2022-06-27T11:37:27.828 INFO:root:teuthology version: 1.1.1.dev376+g082ae7ef 2022-06-27T11:37:27.830 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_vshankar@teuthology --name vshankar-2022-06-27_06:36:09-fs-wip-vshankar-testing-20220627-100931-testing-default-smithi --verbose --archive /home/teuthworker/archive/vshankar-2022-06-27_06:36:09-fs-wip-vshankar-testing-20220627-100931-testing-default-smithi/6900509 --description fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} -- /home/teuthworker/archive/vshankar-2022-06-27_06:36:09-fs-wip-vshankar-testing-20220627-100931-testing-default-smithi/6900509/orig.config.yaml 2022-06-27T11:37:27.967 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2022-06-27T11:37:28.093 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/vshankar-2022-06-27_06:36:09-fs-wip-vshankar-testing-20220627-100931-testing-default-smithi/6900509 branch: wip-vshankar-testing-20220627-100931 description: fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} email: vshankar@redhat.com first_in_suite: false job_id: '6900509' kernel: client: sha1: distro flavor: default kdb: true sha1: 2d518d4a2a7d878b26d7f76b5a3474652fc358a7 last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: vshankar-2022-06-27_06:36:09-fs-wip-vshankar-testing-20220627-100931-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: wip-vshankar-testing-20220627-100931 ceph: cephfs: session_timeout: 300 conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds oft prefetch dirfrags: true mds op complaint time: 180 mds verify scatter: true mds_dir_prefetch: false mds_sleep_rank_change: 5000000.0 mon client directed command retry: 5 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: client mds osd ms inject socket failures: 2500 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon client directed command retry: 5 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: client mds ms inject socket failures: 2500 osd blocked scrub grace period: 3600 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - Replacing daemon mds sha1: 5292901bc999daa0d7ac128764a3f27c4233c252 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: 5292901bc999daa0d7ac128764a3f27c4233c252 kclient: syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-vshankar-testing-20220627-100931 sha1: 5292901bc999daa0d7ac128764a3f27c4233c252 owner: scheduled_vshankar@teuthology priority: 200 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - mds.e - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - mds.d - mds.f - osd.4 - osd.5 - osd.6 - osd.7 sha1: 5292901bc999daa0d7ac128764a3f27c4233c252 sleep_before_teardown: 0 suite: fs suite_branch: wip-vshankar-testing-20220627-100931 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_5292901bc999daa0d7ac128764a3f27c4233c252/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: 5292901bc999daa0d7ac128764a3f27c4233c252 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi195.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDUd5V1StgPqlYh4mQ5I+nK3m+VDwD4/KsWOI5vsXndns8OJ/spYxCbIvlHNGQ+8VPjaOStwHqaVku07YGtorTRaEAhPTpngyMUxndQXOMv4IdSsiAkRj5vHIoTSXqZdaZvG8PBXCONrFecxRKNMh4OMlyRc19bMwXk3ZaG4okDM7H9x6AEkD/gJ+wsjIH7wEDBuvBmmnr2AcxE1ifHHzVq4p4UrkCG6CPULh6qeZdBaG0tmDh1Ct2CO/rCvw81fAxgqdabqfwdw332k2hUrw5sZiOX0mj03ogKP6Sfv3FkuS3m49LwIWaOQCzve5kkgmnU5AuP5Vu0oIKGr0Ug/lC62yv57gNw/yfZXd76RbepIzidZ4pqbXR+ISDZq63hIy8bO/0M7DJWzyCcjc6fUWk3cZJZGRTfUtTwbzzhMeKHuluT4NgREJNw/DKoijPjl0LeM7EjAMdAh0gd5ms/ba9bMIQiWNuWho16WdRi3ZoLL/rkdArEcQp66VyvG/Oze3k= smithi202.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC7GVzKcLLYV3HsMIoNStF8xm4lQJPjMDH+9hR2/ORU8frhqQxcLkXOH72LBSKMISvSaN5/pzutYoF8bVGwUJkbfVeZO/I4Q3SNmA3sgDnmcedLduWGiaqSbIVRMU9ykIs2cQgX2caskfFkgMmjXjb1GXz50ktRDgMEfqNVsp5Xz9g+zkyJLfMCgq3LkhIaY3qYJ1v91KWayXTpLXj+eQ0yTQd6BTBADsrj0l1YYzeNUi8BctL5Sw7TMKGUDFAyulfIGeD3FfPBNtp6HZmDt+ZqDBofqdl3JjC3LELxRbFnVRvFEzbX1udzOcfxIpCs6h77V3jDl0heARsPU3CJsL0yb6EQeqjYiE//NGEMRtTxW4m3BfGU3WzYdhBsBq+Q0xa9rX1CSMYe3BN3r+X6uZAiMJgiAgLPd6ZvlHj8nuQ0Cl+jphhsaTV9r3Sep3p4KJamLCA3hbQGQHeGlV9GxJsK8/hiD4UbbuoUaHJjEZvBgEE+j1sdq6BXGlmtu57OX0k= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - ceph: null - kclient: null - mds_thrash: null - workunit: clients: all: - fs/snaps timeout: 6h teuthology_branch: main teuthology_sha1: 082ae7ef4302fa54665ed0a2535e8e254118dcfd timestamp: 2022-06-27_06:36:09 tube: smithi user: vshankar verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.1713 2022-06-27T11:37:28.094 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_5292901bc999daa0d7ac128764a3f27c4233c252/qa; will attempt to use it 2022-06-27T11:37:28.095 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_5292901bc999daa0d7ac128764a3f27c4233c252/qa/tasks 2022-06-27T11:37:28.096 INFO:teuthology.run_tasks:Running task internal.check_packages... 2022-06-27T11:37:28.099 INFO:teuthology.task.internal:Checking packages... 2022-06-27T11:37:28.124 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '5292901bc999daa0d7ac128764a3f27c4233c252' 2022-06-27T11:37:28.124 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2022-06-27T11:37:28.125 INFO:teuthology.packaging:ref: None 2022-06-27T11:37:28.125 INFO:teuthology.packaging:tag: None 2022-06-27T11:37:28.125 INFO:teuthology.packaging:branch: wip-vshankar-testing-20220627-100931 2022-06-27T11:37:28.126 INFO:teuthology.packaging:sha1: 5292901bc999daa0d7ac128764a3f27c4233c252 2022-06-27T11:37:28.126 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-vshankar-testing-20220627-100931 2022-06-27T11:37:28.456 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-13241.g5292901b 2022-06-27T11:37:28.457 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2022-06-27T11:37:28.464 INFO:teuthology.task.internal:no buildpackages task found 2022-06-27T11:37:28.465 INFO:teuthology.run_tasks:Running task internal.save_config... 2022-06-27T11:37:28.483 INFO:teuthology.task.internal:Saving configuration 2022-06-27T11:37:28.529 INFO:teuthology.run_tasks:Running task internal.check_lock... 2022-06-27T11:37:28.541 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-06-27T11:37:28.577 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi195.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2022-06-27_06:36:09-fs-wip-vshankar-testing-20220627-100931-testing-default-smithi/6900509', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-06-27 11:32:23.363777', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBhozg4tOiwW0oFSK3mhDBlDgUX8GnpUbZUCPS3GfLrFt9ILMHkqUSqzsv0GygQmIphsJuW8Nu1lYTjaoJZE1gg='} 2022-06-27T11:37:28.622 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi202.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2022-06-27_06:36:09-fs-wip-vshankar-testing-20220627-100931-testing-default-smithi/6900509', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-06-27 11:32:23.362038', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOovBArCwmYgLOiZFfjG/eoSjNRTjr9lSwhb5R6a8dEkRuGZvzLMMO/nbjYAiYanypFZTp/XHlZnM9fRGU7mScU='} 2022-06-27T11:37:28.622 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2022-06-27T11:37:28.646 INFO:teuthology.task.internal:roles: ubuntu@smithi195.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2022-06-27T11:37:28.647 INFO:teuthology.task.internal:roles: ubuntu@smithi202.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2022-06-27T11:37:28.648 INFO:teuthology.run_tasks:Running task console_log... 2022-06-27T11:37:28.777 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f1c7ca33b70>, signals=[15]) 2022-06-27T11:37:28.780 INFO:teuthology.run_tasks:Running task internal.connect... 2022-06-27T11:37:28.805 INFO:teuthology.task.internal:Opening connections... 2022-06-27T11:37:28.806 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi195.front.sepia.ceph.com 2022-06-27T11:37:28.808 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-06-27T11:37:28.928 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi202.front.sepia.ceph.com 2022-06-27T11:37:28.930 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi202.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-06-27T11:37:29.008 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2022-06-27T11:37:29.018 DEBUG:teuthology.orchestra.run.smithi195:> uname -m 2022-06-27T11:37:29.042 INFO:teuthology.orchestra.run.smithi195.stdout:x86_64 2022-06-27T11:37:29.042 DEBUG:teuthology.orchestra.run.smithi195:> cat /etc/os-release 2022-06-27T11:37:29.103 INFO:teuthology.orchestra.run.smithi195.stdout:NAME="Red Hat Enterprise Linux" 2022-06-27T11:37:29.103 INFO:teuthology.orchestra.run.smithi195.stdout:VERSION="8.6 (Ootpa)" 2022-06-27T11:37:29.103 INFO:teuthology.orchestra.run.smithi195.stdout:ID="rhel" 2022-06-27T11:37:29.104 INFO:teuthology.orchestra.run.smithi195.stdout:ID_LIKE="fedora" 2022-06-27T11:37:29.104 INFO:teuthology.orchestra.run.smithi195.stdout:VERSION_ID="8.6" 2022-06-27T11:37:29.104 INFO:teuthology.orchestra.run.smithi195.stdout:PLATFORM_ID="platform:el8" 2022-06-27T11:37:29.105 INFO:teuthology.orchestra.run.smithi195.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2022-06-27T11:37:29.106 INFO:teuthology.orchestra.run.smithi195.stdout:ANSI_COLOR="0;31" 2022-06-27T11:37:29.106 INFO:teuthology.orchestra.run.smithi195.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2022-06-27T11:37:29.106 INFO:teuthology.orchestra.run.smithi195.stdout:HOME_URL="https://www.redhat.com/" 2022-06-27T11:37:29.106 INFO:teuthology.orchestra.run.smithi195.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2022-06-27T11:37:29.107 INFO:teuthology.orchestra.run.smithi195.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-06-27T11:37:29.107 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:37:29.107 INFO:teuthology.orchestra.run.smithi195.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2022-06-27T11:37:29.107 INFO:teuthology.orchestra.run.smithi195.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2022-06-27T11:37:29.108 INFO:teuthology.orchestra.run.smithi195.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2022-06-27T11:37:29.108 INFO:teuthology.orchestra.run.smithi195.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2022-06-27T11:37:29.109 INFO:teuthology.lock.ops:Updating smithi195.front.sepia.ceph.com on lock server 2022-06-27T11:37:29.137 DEBUG:teuthology.orchestra.run.smithi202:> uname -m 2022-06-27T11:37:29.185 INFO:teuthology.orchestra.run.smithi202.stdout:x86_64 2022-06-27T11:37:29.186 DEBUG:teuthology.orchestra.run.smithi202:> cat /etc/os-release 2022-06-27T11:37:29.252 INFO:teuthology.orchestra.run.smithi202.stdout:NAME="Red Hat Enterprise Linux" 2022-06-27T11:37:29.252 INFO:teuthology.orchestra.run.smithi202.stdout:VERSION="8.6 (Ootpa)" 2022-06-27T11:37:29.253 INFO:teuthology.orchestra.run.smithi202.stdout:ID="rhel" 2022-06-27T11:37:29.253 INFO:teuthology.orchestra.run.smithi202.stdout:ID_LIKE="fedora" 2022-06-27T11:37:29.254 INFO:teuthology.orchestra.run.smithi202.stdout:VERSION_ID="8.6" 2022-06-27T11:37:29.254 INFO:teuthology.orchestra.run.smithi202.stdout:PLATFORM_ID="platform:el8" 2022-06-27T11:37:29.255 INFO:teuthology.orchestra.run.smithi202.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2022-06-27T11:37:29.255 INFO:teuthology.orchestra.run.smithi202.stdout:ANSI_COLOR="0;31" 2022-06-27T11:37:29.256 INFO:teuthology.orchestra.run.smithi202.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2022-06-27T11:37:29.256 INFO:teuthology.orchestra.run.smithi202.stdout:HOME_URL="https://www.redhat.com/" 2022-06-27T11:37:29.257 INFO:teuthology.orchestra.run.smithi202.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2022-06-27T11:37:29.257 INFO:teuthology.orchestra.run.smithi202.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-06-27T11:37:29.258 INFO:teuthology.orchestra.run.smithi202.stdout: 2022-06-27T11:37:29.258 INFO:teuthology.orchestra.run.smithi202.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2022-06-27T11:37:29.259 INFO:teuthology.orchestra.run.smithi202.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2022-06-27T11:37:29.259 INFO:teuthology.orchestra.run.smithi202.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2022-06-27T11:37:29.259 INFO:teuthology.orchestra.run.smithi202.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2022-06-27T11:37:29.261 INFO:teuthology.lock.ops:Updating smithi202.front.sepia.ceph.com on lock server 2022-06-27T11:37:29.291 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2022-06-27T11:37:29.309 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2022-06-27T11:37:29.317 INFO:teuthology.task.internal:Checking for old test directory... 2022-06-27T11:37:29.317 DEBUG:teuthology.orchestra.run.smithi195:> test '!' -e /home/ubuntu/cephtest 2022-06-27T11:37:29.320 DEBUG:teuthology.orchestra.run.smithi202:> test '!' -e /home/ubuntu/cephtest 2022-06-27T11:37:29.341 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2022-06-27T11:37:29.352 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2022-06-27T11:37:29.353 DEBUG:teuthology.orchestra.run.smithi195:> test -z $(ls -A /var/lib/ceph) 2022-06-27T11:37:29.381 DEBUG:teuthology.orchestra.run.smithi202:> test -z $(ls -A /var/lib/ceph) 2022-06-27T11:37:29.428 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2022-06-27T11:37:29.526 INFO:teuthology.run_tasks:Running task kernel... 2022-06-27T11:37:29.557 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': True, 'sha1': '2d518d4a2a7d878b26d7f76b5a3474652fc358a7'} 2022-06-27T11:37:29.560 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2022-06-27T11:37:29.561 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2022-06-27T11:37:29.562 DEBUG:teuthology.orchestra.run.smithi195:> test -f /run/.containerenv -o -f /.dockerenv 2022-06-27T11:37:29.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-27T11:37:29.592 DEBUG:teuthology.orchestra.run.smithi195:> uname -r 2022-06-27T11:37:29.669 INFO:teuthology.orchestra.run.smithi195.stdout:4.18.0-372.9.1.el8.x86_64 2022-06-27T11:37:29.670 INFO:teuthology.task.kernel:Running kernel on smithi195: 4.18.0-372.9.1.el8.x86_64 2022-06-27T11:37:29.671 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum install -y kernel 2022-06-27T11:37:30.702 INFO:teuthology.orchestra.run.smithi195.stdout:Updating Subscription Management repositories. 2022-06-27T11:37:30.703 INFO:teuthology.orchestra.run.smithi195.stdout:Unable to read consumer identity 2022-06-27T11:37:30.787 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:37:30.788 INFO:teuthology.orchestra.run.smithi195.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2022-06-27T11:37:30.788 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:37:34.813 INFO:teuthology.orchestra.run.smithi195.stdout:Copr repo for python3-asyncssh owned by ceph 1.3 kB/s | 3.5 kB 00:02 2022-06-27T11:37:36.931 INFO:teuthology.orchestra.run.smithi195.stdout:Extra Packages for Enterprise Linux 7.9 MB/s | 11 MB 00:01 2022-06-27T11:37:38.798 INFO:teuthology.orchestra.run.smithi195.stdout:lab-extras 288 kB/s | 24 kB 00:00 2022-06-27T11:37:39.396 INFO:teuthology.orchestra.run.smithi195.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2022-06-27T11:37:39.435 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2022-06-27T11:37:39.436 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2022-06-27T11:37:39.436 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2022-06-27T11:37:39.533 DEBUG:teuthology.orchestra.run.smithi195:> echo no | sudo yum reinstall kernel || true 2022-06-27T11:37:39.864 INFO:teuthology.orchestra.run.smithi195.stdout:Updating Subscription Management repositories. 2022-06-27T11:37:39.865 INFO:teuthology.orchestra.run.smithi195.stdout:Unable to read consumer identity 2022-06-27T11:37:39.917 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:37:39.917 INFO:teuthology.orchestra.run.smithi195.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2022-06-27T11:37:39.918 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:37:40.137 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:00:02 ago on Mon 27 Jun 2022 11:37:38 AM UTC. 2022-06-27T11:37:40.174 INFO:teuthology.orchestra.run.smithi195.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2022-06-27T11:37:40.206 INFO:teuthology.orchestra.run.smithi195.stderr:Error: No packages marked for reinstall. 2022-06-27T11:37:40.251 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum reinstall -y kernel || true 2022-06-27T11:37:40.580 INFO:teuthology.orchestra.run.smithi195.stdout:Updating Subscription Management repositories. 2022-06-27T11:37:40.581 INFO:teuthology.orchestra.run.smithi195.stdout:Unable to read consumer identity 2022-06-27T11:37:40.625 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:37:40.625 INFO:teuthology.orchestra.run.smithi195.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2022-06-27T11:37:40.625 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:37:40.840 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:00:02 ago on Mon 27 Jun 2022 11:37:38 AM UTC. 2022-06-27T11:37:40.880 INFO:teuthology.orchestra.run.smithi195.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2022-06-27T11:37:40.900 INFO:teuthology.orchestra.run.smithi195.stderr:Error: No packages marked for reinstall. 2022-06-27T11:37:40.946 DEBUG:teuthology.orchestra.run.smithi195:> rpm -q kernel --last | head -n 1 2022-06-27T11:37:41.027 INFO:teuthology.orchestra.run.smithi195.stdout:kernel-4.18.0-372.9.1.el8.x86_64 Wed 08 Jun 2022 05:46:35 PM UTC 2022-06-27T11:37:41.029 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-372.9.1.el8.x86_64 2022-06-27T11:37:41.030 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2022-06-27T11:37:41.030 DEBUG:teuthology.parallel:result is None 2022-06-27T11:37:41.031 INFO:teuthology.run_tasks:Running task internal.base... 2022-06-27T11:37:41.053 INFO:teuthology.task.internal:Creating test directory... 2022-06-27T11:37:41.060 DEBUG:teuthology.orchestra.run.smithi195:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-06-27T11:37:41.065 DEBUG:teuthology.orchestra.run.smithi202:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-06-27T11:37:41.091 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2022-06-27T11:37:41.104 INFO:teuthology.run_tasks:Running task internal.archive... 2022-06-27T11:37:41.115 INFO:teuthology.task.internal:Creating archive directory... 2022-06-27T11:37:41.115 DEBUG:teuthology.orchestra.run.smithi195:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-06-27T11:37:41.124 DEBUG:teuthology.orchestra.run.smithi202:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-06-27T11:37:41.174 INFO:teuthology.run_tasks:Running task internal.coredump... 2022-06-27T11:37:41.185 INFO:teuthology.task.internal:Enabling coredump saving... 2022-06-27T11:37:41.185 DEBUG:teuthology.orchestra.run.smithi202:> test -f /run/.containerenv -o -f /.dockerenv 2022-06-27T11:37:41.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-27T11:37:41.227 DEBUG:teuthology.orchestra.run.smithi195:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-06-27T11:37:41.230 DEBUG:teuthology.orchestra.run.smithi202:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-06-27T11:37:41.261 INFO:teuthology.orchestra.run.smithi195.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-06-27T11:37:41.280 INFO:teuthology.orchestra.run.smithi195.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-06-27T11:37:41.309 INFO:teuthology.orchestra.run.smithi202.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-06-27T11:37:41.324 INFO:teuthology.orchestra.run.smithi202.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-06-27T11:37:41.326 INFO:teuthology.run_tasks:Running task internal.sudo... 2022-06-27T11:37:41.335 INFO:teuthology.task.internal:Configuring sudo... 2022-06-27T11:37:41.335 DEBUG:teuthology.orchestra.run.smithi195:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-06-27T11:37:41.339 DEBUG:teuthology.orchestra.run.smithi202:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-06-27T11:37:41.406 INFO:teuthology.run_tasks:Running task internal.syslog... 2022-06-27T11:37:41.426 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2022-06-27T11:37:41.427 DEBUG:teuthology.orchestra.run.smithi195:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-06-27T11:37:41.429 DEBUG:teuthology.orchestra.run.smithi202:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-06-27T11:37:41.475 DEBUG:teuthology.orchestra.run.smithi195:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-06-27T11:37:41.535 DEBUG:teuthology.orchestra.run.smithi195:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2022-06-27T11:37:41.585 DEBUG:teuthology.orchestra.run.smithi195:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-06-27T11:37:41.691 DEBUG:teuthology.orchestra.run.smithi195:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2022-06-27T11:37:41.723 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2022-06-27T11:37:41.728 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-06-27T11:37:41.791 DEBUG:teuthology.orchestra.run.smithi202:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-06-27T11:37:41.841 DEBUG:teuthology.orchestra.run.smithi202:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2022-06-27T11:37:41.937 DEBUG:teuthology.orchestra.run.smithi202:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-06-27T11:37:41.991 DEBUG:teuthology.orchestra.run.smithi202:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2022-06-27T11:37:42.037 DEBUG:teuthology.orchestra.run.smithi202:> set -ex 2022-06-27T11:37:42.038 DEBUG:teuthology.orchestra.run.smithi202:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-06-27T11:37:42.110 DEBUG:teuthology.orchestra.run.smithi195:> sudo service rsyslog restart 2022-06-27T11:37:42.116 DEBUG:teuthology.orchestra.run.smithi202:> sudo service rsyslog restart 2022-06-27T11:37:42.158 INFO:teuthology.orchestra.run.smithi195.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2022-06-27T11:37:42.218 INFO:teuthology.orchestra.run.smithi202.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2022-06-27T11:37:42.589 INFO:teuthology.run_tasks:Running task internal.timer... 2022-06-27T11:37:42.601 INFO:teuthology.task.internal:Starting timer... 2022-06-27T11:37:42.601 INFO:teuthology.run_tasks:Running task pcp... 2022-06-27T11:37:42.619 INFO:teuthology.run_tasks:Running task selinux... 2022-06-27T11:37:42.641 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2022-06-27T11:37:42.643 DEBUG:teuthology.orchestra.run.smithi195:> sudo service auditd rotate 2022-06-27T11:37:42.721 INFO:teuthology.orchestra.run.smithi195.stdout:Rotating logs: 2022-06-27T11:37:42.723 DEBUG:teuthology.orchestra.run.smithi202:> sudo service auditd rotate 2022-06-27T11:37:42.808 INFO:teuthology.orchestra.run.smithi202.stdout:Rotating logs: 2022-06-27T11:37:42.810 DEBUG:teuthology.task.selinux:Getting current SELinux state 2022-06-27T11:37:42.811 DEBUG:teuthology.orchestra.run.smithi195:> /usr/sbin/getenforce 2022-06-27T11:37:42.841 INFO:teuthology.orchestra.run.smithi195.stdout:Permissive 2022-06-27T11:37:42.842 DEBUG:teuthology.orchestra.run.smithi202:> /usr/sbin/getenforce 2022-06-27T11:37:42.873 INFO:teuthology.orchestra.run.smithi202.stdout:Permissive 2022-06-27T11:37:42.874 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi195.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi202.front.sepia.ceph.com': 'permissive'} 2022-06-27T11:37:42.875 DEBUG:teuthology.orchestra.run.smithi195:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|comm="sss_cache"\|context=system_u:system_r:NetworkManager_dispatcher_t:s0\)' 2022-06-27T11:37:42.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-27T11:37:42.912 DEBUG:teuthology.orchestra.run.smithi202:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|comm="sss_cache"\|context=system_u:system_r:NetworkManager_dispatcher_t:s0\)' 2022-06-27T11:37:42.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-27T11:37:42.944 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2022-06-27T11:37:42.945 DEBUG:teuthology.orchestra.run.smithi195:> sudo /usr/sbin/setenforce permissive 2022-06-27T11:37:42.980 DEBUG:teuthology.orchestra.run.smithi202:> sudo /usr/sbin/setenforce permissive 2022-06-27T11:37:43.011 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2022-06-27T11:37:43.025 DEBUG:teuthology.repo_utils:Setting repo remote to git://git.ceph.com/git/ceph-cm-ansible.git 2022-06-27T11:37:43.035 INFO:teuthology.repo_utils:Fetching main from origin 2022-06-27T11:37:43.080 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_main to origin/main 2022-06-27T11:37:43.117 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2022-06-27T11:37:43.120 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi195.front.sepia.ceph.com,smithi202.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_main/cephlab.yml 2022-06-27T11:44:50.138 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi195.front.sepia.ceph.com'), Remote(name='ubuntu@smithi202.front.sepia.ceph.com')] 2022-06-27T11:44:50.142 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi195.front.sepia.ceph.com' 2022-06-27T11:44:50.145 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-06-27T11:44:50.240 DEBUG:teuthology.orchestra.run.smithi195:> true 2022-06-27T11:44:50.308 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi202.front.sepia.ceph.com' 2022-06-27T11:44:50.309 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi202.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-06-27T11:44:50.398 DEBUG:teuthology.orchestra.run.smithi202:> true 2022-06-27T11:44:50.464 INFO:teuthology.run_tasks:Running task clock... 2022-06-27T11:44:50.479 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2022-06-27T11:44:50.479 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-06-27T11:44:50.482 DEBUG:teuthology.orchestra.run.smithi195:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-06-27T11:44:50.484 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-06-27T11:44:50.485 DEBUG:teuthology.orchestra.run.smithi202:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-06-27T11:44:50.512 INFO:teuthology.orchestra.run.smithi195.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2022-06-27T11:44:50.526 INFO:teuthology.orchestra.run.smithi195.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2022-06-27T11:44:50.540 INFO:teuthology.orchestra.run.smithi202.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2022-06-27T11:44:50.556 INFO:teuthology.orchestra.run.smithi202.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2022-06-27T11:44:50.593 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: ntpd: command not found 2022-06-27T11:44:50.604 INFO:teuthology.orchestra.run.smithi195.stdout:506 Cannot talk to daemon 2022-06-27T11:44:50.618 INFO:teuthology.orchestra.run.smithi195.stderr:Failed to start ntp.service: Unit ntp.service not found. 2022-06-27T11:44:50.621 INFO:teuthology.orchestra.run.smithi202.stderr:sudo: ntpd: command not found 2022-06-27T11:44:50.632 INFO:teuthology.orchestra.run.smithi195.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2022-06-27T11:44:50.634 INFO:teuthology.orchestra.run.smithi202.stdout:506 Cannot talk to daemon 2022-06-27T11:44:50.649 INFO:teuthology.orchestra.run.smithi202.stderr:Failed to start ntp.service: Unit ntp.service not found. 2022-06-27T11:44:50.663 INFO:teuthology.orchestra.run.smithi202.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2022-06-27T11:44:50.711 INFO:teuthology.orchestra.run.smithi195.stderr:bash: ntpq: command not found 2022-06-27T11:44:50.715 INFO:teuthology.orchestra.run.smithi195.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2022-06-27T11:44:50.716 INFO:teuthology.orchestra.run.smithi195.stdout:=============================================================================== 2022-06-27T11:44:50.716 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-06-27T11:44:50.717 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-06-27T11:44:50.717 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-06-27T11:44:50.718 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-06-27T11:44:50.756 INFO:teuthology.orchestra.run.smithi202.stderr:bash: ntpq: command not found 2022-06-27T11:44:50.757 INFO:teuthology.orchestra.run.smithi202.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2022-06-27T11:44:50.758 INFO:teuthology.orchestra.run.smithi202.stdout:=============================================================================== 2022-06-27T11:44:50.758 INFO:teuthology.orchestra.run.smithi202.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-06-27T11:44:50.758 INFO:teuthology.orchestra.run.smithi202.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-06-27T11:44:50.759 INFO:teuthology.orchestra.run.smithi202.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-06-27T11:44:50.759 INFO:teuthology.orchestra.run.smithi202.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-06-27T11:44:50.761 INFO:teuthology.run_tasks:Running task install... 2022-06-27T11:44:50.775 DEBUG:teuthology.task.install:project ceph 2022-06-27T11:44:50.776 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5292901bc999daa0d7ac128764a3f27c4233c252'}} 2022-06-27T11:44:50.776 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'flavor': 'default', 'sha1': '5292901bc999daa0d7ac128764a3f27c4233c252'} 2022-06-27T11:44:50.776 INFO:teuthology.task.install:Using flavor: default 2022-06-27T11:44:50.783 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2022-06-27T11:44:50.784 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2022-06-27T11:44:50.785 DEBUG:teuthology.orchestra.run.smithi195:> sudo lsb_release -is 2022-06-27T11:44:50.832 INFO:teuthology.orchestra.run.smithi195.stdout:RedHatEnterprise 2022-06-27T11:44:50.833 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-06-27T11:44:50.834 DEBUG:teuthology.orchestra.run.smithi202:> sudo lsb_release -is 2022-06-27T11:44:50.835 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5292901bc999daa0d7ac128764a3f27c4233c252', 'tag': None, 'wait_for_package': False} 2022-06-27T11:44:50.836 DEBUG:teuthology.orchestra.run.smithi195:> sudo lsb_release -is 2022-06-27T11:44:50.881 INFO:teuthology.orchestra.run.smithi202.stdout:RedHatEnterprise 2022-06-27T11:44:50.882 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-06-27T11:44:50.882 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5292901bc999daa0d7ac128764a3f27c4233c252', 'tag': None, 'wait_for_package': False} 2022-06-27T11:44:50.883 DEBUG:teuthology.orchestra.run.smithi202:> sudo lsb_release -is 2022-06-27T11:44:50.917 INFO:teuthology.orchestra.run.smithi195.stdout:RedHatEnterprise 2022-06-27T11:44:50.918 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-06-27T11:44:50.919 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=5292901bc999daa0d7ac128764a3f27c4233c252 2022-06-27T11:44:50.964 INFO:teuthology.orchestra.run.smithi202.stdout:RedHatEnterprise 2022-06-27T11:44:50.966 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-06-27T11:44:50.966 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=5292901bc999daa0d7ac128764a3f27c4233c252 2022-06-27T11:44:51.086 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20220627-100931/5292901bc999daa0d7ac128764a3f27c4233c252/centos/8/flavors/default/ 2022-06-27T11:44:51.086 INFO:teuthology.task.install.rpm:Package version is 17.0.0-13241.g5292901b 2022-06-27T11:44:51.127 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20220627-100931/5292901bc999daa0d7ac128764a3f27c4233c252/centos/8/flavors/default/ 2022-06-27T11:44:51.127 INFO:teuthology.task.install.rpm:Package version is 17.0.0-13241.g5292901b 2022-06-27T11:44:51.219 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20220627-100931/5292901bc999daa0d7ac128764a3f27c4233c252/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20220627-100931/5292901bc999daa0d7ac128764a3f27c4233c252/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20220627-100931/5292901bc999daa0d7ac128764a3f27c4233c252/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2022-06-27T11:44:51.220 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2022-06-27T11:44:51.220 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/etc/yum.repos.d/ceph.repo 2022-06-27T11:44:51.243 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20220627-100931/5292901bc999daa0d7ac128764a3f27c4233c252/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20220627-100931/5292901bc999daa0d7ac128764a3f27c4233c252/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20220627-100931/5292901bc999daa0d7ac128764a3f27c4233c252/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2022-06-27T11:44:51.244 DEBUG:teuthology.orchestra.run.smithi202:> set -ex 2022-06-27T11:44:51.244 DEBUG:teuthology.orchestra.run.smithi202:> sudo dd of=/etc/yum.repos.d/ceph.repo 2022-06-27T11:44:51.248 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel on remote rpm x86_64 2022-06-27T11:44:51.249 DEBUG:teuthology.orchestra.run.smithi195:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5292901bc999daa0d7ac128764a3f27c4233c252/;g' /etc/yum.repos.d/ceph.repo ; fi 2022-06-27T11:44:51.272 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel on remote rpm x86_64 2022-06-27T11:44:51.273 DEBUG:teuthology.orchestra.run.smithi202:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5292901bc999daa0d7ac128764a3f27c4233c252/;g' /etc/yum.repos.d/ceph.repo ; fi 2022-06-27T11:44:51.317 DEBUG:teuthology.orchestra.run.smithi195:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2022-06-27T11:44:51.339 DEBUG:teuthology.orchestra.run.smithi202:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2022-06-27T11:44:51.394 DEBUG:teuthology.orchestra.run.smithi195:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2022-06-27T11:44:51.416 DEBUG:teuthology.orchestra.run.smithi202:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2022-06-27T11:44:51.459 INFO:teuthology.orchestra.run.smithi195.stdout:check_obsoletes = 1 2022-06-27T11:44:51.461 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum clean all 2022-06-27T11:44:51.483 INFO:teuthology.orchestra.run.smithi202.stdout:check_obsoletes = 1 2022-06-27T11:44:51.485 DEBUG:teuthology.orchestra.run.smithi202:> sudo yum clean all 2022-06-27T11:44:51.833 INFO:teuthology.orchestra.run.smithi195.stdout:Updating Subscription Management repositories. 2022-06-27T11:44:51.857 INFO:teuthology.orchestra.run.smithi202.stdout:Updating Subscription Management repositories. 2022-06-27T11:44:52.063 INFO:teuthology.orchestra.run.smithi195.stdout:43 files removed 2022-06-27T11:44:52.084 INFO:teuthology.orchestra.run.smithi202.stdout:43 files removed 2022-06-27T11:44:52.113 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel 2022-06-27T11:44:52.131 DEBUG:teuthology.orchestra.run.smithi202:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel 2022-06-27T11:44:52.450 INFO:teuthology.orchestra.run.smithi195.stdout:Updating Subscription Management repositories. 2022-06-27T11:44:52.461 INFO:teuthology.orchestra.run.smithi202.stdout:Updating Subscription Management repositories. 2022-06-27T11:44:53.121 INFO:teuthology.orchestra.run.smithi195.stdout:ceph packages for x86_64 240 kB/s | 80 kB 00:00 2022-06-27T11:44:53.174 INFO:teuthology.orchestra.run.smithi202.stdout:ceph packages for x86_64 209 kB/s | 80 kB 00:00 2022-06-27T11:44:53.413 INFO:teuthology.orchestra.run.smithi195.stdout:ceph noarch packages 35 kB/s | 9.1 kB 00:00 2022-06-27T11:44:53.450 INFO:teuthology.orchestra.run.smithi202.stdout:ceph noarch packages 37 kB/s | 9.1 kB 00:00 2022-06-27T11:44:53.681 INFO:teuthology.orchestra.run.smithi195.stdout:ceph source packages 6.1 kB/s | 1.5 kB 00:00 2022-06-27T11:44:53.727 INFO:teuthology.orchestra.run.smithi202.stdout:ceph source packages 5.9 kB/s | 1.5 kB 00:00 2022-06-27T11:44:56.318 INFO:teuthology.orchestra.run.smithi202.stdout:Copr repo for python3-asyncssh owned by ceph 1.3 kB/s | 3.5 kB 00:02 2022-06-27T11:44:56.431 INFO:teuthology.orchestra.run.smithi195.stdout:Copr repo for python3-asyncssh owned by ceph 1.3 kB/s | 3.5 kB 00:02 2022-06-27T11:44:57.472 INFO:teuthology.orchestra.run.smithi202.stdout:Extra Packages for Enterprise Linux 10 MB/s | 11 MB 00:01 2022-06-27T11:44:58.573 INFO:teuthology.orchestra.run.smithi195.stdout:Extra Packages for Enterprise Linux 5.3 MB/s | 11 MB 00:02 2022-06-27T11:45:00.060 INFO:teuthology.orchestra.run.smithi202.stdout:lab-extras 301 kB/s | 24 kB 00:00 2022-06-27T11:45:00.870 INFO:teuthology.orchestra.run.smithi202.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 60 MB/s | 45 MB 00:00 2022-06-27T11:45:01.048 INFO:teuthology.orchestra.run.smithi195.stdout:lab-extras 543 kB/s | 24 kB 00:00 2022-06-27T11:45:01.826 INFO:teuthology.orchestra.run.smithi195.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 63 MB/s | 45 MB 00:00 2022-06-27T11:45:08.443 INFO:teuthology.orchestra.run.smithi202.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 65 MB/s | 48 MB 00:00 2022-06-27T11:45:09.326 INFO:teuthology.orchestra.run.smithi195.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 66 MB/s | 48 MB 00:00 2022-06-27T11:45:18.718 INFO:teuthology.orchestra.run.smithi202.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2022-06-27T11:45:18.719 INFO:teuthology.orchestra.run.smithi202.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2022-06-27T11:45:18.732 INFO:teuthology.orchestra.run.smithi202.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2022-06-27T11:45:18.733 INFO:teuthology.orchestra.run.smithi202.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2022-06-27T11:45:18.734 INFO:teuthology.orchestra.run.smithi202.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2022-06-27T11:45:18.735 INFO:teuthology.orchestra.run.smithi202.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2022-06-27T11:45:18.738 INFO:teuthology.orchestra.run.smithi202.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2022-06-27T11:45:18.741 INFO:teuthology.orchestra.run.smithi202.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2022-06-27T11:45:18.742 INFO:teuthology.orchestra.run.smithi202.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2022-06-27T11:45:18.856 INFO:teuthology.orchestra.run.smithi202.stdout:Dependencies resolved. 2022-06-27T11:45:18.868 INFO:teuthology.orchestra.run.smithi202.stdout:======================================================================================================================= 2022-06-27T11:45:18.868 INFO:teuthology.orchestra.run.smithi202.stdout: Package Arch Version Repository Size 2022-06-27T11:45:18.868 INFO:teuthology.orchestra.run.smithi202.stdout:======================================================================================================================= 2022-06-27T11:45:18.869 INFO:teuthology.orchestra.run.smithi202.stdout:Installing: 2022-06-27T11:45:18.869 INFO:teuthology.orchestra.run.smithi202.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2022-06-27T11:45:18.870 INFO:teuthology.orchestra.run.smithi202.stdout: ceph x86_64 2:17.0.0-13241.g5292901b.el8 ceph 6.5 k 2022-06-27T11:45:18.870 INFO:teuthology.orchestra.run.smithi202.stdout: ceph-base x86_64 2:17.0.0-13241.g5292901b.el8 ceph 5.3 M 2022-06-27T11:45:18.870 INFO:teuthology.orchestra.run.smithi202.stdout: ceph-fuse x86_64 2:17.0.0-13241.g5292901b.el8 ceph 883 k 2022-06-27T11:45:18.871 INFO:teuthology.orchestra.run.smithi202.stdout: ceph-immutable-object-cache x86_64 2:17.0.0-13241.g5292901b.el8 ceph 169 k 2022-06-27T11:45:18.871 INFO:teuthology.orchestra.run.smithi202.stdout: ceph-mgr x86_64 2:17.0.0-13241.g5292901b.el8 ceph 1.5 M 2022-06-27T11:45:18.873 INFO:teuthology.orchestra.run.smithi202.stdout: ceph-mgr-cephadm noarch 2:17.0.0-13241.g5292901b.el8 ceph-noarch 111 k 2022-06-27T11:45:18.873 INFO:teuthology.orchestra.run.smithi202.stdout: ceph-mgr-dashboard noarch 2:17.0.0-13241.g5292901b.el8 ceph-noarch 1.6 M 2022-06-27T11:45:18.873 INFO:teuthology.orchestra.run.smithi202.stdout: ceph-mgr-diskprediction-local noarch 2:17.0.0-13241.g5292901b.el8 ceph-noarch 7.4 M 2022-06-27T11:45:18.874 INFO:teuthology.orchestra.run.smithi202.stdout: ceph-mgr-rook noarch 2:17.0.0-13241.g5292901b.el8 ceph-noarch 47 k 2022-06-27T11:45:18.874 INFO:teuthology.orchestra.run.smithi202.stdout: ceph-radosgw x86_64 2:17.0.0-13241.g5292901b.el8 ceph 17 M 2022-06-27T11:45:18.874 INFO:teuthology.orchestra.run.smithi202.stdout: ceph-test x86_64 2:17.0.0-13241.g5292901b.el8 ceph 53 M 2022-06-27T11:45:18.874 INFO:teuthology.orchestra.run.smithi202.stdout: ceph-volume noarch 2:17.0.0-13241.g5292901b.el8 ceph-noarch 259 k 2022-06-27T11:45:18.875 INFO:teuthology.orchestra.run.smithi202.stdout: cephadm noarch 2:17.0.0-13241.g5292901b.el8 ceph-noarch 79 k 2022-06-27T11:45:18.875 INFO:teuthology.orchestra.run.smithi202.stdout: cephfs-mirror x86_64 2:17.0.0-13241.g5292901b.el8 ceph 273 k 2022-06-27T11:45:18.875 INFO:teuthology.orchestra.run.smithi202.stdout: cephfs-top noarch 2:17.0.0-13241.g5292901b.el8 ceph-noarch 14 k 2022-06-27T11:45:18.875 INFO:teuthology.orchestra.run.smithi202.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2022-06-27T11:45:18.876 INFO:teuthology.orchestra.run.smithi202.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2022-06-27T11:45:18.876 INFO:teuthology.orchestra.run.smithi202.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2022-06-27T11:45:18.876 INFO:teuthology.orchestra.run.smithi202.stdout: libcephfs-devel x86_64 2:17.0.0-13241.g5292901b.el8 ceph 30 k 2022-06-27T11:45:18.877 INFO:teuthology.orchestra.run.smithi202.stdout: libcephfs2 x86_64 2:17.0.0-13241.g5292901b.el8 ceph 796 k 2022-06-27T11:45:18.877 INFO:teuthology.orchestra.run.smithi202.stdout: librados-devel x86_64 2:17.0.0-13241.g5292901b.el8 ceph 125 k 2022-06-27T11:45:18.877 INFO:teuthology.orchestra.run.smithi202.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2022-06-27T11:45:18.878 INFO:teuthology.orchestra.run.smithi202.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2022-06-27T11:45:18.878 INFO:teuthology.orchestra.run.smithi202.stdout: openssl-devel x86_64 1:1.1.1k-6.el8_5 rhel-8-for-x86_64-baseos-rpms 2.3 M 2022-06-27T11:45:18.878 INFO:teuthology.orchestra.run.smithi202.stdout: python3-cephfs x86_64 2:17.0.0-13241.g5292901b.el8 ceph 178 k 2022-06-27T11:45:18.878 INFO:teuthology.orchestra.run.smithi202.stdout: python3-rados x86_64 2:17.0.0-13241.g5292901b.el8 ceph 366 k 2022-06-27T11:45:18.879 INFO:teuthology.orchestra.run.smithi202.stdout: python3-rbd x86_64 2:17.0.0-13241.g5292901b.el8 ceph 346 k 2022-06-27T11:45:18.879 INFO:teuthology.orchestra.run.smithi202.stdout: python3-rgw x86_64 2:17.0.0-13241.g5292901b.el8 ceph 109 k 2022-06-27T11:45:18.879 INFO:teuthology.orchestra.run.smithi202.stdout: rbd-fuse x86_64 2:17.0.0-13241.g5292901b.el8 ceph 93 k 2022-06-27T11:45:18.879 INFO:teuthology.orchestra.run.smithi202.stdout: rbd-mirror x86_64 2:17.0.0-13241.g5292901b.el8 ceph 3.8 M 2022-06-27T11:45:18.880 INFO:teuthology.orchestra.run.smithi202.stdout: rbd-nbd x86_64 2:17.0.0-13241.g5292901b.el8 ceph 175 k 2022-06-27T11:45:18.880 INFO:teuthology.orchestra.run.smithi202.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2022-06-27T11:45:18.880 INFO:teuthology.orchestra.run.smithi202.stdout:Upgrading: 2022-06-27T11:45:18.880 INFO:teuthology.orchestra.run.smithi202.stdout: librados2 x86_64 2:17.0.0-13241.g5292901b.el8 ceph 3.6 M 2022-06-27T11:45:18.881 INFO:teuthology.orchestra.run.smithi202.stdout: librbd1 x86_64 2:17.0.0-13241.g5292901b.el8 ceph 3.9 M 2022-06-27T11:45:18.881 INFO:teuthology.orchestra.run.smithi202.stdout:Installing dependencies: 2022-06-27T11:45:18.881 INFO:teuthology.orchestra.run.smithi202.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2022-06-27T11:45:18.881 INFO:teuthology.orchestra.run.smithi202.stdout: ceph-common x86_64 2:17.0.0-13241.g5292901b.el8 ceph 24 M 2022-06-27T11:45:18.881 INFO:teuthology.orchestra.run.smithi202.stdout: ceph-grafana-dashboards noarch 2:17.0.0-13241.g5292901b.el8 ceph-noarch 23 k 2022-06-27T11:45:18.882 INFO:teuthology.orchestra.run.smithi202.stdout: ceph-mds x86_64 2:17.0.0-13241.g5292901b.el8 ceph 2.3 M 2022-06-27T11:45:18.882 INFO:teuthology.orchestra.run.smithi202.stdout: ceph-mgr-modules-core noarch 2:17.0.0-13241.g5292901b.el8 ceph-noarch 230 k 2022-06-27T11:45:18.882 INFO:teuthology.orchestra.run.smithi202.stdout: ceph-mon x86_64 2:17.0.0-13241.g5292901b.el8 ceph 3.9 M 2022-06-27T11:45:18.882 INFO:teuthology.orchestra.run.smithi202.stdout: ceph-osd x86_64 2:17.0.0-13241.g5292901b.el8 ceph 17 M 2022-06-27T11:45:18.883 INFO:teuthology.orchestra.run.smithi202.stdout: ceph-prometheus-alerts noarch 2:17.0.0-13241.g5292901b.el8 ceph-noarch 15 k 2022-06-27T11:45:18.883 INFO:teuthology.orchestra.run.smithi202.stdout: ceph-selinux x86_64 2:17.0.0-13241.g5292901b.el8 ceph 24 k 2022-06-27T11:45:18.883 INFO:teuthology.orchestra.run.smithi202.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2022-06-27T11:45:18.883 INFO:teuthology.orchestra.run.smithi202.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2022-06-27T11:45:18.884 INFO:teuthology.orchestra.run.smithi202.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2022-06-27T11:45:18.884 INFO:teuthology.orchestra.run.smithi202.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2022-06-27T11:45:18.884 INFO:teuthology.orchestra.run.smithi202.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2022-06-27T11:45:18.884 INFO:teuthology.orchestra.run.smithi202.stdout: libcephsqlite x86_64 2:17.0.0-13241.g5292901b.el8 ceph 170 k 2022-06-27T11:45:18.885 INFO:teuthology.orchestra.run.smithi202.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2022-06-27T11:45:18.885 INFO:teuthology.orchestra.run.smithi202.stdout: libgfortran x86_64 8.5.0-10.el8 rhel-8-for-x86_64-baseos-rpms 643 k 2022-06-27T11:45:18.885 INFO:teuthology.orchestra.run.smithi202.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2022-06-27T11:45:18.885 INFO:teuthology.orchestra.run.smithi202.stdout: libquadmath x86_64 8.5.0-10.el8 rhel-8-for-x86_64-baseos-rpms 171 k 2022-06-27T11:45:18.885 INFO:teuthology.orchestra.run.smithi202.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2022-06-27T11:45:18.886 INFO:teuthology.orchestra.run.smithi202.stdout: libradosstriper1 x86_64 2:17.0.0-13241.g5292901b.el8 ceph 485 k 2022-06-27T11:45:18.886 INFO:teuthology.orchestra.run.smithi202.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2022-06-27T11:45:18.886 INFO:teuthology.orchestra.run.smithi202.stdout: librgw2 x86_64 2:17.0.0-13241.g5292901b.el8 ceph 5.5 M 2022-06-27T11:45:18.887 INFO:teuthology.orchestra.run.smithi202.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2022-06-27T11:45:18.887 INFO:teuthology.orchestra.run.smithi202.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2022-06-27T11:45:18.887 INFO:teuthology.orchestra.run.smithi202.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2022-06-27T11:45:18.887 INFO:teuthology.orchestra.run.smithi202.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2022-06-27T11:45:18.888 INFO:teuthology.orchestra.run.smithi202.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2022-06-27T11:45:18.888 INFO:teuthology.orchestra.run.smithi202.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2022-06-27T11:45:18.888 INFO:teuthology.orchestra.run.smithi202.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2022-06-27T11:45:18.888 INFO:teuthology.orchestra.run.smithi202.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2022-06-27T11:45:18.889 INFO:teuthology.orchestra.run.smithi202.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2022-06-27T11:45:18.889 INFO:teuthology.orchestra.run.smithi202.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2022-06-27T11:45:18.889 INFO:teuthology.orchestra.run.smithi202.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2022-06-27T11:45:18.889 INFO:teuthology.orchestra.run.smithi202.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2022-06-27T11:45:18.890 INFO:teuthology.orchestra.run.smithi202.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2022-06-27T11:45:18.890 INFO:teuthology.orchestra.run.smithi202.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2022-06-27T11:45:18.890 INFO:teuthology.orchestra.run.smithi202.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2022-06-27T11:45:18.890 INFO:teuthology.orchestra.run.smithi202.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2022-06-27T11:45:18.890 INFO:teuthology.orchestra.run.smithi202.stdout: python3-ceph-argparse x86_64 2:17.0.0-13241.g5292901b.el8 ceph 45 k 2022-06-27T11:45:18.891 INFO:teuthology.orchestra.run.smithi202.stdout: python3-ceph-common x86_64 2:17.0.0-13241.g5292901b.el8 ceph 106 k 2022-06-27T11:45:18.891 INFO:teuthology.orchestra.run.smithi202.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2022-06-27T11:45:18.891 INFO:teuthology.orchestra.run.smithi202.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2022-06-27T11:45:18.891 INFO:teuthology.orchestra.run.smithi202.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2022-06-27T11:45:18.892 INFO:teuthology.orchestra.run.smithi202.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2022-06-27T11:45:18.892 INFO:teuthology.orchestra.run.smithi202.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2022-06-27T11:45:18.892 INFO:teuthology.orchestra.run.smithi202.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2022-06-27T11:45:18.892 INFO:teuthology.orchestra.run.smithi202.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2022-06-27T11:45:18.893 INFO:teuthology.orchestra.run.smithi202.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2022-06-27T11:45:18.893 INFO:teuthology.orchestra.run.smithi202.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2022-06-27T11:45:18.893 INFO:teuthology.orchestra.run.smithi202.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2022-06-27T11:45:18.893 INFO:teuthology.orchestra.run.smithi202.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2022-06-27T11:45:18.894 INFO:teuthology.orchestra.run.smithi202.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2022-06-27T11:45:18.894 INFO:teuthology.orchestra.run.smithi202.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2022-06-27T11:45:18.894 INFO:teuthology.orchestra.run.smithi202.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2022-06-27T11:45:18.895 INFO:teuthology.orchestra.run.smithi202.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2022-06-27T11:45:18.895 INFO:teuthology.orchestra.run.smithi202.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2022-06-27T11:45:18.895 INFO:teuthology.orchestra.run.smithi202.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2022-06-27T11:45:18.895 INFO:teuthology.orchestra.run.smithi202.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2022-06-27T11:45:18.896 INFO:teuthology.orchestra.run.smithi202.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2022-06-27T11:45:18.896 INFO:teuthology.orchestra.run.smithi202.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2022-06-27T11:45:18.896 INFO:teuthology.orchestra.run.smithi202.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2022-06-27T11:45:18.896 INFO:teuthology.orchestra.run.smithi202.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2022-06-27T11:45:18.897 INFO:teuthology.orchestra.run.smithi202.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2022-06-27T11:45:18.897 INFO:teuthology.orchestra.run.smithi202.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2022-06-27T11:45:18.897 INFO:teuthology.orchestra.run.smithi202.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2022-06-27T11:45:18.898 INFO:teuthology.orchestra.run.smithi202.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2022-06-27T11:45:18.898 INFO:teuthology.orchestra.run.smithi202.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2022-06-27T11:45:18.898 INFO:teuthology.orchestra.run.smithi202.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2022-06-27T11:45:18.898 INFO:teuthology.orchestra.run.smithi202.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2022-06-27T11:45:18.899 INFO:teuthology.orchestra.run.smithi202.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2022-06-27T11:45:18.899 INFO:teuthology.orchestra.run.smithi202.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2022-06-27T11:45:18.899 INFO:teuthology.orchestra.run.smithi202.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2022-06-27T11:45:18.900 INFO:teuthology.orchestra.run.smithi202.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2022-06-27T11:45:18.900 INFO:teuthology.orchestra.run.smithi202.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2022-06-27T11:45:18.900 INFO:teuthology.orchestra.run.smithi202.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2022-06-27T11:45:18.901 INFO:teuthology.orchestra.run.smithi202.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2022-06-27T11:45:18.901 INFO:teuthology.orchestra.run.smithi202.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2022-06-27T11:45:18.902 INFO:teuthology.orchestra.run.smithi202.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2022-06-27T11:45:18.902 INFO:teuthology.orchestra.run.smithi202.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2022-06-27T11:45:18.903 INFO:teuthology.orchestra.run.smithi202.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2022-06-27T11:45:18.903 INFO:teuthology.orchestra.run.smithi202.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2022-06-27T11:45:18.904 INFO:teuthology.orchestra.run.smithi202.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2022-06-27T11:45:18.904 INFO:teuthology.orchestra.run.smithi202.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2022-06-27T11:45:18.904 INFO:teuthology.orchestra.run.smithi202.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2022-06-27T11:45:18.905 INFO:teuthology.orchestra.run.smithi202.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2022-06-27T11:45:18.905 INFO:teuthology.orchestra.run.smithi202.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2022-06-27T11:45:18.907 INFO:teuthology.orchestra.run.smithi202.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2022-06-27T11:45:18.907 INFO:teuthology.orchestra.run.smithi202.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2022-06-27T11:45:18.907 INFO:teuthology.orchestra.run.smithi202.stdout: 2022-06-27T11:45:18.907 INFO:teuthology.orchestra.run.smithi202.stdout:Transaction Summary 2022-06-27T11:45:18.908 INFO:teuthology.orchestra.run.smithi202.stdout:======================================================================================================================= 2022-06-27T11:45:18.908 INFO:teuthology.orchestra.run.smithi202.stdout:Install 122 Packages 2022-06-27T11:45:18.908 INFO:teuthology.orchestra.run.smithi202.stdout:Upgrade 2 Packages 2022-06-27T11:45:18.909 INFO:teuthology.orchestra.run.smithi202.stdout: 2022-06-27T11:45:18.910 INFO:teuthology.orchestra.run.smithi202.stdout:Total download size: 204 M 2022-06-27T11:45:18.910 INFO:teuthology.orchestra.run.smithi202.stdout:Downloading Packages: 2022-06-27T11:45:19.011 INFO:teuthology.orchestra.run.smithi202.stdout:(1/124): ceph-17.0.0-13241.g5292901b.el8.x86_64 50 kB/s | 6.5 kB 00:00 2022-06-27T11:45:19.229 INFO:teuthology.orchestra.run.smithi202.stdout:(2/124): ceph-fuse-17.0.0-13241.g5292901b.el8.x 4.0 MB/s | 883 kB 00:00 2022-06-27T11:45:19.304 INFO:teuthology.orchestra.run.smithi202.stdout:(3/124): ceph-immutable-object-cache-17.0.0-132 2.2 MB/s | 169 kB 00:00 2022-06-27T11:45:19.340 INFO:teuthology.orchestra.run.smithi195.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2022-06-27T11:45:19.348 INFO:teuthology.orchestra.run.smithi195.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2022-06-27T11:45:19.354 INFO:teuthology.orchestra.run.smithi195.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2022-06-27T11:45:19.355 INFO:teuthology.orchestra.run.smithi195.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2022-06-27T11:45:19.356 INFO:teuthology.orchestra.run.smithi195.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2022-06-27T11:45:19.357 INFO:teuthology.orchestra.run.smithi195.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2022-06-27T11:45:19.359 INFO:teuthology.orchestra.run.smithi195.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2022-06-27T11:45:19.362 INFO:teuthology.orchestra.run.smithi195.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2022-06-27T11:45:19.363 INFO:teuthology.orchestra.run.smithi195.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2022-06-27T11:45:19.479 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2022-06-27T11:45:19.491 INFO:teuthology.orchestra.run.smithi195.stdout:======================================================================================================================= 2022-06-27T11:45:19.491 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2022-06-27T11:45:19.491 INFO:teuthology.orchestra.run.smithi195.stdout:======================================================================================================================= 2022-06-27T11:45:19.492 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2022-06-27T11:45:19.492 INFO:teuthology.orchestra.run.smithi195.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2022-06-27T11:45:19.493 INFO:teuthology.orchestra.run.smithi195.stdout: ceph x86_64 2:17.0.0-13241.g5292901b.el8 ceph 6.5 k 2022-06-27T11:45:19.493 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-base x86_64 2:17.0.0-13241.g5292901b.el8 ceph 5.3 M 2022-06-27T11:45:19.493 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-fuse x86_64 2:17.0.0-13241.g5292901b.el8 ceph 883 k 2022-06-27T11:45:19.494 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-immutable-object-cache x86_64 2:17.0.0-13241.g5292901b.el8 ceph 169 k 2022-06-27T11:45:19.494 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr x86_64 2:17.0.0-13241.g5292901b.el8 ceph 1.5 M 2022-06-27T11:45:19.494 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-cephadm noarch 2:17.0.0-13241.g5292901b.el8 ceph-noarch 111 k 2022-06-27T11:45:19.495 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-dashboard noarch 2:17.0.0-13241.g5292901b.el8 ceph-noarch 1.6 M 2022-06-27T11:45:19.495 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-diskprediction-local noarch 2:17.0.0-13241.g5292901b.el8 ceph-noarch 7.4 M 2022-06-27T11:45:19.495 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-rook noarch 2:17.0.0-13241.g5292901b.el8 ceph-noarch 47 k 2022-06-27T11:45:19.496 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-radosgw x86_64 2:17.0.0-13241.g5292901b.el8 ceph 17 M 2022-06-27T11:45:19.496 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-test x86_64 2:17.0.0-13241.g5292901b.el8 ceph 53 M 2022-06-27T11:45:19.497 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-volume noarch 2:17.0.0-13241.g5292901b.el8 ceph-noarch 259 k 2022-06-27T11:45:19.497 INFO:teuthology.orchestra.run.smithi195.stdout: cephadm noarch 2:17.0.0-13241.g5292901b.el8 ceph-noarch 79 k 2022-06-27T11:45:19.497 INFO:teuthology.orchestra.run.smithi195.stdout: cephfs-mirror x86_64 2:17.0.0-13241.g5292901b.el8 ceph 273 k 2022-06-27T11:45:19.499 INFO:teuthology.orchestra.run.smithi195.stdout: cephfs-top noarch 2:17.0.0-13241.g5292901b.el8 ceph-noarch 14 k 2022-06-27T11:45:19.499 INFO:teuthology.orchestra.run.smithi195.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2022-06-27T11:45:19.499 INFO:teuthology.orchestra.run.smithi195.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2022-06-27T11:45:19.500 INFO:teuthology.orchestra.run.smithi195.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2022-06-27T11:45:19.500 INFO:teuthology.orchestra.run.smithi195.stdout: libcephfs-devel x86_64 2:17.0.0-13241.g5292901b.el8 ceph 30 k 2022-06-27T11:45:19.500 INFO:teuthology.orchestra.run.smithi195.stdout: libcephfs2 x86_64 2:17.0.0-13241.g5292901b.el8 ceph 796 k 2022-06-27T11:45:19.501 INFO:teuthology.orchestra.run.smithi195.stdout: librados-devel x86_64 2:17.0.0-13241.g5292901b.el8 ceph 125 k 2022-06-27T11:45:19.501 INFO:teuthology.orchestra.run.smithi195.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2022-06-27T11:45:19.501 INFO:teuthology.orchestra.run.smithi195.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2022-06-27T11:45:19.502 INFO:teuthology.orchestra.run.smithi195.stdout: openssl-devel x86_64 1:1.1.1k-6.el8_5 rhel-8-for-x86_64-baseos-rpms 2.3 M 2022-06-27T11:45:19.502 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cephfs x86_64 2:17.0.0-13241.g5292901b.el8 ceph 178 k 2022-06-27T11:45:19.502 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rados x86_64 2:17.0.0-13241.g5292901b.el8 ceph 366 k 2022-06-27T11:45:19.502 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rbd x86_64 2:17.0.0-13241.g5292901b.el8 ceph 346 k 2022-06-27T11:45:19.503 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rgw x86_64 2:17.0.0-13241.g5292901b.el8 ceph 109 k 2022-06-27T11:45:19.503 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-fuse x86_64 2:17.0.0-13241.g5292901b.el8 ceph 93 k 2022-06-27T11:45:19.503 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-mirror x86_64 2:17.0.0-13241.g5292901b.el8 ceph 3.8 M 2022-06-27T11:45:19.503 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-nbd x86_64 2:17.0.0-13241.g5292901b.el8 ceph 175 k 2022-06-27T11:45:19.504 INFO:teuthology.orchestra.run.smithi195.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2022-06-27T11:45:19.504 INFO:teuthology.orchestra.run.smithi195.stdout:Upgrading: 2022-06-27T11:45:19.504 INFO:teuthology.orchestra.run.smithi195.stdout: librados2 x86_64 2:17.0.0-13241.g5292901b.el8 ceph 3.6 M 2022-06-27T11:45:19.504 INFO:teuthology.orchestra.run.smithi195.stdout: librbd1 x86_64 2:17.0.0-13241.g5292901b.el8 ceph 3.9 M 2022-06-27T11:45:19.505 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2022-06-27T11:45:19.505 INFO:teuthology.orchestra.run.smithi195.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2022-06-27T11:45:19.505 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-common x86_64 2:17.0.0-13241.g5292901b.el8 ceph 24 M 2022-06-27T11:45:19.505 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-grafana-dashboards noarch 2:17.0.0-13241.g5292901b.el8 ceph-noarch 23 k 2022-06-27T11:45:19.506 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mds x86_64 2:17.0.0-13241.g5292901b.el8 ceph 2.3 M 2022-06-27T11:45:19.506 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-modules-core noarch 2:17.0.0-13241.g5292901b.el8 ceph-noarch 230 k 2022-06-27T11:45:19.506 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mon x86_64 2:17.0.0-13241.g5292901b.el8 ceph 3.9 M 2022-06-27T11:45:19.506 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-osd x86_64 2:17.0.0-13241.g5292901b.el8 ceph 17 M 2022-06-27T11:45:19.507 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-prometheus-alerts noarch 2:17.0.0-13241.g5292901b.el8 ceph-noarch 15 k 2022-06-27T11:45:19.507 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-selinux x86_64 2:17.0.0-13241.g5292901b.el8 ceph 24 k 2022-06-27T11:45:19.507 INFO:teuthology.orchestra.run.smithi195.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2022-06-27T11:45:19.507 INFO:teuthology.orchestra.run.smithi195.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2022-06-27T11:45:19.508 INFO:teuthology.orchestra.run.smithi195.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2022-06-27T11:45:19.508 INFO:teuthology.orchestra.run.smithi195.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2022-06-27T11:45:19.508 INFO:teuthology.orchestra.run.smithi195.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2022-06-27T11:45:19.509 INFO:teuthology.orchestra.run.smithi195.stdout: libcephsqlite x86_64 2:17.0.0-13241.g5292901b.el8 ceph 170 k 2022-06-27T11:45:19.509 INFO:teuthology.orchestra.run.smithi195.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2022-06-27T11:45:19.509 INFO:teuthology.orchestra.run.smithi195.stdout: libgfortran x86_64 8.5.0-10.el8 rhel-8-for-x86_64-baseos-rpms 643 k 2022-06-27T11:45:19.509 INFO:teuthology.orchestra.run.smithi195.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2022-06-27T11:45:19.510 INFO:teuthology.orchestra.run.smithi195.stdout: libquadmath x86_64 8.5.0-10.el8 rhel-8-for-x86_64-baseos-rpms 171 k 2022-06-27T11:45:19.510 INFO:teuthology.orchestra.run.smithi195.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2022-06-27T11:45:19.511 INFO:teuthology.orchestra.run.smithi195.stdout: libradosstriper1 x86_64 2:17.0.0-13241.g5292901b.el8 ceph 485 k 2022-06-27T11:45:19.511 INFO:teuthology.orchestra.run.smithi195.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2022-06-27T11:45:19.511 INFO:teuthology.orchestra.run.smithi195.stdout: librgw2 x86_64 2:17.0.0-13241.g5292901b.el8 ceph 5.5 M 2022-06-27T11:45:19.511 INFO:teuthology.orchestra.run.smithi195.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2022-06-27T11:45:19.511 INFO:teuthology.orchestra.run.smithi195.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2022-06-27T11:45:19.512 INFO:teuthology.orchestra.run.smithi195.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2022-06-27T11:45:19.512 INFO:teuthology.orchestra.run.smithi195.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2022-06-27T11:45:19.512 INFO:teuthology.orchestra.run.smithi195.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2022-06-27T11:45:19.512 INFO:teuthology.orchestra.run.smithi195.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2022-06-27T11:45:19.513 INFO:teuthology.orchestra.run.smithi195.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2022-06-27T11:45:19.513 INFO:teuthology.orchestra.run.smithi195.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2022-06-27T11:45:19.513 INFO:teuthology.orchestra.run.smithi195.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2022-06-27T11:45:19.513 INFO:teuthology.orchestra.run.smithi195.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2022-06-27T11:45:19.514 INFO:teuthology.orchestra.run.smithi195.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2022-06-27T11:45:19.514 INFO:teuthology.orchestra.run.smithi195.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2022-06-27T11:45:19.514 INFO:teuthology.orchestra.run.smithi195.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2022-06-27T11:45:19.514 INFO:teuthology.orchestra.run.smithi195.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2022-06-27T11:45:19.515 INFO:teuthology.orchestra.run.smithi195.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2022-06-27T11:45:19.515 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2022-06-27T11:45:19.515 INFO:teuthology.orchestra.run.smithi195.stdout: python3-ceph-argparse x86_64 2:17.0.0-13241.g5292901b.el8 ceph 45 k 2022-06-27T11:45:19.515 INFO:teuthology.orchestra.run.smithi195.stdout: python3-ceph-common x86_64 2:17.0.0-13241.g5292901b.el8 ceph 106 k 2022-06-27T11:45:19.515 INFO:teuthology.orchestra.run.smithi195.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2022-06-27T11:45:19.516 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2022-06-27T11:45:19.516 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2022-06-27T11:45:19.516 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2022-06-27T11:45:19.516 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2022-06-27T11:45:19.517 INFO:teuthology.orchestra.run.smithi195.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2022-06-27T11:45:19.517 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2022-06-27T11:45:19.517 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2022-06-27T11:45:19.517 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2022-06-27T11:45:19.517 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2022-06-27T11:45:19.518 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2022-06-27T11:45:19.518 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2022-06-27T11:45:19.518 INFO:teuthology.orchestra.run.smithi195.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2022-06-27T11:45:19.518 INFO:teuthology.orchestra.run.smithi195.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2022-06-27T11:45:19.519 INFO:teuthology.orchestra.run.smithi195.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2022-06-27T11:45:19.519 INFO:teuthology.orchestra.run.smithi195.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2022-06-27T11:45:19.519 INFO:teuthology.orchestra.run.smithi195.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2022-06-27T11:45:19.519 INFO:teuthology.orchestra.run.smithi195.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2022-06-27T11:45:19.519 INFO:teuthology.orchestra.run.smithi195.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2022-06-27T11:45:19.520 INFO:teuthology.orchestra.run.smithi195.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2022-06-27T11:45:19.520 INFO:teuthology.orchestra.run.smithi195.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2022-06-27T11:45:19.520 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2022-06-27T11:45:19.520 INFO:teuthology.orchestra.run.smithi195.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2022-06-27T11:45:19.521 INFO:teuthology.orchestra.run.smithi195.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2022-06-27T11:45:19.521 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2022-06-27T11:45:19.521 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2022-06-27T11:45:19.521 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2022-06-27T11:45:19.522 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2022-06-27T11:45:19.522 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2022-06-27T11:45:19.523 INFO:teuthology.orchestra.run.smithi195.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2022-06-27T11:45:19.523 INFO:teuthology.orchestra.run.smithi195.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2022-06-27T11:45:19.523 INFO:teuthology.orchestra.run.smithi195.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2022-06-27T11:45:19.523 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2022-06-27T11:45:19.524 INFO:teuthology.orchestra.run.smithi195.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2022-06-27T11:45:19.524 INFO:teuthology.orchestra.run.smithi195.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2022-06-27T11:45:19.524 INFO:teuthology.orchestra.run.smithi195.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2022-06-27T11:45:19.524 INFO:teuthology.orchestra.run.smithi195.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2022-06-27T11:45:19.524 INFO:teuthology.orchestra.run.smithi195.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2022-06-27T11:45:19.525 INFO:teuthology.orchestra.run.smithi195.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2022-06-27T11:45:19.525 INFO:teuthology.orchestra.run.smithi195.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2022-06-27T11:45:19.525 INFO:teuthology.orchestra.run.smithi195.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2022-06-27T11:45:19.525 INFO:teuthology.orchestra.run.smithi195.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2022-06-27T11:45:19.526 INFO:teuthology.orchestra.run.smithi195.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2022-06-27T11:45:19.526 INFO:teuthology.orchestra.run.smithi195.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2022-06-27T11:45:19.526 INFO:teuthology.orchestra.run.smithi195.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2022-06-27T11:45:19.526 INFO:teuthology.orchestra.run.smithi195.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2022-06-27T11:45:19.526 INFO:teuthology.orchestra.run.smithi195.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2022-06-27T11:45:19.527 INFO:teuthology.orchestra.run.smithi195.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2022-06-27T11:45:19.527 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:45:19.527 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2022-06-27T11:45:19.527 INFO:teuthology.orchestra.run.smithi195.stdout:======================================================================================================================= 2022-06-27T11:45:19.528 INFO:teuthology.orchestra.run.smithi195.stdout:Install 122 Packages 2022-06-27T11:45:19.528 INFO:teuthology.orchestra.run.smithi195.stdout:Upgrade 2 Packages 2022-06-27T11:45:19.528 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:45:19.529 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 204 M 2022-06-27T11:45:19.529 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2022-06-27T11:45:19.668 INFO:teuthology.orchestra.run.smithi195.stdout:(1/124): ceph-17.0.0-13241.g5292901b.el8.x86_64 40 kB/s | 6.5 kB 00:00 2022-06-27T11:45:19.671 INFO:teuthology.orchestra.run.smithi202.stdout:(4/124): ceph-base-17.0.0-13241.g5292901b.el8.x 6.7 MB/s | 5.3 MB 00:00 2022-06-27T11:45:19.713 INFO:teuthology.orchestra.run.smithi202.stdout:(5/124): ceph-mds-17.0.0-13241.g5292901b.el8.x8 5.6 MB/s | 2.3 MB 00:00 2022-06-27T11:45:19.890 INFO:teuthology.orchestra.run.smithi202.stdout:(6/124): ceph-mgr-17.0.0-13241.g5292901b.el8.x8 7.1 MB/s | 1.5 MB 00:00 2022-06-27T11:45:20.422 INFO:teuthology.orchestra.run.smithi202.stdout:(7/124): ceph-mon-17.0.0-13241.g5292901b.el8.x8 5.5 MB/s | 3.9 MB 00:00 2022-06-27T11:45:20.461 INFO:teuthology.orchestra.run.smithi195.stdout:(2/124): ceph-fuse-17.0.0-13241.g5292901b.el8.x 1.1 MB/s | 883 kB 00:00 2022-06-27T11:45:20.618 INFO:teuthology.orchestra.run.smithi195.stdout:(3/124): ceph-immutable-object-cache-17.0.0-132 1.1 MB/s | 169 kB 00:00 2022-06-27T11:45:21.610 INFO:teuthology.orchestra.run.smithi195.stdout:(4/124): ceph-mds-17.0.0-13241.g5292901b.el8.x8 2.3 MB/s | 2.3 MB 00:00 2022-06-27T11:45:21.639 INFO:teuthology.orchestra.run.smithi202.stdout:(8/124): ceph-common-17.0.0-13241.g5292901b.el8 8.5 MB/s | 24 MB 00:02 2022-06-27T11:45:21.739 INFO:teuthology.orchestra.run.smithi202.stdout:(9/124): ceph-selinux-17.0.0-13241.g5292901b.el 243 kB/s | 24 kB 00:00 2022-06-27T11:45:21.744 INFO:teuthology.orchestra.run.smithi195.stdout:(5/124): ceph-base-17.0.0-13241.g5292901b.el8.x 2.4 MB/s | 5.3 MB 00:02 2022-06-27T11:45:22.061 INFO:teuthology.orchestra.run.smithi195.stdout:(6/124): ceph-mgr-17.0.0-13241.g5292901b.el8.x8 3.4 MB/s | 1.5 MB 00:00 2022-06-27T11:45:22.298 INFO:teuthology.orchestra.run.smithi202.stdout:(10/124): ceph-osd-17.0.0-13241.g5292901b.el8.x 7.1 MB/s | 17 MB 00:02 2022-06-27T11:45:22.406 INFO:teuthology.orchestra.run.smithi202.stdout:(11/124): cephfs-mirror-17.0.0-13241.g5292901b. 2.5 MB/s | 273 kB 00:00 2022-06-27T11:45:22.498 INFO:teuthology.orchestra.run.smithi202.stdout:(12/124): libcephfs-devel-17.0.0-13241.g5292901 332 kB/s | 30 kB 00:00 2022-06-27T11:45:22.756 INFO:teuthology.orchestra.run.smithi202.stdout:(13/124): libcephfs2-17.0.0-13241.g5292901b.el8 3.0 MB/s | 796 kB 00:00 2022-06-27T11:45:22.848 INFO:teuthology.orchestra.run.smithi202.stdout:(14/124): libcephsqlite-17.0.0-13241.g5292901b. 1.8 MB/s | 170 kB 00:00 2022-06-27T11:45:22.933 INFO:teuthology.orchestra.run.smithi202.stdout:(15/124): librados-devel-17.0.0-13241.g5292901b 1.5 MB/s | 125 kB 00:00 2022-06-27T11:45:23.149 INFO:teuthology.orchestra.run.smithi202.stdout:(16/124): ceph-radosgw-17.0.0-13241.g5292901b.e 6.1 MB/s | 17 MB 00:02 2022-06-27T11:45:23.174 INFO:teuthology.orchestra.run.smithi202.stdout:(17/124): libradosstriper1-17.0.0-13241.g529290 2.0 MB/s | 485 kB 00:00 2022-06-27T11:45:23.186 INFO:teuthology.orchestra.run.smithi195.stdout:(7/124): ceph-mon-17.0.0-13241.g5292901b.el8.x8 2.7 MB/s | 3.9 MB 00:01 2022-06-27T11:45:23.277 INFO:teuthology.orchestra.run.smithi202.stdout:(18/124): python3-ceph-argparse-17.0.0-13241.g5 447 kB/s | 45 kB 00:00 2022-06-27T11:45:23.374 INFO:teuthology.orchestra.run.smithi202.stdout:(19/124): python3-ceph-common-17.0.0-13241.g529 1.0 MB/s | 106 kB 00:00 2022-06-27T11:45:23.475 INFO:teuthology.orchestra.run.smithi202.stdout:(20/124): python3-cephfs-17.0.0-13241.g5292901b 1.7 MB/s | 178 kB 00:00 2022-06-27T11:45:24.095 INFO:teuthology.orchestra.run.smithi195.stdout:(8/124): ceph-osd-17.0.0-13241.g5292901b.el8.x8 8.4 MB/s | 17 MB 00:02 2022-06-27T11:45:24.117 INFO:teuthology.orchestra.run.smithi202.stdout:(21/124): librgw2-17.0.0-13241.g5292901b.el8.x8 5.7 MB/s | 5.5 MB 00:00 2022-06-27T11:45:24.142 INFO:teuthology.orchestra.run.smithi202.stdout:(22/124): python3-rados-17.0.0-13241.g5292901b. 550 kB/s | 366 kB 00:00 2022-06-27T11:45:24.144 INFO:teuthology.orchestra.run.smithi195.stdout:(9/124): ceph-selinux-17.0.0-13241.g5292901b.el 489 kB/s | 24 kB 00:00 2022-06-27T11:45:24.185 INFO:teuthology.orchestra.run.smithi202.stdout:(23/124): python3-rbd-17.0.0-13241.g5292901b.el 5.1 MB/s | 346 kB 00:00 2022-06-27T11:45:24.209 INFO:teuthology.orchestra.run.smithi202.stdout:(24/124): python3-rgw-17.0.0-13241.g5292901b.el 1.6 MB/s | 109 kB 00:00 2022-06-27T11:45:24.252 INFO:teuthology.orchestra.run.smithi202.stdout:(25/124): rbd-fuse-17.0.0-13241.g5292901b.el8.x 1.4 MB/s | 93 kB 00:00 2022-06-27T11:45:24.368 INFO:teuthology.orchestra.run.smithi202.stdout:(26/124): rbd-nbd-17.0.0-13241.g5292901b.el8.x8 1.5 MB/s | 175 kB 00:00 2022-06-27T11:45:24.451 INFO:teuthology.orchestra.run.smithi202.stdout:(27/124): ceph-grafana-dashboards-17.0.0-13241. 275 kB/s | 23 kB 00:00 2022-06-27T11:45:24.548 INFO:teuthology.orchestra.run.smithi202.stdout:(28/124): ceph-mgr-cephadm-17.0.0-13241.g529290 1.2 MB/s | 111 kB 00:00 2022-06-27T11:45:24.969 INFO:teuthology.orchestra.run.smithi202.stdout:(29/124): ceph-mgr-dashboard-17.0.0-13241.g5292 3.8 MB/s | 1.6 MB 00:00 2022-06-27T11:45:25.253 INFO:teuthology.orchestra.run.smithi202.stdout:(30/124): rbd-mirror-17.0.0-13241.g5292901b.el8 3.6 MB/s | 3.8 MB 00:01 2022-06-27T11:45:25.417 INFO:teuthology.orchestra.run.smithi202.stdout:(31/124): ceph-mgr-modules-core-17.0.0-13241.g5 1.4 MB/s | 230 kB 00:00 2022-06-27T11:45:25.495 INFO:teuthology.orchestra.run.smithi202.stdout:(32/124): ceph-mgr-rook-17.0.0-13241.g5292901b. 606 kB/s | 47 kB 00:00 2022-06-27T11:45:25.587 INFO:teuthology.orchestra.run.smithi202.stdout:(33/124): ceph-prometheus-alerts-17.0.0-13241.g 160 kB/s | 15 kB 00:00 2022-06-27T11:45:25.720 INFO:teuthology.orchestra.run.smithi202.stdout:(34/124): ceph-volume-17.0.0-13241.g5292901b.el 1.9 MB/s | 259 kB 00:00 2022-06-27T11:45:25.804 INFO:teuthology.orchestra.run.smithi202.stdout:(35/124): cephadm-17.0.0-13241.g5292901b.el8.no 948 kB/s | 79 kB 00:00 2022-06-27T11:45:25.879 INFO:teuthology.orchestra.run.smithi202.stdout:(36/124): cephfs-top-17.0.0-13241.g5292901b.el8 192 kB/s | 14 kB 00:00 2022-06-27T11:45:26.038 INFO:teuthology.orchestra.run.smithi202.stdout:(37/124): ceph-mgr-diskprediction-local-17.0.0- 6.9 MB/s | 7.4 MB 00:01 2022-06-27T11:45:26.171 INFO:teuthology.orchestra.run.smithi202.stdout:(38/124): gperftools-libs-2.7-9.el8.x86_64.rpm 1.0 MB/s | 306 kB 00:00 2022-06-27T11:45:26.197 INFO:teuthology.orchestra.run.smithi202.stdout:(39/124): liboath-2.6.2-3.el8.x86_64.rpm 371 kB/s | 59 kB 00:00 2022-06-27T11:45:26.230 INFO:teuthology.orchestra.run.smithi202.stdout:(40/124): libunwind-1.3.1-3.el8.x86_64.rpm 1.3 MB/s | 75 kB 00:00 2022-06-27T11:45:26.289 INFO:teuthology.orchestra.run.smithi202.stdout:(41/124): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 745 kB/s | 44 kB 00:00 2022-06-27T11:45:26.381 INFO:teuthology.orchestra.run.smithi202.stdout:(42/124): python3-beautifulsoup4-4.6.3-2.el8.1. 2.0 MB/s | 185 kB 00:00 2022-06-27T11:45:26.440 INFO:teuthology.orchestra.run.smithi202.stdout:(43/124): python3-cachetools-3.1.1-4.el8.noarch 565 kB/s | 33 kB 00:00 2022-06-27T11:45:26.465 INFO:teuthology.orchestra.run.smithi202.stdout:(44/124): python3-asyncssh-2.7.0-2.el8.noarch.r 1.7 MB/s | 476 kB 00:00 2022-06-27T11:45:26.515 INFO:teuthology.orchestra.run.smithi202.stdout:(45/124): python3-certifi-2018.10.15-7.el8.noar 216 kB/s | 16 kB 00:00 2022-06-27T11:45:26.573 INFO:teuthology.orchestra.run.smithi202.stdout:(46/124): python3-cheroot-8.5.2-1.el8.noarch.rp 1.6 MB/s | 173 kB 00:00 2022-06-27T11:45:26.632 INFO:teuthology.orchestra.run.smithi202.stdout:(47/124): python3-google-auth-1.1.1-10.el8.noar 1.4 MB/s | 81 kB 00:00 2022-06-27T11:45:26.692 INFO:teuthology.orchestra.run.smithi202.stdout:(48/124): python3-jaraco-6.2-6.el8.noarch.rpm 190 kB/s | 11 kB 00:00 2022-06-27T11:45:26.717 INFO:teuthology.orchestra.run.smithi202.stdout:(49/124): python3-cherrypy-18.4.0-1.el8.noarch. 1.9 MB/s | 384 kB 00:00 2022-06-27T11:45:26.750 INFO:teuthology.orchestra.run.smithi202.stdout:(50/124): python3-jaraco-functools-2.0-4.el8.no 300 kB/s | 18 kB 00:00 2022-06-27T11:45:26.809 INFO:teuthology.orchestra.run.smithi202.stdout:(51/124): python3-logutils-0.3.5-11.el8.noarch. 833 kB/s | 49 kB 00:00 2022-06-27T11:45:26.875 INFO:teuthology.orchestra.run.smithi202.stdout:(52/124): python3-more-itertools-7.2.0-3.el8.no 882 kB/s | 59 kB 00:00 2022-06-27T11:45:26.934 INFO:teuthology.orchestra.run.smithi202.stdout:(53/124): python3-natsort-7.1.1-2.el8.noarch.rp 1.0 MB/s | 60 kB 00:00 2022-06-27T11:45:27.167 INFO:teuthology.orchestra.run.smithi202.stdout:(54/124): python3-pecan-1.3.2-9.el8.noarch.rpm 1.2 MB/s | 283 kB 00:00 2022-06-27T11:45:27.219 INFO:teuthology.orchestra.run.smithi202.stdout:(55/124): python3-kubernetes-11.0.0-6.el8.noarc 2.3 MB/s | 1.2 MB 00:00 2022-06-27T11:45:27.252 INFO:teuthology.orchestra.run.smithi202.stdout:(56/124): python3-portend-2.6-1.el8.noarch.rpm 187 kB/s | 16 kB 00:00 2022-06-27T11:45:27.286 INFO:teuthology.orchestra.run.smithi202.stdout:(57/124): python3-repoze-lru-0.7-6.el8.noarch.r 515 kB/s | 34 kB 00:00 2022-06-27T11:45:27.344 INFO:teuthology.orchestra.run.smithi202.stdout:(58/124): python3-rsa-4.8-1.el8.noarch.rpm 1.0 MB/s | 61 kB 00:00 2022-06-27T11:45:27.362 INFO:teuthology.orchestra.run.smithi202.stdout:(59/124): python3-routes-2.4.1-12.el8.noarch.rp 1.8 MB/s | 196 kB 00:00 2022-06-27T11:45:27.411 INFO:teuthology.orchestra.run.smithi202.stdout:(60/124): python3-simplegeneric-0.8.1-17.el8.no 280 kB/s | 19 kB 00:00 2022-06-27T11:45:27.437 INFO:teuthology.orchestra.run.smithi202.stdout:(61/124): python3-singledispatch-3.4.0.3-18.el8 317 kB/s | 24 kB 00:00 2022-06-27T11:45:27.469 INFO:teuthology.orchestra.run.smithi202.stdout:(62/124): python3-tempora-1.14.1-5.el8.noarch.r 499 kB/s | 29 kB 00:00 2022-06-27T11:45:27.503 INFO:teuthology.orchestra.run.smithi202.stdout:(63/124): python3-trustme-0.6.0-4.el8.noarch.rp 412 kB/s | 27 kB 00:00 2022-06-27T11:45:27.595 INFO:teuthology.orchestra.run.smithi202.stdout:(64/124): python3-waitress-1.2.1-2.el8.1.noarch 1.9 MB/s | 241 kB 00:00 2022-06-27T11:45:27.654 INFO:teuthology.orchestra.run.smithi202.stdout:(65/124): python3-websocket-client-0.56.0-5.el8 1.0 MB/s | 61 kB 00:00 2022-06-27T11:45:27.687 INFO:teuthology.orchestra.run.smithi202.stdout:(66/124): python3-webob-1.8.5-1.el8.1.noarch.rp 1.3 MB/s | 251 kB 00:00 2022-06-27T11:45:27.713 INFO:teuthology.orchestra.run.smithi202.stdout:(67/124): python3-webtest-2.0.33-1.el8.noarch.r 1.4 MB/s | 85 kB 00:00 2022-06-27T11:45:27.746 INFO:teuthology.orchestra.run.smithi202.stdout:(68/124): python3-zc-lockfile-2.0-2.el8.noarch. 389 kB/s | 23 kB 00:00 2022-06-27T11:45:27.822 INFO:teuthology.orchestra.run.smithi202.stdout:(69/124): xmlstarlet-1.6.1-11.el8.x86_64.rpm 921 kB/s | 69 kB 00:00 2022-06-27T11:45:27.853 INFO:teuthology.orchestra.run.smithi195.stdout:(10/124): ceph-radosgw-17.0.0-13241.g5292901b.e 3.6 MB/s | 17 MB 00:04 2022-06-27T11:45:27.880 INFO:teuthology.orchestra.run.smithi202.stdout:(70/124): python3-jinja2-2.10.1-3.el8.noarch.rp 9.0 MB/s | 538 kB 00:00 2022-06-27T11:45:27.940 INFO:teuthology.orchestra.run.smithi202.stdout:(71/124): python3-markupsafe-0.23-19.el8.x86_64 655 kB/s | 39 kB 00:00 2022-06-27T11:45:27.945 INFO:teuthology.orchestra.run.smithi195.stdout:(11/124): cephfs-mirror-17.0.0-13241.g5292901b. 2.9 MB/s | 273 kB 00:00 2022-06-27T11:45:27.997 INFO:teuthology.orchestra.run.smithi202.stdout:(72/124): bison-3.0.4-10.el8.x86_64.rpm 12 MB/s | 688 kB 00:00 2022-06-27T11:45:28.012 INFO:teuthology.orchestra.run.smithi195.stdout:(12/124): libcephfs-devel-17.0.0-13241.g5292901 457 kB/s | 30 kB 00:00 2022-06-27T11:45:28.090 INFO:teuthology.orchestra.run.smithi202.stdout:(73/124): python3-numpy-1.14.3-10.el8.x86_64.rp 41 MB/s | 3.7 MB 00:00 2022-06-27T11:45:28.124 INFO:teuthology.orchestra.run.smithi202.stdout:(74/124): python3-prettytable-0.7.2-14.el8.noar 1.3 MB/s | 44 kB 00:00 2022-06-27T11:45:28.173 INFO:teuthology.orchestra.run.smithi202.stdout:(75/124): python3-werkzeug-0.12.2-4.el8.noarch. 9.0 MB/s | 457 kB 00:00 2022-06-27T11:45:28.179 INFO:teuthology.orchestra.run.smithi195.stdout:(13/124): libcephfs2-17.0.0-13241.g5292901b.el8 4.7 MB/s | 796 kB 00:00 2022-06-27T11:45:28.215 INFO:teuthology.orchestra.run.smithi202.stdout:(76/124): python3-mako-1.0.6-13.el8.noarch.rpm 3.7 MB/s | 157 kB 00:00 2022-06-27T11:45:28.237 INFO:teuthology.orchestra.run.smithi195.stdout:(14/124): libcephsqlite-17.0.0-13241.g5292901b. 2.9 MB/s | 170 kB 00:00 2022-06-27T11:45:28.266 INFO:teuthology.orchestra.run.smithi202.stdout:(77/124): libtool-2.4.6-25.el8.x86_64.rpm 14 MB/s | 709 kB 00:00 2022-06-27T11:45:28.304 INFO:teuthology.orchestra.run.smithi195.stdout:(15/124): librados-devel-17.0.0-13241.g5292901b 1.8 MB/s | 125 kB 00:00 2022-06-27T11:45:28.333 INFO:teuthology.orchestra.run.smithi202.stdout:(78/124): python3-numpy-f2py-1.14.3-10.el8.x86_ 3.3 MB/s | 225 kB 00:00 2022-06-27T11:45:28.375 INFO:teuthology.orchestra.run.smithi202.stdout:(79/124): python3-jsonpointer-1.10-11.el8.noarc 479 kB/s | 20 kB 00:00 2022-06-27T11:45:28.416 INFO:teuthology.orchestra.run.smithi202.stdout:(80/124): thrift-0.13.0-2.el8.x86_64.rpm 2.5 MB/s | 1.7 MB 00:00 2022-06-27T11:45:28.446 INFO:teuthology.orchestra.run.smithi195.stdout:(16/124): libradosstriper1-17.0.0-13241.g529290 3.3 MB/s | 485 kB 00:00 2022-06-27T11:45:28.917 INFO:teuthology.orchestra.run.smithi202.stdout:(81/124): ceph-test-17.0.0-13241.g5292901b.el8. 7.4 MB/s | 53 MB 00:07 2022-06-27T11:45:28.950 INFO:teuthology.orchestra.run.smithi202.stdout:(82/124): librdkafka-0.11.4-3.el8.x86_64.rpm 615 kB/s | 354 kB 00:00 2022-06-27T11:45:29.001 INFO:teuthology.orchestra.run.smithi202.stdout:(83/124): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.5 MB/s | 126 kB 00:00 2022-06-27T11:45:29.051 INFO:teuthology.orchestra.run.smithi202.stdout:(84/124): jq-1.6-3.el8.x86_64.rpm 4.0 MB/s | 202 kB 00:00 2022-06-27T11:45:29.093 INFO:teuthology.orchestra.run.smithi202.stdout:(85/124): python3-pytz-2017.2-9.el8.noarch.rpm 1.3 MB/s | 54 kB 00:00 2022-06-27T11:45:29.151 INFO:teuthology.orchestra.run.smithi202.stdout:(86/124): openblas-threads-0.3.15-3.el8.x86_64. 6.6 MB/s | 4.9 MB 00:00 2022-06-27T11:45:29.254 INFO:teuthology.orchestra.run.smithi202.stdout:(87/124): openblas-0.3.15-3.el8.x86_64.rpm 16 MB/s | 4.7 MB 00:00 2022-06-27T11:45:29.302 INFO:teuthology.orchestra.run.smithi202.stdout:(88/124): automake-1.16.1-7.el8.noarch.rpm 4.7 MB/s | 713 kB 00:00 2022-06-27T11:45:29.305 INFO:teuthology.orchestra.run.smithi195.stdout:(17/124): ceph-test-17.0.0-13241.g5292901b.el8. 10 MB/s | 53 MB 00:05 2022-06-27T11:45:29.327 INFO:teuthology.orchestra.run.smithi202.stdout:(89/124): python3-pyasn1-modules-0.3.7-6.el8.no 1.4 MB/s | 110 kB 00:00 2022-06-27T11:45:29.369 INFO:teuthology.orchestra.run.smithi202.stdout:(90/124): python3-jsonpatch-1.21-2.el8.noarch.r 397 kB/s | 27 kB 00:00 2022-06-27T11:45:29.394 INFO:teuthology.orchestra.run.smithi202.stdout:(91/124): lttng-ust-2.8.1-11.el8.x86_64.rpm 4.0 MB/s | 259 kB 00:00 2022-06-27T11:45:29.450 INFO:teuthology.orchestra.run.smithi202.stdout:(92/124): flex-2.6.1-9.el8.x86_64.rpm 4.1 MB/s | 320 kB 00:00 2022-06-27T11:45:29.478 INFO:teuthology.orchestra.run.smithi202.stdout:(93/124): socat-1.7.4.1-1.el8.x86_64.rpm 4.0 MB/s | 323 kB 00:00 2022-06-27T11:45:29.530 INFO:teuthology.orchestra.run.smithi195.stdout:(18/124): ceph-common-17.0.0-13241.g5292901b.el 2.3 MB/s | 24 MB 00:10 2022-06-27T11:45:29.531 INFO:teuthology.orchestra.run.smithi202.stdout:(94/124): python3-pyOpenSSL-19.0.0-1.el8.noarch 2.0 MB/s | 103 kB 00:00 2022-06-27T11:45:29.562 INFO:teuthology.orchestra.run.smithi195.stdout:(19/124): python3-ceph-argparse-17.0.0-13241.g5 179 kB/s | 45 kB 00:00 2022-06-27T11:45:29.587 INFO:teuthology.orchestra.run.smithi202.stdout:(95/124): oniguruma-6.8.2-2.el8.x86_64.rpm 3.1 MB/s | 187 kB 00:00 2022-06-27T11:45:29.656 INFO:teuthology.orchestra.run.smithi195.stdout:(20/124): librgw2-17.0.0-13241.g5292901b.el8.x8 4.6 MB/s | 5.5 MB 00:01 2022-06-27T11:45:29.671 INFO:teuthology.orchestra.run.smithi202.stdout:(96/124): python3-babel-2.5.1-7.el8.noarch.rpm 21 MB/s | 4.8 MB 00:00 2022-06-27T11:45:29.681 INFO:teuthology.orchestra.run.smithi195.stdout:(21/124): python3-ceph-common-17.0.0-13241.g529 705 kB/s | 106 kB 00:00 2022-06-27T11:45:29.698 INFO:teuthology.orchestra.run.smithi195.stdout:(22/124): python3-cephfs-17.0.0-13241.g5292901b 1.2 MB/s | 178 kB 00:00 2022-06-27T11:45:29.704 INFO:teuthology.orchestra.run.smithi202.stdout:(97/124): krb5-devel-1.18.2-14.el8.x86_64.rpm 4.7 MB/s | 560 kB 00:00 2022-06-27T11:45:29.738 INFO:teuthology.orchestra.run.smithi202.stdout:(98/124): python3-oauthlib-2.1.0-1.el8.noarch.r 2.3 MB/s | 155 kB 00:00 2022-06-27T11:45:29.748 INFO:teuthology.orchestra.run.smithi195.stdout:(23/124): python3-rados-17.0.0-13241.g5292901b. 3.9 MB/s | 366 kB 00:00 2022-06-27T11:45:29.772 INFO:teuthology.orchestra.run.smithi202.stdout:(99/124): python3-pycparser-2.14-14.el8.noarch. 1.6 MB/s | 109 kB 00:00 2022-06-27T11:45:29.773 INFO:teuthology.orchestra.run.smithi195.stdout:(24/124): python3-rgw-17.0.0-13241.g5292901b.el 1.4 MB/s | 109 kB 00:00 2022-06-27T11:45:29.807 INFO:teuthology.orchestra.run.smithi195.stdout:(25/124): python3-rbd-17.0.0-13241.g5292901b.el 2.7 MB/s | 346 kB 00:00 2022-06-27T11:45:29.822 INFO:teuthology.orchestra.run.smithi202.stdout:(100/124): python3-cryptography-3.2.1-5.el8.x86 6.9 MB/s | 559 kB 00:00 2022-06-27T11:45:29.834 INFO:teuthology.orchestra.run.smithi195.stdout:(26/124): rbd-fuse-17.0.0-13241.g5292901b.el8.x 1.1 MB/s | 93 kB 00:00 2022-06-27T11:45:29.839 INFO:teuthology.orchestra.run.smithi202.stdout:(101/124): libverto-devel-0.3.0-5.el8.x86_64.rp 267 kB/s | 18 kB 00:00 2022-06-27T11:45:29.883 INFO:teuthology.orchestra.run.smithi195.stdout:(27/124): ceph-grafana-dashboards-17.0.0-13241. 460 kB/s | 23 kB 00:00 2022-06-27T11:45:29.908 INFO:teuthology.orchestra.run.smithi195.stdout:(28/124): rbd-nbd-17.0.0-13241.g5292901b.el8.x8 1.7 MB/s | 175 kB 00:00 2022-06-27T11:45:29.941 INFO:teuthology.orchestra.run.smithi195.stdout:(29/124): ceph-mgr-cephadm-17.0.0-13241.g529290 1.8 MB/s | 111 kB 00:00 2022-06-27T11:45:30.014 INFO:teuthology.orchestra.run.smithi202.stdout:(102/124): python3-scipy-1.0.0-21.module+el8.5. 15 MB/s | 14 MB 00:00 2022-06-27T11:45:30.065 INFO:teuthology.orchestra.run.smithi202.stdout:(103/124): libselinux-devel-2.9-5.el8.x86_64.rp 825 kB/s | 200 kB 00:00 2022-06-27T11:45:30.090 INFO:teuthology.orchestra.run.smithi202.stdout:(104/124): pcre2-utf32-10.32-2.el8.x86_64.rpm 880 kB/s | 220 kB 00:00 2022-06-27T11:45:30.132 INFO:teuthology.orchestra.run.smithi202.stdout:(105/124): libsepol-devel-2.9-3.el8.x86_64.rpm 799 kB/s | 87 kB 00:00 2022-06-27T11:45:30.133 INFO:teuthology.orchestra.run.smithi195.stdout:(30/124): rbd-mirror-17.0.0-13241.g5292901b.el8 10 MB/s | 3.8 MB 00:00 2022-06-27T11:45:30.148 INFO:teuthology.orchestra.run.smithi202.stdout:(106/124): librabbitmq-0.9.0-3.el8.x86_64.rpm 560 kB/s | 47 kB 00:00 2022-06-27T11:45:30.200 INFO:teuthology.orchestra.run.smithi195.stdout:(31/124): ceph-mgr-modules-core-17.0.0-13241.g5 3.4 MB/s | 230 kB 00:00 2022-06-27T11:45:30.211 INFO:teuthology.orchestra.run.smithi202.stdout:(107/124): pcre2-devel-10.32-2.el8.x86_64.rpm 5.4 MB/s | 605 kB 00:00 2022-06-27T11:45:30.232 INFO:teuthology.orchestra.run.smithi202.stdout:(108/124): libaio-devel-0.3.112-1.el8.x86_64.rp 186 kB/s | 19 kB 00:00 2022-06-27T11:45:30.259 INFO:teuthology.orchestra.run.smithi195.stdout:(32/124): ceph-mgr-rook-17.0.0-13241.g5292901b. 805 kB/s | 47 kB 00:00 2022-06-27T11:45:30.266 INFO:teuthology.orchestra.run.smithi202.stdout:(109/124): python3-cffi-1.11.5-5.el8.x86_64.rpm 2.1 MB/s | 238 kB 00:00 2022-06-27T11:45:30.293 INFO:teuthology.orchestra.run.smithi195.stdout:(33/124): ceph-mgr-dashboard-17.0.0-13241.g5292 4.2 MB/s | 1.6 MB 00:00 2022-06-27T11:45:30.316 INFO:teuthology.orchestra.run.smithi202.stdout:(110/124): libuuid-devel-2.32.1-35.el8.x86_64.r 902 kB/s | 98 kB 00:00 2022-06-27T11:45:30.317 INFO:teuthology.orchestra.run.smithi195.stdout:(34/124): ceph-prometheus-alerts-17.0.0-13241.g 251 kB/s | 15 kB 00:00 2022-06-27T11:45:30.351 INFO:teuthology.orchestra.run.smithi195.stdout:(35/124): ceph-volume-17.0.0-13241.g5292901b.el 4.3 MB/s | 259 kB 00:00 2022-06-27T11:45:30.383 INFO:teuthology.orchestra.run.smithi202.stdout:(111/124): zlib-devel-1.2.11-18.el8_5.x86_64.rp 383 kB/s | 58 kB 00:00 2022-06-27T11:45:30.394 INFO:teuthology.orchestra.run.smithi195.stdout:(36/124): cephadm-17.0.0-13241.g5292901b.el8.no 1.0 MB/s | 79 kB 00:00 2022-06-27T11:45:30.408 INFO:teuthology.orchestra.run.smithi202.stdout:(112/124): fuse-2.9.7-15.el8.x86_64.rpm 599 kB/s | 83 kB 00:00 2022-06-27T11:45:30.419 INFO:teuthology.orchestra.run.smithi195.stdout:(37/124): cephfs-top-17.0.0-13241.g5292901b.el8 215 kB/s | 14 kB 00:00 2022-06-27T11:45:30.458 INFO:teuthology.orchestra.run.smithi202.stdout:(113/124): python3-jwt-1.6.1-2.el8.noarch.rpm 309 kB/s | 43 kB 00:00 2022-06-27T11:45:30.484 INFO:teuthology.orchestra.run.smithi202.stdout:(114/124): libcom_err-devel-1.45.6-4.el8.x86_64 384 kB/s | 39 kB 00:00 2022-06-27T11:45:30.509 INFO:teuthology.orchestra.run.smithi202.stdout:(115/124): elfutils-libelf-devel-0.186-1.el8.x8 623 kB/s | 60 kB 00:00 2022-06-27T11:45:30.551 INFO:teuthology.orchestra.run.smithi202.stdout:(116/124): libquadmath-8.5.0-10.el8.x86_64.rpm 1.8 MB/s | 171 kB 00:00 2022-06-27T11:45:30.576 INFO:teuthology.orchestra.run.smithi202.stdout:(117/124): keyutils-libs-devel-1.5.10-9.el8.x86 522 kB/s | 48 kB 00:00 2022-06-27T11:45:30.602 INFO:teuthology.orchestra.run.smithi195.stdout:(38/124): liboath-2.6.2-3.el8.x86_64.rpm 320 kB/s | 59 kB 00:00 2022-06-27T11:45:30.610 INFO:teuthology.orchestra.run.smithi202.stdout:(118/124): libgfortran-8.5.0-10.el8.x86_64.rpm 6.5 MB/s | 643 kB 00:00 2022-06-27T11:45:30.686 INFO:teuthology.orchestra.run.smithi195.stdout:(39/124): libunwind-1.3.1-3.el8.x86_64.rpm 903 kB/s | 75 kB 00:00 2022-06-27T11:45:30.711 INFO:teuthology.orchestra.run.smithi195.stdout:(40/124): gperftools-libs-2.7-9.el8.x86_64.rpm 965 kB/s | 306 kB 00:00 2022-06-27T11:45:30.729 INFO:teuthology.orchestra.run.smithi202.stdout:(119/124): xfsprogs-devel-5.0.0-10.el8.x86_64.r 408 kB/s | 72 kB 00:00 2022-06-27T11:45:30.770 INFO:teuthology.orchestra.run.smithi195.stdout:(41/124): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 748 kB/s | 44 kB 00:00 2022-06-27T11:45:30.902 INFO:teuthology.orchestra.run.smithi202.stdout:(120/124): pcre2-utf16-10.32-2.el8.x86_64.rpm 703 kB/s | 229 kB 00:00 2022-06-27T11:45:30.903 INFO:teuthology.orchestra.run.smithi195.stdout:(42/124): python3-beautifulsoup4-4.6.3-2.el8.1. 1.4 MB/s | 185 kB 00:00 2022-06-27T11:45:30.936 INFO:teuthology.orchestra.run.smithi202.stdout:(121/124): python3-requests-oauthlib-1.0.0-1.el 133 kB/s | 43 kB 00:00 2022-06-27T11:45:30.937 INFO:teuthology.orchestra.run.smithi195.stdout:(43/124): python3-asyncssh-2.7.0-2.el8.noarch.r 1.9 MB/s | 476 kB 00:00 2022-06-27T11:45:30.962 INFO:teuthology.orchestra.run.smithi195.stdout:(44/124): python3-cachetools-3.1.1-4.el8.noarch 564 kB/s | 33 kB 00:00 2022-06-27T11:45:31.004 INFO:teuthology.orchestra.run.smithi202.stdout:(122/124): openssl-devel-1.1.1k-6.el8_5.x86_64. 8.5 MB/s | 2.3 MB 00:00 2022-06-27T11:45:31.046 INFO:teuthology.orchestra.run.smithi195.stdout:(45/124): ceph-mgr-diskprediction-local-17.0.0- 6.7 MB/s | 7.4 MB 00:01 2022-06-27T11:45:31.071 INFO:teuthology.orchestra.run.smithi195.stdout:(46/124): python3-certifi-2018.10.15-7.el8.noar 122 kB/s | 16 kB 00:00 2022-06-27T11:45:31.096 INFO:teuthology.orchestra.run.smithi195.stdout:(47/124): python3-cheroot-8.5.2-1.el8.noarch.rp 1.3 MB/s | 173 kB 00:00 2022-06-27T11:45:31.130 INFO:teuthology.orchestra.run.smithi195.stdout:(48/124): python3-google-auth-1.1.1-10.el8.noar 1.4 MB/s | 81 kB 00:00 2022-06-27T11:45:31.155 INFO:teuthology.orchestra.run.smithi195.stdout:(49/124): python3-jaraco-6.2-6.el8.noarch.rpm 189 kB/s | 11 kB 00:00 2022-06-27T11:45:31.189 INFO:teuthology.orchestra.run.smithi195.stdout:(50/124): python3-jaraco-functools-2.0-4.el8.no 299 kB/s | 18 kB 00:00 2022-06-27T11:45:31.247 INFO:teuthology.orchestra.run.smithi195.stdout:(51/124): python3-logutils-0.3.5-11.el8.noarch. 839 kB/s | 49 kB 00:00 2022-06-27T11:45:31.306 INFO:teuthology.orchestra.run.smithi195.stdout:(52/124): python3-more-itertools-7.2.0-3.el8.no 1.0 MB/s | 59 kB 00:00 2022-06-27T11:45:31.319 INFO:teuthology.orchestra.run.smithi202.stdout:(123/124): librados2-17.0.0-13241.g5292901b.el8 8.8 MB/s | 3.6 MB 00:00 2022-06-27T11:45:31.331 INFO:teuthology.orchestra.run.smithi195.stdout:(53/124): python3-cherrypy-18.4.0-1.el8.noarch. 1.3 MB/s | 384 kB 00:00 2022-06-27T11:45:31.373 INFO:teuthology.orchestra.run.smithi195.stdout:(54/124): python3-natsort-7.1.1-2.el8.noarch.rp 898 kB/s | 60 kB 00:00 2022-06-27T11:45:31.424 INFO:teuthology.orchestra.run.smithi195.stdout:(55/124): python3-pecan-1.3.2-9.el8.noarch.rpm 3.0 MB/s | 283 kB 00:00 2022-06-27T11:45:31.441 INFO:teuthology.orchestra.run.smithi195.stdout:(56/124): python3-portend-2.6-1.el8.noarch.rpm 233 kB/s | 16 kB 00:00 2022-06-27T11:45:31.483 INFO:teuthology.orchestra.run.smithi195.stdout:(57/124): python3-repoze-lru-0.7-6.el8.noarch.r 576 kB/s | 34 kB 00:00 2022-06-27T11:45:31.541 INFO:teuthology.orchestra.run.smithi195.stdout:(58/124): python3-rsa-4.8-1.el8.noarch.rpm 1.0 MB/s | 61 kB 00:00 2022-06-27T11:45:31.553 INFO:teuthology.orchestra.run.smithi202.stdout:(124/124): librbd1-17.0.0-13241.g5292901b.el8.x 6.3 MB/s | 3.9 MB 00:00 2022-06-27T11:45:31.556 INFO:teuthology.orchestra.run.smithi202.stdout:-------------------------------------------------------------------------------- 2022-06-27T11:45:31.556 INFO:teuthology.orchestra.run.smithi202.stdout:Total 16 MB/s | 204 MB 00:12 2022-06-27T11:45:31.575 INFO:teuthology.orchestra.run.smithi195.stdout:(59/124): python3-routes-2.4.1-12.el8.noarch.rp 1.4 MB/s | 196 kB 00:00 2022-06-27T11:45:31.592 INFO:teuthology.orchestra.run.smithi195.stdout:(60/124): python3-simplegeneric-0.8.1-17.el8.no 371 kB/s | 19 kB 00:00 2022-06-27T11:45:31.626 INFO:teuthology.orchestra.run.smithi195.stdout:(61/124): python3-kubernetes-11.0.0-6.el8.noarc 2.5 MB/s | 1.2 MB 00:00 2022-06-27T11:45:31.642 INFO:teuthology.orchestra.run.smithi195.stdout:(62/124): python3-singledispatch-3.4.0.3-18.el8 356 kB/s | 24 kB 00:00 2022-06-27T11:45:31.664 INFO:teuthology.orchestra.run.smithi195.stdout:(63/124): python3-tempora-1.14.1-5.el8.noarch.r 436 kB/s | 29 kB 00:00 2022-06-27T11:45:31.676 INFO:teuthology.orchestra.run.smithi195.stdout:(64/124): python3-trustme-0.6.0-4.el8.noarch.rp 545 kB/s | 27 kB 00:00 2022-06-27T11:45:31.727 INFO:teuthology.orchestra.run.smithi195.stdout:(65/124): python3-websocket-client-0.56.0-5.el8 1.2 MB/s | 61 kB 00:00 2022-06-27T11:45:31.752 INFO:teuthology.orchestra.run.smithi195.stdout:(66/124): python3-webob-1.8.5-1.el8.1.noarch.rp 2.7 MB/s | 251 kB 00:00 2022-06-27T11:45:31.778 INFO:teuthology.orchestra.run.smithi195.stdout:(67/124): python3-waitress-1.2.1-2.el8.1.noarch 1.8 MB/s | 241 kB 00:00 2022-06-27T11:45:31.803 INFO:teuthology.orchestra.run.smithi195.stdout:(68/124): python3-webtest-2.0.33-1.el8.noarch.r 1.1 MB/s | 85 kB 00:00 2022-06-27T11:45:31.819 INFO:teuthology.orchestra.run.smithi195.stdout:(69/124): python3-zc-lockfile-2.0-2.el8.noarch. 342 kB/s | 23 kB 00:00 2022-06-27T11:45:31.861 INFO:teuthology.orchestra.run.smithi195.stdout:(70/124): xmlstarlet-1.6.1-11.el8.x86_64.rpm 1.2 MB/s | 69 kB 00:00 2022-06-27T11:45:31.895 INFO:teuthology.orchestra.run.smithi195.stdout:(71/124): python3-jinja2-2.10.1-3.el8.noarch.rp 7.0 MB/s | 538 kB 00:00 2022-06-27T11:45:31.945 INFO:teuthology.orchestra.run.smithi195.stdout:(72/124): python3-markupsafe-0.23-19.el8.x86_64 465 kB/s | 39 kB 00:00 2022-06-27T11:45:31.980 INFO:teuthology.orchestra.run.smithi195.stdout:(73/124): bison-3.0.4-10.el8.x86_64.rpm 8.1 MB/s | 688 kB 00:00 2022-06-27T11:45:32.029 INFO:teuthology.orchestra.run.smithi195.stdout:(74/124): python3-prettytable-0.7.2-14.el8.noar 960 kB/s | 44 kB 00:00 2022-06-27T11:45:32.104 INFO:teuthology.orchestra.run.smithi195.stdout:(75/124): python3-numpy-1.14.3-10.el8.x86_64.rp 24 MB/s | 3.7 MB 00:00 2022-06-27T11:45:32.138 INFO:teuthology.orchestra.run.smithi195.stdout:(76/124): python3-werkzeug-0.12.2-4.el8.noarch. 4.1 MB/s | 457 kB 00:00 2022-06-27T11:45:32.180 INFO:teuthology.orchestra.run.smithi195.stdout:(77/124): python3-mako-1.0.6-13.el8.noarch.rpm 2.0 MB/s | 157 kB 00:00 2022-06-27T11:45:32.222 INFO:teuthology.orchestra.run.smithi195.stdout:(78/124): libtool-2.4.6-25.el8.x86_64.rpm 8.3 MB/s | 709 kB 00:00 2022-06-27T11:45:32.247 INFO:teuthology.orchestra.run.smithi195.stdout:(79/124): python3-numpy-f2py-1.14.3-10.el8.x86_ 3.5 MB/s | 225 kB 00:00 2022-06-27T11:45:32.277 INFO:teuthology.orchestra.run.smithi202.stdout:Running transaction check 2022-06-27T11:45:32.289 INFO:teuthology.orchestra.run.smithi195.stdout:(80/124): python3-jsonpointer-1.10-11.el8.noarc 299 kB/s | 20 kB 00:00 2022-06-27T11:45:32.331 INFO:teuthology.orchestra.run.smithi195.stdout:(81/124): librdkafka-0.11.4-3.el8.x86_64.rpm 4.3 MB/s | 354 kB 00:00 2022-06-27T11:45:32.382 INFO:teuthology.orchestra.run.smithi195.stdout:(82/124): python3-pyasn1-0.3.7-6.el8.noarch.rpm 2.7 MB/s | 126 kB 00:00 2022-06-27T11:45:32.388 INFO:teuthology.orchestra.run.smithi202.stdout:Transaction check succeeded. 2022-06-27T11:45:32.388 INFO:teuthology.orchestra.run.smithi202.stdout:Running transaction test 2022-06-27T11:45:32.465 INFO:teuthology.orchestra.run.smithi195.stdout:(83/124): openblas-threads-0.3.15-3.el8.x86_64. 28 MB/s | 4.9 MB 00:00 2022-06-27T11:45:32.507 INFO:teuthology.orchestra.run.smithi195.stdout:(84/124): jq-1.6-3.el8.x86_64.rpm 4.7 MB/s | 202 kB 00:00 2022-06-27T11:45:32.573 INFO:teuthology.orchestra.run.smithi195.stdout:(85/124): thrift-0.13.0-2.el8.x86_64.rpm 2.2 MB/s | 1.7 MB 00:00 2022-06-27T11:45:32.658 INFO:teuthology.orchestra.run.smithi195.stdout:(86/124): openblas-0.3.15-3.el8.x86_64.rpm 17 MB/s | 4.7 MB 00:00 2022-06-27T11:45:32.683 INFO:teuthology.orchestra.run.smithi195.stdout:(87/124): python3-pytz-2017.2-9.el8.noarch.rpm 309 kB/s | 54 kB 00:00 2022-06-27T11:45:32.742 INFO:teuthology.orchestra.run.smithi195.stdout:(88/124): automake-1.16.1-7.el8.noarch.rpm 8.4 MB/s | 713 kB 00:00 2022-06-27T11:45:32.767 INFO:teuthology.orchestra.run.smithi195.stdout:(89/124): python3-pyasn1-modules-0.3.7-6.el8.no 1.4 MB/s | 110 kB 00:00 2022-06-27T11:45:32.809 INFO:teuthology.orchestra.run.smithi195.stdout:(90/124): python3-jsonpatch-1.21-2.el8.noarch.r 398 kB/s | 27 kB 00:00 2022-06-27T11:45:32.834 INFO:teuthology.orchestra.run.smithi195.stdout:(91/124): lttng-ust-2.8.1-11.el8.x86_64.rpm 3.8 MB/s | 259 kB 00:00 2022-06-27T11:45:32.884 INFO:teuthology.orchestra.run.smithi195.stdout:(92/124): flex-2.6.1-9.el8.x86_64.rpm 4.1 MB/s | 320 kB 00:00 2022-06-27T11:45:32.910 INFO:teuthology.orchestra.run.smithi195.stdout:(93/124): socat-1.7.4.1-1.el8.x86_64.rpm 4.4 MB/s | 323 kB 00:00 2022-06-27T11:45:32.960 INFO:teuthology.orchestra.run.smithi195.stdout:(94/124): python3-pyOpenSSL-19.0.0-1.el8.noarch 2.2 MB/s | 103 kB 00:00 2022-06-27T11:45:33.002 INFO:teuthology.orchestra.run.smithi195.stdout:(95/124): oniguruma-6.8.2-2.el8.x86_64.rpm 4.4 MB/s | 187 kB 00:00 2022-06-27T11:45:33.077 INFO:teuthology.orchestra.run.smithi195.stdout:(96/124): python3-babel-2.5.1-7.el8.noarch.rpm 25 MB/s | 4.8 MB 00:00 2022-06-27T11:45:33.111 INFO:teuthology.orchestra.run.smithi195.stdout:(97/124): krb5-devel-1.18.2-14.el8.x86_64.rpm 5.0 MB/s | 560 kB 00:00 2022-06-27T11:45:33.155 INFO:teuthology.orchestra.run.smithi195.stdout:(98/124): python3-oauthlib-2.1.0-1.el8.noarch.r 2.0 MB/s | 155 kB 00:00 2022-06-27T11:45:33.178 INFO:teuthology.orchestra.run.smithi195.stdout:(99/124): python3-pycparser-2.14-14.el8.noarch. 1.6 MB/s | 109 kB 00:00 2022-06-27T11:45:33.228 INFO:teuthology.orchestra.run.smithi195.stdout:(100/124): python3-cryptography-3.2.1-5.el8.x86 7.2 MB/s | 559 kB 00:00 2022-06-27T11:45:33.368 INFO:teuthology.orchestra.run.smithi195.stdout:(101/124): python3-scipy-1.0.0-21.module+el8.5. 17 MB/s | 14 MB 00:00 2022-06-27T11:45:33.379 INFO:teuthology.orchestra.run.smithi195.stdout:(102/124): libverto-devel-0.3.0-5.el8.x86_64.rp 91 kB/s | 18 kB 00:00 2022-06-27T11:45:33.429 INFO:teuthology.orchestra.run.smithi195.stdout:(103/124): libselinux-devel-2.9-5.el8.x86_64.rp 997 kB/s | 200 kB 00:00 2022-06-27T11:45:33.436 INFO:teuthology.orchestra.run.smithi202.stdout:Transaction test succeeded. 2022-06-27T11:45:33.439 INFO:teuthology.orchestra.run.smithi202.stdout:Running transaction 2022-06-27T11:45:33.454 INFO:teuthology.orchestra.run.smithi195.stdout:(104/124): libsepol-devel-2.9-3.el8.x86_64.rpm 1.1 MB/s | 87 kB 00:00 2022-06-27T11:45:33.488 INFO:teuthology.orchestra.run.smithi195.stdout:(105/124): pcre2-utf32-10.32-2.el8.x86_64.rpm 2.0 MB/s | 220 kB 00:00 2022-06-27T11:45:33.522 INFO:teuthology.orchestra.run.smithi195.stdout:(106/124): librabbitmq-0.9.0-3.el8.x86_64.rpm 508 kB/s | 47 kB 00:00 2022-06-27T11:45:33.572 INFO:teuthology.orchestra.run.smithi195.stdout:(107/124): pcre2-devel-10.32-2.el8.x86_64.rpm 5.2 MB/s | 605 kB 00:00 2022-06-27T11:45:33.597 INFO:teuthology.orchestra.run.smithi195.stdout:(108/124): libaio-devel-0.3.112-1.el8.x86_64.rp 172 kB/s | 19 kB 00:00 2022-06-27T11:45:33.650 INFO:teuthology.orchestra.run.smithi195.stdout:(109/124): python3-cffi-1.11.5-5.el8.x86_64.rpm 1.9 MB/s | 238 kB 00:00 2022-06-27T11:45:33.702 INFO:teuthology.orchestra.run.smithi195.stdout:(110/124): libuuid-devel-2.32.1-35.el8.x86_64.r 816 kB/s | 98 kB 00:00 2022-06-27T11:45:33.715 INFO:teuthology.orchestra.run.smithi195.stdout:(111/124): zlib-devel-1.2.11-18.el8_5.x86_64.rp 493 kB/s | 58 kB 00:00 2022-06-27T11:45:33.757 INFO:teuthology.orchestra.run.smithi195.stdout:(112/124): fuse-2.9.7-15.el8.x86_64.rpm 846 kB/s | 83 kB 00:00 2022-06-27T11:45:33.798 INFO:teuthology.orchestra.run.smithi195.stdout:(113/124): python3-jwt-1.6.1-2.el8.noarch.rpm 424 kB/s | 43 kB 00:00 2022-06-27T11:45:33.832 INFO:teuthology.orchestra.run.smithi195.stdout:(114/124): libcom_err-devel-1.45.6-4.el8.x86_64 340 kB/s | 39 kB 00:00 2022-06-27T11:45:33.874 INFO:teuthology.orchestra.run.smithi195.stdout:(115/124): elfutils-libelf-devel-0.186-1.el8.x8 527 kB/s | 60 kB 00:00 2022-06-27T11:45:33.908 INFO:teuthology.orchestra.run.smithi195.stdout:(116/124): libquadmath-8.5.0-10.el8.x86_64.rpm 1.5 MB/s | 171 kB 00:00 2022-06-27T11:45:33.933 INFO:teuthology.orchestra.run.smithi195.stdout:(117/124): keyutils-libs-devel-1.5.10-9.el8.x86 499 kB/s | 48 kB 00:00 2022-06-27T11:45:33.994 INFO:teuthology.orchestra.run.smithi195.stdout:(118/124): libgfortran-8.5.0-10.el8.x86_64.rpm 5.6 MB/s | 643 kB 00:00 2022-06-27T11:45:34.017 INFO:teuthology.orchestra.run.smithi195.stdout:(119/124): xfsprogs-devel-5.0.0-10.el8.x86_64.r 690 kB/s | 72 kB 00:00 2022-06-27T11:45:34.042 INFO:teuthology.orchestra.run.smithi195.stdout:(120/124): pcre2-utf16-10.32-2.el8.x86_64.rpm 2.1 MB/s | 229 kB 00:00 2022-06-27T11:45:34.084 INFO:teuthology.orchestra.run.smithi195.stdout:(121/124): python3-requests-oauthlib-1.0.0-1.el 464 kB/s | 43 kB 00:00 2022-06-27T11:45:34.135 INFO:teuthology.orchestra.run.smithi195.stdout:(122/124): openssl-devel-1.1.1k-6.el8_5.x86_64. 20 MB/s | 2.3 MB 00:00 2022-06-27T11:45:34.436 INFO:teuthology.orchestra.run.smithi195.stdout:(123/124): librados2-17.0.0-13241.g5292901b.el8 9.3 MB/s | 3.6 MB 00:00 2022-06-27T11:45:34.622 INFO:teuthology.orchestra.run.smithi195.stdout:(124/124): librbd1-17.0.0-13241.g5292901b.el8.x 7.2 MB/s | 3.9 MB 00:00 2022-06-27T11:45:34.623 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2022-06-27T11:45:34.623 INFO:teuthology.orchestra.run.smithi195.stdout:Total 14 MB/s | 204 MB 00:15 2022-06-27T11:45:35.309 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2022-06-27T11:45:35.421 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2022-06-27T11:45:35.421 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2022-06-27T11:45:36.466 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2022-06-27T11:45:36.470 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2022-06-27T11:45:38.437 INFO:teuthology.orchestra.run.smithi202.stdout: Preparing : 1/1 2022-06-27T11:45:38.634 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/126 2022-06-27T11:45:38.832 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/126 2022-06-27T11:45:39.013 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 3/126 2022-06-27T11:45:39.186 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/126 2022-06-27T11:45:39.571 INFO:teuthology.orchestra.run.smithi202.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/126 2022-06-27T11:45:39.802 INFO:teuthology.orchestra.run.smithi202.stdout: Upgrading : librados2-2:17.0.0-13241.g5292901b.el8.x86_64 5/126 2022-06-27T11:45:40.250 INFO:teuthology.orchestra.run.smithi202.stdout: Running scriptlet: librados2-2:17.0.0-13241.g5292901b.el8.x86_64 5/126 2022-06-27T11:45:40.460 INFO:teuthology.orchestra.run.smithi202.stdout: Upgrading : librbd1-2:17.0.0-13241.g5292901b.el8.x86_64 6/126 2022-06-27T11:45:40.585 INFO:teuthology.orchestra.run.smithi202.stdout: Running scriptlet: librbd1-2:17.0.0-13241.g5292901b.el8.x86_64 6/126 2022-06-27T11:45:40.785 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : libcephfs2-2:17.0.0-13241.g5292901b.el8.x86_64 7/126 2022-06-27T11:45:40.880 INFO:teuthology.orchestra.run.smithi202.stdout: Running scriptlet: libcephfs2-2:17.0.0-13241.g5292901b.el8.x86_64 7/126 2022-06-27T11:45:41.019 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2022-06-27T11:45:41.100 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : python3-rados-2:17.0.0-13241.g5292901b.el8.x86_6 8/126 2022-06-27T11:45:41.208 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/126 2022-06-27T11:45:41.244 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 9/126 2022-06-27T11:45:41.401 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/126 2022-06-27T11:45:41.590 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 3/126 2022-06-27T11:45:41.753 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/126 2022-06-27T11:45:41.823 INFO:teuthology.orchestra.run.smithi202.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 9/126 2022-06-27T11:45:42.126 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : librgw2-2:17.0.0-13241.g5292901b.el8.x86_64 10/126 2022-06-27T11:45:42.136 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/126 2022-06-27T11:45:42.238 INFO:teuthology.orchestra.run.smithi202.stdout: Running scriptlet: librgw2-2:17.0.0-13241.g5292901b.el8.x86_64 10/126 2022-06-27T11:45:42.413 INFO:teuthology.orchestra.run.smithi195.stdout: Upgrading : librados2-2:17.0.0-13241.g5292901b.el8.x86_64 5/126 2022-06-27T11:45:42.447 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/126 2022-06-27T11:45:42.617 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/126 2022-06-27T11:45:42.841 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librados2-2:17.0.0-13241.g5292901b.el8.x86_64 5/126 2022-06-27T11:45:42.843 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : libcephsqlite-2:17.0.0-13241.g5292901b.el8.x86_6 13/126 2022-06-27T11:45:42.938 INFO:teuthology.orchestra.run.smithi202.stdout: Running scriptlet: libcephsqlite-2:17.0.0-13241.g5292901b.el8.x86_6 13/126 2022-06-27T11:45:43.038 INFO:teuthology.orchestra.run.smithi195.stdout: Upgrading : librbd1-2:17.0.0-13241.g5292901b.el8.x86_64 6/126 2022-06-27T11:45:43.093 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : libradosstriper1-2:17.0.0-13241.g5292901b.el8.x8 14/126 2022-06-27T11:45:43.168 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librbd1-2:17.0.0-13241.g5292901b.el8.x86_64 6/126 2022-06-27T11:45:43.191 INFO:teuthology.orchestra.run.smithi202.stdout: Running scriptlet: libradosstriper1-2:17.0.0-13241.g5292901b.el8.x8 14/126 2022-06-27T11:45:43.343 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : libquadmath-8.5.0-10.el8.x86_64 15/126 2022-06-27T11:45:43.355 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libcephfs2-2:17.0.0-13241.g5292901b.el8.x86_64 7/126 2022-06-27T11:45:43.474 INFO:teuthology.orchestra.run.smithi202.stdout: Running scriptlet: libquadmath-8.5.0-10.el8.x86_64 15/126 2022-06-27T11:45:43.476 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: libcephfs2-2:17.0.0-13241.g5292901b.el8.x86_64 7/126 2022-06-27T11:45:43.610 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : libgfortran-8.5.0-10.el8.x86_64 16/126 2022-06-27T11:45:43.669 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-rados-2:17.0.0-13241.g5292901b.el8.x86_6 8/126 2022-06-27T11:45:43.831 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 9/126 2022-06-27T11:45:44.226 INFO:teuthology.orchestra.run.smithi202.stdout: Running scriptlet: libgfortran-8.5.0-10.el8.x86_64 16/126 2022-06-27T11:45:44.370 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 9/126 2022-06-27T11:45:44.410 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 17/126 2022-06-27T11:45:44.613 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : librgw2-2:17.0.0-13241.g5292901b.el8.x86_64 10/126 2022-06-27T11:45:44.719 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librgw2-2:17.0.0-13241.g5292901b.el8.x86_64 10/126 2022-06-27T11:45:44.908 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/126 2022-06-27T11:45:44.991 INFO:teuthology.orchestra.run.smithi202.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 17/126 2022-06-27T11:45:45.078 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/126 2022-06-27T11:45:45.185 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : openblas-0.3.15-3.el8.x86_64 18/126 2022-06-27T11:45:45.297 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libcephsqlite-2:17.0.0-13241.g5292901b.el8.x86_6 13/126 2022-06-27T11:45:45.405 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: libcephsqlite-2:17.0.0-13241.g5292901b.el8.x86_6 13/126 2022-06-27T11:45:45.563 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libradosstriper1-2:17.0.0-13241.g5292901b.el8.x8 14/126 2022-06-27T11:45:45.666 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: libradosstriper1-2:17.0.0-13241.g5292901b.el8.x8 14/126 2022-06-27T11:45:45.821 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libquadmath-8.5.0-10.el8.x86_64 15/126 2022-06-27T11:45:45.882 INFO:teuthology.orchestra.run.smithi202.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 18/126 2022-06-27T11:45:45.957 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: libquadmath-8.5.0-10.el8.x86_64 15/126 2022-06-27T11:45:46.088 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libgfortran-8.5.0-10.el8.x86_64 16/126 2022-06-27T11:45:46.166 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 19/126 2022-06-27T11:45:46.299 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 20/126 2022-06-27T11:45:46.579 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 21/126 2022-06-27T11:45:46.700 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: libgfortran-8.5.0-10.el8.x86_64 16/126 2022-06-27T11:45:46.796 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 22/126 2022-06-27T11:45:46.955 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 17/126 2022-06-27T11:45:47.044 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/126 2022-06-27T11:45:47.549 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 17/126 2022-06-27T11:45:47.723 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 24/126 2022-06-27T11:45:47.754 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : openblas-0.3.15-3.el8.x86_64 18/126 2022-06-27T11:45:47.979 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : python3-ceph-common-2:17.0.0-13241.g5292901b.el8 25/126 2022-06-27T11:45:48.094 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : python3-ceph-argparse-2:17.0.0-13241.g5292901b.e 26/126 2022-06-27T11:45:48.292 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : python3-cephfs-2:17.0.0-13241.g5292901b.el8.x86_ 27/126 2022-06-27T11:45:48.453 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 18/126 2022-06-27T11:45:48.694 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 19/126 2022-06-27T11:45:48.831 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 20/126 2022-06-27T11:45:49.088 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 21/126 2022-06-27T11:45:49.239 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 28/126 2022-06-27T11:45:49.291 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 22/126 2022-06-27T11:45:49.493 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/126 2022-06-27T11:45:49.566 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 29/126 2022-06-27T11:45:49.721 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 24/126 2022-06-27T11:45:49.857 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 30/126 2022-06-27T11:45:49.938 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-ceph-common-2:17.0.0-13241.g5292901b.el8 25/126 2022-06-27T11:45:50.067 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 31/126 2022-06-27T11:45:50.080 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-ceph-argparse-2:17.0.0-13241.g5292901b.e 26/126 2022-06-27T11:45:50.263 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cephfs-2:17.0.0-13241.g5292901b.el8.x86_ 27/126 2022-06-27T11:45:51.198 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 28/126 2022-06-27T11:45:51.493 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 29/126 2022-06-27T11:45:51.734 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 30/126 2022-06-27T11:45:51.821 INFO:teuthology.orchestra.run.smithi202.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 31/126 2022-06-27T11:45:51.938 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 31/126 2022-06-27T11:45:52.144 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 32/126 2022-06-27T11:45:52.378 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : python3-rsa-4.8-1.el8.noarch 33/126 2022-06-27T11:45:52.575 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 34/126 2022-06-27T11:45:52.834 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : python3-rgw-2:17.0.0-13241.g5292901b.el8.x86_64 35/126 2022-06-27T11:45:53.043 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : python3-rbd-2:17.0.0-13241.g5292901b.el8.x86_64 36/126 2022-06-27T11:45:53.211 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : librados-devel-2:17.0.0-13241.g5292901b.el8.x86_ 37/126 2022-06-27T11:45:53.432 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 38/126 2022-06-27T11:45:53.564 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 39/126 2022-06-27T11:45:53.675 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 31/126 2022-06-27T11:45:53.718 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 40/126 2022-06-27T11:45:53.907 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : fuse-2.9.7-15.el8.x86_64 41/126 2022-06-27T11:45:53.997 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 32/126 2022-06-27T11:45:54.081 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 42/126 2022-06-27T11:45:54.190 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-rsa-4.8-1.el8.noarch 33/126 2022-06-27T11:45:54.281 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 43/126 2022-06-27T11:45:54.361 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 34/126 2022-06-27T11:45:54.570 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-rgw-2:17.0.0-13241.g5292901b.el8.x86_64 35/126 2022-06-27T11:45:54.606 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 44/126 2022-06-27T11:45:54.732 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-rbd-2:17.0.0-13241.g5292901b.el8.x86_64 36/126 2022-06-27T11:45:54.874 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 45/126 2022-06-27T11:45:54.927 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : librados-devel-2:17.0.0-13241.g5292901b.el8.x86_ 37/126 2022-06-27T11:45:55.060 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 46/126 2022-06-27T11:45:55.099 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 38/126 2022-06-27T11:45:55.257 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 39/126 2022-06-27T11:45:55.271 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 47/126 2022-06-27T11:45:55.439 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 40/126 2022-06-27T11:45:55.546 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 48/126 2022-06-27T11:45:55.610 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : fuse-2.9.7-15.el8.x86_64 41/126 2022-06-27T11:45:55.771 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 49/126 2022-06-27T11:45:55.815 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 42/126 2022-06-27T11:45:56.014 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 43/126 2022-06-27T11:45:56.167 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 50/126 2022-06-27T11:45:56.317 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 44/126 2022-06-27T11:45:56.486 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 51/126 2022-06-27T11:45:56.582 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 45/126 2022-06-27T11:45:56.709 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 52/126 2022-06-27T11:45:56.780 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 46/126 2022-06-27T11:45:56.921 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 53/126 2022-06-27T11:45:56.959 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 47/126 2022-06-27T11:45:57.166 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 54/126 2022-06-27T11:45:57.240 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 48/126 2022-06-27T11:45:57.438 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 49/126 2022-06-27T11:45:57.500 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 55/126 2022-06-27T11:45:57.716 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 56/126 2022-06-27T11:45:57.842 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 50/126 2022-06-27T11:45:57.925 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 57/126 2022-06-27T11:45:58.139 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 51/126 2022-06-27T11:45:58.173 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 58/126 2022-06-27T11:45:58.316 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 59/126 2022-06-27T11:45:58.337 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 52/126 2022-06-27T11:45:58.398 INFO:teuthology.orchestra.run.smithi202.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 59/126 2022-06-27T11:45:58.547 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 53/126 2022-06-27T11:45:58.640 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : jq-1.6-3.el8.x86_64 60/126 2022-06-27T11:45:58.825 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 54/126 2022-06-27T11:45:58.935 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 61/126 2022-06-27T11:45:59.106 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : automake-1.16.1-7.el8.noarch 62/126 2022-06-27T11:45:59.117 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 55/126 2022-06-27T11:45:59.274 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 63/126 2022-06-27T11:45:59.319 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 56/126 2022-06-27T11:45:59.409 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 64/126 2022-06-27T11:45:59.470 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 57/126 2022-06-27T11:45:59.585 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 65/126 2022-06-27T11:45:59.680 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 58/126 2022-06-27T11:45:59.767 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 66/126 2022-06-27T11:45:59.876 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 59/126 2022-06-27T11:45:59.952 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 59/126 2022-06-27T11:45:59.972 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 67/126 2022-06-27T11:46:00.150 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : jq-1.6-3.el8.x86_64 60/126 2022-06-27T11:46:00.193 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 68/126 2022-06-27T11:46:00.406 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 69/126 2022-06-27T11:46:00.411 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 61/126 2022-06-27T11:46:00.529 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 70/126 2022-06-27T11:46:00.609 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : automake-1.16.1-7.el8.noarch 62/126 2022-06-27T11:46:00.719 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 71/126 2022-06-27T11:46:00.743 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 63/126 2022-06-27T11:46:00.911 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 64/126 2022-06-27T11:46:00.917 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 72/126 2022-06-27T11:46:01.054 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 65/126 2022-06-27T11:46:01.121 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 73/126 2022-06-27T11:46:01.236 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 66/126 2022-06-27T11:46:01.327 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 74/126 2022-06-27T11:46:01.399 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 67/126 2022-06-27T11:46:01.527 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 75/126 2022-06-27T11:46:01.613 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 68/126 2022-06-27T11:46:01.663 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 76/126 2022-06-27T11:46:01.801 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 69/126 2022-06-27T11:46:01.887 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 77/126 2022-06-27T11:46:01.965 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 70/126 2022-06-27T11:46:02.081 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 78/126 2022-06-27T11:46:02.139 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 71/126 2022-06-27T11:46:02.271 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 79/126 2022-06-27T11:46:02.329 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 72/126 2022-06-27T11:46:02.489 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : python3-portend-2.6-1.el8.noarch 80/126 2022-06-27T11:46:02.523 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 73/126 2022-06-27T11:46:02.664 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 81/126 2022-06-27T11:46:02.714 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 74/126 2022-06-27T11:46:02.831 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 82/126 2022-06-27T11:46:02.889 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 75/126 2022-06-27T11:46:03.050 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 76/126 2022-06-27T11:46:03.081 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 83/126 2022-06-27T11:46:03.241 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 77/126 2022-06-27T11:46:03.423 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 78/126 2022-06-27T11:46:03.557 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 79/126 2022-06-27T11:46:03.773 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-portend-2.6-1.el8.noarch 80/126 2022-06-27T11:46:03.984 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 81/126 2022-06-27T11:46:04.186 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 82/126 2022-06-27T11:46:04.187 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 84/126 2022-06-27T11:46:04.378 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 83/126 2022-06-27T11:46:04.452 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 85/126 2022-06-27T11:46:04.653 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 86/126 2022-06-27T11:46:04.991 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 87/126 2022-06-27T11:46:05.273 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 88/126 2022-06-27T11:46:05.448 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 84/126 2022-06-27T11:46:05.480 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 89/126 2022-06-27T11:46:05.657 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 90/126 2022-06-27T11:46:05.704 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 85/126 2022-06-27T11:46:05.909 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 86/126 2022-06-27T11:46:06.258 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 87/126 2022-06-27T11:46:06.518 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 88/126 2022-06-27T11:46:06.704 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 89/126 2022-06-27T11:46:06.860 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 90/126 2022-06-27T11:46:09.418 INFO:teuthology.orchestra.run.smithi202.stdout: Running scriptlet: ceph-common-2:17.0.0-13241.g5292901b.el8.x86_64 91/126 2022-06-27T11:46:09.648 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : ceph-common-2:17.0.0-13241.g5292901b.el8.x86_64 91/126 2022-06-27T11:46:10.215 INFO:teuthology.orchestra.run.smithi202.stdout: Running scriptlet: ceph-common-2:17.0.0-13241.g5292901b.el8.x86_64 91/126 2022-06-27T11:46:10.513 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : ceph-base-2:17.0.0-13241.g5292901b.el8.x86_64 92/126 2022-06-27T11:46:10.594 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-common-2:17.0.0-13241.g5292901b.el8.x86_64 91/126 2022-06-27T11:46:10.600 INFO:teuthology.orchestra.run.smithi202.stdout: Running scriptlet: ceph-base-2:17.0.0-13241.g5292901b.el8.x86_64 92/126 2022-06-27T11:46:10.714 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : ceph-selinux-2:17.0.0-13241.g5292901b.el8.x86_64 93/126 2022-06-27T11:46:10.818 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-common-2:17.0.0-13241.g5292901b.el8.x86_64 91/126 2022-06-27T11:46:11.383 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-common-2:17.0.0-13241.g5292901b.el8.x86_64 91/126 2022-06-27T11:46:11.664 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-base-2:17.0.0-13241.g5292901b.el8.x86_64 92/126 2022-06-27T11:46:11.775 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-base-2:17.0.0-13241.g5292901b.el8.x86_64 92/126 2022-06-27T11:46:11.884 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-selinux-2:17.0.0-13241.g5292901b.el8.x86_64 93/126 2022-06-27T11:46:29.195 INFO:teuthology.orchestra.run.smithi202.stdout: Running scriptlet: ceph-selinux-2:17.0.0-13241.g5292901b.el8.x86_64 93/126 2022-06-27T11:46:29.195 INFO:teuthology.orchestra.run.smithi202.stdout:skipping the directory /sys 2022-06-27T11:46:29.195 INFO:teuthology.orchestra.run.smithi202.stdout:skipping the directory /proc 2022-06-27T11:46:29.196 INFO:teuthology.orchestra.run.smithi202.stdout:skipping the directory /dev 2022-06-27T11:46:29.196 INFO:teuthology.orchestra.run.smithi202.stdout:skipping the directory /run 2022-06-27T11:46:29.196 INFO:teuthology.orchestra.run.smithi202.stdout:skipping the directory /mnt 2022-06-27T11:46:29.196 INFO:teuthology.orchestra.run.smithi202.stdout:skipping the directory /var/tmp 2022-06-27T11:46:29.197 INFO:teuthology.orchestra.run.smithi202.stdout:skipping the directory /home 2022-06-27T11:46:29.197 INFO:teuthology.orchestra.run.smithi202.stdout:skipping the directory /tmp 2022-06-27T11:46:29.197 INFO:teuthology.orchestra.run.smithi202.stdout:skipping the directory /dev 2022-06-27T11:46:29.197 INFO:teuthology.orchestra.run.smithi202.stdout: 2022-06-27T11:46:29.545 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : ceph-volume-2:17.0.0-13241.g5292901b.el8.noarch 94/126 2022-06-27T11:46:30.377 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-selinux-2:17.0.0-13241.g5292901b.el8.x86_64 93/126 2022-06-27T11:46:30.378 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /sys 2022-06-27T11:46:30.378 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /proc 2022-06-27T11:46:30.379 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /dev 2022-06-27T11:46:30.379 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /run 2022-06-27T11:46:30.380 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /mnt 2022-06-27T11:46:30.380 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /var/tmp 2022-06-27T11:46:30.381 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /home 2022-06-27T11:46:30.381 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /tmp 2022-06-27T11:46:30.382 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /dev 2022-06-27T11:46:30.383 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:46:30.715 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-volume-2:17.0.0-13241.g5292901b.el8.noarch 94/126 2022-06-27T11:46:31.054 INFO:teuthology.orchestra.run.smithi202.stdout: Running scriptlet: ceph-volume-2:17.0.0-13241.g5292901b.el8.noarch 94/126 2022-06-27T11:46:31.260 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : ceph-osd-2:17.0.0-13241.g5292901b.el8.x86_64 95/126 2022-06-27T11:46:31.482 INFO:teuthology.orchestra.run.smithi202.stdout: Running scriptlet: ceph-osd-2:17.0.0-13241.g5292901b.el8.x86_64 95/126 2022-06-27T11:46:31.635 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : ceph-mds-2:17.0.0-13241.g5292901b.el8.x86_64 96/126 2022-06-27T11:46:32.018 INFO:teuthology.orchestra.run.smithi202.stdout: Running scriptlet: ceph-mds-2:17.0.0-13241.g5292901b.el8.x86_64 96/126 2022-06-27T11:46:32.169 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : ceph-mon-2:17.0.0-13241.g5292901b.el8.x86_64 97/126 2022-06-27T11:46:32.187 INFO:teuthology.orchestra.run.smithi202.stdout: Running scriptlet: ceph-mon-2:17.0.0-13241.g5292901b.el8.x86_64 97/126 2022-06-27T11:46:32.227 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-volume-2:17.0.0-13241.g5292901b.el8.noarch 94/126 2022-06-27T11:46:32.386 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-osd-2:17.0.0-13241.g5292901b.el8.x86_64 95/126 2022-06-27T11:46:32.609 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-osd-2:17.0.0-13241.g5292901b.el8.x86_64 95/126 2022-06-27T11:46:32.764 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mds-2:17.0.0-13241.g5292901b.el8.x86_64 96/126 2022-06-27T11:46:33.148 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mds-2:17.0.0-13241.g5292901b.el8.x86_64 96/126 2022-06-27T11:46:33.314 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mon-2:17.0.0-13241.g5292901b.el8.x86_64 97/126 2022-06-27T11:46:33.331 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mon-2:17.0.0-13241.g5292901b.el8.x86_64 97/126 2022-06-27T11:46:33.823 INFO:teuthology.orchestra.run.smithi202.stdout: Running scriptlet: cephadm-2:17.0.0-13241.g5292901b.el8.noarch 98/126 2022-06-27T11:46:33.996 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : cephadm-2:17.0.0-13241.g5292901b.el8.noarch 98/126 2022-06-27T11:46:34.141 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : ceph-prometheus-alerts-2:17.0.0-13241.g5292901b. 99/126 2022-06-27T11:46:34.312 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : ceph-grafana-dashboards-2:17.0.0-13241.g5292901b 100/126 2022-06-27T11:46:34.480 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : ceph-mgr-cephadm-2:17.0.0-13241.g5292901b.el8.no 101/126 2022-06-27T11:46:34.683 INFO:teuthology.orchestra.run.smithi202.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.0.0-13241.g5292901b.el8.no 101/126 2022-06-27T11:46:34.810 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: cephadm-2:17.0.0-13241.g5292901b.el8.noarch 98/126 2022-06-27T11:46:34.965 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : cephadm-2:17.0.0-13241.g5292901b.el8.noarch 98/126 2022-06-27T11:46:35.026 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : ceph-mgr-dashboard-2:17.0.0-13241.g5292901b.el8. 102/126 2022-06-27T11:46:35.078 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-prometheus-alerts-2:17.0.0-13241.g5292901b. 99/126 2022-06-27T11:46:35.230 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-grafana-dashboards-2:17.0.0-13241.g5292901b 100/126 2022-06-27T11:46:35.390 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-cephadm-2:17.0.0-13241.g5292901b.el8.no 101/126 2022-06-27T11:46:35.586 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.0.0-13241.g5292901b.el8.no 101/126 2022-06-27T11:46:35.919 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-dashboard-2:17.0.0-13241.g5292901b.el8. 102/126 2022-06-27T11:46:35.984 INFO:teuthology.orchestra.run.smithi202.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.0.0-13241.g5292901b.el8. 102/126 2022-06-27T11:46:36.134 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : ceph-mgr-diskprediction-local-2:17.0.0-13241.g52 103/126 2022-06-27T11:46:36.229 INFO:teuthology.orchestra.run.smithi202.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.0.0-13241.g52 103/126 2022-06-27T11:46:36.629 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : ceph-mgr-modules-core-2:17.0.0-13241.g5292901b.e 104/126 2022-06-27T11:46:36.776 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : ceph-mgr-2:17.0.0-13241.g5292901b.el8.x86_64 105/126 2022-06-27T11:46:36.811 INFO:teuthology.orchestra.run.smithi202.stdout: Running scriptlet: ceph-mgr-2:17.0.0-13241.g5292901b.el8.x86_64 105/126 2022-06-27T11:46:36.874 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.0.0-13241.g5292901b.el8. 102/126 2022-06-27T11:46:36.993 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : ceph-mgr-rook-2:17.0.0-13241.g5292901b.el8.noarc 106/126 2022-06-27T11:46:37.004 INFO:teuthology.orchestra.run.smithi202.stdout: Running scriptlet: ceph-mgr-rook-2:17.0.0-13241.g5292901b.el8.noarc 106/126 2022-06-27T11:46:37.013 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-diskprediction-local-2:17.0.0-13241.g52 103/126 2022-06-27T11:46:37.106 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.0.0-13241.g52 103/126 2022-06-27T11:46:37.464 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-modules-core-2:17.0.0-13241.g5292901b.e 104/126 2022-06-27T11:46:37.635 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-2:17.0.0-13241.g5292901b.el8.x86_64 105/126 2022-06-27T11:46:37.669 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mgr-2:17.0.0-13241.g5292901b.el8.x86_64 105/126 2022-06-27T11:46:37.830 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-rook-2:17.0.0-13241.g5292901b.el8.noarc 106/126 2022-06-27T11:46:37.841 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mgr-rook-2:17.0.0-13241.g5292901b.el8.noarc 106/126 2022-06-27T11:46:38.566 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : ceph-2:17.0.0-13241.g5292901b.el8.x86_64 107/126 2022-06-27T11:46:38.876 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : ceph-radosgw-2:17.0.0-13241.g5292901b.el8.x86_64 108/126 2022-06-27T11:46:38.964 INFO:teuthology.orchestra.run.smithi202.stdout: Running scriptlet: ceph-radosgw-2:17.0.0-13241.g5292901b.el8.x86_64 108/126 2022-06-27T11:46:39.096 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : ceph-immutable-object-cache-2:17.0.0-13241.g5292 109/126 2022-06-27T11:46:39.140 INFO:teuthology.orchestra.run.smithi202.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.0.0-13241.g5292 109/126 2022-06-27T11:46:39.301 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : cephfs-mirror-2:17.0.0-13241.g5292901b.el8.x86_6 110/126 2022-06-27T11:46:39.425 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-2:17.0.0-13241.g5292901b.el8.x86_64 107/126 2022-06-27T11:46:39.686 INFO:teuthology.orchestra.run.smithi202.stdout: Running scriptlet: cephfs-mirror-2:17.0.0-13241.g5292901b.el8.x86_6 110/126 2022-06-27T11:46:39.687 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-radosgw-2:17.0.0-13241.g5292901b.el8.x86_64 108/126 2022-06-27T11:46:39.759 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-radosgw-2:17.0.0-13241.g5292901b.el8.x86_64 108/126 2022-06-27T11:46:39.829 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : rbd-mirror-2:17.0.0-13241.g5292901b.el8.x86_64 111/126 2022-06-27T11:46:39.913 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-immutable-object-cache-2:17.0.0-13241.g5292 109/126 2022-06-27T11:46:39.957 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.0.0-13241.g5292 109/126 2022-06-27T11:46:40.146 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : cephfs-mirror-2:17.0.0-13241.g5292901b.el8.x86_6 110/126 2022-06-27T11:46:40.531 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: cephfs-mirror-2:17.0.0-13241.g5292901b.el8.x86_6 110/126 2022-06-27T11:46:40.680 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : rbd-mirror-2:17.0.0-13241.g5292901b.el8.x86_64 111/126 2022-06-27T11:46:44.795 INFO:teuthology.orchestra.run.smithi202.stdout: Running scriptlet: rbd-mirror-2:17.0.0-13241.g5292901b.el8.x86_64 111/126 2022-06-27T11:46:45.114 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : ceph-test-2:17.0.0-13241.g5292901b.el8.x86_64 112/126 2022-06-27T11:46:45.278 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : libtool-2.4.6-25.el8.x86_64 113/126 2022-06-27T11:46:45.647 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: rbd-mirror-2:17.0.0-13241.g5292901b.el8.x86_64 111/126 2022-06-27T11:46:45.922 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-test-2:17.0.0-13241.g5292901b.el8.x86_64 112/126 2022-06-27T11:46:45.946 INFO:teuthology.orchestra.run.smithi202.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 113/126 2022-06-27T11:46:46.087 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libtool-2.4.6-25.el8.x86_64 113/126 2022-06-27T11:46:46.194 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : openssl-devel-1:1.1.1k-6.el8_5.x86_64 114/126 2022-06-27T11:46:46.434 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 115/126 2022-06-27T11:46:46.588 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : ceph-fuse-2:17.0.0-13241.g5292901b.el8.x86_64 116/126 2022-06-27T11:46:46.729 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : libcephfs-devel-2:17.0.0-13241.g5292901b.el8.x86 117/126 2022-06-27T11:46:46.753 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 113/126 2022-06-27T11:46:46.903 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 118/126 2022-06-27T11:46:47.006 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : openssl-devel-1:1.1.1k-6.el8_5.x86_64 114/126 2022-06-27T11:46:47.075 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : cephfs-top-2:17.0.0-13241.g5292901b.el8.noarch 119/126 2022-06-27T11:46:47.238 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 115/126 2022-06-27T11:46:47.267 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : rbd-fuse-2:17.0.0-13241.g5292901b.el8.x86_64 120/126 2022-06-27T11:46:47.384 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-fuse-2:17.0.0-13241.g5292901b.el8.x86_64 116/126 2022-06-27T11:46:47.451 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : rbd-nbd-2:17.0.0-13241.g5292901b.el8.x86_64 121/126 2022-06-27T11:46:47.508 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libcephfs-devel-2:17.0.0-13241.g5292901b.el8.x86 117/126 2022-06-27T11:46:47.641 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 122/126 2022-06-27T11:46:47.649 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 118/126 2022-06-27T11:46:47.791 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : flex-2.6.1-9.el8.x86_64 123/126 2022-06-27T11:46:47.828 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : cephfs-top-2:17.0.0-13241.g5292901b.el8.noarch 119/126 2022-06-27T11:46:47.873 INFO:teuthology.orchestra.run.smithi202.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 123/126 2022-06-27T11:46:47.990 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : rbd-fuse-2:17.0.0-13241.g5292901b.el8.x86_64 120/126 2022-06-27T11:46:48.066 INFO:teuthology.orchestra.run.smithi202.stdout: Installing : bison-3.0.4-10.el8.x86_64 124/126 2022-06-27T11:46:48.080 INFO:teuthology.orchestra.run.smithi202.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 124/126 2022-06-27T11:46:48.080 INFO:teuthology.orchestra.run.smithi202.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 125/126 2022-06-27T11:46:48.146 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : rbd-nbd-2:17.0.0-13241.g5292901b.el8.x86_64 121/126 2022-06-27T11:46:48.290 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 122/126 2022-06-27T11:46:48.311 INFO:teuthology.orchestra.run.smithi202.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 125/126 2022-06-27T11:46:48.312 INFO:teuthology.orchestra.run.smithi202.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 126/126 2022-06-27T11:46:48.437 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : flex-2.6.1-9.el8.x86_64 123/126 2022-06-27T11:46:48.511 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 123/126 2022-06-27T11:46:48.695 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : bison-3.0.4-10.el8.x86_64 124/126 2022-06-27T11:46:48.707 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 124/126 2022-06-27T11:46:48.708 INFO:teuthology.orchestra.run.smithi195.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 125/126 2022-06-27T11:46:48.914 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 125/126 2022-06-27T11:46:48.915 INFO:teuthology.orchestra.run.smithi195.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 126/126 2022-06-27T11:46:49.046 INFO:teuthology.orchestra.run.smithi202.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 126/126 2022-06-27T11:46:49.046 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : ceph-2:17.0.0-13241.g5292901b.el8.x86_64 1/126 2022-06-27T11:46:49.047 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : ceph-base-2:17.0.0-13241.g5292901b.el8.x86_64 2/126 2022-06-27T11:46:49.047 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : ceph-common-2:17.0.0-13241.g5292901b.el8.x86_64 3/126 2022-06-27T11:46:49.047 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : ceph-fuse-2:17.0.0-13241.g5292901b.el8.x86_64 4/126 2022-06-27T11:46:49.047 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : ceph-immutable-object-cache-2:17.0.0-13241.g5292 5/126 2022-06-27T11:46:49.048 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : ceph-mds-2:17.0.0-13241.g5292901b.el8.x86_64 6/126 2022-06-27T11:46:49.048 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : ceph-mgr-2:17.0.0-13241.g5292901b.el8.x86_64 7/126 2022-06-27T11:46:49.048 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : ceph-mon-2:17.0.0-13241.g5292901b.el8.x86_64 8/126 2022-06-27T11:46:49.048 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : ceph-osd-2:17.0.0-13241.g5292901b.el8.x86_64 9/126 2022-06-27T11:46:49.049 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : ceph-radosgw-2:17.0.0-13241.g5292901b.el8.x86_64 10/126 2022-06-27T11:46:49.049 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : ceph-selinux-2:17.0.0-13241.g5292901b.el8.x86_64 11/126 2022-06-27T11:46:49.050 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : ceph-test-2:17.0.0-13241.g5292901b.el8.x86_64 12/126 2022-06-27T11:46:49.051 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : cephfs-mirror-2:17.0.0-13241.g5292901b.el8.x86_6 13/126 2022-06-27T11:46:49.051 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : libcephfs-devel-2:17.0.0-13241.g5292901b.el8.x86 14/126 2022-06-27T11:46:49.051 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : libcephfs2-2:17.0.0-13241.g5292901b.el8.x86_64 15/126 2022-06-27T11:46:49.051 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : libcephsqlite-2:17.0.0-13241.g5292901b.el8.x86_6 16/126 2022-06-27T11:46:49.052 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : librados-devel-2:17.0.0-13241.g5292901b.el8.x86_ 17/126 2022-06-27T11:46:49.052 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : libradosstriper1-2:17.0.0-13241.g5292901b.el8.x8 18/126 2022-06-27T11:46:49.052 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : librgw2-2:17.0.0-13241.g5292901b.el8.x86_64 19/126 2022-06-27T11:46:49.052 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : python3-ceph-argparse-2:17.0.0-13241.g5292901b.e 20/126 2022-06-27T11:46:49.052 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : python3-ceph-common-2:17.0.0-13241.g5292901b.el8 21/126 2022-06-27T11:46:49.071 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : python3-cephfs-2:17.0.0-13241.g5292901b.el8.x86_ 22/126 2022-06-27T11:46:49.072 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : python3-rados-2:17.0.0-13241.g5292901b.el8.x86_6 23/126 2022-06-27T11:46:49.072 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : python3-rbd-2:17.0.0-13241.g5292901b.el8.x86_64 24/126 2022-06-27T11:46:49.073 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : python3-rgw-2:17.0.0-13241.g5292901b.el8.x86_64 25/126 2022-06-27T11:46:49.073 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : rbd-fuse-2:17.0.0-13241.g5292901b.el8.x86_64 26/126 2022-06-27T11:46:49.073 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : rbd-mirror-2:17.0.0-13241.g5292901b.el8.x86_64 27/126 2022-06-27T11:46:49.074 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : rbd-nbd-2:17.0.0-13241.g5292901b.el8.x86_64 28/126 2022-06-27T11:46:49.074 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : ceph-grafana-dashboards-2:17.0.0-13241.g5292901b 29/126 2022-06-27T11:46:49.075 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : ceph-mgr-cephadm-2:17.0.0-13241.g5292901b.el8.no 30/126 2022-06-27T11:46:49.075 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : ceph-mgr-dashboard-2:17.0.0-13241.g5292901b.el8. 31/126 2022-06-27T11:46:49.076 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.0.0-13241.g52 32/126 2022-06-27T11:46:49.076 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : ceph-mgr-modules-core-2:17.0.0-13241.g5292901b.e 33/126 2022-06-27T11:46:49.077 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : ceph-mgr-rook-2:17.0.0-13241.g5292901b.el8.noarc 34/126 2022-06-27T11:46:49.077 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : ceph-prometheus-alerts-2:17.0.0-13241.g5292901b. 35/126 2022-06-27T11:46:49.077 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : ceph-volume-2:17.0.0-13241.g5292901b.el8.noarch 36/126 2022-06-27T11:46:49.078 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : cephadm-2:17.0.0-13241.g5292901b.el8.noarch 37/126 2022-06-27T11:46:49.078 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : cephfs-top-2:17.0.0-13241.g5292901b.el8.noarch 38/126 2022-06-27T11:46:49.079 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/126 2022-06-27T11:46:49.079 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/126 2022-06-27T11:46:49.079 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/126 2022-06-27T11:46:49.080 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/126 2022-06-27T11:46:49.080 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/126 2022-06-27T11:46:49.081 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/126 2022-06-27T11:46:49.081 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/126 2022-06-27T11:46:49.081 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/126 2022-06-27T11:46:49.082 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/126 2022-06-27T11:46:49.083 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/126 2022-06-27T11:46:49.083 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/126 2022-06-27T11:46:49.083 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/126 2022-06-27T11:46:49.084 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/126 2022-06-27T11:46:49.084 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/126 2022-06-27T11:46:49.084 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/126 2022-06-27T11:46:49.084 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/126 2022-06-27T11:46:49.084 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/126 2022-06-27T11:46:49.085 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/126 2022-06-27T11:46:49.085 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/126 2022-06-27T11:46:49.085 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/126 2022-06-27T11:46:49.086 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/126 2022-06-27T11:46:49.086 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/126 2022-06-27T11:46:49.086 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/126 2022-06-27T11:46:49.086 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/126 2022-06-27T11:46:49.087 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/126 2022-06-27T11:46:49.087 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/126 2022-06-27T11:46:49.087 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 65/126 2022-06-27T11:46:49.087 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/126 2022-06-27T11:46:49.087 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/126 2022-06-27T11:46:49.088 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/126 2022-06-27T11:46:49.088 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/126 2022-06-27T11:46:49.089 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/126 2022-06-27T11:46:49.089 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 71/126 2022-06-27T11:46:49.090 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 72/126 2022-06-27T11:46:49.090 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 73/126 2022-06-27T11:46:49.091 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : bison-3.0.4-10.el8.x86_64 74/126 2022-06-27T11:46:49.091 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 75/126 2022-06-27T11:46:49.092 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 76/126 2022-06-27T11:46:49.092 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 77/126 2022-06-27T11:46:49.092 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 78/126 2022-06-27T11:46:49.093 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 79/126 2022-06-27T11:46:49.093 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 80/126 2022-06-27T11:46:49.094 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 81/126 2022-06-27T11:46:49.094 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 82/126 2022-06-27T11:46:49.095 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 83/126 2022-06-27T11:46:49.095 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 84/126 2022-06-27T11:46:49.095 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 85/126 2022-06-27T11:46:49.096 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : jq-1.6-3.el8.x86_64 86/126 2022-06-27T11:46:49.096 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 87/126 2022-06-27T11:46:49.097 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 88/126 2022-06-27T11:46:49.097 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : automake-1.16.1-7.el8.noarch 89/126 2022-06-27T11:46:49.098 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 90/126 2022-06-27T11:46:49.098 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 91/126 2022-06-27T11:46:49.098 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 92/126 2022-06-27T11:46:49.099 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : flex-2.6.1-9.el8.x86_64 93/126 2022-06-27T11:46:49.099 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 94/126 2022-06-27T11:46:49.100 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 95/126 2022-06-27T11:46:49.100 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 96/126 2022-06-27T11:46:49.101 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 97/126 2022-06-27T11:46:49.101 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 98/126 2022-06-27T11:46:49.101 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 99/126 2022-06-27T11:46:49.102 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 100/126 2022-06-27T11:46:49.102 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 101/126 2022-06-27T11:46:49.103 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 102/126 2022-06-27T11:46:49.103 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 103/126 2022-06-27T11:46:49.103 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 104/126 2022-06-27T11:46:49.104 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 105/126 2022-06-27T11:46:49.104 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 106/126 2022-06-27T11:46:49.105 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 107/126 2022-06-27T11:46:49.105 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 108/126 2022-06-27T11:46:49.105 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 109/126 2022-06-27T11:46:49.106 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 110/126 2022-06-27T11:46:49.106 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 111/126 2022-06-27T11:46:49.106 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 112/126 2022-06-27T11:46:49.107 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 113/126 2022-06-27T11:46:49.107 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 114/126 2022-06-27T11:46:49.108 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 115/126 2022-06-27T11:46:49.108 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : libquadmath-8.5.0-10.el8.x86_64 116/126 2022-06-27T11:46:49.108 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 117/126 2022-06-27T11:46:49.109 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : libgfortran-8.5.0-10.el8.x86_64 118/126 2022-06-27T11:46:49.109 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 119/126 2022-06-27T11:46:49.110 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 120/126 2022-06-27T11:46:49.110 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 121/126 2022-06-27T11:46:49.111 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : openssl-devel-1:1.1.1k-6.el8_5.x86_64 122/126 2022-06-27T11:46:49.111 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : librados2-2:17.0.0-13241.g5292901b.el8.x86_64 123/126 2022-06-27T11:46:49.111 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 124/126 2022-06-27T11:46:49.112 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : librbd1-2:17.0.0-13241.g5292901b.el8.x86_64 125/126 2022-06-27T11:46:49.611 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 126/126 2022-06-27T11:46:49.612 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-2:17.0.0-13241.g5292901b.el8.x86_64 1/126 2022-06-27T11:46:49.612 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-base-2:17.0.0-13241.g5292901b.el8.x86_64 2/126 2022-06-27T11:46:49.612 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-common-2:17.0.0-13241.g5292901b.el8.x86_64 3/126 2022-06-27T11:46:49.613 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-fuse-2:17.0.0-13241.g5292901b.el8.x86_64 4/126 2022-06-27T11:46:49.613 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-immutable-object-cache-2:17.0.0-13241.g5292 5/126 2022-06-27T11:46:49.613 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mds-2:17.0.0-13241.g5292901b.el8.x86_64 6/126 2022-06-27T11:46:49.614 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-2:17.0.0-13241.g5292901b.el8.x86_64 7/126 2022-06-27T11:46:49.614 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mon-2:17.0.0-13241.g5292901b.el8.x86_64 8/126 2022-06-27T11:46:49.614 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-osd-2:17.0.0-13241.g5292901b.el8.x86_64 9/126 2022-06-27T11:46:49.614 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-radosgw-2:17.0.0-13241.g5292901b.el8.x86_64 10/126 2022-06-27T11:46:49.615 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-selinux-2:17.0.0-13241.g5292901b.el8.x86_64 11/126 2022-06-27T11:46:49.615 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-test-2:17.0.0-13241.g5292901b.el8.x86_64 12/126 2022-06-27T11:46:49.615 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : cephfs-mirror-2:17.0.0-13241.g5292901b.el8.x86_6 13/126 2022-06-27T11:46:49.615 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libcephfs-devel-2:17.0.0-13241.g5292901b.el8.x86 14/126 2022-06-27T11:46:49.615 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libcephfs2-2:17.0.0-13241.g5292901b.el8.x86_64 15/126 2022-06-27T11:46:49.616 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libcephsqlite-2:17.0.0-13241.g5292901b.el8.x86_6 16/126 2022-06-27T11:46:49.616 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librados-devel-2:17.0.0-13241.g5292901b.el8.x86_ 17/126 2022-06-27T11:46:49.616 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libradosstriper1-2:17.0.0-13241.g5292901b.el8.x8 18/126 2022-06-27T11:46:49.616 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librgw2-2:17.0.0-13241.g5292901b.el8.x86_64 19/126 2022-06-27T11:46:49.617 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-ceph-argparse-2:17.0.0-13241.g5292901b.e 20/126 2022-06-27T11:46:49.617 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-ceph-common-2:17.0.0-13241.g5292901b.el8 21/126 2022-06-27T11:46:49.617 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cephfs-2:17.0.0-13241.g5292901b.el8.x86_ 22/126 2022-06-27T11:46:49.617 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-rados-2:17.0.0-13241.g5292901b.el8.x86_6 23/126 2022-06-27T11:46:49.617 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-rbd-2:17.0.0-13241.g5292901b.el8.x86_64 24/126 2022-06-27T11:46:49.618 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-rgw-2:17.0.0-13241.g5292901b.el8.x86_64 25/126 2022-06-27T11:46:49.618 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : rbd-fuse-2:17.0.0-13241.g5292901b.el8.x86_64 26/126 2022-06-27T11:46:49.618 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : rbd-mirror-2:17.0.0-13241.g5292901b.el8.x86_64 27/126 2022-06-27T11:46:49.618 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : rbd-nbd-2:17.0.0-13241.g5292901b.el8.x86_64 28/126 2022-06-27T11:46:49.619 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-grafana-dashboards-2:17.0.0-13241.g5292901b 29/126 2022-06-27T11:46:49.619 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-cephadm-2:17.0.0-13241.g5292901b.el8.no 30/126 2022-06-27T11:46:49.619 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-dashboard-2:17.0.0-13241.g5292901b.el8. 31/126 2022-06-27T11:46:49.619 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.0.0-13241.g52 32/126 2022-06-27T11:46:49.619 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-modules-core-2:17.0.0-13241.g5292901b.e 33/126 2022-06-27T11:46:49.620 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-rook-2:17.0.0-13241.g5292901b.el8.noarc 34/126 2022-06-27T11:46:49.620 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-prometheus-alerts-2:17.0.0-13241.g5292901b. 35/126 2022-06-27T11:46:49.620 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-volume-2:17.0.0-13241.g5292901b.el8.noarch 36/126 2022-06-27T11:46:49.620 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : cephadm-2:17.0.0-13241.g5292901b.el8.noarch 37/126 2022-06-27T11:46:49.621 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : cephfs-top-2:17.0.0-13241.g5292901b.el8.noarch 38/126 2022-06-27T11:46:49.621 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/126 2022-06-27T11:46:49.621 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/126 2022-06-27T11:46:49.621 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/126 2022-06-27T11:46:49.621 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/126 2022-06-27T11:46:49.622 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/126 2022-06-27T11:46:49.622 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/126 2022-06-27T11:46:49.622 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/126 2022-06-27T11:46:49.623 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/126 2022-06-27T11:46:49.623 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/126 2022-06-27T11:46:49.623 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/126 2022-06-27T11:46:49.624 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/126 2022-06-27T11:46:49.624 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/126 2022-06-27T11:46:49.624 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/126 2022-06-27T11:46:49.624 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/126 2022-06-27T11:46:49.624 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/126 2022-06-27T11:46:49.625 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/126 2022-06-27T11:46:49.625 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/126 2022-06-27T11:46:49.625 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/126 2022-06-27T11:46:49.625 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/126 2022-06-27T11:46:49.626 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/126 2022-06-27T11:46:49.626 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/126 2022-06-27T11:46:49.626 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/126 2022-06-27T11:46:49.626 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/126 2022-06-27T11:46:49.627 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/126 2022-06-27T11:46:49.627 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/126 2022-06-27T11:46:49.627 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/126 2022-06-27T11:46:49.627 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 65/126 2022-06-27T11:46:49.628 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/126 2022-06-27T11:46:49.628 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/126 2022-06-27T11:46:49.628 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/126 2022-06-27T11:46:49.628 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/126 2022-06-27T11:46:49.629 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/126 2022-06-27T11:46:49.629 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 71/126 2022-06-27T11:46:49.629 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 72/126 2022-06-27T11:46:49.629 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 73/126 2022-06-27T11:46:49.629 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : bison-3.0.4-10.el8.x86_64 74/126 2022-06-27T11:46:49.630 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 75/126 2022-06-27T11:46:49.630 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 76/126 2022-06-27T11:46:49.630 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 77/126 2022-06-27T11:46:49.630 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 78/126 2022-06-27T11:46:49.631 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 79/126 2022-06-27T11:46:49.631 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 80/126 2022-06-27T11:46:49.631 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 81/126 2022-06-27T11:46:49.631 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 82/126 2022-06-27T11:46:49.631 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 83/126 2022-06-27T11:46:49.632 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 84/126 2022-06-27T11:46:49.632 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 85/126 2022-06-27T11:46:49.632 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : jq-1.6-3.el8.x86_64 86/126 2022-06-27T11:46:49.632 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 87/126 2022-06-27T11:46:49.633 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 88/126 2022-06-27T11:46:49.633 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : automake-1.16.1-7.el8.noarch 89/126 2022-06-27T11:46:49.633 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 90/126 2022-06-27T11:46:49.633 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 91/126 2022-06-27T11:46:49.634 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 92/126 2022-06-27T11:46:49.634 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : flex-2.6.1-9.el8.x86_64 93/126 2022-06-27T11:46:49.634 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 94/126 2022-06-27T11:46:49.634 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 95/126 2022-06-27T11:46:49.634 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 96/126 2022-06-27T11:46:49.635 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 97/126 2022-06-27T11:46:49.635 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 98/126 2022-06-27T11:46:49.635 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 99/126 2022-06-27T11:46:49.635 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 100/126 2022-06-27T11:46:49.636 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 101/126 2022-06-27T11:46:49.636 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 102/126 2022-06-27T11:46:49.636 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 103/126 2022-06-27T11:46:49.636 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 104/126 2022-06-27T11:46:49.637 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 105/126 2022-06-27T11:46:49.637 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 106/126 2022-06-27T11:46:49.637 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 107/126 2022-06-27T11:46:49.637 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 108/126 2022-06-27T11:46:49.638 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 109/126 2022-06-27T11:46:49.638 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 110/126 2022-06-27T11:46:49.638 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 111/126 2022-06-27T11:46:49.638 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 112/126 2022-06-27T11:46:49.638 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 113/126 2022-06-27T11:46:49.639 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 114/126 2022-06-27T11:46:49.639 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 115/126 2022-06-27T11:46:49.639 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libquadmath-8.5.0-10.el8.x86_64 116/126 2022-06-27T11:46:49.639 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 117/126 2022-06-27T11:46:49.640 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libgfortran-8.5.0-10.el8.x86_64 118/126 2022-06-27T11:46:49.640 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 119/126 2022-06-27T11:46:49.640 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 120/126 2022-06-27T11:46:49.640 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 121/126 2022-06-27T11:46:49.640 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : openssl-devel-1:1.1.1k-6.el8_5.x86_64 122/126 2022-06-27T11:46:49.641 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librados2-2:17.0.0-13241.g5292901b.el8.x86_64 123/126 2022-06-27T11:46:49.641 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 124/126 2022-06-27T11:46:49.641 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librbd1-2:17.0.0-13241.g5292901b.el8.x86_64 125/126 2022-06-27T11:46:53.468 INFO:teuthology.orchestra.run.smithi202.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 126/126 2022-06-27T11:46:53.469 INFO:teuthology.orchestra.run.smithi202.stdout:Installed products updated. 2022-06-27T11:46:53.679 INFO:teuthology.orchestra.run.smithi202.stdout: 2022-06-27T11:46:53.680 INFO:teuthology.orchestra.run.smithi202.stdout:Upgraded: 2022-06-27T11:46:53.680 INFO:teuthology.orchestra.run.smithi202.stdout: librados2-2:17.0.0-13241.g5292901b.el8.x86_64 2022-06-27T11:46:53.680 INFO:teuthology.orchestra.run.smithi202.stdout: librbd1-2:17.0.0-13241.g5292901b.el8.x86_64 2022-06-27T11:46:53.681 INFO:teuthology.orchestra.run.smithi202.stdout:Installed: 2022-06-27T11:46:53.681 INFO:teuthology.orchestra.run.smithi202.stdout: automake-1.16.1-7.el8.noarch 2022-06-27T11:46:53.682 INFO:teuthology.orchestra.run.smithi202.stdout: bison-3.0.4-10.el8.x86_64 2022-06-27T11:46:53.682 INFO:teuthology.orchestra.run.smithi202.stdout: ceph-2:17.0.0-13241.g5292901b.el8.x86_64 2022-06-27T11:46:53.682 INFO:teuthology.orchestra.run.smithi202.stdout: ceph-base-2:17.0.0-13241.g5292901b.el8.x86_64 2022-06-27T11:46:53.683 INFO:teuthology.orchestra.run.smithi202.stdout: ceph-common-2:17.0.0-13241.g5292901b.el8.x86_64 2022-06-27T11:46:53.683 INFO:teuthology.orchestra.run.smithi202.stdout: ceph-fuse-2:17.0.0-13241.g5292901b.el8.x86_64 2022-06-27T11:46:53.683 INFO:teuthology.orchestra.run.smithi202.stdout: ceph-grafana-dashboards-2:17.0.0-13241.g5292901b.el8.noarch 2022-06-27T11:46:53.683 INFO:teuthology.orchestra.run.smithi202.stdout: ceph-immutable-object-cache-2:17.0.0-13241.g5292901b.el8.x86_64 2022-06-27T11:46:53.684 INFO:teuthology.orchestra.run.smithi202.stdout: ceph-mds-2:17.0.0-13241.g5292901b.el8.x86_64 2022-06-27T11:46:53.684 INFO:teuthology.orchestra.run.smithi202.stdout: ceph-mgr-2:17.0.0-13241.g5292901b.el8.x86_64 2022-06-27T11:46:53.684 INFO:teuthology.orchestra.run.smithi202.stdout: ceph-mgr-cephadm-2:17.0.0-13241.g5292901b.el8.noarch 2022-06-27T11:46:53.685 INFO:teuthology.orchestra.run.smithi202.stdout: ceph-mgr-dashboard-2:17.0.0-13241.g5292901b.el8.noarch 2022-06-27T11:46:53.685 INFO:teuthology.orchestra.run.smithi202.stdout: ceph-mgr-diskprediction-local-2:17.0.0-13241.g5292901b.el8.noarch 2022-06-27T11:46:53.685 INFO:teuthology.orchestra.run.smithi202.stdout: ceph-mgr-modules-core-2:17.0.0-13241.g5292901b.el8.noarch 2022-06-27T11:46:53.685 INFO:teuthology.orchestra.run.smithi202.stdout: ceph-mgr-rook-2:17.0.0-13241.g5292901b.el8.noarch 2022-06-27T11:46:53.686 INFO:teuthology.orchestra.run.smithi202.stdout: ceph-mon-2:17.0.0-13241.g5292901b.el8.x86_64 2022-06-27T11:46:53.686 INFO:teuthology.orchestra.run.smithi202.stdout: ceph-osd-2:17.0.0-13241.g5292901b.el8.x86_64 2022-06-27T11:46:53.686 INFO:teuthology.orchestra.run.smithi202.stdout: ceph-prometheus-alerts-2:17.0.0-13241.g5292901b.el8.noarch 2022-06-27T11:46:53.686 INFO:teuthology.orchestra.run.smithi202.stdout: ceph-radosgw-2:17.0.0-13241.g5292901b.el8.x86_64 2022-06-27T11:46:53.687 INFO:teuthology.orchestra.run.smithi202.stdout: ceph-selinux-2:17.0.0-13241.g5292901b.el8.x86_64 2022-06-27T11:46:53.687 INFO:teuthology.orchestra.run.smithi202.stdout: ceph-test-2:17.0.0-13241.g5292901b.el8.x86_64 2022-06-27T11:46:53.687 INFO:teuthology.orchestra.run.smithi202.stdout: ceph-volume-2:17.0.0-13241.g5292901b.el8.noarch 2022-06-27T11:46:53.688 INFO:teuthology.orchestra.run.smithi202.stdout: cephadm-2:17.0.0-13241.g5292901b.el8.noarch 2022-06-27T11:46:53.688 INFO:teuthology.orchestra.run.smithi202.stdout: cephfs-mirror-2:17.0.0-13241.g5292901b.el8.x86_64 2022-06-27T11:46:53.688 INFO:teuthology.orchestra.run.smithi202.stdout: cephfs-top-2:17.0.0-13241.g5292901b.el8.noarch 2022-06-27T11:46:53.688 INFO:teuthology.orchestra.run.smithi202.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2022-06-27T11:46:53.689 INFO:teuthology.orchestra.run.smithi202.stdout: flex-2.6.1-9.el8.x86_64 2022-06-27T11:46:53.689 INFO:teuthology.orchestra.run.smithi202.stdout: fuse-2.9.7-15.el8.x86_64 2022-06-27T11:46:53.689 INFO:teuthology.orchestra.run.smithi202.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2022-06-27T11:46:53.690 INFO:teuthology.orchestra.run.smithi202.stdout: jq-1.6-3.el8.x86_64 2022-06-27T11:46:53.690 INFO:teuthology.orchestra.run.smithi202.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2022-06-27T11:46:53.691 INFO:teuthology.orchestra.run.smithi202.stdout: krb5-devel-1.18.2-14.el8.x86_64 2022-06-27T11:46:53.691 INFO:teuthology.orchestra.run.smithi202.stdout: libaio-devel-0.3.112-1.el8.x86_64 2022-06-27T11:46:53.691 INFO:teuthology.orchestra.run.smithi202.stdout: libcephfs-devel-2:17.0.0-13241.g5292901b.el8.x86_64 2022-06-27T11:46:53.692 INFO:teuthology.orchestra.run.smithi202.stdout: libcephfs2-2:17.0.0-13241.g5292901b.el8.x86_64 2022-06-27T11:46:53.692 INFO:teuthology.orchestra.run.smithi202.stdout: libcephsqlite-2:17.0.0-13241.g5292901b.el8.x86_64 2022-06-27T11:46:53.693 INFO:teuthology.orchestra.run.smithi202.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2022-06-27T11:46:53.693 INFO:teuthology.orchestra.run.smithi202.stdout: libgfortran-8.5.0-10.el8.x86_64 2022-06-27T11:46:53.693 INFO:teuthology.orchestra.run.smithi202.stdout: liboath-2.6.2-3.el8.x86_64 2022-06-27T11:46:53.693 INFO:teuthology.orchestra.run.smithi202.stdout: libquadmath-8.5.0-10.el8.x86_64 2022-06-27T11:46:53.694 INFO:teuthology.orchestra.run.smithi202.stdout: librabbitmq-0.9.0-3.el8.x86_64 2022-06-27T11:46:53.694 INFO:teuthology.orchestra.run.smithi202.stdout: librados-devel-2:17.0.0-13241.g5292901b.el8.x86_64 2022-06-27T11:46:53.694 INFO:teuthology.orchestra.run.smithi202.stdout: libradosstriper1-2:17.0.0-13241.g5292901b.el8.x86_64 2022-06-27T11:46:53.695 INFO:teuthology.orchestra.run.smithi202.stdout: librdkafka-0.11.4-3.el8.x86_64 2022-06-27T11:46:53.695 INFO:teuthology.orchestra.run.smithi202.stdout: librgw2-2:17.0.0-13241.g5292901b.el8.x86_64 2022-06-27T11:46:53.695 INFO:teuthology.orchestra.run.smithi202.stdout: libselinux-devel-2.9-5.el8.x86_64 2022-06-27T11:46:53.695 INFO:teuthology.orchestra.run.smithi202.stdout: libsepol-devel-2.9-3.el8.x86_64 2022-06-27T11:46:53.696 INFO:teuthology.orchestra.run.smithi202.stdout: libtool-2.4.6-25.el8.x86_64 2022-06-27T11:46:53.697 INFO:teuthology.orchestra.run.smithi202.stdout: libunwind-1.3.1-3.el8.x86_64 2022-06-27T11:46:53.697 INFO:teuthology.orchestra.run.smithi202.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2022-06-27T11:46:53.698 INFO:teuthology.orchestra.run.smithi202.stdout: libverto-devel-0.3.0-5.el8.x86_64 2022-06-27T11:46:53.698 INFO:teuthology.orchestra.run.smithi202.stdout: lttng-ust-2.8.1-11.el8.x86_64 2022-06-27T11:46:53.698 INFO:teuthology.orchestra.run.smithi202.stdout: oniguruma-6.8.2-2.el8.x86_64 2022-06-27T11:46:53.699 INFO:teuthology.orchestra.run.smithi202.stdout: openblas-0.3.15-3.el8.x86_64 2022-06-27T11:46:53.699 INFO:teuthology.orchestra.run.smithi202.stdout: openblas-threads-0.3.15-3.el8.x86_64 2022-06-27T11:46:53.699 INFO:teuthology.orchestra.run.smithi202.stdout: openssl-devel-1:1.1.1k-6.el8_5.x86_64 2022-06-27T11:46:53.700 INFO:teuthology.orchestra.run.smithi202.stdout: pcre2-devel-10.32-2.el8.x86_64 2022-06-27T11:46:53.700 INFO:teuthology.orchestra.run.smithi202.stdout: pcre2-utf16-10.32-2.el8.x86_64 2022-06-27T11:46:53.700 INFO:teuthology.orchestra.run.smithi202.stdout: pcre2-utf32-10.32-2.el8.x86_64 2022-06-27T11:46:53.701 INFO:teuthology.orchestra.run.smithi202.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2022-06-27T11:46:53.701 INFO:teuthology.orchestra.run.smithi202.stdout: python3-babel-2.5.1-7.el8.noarch 2022-06-27T11:46:53.701 INFO:teuthology.orchestra.run.smithi202.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2022-06-27T11:46:53.701 INFO:teuthology.orchestra.run.smithi202.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2022-06-27T11:46:53.701 INFO:teuthology.orchestra.run.smithi202.stdout: python3-cachetools-3.1.1-4.el8.noarch 2022-06-27T11:46:53.702 INFO:teuthology.orchestra.run.smithi202.stdout: python3-ceph-argparse-2:17.0.0-13241.g5292901b.el8.x86_64 2022-06-27T11:46:53.702 INFO:teuthology.orchestra.run.smithi202.stdout: python3-ceph-common-2:17.0.0-13241.g5292901b.el8.x86_64 2022-06-27T11:46:53.702 INFO:teuthology.orchestra.run.smithi202.stdout: python3-cephfs-2:17.0.0-13241.g5292901b.el8.x86_64 2022-06-27T11:46:53.702 INFO:teuthology.orchestra.run.smithi202.stdout: python3-certifi-2018.10.15-7.el8.noarch 2022-06-27T11:46:53.703 INFO:teuthology.orchestra.run.smithi202.stdout: python3-cffi-1.11.5-5.el8.x86_64 2022-06-27T11:46:53.703 INFO:teuthology.orchestra.run.smithi202.stdout: python3-cheroot-8.5.2-1.el8.noarch 2022-06-27T11:46:53.703 INFO:teuthology.orchestra.run.smithi202.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2022-06-27T11:46:53.703 INFO:teuthology.orchestra.run.smithi202.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2022-06-27T11:46:53.704 INFO:teuthology.orchestra.run.smithi202.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2022-06-27T11:46:53.704 INFO:teuthology.orchestra.run.smithi202.stdout: python3-jaraco-6.2-6.el8.noarch 2022-06-27T11:46:53.704 INFO:teuthology.orchestra.run.smithi202.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2022-06-27T11:46:53.704 INFO:teuthology.orchestra.run.smithi202.stdout: python3-jinja2-2.10.1-3.el8.noarch 2022-06-27T11:46:53.704 INFO:teuthology.orchestra.run.smithi202.stdout: python3-jsonpatch-1.21-2.el8.noarch 2022-06-27T11:46:53.705 INFO:teuthology.orchestra.run.smithi202.stdout: python3-jsonpointer-1.10-11.el8.noarch 2022-06-27T11:46:53.705 INFO:teuthology.orchestra.run.smithi202.stdout: python3-jwt-1.6.1-2.el8.noarch 2022-06-27T11:46:53.705 INFO:teuthology.orchestra.run.smithi202.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2022-06-27T11:46:53.705 INFO:teuthology.orchestra.run.smithi202.stdout: python3-logutils-0.3.5-11.el8.noarch 2022-06-27T11:46:53.706 INFO:teuthology.orchestra.run.smithi202.stdout: python3-mako-1.0.6-13.el8.noarch 2022-06-27T11:46:53.706 INFO:teuthology.orchestra.run.smithi202.stdout: python3-markupsafe-0.23-19.el8.x86_64 2022-06-27T11:46:53.706 INFO:teuthology.orchestra.run.smithi202.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2022-06-27T11:46:53.706 INFO:teuthology.orchestra.run.smithi202.stdout: python3-natsort-7.1.1-2.el8.noarch 2022-06-27T11:46:53.707 INFO:teuthology.orchestra.run.smithi202.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2022-06-27T11:46:53.707 INFO:teuthology.orchestra.run.smithi202.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2022-06-27T11:46:53.707 INFO:teuthology.orchestra.run.smithi202.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2022-06-27T11:46:53.707 INFO:teuthology.orchestra.run.smithi202.stdout: python3-pecan-1.3.2-9.el8.noarch 2022-06-27T11:46:53.707 INFO:teuthology.orchestra.run.smithi202.stdout: python3-portend-2.6-1.el8.noarch 2022-06-27T11:46:53.708 INFO:teuthology.orchestra.run.smithi202.stdout: python3-prettytable-0.7.2-14.el8.noarch 2022-06-27T11:46:53.708 INFO:teuthology.orchestra.run.smithi202.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2022-06-27T11:46:53.708 INFO:teuthology.orchestra.run.smithi202.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2022-06-27T11:46:53.708 INFO:teuthology.orchestra.run.smithi202.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2022-06-27T11:46:53.709 INFO:teuthology.orchestra.run.smithi202.stdout: python3-pycparser-2.14-14.el8.noarch 2022-06-27T11:46:53.709 INFO:teuthology.orchestra.run.smithi202.stdout: python3-pytz-2017.2-9.el8.noarch 2022-06-27T11:46:53.709 INFO:teuthology.orchestra.run.smithi202.stdout: python3-rados-2:17.0.0-13241.g5292901b.el8.x86_64 2022-06-27T11:46:53.709 INFO:teuthology.orchestra.run.smithi202.stdout: python3-rbd-2:17.0.0-13241.g5292901b.el8.x86_64 2022-06-27T11:46:53.709 INFO:teuthology.orchestra.run.smithi202.stdout: python3-repoze-lru-0.7-6.el8.noarch 2022-06-27T11:46:53.710 INFO:teuthology.orchestra.run.smithi202.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2022-06-27T11:46:53.710 INFO:teuthology.orchestra.run.smithi202.stdout: python3-rgw-2:17.0.0-13241.g5292901b.el8.x86_64 2022-06-27T11:46:53.710 INFO:teuthology.orchestra.run.smithi202.stdout: python3-routes-2.4.1-12.el8.noarch 2022-06-27T11:46:53.710 INFO:teuthology.orchestra.run.smithi202.stdout: python3-rsa-4.8-1.el8.noarch 2022-06-27T11:46:53.710 INFO:teuthology.orchestra.run.smithi202.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2022-06-27T11:46:53.711 INFO:teuthology.orchestra.run.smithi202.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2022-06-27T11:46:53.711 INFO:teuthology.orchestra.run.smithi202.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2022-06-27T11:46:53.711 INFO:teuthology.orchestra.run.smithi202.stdout: python3-tempora-1.14.1-5.el8.noarch 2022-06-27T11:46:53.711 INFO:teuthology.orchestra.run.smithi202.stdout: python3-trustme-0.6.0-4.el8.noarch 2022-06-27T11:46:53.712 INFO:teuthology.orchestra.run.smithi202.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2022-06-27T11:46:53.712 INFO:teuthology.orchestra.run.smithi202.stdout: python3-webob-1.8.5-1.el8.1.noarch 2022-06-27T11:46:53.712 INFO:teuthology.orchestra.run.smithi202.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2022-06-27T11:46:53.712 INFO:teuthology.orchestra.run.smithi202.stdout: python3-webtest-2.0.33-1.el8.noarch 2022-06-27T11:46:53.712 INFO:teuthology.orchestra.run.smithi202.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2022-06-27T11:46:53.713 INFO:teuthology.orchestra.run.smithi202.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2022-06-27T11:46:53.713 INFO:teuthology.orchestra.run.smithi202.stdout: rbd-fuse-2:17.0.0-13241.g5292901b.el8.x86_64 2022-06-27T11:46:53.713 INFO:teuthology.orchestra.run.smithi202.stdout: rbd-mirror-2:17.0.0-13241.g5292901b.el8.x86_64 2022-06-27T11:46:53.713 INFO:teuthology.orchestra.run.smithi202.stdout: rbd-nbd-2:17.0.0-13241.g5292901b.el8.x86_64 2022-06-27T11:46:53.713 INFO:teuthology.orchestra.run.smithi202.stdout: socat-1.7.4.1-1.el8.x86_64 2022-06-27T11:46:53.714 INFO:teuthology.orchestra.run.smithi202.stdout: thrift-0.13.0-2.el8.x86_64 2022-06-27T11:46:53.714 INFO:teuthology.orchestra.run.smithi202.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2022-06-27T11:46:53.714 INFO:teuthology.orchestra.run.smithi202.stdout: xmlstarlet-1.6.1-11.el8.x86_64 2022-06-27T11:46:53.715 INFO:teuthology.orchestra.run.smithi202.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2022-06-27T11:46:53.715 INFO:teuthology.orchestra.run.smithi202.stdout: 2022-06-27T11:46:53.715 INFO:teuthology.orchestra.run.smithi202.stdout:Complete! 2022-06-27T11:46:53.948 DEBUG:teuthology.parallel:result is None 2022-06-27T11:46:53.950 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 126/126 2022-06-27T11:46:53.951 INFO:teuthology.orchestra.run.smithi195.stdout:Installed products updated. 2022-06-27T11:46:54.167 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:46:54.167 INFO:teuthology.orchestra.run.smithi195.stdout:Upgraded: 2022-06-27T11:46:54.167 INFO:teuthology.orchestra.run.smithi195.stdout: librados2-2:17.0.0-13241.g5292901b.el8.x86_64 2022-06-27T11:46:54.168 INFO:teuthology.orchestra.run.smithi195.stdout: librbd1-2:17.0.0-13241.g5292901b.el8.x86_64 2022-06-27T11:46:54.168 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2022-06-27T11:46:54.168 INFO:teuthology.orchestra.run.smithi195.stdout: automake-1.16.1-7.el8.noarch 2022-06-27T11:46:54.168 INFO:teuthology.orchestra.run.smithi195.stdout: bison-3.0.4-10.el8.x86_64 2022-06-27T11:46:54.169 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-2:17.0.0-13241.g5292901b.el8.x86_64 2022-06-27T11:46:54.169 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-base-2:17.0.0-13241.g5292901b.el8.x86_64 2022-06-27T11:46:54.169 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-common-2:17.0.0-13241.g5292901b.el8.x86_64 2022-06-27T11:46:54.169 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-fuse-2:17.0.0-13241.g5292901b.el8.x86_64 2022-06-27T11:46:54.170 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-grafana-dashboards-2:17.0.0-13241.g5292901b.el8.noarch 2022-06-27T11:46:54.170 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-immutable-object-cache-2:17.0.0-13241.g5292901b.el8.x86_64 2022-06-27T11:46:54.170 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mds-2:17.0.0-13241.g5292901b.el8.x86_64 2022-06-27T11:46:54.170 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-2:17.0.0-13241.g5292901b.el8.x86_64 2022-06-27T11:46:54.171 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-cephadm-2:17.0.0-13241.g5292901b.el8.noarch 2022-06-27T11:46:54.171 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-dashboard-2:17.0.0-13241.g5292901b.el8.noarch 2022-06-27T11:46:54.171 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-diskprediction-local-2:17.0.0-13241.g5292901b.el8.noarch 2022-06-27T11:46:54.171 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-modules-core-2:17.0.0-13241.g5292901b.el8.noarch 2022-06-27T11:46:54.172 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-rook-2:17.0.0-13241.g5292901b.el8.noarch 2022-06-27T11:46:54.172 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mon-2:17.0.0-13241.g5292901b.el8.x86_64 2022-06-27T11:46:54.172 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-osd-2:17.0.0-13241.g5292901b.el8.x86_64 2022-06-27T11:46:54.172 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-prometheus-alerts-2:17.0.0-13241.g5292901b.el8.noarch 2022-06-27T11:46:54.172 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-radosgw-2:17.0.0-13241.g5292901b.el8.x86_64 2022-06-27T11:46:54.173 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-selinux-2:17.0.0-13241.g5292901b.el8.x86_64 2022-06-27T11:46:54.173 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-test-2:17.0.0-13241.g5292901b.el8.x86_64 2022-06-27T11:46:54.173 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-volume-2:17.0.0-13241.g5292901b.el8.noarch 2022-06-27T11:46:54.174 INFO:teuthology.orchestra.run.smithi195.stdout: cephadm-2:17.0.0-13241.g5292901b.el8.noarch 2022-06-27T11:46:54.174 INFO:teuthology.orchestra.run.smithi195.stdout: cephfs-mirror-2:17.0.0-13241.g5292901b.el8.x86_64 2022-06-27T11:46:54.174 INFO:teuthology.orchestra.run.smithi195.stdout: cephfs-top-2:17.0.0-13241.g5292901b.el8.noarch 2022-06-27T11:46:54.175 INFO:teuthology.orchestra.run.smithi195.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2022-06-27T11:46:54.175 INFO:teuthology.orchestra.run.smithi195.stdout: flex-2.6.1-9.el8.x86_64 2022-06-27T11:46:54.175 INFO:teuthology.orchestra.run.smithi195.stdout: fuse-2.9.7-15.el8.x86_64 2022-06-27T11:46:54.175 INFO:teuthology.orchestra.run.smithi195.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2022-06-27T11:46:54.176 INFO:teuthology.orchestra.run.smithi195.stdout: jq-1.6-3.el8.x86_64 2022-06-27T11:46:54.176 INFO:teuthology.orchestra.run.smithi195.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2022-06-27T11:46:54.176 INFO:teuthology.orchestra.run.smithi195.stdout: krb5-devel-1.18.2-14.el8.x86_64 2022-06-27T11:46:54.176 INFO:teuthology.orchestra.run.smithi195.stdout: libaio-devel-0.3.112-1.el8.x86_64 2022-06-27T11:46:54.177 INFO:teuthology.orchestra.run.smithi195.stdout: libcephfs-devel-2:17.0.0-13241.g5292901b.el8.x86_64 2022-06-27T11:46:54.177 INFO:teuthology.orchestra.run.smithi195.stdout: libcephfs2-2:17.0.0-13241.g5292901b.el8.x86_64 2022-06-27T11:46:54.177 INFO:teuthology.orchestra.run.smithi195.stdout: libcephsqlite-2:17.0.0-13241.g5292901b.el8.x86_64 2022-06-27T11:46:54.177 INFO:teuthology.orchestra.run.smithi195.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2022-06-27T11:46:54.178 INFO:teuthology.orchestra.run.smithi195.stdout: libgfortran-8.5.0-10.el8.x86_64 2022-06-27T11:46:54.178 INFO:teuthology.orchestra.run.smithi195.stdout: liboath-2.6.2-3.el8.x86_64 2022-06-27T11:46:54.178 INFO:teuthology.orchestra.run.smithi195.stdout: libquadmath-8.5.0-10.el8.x86_64 2022-06-27T11:46:54.178 INFO:teuthology.orchestra.run.smithi195.stdout: librabbitmq-0.9.0-3.el8.x86_64 2022-06-27T11:46:54.178 INFO:teuthology.orchestra.run.smithi195.stdout: librados-devel-2:17.0.0-13241.g5292901b.el8.x86_64 2022-06-27T11:46:54.179 INFO:teuthology.orchestra.run.smithi195.stdout: libradosstriper1-2:17.0.0-13241.g5292901b.el8.x86_64 2022-06-27T11:46:54.179 INFO:teuthology.orchestra.run.smithi195.stdout: librdkafka-0.11.4-3.el8.x86_64 2022-06-27T11:46:54.179 INFO:teuthology.orchestra.run.smithi195.stdout: librgw2-2:17.0.0-13241.g5292901b.el8.x86_64 2022-06-27T11:46:54.179 INFO:teuthology.orchestra.run.smithi195.stdout: libselinux-devel-2.9-5.el8.x86_64 2022-06-27T11:46:54.180 INFO:teuthology.orchestra.run.smithi195.stdout: libsepol-devel-2.9-3.el8.x86_64 2022-06-27T11:46:54.180 INFO:teuthology.orchestra.run.smithi195.stdout: libtool-2.4.6-25.el8.x86_64 2022-06-27T11:46:54.180 INFO:teuthology.orchestra.run.smithi195.stdout: libunwind-1.3.1-3.el8.x86_64 2022-06-27T11:46:54.180 INFO:teuthology.orchestra.run.smithi195.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2022-06-27T11:46:54.181 INFO:teuthology.orchestra.run.smithi195.stdout: libverto-devel-0.3.0-5.el8.x86_64 2022-06-27T11:46:54.181 INFO:teuthology.orchestra.run.smithi195.stdout: lttng-ust-2.8.1-11.el8.x86_64 2022-06-27T11:46:54.181 INFO:teuthology.orchestra.run.smithi195.stdout: oniguruma-6.8.2-2.el8.x86_64 2022-06-27T11:46:54.181 INFO:teuthology.orchestra.run.smithi195.stdout: openblas-0.3.15-3.el8.x86_64 2022-06-27T11:46:54.182 INFO:teuthology.orchestra.run.smithi195.stdout: openblas-threads-0.3.15-3.el8.x86_64 2022-06-27T11:46:54.182 INFO:teuthology.orchestra.run.smithi195.stdout: openssl-devel-1:1.1.1k-6.el8_5.x86_64 2022-06-27T11:46:54.182 INFO:teuthology.orchestra.run.smithi195.stdout: pcre2-devel-10.32-2.el8.x86_64 2022-06-27T11:46:54.182 INFO:teuthology.orchestra.run.smithi195.stdout: pcre2-utf16-10.32-2.el8.x86_64 2022-06-27T11:46:54.183 INFO:teuthology.orchestra.run.smithi195.stdout: pcre2-utf32-10.32-2.el8.x86_64 2022-06-27T11:46:54.183 INFO:teuthology.orchestra.run.smithi195.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2022-06-27T11:46:54.183 INFO:teuthology.orchestra.run.smithi195.stdout: python3-babel-2.5.1-7.el8.noarch 2022-06-27T11:46:54.183 INFO:teuthology.orchestra.run.smithi195.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2022-06-27T11:46:54.183 INFO:teuthology.orchestra.run.smithi195.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2022-06-27T11:46:54.184 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cachetools-3.1.1-4.el8.noarch 2022-06-27T11:46:54.184 INFO:teuthology.orchestra.run.smithi195.stdout: python3-ceph-argparse-2:17.0.0-13241.g5292901b.el8.x86_64 2022-06-27T11:46:54.184 INFO:teuthology.orchestra.run.smithi195.stdout: python3-ceph-common-2:17.0.0-13241.g5292901b.el8.x86_64 2022-06-27T11:46:54.185 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cephfs-2:17.0.0-13241.g5292901b.el8.x86_64 2022-06-27T11:46:54.185 INFO:teuthology.orchestra.run.smithi195.stdout: python3-certifi-2018.10.15-7.el8.noarch 2022-06-27T11:46:54.185 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cffi-1.11.5-5.el8.x86_64 2022-06-27T11:46:54.185 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cheroot-8.5.2-1.el8.noarch 2022-06-27T11:46:54.186 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2022-06-27T11:46:54.186 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2022-06-27T11:46:54.186 INFO:teuthology.orchestra.run.smithi195.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2022-06-27T11:46:54.187 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-6.2-6.el8.noarch 2022-06-27T11:46:54.187 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2022-06-27T11:46:54.187 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jinja2-2.10.1-3.el8.noarch 2022-06-27T11:46:54.188 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jsonpatch-1.21-2.el8.noarch 2022-06-27T11:46:54.188 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jsonpointer-1.10-11.el8.noarch 2022-06-27T11:46:54.188 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jwt-1.6.1-2.el8.noarch 2022-06-27T11:46:54.188 INFO:teuthology.orchestra.run.smithi195.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2022-06-27T11:46:54.189 INFO:teuthology.orchestra.run.smithi195.stdout: python3-logutils-0.3.5-11.el8.noarch 2022-06-27T11:46:54.189 INFO:teuthology.orchestra.run.smithi195.stdout: python3-mako-1.0.6-13.el8.noarch 2022-06-27T11:46:54.189 INFO:teuthology.orchestra.run.smithi195.stdout: python3-markupsafe-0.23-19.el8.x86_64 2022-06-27T11:46:54.189 INFO:teuthology.orchestra.run.smithi195.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2022-06-27T11:46:54.190 INFO:teuthology.orchestra.run.smithi195.stdout: python3-natsort-7.1.1-2.el8.noarch 2022-06-27T11:46:54.190 INFO:teuthology.orchestra.run.smithi195.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2022-06-27T11:46:54.190 INFO:teuthology.orchestra.run.smithi195.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2022-06-27T11:46:54.190 INFO:teuthology.orchestra.run.smithi195.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2022-06-27T11:46:54.191 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pecan-1.3.2-9.el8.noarch 2022-06-27T11:46:54.191 INFO:teuthology.orchestra.run.smithi195.stdout: python3-portend-2.6-1.el8.noarch 2022-06-27T11:46:54.191 INFO:teuthology.orchestra.run.smithi195.stdout: python3-prettytable-0.7.2-14.el8.noarch 2022-06-27T11:46:54.191 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2022-06-27T11:46:54.192 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2022-06-27T11:46:54.192 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2022-06-27T11:46:54.192 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pycparser-2.14-14.el8.noarch 2022-06-27T11:46:54.192 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pytz-2017.2-9.el8.noarch 2022-06-27T11:46:54.192 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rados-2:17.0.0-13241.g5292901b.el8.x86_64 2022-06-27T11:46:54.193 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rbd-2:17.0.0-13241.g5292901b.el8.x86_64 2022-06-27T11:46:54.193 INFO:teuthology.orchestra.run.smithi195.stdout: python3-repoze-lru-0.7-6.el8.noarch 2022-06-27T11:46:54.193 INFO:teuthology.orchestra.run.smithi195.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2022-06-27T11:46:54.193 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rgw-2:17.0.0-13241.g5292901b.el8.x86_64 2022-06-27T11:46:54.194 INFO:teuthology.orchestra.run.smithi195.stdout: python3-routes-2.4.1-12.el8.noarch 2022-06-27T11:46:54.194 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rsa-4.8-1.el8.noarch 2022-06-27T11:46:54.194 INFO:teuthology.orchestra.run.smithi195.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2022-06-27T11:46:54.194 INFO:teuthology.orchestra.run.smithi195.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2022-06-27T11:46:54.195 INFO:teuthology.orchestra.run.smithi195.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2022-06-27T11:46:54.195 INFO:teuthology.orchestra.run.smithi195.stdout: python3-tempora-1.14.1-5.el8.noarch 2022-06-27T11:46:54.195 INFO:teuthology.orchestra.run.smithi195.stdout: python3-trustme-0.6.0-4.el8.noarch 2022-06-27T11:46:54.195 INFO:teuthology.orchestra.run.smithi195.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2022-06-27T11:46:54.195 INFO:teuthology.orchestra.run.smithi195.stdout: python3-webob-1.8.5-1.el8.1.noarch 2022-06-27T11:46:54.196 INFO:teuthology.orchestra.run.smithi195.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2022-06-27T11:46:54.196 INFO:teuthology.orchestra.run.smithi195.stdout: python3-webtest-2.0.33-1.el8.noarch 2022-06-27T11:46:54.196 INFO:teuthology.orchestra.run.smithi195.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2022-06-27T11:46:54.196 INFO:teuthology.orchestra.run.smithi195.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2022-06-27T11:46:54.197 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-fuse-2:17.0.0-13241.g5292901b.el8.x86_64 2022-06-27T11:46:54.197 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-mirror-2:17.0.0-13241.g5292901b.el8.x86_64 2022-06-27T11:46:54.197 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-nbd-2:17.0.0-13241.g5292901b.el8.x86_64 2022-06-27T11:46:54.197 INFO:teuthology.orchestra.run.smithi195.stdout: socat-1.7.4.1-1.el8.x86_64 2022-06-27T11:46:54.197 INFO:teuthology.orchestra.run.smithi195.stdout: thrift-0.13.0-2.el8.x86_64 2022-06-27T11:46:54.198 INFO:teuthology.orchestra.run.smithi195.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2022-06-27T11:46:54.198 INFO:teuthology.orchestra.run.smithi195.stdout: xmlstarlet-1.6.1-11.el8.x86_64 2022-06-27T11:46:54.198 INFO:teuthology.orchestra.run.smithi195.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2022-06-27T11:46:54.198 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:46:54.199 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2022-06-27T11:46:54.465 DEBUG:teuthology.parallel:result is None 2022-06-27T11:46:54.466 DEBUG:teuthology.orchestra.run.smithi195:> sudo lsb_release -is 2022-06-27T11:46:54.520 INFO:teuthology.orchestra.run.smithi195.stdout:RedHatEnterprise 2022-06-27T11:46:54.522 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-06-27T11:46:54.531 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=5292901bc999daa0d7ac128764a3f27c4233c252 2022-06-27T11:46:54.712 DEBUG:teuthology.orchestra.run.smithi195:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2022-06-27T11:46:54.736 INFO:teuthology.orchestra.run.smithi195.stdout:17.0.0-13241.g5292901b.el8 2022-06-27T11:46:54.739 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-13241.g5292901b.el8 2022-06-27T11:46:54.740 INFO:teuthology.task.install:The correct ceph version 17.0.0-13241.g5292901b is installed. 2022-06-27T11:46:54.741 DEBUG:teuthology.orchestra.run.smithi202:> sudo lsb_release -is 2022-06-27T11:46:54.799 INFO:teuthology.orchestra.run.smithi202.stdout:RedHatEnterprise 2022-06-27T11:46:54.801 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-06-27T11:46:54.801 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=5292901bc999daa0d7ac128764a3f27c4233c252 2022-06-27T11:46:54.966 DEBUG:teuthology.orchestra.run.smithi202:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2022-06-27T11:46:54.989 INFO:teuthology.orchestra.run.smithi202.stdout:17.0.0-13241.g5292901b.el8 2022-06-27T11:46:54.990 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-13241.g5292901b.el8 2022-06-27T11:46:54.990 INFO:teuthology.task.install:The correct ceph version 17.0.0-13241.g5292901b is installed. 2022-06-27T11:46:54.991 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2022-06-27T11:46:54.992 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2022-06-27T11:46:54.992 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-06-27T11:46:55.021 DEBUG:teuthology.orchestra.run.smithi202:> set -ex 2022-06-27T11:46:55.022 DEBUG:teuthology.orchestra.run.smithi202:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-06-27T11:46:55.058 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2022-06-27T11:46:55.059 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2022-06-27T11:46:55.060 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/usr/bin/daemon-helper 2022-06-27T11:46:55.088 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-06-27T11:46:55.155 DEBUG:teuthology.orchestra.run.smithi202:> set -ex 2022-06-27T11:46:55.155 DEBUG:teuthology.orchestra.run.smithi202:> sudo dd of=/usr/bin/daemon-helper 2022-06-27T11:46:55.184 DEBUG:teuthology.orchestra.run.smithi202:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-06-27T11:46:55.251 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2022-06-27T11:46:55.251 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2022-06-27T11:46:55.252 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/usr/bin/adjust-ulimits 2022-06-27T11:46:55.278 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-06-27T11:46:55.345 DEBUG:teuthology.orchestra.run.smithi202:> set -ex 2022-06-27T11:46:55.346 DEBUG:teuthology.orchestra.run.smithi202:> sudo dd of=/usr/bin/adjust-ulimits 2022-06-27T11:46:55.373 DEBUG:teuthology.orchestra.run.smithi202:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-06-27T11:46:55.440 INFO:teuthology.run_tasks:Running task ceph... 2022-06-27T11:46:55.519 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2022-06-27T11:46:55.520 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod 777 /var/log/ceph 2022-06-27T11:46:55.523 DEBUG:teuthology.orchestra.run.smithi202:> sudo chmod 777 /var/log/ceph 2022-06-27T11:46:55.549 INFO:tasks.ceph:Disabling ceph logrotate... 2022-06-27T11:46:55.550 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -f -- /etc/logrotate.d/ceph 2022-06-27T11:46:55.589 DEBUG:teuthology.orchestra.run.smithi202:> sudo rm -f -- /etc/logrotate.d/ceph 2022-06-27T11:46:55.616 INFO:tasks.ceph:Creating extra log directories... 2022-06-27T11:46:55.617 DEBUG:teuthology.orchestra.run.smithi195:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2022-06-27T11:46:55.656 DEBUG:teuthology.orchestra.run.smithi202:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2022-06-27T11:46:55.690 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2022-06-27T11:46:55.690 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2022-06-27T11:46:55.691 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2022-06-27T11:46:55.691 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2022-06-27T11:46:55.691 DEBUG:teuthology.orchestra.run.smithi195:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2022-06-27T11:46:55.736 DEBUG:teuthology.orchestra.run.smithi195:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2022-06-27T11:46:55.848 DEBUG:teuthology.orchestra.run.smithi195:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2022-06-27T11:46:55.876 DEBUG:teuthology.orchestra.run.smithi202:> set -ex 2022-06-27T11:46:55.877 DEBUG:teuthology.orchestra.run.smithi202:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2022-06-27T11:46:55.895 DEBUG:teuthology.orchestra.run.smithi202:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2022-06-27T11:46:56.010 DEBUG:teuthology.orchestra.run.smithi202:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2022-06-27T11:46:56.048 INFO:tasks.ceph:Creating ceph cluster ceph... 2022-06-27T11:46:56.048 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': False, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'Replacing daemon mds'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2022-06-27T11:46:56.049 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/vshankar-2022-06-27_06:36:09-fs-wip-vshankar-testing-20220627-100931-testing-default-smithi/6900509', 'branch': 'wip-vshankar-testing-20220627-100931', 'description': 'fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}}', 'email': 'vshankar@redhat.com', 'first_in_suite': False, 'job_id': '6900509', 'kernel': {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': True, 'sha1': '2d518d4a2a7d878b26d7f76b5a3474652fc358a7'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'vshankar-2022-06-27_06:36:09-fs-wip-vshankar-testing-20220627-100931-testing-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'rhel', 'os_version': '8.6', 'overrides': {'admin_socket': {'branch': 'wip-vshankar-testing-20220627-100931'}, 'ceph': {'cephfs': {'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': False, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'Replacing daemon mds'], 'sha1': '5292901bc999daa0d7ac128764a3f27c4233c252'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '5292901bc999daa0d7ac128764a3f27c4233c252'}}, 'kclient': {'syntax': 'v1'}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-vshankar-testing-20220627-100931', 'sha1': '5292901bc999daa0d7ac128764a3f27c4233c252'}}, 'owner': 'scheduled_vshankar@teuthology', 'priority': 200, 'repo': 'git://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']], 'sha1': '5292901bc999daa0d7ac128764a3f27c4233c252', 'sleep_before_teardown': 0, 'suite': 'fs', 'suite_branch': 'wip-vshankar-testing-20220627-100931', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_5292901bc999daa0d7ac128764a3f27c4233c252/qa', 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph-ci.git', 'suite_sha1': '5292901bc999daa0d7ac128764a3f27c4233c252', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi195.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDUd5V1StgPqlYh4mQ5I+nK3m+VDwD4/KsWOI5vsXndns8OJ/spYxCbIvlHNGQ+8VPjaOStwHqaVku07YGtorTRaEAhPTpngyMUxndQXOMv4IdSsiAkRj5vHIoTSXqZdaZvG8PBXCONrFecxRKNMh4OMlyRc19bMwXk3ZaG4okDM7H9x6AEkD/gJ+wsjIH7wEDBuvBmmnr2AcxE1ifHHzVq4p4UrkCG6CPULh6qeZdBaG0tmDh1Ct2CO/rCvw81fAxgqdabqfwdw332k2hUrw5sZiOX0mj03ogKP6Sfv3FkuS3m49LwIWaOQCzve5kkgmnU5AuP5Vu0oIKGr0Ug/lC62yv57gNw/yfZXd76RbepIzidZ4pqbXR+ISDZq63hIy8bO/0M7DJWzyCcjc6fUWk3cZJZGRTfUtTwbzzhMeKHuluT4NgREJNw/DKoijPjl0LeM7EjAMdAh0gd5ms/ba9bMIQiWNuWho16WdRi3ZoLL/rkdArEcQp66VyvG/Oze3k=', 'smithi202.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC7GVzKcLLYV3HsMIoNStF8xm4lQJPjMDH+9hR2/ORU8frhqQxcLkXOH72LBSKMISvSaN5/pzutYoF8bVGwUJkbfVeZO/I4Q3SNmA3sgDnmcedLduWGiaqSbIVRMU9ykIs2cQgX2caskfFkgMmjXjb1GXz50ktRDgMEfqNVsp5Xz9g+zkyJLfMCgq3LkhIaY3qYJ1v91KWayXTpLXj+eQ0yTQd6BTBADsrj0l1YYzeNUi8BctL5Sw7TMKGUDFAyulfIGeD3FfPBNtp6HZmDt+ZqDBofqdl3JjC3LELxRbFnVRvFEzbX1udzOcfxIpCs6h77V3jDl0heARsPU3CJsL0yb6EQeqjYiE//NGEMRtTxW4m3BfGU3WzYdhBsBq+Q0xa9rX1CSMYe3BN3r+X6uZAiMJgiAgLPd6ZvlHj8nuQ0Cl+jphhsaTV9r3Sep3p4KJamLCA3hbQGQHeGlV9GxJsK8/hiD4UbbuoUaHJjEZvBgEE+j1sdq6BXGlmtu57OX0k='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': True, 'sha1': '2d518d4a2a7d878b26d7f76b5a3474652fc358a7'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'flavor': 'default', 'sha1': '5292901bc999daa0d7ac128764a3f27c4233c252'}}, {'ceph': None}, {'kclient': None}, {'mds_thrash': None}, {'workunit': {'clients': {'all': ['fs/snaps']}, 'timeout': '6h'}}], 'teuthology_branch': 'main', 'teuthology_sha1': '082ae7ef4302fa54665ed0a2535e8e254118dcfd', 'timestamp': '2022-06-27_06:36:09', 'tube': 'smithi', 'user': 'vshankar', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.1713'} 2022-06-27T11:46:56.050 DEBUG:teuthology.orchestra.run.smithi195:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2022-06-27T11:46:56.053 DEBUG:teuthology.orchestra.run.smithi202:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2022-06-27T11:46:56.096 DEBUG:teuthology.orchestra.run.smithi195:> sudo install -d -m0777 -- /var/run/ceph 2022-06-27T11:46:56.109 DEBUG:teuthology.orchestra.run.smithi202:> sudo install -d -m0777 -- /var/run/ceph 2022-06-27T11:46:56.163 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2022-06-27T11:46:56.164 DEBUG:teuthology.orchestra.run.smithi195:> dd if=/scratch_devs of=/dev/stdout 2022-06-27T11:46:56.191 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-06-27T11:46:56.191 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-06-27T11:46:56.249 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-06-27T11:46:56.249 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-06-27T11:46:56.250 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 6h/6d Inode: 345273 Links: 1 2022-06-27T11:46:56.250 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-06-27T11:46:56.250 INFO:teuthology.orchestra.run.smithi195.stdout:Context: system_u:object_r:device_t:s0 2022-06-27T11:46:56.251 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2022-06-27 11:46:49.328648470 +0000 2022-06-27T11:46:56.251 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2022-06-27 11:44:06.384487932 +0000 2022-06-27T11:46:56.251 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2022-06-27 11:44:06.384487932 +0000 2022-06-27T11:46:56.251 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: - 2022-06-27T11:46:56.257 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2022-06-27T11:46:56.258 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2022-06-27T11:46:56.258 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 0.000136779 s, 3.7 MB/s 2022-06-27T11:46:56.260 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-06-27T11:46:56.318 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-06-27T11:46:56.318 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-06-27T11:46:56.319 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 6h/6d Inode: 344633 Links: 1 2022-06-27T11:46:56.319 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-06-27T11:46:56.319 INFO:teuthology.orchestra.run.smithi195.stdout:Context: system_u:object_r:device_t:s0 2022-06-27T11:46:56.319 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2022-06-27 11:46:49.328648470 +0000 2022-06-27T11:46:56.320 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2022-06-27 11:44:06.947477465 +0000 2022-06-27T11:46:56.320 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2022-06-27 11:44:06.947477465 +0000 2022-06-27T11:46:56.320 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: - 2022-06-27T11:46:56.328 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2022-06-27T11:46:56.328 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2022-06-27T11:46:56.329 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 0.000129219 s, 4.0 MB/s 2022-06-27T11:46:56.331 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-06-27T11:46:56.388 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-06-27T11:46:56.389 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-06-27T11:46:56.389 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 6h/6d Inode: 345422 Links: 1 2022-06-27T11:46:56.389 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-06-27T11:46:56.390 INFO:teuthology.orchestra.run.smithi195.stdout:Context: system_u:object_r:device_t:s0 2022-06-27T11:46:56.390 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2022-06-27 11:46:49.328648470 +0000 2022-06-27T11:46:56.391 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2022-06-27 11:44:07.534466551 +0000 2022-06-27T11:46:56.391 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2022-06-27 11:44:07.534466551 +0000 2022-06-27T11:46:56.391 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: - 2022-06-27T11:46:56.397 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2022-06-27T11:46:56.398 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2022-06-27T11:46:56.398 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 0.000116673 s, 4.4 MB/s 2022-06-27T11:46:56.401 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-06-27T11:46:56.457 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-06-27T11:46:56.458 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-06-27T11:46:56.458 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 6h/6d Inode: 342197 Links: 1 2022-06-27T11:46:56.459 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-06-27T11:46:56.459 INFO:teuthology.orchestra.run.smithi195.stdout:Context: system_u:object_r:device_t:s0 2022-06-27T11:46:56.459 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2022-06-27 11:46:49.328648470 +0000 2022-06-27T11:46:56.460 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2022-06-27 11:44:08.067456641 +0000 2022-06-27T11:46:56.460 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2022-06-27 11:44:08.067456641 +0000 2022-06-27T11:46:56.461 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: - 2022-06-27T11:46:56.471 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2022-06-27T11:46:56.471 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2022-06-27T11:46:56.472 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 0.000131471 s, 3.9 MB/s 2022-06-27T11:46:56.472 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2022-06-27T11:46:56.473 DEBUG:teuthology.orchestra.run.smithi202:> set -ex 2022-06-27T11:46:56.473 DEBUG:teuthology.orchestra.run.smithi202:> dd if=/scratch_devs of=/dev/stdout 2022-06-27T11:46:56.492 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-06-27T11:46:56.493 DEBUG:teuthology.orchestra.run.smithi202:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-06-27T11:46:56.556 INFO:teuthology.orchestra.run.smithi202.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-06-27T11:46:56.556 INFO:teuthology.orchestra.run.smithi202.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-06-27T11:46:56.557 INFO:teuthology.orchestra.run.smithi202.stdout:Device: 6h/6d Inode: 345119 Links: 1 2022-06-27T11:46:56.557 INFO:teuthology.orchestra.run.smithi202.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-06-27T11:46:56.558 INFO:teuthology.orchestra.run.smithi202.stdout:Context: system_u:object_r:device_t:s0 2022-06-27T11:46:56.558 INFO:teuthology.orchestra.run.smithi202.stdout:Access: 2022-06-27 11:46:48.774118313 +0000 2022-06-27T11:46:56.558 INFO:teuthology.orchestra.run.smithi202.stdout:Modify: 2022-06-27 11:44:16.781490336 +0000 2022-06-27T11:46:56.559 INFO:teuthology.orchestra.run.smithi202.stdout:Change: 2022-06-27 11:44:16.781490336 +0000 2022-06-27T11:46:56.559 INFO:teuthology.orchestra.run.smithi202.stdout: Birth: - 2022-06-27T11:46:56.560 INFO:teuthology.orchestra.run.smithi202.stderr:1+0 records in 2022-06-27T11:46:56.561 INFO:teuthology.orchestra.run.smithi202.stderr:1+0 records out 2022-06-27T11:46:56.561 INFO:teuthology.orchestra.run.smithi202.stderr:512 bytes copied, 0.000216278 s, 2.4 MB/s 2022-06-27T11:46:56.563 DEBUG:teuthology.orchestra.run.smithi202:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-06-27T11:46:56.621 INFO:teuthology.orchestra.run.smithi202.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-06-27T11:46:56.621 INFO:teuthology.orchestra.run.smithi202.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-06-27T11:46:56.621 INFO:teuthology.orchestra.run.smithi202.stdout:Device: 6h/6d Inode: 345281 Links: 1 2022-06-27T11:46:56.621 INFO:teuthology.orchestra.run.smithi202.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-06-27T11:46:56.622 INFO:teuthology.orchestra.run.smithi202.stdout:Context: system_u:object_r:device_t:s0 2022-06-27T11:46:56.622 INFO:teuthology.orchestra.run.smithi202.stdout:Access: 2022-06-27 11:46:48.774118313 +0000 2022-06-27T11:46:56.622 INFO:teuthology.orchestra.run.smithi202.stdout:Modify: 2022-06-27 11:44:17.363480320 +0000 2022-06-27T11:46:56.622 INFO:teuthology.orchestra.run.smithi202.stdout:Change: 2022-06-27 11:44:17.363480320 +0000 2022-06-27T11:46:56.623 INFO:teuthology.orchestra.run.smithi202.stdout: Birth: - 2022-06-27T11:46:56.630 INFO:teuthology.orchestra.run.smithi202.stderr:1+0 records in 2022-06-27T11:46:56.630 INFO:teuthology.orchestra.run.smithi202.stderr:1+0 records out 2022-06-27T11:46:56.631 INFO:teuthology.orchestra.run.smithi202.stderr:512 bytes copied, 0.00022148 s, 2.3 MB/s 2022-06-27T11:46:56.633 DEBUG:teuthology.orchestra.run.smithi202:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-06-27T11:46:56.700 INFO:teuthology.orchestra.run.smithi202.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-06-27T11:46:56.701 INFO:teuthology.orchestra.run.smithi202.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-06-27T11:46:56.702 INFO:teuthology.orchestra.run.smithi202.stdout:Device: 6h/6d Inode: 343502 Links: 1 2022-06-27T11:46:56.702 INFO:teuthology.orchestra.run.smithi202.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-06-27T11:46:56.702 INFO:teuthology.orchestra.run.smithi202.stdout:Context: system_u:object_r:device_t:s0 2022-06-27T11:46:56.703 INFO:teuthology.orchestra.run.smithi202.stdout:Access: 2022-06-27 11:46:48.774118313 +0000 2022-06-27T11:46:56.703 INFO:teuthology.orchestra.run.smithi202.stdout:Modify: 2022-06-27 11:44:17.989469548 +0000 2022-06-27T11:46:56.704 INFO:teuthology.orchestra.run.smithi202.stdout:Change: 2022-06-27 11:44:17.989469548 +0000 2022-06-27T11:46:56.704 INFO:teuthology.orchestra.run.smithi202.stdout: Birth: - 2022-06-27T11:46:56.711 INFO:teuthology.orchestra.run.smithi202.stderr:1+0 records in 2022-06-27T11:46:56.711 INFO:teuthology.orchestra.run.smithi202.stderr:1+0 records out 2022-06-27T11:46:56.712 INFO:teuthology.orchestra.run.smithi202.stderr:512 bytes copied, 0.000203826 s, 2.5 MB/s 2022-06-27T11:46:56.714 DEBUG:teuthology.orchestra.run.smithi202:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-06-27T11:46:56.771 INFO:teuthology.orchestra.run.smithi202.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-06-27T11:46:56.772 INFO:teuthology.orchestra.run.smithi202.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-06-27T11:46:56.772 INFO:teuthology.orchestra.run.smithi202.stdout:Device: 6h/6d Inode: 340917 Links: 1 2022-06-27T11:46:56.772 INFO:teuthology.orchestra.run.smithi202.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-06-27T11:46:56.773 INFO:teuthology.orchestra.run.smithi202.stdout:Context: system_u:object_r:device_t:s0 2022-06-27T11:46:56.773 INFO:teuthology.orchestra.run.smithi202.stdout:Access: 2022-06-27 11:46:48.774118313 +0000 2022-06-27T11:46:56.773 INFO:teuthology.orchestra.run.smithi202.stdout:Modify: 2022-06-27 11:44:18.599459050 +0000 2022-06-27T11:46:56.773 INFO:teuthology.orchestra.run.smithi202.stdout:Change: 2022-06-27 11:44:18.599459050 +0000 2022-06-27T11:46:56.774 INFO:teuthology.orchestra.run.smithi202.stdout: Birth: - 2022-06-27T11:46:56.781 INFO:teuthology.orchestra.run.smithi202.stderr:1+0 records in 2022-06-27T11:46:56.781 INFO:teuthology.orchestra.run.smithi202.stderr:1+0 records out 2022-06-27T11:46:56.782 INFO:teuthology.orchestra.run.smithi202.stderr:512 bytes copied, 0.000219618 s, 2.3 MB/s 2022-06-27T11:46:56.784 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2022-06-27T11:46:56.784 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi195.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi202.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2022-06-27T11:46:56.785 INFO:tasks.ceph:Generating config... 2022-06-27T11:46:56.787 INFO:tasks.ceph:[client] client mount timeout = 600 2022-06-27T11:46:56.787 INFO:tasks.ceph:[client] debug client = 20 2022-06-27T11:46:56.787 INFO:tasks.ceph:[client] debug ms = 1 2022-06-27T11:46:56.788 INFO:tasks.ceph:[client] rados mon op timeout = 900 2022-06-27T11:46:56.788 INFO:tasks.ceph:[client] rados osd op timeout = 900 2022-06-27T11:46:56.788 INFO:tasks.ceph:[global] ms die on skipped message = False 2022-06-27T11:46:56.789 INFO:tasks.ceph:[mds] debug mds = 20 2022-06-27T11:46:56.789 INFO:tasks.ceph:[mds] debug mds balancer = 20 2022-06-27T11:46:56.789 INFO:tasks.ceph:[mds] debug ms = 1 2022-06-27T11:46:56.790 INFO:tasks.ceph:[mds] mds bal fragment size max = 10000 2022-06-27T11:46:56.791 INFO:tasks.ceph:[mds] mds bal merge size = 5 2022-06-27T11:46:56.792 INFO:tasks.ceph:[mds] mds bal split bits = 3 2022-06-27T11:46:56.792 INFO:tasks.ceph:[mds] mds bal split size = 100 2022-06-27T11:46:56.792 INFO:tasks.ceph:[mds] mds debug frag = True 2022-06-27T11:46:56.793 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2022-06-27T11:46:56.793 INFO:tasks.ceph:[mds] mds oft prefetch dirfrags = True 2022-06-27T11:46:56.793 INFO:tasks.ceph:[mds] mds op complaint time = 180 2022-06-27T11:46:56.794 INFO:tasks.ceph:[mds] mds verify scatter = True 2022-06-27T11:46:56.794 INFO:tasks.ceph:[mds] mds_dir_prefetch = False 2022-06-27T11:46:56.794 INFO:tasks.ceph:[mds] mds_sleep_rank_change = 5000000.0 2022-06-27T11:46:56.795 INFO:tasks.ceph:[mds] mon client directed command retry = 5 2022-06-27T11:46:56.795 INFO:tasks.ceph:[mds] ms inject delay max = 1 2022-06-27T11:46:56.796 INFO:tasks.ceph:[mds] ms inject delay probability = 0.005 2022-06-27T11:46:56.796 INFO:tasks.ceph:[mds] ms inject delay type = client mds osd 2022-06-27T11:46:56.797 INFO:tasks.ceph:[mds] ms inject socket failures = 2500 2022-06-27T11:46:56.797 INFO:tasks.ceph:[mds] osd op complaint time = 180 2022-06-27T11:46:56.798 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2022-06-27T11:46:56.798 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2022-06-27T11:46:56.799 INFO:tasks.ceph:[mgr] debug mgr = 20 2022-06-27T11:46:56.800 INFO:tasks.ceph:[mgr] debug ms = 1 2022-06-27T11:46:56.800 INFO:tasks.ceph:[mon] debug mon = 20 2022-06-27T11:46:56.801 INFO:tasks.ceph:[mon] debug ms = 1 2022-06-27T11:46:56.801 INFO:tasks.ceph:[mon] debug paxos = 20 2022-06-27T11:46:56.802 INFO:tasks.ceph:[mon] mon op complaint time = 120 2022-06-27T11:46:56.802 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2022-06-27T11:46:56.803 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2022-06-27T11:46:56.803 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2022-06-27T11:46:56.803 INFO:tasks.ceph:[osd] debug bluefs = 20 2022-06-27T11:46:56.804 INFO:tasks.ceph:[osd] debug bluestore = 20 2022-06-27T11:46:56.804 INFO:tasks.ceph:[osd] debug ms = 1 2022-06-27T11:46:56.804 INFO:tasks.ceph:[osd] debug osd = 20 2022-06-27T11:46:56.805 INFO:tasks.ceph:[osd] debug rocksdb = 10 2022-06-27T11:46:56.805 INFO:tasks.ceph:[osd] mon client directed command retry = 5 2022-06-27T11:46:56.805 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2022-06-27T11:46:56.806 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2022-06-27T11:46:56.806 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2022-06-27T11:46:56.806 INFO:tasks.ceph:[osd] ms inject delay max = 1 2022-06-27T11:46:56.807 INFO:tasks.ceph:[osd] ms inject delay probability = 0.005 2022-06-27T11:46:56.807 INFO:tasks.ceph:[osd] ms inject delay type = client mds 2022-06-27T11:46:56.807 INFO:tasks.ceph:[osd] ms inject socket failures = 2500 2022-06-27T11:46:56.808 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2022-06-27T11:46:56.808 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2022-06-27T11:46:56.808 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2022-06-27T11:46:56.809 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2022-06-27T11:46:56.809 INFO:tasks.ceph:[osd] osd op complaint time = 180 2022-06-27T11:46:56.810 INFO:tasks.ceph:Setting up mon.a... 2022-06-27T11:46:56.810 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2022-06-27T11:46:56.865 INFO:teuthology.orchestra.run.smithi195.stdout:creating /etc/ceph/ceph.keyring 2022-06-27T11:46:56.868 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2022-06-27T11:46:56.915 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod 0644 /etc/ceph/ceph.keyring 2022-06-27T11:46:56.942 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.195'), ('mon.b', '172.21.15.202'), ('mon.c', '[v2:172.21.15.202:3301,v1:172.21.15.202:6790]')] 2022-06-27T11:46:56.943 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.195,172.21.15.202,[v2:172.21.15.202:3301,v1:172.21.15.202:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.e': {}, 'mds.b': {}, 'mds.d': {}, 'mds.f': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': False, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2022-06-27T11:46:56.945 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2022-06-27T11:46:56.946 DEBUG:teuthology.orchestra.run.smithi195:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2022-06-27T11:46:56.999 DEBUG:teuthology.orchestra.run.smithi195:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.195 --add b 172.21.15.202 --addv c '[v2:172.21.15.202:3301,v1:172.21.15.202:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2022-06-27T11:46:57.086 INFO:teuthology.orchestra.run.smithi195.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2022-06-27T11:46:57.087 INFO:teuthology.orchestra.run.smithi195.stdout:monmaptool: generated fsid 81b3a1d3-6a2b-47e8-8d69-7027cfeae7af 2022-06-27T11:46:57.087 INFO:teuthology.orchestra.run.smithi195.stdout:setting min_mon_release = octopus 2022-06-27T11:46:57.087 INFO:teuthology.orchestra.run.smithi195.stdout:epoch 0 2022-06-27T11:46:57.087 INFO:teuthology.orchestra.run.smithi195.stdout:fsid 81b3a1d3-6a2b-47e8-8d69-7027cfeae7af 2022-06-27T11:46:57.088 INFO:teuthology.orchestra.run.smithi195.stdout:last_changed 2022-06-27T11:46:57.087847+0000 2022-06-27T11:46:57.088 INFO:teuthology.orchestra.run.smithi195.stdout:created 2022-06-27T11:46:57.087847+0000 2022-06-27T11:46:57.088 INFO:teuthology.orchestra.run.smithi195.stdout:min_mon_release 15 (octopus) 2022-06-27T11:46:57.088 INFO:teuthology.orchestra.run.smithi195.stdout:election_strategy: 1 2022-06-27T11:46:57.088 INFO:teuthology.orchestra.run.smithi195.stdout:0: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.a 2022-06-27T11:46:57.089 INFO:teuthology.orchestra.run.smithi195.stdout:1: [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] mon.b 2022-06-27T11:46:57.089 INFO:teuthology.orchestra.run.smithi195.stdout:2: [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] mon.c 2022-06-27T11:46:57.089 INFO:teuthology.orchestra.run.smithi195.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2022-06-27T11:46:57.091 DEBUG:teuthology.orchestra.run.smithi195:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2022-06-27T11:46:57.108 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 81b3a1d3-6a2b-47e8-8d69-7027cfeae7af... 2022-06-27T11:46:57.111 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-06-27T11:46:57.156 DEBUG:teuthology.orchestra.run.smithi202:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-06-27T11:46:57.198 INFO:teuthology.orchestra.run.smithi195.stdout:[global] 2022-06-27T11:46:57.198 INFO:teuthology.orchestra.run.smithi195.stdout: chdir = "" 2022-06-27T11:46:57.198 INFO:teuthology.orchestra.run.smithi195.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-06-27T11:46:57.199 INFO:teuthology.orchestra.run.smithi195.stdout: auth supported = cephx 2022-06-27T11:46:57.199 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:46:57.199 INFO:teuthology.orchestra.run.smithi195.stdout: filestore xattr use omap = true 2022-06-27T11:46:57.199 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:46:57.200 INFO:teuthology.orchestra.run.smithi195.stdout: mon clock drift allowed = 1.000 2022-06-27T11:46:57.200 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:46:57.200 INFO:teuthology.orchestra.run.smithi195.stdout: osd crush chooseleaf type = 0 2022-06-27T11:46:57.200 INFO:teuthology.orchestra.run.smithi195.stdout: auth debug = true 2022-06-27T11:46:57.201 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:46:57.201 INFO:teuthology.orchestra.run.smithi195.stdout: ms die on old message = true 2022-06-27T11:46:57.201 INFO:teuthology.orchestra.run.smithi195.stdout: ms die on bug = true 2022-06-27T11:46:57.201 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:46:57.201 INFO:teuthology.orchestra.run.smithi195.stdout: mon max pg per osd = 10000 # >= luminous 2022-06-27T11:46:57.202 INFO:teuthology.orchestra.run.smithi195.stdout: mon pg warn max object skew = 0 2022-06-27T11:46:57.202 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:46:57.202 INFO:teuthology.orchestra.run.smithi195.stdout: # disable pg_autoscaler by default for new pools 2022-06-27T11:46:57.202 INFO:teuthology.orchestra.run.smithi195.stdout: osd_pool_default_pg_autoscale_mode = off 2022-06-27T11:46:57.203 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:46:57.203 INFO:teuthology.orchestra.run.smithi195.stdout: osd pool default size = 2 2022-06-27T11:46:57.203 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:46:57.203 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd allow primary affinity = true 2022-06-27T11:46:57.204 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd allow pg remap = true 2022-06-27T11:46:57.204 INFO:teuthology.orchestra.run.smithi195.stdout: mon warn on legacy crush tunables = false 2022-06-27T11:46:57.204 INFO:teuthology.orchestra.run.smithi195.stdout: mon warn on crush straw calc version zero = false 2022-06-27T11:46:57.204 INFO:teuthology.orchestra.run.smithi195.stdout: mon warn on no sortbitwise = false 2022-06-27T11:46:57.205 INFO:teuthology.orchestra.run.smithi195.stdout: mon warn on osd down out interval zero = false 2022-06-27T11:46:57.205 INFO:teuthology.orchestra.run.smithi195.stdout: mon warn on too few osds = false 2022-06-27T11:46:57.205 INFO:teuthology.orchestra.run.smithi195.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-06-27T11:46:57.205 INFO:teuthology.orchestra.run.smithi195.stdout: mon_warn_on_pool_no_redundancy = false 2022-06-27T11:46:57.205 INFO:teuthology.orchestra.run.smithi195.stdout: mon_allow_pool_size_one = true 2022-06-27T11:46:57.206 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:46:57.206 INFO:teuthology.orchestra.run.smithi195.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2022-06-27T11:46:57.206 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:46:57.206 INFO:teuthology.orchestra.run.smithi195.stdout: osd default data pool replay window = 5 2022-06-27T11:46:57.207 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:46:57.207 INFO:teuthology.orchestra.run.smithi195.stdout: mon allow pool delete = true 2022-06-27T11:46:57.207 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:46:57.207 INFO:teuthology.orchestra.run.smithi195.stdout: mon cluster log file level = debug 2022-06-27T11:46:57.207 INFO:teuthology.orchestra.run.smithi195.stdout: debug asserts on shutdown = true 2022-06-27T11:46:57.208 INFO:teuthology.orchestra.run.smithi195.stdout: mon health detail to clog = false 2022-06-27T11:46:57.208 INFO:teuthology.orchestra.run.smithi195.stdout: mon host = "172.21.15.195,172.21.15.202,[v2:172.21.15.202:3301,v1:172.21.15.202:6790]" 2022-06-27T11:46:57.208 INFO:teuthology.orchestra.run.smithi195.stdout: ms die on skipped message = False 2022-06-27T11:46:57.208 INFO:teuthology.orchestra.run.smithi195.stdout: fsid = 81b3a1d3-6a2b-47e8-8d69-7027cfeae7af 2022-06-27T11:46:57.209 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:46:57.209 INFO:teuthology.orchestra.run.smithi195.stdout:[osd] 2022-06-27T11:46:57.209 INFO:teuthology.orchestra.run.smithi195.stdout: osd journal size = 100 2022-06-27T11:46:57.209 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:46:57.209 INFO:teuthology.orchestra.run.smithi195.stdout: osd scrub load threshold = 5.0 2022-06-27T11:46:57.210 INFO:teuthology.orchestra.run.smithi195.stdout: osd scrub max interval = 600 2022-06-27T11:46:57.210 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:46:57.210 INFO:teuthology.orchestra.run.smithi195.stdout: osd recover clone overlap = true 2022-06-27T11:46:57.210 INFO:teuthology.orchestra.run.smithi195.stdout: osd recovery max chunk = 1048576 2022-06-27T11:46:57.211 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:46:57.211 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug shutdown = true 2022-06-27T11:46:57.211 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug op order = true 2022-06-27T11:46:57.211 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug verify stray on activate = true 2022-06-27T11:46:57.211 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:46:57.212 INFO:teuthology.orchestra.run.smithi195.stdout: osd open classes on start = true 2022-06-27T11:46:57.212 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug pg log writeout = true 2022-06-27T11:46:57.212 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:46:57.212 INFO:teuthology.orchestra.run.smithi195.stdout: osd deep scrub update digest min age = 30 2022-06-27T11:46:57.213 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:46:57.213 INFO:teuthology.orchestra.run.smithi195.stdout: osd map max advance = 10 2022-06-27T11:46:57.213 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:46:57.213 INFO:teuthology.orchestra.run.smithi195.stdout: journal zero on create = true 2022-06-27T11:46:57.213 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:46:57.214 INFO:teuthology.orchestra.run.smithi195.stdout: filestore ondisk finisher threads = 3 2022-06-27T11:46:57.214 INFO:teuthology.orchestra.run.smithi195.stdout: filestore apply finisher threads = 3 2022-06-27T11:46:57.214 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:46:57.214 INFO:teuthology.orchestra.run.smithi195.stdout: bdev debug aio = true 2022-06-27T11:46:57.215 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug misdirected ops = true 2022-06-27T11:46:57.215 INFO:teuthology.orchestra.run.smithi195.stdout: bluestore block size = 96636764160 2022-06-27T11:46:57.215 INFO:teuthology.orchestra.run.smithi195.stdout: bluestore compression mode = aggressive 2022-06-27T11:46:57.215 INFO:teuthology.orchestra.run.smithi195.stdout: bluestore fsck on mount = True 2022-06-27T11:46:57.216 INFO:teuthology.orchestra.run.smithi195.stdout: debug bluefs = 20 2022-06-27T11:46:57.216 INFO:teuthology.orchestra.run.smithi195.stdout: debug bluestore = 20 2022-06-27T11:46:57.216 INFO:teuthology.orchestra.run.smithi195.stdout: debug ms = 1 2022-06-27T11:46:57.217 INFO:teuthology.orchestra.run.smithi195.stdout: debug osd = 20 2022-06-27T11:46:57.217 INFO:teuthology.orchestra.run.smithi195.stdout: debug rocksdb = 10 2022-06-27T11:46:57.217 INFO:teuthology.orchestra.run.smithi195.stdout: mon client directed command retry = 5 2022-06-27T11:46:57.217 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd backfillfull_ratio = 0.85 2022-06-27T11:46:57.217 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd full ratio = 0.9 2022-06-27T11:46:57.218 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd nearfull ratio = 0.8 2022-06-27T11:46:57.218 INFO:teuthology.orchestra.run.smithi195.stdout: ms inject delay max = 1 2022-06-27T11:46:57.218 INFO:teuthology.orchestra.run.smithi195.stdout: ms inject delay probability = 0.005 2022-06-27T11:46:57.218 INFO:teuthology.orchestra.run.smithi195.stdout: ms inject delay type = client mds 2022-06-27T11:46:57.219 INFO:teuthology.orchestra.run.smithi195.stdout: ms inject socket failures = 2500 2022-06-27T11:46:57.219 INFO:teuthology.orchestra.run.smithi195.stdout: osd blocked scrub grace period = 3600 2022-06-27T11:46:57.219 INFO:teuthology.orchestra.run.smithi195.stdout: osd failsafe full ratio = 0.95 2022-06-27T11:46:57.219 INFO:teuthology.orchestra.run.smithi195.stdout: osd max markdown count = 1000 2022-06-27T11:46:57.220 INFO:teuthology.orchestra.run.smithi195.stdout: osd objectstore = bluestore 2022-06-27T11:46:57.220 INFO:teuthology.orchestra.run.smithi195.stdout: osd op complaint time = 180 2022-06-27T11:46:57.220 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:46:57.220 INFO:teuthology.orchestra.run.smithi195.stdout:[mgr] 2022-06-27T11:46:57.220 INFO:teuthology.orchestra.run.smithi195.stdout: debug ms = 1 2022-06-27T11:46:57.221 INFO:teuthology.orchestra.run.smithi195.stdout: debug mgr = 20 2022-06-27T11:46:57.221 INFO:teuthology.orchestra.run.smithi195.stdout: debug mon = 20 2022-06-27T11:46:57.221 INFO:teuthology.orchestra.run.smithi195.stdout: debug auth = 20 2022-06-27T11:46:57.221 INFO:teuthology.orchestra.run.smithi195.stdout: mon reweight min pgs per osd = 4 2022-06-27T11:46:57.221 INFO:teuthology.orchestra.run.smithi195.stdout: mon reweight min bytes per osd = 10 2022-06-27T11:46:57.222 INFO:teuthology.orchestra.run.smithi195.stdout: mgr/telemetry/nag = false 2022-06-27T11:46:57.222 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:46:57.222 INFO:teuthology.orchestra.run.smithi195.stdout:[mon] 2022-06-27T11:46:57.222 INFO:teuthology.orchestra.run.smithi195.stdout: debug ms = 1 2022-06-27T11:46:57.223 INFO:teuthology.orchestra.run.smithi195.stdout: debug mon = 20 2022-06-27T11:46:57.223 INFO:teuthology.orchestra.run.smithi195.stdout: debug paxos = 20 2022-06-27T11:46:57.223 INFO:teuthology.orchestra.run.smithi195.stdout: debug auth = 20 2022-06-27T11:46:57.223 INFO:teuthology.orchestra.run.smithi195.stdout: mon data avail warn = 5 2022-06-27T11:46:57.223 INFO:teuthology.orchestra.run.smithi195.stdout: mon mgr mkfs grace = 240 2022-06-27T11:46:57.224 INFO:teuthology.orchestra.run.smithi195.stdout: mon reweight min pgs per osd = 4 2022-06-27T11:46:57.224 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd reporter subtree level = osd 2022-06-27T11:46:57.224 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd prime pg temp = true 2022-06-27T11:46:57.224 INFO:teuthology.orchestra.run.smithi195.stdout: mon reweight min bytes per osd = 10 2022-06-27T11:46:57.225 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:46:57.225 INFO:teuthology.orchestra.run.smithi195.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-06-27T11:46:57.225 INFO:teuthology.orchestra.run.smithi195.stdout: auth mon ticket ttl = 660 # 11m 2022-06-27T11:46:57.225 INFO:teuthology.orchestra.run.smithi195.stdout: auth service ticket ttl = 240 # 4m 2022-06-27T11:46:57.225 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:46:57.226 INFO:teuthology.orchestra.run.smithi195.stdout: # don't complain about insecure global_id in the test suite 2022-06-27T11:46:57.226 INFO:teuthology.orchestra.run.smithi195.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-06-27T11:46:57.226 INFO:teuthology.orchestra.run.smithi195.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-06-27T11:46:57.226 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:46:57.226 INFO:teuthology.orchestra.run.smithi195.stdout: # 1m isn't quite enough 2022-06-27T11:46:57.227 INFO:teuthology.orchestra.run.smithi195.stdout: mon_down_mkfs_grace = 2m 2022-06-27T11:46:57.227 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:46:57.227 INFO:teuthology.orchestra.run.smithi195.stdout: mon_warn_on_filestore_osds = false 2022-06-27T11:46:57.227 INFO:teuthology.orchestra.run.smithi195.stdout: mon op complaint time = 120 2022-06-27T11:46:57.228 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:46:57.228 INFO:teuthology.orchestra.run.smithi195.stdout:[client] 2022-06-27T11:46:57.228 INFO:teuthology.orchestra.run.smithi195.stdout: rgw cache enabled = true 2022-06-27T11:46:57.228 INFO:teuthology.orchestra.run.smithi195.stdout: rgw enable ops log = true 2022-06-27T11:46:57.229 INFO:teuthology.orchestra.run.smithi195.stdout: rgw enable usage log = true 2022-06-27T11:46:57.229 INFO:teuthology.orchestra.run.smithi195.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-06-27T11:46:57.229 INFO:teuthology.orchestra.run.smithi195.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-06-27T11:46:57.229 INFO:teuthology.orchestra.run.smithi195.stdout: client mount timeout = 600 2022-06-27T11:46:57.230 INFO:teuthology.orchestra.run.smithi195.stdout: debug client = 20 2022-06-27T11:46:57.230 INFO:teuthology.orchestra.run.smithi195.stdout: debug ms = 1 2022-06-27T11:46:57.230 INFO:teuthology.orchestra.run.smithi195.stdout: rados mon op timeout = 900 2022-06-27T11:46:57.230 INFO:teuthology.orchestra.run.smithi195.stdout: rados osd op timeout = 900 2022-06-27T11:46:57.230 INFO:teuthology.orchestra.run.smithi195.stdout:[mon.a] 2022-06-27T11:46:57.231 INFO:teuthology.orchestra.run.smithi195.stdout:[mon.b] 2022-06-27T11:46:57.231 INFO:teuthology.orchestra.run.smithi195.stdout:[mon.c] 2022-06-27T11:46:57.231 INFO:teuthology.orchestra.run.smithi195.stdout:[mds.a] 2022-06-27T11:46:57.231 INFO:teuthology.orchestra.run.smithi195.stdout:[mds.c] 2022-06-27T11:46:57.232 INFO:teuthology.orchestra.run.smithi195.stdout:[mds.e] 2022-06-27T11:46:57.232 INFO:teuthology.orchestra.run.smithi195.stdout:[mds.b] 2022-06-27T11:46:57.232 INFO:teuthology.orchestra.run.smithi195.stdout:[mds.d] 2022-06-27T11:46:57.232 INFO:teuthology.orchestra.run.smithi195.stdout:[mds.f] 2022-06-27T11:46:57.232 INFO:teuthology.orchestra.run.smithi195.stdout:[mds] 2022-06-27T11:46:57.233 INFO:teuthology.orchestra.run.smithi195.stdout: debug mds = 20 2022-06-27T11:46:57.233 INFO:teuthology.orchestra.run.smithi195.stdout: debug mds balancer = 20 2022-06-27T11:46:57.233 INFO:teuthology.orchestra.run.smithi195.stdout: debug ms = 1 2022-06-27T11:46:57.233 INFO:teuthology.orchestra.run.smithi195.stdout: mds bal fragment size max = 10000 2022-06-27T11:46:57.233 INFO:teuthology.orchestra.run.smithi195.stdout: mds bal merge size = 5 2022-06-27T11:46:57.234 INFO:teuthology.orchestra.run.smithi195.stdout: mds bal split bits = 3 2022-06-27T11:46:57.234 INFO:teuthology.orchestra.run.smithi195.stdout: mds bal split size = 100 2022-06-27T11:46:57.234 INFO:teuthology.orchestra.run.smithi195.stdout: mds debug frag = True 2022-06-27T11:46:57.234 INFO:teuthology.orchestra.run.smithi195.stdout: mds debug scatterstat = True 2022-06-27T11:46:57.235 INFO:teuthology.orchestra.run.smithi195.stdout: mds oft prefetch dirfrags = True 2022-06-27T11:46:57.235 INFO:teuthology.orchestra.run.smithi195.stdout: mds op complaint time = 180 2022-06-27T11:46:57.235 INFO:teuthology.orchestra.run.smithi195.stdout: mds verify scatter = True 2022-06-27T11:46:57.235 INFO:teuthology.orchestra.run.smithi195.stdout: mds_dir_prefetch = False 2022-06-27T11:46:57.236 INFO:teuthology.orchestra.run.smithi195.stdout: mds_sleep_rank_change = 5000000.0 2022-06-27T11:46:57.236 INFO:teuthology.orchestra.run.smithi195.stdout: mon client directed command retry = 5 2022-06-27T11:46:57.236 INFO:teuthology.orchestra.run.smithi195.stdout: ms inject delay max = 1 2022-06-27T11:46:57.236 INFO:teuthology.orchestra.run.smithi195.stdout: ms inject delay probability = 0.005 2022-06-27T11:46:57.236 INFO:teuthology.orchestra.run.smithi195.stdout: ms inject delay type = client mds osd 2022-06-27T11:46:57.237 INFO:teuthology.orchestra.run.smithi195.stdout: ms inject socket failures = 2500 2022-06-27T11:46:57.237 INFO:teuthology.orchestra.run.smithi195.stdout: osd op complaint time = 180 2022-06-27T11:46:57.237 INFO:teuthology.orchestra.run.smithi195.stdout: rados mon op timeout = 900 2022-06-27T11:46:57.237 INFO:teuthology.orchestra.run.smithi195.stdout: rados osd op timeout = 900 2022-06-27T11:46:57.239 INFO:teuthology.orchestra.run.smithi202.stdout:[global] 2022-06-27T11:46:57.239 INFO:teuthology.orchestra.run.smithi202.stdout: chdir = "" 2022-06-27T11:46:57.240 INFO:teuthology.orchestra.run.smithi202.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-06-27T11:46:57.240 INFO:teuthology.orchestra.run.smithi202.stdout: auth supported = cephx 2022-06-27T11:46:57.240 INFO:teuthology.orchestra.run.smithi202.stdout: 2022-06-27T11:46:57.240 INFO:teuthology.orchestra.run.smithi202.stdout: filestore xattr use omap = true 2022-06-27T11:46:57.241 INFO:teuthology.orchestra.run.smithi202.stdout: 2022-06-27T11:46:57.241 INFO:teuthology.orchestra.run.smithi202.stdout: mon clock drift allowed = 1.000 2022-06-27T11:46:57.241 INFO:teuthology.orchestra.run.smithi202.stdout: 2022-06-27T11:46:57.241 INFO:teuthology.orchestra.run.smithi202.stdout: osd crush chooseleaf type = 0 2022-06-27T11:46:57.242 INFO:teuthology.orchestra.run.smithi202.stdout: auth debug = true 2022-06-27T11:46:57.242 INFO:teuthology.orchestra.run.smithi202.stdout: 2022-06-27T11:46:57.242 INFO:teuthology.orchestra.run.smithi202.stdout: ms die on old message = true 2022-06-27T11:46:57.242 INFO:teuthology.orchestra.run.smithi202.stdout: ms die on bug = true 2022-06-27T11:46:57.243 INFO:teuthology.orchestra.run.smithi202.stdout: 2022-06-27T11:46:57.243 INFO:teuthology.orchestra.run.smithi202.stdout: mon max pg per osd = 10000 # >= luminous 2022-06-27T11:46:57.243 INFO:teuthology.orchestra.run.smithi202.stdout: mon pg warn max object skew = 0 2022-06-27T11:46:57.243 INFO:teuthology.orchestra.run.smithi202.stdout: 2022-06-27T11:46:57.244 INFO:teuthology.orchestra.run.smithi202.stdout: # disable pg_autoscaler by default for new pools 2022-06-27T11:46:57.244 INFO:teuthology.orchestra.run.smithi202.stdout: osd_pool_default_pg_autoscale_mode = off 2022-06-27T11:46:57.244 INFO:teuthology.orchestra.run.smithi202.stdout: 2022-06-27T11:46:57.244 INFO:teuthology.orchestra.run.smithi202.stdout: osd pool default size = 2 2022-06-27T11:46:57.244 INFO:teuthology.orchestra.run.smithi202.stdout: 2022-06-27T11:46:57.245 INFO:teuthology.orchestra.run.smithi202.stdout: mon osd allow primary affinity = true 2022-06-27T11:46:57.245 INFO:teuthology.orchestra.run.smithi202.stdout: mon osd allow pg remap = true 2022-06-27T11:46:57.245 INFO:teuthology.orchestra.run.smithi202.stdout: mon warn on legacy crush tunables = false 2022-06-27T11:46:57.245 INFO:teuthology.orchestra.run.smithi202.stdout: mon warn on crush straw calc version zero = false 2022-06-27T11:46:57.246 INFO:teuthology.orchestra.run.smithi202.stdout: mon warn on no sortbitwise = false 2022-06-27T11:46:57.246 INFO:teuthology.orchestra.run.smithi202.stdout: mon warn on osd down out interval zero = false 2022-06-27T11:46:57.246 INFO:teuthology.orchestra.run.smithi202.stdout: mon warn on too few osds = false 2022-06-27T11:46:57.246 INFO:teuthology.orchestra.run.smithi202.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-06-27T11:46:57.247 INFO:teuthology.orchestra.run.smithi202.stdout: mon_warn_on_pool_no_redundancy = false 2022-06-27T11:46:57.247 INFO:teuthology.orchestra.run.smithi202.stdout: mon_allow_pool_size_one = true 2022-06-27T11:46:57.247 INFO:teuthology.orchestra.run.smithi202.stdout: 2022-06-27T11:46:57.247 INFO:teuthology.orchestra.run.smithi202.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2022-06-27T11:46:57.248 INFO:teuthology.orchestra.run.smithi202.stdout: 2022-06-27T11:46:57.248 INFO:teuthology.orchestra.run.smithi202.stdout: osd default data pool replay window = 5 2022-06-27T11:46:57.248 INFO:teuthology.orchestra.run.smithi202.stdout: 2022-06-27T11:46:57.248 INFO:teuthology.orchestra.run.smithi202.stdout: mon allow pool delete = true 2022-06-27T11:46:57.248 INFO:teuthology.orchestra.run.smithi202.stdout: 2022-06-27T11:46:57.249 INFO:teuthology.orchestra.run.smithi202.stdout: mon cluster log file level = debug 2022-06-27T11:46:57.249 INFO:teuthology.orchestra.run.smithi202.stdout: debug asserts on shutdown = true 2022-06-27T11:46:57.249 INFO:teuthology.orchestra.run.smithi202.stdout: mon health detail to clog = false 2022-06-27T11:46:57.249 INFO:teuthology.orchestra.run.smithi202.stdout: mon host = "172.21.15.195,172.21.15.202,[v2:172.21.15.202:3301,v1:172.21.15.202:6790]" 2022-06-27T11:46:57.250 INFO:teuthology.orchestra.run.smithi202.stdout: ms die on skipped message = False 2022-06-27T11:46:57.250 INFO:teuthology.orchestra.run.smithi202.stdout: fsid = 81b3a1d3-6a2b-47e8-8d69-7027cfeae7af 2022-06-27T11:46:57.250 INFO:teuthology.orchestra.run.smithi202.stdout: 2022-06-27T11:46:57.250 INFO:teuthology.orchestra.run.smithi202.stdout:[osd] 2022-06-27T11:46:57.251 INFO:teuthology.orchestra.run.smithi202.stdout: osd journal size = 100 2022-06-27T11:46:57.251 INFO:teuthology.orchestra.run.smithi202.stdout: 2022-06-27T11:46:57.251 INFO:teuthology.orchestra.run.smithi202.stdout: osd scrub load threshold = 5.0 2022-06-27T11:46:57.251 INFO:teuthology.orchestra.run.smithi202.stdout: osd scrub max interval = 600 2022-06-27T11:46:57.251 INFO:teuthology.orchestra.run.smithi202.stdout: 2022-06-27T11:46:57.252 INFO:teuthology.orchestra.run.smithi202.stdout: osd recover clone overlap = true 2022-06-27T11:46:57.252 INFO:teuthology.orchestra.run.smithi202.stdout: osd recovery max chunk = 1048576 2022-06-27T11:46:57.252 INFO:teuthology.orchestra.run.smithi202.stdout: 2022-06-27T11:46:57.252 INFO:teuthology.orchestra.run.smithi202.stdout: osd debug shutdown = true 2022-06-27T11:46:57.253 INFO:teuthology.orchestra.run.smithi202.stdout: osd debug op order = true 2022-06-27T11:46:57.253 INFO:teuthology.orchestra.run.smithi202.stdout: osd debug verify stray on activate = true 2022-06-27T11:46:57.253 INFO:teuthology.orchestra.run.smithi202.stdout: 2022-06-27T11:46:57.253 INFO:teuthology.orchestra.run.smithi202.stdout: osd open classes on start = true 2022-06-27T11:46:57.253 INFO:teuthology.orchestra.run.smithi202.stdout: osd debug pg log writeout = true 2022-06-27T11:46:57.254 INFO:teuthology.orchestra.run.smithi202.stdout: 2022-06-27T11:46:57.254 INFO:teuthology.orchestra.run.smithi202.stdout: osd deep scrub update digest min age = 30 2022-06-27T11:46:57.254 INFO:teuthology.orchestra.run.smithi202.stdout: 2022-06-27T11:46:57.254 INFO:teuthology.orchestra.run.smithi202.stdout: osd map max advance = 10 2022-06-27T11:46:57.255 INFO:teuthology.orchestra.run.smithi202.stdout: 2022-06-27T11:46:57.255 INFO:teuthology.orchestra.run.smithi202.stdout: journal zero on create = true 2022-06-27T11:46:57.255 INFO:teuthology.orchestra.run.smithi202.stdout: 2022-06-27T11:46:57.255 INFO:teuthology.orchestra.run.smithi202.stdout: filestore ondisk finisher threads = 3 2022-06-27T11:46:57.256 INFO:teuthology.orchestra.run.smithi202.stdout: filestore apply finisher threads = 3 2022-06-27T11:46:57.256 INFO:teuthology.orchestra.run.smithi202.stdout: 2022-06-27T11:46:57.256 INFO:teuthology.orchestra.run.smithi202.stdout: bdev debug aio = true 2022-06-27T11:46:57.256 INFO:teuthology.orchestra.run.smithi202.stdout: osd debug misdirected ops = true 2022-06-27T11:46:57.257 INFO:teuthology.orchestra.run.smithi202.stdout: bluestore block size = 96636764160 2022-06-27T11:46:57.257 INFO:teuthology.orchestra.run.smithi202.stdout: bluestore compression mode = aggressive 2022-06-27T11:46:57.257 INFO:teuthology.orchestra.run.smithi202.stdout: bluestore fsck on mount = True 2022-06-27T11:46:57.257 INFO:teuthology.orchestra.run.smithi202.stdout: debug bluefs = 20 2022-06-27T11:46:57.258 INFO:teuthology.orchestra.run.smithi202.stdout: debug bluestore = 20 2022-06-27T11:46:57.258 INFO:teuthology.orchestra.run.smithi202.stdout: debug ms = 1 2022-06-27T11:46:57.258 INFO:teuthology.orchestra.run.smithi202.stdout: debug osd = 20 2022-06-27T11:46:57.258 INFO:teuthology.orchestra.run.smithi202.stdout: debug rocksdb = 10 2022-06-27T11:46:57.258 INFO:teuthology.orchestra.run.smithi202.stdout: mon client directed command retry = 5 2022-06-27T11:46:57.259 INFO:teuthology.orchestra.run.smithi202.stdout: mon osd backfillfull_ratio = 0.85 2022-06-27T11:46:57.259 INFO:teuthology.orchestra.run.smithi202.stdout: mon osd full ratio = 0.9 2022-06-27T11:46:57.259 INFO:teuthology.orchestra.run.smithi202.stdout: mon osd nearfull ratio = 0.8 2022-06-27T11:46:57.259 INFO:teuthology.orchestra.run.smithi202.stdout: ms inject delay max = 1 2022-06-27T11:46:57.260 INFO:teuthology.orchestra.run.smithi202.stdout: ms inject delay probability = 0.005 2022-06-27T11:46:57.260 INFO:teuthology.orchestra.run.smithi202.stdout: ms inject delay type = client mds 2022-06-27T11:46:57.260 INFO:teuthology.orchestra.run.smithi202.stdout: ms inject socket failures = 2500 2022-06-27T11:46:57.260 INFO:teuthology.orchestra.run.smithi202.stdout: osd blocked scrub grace period = 3600 2022-06-27T11:46:57.261 INFO:teuthology.orchestra.run.smithi202.stdout: osd failsafe full ratio = 0.95 2022-06-27T11:46:57.261 INFO:teuthology.orchestra.run.smithi202.stdout: osd max markdown count = 1000 2022-06-27T11:46:57.261 INFO:teuthology.orchestra.run.smithi202.stdout: osd objectstore = bluestore 2022-06-27T11:46:57.261 INFO:teuthology.orchestra.run.smithi202.stdout: osd op complaint time = 180 2022-06-27T11:46:57.261 INFO:teuthology.orchestra.run.smithi202.stdout: 2022-06-27T11:46:57.262 INFO:teuthology.orchestra.run.smithi202.stdout:[mgr] 2022-06-27T11:46:57.262 INFO:teuthology.orchestra.run.smithi202.stdout: debug ms = 1 2022-06-27T11:46:57.262 INFO:teuthology.orchestra.run.smithi202.stdout: debug mgr = 20 2022-06-27T11:46:57.262 INFO:teuthology.orchestra.run.smithi202.stdout: debug mon = 20 2022-06-27T11:46:57.263 INFO:teuthology.orchestra.run.smithi202.stdout: debug auth = 20 2022-06-27T11:46:57.263 INFO:teuthology.orchestra.run.smithi202.stdout: mon reweight min pgs per osd = 4 2022-06-27T11:46:57.263 INFO:teuthology.orchestra.run.smithi202.stdout: mon reweight min bytes per osd = 10 2022-06-27T11:46:57.263 INFO:teuthology.orchestra.run.smithi202.stdout: mgr/telemetry/nag = false 2022-06-27T11:46:57.264 INFO:teuthology.orchestra.run.smithi202.stdout: 2022-06-27T11:46:57.264 INFO:teuthology.orchestra.run.smithi202.stdout:[mon] 2022-06-27T11:46:57.264 INFO:teuthology.orchestra.run.smithi202.stdout: debug ms = 1 2022-06-27T11:46:57.264 INFO:teuthology.orchestra.run.smithi202.stdout: debug mon = 20 2022-06-27T11:46:57.264 INFO:teuthology.orchestra.run.smithi202.stdout: debug paxos = 20 2022-06-27T11:46:57.265 INFO:teuthology.orchestra.run.smithi202.stdout: debug auth = 20 2022-06-27T11:46:57.265 INFO:teuthology.orchestra.run.smithi202.stdout: mon data avail warn = 5 2022-06-27T11:46:57.265 INFO:teuthology.orchestra.run.smithi202.stdout: mon mgr mkfs grace = 240 2022-06-27T11:46:57.265 INFO:teuthology.orchestra.run.smithi202.stdout: mon reweight min pgs per osd = 4 2022-06-27T11:46:57.265 INFO:teuthology.orchestra.run.smithi202.stdout: mon osd reporter subtree level = osd 2022-06-27T11:46:57.266 INFO:teuthology.orchestra.run.smithi202.stdout: mon osd prime pg temp = true 2022-06-27T11:46:57.266 INFO:teuthology.orchestra.run.smithi202.stdout: mon reweight min bytes per osd = 10 2022-06-27T11:46:57.266 INFO:teuthology.orchestra.run.smithi202.stdout: 2022-06-27T11:46:57.266 INFO:teuthology.orchestra.run.smithi202.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-06-27T11:46:57.267 INFO:teuthology.orchestra.run.smithi202.stdout: auth mon ticket ttl = 660 # 11m 2022-06-27T11:46:57.267 INFO:teuthology.orchestra.run.smithi202.stdout: auth service ticket ttl = 240 # 4m 2022-06-27T11:46:57.267 INFO:teuthology.orchestra.run.smithi202.stdout: 2022-06-27T11:46:57.267 INFO:teuthology.orchestra.run.smithi202.stdout: # don't complain about insecure global_id in the test suite 2022-06-27T11:46:57.267 INFO:teuthology.orchestra.run.smithi202.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-06-27T11:46:57.268 INFO:teuthology.orchestra.run.smithi202.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-06-27T11:46:57.268 INFO:teuthology.orchestra.run.smithi202.stdout: 2022-06-27T11:46:57.268 INFO:teuthology.orchestra.run.smithi202.stdout: # 1m isn't quite enough 2022-06-27T11:46:57.268 INFO:teuthology.orchestra.run.smithi202.stdout: mon_down_mkfs_grace = 2m 2022-06-27T11:46:57.269 INFO:teuthology.orchestra.run.smithi202.stdout: 2022-06-27T11:46:57.269 INFO:teuthology.orchestra.run.smithi202.stdout: mon_warn_on_filestore_osds = false 2022-06-27T11:46:57.269 INFO:teuthology.orchestra.run.smithi202.stdout: mon op complaint time = 120 2022-06-27T11:46:57.269 INFO:teuthology.orchestra.run.smithi202.stdout: 2022-06-27T11:46:57.269 INFO:teuthology.orchestra.run.smithi202.stdout:[client] 2022-06-27T11:46:57.270 INFO:teuthology.orchestra.run.smithi202.stdout: rgw cache enabled = true 2022-06-27T11:46:57.270 INFO:teuthology.orchestra.run.smithi202.stdout: rgw enable ops log = true 2022-06-27T11:46:57.270 INFO:teuthology.orchestra.run.smithi202.stdout: rgw enable usage log = true 2022-06-27T11:46:57.270 INFO:teuthology.orchestra.run.smithi202.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-06-27T11:46:57.271 INFO:teuthology.orchestra.run.smithi202.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-06-27T11:46:57.271 INFO:teuthology.orchestra.run.smithi202.stdout: client mount timeout = 600 2022-06-27T11:46:57.271 INFO:teuthology.orchestra.run.smithi202.stdout: debug client = 20 2022-06-27T11:46:57.271 INFO:teuthology.orchestra.run.smithi202.stdout: debug ms = 1 2022-06-27T11:46:57.272 INFO:teuthology.orchestra.run.smithi202.stdout: rados mon op timeout = 900 2022-06-27T11:46:57.272 INFO:teuthology.orchestra.run.smithi202.stdout: rados osd op timeout = 900 2022-06-27T11:46:57.272 INFO:teuthology.orchestra.run.smithi202.stdout:[mon.a] 2022-06-27T11:46:57.272 INFO:teuthology.orchestra.run.smithi202.stdout:[mon.b] 2022-06-27T11:46:57.273 INFO:teuthology.orchestra.run.smithi202.stdout:[mon.c] 2022-06-27T11:46:57.273 INFO:teuthology.orchestra.run.smithi202.stdout:[mds.a] 2022-06-27T11:46:57.273 INFO:teuthology.orchestra.run.smithi202.stdout:[mds.c] 2022-06-27T11:46:57.273 INFO:teuthology.orchestra.run.smithi202.stdout:[mds.e] 2022-06-27T11:46:57.274 INFO:teuthology.orchestra.run.smithi202.stdout:[mds.b] 2022-06-27T11:46:57.274 INFO:teuthology.orchestra.run.smithi202.stdout:[mds.d] 2022-06-27T11:46:57.274 INFO:teuthology.orchestra.run.smithi202.stdout:[mds.f] 2022-06-27T11:46:57.274 INFO:teuthology.orchestra.run.smithi202.stdout:[mds] 2022-06-27T11:46:57.275 INFO:teuthology.orchestra.run.smithi202.stdout: debug mds = 20 2022-06-27T11:46:57.275 INFO:teuthology.orchestra.run.smithi202.stdout: debug mds balancer = 20 2022-06-27T11:46:57.275 INFO:teuthology.orchestra.run.smithi202.stdout: debug ms = 1 2022-06-27T11:46:57.275 INFO:teuthology.orchestra.run.smithi202.stdout: mds bal fragment size max = 10000 2022-06-27T11:46:57.276 INFO:teuthology.orchestra.run.smithi202.stdout: mds bal merge size = 5 2022-06-27T11:46:57.276 INFO:teuthology.orchestra.run.smithi202.stdout: mds bal split bits = 3 2022-06-27T11:46:57.276 INFO:teuthology.orchestra.run.smithi202.stdout: mds bal split size = 100 2022-06-27T11:46:57.276 INFO:teuthology.orchestra.run.smithi202.stdout: mds debug frag = True 2022-06-27T11:46:57.277 INFO:teuthology.orchestra.run.smithi202.stdout: mds debug scatterstat = True 2022-06-27T11:46:57.277 INFO:teuthology.orchestra.run.smithi202.stdout: mds oft prefetch dirfrags = True 2022-06-27T11:46:57.277 INFO:teuthology.orchestra.run.smithi202.stdout: mds op complaint time = 180 2022-06-27T11:46:57.277 INFO:teuthology.orchestra.run.smithi202.stdout: mds verify scatter = True 2022-06-27T11:46:57.277 INFO:teuthology.orchestra.run.smithi202.stdout: mds_dir_prefetch = False 2022-06-27T11:46:57.278 INFO:teuthology.orchestra.run.smithi202.stdout: mds_sleep_rank_change = 5000000.0 2022-06-27T11:46:57.278 INFO:teuthology.orchestra.run.smithi202.stdout: mon client directed command retry = 5 2022-06-27T11:46:57.278 INFO:teuthology.orchestra.run.smithi202.stdout: ms inject delay max = 1 2022-06-27T11:46:57.279 INFO:teuthology.orchestra.run.smithi202.stdout: ms inject delay probability = 0.005 2022-06-27T11:46:57.279 INFO:teuthology.orchestra.run.smithi202.stdout: ms inject delay type = client mds osd 2022-06-27T11:46:57.279 INFO:teuthology.orchestra.run.smithi202.stdout: ms inject socket failures = 2500 2022-06-27T11:46:57.279 INFO:teuthology.orchestra.run.smithi202.stdout: osd op complaint time = 180 2022-06-27T11:46:57.279 INFO:teuthology.orchestra.run.smithi202.stdout: rados mon op timeout = 900 2022-06-27T11:46:57.280 INFO:teuthology.orchestra.run.smithi202.stdout: rados osd op timeout = 900 2022-06-27T11:46:57.280 INFO:tasks.ceph:Creating admin key on mon.a... 2022-06-27T11:46:57.281 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2022-06-27T11:46:57.338 INFO:tasks.ceph:Copying monmap to all nodes... 2022-06-27T11:46:57.339 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2022-06-27T11:46:57.340 DEBUG:teuthology.orchestra.run.smithi195:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2022-06-27T11:46:57.358 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2022-06-27T11:46:57.359 DEBUG:teuthology.orchestra.run.smithi195:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2022-06-27T11:46:57.416 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi195.front.sepia.ceph.com 2022-06-27T11:46:57.416 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2022-06-27T11:46:57.417 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/etc/ceph/ceph.keyring 2022-06-27T11:46:57.417 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod 0644 /etc/ceph/ceph.keyring 2022-06-27T11:46:57.495 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2022-06-27T11:46:57.495 DEBUG:teuthology.orchestra.run.smithi195:> dd of=/home/ubuntu/cephtest/ceph.monmap 2022-06-27T11:46:57.552 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi202.front.sepia.ceph.com 2022-06-27T11:46:57.553 DEBUG:teuthology.orchestra.run.smithi202:> set -ex 2022-06-27T11:46:57.553 DEBUG:teuthology.orchestra.run.smithi202:> sudo dd of=/etc/ceph/ceph.keyring 2022-06-27T11:46:57.554 DEBUG:teuthology.orchestra.run.smithi202:> sudo chmod 0644 /etc/ceph/ceph.keyring 2022-06-27T11:46:57.593 DEBUG:teuthology.orchestra.run.smithi202:> set -ex 2022-06-27T11:46:57.594 DEBUG:teuthology.orchestra.run.smithi202:> dd of=/home/ubuntu/cephtest/ceph.monmap 2022-06-27T11:46:57.651 INFO:tasks.ceph:Setting up mon nodes... 2022-06-27T11:46:57.652 INFO:tasks.ceph:Setting up mgr nodes... 2022-06-27T11:46:57.652 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2022-06-27T11:46:57.706 INFO:teuthology.orchestra.run.smithi195.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2022-06-27T11:46:57.709 DEBUG:teuthology.orchestra.run.smithi202:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2022-06-27T11:46:57.781 INFO:teuthology.orchestra.run.smithi202.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2022-06-27T11:46:57.784 INFO:tasks.ceph:Setting up mds nodes... 2022-06-27T11:46:57.785 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2022-06-27T11:46:57.838 INFO:teuthology.orchestra.run.smithi195.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2022-06-27T11:46:57.841 DEBUG:teuthology.orchestra.run.smithi195:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2022-06-27T11:46:57.869 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2022-06-27T11:46:57.962 INFO:teuthology.orchestra.run.smithi195.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2022-06-27T11:46:57.964 DEBUG:teuthology.orchestra.run.smithi195:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2022-06-27T11:46:57.992 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /var/lib/ceph/mds/ceph-e && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.e /var/lib/ceph/mds/ceph-e/keyring 2022-06-27T11:46:58.084 INFO:teuthology.orchestra.run.smithi195.stdout:creating /var/lib/ceph/mds/ceph-e/keyring 2022-06-27T11:46:58.087 DEBUG:teuthology.orchestra.run.smithi195:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-e 2022-06-27T11:46:58.115 DEBUG:teuthology.orchestra.run.smithi202:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2022-06-27T11:46:58.170 INFO:teuthology.orchestra.run.smithi202.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2022-06-27T11:46:58.173 DEBUG:teuthology.orchestra.run.smithi202:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2022-06-27T11:46:58.199 DEBUG:teuthology.orchestra.run.smithi202:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2022-06-27T11:46:58.293 INFO:teuthology.orchestra.run.smithi202.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2022-06-27T11:46:58.296 DEBUG:teuthology.orchestra.run.smithi202:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2022-06-27T11:46:58.322 DEBUG:teuthology.orchestra.run.smithi202:> sudo mkdir -p /var/lib/ceph/mds/ceph-f && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.f /var/lib/ceph/mds/ceph-f/keyring 2022-06-27T11:46:58.418 INFO:teuthology.orchestra.run.smithi202.stdout:creating /var/lib/ceph/mds/ceph-f/keyring 2022-06-27T11:46:58.422 DEBUG:teuthology.orchestra.run.smithi202:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-f 2022-06-27T11:46:58.454 INFO:tasks.ceph_client:Setting up client nodes... 2022-06-27T11:46:58.455 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2022-06-27T11:46:58.500 INFO:teuthology.orchestra.run.smithi195.stdout:creating /etc/ceph/ceph.client.0.keyring 2022-06-27T11:46:58.513 INFO:tasks.ceph:Running mkfs on osd nodes... 2022-06-27T11:46:58.514 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi195.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi202.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2022-06-27T11:46:58.514 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2022-06-27T11:46:58.581 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2022-06-27T11:46:58.582 INFO:tasks.ceph:role: osd.0 2022-06-27T11:46:58.583 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi195.front.sepia.ceph.com 2022-06-27T11:46:58.583 DEBUG:teuthology.orchestra.run.smithi195:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2022-06-27T11:46:58.655 INFO:teuthology.orchestra.run.smithi195.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2022-06-27T11:46:58.656 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 attr=2, projid32bit=1 2022-06-27T11:46:58.656 INFO:teuthology.orchestra.run.smithi195.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-06-27T11:46:58.656 INFO:teuthology.orchestra.run.smithi195.stdout: = reflink=1 bigtime=0 inobtcount=0 2022-06-27T11:46:58.656 INFO:teuthology.orchestra.run.smithi195.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-06-27T11:46:58.657 INFO:teuthology.orchestra.run.smithi195.stdout: = sunit=0 swidth=0 blks 2022-06-27T11:46:58.657 INFO:teuthology.orchestra.run.smithi195.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-06-27T11:46:58.657 INFO:teuthology.orchestra.run.smithi195.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-06-27T11:46:58.657 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-06-27T11:46:58.658 INFO:teuthology.orchestra.run.smithi195.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-06-27T11:46:58.678 INFO:teuthology.orchestra.run.smithi195.stdout:Discarding blocks...Done. 2022-06-27T11:46:58.680 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi195.front.sepia.ceph.com -o noatime 2022-06-27T11:46:58.680 DEBUG:teuthology.orchestra.run.smithi195:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2022-06-27T11:46:58.761 DEBUG:teuthology.orchestra.run.smithi195:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2022-06-27T11:46:58.835 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2022-06-27T11:46:58.903 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2022-06-27T11:46:58.904 INFO:tasks.ceph:role: osd.1 2022-06-27T11:46:58.904 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi195.front.sepia.ceph.com 2022-06-27T11:46:58.905 DEBUG:teuthology.orchestra.run.smithi195:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2022-06-27T11:46:58.970 INFO:teuthology.orchestra.run.smithi195.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2022-06-27T11:46:58.970 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 attr=2, projid32bit=1 2022-06-27T11:46:58.971 INFO:teuthology.orchestra.run.smithi195.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-06-27T11:46:58.971 INFO:teuthology.orchestra.run.smithi195.stdout: = reflink=1 bigtime=0 inobtcount=0 2022-06-27T11:46:58.971 INFO:teuthology.orchestra.run.smithi195.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-06-27T11:46:58.972 INFO:teuthology.orchestra.run.smithi195.stdout: = sunit=0 swidth=0 blks 2022-06-27T11:46:58.972 INFO:teuthology.orchestra.run.smithi195.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-06-27T11:46:58.972 INFO:teuthology.orchestra.run.smithi195.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-06-27T11:46:58.972 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-06-27T11:46:58.973 INFO:teuthology.orchestra.run.smithi195.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-06-27T11:46:58.992 INFO:teuthology.orchestra.run.smithi195.stdout:Discarding blocks...Done. 2022-06-27T11:46:58.994 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi195.front.sepia.ceph.com -o noatime 2022-06-27T11:46:58.995 DEBUG:teuthology.orchestra.run.smithi195:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2022-06-27T11:46:59.075 DEBUG:teuthology.orchestra.run.smithi195:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2022-06-27T11:46:59.145 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2022-06-27T11:46:59.212 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2022-06-27T11:46:59.213 INFO:tasks.ceph:role: osd.2 2022-06-27T11:46:59.213 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi195.front.sepia.ceph.com 2022-06-27T11:46:59.214 DEBUG:teuthology.orchestra.run.smithi195:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2022-06-27T11:46:59.283 INFO:teuthology.orchestra.run.smithi195.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2022-06-27T11:46:59.283 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 attr=2, projid32bit=1 2022-06-27T11:46:59.283 INFO:teuthology.orchestra.run.smithi195.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-06-27T11:46:59.284 INFO:teuthology.orchestra.run.smithi195.stdout: = reflink=1 bigtime=0 inobtcount=0 2022-06-27T11:46:59.284 INFO:teuthology.orchestra.run.smithi195.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-06-27T11:46:59.284 INFO:teuthology.orchestra.run.smithi195.stdout: = sunit=0 swidth=0 blks 2022-06-27T11:46:59.284 INFO:teuthology.orchestra.run.smithi195.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-06-27T11:46:59.285 INFO:teuthology.orchestra.run.smithi195.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-06-27T11:46:59.285 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-06-27T11:46:59.285 INFO:teuthology.orchestra.run.smithi195.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-06-27T11:46:59.305 INFO:teuthology.orchestra.run.smithi195.stdout:Discarding blocks...Done. 2022-06-27T11:46:59.307 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi195.front.sepia.ceph.com -o noatime 2022-06-27T11:46:59.307 DEBUG:teuthology.orchestra.run.smithi195:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2022-06-27T11:46:59.387 DEBUG:teuthology.orchestra.run.smithi195:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2022-06-27T11:46:59.458 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2022-06-27T11:46:59.525 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2022-06-27T11:46:59.526 INFO:tasks.ceph:role: osd.3 2022-06-27T11:46:59.527 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi195.front.sepia.ceph.com 2022-06-27T11:46:59.527 DEBUG:teuthology.orchestra.run.smithi195:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2022-06-27T11:46:59.591 INFO:teuthology.orchestra.run.smithi195.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2022-06-27T11:46:59.592 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 attr=2, projid32bit=1 2022-06-27T11:46:59.592 INFO:teuthology.orchestra.run.smithi195.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-06-27T11:46:59.593 INFO:teuthology.orchestra.run.smithi195.stdout: = reflink=1 bigtime=0 inobtcount=0 2022-06-27T11:46:59.593 INFO:teuthology.orchestra.run.smithi195.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-06-27T11:46:59.594 INFO:teuthology.orchestra.run.smithi195.stdout: = sunit=0 swidth=0 blks 2022-06-27T11:46:59.595 INFO:teuthology.orchestra.run.smithi195.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-06-27T11:46:59.595 INFO:teuthology.orchestra.run.smithi195.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-06-27T11:46:59.596 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-06-27T11:46:59.596 INFO:teuthology.orchestra.run.smithi195.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-06-27T11:46:59.619 INFO:teuthology.orchestra.run.smithi195.stdout:Discarding blocks...Done. 2022-06-27T11:46:59.620 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi195.front.sepia.ceph.com -o noatime 2022-06-27T11:46:59.621 DEBUG:teuthology.orchestra.run.smithi195:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2022-06-27T11:46:59.700 DEBUG:teuthology.orchestra.run.smithi195:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2022-06-27T11:46:59.779 DEBUG:teuthology.orchestra.run.smithi195:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-06-27T11:46:59.866 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:46:59.866+0000 7fde5adf5440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2022-06-27T11:46:59.867 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:46:59.866+0000 7fde5adf5440 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2022-06-27T11:46:59.867 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:46:59.866+0000 7fde5adf5440 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2022-06-27T11:46:59.868 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:46:59.866+0000 7fde5adf5440 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2022-06-27T11:46:59.868 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:46:59.866+0000 7fde5adf5440 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2022-06-27T11:46:59.868 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:46:59.866+0000 7fde5adf5440 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2022-06-27T11:47:02.740 DEBUG:teuthology.orchestra.run.smithi195:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2022-06-27T11:47:02.774 DEBUG:teuthology.orchestra.run.smithi195:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-06-27T11:47:02.867 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:02.867+0000 7f73524cb440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2022-06-27T11:47:02.867 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:02.867+0000 7f73524cb440 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2022-06-27T11:47:02.867 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:02.867+0000 7f73524cb440 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2022-06-27T11:47:02.867 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:02.867+0000 7f73524cb440 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2022-06-27T11:47:02.868 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:02.867+0000 7f73524cb440 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2022-06-27T11:47:02.868 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:02.867+0000 7f73524cb440 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2022-06-27T11:47:05.727 DEBUG:teuthology.orchestra.run.smithi195:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2022-06-27T11:47:05.758 DEBUG:teuthology.orchestra.run.smithi195:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-06-27T11:47:05.961 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:05.961+0000 7fba9f712440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2022-06-27T11:47:05.962 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:05.961+0000 7fba9f712440 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2022-06-27T11:47:05.962 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:05.961+0000 7fba9f712440 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2022-06-27T11:47:05.962 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:05.961+0000 7fba9f712440 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2022-06-27T11:47:05.963 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:05.961+0000 7fba9f712440 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2022-06-27T11:47:05.963 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:05.961+0000 7fba9f712440 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2022-06-27T11:47:08.816 DEBUG:teuthology.orchestra.run.smithi195:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2022-06-27T11:47:08.849 DEBUG:teuthology.orchestra.run.smithi195:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-06-27T11:47:08.943 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:08.943+0000 7f78861ea440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2022-06-27T11:47:08.944 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:08.943+0000 7f78861ea440 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2022-06-27T11:47:08.945 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:08.943+0000 7f78861ea440 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2022-06-27T11:47:08.948 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:08.943+0000 7f78861ea440 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2022-06-27T11:47:08.948 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:08.943+0000 7f78861ea440 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2022-06-27T11:47:08.948 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:08.944+0000 7f78861ea440 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2022-06-27T11:47:11.802 DEBUG:teuthology.orchestra.run.smithi195:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2022-06-27T11:47:11.834 DEBUG:teuthology.orchestra.run.smithi202:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2022-06-27T11:47:11.868 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2022-06-27T11:47:11.868 INFO:tasks.ceph:role: osd.4 2022-06-27T11:47:11.869 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi202.front.sepia.ceph.com 2022-06-27T11:47:11.869 DEBUG:teuthology.orchestra.run.smithi202:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2022-06-27T11:47:11.947 INFO:teuthology.orchestra.run.smithi202.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2022-06-27T11:47:11.947 INFO:teuthology.orchestra.run.smithi202.stdout: = sectsz=512 attr=2, projid32bit=1 2022-06-27T11:47:11.947 INFO:teuthology.orchestra.run.smithi202.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-06-27T11:47:11.948 INFO:teuthology.orchestra.run.smithi202.stdout: = reflink=1 bigtime=0 inobtcount=0 2022-06-27T11:47:11.948 INFO:teuthology.orchestra.run.smithi202.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-06-27T11:47:11.948 INFO:teuthology.orchestra.run.smithi202.stdout: = sunit=0 swidth=0 blks 2022-06-27T11:47:11.948 INFO:teuthology.orchestra.run.smithi202.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-06-27T11:47:11.949 INFO:teuthology.orchestra.run.smithi202.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-06-27T11:47:11.949 INFO:teuthology.orchestra.run.smithi202.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-06-27T11:47:11.949 INFO:teuthology.orchestra.run.smithi202.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-06-27T11:47:12.387 INFO:teuthology.orchestra.run.smithi202.stdout:Discarding blocks...Done. 2022-06-27T11:47:12.390 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi202.front.sepia.ceph.com -o noatime 2022-06-27T11:47:12.390 DEBUG:teuthology.orchestra.run.smithi202:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2022-06-27T11:47:12.568 DEBUG:teuthology.orchestra.run.smithi202:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2022-06-27T11:47:12.611 DEBUG:teuthology.orchestra.run.smithi202:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2022-06-27T11:47:12.683 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2022-06-27T11:47:12.683 INFO:tasks.ceph:role: osd.5 2022-06-27T11:47:12.684 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi202.front.sepia.ceph.com 2022-06-27T11:47:12.684 DEBUG:teuthology.orchestra.run.smithi202:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2022-06-27T11:47:12.761 INFO:teuthology.orchestra.run.smithi202.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2022-06-27T11:47:12.762 INFO:teuthology.orchestra.run.smithi202.stdout: = sectsz=512 attr=2, projid32bit=1 2022-06-27T11:47:12.762 INFO:teuthology.orchestra.run.smithi202.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-06-27T11:47:12.762 INFO:teuthology.orchestra.run.smithi202.stdout: = reflink=1 bigtime=0 inobtcount=0 2022-06-27T11:47:12.763 INFO:teuthology.orchestra.run.smithi202.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-06-27T11:47:12.763 INFO:teuthology.orchestra.run.smithi202.stdout: = sunit=0 swidth=0 blks 2022-06-27T11:47:12.763 INFO:teuthology.orchestra.run.smithi202.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-06-27T11:47:12.763 INFO:teuthology.orchestra.run.smithi202.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-06-27T11:47:12.764 INFO:teuthology.orchestra.run.smithi202.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-06-27T11:47:12.764 INFO:teuthology.orchestra.run.smithi202.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-06-27T11:47:13.167 INFO:teuthology.orchestra.run.smithi202.stdout:Discarding blocks...Done. 2022-06-27T11:47:13.169 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi202.front.sepia.ceph.com -o noatime 2022-06-27T11:47:13.170 DEBUG:teuthology.orchestra.run.smithi202:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2022-06-27T11:47:13.281 DEBUG:teuthology.orchestra.run.smithi202:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2022-06-27T11:47:13.319 DEBUG:teuthology.orchestra.run.smithi202:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2022-06-27T11:47:13.391 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2022-06-27T11:47:13.392 INFO:tasks.ceph:role: osd.6 2022-06-27T11:47:13.392 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi202.front.sepia.ceph.com 2022-06-27T11:47:13.393 DEBUG:teuthology.orchestra.run.smithi202:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2022-06-27T11:47:13.472 INFO:teuthology.orchestra.run.smithi202.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2022-06-27T11:47:13.472 INFO:teuthology.orchestra.run.smithi202.stdout: = sectsz=512 attr=2, projid32bit=1 2022-06-27T11:47:13.472 INFO:teuthology.orchestra.run.smithi202.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-06-27T11:47:13.473 INFO:teuthology.orchestra.run.smithi202.stdout: = reflink=1 bigtime=0 inobtcount=0 2022-06-27T11:47:13.473 INFO:teuthology.orchestra.run.smithi202.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-06-27T11:47:13.473 INFO:teuthology.orchestra.run.smithi202.stdout: = sunit=0 swidth=0 blks 2022-06-27T11:47:13.474 INFO:teuthology.orchestra.run.smithi202.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-06-27T11:47:13.474 INFO:teuthology.orchestra.run.smithi202.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-06-27T11:47:13.474 INFO:teuthology.orchestra.run.smithi202.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-06-27T11:47:13.474 INFO:teuthology.orchestra.run.smithi202.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-06-27T11:47:13.915 INFO:teuthology.orchestra.run.smithi202.stdout:Discarding blocks...Done. 2022-06-27T11:47:13.917 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi202.front.sepia.ceph.com -o noatime 2022-06-27T11:47:13.917 DEBUG:teuthology.orchestra.run.smithi202:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2022-06-27T11:47:14.055 DEBUG:teuthology.orchestra.run.smithi202:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2022-06-27T11:47:14.096 DEBUG:teuthology.orchestra.run.smithi202:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2022-06-27T11:47:14.170 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2022-06-27T11:47:14.170 INFO:tasks.ceph:role: osd.7 2022-06-27T11:47:14.171 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi202.front.sepia.ceph.com 2022-06-27T11:47:14.171 DEBUG:teuthology.orchestra.run.smithi202:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2022-06-27T11:47:14.250 INFO:teuthology.orchestra.run.smithi202.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2022-06-27T11:47:14.251 INFO:teuthology.orchestra.run.smithi202.stdout: = sectsz=512 attr=2, projid32bit=1 2022-06-27T11:47:14.251 INFO:teuthology.orchestra.run.smithi202.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-06-27T11:47:14.251 INFO:teuthology.orchestra.run.smithi202.stdout: = reflink=1 bigtime=0 inobtcount=0 2022-06-27T11:47:14.251 INFO:teuthology.orchestra.run.smithi202.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-06-27T11:47:14.252 INFO:teuthology.orchestra.run.smithi202.stdout: = sunit=0 swidth=0 blks 2022-06-27T11:47:14.252 INFO:teuthology.orchestra.run.smithi202.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-06-27T11:47:14.252 INFO:teuthology.orchestra.run.smithi202.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-06-27T11:47:14.252 INFO:teuthology.orchestra.run.smithi202.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-06-27T11:47:14.253 INFO:teuthology.orchestra.run.smithi202.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-06-27T11:47:14.698 INFO:teuthology.orchestra.run.smithi202.stdout:Discarding blocks...Done. 2022-06-27T11:47:14.700 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi202.front.sepia.ceph.com -o noatime 2022-06-27T11:47:14.701 DEBUG:teuthology.orchestra.run.smithi202:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2022-06-27T11:47:14.857 DEBUG:teuthology.orchestra.run.smithi202:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2022-06-27T11:47:14.897 DEBUG:teuthology.orchestra.run.smithi202:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-06-27T11:47:14.996 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:14.995+0000 7f3dac563440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2022-06-27T11:47:14.997 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:14.995+0000 7f3dac563440 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2022-06-27T11:47:14.997 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:14.995+0000 7f3dac563440 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2022-06-27T11:47:14.997 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:14.995+0000 7f3dac563440 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2022-06-27T11:47:14.998 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:14.995+0000 7f3dac563440 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2022-06-27T11:47:14.998 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:14.995+0000 7f3dac563440 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2022-06-27T11:47:17.867 DEBUG:teuthology.orchestra.run.smithi202:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2022-06-27T11:47:17.903 DEBUG:teuthology.orchestra.run.smithi202:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-06-27T11:47:17.999 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:17.999+0000 7f4f01237440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2022-06-27T11:47:18.000 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:17.999+0000 7f4f01237440 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2022-06-27T11:47:18.000 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:17.999+0000 7f4f01237440 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2022-06-27T11:47:18.001 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:17.999+0000 7f4f01237440 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2022-06-27T11:47:18.001 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:17.999+0000 7f4f01237440 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2022-06-27T11:47:18.001 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:17.999+0000 7f4f01237440 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2022-06-27T11:47:20.851 DEBUG:teuthology.orchestra.run.smithi202:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2022-06-27T11:47:20.886 DEBUG:teuthology.orchestra.run.smithi202:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-06-27T11:47:20.981 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:20.980+0000 7f05e95fd440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2022-06-27T11:47:20.982 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:20.980+0000 7f05e95fd440 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2022-06-27T11:47:20.982 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:20.980+0000 7f05e95fd440 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2022-06-27T11:47:20.983 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:20.980+0000 7f05e95fd440 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2022-06-27T11:47:20.983 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:20.980+0000 7f05e95fd440 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2022-06-27T11:47:20.983 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:20.981+0000 7f05e95fd440 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2022-06-27T11:47:23.841 DEBUG:teuthology.orchestra.run.smithi202:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2022-06-27T11:47:23.876 DEBUG:teuthology.orchestra.run.smithi202:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-06-27T11:47:23.973 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:23.972+0000 7ff4f79d1440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2022-06-27T11:47:23.974 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:23.973+0000 7ff4f79d1440 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2022-06-27T11:47:23.974 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:23.973+0000 7ff4f79d1440 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2022-06-27T11:47:23.975 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:23.973+0000 7ff4f79d1440 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2022-06-27T11:47:23.975 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:23.973+0000 7ff4f79d1440 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2022-06-27T11:47:23.975 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:23.973+0000 7ff4f79d1440 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2022-06-27T11:47:26.052 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T11:47:26.056 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T11:47:26.855 DEBUG:teuthology.orchestra.run.smithi202:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2022-06-27T11:47:26.885 INFO:tasks.ceph:Reading keys from all nodes... 2022-06-27T11:47:26.886 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2022-06-27T11:47:26.886 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2022-06-27T11:47:26.918 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2022-06-27T11:47:26.919 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2022-06-27T11:47:26.989 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2022-06-27T11:47:26.989 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2022-06-27T11:47:27.059 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2022-06-27T11:47:27.060 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/mds/ceph-e/keyring of=/dev/stdout 2022-06-27T11:47:27.129 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2022-06-27T11:47:27.130 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2022-06-27T11:47:27.214 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2022-06-27T11:47:27.214 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2022-06-27T11:47:27.297 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2022-06-27T11:47:27.297 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2022-06-27T11:47:27.366 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2022-06-27T11:47:27.367 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2022-06-27T11:47:27.434 DEBUG:teuthology.orchestra.run.smithi202:> set -ex 2022-06-27T11:47:27.434 DEBUG:teuthology.orchestra.run.smithi202:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2022-06-27T11:47:27.467 DEBUG:teuthology.orchestra.run.smithi202:> set -ex 2022-06-27T11:47:27.468 DEBUG:teuthology.orchestra.run.smithi202:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2022-06-27T11:47:27.541 DEBUG:teuthology.orchestra.run.smithi202:> set -ex 2022-06-27T11:47:27.542 DEBUG:teuthology.orchestra.run.smithi202:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2022-06-27T11:47:27.617 DEBUG:teuthology.orchestra.run.smithi202:> set -ex 2022-06-27T11:47:27.618 DEBUG:teuthology.orchestra.run.smithi202:> sudo dd if=/var/lib/ceph/mds/ceph-f/keyring of=/dev/stdout 2022-06-27T11:47:27.689 DEBUG:teuthology.orchestra.run.smithi202:> set -ex 2022-06-27T11:47:27.690 DEBUG:teuthology.orchestra.run.smithi202:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2022-06-27T11:47:27.758 DEBUG:teuthology.orchestra.run.smithi202:> set -ex 2022-06-27T11:47:27.759 DEBUG:teuthology.orchestra.run.smithi202:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2022-06-27T11:47:27.826 DEBUG:teuthology.orchestra.run.smithi202:> set -ex 2022-06-27T11:47:27.826 DEBUG:teuthology.orchestra.run.smithi202:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2022-06-27T11:47:27.902 DEBUG:teuthology.orchestra.run.smithi202:> set -ex 2022-06-27T11:47:27.903 DEBUG:teuthology.orchestra.run.smithi202:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2022-06-27T11:47:27.970 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2022-06-27T11:47:27.971 DEBUG:teuthology.orchestra.run.smithi195:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2022-06-27T11:47:27.992 INFO:tasks.ceph:Adding keys to all mons... 2022-06-27T11:47:27.993 DEBUG:teuthology.orchestra.run.smithi195:> sudo tee -a /etc/ceph/ceph.keyring 2022-06-27T11:47:28.037 DEBUG:teuthology.orchestra.run.smithi202:> sudo tee -a /etc/ceph/ceph.keyring 2022-06-27T11:47:28.064 INFO:teuthology.orchestra.run.smithi202.stdout:[mgr.y] 2022-06-27T11:47:28.064 INFO:teuthology.orchestra.run.smithi202.stdout: key = AQCxmLli0rIxKhAAgLRFXPsuUUqiaDB3sTCq0A== 2022-06-27T11:47:28.065 INFO:teuthology.orchestra.run.smithi202.stdout:[mds.a] 2022-06-27T11:47:28.065 INFO:teuthology.orchestra.run.smithi202.stdout: key = AQCxmLlisqsZMhAAkLUNtHd96zkgYjbcNdU4nA== 2022-06-27T11:47:28.066 INFO:teuthology.orchestra.run.smithi202.stdout:[mds.c] 2022-06-27T11:47:28.066 INFO:teuthology.orchestra.run.smithi202.stdout: key = AQCxmLlic1F0ORAAz9pgZBBfWyQ/f5Km31WT+A== 2022-06-27T11:47:28.066 INFO:teuthology.orchestra.run.smithi202.stdout:[mds.e] 2022-06-27T11:47:28.067 INFO:teuthology.orchestra.run.smithi202.stdout: key = AQCymLliWlEiBRAA/Lm7D1zhUcMhJDHs66Rzbw== 2022-06-27T11:47:28.067 INFO:teuthology.orchestra.run.smithi202.stdout:[osd.0] 2022-06-27T11:47:28.068 INFO:teuthology.orchestra.run.smithi202.stdout: key = AQCzmLliVV22MxAACUSkiQASt1HcBKzwFm2pyw== 2022-06-27T11:47:28.068 INFO:teuthology.orchestra.run.smithi202.stdout:[osd.1] 2022-06-27T11:47:28.069 INFO:teuthology.orchestra.run.smithi202.stdout: key = AQC2mLliz82+MxAA8MfPSBa/IFhizDqgrQKPbQ== 2022-06-27T11:47:28.069 INFO:teuthology.orchestra.run.smithi202.stdout:[osd.2] 2022-06-27T11:47:28.070 INFO:teuthology.orchestra.run.smithi202.stdout: key = AQC5mLlidSpeORAA2xOK9X+/73pqHqscXMi8QA== 2022-06-27T11:47:28.070 INFO:teuthology.orchestra.run.smithi202.stdout:[osd.3] 2022-06-27T11:47:28.071 INFO:teuthology.orchestra.run.smithi202.stdout: key = AQC8mLli1hJPOBAAlQWtoh4iMoGIfgL0lsZIUA== 2022-06-27T11:47:28.071 INFO:teuthology.orchestra.run.smithi202.stdout:[mgr.x] 2022-06-27T11:47:28.072 INFO:teuthology.orchestra.run.smithi202.stdout: key = AQCxmLliwmmsLhAACPzoUcvSrsXpZqI7LV1BJg== 2022-06-27T11:47:28.073 INFO:teuthology.orchestra.run.smithi202.stdout:[mds.b] 2022-06-27T11:47:28.074 INFO:teuthology.orchestra.run.smithi202.stdout: key = AQCymLliU1RAChAA5mExsG//0pI2gKn/NJUbhQ== 2022-06-27T11:47:28.074 INFO:teuthology.orchestra.run.smithi202.stdout:[mds.d] 2022-06-27T11:47:28.074 INFO:teuthology.orchestra.run.smithi202.stdout: key = AQCymLlialyPERAACmFDpuHlkMx33uAVDCVydQ== 2022-06-27T11:47:28.074 INFO:teuthology.orchestra.run.smithi202.stdout:[mds.f] 2022-06-27T11:47:28.075 INFO:teuthology.orchestra.run.smithi202.stdout: key = AQCymLlihNfuGBAAWW2aTVEOexWmFcpTLpTn+g== 2022-06-27T11:47:28.075 INFO:teuthology.orchestra.run.smithi202.stdout:[osd.4] 2022-06-27T11:47:28.076 INFO:teuthology.orchestra.run.smithi202.stdout: key = AQDCmLliNBJmOxAA86akZwX001aBSQvMfBDsRA== 2022-06-27T11:47:28.076 INFO:teuthology.orchestra.run.smithi202.stdout:[osd.5] 2022-06-27T11:47:28.076 INFO:teuthology.orchestra.run.smithi202.stdout: key = AQDGmLlictEGABAASLu+di+8T9WymbdfQXVW3g== 2022-06-27T11:47:28.076 INFO:teuthology.orchestra.run.smithi202.stdout:[osd.6] 2022-06-27T11:47:28.076 INFO:teuthology.orchestra.run.smithi202.stdout: key = AQDImLli3GOHOhAA0WxYenACsPhEjhHUxg0I0w== 2022-06-27T11:47:28.077 INFO:teuthology.orchestra.run.smithi202.stdout:[osd.7] 2022-06-27T11:47:28.077 INFO:teuthology.orchestra.run.smithi202.stdout: key = AQDLmLliXY8SOhAAyYoscHtnMLqFmDZohZeXpw== 2022-06-27T11:47:28.077 INFO:teuthology.orchestra.run.smithi202.stdout:[client.0] 2022-06-27T11:47:28.078 INFO:teuthology.orchestra.run.smithi202.stdout: key = AQCymLliXbPwHRAAkX7i3woUHuVYGBVe8ZCUdw== 2022-06-27T11:47:28.079 INFO:teuthology.orchestra.run.smithi195.stdout:[mgr.y] 2022-06-27T11:47:28.079 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQCxmLli0rIxKhAAgLRFXPsuUUqiaDB3sTCq0A== 2022-06-27T11:47:28.079 INFO:teuthology.orchestra.run.smithi195.stdout:[mds.a] 2022-06-27T11:47:28.079 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQCxmLlisqsZMhAAkLUNtHd96zkgYjbcNdU4nA== 2022-06-27T11:47:28.080 INFO:teuthology.orchestra.run.smithi195.stdout:[mds.c] 2022-06-27T11:47:28.080 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQCxmLlic1F0ORAAz9pgZBBfWyQ/f5Km31WT+A== 2022-06-27T11:47:28.080 INFO:teuthology.orchestra.run.smithi195.stdout:[mds.e] 2022-06-27T11:47:28.080 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQCymLliWlEiBRAA/Lm7D1zhUcMhJDHs66Rzbw== 2022-06-27T11:47:28.081 INFO:teuthology.orchestra.run.smithi195.stdout:[osd.0] 2022-06-27T11:47:28.081 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQCzmLliVV22MxAACUSkiQASt1HcBKzwFm2pyw== 2022-06-27T11:47:28.081 INFO:teuthology.orchestra.run.smithi195.stdout:[osd.1] 2022-06-27T11:47:28.081 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQC2mLliz82+MxAA8MfPSBa/IFhizDqgrQKPbQ== 2022-06-27T11:47:28.082 INFO:teuthology.orchestra.run.smithi195.stdout:[osd.2] 2022-06-27T11:47:28.082 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQC5mLlidSpeORAA2xOK9X+/73pqHqscXMi8QA== 2022-06-27T11:47:28.082 INFO:teuthology.orchestra.run.smithi195.stdout:[osd.3] 2022-06-27T11:47:28.083 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQC8mLli1hJPOBAAlQWtoh4iMoGIfgL0lsZIUA== 2022-06-27T11:47:28.083 INFO:teuthology.orchestra.run.smithi195.stdout:[mgr.x] 2022-06-27T11:47:28.083 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQCxmLliwmmsLhAACPzoUcvSrsXpZqI7LV1BJg== 2022-06-27T11:47:28.083 INFO:teuthology.orchestra.run.smithi195.stdout:[mds.b] 2022-06-27T11:47:28.083 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQCymLliU1RAChAA5mExsG//0pI2gKn/NJUbhQ== 2022-06-27T11:47:28.084 INFO:teuthology.orchestra.run.smithi195.stdout:[mds.d] 2022-06-27T11:47:28.084 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQCymLlialyPERAACmFDpuHlkMx33uAVDCVydQ== 2022-06-27T11:47:28.084 INFO:teuthology.orchestra.run.smithi195.stdout:[mds.f] 2022-06-27T11:47:28.084 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQCymLlihNfuGBAAWW2aTVEOexWmFcpTLpTn+g== 2022-06-27T11:47:28.085 INFO:teuthology.orchestra.run.smithi195.stdout:[osd.4] 2022-06-27T11:47:28.085 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQDCmLliNBJmOxAA86akZwX001aBSQvMfBDsRA== 2022-06-27T11:47:28.085 INFO:teuthology.orchestra.run.smithi195.stdout:[osd.5] 2022-06-27T11:47:28.085 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQDGmLlictEGABAASLu+di+8T9WymbdfQXVW3g== 2022-06-27T11:47:28.085 INFO:teuthology.orchestra.run.smithi195.stdout:[osd.6] 2022-06-27T11:47:28.086 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQDImLli3GOHOhAA0WxYenACsPhEjhHUxg0I0w== 2022-06-27T11:47:28.086 INFO:teuthology.orchestra.run.smithi195.stdout:[osd.7] 2022-06-27T11:47:28.086 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQDLmLliXY8SOhAAyYoscHtnMLqFmDZohZeXpw== 2022-06-27T11:47:28.086 INFO:teuthology.orchestra.run.smithi195.stdout:[client.0] 2022-06-27T11:47:28.087 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQCymLliXbPwHRAAkX7i3woUHuVYGBVe8ZCUdw== 2022-06-27T11:47:28.088 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2022-06-27T11:47:28.131 DEBUG:teuthology.orchestra.run.smithi202:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2022-06-27T11:47:28.187 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2022-06-27T11:47:28.191 DEBUG:teuthology.orchestra.run.smithi202:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2022-06-27T11:47:28.254 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2022-06-27T11:47:28.258 DEBUG:teuthology.orchestra.run.smithi202:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2022-06-27T11:47:28.342 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2022-06-27T11:47:28.345 DEBUG:teuthology.orchestra.run.smithi202:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2022-06-27T11:47:28.401 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-06-27T11:47:28.403 DEBUG:teuthology.orchestra.run.smithi202:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-06-27T11:47:28.457 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-06-27T11:47:28.461 DEBUG:teuthology.orchestra.run.smithi202:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-06-27T11:47:28.515 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-06-27T11:47:28.518 DEBUG:teuthology.orchestra.run.smithi202:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-06-27T11:47:28.572 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-06-27T11:47:28.575 DEBUG:teuthology.orchestra.run.smithi202:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-06-27T11:47:28.629 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2022-06-27T11:47:28.632 DEBUG:teuthology.orchestra.run.smithi202:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2022-06-27T11:47:28.687 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2022-06-27T11:47:28.697 DEBUG:teuthology.orchestra.run.smithi202:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2022-06-27T11:47:28.753 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2022-06-27T11:47:28.756 DEBUG:teuthology.orchestra.run.smithi202:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2022-06-27T11:47:28.812 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2022-06-27T11:47:28.815 DEBUG:teuthology.orchestra.run.smithi202:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2022-06-27T11:47:28.872 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-06-27T11:47:28.877 DEBUG:teuthology.orchestra.run.smithi202:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-06-27T11:47:28.932 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-06-27T11:47:28.936 DEBUG:teuthology.orchestra.run.smithi202:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-06-27T11:47:28.991 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-06-27T11:47:28.994 DEBUG:teuthology.orchestra.run.smithi202:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-06-27T11:47:29.048 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-06-27T11:47:29.052 DEBUG:teuthology.orchestra.run.smithi202:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-06-27T11:47:29.107 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-06-27T11:47:29.110 DEBUG:teuthology.orchestra.run.smithi202:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-06-27T11:47:29.167 INFO:tasks.ceph:Running mkfs on mon nodes... 2022-06-27T11:47:29.167 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2022-06-27T11:47:29.195 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2022-06-27T11:47:29.292 DEBUG:teuthology.orchestra.run.smithi195:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2022-06-27T11:47:29.319 DEBUG:teuthology.orchestra.run.smithi202:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2022-06-27T11:47:29.347 DEBUG:teuthology.orchestra.run.smithi202:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2022-06-27T11:47:29.445 DEBUG:teuthology.orchestra.run.smithi202:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2022-06-27T11:47:29.474 DEBUG:teuthology.orchestra.run.smithi202:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2022-06-27T11:47:29.549 DEBUG:teuthology.orchestra.run.smithi202:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2022-06-27T11:47:29.646 DEBUG:teuthology.orchestra.run.smithi202:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2022-06-27T11:47:29.673 DEBUG:teuthology.orchestra.run.smithi195:> rm -- /home/ubuntu/cephtest/ceph.monmap 2022-06-27T11:47:29.676 DEBUG:teuthology.orchestra.run.smithi202:> rm -- /home/ubuntu/cephtest/ceph.monmap 2022-06-27T11:47:29.731 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2022-06-27T11:47:29.732 INFO:tasks.ceph.mon.a:Restarting daemon 2022-06-27T11:47:29.733 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2022-06-27T11:47:29.736 INFO:tasks.ceph.mon.a:Started 2022-06-27T11:47:29.736 INFO:tasks.ceph.mon.b:Restarting daemon 2022-06-27T11:47:29.737 DEBUG:teuthology.orchestra.run.smithi202:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2022-06-27T11:47:29.774 INFO:tasks.ceph.mon.b:Started 2022-06-27T11:47:29.775 INFO:tasks.ceph.mon.c:Restarting daemon 2022-06-27T11:47:29.775 DEBUG:teuthology.orchestra.run.smithi202:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2022-06-27T11:47:29.780 INFO:tasks.ceph.mon.c:Started 2022-06-27T11:47:29.781 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2022-06-27T11:47:29.782 INFO:tasks.ceph.mgr.y:Restarting daemon 2022-06-27T11:47:29.782 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2022-06-27T11:47:29.786 INFO:tasks.ceph.mgr.y:Started 2022-06-27T11:47:29.786 INFO:tasks.ceph.mgr.x:Restarting daemon 2022-06-27T11:47:29.787 DEBUG:teuthology.orchestra.run.smithi202:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2022-06-27T11:47:29.789 INFO:tasks.ceph.mgr.x:Started 2022-06-27T11:47:29.790 INFO:tasks.ceph:Setting crush tunables to default 2022-06-27T11:47:29.790 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd crush tunables default 2022-06-27T11:47:29.916 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:29.913+0000 7fe49c33f700 1 Processor -- start 2022-06-27T11:47:29.917 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:29.913+0000 7fe49c33f700 1 -- start start 2022-06-27T11:47:29.917 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:29.914+0000 7fe49c33f700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fe4940c40d0 0x7fe494157010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:29.917 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:29.914+0000 7fe49c33f700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fe4940c6190 0x7fe494067ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:29.917 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:29.914+0000 7fe49c33f700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe494156b80 con 0x7fe4940c44a0 2022-06-27T11:47:29.918 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:29.914+0000 7fe49c33f700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe49406a040 con 0x7fe4940c6190 2022-06-27T11:47:29.919 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:29.914+0000 7fe49c33f700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe49406a300 con 0x7fe4940c40d0 2022-06-27T11:47:29.919 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:29.914+0000 7fe4998da700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fe4940c6190 0x7fe494067ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:29.919 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:29.914+0000 7fe4998da700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fe4940c6190 0x7fe494067ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:40484/0 (socket says 172.21.15.195:40484) 2022-06-27T11:47:29.919 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:29.914+0000 7fe4998da700 1 -- 172.21.15.195:0/4254052301 learned_addr learned my addr 172.21.15.195:0/4254052301 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:47:29.920 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:29.914+0000 7fe4998da700 1 -- 172.21.15.195:0/4254052301 >> v2:172.21.15.195:3300/0 conn(0x7fe4940c6190 msgr2=0x7fe494067ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2022-06-27T11:47:29.920 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:29.914+0000 7fe4998da700 1 -- 172.21.15.195:0/4254052301 >> v2:172.21.15.195:3300/0 conn(0x7fe4940c6190 msgr2=0x7fe494067ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2022-06-27T11:47:29.921 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:29.914+0000 7fe4998da700 1 --2- 172.21.15.195:0/4254052301 >> v2:172.21.15.195:3300/0 conn(0x7fe4940c6190 0x7fe494067ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2022-06-27T11:47:29.921 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:29.914+0000 7fe49a0db700 1 --2- 172.21.15.195:0/4254052301 >> v2:172.21.15.202:3300/0 conn(0x7fe4940c40d0 0x7fe494157010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:29.921 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:29.914+0000 7fe4998da700 1 --2- 172.21.15.195:0/4254052301 >> v2:172.21.15.195:3300/0 conn(0x7fe4940c6190 0x7fe494067ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2022-06-27T11:47:29.921 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:29.915+0000 7fe49a0db700 1 -- 172.21.15.195:0/4254052301 >> v2:172.21.15.202:3300/0 conn(0x7fe4940c40d0 msgr2=0x7fe494157010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2022-06-27T11:47:29.922 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:29.915+0000 7fe49a0db700 1 -- 172.21.15.195:0/4254052301 >> v2:172.21.15.202:3300/0 conn(0x7fe4940c40d0 msgr2=0x7fe494157010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2022-06-27T11:47:29.922 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:29.915+0000 7fe49a0db700 1 --2- 172.21.15.195:0/4254052301 >> v2:172.21.15.202:3300/0 conn(0x7fe4940c40d0 0x7fe494157010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2022-06-27T11:47:29.922 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:29.915+0000 7fe49a0db700 1 --2- 172.21.15.195:0/4254052301 >> v2:172.21.15.202:3300/0 conn(0x7fe4940c40d0 0x7fe494157010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2022-06-27T11:47:30.115 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:30.115+0000 7fe4998da700 1 --2- 172.21.15.195:0/4254052301 >> v2:172.21.15.195:3300/0 conn(0x7fe4940c6190 0x7fe494067ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:30.116 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:30.115+0000 7fe4998da700 1 -- 172.21.15.195:0/4254052301 >> v2:172.21.15.195:3300/0 conn(0x7fe4940c6190 msgr2=0x7fe494067ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2022-06-27T11:47:30.116 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:30.115+0000 7fe4998da700 1 -- 172.21.15.195:0/4254052301 >> v2:172.21.15.195:3300/0 conn(0x7fe4940c6190 msgr2=0x7fe494067ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2022-06-27T11:47:30.116 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:30.115+0000 7fe4998da700 1 --2- 172.21.15.195:0/4254052301 >> v2:172.21.15.195:3300/0 conn(0x7fe4940c6190 0x7fe494067ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2022-06-27T11:47:30.116 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:30.115+0000 7fe4998da700 1 --2- 172.21.15.195:0/4254052301 >> v2:172.21.15.195:3300/0 conn(0x7fe4940c6190 0x7fe494067ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2022-06-27T11:47:30.117 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:30.116+0000 7fe49a0db700 1 --2- 172.21.15.195:0/4254052301 >> v2:172.21.15.202:3300/0 conn(0x7fe4940c40d0 0x7fe494157010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:30.117 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:30.116+0000 7fe49a0db700 1 -- 172.21.15.195:0/4254052301 >> v2:172.21.15.202:3300/0 conn(0x7fe4940c40d0 msgr2=0x7fe494157010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2022-06-27T11:47:30.117 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:30.116+0000 7fe49a0db700 1 -- 172.21.15.195:0/4254052301 >> v2:172.21.15.202:3300/0 conn(0x7fe4940c40d0 msgr2=0x7fe494157010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2022-06-27T11:47:30.118 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:30.116+0000 7fe49a0db700 1 --2- 172.21.15.195:0/4254052301 >> v2:172.21.15.202:3300/0 conn(0x7fe4940c40d0 0x7fe494157010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2022-06-27T11:47:30.119 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:30.116+0000 7fe49a0db700 1 --2- 172.21.15.195:0/4254052301 >> v2:172.21.15.202:3300/0 conn(0x7fe4940c40d0 0x7fe494157010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2022-06-27T11:47:30.515 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:30.516+0000 7fe4998da700 1 --2- 172.21.15.195:0/4254052301 >> v2:172.21.15.195:3300/0 conn(0x7fe4940c6190 0x7fe494067ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:30.516 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:30.516+0000 7fe4998da700 1 -- 172.21.15.195:0/4254052301 >> v2:172.21.15.195:3300/0 conn(0x7fe4940c6190 msgr2=0x7fe494067ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2022-06-27T11:47:30.516 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:30.516+0000 7fe4998da700 1 -- 172.21.15.195:0/4254052301 >> v2:172.21.15.195:3300/0 conn(0x7fe4940c6190 msgr2=0x7fe494067ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2022-06-27T11:47:30.516 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:30.516+0000 7fe4998da700 1 --2- 172.21.15.195:0/4254052301 >> v2:172.21.15.195:3300/0 conn(0x7fe4940c6190 0x7fe494067ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2022-06-27T11:47:30.517 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:30.516+0000 7fe4998da700 1 --2- 172.21.15.195:0/4254052301 >> v2:172.21.15.195:3300/0 conn(0x7fe4940c6190 0x7fe494067ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2022-06-27T11:47:30.517 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:30.517+0000 7fe49a0db700 1 --2- 172.21.15.195:0/4254052301 >> v2:172.21.15.202:3300/0 conn(0x7fe4940c40d0 0x7fe494157010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:30.518 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:30.517+0000 7fe49a0db700 1 -- 172.21.15.195:0/4254052301 >> v2:172.21.15.202:3300/0 conn(0x7fe4940c40d0 msgr2=0x7fe494157010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2022-06-27T11:47:30.518 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:30.517+0000 7fe49a0db700 1 -- 172.21.15.195:0/4254052301 >> v2:172.21.15.202:3300/0 conn(0x7fe4940c40d0 msgr2=0x7fe494157010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2022-06-27T11:47:30.518 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:30.517+0000 7fe49a0db700 1 --2- 172.21.15.195:0/4254052301 >> v2:172.21.15.202:3300/0 conn(0x7fe4940c40d0 0x7fe494157010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2022-06-27T11:47:30.518 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:30.517+0000 7fe49a0db700 1 --2- 172.21.15.195:0/4254052301 >> v2:172.21.15.202:3300/0 conn(0x7fe4940c40d0 0x7fe494157010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2022-06-27T11:47:31.317 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:31.317+0000 7fe4998da700 1 --2- 172.21.15.195:0/4254052301 >> v2:172.21.15.195:3300/0 conn(0x7fe4940c6190 0x7fe494067ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:31.317 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:31.317+0000 7fe4998da700 1 -- 172.21.15.195:0/4254052301 >> v2:172.21.15.195:3300/0 conn(0x7fe4940c6190 msgr2=0x7fe494067ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2022-06-27T11:47:31.318 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:31.317+0000 7fe4998da700 1 -- 172.21.15.195:0/4254052301 >> v2:172.21.15.195:3300/0 conn(0x7fe4940c6190 msgr2=0x7fe494067ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2022-06-27T11:47:31.318 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:31.317+0000 7fe4998da700 1 --2- 172.21.15.195:0/4254052301 >> v2:172.21.15.195:3300/0 conn(0x7fe4940c6190 0x7fe494067ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2022-06-27T11:47:31.318 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:31.317+0000 7fe4998da700 1 --2- 172.21.15.195:0/4254052301 >> v2:172.21.15.195:3300/0 conn(0x7fe4940c6190 0x7fe494067ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2022-06-27T11:47:31.319 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:31.318+0000 7fe49a0db700 1 --2- 172.21.15.195:0/4254052301 >> v2:172.21.15.202:3300/0 conn(0x7fe4940c40d0 0x7fe494157010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:31.319 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:31.319+0000 7fe49a0db700 1 -- 172.21.15.195:0/4254052301 >> v2:172.21.15.202:3300/0 conn(0x7fe4940c40d0 msgr2=0x7fe494157010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2022-06-27T11:47:31.319 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:31.319+0000 7fe49a0db700 1 -- 172.21.15.195:0/4254052301 >> v2:172.21.15.202:3300/0 conn(0x7fe4940c40d0 msgr2=0x7fe494157010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2022-06-27T11:47:31.320 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:31.319+0000 7fe49a0db700 1 --2- 172.21.15.195:0/4254052301 >> v2:172.21.15.202:3300/0 conn(0x7fe4940c40d0 0x7fe494157010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2022-06-27T11:47:31.320 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:31.319+0000 7fe49a0db700 1 --2- 172.21.15.195:0/4254052301 >> v2:172.21.15.202:3300/0 conn(0x7fe4940c40d0 0x7fe494157010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2022-06-27T11:47:32.916 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:32.914+0000 7fe483fff700 1 -- 172.21.15.195:0/4254052301 >> v2:172.21.15.202:3300/0 conn(0x7fe4940c40d0 msgr2=0x7fe494157010 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2022-06-27T11:47:32.916 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:32.914+0000 7fe483fff700 1 --2- 172.21.15.195:0/4254052301 >> v2:172.21.15.202:3300/0 conn(0x7fe4940c40d0 0x7fe494157010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:32.916 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:32.914+0000 7fe483fff700 1 -- 172.21.15.195:0/4254052301 >> v2:172.21.15.195:3300/0 conn(0x7fe4940c6190 msgr2=0x7fe494067ed0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2022-06-27T11:47:32.918 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:32.914+0000 7fe483fff700 1 --2- 172.21.15.195:0/4254052301 >> v2:172.21.15.195:3300/0 conn(0x7fe4940c6190 0x7fe494067ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:32.918 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:32.914+0000 7fe483fff700 1 -- 172.21.15.195:0/4254052301 >> v1:172.21.15.195:6789/0 conn(0x7fe4940c44a0 legacy=0x7fe49414cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:32.919 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:32.914+0000 7fe483fff700 1 --2- 172.21.15.195:0/4254052301 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe4740071e0 0x7fe474009670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:32.919 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:32.914+0000 7fe483fff700 1 -- 172.21.15.195:0/4254052301 --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe49406a5e0 con 0x7fe4740035c0 2022-06-27T11:47:32.920 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:32.914+0000 7fe483fff700 1 -- 172.21.15.195:0/4254052301 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe49406a040 con 0x7fe4740071e0 2022-06-27T11:47:32.920 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:32.915+0000 7fe4998da700 1 --2- 172.21.15.195:0/4254052301 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe4740071e0 0x7fe474009670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:32.921 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:32.915+0000 7fe4998da700 1 -- 172.21.15.195:0/4254052301 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe4740071e0 msgr2=0x7fe474009670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2022-06-27T11:47:32.921 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:32.915+0000 7fe4998da700 1 -- 172.21.15.195:0/4254052301 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe4740071e0 msgr2=0x7fe474009670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2022-06-27T11:47:32.922 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:32.915+0000 7fe4998da700 1 --2- 172.21.15.195:0/4254052301 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe4740071e0 0x7fe474009670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2022-06-27T11:47:32.923 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:32.915+0000 7fe4998da700 1 --2- 172.21.15.195:0/4254052301 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe4740071e0 0x7fe474009670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2022-06-27T11:47:33.116 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:33.116+0000 7fe4998da700 1 --2- 172.21.15.195:0/4254052301 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe4740071e0 0x7fe474009670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:33.116 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:33.116+0000 7fe4998da700 1 -- 172.21.15.195:0/4254052301 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe4740071e0 msgr2=0x7fe474009670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2022-06-27T11:47:33.116 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:33.116+0000 7fe4998da700 1 -- 172.21.15.195:0/4254052301 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe4740071e0 msgr2=0x7fe474009670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2022-06-27T11:47:33.116 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:33.116+0000 7fe4998da700 1 --2- 172.21.15.195:0/4254052301 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe4740071e0 0x7fe474009670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2022-06-27T11:47:33.117 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:33.116+0000 7fe4998da700 1 --2- 172.21.15.195:0/4254052301 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe4740071e0 0x7fe474009670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2022-06-27T11:47:33.517 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:33.517+0000 7fe4998da700 1 --2- 172.21.15.195:0/4254052301 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe4740071e0 0x7fe474009670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:33.517 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:33.517+0000 7fe4998da700 1 -- 172.21.15.195:0/4254052301 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe4740071e0 msgr2=0x7fe474009670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2022-06-27T11:47:33.517 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:33.517+0000 7fe4998da700 1 -- 172.21.15.195:0/4254052301 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe4740071e0 msgr2=0x7fe474009670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2022-06-27T11:47:33.518 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:33.517+0000 7fe4998da700 1 --2- 172.21.15.195:0/4254052301 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe4740071e0 0x7fe474009670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2022-06-27T11:47:33.518 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:33.517+0000 7fe4998da700 1 --2- 172.21.15.195:0/4254052301 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe4740071e0 0x7fe474009670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2022-06-27T11:47:34.319 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:34.318+0000 7fe4998da700 1 --2- 172.21.15.195:0/4254052301 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe4740071e0 0x7fe474009670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:34.319 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:34.318+0000 7fe4998da700 1 -- 172.21.15.195:0/4254052301 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe4740071e0 msgr2=0x7fe474009670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2022-06-27T11:47:34.319 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:34.318+0000 7fe4998da700 1 -- 172.21.15.195:0/4254052301 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe4740071e0 msgr2=0x7fe474009670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2022-06-27T11:47:34.319 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:34.318+0000 7fe4998da700 1 --2- 172.21.15.195:0/4254052301 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe4740071e0 0x7fe474009670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2022-06-27T11:47:34.320 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:34.318+0000 7fe4998da700 1 --2- 172.21.15.195:0/4254052301 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe4740071e0 0x7fe474009670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2022-06-27T11:47:35.119 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.119+0000 7fe4990d9700 1 -- 172.21.15.195:0/4254052301 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 532163266 0 0) 0x7fe49406a5e0 con 0x7fe4740035c0 2022-06-27T11:47:35.120 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.119+0000 7fe4990d9700 1 -- 172.21.15.195:0/4254052301 --> v1:172.21.15.202:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe47c003430 con 0x7fe4740035c0 2022-06-27T11:47:35.123 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.121+0000 7fe4990d9700 1 -- 172.21.15.195:0/4254052301 <== mon.1 v1:172.21.15.202:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4160983988 0 0) 0x7fe47c003430 con 0x7fe4740035c0 2022-06-27T11:47:35.123 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.121+0000 7fe4990d9700 1 -- 172.21.15.195:0/4254052301 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe4740071e0 msgr2=0x7fe474009670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2022-06-27T11:47:35.124 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.121+0000 7fe4990d9700 1 --2- 172.21.15.195:0/4254052301 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe4740071e0 0x7fe474009670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:35.124 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.121+0000 7fe4990d9700 1 -- 172.21.15.195:0/4254052301 --> v1:172.21.15.202:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe47400ae40 con 0x7fe4740035c0 2022-06-27T11:47:35.124 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.121+0000 7fe4990d9700 1 -- 172.21.15.195:0/4254052301 <== mon.1 v1:172.21.15.202:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7fe48400d1c0 con 0x7fe4740035c0 2022-06-27T11:47:35.124 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.121+0000 7fe4990d9700 1 -- 172.21.15.195:0/4254052301 >> v1:172.21.15.202:6789/0 conn(0x7fe4740035c0 legacy=0x7fe474005a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:35.125 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.122+0000 7fe49a8dc700 1 -- 172.21.15.195:0/4254052301 reap_dead start 2022-06-27T11:47:35.125 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.122+0000 7fe4990d9700 1 --2- 172.21.15.195:0/4254052301 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe47c003920 0x7fe47c003d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:35.125 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.122+0000 7fe4990d9700 1 --2- 172.21.15.195:0/4254052301 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe47c0046d0 0x7fe47c004b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:35.125 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.122+0000 7fe4990d9700 1 --2- 172.21.15.195:0/4254052301 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe47c0051a0 0x7fe47c00b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:35.126 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.122+0000 7fe4990d9700 1 -- 172.21.15.195:0/4254052301 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe49406a040 con 0x7fe47c003920 2022-06-27T11:47:35.126 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.122+0000 7fe4990d9700 1 -- 172.21.15.195:0/4254052301 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe47c00c110 con 0x7fe47c0051a0 2022-06-27T11:47:35.126 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.122+0000 7fe4990d9700 1 -- 172.21.15.195:0/4254052301 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe47c00c2d0 con 0x7fe47c0046d0 2022-06-27T11:47:35.126 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.122+0000 7fe49a0db700 1 --2- 172.21.15.195:0/4254052301 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe47c003920 0x7fe47c003d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:35.127 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.122+0000 7fe4998da700 1 --2- 172.21.15.195:0/4254052301 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe47c0046d0 0x7fe47c004b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:35.127 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.122+0000 7fe49a8dc700 1 --2- 172.21.15.195:0/4254052301 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe47c0051a0 0x7fe47c00b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:35.127 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.122+0000 7fe49a0db700 1 -- 172.21.15.195:0/4254052301 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe47c0046d0 msgr2=0x7fe47c004b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:47:35.127 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.122+0000 7fe49a0db700 1 --2- 172.21.15.195:0/4254052301 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe47c0046d0 0x7fe47c004b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:35.127 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.122+0000 7fe49a0db700 1 -- 172.21.15.195:0/4254052301 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe47c0051a0 msgr2=0x7fe47c00b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:47:35.128 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.122+0000 7fe49a0db700 1 --2- 172.21.15.195:0/4254052301 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe47c0051a0 0x7fe47c00b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:35.128 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.122+0000 7fe49a0db700 1 -- 172.21.15.195:0/4254052301 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe47c003430 con 0x7fe47c003920 2022-06-27T11:47:35.128 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.122+0000 7fe49a0db700 1 --2- 172.21.15.195:0/4254052301 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe47c003920 0x7fe47c003d10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe48400dfc0 tx=0x7fe484008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb9e13ae1fc7d02d server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:35.132 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.124+0000 7fe4990d9700 1 -- 172.21.15.195:0/4254052301 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe484013b80 con 0x7fe47c003920 2022-06-27T11:47:35.132 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.124+0000 7fe4990d9700 1 -- 172.21.15.195:0/4254052301 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe484013d00 con 0x7fe47c003920 2022-06-27T11:47:35.132 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.124+0000 7fe4990d9700 1 -- 172.21.15.195:0/4254052301 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe484004280 con 0x7fe47c003920 2022-06-27T11:47:35.132 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.125+0000 7fe49c33f700 1 -- 172.21.15.195:0/4254052301 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe47c003920 msgr2=0x7fe47c003d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:35.133 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.125+0000 7fe49c33f700 1 --2- 172.21.15.195:0/4254052301 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe47c003920 0x7fe47c003d10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe48400dfc0 tx=0x7fe484008040 comp rx=0 tx=0).stop 2022-06-27T11:47:35.133 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.125+0000 7fe49c33f700 1 -- 172.21.15.195:0/4254052301 shutdown_connections 2022-06-27T11:47:35.133 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.125+0000 7fe49c33f700 1 --2- 172.21.15.195:0/4254052301 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe47c0051a0 0x7fe47c00b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:35.133 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.125+0000 7fe49c33f700 1 --2- 172.21.15.195:0/4254052301 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe47c0046d0 0x7fe47c004b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:35.134 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.125+0000 7fe49c33f700 1 --2- 172.21.15.195:0/4254052301 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe47c003920 0x7fe47c003d10 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:35.134 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.125+0000 7fe49c33f700 1 -- 172.21.15.195:0/4254052301 >> 172.21.15.195:0/4254052301 conn(0x7fe4940ccb80 msgr2=0x7fe4940ccf80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:35.134 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.126+0000 7fe49c33f700 1 -- 172.21.15.195:0/4254052301 shutdown_connections 2022-06-27T11:47:35.134 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.126+0000 7fe49c33f700 1 -- 172.21.15.195:0/4254052301 wait complete. 2022-06-27T11:47:35.135 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.127+0000 7fe49c33f700 1 Processor -- start 2022-06-27T11:47:35.135 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.127+0000 7fe49c33f700 1 -- start start 2022-06-27T11:47:35.135 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.128+0000 7fe49c33f700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe47c003920 0x7fe49416b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:35.135 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.128+0000 7fe49c33f700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe47c0046d0 0x7fe49406e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:35.136 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.128+0000 7fe49c33f700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe47c0051a0 0x7fe49414feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:35.136 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.128+0000 7fe49c33f700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe49406c640 con 0x7fe47c0046d0 2022-06-27T11:47:35.136 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.128+0000 7fe49c33f700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe49406f3c0 con 0x7fe47c003920 2022-06-27T11:47:35.136 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.128+0000 7fe49c33f700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe49406b750 con 0x7fe47c0051a0 2022-06-27T11:47:35.136 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.129+0000 7fe49a0db700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe47c003920 0x7fe49416b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:35.137 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.129+0000 7fe49a0db700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe47c003920 0x7fe49416b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3300/0 says I am v2:172.21.15.195:38680/0 (socket says 172.21.15.195:38680) 2022-06-27T11:47:35.137 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.129+0000 7fe49a8dc700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe47c0051a0 0x7fe49414feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:35.137 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.129+0000 7fe49a0db700 1 -- 172.21.15.195:0/3096547553 learned_addr learned my addr 172.21.15.195:0/3096547553 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:47:35.137 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.129+0000 7fe4998da700 1 --2- 172.21.15.195:0/3096547553 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe47c0046d0 0x7fe49406e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:35.137 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.129+0000 7fe49a0db700 1 -- 172.21.15.195:0/3096547553 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe47c0051a0 msgr2=0x7fe49414feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:35.138 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.129+0000 7fe49a0db700 1 --2- 172.21.15.195:0/3096547553 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe47c0051a0 0x7fe49414feb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:35.138 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.129+0000 7fe49a0db700 1 -- 172.21.15.195:0/3096547553 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe47c0046d0 msgr2=0x7fe49406e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:35.138 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.129+0000 7fe49a0db700 1 --2- 172.21.15.195:0/3096547553 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe47c0046d0 0x7fe49406e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:35.138 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.129+0000 7fe49a0db700 1 -- 172.21.15.195:0/3096547553 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe49406ba30 con 0x7fe47c003920 2022-06-27T11:47:35.139 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.129+0000 7fe49a0db700 1 --2- 172.21.15.195:0/3096547553 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe47c003920 0x7fe49416b910 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe47c00c0e0 tx=0x7fe48401f000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:35.139 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.129+0000 7fe4827fc700 1 -- 172.21.15.195:0/3096547553 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe484026070 con 0x7fe47c003920 2022-06-27T11:47:35.139 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.130+0000 7fe4827fc700 1 -- 172.21.15.195:0/3096547553 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe484013b80 con 0x7fe47c003920 2022-06-27T11:47:35.139 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.130+0000 7fe4827fc700 1 -- 172.21.15.195:0/3096547553 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe48400d3f0 con 0x7fe47c003920 2022-06-27T11:47:35.139 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.130+0000 7fe49c33f700 1 -- 172.21.15.195:0/3096547553 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe49406bd10 con 0x7fe47c003920 2022-06-27T11:47:35.140 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.130+0000 7fe49c33f700 1 -- 172.21.15.195:0/3096547553 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe49406c2d0 con 0x7fe47c003920 2022-06-27T11:47:35.140 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.131+0000 7fe4827fc700 1 -- 172.21.15.195:0/3096547553 <== mon.1 v2:172.21.15.202:3300/0 4 ==== mgrmap(e 1) v1 ==== 517+0+0 (secure 0 0 0) 0x7fe484031070 con 0x7fe47c003920 2022-06-27T11:47:35.140 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.131+0000 7fe4827fc700 1 -- 172.21.15.195:0/3096547553 <== mon.1 v2:172.21.15.202:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fe484000e40 con 0x7fe47c003920 2022-06-27T11:47:35.140 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.131+0000 7fe49c33f700 1 -- 172.21.15.195:0/3096547553 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe49406f3c0 con 0x7fe47c003920 2022-06-27T11:47:35.141 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.135+0000 7fe4827fc700 1 -- 172.21.15.195:0/3096547553 <== mon.1 v2:172.21.15.202:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+70815 (secure 0 0 0) 0x7fe48402f060 con 0x7fe47c003920 2022-06-27T11:47:35.227 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.228+0000 7fe49c33f700 1 -- 172.21.15.195:0/3096547553 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fe4940722c0 con 0x7fe47c003920 2022-06-27T11:47:35.231 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.232+0000 7fe4827fc700 1 -- 172.21.15.195:0/3096547553 <== mon.1 v2:172.21.15.202:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7fe48401a070 con 0x7fe47c003920 2022-06-27T11:47:35.232 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.233+0000 7fe49c33f700 1 -- 172.21.15.195:0/3096547553 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe47c003920 msgr2=0x7fe49416b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:35.233 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.233+0000 7fe49c33f700 1 --2- 172.21.15.195:0/3096547553 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe47c003920 0x7fe49416b910 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe47c00c0e0 tx=0x7fe48401f000 comp rx=0 tx=0).stop 2022-06-27T11:47:35.233 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.233+0000 7fe49c33f700 1 -- 172.21.15.195:0/3096547553 shutdown_connections 2022-06-27T11:47:35.233 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.233+0000 7fe49c33f700 1 --2- 172.21.15.195:0/3096547553 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe47c0051a0 0x7fe49414feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:35.233 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.233+0000 7fe49c33f700 1 --2- 172.21.15.195:0/3096547553 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe47c003920 0x7fe49416b910 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:35.234 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.233+0000 7fe49c33f700 1 --2- 172.21.15.195:0/3096547553 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe47c0046d0 0x7fe49406e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:35.234 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.233+0000 7fe49c33f700 1 -- 172.21.15.195:0/3096547553 >> 172.21.15.195:0/3096547553 conn(0x7fe4940ccb80 msgr2=0x7fe4940c9600 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:35.234 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.233+0000 7fe49c33f700 1 -- 172.21.15.195:0/3096547553 shutdown_connections 2022-06-27T11:47:35.234 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:35.233+0000 7fe49c33f700 1 -- 172.21.15.195:0/3096547553 wait complete. 2022-06-27T11:47:35.235 INFO:teuthology.orchestra.run.smithi195.stderr:adjusted tunables profile to default 2022-06-27T11:47:35.246 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2022-06-27T11:47:35.248 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2022-06-27T11:47:35.248 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2022-06-27T11:47:35.272 INFO:tasks.ceph.mgr.x.smithi202.stderr:2022-06-27T11:47:35.273+0000 7f4976dc5080 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2022-06-27T11:47:35.276 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2022-06-27T11:47:35.277 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2022-06-27T11:47:35.285 INFO:tasks.ceph.mgr.y.smithi195.stderr:2022-06-27T11:47:35.286+0000 7f7d32d42080 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2022-06-27T11:47:35.312 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2022-06-27T11:47:35.312 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2022-06-27T11:47:35.379 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2022-06-27T11:47:35.379 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2022-06-27T11:47:35.423 INFO:tasks.ceph.mgr.x.smithi202.stderr:2022-06-27T11:47:35.424+0000 7f4976dc5080 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2022-06-27T11:47:35.446 DEBUG:teuthology.orchestra.run.smithi202:> set -ex 2022-06-27T11:47:35.447 DEBUG:teuthology.orchestra.run.smithi202:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2022-06-27T11:47:35.451 INFO:tasks.ceph.mgr.y.smithi195.stderr:2022-06-27T11:47:35.452+0000 7f7d32d42080 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2022-06-27T11:47:35.474 DEBUG:teuthology.orchestra.run.smithi202:> set -ex 2022-06-27T11:47:35.475 DEBUG:teuthology.orchestra.run.smithi202:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2022-06-27T11:47:35.541 DEBUG:teuthology.orchestra.run.smithi202:> set -ex 2022-06-27T11:47:35.541 DEBUG:teuthology.orchestra.run.smithi202:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2022-06-27T11:47:35.568 INFO:tasks.ceph.mgr.x.smithi202.stderr:2022-06-27T11:47:35.569+0000 7f4976dc5080 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2022-06-27T11:47:35.594 INFO:tasks.ceph.mgr.y.smithi195.stderr:2022-06-27T11:47:35.594+0000 7f7d32d42080 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2022-06-27T11:47:35.595 DEBUG:teuthology.orchestra.run.smithi202:> set -ex 2022-06-27T11:47:35.596 DEBUG:teuthology.orchestra.run.smithi202:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2022-06-27T11:47:35.636 INFO:tasks.ceph.mgr.x.smithi202.stderr:2022-06-27T11:47:35.637+0000 7f4976dc5080 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2022-06-27T11:47:35.658 INFO:tasks.ceph.mgr.y.smithi195.stderr:2022-06-27T11:47:35.658+0000 7f7d32d42080 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2022-06-27T11:47:35.662 DEBUG:teuthology.orchestra.run.smithi202:> sudo ceph --cluster ceph osd new c64bc51f-d4f3-4bc2-806b-da0dc78d51b0 0 2022-06-27T11:47:35.824 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.825+0000 7f28e5f15700 1 Processor -- start 2022-06-27T11:47:35.824 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.825+0000 7f28e5f15700 1 -- start start 2022-06-27T11:47:35.836 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.825+0000 7f28e5f15700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f28e00c2bf0 0x7f28e00c2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:35.837 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.825+0000 7f28e5f15700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f28e00c61e0 0x7f28e00c48f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:35.837 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.825+0000 7f28e5f15700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28e00bc490 con 0x7f28e00c4e30 2022-06-27T11:47:35.839 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.825+0000 7f28e5f15700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28e00c52b0 con 0x7f28e00c2bf0 2022-06-27T11:47:35.839 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.825+0000 7f28e5f15700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28e006a0e0 con 0x7f28e00c61e0 2022-06-27T11:47:35.840 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.826+0000 7f28df7fe700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f28e00c2bf0 0x7f28e00c2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:35.840 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.826+0000 7f28d6ffd700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f28e00c61e0 0x7f28e00c48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:35.841 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.826+0000 7f28df7fe700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f28e00c2bf0 0x7f28e00c2fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3300/0 says I am v2:172.21.15.202:50084/0 (socket says 172.21.15.202:50084) 2022-06-27T11:47:35.841 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.826+0000 7f28d6ffd700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f28e00c61e0 0x7f28e00c48f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3301/0 says I am v2:172.21.15.202:36940/0 (socket says 172.21.15.202:36940) 2022-06-27T11:47:35.842 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.826+0000 7f28df7fe700 1 -- 172.21.15.202:0/562948084 learned_addr learned my addr 172.21.15.202:0/562948084 (peer_addr_for_me v2:172.21.15.202:0/0) 2022-06-27T11:47:35.842 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.826+0000 7f28df7fe700 1 -- 172.21.15.202:0/562948084 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f28e00c61e0 msgr2=0x7f28e00c48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:47:35.843 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.826+0000 7f28df7fe700 1 --2- 172.21.15.202:0/562948084 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f28e00c61e0 0x7f28e00c48f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:35.843 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.826+0000 7f28df7fe700 1 -- 172.21.15.202:0/562948084 >> v1:172.21.15.195:6789/0 conn(0x7f28e00c4e30 legacy=0x7f28e00c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:35.844 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.826+0000 7f28df7fe700 1 -- 172.21.15.202:0/562948084 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28e00bc6e0 con 0x7f28e00c2bf0 2022-06-27T11:47:35.844 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.826+0000 7f28d6ffd700 1 --2- 172.21.15.202:0/562948084 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f28e00c61e0 0x7f28e00c48f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:47:35.845 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.826+0000 7f28df7fe700 1 --2- 172.21.15.202:0/562948084 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f28e00c2bf0 0x7f28e00c2fc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f28d0006fd0 tx=0x7f28d00090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a7771cba5c2434bb server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:35.845 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.826+0000 7f28deffd700 1 -- 172.21.15.202:0/562948084 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28d0005ad0 con 0x7f28e00c2bf0 2022-06-27T11:47:35.846 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.826+0000 7f28deffd700 1 -- 172.21.15.202:0/562948084 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28d0005c30 con 0x7f28e00c2bf0 2022-06-27T11:47:35.846 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.826+0000 7f28deffd700 1 -- 172.21.15.202:0/562948084 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28d0004030 con 0x7f28e00c2bf0 2022-06-27T11:47:35.846 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.827+0000 7f28e5f15700 1 -- 172.21.15.202:0/562948084 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f28e00c2bf0 msgr2=0x7f28e00c2fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:35.847 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.827+0000 7f28e5f15700 1 --2- 172.21.15.202:0/562948084 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f28e00c2bf0 0x7f28e00c2fc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f28d0006fd0 tx=0x7f28d00090f0 comp rx=0 tx=0).stop 2022-06-27T11:47:35.847 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.827+0000 7f28e5f15700 1 -- 172.21.15.202:0/562948084 shutdown_connections 2022-06-27T11:47:35.847 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.827+0000 7f28e5f15700 1 --2- 172.21.15.202:0/562948084 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f28e00c2bf0 0x7f28e00c2fc0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:35.848 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.827+0000 7f28e5f15700 1 --2- 172.21.15.202:0/562948084 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f28e00c61e0 0x7f28e00c48f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:35.848 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.827+0000 7f28e5f15700 1 -- 172.21.15.202:0/562948084 >> 172.21.15.202:0/562948084 conn(0x7f28e00c9b00 msgr2=0x7f28e00c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:35.848 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.827+0000 7f28e5f15700 1 -- 172.21.15.202:0/562948084 shutdown_connections 2022-06-27T11:47:35.848 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.827+0000 7f28e5f15700 1 -- 172.21.15.202:0/562948084 wait complete. 2022-06-27T11:47:35.849 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.828+0000 7f28e5f15700 1 Processor -- start 2022-06-27T11:47:35.849 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.828+0000 7f28e5f15700 1 -- start start 2022-06-27T11:47:35.849 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.829+0000 7f28e5f15700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f28e00c2bf0 0x7f28e0152080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:35.849 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.829+0000 7f28e5f15700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f28e00c4e30 0x7f28e0152a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:35.849 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.829+0000 7f28e5f15700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f28e00c61e0 0x7f28e01598c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:35.850 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.829+0000 7f28e5f15700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28e01533c0 con 0x7f28e00c4e30 2022-06-27T11:47:35.850 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.829+0000 7f28e5f15700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28e00c2390 con 0x7f28e00c2bf0 2022-06-27T11:47:35.850 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.829+0000 7f28e5f15700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28e0164640 con 0x7f28e00c61e0 2022-06-27T11:47:35.850 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.829+0000 7f28df7fe700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f28e00c2bf0 0x7f28e0152080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:35.851 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.829+0000 7f28df7fe700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f28e00c2bf0 0x7f28e0152080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3300/0 says I am v2:172.21.15.202:50090/0 (socket says 172.21.15.202:50090) 2022-06-27T11:47:35.851 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.829+0000 7f28dffff700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f28e00c61e0 0x7f28e01598c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:35.851 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.829+0000 7f28df7fe700 1 -- 172.21.15.202:0/1742515737 learned_addr learned my addr 172.21.15.202:0/1742515737 (peer_addr_for_me v2:172.21.15.202:0/0) 2022-06-27T11:47:35.851 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.829+0000 7f28d6ffd700 1 --2- 172.21.15.202:0/1742515737 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f28e00c4e30 0x7f28e0152a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:35.851 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.829+0000 7f28df7fe700 1 -- 172.21.15.202:0/1742515737 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f28e00c61e0 msgr2=0x7f28e01598c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:35.852 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.829+0000 7f28df7fe700 1 --2- 172.21.15.202:0/1742515737 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f28e00c61e0 0x7f28e01598c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:35.852 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.829+0000 7f28df7fe700 1 -- 172.21.15.202:0/1742515737 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f28e00c4e30 msgr2=0x7f28e0152a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:35.852 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.829+0000 7f28df7fe700 1 --2- 172.21.15.202:0/1742515737 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f28e00c4e30 0x7f28e0152a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:35.852 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.829+0000 7f28df7fe700 1 -- 172.21.15.202:0/1742515737 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28c80092e0 con 0x7f28e00c2bf0 2022-06-27T11:47:35.853 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.829+0000 7f28dffff700 1 --2- 172.21.15.202:0/1742515737 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f28e00c61e0 0x7f28e01598c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:47:35.853 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.829+0000 7f28df7fe700 1 --2- 172.21.15.202:0/1742515737 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f28e00c2bf0 0x7f28e0152080 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f28e006f810 tx=0x7f28d0015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:35.853 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.830+0000 7f28d7fff700 1 -- 172.21.15.202:0/1742515737 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28d0004560 con 0x7f28e00c2bf0 2022-06-27T11:47:35.853 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.830+0000 7f28d7fff700 1 -- 172.21.15.202:0/1742515737 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28d0003520 con 0x7f28e00c2bf0 2022-06-27T11:47:35.853 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.830+0000 7f28e5f15700 1 -- 172.21.15.202:0/1742515737 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28d0009350 con 0x7f28e00c2bf0 2022-06-27T11:47:35.857 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.830+0000 7f28d7fff700 1 -- 172.21.15.202:0/1742515737 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28d00036d0 con 0x7f28e00c2bf0 2022-06-27T11:47:35.857 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.830+0000 7f28e5f15700 1 -- 172.21.15.202:0/1742515737 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28e00c1e70 con 0x7f28e00c2bf0 2022-06-27T11:47:35.857 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.830+0000 7f28d7fff700 1 -- 172.21.15.202:0/1742515737 <== mon.1 v2:172.21.15.202:3300/0 4 ==== mgrmap(e 1) v1 ==== 517+0+0 (secure 0 0 0) 0x7f28d000f050 con 0x7f28e00c2bf0 2022-06-27T11:47:35.858 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.830+0000 7f28d7fff700 1 -- 172.21.15.202:0/1742515737 <== mon.1 v2:172.21.15.202:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 721+0+0 (secure 0 0 0) 0x7f28d001d790 con 0x7f28e00c2bf0 2022-06-27T11:47:35.858 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.831+0000 7f28e5f15700 1 -- 172.21.15.202:0/1742515737 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28e00722c0 con 0x7f28e00c2bf0 2022-06-27T11:47:35.858 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.833+0000 7f28d7fff700 1 -- 172.21.15.202:0/1742515737 <== mon.1 v2:172.21.15.202:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+70815 (secure 0 0 0) 0x7f28d002c060 con 0x7f28e00c2bf0 2022-06-27T11:47:35.915 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.916+0000 7f28e5f15700 1 -- 172.21.15.202:0/1742515737 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_command({"prefix": "osd new", "uuid": "c64bc51f-d4f3-4bc2-806b-da0dc78d51b0", "id": 0} v 0) v1 -- 0x7f28e0152f90 con 0x7f28e00c2bf0 2022-06-27T11:47:35.918 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.919+0000 7f28d7fff700 1 -- 172.21.15.202:0/1742515737 <== mon.1 v2:172.21.15.202:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c64bc51f-d4f3-4bc2-806b-da0dc78d51b0", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f28d0003b00 con 0x7f28e00c2bf0 2022-06-27T11:47:35.919 INFO:teuthology.orchestra.run.smithi202.stdout:0 2022-06-27T11:47:35.921 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.920+0000 7f28e5f15700 1 -- 172.21.15.202:0/1742515737 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f28e00c2bf0 msgr2=0x7f28e0152080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:35.921 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.920+0000 7f28e5f15700 1 --2- 172.21.15.202:0/1742515737 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f28e00c2bf0 0x7f28e0152080 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f28e006f810 tx=0x7f28d0015000 comp rx=0 tx=0).stop 2022-06-27T11:47:35.921 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.920+0000 7f28e5f15700 1 -- 172.21.15.202:0/1742515737 shutdown_connections 2022-06-27T11:47:35.922 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.920+0000 7f28e5f15700 1 --2- 172.21.15.202:0/1742515737 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f28e00c61e0 0x7f28e01598c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:35.922 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.920+0000 7f28e5f15700 1 --2- 172.21.15.202:0/1742515737 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f28e00c2bf0 0x7f28e0152080 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:35.923 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.920+0000 7f28e5f15700 1 --2- 172.21.15.202:0/1742515737 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f28e00c4e30 0x7f28e0152a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:35.923 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.920+0000 7f28e5f15700 1 -- 172.21.15.202:0/1742515737 >> 172.21.15.202:0/1742515737 conn(0x7f28e00c9b00 msgr2=0x7f28e014da30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:35.924 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.920+0000 7f28e5f15700 1 -- 172.21.15.202:0/1742515737 shutdown_connections 2022-06-27T11:47:35.924 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:35.920+0000 7f28e5f15700 1 -- 172.21.15.202:0/1742515737 wait complete. 2022-06-27T11:47:35.931 DEBUG:teuthology.orchestra.run.smithi202:> sudo ceph --cluster ceph osd new bea71cdf-aa98-493f-9207-66d886a3e30e 1 2022-06-27T11:47:36.070 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.051+0000 7fa50bcfa700 1 Processor -- start 2022-06-27T11:47:36.071 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.052+0000 7fa50bcfa700 1 -- start start 2022-06-27T11:47:36.072 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.052+0000 7fa50bcfa700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fa5040c4160 0x7fa504156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:36.072 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.052+0000 7fa50bcfa700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5040c1a20 con 0x7fa50406f640 2022-06-27T11:47:36.072 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.052+0000 7fa50bcfa700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5040c1130 con 0x7fa5040c39e0 2022-06-27T11:47:36.073 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.052+0000 7fa50bcfa700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa50406ea90 con 0x7fa5040c4160 2022-06-27T11:47:36.073 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.052+0000 7fa509295700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fa5040c4160 0x7fa504156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:36.074 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.052+0000 7fa509295700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fa5040c4160 0x7fa504156ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3300/0 says I am v2:172.21.15.202:50098/0 (socket says 172.21.15.202:50098) 2022-06-27T11:47:36.074 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.053+0000 7fa509295700 1 -- 172.21.15.202:0/3557340948 learned_addr learned my addr 172.21.15.202:0/3557340948 (peer_addr_for_me v2:172.21.15.202:0/0) 2022-06-27T11:47:36.075 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.053+0000 7fa509295700 1 -- 172.21.15.202:0/3557340948 >> v1:172.21.15.202:6789/0 conn(0x7fa5040c39e0 legacy=0x7fa5041535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:36.075 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.053+0000 7fa508a94700 1 -- 172.21.15.202:0/3557340948 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2895122065 0 0) 0x7fa5040c1a20 con 0x7fa50406f640 2022-06-27T11:47:36.076 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.053+0000 7fa509295700 1 -- 172.21.15.202:0/3557340948 >> v1:172.21.15.195:6789/0 conn(0x7fa50406f640 legacy=0x7fa5040c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:36.076 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.053+0000 7fa509295700 1 -- 172.21.15.202:0/3557340948 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa50406ed70 con 0x7fa5040c4160 2022-06-27T11:47:36.077 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.053+0000 7fa509295700 1 --2- 172.21.15.202:0/3557340948 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa5040c4160 0x7fa504156ed0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa4f4009860 tx=0x7fa4f4009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=99818ee1ab935c55 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:36.077 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.053+0000 7fa508a94700 1 -- 172.21.15.202:0/3557340948 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4f400b710 con 0x7fa5040c4160 2022-06-27T11:47:36.077 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.053+0000 7fa508a94700 1 -- 172.21.15.202:0/3557340948 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4f400b870 con 0x7fa5040c4160 2022-06-27T11:47:36.078 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.053+0000 7fa508a94700 1 -- 172.21.15.202:0/3557340948 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4f400bc30 con 0x7fa5040c4160 2022-06-27T11:47:36.078 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.054+0000 7fa50bcfa700 1 -- 172.21.15.202:0/3557340948 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa5040c4160 msgr2=0x7fa504156ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:36.079 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.054+0000 7fa50bcfa700 1 --2- 172.21.15.202:0/3557340948 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa5040c4160 0x7fa504156ed0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa4f4009860 tx=0x7fa4f4009120 comp rx=0 tx=0).stop 2022-06-27T11:47:36.079 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.054+0000 7fa50bcfa700 1 -- 172.21.15.202:0/3557340948 shutdown_connections 2022-06-27T11:47:36.080 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.054+0000 7fa50bcfa700 1 --2- 172.21.15.202:0/3557340948 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa5040c4160 0x7fa504156ed0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:36.080 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.054+0000 7fa50bcfa700 1 -- 172.21.15.202:0/3557340948 >> 172.21.15.202:0/3557340948 conn(0x7fa5040c6350 msgr2=0x7fa5040c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:36.081 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.054+0000 7fa50bcfa700 1 -- 172.21.15.202:0/3557340948 shutdown_connections 2022-06-27T11:47:36.081 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.054+0000 7fa50bcfa700 1 -- 172.21.15.202:0/3557340948 wait complete. 2022-06-27T11:47:36.081 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.055+0000 7fa50bcfa700 1 Processor -- start 2022-06-27T11:47:36.082 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.055+0000 7fa50bcfa700 1 -- start start 2022-06-27T11:47:36.082 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.056+0000 7fa50bcfa700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa50406f640 0x7fa50416b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:36.083 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.056+0000 7fa50bcfa700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa5040c39e0 0x7fa50414f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:36.083 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.056+0000 7fa50bcfa700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa5040c4160 0x7fa50406a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:36.084 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.056+0000 7fa50bcfa700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa50416ac30 con 0x7fa5040c39e0 2022-06-27T11:47:36.084 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.056+0000 7fa50bcfa700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa504068210 con 0x7fa50406f640 2022-06-27T11:47:36.085 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.056+0000 7fa50bcfa700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5040684f0 con 0x7fa5040c4160 2022-06-27T11:47:36.085 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.056+0000 7fa509a96700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa50406f640 0x7fa50416b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:36.086 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.056+0000 7fa509a96700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa50406f640 0x7fa50416b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3300/0 says I am v2:172.21.15.202:50102/0 (socket says 172.21.15.202:50102) 2022-06-27T11:47:36.086 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.056+0000 7fa50a297700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa5040c4160 0x7fa50406a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:36.086 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.056+0000 7fa509a96700 1 -- 172.21.15.202:0/2418672972 learned_addr learned my addr 172.21.15.202:0/2418672972 (peer_addr_for_me v2:172.21.15.202:0/0) 2022-06-27T11:47:36.087 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.056+0000 7fa509295700 1 --2- 172.21.15.202:0/2418672972 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa5040c39e0 0x7fa50414f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:36.087 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.056+0000 7fa509a96700 1 -- 172.21.15.202:0/2418672972 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa5040c4160 msgr2=0x7fa50406a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:36.088 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.056+0000 7fa509a96700 1 --2- 172.21.15.202:0/2418672972 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa5040c4160 0x7fa50406a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:36.088 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.056+0000 7fa509a96700 1 -- 172.21.15.202:0/2418672972 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa5040c39e0 msgr2=0x7fa50414f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:36.089 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.056+0000 7fa509a96700 1 --2- 172.21.15.202:0/2418672972 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa5040c39e0 0x7fa50414f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:36.089 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.056+0000 7fa509a96700 1 -- 172.21.15.202:0/2418672972 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4f40093a0 con 0x7fa50406f640 2022-06-27T11:47:36.090 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.056+0000 7fa509a96700 1 --2- 172.21.15.202:0/2418672972 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa50406f640 0x7fa50416b0c0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa50000c590 tx=0x7fa500007cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:36.090 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.056+0000 7fa4f1ffb700 1 -- 172.21.15.202:0/2418672972 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5000052e0 con 0x7fa50406f640 2022-06-27T11:47:36.091 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.056+0000 7fa4f1ffb700 1 -- 172.21.15.202:0/2418672972 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa500003500 con 0x7fa50406f640 2022-06-27T11:47:36.091 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.056+0000 7fa50bcfa700 1 -- 172.21.15.202:0/2418672972 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5040687d0 con 0x7fa50406f640 2022-06-27T11:47:36.091 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.056+0000 7fa4f1ffb700 1 -- 172.21.15.202:0/2418672972 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5000038c0 con 0x7fa50406f640 2022-06-27T11:47:36.092 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.057+0000 7fa4f1ffb700 1 -- 172.21.15.202:0/2418672972 <== mon.1 v2:172.21.15.202:3300/0 4 ==== mgrmap(e 1) v1 ==== 517+0+0 (secure 0 0 0) 0x7fa500019030 con 0x7fa50406f640 2022-06-27T11:47:36.092 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.057+0000 7fa50bcfa700 1 -- 172.21.15.202:0/2418672972 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa50416a890 con 0x7fa50406f640 2022-06-27T11:47:36.093 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.057+0000 7fa4f1ffb700 1 -- 172.21.15.202:0/2418672972 <== mon.1 v2:172.21.15.202:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 836+0+0 (secure 0 0 0) 0x7fa500012cc0 con 0x7fa50406f640 2022-06-27T11:47:36.093 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.057+0000 7fa50bcfa700 1 -- 172.21.15.202:0/2418672972 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5040722c0 con 0x7fa50406f640 2022-06-27T11:47:36.094 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.060+0000 7fa4f1ffb700 1 -- 172.21.15.202:0/2418672972 <== mon.1 v2:172.21.15.202:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+70815 (secure 0 0 0) 0x7fa500017070 con 0x7fa50406f640 2022-06-27T11:47:36.141 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.142+0000 7fa50bcfa700 1 -- 172.21.15.202:0/2418672972 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_command({"prefix": "osd new", "uuid": "bea71cdf-aa98-493f-9207-66d886a3e30e", "id": 1} v 0) v1 -- 0x7fa50406f240 con 0x7fa50406f640 2022-06-27T11:47:36.144 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.146+0000 7fa4f1ffb700 1 -- 172.21.15.202:0/2418672972 <== mon.1 v2:172.21.15.202:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "bea71cdf-aa98-493f-9207-66d886a3e30e", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7fa500012400 con 0x7fa50406f640 2022-06-27T11:47:36.145 INFO:teuthology.orchestra.run.smithi202.stdout:1 2022-06-27T11:47:36.147 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.147+0000 7fa50bcfa700 1 -- 172.21.15.202:0/2418672972 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa50406f640 msgr2=0x7fa50416b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:36.147 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.147+0000 7fa50bcfa700 1 --2- 172.21.15.202:0/2418672972 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa50406f640 0x7fa50416b0c0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa50000c590 tx=0x7fa500007cb0 comp rx=0 tx=0).stop 2022-06-27T11:47:36.147 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.147+0000 7fa50bcfa700 1 -- 172.21.15.202:0/2418672972 shutdown_connections 2022-06-27T11:47:36.147 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.147+0000 7fa50bcfa700 1 --2- 172.21.15.202:0/2418672972 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa5040c4160 0x7fa50406a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:36.148 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.147+0000 7fa50bcfa700 1 --2- 172.21.15.202:0/2418672972 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa50406f640 0x7fa50416b0c0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:36.148 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.147+0000 7fa50bcfa700 1 --2- 172.21.15.202:0/2418672972 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa5040c39e0 0x7fa50414f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:36.148 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.147+0000 7fa50bcfa700 1 -- 172.21.15.202:0/2418672972 >> 172.21.15.202:0/2418672972 conn(0x7fa5040c6350 msgr2=0x7fa504153370 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:36.148 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.147+0000 7fa50bcfa700 1 -- 172.21.15.202:0/2418672972 shutdown_connections 2022-06-27T11:47:36.148 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.147+0000 7fa50bcfa700 1 -- 172.21.15.202:0/2418672972 wait complete. 2022-06-27T11:47:36.158 DEBUG:teuthology.orchestra.run.smithi202:> sudo ceph --cluster ceph osd new bf6c87a0-a8ea-497a-9ce1-721285888065 2 2022-06-27T11:47:36.240 INFO:tasks.ceph.mgr.y.smithi195.stderr:2022-06-27T11:47:36.241+0000 7f7d32d42080 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2022-06-27T11:47:36.262 INFO:tasks.ceph.mgr.x.smithi202.stderr:2022-06-27T11:47:36.261+0000 7f4976dc5080 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2022-06-27T11:47:36.292 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.283+0000 7f74e9cd1700 1 Processor -- start 2022-06-27T11:47:36.292 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.283+0000 7f74e9cd1700 1 -- start start 2022-06-27T11:47:36.293 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.284+0000 7f74e9cd1700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f74e4150500 0x7f74e4063af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:36.293 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.284+0000 7f74e9cd1700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f74e41508d0 0x7f74e40bbf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:36.293 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.284+0000 7f74e9cd1700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74e4064030 con 0x7f74e406a450 2022-06-27T11:47:36.294 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.284+0000 7f74e9cd1700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74e40c4160 con 0x7f74e41508d0 2022-06-27T11:47:36.294 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.284+0000 7f74e9cd1700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74e40c4440 con 0x7f74e4150500 2022-06-27T11:47:36.294 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.284+0000 7f74e37fe700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f74e4150500 0x7f74e4063af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:36.295 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.284+0000 7f74e37fe700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f74e4150500 0x7f74e4063af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3301/0 says I am v2:172.21.15.202:36962/0 (socket says 172.21.15.202:36962) 2022-06-27T11:47:36.295 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.284+0000 7f74e2ffd700 1 --1- >> v1:172.21.15.202:6789/0 conn(0x7f74e406a450 0x7f74e40c48d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.202:6789/0 says I am v1:172.21.15.202:47476/0 (socket says 172.21.15.202:47476) 2022-06-27T11:47:36.295 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.284+0000 7f74e37fe700 1 -- 172.21.15.202:0/1321852374 learned_addr learned my addr 172.21.15.202:0/1321852374 (peer_addr_for_me v2:172.21.15.202:0/0) 2022-06-27T11:47:36.295 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.284+0000 7f74e3fff700 1 --2- 172.21.15.202:0/1321852374 >> v2:172.21.15.195:3300/0 conn(0x7f74e41508d0 0x7f74e40bbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:36.296 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.285+0000 7f74e27fc700 1 -- 172.21.15.202:0/1321852374 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2780234371 0 0) 0x7f74d8001110 con 0x7f74e406a450 2022-06-27T11:47:36.296 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.285+0000 7f74e27fc700 1 -- 172.21.15.202:0/1321852374 --> v1:172.21.15.202:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74d0003430 con 0x7f74e406a450 2022-06-27T11:47:36.296 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.285+0000 7f74e37fe700 1 -- 172.21.15.202:0/1321852374 >> v2:172.21.15.195:3300/0 conn(0x7f74e41508d0 msgr2=0x7f74e40bbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:47:36.296 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.285+0000 7f74e37fe700 1 --2- 172.21.15.202:0/1321852374 >> v2:172.21.15.195:3300/0 conn(0x7f74e41508d0 0x7f74e40bbf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:36.297 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.285+0000 7f74e37fe700 1 -- 172.21.15.202:0/1321852374 >> v1:172.21.15.202:6789/0 conn(0x7f74e406a450 legacy=0x7f74e40c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:36.297 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.285+0000 7f74e37fe700 1 -- 172.21.15.202:0/1321852374 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74e415a800 con 0x7f74e4150500 2022-06-27T11:47:36.297 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.285+0000 7f74e3fff700 1 --2- 172.21.15.202:0/1321852374 >> v2:172.21.15.195:3300/0 conn(0x7f74e41508d0 0x7f74e40bbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:47:36.298 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.285+0000 7f74e37fe700 1 --2- 172.21.15.202:0/1321852374 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f74e4150500 0x7f74e4063af0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f74d4006fd0 tx=0x7f74d40090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90cd60f7f3ff4fc3 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:36.298 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.285+0000 7f74e27fc700 1 -- 172.21.15.202:0/1321852374 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74d4005ad0 con 0x7f74e4150500 2022-06-27T11:47:36.298 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.285+0000 7f74e27fc700 1 -- 172.21.15.202:0/1321852374 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74d4005c50 con 0x7f74e4150500 2022-06-27T11:47:36.299 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.285+0000 7f74e27fc700 1 -- 172.21.15.202:0/1321852374 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74d4004030 con 0x7f74e4150500 2022-06-27T11:47:36.299 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.286+0000 7f74e9cd1700 1 -- 172.21.15.202:0/1321852374 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f74e4150500 msgr2=0x7f74e4063af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:36.300 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.286+0000 7f74e9cd1700 1 --2- 172.21.15.202:0/1321852374 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f74e4150500 0x7f74e4063af0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f74d4006fd0 tx=0x7f74d40090f0 comp rx=0 tx=0).stop 2022-06-27T11:47:36.300 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.286+0000 7f74e9cd1700 1 -- 172.21.15.202:0/1321852374 shutdown_connections 2022-06-27T11:47:36.300 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.286+0000 7f74e9cd1700 1 --2- 172.21.15.202:0/1321852374 >> v2:172.21.15.195:3300/0 conn(0x7f74e41508d0 0x7f74e40bbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:36.300 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.286+0000 7f74e9cd1700 1 --2- 172.21.15.202:0/1321852374 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f74e4150500 0x7f74e4063af0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:36.301 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.286+0000 7f74e9cd1700 1 -- 172.21.15.202:0/1321852374 >> 172.21.15.202:0/1321852374 conn(0x7f74e40c9b00 msgr2=0x7f74e40c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:36.301 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.286+0000 7f74e9cd1700 1 -- 172.21.15.202:0/1321852374 shutdown_connections 2022-06-27T11:47:36.301 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.286+0000 7f74e9cd1700 1 -- 172.21.15.202:0/1321852374 wait complete. 2022-06-27T11:47:36.301 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.287+0000 7f74e9cd1700 1 Processor -- start 2022-06-27T11:47:36.302 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.287+0000 7f74e9cd1700 1 -- start start 2022-06-27T11:47:36.302 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.288+0000 7f74e9cd1700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f74e406a450 0x7f74e4166280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:36.302 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.288+0000 7f74e9cd1700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f74e4150500 0x7f74e414bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:36.303 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.288+0000 7f74e9cd1700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f74e41508d0 0x7f74e40be0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:36.303 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.288+0000 7f74e9cd1700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74e40c09b0 con 0x7f74e406a450 2022-06-27T11:47:36.303 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.288+0000 7f74e9cd1700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74e4158d60 con 0x7f74e41508d0 2022-06-27T11:47:36.303 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.288+0000 7f74e9cd1700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74e41502b0 con 0x7f74e4150500 2022-06-27T11:47:36.304 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.288+0000 7f74e2ffd700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f74e4150500 0x7f74e414bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:36.304 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.288+0000 7f74e2ffd700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f74e4150500 0x7f74e414bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3301/0 says I am v2:172.21.15.202:36970/0 (socket says 172.21.15.202:36970) 2022-06-27T11:47:36.304 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.288+0000 7f74e2ffd700 1 -- 172.21.15.202:0/943451674 learned_addr learned my addr 172.21.15.202:0/943451674 (peer_addr_for_me v2:172.21.15.202:0/0) 2022-06-27T11:47:36.305 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.288+0000 7f74e3fff700 1 --2- 172.21.15.202:0/943451674 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f74e41508d0 0x7f74e40be0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:36.305 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.289+0000 7f74e37fe700 1 --2- 172.21.15.202:0/943451674 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f74e406a450 0x7f74e4166280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:36.305 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.289+0000 7f74e2ffd700 1 -- 172.21.15.202:0/943451674 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f74e41508d0 msgr2=0x7f74e40be0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:36.306 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.289+0000 7f74e2ffd700 1 --2- 172.21.15.202:0/943451674 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f74e41508d0 0x7f74e40be0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:36.306 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.289+0000 7f74e2ffd700 1 -- 172.21.15.202:0/943451674 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f74e406a450 msgr2=0x7f74e4166280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:36.306 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.289+0000 7f74e2ffd700 1 --2- 172.21.15.202:0/943451674 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f74e406a450 0x7f74e4166280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:36.309 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.289+0000 7f74e2ffd700 1 -- 172.21.15.202:0/943451674 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74dc009a20 con 0x7f74e4150500 2022-06-27T11:47:36.309 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.289+0000 7f74e3fff700 1 --2- 172.21.15.202:0/943451674 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f74e41508d0 0x7f74e40be0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:47:36.310 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.289+0000 7f74e37fe700 1 --2- 172.21.15.202:0/943451674 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f74e406a450 0x7f74e4166280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:47:36.310 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.289+0000 7f74e2ffd700 1 --2- 172.21.15.202:0/943451674 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f74e4150500 0x7f74e414bc20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f74d800c4f0 tx=0x7f74d8007ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:36.311 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.289+0000 7f74cb7fe700 1 -- 172.21.15.202:0/943451674 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74d80072e0 con 0x7f74e4150500 2022-06-27T11:47:36.311 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.289+0000 7f74cb7fe700 1 -- 172.21.15.202:0/943451674 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74d8007460 con 0x7f74e4150500 2022-06-27T11:47:36.311 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.289+0000 7f74cb7fe700 1 -- 172.21.15.202:0/943451674 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74d8004940 con 0x7f74e4150500 2022-06-27T11:47:36.312 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.289+0000 7f74e9cd1700 1 -- 172.21.15.202:0/943451674 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74d4009350 con 0x7f74e4150500 2022-06-27T11:47:36.312 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.289+0000 7f74e9cd1700 1 -- 172.21.15.202:0/943451674 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74e40c06d0 con 0x7f74e4150500 2022-06-27T11:47:36.313 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.289+0000 7f74cb7fe700 1 -- 172.21.15.202:0/943451674 <== mon.2 v2:172.21.15.202:3301/0 4 ==== mgrmap(e 1) v1 ==== 517+0+0 (secure 0 0 0) 0x7f74d8010070 con 0x7f74e4150500 2022-06-27T11:47:36.313 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.289+0000 7f74cb7fe700 1 -- 172.21.15.202:0/943451674 <== mon.2 v2:172.21.15.202:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 951+0+0 (secure 0 0 0) 0x7f74d8013400 con 0x7f74e4150500 2022-06-27T11:47:36.313 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.290+0000 7f74e9cd1700 1 -- 172.21.15.202:0/943451674 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74e40722c0 con 0x7f74e4150500 2022-06-27T11:47:36.314 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.292+0000 7f74cb7fe700 1 -- 172.21.15.202:0/943451674 <== mon.2 v2:172.21.15.202:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+70815 (secure 0 0 0) 0x7f74d8021060 con 0x7f74e4150500 2022-06-27T11:47:36.315 INFO:tasks.ceph.mgr.y.smithi195.stderr:2022-06-27T11:47:36.309+0000 7f7d32d42080 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2022-06-27T11:47:36.333 INFO:tasks.ceph.mgr.x.smithi202.stderr:2022-06-27T11:47:36.334+0000 7f4976dc5080 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2022-06-27T11:47:36.375 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.375+0000 7f74e9cd1700 1 -- 172.21.15.202:0/943451674 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_command({"prefix": "osd new", "uuid": "bf6c87a0-a8ea-497a-9ce1-721285888065", "id": 2} v 0) v1 -- 0x7f74e40ba8e0 con 0x7f74e4150500 2022-06-27T11:47:36.378 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.379+0000 7f74cb7fe700 1 -- 172.21.15.202:0/943451674 <== mon.2 v2:172.21.15.202:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "bf6c87a0-a8ea-497a-9ce1-721285888065", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f74e4064030 con 0x7f74e4150500 2022-06-27T11:47:36.378 INFO:teuthology.orchestra.run.smithi202.stdout:2 2022-06-27T11:47:36.379 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.380+0000 7f74e9cd1700 1 -- 172.21.15.202:0/943451674 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f74e4150500 msgr2=0x7f74e414bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:36.380 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.380+0000 7f74e9cd1700 1 --2- 172.21.15.202:0/943451674 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f74e4150500 0x7f74e414bc20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f74d800c4f0 tx=0x7f74d8007ab0 comp rx=0 tx=0).stop 2022-06-27T11:47:36.380 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.380+0000 7f74e9cd1700 1 -- 172.21.15.202:0/943451674 shutdown_connections 2022-06-27T11:47:36.380 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.380+0000 7f74e9cd1700 1 --2- 172.21.15.202:0/943451674 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f74e406a450 0x7f74e4166280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:36.380 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.380+0000 7f74e9cd1700 1 --2- 172.21.15.202:0/943451674 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f74e41508d0 0x7f74e40be0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:36.381 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.380+0000 7f74e9cd1700 1 --2- 172.21.15.202:0/943451674 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f74e4150500 0x7f74e414bc20 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:36.381 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.380+0000 7f74e9cd1700 1 -- 172.21.15.202:0/943451674 >> 172.21.15.202:0/943451674 conn(0x7f74e40c9b00 msgr2=0x7f74e40c6e40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:36.381 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.380+0000 7f74e9cd1700 1 -- 172.21.15.202:0/943451674 shutdown_connections 2022-06-27T11:47:36.381 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.380+0000 7f74e9cd1700 1 -- 172.21.15.202:0/943451674 wait complete. 2022-06-27T11:47:36.391 DEBUG:teuthology.orchestra.run.smithi202:> sudo ceph --cluster ceph osd new e8aa47ce-af8c-47e7-9654-870cd53229f4 3 2022-06-27T11:47:36.511 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.512+0000 7fe2af0bf700 1 Processor -- start 2022-06-27T11:47:36.511 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.512+0000 7fe2af0bf700 1 -- start start 2022-06-27T11:47:36.514 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.514+0000 7fe2af0bf700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe2a80c61e0 0x7fe2a80c2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:36.514 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.514+0000 7fe2af0bf700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2a80bc490 con 0x7fe2a80c2bf0 2022-06-27T11:47:36.515 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.514+0000 7fe2af0bf700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2a806a0e0 con 0x7fe2a80c5000 2022-06-27T11:47:36.515 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.514+0000 7fe2af0bf700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a80bc6e0 con 0x7fe2a80c61e0 2022-06-27T11:47:36.515 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.514+0000 7fe2a7fff700 1 --1- >> v1:172.21.15.202:6789/0 conn(0x7fe2a80c5000 0x7fe2a80c3900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.202:6789/0 says I am v1:172.21.15.202:47488/0 (socket says 172.21.15.202:47488) 2022-06-27T11:47:36.515 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.514+0000 7fe2a7fff700 1 -- 172.21.15.202:0/2654566287 learned_addr learned my addr 172.21.15.202:0/2654566287 (peer_addr_for_me v1:172.21.15.202:0/0) 2022-06-27T11:47:36.516 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.514+0000 7fe2ad65c700 1 --2- 172.21.15.202:0/2654566287 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe2a80c61e0 0x7fe2a80c2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:36.516 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.514+0000 7fe2a77fe700 1 -- 172.21.15.202:0/2654566287 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 335657212 0 0) 0x7fe2a806a0e0 con 0x7fe2a80c5000 2022-06-27T11:47:36.516 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.514+0000 7fe2a77fe700 1 -- 172.21.15.202:0/2654566287 --> v1:172.21.15.202:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe28c003430 con 0x7fe2a80c5000 2022-06-27T11:47:36.516 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.514+0000 7fe2ad65c700 1 -- 172.21.15.202:0/2654566287 >> v1:172.21.15.202:6789/0 conn(0x7fe2a80c5000 legacy=0x7fe2a80c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:36.516 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.514+0000 7fe2ad65c700 1 -- 172.21.15.202:0/2654566287 >> v1:172.21.15.195:6789/0 conn(0x7fe2a80c2bf0 legacy=0x7fe2a80c48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:36.517 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.514+0000 7fe2ad65c700 1 -- 172.21.15.202:0/2654566287 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2a8158220 con 0x7fe2a80c61e0 2022-06-27T11:47:36.517 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.514+0000 7fe2ad65c700 1 --2- 172.21.15.202:0/2654566287 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe2a80c61e0 0x7fe2a80c2fc0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe2a0009e90 tx=0x7fe2a0009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20877522b8192823 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:36.517 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.514+0000 7fe2a77fe700 1 -- 172.21.15.202:0/2654566287 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2a000e070 con 0x7fe2a80c61e0 2022-06-27T11:47:36.517 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.514+0000 7fe2a77fe700 1 -- 172.21.15.202:0/2654566287 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2a000bc30 con 0x7fe2a80c61e0 2022-06-27T11:47:36.518 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.514+0000 7fe2a77fe700 1 -- 172.21.15.202:0/2654566287 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2a0012430 con 0x7fe2a80c61e0 2022-06-27T11:47:36.518 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.515+0000 7fe2af0bf700 1 -- 172.21.15.202:0/2654566287 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe2a80c61e0 msgr2=0x7fe2a80c2fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:36.518 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.515+0000 7fe2af0bf700 1 --2- 172.21.15.202:0/2654566287 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe2a80c61e0 0x7fe2a80c2fc0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe2a0009e90 tx=0x7fe2a0009780 comp rx=0 tx=0).stop 2022-06-27T11:47:36.518 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.515+0000 7fe2af0bf700 1 -- 172.21.15.202:0/2654566287 shutdown_connections 2022-06-27T11:47:36.518 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.515+0000 7fe2af0bf700 1 --2- 172.21.15.202:0/2654566287 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe2a80c61e0 0x7fe2a80c2fc0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:36.519 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.515+0000 7fe2af0bf700 1 -- 172.21.15.202:0/2654566287 >> 172.21.15.202:0/2654566287 conn(0x7fe2a80c9b00 msgr2=0x7fe2a80c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:36.519 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.515+0000 7fe2af0bf700 1 -- 172.21.15.202:0/2654566287 shutdown_connections 2022-06-27T11:47:36.519 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.515+0000 7fe2af0bf700 1 -- 172.21.15.202:0/2654566287 wait complete. 2022-06-27T11:47:36.522 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.516+0000 7fe2af0bf700 1 Processor -- start 2022-06-27T11:47:36.522 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.516+0000 7fe2af0bf700 1 -- start start 2022-06-27T11:47:36.523 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.517+0000 7fe2af0bf700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe2a80c2bf0 0x7fe2a80bf6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:36.523 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.517+0000 7fe2af0bf700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe2a80c5000 0x7fe2a81697c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:36.523 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.517+0000 7fe2af0bf700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe2a80c61e0 0x7fe2a8068200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:36.523 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.517+0000 7fe2af0bf700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a80bc100 con 0x7fe2a80c2bf0 2022-06-27T11:47:36.523 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.517+0000 7fe2af0bf700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a80bb210 con 0x7fe2a80c5000 2022-06-27T11:47:36.524 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.517+0000 7fe2af0bf700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a80bb4f0 con 0x7fe2a80c61e0 2022-06-27T11:47:36.524 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.517+0000 7fe2a7fff700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe2a80c5000 0x7fe2a81697c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:36.524 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.517+0000 7fe2ad65c700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe2a80c61e0 0x7fe2a8068200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:36.524 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.517+0000 7fe2a7fff700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe2a80c5000 0x7fe2a81697c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3300/0 says I am v2:172.21.15.202:50128/0 (socket says 172.21.15.202:50128) 2022-06-27T11:47:36.525 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.517+0000 7fe2a7fff700 1 -- 172.21.15.202:0/2288138252 learned_addr learned my addr 172.21.15.202:0/2288138252 (peer_addr_for_me v2:172.21.15.202:0/0) 2022-06-27T11:47:36.525 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.517+0000 7fe2ace5b700 1 --2- 172.21.15.202:0/2288138252 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe2a80c2bf0 0x7fe2a80bf6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:36.525 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.518+0000 7fe2ad65c700 1 -- 172.21.15.202:0/2288138252 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe2a80c5000 msgr2=0x7fe2a81697c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:36.525 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.518+0000 7fe2ad65c700 1 --2- 172.21.15.202:0/2288138252 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe2a80c5000 0x7fe2a81697c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:36.525 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.518+0000 7fe2ad65c700 1 -- 172.21.15.202:0/2288138252 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe2a80c2bf0 msgr2=0x7fe2a80bf6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:36.526 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.518+0000 7fe2ad65c700 1 --2- 172.21.15.202:0/2288138252 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe2a80c2bf0 0x7fe2a80bf6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:36.526 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.518+0000 7fe2ad65c700 1 -- 172.21.15.202:0/2288138252 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2a0009a20 con 0x7fe2a80c61e0 2022-06-27T11:47:36.526 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.518+0000 7fe2ad65c700 1 --2- 172.21.15.202:0/2288138252 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe2a80c61e0 0x7fe2a8068200 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe2a806f840 tx=0x7fe2a0019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:36.526 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.518+0000 7fe2a4ff9700 1 -- 172.21.15.202:0/2288138252 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2a00038a0 con 0x7fe2a80c61e0 2022-06-27T11:47:36.527 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.518+0000 7fe2a4ff9700 1 -- 172.21.15.202:0/2288138252 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2a0003a00 con 0x7fe2a80c61e0 2022-06-27T11:47:36.527 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.518+0000 7fe2af0bf700 1 -- 172.21.15.202:0/2288138252 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2a80bb7d0 con 0x7fe2a80c61e0 2022-06-27T11:47:36.527 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.518+0000 7fe2a4ff9700 1 -- 172.21.15.202:0/2288138252 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2a0003e00 con 0x7fe2a80c61e0 2022-06-27T11:47:36.527 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.518+0000 7fe2af0bf700 1 -- 172.21.15.202:0/2288138252 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2a80bbd90 con 0x7fe2a80c61e0 2022-06-27T11:47:36.527 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.518+0000 7fe2a4ff9700 1 -- 172.21.15.202:0/2288138252 <== mon.2 v2:172.21.15.202:3301/0 4 ==== mgrmap(e 1) v1 ==== 517+0+0 (secure 0 0 0) 0x7fe2a002b020 con 0x7fe2a80c61e0 2022-06-27T11:47:36.528 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.518+0000 7fe2a4ff9700 1 -- 172.21.15.202:0/2288138252 <== mon.2 v2:172.21.15.202:3301/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1066+0+0 (secure 0 0 0) 0x7fe2a0004260 con 0x7fe2a80c61e0 2022-06-27T11:47:36.528 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.519+0000 7fe2af0bf700 1 -- 172.21.15.202:0/2288138252 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2a80722c0 con 0x7fe2a80c61e0 2022-06-27T11:47:36.528 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.523+0000 7fe2a4ff9700 1 -- 172.21.15.202:0/2288138252 <== mon.2 v2:172.21.15.202:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+70815 (secure 0 0 0) 0x7fe2a0004710 con 0x7fe2a80c61e0 2022-06-27T11:47:36.622 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.619+0000 7fe2af0bf700 1 -- 172.21.15.202:0/2288138252 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_command({"prefix": "osd new", "uuid": "e8aa47ce-af8c-47e7-9654-870cd53229f4", "id": 3} v 0) v1 -- 0x7fe2a80c5c70 con 0x7fe2a80c61e0 2022-06-27T11:47:36.623 INFO:tasks.ceph.mgr.y.smithi195.stderr:2022-06-27T11:47:36.624+0000 7f7d32d42080 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2022-06-27T11:47:36.629 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.627+0000 7fe2a4ff9700 1 -- 172.21.15.202:0/2288138252 <== mon.2 v2:172.21.15.202:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e8aa47ce-af8c-47e7-9654-870cd53229f4", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7fe2a002e440 con 0x7fe2a80c61e0 2022-06-27T11:47:36.630 INFO:teuthology.orchestra.run.smithi202.stdout:3 2022-06-27T11:47:36.632 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.628+0000 7fe2af0bf700 1 -- 172.21.15.202:0/2288138252 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe2a80c61e0 msgr2=0x7fe2a8068200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:36.632 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.628+0000 7fe2af0bf700 1 --2- 172.21.15.202:0/2288138252 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe2a80c61e0 0x7fe2a8068200 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe2a806f840 tx=0x7fe2a0019000 comp rx=0 tx=0).stop 2022-06-27T11:47:36.633 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.628+0000 7fe2af0bf700 1 -- 172.21.15.202:0/2288138252 shutdown_connections 2022-06-27T11:47:36.633 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.628+0000 7fe2af0bf700 1 --2- 172.21.15.202:0/2288138252 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe2a80c2bf0 0x7fe2a80bf6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:36.634 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.628+0000 7fe2af0bf700 1 --2- 172.21.15.202:0/2288138252 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe2a80c61e0 0x7fe2a8068200 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:36.634 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.628+0000 7fe2af0bf700 1 --2- 172.21.15.202:0/2288138252 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe2a80c5000 0x7fe2a81697c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:36.634 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.628+0000 7fe2af0bf700 1 -- 172.21.15.202:0/2288138252 >> 172.21.15.202:0/2288138252 conn(0x7fe2a80c9b00 msgr2=0x7fe2a814dd50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:36.635 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.628+0000 7fe2af0bf700 1 -- 172.21.15.202:0/2288138252 shutdown_connections 2022-06-27T11:47:36.635 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.628+0000 7fe2af0bf700 1 -- 172.21.15.202:0/2288138252 wait complete. 2022-06-27T11:47:36.639 DEBUG:teuthology.orchestra.run.smithi202:> sudo ceph --cluster ceph osd new 99ffa553-f149-443a-98f5-68b10f77e085 4 2022-06-27T11:47:36.680 INFO:tasks.ceph.mgr.x.smithi202.stderr:2022-06-27T11:47:36.681+0000 7f4976dc5080 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2022-06-27T11:47:36.775 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.762+0000 7eff9d2d6700 1 Processor -- start 2022-06-27T11:47:36.775 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.762+0000 7eff9d2d6700 1 -- start start 2022-06-27T11:47:36.776 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.763+0000 7eff9d2d6700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7eff98150500 0x7eff98063af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:36.777 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.763+0000 7eff9d2d6700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff98064030 con 0x7eff981508d0 2022-06-27T11:47:36.777 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.763+0000 7eff9d2d6700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff980c4160 con 0x7eff9806a450 2022-06-27T11:47:36.778 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.763+0000 7eff9d2d6700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff980c4440 con 0x7eff98150500 2022-06-27T11:47:36.778 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.763+0000 7eff96ffd700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7eff98150500 0x7eff98063af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:36.779 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.763+0000 7eff967fc700 1 --1- >> v1:172.21.15.202:6789/0 conn(0x7eff9806a450 0x7eff980c48d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.202:6789/0 says I am v1:172.21.15.202:47502/0 (socket says 172.21.15.202:47502) 2022-06-27T11:47:36.779 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.763+0000 7eff967fc700 1 -- 172.21.15.202:0/579499139 learned_addr learned my addr 172.21.15.202:0/579499139 (peer_addr_for_me v1:172.21.15.202:0/0) 2022-06-27T11:47:36.780 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.764+0000 7eff96ffd700 1 -- 172.21.15.202:0/579499139 >> v1:172.21.15.202:6789/0 conn(0x7eff9806a450 legacy=0x7eff980c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:36.780 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.764+0000 7eff95ffb700 1 -- 172.21.15.202:0/579499139 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1600447736 0 0) 0x7eff980c4160 con 0x7eff9806a450 2022-06-27T11:47:36.780 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.764+0000 7eff96ffd700 1 -- 172.21.15.202:0/579499139 >> v1:172.21.15.195:6789/0 conn(0x7eff981508d0 legacy=0x7eff980bbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:36.781 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.764+0000 7eff96ffd700 1 -- 172.21.15.202:0/579499139 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff9815a800 con 0x7eff98150500 2022-06-27T11:47:36.781 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.764+0000 7eff96ffd700 1 --2- 172.21.15.202:0/579499139 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7eff98150500 0x7eff98063af0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7eff88009750 tx=0x7eff88009b10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c753ad266fce1d94 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:36.781 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.764+0000 7eff95ffb700 1 -- 172.21.15.202:0/579499139 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff88005ad0 con 0x7eff98150500 2022-06-27T11:47:36.781 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.764+0000 7eff95ffb700 1 -- 172.21.15.202:0/579499139 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff88005c30 con 0x7eff98150500 2022-06-27T11:47:36.782 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.764+0000 7eff95ffb700 1 -- 172.21.15.202:0/579499139 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff88006030 con 0x7eff98150500 2022-06-27T11:47:36.782 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.765+0000 7eff9d2d6700 1 -- 172.21.15.202:0/579499139 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7eff98150500 msgr2=0x7eff98063af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:36.782 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.765+0000 7eff9d2d6700 1 --2- 172.21.15.202:0/579499139 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7eff98150500 0x7eff98063af0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7eff88009750 tx=0x7eff88009b10 comp rx=0 tx=0).stop 2022-06-27T11:47:36.782 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.765+0000 7eff9d2d6700 1 -- 172.21.15.202:0/579499139 shutdown_connections 2022-06-27T11:47:36.783 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.765+0000 7eff9d2d6700 1 --2- 172.21.15.202:0/579499139 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7eff98150500 0x7eff98063af0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:36.783 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.765+0000 7eff9d2d6700 1 -- 172.21.15.202:0/579499139 >> 172.21.15.202:0/579499139 conn(0x7eff980c9b00 msgr2=0x7eff980c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:36.783 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.765+0000 7eff9d2d6700 1 -- 172.21.15.202:0/579499139 shutdown_connections 2022-06-27T11:47:36.784 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.765+0000 7eff9d2d6700 1 -- 172.21.15.202:0/579499139 wait complete. 2022-06-27T11:47:36.784 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.766+0000 7eff9d2d6700 1 Processor -- start 2022-06-27T11:47:36.785 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.766+0000 7eff9d2d6700 1 -- start start 2022-06-27T11:47:36.785 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.767+0000 7eff9d2d6700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7eff9806a450 0x7eff98167010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:36.785 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.767+0000 7eff9d2d6700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7eff98150500 0x7eff980c2740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:36.786 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.767+0000 7eff9d2d6700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7eff981508d0 0x7eff98158b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:36.786 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.767+0000 7eff9d2d6700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff980c7cf0 con 0x7eff98150500 2022-06-27T11:47:36.787 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.767+0000 7eff9d2d6700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff98167550 con 0x7eff981508d0 2022-06-27T11:47:36.787 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.767+0000 7eff9d2d6700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff980c08c0 con 0x7eff9806a450 2022-06-27T11:47:36.788 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.767+0000 7eff96ffd700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7eff9806a450 0x7eff98167010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:36.788 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.767+0000 7eff96ffd700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7eff9806a450 0x7eff98167010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3301/0 says I am v2:172.21.15.202:36992/0 (socket says 172.21.15.202:36992) 2022-06-27T11:47:36.788 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.767+0000 7eff977fe700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7eff981508d0 0x7eff98158b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:36.789 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.767+0000 7eff96ffd700 1 -- 172.21.15.202:0/3893323767 learned_addr learned my addr 172.21.15.202:0/3893323767 (peer_addr_for_me v2:172.21.15.202:0/0) 2022-06-27T11:47:36.789 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.767+0000 7eff967fc700 1 --2- 172.21.15.202:0/3893323767 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7eff98150500 0x7eff980c2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:36.789 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.767+0000 7eff96ffd700 1 -- 172.21.15.202:0/3893323767 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7eff981508d0 msgr2=0x7eff98158b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:36.790 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.767+0000 7eff96ffd700 1 --2- 172.21.15.202:0/3893323767 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7eff981508d0 0x7eff98158b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:36.790 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.767+0000 7eff96ffd700 1 -- 172.21.15.202:0/3893323767 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7eff98150500 msgr2=0x7eff980c2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:36.791 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.767+0000 7eff96ffd700 1 --2- 172.21.15.202:0/3893323767 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7eff98150500 0x7eff980c2740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:36.791 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.767+0000 7eff96ffd700 1 -- 172.21.15.202:0/3893323767 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff88009290 con 0x7eff9806a450 2022-06-27T11:47:36.791 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.767+0000 7eff977fe700 1 --2- 172.21.15.202:0/3893323767 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7eff981508d0 0x7eff98158b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:47:36.792 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.767+0000 7eff96ffd700 1 --2- 172.21.15.202:0/3893323767 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7eff9806a450 0x7eff98167010 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7eff88009b50 tx=0x7eff88016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:36.792 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.767+0000 7eff7effd700 1 -- 172.21.15.202:0/3893323767 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff8801d780 con 0x7eff9806a450 2022-06-27T11:47:36.793 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.767+0000 7eff7effd700 1 -- 172.21.15.202:0/3893323767 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff8801d8e0 con 0x7eff9806a450 2022-06-27T11:47:36.793 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.767+0000 7eff9d2d6700 1 -- 172.21.15.202:0/3893323767 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff980c0b80 con 0x7eff9806a450 2022-06-27T11:47:36.793 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.767+0000 7eff7effd700 1 -- 172.21.15.202:0/3893323767 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff8801dca0 con 0x7eff9806a450 2022-06-27T11:47:36.794 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.768+0000 7eff9d2d6700 1 -- 172.21.15.202:0/3893323767 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff98158060 con 0x7eff9806a450 2022-06-27T11:47:36.794 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.768+0000 7eff7effd700 1 -- 172.21.15.202:0/3893323767 <== mon.2 v2:172.21.15.202:3301/0 4 ==== mgrmap(e 1) v1 ==== 517+0+0 (secure 0 0 0) 0x7eff8800f050 con 0x7eff9806a450 2022-06-27T11:47:36.794 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.768+0000 7eff7effd700 1 -- 172.21.15.202:0/3893323767 <== mon.2 v2:172.21.15.202:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1181+0+0 (secure 0 0 0) 0x7eff88004340 con 0x7eff9806a450 2022-06-27T11:47:36.794 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.769+0000 7eff9d2d6700 1 -- 172.21.15.202:0/3893323767 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff980722c0 con 0x7eff9806a450 2022-06-27T11:47:36.795 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.771+0000 7eff7effd700 1 -- 172.21.15.202:0/3893323767 <== mon.2 v2:172.21.15.202:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+70815 (secure 0 0 0) 0x7eff880444b0 con 0x7eff9806a450 2022-06-27T11:47:36.852 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.853+0000 7eff9d2d6700 1 -- 172.21.15.202:0/3893323767 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_command({"prefix": "osd new", "uuid": "99ffa553-f149-443a-98f5-68b10f77e085", "id": 4} v 0) v1 -- 0x7eff980ba8e0 con 0x7eff9806a450 2022-06-27T11:47:36.860 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.862+0000 7eff7effd700 1 -- 172.21.15.202:0/3893323767 <== mon.2 v2:172.21.15.202:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "99ffa553-f149-443a-98f5-68b10f77e085", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7eff88033070 con 0x7eff9806a450 2022-06-27T11:47:36.861 INFO:teuthology.orchestra.run.smithi202.stdout:4 2022-06-27T11:47:36.863 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.863+0000 7eff9d2d6700 1 -- 172.21.15.202:0/3893323767 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7eff9806a450 msgr2=0x7eff98167010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:36.863 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.863+0000 7eff9d2d6700 1 --2- 172.21.15.202:0/3893323767 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7eff9806a450 0x7eff98167010 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7eff88009b50 tx=0x7eff88016000 comp rx=0 tx=0).stop 2022-06-27T11:47:36.864 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.863+0000 7eff9d2d6700 1 -- 172.21.15.202:0/3893323767 shutdown_connections 2022-06-27T11:47:36.864 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.863+0000 7eff9d2d6700 1 --2- 172.21.15.202:0/3893323767 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7eff981508d0 0x7eff98158b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:36.864 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.863+0000 7eff9d2d6700 1 --2- 172.21.15.202:0/3893323767 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7eff9806a450 0x7eff98167010 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:36.865 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.863+0000 7eff9d2d6700 1 --2- 172.21.15.202:0/3893323767 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7eff98150500 0x7eff980c2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:36.865 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.863+0000 7eff9d2d6700 1 -- 172.21.15.202:0/3893323767 >> 172.21.15.202:0/3893323767 conn(0x7eff980c9b00 msgr2=0x7eff980c6e40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:36.865 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.863+0000 7eff9d2d6700 1 -- 172.21.15.202:0/3893323767 shutdown_connections 2022-06-27T11:47:36.865 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:36.863+0000 7eff9d2d6700 1 -- 172.21.15.202:0/3893323767 wait complete. 2022-06-27T11:47:36.873 DEBUG:teuthology.orchestra.run.smithi202:> sudo ceph --cluster ceph osd new 35409ef0-891b-4c01-82e4-26b8e64f502d 5 2022-06-27T11:47:37.036 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.037+0000 7ffb7ae7b700 1 Processor -- start 2022-06-27T11:47:37.036 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.037+0000 7ffb7ae7b700 1 -- start start 2022-06-27T11:47:37.039 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.038+0000 7ffb7ae7b700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7ffb74150950 0x7ffb74156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:37.040 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.038+0000 7ffb7ae7b700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb740c76b0 con 0x7ffb74150580 2022-06-27T11:47:37.040 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.038+0000 7ffb7ae7b700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb740c8250 con 0x7ffb7406f620 2022-06-27T11:47:37.040 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.038+0000 7ffb7ae7b700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb74157430 con 0x7ffb74150950 2022-06-27T11:47:37.040 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.038+0000 7ffb73fff700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7ffb74150950 0x7ffb74156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:37.041 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.038+0000 7ffb73fff700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7ffb74150950 0x7ffb74156ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3300/0 says I am v2:172.21.15.202:50146/0 (socket says 172.21.15.202:50146) 2022-06-27T11:47:37.041 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.038+0000 7ffb73fff700 1 -- 172.21.15.202:0/472650802 learned_addr learned my addr 172.21.15.202:0/472650802 (peer_addr_for_me v2:172.21.15.202:0/0) 2022-06-27T11:47:37.041 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.038+0000 7ffb73fff700 1 -- 172.21.15.202:0/472650802 >> v1:172.21.15.202:6789/0 conn(0x7ffb7406f620 legacy=0x7ffb7406d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:37.042 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.038+0000 7ffb73fff700 1 -- 172.21.15.202:0/472650802 >> v1:172.21.15.195:6789/0 conn(0x7ffb74150580 legacy=0x7ffb740c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:37.042 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.038+0000 7ffb73fff700 1 -- 172.21.15.202:0/472650802 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb740bad70 con 0x7ffb74150950 2022-06-27T11:47:37.042 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.038+0000 7ffb73fff700 1 --2- 172.21.15.202:0/472650802 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ffb74150950 0x7ffb74156ef0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ffb60009860 tx=0x7ffb60009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6355bee7177b11e9 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:37.042 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.039+0000 7ffb737fe700 1 -- 172.21.15.202:0/472650802 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb6000b680 con 0x7ffb74150950 2022-06-27T11:47:37.043 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.039+0000 7ffb737fe700 1 -- 172.21.15.202:0/472650802 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb6000b7e0 con 0x7ffb74150950 2022-06-27T11:47:37.043 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.039+0000 7ffb737fe700 1 -- 172.21.15.202:0/472650802 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb6000bba0 con 0x7ffb74150950 2022-06-27T11:47:37.043 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.039+0000 7ffb7ae7b700 1 -- 172.21.15.202:0/472650802 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ffb74150950 msgr2=0x7ffb74156ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:37.043 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.039+0000 7ffb7ae7b700 1 --2- 172.21.15.202:0/472650802 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ffb74150950 0x7ffb74156ef0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ffb60009860 tx=0x7ffb60009120 comp rx=0 tx=0).stop 2022-06-27T11:47:37.044 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.039+0000 7ffb7ae7b700 1 -- 172.21.15.202:0/472650802 shutdown_connections 2022-06-27T11:47:37.044 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.039+0000 7ffb7ae7b700 1 --2- 172.21.15.202:0/472650802 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ffb74150950 0x7ffb74156ef0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:37.044 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.039+0000 7ffb7ae7b700 1 -- 172.21.15.202:0/472650802 >> 172.21.15.202:0/472650802 conn(0x7ffb74063af0 msgr2=0x7ffb74063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:37.044 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.039+0000 7ffb7ae7b700 1 -- 172.21.15.202:0/472650802 shutdown_connections 2022-06-27T11:47:37.044 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.039+0000 7ffb7ae7b700 1 -- 172.21.15.202:0/472650802 wait complete. 2022-06-27T11:47:37.045 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.041+0000 7ffb7ae7b700 1 Processor -- start 2022-06-27T11:47:37.048 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.041+0000 7ffb7ae7b700 1 -- start start 2022-06-27T11:47:37.049 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.042+0000 7ffb7ae7b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffb7406f620 0x7ffb740bc740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:37.049 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.042+0000 7ffb7ae7b700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ffb74150580 0x7ffb740c9640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:37.049 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.042+0000 7ffb7ae7b700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ffb74150950 0x7ffb740c9b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:37.050 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.042+0000 7ffb7ae7b700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb7416a5c0 con 0x7ffb7406f620 2022-06-27T11:47:37.050 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.042+0000 7ffb7ae7b700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb7414e1c0 con 0x7ffb74150580 2022-06-27T11:47:37.051 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.042+0000 7ffb7ae7b700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb740c4ce0 con 0x7ffb74150950 2022-06-27T11:47:37.051 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.042+0000 7ffb73fff700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ffb74150580 0x7ffb740c9640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:37.051 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.042+0000 7ffb73fff700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ffb74150580 0x7ffb740c9640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3300/0 says I am v2:172.21.15.202:50152/0 (socket says 172.21.15.202:50152) 2022-06-27T11:47:37.052 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.042+0000 7ffb73fff700 1 -- 172.21.15.202:0/1051128477 learned_addr learned my addr 172.21.15.202:0/1051128477 (peer_addr_for_me v2:172.21.15.202:0/0) 2022-06-27T11:47:37.052 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.042+0000 7ffb79418700 1 --2- 172.21.15.202:0/1051128477 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ffb74150950 0x7ffb740c9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:37.052 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.042+0000 7ffb78c17700 1 --2- 172.21.15.202:0/1051128477 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffb7406f620 0x7ffb740bc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:37.053 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.042+0000 7ffb73fff700 1 -- 172.21.15.202:0/1051128477 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ffb74150950 msgr2=0x7ffb740c9b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:37.053 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.042+0000 7ffb73fff700 1 --2- 172.21.15.202:0/1051128477 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ffb74150950 0x7ffb740c9b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:37.053 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.042+0000 7ffb73fff700 1 -- 172.21.15.202:0/1051128477 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffb7406f620 msgr2=0x7ffb740bc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:37.054 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.042+0000 7ffb73fff700 1 --2- 172.21.15.202:0/1051128477 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffb7406f620 0x7ffb740bc740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:37.054 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.042+0000 7ffb73fff700 1 -- 172.21.15.202:0/1051128477 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb600093a0 con 0x7ffb74150580 2022-06-27T11:47:37.055 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.042+0000 7ffb73fff700 1 --2- 172.21.15.202:0/1051128477 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ffb74150580 0x7ffb740c9640 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ffb74065b00 tx=0x7ffb60017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:37.055 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.043+0000 7ffb70ff9700 1 -- 172.21.15.202:0/1051128477 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb6000b950 con 0x7ffb74150580 2022-06-27T11:47:37.055 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.043+0000 7ffb70ff9700 1 -- 172.21.15.202:0/1051128477 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb6000bad0 con 0x7ffb74150580 2022-06-27T11:47:37.056 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.043+0000 7ffb7ae7b700 1 -- 172.21.15.202:0/1051128477 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb741699b0 con 0x7ffb74150580 2022-06-27T11:47:37.056 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.043+0000 7ffb70ff9700 1 -- 172.21.15.202:0/1051128477 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb6001a680 con 0x7ffb74150580 2022-06-27T11:47:37.056 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.043+0000 7ffb7ae7b700 1 -- 172.21.15.202:0/1051128477 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb74169f70 con 0x7ffb74150580 2022-06-27T11:47:37.057 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.043+0000 7ffb70ff9700 1 -- 172.21.15.202:0/1051128477 <== mon.1 v2:172.21.15.202:3300/0 4 ==== mgrmap(e 1) v1 ==== 517+0+0 (secure 0 0 0) 0x7ffb6000f050 con 0x7ffb74150580 2022-06-27T11:47:37.057 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.043+0000 7ffb70ff9700 1 -- 172.21.15.202:0/1051128477 <== mon.1 v2:172.21.15.202:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1296+0+0 (secure 0 0 0) 0x7ffb60011a40 con 0x7ffb74150580 2022-06-27T11:47:37.057 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.044+0000 7ffb7ae7b700 1 -- 172.21.15.202:0/1051128477 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb7416a700 con 0x7ffb74150580 2022-06-27T11:47:37.058 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.046+0000 7ffb70ff9700 1 -- 172.21.15.202:0/1051128477 <== mon.1 v2:172.21.15.202:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+70815 (secure 0 0 0) 0x7ffb7416a700 con 0x7ffb74150580 2022-06-27T11:47:37.128 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.129+0000 7ffb7ae7b700 1 -- 172.21.15.202:0/1051128477 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_command({"prefix": "osd new", "uuid": "35409ef0-891b-4c01-82e4-26b8e64f502d", "id": 5} v 0) v1 -- 0x7ffb740c9490 con 0x7ffb74150580 2022-06-27T11:47:37.134 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.135+0000 7ffb70ff9700 1 -- 172.21.15.202:0/1051128477 <== mon.1 v2:172.21.15.202:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "35409ef0-891b-4c01-82e4-26b8e64f502d", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7ffb6003d0b0 con 0x7ffb74150580 2022-06-27T11:47:37.134 INFO:teuthology.orchestra.run.smithi202.stdout:5 2022-06-27T11:47:37.136 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.136+0000 7ffb7ae7b700 1 -- 172.21.15.202:0/1051128477 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ffb74150580 msgr2=0x7ffb740c9640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:37.136 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.136+0000 7ffb7ae7b700 1 --2- 172.21.15.202:0/1051128477 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ffb74150580 0x7ffb740c9640 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ffb74065b00 tx=0x7ffb60017000 comp rx=0 tx=0).stop 2022-06-27T11:47:37.136 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.136+0000 7ffb7ae7b700 1 -- 172.21.15.202:0/1051128477 shutdown_connections 2022-06-27T11:47:37.136 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.136+0000 7ffb7ae7b700 1 --2- 172.21.15.202:0/1051128477 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffb7406f620 0x7ffb740bc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:37.137 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.136+0000 7ffb7ae7b700 1 --2- 172.21.15.202:0/1051128477 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ffb74150950 0x7ffb740c9b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:37.137 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.136+0000 7ffb7ae7b700 1 --2- 172.21.15.202:0/1051128477 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ffb74150580 0x7ffb740c9640 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:37.137 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.136+0000 7ffb7ae7b700 1 -- 172.21.15.202:0/1051128477 >> 172.21.15.202:0/1051128477 conn(0x7ffb74063af0 msgr2=0x7ffb7406d730 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:37.137 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.136+0000 7ffb7ae7b700 1 -- 172.21.15.202:0/1051128477 shutdown_connections 2022-06-27T11:47:37.137 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.136+0000 7ffb7ae7b700 1 -- 172.21.15.202:0/1051128477 wait complete. 2022-06-27T11:47:37.147 DEBUG:teuthology.orchestra.run.smithi202:> sudo ceph --cluster ceph osd new 009124c4-8b5f-47d4-bd20-2c19271a2adc 6 2022-06-27T11:47:37.267 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.268+0000 7fb25595b700 1 Processor -- start 2022-06-27T11:47:37.268 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.269+0000 7fb25595b700 1 -- start start 2022-06-27T11:47:37.279 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.269+0000 7fb25595b700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fb2500c2bf0 0x7fb2500c2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:37.280 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.269+0000 7fb25595b700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2500bc490 con 0x7fb2500c5000 2022-06-27T11:47:37.280 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.269+0000 7fb25595b700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb25006a0e0 con 0x7fb2500c61e0 2022-06-27T11:47:37.280 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.269+0000 7fb25595b700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2500bc6e0 con 0x7fb2500c2bf0 2022-06-27T11:47:37.280 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.269+0000 7fb24e7fc700 1 --1- >> v1:172.21.15.202:6789/0 conn(0x7fb2500c61e0 0x7fb2500c48f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.202:6789/0 says I am v1:172.21.15.202:47524/0 (socket says 172.21.15.202:47524) 2022-06-27T11:47:37.281 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.269+0000 7fb24e7fc700 1 -- 172.21.15.202:0/4216633621 learned_addr learned my addr 172.21.15.202:0/4216633621 (peer_addr_for_me v1:172.21.15.202:0/0) 2022-06-27T11:47:37.281 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.270+0000 7fb24effd700 1 --2- 172.21.15.202:0/4216633621 >> v2:172.21.15.195:3300/0 conn(0x7fb2500c2bf0 0x7fb2500c2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:37.281 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.270+0000 7fb24dffb700 1 -- 172.21.15.202:0/4216633621 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2719906084 0 0) 0x7fb25006a0e0 con 0x7fb2500c61e0 2022-06-27T11:47:37.282 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.270+0000 7fb24dffb700 1 -- 172.21.15.202:0/4216633621 --> v1:172.21.15.202:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb23c003430 con 0x7fb2500c61e0 2022-06-27T11:47:37.282 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.270+0000 7fb24effd700 1 -- 172.21.15.202:0/4216633621 >> v1:172.21.15.202:6789/0 conn(0x7fb2500c61e0 legacy=0x7fb2500c48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:37.283 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.270+0000 7fb24effd700 1 -- 172.21.15.202:0/4216633621 >> v1:172.21.15.195:6789/0 conn(0x7fb2500c5000 legacy=0x7fb2500c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:37.283 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.270+0000 7fb24effd700 1 -- 172.21.15.202:0/4216633621 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb250158220 con 0x7fb2500c2bf0 2022-06-27T11:47:37.283 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.270+0000 7fb24effd700 1 --2- 172.21.15.202:0/4216633621 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb2500c2bf0 0x7fb2500c2fc0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb240009720 tx=0x7fb240009a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f6bcfac60718b4 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:37.283 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.270+0000 7fb24dffb700 1 -- 172.21.15.202:0/4216633621 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb240005ad0 con 0x7fb2500c2bf0 2022-06-27T11:47:37.284 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.270+0000 7fb24dffb700 1 -- 172.21.15.202:0/4216633621 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb240005c30 con 0x7fb2500c2bf0 2022-06-27T11:47:37.285 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.270+0000 7fb24dffb700 1 -- 172.21.15.202:0/4216633621 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb240006030 con 0x7fb2500c2bf0 2022-06-27T11:47:37.285 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.271+0000 7fb25595b700 1 -- 172.21.15.202:0/4216633621 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb2500c2bf0 msgr2=0x7fb2500c2fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:37.285 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.271+0000 7fb25595b700 1 --2- 172.21.15.202:0/4216633621 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb2500c2bf0 0x7fb2500c2fc0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb240009720 tx=0x7fb240009a30 comp rx=0 tx=0).stop 2022-06-27T11:47:37.286 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.271+0000 7fb25595b700 1 -- 172.21.15.202:0/4216633621 shutdown_connections 2022-06-27T11:47:37.286 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.271+0000 7fb25595b700 1 --2- 172.21.15.202:0/4216633621 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb2500c2bf0 0x7fb2500c2fc0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:37.287 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.271+0000 7fb25595b700 1 -- 172.21.15.202:0/4216633621 >> 172.21.15.202:0/4216633621 conn(0x7fb2500c9b00 msgr2=0x7fb2500c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:37.287 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.271+0000 7fb25595b700 1 -- 172.21.15.202:0/4216633621 shutdown_connections 2022-06-27T11:47:37.287 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.271+0000 7fb25595b700 1 -- 172.21.15.202:0/4216633621 wait complete. 2022-06-27T11:47:37.288 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.272+0000 7fb25595b700 1 Processor -- start 2022-06-27T11:47:37.288 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.272+0000 7fb25595b700 1 -- start start 2022-06-27T11:47:37.289 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.273+0000 7fb25595b700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb2500c2bf0 0x7fb25014d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:37.289 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.273+0000 7fb25595b700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb2500c5000 0x7fb2501491c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:37.289 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.273+0000 7fb25595b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb2500c61e0 0x7fb250149d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:37.290 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.273+0000 7fb25595b700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb25015a640 con 0x7fb2500c61e0 2022-06-27T11:47:37.290 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.273+0000 7fb25595b700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb250149010 con 0x7fb2500c2bf0 2022-06-27T11:47:37.291 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.273+0000 7fb25595b700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb25014c730 con 0x7fb2500c5000 2022-06-27T11:47:37.291 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.273+0000 7fb24e7fc700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb2500c5000 0x7fb2501491c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:37.292 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.273+0000 7fb24effd700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb2500c2bf0 0x7fb25014d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:37.292 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.273+0000 7fb24e7fc700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb2500c5000 0x7fb2501491c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3301/0 says I am v2:172.21.15.202:37018/0 (socket says 172.21.15.202:37018) 2022-06-27T11:47:37.292 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.273+0000 7fb24effd700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb2500c2bf0 0x7fb25014d210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3300/0 says I am v2:172.21.15.202:50162/0 (socket says 172.21.15.202:50162) 2022-06-27T11:47:37.293 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.273+0000 7fb24e7fc700 1 -- 172.21.15.202:0/993116876 learned_addr learned my addr 172.21.15.202:0/993116876 (peer_addr_for_me v2:172.21.15.202:0/0) 2022-06-27T11:47:37.293 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.273+0000 7fb24f7fe700 1 --2- 172.21.15.202:0/993116876 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb2500c61e0 0x7fb250149d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:37.293 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.273+0000 7fb24e7fc700 1 -- 172.21.15.202:0/993116876 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb2500c2bf0 msgr2=0x7fb25014d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:37.294 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.273+0000 7fb24e7fc700 1 --2- 172.21.15.202:0/993116876 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb2500c2bf0 0x7fb25014d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:37.295 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.273+0000 7fb24e7fc700 1 -- 172.21.15.202:0/993116876 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb2500c61e0 msgr2=0x7fb250149d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:37.295 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.273+0000 7fb24e7fc700 1 --2- 172.21.15.202:0/993116876 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb2500c61e0 0x7fb250149d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:37.295 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.273+0000 7fb24e7fc700 1 -- 172.21.15.202:0/993116876 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb240009290 con 0x7fb2500c5000 2022-06-27T11:47:37.296 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.273+0000 7fb24effd700 1 --2- 172.21.15.202:0/993116876 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb2500c2bf0 0x7fb25014d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:47:37.296 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.273+0000 7fb24f7fe700 1 --2- 172.21.15.202:0/993116876 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb2500c61e0 0x7fb250149d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:47:37.297 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.273+0000 7fb24e7fc700 1 --2- 172.21.15.202:0/993116876 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb2500c5000 0x7fb2501491c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb24400c4f0 tx=0x7fb244007cb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:37.297 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.274+0000 7fb236ffd700 1 -- 172.21.15.202:0/993116876 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb24400ce00 con 0x7fb2500c5000 2022-06-27T11:47:37.298 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.274+0000 7fb236ffd700 1 -- 172.21.15.202:0/993116876 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2440072e0 con 0x7fb2500c5000 2022-06-27T11:47:37.298 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.274+0000 7fb236ffd700 1 -- 172.21.15.202:0/993116876 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2440076a0 con 0x7fb2500c5000 2022-06-27T11:47:37.299 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.274+0000 7fb25595b700 1 -- 172.21.15.202:0/993116876 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb25014c9d0 con 0x7fb2500c5000 2022-06-27T11:47:37.299 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.274+0000 7fb25595b700 1 -- 172.21.15.202:0/993116876 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb250159e40 con 0x7fb2500c5000 2022-06-27T11:47:37.300 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.274+0000 7fb236ffd700 1 -- 172.21.15.202:0/993116876 <== mon.2 v2:172.21.15.202:3301/0 4 ==== mgrmap(e 1) v1 ==== 517+0+0 (secure 0 0 0) 0x7fb244019030 con 0x7fb2500c5000 2022-06-27T11:47:37.300 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.274+0000 7fb236ffd700 1 -- 172.21.15.202:0/993116876 <== mon.2 v2:172.21.15.202:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1411+0+0 (secure 0 0 0) 0x7fb24401cd60 con 0x7fb2500c5000 2022-06-27T11:47:37.301 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.275+0000 7fb25595b700 1 -- 172.21.15.202:0/993116876 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2500722c0 con 0x7fb2500c5000 2022-06-27T11:47:37.301 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.277+0000 7fb236ffd700 1 -- 172.21.15.202:0/993116876 <== mon.2 v2:172.21.15.202:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+70815 (secure 0 0 0) 0x7fb244021060 con 0x7fb2500c5000 2022-06-27T11:47:37.362 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.360+0000 7fb25595b700 1 -- 172.21.15.202:0/993116876 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_command({"prefix": "osd new", "uuid": "009124c4-8b5f-47d4-bd20-2c19271a2adc", "id": 6} v 0) v1 -- 0x7fb25014cff0 con 0x7fb2500c5000 2022-06-27T11:47:37.367 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.368+0000 7fb236ffd700 1 -- 172.21.15.202:0/993116876 <== mon.2 v2:172.21.15.202:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "009124c4-8b5f-47d4-bd20-2c19271a2adc", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7fb24402a0b0 con 0x7fb2500c5000 2022-06-27T11:47:37.367 INFO:teuthology.orchestra.run.smithi202.stdout:6 2022-06-27T11:47:37.369 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.369+0000 7fb25595b700 1 -- 172.21.15.202:0/993116876 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb2500c5000 msgr2=0x7fb2501491c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:37.369 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.369+0000 7fb25595b700 1 --2- 172.21.15.202:0/993116876 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb2500c5000 0x7fb2501491c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb24400c4f0 tx=0x7fb244007cb0 comp rx=0 tx=0).stop 2022-06-27T11:47:37.369 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.369+0000 7fb25595b700 1 -- 172.21.15.202:0/993116876 shutdown_connections 2022-06-27T11:47:37.369 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.369+0000 7fb25595b700 1 --2- 172.21.15.202:0/993116876 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb2500c61e0 0x7fb250149d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:37.370 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.369+0000 7fb25595b700 1 --2- 172.21.15.202:0/993116876 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb2500c2bf0 0x7fb25014d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:37.370 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.369+0000 7fb25595b700 1 --2- 172.21.15.202:0/993116876 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb2500c5000 0x7fb2501491c0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:37.370 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.369+0000 7fb25595b700 1 -- 172.21.15.202:0/993116876 >> 172.21.15.202:0/993116876 conn(0x7fb2500c9b00 msgr2=0x7fb2500c85e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:37.371 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.369+0000 7fb25595b700 1 -- 172.21.15.202:0/993116876 shutdown_connections 2022-06-27T11:47:37.371 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.369+0000 7fb25595b700 1 -- 172.21.15.202:0/993116876 wait complete. 2022-06-27T11:47:37.381 DEBUG:teuthology.orchestra.run.smithi202:> sudo ceph --cluster ceph osd new 7773115e-d3c3-4a0c-8801-d8ce63e0e5a5 7 2022-06-27T11:47:37.502 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.503+0000 7f684a434700 1 Processor -- start 2022-06-27T11:47:37.502 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.503+0000 7f684a434700 1 -- start start 2022-06-27T11:47:37.516 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.504+0000 7f684a434700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f684406f640 0x7f6844156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:37.517 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.504+0000 7f684a434700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f68440c4160 0x7f68440c1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:37.517 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.504+0000 7f684a434700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f68440c39e0 0x7f68441535a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:37.518 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.504+0000 7f684a434700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68440c1a20 con 0x7f684406f640 2022-06-27T11:47:37.518 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.504+0000 7f684a434700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68440c1130 con 0x7f68440c4160 2022-06-27T11:47:37.519 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.504+0000 7f684a434700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f684406ea90 con 0x7f68440c39e0 2022-06-27T11:47:37.519 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.504+0000 7f68437fe700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f68440c4160 0x7f68440c1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:37.519 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.504+0000 7f68489d1700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f68440c39e0 0x7f68441535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:37.520 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.504+0000 7f68437fe700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f68440c4160 0x7f68440c1eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3300/0 says I am v2:172.21.15.202:50170/0 (socket says 172.21.15.202:50170) 2022-06-27T11:47:37.520 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.504+0000 7f68489d1700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f68440c39e0 0x7f68441535a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3301/0 says I am v2:172.21.15.202:37026/0 (socket says 172.21.15.202:37026) 2022-06-27T11:47:37.521 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.504+0000 7f68437fe700 1 -- 172.21.15.202:0/3163134000 learned_addr learned my addr 172.21.15.202:0/3163134000 (peer_addr_for_me v2:172.21.15.202:0/0) 2022-06-27T11:47:37.521 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.504+0000 7f6843fff700 1 --2- 172.21.15.202:0/3163134000 >> v2:172.21.15.195:3300/0 conn(0x7f684406f640 0x7f6844156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:37.521 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.504+0000 7f68437fe700 1 -- 172.21.15.202:0/3163134000 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f68440c39e0 msgr2=0x7f68441535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:47:37.522 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.504+0000 7f68437fe700 1 --2- 172.21.15.202:0/3163134000 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f68440c39e0 0x7f68441535a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:37.522 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.504+0000 7f68437fe700 1 -- 172.21.15.202:0/3163134000 >> v2:172.21.15.195:3300/0 conn(0x7f684406f640 msgr2=0x7f6844156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:47:37.523 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.504+0000 7f68437fe700 1 --2- 172.21.15.202:0/3163134000 >> v2:172.21.15.195:3300/0 conn(0x7f684406f640 0x7f6844156ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:37.523 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.504+0000 7f68437fe700 1 -- 172.21.15.202:0/3163134000 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f684406ed70 con 0x7f68440c4160 2022-06-27T11:47:37.524 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.504+0000 7f68437fe700 1 --2- 172.21.15.202:0/3163134000 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f68440c4160 0x7f68440c1eb0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f684406dd70 tx=0x7f6838009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ad1aeab6d009a03 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:37.524 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.504+0000 7f6842ffd700 1 -- 172.21.15.202:0/3163134000 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6838011630 con 0x7f68440c4160 2022-06-27T11:47:37.524 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.505+0000 7f6842ffd700 1 -- 172.21.15.202:0/3163134000 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6838011790 con 0x7f68440c4160 2022-06-27T11:47:37.525 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.505+0000 7f6842ffd700 1 -- 172.21.15.202:0/3163134000 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6838011b30 con 0x7f68440c4160 2022-06-27T11:47:37.525 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.505+0000 7f684a434700 1 -- 172.21.15.202:0/3163134000 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f68440c4160 msgr2=0x7f68440c1eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:37.526 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.505+0000 7f684a434700 1 --2- 172.21.15.202:0/3163134000 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f68440c4160 0x7f68440c1eb0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f684406dd70 tx=0x7f6838009120 comp rx=0 tx=0).stop 2022-06-27T11:47:37.526 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.505+0000 7f684a434700 1 -- 172.21.15.202:0/3163134000 shutdown_connections 2022-06-27T11:47:37.526 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.505+0000 7f684a434700 1 --2- 172.21.15.202:0/3163134000 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f68440c39e0 0x7f68441535a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:37.527 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.505+0000 7f684a434700 1 --2- 172.21.15.202:0/3163134000 >> v2:172.21.15.195:3300/0 conn(0x7f684406f640 0x7f6844156ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:37.527 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.505+0000 7f684a434700 1 --2- 172.21.15.202:0/3163134000 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f68440c4160 0x7f68440c1eb0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:37.527 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.505+0000 7f684a434700 1 -- 172.21.15.202:0/3163134000 >> 172.21.15.202:0/3163134000 conn(0x7f68440c6350 msgr2=0x7f68440c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:37.528 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.505+0000 7f684a434700 1 -- 172.21.15.202:0/3163134000 shutdown_connections 2022-06-27T11:47:37.528 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.505+0000 7f684a434700 1 -- 172.21.15.202:0/3163134000 wait complete. 2022-06-27T11:47:37.529 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.506+0000 7f684a434700 1 Processor -- start 2022-06-27T11:47:37.529 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.506+0000 7f684a434700 1 -- start start 2022-06-27T11:47:37.529 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.507+0000 7f684a434700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f684406f640 0x7f6844157600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:37.530 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.507+0000 7f684a434700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f68440c39e0 0x7f6844152540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:37.530 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.507+0000 7f684a434700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f68440c4160 0x7f68440c7d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:37.530 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.507+0000 7f684a434700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68440c7950 con 0x7f684406f640 2022-06-27T11:47:37.531 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.507+0000 7f684a434700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6844157b40 con 0x7f68440c39e0 2022-06-27T11:47:37.531 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.507+0000 7f684a434700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6844169e10 con 0x7f68440c4160 2022-06-27T11:47:37.531 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.507+0000 7f68489d1700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f68440c4160 0x7f68440c7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:37.531 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.507+0000 7f68489d1700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f68440c4160 0x7f68440c7d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3301/0 says I am v2:172.21.15.202:37032/0 (socket says 172.21.15.202:37032) 2022-06-27T11:47:37.531 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.507+0000 7f68489d1700 1 -- 172.21.15.202:0/4137829438 learned_addr learned my addr 172.21.15.202:0/4137829438 (peer_addr_for_me v2:172.21.15.202:0/0) 2022-06-27T11:47:37.532 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.507+0000 7f6843fff700 1 --2- 172.21.15.202:0/4137829438 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f684406f640 0x7f6844157600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:37.532 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.507+0000 7f68437fe700 1 --2- 172.21.15.202:0/4137829438 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f68440c39e0 0x7f6844152540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:37.532 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.508+0000 7f6843fff700 1 -- 172.21.15.202:0/4137829438 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f68440c4160 msgr2=0x7f68440c7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:37.533 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.508+0000 7f6843fff700 1 --2- 172.21.15.202:0/4137829438 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f68440c4160 0x7f68440c7d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:37.533 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.508+0000 7f6843fff700 1 -- 172.21.15.202:0/4137829438 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f68440c39e0 msgr2=0x7f6844152540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:37.534 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.508+0000 7f6843fff700 1 --2- 172.21.15.202:0/4137829438 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f68440c39e0 0x7f6844152540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:37.534 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.508+0000 7f6843fff700 1 -- 172.21.15.202:0/4137829438 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68380093a0 con 0x7f684406f640 2022-06-27T11:47:37.535 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.508+0000 7f6843fff700 1 --2- 172.21.15.202:0/4137829438 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f684406f640 0x7f6844157600 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f683400e8b0 tx=0x7f6834009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:37.535 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.508+0000 7f682bfff700 1 -- 172.21.15.202:0/4137829438 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6834013680 con 0x7f684406f640 2022-06-27T11:47:37.535 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.508+0000 7f682bfff700 1 -- 172.21.15.202:0/4137829438 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68340137e0 con 0x7f684406f640 2022-06-27T11:47:37.536 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.508+0000 7f682bfff700 1 -- 172.21.15.202:0/4137829438 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6834013b80 con 0x7f684406f640 2022-06-27T11:47:37.536 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.508+0000 7f684a434700 1 -- 172.21.15.202:0/4137829438 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68440c58a0 con 0x7f684406f640 2022-06-27T11:47:37.536 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.508+0000 7f684a434700 1 -- 172.21.15.202:0/4137829438 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f684416a370 con 0x7f684406f640 2022-06-27T11:47:37.536 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.508+0000 7f682bfff700 1 -- 172.21.15.202:0/4137829438 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 1) v1 ==== 517+0+0 (secure 0 0 0) 0x7f6834019030 con 0x7f684406f640 2022-06-27T11:47:37.537 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.508+0000 7f682bfff700 1 -- 172.21.15.202:0/4137829438 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1526+0+0 (secure 0 0 0) 0x7f68340123d0 con 0x7f684406f640 2022-06-27T11:47:37.537 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.509+0000 7f684a434700 1 -- 172.21.15.202:0/4137829438 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68440722c0 con 0x7f684406f640 2022-06-27T11:47:37.537 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.513+0000 7f682bfff700 1 -- 172.21.15.202:0/4137829438 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+70815 (secure 0 0 0) 0x7f6834021060 con 0x7f684406f640 2022-06-27T11:47:37.594 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.595+0000 7f684a434700 1 -- 172.21.15.202:0/4137829438 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "osd new", "uuid": "7773115e-d3c3-4a0c-8801-d8ce63e0e5a5", "id": 7} v 0) v1 -- 0x7f684406f240 con 0x7f684406f640 2022-06-27T11:47:37.597 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.598+0000 7f682bfff700 1 -- 172.21.15.202:0/4137829438 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7773115e-d3c3-4a0c-8801-d8ce63e0e5a5", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f6834017070 con 0x7f684406f640 2022-06-27T11:47:37.598 INFO:teuthology.orchestra.run.smithi202.stdout:7 2022-06-27T11:47:37.600 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.599+0000 7f684a434700 1 -- 172.21.15.202:0/4137829438 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f684406f640 msgr2=0x7f6844157600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:37.600 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.599+0000 7f684a434700 1 --2- 172.21.15.202:0/4137829438 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f684406f640 0x7f6844157600 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f683400e8b0 tx=0x7f6834009060 comp rx=0 tx=0).stop 2022-06-27T11:47:37.601 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.599+0000 7f684a434700 1 -- 172.21.15.202:0/4137829438 shutdown_connections 2022-06-27T11:47:37.601 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.599+0000 7f684a434700 1 --2- 172.21.15.202:0/4137829438 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f684406f640 0x7f6844157600 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:37.602 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.599+0000 7f684a434700 1 --2- 172.21.15.202:0/4137829438 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f68440c4160 0x7f68440c7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:37.602 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.599+0000 7f684a434700 1 --2- 172.21.15.202:0/4137829438 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f68440c39e0 0x7f6844152540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:37.603 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.599+0000 7f684a434700 1 -- 172.21.15.202:0/4137829438 >> 172.21.15.202:0/4137829438 conn(0x7f68440c6350 msgr2=0x7f6844153370 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:37.603 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.599+0000 7f684a434700 1 -- 172.21.15.202:0/4137829438 shutdown_connections 2022-06-27T11:47:37.604 INFO:teuthology.orchestra.run.smithi202.stderr:2022-06-27T11:47:37.599+0000 7f684a434700 1 -- 172.21.15.202:0/4137829438 wait complete. 2022-06-27T11:47:37.612 INFO:tasks.ceph.osd.0:Restarting daemon 2022-06-27T11:47:37.613 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2022-06-27T11:47:37.617 INFO:tasks.ceph.osd.0:Started 2022-06-27T11:47:37.618 INFO:tasks.ceph.osd.1:Restarting daemon 2022-06-27T11:47:37.618 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2022-06-27T11:47:37.622 INFO:tasks.ceph.osd.1:Started 2022-06-27T11:47:37.622 INFO:tasks.ceph.osd.2:Restarting daemon 2022-06-27T11:47:37.623 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2022-06-27T11:47:37.624 INFO:tasks.ceph.mgr.y.smithi195.stderr:2022-06-27T11:47:37.622+0000 7f7d32d42080 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2022-06-27T11:47:37.626 INFO:tasks.ceph.osd.2:Started 2022-06-27T11:47:37.626 INFO:tasks.ceph.osd.3:Restarting daemon 2022-06-27T11:47:37.627 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2022-06-27T11:47:37.629 INFO:tasks.ceph.osd.3:Started 2022-06-27T11:47:37.630 INFO:tasks.ceph.osd.4:Restarting daemon 2022-06-27T11:47:37.630 DEBUG:teuthology.orchestra.run.smithi202:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2022-06-27T11:47:37.633 INFO:tasks.ceph.osd.4:Started 2022-06-27T11:47:37.633 INFO:tasks.ceph.osd.5:Restarting daemon 2022-06-27T11:47:37.633 DEBUG:teuthology.orchestra.run.smithi202:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2022-06-27T11:47:37.636 INFO:tasks.ceph.osd.5:Started 2022-06-27T11:47:37.637 INFO:tasks.ceph.osd.6:Restarting daemon 2022-06-27T11:47:37.637 DEBUG:teuthology.orchestra.run.smithi202:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2022-06-27T11:47:37.643 INFO:tasks.ceph.osd.6:Started 2022-06-27T11:47:37.643 INFO:tasks.ceph.osd.7:Restarting daemon 2022-06-27T11:47:37.644 DEBUG:teuthology.orchestra.run.smithi202:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2022-06-27T11:47:37.649 INFO:tasks.ceph.osd.7:Started 2022-06-27T11:47:37.649 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-27T11:47:37.732 INFO:tasks.ceph.mgr.y.smithi195.stderr:2022-06-27T11:47:37.732+0000 7f7d32d42080 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2022-06-27T11:47:37.775 INFO:tasks.ceph.mgr.x.smithi202.stderr:2022-06-27T11:47:37.776+0000 7f4976dc5080 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2022-06-27T11:47:37.802 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:37.799+0000 7fb16c965700 1 Processor -- start 2022-06-27T11:47:37.802 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:37.799+0000 7fb16c965700 1 -- start start 2022-06-27T11:47:37.803 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:37.800+0000 7fb16c965700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fb16814e490 0x7fb16806cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:37.803 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:37.800+0000 7fb16c965700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb16814e860 0x7fb1680c3840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:37.803 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:37.800+0000 7fb16c965700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1680c33b0 con 0x7fb1680c3d80 2022-06-27T11:47:37.803 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:37.800+0000 7fb16c965700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb16806d520 con 0x7fb16814e490 2022-06-27T11:47:37.804 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:37.800+0000 7fb16c965700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb16806f220 con 0x7fb16814e860 2022-06-27T11:47:37.804 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:37.800+0000 7fb16659c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fb16814e490 0x7fb16806cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:37.804 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:37.800+0000 7fb16659c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fb16814e490 0x7fb16806cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:40584/0 (socket says 172.21.15.195:40584) 2022-06-27T11:47:37.804 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:37.800+0000 7fb16659c700 1 -- 172.21.15.195:0/3323913066 learned_addr learned my addr 172.21.15.195:0/3323913066 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:47:37.805 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:37.800+0000 7fb165d9b700 1 --2- 172.21.15.195:0/3323913066 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb16814e860 0x7fb1680c3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:37.805 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:37.801+0000 7fb16659c700 1 -- 172.21.15.195:0/3323913066 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb16814e860 msgr2=0x7fb1680c3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:47:37.806 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:37.801+0000 7fb16659c700 1 --2- 172.21.15.195:0/3323913066 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb16814e860 0x7fb1680c3840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:37.806 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:37.801+0000 7fb16559a700 1 -- 172.21.15.195:0/3323913066 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 296643654 0 0) 0x7fb1680c33b0 con 0x7fb1680c3d80 2022-06-27T11:47:37.810 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:37.801+0000 7fb16659c700 1 -- 172.21.15.195:0/3323913066 >> v1:172.21.15.202:6789/0 conn(0x7fb1680c3d80 legacy=0x7fb16806f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:37.810 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:37.801+0000 7fb16659c700 1 -- 172.21.15.195:0/3323913066 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1681542c0 con 0x7fb16814e490 2022-06-27T11:47:37.810 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:37.801+0000 7fb16659c700 1 --2- 172.21.15.195:0/3323913066 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb16814e490 0x7fb16806cfe0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fb158009860 tx=0x7fb158009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=195e24d28211489 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:37.811 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:37.801+0000 7fb16559a700 1 -- 172.21.15.195:0/3323913066 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb158011630 con 0x7fb16814e490 2022-06-27T11:47:37.811 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:37.801+0000 7fb16559a700 1 -- 172.21.15.195:0/3323913066 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb158011790 con 0x7fb16814e490 2022-06-27T11:47:37.811 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:37.801+0000 7fb16559a700 1 -- 172.21.15.195:0/3323913066 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb158011b30 con 0x7fb16814e490 2022-06-27T11:47:37.811 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:37.801+0000 7fb16c965700 1 -- 172.21.15.195:0/3323913066 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb16814e490 msgr2=0x7fb16806cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:37.811 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:37.801+0000 7fb16c965700 1 --2- 172.21.15.195:0/3323913066 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb16814e490 0x7fb16806cfe0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fb158009860 tx=0x7fb158009120 comp rx=0 tx=0).stop 2022-06-27T11:47:37.812 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:37.802+0000 7fb16c965700 1 -- 172.21.15.195:0/3323913066 shutdown_connections 2022-06-27T11:47:37.812 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:37.802+0000 7fb16c965700 1 --2- 172.21.15.195:0/3323913066 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb16814e490 0x7fb16806cfe0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:37.812 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:37.802+0000 7fb16c965700 1 --2- 172.21.15.195:0/3323913066 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb16814e860 0x7fb1680c3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:37.812 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:37.802+0000 7fb16c965700 1 -- 172.21.15.195:0/3323913066 >> 172.21.15.195:0/3323913066 conn(0x7fb168063af0 msgr2=0x7fb168063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:37.812 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:37.802+0000 7fb16c965700 1 -- 172.21.15.195:0/3323913066 shutdown_connections 2022-06-27T11:47:37.813 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:37.802+0000 7fb16c965700 1 -- 172.21.15.195:0/3323913066 wait complete. 2022-06-27T11:47:37.813 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:37.803+0000 7fb16c965700 1 Processor -- start 2022-06-27T11:47:37.813 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:37.803+0000 7fb16c965700 1 -- start start 2022-06-27T11:47:37.817 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:37.803+0000 7fb16c965700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb1680c3d80 0x7fb1681534a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:37.818 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:37.803+0000 7fb16c965700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb16814e490 0x7fb16814faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:37.818 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:37.803+0000 7fb16c965700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb16814e860 0x7fb16814ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:37.819 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:37.803+0000 7fb16c965700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb168150520 con 0x7fb1680c3d80 2022-06-27T11:47:37.819 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:37.803+0000 7fb16c965700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1680bdbd0 con 0x7fb16814e490 2022-06-27T11:47:37.819 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:37.803+0000 7fb16c965700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1680bde20 con 0x7fb16814e860 2022-06-27T11:47:37.820 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:37.803+0000 7fb16659c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb1680c3d80 0x7fb1681534a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:37.820 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:37.803+0000 7fb16659c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb1680c3d80 0x7fb1681534a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:40590/0 (socket says 172.21.15.195:40590) 2022-06-27T11:47:37.820 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:37.804+0000 7fb16659c700 1 -- 172.21.15.195:0/4067277102 learned_addr learned my addr 172.21.15.195:0/4067277102 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:47:37.821 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:37.804+0000 7fb165d9b700 1 --2- 172.21.15.195:0/4067277102 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb16814e490 0x7fb16814faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:37.821 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:37.804+0000 7fb166d9d700 1 --2- 172.21.15.195:0/4067277102 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb16814e860 0x7fb16814ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:37.821 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:37.804+0000 7fb16659c700 1 -- 172.21.15.195:0/4067277102 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb16814e860 msgr2=0x7fb16814ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:37.821 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:37.804+0000 7fb16659c700 1 --2- 172.21.15.195:0/4067277102 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb16814e860 0x7fb16814ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:37.821 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:37.804+0000 7fb16659c700 1 -- 172.21.15.195:0/4067277102 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb16814e490 msgr2=0x7fb16814faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:37.822 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:37.804+0000 7fb16659c700 1 --2- 172.21.15.195:0/4067277102 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb16814e490 0x7fb16814faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:37.822 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:37.804+0000 7fb16659c700 1 -- 172.21.15.195:0/4067277102 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1580093a0 con 0x7fb1680c3d80 2022-06-27T11:47:37.822 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:37.804+0000 7fb16659c700 1 --2- 172.21.15.195:0/4067277102 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb1680c3d80 0x7fb1681534a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb158009f00 tx=0x7fb158017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:37.823 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:37.804+0000 7fb1567fc700 1 -- 172.21.15.195:0/4067277102 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb158011630 con 0x7fb1680c3d80 2022-06-27T11:47:37.823 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:37.804+0000 7fb1567fc700 1 -- 172.21.15.195:0/4067277102 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb158003730 con 0x7fb1680c3d80 2022-06-27T11:47:37.823 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:37.804+0000 7fb1567fc700 1 -- 172.21.15.195:0/4067277102 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb158003af0 con 0x7fb1680c3d80 2022-06-27T11:47:37.823 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:37.804+0000 7fb16c965700 1 -- 172.21.15.195:0/4067277102 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb16814ed00 con 0x7fb1680c3d80 2022-06-27T11:47:37.824 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:37.804+0000 7fb16c965700 1 -- 172.21.15.195:0/4067277102 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb16814f2c0 con 0x7fb1680c3d80 2022-06-27T11:47:37.824 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:37.804+0000 7fb1567fc700 1 -- 172.21.15.195:0/4067277102 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 1) v1 ==== 517+0+0 (secure 0 0 0) 0x7fb15802b030 con 0x7fb1680c3d80 2022-06-27T11:47:37.824 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:37.804+0000 7fb1567fc700 1 -- 172.21.15.195:0/4067277102 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1641+0+0 (secure 0 0 0) 0x7fb15801a450 con 0x7fb1680c3d80 2022-06-27T11:47:37.824 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:37.805+0000 7fb16c965700 1 -- 172.21.15.195:0/4067277102 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1680722c0 con 0x7fb1680c3d80 2022-06-27T11:47:37.824 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:37.807+0000 7fb1567fc700 1 -- 172.21.15.195:0/4067277102 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+70815 (secure 0 0 0) 0x7fb158029070 con 0x7fb1680c3d80 2022-06-27T11:47:37.825 INFO:tasks.ceph.mgr.y.smithi195.stderr:2022-06-27T11:47:37.814+0000 7f7d32d42080 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2022-06-27T11:47:37.857 INFO:tasks.ceph.mgr.x.smithi202.stderr:2022-06-27T11:47:37.859+0000 7f4976dc5080 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2022-06-27T11:47:37.875 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:37.875+0000 7fb16c965700 1 -- 172.21.15.195:0/4067277102 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb1680c41c0 con 0x7fb1680c3d80 2022-06-27T11:47:37.877 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:37.875+0000 7fb1567fc700 1 -- 172.21.15.195:0/4067277102 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7013 (secure 0 0 0) 0x7fb158015020 con 0x7fb1680c3d80 2022-06-27T11:47:37.878 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:47:37.878 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":10,"fsid":"81b3a1d3-6a2b-47e8-8d69-7027cfeae7af","created":"2022-06-27T11:47:35.094715+0000","modified":"2022-06-27T11:47:37.596818+0000","last_up_change":"0.000000","last_in_change":"2022-06-27T11:47:37.596818+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"c64bc51f-d4f3-4bc2-806b-da0dc78d51b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bea71cdf-aa98-493f-9207-66d886a3e30e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bf6c87a0-a8ea-497a-9ce1-721285888065","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8aa47ce-af8c-47e7-9654-870cd53229f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"99ffa553-f149-443a-98f5-68b10f77e085","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"35409ef0-891b-4c01-82e4-26b8e64f502d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"009124c4-8b5f-47d4-bd20-2c19271a2adc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7773115e-d3c3-4a0c-8801-d8ce63e0e5a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-06-27T11:47:37.879 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:37.876+0000 7fb16c965700 1 -- 172.21.15.195:0/4067277102 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb1680c3d80 msgr2=0x7fb1681534a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:37.879 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:37.876+0000 7fb16c965700 1 --2- 172.21.15.195:0/4067277102 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb1680c3d80 0x7fb1681534a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb158009f00 tx=0x7fb158017000 comp rx=0 tx=0).stop 2022-06-27T11:47:37.880 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:37.876+0000 7fb16c965700 1 -- 172.21.15.195:0/4067277102 shutdown_connections 2022-06-27T11:47:37.880 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:37.876+0000 7fb16c965700 1 --2- 172.21.15.195:0/4067277102 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb1680c3d80 0x7fb1681534a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:37.881 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:37.876+0000 7fb16c965700 1 --2- 172.21.15.195:0/4067277102 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb16814e860 0x7fb16814ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:37.881 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:37.876+0000 7fb16c965700 1 --2- 172.21.15.195:0/4067277102 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb16814e490 0x7fb16814faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:37.882 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:37.876+0000 7fb16c965700 1 -- 172.21.15.195:0/4067277102 >> 172.21.15.195:0/4067277102 conn(0x7fb168063af0 msgr2=0x7fb16814d7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:37.882 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:37.876+0000 7fb16c965700 1 -- 172.21.15.195:0/4067277102 shutdown_connections 2022-06-27T11:47:37.882 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:37.876+0000 7fb16c965700 1 -- 172.21.15.195:0/4067277102 wait complete. 2022-06-27T11:47:37.888 INFO:tasks.ceph.ceph_manager.ceph:[] 2022-06-27T11:47:37.889 INFO:tasks.ceph:Waiting for OSDs to come up 2022-06-27T11:47:37.903 INFO:tasks.ceph.mgr.y.smithi195.stderr:2022-06-27T11:47:37.904+0000 7f7d32d42080 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2022-06-27T11:47:37.934 INFO:tasks.ceph.mgr.x.smithi202.stderr:2022-06-27T11:47:37.936+0000 7f4976dc5080 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2022-06-27T11:47:37.990 INFO:tasks.ceph.osd.2.smithi195.stderr:2022-06-27T11:47:37.990+0000 7f088d03f440 -1 Falling back to public interface 2022-06-27T11:47:38.022 INFO:tasks.ceph.mgr.x.smithi202.stderr:2022-06-27T11:47:38.023+0000 7f4976dc5080 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2022-06-27T11:47:38.093 INFO:tasks.ceph.mgr.y.smithi195.stderr:2022-06-27T11:47:38.093+0000 7f7d32d42080 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2022-06-27T11:47:38.210 INFO:tasks.ceph.mgr.x.smithi202.stderr:2022-06-27T11:47:38.210+0000 7f4976dc5080 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2022-06-27T11:47:38.246 INFO:tasks.ceph.osd.1.smithi195.stderr:2022-06-27T11:47:38.246+0000 7fd3413b9440 -1 Falling back to public interface 2022-06-27T11:47:38.247 INFO:tasks.ceph.osd.0.smithi195.stderr:2022-06-27T11:47:38.247+0000 7f5d6d8df440 -1 Falling back to public interface 2022-06-27T11:47:38.250 INFO:tasks.ceph.osd.4.smithi202.stderr:2022-06-27T11:47:38.251+0000 7f4ab95b6440 -1 Falling back to public interface 2022-06-27T11:47:38.255 INFO:tasks.ceph.osd.3.smithi195.stderr:2022-06-27T11:47:38.256+0000 7fad93c17440 -1 Falling back to public interface 2022-06-27T11:47:38.259 INFO:tasks.ceph.osd.6.smithi202.stderr:2022-06-27T11:47:38.260+0000 7f9e34c85440 -1 Falling back to public interface 2022-06-27T11:47:38.259 INFO:tasks.ceph.osd.7.smithi202.stderr:2022-06-27T11:47:38.260+0000 7f4b0df49440 -1 Falling back to public interface 2022-06-27T11:47:38.260 INFO:tasks.ceph.osd.6.smithi202.stderr:2022-06-27T11:47:38.260+0000 7f9e2c249700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6808/0: (98) Address already in use 2022-06-27T11:47:38.263 INFO:tasks.ceph.osd.5.smithi202.stderr:2022-06-27T11:47:38.264+0000 7f4269d04440 -1 Falling back to public interface 2022-06-27T11:47:38.561 INFO:tasks.ceph.mgr.y.smithi195.stderr:2022-06-27T11:47:38.562+0000 7f7d32d42080 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2022-06-27T11:47:38.676 INFO:tasks.ceph.mgr.x.smithi202.stderr:2022-06-27T11:47:38.677+0000 7f4976dc5080 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2022-06-27T11:47:38.696 DEBUG:teuthology.orchestra.run.smithi195:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2022-06-27T11:47:38.812 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.812+0000 7fd89a028700 1 Processor -- start 2022-06-27T11:47:38.812 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.813+0000 7fd89a028700 1 -- start start 2022-06-27T11:47:38.822 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.813+0000 7fd89a028700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd8940c4ea0 0x7fd894156d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:38.824 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.813+0000 7fd89a028700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8941568f0 con 0x7fd8940c58b0 2022-06-27T11:47:38.825 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.813+0000 7fd89a028700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8940c76e0 con 0x7fd8941572c0 2022-06-27T11:47:38.825 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.813+0000 7fd89a028700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd894069090 con 0x7fd8940c4ea0 2022-06-27T11:47:38.825 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.814+0000 7fd8937fe700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fd8940c58b0 0x7fd894067ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:46784/0 (socket says 172.21.15.195:46784) 2022-06-27T11:47:38.826 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.814+0000 7fd8937fe700 1 -- 172.21.15.195:0/2859569582 learned_addr learned my addr 172.21.15.195:0/2859569582 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:47:38.826 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.814+0000 7fd88bfff700 1 --2- 172.21.15.195:0/2859569582 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd8940c4ea0 0x7fd894156d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:38.826 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.814+0000 7fd892ffd700 1 -- 172.21.15.195:0/2859569582 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3491931248 0 0) 0x7fd8941568f0 con 0x7fd8940c58b0 2022-06-27T11:47:38.826 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.814+0000 7fd892ffd700 1 -- 172.21.15.195:0/2859569582 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd874003430 con 0x7fd8940c58b0 2022-06-27T11:47:38.827 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.814+0000 7fd892ffd700 1 -- 172.21.15.195:0/2859569582 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4095067868 0 0) 0x7fd8940c76e0 con 0x7fd8941572c0 2022-06-27T11:47:38.827 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.814+0000 7fd88bfff700 1 -- 172.21.15.195:0/2859569582 >> v1:172.21.15.202:6789/0 conn(0x7fd8941572c0 legacy=0x7fd894153700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:38.827 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.814+0000 7fd88bfff700 1 -- 172.21.15.195:0/2859569582 >> v1:172.21.15.195:6789/0 conn(0x7fd8940c58b0 legacy=0x7fd894067ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:38.827 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.814+0000 7fd88bfff700 1 -- 172.21.15.195:0/2859569582 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd89406c8f0 con 0x7fd8940c4ea0 2022-06-27T11:47:38.828 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.814+0000 7fd88bfff700 1 --2- 172.21.15.195:0/2859569582 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd8940c4ea0 0x7fd894156d80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd87c0097d0 tx=0x7fd87c009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d214253708f4cc3 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:38.828 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.814+0000 7fd892ffd700 1 -- 172.21.15.195:0/2859569582 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd87c011630 con 0x7fd8940c4ea0 2022-06-27T11:47:38.828 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.814+0000 7fd892ffd700 1 -- 172.21.15.195:0/2859569582 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd87c011790 con 0x7fd8940c4ea0 2022-06-27T11:47:38.828 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.814+0000 7fd892ffd700 1 -- 172.21.15.195:0/2859569582 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd87c011b10 con 0x7fd8940c4ea0 2022-06-27T11:47:38.829 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.815+0000 7fd89a028700 1 -- 172.21.15.195:0/2859569582 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd8940c4ea0 msgr2=0x7fd894156d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:38.829 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.815+0000 7fd89a028700 1 --2- 172.21.15.195:0/2859569582 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd8940c4ea0 0x7fd894156d80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd87c0097d0 tx=0x7fd87c009060 comp rx=0 tx=0).stop 2022-06-27T11:47:38.829 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.815+0000 7fd89a028700 1 -- 172.21.15.195:0/2859569582 shutdown_connections 2022-06-27T11:47:38.830 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.815+0000 7fd89a028700 1 --2- 172.21.15.195:0/2859569582 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd8940c4ea0 0x7fd894156d80 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:38.830 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.815+0000 7fd89a028700 1 -- 172.21.15.195:0/2859569582 >> 172.21.15.195:0/2859569582 conn(0x7fd8940c98b0 msgr2=0x7fd8940c9cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:38.830 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.815+0000 7fd89a028700 1 -- 172.21.15.195:0/2859569582 shutdown_connections 2022-06-27T11:47:38.830 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.815+0000 7fd89a028700 1 -- 172.21.15.195:0/2859569582 wait complete. 2022-06-27T11:47:38.831 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.816+0000 7fd89a028700 1 Processor -- start 2022-06-27T11:47:38.831 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.816+0000 7fd89a028700 1 -- start start 2022-06-27T11:47:38.831 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.817+0000 7fd89a028700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd8940c4ea0 0x7fd89414d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:38.831 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.817+0000 7fd89a028700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd8940c58b0 0x7fd8940bd860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:38.832 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.817+0000 7fd89a028700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd8941572c0 0x7fd8940bdda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:38.833 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.817+0000 7fd89a028700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd89406a0a0 con 0x7fd8940c58b0 2022-06-27T11:47:38.833 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.817+0000 7fd89a028700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8940c4cf0 con 0x7fd8941572c0 2022-06-27T11:47:38.834 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.817+0000 7fd89a028700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8940c1200 con 0x7fd8940c4ea0 2022-06-27T11:47:38.834 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.817+0000 7fd88bfff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd8940c58b0 0x7fd8940bd860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:38.835 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.817+0000 7fd8937fe700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd8940c4ea0 0x7fd89414d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:38.835 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.817+0000 7fd88bfff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd8940c58b0 0x7fd8940bd860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:40604/0 (socket says 172.21.15.195:40604) 2022-06-27T11:47:38.835 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.817+0000 7fd893fff700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd8941572c0 0x7fd8940bdda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:38.835 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.817+0000 7fd88bfff700 1 -- 172.21.15.195:0/122383451 learned_addr learned my addr 172.21.15.195:0/122383451 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:47:38.835 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.817+0000 7fd8937fe700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd8940c4ea0 0x7fd89414d220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3301/0 says I am v2:172.21.15.195:33358/0 (socket says 172.21.15.195:33358) 2022-06-27T11:47:38.836 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.817+0000 7fd88bfff700 1 -- 172.21.15.195:0/122383451 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd8940c4ea0 msgr2=0x7fd89414d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:38.836 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.817+0000 7fd88bfff700 1 --2- 172.21.15.195:0/122383451 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd8940c4ea0 0x7fd89414d220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:38.836 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.817+0000 7fd88bfff700 1 -- 172.21.15.195:0/122383451 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd8941572c0 msgr2=0x7fd8940bdda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:38.836 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.817+0000 7fd88bfff700 1 --2- 172.21.15.195:0/122383451 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd8941572c0 0x7fd8940bdda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:38.837 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.817+0000 7fd88bfff700 1 -- 172.21.15.195:0/122383451 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd87c0092e0 con 0x7fd8940c58b0 2022-06-27T11:47:38.837 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.817+0000 7fd8937fe700 1 --2- 172.21.15.195:0/122383451 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd8940c4ea0 0x7fd89414d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:47:38.837 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.817+0000 7fd893fff700 1 --2- 172.21.15.195:0/122383451 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd8941572c0 0x7fd8940bdda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:47:38.837 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.818+0000 7fd88bfff700 1 --2- 172.21.15.195:0/122383451 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd8940c58b0 0x7fd8940bd860 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd87c009e70 tx=0x7fd87c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:38.838 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.818+0000 7fd88b7fe700 1 -- 172.21.15.195:0/122383451 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd87c0045b0 con 0x7fd8940c58b0 2022-06-27T11:47:38.838 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.818+0000 7fd88b7fe700 1 -- 172.21.15.195:0/122383451 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd87c000bc0 con 0x7fd8940c58b0 2022-06-27T11:47:38.838 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.818+0000 7fd89a028700 1 -- 172.21.15.195:0/122383451 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8941678a0 con 0x7fd8940c58b0 2022-06-27T11:47:38.838 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.818+0000 7fd88b7fe700 1 -- 172.21.15.195:0/122383451 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd87c025420 con 0x7fd8940c58b0 2022-06-27T11:47:38.839 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.818+0000 7fd89a028700 1 -- 172.21.15.195:0/122383451 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8940c3660 con 0x7fd8940c58b0 2022-06-27T11:47:38.839 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.818+0000 7fd88b7fe700 1 -- 172.21.15.195:0/122383451 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 1) v1 ==== 517+0+0 (secure 0 0 0) 0x7fd87c02c030 con 0x7fd8940c58b0 2022-06-27T11:47:38.839 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.819+0000 7fd88b7fe700 1 -- 172.21.15.195:0/122383451 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1641+0+0 (secure 0 0 0) 0x7fd87c02ea90 con 0x7fd8940c58b0 2022-06-27T11:47:38.839 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.819+0000 7fd89a028700 1 -- 172.21.15.195:0/122383451 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd894167a50 con 0x7fd8940c58b0 2022-06-27T11:47:38.840 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.821+0000 7fd88b7fe700 1 -- 172.21.15.195:0/122383451 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+70815 (secure 0 0 0) 0x7fd894167a50 con 0x7fd8940c58b0 2022-06-27T11:47:38.844 INFO:tasks.ceph.mgr.y.smithi195.stderr:2022-06-27T11:47:38.844+0000 7f7d32d42080 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2022-06-27T11:47:38.889 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.890+0000 7fd89a028700 1 -- 172.21.15.195:0/122383451 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd89404c470 con 0x7fd8940c58b0 2022-06-27T11:47:38.890 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.890+0000 7fd88b7fe700 1 -- 172.21.15.195:0/122383451 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7013 (secure 0 0 0) 0x7fd87c02a070 con 0x7fd8940c58b0 2022-06-27T11:47:38.891 INFO:teuthology.misc.health.smithi195.stdout: 2022-06-27T11:47:38.891 INFO:teuthology.misc.health.smithi195.stdout:{"epoch":10,"fsid":"81b3a1d3-6a2b-47e8-8d69-7027cfeae7af","created":"2022-06-27T11:47:35.094715+0000","modified":"2022-06-27T11:47:37.596818+0000","last_up_change":"0.000000","last_in_change":"2022-06-27T11:47:37.596818+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"c64bc51f-d4f3-4bc2-806b-da0dc78d51b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bea71cdf-aa98-493f-9207-66d886a3e30e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bf6c87a0-a8ea-497a-9ce1-721285888065","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8aa47ce-af8c-47e7-9654-870cd53229f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"99ffa553-f149-443a-98f5-68b10f77e085","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"35409ef0-891b-4c01-82e4-26b8e64f502d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"009124c4-8b5f-47d4-bd20-2c19271a2adc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7773115e-d3c3-4a0c-8801-d8ce63e0e5a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-06-27T11:47:38.892 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.891+0000 7fd89a028700 1 -- 172.21.15.195:0/122383451 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd8940c58b0 msgr2=0x7fd8940bd860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:38.892 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.891+0000 7fd89a028700 1 --2- 172.21.15.195:0/122383451 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd8940c58b0 0x7fd8940bd860 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd87c009e70 tx=0x7fd87c017000 comp rx=0 tx=0).stop 2022-06-27T11:47:38.893 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.891+0000 7fd89a028700 1 -- 172.21.15.195:0/122383451 shutdown_connections 2022-06-27T11:47:38.893 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.891+0000 7fd89a028700 1 --2- 172.21.15.195:0/122383451 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd8941572c0 0x7fd8940bdda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:38.893 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.891+0000 7fd89a028700 1 --2- 172.21.15.195:0/122383451 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd8940c4ea0 0x7fd89414d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:38.894 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.891+0000 7fd89a028700 1 --2- 172.21.15.195:0/122383451 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd8940c58b0 0x7fd8940bd860 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:38.894 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.891+0000 7fd89a028700 1 -- 172.21.15.195:0/122383451 >> 172.21.15.195:0/122383451 conn(0x7fd8940c98b0 msgr2=0x7fd89406f760 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:38.894 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.891+0000 7fd89a028700 1 -- 172.21.15.195:0/122383451 shutdown_connections 2022-06-27T11:47:38.895 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:38.891+0000 7fd89a028700 1 -- 172.21.15.195:0/122383451 wait complete. 2022-06-27T11:47:38.903 DEBUG:teuthology.misc:0 of 8 OSDs are up 2022-06-27T11:47:38.915 INFO:tasks.ceph.mgr.y.smithi195.stderr:2022-06-27T11:47:38.915+0000 7f7d32d42080 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2022-06-27T11:47:38.955 INFO:tasks.ceph.mgr.x.smithi202.stderr:2022-06-27T11:47:38.956+0000 7f4976dc5080 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2022-06-27T11:47:38.988 INFO:tasks.ceph.mgr.y.smithi195.stderr:2022-06-27T11:47:38.989+0000 7f7d32d42080 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2022-06-27T11:47:39.023 INFO:tasks.ceph.mgr.x.smithi202.stderr:2022-06-27T11:47:39.024+0000 7f4976dc5080 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2022-06-27T11:47:39.062 INFO:tasks.ceph.mgr.y.smithi195.stderr:2022-06-27T11:47:39.063+0000 7f7d32d42080 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2022-06-27T11:47:39.098 INFO:tasks.ceph.mgr.x.smithi202.stderr:2022-06-27T11:47:39.099+0000 7f4976dc5080 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2022-06-27T11:47:39.174 INFO:tasks.ceph.mgr.x.smithi202.stderr:2022-06-27T11:47:39.175+0000 7f4976dc5080 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2022-06-27T11:47:39.223 INFO:tasks.ceph.mgr.y.smithi195.stderr:2022-06-27T11:47:39.223+0000 7f7d32d42080 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2022-06-27T11:47:39.333 INFO:tasks.ceph.mgr.x.smithi202.stderr:2022-06-27T11:47:39.334+0000 7f4976dc5080 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2022-06-27T11:47:39.373 INFO:tasks.ceph.mgr.y.smithi195.stderr:2022-06-27T11:47:39.374+0000 7f7d32d42080 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2022-06-27T11:47:39.484 INFO:tasks.ceph.mgr.x.smithi202.stderr:2022-06-27T11:47:39.486+0000 7f4976dc5080 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2022-06-27T11:47:39.677 INFO:tasks.ceph.mgr.y.smithi195.stderr:2022-06-27T11:47:39.677+0000 7f7d32d42080 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2022-06-27T11:47:39.790 INFO:tasks.ceph.mgr.x.smithi202.stderr:2022-06-27T11:47:39.790+0000 7f4976dc5080 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2022-06-27T11:47:39.812 INFO:tasks.ceph.mgr.y.smithi195.stderr:2022-06-27T11:47:39.813+0000 7f7d32d42080 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2022-06-27T11:47:39.934 INFO:tasks.ceph.mgr.x.smithi202.stderr:2022-06-27T11:47:39.928+0000 7f4976dc5080 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2022-06-27T11:47:40.026 INFO:tasks.ceph.mgr.y.smithi195.stderr:2022-06-27T11:47:40.026+0000 7f7d32d42080 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2022-06-27T11:47:40.140 INFO:tasks.ceph.mgr.x.smithi202.stderr:2022-06-27T11:47:40.141+0000 7f4976dc5080 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2022-06-27T11:47:40.204 INFO:tasks.ceph.mgr.y.smithi195.stderr:2022-06-27T11:47:40.204+0000 7f7d32d42080 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2022-06-27T11:47:40.319 INFO:tasks.ceph.mgr.x.smithi202.stderr:2022-06-27T11:47:40.319+0000 7f4976dc5080 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2022-06-27T11:47:40.319 INFO:tasks.ceph.mgr.y.smithi195.stderr:2022-06-27T11:47:40.319+0000 7f7d32d42080 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2022-06-27T11:47:40.436 INFO:tasks.ceph.mgr.x.smithi202.stderr:2022-06-27T11:47:40.437+0000 7f4976dc5080 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2022-06-27T11:47:41.277 INFO:tasks.ceph.osd.2.smithi195.stderr:2022-06-27T11:47:41.277+0000 7f088d03f440 -1 osd.2 0 log_to_monitors true 2022-06-27T11:47:41.726 INFO:tasks.ceph.osd.0.smithi195.stderr:2022-06-27T11:47:41.726+0000 7f5d6d8df440 -1 osd.0 0 log_to_monitors true 2022-06-27T11:47:41.764 INFO:tasks.ceph.osd.1.smithi195.stderr:2022-06-27T11:47:41.764+0000 7fd3413b9440 -1 osd.1 0 log_to_monitors true 2022-06-27T11:47:41.769 INFO:tasks.ceph.osd.4.smithi202.stderr:2022-06-27T11:47:41.769+0000 7f4ab95b6440 -1 osd.4 0 log_to_monitors true 2022-06-27T11:47:41.784 INFO:tasks.ceph.osd.3.smithi195.stderr:2022-06-27T11:47:41.781+0000 7fad93c17440 -1 osd.3 0 log_to_monitors true 2022-06-27T11:47:41.787 INFO:tasks.ceph.osd.7.smithi202.stderr:2022-06-27T11:47:41.787+0000 7f4b0df49440 -1 osd.7 0 log_to_monitors true 2022-06-27T11:47:41.798 INFO:tasks.ceph.osd.6.smithi202.stderr:2022-06-27T11:47:41.799+0000 7f9e34c85440 -1 osd.6 0 log_to_monitors true 2022-06-27T11:47:41.802 INFO:tasks.ceph.osd.5.smithi202.stderr:2022-06-27T11:47:41.804+0000 7f4269d04440 -1 osd.5 0 log_to_monitors true 2022-06-27T11:47:43.751 INFO:tasks.ceph.osd.2.smithi195.stderr:2022-06-27T11:47:43.751+0000 7f0880dfc700 -1 osd.2 0 waiting for initial osdmap 2022-06-27T11:47:43.765 INFO:tasks.ceph.osd.2.smithi195.stderr:2022-06-27T11:47:43.766+0000 7f0878d7a700 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-06-27T11:47:43.779 INFO:tasks.ceph.osd.0.smithi195.stderr:2022-06-27T11:47:43.780+0000 7f5d6169c700 -1 osd.0 0 waiting for initial osdmap 2022-06-27T11:47:43.790 INFO:tasks.ceph.osd.0.smithi195.stderr:2022-06-27T11:47:43.791+0000 7f5d5961a700 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-06-27T11:47:43.809 INFO:tasks.ceph.osd.7.smithi202.stderr:2022-06-27T11:47:43.811+0000 7f4b01d06700 -1 osd.7 0 waiting for initial osdmap 2022-06-27T11:47:43.812 INFO:tasks.ceph.osd.5.smithi202.stderr:2022-06-27T11:47:43.814+0000 7f425dac1700 -1 osd.5 0 waiting for initial osdmap 2022-06-27T11:47:43.831 INFO:tasks.ceph.osd.5.smithi202.stderr:2022-06-27T11:47:43.833+0000 7f4255a3f700 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-06-27T11:47:43.832 INFO:tasks.ceph.osd.7.smithi202.stderr:2022-06-27T11:47:43.834+0000 7f4af9c84700 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-06-27T11:47:43.833 INFO:tasks.ceph.osd.3.smithi195.stderr:2022-06-27T11:47:43.834+0000 7fad879d4700 -1 osd.3 0 waiting for initial osdmap 2022-06-27T11:47:43.839 INFO:tasks.ceph.osd.6.smithi202.stderr:2022-06-27T11:47:43.840+0000 7f9e28a42700 -1 osd.6 0 waiting for initial osdmap 2022-06-27T11:47:43.841 INFO:tasks.ceph.osd.4.smithi202.stderr:2022-06-27T11:47:43.842+0000 7f4aad373700 -1 osd.4 0 waiting for initial osdmap 2022-06-27T11:47:43.843 INFO:tasks.ceph.osd.1.smithi195.stderr:2022-06-27T11:47:43.843+0000 7fd335176700 -1 osd.1 0 waiting for initial osdmap 2022-06-27T11:47:43.849 INFO:tasks.ceph.osd.3.smithi195.stderr:2022-06-27T11:47:43.850+0000 7fad7f952700 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-06-27T11:47:43.853 INFO:tasks.ceph.osd.1.smithi195.stderr:2022-06-27T11:47:43.854+0000 7fd32d0f4700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-06-27T11:47:43.853 INFO:tasks.ceph.osd.6.smithi202.stderr:2022-06-27T11:47:43.855+0000 7f9e209c0700 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-06-27T11:47:43.856 INFO:tasks.ceph.osd.4.smithi202.stderr:2022-06-27T11:47:43.858+0000 7f4aa52f1700 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-06-27T11:47:45.711 DEBUG:teuthology.orchestra.run.smithi195:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2022-06-27T11:47:45.835 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:45.835+0000 7f7148f42700 1 Processor -- start 2022-06-27T11:47:45.840 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:45.835+0000 7f7148f42700 1 -- start start 2022-06-27T11:47:45.851 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:45.836+0000 7f7148f42700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f71440c8fe0 0x7f7144156850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:45.851 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:45.836+0000 7f7148f42700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7144153360 con 0x7f7144150a80 2022-06-27T11:47:45.852 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:45.836+0000 7f7148f42700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7144153600 con 0x7f71441506b0 2022-06-27T11:47:45.852 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:45.836+0000 7f7148f42700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7144153fa0 con 0x7f71440c8fe0 2022-06-27T11:47:45.852 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:45.836+0000 7f7142d9d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f71440c8fe0 0x7f7144156850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:45.853 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:45.836+0000 7f7141d9b700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f7144150a80 0x7f71440c8090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:47032/0 (socket says 172.21.15.195:47032) 2022-06-27T11:47:45.853 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:45.836+0000 7f7141d9b700 1 -- 172.21.15.195:0/613527008 learned_addr learned my addr 172.21.15.195:0/613527008 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:47:45.854 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:45.836+0000 7f7142d9d700 1 -- 172.21.15.195:0/613527008 >> v1:172.21.15.202:6789/0 conn(0x7f71441506b0 legacy=0x7f71440c76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:45.854 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:45.836+0000 7f7142d9d700 1 -- 172.21.15.195:0/613527008 >> v1:172.21.15.195:6789/0 conn(0x7f7144150a80 legacy=0x7f71440c8090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:45.854 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:45.836+0000 7f714159a700 1 -- 172.21.15.195:0/613527008 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 347601393 0 0) 0x7f7144153600 con 0x7f71441506b0 2022-06-27T11:47:45.855 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:45.837+0000 7f7142d9d700 1 -- 172.21.15.195:0/613527008 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71441538e0 con 0x7f71440c8fe0 2022-06-27T11:47:45.855 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:45.837+0000 7f7142d9d700 1 --2- 172.21.15.195:0/613527008 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f71440c8fe0 0x7f7144156850 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f713c009e60 tx=0x7f713c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4aeb5f781a1fcc0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:45.855 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:45.837+0000 7f714159a700 1 -- 172.21.15.195:0/613527008 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f713c00e070 con 0x7f71440c8fe0 2022-06-27T11:47:45.855 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:45.837+0000 7f714159a700 1 -- 172.21.15.195:0/613527008 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f713c00bb30 con 0x7f71440c8fe0 2022-06-27T11:47:45.856 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:45.837+0000 7f714159a700 1 -- 172.21.15.195:0/613527008 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f713c0038a0 con 0x7f71440c8fe0 2022-06-27T11:47:45.857 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:45.837+0000 7f7148f42700 1 -- 172.21.15.195:0/613527008 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f71440c8fe0 msgr2=0x7f7144156850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:45.857 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:45.837+0000 7f7148f42700 1 --2- 172.21.15.195:0/613527008 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f71440c8fe0 0x7f7144156850 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f713c009e60 tx=0x7f713c009780 comp rx=0 tx=0).stop 2022-06-27T11:47:45.857 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:45.837+0000 7f7148f42700 1 -- 172.21.15.195:0/613527008 shutdown_connections 2022-06-27T11:47:45.858 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:45.837+0000 7f7148f42700 1 --2- 172.21.15.195:0/613527008 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f71440c8fe0 0x7f7144156850 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:45.858 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:45.837+0000 7f7148f42700 1 -- 172.21.15.195:0/613527008 >> 172.21.15.195:0/613527008 conn(0x7f71440c98b0 msgr2=0x7f71440c9cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:45.859 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:45.837+0000 7f7148f42700 1 -- 172.21.15.195:0/613527008 shutdown_connections 2022-06-27T11:47:45.859 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:45.837+0000 7f7148f42700 1 -- 172.21.15.195:0/613527008 wait complete. 2022-06-27T11:47:45.859 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:45.839+0000 7f7148f42700 1 Processor -- start 2022-06-27T11:47:45.860 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:45.839+0000 7f7148f42700 1 -- start start 2022-06-27T11:47:45.860 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:45.839+0000 7f7148f42700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f71440c8fe0 0x7f714416a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:45.860 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:45.839+0000 7f7148f42700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f71441506b0 0x7f71441663c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:45.860 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:45.839+0000 7f7148f42700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7144150a80 0x7f71440c4b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:45.861 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:45.839+0000 7f7148f42700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7144169fe0 con 0x7f71441506b0 2022-06-27T11:47:45.861 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:45.839+0000 7f7148f42700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7144155fa0 con 0x7f7144150a80 2022-06-27T11:47:45.862 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:45.839+0000 7f7148f42700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7144156240 con 0x7f71440c8fe0 2022-06-27T11:47:45.862 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:45.839+0000 7f7141d9b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f71441506b0 0x7f71441663c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:45.863 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:45.839+0000 7f7141d9b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f71441506b0 0x7f71441663c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:40850/0 (socket says 172.21.15.195:40850) 2022-06-27T11:47:45.863 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:45.839+0000 7f7141d9b700 1 -- 172.21.15.195:0/3864588825 learned_addr learned my addr 172.21.15.195:0/3864588825 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:47:45.863 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:45.839+0000 7f7142d9d700 1 --2- 172.21.15.195:0/3864588825 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7144150a80 0x7f71440c4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:45.863 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:45.839+0000 7f714259c700 1 --2- 172.21.15.195:0/3864588825 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f71440c8fe0 0x7f714416a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:45.864 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:45.840+0000 7f7141d9b700 1 -- 172.21.15.195:0/3864588825 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f71440c8fe0 msgr2=0x7f714416a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:45.864 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:45.840+0000 7f7141d9b700 1 --2- 172.21.15.195:0/3864588825 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f71440c8fe0 0x7f714416a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:45.864 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:45.840+0000 7f7141d9b700 1 -- 172.21.15.195:0/3864588825 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7144150a80 msgr2=0x7f71440c4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:45.864 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:45.840+0000 7f7141d9b700 1 --2- 172.21.15.195:0/3864588825 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7144150a80 0x7f71440c4b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:45.865 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:45.840+0000 7f7141d9b700 1 -- 172.21.15.195:0/3864588825 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f713c009a20 con 0x7f71441506b0 2022-06-27T11:47:45.865 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:45.840+0000 7f7141d9b700 1 --2- 172.21.15.195:0/3864588825 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f71441506b0 0x7f71441663c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f713400c510 tx=0x7f7134007d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:45.865 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:45.840+0000 7f71327fc700 1 -- 172.21.15.195:0/3864588825 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7134007430 con 0x7f71441506b0 2022-06-27T11:47:45.865 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:45.840+0000 7f71327fc700 1 -- 172.21.15.195:0/3864588825 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71340075b0 con 0x7f71441506b0 2022-06-27T11:47:45.866 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:45.840+0000 7f7148f42700 1 -- 172.21.15.195:0/3864588825 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7144156520 con 0x7f71441506b0 2022-06-27T11:47:45.866 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:45.840+0000 7f71327fc700 1 -- 172.21.15.195:0/3864588825 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7134004940 con 0x7f71441506b0 2022-06-27T11:47:45.866 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:45.840+0000 7f7148f42700 1 -- 172.21.15.195:0/3864588825 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7144169c70 con 0x7f71441506b0 2022-06-27T11:47:45.867 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:45.841+0000 7f7148f42700 1 -- 172.21.15.195:0/3864588825 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f714416a190 con 0x7f71441506b0 2022-06-27T11:47:45.867 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:45.841+0000 7f71327fc700 1 -- 172.21.15.195:0/3864588825 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 4) v1 ==== 81876+0+0 (secure 0 0 0) 0x7f713401a020 con 0x7f71441506b0 2022-06-27T11:47:45.867 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:45.842+0000 7f71327fc700 1 --2- 172.21.15.195:0/3864588825 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7120063ce0 0x7f7120066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:45.868 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:45.842+0000 7f71327fc700 1 -- 172.21.15.195:0/3864588825 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 4467+0+0 (secure 0 0 0) 0x7f713407fcf0 con 0x7f71441506b0 2022-06-27T11:47:45.868 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:45.842+0000 7f714259c700 1 --2- 172.21.15.195:0/3864588825 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7120063ce0 0x7f7120066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:45.868 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:45.842+0000 7f714259c700 1 --2- 172.21.15.195:0/3864588825 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7120063ce0 0x7f7120066190 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7138001900 tx=0x7f7138009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:45.868 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:45.846+0000 7f71327fc700 1 -- 172.21.15.195:0/3864588825 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f714416a190 con 0x7f71441506b0 2022-06-27T11:47:46.026 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:46.027+0000 7f7148f42700 1 -- 172.21.15.195:0/3864588825 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f714416a190 con 0x7f71441506b0 2022-06-27T11:47:46.028 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:46.027+0000 7f71327fc700 1 -- 172.21.15.195:0/3864588825 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) v1 ==== 74+0+10605 (secure 0 0 0) 0x7f714416a190 con 0x7f71441506b0 2022-06-27T11:47:46.028 INFO:teuthology.misc.health.smithi195.stdout: 2022-06-27T11:47:46.029 INFO:teuthology.misc.health.smithi195.stdout:{"epoch":14,"fsid":"81b3a1d3-6a2b-47e8-8d69-7027cfeae7af","created":"2022-06-27T11:47:35.094715+0000","modified":"2022-06-27T11:47:45.779306+0000","last_up_change":"2022-06-27T11:47:44.777459+0000","last_in_change":"2022-06-27T11:47:37.596818+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"c64bc51f-d4f3-4bc2-806b-da0dc78d51b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6813","nonce":94217},{"type":"v1","addr":"172.21.15.195:6815","nonce":94217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6817","nonce":94217},{"type":"v1","addr":"172.21.15.195:6819","nonce":94217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":94217},{"type":"v1","addr":"172.21.15.195:6823","nonce":94217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6820","nonce":94217},{"type":"v1","addr":"172.21.15.195:6821","nonce":94217}]},"public_addr":"172.21.15.195:6815/94217","cluster_addr":"172.21.15.195:6819/94217","heartbeat_back_addr":"172.21.15.195:6823/94217","heartbeat_front_addr":"172.21.15.195:6821/94217","state":["exists","up"]},{"osd":1,"uuid":"bea71cdf-aa98-493f-9207-66d886a3e30e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":94223},{"type":"v1","addr":"172.21.15.195:6809","nonce":94223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":94223},{"type":"v1","addr":"172.21.15.195:6811","nonce":94223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6816","nonce":94223},{"type":"v1","addr":"172.21.15.195:6818","nonce":94223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":94223},{"type":"v1","addr":"172.21.15.195:6814","nonce":94223}]},"public_addr":"172.21.15.195:6809/94223","cluster_addr":"172.21.15.195:6811/94223","heartbeat_back_addr":"172.21.15.195:6818/94223","heartbeat_front_addr":"172.21.15.195:6814/94223","state":["exists","up"]},{"osd":2,"uuid":"bf6c87a0-a8ea-497a-9ce1-721285888065","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":94221},{"type":"v1","addr":"172.21.15.195:6801","nonce":94221}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":94221},{"type":"v1","addr":"172.21.15.195:6803","nonce":94221}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":94221},{"type":"v1","addr":"172.21.15.195:6807","nonce":94221}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":94221},{"type":"v1","addr":"172.21.15.195:6805","nonce":94221}]},"public_addr":"172.21.15.195:6801/94221","cluster_addr":"172.21.15.195:6803/94221","heartbeat_back_addr":"172.21.15.195:6807/94221","heartbeat_front_addr":"172.21.15.195:6805/94221","state":["exists","up"]},{"osd":3,"uuid":"e8aa47ce-af8c-47e7-9654-870cd53229f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":94232},{"type":"v1","addr":"172.21.15.195:6825","nonce":94232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":94232},{"type":"v1","addr":"172.21.15.195:6827","nonce":94232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":94232},{"type":"v1","addr":"172.21.15.195:6831","nonce":94232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":94232},{"type":"v1","addr":"172.21.15.195:6829","nonce":94232}]},"public_addr":"172.21.15.195:6825/94232","cluster_addr":"172.21.15.195:6827/94232","heartbeat_back_addr":"172.21.15.195:6831/94232","heartbeat_front_addr":"172.21.15.195:6829/94232","state":["exists","up"]},{"osd":4,"uuid":"99ffa553-f149-443a-98f5-68b10f77e085","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6800","nonce":94308},{"type":"v1","addr":"172.21.15.202:6801","nonce":94308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6802","nonce":94308},{"type":"v1","addr":"172.21.15.202:6803","nonce":94308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6806","nonce":94308},{"type":"v1","addr":"172.21.15.202:6807","nonce":94308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6804","nonce":94308},{"type":"v1","addr":"172.21.15.202:6805","nonce":94308}]},"public_addr":"172.21.15.202:6801/94308","cluster_addr":"172.21.15.202:6803/94308","heartbeat_back_addr":"172.21.15.202:6807/94308","heartbeat_front_addr":"172.21.15.202:6805/94308","state":["exists","up"]},{"osd":5,"uuid":"35409ef0-891b-4c01-82e4-26b8e64f502d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6824","nonce":94325},{"type":"v1","addr":"172.21.15.202:6825","nonce":94325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6826","nonce":94325},{"type":"v1","addr":"172.21.15.202:6827","nonce":94325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6830","nonce":94325},{"type":"v1","addr":"172.21.15.202:6831","nonce":94325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6828","nonce":94325},{"type":"v1","addr":"172.21.15.202:6829","nonce":94325}]},"public_addr":"172.21.15.202:6825/94325","cluster_addr":"172.21.15.202:6827/94325","heartbeat_back_addr":"172.21.15.202:6831/94325","heartbeat_front_addr":"172.21.15.202:6829/94325","state":["exists","up"]},{"osd":6,"uuid":"009124c4-8b5f-47d4-bd20-2c19271a2adc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6809","nonce":94309},{"type":"v1","addr":"172.21.15.202:6811","nonce":94309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6813","nonce":94309},{"type":"v1","addr":"172.21.15.202:6815","nonce":94309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6821","nonce":94309},{"type":"v1","addr":"172.21.15.202:6823","nonce":94309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6817","nonce":94309},{"type":"v1","addr":"172.21.15.202:6819","nonce":94309}]},"public_addr":"172.21.15.202:6811/94309","cluster_addr":"172.21.15.202:6815/94309","heartbeat_back_addr":"172.21.15.202:6823/94309","heartbeat_front_addr":"172.21.15.202:6819/94309","state":["exists","up"]},{"osd":7,"uuid":"7773115e-d3c3-4a0c-8801-d8ce63e0e5a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6808","nonce":94310},{"type":"v1","addr":"172.21.15.202:6810","nonce":94310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6812","nonce":94310},{"type":"v1","addr":"172.21.15.202:6814","nonce":94310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6820","nonce":94310},{"type":"v1","addr":"172.21.15.202:6822","nonce":94310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6816","nonce":94310},{"type":"v1","addr":"172.21.15.202:6818","nonce":94310}]},"public_addr":"172.21.15.202:6810/94310","cluster_addr":"172.21.15.202:6814/94310","heartbeat_back_addr":"172.21.15.202:6822/94310","heartbeat_front_addr":"172.21.15.202:6818/94310","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.776182+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.806922+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.237997+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.784371+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.820332+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:43.841414+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.758071+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-06-27T11:47:46.031 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:46.029+0000 7f7148f42700 1 -- 172.21.15.195:0/3864588825 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7120063ce0 msgr2=0x7f7120066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:46.032 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:46.029+0000 7f7148f42700 1 --2- 172.21.15.195:0/3864588825 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7120063ce0 0x7f7120066190 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7138001900 tx=0x7f7138009040 comp rx=0 tx=0).stop 2022-06-27T11:47:46.032 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:46.029+0000 7f7148f42700 1 -- 172.21.15.195:0/3864588825 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f71441506b0 msgr2=0x7f71441663c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:46.033 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:46.029+0000 7f7148f42700 1 --2- 172.21.15.195:0/3864588825 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f71441506b0 0x7f71441663c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f713400c510 tx=0x7f7134007d80 comp rx=0 tx=0).stop 2022-06-27T11:47:46.034 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:46.029+0000 7f7148f42700 1 -- 172.21.15.195:0/3864588825 shutdown_connections 2022-06-27T11:47:46.034 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:46.029+0000 7f7148f42700 1 --2- 172.21.15.195:0/3864588825 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7120063ce0 0x7f7120066190 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:46.035 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:46.029+0000 7f7148f42700 1 --2- 172.21.15.195:0/3864588825 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7144150a80 0x7f71440c4b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:46.035 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:46.029+0000 7f7148f42700 1 --2- 172.21.15.195:0/3864588825 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f71440c8fe0 0x7f714416a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:46.035 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:46.029+0000 7f7148f42700 1 --2- 172.21.15.195:0/3864588825 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f71441506b0 0x7f71441663c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:46.036 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:46.029+0000 7f7148f42700 1 -- 172.21.15.195:0/3864588825 >> 172.21.15.195:0/3864588825 conn(0x7f71440c98b0 msgr2=0x7f71440c3640 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:46.036 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:46.030+0000 7f7148f42700 1 -- 172.21.15.195:0/3864588825 shutdown_connections 2022-06-27T11:47:46.036 INFO:teuthology.misc.health.smithi195.stderr:2022-06-27T11:47:46.030+0000 7f7148f42700 1 -- 172.21.15.195:0/3864588825 wait complete. 2022-06-27T11:47:46.040 DEBUG:teuthology.misc:8 of 8 OSDs are up 2022-06-27T11:47:46.040 INFO:tasks.ceph:Creating RBD pool 2022-06-27T11:47:46.041 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd pool create rbd 8 2022-06-27T11:47:46.166 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.161+0000 7f30e7b26700 1 Processor -- start 2022-06-27T11:47:46.166 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.161+0000 7f30e7b26700 1 -- start start 2022-06-27T11:47:46.166 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.162+0000 7f30e7b26700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f30e00c4160 0x7f30e0156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:46.166 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.162+0000 7f30e7b26700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f30e00c39e0 0x7f30e01535a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:46.167 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.162+0000 7f30e7b26700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30e00c1a20 con 0x7f30e006f640 2022-06-27T11:47:46.167 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.162+0000 7f30e7b26700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30e00c1130 con 0x7f30e00c39e0 2022-06-27T11:47:46.167 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.162+0000 7f30e7b26700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30e006ea90 con 0x7f30e00c4160 2022-06-27T11:47:46.167 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.162+0000 7f30e60c3700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f30e00c39e0 0x7f30e01535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:46.168 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.162+0000 7f30e60c3700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f30e00c39e0 0x7f30e01535a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:40876/0 (socket says 172.21.15.195:40876) 2022-06-27T11:47:46.168 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.162+0000 7f30e60c3700 1 -- 172.21.15.195:0/434466130 learned_addr learned my addr 172.21.15.195:0/434466130 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:47:46.168 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.162+0000 7f30e60c3700 1 -- 172.21.15.195:0/434466130 >> v2:172.21.15.202:3300/0 conn(0x7f30e00c4160 msgr2=0x7f30e0156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:47:46.168 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.162+0000 7f30e60c3700 1 --2- 172.21.15.195:0/434466130 >> v2:172.21.15.202:3300/0 conn(0x7f30e00c4160 0x7f30e0156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:46.168 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.162+0000 7f30e60c3700 1 -- 172.21.15.195:0/434466130 >> v1:172.21.15.202:6789/0 conn(0x7f30e006f640 legacy=0x7f30e00c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:47:46.169 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.162+0000 7f30e60c3700 1 -- 172.21.15.195:0/434466130 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30e006ed70 con 0x7f30e00c39e0 2022-06-27T11:47:46.169 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.162+0000 7f30e60c3700 1 --2- 172.21.15.195:0/434466130 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f30e00c39e0 0x7f30e01535a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f30e006dd70 tx=0x7f30d8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ec9b2ae1cbd532d server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:46.169 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.162+0000 7f30e48c0700 1 -- 172.21.15.195:0/434466130 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30d800e070 con 0x7f30e00c39e0 2022-06-27T11:47:46.169 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.162+0000 7f30e48c0700 1 -- 172.21.15.195:0/434466130 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30d800baa0 con 0x7f30e00c39e0 2022-06-27T11:47:46.170 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.162+0000 7f30e48c0700 1 -- 172.21.15.195:0/434466130 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30d80092e0 con 0x7f30e00c39e0 2022-06-27T11:47:46.170 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.163+0000 7f30e7b26700 1 -- 172.21.15.195:0/434466130 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f30e00c39e0 msgr2=0x7f30e01535a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:46.170 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.163+0000 7f30e7b26700 1 --2- 172.21.15.195:0/434466130 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f30e00c39e0 0x7f30e01535a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f30e006dd70 tx=0x7f30d8009840 comp rx=0 tx=0).stop 2022-06-27T11:47:46.171 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.163+0000 7f30e7b26700 1 -- 172.21.15.195:0/434466130 shutdown_connections 2022-06-27T11:47:46.171 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.163+0000 7f30e7b26700 1 --2- 172.21.15.195:0/434466130 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f30e00c39e0 0x7f30e01535a0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:46.171 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.163+0000 7f30e7b26700 1 --2- 172.21.15.195:0/434466130 >> v2:172.21.15.202:3300/0 conn(0x7f30e00c4160 0x7f30e0156ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:46.172 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.163+0000 7f30e7b26700 1 -- 172.21.15.195:0/434466130 >> 172.21.15.195:0/434466130 conn(0x7f30e00c6350 msgr2=0x7f30e00c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:46.172 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.163+0000 7f30e7b26700 1 -- 172.21.15.195:0/434466130 shutdown_connections 2022-06-27T11:47:46.172 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.163+0000 7f30e7b26700 1 -- 172.21.15.195:0/434466130 wait complete. 2022-06-27T11:47:46.173 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.164+0000 7f30e7b26700 1 Processor -- start 2022-06-27T11:47:46.173 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.164+0000 7f30e7b26700 1 -- start start 2022-06-27T11:47:46.174 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.165+0000 7f30e7b26700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f30e006f640 0x7f30e0166f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:46.174 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.165+0000 7f30e7b26700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f30e00c39e0 0x7f30e00bc0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:46.174 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.165+0000 7f30e7b26700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f30e00c4160 0x7f30e00ba830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:46.175 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.165+0000 7f30e7b26700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30e014e790 con 0x7f30e00c39e0 2022-06-27T11:47:46.175 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.165+0000 7f30e7b26700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30e00bbc20 con 0x7f30e00c4160 2022-06-27T11:47:46.175 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.165+0000 7f30e7b26700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30e00bc5f0 con 0x7f30e006f640 2022-06-27T11:47:46.175 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.165+0000 7f30e50c1700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f30e00c39e0 0x7f30e00bc0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:46.175 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.165+0000 7f30e50c1700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f30e00c39e0 0x7f30e00bc0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:40880/0 (socket says 172.21.15.195:40880) 2022-06-27T11:47:46.176 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.165+0000 7f30e50c1700 1 -- 172.21.15.195:0/636705587 learned_addr learned my addr 172.21.15.195:0/636705587 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:47:46.176 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.165+0000 7f30e60c3700 1 --2- 172.21.15.195:0/636705587 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f30e00c4160 0x7f30e00ba830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:46.176 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.165+0000 7f30e58c2700 1 --2- 172.21.15.195:0/636705587 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f30e006f640 0x7f30e0166f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:46.176 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.165+0000 7f30e50c1700 1 -- 172.21.15.195:0/636705587 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f30e006f640 msgr2=0x7f30e0166f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:46.177 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.165+0000 7f30e50c1700 1 --2- 172.21.15.195:0/636705587 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f30e006f640 0x7f30e0166f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:46.177 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.165+0000 7f30e50c1700 1 -- 172.21.15.195:0/636705587 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f30e00c4160 msgr2=0x7f30e00ba830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:46.177 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.165+0000 7f30e50c1700 1 --2- 172.21.15.195:0/636705587 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f30e00c4160 0x7f30e00ba830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:46.178 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.165+0000 7f30e50c1700 1 -- 172.21.15.195:0/636705587 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30d8009ae0 con 0x7f30e00c39e0 2022-06-27T11:47:46.178 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.165+0000 7f30e50c1700 1 --2- 172.21.15.195:0/636705587 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f30e00c39e0 0x7f30e00bc0b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f30d000b540 tx=0x7f30d00090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:46.179 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.166+0000 7f30cdffb700 1 -- 172.21.15.195:0/636705587 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30d0011630 con 0x7f30e00c39e0 2022-06-27T11:47:46.179 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.166+0000 7f30cdffb700 1 -- 172.21.15.195:0/636705587 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30d0011790 con 0x7f30e00c39e0 2022-06-27T11:47:46.180 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.166+0000 7f30e7b26700 1 -- 172.21.15.195:0/636705587 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30e014ecc0 con 0x7f30e00c39e0 2022-06-27T11:47:46.180 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.166+0000 7f30cdffb700 1 -- 172.21.15.195:0/636705587 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30d0011b50 con 0x7f30e00c39e0 2022-06-27T11:47:46.180 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.166+0000 7f30e7b26700 1 -- 172.21.15.195:0/636705587 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30e00bb5d0 con 0x7f30e00c39e0 2022-06-27T11:47:46.183 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.167+0000 7f30e7b26700 1 -- 172.21.15.195:0/636705587 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30e00722c0 con 0x7f30e00c39e0 2022-06-27T11:47:46.184 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.167+0000 7f30cdffb700 1 -- 172.21.15.195:0/636705587 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 4) v1 ==== 81876+0+0 (secure 0 0 0) 0x7f30d0021070 con 0x7f30e00c39e0 2022-06-27T11:47:46.184 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.167+0000 7f30cdffb700 1 --2- 172.21.15.195:0/636705587 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f30c0063cb0 0x7f30c0066160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:46.185 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.167+0000 7f30cdffb700 1 -- 172.21.15.195:0/636705587 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 4467+0+0 (secure 0 0 0) 0x7f30d007ffb0 con 0x7f30e00c39e0 2022-06-27T11:47:46.185 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.167+0000 7f30e58c2700 1 --2- 172.21.15.195:0/636705587 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f30c0063cb0 0x7f30c0066160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:46.186 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.168+0000 7f30e58c2700 1 --2- 172.21.15.195:0/636705587 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f30c0063cb0 0x7f30c0066160 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f30dc001890 tx=0x7f30dc009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:46.186 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.172+0000 7f30cdffb700 1 -- 172.21.15.195:0/636705587 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f30d004f220 con 0x7f30e00c39e0 2022-06-27T11:47:46.349 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.349+0000 7f30e7b26700 1 -- 172.21.15.195:0/636705587 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f30e00bbd60 con 0x7f30e00c39e0 2022-06-27T11:47:46.782 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.783+0000 7f30cdffb700 1 -- 172.21.15.195:0/636705587 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v15) v1 ==== 109+0+0 (secure 0 0 0) 0x7f30d00150a0 con 0x7f30e00c39e0 2022-06-27T11:47:46.785 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.785+0000 7f30e7b26700 1 -- 172.21.15.195:0/636705587 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f30c0063cb0 msgr2=0x7f30c0066160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:46.786 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.785+0000 7f30e7b26700 1 --2- 172.21.15.195:0/636705587 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f30c0063cb0 0x7f30c0066160 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f30dc001890 tx=0x7f30dc009040 comp rx=0 tx=0).stop 2022-06-27T11:47:46.786 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.785+0000 7f30e7b26700 1 -- 172.21.15.195:0/636705587 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f30e00c39e0 msgr2=0x7f30e00bc0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:46.786 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.785+0000 7f30e7b26700 1 --2- 172.21.15.195:0/636705587 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f30e00c39e0 0x7f30e00bc0b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f30d000b540 tx=0x7f30d00090f0 comp rx=0 tx=0).stop 2022-06-27T11:47:46.786 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.785+0000 7f30e7b26700 1 -- 172.21.15.195:0/636705587 shutdown_connections 2022-06-27T11:47:46.787 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.785+0000 7f30e7b26700 1 --2- 172.21.15.195:0/636705587 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f30c0063cb0 0x7f30c0066160 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:46.787 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.785+0000 7f30e7b26700 1 --2- 172.21.15.195:0/636705587 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f30e00c4160 0x7f30e00ba830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:46.787 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.785+0000 7f30e7b26700 1 --2- 172.21.15.195:0/636705587 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f30e006f640 0x7f30e0166f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:46.787 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.785+0000 7f30e7b26700 1 --2- 172.21.15.195:0/636705587 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f30e00c39e0 0x7f30e00bc0b0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:46.788 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.785+0000 7f30e7b26700 1 -- 172.21.15.195:0/636705587 >> 172.21.15.195:0/636705587 conn(0x7f30e00c6350 msgr2=0x7f30e0152900 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:46.788 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.785+0000 7f30e7b26700 1 -- 172.21.15.195:0/636705587 shutdown_connections 2022-06-27T11:47:46.788 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.785+0000 7f30e7b26700 1 -- 172.21.15.195:0/636705587 wait complete. 2022-06-27T11:47:46.788 INFO:teuthology.orchestra.run.smithi195.stderr:pool 'rbd' created 2022-06-27T11:47:46.797 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2022-06-27T11:47:46.929 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.921+0000 7fba8e19a700 1 Processor -- start 2022-06-27T11:47:46.929 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.922+0000 7fba8e19a700 1 -- start start 2022-06-27T11:47:46.930 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.922+0000 7fba8e19a700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fba88150580 0x7fba88156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:46.930 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.922+0000 7fba8e19a700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fba8806f620 0x7fba8806d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:46.930 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.922+0000 7fba8e19a700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba880c76b0 con 0x7fba88150950 2022-06-27T11:47:46.931 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.922+0000 7fba8e19a700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba88157430 con 0x7fba88150580 2022-06-27T11:47:46.931 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.922+0000 7fba8e19a700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba880c8250 con 0x7fba8806f620 2022-06-27T11:47:46.931 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.922+0000 7fba877fe700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fba88150580 0x7fba88156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:46.931 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.922+0000 7fba86ffd700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fba88150950 0x7fba880c7b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:47084/0 (socket says 172.21.15.195:47084) 2022-06-27T11:47:46.931 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.922+0000 7fba86ffd700 1 -- 172.21.15.195:0/1443696159 learned_addr learned my addr 172.21.15.195:0/1443696159 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:47:46.932 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.923+0000 7fba87fff700 1 --2- 172.21.15.195:0/1443696159 >> v2:172.21.15.202:3300/0 conn(0x7fba8806f620 0x7fba8806d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:46.932 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.923+0000 7fba877fe700 1 -- 172.21.15.195:0/1443696159 >> v2:172.21.15.202:3300/0 conn(0x7fba8806f620 msgr2=0x7fba8806d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:47:46.932 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.923+0000 7fba877fe700 1 --2- 172.21.15.195:0/1443696159 >> v2:172.21.15.202:3300/0 conn(0x7fba8806f620 0x7fba8806d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:46.932 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.923+0000 7fba877fe700 1 -- 172.21.15.195:0/1443696159 >> v1:172.21.15.195:6789/0 conn(0x7fba88150950 legacy=0x7fba880c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:46.933 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.923+0000 7fba867fc700 1 -- 172.21.15.195:0/1443696159 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1221666938 0 0) 0x7fba880c76b0 con 0x7fba88150950 2022-06-27T11:47:46.933 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.923+0000 7fba877fe700 1 -- 172.21.15.195:0/1443696159 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba880bad70 con 0x7fba88150580 2022-06-27T11:47:46.933 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.923+0000 7fba87fff700 1 --2- 172.21.15.195:0/1443696159 >> v2:172.21.15.202:3300/0 conn(0x7fba8806f620 0x7fba8806d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T11:47:46.933 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.923+0000 7fba877fe700 1 --2- 172.21.15.195:0/1443696159 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fba88150580 0x7fba88156ef0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fba880c4f70 tx=0x7fba7c009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ff36ea4edfe25c6 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:46.934 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.923+0000 7fba867fc700 1 -- 172.21.15.195:0/1443696159 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba7c011630 con 0x7fba88150580 2022-06-27T11:47:46.934 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.923+0000 7fba867fc700 1 -- 172.21.15.195:0/1443696159 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba7c011790 con 0x7fba88150580 2022-06-27T11:47:46.934 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.923+0000 7fba867fc700 1 -- 172.21.15.195:0/1443696159 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba7c011b30 con 0x7fba88150580 2022-06-27T11:47:46.934 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.923+0000 7fba8e19a700 1 -- 172.21.15.195:0/1443696159 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fba88150580 msgr2=0x7fba88156ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:46.935 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.923+0000 7fba8e19a700 1 --2- 172.21.15.195:0/1443696159 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fba88150580 0x7fba88156ef0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fba880c4f70 tx=0x7fba7c009060 comp rx=0 tx=0).stop 2022-06-27T11:47:46.935 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.924+0000 7fba8e19a700 1 -- 172.21.15.195:0/1443696159 shutdown_connections 2022-06-27T11:47:46.935 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.924+0000 7fba8e19a700 1 --2- 172.21.15.195:0/1443696159 >> v2:172.21.15.202:3300/0 conn(0x7fba8806f620 0x7fba8806d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:46.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.924+0000 7fba8e19a700 1 --2- 172.21.15.195:0/1443696159 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fba88150580 0x7fba88156ef0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:46.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.924+0000 7fba8e19a700 1 -- 172.21.15.195:0/1443696159 >> 172.21.15.195:0/1443696159 conn(0x7fba88063af0 msgr2=0x7fba88063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:46.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.924+0000 7fba8e19a700 1 -- 172.21.15.195:0/1443696159 shutdown_connections 2022-06-27T11:47:46.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.924+0000 7fba8e19a700 1 -- 172.21.15.195:0/1443696159 wait complete. 2022-06-27T11:47:46.937 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.925+0000 7fba8e19a700 1 Processor -- start 2022-06-27T11:47:46.937 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.925+0000 7fba8e19a700 1 -- start start 2022-06-27T11:47:46.937 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.926+0000 7fba8e19a700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fba8806f620 0x7fba88167180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:46.937 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.926+0000 7fba8e19a700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fba88150580 0x7fba88158ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:46.938 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.926+0000 7fba8e19a700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fba88150950 0x7fba880caad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:46.938 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.926+0000 7fba8e19a700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba88156360 con 0x7fba8806f620 2022-06-27T11:47:46.938 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.926+0000 7fba8e19a700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba8816a3a0 con 0x7fba88150580 2022-06-27T11:47:46.939 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.926+0000 7fba8e19a700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba8816a680 con 0x7fba88150950 2022-06-27T11:47:46.939 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.926+0000 7fba877fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fba8806f620 0x7fba88167180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:46.939 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.926+0000 7fba877fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fba8806f620 0x7fba88167180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:40900/0 (socket says 172.21.15.195:40900) 2022-06-27T11:47:46.940 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.926+0000 7fba877fe700 1 -- 172.21.15.195:0/3328723643 learned_addr learned my addr 172.21.15.195:0/3328723643 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:47:46.940 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.926+0000 7fba86ffd700 1 --2- 172.21.15.195:0/3328723643 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fba88150580 0x7fba88158ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:46.940 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.926+0000 7fba87fff700 1 --2- 172.21.15.195:0/3328723643 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fba88150950 0x7fba880caad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:46.940 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.926+0000 7fba877fe700 1 -- 172.21.15.195:0/3328723643 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fba88150950 msgr2=0x7fba880caad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:46.941 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.926+0000 7fba877fe700 1 --2- 172.21.15.195:0/3328723643 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fba88150950 0x7fba880caad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:46.941 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.926+0000 7fba877fe700 1 -- 172.21.15.195:0/3328723643 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fba88150580 msgr2=0x7fba88158ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:46.941 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.926+0000 7fba877fe700 1 --2- 172.21.15.195:0/3328723643 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fba88150580 0x7fba88158ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:46.941 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.926+0000 7fba877fe700 1 -- 172.21.15.195:0/3328723643 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba7c0092e0 con 0x7fba8806f620 2022-06-27T11:47:46.942 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.926+0000 7fba877fe700 1 --2- 172.21.15.195:0/3328723643 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fba8806f620 0x7fba88167180 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fba7c009ec0 tx=0x7fba7c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:46.942 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.926+0000 7fba6f7fe700 1 -- 172.21.15.195:0/3328723643 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba7c000bc0 con 0x7fba8806f620 2022-06-27T11:47:46.943 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.926+0000 7fba6f7fe700 1 -- 172.21.15.195:0/3328723643 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba7c011900 con 0x7fba8806f620 2022-06-27T11:47:46.943 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.926+0000 7fba8e19a700 1 -- 172.21.15.195:0/3328723643 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba8816a960 con 0x7fba8806f620 2022-06-27T11:47:46.943 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.926+0000 7fba6f7fe700 1 -- 172.21.15.195:0/3328723643 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba7c000d20 con 0x7fba8806f620 2022-06-27T11:47:46.943 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.926+0000 7fba8e19a700 1 -- 172.21.15.195:0/3328723643 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba88166980 con 0x7fba8806f620 2022-06-27T11:47:46.944 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.927+0000 7fba8e19a700 1 -- 172.21.15.195:0/3328723643 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba880722c0 con 0x7fba8806f620 2022-06-27T11:47:46.944 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.928+0000 7fba6f7fe700 1 -- 172.21.15.195:0/3328723643 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 4) v1 ==== 81876+0+0 (secure 0 0 0) 0x7fba7c02c030 con 0x7fba8806f620 2022-06-27T11:47:46.944 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.928+0000 7fba6f7fe700 1 --2- 172.21.15.195:0/3328723643 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fba68063cb0 0x7fba68066160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:46.944 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.928+0000 7fba86ffd700 1 --2- 172.21.15.195:0/3328723643 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fba68063cb0 0x7fba68066160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:46.945 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.928+0000 7fba6f7fe700 1 -- 172.21.15.195:0/3328723643 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 5184+0+0 (secure 0 0 0) 0x7fba7c081a60 con 0x7fba8806f620 2022-06-27T11:47:46.945 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.929+0000 7fba86ffd700 1 --2- 172.21.15.195:0/3328723643 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fba68063cb0 0x7fba68066160 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fba880c4d10 tx=0x7fba78007bf0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:46.946 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:46.933+0000 7fba6f7fe700 1 -- 172.21.15.195:0/3328723643 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fba7c050a10 con 0x7fba8806f620 2022-06-27T11:47:47.113 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.113+0000 7fba8e19a700 1 -- 172.21.15.195:0/3328723643 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fba8806a950 con 0x7fba8806f620 2022-06-27T11:47:47.784 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.784+0000 7fba6f7fe700 1 -- 172.21.15.195:0/3328723643 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v16) v1 ==== 173+0+0 (secure 0 0 0) 0x7fba7c02a0c0 con 0x7fba8806f620 2022-06-27T11:47:47.788 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.788+0000 7fba8e19a700 1 -- 172.21.15.195:0/3328723643 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fba68063cb0 msgr2=0x7fba68066160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:47.788 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.788+0000 7fba8e19a700 1 --2- 172.21.15.195:0/3328723643 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fba68063cb0 0x7fba68066160 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fba880c4d10 tx=0x7fba78007bf0 comp rx=0 tx=0).stop 2022-06-27T11:47:47.789 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.788+0000 7fba8e19a700 1 -- 172.21.15.195:0/3328723643 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fba8806f620 msgr2=0x7fba88167180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:47.789 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.788+0000 7fba8e19a700 1 --2- 172.21.15.195:0/3328723643 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fba8806f620 0x7fba88167180 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fba7c009ec0 tx=0x7fba7c017000 comp rx=0 tx=0).stop 2022-06-27T11:47:47.790 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.788+0000 7fba8e19a700 1 -- 172.21.15.195:0/3328723643 shutdown_connections 2022-06-27T11:47:47.790 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.788+0000 7fba8e19a700 1 --2- 172.21.15.195:0/3328723643 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fba68063cb0 0x7fba68066160 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:47.791 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.788+0000 7fba8e19a700 1 --2- 172.21.15.195:0/3328723643 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fba8806f620 0x7fba88167180 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:47.791 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.788+0000 7fba8e19a700 1 --2- 172.21.15.195:0/3328723643 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fba88150950 0x7fba880caad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:47.791 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.788+0000 7fba8e19a700 1 --2- 172.21.15.195:0/3328723643 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fba88150580 0x7fba88158ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:47.792 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.788+0000 7fba8e19a700 1 -- 172.21.15.195:0/3328723643 >> 172.21.15.195:0/3328723643 conn(0x7fba88063af0 msgr2=0x7fba88069cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:47.792 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.788+0000 7fba8e19a700 1 -- 172.21.15.195:0/3328723643 shutdown_connections 2022-06-27T11:47:47.793 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.788+0000 7fba8e19a700 1 -- 172.21.15.195:0/3328723643 wait complete. 2022-06-27T11:47:47.794 INFO:teuthology.orchestra.run.smithi195.stderr:enabled application 'rbd' on pool 'rbd' 2022-06-27T11:47:47.804 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2022-06-27T11:47:47.805 INFO:tasks.ceph.mds.a:Restarting daemon 2022-06-27T11:47:47.806 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-06-27T11:47:47.809 INFO:tasks.ceph.mds.a:Started 2022-06-27T11:47:47.810 INFO:tasks.ceph.mds.c:Restarting daemon 2022-06-27T11:47:47.810 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2022-06-27T11:47:47.813 INFO:tasks.ceph.mds.c:Started 2022-06-27T11:47:47.814 INFO:tasks.ceph.mds.e:Restarting daemon 2022-06-27T11:47:47.814 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2022-06-27T11:47:47.817 INFO:tasks.ceph.mds.e:Started 2022-06-27T11:47:47.817 INFO:tasks.ceph.mds.b:Restarting daemon 2022-06-27T11:47:47.818 DEBUG:teuthology.orchestra.run.smithi202:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-06-27T11:47:47.823 INFO:tasks.ceph.mds.b:Started 2022-06-27T11:47:47.823 INFO:tasks.ceph.mds.d:Restarting daemon 2022-06-27T11:47:47.823 DEBUG:teuthology.orchestra.run.smithi202:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2022-06-27T11:47:47.826 INFO:tasks.ceph.mds.d:Started 2022-06-27T11:47:47.826 INFO:tasks.ceph.mds.f:Restarting daemon 2022-06-27T11:47:47.827 DEBUG:teuthology.orchestra.run.smithi202:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2022-06-27T11:47:47.829 INFO:tasks.ceph.mds.f:Started 2022-06-27T11:47:47.830 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2022-06-27T11:47:47.831 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-27T11:47:47.900 INFO:tasks.ceph.mds.a.smithi195.stdout:starting mds.a at 2022-06-27T11:47:47.908 INFO:tasks.ceph.mds.c.smithi195.stdout:starting mds.c at 2022-06-27T11:47:47.908 INFO:tasks.ceph.mds.e.smithi195.stdout:starting mds.e at 2022-06-27T11:47:47.915 INFO:tasks.ceph.mds.b.smithi202.stdout:starting mds.b at 2022-06-27T11:47:47.930 INFO:tasks.ceph.mds.d.smithi202.stdout:starting mds.d at 2022-06-27T11:47:47.940 INFO:tasks.ceph.mds.f.smithi202.stdout:starting mds.f at 2022-06-27T11:47:47.967 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.967+0000 7fe20734f700 1 Processor -- start 2022-06-27T11:47:47.968 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.968+0000 7fe20734f700 1 -- start start 2022-06-27T11:47:47.975 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.968+0000 7fe20734f700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fe200067ed0 0x7fe2000c61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:47.975 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.968+0000 7fe20734f700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe20014d8a0 con 0x7fe2000c31d0 2022-06-27T11:47:47.976 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.968+0000 7fe20734f700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2000c3e10 con 0x7fe20006eff0 2022-06-27T11:47:47.976 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.968+0000 7fe20734f700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe20006d970 con 0x7fe200067ed0 2022-06-27T11:47:47.977 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.968+0000 7fe2048ea700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fe2000c31d0 0x7fe20015ba30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:47148/0 (socket says 172.21.15.195:47148) 2022-06-27T11:47:47.977 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.969+0000 7fe2048ea700 1 -- 172.21.15.195:0/1995752540 learned_addr learned my addr 172.21.15.195:0/1995752540 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:47:47.977 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.969+0000 7fe2058ec700 1 --2- 172.21.15.195:0/1995752540 >> v2:172.21.15.202:3300/0 conn(0x7fe200067ed0 0x7fe2000c61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:47.978 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.969+0000 7fe1f3fff700 1 -- 172.21.15.195:0/1995752540 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4090321925 0 0) 0x7fe20014d8a0 con 0x7fe2000c31d0 2022-06-27T11:47:47.978 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.969+0000 7fe1f3fff700 1 -- 172.21.15.195:0/1995752540 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1e8003430 con 0x7fe2000c31d0 2022-06-27T11:47:47.978 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.969+0000 7fe2058ec700 1 -- 172.21.15.195:0/1995752540 >> v1:172.21.15.202:6789/0 conn(0x7fe20006eff0 legacy=0x7fe2000c2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:47.979 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.969+0000 7fe2058ec700 1 -- 172.21.15.195:0/1995752540 >> v1:172.21.15.195:6789/0 conn(0x7fe2000c31d0 legacy=0x7fe20015ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:47.979 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.969+0000 7fe2058ec700 1 -- 172.21.15.195:0/1995752540 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe200153f20 con 0x7fe200067ed0 2022-06-27T11:47:47.979 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.969+0000 7fe2058ec700 1 --2- 172.21.15.195:0/1995752540 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe200067ed0 0x7fe2000c61e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe1f8009e60 tx=0x7fe1f8009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1b979b578e510bbf server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:47.979 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.969+0000 7fe1f3fff700 1 -- 172.21.15.195:0/1995752540 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1f800e070 con 0x7fe200067ed0 2022-06-27T11:47:47.980 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.969+0000 7fe1f3fff700 1 -- 172.21.15.195:0/1995752540 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1f800bbf0 con 0x7fe200067ed0 2022-06-27T11:47:47.980 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.969+0000 7fe1f3fff700 1 -- 172.21.15.195:0/1995752540 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1f8012430 con 0x7fe200067ed0 2022-06-27T11:47:47.980 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.970+0000 7fe20734f700 1 -- 172.21.15.195:0/1995752540 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe200067ed0 msgr2=0x7fe2000c61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:47.981 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.970+0000 7fe20734f700 1 --2- 172.21.15.195:0/1995752540 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe200067ed0 0x7fe2000c61e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe1f8009e60 tx=0x7fe1f8009780 comp rx=0 tx=0).stop 2022-06-27T11:47:47.981 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.970+0000 7fe20734f700 1 -- 172.21.15.195:0/1995752540 shutdown_connections 2022-06-27T11:47:47.981 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.970+0000 7fe20734f700 1 --2- 172.21.15.195:0/1995752540 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe200067ed0 0x7fe2000c61e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:47.981 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.970+0000 7fe20734f700 1 -- 172.21.15.195:0/1995752540 >> 172.21.15.195:0/1995752540 conn(0x7fe2000c9b00 msgr2=0x7fe2000c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:47.982 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.970+0000 7fe20734f700 1 -- 172.21.15.195:0/1995752540 shutdown_connections 2022-06-27T11:47:47.982 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.970+0000 7fe20734f700 1 -- 172.21.15.195:0/1995752540 wait complete. 2022-06-27T11:47:47.982 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.971+0000 7fe20734f700 1 Processor -- start 2022-06-27T11:47:47.983 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.971+0000 7fe20734f700 1 -- start start 2022-06-27T11:47:47.983 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.972+0000 7fe20734f700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe200067ed0 0x7fe2001542a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:47.983 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.972+0000 7fe20734f700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe20006eff0 0x7fe2000bd000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:47.984 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.972+0000 7fe20734f700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe2000bdf80 0x7fe2000be3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:47.984 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.972+0000 7fe20734f700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2001555b0 con 0x7fe20006eff0 2022-06-27T11:47:47.984 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.972+0000 7fe20734f700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe200157a60 con 0x7fe200067ed0 2022-06-27T11:47:47.984 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.972+0000 7fe20734f700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2000be930 con 0x7fe2000bdf80 2022-06-27T11:47:47.985 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.972+0000 7fe2048ea700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe20006eff0 0x7fe2000bd000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:47.985 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.972+0000 7fe2048ea700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe20006eff0 0x7fe2000bd000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:40966/0 (socket says 172.21.15.195:40966) 2022-06-27T11:47:47.986 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.972+0000 7fe2048ea700 1 -- 172.21.15.195:0/3214880302 learned_addr learned my addr 172.21.15.195:0/3214880302 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:47:47.986 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.972+0000 7fe2050eb700 1 --2- 172.21.15.195:0/3214880302 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe200067ed0 0x7fe2001542a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:47.986 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.972+0000 7fe2058ec700 1 --2- 172.21.15.195:0/3214880302 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe2000bdf80 0x7fe2000be3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:47.987 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.972+0000 7fe2048ea700 1 -- 172.21.15.195:0/3214880302 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe2000bdf80 msgr2=0x7fe2000be3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:47.987 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.972+0000 7fe2048ea700 1 --2- 172.21.15.195:0/3214880302 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe2000bdf80 0x7fe2000be3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:47.988 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.972+0000 7fe2048ea700 1 -- 172.21.15.195:0/3214880302 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe200067ed0 msgr2=0x7fe2001542a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:47.988 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.972+0000 7fe2048ea700 1 --2- 172.21.15.195:0/3214880302 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe200067ed0 0x7fe2001542a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:47.988 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.972+0000 7fe2048ea700 1 -- 172.21.15.195:0/3214880302 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1f8009a20 con 0x7fe20006eff0 2022-06-27T11:47:47.989 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.972+0000 7fe2048ea700 1 --2- 172.21.15.195:0/3214880302 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe20006eff0 0x7fe2000bd000 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc00c4c0 tx=0x7fe1fc007cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:47.989 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.972+0000 7fe1f17fa700 1 -- 172.21.15.195:0/3214880302 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1fc00ce10 con 0x7fe20006eff0 2022-06-27T11:47:47.990 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.972+0000 7fe1f17fa700 1 -- 172.21.15.195:0/3214880302 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1fc0072e0 con 0x7fe20006eff0 2022-06-27T11:47:47.990 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.972+0000 7fe20734f700 1 -- 172.21.15.195:0/3214880302 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe200162e30 con 0x7fe20006eff0 2022-06-27T11:47:47.993 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.972+0000 7fe1f17fa700 1 -- 172.21.15.195:0/3214880302 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1fc0076a0 con 0x7fe20006eff0 2022-06-27T11:47:47.993 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.973+0000 7fe20734f700 1 -- 172.21.15.195:0/3214880302 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2001633b0 con 0x7fe20006eff0 2022-06-27T11:47:47.994 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.973+0000 7fe20734f700 1 -- 172.21.15.195:0/3214880302 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2000722c0 con 0x7fe20006eff0 2022-06-27T11:47:47.994 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.974+0000 7fe1f17fa700 1 -- 172.21.15.195:0/3214880302 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 4) v1 ==== 81876+0+0 (secure 0 0 0) 0x7fe1fc019030 con 0x7fe20006eff0 2022-06-27T11:47:47.994 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.974+0000 7fe1f17fa700 1 --2- 172.21.15.195:0/3214880302 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe1e0063cb0 0x7fe1e0066160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:47.995 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.974+0000 7fe2050eb700 1 --2- 172.21.15.195:0/3214880302 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe1e0063cb0 0x7fe1e0066160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:47.995 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.974+0000 7fe1f17fa700 1 -- 172.21.15.195:0/3214880302 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 5206+0+0 (secure 0 0 0) 0x7fe1fc07f5a0 con 0x7fe20006eff0 2022-06-27T11:47:47.996 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.975+0000 7fe2050eb700 1 --2- 172.21.15.195:0/3214880302 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe1e0063cb0 0x7fe1e0066160 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe1f4007fd0 tx=0x7fe1f4009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:47.997 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:47.978+0000 7fe1f17fa700 1 -- 172.21.15.195:0/3214880302 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fe1fc04e530 con 0x7fe20006eff0 2022-06-27T11:47:48.170 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.170+0000 7fe20734f700 1 -- 172.21.15.195:0/3214880302 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe2000c5120 con 0x7fe20006eff0 2022-06-27T11:47:48.171 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.171+0000 7fe1f17fa700 1 -- 172.21.15.195:0/3214880302 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+13722 (secure 0 0 0) 0x7fe1fc017070 con 0x7fe20006eff0 2022-06-27T11:47:48.171 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:47:48.171 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":16,"fsid":"81b3a1d3-6a2b-47e8-8d69-7027cfeae7af","created":"2022-06-27T11:47:35.094715+0000","modified":"2022-06-27T11:47:47.781736+0000","last_up_change":"2022-06-27T11:47:44.777459+0000","last_in_change":"2022-06-27T11:47:37.596818+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2022-06-27T11:47:46.368325+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2022-06-27T11:47:46.386132+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"c64bc51f-d4f3-4bc2-806b-da0dc78d51b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6813","nonce":94217},{"type":"v1","addr":"172.21.15.195:6815","nonce":94217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6817","nonce":94217},{"type":"v1","addr":"172.21.15.195:6819","nonce":94217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":94217},{"type":"v1","addr":"172.21.15.195:6823","nonce":94217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6820","nonce":94217},{"type":"v1","addr":"172.21.15.195:6821","nonce":94217}]},"public_addr":"172.21.15.195:6815/94217","cluster_addr":"172.21.15.195:6819/94217","heartbeat_back_addr":"172.21.15.195:6823/94217","heartbeat_front_addr":"172.21.15.195:6821/94217","state":["exists","up"]},{"osd":1,"uuid":"bea71cdf-aa98-493f-9207-66d886a3e30e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":94223},{"type":"v1","addr":"172.21.15.195:6809","nonce":94223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":94223},{"type":"v1","addr":"172.21.15.195:6811","nonce":94223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6816","nonce":94223},{"type":"v1","addr":"172.21.15.195:6818","nonce":94223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":94223},{"type":"v1","addr":"172.21.15.195:6814","nonce":94223}]},"public_addr":"172.21.15.195:6809/94223","cluster_addr":"172.21.15.195:6811/94223","heartbeat_back_addr":"172.21.15.195:6818/94223","heartbeat_front_addr":"172.21.15.195:6814/94223","state":["exists","up"]},{"osd":2,"uuid":"bf6c87a0-a8ea-497a-9ce1-721285888065","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":94221},{"type":"v1","addr":"172.21.15.195:6801","nonce":94221}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":94221},{"type":"v1","addr":"172.21.15.195:6803","nonce":94221}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":94221},{"type":"v1","addr":"172.21.15.195:6807","nonce":94221}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":94221},{"type":"v1","addr":"172.21.15.195:6805","nonce":94221}]},"public_addr":"172.21.15.195:6801/94221","cluster_addr":"172.21.15.195:6803/94221","heartbeat_back_addr":"172.21.15.195:6807/94221","heartbeat_front_addr":"172.21.15.195:6805/94221","state":["exists","up"]},{"osd":3,"uuid":"e8aa47ce-af8c-47e7-9654-870cd53229f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":94232},{"type":"v1","addr":"172.21.15.195:6825","nonce":94232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":94232},{"type":"v1","addr":"172.21.15.195:6827","nonce":94232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":94232},{"type":"v1","addr":"172.21.15.195:6831","nonce":94232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":94232},{"type":"v1","addr":"172.21.15.195:6829","nonce":94232}]},"public_addr":"172.21.15.195:6825/94232","cluster_addr":"172.21.15.195:6827/94232","heartbeat_back_addr":"172.21.15.195:6831/94232","heartbeat_front_addr":"172.21.15.195:6829/94232","state":["exists","up"]},{"osd":4,"uuid":"99ffa553-f149-443a-98f5-68b10f77e085","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6800","nonce":94308},{"type":"v1","addr":"172.21.15.202:6801","nonce":94308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6802","nonce":94308},{"type":"v1","addr":"172.21.15.202:6803","nonce":94308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6806","nonce":94308},{"type":"v1","addr":"172.21.15.202:6807","nonce":94308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6804","nonce":94308},{"type":"v1","addr":"172.21.15.202:6805","nonce":94308}]},"public_addr":"172.21.15.202:6801/94308","cluster_addr":"172.21.15.202:6803/94308","heartbeat_back_addr":"172.21.15.202:6807/94308","heartbeat_front_addr":"172.21.15.202:6805/94308","state":["exists","up"]},{"osd":5,"uuid":"35409ef0-891b-4c01-82e4-26b8e64f502d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6824","nonce":94325},{"type":"v1","addr":"172.21.15.202:6825","nonce":94325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6826","nonce":94325},{"type":"v1","addr":"172.21.15.202:6827","nonce":94325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6830","nonce":94325},{"type":"v1","addr":"172.21.15.202:6831","nonce":94325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6828","nonce":94325},{"type":"v1","addr":"172.21.15.202:6829","nonce":94325}]},"public_addr":"172.21.15.202:6825/94325","cluster_addr":"172.21.15.202:6827/94325","heartbeat_back_addr":"172.21.15.202:6831/94325","heartbeat_front_addr":"172.21.15.202:6829/94325","state":["exists","up"]},{"osd":6,"uuid":"009124c4-8b5f-47d4-bd20-2c19271a2adc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6809","nonce":94309},{"type":"v1","addr":"172.21.15.202:6811","nonce":94309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6813","nonce":94309},{"type":"v1","addr":"172.21.15.202:6815","nonce":94309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6821","nonce":94309},{"type":"v1","addr":"172.21.15.202:6823","nonce":94309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6817","nonce":94309},{"type":"v1","addr":"172.21.15.202:6819","nonce":94309}]},"public_addr":"172.21.15.202:6811/94309","cluster_addr":"172.21.15.202:6815/94309","heartbeat_back_addr":"172.21.15.202:6823/94309","heartbeat_front_addr":"172.21.15.202:6819/94309","state":["exists","up"]},{"osd":7,"uuid":"7773115e-d3c3-4a0c-8801-d8ce63e0e5a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6808","nonce":94310},{"type":"v1","addr":"172.21.15.202:6810","nonce":94310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6812","nonce":94310},{"type":"v1","addr":"172.21.15.202:6814","nonce":94310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6820","nonce":94310},{"type":"v1","addr":"172.21.15.202:6822","nonce":94310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6816","nonce":94310},{"type":"v1","addr":"172.21.15.202:6818","nonce":94310}]},"public_addr":"172.21.15.202:6810/94310","cluster_addr":"172.21.15.202:6814/94310","heartbeat_back_addr":"172.21.15.202:6822/94310","heartbeat_front_addr":"172.21.15.202:6818/94310","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.776182+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.806922+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.237997+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.784371+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.723615+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.820332+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:43.841414+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.758071+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-06-27T11:47:48.173 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.173+0000 7fe20734f700 1 -- 172.21.15.195:0/3214880302 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe1e0063cb0 msgr2=0x7fe1e0066160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:48.174 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.174+0000 7fe20734f700 1 --2- 172.21.15.195:0/3214880302 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe1e0063cb0 0x7fe1e0066160 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe1f4007fd0 tx=0x7fe1f4009040 comp rx=0 tx=0).stop 2022-06-27T11:47:48.174 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.174+0000 7fe20734f700 1 -- 172.21.15.195:0/3214880302 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe20006eff0 msgr2=0x7fe2000bd000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:48.174 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.174+0000 7fe20734f700 1 --2- 172.21.15.195:0/3214880302 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe20006eff0 0x7fe2000bd000 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc00c4c0 tx=0x7fe1fc007cb0 comp rx=0 tx=0).stop 2022-06-27T11:47:48.174 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.174+0000 7fe20734f700 1 -- 172.21.15.195:0/3214880302 shutdown_connections 2022-06-27T11:47:48.175 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.174+0000 7fe20734f700 1 --2- 172.21.15.195:0/3214880302 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe1e0063cb0 0x7fe1e0066160 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:48.175 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.174+0000 7fe20734f700 1 --2- 172.21.15.195:0/3214880302 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe2000bdf80 0x7fe2000be3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:48.175 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.174+0000 7fe20734f700 1 --2- 172.21.15.195:0/3214880302 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe200067ed0 0x7fe2001542a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:48.175 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.174+0000 7fe20734f700 1 --2- 172.21.15.195:0/3214880302 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe20006eff0 0x7fe2000bd000 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:48.176 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.174+0000 7fe20734f700 1 -- 172.21.15.195:0/3214880302 >> 172.21.15.195:0/3214880302 conn(0x7fe2000c9b00 msgr2=0x7fe20014cb90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:48.176 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.174+0000 7fe20734f700 1 -- 172.21.15.195:0/3214880302 shutdown_connections 2022-06-27T11:47:48.176 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.174+0000 7fe20734f700 1 -- 172.21.15.195:0/3214880302 wait complete. 2022-06-27T11:47:48.186 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2022-06-27T11:47:46.368325+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2022-06-27T11:47:46.386132+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2022-06-27T11:47:48.187 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-06-27T11:47:48.322 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.323+0000 7f65ebbe8700 1 Processor -- start 2022-06-27T11:47:48.323 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.323+0000 7f65ebbe8700 1 -- start start 2022-06-27T11:47:48.330 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.324+0000 7f65ebbe8700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f65e4150500 0x7f65e4063af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:48.330 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.324+0000 7f65ebbe8700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65e4064030 con 0x7f65e41508d0 2022-06-27T11:47:48.331 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.324+0000 7f65ebbe8700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65e40c4160 con 0x7f65e406a450 2022-06-27T11:47:48.331 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.324+0000 7f65ebbe8700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65e40c4440 con 0x7f65e4150500 2022-06-27T11:47:48.331 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.324+0000 7f65ea185700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f65e41508d0 0x7f65e40bbf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:47184/0 (socket says 172.21.15.195:47184) 2022-06-27T11:47:48.332 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.324+0000 7f65ea185700 1 -- 172.21.15.195:0/1453564630 learned_addr learned my addr 172.21.15.195:0/1453564630 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:47:48.332 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.324+0000 7f65e9984700 1 --2- 172.21.15.195:0/1453564630 >> v2:172.21.15.202:3300/0 conn(0x7f65e4150500 0x7f65e4063af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:48.332 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.324+0000 7f65e8982700 1 -- 172.21.15.195:0/1453564630 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3009107701 0 0) 0x7f65dc001490 con 0x7f65e41508d0 2022-06-27T11:47:48.332 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.324+0000 7f65e8982700 1 -- 172.21.15.195:0/1453564630 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65cc003430 con 0x7f65e41508d0 2022-06-27T11:47:48.333 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.324+0000 7f65e8982700 1 -- 172.21.15.195:0/1453564630 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 604325177 0 0) 0x7f65cc003430 con 0x7f65e41508d0 2022-06-27T11:47:48.333 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.324+0000 7f65e9984700 1 -- 172.21.15.195:0/1453564630 >> v1:172.21.15.202:6789/0 conn(0x7f65e406a450 legacy=0x7f65e40c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:48.333 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.324+0000 7f65e9984700 1 -- 172.21.15.195:0/1453564630 >> v1:172.21.15.195:6789/0 conn(0x7f65e41508d0 legacy=0x7f65e40bbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:48.333 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.324+0000 7f65e9984700 1 -- 172.21.15.195:0/1453564630 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65e415a800 con 0x7f65e4150500 2022-06-27T11:47:48.333 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.325+0000 7f65e9984700 1 --2- 172.21.15.195:0/1453564630 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f65e4150500 0x7f65e4063af0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f65e0009720 tx=0x7f65e0009a30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4613be7fdc319dd5 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:48.334 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.325+0000 7f65e8982700 1 -- 172.21.15.195:0/1453564630 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65dc001980 con 0x7f65e4150500 2022-06-27T11:47:48.334 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.325+0000 7f65e8982700 1 -- 172.21.15.195:0/1453564630 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65e0004030 con 0x7f65e4150500 2022-06-27T11:47:48.334 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.325+0000 7f65e8982700 1 -- 172.21.15.195:0/1453564630 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65e00043d0 con 0x7f65e4150500 2022-06-27T11:47:48.334 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.325+0000 7f65ebbe8700 1 -- 172.21.15.195:0/1453564630 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f65e4150500 msgr2=0x7f65e4063af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:48.335 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.325+0000 7f65ebbe8700 1 --2- 172.21.15.195:0/1453564630 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f65e4150500 0x7f65e4063af0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f65e0009720 tx=0x7f65e0009a30 comp rx=0 tx=0).stop 2022-06-27T11:47:48.335 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.325+0000 7f65ebbe8700 1 -- 172.21.15.195:0/1453564630 shutdown_connections 2022-06-27T11:47:48.335 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.325+0000 7f65ebbe8700 1 --2- 172.21.15.195:0/1453564630 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f65e4150500 0x7f65e4063af0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:48.335 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.325+0000 7f65ebbe8700 1 -- 172.21.15.195:0/1453564630 >> 172.21.15.195:0/1453564630 conn(0x7f65e40c9b00 msgr2=0x7f65e40c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:48.335 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.325+0000 7f65ebbe8700 1 -- 172.21.15.195:0/1453564630 shutdown_connections 2022-06-27T11:47:48.336 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.325+0000 7f65ebbe8700 1 -- 172.21.15.195:0/1453564630 wait complete. 2022-06-27T11:47:48.336 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.326+0000 7f65ebbe8700 1 Processor -- start 2022-06-27T11:47:48.336 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.326+0000 7f65ebbe8700 1 -- start start 2022-06-27T11:47:48.336 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.327+0000 7f65ebbe8700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f65e406a450 0x7f65e40c0350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:48.337 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.327+0000 7f65ebbe8700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f65e4150500 0x7f65e40c2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:48.337 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.327+0000 7f65ebbe8700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f65e41508d0 0x7f65e4150d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:48.337 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.327+0000 7f65ebbe8700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e4153880 con 0x7f65e41508d0 2022-06-27T11:47:48.337 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.327+0000 7f65ebbe8700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e4167070 con 0x7f65e4150500 2022-06-27T11:47:48.338 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.327+0000 7f65ebbe8700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65e40c0890 con 0x7f65e406a450 2022-06-27T11:47:48.338 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.327+0000 7f65ea185700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f65e41508d0 0x7f65e4150d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:48.338 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.327+0000 7f65ea185700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f65e41508d0 0x7f65e4150d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41002/0 (socket says 172.21.15.195:41002) 2022-06-27T11:47:48.338 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.327+0000 7f65ea185700 1 -- 172.21.15.195:0/152994510 learned_addr learned my addr 172.21.15.195:0/152994510 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:47:48.339 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.327+0000 7f65e9183700 1 --2- 172.21.15.195:0/152994510 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f65e4150500 0x7f65e40c2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:48.339 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.327+0000 7f65e9984700 1 --2- 172.21.15.195:0/152994510 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f65e406a450 0x7f65e40c0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:48.339 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.327+0000 7f65ea185700 1 -- 172.21.15.195:0/152994510 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f65e406a450 msgr2=0x7f65e40c0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:48.339 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.327+0000 7f65ea185700 1 --2- 172.21.15.195:0/152994510 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f65e406a450 0x7f65e40c0350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:48.340 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.327+0000 7f65ea185700 1 -- 172.21.15.195:0/152994510 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f65e4150500 msgr2=0x7f65e40c2bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:48.340 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.327+0000 7f65ea185700 1 --2- 172.21.15.195:0/152994510 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f65e4150500 0x7f65e40c2bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:48.340 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.327+0000 7f65ea185700 1 -- 172.21.15.195:0/152994510 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65e0009290 con 0x7f65e41508d0 2022-06-27T11:47:48.340 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.328+0000 7f65ea185700 1 --2- 172.21.15.195:0/152994510 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f65e41508d0 0x7f65e4150d90 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f65dc00cb60 tx=0x7f65dc001250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:48.340 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.328+0000 7f65d1ffb700 1 -- 172.21.15.195:0/152994510 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65dc004c80 con 0x7f65e41508d0 2022-06-27T11:47:48.341 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.328+0000 7f65d1ffb700 1 -- 172.21.15.195:0/152994510 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65dc004de0 con 0x7f65e41508d0 2022-06-27T11:47:48.341 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.328+0000 7f65d1ffb700 1 -- 172.21.15.195:0/152994510 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65dc0073a0 con 0x7f65e41508d0 2022-06-27T11:47:48.341 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.328+0000 7f65ebbe8700 1 -- 172.21.15.195:0/152994510 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65e41532c0 con 0x7f65e41508d0 2022-06-27T11:47:48.342 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.328+0000 7f65ebbe8700 1 -- 172.21.15.195:0/152994510 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65e4166a20 con 0x7f65e41508d0 2022-06-27T11:47:48.342 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.329+0000 7f65ebbe8700 1 -- 172.21.15.195:0/152994510 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65e4166880 con 0x7f65e41508d0 2022-06-27T11:47:48.342 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.329+0000 7f65d1ffb700 1 -- 172.21.15.195:0/152994510 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 4) v1 ==== 81876+0+0 (secure 0 0 0) 0x7f65dc01a030 con 0x7f65e41508d0 2022-06-27T11:47:48.342 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.330+0000 7f65d1ffb700 1 --2- 172.21.15.195:0/152994510 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f65c4063cb0 0x7f65c4066160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:48.343 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.330+0000 7f65e9984700 1 --2- 172.21.15.195:0/152994510 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f65c4063cb0 0x7f65c4066160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:48.343 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.330+0000 7f65d1ffb700 1 -- 172.21.15.195:0/152994510 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f65dc07faa0 con 0x7f65e41508d0 2022-06-27T11:47:48.343 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.330+0000 7f65e9984700 1 --2- 172.21.15.195:0/152994510 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f65c4063cb0 0x7f65c4066160 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f65e0009a70 tx=0x7f65e0016000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:48.344 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.334+0000 7f65d1ffb700 1 -- 172.21.15.195:0/152994510 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f65e4166880 con 0x7f65e41508d0 2022-06-27T11:47:48.526 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.526+0000 7f65ebbe8700 1 -- 172.21.15.195:0/152994510 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f65e4064030 con 0x7f65e41508d0 2022-06-27T11:47:48.526 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.526+0000 7f65d1ffb700 1 -- 172.21.15.195:0/152994510 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v16) v1 ==== 92+0+10 (secure 0 0 0) 0x7f65e4064030 con 0x7f65e41508d0 2022-06-27T11:47:48.527 INFO:teuthology.orchestra.run.smithi195.stdout:pg_num: 8 2022-06-27T11:47:48.528 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.528+0000 7f65ebbe8700 1 -- 172.21.15.195:0/152994510 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f65c4063cb0 msgr2=0x7f65c4066160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:48.528 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.528+0000 7f65ebbe8700 1 --2- 172.21.15.195:0/152994510 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f65c4063cb0 0x7f65c4066160 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f65e0009a70 tx=0x7f65e0016000 comp rx=0 tx=0).stop 2022-06-27T11:47:48.529 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.528+0000 7f65ebbe8700 1 -- 172.21.15.195:0/152994510 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f65e41508d0 msgr2=0x7f65e4150d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:48.529 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.528+0000 7f65ebbe8700 1 --2- 172.21.15.195:0/152994510 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f65e41508d0 0x7f65e4150d90 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f65dc00cb60 tx=0x7f65dc001250 comp rx=0 tx=0).stop 2022-06-27T11:47:48.529 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.528+0000 7f65ebbe8700 1 -- 172.21.15.195:0/152994510 shutdown_connections 2022-06-27T11:47:48.529 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.528+0000 7f65ebbe8700 1 --2- 172.21.15.195:0/152994510 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f65c4063cb0 0x7f65c4066160 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:48.529 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.528+0000 7f65ebbe8700 1 --2- 172.21.15.195:0/152994510 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f65e41508d0 0x7f65e4150d90 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:48.530 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.528+0000 7f65ebbe8700 1 --2- 172.21.15.195:0/152994510 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f65e406a450 0x7f65e40c0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:48.530 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.528+0000 7f65ebbe8700 1 --2- 172.21.15.195:0/152994510 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f65e4150500 0x7f65e40c2bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:48.530 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.528+0000 7f65ebbe8700 1 -- 172.21.15.195:0/152994510 >> 172.21.15.195:0/152994510 conn(0x7f65e40c9b00 msgr2=0x7f65e406dc00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:48.530 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.528+0000 7f65ebbe8700 1 -- 172.21.15.195:0/152994510 shutdown_connections 2022-06-27T11:47:48.531 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.528+0000 7f65ebbe8700 1 -- 172.21.15.195:0/152994510 wait complete. 2022-06-27T11:47:48.540 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2022-06-27T11:47:48.664 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.665+0000 7f84a5d67700 1 Processor -- start 2022-06-27T11:47:48.665 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.665+0000 7f84a5d67700 1 -- start start 2022-06-27T11:47:48.668 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.666+0000 7f84a5d67700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f84a0153cb0 0x7f84a0156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:48.668 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.666+0000 7f84a5d67700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84a00c1a20 con 0x7f84a00c4160 2022-06-27T11:47:48.668 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.666+0000 7f84a5d67700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84a00c1130 con 0x7f84a006f640 2022-06-27T11:47:48.668 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.666+0000 7f84a5d67700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84a006ea90 con 0x7f84a0153cb0 2022-06-27T11:47:48.669 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.666+0000 7f849effd700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f84a00c4160 0x7f84a01535a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:47202/0 (socket says 172.21.15.195:47202) 2022-06-27T11:47:48.669 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.666+0000 7f849effd700 1 -- 172.21.15.195:0/91687432 learned_addr learned my addr 172.21.15.195:0/91687432 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:47:48.669 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.666+0000 7f849ffff700 1 --2- 172.21.15.195:0/91687432 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f84a0153cb0 0x7f84a0156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:48.669 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.666+0000 7f849e7fc700 1 -- 172.21.15.195:0/91687432 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1427664144 0 0) 0x7f84a00c1a20 con 0x7f84a00c4160 2022-06-27T11:47:48.670 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.666+0000 7f849e7fc700 1 -- 172.21.15.195:0/91687432 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f848c003430 con 0x7f84a00c4160 2022-06-27T11:47:48.670 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.666+0000 7f849ffff700 1 -- 172.21.15.195:0/91687432 >> v1:172.21.15.202:6789/0 conn(0x7f84a006f640 legacy=0x7f84a00c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:48.670 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.666+0000 7f849e7fc700 1 -- 172.21.15.195:0/91687432 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2307976239 0 0) 0x7f84a00c1130 con 0x7f84a006f640 2022-06-27T11:47:48.670 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.666+0000 7f849ffff700 1 -- 172.21.15.195:0/91687432 >> v1:172.21.15.195:6789/0 conn(0x7f84a00c4160 legacy=0x7f84a01535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:48.671 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.666+0000 7f849ffff700 1 -- 172.21.15.195:0/91687432 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84a006ed70 con 0x7f84a0153cb0 2022-06-27T11:47:48.671 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.666+0000 7f849ffff700 1 --2- 172.21.15.195:0/91687432 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f84a0153cb0 0x7f84a0156ed0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8498009e90 tx=0x7f8498009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ae579b6539dffe8 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:48.671 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.666+0000 7f849e7fc700 1 -- 172.21.15.195:0/91687432 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f849800e070 con 0x7f84a0153cb0 2022-06-27T11:47:48.672 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.666+0000 7f849e7fc700 1 -- 172.21.15.195:0/91687432 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f849800bcc0 con 0x7f84a0153cb0 2022-06-27T11:47:48.672 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.666+0000 7f849e7fc700 1 -- 172.21.15.195:0/91687432 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8498012430 con 0x7f84a0153cb0 2022-06-27T11:47:48.672 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.667+0000 7f84a5d67700 1 -- 172.21.15.195:0/91687432 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f84a0153cb0 msgr2=0x7f84a0156ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:48.673 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.667+0000 7f84a5d67700 1 --2- 172.21.15.195:0/91687432 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f84a0153cb0 0x7f84a0156ed0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8498009e90 tx=0x7f8498009780 comp rx=0 tx=0).stop 2022-06-27T11:47:48.673 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.667+0000 7f84a5d67700 1 -- 172.21.15.195:0/91687432 shutdown_connections 2022-06-27T11:47:48.674 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.667+0000 7f84a5d67700 1 --2- 172.21.15.195:0/91687432 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f84a0153cb0 0x7f84a0156ed0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:48.674 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.667+0000 7f84a5d67700 1 -- 172.21.15.195:0/91687432 >> 172.21.15.195:0/91687432 conn(0x7f84a00c6350 msgr2=0x7f84a00c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:48.675 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.667+0000 7f84a5d67700 1 -- 172.21.15.195:0/91687432 shutdown_connections 2022-06-27T11:47:48.675 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.667+0000 7f84a5d67700 1 -- 172.21.15.195:0/91687432 wait complete. 2022-06-27T11:47:48.676 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.668+0000 7f84a5d67700 1 Processor -- start 2022-06-27T11:47:48.676 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.668+0000 7f84a5d67700 1 -- start start 2022-06-27T11:47:48.682 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.669+0000 7f84a5d67700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f84a006f640 0x7f84a00bde90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:48.682 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.669+0000 7f84a5d67700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f84a00c4160 0x7f84a00baeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:48.683 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.669+0000 7f84a5d67700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f84a0153cb0 0x7f84a00bcbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:48.683 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.669+0000 7f84a5d67700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a00bbe50 con 0x7f84a0153cb0 2022-06-27T11:47:48.683 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.669+0000 7f84a5d67700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a00be3d0 con 0x7f84a006f640 2022-06-27T11:47:48.684 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.669+0000 7f84a5d67700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84a00bc630 con 0x7f84a00c4160 2022-06-27T11:47:48.684 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.669+0000 7f849ffff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f84a0153cb0 0x7f84a00bcbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:48.684 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.669+0000 7f849ffff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f84a0153cb0 0x7f84a00bcbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41020/0 (socket says 172.21.15.195:41020) 2022-06-27T11:47:48.684 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.669+0000 7f849f7fe700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f84a006f640 0x7f84a00bde90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:48.685 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.669+0000 7f849ffff700 1 -- 172.21.15.195:0/1663711450 learned_addr learned my addr 172.21.15.195:0/1663711450 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:47:48.685 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.669+0000 7f849effd700 1 --2- 172.21.15.195:0/1663711450 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f84a00c4160 0x7f84a00baeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:48.685 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.669+0000 7f849ffff700 1 -- 172.21.15.195:0/1663711450 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f84a00c4160 msgr2=0x7f84a00baeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:48.685 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.669+0000 7f849ffff700 1 --2- 172.21.15.195:0/1663711450 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f84a00c4160 0x7f84a00baeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:48.686 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.669+0000 7f849ffff700 1 -- 172.21.15.195:0/1663711450 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f84a006f640 msgr2=0x7f84a00bde90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:48.686 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.669+0000 7f849ffff700 1 --2- 172.21.15.195:0/1663711450 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f84a006f640 0x7f84a00bde90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:48.686 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.669+0000 7f849ffff700 1 -- 172.21.15.195:0/1663711450 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8498009a20 con 0x7f84a0153cb0 2022-06-27T11:47:48.686 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.670+0000 7f849ffff700 1 --2- 172.21.15.195:0/1663711450 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f84a0153cb0 0x7f84a00bcbc0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f84980099f0 tx=0x7f8498019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:48.687 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.670+0000 7f84877fe700 1 -- 172.21.15.195:0/1663711450 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f849801b420 con 0x7f84a0153cb0 2022-06-27T11:47:48.687 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.670+0000 7f84877fe700 1 -- 172.21.15.195:0/1663711450 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f849801b5c0 con 0x7f84a0153cb0 2022-06-27T11:47:48.687 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.670+0000 7f84a5d67700 1 -- 172.21.15.195:0/1663711450 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84a00c00c0 con 0x7f84a0153cb0 2022-06-27T11:47:48.687 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.670+0000 7f84877fe700 1 -- 172.21.15.195:0/1663711450 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f849801b980 con 0x7f84a0153cb0 2022-06-27T11:47:48.688 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.670+0000 7f84a5d67700 1 -- 172.21.15.195:0/1663711450 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84a00ba3d0 con 0x7f84a0153cb0 2022-06-27T11:47:48.688 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.671+0000 7f84a5d67700 1 -- 172.21.15.195:0/1663711450 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84a00722c0 con 0x7f84a0153cb0 2022-06-27T11:47:48.688 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.671+0000 7f84877fe700 1 -- 172.21.15.195:0/1663711450 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 4) v1 ==== 81876+0+0 (secure 0 0 0) 0x7f849802c030 con 0x7f84a0153cb0 2022-06-27T11:47:48.688 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.671+0000 7f84877fe700 1 --2- 172.21.15.195:0/1663711450 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8480063ce0 0x7f8480066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:48.689 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.672+0000 7f84877fe700 1 -- 172.21.15.195:0/1663711450 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f84980821d0 con 0x7f84a0153cb0 2022-06-27T11:47:48.689 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.672+0000 7f849f7fe700 1 --2- 172.21.15.195:0/1663711450 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8480063ce0 0x7f8480066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:48.689 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.672+0000 7f849f7fe700 1 --2- 172.21.15.195:0/1663711450 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8480063ce0 0x7f8480066190 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8490001900 tx=0x7f8490009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:48.689 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.676+0000 7f84877fe700 1 -- 172.21.15.195:0/1663711450 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f8498051160 con 0x7f84a0153cb0 2022-06-27T11:47:48.853 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.853+0000 7f84a5d67700 1 -- 172.21.15.195:0/1663711450 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f84a006f240 con 0x7f84a0153cb0 2022-06-27T11:47:48.854 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.854+0000 7f84877fe700 1 -- 172.21.15.195:0/1663711450 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) v1 ==== 93+0+10 (secure 0 0 0) 0x7f849802a020 con 0x7f84a0153cb0 2022-06-27T11:47:48.855 INFO:teuthology.orchestra.run.smithi195.stdout:pg_num: 1 2022-06-27T11:47:48.857 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.856+0000 7f84a5d67700 1 -- 172.21.15.195:0/1663711450 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8480063ce0 msgr2=0x7f8480066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:48.857 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.856+0000 7f84a5d67700 1 --2- 172.21.15.195:0/1663711450 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8480063ce0 0x7f8480066190 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8490001900 tx=0x7f8490009040 comp rx=0 tx=0).stop 2022-06-27T11:47:48.858 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.856+0000 7f84a5d67700 1 -- 172.21.15.195:0/1663711450 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f84a0153cb0 msgr2=0x7f84a00bcbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:48.858 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.856+0000 7f84a5d67700 1 --2- 172.21.15.195:0/1663711450 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f84a0153cb0 0x7f84a00bcbc0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f84980099f0 tx=0x7f8498019000 comp rx=0 tx=0).stop 2022-06-27T11:47:48.859 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.856+0000 7f84a5d67700 1 -- 172.21.15.195:0/1663711450 shutdown_connections 2022-06-27T11:47:48.859 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.856+0000 7f84a5d67700 1 --2- 172.21.15.195:0/1663711450 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8480063ce0 0x7f8480066190 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:48.860 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.856+0000 7f84a5d67700 1 --2- 172.21.15.195:0/1663711450 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f84a0153cb0 0x7f84a00bcbc0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:48.860 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.856+0000 7f84a5d67700 1 --2- 172.21.15.195:0/1663711450 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f84a006f640 0x7f84a00bde90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:48.861 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.856+0000 7f84a5d67700 1 --2- 172.21.15.195:0/1663711450 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f84a00c4160 0x7f84a00baeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:48.861 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.856+0000 7f84a5d67700 1 -- 172.21.15.195:0/1663711450 >> 172.21.15.195:0/1663711450 conn(0x7f84a00c6350 msgr2=0x7f84a00c7770 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:48.862 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.856+0000 7f84a5d67700 1 -- 172.21.15.195:0/1663711450 shutdown_connections 2022-06-27T11:47:48.862 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.856+0000 7f84a5d67700 1 -- 172.21.15.195:0/1663711450 wait complete. 2022-06-27T11:47:48.868 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:47:49.000 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.991+0000 7f2120927700 1 Processor -- start 2022-06-27T11:47:49.000 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.992+0000 7f2120927700 1 -- start start 2022-06-27T11:47:49.001 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.992+0000 7f2120927700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f211c06fef0 0x7f211c1570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:49.001 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.992+0000 7f2120927700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f211c06fba0 con 0x7f211c0c34a0 2022-06-27T11:47:49.001 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.992+0000 7f2120927700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f211c06f5e0 con 0x7f211c06e850 2022-06-27T11:47:49.001 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.992+0000 7f2120927700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f211c1575e0 con 0x7f211c06fef0 2022-06-27T11:47:49.002 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.993+0000 7f2119d9b700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f211c06fef0 0x7f211c1570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:49.002 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.993+0000 7f2119d9b700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f211c06fef0 0x7f211c1570a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41032/0 (socket says 172.21.15.195:41032) 2022-06-27T11:47:49.002 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.993+0000 7f211ad9d700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f211c0c34a0 0x7f211c0c2920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:47222/0 (socket says 172.21.15.195:47222) 2022-06-27T11:47:49.002 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.993+0000 7f2119d9b700 1 -- 172.21.15.195:0/3539994116 learned_addr learned my addr 172.21.15.195:0/3539994116 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:47:49.003 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.993+0000 7f2119d9b700 1 -- 172.21.15.195:0/3539994116 >> v1:172.21.15.202:6789/0 conn(0x7f211c06e850 legacy=0x7f211c069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:49.003 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.993+0000 7f2119d9b700 1 -- 172.21.15.195:0/3539994116 >> v1:172.21.15.195:6789/0 conn(0x7f211c0c34a0 legacy=0x7f211c0c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:49.003 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.993+0000 7f211959a700 1 -- 172.21.15.195:0/3539994116 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4154076414 0 0) 0x7f211c06f5e0 con 0x7f211c06e850 2022-06-27T11:47:49.003 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.993+0000 7f2119d9b700 1 -- 172.21.15.195:0/3539994116 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f211c06f8c0 con 0x7f211c06fef0 2022-06-27T11:47:49.003 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.993+0000 7f2119d9b700 1 --2- 172.21.15.195:0/3539994116 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f211c06fef0 0x7f211c1570a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f210c009860 tx=0x7f210c009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=200726b4eae031d3 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:49.004 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.993+0000 7f211959a700 1 -- 172.21.15.195:0/3539994116 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f210c004030 con 0x7f211c06fef0 2022-06-27T11:47:49.004 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.993+0000 7f211959a700 1 -- 172.21.15.195:0/3539994116 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f210c004190 con 0x7f211c06fef0 2022-06-27T11:47:49.004 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.993+0000 7f211959a700 1 -- 172.21.15.195:0/3539994116 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f210c011630 con 0x7f211c06fef0 2022-06-27T11:47:49.004 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.994+0000 7f2120927700 1 -- 172.21.15.195:0/3539994116 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f211c06fef0 msgr2=0x7f211c1570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:49.005 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.994+0000 7f2120927700 1 --2- 172.21.15.195:0/3539994116 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f211c06fef0 0x7f211c1570a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f210c009860 tx=0x7f210c009120 comp rx=0 tx=0).stop 2022-06-27T11:47:49.005 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.994+0000 7f2120927700 1 -- 172.21.15.195:0/3539994116 shutdown_connections 2022-06-27T11:47:49.005 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.994+0000 7f2120927700 1 --2- 172.21.15.195:0/3539994116 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f211c06fef0 0x7f211c1570a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:49.005 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.994+0000 7f2120927700 1 -- 172.21.15.195:0/3539994116 >> 172.21.15.195:0/3539994116 conn(0x7f211c0c6350 msgr2=0x7f211c0c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:49.005 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.994+0000 7f2120927700 1 -- 172.21.15.195:0/3539994116 shutdown_connections 2022-06-27T11:47:49.006 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.994+0000 7f2120927700 1 -- 172.21.15.195:0/3539994116 wait complete. 2022-06-27T11:47:49.006 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.995+0000 7f2120927700 1 Processor -- start 2022-06-27T11:47:49.006 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.995+0000 7f2120927700 1 -- start start 2022-06-27T11:47:49.006 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.996+0000 7f2120927700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f211c06e850 0x7f211c0682b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:49.007 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.996+0000 7f2120927700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f211c0c34a0 0x7f211c156b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:49.007 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.996+0000 7f2120927700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f211c0b5f20 0x7f211c155670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:49.007 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.996+0000 7f2120927700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f211c0c0110 con 0x7f211c06e850 2022-06-27T11:47:49.007 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.996+0000 7f2120927700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f211c0c8920 con 0x7f211c0c34a0 2022-06-27T11:47:49.008 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.996+0000 7f2120927700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f211c0b7a10 con 0x7f211c0b5f20 2022-06-27T11:47:49.008 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.996+0000 7f211a59c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f211c06e850 0x7f211c0682b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:49.008 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.996+0000 7f211a59c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f211c06e850 0x7f211c0682b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41036/0 (socket says 172.21.15.195:41036) 2022-06-27T11:47:49.008 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.996+0000 7f211a59c700 1 -- 172.21.15.195:0/1085735864 learned_addr learned my addr 172.21.15.195:0/1085735864 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:47:49.008 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.996+0000 7f211ad9d700 1 --2- 172.21.15.195:0/1085735864 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f211c0b5f20 0x7f211c155670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:49.009 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.996+0000 7f2119d9b700 1 --2- 172.21.15.195:0/1085735864 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f211c0c34a0 0x7f211c156b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:49.009 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.996+0000 7f211a59c700 1 -- 172.21.15.195:0/1085735864 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f211c0b5f20 msgr2=0x7f211c155670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:49.009 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.996+0000 7f211a59c700 1 --2- 172.21.15.195:0/1085735864 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f211c0b5f20 0x7f211c155670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:49.009 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.996+0000 7f211a59c700 1 -- 172.21.15.195:0/1085735864 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f211c0c34a0 msgr2=0x7f211c156b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:49.010 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.996+0000 7f211a59c700 1 --2- 172.21.15.195:0/1085735864 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f211c0c34a0 0x7f211c156b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:49.010 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.996+0000 7f211a59c700 1 -- 172.21.15.195:0/1085735864 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f210c0093a0 con 0x7f211c06e850 2022-06-27T11:47:49.010 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.996+0000 7f211a59c700 1 --2- 172.21.15.195:0/1085735864 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f211c06e850 0x7f211c0682b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f211000c410 tx=0x7f2110007de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:49.010 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.996+0000 7f210a7fc700 1 -- 172.21.15.195:0/1085735864 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f211000ce90 con 0x7f211c06e850 2022-06-27T11:47:49.011 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.996+0000 7f210a7fc700 1 -- 172.21.15.195:0/1085735864 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21100052e0 con 0x7f211c06e850 2022-06-27T11:47:49.011 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.996+0000 7f2120927700 1 -- 172.21.15.195:0/1085735864 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f211c0bfb50 con 0x7f211c06e850 2022-06-27T11:47:49.011 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.996+0000 7f210a7fc700 1 -- 172.21.15.195:0/1085735864 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2110003730 con 0x7f211c06e850 2022-06-27T11:47:49.011 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.996+0000 7f2120927700 1 -- 172.21.15.195:0/1085735864 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f211c0b7cb0 con 0x7f211c06e850 2022-06-27T11:47:49.012 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.997+0000 7f2120927700 1 -- 172.21.15.195:0/1085735864 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f211c0b7850 con 0x7f211c06e850 2022-06-27T11:47:49.012 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.998+0000 7f210a7fc700 1 -- 172.21.15.195:0/1085735864 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 4) v1 ==== 81876+0+0 (secure 0 0 0) 0x7f2110019030 con 0x7f211c06e850 2022-06-27T11:47:49.012 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.998+0000 7f210a7fc700 1 --2- 172.21.15.195:0/1085735864 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f20f8063cb0 0x7f20f8066160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:49.012 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.998+0000 7f210a7fc700 1 -- 172.21.15.195:0/1085735864 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f211007f5c0 con 0x7f211c06e850 2022-06-27T11:47:49.013 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.998+0000 7f2119d9b700 1 --2- 172.21.15.195:0/1085735864 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f20f8063cb0 0x7f20f8066160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:49.013 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:48.999+0000 7f2119d9b700 1 --2- 172.21.15.195:0/1085735864 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f20f8063cb0 0x7f20f8066160 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f210c009160 tx=0x7f210c016000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:49.013 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.002+0000 7f210a7fc700 1 -- 172.21.15.195:0/1085735864 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f211004e550 con 0x7f211c06e850 2022-06-27T11:47:49.224 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.224+0000 7f2120927700 1 -- 172.21.15.195:0/1085735864 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f211c0722c0 con 0x7f211c06e850 2022-06-27T11:47:49.224 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.224+0000 7f210a7fc700 1 -- 172.21.15.195:0/1085735864 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+5051 (secure 0 0 0) 0x7f2110017030 con 0x7f211c06e850 2022-06-27T11:47:49.225 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:47:49.226 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4287,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/864126677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":864126677},{"type":"v1","addr":"172.21.15.202:6833","nonce":864126677}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4293,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/2038169702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2038169702},{"type":"v1","addr":"172.21.15.202:6835","nonce":2038169702}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2022-06-27T11:47:49.228 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.226+0000 7f2120927700 1 -- 172.21.15.195:0/1085735864 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f20f8063cb0 msgr2=0x7f20f8066160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:49.228 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.226+0000 7f2120927700 1 --2- 172.21.15.195:0/1085735864 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f20f8063cb0 0x7f20f8066160 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f210c009160 tx=0x7f210c016000 comp rx=0 tx=0).stop 2022-06-27T11:47:49.229 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.226+0000 7f2120927700 1 -- 172.21.15.195:0/1085735864 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f211c06e850 msgr2=0x7f211c0682b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:49.229 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.226+0000 7f2120927700 1 --2- 172.21.15.195:0/1085735864 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f211c06e850 0x7f211c0682b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f211000c410 tx=0x7f2110007de0 comp rx=0 tx=0).stop 2022-06-27T11:47:49.229 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.226+0000 7f2120927700 1 -- 172.21.15.195:0/1085735864 shutdown_connections 2022-06-27T11:47:49.230 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.226+0000 7f2120927700 1 --2- 172.21.15.195:0/1085735864 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f20f8063cb0 0x7f20f8066160 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:49.230 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.226+0000 7f2120927700 1 --2- 172.21.15.195:0/1085735864 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f211c06e850 0x7f211c0682b0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:49.231 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.226+0000 7f2120927700 1 --2- 172.21.15.195:0/1085735864 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f211c0b5f20 0x7f211c155670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:49.231 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.226+0000 7f2120927700 1 --2- 172.21.15.195:0/1085735864 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f211c0c34a0 0x7f211c156b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:49.231 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.226+0000 7f2120927700 1 -- 172.21.15.195:0/1085735864 >> 172.21.15.195:0/1085735864 conn(0x7f211c0c6350 msgr2=0x7f211c153000 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:49.232 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.226+0000 7f2120927700 1 -- 172.21.15.195:0/1085735864 shutdown_connections 2022-06-27T11:47:49.232 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.226+0000 7f2120927700 1 -- 172.21.15.195:0/1085735864 wait complete. 2022-06-27T11:47:49.232 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 3 2022-06-27T11:47:49.238 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-27T11:47:49.361 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.361+0000 7fcb208a4700 1 Processor -- start 2022-06-27T11:47:49.361 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.361+0000 7fcb208a4700 1 -- start start 2022-06-27T11:47:49.364 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.362+0000 7fcb208a4700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fcb1c150580 0x7fcb1c156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:49.364 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.362+0000 7fcb208a4700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcb1c06f620 0x7fcb1c06d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:49.364 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.362+0000 7fcb208a4700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb1c0c76b0 con 0x7fcb1c150950 2022-06-27T11:47:49.365 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.362+0000 7fcb208a4700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb1c157430 con 0x7fcb1c150580 2022-06-27T11:47:49.365 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.362+0000 7fcb208a4700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb1c0c8250 con 0x7fcb1c06f620 2022-06-27T11:47:49.365 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.362+0000 7fcb1a59c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fcb1c150580 0x7fcb1c156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:49.365 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.363+0000 7fcb1a59c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fcb1c150580 0x7fcb1c156ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41044/0 (socket says 172.21.15.195:41044) 2022-06-27T11:47:49.365 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.363+0000 7fcb19d9b700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fcb1c150950 0x7fcb1c0c7b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:47232/0 (socket says 172.21.15.195:47232) 2022-06-27T11:47:49.366 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.363+0000 7fcb1a59c700 1 -- 172.21.15.195:0/4138037296 learned_addr learned my addr 172.21.15.195:0/4138037296 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:47:49.366 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.363+0000 7fcb1ad9d700 1 --2- 172.21.15.195:0/4138037296 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcb1c06f620 0x7fcb1c06d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:49.366 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.363+0000 7fcb1a59c700 1 -- 172.21.15.195:0/4138037296 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcb1c06f620 msgr2=0x7fcb1c06d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:47:49.367 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.363+0000 7fcb1a59c700 1 --2- 172.21.15.195:0/4138037296 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcb1c06f620 0x7fcb1c06d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:49.367 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.363+0000 7fcb1a59c700 1 -- 172.21.15.195:0/4138037296 >> v1:172.21.15.195:6789/0 conn(0x7fcb1c150950 legacy=0x7fcb1c0c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:49.367 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.363+0000 7fcb1959a700 1 -- 172.21.15.195:0/4138037296 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1132250741 0 0) 0x7fcb1c0c76b0 con 0x7fcb1c150950 2022-06-27T11:47:49.367 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.363+0000 7fcb1a59c700 1 -- 172.21.15.195:0/4138037296 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb1c0bad70 con 0x7fcb1c150580 2022-06-27T11:47:49.368 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.363+0000 7fcb1a59c700 1 --2- 172.21.15.195:0/4138037296 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcb1c150580 0x7fcb1c156ef0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fcb10009860 tx=0x7fcb10009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cc9cb9895848469 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:49.368 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.363+0000 7fcb1959a700 1 -- 172.21.15.195:0/4138037296 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb10011630 con 0x7fcb1c150580 2022-06-27T11:47:49.368 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.363+0000 7fcb1959a700 1 -- 172.21.15.195:0/4138037296 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb10011790 con 0x7fcb1c150580 2022-06-27T11:47:49.368 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.363+0000 7fcb1959a700 1 -- 172.21.15.195:0/4138037296 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb10011b30 con 0x7fcb1c150580 2022-06-27T11:47:49.368 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.364+0000 7fcb208a4700 1 -- 172.21.15.195:0/4138037296 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcb1c150580 msgr2=0x7fcb1c156ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:49.369 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.364+0000 7fcb208a4700 1 --2- 172.21.15.195:0/4138037296 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcb1c150580 0x7fcb1c156ef0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fcb10009860 tx=0x7fcb10009120 comp rx=0 tx=0).stop 2022-06-27T11:47:49.369 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.364+0000 7fcb208a4700 1 -- 172.21.15.195:0/4138037296 shutdown_connections 2022-06-27T11:47:49.369 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.364+0000 7fcb208a4700 1 --2- 172.21.15.195:0/4138037296 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcb1c06f620 0x7fcb1c06d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:49.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.364+0000 7fcb208a4700 1 --2- 172.21.15.195:0/4138037296 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcb1c150580 0x7fcb1c156ef0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:49.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.364+0000 7fcb208a4700 1 -- 172.21.15.195:0/4138037296 >> 172.21.15.195:0/4138037296 conn(0x7fcb1c063af0 msgr2=0x7fcb1c063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:49.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.364+0000 7fcb208a4700 1 -- 172.21.15.195:0/4138037296 shutdown_connections 2022-06-27T11:47:49.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.364+0000 7fcb208a4700 1 -- 172.21.15.195:0/4138037296 wait complete. 2022-06-27T11:47:49.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.365+0000 7fcb208a4700 1 Processor -- start 2022-06-27T11:47:49.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.365+0000 7fcb208a4700 1 -- start start 2022-06-27T11:47:49.379 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.366+0000 7fcb208a4700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcb1c06f620 0x7fcb1c166eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:49.380 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.366+0000 7fcb208a4700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcb1c150580 0x7fcb1c0bd560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:49.380 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.366+0000 7fcb208a4700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcb1c150950 0x7fcb1c0c2d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:49.380 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.366+0000 7fcb208a4700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb1c166a20 con 0x7fcb1c150950 2022-06-27T11:47:49.381 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.366+0000 7fcb208a4700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb1c0c4780 con 0x7fcb1c150580 2022-06-27T11:47:49.381 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.366+0000 7fcb208a4700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb1c166410 con 0x7fcb1c06f620 2022-06-27T11:47:49.381 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.366+0000 7fcb1ad9d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcb1c150950 0x7fcb1c0c2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:49.382 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.366+0000 7fcb1ad9d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcb1c150950 0x7fcb1c0c2d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41052/0 (socket says 172.21.15.195:41052) 2022-06-27T11:47:49.382 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.366+0000 7fcb1ad9d700 1 -- 172.21.15.195:0/2216907745 learned_addr learned my addr 172.21.15.195:0/2216907745 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:47:49.382 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.366+0000 7fcb19d9b700 1 --2- 172.21.15.195:0/2216907745 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcb1c150580 0x7fcb1c0bd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:49.383 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.366+0000 7fcb1a59c700 1 --2- 172.21.15.195:0/2216907745 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcb1c06f620 0x7fcb1c166eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:49.383 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.366+0000 7fcb1ad9d700 1 -- 172.21.15.195:0/2216907745 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcb1c06f620 msgr2=0x7fcb1c166eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:49.383 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.366+0000 7fcb1ad9d700 1 --2- 172.21.15.195:0/2216907745 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcb1c06f620 0x7fcb1c166eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:49.383 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.366+0000 7fcb1ad9d700 1 -- 172.21.15.195:0/2216907745 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcb1c150580 msgr2=0x7fcb1c0bd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:49.384 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.366+0000 7fcb1ad9d700 1 --2- 172.21.15.195:0/2216907745 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcb1c150580 0x7fcb1c0bd560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:49.384 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.366+0000 7fcb1ad9d700 1 -- 172.21.15.195:0/2216907745 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb100093a0 con 0x7fcb1c150950 2022-06-27T11:47:49.384 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.366+0000 7fcb1ad9d700 1 --2- 172.21.15.195:0/2216907745 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcb1c150950 0x7fcb1c0c2d60 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fcb14009f20 tx=0x7fcb1400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:49.384 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.366+0000 7fcb0a7fc700 1 -- 172.21.15.195:0/2216907745 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb1400cc00 con 0x7fcb1c150950 2022-06-27T11:47:49.385 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.366+0000 7fcb0a7fc700 1 -- 172.21.15.195:0/2216907745 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb1400f040 con 0x7fcb1c150950 2022-06-27T11:47:49.385 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.366+0000 7fcb208a4700 1 -- 172.21.15.195:0/2216907745 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb1c0c49f0 con 0x7fcb1c150950 2022-06-27T11:47:49.385 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.366+0000 7fcb0a7fc700 1 -- 172.21.15.195:0/2216907745 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb14014480 con 0x7fcb1c150950 2022-06-27T11:47:49.385 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.366+0000 7fcb208a4700 1 -- 172.21.15.195:0/2216907745 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb1c14edb0 con 0x7fcb1c150950 2022-06-27T11:47:49.385 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.367+0000 7fcb208a4700 1 -- 172.21.15.195:0/2216907745 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb1c166bd0 con 0x7fcb1c150950 2022-06-27T11:47:49.386 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.367+0000 7fcb0a7fc700 1 -- 172.21.15.195:0/2216907745 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fcb14014830 con 0x7fcb1c150950 2022-06-27T11:47:49.386 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.368+0000 7fcb0a7fc700 1 --2- 172.21.15.195:0/2216907745 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcaf8063d00 0x7fcaf80661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:49.386 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.368+0000 7fcb0a7fc700 1 -- 172.21.15.195:0/2216907745 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5206+0+0 (secure 0 0 0) 0x7fcb140814d0 con 0x7fcb1c150950 2022-06-27T11:47:49.386 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.368+0000 7fcb1a59c700 1 --2- 172.21.15.195:0/2216907745 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcaf8063d00 0x7fcaf80661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:49.387 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.368+0000 7fcb1a59c700 1 --2- 172.21.15.195:0/2216907745 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcaf8063d00 0x7fcaf80661b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fcb10009160 tx=0x7fcb10017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:49.387 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.372+0000 7fcb0a7fc700 1 -- 172.21.15.195:0/2216907745 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fcb1c166bd0 con 0x7fcb1c150950 2022-06-27T11:47:49.554 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.555+0000 7fcb208a4700 1 -- 172.21.15.195:0/2216907745 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcb1c166bd0 con 0x7fcb1c150950 2022-06-27T11:47:49.555 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.555+0000 7fcb0a7fc700 1 -- 172.21.15.195:0/2216907745 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+13696 (secure 0 0 0) 0x7fcb1c166bd0 con 0x7fcb1c150950 2022-06-27T11:47:49.555 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:47:49.555 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":17,"fsid":"81b3a1d3-6a2b-47e8-8d69-7027cfeae7af","created":"2022-06-27T11:47:35.094715+0000","modified":"2022-06-27T11:47:48.783203+0000","last_up_change":"2022-06-27T11:47:44.777459+0000","last_in_change":"2022-06-27T11:47:37.596818+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2022-06-27T11:47:46.368325+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2022-06-27T11:47:46.386132+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"c64bc51f-d4f3-4bc2-806b-da0dc78d51b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6813","nonce":94217},{"type":"v1","addr":"172.21.15.195:6815","nonce":94217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6817","nonce":94217},{"type":"v1","addr":"172.21.15.195:6819","nonce":94217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":94217},{"type":"v1","addr":"172.21.15.195:6823","nonce":94217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6820","nonce":94217},{"type":"v1","addr":"172.21.15.195:6821","nonce":94217}]},"public_addr":"172.21.15.195:6815/94217","cluster_addr":"172.21.15.195:6819/94217","heartbeat_back_addr":"172.21.15.195:6823/94217","heartbeat_front_addr":"172.21.15.195:6821/94217","state":["exists","up"]},{"osd":1,"uuid":"bea71cdf-aa98-493f-9207-66d886a3e30e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":94223},{"type":"v1","addr":"172.21.15.195:6809","nonce":94223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":94223},{"type":"v1","addr":"172.21.15.195:6811","nonce":94223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6816","nonce":94223},{"type":"v1","addr":"172.21.15.195:6818","nonce":94223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":94223},{"type":"v1","addr":"172.21.15.195:6814","nonce":94223}]},"public_addr":"172.21.15.195:6809/94223","cluster_addr":"172.21.15.195:6811/94223","heartbeat_back_addr":"172.21.15.195:6818/94223","heartbeat_front_addr":"172.21.15.195:6814/94223","state":["exists","up"]},{"osd":2,"uuid":"bf6c87a0-a8ea-497a-9ce1-721285888065","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":94221},{"type":"v1","addr":"172.21.15.195:6801","nonce":94221}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":94221},{"type":"v1","addr":"172.21.15.195:6803","nonce":94221}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":94221},{"type":"v1","addr":"172.21.15.195:6807","nonce":94221}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":94221},{"type":"v1","addr":"172.21.15.195:6805","nonce":94221}]},"public_addr":"172.21.15.195:6801/94221","cluster_addr":"172.21.15.195:6803/94221","heartbeat_back_addr":"172.21.15.195:6807/94221","heartbeat_front_addr":"172.21.15.195:6805/94221","state":["exists","up"]},{"osd":3,"uuid":"e8aa47ce-af8c-47e7-9654-870cd53229f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":94232},{"type":"v1","addr":"172.21.15.195:6825","nonce":94232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":94232},{"type":"v1","addr":"172.21.15.195:6827","nonce":94232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":94232},{"type":"v1","addr":"172.21.15.195:6831","nonce":94232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":94232},{"type":"v1","addr":"172.21.15.195:6829","nonce":94232}]},"public_addr":"172.21.15.195:6825/94232","cluster_addr":"172.21.15.195:6827/94232","heartbeat_back_addr":"172.21.15.195:6831/94232","heartbeat_front_addr":"172.21.15.195:6829/94232","state":["exists","up"]},{"osd":4,"uuid":"99ffa553-f149-443a-98f5-68b10f77e085","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6800","nonce":94308},{"type":"v1","addr":"172.21.15.202:6801","nonce":94308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6802","nonce":94308},{"type":"v1","addr":"172.21.15.202:6803","nonce":94308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6806","nonce":94308},{"type":"v1","addr":"172.21.15.202:6807","nonce":94308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6804","nonce":94308},{"type":"v1","addr":"172.21.15.202:6805","nonce":94308}]},"public_addr":"172.21.15.202:6801/94308","cluster_addr":"172.21.15.202:6803/94308","heartbeat_back_addr":"172.21.15.202:6807/94308","heartbeat_front_addr":"172.21.15.202:6805/94308","state":["exists","up"]},{"osd":5,"uuid":"35409ef0-891b-4c01-82e4-26b8e64f502d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6824","nonce":94325},{"type":"v1","addr":"172.21.15.202:6825","nonce":94325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6826","nonce":94325},{"type":"v1","addr":"172.21.15.202:6827","nonce":94325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6830","nonce":94325},{"type":"v1","addr":"172.21.15.202:6831","nonce":94325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6828","nonce":94325},{"type":"v1","addr":"172.21.15.202:6829","nonce":94325}]},"public_addr":"172.21.15.202:6825/94325","cluster_addr":"172.21.15.202:6827/94325","heartbeat_back_addr":"172.21.15.202:6831/94325","heartbeat_front_addr":"172.21.15.202:6829/94325","state":["exists","up"]},{"osd":6,"uuid":"009124c4-8b5f-47d4-bd20-2c19271a2adc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6809","nonce":94309},{"type":"v1","addr":"172.21.15.202:6811","nonce":94309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6813","nonce":94309},{"type":"v1","addr":"172.21.15.202:6815","nonce":94309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6821","nonce":94309},{"type":"v1","addr":"172.21.15.202:6823","nonce":94309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6817","nonce":94309},{"type":"v1","addr":"172.21.15.202:6819","nonce":94309}]},"public_addr":"172.21.15.202:6811/94309","cluster_addr":"172.21.15.202:6815/94309","heartbeat_back_addr":"172.21.15.202:6823/94309","heartbeat_front_addr":"172.21.15.202:6819/94309","state":["exists","up"]},{"osd":7,"uuid":"7773115e-d3c3-4a0c-8801-d8ce63e0e5a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6808","nonce":94310},{"type":"v1","addr":"172.21.15.202:6810","nonce":94310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6812","nonce":94310},{"type":"v1","addr":"172.21.15.202:6814","nonce":94310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6820","nonce":94310},{"type":"v1","addr":"172.21.15.202:6822","nonce":94310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6816","nonce":94310},{"type":"v1","addr":"172.21.15.202:6818","nonce":94310}]},"public_addr":"172.21.15.202:6810/94310","cluster_addr":"172.21.15.202:6814/94310","heartbeat_back_addr":"172.21.15.202:6822/94310","heartbeat_front_addr":"172.21.15.202:6818/94310","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.776182+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.806922+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.237997+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.784371+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.723615+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.820332+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:43.841414+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.758071+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-06-27T11:47:49.557 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.557+0000 7fcb208a4700 1 -- 172.21.15.195:0/2216907745 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcaf8063d00 msgr2=0x7fcaf80661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:49.558 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.557+0000 7fcb208a4700 1 --2- 172.21.15.195:0/2216907745 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcaf8063d00 0x7fcaf80661b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fcb10009160 tx=0x7fcb10017000 comp rx=0 tx=0).stop 2022-06-27T11:47:49.558 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.557+0000 7fcb208a4700 1 -- 172.21.15.195:0/2216907745 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcb1c150950 msgr2=0x7fcb1c0c2d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:49.558 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.557+0000 7fcb208a4700 1 --2- 172.21.15.195:0/2216907745 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcb1c150950 0x7fcb1c0c2d60 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fcb14009f20 tx=0x7fcb1400b040 comp rx=0 tx=0).stop 2022-06-27T11:47:49.558 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.557+0000 7fcb208a4700 1 -- 172.21.15.195:0/2216907745 shutdown_connections 2022-06-27T11:47:49.559 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.557+0000 7fcb208a4700 1 --2- 172.21.15.195:0/2216907745 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcaf8063d00 0x7fcaf80661b0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:49.559 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.557+0000 7fcb208a4700 1 --2- 172.21.15.195:0/2216907745 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcb1c150950 0x7fcb1c0c2d60 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:49.559 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.557+0000 7fcb208a4700 1 --2- 172.21.15.195:0/2216907745 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcb1c06f620 0x7fcb1c166eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:49.559 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.557+0000 7fcb208a4700 1 --2- 172.21.15.195:0/2216907745 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcb1c150580 0x7fcb1c0bd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:49.560 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.557+0000 7fcb208a4700 1 -- 172.21.15.195:0/2216907745 >> 172.21.15.195:0/2216907745 conn(0x7fcb1c063af0 msgr2=0x7fcb1c06d470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:49.560 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.558+0000 7fcb208a4700 1 -- 172.21.15.195:0/2216907745 shutdown_connections 2022-06-27T11:47:49.560 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.558+0000 7fcb208a4700 1 -- 172.21.15.195:0/2216907745 wait complete. 2022-06-27T11:47:49.570 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2022-06-27T11:47:46.368325+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2022-06-27T11:47:46.386132+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2022-06-27T11:47:49.571 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-06-27T11:47:49.694 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.694+0000 7fd3bfe30700 1 Processor -- start 2022-06-27T11:47:49.694 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.694+0000 7fd3bfe30700 1 -- start start 2022-06-27T11:47:49.697 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.695+0000 7fd3bfe30700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd3b80688a0 0x7fd3b80c28c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:49.697 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.695+0000 7fd3bfe30700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3b80c4b10 con 0x7fd3b8156100 2022-06-27T11:47:49.698 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.695+0000 7fd3bfe30700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3b8068f20 con 0x7fd3b806a440 2022-06-27T11:47:49.698 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.695+0000 7fd3bfe30700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3b80c3dc0 con 0x7fd3b80688a0 2022-06-27T11:47:49.699 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.695+0000 7fd3be3cd700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd3b80688a0 0x7fd3b80c28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:49.699 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.695+0000 7fd3bdbcc700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fd3b8156100 0x7fd3b80c4fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:47250/0 (socket says 172.21.15.195:47250) 2022-06-27T11:47:49.699 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.695+0000 7fd3be3cd700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd3b80688a0 0x7fd3b80c28c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41064/0 (socket says 172.21.15.195:41064) 2022-06-27T11:47:49.699 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.695+0000 7fd3bdbcc700 1 -- 172.21.15.195:0/599069111 learned_addr learned my addr 172.21.15.195:0/599069111 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:47:49.700 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.696+0000 7fd3be3cd700 1 -- 172.21.15.195:0/599069111 >> v1:172.21.15.202:6789/0 conn(0x7fd3b806a440 legacy=0x7fd3b81564d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:49.700 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.696+0000 7fd3be3cd700 1 -- 172.21.15.195:0/599069111 >> v1:172.21.15.195:6789/0 conn(0x7fd3b8156100 legacy=0x7fd3b80c4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:49.700 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.696+0000 7fd3bcbca700 1 -- 172.21.15.195:0/599069111 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 485464418 0 0) 0x7fd3b80c4b10 con 0x7fd3b8156100 2022-06-27T11:47:49.700 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.696+0000 7fd3be3cd700 1 -- 172.21.15.195:0/599069111 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3b80c56b0 con 0x7fd3b80688a0 2022-06-27T11:47:49.701 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.696+0000 7fd3be3cd700 1 --2- 172.21.15.195:0/599069111 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd3b80688a0 0x7fd3b80c28c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd3b000d0d0 tx=0x7fd3b000b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25ef9f9115ebfd20 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:49.701 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.696+0000 7fd3bcbca700 1 -- 172.21.15.195:0/599069111 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3b0010070 con 0x7fd3b80688a0 2022-06-27T11:47:49.701 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.696+0000 7fd3bcbca700 1 -- 172.21.15.195:0/599069111 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3b000ddf0 con 0x7fd3b80688a0 2022-06-27T11:47:49.702 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.696+0000 7fd3bcbca700 1 -- 172.21.15.195:0/599069111 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3b000b320 con 0x7fd3b80688a0 2022-06-27T11:47:49.702 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.696+0000 7fd3bfe30700 1 -- 172.21.15.195:0/599069111 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd3b80688a0 msgr2=0x7fd3b80c28c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:49.702 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.697+0000 7fd3bfe30700 1 --2- 172.21.15.195:0/599069111 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd3b80688a0 0x7fd3b80c28c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd3b000d0d0 tx=0x7fd3b000b990 comp rx=0 tx=0).stop 2022-06-27T11:47:49.702 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.697+0000 7fd3bfe30700 1 -- 172.21.15.195:0/599069111 shutdown_connections 2022-06-27T11:47:49.703 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.697+0000 7fd3bfe30700 1 --2- 172.21.15.195:0/599069111 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd3b80688a0 0x7fd3b80c28c0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:49.703 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.697+0000 7fd3bfe30700 1 -- 172.21.15.195:0/599069111 >> 172.21.15.195:0/599069111 conn(0x7fd3b80c9b00 msgr2=0x7fd3b80c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:49.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.697+0000 7fd3bfe30700 1 -- 172.21.15.195:0/599069111 shutdown_connections 2022-06-27T11:47:49.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.697+0000 7fd3bfe30700 1 -- 172.21.15.195:0/599069111 wait complete. 2022-06-27T11:47:49.708 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.698+0000 7fd3bfe30700 1 Processor -- start 2022-06-27T11:47:49.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.698+0000 7fd3bfe30700 1 -- start start 2022-06-27T11:47:49.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.699+0000 7fd3bfe30700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd3b80688a0 0x7fd3b80c81a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:49.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.699+0000 7fd3bfe30700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd3b806a440 0x7fd3b81482d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:49.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.699+0000 7fd3bfe30700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd3b8156100 0x7fd3b8148810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:49.710 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.699+0000 7fd3bfe30700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b814a250 con 0x7fd3b806a440 2022-06-27T11:47:49.710 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.699+0000 7fd3bfe30700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b80bfeb0 con 0x7fd3b80688a0 2022-06-27T11:47:49.710 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.699+0000 7fd3bfe30700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b814e4d0 con 0x7fd3b8156100 2022-06-27T11:47:49.710 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.699+0000 7fd3bd3cb700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd3b806a440 0x7fd3b81482d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:49.711 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.699+0000 7fd3bdbcc700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd3b80688a0 0x7fd3b80c81a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:49.711 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.699+0000 7fd3bd3cb700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd3b806a440 0x7fd3b81482d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41068/0 (socket says 172.21.15.195:41068) 2022-06-27T11:47:49.712 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.699+0000 7fd3bd3cb700 1 -- 172.21.15.195:0/2340393688 learned_addr learned my addr 172.21.15.195:0/2340393688 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:47:49.712 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.699+0000 7fd3be3cd700 1 --2- 172.21.15.195:0/2340393688 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd3b8156100 0x7fd3b8148810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:49.712 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.699+0000 7fd3bd3cb700 1 -- 172.21.15.195:0/2340393688 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd3b8156100 msgr2=0x7fd3b8148810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:49.713 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.699+0000 7fd3bd3cb700 1 --2- 172.21.15.195:0/2340393688 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd3b8156100 0x7fd3b8148810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:49.713 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.699+0000 7fd3bd3cb700 1 -- 172.21.15.195:0/2340393688 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd3b80688a0 msgr2=0x7fd3b80c81a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:49.714 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.699+0000 7fd3bd3cb700 1 --2- 172.21.15.195:0/2340393688 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd3b80688a0 0x7fd3b80c81a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:49.714 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.699+0000 7fd3bd3cb700 1 -- 172.21.15.195:0/2340393688 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3b000bc30 con 0x7fd3b806a440 2022-06-27T11:47:49.714 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.700+0000 7fd3bd3cb700 1 --2- 172.21.15.195:0/2340393688 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd3b806a440 0x7fd3b81482d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd3b400c4f0 tx=0x7fd3b4007d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:49.715 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.700+0000 7fd3a5ffb700 1 -- 172.21.15.195:0/2340393688 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3b400ce90 con 0x7fd3b806a440 2022-06-27T11:47:49.715 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.700+0000 7fd3a5ffb700 1 -- 172.21.15.195:0/2340393688 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3b40050d0 con 0x7fd3b806a440 2022-06-27T11:47:49.715 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.700+0000 7fd3bfe30700 1 -- 172.21.15.195:0/2340393688 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3b814d620 con 0x7fd3b806a440 2022-06-27T11:47:49.715 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.700+0000 7fd3a5ffb700 1 -- 172.21.15.195:0/2340393688 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3b4011590 con 0x7fd3b806a440 2022-06-27T11:47:49.716 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.700+0000 7fd3bfe30700 1 -- 172.21.15.195:0/2340393688 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3b814dba0 con 0x7fd3b806a440 2022-06-27T11:47:49.716 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.701+0000 7fd3bfe30700 1 -- 172.21.15.195:0/2340393688 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3b814e610 con 0x7fd3b806a440 2022-06-27T11:47:49.717 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.701+0000 7fd3a5ffb700 1 -- 172.21.15.195:0/2340393688 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fd3b4019030 con 0x7fd3b806a440 2022-06-27T11:47:49.718 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.702+0000 7fd3a5ffb700 1 --2- 172.21.15.195:0/2340393688 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd398063d00 0x7fd3980661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:49.718 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.702+0000 7fd3a5ffb700 1 -- 172.21.15.195:0/2340393688 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5206+0+0 (secure 0 0 0) 0x7fd3b407f5e0 con 0x7fd3b806a440 2022-06-27T11:47:49.718 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.702+0000 7fd3bdbcc700 1 --2- 172.21.15.195:0/2340393688 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd398063d00 0x7fd3980661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:49.719 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.702+0000 7fd3bdbcc700 1 --2- 172.21.15.195:0/2340393688 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd398063d00 0x7fd3980661b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd3a80010e0 tx=0x7fd3a8009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:49.719 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.706+0000 7fd3a5ffb700 1 -- 172.21.15.195:0/2340393688 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fd3b814e610 con 0x7fd3b806a440 2022-06-27T11:47:49.896 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.897+0000 7fd3bfe30700 1 -- 172.21.15.195:0/2340393688 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fd3b814e610 con 0x7fd3b806a440 2022-06-27T11:47:49.897 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.897+0000 7fd3a5ffb700 1 -- 172.21.15.195:0/2340393688 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v17) v1 ==== 92+0+10 (secure 0 0 0) 0x7fd3b814e610 con 0x7fd3b806a440 2022-06-27T11:47:49.897 INFO:teuthology.orchestra.run.smithi195.stdout:pg_num: 8 2022-06-27T11:47:49.900 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.899+0000 7fd3bfe30700 1 -- 172.21.15.195:0/2340393688 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd398063d00 msgr2=0x7fd3980661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:49.900 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.899+0000 7fd3bfe30700 1 --2- 172.21.15.195:0/2340393688 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd398063d00 0x7fd3980661b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd3a80010e0 tx=0x7fd3a8009040 comp rx=0 tx=0).stop 2022-06-27T11:47:49.901 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.899+0000 7fd3bfe30700 1 -- 172.21.15.195:0/2340393688 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd3b806a440 msgr2=0x7fd3b81482d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:49.901 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.899+0000 7fd3bfe30700 1 --2- 172.21.15.195:0/2340393688 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd3b806a440 0x7fd3b81482d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd3b400c4f0 tx=0x7fd3b4007d40 comp rx=0 tx=0).stop 2022-06-27T11:47:49.901 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.899+0000 7fd3bfe30700 1 -- 172.21.15.195:0/2340393688 shutdown_connections 2022-06-27T11:47:49.906 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.899+0000 7fd3bfe30700 1 --2- 172.21.15.195:0/2340393688 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd398063d00 0x7fd3980661b0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:49.907 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.899+0000 7fd3bfe30700 1 --2- 172.21.15.195:0/2340393688 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd3b8156100 0x7fd3b8148810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:49.907 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.899+0000 7fd3bfe30700 1 --2- 172.21.15.195:0/2340393688 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd3b80688a0 0x7fd3b80c81a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:49.907 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.899+0000 7fd3bfe30700 1 --2- 172.21.15.195:0/2340393688 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd3b806a440 0x7fd3b81482d0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:49.908 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.899+0000 7fd3bfe30700 1 -- 172.21.15.195:0/2340393688 >> 172.21.15.195:0/2340393688 conn(0x7fd3b80c9b00 msgr2=0x7fd3b80c06f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:49.908 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.899+0000 7fd3bfe30700 1 -- 172.21.15.195:0/2340393688 shutdown_connections 2022-06-27T11:47:49.909 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:49.899+0000 7fd3bfe30700 1 -- 172.21.15.195:0/2340393688 wait complete. 2022-06-27T11:47:49.911 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2022-06-27T11:47:50.037 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.038+0000 7f13d621b700 1 Processor -- start 2022-06-27T11:47:50.038 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.038+0000 7f13d621b700 1 -- start start 2022-06-27T11:47:50.043 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.039+0000 7f13d621b700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f13d006e850 0x7f13d01570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:50.044 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.039+0000 7f13d621b700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f13d00c34a0 0x7f13d00c2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:50.044 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.039+0000 7f13d621b700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13d006fba0 con 0x7f13d006fef0 2022-06-27T11:47:50.044 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.039+0000 7f13d621b700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13d01575e0 con 0x7f13d00c34a0 2022-06-27T11:47:50.044 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.039+0000 7f13d621b700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13d006f5e0 con 0x7f13d006e850 2022-06-27T11:47:50.045 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.039+0000 7f13ceffd700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f13d006fef0 0x7f13d0069de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:47270/0 (socket says 172.21.15.195:47270) 2022-06-27T11:47:50.045 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.039+0000 7f13ceffd700 1 -- 172.21.15.195:0/730660565 learned_addr learned my addr 172.21.15.195:0/730660565 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:47:50.045 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.039+0000 7f13cf7fe700 1 --2- 172.21.15.195:0/730660565 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f13d006e850 0x7f13d01570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:50.045 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.039+0000 7f13cffff700 1 --2- 172.21.15.195:0/730660565 >> v2:172.21.15.202:3300/0 conn(0x7f13d00c34a0 0x7f13d00c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:50.046 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.039+0000 7f13ce7fc700 1 -- 172.21.15.195:0/730660565 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1379897166 0 0) 0x7f13d006fba0 con 0x7f13d006fef0 2022-06-27T11:47:50.046 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.039+0000 7f13ce7fc700 1 -- 172.21.15.195:0/730660565 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13bc003430 con 0x7f13d006fef0 2022-06-27T11:47:50.046 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.039+0000 7f13cf7fe700 1 -- 172.21.15.195:0/730660565 >> v2:172.21.15.202:3300/0 conn(0x7f13d00c34a0 msgr2=0x7f13d00c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:47:50.046 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.039+0000 7f13cf7fe700 1 --2- 172.21.15.195:0/730660565 >> v2:172.21.15.202:3300/0 conn(0x7f13d00c34a0 0x7f13d00c2920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:50.047 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.039+0000 7f13cf7fe700 1 -- 172.21.15.195:0/730660565 >> v1:172.21.15.195:6789/0 conn(0x7f13d006fef0 legacy=0x7f13d0069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:50.047 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.039+0000 7f13cf7fe700 1 -- 172.21.15.195:0/730660565 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13d006f8c0 con 0x7f13d006e850 2022-06-27T11:47:50.047 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.039+0000 7f13cffff700 1 --2- 172.21.15.195:0/730660565 >> v2:172.21.15.202:3300/0 conn(0x7f13d00c34a0 0x7f13d00c2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:47:50.047 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.039+0000 7f13cf7fe700 1 --2- 172.21.15.195:0/730660565 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f13d006e850 0x7f13d01570a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f13c40097a0 tx=0x7f13c4009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9060e2de501bcf75 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:50.048 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.040+0000 7f13ce7fc700 1 -- 172.21.15.195:0/730660565 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c4011630 con 0x7f13d006e850 2022-06-27T11:47:50.048 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.040+0000 7f13ce7fc700 1 -- 172.21.15.195:0/730660565 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13c4011790 con 0x7f13d006e850 2022-06-27T11:47:50.048 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.040+0000 7f13ce7fc700 1 -- 172.21.15.195:0/730660565 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c4011b30 con 0x7f13d006e850 2022-06-27T11:47:50.048 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.040+0000 7f13d621b700 1 -- 172.21.15.195:0/730660565 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f13d006e850 msgr2=0x7f13d01570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:50.049 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.040+0000 7f13d621b700 1 --2- 172.21.15.195:0/730660565 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f13d006e850 0x7f13d01570a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f13c40097a0 tx=0x7f13c4009060 comp rx=0 tx=0).stop 2022-06-27T11:47:50.049 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.040+0000 7f13d621b700 1 -- 172.21.15.195:0/730660565 shutdown_connections 2022-06-27T11:47:50.049 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.040+0000 7f13d621b700 1 --2- 172.21.15.195:0/730660565 >> v2:172.21.15.202:3300/0 conn(0x7f13d00c34a0 0x7f13d00c2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:50.049 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.040+0000 7f13d621b700 1 --2- 172.21.15.195:0/730660565 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f13d006e850 0x7f13d01570a0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:50.050 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.040+0000 7f13d621b700 1 -- 172.21.15.195:0/730660565 >> 172.21.15.195:0/730660565 conn(0x7f13d00c6350 msgr2=0x7f13d00c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:50.050 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.040+0000 7f13d621b700 1 -- 172.21.15.195:0/730660565 shutdown_connections 2022-06-27T11:47:50.050 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.040+0000 7f13d621b700 1 -- 172.21.15.195:0/730660565 wait complete. 2022-06-27T11:47:50.051 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.041+0000 7f13d621b700 1 Processor -- start 2022-06-27T11:47:50.051 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.041+0000 7f13d621b700 1 -- start start 2022-06-27T11:47:50.051 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.042+0000 7f13d621b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f13d006e850 0x7f13d00c0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:50.051 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.042+0000 7f13d621b700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f13d00c34a0 0x7f13d0151010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:50.051 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.042+0000 7f13d621b700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f13d0153ce0 0x7f13d0155850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:50.052 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.042+0000 7f13d621b700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13d0153850 con 0x7f13d006e850 2022-06-27T11:47:50.052 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.042+0000 7f13d621b700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13d01562b0 con 0x7f13d0153ce0 2022-06-27T11:47:50.052 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.042+0000 7f13d621b700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13d0167220 con 0x7f13d00c34a0 2022-06-27T11:47:50.053 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.042+0000 7f13cf7fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f13d006e850 0x7f13d00c0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:50.053 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.042+0000 7f13cf7fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f13d006e850 0x7f13d00c0630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41086/0 (socket says 172.21.15.195:41086) 2022-06-27T11:47:50.053 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.042+0000 7f13cf7fe700 1 -- 172.21.15.195:0/2787326811 learned_addr learned my addr 172.21.15.195:0/2787326811 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:47:50.054 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.042+0000 7f13ceffd700 1 --2- 172.21.15.195:0/2787326811 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f13d00c34a0 0x7f13d0151010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:50.054 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.042+0000 7f13cffff700 1 --2- 172.21.15.195:0/2787326811 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f13d0153ce0 0x7f13d0155850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:50.054 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.043+0000 7f13cf7fe700 1 -- 172.21.15.195:0/2787326811 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f13d00c34a0 msgr2=0x7f13d0151010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:50.054 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.043+0000 7f13cf7fe700 1 --2- 172.21.15.195:0/2787326811 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f13d00c34a0 0x7f13d0151010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:50.055 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.043+0000 7f13cf7fe700 1 -- 172.21.15.195:0/2787326811 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f13d0153ce0 msgr2=0x7f13d0155850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:50.055 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.043+0000 7f13cf7fe700 1 --2- 172.21.15.195:0/2787326811 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f13d0153ce0 0x7f13d0155850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:50.055 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.043+0000 7f13cf7fe700 1 -- 172.21.15.195:0/2787326811 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13c8009a20 con 0x7f13d006e850 2022-06-27T11:47:50.055 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.043+0000 7f13cf7fe700 1 --2- 172.21.15.195:0/2787326811 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f13d006e850 0x7f13d00c0630 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f13c4009d90 tx=0x7f13c4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:50.056 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.043+0000 7f13b77fe700 1 -- 172.21.15.195:0/2787326811 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c40037b0 con 0x7f13d006e850 2022-06-27T11:47:50.056 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.043+0000 7f13b77fe700 1 -- 172.21.15.195:0/2787326811 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13c4003910 con 0x7f13d006e850 2022-06-27T11:47:50.056 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.043+0000 7f13d621b700 1 -- 172.21.15.195:0/2787326811 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13c40092e0 con 0x7f13d006e850 2022-06-27T11:47:50.056 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.043+0000 7f13b77fe700 1 -- 172.21.15.195:0/2787326811 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c4003cd0 con 0x7f13d006e850 2022-06-27T11:47:50.057 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.043+0000 7f13d621b700 1 -- 172.21.15.195:0/2787326811 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13d0153200 con 0x7f13d006e850 2022-06-27T11:47:50.057 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.044+0000 7f13d621b700 1 -- 172.21.15.195:0/2787326811 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13d0153060 con 0x7f13d006e850 2022-06-27T11:47:50.058 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.044+0000 7f13b77fe700 1 -- 172.21.15.195:0/2787326811 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f13c400f050 con 0x7f13d006e850 2022-06-27T11:47:50.058 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.045+0000 7f13b77fe700 1 --2- 172.21.15.195:0/2787326811 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f13b0063d00 0x7f13b00661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:50.059 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.045+0000 7f13b77fe700 1 -- 172.21.15.195:0/2787326811 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f13c4081880 con 0x7f13d006e850 2022-06-27T11:47:50.059 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.045+0000 7f13ceffd700 1 --2- 172.21.15.195:0/2787326811 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f13b0063d00 0x7f13b00661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:50.059 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.045+0000 7f13ceffd700 1 --2- 172.21.15.195:0/2787326811 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f13b0063d00 0x7f13b00661b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f13c0007fd0 tx=0x7f13c0009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:50.059 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.049+0000 7f13b77fe700 1 -- 172.21.15.195:0/2787326811 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f13d0153060 con 0x7f13d006e850 2022-06-27T11:47:50.226 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.226+0000 7f13d621b700 1 -- 172.21.15.195:0/2787326811 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f13d0153b30 con 0x7f13d006e850 2022-06-27T11:47:50.226 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.227+0000 7f13b77fe700 1 -- 172.21.15.195:0/2787326811 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) v1 ==== 93+0+10 (secure 0 0 0) 0x7f13c4015030 con 0x7f13d006e850 2022-06-27T11:47:50.227 INFO:teuthology.orchestra.run.smithi195.stdout:pg_num: 1 2022-06-27T11:47:50.229 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.229+0000 7f13d621b700 1 -- 172.21.15.195:0/2787326811 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f13b0063d00 msgr2=0x7f13b00661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:50.229 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.229+0000 7f13d621b700 1 --2- 172.21.15.195:0/2787326811 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f13b0063d00 0x7f13b00661b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f13c0007fd0 tx=0x7f13c0009040 comp rx=0 tx=0).stop 2022-06-27T11:47:50.229 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.229+0000 7f13d621b700 1 -- 172.21.15.195:0/2787326811 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f13d006e850 msgr2=0x7f13d00c0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:50.230 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.229+0000 7f13d621b700 1 --2- 172.21.15.195:0/2787326811 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f13d006e850 0x7f13d00c0630 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f13c4009d90 tx=0x7f13c4017000 comp rx=0 tx=0).stop 2022-06-27T11:47:50.230 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.229+0000 7f13d621b700 1 -- 172.21.15.195:0/2787326811 shutdown_connections 2022-06-27T11:47:50.230 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.229+0000 7f13d621b700 1 --2- 172.21.15.195:0/2787326811 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f13b0063d00 0x7f13b00661b0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:50.230 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.229+0000 7f13d621b700 1 --2- 172.21.15.195:0/2787326811 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f13d006e850 0x7f13d00c0630 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:50.231 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.229+0000 7f13d621b700 1 --2- 172.21.15.195:0/2787326811 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f13d0153ce0 0x7f13d0155850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:50.231 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.229+0000 7f13d621b700 1 --2- 172.21.15.195:0/2787326811 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f13d00c34a0 0x7f13d0151010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:50.231 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.229+0000 7f13d621b700 1 -- 172.21.15.195:0/2787326811 >> 172.21.15.195:0/2787326811 conn(0x7f13d00c6350 msgr2=0x7f13d0150550 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:50.231 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.229+0000 7f13d621b700 1 -- 172.21.15.195:0/2787326811 shutdown_connections 2022-06-27T11:47:50.231 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.229+0000 7f13d621b700 1 -- 172.21.15.195:0/2787326811 wait complete. 2022-06-27T11:47:50.241 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-06-27T11:47:50.361 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.362+0000 7fa439b05700 1 Processor -- start 2022-06-27T11:47:50.362 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.362+0000 7fa439b05700 1 -- start start 2022-06-27T11:47:50.368 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.362+0000 7fa439b05700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa43406a450 0x7fa434063af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:50.369 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.362+0000 7fa439b05700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fa4341508d0 0x7fa4340bbf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:50.369 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.362+0000 7fa439b05700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa434064030 con 0x7fa434150500 2022-06-27T11:47:50.369 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.362+0000 7fa439b05700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4340c4160 con 0x7fa4341508d0 2022-06-27T11:47:50.369 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.362+0000 7fa439b05700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4340c4440 con 0x7fa43406a450 2022-06-27T11:47:50.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.363+0000 7fa433fff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fa4341508d0 0x7fa4340bbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:50.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.363+0000 7fa433fff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fa4341508d0 0x7fa4340bbf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41098/0 (socket says 172.21.15.195:41098) 2022-06-27T11:47:50.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.363+0000 7fa433fff700 1 -- 172.21.15.195:0/1054653200 learned_addr learned my addr 172.21.15.195:0/1054653200 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:47:50.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.363+0000 7fa432ffd700 1 --2- 172.21.15.195:0/1054653200 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa43406a450 0x7fa434063af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:50.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.363+0000 7fa433fff700 1 -- 172.21.15.195:0/1054653200 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa43406a450 msgr2=0x7fa434063af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:47:50.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.363+0000 7fa433fff700 1 --2- 172.21.15.195:0/1054653200 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa43406a450 0x7fa434063af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:50.372 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.363+0000 7fa433fff700 1 -- 172.21.15.195:0/1054653200 >> v1:172.21.15.202:6789/0 conn(0x7fa434150500 legacy=0x7fa4340c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:50.372 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.363+0000 7fa433fff700 1 -- 172.21.15.195:0/1054653200 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa43415a800 con 0x7fa4341508d0 2022-06-27T11:47:50.372 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.363+0000 7fa433fff700 1 --2- 172.21.15.195:0/1054653200 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa4341508d0 0x7fa4340bbf50 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa42c009f20 tx=0x7fa42c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52020c1bc0cb2ed0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:50.372 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.363+0000 7fa4327fc700 1 -- 172.21.15.195:0/1054653200 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa42c00e070 con 0x7fa4341508d0 2022-06-27T11:47:50.372 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.363+0000 7fa4327fc700 1 -- 172.21.15.195:0/1054653200 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa42c00b9d0 con 0x7fa4341508d0 2022-06-27T11:47:50.373 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.363+0000 7fa4327fc700 1 -- 172.21.15.195:0/1054653200 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa42c0092e0 con 0x7fa4341508d0 2022-06-27T11:47:50.373 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.364+0000 7fa439b05700 1 -- 172.21.15.195:0/1054653200 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa4341508d0 msgr2=0x7fa4340bbf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:50.373 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.364+0000 7fa439b05700 1 --2- 172.21.15.195:0/1054653200 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa4341508d0 0x7fa4340bbf50 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa42c009f20 tx=0x7fa42c009840 comp rx=0 tx=0).stop 2022-06-27T11:47:50.373 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.364+0000 7fa439b05700 1 -- 172.21.15.195:0/1054653200 shutdown_connections 2022-06-27T11:47:50.374 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.364+0000 7fa439b05700 1 --2- 172.21.15.195:0/1054653200 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa4341508d0 0x7fa4340bbf50 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:50.374 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.364+0000 7fa439b05700 1 --2- 172.21.15.195:0/1054653200 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa43406a450 0x7fa434063af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:50.374 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.364+0000 7fa439b05700 1 -- 172.21.15.195:0/1054653200 >> 172.21.15.195:0/1054653200 conn(0x7fa4340c9b00 msgr2=0x7fa4340c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:50.374 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.364+0000 7fa439b05700 1 -- 172.21.15.195:0/1054653200 shutdown_connections 2022-06-27T11:47:50.374 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.364+0000 7fa439b05700 1 -- 172.21.15.195:0/1054653200 wait complete. 2022-06-27T11:47:50.375 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.365+0000 7fa439b05700 1 Processor -- start 2022-06-27T11:47:50.375 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.365+0000 7fa439b05700 1 -- start start 2022-06-27T11:47:50.375 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.366+0000 7fa439b05700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa43406a450 0x7fa4340c0350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:50.376 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.366+0000 7fa439b05700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa434150500 0x7fa4340685c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:50.376 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.366+0000 7fa439b05700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa4341508d0 0x7fa434159240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:50.377 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.366+0000 7fa439b05700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4340be410 con 0x7fa43406a450 2022-06-27T11:47:50.377 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.366+0000 7fa439b05700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4340bd290 con 0x7fa4341508d0 2022-06-27T11:47:50.377 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.366+0000 7fa439b05700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4340bde90 con 0x7fa434150500 2022-06-27T11:47:50.378 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.366+0000 7fa4337fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa43406a450 0x7fa4340c0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:50.378 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.366+0000 7fa4337fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa43406a450 0x7fa4340c0350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41100/0 (socket says 172.21.15.195:41100) 2022-06-27T11:47:50.379 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.366+0000 7fa4337fe700 1 -- 172.21.15.195:0/2506443346 learned_addr learned my addr 172.21.15.195:0/2506443346 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:47:50.379 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.366+0000 7fa432ffd700 1 --2- 172.21.15.195:0/2506443346 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa434150500 0x7fa4340685c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:50.380 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.366+0000 7fa433fff700 1 --2- 172.21.15.195:0/2506443346 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa4341508d0 0x7fa434159240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:50.380 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.366+0000 7fa4337fe700 1 -- 172.21.15.195:0/2506443346 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa434150500 msgr2=0x7fa4340685c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:50.380 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.366+0000 7fa4337fe700 1 --2- 172.21.15.195:0/2506443346 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa434150500 0x7fa4340685c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:50.381 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.366+0000 7fa4337fe700 1 -- 172.21.15.195:0/2506443346 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa4341508d0 msgr2=0x7fa434159240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:50.381 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.366+0000 7fa4337fe700 1 --2- 172.21.15.195:0/2506443346 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa4341508d0 0x7fa434159240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:50.382 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.366+0000 7fa4337fe700 1 -- 172.21.15.195:0/2506443346 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa42c009ae0 con 0x7fa43406a450 2022-06-27T11:47:50.383 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.366+0000 7fa432ffd700 1 --2- 172.21.15.195:0/2506443346 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa434150500 0x7fa4340685c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:47:50.383 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.366+0000 7fa4337fe700 1 --2- 172.21.15.195:0/2506443346 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa43406a450 0x7fa4340c0350 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa42400c4e0 tx=0x7fa424007e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:50.384 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.366+0000 7fa41b7fe700 1 -- 172.21.15.195:0/2506443346 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa42400ce50 con 0x7fa43406a450 2022-06-27T11:47:50.384 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.366+0000 7fa41b7fe700 1 -- 172.21.15.195:0/2506443346 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4240050d0 con 0x7fa43406a450 2022-06-27T11:47:50.385 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.366+0000 7fa439b05700 1 -- 172.21.15.195:0/2506443346 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4340be130 con 0x7fa43406a450 2022-06-27T11:47:50.385 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.367+0000 7fa41b7fe700 1 -- 172.21.15.195:0/2506443346 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa424011590 con 0x7fa43406a450 2022-06-27T11:47:50.385 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.367+0000 7fa439b05700 1 -- 172.21.15.195:0/2506443346 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa434158d20 con 0x7fa43406a450 2022-06-27T11:47:50.386 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.367+0000 7fa439b05700 1 -- 172.21.15.195:0/2506443346 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa434069fb0 con 0x7fa43406a450 2022-06-27T11:47:50.386 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.368+0000 7fa41b7fe700 1 -- 172.21.15.195:0/2506443346 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fa424019030 con 0x7fa43406a450 2022-06-27T11:47:50.387 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.368+0000 7fa41b7fe700 1 --2- 172.21.15.195:0/2506443346 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa414063d00 0x7fa4140661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:50.387 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.368+0000 7fa41b7fe700 1 -- 172.21.15.195:0/2506443346 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5206+0+0 (secure 0 0 0) 0x7fa42407f420 con 0x7fa43406a450 2022-06-27T11:47:50.388 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.368+0000 7fa432ffd700 1 --2- 172.21.15.195:0/2506443346 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa414063d00 0x7fa4140661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:50.388 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.368+0000 7fa432ffd700 1 --2- 172.21.15.195:0/2506443346 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa414063d00 0x7fa4140661b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa4340bd4f0 tx=0x7fa428009060 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:50.389 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.372+0000 7fa41b7fe700 1 -- 172.21.15.195:0/2506443346 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fa42404e360 con 0x7fa43406a450 2022-06-27T11:47:50.555 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.555+0000 7fa439b05700 1 -- 172.21.15.195:0/2506443346 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fa434064030 con 0x7fa43406a450 2022-06-27T11:47:50.555 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.555+0000 7fa41b7fe700 1 -- 172.21.15.195:0/2506443346 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v17) v1 ==== 84+0+126 (secure 0 0 0) 0x7fa434064030 con 0x7fa43406a450 2022-06-27T11:47:50.556 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:47:50.556 INFO:teuthology.orchestra.run.smithi195.stdout:[ 2022-06-27T11:47:50.557 INFO:teuthology.orchestra.run.smithi195.stdout: { 2022-06-27T11:47:50.557 INFO:teuthology.orchestra.run.smithi195.stdout: "poolnum": 1, 2022-06-27T11:47:50.558 INFO:teuthology.orchestra.run.smithi195.stdout: "poolname": "rbd" 2022-06-27T11:47:50.558 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2022-06-27T11:47:50.559 INFO:teuthology.orchestra.run.smithi195.stdout: { 2022-06-27T11:47:50.559 INFO:teuthology.orchestra.run.smithi195.stdout: "poolnum": 2, 2022-06-27T11:47:50.559 INFO:teuthology.orchestra.run.smithi195.stdout: "poolname": ".mgr" 2022-06-27T11:47:50.560 INFO:teuthology.orchestra.run.smithi195.stdout: } 2022-06-27T11:47:50.560 INFO:teuthology.orchestra.run.smithi195.stdout:] 2022-06-27T11:47:50.562 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.557+0000 7fa439b05700 1 -- 172.21.15.195:0/2506443346 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa414063d00 msgr2=0x7fa4140661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:50.563 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.557+0000 7fa439b05700 1 --2- 172.21.15.195:0/2506443346 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa414063d00 0x7fa4140661b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa4340bd4f0 tx=0x7fa428009060 comp rx=0 tx=0).stop 2022-06-27T11:47:50.563 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.557+0000 7fa439b05700 1 -- 172.21.15.195:0/2506443346 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa43406a450 msgr2=0x7fa4340c0350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:50.564 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.557+0000 7fa439b05700 1 --2- 172.21.15.195:0/2506443346 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa43406a450 0x7fa4340c0350 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa42400c4e0 tx=0x7fa424007e30 comp rx=0 tx=0).stop 2022-06-27T11:47:50.564 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.557+0000 7fa439b05700 1 -- 172.21.15.195:0/2506443346 shutdown_connections 2022-06-27T11:47:50.564 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.557+0000 7fa439b05700 1 --2- 172.21.15.195:0/2506443346 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa414063d00 0x7fa4140661b0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:50.565 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.557+0000 7fa439b05700 1 --2- 172.21.15.195:0/2506443346 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa43406a450 0x7fa4340c0350 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:50.565 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.557+0000 7fa439b05700 1 --2- 172.21.15.195:0/2506443346 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa4341508d0 0x7fa434159240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:50.566 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.557+0000 7fa439b05700 1 --2- 172.21.15.195:0/2506443346 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa434150500 0x7fa4340685c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:50.566 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.557+0000 7fa439b05700 1 -- 172.21.15.195:0/2506443346 >> 172.21.15.195:0/2506443346 conn(0x7fa4340c9b00 msgr2=0x7fa43414fbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:50.567 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.557+0000 7fa439b05700 1 -- 172.21.15.195:0/2506443346 shutdown_connections 2022-06-27T11:47:50.567 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.557+0000 7fa439b05700 1 -- 172.21.15.195:0/2506443346 wait complete. 2022-06-27T11:47:50.568 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-06-27T11:47:50.569 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2022-06-27T11:47:50.693 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.693+0000 7f4baa665700 1 Processor -- start 2022-06-27T11:47:50.693 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.693+0000 7f4baa665700 1 -- start start 2022-06-27T11:47:50.702 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.694+0000 7f4baa665700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f4ba4156100 0x7f4ba40688a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:50.702 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.694+0000 7f4baa665700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4ba406a440 0x7f4ba40c4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:50.703 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.694+0000 7f4baa665700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ba40c4b10 con 0x7f4ba41564d0 2022-06-27T11:47:50.703 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.694+0000 7f4baa665700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ba40c3dc0 con 0x7f4ba4156100 2022-06-27T11:47:50.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.694+0000 7f4baa665700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba4068f20 con 0x7f4ba406a440 2022-06-27T11:47:50.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.694+0000 7f4ba37fe700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4ba406a440 0x7f4ba40c4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:50.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.694+0000 7f4ba3fff700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f4ba4156100 0x7f4ba40688a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:50.705 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.694+0000 7f4ba3fff700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f4ba4156100 0x7f4ba40688a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3300/0 says I am v2:172.21.15.195:39234/0 (socket says 172.21.15.195:39234) 2022-06-27T11:47:50.705 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.694+0000 7f4ba37fe700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4ba406a440 0x7f4ba40c4fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3301/0 says I am v2:172.21.15.195:33864/0 (socket says 172.21.15.195:33864) 2022-06-27T11:47:50.706 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.694+0000 7f4ba8c02700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f4ba41564d0 0x7f4ba40c28c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:47300/0 (socket says 172.21.15.195:47300) 2022-06-27T11:47:50.706 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.694+0000 7f4ba3fff700 1 -- 172.21.15.195:0/1662835155 learned_addr learned my addr 172.21.15.195:0/1662835155 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:47:50.707 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.694+0000 7f4ba2ffd700 1 -- 172.21.15.195:0/1662835155 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2733273358 0 0) 0x7f4ba40c4b10 con 0x7f4ba41564d0 2022-06-27T11:47:50.707 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.695+0000 7f4ba2ffd700 1 -- 172.21.15.195:0/1662835155 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b90003430 con 0x7f4ba41564d0 2022-06-27T11:47:50.707 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.695+0000 7f4ba37fe700 1 -- 172.21.15.195:0/1662835155 >> v2:172.21.15.202:3300/0 conn(0x7f4ba4156100 msgr2=0x7f4ba40688a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:47:50.708 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.695+0000 7f4ba37fe700 1 --2- 172.21.15.195:0/1662835155 >> v2:172.21.15.202:3300/0 conn(0x7f4ba4156100 0x7f4ba40688a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:50.708 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.695+0000 7f4ba37fe700 1 -- 172.21.15.195:0/1662835155 >> v1:172.21.15.195:6789/0 conn(0x7f4ba41564d0 legacy=0x7f4ba40c28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:50.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.695+0000 7f4ba37fe700 1 -- 172.21.15.195:0/1662835155 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ba40c6360 con 0x7f4ba406a440 2022-06-27T11:47:50.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.695+0000 7f4ba3fff700 1 --2- 172.21.15.195:0/1662835155 >> v2:172.21.15.202:3300/0 conn(0x7f4ba4156100 0x7f4ba40688a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:47:50.710 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.695+0000 7f4ba37fe700 1 --2- 172.21.15.195:0/1662835155 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4ba406a440 0x7f4ba40c4fa0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4b9400a940 tx=0x7f4b9400a200 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ad578d2fe89432b server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:50.710 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.695+0000 7f4ba2ffd700 1 -- 172.21.15.195:0/1662835155 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b9400c710 con 0x7f4ba406a440 2022-06-27T11:47:50.711 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.695+0000 7f4ba2ffd700 1 -- 172.21.15.195:0/1662835155 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b9400c870 con 0x7f4ba406a440 2022-06-27T11:47:50.711 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.695+0000 7f4ba2ffd700 1 -- 172.21.15.195:0/1662835155 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b9400cc30 con 0x7f4ba406a440 2022-06-27T11:47:50.712 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.695+0000 7f4baa665700 1 -- 172.21.15.195:0/1662835155 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4ba406a440 msgr2=0x7f4ba40c4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:50.713 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.695+0000 7f4baa665700 1 --2- 172.21.15.195:0/1662835155 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4ba406a440 0x7f4ba40c4fa0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4b9400a940 tx=0x7f4b9400a200 comp rx=0 tx=0).stop 2022-06-27T11:47:50.713 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.695+0000 7f4baa665700 1 -- 172.21.15.195:0/1662835155 shutdown_connections 2022-06-27T11:47:50.714 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.695+0000 7f4baa665700 1 --2- 172.21.15.195:0/1662835155 >> v2:172.21.15.202:3300/0 conn(0x7f4ba4156100 0x7f4ba40688a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:50.714 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.695+0000 7f4baa665700 1 --2- 172.21.15.195:0/1662835155 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4ba406a440 0x7f4ba40c4fa0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:50.714 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.696+0000 7f4baa665700 1 -- 172.21.15.195:0/1662835155 >> 172.21.15.195:0/1662835155 conn(0x7f4ba40c9b00 msgr2=0x7f4ba40c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:50.715 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.696+0000 7f4baa665700 1 -- 172.21.15.195:0/1662835155 shutdown_connections 2022-06-27T11:47:50.715 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.696+0000 7f4baa665700 1 -- 172.21.15.195:0/1662835155 wait complete. 2022-06-27T11:47:50.716 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.697+0000 7f4baa665700 1 Processor -- start 2022-06-27T11:47:50.716 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.697+0000 7f4baa665700 1 -- start start 2022-06-27T11:47:50.717 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.697+0000 7f4baa665700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4ba406a440 0x7f4ba4147f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:50.717 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.697+0000 7f4baa665700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4ba4156100 0x7f4ba41418c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:50.717 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.697+0000 7f4baa665700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4ba41564d0 0x7f4ba416a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:50.718 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.698+0000 7f4baa665700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba414d100 con 0x7f4ba41564d0 2022-06-27T11:47:50.718 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.698+0000 7f4baa665700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba41588f0 con 0x7f4ba406a440 2022-06-27T11:47:50.718 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.698+0000 7f4baa665700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba40c2030 con 0x7f4ba4156100 2022-06-27T11:47:50.719 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.698+0000 7f4ba8c02700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4ba41564d0 0x7f4ba416a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:50.719 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.698+0000 7f4ba8c02700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4ba41564d0 0x7f4ba416a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41118/0 (socket says 172.21.15.195:41118) 2022-06-27T11:47:50.719 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.698+0000 7f4ba8c02700 1 -- 172.21.15.195:0/2442440571 learned_addr learned my addr 172.21.15.195:0/2442440571 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:47:50.720 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.698+0000 7f4ba3fff700 1 --2- 172.21.15.195:0/2442440571 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4ba406a440 0x7f4ba4147f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:50.720 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.698+0000 7f4ba37fe700 1 --2- 172.21.15.195:0/2442440571 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4ba4156100 0x7f4ba41418c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:50.720 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.698+0000 7f4ba8c02700 1 -- 172.21.15.195:0/2442440571 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4ba4156100 msgr2=0x7f4ba41418c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:50.721 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.698+0000 7f4ba8c02700 1 --2- 172.21.15.195:0/2442440571 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4ba4156100 0x7f4ba41418c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:50.721 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.698+0000 7f4ba8c02700 1 -- 172.21.15.195:0/2442440571 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4ba406a440 msgr2=0x7f4ba4147f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:50.722 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.698+0000 7f4ba8c02700 1 --2- 172.21.15.195:0/2442440571 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4ba406a440 0x7f4ba4147f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:50.722 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.698+0000 7f4ba8c02700 1 -- 172.21.15.195:0/2442440571 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b98009350 con 0x7f4ba41564d0 2022-06-27T11:47:50.722 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.698+0000 7f4ba8c02700 1 --2- 172.21.15.195:0/2442440571 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4ba41564d0 0x7f4ba416a0d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c00c790 tx=0x7f4b9c0015b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:50.722 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.698+0000 7f4b8bfff700 1 -- 172.21.15.195:0/2442440571 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b9c005cb0 con 0x7f4ba41564d0 2022-06-27T11:47:50.722 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.698+0000 7f4b8bfff700 1 -- 172.21.15.195:0/2442440571 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b9c005e10 con 0x7f4ba41564d0 2022-06-27T11:47:50.723 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.698+0000 7f4baa665700 1 -- 172.21.15.195:0/2442440571 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b9400a480 con 0x7f4ba41564d0 2022-06-27T11:47:50.723 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.698+0000 7f4b8bfff700 1 -- 172.21.15.195:0/2442440571 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b9c0072e0 con 0x7f4ba41564d0 2022-06-27T11:47:50.723 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.698+0000 7f4baa665700 1 -- 172.21.15.195:0/2442440571 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ba414be10 con 0x7f4ba41564d0 2022-06-27T11:47:50.723 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.699+0000 7f4baa665700 1 -- 172.21.15.195:0/2442440571 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ba40722c0 con 0x7f4ba41564d0 2022-06-27T11:47:50.724 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.700+0000 7f4b8bfff700 1 -- 172.21.15.195:0/2442440571 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f4b9c019030 con 0x7f4ba41564d0 2022-06-27T11:47:50.724 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.700+0000 7f4b8bfff700 1 --2- 172.21.15.195:0/2442440571 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4b84063d00 0x7f4b840661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:50.724 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.700+0000 7f4b8bfff700 1 -- 172.21.15.195:0/2442440571 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f4b9c07f860 con 0x7f4ba41564d0 2022-06-27T11:47:50.724 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.700+0000 7f4ba3fff700 1 --2- 172.21.15.195:0/2442440571 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4b84063d00 0x7f4b840661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:50.724 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.700+0000 7f4ba3fff700 1 --2- 172.21.15.195:0/2442440571 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4b84063d00 0x7f4b840661b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4b98006fd0 tx=0x7f4b980090f0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:50.725 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.704+0000 7f4b8bfff700 1 -- 172.21.15.195:0/2442440571 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f4b9c049c90 con 0x7f4ba41564d0 2022-06-27T11:47:50.882 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.882+0000 7f4baa665700 1 -- 172.21.15.195:0/2442440571 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f4ba40c6640 con 0x7f4ba41564d0 2022-06-27T11:47:50.954 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.955+0000 7f4b8bfff700 1 -- 172.21.15.195:0/2442440571 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v18) v1 ==== 138+0+0 (secure 0 0 0) 0x7f4b9c017020 con 0x7f4ba41564d0 2022-06-27T11:47:50.957 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.958+0000 7f4baa665700 1 -- 172.21.15.195:0/2442440571 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4b84063d00 msgr2=0x7f4b840661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:50.958 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.958+0000 7f4baa665700 1 --2- 172.21.15.195:0/2442440571 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4b84063d00 0x7f4b840661b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4b98006fd0 tx=0x7f4b980090f0 comp rx=0 tx=0).stop 2022-06-27T11:47:50.958 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.958+0000 7f4baa665700 1 -- 172.21.15.195:0/2442440571 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4ba41564d0 msgr2=0x7f4ba416a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:50.958 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.958+0000 7f4baa665700 1 --2- 172.21.15.195:0/2442440571 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4ba41564d0 0x7f4ba416a0d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c00c790 tx=0x7f4b9c0015b0 comp rx=0 tx=0).stop 2022-06-27T11:47:50.959 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.959+0000 7f4baa665700 1 -- 172.21.15.195:0/2442440571 shutdown_connections 2022-06-27T11:47:50.960 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.959+0000 7f4baa665700 1 --2- 172.21.15.195:0/2442440571 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4b84063d00 0x7f4b840661b0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:50.960 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.959+0000 7f4baa665700 1 --2- 172.21.15.195:0/2442440571 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4ba41564d0 0x7f4ba416a0d0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:50.960 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.959+0000 7f4baa665700 1 --2- 172.21.15.195:0/2442440571 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4ba406a440 0x7f4ba4147f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:50.960 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.959+0000 7f4baa665700 1 --2- 172.21.15.195:0/2442440571 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4ba4156100 0x7f4ba41418c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:50.961 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.959+0000 7f4baa665700 1 -- 172.21.15.195:0/2442440571 >> 172.21.15.195:0/2442440571 conn(0x7f4ba40c9b00 msgr2=0x7f4ba40c0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:50.961 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.959+0000 7f4baa665700 1 -- 172.21.15.195:0/2442440571 shutdown_connections 2022-06-27T11:47:50.961 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:50.960+0000 7f4baa665700 1 -- 172.21.15.195:0/2442440571 wait complete. 2022-06-27T11:47:50.962 INFO:teuthology.orchestra.run.smithi195.stderr:pool 'cephfs_metadata' created 2022-06-27T11:47:50.973 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2022-06-27T11:47:51.100 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.101+0000 7fe6380dd700 1 Processor -- start 2022-06-27T11:47:51.101 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.101+0000 7fe6380dd700 1 -- start start 2022-06-27T11:47:51.104 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.102+0000 7fe6380dd700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fe630157230 0x7fe630154ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:51.104 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.102+0000 7fe6380dd700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe63006d0e0 0x7fe63006fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:51.104 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.102+0000 7fe6380dd700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe630154630 con 0x7fe630155000 2022-06-27T11:47:51.104 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.102+0000 7fe6380dd700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6300c2680 con 0x7fe630157230 2022-06-27T11:47:51.105 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.102+0000 7fe6380dd700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6300c4cd0 con 0x7fe63006d0e0 2022-06-27T11:47:51.105 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.102+0000 7fe63667a700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe63006d0e0 0x7fe63006fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:51.105 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.102+0000 7fe635e79700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fe630157230 0x7fe630154ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:51.105 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.102+0000 7fe63667a700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe63006d0e0 0x7fe63006fda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3301/0 says I am v2:172.21.15.195:33882/0 (socket says 172.21.15.195:33882) 2022-06-27T11:47:51.106 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.102+0000 7fe635e79700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fe630157230 0x7fe630154ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3300/0 says I am v2:172.21.15.195:39248/0 (socket says 172.21.15.195:39248) 2022-06-27T11:47:51.106 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.102+0000 7fe63667a700 1 -- 172.21.15.195:0/2217572840 learned_addr learned my addr 172.21.15.195:0/2217572840 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:47:51.106 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.102+0000 7fe634e77700 1 -- 172.21.15.195:0/2217572840 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1175048623 0 0) 0x7fe630154630 con 0x7fe630155000 2022-06-27T11:47:51.106 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.102+0000 7fe634e77700 1 -- 172.21.15.195:0/2217572840 --> v1:172.21.15.202:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe618003430 con 0x7fe630155000 2022-06-27T11:47:51.106 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.102+0000 7fe63667a700 1 -- 172.21.15.195:0/2217572840 >> v2:172.21.15.202:3300/0 conn(0x7fe630157230 msgr2=0x7fe630154ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:47:51.107 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.102+0000 7fe63667a700 1 --2- 172.21.15.195:0/2217572840 >> v2:172.21.15.202:3300/0 conn(0x7fe630157230 0x7fe630154ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:51.107 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.102+0000 7fe63667a700 1 -- 172.21.15.195:0/2217572840 >> v1:172.21.15.202:6789/0 conn(0x7fe630155000 legacy=0x7fe63006f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:51.107 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.102+0000 7fe63667a700 1 -- 172.21.15.195:0/2217572840 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6301557e0 con 0x7fe63006d0e0 2022-06-27T11:47:51.107 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.102+0000 7fe63667a700 1 --2- 172.21.15.195:0/2217572840 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe63006d0e0 0x7fe63006fda0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe628009f20 tx=0x7fe628009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31e21ff321b74bb5 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:51.108 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.102+0000 7fe634e77700 1 -- 172.21.15.195:0/2217572840 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe62800e070 con 0x7fe63006d0e0 2022-06-27T11:47:51.108 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.102+0000 7fe634e77700 1 -- 172.21.15.195:0/2217572840 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe62800b970 con 0x7fe63006d0e0 2022-06-27T11:47:51.109 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.102+0000 7fe634e77700 1 -- 172.21.15.195:0/2217572840 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6280092e0 con 0x7fe63006d0e0 2022-06-27T11:47:51.109 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.103+0000 7fe6380dd700 1 -- 172.21.15.195:0/2217572840 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe63006d0e0 msgr2=0x7fe63006fda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:51.110 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.103+0000 7fe6380dd700 1 --2- 172.21.15.195:0/2217572840 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe63006d0e0 0x7fe63006fda0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe628009f20 tx=0x7fe628009840 comp rx=0 tx=0).stop 2022-06-27T11:47:51.110 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.103+0000 7fe6380dd700 1 -- 172.21.15.195:0/2217572840 shutdown_connections 2022-06-27T11:47:51.110 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.103+0000 7fe6380dd700 1 --2- 172.21.15.195:0/2217572840 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe63006d0e0 0x7fe63006fda0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:51.111 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.103+0000 7fe6380dd700 1 --2- 172.21.15.195:0/2217572840 >> v2:172.21.15.202:3300/0 conn(0x7fe630157230 0x7fe630154ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:51.111 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.103+0000 7fe6380dd700 1 -- 172.21.15.195:0/2217572840 >> 172.21.15.195:0/2217572840 conn(0x7fe63006c250 msgr2=0x7fe630063af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:51.112 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.103+0000 7fe6380dd700 1 -- 172.21.15.195:0/2217572840 shutdown_connections 2022-06-27T11:47:51.112 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.103+0000 7fe6380dd700 1 -- 172.21.15.195:0/2217572840 wait complete. 2022-06-27T11:47:51.113 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.104+0000 7fe6380dd700 1 Processor -- start 2022-06-27T11:47:51.113 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.104+0000 7fe6380dd700 1 -- start start 2022-06-27T11:47:51.119 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.105+0000 7fe6380dd700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe63006d0e0 0x7fe6300c62c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:51.120 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.105+0000 7fe6380dd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe630155000 0x7fe6300c08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:51.120 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.105+0000 7fe6380dd700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe630157230 0x7fe6300c0de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:51.121 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.105+0000 7fe6380dd700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe630158820 con 0x7fe630155000 2022-06-27T11:47:51.121 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.105+0000 7fe6380dd700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6300bf710 con 0x7fe63006d0e0 2022-06-27T11:47:51.121 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.105+0000 7fe6380dd700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6300c0410 con 0x7fe630157230 2022-06-27T11:47:51.122 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.105+0000 7fe635e79700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe63006d0e0 0x7fe6300c62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:51.122 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.105+0000 7fe63667a700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe630157230 0x7fe6300c0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:51.123 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.105+0000 7fe635e79700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe63006d0e0 0x7fe6300c62c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3300/0 says I am v2:172.21.15.195:39254/0 (socket says 172.21.15.195:39254) 2022-06-27T11:47:51.123 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.105+0000 7fe63667a700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe630157230 0x7fe6300c0de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3301/0 says I am v2:172.21.15.195:33886/0 (socket says 172.21.15.195:33886) 2022-06-27T11:47:51.123 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.105+0000 7fe635e79700 1 -- 172.21.15.195:0/4008533221 learned_addr learned my addr 172.21.15.195:0/4008533221 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:47:51.124 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.105+0000 7fe635678700 1 --2- 172.21.15.195:0/4008533221 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe630155000 0x7fe6300c08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:51.124 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.105+0000 7fe63667a700 1 -- 172.21.15.195:0/4008533221 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe63006d0e0 msgr2=0x7fe6300c62c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:51.125 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.105+0000 7fe63667a700 1 --2- 172.21.15.195:0/4008533221 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe63006d0e0 0x7fe6300c62c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:51.125 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.105+0000 7fe63667a700 1 -- 172.21.15.195:0/4008533221 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe630155000 msgr2=0x7fe6300c08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:51.126 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.105+0000 7fe63667a700 1 --2- 172.21.15.195:0/4008533221 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe630155000 0x7fe6300c08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:51.126 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.105+0000 7fe63667a700 1 -- 172.21.15.195:0/4008533221 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe628009ae0 con 0x7fe630157230 2022-06-27T11:47:51.126 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.105+0000 7fe635e79700 1 --2- 172.21.15.195:0/4008533221 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe63006d0e0 0x7fe6300c62c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:47:51.127 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.106+0000 7fe63667a700 1 --2- 172.21.15.195:0/4008533221 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe630157230 0x7fe6300c0de0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe628015000 tx=0x7fe628018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:51.127 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.106+0000 7fe61dffb700 1 -- 172.21.15.195:0/4008533221 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6280092e0 con 0x7fe630157230 2022-06-27T11:47:51.128 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.106+0000 7fe61dffb700 1 -- 172.21.15.195:0/4008533221 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe628004040 con 0x7fe630157230 2022-06-27T11:47:51.129 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.106+0000 7fe6380dd700 1 -- 172.21.15.195:0/4008533221 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6300c9f70 con 0x7fe630157230 2022-06-27T11:47:51.129 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.106+0000 7fe61dffb700 1 -- 172.21.15.195:0/4008533221 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe628004400 con 0x7fe630157230 2022-06-27T11:47:51.129 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.106+0000 7fe6380dd700 1 -- 172.21.15.195:0/4008533221 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe630164880 con 0x7fe630157230 2022-06-27T11:47:51.130 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.107+0000 7fe6380dd700 1 -- 172.21.15.195:0/4008533221 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6300722c0 con 0x7fe630157230 2022-06-27T11:47:51.130 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.107+0000 7fe61dffb700 1 -- 172.21.15.195:0/4008533221 <== mon.2 v2:172.21.15.202:3301/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fe6280047b0 con 0x7fe630157230 2022-06-27T11:47:51.131 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.108+0000 7fe61dffb700 1 --2- 172.21.15.195:0/4008533221 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe610063d00 0x7fe6100661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:51.131 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.108+0000 7fe61dffb700 1 -- 172.21.15.195:0/4008533221 <== mon.2 v2:172.21.15.202:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5576+0+0 (secure 0 0 0) 0x7fe628082410 con 0x7fe630157230 2022-06-27T11:47:51.131 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.108+0000 7fe635e79700 1 --2- 172.21.15.195:0/4008533221 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe610063d00 0x7fe6100661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:51.132 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.108+0000 7fe635e79700 1 --2- 172.21.15.195:0/4008533221 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe610063d00 0x7fe6100661b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe62c0113e0 tx=0x7fe62c009060 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:51.132 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.117+0000 7fe61dffb700 1 -- 172.21.15.195:0/4008533221 <== mon.2 v2:172.21.15.202:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fe628051180 con 0x7fe630157230 2022-06-27T11:47:51.294 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.295+0000 7fe6380dd700 1 -- 172.21.15.195:0/4008533221 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fe6300c0550 con 0x7fe630157230 2022-06-27T11:47:51.962 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.962+0000 7fe61dffb700 1 -- 172.21.15.195:0/4008533221 <== mon.2 v2:172.21.15.202:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v19) v1 ==== 170+0+0 (secure 0 0 0) 0x7fe62802a0d0 con 0x7fe630157230 2022-06-27T11:47:51.968 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.969+0000 7fe6380dd700 1 -- 172.21.15.195:0/4008533221 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe610063d00 msgr2=0x7fe6100661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:51.969 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.969+0000 7fe6380dd700 1 --2- 172.21.15.195:0/4008533221 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe610063d00 0x7fe6100661b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe62c0113e0 tx=0x7fe62c009060 comp rx=0 tx=0).stop 2022-06-27T11:47:51.969 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.969+0000 7fe6380dd700 1 -- 172.21.15.195:0/4008533221 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe630157230 msgr2=0x7fe6300c0de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:51.970 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.969+0000 7fe6380dd700 1 --2- 172.21.15.195:0/4008533221 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe630157230 0x7fe6300c0de0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe628015000 tx=0x7fe628018000 comp rx=0 tx=0).stop 2022-06-27T11:47:51.979 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.979+0000 7fe6380dd700 1 -- 172.21.15.195:0/4008533221 shutdown_connections 2022-06-27T11:47:51.980 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.979+0000 7fe6380dd700 1 --2- 172.21.15.195:0/4008533221 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe610063d00 0x7fe6100661b0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:51.981 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.979+0000 7fe6380dd700 1 --2- 172.21.15.195:0/4008533221 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe630157230 0x7fe6300c0de0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:51.981 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.979+0000 7fe6380dd700 1 --2- 172.21.15.195:0/4008533221 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe63006d0e0 0x7fe6300c62c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:51.982 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.979+0000 7fe6380dd700 1 --2- 172.21.15.195:0/4008533221 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe630155000 0x7fe6300c08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:51.982 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.979+0000 7fe6380dd700 1 -- 172.21.15.195:0/4008533221 >> 172.21.15.195:0/4008533221 conn(0x7fe63006c250 msgr2=0x7fe630152dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:51.983 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.979+0000 7fe6380dd700 1 -- 172.21.15.195:0/4008533221 shutdown_connections 2022-06-27T11:47:51.983 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:51.979+0000 7fe6380dd700 1 -- 172.21.15.195:0/4008533221 wait complete. 2022-06-27T11:47:51.984 INFO:teuthology.orchestra.run.smithi195.stderr:pool 'cephfs_data' created 2022-06-27T11:47:51.993 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2022-06-27T11:47:52.118 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.118+0000 7f1f799a5700 1 Processor -- start 2022-06-27T11:47:52.119 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.118+0000 7f1f799a5700 1 -- start start 2022-06-27T11:47:52.120 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.119+0000 7f1f799a5700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1f740c4160 0x7f1f74156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:52.120 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.119+0000 7f1f799a5700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f1f740c39e0 0x7f1f741535a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:52.120 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.119+0000 7f1f799a5700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f740c1a20 con 0x7f1f7406f640 2022-06-27T11:47:52.120 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.119+0000 7f1f799a5700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f740c1130 con 0x7f1f740c4160 2022-06-27T11:47:52.121 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.119+0000 7f1f799a5700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f7406ea90 con 0x7f1f740c39e0 2022-06-27T11:47:52.121 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.119+0000 7f1f727fc700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1f740c4160 0x7f1f74156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:52.121 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.119+0000 7f1f727fc700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1f740c4160 0x7f1f74156ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41138/0 (socket says 172.21.15.195:41138) 2022-06-27T11:47:52.121 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.119+0000 7f1f727fc700 1 -- 172.21.15.195:0/2325535205 learned_addr learned my addr 172.21.15.195:0/2325535205 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:47:52.122 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.119+0000 7f1f737fe700 1 --2- 172.21.15.195:0/2325535205 >> v2:172.21.15.202:3300/0 conn(0x7f1f740c39e0 0x7f1f741535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:52.122 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.120+0000 7f1f71ffb700 1 -- 172.21.15.195:0/2325535205 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3118702652 0 0) 0x7f1f740c1a20 con 0x7f1f7406f640 2022-06-27T11:47:52.122 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.120+0000 7f1f71ffb700 1 -- 172.21.15.195:0/2325535205 --> v1:172.21.15.202:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f60003430 con 0x7f1f7406f640 2022-06-27T11:47:52.137 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.120+0000 7f1f727fc700 1 -- 172.21.15.195:0/2325535205 >> v2:172.21.15.202:3300/0 conn(0x7f1f740c39e0 msgr2=0x7f1f741535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:47:52.138 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.120+0000 7f1f727fc700 1 --2- 172.21.15.195:0/2325535205 >> v2:172.21.15.202:3300/0 conn(0x7f1f740c39e0 0x7f1f741535a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:52.139 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.120+0000 7f1f727fc700 1 -- 172.21.15.195:0/2325535205 >> v1:172.21.15.202:6789/0 conn(0x7f1f7406f640 legacy=0x7f1f740c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:52.139 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.120+0000 7f1f727fc700 1 -- 172.21.15.195:0/2325535205 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f7406ed70 con 0x7f1f740c4160 2022-06-27T11:47:52.139 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.120+0000 7f1f727fc700 1 --2- 172.21.15.195:0/2325535205 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1f740c4160 0x7f1f74156ed0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1f74065ef0 tx=0x7f1f64009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=488ae6260bed49fe server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:52.140 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.120+0000 7f1f71ffb700 1 -- 172.21.15.195:0/2325535205 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f64011630 con 0x7f1f740c4160 2022-06-27T11:47:52.140 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.120+0000 7f1f71ffb700 1 -- 172.21.15.195:0/2325535205 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f64011790 con 0x7f1f740c4160 2022-06-27T11:47:52.141 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.120+0000 7f1f71ffb700 1 -- 172.21.15.195:0/2325535205 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f64011b30 con 0x7f1f740c4160 2022-06-27T11:47:52.141 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.121+0000 7f1f799a5700 1 -- 172.21.15.195:0/2325535205 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1f740c4160 msgr2=0x7f1f74156ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:52.142 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.121+0000 7f1f799a5700 1 --2- 172.21.15.195:0/2325535205 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1f740c4160 0x7f1f74156ed0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1f74065ef0 tx=0x7f1f64009120 comp rx=0 tx=0).stop 2022-06-27T11:47:52.142 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.121+0000 7f1f799a5700 1 -- 172.21.15.195:0/2325535205 shutdown_connections 2022-06-27T11:47:52.142 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.121+0000 7f1f799a5700 1 --2- 172.21.15.195:0/2325535205 >> v2:172.21.15.202:3300/0 conn(0x7f1f740c39e0 0x7f1f741535a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:52.143 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.121+0000 7f1f799a5700 1 --2- 172.21.15.195:0/2325535205 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1f740c4160 0x7f1f74156ed0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:52.143 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.121+0000 7f1f799a5700 1 -- 172.21.15.195:0/2325535205 >> 172.21.15.195:0/2325535205 conn(0x7f1f740c6350 msgr2=0x7f1f740c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:52.143 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.121+0000 7f1f799a5700 1 -- 172.21.15.195:0/2325535205 shutdown_connections 2022-06-27T11:47:52.144 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.121+0000 7f1f799a5700 1 -- 172.21.15.195:0/2325535205 wait complete. 2022-06-27T11:47:52.144 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.122+0000 7f1f799a5700 1 Processor -- start 2022-06-27T11:47:52.145 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.122+0000 7f1f799a5700 1 -- start start 2022-06-27T11:47:52.145 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.123+0000 7f1f799a5700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1f7406f640 0x7f1f74158ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:52.146 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.123+0000 7f1f799a5700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1f740c39e0 0x7f1f74153550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:52.146 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.123+0000 7f1f799a5700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1f740c4160 0x7f1f74166b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:52.146 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.123+0000 7f1f799a5700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f740bbcf0 con 0x7f1f740c4160 2022-06-27T11:47:52.147 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.123+0000 7f1f799a5700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f740bafe0 con 0x7f1f7406f640 2022-06-27T11:47:52.147 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.123+0000 7f1f799a5700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f740703d0 con 0x7f1f740c39e0 2022-06-27T11:47:52.148 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.123+0000 7f1f737fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1f740c4160 0x7f1f74166b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:52.148 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.123+0000 7f1f72ffd700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1f7406f640 0x7f1f74158ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:52.149 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.123+0000 7f1f737fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1f740c4160 0x7f1f74166b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41146/0 (socket says 172.21.15.195:41146) 2022-06-27T11:47:52.149 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.123+0000 7f1f737fe700 1 -- 172.21.15.195:0/1462616880 learned_addr learned my addr 172.21.15.195:0/1462616880 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:47:52.150 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.123+0000 7f1f727fc700 1 --2- 172.21.15.195:0/1462616880 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1f740c39e0 0x7f1f74153550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:52.150 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.123+0000 7f1f737fe700 1 -- 172.21.15.195:0/1462616880 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1f740c39e0 msgr2=0x7f1f74153550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:52.151 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.123+0000 7f1f737fe700 1 --2- 172.21.15.195:0/1462616880 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1f740c39e0 0x7f1f74153550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:52.151 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.123+0000 7f1f737fe700 1 -- 172.21.15.195:0/1462616880 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1f7406f640 msgr2=0x7f1f74158ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:52.151 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.123+0000 7f1f737fe700 1 --2- 172.21.15.195:0/1462616880 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1f7406f640 0x7f1f74158ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:52.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.123+0000 7f1f737fe700 1 -- 172.21.15.195:0/1462616880 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f640093a0 con 0x7f1f740c4160 2022-06-27T11:47:52.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.123+0000 7f1f737fe700 1 --2- 172.21.15.195:0/1462616880 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1f740c4160 0x7f1f74166b20 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c009ec0 tx=0x7f1f6c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:52.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.123+0000 7f1f5affd700 1 -- 172.21.15.195:0/1462616880 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f6c00cc00 con 0x7f1f740c4160 2022-06-27T11:47:52.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.123+0000 7f1f5affd700 1 -- 172.21.15.195:0/1462616880 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f6c00f040 con 0x7f1f740c4160 2022-06-27T11:47:52.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.123+0000 7f1f799a5700 1 -- 172.21.15.195:0/1462616880 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f741595a0 con 0x7f1f740c4160 2022-06-27T11:47:52.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.123+0000 7f1f5affd700 1 -- 172.21.15.195:0/1462616880 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f6c014480 con 0x7f1f740c4160 2022-06-27T11:47:52.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.123+0000 7f1f799a5700 1 -- 172.21.15.195:0/1462616880 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f740ba730 con 0x7f1f740c4160 2022-06-27T11:47:52.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.124+0000 7f1f799a5700 1 -- 172.21.15.195:0/1462616880 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f74166820 con 0x7f1f740c4160 2022-06-27T11:47:52.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.125+0000 7f1f5affd700 1 -- 172.21.15.195:0/1462616880 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f1f6c014830 con 0x7f1f740c4160 2022-06-27T11:47:52.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.125+0000 7f1f5affd700 1 --2- 172.21.15.195:0/1462616880 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1f54063d00 0x7f1f540661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:52.155 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.125+0000 7f1f5affd700 1 -- 172.21.15.195:0/1462616880 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5958+0+0 (secure 0 0 0) 0x7f1f6c081410 con 0x7f1f740c4160 2022-06-27T11:47:52.155 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.125+0000 7f1f72ffd700 1 --2- 172.21.15.195:0/1462616880 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1f54063d00 0x7f1f540661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:52.155 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.125+0000 7f1f72ffd700 1 --2- 172.21.15.195:0/1462616880 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1f54063d00 0x7f1f540661b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1f680018b0 tx=0x7f1f68009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:52.155 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.130+0000 7f1f5affd700 1 -- 172.21.15.195:0/1462616880 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f1f6c050060 con 0x7f1f740c4160 2022-06-27T11:47:52.355 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.356+0000 7f1f799a5700 1 -- 172.21.15.195:0/1462616880 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f1f740722c0 con 0x7f1f740c4160 2022-06-27T11:47:52.361 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.360+0000 7f1f5affd700 1 -- 172.21.15.195:0/1462616880 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 3 and data pool 4 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f1f6c053680 con 0x7f1f740c4160 2022-06-27T11:47:52.363 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.364+0000 7f1f799a5700 1 -- 172.21.15.195:0/1462616880 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1f54063d00 msgr2=0x7f1f540661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:52.364 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.364+0000 7f1f799a5700 1 --2- 172.21.15.195:0/1462616880 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1f54063d00 0x7f1f540661b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1f680018b0 tx=0x7f1f68009040 comp rx=0 tx=0).stop 2022-06-27T11:47:52.364 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.364+0000 7f1f799a5700 1 -- 172.21.15.195:0/1462616880 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1f740c4160 msgr2=0x7f1f74166b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:52.364 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.364+0000 7f1f799a5700 1 --2- 172.21.15.195:0/1462616880 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1f740c4160 0x7f1f74166b20 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c009ec0 tx=0x7f1f6c00b040 comp rx=0 tx=0).stop 2022-06-27T11:47:52.367 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.368+0000 7f1f799a5700 1 -- 172.21.15.195:0/1462616880 shutdown_connections 2022-06-27T11:47:52.367 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.368+0000 7f1f799a5700 1 --2- 172.21.15.195:0/1462616880 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1f54063d00 0x7f1f540661b0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:52.368 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.368+0000 7f1f799a5700 1 --2- 172.21.15.195:0/1462616880 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1f740c4160 0x7f1f74166b20 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:52.368 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.368+0000 7f1f799a5700 1 --2- 172.21.15.195:0/1462616880 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1f7406f640 0x7f1f74158ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:52.368 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.368+0000 7f1f799a5700 1 --2- 172.21.15.195:0/1462616880 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1f740c39e0 0x7f1f74153550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:52.368 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.368+0000 7f1f799a5700 1 -- 172.21.15.195:0/1462616880 >> 172.21.15.195:0/1462616880 conn(0x7f1f740c6350 msgr2=0x7f1f741554c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:52.369 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.369+0000 7f1f799a5700 1 -- 172.21.15.195:0/1462616880 shutdown_connections 2022-06-27T11:47:52.369 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.369+0000 7f1f799a5700 1 -- 172.21.15.195:0/1462616880 wait complete. 2022-06-27T11:47:52.370 INFO:teuthology.orchestra.run.smithi195.stderr:new fs with metadata pool 3 and data pool 4 2022-06-27T11:47:52.390 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-27T11:47:52.531 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.510+0000 7f14a6d94700 1 Processor -- start 2022-06-27T11:47:52.532 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.510+0000 7f14a6d94700 1 -- start start 2022-06-27T11:47:52.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.511+0000 7f14a6d94700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f14a00c4160 0x7f14a0156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:52.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.511+0000 7f14a6d94700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f14a00c39e0 0x7f14a01535a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:52.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.511+0000 7f14a6d94700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14a00c1a20 con 0x7f14a006f640 2022-06-27T11:47:52.534 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.511+0000 7f14a6d94700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14a00c1130 con 0x7f14a00c4160 2022-06-27T11:47:52.534 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.511+0000 7f14a6d94700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14a006ea90 con 0x7f14a00c39e0 2022-06-27T11:47:52.535 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.511+0000 7f14a5331700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f14a00c39e0 0x7f14a01535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:52.535 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.511+0000 7f149ffff700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f14a00c4160 0x7f14a0156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:52.536 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.511+0000 7f14a4b30700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f14a006f640 0x7f14a00c1eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:47338/0 (socket says 172.21.15.195:47338) 2022-06-27T11:47:52.536 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.511+0000 7f14a4b30700 1 -- 172.21.15.195:0/2279658218 learned_addr learned my addr 172.21.15.195:0/2279658218 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:47:52.537 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.511+0000 7f149f7fe700 1 -- 172.21.15.195:0/2279658218 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4209350137 0 0) 0x7f14a00c1a20 con 0x7f14a006f640 2022-06-27T11:47:52.537 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.511+0000 7f149f7fe700 1 -- 172.21.15.195:0/2279658218 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1484003430 con 0x7f14a006f640 2022-06-27T11:47:52.537 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.511+0000 7f14a5331700 1 -- 172.21.15.195:0/2279658218 >> v2:172.21.15.202:3300/0 conn(0x7f14a00c4160 msgr2=0x7f14a0156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:47:52.538 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.511+0000 7f14a5331700 1 --2- 172.21.15.195:0/2279658218 >> v2:172.21.15.202:3300/0 conn(0x7f14a00c4160 0x7f14a0156ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:52.538 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.511+0000 7f14a5331700 1 -- 172.21.15.195:0/2279658218 >> v1:172.21.15.195:6789/0 conn(0x7f14a006f640 legacy=0x7f14a00c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:52.539 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.511+0000 7f14a5331700 1 -- 172.21.15.195:0/2279658218 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14a006ed70 con 0x7f14a00c39e0 2022-06-27T11:47:52.539 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.512+0000 7f14a5331700 1 --2- 172.21.15.195:0/2279658218 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f14a00c39e0 0x7f14a01535a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1498009e60 tx=0x7f1498009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2864c44b3dcd3cb server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:52.540 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.512+0000 7f149f7fe700 1 -- 172.21.15.195:0/2279658218 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f149800e070 con 0x7f14a00c39e0 2022-06-27T11:47:52.540 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.512+0000 7f149f7fe700 1 -- 172.21.15.195:0/2279658218 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f149800bb40 con 0x7f14a00c39e0 2022-06-27T11:47:52.540 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.512+0000 7f149f7fe700 1 -- 172.21.15.195:0/2279658218 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1498012430 con 0x7f14a00c39e0 2022-06-27T11:47:52.541 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.512+0000 7f14a6d94700 1 -- 172.21.15.195:0/2279658218 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f14a00c39e0 msgr2=0x7f14a01535a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:52.541 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.512+0000 7f14a6d94700 1 --2- 172.21.15.195:0/2279658218 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f14a00c39e0 0x7f14a01535a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1498009e60 tx=0x7f1498009780 comp rx=0 tx=0).stop 2022-06-27T11:47:52.541 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.512+0000 7f14a6d94700 1 -- 172.21.15.195:0/2279658218 shutdown_connections 2022-06-27T11:47:52.541 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.512+0000 7f14a6d94700 1 --2- 172.21.15.195:0/2279658218 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f14a00c39e0 0x7f14a01535a0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:52.542 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.512+0000 7f14a6d94700 1 --2- 172.21.15.195:0/2279658218 >> v2:172.21.15.202:3300/0 conn(0x7f14a00c4160 0x7f14a0156ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:52.542 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.512+0000 7f14a6d94700 1 -- 172.21.15.195:0/2279658218 >> 172.21.15.195:0/2279658218 conn(0x7f14a00c6350 msgr2=0x7f14a00c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:52.542 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.512+0000 7f14a6d94700 1 -- 172.21.15.195:0/2279658218 shutdown_connections 2022-06-27T11:47:52.542 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.512+0000 7f14a6d94700 1 -- 172.21.15.195:0/2279658218 wait complete. 2022-06-27T11:47:52.543 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.513+0000 7f14a6d94700 1 Processor -- start 2022-06-27T11:47:52.543 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.514+0000 7f14a6d94700 1 -- start start 2022-06-27T11:47:52.543 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.514+0000 7f14a6d94700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f14a006f640 0x7f14a0152a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:52.543 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.514+0000 7f14a6d94700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f14a00c39e0 0x7f14a0153550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:52.544 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.514+0000 7f14a6d94700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f14a00c4160 0x7f14a01664d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:52.544 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.514+0000 7f14a6d94700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a0151fe0 con 0x7f14a00c4160 2022-06-27T11:47:52.544 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.514+0000 7f14a6d94700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a00c08a0 con 0x7f14a006f640 2022-06-27T11:47:52.544 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.514+0000 7f14a6d94700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14a01510f0 con 0x7f14a00c39e0 2022-06-27T11:47:52.544 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.514+0000 7f14a5331700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f14a00c4160 0x7f14a01664d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:52.545 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.514+0000 7f14a5331700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f14a00c4160 0x7f14a01664d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41160/0 (socket says 172.21.15.195:41160) 2022-06-27T11:47:52.545 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.514+0000 7f14a5331700 1 -- 172.21.15.195:0/1611454721 learned_addr learned my addr 172.21.15.195:0/1611454721 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:47:52.545 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.514+0000 7f14a4b30700 1 --2- 172.21.15.195:0/1611454721 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f14a006f640 0x7f14a0152a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:52.545 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.514+0000 7f149ffff700 1 --2- 172.21.15.195:0/1611454721 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f14a00c39e0 0x7f14a0153550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:52.546 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.515+0000 7f14a5331700 1 -- 172.21.15.195:0/1611454721 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f14a00c39e0 msgr2=0x7f14a0153550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:52.546 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.515+0000 7f14a5331700 1 --2- 172.21.15.195:0/1611454721 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f14a00c39e0 0x7f14a0153550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:52.546 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.515+0000 7f14a5331700 1 -- 172.21.15.195:0/1611454721 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f14a006f640 msgr2=0x7f14a0152a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:52.547 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.515+0000 7f14a5331700 1 --2- 172.21.15.195:0/1611454721 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f14a006f640 0x7f14a0152a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:52.547 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.515+0000 7f14a5331700 1 -- 172.21.15.195:0/1611454721 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1498009a20 con 0x7f14a00c4160 2022-06-27T11:47:52.547 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.515+0000 7f14a5331700 1 --2- 172.21.15.195:0/1611454721 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f14a00c4160 0x7f14a01664d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f14a0065ef0 tx=0x7f1498019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:52.547 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.515+0000 7f149cff9700 1 -- 172.21.15.195:0/1611454721 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14980127a0 con 0x7f14a00c4160 2022-06-27T11:47:52.548 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.515+0000 7f149cff9700 1 -- 172.21.15.195:0/1611454721 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1498012900 con 0x7f14a00c4160 2022-06-27T11:47:52.548 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.515+0000 7f14a6d94700 1 -- 172.21.15.195:0/1611454721 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14a01513b0 con 0x7f14a00c4160 2022-06-27T11:47:52.548 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.515+0000 7f149cff9700 1 -- 172.21.15.195:0/1611454721 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1498012cc0 con 0x7f14a00c4160 2022-06-27T11:47:52.548 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.515+0000 7f14a6d94700 1 -- 172.21.15.195:0/1611454721 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14a0151c70 con 0x7f14a00c4160 2022-06-27T11:47:52.549 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.516+0000 7f14a6d94700 1 -- 172.21.15.195:0/1611454721 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14a0151e30 con 0x7f14a00c4160 2022-06-27T11:47:52.549 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.516+0000 7f149cff9700 1 -- 172.21.15.195:0/1611454721 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f1498004250 con 0x7f14a00c4160 2022-06-27T11:47:52.549 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.517+0000 7f149cff9700 1 --2- 172.21.15.195:0/1611454721 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1488063d00 0x7f14880661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:52.550 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.517+0000 7f149cff9700 1 -- 172.21.15.195:0/1611454721 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f14980830c0 con 0x7f14a00c4160 2022-06-27T11:47:52.551 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.517+0000 7f14a4b30700 1 --2- 172.21.15.195:0/1611454721 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1488063d00 0x7f14880661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:52.551 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.517+0000 7f14a4b30700 1 --2- 172.21.15.195:0/1611454721 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1488063d00 0x7f14880661b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1494007fd0 tx=0x7f1494009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:52.551 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.521+0000 7f149cff9700 1 -- 172.21.15.195:0/1611454721 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f1498051cb0 con 0x7f14a00c4160 2022-06-27T11:47:52.700 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.700+0000 7f14a6d94700 1 -- 172.21.15.195:0/1611454721 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f14a0152370 con 0x7f14a00c4160 2022-06-27T11:47:52.700 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.701+0000 7f149cff9700 1 -- 172.21.15.195:0/1611454721 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16927 (secure 0 0 0) 0x7f14a0152370 con 0x7f14a00c4160 2022-06-27T11:47:52.701 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:47:52.701 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":20,"fsid":"81b3a1d3-6a2b-47e8-8d69-7027cfeae7af","created":"2022-06-27T11:47:35.094715+0000","modified":"2022-06-27T11:47:52.357771+0000","last_up_change":"2022-06-27T11:47:44.777459+0000","last_in_change":"2022-06-27T11:47:37.596818+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2022-06-27T11:47:46.368325+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2022-06-27T11:47:46.386132+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-06-27T11:47:50.901270+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-06-27T11:47:51.314719+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c64bc51f-d4f3-4bc2-806b-da0dc78d51b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6813","nonce":94217},{"type":"v1","addr":"172.21.15.195:6815","nonce":94217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6817","nonce":94217},{"type":"v1","addr":"172.21.15.195:6819","nonce":94217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":94217},{"type":"v1","addr":"172.21.15.195:6823","nonce":94217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6820","nonce":94217},{"type":"v1","addr":"172.21.15.195:6821","nonce":94217}]},"public_addr":"172.21.15.195:6815/94217","cluster_addr":"172.21.15.195:6819/94217","heartbeat_back_addr":"172.21.15.195:6823/94217","heartbeat_front_addr":"172.21.15.195:6821/94217","state":["exists","up"]},{"osd":1,"uuid":"bea71cdf-aa98-493f-9207-66d886a3e30e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":94223},{"type":"v1","addr":"172.21.15.195:6809","nonce":94223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":94223},{"type":"v1","addr":"172.21.15.195:6811","nonce":94223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6816","nonce":94223},{"type":"v1","addr":"172.21.15.195:6818","nonce":94223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":94223},{"type":"v1","addr":"172.21.15.195:6814","nonce":94223}]},"public_addr":"172.21.15.195:6809/94223","cluster_addr":"172.21.15.195:6811/94223","heartbeat_back_addr":"172.21.15.195:6818/94223","heartbeat_front_addr":"172.21.15.195:6814/94223","state":["exists","up"]},{"osd":2,"uuid":"bf6c87a0-a8ea-497a-9ce1-721285888065","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":94221},{"type":"v1","addr":"172.21.15.195:6801","nonce":94221}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":94221},{"type":"v1","addr":"172.21.15.195:6803","nonce":94221}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":94221},{"type":"v1","addr":"172.21.15.195:6807","nonce":94221}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":94221},{"type":"v1","addr":"172.21.15.195:6805","nonce":94221}]},"public_addr":"172.21.15.195:6801/94221","cluster_addr":"172.21.15.195:6803/94221","heartbeat_back_addr":"172.21.15.195:6807/94221","heartbeat_front_addr":"172.21.15.195:6805/94221","state":["exists","up"]},{"osd":3,"uuid":"e8aa47ce-af8c-47e7-9654-870cd53229f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":94232},{"type":"v1","addr":"172.21.15.195:6825","nonce":94232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":94232},{"type":"v1","addr":"172.21.15.195:6827","nonce":94232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":94232},{"type":"v1","addr":"172.21.15.195:6831","nonce":94232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":94232},{"type":"v1","addr":"172.21.15.195:6829","nonce":94232}]},"public_addr":"172.21.15.195:6825/94232","cluster_addr":"172.21.15.195:6827/94232","heartbeat_back_addr":"172.21.15.195:6831/94232","heartbeat_front_addr":"172.21.15.195:6829/94232","state":["exists","up"]},{"osd":4,"uuid":"99ffa553-f149-443a-98f5-68b10f77e085","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6800","nonce":94308},{"type":"v1","addr":"172.21.15.202:6801","nonce":94308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6802","nonce":94308},{"type":"v1","addr":"172.21.15.202:6803","nonce":94308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6806","nonce":94308},{"type":"v1","addr":"172.21.15.202:6807","nonce":94308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6804","nonce":94308},{"type":"v1","addr":"172.21.15.202:6805","nonce":94308}]},"public_addr":"172.21.15.202:6801/94308","cluster_addr":"172.21.15.202:6803/94308","heartbeat_back_addr":"172.21.15.202:6807/94308","heartbeat_front_addr":"172.21.15.202:6805/94308","state":["exists","up"]},{"osd":5,"uuid":"35409ef0-891b-4c01-82e4-26b8e64f502d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6824","nonce":94325},{"type":"v1","addr":"172.21.15.202:6825","nonce":94325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6826","nonce":94325},{"type":"v1","addr":"172.21.15.202:6827","nonce":94325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6830","nonce":94325},{"type":"v1","addr":"172.21.15.202:6831","nonce":94325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6828","nonce":94325},{"type":"v1","addr":"172.21.15.202:6829","nonce":94325}]},"public_addr":"172.21.15.202:6825/94325","cluster_addr":"172.21.15.202:6827/94325","heartbeat_back_addr":"172.21.15.202:6831/94325","heartbeat_front_addr":"172.21.15.202:6829/94325","state":["exists","up"]},{"osd":6,"uuid":"009124c4-8b5f-47d4-bd20-2c19271a2adc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6809","nonce":94309},{"type":"v1","addr":"172.21.15.202:6811","nonce":94309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6813","nonce":94309},{"type":"v1","addr":"172.21.15.202:6815","nonce":94309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6821","nonce":94309},{"type":"v1","addr":"172.21.15.202:6823","nonce":94309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6817","nonce":94309},{"type":"v1","addr":"172.21.15.202:6819","nonce":94309}]},"public_addr":"172.21.15.202:6811/94309","cluster_addr":"172.21.15.202:6815/94309","heartbeat_back_addr":"172.21.15.202:6823/94309","heartbeat_front_addr":"172.21.15.202:6819/94309","state":["exists","up"]},{"osd":7,"uuid":"7773115e-d3c3-4a0c-8801-d8ce63e0e5a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6808","nonce":94310},{"type":"v1","addr":"172.21.15.202:6810","nonce":94310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6812","nonce":94310},{"type":"v1","addr":"172.21.15.202:6814","nonce":94310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6820","nonce":94310},{"type":"v1","addr":"172.21.15.202:6822","nonce":94310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6816","nonce":94310},{"type":"v1","addr":"172.21.15.202:6818","nonce":94310}]},"public_addr":"172.21.15.202:6810/94310","cluster_addr":"172.21.15.202:6814/94310","heartbeat_back_addr":"172.21.15.202:6822/94310","heartbeat_front_addr":"172.21.15.202:6818/94310","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.776182+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.806922+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.237997+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.784371+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.723615+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.820332+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:43.841414+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.758071+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-06-27T11:47:52.702 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.703+0000 7f14a6d94700 1 -- 172.21.15.195:0/1611454721 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1488063d00 msgr2=0x7f14880661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:52.703 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.703+0000 7f14a6d94700 1 --2- 172.21.15.195:0/1611454721 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1488063d00 0x7f14880661b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1494007fd0 tx=0x7f1494009040 comp rx=0 tx=0).stop 2022-06-27T11:47:52.703 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.703+0000 7f14a6d94700 1 -- 172.21.15.195:0/1611454721 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f14a00c4160 msgr2=0x7f14a01664d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:52.703 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.703+0000 7f14a6d94700 1 --2- 172.21.15.195:0/1611454721 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f14a00c4160 0x7f14a01664d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f14a0065ef0 tx=0x7f1498019000 comp rx=0 tx=0).stop 2022-06-27T11:47:52.703 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.703+0000 7f14a6d94700 1 -- 172.21.15.195:0/1611454721 shutdown_connections 2022-06-27T11:47:52.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.703+0000 7f14a6d94700 1 --2- 172.21.15.195:0/1611454721 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1488063d00 0x7f14880661b0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:52.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.703+0000 7f14a6d94700 1 --2- 172.21.15.195:0/1611454721 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f14a00c4160 0x7f14a01664d0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:52.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.703+0000 7f14a6d94700 1 --2- 172.21.15.195:0/1611454721 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f14a006f640 0x7f14a0152a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:52.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.703+0000 7f14a6d94700 1 --2- 172.21.15.195:0/1611454721 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f14a00c39e0 0x7f14a0153550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:52.705 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.703+0000 7f14a6d94700 1 -- 172.21.15.195:0/1611454721 >> 172.21.15.195:0/1611454721 conn(0x7f14a00c6350 msgr2=0x7f14a0155730 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:52.705 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.703+0000 7f14a6d94700 1 -- 172.21.15.195:0/1611454721 shutdown_connections 2022-06-27T11:47:52.705 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.703+0000 7f14a6d94700 1 -- 172.21.15.195:0/1611454721 wait complete. 2022-06-27T11:47:52.715 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-27T11:47:52.835 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.835+0000 7fbcabbbb700 1 Processor -- start 2022-06-27T11:47:52.835 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.836+0000 7fbcabbbb700 1 -- start start 2022-06-27T11:47:52.843 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.836+0000 7fbcabbbb700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fbca41508d0 0x7fbca4063af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:52.843 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.836+0000 7fbcabbbb700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbca4064030 con 0x7fbca4150500 2022-06-27T11:47:52.844 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.836+0000 7fbcabbbb700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbca40c4160 con 0x7fbca406a450 2022-06-27T11:47:52.844 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.836+0000 7fbcabbbb700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbca40c4440 con 0x7fbca41508d0 2022-06-27T11:47:52.844 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.837+0000 7fbcaa158700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fbca41508d0 0x7fbca4063af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:52.845 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.837+0000 7fbca9957700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fbca4150500 0x7fbca40c48d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:47352/0 (socket says 172.21.15.195:47352) 2022-06-27T11:47:52.845 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.837+0000 7fbcaa158700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fbca41508d0 0x7fbca4063af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41168/0 (socket says 172.21.15.195:41168) 2022-06-27T11:47:52.845 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.837+0000 7fbca9957700 1 -- 172.21.15.195:0/375199700 learned_addr learned my addr 172.21.15.195:0/375199700 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:47:52.845 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.837+0000 7fbcaa158700 1 -- 172.21.15.195:0/375199700 >> v1:172.21.15.202:6789/0 conn(0x7fbca406a450 legacy=0x7fbca40bbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:52.846 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.837+0000 7fbcaa158700 1 -- 172.21.15.195:0/375199700 >> v1:172.21.15.195:6789/0 conn(0x7fbca4150500 legacy=0x7fbca40c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:52.846 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.837+0000 7fbcaa158700 1 -- 172.21.15.195:0/375199700 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbca415a800 con 0x7fbca41508d0 2022-06-27T11:47:52.846 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.837+0000 7fbcaa158700 1 --2- 172.21.15.195:0/375199700 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbca41508d0 0x7fbca4063af0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c009e60 tx=0x7fbc9c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0f803e1d1b12c2b server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:52.846 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.837+0000 7fbca8955700 1 -- 172.21.15.195:0/375199700 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc9c00e070 con 0x7fbca41508d0 2022-06-27T11:47:52.847 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.837+0000 7fbca8955700 1 -- 172.21.15.195:0/375199700 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc9c00bc30 con 0x7fbca41508d0 2022-06-27T11:47:52.847 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.837+0000 7fbca8955700 1 -- 172.21.15.195:0/375199700 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc9c012430 con 0x7fbca41508d0 2022-06-27T11:47:52.847 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.838+0000 7fbcabbbb700 1 -- 172.21.15.195:0/375199700 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbca41508d0 msgr2=0x7fbca4063af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:52.848 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.838+0000 7fbcabbbb700 1 --2- 172.21.15.195:0/375199700 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbca41508d0 0x7fbca4063af0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c009e60 tx=0x7fbc9c009780 comp rx=0 tx=0).stop 2022-06-27T11:47:52.848 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.838+0000 7fbcabbbb700 1 -- 172.21.15.195:0/375199700 shutdown_connections 2022-06-27T11:47:52.849 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.838+0000 7fbcabbbb700 1 --2- 172.21.15.195:0/375199700 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbca41508d0 0x7fbca4063af0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:52.849 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.838+0000 7fbcabbbb700 1 -- 172.21.15.195:0/375199700 >> 172.21.15.195:0/375199700 conn(0x7fbca40c9b00 msgr2=0x7fbca40c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:52.849 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.838+0000 7fbcabbbb700 1 -- 172.21.15.195:0/375199700 shutdown_connections 2022-06-27T11:47:52.850 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.838+0000 7fbcabbbb700 1 -- 172.21.15.195:0/375199700 wait complete. 2022-06-27T11:47:52.850 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.839+0000 7fbcabbbb700 1 Processor -- start 2022-06-27T11:47:52.850 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.839+0000 7fbcabbbb700 1 -- start start 2022-06-27T11:47:52.851 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.839+0000 7fbcabbbb700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbca406a450 0x7fbca406de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:52.851 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.839+0000 7fbcabbbb700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbca4150500 0x7fbca40c13c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:52.851 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.840+0000 7fbcabbbb700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbca41508d0 0x7fbca40c1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:52.852 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.840+0000 7fbcabbbb700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbca40c33e0 con 0x7fbca406a450 2022-06-27T11:47:52.852 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.840+0000 7fbcabbbb700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbca4166140 con 0x7fbca41508d0 2022-06-27T11:47:52.852 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.840+0000 7fbcabbbb700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbca4166c80 con 0x7fbca4150500 2022-06-27T11:47:52.852 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.840+0000 7fbca9957700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbca406a450 0x7fbca406de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:52.852 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.840+0000 7fbca9957700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbca406a450 0x7fbca406de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41170/0 (socket says 172.21.15.195:41170) 2022-06-27T11:47:52.853 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.840+0000 7fbca9957700 1 -- 172.21.15.195:0/4258542724 learned_addr learned my addr 172.21.15.195:0/4258542724 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:47:52.853 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.840+0000 7fbcaa158700 1 --2- 172.21.15.195:0/4258542724 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbca41508d0 0x7fbca40c1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:52.854 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.840+0000 7fbca9156700 1 --2- 172.21.15.195:0/4258542724 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbca4150500 0x7fbca40c13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:52.854 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.840+0000 7fbca9957700 1 -- 172.21.15.195:0/4258542724 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbca4150500 msgr2=0x7fbca40c13c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:52.854 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.840+0000 7fbca9957700 1 --2- 172.21.15.195:0/4258542724 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbca4150500 0x7fbca40c13c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:52.855 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.840+0000 7fbca9957700 1 -- 172.21.15.195:0/4258542724 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbca41508d0 msgr2=0x7fbca40c1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:52.855 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.840+0000 7fbca9957700 1 --2- 172.21.15.195:0/4258542724 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbca41508d0 0x7fbca40c1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:52.856 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.840+0000 7fbca9957700 1 -- 172.21.15.195:0/4258542724 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc9c009a20 con 0x7fbca406a450 2022-06-27T11:47:52.856 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.840+0000 7fbca9957700 1 --2- 172.21.15.195:0/4258542724 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbca406a450 0x7fbca406de90 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fbc9400c4c0 tx=0x7fbc94007ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:52.856 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.840+0000 7fbc91ffb700 1 -- 172.21.15.195:0/4258542724 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc9400ce30 con 0x7fbca406a450 2022-06-27T11:47:52.856 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.840+0000 7fbc91ffb700 1 -- 172.21.15.195:0/4258542724 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc940072e0 con 0x7fbca406a450 2022-06-27T11:47:52.857 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.840+0000 7fbc91ffb700 1 -- 172.21.15.195:0/4258542724 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc940056b0 con 0x7fbca406a450 2022-06-27T11:47:52.857 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.840+0000 7fbcabbbb700 1 -- 172.21.15.195:0/4258542724 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbca40c3100 con 0x7fbca406a450 2022-06-27T11:47:52.857 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.841+0000 7fbcabbbb700 1 -- 172.21.15.195:0/4258542724 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbca4166630 con 0x7fbca406a450 2022-06-27T11:47:52.857 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.841+0000 7fbcabbbb700 1 -- 172.21.15.195:0/4258542724 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbca4166f60 con 0x7fbca406a450 2022-06-27T11:47:52.858 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.842+0000 7fbc91ffb700 1 -- 172.21.15.195:0/4258542724 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fbc94019030 con 0x7fbca406a450 2022-06-27T11:47:52.858 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.842+0000 7fbc91ffb700 1 --2- 172.21.15.195:0/4258542724 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbc84063d00 0x7fbc840661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:52.858 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.842+0000 7fbc91ffb700 1 -- 172.21.15.195:0/4258542724 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fbc9407f820 con 0x7fbca406a450 2022-06-27T11:47:52.859 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.842+0000 7fbca9156700 1 --2- 172.21.15.195:0/4258542724 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbc84063d00 0x7fbc840661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:52.859 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.842+0000 7fbca9156700 1 --2- 172.21.15.195:0/4258542724 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbc84063d00 0x7fbc840661b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fbca4154030 tx=0x7fbca0009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:52.860 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:52.846+0000 7fbc91ffb700 1 -- 172.21.15.195:0/4258542724 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fbc9404e410 con 0x7fbca406a450 2022-06-27T11:47:53.026 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.027+0000 7fbcabbbb700 1 -- 172.21.15.195:0/4258542724 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbca4064030 con 0x7fbca406a450 2022-06-27T11:47:53.027 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.028+0000 7fbc91ffb700 1 -- 172.21.15.195:0/4258542724 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16927 (secure 0 0 0) 0x7fbca4064030 con 0x7fbca406a450 2022-06-27T11:47:53.028 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:47:53.028 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":20,"fsid":"81b3a1d3-6a2b-47e8-8d69-7027cfeae7af","created":"2022-06-27T11:47:35.094715+0000","modified":"2022-06-27T11:47:52.357771+0000","last_up_change":"2022-06-27T11:47:44.777459+0000","last_in_change":"2022-06-27T11:47:37.596818+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2022-06-27T11:47:46.368325+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2022-06-27T11:47:46.386132+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-06-27T11:47:50.901270+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-06-27T11:47:51.314719+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c64bc51f-d4f3-4bc2-806b-da0dc78d51b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6813","nonce":94217},{"type":"v1","addr":"172.21.15.195:6815","nonce":94217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6817","nonce":94217},{"type":"v1","addr":"172.21.15.195:6819","nonce":94217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":94217},{"type":"v1","addr":"172.21.15.195:6823","nonce":94217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6820","nonce":94217},{"type":"v1","addr":"172.21.15.195:6821","nonce":94217}]},"public_addr":"172.21.15.195:6815/94217","cluster_addr":"172.21.15.195:6819/94217","heartbeat_back_addr":"172.21.15.195:6823/94217","heartbeat_front_addr":"172.21.15.195:6821/94217","state":["exists","up"]},{"osd":1,"uuid":"bea71cdf-aa98-493f-9207-66d886a3e30e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":94223},{"type":"v1","addr":"172.21.15.195:6809","nonce":94223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":94223},{"type":"v1","addr":"172.21.15.195:6811","nonce":94223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6816","nonce":94223},{"type":"v1","addr":"172.21.15.195:6818","nonce":94223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":94223},{"type":"v1","addr":"172.21.15.195:6814","nonce":94223}]},"public_addr":"172.21.15.195:6809/94223","cluster_addr":"172.21.15.195:6811/94223","heartbeat_back_addr":"172.21.15.195:6818/94223","heartbeat_front_addr":"172.21.15.195:6814/94223","state":["exists","up"]},{"osd":2,"uuid":"bf6c87a0-a8ea-497a-9ce1-721285888065","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":94221},{"type":"v1","addr":"172.21.15.195:6801","nonce":94221}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":94221},{"type":"v1","addr":"172.21.15.195:6803","nonce":94221}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":94221},{"type":"v1","addr":"172.21.15.195:6807","nonce":94221}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":94221},{"type":"v1","addr":"172.21.15.195:6805","nonce":94221}]},"public_addr":"172.21.15.195:6801/94221","cluster_addr":"172.21.15.195:6803/94221","heartbeat_back_addr":"172.21.15.195:6807/94221","heartbeat_front_addr":"172.21.15.195:6805/94221","state":["exists","up"]},{"osd":3,"uuid":"e8aa47ce-af8c-47e7-9654-870cd53229f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":94232},{"type":"v1","addr":"172.21.15.195:6825","nonce":94232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":94232},{"type":"v1","addr":"172.21.15.195:6827","nonce":94232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":94232},{"type":"v1","addr":"172.21.15.195:6831","nonce":94232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":94232},{"type":"v1","addr":"172.21.15.195:6829","nonce":94232}]},"public_addr":"172.21.15.195:6825/94232","cluster_addr":"172.21.15.195:6827/94232","heartbeat_back_addr":"172.21.15.195:6831/94232","heartbeat_front_addr":"172.21.15.195:6829/94232","state":["exists","up"]},{"osd":4,"uuid":"99ffa553-f149-443a-98f5-68b10f77e085","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6800","nonce":94308},{"type":"v1","addr":"172.21.15.202:6801","nonce":94308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6802","nonce":94308},{"type":"v1","addr":"172.21.15.202:6803","nonce":94308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6806","nonce":94308},{"type":"v1","addr":"172.21.15.202:6807","nonce":94308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6804","nonce":94308},{"type":"v1","addr":"172.21.15.202:6805","nonce":94308}]},"public_addr":"172.21.15.202:6801/94308","cluster_addr":"172.21.15.202:6803/94308","heartbeat_back_addr":"172.21.15.202:6807/94308","heartbeat_front_addr":"172.21.15.202:6805/94308","state":["exists","up"]},{"osd":5,"uuid":"35409ef0-891b-4c01-82e4-26b8e64f502d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6824","nonce":94325},{"type":"v1","addr":"172.21.15.202:6825","nonce":94325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6826","nonce":94325},{"type":"v1","addr":"172.21.15.202:6827","nonce":94325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6830","nonce":94325},{"type":"v1","addr":"172.21.15.202:6831","nonce":94325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6828","nonce":94325},{"type":"v1","addr":"172.21.15.202:6829","nonce":94325}]},"public_addr":"172.21.15.202:6825/94325","cluster_addr":"172.21.15.202:6827/94325","heartbeat_back_addr":"172.21.15.202:6831/94325","heartbeat_front_addr":"172.21.15.202:6829/94325","state":["exists","up"]},{"osd":6,"uuid":"009124c4-8b5f-47d4-bd20-2c19271a2adc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6809","nonce":94309},{"type":"v1","addr":"172.21.15.202:6811","nonce":94309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6813","nonce":94309},{"type":"v1","addr":"172.21.15.202:6815","nonce":94309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6821","nonce":94309},{"type":"v1","addr":"172.21.15.202:6823","nonce":94309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6817","nonce":94309},{"type":"v1","addr":"172.21.15.202:6819","nonce":94309}]},"public_addr":"172.21.15.202:6811/94309","cluster_addr":"172.21.15.202:6815/94309","heartbeat_back_addr":"172.21.15.202:6823/94309","heartbeat_front_addr":"172.21.15.202:6819/94309","state":["exists","up"]},{"osd":7,"uuid":"7773115e-d3c3-4a0c-8801-d8ce63e0e5a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6808","nonce":94310},{"type":"v1","addr":"172.21.15.202:6810","nonce":94310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6812","nonce":94310},{"type":"v1","addr":"172.21.15.202:6814","nonce":94310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6820","nonce":94310},{"type":"v1","addr":"172.21.15.202:6822","nonce":94310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6816","nonce":94310},{"type":"v1","addr":"172.21.15.202:6818","nonce":94310}]},"public_addr":"172.21.15.202:6810/94310","cluster_addr":"172.21.15.202:6814/94310","heartbeat_back_addr":"172.21.15.202:6822/94310","heartbeat_front_addr":"172.21.15.202:6818/94310","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.776182+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.806922+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.237997+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.784371+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.723615+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.820332+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:43.841414+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.758071+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-06-27T11:47:53.029 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.030+0000 7fbcabbbb700 1 -- 172.21.15.195:0/4258542724 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbc84063d00 msgr2=0x7fbc840661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:53.030 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.030+0000 7fbcabbbb700 1 --2- 172.21.15.195:0/4258542724 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbc84063d00 0x7fbc840661b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fbca4154030 tx=0x7fbca0009040 comp rx=0 tx=0).stop 2022-06-27T11:47:53.030 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.030+0000 7fbcabbbb700 1 -- 172.21.15.195:0/4258542724 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbca406a450 msgr2=0x7fbca406de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:53.030 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.030+0000 7fbcabbbb700 1 --2- 172.21.15.195:0/4258542724 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbca406a450 0x7fbca406de90 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fbc9400c4c0 tx=0x7fbc94007ca0 comp rx=0 tx=0).stop 2022-06-27T11:47:53.030 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.030+0000 7fbcabbbb700 1 -- 172.21.15.195:0/4258542724 shutdown_connections 2022-06-27T11:47:53.031 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.030+0000 7fbcabbbb700 1 --2- 172.21.15.195:0/4258542724 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbc84063d00 0x7fbc840661b0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:53.031 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.030+0000 7fbcabbbb700 1 --2- 172.21.15.195:0/4258542724 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbca406a450 0x7fbca406de90 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:53.032 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.030+0000 7fbcabbbb700 1 --2- 172.21.15.195:0/4258542724 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbca41508d0 0x7fbca40c1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:53.032 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.030+0000 7fbcabbbb700 1 --2- 172.21.15.195:0/4258542724 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbca4150500 0x7fbca40c13c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:53.033 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.030+0000 7fbcabbbb700 1 -- 172.21.15.195:0/4258542724 >> 172.21.15.195:0/4258542724 conn(0x7fbca40c9b00 msgr2=0x7fbca40c58c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:53.033 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.030+0000 7fbcabbbb700 1 -- 172.21.15.195:0/4258542724 shutdown_connections 2022-06-27T11:47:53.034 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.030+0000 7fbcabbbb700 1 -- 172.21.15.195:0/4258542724 wait complete. 2022-06-27T11:47:53.042 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-06-27T11:47:53.168 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.166+0000 7f1d81b1c700 1 Processor -- start 2022-06-27T11:47:53.169 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.166+0000 7f1d81b1c700 1 -- start start 2022-06-27T11:47:53.169 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.167+0000 7f1d81b1c700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1d7c150500 0x7f1d7c063af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:53.169 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.167+0000 7f1d81b1c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1d7c06a450 0x7f1d7c0c48d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:53.170 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.167+0000 7f1d81b1c700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d7c064030 con 0x7f1d7c1508d0 2022-06-27T11:47:53.170 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.167+0000 7f1d81b1c700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d7c0c4160 con 0x7f1d7c06a450 2022-06-27T11:47:53.170 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.167+0000 7f1d81b1c700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7c0c4440 con 0x7f1d7c150500 2022-06-27T11:47:53.170 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.167+0000 7f1d7affd700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1d7c06a450 0x7f1d7c0c48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:53.171 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.167+0000 7f1d7affd700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1d7c06a450 0x7f1d7c0c48d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41180/0 (socket says 172.21.15.195:41180) 2022-06-27T11:47:53.171 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.167+0000 7f1d7affd700 1 -- 172.21.15.195:0/189149426 learned_addr learned my addr 172.21.15.195:0/189149426 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:47:53.171 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.167+0000 7f1d7b7fe700 1 --2- 172.21.15.195:0/189149426 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1d7c150500 0x7f1d7c063af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:53.172 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.167+0000 7f1d7affd700 1 -- 172.21.15.195:0/189149426 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1d7c150500 msgr2=0x7f1d7c063af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:47:53.172 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.167+0000 7f1d7affd700 1 --2- 172.21.15.195:0/189149426 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1d7c150500 0x7f1d7c063af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:53.173 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.167+0000 7f1d7affd700 1 -- 172.21.15.195:0/189149426 >> v1:172.21.15.202:6789/0 conn(0x7f1d7c1508d0 legacy=0x7f1d7c0bbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:53.173 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.167+0000 7f1d7affd700 1 -- 172.21.15.195:0/189149426 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d7c15a800 con 0x7f1d7c06a450 2022-06-27T11:47:53.173 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.167+0000 7f1d7affd700 1 --2- 172.21.15.195:0/189149426 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1d7c06a450 0x7f1d7c0c48d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1d64009860 tx=0x7f1d64009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c33ad05b87a265b2 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:53.173 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.167+0000 7f1d7a7fc700 1 -- 172.21.15.195:0/189149426 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d64011630 con 0x7f1d7c06a450 2022-06-27T11:47:53.174 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.167+0000 7f1d7a7fc700 1 -- 172.21.15.195:0/189149426 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d64011790 con 0x7f1d7c06a450 2022-06-27T11:47:53.174 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.167+0000 7f1d7a7fc700 1 -- 172.21.15.195:0/189149426 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d64011b30 con 0x7f1d7c06a450 2022-06-27T11:47:53.174 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.168+0000 7f1d81b1c700 1 -- 172.21.15.195:0/189149426 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1d7c06a450 msgr2=0x7f1d7c0c48d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:53.174 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.168+0000 7f1d81b1c700 1 --2- 172.21.15.195:0/189149426 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1d7c06a450 0x7f1d7c0c48d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1d64009860 tx=0x7f1d64009120 comp rx=0 tx=0).stop 2022-06-27T11:47:53.175 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.168+0000 7f1d81b1c700 1 -- 172.21.15.195:0/189149426 shutdown_connections 2022-06-27T11:47:53.175 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.168+0000 7f1d81b1c700 1 --2- 172.21.15.195:0/189149426 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1d7c150500 0x7f1d7c063af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:53.175 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.168+0000 7f1d81b1c700 1 --2- 172.21.15.195:0/189149426 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1d7c06a450 0x7f1d7c0c48d0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:53.175 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.168+0000 7f1d81b1c700 1 -- 172.21.15.195:0/189149426 >> 172.21.15.195:0/189149426 conn(0x7f1d7c0c9b00 msgr2=0x7f1d7c0c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:53.176 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.168+0000 7f1d81b1c700 1 -- 172.21.15.195:0/189149426 shutdown_connections 2022-06-27T11:47:53.176 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.168+0000 7f1d81b1c700 1 -- 172.21.15.195:0/189149426 wait complete. 2022-06-27T11:47:53.179 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.169+0000 7f1d81b1c700 1 Processor -- start 2022-06-27T11:47:53.180 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.169+0000 7f1d81b1c700 1 -- start start 2022-06-27T11:47:53.180 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.170+0000 7f1d81b1c700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1d7c06a450 0x7f1d7c166f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:53.180 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.170+0000 7f1d81b1c700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1d7c150500 0x7f1d7c0c25b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:53.180 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.170+0000 7f1d81b1c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1d7c1508d0 0x7f1d7c0c50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:53.181 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.170+0000 7f1d81b1c700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7c1674c0 con 0x7f1d7c1508d0 2022-06-27T11:47:53.181 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.170+0000 7f1d81b1c700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7c0c1de0 con 0x7f1d7c06a450 2022-06-27T11:47:53.181 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.170+0000 7f1d81b1c700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7c0c31b0 con 0x7f1d7c150500 2022-06-27T11:47:53.181 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.170+0000 7f1d7bfff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1d7c1508d0 0x7f1d7c0c50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:53.182 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.170+0000 7f1d7bfff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1d7c1508d0 0x7f1d7c0c50d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41188/0 (socket says 172.21.15.195:41188) 2022-06-27T11:47:53.182 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.170+0000 7f1d7bfff700 1 -- 172.21.15.195:0/2906001503 learned_addr learned my addr 172.21.15.195:0/2906001503 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:47:53.182 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.170+0000 7f1d7b7fe700 1 --2- 172.21.15.195:0/2906001503 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1d7c06a450 0x7f1d7c166f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:53.182 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.170+0000 7f1d7affd700 1 --2- 172.21.15.195:0/2906001503 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1d7c150500 0x7f1d7c0c25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:53.183 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.170+0000 7f1d7bfff700 1 -- 172.21.15.195:0/2906001503 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1d7c150500 msgr2=0x7f1d7c0c25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:53.183 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.170+0000 7f1d7bfff700 1 --2- 172.21.15.195:0/2906001503 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1d7c150500 0x7f1d7c0c25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:53.183 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.170+0000 7f1d7bfff700 1 -- 172.21.15.195:0/2906001503 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1d7c06a450 msgr2=0x7f1d7c166f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:53.183 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.170+0000 7f1d7bfff700 1 --2- 172.21.15.195:0/2906001503 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1d7c06a450 0x7f1d7c166f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:53.184 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.170+0000 7f1d7bfff700 1 -- 172.21.15.195:0/2906001503 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d640093a0 con 0x7f1d7c1508d0 2022-06-27T11:47:53.184 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.171+0000 7f1d7bfff700 1 --2- 172.21.15.195:0/2906001503 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1d7c1508d0 0x7f1d7c0c50d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1d7400c470 tx=0x7f1d74007db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:53.184 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.171+0000 7f1d737fe700 1 -- 172.21.15.195:0/2906001503 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d740050d0 con 0x7f1d7c1508d0 2022-06-27T11:47:53.184 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.171+0000 7f1d737fe700 1 -- 172.21.15.195:0/2906001503 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d74005230 con 0x7f1d7c1508d0 2022-06-27T11:47:53.184 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.171+0000 7f1d81b1c700 1 -- 172.21.15.195:0/2906001503 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d7c0c2ed0 con 0x7f1d7c1508d0 2022-06-27T11:47:53.185 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.171+0000 7f1d737fe700 1 -- 172.21.15.195:0/2906001503 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d74003730 con 0x7f1d7c1508d0 2022-06-27T11:47:53.185 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.171+0000 7f1d81b1c700 1 -- 172.21.15.195:0/2906001503 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d7c0c1790 con 0x7f1d7c1508d0 2022-06-27T11:47:53.185 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.172+0000 7f1d81b1c700 1 -- 172.21.15.195:0/2906001503 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d7c0722c0 con 0x7f1d7c1508d0 2022-06-27T11:47:53.185 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.172+0000 7f1d737fe700 1 -- 172.21.15.195:0/2906001503 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f1d74019030 con 0x7f1d7c1508d0 2022-06-27T11:47:53.186 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.172+0000 7f1d737fe700 1 --2- 172.21.15.195:0/2906001503 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1d60063d00 0x7f1d600661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:53.186 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.172+0000 7f1d737fe700 1 -- 172.21.15.195:0/2906001503 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f1d7407f640 con 0x7f1d7c1508d0 2022-06-27T11:47:53.186 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.172+0000 7f1d7b7fe700 1 --2- 172.21.15.195:0/2906001503 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1d60063d00 0x7f1d600661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:53.186 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.173+0000 7f1d7b7fe700 1 --2- 172.21.15.195:0/2906001503 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1d60063d00 0x7f1d600661b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c00f5b0 tx=0x7f1d6c00b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:53.187 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.176+0000 7f1d737fe700 1 -- 172.21.15.195:0/2906001503 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f1d7404e230 con 0x7f1d7c1508d0 2022-06-27T11:47:53.398 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:53.398+0000 7f1d81b1c700 1 -- 172.21.15.195:0/2906001503 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f1d7c064030 con 0x7f1d7c1508d0 2022-06-27T11:47:54.362 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.362+0000 7f1d737fe700 1 -- 172.21.15.195:0/2906001503 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v5) v1 ==== 118+0+0 (secure 0 0 0) 0x7f1d7c064030 con 0x7f1d7c1508d0 2022-06-27T11:47:54.365 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.365+0000 7f1d81b1c700 1 -- 172.21.15.195:0/2906001503 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1d60063d00 msgr2=0x7f1d600661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:54.366 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.365+0000 7f1d81b1c700 1 --2- 172.21.15.195:0/2906001503 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1d60063d00 0x7f1d600661b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c00f5b0 tx=0x7f1d6c00b040 comp rx=0 tx=0).stop 2022-06-27T11:47:54.366 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.365+0000 7f1d81b1c700 1 -- 172.21.15.195:0/2906001503 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1d7c1508d0 msgr2=0x7f1d7c0c50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:54.366 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.365+0000 7f1d81b1c700 1 --2- 172.21.15.195:0/2906001503 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1d7c1508d0 0x7f1d7c0c50d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1d7400c470 tx=0x7f1d74007db0 comp rx=0 tx=0).stop 2022-06-27T11:47:54.366 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.365+0000 7f1d81b1c700 1 -- 172.21.15.195:0/2906001503 shutdown_connections 2022-06-27T11:47:54.367 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.365+0000 7f1d81b1c700 1 --2- 172.21.15.195:0/2906001503 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1d60063d00 0x7f1d600661b0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:54.367 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.365+0000 7f1d81b1c700 1 --2- 172.21.15.195:0/2906001503 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1d7c1508d0 0x7f1d7c0c50d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:54.367 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.365+0000 7f1d81b1c700 1 --2- 172.21.15.195:0/2906001503 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1d7c06a450 0x7f1d7c166f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:54.367 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.365+0000 7f1d81b1c700 1 --2- 172.21.15.195:0/2906001503 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1d7c150500 0x7f1d7c0c25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:54.368 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.365+0000 7f1d81b1c700 1 -- 172.21.15.195:0/2906001503 >> 172.21.15.195:0/2906001503 conn(0x7f1d7c0c9b00 msgr2=0x7f1d7c14edc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:54.368 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.366+0000 7f1d81b1c700 1 -- 172.21.15.195:0/2906001503 shutdown_connections 2022-06-27T11:47:54.368 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.366+0000 7f1d81b1c700 1 -- 172.21.15.195:0/2906001503 wait complete. 2022-06-27T11:47:54.380 DEBUG:tasks.cephfs.filesystem:fs_config: {'session_timeout': 300} 2022-06-27T11:47:54.380 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2022-06-27T11:47:54.510 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.510+0000 7f271ddfa700 1 Processor -- start 2022-06-27T11:47:54.510 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.510+0000 7f271ddfa700 1 -- start start 2022-06-27T11:47:54.517 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.511+0000 7f271ddfa700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f271806fef0 0x7f27181570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:54.517 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.511+0000 7f271ddfa700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f27180c34a0 0x7f27180c2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:54.518 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.511+0000 7f271ddfa700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f271806fba0 con 0x7f271806e850 2022-06-27T11:47:54.518 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.511+0000 7f271ddfa700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27181575e0 con 0x7f271806fef0 2022-06-27T11:47:54.519 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.511+0000 7f271ddfa700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f271806f5e0 con 0x7f27180c34a0 2022-06-27T11:47:54.519 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.511+0000 7f2716ffd700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f271806fef0 0x7f27181570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:54.520 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.511+0000 7f27177fe700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f271806e850 0x7f2718069de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:47380/0 (socket says 172.21.15.195:47380) 2022-06-27T11:47:54.520 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.511+0000 7f27177fe700 1 -- 172.21.15.195:0/2257048075 learned_addr learned my addr 172.21.15.195:0/2257048075 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:47:54.521 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.511+0000 7f2717fff700 1 --2- 172.21.15.195:0/2257048075 >> v2:172.21.15.202:3300/0 conn(0x7f27180c34a0 0x7f27180c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:54.522 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.511+0000 7f2716ffd700 1 -- 172.21.15.195:0/2257048075 >> v2:172.21.15.202:3300/0 conn(0x7f27180c34a0 msgr2=0x7f27180c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:47:54.522 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.511+0000 7f2716ffd700 1 --2- 172.21.15.195:0/2257048075 >> v2:172.21.15.202:3300/0 conn(0x7f27180c34a0 0x7f27180c2920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:54.522 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.511+0000 7f2716ffd700 1 -- 172.21.15.195:0/2257048075 >> v1:172.21.15.195:6789/0 conn(0x7f271806e850 legacy=0x7f2718069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:54.523 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.511+0000 7f2716ffd700 1 -- 172.21.15.195:0/2257048075 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f271806f8c0 con 0x7f271806fef0 2022-06-27T11:47:54.523 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.511+0000 7f27167fc700 1 -- 172.21.15.195:0/2257048075 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 193983445 0 0) 0x7f271806fba0 con 0x7f271806e850 2022-06-27T11:47:54.524 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.512+0000 7f2717fff700 1 --2- 172.21.15.195:0/2257048075 >> v2:172.21.15.202:3300/0 conn(0x7f27180c34a0 0x7f27180c2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T11:47:54.524 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.512+0000 7f2716ffd700 1 --2- 172.21.15.195:0/2257048075 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f271806fef0 0x7f27181570a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f27180c1420 tx=0x7f2708009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee88b225e6b0b7cc server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:54.524 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.512+0000 7f27167fc700 1 -- 172.21.15.195:0/2257048075 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2708011630 con 0x7f271806fef0 2022-06-27T11:47:54.525 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.512+0000 7f27167fc700 1 -- 172.21.15.195:0/2257048075 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2708011790 con 0x7f271806fef0 2022-06-27T11:47:54.525 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.512+0000 7f27167fc700 1 -- 172.21.15.195:0/2257048075 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2708011b30 con 0x7f271806fef0 2022-06-27T11:47:54.525 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.512+0000 7f271ddfa700 1 -- 172.21.15.195:0/2257048075 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f271806fef0 msgr2=0x7f27181570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:54.526 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.512+0000 7f271ddfa700 1 --2- 172.21.15.195:0/2257048075 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f271806fef0 0x7f27181570a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f27180c1420 tx=0x7f2708009060 comp rx=0 tx=0).stop 2022-06-27T11:47:54.526 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.512+0000 7f271ddfa700 1 -- 172.21.15.195:0/2257048075 shutdown_connections 2022-06-27T11:47:54.526 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.512+0000 7f271ddfa700 1 --2- 172.21.15.195:0/2257048075 >> v2:172.21.15.202:3300/0 conn(0x7f27180c34a0 0x7f27180c2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:54.526 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.512+0000 7f271ddfa700 1 --2- 172.21.15.195:0/2257048075 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f271806fef0 0x7f27181570a0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:54.526 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.512+0000 7f271ddfa700 1 -- 172.21.15.195:0/2257048075 >> 172.21.15.195:0/2257048075 conn(0x7f27180c6350 msgr2=0x7f27180c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:54.527 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.512+0000 7f271ddfa700 1 -- 172.21.15.195:0/2257048075 shutdown_connections 2022-06-27T11:47:54.527 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.512+0000 7f271ddfa700 1 -- 172.21.15.195:0/2257048075 wait complete. 2022-06-27T11:47:54.527 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.513+0000 7f271ddfa700 1 Processor -- start 2022-06-27T11:47:54.527 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.514+0000 7f271ddfa700 1 -- start start 2022-06-27T11:47:54.528 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.514+0000 7f271ddfa700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f271806e850 0x7f2718153c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:54.528 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.514+0000 7f271ddfa700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f27180c34a0 0x7f271814dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:54.528 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.514+0000 7f271ddfa700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f27180c8400 0x7f271814e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:54.528 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.514+0000 7f271ddfa700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27180bd770 con 0x7f27180c8400 2022-06-27T11:47:54.528 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.514+0000 7f271ddfa700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2718153220 con 0x7f27180c34a0 2022-06-27T11:47:54.529 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.514+0000 7f271ddfa700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27181537e0 con 0x7f271806e850 2022-06-27T11:47:54.529 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.514+0000 7f2717fff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f27180c8400 0x7f271814e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:54.529 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.514+0000 7f2717fff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f27180c8400 0x7f271814e290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41202/0 (socket says 172.21.15.195:41202) 2022-06-27T11:47:54.529 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.514+0000 7f2717fff700 1 -- 172.21.15.195:0/781686849 learned_addr learned my addr 172.21.15.195:0/781686849 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:47:54.529 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.514+0000 7f2716ffd700 1 --2- 172.21.15.195:0/781686849 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f27180c34a0 0x7f271814dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:54.530 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.514+0000 7f27177fe700 1 --2- 172.21.15.195:0/781686849 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f271806e850 0x7f2718153c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:54.530 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.515+0000 7f2717fff700 1 -- 172.21.15.195:0/781686849 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f271806e850 msgr2=0x7f2718153c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:54.531 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.515+0000 7f2717fff700 1 --2- 172.21.15.195:0/781686849 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f271806e850 0x7f2718153c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:54.531 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.515+0000 7f2717fff700 1 -- 172.21.15.195:0/781686849 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f27180c34a0 msgr2=0x7f271814dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:54.531 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.515+0000 7f2717fff700 1 --2- 172.21.15.195:0/781686849 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f27180c34a0 0x7f271814dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:54.531 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.515+0000 7f2717fff700 1 -- 172.21.15.195:0/781686849 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27080092e0 con 0x7f27180c8400 2022-06-27T11:47:54.532 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.515+0000 7f2717fff700 1 --2- 172.21.15.195:0/781686849 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f27180c8400 0x7f271814e290 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f271000d1b0 tx=0x7f2710009a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:54.532 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.515+0000 7f26ff7fe700 1 -- 172.21.15.195:0/781686849 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f271000dcf0 con 0x7f27180c8400 2022-06-27T11:47:54.532 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.515+0000 7f26ff7fe700 1 -- 172.21.15.195:0/781686849 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f271000f040 con 0x7f27180c8400 2022-06-27T11:47:54.532 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.515+0000 7f26ff7fe700 1 -- 172.21.15.195:0/781686849 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2710003af0 con 0x7f27180c8400 2022-06-27T11:47:54.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.515+0000 7f271ddfa700 1 -- 172.21.15.195:0/781686849 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f271814e7d0 con 0x7f27180c8400 2022-06-27T11:47:54.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.515+0000 7f271ddfa700 1 -- 172.21.15.195:0/781686849 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2718164da0 con 0x7f27180c8400 2022-06-27T11:47:54.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.516+0000 7f271ddfa700 1 -- 172.21.15.195:0/781686849 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27180722c0 con 0x7f27180c8400 2022-06-27T11:47:54.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.516+0000 7f26ff7fe700 1 -- 172.21.15.195:0/781686849 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f271001c030 con 0x7f27180c8400 2022-06-27T11:47:54.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.517+0000 7f26ff7fe700 1 --2- 172.21.15.195:0/781686849 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f26f8063d00 0x7f26f80661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:54.534 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.517+0000 7f26ff7fe700 1 -- 172.21.15.195:0/781686849 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f27100808f0 con 0x7f27180c8400 2022-06-27T11:47:54.534 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.517+0000 7f27177fe700 1 --2- 172.21.15.195:0/781686849 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f26f8063d00 0x7f26f80661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:54.534 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.517+0000 7f27177fe700 1 --2- 172.21.15.195:0/781686849 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f26f8063d00 0x7f26f80661b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f270c007fd0 tx=0x7f270c007d10 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:54.535 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.521+0000 7f26ff7fe700 1 -- 172.21.15.195:0/781686849 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f271004f4e0 con 0x7f27180c8400 2022-06-27T11:47:54.739 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:54.740+0000 7f271ddfa700 1 -- 172.21.15.195:0/781686849 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f2718153920 con 0x7f27180c8400 2022-06-27T11:47:55.366 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.366+0000 7f26ff7fe700 1 -- 172.21.15.195:0/781686849 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v6) v1 ==== 122+0+0 (secure 0 0 0) 0x7f271001a070 con 0x7f27180c8400 2022-06-27T11:47:55.369 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.369+0000 7f271ddfa700 1 -- 172.21.15.195:0/781686849 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f26f8063d00 msgr2=0x7f26f80661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:55.369 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.369+0000 7f271ddfa700 1 --2- 172.21.15.195:0/781686849 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f26f8063d00 0x7f26f80661b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f270c007fd0 tx=0x7f270c007d10 comp rx=0 tx=0).stop 2022-06-27T11:47:55.369 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.369+0000 7f271ddfa700 1 -- 172.21.15.195:0/781686849 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f27180c8400 msgr2=0x7f271814e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:55.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.369+0000 7f271ddfa700 1 --2- 172.21.15.195:0/781686849 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f27180c8400 0x7f271814e290 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f271000d1b0 tx=0x7f2710009a50 comp rx=0 tx=0).stop 2022-06-27T11:47:55.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.369+0000 7f271ddfa700 1 -- 172.21.15.195:0/781686849 shutdown_connections 2022-06-27T11:47:55.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.369+0000 7f271ddfa700 1 --2- 172.21.15.195:0/781686849 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f26f8063d00 0x7f26f80661b0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:55.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.369+0000 7f271ddfa700 1 --2- 172.21.15.195:0/781686849 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f27180c8400 0x7f271814e290 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:55.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.369+0000 7f271ddfa700 1 --2- 172.21.15.195:0/781686849 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f271806e850 0x7f2718153c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:55.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.369+0000 7f271ddfa700 1 --2- 172.21.15.195:0/781686849 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f27180c34a0 0x7f271814dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:55.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.369+0000 7f271ddfa700 1 -- 172.21.15.195:0/781686849 >> 172.21.15.195:0/781686849 conn(0x7f27180c6350 msgr2=0x7f27180c2ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:55.372 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.369+0000 7f271ddfa700 1 -- 172.21.15.195:0/781686849 shutdown_connections 2022-06-27T11:47:55.372 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.369+0000 7f271ddfa700 1 -- 172.21.15.195:0/781686849 wait complete. 2022-06-27T11:47:55.382 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2022-06-27T11:47:55.516 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.508+0000 7fbd87add700 1 Processor -- start 2022-06-27T11:47:55.517 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.508+0000 7fbd87add700 1 -- start start 2022-06-27T11:47:55.517 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.509+0000 7fbd87add700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbd80153cb0 0x7fbd80156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:55.517 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.509+0000 7fbd87add700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd800c1a20 con 0x7fbd800c4160 2022-06-27T11:47:55.518 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.509+0000 7fbd87add700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd800c1130 con 0x7fbd8006f640 2022-06-27T11:47:55.518 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.509+0000 7fbd87add700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd8006ea90 con 0x7fbd80153cb0 2022-06-27T11:47:55.519 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.509+0000 7fbd85078700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fbd800c4160 0x7fbd801535a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:47394/0 (socket says 172.21.15.195:47394) 2022-06-27T11:47:55.519 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.509+0000 7fbd85078700 1 -- 172.21.15.195:0/1064891807 learned_addr learned my addr 172.21.15.195:0/1064891807 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:47:55.519 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.509+0000 7fbd8607a700 1 --2- 172.21.15.195:0/1064891807 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbd80153cb0 0x7fbd80156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:55.520 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.509+0000 7fbd84877700 1 -- 172.21.15.195:0/1064891807 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3501679994 0 0) 0x7fbd800c1a20 con 0x7fbd800c4160 2022-06-27T11:47:55.520 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.509+0000 7fbd84877700 1 -- 172.21.15.195:0/1064891807 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd68003430 con 0x7fbd800c4160 2022-06-27T11:47:55.520 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.509+0000 7fbd84877700 1 -- 172.21.15.195:0/1064891807 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3991942808 0 0) 0x7fbd800c1130 con 0x7fbd8006f640 2022-06-27T11:47:55.521 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.509+0000 7fbd8607a700 1 -- 172.21.15.195:0/1064891807 >> v1:172.21.15.202:6789/0 conn(0x7fbd8006f640 legacy=0x7fbd800c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:55.521 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.509+0000 7fbd8607a700 1 -- 172.21.15.195:0/1064891807 >> v1:172.21.15.195:6789/0 conn(0x7fbd800c4160 legacy=0x7fbd801535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:55.522 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.509+0000 7fbd8607a700 1 -- 172.21.15.195:0/1064891807 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd8006ed70 con 0x7fbd80153cb0 2022-06-27T11:47:55.522 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.509+0000 7fbd8607a700 1 --2- 172.21.15.195:0/1064891807 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbd80153cb0 0x7fbd80156ed0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fbd78009e90 tx=0x7fbd78009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14d38c01c5cbd598 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:55.522 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.510+0000 7fbd84877700 1 -- 172.21.15.195:0/1064891807 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd7800e070 con 0x7fbd80153cb0 2022-06-27T11:47:55.522 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.510+0000 7fbd84877700 1 -- 172.21.15.195:0/1064891807 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd7800bcc0 con 0x7fbd80153cb0 2022-06-27T11:47:55.523 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.510+0000 7fbd84877700 1 -- 172.21.15.195:0/1064891807 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd78012430 con 0x7fbd80153cb0 2022-06-27T11:47:55.523 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.510+0000 7fbd87add700 1 -- 172.21.15.195:0/1064891807 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbd80153cb0 msgr2=0x7fbd80156ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:55.523 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.510+0000 7fbd87add700 1 --2- 172.21.15.195:0/1064891807 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbd80153cb0 0x7fbd80156ed0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fbd78009e90 tx=0x7fbd78009780 comp rx=0 tx=0).stop 2022-06-27T11:47:55.524 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.510+0000 7fbd87add700 1 -- 172.21.15.195:0/1064891807 shutdown_connections 2022-06-27T11:47:55.524 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.510+0000 7fbd87add700 1 --2- 172.21.15.195:0/1064891807 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbd80153cb0 0x7fbd80156ed0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:55.524 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.510+0000 7fbd87add700 1 -- 172.21.15.195:0/1064891807 >> 172.21.15.195:0/1064891807 conn(0x7fbd800c6350 msgr2=0x7fbd800c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:55.524 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.510+0000 7fbd87add700 1 -- 172.21.15.195:0/1064891807 shutdown_connections 2022-06-27T11:47:55.525 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.510+0000 7fbd87add700 1 -- 172.21.15.195:0/1064891807 wait complete. 2022-06-27T11:47:55.525 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.511+0000 7fbd87add700 1 Processor -- start 2022-06-27T11:47:55.525 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.511+0000 7fbd87add700 1 -- start start 2022-06-27T11:47:55.525 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.512+0000 7fbd87add700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbd8006f640 0x7fbd800bde90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:55.525 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.512+0000 7fbd87add700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbd800c4160 0x7fbd800baeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:55.526 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.512+0000 7fbd87add700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbd80153cb0 0x7fbd800bcbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:55.526 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.512+0000 7fbd87add700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd800bbe50 con 0x7fbd80153cb0 2022-06-27T11:47:55.527 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.512+0000 7fbd87add700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd800be3d0 con 0x7fbd800c4160 2022-06-27T11:47:55.527 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.512+0000 7fbd87add700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd800bc630 con 0x7fbd8006f640 2022-06-27T11:47:55.527 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.512+0000 7fbd8607a700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbd80153cb0 0x7fbd800bcbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:55.527 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.512+0000 7fbd8607a700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbd80153cb0 0x7fbd800bcbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41214/0 (socket says 172.21.15.195:41214) 2022-06-27T11:47:55.527 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.512+0000 7fbd8607a700 1 -- 172.21.15.195:0/757227713 learned_addr learned my addr 172.21.15.195:0/757227713 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:47:55.528 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.512+0000 7fbd85078700 1 --2- 172.21.15.195:0/757227713 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbd800c4160 0x7fbd800baeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:55.528 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.512+0000 7fbd85879700 1 --2- 172.21.15.195:0/757227713 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbd8006f640 0x7fbd800bde90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:55.529 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.512+0000 7fbd8607a700 1 -- 172.21.15.195:0/757227713 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbd8006f640 msgr2=0x7fbd800bde90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:55.529 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.512+0000 7fbd8607a700 1 --2- 172.21.15.195:0/757227713 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbd8006f640 0x7fbd800bde90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:55.529 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.512+0000 7fbd8607a700 1 -- 172.21.15.195:0/757227713 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbd800c4160 msgr2=0x7fbd800baeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:55.529 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.512+0000 7fbd8607a700 1 --2- 172.21.15.195:0/757227713 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbd800c4160 0x7fbd800baeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:55.530 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.513+0000 7fbd8607a700 1 -- 172.21.15.195:0/757227713 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd78009a20 con 0x7fbd80153cb0 2022-06-27T11:47:55.530 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.513+0000 7fbd8607a700 1 --2- 172.21.15.195:0/757227713 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbd80153cb0 0x7fbd800bcbc0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fbd780099f0 tx=0x7fbd78019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:55.530 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.513+0000 7fbd6dffb700 1 -- 172.21.15.195:0/757227713 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd7801b420 con 0x7fbd80153cb0 2022-06-27T11:47:55.530 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.513+0000 7fbd6dffb700 1 -- 172.21.15.195:0/757227713 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd7801b5c0 con 0x7fbd80153cb0 2022-06-27T11:47:55.531 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.513+0000 7fbd87add700 1 -- 172.21.15.195:0/757227713 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd800c00c0 con 0x7fbd80153cb0 2022-06-27T11:47:55.531 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.513+0000 7fbd6dffb700 1 -- 172.21.15.195:0/757227713 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd7801b980 con 0x7fbd80153cb0 2022-06-27T11:47:55.531 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.513+0000 7fbd87add700 1 -- 172.21.15.195:0/757227713 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd800ba3d0 con 0x7fbd80153cb0 2022-06-27T11:47:55.531 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.514+0000 7fbd87add700 1 -- 172.21.15.195:0/757227713 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd800722c0 con 0x7fbd80153cb0 2022-06-27T11:47:55.531 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.514+0000 7fbd6dffb700 1 -- 172.21.15.195:0/757227713 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fbd7802c030 con 0x7fbd80153cb0 2022-06-27T11:47:55.532 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.515+0000 7fbd6dffb700 1 --2- 172.21.15.195:0/757227713 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbd60063d30 0x7fbd600661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:55.532 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.515+0000 7fbd6dffb700 1 -- 172.21.15.195:0/757227713 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fbd78082590 con 0x7fbd80153cb0 2022-06-27T11:47:55.532 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.515+0000 7fbd85879700 1 --2- 172.21.15.195:0/757227713 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbd60063d30 0x7fbd600661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:55.532 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.515+0000 7fbd85879700 1 --2- 172.21.15.195:0/757227713 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbd60063d30 0x7fbd600661e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fbd70001900 tx=0x7fbd70009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:55.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.519+0000 7fbd6dffb700 1 -- 172.21.15.195:0/757227713 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fbd78051180 con 0x7fbd80153cb0 2022-06-27T11:47:55.746 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:55.747+0000 7fbd87add700 1 -- 172.21.15.195:0/757227713 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7fbd8006f240 con 0x7fbd80153cb0 2022-06-27T11:47:56.142 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T11:47:56.153 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T11:47:56.373 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.374+0000 7fbd6dffb700 1 -- 172.21.15.195:0/757227713 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v7) v1 ==== 115+0+0 (secure 0 0 0) 0x7fbd7802a020 con 0x7fbd80153cb0 2022-06-27T11:47:56.376 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.376+0000 7fbd87add700 1 -- 172.21.15.195:0/757227713 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbd60063d30 msgr2=0x7fbd600661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:56.376 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.376+0000 7fbd87add700 1 --2- 172.21.15.195:0/757227713 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbd60063d30 0x7fbd600661e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fbd70001900 tx=0x7fbd70009040 comp rx=0 tx=0).stop 2022-06-27T11:47:56.376 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.376+0000 7fbd87add700 1 -- 172.21.15.195:0/757227713 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbd80153cb0 msgr2=0x7fbd800bcbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:56.377 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.376+0000 7fbd87add700 1 --2- 172.21.15.195:0/757227713 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbd80153cb0 0x7fbd800bcbc0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fbd780099f0 tx=0x7fbd78019000 comp rx=0 tx=0).stop 2022-06-27T11:47:56.377 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.376+0000 7fbd87add700 1 -- 172.21.15.195:0/757227713 shutdown_connections 2022-06-27T11:47:56.377 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.376+0000 7fbd87add700 1 --2- 172.21.15.195:0/757227713 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbd60063d30 0x7fbd600661e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:56.378 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.376+0000 7fbd87add700 1 --2- 172.21.15.195:0/757227713 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbd80153cb0 0x7fbd800bcbc0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:56.378 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.376+0000 7fbd87add700 1 --2- 172.21.15.195:0/757227713 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbd8006f640 0x7fbd800bde90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:56.379 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.376+0000 7fbd87add700 1 --2- 172.21.15.195:0/757227713 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbd800c4160 0x7fbd800baeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:56.379 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.376+0000 7fbd87add700 1 -- 172.21.15.195:0/757227713 >> 172.21.15.195:0/757227713 conn(0x7fbd800c6350 msgr2=0x7fbd800c7770 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:56.379 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.376+0000 7fbd87add700 1 -- 172.21.15.195:0/757227713 shutdown_connections 2022-06-27T11:47:56.380 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.376+0000 7fbd87add700 1 -- 172.21.15.195:0/757227713 wait complete. 2022-06-27T11:47:56.389 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2022-06-27T11:47:56.390 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:47:56.512 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.512+0000 7ff31466a700 1 Processor -- start 2022-06-27T11:47:56.512 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.512+0000 7ff31466a700 1 -- start start 2022-06-27T11:47:56.521 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.513+0000 7ff31466a700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7ff30c0c3030 0x7ff30c1570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:56.522 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.513+0000 7ff31466a700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff30c06fba0 con 0x7ff30c06e850 2022-06-27T11:47:56.522 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.513+0000 7ff31466a700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff30c06f5e0 con 0x7ff30c06fef0 2022-06-27T11:47:56.523 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.513+0000 7ff31466a700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff30c1575e0 con 0x7ff30c0c3030 2022-06-27T11:47:56.523 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.513+0000 7ff312c07700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7ff30c0c3030 0x7ff30c1570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:56.523 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.513+0000 7ff312406700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7ff30c06e850 0x7ff30c069de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:47406/0 (socket says 172.21.15.195:47406) 2022-06-27T11:47:56.523 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.513+0000 7ff312406700 1 -- 172.21.15.195:0/3606779194 learned_addr learned my addr 172.21.15.195:0/3606779194 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:47:56.524 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.513+0000 7ff312c07700 1 -- 172.21.15.195:0/3606779194 >> v1:172.21.15.202:6789/0 conn(0x7ff30c06fef0 legacy=0x7ff30c0c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:56.524 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.513+0000 7ff312c07700 1 -- 172.21.15.195:0/3606779194 >> v1:172.21.15.195:6789/0 conn(0x7ff30c06e850 legacy=0x7ff30c069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:56.524 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.513+0000 7ff311404700 1 -- 172.21.15.195:0/3606779194 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3044119846 0 0) 0x7ff30c06fba0 con 0x7ff30c06e850 2022-06-27T11:47:56.525 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.513+0000 7ff312c07700 1 -- 172.21.15.195:0/3606779194 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff30c06f8c0 con 0x7ff30c0c3030 2022-06-27T11:47:56.525 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.513+0000 7ff312c07700 1 --2- 172.21.15.195:0/3606779194 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff30c0c3030 0x7ff30c1570a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff304009e60 tx=0x7ff304009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77f727342682f5fa server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:56.525 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.514+0000 7ff311404700 1 -- 172.21.15.195:0/3606779194 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff30400e070 con 0x7ff30c0c3030 2022-06-27T11:47:56.525 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.514+0000 7ff311404700 1 -- 172.21.15.195:0/3606779194 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff30400bcc0 con 0x7ff30c0c3030 2022-06-27T11:47:56.526 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.514+0000 7ff311404700 1 -- 172.21.15.195:0/3606779194 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff304012680 con 0x7ff30c0c3030 2022-06-27T11:47:56.526 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.514+0000 7ff31466a700 1 -- 172.21.15.195:0/3606779194 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff30c0c3030 msgr2=0x7ff30c1570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:56.526 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.514+0000 7ff31466a700 1 --2- 172.21.15.195:0/3606779194 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff30c0c3030 0x7ff30c1570a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff304009e60 tx=0x7ff304009780 comp rx=0 tx=0).stop 2022-06-27T11:47:56.527 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.514+0000 7ff31466a700 1 -- 172.21.15.195:0/3606779194 shutdown_connections 2022-06-27T11:47:56.527 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.514+0000 7ff31466a700 1 --2- 172.21.15.195:0/3606779194 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff30c0c3030 0x7ff30c1570a0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:56.527 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.514+0000 7ff31466a700 1 -- 172.21.15.195:0/3606779194 >> 172.21.15.195:0/3606779194 conn(0x7ff30c0c6350 msgr2=0x7ff30c0c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:56.527 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.514+0000 7ff31466a700 1 -- 172.21.15.195:0/3606779194 shutdown_connections 2022-06-27T11:47:56.528 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.514+0000 7ff31466a700 1 -- 172.21.15.195:0/3606779194 wait complete. 2022-06-27T11:47:56.528 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.516+0000 7ff31466a700 1 Processor -- start 2022-06-27T11:47:56.528 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.516+0000 7ff31466a700 1 -- start start 2022-06-27T11:47:56.529 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.517+0000 7ff31466a700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff30c06e850 0x7ff30c163bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:56.529 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.517+0000 7ff31466a700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff30c0c3030 0x7ff30c156dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:56.529 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.517+0000 7ff31466a700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff30c152190 0x7ff30c0b6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:56.530 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.517+0000 7ff31466a700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff30c1640f0 con 0x7ff30c06e850 2022-06-27T11:47:56.530 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.517+0000 7ff31466a700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff30c155db0 con 0x7ff30c0c3030 2022-06-27T11:47:56.530 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.517+0000 7ff31466a700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff30c156930 con 0x7ff30c152190 2022-06-27T11:47:56.530 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.517+0000 7ff312406700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff30c06e850 0x7ff30c163bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:56.530 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.517+0000 7ff312406700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff30c06e850 0x7ff30c163bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41224/0 (socket says 172.21.15.195:41224) 2022-06-27T11:47:56.531 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.517+0000 7ff312406700 1 -- 172.21.15.195:0/4180005894 learned_addr learned my addr 172.21.15.195:0/4180005894 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:47:56.531 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.517+0000 7ff311c05700 1 --2- 172.21.15.195:0/4180005894 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff30c0c3030 0x7ff30c156dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:56.531 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.517+0000 7ff312c07700 1 --2- 172.21.15.195:0/4180005894 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff30c152190 0x7ff30c0b6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:56.531 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.517+0000 7ff312406700 1 -- 172.21.15.195:0/4180005894 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff30c152190 msgr2=0x7ff30c0b6ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:56.532 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.517+0000 7ff312406700 1 --2- 172.21.15.195:0/4180005894 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff30c152190 0x7ff30c0b6ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:56.532 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.517+0000 7ff312406700 1 -- 172.21.15.195:0/4180005894 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff30c0c3030 msgr2=0x7ff30c156dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:56.532 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.517+0000 7ff312406700 1 --2- 172.21.15.195:0/4180005894 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff30c0c3030 0x7ff30c156dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:56.532 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.517+0000 7ff312406700 1 -- 172.21.15.195:0/4180005894 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff304009a20 con 0x7ff30c06e850 2022-06-27T11:47:56.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.517+0000 7ff312406700 1 --2- 172.21.15.195:0/4180005894 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff30c06e850 0x7ff30c163bb0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff30800c7b0 tx=0x7ff308007e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:56.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.517+0000 7ff2fa7fc700 1 -- 172.21.15.195:0/4180005894 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff308007450 con 0x7ff30c06e850 2022-06-27T11:47:56.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.517+0000 7ff2fa7fc700 1 -- 172.21.15.195:0/4180005894 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3080046f0 con 0x7ff30c06e850 2022-06-27T11:47:56.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.517+0000 7ff31466a700 1 -- 172.21.15.195:0/4180005894 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff30c0c6a00 con 0x7ff30c06e850 2022-06-27T11:47:56.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.517+0000 7ff2fa7fc700 1 -- 172.21.15.195:0/4180005894 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff308004ab0 con 0x7ff30c06e850 2022-06-27T11:47:56.534 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.517+0000 7ff31466a700 1 -- 172.21.15.195:0/4180005894 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff30c156000 con 0x7ff30c06e850 2022-06-27T11:47:56.534 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.518+0000 7ff31466a700 1 -- 172.21.15.195:0/4180005894 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff30c0722c0 con 0x7ff30c06e850 2022-06-27T11:47:56.534 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.519+0000 7ff2fa7fc700 1 -- 172.21.15.195:0/4180005894 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7ff30800c020 con 0x7ff30c06e850 2022-06-27T11:47:56.534 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.519+0000 7ff2fa7fc700 1 --2- 172.21.15.195:0/4180005894 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff2ec063d00 0x7ff2ec0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:56.535 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.519+0000 7ff2fa7fc700 1 -- 172.21.15.195:0/4180005894 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7ff308082ae0 con 0x7ff30c06e850 2022-06-27T11:47:56.535 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.519+0000 7ff311c05700 1 --2- 172.21.15.195:0/4180005894 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff2ec063d00 0x7ff2ec0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:56.536 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.519+0000 7ff311c05700 1 --2- 172.21.15.195:0/4180005894 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff2ec063d00 0x7ff2ec0661b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff30c0c9360 tx=0x7ff2fc009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:56.537 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.523+0000 7ff2fa7fc700 1 -- 172.21.15.195:0/4180005894 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7ff3080516d0 con 0x7ff30c06e850 2022-06-27T11:47:56.743 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.744+0000 7ff31466a700 1 -- 172.21.15.195:0/4180005894 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff30c06f240 con 0x7ff30c06e850 2022-06-27T11:47:56.745 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.745+0000 7ff2fa7fc700 1 -- 172.21.15.195:0/4180005894 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+6121 (secure 0 0 0) 0x7ff308015030 con 0x7ff30c06e850 2022-06-27T11:47:56.745 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:47:56.746 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4287,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/864126677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":864126677},{"type":"v1","addr":"172.21.15.202:6833","nonce":864126677}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:47:56.366608+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"d","rank":0,"incarnation":4,"state":"up:creating","state_seq":1,"addr":"172.21.15.202:6835/2038169702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2038169702},{"type":"v1","addr":"172.21.15.202:6835","nonce":2038169702}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:47:56.751 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.747+0000 7ff31466a700 1 -- 172.21.15.195:0/4180005894 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff2ec063d00 msgr2=0x7ff2ec0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:56.751 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.747+0000 7ff31466a700 1 --2- 172.21.15.195:0/4180005894 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff2ec063d00 0x7ff2ec0661b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff30c0c9360 tx=0x7ff2fc009040 comp rx=0 tx=0).stop 2022-06-27T11:47:56.752 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.747+0000 7ff31466a700 1 -- 172.21.15.195:0/4180005894 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff30c06e850 msgr2=0x7ff30c163bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:56.752 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.747+0000 7ff31466a700 1 --2- 172.21.15.195:0/4180005894 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff30c06e850 0x7ff30c163bb0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff30800c7b0 tx=0x7ff308007e60 comp rx=0 tx=0).stop 2022-06-27T11:47:56.753 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.747+0000 7ff31466a700 1 -- 172.21.15.195:0/4180005894 shutdown_connections 2022-06-27T11:47:56.753 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.747+0000 7ff31466a700 1 --2- 172.21.15.195:0/4180005894 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff2ec063d00 0x7ff2ec0661b0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:56.753 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.747+0000 7ff31466a700 1 --2- 172.21.15.195:0/4180005894 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff30c06e850 0x7ff30c163bb0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:56.754 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.747+0000 7ff31466a700 1 --2- 172.21.15.195:0/4180005894 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff30c152190 0x7ff30c0b6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:56.754 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.747+0000 7ff31466a700 1 --2- 172.21.15.195:0/4180005894 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff30c0c3030 0x7ff30c156dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:56.755 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.747+0000 7ff31466a700 1 -- 172.21.15.195:0/4180005894 >> 172.21.15.195:0/4180005894 conn(0x7ff30c0c6350 msgr2=0x7ff30c1505e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:56.756 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.747+0000 7ff31466a700 1 -- 172.21.15.195:0/4180005894 shutdown_connections 2022-06-27T11:47:56.756 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.747+0000 7ff31466a700 1 -- 172.21.15.195:0/4180005894 wait complete. 2022-06-27T11:47:56.757 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 7 2022-06-27T11:47:56.759 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-27T11:47:56.889 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.880+0000 7f94030c3700 1 Processor -- start 2022-06-27T11:47:56.889 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.880+0000 7f94030c3700 1 -- start start 2022-06-27T11:47:56.889 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.881+0000 7f94030c3700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f93fc06e850 0x7f93fc1570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:56.890 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.881+0000 7f94030c3700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f93fc0c34a0 0x7f93fc0c2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:56.890 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.881+0000 7f94030c3700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93fc06fba0 con 0x7f93fc06fef0 2022-06-27T11:47:56.890 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.881+0000 7f94030c3700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93fc1575e0 con 0x7f93fc0c34a0 2022-06-27T11:47:56.890 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.881+0000 7f94030c3700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93fc06f5e0 con 0x7f93fc06e850 2022-06-27T11:47:56.891 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.881+0000 7f9401660700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f93fc0c34a0 0x7f93fc0c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:56.891 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.881+0000 7f9400e5f700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f93fc06e850 0x7f93fc1570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:56.891 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.881+0000 7f93fbfff700 1 --1- >> v1:172.21.15.202:6789/0 conn(0x7f93fc06fef0 0x7f93fc069de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.202:6789/0 says I am v1:172.21.15.195:49282/0 (socket says 172.21.15.195:49282) 2022-06-27T11:47:56.891 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.881+0000 7f93fbfff700 1 -- 172.21.15.195:0/2291679478 learned_addr learned my addr 172.21.15.195:0/2291679478 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:47:56.892 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.882+0000 7f9401660700 1 -- 172.21.15.195:0/2291679478 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f93fc06e850 msgr2=0x7f93fc1570a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:47:56.892 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.882+0000 7f9401660700 1 --2- 172.21.15.195:0/2291679478 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f93fc06e850 0x7f93fc1570a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:56.892 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.882+0000 7f9401660700 1 -- 172.21.15.195:0/2291679478 >> v1:172.21.15.202:6789/0 conn(0x7f93fc06fef0 legacy=0x7f93fc069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:56.892 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.882+0000 7f9401660700 1 -- 172.21.15.195:0/2291679478 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93fc06f8c0 con 0x7f93fc0c34a0 2022-06-27T11:47:56.892 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.882+0000 7f9400e5f700 1 --2- 172.21.15.195:0/2291679478 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f93fc06e850 0x7f93fc1570a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:47:56.893 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.882+0000 7f9401660700 1 --2- 172.21.15.195:0/2291679478 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f93fc0c34a0 0x7f93fc0c2920 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f93f4009e60 tx=0x7f93f4009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1b5a6dbb2e4b0869 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:56.893 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.882+0000 7f93fb7fe700 1 -- 172.21.15.195:0/2291679478 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93f400e070 con 0x7f93fc0c34a0 2022-06-27T11:47:56.893 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.882+0000 7f93fb7fe700 1 -- 172.21.15.195:0/2291679478 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93f400bbf0 con 0x7f93fc0c34a0 2022-06-27T11:47:56.893 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.882+0000 7f93fb7fe700 1 -- 172.21.15.195:0/2291679478 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93f4012430 con 0x7f93fc0c34a0 2022-06-27T11:47:56.894 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.883+0000 7f94030c3700 1 -- 172.21.15.195:0/2291679478 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f93fc0c34a0 msgr2=0x7f93fc0c2920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:56.894 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.883+0000 7f94030c3700 1 --2- 172.21.15.195:0/2291679478 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f93fc0c34a0 0x7f93fc0c2920 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f93f4009e60 tx=0x7f93f4009780 comp rx=0 tx=0).stop 2022-06-27T11:47:56.894 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.883+0000 7f94030c3700 1 -- 172.21.15.195:0/2291679478 shutdown_connections 2022-06-27T11:47:56.894 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.883+0000 7f94030c3700 1 --2- 172.21.15.195:0/2291679478 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f93fc0c34a0 0x7f93fc0c2920 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:56.894 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.883+0000 7f94030c3700 1 --2- 172.21.15.195:0/2291679478 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f93fc06e850 0x7f93fc1570a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:56.895 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.883+0000 7f94030c3700 1 -- 172.21.15.195:0/2291679478 >> 172.21.15.195:0/2291679478 conn(0x7f93fc0c6350 msgr2=0x7f93fc0c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:56.895 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.883+0000 7f94030c3700 1 -- 172.21.15.195:0/2291679478 shutdown_connections 2022-06-27T11:47:56.895 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.883+0000 7f94030c3700 1 -- 172.21.15.195:0/2291679478 wait complete. 2022-06-27T11:47:56.895 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.884+0000 7f94030c3700 1 Processor -- start 2022-06-27T11:47:56.896 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.884+0000 7f94030c3700 1 -- start start 2022-06-27T11:47:56.896 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.885+0000 7f94030c3700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f93fc06e850 0x7f93fc14af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:56.896 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.885+0000 7f94030c3700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f93fc0c34a0 0x7f93fc1523a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:56.896 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.885+0000 7f94030c3700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f93fc149bd0 0x7f93fc1528e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:56.896 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.885+0000 7f94030c3700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93fc151f10 con 0x7f93fc06e850 2022-06-27T11:47:56.897 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.885+0000 7f94030c3700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93fc14a150 con 0x7f93fc149bd0 2022-06-27T11:47:56.897 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.885+0000 7f94030c3700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93fc151300 con 0x7f93fc0c34a0 2022-06-27T11:47:56.897 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.885+0000 7f9400e5f700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f93fc06e850 0x7f93fc14af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:56.897 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.885+0000 7f9400e5f700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f93fc06e850 0x7f93fc14af90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41238/0 (socket says 172.21.15.195:41238) 2022-06-27T11:47:56.897 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.885+0000 7f9400e5f700 1 -- 172.21.15.195:0/3787003768 learned_addr learned my addr 172.21.15.195:0/3787003768 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:47:56.898 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.885+0000 7f93fbfff700 1 --2- 172.21.15.195:0/3787003768 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f93fc0c34a0 0x7f93fc1523a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:56.898 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.885+0000 7f9401660700 1 --2- 172.21.15.195:0/3787003768 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f93fc149bd0 0x7f93fc1528e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:56.898 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.885+0000 7f9400e5f700 1 -- 172.21.15.195:0/3787003768 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f93fc0c34a0 msgr2=0x7f93fc1523a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:56.898 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.885+0000 7f9400e5f700 1 --2- 172.21.15.195:0/3787003768 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f93fc0c34a0 0x7f93fc1523a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:56.899 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.885+0000 7f9400e5f700 1 -- 172.21.15.195:0/3787003768 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f93fc149bd0 msgr2=0x7f93fc1528e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:56.899 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.885+0000 7f9400e5f700 1 --2- 172.21.15.195:0/3787003768 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f93fc149bd0 0x7f93fc1528e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:56.899 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.885+0000 7f9400e5f700 1 -- 172.21.15.195:0/3787003768 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93f00092e0 con 0x7f93fc06e850 2022-06-27T11:47:56.899 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.885+0000 7f9400e5f700 1 --2- 172.21.15.195:0/3787003768 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f93fc06e850 0x7f93fc14af90 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f93f000ea50 tx=0x7f93f0009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:56.900 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.885+0000 7f93f8ff9700 1 -- 172.21.15.195:0/3787003768 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93f000c640 con 0x7f93fc06e850 2022-06-27T11:47:56.900 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.885+0000 7f93f8ff9700 1 -- 172.21.15.195:0/3787003768 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93f000c7a0 con 0x7f93fc06e850 2022-06-27T11:47:56.900 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.885+0000 7f94030c3700 1 -- 172.21.15.195:0/3787003768 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93f4009a20 con 0x7f93fc06e850 2022-06-27T11:47:56.900 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.885+0000 7f93f8ff9700 1 -- 172.21.15.195:0/3787003768 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93f000cb20 con 0x7f93fc06e850 2022-06-27T11:47:56.900 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.885+0000 7f94030c3700 1 -- 172.21.15.195:0/3787003768 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93fc1518c0 con 0x7f93fc06e850 2022-06-27T11:47:56.901 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.886+0000 7f94030c3700 1 -- 172.21.15.195:0/3787003768 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93fc0722c0 con 0x7f93fc06e850 2022-06-27T11:47:56.901 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.887+0000 7f93f8ff9700 1 -- 172.21.15.195:0/3787003768 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f93f0019030 con 0x7f93fc06e850 2022-06-27T11:47:56.901 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.887+0000 7f93f8ff9700 1 --2- 172.21.15.195:0/3787003768 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f93e4063d00 0x7f93e40661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:56.901 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.887+0000 7f93fbfff700 1 --2- 172.21.15.195:0/3787003768 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f93e4063d00 0x7f93e40661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:56.901 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.887+0000 7f93f8ff9700 1 -- 172.21.15.195:0/3787003768 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f93f0080850 con 0x7f93fc06e850 2022-06-27T11:47:56.902 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.887+0000 7f93fbfff700 1 --2- 172.21.15.195:0/3787003768 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f93e4063d00 0x7f93e40661b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f93fc0c33b0 tx=0x7f93e8009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:56.902 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:56.891+0000 7f93f8ff9700 1 -- 172.21.15.195:0/3787003768 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f93f004f3e0 con 0x7f93fc06e850 2022-06-27T11:47:57.068 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.067+0000 7f94030c3700 1 -- 172.21.15.195:0/3787003768 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f93fc06b2a0 con 0x7f93fc06e850 2022-06-27T11:47:57.068 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.068+0000 7f93f8ff9700 1 -- 172.21.15.195:0/3787003768 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16914 (secure 0 0 0) 0x7f93f0017090 con 0x7f93fc06e850 2022-06-27T11:47:57.069 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:47:57.069 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":21,"fsid":"81b3a1d3-6a2b-47e8-8d69-7027cfeae7af","created":"2022-06-27T11:47:35.094715+0000","modified":"2022-06-27T11:47:53.359270+0000","last_up_change":"2022-06-27T11:47:44.777459+0000","last_in_change":"2022-06-27T11:47:37.596818+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2022-06-27T11:47:46.368325+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2022-06-27T11:47:46.386132+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-06-27T11:47:50.901270+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-06-27T11:47:51.314719+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c64bc51f-d4f3-4bc2-806b-da0dc78d51b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6813","nonce":94217},{"type":"v1","addr":"172.21.15.195:6815","nonce":94217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6817","nonce":94217},{"type":"v1","addr":"172.21.15.195:6819","nonce":94217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":94217},{"type":"v1","addr":"172.21.15.195:6823","nonce":94217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6820","nonce":94217},{"type":"v1","addr":"172.21.15.195:6821","nonce":94217}]},"public_addr":"172.21.15.195:6815/94217","cluster_addr":"172.21.15.195:6819/94217","heartbeat_back_addr":"172.21.15.195:6823/94217","heartbeat_front_addr":"172.21.15.195:6821/94217","state":["exists","up"]},{"osd":1,"uuid":"bea71cdf-aa98-493f-9207-66d886a3e30e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":94223},{"type":"v1","addr":"172.21.15.195:6809","nonce":94223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":94223},{"type":"v1","addr":"172.21.15.195:6811","nonce":94223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6816","nonce":94223},{"type":"v1","addr":"172.21.15.195:6818","nonce":94223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":94223},{"type":"v1","addr":"172.21.15.195:6814","nonce":94223}]},"public_addr":"172.21.15.195:6809/94223","cluster_addr":"172.21.15.195:6811/94223","heartbeat_back_addr":"172.21.15.195:6818/94223","heartbeat_front_addr":"172.21.15.195:6814/94223","state":["exists","up"]},{"osd":2,"uuid":"bf6c87a0-a8ea-497a-9ce1-721285888065","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":94221},{"type":"v1","addr":"172.21.15.195:6801","nonce":94221}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":94221},{"type":"v1","addr":"172.21.15.195:6803","nonce":94221}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":94221},{"type":"v1","addr":"172.21.15.195:6807","nonce":94221}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":94221},{"type":"v1","addr":"172.21.15.195:6805","nonce":94221}]},"public_addr":"172.21.15.195:6801/94221","cluster_addr":"172.21.15.195:6803/94221","heartbeat_back_addr":"172.21.15.195:6807/94221","heartbeat_front_addr":"172.21.15.195:6805/94221","state":["exists","up"]},{"osd":3,"uuid":"e8aa47ce-af8c-47e7-9654-870cd53229f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":94232},{"type":"v1","addr":"172.21.15.195:6825","nonce":94232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":94232},{"type":"v1","addr":"172.21.15.195:6827","nonce":94232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":94232},{"type":"v1","addr":"172.21.15.195:6831","nonce":94232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":94232},{"type":"v1","addr":"172.21.15.195:6829","nonce":94232}]},"public_addr":"172.21.15.195:6825/94232","cluster_addr":"172.21.15.195:6827/94232","heartbeat_back_addr":"172.21.15.195:6831/94232","heartbeat_front_addr":"172.21.15.195:6829/94232","state":["exists","up"]},{"osd":4,"uuid":"99ffa553-f149-443a-98f5-68b10f77e085","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6800","nonce":94308},{"type":"v1","addr":"172.21.15.202:6801","nonce":94308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6802","nonce":94308},{"type":"v1","addr":"172.21.15.202:6803","nonce":94308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6806","nonce":94308},{"type":"v1","addr":"172.21.15.202:6807","nonce":94308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6804","nonce":94308},{"type":"v1","addr":"172.21.15.202:6805","nonce":94308}]},"public_addr":"172.21.15.202:6801/94308","cluster_addr":"172.21.15.202:6803/94308","heartbeat_back_addr":"172.21.15.202:6807/94308","heartbeat_front_addr":"172.21.15.202:6805/94308","state":["exists","up"]},{"osd":5,"uuid":"35409ef0-891b-4c01-82e4-26b8e64f502d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6824","nonce":94325},{"type":"v1","addr":"172.21.15.202:6825","nonce":94325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6826","nonce":94325},{"type":"v1","addr":"172.21.15.202:6827","nonce":94325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6830","nonce":94325},{"type":"v1","addr":"172.21.15.202:6831","nonce":94325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6828","nonce":94325},{"type":"v1","addr":"172.21.15.202:6829","nonce":94325}]},"public_addr":"172.21.15.202:6825/94325","cluster_addr":"172.21.15.202:6827/94325","heartbeat_back_addr":"172.21.15.202:6831/94325","heartbeat_front_addr":"172.21.15.202:6829/94325","state":["exists","up"]},{"osd":6,"uuid":"009124c4-8b5f-47d4-bd20-2c19271a2adc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6809","nonce":94309},{"type":"v1","addr":"172.21.15.202:6811","nonce":94309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6813","nonce":94309},{"type":"v1","addr":"172.21.15.202:6815","nonce":94309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6821","nonce":94309},{"type":"v1","addr":"172.21.15.202:6823","nonce":94309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6817","nonce":94309},{"type":"v1","addr":"172.21.15.202:6819","nonce":94309}]},"public_addr":"172.21.15.202:6811/94309","cluster_addr":"172.21.15.202:6815/94309","heartbeat_back_addr":"172.21.15.202:6823/94309","heartbeat_front_addr":"172.21.15.202:6819/94309","state":["exists","up"]},{"osd":7,"uuid":"7773115e-d3c3-4a0c-8801-d8ce63e0e5a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6808","nonce":94310},{"type":"v1","addr":"172.21.15.202:6810","nonce":94310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6812","nonce":94310},{"type":"v1","addr":"172.21.15.202:6814","nonce":94310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6820","nonce":94310},{"type":"v1","addr":"172.21.15.202:6822","nonce":94310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6816","nonce":94310},{"type":"v1","addr":"172.21.15.202:6818","nonce":94310}]},"public_addr":"172.21.15.202:6810/94310","cluster_addr":"172.21.15.202:6814/94310","heartbeat_back_addr":"172.21.15.202:6822/94310","heartbeat_front_addr":"172.21.15.202:6818/94310","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.776182+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.806922+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.237997+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.784371+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.723615+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.820332+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:43.841414+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.758071+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-06-27T11:47:57.072 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.070+0000 7f94030c3700 1 -- 172.21.15.195:0/3787003768 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f93e4063d00 msgr2=0x7f93e40661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:57.073 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.070+0000 7f94030c3700 1 --2- 172.21.15.195:0/3787003768 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f93e4063d00 0x7f93e40661b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f93fc0c33b0 tx=0x7f93e8009040 comp rx=0 tx=0).stop 2022-06-27T11:47:57.074 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.070+0000 7f94030c3700 1 -- 172.21.15.195:0/3787003768 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f93fc06e850 msgr2=0x7f93fc14af90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:57.074 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.070+0000 7f94030c3700 1 --2- 172.21.15.195:0/3787003768 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f93fc06e850 0x7f93fc14af90 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f93f000ea50 tx=0x7f93f0009060 comp rx=0 tx=0).stop 2022-06-27T11:47:57.074 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.070+0000 7f94030c3700 1 -- 172.21.15.195:0/3787003768 shutdown_connections 2022-06-27T11:47:57.075 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.070+0000 7f94030c3700 1 --2- 172.21.15.195:0/3787003768 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f93e4063d00 0x7f93e40661b0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:57.075 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.070+0000 7f94030c3700 1 --2- 172.21.15.195:0/3787003768 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f93fc06e850 0x7f93fc14af90 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:57.076 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.070+0000 7f94030c3700 1 --2- 172.21.15.195:0/3787003768 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f93fc149bd0 0x7f93fc1528e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:57.076 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.070+0000 7f94030c3700 1 --2- 172.21.15.195:0/3787003768 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f93fc0c34a0 0x7f93fc1523a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:57.076 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.070+0000 7f94030c3700 1 -- 172.21.15.195:0/3787003768 >> 172.21.15.195:0/3787003768 conn(0x7f93fc0c6350 msgr2=0x7f93fc068a10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:57.077 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.070+0000 7f94030c3700 1 -- 172.21.15.195:0/3787003768 shutdown_connections 2022-06-27T11:47:57.077 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.070+0000 7f94030c3700 1 -- 172.21.15.195:0/3787003768 wait complete. 2022-06-27T11:47:57.082 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2022-06-27T11:47:57.082 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2022-06-27T11:47:57.203 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.203+0000 7f3a8b2e9700 1 Processor -- start 2022-06-27T11:47:57.203 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.203+0000 7f3a8b2e9700 1 -- start start 2022-06-27T11:47:57.208 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.204+0000 7f3a8b2e9700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3a8406f640 0x7f3a84156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:57.208 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.204+0000 7f3a8b2e9700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f3a840c4160 0x7f3a840c1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:57.209 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.204+0000 7f3a8b2e9700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f3a840c39e0 0x7f3a841535a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:57.209 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.204+0000 7f3a8b2e9700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a840c1a20 con 0x7f3a840c4160 2022-06-27T11:47:57.209 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.204+0000 7f3a8b2e9700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a840c1130 con 0x7f3a840c39e0 2022-06-27T11:47:57.209 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.204+0000 7f3a8b2e9700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a8406ea90 con 0x7f3a8406f640 2022-06-27T11:47:57.209 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.204+0000 7f3a88884700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f3a840c4160 0x7f3a840c1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:57.210 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.204+0000 7f3a88884700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f3a840c4160 0x7f3a840c1eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41246/0 (socket says 172.21.15.195:41246) 2022-06-27T11:47:57.210 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.204+0000 7f3a88884700 1 -- 172.21.15.195:0/3671354760 learned_addr learned my addr 172.21.15.195:0/3671354760 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:47:57.210 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.204+0000 7f3a89886700 1 --2- 172.21.15.195:0/3671354760 >> v2:172.21.15.202:3300/0 conn(0x7f3a840c39e0 0x7f3a841535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:57.211 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.204+0000 7f3a89085700 1 --2- 172.21.15.195:0/3671354760 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3a8406f640 0x7f3a84156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:57.211 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.205+0000 7f3a88884700 1 -- 172.21.15.195:0/3671354760 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3a8406f640 msgr2=0x7f3a84156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:47:57.211 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.205+0000 7f3a88884700 1 --2- 172.21.15.195:0/3671354760 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3a8406f640 0x7f3a84156ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:57.211 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.205+0000 7f3a88884700 1 -- 172.21.15.195:0/3671354760 >> v2:172.21.15.202:3300/0 conn(0x7f3a840c39e0 msgr2=0x7f3a841535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:47:57.211 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.205+0000 7f3a88884700 1 --2- 172.21.15.195:0/3671354760 >> v2:172.21.15.202:3300/0 conn(0x7f3a840c39e0 0x7f3a841535a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:57.212 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.205+0000 7f3a88884700 1 -- 172.21.15.195:0/3671354760 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a8406ed70 con 0x7f3a840c4160 2022-06-27T11:47:57.212 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.205+0000 7f3a88884700 1 --2- 172.21.15.195:0/3671354760 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3a840c4160 0x7f3a840c1eb0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f3a84065ec0 tx=0x7f3a80009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3955608dc3175128 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:57.212 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.205+0000 7f3a73fff700 1 -- 172.21.15.195:0/3671354760 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a80011630 con 0x7f3a840c4160 2022-06-27T11:47:57.212 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.205+0000 7f3a73fff700 1 -- 172.21.15.195:0/3671354760 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a80011790 con 0x7f3a840c4160 2022-06-27T11:47:57.213 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.205+0000 7f3a73fff700 1 -- 172.21.15.195:0/3671354760 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a80011b30 con 0x7f3a840c4160 2022-06-27T11:47:57.213 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.205+0000 7f3a8b2e9700 1 -- 172.21.15.195:0/3671354760 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3a840c4160 msgr2=0x7f3a840c1eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:57.213 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.205+0000 7f3a8b2e9700 1 --2- 172.21.15.195:0/3671354760 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3a840c4160 0x7f3a840c1eb0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f3a84065ec0 tx=0x7f3a80009120 comp rx=0 tx=0).stop 2022-06-27T11:47:57.213 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.205+0000 7f3a8b2e9700 1 -- 172.21.15.195:0/3671354760 shutdown_connections 2022-06-27T11:47:57.213 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.205+0000 7f3a8b2e9700 1 --2- 172.21.15.195:0/3671354760 >> v2:172.21.15.202:3300/0 conn(0x7f3a840c39e0 0x7f3a841535a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:57.214 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.205+0000 7f3a8b2e9700 1 --2- 172.21.15.195:0/3671354760 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3a8406f640 0x7f3a84156ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:57.214 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.205+0000 7f3a8b2e9700 1 --2- 172.21.15.195:0/3671354760 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3a840c4160 0x7f3a840c1eb0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:57.214 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.205+0000 7f3a8b2e9700 1 -- 172.21.15.195:0/3671354760 >> 172.21.15.195:0/3671354760 conn(0x7f3a840c6350 msgr2=0x7f3a840c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:57.214 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.205+0000 7f3a8b2e9700 1 -- 172.21.15.195:0/3671354760 shutdown_connections 2022-06-27T11:47:57.215 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.205+0000 7f3a8b2e9700 1 -- 172.21.15.195:0/3671354760 wait complete. 2022-06-27T11:47:57.215 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.206+0000 7f3a8b2e9700 1 Processor -- start 2022-06-27T11:47:57.215 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.206+0000 7f3a8b2e9700 1 -- start start 2022-06-27T11:47:57.215 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.207+0000 7f3a8b2e9700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3a8406f640 0x7f3a8416a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:57.216 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.207+0000 7f3a8b2e9700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3a840c39e0 0x7f3a84167ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:57.216 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.207+0000 7f3a8b2e9700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3a840c4160 0x7f3a84168530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:57.216 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.207+0000 7f3a8b2e9700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a840bedb0 con 0x7f3a840c39e0 2022-06-27T11:47:57.216 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.207+0000 7f3a8b2e9700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a8416ac00 con 0x7f3a840c4160 2022-06-27T11:47:57.216 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.207+0000 7f3a8b2e9700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a840be760 con 0x7f3a8406f640 2022-06-27T11:47:57.217 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.207+0000 7f3a88884700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3a840c39e0 0x7f3a84167ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:57.217 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.207+0000 7f3a88884700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3a840c39e0 0x7f3a84167ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41254/0 (socket says 172.21.15.195:41254) 2022-06-27T11:47:57.217 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.207+0000 7f3a88884700 1 -- 172.21.15.195:0/3888788467 learned_addr learned my addr 172.21.15.195:0/3888788467 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:47:57.217 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.207+0000 7f3a89085700 1 --2- 172.21.15.195:0/3888788467 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3a8406f640 0x7f3a8416a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:57.218 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.207+0000 7f3a89886700 1 --2- 172.21.15.195:0/3888788467 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3a840c4160 0x7f3a84168530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:57.218 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.208+0000 7f3a88884700 1 -- 172.21.15.195:0/3888788467 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3a8406f640 msgr2=0x7f3a8416a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:57.218 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.208+0000 7f3a88884700 1 --2- 172.21.15.195:0/3888788467 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3a8406f640 0x7f3a8416a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:57.218 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.208+0000 7f3a88884700 1 -- 172.21.15.195:0/3888788467 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3a840c4160 msgr2=0x7f3a84168530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:57.218 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.208+0000 7f3a88884700 1 --2- 172.21.15.195:0/3888788467 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3a840c4160 0x7f3a84168530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:57.219 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.208+0000 7f3a88884700 1 -- 172.21.15.195:0/3888788467 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a800093a0 con 0x7f3a840c39e0 2022-06-27T11:47:57.219 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.208+0000 7f3a89085700 1 --2- 172.21.15.195:0/3888788467 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3a8406f640 0x7f3a8416a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:47:57.219 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.208+0000 7f3a88884700 1 --2- 172.21.15.195:0/3888788467 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3a840c39e0 0x7f3a84167ff0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3a80009160 tx=0x7f3a80017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:57.219 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.208+0000 7f3a717fa700 1 -- 172.21.15.195:0/3888788467 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a80011900 con 0x7f3a840c39e0 2022-06-27T11:47:57.220 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.208+0000 7f3a717fa700 1 -- 172.21.15.195:0/3888788467 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a80011a80 con 0x7f3a840c39e0 2022-06-27T11:47:57.220 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.208+0000 7f3a8b2e9700 1 -- 172.21.15.195:0/3888788467 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a840bea40 con 0x7f3a840c39e0 2022-06-27T11:47:57.221 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.208+0000 7f3a717fa700 1 -- 172.21.15.195:0/3888788467 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a800042a0 con 0x7f3a840c39e0 2022-06-27T11:47:57.221 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.208+0000 7f3a8b2e9700 1 -- 172.21.15.195:0/3888788467 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a840c05b0 con 0x7f3a840c39e0 2022-06-27T11:47:57.221 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.209+0000 7f3a8b2e9700 1 -- 172.21.15.195:0/3888788467 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a840c07a0 con 0x7f3a840c39e0 2022-06-27T11:47:57.222 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.209+0000 7f3a717fa700 1 -- 172.21.15.195:0/3888788467 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f3a8002b030 con 0x7f3a840c39e0 2022-06-27T11:47:57.223 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.210+0000 7f3a717fa700 1 --2- 172.21.15.195:0/3888788467 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3a64063d30 0x7f3a640661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:57.223 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.210+0000 7f3a89085700 1 --2- 172.21.15.195:0/3888788467 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3a64063d30 0x7f3a640661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:57.223 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.210+0000 7f3a717fa700 1 -- 172.21.15.195:0/3888788467 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f3a80081c80 con 0x7f3a840c39e0 2022-06-27T11:47:57.223 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.210+0000 7f3a89085700 1 --2- 172.21.15.195:0/3888788467 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3a64063d30 0x7f3a640661e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3a740113e0 tx=0x7f3a74009060 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:57.223 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.214+0000 7f3a717fa700 1 -- 172.21.15.195:0/3888788467 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f3a80050870 con 0x7f3a840c39e0 2022-06-27T11:47:57.395 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.395+0000 7f3a8b2e9700 1 -- 172.21.15.195:0/3888788467 --> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3a84167e80 con 0x7f3a64063d30 2022-06-27T11:47:57.403 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.403+0000 7f3a717fa700 1 -- 172.21.15.195:0/3888788467 <== mgr.4098 v2:172.21.15.195:6832/93955 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+284966 (secure 0 0 0) 0x7f3a84167e80 con 0x7f3a64063d30 2022-06-27T11:47:57.407 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:47:57.410 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.407+0000 7f3a8b2e9700 1 -- 172.21.15.195:0/3888788467 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3a64063d30 msgr2=0x7f3a640661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:57.410 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.407+0000 7f3a8b2e9700 1 --2- 172.21.15.195:0/3888788467 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3a64063d30 0x7f3a640661e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3a740113e0 tx=0x7f3a74009060 comp rx=0 tx=0).stop 2022-06-27T11:47:57.411 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.407+0000 7f3a8b2e9700 1 -- 172.21.15.195:0/3888788467 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3a840c39e0 msgr2=0x7f3a84167ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:57.411 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.407+0000 7f3a8b2e9700 1 --2- 172.21.15.195:0/3888788467 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3a840c39e0 0x7f3a84167ff0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3a80009160 tx=0x7f3a80017000 comp rx=0 tx=0).stop 2022-06-27T11:47:57.411 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.407+0000 7f3a8b2e9700 1 -- 172.21.15.195:0/3888788467 shutdown_connections 2022-06-27T11:47:57.411 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.407+0000 7f3a8b2e9700 1 --2- 172.21.15.195:0/3888788467 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3a64063d30 0x7f3a640661e0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:57.412 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.407+0000 7f3a8b2e9700 1 --2- 172.21.15.195:0/3888788467 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3a840c4160 0x7f3a84168530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:57.412 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.407+0000 7f3a8b2e9700 1 --2- 172.21.15.195:0/3888788467 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3a8406f640 0x7f3a8416a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:57.412 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.407+0000 7f3a8b2e9700 1 --2- 172.21.15.195:0/3888788467 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3a840c39e0 0x7f3a84167ff0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:57.412 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.407+0000 7f3a8b2e9700 1 -- 172.21.15.195:0/3888788467 >> 172.21.15.195:0/3888788467 conn(0x7f3a840c6350 msgr2=0x7f3a840c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:57.412 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.407+0000 7f3a8b2e9700 1 -- 172.21.15.195:0/3888788467 shutdown_connections 2022-06-27T11:47:57.413 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.407+0000 7f3a8b2e9700 1 -- 172.21.15.195:0/3888788467 wait complete. 2022-06-27T11:47:57.413 INFO:teuthology.orchestra.run.smithi195.stderr:dumped all 2022-06-27T11:47:57.419 INFO:teuthology.orchestra.run.smithi195.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2022-06-27T11:47:56.340350+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":144,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":68628,"kb_used_data":1804,"kb_used_omap":0,"kb_used_meta":66752,"kb_avail":754906092,"statfs":{"total":773094113280,"available":773023838208,"internally_reserved":0,"allocated":1847296,"data_stored":1802801,"data_compressed":64756,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":68354048},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":14,"num_write_kb":262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.385895"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384962+0000","last_change":"2022-06-27T11:47:52.381823+0000","last_active":"2022-06-27T11:47:52.384962+0000","last_peered":"2022-06-27T11:47:52.384962+0000","last_clean":"2022-06-27T11:47:52.384962+0000","last_became_active":"2022-06-27T11:47:52.381551+0000","last_became_peered":"2022-06-27T11:47:52.381551+0000","last_unstale":"2022-06-27T11:47:52.384962+0000","last_undegraded":"2022-06-27T11:47:52.384962+0000","last_fullsized":"2022-06-27T11:47:52.384962+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:17:34.164061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.378135+0000","last_change":"2022-06-27T11:47:52.374939+0000","last_active":"2022-06-27T11:47:52.378135+0000","last_peered":"2022-06-27T11:47:52.378135+0000","last_clean":"2022-06-27T11:47:52.378135+0000","last_became_active":"2022-06-27T11:47:52.374620+0000","last_became_peered":"2022-06-27T11:47:52.374620+0000","last_unstale":"2022-06-27T11:47:52.378135+0000","last_undegraded":"2022-06-27T11:47:52.378135+0000","last_fullsized":"2022-06-27T11:47:52.378135+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:56:27.254098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384127+0000","last_change":"2022-06-27T11:47:52.380802+0000","last_active":"2022-06-27T11:47:52.384127+0000","last_peered":"2022-06-27T11:47:52.384127+0000","last_clean":"2022-06-27T11:47:52.384127+0000","last_became_active":"2022-06-27T11:47:52.380500+0000","last_became_peered":"2022-06-27T11:47:52.380500+0000","last_unstale":"2022-06-27T11:47:52.384127+0000","last_undegraded":"2022-06-27T11:47:52.384127+0000","last_fullsized":"2022-06-27T11:47:52.384127+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:06:42.889598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384114+0000","last_change":"2022-06-27T11:47:52.380943+0000","last_active":"2022-06-27T11:47:52.384114+0000","last_peered":"2022-06-27T11:47:52.384114+0000","last_clean":"2022-06-27T11:47:52.384114+0000","last_became_active":"2022-06-27T11:47:52.380648+0000","last_became_peered":"2022-06-27T11:47:52.380648+0000","last_unstale":"2022-06-27T11:47:52.384114+0000","last_undegraded":"2022-06-27T11:47:52.384114+0000","last_fullsized":"2022-06-27T11:47:52.384114+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:14:19.729605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.386875+0000","last_change":"2022-06-27T11:47:52.386144+0000","last_active":"2022-06-27T11:47:52.386875+0000","last_peered":"2022-06-27T11:47:52.386875+0000","last_clean":"2022-06-27T11:47:52.386875+0000","last_became_active":"2022-06-27T11:47:52.385586+0000","last_became_peered":"2022-06-27T11:47:52.385586+0000","last_unstale":"2022-06-27T11:47:52.386875+0000","last_undegraded":"2022-06-27T11:47:52.386875+0000","last_fullsized":"2022-06-27T11:47:52.386875+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T14:30:03.660749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.387182+0000","last_change":"2022-06-27T11:47:52.381906+0000","last_active":"2022-06-27T11:47:52.387182+0000","last_peered":"2022-06-27T11:47:52.387182+0000","last_clean":"2022-06-27T11:47:52.387182+0000","last_became_active":"2022-06-27T11:47:52.381279+0000","last_became_peered":"2022-06-27T11:47:52.381279+0000","last_unstale":"2022-06-27T11:47:52.387182+0000","last_undegraded":"2022-06-27T11:47:52.387182+0000","last_fullsized":"2022-06-27T11:47:52.387182+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T18:41:03.991762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.383137+0000","last_change":"2022-06-27T11:47:52.377128+0000","last_active":"2022-06-27T11:47:52.383137+0000","last_peered":"2022-06-27T11:47:52.383137+0000","last_clean":"2022-06-27T11:47:52.383137+0000","last_became_active":"2022-06-27T11:47:52.376499+0000","last_became_peered":"2022-06-27T11:47:52.376499+0000","last_unstale":"2022-06-27T11:47:52.383137+0000","last_undegraded":"2022-06-27T11:47:52.383137+0000","last_fullsized":"2022-06-27T11:47:52.383137+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:00:13.208145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.378171+0000","last_change":"2022-06-27T11:47:52.374169+0000","last_active":"2022-06-27T11:47:52.378171+0000","last_peered":"2022-06-27T11:47:52.378171+0000","last_clean":"2022-06-27T11:47:52.378171+0000","last_became_active":"2022-06-27T11:47:52.373917+0000","last_became_peered":"2022-06-27T11:47:52.373917+0000","last_unstale":"2022-06-27T11:47:52.378171+0000","last_undegraded":"2022-06-27T11:47:52.378171+0000","last_fullsized":"2022-06-27T11:47:52.378171+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:43:52.741254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384769+0000","last_change":"2022-06-27T11:47:52.382039+0000","last_active":"2022-06-27T11:47:52.384769+0000","last_peered":"2022-06-27T11:47:52.384769+0000","last_clean":"2022-06-27T11:47:52.384769+0000","last_became_active":"2022-06-27T11:47:52.381202+0000","last_became_peered":"2022-06-27T11:47:52.381202+0000","last_unstale":"2022-06-27T11:47:52.384769+0000","last_undegraded":"2022-06-27T11:47:52.384769+0000","last_fullsized":"2022-06-27T11:47:52.384769+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:13:02.553823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.383894+0000","last_change":"2022-06-27T11:47:52.380839+0000","last_active":"2022-06-27T11:47:52.383894+0000","last_peered":"2022-06-27T11:47:52.383894+0000","last_clean":"2022-06-27T11:47:52.383894+0000","last_became_active":"2022-06-27T11:47:52.380527+0000","last_became_peered":"2022-06-27T11:47:52.380527+0000","last_unstale":"2022-06-27T11:47:52.383894+0000","last_undegraded":"2022-06-27T11:47:52.383894+0000","last_fullsized":"2022-06-27T11:47:52.383894+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T23:18:07.086598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.374783+0000","last_change":"2022-06-27T11:47:52.371267+0000","last_active":"2022-06-27T11:47:52.374783+0000","last_peered":"2022-06-27T11:47:52.374783+0000","last_clean":"2022-06-27T11:47:52.374783+0000","last_became_active":"2022-06-27T11:47:52.370921+0000","last_became_peered":"2022-06-27T11:47:52.370921+0000","last_unstale":"2022-06-27T11:47:52.374783+0000","last_undegraded":"2022-06-27T11:47:52.374783+0000","last_fullsized":"2022-06-27T11:47:52.374783+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T19:22:39.648715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.378994+0000","last_change":"2022-06-27T11:47:52.375381+0000","last_active":"2022-06-27T11:47:52.378994+0000","last_peered":"2022-06-27T11:47:52.378994+0000","last_clean":"2022-06-27T11:47:52.378994+0000","last_became_active":"2022-06-27T11:47:52.375089+0000","last_became_peered":"2022-06-27T11:47:52.375089+0000","last_unstale":"2022-06-27T11:47:52.378994+0000","last_undegraded":"2022-06-27T11:47:52.378994+0000","last_fullsized":"2022-06-27T11:47:52.378994+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:43:14.830424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384993+0000","last_change":"2022-06-27T11:47:52.381690+0000","last_active":"2022-06-27T11:47:52.384993+0000","last_peered":"2022-06-27T11:47:52.384993+0000","last_clean":"2022-06-27T11:47:52.384993+0000","last_became_active":"2022-06-27T11:47:52.381472+0000","last_became_peered":"2022-06-27T11:47:52.381472+0000","last_unstale":"2022-06-27T11:47:52.384993+0000","last_undegraded":"2022-06-27T11:47:52.384993+0000","last_fullsized":"2022-06-27T11:47:52.384993+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:14:56.919536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.378838+0000","last_change":"2022-06-27T11:47:52.376063+0000","last_active":"2022-06-27T11:47:52.378838+0000","last_peered":"2022-06-27T11:47:52.378838+0000","last_clean":"2022-06-27T11:47:52.378838+0000","last_became_active":"2022-06-27T11:47:52.375778+0000","last_became_peered":"2022-06-27T11:47:52.375778+0000","last_unstale":"2022-06-27T11:47:52.378838+0000","last_undegraded":"2022-06-27T11:47:52.378838+0000","last_fullsized":"2022-06-27T11:47:52.378838+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T23:16:52.221967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.383160+0000","last_change":"2022-06-27T11:47:52.377297+0000","last_active":"2022-06-27T11:47:52.383160+0000","last_peered":"2022-06-27T11:47:52.383160+0000","last_clean":"2022-06-27T11:47:52.383160+0000","last_became_active":"2022-06-27T11:47:52.376855+0000","last_became_peered":"2022-06-27T11:47:52.376855+0000","last_unstale":"2022-06-27T11:47:52.383160+0000","last_undegraded":"2022-06-27T11:47:52.383160+0000","last_fullsized":"2022-06-27T11:47:52.383160+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:33:39.990522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.387168+0000","last_change":"2022-06-27T11:47:52.381821+0000","last_active":"2022-06-27T11:47:52.387168+0000","last_peered":"2022-06-27T11:47:52.387168+0000","last_clean":"2022-06-27T11:47:52.387168+0000","last_became_active":"2022-06-27T11:47:52.381548+0000","last_became_peered":"2022-06-27T11:47:52.381548+0000","last_unstale":"2022-06-27T11:47:52.387168+0000","last_undegraded":"2022-06-27T11:47:52.387168+0000","last_fullsized":"2022-06-27T11:47:52.387168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T23:14:53.320016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384753+0000","last_change":"2022-06-27T11:47:52.381811+0000","last_active":"2022-06-27T11:47:52.384753+0000","last_peered":"2022-06-27T11:47:52.384753+0000","last_clean":"2022-06-27T11:47:52.384753+0000","last_became_active":"2022-06-27T11:47:52.381554+0000","last_became_peered":"2022-06-27T11:47:52.381554+0000","last_unstale":"2022-06-27T11:47:52.384753+0000","last_undegraded":"2022-06-27T11:47:52.384753+0000","last_fullsized":"2022-06-27T11:47:52.384753+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T14:19:38.519380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.387218+0000","last_change":"2022-06-27T11:47:52.382952+0000","last_active":"2022-06-27T11:47:52.387218+0000","last_peered":"2022-06-27T11:47:52.387218+0000","last_clean":"2022-06-27T11:47:52.387218+0000","last_became_active":"2022-06-27T11:47:52.382691+0000","last_became_peered":"2022-06-27T11:47:52.382691+0000","last_unstale":"2022-06-27T11:47:52.387218+0000","last_undegraded":"2022-06-27T11:47:52.387218+0000","last_fullsized":"2022-06-27T11:47:52.387218+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:28:54.743970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.383682+0000","last_change":"2022-06-27T11:47:52.379083+0000","last_active":"2022-06-27T11:47:52.383682+0000","last_peered":"2022-06-27T11:47:52.383682+0000","last_clean":"2022-06-27T11:47:52.383682+0000","last_became_active":"2022-06-27T11:47:52.378724+0000","last_became_peered":"2022-06-27T11:47:52.378724+0000","last_unstale":"2022-06-27T11:47:52.383682+0000","last_undegraded":"2022-06-27T11:47:52.383682+0000","last_fullsized":"2022-06-27T11:47:52.383682+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T21:25:07.786935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.383275+0000","last_change":"2022-06-27T11:47:52.377947+0000","last_active":"2022-06-27T11:47:52.383275+0000","last_peered":"2022-06-27T11:47:52.383275+0000","last_clean":"2022-06-27T11:47:52.383275+0000","last_became_active":"2022-06-27T11:47:52.377652+0000","last_became_peered":"2022-06-27T11:47:52.377652+0000","last_unstale":"2022-06-27T11:47:52.383275+0000","last_undegraded":"2022-06-27T11:47:52.383275+0000","last_fullsized":"2022-06-27T11:47:52.383275+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T20:48:28.642052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384444+0000","last_change":"2022-06-27T11:47:52.381920+0000","last_active":"2022-06-27T11:47:52.384444+0000","last_peered":"2022-06-27T11:47:52.384444+0000","last_clean":"2022-06-27T11:47:52.384444+0000","last_became_active":"2022-06-27T11:47:52.381477+0000","last_became_peered":"2022-06-27T11:47:52.381477+0000","last_unstale":"2022-06-27T11:47:52.384444+0000","last_undegraded":"2022-06-27T11:47:52.384444+0000","last_fullsized":"2022-06-27T11:47:52.384444+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T20:42:25.529336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.386345+0000","last_change":"2022-06-27T11:47:52.385231+0000","last_active":"2022-06-27T11:47:52.386345+0000","last_peered":"2022-06-27T11:47:52.386345+0000","last_clean":"2022-06-27T11:47:52.386345+0000","last_became_active":"2022-06-27T11:47:52.384972+0000","last_became_peered":"2022-06-27T11:47:52.384972+0000","last_unstale":"2022-06-27T11:47:52.386345+0000","last_undegraded":"2022-06-27T11:47:52.386345+0000","last_fullsized":"2022-06-27T11:47:52.386345+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T17:37:26.084788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.385330+0000","last_change":"2022-06-27T11:47:52.381358+0000","last_active":"2022-06-27T11:47:52.385330+0000","last_peered":"2022-06-27T11:47:52.385330+0000","last_clean":"2022-06-27T11:47:52.385330+0000","last_became_active":"2022-06-27T11:47:52.380997+0000","last_became_peered":"2022-06-27T11:47:52.380997+0000","last_unstale":"2022-06-27T11:47:52.385330+0000","last_undegraded":"2022-06-27T11:47:52.385330+0000","last_fullsized":"2022-06-27T11:47:52.385330+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T21:44:16.061234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384432+0000","last_change":"2022-06-27T11:47:52.381899+0000","last_active":"2022-06-27T11:47:52.384432+0000","last_peered":"2022-06-27T11:47:52.384432+0000","last_clean":"2022-06-27T11:47:52.384432+0000","last_became_active":"2022-06-27T11:47:52.381645+0000","last_became_peered":"2022-06-27T11:47:52.381645+0000","last_unstale":"2022-06-27T11:47:52.384432+0000","last_undegraded":"2022-06-27T11:47:52.384432+0000","last_fullsized":"2022-06-27T11:47:52.384432+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:05:59.742023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.385053+0000","last_change":"2022-06-27T11:47:52.378736+0000","last_active":"2022-06-27T11:47:52.385053+0000","last_peered":"2022-06-27T11:47:52.385053+0000","last_clean":"2022-06-27T11:47:52.385053+0000","last_became_active":"2022-06-27T11:47:52.378475+0000","last_became_peered":"2022-06-27T11:47:52.378475+0000","last_unstale":"2022-06-27T11:47:52.385053+0000","last_undegraded":"2022-06-27T11:47:52.385053+0000","last_fullsized":"2022-06-27T11:47:52.385053+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T12:42:12.528716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384922+0000","last_change":"2022-06-27T11:47:52.378044+0000","last_active":"2022-06-27T11:47:52.384922+0000","last_peered":"2022-06-27T11:47:52.384922+0000","last_clean":"2022-06-27T11:47:52.384922+0000","last_became_active":"2022-06-27T11:47:52.377791+0000","last_became_peered":"2022-06-27T11:47:52.377791+0000","last_unstale":"2022-06-27T11:47:52.384922+0000","last_undegraded":"2022-06-27T11:47:52.384922+0000","last_fullsized":"2022-06-27T11:47:52.384922+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:21:45.615162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384115+0000","last_change":"2022-06-27T11:47:52.382267+0000","last_active":"2022-06-27T11:47:52.384115+0000","last_peered":"2022-06-27T11:47:52.384115+0000","last_clean":"2022-06-27T11:47:52.384115+0000","last_became_active":"2022-06-27T11:47:52.381985+0000","last_became_peered":"2022-06-27T11:47:52.381985+0000","last_unstale":"2022-06-27T11:47:52.384115+0000","last_undegraded":"2022-06-27T11:47:52.384115+0000","last_fullsized":"2022-06-27T11:47:52.384115+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:34:09.509759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.365170+0000","last_change":"2022-06-27T11:47:51.982183+0000","last_active":"2022-06-27T11:47:52.365170+0000","last_peered":"2022-06-27T11:47:52.365170+0000","last_clean":"2022-06-27T11:47:52.365170+0000","last_became_active":"2022-06-27T11:47:51.981339+0000","last_became_peered":"2022-06-27T11:47:51.981339+0000","last_unstale":"2022-06-27T11:47:52.365170+0000","last_undegraded":"2022-06-27T11:47:52.365170+0000","last_fullsized":"2022-06-27T11:47:52.365170+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:08:09.223947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384452+0000","last_change":"2022-06-27T11:47:52.377197+0000","last_active":"2022-06-27T11:47:52.384452+0000","last_peered":"2022-06-27T11:47:52.384452+0000","last_clean":"2022-06-27T11:47:52.384452+0000","last_became_active":"2022-06-27T11:47:52.376925+0000","last_became_peered":"2022-06-27T11:47:52.376925+0000","last_unstale":"2022-06-27T11:47:52.384452+0000","last_undegraded":"2022-06-27T11:47:52.384452+0000","last_fullsized":"2022-06-27T11:47:52.384452+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:39:27.103978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.364221+0000","last_change":"2022-06-27T11:47:51.981074+0000","last_active":"2022-06-27T11:47:52.364221+0000","last_peered":"2022-06-27T11:47:52.364221+0000","last_clean":"2022-06-27T11:47:52.364221+0000","last_became_active":"2022-06-27T11:47:51.980636+0000","last_became_peered":"2022-06-27T11:47:51.980636+0000","last_unstale":"2022-06-27T11:47:52.364221+0000","last_undegraded":"2022-06-27T11:47:52.364221+0000","last_fullsized":"2022-06-27T11:47:52.364221+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T23:32:52.850801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384380+0000","last_change":"2022-06-27T11:47:52.378415+0000","last_active":"2022-06-27T11:47:52.384380+0000","last_peered":"2022-06-27T11:47:52.384380+0000","last_clean":"2022-06-27T11:47:52.384380+0000","last_became_active":"2022-06-27T11:47:52.378123+0000","last_became_peered":"2022-06-27T11:47:52.378123+0000","last_unstale":"2022-06-27T11:47:52.384380+0000","last_undegraded":"2022-06-27T11:47:52.384380+0000","last_fullsized":"2022-06-27T11:47:52.384380+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T11:54:37.127828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.369137+0000","last_change":"2022-06-27T11:47:51.975684+0000","last_active":"2022-06-27T11:47:52.369137+0000","last_peered":"2022-06-27T11:47:52.369137+0000","last_clean":"2022-06-27T11:47:52.369137+0000","last_became_active":"2022-06-27T11:47:51.974902+0000","last_became_peered":"2022-06-27T11:47:51.974902+0000","last_unstale":"2022-06-27T11:47:52.369137+0000","last_undegraded":"2022-06-27T11:47:52.369137+0000","last_fullsized":"2022-06-27T11:47:52.369137+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:51:45.780798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.378008+0000","last_change":"2022-06-27T11:47:52.376089+0000","last_active":"2022-06-27T11:47:52.378008+0000","last_peered":"2022-06-27T11:47:52.378008+0000","last_clean":"2022-06-27T11:47:52.378008+0000","last_became_active":"2022-06-27T11:47:52.375780+0000","last_became_peered":"2022-06-27T11:47:52.375780+0000","last_unstale":"2022-06-27T11:47:52.378008+0000","last_undegraded":"2022-06-27T11:47:52.378008+0000","last_fullsized":"2022-06-27T11:47:52.378008+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T14:17:53.942880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.371313+0000","last_change":"2022-06-27T11:47:51.974071+0000","last_active":"2022-06-27T11:47:52.371313+0000","last_peered":"2022-06-27T11:47:52.371313+0000","last_clean":"2022-06-27T11:47:52.371313+0000","last_became_active":"2022-06-27T11:47:51.973682+0000","last_became_peered":"2022-06-27T11:47:51.973682+0000","last_unstale":"2022-06-27T11:47:52.371313+0000","last_undegraded":"2022-06-27T11:47:52.371313+0000","last_fullsized":"2022-06-27T11:47:52.371313+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T17:40:03.705083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384841+0000","last_change":"2022-06-27T11:47:52.377716+0000","last_active":"2022-06-27T11:47:52.384841+0000","last_peered":"2022-06-27T11:47:52.384841+0000","last_clean":"2022-06-27T11:47:52.384841+0000","last_became_active":"2022-06-27T11:47:52.377464+0000","last_became_peered":"2022-06-27T11:47:52.377464+0000","last_unstale":"2022-06-27T11:47:52.384841+0000","last_undegraded":"2022-06-27T11:47:52.384841+0000","last_fullsized":"2022-06-27T11:47:52.384841+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:39:36.234204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.368885+0000","last_change":"2022-06-27T11:47:51.975895+0000","last_active":"2022-06-27T11:47:52.368885+0000","last_peered":"2022-06-27T11:47:52.368885+0000","last_clean":"2022-06-27T11:47:52.368885+0000","last_became_active":"2022-06-27T11:47:51.975601+0000","last_became_peered":"2022-06-27T11:47:51.975601+0000","last_unstale":"2022-06-27T11:47:52.368885+0000","last_undegraded":"2022-06-27T11:47:52.368885+0000","last_fullsized":"2022-06-27T11:47:52.368885+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:43:55.833399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.385349+0000","last_change":"2022-06-27T11:47:52.383063+0000","last_active":"2022-06-27T11:47:52.385349+0000","last_peered":"2022-06-27T11:47:52.385349+0000","last_clean":"2022-06-27T11:47:52.385349+0000","last_became_active":"2022-06-27T11:47:52.382390+0000","last_became_peered":"2022-06-27T11:47:52.382390+0000","last_unstale":"2022-06-27T11:47:52.385349+0000","last_undegraded":"2022-06-27T11:47:52.385349+0000","last_fullsized":"2022-06-27T11:47:52.385349+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:40:10.929542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.369710+0000","last_change":"2022-06-27T11:47:51.980627+0000","last_active":"2022-06-27T11:47:52.369710+0000","last_peered":"2022-06-27T11:47:52.369710+0000","last_clean":"2022-06-27T11:47:52.369710+0000","last_became_active":"2022-06-27T11:47:51.980265+0000","last_became_peered":"2022-06-27T11:47:51.980265+0000","last_unstale":"2022-06-27T11:47:52.369710+0000","last_undegraded":"2022-06-27T11:47:52.369710+0000","last_fullsized":"2022-06-27T11:47:52.369710+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T14:18:32.599085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384584+0000","last_change":"2022-06-27T11:47:52.377475+0000","last_active":"2022-06-27T11:47:52.384584+0000","last_peered":"2022-06-27T11:47:52.384584+0000","last_clean":"2022-06-27T11:47:52.384584+0000","last_became_active":"2022-06-27T11:47:52.377220+0000","last_became_peered":"2022-06-27T11:47:52.377220+0000","last_unstale":"2022-06-27T11:47:52.384584+0000","last_undegraded":"2022-06-27T11:47:52.384584+0000","last_fullsized":"2022-06-27T11:47:52.384584+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T23:27:26.525688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.363531+0000","last_change":"2022-06-27T11:47:51.978195+0000","last_active":"2022-06-27T11:47:52.363531+0000","last_peered":"2022-06-27T11:47:52.363531+0000","last_clean":"2022-06-27T11:47:52.363531+0000","last_became_active":"2022-06-27T11:47:51.974673+0000","last_became_peered":"2022-06-27T11:47:51.974673+0000","last_unstale":"2022-06-27T11:47:52.363531+0000","last_undegraded":"2022-06-27T11:47:52.363531+0000","last_fullsized":"2022-06-27T11:47:52.363531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T18:09:43.518644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.382204+0000","last_change":"2022-06-27T11:47:52.378740+0000","last_active":"2022-06-27T11:47:52.382204+0000","last_peered":"2022-06-27T11:47:52.382204+0000","last_clean":"2022-06-27T11:47:52.382204+0000","last_became_active":"2022-06-27T11:47:52.378451+0000","last_became_peered":"2022-06-27T11:47:52.378451+0000","last_unstale":"2022-06-27T11:47:52.382204+0000","last_undegraded":"2022-06-27T11:47:52.382204+0000","last_fullsized":"2022-06-27T11:47:52.382204+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:04:47.829409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.366740+0000","last_change":"2022-06-27T11:47:51.983039+0000","last_active":"2022-06-27T11:47:52.366740+0000","last_peered":"2022-06-27T11:47:52.366740+0000","last_clean":"2022-06-27T11:47:52.366740+0000","last_became_active":"2022-06-27T11:47:51.982683+0000","last_became_peered":"2022-06-27T11:47:51.982683+0000","last_unstale":"2022-06-27T11:47:52.366740+0000","last_undegraded":"2022-06-27T11:47:52.366740+0000","last_fullsized":"2022-06-27T11:47:52.366740+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T17:10:16.876753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384742+0000","last_change":"2022-06-27T11:47:52.374884+0000","last_active":"2022-06-27T11:47:52.384742+0000","last_peered":"2022-06-27T11:47:52.384742+0000","last_clean":"2022-06-27T11:47:52.384742+0000","last_became_active":"2022-06-27T11:47:52.374632+0000","last_became_peered":"2022-06-27T11:47:52.374632+0000","last_unstale":"2022-06-27T11:47:52.384742+0000","last_undegraded":"2022-06-27T11:47:52.384742+0000","last_fullsized":"2022-06-27T11:47:52.384742+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T21:33:54.724089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.374167+0000","last_change":"2022-06-27T11:47:51.976772+0000","last_active":"2022-06-27T11:47:52.374167+0000","last_peered":"2022-06-27T11:47:52.374167+0000","last_clean":"2022-06-27T11:47:52.374167+0000","last_became_active":"2022-06-27T11:47:51.976475+0000","last_became_peered":"2022-06-27T11:47:51.976475+0000","last_unstale":"2022-06-27T11:47:52.374167+0000","last_undegraded":"2022-06-27T11:47:52.374167+0000","last_fullsized":"2022-06-27T11:47:52.374167+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T23:39:10.919159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.377943+0000","last_change":"2022-06-27T11:47:52.373748+0000","last_active":"2022-06-27T11:47:52.377943+0000","last_peered":"2022-06-27T11:47:52.377943+0000","last_clean":"2022-06-27T11:47:52.377943+0000","last_became_active":"2022-06-27T11:47:52.373459+0000","last_became_peered":"2022-06-27T11:47:52.373459+0000","last_unstale":"2022-06-27T11:47:52.377943+0000","last_undegraded":"2022-06-27T11:47:52.377943+0000","last_fullsized":"2022-06-27T11:47:52.377943+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T18:27:22.254416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.367083+0000","last_change":"2022-06-27T11:47:51.982926+0000","last_active":"2022-06-27T11:47:52.367083+0000","last_peered":"2022-06-27T11:47:52.367083+0000","last_clean":"2022-06-27T11:47:52.367083+0000","last_became_active":"2022-06-27T11:47:51.982638+0000","last_became_peered":"2022-06-27T11:47:51.982638+0000","last_unstale":"2022-06-27T11:47:52.367083+0000","last_undegraded":"2022-06-27T11:47:52.367083+0000","last_fullsized":"2022-06-27T11:47:52.367083+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:34:04.834822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384791+0000","last_change":"2022-06-27T11:47:52.381623+0000","last_active":"2022-06-27T11:47:52.384791+0000","last_peered":"2022-06-27T11:47:52.384791+0000","last_clean":"2022-06-27T11:47:52.384791+0000","last_became_active":"2022-06-27T11:47:52.381258+0000","last_became_peered":"2022-06-27T11:47:52.381258+0000","last_unstale":"2022-06-27T11:47:52.384791+0000","last_undegraded":"2022-06-27T11:47:52.384791+0000","last_fullsized":"2022-06-27T11:47:52.384791+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:39:07.735224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.369446+0000","last_change":"2022-06-27T11:47:51.973125+0000","last_active":"2022-06-27T11:47:52.369446+0000","last_peered":"2022-06-27T11:47:52.369446+0000","last_clean":"2022-06-27T11:47:52.369446+0000","last_became_active":"2022-06-27T11:47:51.972806+0000","last_became_peered":"2022-06-27T11:47:51.972806+0000","last_unstale":"2022-06-27T11:47:52.369446+0000","last_undegraded":"2022-06-27T11:47:52.369446+0000","last_fullsized":"2022-06-27T11:47:52.369446+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:49:23.881991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384422+0000","last_change":"2022-06-27T11:47:52.376101+0000","last_active":"2022-06-27T11:47:52.384422+0000","last_peered":"2022-06-27T11:47:52.384422+0000","last_clean":"2022-06-27T11:47:52.384422+0000","last_became_active":"2022-06-27T11:47:52.375774+0000","last_became_peered":"2022-06-27T11:47:52.375774+0000","last_unstale":"2022-06-27T11:47:52.384422+0000","last_undegraded":"2022-06-27T11:47:52.384422+0000","last_fullsized":"2022-06-27T11:47:52.384422+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T18:21:13.049096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.367430+0000","last_change":"2022-06-27T11:47:51.981981+0000","last_active":"2022-06-27T11:47:52.367430+0000","last_peered":"2022-06-27T11:47:52.367430+0000","last_clean":"2022-06-27T11:47:52.367430+0000","last_became_active":"2022-06-27T11:47:51.981749+0000","last_became_peered":"2022-06-27T11:47:51.981749+0000","last_unstale":"2022-06-27T11:47:52.367430+0000","last_undegraded":"2022-06-27T11:47:52.367430+0000","last_fullsized":"2022-06-27T11:47:52.367430+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:28:52.457719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.378190+0000","last_change":"2022-06-27T11:47:52.374223+0000","last_active":"2022-06-27T11:47:52.378190+0000","last_peered":"2022-06-27T11:47:52.378190+0000","last_clean":"2022-06-27T11:47:52.378190+0000","last_became_active":"2022-06-27T11:47:52.373932+0000","last_became_peered":"2022-06-27T11:47:52.373932+0000","last_unstale":"2022-06-27T11:47:52.378190+0000","last_undegraded":"2022-06-27T11:47:52.378190+0000","last_fullsized":"2022-06-27T11:47:52.378190+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:48:33.254458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.367357+0000","last_change":"2022-06-27T11:47:51.982895+0000","last_active":"2022-06-27T11:47:52.367357+0000","last_peered":"2022-06-27T11:47:52.367357+0000","last_clean":"2022-06-27T11:47:52.367357+0000","last_became_active":"2022-06-27T11:47:51.982619+0000","last_became_peered":"2022-06-27T11:47:51.982619+0000","last_unstale":"2022-06-27T11:47:52.367357+0000","last_undegraded":"2022-06-27T11:47:52.367357+0000","last_fullsized":"2022-06-27T11:47:52.367357+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:22:01.241112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.377169+0000","last_change":"2022-06-27T11:47:52.375225+0000","last_active":"2022-06-27T11:47:52.377169+0000","last_peered":"2022-06-27T11:47:52.377169+0000","last_clean":"2022-06-27T11:47:52.377169+0000","last_became_active":"2022-06-27T11:47:52.374925+0000","last_became_peered":"2022-06-27T11:47:52.374925+0000","last_unstale":"2022-06-27T11:47:52.377169+0000","last_undegraded":"2022-06-27T11:47:52.377169+0000","last_fullsized":"2022-06-27T11:47:52.377169+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:54:31.945902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.369615+0000","last_change":"2022-06-27T11:47:51.975202+0000","last_active":"2022-06-27T11:47:52.369615+0000","last_peered":"2022-06-27T11:47:52.369615+0000","last_clean":"2022-06-27T11:47:52.369615+0000","last_became_active":"2022-06-27T11:47:51.974968+0000","last_became_peered":"2022-06-27T11:47:51.974968+0000","last_unstale":"2022-06-27T11:47:52.369615+0000","last_undegraded":"2022-06-27T11:47:52.369615+0000","last_fullsized":"2022-06-27T11:47:52.369615+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T21:16:30.362248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.378431+0000","last_change":"2022-06-27T11:47:52.377794+0000","last_active":"2022-06-27T11:47:52.378431+0000","last_peered":"2022-06-27T11:47:52.378431+0000","last_clean":"2022-06-27T11:47:52.378431+0000","last_became_active":"2022-06-27T11:47:52.377387+0000","last_became_peered":"2022-06-27T11:47:52.377387+0000","last_unstale":"2022-06-27T11:47:52.378431+0000","last_undegraded":"2022-06-27T11:47:52.378431+0000","last_fullsized":"2022-06-27T11:47:52.378431+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T17:00:00.924967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.368792+0000","last_change":"2022-06-27T11:47:51.976575+0000","last_active":"2022-06-27T11:47:52.368792+0000","last_peered":"2022-06-27T11:47:52.368792+0000","last_clean":"2022-06-27T11:47:52.368792+0000","last_became_active":"2022-06-27T11:47:51.976300+0000","last_became_peered":"2022-06-27T11:47:51.976300+0000","last_unstale":"2022-06-27T11:47:52.368792+0000","last_undegraded":"2022-06-27T11:47:52.368792+0000","last_fullsized":"2022-06-27T11:47:52.368792+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:08:46.067375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.387174+0000","last_change":"2022-06-27T11:47:52.386298+0000","last_active":"2022-06-27T11:47:52.387174+0000","last_peered":"2022-06-27T11:47:52.387174+0000","last_clean":"2022-06-27T11:47:52.387174+0000","last_became_active":"2022-06-27T11:47:52.385918+0000","last_became_peered":"2022-06-27T11:47:52.385918+0000","last_unstale":"2022-06-27T11:47:52.387174+0000","last_undegraded":"2022-06-27T11:47:52.387174+0000","last_fullsized":"2022-06-27T11:47:52.387174+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:21:51.780895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.369604+0000","last_change":"2022-06-27T11:47:51.973817+0000","last_active":"2022-06-27T11:47:52.369604+0000","last_peered":"2022-06-27T11:47:52.369604+0000","last_clean":"2022-06-27T11:47:52.369604+0000","last_became_active":"2022-06-27T11:47:51.973515+0000","last_became_peered":"2022-06-27T11:47:51.973515+0000","last_unstale":"2022-06-27T11:47:52.369604+0000","last_undegraded":"2022-06-27T11:47:52.369604+0000","last_fullsized":"2022-06-27T11:47:52.369604+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T21:01:45.170590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.374582+0000","last_change":"2022-06-27T11:47:52.371141+0000","last_active":"2022-06-27T11:47:52.374582+0000","last_peered":"2022-06-27T11:47:52.374582+0000","last_clean":"2022-06-27T11:47:52.374582+0000","last_became_active":"2022-06-27T11:47:52.370796+0000","last_became_peered":"2022-06-27T11:47:52.370796+0000","last_unstale":"2022-06-27T11:47:52.374582+0000","last_undegraded":"2022-06-27T11:47:52.374582+0000","last_fullsized":"2022-06-27T11:47:52.374582+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T23:16:39.266615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.369465+0000","last_change":"2022-06-27T11:47:51.983071+0000","last_active":"2022-06-27T11:47:52.369465+0000","last_peered":"2022-06-27T11:47:52.369465+0000","last_clean":"2022-06-27T11:47:52.369465+0000","last_became_active":"2022-06-27T11:47:51.982756+0000","last_became_peered":"2022-06-27T11:47:51.982756+0000","last_unstale":"2022-06-27T11:47:52.369465+0000","last_undegraded":"2022-06-27T11:47:52.369465+0000","last_fullsized":"2022-06-27T11:47:52.369465+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T17:14:00.737599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.387222+0000","last_change":"2022-06-27T11:47:52.382943+0000","last_active":"2022-06-27T11:47:52.387222+0000","last_peered":"2022-06-27T11:47:52.387222+0000","last_clean":"2022-06-27T11:47:52.387222+0000","last_became_active":"2022-06-27T11:47:52.382625+0000","last_became_peered":"2022-06-27T11:47:52.382625+0000","last_unstale":"2022-06-27T11:47:52.387222+0000","last_undegraded":"2022-06-27T11:47:52.387222+0000","last_fullsized":"2022-06-27T11:47:52.387222+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:01:18.570554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.362681+0000","last_change":"2022-06-27T11:47:51.978856+0000","last_active":"2022-06-27T11:47:52.362681+0000","last_peered":"2022-06-27T11:47:52.362681+0000","last_clean":"2022-06-27T11:47:52.362681+0000","last_became_active":"2022-06-27T11:47:51.978541+0000","last_became_peered":"2022-06-27T11:47:51.978541+0000","last_unstale":"2022-06-27T11:47:52.362681+0000","last_undegraded":"2022-06-27T11:47:52.362681+0000","last_fullsized":"2022-06-27T11:47:52.362681+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:36:24.190996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.378203+0000","last_change":"2022-06-27T11:47:52.376925+0000","last_active":"2022-06-27T11:47:52.378203+0000","last_peered":"2022-06-27T11:47:52.378203+0000","last_clean":"2022-06-27T11:47:52.378203+0000","last_became_active":"2022-06-27T11:47:52.376646+0000","last_became_peered":"2022-06-27T11:47:52.376646+0000","last_unstale":"2022-06-27T11:47:52.378203+0000","last_undegraded":"2022-06-27T11:47:52.378203+0000","last_fullsized":"2022-06-27T11:47:52.378203+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:17:51.913392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.369021+0000","last_change":"2022-06-27T11:47:51.979355+0000","last_active":"2022-06-27T11:47:52.369021+0000","last_peered":"2022-06-27T11:47:52.369021+0000","last_clean":"2022-06-27T11:47:52.369021+0000","last_became_active":"2022-06-27T11:47:51.977591+0000","last_became_peered":"2022-06-27T11:47:51.977591+0000","last_unstale":"2022-06-27T11:47:52.369021+0000","last_undegraded":"2022-06-27T11:47:52.369021+0000","last_fullsized":"2022-06-27T11:47:52.369021+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T11:50:43.086014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.385028+0000","last_change":"2022-06-27T11:47:52.375271+0000","last_active":"2022-06-27T11:47:52.385028+0000","last_peered":"2022-06-27T11:47:52.385028+0000","last_clean":"2022-06-27T11:47:52.385028+0000","last_became_active":"2022-06-27T11:47:52.374985+0000","last_became_peered":"2022-06-27T11:47:52.374985+0000","last_unstale":"2022-06-27T11:47:52.385028+0000","last_undegraded":"2022-06-27T11:47:52.385028+0000","last_fullsized":"2022-06-27T11:47:52.385028+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T17:58:40.711409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.369774+0000","last_change":"2022-06-27T11:47:51.979723+0000","last_active":"2022-06-27T11:47:52.369774+0000","last_peered":"2022-06-27T11:47:52.369774+0000","last_clean":"2022-06-27T11:47:52.369774+0000","last_became_active":"2022-06-27T11:47:51.977669+0000","last_became_peered":"2022-06-27T11:47:51.977669+0000","last_unstale":"2022-06-27T11:47:52.369774+0000","last_undegraded":"2022-06-27T11:47:52.369774+0000","last_fullsized":"2022-06-27T11:47:52.369774+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T14:35:14.770254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.363444+0000","last_change":"2022-06-27T11:47:48.388880+0000","last_active":"2022-06-27T11:47:52.363444+0000","last_peered":"2022-06-27T11:47:52.363444+0000","last_clean":"2022-06-27T11:47:52.363444+0000","last_became_active":"2022-06-27T11:47:48.388639+0000","last_became_peered":"2022-06-27T11:47:48.388639+0000","last_unstale":"2022-06-27T11:47:52.363444+0000","last_undegraded":"2022-06-27T11:47:52.363444+0000","last_fullsized":"2022-06-27T11:47:52.363444+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_clean_scrub_stamp":"2022-06-27T11:47:46.780512+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:31:10.549485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.380026+0000","last_change":"2022-06-27T11:47:52.373143+0000","last_active":"2022-06-27T11:47:52.380026+0000","last_peered":"2022-06-27T11:47:52.380026+0000","last_clean":"2022-06-27T11:47:52.380026+0000","last_became_active":"2022-06-27T11:47:52.372850+0000","last_became_peered":"2022-06-27T11:47:52.372850+0000","last_unstale":"2022-06-27T11:47:52.380026+0000","last_undegraded":"2022-06-27T11:47:52.380026+0000","last_fullsized":"2022-06-27T11:47:52.380026+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:48:44.251459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.369716+0000","last_change":"2022-06-27T11:47:47.793307+0000","last_active":"2022-06-27T11:47:52.369716+0000","last_peered":"2022-06-27T11:47:52.369716+0000","last_clean":"2022-06-27T11:47:52.369716+0000","last_became_active":"2022-06-27T11:47:47.792958+0000","last_became_peered":"2022-06-27T11:47:47.792958+0000","last_unstale":"2022-06-27T11:47:52.369716+0000","last_undegraded":"2022-06-27T11:47:52.369716+0000","last_fullsized":"2022-06-27T11:47:52.369716+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_clean_scrub_stamp":"2022-06-27T11:47:46.780512+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:08:35.128804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.372474+0000","last_change":"2022-06-27T11:47:51.976872+0000","last_active":"2022-06-27T11:47:52.372474+0000","last_peered":"2022-06-27T11:47:52.372474+0000","last_clean":"2022-06-27T11:47:52.372474+0000","last_became_active":"2022-06-27T11:47:51.976546+0000","last_became_peered":"2022-06-27T11:47:51.976546+0000","last_unstale":"2022-06-27T11:47:52.372474+0000","last_undegraded":"2022-06-27T11:47:52.372474+0000","last_fullsized":"2022-06-27T11:47:52.372474+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T14:44:43.349688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.375090+0000","last_change":"2022-06-27T11:47:52.372794+0000","last_active":"2022-06-27T11:47:52.375090+0000","last_peered":"2022-06-27T11:47:52.375090+0000","last_clean":"2022-06-27T11:47:52.375090+0000","last_became_active":"2022-06-27T11:47:52.372486+0000","last_became_peered":"2022-06-27T11:47:52.372486+0000","last_unstale":"2022-06-27T11:47:52.375090+0000","last_undegraded":"2022-06-27T11:47:52.375090+0000","last_fullsized":"2022-06-27T11:47:52.375090+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:23:56.307995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.363787+0000","last_change":"2022-06-27T11:47:47.792631+0000","last_active":"2022-06-27T11:47:52.363787+0000","last_peered":"2022-06-27T11:47:52.363787+0000","last_clean":"2022-06-27T11:47:52.363787+0000","last_became_active":"2022-06-27T11:47:47.792282+0000","last_became_peered":"2022-06-27T11:47:47.792282+0000","last_unstale":"2022-06-27T11:47:52.363787+0000","last_undegraded":"2022-06-27T11:47:52.363787+0000","last_fullsized":"2022-06-27T11:47:52.363787+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_clean_scrub_stamp":"2022-06-27T11:47:46.780512+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:41:25.449379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"17'117","reported_seq":163,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.368104+0000","last_change":"2022-06-27T11:47:47.791697+0000","last_active":"2022-06-27T11:47:52.368104+0000","last_peered":"2022-06-27T11:47:52.368104+0000","last_clean":"2022-06-27T11:47:52.368104+0000","last_became_active":"2022-06-27T11:47:47.790645+0000","last_became_peered":"2022-06-27T11:47:47.790645+0000","last_unstale":"2022-06-27T11:47:52.368104+0000","last_undegraded":"2022-06-27T11:47:52.368104+0000","last_fullsized":"2022-06-27T11:47:52.368104+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_clean_scrub_stamp":"2022-06-27T11:47:46.780512+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:11:09.344666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.367962+0000","last_change":"2022-06-27T11:47:51.982728+0000","last_active":"2022-06-27T11:47:52.367962+0000","last_peered":"2022-06-27T11:47:52.367962+0000","last_clean":"2022-06-27T11:47:52.367962+0000","last_became_active":"2022-06-27T11:47:51.982445+0000","last_became_peered":"2022-06-27T11:47:51.982445+0000","last_unstale":"2022-06-27T11:47:52.367962+0000","last_undegraded":"2022-06-27T11:47:52.367962+0000","last_fullsized":"2022-06-27T11:47:52.367962+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:42:04.682687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.386922+0000","last_change":"2022-06-27T11:47:52.386015+0000","last_active":"2022-06-27T11:47:52.386922+0000","last_peered":"2022-06-27T11:47:52.386922+0000","last_clean":"2022-06-27T11:47:52.386922+0000","last_became_active":"2022-06-27T11:47:52.385731+0000","last_became_peered":"2022-06-27T11:47:52.385731+0000","last_unstale":"2022-06-27T11:47:52.386922+0000","last_undegraded":"2022-06-27T11:47:52.386922+0000","last_fullsized":"2022-06-27T11:47:52.386922+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T11:58:50.483766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.366627+0000","last_change":"2022-06-27T11:47:47.807699+0000","last_active":"2022-06-27T11:47:52.366627+0000","last_peered":"2022-06-27T11:47:52.366627+0000","last_clean":"2022-06-27T11:47:52.366627+0000","last_became_active":"2022-06-27T11:47:47.807422+0000","last_became_peered":"2022-06-27T11:47:47.807422+0000","last_unstale":"2022-06-27T11:47:52.366627+0000","last_undegraded":"2022-06-27T11:47:52.366627+0000","last_fullsized":"2022-06-27T11:47:52.366627+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_clean_scrub_stamp":"2022-06-27T11:47:46.780512+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:15:00.549295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.363922+0000","last_change":"2022-06-27T11:47:51.974468+0000","last_active":"2022-06-27T11:47:52.363922+0000","last_peered":"2022-06-27T11:47:52.363922+0000","last_clean":"2022-06-27T11:47:52.363922+0000","last_became_active":"2022-06-27T11:47:51.974160+0000","last_became_peered":"2022-06-27T11:47:51.974160+0000","last_unstale":"2022-06-27T11:47:52.363922+0000","last_undegraded":"2022-06-27T11:47:52.363922+0000","last_fullsized":"2022-06-27T11:47:52.363922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T19:14:45.694282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.380162+0000","last_change":"2022-06-27T11:47:52.373931+0000","last_active":"2022-06-27T11:47:52.380162+0000","last_peered":"2022-06-27T11:47:52.380162+0000","last_clean":"2022-06-27T11:47:52.380162+0000","last_became_active":"2022-06-27T11:47:52.373562+0000","last_became_peered":"2022-06-27T11:47:52.373562+0000","last_unstale":"2022-06-27T11:47:52.380162+0000","last_undegraded":"2022-06-27T11:47:52.380162+0000","last_fullsized":"2022-06-27T11:47:52.380162+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T14:09:22.859955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.367866+0000","last_change":"2022-06-27T11:47:47.807650+0000","last_active":"2022-06-27T11:47:52.367866+0000","last_peered":"2022-06-27T11:47:52.367866+0000","last_clean":"2022-06-27T11:47:52.367866+0000","last_became_active":"2022-06-27T11:47:47.807375+0000","last_became_peered":"2022-06-27T11:47:47.807375+0000","last_unstale":"2022-06-27T11:47:52.367866+0000","last_undegraded":"2022-06-27T11:47:52.367866+0000","last_fullsized":"2022-06-27T11:47:52.367866+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_clean_scrub_stamp":"2022-06-27T11:47:46.780512+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T14:10:48.067284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.368898+0000","last_change":"2022-06-27T11:47:51.972154+0000","last_active":"2022-06-27T11:47:52.368898+0000","last_peered":"2022-06-27T11:47:52.368898+0000","last_clean":"2022-06-27T11:47:52.368898+0000","last_became_active":"2022-06-27T11:47:51.971762+0000","last_became_peered":"2022-06-27T11:47:51.971762+0000","last_unstale":"2022-06-27T11:47:52.368898+0000","last_undegraded":"2022-06-27T11:47:52.368898+0000","last_fullsized":"2022-06-27T11:47:52.368898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T18:15:14.271005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.379879+0000","last_change":"2022-06-27T11:47:52.378294+0000","last_active":"2022-06-27T11:47:52.379879+0000","last_peered":"2022-06-27T11:47:52.379879+0000","last_clean":"2022-06-27T11:47:52.379879+0000","last_became_active":"2022-06-27T11:47:52.377704+0000","last_became_peered":"2022-06-27T11:47:52.377704+0000","last_unstale":"2022-06-27T11:47:52.379879+0000","last_undegraded":"2022-06-27T11:47:52.379879+0000","last_fullsized":"2022-06-27T11:47:52.379879+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:26:34.474081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.364180+0000","last_change":"2022-06-27T11:47:47.807600+0000","last_active":"2022-06-27T11:47:52.364180+0000","last_peered":"2022-06-27T11:47:52.364180+0000","last_clean":"2022-06-27T11:47:52.364180+0000","last_became_active":"2022-06-27T11:47:47.807284+0000","last_became_peered":"2022-06-27T11:47:47.807284+0000","last_unstale":"2022-06-27T11:47:52.364180+0000","last_undegraded":"2022-06-27T11:47:52.364180+0000","last_fullsized":"2022-06-27T11:47:52.364180+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_clean_scrub_stamp":"2022-06-27T11:47:46.780512+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T20:57:06.487027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.362687+0000","last_change":"2022-06-27T11:47:51.980734+0000","last_active":"2022-06-27T11:47:52.362687+0000","last_peered":"2022-06-27T11:47:52.362687+0000","last_clean":"2022-06-27T11:47:52.362687+0000","last_became_active":"2022-06-27T11:47:51.980476+0000","last_became_peered":"2022-06-27T11:47:51.980476+0000","last_unstale":"2022-06-27T11:47:52.362687+0000","last_undegraded":"2022-06-27T11:47:52.362687+0000","last_fullsized":"2022-06-27T11:47:52.362687+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T21:32:16.435472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.385478+0000","last_change":"2022-06-27T11:47:52.379902+0000","last_active":"2022-06-27T11:47:52.385478+0000","last_peered":"2022-06-27T11:47:52.385478+0000","last_clean":"2022-06-27T11:47:52.385478+0000","last_became_active":"2022-06-27T11:47:52.379359+0000","last_became_peered":"2022-06-27T11:47:52.379359+0000","last_unstale":"2022-06-27T11:47:52.385478+0000","last_undegraded":"2022-06-27T11:47:52.385478+0000","last_fullsized":"2022-06-27T11:47:52.385478+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:41:27.884083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.370459+0000","last_change":"2022-06-27T11:47:47.807607+0000","last_active":"2022-06-27T11:47:52.370459+0000","last_peered":"2022-06-27T11:47:52.370459+0000","last_clean":"2022-06-27T11:47:52.370459+0000","last_became_active":"2022-06-27T11:47:47.807293+0000","last_became_peered":"2022-06-27T11:47:47.807293+0000","last_unstale":"2022-06-27T11:47:52.370459+0000","last_undegraded":"2022-06-27T11:47:52.370459+0000","last_fullsized":"2022-06-27T11:47:52.370459+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_clean_scrub_stamp":"2022-06-27T11:47:46.780512+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:00:35.318372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.364133+0000","last_change":"2022-06-27T11:47:51.976055+0000","last_active":"2022-06-27T11:47:52.364133+0000","last_peered":"2022-06-27T11:47:52.364133+0000","last_clean":"2022-06-27T11:47:52.364133+0000","last_became_active":"2022-06-27T11:47:51.975303+0000","last_became_peered":"2022-06-27T11:47:51.975303+0000","last_unstale":"2022-06-27T11:47:52.364133+0000","last_undegraded":"2022-06-27T11:47:52.364133+0000","last_fullsized":"2022-06-27T11:47:52.364133+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:27:32.125525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384750+0000","last_change":"2022-06-27T11:47:52.380775+0000","last_active":"2022-06-27T11:47:52.384750+0000","last_peered":"2022-06-27T11:47:52.384750+0000","last_clean":"2022-06-27T11:47:52.384750+0000","last_became_active":"2022-06-27T11:47:52.380439+0000","last_became_peered":"2022-06-27T11:47:52.380439+0000","last_unstale":"2022-06-27T11:47:52.384750+0000","last_undegraded":"2022-06-27T11:47:52.384750+0000","last_fullsized":"2022-06-27T11:47:52.384750+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:03:44.939329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.370743+0000","last_change":"2022-06-27T11:47:47.792314+0000","last_active":"2022-06-27T11:47:52.370743+0000","last_peered":"2022-06-27T11:47:52.370743+0000","last_clean":"2022-06-27T11:47:52.370743+0000","last_became_active":"2022-06-27T11:47:47.792032+0000","last_became_peered":"2022-06-27T11:47:47.792032+0000","last_unstale":"2022-06-27T11:47:52.370743+0000","last_undegraded":"2022-06-27T11:47:52.370743+0000","last_fullsized":"2022-06-27T11:47:52.370743+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_clean_scrub_stamp":"2022-06-27T11:47:46.780512+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T14:13:03.591775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.369208+0000","last_change":"2022-06-27T11:47:51.978639+0000","last_active":"2022-06-27T11:47:52.369208+0000","last_peered":"2022-06-27T11:47:52.369208+0000","last_clean":"2022-06-27T11:47:52.369208+0000","last_became_active":"2022-06-27T11:47:51.978341+0000","last_became_peered":"2022-06-27T11:47:51.978341+0000","last_unstale":"2022-06-27T11:47:52.369208+0000","last_undegraded":"2022-06-27T11:47:52.369208+0000","last_fullsized":"2022-06-27T11:47:52.369208+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T21:34:48.070558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.381133+0000","last_change":"2022-06-27T11:47:52.373588+0000","last_active":"2022-06-27T11:47:52.381133+0000","last_peered":"2022-06-27T11:47:52.381133+0000","last_clean":"2022-06-27T11:47:52.381133+0000","last_became_active":"2022-06-27T11:47:52.373208+0000","last_became_peered":"2022-06-27T11:47:52.373208+0000","last_unstale":"2022-06-27T11:47:52.381133+0000","last_undegraded":"2022-06-27T11:47:52.381133+0000","last_fullsized":"2022-06-27T11:47:52.381133+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T20:27:28.252147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.374017+0000","last_change":"2022-06-27T11:47:51.982051+0000","last_active":"2022-06-27T11:47:52.374017+0000","last_peered":"2022-06-27T11:47:52.374017+0000","last_clean":"2022-06-27T11:47:52.374017+0000","last_became_active":"2022-06-27T11:47:51.981742+0000","last_became_peered":"2022-06-27T11:47:51.981742+0000","last_unstale":"2022-06-27T11:47:52.374017+0000","last_undegraded":"2022-06-27T11:47:52.374017+0000","last_fullsized":"2022-06-27T11:47:52.374017+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:39:56.937086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.380333+0000","last_change":"2022-06-27T11:47:52.378192+0000","last_active":"2022-06-27T11:47:52.380333+0000","last_peered":"2022-06-27T11:47:52.380333+0000","last_clean":"2022-06-27T11:47:52.380333+0000","last_became_active":"2022-06-27T11:47:52.377919+0000","last_became_peered":"2022-06-27T11:47:52.377919+0000","last_unstale":"2022-06-27T11:47:52.380333+0000","last_undegraded":"2022-06-27T11:47:52.380333+0000","last_fullsized":"2022-06-27T11:47:52.380333+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T18:51:35.757778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.364197+0000","last_change":"2022-06-27T11:47:51.981101+0000","last_active":"2022-06-27T11:47:52.364197+0000","last_peered":"2022-06-27T11:47:52.364197+0000","last_clean":"2022-06-27T11:47:52.364197+0000","last_became_active":"2022-06-27T11:47:51.980793+0000","last_became_peered":"2022-06-27T11:47:51.980793+0000","last_unstale":"2022-06-27T11:47:52.364197+0000","last_undegraded":"2022-06-27T11:47:52.364197+0000","last_fullsized":"2022-06-27T11:47:52.364197+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:28:10.206148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.372181+0000","last_change":"2022-06-27T11:47:52.370334+0000","last_active":"2022-06-27T11:47:52.372181+0000","last_peered":"2022-06-27T11:47:52.372181+0000","last_clean":"2022-06-27T11:47:52.372181+0000","last_became_active":"2022-06-27T11:47:52.370028+0000","last_became_peered":"2022-06-27T11:47:52.370028+0000","last_unstale":"2022-06-27T11:47:52.372181+0000","last_undegraded":"2022-06-27T11:47:52.372181+0000","last_fullsized":"2022-06-27T11:47:52.372181+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T17:28:12.357757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.369303+0000","last_change":"2022-06-27T11:47:51.981186+0000","last_active":"2022-06-27T11:47:52.369303+0000","last_peered":"2022-06-27T11:47:52.369303+0000","last_clean":"2022-06-27T11:47:52.369303+0000","last_became_active":"2022-06-27T11:47:51.980921+0000","last_became_peered":"2022-06-27T11:47:51.980921+0000","last_unstale":"2022-06-27T11:47:52.369303+0000","last_undegraded":"2022-06-27T11:47:52.369303+0000","last_fullsized":"2022-06-27T11:47:52.369303+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:04:40.732299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.387201+0000","last_change":"2022-06-27T11:47:52.382091+0000","last_active":"2022-06-27T11:47:52.387201+0000","last_peered":"2022-06-27T11:47:52.387201+0000","last_clean":"2022-06-27T11:47:52.387201+0000","last_became_active":"2022-06-27T11:47:52.381633+0000","last_became_peered":"2022-06-27T11:47:52.381633+0000","last_unstale":"2022-06-27T11:47:52.387201+0000","last_undegraded":"2022-06-27T11:47:52.387201+0000","last_fullsized":"2022-06-27T11:47:52.387201+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:34:44.030534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.369654+0000","last_change":"2022-06-27T11:47:51.972963+0000","last_active":"2022-06-27T11:47:52.369654+0000","last_peered":"2022-06-27T11:47:52.369654+0000","last_clean":"2022-06-27T11:47:52.369654+0000","last_became_active":"2022-06-27T11:47:51.972686+0000","last_became_peered":"2022-06-27T11:47:51.972686+0000","last_unstale":"2022-06-27T11:47:52.369654+0000","last_undegraded":"2022-06-27T11:47:52.369654+0000","last_fullsized":"2022-06-27T11:47:52.369654+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T12:11:38.008538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.385365+0000","last_change":"2022-06-27T11:47:52.383414+0000","last_active":"2022-06-27T11:47:52.385365+0000","last_peered":"2022-06-27T11:47:52.385365+0000","last_clean":"2022-06-27T11:47:52.385365+0000","last_became_active":"2022-06-27T11:47:52.383070+0000","last_became_peered":"2022-06-27T11:47:52.383070+0000","last_unstale":"2022-06-27T11:47:52.385365+0000","last_undegraded":"2022-06-27T11:47:52.385365+0000","last_fullsized":"2022-06-27T11:47:52.385365+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:11:28.410045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.371392+0000","last_change":"2022-06-27T11:47:51.971664+0000","last_active":"2022-06-27T11:47:52.371392+0000","last_peered":"2022-06-27T11:47:52.371392+0000","last_clean":"2022-06-27T11:47:52.371392+0000","last_became_active":"2022-06-27T11:47:51.971385+0000","last_became_peered":"2022-06-27T11:47:51.971385+0000","last_unstale":"2022-06-27T11:47:52.371392+0000","last_undegraded":"2022-06-27T11:47:52.371392+0000","last_fullsized":"2022-06-27T11:47:52.371392+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T18:44:38.394388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.379723+0000","last_change":"2022-06-27T11:47:52.377305+0000","last_active":"2022-06-27T11:47:52.379723+0000","last_peered":"2022-06-27T11:47:52.379723+0000","last_clean":"2022-06-27T11:47:52.379723+0000","last_became_active":"2022-06-27T11:47:52.377012+0000","last_became_peered":"2022-06-27T11:47:52.377012+0000","last_unstale":"2022-06-27T11:47:52.379723+0000","last_undegraded":"2022-06-27T11:47:52.379723+0000","last_fullsized":"2022-06-27T11:47:52.379723+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:15:14.505656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.377592+0000","last_change":"2022-06-27T11:47:52.376074+0000","last_active":"2022-06-27T11:47:52.377592+0000","last_peered":"2022-06-27T11:47:52.377592+0000","last_clean":"2022-06-27T11:47:52.377592+0000","last_became_active":"2022-06-27T11:47:52.375836+0000","last_became_peered":"2022-06-27T11:47:52.375836+0000","last_unstale":"2022-06-27T11:47:52.377592+0000","last_undegraded":"2022-06-27T11:47:52.377592+0000","last_fullsized":"2022-06-27T11:47:52.377592+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T18:46:26.900833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384440+0000","last_change":"2022-06-27T11:47:52.376668+0000","last_active":"2022-06-27T11:47:52.384440+0000","last_peered":"2022-06-27T11:47:52.384440+0000","last_clean":"2022-06-27T11:47:52.384440+0000","last_became_active":"2022-06-27T11:47:52.376386+0000","last_became_peered":"2022-06-27T11:47:52.376386+0000","last_unstale":"2022-06-27T11:47:52.384440+0000","last_undegraded":"2022-06-27T11:47:52.384440+0000","last_fullsized":"2022-06-27T11:47:52.384440+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T11:48:35.874748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.379744+0000","last_change":"2022-06-27T11:47:52.377793+0000","last_active":"2022-06-27T11:47:52.379744+0000","last_peered":"2022-06-27T11:47:52.379744+0000","last_clean":"2022-06-27T11:47:52.379744+0000","last_became_active":"2022-06-27T11:47:52.377318+0000","last_became_peered":"2022-06-27T11:47:52.377318+0000","last_unstale":"2022-06-27T11:47:52.379744+0000","last_undegraded":"2022-06-27T11:47:52.379744+0000","last_fullsized":"2022-06-27T11:47:52.379744+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T17:54:22.511301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384925+0000","last_change":"2022-06-27T11:47:52.374638+0000","last_active":"2022-06-27T11:47:52.384925+0000","last_peered":"2022-06-27T11:47:52.384925+0000","last_clean":"2022-06-27T11:47:52.384925+0000","last_became_active":"2022-06-27T11:47:52.374352+0000","last_became_peered":"2022-06-27T11:47:52.374352+0000","last_unstale":"2022-06-27T11:47:52.384925+0000","last_undegraded":"2022-06-27T11:47:52.384925+0000","last_fullsized":"2022-06-27T11:47:52.384925+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:09:59.369369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.375769+0000","last_change":"2022-06-27T11:47:52.373288+0000","last_active":"2022-06-27T11:47:52.375769+0000","last_peered":"2022-06-27T11:47:52.375769+0000","last_clean":"2022-06-27T11:47:52.375769+0000","last_became_active":"2022-06-27T11:47:52.372891+0000","last_became_peered":"2022-06-27T11:47:52.372891+0000","last_unstale":"2022-06-27T11:47:52.375769+0000","last_undegraded":"2022-06-27T11:47:52.375769+0000","last_fullsized":"2022-06-27T11:47:52.375769+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T23:16:08.088450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64756,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574852,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8988,"kb_used_data":276,"kb_used_omap":0,"kb_used_meta":8704,"kb_avail":94362852,"statfs":{"total":96636764160,"available":96627560448,"internally_reserved":0,"allocated":282624,"data_stored":669207,"data_compressed":32378,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":8912896},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45000000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30599999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38800000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34000000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574852,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8408,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":8192,"kb_avail":94363432,"statfs":{"total":96636764160,"available":96628154368,"internally_reserved":0,"allocated":212992,"data_stored":77836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8388608},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61599999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54000000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.497}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28999999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33400000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.374}]}]},{"osd":1,"up_from":13,"seq":55834574852,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8996,"kb_used_data":276,"kb_used_omap":0,"kb_used_meta":8704,"kb_avail":94362844,"statfs":{"total":96636764160,"available":96627552256,"internally_reserved":0,"allocated":282624,"data_stored":669207,"data_compressed":32378,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":8912896},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34499999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42799999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.442}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52500000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51900000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574852,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8464,"kb_used_data":200,"kb_used_omap":0,"kb_used_meta":8256,"kb_avail":94363376,"statfs":{"total":96636764160,"available":96628097024,"internally_reserved":0,"allocated":204800,"data_stored":72198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8454144},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34300000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29299999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28299999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51300000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48699999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52100000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8412,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":8192,"kb_avail":94363428,"statfs":{"total":96636764160,"available":96628150272,"internally_reserved":0,"allocated":217088,"data_stored":78839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8388608},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.438}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52400000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70199999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53700000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60099999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8472,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":8256,"kb_avail":94363368,"statfs":{"total":96636764160,"available":96628088832,"internally_reserved":0,"allocated":212992,"data_stored":77836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8454144},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33000000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.376}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59999999999999998}]}]},{"osd":4,"up_from":13,"seq":55834574852,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8476,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":8256,"kb_avail":94363364,"statfs":{"total":96636764160,"available":96628084736,"internally_reserved":0,"allocated":217088,"data_stored":78839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8454144},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39700000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45200000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45800000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48899999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8412,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":8192,"kb_avail":94363428,"statfs":{"total":96636764160,"available":96628150272,"internally_reserved":0,"allocated":217088,"data_stored":78839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8388608},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56499999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.629}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68700000000000006}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32300000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34799999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.441}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32378,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32378,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-06-27T11:47:57.426 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2022-06-27T11:47:57.578 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.579+0000 7f06ee3eb700 1 Processor -- start 2022-06-27T11:47:57.579 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.579+0000 7f06ee3eb700 1 -- start start 2022-06-27T11:47:57.588 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.579+0000 7f06ee3eb700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f06e80c2bf0 0x7f06e80c2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:57.588 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.579+0000 7f06ee3eb700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f06e80c5000 0x7f06e80c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:57.588 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.579+0000 7f06ee3eb700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06e80bc490 con 0x7f06e80c61e0 2022-06-27T11:47:57.589 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.579+0000 7f06ee3eb700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06e806a0e0 con 0x7f06e80c2bf0 2022-06-27T11:47:57.589 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.579+0000 7f06ee3eb700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06e80bc6e0 con 0x7f06e80c5000 2022-06-27T11:47:57.589 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.580+0000 7f06e7fff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f06e80c2bf0 0x7f06e80c2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:57.589 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.580+0000 7f06e7fff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f06e80c2bf0 0x7f06e80c2fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41260/0 (socket says 172.21.15.195:41260) 2022-06-27T11:47:57.590 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.580+0000 7f06e7fff700 1 -- 172.21.15.195:0/2057235149 learned_addr learned my addr 172.21.15.195:0/2057235149 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:47:57.590 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.580+0000 7f06ec988700 1 --2- 172.21.15.195:0/2057235149 >> v2:172.21.15.202:3300/0 conn(0x7f06e80c5000 0x7f06e80c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:57.590 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.580+0000 7f06e7fff700 1 -- 172.21.15.195:0/2057235149 >> v2:172.21.15.202:3300/0 conn(0x7f06e80c5000 msgr2=0x7f06e80c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:47:57.590 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.580+0000 7f06e7fff700 1 --2- 172.21.15.195:0/2057235149 >> v2:172.21.15.202:3300/0 conn(0x7f06e80c5000 0x7f06e80c3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:57.591 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.580+0000 7f06e7fff700 1 -- 172.21.15.195:0/2057235149 >> v1:172.21.15.202:6789/0 conn(0x7f06e80c61e0 legacy=0x7f06e80c48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:57.591 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.580+0000 7f06e7fff700 1 -- 172.21.15.195:0/2057235149 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06e8158220 con 0x7f06e80c2bf0 2022-06-27T11:47:57.591 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.580+0000 7f06e7fff700 1 --2- 172.21.15.195:0/2057235149 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f06e80c2bf0 0x7f06e80c2fc0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f06e80bc9a0 tx=0x7f06d80090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27074dc784ea3d80 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:57.591 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.580+0000 7f06e6ffd700 1 -- 172.21.15.195:0/2057235149 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06d8005ad0 con 0x7f06e80c2bf0 2022-06-27T11:47:57.592 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.580+0000 7f06e6ffd700 1 -- 172.21.15.195:0/2057235149 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06d8005c30 con 0x7f06e80c2bf0 2022-06-27T11:47:57.592 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.580+0000 7f06e6ffd700 1 -- 172.21.15.195:0/2057235149 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06d8004030 con 0x7f06e80c2bf0 2022-06-27T11:47:57.592 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.581+0000 7f06ee3eb700 1 -- 172.21.15.195:0/2057235149 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f06e80c2bf0 msgr2=0x7f06e80c2fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:57.592 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.581+0000 7f06ee3eb700 1 --2- 172.21.15.195:0/2057235149 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f06e80c2bf0 0x7f06e80c2fc0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f06e80bc9a0 tx=0x7f06d80090f0 comp rx=0 tx=0).stop 2022-06-27T11:47:57.593 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.581+0000 7f06ee3eb700 1 -- 172.21.15.195:0/2057235149 shutdown_connections 2022-06-27T11:47:57.593 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.581+0000 7f06ee3eb700 1 --2- 172.21.15.195:0/2057235149 >> v2:172.21.15.202:3300/0 conn(0x7f06e80c5000 0x7f06e80c3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:57.593 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.581+0000 7f06ee3eb700 1 --2- 172.21.15.195:0/2057235149 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f06e80c2bf0 0x7f06e80c2fc0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:57.593 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.581+0000 7f06ee3eb700 1 -- 172.21.15.195:0/2057235149 >> 172.21.15.195:0/2057235149 conn(0x7f06e80c9b00 msgr2=0x7f06e80c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:57.594 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.581+0000 7f06ee3eb700 1 -- 172.21.15.195:0/2057235149 shutdown_connections 2022-06-27T11:47:57.594 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.581+0000 7f06ee3eb700 1 -- 172.21.15.195:0/2057235149 wait complete. 2022-06-27T11:47:57.594 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.582+0000 7f06ee3eb700 1 Processor -- start 2022-06-27T11:47:57.594 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.582+0000 7f06ee3eb700 1 -- start start 2022-06-27T11:47:57.594 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.583+0000 7f06ee3eb700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f06e80c2bf0 0x7f06e8164de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:57.595 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.583+0000 7f06ee3eb700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f06e80c5000 0x7f06e81677c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:57.595 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.583+0000 7f06ee3eb700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f06e80c61e0 0x7f06e80c1b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:57.595 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.583+0000 7f06ee3eb700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06e8164950 con 0x7f06e80c5000 2022-06-27T11:47:57.595 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.583+0000 7f06ee3eb700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06e80be7a0 con 0x7f06e80c61e0 2022-06-27T11:47:57.596 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.583+0000 7f06ee3eb700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06e80bd230 con 0x7f06e80c2bf0 2022-06-27T11:47:57.596 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.583+0000 7f06e77fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f06e80c5000 0x7f06e81677c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:57.596 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.583+0000 7f06e77fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f06e80c5000 0x7f06e81677c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41266/0 (socket says 172.21.15.195:41266) 2022-06-27T11:47:57.596 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.583+0000 7f06e77fe700 1 -- 172.21.15.195:0/1529588069 learned_addr learned my addr 172.21.15.195:0/1529588069 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:47:57.596 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.583+0000 7f06ec988700 1 --2- 172.21.15.195:0/1529588069 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f06e80c61e0 0x7f06e80c1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:57.597 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.583+0000 7f06e7fff700 1 --2- 172.21.15.195:0/1529588069 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f06e80c2bf0 0x7f06e8164de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:57.597 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.583+0000 7f06e77fe700 1 -- 172.21.15.195:0/1529588069 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f06e80c2bf0 msgr2=0x7f06e8164de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:57.597 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.583+0000 7f06e77fe700 1 --2- 172.21.15.195:0/1529588069 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f06e80c2bf0 0x7f06e8164de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:57.597 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.583+0000 7f06e77fe700 1 -- 172.21.15.195:0/1529588069 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f06e80c61e0 msgr2=0x7f06e80c1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:57.598 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.583+0000 7f06e77fe700 1 --2- 172.21.15.195:0/1529588069 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f06e80c61e0 0x7f06e80c1b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:57.598 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.583+0000 7f06e77fe700 1 -- 172.21.15.195:0/1529588069 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06d8009350 con 0x7f06e80c5000 2022-06-27T11:47:57.598 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.583+0000 7f06ec988700 1 --2- 172.21.15.195:0/1529588069 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f06e80c61e0 0x7f06e80c1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:47:57.598 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.583+0000 7f06e77fe700 1 --2- 172.21.15.195:0/1529588069 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f06e80c5000 0x7f06e81677c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f06dc00c410 tx=0x7f06dc007e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:57.598 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.584+0000 7f06cffff700 1 -- 172.21.15.195:0/1529588069 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06dc00ce50 con 0x7f06e80c5000 2022-06-27T11:47:57.599 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.584+0000 7f06cffff700 1 -- 172.21.15.195:0/1529588069 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06dc0050d0 con 0x7f06e80c5000 2022-06-27T11:47:57.599 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.584+0000 7f06ee3eb700 1 -- 172.21.15.195:0/1529588069 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06e80bd510 con 0x7f06e80c5000 2022-06-27T11:47:57.599 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.584+0000 7f06cffff700 1 -- 172.21.15.195:0/1529588069 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06dc011590 con 0x7f06e80c5000 2022-06-27T11:47:57.599 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.584+0000 7f06ee3eb700 1 -- 172.21.15.195:0/1529588069 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06e81645e0 con 0x7f06e80c5000 2022-06-27T11:47:57.599 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.585+0000 7f06ee3eb700 1 -- 172.21.15.195:0/1529588069 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06e8164b00 con 0x7f06e80c5000 2022-06-27T11:47:57.600 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.585+0000 7f06cffff700 1 -- 172.21.15.195:0/1529588069 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f06dc019030 con 0x7f06e80c5000 2022-06-27T11:47:57.600 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.585+0000 7f06cffff700 1 --2- 172.21.15.195:0/1529588069 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f06c8063d30 0x7f06c80661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:57.600 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.585+0000 7f06cffff700 1 -- 172.21.15.195:0/1529588069 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f06dc07f8f0 con 0x7f06e80c5000 2022-06-27T11:47:57.600 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.585+0000 7f06e7fff700 1 --2- 172.21.15.195:0/1529588069 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f06c8063d30 0x7f06c80661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:57.601 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.586+0000 7f06e7fff700 1 --2- 172.21.15.195:0/1529588069 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f06c8063d30 0x7f06c80661e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f06d8005fd0 tx=0x7f06d8015000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:57.601 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.590+0000 7f06cffff700 1 -- 172.21.15.195:0/1529588069 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f06e8164b00 con 0x7f06e80c5000 2022-06-27T11:47:57.761 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.761+0000 7f06ee3eb700 1 -- 172.21.15.195:0/1529588069 --> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f06e80722c0 con 0x7f06c8063d30 2022-06-27T11:47:57.769 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.769+0000 7f06cffff700 1 -- 172.21.15.195:0/1529588069 <== mgr.4098 v2:172.21.15.195:6832/93955 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+284966 (secure 0 0 0) 0x7f06e80722c0 con 0x7f06c8063d30 2022-06-27T11:47:57.775 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:47:57.779 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.772+0000 7f06ee3eb700 1 -- 172.21.15.195:0/1529588069 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f06c8063d30 msgr2=0x7f06c80661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:57.779 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.772+0000 7f06ee3eb700 1 --2- 172.21.15.195:0/1529588069 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f06c8063d30 0x7f06c80661e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f06d8005fd0 tx=0x7f06d8015000 comp rx=0 tx=0).stop 2022-06-27T11:47:57.780 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.772+0000 7f06ee3eb700 1 -- 172.21.15.195:0/1529588069 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f06e80c5000 msgr2=0x7f06e81677c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:57.780 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.772+0000 7f06ee3eb700 1 --2- 172.21.15.195:0/1529588069 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f06e80c5000 0x7f06e81677c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f06dc00c410 tx=0x7f06dc007e40 comp rx=0 tx=0).stop 2022-06-27T11:47:57.780 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.772+0000 7f06ee3eb700 1 -- 172.21.15.195:0/1529588069 shutdown_connections 2022-06-27T11:47:57.781 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.772+0000 7f06ee3eb700 1 --2- 172.21.15.195:0/1529588069 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f06c8063d30 0x7f06c80661e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:57.781 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.772+0000 7f06ee3eb700 1 --2- 172.21.15.195:0/1529588069 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f06e80c61e0 0x7f06e80c1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:57.782 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.772+0000 7f06ee3eb700 1 --2- 172.21.15.195:0/1529588069 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f06e80c2bf0 0x7f06e8164de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:57.782 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.772+0000 7f06ee3eb700 1 --2- 172.21.15.195:0/1529588069 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f06e80c5000 0x7f06e81677c0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:57.782 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.772+0000 7f06ee3eb700 1 -- 172.21.15.195:0/1529588069 >> 172.21.15.195:0/1529588069 conn(0x7f06e80c9b00 msgr2=0x7f06e806dcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:57.783 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.773+0000 7f06ee3eb700 1 -- 172.21.15.195:0/1529588069 shutdown_connections 2022-06-27T11:47:57.783 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.773+0000 7f06ee3eb700 1 -- 172.21.15.195:0/1529588069 wait complete. 2022-06-27T11:47:57.784 INFO:teuthology.orchestra.run.smithi195.stderr:dumped all 2022-06-27T11:47:57.785 INFO:teuthology.orchestra.run.smithi195.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2022-06-27T11:47:56.340350+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":144,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":68628,"kb_used_data":1804,"kb_used_omap":0,"kb_used_meta":66752,"kb_avail":754906092,"statfs":{"total":773094113280,"available":773023838208,"internally_reserved":0,"allocated":1847296,"data_stored":1802801,"data_compressed":64756,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":68354048},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":14,"num_write_kb":262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.385895"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384962+0000","last_change":"2022-06-27T11:47:52.381823+0000","last_active":"2022-06-27T11:47:52.384962+0000","last_peered":"2022-06-27T11:47:52.384962+0000","last_clean":"2022-06-27T11:47:52.384962+0000","last_became_active":"2022-06-27T11:47:52.381551+0000","last_became_peered":"2022-06-27T11:47:52.381551+0000","last_unstale":"2022-06-27T11:47:52.384962+0000","last_undegraded":"2022-06-27T11:47:52.384962+0000","last_fullsized":"2022-06-27T11:47:52.384962+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:17:34.164061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.378135+0000","last_change":"2022-06-27T11:47:52.374939+0000","last_active":"2022-06-27T11:47:52.378135+0000","last_peered":"2022-06-27T11:47:52.378135+0000","last_clean":"2022-06-27T11:47:52.378135+0000","last_became_active":"2022-06-27T11:47:52.374620+0000","last_became_peered":"2022-06-27T11:47:52.374620+0000","last_unstale":"2022-06-27T11:47:52.378135+0000","last_undegraded":"2022-06-27T11:47:52.378135+0000","last_fullsized":"2022-06-27T11:47:52.378135+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:56:27.254098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384127+0000","last_change":"2022-06-27T11:47:52.380802+0000","last_active":"2022-06-27T11:47:52.384127+0000","last_peered":"2022-06-27T11:47:52.384127+0000","last_clean":"2022-06-27T11:47:52.384127+0000","last_became_active":"2022-06-27T11:47:52.380500+0000","last_became_peered":"2022-06-27T11:47:52.380500+0000","last_unstale":"2022-06-27T11:47:52.384127+0000","last_undegraded":"2022-06-27T11:47:52.384127+0000","last_fullsized":"2022-06-27T11:47:52.384127+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:06:42.889598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384114+0000","last_change":"2022-06-27T11:47:52.380943+0000","last_active":"2022-06-27T11:47:52.384114+0000","last_peered":"2022-06-27T11:47:52.384114+0000","last_clean":"2022-06-27T11:47:52.384114+0000","last_became_active":"2022-06-27T11:47:52.380648+0000","last_became_peered":"2022-06-27T11:47:52.380648+0000","last_unstale":"2022-06-27T11:47:52.384114+0000","last_undegraded":"2022-06-27T11:47:52.384114+0000","last_fullsized":"2022-06-27T11:47:52.384114+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:14:19.729605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.386875+0000","last_change":"2022-06-27T11:47:52.386144+0000","last_active":"2022-06-27T11:47:52.386875+0000","last_peered":"2022-06-27T11:47:52.386875+0000","last_clean":"2022-06-27T11:47:52.386875+0000","last_became_active":"2022-06-27T11:47:52.385586+0000","last_became_peered":"2022-06-27T11:47:52.385586+0000","last_unstale":"2022-06-27T11:47:52.386875+0000","last_undegraded":"2022-06-27T11:47:52.386875+0000","last_fullsized":"2022-06-27T11:47:52.386875+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T14:30:03.660749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.387182+0000","last_change":"2022-06-27T11:47:52.381906+0000","last_active":"2022-06-27T11:47:52.387182+0000","last_peered":"2022-06-27T11:47:52.387182+0000","last_clean":"2022-06-27T11:47:52.387182+0000","last_became_active":"2022-06-27T11:47:52.381279+0000","last_became_peered":"2022-06-27T11:47:52.381279+0000","last_unstale":"2022-06-27T11:47:52.387182+0000","last_undegraded":"2022-06-27T11:47:52.387182+0000","last_fullsized":"2022-06-27T11:47:52.387182+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T18:41:03.991762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.383137+0000","last_change":"2022-06-27T11:47:52.377128+0000","last_active":"2022-06-27T11:47:52.383137+0000","last_peered":"2022-06-27T11:47:52.383137+0000","last_clean":"2022-06-27T11:47:52.383137+0000","last_became_active":"2022-06-27T11:47:52.376499+0000","last_became_peered":"2022-06-27T11:47:52.376499+0000","last_unstale":"2022-06-27T11:47:52.383137+0000","last_undegraded":"2022-06-27T11:47:52.383137+0000","last_fullsized":"2022-06-27T11:47:52.383137+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:00:13.208145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.378171+0000","last_change":"2022-06-27T11:47:52.374169+0000","last_active":"2022-06-27T11:47:52.378171+0000","last_peered":"2022-06-27T11:47:52.378171+0000","last_clean":"2022-06-27T11:47:52.378171+0000","last_became_active":"2022-06-27T11:47:52.373917+0000","last_became_peered":"2022-06-27T11:47:52.373917+0000","last_unstale":"2022-06-27T11:47:52.378171+0000","last_undegraded":"2022-06-27T11:47:52.378171+0000","last_fullsized":"2022-06-27T11:47:52.378171+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:43:52.741254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384769+0000","last_change":"2022-06-27T11:47:52.382039+0000","last_active":"2022-06-27T11:47:52.384769+0000","last_peered":"2022-06-27T11:47:52.384769+0000","last_clean":"2022-06-27T11:47:52.384769+0000","last_became_active":"2022-06-27T11:47:52.381202+0000","last_became_peered":"2022-06-27T11:47:52.381202+0000","last_unstale":"2022-06-27T11:47:52.384769+0000","last_undegraded":"2022-06-27T11:47:52.384769+0000","last_fullsized":"2022-06-27T11:47:52.384769+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:13:02.553823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.383894+0000","last_change":"2022-06-27T11:47:52.380839+0000","last_active":"2022-06-27T11:47:52.383894+0000","last_peered":"2022-06-27T11:47:52.383894+0000","last_clean":"2022-06-27T11:47:52.383894+0000","last_became_active":"2022-06-27T11:47:52.380527+0000","last_became_peered":"2022-06-27T11:47:52.380527+0000","last_unstale":"2022-06-27T11:47:52.383894+0000","last_undegraded":"2022-06-27T11:47:52.383894+0000","last_fullsized":"2022-06-27T11:47:52.383894+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T23:18:07.086598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.374783+0000","last_change":"2022-06-27T11:47:52.371267+0000","last_active":"2022-06-27T11:47:52.374783+0000","last_peered":"2022-06-27T11:47:52.374783+0000","last_clean":"2022-06-27T11:47:52.374783+0000","last_became_active":"2022-06-27T11:47:52.370921+0000","last_became_peered":"2022-06-27T11:47:52.370921+0000","last_unstale":"2022-06-27T11:47:52.374783+0000","last_undegraded":"2022-06-27T11:47:52.374783+0000","last_fullsized":"2022-06-27T11:47:52.374783+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T19:22:39.648715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.378994+0000","last_change":"2022-06-27T11:47:52.375381+0000","last_active":"2022-06-27T11:47:52.378994+0000","last_peered":"2022-06-27T11:47:52.378994+0000","last_clean":"2022-06-27T11:47:52.378994+0000","last_became_active":"2022-06-27T11:47:52.375089+0000","last_became_peered":"2022-06-27T11:47:52.375089+0000","last_unstale":"2022-06-27T11:47:52.378994+0000","last_undegraded":"2022-06-27T11:47:52.378994+0000","last_fullsized":"2022-06-27T11:47:52.378994+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:43:14.830424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384993+0000","last_change":"2022-06-27T11:47:52.381690+0000","last_active":"2022-06-27T11:47:52.384993+0000","last_peered":"2022-06-27T11:47:52.384993+0000","last_clean":"2022-06-27T11:47:52.384993+0000","last_became_active":"2022-06-27T11:47:52.381472+0000","last_became_peered":"2022-06-27T11:47:52.381472+0000","last_unstale":"2022-06-27T11:47:52.384993+0000","last_undegraded":"2022-06-27T11:47:52.384993+0000","last_fullsized":"2022-06-27T11:47:52.384993+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:14:56.919536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.378838+0000","last_change":"2022-06-27T11:47:52.376063+0000","last_active":"2022-06-27T11:47:52.378838+0000","last_peered":"2022-06-27T11:47:52.378838+0000","last_clean":"2022-06-27T11:47:52.378838+0000","last_became_active":"2022-06-27T11:47:52.375778+0000","last_became_peered":"2022-06-27T11:47:52.375778+0000","last_unstale":"2022-06-27T11:47:52.378838+0000","last_undegraded":"2022-06-27T11:47:52.378838+0000","last_fullsized":"2022-06-27T11:47:52.378838+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T23:16:52.221967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.383160+0000","last_change":"2022-06-27T11:47:52.377297+0000","last_active":"2022-06-27T11:47:52.383160+0000","last_peered":"2022-06-27T11:47:52.383160+0000","last_clean":"2022-06-27T11:47:52.383160+0000","last_became_active":"2022-06-27T11:47:52.376855+0000","last_became_peered":"2022-06-27T11:47:52.376855+0000","last_unstale":"2022-06-27T11:47:52.383160+0000","last_undegraded":"2022-06-27T11:47:52.383160+0000","last_fullsized":"2022-06-27T11:47:52.383160+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:33:39.990522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.387168+0000","last_change":"2022-06-27T11:47:52.381821+0000","last_active":"2022-06-27T11:47:52.387168+0000","last_peered":"2022-06-27T11:47:52.387168+0000","last_clean":"2022-06-27T11:47:52.387168+0000","last_became_active":"2022-06-27T11:47:52.381548+0000","last_became_peered":"2022-06-27T11:47:52.381548+0000","last_unstale":"2022-06-27T11:47:52.387168+0000","last_undegraded":"2022-06-27T11:47:52.387168+0000","last_fullsized":"2022-06-27T11:47:52.387168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T23:14:53.320016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384753+0000","last_change":"2022-06-27T11:47:52.381811+0000","last_active":"2022-06-27T11:47:52.384753+0000","last_peered":"2022-06-27T11:47:52.384753+0000","last_clean":"2022-06-27T11:47:52.384753+0000","last_became_active":"2022-06-27T11:47:52.381554+0000","last_became_peered":"2022-06-27T11:47:52.381554+0000","last_unstale":"2022-06-27T11:47:52.384753+0000","last_undegraded":"2022-06-27T11:47:52.384753+0000","last_fullsized":"2022-06-27T11:47:52.384753+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T14:19:38.519380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.387218+0000","last_change":"2022-06-27T11:47:52.382952+0000","last_active":"2022-06-27T11:47:52.387218+0000","last_peered":"2022-06-27T11:47:52.387218+0000","last_clean":"2022-06-27T11:47:52.387218+0000","last_became_active":"2022-06-27T11:47:52.382691+0000","last_became_peered":"2022-06-27T11:47:52.382691+0000","last_unstale":"2022-06-27T11:47:52.387218+0000","last_undegraded":"2022-06-27T11:47:52.387218+0000","last_fullsized":"2022-06-27T11:47:52.387218+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:28:54.743970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.383682+0000","last_change":"2022-06-27T11:47:52.379083+0000","last_active":"2022-06-27T11:47:52.383682+0000","last_peered":"2022-06-27T11:47:52.383682+0000","last_clean":"2022-06-27T11:47:52.383682+0000","last_became_active":"2022-06-27T11:47:52.378724+0000","last_became_peered":"2022-06-27T11:47:52.378724+0000","last_unstale":"2022-06-27T11:47:52.383682+0000","last_undegraded":"2022-06-27T11:47:52.383682+0000","last_fullsized":"2022-06-27T11:47:52.383682+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T21:25:07.786935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.383275+0000","last_change":"2022-06-27T11:47:52.377947+0000","last_active":"2022-06-27T11:47:52.383275+0000","last_peered":"2022-06-27T11:47:52.383275+0000","last_clean":"2022-06-27T11:47:52.383275+0000","last_became_active":"2022-06-27T11:47:52.377652+0000","last_became_peered":"2022-06-27T11:47:52.377652+0000","last_unstale":"2022-06-27T11:47:52.383275+0000","last_undegraded":"2022-06-27T11:47:52.383275+0000","last_fullsized":"2022-06-27T11:47:52.383275+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T20:48:28.642052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384444+0000","last_change":"2022-06-27T11:47:52.381920+0000","last_active":"2022-06-27T11:47:52.384444+0000","last_peered":"2022-06-27T11:47:52.384444+0000","last_clean":"2022-06-27T11:47:52.384444+0000","last_became_active":"2022-06-27T11:47:52.381477+0000","last_became_peered":"2022-06-27T11:47:52.381477+0000","last_unstale":"2022-06-27T11:47:52.384444+0000","last_undegraded":"2022-06-27T11:47:52.384444+0000","last_fullsized":"2022-06-27T11:47:52.384444+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T20:42:25.529336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.386345+0000","last_change":"2022-06-27T11:47:52.385231+0000","last_active":"2022-06-27T11:47:52.386345+0000","last_peered":"2022-06-27T11:47:52.386345+0000","last_clean":"2022-06-27T11:47:52.386345+0000","last_became_active":"2022-06-27T11:47:52.384972+0000","last_became_peered":"2022-06-27T11:47:52.384972+0000","last_unstale":"2022-06-27T11:47:52.386345+0000","last_undegraded":"2022-06-27T11:47:52.386345+0000","last_fullsized":"2022-06-27T11:47:52.386345+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T17:37:26.084788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.385330+0000","last_change":"2022-06-27T11:47:52.381358+0000","last_active":"2022-06-27T11:47:52.385330+0000","last_peered":"2022-06-27T11:47:52.385330+0000","last_clean":"2022-06-27T11:47:52.385330+0000","last_became_active":"2022-06-27T11:47:52.380997+0000","last_became_peered":"2022-06-27T11:47:52.380997+0000","last_unstale":"2022-06-27T11:47:52.385330+0000","last_undegraded":"2022-06-27T11:47:52.385330+0000","last_fullsized":"2022-06-27T11:47:52.385330+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T21:44:16.061234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384432+0000","last_change":"2022-06-27T11:47:52.381899+0000","last_active":"2022-06-27T11:47:52.384432+0000","last_peered":"2022-06-27T11:47:52.384432+0000","last_clean":"2022-06-27T11:47:52.384432+0000","last_became_active":"2022-06-27T11:47:52.381645+0000","last_became_peered":"2022-06-27T11:47:52.381645+0000","last_unstale":"2022-06-27T11:47:52.384432+0000","last_undegraded":"2022-06-27T11:47:52.384432+0000","last_fullsized":"2022-06-27T11:47:52.384432+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:05:59.742023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.385053+0000","last_change":"2022-06-27T11:47:52.378736+0000","last_active":"2022-06-27T11:47:52.385053+0000","last_peered":"2022-06-27T11:47:52.385053+0000","last_clean":"2022-06-27T11:47:52.385053+0000","last_became_active":"2022-06-27T11:47:52.378475+0000","last_became_peered":"2022-06-27T11:47:52.378475+0000","last_unstale":"2022-06-27T11:47:52.385053+0000","last_undegraded":"2022-06-27T11:47:52.385053+0000","last_fullsized":"2022-06-27T11:47:52.385053+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T12:42:12.528716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384922+0000","last_change":"2022-06-27T11:47:52.378044+0000","last_active":"2022-06-27T11:47:52.384922+0000","last_peered":"2022-06-27T11:47:52.384922+0000","last_clean":"2022-06-27T11:47:52.384922+0000","last_became_active":"2022-06-27T11:47:52.377791+0000","last_became_peered":"2022-06-27T11:47:52.377791+0000","last_unstale":"2022-06-27T11:47:52.384922+0000","last_undegraded":"2022-06-27T11:47:52.384922+0000","last_fullsized":"2022-06-27T11:47:52.384922+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:21:45.615162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384115+0000","last_change":"2022-06-27T11:47:52.382267+0000","last_active":"2022-06-27T11:47:52.384115+0000","last_peered":"2022-06-27T11:47:52.384115+0000","last_clean":"2022-06-27T11:47:52.384115+0000","last_became_active":"2022-06-27T11:47:52.381985+0000","last_became_peered":"2022-06-27T11:47:52.381985+0000","last_unstale":"2022-06-27T11:47:52.384115+0000","last_undegraded":"2022-06-27T11:47:52.384115+0000","last_fullsized":"2022-06-27T11:47:52.384115+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:34:09.509759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.365170+0000","last_change":"2022-06-27T11:47:51.982183+0000","last_active":"2022-06-27T11:47:52.365170+0000","last_peered":"2022-06-27T11:47:52.365170+0000","last_clean":"2022-06-27T11:47:52.365170+0000","last_became_active":"2022-06-27T11:47:51.981339+0000","last_became_peered":"2022-06-27T11:47:51.981339+0000","last_unstale":"2022-06-27T11:47:52.365170+0000","last_undegraded":"2022-06-27T11:47:52.365170+0000","last_fullsized":"2022-06-27T11:47:52.365170+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:08:09.223947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384452+0000","last_change":"2022-06-27T11:47:52.377197+0000","last_active":"2022-06-27T11:47:52.384452+0000","last_peered":"2022-06-27T11:47:52.384452+0000","last_clean":"2022-06-27T11:47:52.384452+0000","last_became_active":"2022-06-27T11:47:52.376925+0000","last_became_peered":"2022-06-27T11:47:52.376925+0000","last_unstale":"2022-06-27T11:47:52.384452+0000","last_undegraded":"2022-06-27T11:47:52.384452+0000","last_fullsized":"2022-06-27T11:47:52.384452+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:39:27.103978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.364221+0000","last_change":"2022-06-27T11:47:51.981074+0000","last_active":"2022-06-27T11:47:52.364221+0000","last_peered":"2022-06-27T11:47:52.364221+0000","last_clean":"2022-06-27T11:47:52.364221+0000","last_became_active":"2022-06-27T11:47:51.980636+0000","last_became_peered":"2022-06-27T11:47:51.980636+0000","last_unstale":"2022-06-27T11:47:52.364221+0000","last_undegraded":"2022-06-27T11:47:52.364221+0000","last_fullsized":"2022-06-27T11:47:52.364221+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T23:32:52.850801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384380+0000","last_change":"2022-06-27T11:47:52.378415+0000","last_active":"2022-06-27T11:47:52.384380+0000","last_peered":"2022-06-27T11:47:52.384380+0000","last_clean":"2022-06-27T11:47:52.384380+0000","last_became_active":"2022-06-27T11:47:52.378123+0000","last_became_peered":"2022-06-27T11:47:52.378123+0000","last_unstale":"2022-06-27T11:47:52.384380+0000","last_undegraded":"2022-06-27T11:47:52.384380+0000","last_fullsized":"2022-06-27T11:47:52.384380+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T11:54:37.127828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.369137+0000","last_change":"2022-06-27T11:47:51.975684+0000","last_active":"2022-06-27T11:47:52.369137+0000","last_peered":"2022-06-27T11:47:52.369137+0000","last_clean":"2022-06-27T11:47:52.369137+0000","last_became_active":"2022-06-27T11:47:51.974902+0000","last_became_peered":"2022-06-27T11:47:51.974902+0000","last_unstale":"2022-06-27T11:47:52.369137+0000","last_undegraded":"2022-06-27T11:47:52.369137+0000","last_fullsized":"2022-06-27T11:47:52.369137+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:51:45.780798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.378008+0000","last_change":"2022-06-27T11:47:52.376089+0000","last_active":"2022-06-27T11:47:52.378008+0000","last_peered":"2022-06-27T11:47:52.378008+0000","last_clean":"2022-06-27T11:47:52.378008+0000","last_became_active":"2022-06-27T11:47:52.375780+0000","last_became_peered":"2022-06-27T11:47:52.375780+0000","last_unstale":"2022-06-27T11:47:52.378008+0000","last_undegraded":"2022-06-27T11:47:52.378008+0000","last_fullsized":"2022-06-27T11:47:52.378008+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T14:17:53.942880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.371313+0000","last_change":"2022-06-27T11:47:51.974071+0000","last_active":"2022-06-27T11:47:52.371313+0000","last_peered":"2022-06-27T11:47:52.371313+0000","last_clean":"2022-06-27T11:47:52.371313+0000","last_became_active":"2022-06-27T11:47:51.973682+0000","last_became_peered":"2022-06-27T11:47:51.973682+0000","last_unstale":"2022-06-27T11:47:52.371313+0000","last_undegraded":"2022-06-27T11:47:52.371313+0000","last_fullsized":"2022-06-27T11:47:52.371313+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T17:40:03.705083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384841+0000","last_change":"2022-06-27T11:47:52.377716+0000","last_active":"2022-06-27T11:47:52.384841+0000","last_peered":"2022-06-27T11:47:52.384841+0000","last_clean":"2022-06-27T11:47:52.384841+0000","last_became_active":"2022-06-27T11:47:52.377464+0000","last_became_peered":"2022-06-27T11:47:52.377464+0000","last_unstale":"2022-06-27T11:47:52.384841+0000","last_undegraded":"2022-06-27T11:47:52.384841+0000","last_fullsized":"2022-06-27T11:47:52.384841+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:39:36.234204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.368885+0000","last_change":"2022-06-27T11:47:51.975895+0000","last_active":"2022-06-27T11:47:52.368885+0000","last_peered":"2022-06-27T11:47:52.368885+0000","last_clean":"2022-06-27T11:47:52.368885+0000","last_became_active":"2022-06-27T11:47:51.975601+0000","last_became_peered":"2022-06-27T11:47:51.975601+0000","last_unstale":"2022-06-27T11:47:52.368885+0000","last_undegraded":"2022-06-27T11:47:52.368885+0000","last_fullsized":"2022-06-27T11:47:52.368885+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:43:55.833399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.385349+0000","last_change":"2022-06-27T11:47:52.383063+0000","last_active":"2022-06-27T11:47:52.385349+0000","last_peered":"2022-06-27T11:47:52.385349+0000","last_clean":"2022-06-27T11:47:52.385349+0000","last_became_active":"2022-06-27T11:47:52.382390+0000","last_became_peered":"2022-06-27T11:47:52.382390+0000","last_unstale":"2022-06-27T11:47:52.385349+0000","last_undegraded":"2022-06-27T11:47:52.385349+0000","last_fullsized":"2022-06-27T11:47:52.385349+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:40:10.929542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.369710+0000","last_change":"2022-06-27T11:47:51.980627+0000","last_active":"2022-06-27T11:47:52.369710+0000","last_peered":"2022-06-27T11:47:52.369710+0000","last_clean":"2022-06-27T11:47:52.369710+0000","last_became_active":"2022-06-27T11:47:51.980265+0000","last_became_peered":"2022-06-27T11:47:51.980265+0000","last_unstale":"2022-06-27T11:47:52.369710+0000","last_undegraded":"2022-06-27T11:47:52.369710+0000","last_fullsized":"2022-06-27T11:47:52.369710+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T14:18:32.599085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384584+0000","last_change":"2022-06-27T11:47:52.377475+0000","last_active":"2022-06-27T11:47:52.384584+0000","last_peered":"2022-06-27T11:47:52.384584+0000","last_clean":"2022-06-27T11:47:52.384584+0000","last_became_active":"2022-06-27T11:47:52.377220+0000","last_became_peered":"2022-06-27T11:47:52.377220+0000","last_unstale":"2022-06-27T11:47:52.384584+0000","last_undegraded":"2022-06-27T11:47:52.384584+0000","last_fullsized":"2022-06-27T11:47:52.384584+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T23:27:26.525688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.363531+0000","last_change":"2022-06-27T11:47:51.978195+0000","last_active":"2022-06-27T11:47:52.363531+0000","last_peered":"2022-06-27T11:47:52.363531+0000","last_clean":"2022-06-27T11:47:52.363531+0000","last_became_active":"2022-06-27T11:47:51.974673+0000","last_became_peered":"2022-06-27T11:47:51.974673+0000","last_unstale":"2022-06-27T11:47:52.363531+0000","last_undegraded":"2022-06-27T11:47:52.363531+0000","last_fullsized":"2022-06-27T11:47:52.363531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T18:09:43.518644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.382204+0000","last_change":"2022-06-27T11:47:52.378740+0000","last_active":"2022-06-27T11:47:52.382204+0000","last_peered":"2022-06-27T11:47:52.382204+0000","last_clean":"2022-06-27T11:47:52.382204+0000","last_became_active":"2022-06-27T11:47:52.378451+0000","last_became_peered":"2022-06-27T11:47:52.378451+0000","last_unstale":"2022-06-27T11:47:52.382204+0000","last_undegraded":"2022-06-27T11:47:52.382204+0000","last_fullsized":"2022-06-27T11:47:52.382204+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:04:47.829409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.366740+0000","last_change":"2022-06-27T11:47:51.983039+0000","last_active":"2022-06-27T11:47:52.366740+0000","last_peered":"2022-06-27T11:47:52.366740+0000","last_clean":"2022-06-27T11:47:52.366740+0000","last_became_active":"2022-06-27T11:47:51.982683+0000","last_became_peered":"2022-06-27T11:47:51.982683+0000","last_unstale":"2022-06-27T11:47:52.366740+0000","last_undegraded":"2022-06-27T11:47:52.366740+0000","last_fullsized":"2022-06-27T11:47:52.366740+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T17:10:16.876753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384742+0000","last_change":"2022-06-27T11:47:52.374884+0000","last_active":"2022-06-27T11:47:52.384742+0000","last_peered":"2022-06-27T11:47:52.384742+0000","last_clean":"2022-06-27T11:47:52.384742+0000","last_became_active":"2022-06-27T11:47:52.374632+0000","last_became_peered":"2022-06-27T11:47:52.374632+0000","last_unstale":"2022-06-27T11:47:52.384742+0000","last_undegraded":"2022-06-27T11:47:52.384742+0000","last_fullsized":"2022-06-27T11:47:52.384742+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T21:33:54.724089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.374167+0000","last_change":"2022-06-27T11:47:51.976772+0000","last_active":"2022-06-27T11:47:52.374167+0000","last_peered":"2022-06-27T11:47:52.374167+0000","last_clean":"2022-06-27T11:47:52.374167+0000","last_became_active":"2022-06-27T11:47:51.976475+0000","last_became_peered":"2022-06-27T11:47:51.976475+0000","last_unstale":"2022-06-27T11:47:52.374167+0000","last_undegraded":"2022-06-27T11:47:52.374167+0000","last_fullsized":"2022-06-27T11:47:52.374167+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T23:39:10.919159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.377943+0000","last_change":"2022-06-27T11:47:52.373748+0000","last_active":"2022-06-27T11:47:52.377943+0000","last_peered":"2022-06-27T11:47:52.377943+0000","last_clean":"2022-06-27T11:47:52.377943+0000","last_became_active":"2022-06-27T11:47:52.373459+0000","last_became_peered":"2022-06-27T11:47:52.373459+0000","last_unstale":"2022-06-27T11:47:52.377943+0000","last_undegraded":"2022-06-27T11:47:52.377943+0000","last_fullsized":"2022-06-27T11:47:52.377943+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T18:27:22.254416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.367083+0000","last_change":"2022-06-27T11:47:51.982926+0000","last_active":"2022-06-27T11:47:52.367083+0000","last_peered":"2022-06-27T11:47:52.367083+0000","last_clean":"2022-06-27T11:47:52.367083+0000","last_became_active":"2022-06-27T11:47:51.982638+0000","last_became_peered":"2022-06-27T11:47:51.982638+0000","last_unstale":"2022-06-27T11:47:52.367083+0000","last_undegraded":"2022-06-27T11:47:52.367083+0000","last_fullsized":"2022-06-27T11:47:52.367083+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:34:04.834822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384791+0000","last_change":"2022-06-27T11:47:52.381623+0000","last_active":"2022-06-27T11:47:52.384791+0000","last_peered":"2022-06-27T11:47:52.384791+0000","last_clean":"2022-06-27T11:47:52.384791+0000","last_became_active":"2022-06-27T11:47:52.381258+0000","last_became_peered":"2022-06-27T11:47:52.381258+0000","last_unstale":"2022-06-27T11:47:52.384791+0000","last_undegraded":"2022-06-27T11:47:52.384791+0000","last_fullsized":"2022-06-27T11:47:52.384791+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:39:07.735224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.369446+0000","last_change":"2022-06-27T11:47:51.973125+0000","last_active":"2022-06-27T11:47:52.369446+0000","last_peered":"2022-06-27T11:47:52.369446+0000","last_clean":"2022-06-27T11:47:52.369446+0000","last_became_active":"2022-06-27T11:47:51.972806+0000","last_became_peered":"2022-06-27T11:47:51.972806+0000","last_unstale":"2022-06-27T11:47:52.369446+0000","last_undegraded":"2022-06-27T11:47:52.369446+0000","last_fullsized":"2022-06-27T11:47:52.369446+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:49:23.881991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384422+0000","last_change":"2022-06-27T11:47:52.376101+0000","last_active":"2022-06-27T11:47:52.384422+0000","last_peered":"2022-06-27T11:47:52.384422+0000","last_clean":"2022-06-27T11:47:52.384422+0000","last_became_active":"2022-06-27T11:47:52.375774+0000","last_became_peered":"2022-06-27T11:47:52.375774+0000","last_unstale":"2022-06-27T11:47:52.384422+0000","last_undegraded":"2022-06-27T11:47:52.384422+0000","last_fullsized":"2022-06-27T11:47:52.384422+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T18:21:13.049096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.367430+0000","last_change":"2022-06-27T11:47:51.981981+0000","last_active":"2022-06-27T11:47:52.367430+0000","last_peered":"2022-06-27T11:47:52.367430+0000","last_clean":"2022-06-27T11:47:52.367430+0000","last_became_active":"2022-06-27T11:47:51.981749+0000","last_became_peered":"2022-06-27T11:47:51.981749+0000","last_unstale":"2022-06-27T11:47:52.367430+0000","last_undegraded":"2022-06-27T11:47:52.367430+0000","last_fullsized":"2022-06-27T11:47:52.367430+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:28:52.457719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.378190+0000","last_change":"2022-06-27T11:47:52.374223+0000","last_active":"2022-06-27T11:47:52.378190+0000","last_peered":"2022-06-27T11:47:52.378190+0000","last_clean":"2022-06-27T11:47:52.378190+0000","last_became_active":"2022-06-27T11:47:52.373932+0000","last_became_peered":"2022-06-27T11:47:52.373932+0000","last_unstale":"2022-06-27T11:47:52.378190+0000","last_undegraded":"2022-06-27T11:47:52.378190+0000","last_fullsized":"2022-06-27T11:47:52.378190+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:48:33.254458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.367357+0000","last_change":"2022-06-27T11:47:51.982895+0000","last_active":"2022-06-27T11:47:52.367357+0000","last_peered":"2022-06-27T11:47:52.367357+0000","last_clean":"2022-06-27T11:47:52.367357+0000","last_became_active":"2022-06-27T11:47:51.982619+0000","last_became_peered":"2022-06-27T11:47:51.982619+0000","last_unstale":"2022-06-27T11:47:52.367357+0000","last_undegraded":"2022-06-27T11:47:52.367357+0000","last_fullsized":"2022-06-27T11:47:52.367357+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:22:01.241112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.377169+0000","last_change":"2022-06-27T11:47:52.375225+0000","last_active":"2022-06-27T11:47:52.377169+0000","last_peered":"2022-06-27T11:47:52.377169+0000","last_clean":"2022-06-27T11:47:52.377169+0000","last_became_active":"2022-06-27T11:47:52.374925+0000","last_became_peered":"2022-06-27T11:47:52.374925+0000","last_unstale":"2022-06-27T11:47:52.377169+0000","last_undegraded":"2022-06-27T11:47:52.377169+0000","last_fullsized":"2022-06-27T11:47:52.377169+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:54:31.945902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.369615+0000","last_change":"2022-06-27T11:47:51.975202+0000","last_active":"2022-06-27T11:47:52.369615+0000","last_peered":"2022-06-27T11:47:52.369615+0000","last_clean":"2022-06-27T11:47:52.369615+0000","last_became_active":"2022-06-27T11:47:51.974968+0000","last_became_peered":"2022-06-27T11:47:51.974968+0000","last_unstale":"2022-06-27T11:47:52.369615+0000","last_undegraded":"2022-06-27T11:47:52.369615+0000","last_fullsized":"2022-06-27T11:47:52.369615+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T21:16:30.362248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.378431+0000","last_change":"2022-06-27T11:47:52.377794+0000","last_active":"2022-06-27T11:47:52.378431+0000","last_peered":"2022-06-27T11:47:52.378431+0000","last_clean":"2022-06-27T11:47:52.378431+0000","last_became_active":"2022-06-27T11:47:52.377387+0000","last_became_peered":"2022-06-27T11:47:52.377387+0000","last_unstale":"2022-06-27T11:47:52.378431+0000","last_undegraded":"2022-06-27T11:47:52.378431+0000","last_fullsized":"2022-06-27T11:47:52.378431+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T17:00:00.924967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.368792+0000","last_change":"2022-06-27T11:47:51.976575+0000","last_active":"2022-06-27T11:47:52.368792+0000","last_peered":"2022-06-27T11:47:52.368792+0000","last_clean":"2022-06-27T11:47:52.368792+0000","last_became_active":"2022-06-27T11:47:51.976300+0000","last_became_peered":"2022-06-27T11:47:51.976300+0000","last_unstale":"2022-06-27T11:47:52.368792+0000","last_undegraded":"2022-06-27T11:47:52.368792+0000","last_fullsized":"2022-06-27T11:47:52.368792+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:08:46.067375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.387174+0000","last_change":"2022-06-27T11:47:52.386298+0000","last_active":"2022-06-27T11:47:52.387174+0000","last_peered":"2022-06-27T11:47:52.387174+0000","last_clean":"2022-06-27T11:47:52.387174+0000","last_became_active":"2022-06-27T11:47:52.385918+0000","last_became_peered":"2022-06-27T11:47:52.385918+0000","last_unstale":"2022-06-27T11:47:52.387174+0000","last_undegraded":"2022-06-27T11:47:52.387174+0000","last_fullsized":"2022-06-27T11:47:52.387174+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:21:51.780895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.369604+0000","last_change":"2022-06-27T11:47:51.973817+0000","last_active":"2022-06-27T11:47:52.369604+0000","last_peered":"2022-06-27T11:47:52.369604+0000","last_clean":"2022-06-27T11:47:52.369604+0000","last_became_active":"2022-06-27T11:47:51.973515+0000","last_became_peered":"2022-06-27T11:47:51.973515+0000","last_unstale":"2022-06-27T11:47:52.369604+0000","last_undegraded":"2022-06-27T11:47:52.369604+0000","last_fullsized":"2022-06-27T11:47:52.369604+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T21:01:45.170590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.374582+0000","last_change":"2022-06-27T11:47:52.371141+0000","last_active":"2022-06-27T11:47:52.374582+0000","last_peered":"2022-06-27T11:47:52.374582+0000","last_clean":"2022-06-27T11:47:52.374582+0000","last_became_active":"2022-06-27T11:47:52.370796+0000","last_became_peered":"2022-06-27T11:47:52.370796+0000","last_unstale":"2022-06-27T11:47:52.374582+0000","last_undegraded":"2022-06-27T11:47:52.374582+0000","last_fullsized":"2022-06-27T11:47:52.374582+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T23:16:39.266615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.369465+0000","last_change":"2022-06-27T11:47:51.983071+0000","last_active":"2022-06-27T11:47:52.369465+0000","last_peered":"2022-06-27T11:47:52.369465+0000","last_clean":"2022-06-27T11:47:52.369465+0000","last_became_active":"2022-06-27T11:47:51.982756+0000","last_became_peered":"2022-06-27T11:47:51.982756+0000","last_unstale":"2022-06-27T11:47:52.369465+0000","last_undegraded":"2022-06-27T11:47:52.369465+0000","last_fullsized":"2022-06-27T11:47:52.369465+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T17:14:00.737599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.387222+0000","last_change":"2022-06-27T11:47:52.382943+0000","last_active":"2022-06-27T11:47:52.387222+0000","last_peered":"2022-06-27T11:47:52.387222+0000","last_clean":"2022-06-27T11:47:52.387222+0000","last_became_active":"2022-06-27T11:47:52.382625+0000","last_became_peered":"2022-06-27T11:47:52.382625+0000","last_unstale":"2022-06-27T11:47:52.387222+0000","last_undegraded":"2022-06-27T11:47:52.387222+0000","last_fullsized":"2022-06-27T11:47:52.387222+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:01:18.570554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.362681+0000","last_change":"2022-06-27T11:47:51.978856+0000","last_active":"2022-06-27T11:47:52.362681+0000","last_peered":"2022-06-27T11:47:52.362681+0000","last_clean":"2022-06-27T11:47:52.362681+0000","last_became_active":"2022-06-27T11:47:51.978541+0000","last_became_peered":"2022-06-27T11:47:51.978541+0000","last_unstale":"2022-06-27T11:47:52.362681+0000","last_undegraded":"2022-06-27T11:47:52.362681+0000","last_fullsized":"2022-06-27T11:47:52.362681+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:36:24.190996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.378203+0000","last_change":"2022-06-27T11:47:52.376925+0000","last_active":"2022-06-27T11:47:52.378203+0000","last_peered":"2022-06-27T11:47:52.378203+0000","last_clean":"2022-06-27T11:47:52.378203+0000","last_became_active":"2022-06-27T11:47:52.376646+0000","last_became_peered":"2022-06-27T11:47:52.376646+0000","last_unstale":"2022-06-27T11:47:52.378203+0000","last_undegraded":"2022-06-27T11:47:52.378203+0000","last_fullsized":"2022-06-27T11:47:52.378203+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:17:51.913392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.369021+0000","last_change":"2022-06-27T11:47:51.979355+0000","last_active":"2022-06-27T11:47:52.369021+0000","last_peered":"2022-06-27T11:47:52.369021+0000","last_clean":"2022-06-27T11:47:52.369021+0000","last_became_active":"2022-06-27T11:47:51.977591+0000","last_became_peered":"2022-06-27T11:47:51.977591+0000","last_unstale":"2022-06-27T11:47:52.369021+0000","last_undegraded":"2022-06-27T11:47:52.369021+0000","last_fullsized":"2022-06-27T11:47:52.369021+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T11:50:43.086014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.385028+0000","last_change":"2022-06-27T11:47:52.375271+0000","last_active":"2022-06-27T11:47:52.385028+0000","last_peered":"2022-06-27T11:47:52.385028+0000","last_clean":"2022-06-27T11:47:52.385028+0000","last_became_active":"2022-06-27T11:47:52.374985+0000","last_became_peered":"2022-06-27T11:47:52.374985+0000","last_unstale":"2022-06-27T11:47:52.385028+0000","last_undegraded":"2022-06-27T11:47:52.385028+0000","last_fullsized":"2022-06-27T11:47:52.385028+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T17:58:40.711409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.369774+0000","last_change":"2022-06-27T11:47:51.979723+0000","last_active":"2022-06-27T11:47:52.369774+0000","last_peered":"2022-06-27T11:47:52.369774+0000","last_clean":"2022-06-27T11:47:52.369774+0000","last_became_active":"2022-06-27T11:47:51.977669+0000","last_became_peered":"2022-06-27T11:47:51.977669+0000","last_unstale":"2022-06-27T11:47:52.369774+0000","last_undegraded":"2022-06-27T11:47:52.369774+0000","last_fullsized":"2022-06-27T11:47:52.369774+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T14:35:14.770254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.363444+0000","last_change":"2022-06-27T11:47:48.388880+0000","last_active":"2022-06-27T11:47:52.363444+0000","last_peered":"2022-06-27T11:47:52.363444+0000","last_clean":"2022-06-27T11:47:52.363444+0000","last_became_active":"2022-06-27T11:47:48.388639+0000","last_became_peered":"2022-06-27T11:47:48.388639+0000","last_unstale":"2022-06-27T11:47:52.363444+0000","last_undegraded":"2022-06-27T11:47:52.363444+0000","last_fullsized":"2022-06-27T11:47:52.363444+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_clean_scrub_stamp":"2022-06-27T11:47:46.780512+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:31:10.549485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.380026+0000","last_change":"2022-06-27T11:47:52.373143+0000","last_active":"2022-06-27T11:47:52.380026+0000","last_peered":"2022-06-27T11:47:52.380026+0000","last_clean":"2022-06-27T11:47:52.380026+0000","last_became_active":"2022-06-27T11:47:52.372850+0000","last_became_peered":"2022-06-27T11:47:52.372850+0000","last_unstale":"2022-06-27T11:47:52.380026+0000","last_undegraded":"2022-06-27T11:47:52.380026+0000","last_fullsized":"2022-06-27T11:47:52.380026+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:48:44.251459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.369716+0000","last_change":"2022-06-27T11:47:47.793307+0000","last_active":"2022-06-27T11:47:52.369716+0000","last_peered":"2022-06-27T11:47:52.369716+0000","last_clean":"2022-06-27T11:47:52.369716+0000","last_became_active":"2022-06-27T11:47:47.792958+0000","last_became_peered":"2022-06-27T11:47:47.792958+0000","last_unstale":"2022-06-27T11:47:52.369716+0000","last_undegraded":"2022-06-27T11:47:52.369716+0000","last_fullsized":"2022-06-27T11:47:52.369716+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_clean_scrub_stamp":"2022-06-27T11:47:46.780512+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:08:35.128804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.372474+0000","last_change":"2022-06-27T11:47:51.976872+0000","last_active":"2022-06-27T11:47:52.372474+0000","last_peered":"2022-06-27T11:47:52.372474+0000","last_clean":"2022-06-27T11:47:52.372474+0000","last_became_active":"2022-06-27T11:47:51.976546+0000","last_became_peered":"2022-06-27T11:47:51.976546+0000","last_unstale":"2022-06-27T11:47:52.372474+0000","last_undegraded":"2022-06-27T11:47:52.372474+0000","last_fullsized":"2022-06-27T11:47:52.372474+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T14:44:43.349688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.375090+0000","last_change":"2022-06-27T11:47:52.372794+0000","last_active":"2022-06-27T11:47:52.375090+0000","last_peered":"2022-06-27T11:47:52.375090+0000","last_clean":"2022-06-27T11:47:52.375090+0000","last_became_active":"2022-06-27T11:47:52.372486+0000","last_became_peered":"2022-06-27T11:47:52.372486+0000","last_unstale":"2022-06-27T11:47:52.375090+0000","last_undegraded":"2022-06-27T11:47:52.375090+0000","last_fullsized":"2022-06-27T11:47:52.375090+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:23:56.307995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.363787+0000","last_change":"2022-06-27T11:47:47.792631+0000","last_active":"2022-06-27T11:47:52.363787+0000","last_peered":"2022-06-27T11:47:52.363787+0000","last_clean":"2022-06-27T11:47:52.363787+0000","last_became_active":"2022-06-27T11:47:47.792282+0000","last_became_peered":"2022-06-27T11:47:47.792282+0000","last_unstale":"2022-06-27T11:47:52.363787+0000","last_undegraded":"2022-06-27T11:47:52.363787+0000","last_fullsized":"2022-06-27T11:47:52.363787+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_clean_scrub_stamp":"2022-06-27T11:47:46.780512+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:41:25.449379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"17'117","reported_seq":163,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.368104+0000","last_change":"2022-06-27T11:47:47.791697+0000","last_active":"2022-06-27T11:47:52.368104+0000","last_peered":"2022-06-27T11:47:52.368104+0000","last_clean":"2022-06-27T11:47:52.368104+0000","last_became_active":"2022-06-27T11:47:47.790645+0000","last_became_peered":"2022-06-27T11:47:47.790645+0000","last_unstale":"2022-06-27T11:47:52.368104+0000","last_undegraded":"2022-06-27T11:47:52.368104+0000","last_fullsized":"2022-06-27T11:47:52.368104+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_clean_scrub_stamp":"2022-06-27T11:47:46.780512+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:11:09.344666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.367962+0000","last_change":"2022-06-27T11:47:51.982728+0000","last_active":"2022-06-27T11:47:52.367962+0000","last_peered":"2022-06-27T11:47:52.367962+0000","last_clean":"2022-06-27T11:47:52.367962+0000","last_became_active":"2022-06-27T11:47:51.982445+0000","last_became_peered":"2022-06-27T11:47:51.982445+0000","last_unstale":"2022-06-27T11:47:52.367962+0000","last_undegraded":"2022-06-27T11:47:52.367962+0000","last_fullsized":"2022-06-27T11:47:52.367962+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:42:04.682687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.386922+0000","last_change":"2022-06-27T11:47:52.386015+0000","last_active":"2022-06-27T11:47:52.386922+0000","last_peered":"2022-06-27T11:47:52.386922+0000","last_clean":"2022-06-27T11:47:52.386922+0000","last_became_active":"2022-06-27T11:47:52.385731+0000","last_became_peered":"2022-06-27T11:47:52.385731+0000","last_unstale":"2022-06-27T11:47:52.386922+0000","last_undegraded":"2022-06-27T11:47:52.386922+0000","last_fullsized":"2022-06-27T11:47:52.386922+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T11:58:50.483766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.366627+0000","last_change":"2022-06-27T11:47:47.807699+0000","last_active":"2022-06-27T11:47:52.366627+0000","last_peered":"2022-06-27T11:47:52.366627+0000","last_clean":"2022-06-27T11:47:52.366627+0000","last_became_active":"2022-06-27T11:47:47.807422+0000","last_became_peered":"2022-06-27T11:47:47.807422+0000","last_unstale":"2022-06-27T11:47:52.366627+0000","last_undegraded":"2022-06-27T11:47:52.366627+0000","last_fullsized":"2022-06-27T11:47:52.366627+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_clean_scrub_stamp":"2022-06-27T11:47:46.780512+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:15:00.549295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.363922+0000","last_change":"2022-06-27T11:47:51.974468+0000","last_active":"2022-06-27T11:47:52.363922+0000","last_peered":"2022-06-27T11:47:52.363922+0000","last_clean":"2022-06-27T11:47:52.363922+0000","last_became_active":"2022-06-27T11:47:51.974160+0000","last_became_peered":"2022-06-27T11:47:51.974160+0000","last_unstale":"2022-06-27T11:47:52.363922+0000","last_undegraded":"2022-06-27T11:47:52.363922+0000","last_fullsized":"2022-06-27T11:47:52.363922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T19:14:45.694282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.380162+0000","last_change":"2022-06-27T11:47:52.373931+0000","last_active":"2022-06-27T11:47:52.380162+0000","last_peered":"2022-06-27T11:47:52.380162+0000","last_clean":"2022-06-27T11:47:52.380162+0000","last_became_active":"2022-06-27T11:47:52.373562+0000","last_became_peered":"2022-06-27T11:47:52.373562+0000","last_unstale":"2022-06-27T11:47:52.380162+0000","last_undegraded":"2022-06-27T11:47:52.380162+0000","last_fullsized":"2022-06-27T11:47:52.380162+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T14:09:22.859955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.367866+0000","last_change":"2022-06-27T11:47:47.807650+0000","last_active":"2022-06-27T11:47:52.367866+0000","last_peered":"2022-06-27T11:47:52.367866+0000","last_clean":"2022-06-27T11:47:52.367866+0000","last_became_active":"2022-06-27T11:47:47.807375+0000","last_became_peered":"2022-06-27T11:47:47.807375+0000","last_unstale":"2022-06-27T11:47:52.367866+0000","last_undegraded":"2022-06-27T11:47:52.367866+0000","last_fullsized":"2022-06-27T11:47:52.367866+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_clean_scrub_stamp":"2022-06-27T11:47:46.780512+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T14:10:48.067284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.368898+0000","last_change":"2022-06-27T11:47:51.972154+0000","last_active":"2022-06-27T11:47:52.368898+0000","last_peered":"2022-06-27T11:47:52.368898+0000","last_clean":"2022-06-27T11:47:52.368898+0000","last_became_active":"2022-06-27T11:47:51.971762+0000","last_became_peered":"2022-06-27T11:47:51.971762+0000","last_unstale":"2022-06-27T11:47:52.368898+0000","last_undegraded":"2022-06-27T11:47:52.368898+0000","last_fullsized":"2022-06-27T11:47:52.368898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T18:15:14.271005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.379879+0000","last_change":"2022-06-27T11:47:52.378294+0000","last_active":"2022-06-27T11:47:52.379879+0000","last_peered":"2022-06-27T11:47:52.379879+0000","last_clean":"2022-06-27T11:47:52.379879+0000","last_became_active":"2022-06-27T11:47:52.377704+0000","last_became_peered":"2022-06-27T11:47:52.377704+0000","last_unstale":"2022-06-27T11:47:52.379879+0000","last_undegraded":"2022-06-27T11:47:52.379879+0000","last_fullsized":"2022-06-27T11:47:52.379879+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:26:34.474081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.364180+0000","last_change":"2022-06-27T11:47:47.807600+0000","last_active":"2022-06-27T11:47:52.364180+0000","last_peered":"2022-06-27T11:47:52.364180+0000","last_clean":"2022-06-27T11:47:52.364180+0000","last_became_active":"2022-06-27T11:47:47.807284+0000","last_became_peered":"2022-06-27T11:47:47.807284+0000","last_unstale":"2022-06-27T11:47:52.364180+0000","last_undegraded":"2022-06-27T11:47:52.364180+0000","last_fullsized":"2022-06-27T11:47:52.364180+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_clean_scrub_stamp":"2022-06-27T11:47:46.780512+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T20:57:06.487027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.362687+0000","last_change":"2022-06-27T11:47:51.980734+0000","last_active":"2022-06-27T11:47:52.362687+0000","last_peered":"2022-06-27T11:47:52.362687+0000","last_clean":"2022-06-27T11:47:52.362687+0000","last_became_active":"2022-06-27T11:47:51.980476+0000","last_became_peered":"2022-06-27T11:47:51.980476+0000","last_unstale":"2022-06-27T11:47:52.362687+0000","last_undegraded":"2022-06-27T11:47:52.362687+0000","last_fullsized":"2022-06-27T11:47:52.362687+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T21:32:16.435472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.385478+0000","last_change":"2022-06-27T11:47:52.379902+0000","last_active":"2022-06-27T11:47:52.385478+0000","last_peered":"2022-06-27T11:47:52.385478+0000","last_clean":"2022-06-27T11:47:52.385478+0000","last_became_active":"2022-06-27T11:47:52.379359+0000","last_became_peered":"2022-06-27T11:47:52.379359+0000","last_unstale":"2022-06-27T11:47:52.385478+0000","last_undegraded":"2022-06-27T11:47:52.385478+0000","last_fullsized":"2022-06-27T11:47:52.385478+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:41:27.884083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.370459+0000","last_change":"2022-06-27T11:47:47.807607+0000","last_active":"2022-06-27T11:47:52.370459+0000","last_peered":"2022-06-27T11:47:52.370459+0000","last_clean":"2022-06-27T11:47:52.370459+0000","last_became_active":"2022-06-27T11:47:47.807293+0000","last_became_peered":"2022-06-27T11:47:47.807293+0000","last_unstale":"2022-06-27T11:47:52.370459+0000","last_undegraded":"2022-06-27T11:47:52.370459+0000","last_fullsized":"2022-06-27T11:47:52.370459+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_clean_scrub_stamp":"2022-06-27T11:47:46.780512+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:00:35.318372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.364133+0000","last_change":"2022-06-27T11:47:51.976055+0000","last_active":"2022-06-27T11:47:52.364133+0000","last_peered":"2022-06-27T11:47:52.364133+0000","last_clean":"2022-06-27T11:47:52.364133+0000","last_became_active":"2022-06-27T11:47:51.975303+0000","last_became_peered":"2022-06-27T11:47:51.975303+0000","last_unstale":"2022-06-27T11:47:52.364133+0000","last_undegraded":"2022-06-27T11:47:52.364133+0000","last_fullsized":"2022-06-27T11:47:52.364133+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:27:32.125525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384750+0000","last_change":"2022-06-27T11:47:52.380775+0000","last_active":"2022-06-27T11:47:52.384750+0000","last_peered":"2022-06-27T11:47:52.384750+0000","last_clean":"2022-06-27T11:47:52.384750+0000","last_became_active":"2022-06-27T11:47:52.380439+0000","last_became_peered":"2022-06-27T11:47:52.380439+0000","last_unstale":"2022-06-27T11:47:52.384750+0000","last_undegraded":"2022-06-27T11:47:52.384750+0000","last_fullsized":"2022-06-27T11:47:52.384750+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:03:44.939329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.370743+0000","last_change":"2022-06-27T11:47:47.792314+0000","last_active":"2022-06-27T11:47:52.370743+0000","last_peered":"2022-06-27T11:47:52.370743+0000","last_clean":"2022-06-27T11:47:52.370743+0000","last_became_active":"2022-06-27T11:47:47.792032+0000","last_became_peered":"2022-06-27T11:47:47.792032+0000","last_unstale":"2022-06-27T11:47:52.370743+0000","last_undegraded":"2022-06-27T11:47:52.370743+0000","last_fullsized":"2022-06-27T11:47:52.370743+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_clean_scrub_stamp":"2022-06-27T11:47:46.780512+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T14:13:03.591775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.369208+0000","last_change":"2022-06-27T11:47:51.978639+0000","last_active":"2022-06-27T11:47:52.369208+0000","last_peered":"2022-06-27T11:47:52.369208+0000","last_clean":"2022-06-27T11:47:52.369208+0000","last_became_active":"2022-06-27T11:47:51.978341+0000","last_became_peered":"2022-06-27T11:47:51.978341+0000","last_unstale":"2022-06-27T11:47:52.369208+0000","last_undegraded":"2022-06-27T11:47:52.369208+0000","last_fullsized":"2022-06-27T11:47:52.369208+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T21:34:48.070558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.381133+0000","last_change":"2022-06-27T11:47:52.373588+0000","last_active":"2022-06-27T11:47:52.381133+0000","last_peered":"2022-06-27T11:47:52.381133+0000","last_clean":"2022-06-27T11:47:52.381133+0000","last_became_active":"2022-06-27T11:47:52.373208+0000","last_became_peered":"2022-06-27T11:47:52.373208+0000","last_unstale":"2022-06-27T11:47:52.381133+0000","last_undegraded":"2022-06-27T11:47:52.381133+0000","last_fullsized":"2022-06-27T11:47:52.381133+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T20:27:28.252147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.374017+0000","last_change":"2022-06-27T11:47:51.982051+0000","last_active":"2022-06-27T11:47:52.374017+0000","last_peered":"2022-06-27T11:47:52.374017+0000","last_clean":"2022-06-27T11:47:52.374017+0000","last_became_active":"2022-06-27T11:47:51.981742+0000","last_became_peered":"2022-06-27T11:47:51.981742+0000","last_unstale":"2022-06-27T11:47:52.374017+0000","last_undegraded":"2022-06-27T11:47:52.374017+0000","last_fullsized":"2022-06-27T11:47:52.374017+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:39:56.937086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.380333+0000","last_change":"2022-06-27T11:47:52.378192+0000","last_active":"2022-06-27T11:47:52.380333+0000","last_peered":"2022-06-27T11:47:52.380333+0000","last_clean":"2022-06-27T11:47:52.380333+0000","last_became_active":"2022-06-27T11:47:52.377919+0000","last_became_peered":"2022-06-27T11:47:52.377919+0000","last_unstale":"2022-06-27T11:47:52.380333+0000","last_undegraded":"2022-06-27T11:47:52.380333+0000","last_fullsized":"2022-06-27T11:47:52.380333+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T18:51:35.757778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.364197+0000","last_change":"2022-06-27T11:47:51.981101+0000","last_active":"2022-06-27T11:47:52.364197+0000","last_peered":"2022-06-27T11:47:52.364197+0000","last_clean":"2022-06-27T11:47:52.364197+0000","last_became_active":"2022-06-27T11:47:51.980793+0000","last_became_peered":"2022-06-27T11:47:51.980793+0000","last_unstale":"2022-06-27T11:47:52.364197+0000","last_undegraded":"2022-06-27T11:47:52.364197+0000","last_fullsized":"2022-06-27T11:47:52.364197+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:28:10.206148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.372181+0000","last_change":"2022-06-27T11:47:52.370334+0000","last_active":"2022-06-27T11:47:52.372181+0000","last_peered":"2022-06-27T11:47:52.372181+0000","last_clean":"2022-06-27T11:47:52.372181+0000","last_became_active":"2022-06-27T11:47:52.370028+0000","last_became_peered":"2022-06-27T11:47:52.370028+0000","last_unstale":"2022-06-27T11:47:52.372181+0000","last_undegraded":"2022-06-27T11:47:52.372181+0000","last_fullsized":"2022-06-27T11:47:52.372181+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T17:28:12.357757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.369303+0000","last_change":"2022-06-27T11:47:51.981186+0000","last_active":"2022-06-27T11:47:52.369303+0000","last_peered":"2022-06-27T11:47:52.369303+0000","last_clean":"2022-06-27T11:47:52.369303+0000","last_became_active":"2022-06-27T11:47:51.980921+0000","last_became_peered":"2022-06-27T11:47:51.980921+0000","last_unstale":"2022-06-27T11:47:52.369303+0000","last_undegraded":"2022-06-27T11:47:52.369303+0000","last_fullsized":"2022-06-27T11:47:52.369303+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:04:40.732299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.387201+0000","last_change":"2022-06-27T11:47:52.382091+0000","last_active":"2022-06-27T11:47:52.387201+0000","last_peered":"2022-06-27T11:47:52.387201+0000","last_clean":"2022-06-27T11:47:52.387201+0000","last_became_active":"2022-06-27T11:47:52.381633+0000","last_became_peered":"2022-06-27T11:47:52.381633+0000","last_unstale":"2022-06-27T11:47:52.387201+0000","last_undegraded":"2022-06-27T11:47:52.387201+0000","last_fullsized":"2022-06-27T11:47:52.387201+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:34:44.030534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.369654+0000","last_change":"2022-06-27T11:47:51.972963+0000","last_active":"2022-06-27T11:47:52.369654+0000","last_peered":"2022-06-27T11:47:52.369654+0000","last_clean":"2022-06-27T11:47:52.369654+0000","last_became_active":"2022-06-27T11:47:51.972686+0000","last_became_peered":"2022-06-27T11:47:51.972686+0000","last_unstale":"2022-06-27T11:47:52.369654+0000","last_undegraded":"2022-06-27T11:47:52.369654+0000","last_fullsized":"2022-06-27T11:47:52.369654+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T12:11:38.008538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.385365+0000","last_change":"2022-06-27T11:47:52.383414+0000","last_active":"2022-06-27T11:47:52.385365+0000","last_peered":"2022-06-27T11:47:52.385365+0000","last_clean":"2022-06-27T11:47:52.385365+0000","last_became_active":"2022-06-27T11:47:52.383070+0000","last_became_peered":"2022-06-27T11:47:52.383070+0000","last_unstale":"2022-06-27T11:47:52.385365+0000","last_undegraded":"2022-06-27T11:47:52.385365+0000","last_fullsized":"2022-06-27T11:47:52.385365+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:11:28.410045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.371392+0000","last_change":"2022-06-27T11:47:51.971664+0000","last_active":"2022-06-27T11:47:52.371392+0000","last_peered":"2022-06-27T11:47:52.371392+0000","last_clean":"2022-06-27T11:47:52.371392+0000","last_became_active":"2022-06-27T11:47:51.971385+0000","last_became_peered":"2022-06-27T11:47:51.971385+0000","last_unstale":"2022-06-27T11:47:52.371392+0000","last_undegraded":"2022-06-27T11:47:52.371392+0000","last_fullsized":"2022-06-27T11:47:52.371392+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T18:44:38.394388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.379723+0000","last_change":"2022-06-27T11:47:52.377305+0000","last_active":"2022-06-27T11:47:52.379723+0000","last_peered":"2022-06-27T11:47:52.379723+0000","last_clean":"2022-06-27T11:47:52.379723+0000","last_became_active":"2022-06-27T11:47:52.377012+0000","last_became_peered":"2022-06-27T11:47:52.377012+0000","last_unstale":"2022-06-27T11:47:52.379723+0000","last_undegraded":"2022-06-27T11:47:52.379723+0000","last_fullsized":"2022-06-27T11:47:52.379723+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:15:14.505656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.377592+0000","last_change":"2022-06-27T11:47:52.376074+0000","last_active":"2022-06-27T11:47:52.377592+0000","last_peered":"2022-06-27T11:47:52.377592+0000","last_clean":"2022-06-27T11:47:52.377592+0000","last_became_active":"2022-06-27T11:47:52.375836+0000","last_became_peered":"2022-06-27T11:47:52.375836+0000","last_unstale":"2022-06-27T11:47:52.377592+0000","last_undegraded":"2022-06-27T11:47:52.377592+0000","last_fullsized":"2022-06-27T11:47:52.377592+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T18:46:26.900833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384440+0000","last_change":"2022-06-27T11:47:52.376668+0000","last_active":"2022-06-27T11:47:52.384440+0000","last_peered":"2022-06-27T11:47:52.384440+0000","last_clean":"2022-06-27T11:47:52.384440+0000","last_became_active":"2022-06-27T11:47:52.376386+0000","last_became_peered":"2022-06-27T11:47:52.376386+0000","last_unstale":"2022-06-27T11:47:52.384440+0000","last_undegraded":"2022-06-27T11:47:52.384440+0000","last_fullsized":"2022-06-27T11:47:52.384440+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T11:48:35.874748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.379744+0000","last_change":"2022-06-27T11:47:52.377793+0000","last_active":"2022-06-27T11:47:52.379744+0000","last_peered":"2022-06-27T11:47:52.379744+0000","last_clean":"2022-06-27T11:47:52.379744+0000","last_became_active":"2022-06-27T11:47:52.377318+0000","last_became_peered":"2022-06-27T11:47:52.377318+0000","last_unstale":"2022-06-27T11:47:52.379744+0000","last_undegraded":"2022-06-27T11:47:52.379744+0000","last_fullsized":"2022-06-27T11:47:52.379744+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T17:54:22.511301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384925+0000","last_change":"2022-06-27T11:47:52.374638+0000","last_active":"2022-06-27T11:47:52.384925+0000","last_peered":"2022-06-27T11:47:52.384925+0000","last_clean":"2022-06-27T11:47:52.384925+0000","last_became_active":"2022-06-27T11:47:52.374352+0000","last_became_peered":"2022-06-27T11:47:52.374352+0000","last_unstale":"2022-06-27T11:47:52.384925+0000","last_undegraded":"2022-06-27T11:47:52.384925+0000","last_fullsized":"2022-06-27T11:47:52.384925+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:09:59.369369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.375769+0000","last_change":"2022-06-27T11:47:52.373288+0000","last_active":"2022-06-27T11:47:52.375769+0000","last_peered":"2022-06-27T11:47:52.375769+0000","last_clean":"2022-06-27T11:47:52.375769+0000","last_became_active":"2022-06-27T11:47:52.372891+0000","last_became_peered":"2022-06-27T11:47:52.372891+0000","last_unstale":"2022-06-27T11:47:52.375769+0000","last_undegraded":"2022-06-27T11:47:52.375769+0000","last_fullsized":"2022-06-27T11:47:52.375769+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T23:16:08.088450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64756,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574852,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8988,"kb_used_data":276,"kb_used_omap":0,"kb_used_meta":8704,"kb_avail":94362852,"statfs":{"total":96636764160,"available":96627560448,"internally_reserved":0,"allocated":282624,"data_stored":669207,"data_compressed":32378,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":8912896},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45000000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30599999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38800000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34000000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574852,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8408,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":8192,"kb_avail":94363432,"statfs":{"total":96636764160,"available":96628154368,"internally_reserved":0,"allocated":212992,"data_stored":77836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8388608},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61599999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54000000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.497}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28999999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33400000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.374}]}]},{"osd":1,"up_from":13,"seq":55834574852,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8996,"kb_used_data":276,"kb_used_omap":0,"kb_used_meta":8704,"kb_avail":94362844,"statfs":{"total":96636764160,"available":96627552256,"internally_reserved":0,"allocated":282624,"data_stored":669207,"data_compressed":32378,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":8912896},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34499999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42799999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.442}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52500000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51900000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574852,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8464,"kb_used_data":200,"kb_used_omap":0,"kb_used_meta":8256,"kb_avail":94363376,"statfs":{"total":96636764160,"available":96628097024,"internally_reserved":0,"allocated":204800,"data_stored":72198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8454144},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34300000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29299999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28299999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51300000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48699999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52100000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8412,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":8192,"kb_avail":94363428,"statfs":{"total":96636764160,"available":96628150272,"internally_reserved":0,"allocated":217088,"data_stored":78839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8388608},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.438}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52400000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70199999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53700000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60099999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8472,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":8256,"kb_avail":94363368,"statfs":{"total":96636764160,"available":96628088832,"internally_reserved":0,"allocated":212992,"data_stored":77836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8454144},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33000000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.376}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59999999999999998}]}]},{"osd":4,"up_from":13,"seq":55834574852,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8476,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":8256,"kb_avail":94363364,"statfs":{"total":96636764160,"available":96628084736,"internally_reserved":0,"allocated":217088,"data_stored":78839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8454144},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39700000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45200000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45800000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48899999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8412,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":8192,"kb_avail":94363428,"statfs":{"total":96636764160,"available":96628150272,"internally_reserved":0,"allocated":217088,"data_stored":78839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8388608},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56499999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.629}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68700000000000006}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32300000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34799999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.441}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32378,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32378,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-06-27T11:47:57.791 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2022-06-27T11:47:57.792 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2022-06-27T11:47:57.792 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2022-06-27T11:47:57.793 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2022-06-27T11:47:57.793 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2022-06-27T11:47:57.945 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.945+0000 7f558f908700 1 Processor -- start 2022-06-27T11:47:57.946 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.945+0000 7f558f908700 1 -- start start 2022-06-27T11:47:57.958 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.946+0000 7f558f908700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f558814e490 0x7f558806cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:57.958 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.946+0000 7f558f908700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f558814e860 0x7f55880c3840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:57.959 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.946+0000 7f558f908700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55880c33b0 con 0x7f55880c3d80 2022-06-27T11:47:57.960 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.946+0000 7f558f908700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f558806d520 con 0x7f558814e860 2022-06-27T11:47:57.960 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.946+0000 7f558f908700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f558806f220 con 0x7f558814e490 2022-06-27T11:47:57.961 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.946+0000 7f558cea3700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f558814e860 0x7f55880c3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:57.962 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.946+0000 7f558dea5700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f55880c3d80 0x7f558806f620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:47466/0 (socket says 172.21.15.195:47466) 2022-06-27T11:47:57.962 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.946+0000 7f558dea5700 1 -- 172.21.15.195:0/3177076272 learned_addr learned my addr 172.21.15.195:0/3177076272 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:47:57.962 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.946+0000 7f558d6a4700 1 --2- 172.21.15.195:0/3177076272 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f558814e490 0x7f558806cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:57.963 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.947+0000 7f558cea3700 1 -- 172.21.15.195:0/3177076272 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f558814e490 msgr2=0x7f558806cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:47:57.964 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.947+0000 7f558cea3700 1 --2- 172.21.15.195:0/3177076272 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f558814e490 0x7f558806cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:57.964 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.947+0000 7f558cea3700 1 -- 172.21.15.195:0/3177076272 >> v1:172.21.15.195:6789/0 conn(0x7f55880c3d80 legacy=0x7f558806f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:57.965 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.947+0000 7f558cea3700 1 -- 172.21.15.195:0/3177076272 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55881542c0 con 0x7f558814e860 2022-06-27T11:47:57.965 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.947+0000 7f5577fff700 1 -- 172.21.15.195:0/3177076272 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 51920707 0 0) 0x7f55880c33b0 con 0x7f55880c3d80 2022-06-27T11:47:57.966 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.947+0000 7f558cea3700 1 --2- 172.21.15.195:0/3177076272 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f558814e860 0x7f55880c3840 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f55840097a0 tx=0x7f5584009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ee57e7f3e12e53c server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:57.966 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.947+0000 7f5577fff700 1 -- 172.21.15.195:0/3177076272 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5584011630 con 0x7f558814e860 2022-06-27T11:47:57.967 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.947+0000 7f5577fff700 1 -- 172.21.15.195:0/3177076272 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5584011790 con 0x7f558814e860 2022-06-27T11:47:57.967 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.947+0000 7f5577fff700 1 -- 172.21.15.195:0/3177076272 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5584011b30 con 0x7f558814e860 2022-06-27T11:47:57.968 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.948+0000 7f558f908700 1 -- 172.21.15.195:0/3177076272 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f558814e860 msgr2=0x7f55880c3840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:57.968 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.948+0000 7f558f908700 1 --2- 172.21.15.195:0/3177076272 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f558814e860 0x7f55880c3840 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f55840097a0 tx=0x7f5584009060 comp rx=0 tx=0).stop 2022-06-27T11:47:57.969 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.948+0000 7f558f908700 1 -- 172.21.15.195:0/3177076272 shutdown_connections 2022-06-27T11:47:57.969 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.948+0000 7f558f908700 1 --2- 172.21.15.195:0/3177076272 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f558814e490 0x7f558806cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:57.969 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.948+0000 7f558f908700 1 --2- 172.21.15.195:0/3177076272 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f558814e860 0x7f55880c3840 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:57.970 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.948+0000 7f558f908700 1 -- 172.21.15.195:0/3177076272 >> 172.21.15.195:0/3177076272 conn(0x7f5588063af0 msgr2=0x7f5588063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:57.970 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.948+0000 7f558f908700 1 -- 172.21.15.195:0/3177076272 shutdown_connections 2022-06-27T11:47:57.970 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.948+0000 7f558f908700 1 -- 172.21.15.195:0/3177076272 wait complete. 2022-06-27T11:47:57.970 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.949+0000 7f558f908700 1 Processor -- start 2022-06-27T11:47:57.971 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.949+0000 7f558f908700 1 -- start start 2022-06-27T11:47:57.971 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.950+0000 7f558f908700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f55880c3d80 0x7f5588156c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:57.971 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.950+0000 7f558f908700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f558814e490 0x7f5588166dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:57.971 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.950+0000 7f558f908700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f558814e860 0x7f5588167310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:57.972 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.950+0000 7f558f908700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5588157eb0 con 0x7f558814e860 2022-06-27T11:47:57.972 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.950+0000 7f558f908700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5588167850 con 0x7f558814e490 2022-06-27T11:47:57.972 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.950+0000 7f558f908700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55881567d0 con 0x7f55880c3d80 2022-06-27T11:47:57.972 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.950+0000 7f558dea5700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f558814e860 0x7f5588167310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:57.973 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.950+0000 7f558dea5700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f558814e860 0x7f5588167310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41284/0 (socket says 172.21.15.195:41284) 2022-06-27T11:47:57.973 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.950+0000 7f558dea5700 1 -- 172.21.15.195:0/3359266833 learned_addr learned my addr 172.21.15.195:0/3359266833 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:47:57.973 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.950+0000 7f558d6a4700 1 --2- 172.21.15.195:0/3359266833 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f55880c3d80 0x7f5588156c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:57.973 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.950+0000 7f558cea3700 1 --2- 172.21.15.195:0/3359266833 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f558814e490 0x7f5588166dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:57.974 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.950+0000 7f558dea5700 1 -- 172.21.15.195:0/3359266833 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f55880c3d80 msgr2=0x7f5588156c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:57.974 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.950+0000 7f558dea5700 1 --2- 172.21.15.195:0/3359266833 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f55880c3d80 0x7f5588156c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:57.974 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.950+0000 7f558dea5700 1 -- 172.21.15.195:0/3359266833 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f558814e490 msgr2=0x7f5588166dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:57.974 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.950+0000 7f558dea5700 1 --2- 172.21.15.195:0/3359266833 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f558814e490 0x7f5588166dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:57.975 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.950+0000 7f558dea5700 1 -- 172.21.15.195:0/3359266833 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55840092e0 con 0x7f558814e860 2022-06-27T11:47:57.975 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.950+0000 7f558dea5700 1 --2- 172.21.15.195:0/3359266833 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f558814e860 0x7f5588167310 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f558000c870 tx=0x7f5580001540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:57.975 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.950+0000 7f55757fa700 1 -- 172.21.15.195:0/3359266833 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5580004c80 con 0x7f558814e860 2022-06-27T11:47:57.975 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.950+0000 7f55757fa700 1 -- 172.21.15.195:0/3359266833 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5580004e00 con 0x7f558814e860 2022-06-27T11:47:57.976 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.950+0000 7f558f908700 1 -- 172.21.15.195:0/3359266833 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55880c7ed0 con 0x7f558814e860 2022-06-27T11:47:57.976 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.950+0000 7f55757fa700 1 -- 172.21.15.195:0/3359266833 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55800073a0 con 0x7f558814e860 2022-06-27T11:47:57.976 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.950+0000 7f558f908700 1 -- 172.21.15.195:0/3359266833 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5588156180 con 0x7f558814e860 2022-06-27T11:47:57.976 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.951+0000 7f558f908700 1 -- 172.21.15.195:0/3359266833 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55880722c0 con 0x7f558814e860 2022-06-27T11:47:57.976 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.952+0000 7f55757fa700 1 -- 172.21.15.195:0/3359266833 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f558001a020 con 0x7f558814e860 2022-06-27T11:47:57.977 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.952+0000 7f55757fa700 1 --2- 172.21.15.195:0/3359266833 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5568063d30 0x7f55680661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:57.977 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.952+0000 7f55757fa700 1 -- 172.21.15.195:0/3359266833 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f55800804c0 con 0x7f558814e860 2022-06-27T11:47:57.977 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.952+0000 7f558d6a4700 1 --2- 172.21.15.195:0/3359266833 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5568063d30 0x7f55680661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:57.977 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.952+0000 7f558d6a4700 1 --2- 172.21.15.195:0/3359266833 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5568063d30 0x7f55680661e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5578009c70 tx=0x7f5578009060 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:57.978 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:57.956+0000 7f55757fa700 1 -- 172.21.15.195:0/3359266833 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f558004f0b0 con 0x7f558814e860 2022-06-27T11:47:58.185 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.186+0000 7f558f908700 1 -- 172.21.15.195:0/3359266833 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f5588156910 con 0x7f558814e860 2022-06-27T11:47:58.188 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.188+0000 7f55757fa700 1 -- 172.21.15.195:0/3359266833 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+158612 (secure 0 0 0) 0x7f55800526d0 con 0x7f558814e860 2022-06-27T11:47:58.190 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:47:58.192 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.191+0000 7f558f908700 1 -- 172.21.15.195:0/3359266833 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5568063d30 msgr2=0x7f55680661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:58.192 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.191+0000 7f558f908700 1 --2- 172.21.15.195:0/3359266833 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5568063d30 0x7f55680661e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5578009c70 tx=0x7f5578009060 comp rx=0 tx=0).stop 2022-06-27T11:47:58.192 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.191+0000 7f558f908700 1 -- 172.21.15.195:0/3359266833 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f558814e860 msgr2=0x7f5588167310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:58.193 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.191+0000 7f558f908700 1 --2- 172.21.15.195:0/3359266833 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f558814e860 0x7f5588167310 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f558000c870 tx=0x7f5580001540 comp rx=0 tx=0).stop 2022-06-27T11:47:58.193 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.191+0000 7f558f908700 1 -- 172.21.15.195:0/3359266833 shutdown_connections 2022-06-27T11:47:58.193 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.191+0000 7f558f908700 1 --2- 172.21.15.195:0/3359266833 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5568063d30 0x7f55680661e0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:58.193 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.191+0000 7f558f908700 1 --2- 172.21.15.195:0/3359266833 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f558814e860 0x7f5588167310 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:58.194 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.191+0000 7f558f908700 1 --2- 172.21.15.195:0/3359266833 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f55880c3d80 0x7f5588156c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:58.194 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.191+0000 7f558f908700 1 --2- 172.21.15.195:0/3359266833 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f558814e490 0x7f5588166dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:58.194 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.191+0000 7f558f908700 1 -- 172.21.15.195:0/3359266833 >> 172.21.15.195:0/3359266833 conn(0x7f5588063af0 msgr2=0x7f55881529a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:58.194 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.191+0000 7f558f908700 1 -- 172.21.15.195:0/3359266833 shutdown_connections 2022-06-27T11:47:58.195 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.191+0000 7f558f908700 1 -- 172.21.15.195:0/3359266833 wait complete. 2022-06-27T11:47:58.203 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":5,"active_gid":4098,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":93955},{"type":"v1","addr":"172.21.15.195:6833","nonce":93955}]},"active_addr":"172.21.15.195:6833/93955","active_change":"2022-06-27T11:47:40.332795+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":4100,"name":"x","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.195:0","nonce":1730574404}]},{"addrvec":[{"type":"v2","addr":"172.21.15.195:0","nonce":2546867354}]},{"addrvec":[{"type":"v2","addr":"172.21.15.195:0","nonce":4262115391}]},{"addrvec":[{"type":"v2","addr":"172.21.15.195:0","nonce":900367976}]}]}} 2022-06-27T11:47:58.207 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2022-06-27T11:47:58.207 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2022-06-27T11:47:58.208 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-27T11:47:58.328 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.328+0000 7f79731d5700 1 Processor -- start 2022-06-27T11:47:58.328 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.328+0000 7f79731d5700 1 -- start start 2022-06-27T11:47:58.335 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.329+0000 7f79731d5700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f796c06e850 0x7f796c1570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:58.336 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.329+0000 7f79731d5700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f796c0c34a0 0x7f796c0c2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:58.336 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.329+0000 7f79731d5700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f796c06fba0 con 0x7f796c06fef0 2022-06-27T11:47:58.336 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.329+0000 7f79731d5700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f796c1575e0 con 0x7f796c0c34a0 2022-06-27T11:47:58.336 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.329+0000 7f79731d5700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f796c06f5e0 con 0x7f796c06e850 2022-06-27T11:47:58.337 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.329+0000 7f7971772700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f796c0c34a0 0x7f796c0c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:58.337 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.329+0000 7f7971772700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f796c0c34a0 0x7f796c0c2920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41292/0 (socket says 172.21.15.195:41292) 2022-06-27T11:47:58.337 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.329+0000 7f7971772700 1 -- 172.21.15.195:0/2744987206 learned_addr learned my addr 172.21.15.195:0/2744987206 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:47:58.337 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.329+0000 7f7970f71700 1 --2- 172.21.15.195:0/2744987206 >> v2:172.21.15.202:3300/0 conn(0x7f796c06e850 0x7f796c1570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:58.338 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.329+0000 7f7971772700 1 -- 172.21.15.195:0/2744987206 >> v2:172.21.15.202:3300/0 conn(0x7f796c06e850 msgr2=0x7f796c1570a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:47:58.338 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.329+0000 7f7971772700 1 --2- 172.21.15.195:0/2744987206 >> v2:172.21.15.202:3300/0 conn(0x7f796c06e850 0x7f796c1570a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:58.338 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.329+0000 7f7971772700 1 -- 172.21.15.195:0/2744987206 >> v1:172.21.15.202:6789/0 conn(0x7f796c06fef0 legacy=0x7f796c069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:58.338 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.329+0000 7f7971772700 1 -- 172.21.15.195:0/2744987206 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f796c06f8c0 con 0x7f796c0c34a0 2022-06-27T11:47:58.339 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.330+0000 7f7971772700 1 --2- 172.21.15.195:0/2744987206 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f796c0c34a0 0x7f796c0c2920 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f796c06e120 tx=0x7f796400ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73286b9f78c378f9 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:58.339 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.330+0000 7f796b7fe700 1 -- 172.21.15.195:0/2744987206 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7964010070 con 0x7f796c0c34a0 2022-06-27T11:47:58.339 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.330+0000 7f796b7fe700 1 -- 172.21.15.195:0/2744987206 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f796400de70 con 0x7f796c0c34a0 2022-06-27T11:47:58.339 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.330+0000 7f796b7fe700 1 -- 172.21.15.195:0/2744987206 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f796400b300 con 0x7f796c0c34a0 2022-06-27T11:47:58.340 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.330+0000 7f79731d5700 1 -- 172.21.15.195:0/2744987206 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f796c0c34a0 msgr2=0x7f796c0c2920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:58.340 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.330+0000 7f79731d5700 1 --2- 172.21.15.195:0/2744987206 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f796c0c34a0 0x7f796c0c2920 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f796c06e120 tx=0x7f796400ba50 comp rx=0 tx=0).stop 2022-06-27T11:47:58.340 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.330+0000 7f79731d5700 1 -- 172.21.15.195:0/2744987206 shutdown_connections 2022-06-27T11:47:58.340 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.330+0000 7f79731d5700 1 --2- 172.21.15.195:0/2744987206 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f796c0c34a0 0x7f796c0c2920 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:58.341 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.330+0000 7f79731d5700 1 --2- 172.21.15.195:0/2744987206 >> v2:172.21.15.202:3300/0 conn(0x7f796c06e850 0x7f796c1570a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:58.341 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.330+0000 7f79731d5700 1 -- 172.21.15.195:0/2744987206 >> 172.21.15.195:0/2744987206 conn(0x7f796c0c6350 msgr2=0x7f796c0c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:58.341 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.330+0000 7f79731d5700 1 -- 172.21.15.195:0/2744987206 shutdown_connections 2022-06-27T11:47:58.341 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.330+0000 7f79731d5700 1 -- 172.21.15.195:0/2744987206 wait complete. 2022-06-27T11:47:58.342 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.331+0000 7f79731d5700 1 Processor -- start 2022-06-27T11:47:58.342 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.332+0000 7f79731d5700 1 -- start start 2022-06-27T11:47:58.342 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.332+0000 7f79731d5700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f796c06e850 0x7f796c0c8170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:58.342 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.332+0000 7f79731d5700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f796c0c34a0 0x7f796c160530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:58.343 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.332+0000 7f79731d5700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f796c0b9970 0x7f796c160a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:58.343 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.333+0000 7f79731d5700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f796c161d20 con 0x7f796c06e850 2022-06-27T11:47:58.343 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.333+0000 7f79731d5700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f796c1600a0 con 0x7f796c0c34a0 2022-06-27T11:47:58.343 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.333+0000 7f79731d5700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f796c153500 con 0x7f796c0b9970 2022-06-27T11:47:58.343 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.333+0000 7f7970f71700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f796c06e850 0x7f796c0c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:58.344 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.333+0000 7f7970f71700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f796c06e850 0x7f796c0c8170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41294/0 (socket says 172.21.15.195:41294) 2022-06-27T11:47:58.344 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.333+0000 7f7970f71700 1 -- 172.21.15.195:0/1406217019 learned_addr learned my addr 172.21.15.195:0/1406217019 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:47:58.344 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.333+0000 7f796bfff700 1 --2- 172.21.15.195:0/1406217019 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f796c0c34a0 0x7f796c160530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:58.344 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.333+0000 7f7971772700 1 --2- 172.21.15.195:0/1406217019 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f796c0b9970 0x7f796c160a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:58.345 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.333+0000 7f7970f71700 1 -- 172.21.15.195:0/1406217019 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f796c0b9970 msgr2=0x7f796c160a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:58.345 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.333+0000 7f7970f71700 1 --2- 172.21.15.195:0/1406217019 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f796c0b9970 0x7f796c160a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:58.345 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.333+0000 7f7970f71700 1 -- 172.21.15.195:0/1406217019 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f796c0c34a0 msgr2=0x7f796c160530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:58.345 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.333+0000 7f7970f71700 1 --2- 172.21.15.195:0/1406217019 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f796c0c34a0 0x7f796c160530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:58.346 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.333+0000 7f7970f71700 1 -- 172.21.15.195:0/1406217019 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f796400bcf0 con 0x7f796c06e850 2022-06-27T11:47:58.346 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.333+0000 7f7970f71700 1 --2- 172.21.15.195:0/1406217019 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f796c06e850 0x7f796c0c8170 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f796000e930 tx=0x7f7960009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:58.346 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.333+0000 7f7968ff9700 1 -- 172.21.15.195:0/1406217019 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7960013680 con 0x7f796c06e850 2022-06-27T11:47:58.346 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.333+0000 7f7968ff9700 1 -- 172.21.15.195:0/1406217019 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79600137e0 con 0x7f796c06e850 2022-06-27T11:47:58.347 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.333+0000 7f79731d5700 1 -- 172.21.15.195:0/1406217019 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f796c068970 con 0x7f796c06e850 2022-06-27T11:47:58.347 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.333+0000 7f7968ff9700 1 -- 172.21.15.195:0/1406217019 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7960013b80 con 0x7f796c06e850 2022-06-27T11:47:58.347 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.333+0000 7f79731d5700 1 -- 172.21.15.195:0/1406217019 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f796c15fa50 con 0x7f796c06e850 2022-06-27T11:47:58.347 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.334+0000 7f79731d5700 1 -- 172.21.15.195:0/1406217019 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f796c0722c0 con 0x7f796c06e850 2022-06-27T11:47:58.348 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.335+0000 7f7968ff9700 1 -- 172.21.15.195:0/1406217019 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f7960019030 con 0x7f796c06e850 2022-06-27T11:47:58.348 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.335+0000 7f7968ff9700 1 --2- 172.21.15.195:0/1406217019 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7954063d00 0x7f79540661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:58.348 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.335+0000 7f7968ff9700 1 -- 172.21.15.195:0/1406217019 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f7960080760 con 0x7f796c06e850 2022-06-27T11:47:58.348 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.335+0000 7f796bfff700 1 --2- 172.21.15.195:0/1406217019 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7954063d00 0x7f79540661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:58.348 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.335+0000 7f796bfff700 1 --2- 172.21.15.195:0/1406217019 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7954063d00 0x7f79540661b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f796c069b30 tx=0x7f7958009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:58.349 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.340+0000 7f7968ff9700 1 -- 172.21.15.195:0/1406217019 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f796004f2f0 con 0x7f796c06e850 2022-06-27T11:47:58.521 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.521+0000 7f79731d5700 1 -- 172.21.15.195:0/1406217019 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f796c06f240 con 0x7f796c06e850 2022-06-27T11:47:58.522 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.522+0000 7f7968ff9700 1 -- 172.21.15.195:0/1406217019 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16914 (secure 0 0 0) 0x7f7960052910 con 0x7f796c06e850 2022-06-27T11:47:58.522 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:47:58.523 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":21,"fsid":"81b3a1d3-6a2b-47e8-8d69-7027cfeae7af","created":"2022-06-27T11:47:35.094715+0000","modified":"2022-06-27T11:47:53.359270+0000","last_up_change":"2022-06-27T11:47:44.777459+0000","last_in_change":"2022-06-27T11:47:37.596818+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2022-06-27T11:47:46.368325+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2022-06-27T11:47:46.386132+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-06-27T11:47:50.901270+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-06-27T11:47:51.314719+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c64bc51f-d4f3-4bc2-806b-da0dc78d51b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6813","nonce":94217},{"type":"v1","addr":"172.21.15.195:6815","nonce":94217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6817","nonce":94217},{"type":"v1","addr":"172.21.15.195:6819","nonce":94217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":94217},{"type":"v1","addr":"172.21.15.195:6823","nonce":94217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6820","nonce":94217},{"type":"v1","addr":"172.21.15.195:6821","nonce":94217}]},"public_addr":"172.21.15.195:6815/94217","cluster_addr":"172.21.15.195:6819/94217","heartbeat_back_addr":"172.21.15.195:6823/94217","heartbeat_front_addr":"172.21.15.195:6821/94217","state":["exists","up"]},{"osd":1,"uuid":"bea71cdf-aa98-493f-9207-66d886a3e30e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":94223},{"type":"v1","addr":"172.21.15.195:6809","nonce":94223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":94223},{"type":"v1","addr":"172.21.15.195:6811","nonce":94223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6816","nonce":94223},{"type":"v1","addr":"172.21.15.195:6818","nonce":94223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":94223},{"type":"v1","addr":"172.21.15.195:6814","nonce":94223}]},"public_addr":"172.21.15.195:6809/94223","cluster_addr":"172.21.15.195:6811/94223","heartbeat_back_addr":"172.21.15.195:6818/94223","heartbeat_front_addr":"172.21.15.195:6814/94223","state":["exists","up"]},{"osd":2,"uuid":"bf6c87a0-a8ea-497a-9ce1-721285888065","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":94221},{"type":"v1","addr":"172.21.15.195:6801","nonce":94221}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":94221},{"type":"v1","addr":"172.21.15.195:6803","nonce":94221}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":94221},{"type":"v1","addr":"172.21.15.195:6807","nonce":94221}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":94221},{"type":"v1","addr":"172.21.15.195:6805","nonce":94221}]},"public_addr":"172.21.15.195:6801/94221","cluster_addr":"172.21.15.195:6803/94221","heartbeat_back_addr":"172.21.15.195:6807/94221","heartbeat_front_addr":"172.21.15.195:6805/94221","state":["exists","up"]},{"osd":3,"uuid":"e8aa47ce-af8c-47e7-9654-870cd53229f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":94232},{"type":"v1","addr":"172.21.15.195:6825","nonce":94232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":94232},{"type":"v1","addr":"172.21.15.195:6827","nonce":94232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":94232},{"type":"v1","addr":"172.21.15.195:6831","nonce":94232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":94232},{"type":"v1","addr":"172.21.15.195:6829","nonce":94232}]},"public_addr":"172.21.15.195:6825/94232","cluster_addr":"172.21.15.195:6827/94232","heartbeat_back_addr":"172.21.15.195:6831/94232","heartbeat_front_addr":"172.21.15.195:6829/94232","state":["exists","up"]},{"osd":4,"uuid":"99ffa553-f149-443a-98f5-68b10f77e085","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6800","nonce":94308},{"type":"v1","addr":"172.21.15.202:6801","nonce":94308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6802","nonce":94308},{"type":"v1","addr":"172.21.15.202:6803","nonce":94308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6806","nonce":94308},{"type":"v1","addr":"172.21.15.202:6807","nonce":94308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6804","nonce":94308},{"type":"v1","addr":"172.21.15.202:6805","nonce":94308}]},"public_addr":"172.21.15.202:6801/94308","cluster_addr":"172.21.15.202:6803/94308","heartbeat_back_addr":"172.21.15.202:6807/94308","heartbeat_front_addr":"172.21.15.202:6805/94308","state":["exists","up"]},{"osd":5,"uuid":"35409ef0-891b-4c01-82e4-26b8e64f502d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6824","nonce":94325},{"type":"v1","addr":"172.21.15.202:6825","nonce":94325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6826","nonce":94325},{"type":"v1","addr":"172.21.15.202:6827","nonce":94325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6830","nonce":94325},{"type":"v1","addr":"172.21.15.202:6831","nonce":94325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6828","nonce":94325},{"type":"v1","addr":"172.21.15.202:6829","nonce":94325}]},"public_addr":"172.21.15.202:6825/94325","cluster_addr":"172.21.15.202:6827/94325","heartbeat_back_addr":"172.21.15.202:6831/94325","heartbeat_front_addr":"172.21.15.202:6829/94325","state":["exists","up"]},{"osd":6,"uuid":"009124c4-8b5f-47d4-bd20-2c19271a2adc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6809","nonce":94309},{"type":"v1","addr":"172.21.15.202:6811","nonce":94309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6813","nonce":94309},{"type":"v1","addr":"172.21.15.202:6815","nonce":94309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6821","nonce":94309},{"type":"v1","addr":"172.21.15.202:6823","nonce":94309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6817","nonce":94309},{"type":"v1","addr":"172.21.15.202:6819","nonce":94309}]},"public_addr":"172.21.15.202:6811/94309","cluster_addr":"172.21.15.202:6815/94309","heartbeat_back_addr":"172.21.15.202:6823/94309","heartbeat_front_addr":"172.21.15.202:6819/94309","state":["exists","up"]},{"osd":7,"uuid":"7773115e-d3c3-4a0c-8801-d8ce63e0e5a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6808","nonce":94310},{"type":"v1","addr":"172.21.15.202:6810","nonce":94310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6812","nonce":94310},{"type":"v1","addr":"172.21.15.202:6814","nonce":94310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6820","nonce":94310},{"type":"v1","addr":"172.21.15.202:6822","nonce":94310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6816","nonce":94310},{"type":"v1","addr":"172.21.15.202:6818","nonce":94310}]},"public_addr":"172.21.15.202:6810/94310","cluster_addr":"172.21.15.202:6814/94310","heartbeat_back_addr":"172.21.15.202:6822/94310","heartbeat_front_addr":"172.21.15.202:6818/94310","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.776182+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.806922+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.237997+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.784371+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.723615+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.820332+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:43.841414+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.758071+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-06-27T11:47:58.525 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.524+0000 7f79731d5700 1 -- 172.21.15.195:0/1406217019 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7954063d00 msgr2=0x7f79540661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:58.526 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.524+0000 7f79731d5700 1 --2- 172.21.15.195:0/1406217019 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7954063d00 0x7f79540661b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f796c069b30 tx=0x7f7958009040 comp rx=0 tx=0).stop 2022-06-27T11:47:58.526 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.524+0000 7f79731d5700 1 -- 172.21.15.195:0/1406217019 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f796c06e850 msgr2=0x7f796c0c8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:58.526 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.524+0000 7f79731d5700 1 --2- 172.21.15.195:0/1406217019 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f796c06e850 0x7f796c0c8170 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f796000e930 tx=0x7f7960009060 comp rx=0 tx=0).stop 2022-06-27T11:47:58.527 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.524+0000 7f79731d5700 1 -- 172.21.15.195:0/1406217019 shutdown_connections 2022-06-27T11:47:58.528 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.524+0000 7f79731d5700 1 --2- 172.21.15.195:0/1406217019 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7954063d00 0x7f79540661b0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:58.529 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.524+0000 7f79731d5700 1 --2- 172.21.15.195:0/1406217019 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f796c06e850 0x7f796c0c8170 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:58.529 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.524+0000 7f79731d5700 1 --2- 172.21.15.195:0/1406217019 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f796c0b9970 0x7f796c160a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:58.529 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.524+0000 7f79731d5700 1 --2- 172.21.15.195:0/1406217019 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f796c0c34a0 0x7f796c160530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:58.530 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.524+0000 7f79731d5700 1 -- 172.21.15.195:0/1406217019 >> 172.21.15.195:0/1406217019 conn(0x7f796c0c6350 msgr2=0x7f796c1505e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:58.530 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.524+0000 7f79731d5700 1 -- 172.21.15.195:0/1406217019 shutdown_connections 2022-06-27T11:47:58.531 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.524+0000 7f79731d5700 1 -- 172.21.15.195:0/1406217019 wait complete. 2022-06-27T11:47:58.537 INFO:tasks.ceph.ceph_manager.ceph:all up! 2022-06-27T11:47:58.539 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-27T11:47:58.658 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.659+0000 7ff0f69e1700 1 Processor -- start 2022-06-27T11:47:58.659 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.659+0000 7ff0f69e1700 1 -- start start 2022-06-27T11:47:58.671 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.660+0000 7ff0f69e1700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7ff0f00c4160 0x7ff0f0156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:58.671 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.660+0000 7ff0f69e1700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7ff0f00c39e0 0x7ff0f01535a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:58.671 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.660+0000 7ff0f69e1700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0f00c1a20 con 0x7ff0f006f640 2022-06-27T11:47:58.672 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.660+0000 7ff0f69e1700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0f00c1130 con 0x7ff0f00c39e0 2022-06-27T11:47:58.672 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.660+0000 7ff0f69e1700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0f006ea90 con 0x7ff0f00c4160 2022-06-27T11:47:58.672 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.660+0000 7ff0f4f7e700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7ff0f00c39e0 0x7ff0f01535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:58.672 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.660+0000 7ff0effff700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7ff0f006f640 0x7ff0f00c1eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:47490/0 (socket says 172.21.15.195:47490) 2022-06-27T11:47:58.673 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.660+0000 7ff0effff700 1 -- 172.21.15.195:0/1195463976 learned_addr learned my addr 172.21.15.195:0/1195463976 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:47:58.673 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.660+0000 7ff0ef7fe700 1 --2- 172.21.15.195:0/1195463976 >> v2:172.21.15.202:3300/0 conn(0x7ff0f00c4160 0x7ff0f0156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:58.673 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.660+0000 7ff0f4f7e700 1 -- 172.21.15.195:0/1195463976 >> v2:172.21.15.202:3300/0 conn(0x7ff0f00c4160 msgr2=0x7ff0f0156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:47:58.674 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.660+0000 7ff0f4f7e700 1 --2- 172.21.15.195:0/1195463976 >> v2:172.21.15.202:3300/0 conn(0x7ff0f00c4160 0x7ff0f0156ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:58.674 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.660+0000 7ff0f4f7e700 1 -- 172.21.15.195:0/1195463976 >> v1:172.21.15.195:6789/0 conn(0x7ff0f006f640 legacy=0x7ff0f00c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:58.675 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.660+0000 7ff0f4f7e700 1 -- 172.21.15.195:0/1195463976 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0f006ed70 con 0x7ff0f00c39e0 2022-06-27T11:47:58.675 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.660+0000 7ff0eeffd700 1 -- 172.21.15.195:0/1195463976 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4141460402 0 0) 0x7ff0f00c1a20 con 0x7ff0f006f640 2022-06-27T11:47:58.675 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.660+0000 7ff0f4f7e700 1 --2- 172.21.15.195:0/1195463976 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff0f00c39e0 0x7ff0f01535a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ff0f006dd70 tx=0x7ff0e8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2498dc7d4d045bf8 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:58.675 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.660+0000 7ff0eeffd700 1 -- 172.21.15.195:0/1195463976 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e800e070 con 0x7ff0f00c39e0 2022-06-27T11:47:58.676 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.661+0000 7ff0eeffd700 1 -- 172.21.15.195:0/1195463976 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0e800bbf0 con 0x7ff0f00c39e0 2022-06-27T11:47:58.676 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.661+0000 7ff0eeffd700 1 -- 172.21.15.195:0/1195463976 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e8012430 con 0x7ff0f00c39e0 2022-06-27T11:47:58.676 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.661+0000 7ff0f69e1700 1 -- 172.21.15.195:0/1195463976 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff0f00c39e0 msgr2=0x7ff0f01535a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:58.676 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.661+0000 7ff0f69e1700 1 --2- 172.21.15.195:0/1195463976 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff0f00c39e0 0x7ff0f01535a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ff0f006dd70 tx=0x7ff0e8009780 comp rx=0 tx=0).stop 2022-06-27T11:47:58.677 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.661+0000 7ff0f69e1700 1 -- 172.21.15.195:0/1195463976 shutdown_connections 2022-06-27T11:47:58.677 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.661+0000 7ff0f69e1700 1 --2- 172.21.15.195:0/1195463976 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff0f00c39e0 0x7ff0f01535a0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:58.678 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.661+0000 7ff0f69e1700 1 --2- 172.21.15.195:0/1195463976 >> v2:172.21.15.202:3300/0 conn(0x7ff0f00c4160 0x7ff0f0156ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:58.679 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.661+0000 7ff0f69e1700 1 -- 172.21.15.195:0/1195463976 >> 172.21.15.195:0/1195463976 conn(0x7ff0f00c6350 msgr2=0x7ff0f00c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:58.679 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.661+0000 7ff0f69e1700 1 -- 172.21.15.195:0/1195463976 shutdown_connections 2022-06-27T11:47:58.680 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.661+0000 7ff0f69e1700 1 -- 172.21.15.195:0/1195463976 wait complete. 2022-06-27T11:47:58.681 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.662+0000 7ff0f69e1700 1 Processor -- start 2022-06-27T11:47:58.681 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.662+0000 7ff0f69e1700 1 -- start start 2022-06-27T11:47:58.682 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.663+0000 7ff0f69e1700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff0f006f640 0x7ff0f014ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:58.684 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.663+0000 7ff0f69e1700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff0f00c39e0 0x7ff0f0167250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:58.684 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.663+0000 7ff0f69e1700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff0f00c4160 0x7ff0f0154300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:58.684 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.663+0000 7ff0f69e1700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f00c9770 con 0x7ff0f00c39e0 2022-06-27T11:47:58.685 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.663+0000 7ff0f69e1700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f014f0b0 con 0x7ff0f00c4160 2022-06-27T11:47:58.685 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.663+0000 7ff0f69e1700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f00c8ed0 con 0x7ff0f006f640 2022-06-27T11:47:58.685 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.663+0000 7ff0ef7fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff0f00c39e0 0x7ff0f0167250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:58.685 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.663+0000 7ff0ef7fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff0f00c39e0 0x7ff0f0167250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41310/0 (socket says 172.21.15.195:41310) 2022-06-27T11:47:58.685 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.663+0000 7ff0ef7fe700 1 -- 172.21.15.195:0/910437802 learned_addr learned my addr 172.21.15.195:0/910437802 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:47:58.686 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.663+0000 7ff0f4f7e700 1 --2- 172.21.15.195:0/910437802 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff0f00c4160 0x7ff0f0154300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:58.686 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.663+0000 7ff0effff700 1 --2- 172.21.15.195:0/910437802 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff0f006f640 0x7ff0f014ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:58.686 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.663+0000 7ff0ef7fe700 1 -- 172.21.15.195:0/910437802 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff0f006f640 msgr2=0x7ff0f014ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:58.687 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.663+0000 7ff0ef7fe700 1 --2- 172.21.15.195:0/910437802 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff0f006f640 0x7ff0f014ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:58.687 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.663+0000 7ff0ef7fe700 1 -- 172.21.15.195:0/910437802 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff0f00c4160 msgr2=0x7ff0f0154300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:58.687 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.663+0000 7ff0ef7fe700 1 --2- 172.21.15.195:0/910437802 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff0f00c4160 0x7ff0f0154300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:58.687 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.663+0000 7ff0ef7fe700 1 -- 172.21.15.195:0/910437802 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0e8009a20 con 0x7ff0f00c39e0 2022-06-27T11:47:58.688 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.664+0000 7ff0ef7fe700 1 --2- 172.21.15.195:0/910437802 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff0f00c39e0 0x7ff0f0167250 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff0e000e900 tx=0x7ff0e0009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:58.688 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.664+0000 7ff0d7fff700 1 -- 172.21.15.195:0/910437802 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e0013680 con 0x7ff0f00c39e0 2022-06-27T11:47:58.688 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.664+0000 7ff0d7fff700 1 -- 172.21.15.195:0/910437802 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0e00137e0 con 0x7ff0f00c39e0 2022-06-27T11:47:58.688 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.664+0000 7ff0f69e1700 1 -- 172.21.15.195:0/910437802 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0f00c91b0 con 0x7ff0f00c39e0 2022-06-27T11:47:58.689 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.664+0000 7ff0d7fff700 1 -- 172.21.15.195:0/910437802 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e0013b80 con 0x7ff0f00c39e0 2022-06-27T11:47:58.689 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.664+0000 7ff0f69e1700 1 -- 172.21.15.195:0/910437802 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0f014d490 con 0x7ff0f00c39e0 2022-06-27T11:47:58.689 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.665+0000 7ff0f69e1700 1 -- 172.21.15.195:0/910437802 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0f014d2f0 con 0x7ff0f00c39e0 2022-06-27T11:47:58.689 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.665+0000 7ff0d7fff700 1 -- 172.21.15.195:0/910437802 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7ff0e0019030 con 0x7ff0f00c39e0 2022-06-27T11:47:58.690 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.665+0000 7ff0d7fff700 1 --2- 172.21.15.195:0/910437802 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff0d0063d00 0x7ff0d00661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:58.691 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.665+0000 7ff0d7fff700 1 -- 172.21.15.195:0/910437802 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7ff0e0080730 con 0x7ff0f00c39e0 2022-06-27T11:47:58.691 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.665+0000 7ff0effff700 1 --2- 172.21.15.195:0/910437802 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff0d0063d00 0x7ff0d00661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:58.692 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.666+0000 7ff0effff700 1 --2- 172.21.15.195:0/910437802 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff0d0063d00 0x7ff0d00661b0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff0e4004810 tx=0x7ff0e4007bf0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:58.692 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.670+0000 7ff0d7fff700 1 -- 172.21.15.195:0/910437802 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7ff0f014d2f0 con 0x7ff0f00c39e0 2022-06-27T11:47:58.846 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.847+0000 7ff0f69e1700 1 -- 172.21.15.195:0/910437802 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff0f014d2f0 con 0x7ff0f00c39e0 2022-06-27T11:47:58.848 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.848+0000 7ff0d7fff700 1 -- 172.21.15.195:0/910437802 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16914 (secure 0 0 0) 0x7ff0f014d2f0 con 0x7ff0f00c39e0 2022-06-27T11:47:58.849 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:47:58.849 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":21,"fsid":"81b3a1d3-6a2b-47e8-8d69-7027cfeae7af","created":"2022-06-27T11:47:35.094715+0000","modified":"2022-06-27T11:47:53.359270+0000","last_up_change":"2022-06-27T11:47:44.777459+0000","last_in_change":"2022-06-27T11:47:37.596818+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2022-06-27T11:47:46.368325+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2022-06-27T11:47:46.386132+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-06-27T11:47:50.901270+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-06-27T11:47:51.314719+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c64bc51f-d4f3-4bc2-806b-da0dc78d51b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6813","nonce":94217},{"type":"v1","addr":"172.21.15.195:6815","nonce":94217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6817","nonce":94217},{"type":"v1","addr":"172.21.15.195:6819","nonce":94217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":94217},{"type":"v1","addr":"172.21.15.195:6823","nonce":94217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6820","nonce":94217},{"type":"v1","addr":"172.21.15.195:6821","nonce":94217}]},"public_addr":"172.21.15.195:6815/94217","cluster_addr":"172.21.15.195:6819/94217","heartbeat_back_addr":"172.21.15.195:6823/94217","heartbeat_front_addr":"172.21.15.195:6821/94217","state":["exists","up"]},{"osd":1,"uuid":"bea71cdf-aa98-493f-9207-66d886a3e30e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":94223},{"type":"v1","addr":"172.21.15.195:6809","nonce":94223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":94223},{"type":"v1","addr":"172.21.15.195:6811","nonce":94223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6816","nonce":94223},{"type":"v1","addr":"172.21.15.195:6818","nonce":94223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":94223},{"type":"v1","addr":"172.21.15.195:6814","nonce":94223}]},"public_addr":"172.21.15.195:6809/94223","cluster_addr":"172.21.15.195:6811/94223","heartbeat_back_addr":"172.21.15.195:6818/94223","heartbeat_front_addr":"172.21.15.195:6814/94223","state":["exists","up"]},{"osd":2,"uuid":"bf6c87a0-a8ea-497a-9ce1-721285888065","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":94221},{"type":"v1","addr":"172.21.15.195:6801","nonce":94221}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":94221},{"type":"v1","addr":"172.21.15.195:6803","nonce":94221}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":94221},{"type":"v1","addr":"172.21.15.195:6807","nonce":94221}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":94221},{"type":"v1","addr":"172.21.15.195:6805","nonce":94221}]},"public_addr":"172.21.15.195:6801/94221","cluster_addr":"172.21.15.195:6803/94221","heartbeat_back_addr":"172.21.15.195:6807/94221","heartbeat_front_addr":"172.21.15.195:6805/94221","state":["exists","up"]},{"osd":3,"uuid":"e8aa47ce-af8c-47e7-9654-870cd53229f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":94232},{"type":"v1","addr":"172.21.15.195:6825","nonce":94232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":94232},{"type":"v1","addr":"172.21.15.195:6827","nonce":94232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":94232},{"type":"v1","addr":"172.21.15.195:6831","nonce":94232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":94232},{"type":"v1","addr":"172.21.15.195:6829","nonce":94232}]},"public_addr":"172.21.15.195:6825/94232","cluster_addr":"172.21.15.195:6827/94232","heartbeat_back_addr":"172.21.15.195:6831/94232","heartbeat_front_addr":"172.21.15.195:6829/94232","state":["exists","up"]},{"osd":4,"uuid":"99ffa553-f149-443a-98f5-68b10f77e085","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6800","nonce":94308},{"type":"v1","addr":"172.21.15.202:6801","nonce":94308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6802","nonce":94308},{"type":"v1","addr":"172.21.15.202:6803","nonce":94308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6806","nonce":94308},{"type":"v1","addr":"172.21.15.202:6807","nonce":94308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6804","nonce":94308},{"type":"v1","addr":"172.21.15.202:6805","nonce":94308}]},"public_addr":"172.21.15.202:6801/94308","cluster_addr":"172.21.15.202:6803/94308","heartbeat_back_addr":"172.21.15.202:6807/94308","heartbeat_front_addr":"172.21.15.202:6805/94308","state":["exists","up"]},{"osd":5,"uuid":"35409ef0-891b-4c01-82e4-26b8e64f502d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6824","nonce":94325},{"type":"v1","addr":"172.21.15.202:6825","nonce":94325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6826","nonce":94325},{"type":"v1","addr":"172.21.15.202:6827","nonce":94325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6830","nonce":94325},{"type":"v1","addr":"172.21.15.202:6831","nonce":94325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6828","nonce":94325},{"type":"v1","addr":"172.21.15.202:6829","nonce":94325}]},"public_addr":"172.21.15.202:6825/94325","cluster_addr":"172.21.15.202:6827/94325","heartbeat_back_addr":"172.21.15.202:6831/94325","heartbeat_front_addr":"172.21.15.202:6829/94325","state":["exists","up"]},{"osd":6,"uuid":"009124c4-8b5f-47d4-bd20-2c19271a2adc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6809","nonce":94309},{"type":"v1","addr":"172.21.15.202:6811","nonce":94309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6813","nonce":94309},{"type":"v1","addr":"172.21.15.202:6815","nonce":94309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6821","nonce":94309},{"type":"v1","addr":"172.21.15.202:6823","nonce":94309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6817","nonce":94309},{"type":"v1","addr":"172.21.15.202:6819","nonce":94309}]},"public_addr":"172.21.15.202:6811/94309","cluster_addr":"172.21.15.202:6815/94309","heartbeat_back_addr":"172.21.15.202:6823/94309","heartbeat_front_addr":"172.21.15.202:6819/94309","state":["exists","up"]},{"osd":7,"uuid":"7773115e-d3c3-4a0c-8801-d8ce63e0e5a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6808","nonce":94310},{"type":"v1","addr":"172.21.15.202:6810","nonce":94310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6812","nonce":94310},{"type":"v1","addr":"172.21.15.202:6814","nonce":94310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6820","nonce":94310},{"type":"v1","addr":"172.21.15.202:6822","nonce":94310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6816","nonce":94310},{"type":"v1","addr":"172.21.15.202:6818","nonce":94310}]},"public_addr":"172.21.15.202:6810/94310","cluster_addr":"172.21.15.202:6814/94310","heartbeat_back_addr":"172.21.15.202:6822/94310","heartbeat_front_addr":"172.21.15.202:6818/94310","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.776182+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.806922+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.237997+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.784371+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.723615+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.820332+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:43.841414+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.758071+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-06-27T11:47:58.851 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.849+0000 7ff0f69e1700 1 -- 172.21.15.195:0/910437802 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff0d0063d00 msgr2=0x7ff0d00661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:58.851 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.849+0000 7ff0f69e1700 1 --2- 172.21.15.195:0/910437802 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff0d0063d00 0x7ff0d00661b0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff0e4004810 tx=0x7ff0e4007bf0 comp rx=0 tx=0).stop 2022-06-27T11:47:58.851 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.849+0000 7ff0f69e1700 1 -- 172.21.15.195:0/910437802 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff0f00c39e0 msgr2=0x7ff0f0167250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:58.851 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.849+0000 7ff0f69e1700 1 --2- 172.21.15.195:0/910437802 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff0f00c39e0 0x7ff0f0167250 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff0e000e900 tx=0x7ff0e0009060 comp rx=0 tx=0).stop 2022-06-27T11:47:58.852 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.849+0000 7ff0f69e1700 1 -- 172.21.15.195:0/910437802 shutdown_connections 2022-06-27T11:47:58.852 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.849+0000 7ff0f69e1700 1 --2- 172.21.15.195:0/910437802 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff0d0063d00 0x7ff0d00661b0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:58.852 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.849+0000 7ff0f69e1700 1 --2- 172.21.15.195:0/910437802 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff0f00c4160 0x7ff0f0154300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:58.853 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.849+0000 7ff0f69e1700 1 --2- 172.21.15.195:0/910437802 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff0f006f640 0x7ff0f014ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:58.853 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.850+0000 7ff0f69e1700 1 --2- 172.21.15.195:0/910437802 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff0f00c39e0 0x7ff0f0167250 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:58.853 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.850+0000 7ff0f69e1700 1 -- 172.21.15.195:0/910437802 >> 172.21.15.195:0/910437802 conn(0x7ff0f00c6350 msgr2=0x7ff0f006c7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:58.853 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.850+0000 7ff0f69e1700 1 -- 172.21.15.195:0/910437802 shutdown_connections 2022-06-27T11:47:58.854 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:58.850+0000 7ff0f69e1700 1 -- 172.21.15.195:0/910437802 wait complete. 2022-06-27T11:47:58.863 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2022-06-27T11:47:58.864 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2022-06-27T11:47:58.864 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2022-06-27T11:47:58.865 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2022-06-27T11:47:58.866 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2022-06-27T11:47:58.867 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2022-06-27T11:47:58.867 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2022-06-27T11:47:58.869 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2022-06-27T11:47:59.069 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.070+0000 7fcda7d57700 1 Processor -- start 2022-06-27T11:47:59.071 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.070+0000 7fcda7d57700 1 -- start start 2022-06-27T11:47:59.071 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.072+0000 7fcda7d57700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcda0157010 0x7fcda0067ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.072 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.072+0000 7fcda7d57700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcda0156b80 con 0x7fcda00c40d0 2022-06-27T11:47:59.072 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.072+0000 7fcda7d57700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcda006a040 con 0x7fcda00c6190 2022-06-27T11:47:59.072 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.072+0000 7fcda7d57700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcda006a300 con 0x7fcda0157010 2022-06-27T11:47:59.072 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.072+0000 7fcda5af3700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fcda00c40d0 0x7fcda00c44a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:47504/0 (socket says 172.21.15.195:47504) 2022-06-27T11:47:59.073 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.072+0000 7fcda5af3700 1 -- 172.21.15.195:0/2626437273 learned_addr learned my addr 172.21.15.195:0/2626437273 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:47:59.075 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.074+0000 7fcda4af1700 1 -- 172.21.15.195:0/2626437273 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3120627266 0 0) 0x7fcda0156b80 con 0x7fcda00c40d0 2022-06-27T11:47:59.075 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.074+0000 7fcda4af1700 1 -- 172.21.15.195:0/2626437273 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd88003430 con 0x7fcda00c40d0 2022-06-27T11:47:59.075 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.074+0000 7fcda4af1700 1 -- 172.21.15.195:0/2626437273 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1562212400 0 0) 0x7fcd88003430 con 0x7fcda00c40d0 2022-06-27T11:47:59.075 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.074+0000 7fcda4af1700 1 -- 172.21.15.195:0/2626437273 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcda0157010 msgr2=0x7fcda0067ed0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2022-06-27T11:47:59.076 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.074+0000 7fcda4af1700 1 --2- 172.21.15.195:0/2626437273 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcda0157010 0x7fcda0067ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.076 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.074+0000 7fcda4af1700 1 -- 172.21.15.195:0/2626437273 >> v1:172.21.15.202:6789/0 conn(0x7fcda00c6190 legacy=0x7fcda014cb80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2022-06-27T11:47:59.076 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.074+0000 7fcda4af1700 1 -- 172.21.15.195:0/2626437273 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcda006a5e0 con 0x7fcda00c40d0 2022-06-27T11:47:59.076 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.074+0000 7fcda4af1700 1 -- 172.21.15.195:0/2626437273 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7fcd980017f0 con 0x7fcda00c40d0 2022-06-27T11:47:59.077 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.074+0000 7fcda4af1700 1 -- 172.21.15.195:0/2626437273 >> v1:172.21.15.195:6789/0 conn(0x7fcda00c40d0 legacy=0x7fcda00c44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.077 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.074+0000 7fcda4af1700 1 --2- 172.21.15.195:0/2626437273 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcd880039d0 0x7fcd88003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.077 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.074+0000 7fcda4af1700 1 --2- 172.21.15.195:0/2626437273 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcda0157010 0x7fcd880046d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.077 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.074+0000 7fcda4af1700 1 --2- 172.21.15.195:0/2626437273 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcd88004e10 0x7fcd8800b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.078 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.074+0000 7fcda4af1700 1 -- 172.21.15.195:0/2626437273 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcda006a300 con 0x7fcd880039d0 2022-06-27T11:47:59.078 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.074+0000 7fcda4af1700 1 -- 172.21.15.195:0/2626437273 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd88005460 con 0x7fcd88004e10 2022-06-27T11:47:59.078 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.074+0000 7fcda4af1700 1 -- 172.21.15.195:0/2626437273 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd88014670 con 0x7fcda0157010 2022-06-27T11:47:59.078 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.074+0000 7fcda52f2700 1 --2- 172.21.15.195:0/2626437273 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcda0157010 0x7fcd880046d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.079 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.074+0000 7fcda62f4700 1 --2- 172.21.15.195:0/2626437273 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcd88004e10 0x7fcd8800b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.079 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.075+0000 7fcda52f2700 1 -- 172.21.15.195:0/2626437273 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcd88004e10 msgr2=0x7fcd8800b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:47:59.079 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.075+0000 7fcda52f2700 1 --2- 172.21.15.195:0/2626437273 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcd88004e10 0x7fcd8800b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.079 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.075+0000 7fcda52f2700 1 -- 172.21.15.195:0/2626437273 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcd880039d0 msgr2=0x7fcd88003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:47:59.079 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.075+0000 7fcda52f2700 1 --2- 172.21.15.195:0/2626437273 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcd880039d0 0x7fcd88003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.080 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.075+0000 7fcda52f2700 1 -- 172.21.15.195:0/2626437273 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcda006a5e0 con 0x7fcda0157010 2022-06-27T11:47:59.080 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.075+0000 7fcda52f2700 1 --2- 172.21.15.195:0/2626437273 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcda0157010 0x7fcd880046d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fcd900097e0 tx=0x7fcd90006db0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0ba8f116d25bff8 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:59.081 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.076+0000 7fcda4af1700 1 -- 172.21.15.195:0/2626437273 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd90011440 con 0x7fcda0157010 2022-06-27T11:47:59.081 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.076+0000 7fcda4af1700 1 -- 172.21.15.195:0/2626437273 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd900115c0 con 0x7fcda0157010 2022-06-27T11:47:59.081 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.076+0000 7fcda4af1700 1 -- 172.21.15.195:0/2626437273 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd90011980 con 0x7fcda0157010 2022-06-27T11:47:59.082 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.077+0000 7f249c93b700 1 Processor -- start 2022-06-27T11:47:59.082 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.077+0000 7f249c93b700 1 -- start start 2022-06-27T11:47:59.083 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.077+0000 7fcda7d57700 1 -- 172.21.15.195:0/2626437273 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcda0157010 msgr2=0x7fcd880046d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.084 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.077+0000 7fcda7d57700 1 --2- 172.21.15.195:0/2626437273 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcda0157010 0x7fcd880046d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fcd900097e0 tx=0x7fcd90006db0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.084 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.078+0000 7fcda62f4700 1 -- 172.21.15.195:0/2626437273 reap_dead start 2022-06-27T11:47:59.084 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.078+0000 7fcda7d57700 1 -- 172.21.15.195:0/2626437273 shutdown_connections 2022-06-27T11:47:59.085 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.078+0000 7fcda7d57700 1 -- 172.21.15.195:0/2626437273 >> 172.21.15.195:0/2626437273 conn(0x7fcda00ccb80 msgr2=0x7fcda00ccf80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:59.085 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.078+0000 7fcda7d57700 1 -- 172.21.15.195:0/2626437273 shutdown_connections 2022-06-27T11:47:59.085 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.078+0000 7fcda7d57700 1 -- 172.21.15.195:0/2626437273 wait complete. 2022-06-27T11:47:59.085 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.079+0000 7f249c93b700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f24980c3d70 0x7f24980c2610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.086 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.079+0000 7f249c93b700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2498153630 0x7f2498153ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.086 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.079+0000 7f249c93b700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24980c2b50 con 0x7f24980c4140 2022-06-27T11:47:59.086 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.079+0000 7f249c93b700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24980c37e0 con 0x7f24980c3d70 2022-06-27T11:47:59.087 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.079+0000 7f249c93b700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2498154010 con 0x7f2498153630 2022-06-27T11:47:59.087 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.079+0000 7f2495d9b700 1 --1- >> v1:172.21.15.202:6789/0 conn(0x7f24980c4140 0x7f24980c3010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.202:6789/0 says I am v1:172.21.15.195:49378/0 (socket says 172.21.15.195:49378) 2022-06-27T11:47:59.087 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.079+0000 7f2495d9b700 1 -- 172.21.15.195:0/2217793621 learned_addr learned my addr 172.21.15.195:0/2217793621 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:47:59.087 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.079+0000 7f249659c700 1 --2- 172.21.15.195:0/2217793621 >> v2:172.21.15.195:3300/0 conn(0x7f24980c3d70 0x7f24980c2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.087 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.080+0000 7f249559a700 1 -- 172.21.15.195:0/2217793621 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1482407939 0 0) 0x7f24980c2b50 con 0x7f24980c4140 2022-06-27T11:47:59.088 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.080+0000 7f249559a700 1 -- 172.21.15.195:0/2217793621 --> v1:172.21.15.202:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f247c003430 con 0x7f24980c4140 2022-06-27T11:47:59.088 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.080+0000 7f249659c700 1 -- 172.21.15.195:0/2217793621 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2498153630 msgr2=0x7f2498153ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2022-06-27T11:47:59.088 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.080+0000 7f249659c700 1 --2- 172.21.15.195:0/2217793621 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2498153630 0x7f2498153ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.088 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.080+0000 7f249659c700 1 -- 172.21.15.195:0/2217793621 >> v1:172.21.15.202:6789/0 conn(0x7f24980c4140 legacy=0x7f24980c3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.089 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.080+0000 7f249659c700 1 -- 172.21.15.195:0/2217793621 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24980c4d90 con 0x7f24980c3d70 2022-06-27T11:47:59.089 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.079+0000 7fcda7d57700 1 Processor -- start 2022-06-27T11:47:59.089 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.080+0000 7f249659c700 1 --2- 172.21.15.195:0/2217793621 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f24980c3d70 0x7f24980c2610 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f248c0097a0 tx=0x7f248c009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c22e9758f8eda266 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:59.090 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.079+0000 7f561442f700 1 Processor -- start 2022-06-27T11:47:59.090 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.079+0000 7f561442f700 1 -- start start 2022-06-27T11:47:59.091 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.081+0000 7f249559a700 1 -- 172.21.15.195:0/2217793621 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f248c011630 con 0x7f24980c3d70 2022-06-27T11:47:59.091 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.081+0000 7f249559a700 1 -- 172.21.15.195:0/2217793621 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f248c011790 con 0x7f24980c3d70 2022-06-27T11:47:59.091 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.081+0000 7f249559a700 1 -- 172.21.15.195:0/2217793621 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f248c011b30 con 0x7f24980c3d70 2022-06-27T11:47:59.092 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.079+0000 7fcda7d57700 1 -- start start 2022-06-27T11:47:59.092 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.080+0000 7fcda7d57700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcda00c97e0 0x7fcda0162440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.093 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.080+0000 7fcda7d57700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcda00c9bb0 0x7fcda006d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.093 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.080+0000 7fcda7d57700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcda006cae0 0x7fcda00b1a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.093 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.080+0000 7fcda7d57700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcda00ab150 con 0x7fcda006cae0 2022-06-27T11:47:59.093 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.080+0000 7fcda7d57700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcda006c650 con 0x7fcda00c9bb0 2022-06-27T11:47:59.093 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.080+0000 7fcda7d57700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcda006db90 con 0x7fcda00c97e0 2022-06-27T11:47:59.094 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.081+0000 7fcda5af3700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcda00c97e0 0x7fcda0162440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.094 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.081+0000 7fcda52f2700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcda00c9bb0 0x7fcda006d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.095 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.081+0000 7fcda5af3700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcda00c97e0 0x7fcda0162440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3301/0 says I am v2:172.21.15.195:34096/0 (socket says 172.21.15.195:34096) 2022-06-27T11:47:59.095 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.081+0000 7fcda5af3700 1 -- 172.21.15.195:0/1338727304 learned_addr learned my addr 172.21.15.195:0/1338727304 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:47:59.095 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.080+0000 7f561442f700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f560c0c6590 0x7f560c15b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.096 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.080+0000 7f561442f700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f560c067ed0 con 0x7f560c06a0c0 2022-06-27T11:47:59.096 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.080+0000 7f561442f700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f560c068120 con 0x7f560c069cf0 2022-06-27T11:47:59.096 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.080+0000 7f561442f700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f560c0c48a0 con 0x7f560c0c6590 2022-06-27T11:47:59.096 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.081+0000 7f56121cb700 1 --1- >> v1:172.21.15.202:6789/0 conn(0x7f560c069cf0 0x7f560c0c4190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.202:6789/0 says I am v1:172.21.15.195:49380/0 (socket says 172.21.15.195:49380) 2022-06-27T11:47:59.097 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.081+0000 7f56121cb700 1 -- 172.21.15.195:0/176791578 learned_addr learned my addr 172.21.15.195:0/176791578 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:47:59.097 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.081+0000 7f56119ca700 1 --2- 172.21.15.195:0/176791578 >> v2:172.21.15.202:3300/0 conn(0x7f560c0c6590 0x7f560c15b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.097 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.081+0000 7fcda52f2700 1 -- 172.21.15.195:0/1338727304 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcda00c97e0 msgr2=0x7fcda0162440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.098 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.081+0000 7fcda52f2700 1 --2- 172.21.15.195:0/1338727304 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcda00c97e0 0x7fcda0162440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.098 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.081+0000 7fcda52f2700 1 -- 172.21.15.195:0/1338727304 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcda006cae0 msgr2=0x7fcda00b1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.098 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.081+0000 7fcda52f2700 1 --2- 172.21.15.195:0/1338727304 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcda006cae0 0x7fcda00b1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.098 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.081+0000 7fcda52f2700 1 -- 172.21.15.195:0/1338727304 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcda0163320 con 0x7fcda00c9bb0 2022-06-27T11:47:59.099 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.081+0000 7fe9d7435700 1 Processor -- start 2022-06-27T11:47:59.099 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.081+0000 7f56119ca700 1 -- 172.21.15.195:0/176791578 >> v1:172.21.15.202:6789/0 conn(0x7f560c069cf0 legacy=0x7f560c0c4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.100 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.081+0000 7f56119ca700 1 -- 172.21.15.195:0/176791578 >> v1:172.21.15.195:6789/0 conn(0x7f560c06a0c0 legacy=0x7f560c154650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.100 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.081+0000 7f56119ca700 1 -- 172.21.15.195:0/176791578 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f560c068400 con 0x7f560c0c6590 2022-06-27T11:47:59.100 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.082+0000 7f56119ca700 1 --2- 172.21.15.195:0/176791578 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f560c0c6590 0x7f560c15b310 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f55fc0097a0 tx=0x7f55fc009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=822a0c7a248c5a53 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:59.101 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.082+0000 7f249c93b700 1 -- 172.21.15.195:0/2217793621 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f24980c3d70 msgr2=0x7f24980c2610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.101 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.082+0000 7f249c93b700 1 --2- 172.21.15.195:0/2217793621 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f24980c3d70 0x7f24980c2610 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f248c0097a0 tx=0x7f248c009060 comp rx=0 tx=0).stop 2022-06-27T11:47:59.101 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.082+0000 7fe9d7435700 1 -- start start 2022-06-27T11:47:59.103 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.082+0000 7f56111c9700 1 -- 172.21.15.195:0/176791578 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55fc010590 con 0x7f560c0c6590 2022-06-27T11:47:59.103 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.082+0000 7f56111c9700 1 -- 172.21.15.195:0/176791578 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55fc0106f0 con 0x7f560c0c6590 2022-06-27T11:47:59.103 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.082+0000 7f56111c9700 1 -- 172.21.15.195:0/176791578 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55fc010ab0 con 0x7f560c0c6590 2022-06-27T11:47:59.104 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.083+0000 7f58f0922700 1 Processor -- start 2022-06-27T11:47:59.104 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.083+0000 7f58f0922700 1 -- start start 2022-06-27T11:47:59.105 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.083+0000 7fe9d7435700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fe9d0125950 0x7fe9d00c3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.105 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.083+0000 7fe9d7435700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9d0156770 con 0x7fe9d00c2610 2022-06-27T11:47:59.106 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.083+0000 7fe9d7435700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9d0156a50 con 0x7fe9d00c3b90 2022-06-27T11:47:59.106 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.083+0000 7fe9d7435700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9d00c37e0 con 0x7fe9d0125950 2022-06-27T11:47:59.106 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.083+0000 7fe9d51d1700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fe9d00c2610 0x7fe9d00c3010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:47532/0 (socket says 172.21.15.195:47532) 2022-06-27T11:47:59.106 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.083+0000 7fe9d51d1700 1 -- 172.21.15.195:0/3491633377 learned_addr learned my addr 172.21.15.195:0/3491633377 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:47:59.107 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.083+0000 7fe9d59d2700 1 --2- 172.21.15.195:0/3491633377 >> v2:172.21.15.202:3300/0 conn(0x7fe9d0125950 0x7fe9d00c3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.107 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.084+0000 7fe9bffff700 1 -- 172.21.15.195:0/3491633377 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1198627379 0 0) 0x7fe9d0156770 con 0x7fe9d00c2610 2022-06-27T11:47:59.107 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.084+0000 7fe9bffff700 1 -- 172.21.15.195:0/3491633377 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9b8003430 con 0x7fe9d00c2610 2022-06-27T11:47:59.108 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.084+0000 7fe9d59d2700 1 -- 172.21.15.195:0/3491633377 >> v1:172.21.15.202:6789/0 conn(0x7fe9d00c3b90 legacy=0x7fe9d0153630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:47:59.108 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.084+0000 7fe9d59d2700 1 -- 172.21.15.195:0/3491633377 >> v1:172.21.15.195:6789/0 conn(0x7fe9d00c2610 legacy=0x7fe9d00c3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.108 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.084+0000 7fe9d59d2700 1 -- 172.21.15.195:0/3491633377 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9d0156d30 con 0x7fe9d0125950 2022-06-27T11:47:59.108 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.084+0000 7fe9d59d2700 1 --2- 172.21.15.195:0/3491633377 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe9d0125950 0x7fe9d00c3fc0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe9c8009e60 tx=0x7fe9c8009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=14a4a49b37d37e47 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:59.109 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.084+0000 7fcda52f2700 1 --2- 172.21.15.195:0/1338727304 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcda00c9bb0 0x7fcda006d650 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fcd90015000 tx=0x7fcd90016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:59.109 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.084+0000 7fcd8dffb700 1 -- 172.21.15.195:0/1338727304 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd90011cf0 con 0x7fcda00c9bb0 2022-06-27T11:47:59.109 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.084+0000 7fcda7d57700 1 -- 172.21.15.195:0/1338727304 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcda0162a80 con 0x7fcda00c9bb0 2022-06-27T11:47:59.110 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.084+0000 7fe9bffff700 1 -- 172.21.15.195:0/3491633377 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9c800e070 con 0x7fe9d0125950 2022-06-27T11:47:59.111 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.084+0000 7f249c93b700 1 -- 172.21.15.195:0/2217793621 shutdown_connections 2022-06-27T11:47:59.111 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.085+0000 7f249c93b700 1 --2- 172.21.15.195:0/2217793621 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2498153630 0x7f2498153ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.111 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.085+0000 7f249c93b700 1 --2- 172.21.15.195:0/2217793621 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f24980c3d70 0x7f24980c2610 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.111 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.085+0000 7f249c93b700 1 -- 172.21.15.195:0/2217793621 >> 172.21.15.195:0/2217793621 conn(0x7f24980c8740 msgr2=0x7f24980c8b10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:59.112 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.084+0000 7fe9bffff700 1 -- 172.21.15.195:0/3491633377 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9c800bbb0 con 0x7fe9d0125950 2022-06-27T11:47:59.112 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.084+0000 7fe9bffff700 1 -- 172.21.15.195:0/3491633377 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9c80038a0 con 0x7fe9d0125950 2022-06-27T11:47:59.113 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.084+0000 7fcda7d57700 1 -- 172.21.15.195:0/1338727304 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcda0163040 con 0x7fcda00c9bb0 2022-06-27T11:47:59.114 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.084+0000 7f58f0922700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f58ec0c3d70 0x7f58ec0c0d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.114 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.084+0000 7f58f0922700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58ec0c08d0 con 0x7f58ec0c4ae0 2022-06-27T11:47:59.115 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.084+0000 7f58f0922700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58ec0c33c0 con 0x7f58ec0c4140 2022-06-27T11:47:59.115 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.084+0000 7f58f0922700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58ec0685e0 con 0x7f58ec0c3d70 2022-06-27T11:47:59.115 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.084+0000 7f58e9d9b700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f58ec0c4ae0 0x7f58ec068760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:47540/0 (socket says 172.21.15.195:47540) 2022-06-27T11:47:59.115 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.084+0000 7f58e9d9b700 1 -- 172.21.15.195:0/649638938 learned_addr learned my addr 172.21.15.195:0/649638938 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:47:59.116 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.084+0000 7f58ead9d700 1 --1- 172.21.15.195:0/649638938 >> v1:172.21.15.202:6789/0 conn(0x7f58ec0c4140 0x7f58ec067ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.202:6789/0 says I am v1:172.21.15.195:49402/0 (socket says 172.21.15.195:49402) 2022-06-27T11:47:59.116 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.084+0000 7f58ea59c700 1 --2- 172.21.15.195:0/649638938 >> v2:172.21.15.195:3300/0 conn(0x7f58ec0c3d70 0x7f58ec0c0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.116 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.085+0000 7f58e959a700 1 -- 172.21.15.195:0/649638938 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 317997187 0 0) 0x7f58ec0c33c0 con 0x7f58ec0c4140 2022-06-27T11:47:59.116 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.085+0000 7f58e959a700 1 -- 172.21.15.195:0/649638938 --> v1:172.21.15.202:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58d0003430 con 0x7f58ec0c4140 2022-06-27T11:47:59.117 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.085+0000 7f58e959a700 1 -- 172.21.15.195:0/649638938 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2352999626 0 0) 0x7f58ec0c08d0 con 0x7f58ec0c4ae0 2022-06-27T11:47:59.117 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.085+0000 7f58e959a700 1 -- 172.21.15.195:0/649638938 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58ec0c33c0 con 0x7f58ec0c4ae0 2022-06-27T11:47:59.117 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.085+0000 7f58e959a700 1 -- 172.21.15.195:0/649638938 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3518607728 0 0) 0x7f58ec0c33c0 con 0x7f58ec0c4ae0 2022-06-27T11:47:59.117 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.085+0000 7f58e959a700 1 -- 172.21.15.195:0/649638938 >> v2:172.21.15.195:3300/0 conn(0x7f58ec0c3d70 msgr2=0x7f58ec0c0d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:47:59.118 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.085+0000 7f58e959a700 1 --2- 172.21.15.195:0/649638938 >> v2:172.21.15.195:3300/0 conn(0x7f58ec0c3d70 0x7f58ec0c0d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.118 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.085+0000 7f58e959a700 1 -- 172.21.15.195:0/649638938 >> v1:172.21.15.202:6789/0 conn(0x7f58ec0c4140 legacy=0x7f58ec067ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.118 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.085+0000 7f58e959a700 1 -- 172.21.15.195:0/649638938 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58ec0c3660 con 0x7f58ec0c4ae0 2022-06-27T11:47:59.118 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.085+0000 7f58e959a700 1 -- 172.21.15.195:0/649638938 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7f58e00013a0 con 0x7f58ec0c4ae0 2022-06-27T11:47:59.119 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.085+0000 7f58e959a700 1 -- 172.21.15.195:0/649638938 >> v1:172.21.15.195:6789/0 conn(0x7f58ec0c4ae0 legacy=0x7f58ec068760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.119 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.085+0000 7f58e959a700 1 --2- 172.21.15.195:0/649638938 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f58d0004b60 0x7f58d0004f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.119 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.085+0000 7f58e959a700 1 --2- 172.21.15.195:0/649638938 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f58d0003670 0x7f58d0003ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.120 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.085+0000 7f58e959a700 1 --2- 172.21.15.195:0/649638938 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f58d0004140 0x7f58d000b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.120 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.085+0000 7f58e959a700 1 -- 172.21.15.195:0/649638938 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58ec0685e0 con 0x7f58d0004b60 2022-06-27T11:47:59.121 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.085+0000 7f58e959a700 1 -- 172.21.15.195:0/649638938 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58d000c070 con 0x7f58d0003670 2022-06-27T11:47:59.121 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.085+0000 7f58e959a700 1 -- 172.21.15.195:0/649638938 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58d000c230 con 0x7f58d0004140 2022-06-27T11:47:59.122 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.085+0000 7fcd8dffb700 1 -- 172.21.15.195:0/1338727304 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd90011e70 con 0x7fcda00c9bb0 2022-06-27T11:47:59.123 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.085+0000 7fcd8dffb700 1 -- 172.21.15.195:0/1338727304 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd90011960 con 0x7fcda00c9bb0 2022-06-27T11:47:59.123 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.085+0000 7f58e9d9b700 1 --2- 172.21.15.195:0/649638938 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f58d0003670 0x7f58d0003ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.124 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.086+0000 7fcda7d57700 1 -- 172.21.15.195:0/1338727304 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fcd7c000fd0 con 0x7fcda00c9bb0 2022-06-27T11:47:59.125 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.086+0000 7fe9d7435700 1 -- 172.21.15.195:0/3491633377 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe9d0125950 msgr2=0x7fe9d00c3fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.125 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.086+0000 7fe9d7435700 1 --2- 172.21.15.195:0/3491633377 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe9d0125950 0x7fe9d00c3fc0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe9c8009e60 tx=0x7fe9c8009780 comp rx=0 tx=0).stop 2022-06-27T11:47:59.126 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.086+0000 7f58ead9d700 1 --2- 172.21.15.195:0/649638938 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f58d0004140 0x7f58d000b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.126 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.086+0000 7f58ea59c700 1 --2- 172.21.15.195:0/649638938 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f58d0004b60 0x7f58d0004f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.126 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.086+0000 7f58e9d9b700 1 -- 172.21.15.195:0/649638938 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f58d0004140 msgr2=0x7f58d000b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:47:59.126 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.086+0000 7f58e9d9b700 1 --2- 172.21.15.195:0/649638938 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f58d0004140 0x7f58d000b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.127 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.086+0000 7f58e9d9b700 1 -- 172.21.15.195:0/649638938 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f58d0004b60 msgr2=0x7f58d0004f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:47:59.127 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.086+0000 7f58e9d9b700 1 --2- 172.21.15.195:0/649638938 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f58d0004b60 0x7f58d0004f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.127 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.086+0000 7f58e9d9b700 1 -- 172.21.15.195:0/649638938 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58ec0c3660 con 0x7f58d0003670 2022-06-27T11:47:59.127 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.086+0000 7f58ead9d700 1 -- 172.21.15.195:0/649638938 reap_dead start 2022-06-27T11:47:59.127 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.086+0000 7f58e9d9b700 1 --2- 172.21.15.195:0/649638938 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f58d0003670 0x7f58d0003ae0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f58e000c110 tx=0x7f58e0001d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b950c759e770e97 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:59.128 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.086+0000 7f58e959a700 1 -- 172.21.15.195:0/649638938 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e000c9c0 con 0x7f58d0003670 2022-06-27T11:47:59.128 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.086+0000 7f58e959a700 1 -- 172.21.15.195:0/649638938 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58e000cb20 con 0x7f58d0003670 2022-06-27T11:47:59.128 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.086+0000 7f58e959a700 1 -- 172.21.15.195:0/649638938 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e00070d0 con 0x7f58d0003670 2022-06-27T11:47:59.128 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.087+0000 7f249c93b700 1 -- 172.21.15.195:0/2217793621 shutdown_connections 2022-06-27T11:47:59.129 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.086+0000 7fcd8dffb700 1 -- 172.21.15.195:0/1338727304 <== mon.1 v2:172.21.15.202:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fcd9000f070 con 0x7fcda00c9bb0 2022-06-27T11:47:59.129 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.087+0000 7fe9d7435700 1 -- 172.21.15.195:0/3491633377 shutdown_connections 2022-06-27T11:47:59.130 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.087+0000 7fe9d7435700 1 --2- 172.21.15.195:0/3491633377 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe9d0125950 0x7fe9d00c3fc0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.130 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.087+0000 7fe9d7435700 1 -- 172.21.15.195:0/3491633377 >> 172.21.15.195:0/3491633377 conn(0x7fe9d00c8740 msgr2=0x7fe9d00c8b10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:59.130 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.087+0000 7fcd8dffb700 1 --2- 172.21.15.195:0/1338727304 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcd80063d00 0x7fcd800661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.131 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.087+0000 7f249c93b700 1 -- 172.21.15.195:0/2217793621 wait complete. 2022-06-27T11:47:59.132 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.087+0000 7fcd8dffb700 1 -- 172.21.15.195:0/1338727304 <== mon.1 v2:172.21.15.202:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fcd90081e70 con 0x7fcda00c9bb0 2022-06-27T11:47:59.132 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.087+0000 7fcd8dffb700 1 --2- 172.21.15.195:0/1338727304 >> [v2:172.21.15.202:6808/94310,v1:172.21.15.202:6810/94310] conn(0x7fcd8006c6d0 0x7fcd8006eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.132 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.087+0000 7fcd8dffb700 1 -- 172.21.15.195:0/1338727304 --> [v2:172.21.15.202:6808/94310,v1:172.21.15.202:6810/94310] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcd8006f190 con 0x7fcd8006c6d0 2022-06-27T11:47:59.132 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.087+0000 7fcd8dffb700 1 -- 172.21.15.195:0/1338727304 <== mon.1 v2:172.21.15.202:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcd900874d0 con 0x7fcda00c9bb0 2022-06-27T11:47:59.133 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.087+0000 7fcda62f4700 1 --2- 172.21.15.195:0/1338727304 >> [v2:172.21.15.202:6808/94310,v1:172.21.15.202:6810/94310] conn(0x7fcd8006c6d0 0x7fcd8006eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.133 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.088+0000 7fcda5af3700 1 --2- 172.21.15.195:0/1338727304 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcd80063d00 0x7fcd800661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.133 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.088+0000 7f561442f700 1 -- 172.21.15.195:0/176791578 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f560c0c6590 msgr2=0x7f560c15b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.133 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.088+0000 7f561442f700 1 --2- 172.21.15.195:0/176791578 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f560c0c6590 0x7f560c15b310 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f55fc0097a0 tx=0x7f55fc009060 comp rx=0 tx=0).stop 2022-06-27T11:47:59.134 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.087+0000 7f58f0922700 1 -- 172.21.15.195:0/649638938 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f58d0003670 msgr2=0x7f58d0003ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.134 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.087+0000 7f58f0922700 1 --2- 172.21.15.195:0/649638938 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f58d0003670 0x7f58d0003ae0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f58e000c110 tx=0x7f58e0001d10 comp rx=0 tx=0).stop 2022-06-27T11:47:59.135 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.088+0000 7fe9d7435700 1 -- 172.21.15.195:0/3491633377 shutdown_connections 2022-06-27T11:47:59.135 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.088+0000 7fcda62f4700 1 --2- 172.21.15.195:0/1338727304 >> [v2:172.21.15.202:6808/94310,v1:172.21.15.202:6810/94310] conn(0x7fcd8006c6d0 0x7fcd8006eae0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:59.136 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.088+0000 7f58f0922700 1 -- 172.21.15.195:0/649638938 shutdown_connections 2022-06-27T11:47:59.146 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.088+0000 7f58f0922700 1 --2- 172.21.15.195:0/649638938 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f58d0003670 0x7f58d0003ae0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.146 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.088+0000 7f58f0922700 1 -- 172.21.15.195:0/649638938 >> 172.21.15.195:0/649638938 conn(0x7f58ec0c8740 msgr2=0x7f58ec0c8b10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:59.146 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.088+0000 7f58f0922700 1 -- 172.21.15.195:0/649638938 shutdown_connections 2022-06-27T11:47:59.147 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.088+0000 7f561442f700 1 -- 172.21.15.195:0/176791578 shutdown_connections 2022-06-27T11:47:59.147 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.088+0000 7f561442f700 1 --2- 172.21.15.195:0/176791578 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f560c0c6590 0x7f560c15b310 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.147 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.088+0000 7f561442f700 1 -- 172.21.15.195:0/176791578 >> 172.21.15.195:0/176791578 conn(0x7f560c0c9b00 msgr2=0x7f560c0c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:59.148 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.088+0000 7f58f0922700 1 -- 172.21.15.195:0/649638938 wait complete. 2022-06-27T11:47:59.148 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.089+0000 7f249c93b700 1 Processor -- start 2022-06-27T11:47:59.149 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.089+0000 7f249c93b700 1 -- start start 2022-06-27T11:47:59.149 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.088+0000 7fe9d7435700 1 -- 172.21.15.195:0/3491633377 wait complete. 2022-06-27T11:47:59.149 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.090+0000 7fe9d7435700 1 Processor -- start 2022-06-27T11:47:59.150 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.090+0000 7f58f0922700 1 Processor -- start 2022-06-27T11:47:59.150 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.089+0000 7fcd8dffb700 1 -- 172.21.15.195:0/1338727304 <== osd.7 v2:172.21.15.202:6808/94310 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24062 (crc 0 0 0) 0x7fcd8006f190 con 0x7fcd8006c6d0 2022-06-27T11:47:59.151 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.090+0000 7f249c93b700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f24980c3d70 0x7f24980bde60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.151 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.090+0000 7f249c93b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f24980c4140 0x7f249816e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.151 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.090+0000 7f249c93b700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2498153630 0x7f249816e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.090+0000 7f249c93b700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2498121d50 con 0x7f24980c4140 2022-06-27T11:47:59.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.090+0000 7f249c93b700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24980bf3f0 con 0x7f24980c3d70 2022-06-27T11:47:59.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.090+0000 7f249c93b700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24980bf6d0 con 0x7f2498153630 2022-06-27T11:47:59.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.090+0000 7f249659c700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f24980c3d70 0x7f24980bde60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.090+0000 7f249659c700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f24980c3d70 0x7f24980bde60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3300/0 says I am v2:172.21.15.195:39492/0 (socket says 172.21.15.195:39492) 2022-06-27T11:47:59.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.090+0000 7f249659c700 1 -- 172.21.15.195:0/2103073482 learned_addr learned my addr 172.21.15.195:0/2103073482 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:47:59.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.090+0000 7f2496d9d700 1 --2- 172.21.15.195:0/2103073482 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2498153630 0x7f249816e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.091+0000 7f249659c700 1 -- 172.21.15.195:0/2103073482 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2498153630 msgr2=0x7f249816e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.091+0000 7f249659c700 1 --2- 172.21.15.195:0/2103073482 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2498153630 0x7f249816e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.091+0000 7f249659c700 1 -- 172.21.15.195:0/2103073482 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f24980c4140 msgr2=0x7f249816e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.091+0000 7f249659c700 1 --2- 172.21.15.195:0/2103073482 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f24980c4140 0x7f249816e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.091+0000 7f249659c700 1 -- 172.21.15.195:0/2103073482 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f248c0092e0 con 0x7f24980c3d70 2022-06-27T11:47:59.155 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.090+0000 7fe9d7435700 1 -- start start 2022-06-27T11:47:59.155 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.091+0000 7fe9d7435700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe9d00c2610 0x7fe9d0158c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.155 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.091+0000 7fe9d7435700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe9d0125950 0x7fe9d0151760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.155 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.091+0000 7fe9d7435700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe9d0159180 0x7fe9d0151ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.156 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.091+0000 7fe9d7435700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d0124680 con 0x7fe9d00c2610 2022-06-27T11:47:59.156 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.091+0000 7fe9d7435700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d006fba0 con 0x7fe9d0125950 2022-06-27T11:47:59.156 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.091+0000 7fe9d7435700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d016ac90 con 0x7fe9d0159180 2022-06-27T11:47:59.156 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.092+0000 7fe9d51d1700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe9d00c2610 0x7fe9d0158c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.157 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.090+0000 7f58f0922700 1 -- start start 2022-06-27T11:47:59.157 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.091+0000 7f249659c700 1 --2- 172.21.15.195:0/2103073482 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f24980c3d70 0x7f24980bde60 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f248c009e40 tx=0x7f248c017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:59.158 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.091+0000 7f58f0922700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f58d0003670 0x7f58ec0c6820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.158 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.091+0000 7f58f0922700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f58ec172cf0 0x7f58ec0b6690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.159 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.091+0000 7f58f0922700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f58ec0b6bd0 0x7f58ec169540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.159 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.091+0000 7f58f0922700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58ec167fb0 con 0x7f58d0003670 2022-06-27T11:47:59.159 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.092+0000 7f58f0922700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58ec0b5f20 con 0x7f58ec172cf0 2022-06-27T11:47:59.159 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.092+0000 7f58f0922700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58ec173230 con 0x7f58ec0b6bd0 2022-06-27T11:47:59.160 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.092+0000 7fe9d51d1700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe9d00c2610 0x7fe9d0158c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41372/0 (socket says 172.21.15.195:41372) 2022-06-27T11:47:59.160 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.092+0000 7fe9d51d1700 1 -- 172.21.15.195:0/1080953889 learned_addr learned my addr 172.21.15.195:0/1080953889 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:47:59.160 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.092+0000 7fe9d49d0700 1 --2- 172.21.15.195:0/1080953889 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe9d0125950 0x7fe9d0151760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.161 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.092+0000 7fe9d59d2700 1 --2- 172.21.15.195:0/1080953889 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe9d0159180 0x7fe9d0151ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.161 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.092+0000 7f58ead9d700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f58ec0b6bd0 0x7f58ec169540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.162 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.092+0000 7f58ead9d700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f58ec0b6bd0 0x7f58ec169540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3301/0 says I am v2:172.21.15.195:34134/0 (socket says 172.21.15.195:34134) 2022-06-27T11:47:59.162 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.092+0000 7f58ead9d700 1 -- 172.21.15.195:0/3177104790 learned_addr learned my addr 172.21.15.195:0/3177104790 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:47:59.162 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.092+0000 7fe9d51d1700 1 -- 172.21.15.195:0/1080953889 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe9d0159180 msgr2=0x7fe9d0151ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.162 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.092+0000 7fe9d51d1700 1 --2- 172.21.15.195:0/1080953889 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe9d0159180 0x7fe9d0151ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.163 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.092+0000 7fe9d51d1700 1 -- 172.21.15.195:0/1080953889 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe9d0125950 msgr2=0x7fe9d0151760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.163 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.092+0000 7fcda5af3700 1 --2- 172.21.15.195:0/1338727304 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcd80063d00 0x7fcd800661b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fcd980014a0 tx=0x7fcd98007d50 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:59.164 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.092+0000 7fe9d51d1700 1 --2- 172.21.15.195:0/1080953889 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe9d0125950 0x7fe9d0151760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.164 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.092+0000 7fe9d51d1700 1 -- 172.21.15.195:0/1080953889 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c8009a20 con 0x7fe9d00c2610 2022-06-27T11:47:59.164 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.092+0000 7fe9d51d1700 1 --2- 172.21.15.195:0/1080953889 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe9d00c2610 0x7fe9d0158c40 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe9cc00c4b0 tx=0x7fe9cc007c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:59.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.092+0000 7f58e9d9b700 1 --2- 172.21.15.195:0/3177104790 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f58ec172cf0 0x7f58ec0b6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.092+0000 7f561442f700 1 -- 172.21.15.195:0/176791578 shutdown_connections 2022-06-27T11:47:59.166 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.093+0000 7fe9bd7fa700 1 -- 172.21.15.195:0/1080953889 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9cc00ce00 con 0x7fe9d00c2610 2022-06-27T11:47:59.166 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.093+0000 7fe9bd7fa700 1 -- 172.21.15.195:0/1080953889 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9cc0072e0 con 0x7fe9d00c2610 2022-06-27T11:47:59.166 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.093+0000 7fe9bd7fa700 1 -- 172.21.15.195:0/1080953889 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9cc0076a0 con 0x7fe9d00c2610 2022-06-27T11:47:59.167 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.093+0000 7f58ea59c700 1 --2- 172.21.15.195:0/3177104790 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f58d0003670 0x7f58ec0c6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.167 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.093+0000 7f561442f700 1 -- 172.21.15.195:0/176791578 wait complete. 2022-06-27T11:47:59.167 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.094+0000 7f561442f700 1 Processor -- start 2022-06-27T11:47:59.168 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.093+0000 7f58ea59c700 1 -- 172.21.15.195:0/3177104790 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f58ec0b6bd0 msgr2=0x7f58ec169540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.168 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.093+0000 7f58ea59c700 1 --2- 172.21.15.195:0/3177104790 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f58ec0b6bd0 0x7f58ec169540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.169 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.093+0000 7f58ea59c700 1 -- 172.21.15.195:0/3177104790 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f58ec172cf0 msgr2=0x7f58ec0b6690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.169 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.093+0000 7f58ea59c700 1 --2- 172.21.15.195:0/3177104790 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f58ec172cf0 0x7f58ec0b6690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.169 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.093+0000 7f58ea59c700 1 -- 172.21.15.195:0/3177104790 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58ec173510 con 0x7f58d0003670 2022-06-27T11:47:59.169 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.094+0000 7f58ea59c700 1 --2- 172.21.15.195:0/3177104790 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f58d0003670 0x7f58ec0c6820 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f58ec0c2d10 tx=0x7f58dc008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:59.170 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.093+0000 7fe9d7435700 1 -- 172.21.15.195:0/1080953889 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9d006fdd0 con 0x7fe9d00c2610 2022-06-27T11:47:59.170 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.094+0000 7fe9d7435700 1 -- 172.21.15.195:0/1080953889 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9d01242a0 con 0x7fe9d00c2610 2022-06-27T11:47:59.170 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.094+0000 7f561442f700 1 -- start start 2022-06-27T11:47:59.171 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.095+0000 7f58da7fc700 1 -- 172.21.15.195:0/3177104790 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58dc00b040 con 0x7f58d0003670 2022-06-27T11:47:59.171 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.095+0000 7f58da7fc700 1 -- 172.21.15.195:0/3177104790 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58dc017070 con 0x7f58d0003670 2022-06-27T11:47:59.172 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.095+0000 7f58da7fc700 1 -- 172.21.15.195:0/3177104790 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58dc004240 con 0x7f58d0003670 2022-06-27T11:47:59.172 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.095+0000 7f58f0922700 1 -- 172.21.15.195:0/3177104790 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58ec1737f0 con 0x7f58d0003670 2022-06-27T11:47:59.172 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.095+0000 7f58f0922700 1 -- 172.21.15.195:0/3177104790 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58ec0b6170 con 0x7f58d0003670 2022-06-27T11:47:59.173 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.095+0000 7f561442f700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f560c069cf0 0x7f560c153a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.173 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.095+0000 7f561442f700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f560c06a0c0 0x7f560c06e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.174 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.095+0000 7f561442f700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f560c0c6590 0x7f560c14e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.174 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.095+0000 7f561442f700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f560c16ed30 con 0x7f560c069cf0 2022-06-27T11:47:59.174 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.095+0000 7f561442f700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f560c1540a0 con 0x7f560c06a0c0 2022-06-27T11:47:59.174 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.095+0000 7f561442f700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f560c16e0b0 con 0x7f560c0c6590 2022-06-27T11:47:59.174 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.096+0000 7f56119ca700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f560c06a0c0 0x7f560c06e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.175 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.096+0000 7f56119ca700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f560c06a0c0 0x7f560c06e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3300/0 says I am v2:172.21.15.195:39510/0 (socket says 172.21.15.195:39510) 2022-06-27T11:47:59.175 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.096+0000 7f56119ca700 1 -- 172.21.15.195:0/3650733220 learned_addr learned my addr 172.21.15.195:0/3650733220 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:47:59.175 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.096+0000 7f56129cc700 1 --2- 172.21.15.195:0/3650733220 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f560c0c6590 0x7f560c14e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.175 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.096+0000 7f56121cb700 1 --2- 172.21.15.195:0/3650733220 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f560c069cf0 0x7f560c153a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.176 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.096+0000 7f58f0922700 1 -- 172.21.15.195:0/3177104790 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f58ec0722c0 con 0x7f58d0003670 2022-06-27T11:47:59.176 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.096+0000 7f56119ca700 1 -- 172.21.15.195:0/3650733220 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f560c0c6590 msgr2=0x7f560c14e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.177 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.096+0000 7f56119ca700 1 --2- 172.21.15.195:0/3650733220 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f560c0c6590 0x7f560c14e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.177 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.096+0000 7f56119ca700 1 -- 172.21.15.195:0/3650733220 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f560c069cf0 msgr2=0x7f560c153a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.177 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.096+0000 7f56119ca700 1 --2- 172.21.15.195:0/3650733220 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f560c069cf0 0x7f560c153a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.177 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.096+0000 7f56119ca700 1 -- 172.21.15.195:0/3650733220 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55fc0092e0 con 0x7f560c06a0c0 2022-06-27T11:47:59.177 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.096+0000 7f56129cc700 1 --2- 172.21.15.195:0/3650733220 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f560c0c6590 0x7f560c14e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T11:47:59.178 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.096+0000 7fe9bd7fa700 1 -- 172.21.15.195:0/1080953889 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fe9cc019030 con 0x7fe9d00c2610 2022-06-27T11:47:59.178 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.096+0000 7fe67720e700 1 Processor -- start 2022-06-27T11:47:59.179 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.096+0000 7f56119ca700 1 --2- 172.21.15.195:0/3650733220 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f560c06a0c0 0x7f560c06e650 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f55fc009ef0 tx=0x7f55fc018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:59.180 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.097+0000 7fe9d7435700 1 -- 172.21.15.195:0/1080953889 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe9d00722c0 con 0x7fe9d00c2610 2022-06-27T11:47:59.180 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.097+0000 7fe9bd7fa700 1 --2- 172.21.15.195:0/1080953889 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe9b0063d80 0x7fe9b0066230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.180 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.097+0000 7fe9bd7fa700 1 -- 172.21.15.195:0/1080953889 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fe9cc07fa70 con 0x7fe9d00c2610 2022-06-27T11:47:59.180 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.097+0000 7fe9bd7fa700 1 --2- 172.21.15.195:0/1080953889 >> [v2:172.21.15.195:6808/94223,v1:172.21.15.195:6809/94223] conn(0x7fe9b006c6d0 0x7fe9b006eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.181 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.097+0000 7fe9bd7fa700 1 -- 172.21.15.195:0/1080953889 --> [v2:172.21.15.195:6808/94223,v1:172.21.15.195:6809/94223] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe9b006f190 con 0x7fe9b006c6d0 2022-06-27T11:47:59.181 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.097+0000 7fe9d49d0700 1 --2- 172.21.15.195:0/1080953889 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe9b0063d80 0x7fe9b0066230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.181 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.097+0000 7fe9d59d2700 1 --2- 172.21.15.195:0/1080953889 >> [v2:172.21.15.195:6808/94223,v1:172.21.15.195:6809/94223] conn(0x7fe9b006c6d0 0x7fe9b006eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.182 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.097+0000 7f24867fc700 1 -- 172.21.15.195:0/2103073482 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f248c0037b0 con 0x7f24980c3d70 2022-06-27T11:47:59.182 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.097+0000 7f24867fc700 1 -- 172.21.15.195:0/2103073482 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f248c003910 con 0x7f24980c3d70 2022-06-27T11:47:59.182 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.097+0000 7f24867fc700 1 -- 172.21.15.195:0/2103073482 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f248c003cd0 con 0x7f24980c3d70 2022-06-27T11:47:59.183 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.098+0000 7fe9d49d0700 1 --2- 172.21.15.195:0/1080953889 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe9b0063d80 0x7fe9b0066230 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe9d00c9a30 tx=0x7fe9c0009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:59.183 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.098+0000 7fe9d59d2700 1 --2- 172.21.15.195:0/1080953889 >> [v2:172.21.15.195:6808/94223,v1:172.21.15.195:6809/94223] conn(0x7fe9b006c6d0 0x7fe9b006eae0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:59.183 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.099+0000 7fe9bd7fa700 1 -- 172.21.15.195:0/1080953889 <== osd.1 v2:172.21.15.195:6808/94223 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24062 (crc 0 0 0) 0x7fe9b006f190 con 0x7fe9b006c6d0 2022-06-27T11:47:59.184 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.098+0000 7f56027fc700 1 -- 172.21.15.195:0/3650733220 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55fc006950 con 0x7f560c06a0c0 2022-06-27T11:47:59.184 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.098+0000 7f56027fc700 1 -- 172.21.15.195:0/3650733220 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55fc020070 con 0x7f560c06a0c0 2022-06-27T11:47:59.184 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.098+0000 7f56027fc700 1 -- 172.21.15.195:0/3650733220 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55fc003750 con 0x7f560c06a0c0 2022-06-27T11:47:59.185 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.098+0000 7f561442f700 1 -- 172.21.15.195:0/3650733220 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f560c16e390 con 0x7f560c06a0c0 2022-06-27T11:47:59.185 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.099+0000 7f561442f700 1 -- 172.21.15.195:0/3650733220 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f560c16e950 con 0x7f560c06a0c0 2022-06-27T11:47:59.185 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.096+0000 7fe67720e700 1 -- start start 2022-06-27T11:47:59.185 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.098+0000 7fe67720e700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fe6700c3980 0x7fe6700c0a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.186 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.098+0000 7fe67720e700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe670155b50 con 0x7fe6700c41b0 2022-06-27T11:47:59.186 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.098+0000 7fe67720e700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe670155df0 con 0x7fe6700c4580 2022-06-27T11:47:59.186 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.098+0000 7fe67720e700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6701527a0 con 0x7fe6700c3980 2022-06-27T11:47:59.186 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.099+0000 7fe6757ab700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fe6700c3980 0x7fe6700c0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.187 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.099+0000 7fe6757ab700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fe6700c3980 0x7fe6700c0a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3300/0 says I am v2:172.21.15.195:39524/0 (socket says 172.21.15.195:39524) 2022-06-27T11:47:59.187 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.099+0000 7fe6757ab700 1 -- 172.21.15.195:0/4152211537 learned_addr learned my addr 172.21.15.195:0/4152211537 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:47:59.187 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.098+0000 7f249c93b700 1 -- 172.21.15.195:0/2103073482 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24980bf9b0 con 0x7f24980c3d70 2022-06-27T11:47:59.188 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.098+0000 7f249c93b700 1 -- 172.21.15.195:0/2103073482 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24980c94a0 con 0x7f24980c3d70 2022-06-27T11:47:59.188 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.100+0000 7f24867fc700 1 -- 172.21.15.195:0/2103073482 <== mon.1 v2:172.21.15.202:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f248c02a020 con 0x7f24980c3d70 2022-06-27T11:47:59.188 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.100+0000 7f561442f700 1 -- 172.21.15.195:0/3650733220 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f560c0722c0 con 0x7f560c06a0c0 2022-06-27T11:47:59.189 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.100+0000 7f58da7fc700 1 -- 172.21.15.195:0/3177104790 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f58dc023070 con 0x7f58d0003670 2022-06-27T11:47:59.189 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.100+0000 7f24867fc700 1 --2- 172.21.15.195:0/2103073482 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2474063e80 0x7f2474066330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.189 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.100+0000 7f24867fc700 1 -- 172.21.15.195:0/2103073482 <== mon.1 v2:172.21.15.202:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f248c081fc0 con 0x7f24980c3d70 2022-06-27T11:47:59.190 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.100+0000 7fe6757ab700 1 -- 172.21.15.195:0/4152211537 >> v1:172.21.15.202:6789/0 conn(0x7fe6700c4580 legacy=0x7fe6700c3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.190 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.100+0000 7fe6757ab700 1 -- 172.21.15.195:0/4152211537 >> v1:172.21.15.195:6789/0 conn(0x7fe6700c41b0 legacy=0x7fe670152bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.190 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.100+0000 7fe6757ab700 1 -- 172.21.15.195:0/4152211537 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6701560d0 con 0x7fe6700c3980 2022-06-27T11:47:59.191 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.100+0000 7f56027fc700 1 -- 172.21.15.195:0/3650733220 <== mon.1 v2:172.21.15.202:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f55fc02b020 con 0x7f560c06a0c0 2022-06-27T11:47:59.191 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.100+0000 7f58da7fc700 1 --2- 172.21.15.195:0/3177104790 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f58c8063e50 0x7f58c8066300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.192 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.101+0000 7f58da7fc700 1 -- 172.21.15.195:0/3177104790 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f58dc0802c0 con 0x7f58d0003670 2022-06-27T11:47:59.192 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.100+0000 7fdb3e5ea700 1 Processor -- start 2022-06-27T11:47:59.193 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.101+0000 7f58da7fc700 1 --2- 172.21.15.195:0/3177104790 >> [v2:172.21.15.202:6800/94308,v1:172.21.15.202:6801/94308] conn(0x7f58c806c820 0x7f58c806ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.193 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.101+0000 7f58da7fc700 1 -- 172.21.15.195:0/3177104790 --> [v2:172.21.15.202:6800/94308,v1:172.21.15.202:6801/94308] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f58c806f2e0 con 0x7f58c806c820 2022-06-27T11:47:59.193 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.101+0000 7f58da7fc700 1 -- 172.21.15.195:0/3177104790 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f58dc0805f0 con 0x7f58d0003670 2022-06-27T11:47:59.194 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.101+0000 7f56027fc700 1 --2- 172.21.15.195:0/3650733220 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f55f4063e50 0x7f55f4066300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.194 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.101+0000 7f56027fc700 1 -- 172.21.15.195:0/3650733220 <== mon.1 v2:172.21.15.202:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f55fc0822e0 con 0x7f560c06a0c0 2022-06-27T11:47:59.194 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.101+0000 7f249c93b700 1 --2- 172.21.15.195:0/2103073482 >> [v2:172.21.15.195:6800/94221,v1:172.21.15.195:6801/94221] conn(0x7f249814eee0 0x7f249814f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.195 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.101+0000 7f249c93b700 1 -- 172.21.15.195:0/2103073482 --> [v2:172.21.15.195:6800/94221,v1:172.21.15.195:6801/94221] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f24980c3970 con 0x7f249814eee0 2022-06-27T11:47:59.195 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.101+0000 7f2496d9d700 1 --2- 172.21.15.195:0/2103073482 >> [v2:172.21.15.195:6800/94221,v1:172.21.15.195:6801/94221] conn(0x7f249814eee0 0x7f249814f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.195 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.101+0000 7f58ead9d700 1 --2- 172.21.15.195:0/3177104790 >> [v2:172.21.15.202:6800/94308,v1:172.21.15.202:6801/94308] conn(0x7f58c806c820 0x7f58c806ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.196 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.101+0000 7fe6757ab700 1 --2- 172.21.15.195:0/4152211537 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe6700c3980 0x7fe6700c0a20 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe664009160 tx=0x7fe6640099b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=711a16948b1a40af server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:59.196 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.102+0000 7f58e9d9b700 1 --2- 172.21.15.195:0/3177104790 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f58c8063e50 0x7f58c8066300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.197 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.102+0000 7f2495d9b700 1 --2- 172.21.15.195:0/2103073482 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2474063e80 0x7f2474066330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.197 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.101+0000 7f56027fc700 1 --2- 172.21.15.195:0/3650733220 >> [v2:172.21.15.202:6809/94309,v1:172.21.15.202:6811/94309] conn(0x7f55f406c820 0x7f55f406ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.197 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.101+0000 7f56027fc700 1 -- 172.21.15.195:0/3650733220 --> [v2:172.21.15.202:6809/94309,v1:172.21.15.202:6811/94309] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f55f406f2e0 con 0x7f55f406c820 2022-06-27T11:47:59.198 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.102+0000 7fe66f7fe700 1 -- 172.21.15.195:0/4152211537 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe66400c850 con 0x7fe6700c3980 2022-06-27T11:47:59.198 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.101+0000 7f56027fc700 1 -- 172.21.15.195:0/3650733220 <== mon.1 v2:172.21.15.202:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f55fc084880 con 0x7f560c06a0c0 2022-06-27T11:47:59.199 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.102+0000 7fe66f7fe700 1 -- 172.21.15.195:0/4152211537 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe66400c9b0 con 0x7fe6700c3980 2022-06-27T11:47:59.199 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.102+0000 7fe66f7fe700 1 -- 172.21.15.195:0/4152211537 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe66400cd50 con 0x7fe6700c3980 2022-06-27T11:47:59.200 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.103+0000 7fe9bd7fa700 1 -- 172.21.15.195:0/1080953889 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe9cc04e570 con 0x7fe9d00c2610 2022-06-27T11:47:59.200 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.102+0000 7f2496d9d700 1 --2- 172.21.15.195:0/2103073482 >> [v2:172.21.15.195:6800/94221,v1:172.21.15.195:6801/94221] conn(0x7f249814eee0 0x7f249814f2b0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:59.200 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.103+0000 7fe67720e700 1 -- 172.21.15.195:0/4152211537 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe6700c3980 msgr2=0x7fe6700c0a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.201 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.103+0000 7fe67720e700 1 --2- 172.21.15.195:0/4152211537 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe6700c3980 0x7fe6700c0a20 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe664009160 tx=0x7fe6640099b0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.201 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.103+0000 7fe67720e700 1 -- 172.21.15.195:0/4152211537 shutdown_connections 2022-06-27T11:47:59.201 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.103+0000 7fe67720e700 1 --2- 172.21.15.195:0/4152211537 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe6700c3980 0x7fe6700c0a20 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.201 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.103+0000 7fe67720e700 1 -- 172.21.15.195:0/4152211537 >> 172.21.15.195:0/4152211537 conn(0x7fe6700c8740 msgr2=0x7fe6700c8b10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:59.202 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.103+0000 7fe67720e700 1 -- 172.21.15.195:0/4152211537 shutdown_connections 2022-06-27T11:47:59.202 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.103+0000 7f58e9d9b700 1 --2- 172.21.15.195:0/3177104790 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f58c8063e50 0x7f58c8066300 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f58e000c110 tx=0x7f58e0015000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:59.203 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.101+0000 7fdb3e5ea700 1 -- start start 2022-06-27T11:47:59.203 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.102+0000 7fdb3e5ea700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fdb380c4580 0x7fdb380c3270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.203 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.102+0000 7fdb3e5ea700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb38155b50 con 0x7fdb380c41b0 2022-06-27T11:47:59.203 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.102+0000 7fdb3e5ea700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb38155df0 con 0x7fdb380c37b0 2022-06-27T11:47:59.204 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.102+0000 7fdb3e5ea700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb381527a0 con 0x7fdb380c4580 2022-06-27T11:47:59.204 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.102+0000 7fdb377fe700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fdb380c4580 0x7fdb380c3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.204 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.103+0000 7fdb377fe700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fdb380c4580 0x7fdb380c3270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41414/0 (socket says 172.21.15.195:41414) 2022-06-27T11:47:59.204 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.103+0000 7fdb377fe700 1 -- 172.21.15.195:0/1147266857 learned_addr learned my addr 172.21.15.195:0/1147266857 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:47:59.204 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.103+0000 7fdb377fe700 1 -- 172.21.15.195:0/1147266857 >> v1:172.21.15.202:6789/0 conn(0x7fdb380c37b0 legacy=0x7fdb380c0a20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2022-06-27T11:47:59.205 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.103+0000 7fdb377fe700 1 -- 172.21.15.195:0/1147266857 >> v1:172.21.15.195:6789/0 conn(0x7fdb380c41b0 legacy=0x7fdb38152bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:47:59.205 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.103+0000 7fdb377fe700 1 -- 172.21.15.195:0/1147266857 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb381560d0 con 0x7fdb380c4580 2022-06-27T11:47:59.205 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.102+0000 7f56121cb700 1 --2- 172.21.15.195:0/3650733220 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f55f4063e50 0x7f55f4066300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.206 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.102+0000 7f56129cc700 1 --2- 172.21.15.195:0/3650733220 >> [v2:172.21.15.202:6809/94309,v1:172.21.15.202:6811/94309] conn(0x7f55f406c820 0x7f55f406ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.206 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.103+0000 7f56121cb700 1 --2- 172.21.15.195:0/3650733220 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f55f4063e50 0x7f55f4066300 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5608000c00 tx=0x7f5608007b20 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:59.206 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.103+0000 7f56129cc700 1 --2- 172.21.15.195:0/3650733220 >> [v2:172.21.15.202:6809/94309,v1:172.21.15.202:6811/94309] conn(0x7f55f406c820 0x7f55f406ec30 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:59.206 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.103+0000 7f2495d9b700 1 --2- 172.21.15.195:0/2103073482 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2474063e80 0x7f2474066330 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2488007fd0 tx=0x7f2488007890 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:59.207 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.103+0000 7fdb377fe700 1 --2- 172.21.15.195:0/1147266857 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdb380c4580 0x7fdb380c3270 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fdb2c008870 tx=0x7fdb2c008190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d7564f7369b4ceb server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:59.207 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.103+0000 7fdb36ffd700 1 -- 172.21.15.195:0/1147266857 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb2c010070 con 0x7fdb380c4580 2022-06-27T11:47:59.208 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.104+0000 7fdb36ffd700 1 -- 172.21.15.195:0/1147266857 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb2c004030 con 0x7fdb380c4580 2022-06-27T11:47:59.208 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.104+0000 7fdb36ffd700 1 -- 172.21.15.195:0/1147266857 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb2c0043d0 con 0x7fdb380c4580 2022-06-27T11:47:59.208 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.104+0000 7f56027fc700 1 -- 172.21.15.195:0/3650733220 <== osd.6 v2:172.21.15.202:6809/94309 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24062 (crc 0 0 0) 0x7f55f406f2e0 con 0x7f55f406c820 2022-06-27T11:47:59.209 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.104+0000 7f24867fc700 1 -- 172.21.15.195:0/2103073482 <== osd.2 v2:172.21.15.195:6800/94221 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24062 (crc 0 0 0) 0x7f24980c3970 con 0x7f249814eee0 2022-06-27T11:47:59.209 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.103+0000 7fe67720e700 1 -- 172.21.15.195:0/4152211537 wait complete. 2022-06-27T11:47:59.210 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.104+0000 7fe67720e700 1 Processor -- start 2022-06-27T11:47:59.210 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.104+0000 7fdb3e5ea700 1 -- 172.21.15.195:0/1147266857 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdb380c4580 msgr2=0x7fdb380c3270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.210 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.104+0000 7fdb3e5ea700 1 --2- 172.21.15.195:0/1147266857 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdb380c4580 0x7fdb380c3270 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fdb2c008870 tx=0x7fdb2c008190 comp rx=0 tx=0).stop 2022-06-27T11:47:59.211 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.105+0000 7fe67720e700 1 -- start start 2022-06-27T11:47:59.211 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.105+0000 7fe67720e700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe6700c3980 0x7fe670174f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.211 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.105+0000 7fe67720e700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe6700c41b0 0x7fe670175440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.212 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.105+0000 7fe67720e700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe6700c4580 0x7fe6701691a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.212 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.105+0000 7fe67720e700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe670174a70 con 0x7fe6700c4580 2022-06-27T11:47:59.212 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.105+0000 7fe67720e700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe670174770 con 0x7fe6700c3980 2022-06-27T11:47:59.212 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.105+0000 7fe67720e700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6701650f0 con 0x7fe6700c41b0 2022-06-27T11:47:59.213 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.106+0000 7fdb3e5ea700 1 -- 172.21.15.195:0/1147266857 shutdown_connections 2022-06-27T11:47:59.213 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.106+0000 7fdb3e5ea700 1 --2- 172.21.15.195:0/1147266857 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdb380c4580 0x7fdb380c3270 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.213 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.106+0000 7fdb3e5ea700 1 -- 172.21.15.195:0/1147266857 >> 172.21.15.195:0/1147266857 conn(0x7fdb380c8740 msgr2=0x7fdb380c8b10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:59.214 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.106+0000 7fe674faa700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe6700c3980 0x7fe670174f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.214 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.106+0000 7fdb3e5ea700 1 -- 172.21.15.195:0/1147266857 shutdown_connections 2022-06-27T11:47:59.215 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.106+0000 7fdb3e5ea700 1 -- 172.21.15.195:0/1147266857 wait complete. 2022-06-27T11:47:59.215 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.106+0000 7fe674faa700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe6700c3980 0x7fe670174f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3300/0 says I am v2:172.21.15.195:39542/0 (socket says 172.21.15.195:39542) 2022-06-27T11:47:59.215 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.106+0000 7fe674faa700 1 -- 172.21.15.195:0/1938617891 learned_addr learned my addr 172.21.15.195:0/1938617891 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:47:59.216 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.106+0000 7fe66ffff700 1 --2- 172.21.15.195:0/1938617891 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe6700c41b0 0x7fe670175440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.216 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.106+0000 7fe6757ab700 1 --2- 172.21.15.195:0/1938617891 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe6700c4580 0x7fe6701691a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.217 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.107+0000 7fdb3e5ea700 1 Processor -- start 2022-06-27T11:47:59.217 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.108+0000 7fdb3e5ea700 1 -- start start 2022-06-27T11:47:59.217 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.109+0000 7fdb3e5ea700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdb380c37b0 0x7fdb3814f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.217 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.109+0000 7fdb3e5ea700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fdb380c41b0 0x7fdb3814f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.218 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.109+0000 7fdb3e5ea700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fdb380c4580 0x7fdb380c78b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.218 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.109+0000 7fdb3e5ea700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb3814ed20 con 0x7fdb380c37b0 2022-06-27T11:47:59.218 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.109+0000 7fdb3e5ea700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb3816bf00 con 0x7fdb380c41b0 2022-06-27T11:47:59.219 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.106+0000 7fe66ffff700 1 -- 172.21.15.195:0/1938617891 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe6700c3980 msgr2=0x7fe670174f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.219 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.106+0000 7fe66ffff700 1 --2- 172.21.15.195:0/1938617891 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe6700c3980 0x7fe670174f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.219 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.109+0000 7fdb3e5ea700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb3816b9d0 con 0x7fdb380c4580 2022-06-27T11:47:59.220 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.106+0000 7fe66ffff700 1 -- 172.21.15.195:0/1938617891 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe6700c4580 msgr2=0x7fe6701691a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.220 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.106+0000 7fe66ffff700 1 --2- 172.21.15.195:0/1938617891 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe6700c4580 0x7fe6701691a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.220 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.106+0000 7fe66ffff700 1 -- 172.21.15.195:0/1938617891 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe664009c30 con 0x7fe6700c41b0 2022-06-27T11:47:59.220 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.106+0000 7fe66ffff700 1 --2- 172.21.15.195:0/1938617891 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe6700c41b0 0x7fe670175440 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe66000c470 tx=0x7fe660007d80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:59.221 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.107+0000 7fe66cff9700 1 -- 172.21.15.195:0/1938617891 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe66000ce30 con 0x7fe6700c41b0 2022-06-27T11:47:59.221 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.109+0000 7fdb37fff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdb380c37b0 0x7fdb3814f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.221 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.109+0000 7fdb37fff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdb380c37b0 0x7fdb3814f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41422/0 (socket says 172.21.15.195:41422) 2022-06-27T11:47:59.222 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.109+0000 7fdb37fff700 1 -- 172.21.15.195:0/3470242251 learned_addr learned my addr 172.21.15.195:0/3470242251 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:47:59.222 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.107+0000 7fe66cff9700 1 -- 172.21.15.195:0/1938617891 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6600050d0 con 0x7fe6700c41b0 2022-06-27T11:47:59.223 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.107+0000 7fe67720e700 1 -- 172.21.15.195:0/1938617891 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6700c7000 con 0x7fe6700c41b0 2022-06-27T11:47:59.223 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.107+0000 7fe66cff9700 1 -- 172.21.15.195:0/1938617891 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe660011590 con 0x7fe6700c41b0 2022-06-27T11:47:59.223 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.107+0000 7fe67720e700 1 -- 172.21.15.195:0/1938617891 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6700c7e30 con 0x7fe6700c41b0 2022-06-27T11:47:59.223 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.108+0000 7fe66cff9700 1 -- 172.21.15.195:0/1938617891 <== mon.2 v2:172.21.15.202:3301/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fe660019030 con 0x7fe6700c41b0 2022-06-27T11:47:59.224 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.109+0000 7fe66cff9700 1 --2- 172.21.15.195:0/1938617891 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe650063dd0 0x7fe650066280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.224 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.109+0000 7fe66cff9700 1 -- 172.21.15.195:0/1938617891 <== mon.2 v2:172.21.15.202:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fe66007f800 con 0x7fe6700c41b0 2022-06-27T11:47:59.224 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.109+0000 7fdb3cb87700 1 --2- 172.21.15.195:0/3470242251 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fdb380c4580 0x7fdb380c78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.225 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.109+0000 7fdb377fe700 1 --2- 172.21.15.195:0/3470242251 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fdb380c41b0 0x7fdb3814f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.225 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.110+0000 7f58ead9d700 1 --2- 172.21.15.195:0/3177104790 >> [v2:172.21.15.202:6800/94308,v1:172.21.15.202:6801/94308] conn(0x7f58c806c820 0x7f58c806ec30 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:59.227 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.110+0000 7fdb3cb87700 1 -- 172.21.15.195:0/3470242251 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fdb380c41b0 msgr2=0x7fdb3814f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.227 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.110+0000 7fdb3cb87700 1 --2- 172.21.15.195:0/3470242251 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fdb380c41b0 0x7fdb3814f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.227 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.110+0000 7fdb3cb87700 1 -- 172.21.15.195:0/3470242251 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdb380c37b0 msgr2=0x7fdb3814f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.228 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.110+0000 7fdb3cb87700 1 --2- 172.21.15.195:0/3470242251 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdb380c37b0 0x7fdb3814f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.228 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.110+0000 7fdb3cb87700 1 -- 172.21.15.195:0/3470242251 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb2c008430 con 0x7fdb380c4580 2022-06-27T11:47:59.228 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.110+0000 7fdb3cb87700 1 --2- 172.21.15.195:0/3470242251 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fdb380c4580 0x7fdb380c78b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fdb3000b870 tx=0x7fdb30009290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:59.228 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.110+0000 7fdb1ffff700 1 -- 172.21.15.195:0/3470242251 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb30010680 con 0x7fdb380c4580 2022-06-27T11:47:59.228 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.110+0000 7fdb1ffff700 1 -- 172.21.15.195:0/3470242251 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb30010800 con 0x7fdb380c4580 2022-06-27T11:47:59.229 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.110+0000 7fdb1ffff700 1 -- 172.21.15.195:0/3470242251 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb30010bc0 con 0x7fdb380c4580 2022-06-27T11:47:59.229 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.110+0000 7fdb3e5ea700 1 -- 172.21.15.195:0/3470242251 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb381496c0 con 0x7fdb380c4580 2022-06-27T11:47:59.229 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.110+0000 7fdb3e5ea700 1 -- 172.21.15.195:0/3470242251 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb380c5520 con 0x7fdb380c4580 2022-06-27T11:47:59.229 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.112+0000 7fdb3e5ea700 1 -- 172.21.15.195:0/3470242251 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fdb3814eaf0 con 0x7fdb380c4580 2022-06-27T11:47:59.229 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.112+0000 7fdb1ffff700 1 -- 172.21.15.195:0/3470242251 <== mon.2 v2:172.21.15.202:3301/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fdb30020070 con 0x7fdb380c4580 2022-06-27T11:47:59.230 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.112+0000 7fdb1ffff700 1 --2- 172.21.15.195:0/3470242251 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fdb18063d00 0x7fdb180661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.230 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.112+0000 7fdb1ffff700 1 -- 172.21.15.195:0/3470242251 <== mon.2 v2:172.21.15.202:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fdb30080730 con 0x7fdb380c4580 2022-06-27T11:47:59.230 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.112+0000 7fdb37fff700 1 --2- 172.21.15.195:0/3470242251 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fdb18063d00 0x7fdb180661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.230 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.113+0000 7fdb1ffff700 1 --2- 172.21.15.195:0/3470242251 >> [v2:172.21.15.202:6824/94325,v1:172.21.15.202:6825/94325] conn(0x7fdb1806c6d0 0x7fdb1806eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.231 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.113+0000 7fdb1ffff700 1 -- 172.21.15.195:0/3470242251 --> [v2:172.21.15.202:6824/94325,v1:172.21.15.202:6825/94325] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdb1806f190 con 0x7fdb1806c6d0 2022-06-27T11:47:59.231 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.113+0000 7fdb1ffff700 1 -- 172.21.15.195:0/3470242251 <== mon.2 v2:172.21.15.202:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdb30082d80 con 0x7fdb380c4580 2022-06-27T11:47:59.231 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.113+0000 7fdb377fe700 1 --2- 172.21.15.195:0/3470242251 >> [v2:172.21.15.202:6824/94325,v1:172.21.15.202:6825/94325] conn(0x7fdb1806c6d0 0x7fdb1806eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.231 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.113+0000 7fdb37fff700 1 --2- 172.21.15.195:0/3470242251 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fdb18063d00 0x7fdb180661b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fdb28004e10 tx=0x7fdb28009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:59.231 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.113+0000 7fdb377fe700 1 --2- 172.21.15.195:0/3470242251 >> [v2:172.21.15.202:6824/94325,v1:172.21.15.202:6825/94325] conn(0x7fdb1806c6d0 0x7fdb1806eae0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:59.232 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.114+0000 7f58da7fc700 1 -- 172.21.15.195:0/3177104790 <== osd.4 v2:172.21.15.202:6800/94308 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24062 (crc 0 0 0) 0x7f58c806f2e0 con 0x7f58c806c820 2022-06-27T11:47:59.232 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.114+0000 7fdb1ffff700 1 -- 172.21.15.195:0/3470242251 <== osd.5 v2:172.21.15.202:6824/94325 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24062 (crc 0 0 0) 0x7fdb1806f190 con 0x7fdb1806c6d0 2022-06-27T11:47:59.233 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.110+0000 7fe674faa700 1 --2- 172.21.15.195:0/1938617891 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe650063dd0 0x7fe650066280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.233 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.110+0000 7fe67720e700 1 --2- 172.21.15.195:0/1938617891 >> [v2:172.21.15.195:6824/94232,v1:172.21.15.195:6825/94232] conn(0x7fe64c0015f0 0x7fe64c003aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.234 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.110+0000 7fe67720e700 1 -- 172.21.15.195:0/1938617891 --> [v2:172.21.15.195:6824/94232,v1:172.21.15.195:6825/94232] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe64c006a50 con 0x7fe64c0015f0 2022-06-27T11:47:59.234 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.110+0000 7fe6757ab700 1 --2- 172.21.15.195:0/1938617891 >> [v2:172.21.15.195:6824/94232,v1:172.21.15.195:6825/94232] conn(0x7fe64c0015f0 0x7fe64c003aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.234 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.111+0000 7fe6757ab700 1 --2- 172.21.15.195:0/1938617891 >> [v2:172.21.15.195:6824/94232,v1:172.21.15.195:6825/94232] conn(0x7fe64c0015f0 0x7fe64c003aa0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:59.234 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.112+0000 7fe674faa700 1 --2- 172.21.15.195:0/1938617891 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe650063dd0 0x7fe650066280 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe668001e30 tx=0x7fe668009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:59.234 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.112+0000 7fe66cff9700 1 -- 172.21.15.195:0/1938617891 <== osd.3 v2:172.21.15.195:6824/94232 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24062 (crc 0 0 0) 0x7fe64c006a50 con 0x7fe64c0015f0 2022-06-27T11:47:59.235 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.116+0000 7fcda7d57700 1 -- 172.21.15.195:0/1338727304 --> [v2:172.21.15.202:6808/94310,v1:172.21.15.202:6810/94310] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fcd7c002bf0 con 0x7fcd8006c6d0 2022-06-27T11:47:59.235 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.117+0000 7fcd8dffb700 1 -- 172.21.15.195:0/1338727304 <== osd.7 v2:172.21.15.202:6808/94310 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fcd7c002bf0 con 0x7fcd8006c6d0 2022-06-27T11:47:59.236 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.117+0000 7fcda7d57700 1 -- 172.21.15.195:0/1338727304 >> [v2:172.21.15.202:6808/94310,v1:172.21.15.202:6810/94310] conn(0x7fcd8006c6d0 msgr2=0x7fcd8006eae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.236 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.117+0000 7fcda7d57700 1 --2- 172.21.15.195:0/1338727304 >> [v2:172.21.15.202:6808/94310,v1:172.21.15.202:6810/94310] conn(0x7fcd8006c6d0 0x7fcd8006eae0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.237 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.117+0000 7fcda7d57700 1 -- 172.21.15.195:0/1338727304 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcd80063d00 msgr2=0x7fcd800661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.237 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.117+0000 7fcda7d57700 1 --2- 172.21.15.195:0/1338727304 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcd80063d00 0x7fcd800661b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fcd980014a0 tx=0x7fcd98007d50 comp rx=0 tx=0).stop 2022-06-27T11:47:59.237 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.117+0000 7fcda7d57700 1 -- 172.21.15.195:0/1338727304 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcda00c9bb0 msgr2=0x7fcda006d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.237 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.117+0000 7fcda7d57700 1 --2- 172.21.15.195:0/1338727304 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcda00c9bb0 0x7fcda006d650 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fcd90015000 tx=0x7fcd90016000 comp rx=0 tx=0).stop 2022-06-27T11:47:59.237 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.117+0000 7fcda62f4700 1 -- 172.21.15.195:0/1338727304 reap_dead start 2022-06-27T11:47:59.238 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.117+0000 7fcda7d57700 1 -- 172.21.15.195:0/1338727304 shutdown_connections 2022-06-27T11:47:59.238 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.117+0000 7fcda7d57700 1 -- 172.21.15.195:0/1338727304 >> 172.21.15.195:0/1338727304 conn(0x7fcda00ccb80 msgr2=0x7fcda0155b10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:59.239 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.124+0000 7fe4cce65700 1 Processor -- start 2022-06-27T11:47:59.239 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.124+0000 7fe4cce65700 1 -- start start 2022-06-27T11:47:59.240 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.124+0000 7fe4cce65700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe4c806e850 0x7fe4c81570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.240 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.124+0000 7fe4cce65700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fe4c80c34a0 0x7fe4c80c2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.240 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.125+0000 7fe4cce65700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4c806fba0 con 0x7fe4c806fef0 2022-06-27T11:47:59.240 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.125+0000 7fe4cce65700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4c81575e0 con 0x7fe4c80c34a0 2022-06-27T11:47:59.240 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.125+0000 7fe4cce65700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c806f5e0 con 0x7fe4c806e850 2022-06-27T11:47:59.241 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.125+0000 7fe4c6d9d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fe4c80c34a0 0x7fe4c80c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.241 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.125+0000 7fe4c6d9d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fe4c80c34a0 0x7fe4c80c2920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41438/0 (socket says 172.21.15.195:41438) 2022-06-27T11:47:59.241 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.125+0000 7fe4c6d9d700 1 -- 172.21.15.195:0/1991380206 learned_addr learned my addr 172.21.15.195:0/1991380206 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:47:59.241 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.125+0000 7fe4c6d9d700 1 -- 172.21.15.195:0/1991380206 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe4c806e850 msgr2=0x7fe4c81570a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:47:59.241 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.125+0000 7fe4c6d9d700 1 --2- 172.21.15.195:0/1991380206 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe4c806e850 0x7fe4c81570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.242 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.125+0000 7fe4c6d9d700 1 -- 172.21.15.195:0/1991380206 >> v1:172.21.15.195:6789/0 conn(0x7fe4c806fef0 legacy=0x7fe4c8069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:47:59.242 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.125+0000 7fe4c6d9d700 1 -- 172.21.15.195:0/1991380206 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4c806f8c0 con 0x7fe4c80c34a0 2022-06-27T11:47:59.242 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.125+0000 7fe4c6d9d700 1 --2- 172.21.15.195:0/1991380206 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe4c80c34a0 0x7fe4c80c2920 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe4c0009f20 tx=0x7fe4c0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5b96b124965e062 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:59.243 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.127+0000 7fcda7d57700 1 -- 172.21.15.195:0/1338727304 shutdown_connections 2022-06-27T11:47:59.243 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.127+0000 7fcda7d57700 1 -- 172.21.15.195:0/1338727304 wait complete. 2022-06-27T11:47:59.244 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.131+0000 7fe9d7435700 1 -- 172.21.15.195:0/1080953889 --> [v2:172.21.15.195:6808/94223,v1:172.21.15.195:6809/94223] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe9d00c3970 con 0x7fe9b006c6d0 2022-06-27T11:47:59.244 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.131+0000 7fe9bd7fa700 1 -- 172.21.15.195:0/1080953889 <== osd.1 v2:172.21.15.195:6808/94223 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fe9d00c3970 con 0x7fe9b006c6d0 2022-06-27T11:47:59.244 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.132+0000 7fe9d7435700 1 -- 172.21.15.195:0/1080953889 >> [v2:172.21.15.195:6808/94223,v1:172.21.15.195:6809/94223] conn(0x7fe9b006c6d0 msgr2=0x7fe9b006eae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.245 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.132+0000 7fe9d7435700 1 --2- 172.21.15.195:0/1080953889 >> [v2:172.21.15.195:6808/94223,v1:172.21.15.195:6809/94223] conn(0x7fe9b006c6d0 0x7fe9b006eae0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.245 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.134+0000 7f561442f700 1 -- 172.21.15.195:0/3650733220 --> [v2:172.21.15.202:6809/94309,v1:172.21.15.202:6811/94309] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f560c063af0 con 0x7f55f406c820 2022-06-27T11:47:59.246 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.134+0000 7f56027fc700 1 -- 172.21.15.195:0/3650733220 <== osd.6 v2:172.21.15.202:6809/94309 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f560c063af0 con 0x7f55f406c820 2022-06-27T11:47:59.246 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.135+0000 7f561442f700 1 -- 172.21.15.195:0/3650733220 >> [v2:172.21.15.202:6809/94309,v1:172.21.15.202:6811/94309] conn(0x7f55f406c820 msgr2=0x7f55f406ec30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.247 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.135+0000 7f561442f700 1 --2- 172.21.15.195:0/3650733220 >> [v2:172.21.15.202:6809/94309,v1:172.21.15.202:6811/94309] conn(0x7f55f406c820 0x7f55f406ec30 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.247 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.135+0000 7f561442f700 1 -- 172.21.15.195:0/3650733220 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f55f4063e50 msgr2=0x7f55f4066300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.247 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.135+0000 7f561442f700 1 --2- 172.21.15.195:0/3650733220 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f55f4063e50 0x7f55f4066300 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5608000c00 tx=0x7f5608007b20 comp rx=0 tx=0).stop 2022-06-27T11:47:59.247 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.135+0000 7f561442f700 1 -- 172.21.15.195:0/3650733220 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f560c06a0c0 msgr2=0x7f560c06e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.247 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.135+0000 7f561442f700 1 --2- 172.21.15.195:0/3650733220 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f560c06a0c0 0x7f560c06e650 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f55fc009ef0 tx=0x7f55fc018000 comp rx=0 tx=0).stop 2022-06-27T11:47:59.248 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.135+0000 7f56129cc700 1 -- 172.21.15.195:0/3650733220 reap_dead start 2022-06-27T11:47:59.248 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.136+0000 7f249c93b700 1 -- 172.21.15.195:0/2103073482 --> [v2:172.21.15.195:6800/94221,v1:172.21.15.195:6801/94221] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2498065d50 con 0x7f249814eee0 2022-06-27T11:47:59.249 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.137+0000 7f24867fc700 1 -- 172.21.15.195:0/2103073482 <== osd.2 v2:172.21.15.195:6800/94221 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f2498065d50 con 0x7f249814eee0 2022-06-27T11:47:59.249 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.137+0000 7f249c93b700 1 -- 172.21.15.195:0/2103073482 >> [v2:172.21.15.195:6800/94221,v1:172.21.15.195:6801/94221] conn(0x7f249814eee0 msgr2=0x7f249814f2b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.250 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.137+0000 7f249c93b700 1 --2- 172.21.15.195:0/2103073482 >> [v2:172.21.15.195:6800/94221,v1:172.21.15.195:6801/94221] conn(0x7f249814eee0 0x7f249814f2b0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.250 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.137+0000 7f249c93b700 1 -- 172.21.15.195:0/2103073482 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2474063e80 msgr2=0x7f2474066330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.250 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.137+0000 7f249c93b700 1 --2- 172.21.15.195:0/2103073482 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2474063e80 0x7f2474066330 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2488007fd0 tx=0x7f2488007890 comp rx=0 tx=0).stop 2022-06-27T11:47:59.250 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.137+0000 7f249c93b700 1 -- 172.21.15.195:0/2103073482 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f24980c3d70 msgr2=0x7f24980bde60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.251 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.137+0000 7f249c93b700 1 --2- 172.21.15.195:0/2103073482 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f24980c3d70 0x7f24980bde60 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f248c009e40 tx=0x7f248c017000 comp rx=0 tx=0).stop 2022-06-27T11:47:59.251 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.137+0000 7f2496d9d700 1 -- 172.21.15.195:0/2103073482 reap_dead start 2022-06-27T11:47:59.251 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.137+0000 7f561442f700 1 -- 172.21.15.195:0/3650733220 shutdown_connections 2022-06-27T11:47:59.252 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.137+0000 7f561442f700 1 -- 172.21.15.195:0/3650733220 >> 172.21.15.195:0/3650733220 conn(0x7f560c0c9b00 msgr2=0x7f560c150460 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:59.252 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.138+0000 7f561442f700 1 -- 172.21.15.195:0/3650733220 shutdown_connections 2022-06-27T11:47:59.252 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.138+0000 7f561442f700 1 -- 172.21.15.195:0/3650733220 wait complete. 2022-06-27T11:47:59.253 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.134+0000 7fe4c559a700 1 -- 172.21.15.195:0/1991380206 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4c000e070 con 0x7fe4c80c34a0 2022-06-27T11:47:59.253 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.134+0000 7fe4c559a700 1 -- 172.21.15.195:0/1991380206 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4c000bad0 con 0x7fe4c80c34a0 2022-06-27T11:47:59.253 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.134+0000 7fe4c559a700 1 -- 172.21.15.195:0/1991380206 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4c00092e0 con 0x7fe4c80c34a0 2022-06-27T11:47:59.253 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.135+0000 7fe4cce65700 1 -- 172.21.15.195:0/1991380206 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe4c80c34a0 msgr2=0x7fe4c80c2920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.254 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.135+0000 7fe4cce65700 1 --2- 172.21.15.195:0/1991380206 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe4c80c34a0 0x7fe4c80c2920 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe4c0009f20 tx=0x7fe4c0009840 comp rx=0 tx=0).stop 2022-06-27T11:47:59.254 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.135+0000 7fe4cce65700 1 -- 172.21.15.195:0/1991380206 shutdown_connections 2022-06-27T11:47:59.254 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.135+0000 7fe4cce65700 1 --2- 172.21.15.195:0/1991380206 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe4c80c34a0 0x7fe4c80c2920 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.254 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.135+0000 7fe4cce65700 1 --2- 172.21.15.195:0/1991380206 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe4c806e850 0x7fe4c81570a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.255 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.135+0000 7fe4cce65700 1 -- 172.21.15.195:0/1991380206 >> 172.21.15.195:0/1991380206 conn(0x7fe4c80c6350 msgr2=0x7fe4c80c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:59.255 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.135+0000 7fe4cce65700 1 -- 172.21.15.195:0/1991380206 shutdown_connections 2022-06-27T11:47:59.255 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.136+0000 7fe4cce65700 1 -- 172.21.15.195:0/1991380206 wait complete. 2022-06-27T11:47:59.255 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.137+0000 7fe4cce65700 1 Processor -- start 2022-06-27T11:47:59.255 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.137+0000 7fe4cce65700 1 -- start start 2022-06-27T11:47:59.256 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.138+0000 7fe4cce65700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe4c806e850 0x7fe4c8136940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.256 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.138+0000 7fe4cce65700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe4c80c34a0 0x7fe4c8137600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.256 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.141+0000 7fe9d7435700 1 -- 172.21.15.195:0/1080953889 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe9b0063d80 msgr2=0x7fe9b0066230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.257 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.141+0000 7fe9d7435700 1 --2- 172.21.15.195:0/1080953889 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe9b0063d80 0x7fe9b0066230 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe9d00c9a30 tx=0x7fe9c0009040 comp rx=0 tx=0).stop 2022-06-27T11:47:59.257 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.141+0000 7fe9d7435700 1 -- 172.21.15.195:0/1080953889 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe9d00c2610 msgr2=0x7fe9d0158c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.257 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.141+0000 7fe9d7435700 1 --2- 172.21.15.195:0/1080953889 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe9d00c2610 0x7fe9d0158c40 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe9cc00c4b0 tx=0x7fe9cc007c70 comp rx=0 tx=0).stop 2022-06-27T11:47:59.257 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.141+0000 7fe9d59d2700 1 -- 172.21.15.195:0/1080953889 reap_dead start 2022-06-27T11:47:59.258 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.141+0000 7fe9d7435700 1 -- 172.21.15.195:0/1080953889 shutdown_connections 2022-06-27T11:47:59.258 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.141+0000 7fe9d7435700 1 -- 172.21.15.195:0/1080953889 >> 172.21.15.195:0/1080953889 conn(0x7fe9d00c8740 msgr2=0x7fe9d0153400 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:59.259 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.138+0000 7fe4cce65700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe4c8137b40 0x7fe4c813ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.259 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.138+0000 7fe4cce65700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c81564d0 con 0x7fe4c8137b40 2022-06-27T11:47:59.259 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.138+0000 7fe4cce65700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c8136e80 con 0x7fe4c80c34a0 2022-06-27T11:47:59.259 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.138+0000 7fe4cce65700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c813c750 con 0x7fe4c806e850 2022-06-27T11:47:59.260 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.138+0000 7fe4c5d9b700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe4c80c34a0 0x7fe4c8137600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.260 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.139+0000 7fe4c659c700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe4c806e850 0x7fe4c8136940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.260 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.139+0000 7fe4c5d9b700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe4c80c34a0 0x7fe4c8137600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3300/0 says I am v2:172.21.15.195:39568/0 (socket says 172.21.15.195:39568) 2022-06-27T11:47:59.260 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.139+0000 7fe4c5d9b700 1 -- 172.21.15.195:0/4081562564 learned_addr learned my addr 172.21.15.195:0/4081562564 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:47:59.260 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.139+0000 7fe4c6d9d700 1 --2- 172.21.15.195:0/4081562564 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe4c8137b40 0x7fe4c813ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.261 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.139+0000 7fe4c5d9b700 1 -- 172.21.15.195:0/4081562564 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe4c806e850 msgr2=0x7fe4c8136940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.261 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.139+0000 7fe4c5d9b700 1 --2- 172.21.15.195:0/4081562564 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe4c806e850 0x7fe4c8136940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.261 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.139+0000 7fe4c5d9b700 1 -- 172.21.15.195:0/4081562564 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe4c8137b40 msgr2=0x7fe4c813ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.261 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.139+0000 7fe4c5d9b700 1 --2- 172.21.15.195:0/4081562564 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe4c8137b40 0x7fe4c813ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.262 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.139+0000 7fe4c5d9b700 1 -- 172.21.15.195:0/4081562564 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4c0009ae0 con 0x7fe4c80c34a0 2022-06-27T11:47:59.262 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.139+0000 7fe4c5d9b700 1 --2- 172.21.15.195:0/4081562564 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe4c80c34a0 0x7fe4c8137600 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc00c4a0 tx=0x7fe4bc007da0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:59.262 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.139+0000 7fe4b67fc700 1 -- 172.21.15.195:0/4081562564 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4bc00cea0 con 0x7fe4c80c34a0 2022-06-27T11:47:59.262 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.139+0000 7fe4b67fc700 1 -- 172.21.15.195:0/4081562564 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4bc0052e0 con 0x7fe4c80c34a0 2022-06-27T11:47:59.263 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.139+0000 7fe4cce65700 1 -- 172.21.15.195:0/4081562564 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4c8155c60 con 0x7fe4c80c34a0 2022-06-27T11:47:59.263 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.139+0000 7fe4b67fc700 1 -- 172.21.15.195:0/4081562564 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4bc003750 con 0x7fe4c80c34a0 2022-06-27T11:47:59.263 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.139+0000 7fe4cce65700 1 -- 172.21.15.195:0/4081562564 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4c8154c50 con 0x7fe4c80c34a0 2022-06-27T11:47:59.263 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.141+0000 7fe4cce65700 1 -- 172.21.15.195:0/4081562564 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe4c80722c0 con 0x7fe4c80c34a0 2022-06-27T11:47:59.264 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.141+0000 7fe9d7435700 1 -- 172.21.15.195:0/1080953889 shutdown_connections 2022-06-27T11:47:59.265 INFO:teuthology.orchestra.run.smithi195.stdout:55834574853 2022-06-27T11:47:59.265 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2022-06-27T11:47:59.266 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.142+0000 7f249c93b700 1 -- 172.21.15.195:0/2103073482 shutdown_connections 2022-06-27T11:47:59.266 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.142+0000 7f249c93b700 1 -- 172.21.15.195:0/2103073482 >> 172.21.15.195:0/2103073482 conn(0x7f24980c8740 msgr2=0x7f24980c8b10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:59.267 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.142+0000 7fe4b67fc700 1 -- 172.21.15.195:0/4081562564 <== mon.1 v2:172.21.15.202:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fe4bc019030 con 0x7fe4c80c34a0 2022-06-27T11:47:59.267 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.142+0000 7fe9d7435700 1 -- 172.21.15.195:0/1080953889 wait complete. 2022-06-27T11:47:59.268 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.142+0000 7fe4b67fc700 1 --2- 172.21.15.195:0/4081562564 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe4a4063d00 0x7fe4a40661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.268 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.142+0000 7fe4b67fc700 1 -- 172.21.15.195:0/4081562564 <== mon.1 v2:172.21.15.202:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fe4bc07f820 con 0x7fe4c80c34a0 2022-06-27T11:47:59.268 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.143+0000 7fe4b67fc700 1 --2- 172.21.15.195:0/4081562564 >> [v2:172.21.15.195:6813/94217,v1:172.21.15.195:6815/94217] conn(0x7fe4a406c6d0 0x7fe4a406eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.268 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.143+0000 7fe4b67fc700 1 -- 172.21.15.195:0/4081562564 --> [v2:172.21.15.195:6813/94217,v1:172.21.15.195:6815/94217] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe4a406f190 con 0x7fe4a406c6d0 2022-06-27T11:47:59.269 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.143+0000 7fe4b67fc700 1 -- 172.21.15.195:0/4081562564 <== mon.1 v2:172.21.15.202:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe4bc084220 con 0x7fe4c80c34a0 2022-06-27T11:47:59.269 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.142+0000 7f58f0922700 1 -- 172.21.15.195:0/3177104790 --> [v2:172.21.15.202:6800/94308,v1:172.21.15.202:6801/94308] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f58ec0b8130 con 0x7f58c806c820 2022-06-27T11:47:59.269 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.143+0000 7fe4c6d9d700 1 --2- 172.21.15.195:0/4081562564 >> [v2:172.21.15.195:6813/94217,v1:172.21.15.195:6815/94217] conn(0x7fe4a406c6d0 0x7fe4a406eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.270 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.143+0000 7f58da7fc700 1 -- 172.21.15.195:0/3177104790 <== osd.4 v2:172.21.15.202:6800/94308 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f58ec0b8130 con 0x7f58c806c820 2022-06-27T11:47:59.271 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.143+0000 7fe4c6d9d700 1 --2- 172.21.15.195:0/4081562564 >> [v2:172.21.15.195:6813/94217,v1:172.21.15.195:6815/94217] conn(0x7fe4a406c6d0 0x7fe4a406eae0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:59.271 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.143+0000 7fe4c659c700 1 --2- 172.21.15.195:0/4081562564 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe4a4063d00 0x7fe4a40661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.271 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.144+0000 7fe4c659c700 1 --2- 172.21.15.195:0/4081562564 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe4a4063d00 0x7fe4a40661b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe4b8009a00 tx=0x7fe4b8006f50 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:59.272 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.145+0000 7fe4b67fc700 1 -- 172.21.15.195:0/4081562564 <== osd.0 v2:172.21.15.195:6813/94217 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24062 (crc 0 0 0) 0x7fe4a406f190 con 0x7fe4a406c6d0 2022-06-27T11:47:59.273 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.146+0000 7f58f0922700 1 -- 172.21.15.195:0/3177104790 >> [v2:172.21.15.202:6800/94308,v1:172.21.15.202:6801/94308] conn(0x7f58c806c820 msgr2=0x7f58c806ec30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.273 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.146+0000 7f58f0922700 1 --2- 172.21.15.195:0/3177104790 >> [v2:172.21.15.202:6800/94308,v1:172.21.15.202:6801/94308] conn(0x7f58c806c820 0x7f58c806ec30 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.273 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.146+0000 7f58f0922700 1 -- 172.21.15.195:0/3177104790 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f58c8063e50 msgr2=0x7f58c8066300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.274 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.146+0000 7f58f0922700 1 --2- 172.21.15.195:0/3177104790 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f58c8063e50 0x7f58c8066300 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f58e000c110 tx=0x7f58e0015000 comp rx=0 tx=0).stop 2022-06-27T11:47:59.274 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.146+0000 7f58f0922700 1 -- 172.21.15.195:0/3177104790 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f58d0003670 msgr2=0x7f58ec0c6820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.274 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.146+0000 7f58f0922700 1 --2- 172.21.15.195:0/3177104790 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f58d0003670 0x7f58ec0c6820 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f58ec0c2d10 tx=0x7f58dc008040 comp rx=0 tx=0).stop 2022-06-27T11:47:59.274 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.146+0000 7f58ead9d700 1 -- 172.21.15.195:0/3177104790 reap_dead start 2022-06-27T11:47:59.275 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.146+0000 7f58f0922700 1 -- 172.21.15.195:0/3177104790 shutdown_connections 2022-06-27T11:47:59.275 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.146+0000 7f58f0922700 1 -- 172.21.15.195:0/3177104790 >> 172.21.15.195:0/3177104790 conn(0x7f58ec0c8740 msgr2=0x7f58ec0c8b10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:59.275 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.146+0000 7f58f0922700 1 -- 172.21.15.195:0/3177104790 shutdown_connections 2022-06-27T11:47:59.275 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.146+0000 7f58f0922700 1 -- 172.21.15.195:0/3177104790 wait complete. 2022-06-27T11:47:59.276 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.147+0000 7fe67720e700 1 -- 172.21.15.195:0/1938617891 --> [v2:172.21.15.195:6824/94232,v1:172.21.15.195:6825/94232] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe64c005ae0 con 0x7fe64c0015f0 2022-06-27T11:47:59.276 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.147+0000 7fe66cff9700 1 -- 172.21.15.195:0/1938617891 <== osd.3 v2:172.21.15.195:6824/94232 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fe64c005ae0 con 0x7fe64c0015f0 2022-06-27T11:47:59.277 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.148+0000 7f249c93b700 1 -- 172.21.15.195:0/2103073482 shutdown_connections 2022-06-27T11:47:59.277 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.148+0000 7fe67720e700 1 -- 172.21.15.195:0/1938617891 >> [v2:172.21.15.195:6824/94232,v1:172.21.15.195:6825/94232] conn(0x7fe64c0015f0 msgr2=0x7fe64c003aa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.277 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.148+0000 7fe67720e700 1 --2- 172.21.15.195:0/1938617891 >> [v2:172.21.15.195:6824/94232,v1:172.21.15.195:6825/94232] conn(0x7fe64c0015f0 0x7fe64c003aa0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.278 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.149+0000 7f249c93b700 1 -- 172.21.15.195:0/2103073482 wait complete. 2022-06-27T11:47:59.279 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.151+0000 7fe67720e700 1 -- 172.21.15.195:0/1938617891 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe650063dd0 msgr2=0x7fe650066280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.279 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.151+0000 7fe67720e700 1 --2- 172.21.15.195:0/1938617891 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe650063dd0 0x7fe650066280 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe668001e30 tx=0x7fe668009040 comp rx=0 tx=0).stop 2022-06-27T11:47:59.279 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.151+0000 7fe67720e700 1 -- 172.21.15.195:0/1938617891 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe6700c41b0 msgr2=0x7fe670175440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.279 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.151+0000 7fe67720e700 1 --2- 172.21.15.195:0/1938617891 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe6700c41b0 0x7fe670175440 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe66000c470 tx=0x7fe660007d80 comp rx=0 tx=0).stop 2022-06-27T11:47:59.280 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.151+0000 7fe6757ab700 1 -- 172.21.15.195:0/1938617891 reap_dead start 2022-06-27T11:47:59.280 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.151+0000 7fe67720e700 1 -- 172.21.15.195:0/1938617891 shutdown_connections 2022-06-27T11:47:59.280 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.151+0000 7fe67720e700 1 -- 172.21.15.195:0/1938617891 >> 172.21.15.195:0/1938617891 conn(0x7fe6700c8740 msgr2=0x7fe6701544c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:59.280 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.151+0000 7fe67720e700 1 -- 172.21.15.195:0/1938617891 shutdown_connections 2022-06-27T11:47:59.281 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.151+0000 7fe67720e700 1 -- 172.21.15.195:0/1938617891 wait complete. 2022-06-27T11:47:59.281 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.154+0000 7fdb3e5ea700 1 -- 172.21.15.195:0/3470242251 --> [v2:172.21.15.202:6824/94325,v1:172.21.15.202:6825/94325] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fdb380722c0 con 0x7fdb1806c6d0 2022-06-27T11:47:59.282 INFO:teuthology.orchestra.run.smithi195.stdout:55834574853 2022-06-27T11:47:59.283 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2022-06-27T11:47:59.283 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.155+0000 7fdb1ffff700 1 -- 172.21.15.195:0/3470242251 <== osd.5 v2:172.21.15.202:6824/94325 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fdb380722c0 con 0x7fdb1806c6d0 2022-06-27T11:47:59.284 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.155+0000 7fdb3e5ea700 1 -- 172.21.15.195:0/3470242251 >> [v2:172.21.15.202:6824/94325,v1:172.21.15.202:6825/94325] conn(0x7fdb1806c6d0 msgr2=0x7fdb1806eae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.285 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.155+0000 7fdb3e5ea700 1 --2- 172.21.15.195:0/3470242251 >> [v2:172.21.15.202:6824/94325,v1:172.21.15.202:6825/94325] conn(0x7fdb1806c6d0 0x7fdb1806eae0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.285 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.155+0000 7fdb3e5ea700 1 -- 172.21.15.195:0/3470242251 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fdb18063d00 msgr2=0x7fdb180661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.285 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.155+0000 7fdb3e5ea700 1 --2- 172.21.15.195:0/3470242251 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fdb18063d00 0x7fdb180661b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fdb28004e10 tx=0x7fdb28009040 comp rx=0 tx=0).stop 2022-06-27T11:47:59.285 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.155+0000 7fdb3e5ea700 1 -- 172.21.15.195:0/3470242251 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fdb380c4580 msgr2=0x7fdb380c78b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.285 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.155+0000 7fdb3e5ea700 1 --2- 172.21.15.195:0/3470242251 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fdb380c4580 0x7fdb380c78b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fdb3000b870 tx=0x7fdb30009290 comp rx=0 tx=0).stop 2022-06-27T11:47:59.286 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.155+0000 7fdb3cb87700 1 -- 172.21.15.195:0/3470242251 reap_dead start 2022-06-27T11:47:59.286 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.155+0000 7fdb3e5ea700 1 -- 172.21.15.195:0/3470242251 shutdown_connections 2022-06-27T11:47:59.286 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.155+0000 7fdb3e5ea700 1 -- 172.21.15.195:0/3470242251 >> 172.21.15.195:0/3470242251 conn(0x7fdb380c8740 msgr2=0x7fdb38152bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:59.286 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.155+0000 7fdb3e5ea700 1 -- 172.21.15.195:0/3470242251 shutdown_connections 2022-06-27T11:47:59.287 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.155+0000 7fdb3e5ea700 1 -- 172.21.15.195:0/3470242251 wait complete. 2022-06-27T11:47:59.287 INFO:teuthology.orchestra.run.smithi195.stdout:55834574853 2022-06-27T11:47:59.288 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2022-06-27T11:47:59.289 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.166+0000 7fe4cce65700 1 -- 172.21.15.195:0/4081562564 --> [v2:172.21.15.195:6813/94217,v1:172.21.15.195:6815/94217] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe4c804c470 con 0x7fe4a406c6d0 2022-06-27T11:47:59.289 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.167+0000 7fe4b67fc700 1 -- 172.21.15.195:0/4081562564 <== osd.0 v2:172.21.15.195:6813/94217 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fe4c804c470 con 0x7fe4a406c6d0 2022-06-27T11:47:59.290 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.167+0000 7fe4cce65700 1 -- 172.21.15.195:0/4081562564 >> [v2:172.21.15.195:6813/94217,v1:172.21.15.195:6815/94217] conn(0x7fe4a406c6d0 msgr2=0x7fe4a406eae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.290 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.167+0000 7fe4cce65700 1 --2- 172.21.15.195:0/4081562564 >> [v2:172.21.15.195:6813/94217,v1:172.21.15.195:6815/94217] conn(0x7fe4a406c6d0 0x7fe4a406eae0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.291 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.167+0000 7fe4cce65700 1 -- 172.21.15.195:0/4081562564 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe4a4063d00 msgr2=0x7fe4a40661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.291 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.167+0000 7fe4cce65700 1 --2- 172.21.15.195:0/4081562564 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe4a4063d00 0x7fe4a40661b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe4b8009a00 tx=0x7fe4b8006f50 comp rx=0 tx=0).stop 2022-06-27T11:47:59.291 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.167+0000 7fe4cce65700 1 -- 172.21.15.195:0/4081562564 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe4c80c34a0 msgr2=0x7fe4c8137600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.291 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.167+0000 7fe4cce65700 1 --2- 172.21.15.195:0/4081562564 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe4c80c34a0 0x7fe4c8137600 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc00c4a0 tx=0x7fe4bc007da0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.291 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.168+0000 7fe4c6d9d700 1 -- 172.21.15.195:0/4081562564 reap_dead start 2022-06-27T11:47:59.292 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.168+0000 7fe4cce65700 1 -- 172.21.15.195:0/4081562564 shutdown_connections 2022-06-27T11:47:59.292 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.168+0000 7fe4cce65700 1 -- 172.21.15.195:0/4081562564 >> 172.21.15.195:0/4081562564 conn(0x7fe4c80c6350 msgr2=0x7fe4c80c2820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:59.292 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.168+0000 7fe4cce65700 1 -- 172.21.15.195:0/4081562564 shutdown_connections 2022-06-27T11:47:59.292 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.168+0000 7fe4cce65700 1 -- 172.21.15.195:0/4081562564 wait complete. 2022-06-27T11:47:59.293 INFO:teuthology.orchestra.run.smithi195.stdout:55834574853 2022-06-27T11:47:59.294 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2022-06-27T11:47:59.295 INFO:teuthology.orchestra.run.smithi195.stdout:55834574854 2022-06-27T11:47:59.295 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2022-06-27T11:47:59.296 INFO:teuthology.orchestra.run.smithi195.stdout:55834574853 2022-06-27T11:47:59.297 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2022-06-27T11:47:59.298 INFO:teuthology.orchestra.run.smithi195.stdout:55834574854 2022-06-27T11:47:59.298 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2022-06-27T11:47:59.299 INFO:teuthology.orchestra.run.smithi195.stdout:55834574853 2022-06-27T11:47:59.300 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2022-06-27T11:47:59.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.531+0000 7f92518ae700 1 Processor -- start 2022-06-27T11:47:59.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.531+0000 7f92518ae700 1 -- start start 2022-06-27T11:47:59.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.532+0000 7f92518ae700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f924c0c4580 0x7f924c0c3270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.532+0000 7f92518ae700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f924c0c37b0 0x7f924c0c0a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.534 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.532+0000 7f92518ae700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f924c155b50 con 0x7f924c0c41b0 2022-06-27T11:47:59.534 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.533+0000 7f92518ae700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f924c1527a0 con 0x7f924c0c4580 2022-06-27T11:47:59.534 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.533+0000 7f92518ae700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f924c0c3c30 con 0x7f924c0c37b0 2022-06-27T11:47:59.534 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.533+0000 7f924b7fe700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f924c0c37b0 0x7f924c0c0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.535 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.533+0000 7f924b7fe700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f924c0c37b0 0x7f924c0c0a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3301/0 says I am v2:172.21.15.195:34210/0 (socket says 172.21.15.195:34210) 2022-06-27T11:47:59.535 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.533+0000 7f924b7fe700 1 -- 172.21.15.195:0/2234547880 learned_addr learned my addr 172.21.15.195:0/2234547880 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:47:59.535 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.533+0000 7f924a7fc700 1 --2- 172.21.15.195:0/2234547880 >> v2:172.21.15.195:3300/0 conn(0x7f924c0c4580 0x7f924c0c3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.535 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.533+0000 7f924b7fe700 1 -- 172.21.15.195:0/2234547880 >> v2:172.21.15.195:3300/0 conn(0x7f924c0c4580 msgr2=0x7f924c0c3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:47:59.536 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.533+0000 7f924b7fe700 1 --2- 172.21.15.195:0/2234547880 >> v2:172.21.15.195:3300/0 conn(0x7f924c0c4580 0x7f924c0c3270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.536 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.533+0000 7f924b7fe700 1 -- 172.21.15.195:0/2234547880 >> v1:172.21.15.202:6789/0 conn(0x7f924c0c41b0 legacy=0x7f924c152bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.536 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.533+0000 7f924b7fe700 1 -- 172.21.15.195:0/2234547880 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f924c155df0 con 0x7f924c0c37b0 2022-06-27T11:47:59.536 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.533+0000 7f924b7fe700 1 --2- 172.21.15.195:0/2234547880 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f924c0c37b0 0x7f924c0c0a20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f924400d1a0 tx=0x7f9244007690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec3f636a3a35bf1b server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:59.537 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.533+0000 7fe9415c3700 1 Processor -- start 2022-06-27T11:47:59.537 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.533+0000 7fe9415c3700 1 -- start start 2022-06-27T11:47:59.538 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.534+0000 7f9249ffb700 1 -- 172.21.15.195:0/2234547880 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9244010070 con 0x7f924c0c37b0 2022-06-27T11:47:59.538 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.534+0000 7f9249ffb700 1 -- 172.21.15.195:0/2234547880 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f924400dd40 con 0x7f924c0c37b0 2022-06-27T11:47:59.538 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.534+0000 7f9249ffb700 1 -- 172.21.15.195:0/2234547880 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9244007d90 con 0x7f924c0c37b0 2022-06-27T11:47:59.539 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.534+0000 7fe2466a0700 1 Processor -- start 2022-06-27T11:47:59.539 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.535+0000 7fd5412ca700 1 Processor -- start 2022-06-27T11:47:59.540 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.535+0000 7fe2466a0700 1 -- start start 2022-06-27T11:47:59.541 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.535+0000 7f92518ae700 1 -- 172.21.15.195:0/2234547880 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f924c0c37b0 msgr2=0x7f924c0c0a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.541 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.535+0000 7f92518ae700 1 --2- 172.21.15.195:0/2234547880 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f924c0c37b0 0x7f924c0c0a20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f924400d1a0 tx=0x7f9244007690 comp rx=0 tx=0).stop 2022-06-27T11:47:59.541 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.535+0000 7f92518ae700 1 -- 172.21.15.195:0/2234547880 shutdown_connections 2022-06-27T11:47:59.541 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.535+0000 7f92518ae700 1 --2- 172.21.15.195:0/2234547880 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f924c0c37b0 0x7f924c0c0a20 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.542 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.535+0000 7f92518ae700 1 --2- 172.21.15.195:0/2234547880 >> v2:172.21.15.195:3300/0 conn(0x7f924c0c4580 0x7f924c0c3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.542 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.535+0000 7f92518ae700 1 -- 172.21.15.195:0/2234547880 >> 172.21.15.195:0/2234547880 conn(0x7f924c0c8740 msgr2=0x7f924c0c8b10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:59.542 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.536+0000 7fe9415c3700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe93c0c31f0 0x7fe93c15b640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.543 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.536+0000 7fe9415c3700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fe93c0c35c0 0x7fe93c067ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.543 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.536+0000 7fe9415c3700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe93c15b1b0 con 0x7fe93c0c49b0 2022-06-27T11:47:59.543 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.536+0000 7fe9415c3700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe93c069080 con 0x7fe93c0c35c0 2022-06-27T11:47:59.543 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.536+0000 7fe9415c3700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe93c15aba0 con 0x7fe93c0c31f0 2022-06-27T11:47:59.544 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.536+0000 7fe93b7fe700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fe93c0c49b0 0x7fe93c0b7f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:47646/0 (socket says 172.21.15.195:47646) 2022-06-27T11:47:59.544 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.536+0000 7fe93b7fe700 1 -- 172.21.15.195:0/584574401 learned_addr learned my addr 172.21.15.195:0/584574401 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:47:59.544 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.536+0000 7fe2466a0700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fe24006b2e0 0x7fe2400c2610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.544 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.536+0000 7fe2466a0700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe2400c4bf0 0x7fe2400c3b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.545 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.536+0000 7fe2466a0700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe24015eab0 con 0x7fe2400c3010 2022-06-27T11:47:59.545 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.536+0000 7fe2466a0700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2400c1d50 con 0x7fe24006b2e0 2022-06-27T11:47:59.545 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.536+0000 7fe2466a0700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2400c2b50 con 0x7fe2400c4bf0 2022-06-27T11:47:59.546 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.536+0000 7fe93a7fc700 1 --2- 172.21.15.195:0/584574401 >> v2:172.21.15.202:3300/0 conn(0x7fe93c0c35c0 0x7fe93c067ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.546 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.536+0000 7fe93affd700 1 --2- 172.21.15.195:0/584574401 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe93c0c31f0 0x7fe93c15b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.547 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.535+0000 7fd5412ca700 1 -- start start 2022-06-27T11:47:59.547 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.536+0000 7fd5412ca700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd53c154f60 0x7fd53c0c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.547 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.536+0000 7fd5412ca700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd53c0c54b0 con 0x7fd53c0680b0 2022-06-27T11:47:59.547 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.536+0000 7fd5412ca700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd53c0c6050 con 0x7fd53c069050 2022-06-27T11:47:59.548 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.536+0000 7fd5412ca700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd53c15b070 con 0x7fd53c154f60 2022-06-27T11:47:59.548 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.536+0000 7fe244c3d700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fe2400c3010 0x7fe240067ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:47652/0 (socket says 172.21.15.195:47652) 2022-06-27T11:47:59.548 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.536+0000 7fe244c3d700 1 -- 172.21.15.195:0/183084986 learned_addr learned my addr 172.21.15.195:0/183084986 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:47:59.549 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.535+0000 7f03d28aa700 1 Processor -- start 2022-06-27T11:47:59.549 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.535+0000 7f03d28aa700 1 -- start start 2022-06-27T11:47:59.549 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.536+0000 7f03d28aa700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f03cc06a690 0x7f03cc15a890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.550 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.536+0000 7f03d28aa700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03cc15ae90 con 0x7f03cc06aa60 2022-06-27T11:47:59.550 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.536+0000 7f03d28aa700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03cc15b170 con 0x7f03cc06d050 2022-06-27T11:47:59.550 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.536+0000 7f03d28aa700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03cc069be0 con 0x7f03cc06a690 2022-06-27T11:47:59.550 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.536+0000 7f03cbfff700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f03cc06a690 0x7f03cc15a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.551 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.536+0000 7f03cb7fe700 1 --1- >> v1:172.21.15.202:6789/0 conn(0x7f03cc06d050 0x7f03cc0c7c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.202:6789/0 says I am v1:172.21.15.195:49522/0 (socket says 172.21.15.195:49522) 2022-06-27T11:47:59.551 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.537+0000 7f03cb7fe700 1 -- 172.21.15.195:0/4287950123 learned_addr learned my addr 172.21.15.195:0/4287950123 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:47:59.551 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.537+0000 7f03cbfff700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f03cc06a690 0x7f03cc15a890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3301/0 says I am v2:172.21.15.195:34226/0 (socket says 172.21.15.195:34226) 2022-06-27T11:47:59.551 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.536+0000 7f92518ae700 1 -- 172.21.15.195:0/2234547880 shutdown_connections 2022-06-27T11:47:59.552 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.536+0000 7fd53affd700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fd53c0680b0 0x7fd53c0c5940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:47654/0 (socket says 172.21.15.195:47654) 2022-06-27T11:47:59.552 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.536+0000 7fd53affd700 1 -- 172.21.15.195:0/2786723916 learned_addr learned my addr 172.21.15.195:0/2786723916 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:47:59.553 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.537+0000 7fe23ffff700 1 --2- 172.21.15.195:0/183084986 >> v2:172.21.15.202:3300/0 conn(0x7fe24006b2e0 0x7fe2400c2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.553 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.537+0000 7fd539ffb700 1 -- 172.21.15.195:0/2786723916 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1305191790 0 0) 0x7fd53c0c6050 con 0x7fd53c069050 2022-06-27T11:47:59.553 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.537+0000 7fd539ffb700 1 -- 172.21.15.195:0/2786723916 --> v1:172.21.15.202:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd528003430 con 0x7fd53c069050 2022-06-27T11:47:59.554 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.537+0000 7f03cbfff700 1 -- 172.21.15.195:0/4287950123 >> v1:172.21.15.202:6789/0 conn(0x7f03cc06d050 legacy=0x7f03cc0c7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.554 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.537+0000 7f03cbfff700 1 -- 172.21.15.195:0/4287950123 >> v1:172.21.15.195:6789/0 conn(0x7f03cc06aa60 legacy=0x7f03cc154700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.554 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.537+0000 7f03cbfff700 1 -- 172.21.15.195:0/4287950123 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03cc15b450 con 0x7f03cc06a690 2022-06-27T11:47:59.555 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.537+0000 7fe23ffff700 1 -- 172.21.15.195:0/183084986 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe2400c4bf0 msgr2=0x7fe2400c3b90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2022-06-27T11:47:59.556 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.537+0000 7fe23ffff700 1 --2- 172.21.15.195:0/183084986 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe2400c4bf0 0x7fe2400c3b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.556 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.537+0000 7fe23ffff700 1 -- 172.21.15.195:0/183084986 >> v1:172.21.15.195:6789/0 conn(0x7fe2400c3010 legacy=0x7fe240067ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:47:59.556 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.537+0000 7fe23ffff700 1 -- 172.21.15.195:0/183084986 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2400c35e0 con 0x7fe24006b2e0 2022-06-27T11:47:59.557 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.537+0000 7f03cbfff700 1 --2- 172.21.15.195:0/4287950123 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f03cc06a690 0x7f03cc15a890 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f03c401ae90 tx=0x7f03c401a780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4924b9d4efede90 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:59.558 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.537+0000 7fd539ffb700 1 -- 172.21.15.195:0/2786723916 <== mon.1 v1:172.21.15.202:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 889241114 0 0) 0x7fd528003430 con 0x7fd53c069050 2022-06-27T11:47:59.558 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.537+0000 7fd539ffb700 1 -- 172.21.15.195:0/2786723916 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd53c154f60 msgr2=0x7fd53c0c49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:47:59.558 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.537+0000 7fd539ffb700 1 --2- 172.21.15.195:0/2786723916 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd53c154f60 0x7fd53c0c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.559 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.537+0000 7fd539ffb700 1 -- 172.21.15.195:0/2786723916 >> v1:172.21.15.195:6789/0 conn(0x7fd53c0680b0 legacy=0x7fd53c0c5940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.559 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.537+0000 7fd539ffb700 1 -- 172.21.15.195:0/2786723916 --> v1:172.21.15.202:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd53c15b350 con 0x7fd53c069050 2022-06-27T11:47:59.559 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.537+0000 7fd539ffb700 1 -- 172.21.15.195:0/2786723916 <== mon.1 v1:172.21.15.202:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7fd5300014b0 con 0x7fd53c069050 2022-06-27T11:47:59.560 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.537+0000 7fd539ffb700 1 -- 172.21.15.195:0/2786723916 >> v1:172.21.15.202:6789/0 conn(0x7fd53c069050 legacy=0x7fd53c154850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.560 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.537+0000 7fd539ffb700 1 --2- 172.21.15.195:0/2786723916 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd528003920 0x7fd528003d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.560 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.537+0000 7fd539ffb700 1 --2- 172.21.15.195:0/2786723916 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd5280046d0 0x7fd528004b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.560 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.537+0000 7fd539ffb700 1 --2- 172.21.15.195:0/2786723916 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd528005150 0x7fd52800b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.561 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.537+0000 7fd539ffb700 1 -- 172.21.15.195:0/2786723916 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd53c15b070 con 0x7fd528005150 2022-06-27T11:47:59.561 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.537+0000 7fd539ffb700 1 -- 172.21.15.195:0/2786723916 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd52800c050 con 0x7fd528003920 2022-06-27T11:47:59.561 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.537+0000 7fd539ffb700 1 -- 172.21.15.195:0/2786723916 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd52800c210 con 0x7fd5280046d0 2022-06-27T11:47:59.561 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.537+0000 7fe23ffff700 1 --2- 172.21.15.195:0/183084986 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe24006b2e0 0x7fe2400c2610 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe2300097a0 tx=0x7fe230009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b65ec6a13723aa3 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:59.562 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.537+0000 7fd53affd700 1 --2- 172.21.15.195:0/2786723916 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd528003920 0x7fd528003d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.562 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.537+0000 7fe23f7fe700 1 -- 172.21.15.195:0/183084986 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe230004030 con 0x7fe24006b2e0 2022-06-27T11:47:59.563 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.538+0000 7fe23f7fe700 1 -- 172.21.15.195:0/183084986 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe230004190 con 0x7fe24006b2e0 2022-06-27T11:47:59.563 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.538+0000 7fe23f7fe700 1 -- 172.21.15.195:0/183084986 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe230011630 con 0x7fe24006b2e0 2022-06-27T11:47:59.563 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.537+0000 7f03caffd700 1 -- 172.21.15.195:0/4287950123 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c401f070 con 0x7f03cc06a690 2022-06-27T11:47:59.564 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.537+0000 7f03caffd700 1 -- 172.21.15.195:0/4287950123 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03c401cc90 con 0x7f03cc06a690 2022-06-27T11:47:59.564 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.537+0000 7f03caffd700 1 -- 172.21.15.195:0/4287950123 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c4023680 con 0x7f03cc06a690 2022-06-27T11:47:59.564 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.537+0000 7fd53a7fc700 1 --2- 172.21.15.195:0/2786723916 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd5280046d0 0x7fd528004b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.565 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.538+0000 7fd53affd700 1 -- 172.21.15.195:0/2786723916 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd5280046d0 msgr2=0x7fd528004b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:47:59.565 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.538+0000 7fd53affd700 1 --2- 172.21.15.195:0/2786723916 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd5280046d0 0x7fd528004b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.565 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.538+0000 7fd53affd700 1 -- 172.21.15.195:0/2786723916 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd528005150 msgr2=0x7fd52800b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:47:59.565 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.538+0000 7fd53affd700 1 --2- 172.21.15.195:0/2786723916 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd528005150 0x7fd52800b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.566 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.538+0000 7fd53affd700 1 -- 172.21.15.195:0/2786723916 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd53c15b350 con 0x7fd528003920 2022-06-27T11:47:59.566 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.536+0000 7fe939ffb700 1 -- 172.21.15.195:0/584574401 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4146075231 0 0) 0x7fe93c15b1b0 con 0x7fe93c0c49b0 2022-06-27T11:47:59.566 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.536+0000 7fe939ffb700 1 -- 172.21.15.195:0/584574401 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe928003430 con 0x7fe93c0c49b0 2022-06-27T11:47:59.567 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.536+0000 7fe93affd700 1 -- 172.21.15.195:0/584574401 >> v2:172.21.15.202:3300/0 conn(0x7fe93c0c35c0 msgr2=0x7fe93c067ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:47:59.567 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.536+0000 7fe93affd700 1 --2- 172.21.15.195:0/584574401 >> v2:172.21.15.202:3300/0 conn(0x7fe93c0c35c0 0x7fe93c067ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.567 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.536+0000 7fe93affd700 1 -- 172.21.15.195:0/584574401 >> v1:172.21.15.195:6789/0 conn(0x7fe93c0c49b0 legacy=0x7fe93c0b7f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.567 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.536+0000 7fe93affd700 1 -- 172.21.15.195:0/584574401 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe93c06b0e0 con 0x7fe93c0c31f0 2022-06-27T11:47:59.568 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.538+0000 7fd53affd700 1 --2- 172.21.15.195:0/2786723916 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd528003920 0x7fd528003d30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd534001f10 tx=0x7fd534007f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e2fa60dc3bf44e8d server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:59.568 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.538+0000 7f03d28aa700 1 -- 172.21.15.195:0/4287950123 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f03cc06a690 msgr2=0x7f03cc15a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.568 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.538+0000 7f03d28aa700 1 --2- 172.21.15.195:0/4287950123 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f03cc06a690 0x7f03cc15a890 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f03c401ae90 tx=0x7f03c401a780 comp rx=0 tx=0).stop 2022-06-27T11:47:59.569 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.537+0000 7f92518ae700 1 -- 172.21.15.195:0/2234547880 wait complete. 2022-06-27T11:47:59.569 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.538+0000 7f92518ae700 1 Processor -- start 2022-06-27T11:47:59.570 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.538+0000 7fe95fb60700 1 Processor -- start 2022-06-27T11:47:59.570 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.538+0000 7fe95fb60700 1 -- start start 2022-06-27T11:47:59.570 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.540+0000 7fe95fb60700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fe9580c6590 0x7fe95815b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.570 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.540+0000 7fe95fb60700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe95806a0c0 0x7fe958154650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.571 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.540+0000 7fe95fb60700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe958067ed0 con 0x7fe958069cf0 2022-06-27T11:47:59.571 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.540+0000 7fe95fb60700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9580c48a0 con 0x7fe9580c6590 2022-06-27T11:47:59.571 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.540+0000 7fe95fb60700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe958068120 con 0x7fe95806a0c0 2022-06-27T11:47:59.571 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.540+0000 7fe95d8fc700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fe958069cf0 0x7fe9580c4190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:47686/0 (socket says 172.21.15.195:47686) 2022-06-27T11:47:59.571 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.540+0000 7fe95d8fc700 1 -- 172.21.15.195:0/714907992 learned_addr learned my addr 172.21.15.195:0/714907992 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:47:59.572 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.540+0000 7fe95e0fd700 1 --2- 172.21.15.195:0/714907992 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe95806a0c0 0x7fe958154650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.572 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.539+0000 7f03d28aa700 1 -- 172.21.15.195:0/4287950123 shutdown_connections 2022-06-27T11:47:59.572 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.539+0000 7f03d28aa700 1 --2- 172.21.15.195:0/4287950123 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f03cc06a690 0x7f03cc15a890 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.573 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.539+0000 7f03d28aa700 1 -- 172.21.15.195:0/4287950123 >> 172.21.15.195:0/4287950123 conn(0x7f03cc063af0 msgr2=0x7f03cc063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:59.573 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.539+0000 7f03d28aa700 1 -- 172.21.15.195:0/4287950123 shutdown_connections 2022-06-27T11:47:59.573 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.539+0000 7f03d28aa700 1 -- 172.21.15.195:0/4287950123 wait complete. 2022-06-27T11:47:59.574 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.541+0000 7f03d28aa700 1 Processor -- start 2022-06-27T11:47:59.574 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.539+0000 7f92518ae700 1 -- start start 2022-06-27T11:47:59.574 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.540+0000 7f92518ae700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f924c0c37b0 0x7f924c0b70a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.575 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.540+0000 7f92518ae700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f924c0c41b0 0x7f924c0b75e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.575 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.540+0000 7f92518ae700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f924c0c4580 0x7f924c16acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.575 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.540+0000 7f92518ae700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f924c14fe70 con 0x7f924c0c41b0 2022-06-27T11:47:59.575 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.540+0000 7f92518ae700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f924c1498c0 con 0x7f924c0c4580 2022-06-27T11:47:59.576 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.540+0000 7f92518ae700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f924c14d660 con 0x7f924c0c37b0 2022-06-27T11:47:59.576 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.540+0000 7f924a7fc700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f924c0c41b0 0x7f924c0b75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.576 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.540+0000 7f924a7fc700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f924c0c41b0 0x7f924c0b75e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41492/0 (socket says 172.21.15.195:41492) 2022-06-27T11:47:59.576 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.540+0000 7f924a7fc700 1 -- 172.21.15.195:0/1059887574 learned_addr learned my addr 172.21.15.195:0/1059887574 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:47:59.576 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.540+0000 7f924b7fe700 1 --2- 172.21.15.195:0/1059887574 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f924c0c4580 0x7f924c16acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.577 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.541+0000 7f924affd700 1 --2- 172.21.15.195:0/1059887574 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f924c0c37b0 0x7f924c0b70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.577 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.538+0000 7fd539ffb700 1 -- 172.21.15.195:0/2786723916 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5340018d0 con 0x7fd528003920 2022-06-27T11:47:59.577 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.538+0000 7fd539ffb700 1 -- 172.21.15.195:0/2786723916 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd534004b30 con 0x7fd528003920 2022-06-27T11:47:59.578 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.538+0000 7fd539ffb700 1 -- 172.21.15.195:0/2786723916 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd53400c6c0 con 0x7fd528003920 2022-06-27T11:47:59.578 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.539+0000 7fd5412ca700 1 -- 172.21.15.195:0/2786723916 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd528003920 msgr2=0x7fd528003d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.578 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.539+0000 7fd5412ca700 1 --2- 172.21.15.195:0/2786723916 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd528003920 0x7fd528003d30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd534001f10 tx=0x7fd534007f90 comp rx=0 tx=0).stop 2022-06-27T11:47:59.579 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.539+0000 7fd53b7fe700 1 -- 172.21.15.195:0/2786723916 reap_dead start 2022-06-27T11:47:59.579 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.539+0000 7fd5412ca700 1 -- 172.21.15.195:0/2786723916 shutdown_connections 2022-06-27T11:47:59.579 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.539+0000 7fd5412ca700 1 -- 172.21.15.195:0/2786723916 >> 172.21.15.195:0/2786723916 conn(0x7fd53c0ccb90 msgr2=0x7fd53c0ccf90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:59.579 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.540+0000 7fd5412ca700 1 -- 172.21.15.195:0/2786723916 shutdown_connections 2022-06-27T11:47:59.579 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.540+0000 7fd5412ca700 1 -- 172.21.15.195:0/2786723916 wait complete. 2022-06-27T11:47:59.580 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.541+0000 7fe93affd700 1 --2- 172.21.15.195:0/584574401 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe93c0c31f0 0x7fe93c15b640 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe9300097a0 tx=0x7fe930009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=360e6bd7815b2d8a server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:59.580 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.539+0000 7fe2466a0700 1 -- 172.21.15.195:0/183084986 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe24006b2e0 msgr2=0x7fe2400c2610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.581 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.539+0000 7fe2466a0700 1 --2- 172.21.15.195:0/183084986 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe24006b2e0 0x7fe2400c2610 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe2300097a0 tx=0x7fe230009060 comp rx=0 tx=0).stop 2022-06-27T11:47:59.581 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.539+0000 7fe2466a0700 1 -- 172.21.15.195:0/183084986 shutdown_connections 2022-06-27T11:47:59.581 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.539+0000 7fe2466a0700 1 --2- 172.21.15.195:0/183084986 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe24006b2e0 0x7fe2400c2610 secure :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe2300097a0 tx=0x7fe230009060 comp rx=0 tx=0).stop 2022-06-27T11:47:59.581 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.539+0000 7fe2466a0700 1 --2- 172.21.15.195:0/183084986 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe2400c4bf0 0x7fe2400c3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.582 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.539+0000 7fe2466a0700 1 -- 172.21.15.195:0/183084986 >> 172.21.15.195:0/183084986 conn(0x7fe2400c8740 msgr2=0x7fe2400c8b10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:59.582 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.539+0000 7fe2466a0700 1 -- 172.21.15.195:0/183084986 shutdown_connections 2022-06-27T11:47:59.582 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.539+0000 7fe2466a0700 1 -- 172.21.15.195:0/183084986 wait complete. 2022-06-27T11:47:59.582 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.541+0000 7fe2466a0700 1 Processor -- start 2022-06-27T11:47:59.583 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.541+0000 7fe2466a0700 1 -- start start 2022-06-27T11:47:59.583 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.541+0000 7f924a7fc700 1 -- 172.21.15.195:0/1059887574 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f924c0c37b0 msgr2=0x7f924c0b70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.583 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.541+0000 7f924a7fc700 1 --2- 172.21.15.195:0/1059887574 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f924c0c37b0 0x7f924c0b70a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.584 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.541+0000 7f924a7fc700 1 -- 172.21.15.195:0/1059887574 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f924c0c4580 msgr2=0x7f924c16acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.584 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.541+0000 7f924a7fc700 1 --2- 172.21.15.195:0/1059887574 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f924c0c4580 0x7f924c16acf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.584 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.541+0000 7f924a7fc700 1 -- 172.21.15.195:0/1059887574 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f924400be70 con 0x7f924c0c41b0 2022-06-27T11:47:59.584 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.541+0000 7fe95d0fb700 1 --2- 172.21.15.195:0/714907992 >> v2:172.21.15.195:3300/0 conn(0x7fe9580c6590 0x7fe95815b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.585 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.541+0000 7f03d28aa700 1 -- start start 2022-06-27T11:47:59.585 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.541+0000 7f924a7fc700 1 --2- 172.21.15.195:0/1059887574 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f924c0c41b0 0x7f924c0b75e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f924000d8f0 tx=0x7f9240008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:59.586 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.541+0000 7fd5412ca700 1 Processor -- start 2022-06-27T11:47:59.587 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.541+0000 7fe95e0fd700 1 -- 172.21.15.195:0/714907992 >> v2:172.21.15.195:3300/0 conn(0x7fe9580c6590 msgr2=0x7fe95815b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:47:59.587 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.541+0000 7fe95e0fd700 1 --2- 172.21.15.195:0/714907992 >> v2:172.21.15.195:3300/0 conn(0x7fe9580c6590 0x7fe95815b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.587 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.541+0000 7fe95e0fd700 1 -- 172.21.15.195:0/714907992 >> v1:172.21.15.195:6789/0 conn(0x7fe958069cf0 legacy=0x7fe9580c4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.588 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.541+0000 7fe95e0fd700 1 -- 172.21.15.195:0/714907992 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe958068400 con 0x7fe95806a0c0 2022-06-27T11:47:59.588 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.542+0000 7f03d28aa700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f03cc06a690 0x7f03cc1573c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.588 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.542+0000 7f03d28aa700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f03cc06aa60 0x7f03cc149930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.589 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.542+0000 7f03d28aa700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f03cc06d050 0x7f03cc15d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.589 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.542+0000 7f03d28aa700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03cc155860 con 0x7f03cc06a690 2022-06-27T11:47:59.589 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.542+0000 7f03d28aa700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03cc157900 con 0x7f03cc06aa60 2022-06-27T11:47:59.590 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.542+0000 7f03d28aa700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03cc14ce90 con 0x7f03cc06d050 2022-06-27T11:47:59.590 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.542+0000 7f03cb7fe700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f03cc06aa60 0x7f03cc149930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.590 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.542+0000 7f03cb7fe700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f03cc06aa60 0x7f03cc149930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3300/0 says I am v2:172.21.15.195:39634/0 (socket says 172.21.15.195:39634) 2022-06-27T11:47:59.591 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.542+0000 7f03cb7fe700 1 -- 172.21.15.195:0/4105060000 learned_addr learned my addr 172.21.15.195:0/4105060000 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:47:59.591 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.542+0000 7f9232ffd700 1 -- 172.21.15.195:0/1059887574 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92400125f0 con 0x7f924c0c41b0 2022-06-27T11:47:59.592 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.542+0000 7f9232ffd700 1 -- 172.21.15.195:0/1059887574 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9240012750 con 0x7f924c0c41b0 2022-06-27T11:47:59.592 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.542+0000 7f9232ffd700 1 -- 172.21.15.195:0/1059887574 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9240012af0 con 0x7f924c0c41b0 2022-06-27T11:47:59.592 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.543+0000 7f92518ae700 1 -- 172.21.15.195:0/1059887574 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f924c16f0d0 con 0x7f924c0c41b0 2022-06-27T11:47:59.592 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.543+0000 7f92518ae700 1 -- 172.21.15.195:0/1059887574 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f924c14d890 con 0x7f924c0c41b0 2022-06-27T11:47:59.593 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.542+0000 7fd5412ca700 1 -- start start 2022-06-27T11:47:59.593 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.543+0000 7fd5412ca700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd528003920 0x7fd53c0cbef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.593 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.543+0000 7fd5412ca700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd53c155670 0x7fd53c16a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.594 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.543+0000 7fd5412ca700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd53c15dd20 0x7fd53c16ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.594 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.543+0000 7fd5412ca700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd53c155b60 con 0x7fd528003920 2022-06-27T11:47:59.594 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.543+0000 7fd5412ca700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd53c16e780 con 0x7fd53c15dd20 2022-06-27T11:47:59.594 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.543+0000 7fd5412ca700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd53c16db70 con 0x7fd53c155670 2022-06-27T11:47:59.595 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.542+0000 7fe939ffb700 1 -- 172.21.15.195:0/584574401 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe930011630 con 0x7fe93c0c31f0 2022-06-27T11:47:59.595 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.542+0000 7fe939ffb700 1 -- 172.21.15.195:0/584574401 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe930011790 con 0x7fe93c0c31f0 2022-06-27T11:47:59.595 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.542+0000 7fe939ffb700 1 -- 172.21.15.195:0/584574401 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe930011b30 con 0x7fe93c0c31f0 2022-06-27T11:47:59.595 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.542+0000 7fe9415c3700 1 -- 172.21.15.195:0/584574401 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe93c0c31f0 msgr2=0x7fe93c15b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.596 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.542+0000 7fe9415c3700 1 --2- 172.21.15.195:0/584574401 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe93c0c31f0 0x7fe93c15b640 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe9300097a0 tx=0x7fe930009060 comp rx=0 tx=0).stop 2022-06-27T11:47:59.596 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.543+0000 7fe9415c3700 1 -- 172.21.15.195:0/584574401 shutdown_connections 2022-06-27T11:47:59.596 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.543+0000 7fe9415c3700 1 --2- 172.21.15.195:0/584574401 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe93c0c31f0 0x7fe93c15b640 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.596 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.543+0000 7fe9415c3700 1 --2- 172.21.15.195:0/584574401 >> v2:172.21.15.202:3300/0 conn(0x7fe93c0c35c0 0x7fe93c067ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.597 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.543+0000 7fe9415c3700 1 -- 172.21.15.195:0/584574401 >> 172.21.15.195:0/584574401 conn(0x7fe93c0c8740 msgr2=0x7fe93c0c8b10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:59.597 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.543+0000 7fe9415c3700 1 -- 172.21.15.195:0/584574401 shutdown_connections 2022-06-27T11:47:59.598 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.542+0000 7fe2466a0700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe2400c3010 0x7fe24006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.598 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.542+0000 7fe2466a0700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe2400c4bf0 0x7fe2400b7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.599 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.542+0000 7fe2466a0700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe2401518a0 0x7fe24014cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.599 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.542+0000 7fe2466a0700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2401537e0 con 0x7fe2401518a0 2022-06-27T11:47:59.599 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.542+0000 7fe2466a0700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe24016f6e0 con 0x7fe2400c3010 2022-06-27T11:47:59.600 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.542+0000 7fe2466a0700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe24006d800 con 0x7fe2400c4bf0 2022-06-27T11:47:59.600 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.542+0000 7fe23ffff700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe2400c3010 0x7fe24006d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.601 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.542+0000 7fe237fff700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe2400c4bf0 0x7fe2400b7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.601 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.542+0000 7fe23ffff700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe2400c3010 0x7fe24006d2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3300/0 says I am v2:172.21.15.195:39626/0 (socket says 172.21.15.195:39626) 2022-06-27T11:47:59.602 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.542+0000 7fe23ffff700 1 -- 172.21.15.195:0/123644676 learned_addr learned my addr 172.21.15.195:0/123644676 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:47:59.602 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.542+0000 7fe244c3d700 1 --2- 172.21.15.195:0/123644676 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe2401518a0 0x7fe24014cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.602 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.542+0000 7fe23ffff700 1 -- 172.21.15.195:0/123644676 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe2400c4bf0 msgr2=0x7fe2400b7150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.603 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.542+0000 7fe23ffff700 1 --2- 172.21.15.195:0/123644676 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe2400c4bf0 0x7fe2400b7150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.603 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.542+0000 7fe23ffff700 1 -- 172.21.15.195:0/123644676 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe2401518a0 msgr2=0x7fe24014cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.604 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.542+0000 7fe23ffff700 1 --2- 172.21.15.195:0/123644676 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe2401518a0 0x7fe24014cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.604 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.542+0000 7fe23ffff700 1 -- 172.21.15.195:0/123644676 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2300092e0 con 0x7fe2400c3010 2022-06-27T11:47:59.605 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.542+0000 7fe237fff700 1 --2- 172.21.15.195:0/123644676 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe2400c4bf0 0x7fe2400b7150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:47:59.605 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.543+0000 7fe23ffff700 1 --2- 172.21.15.195:0/123644676 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe2400c3010 0x7fe24006d2c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe24006cd10 tx=0x7fe230016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:59.606 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.543+0000 7f03cb7fe700 1 -- 172.21.15.195:0/4105060000 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f03cc06d050 msgr2=0x7f03cc15d130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2022-06-27T11:47:59.606 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.543+0000 7f03cb7fe700 1 --2- 172.21.15.195:0/4105060000 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f03cc06d050 0x7f03cc15d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.607 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.543+0000 7f03cb7fe700 1 -- 172.21.15.195:0/4105060000 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f03cc06a690 msgr2=0x7f03cc1573c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.607 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.543+0000 7f03cb7fe700 1 --2- 172.21.15.195:0/4105060000 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f03cc06a690 0x7f03cc1573c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.608 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.543+0000 7f03cb7fe700 1 -- 172.21.15.195:0/4105060000 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03c401aa20 con 0x7f03cc06aa60 2022-06-27T11:47:59.609 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.543+0000 7fd53b7fe700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd53c15dd20 0x7fd53c16ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.609 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.543+0000 7fd53b7fe700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd53c15dd20 0x7fd53c16ac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3300/0 says I am v2:172.21.15.195:39638/0 (socket says 172.21.15.195:39638) 2022-06-27T11:47:59.609 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.543+0000 7fd53b7fe700 1 -- 172.21.15.195:0/2471686570 learned_addr learned my addr 172.21.15.195:0/2471686570 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:47:59.610 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.543+0000 7fd53a7fc700 1 --2- 172.21.15.195:0/2471686570 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd53c155670 0x7fd53c16a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.610 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.543+0000 7fe95e0fd700 1 --2- 172.21.15.195:0/714907992 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe95806a0c0 0x7fe958154650 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe9480097a0 tx=0x7fe948009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c3007e099eb7d24 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:59.611 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.543+0000 7fd53b7fe700 1 -- 172.21.15.195:0/2471686570 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd53c155670 msgr2=0x7fd53c16a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.612 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.544+0000 7fd53b7fe700 1 --2- 172.21.15.195:0/2471686570 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd53c155670 0x7fd53c16a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.612 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.544+0000 7fd53b7fe700 1 -- 172.21.15.195:0/2471686570 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd528003920 msgr2=0x7fd53c0cbef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.612 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.544+0000 7fd53b7fe700 1 --2- 172.21.15.195:0/2471686570 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd528003920 0x7fd53c0cbef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.613 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.544+0000 7fd53b7fe700 1 -- 172.21.15.195:0/2471686570 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd53c16de50 con 0x7fd53c15dd20 2022-06-27T11:47:59.614 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.543+0000 7fe23cff9700 1 -- 172.21.15.195:0/123644676 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2300038b0 con 0x7fe2400c3010 2022-06-27T11:47:59.614 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.543+0000 7fe23cff9700 1 -- 172.21.15.195:0/123644676 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe230003a30 con 0x7fe2400c3010 2022-06-27T11:47:59.614 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.543+0000 7fe23cff9700 1 -- 172.21.15.195:0/123644676 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe230003df0 con 0x7fe2400c3010 2022-06-27T11:47:59.615 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.543+0000 7fe2466a0700 1 -- 172.21.15.195:0/123644676 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe24016f400 con 0x7fe2400c3010 2022-06-27T11:47:59.615 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.543+0000 7fe2466a0700 1 -- 172.21.15.195:0/123644676 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe24016b780 con 0x7fe2400c3010 2022-06-27T11:47:59.616 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.544+0000 7f03cb7fe700 1 --2- 172.21.15.195:0/4105060000 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f03cc06aa60 0x7f03cc149930 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f03bc00c440 tx=0x7f03bc007e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:59.616 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.543+0000 7fe9415c3700 1 -- 172.21.15.195:0/584574401 wait complete. 2022-06-27T11:47:59.617 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.544+0000 7fe9415c3700 1 Processor -- start 2022-06-27T11:47:59.617 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.544+0000 7fe95c8fa700 1 -- 172.21.15.195:0/714907992 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe948011630 con 0x7fe95806a0c0 2022-06-27T11:47:59.618 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.544+0000 7fe95c8fa700 1 -- 172.21.15.195:0/714907992 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe948011790 con 0x7fe95806a0c0 2022-06-27T11:47:59.618 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.544+0000 7fe95c8fa700 1 -- 172.21.15.195:0/714907992 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe948011b30 con 0x7fe95806a0c0 2022-06-27T11:47:59.619 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.545+0000 7fe95fb60700 1 -- 172.21.15.195:0/714907992 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe95806a0c0 msgr2=0x7fe958154650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.619 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.545+0000 7fe95fb60700 1 --2- 172.21.15.195:0/714907992 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe95806a0c0 0x7fe958154650 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe9480097a0 tx=0x7fe948009060 comp rx=0 tx=0).stop 2022-06-27T11:47:59.619 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.545+0000 7fe95fb60700 1 -- 172.21.15.195:0/714907992 shutdown_connections 2022-06-27T11:47:59.620 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.545+0000 7fe95fb60700 1 --2- 172.21.15.195:0/714907992 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe95806a0c0 0x7fe958154650 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.620 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.545+0000 7fe95fb60700 1 --2- 172.21.15.195:0/714907992 >> v2:172.21.15.195:3300/0 conn(0x7fe9580c6590 0x7fe95815b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.620 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.545+0000 7fe95fb60700 1 -- 172.21.15.195:0/714907992 >> 172.21.15.195:0/714907992 conn(0x7fe9580c9b00 msgr2=0x7fe9580c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:59.621 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.545+0000 7f92518ae700 1 -- 172.21.15.195:0/1059887574 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f924c0c7920 con 0x7f924c0c41b0 2022-06-27T11:47:59.622 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.546+0000 7f9232ffd700 1 -- 172.21.15.195:0/1059887574 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f924001e460 con 0x7f924c0c41b0 2022-06-27T11:47:59.622 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.546+0000 7f9232ffd700 1 --2- 172.21.15.195:0/1059887574 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f922c063db0 0x7f922c066260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.622 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.546+0000 7f9232ffd700 1 -- 172.21.15.195:0/1059887574 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f9240080a00 con 0x7f924c0c41b0 2022-06-27T11:47:59.623 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.545+0000 7fd53b7fe700 1 --2- 172.21.15.195:0/2471686570 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd53c15dd20 0x7fd53c16ac50 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd53c155330 tx=0x7fd52c007db0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:59.623 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.545+0000 7fd522ffd700 1 -- 172.21.15.195:0/2471686570 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd52c001cf0 con 0x7fd53c15dd20 2022-06-27T11:47:59.624 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.545+0000 7fd522ffd700 1 -- 172.21.15.195:0/2471686570 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd52c005870 con 0x7fd53c15dd20 2022-06-27T11:47:59.624 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.545+0000 7fd522ffd700 1 -- 172.21.15.195:0/2471686570 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd52c005c30 con 0x7fd53c15dd20 2022-06-27T11:47:59.625 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.545+0000 7fd5412ca700 1 -- 172.21.15.195:0/2471686570 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd53c16e130 con 0x7fd53c15dd20 2022-06-27T11:47:59.625 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.545+0000 7fd5412ca700 1 -- 172.21.15.195:0/2471686570 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd53c158050 con 0x7fd53c15dd20 2022-06-27T11:47:59.626 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.546+0000 7fe9782fd700 1 Processor -- start 2022-06-27T11:47:59.626 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.546+0000 7fe9782fd700 1 -- start start 2022-06-27T11:47:59.627 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.545+0000 7fe9415c3700 1 -- start start 2022-06-27T11:47:59.627 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.546+0000 7fe9415c3700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe93c0c31f0 0x7fe93c06fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.628 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.546+0000 7fe9415c3700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe93c0c35c0 0x7fe93c0b70a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.628 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.546+0000 7fe9415c3700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe93c0c49b0 0x7fe93c0bbce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.628 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.546+0000 7fe9415c3700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe93c070310 con 0x7fe93c0c35c0 2022-06-27T11:47:59.628 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.546+0000 7fe9415c3700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe93c06d870 con 0x7fe93c0c31f0 2022-06-27T11:47:59.629 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.546+0000 7fe9415c3700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe93c06ea70 con 0x7fe93c0c49b0 2022-06-27T11:47:59.629 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.546+0000 7fe93b7fe700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe93c0c49b0 0x7fe93c0bbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.629 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.546+0000 7fe93b7fe700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe93c0c49b0 0x7fe93c0bbce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3301/0 says I am v2:172.21.15.195:34274/0 (socket says 172.21.15.195:34274) 2022-06-27T11:47:59.630 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.546+0000 7fe93b7fe700 1 -- 172.21.15.195:0/746803739 learned_addr learned my addr 172.21.15.195:0/746803739 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:47:59.630 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.546+0000 7fe93affd700 1 --2- 172.21.15.195:0/746803739 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe93c0c31f0 0x7fe93c06fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.631 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.545+0000 7fe235ffb700 1 -- 172.21.15.195:0/123644676 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe24016b300 con 0x7fe2400c3010 2022-06-27T11:47:59.631 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.547+0000 7fe95fb60700 1 -- 172.21.15.195:0/714907992 shutdown_connections 2022-06-27T11:47:59.632 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.547+0000 7fd5412ca700 1 -- 172.21.15.195:0/2471686570 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd53c0722c0 con 0x7fd53c15dd20 2022-06-27T11:47:59.634 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.546+0000 7fed059ac700 1 Processor -- start 2022-06-27T11:47:59.634 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.546+0000 7fed059ac700 1 -- start start 2022-06-27T11:47:59.635 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.547+0000 7fed059ac700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fed00069cf0 0x7fed0015b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.635 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.547+0000 7fed059ac700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fed000c6590 0x7fed000c4190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.635 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.547+0000 7fed059ac700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fed0006a0c0 0x7fed00154650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.636 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.547+0000 7fed059ac700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed00067ed0 con 0x7fed000c6590 2022-06-27T11:47:59.636 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.547+0000 7fed059ac700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed00068120 con 0x7fed00069cf0 2022-06-27T11:47:59.636 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.547+0000 7fed059ac700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed00068400 con 0x7fed0006a0c0 2022-06-27T11:47:59.637 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.547+0000 7fecfeffd700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fed00069cf0 0x7fed0015b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.637 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.547+0000 7fecfeffd700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fed00069cf0 0x7fed0015b310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3300/0 says I am v2:172.21.15.195:39648/0 (socket says 172.21.15.195:39648) 2022-06-27T11:47:59.637 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.547+0000 7fecfeffd700 1 -- 172.21.15.195:0/3325645324 learned_addr learned my addr 172.21.15.195:0/3325645324 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:47:59.637 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.547+0000 7fecfe7fc700 1 --2- 172.21.15.195:0/3325645324 >> v2:172.21.15.195:3300/0 conn(0x7fed000c6590 0x7fed000c4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.638 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.548+0000 7fecfeffd700 1 -- 172.21.15.195:0/3325645324 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fed0006a0c0 msgr2=0x7fed00154650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:47:59.638 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.548+0000 7fecfeffd700 1 --2- 172.21.15.195:0/3325645324 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fed0006a0c0 0x7fed00154650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.638 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.548+0000 7fecfeffd700 1 -- 172.21.15.195:0/3325645324 >> v2:172.21.15.195:3300/0 conn(0x7fed000c6590 msgr2=0x7fed000c4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:47:59.638 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.548+0000 7fecfeffd700 1 --2- 172.21.15.195:0/3325645324 >> v2:172.21.15.195:3300/0 conn(0x7fed000c6590 0x7fed000c4190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.638 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.548+0000 7fecfeffd700 1 -- 172.21.15.195:0/3325645324 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed00063af0 con 0x7fed00069cf0 2022-06-27T11:47:59.639 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.548+0000 7fe93b7fe700 1 -- 172.21.15.195:0/746803739 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe93c0c31f0 msgr2=0x7fe93c06fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.639 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.548+0000 7fe93b7fe700 1 --2- 172.21.15.195:0/746803739 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe93c0c31f0 0x7fe93c06fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.639 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.548+0000 7fe93b7fe700 1 -- 172.21.15.195:0/746803739 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe93c0c35c0 msgr2=0x7fe93c0b70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.640 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.548+0000 7fe93b7fe700 1 --2- 172.21.15.195:0/746803739 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe93c0c35c0 0x7fe93c0b70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.640 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.548+0000 7fe93b7fe700 1 -- 172.21.15.195:0/746803739 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9300092e0 con 0x7fe93c0c49b0 2022-06-27T11:47:59.640 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.547+0000 7fe23cff9700 1 -- 172.21.15.195:0/123644676 <== mon.1 v2:172.21.15.202:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fe230023070 con 0x7fe2400c3010 2022-06-27T11:47:59.641 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.547+0000 7fe23cff9700 1 --2- 172.21.15.195:0/123644676 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe224063db0 0x7fe224066260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.641 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.547+0000 7fe23cff9700 1 -- 172.21.15.195:0/123644676 <== mon.1 v2:172.21.15.202:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fe230081e90 con 0x7fe2400c3010 2022-06-27T11:47:59.641 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.548+0000 7fecfeffd700 1 --2- 172.21.15.195:0/3325645324 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fed00069cf0 0x7fed0015b310 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fed0006f840 tx=0x7fecf40090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=676836f838427e00 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:59.642 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.545+0000 7f03c3fff700 1 -- 172.21.15.195:0/4105060000 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03bc0072e0 con 0x7f03cc06aa60 2022-06-27T11:47:59.642 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.545+0000 7f03c3fff700 1 -- 172.21.15.195:0/4105060000 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03bc007440 con 0x7f03cc06aa60 2022-06-27T11:47:59.643 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.545+0000 7f03c3fff700 1 -- 172.21.15.195:0/4105060000 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03bc005030 con 0x7f03cc06aa60 2022-06-27T11:47:59.643 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.545+0000 7f03d28aa700 1 -- 172.21.15.195:0/4105060000 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03cc14d6f0 con 0x7f03cc06aa60 2022-06-27T11:47:59.643 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.545+0000 7f03d28aa700 1 -- 172.21.15.195:0/4105060000 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03cc0cb5e0 con 0x7f03cc06aa60 2022-06-27T11:47:59.643 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.547+0000 7f03d28aa700 1 -- 172.21.15.195:0/4105060000 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03cc0722c0 con 0x7f03cc06aa60 2022-06-27T11:47:59.644 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.548+0000 7fe9782fd700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fe9700680b0 0x7fe9700c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.644 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.548+0000 7fe9782fd700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe970069050 0x7fe97015a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.645 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.548+0000 7fd522ffd700 1 -- 172.21.15.195:0/2471686570 <== mon.1 v2:172.21.15.202:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fd52c010030 con 0x7fd53c15dd20 2022-06-27T11:47:59.645 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.548+0000 7fe93b7fe700 1 --2- 172.21.15.195:0/746803739 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe93c0c49b0 0x7fe93c0bbce0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe92c0010e0 tx=0x7fe92c007c70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:59.646 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.548+0000 7fe237fff700 1 --2- 172.21.15.195:0/123644676 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe224063db0 0x7fe224066260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.646 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.548+0000 7fe9782fd700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9700c54b0 con 0x7fe970068480 2022-06-27T11:47:59.646 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.548+0000 7fe9782fd700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9700c6050 con 0x7fe9700680b0 2022-06-27T11:47:59.647 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.548+0000 7fe9782fd700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe97015b070 con 0x7fe970069050 2022-06-27T11:47:59.647 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.548+0000 7fe95fb60700 1 -- 172.21.15.195:0/714907992 wait complete. 2022-06-27T11:47:59.648 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.548+0000 7f924affd700 1 --2- 172.21.15.195:0/1059887574 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f922c063db0 0x7f922c066260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.648 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.548+0000 7fe975898700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe970069050 0x7fe97015a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.649 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.548+0000 7fd522ffd700 1 --2- 172.21.15.195:0/2471686570 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd51c063e80 0x7fd51c066330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.649 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.548+0000 7fd522ffd700 1 -- 172.21.15.195:0/2471686570 <== mon.1 v2:172.21.15.202:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fd52c07fc60 con 0x7fd53c15dd20 2022-06-27T11:47:59.649 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.548+0000 7fd53affd700 1 --2- 172.21.15.195:0/2471686570 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd51c063e80 0x7fd51c066330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.649 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.549+0000 7f924affd700 1 --2- 172.21.15.195:0/1059887574 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f922c063db0 0x7f922c066260 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f923c0018a0 tx=0x7f923c009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:59.650 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.549+0000 7fd53affd700 1 --2- 172.21.15.195:0/2471686570 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd51c063e80 0x7fd51c066330 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd534001f10 tx=0x7fd534015000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:59.651 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.549+0000 7fe237fff700 1 --2- 172.21.15.195:0/123644676 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe224063db0 0x7fe224066260 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe2400b86b0 tx=0x7fe228008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:59.652 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.548+0000 7fe975898700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe970069050 0x7fe97015a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3301/0 says I am v2:172.21.15.195:34288/0 (socket says 172.21.15.195:34288) 2022-06-27T11:47:59.652 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.548+0000 7fe975898700 1 -- 172.21.15.195:0/2111064136 learned_addr learned my addr 172.21.15.195:0/2111064136 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:47:59.653 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.548+0000 7fe922ffd700 1 -- 172.21.15.195:0/746803739 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe92c0072e0 con 0x7fe93c0c49b0 2022-06-27T11:47:59.654 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.548+0000 7fe922ffd700 1 -- 172.21.15.195:0/746803739 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe92c007440 con 0x7fe93c0c49b0 2022-06-27T11:47:59.654 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.548+0000 7fe922ffd700 1 -- 172.21.15.195:0/746803739 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe92c0057c0 con 0x7fe93c0c49b0 2022-06-27T11:47:59.655 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.548+0000 7fe9415c3700 1 -- 172.21.15.195:0/746803739 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe93c159db0 con 0x7fe93c0c49b0 2022-06-27T11:47:59.655 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.548+0000 7fe9415c3700 1 -- 172.21.15.195:0/746803739 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe93c06e140 con 0x7fe93c0c49b0 2022-06-27T11:47:59.656 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.549+0000 7fe975898700 1 -- 172.21.15.195:0/2111064136 >> v2:172.21.15.195:3300/0 conn(0x7fe9700680b0 msgr2=0x7fe9700c49a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2022-06-27T11:47:59.656 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.549+0000 7fe975898700 1 --2- 172.21.15.195:0/2111064136 >> v2:172.21.15.195:3300/0 conn(0x7fe9700680b0 0x7fe9700c49a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.656 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.549+0000 7fe975898700 1 -- 172.21.15.195:0/2111064136 >> v1:172.21.15.195:6789/0 conn(0x7fe970068480 legacy=0x7fe9700c5940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:47:59.657 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.549+0000 7fe975898700 1 -- 172.21.15.195:0/2111064136 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe97015b350 con 0x7fe970069050 2022-06-27T11:47:59.657 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.549+0000 7fe9415c3700 1 -- 172.21.15.195:0/746803739 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe93c0722c0 con 0x7fe93c0c49b0 2022-06-27T11:47:59.658 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.549+0000 7fe975898700 1 --2- 172.21.15.195:0/2111064136 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe970069050 0x7fe97015a970 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe968009f20 tx=0x7fe968009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd02aafd8047f4d8 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:59.659 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.550+0000 7fe95fb60700 1 Processor -- start 2022-06-27T11:47:59.659 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.550+0000 7fe95fb60700 1 -- start start 2022-06-27T11:47:59.660 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.550+0000 7fe975097700 1 -- 172.21.15.195:0/2111064136 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe96800e070 con 0x7fe970069050 2022-06-27T11:47:59.660 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.550+0000 7fe975097700 1 -- 172.21.15.195:0/2111064136 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe96800ba40 con 0x7fe970069050 2022-06-27T11:47:59.660 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.550+0000 7fe975097700 1 -- 172.21.15.195:0/2111064136 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9680092e0 con 0x7fe970069050 2022-06-27T11:47:59.661 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.550+0000 7fe95fb60700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe958069cf0 0x7fe958141390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.662 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.550+0000 7fe95fb60700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe95806a0c0 0x7fe95813f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.662 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.550+0000 7fe95fb60700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe9580c6590 0x7fe95813f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.663 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.550+0000 7fe95fb60700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe95813e700 con 0x7fe958069cf0 2022-06-27T11:47:59.663 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.550+0000 7fe95fb60700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe958139fc0 con 0x7fe95806a0c0 2022-06-27T11:47:59.664 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.550+0000 7fe95fb60700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe958134130 con 0x7fe9580c6590 2022-06-27T11:47:59.664 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.550+0000 7fe95d8fc700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe958069cf0 0x7fe958141390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.664 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.550+0000 7fe95d8fc700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe958069cf0 0x7fe958141390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41538/0 (socket says 172.21.15.195:41538) 2022-06-27T11:47:59.664 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.550+0000 7fe95d8fc700 1 -- 172.21.15.195:0/3987743196 learned_addr learned my addr 172.21.15.195:0/3987743196 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:47:59.664 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.550+0000 7fe95d0fb700 1 --2- 172.21.15.195:0/3987743196 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe95806a0c0 0x7fe95813f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.665 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.551+0000 7fe9782fd700 1 -- 172.21.15.195:0/2111064136 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe970069050 msgr2=0x7fe97015a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.665 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.551+0000 7fe9782fd700 1 --2- 172.21.15.195:0/2111064136 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe970069050 0x7fe97015a970 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe968009f20 tx=0x7fe968009840 comp rx=0 tx=0).stop 2022-06-27T11:47:59.666 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.551+0000 7fe9782fd700 1 -- 172.21.15.195:0/2111064136 shutdown_connections 2022-06-27T11:47:59.666 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.551+0000 7fe95e0fd700 1 --2- 172.21.15.195:0/3987743196 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe9580c6590 0x7fe95813f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.667 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.551+0000 7fe95d8fc700 1 -- 172.21.15.195:0/3987743196 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe9580c6590 msgr2=0x7fe95813f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.667 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.551+0000 7fe9782fd700 1 --2- 172.21.15.195:0/2111064136 >> v2:172.21.15.195:3300/0 conn(0x7fe9700680b0 0x7fe9700c49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.667 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.551+0000 7fe9782fd700 1 --2- 172.21.15.195:0/2111064136 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe970069050 0x7fe97015a970 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.668 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.551+0000 7fe9782fd700 1 -- 172.21.15.195:0/2111064136 >> 172.21.15.195:0/2111064136 conn(0x7fe9700ccb90 msgr2=0x7fe9700ccf90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:59.668 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.551+0000 7fe95d8fc700 1 --2- 172.21.15.195:0/3987743196 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe9580c6590 0x7fe95813f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.668 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.551+0000 7fe95d8fc700 1 -- 172.21.15.195:0/3987743196 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe95806a0c0 msgr2=0x7fe95813f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.669 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.551+0000 7fe95d8fc700 1 --2- 172.21.15.195:0/3987743196 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe95806a0c0 0x7fe95813f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.669 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.551+0000 7fe95d8fc700 1 -- 172.21.15.195:0/3987743196 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9480092e0 con 0x7fe958069cf0 2022-06-27T11:47:59.669 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.551+0000 7fe9782fd700 1 -- 172.21.15.195:0/2111064136 shutdown_connections 2022-06-27T11:47:59.670 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.551+0000 7fe922ffd700 1 -- 172.21.15.195:0/746803739 <== mon.2 v2:172.21.15.202:3301/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fe92c019030 con 0x7fe93c0c49b0 2022-06-27T11:47:59.670 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.551+0000 7fe95d8fc700 1 --2- 172.21.15.195:0/3987743196 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe958069cf0 0x7fe958141390 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe95400c4c0 tx=0x7fe954007d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:59.671 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.551+0000 7fe922ffd700 1 --2- 172.21.15.195:0/746803739 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe91c063dd0 0x7fe91c066280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.671 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.551+0000 7fe922ffd700 1 -- 172.21.15.195:0/746803739 <== mon.2 v2:172.21.15.202:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fe92c080120 con 0x7fe93c0c49b0 2022-06-27T11:47:59.672 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.551+0000 7fe93affd700 1 --2- 172.21.15.195:0/746803739 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe91c063dd0 0x7fe91c066280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.672 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.551+0000 7fe9782fd700 1 -- 172.21.15.195:0/2111064136 wait complete. 2022-06-27T11:47:59.673 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.552+0000 7fe93affd700 1 --2- 172.21.15.195:0/746803739 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe91c063dd0 0x7fe91c066280 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe9300097a0 tx=0x7fe930017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:59.673 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.552+0000 7fe23cff9700 1 -- 172.21.15.195:0/123644676 <== mon.1 v2:172.21.15.202:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fe24016b300 con 0x7fe2400c3010 2022-06-27T11:47:59.674 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.553+0000 7fe9782fd700 1 Processor -- start 2022-06-27T11:47:59.674 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.553+0000 7fe9782fd700 1 -- start start 2022-06-27T11:47:59.675 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.553+0000 7f03c3fff700 1 -- 172.21.15.195:0/4105060000 <== mon.1 v2:172.21.15.202:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f03bc010070 con 0x7f03cc06aa60 2022-06-27T11:47:59.676 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.553+0000 7fe9782fd700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe9700680b0 0x7fe97016b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.676 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.553+0000 7fe9782fd700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe970068480 0x7fe970159a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.676 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.553+0000 7fe9782fd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe970069050 0x7fe9700c44b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.676 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.553+0000 7fe9782fd700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9701550c0 con 0x7fe970069050 2022-06-27T11:47:59.677 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.553+0000 7fe9782fd700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9700c1fb0 con 0x7fe970068480 2022-06-27T11:47:59.677 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.553+0000 7fe9782fd700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9700c0e90 con 0x7fe9700680b0 2022-06-27T11:47:59.677 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.553+0000 7f03c3fff700 1 --2- 172.21.15.195:0/4105060000 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f03b0063d30 0x7f03b00661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.678 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.553+0000 7fe976099700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe9700680b0 0x7fe97016b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.678 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.553+0000 7fe97689a700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe970069050 0x7fe9700c44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.678 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.554+0000 7f03cbfff700 1 --2- 172.21.15.195:0/4105060000 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f03b0063d30 0x7f03b00661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.679 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.553+0000 7fe976099700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe9700680b0 0x7fe97016b610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3301/0 says I am v2:172.21.15.195:34302/0 (socket says 172.21.15.195:34302) 2022-06-27T11:47:59.680 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.553+0000 7fe97689a700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe970069050 0x7fe9700c44b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41548/0 (socket says 172.21.15.195:41548) 2022-06-27T11:47:59.680 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.553+0000 7fe976099700 1 -- 172.21.15.195:0/264199571 learned_addr learned my addr 172.21.15.195:0/264199571 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:47:59.680 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.554+0000 7fe976099700 1 -- 172.21.15.195:0/264199571 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe970068480 msgr2=0x7fe970159a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2022-06-27T11:47:59.681 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.554+0000 7fe976099700 1 --2- 172.21.15.195:0/264199571 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe970068480 0x7fe970159a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.681 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.554+0000 7fe976099700 1 -- 172.21.15.195:0/264199571 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe970069050 msgr2=0x7fe9700c44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.681 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.554+0000 7fe976099700 1 --2- 172.21.15.195:0/264199571 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe970069050 0x7fe9700c44b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.681 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.554+0000 7fe976099700 1 -- 172.21.15.195:0/264199571 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe968009ae0 con 0x7fe9700680b0 2022-06-27T11:47:59.682 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.554+0000 7fe97689a700 1 --2- 172.21.15.195:0/264199571 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe970069050 0x7fe9700c44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:47:59.682 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.554+0000 7f03cbfff700 1 --2- 172.21.15.195:0/4105060000 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f03b0063d30 0x7f03b00661e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f03c401ade0 tx=0x7f03c402a000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:59.683 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.554+0000 7fe976099700 1 --2- 172.21.15.195:0/264199571 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe9700680b0 0x7fe97016b610 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe96000b4c0 tx=0x7fe960006f50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:59.684 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.554+0000 7fe94dffb700 1 -- 172.21.15.195:0/3987743196 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9540072e0 con 0x7fe958069cf0 2022-06-27T11:47:59.684 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.554+0000 7fe94dffb700 1 -- 172.21.15.195:0/3987743196 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe954007440 con 0x7fe958069cf0 2022-06-27T11:47:59.685 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.554+0000 7fe94dffb700 1 -- 172.21.15.195:0/3987743196 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9540057c0 con 0x7fe958069cf0 2022-06-27T11:47:59.685 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.554+0000 7fe95fb60700 1 -- 172.21.15.195:0/3987743196 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe95813fd30 con 0x7fe958069cf0 2022-06-27T11:47:59.685 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.554+0000 7f9232ffd700 1 -- 172.21.15.195:0/1059887574 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f924004f4a0 con 0x7f924c0c41b0 2022-06-27T11:47:59.686 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.554+0000 7fe95fb60700 1 -- 172.21.15.195:0/3987743196 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9580c86e0 con 0x7fe958069cf0 2022-06-27T11:47:59.687 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.555+0000 7fe95e7fc700 1 -- 172.21.15.195:0/264199571 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe96000be80 con 0x7fe9700680b0 2022-06-27T11:47:59.687 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.555+0000 7fe95e7fc700 1 -- 172.21.15.195:0/264199571 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe960011630 con 0x7fe9700680b0 2022-06-27T11:47:59.688 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.555+0000 7fe95e7fc700 1 -- 172.21.15.195:0/264199571 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9600119f0 con 0x7fe9700680b0 2022-06-27T11:47:59.688 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.555+0000 7fe9782fd700 1 -- 172.21.15.195:0/264199571 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe970154d70 con 0x7fe9700680b0 2022-06-27T11:47:59.689 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.555+0000 7fe9782fd700 1 -- 172.21.15.195:0/264199571 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9700c13a0 con 0x7fe9700680b0 2022-06-27T11:47:59.689 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.556+0000 7fe95fb60700 1 -- 172.21.15.195:0/3987743196 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9580722c0 con 0x7fe958069cf0 2022-06-27T11:47:59.690 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.556+0000 7fecfdffb700 1 -- 172.21.15.195:0/3325645324 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf4005ad0 con 0x7fed00069cf0 2022-06-27T11:47:59.691 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.556+0000 7fd522ffd700 1 -- 172.21.15.195:0/2471686570 <== mon.1 v2:172.21.15.202:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fd52c04e780 con 0x7fd53c15dd20 2022-06-27T11:47:59.692 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.556+0000 7fecfdffb700 1 -- 172.21.15.195:0/3325645324 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecf4005c50 con 0x7fed00069cf0 2022-06-27T11:47:59.692 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.556+0000 7fecfdffb700 1 -- 172.21.15.195:0/3325645324 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf4004030 con 0x7fed00069cf0 2022-06-27T11:47:59.693 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.556+0000 7fe94dffb700 1 -- 172.21.15.195:0/3987743196 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fe954019030 con 0x7fe958069cf0 2022-06-27T11:47:59.693 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.556+0000 7f03c3fff700 1 -- 172.21.15.195:0/4105060000 <== mon.1 v2:172.21.15.202:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f03bc053460 con 0x7f03cc06aa60 2022-06-27T11:47:59.694 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.557+0000 7fe94dffb700 1 --2- 172.21.15.195:0/3987743196 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe940063d30 0x7fe9400661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.695 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.557+0000 7fe94dffb700 1 -- 172.21.15.195:0/3987743196 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fe95407f9a0 con 0x7fe958069cf0 2022-06-27T11:47:59.695 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.557+0000 7fed059ac700 1 -- 172.21.15.195:0/3325645324 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fed00069cf0 msgr2=0x7fed0015b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.695 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.557+0000 7fed059ac700 1 --2- 172.21.15.195:0/3325645324 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fed00069cf0 0x7fed0015b310 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fed0006f840 tx=0x7fecf40090f0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.696 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.557+0000 7fe95e7fc700 1 -- 172.21.15.195:0/264199571 <== mon.2 v2:172.21.15.202:3301/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fe96000f050 con 0x7fe9700680b0 2022-06-27T11:47:59.697 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.557+0000 7fe95d0fb700 1 --2- 172.21.15.195:0/3987743196 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe940063d30 0x7fe9400661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.697 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.557+0000 7fe9782fd700 1 -- 172.21.15.195:0/264199571 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9700722c0 con 0x7fe9700680b0 2022-06-27T11:47:59.697 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.557+0000 7fe95e7fc700 1 --2- 172.21.15.195:0/264199571 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe950063d80 0x7fe950066230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.698 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.557+0000 7fe95d0fb700 1 --2- 172.21.15.195:0/3987743196 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe940063d30 0x7fe9400661e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe958153ed0 tx=0x7fe9500097e0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:59.698 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.557+0000 7fe95e7fc700 1 -- 172.21.15.195:0/264199571 <== mon.2 v2:172.21.15.202:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fe960080060 con 0x7fe9700680b0 2022-06-27T11:47:59.700 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.558+0000 7fed059ac700 1 -- 172.21.15.195:0/3325645324 shutdown_connections 2022-06-27T11:47:59.700 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.558+0000 7fed059ac700 1 --2- 172.21.15.195:0/3325645324 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fed0006a0c0 0x7fed00154650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.700 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.558+0000 7fed059ac700 1 --2- 172.21.15.195:0/3325645324 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fed00069cf0 0x7fed0015b310 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.701 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.558+0000 7fed059ac700 1 --2- 172.21.15.195:0/3325645324 >> v2:172.21.15.195:3300/0 conn(0x7fed000c6590 0x7fed000c4190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.701 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.558+0000 7fed059ac700 1 -- 172.21.15.195:0/3325645324 >> 172.21.15.195:0/3325645324 conn(0x7fed000c9b00 msgr2=0x7fed000c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:59.701 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.558+0000 7fed059ac700 1 -- 172.21.15.195:0/3325645324 shutdown_connections 2022-06-27T11:47:59.701 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.558+0000 7fed059ac700 1 -- 172.21.15.195:0/3325645324 wait complete. 2022-06-27T11:47:59.702 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.560+0000 7fed059ac700 1 Processor -- start 2022-06-27T11:47:59.702 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.560+0000 7fed059ac700 1 -- start start 2022-06-27T11:47:59.702 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.560+0000 7f03c3fff700 1 -- 172.21.15.195:0/4105060000 <== mon.1 v2:172.21.15.202:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f03bc04f170 con 0x7f03cc06aa60 2022-06-27T11:47:59.703 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.561+0000 7fe922ffd700 1 -- 172.21.15.195:0/746803739 <== mon.2 v2:172.21.15.202:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fe92c04ee80 con 0x7fe93c0c49b0 2022-06-27T11:47:59.703 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.561+0000 7fed059ac700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fed00069cf0 0x7fed00158b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.561+0000 7fed059ac700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fed0006a0c0 0x7fed0015a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.561+0000 7fed059ac700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fed000c6590 0x7fed000c0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.561+0000 7fed059ac700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed000c3160 con 0x7fed000c6590 2022-06-27T11:47:59.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.561+0000 7fed059ac700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed0015a9c0 con 0x7fed00069cf0 2022-06-27T11:47:59.705 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.561+0000 7fed059ac700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed000c5da0 con 0x7fed0006a0c0 2022-06-27T11:47:59.705 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.561+0000 7fe975898700 1 --2- 172.21.15.195:0/264199571 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe950063d80 0x7fe950066230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.707 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.561+0000 7fecfe7fc700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fed0006a0c0 0x7fed0015a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.707 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.561+0000 7fecfe7fc700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fed0006a0c0 0x7fed0015a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3301/0 says I am v2:172.21.15.195:34314/0 (socket says 172.21.15.195:34314) 2022-06-27T11:47:59.708 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.561+0000 7fecfe7fc700 1 -- 172.21.15.195:0/29621834 learned_addr learned my addr 172.21.15.195:0/29621834 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:47:59.708 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.561+0000 7fecfeffd700 1 --2- 172.21.15.195:0/29621834 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fed00069cf0 0x7fed00158b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.562+0000 7fecff7fe700 1 --2- 172.21.15.195:0/29621834 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fed000c6590 0x7fed000c0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.562+0000 7fecfe7fc700 1 -- 172.21.15.195:0/29621834 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fed00069cf0 msgr2=0x7fed00158b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.562+0000 7fecfe7fc700 1 --2- 172.21.15.195:0/29621834 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fed00069cf0 0x7fed00158b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.710 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.562+0000 7fecfe7fc700 1 -- 172.21.15.195:0/29621834 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fed000c6590 msgr2=0x7fed000c0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.710 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.562+0000 7fecfe7fc700 1 --2- 172.21.15.195:0/29621834 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fed000c6590 0x7fed000c0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.711 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.562+0000 7fecfe7fc700 1 -- 172.21.15.195:0/29621834 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecf4009350 con 0x7fed0006a0c0 2022-06-27T11:47:59.711 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.562+0000 7fe975898700 1 --2- 172.21.15.195:0/264199571 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe950063d80 0x7fe950066230 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe97004ce30 tx=0x7fe968017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:59.713 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.562+0000 7fecfe7fc700 1 --2- 172.21.15.195:0/29621834 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fed0006a0c0 0x7fed0015a480 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fecf000e930 tx=0x7fecf0009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:59.713 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.562+0000 7fece6ffd700 1 -- 172.21.15.195:0/29621834 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf0013680 con 0x7fed0006a0c0 2022-06-27T11:47:59.714 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.562+0000 7fece6ffd700 1 -- 172.21.15.195:0/29621834 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecf00137e0 con 0x7fed0006a0c0 2022-06-27T11:47:59.714 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.562+0000 7fece6ffd700 1 -- 172.21.15.195:0/29621834 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf0013b80 con 0x7fed0006a0c0 2022-06-27T11:47:59.714 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.563+0000 7fed059ac700 1 -- 172.21.15.195:0/29621834 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed000c2c00 con 0x7fed0006a0c0 2022-06-27T11:47:59.715 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.563+0000 7fed059ac700 1 -- 172.21.15.195:0/29621834 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed00141cf0 con 0x7fed0006a0c0 2022-06-27T11:47:59.715 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.566+0000 7fece6ffd700 1 -- 172.21.15.195:0/29621834 <== mon.2 v2:172.21.15.202:3301/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fecf0019030 con 0x7fed0006a0c0 2022-06-27T11:47:59.715 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.567+0000 7fece6ffd700 1 --2- 172.21.15.195:0/29621834 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fece0063d80 0x7fece0066230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:47:59.715 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.567+0000 7fece6ffd700 1 -- 172.21.15.195:0/29621834 <== mon.2 v2:172.21.15.202:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fecf0080770 con 0x7fed0006a0c0 2022-06-27T11:47:59.716 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.568+0000 7fecfeffd700 1 --2- 172.21.15.195:0/29621834 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fece0063d80 0x7fece0066230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:47:59.716 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.562+0000 7fe95e7fc700 1 -- 172.21.15.195:0/264199571 <== mon.2 v2:172.21.15.202:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fe960052590 con 0x7fe9700680b0 2022-06-27T11:47:59.716 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.569+0000 7fe94dffb700 1 -- 172.21.15.195:0/3987743196 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fe95404e590 con 0x7fe958069cf0 2022-06-27T11:47:59.717 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.579+0000 7fecfeffd700 1 --2- 172.21.15.195:0/29621834 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fece0063d80 0x7fece0066230 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fecf4005fd0 tx=0x7fecf4015000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:47:59.717 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.579+0000 7fed059ac700 1 -- 172.21.15.195:0/29621834 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed00141b50 con 0x7fed0006a0c0 2022-06-27T11:47:59.718 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.584+0000 7fece6ffd700 1 -- 172.21.15.195:0/29621834 <== mon.2 v2:172.21.15.202:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fed00141b50 con 0x7fed0006a0c0 2022-06-27T11:47:59.929 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.930+0000 7fe235ffb700 1 -- 172.21.15.195:0/123644676 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fe24016b300 con 0x7fe2400c3010 2022-06-27T11:47:59.930 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.930+0000 7fd5412ca700 1 -- 172.21.15.195:0/2471686570 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fd53c16e8c0 con 0x7fd53c15dd20 2022-06-27T11:47:59.930 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.930+0000 7f92518ae700 1 -- 172.21.15.195:0/1059887574 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f924c1560d0 con 0x7f924c0c41b0 2022-06-27T11:47:59.931 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.930+0000 7fe23cff9700 1 -- 172.21.15.195:0/123644676 <== mon.1 v2:172.21.15.202:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe24016b300 con 0x7fe2400c3010 2022-06-27T11:47:59.931 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.930+0000 7f9232ffd700 1 -- 172.21.15.195:0/1059887574 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9240012e60 con 0x7f924c0c41b0 2022-06-27T11:47:59.932 INFO:teuthology.orchestra.run.smithi195.stdout:55834574852 2022-06-27T11:47:59.932 INFO:teuthology.orchestra.run.smithi195.stdout:55834574852 2022-06-27T11:47:59.934 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.935+0000 7f92518ae700 1 -- 172.21.15.195:0/1059887574 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f922c063db0 msgr2=0x7f922c066260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.935 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.935+0000 7f92518ae700 1 --2- 172.21.15.195:0/1059887574 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f922c063db0 0x7f922c066260 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f923c0018a0 tx=0x7f923c009040 comp rx=0 tx=0).stop 2022-06-27T11:47:59.935 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.935+0000 7f92518ae700 1 -- 172.21.15.195:0/1059887574 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f924c0c41b0 msgr2=0x7f924c0b75e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.935 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.935+0000 7f92518ae700 1 --2- 172.21.15.195:0/1059887574 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f924c0c41b0 0x7f924c0b75e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f924000d8f0 tx=0x7f9240008040 comp rx=0 tx=0).stop 2022-06-27T11:47:59.935 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.935+0000 7f92518ae700 1 -- 172.21.15.195:0/1059887574 shutdown_connections 2022-06-27T11:47:59.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.935+0000 7f92518ae700 1 --2- 172.21.15.195:0/1059887574 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f922c063db0 0x7f922c066260 secure :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f923c0018a0 tx=0x7f923c009040 comp rx=0 tx=0).stop 2022-06-27T11:47:59.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.935+0000 7f92518ae700 1 --2- 172.21.15.195:0/1059887574 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f924c0c4580 0x7f924c16acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.935+0000 7f92518ae700 1 --2- 172.21.15.195:0/1059887574 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f924c0c37b0 0x7f924c0b70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.935+0000 7f92518ae700 1 --2- 172.21.15.195:0/1059887574 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f924c0c41b0 0x7f924c0b75e0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.937 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.935+0000 7f92518ae700 1 -- 172.21.15.195:0/1059887574 >> 172.21.15.195:0/1059887574 conn(0x7f924c0c8740 msgr2=0x7f924c0bfcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:59.937 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.935+0000 7fe235ffb700 1 -- 172.21.15.195:0/123644676 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe224063db0 msgr2=0x7fe224066260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.937 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.935+0000 7fe235ffb700 1 --2- 172.21.15.195:0/123644676 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe224063db0 0x7fe224066260 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe2400b86b0 tx=0x7fe228008040 comp rx=0 tx=0).stop 2022-06-27T11:47:59.938 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.935+0000 7fe235ffb700 1 -- 172.21.15.195:0/123644676 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe2400c3010 msgr2=0x7fe24006d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.938 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.935+0000 7fe235ffb700 1 --2- 172.21.15.195:0/123644676 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe2400c3010 0x7fe24006d2c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe24006cd10 tx=0x7fe230016000 comp rx=0 tx=0).stop 2022-06-27T11:47:59.938 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.935+0000 7fd522ffd700 1 -- 172.21.15.195:0/2471686570 <== mon.1 v2:172.21.15.202:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd53c16e8c0 con 0x7fd53c15dd20 2022-06-27T11:47:59.939 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.935+0000 7fe235ffb700 1 -- 172.21.15.195:0/123644676 shutdown_connections 2022-06-27T11:47:59.939 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.935+0000 7fe235ffb700 1 --2- 172.21.15.195:0/123644676 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe224063db0 0x7fe224066260 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.939 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.935+0000 7fe235ffb700 1 --2- 172.21.15.195:0/123644676 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe2401518a0 0x7fe24014cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.940 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.935+0000 7fe235ffb700 1 --2- 172.21.15.195:0/123644676 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe2400c3010 0x7fe24006d2c0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.940 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.935+0000 7fe235ffb700 1 --2- 172.21.15.195:0/123644676 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe2400c4bf0 0x7fe2400b7150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.940 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.935+0000 7fe235ffb700 1 -- 172.21.15.195:0/123644676 >> 172.21.15.195:0/123644676 conn(0x7fe2400c8740 msgr2=0x7fe24015bcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:59.941 INFO:teuthology.orchestra.run.smithi195.stdout:55834574853 2022-06-27T11:47:59.941 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.935+0000 7fe235ffb700 1 -- 172.21.15.195:0/123644676 shutdown_connections 2022-06-27T11:47:59.941 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.935+0000 7fe235ffb700 1 -- 172.21.15.195:0/123644676 wait complete. 2022-06-27T11:47:59.942 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.936+0000 7f92518ae700 1 -- 172.21.15.195:0/1059887574 shutdown_connections 2022-06-27T11:47:59.942 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.936+0000 7f92518ae700 1 -- 172.21.15.195:0/1059887574 wait complete. 2022-06-27T11:47:59.943 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.938+0000 7fd5412ca700 1 -- 172.21.15.195:0/2471686570 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd51c063e80 msgr2=0x7fd51c066330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.943 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.938+0000 7fd5412ca700 1 --2- 172.21.15.195:0/2471686570 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd51c063e80 0x7fd51c066330 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd534001f10 tx=0x7fd534015000 comp rx=0 tx=0).stop 2022-06-27T11:47:59.943 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.938+0000 7fd5412ca700 1 -- 172.21.15.195:0/2471686570 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd53c15dd20 msgr2=0x7fd53c16ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.944 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.938+0000 7fd5412ca700 1 --2- 172.21.15.195:0/2471686570 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd53c15dd20 0x7fd53c16ac50 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd53c155330 tx=0x7fd52c007db0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.944 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.938+0000 7fd5412ca700 1 -- 172.21.15.195:0/2471686570 shutdown_connections 2022-06-27T11:47:59.944 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.938+0000 7fd5412ca700 1 --2- 172.21.15.195:0/2471686570 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd51c063e80 0x7fd51c066330 secure :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd534001f10 tx=0x7fd534015000 comp rx=0 tx=0).stop 2022-06-27T11:47:59.944 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.938+0000 7fd5412ca700 1 --2- 172.21.15.195:0/2471686570 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd528003920 0x7fd53c0cbef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.945 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.938+0000 7fd5412ca700 1 --2- 172.21.15.195:0/2471686570 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd53c15dd20 0x7fd53c16ac50 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.945 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.938+0000 7fd5412ca700 1 --2- 172.21.15.195:0/2471686570 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd53c155670 0x7fd53c16a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.945 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.938+0000 7fd5412ca700 1 -- 172.21.15.195:0/2471686570 >> 172.21.15.195:0/2471686570 conn(0x7fd53c0ccb90 msgr2=0x7fd53c151ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:59.945 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.941+0000 7fd5412ca700 1 -- 172.21.15.195:0/2471686570 shutdown_connections 2022-06-27T11:47:59.946 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.941+0000 7fe9782fd700 1 -- 172.21.15.195:0/264199571 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fe9700c9330 con 0x7fe9700680b0 2022-06-27T11:47:59.947 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.942+0000 7fd5412ca700 1 -- 172.21.15.195:0/2471686570 wait complete. 2022-06-27T11:47:59.947 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.942+0000 7fe95e7fc700 1 -- 172.21.15.195:0/264199571 <== mon.2 v2:172.21.15.202:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe960052180 con 0x7fe9700680b0 2022-06-27T11:47:59.948 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.944+0000 7f03d28aa700 1 -- 172.21.15.195:0/4105060000 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f03cc14d880 con 0x7f03cc06aa60 2022-06-27T11:47:59.948 INFO:teuthology.orchestra.run.smithi195.stdout:55834574852 2022-06-27T11:47:59.949 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.944+0000 7fe9415c3700 1 -- 172.21.15.195:0/746803739 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fe93c065d50 con 0x7fe93c0c49b0 2022-06-27T11:47:59.949 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.945+0000 7fe922ffd700 1 -- 172.21.15.195:0/746803739 <== mon.2 v2:172.21.15.202:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe92c017020 con 0x7fe93c0c49b0 2022-06-27T11:47:59.949 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.948+0000 7f03c3fff700 1 -- 172.21.15.195:0/4105060000 <== mon.1 v2:172.21.15.202:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f03bc021020 con 0x7f03cc06aa60 2022-06-27T11:47:59.950 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.945+0000 7fe9782fd700 1 -- 172.21.15.195:0/264199571 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe950063d80 msgr2=0x7fe950066230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.950 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.945+0000 7fe9782fd700 1 --2- 172.21.15.195:0/264199571 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe950063d80 0x7fe950066230 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe97004ce30 tx=0x7fe968017000 comp rx=0 tx=0).stop 2022-06-27T11:47:59.950 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.946+0000 7fe9782fd700 1 -- 172.21.15.195:0/264199571 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe9700680b0 msgr2=0x7fe97016b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.951 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.946+0000 7fe9782fd700 1 --2- 172.21.15.195:0/264199571 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe9700680b0 0x7fe97016b610 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe96000b4c0 tx=0x7fe960006f50 comp rx=0 tx=0).stop 2022-06-27T11:47:59.951 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.948+0000 7fe9782fd700 1 -- 172.21.15.195:0/264199571 shutdown_connections 2022-06-27T11:47:59.951 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.948+0000 7fe9782fd700 1 --2- 172.21.15.195:0/264199571 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe950063d80 0x7fe950066230 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.951 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.948+0000 7fe9782fd700 1 --2- 172.21.15.195:0/264199571 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe970069050 0x7fe9700c44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.952 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.948+0000 7fe9782fd700 1 --2- 172.21.15.195:0/264199571 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe9700680b0 0x7fe97016b610 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.952 INFO:teuthology.orchestra.run.smithi195.stdout:55834574853 2022-06-27T11:47:59.953 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.948+0000 7fe9782fd700 1 --2- 172.21.15.195:0/264199571 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe970068480 0x7fe970159a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.953 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.948+0000 7fe9782fd700 1 -- 172.21.15.195:0/264199571 >> 172.21.15.195:0/264199571 conn(0x7fe9700ccb90 msgr2=0x7fe97015cc50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:59.953 INFO:teuthology.orchestra.run.smithi195.stdout:55834574852 2022-06-27T11:47:59.954 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.951+0000 7fe95fb60700 1 -- 172.21.15.195:0/3987743196 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fe95813a6c0 con 0x7fe958069cf0 2022-06-27T11:47:59.954 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.952+0000 7fe94dffb700 1 -- 172.21.15.195:0/3987743196 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe954017070 con 0x7fe958069cf0 2022-06-27T11:47:59.955 INFO:teuthology.orchestra.run.smithi195.stdout:55834574852 2022-06-27T11:47:59.955 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.950+0000 7fe9782fd700 1 -- 172.21.15.195:0/264199571 shutdown_connections 2022-06-27T11:47:59.955 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.950+0000 7fe9782fd700 1 -- 172.21.15.195:0/264199571 wait complete. 2022-06-27T11:47:59.956 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.952+0000 7fe9415c3700 1 -- 172.21.15.195:0/746803739 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe91c063dd0 msgr2=0x7fe91c066280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.956 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.952+0000 7fe9415c3700 1 --2- 172.21.15.195:0/746803739 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe91c063dd0 0x7fe91c066280 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe9300097a0 tx=0x7fe930017000 comp rx=0 tx=0).stop 2022-06-27T11:47:59.956 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.952+0000 7fe9415c3700 1 -- 172.21.15.195:0/746803739 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe93c0c49b0 msgr2=0x7fe93c0bbce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.957 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.952+0000 7fe9415c3700 1 --2- 172.21.15.195:0/746803739 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe93c0c49b0 0x7fe93c0bbce0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe92c0010e0 tx=0x7fe92c007c70 comp rx=0 tx=0).stop 2022-06-27T11:47:59.957 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.952+0000 7fe9415c3700 1 -- 172.21.15.195:0/746803739 shutdown_connections 2022-06-27T11:47:59.957 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.952+0000 7fe9415c3700 1 --2- 172.21.15.195:0/746803739 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe91c063dd0 0x7fe91c066280 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.957 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.952+0000 7fe9415c3700 1 --2- 172.21.15.195:0/746803739 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe93c0c49b0 0x7fe93c0bbce0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.958 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.952+0000 7fe9415c3700 1 --2- 172.21.15.195:0/746803739 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe93c0c31f0 0x7fe93c06fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.958 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.952+0000 7fe9415c3700 1 --2- 172.21.15.195:0/746803739 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe93c0c35c0 0x7fe93c0b70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.958 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.952+0000 7fe9415c3700 1 -- 172.21.15.195:0/746803739 >> 172.21.15.195:0/746803739 conn(0x7fe93c0c8740 msgr2=0x7fe93c0bfba0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:59.959 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.953+0000 7f03d28aa700 1 -- 172.21.15.195:0/4105060000 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f03b0063d30 msgr2=0x7f03b00661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.959 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.953+0000 7f03d28aa700 1 --2- 172.21.15.195:0/4105060000 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f03b0063d30 0x7f03b00661e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f03c401ade0 tx=0x7f03c402a000 comp rx=0 tx=0).stop 2022-06-27T11:47:59.959 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.953+0000 7f03d28aa700 1 -- 172.21.15.195:0/4105060000 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f03cc06aa60 msgr2=0x7f03cc149930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.960 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.953+0000 7f03d28aa700 1 --2- 172.21.15.195:0/4105060000 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f03cc06aa60 0x7f03cc149930 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f03bc00c440 tx=0x7f03bc007e00 comp rx=0 tx=0).stop 2022-06-27T11:47:59.960 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.953+0000 7fe9415c3700 1 -- 172.21.15.195:0/746803739 shutdown_connections 2022-06-27T11:47:59.961 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.953+0000 7fe9415c3700 1 -- 172.21.15.195:0/746803739 wait complete. 2022-06-27T11:47:59.962 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.953+0000 7f03d28aa700 1 -- 172.21.15.195:0/4105060000 shutdown_connections 2022-06-27T11:47:59.962 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.953+0000 7f03d28aa700 1 --2- 172.21.15.195:0/4105060000 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f03b0063d30 0x7f03b00661e0 secure :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f03c401ade0 tx=0x7f03c402a000 comp rx=0 tx=0).stop 2022-06-27T11:47:59.962 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.953+0000 7f03d28aa700 1 --2- 172.21.15.195:0/4105060000 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f03cc06a690 0x7f03cc1573c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.963 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.953+0000 7f03d28aa700 1 --2- 172.21.15.195:0/4105060000 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f03cc06d050 0x7f03cc15d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.963 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.953+0000 7f03d28aa700 1 --2- 172.21.15.195:0/4105060000 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f03cc06aa60 0x7f03cc149930 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.963 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.953+0000 7f03d28aa700 1 -- 172.21.15.195:0/4105060000 >> 172.21.15.195:0/4105060000 conn(0x7f03cc063af0 msgr2=0x7f03cc153800 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:59.963 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.956+0000 7f03d28aa700 1 -- 172.21.15.195:0/4105060000 shutdown_connections 2022-06-27T11:47:59.964 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.956+0000 7f03d28aa700 1 -- 172.21.15.195:0/4105060000 wait complete. 2022-06-27T11:47:59.965 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.0 2022-06-27T11:47:59.966 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.4 2022-06-27T11:47:59.966 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.958+0000 7fe95fb60700 1 -- 172.21.15.195:0/3987743196 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe940063d30 msgr2=0x7fe9400661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.967 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.958+0000 7fe95fb60700 1 --2- 172.21.15.195:0/3987743196 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe940063d30 0x7fe9400661e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe958153ed0 tx=0x7fe9500097e0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.967 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.958+0000 7fe95fb60700 1 -- 172.21.15.195:0/3987743196 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe958069cf0 msgr2=0x7fe958141390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.967 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.958+0000 7fe95fb60700 1 --2- 172.21.15.195:0/3987743196 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe958069cf0 0x7fe958141390 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe95400c4c0 tx=0x7fe954007d80 comp rx=0 tx=0).stop 2022-06-27T11:47:59.967 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.963+0000 7fe95fb60700 1 -- 172.21.15.195:0/3987743196 shutdown_connections 2022-06-27T11:47:59.968 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.963+0000 7fe95fb60700 1 --2- 172.21.15.195:0/3987743196 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe940063d30 0x7fe9400661e0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.968 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.963+0000 7fe95fb60700 1 --2- 172.21.15.195:0/3987743196 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe958069cf0 0x7fe958141390 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.968 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.963+0000 7fe95fb60700 1 --2- 172.21.15.195:0/3987743196 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe9580c6590 0x7fe95813f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.968 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.963+0000 7fe95fb60700 1 --2- 172.21.15.195:0/3987743196 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe95806a0c0 0x7fe95813f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.968 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.963+0000 7fe95fb60700 1 -- 172.21.15.195:0/3987743196 >> 172.21.15.195:0/3987743196 conn(0x7fe9580c9b00 msgr2=0x7fe95806bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:59.969 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.963+0000 7fe95fb60700 1 -- 172.21.15.195:0/3987743196 shutdown_connections 2022-06-27T11:47:59.969 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.963+0000 7fe95fb60700 1 -- 172.21.15.195:0/3987743196 wait complete. 2022-06-27T11:47:59.970 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.2 2022-06-27T11:47:59.971 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.966+0000 7fed059ac700 1 -- 172.21.15.195:0/29621834 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fed00141b50 con 0x7fed0006a0c0 2022-06-27T11:47:59.971 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.967+0000 7fece6ffd700 1 -- 172.21.15.195:0/29621834 <== mon.2 v2:172.21.15.202:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fed00141b50 con 0x7fed0006a0c0 2022-06-27T11:47:59.971 INFO:teuthology.orchestra.run.smithi195.stdout:55834574852 2022-06-27T11:47:59.972 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.972+0000 7fed059ac700 1 -- 172.21.15.195:0/29621834 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fece0063d80 msgr2=0x7fece0066230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.972 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.972+0000 7fed059ac700 1 --2- 172.21.15.195:0/29621834 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fece0063d80 0x7fece0066230 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fecf4005fd0 tx=0x7fecf4015000 comp rx=0 tx=0).stop 2022-06-27T11:47:59.972 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.972+0000 7fed059ac700 1 -- 172.21.15.195:0/29621834 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fed0006a0c0 msgr2=0x7fed0015a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:47:59.973 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.972+0000 7fed059ac700 1 --2- 172.21.15.195:0/29621834 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fed0006a0c0 0x7fed0015a480 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fecf000e930 tx=0x7fecf0009060 comp rx=0 tx=0).stop 2022-06-27T11:47:59.974 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.3 2022-06-27T11:47:59.975 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.973+0000 7fed059ac700 1 -- 172.21.15.195:0/29621834 shutdown_connections 2022-06-27T11:47:59.975 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.973+0000 7fed059ac700 1 --2- 172.21.15.195:0/29621834 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fece0063d80 0x7fece0066230 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.975 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.973+0000 7fed059ac700 1 --2- 172.21.15.195:0/29621834 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fed000c6590 0x7fed000c0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.975 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.973+0000 7fed059ac700 1 --2- 172.21.15.195:0/29621834 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fed00069cf0 0x7fed00158b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.975 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.973+0000 7fed059ac700 1 --2- 172.21.15.195:0/29621834 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fed0006a0c0 0x7fed0015a480 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:47:59.976 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.973+0000 7fed059ac700 1 -- 172.21.15.195:0/29621834 >> 172.21.15.195:0/29621834 conn(0x7fed000c9b00 msgr2=0x7fed00153410 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:47:59.976 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.973+0000 7fed059ac700 1 -- 172.21.15.195:0/29621834 shutdown_connections 2022-06-27T11:47:59.976 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:47:59.973+0000 7fed059ac700 1 -- 172.21.15.195:0/29621834 wait complete. 2022-06-27T11:47:59.977 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.5 2022-06-27T11:47:59.978 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.7 2022-06-27T11:47:59.983 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.6 2022-06-27T11:47:59.988 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.1 2022-06-27T11:48:00.966 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2022-06-27T11:48:00.967 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2022-06-27T11:48:00.971 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2022-06-27T11:48:00.975 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2022-06-27T11:48:00.978 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2022-06-27T11:48:00.980 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2022-06-27T11:48:00.984 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2022-06-27T11:48:00.990 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2022-06-27T11:48:01.167 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.167+0000 7f759cf4e700 1 Processor -- start 2022-06-27T11:48:01.167 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.167+0000 7f759cf4e700 1 -- start start 2022-06-27T11:48:01.168 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.169+0000 7f759cf4e700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f75980c41b0 0x7f7598152bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:01.168 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.169+0000 7f759cf4e700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f75980c4580 0x7f7598153130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:01.169 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.169+0000 7f759cf4e700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7598155b50 con 0x7f75980c3270 2022-06-27T11:48:01.170 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.169+0000 7f759cf4e700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75981527a0 con 0x7f75980c41b0 2022-06-27T11:48:01.170 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.169+0000 7f759cf4e700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7598155df0 con 0x7f75980c4580 2022-06-27T11:48:01.170 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.169+0000 7f759659c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f75980c41b0 0x7f7598152bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:01.170 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.169+0000 7f759659c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f75980c41b0 0x7f7598152bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41564/0 (socket says 172.21.15.195:41564) 2022-06-27T11:48:01.171 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.169+0000 7f759659c700 1 -- 172.21.15.195:0/4091381608 learned_addr learned my addr 172.21.15.195:0/4091381608 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:48:01.171 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.170+0000 7f759659c700 1 -- 172.21.15.195:0/4091381608 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f75980c4580 msgr2=0x7f7598153130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:01.171 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.170+0000 7f759659c700 1 --2- 172.21.15.195:0/4091381608 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f75980c4580 0x7f7598153130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.171 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.170+0000 7f759659c700 1 -- 172.21.15.195:0/4091381608 >> v1:172.21.15.195:6789/0 conn(0x7f75980c3270 legacy=0x7f75980c0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:01.172 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.170+0000 7f759659c700 1 -- 172.21.15.195:0/4091381608 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75981560d0 con 0x7f75980c41b0 2022-06-27T11:48:01.172 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.171+0000 7f759659c700 1 --2- 172.21.15.195:0/4091381608 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f75980c41b0 0x7f7598152bf0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7588008870 tx=0x7f7588008190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1effaf2f4c8e53ae server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:01.175 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.176+0000 7f759559a700 1 -- 172.21.15.195:0/4091381608 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7588010070 con 0x7f75980c41b0 2022-06-27T11:48:01.176 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.176+0000 7f759559a700 1 -- 172.21.15.195:0/4091381608 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7588004030 con 0x7f75980c41b0 2022-06-27T11:48:01.176 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.176+0000 7f759559a700 1 -- 172.21.15.195:0/4091381608 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75880043d0 con 0x7f75980c41b0 2022-06-27T11:48:01.180 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.180+0000 7f759cf4e700 1 -- 172.21.15.195:0/4091381608 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f75980c41b0 msgr2=0x7f7598152bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:01.180 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.180+0000 7f759cf4e700 1 --2- 172.21.15.195:0/4091381608 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f75980c41b0 0x7f7598152bf0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7588008870 tx=0x7f7588008190 comp rx=0 tx=0).stop 2022-06-27T11:48:01.181 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.180+0000 7f759cf4e700 1 -- 172.21.15.195:0/4091381608 shutdown_connections 2022-06-27T11:48:01.181 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.180+0000 7f759cf4e700 1 --2- 172.21.15.195:0/4091381608 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f75980c41b0 0x7f7598152bf0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.181 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.180+0000 7f759cf4e700 1 --2- 172.21.15.195:0/4091381608 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f75980c4580 0x7f7598153130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.181 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.180+0000 7f759cf4e700 1 -- 172.21.15.195:0/4091381608 >> 172.21.15.195:0/4091381608 conn(0x7f75980c8740 msgr2=0x7f75980c8b10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:01.182 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.181+0000 7f759cf4e700 1 -- 172.21.15.195:0/4091381608 shutdown_connections 2022-06-27T11:48:01.182 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.181+0000 7f759cf4e700 1 -- 172.21.15.195:0/4091381608 wait complete. 2022-06-27T11:48:01.183 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.183+0000 7f759cf4e700 1 Processor -- start 2022-06-27T11:48:01.183 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.184+0000 7f80c2db8700 1 Processor -- start 2022-06-27T11:48:01.184 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.183+0000 7f759cf4e700 1 -- start start 2022-06-27T11:48:01.185 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.184+0000 7f759cf4e700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f75980c3270 0x7f7598150450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:01.185 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.184+0000 7f759cf4e700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f75980c41b0 0x7f75980bfeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:01.185 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.184+0000 7f759cf4e700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f75980c4580 0x7f759816c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:01.185 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.184+0000 7f759cf4e700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75980c8450 con 0x7f75980c41b0 2022-06-27T11:48:01.186 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.184+0000 7f759cf4e700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7598161770 con 0x7f75980c4580 2022-06-27T11:48:01.186 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.184+0000 7f759cf4e700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7598161a50 con 0x7f75980c3270 2022-06-27T11:48:01.186 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.184+0000 7f7595d9b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f75980c41b0 0x7f75980bfeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:01.187 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.185+0000 7f7595d9b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f75980c41b0 0x7f75980bfeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41570/0 (socket says 172.21.15.195:41570) 2022-06-27T11:48:01.187 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.185+0000 7f7595d9b700 1 -- 172.21.15.195:0/146907795 learned_addr learned my addr 172.21.15.195:0/146907795 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:48:01.187 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.185+0000 7f759659c700 1 --2- 172.21.15.195:0/146907795 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f75980c3270 0x7f7598150450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:01.187 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.185+0000 7f7595d9b700 1 -- 172.21.15.195:0/146907795 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f75980c3270 msgr2=0x7f7598150450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:01.188 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.185+0000 7f7595d9b700 1 --2- 172.21.15.195:0/146907795 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f75980c3270 0x7f7598150450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.188 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.185+0000 7f7595d9b700 1 -- 172.21.15.195:0/146907795 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f75980c4580 msgr2=0x7f759816c100 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2022-06-27T11:48:01.188 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.185+0000 7f7595d9b700 1 --2- 172.21.15.195:0/146907795 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f75980c4580 0x7f759816c100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.189 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.185+0000 7f7595d9b700 1 -- 172.21.15.195:0/146907795 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7588008430 con 0x7f75980c41b0 2022-06-27T11:48:01.190 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.186+0000 7f7595d9b700 1 --2- 172.21.15.195:0/146907795 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f75980c41b0 0x7f75980bfeb0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f759000b7d0 tx=0x7f7590009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:01.191 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.184+0000 7f80c2db8700 1 -- start start 2022-06-27T11:48:01.191 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.185+0000 7f80c2db8700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f80bc0c4580 0x7f80bc0c3270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:01.192 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.185+0000 7f80c2db8700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f80bc0c37b0 0x7f80bc0c0a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:01.192 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.185+0000 7f80c2db8700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80bc155b50 con 0x7f80bc0c41b0 2022-06-27T11:48:01.192 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.185+0000 7f80c2db8700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80bc1527a0 con 0x7f80bc0c37b0 2022-06-27T11:48:01.192 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.185+0000 7f80c2db8700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80bc0c3c30 con 0x7f80bc0c4580 2022-06-27T11:48:01.193 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.186+0000 7f80bbfff700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f80bc0c4580 0x7f80bc0c3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:01.193 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.186+0000 7f80bbfff700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f80bc0c4580 0x7f80bc0c3270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3300/0 says I am v2:172.21.15.195:39704/0 (socket says 172.21.15.195:39704) 2022-06-27T11:48:01.193 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.186+0000 7f80bbfff700 1 -- 172.21.15.195:0/2849536208 learned_addr learned my addr 172.21.15.195:0/2849536208 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:48:01.193 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.186+0000 7f80bbfff700 1 -- 172.21.15.195:0/2849536208 >> v2:172.21.15.195:3300/0 conn(0x7f80bc0c37b0 msgr2=0x7f80bc0c0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:01.194 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.186+0000 7f80bbfff700 1 --2- 172.21.15.195:0/2849536208 >> v2:172.21.15.195:3300/0 conn(0x7f80bc0c37b0 0x7f80bc0c0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.194 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.186+0000 7f80bbfff700 1 -- 172.21.15.195:0/2849536208 >> v1:172.21.15.202:6789/0 conn(0x7f80bc0c41b0 legacy=0x7f80bc152bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:01.194 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.186+0000 7f80bbfff700 1 -- 172.21.15.195:0/2849536208 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80bc155df0 con 0x7f80bc0c4580 2022-06-27T11:48:01.194 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.186+0000 7f80bbfff700 1 --2- 172.21.15.195:0/2849536208 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f80bc0c4580 0x7f80bc0c3270 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f80bc0694d0 tx=0x7f80ac0099b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=96e6e4a3c5c0ab7 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:01.195 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.186+0000 7f75867fc700 1 -- 172.21.15.195:0/146907795 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75900092e0 con 0x7f75980c41b0 2022-06-27T11:48:01.195 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.186+0000 7f75867fc700 1 -- 172.21.15.195:0/146907795 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7590007400 con 0x7f75980c41b0 2022-06-27T11:48:01.196 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.186+0000 7f75867fc700 1 -- 172.21.15.195:0/146907795 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75900077a0 con 0x7f75980c41b0 2022-06-27T11:48:01.196 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.187+0000 7f759cf4e700 1 -- 172.21.15.195:0/146907795 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7598161d30 con 0x7f75980c41b0 2022-06-27T11:48:01.196 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.187+0000 7f759cf4e700 1 -- 172.21.15.195:0/146907795 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75981622f0 con 0x7f75980c41b0 2022-06-27T11:48:01.197 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.188+0000 7f80bb7fe700 1 -- 172.21.15.195:0/2849536208 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80ac004030 con 0x7f80bc0c4580 2022-06-27T11:48:01.198 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.188+0000 7f80bb7fe700 1 -- 172.21.15.195:0/2849536208 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80ac0041b0 con 0x7f80bc0c4580 2022-06-27T11:48:01.198 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.188+0000 7f80bb7fe700 1 -- 172.21.15.195:0/2849536208 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80ac00f750 con 0x7f80bc0c4580 2022-06-27T11:48:01.198 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.189+0000 7f80c2db8700 1 -- 172.21.15.195:0/2849536208 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f80bc0c4580 msgr2=0x7f80bc0c3270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:01.199 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.189+0000 7f80c2db8700 1 --2- 172.21.15.195:0/2849536208 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f80bc0c4580 0x7f80bc0c3270 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f80bc0694d0 tx=0x7f80ac0099b0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.199 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.189+0000 7f80c2db8700 1 -- 172.21.15.195:0/2849536208 shutdown_connections 2022-06-27T11:48:01.200 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.189+0000 7f80c2db8700 1 --2- 172.21.15.195:0/2849536208 >> v2:172.21.15.195:3300/0 conn(0x7f80bc0c37b0 0x7f80bc0c0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.200 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.189+0000 7f80c2db8700 1 --2- 172.21.15.195:0/2849536208 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f80bc0c4580 0x7f80bc0c3270 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.200 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.189+0000 7f80c2db8700 1 -- 172.21.15.195:0/2849536208 >> 172.21.15.195:0/2849536208 conn(0x7f80bc0c8740 msgr2=0x7f80bc0c8b10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:01.200 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.190+0000 7f80c2db8700 1 -- 172.21.15.195:0/2849536208 shutdown_connections 2022-06-27T11:48:01.200 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.190+0000 7f80c2db8700 1 -- 172.21.15.195:0/2849536208 wait complete. 2022-06-27T11:48:01.201 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.189+0000 7f759cf4e700 1 -- 172.21.15.195:0/146907795 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7598162620 con 0x7f75980c41b0 2022-06-27T11:48:01.201 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.190+0000 7f75867fc700 1 -- 172.21.15.195:0/146907795 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f7590020070 con 0x7f75980c41b0 2022-06-27T11:48:01.202 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.190+0000 7f75867fc700 1 --2- 172.21.15.195:0/146907795 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7574063e50 0x7f7574066300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:01.202 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.190+0000 7f75867fc700 1 -- 172.21.15.195:0/146907795 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f7590080740 con 0x7f75980c41b0 2022-06-27T11:48:01.202 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.191+0000 7f80c2db8700 1 Processor -- start 2022-06-27T11:48:01.203 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.192+0000 7f759659c700 1 --2- 172.21.15.195:0/146907795 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7574063e50 0x7f7574066300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:01.203 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.192+0000 7f80c2db8700 1 -- start start 2022-06-27T11:48:01.204 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.192+0000 7f759659c700 1 --2- 172.21.15.195:0/146907795 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7574063e50 0x7f7574066300 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f75880081d0 tx=0x7f758801b000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:01.205 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.193+0000 7f80c2db8700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f80bc0c37b0 0x7f80bc0c6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:01.205 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.193+0000 7f80c2db8700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f80bc0c41b0 0x7f80bc1631d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:01.205 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.193+0000 7f80c2db8700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f80bc0c4580 0x7f80bc14d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:01.206 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.193+0000 7f80c2db8700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80bc14dee0 con 0x7f80bc0c4580 2022-06-27T11:48:01.206 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.193+0000 7f80c2db8700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80bc14dba0 con 0x7f80bc0c41b0 2022-06-27T11:48:01.206 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.193+0000 7f80c2db8700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80bc162710 con 0x7f80bc0c37b0 2022-06-27T11:48:01.206 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.193+0000 7f80c1355700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f80bc0c4580 0x7f80bc14d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:01.207 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.193+0000 7f80c1355700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f80bc0c4580 0x7f80bc14d660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41584/0 (socket says 172.21.15.195:41584) 2022-06-27T11:48:01.207 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.193+0000 7f80c1355700 1 -- 172.21.15.195:0/516673139 learned_addr learned my addr 172.21.15.195:0/516673139 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:48:01.207 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.193+0000 7f80c0b54700 1 --2- 172.21.15.195:0/516673139 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f80bc0c37b0 0x7f80bc0c6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:01.207 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.194+0000 7f80c1355700 1 -- 172.21.15.195:0/516673139 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f80bc0c37b0 msgr2=0x7f80bc0c6550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:01.208 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.194+0000 7f80c1355700 1 --2- 172.21.15.195:0/516673139 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f80bc0c37b0 0x7f80bc0c6550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.208 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.194+0000 7f80c1355700 1 -- 172.21.15.195:0/516673139 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f80bc0c41b0 msgr2=0x7f80bc1631d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2022-06-27T11:48:01.208 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.194+0000 7f80c1355700 1 --2- 172.21.15.195:0/516673139 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f80bc0c41b0 0x7f80bc1631d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.208 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.194+0000 7f80c1355700 1 -- 172.21.15.195:0/516673139 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80ac009c30 con 0x7f80bc0c4580 2022-06-27T11:48:01.208 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.194+0000 7f80c1355700 1 --2- 172.21.15.195:0/516673139 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f80bc0c4580 0x7f80bc14d660 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f80b000b4f0 tx=0x7f80b00090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:01.209 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.196+0000 7f80b8ff9700 1 -- 172.21.15.195:0/516673139 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80b0010770 con 0x7f80bc0c4580 2022-06-27T11:48:01.209 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.196+0000 7f80b8ff9700 1 -- 172.21.15.195:0/516673139 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80b00108d0 con 0x7f80bc0c4580 2022-06-27T11:48:01.209 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.196+0000 7f80b8ff9700 1 -- 172.21.15.195:0/516673139 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80b0010c90 con 0x7f80bc0c4580 2022-06-27T11:48:01.210 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.197+0000 7f75867fc700 1 -- 172.21.15.195:0/146907795 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f759004f2c0 con 0x7f75980c41b0 2022-06-27T11:48:01.210 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.197+0000 7f80c2db8700 1 -- 172.21.15.195:0/516673139 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80bc14e160 con 0x7f80bc0c4580 2022-06-27T11:48:01.210 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.197+0000 7f80c2db8700 1 -- 172.21.15.195:0/516673139 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80bc1629d0 con 0x7f80bc0c4580 2022-06-27T11:48:01.211 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.198+0000 7f80c2db8700 1 -- 172.21.15.195:0/516673139 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80bc0722c0 con 0x7f80bc0c4580 2022-06-27T11:48:01.218 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.217+0000 7f80b8ff9700 1 -- 172.21.15.195:0/516673139 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f80b001f030 con 0x7f80bc0c4580 2022-06-27T11:48:01.219 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.218+0000 7f80b8ff9700 1 --2- 172.21.15.195:0/516673139 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f80a8063ef0 0x7f80a80663a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:01.219 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.218+0000 7f80b8ff9700 1 -- 172.21.15.195:0/516673139 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f80b0084850 con 0x7f80bc0c4580 2022-06-27T11:48:01.219 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.218+0000 7f80b8ff9700 1 -- 172.21.15.195:0/516673139 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f80b00add30 con 0x7f80bc0c4580 2022-06-27T11:48:01.219 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.218+0000 7f80c0b54700 1 --2- 172.21.15.195:0/516673139 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f80a8063ef0 0x7f80a80663a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:01.220 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.219+0000 7f80c0b54700 1 --2- 172.21.15.195:0/516673139 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f80a8063ef0 0x7f80a80663a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f80b4007fd0 tx=0x7f80b4009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:01.229 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.226+0000 7f08e15ee700 1 Processor -- start 2022-06-27T11:48:01.229 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.226+0000 7f08e15ee700 1 -- start start 2022-06-27T11:48:01.230 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.227+0000 7f08e15ee700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f08dc069cf0 0x7f08dc15b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:01.230 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.227+0000 7f08e15ee700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f08dc0c6590 0x7f08dc0c4190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:01.230 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.227+0000 7f08e15ee700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08dc067ed0 con 0x7f08dc06a0c0 2022-06-27T11:48:01.231 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.227+0000 7f08e15ee700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08dc068120 con 0x7f08dc0c6590 2022-06-27T11:48:01.231 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.227+0000 7f08e15ee700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08dc068400 con 0x7f08dc069cf0 2022-06-27T11:48:01.231 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.228+0000 7f08daffd700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f08dc069cf0 0x7f08dc15b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:01.232 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.228+0000 7f08daffd700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f08dc069cf0 0x7f08dc15b310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3300/0 says I am v2:172.21.15.195:39722/0 (socket says 172.21.15.195:39722) 2022-06-27T11:48:01.232 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.228+0000 7f08daffd700 1 -- 172.21.15.195:0/3054763395 learned_addr learned my addr 172.21.15.195:0/3054763395 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:48:01.233 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.229+0000 7f08daffd700 1 -- 172.21.15.195:0/3054763395 >> v2:172.21.15.195:3300/0 conn(0x7f08dc0c6590 msgr2=0x7f08dc0c4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:01.233 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.229+0000 7f08daffd700 1 --2- 172.21.15.195:0/3054763395 >> v2:172.21.15.195:3300/0 conn(0x7f08dc0c6590 0x7f08dc0c4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.233 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.229+0000 7f08daffd700 1 -- 172.21.15.195:0/3054763395 >> v1:172.21.15.195:6789/0 conn(0x7f08dc06a0c0 legacy=0x7f08dc154650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:01.234 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.229+0000 7f08daffd700 1 -- 172.21.15.195:0/3054763395 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08dc063af0 con 0x7f08dc069cf0 2022-06-27T11:48:01.234 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.229+0000 7f08daffd700 1 --2- 172.21.15.195:0/3054763395 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f08dc069cf0 0x7f08dc15b310 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f08dc06f840 tx=0x7f08d4009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=548f6cb0561519db server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:01.236 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.223+0000 7f93d8070700 1 Processor -- start 2022-06-27T11:48:01.237 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.226+0000 7f93d8070700 1 -- start start 2022-06-27T11:48:01.237 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.227+0000 7f93d8070700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f93d00c41b0 0x7f93d0152bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:01.237 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.227+0000 7f93d8070700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f93d00c3980 0x7f93d0153130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:01.237 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.227+0000 7f93d8070700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93d0155b50 con 0x7f93d00c4580 2022-06-27T11:48:01.238 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.227+0000 7f93d8070700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93d01527a0 con 0x7f93d00c3980 2022-06-27T11:48:01.238 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.227+0000 7f93d8070700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93d0155df0 con 0x7f93d00c41b0 2022-06-27T11:48:01.238 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.228+0000 7f93d5e0c700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f93d00c41b0 0x7f93d0152bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:01.239 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.228+0000 7f93d5e0c700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f93d00c41b0 0x7f93d0152bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3301/0 says I am v2:172.21.15.195:34348/0 (socket says 172.21.15.195:34348) 2022-06-27T11:48:01.239 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.228+0000 7f93d5e0c700 1 -- 172.21.15.195:0/2348468545 learned_addr learned my addr 172.21.15.195:0/2348468545 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:48:01.239 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.229+0000 7f93d5e0c700 1 -- 172.21.15.195:0/2348468545 >> v2:172.21.15.195:3300/0 conn(0x7f93d00c3980 msgr2=0x7f93d0153130 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2022-06-27T11:48:01.239 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.229+0000 7f93d5e0c700 1 --2- 172.21.15.195:0/2348468545 >> v2:172.21.15.195:3300/0 conn(0x7f93d00c3980 0x7f93d0153130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.239 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.229+0000 7f93d5e0c700 1 -- 172.21.15.195:0/2348468545 >> v1:172.21.15.195:6789/0 conn(0x7f93d00c4580 legacy=0x7f93d00c3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:01.240 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.229+0000 7f93d5e0c700 1 -- 172.21.15.195:0/2348468545 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93d01560d0 con 0x7f93d00c41b0 2022-06-27T11:48:01.240 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.229+0000 7f93d5e0c700 1 --2- 172.21.15.195:0/2348468545 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f93d00c41b0 0x7f93d0152bf0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f93cc00a910 tx=0x7f93cc00a1b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=391fdd41d16f78f9 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:01.240 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.229+0000 7f93d4e0a700 1 -- 172.21.15.195:0/2348468545 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93cc00ab20 con 0x7f93d00c41b0 2022-06-27T11:48:01.240 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.229+0000 7f93d4e0a700 1 -- 172.21.15.195:0/2348468545 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93cc00ac80 con 0x7f93d00c41b0 2022-06-27T11:48:01.240 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.229+0000 7f93d4e0a700 1 -- 172.21.15.195:0/2348468545 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93cc004030 con 0x7f93d00c41b0 2022-06-27T11:48:01.241 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.230+0000 7f93d8070700 1 -- 172.21.15.195:0/2348468545 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f93d00c41b0 msgr2=0x7f93d0152bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:01.241 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.230+0000 7f93d8070700 1 --2- 172.21.15.195:0/2348468545 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f93d00c41b0 0x7f93d0152bf0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f93cc00a910 tx=0x7f93cc00a1b0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.241 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.230+0000 7f93d8070700 1 -- 172.21.15.195:0/2348468545 shutdown_connections 2022-06-27T11:48:01.241 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.230+0000 7f93d8070700 1 --2- 172.21.15.195:0/2348468545 >> v2:172.21.15.195:3300/0 conn(0x7f93d00c3980 0x7f93d0153130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.242 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.230+0000 7f93d8070700 1 --2- 172.21.15.195:0/2348468545 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f93d00c41b0 0x7f93d0152bf0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.242 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.230+0000 7f93d8070700 1 -- 172.21.15.195:0/2348468545 >> 172.21.15.195:0/2348468545 conn(0x7f93d00c8740 msgr2=0x7f93d00c8b10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:01.243 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.231+0000 7f08d9ffb700 1 -- 172.21.15.195:0/3054763395 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d400e070 con 0x7f08dc069cf0 2022-06-27T11:48:01.243 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.231+0000 7f08d9ffb700 1 -- 172.21.15.195:0/3054763395 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08d400ba90 con 0x7f08dc069cf0 2022-06-27T11:48:01.243 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.231+0000 7f08d9ffb700 1 -- 172.21.15.195:0/3054763395 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d40092e0 con 0x7f08dc069cf0 2022-06-27T11:48:01.244 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.232+0000 7f08e15ee700 1 -- 172.21.15.195:0/3054763395 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f08dc069cf0 msgr2=0x7f08dc15b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:01.244 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.232+0000 7f08e15ee700 1 --2- 172.21.15.195:0/3054763395 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f08dc069cf0 0x7f08dc15b310 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f08dc06f840 tx=0x7f08d4009840 comp rx=0 tx=0).stop 2022-06-27T11:48:01.245 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.231+0000 7f93d8070700 1 -- 172.21.15.195:0/2348468545 shutdown_connections 2022-06-27T11:48:01.245 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.232+0000 7f93d8070700 1 -- 172.21.15.195:0/2348468545 wait complete. 2022-06-27T11:48:01.246 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.232+0000 7f08e15ee700 1 -- 172.21.15.195:0/3054763395 shutdown_connections 2022-06-27T11:48:01.247 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.232+0000 7f08e15ee700 1 --2- 172.21.15.195:0/3054763395 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f08dc069cf0 0x7f08dc15b310 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.247 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.232+0000 7f08e15ee700 1 --2- 172.21.15.195:0/3054763395 >> v2:172.21.15.195:3300/0 conn(0x7f08dc0c6590 0x7f08dc0c4190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.247 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.232+0000 7f08e15ee700 1 -- 172.21.15.195:0/3054763395 >> 172.21.15.195:0/3054763395 conn(0x7f08dc0c9b00 msgr2=0x7f08dc0c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:01.248 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.232+0000 7f08e15ee700 1 -- 172.21.15.195:0/3054763395 shutdown_connections 2022-06-27T11:48:01.248 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.232+0000 7f08e15ee700 1 -- 172.21.15.195:0/3054763395 wait complete. 2022-06-27T11:48:01.248 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.233+0000 7f93d8070700 1 Processor -- start 2022-06-27T11:48:01.249 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.233+0000 7f08e15ee700 1 Processor -- start 2022-06-27T11:48:01.249 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.234+0000 7f08e15ee700 1 -- start start 2022-06-27T11:48:01.250 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.233+0000 7f93d8070700 1 -- start start 2022-06-27T11:48:01.250 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.234+0000 7f93d8070700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f93d00c3980 0x7f93d00b7680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:01.250 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.234+0000 7f93d8070700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f93d00c41b0 0x7f93d016eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:01.251 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.234+0000 7f93d8070700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f93d00c4580 0x7f93d01516f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:01.251 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.234+0000 7f93d8070700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93d0148c60 con 0x7f93d00c3980 2022-06-27T11:48:01.251 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.235+0000 7f93d8070700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93d00b7c80 con 0x7f93d00c4580 2022-06-27T11:48:01.251 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.235+0000 7f93d8070700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93d0148050 con 0x7f93d00c41b0 2022-06-27T11:48:01.252 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.235+0000 7f08e15ee700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f08dc069cf0 0x7f08dc15f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:01.252 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.235+0000 7f08e15ee700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f08dc06a0c0 0x7f08dc0c4a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:01.253 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.235+0000 7f08e15ee700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f08dc0c6590 0x7f08dc0c1c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:01.253 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.235+0000 7f08e15ee700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08dc16ad30 con 0x7f08dc069cf0 2022-06-27T11:48:01.253 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.235+0000 7f08e15ee700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08dc0c33d0 con 0x7f08dc0c6590 2022-06-27T11:48:01.253 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.235+0000 7f08e15ee700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08dc16bb50 con 0x7f08dc06a0c0 2022-06-27T11:48:01.254 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.235+0000 7f08daffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f08dc069cf0 0x7f08dc15f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:01.254 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.235+0000 7f08daffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f08dc069cf0 0x7f08dc15f500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41608/0 (socket says 172.21.15.195:41608) 2022-06-27T11:48:01.255 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.235+0000 7f08daffd700 1 -- 172.21.15.195:0/2140847956 learned_addr learned my addr 172.21.15.195:0/2140847956 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:48:01.255 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.235+0000 7f08db7fe700 1 --2- 172.21.15.195:0/2140847956 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f08dc0c6590 0x7f08dc0c1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:01.255 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.235+0000 7f08da7fc700 1 --2- 172.21.15.195:0/2140847956 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f08dc06a0c0 0x7f08dc0c4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:01.256 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.235+0000 7f93d660d700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f93d00c4580 0x7f93d01516f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:01.257 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.235+0000 7f93d560b700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f93d00c41b0 0x7f93d016eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:01.257 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.235+0000 7f93d560b700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f93d00c41b0 0x7f93d016eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3301/0 says I am v2:172.21.15.195:34358/0 (socket says 172.21.15.195:34358) 2022-06-27T11:48:01.257 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.235+0000 7f93d560b700 1 -- 172.21.15.195:0/2165924106 learned_addr learned my addr 172.21.15.195:0/2165924106 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:48:01.258 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.235+0000 7f93d5e0c700 1 --2- 172.21.15.195:0/2165924106 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f93d00c3980 0x7f93d00b7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:01.258 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.236+0000 7f08db7fe700 1 -- 172.21.15.195:0/2140847956 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f08dc06a0c0 msgr2=0x7f08dc0c4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:01.259 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.236+0000 7f08db7fe700 1 --2- 172.21.15.195:0/2140847956 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f08dc06a0c0 0x7f08dc0c4a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.259 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.236+0000 7f08db7fe700 1 -- 172.21.15.195:0/2140847956 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f08dc069cf0 msgr2=0x7f08dc15f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:01.259 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.236+0000 7f08db7fe700 1 --2- 172.21.15.195:0/2140847956 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f08dc069cf0 0x7f08dc15f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.259 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.236+0000 7f08db7fe700 1 -- 172.21.15.195:0/2140847956 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08d4009ae0 con 0x7f08dc0c6590 2022-06-27T11:48:01.261 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.236+0000 7f93d560b700 1 -- 172.21.15.195:0/2165924106 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f93d00c4580 msgr2=0x7f93d01516f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:01.261 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.236+0000 7f93d560b700 1 --2- 172.21.15.195:0/2165924106 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f93d00c4580 0x7f93d01516f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.262 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.236+0000 7f93d560b700 1 -- 172.21.15.195:0/2165924106 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f93d00c3980 msgr2=0x7f93d00b7680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:01.262 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.236+0000 7f93d560b700 1 --2- 172.21.15.195:0/2165924106 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f93d00c3980 0x7f93d00b7680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.262 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.236+0000 7f93d560b700 1 -- 172.21.15.195:0/2165924106 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93cc00a450 con 0x7f93d00c41b0 2022-06-27T11:48:01.263 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.236+0000 7f93d5e0c700 1 --2- 172.21.15.195:0/2165924106 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f93d00c3980 0x7f93d00b7680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:48:01.263 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.236+0000 7f93d560b700 1 --2- 172.21.15.195:0/2165924106 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f93d00c41b0 0x7f93d016eec0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f93c800b790 tx=0x7f93c8009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:01.264 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.236+0000 7f08db7fe700 1 --2- 172.21.15.195:0/2140847956 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f08dc0c6590 0x7f08dc0c1c00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f08d000c490 tx=0x7f08d0007d40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:01.264 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.236+0000 7f704747c700 1 Processor -- start 2022-06-27T11:48:01.266 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.236+0000 7f08c2ffd700 1 -- 172.21.15.195:0/2140847956 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d00050d0 con 0x7f08dc0c6590 2022-06-27T11:48:01.266 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.236+0000 7f08e15ee700 1 -- 172.21.15.195:0/2140847956 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08dc16af40 con 0x7f08dc0c6590 2022-06-27T11:48:01.267 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.236+0000 7f08e15ee700 1 -- 172.21.15.195:0/2140847956 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08dc16b500 con 0x7f08dc0c6590 2022-06-27T11:48:01.267 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.237+0000 7f08e15ee700 1 -- 172.21.15.195:0/2140847956 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08dc0722c0 con 0x7f08dc0c6590 2022-06-27T11:48:01.269 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.236+0000 7f704747c700 1 -- start start 2022-06-27T11:48:01.269 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.237+0000 7f704747c700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f70400c4580 0x7f70400c3270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:01.270 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.237+0000 7f704747c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f70400c37b0 0x7f70400c0a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:01.270 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.237+0000 7f704747c700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7040155b50 con 0x7f70400c41b0 2022-06-27T11:48:01.271 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.237+0000 7f704747c700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70401527a0 con 0x7f70400c37b0 2022-06-27T11:48:01.271 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.237+0000 7f704747c700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70400c3c30 con 0x7f70400c4580 2022-06-27T11:48:01.271 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.238+0000 7f7045218700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f70400c41b0 0x7f7040152bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:47802/0 (socket says 172.21.15.195:47802) 2022-06-27T11:48:01.272 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.238+0000 7f7045218700 1 -- 172.21.15.195:0/4234889812 learned_addr learned my addr 172.21.15.195:0/4234889812 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:48:01.272 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.238+0000 7f7045a19700 1 --2- 172.21.15.195:0/4234889812 >> v2:172.21.15.195:3300/0 conn(0x7f70400c37b0 0x7f70400c0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:01.273 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.238+0000 7f08c2ffd700 1 -- 172.21.15.195:0/2140847956 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08d0005250 con 0x7f08dc0c6590 2022-06-27T11:48:01.274 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.238+0000 7f08c2ffd700 1 -- 172.21.15.195:0/2140847956 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d00053c0 con 0x7f08dc0c6590 2022-06-27T11:48:01.275 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.238+0000 7f7045a19700 1 -- 172.21.15.195:0/4234889812 >> v2:172.21.15.202:3300/0 conn(0x7f70400c4580 msgr2=0x7f70400c3270 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2022-06-27T11:48:01.275 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.238+0000 7f7045a19700 1 --2- 172.21.15.195:0/4234889812 >> v2:172.21.15.202:3300/0 conn(0x7f70400c4580 0x7f70400c3270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.276 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.238+0000 7f7045a19700 1 -- 172.21.15.195:0/4234889812 >> v1:172.21.15.195:6789/0 conn(0x7f70400c41b0 legacy=0x7f7040152bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:01.276 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.238+0000 7f7045a19700 1 -- 172.21.15.195:0/4234889812 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7040155df0 con 0x7f70400c37b0 2022-06-27T11:48:01.277 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.238+0000 7f93bdffb700 1 -- 172.21.15.195:0/2165924106 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c8011690 con 0x7f93d00c41b0 2022-06-27T11:48:01.277 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.238+0000 7f93bdffb700 1 -- 172.21.15.195:0/2165924106 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93c80117f0 con 0x7f93d00c41b0 2022-06-27T11:48:01.278 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.238+0000 7f93d8070700 1 -- 172.21.15.195:0/2165924106 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93d0148330 con 0x7f93d00c41b0 2022-06-27T11:48:01.278 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.238+0000 7f93bdffb700 1 -- 172.21.15.195:0/2165924106 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c8011bb0 con 0x7f93d00c41b0 2022-06-27T11:48:01.279 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.238+0000 7f93d8070700 1 -- 172.21.15.195:0/2165924106 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93d01488f0 con 0x7f93d00c41b0 2022-06-27T11:48:01.280 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.239+0000 7f7045a19700 1 --2- 172.21.15.195:0/4234889812 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f70400c37b0 0x7f70400c0a20 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f70400694d0 tx=0x7f703c0080e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e1b38be61e8a592 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:01.281 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.239+0000 7f4c48870700 1 Processor -- start 2022-06-27T11:48:01.282 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.239+0000 7f4c48870700 1 -- start start 2022-06-27T11:48:01.284 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.239+0000 7f702ffff700 1 -- 172.21.15.195:0/4234889812 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f703c010070 con 0x7f70400c37b0 2022-06-27T11:48:01.284 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.239+0000 7f702ffff700 1 -- 172.21.15.195:0/4234889812 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f703c004030 con 0x7f70400c37b0 2022-06-27T11:48:01.284 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.239+0000 7f702ffff700 1 -- 172.21.15.195:0/4234889812 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f703c0043d0 con 0x7f70400c37b0 2022-06-27T11:48:01.285 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.240+0000 7f704747c700 1 -- 172.21.15.195:0/4234889812 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f70400c37b0 msgr2=0x7f70400c0a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:01.285 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.240+0000 7f704747c700 1 --2- 172.21.15.195:0/4234889812 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f70400c37b0 0x7f70400c0a20 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f70400694d0 tx=0x7f703c0080e0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.286 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.240+0000 7f704747c700 1 -- 172.21.15.195:0/4234889812 shutdown_connections 2022-06-27T11:48:01.286 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.240+0000 7f704747c700 1 --2- 172.21.15.195:0/4234889812 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f70400c37b0 0x7f70400c0a20 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.287 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.241+0000 7f704747c700 1 --2- 172.21.15.195:0/4234889812 >> v2:172.21.15.202:3300/0 conn(0x7f70400c4580 0x7f70400c3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.287 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.241+0000 7f704747c700 1 -- 172.21.15.195:0/4234889812 >> 172.21.15.195:0/4234889812 conn(0x7f70400c8740 msgr2=0x7f70400c8b10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:01.288 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.241+0000 7f704747c700 1 -- 172.21.15.195:0/4234889812 shutdown_connections 2022-06-27T11:48:01.288 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.241+0000 7f704747c700 1 -- 172.21.15.195:0/4234889812 wait complete. 2022-06-27T11:48:01.289 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.240+0000 7f08c2ffd700 1 -- 172.21.15.195:0/2140847956 <== mon.1 v2:172.21.15.202:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f08d0019030 con 0x7f08dc0c6590 2022-06-27T11:48:01.289 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.240+0000 7f08c2ffd700 1 --2- 172.21.15.195:0/2140847956 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f08bc063d30 0x7f08bc0661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:01.290 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.240+0000 7f08c2ffd700 1 -- 172.21.15.195:0/2140847956 <== mon.1 v2:172.21.15.202:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f08d007fce0 con 0x7f08dc0c6590 2022-06-27T11:48:01.290 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.240+0000 7f08daffd700 1 --2- 172.21.15.195:0/2140847956 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f08bc063d30 0x7f08bc0661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:01.291 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.241+0000 7f08daffd700 1 --2- 172.21.15.195:0/2140847956 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f08bc063d30 0x7f08bc0661e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f08d40011d0 tx=0x7f08d4017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:01.292 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.241+0000 7f4c48870700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f4c44154c90 0x7f4c441580c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:01.293 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.241+0000 7f4c48870700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f4c44155060 0x7f4c440c1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:01.293 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.241+0000 7f4c48870700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c44157c30 con 0x7f4c44154310 2022-06-27T11:48:01.294 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.241+0000 7f4c48870700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c44151dd0 con 0x7f4c44155060 2022-06-27T11:48:01.294 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.241+0000 7f4c48870700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c44157900 con 0x7f4c44154c90 2022-06-27T11:48:01.295 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.241+0000 7f4c41d9b700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f4c44155060 0x7f4c440c1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:01.295 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.241+0000 7f4c41d9b700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f4c44155060 0x7f4c440c1dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41622/0 (socket says 172.21.15.195:41622) 2022-06-27T11:48:01.296 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.241+0000 7f4c41d9b700 1 -- 172.21.15.195:0/2982400017 learned_addr learned my addr 172.21.15.195:0/2982400017 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:48:01.297 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.242+0000 7f4c41d9b700 1 -- 172.21.15.195:0/2982400017 >> v2:172.21.15.202:3300/0 conn(0x7f4c44154c90 msgr2=0x7f4c441580c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2022-06-27T11:48:01.297 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.242+0000 7f4c41d9b700 1 --2- 172.21.15.195:0/2982400017 >> v2:172.21.15.202:3300/0 conn(0x7f4c44154c90 0x7f4c441580c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.298 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.242+0000 7f4c41d9b700 1 -- 172.21.15.195:0/2982400017 >> v1:172.21.15.202:6789/0 conn(0x7f4c44154310 legacy=0x7f4c4406aca0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2022-06-27T11:48:01.298 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.242+0000 7f4c41d9b700 1 -- 172.21.15.195:0/2982400017 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c440c39a0 con 0x7f4c44155060 2022-06-27T11:48:01.298 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.242+0000 7f4c41d9b700 1 --2- 172.21.15.195:0/2982400017 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4c44155060 0x7f4c440c1dd0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f4c4406caf0 tx=0x7f4c3c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5131b7d069702da7 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:01.299 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.242+0000 7f93bdffb700 1 -- 172.21.15.195:0/2165924106 <== mon.2 v2:172.21.15.202:3301/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f93c8017020 con 0x7f93d00c41b0 2022-06-27T11:48:01.300 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.242+0000 7f704747c700 1 Processor -- start 2022-06-27T11:48:01.301 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.242+0000 7f704747c700 1 -- start start 2022-06-27T11:48:01.302 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.242+0000 7f93bdffb700 1 --2- 172.21.15.195:0/2165924106 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f93b0063e80 0x7f93b0066330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:01.302 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.243+0000 7f93bdffb700 1 -- 172.21.15.195:0/2165924106 <== mon.2 v2:172.21.15.202:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f93c80803c0 con 0x7f93d00c41b0 2022-06-27T11:48:01.303 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.243+0000 7f4c4159a700 1 -- 172.21.15.195:0/2982400017 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c3c00e070 con 0x7f4c44155060 2022-06-27T11:48:01.304 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.243+0000 7f4c4159a700 1 -- 172.21.15.195:0/2982400017 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c3c00baa0 con 0x7f4c44155060 2022-06-27T11:48:01.304 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.243+0000 7f4c4159a700 1 -- 172.21.15.195:0/2982400017 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c3c0092e0 con 0x7f4c44155060 2022-06-27T11:48:01.305 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.243+0000 7f93d5e0c700 1 --2- 172.21.15.195:0/2165924106 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f93b0063e80 0x7f93b0066330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:01.306 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.243+0000 7f704747c700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f70400c37b0 0x7f704014f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:01.307 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.243+0000 7f704747c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f70400c41b0 0x7f7040169c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:01.307 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.243+0000 7f704747c700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f70400c4580 0x7f704016a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:01.308 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.243+0000 7f704747c700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f704016a6d0 con 0x7f70400c41b0 2022-06-27T11:48:01.308 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.243+0000 7f704747c700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f704016a8c0 con 0x7f70400c37b0 2022-06-27T11:48:01.309 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.243+0000 7f704747c700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f704014f180 con 0x7f70400c4580 2022-06-27T11:48:01.309 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.244+0000 7f7045a19700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f70400c4580 0x7f704016a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:01.310 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.244+0000 7f7045a19700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f70400c4580 0x7f704016a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3301/0 says I am v2:172.21.15.195:34386/0 (socket says 172.21.15.195:34386) 2022-06-27T11:48:01.310 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.244+0000 7f7045a19700 1 -- 172.21.15.195:0/3890985076 learned_addr learned my addr 172.21.15.195:0/3890985076 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:48:01.310 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.244+0000 7f7045218700 1 --2- 172.21.15.195:0/3890985076 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f70400c37b0 0x7f704014f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:01.311 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.244+0000 7f08c2ffd700 1 -- 172.21.15.195:0/2140847956 <== mon.1 v2:172.21.15.202:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f08d004e950 con 0x7f08dc0c6590 2022-06-27T11:48:01.312 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.244+0000 7f4c48870700 1 -- 172.21.15.195:0/2982400017 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4c44155060 msgr2=0x7f4c440c1dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:01.313 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.244+0000 7f4c48870700 1 --2- 172.21.15.195:0/2982400017 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4c44155060 0x7f4c440c1dd0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f4c4406caf0 tx=0x7f4c3c009840 comp rx=0 tx=0).stop 2022-06-27T11:48:01.313 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.244+0000 7f93d5e0c700 1 --2- 172.21.15.195:0/2165924106 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f93b0063e80 0x7f93b0066330 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f93cc00a910 tx=0x7f93cc01a000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:01.314 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.244+0000 7f7045a19700 1 -- 172.21.15.195:0/3890985076 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f70400c37b0 msgr2=0x7f704014f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:01.315 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.244+0000 7f7045a19700 1 --2- 172.21.15.195:0/3890985076 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f70400c37b0 0x7f704014f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.315 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.244+0000 7f7045a19700 1 -- 172.21.15.195:0/3890985076 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f70400c41b0 msgr2=0x7f7040169c50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2022-06-27T11:48:01.316 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.244+0000 7f7045a19700 1 --2- 172.21.15.195:0/3890985076 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f70400c41b0 0x7f7040169c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.316 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.244+0000 7f7045a19700 1 -- 172.21.15.195:0/3890985076 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f703c008360 con 0x7f70400c4580 2022-06-27T11:48:01.317 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.245+0000 7f93d8070700 1 -- 172.21.15.195:0/2165924106 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93d0148ab0 con 0x7f93d00c41b0 2022-06-27T11:48:01.318 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.245+0000 7f4c48870700 1 -- 172.21.15.195:0/2982400017 shutdown_connections 2022-06-27T11:48:01.319 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.245+0000 7f4c48870700 1 --2- 172.21.15.195:0/2982400017 >> v2:172.21.15.202:3300/0 conn(0x7f4c44154c90 0x7f4c441580c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.319 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.245+0000 7f4c48870700 1 --2- 172.21.15.195:0/2982400017 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4c44155060 0x7f4c440c1dd0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.320 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.245+0000 7f4c48870700 1 -- 172.21.15.195:0/2982400017 >> 172.21.15.195:0/2982400017 conn(0x7f4c440c8740 msgr2=0x7f4c440c8b10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:01.320 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.245+0000 7f4c48870700 1 -- 172.21.15.195:0/2982400017 shutdown_connections 2022-06-27T11:48:01.321 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.245+0000 7f4c48870700 1 -- 172.21.15.195:0/2982400017 wait complete. 2022-06-27T11:48:01.322 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.244+0000 7f150e72d700 1 Processor -- start 2022-06-27T11:48:01.323 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.244+0000 7f150e72d700 1 -- start start 2022-06-27T11:48:01.324 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.245+0000 7f7045a19700 1 --2- 172.21.15.195:0/3890985076 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f70400c4580 0x7f704016a190 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f703c008120 tx=0x7f703c01b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:01.326 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.246+0000 7f150e72d700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f15080c0d60 0x7f15080b5f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:01.326 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.246+0000 7f150e72d700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f15080c2610 0x7f1508166b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:01.327 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.246+0000 7f150e72d700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15080c08d0 con 0x7f15080c4ae0 2022-06-27T11:48:01.327 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.246+0000 7f150e72d700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15080c3660 con 0x7f15080c0d60 2022-06-27T11:48:01.327 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.246+0000 7f150e72d700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15080bffa0 con 0x7f15080c2610 2022-06-27T11:48:01.328 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.246+0000 7f1507fff700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f15080c4ae0 0x7f15080c3b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:47822/0 (socket says 172.21.15.195:47822) 2022-06-27T11:48:01.329 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.246+0000 7f1507fff700 1 -- 172.21.15.195:0/3756924483 learned_addr learned my addr 172.21.15.195:0/3756924483 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:48:01.329 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.246+0000 7f150ccca700 1 --2- 172.21.15.195:0/3756924483 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f15080c2610 0x7f1508166b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:01.329 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.246+0000 7f15077fe700 1 --2- 172.21.15.195:0/3756924483 >> v2:172.21.15.202:3300/0 conn(0x7f15080c0d60 0x7f15080b5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:01.330 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.246+0000 7f150ccca700 1 -- 172.21.15.195:0/3756924483 >> v2:172.21.15.202:3300/0 conn(0x7f15080c0d60 msgr2=0x7f15080b5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:01.330 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.246+0000 7f150ccca700 1 --2- 172.21.15.195:0/3756924483 >> v2:172.21.15.202:3300/0 conn(0x7f15080c0d60 0x7f15080b5f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.331 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.246+0000 7f150ccca700 1 -- 172.21.15.195:0/3756924483 >> v1:172.21.15.195:6789/0 conn(0x7f15080c4ae0 legacy=0x7f15080c3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:01.331 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.246+0000 7f150ccca700 1 -- 172.21.15.195:0/3756924483 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15080c0280 con 0x7f15080c2610 2022-06-27T11:48:01.331 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.247+0000 7f150ccca700 1 --2- 172.21.15.195:0/3756924483 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f15080c2610 0x7f1508166b70 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f14fc0097a0 tx=0x7f14fc009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=643db6da8e6a59ba server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:01.335 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.247+0000 7f4c48870700 1 Processor -- start 2022-06-27T11:48:01.336 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.247+0000 7f4c48870700 1 -- start start 2022-06-27T11:48:01.336 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.248+0000 7f4c48870700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4c44154310 0x7f4c4414f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:01.337 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.248+0000 7f4c48870700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4c44154c90 0x7f4c441564a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:01.337 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.248+0000 7f4c48870700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4c4416ee40 0x7f4c4416f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:01.338 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.248+0000 7f4c48870700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c440bcba0 con 0x7f4c44154310 2022-06-27T11:48:01.338 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.248+0000 7f4c48870700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c440bfbc0 con 0x7f4c4416ee40 2022-06-27T11:48:01.339 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.248+0000 7f4c48870700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c4416c180 con 0x7f4c44154c90 2022-06-27T11:48:01.339 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.248+0000 7f4c4259c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4c44154310 0x7f4c4414f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:01.340 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.248+0000 7f4c4259c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4c44154310 0x7f4c4414f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41640/0 (socket says 172.21.15.195:41640) 2022-06-27T11:48:01.340 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.248+0000 7f4c4259c700 1 -- 172.21.15.195:0/3157918351 learned_addr learned my addr 172.21.15.195:0/3157918351 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:48:01.340 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.248+0000 7f4c41d9b700 1 --2- 172.21.15.195:0/3157918351 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4c44154c90 0x7f4c441564a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:01.341 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.249+0000 7f4c42d9d700 1 --2- 172.21.15.195:0/3157918351 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4c4416ee40 0x7f4c4416f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:01.341 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.249+0000 7f4c4259c700 1 -- 172.21.15.195:0/3157918351 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4c44154c90 msgr2=0x7f4c441564a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:01.342 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.249+0000 7f4c4259c700 1 --2- 172.21.15.195:0/3157918351 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4c44154c90 0x7f4c441564a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.342 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.249+0000 7f4c4259c700 1 -- 172.21.15.195:0/3157918351 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4c4416ee40 msgr2=0x7f4c4416f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:01.342 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.249+0000 7f4c4259c700 1 --2- 172.21.15.195:0/3157918351 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4c4416ee40 0x7f4c4416f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.343 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.249+0000 7f4c4259c700 1 -- 172.21.15.195:0/3157918351 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c3c009ae0 con 0x7f4c44154310 2022-06-27T11:48:01.343 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.249+0000 7f4c4259c700 1 --2- 172.21.15.195:0/3157918351 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4c44154310 0x7f4c4414f000 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f4c3800b4f0 tx=0x7f4c38009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:01.344 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.249+0000 7f4c327fc700 1 -- 172.21.15.195:0/3157918351 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c3800be60 con 0x7f4c44154310 2022-06-27T11:48:01.344 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.249+0000 7f4c327fc700 1 -- 172.21.15.195:0/3157918351 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c38011630 con 0x7f4c44154310 2022-06-27T11:48:01.344 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.249+0000 7f4c48870700 1 -- 172.21.15.195:0/3157918351 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c4416e9b0 con 0x7f4c44154310 2022-06-27T11:48:01.345 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.250+0000 7f93bdffb700 1 -- 172.21.15.195:0/2165924106 <== mon.2 v2:172.21.15.202:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f93c8052970 con 0x7f93d00c41b0 2022-06-27T11:48:01.347 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.249+0000 7f4c48870700 1 -- 172.21.15.195:0/3157918351 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c4416b600 con 0x7f4c44154310 2022-06-27T11:48:01.347 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.251+0000 7f4c327fc700 1 -- 172.21.15.195:0/3157918351 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c380119d0 con 0x7f4c44154310 2022-06-27T11:48:01.348 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.251+0000 7f4c327fc700 1 -- 172.21.15.195:0/3157918351 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f4c38021070 con 0x7f4c44154310 2022-06-27T11:48:01.348 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.252+0000 7f4c327fc700 1 --2- 172.21.15.195:0/3157918351 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4c20063d00 0x7f4c200661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:01.349 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.252+0000 7f4c327fc700 1 -- 172.21.15.195:0/3157918351 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f4c38080580 con 0x7f4c44154310 2022-06-27T11:48:01.349 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.252+0000 7f4c41d9b700 1 --2- 172.21.15.195:0/3157918351 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4c20063d00 0x7f4c200661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:01.350 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.250+0000 7f1506ffd700 1 -- 172.21.15.195:0/3756924483 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14fc004030 con 0x7f15080c2610 2022-06-27T11:48:01.351 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.250+0000 7f1506ffd700 1 -- 172.21.15.195:0/3756924483 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14fc004190 con 0x7f15080c2610 2022-06-27T11:48:01.351 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.250+0000 7f1506ffd700 1 -- 172.21.15.195:0/3756924483 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14fc0105b0 con 0x7f15080c2610 2022-06-27T11:48:01.352 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.251+0000 7f150e72d700 1 -- 172.21.15.195:0/3756924483 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f15080c2610 msgr2=0x7f1508166b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:01.352 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.251+0000 7f150e72d700 1 --2- 172.21.15.195:0/3756924483 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f15080c2610 0x7f1508166b70 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f14fc0097a0 tx=0x7f14fc009060 comp rx=0 tx=0).stop 2022-06-27T11:48:01.352 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.252+0000 7f150e72d700 1 -- 172.21.15.195:0/3756924483 shutdown_connections 2022-06-27T11:48:01.353 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.252+0000 7f150e72d700 1 --2- 172.21.15.195:0/3756924483 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f15080c2610 0x7f1508166b70 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.353 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.252+0000 7f150e72d700 1 --2- 172.21.15.195:0/3756924483 >> v2:172.21.15.202:3300/0 conn(0x7f15080c0d60 0x7f15080b5f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.354 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.252+0000 7f150e72d700 1 -- 172.21.15.195:0/3756924483 >> 172.21.15.195:0/3756924483 conn(0x7f15080c8740 msgr2=0x7f15080c8b10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:01.354 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.252+0000 7f150e72d700 1 -- 172.21.15.195:0/3756924483 shutdown_connections 2022-06-27T11:48:01.355 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.252+0000 7f150e72d700 1 -- 172.21.15.195:0/3756924483 wait complete. 2022-06-27T11:48:01.356 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.253+0000 7f702d7fa700 1 -- 172.21.15.195:0/3890985076 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f703c00dea0 con 0x7f70400c4580 2022-06-27T11:48:01.356 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.253+0000 7f702d7fa700 1 -- 172.21.15.195:0/3890985076 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f703c003730 con 0x7f70400c4580 2022-06-27T11:48:01.356 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.253+0000 7f704747c700 1 -- 172.21.15.195:0/3890985076 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70401502b0 con 0x7f70400c4580 2022-06-27T11:48:01.357 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.253+0000 7f704747c700 1 -- 172.21.15.195:0/3890985076 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7040150010 con 0x7f70400c4580 2022-06-27T11:48:01.357 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.253+0000 7f702d7fa700 1 -- 172.21.15.195:0/3890985076 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f703c003ad0 con 0x7f70400c4580 2022-06-27T11:48:01.358 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.253+0000 7f150e72d700 1 Processor -- start 2022-06-27T11:48:01.359 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.254+0000 7f4c41d9b700 1 --2- 172.21.15.195:0/3157918351 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4c20063d00 0x7f4c200661b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4c3c009880 tx=0x7f4c3c017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:01.359 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.255+0000 7f4c48870700 1 -- 172.21.15.195:0/3157918351 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c24005170 con 0x7f4c44154310 2022-06-27T11:48:01.361 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.254+0000 7f150e72d700 1 -- start start 2022-06-27T11:48:01.362 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.255+0000 7f150e72d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f15080c0d60 0x7f1508172740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:01.362 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.255+0000 7f150e72d700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f15080c2610 0x7f1508177e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:01.362 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.255+0000 7f150e72d700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f15080c4ae0 0x7f1508187b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:01.363 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.255+0000 7f150e72d700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1508187640 con 0x7f15080c0d60 2022-06-27T11:48:01.363 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.255+0000 7f150e72d700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f150806d440 con 0x7f15080c2610 2022-06-27T11:48:01.364 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.255+0000 7f150e72d700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1508070510 con 0x7f15080c4ae0 2022-06-27T11:48:01.364 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.255+0000 7f150ccca700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f15080c4ae0 0x7f1508187b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:01.364 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.255+0000 7f1507fff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f15080c0d60 0x7f1508172740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:01.365 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.256+0000 7f1507fff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f15080c0d60 0x7f1508172740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41648/0 (socket says 172.21.15.195:41648) 2022-06-27T11:48:01.365 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.256+0000 7f1507fff700 1 -- 172.21.15.195:0/322952298 learned_addr learned my addr 172.21.15.195:0/322952298 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:48:01.366 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.256+0000 7f1507fff700 1 -- 172.21.15.195:0/322952298 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f15080c4ae0 msgr2=0x7f1508187b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:01.366 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.256+0000 7f1507fff700 1 --2- 172.21.15.195:0/322952298 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f15080c4ae0 0x7f1508187b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.367 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.256+0000 7f1507fff700 1 -- 172.21.15.195:0/322952298 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f15080c2610 msgr2=0x7f1508177e80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2022-06-27T11:48:01.367 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.256+0000 7f1507fff700 1 --2- 172.21.15.195:0/322952298 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f15080c2610 0x7f1508177e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.367 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.256+0000 7f1507fff700 1 -- 172.21.15.195:0/322952298 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14fc0092e0 con 0x7f15080c0d60 2022-06-27T11:48:01.369 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.255+0000 7f702d7fa700 1 -- 172.21.15.195:0/3890985076 <== mon.2 v2:172.21.15.202:3301/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f703c02f070 con 0x7f70400c4580 2022-06-27T11:48:01.369 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.256+0000 7f702d7fa700 1 --2- 172.21.15.195:0/3890985076 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7020063e50 0x7f7020066300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:01.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.256+0000 7f702d7fa700 1 -- 172.21.15.195:0/3890985076 <== mon.2 v2:172.21.15.202:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f703c081f20 con 0x7f70400c4580 2022-06-27T11:48:01.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.256+0000 7f7045218700 1 --2- 172.21.15.195:0/3890985076 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7020063e50 0x7f7020066300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:01.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.256+0000 7f7045218700 1 --2- 172.21.15.195:0/3890985076 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7020063e50 0x7f7020066300 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7030007650 tx=0x7f7030000bc0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:01.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.259+0000 7f704747c700 1 -- 172.21.15.195:0/3890985076 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f701c005170 con 0x7f70400c4580 2022-06-27T11:48:01.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.263+0000 7f1507fff700 1 --2- 172.21.15.195:0/322952298 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f15080c0d60 0x7f1508172740 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f150000c960 tx=0x7f1500001690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:01.372 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.264+0000 7f702d7fa700 1 -- 172.21.15.195:0/3890985076 <== mon.2 v2:172.21.15.202:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f703c02d080 con 0x7f70400c4580 2022-06-27T11:48:01.373 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.265+0000 7f14effff700 1 -- 172.21.15.195:0/322952298 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1500004c80 con 0x7f15080c0d60 2022-06-27T11:48:01.374 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.265+0000 7f14effff700 1 -- 172.21.15.195:0/322952298 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1500004de0 con 0x7f15080c0d60 2022-06-27T11:48:01.374 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.265+0000 7f14effff700 1 -- 172.21.15.195:0/322952298 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15000073a0 con 0x7f15080c0d60 2022-06-27T11:48:01.375 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.266+0000 7f4c327fc700 1 -- 172.21.15.195:0/3157918351 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f4c38015080 con 0x7f4c44154310 2022-06-27T11:48:01.376 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.266+0000 7f150e72d700 1 -- 172.21.15.195:0/322952298 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f150806d700 con 0x7f15080c0d60 2022-06-27T11:48:01.376 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.266+0000 7f150e72d700 1 -- 172.21.15.195:0/322952298 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f150806dcc0 con 0x7f15080c0d60 2022-06-27T11:48:01.376 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.267+0000 7f150e72d700 1 -- 172.21.15.195:0/322952298 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15080722c0 con 0x7f15080c0d60 2022-06-27T11:48:01.377 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.268+0000 7f14effff700 1 -- 172.21.15.195:0/322952298 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f1500010070 con 0x7f15080c0d60 2022-06-27T11:48:01.377 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.269+0000 7f14effff700 1 --2- 172.21.15.195:0/322952298 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f14e8063d00 0x7f14e80661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:01.377 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.269+0000 7f14effff700 1 -- 172.21.15.195:0/322952298 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f150007fa60 con 0x7f15080c0d60 2022-06-27T11:48:01.378 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.269+0000 7f15077fe700 1 --2- 172.21.15.195:0/322952298 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f14e8063d00 0x7f14e80661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:01.378 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.270+0000 7f15077fe700 1 --2- 172.21.15.195:0/322952298 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f14e8063d00 0x7f14e80661b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f15080893d0 tx=0x7f14f8009060 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:01.378 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.278+0000 7f14effff700 1 -- 172.21.15.195:0/322952298 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f150004e6d0 con 0x7f15080c0d60 2022-06-27T11:48:01.381 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.284+0000 7fcdd683c700 1 Processor -- start 2022-06-27T11:48:01.382 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.284+0000 7fcdd683c700 1 -- start start 2022-06-27T11:48:01.382 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.286+0000 7fcdd683c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fcdd015b640 0x7fcdd0067ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:01.382 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.286+0000 7fcdd683c700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdd015b1b0 con 0x7fcdd00c31f0 2022-06-27T11:48:01.383 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.286+0000 7fcdd683c700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdd015aba0 con 0x7fcdd00c35c0 2022-06-27T11:48:01.383 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.286+0000 7fcdd683c700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdd0069080 con 0x7fcdd015b640 2022-06-27T11:48:01.383 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.287+0000 7fcdcf7fe700 1 --1- >> v1:172.21.15.202:6789/0 conn(0x7fcdd00c35c0 0x7fcdd00b7f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.202:6789/0 says I am v1:172.21.15.195:49706/0 (socket says 172.21.15.195:49706) 2022-06-27T11:48:01.384 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.287+0000 7fcdcf7fe700 1 -- 172.21.15.195:0/1142268566 learned_addr learned my addr 172.21.15.195:0/1142268566 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:48:01.384 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.287+0000 7fcdd4dd9700 1 --2- 172.21.15.195:0/1142268566 >> v2:172.21.15.195:3300/0 conn(0x7fcdd015b640 0x7fcdd0067ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:01.384 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.287+0000 7fcdd4dd9700 1 -- 172.21.15.195:0/1142268566 >> v1:172.21.15.202:6789/0 conn(0x7fcdd00c35c0 legacy=0x7fcdd00b7f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:01.385 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.287+0000 7fcdd4dd9700 1 -- 172.21.15.195:0/1142268566 >> v1:172.21.15.195:6789/0 conn(0x7fcdd00c31f0 legacy=0x7fcdd00c49b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:01.385 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.287+0000 7fcdd4dd9700 1 -- 172.21.15.195:0/1142268566 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdd006b0e0 con 0x7fcdd015b640 2022-06-27T11:48:01.385 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.288+0000 7fcdd4dd9700 1 --2- 172.21.15.195:0/1142268566 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcdd015b640 0x7fcdd0067ed0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fcdc40097a0 tx=0x7fcdc4009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21d8a2f72404266a server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:01.386 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.288+0000 7fcdceffd700 1 -- 172.21.15.195:0/1142268566 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc4004030 con 0x7fcdd015b640 2022-06-27T11:48:01.386 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.288+0000 7fcdceffd700 1 -- 172.21.15.195:0/1142268566 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdc4004190 con 0x7fcdd015b640 2022-06-27T11:48:01.387 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.288+0000 7fcdceffd700 1 -- 172.21.15.195:0/1142268566 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc4011630 con 0x7fcdd015b640 2022-06-27T11:48:01.387 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.289+0000 7fcdd683c700 1 -- 172.21.15.195:0/1142268566 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcdd015b640 msgr2=0x7fcdd0067ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:01.387 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.289+0000 7fcdd683c700 1 --2- 172.21.15.195:0/1142268566 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcdd015b640 0x7fcdd0067ed0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fcdc40097a0 tx=0x7fcdc4009060 comp rx=0 tx=0).stop 2022-06-27T11:48:01.388 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.292+0000 7fcdd683c700 1 -- 172.21.15.195:0/1142268566 shutdown_connections 2022-06-27T11:48:01.388 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.292+0000 7fcdd683c700 1 --2- 172.21.15.195:0/1142268566 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcdd015b640 0x7fcdd0067ed0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.388 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.292+0000 7fcdd683c700 1 -- 172.21.15.195:0/1142268566 >> 172.21.15.195:0/1142268566 conn(0x7fcdd00c8740 msgr2=0x7fcdd00c8b10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:01.389 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.292+0000 7fcdd683c700 1 -- 172.21.15.195:0/1142268566 shutdown_connections 2022-06-27T11:48:01.389 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.292+0000 7fcdd683c700 1 -- 172.21.15.195:0/1142268566 wait complete. 2022-06-27T11:48:01.389 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.294+0000 7fcdd683c700 1 Processor -- start 2022-06-27T11:48:01.390 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.295+0000 7fcdd683c700 1 -- start start 2022-06-27T11:48:01.390 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.296+0000 7fcdd683c700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcdd00c31f0 0x7fcdd01475f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:01.390 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.296+0000 7fcdd683c700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcdd00c35c0 0x7fcdd0159ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:01.391 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.296+0000 7fcdd683c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcdd015b640 0x7fcdd015a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:01.391 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.296+0000 7fcdd683c700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd006e6a0 con 0x7fcdd015b640 2022-06-27T11:48:01.391 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.296+0000 7fcdd683c700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd0149410 con 0x7fcdd00c31f0 2022-06-27T11:48:01.392 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.296+0000 7fcdd683c700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd01496b0 con 0x7fcdd00c35c0 2022-06-27T11:48:01.392 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.297+0000 7fcdcf7fe700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcdd00c35c0 0x7fcdd0159ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:01.392 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.297+0000 7fcdcffff700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcdd00c31f0 0x7fcdd01475f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:01.393 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.297+0000 7fcdcffff700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcdd00c31f0 0x7fcdd01475f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3300/0 says I am v2:172.21.15.195:39788/0 (socket says 172.21.15.195:39788) 2022-06-27T11:48:01.393 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.297+0000 7fcdcffff700 1 -- 172.21.15.195:0/2721321495 learned_addr learned my addr 172.21.15.195:0/2721321495 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:48:01.393 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.299+0000 7fcdcffff700 1 -- 172.21.15.195:0/2721321495 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcdd00c35c0 msgr2=0x7fcdd0159ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:01.394 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.299+0000 7fcdcffff700 1 --2- 172.21.15.195:0/2721321495 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcdd00c35c0 0x7fcdd0159ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.394 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.299+0000 7fcdcffff700 1 -- 172.21.15.195:0/2721321495 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcdd015b640 msgr2=0x7fcdd015a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:01.394 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.299+0000 7fcdcffff700 1 --2- 172.21.15.195:0/2721321495 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcdd015b640 0x7fcdd015a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.395 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.299+0000 7fcdcffff700 1 -- 172.21.15.195:0/2721321495 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdc40092e0 con 0x7fcdd00c31f0 2022-06-27T11:48:01.395 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.299+0000 7fcdcffff700 1 --2- 172.21.15.195:0/2721321495 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcdd00c31f0 0x7fcdd01475f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fcdc000c460 tx=0x7fcdc0007e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:01.395 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.300+0000 7fcdb7fff700 1 -- 172.21.15.195:0/2721321495 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc00053b0 con 0x7fcdd00c31f0 2022-06-27T11:48:01.396 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.300+0000 7fcdb7fff700 1 -- 172.21.15.195:0/2721321495 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdc0003500 con 0x7fcdd00c31f0 2022-06-27T11:48:01.396 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.300+0000 7fcdb7fff700 1 -- 172.21.15.195:0/2721321495 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc00038a0 con 0x7fcdd00c31f0 2022-06-27T11:48:01.396 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.300+0000 7fcdd683c700 1 -- 172.21.15.195:0/2721321495 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdd006dd70 con 0x7fcdd00c31f0 2022-06-27T11:48:01.397 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.300+0000 7fcdd683c700 1 -- 172.21.15.195:0/2721321495 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdd006e330 con 0x7fcdd00c31f0 2022-06-27T11:48:01.397 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.303+0000 7fcdb7fff700 1 -- 172.21.15.195:0/2721321495 <== mon.1 v2:172.21.15.202:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fcdc0019030 con 0x7fcdd00c31f0 2022-06-27T11:48:01.397 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.304+0000 7fcdb7fff700 1 --2- 172.21.15.195:0/2721321495 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcdb0063e20 0x7fcdb00662d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:01.397 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.304+0000 7fcdb7fff700 1 -- 172.21.15.195:0/2721321495 <== mon.1 v2:172.21.15.202:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fcdc007fbf0 con 0x7fcdd00c31f0 2022-06-27T11:48:01.397 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.304+0000 7fcdcf7fe700 1 --2- 172.21.15.195:0/2721321495 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcdb0063e20 0x7fcdb00662d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:01.398 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.305+0000 7fcdcf7fe700 1 --2- 172.21.15.195:0/2721321495 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcdb0063e20 0x7fcdb00662d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fcdd0072940 tx=0x7fcdc8008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:01.398 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.305+0000 7fcdd683c700 1 -- 172.21.15.195:0/2721321495 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcda8005170 con 0x7fcdd00c31f0 2022-06-27T11:48:01.398 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.311+0000 7fcdb7fff700 1 -- 172.21.15.195:0/2721321495 <== mon.1 v2:172.21.15.202:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fcdc004e770 con 0x7fcdd00c31f0 2022-06-27T11:48:01.558 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.557+0000 7f759cf4e700 1 -- 172.21.15.195:0/146907795 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f75980722c0 con 0x7f75980c41b0 2022-06-27T11:48:01.559 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.558+0000 7f75867fc700 1 -- 172.21.15.195:0/146907795 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7590014020 con 0x7f75980c41b0 2022-06-27T11:48:01.560 INFO:teuthology.orchestra.run.smithi195.stdout:55834574854 2022-06-27T11:48:01.564 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.561+0000 7f759cf4e700 1 -- 172.21.15.195:0/146907795 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7574063e50 msgr2=0x7f7574066300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:01.564 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.561+0000 7f759cf4e700 1 --2- 172.21.15.195:0/146907795 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7574063e50 0x7f7574066300 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f75880081d0 tx=0x7f758801b000 comp rx=0 tx=0).stop 2022-06-27T11:48:01.565 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.561+0000 7f759cf4e700 1 -- 172.21.15.195:0/146907795 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f75980c41b0 msgr2=0x7f75980bfeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:01.565 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.561+0000 7f759cf4e700 1 --2- 172.21.15.195:0/146907795 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f75980c41b0 0x7f75980bfeb0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f759000b7d0 tx=0x7f7590009840 comp rx=0 tx=0).stop 2022-06-27T11:48:01.566 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.562+0000 7f759cf4e700 1 -- 172.21.15.195:0/146907795 shutdown_connections 2022-06-27T11:48:01.566 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.562+0000 7f759cf4e700 1 --2- 172.21.15.195:0/146907795 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7574063e50 0x7f7574066300 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.566 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.562+0000 7f759cf4e700 1 --2- 172.21.15.195:0/146907795 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f75980c4580 0x7f759816c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.567 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.562+0000 7f759cf4e700 1 --2- 172.21.15.195:0/146907795 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f75980c3270 0x7f7598150450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.567 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.562+0000 7f759cf4e700 1 --2- 172.21.15.195:0/146907795 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f75980c41b0 0x7f75980bfeb0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.568 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.562+0000 7f759cf4e700 1 -- 172.21.15.195:0/146907795 >> 172.21.15.195:0/146907795 conn(0x7f75980c8740 msgr2=0x7f7598154540 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:01.569 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.562+0000 7f759cf4e700 1 -- 172.21.15.195:0/146907795 shutdown_connections 2022-06-27T11:48:01.569 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.562+0000 7f759cf4e700 1 -- 172.21.15.195:0/146907795 wait complete. 2022-06-27T11:48:01.580 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574854 for osd.0 2022-06-27T11:48:01.581 DEBUG:teuthology.parallel:result is None 2022-06-27T11:48:01.591 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.586+0000 7f80c2db8700 1 -- 172.21.15.195:0/516673139 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f80bc1560d0 con 0x7f80bc0c4580 2022-06-27T11:48:01.591 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.587+0000 7f80b8ff9700 1 -- 172.21.15.195:0/516673139 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f80b0027020 con 0x7f80bc0c4580 2022-06-27T11:48:01.592 INFO:teuthology.orchestra.run.smithi195.stdout:55834574854 2022-06-27T11:48:01.594 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.594+0000 7f80c2db8700 1 -- 172.21.15.195:0/516673139 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f80a8063ef0 msgr2=0x7f80a80663a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:01.594 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.594+0000 7f80c2db8700 1 --2- 172.21.15.195:0/516673139 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f80a8063ef0 0x7f80a80663a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f80b4007fd0 tx=0x7f80b4009040 comp rx=0 tx=0).stop 2022-06-27T11:48:01.594 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.594+0000 7f80c2db8700 1 -- 172.21.15.195:0/516673139 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f80bc0c4580 msgr2=0x7f80bc14d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:01.594 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.594+0000 7f80c2db8700 1 --2- 172.21.15.195:0/516673139 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f80bc0c4580 0x7f80bc14d660 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f80b000b4f0 tx=0x7f80b00090f0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.595 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.594+0000 7f80c2db8700 1 -- 172.21.15.195:0/516673139 shutdown_connections 2022-06-27T11:48:01.595 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.594+0000 7f80c2db8700 1 --2- 172.21.15.195:0/516673139 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f80a8063ef0 0x7f80a80663a0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.595 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.594+0000 7f80c2db8700 1 --2- 172.21.15.195:0/516673139 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f80bc0c4580 0x7f80bc14d660 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.596 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.594+0000 7f80c2db8700 1 --2- 172.21.15.195:0/516673139 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f80bc0c37b0 0x7f80bc0c6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.596 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.594+0000 7f80c2db8700 1 --2- 172.21.15.195:0/516673139 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f80bc0c41b0 0x7f80bc1631d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.596 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.594+0000 7f80c2db8700 1 -- 172.21.15.195:0/516673139 >> 172.21.15.195:0/516673139 conn(0x7f80bc0c8740 msgr2=0x7f80bc154b70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:01.596 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.594+0000 7f80c2db8700 1 -- 172.21.15.195:0/516673139 shutdown_connections 2022-06-27T11:48:01.597 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.594+0000 7f80c2db8700 1 -- 172.21.15.195:0/516673139 wait complete. 2022-06-27T11:48:01.611 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.611+0000 7f08e15ee700 1 -- 172.21.15.195:0/2140847956 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f08dc063dd0 con 0x7f08dc0c6590 2022-06-27T11:48:01.611 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.611+0000 7f08c2ffd700 1 -- 172.21.15.195:0/2140847956 <== mon.1 v2:172.21.15.202:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f08d0017020 con 0x7f08dc0c6590 2022-06-27T11:48:01.613 INFO:teuthology.orchestra.run.smithi195.stdout:55834574855 2022-06-27T11:48:01.614 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574854 for osd.4 2022-06-27T11:48:01.615 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.614+0000 7f08e15ee700 1 -- 172.21.15.195:0/2140847956 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f08bc063d30 msgr2=0x7f08bc0661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:01.615 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.614+0000 7f08e15ee700 1 --2- 172.21.15.195:0/2140847956 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f08bc063d30 0x7f08bc0661e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f08d40011d0 tx=0x7f08d4017000 comp rx=0 tx=0).stop 2022-06-27T11:48:01.615 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.614+0000 7f08e15ee700 1 -- 172.21.15.195:0/2140847956 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f08dc0c6590 msgr2=0x7f08dc0c1c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:01.615 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.614+0000 7f08e15ee700 1 --2- 172.21.15.195:0/2140847956 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f08dc0c6590 0x7f08dc0c1c00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f08d000c490 tx=0x7f08d0007d40 comp rx=0 tx=0).stop 2022-06-27T11:48:01.616 DEBUG:teuthology.parallel:result is None 2022-06-27T11:48:01.617 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.616+0000 7f08e15ee700 1 -- 172.21.15.195:0/2140847956 shutdown_connections 2022-06-27T11:48:01.617 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.616+0000 7f08e15ee700 1 --2- 172.21.15.195:0/2140847956 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f08bc063d30 0x7f08bc0661e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.618 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.616+0000 7f08e15ee700 1 --2- 172.21.15.195:0/2140847956 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f08dc069cf0 0x7f08dc15f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.618 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.616+0000 7f08e15ee700 1 --2- 172.21.15.195:0/2140847956 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f08dc0c6590 0x7f08dc0c1c00 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.619 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.616+0000 7f08e15ee700 1 --2- 172.21.15.195:0/2140847956 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f08dc06a0c0 0x7f08dc0c4a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.619 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.616+0000 7f08e15ee700 1 -- 172.21.15.195:0/2140847956 >> 172.21.15.195:0/2140847956 conn(0x7f08dc0c9b00 msgr2=0x7f08dc150190 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:01.620 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.617+0000 7f08e15ee700 1 -- 172.21.15.195:0/2140847956 shutdown_connections 2022-06-27T11:48:01.620 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.617+0000 7f08e15ee700 1 -- 172.21.15.195:0/2140847956 wait complete. 2022-06-27T11:48:01.623 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.623+0000 7f704747c700 1 -- 172.21.15.195:0/3890985076 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f701c005840 con 0x7f70400c4580 2022-06-27T11:48:01.623 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.624+0000 7f702d7fa700 1 -- 172.21.15.195:0/3890985076 <== mon.2 v2:172.21.15.202:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f703c050aa0 con 0x7f70400c4580 2022-06-27T11:48:01.624 INFO:teuthology.orchestra.run.smithi195.stdout:55834574854 2022-06-27T11:48:01.627 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.628+0000 7f704747c700 1 -- 172.21.15.195:0/3890985076 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7020063e50 msgr2=0x7f7020066300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:01.627 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.628+0000 7f704747c700 1 --2- 172.21.15.195:0/3890985076 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7020063e50 0x7f7020066300 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7030007650 tx=0x7f7030000bc0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.627 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.628+0000 7f704747c700 1 -- 172.21.15.195:0/3890985076 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f70400c4580 msgr2=0x7f704016a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:01.628 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.628+0000 7f704747c700 1 --2- 172.21.15.195:0/3890985076 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f70400c4580 0x7f704016a190 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f703c008120 tx=0x7f703c01b000 comp rx=0 tx=0).stop 2022-06-27T11:48:01.629 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.629+0000 7f704747c700 1 -- 172.21.15.195:0/3890985076 shutdown_connections 2022-06-27T11:48:01.629 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.629+0000 7f704747c700 1 --2- 172.21.15.195:0/3890985076 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7020063e50 0x7f7020066300 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.629 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.629+0000 7f704747c700 1 --2- 172.21.15.195:0/3890985076 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f70400c4580 0x7f704016a190 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.629 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.629+0000 7f704747c700 1 --2- 172.21.15.195:0/3890985076 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f70400c37b0 0x7f704014f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.630 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.629+0000 7f704747c700 1 --2- 172.21.15.195:0/3890985076 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f70400c41b0 0x7f7040169c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.630 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.629+0000 7f704747c700 1 -- 172.21.15.195:0/3890985076 >> 172.21.15.195:0/3890985076 conn(0x7f70400c8740 msgr2=0x7f70401544b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:01.631 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.631+0000 7f704747c700 1 -- 172.21.15.195:0/3890985076 shutdown_connections 2022-06-27T11:48:01.631 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.631+0000 7f704747c700 1 -- 172.21.15.195:0/3890985076 wait complete. 2022-06-27T11:48:01.635 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574855 for osd.2 2022-06-27T11:48:01.635 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.635+0000 7f150e72d700 1 -- 172.21.15.195:0/322952298 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f15080c3970 con 0x7f15080c0d60 2022-06-27T11:48:01.635 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.635+0000 7f14effff700 1 -- 172.21.15.195:0/322952298 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1500018070 con 0x7f15080c0d60 2022-06-27T11:48:01.636 DEBUG:teuthology.parallel:result is None 2022-06-27T11:48:01.636 INFO:teuthology.orchestra.run.smithi195.stdout:55834574854 2022-06-27T11:48:01.637 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.637+0000 7f150e72d700 1 -- 172.21.15.195:0/322952298 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f14e8063d00 msgr2=0x7f14e80661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:01.638 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.637+0000 7f150e72d700 1 --2- 172.21.15.195:0/322952298 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f14e8063d00 0x7f14e80661b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f15080893d0 tx=0x7f14f8009060 comp rx=0 tx=0).stop 2022-06-27T11:48:01.638 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.637+0000 7f150e72d700 1 -- 172.21.15.195:0/322952298 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f15080c0d60 msgr2=0x7f1508172740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:01.638 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.637+0000 7f150e72d700 1 --2- 172.21.15.195:0/322952298 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f15080c0d60 0x7f1508172740 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f150000c960 tx=0x7f1500001690 comp rx=0 tx=0).stop 2022-06-27T11:48:01.641 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.641+0000 7f150e72d700 1 -- 172.21.15.195:0/322952298 shutdown_connections 2022-06-27T11:48:01.641 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.641+0000 7f150e72d700 1 --2- 172.21.15.195:0/322952298 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f14e8063d00 0x7f14e80661b0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.641 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.641+0000 7f150e72d700 1 --2- 172.21.15.195:0/322952298 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f15080c0d60 0x7f1508172740 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.641 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.641+0000 7f150e72d700 1 --2- 172.21.15.195:0/322952298 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f15080c4ae0 0x7f1508187b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.642 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.641+0000 7f150e72d700 1 --2- 172.21.15.195:0/322952298 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f15080c2610 0x7f1508177e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.642 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.641+0000 7f150e72d700 1 -- 172.21.15.195:0/322952298 >> 172.21.15.195:0/322952298 conn(0x7f15080c8740 msgr2=0x7f15080c2be0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:01.642 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.641+0000 7f150e72d700 1 -- 172.21.15.195:0/322952298 shutdown_connections 2022-06-27T11:48:01.642 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.641+0000 7f150e72d700 1 -- 172.21.15.195:0/322952298 wait complete. 2022-06-27T11:48:01.646 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574854 for osd.5 2022-06-27T11:48:01.646 DEBUG:teuthology.parallel:result is None 2022-06-27T11:48:01.654 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574854 for osd.1 2022-06-27T11:48:01.654 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.654+0000 7f93d8070700 1 -- 172.21.15.195:0/2165924106 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f93d0148dd0 con 0x7f93d00c41b0 2022-06-27T11:48:01.655 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.655+0000 7f93bdffb700 1 -- 172.21.15.195:0/2165924106 <== mon.2 v2:172.21.15.202:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f93c8052560 con 0x7f93d00c41b0 2022-06-27T11:48:01.655 DEBUG:teuthology.parallel:result is None 2022-06-27T11:48:01.655 INFO:teuthology.orchestra.run.smithi195.stdout:55834574854 2022-06-27T11:48:01.657 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.658+0000 7f93d8070700 1 -- 172.21.15.195:0/2165924106 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f93b0063e80 msgr2=0x7f93b0066330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:01.658 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.658+0000 7f93d8070700 1 --2- 172.21.15.195:0/2165924106 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f93b0063e80 0x7f93b0066330 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f93cc00a910 tx=0x7f93cc01a000 comp rx=0 tx=0).stop 2022-06-27T11:48:01.658 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.658+0000 7f93d8070700 1 -- 172.21.15.195:0/2165924106 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f93d00c41b0 msgr2=0x7f93d016eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:01.658 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.658+0000 7f93d8070700 1 --2- 172.21.15.195:0/2165924106 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f93d00c41b0 0x7f93d016eec0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f93c800b790 tx=0x7f93c8009310 comp rx=0 tx=0).stop 2022-06-27T11:48:01.658 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.658+0000 7f93d8070700 1 -- 172.21.15.195:0/2165924106 shutdown_connections 2022-06-27T11:48:01.659 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.658+0000 7f93d8070700 1 --2- 172.21.15.195:0/2165924106 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f93b0063e80 0x7f93b0066330 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.659 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.658+0000 7f93d8070700 1 --2- 172.21.15.195:0/2165924106 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f93d00c3980 0x7f93d00b7680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.659 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.658+0000 7f93d8070700 1 --2- 172.21.15.195:0/2165924106 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f93d00c4580 0x7f93d01516f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.660 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.658+0000 7f93d8070700 1 --2- 172.21.15.195:0/2165924106 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f93d00c41b0 0x7f93d016eec0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.660 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.658+0000 7f93d8070700 1 -- 172.21.15.195:0/2165924106 >> 172.21.15.195:0/2165924106 conn(0x7f93d00c8740 msgr2=0x7f93d0150960 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:01.660 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.658+0000 7f93d8070700 1 -- 172.21.15.195:0/2165924106 shutdown_connections 2022-06-27T11:48:01.661 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.658+0000 7f93d8070700 1 -- 172.21.15.195:0/2165924106 wait complete. 2022-06-27T11:48:01.662 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.662+0000 7fcdd683c700 1 -- 172.21.15.195:0/2721321495 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fcda8005840 con 0x7fcdd00c31f0 2022-06-27T11:48:01.663 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.663+0000 7fcdb7fff700 1 -- 172.21.15.195:0/2721321495 <== mon.1 v2:172.21.15.202:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fcdc0017020 con 0x7fcdd00c31f0 2022-06-27T11:48:01.663 INFO:teuthology.orchestra.run.smithi195.stdout:55834574855 2022-06-27T11:48:01.666 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.665+0000 7fcdd683c700 1 -- 172.21.15.195:0/2721321495 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcdb0063e20 msgr2=0x7fcdb00662d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:01.666 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.665+0000 7fcdd683c700 1 --2- 172.21.15.195:0/2721321495 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcdb0063e20 0x7fcdb00662d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fcdd0072940 tx=0x7fcdc8008040 comp rx=0 tx=0).stop 2022-06-27T11:48:01.667 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.665+0000 7fcdd683c700 1 -- 172.21.15.195:0/2721321495 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcdd00c31f0 msgr2=0x7fcdd01475f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:01.667 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.665+0000 7fcdd683c700 1 --2- 172.21.15.195:0/2721321495 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcdd00c31f0 0x7fcdd01475f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fcdc000c460 tx=0x7fcdc0007e00 comp rx=0 tx=0).stop 2022-06-27T11:48:01.667 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.665+0000 7fcdd683c700 1 -- 172.21.15.195:0/2721321495 shutdown_connections 2022-06-27T11:48:01.668 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.665+0000 7fcdd683c700 1 --2- 172.21.15.195:0/2721321495 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcdb0063e20 0x7fcdb00662d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.668 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.665+0000 7fcdd683c700 1 --2- 172.21.15.195:0/2721321495 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcdd015b640 0x7fcdd015a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.669 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.665+0000 7fcdd683c700 1 --2- 172.21.15.195:0/2721321495 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcdd00c31f0 0x7fcdd01475f0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.669 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.665+0000 7fcdd683c700 1 --2- 172.21.15.195:0/2721321495 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcdd00c35c0 0x7fcdd0159ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.670 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.665+0000 7fcdd683c700 1 -- 172.21.15.195:0/2721321495 >> 172.21.15.195:0/2721321495 conn(0x7fcdd00c8740 msgr2=0x7fcdd00b8150 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:01.670 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.665+0000 7fcdd683c700 1 -- 172.21.15.195:0/2721321495 shutdown_connections 2022-06-27T11:48:01.671 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.665+0000 7fcdd683c700 1 -- 172.21.15.195:0/2721321495 wait complete. 2022-06-27T11:48:01.673 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.671+0000 7f4c48870700 1 -- 172.21.15.195:0/3157918351 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f4c24005840 con 0x7f4c44154310 2022-06-27T11:48:01.673 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.672+0000 7f4c327fc700 1 -- 172.21.15.195:0/3157918351 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4c3804f170 con 0x7f4c44154310 2022-06-27T11:48:01.674 INFO:teuthology.orchestra.run.smithi195.stdout:55834574854 2022-06-27T11:48:01.676 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.675+0000 7f4c48870700 1 -- 172.21.15.195:0/3157918351 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4c20063d00 msgr2=0x7f4c200661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:01.676 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.675+0000 7f4c48870700 1 --2- 172.21.15.195:0/3157918351 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4c20063d00 0x7f4c200661b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4c3c009880 tx=0x7f4c3c017000 comp rx=0 tx=0).stop 2022-06-27T11:48:01.677 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.675+0000 7f4c48870700 1 -- 172.21.15.195:0/3157918351 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4c44154310 msgr2=0x7f4c4414f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:01.678 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.675+0000 7f4c48870700 1 --2- 172.21.15.195:0/3157918351 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4c44154310 0x7f4c4414f000 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f4c3800b4f0 tx=0x7f4c38009060 comp rx=0 tx=0).stop 2022-06-27T11:48:01.678 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.675+0000 7f4c48870700 1 -- 172.21.15.195:0/3157918351 shutdown_connections 2022-06-27T11:48:01.679 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.675+0000 7f4c48870700 1 --2- 172.21.15.195:0/3157918351 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4c20063d00 0x7f4c200661b0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.679 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.675+0000 7f4c48870700 1 --2- 172.21.15.195:0/3157918351 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4c44154310 0x7f4c4414f000 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.680 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.675+0000 7f4c48870700 1 --2- 172.21.15.195:0/3157918351 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4c4416ee40 0x7f4c4416f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.680 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.675+0000 7f4c48870700 1 --2- 172.21.15.195:0/3157918351 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4c44154c90 0x7f4c441564a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.680 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.675+0000 7f4c48870700 1 -- 172.21.15.195:0/3157918351 >> 172.21.15.195:0/3157918351 conn(0x7f4c440c8740 msgr2=0x7f4c440b85e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:01.681 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.675+0000 7f4c48870700 1 -- 172.21.15.195:0/3157918351 shutdown_connections 2022-06-27T11:48:01.681 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.675+0000 7f4c48870700 1 -- 172.21.15.195:0/3157918351 wait complete. 2022-06-27T11:48:01.683 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574854 for osd.6 2022-06-27T11:48:01.684 DEBUG:teuthology.parallel:result is None 2022-06-27T11:48:01.685 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574855 for osd.3 2022-06-27T11:48:01.686 DEBUG:teuthology.parallel:result is None 2022-06-27T11:48:01.688 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574854 for osd.7 2022-06-27T11:48:01.688 DEBUG:teuthology.parallel:result is None 2022-06-27T11:48:01.689 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2022-06-27T11:48:01.690 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2022-06-27T11:48:01.856 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.846+0000 7fb29ce41700 1 Processor -- start 2022-06-27T11:48:01.856 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.846+0000 7fb29ce41700 1 -- start start 2022-06-27T11:48:01.857 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.847+0000 7fb29ce41700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fb2980b8520 0x7fb298153de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:01.857 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.847+0000 7fb29ce41700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb2980c3010 0x7fb2980c3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:01.857 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.847+0000 7fb29ce41700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb298068a10 con 0x7fb298068ea0 2022-06-27T11:48:01.858 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.847+0000 7fb29ce41700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb298067ed0 con 0x7fb2980b8520 2022-06-27T11:48:01.859 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.847+0000 7fb29ce41700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2980692d0 con 0x7fb2980c3010 2022-06-27T11:48:01.860 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.847+0000 7fb295d9b700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fb298068ea0 0x7fb2980c45f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:47858/0 (socket says 172.21.15.195:47858) 2022-06-27T11:48:01.860 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.847+0000 7fb295d9b700 1 -- 172.21.15.195:0/1843960821 learned_addr learned my addr 172.21.15.195:0/1843960821 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:48:01.860 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.848+0000 7fb296d9d700 1 --2- 172.21.15.195:0/1843960821 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb2980c3010 0x7fb2980c3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:01.861 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.848+0000 7fb29659c700 1 --2- 172.21.15.195:0/1843960821 >> v2:172.21.15.202:3300/0 conn(0x7fb2980b8520 0x7fb298153de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:01.861 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.848+0000 7fb29559a700 1 -- 172.21.15.195:0/1843960821 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1169970083 0 0) 0x7fb298068a10 con 0x7fb298068ea0 2022-06-27T11:48:01.861 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.848+0000 7fb29559a700 1 -- 172.21.15.195:0/1843960821 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb27c003430 con 0x7fb298068ea0 2022-06-27T11:48:01.862 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.848+0000 7fb29559a700 1 -- 172.21.15.195:0/1843960821 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1197133867 0 0) 0x7fb27c003430 con 0x7fb298068ea0 2022-06-27T11:48:01.862 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.848+0000 7fb296d9d700 1 -- 172.21.15.195:0/1843960821 >> v2:172.21.15.202:3300/0 conn(0x7fb2980b8520 msgr2=0x7fb298153de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:01.863 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.848+0000 7fb296d9d700 1 --2- 172.21.15.195:0/1843960821 >> v2:172.21.15.202:3300/0 conn(0x7fb2980b8520 0x7fb298153de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.863 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.848+0000 7fb296d9d700 1 -- 172.21.15.195:0/1843960821 >> v1:172.21.15.195:6789/0 conn(0x7fb298068ea0 legacy=0x7fb2980c45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:01.863 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.848+0000 7fb296d9d700 1 -- 172.21.15.195:0/1843960821 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2980c3dd0 con 0x7fb2980c3010 2022-06-27T11:48:01.863 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.848+0000 7fb296d9d700 1 --2- 172.21.15.195:0/1843960821 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb2980c3010 0x7fb2980c3490 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb290009e60 tx=0x7fb290009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8174d34e33e84ee1 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:01.864 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.848+0000 7fb29559a700 1 -- 172.21.15.195:0/1843960821 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb29000e070 con 0x7fb2980c3010 2022-06-27T11:48:01.864 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.848+0000 7fb29559a700 1 -- 172.21.15.195:0/1843960821 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2900038a0 con 0x7fb2980c3010 2022-06-27T11:48:01.864 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.848+0000 7fb29559a700 1 -- 172.21.15.195:0/1843960821 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb28c001470 con 0x7fb2980c3010 2022-06-27T11:48:01.864 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.849+0000 7fb29ce41700 1 -- 172.21.15.195:0/1843960821 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb2980c3010 msgr2=0x7fb2980c3490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:01.865 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.849+0000 7fb29ce41700 1 --2- 172.21.15.195:0/1843960821 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb2980c3010 0x7fb2980c3490 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb290009e60 tx=0x7fb290009780 comp rx=0 tx=0).stop 2022-06-27T11:48:01.865 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.849+0000 7fb29ce41700 1 -- 172.21.15.195:0/1843960821 shutdown_connections 2022-06-27T11:48:01.865 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.849+0000 7fb29ce41700 1 --2- 172.21.15.195:0/1843960821 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb2980c3010 0x7fb2980c3490 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.865 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.849+0000 7fb29ce41700 1 --2- 172.21.15.195:0/1843960821 >> v2:172.21.15.202:3300/0 conn(0x7fb2980b8520 0x7fb298153de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.865 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.849+0000 7fb29ce41700 1 -- 172.21.15.195:0/1843960821 >> 172.21.15.195:0/1843960821 conn(0x7fb2980c8740 msgr2=0x7fb2980c8b10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:01.866 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.849+0000 7fb29ce41700 1 -- 172.21.15.195:0/1843960821 shutdown_connections 2022-06-27T11:48:01.866 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.849+0000 7fb29ce41700 1 -- 172.21.15.195:0/1843960821 wait complete. 2022-06-27T11:48:01.866 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.850+0000 7fb29ce41700 1 Processor -- start 2022-06-27T11:48:01.866 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.850+0000 7fb29ce41700 1 -- start start 2022-06-27T11:48:01.867 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.851+0000 7fb29ce41700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb298068ea0 0x7fb2980ba710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:01.867 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.851+0000 7fb29ce41700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb2980b8520 0x7fb2980bac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:01.867 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.851+0000 7fb29ce41700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb2980c3010 0x7fb2980c7960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:01.867 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.851+0000 7fb29ce41700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2980b9fa0 con 0x7fb298068ea0 2022-06-27T11:48:01.867 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.851+0000 7fb29ce41700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2980bbc20 con 0x7fb2980c3010 2022-06-27T11:48:01.868 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.851+0000 7fb29ce41700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2981645e0 con 0x7fb2980b8520 2022-06-27T11:48:01.868 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.851+0000 7fb29659c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb298068ea0 0x7fb2980ba710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:01.868 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.851+0000 7fb29659c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb298068ea0 0x7fb2980ba710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41676/0 (socket says 172.21.15.195:41676) 2022-06-27T11:48:01.868 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.851+0000 7fb29659c700 1 -- 172.21.15.195:0/371027415 learned_addr learned my addr 172.21.15.195:0/371027415 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:48:01.869 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.851+0000 7fb296d9d700 1 --2- 172.21.15.195:0/371027415 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb2980c3010 0x7fb2980c7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:01.869 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.851+0000 7fb295d9b700 1 --2- 172.21.15.195:0/371027415 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb2980b8520 0x7fb2980bac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:01.869 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.851+0000 7fb29659c700 1 -- 172.21.15.195:0/371027415 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb2980b8520 msgr2=0x7fb2980bac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:01.869 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.851+0000 7fb29659c700 1 --2- 172.21.15.195:0/371027415 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb2980b8520 0x7fb2980bac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.870 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.851+0000 7fb29659c700 1 -- 172.21.15.195:0/371027415 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb2980c3010 msgr2=0x7fb2980c7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:01.870 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.851+0000 7fb29659c700 1 --2- 172.21.15.195:0/371027415 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb2980c3010 0x7fb2980c7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:01.870 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.851+0000 7fb29659c700 1 -- 172.21.15.195:0/371027415 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb290009a20 con 0x7fb298068ea0 2022-06-27T11:48:01.870 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.851+0000 7fb295d9b700 1 --2- 172.21.15.195:0/371027415 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb2980b8520 0x7fb2980bac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T11:48:01.870 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.851+0000 7fb29659c700 1 --2- 172.21.15.195:0/371027415 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb298068ea0 0x7fb2980ba710 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb28800e970 tx=0x7fb288009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:01.871 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.851+0000 7fb2867fc700 1 -- 172.21.15.195:0/371027415 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb28800ee60 con 0x7fb298068ea0 2022-06-27T11:48:01.871 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.851+0000 7fb2867fc700 1 -- 172.21.15.195:0/371027415 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb28800c8e0 con 0x7fb298068ea0 2022-06-27T11:48:01.871 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.851+0000 7fb29ce41700 1 -- 172.21.15.195:0/371027415 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb298165120 con 0x7fb298068ea0 2022-06-27T11:48:01.871 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.851+0000 7fb2867fc700 1 -- 172.21.15.195:0/371027415 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb28800cc60 con 0x7fb298068ea0 2022-06-27T11:48:01.872 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.852+0000 7fb29ce41700 1 -- 172.21.15.195:0/371027415 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2981647f0 con 0x7fb298068ea0 2022-06-27T11:48:01.872 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.852+0000 7fb29ce41700 1 -- 172.21.15.195:0/371027415 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2980722c0 con 0x7fb298068ea0 2022-06-27T11:48:01.872 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.853+0000 7fb2867fc700 1 -- 172.21.15.195:0/371027415 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fb288019030 con 0x7fb298068ea0 2022-06-27T11:48:01.872 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.853+0000 7fb2867fc700 1 --2- 172.21.15.195:0/371027415 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb274063d00 0x7fb2740661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:01.873 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.853+0000 7fb2867fc700 1 -- 172.21.15.195:0/371027415 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fb288080930 con 0x7fb298068ea0 2022-06-27T11:48:01.873 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.853+0000 7fb295d9b700 1 --2- 172.21.15.195:0/371027415 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb274063d00 0x7fb2740661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:01.873 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.853+0000 7fb295d9b700 1 --2- 172.21.15.195:0/371027415 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb274063d00 0x7fb2740661b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb2980b7df0 tx=0x7fb28c0015c0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:01.874 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:01.857+0000 7fb2867fc700 1 -- 172.21.15.195:0/371027415 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fb28804f520 con 0x7fb298068ea0 2022-06-27T11:48:02.036 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.036+0000 7fb29ce41700 1 -- 172.21.15.195:0/371027415 --> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb298152980 con 0x7fb274063d00 2022-06-27T11:48:02.043 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.044+0000 7fb2867fc700 1 -- 172.21.15.195:0/371027415 <== mgr.4098 v2:172.21.15.195:6832/93955 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+284879 (secure 0 0 0) 0x7fb298152980 con 0x7fb274063d00 2022-06-27T11:48:02.047 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:48:02.050 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.047+0000 7fb29ce41700 1 -- 172.21.15.195:0/371027415 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb274063d00 msgr2=0x7fb2740661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:02.051 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.047+0000 7fb29ce41700 1 --2- 172.21.15.195:0/371027415 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb274063d00 0x7fb2740661b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb2980b7df0 tx=0x7fb28c0015c0 comp rx=0 tx=0).stop 2022-06-27T11:48:02.051 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.047+0000 7fb29ce41700 1 -- 172.21.15.195:0/371027415 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb298068ea0 msgr2=0x7fb2980ba710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:02.051 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.047+0000 7fb29ce41700 1 --2- 172.21.15.195:0/371027415 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb298068ea0 0x7fb2980ba710 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb28800e970 tx=0x7fb288009060 comp rx=0 tx=0).stop 2022-06-27T11:48:02.051 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.047+0000 7fb29ce41700 1 -- 172.21.15.195:0/371027415 shutdown_connections 2022-06-27T11:48:02.052 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.047+0000 7fb29ce41700 1 --2- 172.21.15.195:0/371027415 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb274063d00 0x7fb2740661b0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:02.052 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.047+0000 7fb29ce41700 1 --2- 172.21.15.195:0/371027415 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb298068ea0 0x7fb2980ba710 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:02.052 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.047+0000 7fb29ce41700 1 --2- 172.21.15.195:0/371027415 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb2980c3010 0x7fb2980c7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:02.052 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.047+0000 7fb29ce41700 1 --2- 172.21.15.195:0/371027415 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb2980b8520 0x7fb2980bac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:02.053 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.047+0000 7fb29ce41700 1 -- 172.21.15.195:0/371027415 >> 172.21.15.195:0/371027415 conn(0x7fb2980c8740 msgr2=0x7fb2980c8b10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:02.053 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.047+0000 7fb29ce41700 1 -- 172.21.15.195:0/371027415 shutdown_connections 2022-06-27T11:48:02.053 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.047+0000 7fb29ce41700 1 -- 172.21.15.195:0/371027415 wait complete. 2022-06-27T11:48:02.053 INFO:teuthology.orchestra.run.smithi195.stderr:dumped all 2022-06-27T11:48:02.059 INFO:teuthology.orchestra.run.smithi195.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2022-06-27T11:48:00.341268+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592698,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":210,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":68768,"kb_used_data":1880,"kb_used_omap":0,"kb_used_meta":66816,"kb_avail":754905952,"statfs":{"total":773094113280,"available":773023694848,"internally_reserved":0,"allocated":1925120,"data_stored":1814102,"data_compressed":64756,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":68419584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":58,"num_write_kb":276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001822"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384962+0000","last_change":"2022-06-27T11:47:52.381823+0000","last_active":"2022-06-27T11:47:52.384962+0000","last_peered":"2022-06-27T11:47:52.384962+0000","last_clean":"2022-06-27T11:47:52.384962+0000","last_became_active":"2022-06-27T11:47:52.381551+0000","last_became_peered":"2022-06-27T11:47:52.381551+0000","last_unstale":"2022-06-27T11:47:52.384962+0000","last_undegraded":"2022-06-27T11:47:52.384962+0000","last_fullsized":"2022-06-27T11:47:52.384962+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:17:34.164061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.378135+0000","last_change":"2022-06-27T11:47:52.374939+0000","last_active":"2022-06-27T11:47:52.378135+0000","last_peered":"2022-06-27T11:47:52.378135+0000","last_clean":"2022-06-27T11:47:52.378135+0000","last_became_active":"2022-06-27T11:47:52.374620+0000","last_became_peered":"2022-06-27T11:47:52.374620+0000","last_unstale":"2022-06-27T11:47:52.378135+0000","last_undegraded":"2022-06-27T11:47:52.378135+0000","last_fullsized":"2022-06-27T11:47:52.378135+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:56:27.254098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384127+0000","last_change":"2022-06-27T11:47:52.380802+0000","last_active":"2022-06-27T11:47:52.384127+0000","last_peered":"2022-06-27T11:47:52.384127+0000","last_clean":"2022-06-27T11:47:52.384127+0000","last_became_active":"2022-06-27T11:47:52.380500+0000","last_became_peered":"2022-06-27T11:47:52.380500+0000","last_unstale":"2022-06-27T11:47:52.384127+0000","last_undegraded":"2022-06-27T11:47:52.384127+0000","last_fullsized":"2022-06-27T11:47:52.384127+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:06:42.889598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384114+0000","last_change":"2022-06-27T11:47:52.380943+0000","last_active":"2022-06-27T11:47:52.384114+0000","last_peered":"2022-06-27T11:47:52.384114+0000","last_clean":"2022-06-27T11:47:52.384114+0000","last_became_active":"2022-06-27T11:47:52.380648+0000","last_became_peered":"2022-06-27T11:47:52.380648+0000","last_unstale":"2022-06-27T11:47:52.384114+0000","last_undegraded":"2022-06-27T11:47:52.384114+0000","last_fullsized":"2022-06-27T11:47:52.384114+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:14:19.729605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.386875+0000","last_change":"2022-06-27T11:47:52.386144+0000","last_active":"2022-06-27T11:47:52.386875+0000","last_peered":"2022-06-27T11:47:52.386875+0000","last_clean":"2022-06-27T11:47:52.386875+0000","last_became_active":"2022-06-27T11:47:52.385586+0000","last_became_peered":"2022-06-27T11:47:52.385586+0000","last_unstale":"2022-06-27T11:47:52.386875+0000","last_undegraded":"2022-06-27T11:47:52.386875+0000","last_fullsized":"2022-06-27T11:47:52.386875+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T14:30:03.660749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.387182+0000","last_change":"2022-06-27T11:47:52.381906+0000","last_active":"2022-06-27T11:47:52.387182+0000","last_peered":"2022-06-27T11:47:52.387182+0000","last_clean":"2022-06-27T11:47:52.387182+0000","last_became_active":"2022-06-27T11:47:52.381279+0000","last_became_peered":"2022-06-27T11:47:52.381279+0000","last_unstale":"2022-06-27T11:47:52.387182+0000","last_undegraded":"2022-06-27T11:47:52.387182+0000","last_fullsized":"2022-06-27T11:47:52.387182+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T18:41:03.991762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.383137+0000","last_change":"2022-06-27T11:47:52.377128+0000","last_active":"2022-06-27T11:47:52.383137+0000","last_peered":"2022-06-27T11:47:52.383137+0000","last_clean":"2022-06-27T11:47:52.383137+0000","last_became_active":"2022-06-27T11:47:52.376499+0000","last_became_peered":"2022-06-27T11:47:52.376499+0000","last_unstale":"2022-06-27T11:47:52.383137+0000","last_undegraded":"2022-06-27T11:47:52.383137+0000","last_fullsized":"2022-06-27T11:47:52.383137+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:00:13.208145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.378171+0000","last_change":"2022-06-27T11:47:52.374169+0000","last_active":"2022-06-27T11:47:52.378171+0000","last_peered":"2022-06-27T11:47:52.378171+0000","last_clean":"2022-06-27T11:47:52.378171+0000","last_became_active":"2022-06-27T11:47:52.373917+0000","last_became_peered":"2022-06-27T11:47:52.373917+0000","last_unstale":"2022-06-27T11:47:52.378171+0000","last_undegraded":"2022-06-27T11:47:52.378171+0000","last_fullsized":"2022-06-27T11:47:52.378171+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:43:52.741254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384769+0000","last_change":"2022-06-27T11:47:52.382039+0000","last_active":"2022-06-27T11:47:52.384769+0000","last_peered":"2022-06-27T11:47:52.384769+0000","last_clean":"2022-06-27T11:47:52.384769+0000","last_became_active":"2022-06-27T11:47:52.381202+0000","last_became_peered":"2022-06-27T11:47:52.381202+0000","last_unstale":"2022-06-27T11:47:52.384769+0000","last_undegraded":"2022-06-27T11:47:52.384769+0000","last_fullsized":"2022-06-27T11:47:52.384769+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:13:02.553823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.383894+0000","last_change":"2022-06-27T11:47:52.380839+0000","last_active":"2022-06-27T11:47:52.383894+0000","last_peered":"2022-06-27T11:47:52.383894+0000","last_clean":"2022-06-27T11:47:52.383894+0000","last_became_active":"2022-06-27T11:47:52.380527+0000","last_became_peered":"2022-06-27T11:47:52.380527+0000","last_unstale":"2022-06-27T11:47:52.383894+0000","last_undegraded":"2022-06-27T11:47:52.383894+0000","last_fullsized":"2022-06-27T11:47:52.383894+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T23:18:07.086598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.374783+0000","last_change":"2022-06-27T11:47:52.371267+0000","last_active":"2022-06-27T11:47:52.374783+0000","last_peered":"2022-06-27T11:47:52.374783+0000","last_clean":"2022-06-27T11:47:52.374783+0000","last_became_active":"2022-06-27T11:47:52.370921+0000","last_became_peered":"2022-06-27T11:47:52.370921+0000","last_unstale":"2022-06-27T11:47:52.374783+0000","last_undegraded":"2022-06-27T11:47:52.374783+0000","last_fullsized":"2022-06-27T11:47:52.374783+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T19:22:39.648715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.378994+0000","last_change":"2022-06-27T11:47:52.375381+0000","last_active":"2022-06-27T11:47:52.378994+0000","last_peered":"2022-06-27T11:47:52.378994+0000","last_clean":"2022-06-27T11:47:52.378994+0000","last_became_active":"2022-06-27T11:47:52.375089+0000","last_became_peered":"2022-06-27T11:47:52.375089+0000","last_unstale":"2022-06-27T11:47:52.378994+0000","last_undegraded":"2022-06-27T11:47:52.378994+0000","last_fullsized":"2022-06-27T11:47:52.378994+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:43:14.830424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384993+0000","last_change":"2022-06-27T11:47:52.381690+0000","last_active":"2022-06-27T11:47:52.384993+0000","last_peered":"2022-06-27T11:47:52.384993+0000","last_clean":"2022-06-27T11:47:52.384993+0000","last_became_active":"2022-06-27T11:47:52.381472+0000","last_became_peered":"2022-06-27T11:47:52.381472+0000","last_unstale":"2022-06-27T11:47:52.384993+0000","last_undegraded":"2022-06-27T11:47:52.384993+0000","last_fullsized":"2022-06-27T11:47:52.384993+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:14:56.919536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.378838+0000","last_change":"2022-06-27T11:47:52.376063+0000","last_active":"2022-06-27T11:47:52.378838+0000","last_peered":"2022-06-27T11:47:52.378838+0000","last_clean":"2022-06-27T11:47:52.378838+0000","last_became_active":"2022-06-27T11:47:52.375778+0000","last_became_peered":"2022-06-27T11:47:52.375778+0000","last_unstale":"2022-06-27T11:47:52.378838+0000","last_undegraded":"2022-06-27T11:47:52.378838+0000","last_fullsized":"2022-06-27T11:47:52.378838+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T23:16:52.221967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.383160+0000","last_change":"2022-06-27T11:47:52.377297+0000","last_active":"2022-06-27T11:47:52.383160+0000","last_peered":"2022-06-27T11:47:52.383160+0000","last_clean":"2022-06-27T11:47:52.383160+0000","last_became_active":"2022-06-27T11:47:52.376855+0000","last_became_peered":"2022-06-27T11:47:52.376855+0000","last_unstale":"2022-06-27T11:47:52.383160+0000","last_undegraded":"2022-06-27T11:47:52.383160+0000","last_fullsized":"2022-06-27T11:47:52.383160+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:33:39.990522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.387168+0000","last_change":"2022-06-27T11:47:52.381821+0000","last_active":"2022-06-27T11:47:52.387168+0000","last_peered":"2022-06-27T11:47:52.387168+0000","last_clean":"2022-06-27T11:47:52.387168+0000","last_became_active":"2022-06-27T11:47:52.381548+0000","last_became_peered":"2022-06-27T11:47:52.381548+0000","last_unstale":"2022-06-27T11:47:52.387168+0000","last_undegraded":"2022-06-27T11:47:52.387168+0000","last_fullsized":"2022-06-27T11:47:52.387168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T23:14:53.320016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384753+0000","last_change":"2022-06-27T11:47:52.381811+0000","last_active":"2022-06-27T11:47:52.384753+0000","last_peered":"2022-06-27T11:47:52.384753+0000","last_clean":"2022-06-27T11:47:52.384753+0000","last_became_active":"2022-06-27T11:47:52.381554+0000","last_became_peered":"2022-06-27T11:47:52.381554+0000","last_unstale":"2022-06-27T11:47:52.384753+0000","last_undegraded":"2022-06-27T11:47:52.384753+0000","last_fullsized":"2022-06-27T11:47:52.384753+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T14:19:38.519380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.387218+0000","last_change":"2022-06-27T11:47:52.382952+0000","last_active":"2022-06-27T11:47:52.387218+0000","last_peered":"2022-06-27T11:47:52.387218+0000","last_clean":"2022-06-27T11:47:52.387218+0000","last_became_active":"2022-06-27T11:47:52.382691+0000","last_became_peered":"2022-06-27T11:47:52.382691+0000","last_unstale":"2022-06-27T11:47:52.387218+0000","last_undegraded":"2022-06-27T11:47:52.387218+0000","last_fullsized":"2022-06-27T11:47:52.387218+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:28:54.743970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.383682+0000","last_change":"2022-06-27T11:47:52.379083+0000","last_active":"2022-06-27T11:47:52.383682+0000","last_peered":"2022-06-27T11:47:52.383682+0000","last_clean":"2022-06-27T11:47:52.383682+0000","last_became_active":"2022-06-27T11:47:52.378724+0000","last_became_peered":"2022-06-27T11:47:52.378724+0000","last_unstale":"2022-06-27T11:47:52.383682+0000","last_undegraded":"2022-06-27T11:47:52.383682+0000","last_fullsized":"2022-06-27T11:47:52.383682+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T21:25:07.786935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.383275+0000","last_change":"2022-06-27T11:47:52.377947+0000","last_active":"2022-06-27T11:47:52.383275+0000","last_peered":"2022-06-27T11:47:52.383275+0000","last_clean":"2022-06-27T11:47:52.383275+0000","last_became_active":"2022-06-27T11:47:52.377652+0000","last_became_peered":"2022-06-27T11:47:52.377652+0000","last_unstale":"2022-06-27T11:47:52.383275+0000","last_undegraded":"2022-06-27T11:47:52.383275+0000","last_fullsized":"2022-06-27T11:47:52.383275+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T20:48:28.642052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384444+0000","last_change":"2022-06-27T11:47:52.381920+0000","last_active":"2022-06-27T11:47:52.384444+0000","last_peered":"2022-06-27T11:47:52.384444+0000","last_clean":"2022-06-27T11:47:52.384444+0000","last_became_active":"2022-06-27T11:47:52.381477+0000","last_became_peered":"2022-06-27T11:47:52.381477+0000","last_unstale":"2022-06-27T11:47:52.384444+0000","last_undegraded":"2022-06-27T11:47:52.384444+0000","last_fullsized":"2022-06-27T11:47:52.384444+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T20:42:25.529336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.386345+0000","last_change":"2022-06-27T11:47:52.385231+0000","last_active":"2022-06-27T11:47:52.386345+0000","last_peered":"2022-06-27T11:47:52.386345+0000","last_clean":"2022-06-27T11:47:52.386345+0000","last_became_active":"2022-06-27T11:47:52.384972+0000","last_became_peered":"2022-06-27T11:47:52.384972+0000","last_unstale":"2022-06-27T11:47:52.386345+0000","last_undegraded":"2022-06-27T11:47:52.386345+0000","last_fullsized":"2022-06-27T11:47:52.386345+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T17:37:26.084788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.385330+0000","last_change":"2022-06-27T11:47:52.381358+0000","last_active":"2022-06-27T11:47:52.385330+0000","last_peered":"2022-06-27T11:47:52.385330+0000","last_clean":"2022-06-27T11:47:52.385330+0000","last_became_active":"2022-06-27T11:47:52.380997+0000","last_became_peered":"2022-06-27T11:47:52.380997+0000","last_unstale":"2022-06-27T11:47:52.385330+0000","last_undegraded":"2022-06-27T11:47:52.385330+0000","last_fullsized":"2022-06-27T11:47:52.385330+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T21:44:16.061234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384432+0000","last_change":"2022-06-27T11:47:52.381899+0000","last_active":"2022-06-27T11:47:52.384432+0000","last_peered":"2022-06-27T11:47:52.384432+0000","last_clean":"2022-06-27T11:47:52.384432+0000","last_became_active":"2022-06-27T11:47:52.381645+0000","last_became_peered":"2022-06-27T11:47:52.381645+0000","last_unstale":"2022-06-27T11:47:52.384432+0000","last_undegraded":"2022-06-27T11:47:52.384432+0000","last_fullsized":"2022-06-27T11:47:52.384432+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:05:59.742023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.385053+0000","last_change":"2022-06-27T11:47:52.378736+0000","last_active":"2022-06-27T11:47:52.385053+0000","last_peered":"2022-06-27T11:47:52.385053+0000","last_clean":"2022-06-27T11:47:52.385053+0000","last_became_active":"2022-06-27T11:47:52.378475+0000","last_became_peered":"2022-06-27T11:47:52.378475+0000","last_unstale":"2022-06-27T11:47:52.385053+0000","last_undegraded":"2022-06-27T11:47:52.385053+0000","last_fullsized":"2022-06-27T11:47:52.385053+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T12:42:12.528716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384922+0000","last_change":"2022-06-27T11:47:52.378044+0000","last_active":"2022-06-27T11:47:52.384922+0000","last_peered":"2022-06-27T11:47:52.384922+0000","last_clean":"2022-06-27T11:47:52.384922+0000","last_became_active":"2022-06-27T11:47:52.377791+0000","last_became_peered":"2022-06-27T11:47:52.377791+0000","last_unstale":"2022-06-27T11:47:52.384922+0000","last_undegraded":"2022-06-27T11:47:52.384922+0000","last_fullsized":"2022-06-27T11:47:52.384922+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:21:45.615162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384115+0000","last_change":"2022-06-27T11:47:52.382267+0000","last_active":"2022-06-27T11:47:52.384115+0000","last_peered":"2022-06-27T11:47:52.384115+0000","last_clean":"2022-06-27T11:47:52.384115+0000","last_became_active":"2022-06-27T11:47:52.381985+0000","last_became_peered":"2022-06-27T11:47:52.381985+0000","last_unstale":"2022-06-27T11:47:52.384115+0000","last_undegraded":"2022-06-27T11:47:52.384115+0000","last_fullsized":"2022-06-27T11:47:52.384115+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:34:09.509759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:53.909338+0000","last_change":"2022-06-27T11:47:51.982183+0000","last_active":"2022-06-27T11:47:53.909338+0000","last_peered":"2022-06-27T11:47:53.909338+0000","last_clean":"2022-06-27T11:47:53.909338+0000","last_became_active":"2022-06-27T11:47:51.981339+0000","last_became_peered":"2022-06-27T11:47:51.981339+0000","last_unstale":"2022-06-27T11:47:53.909338+0000","last_undegraded":"2022-06-27T11:47:53.909338+0000","last_fullsized":"2022-06-27T11:47:53.909338+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:08:09.223947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384452+0000","last_change":"2022-06-27T11:47:52.377197+0000","last_active":"2022-06-27T11:47:52.384452+0000","last_peered":"2022-06-27T11:47:52.384452+0000","last_clean":"2022-06-27T11:47:52.384452+0000","last_became_active":"2022-06-27T11:47:52.376925+0000","last_became_peered":"2022-06-27T11:47:52.376925+0000","last_unstale":"2022-06-27T11:47:52.384452+0000","last_undegraded":"2022-06-27T11:47:52.384452+0000","last_fullsized":"2022-06-27T11:47:52.384452+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:39:27.103978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.364221+0000","last_change":"2022-06-27T11:47:51.981074+0000","last_active":"2022-06-27T11:47:52.364221+0000","last_peered":"2022-06-27T11:47:52.364221+0000","last_clean":"2022-06-27T11:47:52.364221+0000","last_became_active":"2022-06-27T11:47:51.980636+0000","last_became_peered":"2022-06-27T11:47:51.980636+0000","last_unstale":"2022-06-27T11:47:52.364221+0000","last_undegraded":"2022-06-27T11:47:52.364221+0000","last_fullsized":"2022-06-27T11:47:52.364221+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T23:32:52.850801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384380+0000","last_change":"2022-06-27T11:47:52.378415+0000","last_active":"2022-06-27T11:47:52.384380+0000","last_peered":"2022-06-27T11:47:52.384380+0000","last_clean":"2022-06-27T11:47:52.384380+0000","last_became_active":"2022-06-27T11:47:52.378123+0000","last_became_peered":"2022-06-27T11:47:52.378123+0000","last_unstale":"2022-06-27T11:47:52.384380+0000","last_undegraded":"2022-06-27T11:47:52.384380+0000","last_fullsized":"2022-06-27T11:47:52.384380+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T11:54:37.127828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.369137+0000","last_change":"2022-06-27T11:47:51.975684+0000","last_active":"2022-06-27T11:47:52.369137+0000","last_peered":"2022-06-27T11:47:52.369137+0000","last_clean":"2022-06-27T11:47:52.369137+0000","last_became_active":"2022-06-27T11:47:51.974902+0000","last_became_peered":"2022-06-27T11:47:51.974902+0000","last_unstale":"2022-06-27T11:47:52.369137+0000","last_undegraded":"2022-06-27T11:47:52.369137+0000","last_fullsized":"2022-06-27T11:47:52.369137+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:51:45.780798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.378008+0000","last_change":"2022-06-27T11:47:52.376089+0000","last_active":"2022-06-27T11:47:52.378008+0000","last_peered":"2022-06-27T11:47:52.378008+0000","last_clean":"2022-06-27T11:47:52.378008+0000","last_became_active":"2022-06-27T11:47:52.375780+0000","last_became_peered":"2022-06-27T11:47:52.375780+0000","last_unstale":"2022-06-27T11:47:52.378008+0000","last_undegraded":"2022-06-27T11:47:52.378008+0000","last_fullsized":"2022-06-27T11:47:52.378008+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T14:17:53.942880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.371313+0000","last_change":"2022-06-27T11:47:51.974071+0000","last_active":"2022-06-27T11:47:52.371313+0000","last_peered":"2022-06-27T11:47:52.371313+0000","last_clean":"2022-06-27T11:47:52.371313+0000","last_became_active":"2022-06-27T11:47:51.973682+0000","last_became_peered":"2022-06-27T11:47:51.973682+0000","last_unstale":"2022-06-27T11:47:52.371313+0000","last_undegraded":"2022-06-27T11:47:52.371313+0000","last_fullsized":"2022-06-27T11:47:52.371313+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T17:40:03.705083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384841+0000","last_change":"2022-06-27T11:47:52.377716+0000","last_active":"2022-06-27T11:47:52.384841+0000","last_peered":"2022-06-27T11:47:52.384841+0000","last_clean":"2022-06-27T11:47:52.384841+0000","last_became_active":"2022-06-27T11:47:52.377464+0000","last_became_peered":"2022-06-27T11:47:52.377464+0000","last_unstale":"2022-06-27T11:47:52.384841+0000","last_undegraded":"2022-06-27T11:47:52.384841+0000","last_fullsized":"2022-06-27T11:47:52.384841+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:39:36.234204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.368885+0000","last_change":"2022-06-27T11:47:51.975895+0000","last_active":"2022-06-27T11:47:52.368885+0000","last_peered":"2022-06-27T11:47:52.368885+0000","last_clean":"2022-06-27T11:47:52.368885+0000","last_became_active":"2022-06-27T11:47:51.975601+0000","last_became_peered":"2022-06-27T11:47:51.975601+0000","last_unstale":"2022-06-27T11:47:52.368885+0000","last_undegraded":"2022-06-27T11:47:52.368885+0000","last_fullsized":"2022-06-27T11:47:52.368885+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:43:55.833399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.385349+0000","last_change":"2022-06-27T11:47:52.383063+0000","last_active":"2022-06-27T11:47:52.385349+0000","last_peered":"2022-06-27T11:47:52.385349+0000","last_clean":"2022-06-27T11:47:52.385349+0000","last_became_active":"2022-06-27T11:47:52.382390+0000","last_became_peered":"2022-06-27T11:47:52.382390+0000","last_unstale":"2022-06-27T11:47:52.385349+0000","last_undegraded":"2022-06-27T11:47:52.385349+0000","last_fullsized":"2022-06-27T11:47:52.385349+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:40:10.929542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"21'3","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:57.367279+0000","last_change":"2022-06-27T11:47:51.980627+0000","last_active":"2022-06-27T11:47:57.367279+0000","last_peered":"2022-06-27T11:47:57.367279+0000","last_clean":"2022-06-27T11:47:57.367279+0000","last_became_active":"2022-06-27T11:47:51.980265+0000","last_became_peered":"2022-06-27T11:47:51.980265+0000","last_unstale":"2022-06-27T11:47:57.367279+0000","last_undegraded":"2022-06-27T11:47:57.367279+0000","last_fullsized":"2022-06-27T11:47:57.367279+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T14:18:32.599085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384584+0000","last_change":"2022-06-27T11:47:52.377475+0000","last_active":"2022-06-27T11:47:52.384584+0000","last_peered":"2022-06-27T11:47:52.384584+0000","last_clean":"2022-06-27T11:47:52.384584+0000","last_became_active":"2022-06-27T11:47:52.377220+0000","last_became_peered":"2022-06-27T11:47:52.377220+0000","last_unstale":"2022-06-27T11:47:52.384584+0000","last_undegraded":"2022-06-27T11:47:52.384584+0000","last_fullsized":"2022-06-27T11:47:52.384584+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T23:27:26.525688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"21'6","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:57.368255+0000","last_change":"2022-06-27T11:47:51.978195+0000","last_active":"2022-06-27T11:47:57.368255+0000","last_peered":"2022-06-27T11:47:57.368255+0000","last_clean":"2022-06-27T11:47:57.368255+0000","last_became_active":"2022-06-27T11:47:51.974673+0000","last_became_peered":"2022-06-27T11:47:51.974673+0000","last_unstale":"2022-06-27T11:47:57.368255+0000","last_undegraded":"2022-06-27T11:47:57.368255+0000","last_fullsized":"2022-06-27T11:47:57.368255+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T18:09:43.518644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.382204+0000","last_change":"2022-06-27T11:47:52.378740+0000","last_active":"2022-06-27T11:47:52.382204+0000","last_peered":"2022-06-27T11:47:52.382204+0000","last_clean":"2022-06-27T11:47:52.382204+0000","last_became_active":"2022-06-27T11:47:52.378451+0000","last_became_peered":"2022-06-27T11:47:52.378451+0000","last_unstale":"2022-06-27T11:47:52.382204+0000","last_undegraded":"2022-06-27T11:47:52.382204+0000","last_fullsized":"2022-06-27T11:47:52.382204+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:04:47.829409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.13","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:57.368186+0000","last_change":"2022-06-27T11:47:51.983039+0000","last_active":"2022-06-27T11:47:57.368186+0000","last_peered":"2022-06-27T11:47:57.368186+0000","last_clean":"2022-06-27T11:47:57.368186+0000","last_became_active":"2022-06-27T11:47:51.982683+0000","last_became_peered":"2022-06-27T11:47:51.982683+0000","last_unstale":"2022-06-27T11:47:57.368186+0000","last_undegraded":"2022-06-27T11:47:57.368186+0000","last_fullsized":"2022-06-27T11:47:57.368186+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T17:10:16.876753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384742+0000","last_change":"2022-06-27T11:47:52.374884+0000","last_active":"2022-06-27T11:47:52.384742+0000","last_peered":"2022-06-27T11:47:52.384742+0000","last_clean":"2022-06-27T11:47:52.384742+0000","last_became_active":"2022-06-27T11:47:52.374632+0000","last_became_peered":"2022-06-27T11:47:52.374632+0000","last_unstale":"2022-06-27T11:47:52.384742+0000","last_undegraded":"2022-06-27T11:47:52.384742+0000","last_fullsized":"2022-06-27T11:47:52.384742+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T21:33:54.724089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.374167+0000","last_change":"2022-06-27T11:47:51.976772+0000","last_active":"2022-06-27T11:47:52.374167+0000","last_peered":"2022-06-27T11:47:52.374167+0000","last_clean":"2022-06-27T11:47:52.374167+0000","last_became_active":"2022-06-27T11:47:51.976475+0000","last_became_peered":"2022-06-27T11:47:51.976475+0000","last_unstale":"2022-06-27T11:47:52.374167+0000","last_undegraded":"2022-06-27T11:47:52.374167+0000","last_fullsized":"2022-06-27T11:47:52.374167+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T23:39:10.919159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.377943+0000","last_change":"2022-06-27T11:47:52.373748+0000","last_active":"2022-06-27T11:47:52.377943+0000","last_peered":"2022-06-27T11:47:52.377943+0000","last_clean":"2022-06-27T11:47:52.377943+0000","last_became_active":"2022-06-27T11:47:52.373459+0000","last_became_peered":"2022-06-27T11:47:52.373459+0000","last_unstale":"2022-06-27T11:47:52.377943+0000","last_undegraded":"2022-06-27T11:47:52.377943+0000","last_fullsized":"2022-06-27T11:47:52.377943+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T18:27:22.254416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.367083+0000","last_change":"2022-06-27T11:47:51.982926+0000","last_active":"2022-06-27T11:47:52.367083+0000","last_peered":"2022-06-27T11:47:52.367083+0000","last_clean":"2022-06-27T11:47:52.367083+0000","last_became_active":"2022-06-27T11:47:51.982638+0000","last_became_peered":"2022-06-27T11:47:51.982638+0000","last_unstale":"2022-06-27T11:47:52.367083+0000","last_undegraded":"2022-06-27T11:47:52.367083+0000","last_fullsized":"2022-06-27T11:47:52.367083+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:34:04.834822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384791+0000","last_change":"2022-06-27T11:47:52.381623+0000","last_active":"2022-06-27T11:47:52.384791+0000","last_peered":"2022-06-27T11:47:52.384791+0000","last_clean":"2022-06-27T11:47:52.384791+0000","last_became_active":"2022-06-27T11:47:52.381258+0000","last_became_peered":"2022-06-27T11:47:52.381258+0000","last_unstale":"2022-06-27T11:47:52.384791+0000","last_undegraded":"2022-06-27T11:47:52.384791+0000","last_fullsized":"2022-06-27T11:47:52.384791+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:39:07.735224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"21'8","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:57.369384+0000","last_change":"2022-06-27T11:47:51.973125+0000","last_active":"2022-06-27T11:47:57.369384+0000","last_peered":"2022-06-27T11:47:57.369384+0000","last_clean":"2022-06-27T11:47:57.369384+0000","last_became_active":"2022-06-27T11:47:51.972806+0000","last_became_peered":"2022-06-27T11:47:51.972806+0000","last_unstale":"2022-06-27T11:47:57.369384+0000","last_undegraded":"2022-06-27T11:47:57.369384+0000","last_fullsized":"2022-06-27T11:47:57.369384+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:49:23.881991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384422+0000","last_change":"2022-06-27T11:47:52.376101+0000","last_active":"2022-06-27T11:47:52.384422+0000","last_peered":"2022-06-27T11:47:52.384422+0000","last_clean":"2022-06-27T11:47:52.384422+0000","last_became_active":"2022-06-27T11:47:52.375774+0000","last_became_peered":"2022-06-27T11:47:52.375774+0000","last_unstale":"2022-06-27T11:47:52.384422+0000","last_undegraded":"2022-06-27T11:47:52.384422+0000","last_fullsized":"2022-06-27T11:47:52.384422+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T18:21:13.049096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.f","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:53.492124+0000","last_change":"2022-06-27T11:47:51.981981+0000","last_active":"2022-06-27T11:47:53.492124+0000","last_peered":"2022-06-27T11:47:53.492124+0000","last_clean":"2022-06-27T11:47:53.492124+0000","last_became_active":"2022-06-27T11:47:51.981749+0000","last_became_peered":"2022-06-27T11:47:51.981749+0000","last_unstale":"2022-06-27T11:47:53.492124+0000","last_undegraded":"2022-06-27T11:47:53.492124+0000","last_fullsized":"2022-06-27T11:47:53.492124+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:28:52.457719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.378190+0000","last_change":"2022-06-27T11:47:52.374223+0000","last_active":"2022-06-27T11:47:52.378190+0000","last_peered":"2022-06-27T11:47:52.378190+0000","last_clean":"2022-06-27T11:47:52.378190+0000","last_became_active":"2022-06-27T11:47:52.373932+0000","last_became_peered":"2022-06-27T11:47:52.373932+0000","last_unstale":"2022-06-27T11:47:52.378190+0000","last_undegraded":"2022-06-27T11:47:52.378190+0000","last_fullsized":"2022-06-27T11:47:52.378190+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:48:33.254458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:57.368190+0000","last_change":"2022-06-27T11:47:51.982895+0000","last_active":"2022-06-27T11:47:57.368190+0000","last_peered":"2022-06-27T11:47:57.368190+0000","last_clean":"2022-06-27T11:47:57.368190+0000","last_became_active":"2022-06-27T11:47:51.982619+0000","last_became_peered":"2022-06-27T11:47:51.982619+0000","last_unstale":"2022-06-27T11:47:57.368190+0000","last_undegraded":"2022-06-27T11:47:57.368190+0000","last_fullsized":"2022-06-27T11:47:57.368190+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:22:01.241112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.377169+0000","last_change":"2022-06-27T11:47:52.375225+0000","last_active":"2022-06-27T11:47:52.377169+0000","last_peered":"2022-06-27T11:47:52.377169+0000","last_clean":"2022-06-27T11:47:52.377169+0000","last_became_active":"2022-06-27T11:47:52.374925+0000","last_became_peered":"2022-06-27T11:47:52.374925+0000","last_unstale":"2022-06-27T11:47:52.377169+0000","last_undegraded":"2022-06-27T11:47:52.377169+0000","last_fullsized":"2022-06-27T11:47:52.377169+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:54:31.945902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:53.492085+0000","last_change":"2022-06-27T11:47:51.975202+0000","last_active":"2022-06-27T11:47:53.492085+0000","last_peered":"2022-06-27T11:47:53.492085+0000","last_clean":"2022-06-27T11:47:53.492085+0000","last_became_active":"2022-06-27T11:47:51.974968+0000","last_became_peered":"2022-06-27T11:47:51.974968+0000","last_unstale":"2022-06-27T11:47:53.492085+0000","last_undegraded":"2022-06-27T11:47:53.492085+0000","last_fullsized":"2022-06-27T11:47:53.492085+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T21:16:30.362248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.378431+0000","last_change":"2022-06-27T11:47:52.377794+0000","last_active":"2022-06-27T11:47:52.378431+0000","last_peered":"2022-06-27T11:47:52.378431+0000","last_clean":"2022-06-27T11:47:52.378431+0000","last_became_active":"2022-06-27T11:47:52.377387+0000","last_became_peered":"2022-06-27T11:47:52.377387+0000","last_unstale":"2022-06-27T11:47:52.378431+0000","last_undegraded":"2022-06-27T11:47:52.378431+0000","last_fullsized":"2022-06-27T11:47:52.378431+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T17:00:00.924967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.368792+0000","last_change":"2022-06-27T11:47:51.976575+0000","last_active":"2022-06-27T11:47:52.368792+0000","last_peered":"2022-06-27T11:47:52.368792+0000","last_clean":"2022-06-27T11:47:52.368792+0000","last_became_active":"2022-06-27T11:47:51.976300+0000","last_became_peered":"2022-06-27T11:47:51.976300+0000","last_unstale":"2022-06-27T11:47:52.368792+0000","last_undegraded":"2022-06-27T11:47:52.368792+0000","last_fullsized":"2022-06-27T11:47:52.368792+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:08:46.067375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.387174+0000","last_change":"2022-06-27T11:47:52.386298+0000","last_active":"2022-06-27T11:47:52.387174+0000","last_peered":"2022-06-27T11:47:52.387174+0000","last_clean":"2022-06-27T11:47:52.387174+0000","last_became_active":"2022-06-27T11:47:52.385918+0000","last_became_peered":"2022-06-27T11:47:52.385918+0000","last_unstale":"2022-06-27T11:47:52.387174+0000","last_undegraded":"2022-06-27T11:47:52.387174+0000","last_fullsized":"2022-06-27T11:47:52.387174+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:21:51.780895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:57.364776+0000","last_change":"2022-06-27T11:47:51.973817+0000","last_active":"2022-06-27T11:47:57.364776+0000","last_peered":"2022-06-27T11:47:57.364776+0000","last_clean":"2022-06-27T11:47:57.364776+0000","last_became_active":"2022-06-27T11:47:51.973515+0000","last_became_peered":"2022-06-27T11:47:51.973515+0000","last_unstale":"2022-06-27T11:47:57.364776+0000","last_undegraded":"2022-06-27T11:47:57.364776+0000","last_fullsized":"2022-06-27T11:47:57.364776+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T21:01:45.170590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.374582+0000","last_change":"2022-06-27T11:47:52.371141+0000","last_active":"2022-06-27T11:47:52.374582+0000","last_peered":"2022-06-27T11:47:52.374582+0000","last_clean":"2022-06-27T11:47:52.374582+0000","last_became_active":"2022-06-27T11:47:52.370796+0000","last_became_peered":"2022-06-27T11:47:52.370796+0000","last_unstale":"2022-06-27T11:47:52.374582+0000","last_undegraded":"2022-06-27T11:47:52.374582+0000","last_fullsized":"2022-06-27T11:47:52.374582+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T23:16:39.266615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.369465+0000","last_change":"2022-06-27T11:47:51.983071+0000","last_active":"2022-06-27T11:47:52.369465+0000","last_peered":"2022-06-27T11:47:52.369465+0000","last_clean":"2022-06-27T11:47:52.369465+0000","last_became_active":"2022-06-27T11:47:51.982756+0000","last_became_peered":"2022-06-27T11:47:51.982756+0000","last_unstale":"2022-06-27T11:47:52.369465+0000","last_undegraded":"2022-06-27T11:47:52.369465+0000","last_fullsized":"2022-06-27T11:47:52.369465+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T17:14:00.737599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.387222+0000","last_change":"2022-06-27T11:47:52.382943+0000","last_active":"2022-06-27T11:47:52.387222+0000","last_peered":"2022-06-27T11:47:52.387222+0000","last_clean":"2022-06-27T11:47:52.387222+0000","last_became_active":"2022-06-27T11:47:52.382625+0000","last_became_peered":"2022-06-27T11:47:52.382625+0000","last_unstale":"2022-06-27T11:47:52.387222+0000","last_undegraded":"2022-06-27T11:47:52.387222+0000","last_fullsized":"2022-06-27T11:47:52.387222+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:01:18.570554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.362681+0000","last_change":"2022-06-27T11:47:51.978856+0000","last_active":"2022-06-27T11:47:52.362681+0000","last_peered":"2022-06-27T11:47:52.362681+0000","last_clean":"2022-06-27T11:47:52.362681+0000","last_became_active":"2022-06-27T11:47:51.978541+0000","last_became_peered":"2022-06-27T11:47:51.978541+0000","last_unstale":"2022-06-27T11:47:52.362681+0000","last_undegraded":"2022-06-27T11:47:52.362681+0000","last_fullsized":"2022-06-27T11:47:52.362681+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:36:24.190996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.378203+0000","last_change":"2022-06-27T11:47:52.376925+0000","last_active":"2022-06-27T11:47:52.378203+0000","last_peered":"2022-06-27T11:47:52.378203+0000","last_clean":"2022-06-27T11:47:52.378203+0000","last_became_active":"2022-06-27T11:47:52.376646+0000","last_became_peered":"2022-06-27T11:47:52.376646+0000","last_unstale":"2022-06-27T11:47:52.378203+0000","last_undegraded":"2022-06-27T11:47:52.378203+0000","last_fullsized":"2022-06-27T11:47:52.378203+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:17:51.913392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.369021+0000","last_change":"2022-06-27T11:47:51.979355+0000","last_active":"2022-06-27T11:47:52.369021+0000","last_peered":"2022-06-27T11:47:52.369021+0000","last_clean":"2022-06-27T11:47:52.369021+0000","last_became_active":"2022-06-27T11:47:51.977591+0000","last_became_peered":"2022-06-27T11:47:51.977591+0000","last_unstale":"2022-06-27T11:47:52.369021+0000","last_undegraded":"2022-06-27T11:47:52.369021+0000","last_fullsized":"2022-06-27T11:47:52.369021+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T11:50:43.086014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.385028+0000","last_change":"2022-06-27T11:47:52.375271+0000","last_active":"2022-06-27T11:47:52.385028+0000","last_peered":"2022-06-27T11:47:52.385028+0000","last_clean":"2022-06-27T11:47:52.385028+0000","last_became_active":"2022-06-27T11:47:52.374985+0000","last_became_peered":"2022-06-27T11:47:52.374985+0000","last_unstale":"2022-06-27T11:47:52.385028+0000","last_undegraded":"2022-06-27T11:47:52.385028+0000","last_fullsized":"2022-06-27T11:47:52.385028+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T17:58:40.711409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:57.367534+0000","last_change":"2022-06-27T11:47:51.979723+0000","last_active":"2022-06-27T11:47:57.367534+0000","last_peered":"2022-06-27T11:47:57.367534+0000","last_clean":"2022-06-27T11:47:57.367534+0000","last_became_active":"2022-06-27T11:47:51.977669+0000","last_became_peered":"2022-06-27T11:47:51.977669+0000","last_unstale":"2022-06-27T11:47:57.367534+0000","last_undegraded":"2022-06-27T11:47:57.367534+0000","last_fullsized":"2022-06-27T11:47:57.367534+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T14:35:14.770254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.363444+0000","last_change":"2022-06-27T11:47:48.388880+0000","last_active":"2022-06-27T11:47:52.363444+0000","last_peered":"2022-06-27T11:47:52.363444+0000","last_clean":"2022-06-27T11:47:52.363444+0000","last_became_active":"2022-06-27T11:47:48.388639+0000","last_became_peered":"2022-06-27T11:47:48.388639+0000","last_unstale":"2022-06-27T11:47:52.363444+0000","last_undegraded":"2022-06-27T11:47:52.363444+0000","last_fullsized":"2022-06-27T11:47:52.363444+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_clean_scrub_stamp":"2022-06-27T11:47:46.780512+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:31:10.549485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.380026+0000","last_change":"2022-06-27T11:47:52.373143+0000","last_active":"2022-06-27T11:47:52.380026+0000","last_peered":"2022-06-27T11:47:52.380026+0000","last_clean":"2022-06-27T11:47:52.380026+0000","last_became_active":"2022-06-27T11:47:52.372850+0000","last_became_peered":"2022-06-27T11:47:52.372850+0000","last_unstale":"2022-06-27T11:47:52.380026+0000","last_undegraded":"2022-06-27T11:47:52.380026+0000","last_fullsized":"2022-06-27T11:47:52.380026+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:48:44.251459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.369716+0000","last_change":"2022-06-27T11:47:47.793307+0000","last_active":"2022-06-27T11:47:52.369716+0000","last_peered":"2022-06-27T11:47:52.369716+0000","last_clean":"2022-06-27T11:47:52.369716+0000","last_became_active":"2022-06-27T11:47:47.792958+0000","last_became_peered":"2022-06-27T11:47:47.792958+0000","last_unstale":"2022-06-27T11:47:52.369716+0000","last_undegraded":"2022-06-27T11:47:52.369716+0000","last_fullsized":"2022-06-27T11:47:52.369716+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_clean_scrub_stamp":"2022-06-27T11:47:46.780512+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:08:35.128804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.372474+0000","last_change":"2022-06-27T11:47:51.976872+0000","last_active":"2022-06-27T11:47:52.372474+0000","last_peered":"2022-06-27T11:47:52.372474+0000","last_clean":"2022-06-27T11:47:52.372474+0000","last_became_active":"2022-06-27T11:47:51.976546+0000","last_became_peered":"2022-06-27T11:47:51.976546+0000","last_unstale":"2022-06-27T11:47:52.372474+0000","last_undegraded":"2022-06-27T11:47:52.372474+0000","last_fullsized":"2022-06-27T11:47:52.372474+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T14:44:43.349688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.375090+0000","last_change":"2022-06-27T11:47:52.372794+0000","last_active":"2022-06-27T11:47:52.375090+0000","last_peered":"2022-06-27T11:47:52.375090+0000","last_clean":"2022-06-27T11:47:52.375090+0000","last_became_active":"2022-06-27T11:47:52.372486+0000","last_became_peered":"2022-06-27T11:47:52.372486+0000","last_unstale":"2022-06-27T11:47:52.375090+0000","last_undegraded":"2022-06-27T11:47:52.375090+0000","last_fullsized":"2022-06-27T11:47:52.375090+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:23:56.307995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.363787+0000","last_change":"2022-06-27T11:47:47.792631+0000","last_active":"2022-06-27T11:47:52.363787+0000","last_peered":"2022-06-27T11:47:52.363787+0000","last_clean":"2022-06-27T11:47:52.363787+0000","last_became_active":"2022-06-27T11:47:47.792282+0000","last_became_peered":"2022-06-27T11:47:47.792282+0000","last_unstale":"2022-06-27T11:47:52.363787+0000","last_undegraded":"2022-06-27T11:47:52.363787+0000","last_fullsized":"2022-06-27T11:47:52.363787+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_clean_scrub_stamp":"2022-06-27T11:47:46.780512+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:41:25.449379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"17'117","reported_seq":163,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.368104+0000","last_change":"2022-06-27T11:47:47.791697+0000","last_active":"2022-06-27T11:47:52.368104+0000","last_peered":"2022-06-27T11:47:52.368104+0000","last_clean":"2022-06-27T11:47:52.368104+0000","last_became_active":"2022-06-27T11:47:47.790645+0000","last_became_peered":"2022-06-27T11:47:47.790645+0000","last_unstale":"2022-06-27T11:47:52.368104+0000","last_undegraded":"2022-06-27T11:47:52.368104+0000","last_fullsized":"2022-06-27T11:47:52.368104+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_clean_scrub_stamp":"2022-06-27T11:47:46.780512+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:11:09.344666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.367962+0000","last_change":"2022-06-27T11:47:51.982728+0000","last_active":"2022-06-27T11:47:52.367962+0000","last_peered":"2022-06-27T11:47:52.367962+0000","last_clean":"2022-06-27T11:47:52.367962+0000","last_became_active":"2022-06-27T11:47:51.982445+0000","last_became_peered":"2022-06-27T11:47:51.982445+0000","last_unstale":"2022-06-27T11:47:52.367962+0000","last_undegraded":"2022-06-27T11:47:52.367962+0000","last_fullsized":"2022-06-27T11:47:52.367962+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:42:04.682687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.386922+0000","last_change":"2022-06-27T11:47:52.386015+0000","last_active":"2022-06-27T11:47:52.386922+0000","last_peered":"2022-06-27T11:47:52.386922+0000","last_clean":"2022-06-27T11:47:52.386922+0000","last_became_active":"2022-06-27T11:47:52.385731+0000","last_became_peered":"2022-06-27T11:47:52.385731+0000","last_unstale":"2022-06-27T11:47:52.386922+0000","last_undegraded":"2022-06-27T11:47:52.386922+0000","last_fullsized":"2022-06-27T11:47:52.386922+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T11:58:50.483766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.366627+0000","last_change":"2022-06-27T11:47:47.807699+0000","last_active":"2022-06-27T11:47:52.366627+0000","last_peered":"2022-06-27T11:47:52.366627+0000","last_clean":"2022-06-27T11:47:52.366627+0000","last_became_active":"2022-06-27T11:47:47.807422+0000","last_became_peered":"2022-06-27T11:47:47.807422+0000","last_unstale":"2022-06-27T11:47:52.366627+0000","last_undegraded":"2022-06-27T11:47:52.366627+0000","last_fullsized":"2022-06-27T11:47:52.366627+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_clean_scrub_stamp":"2022-06-27T11:47:46.780512+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:15:00.549295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.363922+0000","last_change":"2022-06-27T11:47:51.974468+0000","last_active":"2022-06-27T11:47:52.363922+0000","last_peered":"2022-06-27T11:47:52.363922+0000","last_clean":"2022-06-27T11:47:52.363922+0000","last_became_active":"2022-06-27T11:47:51.974160+0000","last_became_peered":"2022-06-27T11:47:51.974160+0000","last_unstale":"2022-06-27T11:47:52.363922+0000","last_undegraded":"2022-06-27T11:47:52.363922+0000","last_fullsized":"2022-06-27T11:47:52.363922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T19:14:45.694282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.380162+0000","last_change":"2022-06-27T11:47:52.373931+0000","last_active":"2022-06-27T11:47:52.380162+0000","last_peered":"2022-06-27T11:47:52.380162+0000","last_clean":"2022-06-27T11:47:52.380162+0000","last_became_active":"2022-06-27T11:47:52.373562+0000","last_became_peered":"2022-06-27T11:47:52.373562+0000","last_unstale":"2022-06-27T11:47:52.380162+0000","last_undegraded":"2022-06-27T11:47:52.380162+0000","last_fullsized":"2022-06-27T11:47:52.380162+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T14:09:22.859955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.367866+0000","last_change":"2022-06-27T11:47:47.807650+0000","last_active":"2022-06-27T11:47:52.367866+0000","last_peered":"2022-06-27T11:47:52.367866+0000","last_clean":"2022-06-27T11:47:52.367866+0000","last_became_active":"2022-06-27T11:47:47.807375+0000","last_became_peered":"2022-06-27T11:47:47.807375+0000","last_unstale":"2022-06-27T11:47:52.367866+0000","last_undegraded":"2022-06-27T11:47:52.367866+0000","last_fullsized":"2022-06-27T11:47:52.367866+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_clean_scrub_stamp":"2022-06-27T11:47:46.780512+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T14:10:48.067284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.368898+0000","last_change":"2022-06-27T11:47:51.972154+0000","last_active":"2022-06-27T11:47:52.368898+0000","last_peered":"2022-06-27T11:47:52.368898+0000","last_clean":"2022-06-27T11:47:52.368898+0000","last_became_active":"2022-06-27T11:47:51.971762+0000","last_became_peered":"2022-06-27T11:47:51.971762+0000","last_unstale":"2022-06-27T11:47:52.368898+0000","last_undegraded":"2022-06-27T11:47:52.368898+0000","last_fullsized":"2022-06-27T11:47:52.368898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T18:15:14.271005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.379879+0000","last_change":"2022-06-27T11:47:52.378294+0000","last_active":"2022-06-27T11:47:52.379879+0000","last_peered":"2022-06-27T11:47:52.379879+0000","last_clean":"2022-06-27T11:47:52.379879+0000","last_became_active":"2022-06-27T11:47:52.377704+0000","last_became_peered":"2022-06-27T11:47:52.377704+0000","last_unstale":"2022-06-27T11:47:52.379879+0000","last_undegraded":"2022-06-27T11:47:52.379879+0000","last_fullsized":"2022-06-27T11:47:52.379879+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:26:34.474081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.364180+0000","last_change":"2022-06-27T11:47:47.807600+0000","last_active":"2022-06-27T11:47:52.364180+0000","last_peered":"2022-06-27T11:47:52.364180+0000","last_clean":"2022-06-27T11:47:52.364180+0000","last_became_active":"2022-06-27T11:47:47.807284+0000","last_became_peered":"2022-06-27T11:47:47.807284+0000","last_unstale":"2022-06-27T11:47:52.364180+0000","last_undegraded":"2022-06-27T11:47:52.364180+0000","last_fullsized":"2022-06-27T11:47:52.364180+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_clean_scrub_stamp":"2022-06-27T11:47:46.780512+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T20:57:06.487027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.362687+0000","last_change":"2022-06-27T11:47:51.980734+0000","last_active":"2022-06-27T11:47:52.362687+0000","last_peered":"2022-06-27T11:47:52.362687+0000","last_clean":"2022-06-27T11:47:52.362687+0000","last_became_active":"2022-06-27T11:47:51.980476+0000","last_became_peered":"2022-06-27T11:47:51.980476+0000","last_unstale":"2022-06-27T11:47:52.362687+0000","last_undegraded":"2022-06-27T11:47:52.362687+0000","last_fullsized":"2022-06-27T11:47:52.362687+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T21:32:16.435472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.385478+0000","last_change":"2022-06-27T11:47:52.379902+0000","last_active":"2022-06-27T11:47:52.385478+0000","last_peered":"2022-06-27T11:47:52.385478+0000","last_clean":"2022-06-27T11:47:52.385478+0000","last_became_active":"2022-06-27T11:47:52.379359+0000","last_became_peered":"2022-06-27T11:47:52.379359+0000","last_unstale":"2022-06-27T11:47:52.385478+0000","last_undegraded":"2022-06-27T11:47:52.385478+0000","last_fullsized":"2022-06-27T11:47:52.385478+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:41:27.884083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.370459+0000","last_change":"2022-06-27T11:47:47.807607+0000","last_active":"2022-06-27T11:47:52.370459+0000","last_peered":"2022-06-27T11:47:52.370459+0000","last_clean":"2022-06-27T11:47:52.370459+0000","last_became_active":"2022-06-27T11:47:47.807293+0000","last_became_peered":"2022-06-27T11:47:47.807293+0000","last_unstale":"2022-06-27T11:47:52.370459+0000","last_undegraded":"2022-06-27T11:47:52.370459+0000","last_fullsized":"2022-06-27T11:47:52.370459+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_clean_scrub_stamp":"2022-06-27T11:47:46.780512+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:00:35.318372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.364133+0000","last_change":"2022-06-27T11:47:51.976055+0000","last_active":"2022-06-27T11:47:52.364133+0000","last_peered":"2022-06-27T11:47:52.364133+0000","last_clean":"2022-06-27T11:47:52.364133+0000","last_became_active":"2022-06-27T11:47:51.975303+0000","last_became_peered":"2022-06-27T11:47:51.975303+0000","last_unstale":"2022-06-27T11:47:52.364133+0000","last_undegraded":"2022-06-27T11:47:52.364133+0000","last_fullsized":"2022-06-27T11:47:52.364133+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:27:32.125525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384750+0000","last_change":"2022-06-27T11:47:52.380775+0000","last_active":"2022-06-27T11:47:52.384750+0000","last_peered":"2022-06-27T11:47:52.384750+0000","last_clean":"2022-06-27T11:47:52.384750+0000","last_became_active":"2022-06-27T11:47:52.380439+0000","last_became_peered":"2022-06-27T11:47:52.380439+0000","last_unstale":"2022-06-27T11:47:52.384750+0000","last_undegraded":"2022-06-27T11:47:52.384750+0000","last_fullsized":"2022-06-27T11:47:52.384750+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:03:44.939329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.370743+0000","last_change":"2022-06-27T11:47:47.792314+0000","last_active":"2022-06-27T11:47:52.370743+0000","last_peered":"2022-06-27T11:47:52.370743+0000","last_clean":"2022-06-27T11:47:52.370743+0000","last_became_active":"2022-06-27T11:47:47.792032+0000","last_became_peered":"2022-06-27T11:47:47.792032+0000","last_unstale":"2022-06-27T11:47:52.370743+0000","last_undegraded":"2022-06-27T11:47:52.370743+0000","last_fullsized":"2022-06-27T11:47:52.370743+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_clean_scrub_stamp":"2022-06-27T11:47:46.780512+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T14:13:03.591775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.369208+0000","last_change":"2022-06-27T11:47:51.978639+0000","last_active":"2022-06-27T11:47:52.369208+0000","last_peered":"2022-06-27T11:47:52.369208+0000","last_clean":"2022-06-27T11:47:52.369208+0000","last_became_active":"2022-06-27T11:47:51.978341+0000","last_became_peered":"2022-06-27T11:47:51.978341+0000","last_unstale":"2022-06-27T11:47:52.369208+0000","last_undegraded":"2022-06-27T11:47:52.369208+0000","last_fullsized":"2022-06-27T11:47:52.369208+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T21:34:48.070558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.381133+0000","last_change":"2022-06-27T11:47:52.373588+0000","last_active":"2022-06-27T11:47:52.381133+0000","last_peered":"2022-06-27T11:47:52.381133+0000","last_clean":"2022-06-27T11:47:52.381133+0000","last_became_active":"2022-06-27T11:47:52.373208+0000","last_became_peered":"2022-06-27T11:47:52.373208+0000","last_unstale":"2022-06-27T11:47:52.381133+0000","last_undegraded":"2022-06-27T11:47:52.381133+0000","last_fullsized":"2022-06-27T11:47:52.381133+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T20:27:28.252147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:53.910019+0000","last_change":"2022-06-27T11:47:51.982051+0000","last_active":"2022-06-27T11:47:53.910019+0000","last_peered":"2022-06-27T11:47:53.910019+0000","last_clean":"2022-06-27T11:47:53.910019+0000","last_became_active":"2022-06-27T11:47:51.981742+0000","last_became_peered":"2022-06-27T11:47:51.981742+0000","last_unstale":"2022-06-27T11:47:53.910019+0000","last_undegraded":"2022-06-27T11:47:53.910019+0000","last_fullsized":"2022-06-27T11:47:53.910019+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:39:56.937086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.380333+0000","last_change":"2022-06-27T11:47:52.378192+0000","last_active":"2022-06-27T11:47:52.380333+0000","last_peered":"2022-06-27T11:47:52.380333+0000","last_clean":"2022-06-27T11:47:52.380333+0000","last_became_active":"2022-06-27T11:47:52.377919+0000","last_became_peered":"2022-06-27T11:47:52.377919+0000","last_unstale":"2022-06-27T11:47:52.380333+0000","last_undegraded":"2022-06-27T11:47:52.380333+0000","last_fullsized":"2022-06-27T11:47:52.380333+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T18:51:35.757778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:57.368777+0000","last_change":"2022-06-27T11:47:51.981101+0000","last_active":"2022-06-27T11:47:57.368777+0000","last_peered":"2022-06-27T11:47:57.368777+0000","last_clean":"2022-06-27T11:47:57.368777+0000","last_became_active":"2022-06-27T11:47:51.980793+0000","last_became_peered":"2022-06-27T11:47:51.980793+0000","last_unstale":"2022-06-27T11:47:57.368777+0000","last_undegraded":"2022-06-27T11:47:57.368777+0000","last_fullsized":"2022-06-27T11:47:57.368777+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:28:10.206148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.372181+0000","last_change":"2022-06-27T11:47:52.370334+0000","last_active":"2022-06-27T11:47:52.372181+0000","last_peered":"2022-06-27T11:47:52.372181+0000","last_clean":"2022-06-27T11:47:52.372181+0000","last_became_active":"2022-06-27T11:47:52.370028+0000","last_became_peered":"2022-06-27T11:47:52.370028+0000","last_unstale":"2022-06-27T11:47:52.372181+0000","last_undegraded":"2022-06-27T11:47:52.372181+0000","last_fullsized":"2022-06-27T11:47:52.372181+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T17:28:12.357757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:57.369040+0000","last_change":"2022-06-27T11:47:51.981186+0000","last_active":"2022-06-27T11:47:57.369040+0000","last_peered":"2022-06-27T11:47:57.369040+0000","last_clean":"2022-06-27T11:47:57.369040+0000","last_became_active":"2022-06-27T11:47:51.980921+0000","last_became_peered":"2022-06-27T11:47:51.980921+0000","last_unstale":"2022-06-27T11:47:57.369040+0000","last_undegraded":"2022-06-27T11:47:57.369040+0000","last_fullsized":"2022-06-27T11:47:57.369040+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:04:40.732299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.387201+0000","last_change":"2022-06-27T11:47:52.382091+0000","last_active":"2022-06-27T11:47:52.387201+0000","last_peered":"2022-06-27T11:47:52.387201+0000","last_clean":"2022-06-27T11:47:52.387201+0000","last_became_active":"2022-06-27T11:47:52.381633+0000","last_became_peered":"2022-06-27T11:47:52.381633+0000","last_unstale":"2022-06-27T11:47:52.387201+0000","last_undegraded":"2022-06-27T11:47:52.387201+0000","last_fullsized":"2022-06-27T11:47:52.387201+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:34:44.030534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:53.910420+0000","last_change":"2022-06-27T11:47:51.972963+0000","last_active":"2022-06-27T11:47:53.910420+0000","last_peered":"2022-06-27T11:47:53.910420+0000","last_clean":"2022-06-27T11:47:53.910420+0000","last_became_active":"2022-06-27T11:47:51.972686+0000","last_became_peered":"2022-06-27T11:47:51.972686+0000","last_unstale":"2022-06-27T11:47:53.910420+0000","last_undegraded":"2022-06-27T11:47:53.910420+0000","last_fullsized":"2022-06-27T11:47:53.910420+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T12:11:38.008538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.385365+0000","last_change":"2022-06-27T11:47:52.383414+0000","last_active":"2022-06-27T11:47:52.385365+0000","last_peered":"2022-06-27T11:47:52.385365+0000","last_clean":"2022-06-27T11:47:52.385365+0000","last_became_active":"2022-06-27T11:47:52.383070+0000","last_became_peered":"2022-06-27T11:47:52.383070+0000","last_unstale":"2022-06-27T11:47:52.385365+0000","last_undegraded":"2022-06-27T11:47:52.385365+0000","last_fullsized":"2022-06-27T11:47:52.385365+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:11:28.410045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"21'4","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:57.370018+0000","last_change":"2022-06-27T11:47:51.971664+0000","last_active":"2022-06-27T11:47:57.370018+0000","last_peered":"2022-06-27T11:47:57.370018+0000","last_clean":"2022-06-27T11:47:57.370018+0000","last_became_active":"2022-06-27T11:47:51.971385+0000","last_became_peered":"2022-06-27T11:47:51.971385+0000","last_unstale":"2022-06-27T11:47:57.370018+0000","last_undegraded":"2022-06-27T11:47:57.370018+0000","last_fullsized":"2022-06-27T11:47:57.370018+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T18:44:38.394388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.379723+0000","last_change":"2022-06-27T11:47:52.377305+0000","last_active":"2022-06-27T11:47:52.379723+0000","last_peered":"2022-06-27T11:47:52.379723+0000","last_clean":"2022-06-27T11:47:52.379723+0000","last_became_active":"2022-06-27T11:47:52.377012+0000","last_became_peered":"2022-06-27T11:47:52.377012+0000","last_unstale":"2022-06-27T11:47:52.379723+0000","last_undegraded":"2022-06-27T11:47:52.379723+0000","last_fullsized":"2022-06-27T11:47:52.379723+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:15:14.505656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.377592+0000","last_change":"2022-06-27T11:47:52.376074+0000","last_active":"2022-06-27T11:47:52.377592+0000","last_peered":"2022-06-27T11:47:52.377592+0000","last_clean":"2022-06-27T11:47:52.377592+0000","last_became_active":"2022-06-27T11:47:52.375836+0000","last_became_peered":"2022-06-27T11:47:52.375836+0000","last_unstale":"2022-06-27T11:47:52.377592+0000","last_undegraded":"2022-06-27T11:47:52.377592+0000","last_fullsized":"2022-06-27T11:47:52.377592+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T18:46:26.900833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384440+0000","last_change":"2022-06-27T11:47:52.376668+0000","last_active":"2022-06-27T11:47:52.384440+0000","last_peered":"2022-06-27T11:47:52.384440+0000","last_clean":"2022-06-27T11:47:52.384440+0000","last_became_active":"2022-06-27T11:47:52.376386+0000","last_became_peered":"2022-06-27T11:47:52.376386+0000","last_unstale":"2022-06-27T11:47:52.384440+0000","last_undegraded":"2022-06-27T11:47:52.384440+0000","last_fullsized":"2022-06-27T11:47:52.384440+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T11:48:35.874748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.379744+0000","last_change":"2022-06-27T11:47:52.377793+0000","last_active":"2022-06-27T11:47:52.379744+0000","last_peered":"2022-06-27T11:47:52.379744+0000","last_clean":"2022-06-27T11:47:52.379744+0000","last_became_active":"2022-06-27T11:47:52.377318+0000","last_became_peered":"2022-06-27T11:47:52.377318+0000","last_unstale":"2022-06-27T11:47:52.379744+0000","last_undegraded":"2022-06-27T11:47:52.379744+0000","last_fullsized":"2022-06-27T11:47:52.379744+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T17:54:22.511301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384925+0000","last_change":"2022-06-27T11:47:52.374638+0000","last_active":"2022-06-27T11:47:52.384925+0000","last_peered":"2022-06-27T11:47:52.384925+0000","last_clean":"2022-06-27T11:47:52.384925+0000","last_became_active":"2022-06-27T11:47:52.374352+0000","last_became_peered":"2022-06-27T11:47:52.374352+0000","last_unstale":"2022-06-27T11:47:52.384925+0000","last_undegraded":"2022-06-27T11:47:52.384925+0000","last_fullsized":"2022-06-27T11:47:52.384925+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:09:59.369369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.375769+0000","last_change":"2022-06-27T11:47:52.373288+0000","last_active":"2022-06-27T11:47:52.375769+0000","last_peered":"2022-06-27T11:47:52.375769+0000","last_clean":"2022-06-27T11:47:52.375769+0000","last_became_active":"2022-06-27T11:47:52.372891+0000","last_became_peered":"2022-06-27T11:47:52.372891+0000","last_unstale":"2022-06-27T11:47:52.375769+0000","last_undegraded":"2022-06-27T11:47:52.375769+0000","last_fullsized":"2022-06-27T11:47:52.375769+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T23:16:08.088450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64756,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9016,"kb_used_data":304,"kb_used_omap":0,"kb_used_meta":8704,"kb_avail":94362824,"statfs":{"total":96636764160,"available":96627531776,"internally_reserved":0,"allocated":311296,"data_stored":670971,"data_compressed":32378,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":8912896},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39400000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.316}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40899999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28699999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42999999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46300000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8408,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":8192,"kb_avail":94363432,"statfs":{"total":96636764160,"available":96628154368,"internally_reserved":0,"allocated":212992,"data_stored":77836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8388608},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47999999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.37}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8996,"kb_used_data":276,"kb_used_omap":0,"kb_used_meta":8704,"kb_avail":94362844,"statfs":{"total":96636764160,"available":96627552256,"internally_reserved":0,"allocated":282624,"data_stored":669207,"data_compressed":32378,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":8912896},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28599999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39700000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8480,"kb_used_data":216,"kb_used_omap":0,"kb_used_meta":8256,"kb_avail":94363360,"statfs":{"total":96636764160,"available":96628080640,"internally_reserved":0,"allocated":221184,"data_stored":79405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8454144},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.377}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26200000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35499999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34699999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42999999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38900000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8412,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":8192,"kb_avail":94363428,"statfs":{"total":96636764160,"available":96628150272,"internally_reserved":0,"allocated":217088,"data_stored":78839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8388608},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.252}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46000000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41699999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]}]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8476,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":8256,"kb_avail":94363364,"statfs":{"total":96636764160,"available":96628084736,"internally_reserved":0,"allocated":217088,"data_stored":77926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8454144},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.484}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3620000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.205}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.113}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.339}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.172000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.231999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8560,"kb_used_data":232,"kb_used_omap":0,"kb_used_meta":8320,"kb_avail":94363280,"statfs":{"total":96636764160,"available":96627998720,"internally_reserved":0,"allocated":237568,"data_stored":80443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8519680},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39700000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45200000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45800000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48899999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8420,"kb_used_data":220,"kb_used_omap":0,"kb_used_meta":8192,"kb_avail":94363420,"statfs":{"total":96636764160,"available":96628142080,"internally_reserved":0,"allocated":225280,"data_stored":79475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8388608},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68100000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.702}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.8689999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78100000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46400000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32378,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32378,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-06-27T11:48:02.065 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2022-06-27T11:48:02.218 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.219+0000 7f79b372d700 1 Processor -- start 2022-06-27T11:48:02.219 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.219+0000 7f79b372d700 1 -- start start 2022-06-27T11:48:02.222 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.220+0000 7f79b372d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f79ac06fef0 0x7f79ac1570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:02.222 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.220+0000 7f79b372d700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79ac06fba0 con 0x7f79ac0c34a0 2022-06-27T11:48:02.222 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.220+0000 7f79b372d700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79ac06f5e0 con 0x7f79ac06e850 2022-06-27T11:48:02.223 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.220+0000 7f79b372d700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79ac1575e0 con 0x7f79ac06fef0 2022-06-27T11:48:02.223 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.220+0000 7f79b0cc8700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f79ac06fef0 0x7f79ac1570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:02.223 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.220+0000 7f79b1cca700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f79ac0c34a0 0x7f79ac0c2920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:47876/0 (socket says 172.21.15.195:47876) 2022-06-27T11:48:02.224 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.220+0000 7f79b1cca700 1 -- 172.21.15.195:0/3119891761 learned_addr learned my addr 172.21.15.195:0/3119891761 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:48:02.224 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.220+0000 7f79b0cc8700 1 -- 172.21.15.195:0/3119891761 >> v1:172.21.15.202:6789/0 conn(0x7f79ac06e850 legacy=0x7f79ac069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:02.224 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.220+0000 7f79b0cc8700 1 -- 172.21.15.195:0/3119891761 >> v1:172.21.15.195:6789/0 conn(0x7f79ac0c34a0 legacy=0x7f79ac0c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:02.224 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.220+0000 7f799bfff700 1 -- 172.21.15.195:0/3119891761 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2045190769 0 0) 0x7f79ac06fba0 con 0x7f79ac0c34a0 2022-06-27T11:48:02.225 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.220+0000 7f79b0cc8700 1 -- 172.21.15.195:0/3119891761 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79ac06f8c0 con 0x7f79ac06fef0 2022-06-27T11:48:02.225 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.220+0000 7f79b0cc8700 1 --2- 172.21.15.195:0/3119891761 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f79ac06fef0 0x7f79ac1570a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f799c0097a0 tx=0x7f799c009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c7c522259f8caf0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:02.225 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.220+0000 7f799bfff700 1 -- 172.21.15.195:0/3119891761 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f799c004030 con 0x7f79ac06fef0 2022-06-27T11:48:02.225 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.221+0000 7f799bfff700 1 -- 172.21.15.195:0/3119891761 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f799c004190 con 0x7f79ac06fef0 2022-06-27T11:48:02.226 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.221+0000 7f799bfff700 1 -- 172.21.15.195:0/3119891761 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f799c011630 con 0x7f79ac06fef0 2022-06-27T11:48:02.226 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.221+0000 7f79b372d700 1 -- 172.21.15.195:0/3119891761 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f79ac06fef0 msgr2=0x7f79ac1570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:02.226 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.221+0000 7f79b372d700 1 --2- 172.21.15.195:0/3119891761 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f79ac06fef0 0x7f79ac1570a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f799c0097a0 tx=0x7f799c009060 comp rx=0 tx=0).stop 2022-06-27T11:48:02.226 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.221+0000 7f79b372d700 1 -- 172.21.15.195:0/3119891761 shutdown_connections 2022-06-27T11:48:02.226 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.221+0000 7f79b372d700 1 --2- 172.21.15.195:0/3119891761 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f79ac06fef0 0x7f79ac1570a0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:02.227 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.221+0000 7f79b372d700 1 -- 172.21.15.195:0/3119891761 >> 172.21.15.195:0/3119891761 conn(0x7f79ac0c6350 msgr2=0x7f79ac0c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:02.227 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.221+0000 7f79b372d700 1 -- 172.21.15.195:0/3119891761 shutdown_connections 2022-06-27T11:48:02.227 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.221+0000 7f79b372d700 1 -- 172.21.15.195:0/3119891761 wait complete. 2022-06-27T11:48:02.227 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.222+0000 7f79b372d700 1 Processor -- start 2022-06-27T11:48:02.228 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.222+0000 7f79b372d700 1 -- start start 2022-06-27T11:48:02.231 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.223+0000 7f79b372d700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f79ac06e850 0x7f79ac0c6900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:02.231 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.223+0000 7f79b372d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f79ac0c34a0 0x7f79ac165a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:02.231 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.223+0000 7f79b372d700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f79ac156500 0x7f79ac165f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:02.231 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.223+0000 7f79b372d700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79ac165580 con 0x7f79ac0c34a0 2022-06-27T11:48:02.232 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.223+0000 7f79b372d700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79ac0b5fc0 con 0x7f79ac06e850 2022-06-27T11:48:02.232 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.223+0000 7f79b372d700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79ac162ab0 con 0x7f79ac156500 2022-06-27T11:48:02.232 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.223+0000 7f79b0cc8700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f79ac0c34a0 0x7f79ac165a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:02.232 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.223+0000 7f79b0cc8700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f79ac0c34a0 0x7f79ac165a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41692/0 (socket says 172.21.15.195:41692) 2022-06-27T11:48:02.233 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.223+0000 7f79b0cc8700 1 -- 172.21.15.195:0/3662662409 learned_addr learned my addr 172.21.15.195:0/3662662409 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:48:02.233 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.223+0000 7f79b1cca700 1 --2- 172.21.15.195:0/3662662409 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f79ac156500 0x7f79ac165f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:02.233 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.223+0000 7f79b14c9700 1 --2- 172.21.15.195:0/3662662409 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f79ac06e850 0x7f79ac0c6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:02.233 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.223+0000 7f79b0cc8700 1 -- 172.21.15.195:0/3662662409 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f79ac156500 msgr2=0x7f79ac165f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:02.233 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.223+0000 7f79b0cc8700 1 --2- 172.21.15.195:0/3662662409 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f79ac156500 0x7f79ac165f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:02.234 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.223+0000 7f79b0cc8700 1 -- 172.21.15.195:0/3662662409 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f79ac06e850 msgr2=0x7f79ac0c6900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:02.234 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.223+0000 7f79b0cc8700 1 --2- 172.21.15.195:0/3662662409 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f79ac06e850 0x7f79ac0c6900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:02.234 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.224+0000 7f79b0cc8700 1 -- 172.21.15.195:0/3662662409 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f799c0092e0 con 0x7f79ac0c34a0 2022-06-27T11:48:02.234 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.224+0000 7f79b1cca700 1 --2- 172.21.15.195:0/3662662409 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f79ac156500 0x7f79ac165f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T11:48:02.235 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.224+0000 7f79b0cc8700 1 --2- 172.21.15.195:0/3662662409 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f79ac0c34a0 0x7f79ac165a10 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f799c009ef0 tx=0x7f799c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:02.235 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.224+0000 7f79997fa700 1 -- 172.21.15.195:0/3662662409 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f799c003880 con 0x7f79ac0c34a0 2022-06-27T11:48:02.235 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.224+0000 7f79997fa700 1 -- 172.21.15.195:0/3662662409 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f799c003a00 con 0x7f79ac0c34a0 2022-06-27T11:48:02.235 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.224+0000 7f79997fa700 1 -- 172.21.15.195:0/3662662409 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f799c003dc0 con 0x7f79ac0c34a0 2022-06-27T11:48:02.236 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.224+0000 7f79b372d700 1 -- 172.21.15.195:0/3662662409 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79ac0b6260 con 0x7f79ac0c34a0 2022-06-27T11:48:02.236 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.224+0000 7f79b372d700 1 -- 172.21.15.195:0/3662662409 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79ac162d90 con 0x7f79ac0c34a0 2022-06-27T11:48:02.236 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.225+0000 7f79b372d700 1 -- 172.21.15.195:0/3662662409 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79ac1656f0 con 0x7f79ac0c34a0 2022-06-27T11:48:02.236 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.225+0000 7f79997fa700 1 -- 172.21.15.195:0/3662662409 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f799c017020 con 0x7f79ac0c34a0 2022-06-27T11:48:02.236 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.226+0000 7f79997fa700 1 --2- 172.21.15.195:0/3662662409 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f798c063d00 0x7f798c0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:02.237 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.226+0000 7f79997fa700 1 -- 172.21.15.195:0/3662662409 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f799c082290 con 0x7f79ac0c34a0 2022-06-27T11:48:02.237 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.226+0000 7f79b14c9700 1 --2- 172.21.15.195:0/3662662409 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f798c063d00 0x7f798c0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:02.237 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.226+0000 7f79b14c9700 1 --2- 172.21.15.195:0/3662662409 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f798c063d00 0x7f798c0661b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f79a800b990 tx=0x7f79a8008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:02.237 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.230+0000 7f79997fa700 1 -- 172.21.15.195:0/3662662409 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f799c050e80 con 0x7f79ac0c34a0 2022-06-27T11:48:02.406 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.406+0000 7f79b372d700 1 -- 172.21.15.195:0/3662662409 --> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f79ac1656c0 con 0x7f798c063d00 2022-06-27T11:48:02.415 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.414+0000 7f79997fa700 1 -- 172.21.15.195:0/3662662409 <== mgr.4098 v2:172.21.15.195:6832/93955 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+284879 (secure 0 0 0) 0x7f79ac1656c0 con 0x7f798c063d00 2022-06-27T11:48:02.419 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:48:02.423 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.418+0000 7f79b372d700 1 -- 172.21.15.195:0/3662662409 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f798c063d00 msgr2=0x7f798c0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:02.424 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.418+0000 7f79b372d700 1 --2- 172.21.15.195:0/3662662409 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f798c063d00 0x7f798c0661b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f79a800b990 tx=0x7f79a8008040 comp rx=0 tx=0).stop 2022-06-27T11:48:02.424 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.418+0000 7f79b372d700 1 -- 172.21.15.195:0/3662662409 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f79ac0c34a0 msgr2=0x7f79ac165a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:02.424 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.418+0000 7f79b372d700 1 --2- 172.21.15.195:0/3662662409 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f79ac0c34a0 0x7f79ac165a10 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f799c009ef0 tx=0x7f799c016000 comp rx=0 tx=0).stop 2022-06-27T11:48:02.425 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.418+0000 7f79b372d700 1 -- 172.21.15.195:0/3662662409 shutdown_connections 2022-06-27T11:48:02.425 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.418+0000 7f79b372d700 1 --2- 172.21.15.195:0/3662662409 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f798c063d00 0x7f798c0661b0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:02.425 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.418+0000 7f79b372d700 1 --2- 172.21.15.195:0/3662662409 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f79ac156500 0x7f79ac165f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:02.426 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.418+0000 7f79b372d700 1 --2- 172.21.15.195:0/3662662409 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f79ac06e850 0x7f79ac0c6900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:02.426 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.418+0000 7f79b372d700 1 --2- 172.21.15.195:0/3662662409 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f79ac0c34a0 0x7f79ac165a10 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:02.426 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.418+0000 7f79b372d700 1 -- 172.21.15.195:0/3662662409 >> 172.21.15.195:0/3662662409 conn(0x7f79ac0c6350 msgr2=0x7f79ac157720 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:02.427 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.418+0000 7f79b372d700 1 -- 172.21.15.195:0/3662662409 shutdown_connections 2022-06-27T11:48:02.427 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.418+0000 7f79b372d700 1 -- 172.21.15.195:0/3662662409 wait complete. 2022-06-27T11:48:02.427 INFO:teuthology.orchestra.run.smithi195.stderr:dumped all 2022-06-27T11:48:02.429 INFO:teuthology.orchestra.run.smithi195.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2022-06-27T11:48:02.341786+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592698,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":210,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":68768,"kb_used_data":1880,"kb_used_omap":0,"kb_used_meta":66816,"kb_avail":754905952,"statfs":{"total":773094113280,"available":773023694848,"internally_reserved":0,"allocated":1925120,"data_stored":1814102,"data_compressed":64756,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":68419584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":58,"num_write_kb":276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.981940"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384962+0000","last_change":"2022-06-27T11:47:52.381823+0000","last_active":"2022-06-27T11:47:52.384962+0000","last_peered":"2022-06-27T11:47:52.384962+0000","last_clean":"2022-06-27T11:47:52.384962+0000","last_became_active":"2022-06-27T11:47:52.381551+0000","last_became_peered":"2022-06-27T11:47:52.381551+0000","last_unstale":"2022-06-27T11:47:52.384962+0000","last_undegraded":"2022-06-27T11:47:52.384962+0000","last_fullsized":"2022-06-27T11:47:52.384962+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:17:34.164061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.378135+0000","last_change":"2022-06-27T11:47:52.374939+0000","last_active":"2022-06-27T11:47:52.378135+0000","last_peered":"2022-06-27T11:47:52.378135+0000","last_clean":"2022-06-27T11:47:52.378135+0000","last_became_active":"2022-06-27T11:47:52.374620+0000","last_became_peered":"2022-06-27T11:47:52.374620+0000","last_unstale":"2022-06-27T11:47:52.378135+0000","last_undegraded":"2022-06-27T11:47:52.378135+0000","last_fullsized":"2022-06-27T11:47:52.378135+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:56:27.254098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384127+0000","last_change":"2022-06-27T11:47:52.380802+0000","last_active":"2022-06-27T11:47:52.384127+0000","last_peered":"2022-06-27T11:47:52.384127+0000","last_clean":"2022-06-27T11:47:52.384127+0000","last_became_active":"2022-06-27T11:47:52.380500+0000","last_became_peered":"2022-06-27T11:47:52.380500+0000","last_unstale":"2022-06-27T11:47:52.384127+0000","last_undegraded":"2022-06-27T11:47:52.384127+0000","last_fullsized":"2022-06-27T11:47:52.384127+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:06:42.889598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384114+0000","last_change":"2022-06-27T11:47:52.380943+0000","last_active":"2022-06-27T11:47:52.384114+0000","last_peered":"2022-06-27T11:47:52.384114+0000","last_clean":"2022-06-27T11:47:52.384114+0000","last_became_active":"2022-06-27T11:47:52.380648+0000","last_became_peered":"2022-06-27T11:47:52.380648+0000","last_unstale":"2022-06-27T11:47:52.384114+0000","last_undegraded":"2022-06-27T11:47:52.384114+0000","last_fullsized":"2022-06-27T11:47:52.384114+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:14:19.729605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.386875+0000","last_change":"2022-06-27T11:47:52.386144+0000","last_active":"2022-06-27T11:47:52.386875+0000","last_peered":"2022-06-27T11:47:52.386875+0000","last_clean":"2022-06-27T11:47:52.386875+0000","last_became_active":"2022-06-27T11:47:52.385586+0000","last_became_peered":"2022-06-27T11:47:52.385586+0000","last_unstale":"2022-06-27T11:47:52.386875+0000","last_undegraded":"2022-06-27T11:47:52.386875+0000","last_fullsized":"2022-06-27T11:47:52.386875+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T14:30:03.660749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.387182+0000","last_change":"2022-06-27T11:47:52.381906+0000","last_active":"2022-06-27T11:47:52.387182+0000","last_peered":"2022-06-27T11:47:52.387182+0000","last_clean":"2022-06-27T11:47:52.387182+0000","last_became_active":"2022-06-27T11:47:52.381279+0000","last_became_peered":"2022-06-27T11:47:52.381279+0000","last_unstale":"2022-06-27T11:47:52.387182+0000","last_undegraded":"2022-06-27T11:47:52.387182+0000","last_fullsized":"2022-06-27T11:47:52.387182+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T18:41:03.991762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.383137+0000","last_change":"2022-06-27T11:47:52.377128+0000","last_active":"2022-06-27T11:47:52.383137+0000","last_peered":"2022-06-27T11:47:52.383137+0000","last_clean":"2022-06-27T11:47:52.383137+0000","last_became_active":"2022-06-27T11:47:52.376499+0000","last_became_peered":"2022-06-27T11:47:52.376499+0000","last_unstale":"2022-06-27T11:47:52.383137+0000","last_undegraded":"2022-06-27T11:47:52.383137+0000","last_fullsized":"2022-06-27T11:47:52.383137+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:00:13.208145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.378171+0000","last_change":"2022-06-27T11:47:52.374169+0000","last_active":"2022-06-27T11:47:52.378171+0000","last_peered":"2022-06-27T11:47:52.378171+0000","last_clean":"2022-06-27T11:47:52.378171+0000","last_became_active":"2022-06-27T11:47:52.373917+0000","last_became_peered":"2022-06-27T11:47:52.373917+0000","last_unstale":"2022-06-27T11:47:52.378171+0000","last_undegraded":"2022-06-27T11:47:52.378171+0000","last_fullsized":"2022-06-27T11:47:52.378171+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:43:52.741254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384769+0000","last_change":"2022-06-27T11:47:52.382039+0000","last_active":"2022-06-27T11:47:52.384769+0000","last_peered":"2022-06-27T11:47:52.384769+0000","last_clean":"2022-06-27T11:47:52.384769+0000","last_became_active":"2022-06-27T11:47:52.381202+0000","last_became_peered":"2022-06-27T11:47:52.381202+0000","last_unstale":"2022-06-27T11:47:52.384769+0000","last_undegraded":"2022-06-27T11:47:52.384769+0000","last_fullsized":"2022-06-27T11:47:52.384769+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:13:02.553823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.383894+0000","last_change":"2022-06-27T11:47:52.380839+0000","last_active":"2022-06-27T11:47:52.383894+0000","last_peered":"2022-06-27T11:47:52.383894+0000","last_clean":"2022-06-27T11:47:52.383894+0000","last_became_active":"2022-06-27T11:47:52.380527+0000","last_became_peered":"2022-06-27T11:47:52.380527+0000","last_unstale":"2022-06-27T11:47:52.383894+0000","last_undegraded":"2022-06-27T11:47:52.383894+0000","last_fullsized":"2022-06-27T11:47:52.383894+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T23:18:07.086598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.374783+0000","last_change":"2022-06-27T11:47:52.371267+0000","last_active":"2022-06-27T11:47:52.374783+0000","last_peered":"2022-06-27T11:47:52.374783+0000","last_clean":"2022-06-27T11:47:52.374783+0000","last_became_active":"2022-06-27T11:47:52.370921+0000","last_became_peered":"2022-06-27T11:47:52.370921+0000","last_unstale":"2022-06-27T11:47:52.374783+0000","last_undegraded":"2022-06-27T11:47:52.374783+0000","last_fullsized":"2022-06-27T11:47:52.374783+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T19:22:39.648715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.378994+0000","last_change":"2022-06-27T11:47:52.375381+0000","last_active":"2022-06-27T11:47:52.378994+0000","last_peered":"2022-06-27T11:47:52.378994+0000","last_clean":"2022-06-27T11:47:52.378994+0000","last_became_active":"2022-06-27T11:47:52.375089+0000","last_became_peered":"2022-06-27T11:47:52.375089+0000","last_unstale":"2022-06-27T11:47:52.378994+0000","last_undegraded":"2022-06-27T11:47:52.378994+0000","last_fullsized":"2022-06-27T11:47:52.378994+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:43:14.830424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384993+0000","last_change":"2022-06-27T11:47:52.381690+0000","last_active":"2022-06-27T11:47:52.384993+0000","last_peered":"2022-06-27T11:47:52.384993+0000","last_clean":"2022-06-27T11:47:52.384993+0000","last_became_active":"2022-06-27T11:47:52.381472+0000","last_became_peered":"2022-06-27T11:47:52.381472+0000","last_unstale":"2022-06-27T11:47:52.384993+0000","last_undegraded":"2022-06-27T11:47:52.384993+0000","last_fullsized":"2022-06-27T11:47:52.384993+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:14:56.919536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.378838+0000","last_change":"2022-06-27T11:47:52.376063+0000","last_active":"2022-06-27T11:47:52.378838+0000","last_peered":"2022-06-27T11:47:52.378838+0000","last_clean":"2022-06-27T11:47:52.378838+0000","last_became_active":"2022-06-27T11:47:52.375778+0000","last_became_peered":"2022-06-27T11:47:52.375778+0000","last_unstale":"2022-06-27T11:47:52.378838+0000","last_undegraded":"2022-06-27T11:47:52.378838+0000","last_fullsized":"2022-06-27T11:47:52.378838+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T23:16:52.221967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.383160+0000","last_change":"2022-06-27T11:47:52.377297+0000","last_active":"2022-06-27T11:47:52.383160+0000","last_peered":"2022-06-27T11:47:52.383160+0000","last_clean":"2022-06-27T11:47:52.383160+0000","last_became_active":"2022-06-27T11:47:52.376855+0000","last_became_peered":"2022-06-27T11:47:52.376855+0000","last_unstale":"2022-06-27T11:47:52.383160+0000","last_undegraded":"2022-06-27T11:47:52.383160+0000","last_fullsized":"2022-06-27T11:47:52.383160+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:33:39.990522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.387168+0000","last_change":"2022-06-27T11:47:52.381821+0000","last_active":"2022-06-27T11:47:52.387168+0000","last_peered":"2022-06-27T11:47:52.387168+0000","last_clean":"2022-06-27T11:47:52.387168+0000","last_became_active":"2022-06-27T11:47:52.381548+0000","last_became_peered":"2022-06-27T11:47:52.381548+0000","last_unstale":"2022-06-27T11:47:52.387168+0000","last_undegraded":"2022-06-27T11:47:52.387168+0000","last_fullsized":"2022-06-27T11:47:52.387168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T23:14:53.320016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384753+0000","last_change":"2022-06-27T11:47:52.381811+0000","last_active":"2022-06-27T11:47:52.384753+0000","last_peered":"2022-06-27T11:47:52.384753+0000","last_clean":"2022-06-27T11:47:52.384753+0000","last_became_active":"2022-06-27T11:47:52.381554+0000","last_became_peered":"2022-06-27T11:47:52.381554+0000","last_unstale":"2022-06-27T11:47:52.384753+0000","last_undegraded":"2022-06-27T11:47:52.384753+0000","last_fullsized":"2022-06-27T11:47:52.384753+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T14:19:38.519380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.387218+0000","last_change":"2022-06-27T11:47:52.382952+0000","last_active":"2022-06-27T11:47:52.387218+0000","last_peered":"2022-06-27T11:47:52.387218+0000","last_clean":"2022-06-27T11:47:52.387218+0000","last_became_active":"2022-06-27T11:47:52.382691+0000","last_became_peered":"2022-06-27T11:47:52.382691+0000","last_unstale":"2022-06-27T11:47:52.387218+0000","last_undegraded":"2022-06-27T11:47:52.387218+0000","last_fullsized":"2022-06-27T11:47:52.387218+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:28:54.743970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.383682+0000","last_change":"2022-06-27T11:47:52.379083+0000","last_active":"2022-06-27T11:47:52.383682+0000","last_peered":"2022-06-27T11:47:52.383682+0000","last_clean":"2022-06-27T11:47:52.383682+0000","last_became_active":"2022-06-27T11:47:52.378724+0000","last_became_peered":"2022-06-27T11:47:52.378724+0000","last_unstale":"2022-06-27T11:47:52.383682+0000","last_undegraded":"2022-06-27T11:47:52.383682+0000","last_fullsized":"2022-06-27T11:47:52.383682+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T21:25:07.786935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.383275+0000","last_change":"2022-06-27T11:47:52.377947+0000","last_active":"2022-06-27T11:47:52.383275+0000","last_peered":"2022-06-27T11:47:52.383275+0000","last_clean":"2022-06-27T11:47:52.383275+0000","last_became_active":"2022-06-27T11:47:52.377652+0000","last_became_peered":"2022-06-27T11:47:52.377652+0000","last_unstale":"2022-06-27T11:47:52.383275+0000","last_undegraded":"2022-06-27T11:47:52.383275+0000","last_fullsized":"2022-06-27T11:47:52.383275+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T20:48:28.642052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384444+0000","last_change":"2022-06-27T11:47:52.381920+0000","last_active":"2022-06-27T11:47:52.384444+0000","last_peered":"2022-06-27T11:47:52.384444+0000","last_clean":"2022-06-27T11:47:52.384444+0000","last_became_active":"2022-06-27T11:47:52.381477+0000","last_became_peered":"2022-06-27T11:47:52.381477+0000","last_unstale":"2022-06-27T11:47:52.384444+0000","last_undegraded":"2022-06-27T11:47:52.384444+0000","last_fullsized":"2022-06-27T11:47:52.384444+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T20:42:25.529336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.386345+0000","last_change":"2022-06-27T11:47:52.385231+0000","last_active":"2022-06-27T11:47:52.386345+0000","last_peered":"2022-06-27T11:47:52.386345+0000","last_clean":"2022-06-27T11:47:52.386345+0000","last_became_active":"2022-06-27T11:47:52.384972+0000","last_became_peered":"2022-06-27T11:47:52.384972+0000","last_unstale":"2022-06-27T11:47:52.386345+0000","last_undegraded":"2022-06-27T11:47:52.386345+0000","last_fullsized":"2022-06-27T11:47:52.386345+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T17:37:26.084788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.385330+0000","last_change":"2022-06-27T11:47:52.381358+0000","last_active":"2022-06-27T11:47:52.385330+0000","last_peered":"2022-06-27T11:47:52.385330+0000","last_clean":"2022-06-27T11:47:52.385330+0000","last_became_active":"2022-06-27T11:47:52.380997+0000","last_became_peered":"2022-06-27T11:47:52.380997+0000","last_unstale":"2022-06-27T11:47:52.385330+0000","last_undegraded":"2022-06-27T11:47:52.385330+0000","last_fullsized":"2022-06-27T11:47:52.385330+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T21:44:16.061234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384432+0000","last_change":"2022-06-27T11:47:52.381899+0000","last_active":"2022-06-27T11:47:52.384432+0000","last_peered":"2022-06-27T11:47:52.384432+0000","last_clean":"2022-06-27T11:47:52.384432+0000","last_became_active":"2022-06-27T11:47:52.381645+0000","last_became_peered":"2022-06-27T11:47:52.381645+0000","last_unstale":"2022-06-27T11:47:52.384432+0000","last_undegraded":"2022-06-27T11:47:52.384432+0000","last_fullsized":"2022-06-27T11:47:52.384432+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:05:59.742023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.385053+0000","last_change":"2022-06-27T11:47:52.378736+0000","last_active":"2022-06-27T11:47:52.385053+0000","last_peered":"2022-06-27T11:47:52.385053+0000","last_clean":"2022-06-27T11:47:52.385053+0000","last_became_active":"2022-06-27T11:47:52.378475+0000","last_became_peered":"2022-06-27T11:47:52.378475+0000","last_unstale":"2022-06-27T11:47:52.385053+0000","last_undegraded":"2022-06-27T11:47:52.385053+0000","last_fullsized":"2022-06-27T11:47:52.385053+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T12:42:12.528716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384922+0000","last_change":"2022-06-27T11:47:52.378044+0000","last_active":"2022-06-27T11:47:52.384922+0000","last_peered":"2022-06-27T11:47:52.384922+0000","last_clean":"2022-06-27T11:47:52.384922+0000","last_became_active":"2022-06-27T11:47:52.377791+0000","last_became_peered":"2022-06-27T11:47:52.377791+0000","last_unstale":"2022-06-27T11:47:52.384922+0000","last_undegraded":"2022-06-27T11:47:52.384922+0000","last_fullsized":"2022-06-27T11:47:52.384922+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:21:45.615162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384115+0000","last_change":"2022-06-27T11:47:52.382267+0000","last_active":"2022-06-27T11:47:52.384115+0000","last_peered":"2022-06-27T11:47:52.384115+0000","last_clean":"2022-06-27T11:47:52.384115+0000","last_became_active":"2022-06-27T11:47:52.381985+0000","last_became_peered":"2022-06-27T11:47:52.381985+0000","last_unstale":"2022-06-27T11:47:52.384115+0000","last_undegraded":"2022-06-27T11:47:52.384115+0000","last_fullsized":"2022-06-27T11:47:52.384115+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:34:09.509759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:53.909338+0000","last_change":"2022-06-27T11:47:51.982183+0000","last_active":"2022-06-27T11:47:53.909338+0000","last_peered":"2022-06-27T11:47:53.909338+0000","last_clean":"2022-06-27T11:47:53.909338+0000","last_became_active":"2022-06-27T11:47:51.981339+0000","last_became_peered":"2022-06-27T11:47:51.981339+0000","last_unstale":"2022-06-27T11:47:53.909338+0000","last_undegraded":"2022-06-27T11:47:53.909338+0000","last_fullsized":"2022-06-27T11:47:53.909338+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:08:09.223947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384452+0000","last_change":"2022-06-27T11:47:52.377197+0000","last_active":"2022-06-27T11:47:52.384452+0000","last_peered":"2022-06-27T11:47:52.384452+0000","last_clean":"2022-06-27T11:47:52.384452+0000","last_became_active":"2022-06-27T11:47:52.376925+0000","last_became_peered":"2022-06-27T11:47:52.376925+0000","last_unstale":"2022-06-27T11:47:52.384452+0000","last_undegraded":"2022-06-27T11:47:52.384452+0000","last_fullsized":"2022-06-27T11:47:52.384452+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:39:27.103978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.364221+0000","last_change":"2022-06-27T11:47:51.981074+0000","last_active":"2022-06-27T11:47:52.364221+0000","last_peered":"2022-06-27T11:47:52.364221+0000","last_clean":"2022-06-27T11:47:52.364221+0000","last_became_active":"2022-06-27T11:47:51.980636+0000","last_became_peered":"2022-06-27T11:47:51.980636+0000","last_unstale":"2022-06-27T11:47:52.364221+0000","last_undegraded":"2022-06-27T11:47:52.364221+0000","last_fullsized":"2022-06-27T11:47:52.364221+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T23:32:52.850801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384380+0000","last_change":"2022-06-27T11:47:52.378415+0000","last_active":"2022-06-27T11:47:52.384380+0000","last_peered":"2022-06-27T11:47:52.384380+0000","last_clean":"2022-06-27T11:47:52.384380+0000","last_became_active":"2022-06-27T11:47:52.378123+0000","last_became_peered":"2022-06-27T11:47:52.378123+0000","last_unstale":"2022-06-27T11:47:52.384380+0000","last_undegraded":"2022-06-27T11:47:52.384380+0000","last_fullsized":"2022-06-27T11:47:52.384380+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T11:54:37.127828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.369137+0000","last_change":"2022-06-27T11:47:51.975684+0000","last_active":"2022-06-27T11:47:52.369137+0000","last_peered":"2022-06-27T11:47:52.369137+0000","last_clean":"2022-06-27T11:47:52.369137+0000","last_became_active":"2022-06-27T11:47:51.974902+0000","last_became_peered":"2022-06-27T11:47:51.974902+0000","last_unstale":"2022-06-27T11:47:52.369137+0000","last_undegraded":"2022-06-27T11:47:52.369137+0000","last_fullsized":"2022-06-27T11:47:52.369137+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:51:45.780798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.378008+0000","last_change":"2022-06-27T11:47:52.376089+0000","last_active":"2022-06-27T11:47:52.378008+0000","last_peered":"2022-06-27T11:47:52.378008+0000","last_clean":"2022-06-27T11:47:52.378008+0000","last_became_active":"2022-06-27T11:47:52.375780+0000","last_became_peered":"2022-06-27T11:47:52.375780+0000","last_unstale":"2022-06-27T11:47:52.378008+0000","last_undegraded":"2022-06-27T11:47:52.378008+0000","last_fullsized":"2022-06-27T11:47:52.378008+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T14:17:53.942880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.371313+0000","last_change":"2022-06-27T11:47:51.974071+0000","last_active":"2022-06-27T11:47:52.371313+0000","last_peered":"2022-06-27T11:47:52.371313+0000","last_clean":"2022-06-27T11:47:52.371313+0000","last_became_active":"2022-06-27T11:47:51.973682+0000","last_became_peered":"2022-06-27T11:47:51.973682+0000","last_unstale":"2022-06-27T11:47:52.371313+0000","last_undegraded":"2022-06-27T11:47:52.371313+0000","last_fullsized":"2022-06-27T11:47:52.371313+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T17:40:03.705083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384841+0000","last_change":"2022-06-27T11:47:52.377716+0000","last_active":"2022-06-27T11:47:52.384841+0000","last_peered":"2022-06-27T11:47:52.384841+0000","last_clean":"2022-06-27T11:47:52.384841+0000","last_became_active":"2022-06-27T11:47:52.377464+0000","last_became_peered":"2022-06-27T11:47:52.377464+0000","last_unstale":"2022-06-27T11:47:52.384841+0000","last_undegraded":"2022-06-27T11:47:52.384841+0000","last_fullsized":"2022-06-27T11:47:52.384841+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:39:36.234204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.368885+0000","last_change":"2022-06-27T11:47:51.975895+0000","last_active":"2022-06-27T11:47:52.368885+0000","last_peered":"2022-06-27T11:47:52.368885+0000","last_clean":"2022-06-27T11:47:52.368885+0000","last_became_active":"2022-06-27T11:47:51.975601+0000","last_became_peered":"2022-06-27T11:47:51.975601+0000","last_unstale":"2022-06-27T11:47:52.368885+0000","last_undegraded":"2022-06-27T11:47:52.368885+0000","last_fullsized":"2022-06-27T11:47:52.368885+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:43:55.833399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.385349+0000","last_change":"2022-06-27T11:47:52.383063+0000","last_active":"2022-06-27T11:47:52.385349+0000","last_peered":"2022-06-27T11:47:52.385349+0000","last_clean":"2022-06-27T11:47:52.385349+0000","last_became_active":"2022-06-27T11:47:52.382390+0000","last_became_peered":"2022-06-27T11:47:52.382390+0000","last_unstale":"2022-06-27T11:47:52.385349+0000","last_undegraded":"2022-06-27T11:47:52.385349+0000","last_fullsized":"2022-06-27T11:47:52.385349+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:40:10.929542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"21'3","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:57.367279+0000","last_change":"2022-06-27T11:47:51.980627+0000","last_active":"2022-06-27T11:47:57.367279+0000","last_peered":"2022-06-27T11:47:57.367279+0000","last_clean":"2022-06-27T11:47:57.367279+0000","last_became_active":"2022-06-27T11:47:51.980265+0000","last_became_peered":"2022-06-27T11:47:51.980265+0000","last_unstale":"2022-06-27T11:47:57.367279+0000","last_undegraded":"2022-06-27T11:47:57.367279+0000","last_fullsized":"2022-06-27T11:47:57.367279+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T14:18:32.599085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384584+0000","last_change":"2022-06-27T11:47:52.377475+0000","last_active":"2022-06-27T11:47:52.384584+0000","last_peered":"2022-06-27T11:47:52.384584+0000","last_clean":"2022-06-27T11:47:52.384584+0000","last_became_active":"2022-06-27T11:47:52.377220+0000","last_became_peered":"2022-06-27T11:47:52.377220+0000","last_unstale":"2022-06-27T11:47:52.384584+0000","last_undegraded":"2022-06-27T11:47:52.384584+0000","last_fullsized":"2022-06-27T11:47:52.384584+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T23:27:26.525688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"21'6","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:57.368255+0000","last_change":"2022-06-27T11:47:51.978195+0000","last_active":"2022-06-27T11:47:57.368255+0000","last_peered":"2022-06-27T11:47:57.368255+0000","last_clean":"2022-06-27T11:47:57.368255+0000","last_became_active":"2022-06-27T11:47:51.974673+0000","last_became_peered":"2022-06-27T11:47:51.974673+0000","last_unstale":"2022-06-27T11:47:57.368255+0000","last_undegraded":"2022-06-27T11:47:57.368255+0000","last_fullsized":"2022-06-27T11:47:57.368255+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T18:09:43.518644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.382204+0000","last_change":"2022-06-27T11:47:52.378740+0000","last_active":"2022-06-27T11:47:52.382204+0000","last_peered":"2022-06-27T11:47:52.382204+0000","last_clean":"2022-06-27T11:47:52.382204+0000","last_became_active":"2022-06-27T11:47:52.378451+0000","last_became_peered":"2022-06-27T11:47:52.378451+0000","last_unstale":"2022-06-27T11:47:52.382204+0000","last_undegraded":"2022-06-27T11:47:52.382204+0000","last_fullsized":"2022-06-27T11:47:52.382204+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:04:47.829409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.13","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:57.368186+0000","last_change":"2022-06-27T11:47:51.983039+0000","last_active":"2022-06-27T11:47:57.368186+0000","last_peered":"2022-06-27T11:47:57.368186+0000","last_clean":"2022-06-27T11:47:57.368186+0000","last_became_active":"2022-06-27T11:47:51.982683+0000","last_became_peered":"2022-06-27T11:47:51.982683+0000","last_unstale":"2022-06-27T11:47:57.368186+0000","last_undegraded":"2022-06-27T11:47:57.368186+0000","last_fullsized":"2022-06-27T11:47:57.368186+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T17:10:16.876753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384742+0000","last_change":"2022-06-27T11:47:52.374884+0000","last_active":"2022-06-27T11:47:52.384742+0000","last_peered":"2022-06-27T11:47:52.384742+0000","last_clean":"2022-06-27T11:47:52.384742+0000","last_became_active":"2022-06-27T11:47:52.374632+0000","last_became_peered":"2022-06-27T11:47:52.374632+0000","last_unstale":"2022-06-27T11:47:52.384742+0000","last_undegraded":"2022-06-27T11:47:52.384742+0000","last_fullsized":"2022-06-27T11:47:52.384742+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T21:33:54.724089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.374167+0000","last_change":"2022-06-27T11:47:51.976772+0000","last_active":"2022-06-27T11:47:52.374167+0000","last_peered":"2022-06-27T11:47:52.374167+0000","last_clean":"2022-06-27T11:47:52.374167+0000","last_became_active":"2022-06-27T11:47:51.976475+0000","last_became_peered":"2022-06-27T11:47:51.976475+0000","last_unstale":"2022-06-27T11:47:52.374167+0000","last_undegraded":"2022-06-27T11:47:52.374167+0000","last_fullsized":"2022-06-27T11:47:52.374167+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T23:39:10.919159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.377943+0000","last_change":"2022-06-27T11:47:52.373748+0000","last_active":"2022-06-27T11:47:52.377943+0000","last_peered":"2022-06-27T11:47:52.377943+0000","last_clean":"2022-06-27T11:47:52.377943+0000","last_became_active":"2022-06-27T11:47:52.373459+0000","last_became_peered":"2022-06-27T11:47:52.373459+0000","last_unstale":"2022-06-27T11:47:52.377943+0000","last_undegraded":"2022-06-27T11:47:52.377943+0000","last_fullsized":"2022-06-27T11:47:52.377943+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T18:27:22.254416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.367083+0000","last_change":"2022-06-27T11:47:51.982926+0000","last_active":"2022-06-27T11:47:52.367083+0000","last_peered":"2022-06-27T11:47:52.367083+0000","last_clean":"2022-06-27T11:47:52.367083+0000","last_became_active":"2022-06-27T11:47:51.982638+0000","last_became_peered":"2022-06-27T11:47:51.982638+0000","last_unstale":"2022-06-27T11:47:52.367083+0000","last_undegraded":"2022-06-27T11:47:52.367083+0000","last_fullsized":"2022-06-27T11:47:52.367083+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:34:04.834822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384791+0000","last_change":"2022-06-27T11:47:52.381623+0000","last_active":"2022-06-27T11:47:52.384791+0000","last_peered":"2022-06-27T11:47:52.384791+0000","last_clean":"2022-06-27T11:47:52.384791+0000","last_became_active":"2022-06-27T11:47:52.381258+0000","last_became_peered":"2022-06-27T11:47:52.381258+0000","last_unstale":"2022-06-27T11:47:52.384791+0000","last_undegraded":"2022-06-27T11:47:52.384791+0000","last_fullsized":"2022-06-27T11:47:52.384791+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:39:07.735224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"21'8","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:57.369384+0000","last_change":"2022-06-27T11:47:51.973125+0000","last_active":"2022-06-27T11:47:57.369384+0000","last_peered":"2022-06-27T11:47:57.369384+0000","last_clean":"2022-06-27T11:47:57.369384+0000","last_became_active":"2022-06-27T11:47:51.972806+0000","last_became_peered":"2022-06-27T11:47:51.972806+0000","last_unstale":"2022-06-27T11:47:57.369384+0000","last_undegraded":"2022-06-27T11:47:57.369384+0000","last_fullsized":"2022-06-27T11:47:57.369384+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:49:23.881991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384422+0000","last_change":"2022-06-27T11:47:52.376101+0000","last_active":"2022-06-27T11:47:52.384422+0000","last_peered":"2022-06-27T11:47:52.384422+0000","last_clean":"2022-06-27T11:47:52.384422+0000","last_became_active":"2022-06-27T11:47:52.375774+0000","last_became_peered":"2022-06-27T11:47:52.375774+0000","last_unstale":"2022-06-27T11:47:52.384422+0000","last_undegraded":"2022-06-27T11:47:52.384422+0000","last_fullsized":"2022-06-27T11:47:52.384422+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T18:21:13.049096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.f","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:53.492124+0000","last_change":"2022-06-27T11:47:51.981981+0000","last_active":"2022-06-27T11:47:53.492124+0000","last_peered":"2022-06-27T11:47:53.492124+0000","last_clean":"2022-06-27T11:47:53.492124+0000","last_became_active":"2022-06-27T11:47:51.981749+0000","last_became_peered":"2022-06-27T11:47:51.981749+0000","last_unstale":"2022-06-27T11:47:53.492124+0000","last_undegraded":"2022-06-27T11:47:53.492124+0000","last_fullsized":"2022-06-27T11:47:53.492124+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:28:52.457719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.378190+0000","last_change":"2022-06-27T11:47:52.374223+0000","last_active":"2022-06-27T11:47:52.378190+0000","last_peered":"2022-06-27T11:47:52.378190+0000","last_clean":"2022-06-27T11:47:52.378190+0000","last_became_active":"2022-06-27T11:47:52.373932+0000","last_became_peered":"2022-06-27T11:47:52.373932+0000","last_unstale":"2022-06-27T11:47:52.378190+0000","last_undegraded":"2022-06-27T11:47:52.378190+0000","last_fullsized":"2022-06-27T11:47:52.378190+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:48:33.254458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:57.368190+0000","last_change":"2022-06-27T11:47:51.982895+0000","last_active":"2022-06-27T11:47:57.368190+0000","last_peered":"2022-06-27T11:47:57.368190+0000","last_clean":"2022-06-27T11:47:57.368190+0000","last_became_active":"2022-06-27T11:47:51.982619+0000","last_became_peered":"2022-06-27T11:47:51.982619+0000","last_unstale":"2022-06-27T11:47:57.368190+0000","last_undegraded":"2022-06-27T11:47:57.368190+0000","last_fullsized":"2022-06-27T11:47:57.368190+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:22:01.241112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.377169+0000","last_change":"2022-06-27T11:47:52.375225+0000","last_active":"2022-06-27T11:47:52.377169+0000","last_peered":"2022-06-27T11:47:52.377169+0000","last_clean":"2022-06-27T11:47:52.377169+0000","last_became_active":"2022-06-27T11:47:52.374925+0000","last_became_peered":"2022-06-27T11:47:52.374925+0000","last_unstale":"2022-06-27T11:47:52.377169+0000","last_undegraded":"2022-06-27T11:47:52.377169+0000","last_fullsized":"2022-06-27T11:47:52.377169+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:54:31.945902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:53.492085+0000","last_change":"2022-06-27T11:47:51.975202+0000","last_active":"2022-06-27T11:47:53.492085+0000","last_peered":"2022-06-27T11:47:53.492085+0000","last_clean":"2022-06-27T11:47:53.492085+0000","last_became_active":"2022-06-27T11:47:51.974968+0000","last_became_peered":"2022-06-27T11:47:51.974968+0000","last_unstale":"2022-06-27T11:47:53.492085+0000","last_undegraded":"2022-06-27T11:47:53.492085+0000","last_fullsized":"2022-06-27T11:47:53.492085+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T21:16:30.362248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.378431+0000","last_change":"2022-06-27T11:47:52.377794+0000","last_active":"2022-06-27T11:47:52.378431+0000","last_peered":"2022-06-27T11:47:52.378431+0000","last_clean":"2022-06-27T11:47:52.378431+0000","last_became_active":"2022-06-27T11:47:52.377387+0000","last_became_peered":"2022-06-27T11:47:52.377387+0000","last_unstale":"2022-06-27T11:47:52.378431+0000","last_undegraded":"2022-06-27T11:47:52.378431+0000","last_fullsized":"2022-06-27T11:47:52.378431+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T17:00:00.924967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.368792+0000","last_change":"2022-06-27T11:47:51.976575+0000","last_active":"2022-06-27T11:47:52.368792+0000","last_peered":"2022-06-27T11:47:52.368792+0000","last_clean":"2022-06-27T11:47:52.368792+0000","last_became_active":"2022-06-27T11:47:51.976300+0000","last_became_peered":"2022-06-27T11:47:51.976300+0000","last_unstale":"2022-06-27T11:47:52.368792+0000","last_undegraded":"2022-06-27T11:47:52.368792+0000","last_fullsized":"2022-06-27T11:47:52.368792+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:08:46.067375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.387174+0000","last_change":"2022-06-27T11:47:52.386298+0000","last_active":"2022-06-27T11:47:52.387174+0000","last_peered":"2022-06-27T11:47:52.387174+0000","last_clean":"2022-06-27T11:47:52.387174+0000","last_became_active":"2022-06-27T11:47:52.385918+0000","last_became_peered":"2022-06-27T11:47:52.385918+0000","last_unstale":"2022-06-27T11:47:52.387174+0000","last_undegraded":"2022-06-27T11:47:52.387174+0000","last_fullsized":"2022-06-27T11:47:52.387174+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:21:51.780895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:57.364776+0000","last_change":"2022-06-27T11:47:51.973817+0000","last_active":"2022-06-27T11:47:57.364776+0000","last_peered":"2022-06-27T11:47:57.364776+0000","last_clean":"2022-06-27T11:47:57.364776+0000","last_became_active":"2022-06-27T11:47:51.973515+0000","last_became_peered":"2022-06-27T11:47:51.973515+0000","last_unstale":"2022-06-27T11:47:57.364776+0000","last_undegraded":"2022-06-27T11:47:57.364776+0000","last_fullsized":"2022-06-27T11:47:57.364776+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T21:01:45.170590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.374582+0000","last_change":"2022-06-27T11:47:52.371141+0000","last_active":"2022-06-27T11:47:52.374582+0000","last_peered":"2022-06-27T11:47:52.374582+0000","last_clean":"2022-06-27T11:47:52.374582+0000","last_became_active":"2022-06-27T11:47:52.370796+0000","last_became_peered":"2022-06-27T11:47:52.370796+0000","last_unstale":"2022-06-27T11:47:52.374582+0000","last_undegraded":"2022-06-27T11:47:52.374582+0000","last_fullsized":"2022-06-27T11:47:52.374582+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T23:16:39.266615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.369465+0000","last_change":"2022-06-27T11:47:51.983071+0000","last_active":"2022-06-27T11:47:52.369465+0000","last_peered":"2022-06-27T11:47:52.369465+0000","last_clean":"2022-06-27T11:47:52.369465+0000","last_became_active":"2022-06-27T11:47:51.982756+0000","last_became_peered":"2022-06-27T11:47:51.982756+0000","last_unstale":"2022-06-27T11:47:52.369465+0000","last_undegraded":"2022-06-27T11:47:52.369465+0000","last_fullsized":"2022-06-27T11:47:52.369465+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T17:14:00.737599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.387222+0000","last_change":"2022-06-27T11:47:52.382943+0000","last_active":"2022-06-27T11:47:52.387222+0000","last_peered":"2022-06-27T11:47:52.387222+0000","last_clean":"2022-06-27T11:47:52.387222+0000","last_became_active":"2022-06-27T11:47:52.382625+0000","last_became_peered":"2022-06-27T11:47:52.382625+0000","last_unstale":"2022-06-27T11:47:52.387222+0000","last_undegraded":"2022-06-27T11:47:52.387222+0000","last_fullsized":"2022-06-27T11:47:52.387222+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:01:18.570554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.362681+0000","last_change":"2022-06-27T11:47:51.978856+0000","last_active":"2022-06-27T11:47:52.362681+0000","last_peered":"2022-06-27T11:47:52.362681+0000","last_clean":"2022-06-27T11:47:52.362681+0000","last_became_active":"2022-06-27T11:47:51.978541+0000","last_became_peered":"2022-06-27T11:47:51.978541+0000","last_unstale":"2022-06-27T11:47:52.362681+0000","last_undegraded":"2022-06-27T11:47:52.362681+0000","last_fullsized":"2022-06-27T11:47:52.362681+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:36:24.190996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.378203+0000","last_change":"2022-06-27T11:47:52.376925+0000","last_active":"2022-06-27T11:47:52.378203+0000","last_peered":"2022-06-27T11:47:52.378203+0000","last_clean":"2022-06-27T11:47:52.378203+0000","last_became_active":"2022-06-27T11:47:52.376646+0000","last_became_peered":"2022-06-27T11:47:52.376646+0000","last_unstale":"2022-06-27T11:47:52.378203+0000","last_undegraded":"2022-06-27T11:47:52.378203+0000","last_fullsized":"2022-06-27T11:47:52.378203+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:17:51.913392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.369021+0000","last_change":"2022-06-27T11:47:51.979355+0000","last_active":"2022-06-27T11:47:52.369021+0000","last_peered":"2022-06-27T11:47:52.369021+0000","last_clean":"2022-06-27T11:47:52.369021+0000","last_became_active":"2022-06-27T11:47:51.977591+0000","last_became_peered":"2022-06-27T11:47:51.977591+0000","last_unstale":"2022-06-27T11:47:52.369021+0000","last_undegraded":"2022-06-27T11:47:52.369021+0000","last_fullsized":"2022-06-27T11:47:52.369021+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T11:50:43.086014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.385028+0000","last_change":"2022-06-27T11:47:52.375271+0000","last_active":"2022-06-27T11:47:52.385028+0000","last_peered":"2022-06-27T11:47:52.385028+0000","last_clean":"2022-06-27T11:47:52.385028+0000","last_became_active":"2022-06-27T11:47:52.374985+0000","last_became_peered":"2022-06-27T11:47:52.374985+0000","last_unstale":"2022-06-27T11:47:52.385028+0000","last_undegraded":"2022-06-27T11:47:52.385028+0000","last_fullsized":"2022-06-27T11:47:52.385028+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T17:58:40.711409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:57.367534+0000","last_change":"2022-06-27T11:47:51.979723+0000","last_active":"2022-06-27T11:47:57.367534+0000","last_peered":"2022-06-27T11:47:57.367534+0000","last_clean":"2022-06-27T11:47:57.367534+0000","last_became_active":"2022-06-27T11:47:51.977669+0000","last_became_peered":"2022-06-27T11:47:51.977669+0000","last_unstale":"2022-06-27T11:47:57.367534+0000","last_undegraded":"2022-06-27T11:47:57.367534+0000","last_fullsized":"2022-06-27T11:47:57.367534+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T14:35:14.770254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.363444+0000","last_change":"2022-06-27T11:47:48.388880+0000","last_active":"2022-06-27T11:47:52.363444+0000","last_peered":"2022-06-27T11:47:52.363444+0000","last_clean":"2022-06-27T11:47:52.363444+0000","last_became_active":"2022-06-27T11:47:48.388639+0000","last_became_peered":"2022-06-27T11:47:48.388639+0000","last_unstale":"2022-06-27T11:47:52.363444+0000","last_undegraded":"2022-06-27T11:47:52.363444+0000","last_fullsized":"2022-06-27T11:47:52.363444+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_clean_scrub_stamp":"2022-06-27T11:47:46.780512+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:31:10.549485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.380026+0000","last_change":"2022-06-27T11:47:52.373143+0000","last_active":"2022-06-27T11:47:52.380026+0000","last_peered":"2022-06-27T11:47:52.380026+0000","last_clean":"2022-06-27T11:47:52.380026+0000","last_became_active":"2022-06-27T11:47:52.372850+0000","last_became_peered":"2022-06-27T11:47:52.372850+0000","last_unstale":"2022-06-27T11:47:52.380026+0000","last_undegraded":"2022-06-27T11:47:52.380026+0000","last_fullsized":"2022-06-27T11:47:52.380026+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:48:44.251459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.369716+0000","last_change":"2022-06-27T11:47:47.793307+0000","last_active":"2022-06-27T11:47:52.369716+0000","last_peered":"2022-06-27T11:47:52.369716+0000","last_clean":"2022-06-27T11:47:52.369716+0000","last_became_active":"2022-06-27T11:47:47.792958+0000","last_became_peered":"2022-06-27T11:47:47.792958+0000","last_unstale":"2022-06-27T11:47:52.369716+0000","last_undegraded":"2022-06-27T11:47:52.369716+0000","last_fullsized":"2022-06-27T11:47:52.369716+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_clean_scrub_stamp":"2022-06-27T11:47:46.780512+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:08:35.128804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.372474+0000","last_change":"2022-06-27T11:47:51.976872+0000","last_active":"2022-06-27T11:47:52.372474+0000","last_peered":"2022-06-27T11:47:52.372474+0000","last_clean":"2022-06-27T11:47:52.372474+0000","last_became_active":"2022-06-27T11:47:51.976546+0000","last_became_peered":"2022-06-27T11:47:51.976546+0000","last_unstale":"2022-06-27T11:47:52.372474+0000","last_undegraded":"2022-06-27T11:47:52.372474+0000","last_fullsized":"2022-06-27T11:47:52.372474+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T14:44:43.349688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.375090+0000","last_change":"2022-06-27T11:47:52.372794+0000","last_active":"2022-06-27T11:47:52.375090+0000","last_peered":"2022-06-27T11:47:52.375090+0000","last_clean":"2022-06-27T11:47:52.375090+0000","last_became_active":"2022-06-27T11:47:52.372486+0000","last_became_peered":"2022-06-27T11:47:52.372486+0000","last_unstale":"2022-06-27T11:47:52.375090+0000","last_undegraded":"2022-06-27T11:47:52.375090+0000","last_fullsized":"2022-06-27T11:47:52.375090+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:23:56.307995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.363787+0000","last_change":"2022-06-27T11:47:47.792631+0000","last_active":"2022-06-27T11:47:52.363787+0000","last_peered":"2022-06-27T11:47:52.363787+0000","last_clean":"2022-06-27T11:47:52.363787+0000","last_became_active":"2022-06-27T11:47:47.792282+0000","last_became_peered":"2022-06-27T11:47:47.792282+0000","last_unstale":"2022-06-27T11:47:52.363787+0000","last_undegraded":"2022-06-27T11:47:52.363787+0000","last_fullsized":"2022-06-27T11:47:52.363787+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_clean_scrub_stamp":"2022-06-27T11:47:46.780512+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:41:25.449379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"17'117","reported_seq":163,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.368104+0000","last_change":"2022-06-27T11:47:47.791697+0000","last_active":"2022-06-27T11:47:52.368104+0000","last_peered":"2022-06-27T11:47:52.368104+0000","last_clean":"2022-06-27T11:47:52.368104+0000","last_became_active":"2022-06-27T11:47:47.790645+0000","last_became_peered":"2022-06-27T11:47:47.790645+0000","last_unstale":"2022-06-27T11:47:52.368104+0000","last_undegraded":"2022-06-27T11:47:52.368104+0000","last_fullsized":"2022-06-27T11:47:52.368104+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_clean_scrub_stamp":"2022-06-27T11:47:46.780512+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:11:09.344666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.367962+0000","last_change":"2022-06-27T11:47:51.982728+0000","last_active":"2022-06-27T11:47:52.367962+0000","last_peered":"2022-06-27T11:47:52.367962+0000","last_clean":"2022-06-27T11:47:52.367962+0000","last_became_active":"2022-06-27T11:47:51.982445+0000","last_became_peered":"2022-06-27T11:47:51.982445+0000","last_unstale":"2022-06-27T11:47:52.367962+0000","last_undegraded":"2022-06-27T11:47:52.367962+0000","last_fullsized":"2022-06-27T11:47:52.367962+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:42:04.682687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.386922+0000","last_change":"2022-06-27T11:47:52.386015+0000","last_active":"2022-06-27T11:47:52.386922+0000","last_peered":"2022-06-27T11:47:52.386922+0000","last_clean":"2022-06-27T11:47:52.386922+0000","last_became_active":"2022-06-27T11:47:52.385731+0000","last_became_peered":"2022-06-27T11:47:52.385731+0000","last_unstale":"2022-06-27T11:47:52.386922+0000","last_undegraded":"2022-06-27T11:47:52.386922+0000","last_fullsized":"2022-06-27T11:47:52.386922+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T11:58:50.483766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.366627+0000","last_change":"2022-06-27T11:47:47.807699+0000","last_active":"2022-06-27T11:47:52.366627+0000","last_peered":"2022-06-27T11:47:52.366627+0000","last_clean":"2022-06-27T11:47:52.366627+0000","last_became_active":"2022-06-27T11:47:47.807422+0000","last_became_peered":"2022-06-27T11:47:47.807422+0000","last_unstale":"2022-06-27T11:47:52.366627+0000","last_undegraded":"2022-06-27T11:47:52.366627+0000","last_fullsized":"2022-06-27T11:47:52.366627+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_clean_scrub_stamp":"2022-06-27T11:47:46.780512+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:15:00.549295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.363922+0000","last_change":"2022-06-27T11:47:51.974468+0000","last_active":"2022-06-27T11:47:52.363922+0000","last_peered":"2022-06-27T11:47:52.363922+0000","last_clean":"2022-06-27T11:47:52.363922+0000","last_became_active":"2022-06-27T11:47:51.974160+0000","last_became_peered":"2022-06-27T11:47:51.974160+0000","last_unstale":"2022-06-27T11:47:52.363922+0000","last_undegraded":"2022-06-27T11:47:52.363922+0000","last_fullsized":"2022-06-27T11:47:52.363922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T19:14:45.694282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.380162+0000","last_change":"2022-06-27T11:47:52.373931+0000","last_active":"2022-06-27T11:47:52.380162+0000","last_peered":"2022-06-27T11:47:52.380162+0000","last_clean":"2022-06-27T11:47:52.380162+0000","last_became_active":"2022-06-27T11:47:52.373562+0000","last_became_peered":"2022-06-27T11:47:52.373562+0000","last_unstale":"2022-06-27T11:47:52.380162+0000","last_undegraded":"2022-06-27T11:47:52.380162+0000","last_fullsized":"2022-06-27T11:47:52.380162+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T14:09:22.859955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.367866+0000","last_change":"2022-06-27T11:47:47.807650+0000","last_active":"2022-06-27T11:47:52.367866+0000","last_peered":"2022-06-27T11:47:52.367866+0000","last_clean":"2022-06-27T11:47:52.367866+0000","last_became_active":"2022-06-27T11:47:47.807375+0000","last_became_peered":"2022-06-27T11:47:47.807375+0000","last_unstale":"2022-06-27T11:47:52.367866+0000","last_undegraded":"2022-06-27T11:47:52.367866+0000","last_fullsized":"2022-06-27T11:47:52.367866+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_clean_scrub_stamp":"2022-06-27T11:47:46.780512+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T14:10:48.067284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.368898+0000","last_change":"2022-06-27T11:47:51.972154+0000","last_active":"2022-06-27T11:47:52.368898+0000","last_peered":"2022-06-27T11:47:52.368898+0000","last_clean":"2022-06-27T11:47:52.368898+0000","last_became_active":"2022-06-27T11:47:51.971762+0000","last_became_peered":"2022-06-27T11:47:51.971762+0000","last_unstale":"2022-06-27T11:47:52.368898+0000","last_undegraded":"2022-06-27T11:47:52.368898+0000","last_fullsized":"2022-06-27T11:47:52.368898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T18:15:14.271005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.379879+0000","last_change":"2022-06-27T11:47:52.378294+0000","last_active":"2022-06-27T11:47:52.379879+0000","last_peered":"2022-06-27T11:47:52.379879+0000","last_clean":"2022-06-27T11:47:52.379879+0000","last_became_active":"2022-06-27T11:47:52.377704+0000","last_became_peered":"2022-06-27T11:47:52.377704+0000","last_unstale":"2022-06-27T11:47:52.379879+0000","last_undegraded":"2022-06-27T11:47:52.379879+0000","last_fullsized":"2022-06-27T11:47:52.379879+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:26:34.474081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.364180+0000","last_change":"2022-06-27T11:47:47.807600+0000","last_active":"2022-06-27T11:47:52.364180+0000","last_peered":"2022-06-27T11:47:52.364180+0000","last_clean":"2022-06-27T11:47:52.364180+0000","last_became_active":"2022-06-27T11:47:47.807284+0000","last_became_peered":"2022-06-27T11:47:47.807284+0000","last_unstale":"2022-06-27T11:47:52.364180+0000","last_undegraded":"2022-06-27T11:47:52.364180+0000","last_fullsized":"2022-06-27T11:47:52.364180+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_clean_scrub_stamp":"2022-06-27T11:47:46.780512+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T20:57:06.487027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.362687+0000","last_change":"2022-06-27T11:47:51.980734+0000","last_active":"2022-06-27T11:47:52.362687+0000","last_peered":"2022-06-27T11:47:52.362687+0000","last_clean":"2022-06-27T11:47:52.362687+0000","last_became_active":"2022-06-27T11:47:51.980476+0000","last_became_peered":"2022-06-27T11:47:51.980476+0000","last_unstale":"2022-06-27T11:47:52.362687+0000","last_undegraded":"2022-06-27T11:47:52.362687+0000","last_fullsized":"2022-06-27T11:47:52.362687+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T21:32:16.435472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.385478+0000","last_change":"2022-06-27T11:47:52.379902+0000","last_active":"2022-06-27T11:47:52.385478+0000","last_peered":"2022-06-27T11:47:52.385478+0000","last_clean":"2022-06-27T11:47:52.385478+0000","last_became_active":"2022-06-27T11:47:52.379359+0000","last_became_peered":"2022-06-27T11:47:52.379359+0000","last_unstale":"2022-06-27T11:47:52.385478+0000","last_undegraded":"2022-06-27T11:47:52.385478+0000","last_fullsized":"2022-06-27T11:47:52.385478+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:41:27.884083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.370459+0000","last_change":"2022-06-27T11:47:47.807607+0000","last_active":"2022-06-27T11:47:52.370459+0000","last_peered":"2022-06-27T11:47:52.370459+0000","last_clean":"2022-06-27T11:47:52.370459+0000","last_became_active":"2022-06-27T11:47:47.807293+0000","last_became_peered":"2022-06-27T11:47:47.807293+0000","last_unstale":"2022-06-27T11:47:52.370459+0000","last_undegraded":"2022-06-27T11:47:52.370459+0000","last_fullsized":"2022-06-27T11:47:52.370459+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_clean_scrub_stamp":"2022-06-27T11:47:46.780512+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:00:35.318372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.364133+0000","last_change":"2022-06-27T11:47:51.976055+0000","last_active":"2022-06-27T11:47:52.364133+0000","last_peered":"2022-06-27T11:47:52.364133+0000","last_clean":"2022-06-27T11:47:52.364133+0000","last_became_active":"2022-06-27T11:47:51.975303+0000","last_became_peered":"2022-06-27T11:47:51.975303+0000","last_unstale":"2022-06-27T11:47:52.364133+0000","last_undegraded":"2022-06-27T11:47:52.364133+0000","last_fullsized":"2022-06-27T11:47:52.364133+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:27:32.125525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384750+0000","last_change":"2022-06-27T11:47:52.380775+0000","last_active":"2022-06-27T11:47:52.384750+0000","last_peered":"2022-06-27T11:47:52.384750+0000","last_clean":"2022-06-27T11:47:52.384750+0000","last_became_active":"2022-06-27T11:47:52.380439+0000","last_became_peered":"2022-06-27T11:47:52.380439+0000","last_unstale":"2022-06-27T11:47:52.384750+0000","last_undegraded":"2022-06-27T11:47:52.384750+0000","last_fullsized":"2022-06-27T11:47:52.384750+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:03:44.939329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.370743+0000","last_change":"2022-06-27T11:47:47.792314+0000","last_active":"2022-06-27T11:47:52.370743+0000","last_peered":"2022-06-27T11:47:52.370743+0000","last_clean":"2022-06-27T11:47:52.370743+0000","last_became_active":"2022-06-27T11:47:47.792032+0000","last_became_peered":"2022-06-27T11:47:47.792032+0000","last_unstale":"2022-06-27T11:47:52.370743+0000","last_undegraded":"2022-06-27T11:47:52.370743+0000","last_fullsized":"2022-06-27T11:47:52.370743+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_clean_scrub_stamp":"2022-06-27T11:47:46.780512+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T14:13:03.591775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.369208+0000","last_change":"2022-06-27T11:47:51.978639+0000","last_active":"2022-06-27T11:47:52.369208+0000","last_peered":"2022-06-27T11:47:52.369208+0000","last_clean":"2022-06-27T11:47:52.369208+0000","last_became_active":"2022-06-27T11:47:51.978341+0000","last_became_peered":"2022-06-27T11:47:51.978341+0000","last_unstale":"2022-06-27T11:47:52.369208+0000","last_undegraded":"2022-06-27T11:47:52.369208+0000","last_fullsized":"2022-06-27T11:47:52.369208+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T21:34:48.070558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.381133+0000","last_change":"2022-06-27T11:47:52.373588+0000","last_active":"2022-06-27T11:47:52.381133+0000","last_peered":"2022-06-27T11:47:52.381133+0000","last_clean":"2022-06-27T11:47:52.381133+0000","last_became_active":"2022-06-27T11:47:52.373208+0000","last_became_peered":"2022-06-27T11:47:52.373208+0000","last_unstale":"2022-06-27T11:47:52.381133+0000","last_undegraded":"2022-06-27T11:47:52.381133+0000","last_fullsized":"2022-06-27T11:47:52.381133+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T20:27:28.252147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:53.910019+0000","last_change":"2022-06-27T11:47:51.982051+0000","last_active":"2022-06-27T11:47:53.910019+0000","last_peered":"2022-06-27T11:47:53.910019+0000","last_clean":"2022-06-27T11:47:53.910019+0000","last_became_active":"2022-06-27T11:47:51.981742+0000","last_became_peered":"2022-06-27T11:47:51.981742+0000","last_unstale":"2022-06-27T11:47:53.910019+0000","last_undegraded":"2022-06-27T11:47:53.910019+0000","last_fullsized":"2022-06-27T11:47:53.910019+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:39:56.937086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.380333+0000","last_change":"2022-06-27T11:47:52.378192+0000","last_active":"2022-06-27T11:47:52.380333+0000","last_peered":"2022-06-27T11:47:52.380333+0000","last_clean":"2022-06-27T11:47:52.380333+0000","last_became_active":"2022-06-27T11:47:52.377919+0000","last_became_peered":"2022-06-27T11:47:52.377919+0000","last_unstale":"2022-06-27T11:47:52.380333+0000","last_undegraded":"2022-06-27T11:47:52.380333+0000","last_fullsized":"2022-06-27T11:47:52.380333+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T18:51:35.757778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:57.368777+0000","last_change":"2022-06-27T11:47:51.981101+0000","last_active":"2022-06-27T11:47:57.368777+0000","last_peered":"2022-06-27T11:47:57.368777+0000","last_clean":"2022-06-27T11:47:57.368777+0000","last_became_active":"2022-06-27T11:47:51.980793+0000","last_became_peered":"2022-06-27T11:47:51.980793+0000","last_unstale":"2022-06-27T11:47:57.368777+0000","last_undegraded":"2022-06-27T11:47:57.368777+0000","last_fullsized":"2022-06-27T11:47:57.368777+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:28:10.206148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.372181+0000","last_change":"2022-06-27T11:47:52.370334+0000","last_active":"2022-06-27T11:47:52.372181+0000","last_peered":"2022-06-27T11:47:52.372181+0000","last_clean":"2022-06-27T11:47:52.372181+0000","last_became_active":"2022-06-27T11:47:52.370028+0000","last_became_peered":"2022-06-27T11:47:52.370028+0000","last_unstale":"2022-06-27T11:47:52.372181+0000","last_undegraded":"2022-06-27T11:47:52.372181+0000","last_fullsized":"2022-06-27T11:47:52.372181+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T17:28:12.357757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:57.369040+0000","last_change":"2022-06-27T11:47:51.981186+0000","last_active":"2022-06-27T11:47:57.369040+0000","last_peered":"2022-06-27T11:47:57.369040+0000","last_clean":"2022-06-27T11:47:57.369040+0000","last_became_active":"2022-06-27T11:47:51.980921+0000","last_became_peered":"2022-06-27T11:47:51.980921+0000","last_unstale":"2022-06-27T11:47:57.369040+0000","last_undegraded":"2022-06-27T11:47:57.369040+0000","last_fullsized":"2022-06-27T11:47:57.369040+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:04:40.732299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.387201+0000","last_change":"2022-06-27T11:47:52.382091+0000","last_active":"2022-06-27T11:47:52.387201+0000","last_peered":"2022-06-27T11:47:52.387201+0000","last_clean":"2022-06-27T11:47:52.387201+0000","last_became_active":"2022-06-27T11:47:52.381633+0000","last_became_peered":"2022-06-27T11:47:52.381633+0000","last_unstale":"2022-06-27T11:47:52.387201+0000","last_undegraded":"2022-06-27T11:47:52.387201+0000","last_fullsized":"2022-06-27T11:47:52.387201+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:34:44.030534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:53.910420+0000","last_change":"2022-06-27T11:47:51.972963+0000","last_active":"2022-06-27T11:47:53.910420+0000","last_peered":"2022-06-27T11:47:53.910420+0000","last_clean":"2022-06-27T11:47:53.910420+0000","last_became_active":"2022-06-27T11:47:51.972686+0000","last_became_peered":"2022-06-27T11:47:51.972686+0000","last_unstale":"2022-06-27T11:47:53.910420+0000","last_undegraded":"2022-06-27T11:47:53.910420+0000","last_fullsized":"2022-06-27T11:47:53.910420+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T12:11:38.008538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.385365+0000","last_change":"2022-06-27T11:47:52.383414+0000","last_active":"2022-06-27T11:47:52.385365+0000","last_peered":"2022-06-27T11:47:52.385365+0000","last_clean":"2022-06-27T11:47:52.385365+0000","last_became_active":"2022-06-27T11:47:52.383070+0000","last_became_peered":"2022-06-27T11:47:52.383070+0000","last_unstale":"2022-06-27T11:47:52.385365+0000","last_undegraded":"2022-06-27T11:47:52.385365+0000","last_fullsized":"2022-06-27T11:47:52.385365+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:11:28.410045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"21'4","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:57.370018+0000","last_change":"2022-06-27T11:47:51.971664+0000","last_active":"2022-06-27T11:47:57.370018+0000","last_peered":"2022-06-27T11:47:57.370018+0000","last_clean":"2022-06-27T11:47:57.370018+0000","last_became_active":"2022-06-27T11:47:51.971385+0000","last_became_peered":"2022-06-27T11:47:51.971385+0000","last_unstale":"2022-06-27T11:47:57.370018+0000","last_undegraded":"2022-06-27T11:47:57.370018+0000","last_fullsized":"2022-06-27T11:47:57.370018+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T18:44:38.394388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.379723+0000","last_change":"2022-06-27T11:47:52.377305+0000","last_active":"2022-06-27T11:47:52.379723+0000","last_peered":"2022-06-27T11:47:52.379723+0000","last_clean":"2022-06-27T11:47:52.379723+0000","last_became_active":"2022-06-27T11:47:52.377012+0000","last_became_peered":"2022-06-27T11:47:52.377012+0000","last_unstale":"2022-06-27T11:47:52.379723+0000","last_undegraded":"2022-06-27T11:47:52.379723+0000","last_fullsized":"2022-06-27T11:47:52.379723+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:15:14.505656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.377592+0000","last_change":"2022-06-27T11:47:52.376074+0000","last_active":"2022-06-27T11:47:52.377592+0000","last_peered":"2022-06-27T11:47:52.377592+0000","last_clean":"2022-06-27T11:47:52.377592+0000","last_became_active":"2022-06-27T11:47:52.375836+0000","last_became_peered":"2022-06-27T11:47:52.375836+0000","last_unstale":"2022-06-27T11:47:52.377592+0000","last_undegraded":"2022-06-27T11:47:52.377592+0000","last_fullsized":"2022-06-27T11:47:52.377592+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T18:46:26.900833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384440+0000","last_change":"2022-06-27T11:47:52.376668+0000","last_active":"2022-06-27T11:47:52.384440+0000","last_peered":"2022-06-27T11:47:52.384440+0000","last_clean":"2022-06-27T11:47:52.384440+0000","last_became_active":"2022-06-27T11:47:52.376386+0000","last_became_peered":"2022-06-27T11:47:52.376386+0000","last_unstale":"2022-06-27T11:47:52.384440+0000","last_undegraded":"2022-06-27T11:47:52.384440+0000","last_fullsized":"2022-06-27T11:47:52.384440+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T11:48:35.874748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.379744+0000","last_change":"2022-06-27T11:47:52.377793+0000","last_active":"2022-06-27T11:47:52.379744+0000","last_peered":"2022-06-27T11:47:52.379744+0000","last_clean":"2022-06-27T11:47:52.379744+0000","last_became_active":"2022-06-27T11:47:52.377318+0000","last_became_peered":"2022-06-27T11:47:52.377318+0000","last_unstale":"2022-06-27T11:47:52.379744+0000","last_undegraded":"2022-06-27T11:47:52.379744+0000","last_fullsized":"2022-06-27T11:47:52.379744+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T17:54:22.511301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384925+0000","last_change":"2022-06-27T11:47:52.374638+0000","last_active":"2022-06-27T11:47:52.384925+0000","last_peered":"2022-06-27T11:47:52.384925+0000","last_clean":"2022-06-27T11:47:52.384925+0000","last_became_active":"2022-06-27T11:47:52.374352+0000","last_became_peered":"2022-06-27T11:47:52.374352+0000","last_unstale":"2022-06-27T11:47:52.384925+0000","last_undegraded":"2022-06-27T11:47:52.384925+0000","last_fullsized":"2022-06-27T11:47:52.384925+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:09:59.369369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.375769+0000","last_change":"2022-06-27T11:47:52.373288+0000","last_active":"2022-06-27T11:47:52.375769+0000","last_peered":"2022-06-27T11:47:52.375769+0000","last_clean":"2022-06-27T11:47:52.375769+0000","last_became_active":"2022-06-27T11:47:52.372891+0000","last_became_peered":"2022-06-27T11:47:52.372891+0000","last_unstale":"2022-06-27T11:47:52.375769+0000","last_undegraded":"2022-06-27T11:47:52.375769+0000","last_fullsized":"2022-06-27T11:47:52.375769+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T23:16:08.088450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64756,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9016,"kb_used_data":304,"kb_used_omap":0,"kb_used_meta":8704,"kb_avail":94362824,"statfs":{"total":96636764160,"available":96627531776,"internally_reserved":0,"allocated":311296,"data_stored":670971,"data_compressed":32378,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":8912896},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39400000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.316}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40899999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28699999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42999999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46300000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8408,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":8192,"kb_avail":94363432,"statfs":{"total":96636764160,"available":96628154368,"internally_reserved":0,"allocated":212992,"data_stored":77836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8388608},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47999999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.37}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8996,"kb_used_data":276,"kb_used_omap":0,"kb_used_meta":8704,"kb_avail":94362844,"statfs":{"total":96636764160,"available":96627552256,"internally_reserved":0,"allocated":282624,"data_stored":669207,"data_compressed":32378,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":8912896},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28599999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39700000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8480,"kb_used_data":216,"kb_used_omap":0,"kb_used_meta":8256,"kb_avail":94363360,"statfs":{"total":96636764160,"available":96628080640,"internally_reserved":0,"allocated":221184,"data_stored":79405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8454144},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.377}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26200000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35499999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34699999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42999999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38900000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8412,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":8192,"kb_avail":94363428,"statfs":{"total":96636764160,"available":96628150272,"internally_reserved":0,"allocated":217088,"data_stored":78839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8388608},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.252}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46000000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41699999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]}]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8476,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":8256,"kb_avail":94363364,"statfs":{"total":96636764160,"available":96628084736,"internally_reserved":0,"allocated":217088,"data_stored":77926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8454144},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.484}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3620000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.205}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.113}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.339}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.172000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.231999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8560,"kb_used_data":232,"kb_used_omap":0,"kb_used_meta":8320,"kb_avail":94363280,"statfs":{"total":96636764160,"available":96627998720,"internally_reserved":0,"allocated":237568,"data_stored":80443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8519680},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39700000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45200000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45800000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48899999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8420,"kb_used_data":220,"kb_used_omap":0,"kb_used_meta":8192,"kb_avail":94363420,"statfs":{"total":96636764160,"available":96628142080,"internally_reserved":0,"allocated":225280,"data_stored":79475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8388608},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68100000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.702}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.8689999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78100000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46400000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32378,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32378,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-06-27T11:48:02.436 INFO:tasks.ceph.ceph_manager.ceph:clean! 2022-06-27T11:48:02.437 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2022-06-27T11:48:02.437 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2022-06-27T11:48:02.437 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2022-06-27T11:48:02.590 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.590+0000 7f3228cca700 1 Processor -- start 2022-06-27T11:48:02.590 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.590+0000 7f3228cca700 1 -- start start 2022-06-27T11:48:02.593 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.591+0000 7f3228cca700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f322406eff0 0x7f32240c61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:02.593 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.591+0000 7f3228cca700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f322414d8a0 con 0x7f3224067ed0 2022-06-27T11:48:02.593 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.591+0000 7f3228cca700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32240c3e10 con 0x7f32240c31d0 2022-06-27T11:48:02.594 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.591+0000 7f3228cca700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f322406d970 con 0x7f322406eff0 2022-06-27T11:48:02.594 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.591+0000 7f322259c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f322406eff0 0x7f32240c61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:02.594 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.591+0000 7f3221d9b700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f3224067ed0 0x7f32240c2ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:47886/0 (socket says 172.21.15.195:47886) 2022-06-27T11:48:02.594 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.591+0000 7f3221d9b700 1 -- 172.21.15.195:0/2935753191 learned_addr learned my addr 172.21.15.195:0/2935753191 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:48:02.595 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.592+0000 7f322259c700 1 -- 172.21.15.195:0/2935753191 >> v1:172.21.15.202:6789/0 conn(0x7f32240c31d0 legacy=0x7f322415da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:02.595 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.592+0000 7f322259c700 1 -- 172.21.15.195:0/2935753191 >> v1:172.21.15.195:6789/0 conn(0x7f3224067ed0 legacy=0x7f32240c2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:02.595 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.592+0000 7f322259c700 1 -- 172.21.15.195:0/2935753191 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3224153f20 con 0x7f322406eff0 2022-06-27T11:48:02.595 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.592+0000 7f322159a700 1 -- 172.21.15.195:0/2935753191 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3814008737 0 0) 0x7f322414d8a0 con 0x7f3224067ed0 2022-06-27T11:48:02.596 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.592+0000 7f322259c700 1 --2- 172.21.15.195:0/2935753191 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f322406eff0 0x7f32240c61e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3214009720 tx=0x7f3214009ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb222bdd1456da44 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:02.596 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.592+0000 7f322159a700 1 -- 172.21.15.195:0/2935753191 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3214005ad0 con 0x7f322406eff0 2022-06-27T11:48:02.596 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.592+0000 7f322159a700 1 -- 172.21.15.195:0/2935753191 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3214005c30 con 0x7f322406eff0 2022-06-27T11:48:02.596 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.592+0000 7f322159a700 1 -- 172.21.15.195:0/2935753191 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3214004030 con 0x7f322406eff0 2022-06-27T11:48:02.596 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.592+0000 7f3228cca700 1 -- 172.21.15.195:0/2935753191 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f322406eff0 msgr2=0x7f32240c61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:02.597 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.592+0000 7f3228cca700 1 --2- 172.21.15.195:0/2935753191 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f322406eff0 0x7f32240c61e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3214009720 tx=0x7f3214009ae0 comp rx=0 tx=0).stop 2022-06-27T11:48:02.597 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.592+0000 7f3228cca700 1 -- 172.21.15.195:0/2935753191 shutdown_connections 2022-06-27T11:48:02.597 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.592+0000 7f3228cca700 1 --2- 172.21.15.195:0/2935753191 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f322406eff0 0x7f32240c61e0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:02.598 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.592+0000 7f3228cca700 1 -- 172.21.15.195:0/2935753191 >> 172.21.15.195:0/2935753191 conn(0x7f32240c9b00 msgr2=0x7f32240c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:02.598 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.593+0000 7f3228cca700 1 -- 172.21.15.195:0/2935753191 shutdown_connections 2022-06-27T11:48:02.599 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.593+0000 7f3228cca700 1 -- 172.21.15.195:0/2935753191 wait complete. 2022-06-27T11:48:02.599 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.594+0000 7f3228cca700 1 Processor -- start 2022-06-27T11:48:02.600 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.594+0000 7f3228cca700 1 -- start start 2022-06-27T11:48:02.604 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.594+0000 7f3228cca700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3224067ed0 0x7f322406dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:02.604 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.594+0000 7f3228cca700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f322406eff0 0x7f3224169ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:02.604 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.594+0000 7f3228cca700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3224069390 0x7f322416a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:02.605 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.594+0000 7f3228cca700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3224154340 con 0x7f3224067ed0 2022-06-27T11:48:02.605 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.594+0000 7f3228cca700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32241545b0 con 0x7f3224069390 2022-06-27T11:48:02.605 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.594+0000 7f3228cca700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3224154890 con 0x7f322406eff0 2022-06-27T11:48:02.606 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.594+0000 7f322259c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3224067ed0 0x7f322406dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:02.606 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.594+0000 7f322259c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3224067ed0 0x7f322406dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41704/0 (socket says 172.21.15.195:41704) 2022-06-27T11:48:02.606 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.594+0000 7f322259c700 1 -- 172.21.15.195:0/2656480211 learned_addr learned my addr 172.21.15.195:0/2656480211 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:48:02.606 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.595+0000 7f322259c700 1 -- 172.21.15.195:0/2656480211 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f322406eff0 msgr2=0x7f3224169ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:02.607 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.595+0000 7f322259c700 1 --2- 172.21.15.195:0/2656480211 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f322406eff0 0x7f3224169ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:02.607 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.595+0000 7f322259c700 1 -- 172.21.15.195:0/2656480211 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3224069390 msgr2=0x7f322416a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:02.608 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.595+0000 7f322259c700 1 --2- 172.21.15.195:0/2656480211 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3224069390 0x7f322416a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:02.608 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.595+0000 7f322259c700 1 -- 172.21.15.195:0/2656480211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3214009290 con 0x7f3224067ed0 2022-06-27T11:48:02.609 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.595+0000 7f322259c700 1 --2- 172.21.15.195:0/2656480211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3224067ed0 0x7f322406dee0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3214009b20 tx=0x7f3214015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:02.609 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.595+0000 7f321a7fc700 1 -- 172.21.15.195:0/2656480211 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3214000cc0 con 0x7f3224067ed0 2022-06-27T11:48:02.609 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.595+0000 7f321a7fc700 1 -- 172.21.15.195:0/2656480211 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f321401d650 con 0x7f3224067ed0 2022-06-27T11:48:02.610 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.595+0000 7f3228cca700 1 -- 172.21.15.195:0/2656480211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3224154b70 con 0x7f3224067ed0 2022-06-27T11:48:02.610 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.595+0000 7f321a7fc700 1 -- 172.21.15.195:0/2656480211 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f321401d800 con 0x7f3224067ed0 2022-06-27T11:48:02.610 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.595+0000 7f3228cca700 1 -- 172.21.15.195:0/2656480211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f322406f630 con 0x7f3224067ed0 2022-06-27T11:48:02.610 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.596+0000 7f3228cca700 1 -- 172.21.15.195:0/2656480211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3224169d10 con 0x7f3224067ed0 2022-06-27T11:48:02.611 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.596+0000 7f321a7fc700 1 -- 172.21.15.195:0/2656480211 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f321400f050 con 0x7f3224067ed0 2022-06-27T11:48:02.611 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.597+0000 7f321a7fc700 1 --2- 172.21.15.195:0/2656480211 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3208063d00 0x7f32080661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:02.611 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.597+0000 7f321a7fc700 1 -- 172.21.15.195:0/2656480211 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f3214082d70 con 0x7f3224067ed0 2022-06-27T11:48:02.612 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.597+0000 7f3221d9b700 1 --2- 172.21.15.195:0/2656480211 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3208063d00 0x7f32080661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:02.612 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.597+0000 7f3221d9b700 1 --2- 172.21.15.195:0/2656480211 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3208063d00 0x7f32080661b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3224063f50 tx=0x7f320c007990 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:02.612 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.601+0000 7f321a7fc700 1 -- 172.21.15.195:0/2656480211 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f3214051870 con 0x7f3224067ed0 2022-06-27T11:48:02.829 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.829+0000 7f3228cca700 1 -- 172.21.15.195:0/2656480211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3224156300 con 0x7f3224067ed0 2022-06-27T11:48:02.829 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.830+0000 7f321a7fc700 1 -- 172.21.15.195:0/2656480211 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f3224156300 con 0x7f3224067ed0 2022-06-27T11:48:02.830 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:48:02.830 INFO:teuthology.orchestra.run.smithi195.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2022-06-27T11:48:02.831 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.832+0000 7f3228cca700 1 -- 172.21.15.195:0/2656480211 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3208063d00 msgr2=0x7f32080661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:02.832 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.832+0000 7f3228cca700 1 --2- 172.21.15.195:0/2656480211 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3208063d00 0x7f32080661b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3224063f50 tx=0x7f320c007990 comp rx=0 tx=0).stop 2022-06-27T11:48:02.832 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.832+0000 7f3228cca700 1 -- 172.21.15.195:0/2656480211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3224067ed0 msgr2=0x7f322406dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:02.832 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.832+0000 7f3228cca700 1 --2- 172.21.15.195:0/2656480211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3224067ed0 0x7f322406dee0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3214009b20 tx=0x7f3214015000 comp rx=0 tx=0).stop 2022-06-27T11:48:02.832 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.832+0000 7f3228cca700 1 -- 172.21.15.195:0/2656480211 shutdown_connections 2022-06-27T11:48:02.832 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.832+0000 7f3228cca700 1 --2- 172.21.15.195:0/2656480211 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3208063d00 0x7f32080661b0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:02.833 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.832+0000 7f3228cca700 1 --2- 172.21.15.195:0/2656480211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3224067ed0 0x7f322406dee0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:02.833 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.832+0000 7f3228cca700 1 --2- 172.21.15.195:0/2656480211 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3224069390 0x7f322416a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:02.833 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.832+0000 7f3228cca700 1 --2- 172.21.15.195:0/2656480211 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f322406eff0 0x7f3224169ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:02.833 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.832+0000 7f3228cca700 1 -- 172.21.15.195:0/2656480211 >> 172.21.15.195:0/2656480211 conn(0x7f32240c9b00 msgr2=0x7f322414cca0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:02.834 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.832+0000 7f3228cca700 1 -- 172.21.15.195:0/2656480211 shutdown_connections 2022-06-27T11:48:02.834 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.832+0000 7f3228cca700 1 -- 172.21.15.195:0/2656480211 wait complete. 2022-06-27T11:48:02.844 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2022-06-27T11:48:02.845 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-27T11:48:02.965 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.966+0000 7f21eff2b700 1 Processor -- start 2022-06-27T11:48:02.966 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.966+0000 7f21eff2b700 1 -- start start 2022-06-27T11:48:02.974 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.967+0000 7f21eff2b700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f21e8156100 0x7f21e80688a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:02.974 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.967+0000 7f21eff2b700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f21e806a440 0x7f21e80c4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:02.975 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.967+0000 7f21eff2b700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21e80c4b10 con 0x7f21e81564d0 2022-06-27T11:48:02.975 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.967+0000 7f21eff2b700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21e80c3dc0 con 0x7f21e806a440 2022-06-27T11:48:02.975 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.967+0000 7f21eff2b700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21e8068f20 con 0x7f21e8156100 2022-06-27T11:48:02.976 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.967+0000 7f21ed4c6700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f21e806a440 0x7f21e80c4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:02.976 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.967+0000 7f21ed4c6700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f21e806a440 0x7f21e80c4fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41712/0 (socket says 172.21.15.195:41712) 2022-06-27T11:48:02.976 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.967+0000 7f21ed4c6700 1 -- 172.21.15.195:0/3481434751 learned_addr learned my addr 172.21.15.195:0/3481434751 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:48:02.976 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.967+0000 7f21edcc7700 1 --2- 172.21.15.195:0/3481434751 >> v2:172.21.15.202:3300/0 conn(0x7f21e8156100 0x7f21e80688a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:02.977 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.967+0000 7f21ed4c6700 1 -- 172.21.15.195:0/3481434751 >> v2:172.21.15.202:3300/0 conn(0x7f21e8156100 msgr2=0x7f21e80688a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:02.977 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.967+0000 7f21ed4c6700 1 --2- 172.21.15.195:0/3481434751 >> v2:172.21.15.202:3300/0 conn(0x7f21e8156100 0x7f21e80688a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:02.978 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.967+0000 7f21ed4c6700 1 -- 172.21.15.195:0/3481434751 >> v1:172.21.15.202:6789/0 conn(0x7f21e81564d0 legacy=0x7f21e80c28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:02.978 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.967+0000 7f21ed4c6700 1 -- 172.21.15.195:0/3481434751 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21e80c6360 con 0x7f21e806a440 2022-06-27T11:48:02.978 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.967+0000 7f21ed4c6700 1 --2- 172.21.15.195:0/3481434751 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f21e806a440 0x7f21e80c4fa0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f21e8070770 tx=0x7f21d8009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f5ee4ba607b584e server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:02.979 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.968+0000 7f21eccc5700 1 -- 172.21.15.195:0/3481434751 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d8004030 con 0x7f21e806a440 2022-06-27T11:48:02.979 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.968+0000 7f21eccc5700 1 -- 172.21.15.195:0/3481434751 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21d8004190 con 0x7f21e806a440 2022-06-27T11:48:02.979 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.968+0000 7f21eccc5700 1 -- 172.21.15.195:0/3481434751 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d8011630 con 0x7f21e806a440 2022-06-27T11:48:02.979 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.968+0000 7f21eff2b700 1 -- 172.21.15.195:0/3481434751 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f21e806a440 msgr2=0x7f21e80c4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:02.980 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.968+0000 7f21eff2b700 1 --2- 172.21.15.195:0/3481434751 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f21e806a440 0x7f21e80c4fa0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f21e8070770 tx=0x7f21d8009120 comp rx=0 tx=0).stop 2022-06-27T11:48:02.980 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.968+0000 7f21eff2b700 1 -- 172.21.15.195:0/3481434751 shutdown_connections 2022-06-27T11:48:02.980 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.968+0000 7f21eff2b700 1 --2- 172.21.15.195:0/3481434751 >> v2:172.21.15.202:3300/0 conn(0x7f21e8156100 0x7f21e80688a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:02.980 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.968+0000 7f21eff2b700 1 --2- 172.21.15.195:0/3481434751 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f21e806a440 0x7f21e80c4fa0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:02.981 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.968+0000 7f21eff2b700 1 -- 172.21.15.195:0/3481434751 >> 172.21.15.195:0/3481434751 conn(0x7f21e80c9b00 msgr2=0x7f21e80c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:02.981 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.968+0000 7f21eff2b700 1 -- 172.21.15.195:0/3481434751 shutdown_connections 2022-06-27T11:48:02.981 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.968+0000 7f21eff2b700 1 -- 172.21.15.195:0/3481434751 wait complete. 2022-06-27T11:48:02.982 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.969+0000 7f21eff2b700 1 Processor -- start 2022-06-27T11:48:02.982 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.969+0000 7f21eff2b700 1 -- start start 2022-06-27T11:48:02.983 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.970+0000 7f21eff2b700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f21e806a440 0x7f21e814ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:02.983 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.970+0000 7f21eff2b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f21e8156100 0x7f21e81550c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:02.983 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.970+0000 7f21eff2b700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f21e81564d0 0x7f21e8155600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:02.984 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.970+0000 7f21eff2b700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21e8152cc0 con 0x7f21e8156100 2022-06-27T11:48:02.984 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.970+0000 7f21eff2b700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21e81668a0 con 0x7f21e806a440 2022-06-27T11:48:02.984 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.970+0000 7f21eff2b700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21e8154c30 con 0x7f21e81564d0 2022-06-27T11:48:02.985 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.970+0000 7f21ed4c6700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f21e8156100 0x7f21e81550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:02.985 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.970+0000 7f21ed4c6700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f21e8156100 0x7f21e81550c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41718/0 (socket says 172.21.15.195:41718) 2022-06-27T11:48:02.985 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.970+0000 7f21ed4c6700 1 -- 172.21.15.195:0/696621824 learned_addr learned my addr 172.21.15.195:0/696621824 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:48:02.985 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.970+0000 7f21edcc7700 1 --2- 172.21.15.195:0/696621824 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f21e806a440 0x7f21e814ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:02.986 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.970+0000 7f21ee4c8700 1 --2- 172.21.15.195:0/696621824 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f21e81564d0 0x7f21e8155600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:02.986 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.970+0000 7f21ed4c6700 1 -- 172.21.15.195:0/696621824 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f21e81564d0 msgr2=0x7f21e8155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:02.986 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.970+0000 7f21ed4c6700 1 --2- 172.21.15.195:0/696621824 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f21e81564d0 0x7f21e8155600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:02.986 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.970+0000 7f21ed4c6700 1 -- 172.21.15.195:0/696621824 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f21e806a440 msgr2=0x7f21e814ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:02.987 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.970+0000 7f21ed4c6700 1 --2- 172.21.15.195:0/696621824 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f21e806a440 0x7f21e814ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:02.987 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.970+0000 7f21ed4c6700 1 -- 172.21.15.195:0/696621824 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21d80093a0 con 0x7f21e8156100 2022-06-27T11:48:02.987 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.971+0000 7f21edcc7700 1 --2- 172.21.15.195:0/696621824 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f21e806a440 0x7f21e814ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T11:48:02.988 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.971+0000 7f21ed4c6700 1 --2- 172.21.15.195:0/696621824 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f21e8156100 0x7f21e81550c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f21e806cb90 tx=0x7f21d8016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:02.988 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.971+0000 7f21d5ffb700 1 -- 172.21.15.195:0/696621824 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d8003880 con 0x7f21e8156100 2022-06-27T11:48:02.988 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.971+0000 7f21d5ffb700 1 -- 172.21.15.195:0/696621824 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21d80039e0 con 0x7f21e8156100 2022-06-27T11:48:02.988 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.971+0000 7f21d5ffb700 1 -- 172.21.15.195:0/696621824 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d8003dc0 con 0x7f21e8156100 2022-06-27T11:48:02.989 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.971+0000 7f21eff2b700 1 -- 172.21.15.195:0/696621824 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21e806dd30 con 0x7f21e8156100 2022-06-27T11:48:02.989 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.971+0000 7f21eff2b700 1 -- 172.21.15.195:0/696621824 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21e8166d90 con 0x7f21e8156100 2022-06-27T11:48:02.989 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.972+0000 7f21eff2b700 1 -- 172.21.15.195:0/696621824 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21e80c8e50 con 0x7f21e8156100 2022-06-27T11:48:02.989 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.972+0000 7f21d5ffb700 1 -- 172.21.15.195:0/696621824 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f21d802b030 con 0x7f21e8156100 2022-06-27T11:48:02.990 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.973+0000 7f21d5ffb700 1 --2- 172.21.15.195:0/696621824 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f21c8063d30 0x7f21c80661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:02.990 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.973+0000 7f21edcc7700 1 --2- 172.21.15.195:0/696621824 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f21c8063d30 0x7f21c80661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:02.990 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.973+0000 7f21d5ffb700 1 -- 172.21.15.195:0/696621824 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f21d8082310 con 0x7f21e8156100 2022-06-27T11:48:02.990 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.973+0000 7f21edcc7700 1 --2- 172.21.15.195:0/696621824 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f21c8063d30 0x7f21c80661e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f21e400aa50 tx=0x7f21e4008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:02.991 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:02.977+0000 7f21d5ffb700 1 -- 172.21.15.195:0/696621824 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f21d8050f00 con 0x7f21e8156100 2022-06-27T11:48:03.157 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.158+0000 7f21eff2b700 1 -- 172.21.15.195:0/696621824 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e80722c0 con 0x7f21e8156100 2022-06-27T11:48:03.159 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.159+0000 7f21d5ffb700 1 -- 172.21.15.195:0/696621824 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16914 (secure 0 0 0) 0x7f21d8029090 con 0x7f21e8156100 2022-06-27T11:48:03.159 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:48:03.160 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":21,"fsid":"81b3a1d3-6a2b-47e8-8d69-7027cfeae7af","created":"2022-06-27T11:47:35.094715+0000","modified":"2022-06-27T11:47:53.359270+0000","last_up_change":"2022-06-27T11:47:44.777459+0000","last_in_change":"2022-06-27T11:47:37.596818+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2022-06-27T11:47:46.368325+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2022-06-27T11:47:46.386132+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-06-27T11:47:50.901270+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-06-27T11:47:51.314719+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c64bc51f-d4f3-4bc2-806b-da0dc78d51b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6813","nonce":94217},{"type":"v1","addr":"172.21.15.195:6815","nonce":94217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6817","nonce":94217},{"type":"v1","addr":"172.21.15.195:6819","nonce":94217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":94217},{"type":"v1","addr":"172.21.15.195:6823","nonce":94217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6820","nonce":94217},{"type":"v1","addr":"172.21.15.195:6821","nonce":94217}]},"public_addr":"172.21.15.195:6815/94217","cluster_addr":"172.21.15.195:6819/94217","heartbeat_back_addr":"172.21.15.195:6823/94217","heartbeat_front_addr":"172.21.15.195:6821/94217","state":["exists","up"]},{"osd":1,"uuid":"bea71cdf-aa98-493f-9207-66d886a3e30e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":94223},{"type":"v1","addr":"172.21.15.195:6809","nonce":94223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":94223},{"type":"v1","addr":"172.21.15.195:6811","nonce":94223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6816","nonce":94223},{"type":"v1","addr":"172.21.15.195:6818","nonce":94223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":94223},{"type":"v1","addr":"172.21.15.195:6814","nonce":94223}]},"public_addr":"172.21.15.195:6809/94223","cluster_addr":"172.21.15.195:6811/94223","heartbeat_back_addr":"172.21.15.195:6818/94223","heartbeat_front_addr":"172.21.15.195:6814/94223","state":["exists","up"]},{"osd":2,"uuid":"bf6c87a0-a8ea-497a-9ce1-721285888065","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":94221},{"type":"v1","addr":"172.21.15.195:6801","nonce":94221}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":94221},{"type":"v1","addr":"172.21.15.195:6803","nonce":94221}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":94221},{"type":"v1","addr":"172.21.15.195:6807","nonce":94221}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":94221},{"type":"v1","addr":"172.21.15.195:6805","nonce":94221}]},"public_addr":"172.21.15.195:6801/94221","cluster_addr":"172.21.15.195:6803/94221","heartbeat_back_addr":"172.21.15.195:6807/94221","heartbeat_front_addr":"172.21.15.195:6805/94221","state":["exists","up"]},{"osd":3,"uuid":"e8aa47ce-af8c-47e7-9654-870cd53229f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":94232},{"type":"v1","addr":"172.21.15.195:6825","nonce":94232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":94232},{"type":"v1","addr":"172.21.15.195:6827","nonce":94232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":94232},{"type":"v1","addr":"172.21.15.195:6831","nonce":94232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":94232},{"type":"v1","addr":"172.21.15.195:6829","nonce":94232}]},"public_addr":"172.21.15.195:6825/94232","cluster_addr":"172.21.15.195:6827/94232","heartbeat_back_addr":"172.21.15.195:6831/94232","heartbeat_front_addr":"172.21.15.195:6829/94232","state":["exists","up"]},{"osd":4,"uuid":"99ffa553-f149-443a-98f5-68b10f77e085","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6800","nonce":94308},{"type":"v1","addr":"172.21.15.202:6801","nonce":94308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6802","nonce":94308},{"type":"v1","addr":"172.21.15.202:6803","nonce":94308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6806","nonce":94308},{"type":"v1","addr":"172.21.15.202:6807","nonce":94308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6804","nonce":94308},{"type":"v1","addr":"172.21.15.202:6805","nonce":94308}]},"public_addr":"172.21.15.202:6801/94308","cluster_addr":"172.21.15.202:6803/94308","heartbeat_back_addr":"172.21.15.202:6807/94308","heartbeat_front_addr":"172.21.15.202:6805/94308","state":["exists","up"]},{"osd":5,"uuid":"35409ef0-891b-4c01-82e4-26b8e64f502d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6824","nonce":94325},{"type":"v1","addr":"172.21.15.202:6825","nonce":94325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6826","nonce":94325},{"type":"v1","addr":"172.21.15.202:6827","nonce":94325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6830","nonce":94325},{"type":"v1","addr":"172.21.15.202:6831","nonce":94325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6828","nonce":94325},{"type":"v1","addr":"172.21.15.202:6829","nonce":94325}]},"public_addr":"172.21.15.202:6825/94325","cluster_addr":"172.21.15.202:6827/94325","heartbeat_back_addr":"172.21.15.202:6831/94325","heartbeat_front_addr":"172.21.15.202:6829/94325","state":["exists","up"]},{"osd":6,"uuid":"009124c4-8b5f-47d4-bd20-2c19271a2adc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6809","nonce":94309},{"type":"v1","addr":"172.21.15.202:6811","nonce":94309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6813","nonce":94309},{"type":"v1","addr":"172.21.15.202:6815","nonce":94309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6821","nonce":94309},{"type":"v1","addr":"172.21.15.202:6823","nonce":94309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6817","nonce":94309},{"type":"v1","addr":"172.21.15.202:6819","nonce":94309}]},"public_addr":"172.21.15.202:6811/94309","cluster_addr":"172.21.15.202:6815/94309","heartbeat_back_addr":"172.21.15.202:6823/94309","heartbeat_front_addr":"172.21.15.202:6819/94309","state":["exists","up"]},{"osd":7,"uuid":"7773115e-d3c3-4a0c-8801-d8ce63e0e5a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6808","nonce":94310},{"type":"v1","addr":"172.21.15.202:6810","nonce":94310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6812","nonce":94310},{"type":"v1","addr":"172.21.15.202:6814","nonce":94310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6820","nonce":94310},{"type":"v1","addr":"172.21.15.202:6822","nonce":94310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6816","nonce":94310},{"type":"v1","addr":"172.21.15.202:6818","nonce":94310}]},"public_addr":"172.21.15.202:6810/94310","cluster_addr":"172.21.15.202:6814/94310","heartbeat_back_addr":"172.21.15.202:6822/94310","heartbeat_front_addr":"172.21.15.202:6818/94310","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.776182+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.806922+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.237997+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.784371+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.723615+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.820332+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:43.841414+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.758071+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-06-27T11:48:03.162 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.161+0000 7f21eff2b700 1 -- 172.21.15.195:0/696621824 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f21c8063d30 msgr2=0x7f21c80661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:03.162 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.161+0000 7f21eff2b700 1 --2- 172.21.15.195:0/696621824 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f21c8063d30 0x7f21c80661e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f21e400aa50 tx=0x7f21e4008040 comp rx=0 tx=0).stop 2022-06-27T11:48:03.163 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.161+0000 7f21eff2b700 1 -- 172.21.15.195:0/696621824 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f21e8156100 msgr2=0x7f21e81550c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:03.163 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.161+0000 7f21eff2b700 1 --2- 172.21.15.195:0/696621824 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f21e8156100 0x7f21e81550c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f21e806cb90 tx=0x7f21d8016000 comp rx=0 tx=0).stop 2022-06-27T11:48:03.163 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.161+0000 7f21eff2b700 1 -- 172.21.15.195:0/696621824 shutdown_connections 2022-06-27T11:48:03.164 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.161+0000 7f21eff2b700 1 --2- 172.21.15.195:0/696621824 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f21c8063d30 0x7f21c80661e0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:03.164 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.161+0000 7f21eff2b700 1 --2- 172.21.15.195:0/696621824 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f21e81564d0 0x7f21e8155600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:03.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.161+0000 7f21eff2b700 1 --2- 172.21.15.195:0/696621824 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f21e806a440 0x7f21e814ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:03.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.161+0000 7f21eff2b700 1 --2- 172.21.15.195:0/696621824 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f21e8156100 0x7f21e81550c0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:03.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.161+0000 7f21eff2b700 1 -- 172.21.15.195:0/696621824 >> 172.21.15.195:0/696621824 conn(0x7f21e80c9b00 msgr2=0x7f21e80c0390 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:03.166 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.161+0000 7f21eff2b700 1 -- 172.21.15.195:0/696621824 shutdown_connections 2022-06-27T11:48:03.166 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.161+0000 7f21eff2b700 1 -- 172.21.15.195:0/696621824 wait complete. 2022-06-27T11:48:03.173 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2022-06-27T11:47:46.368325+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2022-06-27T11:47:46.386132+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2022-06-27T11:47:50.901270+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2022-06-27T11:47:51.314719+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-06-27T11:48:03.174 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-06-27T11:48:03.296 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.297+0000 7f84d73a5700 1 Processor -- start 2022-06-27T11:48:03.297 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.297+0000 7f84d73a5700 1 -- start start 2022-06-27T11:48:03.304 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.298+0000 7f84d73a5700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f84d006f640 0x7f84d0156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:03.305 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.298+0000 7f84d73a5700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84d00c1a20 con 0x7f84d00c39e0 2022-06-27T11:48:03.305 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.298+0000 7f84d73a5700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84d00c1130 con 0x7f84d00c4160 2022-06-27T11:48:03.305 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.298+0000 7f84d73a5700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84d006ea90 con 0x7f84d006f640 2022-06-27T11:48:03.306 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.298+0000 7f84d5942700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f84d00c39e0 0x7f84d01535a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:47918/0 (socket says 172.21.15.195:47918) 2022-06-27T11:48:03.306 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.298+0000 7f84d5942700 1 -- 172.21.15.195:0/58729199 learned_addr learned my addr 172.21.15.195:0/58729199 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:48:03.307 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.298+0000 7f84d5141700 1 --2- 172.21.15.195:0/58729199 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f84d006f640 0x7f84d0156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:03.307 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.298+0000 7f84bffff700 1 -- 172.21.15.195:0/58729199 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1169818961 0 0) 0x7f84d00c1a20 con 0x7f84d00c39e0 2022-06-27T11:48:03.308 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.298+0000 7f84bffff700 1 -- 172.21.15.195:0/58729199 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84b8003430 con 0x7f84d00c39e0 2022-06-27T11:48:03.308 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.299+0000 7f84bffff700 1 -- 172.21.15.195:0/58729199 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 806596728 0 0) 0x7f84d00c1130 con 0x7f84d00c4160 2022-06-27T11:48:03.308 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.299+0000 7f84bffff700 1 -- 172.21.15.195:0/58729199 --> v1:172.21.15.202:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84d00c1a20 con 0x7f84d00c4160 2022-06-27T11:48:03.309 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.299+0000 7f84bffff700 1 -- 172.21.15.195:0/58729199 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2916987436 0 0) 0x7f84b8003430 con 0x7f84d00c39e0 2022-06-27T11:48:03.309 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.299+0000 7f84bffff700 1 -- 172.21.15.195:0/58729199 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f84d006f640 msgr2=0x7f84d0156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:03.309 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.299+0000 7f84bffff700 1 --2- 172.21.15.195:0/58729199 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f84d006f640 0x7f84d0156ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:03.310 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.299+0000 7f84bffff700 1 -- 172.21.15.195:0/58729199 >> v1:172.21.15.202:6789/0 conn(0x7f84d00c4160 legacy=0x7f84d00c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:03.310 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.299+0000 7f84bffff700 1 -- 172.21.15.195:0/58729199 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84d006ed70 con 0x7f84d00c39e0 2022-06-27T11:48:03.310 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.299+0000 7f84bffff700 1 -- 172.21.15.195:0/58729199 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7f84c80017f0 con 0x7f84d00c39e0 2022-06-27T11:48:03.311 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.299+0000 7f84bffff700 1 -- 172.21.15.195:0/58729199 >> v1:172.21.15.195:6789/0 conn(0x7f84d00c39e0 legacy=0x7f84d01535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:03.311 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.299+0000 7f84bffff700 1 --2- 172.21.15.195:0/58729199 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f84b8003ae0 0x7f84b8003ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:03.311 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.299+0000 7f84bffff700 1 --2- 172.21.15.195:0/58729199 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f84d006f640 0x7f84b80046d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:03.312 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.299+0000 7f84bffff700 1 --2- 172.21.15.195:0/58729199 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f84b8004c10 0x7f84b8005080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:03.312 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.299+0000 7f84bffff700 1 -- 172.21.15.195:0/58729199 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84d006ea90 con 0x7f84b8003ae0 2022-06-27T11:48:03.313 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.299+0000 7f84bffff700 1 -- 172.21.15.195:0/58729199 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84b800ba00 con 0x7f84b8004c10 2022-06-27T11:48:03.313 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.299+0000 7f84bffff700 1 -- 172.21.15.195:0/58729199 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84b800bbc0 con 0x7f84d006f640 2022-06-27T11:48:03.313 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.299+0000 7f84d5141700 1 --2- 172.21.15.195:0/58729199 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f84b8003ae0 0x7f84b8003ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:03.314 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.299+0000 7f84d5942700 1 --2- 172.21.15.195:0/58729199 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f84b8004c10 0x7f84b8005080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:03.314 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.299+0000 7f84d4940700 1 --2- 172.21.15.195:0/58729199 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f84d006f640 0x7f84b80046d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:03.314 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.299+0000 7f84d5141700 1 -- 172.21.15.195:0/58729199 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f84d006f640 msgr2=0x7f84b80046d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:03.315 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.299+0000 7f84d5141700 1 --2- 172.21.15.195:0/58729199 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f84d006f640 0x7f84b80046d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:03.315 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.299+0000 7f84d5141700 1 -- 172.21.15.195:0/58729199 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f84b8004c10 msgr2=0x7f84b8005080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:03.316 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.299+0000 7f84d5141700 1 --2- 172.21.15.195:0/58729199 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f84b8004c10 0x7f84b8005080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:03.316 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.299+0000 7f84d5141700 1 -- 172.21.15.195:0/58729199 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84b8003430 con 0x7f84b8003ae0 2022-06-27T11:48:03.316 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.299+0000 7f84d4940700 1 --2- 172.21.15.195:0/58729199 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f84d006f640 0x7f84b80046d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T11:48:03.317 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.299+0000 7f84d5141700 1 --2- 172.21.15.195:0/58729199 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f84b8003ae0 0x7f84b8003ed0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f84c0009ac0 tx=0x7f84c0009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d79cb0378182b75 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:03.317 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.299+0000 7f84bffff700 1 -- 172.21.15.195:0/58729199 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c0009470 con 0x7f84b8003ae0 2022-06-27T11:48:03.317 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.299+0000 7f84bffff700 1 -- 172.21.15.195:0/58729199 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84c000e640 con 0x7f84b8003ae0 2022-06-27T11:48:03.318 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.299+0000 7f84bffff700 1 -- 172.21.15.195:0/58729199 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c000e9e0 con 0x7f84b8003ae0 2022-06-27T11:48:03.318 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.300+0000 7f84d73a5700 1 -- 172.21.15.195:0/58729199 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f84b8003ae0 msgr2=0x7f84b8003ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:03.319 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.300+0000 7f84d73a5700 1 --2- 172.21.15.195:0/58729199 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f84b8003ae0 0x7f84b8003ed0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f84c0009ac0 tx=0x7f84c0009060 comp rx=0 tx=0).stop 2022-06-27T11:48:03.319 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.300+0000 7f84d5942700 1 -- 172.21.15.195:0/58729199 reap_dead start 2022-06-27T11:48:03.319 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.300+0000 7f84d73a5700 1 -- 172.21.15.195:0/58729199 shutdown_connections 2022-06-27T11:48:03.320 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.300+0000 7f84d73a5700 1 -- 172.21.15.195:0/58729199 >> 172.21.15.195:0/58729199 conn(0x7f84d00c6350 msgr2=0x7f84d00c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:03.320 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.300+0000 7f84d73a5700 1 -- 172.21.15.195:0/58729199 shutdown_connections 2022-06-27T11:48:03.321 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.300+0000 7f84d73a5700 1 -- 172.21.15.195:0/58729199 wait complete. 2022-06-27T11:48:03.321 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.301+0000 7f84d73a5700 1 Processor -- start 2022-06-27T11:48:03.322 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.301+0000 7f84d73a5700 1 -- start start 2022-06-27T11:48:03.322 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.302+0000 7f84d73a5700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f84d00c8600 0x7f84d01547a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:03.322 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.302+0000 7f84d73a5700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f84d00c89d0 0x7f84d0169c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:03.323 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.302+0000 7f84d73a5700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f84d016a150 0x7f84d0158780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:03.323 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.302+0000 7f84d73a5700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84d00bd280 con 0x7f84d00c89d0 2022-06-27T11:48:03.324 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.302+0000 7f84d73a5700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84d0154ce0 con 0x7f84d016a150 2022-06-27T11:48:03.324 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.302+0000 7f84d73a5700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84d016a5a0 con 0x7f84d00c8600 2022-06-27T11:48:03.324 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.302+0000 7f84d4940700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f84d00c89d0 0x7f84d0169c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:03.325 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.302+0000 7f84d4940700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f84d00c89d0 0x7f84d0169c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41738/0 (socket says 172.21.15.195:41738) 2022-06-27T11:48:03.325 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.302+0000 7f84d4940700 1 -- 172.21.15.195:0/3242232246 learned_addr learned my addr 172.21.15.195:0/3242232246 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:48:03.326 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.302+0000 7f84d5942700 1 --2- 172.21.15.195:0/3242232246 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f84d016a150 0x7f84d0158780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:03.326 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.302+0000 7f84d5141700 1 --2- 172.21.15.195:0/3242232246 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f84d00c8600 0x7f84d01547a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:03.326 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.302+0000 7f84d4940700 1 -- 172.21.15.195:0/3242232246 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f84d00c8600 msgr2=0x7f84d01547a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:03.326 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.302+0000 7f84d4940700 1 --2- 172.21.15.195:0/3242232246 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f84d00c8600 0x7f84d01547a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:03.327 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.302+0000 7f84d4940700 1 -- 172.21.15.195:0/3242232246 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f84d016a150 msgr2=0x7f84d0158780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:03.327 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.302+0000 7f84d4940700 1 --2- 172.21.15.195:0/3242232246 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f84d016a150 0x7f84d0158780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:03.327 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.302+0000 7f84d4940700 1 -- 172.21.15.195:0/3242232246 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84d00bc670 con 0x7f84d00c89d0 2022-06-27T11:48:03.327 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.302+0000 7f84d4940700 1 --2- 172.21.15.195:0/3242232246 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f84d00c89d0 0x7f84d0169c10 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f84cc00e870 tx=0x7f84cc009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:03.328 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.303+0000 7f84bd7fa700 1 -- 172.21.15.195:0/3242232246 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84cc0052e0 con 0x7f84d00c89d0 2022-06-27T11:48:03.328 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.303+0000 7f84bd7fa700 1 -- 172.21.15.195:0/3242232246 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84cc003500 con 0x7f84d00c89d0 2022-06-27T11:48:03.328 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.303+0000 7f84d73a5700 1 -- 172.21.15.195:0/3242232246 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84d00bc950 con 0x7f84d00c89d0 2022-06-27T11:48:03.328 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.303+0000 7f84bd7fa700 1 -- 172.21.15.195:0/3242232246 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84cc0038c0 con 0x7f84d00c89d0 2022-06-27T11:48:03.329 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.303+0000 7f84d73a5700 1 -- 172.21.15.195:0/3242232246 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84d00bcf10 con 0x7f84d00c89d0 2022-06-27T11:48:03.329 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.304+0000 7f84d73a5700 1 -- 172.21.15.195:0/3242232246 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84d00722c0 con 0x7f84d00c89d0 2022-06-27T11:48:03.329 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.304+0000 7f84bd7fa700 1 -- 172.21.15.195:0/3242232246 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f84cc01b030 con 0x7f84d00c89d0 2022-06-27T11:48:03.329 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.304+0000 7f84bd7fa700 1 --2- 172.21.15.195:0/3242232246 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f84b0063d30 0x7f84b00661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:03.330 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.305+0000 7f84bd7fa700 1 -- 172.21.15.195:0/3242232246 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f84cc080030 con 0x7f84d00c89d0 2022-06-27T11:48:03.330 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.305+0000 7f84d5141700 1 --2- 172.21.15.195:0/3242232246 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f84b0063d30 0x7f84b00661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:03.330 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.305+0000 7f84d5141700 1 --2- 172.21.15.195:0/3242232246 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f84b0063d30 0x7f84b00661e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f84c000a010 tx=0x7f84c001a000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:03.331 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.309+0000 7f84bd7fa700 1 -- 172.21.15.195:0/3242232246 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f84cc04ecd0 con 0x7f84d00c89d0 2022-06-27T11:48:03.488 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.488+0000 7f84d73a5700 1 -- 172.21.15.195:0/3242232246 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f84d00bd0d0 con 0x7f84d00c89d0 2022-06-27T11:48:03.488 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.488+0000 7f84bd7fa700 1 -- 172.21.15.195:0/3242232246 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v21) v1 ==== 92+0+10 (secure 0 0 0) 0x7f84cc0522f0 con 0x7f84d00c89d0 2022-06-27T11:48:03.488 INFO:teuthology.orchestra.run.smithi195.stdout:pg_num: 8 2022-06-27T11:48:03.491 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.490+0000 7f84d73a5700 1 -- 172.21.15.195:0/3242232246 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f84b0063d30 msgr2=0x7f84b00661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:03.491 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.490+0000 7f84d73a5700 1 --2- 172.21.15.195:0/3242232246 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f84b0063d30 0x7f84b00661e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f84c000a010 tx=0x7f84c001a000 comp rx=0 tx=0).stop 2022-06-27T11:48:03.491 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.490+0000 7f84d73a5700 1 -- 172.21.15.195:0/3242232246 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f84d00c89d0 msgr2=0x7f84d0169c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:03.492 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.490+0000 7f84d73a5700 1 --2- 172.21.15.195:0/3242232246 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f84d00c89d0 0x7f84d0169c10 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f84cc00e870 tx=0x7f84cc009040 comp rx=0 tx=0).stop 2022-06-27T11:48:03.492 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.490+0000 7f84d73a5700 1 -- 172.21.15.195:0/3242232246 shutdown_connections 2022-06-27T11:48:03.492 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.490+0000 7f84d73a5700 1 --2- 172.21.15.195:0/3242232246 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f84b0063d30 0x7f84b00661e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:03.492 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.490+0000 7f84d73a5700 1 --2- 172.21.15.195:0/3242232246 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f84d016a150 0x7f84d0158780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:03.493 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.490+0000 7f84d73a5700 1 --2- 172.21.15.195:0/3242232246 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f84d00c8600 0x7f84d01547a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:03.493 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.490+0000 7f84d73a5700 1 --2- 172.21.15.195:0/3242232246 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f84d00c89d0 0x7f84d0169c10 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:03.493 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.491+0000 7f84d73a5700 1 -- 172.21.15.195:0/3242232246 >> 172.21.15.195:0/3242232246 conn(0x7f84d00c6350 msgr2=0x7f84d01505e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:03.493 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.491+0000 7f84d73a5700 1 -- 172.21.15.195:0/3242232246 shutdown_connections 2022-06-27T11:48:03.494 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.491+0000 7f84d73a5700 1 -- 172.21.15.195:0/3242232246 wait complete. 2022-06-27T11:48:03.502 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2022-06-27T11:48:03.625 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.626+0000 7f4d0e644700 1 Processor -- start 2022-06-27T11:48:03.627 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.626+0000 7f4d0e644700 1 -- start start 2022-06-27T11:48:03.638 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.627+0000 7f4d0e644700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4d0806a440 0x7f4d080688a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:03.640 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.627+0000 7f4d0e644700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f4d081564d0 0x7f4d08156950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:03.640 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.627+0000 7f4d0e644700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d080c4b10 con 0x7f4d08156100 2022-06-27T11:48:03.640 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.627+0000 7f4d0e644700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d080c3dc0 con 0x7f4d081564d0 2022-06-27T11:48:03.640 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.627+0000 7f4d0e644700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d08068f20 con 0x7f4d0806a440 2022-06-27T11:48:03.640 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.627+0000 7f4d07fff700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f4d08156100 0x7f4d080c4fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:47938/0 (socket says 172.21.15.195:47938) 2022-06-27T11:48:03.641 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.627+0000 7f4d07fff700 1 -- 172.21.15.195:0/1080032829 learned_addr learned my addr 172.21.15.195:0/1080032829 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:48:03.641 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.627+0000 7f4d0cbe1700 1 --2- 172.21.15.195:0/1080032829 >> v2:172.21.15.202:3300/0 conn(0x7f4d081564d0 0x7f4d08156950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:03.641 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.627+0000 7f4d077fe700 1 --2- 172.21.15.195:0/1080032829 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4d0806a440 0x7f4d080688a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:03.641 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.627+0000 7f4d06ffd700 1 -- 172.21.15.195:0/1080032829 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 143053073 0 0) 0x7f4d080c4b10 con 0x7f4d08156100 2022-06-27T11:48:03.642 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.627+0000 7f4d06ffd700 1 -- 172.21.15.195:0/1080032829 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cf4003430 con 0x7f4d08156100 2022-06-27T11:48:03.642 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.627+0000 7f4d0cbe1700 1 -- 172.21.15.195:0/1080032829 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4d0806a440 msgr2=0x7f4d080688a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:03.642 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.627+0000 7f4d0cbe1700 1 --2- 172.21.15.195:0/1080032829 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4d0806a440 0x7f4d080688a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:03.642 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.627+0000 7f4d06ffd700 1 -- 172.21.15.195:0/1080032829 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1347638611 0 0) 0x7f4cf4003430 con 0x7f4d08156100 2022-06-27T11:48:03.643 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.627+0000 7f4d0cbe1700 1 -- 172.21.15.195:0/1080032829 >> v1:172.21.15.195:6789/0 conn(0x7f4d08156100 legacy=0x7f4d080c4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:03.643 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.627+0000 7f4d0cbe1700 1 -- 172.21.15.195:0/1080032829 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d080c56b0 con 0x7f4d081564d0 2022-06-27T11:48:03.643 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.627+0000 7f4d077fe700 1 --2- 172.21.15.195:0/1080032829 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4d0806a440 0x7f4d080688a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:48:03.643 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.628+0000 7f4d0cbe1700 1 --2- 172.21.15.195:0/1080032829 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4d081564d0 0x7f4d08156950 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4d00009e60 tx=0x7f4d00009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df971d47f5adc4a6 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:03.643 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.628+0000 7f4d06ffd700 1 -- 172.21.15.195:0/1080032829 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d0000e070 con 0x7f4d081564d0 2022-06-27T11:48:03.644 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.628+0000 7f4d06ffd700 1 -- 172.21.15.195:0/1080032829 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d000038a0 con 0x7f4d081564d0 2022-06-27T11:48:03.644 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.628+0000 7f4d06ffd700 1 -- 172.21.15.195:0/1080032829 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cf80015f0 con 0x7f4d081564d0 2022-06-27T11:48:03.644 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.628+0000 7f4d0e644700 1 -- 172.21.15.195:0/1080032829 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4d081564d0 msgr2=0x7f4d08156950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:03.644 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.628+0000 7f4d0e644700 1 --2- 172.21.15.195:0/1080032829 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4d081564d0 0x7f4d08156950 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4d00009e60 tx=0x7f4d00009780 comp rx=0 tx=0).stop 2022-06-27T11:48:03.644 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.628+0000 7f4d0e644700 1 -- 172.21.15.195:0/1080032829 shutdown_connections 2022-06-27T11:48:03.645 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.628+0000 7f4d0e644700 1 --2- 172.21.15.195:0/1080032829 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4d081564d0 0x7f4d08156950 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:03.645 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.628+0000 7f4d0e644700 1 --2- 172.21.15.195:0/1080032829 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4d0806a440 0x7f4d080688a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:03.645 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.628+0000 7f4d0e644700 1 -- 172.21.15.195:0/1080032829 >> 172.21.15.195:0/1080032829 conn(0x7f4d080c9b00 msgr2=0x7f4d080c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:03.646 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.628+0000 7f4d0e644700 1 -- 172.21.15.195:0/1080032829 shutdown_connections 2022-06-27T11:48:03.646 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.628+0000 7f4d0e644700 1 -- 172.21.15.195:0/1080032829 wait complete. 2022-06-27T11:48:03.646 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.629+0000 7f4d0e644700 1 Processor -- start 2022-06-27T11:48:03.646 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.630+0000 7f4d0e644700 1 -- start start 2022-06-27T11:48:03.647 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.630+0000 7f4d0e644700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4d0806a440 0x7f4d081490c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:03.647 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.630+0000 7f4d0e644700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4d08156100 0x7f4d081551d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:03.647 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.630+0000 7f4d0e644700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4d081564d0 0x7f4d08155710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:03.647 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.630+0000 7f4d0e644700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d081518c0 con 0x7f4d08156100 2022-06-27T11:48:03.647 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.630+0000 7f4d0e644700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d08152400 con 0x7f4d0806a440 2022-06-27T11:48:03.648 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.630+0000 7f4d0e644700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d08152b00 con 0x7f4d081564d0 2022-06-27T11:48:03.648 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.630+0000 7f4d077fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4d08156100 0x7f4d081551d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:03.648 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.630+0000 7f4d077fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4d08156100 0x7f4d081551d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41758/0 (socket says 172.21.15.195:41758) 2022-06-27T11:48:03.648 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.630+0000 7f4d077fe700 1 -- 172.21.15.195:0/2489542835 learned_addr learned my addr 172.21.15.195:0/2489542835 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:48:03.649 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.631+0000 7f4d0cbe1700 1 --2- 172.21.15.195:0/2489542835 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4d081564d0 0x7f4d08155710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:03.649 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.631+0000 7f4d07fff700 1 --2- 172.21.15.195:0/2489542835 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4d0806a440 0x7f4d081490c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:03.649 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.631+0000 7f4d077fe700 1 -- 172.21.15.195:0/2489542835 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4d081564d0 msgr2=0x7f4d08155710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:03.649 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.631+0000 7f4d077fe700 1 --2- 172.21.15.195:0/2489542835 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4d081564d0 0x7f4d08155710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:03.650 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.631+0000 7f4d077fe700 1 -- 172.21.15.195:0/2489542835 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4d0806a440 msgr2=0x7f4d081490c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:03.650 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.631+0000 7f4d077fe700 1 --2- 172.21.15.195:0/2489542835 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4d0806a440 0x7f4d081490c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:03.650 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.631+0000 7f4d077fe700 1 -- 172.21.15.195:0/2489542835 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cfc0092e0 con 0x7f4d08156100 2022-06-27T11:48:03.650 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.631+0000 7f4d077fe700 1 --2- 172.21.15.195:0/2489542835 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4d08156100 0x7f4d081551d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f4cfc00eaa0 tx=0x7f4cfc009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:03.650 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.631+0000 7f4ceffff700 1 -- 172.21.15.195:0/2489542835 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cfc00c920 con 0x7f4d08156100 2022-06-27T11:48:03.651 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.631+0000 7f4ceffff700 1 -- 172.21.15.195:0/2489542835 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cfc00ca80 con 0x7f4d08156100 2022-06-27T11:48:03.651 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.631+0000 7f4d0e644700 1 -- 172.21.15.195:0/2489542835 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d00009a20 con 0x7f4d08156100 2022-06-27T11:48:03.651 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.631+0000 7f4ceffff700 1 -- 172.21.15.195:0/2489542835 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cfc00ce00 con 0x7f4d08156100 2022-06-27T11:48:03.651 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.631+0000 7f4d0e644700 1 -- 172.21.15.195:0/2489542835 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d08151db0 con 0x7f4d08156100 2022-06-27T11:48:03.652 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.632+0000 7f4d0e644700 1 -- 172.21.15.195:0/2489542835 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d08152790 con 0x7f4d08156100 2022-06-27T11:48:03.652 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.632+0000 7f4ceffff700 1 -- 172.21.15.195:0/2489542835 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f4cfc023070 con 0x7f4d08156100 2022-06-27T11:48:03.652 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.633+0000 7f4ceffff700 1 --2- 172.21.15.195:0/2489542835 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4ce8063d30 0x7f4ce80661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:03.653 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.633+0000 7f4ceffff700 1 -- 172.21.15.195:0/2489542835 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f4cfc080dc0 con 0x7f4d08156100 2022-06-27T11:48:03.653 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.633+0000 7f4d07fff700 1 --2- 172.21.15.195:0/2489542835 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4ce8063d30 0x7f4ce80661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:03.654 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.633+0000 7f4d07fff700 1 --2- 172.21.15.195:0/2489542835 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4ce8063d30 0x7f4ce80661e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4cf80017a0 tx=0x7f4cf8009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:03.654 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.637+0000 7f4ceffff700 1 -- 172.21.15.195:0/2489542835 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f4cfc04f950 con 0x7f4d08156100 2022-06-27T11:48:03.815 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.815+0000 7f4d0e644700 1 -- 172.21.15.195:0/2489542835 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4d080722c0 con 0x7f4d08156100 2022-06-27T11:48:03.815 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.815+0000 7f4ceffff700 1 -- 172.21.15.195:0/2489542835 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4cfc052f70 con 0x7f4d08156100 2022-06-27T11:48:03.815 INFO:teuthology.orchestra.run.smithi195.stdout:pg_num: 1 2022-06-27T11:48:03.818 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.817+0000 7f4d0e644700 1 -- 172.21.15.195:0/2489542835 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4ce8063d30 msgr2=0x7f4ce80661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:03.819 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.817+0000 7f4d0e644700 1 --2- 172.21.15.195:0/2489542835 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4ce8063d30 0x7f4ce80661e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4cf80017a0 tx=0x7f4cf8009040 comp rx=0 tx=0).stop 2022-06-27T11:48:03.819 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.817+0000 7f4d0e644700 1 -- 172.21.15.195:0/2489542835 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4d08156100 msgr2=0x7f4d081551d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:03.820 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.817+0000 7f4d0e644700 1 --2- 172.21.15.195:0/2489542835 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4d08156100 0x7f4d081551d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f4cfc00eaa0 tx=0x7f4cfc009060 comp rx=0 tx=0).stop 2022-06-27T11:48:03.820 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.817+0000 7f4d0e644700 1 -- 172.21.15.195:0/2489542835 shutdown_connections 2022-06-27T11:48:03.820 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.817+0000 7f4d0e644700 1 --2- 172.21.15.195:0/2489542835 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4ce8063d30 0x7f4ce80661e0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:03.821 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.817+0000 7f4d0e644700 1 --2- 172.21.15.195:0/2489542835 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4d081564d0 0x7f4d08155710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:03.821 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.817+0000 7f4d0e644700 1 --2- 172.21.15.195:0/2489542835 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4d0806a440 0x7f4d081490c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:03.822 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.817+0000 7f4d0e644700 1 --2- 172.21.15.195:0/2489542835 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4d08156100 0x7f4d081551d0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:03.822 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.817+0000 7f4d0e644700 1 -- 172.21.15.195:0/2489542835 >> 172.21.15.195:0/2489542835 conn(0x7f4d080c9b00 msgr2=0x7f4d080c0750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:03.822 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.818+0000 7f4d0e644700 1 -- 172.21.15.195:0/2489542835 shutdown_connections 2022-06-27T11:48:03.823 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.818+0000 7f4d0e644700 1 -- 172.21.15.195:0/2489542835 wait complete. 2022-06-27T11:48:03.830 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-06-27T11:48:03.926 INFO:tasks.ceph.mds.d.smithi202.stderr:2022-06-27T11:48:03.926+0000 7f1e2af35700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2022-06-27T11:48:03.958 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.955+0000 7ff8f60b0700 1 Processor -- start 2022-06-27T11:48:03.959 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.955+0000 7ff8f60b0700 1 -- start start 2022-06-27T11:48:03.959 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.956+0000 7ff8f60b0700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff8f006a440 0x7ff8f00688a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:03.959 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.956+0000 7ff8f60b0700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7ff8f01564d0 0x7ff8f0156950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:03.959 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.956+0000 7ff8f60b0700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8f00c4b10 con 0x7ff8f0156100 2022-06-27T11:48:03.960 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.956+0000 7ff8f60b0700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8f00c3dc0 con 0x7ff8f01564d0 2022-06-27T11:48:03.960 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.956+0000 7ff8f60b0700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f0068f20 con 0x7ff8f006a440 2022-06-27T11:48:03.960 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.956+0000 7ff8effff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7ff8f01564d0 0x7ff8f0156950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:03.960 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.956+0000 7ff8ef7fe700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7ff8f0156100 0x7ff8f00c4fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:47952/0 (socket says 172.21.15.195:47952) 2022-06-27T11:48:03.961 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.956+0000 7ff8ef7fe700 1 -- 172.21.15.195:0/3208182314 learned_addr learned my addr 172.21.15.195:0/3208182314 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:48:03.961 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.957+0000 7ff8eeffd700 1 --2- 172.21.15.195:0/3208182314 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff8f006a440 0x7ff8f00688a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:03.961 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.957+0000 7ff8effff700 1 -- 172.21.15.195:0/3208182314 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff8f006a440 msgr2=0x7ff8f00688a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:03.961 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.957+0000 7ff8effff700 1 --2- 172.21.15.195:0/3208182314 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff8f006a440 0x7ff8f00688a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:03.962 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.957+0000 7ff8effff700 1 -- 172.21.15.195:0/3208182314 >> v1:172.21.15.195:6789/0 conn(0x7ff8f0156100 legacy=0x7ff8f00c4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:03.962 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.957+0000 7ff8effff700 1 -- 172.21.15.195:0/3208182314 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8f00c56b0 con 0x7ff8f01564d0 2022-06-27T11:48:03.962 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.957+0000 7ff8ee7fc700 1 -- 172.21.15.195:0/3208182314 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4275117102 0 0) 0x7ff8f00c4b10 con 0x7ff8f0156100 2022-06-27T11:48:03.963 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.957+0000 7ff8effff700 1 --2- 172.21.15.195:0/3208182314 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff8f01564d0 0x7ff8f0156950 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff8e8009e60 tx=0x7ff8e8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e3c74b78d34fa64 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:03.963 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.957+0000 7ff8ee7fc700 1 -- 172.21.15.195:0/3208182314 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8e800e070 con 0x7ff8f01564d0 2022-06-27T11:48:03.964 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.957+0000 7ff8ee7fc700 1 -- 172.21.15.195:0/3208182314 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8e800bc30 con 0x7ff8f01564d0 2022-06-27T11:48:03.964 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.957+0000 7ff8ee7fc700 1 -- 172.21.15.195:0/3208182314 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8e8012430 con 0x7ff8f01564d0 2022-06-27T11:48:03.965 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.957+0000 7ff8f60b0700 1 -- 172.21.15.195:0/3208182314 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff8f01564d0 msgr2=0x7ff8f0156950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:03.965 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.957+0000 7ff8f60b0700 1 --2- 172.21.15.195:0/3208182314 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff8f01564d0 0x7ff8f0156950 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff8e8009e60 tx=0x7ff8e8009780 comp rx=0 tx=0).stop 2022-06-27T11:48:03.965 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.957+0000 7ff8f60b0700 1 -- 172.21.15.195:0/3208182314 shutdown_connections 2022-06-27T11:48:03.965 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.957+0000 7ff8f60b0700 1 --2- 172.21.15.195:0/3208182314 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff8f01564d0 0x7ff8f0156950 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:03.965 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.957+0000 7ff8f60b0700 1 --2- 172.21.15.195:0/3208182314 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff8f006a440 0x7ff8f00688a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:03.966 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.958+0000 7ff8f60b0700 1 -- 172.21.15.195:0/3208182314 >> 172.21.15.195:0/3208182314 conn(0x7ff8f00c9b00 msgr2=0x7ff8f00c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:03.966 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.958+0000 7ff8f60b0700 1 -- 172.21.15.195:0/3208182314 shutdown_connections 2022-06-27T11:48:03.966 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.958+0000 7ff8f60b0700 1 -- 172.21.15.195:0/3208182314 wait complete. 2022-06-27T11:48:03.966 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.959+0000 7ff8f60b0700 1 Processor -- start 2022-06-27T11:48:03.967 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.959+0000 7ff8f60b0700 1 -- start start 2022-06-27T11:48:03.970 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.959+0000 7ff8f60b0700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff8f006a440 0x7ff8f014a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:03.970 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.959+0000 7ff8f60b0700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff8f0156100 0x7ff8f014af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:03.970 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.959+0000 7ff8f60b0700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff8f01564d0 0x7ff8f00bda30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:03.971 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.959+0000 7ff8f60b0700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f014f650 con 0x7ff8f006a440 2022-06-27T11:48:03.971 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.959+0000 7ff8f60b0700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f00bd5a0 con 0x7ff8f0156100 2022-06-27T11:48:03.971 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.959+0000 7ff8f60b0700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f014f090 con 0x7ff8f01564d0 2022-06-27T11:48:03.971 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.960+0000 7ff8ef7fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff8f006a440 0x7ff8f014a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:03.971 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.960+0000 7ff8ef7fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff8f006a440 0x7ff8f014a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41770/0 (socket says 172.21.15.195:41770) 2022-06-27T11:48:03.972 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.960+0000 7ff8ef7fe700 1 -- 172.21.15.195:0/3616192407 learned_addr learned my addr 172.21.15.195:0/3616192407 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:48:03.972 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.960+0000 7ff8eeffd700 1 --2- 172.21.15.195:0/3616192407 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff8f0156100 0x7ff8f014af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:03.972 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.960+0000 7ff8effff700 1 --2- 172.21.15.195:0/3616192407 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff8f01564d0 0x7ff8f00bda30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:03.972 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.960+0000 7ff8ef7fe700 1 -- 172.21.15.195:0/3616192407 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff8f01564d0 msgr2=0x7ff8f00bda30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:03.973 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.960+0000 7ff8ef7fe700 1 --2- 172.21.15.195:0/3616192407 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff8f01564d0 0x7ff8f00bda30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:03.973 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.960+0000 7ff8ef7fe700 1 -- 172.21.15.195:0/3616192407 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff8f0156100 msgr2=0x7ff8f014af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:03.973 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.960+0000 7ff8ef7fe700 1 --2- 172.21.15.195:0/3616192407 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff8f0156100 0x7ff8f014af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:03.974 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.960+0000 7ff8ef7fe700 1 -- 172.21.15.195:0/3616192407 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8e8009a20 con 0x7ff8f006a440 2022-06-27T11:48:03.975 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.960+0000 7ff8eeffd700 1 --2- 172.21.15.195:0/3616192407 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff8f0156100 0x7ff8f014af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T11:48:03.975 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.960+0000 7ff8ef7fe700 1 --2- 172.21.15.195:0/3616192407 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff8f006a440 0x7ff8f014a9d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff8e400c810 tx=0x7ff8e4007b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:03.975 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.960+0000 7ff8d77fe700 1 -- 172.21.15.195:0/3616192407 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8e40072e0 con 0x7ff8f006a440 2022-06-27T11:48:03.976 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.960+0000 7ff8d77fe700 1 -- 172.21.15.195:0/3616192407 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8e4007440 con 0x7ff8f006a440 2022-06-27T11:48:03.976 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.960+0000 7ff8d77fe700 1 -- 172.21.15.195:0/3616192407 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8e4005900 con 0x7ff8f006a440 2022-06-27T11:48:03.976 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.960+0000 7ff8f60b0700 1 -- 172.21.15.195:0/3616192407 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8f014f370 con 0x7ff8f006a440 2022-06-27T11:48:03.977 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.960+0000 7ff8f60b0700 1 -- 172.21.15.195:0/3616192407 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8f00bd230 con 0x7ff8f006a440 2022-06-27T11:48:03.977 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.961+0000 7ff8f60b0700 1 -- 172.21.15.195:0/3616192407 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8f00c8e50 con 0x7ff8f006a440 2022-06-27T11:48:03.977 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.961+0000 7ff8d77fe700 1 -- 172.21.15.195:0/3616192407 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7ff8e4019030 con 0x7ff8f006a440 2022-06-27T11:48:03.977 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.962+0000 7ff8d77fe700 1 --2- 172.21.15.195:0/3616192407 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff8d0063d00 0x7ff8d00661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:03.978 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.962+0000 7ff8d77fe700 1 -- 172.21.15.195:0/3616192407 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7ff8e407fbb0 con 0x7ff8f006a440 2022-06-27T11:48:03.978 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.962+0000 7ff8eeffd700 1 --2- 172.21.15.195:0/3616192407 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff8d0063d00 0x7ff8d00661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:03.978 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.962+0000 7ff8eeffd700 1 --2- 172.21.15.195:0/3616192407 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff8d0063d00 0x7ff8d00661b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff8e0006fd0 tx=0x7ff8e0008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:03.978 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:03.966+0000 7ff8d77fe700 1 -- 172.21.15.195:0/3616192407 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7ff8e4086080 con 0x7ff8f006a440 2022-06-27T11:48:04.144 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.144+0000 7ff8f60b0700 1 -- 172.21.15.195:0/3616192407 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff8f00722c0 con 0x7ff8f006a440 2022-06-27T11:48:04.146 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.144+0000 7ff8d77fe700 1 -- 172.21.15.195:0/3616192407 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff8e4017020 con 0x7ff8f006a440 2022-06-27T11:48:04.146 INFO:teuthology.orchestra.run.smithi195.stdout:pg_num: 32 2022-06-27T11:48:04.148 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.147+0000 7ff8f60b0700 1 -- 172.21.15.195:0/3616192407 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff8d0063d00 msgr2=0x7ff8d00661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:04.149 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.147+0000 7ff8f60b0700 1 --2- 172.21.15.195:0/3616192407 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff8d0063d00 0x7ff8d00661b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff8e0006fd0 tx=0x7ff8e0008040 comp rx=0 tx=0).stop 2022-06-27T11:48:04.149 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.147+0000 7ff8f60b0700 1 -- 172.21.15.195:0/3616192407 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff8f006a440 msgr2=0x7ff8f014a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:04.150 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.147+0000 7ff8f60b0700 1 --2- 172.21.15.195:0/3616192407 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff8f006a440 0x7ff8f014a9d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff8e400c810 tx=0x7ff8e4007b40 comp rx=0 tx=0).stop 2022-06-27T11:48:04.150 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.147+0000 7ff8f60b0700 1 -- 172.21.15.195:0/3616192407 shutdown_connections 2022-06-27T11:48:04.150 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.147+0000 7ff8f60b0700 1 --2- 172.21.15.195:0/3616192407 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff8d0063d00 0x7ff8d00661b0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:04.151 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.147+0000 7ff8f60b0700 1 --2- 172.21.15.195:0/3616192407 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff8f006a440 0x7ff8f014a9d0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:04.151 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.147+0000 7ff8f60b0700 1 --2- 172.21.15.195:0/3616192407 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff8f01564d0 0x7ff8f00bda30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:04.151 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.147+0000 7ff8f60b0700 1 --2- 172.21.15.195:0/3616192407 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff8f0156100 0x7ff8f014af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:04.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.147+0000 7ff8f60b0700 1 -- 172.21.15.195:0/3616192407 >> 172.21.15.195:0/3616192407 conn(0x7ff8f00c9b00 msgr2=0x7ff8f006e640 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:04.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.147+0000 7ff8f60b0700 1 -- 172.21.15.195:0/3616192407 shutdown_connections 2022-06-27T11:48:04.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.147+0000 7ff8f60b0700 1 -- 172.21.15.195:0/3616192407 wait complete. 2022-06-27T11:48:04.159 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-06-27T11:48:04.282 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.282+0000 7f759e127700 1 Processor -- start 2022-06-27T11:48:04.282 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.282+0000 7f759e127700 1 -- start start 2022-06-27T11:48:04.286 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.283+0000 7f759e127700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f759806e850 0x7f75981570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:04.286 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.283+0000 7f759e127700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f75980c34a0 0x7f75980c2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:04.286 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.283+0000 7f759e127700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f759806fba0 con 0x7f759806fef0 2022-06-27T11:48:04.287 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.283+0000 7f759e127700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75981575e0 con 0x7f759806e850 2022-06-27T11:48:04.287 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.283+0000 7f759e127700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f759806f5e0 con 0x7f75980c34a0 2022-06-27T11:48:04.287 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.283+0000 7f7596ffd700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f759806fef0 0x7f7598069de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:47966/0 (socket says 172.21.15.195:47966) 2022-06-27T11:48:04.287 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.283+0000 7f7596ffd700 1 -- 172.21.15.195:0/1655220652 learned_addr learned my addr 172.21.15.195:0/1655220652 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:48:04.288 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.283+0000 7f75977fe700 1 --2- 172.21.15.195:0/1655220652 >> v2:172.21.15.202:3300/0 conn(0x7f759806e850 0x7f75981570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:04.288 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.283+0000 7f75967fc700 1 -- 172.21.15.195:0/1655220652 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1821371470 0 0) 0x7f759806fba0 con 0x7f759806fef0 2022-06-27T11:48:04.289 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.283+0000 7f7597fff700 1 --2- 172.21.15.195:0/1655220652 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f75980c34a0 0x7f75980c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:04.289 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.283+0000 7f75967fc700 1 -- 172.21.15.195:0/1655220652 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7578003430 con 0x7f759806fef0 2022-06-27T11:48:04.289 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.284+0000 7f75967fc700 1 -- 172.21.15.195:0/1655220652 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 93634600 0 0) 0x7f7578003430 con 0x7f759806fef0 2022-06-27T11:48:04.290 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.284+0000 7f75967fc700 1 -- 172.21.15.195:0/1655220652 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f75980c34a0 msgr2=0x7f75980c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:04.290 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.284+0000 7f75967fc700 1 --2- 172.21.15.195:0/1655220652 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f75980c34a0 0x7f75980c2920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:04.290 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.284+0000 7f75967fc700 1 -- 172.21.15.195:0/1655220652 >> v2:172.21.15.202:3300/0 conn(0x7f759806e850 msgr2=0x7f75981570a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:04.290 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.284+0000 7f75967fc700 1 --2- 172.21.15.195:0/1655220652 >> v2:172.21.15.202:3300/0 conn(0x7f759806e850 0x7f75981570a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:04.291 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.284+0000 7f75967fc700 1 -- 172.21.15.195:0/1655220652 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f759806f8c0 con 0x7f759806fef0 2022-06-27T11:48:04.291 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.284+0000 7f75967fc700 1 -- 172.21.15.195:0/1655220652 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7f7580001470 con 0x7f759806fef0 2022-06-27T11:48:04.291 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.284+0000 7f75967fc700 1 -- 172.21.15.195:0/1655220652 >> v1:172.21.15.195:6789/0 conn(0x7f759806fef0 legacy=0x7f7598069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:04.291 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.284+0000 7f75967fc700 1 --2- 172.21.15.195:0/1655220652 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7578003920 0x7f7578003d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:04.292 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.284+0000 7f75967fc700 1 --2- 172.21.15.195:0/1655220652 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f75980c34a0 0x7f75780046d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:04.292 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.284+0000 7f75967fc700 1 --2- 172.21.15.195:0/1655220652 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7578004c10 0x7f7578005080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:04.292 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.284+0000 7f75967fc700 1 -- 172.21.15.195:0/1655220652 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75981575e0 con 0x7f7578004c10 2022-06-27T11:48:04.292 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.284+0000 7f75967fc700 1 -- 172.21.15.195:0/1655220652 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f759806f5e0 con 0x7f7578003920 2022-06-27T11:48:04.292 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.284+0000 7f75967fc700 1 -- 172.21.15.195:0/1655220652 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f757800b920 con 0x7f75980c34a0 2022-06-27T11:48:04.293 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.284+0000 7f7597fff700 1 --2- 172.21.15.195:0/1655220652 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7578004c10 0x7f7578005080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:04.293 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.284+0000 7f7596ffd700 1 --2- 172.21.15.195:0/1655220652 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f75980c34a0 0x7f75780046d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:04.293 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.284+0000 7f75977fe700 1 --2- 172.21.15.195:0/1655220652 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7578003920 0x7f7578003d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:04.293 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.284+0000 7f7597fff700 1 -- 172.21.15.195:0/1655220652 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f75980c34a0 msgr2=0x7f75780046d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:04.294 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.284+0000 7f7597fff700 1 --2- 172.21.15.195:0/1655220652 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f75980c34a0 0x7f75780046d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:04.294 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.284+0000 7f7597fff700 1 -- 172.21.15.195:0/1655220652 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7578003920 msgr2=0x7f7578003d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:04.294 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.284+0000 7f7597fff700 1 --2- 172.21.15.195:0/1655220652 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7578003920 0x7f7578003d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:04.294 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.284+0000 7f7597fff700 1 -- 172.21.15.195:0/1655220652 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7578003430 con 0x7f7578004c10 2022-06-27T11:48:04.295 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.284+0000 7f7597fff700 1 --2- 172.21.15.195:0/1655220652 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7578004c10 0x7f7578005080 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f7588009ac0 tx=0x7f7588009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a445f9e34b22c53 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:04.295 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.284+0000 7f75967fc700 1 -- 172.21.15.195:0/1655220652 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7588009470 con 0x7f7578004c10 2022-06-27T11:48:04.295 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.284+0000 7f75967fc700 1 -- 172.21.15.195:0/1655220652 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f758800e5e0 con 0x7f7578004c10 2022-06-27T11:48:04.296 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.284+0000 7f75967fc700 1 -- 172.21.15.195:0/1655220652 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f758800e980 con 0x7f7578004c10 2022-06-27T11:48:04.296 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.285+0000 7f759e127700 1 -- 172.21.15.195:0/1655220652 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7578004c10 msgr2=0x7f7578005080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:04.297 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.285+0000 7f759e127700 1 --2- 172.21.15.195:0/1655220652 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7578004c10 0x7f7578005080 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f7588009ac0 tx=0x7f7588009060 comp rx=0 tx=0).stop 2022-06-27T11:48:04.297 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.285+0000 7f7597fff700 1 -- 172.21.15.195:0/1655220652 reap_dead start 2022-06-27T11:48:04.297 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.285+0000 7f759e127700 1 -- 172.21.15.195:0/1655220652 shutdown_connections 2022-06-27T11:48:04.298 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.285+0000 7f759e127700 1 -- 172.21.15.195:0/1655220652 >> 172.21.15.195:0/1655220652 conn(0x7f75980c6350 msgr2=0x7f75980c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:04.298 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.285+0000 7f759e127700 1 -- 172.21.15.195:0/1655220652 shutdown_connections 2022-06-27T11:48:04.299 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.285+0000 7f759e127700 1 -- 172.21.15.195:0/1655220652 wait complete. 2022-06-27T11:48:04.300 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.286+0000 7f759e127700 1 Processor -- start 2022-06-27T11:48:04.300 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.286+0000 7f759e127700 1 -- start start 2022-06-27T11:48:04.307 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.287+0000 7f759e127700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7578004c10 0x7f759814ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:04.307 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.287+0000 7f759e127700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f759814f510 0x7f7598162500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:04.307 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.287+0000 7f759e127700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7598162a40 0x7f75980c7c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:04.308 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.287+0000 7f759e127700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75980be200 con 0x7f7598162a40 2022-06-27T11:48:04.308 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.287+0000 7f759e127700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7598161dd0 con 0x7f759814f510 2022-06-27T11:48:04.308 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.287+0000 7f759e127700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7598162070 con 0x7f7578004c10 2022-06-27T11:48:04.308 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.287+0000 7f7597fff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7598162a40 0x7f75980c7c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:04.309 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.287+0000 7f7597fff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7598162a40 0x7f75980c7c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41794/0 (socket says 172.21.15.195:41794) 2022-06-27T11:48:04.309 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.287+0000 7f7597fff700 1 -- 172.21.15.195:0/1603471132 learned_addr learned my addr 172.21.15.195:0/1603471132 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:48:04.309 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.287+0000 7f75977fe700 1 --2- 172.21.15.195:0/1603471132 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7578004c10 0x7f759814ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:04.309 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.287+0000 7f7596ffd700 1 --2- 172.21.15.195:0/1603471132 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f759814f510 0x7f7598162500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:04.309 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.287+0000 7f7597fff700 1 -- 172.21.15.195:0/1603471132 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7578004c10 msgr2=0x7f759814ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:04.310 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.287+0000 7f7597fff700 1 --2- 172.21.15.195:0/1603471132 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7578004c10 0x7f759814ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:04.310 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.287+0000 7f7597fff700 1 -- 172.21.15.195:0/1603471132 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f759814f510 msgr2=0x7f7598162500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:04.310 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.287+0000 7f7597fff700 1 --2- 172.21.15.195:0/1603471132 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f759814f510 0x7f7598162500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:04.310 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.287+0000 7f7597fff700 1 -- 172.21.15.195:0/1603471132 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75980bd6e0 con 0x7f7598162a40 2022-06-27T11:48:04.311 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.287+0000 7f75977fe700 1 --2- 172.21.15.195:0/1603471132 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7578004c10 0x7f759814ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T11:48:04.311 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.287+0000 7f7596ffd700 1 --2- 172.21.15.195:0/1603471132 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f759814f510 0x7f7598162500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T11:48:04.311 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.288+0000 7f7597fff700 1 --2- 172.21.15.195:0/1603471132 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7598162a40 0x7f75980c7c70 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f75780045f0 tx=0x7f758801a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:04.311 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.288+0000 7f758f7fe700 1 -- 172.21.15.195:0/1603471132 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7588021070 con 0x7f7598162a40 2022-06-27T11:48:04.311 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.288+0000 7f758f7fe700 1 -- 172.21.15.195:0/1603471132 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7588009470 con 0x7f7598162a40 2022-06-27T11:48:04.312 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.288+0000 7f759e127700 1 -- 172.21.15.195:0/1603471132 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75981653a0 con 0x7f7598162a40 2022-06-27T11:48:04.312 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.288+0000 7f758f7fe700 1 -- 172.21.15.195:0/1603471132 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7588003980 con 0x7f7598162a40 2022-06-27T11:48:04.312 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.288+0000 7f759e127700 1 -- 172.21.15.195:0/1603471132 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75980bdc40 con 0x7f7598162a40 2022-06-27T11:48:04.312 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.289+0000 7f759e127700 1 -- 172.21.15.195:0/1603471132 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75980c58c0 con 0x7f7598162a40 2022-06-27T11:48:04.313 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.289+0000 7f758f7fe700 1 -- 172.21.15.195:0/1603471132 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f7588019030 con 0x7f7598162a40 2022-06-27T11:48:04.313 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.289+0000 7f758f7fe700 1 --2- 172.21.15.195:0/1603471132 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f757c063d20 0x7f757c0661d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:04.313 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.290+0000 7f758f7fe700 1 -- 172.21.15.195:0/1603471132 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f7588081e00 con 0x7f7598162a40 2022-06-27T11:48:04.313 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.290+0000 7f75977fe700 1 --2- 172.21.15.195:0/1603471132 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f757c063d20 0x7f757c0661d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:04.313 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.290+0000 7f75977fe700 1 --2- 172.21.15.195:0/1603471132 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f757c063d20 0x7f757c0661d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f75900094d0 tx=0x7f759000b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:04.314 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.294+0000 7f758f7fe700 1 -- 172.21.15.195:0/1603471132 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f75880509f0 con 0x7f7598162a40 2022-06-27T11:48:04.474 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.474+0000 7f759e127700 1 -- 172.21.15.195:0/1603471132 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f75980722c0 con 0x7f7598162a40 2022-06-27T11:48:04.475 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.475+0000 7f758f7fe700 1 -- 172.21.15.195:0/1603471132 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f758802b0a0 con 0x7f7598162a40 2022-06-27T11:48:04.476 INFO:teuthology.orchestra.run.smithi195.stdout:pg_num: 64 2022-06-27T11:48:04.477 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.477+0000 7f759e127700 1 -- 172.21.15.195:0/1603471132 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f757c063d20 msgr2=0x7f757c0661d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:04.477 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.477+0000 7f759e127700 1 --2- 172.21.15.195:0/1603471132 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f757c063d20 0x7f757c0661d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f75900094d0 tx=0x7f759000b040 comp rx=0 tx=0).stop 2022-06-27T11:48:04.477 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.477+0000 7f759e127700 1 -- 172.21.15.195:0/1603471132 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7598162a40 msgr2=0x7f75980c7c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:04.478 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.477+0000 7f759e127700 1 --2- 172.21.15.195:0/1603471132 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7598162a40 0x7f75980c7c70 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f75780045f0 tx=0x7f758801a000 comp rx=0 tx=0).stop 2022-06-27T11:48:04.478 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.477+0000 7f759e127700 1 -- 172.21.15.195:0/1603471132 shutdown_connections 2022-06-27T11:48:04.478 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.477+0000 7f759e127700 1 --2- 172.21.15.195:0/1603471132 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f757c063d20 0x7f757c0661d0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:04.478 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.477+0000 7f759e127700 1 --2- 172.21.15.195:0/1603471132 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7598162a40 0x7f75980c7c70 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:04.479 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.477+0000 7f759e127700 1 --2- 172.21.15.195:0/1603471132 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7578004c10 0x7f759814ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:04.479 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.477+0000 7f759e127700 1 --2- 172.21.15.195:0/1603471132 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f759814f510 0x7f7598162500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:04.479 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.477+0000 7f759e127700 1 -- 172.21.15.195:0/1603471132 >> 172.21.15.195:0/1603471132 conn(0x7f75980c6350 msgr2=0x7f7598156500 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:04.479 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.477+0000 7f759e127700 1 -- 172.21.15.195:0/1603471132 shutdown_connections 2022-06-27T11:48:04.480 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.477+0000 7f759e127700 1 -- 172.21.15.195:0/1603471132 wait complete. 2022-06-27T11:48:04.488 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:48:04.613 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.609+0000 7f0d731ed700 1 Processor -- start 2022-06-27T11:48:04.613 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.609+0000 7f0d731ed700 1 -- start start 2022-06-27T11:48:04.614 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.610+0000 7f0d731ed700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0d6c06e850 0x7f0d6c1570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:04.614 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.610+0000 7f0d731ed700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f0d6c06fef0 0x7f0d6c069de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:04.615 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.610+0000 7f0d731ed700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f0d6c0c34a0 0x7f0d6c06a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:04.615 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.610+0000 7f0d731ed700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d6c1575e0 con 0x7f0d6c0c34a0 2022-06-27T11:48:04.616 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.610+0000 7f0d731ed700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d6c06fba0 con 0x7f0d6c06fef0 2022-06-27T11:48:04.616 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.610+0000 7f0d731ed700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d6c06f5e0 con 0x7f0d6c06e850 2022-06-27T11:48:04.617 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.610+0000 7f0d7178a700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f0d6c0c34a0 0x7f0d6c06a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:04.617 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.610+0000 7f0d7178a700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f0d6c0c34a0 0x7f0d6c06a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41802/0 (socket says 172.21.15.195:41802) 2022-06-27T11:48:04.617 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.610+0000 7f0d7178a700 1 -- 172.21.15.195:0/1371054541 learned_addr learned my addr 172.21.15.195:0/1371054541 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:48:04.618 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.610+0000 7f0d6bfff700 1 --2- 172.21.15.195:0/1371054541 >> v2:172.21.15.202:3300/0 conn(0x7f0d6c06fef0 0x7f0d6c069de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:04.618 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.610+0000 7f0d70f89700 1 --2- 172.21.15.195:0/1371054541 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0d6c06e850 0x7f0d6c1570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:04.619 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.610+0000 7f0d7178a700 1 -- 172.21.15.195:0/1371054541 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0d6c06e850 msgr2=0x7f0d6c1570a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:04.619 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.610+0000 7f0d7178a700 1 --2- 172.21.15.195:0/1371054541 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0d6c06e850 0x7f0d6c1570a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:04.619 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.610+0000 7f0d7178a700 1 -- 172.21.15.195:0/1371054541 >> v2:172.21.15.202:3300/0 conn(0x7f0d6c06fef0 msgr2=0x7f0d6c069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:04.619 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.610+0000 7f0d7178a700 1 --2- 172.21.15.195:0/1371054541 >> v2:172.21.15.202:3300/0 conn(0x7f0d6c06fef0 0x7f0d6c069de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:04.620 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.610+0000 7f0d7178a700 1 -- 172.21.15.195:0/1371054541 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d6c06f8c0 con 0x7f0d6c0c34a0 2022-06-27T11:48:04.620 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.610+0000 7f0d6bfff700 1 --2- 172.21.15.195:0/1371054541 >> v2:172.21.15.202:3300/0 conn(0x7f0d6c06fef0 0x7f0d6c069de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T11:48:04.620 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.610+0000 7f0d7178a700 1 --2- 172.21.15.195:0/1371054541 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0d6c0c34a0 0x7f0d6c06a320 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c06e240 tx=0x7f0d64009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f604b409859ceb1 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:04.620 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.610+0000 7f0d6b7fe700 1 -- 172.21.15.195:0/1371054541 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d6400e070 con 0x7f0d6c0c34a0 2022-06-27T11:48:04.626 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.610+0000 7f0d6b7fe700 1 -- 172.21.15.195:0/1371054541 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d6400bae0 con 0x7f0d6c0c34a0 2022-06-27T11:48:04.626 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.610+0000 7f0d6b7fe700 1 -- 172.21.15.195:0/1371054541 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d640092e0 con 0x7f0d6c0c34a0 2022-06-27T11:48:04.627 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.611+0000 7f0d731ed700 1 -- 172.21.15.195:0/1371054541 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0d6c0c34a0 msgr2=0x7f0d6c06a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:04.627 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.611+0000 7f0d731ed700 1 --2- 172.21.15.195:0/1371054541 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0d6c0c34a0 0x7f0d6c06a320 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c06e240 tx=0x7f0d64009840 comp rx=0 tx=0).stop 2022-06-27T11:48:04.628 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.611+0000 7f0d731ed700 1 -- 172.21.15.195:0/1371054541 shutdown_connections 2022-06-27T11:48:04.628 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.611+0000 7f0d731ed700 1 --2- 172.21.15.195:0/1371054541 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0d6c0c34a0 0x7f0d6c06a320 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:04.629 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.611+0000 7f0d731ed700 1 --2- 172.21.15.195:0/1371054541 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0d6c06e850 0x7f0d6c1570a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:04.629 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.611+0000 7f0d731ed700 1 --2- 172.21.15.195:0/1371054541 >> v2:172.21.15.202:3300/0 conn(0x7f0d6c06fef0 0x7f0d6c069de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:04.629 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.611+0000 7f0d731ed700 1 -- 172.21.15.195:0/1371054541 >> 172.21.15.195:0/1371054541 conn(0x7f0d6c0c6350 msgr2=0x7f0d6c0c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:04.629 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.611+0000 7f0d731ed700 1 -- 172.21.15.195:0/1371054541 shutdown_connections 2022-06-27T11:48:04.630 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.611+0000 7f0d731ed700 1 -- 172.21.15.195:0/1371054541 wait complete. 2022-06-27T11:48:04.630 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.612+0000 7f0d731ed700 1 Processor -- start 2022-06-27T11:48:04.630 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.612+0000 7f0d731ed700 1 -- start start 2022-06-27T11:48:04.631 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.613+0000 7f0d731ed700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0d6c06e850 0x7f0d6c1558a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:04.631 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.613+0000 7f0d731ed700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0d6c0c34a0 0x7f0d6c0c4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:04.632 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.613+0000 7f0d731ed700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0d6c156a30 0x7f0d6c148ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:04.632 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.613+0000 7f0d731ed700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d6c155410 con 0x7f0d6c0c34a0 2022-06-27T11:48:04.633 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.613+0000 7f0d731ed700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d6c150b20 con 0x7f0d6c06e850 2022-06-27T11:48:04.634 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.613+0000 7f0d731ed700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d6c0c0100 con 0x7f0d6c156a30 2022-06-27T11:48:04.634 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.613+0000 7f0d6bfff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0d6c0c34a0 0x7f0d6c0c4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:04.635 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.613+0000 7f0d6bfff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0d6c0c34a0 0x7f0d6c0c4ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41806/0 (socket says 172.21.15.195:41806) 2022-06-27T11:48:04.636 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.613+0000 7f0d6bfff700 1 -- 172.21.15.195:0/472504971 learned_addr learned my addr 172.21.15.195:0/472504971 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:48:04.636 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.613+0000 7f0d70f89700 1 --2- 172.21.15.195:0/472504971 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0d6c06e850 0x7f0d6c1558a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:04.636 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.613+0000 7f0d7178a700 1 --2- 172.21.15.195:0/472504971 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0d6c156a30 0x7f0d6c148ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:04.637 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.613+0000 7f0d6bfff700 1 -- 172.21.15.195:0/472504971 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0d6c156a30 msgr2=0x7f0d6c148ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:04.637 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.613+0000 7f0d6bfff700 1 --2- 172.21.15.195:0/472504971 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0d6c156a30 0x7f0d6c148ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:04.637 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.613+0000 7f0d6bfff700 1 -- 172.21.15.195:0/472504971 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0d6c06e850 msgr2=0x7f0d6c1558a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:04.638 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.613+0000 7f0d6bfff700 1 --2- 172.21.15.195:0/472504971 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0d6c06e850 0x7f0d6c1558a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:04.638 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.613+0000 7f0d6bfff700 1 -- 172.21.15.195:0/472504971 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d64009ae0 con 0x7f0d6c0c34a0 2022-06-27T11:48:04.638 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.614+0000 7f0d6bfff700 1 --2- 172.21.15.195:0/472504971 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0d6c0c34a0 0x7f0d6c0c4ec0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f0d5800c8b0 tx=0x7f0d58008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:04.639 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.614+0000 7f0d68ff9700 1 -- 172.21.15.195:0/472504971 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d58011600 con 0x7f0d6c0c34a0 2022-06-27T11:48:04.640 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.614+0000 7f0d68ff9700 1 -- 172.21.15.195:0/472504971 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d58011760 con 0x7f0d6c0c34a0 2022-06-27T11:48:04.640 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.614+0000 7f0d731ed700 1 -- 172.21.15.195:0/472504971 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d6c0c0380 con 0x7f0d6c0c34a0 2022-06-27T11:48:04.640 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.614+0000 7f0d68ff9700 1 -- 172.21.15.195:0/472504971 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d58011b20 con 0x7f0d6c0c34a0 2022-06-27T11:48:04.640 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.614+0000 7f0d731ed700 1 -- 172.21.15.195:0/472504971 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d6c0bd980 con 0x7f0d6c0c34a0 2022-06-27T11:48:04.641 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.615+0000 7f0d731ed700 1 -- 172.21.15.195:0/472504971 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d6c1555c0 con 0x7f0d6c0c34a0 2022-06-27T11:48:04.641 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.615+0000 7f0d68ff9700 1 -- 172.21.15.195:0/472504971 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f0d58021070 con 0x7f0d6c0c34a0 2022-06-27T11:48:04.641 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.615+0000 7f0d68ff9700 1 --2- 172.21.15.195:0/472504971 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0d54063d00 0x7f0d540661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:04.641 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.616+0000 7f0d70f89700 1 --2- 172.21.15.195:0/472504971 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0d54063d00 0x7f0d540661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:04.642 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.616+0000 7f0d68ff9700 1 -- 172.21.15.195:0/472504971 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f0d580805e0 con 0x7f0d6c0c34a0 2022-06-27T11:48:04.642 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.616+0000 7f0d70f89700 1 --2- 172.21.15.195:0/472504971 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0d54063d00 0x7f0d540661b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0d60006fd0 tx=0x7f0d60008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:04.642 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.620+0000 7f0d68ff9700 1 -- 172.21.15.195:0/472504971 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f0d6c1555c0 con 0x7f0d6c0c34a0 2022-06-27T11:48:04.851 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.849+0000 7f0d731ed700 1 -- 172.21.15.195:0/472504971 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0d6c0722c0 con 0x7f0d6c0c34a0 2022-06-27T11:48:04.852 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.850+0000 7f0d68ff9700 1 -- 172.21.15.195:0/472504971 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6119 (secure 0 0 0) 0x7f0d58015090 con 0x7f0d6c0c34a0 2022-06-27T11:48:04.852 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:48:04.853 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4287,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/864126677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":864126677},{"type":"v1","addr":"172.21.15.202:6833","nonce":864126677}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:47:57.373108+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.202:6835/2038169702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2038169702},{"type":"v1","addr":"172.21.15.202:6835","nonce":2038169702}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:48:04.854 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.852+0000 7f0d731ed700 1 -- 172.21.15.195:0/472504971 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0d54063d00 msgr2=0x7f0d540661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:04.854 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.852+0000 7f0d731ed700 1 --2- 172.21.15.195:0/472504971 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0d54063d00 0x7f0d540661b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0d60006fd0 tx=0x7f0d60008040 comp rx=0 tx=0).stop 2022-06-27T11:48:04.854 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.852+0000 7f0d731ed700 1 -- 172.21.15.195:0/472504971 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0d6c0c34a0 msgr2=0x7f0d6c0c4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:04.854 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.852+0000 7f0d731ed700 1 --2- 172.21.15.195:0/472504971 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0d6c0c34a0 0x7f0d6c0c4ec0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f0d5800c8b0 tx=0x7f0d58008040 comp rx=0 tx=0).stop 2022-06-27T11:48:04.855 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.852+0000 7f0d731ed700 1 -- 172.21.15.195:0/472504971 shutdown_connections 2022-06-27T11:48:04.855 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.852+0000 7f0d731ed700 1 --2- 172.21.15.195:0/472504971 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0d54063d00 0x7f0d540661b0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:04.855 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.852+0000 7f0d731ed700 1 --2- 172.21.15.195:0/472504971 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0d6c156a30 0x7f0d6c148ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:04.855 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.852+0000 7f0d731ed700 1 --2- 172.21.15.195:0/472504971 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0d6c06e850 0x7f0d6c1558a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:04.856 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.852+0000 7f0d731ed700 1 --2- 172.21.15.195:0/472504971 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0d6c0c34a0 0x7f0d6c0c4ec0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:04.856 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.852+0000 7f0d731ed700 1 -- 172.21.15.195:0/472504971 >> 172.21.15.195:0/472504971 conn(0x7f0d6c0c6350 msgr2=0x7f0d6c0c7cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:04.856 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.853+0000 7f0d731ed700 1 -- 172.21.15.195:0/472504971 shutdown_connections 2022-06-27T11:48:04.856 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:04.853+0000 7f0d731ed700 1 -- 172.21.15.195:0/472504971 wait complete. 2022-06-27T11:48:04.857 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 8 2022-06-27T11:48:05.471 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-06-27T11:47:52.358095+0000', 'modified': '2022-06-27T11:47:57.373108+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4293}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4293': {'gid': 4293, 'name': 'd', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.202:6835/2038169702', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.202:6834', 'nonce': 2038169702}, {'type': 'v1', 'addr': '172.21.15.202:6835', 'nonce': 2038169702}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [4], 'metadata_pool': 3, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-06-27T11:48:05.471 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-06-27T11:48:05.472 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2022-06-27T11:48:05.597 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.598+0000 7f41137a9700 1 Processor -- start 2022-06-27T11:48:05.599 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.598+0000 7f41137a9700 1 -- start start 2022-06-27T11:48:05.617 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.599+0000 7f41137a9700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f410c06f620 0x7f410c156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:05.617 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.599+0000 7f41137a9700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f410c0c76b0 con 0x7f410c150950 2022-06-27T11:48:05.617 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.599+0000 7f41137a9700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f410c0c8250 con 0x7f410c150580 2022-06-27T11:48:05.619 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.599+0000 7f41137a9700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f410c157430 con 0x7f410c06f620 2022-06-27T11:48:05.619 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.599+0000 7f4111d46700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f410c06f620 0x7f410c156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:05.619 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.599+0000 7f4110d44700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f410c150950 0x7f410c06d6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:48002/0 (socket says 172.21.15.195:48002) 2022-06-27T11:48:05.620 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.599+0000 7f4110d44700 1 -- 172.21.15.195:0/1414772955 learned_addr learned my addr 172.21.15.195:0/1414772955 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:48:05.620 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.599+0000 7f4111d46700 1 -- 172.21.15.195:0/1414772955 >> v1:172.21.15.202:6789/0 conn(0x7f410c150580 legacy=0x7f410c0c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:05.621 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.599+0000 7f4111d46700 1 -- 172.21.15.195:0/1414772955 >> v1:172.21.15.195:6789/0 conn(0x7f410c150950 legacy=0x7f410c06d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:05.621 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.599+0000 7f4111d46700 1 -- 172.21.15.195:0/1414772955 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f410c0bad70 con 0x7f410c06f620 2022-06-27T11:48:05.622 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.599+0000 7f40fbfff700 1 -- 172.21.15.195:0/1414772955 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 716590809 0 0) 0x7f410c0c76b0 con 0x7f410c150950 2022-06-27T11:48:05.622 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.599+0000 7f4111d46700 1 --2- 172.21.15.195:0/1414772955 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f410c06f620 0x7f410c156ef0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4104009e60 tx=0x7f4104009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8a83ba57729b8f1 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:05.623 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.599+0000 7f40fbfff700 1 -- 172.21.15.195:0/1414772955 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f410400e070 con 0x7f410c06f620 2022-06-27T11:48:05.623 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.599+0000 7f40fbfff700 1 -- 172.21.15.195:0/1414772955 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f410400bbf0 con 0x7f410c06f620 2022-06-27T11:48:05.624 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.599+0000 7f40fbfff700 1 -- 172.21.15.195:0/1414772955 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41040038a0 con 0x7f410c06f620 2022-06-27T11:48:05.624 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.600+0000 7f41137a9700 1 -- 172.21.15.195:0/1414772955 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f410c06f620 msgr2=0x7f410c156ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:05.624 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.600+0000 7f41137a9700 1 --2- 172.21.15.195:0/1414772955 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f410c06f620 0x7f410c156ef0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4104009e60 tx=0x7f4104009780 comp rx=0 tx=0).stop 2022-06-27T11:48:05.625 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.600+0000 7f41137a9700 1 -- 172.21.15.195:0/1414772955 shutdown_connections 2022-06-27T11:48:05.626 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.600+0000 7f41137a9700 1 --2- 172.21.15.195:0/1414772955 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f410c06f620 0x7f410c156ef0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:05.626 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.600+0000 7f41137a9700 1 -- 172.21.15.195:0/1414772955 >> 172.21.15.195:0/1414772955 conn(0x7f410c063af0 msgr2=0x7f410c063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:05.626 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.600+0000 7f41137a9700 1 -- 172.21.15.195:0/1414772955 shutdown_connections 2022-06-27T11:48:05.627 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.600+0000 7f41137a9700 1 -- 172.21.15.195:0/1414772955 wait complete. 2022-06-27T11:48:05.627 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.601+0000 7f41137a9700 1 Processor -- start 2022-06-27T11:48:05.627 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.601+0000 7f41137a9700 1 -- start start 2022-06-27T11:48:05.628 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.602+0000 7f41137a9700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f410c06f620 0x7f410c14cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:05.628 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.602+0000 7f41137a9700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f410c150580 0x7f410c15b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:05.629 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.602+0000 7f41137a9700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f410c150950 0x7f410c15bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:05.629 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.602+0000 7f41137a9700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f410c140c30 con 0x7f410c06f620 2022-06-27T11:48:05.630 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.602+0000 7f41137a9700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f410c1494f0 con 0x7f410c150580 2022-06-27T11:48:05.630 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.602+0000 7f41137a9700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f410c15ab80 con 0x7f410c150950 2022-06-27T11:48:05.631 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.602+0000 7f4111545700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f410c06f620 0x7f410c14cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:05.631 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.602+0000 7f4111545700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f410c06f620 0x7f410c14cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41818/0 (socket says 172.21.15.195:41818) 2022-06-27T11:48:05.632 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.602+0000 7f4111545700 1 -- 172.21.15.195:0/4116983290 learned_addr learned my addr 172.21.15.195:0/4116983290 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:48:05.632 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.602+0000 7f4110d44700 1 --2- 172.21.15.195:0/4116983290 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f410c150580 0x7f410c15b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:05.633 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.602+0000 7f4111d46700 1 --2- 172.21.15.195:0/4116983290 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f410c150950 0x7f410c15bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:05.633 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.602+0000 7f4111545700 1 -- 172.21.15.195:0/4116983290 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f410c150950 msgr2=0x7f410c15bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:05.634 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.602+0000 7f4111545700 1 --2- 172.21.15.195:0/4116983290 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f410c150950 0x7f410c15bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:05.634 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.602+0000 7f4111545700 1 -- 172.21.15.195:0/4116983290 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f410c150580 msgr2=0x7f410c15b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:05.634 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.602+0000 7f4111545700 1 --2- 172.21.15.195:0/4116983290 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f410c150580 0x7f410c15b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:05.635 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.602+0000 7f4111545700 1 -- 172.21.15.195:0/4116983290 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4104009a20 con 0x7f410c06f620 2022-06-27T11:48:05.635 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.602+0000 7f4110d44700 1 --2- 172.21.15.195:0/4116983290 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f410c150580 0x7f410c15b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T11:48:05.635 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.602+0000 7f4111d46700 1 --2- 172.21.15.195:0/4116983290 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f410c150950 0x7f410c15bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T11:48:05.636 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.602+0000 7f4111545700 1 --2- 172.21.15.195:0/4116983290 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f410c06f620 0x7f410c14cc30 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f410800c4f0 tx=0x7f4108007e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:05.636 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.603+0000 7f40f97fa700 1 -- 172.21.15.195:0/4116983290 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4108011590 con 0x7f410c06f620 2022-06-27T11:48:05.637 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.603+0000 7f40f97fa700 1 -- 172.21.15.195:0/4116983290 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41080116f0 con 0x7f410c06f620 2022-06-27T11:48:05.637 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.603+0000 7f40f97fa700 1 -- 172.21.15.195:0/4116983290 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4108011ab0 con 0x7f410c06f620 2022-06-27T11:48:05.637 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.603+0000 7f41137a9700 1 -- 172.21.15.195:0/4116983290 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f410c158f20 con 0x7f410c06f620 2022-06-27T11:48:05.638 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.603+0000 7f41137a9700 1 -- 172.21.15.195:0/4116983290 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f410c1594e0 con 0x7f410c06f620 2022-06-27T11:48:05.638 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.604+0000 7f40f97fa700 1 -- 172.21.15.195:0/4116983290 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f4108010050 con 0x7f410c06f620 2022-06-27T11:48:05.638 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.604+0000 7f40f97fa700 1 --2- 172.21.15.195:0/4116983290 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f40ec063d00 0x7f40ec0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:05.639 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.604+0000 7f40f97fa700 1 -- 172.21.15.195:0/4116983290 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f4108080400 con 0x7f410c06f620 2022-06-27T11:48:05.639 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.604+0000 7f4110d44700 1 --2- 172.21.15.195:0/4116983290 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f40ec063d00 0x7f40ec0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:05.640 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.605+0000 7f4110d44700 1 --2- 172.21.15.195:0/4116983290 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f40ec063d00 0x7f40ec0661b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f40fc0076e0 tx=0x7f40fc007bd0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:05.640 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.606+0000 7f411fdb8b80 1 Processor -- start 2022-06-27T11:48:05.640 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.606+0000 7f411fdb8b80 1 -- start start 2022-06-27T11:48:05.641 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.607+0000 7f411fdb8b80 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x55c64ae77360 0x55c64ae78d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:05.641 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.607+0000 7f411fdb8b80 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c64ae74110 0x55c64aeb2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:05.642 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.607+0000 7f411fdb8b80 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x55c64ae77730 0x55c64aeb2990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:05.642 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.607+0000 7f411fdb8b80 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c64ad651c0 con 0x55c64ae74110 2022-06-27T11:48:05.642 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.607+0000 7f411fdb8b80 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c64adcecb0 con 0x55c64ae77730 2022-06-27T11:48:05.643 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.607+0000 7f411fdb8b80 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c64adc84b0 con 0x55c64ae77360 2022-06-27T11:48:05.643 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.607+0000 7f4110d44700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c64ae74110 0x55c64aeb2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:05.644 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.607+0000 7f4110d44700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c64ae74110 0x55c64aeb2450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41830/0 (socket says 172.21.15.195:41830) 2022-06-27T11:48:05.644 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.607+0000 7f4110d44700 1 -- 172.21.15.195:0/3411554959 learned_addr learned my addr 172.21.15.195:0/3411554959 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:48:05.644 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.607+0000 7f4111d46700 1 --2- 172.21.15.195:0/3411554959 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x55c64ae77730 0x55c64aeb2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:05.645 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.607+0000 7f4111545700 1 --2- 172.21.15.195:0/3411554959 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x55c64ae77360 0x55c64ae78d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:05.645 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.607+0000 7f4110d44700 1 -- 172.21.15.195:0/3411554959 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x55c64ae77360 msgr2=0x55c64ae78d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:05.646 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.607+0000 7f4110d44700 1 --2- 172.21.15.195:0/3411554959 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x55c64ae77360 0x55c64ae78d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:05.646 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.607+0000 7f4110d44700 1 -- 172.21.15.195:0/3411554959 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x55c64ae77730 msgr2=0x55c64aeb2990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:05.646 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.607+0000 7f4110d44700 1 --2- 172.21.15.195:0/3411554959 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x55c64ae77730 0x55c64aeb2990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:05.647 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.607+0000 7f4110d44700 1 -- 172.21.15.195:0/3411554959 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c64ac51360 con 0x55c64ae74110 2022-06-27T11:48:05.647 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.607+0000 7f4110d44700 1 --2- 172.21.15.195:0/3411554959 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c64ae74110 0x55c64aeb2450 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f40fc001620 tx=0x7f40fc013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bedbd4c3a99491a server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:05.648 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.608+0000 7f40f27fc700 1 -- 172.21.15.195:0/3411554959 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40fc013630 con 0x55c64ae74110 2022-06-27T11:48:05.648 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.608+0000 7f40f27fc700 1 -- 172.21.15.195:0/3411554959 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40fc013790 con 0x55c64ae74110 2022-06-27T11:48:05.649 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.608+0000 7f40f27fc700 1 -- 172.21.15.195:0/3411554959 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40fc013b10 con 0x55c64ae74110 2022-06-27T11:48:05.649 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.608+0000 7f411fdb8b80 1 -- 172.21.15.195:0/3411554959 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c64ae74110 msgr2=0x55c64aeb2450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:05.649 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.608+0000 7f411fdb8b80 1 --2- 172.21.15.195:0/3411554959 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c64ae74110 0x55c64aeb2450 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f40fc001620 tx=0x7f40fc013000 comp rx=0 tx=0).stop 2022-06-27T11:48:05.650 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.608+0000 7f411fdb8b80 1 -- 172.21.15.195:0/3411554959 shutdown_connections 2022-06-27T11:48:05.650 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.608+0000 7f411fdb8b80 1 --2- 172.21.15.195:0/3411554959 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x55c64ae77730 0x55c64aeb2990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:05.651 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.608+0000 7f411fdb8b80 1 --2- 172.21.15.195:0/3411554959 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x55c64ae77360 0x55c64ae78d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:05.651 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.608+0000 7f411fdb8b80 1 --2- 172.21.15.195:0/3411554959 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c64ae74110 0x55c64aeb2450 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:05.651 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.608+0000 7f411fdb8b80 1 -- 172.21.15.195:0/3411554959 >> 172.21.15.195:0/3411554959 conn(0x55c64ae6e9d0 msgr2=0x55c64aebca90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:05.652 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.608+0000 7f411fdb8b80 1 -- 172.21.15.195:0/3411554959 shutdown_connections 2022-06-27T11:48:05.652 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.608+0000 7f411fdb8b80 1 -- 172.21.15.195:0/3411554959 wait complete. 2022-06-27T11:48:05.653 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.609+0000 7f411fdb8b80 1 -- start start 2022-06-27T11:48:05.653 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.609+0000 7f411fdb8b80 1 Processor -- start 2022-06-27T11:48:05.653 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.610+0000 7f411fdb8b80 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x55c64ae74110 0x55c64aeb2780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:05.654 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.610+0000 7f411fdb8b80 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x55c64ae77360 0x55c64aecc380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:05.654 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.610+0000 7f411fdb8b80 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c64ae77730 0x55c64aecc8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:05.655 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.610+0000 7f411fdb8b80 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c64acb6e40 con 0x55c64ae77730 2022-06-27T11:48:05.655 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.610+0000 7f411fdb8b80 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c64aebadb0 con 0x55c64ae74110 2022-06-27T11:48:05.655 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.610+0000 7f411fdb8b80 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c64aec19f0 con 0x55c64ae77360 2022-06-27T11:48:05.655 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.610+0000 7f411fdb8b80 10 client.0 _ll_register_callbacks cb 0x55c64ac74fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-06-27T11:48:05.656 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.610+0000 7f4111d46700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c64ae77730 0x55c64aecc8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:05.656 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.610+0000 7f4111d46700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c64ae77730 0x55c64aecc8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41836/0 (socket says 172.21.15.195:41836) 2022-06-27T11:48:05.656 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.610+0000 7f4111d46700 1 -- 172.21.15.195:0/4062122557 learned_addr learned my addr 172.21.15.195:0/4062122557 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:48:05.656 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.610+0000 7f4111545700 1 --2- 172.21.15.195:0/4062122557 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x55c64ae74110 0x55c64aeb2780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:05.657 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.610+0000 7f4110d44700 1 --2- 172.21.15.195:0/4062122557 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x55c64ae77360 0x55c64aecc380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:05.657 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.610+0000 7f4111d46700 1 -- 172.21.15.195:0/4062122557 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x55c64ae77360 msgr2=0x55c64aecc380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:05.658 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.610+0000 7f4111d46700 1 --2- 172.21.15.195:0/4062122557 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x55c64ae77360 0x55c64aecc380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:05.658 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.610+0000 7f4111d46700 1 -- 172.21.15.195:0/4062122557 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x55c64ae74110 msgr2=0x55c64aeb2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:05.658 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.610+0000 7f4111d46700 1 --2- 172.21.15.195:0/4062122557 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x55c64ae74110 0x55c64aeb2780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:05.659 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.610+0000 7f4111d46700 1 -- 172.21.15.195:0/4062122557 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40fc00f040 con 0x55c64ae77730 2022-06-27T11:48:05.659 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.610+0000 7f4111545700 1 --2- 172.21.15.195:0/4062122557 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x55c64ae74110 0x55c64aeb2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:48:05.660 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.610+0000 7f4111d46700 1 --2- 172.21.15.195:0/4062122557 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c64ae77730 0x55c64aecc8c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f410400b6c0 tx=0x7f4104018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c750177c992edc6c server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:05.660 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.610+0000 7f40e77fe700 10 client.0 ms_handle_connect on v2:172.21.15.195:3300/0 2022-06-27T11:48:05.660 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.610+0000 7f40e77fe700 1 -- 172.21.15.195:0/4062122557 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4104022070 con 0x55c64ae77730 2022-06-27T11:48:05.661 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.611+0000 7f40e77fe700 1 -- 172.21.15.195:0/4062122557 <== mon.0 v2:172.21.15.195:3300/0 2 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f41040047a0 con 0x55c64ae77730 2022-06-27T11:48:05.661 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.611+0000 7f411fdb8b80 1 -- 172.21.15.195:0/4062122557 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c64ae6c190 con 0x55c64ae77730 2022-06-27T11:48:05.661 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.611+0000 7f40e77fe700 1 -- 172.21.15.195:0/4062122557 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4104027e40 con 0x55c64ae77730 2022-06-27T11:48:05.662 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.611+0000 7f411fdb8b80 10 client.4653 fetch_fsmap learned FSMap version 8 2022-06-27T11:48:05.663 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.611+0000 7f411fdb8b80 1 -- 172.21.15.195:0/4062122557 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x55c64abfdb10 con 0x55c64ae77730 2022-06-27T11:48:05.663 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.611+0000 7f40e77fe700 1 -- 172.21.15.195:0/4062122557 <== mon.0 v2:172.21.15.195:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55c64ae6c190 con 0x55c64ae77730 2022-06-27T11:48:05.663 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.611+0000 7f40e77fe700 1 -- 172.21.15.195:0/4062122557 <== mon.0 v2:172.21.15.195:3300/0 5 ==== fsmap(e 8) v1 ==== 4190+0+0 (secure 0 0 0) 0x7f410400b050 con 0x55c64ae77730 2022-06-27T11:48:05.664 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.611+0000 7f411fdb8b80 10 client.4653 fetch_fsmap finished waiting for FSMap version 8 2022-06-27T11:48:05.664 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.611+0000 7f411fdb8b80 10 client.4653 resolve_mds: resolved name 'd' to daemon mds.d 2022-06-27T11:48:05.664 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.611+0000 7f411fdb8b80 20 client.4653 populate_metadata read hostname 'smithi195' 2022-06-27T11:48:05.665 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.611+0000 7f411fdb8b80 1 --2- 172.21.15.195:0/4062122557 >> [v2:172.21.15.202:6834/2038169702,v1:172.21.15.202:6835/2038169702] conn(0x55c64aea50a0 0x55c64aea5470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:05.665 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.611+0000 7f411fdb8b80 4 client.4653 mds_command: new command op to 4293 tid=0[{"prefix": "get_command_descriptions"}] 2022-06-27T11:48:05.666 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.611+0000 7f411fdb8b80 1 -- 172.21.15.195:0/4062122557 --> [v2:172.21.15.202:6834/2038169702,v1:172.21.15.202:6835/2038169702] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c64adc1a20 con 0x55c64aea50a0 2022-06-27T11:48:05.666 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.611+0000 7f4111545700 1 --2- 172.21.15.195:0/4062122557 >> [v2:172.21.15.202:6834/2038169702,v1:172.21.15.202:6835/2038169702] conn(0x55c64aea50a0 0x55c64aea5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:05.682 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.612+0000 7f40e77fe700 10 client.4653 ms_handle_connect on v2:172.21.15.202:6834/2038169702 2022-06-27T11:48:05.682 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.612+0000 7f4111545700 1 --2- 172.21.15.195:0/4062122557 >> [v2:172.21.15.202:6834/2038169702,v1:172.21.15.202:6835/2038169702] conn(0x55c64aea50a0 0x55c64aea5470 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fcf3e9146ecab307 server_cookie=ee905b86d49450de in_seq=0 out_seq=0 2022-06-27T11:48:05.682 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.612+0000 7f40e77fe700 1 -- 172.21.15.195:0/4062122557 <== mds.0 v2:172.21.15.202:6834/2038169702 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17402 (crc 0 0 0) 0x55c64adc1a20 con 0x55c64aea50a0 2022-06-27T11:48:05.683 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.612+0000 7f40e77fe700 10 client.4653 handle_command_reply: tid=0 2022-06-27T11:48:05.683 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.612+0000 7f411fdb8b80 1 client.4653 shutdown 2022-06-27T11:48:05.683 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.612+0000 7f411fdb8b80 1 -- 172.21.15.195:0/4062122557 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c64ae77730 msgr2=0x55c64aecc8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:05.684 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.612+0000 7f411fdb8b80 1 --2- 172.21.15.195:0/4062122557 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c64ae77730 0x55c64aecc8c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f410400b6c0 tx=0x7f4104018000 comp rx=0 tx=0).stop 2022-06-27T11:48:05.684 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.612+0000 7f411fdb8b80 1 -- 172.21.15.195:0/4062122557 shutdown_connections 2022-06-27T11:48:05.684 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.612+0000 7f411fdb8b80 1 --2- 172.21.15.195:0/4062122557 >> [v2:172.21.15.202:6834/2038169702,v1:172.21.15.202:6835/2038169702] conn(0x55c64aea50a0 0x55c64aea5470 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:05.685 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.612+0000 7f411fdb8b80 1 --2- 172.21.15.195:0/4062122557 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c64ae77730 0x55c64aecc8c0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:05.685 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.612+0000 7f40e77fe700 0 client.4653 ms_handle_reset on v2:172.21.15.202:6834/2038169702 2022-06-27T11:48:05.685 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.612+0000 7f411fdb8b80 1 --2- 172.21.15.195:0/4062122557 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x55c64ae74110 0x55c64aeb2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:05.686 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.612+0000 7f411fdb8b80 1 --2- 172.21.15.195:0/4062122557 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x55c64ae77360 0x55c64aecc380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:05.686 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.612+0000 7f411fdb8b80 1 -- 172.21.15.195:0/4062122557 >> 172.21.15.195:0/4062122557 conn(0x55c64ae6e9d0 msgr2=0x55c64aebca70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:05.687 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.613+0000 7f411fdb8b80 1 -- 172.21.15.195:0/4062122557 shutdown_connections 2022-06-27T11:48:05.687 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.613+0000 7f411fdb8b80 1 -- 172.21.15.195:0/4062122557 wait complete. 2022-06-27T11:48:05.687 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.613+0000 7f411fdb8b80 20 client.4653 trim_cache size 0 max 16384 2022-06-27T11:48:05.688 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.630+0000 7f411fdb8b80 1 Processor -- start 2022-06-27T11:48:05.688 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.630+0000 7f411fdb8b80 1 -- start start 2022-06-27T11:48:05.688 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.630+0000 7f411fdb8b80 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c64ae74110 0x55c64ae79e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:05.689 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.630+0000 7f411fdb8b80 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x55c64ae77360 0x55c64ae781c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:05.689 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.630+0000 7f411fdb8b80 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x55c64ae77730 0x55c64aeb2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:05.689 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.630+0000 7f411fdb8b80 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c64aeb72d0 con 0x55c64ae74110 2022-06-27T11:48:05.690 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.630+0000 7f411fdb8b80 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c64aec0970 con 0x55c64ae77360 2022-06-27T11:48:05.690 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.630+0000 7f411fdb8b80 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c64aea2920 con 0x55c64ae77730 2022-06-27T11:48:05.691 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.630+0000 7f4111545700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c64ae74110 0x55c64ae79e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:05.691 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.630+0000 7f4111545700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c64ae74110 0x55c64ae79e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41840/0 (socket says 172.21.15.195:41840) 2022-06-27T11:48:05.691 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.630+0000 7f4111545700 1 -- 172.21.15.195:0/929548287 learned_addr learned my addr 172.21.15.195:0/929548287 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:48:05.691 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.630+0000 7f4110d44700 1 --2- 172.21.15.195:0/929548287 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x55c64ae77360 0x55c64ae781c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:05.692 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.630+0000 7f4111d46700 1 --2- 172.21.15.195:0/929548287 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x55c64ae77730 0x55c64aeb2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:05.692 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.631+0000 7f4111545700 1 -- 172.21.15.195:0/929548287 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x55c64ae77730 msgr2=0x55c64aeb2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:05.692 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.631+0000 7f4111545700 1 --2- 172.21.15.195:0/929548287 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x55c64ae77730 0x55c64aeb2590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:05.692 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.631+0000 7f4111545700 1 -- 172.21.15.195:0/929548287 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x55c64ae77360 msgr2=0x55c64ae781c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:05.693 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.631+0000 7f4111545700 1 --2- 172.21.15.195:0/929548287 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x55c64ae77360 0x55c64ae781c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:05.693 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.631+0000 7f4111545700 1 -- 172.21.15.195:0/929548287 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c64abfbfe0 con 0x55c64ae74110 2022-06-27T11:48:05.693 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.631+0000 7f4111545700 1 --2- 172.21.15.195:0/929548287 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c64ae74110 0x55c64ae79e10 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f410800cc40 tx=0x7f4108066040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=990d2712f8ce6d07 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:05.693 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.631+0000 7f40f2ffd700 1 -- 172.21.15.195:0/929548287 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4108050e50 con 0x55c64ae74110 2022-06-27T11:48:05.694 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.631+0000 7f40f2ffd700 1 -- 172.21.15.195:0/929548287 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4108051050 con 0x55c64ae74110 2022-06-27T11:48:05.694 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.631+0000 7f40f2ffd700 1 -- 172.21.15.195:0/929548287 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f410800e290 con 0x55c64ae74110 2022-06-27T11:48:05.694 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.631+0000 7f411fdb8b80 1 -- 172.21.15.195:0/929548287 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c64ae74110 msgr2=0x55c64ae79e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:05.694 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.631+0000 7f411fdb8b80 1 --2- 172.21.15.195:0/929548287 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c64ae74110 0x55c64ae79e10 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f410800cc40 tx=0x7f4108066040 comp rx=0 tx=0).stop 2022-06-27T11:48:05.695 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.631+0000 7f411fdb8b80 1 -- 172.21.15.195:0/929548287 shutdown_connections 2022-06-27T11:48:05.695 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.631+0000 7f411fdb8b80 1 --2- 172.21.15.195:0/929548287 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c64ae74110 0x55c64ae79e10 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:05.695 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.631+0000 7f411fdb8b80 1 --2- 172.21.15.195:0/929548287 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x55c64ae77730 0x55c64aeb2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:05.695 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.631+0000 7f411fdb8b80 1 --2- 172.21.15.195:0/929548287 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x55c64ae77360 0x55c64ae781c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:05.695 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.631+0000 7f411fdb8b80 1 -- 172.21.15.195:0/929548287 >> 172.21.15.195:0/929548287 conn(0x55c64ae6e9d0 msgr2=0x55c64aebc260 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:05.696 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.631+0000 7f411fdb8b80 1 -- 172.21.15.195:0/929548287 shutdown_connections 2022-06-27T11:48:05.696 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.631+0000 7f411fdb8b80 1 -- 172.21.15.195:0/929548287 wait complete. 2022-06-27T11:48:05.696 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.632+0000 7f411fdb8b80 1 -- start start 2022-06-27T11:48:05.696 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.632+0000 7f411fdb8b80 1 Processor -- start 2022-06-27T11:48:05.696 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.633+0000 7f411fdb8b80 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x55c64ae74110 0x55c64aeb1400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:05.697 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.633+0000 7f411fdb8b80 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c64ae77360 0x55c64aea5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:05.697 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.633+0000 7f411fdb8b80 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x55c64ae77730 0x55c64af17980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:05.697 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.633+0000 7f411fdb8b80 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c64aea8e70 con 0x55c64ae77360 2022-06-27T11:48:05.697 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.633+0000 7f411fdb8b80 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c64aebcda0 con 0x55c64ae74110 2022-06-27T11:48:05.698 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.633+0000 7f411fdb8b80 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c64aebd060 con 0x55c64ae77730 2022-06-27T11:48:05.698 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.633+0000 7f411fdb8b80 10 client.0 _ll_register_callbacks cb 0x7f4104003f20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-06-27T11:48:05.698 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.633+0000 7f4110d44700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c64ae77360 0x55c64aea5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:05.698 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.633+0000 7f4110d44700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c64ae77360 0x55c64aea5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41848/0 (socket says 172.21.15.195:41848) 2022-06-27T11:48:05.698 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.633+0000 7f4110d44700 1 -- 172.21.15.195:0/201924277 learned_addr learned my addr 172.21.15.195:0/201924277 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:48:05.699 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.633+0000 7f4111545700 1 --2- 172.21.15.195:0/201924277 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x55c64ae74110 0x55c64aeb1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:05.699 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.633+0000 7f4111d46700 1 --2- 172.21.15.195:0/201924277 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x55c64ae77730 0x55c64af17980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:05.699 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.633+0000 7f4110d44700 1 -- 172.21.15.195:0/201924277 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x55c64ae77730 msgr2=0x55c64af17980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:05.699 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.633+0000 7f4110d44700 1 --2- 172.21.15.195:0/201924277 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x55c64ae77730 0x55c64af17980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:05.700 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.633+0000 7f4110d44700 1 -- 172.21.15.195:0/201924277 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x55c64ae74110 msgr2=0x55c64aeb1400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:05.700 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.633+0000 7f4110d44700 1 --2- 172.21.15.195:0/201924277 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x55c64ae74110 0x55c64aeb1400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:05.700 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.633+0000 7f4110d44700 1 -- 172.21.15.195:0/201924277 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f410c1594e0 con 0x55c64ae77360 2022-06-27T11:48:05.700 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.633+0000 7f4110d44700 1 --2- 172.21.15.195:0/201924277 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c64ae77360 0x55c64aea5700 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f40fc001490 tx=0x7f40fc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4f569d5c41b8e9c server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:05.700 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.633+0000 7f40e77fe700 10 client.0 ms_handle_connect on v2:172.21.15.195:3300/0 2022-06-27T11:48:05.701 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.634+0000 7f40e77fe700 1 -- 172.21.15.195:0/201924277 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40fc028040 con 0x55c64ae77360 2022-06-27T11:48:05.701 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.634+0000 7f40e77fe700 1 -- 172.21.15.195:0/201924277 <== mon.0 v2:172.21.15.195:3300/0 2 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f40fc013b10 con 0x55c64ae77360 2022-06-27T11:48:05.701 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.634+0000 7f411fdb8b80 1 -- 172.21.15.195:0/201924277 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f40fc013790 con 0x55c64ae77360 2022-06-27T11:48:05.701 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.634+0000 7f40e77fe700 1 -- 172.21.15.195:0/201924277 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7f40fc021660 con 0x55c64ae77360 2022-06-27T11:48:05.702 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.634+0000 7f411fdb8b80 10 client.4659 fetch_fsmap learned FSMap version 8 2022-06-27T11:48:05.702 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.634+0000 7f411fdb8b80 1 -- 172.21.15.195:0/201924277 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x55c64acca7b0 con 0x55c64ae77360 2022-06-27T11:48:05.702 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.634+0000 7f40e77fe700 1 -- 172.21.15.195:0/201924277 <== mon.0 v2:172.21.15.195:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40fc013790 con 0x55c64ae77360 2022-06-27T11:48:05.702 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.634+0000 7f40e77fe700 1 -- 172.21.15.195:0/201924277 <== mon.0 v2:172.21.15.195:3300/0 5 ==== fsmap(e 8) v1 ==== 4190+0+0 (secure 0 0 0) 0x7f40fc0217c0 con 0x55c64ae77360 2022-06-27T11:48:05.702 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.634+0000 7f411fdb8b80 10 client.4659 fetch_fsmap finished waiting for FSMap version 8 2022-06-27T11:48:05.703 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.634+0000 7f411fdb8b80 10 client.4659 resolve_mds: resolved name 'd' to daemon mds.d 2022-06-27T11:48:05.703 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.634+0000 7f411fdb8b80 20 client.4659 populate_metadata read hostname 'smithi195' 2022-06-27T11:48:05.703 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.634+0000 7f411fdb8b80 1 --2- 172.21.15.195:0/201924277 >> [v2:172.21.15.202:6834/2038169702,v1:172.21.15.202:6835/2038169702] conn(0x55c64aead3a0 0x55c64aead770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:05.703 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.634+0000 7f411fdb8b80 4 client.4659 mds_command: new command op to 4293 tid=0[{"prefix": "status"}] 2022-06-27T11:48:05.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.634+0000 7f411fdb8b80 1 -- 172.21.15.195:0/201924277 --> [v2:172.21.15.202:6834/2038169702,v1:172.21.15.202:6835/2038169702] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55c64adc1800 con 0x55c64aead3a0 2022-06-27T11:48:05.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.634+0000 7f4111545700 1 --2- 172.21.15.195:0/201924277 >> [v2:172.21.15.202:6834/2038169702,v1:172.21.15.202:6835/2038169702] conn(0x55c64aead3a0 0x55c64aead770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:05.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.634+0000 7f40e77fe700 10 client.4659 ms_handle_connect on v2:172.21.15.202:6834/2038169702 2022-06-27T11:48:05.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.634+0000 7f4111545700 1 --2- 172.21.15.195:0/201924277 >> [v2:172.21.15.202:6834/2038169702,v1:172.21.15.202:6835/2038169702] conn(0x55c64aead3a0 0x55c64aead770 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c89741a76256dac server_cookie=adbdc6bed591ef9e in_seq=0 out_seq=0 2022-06-27T11:48:05.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.635+0000 7f4111545700 1 -- 172.21.15.195:0/201924277 >> [v2:172.21.15.202:6834/2038169702,v1:172.21.15.202:6835/2038169702] conn(0x55c64aead3a0 msgr2=0x55c64aead770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 18 2022-06-27T11:48:05.705 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.635+0000 7f4111545700 1 -- 172.21.15.195:0/201924277 >> [v2:172.21.15.202:6834/2038169702,v1:172.21.15.202:6835/2038169702] conn(0x55c64aead3a0 msgr2=0x55c64aead770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2022-06-27T11:48:05.705 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.635+0000 7f4111545700 1 --2- 172.21.15.195:0/201924277 >> [v2:172.21.15.202:6834/2038169702,v1:172.21.15.202:6835/2038169702] conn(0x55c64aead3a0 0x55c64aead770 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2022-06-27T11:48:05.705 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.635+0000 7f4111545700 1 --2- 172.21.15.195:0/201924277 >> [v2:172.21.15.202:6834/2038169702,v1:172.21.15.202:6835/2038169702] conn(0x55c64aead3a0 0x55c64aead770 unknown :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault initiating reconnect 2022-06-27T11:48:05.705 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.635+0000 7f4111545700 1 --2- 172.21.15.195:0/201924277 >> [v2:172.21.15.202:6834/2038169702,v1:172.21.15.202:6835/2038169702] conn(0x55c64aead3a0 0x55c64aead770 unknown :-1 s=BANNER_CONNECTING pgs=15 cs=1 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:05.706 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.635+0000 7f40e77fe700 10 client.4659 ms_handle_connect on v2:172.21.15.202:6834/2038169702 2022-06-27T11:48:05.706 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.635+0000 7f4111545700 1 --2- 172.21.15.195:0/201924277 >> [v2:172.21.15.202:6834/2038169702,v1:172.21.15.202:6835/2038169702] conn(0x55c64aead3a0 0x55c64aead770 crc :-1 s=READY pgs=15 cs=1 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c89741a76256dac server_cookie=adbdc6bed591ef9e in_seq=0 out_seq=1 2022-06-27T11:48:05.706 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.635+0000 7f40e77fe700 1 -- 172.21.15.195:0/201924277 <== mds.0 v2:172.21.15.202:6834/2038169702 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x55c64adc1800 con 0x55c64aead3a0 2022-06-27T11:48:05.706 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.635+0000 7f40e77fe700 10 client.4659 handle_command_reply: tid=0 2022-06-27T11:48:05.706 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.635+0000 7f411fdb8b80 1 client.4659 shutdown 2022-06-27T11:48:05.707 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.635+0000 7f411fdb8b80 1 -- 172.21.15.195:0/201924277 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c64ae77360 msgr2=0x55c64aea5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:05.707 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.635+0000 7f411fdb8b80 1 --2- 172.21.15.195:0/201924277 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c64ae77360 0x55c64aea5700 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f40fc001490 tx=0x7f40fc017000 comp rx=0 tx=0).stop 2022-06-27T11:48:05.707 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.635+0000 7f411fdb8b80 1 -- 172.21.15.195:0/201924277 shutdown_connections 2022-06-27T11:48:05.707 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.635+0000 7f411fdb8b80 1 --2- 172.21.15.195:0/201924277 >> [v2:172.21.15.202:6834/2038169702,v1:172.21.15.202:6835/2038169702] conn(0x55c64aead3a0 0x55c64aead770 crc :-1 s=READY pgs=15 cs=1 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:05.707 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.635+0000 7f411fdb8b80 1 --2- 172.21.15.195:0/201924277 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x55c64ae77730 0x55c64af17980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:05.708 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.635+0000 7f40e77fe700 0 client.4659 ms_handle_reset on v2:172.21.15.202:6834/2038169702 2022-06-27T11:48:05.708 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.635+0000 7f411fdb8b80 1 --2- 172.21.15.195:0/201924277 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x55c64ae74110 0x55c64aeb1400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:05.708 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.635+0000 7f411fdb8b80 1 --2- 172.21.15.195:0/201924277 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c64ae77360 0x55c64aea5700 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:05.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.635+0000 7f411fdb8b80 1 -- 172.21.15.195:0/201924277 >> 172.21.15.195:0/201924277 conn(0x55c64ae6e9d0 msgr2=0x55c64ae79e10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:05.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.635+0000 7f411fdb8b80 1 -- 172.21.15.195:0/201924277 shutdown_connections 2022-06-27T11:48:05.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.635+0000 7f411fdb8b80 1 -- 172.21.15.195:0/201924277 wait complete. 2022-06-27T11:48:05.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.636+0000 7f411fdb8b80 20 client.4659 trim_cache size 0 max 16384 2022-06-27T11:48:05.710 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2022-06-27T11:48:05.710 INFO:teuthology.orchestra.run.smithi195.stdout: "cluster_fsid": "81b3a1d3-6a2b-47e8-8d69-7027cfeae7af", 2022-06-27T11:48:05.710 INFO:teuthology.orchestra.run.smithi195.stdout: "whoami": 0, 2022-06-27T11:48:05.711 INFO:teuthology.orchestra.run.smithi195.stdout: "id": 4293, 2022-06-27T11:48:05.711 INFO:teuthology.orchestra.run.smithi195.stdout: "want_state": "up:active", 2022-06-27T11:48:05.711 INFO:teuthology.orchestra.run.smithi195.stdout: "state": "up:active", 2022-06-27T11:48:05.711 INFO:teuthology.orchestra.run.smithi195.stdout: "fs_name": "cephfs", 2022-06-27T11:48:05.711 INFO:teuthology.orchestra.run.smithi195.stdout: "rank_uptime": 13.276440765, 2022-06-27T11:48:05.712 INFO:teuthology.orchestra.run.smithi195.stdout: "mdsmap_epoch": 8, 2022-06-27T11:48:05.712 INFO:teuthology.orchestra.run.smithi195.stdout: "osdmap_epoch": 21, 2022-06-27T11:48:05.712 INFO:teuthology.orchestra.run.smithi195.stdout: "osdmap_epoch_barrier": 21, 2022-06-27T11:48:05.712 INFO:teuthology.orchestra.run.smithi195.stdout: "uptime": 17.702162090000002 2022-06-27T11:48:05.713 INFO:teuthology.orchestra.run.smithi195.stdout:} 2022-06-27T11:48:05.714 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.636+0000 7f41137a9700 1 -- 172.21.15.195:0/4116983290 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f40ec063d00 msgr2=0x7f40ec0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:05.715 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.636+0000 7f41137a9700 1 --2- 172.21.15.195:0/4116983290 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f40ec063d00 0x7f40ec0661b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f40fc0076e0 tx=0x7f40fc007bd0 comp rx=0 tx=0).stop 2022-06-27T11:48:05.715 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.636+0000 7f41137a9700 1 -- 172.21.15.195:0/4116983290 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f410c06f620 msgr2=0x7f410c14cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:05.715 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.636+0000 7f41137a9700 1 --2- 172.21.15.195:0/4116983290 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f410c06f620 0x7f410c14cc30 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f410800c4f0 tx=0x7f4108007e20 comp rx=0 tx=0).stop 2022-06-27T11:48:05.715 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.636+0000 7f41137a9700 1 -- 172.21.15.195:0/4116983290 shutdown_connections 2022-06-27T11:48:05.716 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.636+0000 7f41137a9700 1 --2- 172.21.15.195:0/4116983290 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f40ec063d00 0x7f40ec0661b0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:05.716 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.636+0000 7f41137a9700 1 --2- 172.21.15.195:0/4116983290 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f410c06f620 0x7f410c14cc30 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:05.716 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.636+0000 7f41137a9700 1 --2- 172.21.15.195:0/4116983290 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f410c150950 0x7f410c15bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:05.716 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.636+0000 7f41137a9700 1 --2- 172.21.15.195:0/4116983290 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f410c150580 0x7f410c15b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:05.717 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.636+0000 7f41137a9700 1 -- 172.21.15.195:0/4116983290 >> 172.21.15.195:0/4116983290 conn(0x7f410c063af0 msgr2=0x7f410c1458d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:05.717 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.636+0000 7f41137a9700 1 -- 172.21.15.195:0/4116983290 shutdown_connections 2022-06-27T11:48:05.717 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.636+0000 7f41137a9700 1 -- 172.21.15.195:0/4116983290 wait complete. 2022-06-27T11:48:05.717 INFO:teuthology.run_tasks:Running task kclient... 2022-06-27T11:48:05.741 INFO:tasks.kclient:Mounting kernel clients... 2022-06-27T11:48:05.741 INFO:tasks.kclient:config is {'client.0': {}} 2022-06-27T11:48:05.742 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-06-27T11:48:05.743 DEBUG:teuthology.orchestra.run.smithi195:> ip netns list 2022-06-27T11:48:05.761 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-06-27T11:48:05.761 DEBUG:teuthology.orchestra.run.smithi195:> sudo ip link delete ceph-brx 2022-06-27T11:48:05.829 INFO:teuthology.orchestra.run.smithi195.stderr:Cannot find device "ceph-brx" 2022-06-27T11:48:05.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-27T11:48:05.832 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2022-06-27T11:48:05.833 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2022-06-27T11:48:05.992 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.992+0000 7fd6a7cf7700 1 Processor -- start 2022-06-27T11:48:05.993 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.993+0000 7fd6a7cf7700 1 -- start start 2022-06-27T11:48:06.007 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.993+0000 7fd6a7cf7700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fd6a014e490 0x7fd6a006cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:06.008 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.993+0000 7fd6a7cf7700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd6a014e860 0x7fd6a00c3840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:06.008 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.993+0000 7fd6a7cf7700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6a00c33b0 con 0x7fd6a00c3d80 2022-06-27T11:48:06.009 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.993+0000 7fd6a7cf7700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6a006d520 con 0x7fd6a014e490 2022-06-27T11:48:06.009 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.993+0000 7fd6a7cf7700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a006f220 con 0x7fd6a014e860 2022-06-27T11:48:06.010 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.994+0000 7fd6a6294700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fd6a00c3d80 0x7fd6a006f620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:48048/0 (socket says 172.21.15.195:48048) 2022-06-27T11:48:06.010 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.994+0000 7fd6a6294700 1 -- 172.21.15.195:0/3563581277 learned_addr learned my addr 172.21.15.195:0/3563581277 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:48:06.010 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.994+0000 7fd6a5292700 1 --2- 172.21.15.195:0/3563581277 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd6a014e860 0x7fd6a00c3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:06.011 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.994+0000 7fd6a5a93700 1 --2- 172.21.15.195:0/3563581277 >> v2:172.21.15.202:3300/0 conn(0x7fd6a014e490 0x7fd6a006cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:06.011 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.994+0000 7fd6a4a91700 1 -- 172.21.15.195:0/3563581277 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3512840881 0 0) 0x7fd6a00c33b0 con 0x7fd6a00c3d80 2022-06-27T11:48:06.011 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.994+0000 7fd6a4a91700 1 -- 172.21.15.195:0/3563581277 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd688003430 con 0x7fd6a00c3d80 2022-06-27T11:48:06.012 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.994+0000 7fd6a4a91700 1 -- 172.21.15.195:0/3563581277 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2384985533 0 0) 0x7fd688003430 con 0x7fd6a00c3d80 2022-06-27T11:48:06.012 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.994+0000 7fd6a4a91700 1 -- 172.21.15.195:0/3563581277 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd6a014e860 msgr2=0x7fd6a00c3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:06.012 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.994+0000 7fd6a4a91700 1 --2- 172.21.15.195:0/3563581277 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd6a014e860 0x7fd6a00c3840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:06.013 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.994+0000 7fd6a4a91700 1 -- 172.21.15.195:0/3563581277 >> v2:172.21.15.202:3300/0 conn(0x7fd6a014e490 msgr2=0x7fd6a006cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:06.013 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.994+0000 7fd6a4a91700 1 --2- 172.21.15.195:0/3563581277 >> v2:172.21.15.202:3300/0 conn(0x7fd6a014e490 0x7fd6a006cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:06.014 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.994+0000 7fd6a4a91700 1 -- 172.21.15.195:0/3563581277 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6a01542c0 con 0x7fd6a00c3d80 2022-06-27T11:48:06.014 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.994+0000 7fd6a4a91700 1 -- 172.21.15.195:0/3563581277 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7fd6980017f0 con 0x7fd6a00c3d80 2022-06-27T11:48:06.015 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.994+0000 7fd6a5292700 1 --2- 172.21.15.195:0/3563581277 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd6a014e860 0x7fd6a00c3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:48:06.015 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.994+0000 7fd6a5a93700 1 --2- 172.21.15.195:0/3563581277 >> v2:172.21.15.202:3300/0 conn(0x7fd6a014e490 0x7fd6a006cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:48:06.016 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.994+0000 7fd6a4a91700 1 -- 172.21.15.195:0/3563581277 >> v1:172.21.15.195:6789/0 conn(0x7fd6a00c3d80 legacy=0x7fd6a006f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:06.016 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.994+0000 7fd6a4a91700 1 --2- 172.21.15.195:0/3563581277 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd688003920 0x7fd688003d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:06.016 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.994+0000 7fd6a4a91700 1 --2- 172.21.15.195:0/3563581277 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd6880046d0 0x7fd688004b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:06.016 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.994+0000 7fd6a4a91700 1 --2- 172.21.15.195:0/3563581277 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd6a014e860 0x7fd688005080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:06.017 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.994+0000 7fd6a4a91700 1 -- 172.21.15.195:0/3563581277 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a006d520 con 0x7fd6880046d0 2022-06-27T11:48:06.017 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.994+0000 7fd6a4a91700 1 -- 172.21.15.195:0/3563581277 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a006f220 con 0x7fd688003920 2022-06-27T11:48:06.017 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.994+0000 7fd6a4a91700 1 -- 172.21.15.195:0/3563581277 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd68800b920 con 0x7fd6a014e860 2022-06-27T11:48:06.017 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.994+0000 7fd6a5292700 1 --2- 172.21.15.195:0/3563581277 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd6880046d0 0x7fd688004b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:06.018 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.994+0000 7fd6a5a93700 1 --2- 172.21.15.195:0/3563581277 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd688003920 0x7fd688003d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:06.018 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.994+0000 7fd6a6294700 1 --2- 172.21.15.195:0/3563581277 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd6a014e860 0x7fd688005080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:06.018 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.995+0000 7fd6a5292700 1 -- 172.21.15.195:0/3563581277 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd6a014e860 msgr2=0x7fd688005080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:06.018 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.995+0000 7fd6a5292700 1 --2- 172.21.15.195:0/3563581277 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd6a014e860 0x7fd688005080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:06.019 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.995+0000 7fd6a5292700 1 -- 172.21.15.195:0/3563581277 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd688003920 msgr2=0x7fd688003d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:06.019 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.995+0000 7fd6a5292700 1 --2- 172.21.15.195:0/3563581277 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd688003920 0x7fd688003d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:06.019 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.995+0000 7fd6a5292700 1 -- 172.21.15.195:0/3563581277 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd69c0092e0 con 0x7fd6880046d0 2022-06-27T11:48:06.020 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.995+0000 7fd6a6294700 1 --2- 172.21.15.195:0/3563581277 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd6a014e860 0x7fd688005080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T11:48:06.020 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.995+0000 7fd6a5292700 1 --2- 172.21.15.195:0/3563581277 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd6880046d0 0x7fd688004b40 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd6a014ec30 tx=0x7fd69c009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d11487d27173614d server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:06.020 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.995+0000 7fd6a4a91700 1 -- 172.21.15.195:0/3563581277 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd69c009600 con 0x7fd6880046d0 2022-06-27T11:48:06.020 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.995+0000 7fd6a4a91700 1 -- 172.21.15.195:0/3563581277 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd69c00e710 con 0x7fd6880046d0 2022-06-27T11:48:06.021 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.995+0000 7fd6a4a91700 1 -- 172.21.15.195:0/3563581277 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd69c00eab0 con 0x7fd6880046d0 2022-06-27T11:48:06.021 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.995+0000 7fd6a7cf7700 1 -- 172.21.15.195:0/3563581277 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd6880046d0 msgr2=0x7fd688004b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:06.021 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.995+0000 7fd6a7cf7700 1 --2- 172.21.15.195:0/3563581277 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd6880046d0 0x7fd688004b40 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd6a014ec30 tx=0x7fd69c009060 comp rx=0 tx=0).stop 2022-06-27T11:48:06.021 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.995+0000 7fd6a6294700 1 -- 172.21.15.195:0/3563581277 reap_dead start 2022-06-27T11:48:06.022 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.995+0000 7fd6a7cf7700 1 -- 172.21.15.195:0/3563581277 shutdown_connections 2022-06-27T11:48:06.022 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.995+0000 7fd6a7cf7700 1 -- 172.21.15.195:0/3563581277 >> 172.21.15.195:0/3563581277 conn(0x7fd6a0063af0 msgr2=0x7fd6a0063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:06.022 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.995+0000 7fd6a7cf7700 1 -- 172.21.15.195:0/3563581277 shutdown_connections 2022-06-27T11:48:06.022 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.995+0000 7fd6a7cf7700 1 -- 172.21.15.195:0/3563581277 wait complete. 2022-06-27T11:48:06.023 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.996+0000 7fd6a7cf7700 1 Processor -- start 2022-06-27T11:48:06.023 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.997+0000 7fd6a7cf7700 1 -- start start 2022-06-27T11:48:06.023 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.997+0000 7fd6a7cf7700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd6880046d0 0x7fd6a016a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:06.024 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.997+0000 7fd6a7cf7700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd6a014b9a0 0x7fd6a00c9000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:06.024 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.997+0000 7fd6a7cf7700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd6a01507f0 0x7fd6a0149260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:06.025 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.997+0000 7fd6a7cf7700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a0151680 con 0x7fd6a014b9a0 2022-06-27T11:48:06.025 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.997+0000 7fd6a7cf7700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a0150360 con 0x7fd6880046d0 2022-06-27T11:48:06.025 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.997+0000 7fd6a7cf7700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a014fd50 con 0x7fd6a01507f0 2022-06-27T11:48:06.026 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.997+0000 7fd6a5292700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd6a014b9a0 0x7fd6a00c9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:06.026 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.997+0000 7fd6a5292700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd6a014b9a0 0x7fd6a00c9000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41870/0 (socket says 172.21.15.195:41870) 2022-06-27T11:48:06.026 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.997+0000 7fd6a5292700 1 -- 172.21.15.195:0/3621807838 learned_addr learned my addr 172.21.15.195:0/3621807838 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:48:06.026 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.997+0000 7fd6a5a93700 1 --2- 172.21.15.195:0/3621807838 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd6880046d0 0x7fd6a016a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:06.026 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.997+0000 7fd6a6294700 1 --2- 172.21.15.195:0/3621807838 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd6a01507f0 0x7fd6a0149260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:06.027 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.998+0000 7fd6a5292700 1 -- 172.21.15.195:0/3621807838 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd6a01507f0 msgr2=0x7fd6a0149260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:06.027 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.998+0000 7fd6a5292700 1 --2- 172.21.15.195:0/3621807838 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd6a01507f0 0x7fd6a0149260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:06.027 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.998+0000 7fd6a5292700 1 -- 172.21.15.195:0/3621807838 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd6880046d0 msgr2=0x7fd6a016a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:06.027 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.998+0000 7fd6a5292700 1 --2- 172.21.15.195:0/3621807838 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd6880046d0 0x7fd6a016a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:06.028 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.998+0000 7fd6a5292700 1 -- 172.21.15.195:0/3621807838 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6a01568a0 con 0x7fd6a014b9a0 2022-06-27T11:48:06.028 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.998+0000 7fd6a5292700 1 --2- 172.21.15.195:0/3621807838 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd6a014b9a0 0x7fd6a00c9000 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd688004550 tx=0x7fd69c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:06.028 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.998+0000 7fd68dffb700 1 -- 172.21.15.195:0/3621807838 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd69c022070 con 0x7fd6a014b9a0 2022-06-27T11:48:06.028 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.998+0000 7fd68dffb700 1 -- 172.21.15.195:0/3621807838 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd69c000bc0 con 0x7fd6a014b9a0 2022-06-27T11:48:06.028 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.998+0000 7fd6a7cf7700 1 -- 172.21.15.195:0/3621807838 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6a01518d0 con 0x7fd6a014b9a0 2022-06-27T11:48:06.029 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.998+0000 7fd68dffb700 1 -- 172.21.15.195:0/3621807838 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd69c01c450 con 0x7fd6a014b9a0 2022-06-27T11:48:06.029 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.998+0000 7fd6a7cf7700 1 -- 172.21.15.195:0/3621807838 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6a00c9e90 con 0x7fd6a014b9a0 2022-06-27T11:48:06.029 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.999+0000 7fd6a7cf7700 1 -- 172.21.15.195:0/3621807838 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6a01540e0 con 0x7fd6a014b9a0 2022-06-27T11:48:06.029 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:05.999+0000 7fd68dffb700 1 -- 172.21.15.195:0/3621807838 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fd69c02d030 con 0x7fd6a014b9a0 2022-06-27T11:48:06.029 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:06.000+0000 7fd68dffb700 1 --2- 172.21.15.195:0/3621807838 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd680063d00 0x7fd6800661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:06.030 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:06.000+0000 7fd68dffb700 1 -- 172.21.15.195:0/3621807838 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fd69c081be0 con 0x7fd6a014b9a0 2022-06-27T11:48:06.030 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:06.000+0000 7fd6a5a93700 1 --2- 172.21.15.195:0/3621807838 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd680063d00 0x7fd6800661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:06.030 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:06.000+0000 7fd6a5a93700 1 --2- 172.21.15.195:0/3621807838 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd680063d00 0x7fd6800661b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd690006dc0 tx=0x7fd690009060 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:06.031 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:06.004+0000 7fd68dffb700 1 -- 172.21.15.195:0/3621807838 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fd69c050860 con 0x7fd6a014b9a0 2022-06-27T11:48:06.227 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:06.228+0000 7fd6a7cf7700 1 -- 172.21.15.195:0/3621807838 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fd6a0150fe0 con 0x7fd6a014b9a0 2022-06-27T11:48:06.228 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:06.228+0000 7fd68dffb700 1 -- 172.21.15.195:0/3621807838 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7fd69c02b020 con 0x7fd6a014b9a0 2022-06-27T11:48:06.228 INFO:teuthology.orchestra.run.smithi195.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2022-06-27T11:48:06.230 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:06.230+0000 7fd6a7cf7700 1 -- 172.21.15.195:0/3621807838 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd680063d00 msgr2=0x7fd6800661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:06.231 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:06.230+0000 7fd6a7cf7700 1 --2- 172.21.15.195:0/3621807838 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd680063d00 0x7fd6800661b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd690006dc0 tx=0x7fd690009060 comp rx=0 tx=0).stop 2022-06-27T11:48:06.231 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:06.230+0000 7fd6a7cf7700 1 -- 172.21.15.195:0/3621807838 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd6a014b9a0 msgr2=0x7fd6a00c9000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:06.231 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:06.230+0000 7fd6a7cf7700 1 --2- 172.21.15.195:0/3621807838 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd6a014b9a0 0x7fd6a00c9000 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd688004550 tx=0x7fd69c017000 comp rx=0 tx=0).stop 2022-06-27T11:48:06.231 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:06.230+0000 7fd6a7cf7700 1 -- 172.21.15.195:0/3621807838 shutdown_connections 2022-06-27T11:48:06.232 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:06.230+0000 7fd6a7cf7700 1 --2- 172.21.15.195:0/3621807838 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd680063d00 0x7fd6800661b0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:06.232 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:06.230+0000 7fd6a7cf7700 1 --2- 172.21.15.195:0/3621807838 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd6a01507f0 0x7fd6a0149260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:06.232 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:06.230+0000 7fd6a7cf7700 1 --2- 172.21.15.195:0/3621807838 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd6880046d0 0x7fd6a016a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:06.232 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:06.230+0000 7fd6a7cf7700 1 --2- 172.21.15.195:0/3621807838 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd6a014b9a0 0x7fd6a00c9000 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:06.233 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:06.230+0000 7fd6a7cf7700 1 -- 172.21.15.195:0/3621807838 >> 172.21.15.195:0/3621807838 conn(0x7fd6a0063af0 msgr2=0x7fd6a00c59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:06.233 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:06.230+0000 7fd6a7cf7700 1 -- 172.21.15.195:0/3621807838 shutdown_connections 2022-06-27T11:48:06.234 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:06.230+0000 7fd6a7cf7700 1 -- 172.21.15.195:0/3621807838 wait complete. 2022-06-27T11:48:06.242 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2022-06-27T11:48:06.243 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2022-06-27T11:48:06.243 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi195.front.sepia.ceph.com 2022-06-27T11:48:06.243 INFO:tasks.cephfs.mount:self.client.name = client.0 2022-06-27T11:48:06.243 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2022-06-27T11:48:06.244 INFO:tasks.cephfs.mount:self.cephfs_name = None 2022-06-27T11:48:06.244 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2022-06-27T11:48:06.244 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2022-06-27T11:48:06.245 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2022-06-27T11:48:06.246 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-06-27T11:48:06.246 DEBUG:teuthology.orchestra.run.smithi195:> ip addr 2022-06-27T11:48:06.264 INFO:teuthology.orchestra.run.smithi195.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2022-06-27T11:48:06.264 INFO:teuthology.orchestra.run.smithi195.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2022-06-27T11:48:06.265 INFO:teuthology.orchestra.run.smithi195.stdout: inet 127.0.0.1/8 scope host lo 2022-06-27T11:48:06.265 INFO:teuthology.orchestra.run.smithi195.stdout: valid_lft forever preferred_lft forever 2022-06-27T11:48:06.266 INFO:teuthology.orchestra.run.smithi195.stdout: inet6 ::1/128 scope host 2022-06-27T11:48:06.266 INFO:teuthology.orchestra.run.smithi195.stdout: valid_lft forever preferred_lft forever 2022-06-27T11:48:06.267 INFO:teuthology.orchestra.run.smithi195.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2022-06-27T11:48:06.267 INFO:teuthology.orchestra.run.smithi195.stdout: link/ether 0c:c4:7a:d9:36:f0 brd ff:ff:ff:ff:ff:ff 2022-06-27T11:48:06.267 INFO:teuthology.orchestra.run.smithi195.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2022-06-27T11:48:06.268 INFO:teuthology.orchestra.run.smithi195.stdout: link/ether 0c:c4:7a:d9:36:f1 brd ff:ff:ff:ff:ff:ff 2022-06-27T11:48:06.268 INFO:teuthology.orchestra.run.smithi195.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2022-06-27T11:48:06.269 INFO:teuthology.orchestra.run.smithi195.stdout: link/ether 0c:c4:7a:88:80:80 brd ff:ff:ff:ff:ff:ff 2022-06-27T11:48:06.269 INFO:teuthology.orchestra.run.smithi195.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2022-06-27T11:48:06.270 INFO:teuthology.orchestra.run.smithi195.stdout: link/ether 0c:c4:7a:88:80:81 brd ff:ff:ff:ff:ff:ff 2022-06-27T11:48:06.270 INFO:teuthology.orchestra.run.smithi195.stdout: inet 172.21.15.195/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2022-06-27T11:48:06.271 INFO:teuthology.orchestra.run.smithi195.stdout: valid_lft 42525sec preferred_lft 42525sec 2022-06-27T11:48:06.271 INFO:teuthology.orchestra.run.smithi195.stdout: inet6 fe80::ec4:7aff:fe88:8081/64 scope link 2022-06-27T11:48:06.271 INFO:teuthology.orchestra.run.smithi195.stdout: valid_lft forever preferred_lft forever 2022-06-27T11:48:06.272 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2022-06-27T11:48:06.273 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-06-27T11:48:06.273 DEBUG:teuthology.orchestra.run.smithi195:> (cd / && exec bash -c ' 2022-06-27T11:48:06.274 DEBUG:teuthology.orchestra.run.smithi195:> set -e 2022-06-27T11:48:06.274 DEBUG:teuthology.orchestra.run.smithi195:> sudo ip link add name ceph-brx type bridge 2022-06-27T11:48:06.275 DEBUG:teuthology.orchestra.run.smithi195:> sudo ip addr flush dev ceph-brx 2022-06-27T11:48:06.275 DEBUG:teuthology.orchestra.run.smithi195:> sudo ip link set ceph-brx up 2022-06-27T11:48:06.276 DEBUG:teuthology.orchestra.run.smithi195:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2022-06-27T11:48:06.276 DEBUG:teuthology.orchestra.run.smithi195:> ') 2022-06-27T11:48:06.415 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-06-27T11:48:06.416 DEBUG:teuthology.orchestra.run.smithi195:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2022-06-27T11:48:06.445 INFO:teuthology.orchestra.run.smithi195.stdout:1 2022-06-27T11:48:06.447 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-06-27T11:48:06.448 DEBUG:teuthology.orchestra.run.smithi195:> route 2022-06-27T11:48:06.519 INFO:teuthology.orchestra.run.smithi195.stdout:Kernel IP routing table 2022-06-27T11:48:06.520 INFO:teuthology.orchestra.run.smithi195.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2022-06-27T11:48:06.520 INFO:teuthology.orchestra.run.smithi195.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2022-06-27T11:48:06.520 INFO:teuthology.orchestra.run.smithi195.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2022-06-27T11:48:06.521 INFO:teuthology.orchestra.run.smithi195.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2022-06-27T11:48:06.521 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-06-27T11:48:06.522 DEBUG:teuthology.orchestra.run.smithi195:> (cd / && exec bash -c ' 2022-06-27T11:48:06.522 DEBUG:teuthology.orchestra.run.smithi195:> set -e 2022-06-27T11:48:06.523 DEBUG:teuthology.orchestra.run.smithi195:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2022-06-27T11:48:06.523 DEBUG:teuthology.orchestra.run.smithi195:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2022-06-27T11:48:06.523 DEBUG:teuthology.orchestra.run.smithi195:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2022-06-27T11:48:06.524 DEBUG:teuthology.orchestra.run.smithi195:> ') 2022-06-27T11:48:06.746 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-06-27T11:48:06.747 DEBUG:teuthology.orchestra.run.smithi195:> ip netns list 2022-06-27T11:48:06.764 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-06-27T11:48:06.765 DEBUG:teuthology.orchestra.run.smithi195:> ip netns list-id 2022-06-27T11:48:06.821 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-06-27T11:48:06.822 DEBUG:teuthology.orchestra.run.smithi195:> (cd / && exec bash -c ' 2022-06-27T11:48:06.822 DEBUG:teuthology.orchestra.run.smithi195:> set -e 2022-06-27T11:48:06.822 DEBUG:teuthology.orchestra.run.smithi195:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2022-06-27T11:48:06.823 DEBUG:teuthology.orchestra.run.smithi195:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2022-06-27T11:48:06.823 DEBUG:teuthology.orchestra.run.smithi195:> ') 2022-06-27T11:48:06.923 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2022-06-27T11:48:06.924 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-06-27T11:48:06.924 DEBUG:teuthology.orchestra.run.smithi195:> (cd / && exec bash -c ' 2022-06-27T11:48:06.925 DEBUG:teuthology.orchestra.run.smithi195:> set -e 2022-06-27T11:48:06.925 DEBUG:teuthology.orchestra.run.smithi195:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2022-06-27T11:48:06.925 DEBUG:teuthology.orchestra.run.smithi195:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2022-06-27T11:48:06.926 DEBUG:teuthology.orchestra.run.smithi195:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2022-06-27T11:48:06.926 DEBUG:teuthology.orchestra.run.smithi195:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2022-06-27T11:48:06.926 DEBUG:teuthology.orchestra.run.smithi195:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2022-06-27T11:48:06.927 DEBUG:teuthology.orchestra.run.smithi195:> ') 2022-06-27T11:48:07.139 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-06-27T11:48:07.140 DEBUG:teuthology.orchestra.run.smithi195:> (cd / && exec bash -c ' 2022-06-27T11:48:07.141 DEBUG:teuthology.orchestra.run.smithi195:> set -e 2022-06-27T11:48:07.141 DEBUG:teuthology.orchestra.run.smithi195:> sudo ip link set brx.0 up 2022-06-27T11:48:07.142 DEBUG:teuthology.orchestra.run.smithi195:> sudo ip link set dev brx.0 master ceph-brx 2022-06-27T11:48:07.143 DEBUG:teuthology.orchestra.run.smithi195:> ') 2022-06-27T11:48:07.244 INFO:teuthology.orchestra.run:Running command with timeout 60 2022-06-27T11:48:07.245 DEBUG:teuthology.orchestra.run.smithi195:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2022-06-27T11:48:07.264 INFO:teuthology.orchestra.run.smithi195.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2022-06-27T11:48:07.268 INFO:teuthology.orchestra.run:Running command with timeout 60 2022-06-27T11:48:07.269 DEBUG:teuthology.orchestra.run.smithi195:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2022-06-27T11:48:07.325 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2022-06-27T11:48:07.326 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-06-27T11:48:07.327 DEBUG:teuthology.orchestra.run.smithi195:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2022-06-27T11:48:07.429 INFO:teuthology.orchestra.run.smithi195.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2022-06-27T11:48:07.429 INFO:teuthology.orchestra.run.smithi195.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2022-06-27T11:48:07.430 INFO:teuthology.orchestra.run.smithi195.stdout:invalid new device string format 2022-06-27T11:48:07.521 INFO:teuthology.orchestra.run.smithi195.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2022-06-27T11:48:07.521 INFO:teuthology.orchestra.run.smithi195.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2022-06-27T11:48:07.522 INFO:teuthology.orchestra.run.smithi195.stdout:invalid new device string format 2022-06-27T11:48:07.522 INFO:teuthology.orchestra.run.smithi195.stdout:mount.ceph: resolved to: "172.21.15.195:3300,172.21.15.202:3300,172.21.15.202:3301" 2022-06-27T11:48:07.523 INFO:teuthology.orchestra.run.smithi195.stdout:mount.ceph: switching to using v1 address with old syntax 2022-06-27T11:48:07.588 INFO:teuthology.orchestra.run.smithi195.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2022-06-27T11:48:07.588 INFO:teuthology.orchestra.run.smithi195.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2022-06-27T11:48:07.589 INFO:teuthology.orchestra.run.smithi195.stdout:invalid new device string format 2022-06-27T11:48:07.589 INFO:teuthology.orchestra.run.smithi195.stdout:mount.ceph: resolved to: "172.21.15.195:3300,172.21.15.202:3300,172.21.15.202:3301" 2022-06-27T11:48:07.589 INFO:teuthology.orchestra.run.smithi195.stdout:mount.ceph: switching to using v1 address with old syntax 2022-06-27T11:48:07.590 INFO:teuthology.orchestra.run.smithi195.stdout:mount.ceph: resolved to: "172.21.15.195:6789,172.21.15.202:6789,172.21.15.202:6790" 2022-06-27T11:48:07.590 INFO:teuthology.orchestra.run.smithi195.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.195:6789,172.21.15.202:6789,172.21.15.202:6790:/ 2022-06-27T11:48:07.590 INFO:teuthology.orchestra.run.smithi195.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=81b3a1d3-6a2b-47e8-8d69-7027cfeae7af" will pass to kernel 2022-06-27T11:48:07.591 INFO:tasks.cephfs.kernel_mount:mount command passed 2022-06-27T11:48:07.592 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-06-27T11:48:07.592 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2022-06-27T11:48:07.620 INFO:teuthology.run_tasks:Running task mds_thrash... 2022-06-27T11:48:07.634 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-27T11:48:07.781 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.782+0000 7f093f0d1700 1 Processor -- start 2022-06-27T11:48:07.782 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.782+0000 7f093f0d1700 1 -- start start 2022-06-27T11:48:07.790 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.783+0000 7f093f0d1700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f093806e850 0x7f09381570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:07.791 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.783+0000 7f093f0d1700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f093806fba0 con 0x7f09380c34a0 2022-06-27T11:48:07.791 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.783+0000 7f093f0d1700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f093806f5e0 con 0x7f093806fef0 2022-06-27T11:48:07.791 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.783+0000 7f093f0d1700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09381575e0 con 0x7f093806e850 2022-06-27T11:48:07.791 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.783+0000 7f093d66e700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f09380c34a0 0x7f09380c2920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:48072/0 (socket says 172.21.15.195:48072) 2022-06-27T11:48:07.792 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.783+0000 7f093d66e700 1 -- 172.21.15.195:0/3731791869 learned_addr learned my addr 172.21.15.195:0/3731791869 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:48:07.792 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.783+0000 7f093ce6d700 1 --2- 172.21.15.195:0/3731791869 >> v2:172.21.15.202:3300/0 conn(0x7f093806e850 0x7f09381570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:07.792 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.783+0000 7f09377fe700 1 -- 172.21.15.195:0/3731791869 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3413479585 0 0) 0x7f093806fba0 con 0x7f09380c34a0 2022-06-27T11:48:07.793 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.783+0000 7f09377fe700 1 -- 172.21.15.195:0/3731791869 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f091c003430 con 0x7f09380c34a0 2022-06-27T11:48:07.793 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.783+0000 7f09377fe700 1 -- 172.21.15.195:0/3731791869 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4212239028 0 0) 0x7f091c003430 con 0x7f09380c34a0 2022-06-27T11:48:07.793 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.783+0000 7f09377fe700 1 -- 172.21.15.195:0/3731791869 >> v2:172.21.15.202:3300/0 conn(0x7f093806e850 msgr2=0x7f09381570a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:07.793 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.783+0000 7f09377fe700 1 --2- 172.21.15.195:0/3731791869 >> v2:172.21.15.202:3300/0 conn(0x7f093806e850 0x7f09381570a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:07.793 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.783+0000 7f09377fe700 1 -- 172.21.15.195:0/3731791869 >> v1:172.21.15.202:6789/0 conn(0x7f093806fef0 legacy=0x7f0938069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:07.794 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.783+0000 7f09377fe700 1 -- 172.21.15.195:0/3731791869 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f093806f8c0 con 0x7f09380c34a0 2022-06-27T11:48:07.794 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.783+0000 7f09377fe700 1 -- 172.21.15.195:0/3731791869 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7f09300017f0 con 0x7f09380c34a0 2022-06-27T11:48:07.794 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.783+0000 7f09377fe700 1 -- 172.21.15.195:0/3731791869 >> v1:172.21.15.195:6789/0 conn(0x7f09380c34a0 legacy=0x7f09380c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:07.794 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.783+0000 7f09377fe700 1 --2- 172.21.15.195:0/3731791869 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f091c003920 0x7f091c003d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:07.795 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.783+0000 7f09377fe700 1 --2- 172.21.15.195:0/3731791869 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f091c0046d0 0x7f091c004b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:07.795 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.783+0000 7f09377fe700 1 --2- 172.21.15.195:0/3731791869 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f091c005080 0x7f091c00b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:07.795 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.783+0000 7f09377fe700 1 -- 172.21.15.195:0/3731791869 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09381575e0 con 0x7f091c0046d0 2022-06-27T11:48:07.795 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.783+0000 7f09377fe700 1 -- 172.21.15.195:0/3731791869 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f091c00bf90 con 0x7f091c003920 2022-06-27T11:48:07.795 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.783+0000 7f09377fe700 1 -- 172.21.15.195:0/3731791869 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f091c00c150 con 0x7f091c005080 2022-06-27T11:48:07.796 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.783+0000 7f0937fff700 1 --2- 172.21.15.195:0/3731791869 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f091c0046d0 0x7f091c004b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:07.796 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.783+0000 7f093ce6d700 1 --2- 172.21.15.195:0/3731791869 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f091c003920 0x7f091c003d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:07.796 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.784+0000 7f093d66e700 1 --2- 172.21.15.195:0/3731791869 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f091c005080 0x7f091c00b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:07.796 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.784+0000 7f0937fff700 1 -- 172.21.15.195:0/3731791869 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f091c005080 msgr2=0x7f091c00b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:07.797 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.784+0000 7f0937fff700 1 --2- 172.21.15.195:0/3731791869 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f091c005080 0x7f091c00b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:07.797 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.784+0000 7f0937fff700 1 -- 172.21.15.195:0/3731791869 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f091c003920 msgr2=0x7f091c003d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:07.797 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.784+0000 7f0937fff700 1 --2- 172.21.15.195:0/3731791869 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f091c003920 0x7f091c003d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:07.797 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.784+0000 7f0937fff700 1 -- 172.21.15.195:0/3731791869 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f091c003430 con 0x7f091c0046d0 2022-06-27T11:48:07.797 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.784+0000 7f093d66e700 1 --2- 172.21.15.195:0/3731791869 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f091c005080 0x7f091c00b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T11:48:07.798 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.784+0000 7f093d66e700 1 -- 172.21.15.195:0/3731791869 reap_dead start 2022-06-27T11:48:07.798 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.784+0000 7f0937fff700 1 --2- 172.21.15.195:0/3731791869 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f091c0046d0 0x7f091c004b40 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f09240017a0 tx=0x7f0924007bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a2f8a5a858c02d3 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:07.798 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.784+0000 7f09377fe700 1 -- 172.21.15.195:0/3731791869 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09240050d0 con 0x7f091c0046d0 2022-06-27T11:48:07.798 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.784+0000 7f09377fe700 1 -- 172.21.15.195:0/3731791869 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0924005250 con 0x7f091c0046d0 2022-06-27T11:48:07.799 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.784+0000 7f09377fe700 1 -- 172.21.15.195:0/3731791869 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0924003750 con 0x7f091c0046d0 2022-06-27T11:48:07.799 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.784+0000 7f093f0d1700 1 -- 172.21.15.195:0/3731791869 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f091c0046d0 msgr2=0x7f091c004b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:07.799 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.784+0000 7f093f0d1700 1 --2- 172.21.15.195:0/3731791869 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f091c0046d0 0x7f091c004b40 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f09240017a0 tx=0x7f0924007bf0 comp rx=0 tx=0).stop 2022-06-27T11:48:07.799 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.784+0000 7f093f0d1700 1 -- 172.21.15.195:0/3731791869 shutdown_connections 2022-06-27T11:48:07.800 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.784+0000 7f093f0d1700 1 --2- 172.21.15.195:0/3731791869 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f091c0046d0 0x7f091c004b40 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:07.800 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.784+0000 7f093f0d1700 1 -- 172.21.15.195:0/3731791869 >> 172.21.15.195:0/3731791869 conn(0x7f09380c6350 msgr2=0x7f09380c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:07.800 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.784+0000 7f093f0d1700 1 -- 172.21.15.195:0/3731791869 shutdown_connections 2022-06-27T11:48:07.800 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.785+0000 7f093f0d1700 1 -- 172.21.15.195:0/3731791869 wait complete. 2022-06-27T11:48:07.800 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.786+0000 7f093f0d1700 1 Processor -- start 2022-06-27T11:48:07.801 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.786+0000 7f093f0d1700 1 -- start start 2022-06-27T11:48:07.801 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.787+0000 7f093f0d1700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f091c0046d0 0x7f09381553a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:07.801 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.787+0000 7f093f0d1700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f09381542e0 0x7f0938153410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:07.801 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.787+0000 7f093f0d1700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0938154f20 0x7f09381511f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:07.802 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.787+0000 7f093f0d1700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0938154960 con 0x7f0938154f20 2022-06-27T11:48:07.802 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.787+0000 7f093f0d1700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0938153e40 con 0x7f09381542e0 2022-06-27T11:48:07.802 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.787+0000 7f093f0d1700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0938154bb0 con 0x7f091c0046d0 2022-06-27T11:48:07.802 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.787+0000 7f093d66e700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0938154f20 0x7f09381511f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:07.802 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.787+0000 7f093d66e700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0938154f20 0x7f09381511f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41896/0 (socket says 172.21.15.195:41896) 2022-06-27T11:48:07.803 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.787+0000 7f093d66e700 1 -- 172.21.15.195:0/443391809 learned_addr learned my addr 172.21.15.195:0/443391809 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:48:07.803 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.787+0000 7f093ce6d700 1 --2- 172.21.15.195:0/443391809 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f091c0046d0 0x7f09381553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:07.803 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.787+0000 7f0937fff700 1 --2- 172.21.15.195:0/443391809 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f09381542e0 0x7f0938153410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:07.803 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.788+0000 7f093d66e700 1 -- 172.21.15.195:0/443391809 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f091c0046d0 msgr2=0x7f09381553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:07.804 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.788+0000 7f093d66e700 1 --2- 172.21.15.195:0/443391809 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f091c0046d0 0x7f09381553a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:07.804 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.788+0000 7f093d66e700 1 -- 172.21.15.195:0/443391809 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f09381542e0 msgr2=0x7f0938153410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:07.804 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.788+0000 7f093d66e700 1 --2- 172.21.15.195:0/443391809 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f09381542e0 0x7f0938153410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:07.805 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.788+0000 7f093d66e700 1 -- 172.21.15.195:0/443391809 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0938156c70 con 0x7f0938154f20 2022-06-27T11:48:07.805 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.788+0000 7f093d66e700 1 --2- 172.21.15.195:0/443391809 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0938154f20 0x7f09381511f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f09380c3870 tx=0x7f0930001490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:07.805 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.788+0000 7f0934ff9700 1 -- 172.21.15.195:0/443391809 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09300079e0 con 0x7f0938154f20 2022-06-27T11:48:07.805 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.788+0000 7f0934ff9700 1 -- 172.21.15.195:0/443391809 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f093000e8c0 con 0x7f0938154f20 2022-06-27T11:48:07.806 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.788+0000 7f093f0d1700 1 -- 172.21.15.195:0/443391809 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0938156f50 con 0x7f0938154f20 2022-06-27T11:48:07.806 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.788+0000 7f0934ff9700 1 -- 172.21.15.195:0/443391809 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f093000ec60 con 0x7f0938154f20 2022-06-27T11:48:07.806 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.788+0000 7f093f0d1700 1 -- 172.21.15.195:0/443391809 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0938152ef0 con 0x7f0938154f20 2022-06-27T11:48:07.806 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.789+0000 7f093f0d1700 1 -- 172.21.15.195:0/443391809 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09380722c0 con 0x7f0938154f20 2022-06-27T11:48:07.807 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.789+0000 7f0934ff9700 1 -- 172.21.15.195:0/443391809 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f0930013630 con 0x7f0938154f20 2022-06-27T11:48:07.807 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.789+0000 7f0934ff9700 1 --2- 172.21.15.195:0/443391809 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0920063d00 0x7f09200661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:07.807 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.790+0000 7f0934ff9700 1 -- 172.21.15.195:0/443391809 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f0930080ad0 con 0x7f0938154f20 2022-06-27T11:48:07.807 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.790+0000 7f093ce6d700 1 --2- 172.21.15.195:0/443391809 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0920063d00 0x7f09200661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:07.807 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.790+0000 7f093ce6d700 1 --2- 172.21.15.195:0/443391809 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0920063d00 0x7f09200661b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f092c006dc0 tx=0x7f092c009060 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:07.808 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.794+0000 7f0934ff9700 1 -- 172.21.15.195:0/443391809 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f093004f6c0 con 0x7f0938154f20 2022-06-27T11:48:07.977 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.978+0000 7f093f0d1700 1 -- 172.21.15.195:0/443391809 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f09380c5830 con 0x7f0938154f20 2022-06-27T11:48:07.978 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.978+0000 7f0934ff9700 1 -- 172.21.15.195:0/443391809 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16914 (secure 0 0 0) 0x7f0930052ce0 con 0x7f0938154f20 2022-06-27T11:48:07.978 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:48:07.979 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":21,"fsid":"81b3a1d3-6a2b-47e8-8d69-7027cfeae7af","created":"2022-06-27T11:47:35.094715+0000","modified":"2022-06-27T11:47:53.359270+0000","last_up_change":"2022-06-27T11:47:44.777459+0000","last_in_change":"2022-06-27T11:47:37.596818+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2022-06-27T11:47:46.368325+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2022-06-27T11:47:46.386132+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-06-27T11:47:50.901270+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-06-27T11:47:51.314719+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c64bc51f-d4f3-4bc2-806b-da0dc78d51b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6813","nonce":94217},{"type":"v1","addr":"172.21.15.195:6815","nonce":94217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6817","nonce":94217},{"type":"v1","addr":"172.21.15.195:6819","nonce":94217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":94217},{"type":"v1","addr":"172.21.15.195:6823","nonce":94217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6820","nonce":94217},{"type":"v1","addr":"172.21.15.195:6821","nonce":94217}]},"public_addr":"172.21.15.195:6815/94217","cluster_addr":"172.21.15.195:6819/94217","heartbeat_back_addr":"172.21.15.195:6823/94217","heartbeat_front_addr":"172.21.15.195:6821/94217","state":["exists","up"]},{"osd":1,"uuid":"bea71cdf-aa98-493f-9207-66d886a3e30e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":94223},{"type":"v1","addr":"172.21.15.195:6809","nonce":94223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":94223},{"type":"v1","addr":"172.21.15.195:6811","nonce":94223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6816","nonce":94223},{"type":"v1","addr":"172.21.15.195:6818","nonce":94223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":94223},{"type":"v1","addr":"172.21.15.195:6814","nonce":94223}]},"public_addr":"172.21.15.195:6809/94223","cluster_addr":"172.21.15.195:6811/94223","heartbeat_back_addr":"172.21.15.195:6818/94223","heartbeat_front_addr":"172.21.15.195:6814/94223","state":["exists","up"]},{"osd":2,"uuid":"bf6c87a0-a8ea-497a-9ce1-721285888065","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":94221},{"type":"v1","addr":"172.21.15.195:6801","nonce":94221}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":94221},{"type":"v1","addr":"172.21.15.195:6803","nonce":94221}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":94221},{"type":"v1","addr":"172.21.15.195:6807","nonce":94221}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":94221},{"type":"v1","addr":"172.21.15.195:6805","nonce":94221}]},"public_addr":"172.21.15.195:6801/94221","cluster_addr":"172.21.15.195:6803/94221","heartbeat_back_addr":"172.21.15.195:6807/94221","heartbeat_front_addr":"172.21.15.195:6805/94221","state":["exists","up"]},{"osd":3,"uuid":"e8aa47ce-af8c-47e7-9654-870cd53229f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":94232},{"type":"v1","addr":"172.21.15.195:6825","nonce":94232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":94232},{"type":"v1","addr":"172.21.15.195:6827","nonce":94232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":94232},{"type":"v1","addr":"172.21.15.195:6831","nonce":94232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":94232},{"type":"v1","addr":"172.21.15.195:6829","nonce":94232}]},"public_addr":"172.21.15.195:6825/94232","cluster_addr":"172.21.15.195:6827/94232","heartbeat_back_addr":"172.21.15.195:6831/94232","heartbeat_front_addr":"172.21.15.195:6829/94232","state":["exists","up"]},{"osd":4,"uuid":"99ffa553-f149-443a-98f5-68b10f77e085","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6800","nonce":94308},{"type":"v1","addr":"172.21.15.202:6801","nonce":94308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6802","nonce":94308},{"type":"v1","addr":"172.21.15.202:6803","nonce":94308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6806","nonce":94308},{"type":"v1","addr":"172.21.15.202:6807","nonce":94308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6804","nonce":94308},{"type":"v1","addr":"172.21.15.202:6805","nonce":94308}]},"public_addr":"172.21.15.202:6801/94308","cluster_addr":"172.21.15.202:6803/94308","heartbeat_back_addr":"172.21.15.202:6807/94308","heartbeat_front_addr":"172.21.15.202:6805/94308","state":["exists","up"]},{"osd":5,"uuid":"35409ef0-891b-4c01-82e4-26b8e64f502d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6824","nonce":94325},{"type":"v1","addr":"172.21.15.202:6825","nonce":94325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6826","nonce":94325},{"type":"v1","addr":"172.21.15.202:6827","nonce":94325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6830","nonce":94325},{"type":"v1","addr":"172.21.15.202:6831","nonce":94325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6828","nonce":94325},{"type":"v1","addr":"172.21.15.202:6829","nonce":94325}]},"public_addr":"172.21.15.202:6825/94325","cluster_addr":"172.21.15.202:6827/94325","heartbeat_back_addr":"172.21.15.202:6831/94325","heartbeat_front_addr":"172.21.15.202:6829/94325","state":["exists","up"]},{"osd":6,"uuid":"009124c4-8b5f-47d4-bd20-2c19271a2adc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6809","nonce":94309},{"type":"v1","addr":"172.21.15.202:6811","nonce":94309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6813","nonce":94309},{"type":"v1","addr":"172.21.15.202:6815","nonce":94309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6821","nonce":94309},{"type":"v1","addr":"172.21.15.202:6823","nonce":94309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6817","nonce":94309},{"type":"v1","addr":"172.21.15.202:6819","nonce":94309}]},"public_addr":"172.21.15.202:6811/94309","cluster_addr":"172.21.15.202:6815/94309","heartbeat_back_addr":"172.21.15.202:6823/94309","heartbeat_front_addr":"172.21.15.202:6819/94309","state":["exists","up"]},{"osd":7,"uuid":"7773115e-d3c3-4a0c-8801-d8ce63e0e5a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6808","nonce":94310},{"type":"v1","addr":"172.21.15.202:6810","nonce":94310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6812","nonce":94310},{"type":"v1","addr":"172.21.15.202:6814","nonce":94310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6820","nonce":94310},{"type":"v1","addr":"172.21.15.202:6822","nonce":94310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6816","nonce":94310},{"type":"v1","addr":"172.21.15.202:6818","nonce":94310}]},"public_addr":"172.21.15.202:6810/94310","cluster_addr":"172.21.15.202:6814/94310","heartbeat_back_addr":"172.21.15.202:6822/94310","heartbeat_front_addr":"172.21.15.202:6818/94310","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.776182+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.806922+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.237997+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.784371+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.723615+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.820332+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:43.841414+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.758071+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-06-27T11:48:07.980 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.981+0000 7f093f0d1700 1 -- 172.21.15.195:0/443391809 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0920063d00 msgr2=0x7f09200661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:07.981 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.981+0000 7f093f0d1700 1 --2- 172.21.15.195:0/443391809 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0920063d00 0x7f09200661b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f092c006dc0 tx=0x7f092c009060 comp rx=0 tx=0).stop 2022-06-27T11:48:07.981 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.981+0000 7f093f0d1700 1 -- 172.21.15.195:0/443391809 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0938154f20 msgr2=0x7f09381511f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:07.981 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.981+0000 7f093f0d1700 1 --2- 172.21.15.195:0/443391809 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0938154f20 0x7f09381511f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f09380c3870 tx=0x7f0930001490 comp rx=0 tx=0).stop 2022-06-27T11:48:07.982 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.981+0000 7f093f0d1700 1 -- 172.21.15.195:0/443391809 shutdown_connections 2022-06-27T11:48:07.982 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.981+0000 7f093f0d1700 1 --2- 172.21.15.195:0/443391809 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0920063d00 0x7f09200661b0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:07.982 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.981+0000 7f093f0d1700 1 --2- 172.21.15.195:0/443391809 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0938154f20 0x7f09381511f0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:07.982 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.981+0000 7f093f0d1700 1 --2- 172.21.15.195:0/443391809 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f091c0046d0 0x7f09381553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:07.983 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.981+0000 7f093f0d1700 1 --2- 172.21.15.195:0/443391809 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f09381542e0 0x7f0938153410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:07.983 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.981+0000 7f093f0d1700 1 -- 172.21.15.195:0/443391809 >> 172.21.15.195:0/443391809 conn(0x7f09380c6350 msgr2=0x7f09380c4c10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:07.983 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.981+0000 7f093f0d1700 1 -- 172.21.15.195:0/443391809 shutdown_connections 2022-06-27T11:48:07.984 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:07.981+0000 7f093f0d1700 1 -- 172.21.15.195:0/443391809 wait complete. 2022-06-27T11:48:07.993 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2022-06-27T11:47:46.368325+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2022-06-27T11:47:46.386132+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2022-06-27T11:47:50.901270+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2022-06-27T11:47:51.314719+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-06-27T11:48:07.993 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-06-27T11:48:08.125 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.117+0000 7f5f2faf9700 1 Processor -- start 2022-06-27T11:48:08.125 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.117+0000 7f5f2faf9700 1 -- start start 2022-06-27T11:48:08.126 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.118+0000 7f5f2faf9700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5f2806fef0 0x7f5f281570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:08.126 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.118+0000 7f5f2faf9700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f5f280c34a0 0x7f5f280c2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:08.126 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.118+0000 7f5f2faf9700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f2806fba0 con 0x7f5f2806e850 2022-06-27T11:48:08.127 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.118+0000 7f5f2faf9700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f281575e0 con 0x7f5f280c34a0 2022-06-27T11:48:08.127 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.118+0000 7f5f2faf9700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2806f5e0 con 0x7f5f2806fef0 2022-06-27T11:48:08.127 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.118+0000 7f5f2d895700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f5f2806e850 0x7f5f28069de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:48096/0 (socket says 172.21.15.195:48096) 2022-06-27T11:48:08.127 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.118+0000 7f5f2d895700 1 -- 172.21.15.195:0/2923127130 learned_addr learned my addr 172.21.15.195:0/2923127130 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:48:08.128 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.119+0000 7f5f2e096700 1 --2- 172.21.15.195:0/2923127130 >> v2:172.21.15.202:3300/0 conn(0x7f5f280c34a0 0x7f5f280c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:08.128 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.119+0000 7f5f2d094700 1 --2- 172.21.15.195:0/2923127130 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5f2806fef0 0x7f5f281570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:08.128 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.119+0000 7f5f2c893700 1 -- 172.21.15.195:0/2923127130 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4082024749 0 0) 0x7f5f2806fba0 con 0x7f5f2806e850 2022-06-27T11:48:08.128 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.119+0000 7f5f2c893700 1 -- 172.21.15.195:0/2923127130 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f10003430 con 0x7f5f2806e850 2022-06-27T11:48:08.129 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.119+0000 7f5f2d094700 1 -- 172.21.15.195:0/2923127130 >> v2:172.21.15.202:3300/0 conn(0x7f5f280c34a0 msgr2=0x7f5f280c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:08.129 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.119+0000 7f5f2d094700 1 --2- 172.21.15.195:0/2923127130 >> v2:172.21.15.202:3300/0 conn(0x7f5f280c34a0 0x7f5f280c2920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:08.129 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.119+0000 7f5f2d094700 1 -- 172.21.15.195:0/2923127130 >> v1:172.21.15.195:6789/0 conn(0x7f5f2806e850 legacy=0x7f5f28069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:08.129 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.119+0000 7f5f2c893700 1 -- 172.21.15.195:0/2923127130 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1075889507 0 0) 0x7f5f10003430 con 0x7f5f2806e850 2022-06-27T11:48:08.130 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.119+0000 7f5f2d094700 1 -- 172.21.15.195:0/2923127130 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f2806f8c0 con 0x7f5f2806fef0 2022-06-27T11:48:08.130 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.119+0000 7f5f2e096700 1 --2- 172.21.15.195:0/2923127130 >> v2:172.21.15.202:3300/0 conn(0x7f5f280c34a0 0x7f5f280c2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:48:08.130 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.119+0000 7f5f2d094700 1 --2- 172.21.15.195:0/2923127130 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5f2806fef0 0x7f5f281570a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5f240097a0 tx=0x7f5f24009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=563777d44e5bbff6 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:08.130 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.119+0000 7f5f2c893700 1 -- 172.21.15.195:0/2923127130 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f24011630 con 0x7f5f2806fef0 2022-06-27T11:48:08.131 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.119+0000 7f5f2c893700 1 -- 172.21.15.195:0/2923127130 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f24011790 con 0x7f5f2806fef0 2022-06-27T11:48:08.131 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.119+0000 7f5f2c893700 1 -- 172.21.15.195:0/2923127130 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f24011b30 con 0x7f5f2806fef0 2022-06-27T11:48:08.131 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.120+0000 7f5f2faf9700 1 -- 172.21.15.195:0/2923127130 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5f2806fef0 msgr2=0x7f5f281570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:08.131 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.120+0000 7f5f2faf9700 1 --2- 172.21.15.195:0/2923127130 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5f2806fef0 0x7f5f281570a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5f240097a0 tx=0x7f5f24009060 comp rx=0 tx=0).stop 2022-06-27T11:48:08.132 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.120+0000 7f5f2faf9700 1 -- 172.21.15.195:0/2923127130 shutdown_connections 2022-06-27T11:48:08.132 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.120+0000 7f5f2faf9700 1 --2- 172.21.15.195:0/2923127130 >> v2:172.21.15.202:3300/0 conn(0x7f5f280c34a0 0x7f5f280c2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:08.132 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.120+0000 7f5f2faf9700 1 --2- 172.21.15.195:0/2923127130 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5f2806fef0 0x7f5f281570a0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:08.132 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.120+0000 7f5f2faf9700 1 -- 172.21.15.195:0/2923127130 >> 172.21.15.195:0/2923127130 conn(0x7f5f280c6350 msgr2=0x7f5f280c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:08.132 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.120+0000 7f5f2faf9700 1 -- 172.21.15.195:0/2923127130 shutdown_connections 2022-06-27T11:48:08.133 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.120+0000 7f5f2faf9700 1 -- 172.21.15.195:0/2923127130 wait complete. 2022-06-27T11:48:08.133 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.121+0000 7f5f2faf9700 1 Processor -- start 2022-06-27T11:48:08.133 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.121+0000 7f5f2faf9700 1 -- start start 2022-06-27T11:48:08.133 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.122+0000 7f5f2faf9700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5f2806e850 0x7f5f2816b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:08.134 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.122+0000 7f5f2faf9700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5f280c34a0 0x7f5f280bff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:08.134 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.122+0000 7f5f2faf9700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5f2814a4a0 0x7f5f28069c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:08.134 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.122+0000 7f5f2faf9700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2814af30 con 0x7f5f280c34a0 2022-06-27T11:48:08.134 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.122+0000 7f5f2faf9700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2814fc50 con 0x7f5f2806e850 2022-06-27T11:48:08.135 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.122+0000 7f5f2faf9700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f280c04a0 con 0x7f5f2814a4a0 2022-06-27T11:48:08.135 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.122+0000 7f5f2d094700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5f280c34a0 0x7f5f280bff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:08.135 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.122+0000 7f5f2d094700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5f280c34a0 0x7f5f280bff60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41916/0 (socket says 172.21.15.195:41916) 2022-06-27T11:48:08.136 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.122+0000 7f5f2d094700 1 -- 172.21.15.195:0/1173842362 learned_addr learned my addr 172.21.15.195:0/1173842362 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:48:08.136 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.122+0000 7f5f2d895700 1 --2- 172.21.15.195:0/1173842362 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5f2806e850 0x7f5f2816b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:08.136 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.122+0000 7f5f2e096700 1 --2- 172.21.15.195:0/1173842362 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5f2814a4a0 0x7f5f28069c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:08.136 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.122+0000 7f5f2d094700 1 -- 172.21.15.195:0/1173842362 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5f2814a4a0 msgr2=0x7f5f28069c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:08.137 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.122+0000 7f5f2d094700 1 --2- 172.21.15.195:0/1173842362 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5f2814a4a0 0x7f5f28069c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:08.137 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.122+0000 7f5f2d094700 1 -- 172.21.15.195:0/1173842362 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5f2806e850 msgr2=0x7f5f2816b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:08.137 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.122+0000 7f5f2d094700 1 --2- 172.21.15.195:0/1173842362 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5f2806e850 0x7f5f2816b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:08.137 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.122+0000 7f5f2d094700 1 -- 172.21.15.195:0/1173842362 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f20009a20 con 0x7f5f280c34a0 2022-06-27T11:48:08.138 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.122+0000 7f5f2d094700 1 --2- 172.21.15.195:0/1173842362 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5f280c34a0 0x7f5f280bff60 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f5f24009d90 tx=0x7f5f24017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:08.138 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.123+0000 7f5f15ffb700 1 -- 172.21.15.195:0/1173842362 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f24011900 con 0x7f5f280c34a0 2022-06-27T11:48:08.138 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.123+0000 7f5f15ffb700 1 -- 172.21.15.195:0/1173842362 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f24011a60 con 0x7f5f280c34a0 2022-06-27T11:48:08.138 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.123+0000 7f5f2faf9700 1 -- 172.21.15.195:0/1173842362 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f240092e0 con 0x7f5f280c34a0 2022-06-27T11:48:08.139 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.123+0000 7f5f15ffb700 1 -- 172.21.15.195:0/1173842362 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f24023600 con 0x7f5f280c34a0 2022-06-27T11:48:08.139 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.123+0000 7f5f2faf9700 1 -- 172.21.15.195:0/1173842362 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f2814b460 con 0x7f5f280c34a0 2022-06-27T11:48:08.140 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.124+0000 7f5f2faf9700 1 -- 172.21.15.195:0/1173842362 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f280722c0 con 0x7f5f280c34a0 2022-06-27T11:48:08.140 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.124+0000 7f5f15ffb700 1 -- 172.21.15.195:0/1173842362 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f5f24029020 con 0x7f5f280c34a0 2022-06-27T11:48:08.140 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.124+0000 7f5f15ffb700 1 --2- 172.21.15.195:0/1173842362 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5f08063d00 0x7f5f080661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:08.141 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.125+0000 7f5f15ffb700 1 -- 172.21.15.195:0/1173842362 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f5f240817a0 con 0x7f5f280c34a0 2022-06-27T11:48:08.141 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.125+0000 7f5f2d895700 1 --2- 172.21.15.195:0/1173842362 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5f08063d00 0x7f5f080661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:08.141 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.125+0000 7f5f2d895700 1 --2- 172.21.15.195:0/1173842362 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5f08063d00 0x7f5f080661b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5f180011e0 tx=0x7f5f18007bd0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:08.143 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.129+0000 7f5f15ffb700 1 -- 172.21.15.195:0/1173842362 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f5f24050450 con 0x7f5f280c34a0 2022-06-27T11:48:08.305 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.305+0000 7f5f2faf9700 1 -- 172.21.15.195:0/1173842362 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f5f2806f240 con 0x7f5f280c34a0 2022-06-27T11:48:08.305 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.305+0000 7f5f15ffb700 1 -- 172.21.15.195:0/1173842362 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v21) v1 ==== 92+0+10 (secure 0 0 0) 0x7f5f240150a0 con 0x7f5f280c34a0 2022-06-27T11:48:08.306 INFO:teuthology.orchestra.run.smithi195.stdout:pg_num: 8 2022-06-27T11:48:08.308 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.307+0000 7f5f2faf9700 1 -- 172.21.15.195:0/1173842362 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5f08063d00 msgr2=0x7f5f080661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:08.309 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.307+0000 7f5f2faf9700 1 --2- 172.21.15.195:0/1173842362 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5f08063d00 0x7f5f080661b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5f180011e0 tx=0x7f5f18007bd0 comp rx=0 tx=0).stop 2022-06-27T11:48:08.310 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.307+0000 7f5f2faf9700 1 -- 172.21.15.195:0/1173842362 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5f280c34a0 msgr2=0x7f5f280bff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:08.311 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.307+0000 7f5f2faf9700 1 --2- 172.21.15.195:0/1173842362 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5f280c34a0 0x7f5f280bff60 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f5f24009d90 tx=0x7f5f24017000 comp rx=0 tx=0).stop 2022-06-27T11:48:08.311 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.307+0000 7f5f2faf9700 1 -- 172.21.15.195:0/1173842362 shutdown_connections 2022-06-27T11:48:08.312 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.307+0000 7f5f2faf9700 1 --2- 172.21.15.195:0/1173842362 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5f08063d00 0x7f5f080661b0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:08.312 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.307+0000 7f5f2faf9700 1 --2- 172.21.15.195:0/1173842362 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5f2814a4a0 0x7f5f28069c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:08.312 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.307+0000 7f5f2faf9700 1 --2- 172.21.15.195:0/1173842362 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5f2806e850 0x7f5f2816b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:08.313 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.307+0000 7f5f2faf9700 1 --2- 172.21.15.195:0/1173842362 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5f280c34a0 0x7f5f280bff60 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:08.313 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.307+0000 7f5f2faf9700 1 -- 172.21.15.195:0/1173842362 >> 172.21.15.195:0/1173842362 conn(0x7f5f280c6350 msgr2=0x7f5f28156d50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:08.314 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.307+0000 7f5f2faf9700 1 -- 172.21.15.195:0/1173842362 shutdown_connections 2022-06-27T11:48:08.314 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.307+0000 7f5f2faf9700 1 -- 172.21.15.195:0/1173842362 wait complete. 2022-06-27T11:48:08.320 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2022-06-27T11:48:08.451 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.452+0000 7f764fbf3700 1 Processor -- start 2022-06-27T11:48:08.452 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.452+0000 7f764fbf3700 1 -- start start 2022-06-27T11:48:08.455 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.453+0000 7f764fbf3700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f7648150580 0x7f7648156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:08.455 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.453+0000 7f764fbf3700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7648150950 0x7f76480c7b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:08.455 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.453+0000 7f764fbf3700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76480c76b0 con 0x7f764806f620 2022-06-27T11:48:08.455 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.453+0000 7f764fbf3700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7648157430 con 0x7f7648150580 2022-06-27T11:48:08.456 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.453+0000 7f764fbf3700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76480bad70 con 0x7f7648150950 2022-06-27T11:48:08.456 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.453+0000 7f764d98f700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f7648150580 0x7f7648156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:08.456 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.453+0000 7f764e190700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f764806f620 0x7f764806d6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:48122/0 (socket says 172.21.15.195:48122) 2022-06-27T11:48:08.456 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.453+0000 7f764e190700 1 -- 172.21.15.195:0/3435493941 learned_addr learned my addr 172.21.15.195:0/3435493941 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:48:08.457 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.453+0000 7f764d18e700 1 --2- 172.21.15.195:0/3435493941 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7648150950 0x7f76480c7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:08.457 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.453+0000 7f764d98f700 1 -- 172.21.15.195:0/3435493941 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7648150950 msgr2=0x7f76480c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:08.457 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.453+0000 7f764d98f700 1 --2- 172.21.15.195:0/3435493941 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7648150950 0x7f76480c7b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:08.457 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.453+0000 7f764d98f700 1 -- 172.21.15.195:0/3435493941 >> v1:172.21.15.195:6789/0 conn(0x7f764806f620 legacy=0x7f764806d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:08.458 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.453+0000 7f764d98f700 1 -- 172.21.15.195:0/3435493941 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76480ba4d0 con 0x7f7648150580 2022-06-27T11:48:08.458 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.453+0000 7f764c98d700 1 -- 172.21.15.195:0/3435493941 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1312664550 0 0) 0x7f76480c76b0 con 0x7f764806f620 2022-06-27T11:48:08.458 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.453+0000 7f764d98f700 1 --2- 172.21.15.195:0/3435493941 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7648150580 0x7f7648156ef0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f76440097a0 tx=0x7f7644009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b9684731f49d240 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:08.458 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.454+0000 7f764c98d700 1 -- 172.21.15.195:0/3435493941 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7644011630 con 0x7f7648150580 2022-06-27T11:48:08.459 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.454+0000 7f764c98d700 1 -- 172.21.15.195:0/3435493941 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7644011790 con 0x7f7648150580 2022-06-27T11:48:08.459 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.454+0000 7f764c98d700 1 -- 172.21.15.195:0/3435493941 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7644011b30 con 0x7f7648150580 2022-06-27T11:48:08.459 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.454+0000 7f764fbf3700 1 -- 172.21.15.195:0/3435493941 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7648150580 msgr2=0x7f7648156ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:08.459 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.454+0000 7f764fbf3700 1 --2- 172.21.15.195:0/3435493941 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7648150580 0x7f7648156ef0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f76440097a0 tx=0x7f7644009060 comp rx=0 tx=0).stop 2022-06-27T11:48:08.460 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.454+0000 7f764fbf3700 1 -- 172.21.15.195:0/3435493941 shutdown_connections 2022-06-27T11:48:08.460 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.454+0000 7f764fbf3700 1 --2- 172.21.15.195:0/3435493941 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7648150580 0x7f7648156ef0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:08.460 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.454+0000 7f764fbf3700 1 --2- 172.21.15.195:0/3435493941 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7648150950 0x7f76480c7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:08.461 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.454+0000 7f764fbf3700 1 -- 172.21.15.195:0/3435493941 >> 172.21.15.195:0/3435493941 conn(0x7f7648063af0 msgr2=0x7f7648063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:08.461 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.454+0000 7f764fbf3700 1 -- 172.21.15.195:0/3435493941 shutdown_connections 2022-06-27T11:48:08.461 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.454+0000 7f764fbf3700 1 -- 172.21.15.195:0/3435493941 wait complete. 2022-06-27T11:48:08.461 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.455+0000 7f764fbf3700 1 Processor -- start 2022-06-27T11:48:08.462 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.455+0000 7f764fbf3700 1 -- start start 2022-06-27T11:48:08.465 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.456+0000 7f764fbf3700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f764806f620 0x7f76480bb5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:08.465 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.456+0000 7f764fbf3700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7648150580 0x7f76480bdc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:08.466 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.456+0000 7f764fbf3700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7648150950 0x7f7648069da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:08.466 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.456+0000 7f764fbf3700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f764816a900 con 0x7f764806f620 2022-06-27T11:48:08.466 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.456+0000 7f764fbf3700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f764816ad20 con 0x7f7648150950 2022-06-27T11:48:08.466 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.456+0000 7f764fbf3700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76480c4dc0 con 0x7f7648150580 2022-06-27T11:48:08.467 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.456+0000 7f764d98f700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f764806f620 0x7f76480bb5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:08.467 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.456+0000 7f764d98f700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f764806f620 0x7f76480bb5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41936/0 (socket says 172.21.15.195:41936) 2022-06-27T11:48:08.467 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.456+0000 7f764d98f700 1 -- 172.21.15.195:0/2503897621 learned_addr learned my addr 172.21.15.195:0/2503897621 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:48:08.467 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.456+0000 7f764d18e700 1 --2- 172.21.15.195:0/2503897621 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7648150580 0x7f76480bdc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:08.468 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.456+0000 7f764e190700 1 --2- 172.21.15.195:0/2503897621 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7648150950 0x7f7648069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:08.468 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.456+0000 7f764d98f700 1 -- 172.21.15.195:0/2503897621 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7648150580 msgr2=0x7f76480bdc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:08.468 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.456+0000 7f764d98f700 1 --2- 172.21.15.195:0/2503897621 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7648150580 0x7f76480bdc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:08.468 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.456+0000 7f764d98f700 1 -- 172.21.15.195:0/2503897621 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7648150950 msgr2=0x7f7648069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:08.469 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.456+0000 7f764d98f700 1 --2- 172.21.15.195:0/2503897621 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7648150950 0x7f7648069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:08.469 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.456+0000 7f764d98f700 1 -- 172.21.15.195:0/2503897621 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76440092e0 con 0x7f764806f620 2022-06-27T11:48:08.470 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.456+0000 7f764d18e700 1 --2- 172.21.15.195:0/2503897621 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7648150580 0x7f76480bdc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T11:48:08.470 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.456+0000 7f764e190700 1 --2- 172.21.15.195:0/2503897621 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7648150950 0x7f7648069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T11:48:08.470 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.457+0000 7f764d98f700 1 --2- 172.21.15.195:0/2503897621 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f764806f620 0x7f76480bb5b0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f7644011c60 tx=0x7f7644017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:08.471 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.457+0000 7f7635ffb700 1 -- 172.21.15.195:0/2503897621 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7644003900 con 0x7f764806f620 2022-06-27T11:48:08.471 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.457+0000 7f7635ffb700 1 -- 172.21.15.195:0/2503897621 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7644003a60 con 0x7f764806f620 2022-06-27T11:48:08.471 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.457+0000 7f764fbf3700 1 -- 172.21.15.195:0/2503897621 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76480c4b20 con 0x7f764806f620 2022-06-27T11:48:08.471 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.457+0000 7f7635ffb700 1 -- 172.21.15.195:0/2503897621 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7644003e20 con 0x7f764806f620 2022-06-27T11:48:08.471 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.457+0000 7f764fbf3700 1 -- 172.21.15.195:0/2503897621 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7648169f60 con 0x7f764806f620 2022-06-27T11:48:08.472 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.458+0000 7f764fbf3700 1 -- 172.21.15.195:0/2503897621 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f764816ab70 con 0x7f764806f620 2022-06-27T11:48:08.472 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.458+0000 7f7635ffb700 1 -- 172.21.15.195:0/2503897621 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f764402a020 con 0x7f764806f620 2022-06-27T11:48:08.472 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.459+0000 7f7635ffb700 1 --2- 172.21.15.195:0/2503897621 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7628063d00 0x7f76280661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:08.472 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.459+0000 7f7635ffb700 1 -- 172.21.15.195:0/2503897621 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f7644082080 con 0x7f764806f620 2022-06-27T11:48:08.472 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.459+0000 7f764d18e700 1 --2- 172.21.15.195:0/2503897621 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7628063d00 0x7f76280661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:08.473 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.459+0000 7f764d18e700 1 --2- 172.21.15.195:0/2503897621 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7628063d00 0x7f76280661b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7638006fd0 tx=0x7f7638008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:08.473 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.463+0000 7f7635ffb700 1 -- 172.21.15.195:0/2503897621 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f76440b1c90 con 0x7f764806f620 2022-06-27T11:48:08.640 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.641+0000 7f764fbf3700 1 -- 172.21.15.195:0/2503897621 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f76480722c0 con 0x7f764806f620 2022-06-27T11:48:08.651 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.641+0000 7f7635ffb700 1 -- 172.21.15.195:0/2503897621 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7644050b20 con 0x7f764806f620 2022-06-27T11:48:08.652 INFO:teuthology.orchestra.run.smithi195.stdout:pg_num: 1 2022-06-27T11:48:08.655 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.643+0000 7f764fbf3700 1 -- 172.21.15.195:0/2503897621 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7628063d00 msgr2=0x7f76280661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:08.656 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.643+0000 7f764fbf3700 1 --2- 172.21.15.195:0/2503897621 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7628063d00 0x7f76280661b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7638006fd0 tx=0x7f7638008040 comp rx=0 tx=0).stop 2022-06-27T11:48:08.656 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.643+0000 7f764fbf3700 1 -- 172.21.15.195:0/2503897621 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f764806f620 msgr2=0x7f76480bb5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:08.657 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.643+0000 7f764fbf3700 1 --2- 172.21.15.195:0/2503897621 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f764806f620 0x7f76480bb5b0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f7644011c60 tx=0x7f7644017000 comp rx=0 tx=0).stop 2022-06-27T11:48:08.657 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.643+0000 7f764fbf3700 1 -- 172.21.15.195:0/2503897621 shutdown_connections 2022-06-27T11:48:08.658 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.643+0000 7f764fbf3700 1 --2- 172.21.15.195:0/2503897621 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7628063d00 0x7f76280661b0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:08.658 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.643+0000 7f764fbf3700 1 --2- 172.21.15.195:0/2503897621 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f764806f620 0x7f76480bb5b0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:08.659 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.643+0000 7f764fbf3700 1 --2- 172.21.15.195:0/2503897621 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7648150950 0x7f7648069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:08.661 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.643+0000 7f764fbf3700 1 --2- 172.21.15.195:0/2503897621 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7648150580 0x7f76480bdc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:08.661 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.643+0000 7f764fbf3700 1 -- 172.21.15.195:0/2503897621 >> 172.21.15.195:0/2503897621 conn(0x7f7648063af0 msgr2=0x7f764814e120 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:08.662 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.643+0000 7f764fbf3700 1 -- 172.21.15.195:0/2503897621 shutdown_connections 2022-06-27T11:48:08.662 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.643+0000 7f764fbf3700 1 -- 172.21.15.195:0/2503897621 wait complete. 2022-06-27T11:48:08.663 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-06-27T11:48:08.801 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.788+0000 7f9b91af7700 1 Processor -- start 2022-06-27T11:48:08.801 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.788+0000 7f9b91af7700 1 -- start start 2022-06-27T11:48:08.802 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.789+0000 7f9b91af7700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f9b8c156100 0x7f9b8c0688a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:08.802 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.789+0000 7f9b91af7700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9b8c06a440 0x7f9b8c0c4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:08.803 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.789+0000 7f9b91af7700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f9b8c1564d0 0x7f9b8c156950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:08.803 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.789+0000 7f9b91af7700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b8c0c3dc0 con 0x7f9b8c1564d0 2022-06-27T11:48:08.804 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.789+0000 7f9b91af7700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b8c0c4b10 con 0x7f9b8c156100 2022-06-27T11:48:08.804 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.789+0000 7f9b91af7700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b8c068f20 con 0x7f9b8c06a440 2022-06-27T11:48:08.805 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.789+0000 7f9b8bfff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f9b8c1564d0 0x7f9b8c156950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:08.807 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.789+0000 7f9b8bfff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f9b8c1564d0 0x7f9b8c156950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41948/0 (socket says 172.21.15.195:41948) 2022-06-27T11:48:08.807 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.789+0000 7f9b8bfff700 1 -- 172.21.15.195:0/868180448 learned_addr learned my addr 172.21.15.195:0/868180448 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:48:08.808 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.789+0000 7f9b8b7fe700 1 --2- 172.21.15.195:0/868180448 >> v2:172.21.15.202:3300/0 conn(0x7f9b8c156100 0x7f9b8c0688a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:08.808 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.789+0000 7f9b8affd700 1 --2- 172.21.15.195:0/868180448 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9b8c06a440 0x7f9b8c0c4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:08.809 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.789+0000 7f9b8bfff700 1 -- 172.21.15.195:0/868180448 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9b8c06a440 msgr2=0x7f9b8c0c4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:08.809 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.789+0000 7f9b8bfff700 1 --2- 172.21.15.195:0/868180448 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9b8c06a440 0x7f9b8c0c4fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:08.809 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.789+0000 7f9b8bfff700 1 -- 172.21.15.195:0/868180448 >> v2:172.21.15.202:3300/0 conn(0x7f9b8c156100 msgr2=0x7f9b8c0688a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:08.810 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.789+0000 7f9b8bfff700 1 --2- 172.21.15.195:0/868180448 >> v2:172.21.15.202:3300/0 conn(0x7f9b8c156100 0x7f9b8c0688a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:08.810 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.789+0000 7f9b8bfff700 1 -- 172.21.15.195:0/868180448 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b8c0c6360 con 0x7f9b8c1564d0 2022-06-27T11:48:08.811 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.789+0000 7f9b8bfff700 1 --2- 172.21.15.195:0/868180448 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9b8c1564d0 0x7f9b8c156950 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c070770 tx=0x7f9b84009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ebd8c7801adbc86 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:08.811 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.790+0000 7f9b8a7fc700 1 -- 172.21.15.195:0/868180448 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b8400e070 con 0x7f9b8c1564d0 2022-06-27T11:48:08.812 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.790+0000 7f9b8a7fc700 1 -- 172.21.15.195:0/868180448 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b8400b970 con 0x7f9b8c1564d0 2022-06-27T11:48:08.812 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.790+0000 7f9b8a7fc700 1 -- 172.21.15.195:0/868180448 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b840092e0 con 0x7f9b8c1564d0 2022-06-27T11:48:08.813 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.790+0000 7f9b91af7700 1 -- 172.21.15.195:0/868180448 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9b8c1564d0 msgr2=0x7f9b8c156950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:08.813 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.790+0000 7f9b91af7700 1 --2- 172.21.15.195:0/868180448 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9b8c1564d0 0x7f9b8c156950 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c070770 tx=0x7f9b84009840 comp rx=0 tx=0).stop 2022-06-27T11:48:08.813 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.790+0000 7f9b91af7700 1 -- 172.21.15.195:0/868180448 shutdown_connections 2022-06-27T11:48:08.814 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.790+0000 7f9b91af7700 1 --2- 172.21.15.195:0/868180448 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9b8c1564d0 0x7f9b8c156950 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:08.814 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.790+0000 7f9b91af7700 1 --2- 172.21.15.195:0/868180448 >> v2:172.21.15.202:3300/0 conn(0x7f9b8c156100 0x7f9b8c0688a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:08.815 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.790+0000 7f9b91af7700 1 --2- 172.21.15.195:0/868180448 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9b8c06a440 0x7f9b8c0c4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:08.815 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.790+0000 7f9b91af7700 1 -- 172.21.15.195:0/868180448 >> 172.21.15.195:0/868180448 conn(0x7f9b8c0c9b00 msgr2=0x7f9b8c0c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:08.816 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.790+0000 7f9b91af7700 1 -- 172.21.15.195:0/868180448 shutdown_connections 2022-06-27T11:48:08.816 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.790+0000 7f9b91af7700 1 -- 172.21.15.195:0/868180448 wait complete. 2022-06-27T11:48:08.816 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.791+0000 7f9b91af7700 1 Processor -- start 2022-06-27T11:48:08.818 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.791+0000 7f9b91af7700 1 -- start start 2022-06-27T11:48:08.818 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.792+0000 7f9b91af7700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9b8c06a440 0x7f9b8c146eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:08.819 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.792+0000 7f9b91af7700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9b8c156100 0x7f9b8c1696f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:08.819 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.792+0000 7f9b91af7700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9b8c1564d0 0x7f9b8c0c2690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:08.819 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.792+0000 7f9b91af7700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b8c15c280 con 0x7f9b8c1564d0 2022-06-27T11:48:08.820 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.792+0000 7f9b91af7700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b8c166b60 con 0x7f9b8c156100 2022-06-27T11:48:08.820 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.792+0000 7f9b91af7700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b8c166ff0 con 0x7f9b8c06a440 2022-06-27T11:48:08.821 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.792+0000 7f9b8bfff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9b8c1564d0 0x7f9b8c0c2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:08.821 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.792+0000 7f9b8bfff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9b8c1564d0 0x7f9b8c0c2690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41954/0 (socket says 172.21.15.195:41954) 2022-06-27T11:48:08.822 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.792+0000 7f9b8bfff700 1 -- 172.21.15.195:0/3485919256 learned_addr learned my addr 172.21.15.195:0/3485919256 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:48:08.822 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.792+0000 7f9b8b7fe700 1 --2- 172.21.15.195:0/3485919256 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9b8c06a440 0x7f9b8c146eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:08.823 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.792+0000 7f9b8affd700 1 --2- 172.21.15.195:0/3485919256 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9b8c156100 0x7f9b8c1696f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:08.823 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.792+0000 7f9b8bfff700 1 -- 172.21.15.195:0/3485919256 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9b8c06a440 msgr2=0x7f9b8c146eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:08.823 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.792+0000 7f9b8bfff700 1 --2- 172.21.15.195:0/3485919256 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9b8c06a440 0x7f9b8c146eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:08.824 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.792+0000 7f9b8bfff700 1 -- 172.21.15.195:0/3485919256 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9b8c156100 msgr2=0x7f9b8c1696f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:08.824 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.792+0000 7f9b8bfff700 1 --2- 172.21.15.195:0/3485919256 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9b8c156100 0x7f9b8c1696f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:08.825 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.792+0000 7f9b8bfff700 1 -- 172.21.15.195:0/3485919256 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b84009ae0 con 0x7f9b8c1564d0 2022-06-27T11:48:08.825 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.793+0000 7f9b8bfff700 1 --2- 172.21.15.195:0/3485919256 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9b8c1564d0 0x7f9b8c0c2690 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9b8400b610 tx=0x7f9b84018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:08.826 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.793+0000 7f9b737fe700 1 -- 172.21.15.195:0/3485919256 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b84009940 con 0x7f9b8c1564d0 2022-06-27T11:48:08.826 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.793+0000 7f9b737fe700 1 -- 172.21.15.195:0/3485919256 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b84004250 con 0x7f9b8c1564d0 2022-06-27T11:48:08.827 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.793+0000 7f9b91af7700 1 -- 172.21.15.195:0/3485919256 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b8c15c4b0 con 0x7f9b8c1564d0 2022-06-27T11:48:08.827 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.793+0000 7f9b737fe700 1 -- 172.21.15.195:0/3485919256 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b840045f0 con 0x7f9b8c1564d0 2022-06-27T11:48:08.827 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.793+0000 7f9b91af7700 1 -- 172.21.15.195:0/3485919256 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b8c166230 con 0x7f9b8c1564d0 2022-06-27T11:48:08.828 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.794+0000 7f9b91af7700 1 -- 172.21.15.195:0/3485919256 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b8c0722c0 con 0x7f9b8c1564d0 2022-06-27T11:48:08.828 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.794+0000 7f9b737fe700 1 -- 172.21.15.195:0/3485919256 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f9b8401d420 con 0x7f9b8c1564d0 2022-06-27T11:48:08.829 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.795+0000 7f9b737fe700 1 --2- 172.21.15.195:0/3485919256 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9b6c063d00 0x7f9b6c0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:08.829 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.795+0000 7f9b737fe700 1 -- 172.21.15.195:0/3485919256 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f9b84082700 con 0x7f9b8c1564d0 2022-06-27T11:48:08.829 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.795+0000 7f9b8b7fe700 1 --2- 172.21.15.195:0/3485919256 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9b6c063d00 0x7f9b6c0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:08.829 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.795+0000 7f9b8b7fe700 1 --2- 172.21.15.195:0/3485919256 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9b6c063d00 0x7f9b6c0661b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9b80009c00 tx=0x7f9b80008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:08.830 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.799+0000 7f9b737fe700 1 -- 172.21.15.195:0/3485919256 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f9b84051290 con 0x7f9b8c1564d0 2022-06-27T11:48:08.980 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.980+0000 7f9b91af7700 1 -- 172.21.15.195:0/3485919256 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9b8c0c6640 con 0x7f9b8c1564d0 2022-06-27T11:48:08.980 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.980+0000 7f9b737fe700 1 -- 172.21.15.195:0/3485919256 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9b840548b0 con 0x7f9b8c1564d0 2022-06-27T11:48:08.981 INFO:teuthology.orchestra.run.smithi195.stdout:pg_num: 32 2022-06-27T11:48:08.984 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.982+0000 7f9b91af7700 1 -- 172.21.15.195:0/3485919256 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9b6c063d00 msgr2=0x7f9b6c0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:08.984 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.982+0000 7f9b91af7700 1 --2- 172.21.15.195:0/3485919256 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9b6c063d00 0x7f9b6c0661b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9b80009c00 tx=0x7f9b80008040 comp rx=0 tx=0).stop 2022-06-27T11:48:08.984 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.982+0000 7f9b91af7700 1 -- 172.21.15.195:0/3485919256 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9b8c1564d0 msgr2=0x7f9b8c0c2690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:08.985 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.982+0000 7f9b91af7700 1 --2- 172.21.15.195:0/3485919256 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9b8c1564d0 0x7f9b8c0c2690 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9b8400b610 tx=0x7f9b84018000 comp rx=0 tx=0).stop 2022-06-27T11:48:08.985 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.982+0000 7f9b91af7700 1 -- 172.21.15.195:0/3485919256 shutdown_connections 2022-06-27T11:48:08.986 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.983+0000 7f9b91af7700 1 --2- 172.21.15.195:0/3485919256 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9b6c063d00 0x7f9b6c0661b0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:08.986 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.983+0000 7f9b91af7700 1 --2- 172.21.15.195:0/3485919256 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9b8c1564d0 0x7f9b8c0c2690 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:08.988 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.983+0000 7f9b91af7700 1 --2- 172.21.15.195:0/3485919256 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9b8c06a440 0x7f9b8c146eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:08.989 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.983+0000 7f9b91af7700 1 --2- 172.21.15.195:0/3485919256 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9b8c156100 0x7f9b8c1696f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:08.989 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.983+0000 7f9b91af7700 1 -- 172.21.15.195:0/3485919256 >> 172.21.15.195:0/3485919256 conn(0x7f9b8c0c9b00 msgr2=0x7f9b8c06f000 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:08.989 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.983+0000 7f9b91af7700 1 -- 172.21.15.195:0/3485919256 shutdown_connections 2022-06-27T11:48:08.990 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:08.983+0000 7f9b91af7700 1 -- 172.21.15.195:0/3485919256 wait complete. 2022-06-27T11:48:08.995 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-06-27T11:48:09.120 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.120+0000 7fada86e3700 1 Processor -- start 2022-06-27T11:48:09.121 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.120+0000 7fada86e3700 1 -- start start 2022-06-27T11:48:09.129 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.121+0000 7fada86e3700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fada01508d0 0x7fada0063af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:09.130 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.121+0000 7fada86e3700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fada0064030 con 0x7fada006a450 2022-06-27T11:48:09.130 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.121+0000 7fada86e3700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fada00c4160 con 0x7fada0150500 2022-06-27T11:48:09.130 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.121+0000 7fada86e3700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fada00c4440 con 0x7fada01508d0 2022-06-27T11:48:09.131 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.121+0000 7fada5c7e700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fada006a450 0x7fada00bbf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:48146/0 (socket says 172.21.15.195:48146) 2022-06-27T11:48:09.131 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.121+0000 7fada5c7e700 1 -- 172.21.15.195:0/3772850087 learned_addr learned my addr 172.21.15.195:0/3772850087 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:48:09.131 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.121+0000 7fada6c80700 1 --2- 172.21.15.195:0/3772850087 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fada01508d0 0x7fada0063af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:09.131 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.121+0000 7fada547d700 1 -- 172.21.15.195:0/3772850087 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2945248509 0 0) 0x7fad9c001110 con 0x7fada006a450 2022-06-27T11:48:09.132 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.121+0000 7fada547d700 1 -- 172.21.15.195:0/3772850087 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad88003430 con 0x7fada006a450 2022-06-27T11:48:09.132 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.121+0000 7fada547d700 1 -- 172.21.15.195:0/3772850087 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2704534595 0 0) 0x7fada00c4160 con 0x7fada0150500 2022-06-27T11:48:09.132 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.121+0000 7fada547d700 1 -- 172.21.15.195:0/3772850087 --> v1:172.21.15.202:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad9c001110 con 0x7fada0150500 2022-06-27T11:48:09.132 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.121+0000 7fada547d700 1 -- 172.21.15.195:0/3772850087 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3688356434 0 0) 0x7fad88003430 con 0x7fada006a450 2022-06-27T11:48:09.133 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.121+0000 7fada6c80700 1 -- 172.21.15.195:0/3772850087 >> v1:172.21.15.202:6789/0 conn(0x7fada0150500 legacy=0x7fada00c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:09.133 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.121+0000 7fada6c80700 1 -- 172.21.15.195:0/3772850087 >> v1:172.21.15.195:6789/0 conn(0x7fada006a450 legacy=0x7fada00bbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:09.133 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.121+0000 7fada6c80700 1 -- 172.21.15.195:0/3772850087 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fada015a800 con 0x7fada01508d0 2022-06-27T11:48:09.133 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.121+0000 7fada6c80700 1 --2- 172.21.15.195:0/3772850087 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fada01508d0 0x7fada0063af0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fad98009e90 tx=0x7fad98009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2616412fe5cad098 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:09.134 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.122+0000 7fada547d700 1 -- 172.21.15.195:0/3772850087 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad9800e070 con 0x7fada01508d0 2022-06-27T11:48:09.134 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.122+0000 7fada547d700 1 -- 172.21.15.195:0/3772850087 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad980038a0 con 0x7fada01508d0 2022-06-27T11:48:09.134 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.122+0000 7fada547d700 1 -- 172.21.15.195:0/3772850087 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad9c001600 con 0x7fada01508d0 2022-06-27T11:48:09.135 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.122+0000 7fada86e3700 1 -- 172.21.15.195:0/3772850087 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fada01508d0 msgr2=0x7fada0063af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:09.135 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.122+0000 7fada86e3700 1 --2- 172.21.15.195:0/3772850087 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fada01508d0 0x7fada0063af0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fad98009e90 tx=0x7fad98009780 comp rx=0 tx=0).stop 2022-06-27T11:48:09.136 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.122+0000 7fada86e3700 1 -- 172.21.15.195:0/3772850087 shutdown_connections 2022-06-27T11:48:09.136 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.122+0000 7fada86e3700 1 --2- 172.21.15.195:0/3772850087 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fada01508d0 0x7fada0063af0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:09.136 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.122+0000 7fada86e3700 1 -- 172.21.15.195:0/3772850087 >> 172.21.15.195:0/3772850087 conn(0x7fada00c9b00 msgr2=0x7fada00c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:09.137 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.122+0000 7fada86e3700 1 -- 172.21.15.195:0/3772850087 shutdown_connections 2022-06-27T11:48:09.137 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.122+0000 7fada86e3700 1 -- 172.21.15.195:0/3772850087 wait complete. 2022-06-27T11:48:09.137 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.123+0000 7fada86e3700 1 Processor -- start 2022-06-27T11:48:09.137 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.123+0000 7fada86e3700 1 -- start start 2022-06-27T11:48:09.138 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.124+0000 7fada86e3700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fada006a450 0x7fada0067ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:09.138 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.124+0000 7fada86e3700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fada0150500 0x7fada0167050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:09.138 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.124+0000 7fada86e3700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fada01508d0 0x7fada0149d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:09.139 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.124+0000 7fada86e3700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fada006a2a0 con 0x7fada0150500 2022-06-27T11:48:09.139 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.124+0000 7fada86e3700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fada01665d0 con 0x7fada006a450 2022-06-27T11:48:09.139 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.124+0000 7fada86e3700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fada0157700 con 0x7fada01508d0 2022-06-27T11:48:09.139 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.124+0000 7fada5c7e700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fada0150500 0x7fada0167050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:09.139 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.124+0000 7fada5c7e700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fada0150500 0x7fada0167050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41964/0 (socket says 172.21.15.195:41964) 2022-06-27T11:48:09.140 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.124+0000 7fada5c7e700 1 -- 172.21.15.195:0/578823299 learned_addr learned my addr 172.21.15.195:0/578823299 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:48:09.140 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.124+0000 7fada647f700 1 --2- 172.21.15.195:0/578823299 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fada006a450 0x7fada0067ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:09.140 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.124+0000 7fada6c80700 1 --2- 172.21.15.195:0/578823299 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fada01508d0 0x7fada0149d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:09.140 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.124+0000 7fada5c7e700 1 -- 172.21.15.195:0/578823299 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fada01508d0 msgr2=0x7fada0149d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:09.141 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.124+0000 7fada5c7e700 1 --2- 172.21.15.195:0/578823299 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fada01508d0 0x7fada0149d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:09.141 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.124+0000 7fada5c7e700 1 -- 172.21.15.195:0/578823299 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fada006a450 msgr2=0x7fada0067ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:09.141 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.124+0000 7fada5c7e700 1 --2- 172.21.15.195:0/578823299 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fada006a450 0x7fada0067ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:09.142 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.124+0000 7fada5c7e700 1 -- 172.21.15.195:0/578823299 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad98009a20 con 0x7fada0150500 2022-06-27T11:48:09.142 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.124+0000 7fada647f700 1 --2- 172.21.15.195:0/578823299 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fada006a450 0x7fada0067ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T11:48:09.143 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.125+0000 7fada5c7e700 1 --2- 172.21.15.195:0/578823299 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fada0150500 0x7fada0167050 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fad9c00c960 tx=0x7fad9c001c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:09.143 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.125+0000 7fad8e7fc700 1 -- 172.21.15.195:0/578823299 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad9c007360 con 0x7fada0150500 2022-06-27T11:48:09.143 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.125+0000 7fad8e7fc700 1 -- 172.21.15.195:0/578823299 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad9c0074c0 con 0x7fada0150500 2022-06-27T11:48:09.144 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.125+0000 7fada86e3700 1 -- 172.21.15.195:0/578823299 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fada014f8a0 con 0x7fada0150500 2022-06-27T11:48:09.144 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.125+0000 7fad8e7fc700 1 -- 172.21.15.195:0/578823299 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad9c004940 con 0x7fada0150500 2022-06-27T11:48:09.144 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.125+0000 7fada86e3700 1 -- 172.21.15.195:0/578823299 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fada014fe60 con 0x7fada0150500 2022-06-27T11:48:09.144 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.126+0000 7fada86e3700 1 -- 172.21.15.195:0/578823299 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fada00722c0 con 0x7fada0150500 2022-06-27T11:48:09.145 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.126+0000 7fad8e7fc700 1 -- 172.21.15.195:0/578823299 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fad9c01a030 con 0x7fada0150500 2022-06-27T11:48:09.145 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.126+0000 7fad8e7fc700 1 --2- 172.21.15.195:0/578823299 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fad80063d00 0x7fad800661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:09.145 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.126+0000 7fad8e7fc700 1 -- 172.21.15.195:0/578823299 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fad9c07fe00 con 0x7fada0150500 2022-06-27T11:48:09.145 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.127+0000 7fada647f700 1 --2- 172.21.15.195:0/578823299 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fad80063d00 0x7fad800661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:09.146 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.127+0000 7fada647f700 1 --2- 172.21.15.195:0/578823299 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fad80063d00 0x7fad800661b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fad90001170 tx=0x7fad90009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:09.147 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.131+0000 7fad8e7fc700 1 -- 172.21.15.195:0/578823299 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fad9c04ea70 con 0x7fada0150500 2022-06-27T11:48:09.308 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.309+0000 7fada86e3700 1 -- 172.21.15.195:0/578823299 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fada0064030 con 0x7fada0150500 2022-06-27T11:48:09.309 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.309+0000 7fad8e7fc700 1 -- 172.21.15.195:0/578823299 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fada0064030 con 0x7fada0150500 2022-06-27T11:48:09.309 INFO:teuthology.orchestra.run.smithi195.stdout:pg_num: 64 2022-06-27T11:48:09.312 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.311+0000 7fada86e3700 1 -- 172.21.15.195:0/578823299 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fad80063d00 msgr2=0x7fad800661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:09.312 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.311+0000 7fada86e3700 1 --2- 172.21.15.195:0/578823299 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fad80063d00 0x7fad800661b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fad90001170 tx=0x7fad90009040 comp rx=0 tx=0).stop 2022-06-27T11:48:09.313 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.311+0000 7fada86e3700 1 -- 172.21.15.195:0/578823299 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fada0150500 msgr2=0x7fada0167050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:09.314 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.311+0000 7fada86e3700 1 --2- 172.21.15.195:0/578823299 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fada0150500 0x7fada0167050 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fad9c00c960 tx=0x7fad9c001c60 comp rx=0 tx=0).stop 2022-06-27T11:48:09.314 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.311+0000 7fada86e3700 1 -- 172.21.15.195:0/578823299 shutdown_connections 2022-06-27T11:48:09.315 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.311+0000 7fada86e3700 1 --2- 172.21.15.195:0/578823299 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fad80063d00 0x7fad800661b0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:09.315 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.311+0000 7fada86e3700 1 --2- 172.21.15.195:0/578823299 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fada01508d0 0x7fada0149d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:09.316 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.311+0000 7fada86e3700 1 --2- 172.21.15.195:0/578823299 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fada006a450 0x7fada0067ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:09.316 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.311+0000 7fada86e3700 1 --2- 172.21.15.195:0/578823299 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fada0150500 0x7fada0167050 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:09.317 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.311+0000 7fada86e3700 1 -- 172.21.15.195:0/578823299 >> 172.21.15.195:0/578823299 conn(0x7fada00c9b00 msgr2=0x7fada014b040 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:09.317 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.311+0000 7fada86e3700 1 -- 172.21.15.195:0/578823299 shutdown_connections 2022-06-27T11:48:09.317 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.311+0000 7fada86e3700 1 -- 172.21.15.195:0/578823299 wait complete. 2022-06-27T11:48:09.323 INFO:tasks.mds_thrash:mds thrasher using random seed: 1656330489 2022-06-27T11:48:09.323 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-27T11:48:09.444 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.444+0000 7fd9afb23700 1 Processor -- start 2022-06-27T11:48:09.444 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.444+0000 7fd9afb23700 1 -- start start 2022-06-27T11:48:09.453 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.445+0000 7fd9afb23700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd9a806fef0 0x7fd9a81570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:09.453 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.445+0000 7fd9afb23700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fd9a80c34a0 0x7fd9a80c2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:09.454 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.445+0000 7fd9afb23700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9a806fba0 con 0x7fd9a806e850 2022-06-27T11:48:09.454 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.445+0000 7fd9afb23700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9a81575e0 con 0x7fd9a80c34a0 2022-06-27T11:48:09.454 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.445+0000 7fd9afb23700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a806f5e0 con 0x7fd9a806fef0 2022-06-27T11:48:09.455 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.445+0000 7fd9ad8bf700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fd9a806e850 0x7fd9a8069de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:48162/0 (socket says 172.21.15.195:48162) 2022-06-27T11:48:09.455 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.445+0000 7fd9ad8bf700 1 -- 172.21.15.195:0/1836639072 learned_addr learned my addr 172.21.15.195:0/1836639072 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:48:09.455 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.445+0000 7fd9ad0be700 1 --2- 172.21.15.195:0/1836639072 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd9a806fef0 0x7fd9a81570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:09.455 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.445+0000 7fd9ae0c0700 1 --2- 172.21.15.195:0/1836639072 >> v2:172.21.15.202:3300/0 conn(0x7fd9a80c34a0 0x7fd9a80c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:09.455 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.445+0000 7fd9ac8bd700 1 -- 172.21.15.195:0/1836639072 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3616625980 0 0) 0x7fd9a806fba0 con 0x7fd9a806e850 2022-06-27T11:48:09.456 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.445+0000 7fd9ac8bd700 1 -- 172.21.15.195:0/1836639072 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd990003430 con 0x7fd9a806e850 2022-06-27T11:48:09.456 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.446+0000 7fd9ac8bd700 1 -- 172.21.15.195:0/1836639072 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2645549040 0 0) 0x7fd990003430 con 0x7fd9a806e850 2022-06-27T11:48:09.456 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.446+0000 7fd9ac8bd700 1 -- 172.21.15.195:0/1836639072 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd9a806fef0 msgr2=0x7fd9a81570a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:09.456 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.446+0000 7fd9ac8bd700 1 --2- 172.21.15.195:0/1836639072 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd9a806fef0 0x7fd9a81570a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:09.458 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.446+0000 7fd9ac8bd700 1 -- 172.21.15.195:0/1836639072 >> v2:172.21.15.202:3300/0 conn(0x7fd9a80c34a0 msgr2=0x7fd9a80c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:09.459 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.446+0000 7fd9ac8bd700 1 --2- 172.21.15.195:0/1836639072 >> v2:172.21.15.202:3300/0 conn(0x7fd9a80c34a0 0x7fd9a80c2920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:09.459 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.446+0000 7fd9ac8bd700 1 -- 172.21.15.195:0/1836639072 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9a806f8c0 con 0x7fd9a806e850 2022-06-27T11:48:09.459 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.446+0000 7fd9ac8bd700 1 -- 172.21.15.195:0/1836639072 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7fd998001470 con 0x7fd9a806e850 2022-06-27T11:48:09.459 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.446+0000 7fd9ac8bd700 1 -- 172.21.15.195:0/1836639072 >> v1:172.21.15.195:6789/0 conn(0x7fd9a806e850 legacy=0x7fd9a8069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:09.460 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.446+0000 7fd9ac8bd700 1 --2- 172.21.15.195:0/1836639072 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd990003920 0x7fd990003d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:09.460 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.446+0000 7fd9ac8bd700 1 --2- 172.21.15.195:0/1836639072 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd9900046d0 0x7fd990004b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:09.460 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.446+0000 7fd9ac8bd700 1 --2- 172.21.15.195:0/1836639072 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd990005040 0x7fd99000b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:09.461 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.446+0000 7fd9ac8bd700 1 -- 172.21.15.195:0/1836639072 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a81575e0 con 0x7fd990005040 2022-06-27T11:48:09.461 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.446+0000 7fd9ac8bd700 1 -- 172.21.15.195:0/1836639072 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a806f5e0 con 0x7fd990003920 2022-06-27T11:48:09.461 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.446+0000 7fd9ac8bd700 1 -- 172.21.15.195:0/1836639072 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd99000be60 con 0x7fd9900046d0 2022-06-27T11:48:09.461 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.446+0000 7fd9ae0c0700 1 --2- 172.21.15.195:0/1836639072 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd990005040 0x7fd99000b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:09.462 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.446+0000 7fd9ad8bf700 1 --2- 172.21.15.195:0/1836639072 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd990003920 0x7fd990003d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:09.463 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.446+0000 7fd9ad0be700 1 --2- 172.21.15.195:0/1836639072 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd9900046d0 0x7fd990004b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:09.463 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.446+0000 7fd9ae0c0700 1 -- 172.21.15.195:0/1836639072 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd9900046d0 msgr2=0x7fd990004b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:09.463 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.446+0000 7fd9ae0c0700 1 --2- 172.21.15.195:0/1836639072 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd9900046d0 0x7fd990004b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:09.463 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.446+0000 7fd9ae0c0700 1 -- 172.21.15.195:0/1836639072 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd990003920 msgr2=0x7fd990003d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:09.464 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.446+0000 7fd9ae0c0700 1 --2- 172.21.15.195:0/1836639072 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd990003920 0x7fd990003d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:09.464 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.446+0000 7fd9ae0c0700 1 -- 172.21.15.195:0/1836639072 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd990003430 con 0x7fd990005040 2022-06-27T11:48:09.464 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.446+0000 7fd9ae0c0700 1 --2- 172.21.15.195:0/1836639072 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd990005040 0x7fd99000b920 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fd9a000e240 tx=0x7fd9a000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dca37327eb30345f server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:09.464 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.446+0000 7fd9ac8bd700 1 -- 172.21.15.195:0/1836639072 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a000eaf0 con 0x7fd990005040 2022-06-27T11:48:09.464 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.446+0000 7fd9ac8bd700 1 -- 172.21.15.195:0/1836639072 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9a000f040 con 0x7fd990005040 2022-06-27T11:48:09.465 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.446+0000 7fd9ac8bd700 1 -- 172.21.15.195:0/1836639072 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a000eea0 con 0x7fd990005040 2022-06-27T11:48:09.465 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.447+0000 7fd9afb23700 1 -- 172.21.15.195:0/1836639072 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd990005040 msgr2=0x7fd99000b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:09.465 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.447+0000 7fd9afb23700 1 --2- 172.21.15.195:0/1836639072 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd990005040 0x7fd99000b920 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fd9a000e240 tx=0x7fd9a000b040 comp rx=0 tx=0).stop 2022-06-27T11:48:09.466 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.447+0000 7fd9ae0c0700 1 -- 172.21.15.195:0/1836639072 reap_dead start 2022-06-27T11:48:09.466 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.447+0000 7fd9afb23700 1 -- 172.21.15.195:0/1836639072 shutdown_connections 2022-06-27T11:48:09.466 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.447+0000 7fd9afb23700 1 -- 172.21.15.195:0/1836639072 >> 172.21.15.195:0/1836639072 conn(0x7fd9a80c6350 msgr2=0x7fd9a80c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:09.467 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.447+0000 7fd9afb23700 1 -- 172.21.15.195:0/1836639072 shutdown_connections 2022-06-27T11:48:09.468 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.447+0000 7fd9afb23700 1 -- 172.21.15.195:0/1836639072 wait complete. 2022-06-27T11:48:09.468 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.449+0000 7fd9afb23700 1 Processor -- start 2022-06-27T11:48:09.469 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.449+0000 7fd9afb23700 1 -- start start 2022-06-27T11:48:09.469 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.449+0000 7fd9afb23700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd9a81547f0 0x7fd9a8156d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:09.469 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.449+0000 7fd9afb23700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd9a80c0390 0x7fd9a8148470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:09.470 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.449+0000 7fd9afb23700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd9a8154bc0 0x7fd9a81489b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:09.470 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.449+0000 7fd9afb23700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a8148ef0 con 0x7fd9a80c0390 2022-06-27T11:48:09.470 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.449+0000 7fd9afb23700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a814b880 con 0x7fd9a81547f0 2022-06-27T11:48:09.471 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.449+0000 7fd9afb23700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a8149570 con 0x7fd9a8154bc0 2022-06-27T11:48:09.471 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.450+0000 7fd9ad0be700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd9a80c0390 0x7fd9a8148470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:09.471 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.450+0000 7fd9ad0be700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd9a80c0390 0x7fd9a8148470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41984/0 (socket says 172.21.15.195:41984) 2022-06-27T11:48:09.471 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.450+0000 7fd9ad0be700 1 -- 172.21.15.195:0/902946001 learned_addr learned my addr 172.21.15.195:0/902946001 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:48:09.472 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.450+0000 7fd9ad8bf700 1 --2- 172.21.15.195:0/902946001 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd9a81547f0 0x7fd9a8156d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:09.472 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.450+0000 7fd9ae0c0700 1 --2- 172.21.15.195:0/902946001 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd9a8154bc0 0x7fd9a81489b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:09.473 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.450+0000 7fd9ad0be700 1 -- 172.21.15.195:0/902946001 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd9a8154bc0 msgr2=0x7fd9a81489b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:09.473 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.450+0000 7fd9ad0be700 1 --2- 172.21.15.195:0/902946001 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd9a8154bc0 0x7fd9a81489b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:09.473 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.450+0000 7fd9ad0be700 1 -- 172.21.15.195:0/902946001 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd9a81547f0 msgr2=0x7fd9a8156d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:09.474 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.450+0000 7fd9ad0be700 1 --2- 172.21.15.195:0/902946001 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd9a81547f0 0x7fd9a8156d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:09.474 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.450+0000 7fd9ad0be700 1 -- 172.21.15.195:0/902946001 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9a8149850 con 0x7fd9a80c0390 2022-06-27T11:48:09.475 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.450+0000 7fd9ad8bf700 1 --2- 172.21.15.195:0/902946001 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd9a81547f0 0x7fd9a8156d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T11:48:09.475 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.450+0000 7fd9ae0c0700 1 --2- 172.21.15.195:0/902946001 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd9a8154bc0 0x7fd9a81489b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T11:48:09.475 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.450+0000 7fd9ad0be700 1 --2- 172.21.15.195:0/902946001 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd9a80c0390 0x7fd9a8148470 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fd9a40119d0 tx=0x7fd9a4009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:09.476 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.450+0000 7fd995ffb700 1 -- 172.21.15.195:0/902946001 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a4004240 con 0x7fd9a80c0390 2022-06-27T11:48:09.476 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.450+0000 7fd995ffb700 1 -- 172.21.15.195:0/902946001 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9a40043a0 con 0x7fd9a80c0390 2022-06-27T11:48:09.476 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.450+0000 7fd995ffb700 1 -- 172.21.15.195:0/902946001 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a400f940 con 0x7fd9a80c0390 2022-06-27T11:48:09.477 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.450+0000 7fd9afb23700 1 -- 172.21.15.195:0/902946001 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9a814b230 con 0x7fd9a80c0390 2022-06-27T11:48:09.477 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.450+0000 7fd9afb23700 1 -- 172.21.15.195:0/902946001 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9a814af30 con 0x7fd9a80c0390 2022-06-27T11:48:09.478 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.451+0000 7fd9afb23700 1 -- 172.21.15.195:0/902946001 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9a814ba30 con 0x7fd9a80c0390 2022-06-27T11:48:09.478 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.452+0000 7fd995ffb700 1 -- 172.21.15.195:0/902946001 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fd9a4024070 con 0x7fd9a80c0390 2022-06-27T11:48:09.478 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.452+0000 7fd995ffb700 1 --2- 172.21.15.195:0/902946001 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd988063d20 0x7fd9880661d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:09.479 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.452+0000 7fd995ffb700 1 -- 172.21.15.195:0/902946001 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fd9a4018030 con 0x7fd9a80c0390 2022-06-27T11:48:09.479 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.452+0000 7fd9ad8bf700 1 --2- 172.21.15.195:0/902946001 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd988063d20 0x7fd9880661d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:09.479 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.452+0000 7fd9ad8bf700 1 --2- 172.21.15.195:0/902946001 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd988063d20 0x7fd9880661d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd998000e10 tx=0x7fd998009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:09.480 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.456+0000 7fd995ffb700 1 -- 172.21.15.195:0/902946001 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fd9a814ba30 con 0x7fd9a80c0390 2022-06-27T11:48:09.632 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.632+0000 7fd9afb23700 1 -- 172.21.15.195:0/902946001 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd9a814ba30 con 0x7fd9a80c0390 2022-06-27T11:48:09.633 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.633+0000 7fd995ffb700 1 -- 172.21.15.195:0/902946001 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16914 (secure 0 0 0) 0x7fd9a814ba30 con 0x7fd9a80c0390 2022-06-27T11:48:09.633 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:48:09.634 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":21,"fsid":"81b3a1d3-6a2b-47e8-8d69-7027cfeae7af","created":"2022-06-27T11:47:35.094715+0000","modified":"2022-06-27T11:47:53.359270+0000","last_up_change":"2022-06-27T11:47:44.777459+0000","last_in_change":"2022-06-27T11:47:37.596818+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2022-06-27T11:47:46.368325+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2022-06-27T11:47:46.386132+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-06-27T11:47:50.901270+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-06-27T11:47:51.314719+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c64bc51f-d4f3-4bc2-806b-da0dc78d51b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6813","nonce":94217},{"type":"v1","addr":"172.21.15.195:6815","nonce":94217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6817","nonce":94217},{"type":"v1","addr":"172.21.15.195:6819","nonce":94217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":94217},{"type":"v1","addr":"172.21.15.195:6823","nonce":94217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6820","nonce":94217},{"type":"v1","addr":"172.21.15.195:6821","nonce":94217}]},"public_addr":"172.21.15.195:6815/94217","cluster_addr":"172.21.15.195:6819/94217","heartbeat_back_addr":"172.21.15.195:6823/94217","heartbeat_front_addr":"172.21.15.195:6821/94217","state":["exists","up"]},{"osd":1,"uuid":"bea71cdf-aa98-493f-9207-66d886a3e30e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":94223},{"type":"v1","addr":"172.21.15.195:6809","nonce":94223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":94223},{"type":"v1","addr":"172.21.15.195:6811","nonce":94223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6816","nonce":94223},{"type":"v1","addr":"172.21.15.195:6818","nonce":94223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":94223},{"type":"v1","addr":"172.21.15.195:6814","nonce":94223}]},"public_addr":"172.21.15.195:6809/94223","cluster_addr":"172.21.15.195:6811/94223","heartbeat_back_addr":"172.21.15.195:6818/94223","heartbeat_front_addr":"172.21.15.195:6814/94223","state":["exists","up"]},{"osd":2,"uuid":"bf6c87a0-a8ea-497a-9ce1-721285888065","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":94221},{"type":"v1","addr":"172.21.15.195:6801","nonce":94221}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":94221},{"type":"v1","addr":"172.21.15.195:6803","nonce":94221}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":94221},{"type":"v1","addr":"172.21.15.195:6807","nonce":94221}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":94221},{"type":"v1","addr":"172.21.15.195:6805","nonce":94221}]},"public_addr":"172.21.15.195:6801/94221","cluster_addr":"172.21.15.195:6803/94221","heartbeat_back_addr":"172.21.15.195:6807/94221","heartbeat_front_addr":"172.21.15.195:6805/94221","state":["exists","up"]},{"osd":3,"uuid":"e8aa47ce-af8c-47e7-9654-870cd53229f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":94232},{"type":"v1","addr":"172.21.15.195:6825","nonce":94232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":94232},{"type":"v1","addr":"172.21.15.195:6827","nonce":94232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":94232},{"type":"v1","addr":"172.21.15.195:6831","nonce":94232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":94232},{"type":"v1","addr":"172.21.15.195:6829","nonce":94232}]},"public_addr":"172.21.15.195:6825/94232","cluster_addr":"172.21.15.195:6827/94232","heartbeat_back_addr":"172.21.15.195:6831/94232","heartbeat_front_addr":"172.21.15.195:6829/94232","state":["exists","up"]},{"osd":4,"uuid":"99ffa553-f149-443a-98f5-68b10f77e085","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6800","nonce":94308},{"type":"v1","addr":"172.21.15.202:6801","nonce":94308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6802","nonce":94308},{"type":"v1","addr":"172.21.15.202:6803","nonce":94308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6806","nonce":94308},{"type":"v1","addr":"172.21.15.202:6807","nonce":94308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6804","nonce":94308},{"type":"v1","addr":"172.21.15.202:6805","nonce":94308}]},"public_addr":"172.21.15.202:6801/94308","cluster_addr":"172.21.15.202:6803/94308","heartbeat_back_addr":"172.21.15.202:6807/94308","heartbeat_front_addr":"172.21.15.202:6805/94308","state":["exists","up"]},{"osd":5,"uuid":"35409ef0-891b-4c01-82e4-26b8e64f502d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6824","nonce":94325},{"type":"v1","addr":"172.21.15.202:6825","nonce":94325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6826","nonce":94325},{"type":"v1","addr":"172.21.15.202:6827","nonce":94325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6830","nonce":94325},{"type":"v1","addr":"172.21.15.202:6831","nonce":94325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6828","nonce":94325},{"type":"v1","addr":"172.21.15.202:6829","nonce":94325}]},"public_addr":"172.21.15.202:6825/94325","cluster_addr":"172.21.15.202:6827/94325","heartbeat_back_addr":"172.21.15.202:6831/94325","heartbeat_front_addr":"172.21.15.202:6829/94325","state":["exists","up"]},{"osd":6,"uuid":"009124c4-8b5f-47d4-bd20-2c19271a2adc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6809","nonce":94309},{"type":"v1","addr":"172.21.15.202:6811","nonce":94309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6813","nonce":94309},{"type":"v1","addr":"172.21.15.202:6815","nonce":94309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6821","nonce":94309},{"type":"v1","addr":"172.21.15.202:6823","nonce":94309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6817","nonce":94309},{"type":"v1","addr":"172.21.15.202:6819","nonce":94309}]},"public_addr":"172.21.15.202:6811/94309","cluster_addr":"172.21.15.202:6815/94309","heartbeat_back_addr":"172.21.15.202:6823/94309","heartbeat_front_addr":"172.21.15.202:6819/94309","state":["exists","up"]},{"osd":7,"uuid":"7773115e-d3c3-4a0c-8801-d8ce63e0e5a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6808","nonce":94310},{"type":"v1","addr":"172.21.15.202:6810","nonce":94310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6812","nonce":94310},{"type":"v1","addr":"172.21.15.202:6814","nonce":94310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6820","nonce":94310},{"type":"v1","addr":"172.21.15.202:6822","nonce":94310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6816","nonce":94310},{"type":"v1","addr":"172.21.15.202:6818","nonce":94310}]},"public_addr":"172.21.15.202:6810/94310","cluster_addr":"172.21.15.202:6814/94310","heartbeat_back_addr":"172.21.15.202:6822/94310","heartbeat_front_addr":"172.21.15.202:6818/94310","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.776182+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.806922+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.237997+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.784371+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.723615+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.820332+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:43.841414+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.758071+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-06-27T11:48:09.641 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.636+0000 7fd9afb23700 1 -- 172.21.15.195:0/902946001 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd988063d20 msgr2=0x7fd9880661d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:09.641 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.636+0000 7fd9afb23700 1 --2- 172.21.15.195:0/902946001 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd988063d20 0x7fd9880661d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd998000e10 tx=0x7fd998009040 comp rx=0 tx=0).stop 2022-06-27T11:48:09.642 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.636+0000 7fd9afb23700 1 -- 172.21.15.195:0/902946001 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd9a80c0390 msgr2=0x7fd9a8148470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:09.642 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.636+0000 7fd9afb23700 1 --2- 172.21.15.195:0/902946001 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd9a80c0390 0x7fd9a8148470 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fd9a40119d0 tx=0x7fd9a4009060 comp rx=0 tx=0).stop 2022-06-27T11:48:09.642 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.636+0000 7fd9afb23700 1 -- 172.21.15.195:0/902946001 shutdown_connections 2022-06-27T11:48:09.642 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.636+0000 7fd9afb23700 1 --2- 172.21.15.195:0/902946001 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd988063d20 0x7fd9880661d0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:09.643 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.636+0000 7fd9afb23700 1 --2- 172.21.15.195:0/902946001 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd9a8154bc0 0x7fd9a81489b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:09.643 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.636+0000 7fd9afb23700 1 --2- 172.21.15.195:0/902946001 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd9a81547f0 0x7fd9a8156d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:09.643 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.636+0000 7fd9afb23700 1 --2- 172.21.15.195:0/902946001 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd9a80c0390 0x7fd9a8148470 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:09.643 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.636+0000 7fd9afb23700 1 -- 172.21.15.195:0/902946001 >> 172.21.15.195:0/902946001 conn(0x7fd9a80c6350 msgr2=0x7fd9a8068b90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:09.644 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.636+0000 7fd9afb23700 1 -- 172.21.15.195:0/902946001 shutdown_connections 2022-06-27T11:48:09.644 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.636+0000 7fd9afb23700 1 -- 172.21.15.195:0/902946001 wait complete. 2022-06-27T11:48:09.647 INFO:tasks.mds_thrash.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2022-06-27T11:47:46.368325+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2022-06-27T11:47:46.386132+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2022-06-27T11:47:50.901270+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2022-06-27T11:47:51.314719+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-06-27T11:48:09.649 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-06-27T11:48:09.773 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.773+0000 7f38c33ca700 1 Processor -- start 2022-06-27T11:48:09.773 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.773+0000 7f38c33ca700 1 -- start start 2022-06-27T11:48:09.784 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.774+0000 7f38c33ca700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f38bc14e490 0x7f38bc06cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:09.785 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.774+0000 7f38c33ca700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f38bc0c3f50 0x7f38bc157890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:09.785 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.774+0000 7f38c33ca700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38bc0c33b0 con 0x7f38bc14e860 2022-06-27T11:48:09.786 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.774+0000 7f38c33ca700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38bc06d520 con 0x7f38bc14e490 2022-06-27T11:48:09.786 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.774+0000 7f38c33ca700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38bc06f220 con 0x7f38bc0c3f50 2022-06-27T11:48:09.786 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.774+0000 7f38c1166700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f38bc14e490 0x7f38bc06cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:09.787 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.774+0000 7f38c0965700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f38bc14e860 0x7f38bc0c3840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:48180/0 (socket says 172.21.15.195:48180) 2022-06-27T11:48:09.787 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.774+0000 7f38c0965700 1 -- 172.21.15.195:0/3340400973 learned_addr learned my addr 172.21.15.195:0/3340400973 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:48:09.787 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.774+0000 7f38c1967700 1 --2- 172.21.15.195:0/3340400973 >> v2:172.21.15.202:3300/0 conn(0x7f38bc0c3f50 0x7f38bc157890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:09.788 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.774+0000 7f38c1166700 1 -- 172.21.15.195:0/3340400973 >> v2:172.21.15.202:3300/0 conn(0x7f38bc0c3f50 msgr2=0x7f38bc157890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:09.788 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.774+0000 7f38c1166700 1 --2- 172.21.15.195:0/3340400973 >> v2:172.21.15.202:3300/0 conn(0x7f38bc0c3f50 0x7f38bc157890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:09.789 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.774+0000 7f38c1166700 1 -- 172.21.15.195:0/3340400973 >> v1:172.21.15.195:6789/0 conn(0x7f38bc14e860 legacy=0x7f38bc0c3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:09.789 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.774+0000 7f38abfff700 1 -- 172.21.15.195:0/3340400973 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 303508394 0 0) 0x7f38bc0c33b0 con 0x7f38bc14e860 2022-06-27T11:48:09.789 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.774+0000 7f38c1166700 1 -- 172.21.15.195:0/3340400973 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38bc1542c0 con 0x7f38bc14e490 2022-06-27T11:48:09.790 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.775+0000 7f38c1166700 1 --2- 172.21.15.195:0/3340400973 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f38bc14e490 0x7f38bc06cfe0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f38bc06cd00 tx=0x7f38ac009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a779812b963b520d server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:09.790 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.775+0000 7f38abfff700 1 -- 172.21.15.195:0/3340400973 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38ac011630 con 0x7f38bc14e490 2022-06-27T11:48:09.790 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.775+0000 7f38abfff700 1 -- 172.21.15.195:0/3340400973 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38ac011790 con 0x7f38bc14e490 2022-06-27T11:48:09.791 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.775+0000 7f38abfff700 1 -- 172.21.15.195:0/3340400973 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38ac011b30 con 0x7f38bc14e490 2022-06-27T11:48:09.791 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.775+0000 7f38c33ca700 1 -- 172.21.15.195:0/3340400973 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f38bc14e490 msgr2=0x7f38bc06cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:09.791 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.775+0000 7f38c33ca700 1 --2- 172.21.15.195:0/3340400973 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f38bc14e490 0x7f38bc06cfe0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f38bc06cd00 tx=0x7f38ac009060 comp rx=0 tx=0).stop 2022-06-27T11:48:09.791 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.775+0000 7f38c33ca700 1 -- 172.21.15.195:0/3340400973 shutdown_connections 2022-06-27T11:48:09.791 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.775+0000 7f38c33ca700 1 --2- 172.21.15.195:0/3340400973 >> v2:172.21.15.202:3300/0 conn(0x7f38bc0c3f50 0x7f38bc157890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:09.792 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.775+0000 7f38c33ca700 1 --2- 172.21.15.195:0/3340400973 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f38bc14e490 0x7f38bc06cfe0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:09.792 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.775+0000 7f38c33ca700 1 -- 172.21.15.195:0/3340400973 >> 172.21.15.195:0/3340400973 conn(0x7f38bc063af0 msgr2=0x7f38bc063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:09.792 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.775+0000 7f38c33ca700 1 -- 172.21.15.195:0/3340400973 shutdown_connections 2022-06-27T11:48:09.792 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.775+0000 7f38c33ca700 1 -- 172.21.15.195:0/3340400973 wait complete. 2022-06-27T11:48:09.793 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.776+0000 7f38c33ca700 1 Processor -- start 2022-06-27T11:48:09.793 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.776+0000 7f38c33ca700 1 -- start start 2022-06-27T11:48:09.793 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.777+0000 7f38c33ca700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f38bc0c3f50 0x7f38bc167bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:09.793 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.777+0000 7f38c33ca700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f38bc14e490 0x7f38bc152400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:09.793 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.777+0000 7f38c33ca700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f38bc14e860 0x7f38bc152940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:09.794 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.777+0000 7f38c33ca700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38bc0c1450 con 0x7f38bc0c3f50 2022-06-27T11:48:09.794 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.777+0000 7f38c33ca700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38bc168580 con 0x7f38bc14e490 2022-06-27T11:48:09.794 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.777+0000 7f38c33ca700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38bc0c16f0 con 0x7f38bc14e860 2022-06-27T11:48:09.794 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.777+0000 7f38c1166700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f38bc0c3f50 0x7f38bc167bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:09.795 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.777+0000 7f38c1166700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f38bc0c3f50 0x7f38bc167bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:41996/0 (socket says 172.21.15.195:41996) 2022-06-27T11:48:09.795 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.777+0000 7f38c1166700 1 -- 172.21.15.195:0/3048191786 learned_addr learned my addr 172.21.15.195:0/3048191786 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:48:09.795 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.778+0000 7f38c0965700 1 --2- 172.21.15.195:0/3048191786 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f38bc14e490 0x7f38bc152400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:09.795 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.778+0000 7f38c1967700 1 --2- 172.21.15.195:0/3048191786 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f38bc14e860 0x7f38bc152940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:09.796 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.778+0000 7f38c1166700 1 -- 172.21.15.195:0/3048191786 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f38bc14e860 msgr2=0x7f38bc152940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:09.796 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.778+0000 7f38c1166700 1 --2- 172.21.15.195:0/3048191786 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f38bc14e860 0x7f38bc152940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:09.796 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.778+0000 7f38c1166700 1 -- 172.21.15.195:0/3048191786 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f38bc14e490 msgr2=0x7f38bc152400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:09.796 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.778+0000 7f38c1166700 1 --2- 172.21.15.195:0/3048191786 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f38bc14e490 0x7f38bc152400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:09.797 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.778+0000 7f38c1166700 1 -- 172.21.15.195:0/3048191786 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38ac0092e0 con 0x7f38bc0c3f50 2022-06-27T11:48:09.797 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.778+0000 7f38c0965700 1 --2- 172.21.15.195:0/3048191786 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f38bc14e490 0x7f38bc152400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T11:48:09.797 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.778+0000 7f38c1166700 1 --2- 172.21.15.195:0/3048191786 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f38bc0c3f50 0x7f38bc167bd0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f38ac011fd0 tx=0x7f38ac017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:09.797 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.778+0000 7f38a97fa700 1 -- 172.21.15.195:0/3048191786 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38ac003a90 con 0x7f38bc0c3f50 2022-06-27T11:48:09.797 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.778+0000 7f38a97fa700 1 -- 172.21.15.195:0/3048191786 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38ac003c10 con 0x7f38bc0c3f50 2022-06-27T11:48:09.798 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.778+0000 7f38a97fa700 1 -- 172.21.15.195:0/3048191786 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38ac024440 con 0x7f38bc0c3f50 2022-06-27T11:48:09.798 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.778+0000 7f38c33ca700 1 -- 172.21.15.195:0/3048191786 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38bc0bf310 con 0x7f38bc0c3f50 2022-06-27T11:48:09.798 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.778+0000 7f38c33ca700 1 -- 172.21.15.195:0/3048191786 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38bc14f980 con 0x7f38bc0c3f50 2022-06-27T11:48:09.798 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.779+0000 7f38c33ca700 1 -- 172.21.15.195:0/3048191786 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38bc0bf170 con 0x7f38bc0c3f50 2022-06-27T11:48:09.799 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.779+0000 7f38a97fa700 1 -- 172.21.15.195:0/3048191786 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f38ac02a020 con 0x7f38bc0c3f50 2022-06-27T11:48:09.799 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.780+0000 7f38a97fa700 1 --2- 172.21.15.195:0/3048191786 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f389c063d00 0x7f389c0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:09.799 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.780+0000 7f38a97fa700 1 -- 172.21.15.195:0/3048191786 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f38ac082030 con 0x7f38bc0c3f50 2022-06-27T11:48:09.799 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.780+0000 7f38c0965700 1 --2- 172.21.15.195:0/3048191786 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f389c063d00 0x7f389c0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:09.800 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.780+0000 7f38c0965700 1 --2- 172.21.15.195:0/3048191786 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f389c063d00 0x7f389c0661b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f38bc072980 tx=0x7f38b8009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:09.800 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.784+0000 7f38a97fa700 1 -- 172.21.15.195:0/3048191786 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f38bc0bf170 con 0x7f38bc0c3f50 2022-06-27T11:48:09.962 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.963+0000 7f38c33ca700 1 -- 172.21.15.195:0/3048191786 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f38bc0bf170 con 0x7f38bc0c3f50 2022-06-27T11:48:09.963 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.963+0000 7f38a97fa700 1 -- 172.21.15.195:0/3048191786 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v21) v1 ==== 92+0+10 (secure 0 0 0) 0x7f38bc0bf170 con 0x7f38bc0c3f50 2022-06-27T11:48:09.963 INFO:teuthology.orchestra.run.smithi195.stdout:pg_num: 8 2022-06-27T11:48:09.965 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.965+0000 7f38c33ca700 1 -- 172.21.15.195:0/3048191786 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f389c063d00 msgr2=0x7f389c0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:09.965 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.965+0000 7f38c33ca700 1 --2- 172.21.15.195:0/3048191786 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f389c063d00 0x7f389c0661b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f38bc072980 tx=0x7f38b8009040 comp rx=0 tx=0).stop 2022-06-27T11:48:09.965 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.965+0000 7f38c33ca700 1 -- 172.21.15.195:0/3048191786 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f38bc0c3f50 msgr2=0x7f38bc167bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:09.966 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.965+0000 7f38c33ca700 1 --2- 172.21.15.195:0/3048191786 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f38bc0c3f50 0x7f38bc167bd0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f38ac011fd0 tx=0x7f38ac017000 comp rx=0 tx=0).stop 2022-06-27T11:48:09.966 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.965+0000 7f38c33ca700 1 -- 172.21.15.195:0/3048191786 shutdown_connections 2022-06-27T11:48:09.966 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.965+0000 7f38c33ca700 1 --2- 172.21.15.195:0/3048191786 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f389c063d00 0x7f389c0661b0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:09.966 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.965+0000 7f38c33ca700 1 --2- 172.21.15.195:0/3048191786 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f38bc0c3f50 0x7f38bc167bd0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:09.967 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.965+0000 7f38c33ca700 1 --2- 172.21.15.195:0/3048191786 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f38bc14e860 0x7f38bc152940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:09.967 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.965+0000 7f38c33ca700 1 --2- 172.21.15.195:0/3048191786 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f38bc14e490 0x7f38bc152400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:09.967 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.965+0000 7f38c33ca700 1 -- 172.21.15.195:0/3048191786 >> 172.21.15.195:0/3048191786 conn(0x7f38bc063af0 msgr2=0x7f38bc14d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:09.967 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.965+0000 7f38c33ca700 1 -- 172.21.15.195:0/3048191786 shutdown_connections 2022-06-27T11:48:09.968 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:09.965+0000 7f38c33ca700 1 -- 172.21.15.195:0/3048191786 wait complete. 2022-06-27T11:48:09.978 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2022-06-27T11:48:10.101 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.102+0000 7f8555b71700 1 Processor -- start 2022-06-27T11:48:10.102 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.102+0000 7f8555b71700 1 -- start start 2022-06-27T11:48:10.111 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.102+0000 7f8555b71700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f855006fef0 0x7f85501570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:10.111 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.102+0000 7f8555b71700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f855006fba0 con 0x7f855006e850 2022-06-27T11:48:10.111 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.103+0000 7f8555b71700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f855006f5e0 con 0x7f85500c34a0 2022-06-27T11:48:10.112 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.103+0000 7f8555b71700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85501575e0 con 0x7f855006fef0 2022-06-27T11:48:10.112 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.103+0000 7f854f7fe700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f855006e850 0x7f8550069de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:48190/0 (socket says 172.21.15.195:48190) 2022-06-27T11:48:10.113 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.103+0000 7f854f7fe700 1 -- 172.21.15.195:0/3333440758 learned_addr learned my addr 172.21.15.195:0/3333440758 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:48:10.113 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.103+0000 7f8547fff700 1 --2- 172.21.15.195:0/3333440758 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f855006fef0 0x7f85501570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:10.114 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.103+0000 7f854effd700 1 -- 172.21.15.195:0/3333440758 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3573594244 0 0) 0x7f855006fba0 con 0x7f855006e850 2022-06-27T11:48:10.114 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.103+0000 7f854effd700 1 -- 172.21.15.195:0/3333440758 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8530003430 con 0x7f855006e850 2022-06-27T11:48:10.115 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.103+0000 7f854effd700 1 -- 172.21.15.195:0/3333440758 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 981337292 0 0) 0x7f855006f5e0 con 0x7f85500c34a0 2022-06-27T11:48:10.115 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.103+0000 7f854effd700 1 -- 172.21.15.195:0/3333440758 --> v1:172.21.15.202:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f855006fba0 con 0x7f85500c34a0 2022-06-27T11:48:10.115 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.103+0000 7f854effd700 1 -- 172.21.15.195:0/3333440758 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1801700549 0 0) 0x7f8530003430 con 0x7f855006e850 2022-06-27T11:48:10.116 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.103+0000 7f854effd700 1 -- 172.21.15.195:0/3333440758 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f855006fef0 msgr2=0x7f85501570a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:10.116 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.103+0000 7f854effd700 1 --2- 172.21.15.195:0/3333440758 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f855006fef0 0x7f85501570a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:10.117 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.103+0000 7f854effd700 1 -- 172.21.15.195:0/3333440758 >> v1:172.21.15.202:6789/0 conn(0x7f85500c34a0 legacy=0x7f85500c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:10.117 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.103+0000 7f854effd700 1 -- 172.21.15.195:0/3333440758 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f855006f8c0 con 0x7f855006e850 2022-06-27T11:48:10.118 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.103+0000 7f854effd700 1 -- 172.21.15.195:0/3333440758 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7f8540001470 con 0x7f855006e850 2022-06-27T11:48:10.118 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.103+0000 7f854effd700 1 -- 172.21.15.195:0/3333440758 >> v1:172.21.15.195:6789/0 conn(0x7f855006e850 legacy=0x7f8550069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:10.119 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.103+0000 7f854effd700 1 --2- 172.21.15.195:0/3333440758 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8530003ae0 0x7f8530003ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:10.119 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.103+0000 7f854effd700 1 --2- 172.21.15.195:0/3333440758 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f85300046d0 0x7f8530004b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:10.119 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.103+0000 7f854effd700 1 --2- 172.21.15.195:0/3333440758 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8530005080 0x7f853000b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:10.120 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.103+0000 7f854effd700 1 -- 172.21.15.195:0/3333440758 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85501575e0 con 0x7f85300046d0 2022-06-27T11:48:10.120 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.103+0000 7f854effd700 1 -- 172.21.15.195:0/3333440758 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f853000be60 con 0x7f8530003ae0 2022-06-27T11:48:10.121 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.103+0000 7f854effd700 1 -- 172.21.15.195:0/3333440758 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f853000bfa0 con 0x7f8530005080 2022-06-27T11:48:10.121 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.103+0000 7f854f7fe700 1 --2- 172.21.15.195:0/3333440758 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8530003ae0 0x7f8530003ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:10.121 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.103+0000 7f8547fff700 1 --2- 172.21.15.195:0/3333440758 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f85300046d0 0x7f8530004b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:10.121 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.104+0000 7f854ffff700 1 --2- 172.21.15.195:0/3333440758 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8530005080 0x7f853000b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:10.122 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.104+0000 7f8547fff700 1 -- 172.21.15.195:0/3333440758 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8530005080 msgr2=0x7f853000b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:10.122 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.104+0000 7f8547fff700 1 --2- 172.21.15.195:0/3333440758 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8530005080 0x7f853000b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:10.122 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.104+0000 7f8547fff700 1 -- 172.21.15.195:0/3333440758 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8530003ae0 msgr2=0x7f8530003ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:10.123 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.104+0000 7f8547fff700 1 --2- 172.21.15.195:0/3333440758 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8530003ae0 0x7f8530003ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:10.123 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.104+0000 7f8547fff700 1 -- 172.21.15.195:0/3333440758 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8530003430 con 0x7f85300046d0 2022-06-27T11:48:10.123 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.104+0000 7f854f7fe700 1 --2- 172.21.15.195:0/3333440758 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8530003ae0 0x7f8530003ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:48:10.123 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.104+0000 7f8547fff700 1 --2- 172.21.15.195:0/3333440758 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f85300046d0 0x7f8530004b40 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f8538009af0 tx=0x7f8538009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c74c20b790e65da server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:10.124 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.104+0000 7f854effd700 1 -- 172.21.15.195:0/3333440758 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8538009470 con 0x7f85300046d0 2022-06-27T11:48:10.124 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.104+0000 7f854effd700 1 -- 172.21.15.195:0/3333440758 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f853800e640 con 0x7f85300046d0 2022-06-27T11:48:10.124 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.104+0000 7f854effd700 1 -- 172.21.15.195:0/3333440758 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f853800e9e0 con 0x7f85300046d0 2022-06-27T11:48:10.124 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.104+0000 7f8555b71700 1 -- 172.21.15.195:0/3333440758 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f85300046d0 msgr2=0x7f8530004b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:10.125 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.105+0000 7f8555b71700 1 --2- 172.21.15.195:0/3333440758 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f85300046d0 0x7f8530004b40 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f8538009af0 tx=0x7f8538009060 comp rx=0 tx=0).stop 2022-06-27T11:48:10.125 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.105+0000 7f854ffff700 1 -- 172.21.15.195:0/3333440758 reap_dead start 2022-06-27T11:48:10.125 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.105+0000 7f8555b71700 1 -- 172.21.15.195:0/3333440758 shutdown_connections 2022-06-27T11:48:10.125 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.105+0000 7f8555b71700 1 -- 172.21.15.195:0/3333440758 >> 172.21.15.195:0/3333440758 conn(0x7f85500c6350 msgr2=0x7f85500c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:10.126 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.105+0000 7f8555b71700 1 -- 172.21.15.195:0/3333440758 shutdown_connections 2022-06-27T11:48:10.126 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.105+0000 7f8555b71700 1 -- 172.21.15.195:0/3333440758 wait complete. 2022-06-27T11:48:10.126 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.106+0000 7f8555b71700 1 Processor -- start 2022-06-27T11:48:10.126 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.106+0000 7f8555b71700 1 -- start start 2022-06-27T11:48:10.127 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.106+0000 7f8555b71700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f85300046d0 0x7f8550153100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:10.127 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.106+0000 7f8555b71700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f85500687a0 0x7f8550166ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:10.128 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.106+0000 7f8555b71700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8550154e60 0x7f8550148460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:10.128 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.106+0000 7f8555b71700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8550154a40 con 0x7f85500687a0 2022-06-27T11:48:10.128 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.106+0000 7f8555b71700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85501670e0 con 0x7f8550154e60 2022-06-27T11:48:10.128 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.106+0000 7f8555b71700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8550153ce0 con 0x7f85300046d0 2022-06-27T11:48:10.129 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.106+0000 7f8547fff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f85500687a0 0x7f8550166ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:10.129 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.106+0000 7f8547fff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f85500687a0 0x7f8550166ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:42016/0 (socket says 172.21.15.195:42016) 2022-06-27T11:48:10.129 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.107+0000 7f8547fff700 1 -- 172.21.15.195:0/1979557323 learned_addr learned my addr 172.21.15.195:0/1979557323 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:48:10.129 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.107+0000 7f854f7fe700 1 --2- 172.21.15.195:0/1979557323 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f85300046d0 0x7f8550153100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:10.129 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.107+0000 7f854ffff700 1 --2- 172.21.15.195:0/1979557323 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8550154e60 0x7f8550148460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:10.130 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.107+0000 7f8547fff700 1 -- 172.21.15.195:0/1979557323 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f85300046d0 msgr2=0x7f8550153100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:10.130 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.107+0000 7f8547fff700 1 --2- 172.21.15.195:0/1979557323 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f85300046d0 0x7f8550153100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:10.130 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.107+0000 7f8547fff700 1 -- 172.21.15.195:0/1979557323 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8550154e60 msgr2=0x7f8550148460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:10.130 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.107+0000 7f8547fff700 1 --2- 172.21.15.195:0/1979557323 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8550154e60 0x7f8550148460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:10.131 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.107+0000 7f8547fff700 1 -- 172.21.15.195:0/1979557323 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8550153720 con 0x7f85500687a0 2022-06-27T11:48:10.131 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.107+0000 7f8547fff700 1 --2- 172.21.15.195:0/1979557323 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f85500687a0 0x7f8550166ba0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f8538006d90 tx=0x7f853801a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:10.131 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.107+0000 7f85477fe700 1 -- 172.21.15.195:0/1979557323 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8538022070 con 0x7f85500687a0 2022-06-27T11:48:10.131 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.107+0000 7f85477fe700 1 -- 172.21.15.195:0/1979557323 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8538003520 con 0x7f85500687a0 2022-06-27T11:48:10.132 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.107+0000 7f85477fe700 1 -- 172.21.15.195:0/1979557323 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85380038c0 con 0x7f85500687a0 2022-06-27T11:48:10.132 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.107+0000 7f8555b71700 1 -- 172.21.15.195:0/1979557323 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8550153a00 con 0x7f85500687a0 2022-06-27T11:48:10.132 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.107+0000 7f8555b71700 1 -- 172.21.15.195:0/1979557323 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8550154660 con 0x7f85500687a0 2022-06-27T11:48:10.132 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.108+0000 7f8555b71700 1 -- 172.21.15.195:0/1979557323 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85500722c0 con 0x7f85500687a0 2022-06-27T11:48:10.132 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.108+0000 7f85477fe700 1 -- 172.21.15.195:0/1979557323 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f8538019030 con 0x7f85500687a0 2022-06-27T11:48:10.133 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.109+0000 7f85477fe700 1 --2- 172.21.15.195:0/1979557323 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8534063cf0 0x7f85340661a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:10.133 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.109+0000 7f85477fe700 1 -- 172.21.15.195:0/1979557323 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f8538081a90 con 0x7f85500687a0 2022-06-27T11:48:10.133 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.109+0000 7f854f7fe700 1 --2- 172.21.15.195:0/1979557323 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8534063cf0 0x7f85340661a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:10.133 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.109+0000 7f854f7fe700 1 --2- 172.21.15.195:0/1979557323 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8534063cf0 0x7f85340661a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8540007f20 tx=0x7f8540001df0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:10.134 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.113+0000 7f85477fe700 1 -- 172.21.15.195:0/1979557323 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f8538050740 con 0x7f85500687a0 2022-06-27T11:48:10.291 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.291+0000 7f8555b71700 1 -- 172.21.15.195:0/1979557323 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f85501547f0 con 0x7f85500687a0 2022-06-27T11:48:10.292 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.292+0000 7f85477fe700 1 -- 172.21.15.195:0/1979557323 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8538017070 con 0x7f85500687a0 2022-06-27T11:48:10.293 INFO:teuthology.orchestra.run.smithi195.stdout:pg_num: 1 2022-06-27T11:48:10.294 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.293+0000 7f8555b71700 1 -- 172.21.15.195:0/1979557323 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8534063cf0 msgr2=0x7f85340661a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:10.295 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.293+0000 7f8555b71700 1 --2- 172.21.15.195:0/1979557323 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8534063cf0 0x7f85340661a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8540007f20 tx=0x7f8540001df0 comp rx=0 tx=0).stop 2022-06-27T11:48:10.295 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.293+0000 7f8555b71700 1 -- 172.21.15.195:0/1979557323 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f85500687a0 msgr2=0x7f8550166ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:10.296 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.293+0000 7f8555b71700 1 --2- 172.21.15.195:0/1979557323 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f85500687a0 0x7f8550166ba0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f8538006d90 tx=0x7f853801a000 comp rx=0 tx=0).stop 2022-06-27T11:48:10.296 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.293+0000 7f8555b71700 1 -- 172.21.15.195:0/1979557323 shutdown_connections 2022-06-27T11:48:10.296 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.293+0000 7f8555b71700 1 --2- 172.21.15.195:0/1979557323 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8534063cf0 0x7f85340661a0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:10.297 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.293+0000 7f8555b71700 1 --2- 172.21.15.195:0/1979557323 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8550154e60 0x7f8550148460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:10.297 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.293+0000 7f8555b71700 1 --2- 172.21.15.195:0/1979557323 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f85300046d0 0x7f8550153100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:10.298 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.293+0000 7f8555b71700 1 --2- 172.21.15.195:0/1979557323 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f85500687a0 0x7f8550166ba0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:10.298 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.293+0000 7f8555b71700 1 -- 172.21.15.195:0/1979557323 >> 172.21.15.195:0/1979557323 conn(0x7f85500c6350 msgr2=0x7f85500c68a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:10.299 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.293+0000 7f8555b71700 1 -- 172.21.15.195:0/1979557323 shutdown_connections 2022-06-27T11:48:10.299 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.293+0000 7f8555b71700 1 -- 172.21.15.195:0/1979557323 wait complete. 2022-06-27T11:48:10.305 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-06-27T11:48:10.439 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.431+0000 7f0c0e33b700 1 Processor -- start 2022-06-27T11:48:10.439 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.431+0000 7f0c0e33b700 1 -- start start 2022-06-27T11:48:10.439 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.431+0000 7f0c0e33b700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f0c0806f640 0x7f0c08156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:10.440 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.432+0000 7f0c0e33b700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c080c1a20 con 0x7f0c080c4160 2022-06-27T11:48:10.440 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.432+0000 7f0c0e33b700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c080c1130 con 0x7f0c080c39e0 2022-06-27T11:48:10.440 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.432+0000 7f0c0e33b700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c0806ea90 con 0x7f0c0806f640 2022-06-27T11:48:10.440 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.432+0000 7f0c077fe700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f0c080c4160 0x7f0c080c1eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:48210/0 (socket says 172.21.15.195:48210) 2022-06-27T11:48:10.441 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.432+0000 7f0c077fe700 1 -- 172.21.15.195:0/334002040 learned_addr learned my addr 172.21.15.195:0/334002040 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:48:10.441 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.432+0000 7f0c06ffd700 1 -- 172.21.15.195:0/334002040 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3252462634 0 0) 0x7f0c080c1a20 con 0x7f0c080c4160 2022-06-27T11:48:10.441 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.432+0000 7f0c07fff700 1 --2- 172.21.15.195:0/334002040 >> v2:172.21.15.202:3300/0 conn(0x7f0c0806f640 0x7f0c08156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:10.442 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.432+0000 7f0c06ffd700 1 -- 172.21.15.195:0/334002040 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bf4003430 con 0x7f0c080c4160 2022-06-27T11:48:10.442 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.432+0000 7f0c06ffd700 1 -- 172.21.15.195:0/334002040 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2244002926 0 0) 0x7f0bf4003430 con 0x7f0c080c4160 2022-06-27T11:48:10.442 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.432+0000 7f0c06ffd700 1 -- 172.21.15.195:0/334002040 >> v2:172.21.15.202:3300/0 conn(0x7f0c0806f640 msgr2=0x7f0c08156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:10.442 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.432+0000 7f0c06ffd700 1 --2- 172.21.15.195:0/334002040 >> v2:172.21.15.202:3300/0 conn(0x7f0c0806f640 0x7f0c08156ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:10.442 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.432+0000 7f0c06ffd700 1 -- 172.21.15.195:0/334002040 >> v1:172.21.15.202:6789/0 conn(0x7f0c080c39e0 legacy=0x7f0c081535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:10.443 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.432+0000 7f0c06ffd700 1 -- 172.21.15.195:0/334002040 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c0806ed70 con 0x7f0c080c4160 2022-06-27T11:48:10.443 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.432+0000 7f0c06ffd700 1 -- 172.21.15.195:0/334002040 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7f0bfc001470 con 0x7f0c080c4160 2022-06-27T11:48:10.443 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.432+0000 7f0c06ffd700 1 -- 172.21.15.195:0/334002040 >> v1:172.21.15.195:6789/0 conn(0x7f0c080c4160 legacy=0x7f0c080c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:10.443 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.432+0000 7f0c06ffd700 1 --2- 172.21.15.195:0/334002040 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0bf4003920 0x7f0bf4003d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:10.444 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.432+0000 7f0c06ffd700 1 --2- 172.21.15.195:0/334002040 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0bf40046d0 0x7f0bf4004b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:10.444 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.432+0000 7f0c06ffd700 1 --2- 172.21.15.195:0/334002040 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0bf4005080 0x7f0bf400b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:10.444 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.432+0000 7f0c06ffd700 1 -- 172.21.15.195:0/334002040 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0806ea90 con 0x7f0bf4005080 2022-06-27T11:48:10.444 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.432+0000 7f0c06ffd700 1 -- 172.21.15.195:0/334002040 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf400bf90 con 0x7f0bf4003920 2022-06-27T11:48:10.444 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.432+0000 7f0c06ffd700 1 -- 172.21.15.195:0/334002040 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf400c150 con 0x7f0bf40046d0 2022-06-27T11:48:10.445 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.432+0000 7f0c0c8d8700 1 --2- 172.21.15.195:0/334002040 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0bf4005080 0x7f0bf400b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:10.445 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.432+0000 7f0c077fe700 1 --2- 172.21.15.195:0/334002040 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0bf40046d0 0x7f0bf4004b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:10.445 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.433+0000 7f0c07fff700 1 --2- 172.21.15.195:0/334002040 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0bf4003920 0x7f0bf4003d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:10.445 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.433+0000 7f0c0c8d8700 1 -- 172.21.15.195:0/334002040 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0bf40046d0 msgr2=0x7f0bf4004b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:10.445 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.433+0000 7f0c0c8d8700 1 --2- 172.21.15.195:0/334002040 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0bf40046d0 0x7f0bf4004b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:10.446 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.433+0000 7f0c0c8d8700 1 -- 172.21.15.195:0/334002040 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0bf4003920 msgr2=0x7f0bf4003d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:10.446 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.433+0000 7f0c0c8d8700 1 --2- 172.21.15.195:0/334002040 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0bf4003920 0x7f0bf4003d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:10.446 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.433+0000 7f0c0c8d8700 1 -- 172.21.15.195:0/334002040 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bf4003430 con 0x7f0bf4005080 2022-06-27T11:48:10.446 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.433+0000 7f0c077fe700 1 --2- 172.21.15.195:0/334002040 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0bf40046d0 0x7f0bf4004b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T11:48:10.447 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.433+0000 7f0c0c8d8700 1 -- 172.21.15.195:0/334002040 reap_dead start 2022-06-27T11:48:10.447 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.433+0000 7f0c0c8d8700 1 --2- 172.21.15.195:0/334002040 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0bf4005080 0x7f0bf400b920 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f0c00001e60 tx=0x7f0c00007f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c4946838c4607f4 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:10.447 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.433+0000 7f0c06ffd700 1 -- 172.21.15.195:0/334002040 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c00003880 con 0x7f0bf4005080 2022-06-27T11:48:10.447 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.433+0000 7f0c06ffd700 1 -- 172.21.15.195:0/334002040 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c000039e0 con 0x7f0bf4005080 2022-06-27T11:48:10.447 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.433+0000 7f0c06ffd700 1 -- 172.21.15.195:0/334002040 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c00003da0 con 0x7f0bf4005080 2022-06-27T11:48:10.448 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.433+0000 7f0c0e33b700 1 -- 172.21.15.195:0/334002040 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0bf4005080 msgr2=0x7f0bf400b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:10.448 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.433+0000 7f0c0e33b700 1 --2- 172.21.15.195:0/334002040 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0bf4005080 0x7f0bf400b920 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f0c00001e60 tx=0x7f0c00007f30 comp rx=0 tx=0).stop 2022-06-27T11:48:10.448 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.433+0000 7f0c0e33b700 1 -- 172.21.15.195:0/334002040 shutdown_connections 2022-06-27T11:48:10.448 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.433+0000 7f0c0e33b700 1 --2- 172.21.15.195:0/334002040 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0bf4005080 0x7f0bf400b920 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:10.449 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.433+0000 7f0c0e33b700 1 -- 172.21.15.195:0/334002040 >> 172.21.15.195:0/334002040 conn(0x7f0c080c6350 msgr2=0x7f0c080c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:10.449 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.433+0000 7f0c0e33b700 1 -- 172.21.15.195:0/334002040 shutdown_connections 2022-06-27T11:48:10.449 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.434+0000 7f0c0e33b700 1 -- 172.21.15.195:0/334002040 wait complete. 2022-06-27T11:48:10.449 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.435+0000 7f0c0e33b700 1 Processor -- start 2022-06-27T11:48:10.449 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.435+0000 7f0c0e33b700 1 -- start start 2022-06-27T11:48:10.450 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.435+0000 7f0c0e33b700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0bf4005080 0x7f0c0816a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:10.450 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.435+0000 7f0c0e33b700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0c08158da0 0x7f0c0815bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:10.450 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.435+0000 7f0c0e33b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0c081561d0 0x7f0c08156640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:10.450 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.435+0000 7f0c0e33b700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c080bac30 con 0x7f0c081561d0 2022-06-27T11:48:10.450 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.435+0000 7f0c0e33b700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c080c0c20 con 0x7f0c08158da0 2022-06-27T11:48:10.451 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.435+0000 7f0c0e33b700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c08159650 con 0x7f0bf4005080 2022-06-27T11:48:10.451 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.436+0000 7f0c0c8d8700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0c081561d0 0x7f0c08156640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:10.451 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.436+0000 7f0c0c8d8700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0c081561d0 0x7f0c08156640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:42038/0 (socket says 172.21.15.195:42038) 2022-06-27T11:48:10.451 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.436+0000 7f0c0c8d8700 1 -- 172.21.15.195:0/3453285999 learned_addr learned my addr 172.21.15.195:0/3453285999 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:48:10.452 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.436+0000 7f0c077fe700 1 --2- 172.21.15.195:0/3453285999 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0c08158da0 0x7f0c0815bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:10.452 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.436+0000 7f0c07fff700 1 --2- 172.21.15.195:0/3453285999 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0bf4005080 0x7f0c0816a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:10.452 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.436+0000 7f0c0c8d8700 1 -- 172.21.15.195:0/3453285999 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0bf4005080 msgr2=0x7f0c0816a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:10.452 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.436+0000 7f0c0c8d8700 1 --2- 172.21.15.195:0/3453285999 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0bf4005080 0x7f0c0816a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:10.452 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.436+0000 7f0c0c8d8700 1 -- 172.21.15.195:0/3453285999 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0c08158da0 msgr2=0x7f0c0815bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:10.453 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.436+0000 7f0c0c8d8700 1 --2- 172.21.15.195:0/3453285999 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0c08158da0 0x7f0c0815bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:10.453 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.436+0000 7f0c0c8d8700 1 -- 172.21.15.195:0/3453285999 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c08167140 con 0x7f0c081561d0 2022-06-27T11:48:10.453 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.436+0000 7f0c0c8d8700 1 --2- 172.21.15.195:0/3453285999 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0c081561d0 0x7f0c08156640 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0c0806dda0 tx=0x7f0c00014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:10.453 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.436+0000 7f0beffff700 1 -- 172.21.15.195:0/3453285999 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c00003880 con 0x7f0c081561d0 2022-06-27T11:48:10.454 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.436+0000 7f0beffff700 1 -- 172.21.15.195:0/3453285999 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c00001750 con 0x7f0c081561d0 2022-06-27T11:48:10.454 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.436+0000 7f0c0e33b700 1 -- 172.21.15.195:0/3453285999 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c08156b80 con 0x7f0c081561d0 2022-06-27T11:48:10.454 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.436+0000 7f0beffff700 1 -- 172.21.15.195:0/3453285999 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c0000ca30 con 0x7f0c081561d0 2022-06-27T11:48:10.454 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.436+0000 7f0c0e33b700 1 -- 172.21.15.195:0/3453285999 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c08166810 con 0x7f0c081561d0 2022-06-27T11:48:10.454 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.437+0000 7f0c0e33b700 1 -- 172.21.15.195:0/3453285999 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c080722c0 con 0x7f0c081561d0 2022-06-27T11:48:10.455 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.437+0000 7f0beffff700 1 -- 172.21.15.195:0/3453285999 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f0c0001b040 con 0x7f0c081561d0 2022-06-27T11:48:10.455 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.438+0000 7f0beffff700 1 --2- 172.21.15.195:0/3453285999 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0be8063d30 0x7f0be80661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:10.455 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.438+0000 7f0beffff700 1 -- 172.21.15.195:0/3453285999 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f0c00081bf0 con 0x7f0c081561d0 2022-06-27T11:48:10.456 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.438+0000 7f0c07fff700 1 --2- 172.21.15.195:0/3453285999 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0be8063d30 0x7f0be80661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:10.456 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.438+0000 7f0c07fff700 1 --2- 172.21.15.195:0/3453285999 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0be8063d30 0x7f0be80661e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0bf8006dc0 tx=0x7f0bf8009060 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:10.457 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.442+0000 7f0beffff700 1 -- 172.21.15.195:0/3453285999 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f0c000507e0 con 0x7f0c081561d0 2022-06-27T11:48:10.618 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.618+0000 7f0c0e33b700 1 -- 172.21.15.195:0/3453285999 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0c080722c0 con 0x7f0c081561d0 2022-06-27T11:48:10.618 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.618+0000 7f0beffff700 1 -- 172.21.15.195:0/3453285999 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0c00053e00 con 0x7f0c081561d0 2022-06-27T11:48:10.618 INFO:teuthology.orchestra.run.smithi195.stdout:pg_num: 32 2022-06-27T11:48:10.620 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.620+0000 7f0c0e33b700 1 -- 172.21.15.195:0/3453285999 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0be8063d30 msgr2=0x7f0be80661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:10.620 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.620+0000 7f0c0e33b700 1 --2- 172.21.15.195:0/3453285999 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0be8063d30 0x7f0be80661e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0bf8006dc0 tx=0x7f0bf8009060 comp rx=0 tx=0).stop 2022-06-27T11:48:10.620 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.620+0000 7f0c0e33b700 1 -- 172.21.15.195:0/3453285999 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0c081561d0 msgr2=0x7f0c08156640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:10.621 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.620+0000 7f0c0e33b700 1 --2- 172.21.15.195:0/3453285999 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0c081561d0 0x7f0c08156640 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0c0806dda0 tx=0x7f0c00014000 comp rx=0 tx=0).stop 2022-06-27T11:48:10.621 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.620+0000 7f0c0e33b700 1 -- 172.21.15.195:0/3453285999 shutdown_connections 2022-06-27T11:48:10.621 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.620+0000 7f0c0e33b700 1 --2- 172.21.15.195:0/3453285999 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0be8063d30 0x7f0be80661e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:10.621 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.620+0000 7f0c0e33b700 1 --2- 172.21.15.195:0/3453285999 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0c081561d0 0x7f0c08156640 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:10.622 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.620+0000 7f0c0e33b700 1 --2- 172.21.15.195:0/3453285999 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0bf4005080 0x7f0c0816a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:10.622 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.620+0000 7f0c0e33b700 1 --2- 172.21.15.195:0/3453285999 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0c08158da0 0x7f0c0815bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:10.622 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.620+0000 7f0c0e33b700 1 -- 172.21.15.195:0/3453285999 >> 172.21.15.195:0/3453285999 conn(0x7f0c080c6350 msgr2=0x7f0c081506e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:10.622 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.620+0000 7f0c0e33b700 1 -- 172.21.15.195:0/3453285999 shutdown_connections 2022-06-27T11:48:10.623 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.620+0000 7f0c0e33b700 1 -- 172.21.15.195:0/3453285999 wait complete. 2022-06-27T11:48:10.632 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-06-27T11:48:10.755 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.755+0000 7fdd71cdf700 1 Processor -- start 2022-06-27T11:48:10.755 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.755+0000 7fdd71cdf700 1 -- start start 2022-06-27T11:48:10.764 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.756+0000 7fdd71cdf700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fdd6c150500 0x7fdd6c063af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:10.764 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.756+0000 7fdd71cdf700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fdd6c1508d0 0x7fdd6c0bbf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:10.765 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.756+0000 7fdd71cdf700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd6c064030 con 0x7fdd6c06a450 2022-06-27T11:48:10.765 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.756+0000 7fdd71cdf700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd6c0c4160 con 0x7fdd6c150500 2022-06-27T11:48:10.765 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.756+0000 7fdd71cdf700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd6c0c4440 con 0x7fdd6c1508d0 2022-06-27T11:48:10.765 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.756+0000 7fdd6affd700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fdd6c06a450 0x7fdd6c0c48d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:48230/0 (socket says 172.21.15.195:48230) 2022-06-27T11:48:10.766 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.756+0000 7fdd6affd700 1 -- 172.21.15.195:0/3166019612 learned_addr learned my addr 172.21.15.195:0/3166019612 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:48:10.766 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.757+0000 7fdd6b7fe700 1 --2- 172.21.15.195:0/3166019612 >> v2:172.21.15.202:3300/0 conn(0x7fdd6c150500 0x7fdd6c063af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:10.766 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.757+0000 7fdd6bfff700 1 --2- 172.21.15.195:0/3166019612 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fdd6c1508d0 0x7fdd6c0bbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:10.767 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.757+0000 7fdd6a7fc700 1 -- 172.21.15.195:0/3166019612 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3520580908 0 0) 0x7fdd54001110 con 0x7fdd6c06a450 2022-06-27T11:48:10.767 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.757+0000 7fdd6a7fc700 1 -- 172.21.15.195:0/3166019612 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd4c003430 con 0x7fdd6c06a450 2022-06-27T11:48:10.767 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.757+0000 7fdd6b7fe700 1 -- 172.21.15.195:0/3166019612 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fdd6c1508d0 msgr2=0x7fdd6c0bbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:10.768 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.757+0000 7fdd6b7fe700 1 --2- 172.21.15.195:0/3166019612 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fdd6c1508d0 0x7fdd6c0bbf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:10.768 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.757+0000 7fdd6a7fc700 1 -- 172.21.15.195:0/3166019612 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 710196164 0 0) 0x7fdd4c003430 con 0x7fdd6c06a450 2022-06-27T11:48:10.769 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.757+0000 7fdd6b7fe700 1 -- 172.21.15.195:0/3166019612 >> v1:172.21.15.195:6789/0 conn(0x7fdd6c06a450 legacy=0x7fdd6c0c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:10.769 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.757+0000 7fdd6b7fe700 1 -- 172.21.15.195:0/3166019612 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd6c15a800 con 0x7fdd6c150500 2022-06-27T11:48:10.770 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.757+0000 7fdd6b7fe700 1 --2- 172.21.15.195:0/3166019612 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fdd6c150500 0x7fdd6c063af0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c009720 tx=0x7fdd5c009a30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2fd802f45df4ac8d server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:10.770 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.757+0000 7fdd6a7fc700 1 -- 172.21.15.195:0/3166019612 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd54001600 con 0x7fdd6c150500 2022-06-27T11:48:10.770 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.757+0000 7fdd6a7fc700 1 -- 172.21.15.195:0/3166019612 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd5c004030 con 0x7fdd6c150500 2022-06-27T11:48:10.771 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.757+0000 7fdd6a7fc700 1 -- 172.21.15.195:0/3166019612 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd5c0043d0 con 0x7fdd6c150500 2022-06-27T11:48:10.771 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.758+0000 7fdd71cdf700 1 -- 172.21.15.195:0/3166019612 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fdd6c150500 msgr2=0x7fdd6c063af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:10.772 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.758+0000 7fdd71cdf700 1 --2- 172.21.15.195:0/3166019612 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fdd6c150500 0x7fdd6c063af0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c009720 tx=0x7fdd5c009a30 comp rx=0 tx=0).stop 2022-06-27T11:48:10.772 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.758+0000 7fdd71cdf700 1 -- 172.21.15.195:0/3166019612 shutdown_connections 2022-06-27T11:48:10.773 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.758+0000 7fdd71cdf700 1 --2- 172.21.15.195:0/3166019612 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fdd6c1508d0 0x7fdd6c0bbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:10.773 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.758+0000 7fdd71cdf700 1 --2- 172.21.15.195:0/3166019612 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fdd6c150500 0x7fdd6c063af0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:10.774 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.758+0000 7fdd71cdf700 1 -- 172.21.15.195:0/3166019612 >> 172.21.15.195:0/3166019612 conn(0x7fdd6c0c9b00 msgr2=0x7fdd6c0c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:10.774 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.758+0000 7fdd71cdf700 1 -- 172.21.15.195:0/3166019612 shutdown_connections 2022-06-27T11:48:10.775 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.758+0000 7fdd71cdf700 1 -- 172.21.15.195:0/3166019612 wait complete. 2022-06-27T11:48:10.775 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.759+0000 7fdd71cdf700 1 Processor -- start 2022-06-27T11:48:10.776 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.759+0000 7fdd71cdf700 1 -- start start 2022-06-27T11:48:10.776 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.760+0000 7fdd71cdf700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fdd6c06a450 0x7fdd6c0c21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:10.776 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.760+0000 7fdd71cdf700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdd6c150500 0x7fdd6c16a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:10.777 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.760+0000 7fdd71cdf700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fdd6c1508d0 0x7fdd6c15bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:10.777 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.760+0000 7fdd71cdf700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd6c0c2720 con 0x7fdd6c150500 2022-06-27T11:48:10.778 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.760+0000 7fdd71cdf700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd6c16a130 con 0x7fdd6c1508d0 2022-06-27T11:48:10.779 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.760+0000 7fdd71cdf700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd6c068f10 con 0x7fdd6c06a450 2022-06-27T11:48:10.779 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.760+0000 7fdd6affd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdd6c150500 0x7fdd6c16a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:10.779 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.760+0000 7fdd6affd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdd6c150500 0x7fdd6c16a5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:42050/0 (socket says 172.21.15.195:42050) 2022-06-27T11:48:10.780 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.760+0000 7fdd6affd700 1 -- 172.21.15.195:0/955632085 learned_addr learned my addr 172.21.15.195:0/955632085 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:48:10.780 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.760+0000 7fdd6bfff700 1 --2- 172.21.15.195:0/955632085 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fdd6c1508d0 0x7fdd6c15bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:10.780 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.760+0000 7fdd6b7fe700 1 --2- 172.21.15.195:0/955632085 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fdd6c06a450 0x7fdd6c0c21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:10.781 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.760+0000 7fdd6affd700 1 -- 172.21.15.195:0/955632085 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fdd6c06a450 msgr2=0x7fdd6c0c21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:10.781 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.760+0000 7fdd6affd700 1 --2- 172.21.15.195:0/955632085 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fdd6c06a450 0x7fdd6c0c21e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:10.782 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.760+0000 7fdd6affd700 1 -- 172.21.15.195:0/955632085 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fdd6c1508d0 msgr2=0x7fdd6c15bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:10.782 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.760+0000 7fdd6affd700 1 --2- 172.21.15.195:0/955632085 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fdd6c1508d0 0x7fdd6c15bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:10.783 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.760+0000 7fdd6affd700 1 -- 172.21.15.195:0/955632085 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd5c009290 con 0x7fdd6c150500 2022-06-27T11:48:10.783 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.760+0000 7fdd6affd700 1 --2- 172.21.15.195:0/955632085 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdd6c150500 0x7fdd6c16a5c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fdd5400c810 tx=0x7fdd54000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:10.784 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.761+0000 7fdd637fe700 1 -- 172.21.15.195:0/955632085 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd54007360 con 0x7fdd6c150500 2022-06-27T11:48:10.784 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.761+0000 7fdd637fe700 1 -- 172.21.15.195:0/955632085 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd540074e0 con 0x7fdd6c150500 2022-06-27T11:48:10.785 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.761+0000 7fdd71cdf700 1 -- 172.21.15.195:0/955632085 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd6c169e00 con 0x7fdd6c150500 2022-06-27T11:48:10.785 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.761+0000 7fdd637fe700 1 -- 172.21.15.195:0/955632085 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd54004940 con 0x7fdd6c150500 2022-06-27T11:48:10.785 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.761+0000 7fdd71cdf700 1 -- 172.21.15.195:0/955632085 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd6c0bee20 con 0x7fdd6c150500 2022-06-27T11:48:10.786 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.762+0000 7fdd71cdf700 1 -- 172.21.15.195:0/955632085 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd6c0722c0 con 0x7fdd6c150500 2022-06-27T11:48:10.786 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.762+0000 7fdd637fe700 1 -- 172.21.15.195:0/955632085 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fdd5401a030 con 0x7fdd6c150500 2022-06-27T11:48:10.786 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.762+0000 7fdd637fe700 1 --2- 172.21.15.195:0/955632085 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fdd50063d00 0x7fdd500661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:10.787 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.762+0000 7fdd637fe700 1 -- 172.21.15.195:0/955632085 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fdd5407fd00 con 0x7fdd6c150500 2022-06-27T11:48:10.787 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.762+0000 7fdd6b7fe700 1 --2- 172.21.15.195:0/955632085 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fdd50063d00 0x7fdd500661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:10.788 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.763+0000 7fdd6b7fe700 1 --2- 172.21.15.195:0/955632085 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fdd50063d00 0x7fdd500661b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c009a70 tx=0x7fdd5c015000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:10.789 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.767+0000 7fdd637fe700 1 -- 172.21.15.195:0/955632085 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fdd54083030 con 0x7fdd6c150500 2022-06-27T11:48:10.948 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.949+0000 7fdd71cdf700 1 -- 172.21.15.195:0/955632085 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fdd6c064030 con 0x7fdd6c150500 2022-06-27T11:48:10.949 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.949+0000 7fdd637fe700 1 -- 172.21.15.195:0/955632085 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fdd6c064030 con 0x7fdd6c150500 2022-06-27T11:48:10.949 INFO:teuthology.orchestra.run.smithi195.stdout:pg_num: 64 2022-06-27T11:48:10.952 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.951+0000 7fdd71cdf700 1 -- 172.21.15.195:0/955632085 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fdd50063d00 msgr2=0x7fdd500661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:10.952 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.951+0000 7fdd71cdf700 1 --2- 172.21.15.195:0/955632085 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fdd50063d00 0x7fdd500661b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c009a70 tx=0x7fdd5c015000 comp rx=0 tx=0).stop 2022-06-27T11:48:10.952 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.951+0000 7fdd71cdf700 1 -- 172.21.15.195:0/955632085 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdd6c150500 msgr2=0x7fdd6c16a5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:10.953 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.951+0000 7fdd71cdf700 1 --2- 172.21.15.195:0/955632085 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdd6c150500 0x7fdd6c16a5c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fdd5400c810 tx=0x7fdd54000ed0 comp rx=0 tx=0).stop 2022-06-27T11:48:10.953 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.952+0000 7fdd71cdf700 1 -- 172.21.15.195:0/955632085 shutdown_connections 2022-06-27T11:48:10.953 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.952+0000 7fdd71cdf700 1 --2- 172.21.15.195:0/955632085 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fdd50063d00 0x7fdd500661b0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:10.953 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.952+0000 7fdd71cdf700 1 --2- 172.21.15.195:0/955632085 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fdd6c1508d0 0x7fdd6c15bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:10.954 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.952+0000 7fdd71cdf700 1 --2- 172.21.15.195:0/955632085 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fdd6c06a450 0x7fdd6c0c21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:10.954 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.952+0000 7fdd71cdf700 1 --2- 172.21.15.195:0/955632085 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdd6c150500 0x7fdd6c16a5c0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:10.954 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.952+0000 7fdd71cdf700 1 -- 172.21.15.195:0/955632085 >> 172.21.15.195:0/955632085 conn(0x7fdd6c0c9b00 msgr2=0x7fdd6c067ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:10.955 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.952+0000 7fdd71cdf700 1 -- 172.21.15.195:0/955632085 shutdown_connections 2022-06-27T11:48:10.955 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:10.952+0000 7fdd71cdf700 1 -- 172.21.15.195:0/955632085 wait complete. 2022-06-27T11:48:10.964 INFO:tasks.mds_thrash:Wait for all MDSs to reach steady state... 2022-06-27T11:48:10.964 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:48:11.084 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.084+0000 7f41b1122700 1 Processor -- start 2022-06-27T11:48:11.084 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.085+0000 7f41b1122700 1 -- start start 2022-06-27T11:48:11.089 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.085+0000 7f41b1122700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f41ac06eff0 0x7f41ac0c61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:11.090 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.085+0000 7f41b1122700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f41ac0c31d0 0x7f41ac0c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:11.090 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.085+0000 7f41b1122700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41ac14d8a0 con 0x7f41ac067ed0 2022-06-27T11:48:11.090 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.085+0000 7f41b1122700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41ac0c47e0 con 0x7f41ac0c31d0 2022-06-27T11:48:11.091 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.085+0000 7f41b1122700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41ac0c3e10 con 0x7f41ac06eff0 2022-06-27T11:48:11.091 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.085+0000 7f41ab59e700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f41ac0c31d0 0x7f41ac0c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:11.091 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.085+0000 7f41ab59e700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f41ac0c31d0 0x7f41ac0c42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:42058/0 (socket says 172.21.15.195:42058) 2022-06-27T11:48:11.091 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.085+0000 7f41ab59e700 1 -- 172.21.15.195:0/1619000213 learned_addr learned my addr 172.21.15.195:0/1619000213 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:48:11.092 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.086+0000 7f41aad9d700 1 --2- 172.21.15.195:0/1619000213 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f41ac06eff0 0x7f41ac0c61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:11.092 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.086+0000 7f41ab59e700 1 -- 172.21.15.195:0/1619000213 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f41ac06eff0 msgr2=0x7f41ac0c61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:11.092 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.086+0000 7f41ab59e700 1 --2- 172.21.15.195:0/1619000213 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f41ac06eff0 0x7f41ac0c61e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:11.092 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.086+0000 7f41ab59e700 1 -- 172.21.15.195:0/1619000213 >> v1:172.21.15.202:6789/0 conn(0x7f41ac067ed0 legacy=0x7f41ac0c2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:11.092 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.086+0000 7f41ab59e700 1 -- 172.21.15.195:0/1619000213 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41ac06d970 con 0x7f41ac0c31d0 2022-06-27T11:48:11.093 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.086+0000 7f41ab59e700 1 --2- 172.21.15.195:0/1619000213 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f41ac0c31d0 0x7f41ac0c42a0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f41a4009f20 tx=0x7f41a4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40463f7986cace26 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:11.093 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.086+0000 7f41a9d9b700 1 -- 172.21.15.195:0/1619000213 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41a400e070 con 0x7f41ac0c31d0 2022-06-27T11:48:11.093 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.086+0000 7f41a9d9b700 1 -- 172.21.15.195:0/1619000213 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41a400ba60 con 0x7f41ac0c31d0 2022-06-27T11:48:11.093 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.086+0000 7f41a9d9b700 1 -- 172.21.15.195:0/1619000213 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41a40092e0 con 0x7f41ac0c31d0 2022-06-27T11:48:11.094 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.086+0000 7f41b1122700 1 -- 172.21.15.195:0/1619000213 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f41ac0c31d0 msgr2=0x7f41ac0c42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:11.094 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.086+0000 7f41b1122700 1 --2- 172.21.15.195:0/1619000213 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f41ac0c31d0 0x7f41ac0c42a0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f41a4009f20 tx=0x7f41a4009840 comp rx=0 tx=0).stop 2022-06-27T11:48:11.094 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.087+0000 7f41b1122700 1 -- 172.21.15.195:0/1619000213 shutdown_connections 2022-06-27T11:48:11.094 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.087+0000 7f41b1122700 1 --2- 172.21.15.195:0/1619000213 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f41ac0c31d0 0x7f41ac0c42a0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:11.095 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.087+0000 7f41b1122700 1 --2- 172.21.15.195:0/1619000213 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f41ac06eff0 0x7f41ac0c61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:11.095 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.087+0000 7f41b1122700 1 -- 172.21.15.195:0/1619000213 >> 172.21.15.195:0/1619000213 conn(0x7f41ac0c9b00 msgr2=0x7f41ac0c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:11.095 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.087+0000 7f41b1122700 1 -- 172.21.15.195:0/1619000213 shutdown_connections 2022-06-27T11:48:11.095 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.087+0000 7f41b1122700 1 -- 172.21.15.195:0/1619000213 wait complete. 2022-06-27T11:48:11.095 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.088+0000 7f41b1122700 1 Processor -- start 2022-06-27T11:48:11.096 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.088+0000 7f41b1122700 1 -- start start 2022-06-27T11:48:11.096 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.089+0000 7f41b1122700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f41ac067ed0 0x7f41ac1402a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:11.096 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.089+0000 7f41b1122700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f41ac06eff0 0x7f41ac141eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:11.096 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.089+0000 7f41b1122700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f41ac146590 0x7f41ac1469e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:11.097 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.089+0000 7f41b1122700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41ac141280 con 0x7f41ac067ed0 2022-06-27T11:48:11.097 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.089+0000 7f41b1122700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41ac1407e0 con 0x7f41ac146590 2022-06-27T11:48:11.097 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.089+0000 7f41b1122700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41ac142b00 con 0x7f41ac06eff0 2022-06-27T11:48:11.097 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.089+0000 7f41aad9d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f41ac067ed0 0x7f41ac1402a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:11.098 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.089+0000 7f41aad9d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f41ac067ed0 0x7f41ac1402a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:42060/0 (socket says 172.21.15.195:42060) 2022-06-27T11:48:11.098 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.089+0000 7f41aad9d700 1 -- 172.21.15.195:0/2242102830 learned_addr learned my addr 172.21.15.195:0/2242102830 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:48:11.098 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.089+0000 7f41ab59e700 1 --2- 172.21.15.195:0/2242102830 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f41ac146590 0x7f41ac1469e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:11.098 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.089+0000 7f41aa59c700 1 --2- 172.21.15.195:0/2242102830 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f41ac06eff0 0x7f41ac141eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:11.098 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.089+0000 7f41aad9d700 1 -- 172.21.15.195:0/2242102830 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f41ac06eff0 msgr2=0x7f41ac141eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:11.099 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.089+0000 7f41aad9d700 1 --2- 172.21.15.195:0/2242102830 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f41ac06eff0 0x7f41ac141eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:11.099 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.089+0000 7f41aad9d700 1 -- 172.21.15.195:0/2242102830 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f41ac146590 msgr2=0x7f41ac1469e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:11.099 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.089+0000 7f41aad9d700 1 --2- 172.21.15.195:0/2242102830 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f41ac146590 0x7f41ac1469e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:11.099 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.089+0000 7f41aad9d700 1 -- 172.21.15.195:0/2242102830 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41a4009ae0 con 0x7f41ac067ed0 2022-06-27T11:48:11.100 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.089+0000 7f41aad9d700 1 --2- 172.21.15.195:0/2242102830 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f41ac067ed0 0x7f41ac1402a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f41a000e990 tx=0x7f41a0009550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:11.100 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.089+0000 7f4192ffd700 1 -- 172.21.15.195:0/2242102830 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41a0013680 con 0x7f41ac067ed0 2022-06-27T11:48:11.100 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.089+0000 7f4192ffd700 1 -- 172.21.15.195:0/2242102830 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41a00137e0 con 0x7f41ac067ed0 2022-06-27T11:48:11.100 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.089+0000 7f41b1122700 1 -- 172.21.15.195:0/2242102830 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41ac1423f0 con 0x7f41ac067ed0 2022-06-27T11:48:11.101 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.089+0000 7f4192ffd700 1 -- 172.21.15.195:0/2242102830 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41a0013b60 con 0x7f41ac067ed0 2022-06-27T11:48:11.101 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.089+0000 7f41b1122700 1 -- 172.21.15.195:0/2242102830 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41ac1409e0 con 0x7f41ac067ed0 2022-06-27T11:48:11.101 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.090+0000 7f41b1122700 1 -- 172.21.15.195:0/2242102830 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41ac0722c0 con 0x7f41ac067ed0 2022-06-27T11:48:11.102 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.091+0000 7f4192ffd700 1 -- 172.21.15.195:0/2242102830 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f41a0023070 con 0x7f41ac067ed0 2022-06-27T11:48:11.103 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.091+0000 7f4192ffd700 1 --2- 172.21.15.195:0/2242102830 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f418c063cd0 0x7f418c066180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:11.103 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.091+0000 7f4192ffd700 1 -- 172.21.15.195:0/2242102830 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f41a0080bc0 con 0x7f41ac067ed0 2022-06-27T11:48:11.103 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.091+0000 7f41aa59c700 1 --2- 172.21.15.195:0/2242102830 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f418c063cd0 0x7f418c066180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:11.103 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.092+0000 7f41aa59c700 1 --2- 172.21.15.195:0/2242102830 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f418c063cd0 0x7f418c066180 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f41ac13e570 tx=0x7f419c009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:11.104 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.095+0000 7f4192ffd700 1 -- 172.21.15.195:0/2242102830 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f41a004f7b0 con 0x7f41ac067ed0 2022-06-27T11:48:11.318 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.318+0000 7f41b1122700 1 -- 172.21.15.195:0/2242102830 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f41ac153f20 con 0x7f41ac067ed0 2022-06-27T11:48:11.319 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.319+0000 7f4192ffd700 1 -- 172.21.15.195:0/2242102830 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6119 (secure 0 0 0) 0x7f41a0021090 con 0x7f41ac067ed0 2022-06-27T11:48:11.320 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:48:11.320 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4287,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/864126677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":864126677},{"type":"v1","addr":"172.21.15.202:6833","nonce":864126677}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:47:57.373108+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.202:6835/2038169702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2038169702},{"type":"v1","addr":"172.21.15.202:6835","nonce":2038169702}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:48:11.323 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.321+0000 7f41b1122700 1 -- 172.21.15.195:0/2242102830 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f418c063cd0 msgr2=0x7f418c066180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:11.324 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.321+0000 7f41b1122700 1 --2- 172.21.15.195:0/2242102830 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f418c063cd0 0x7f418c066180 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f41ac13e570 tx=0x7f419c009040 comp rx=0 tx=0).stop 2022-06-27T11:48:11.324 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.321+0000 7f41b1122700 1 -- 172.21.15.195:0/2242102830 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f41ac067ed0 msgr2=0x7f41ac1402a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:11.325 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.321+0000 7f41b1122700 1 --2- 172.21.15.195:0/2242102830 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f41ac067ed0 0x7f41ac1402a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f41a000e990 tx=0x7f41a0009550 comp rx=0 tx=0).stop 2022-06-27T11:48:11.325 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.321+0000 7f41b1122700 1 -- 172.21.15.195:0/2242102830 shutdown_connections 2022-06-27T11:48:11.326 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.321+0000 7f41b1122700 1 --2- 172.21.15.195:0/2242102830 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f418c063cd0 0x7f418c066180 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:11.326 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.321+0000 7f41b1122700 1 --2- 172.21.15.195:0/2242102830 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f41ac067ed0 0x7f41ac1402a0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:11.327 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.321+0000 7f41b1122700 1 --2- 172.21.15.195:0/2242102830 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f41ac146590 0x7f41ac1469e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:11.327 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.321+0000 7f41b1122700 1 --2- 172.21.15.195:0/2242102830 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f41ac06eff0 0x7f41ac141eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:11.327 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.321+0000 7f41b1122700 1 -- 172.21.15.195:0/2242102830 >> 172.21.15.195:0/2242102830 conn(0x7f41ac0c9b00 msgr2=0x7f41ac0682e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:11.328 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.321+0000 7f41b1122700 1 -- 172.21.15.195:0/2242102830 shutdown_connections 2022-06-27T11:48:11.328 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.321+0000 7f41b1122700 1 -- 172.21.15.195:0/2242102830 wait complete. 2022-06-27T11:48:11.329 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 8 2022-06-27T11:48:11.333 INFO:tasks.mds_thrash:Ready to start thrashing 2022-06-27T11:48:11.334 INFO:tasks.mds_thrash.ceph_manager:waiting for clean 2022-06-27T11:48:11.335 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2022-06-27T11:48:11.456 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.456+0000 7f7f4f7bf700 1 Processor -- start 2022-06-27T11:48:11.457 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.457+0000 7f7f4f7bf700 1 -- start start 2022-06-27T11:48:11.463 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.457+0000 7f7f4f7bf700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f7f48156100 0x7f7f480688a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:11.463 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.457+0000 7f7f4f7bf700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7f4806a440 0x7f7f480c4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:11.464 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.457+0000 7f7f4f7bf700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f480c4b10 con 0x7f7f481564d0 2022-06-27T11:48:11.464 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.457+0000 7f7f4f7bf700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f480c3dc0 con 0x7f7f48156100 2022-06-27T11:48:11.464 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.457+0000 7f7f4f7bf700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f48068f20 con 0x7f7f4806a440 2022-06-27T11:48:11.464 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.458+0000 7f7f4cd5a700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7f4806a440 0x7f7f480c4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:11.464 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.458+0000 7f7f4d55b700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f7f48156100 0x7f7f480688a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:11.465 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.458+0000 7f7f4d55b700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f7f48156100 0x7f7f480688a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3300/0 says I am v2:172.21.15.195:40194/0 (socket says 172.21.15.195:40194) 2022-06-27T11:48:11.465 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.458+0000 7f7f4dd5c700 1 --1- >> v1:172.21.15.202:6789/0 conn(0x7f7f481564d0 0x7f7f480c28c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.202:6789/0 says I am v1:172.21.15.195:50120/0 (socket says 172.21.15.195:50120) 2022-06-27T11:48:11.465 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.458+0000 7f7f4cd5a700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7f4806a440 0x7f7f480c4fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3301/0 says I am v2:172.21.15.195:34826/0 (socket says 172.21.15.195:34826) 2022-06-27T11:48:11.465 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.458+0000 7f7f4d55b700 1 -- 172.21.15.195:0/2933566020 learned_addr learned my addr 172.21.15.195:0/2933566020 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:48:11.466 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.458+0000 7f7f4d55b700 1 -- 172.21.15.195:0/2933566020 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7f4806a440 msgr2=0x7f7f480c4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:11.466 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.458+0000 7f7f4d55b700 1 --2- 172.21.15.195:0/2933566020 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7f4806a440 0x7f7f480c4fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:11.466 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.458+0000 7f7f4d55b700 1 -- 172.21.15.195:0/2933566020 >> v1:172.21.15.202:6789/0 conn(0x7f7f481564d0 legacy=0x7f7f480c28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:11.466 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.458+0000 7f7f4d55b700 1 -- 172.21.15.195:0/2933566020 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f480c6360 con 0x7f7f48156100 2022-06-27T11:48:11.467 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.458+0000 7f7f37fff700 1 -- 172.21.15.195:0/2933566020 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 934992710 0 0) 0x7f7f480c4b10 con 0x7f7f481564d0 2022-06-27T11:48:11.467 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.458+0000 7f7f4cd5a700 1 --2- 172.21.15.195:0/2933566020 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7f4806a440 0x7f7f480c4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:48:11.468 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.458+0000 7f7f4d55b700 1 --2- 172.21.15.195:0/2933566020 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7f48156100 0x7f7f480688a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7f38006fd0 tx=0x7f7f380090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=882b6e85961e9a4f server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:11.468 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.459+0000 7f7f37fff700 1 -- 172.21.15.195:0/2933566020 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f38005ad0 con 0x7f7f48156100 2022-06-27T11:48:11.468 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.459+0000 7f7f37fff700 1 -- 172.21.15.195:0/2933566020 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f38005c30 con 0x7f7f48156100 2022-06-27T11:48:11.468 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.459+0000 7f7f37fff700 1 -- 172.21.15.195:0/2933566020 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f38004030 con 0x7f7f48156100 2022-06-27T11:48:11.468 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.459+0000 7f7f4f7bf700 1 -- 172.21.15.195:0/2933566020 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7f48156100 msgr2=0x7f7f480688a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:11.469 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.459+0000 7f7f4f7bf700 1 --2- 172.21.15.195:0/2933566020 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7f48156100 0x7f7f480688a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7f38006fd0 tx=0x7f7f380090f0 comp rx=0 tx=0).stop 2022-06-27T11:48:11.469 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.459+0000 7f7f4f7bf700 1 -- 172.21.15.195:0/2933566020 shutdown_connections 2022-06-27T11:48:11.469 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.459+0000 7f7f4f7bf700 1 --2- 172.21.15.195:0/2933566020 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7f48156100 0x7f7f480688a0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:11.469 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.459+0000 7f7f4f7bf700 1 --2- 172.21.15.195:0/2933566020 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7f4806a440 0x7f7f480c4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:11.470 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.459+0000 7f7f4f7bf700 1 -- 172.21.15.195:0/2933566020 >> 172.21.15.195:0/2933566020 conn(0x7f7f480c9b00 msgr2=0x7f7f480c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:11.470 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.459+0000 7f7f4f7bf700 1 -- 172.21.15.195:0/2933566020 shutdown_connections 2022-06-27T11:48:11.470 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.459+0000 7f7f4f7bf700 1 -- 172.21.15.195:0/2933566020 wait complete. 2022-06-27T11:48:11.470 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.460+0000 7f7f4f7bf700 1 Processor -- start 2022-06-27T11:48:11.471 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.460+0000 7f7f4f7bf700 1 -- start start 2022-06-27T11:48:11.471 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.461+0000 7f7f4f7bf700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7f4806a440 0x7f7f480bf890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:11.471 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.461+0000 7f7f4f7bf700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7f48156100 0x7f7f480c2870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:11.471 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.461+0000 7f7f4f7bf700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7f481564d0 0x7f7f480bd130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:11.472 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.461+0000 7f7f4f7bf700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f48155dd0 con 0x7f7f48156100 2022-06-27T11:48:11.472 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.461+0000 7f7f4f7bf700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f480c8e70 con 0x7f7f481564d0 2022-06-27T11:48:11.472 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.461+0000 7f7f4f7bf700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f48155940 con 0x7f7f4806a440 2022-06-27T11:48:11.472 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.461+0000 7f7f4cd5a700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7f48156100 0x7f7f480c2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:11.473 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.461+0000 7f7f4cd5a700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7f48156100 0x7f7f480c2870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:42074/0 (socket says 172.21.15.195:42074) 2022-06-27T11:48:11.473 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.461+0000 7f7f4cd5a700 1 -- 172.21.15.195:0/2525608033 learned_addr learned my addr 172.21.15.195:0/2525608033 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:48:11.473 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.461+0000 7f7f4dd5c700 1 --2- 172.21.15.195:0/2525608033 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7f481564d0 0x7f7f480bd130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:11.473 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.461+0000 7f7f4d55b700 1 --2- 172.21.15.195:0/2525608033 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7f4806a440 0x7f7f480bf890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:11.474 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.462+0000 7f7f4cd5a700 1 -- 172.21.15.195:0/2525608033 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7f4806a440 msgr2=0x7f7f480bf890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:11.474 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.462+0000 7f7f4cd5a700 1 --2- 172.21.15.195:0/2525608033 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7f4806a440 0x7f7f480bf890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:11.474 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.462+0000 7f7f4cd5a700 1 -- 172.21.15.195:0/2525608033 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7f481564d0 msgr2=0x7f7f480bd130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:11.474 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.462+0000 7f7f4cd5a700 1 --2- 172.21.15.195:0/2525608033 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7f481564d0 0x7f7f480bd130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:11.475 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.462+0000 7f7f4cd5a700 1 -- 172.21.15.195:0/2525608033 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f440092e0 con 0x7f7f48156100 2022-06-27T11:48:11.475 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.462+0000 7f7f4cd5a700 1 --2- 172.21.15.195:0/2525608033 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7f48156100 0x7f7f480c2870 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f7f4400eaa0 tx=0x7f7f44009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:11.475 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.462+0000 7f7f357fa700 1 -- 172.21.15.195:0/2525608033 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f4400eee0 con 0x7f7f48156100 2022-06-27T11:48:11.475 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.462+0000 7f7f357fa700 1 -- 172.21.15.195:0/2525608033 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f4400c850 con 0x7f7f48156100 2022-06-27T11:48:11.476 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.462+0000 7f7f4f7bf700 1 -- 172.21.15.195:0/2525608033 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f38009350 con 0x7f7f48156100 2022-06-27T11:48:11.476 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.462+0000 7f7f357fa700 1 -- 172.21.15.195:0/2525608033 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f4400cbd0 con 0x7f7f48156100 2022-06-27T11:48:11.476 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.462+0000 7f7f4f7bf700 1 -- 172.21.15.195:0/2525608033 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f48155010 con 0x7f7f48156100 2022-06-27T11:48:11.476 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.463+0000 7f7f4f7bf700 1 -- 172.21.15.195:0/2525608033 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f480722c0 con 0x7f7f48156100 2022-06-27T11:48:11.476 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.463+0000 7f7f357fa700 1 -- 172.21.15.195:0/2525608033 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f7f44019030 con 0x7f7f48156100 2022-06-27T11:48:11.477 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.464+0000 7f7f357fa700 1 --2- 172.21.15.195:0/2525608033 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7f28063d00 0x7f7f280661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:11.477 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.464+0000 7f7f357fa700 1 -- 172.21.15.195:0/2525608033 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f7f440808f0 con 0x7f7f48156100 2022-06-27T11:48:11.477 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.464+0000 7f7f4d55b700 1 --2- 172.21.15.195:0/2525608033 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7f28063d00 0x7f7f280661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:11.478 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.464+0000 7f7f4d55b700 1 --2- 172.21.15.195:0/2525608033 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7f28063d00 0x7f7f280661b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7f38009130 tx=0x7f7f38015000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:11.478 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.468+0000 7f7f357fa700 1 -- 172.21.15.195:0/2525608033 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f7f4404f480 con 0x7f7f48156100 2022-06-27T11:48:11.642 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.640+0000 7f7f4f7bf700 1 -- 172.21.15.195:0/2525608033 --> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7f480c6640 con 0x7f7f28063d00 2022-06-27T11:48:11.647 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.648+0000 7f7f357fa700 1 -- 172.21.15.195:0/2525608033 <== mgr.4098 v2:172.21.15.195:6832/93955 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+284935 (secure 0 0 0) 0x7f7f480c6640 con 0x7f7f28063d00 2022-06-27T11:48:11.652 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:48:11.654 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.651+0000 7f7f4f7bf700 1 -- 172.21.15.195:0/2525608033 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7f28063d00 msgr2=0x7f7f280661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:11.655 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.651+0000 7f7f4f7bf700 1 --2- 172.21.15.195:0/2525608033 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7f28063d00 0x7f7f280661b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7f38009130 tx=0x7f7f38015000 comp rx=0 tx=0).stop 2022-06-27T11:48:11.655 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.651+0000 7f7f4f7bf700 1 -- 172.21.15.195:0/2525608033 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7f48156100 msgr2=0x7f7f480c2870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:11.655 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.651+0000 7f7f4f7bf700 1 --2- 172.21.15.195:0/2525608033 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7f48156100 0x7f7f480c2870 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f7f4400eaa0 tx=0x7f7f44009060 comp rx=0 tx=0).stop 2022-06-27T11:48:11.655 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.651+0000 7f7f4f7bf700 1 -- 172.21.15.195:0/2525608033 shutdown_connections 2022-06-27T11:48:11.656 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.651+0000 7f7f4f7bf700 1 --2- 172.21.15.195:0/2525608033 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7f28063d00 0x7f7f280661b0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:11.656 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.651+0000 7f7f4f7bf700 1 --2- 172.21.15.195:0/2525608033 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7f481564d0 0x7f7f480bd130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:11.657 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.651+0000 7f7f4f7bf700 1 --2- 172.21.15.195:0/2525608033 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7f4806a440 0x7f7f480bf890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:11.657 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.651+0000 7f7f4f7bf700 1 --2- 172.21.15.195:0/2525608033 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7f48156100 0x7f7f480c2870 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:11.657 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.651+0000 7f7f4f7bf700 1 -- 172.21.15.195:0/2525608033 >> 172.21.15.195:0/2525608033 conn(0x7f7f480c9b00 msgr2=0x7f7f4806ebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:11.657 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.651+0000 7f7f4f7bf700 1 -- 172.21.15.195:0/2525608033 shutdown_connections 2022-06-27T11:48:11.658 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.651+0000 7f7f4f7bf700 1 -- 172.21.15.195:0/2525608033 wait complete. 2022-06-27T11:48:11.658 INFO:teuthology.orchestra.run.smithi195.stderr:dumped all 2022-06-27T11:48:11.663 INFO:teuthology.orchestra.run.smithi195.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2022-06-27T11:48:10.343250+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592958,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":205,"num_write_kb":2431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":210,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":68760,"kb_used_data":1880,"kb_used_omap":0,"kb_used_meta":66816,"kb_avail":754905960,"statfs":{"total":773094113280,"available":773023703040,"internally_reserved":0,"allocated":1925120,"data_stored":1814622,"data_compressed":64756,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":68419584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2590,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002303"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384962+0000","last_change":"2022-06-27T11:47:52.381823+0000","last_active":"2022-06-27T11:47:52.384962+0000","last_peered":"2022-06-27T11:47:52.384962+0000","last_clean":"2022-06-27T11:47:52.384962+0000","last_became_active":"2022-06-27T11:47:52.381551+0000","last_became_peered":"2022-06-27T11:47:52.381551+0000","last_unstale":"2022-06-27T11:47:52.384962+0000","last_undegraded":"2022-06-27T11:47:52.384962+0000","last_fullsized":"2022-06-27T11:47:52.384962+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:17:34.164061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.378135+0000","last_change":"2022-06-27T11:47:52.374939+0000","last_active":"2022-06-27T11:47:52.378135+0000","last_peered":"2022-06-27T11:47:52.378135+0000","last_clean":"2022-06-27T11:47:52.378135+0000","last_became_active":"2022-06-27T11:47:52.374620+0000","last_became_peered":"2022-06-27T11:47:52.374620+0000","last_unstale":"2022-06-27T11:47:52.378135+0000","last_undegraded":"2022-06-27T11:47:52.378135+0000","last_fullsized":"2022-06-27T11:47:52.378135+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:56:27.254098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384127+0000","last_change":"2022-06-27T11:47:52.380802+0000","last_active":"2022-06-27T11:47:52.384127+0000","last_peered":"2022-06-27T11:47:52.384127+0000","last_clean":"2022-06-27T11:47:52.384127+0000","last_became_active":"2022-06-27T11:47:52.380500+0000","last_became_peered":"2022-06-27T11:47:52.380500+0000","last_unstale":"2022-06-27T11:47:52.384127+0000","last_undegraded":"2022-06-27T11:47:52.384127+0000","last_fullsized":"2022-06-27T11:47:52.384127+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:06:42.889598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384114+0000","last_change":"2022-06-27T11:47:52.380943+0000","last_active":"2022-06-27T11:47:52.384114+0000","last_peered":"2022-06-27T11:47:52.384114+0000","last_clean":"2022-06-27T11:47:52.384114+0000","last_became_active":"2022-06-27T11:47:52.380648+0000","last_became_peered":"2022-06-27T11:47:52.380648+0000","last_unstale":"2022-06-27T11:47:52.384114+0000","last_undegraded":"2022-06-27T11:47:52.384114+0000","last_fullsized":"2022-06-27T11:47:52.384114+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:14:19.729605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.386875+0000","last_change":"2022-06-27T11:47:52.386144+0000","last_active":"2022-06-27T11:47:52.386875+0000","last_peered":"2022-06-27T11:47:52.386875+0000","last_clean":"2022-06-27T11:47:52.386875+0000","last_became_active":"2022-06-27T11:47:52.385586+0000","last_became_peered":"2022-06-27T11:47:52.385586+0000","last_unstale":"2022-06-27T11:47:52.386875+0000","last_undegraded":"2022-06-27T11:47:52.386875+0000","last_fullsized":"2022-06-27T11:47:52.386875+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T14:30:03.660749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.387182+0000","last_change":"2022-06-27T11:47:52.381906+0000","last_active":"2022-06-27T11:47:52.387182+0000","last_peered":"2022-06-27T11:47:52.387182+0000","last_clean":"2022-06-27T11:47:52.387182+0000","last_became_active":"2022-06-27T11:47:52.381279+0000","last_became_peered":"2022-06-27T11:47:52.381279+0000","last_unstale":"2022-06-27T11:47:52.387182+0000","last_undegraded":"2022-06-27T11:47:52.387182+0000","last_fullsized":"2022-06-27T11:47:52.387182+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T18:41:03.991762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.383137+0000","last_change":"2022-06-27T11:47:52.377128+0000","last_active":"2022-06-27T11:47:52.383137+0000","last_peered":"2022-06-27T11:47:52.383137+0000","last_clean":"2022-06-27T11:47:52.383137+0000","last_became_active":"2022-06-27T11:47:52.376499+0000","last_became_peered":"2022-06-27T11:47:52.376499+0000","last_unstale":"2022-06-27T11:47:52.383137+0000","last_undegraded":"2022-06-27T11:47:52.383137+0000","last_fullsized":"2022-06-27T11:47:52.383137+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:00:13.208145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.378171+0000","last_change":"2022-06-27T11:47:52.374169+0000","last_active":"2022-06-27T11:47:52.378171+0000","last_peered":"2022-06-27T11:47:52.378171+0000","last_clean":"2022-06-27T11:47:52.378171+0000","last_became_active":"2022-06-27T11:47:52.373917+0000","last_became_peered":"2022-06-27T11:47:52.373917+0000","last_unstale":"2022-06-27T11:47:52.378171+0000","last_undegraded":"2022-06-27T11:47:52.378171+0000","last_fullsized":"2022-06-27T11:47:52.378171+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:43:52.741254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384769+0000","last_change":"2022-06-27T11:47:52.382039+0000","last_active":"2022-06-27T11:47:52.384769+0000","last_peered":"2022-06-27T11:47:52.384769+0000","last_clean":"2022-06-27T11:47:52.384769+0000","last_became_active":"2022-06-27T11:47:52.381202+0000","last_became_peered":"2022-06-27T11:47:52.381202+0000","last_unstale":"2022-06-27T11:47:52.384769+0000","last_undegraded":"2022-06-27T11:47:52.384769+0000","last_fullsized":"2022-06-27T11:47:52.384769+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:13:02.553823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.383894+0000","last_change":"2022-06-27T11:47:52.380839+0000","last_active":"2022-06-27T11:47:52.383894+0000","last_peered":"2022-06-27T11:47:52.383894+0000","last_clean":"2022-06-27T11:47:52.383894+0000","last_became_active":"2022-06-27T11:47:52.380527+0000","last_became_peered":"2022-06-27T11:47:52.380527+0000","last_unstale":"2022-06-27T11:47:52.383894+0000","last_undegraded":"2022-06-27T11:47:52.383894+0000","last_fullsized":"2022-06-27T11:47:52.383894+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T23:18:07.086598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.374783+0000","last_change":"2022-06-27T11:47:52.371267+0000","last_active":"2022-06-27T11:47:52.374783+0000","last_peered":"2022-06-27T11:47:52.374783+0000","last_clean":"2022-06-27T11:47:52.374783+0000","last_became_active":"2022-06-27T11:47:52.370921+0000","last_became_peered":"2022-06-27T11:47:52.370921+0000","last_unstale":"2022-06-27T11:47:52.374783+0000","last_undegraded":"2022-06-27T11:47:52.374783+0000","last_fullsized":"2022-06-27T11:47:52.374783+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T19:22:39.648715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.378994+0000","last_change":"2022-06-27T11:47:52.375381+0000","last_active":"2022-06-27T11:47:52.378994+0000","last_peered":"2022-06-27T11:47:52.378994+0000","last_clean":"2022-06-27T11:47:52.378994+0000","last_became_active":"2022-06-27T11:47:52.375089+0000","last_became_peered":"2022-06-27T11:47:52.375089+0000","last_unstale":"2022-06-27T11:47:52.378994+0000","last_undegraded":"2022-06-27T11:47:52.378994+0000","last_fullsized":"2022-06-27T11:47:52.378994+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:43:14.830424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384993+0000","last_change":"2022-06-27T11:47:52.381690+0000","last_active":"2022-06-27T11:47:52.384993+0000","last_peered":"2022-06-27T11:47:52.384993+0000","last_clean":"2022-06-27T11:47:52.384993+0000","last_became_active":"2022-06-27T11:47:52.381472+0000","last_became_peered":"2022-06-27T11:47:52.381472+0000","last_unstale":"2022-06-27T11:47:52.384993+0000","last_undegraded":"2022-06-27T11:47:52.384993+0000","last_fullsized":"2022-06-27T11:47:52.384993+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:14:56.919536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.378838+0000","last_change":"2022-06-27T11:47:52.376063+0000","last_active":"2022-06-27T11:47:52.378838+0000","last_peered":"2022-06-27T11:47:52.378838+0000","last_clean":"2022-06-27T11:47:52.378838+0000","last_became_active":"2022-06-27T11:47:52.375778+0000","last_became_peered":"2022-06-27T11:47:52.375778+0000","last_unstale":"2022-06-27T11:47:52.378838+0000","last_undegraded":"2022-06-27T11:47:52.378838+0000","last_fullsized":"2022-06-27T11:47:52.378838+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T23:16:52.221967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.383160+0000","last_change":"2022-06-27T11:47:52.377297+0000","last_active":"2022-06-27T11:47:52.383160+0000","last_peered":"2022-06-27T11:47:52.383160+0000","last_clean":"2022-06-27T11:47:52.383160+0000","last_became_active":"2022-06-27T11:47:52.376855+0000","last_became_peered":"2022-06-27T11:47:52.376855+0000","last_unstale":"2022-06-27T11:47:52.383160+0000","last_undegraded":"2022-06-27T11:47:52.383160+0000","last_fullsized":"2022-06-27T11:47:52.383160+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:33:39.990522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.387168+0000","last_change":"2022-06-27T11:47:52.381821+0000","last_active":"2022-06-27T11:47:52.387168+0000","last_peered":"2022-06-27T11:47:52.387168+0000","last_clean":"2022-06-27T11:47:52.387168+0000","last_became_active":"2022-06-27T11:47:52.381548+0000","last_became_peered":"2022-06-27T11:47:52.381548+0000","last_unstale":"2022-06-27T11:47:52.387168+0000","last_undegraded":"2022-06-27T11:47:52.387168+0000","last_fullsized":"2022-06-27T11:47:52.387168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T23:14:53.320016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384753+0000","last_change":"2022-06-27T11:47:52.381811+0000","last_active":"2022-06-27T11:47:52.384753+0000","last_peered":"2022-06-27T11:47:52.384753+0000","last_clean":"2022-06-27T11:47:52.384753+0000","last_became_active":"2022-06-27T11:47:52.381554+0000","last_became_peered":"2022-06-27T11:47:52.381554+0000","last_unstale":"2022-06-27T11:47:52.384753+0000","last_undegraded":"2022-06-27T11:47:52.384753+0000","last_fullsized":"2022-06-27T11:47:52.384753+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T14:19:38.519380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.387218+0000","last_change":"2022-06-27T11:47:52.382952+0000","last_active":"2022-06-27T11:47:52.387218+0000","last_peered":"2022-06-27T11:47:52.387218+0000","last_clean":"2022-06-27T11:47:52.387218+0000","last_became_active":"2022-06-27T11:47:52.382691+0000","last_became_peered":"2022-06-27T11:47:52.382691+0000","last_unstale":"2022-06-27T11:47:52.387218+0000","last_undegraded":"2022-06-27T11:47:52.387218+0000","last_fullsized":"2022-06-27T11:47:52.387218+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:28:54.743970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.383682+0000","last_change":"2022-06-27T11:47:52.379083+0000","last_active":"2022-06-27T11:47:52.383682+0000","last_peered":"2022-06-27T11:47:52.383682+0000","last_clean":"2022-06-27T11:47:52.383682+0000","last_became_active":"2022-06-27T11:47:52.378724+0000","last_became_peered":"2022-06-27T11:47:52.378724+0000","last_unstale":"2022-06-27T11:47:52.383682+0000","last_undegraded":"2022-06-27T11:47:52.383682+0000","last_fullsized":"2022-06-27T11:47:52.383682+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T21:25:07.786935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.383275+0000","last_change":"2022-06-27T11:47:52.377947+0000","last_active":"2022-06-27T11:47:52.383275+0000","last_peered":"2022-06-27T11:47:52.383275+0000","last_clean":"2022-06-27T11:47:52.383275+0000","last_became_active":"2022-06-27T11:47:52.377652+0000","last_became_peered":"2022-06-27T11:47:52.377652+0000","last_unstale":"2022-06-27T11:47:52.383275+0000","last_undegraded":"2022-06-27T11:47:52.383275+0000","last_fullsized":"2022-06-27T11:47:52.383275+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T20:48:28.642052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384444+0000","last_change":"2022-06-27T11:47:52.381920+0000","last_active":"2022-06-27T11:47:52.384444+0000","last_peered":"2022-06-27T11:47:52.384444+0000","last_clean":"2022-06-27T11:47:52.384444+0000","last_became_active":"2022-06-27T11:47:52.381477+0000","last_became_peered":"2022-06-27T11:47:52.381477+0000","last_unstale":"2022-06-27T11:47:52.384444+0000","last_undegraded":"2022-06-27T11:47:52.384444+0000","last_fullsized":"2022-06-27T11:47:52.384444+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T20:42:25.529336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.386345+0000","last_change":"2022-06-27T11:47:52.385231+0000","last_active":"2022-06-27T11:47:52.386345+0000","last_peered":"2022-06-27T11:47:52.386345+0000","last_clean":"2022-06-27T11:47:52.386345+0000","last_became_active":"2022-06-27T11:47:52.384972+0000","last_became_peered":"2022-06-27T11:47:52.384972+0000","last_unstale":"2022-06-27T11:47:52.386345+0000","last_undegraded":"2022-06-27T11:47:52.386345+0000","last_fullsized":"2022-06-27T11:47:52.386345+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T17:37:26.084788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.385330+0000","last_change":"2022-06-27T11:47:52.381358+0000","last_active":"2022-06-27T11:47:52.385330+0000","last_peered":"2022-06-27T11:47:52.385330+0000","last_clean":"2022-06-27T11:47:52.385330+0000","last_became_active":"2022-06-27T11:47:52.380997+0000","last_became_peered":"2022-06-27T11:47:52.380997+0000","last_unstale":"2022-06-27T11:47:52.385330+0000","last_undegraded":"2022-06-27T11:47:52.385330+0000","last_fullsized":"2022-06-27T11:47:52.385330+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T21:44:16.061234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384432+0000","last_change":"2022-06-27T11:47:52.381899+0000","last_active":"2022-06-27T11:47:52.384432+0000","last_peered":"2022-06-27T11:47:52.384432+0000","last_clean":"2022-06-27T11:47:52.384432+0000","last_became_active":"2022-06-27T11:47:52.381645+0000","last_became_peered":"2022-06-27T11:47:52.381645+0000","last_unstale":"2022-06-27T11:47:52.384432+0000","last_undegraded":"2022-06-27T11:47:52.384432+0000","last_fullsized":"2022-06-27T11:47:52.384432+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:05:59.742023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.385053+0000","last_change":"2022-06-27T11:47:52.378736+0000","last_active":"2022-06-27T11:47:52.385053+0000","last_peered":"2022-06-27T11:47:52.385053+0000","last_clean":"2022-06-27T11:47:52.385053+0000","last_became_active":"2022-06-27T11:47:52.378475+0000","last_became_peered":"2022-06-27T11:47:52.378475+0000","last_unstale":"2022-06-27T11:47:52.385053+0000","last_undegraded":"2022-06-27T11:47:52.385053+0000","last_fullsized":"2022-06-27T11:47:52.385053+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T12:42:12.528716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384922+0000","last_change":"2022-06-27T11:47:52.378044+0000","last_active":"2022-06-27T11:47:52.384922+0000","last_peered":"2022-06-27T11:47:52.384922+0000","last_clean":"2022-06-27T11:47:52.384922+0000","last_became_active":"2022-06-27T11:47:52.377791+0000","last_became_peered":"2022-06-27T11:47:52.377791+0000","last_unstale":"2022-06-27T11:47:52.384922+0000","last_undegraded":"2022-06-27T11:47:52.384922+0000","last_fullsized":"2022-06-27T11:47:52.384922+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:21:45.615162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384115+0000","last_change":"2022-06-27T11:47:52.382267+0000","last_active":"2022-06-27T11:47:52.384115+0000","last_peered":"2022-06-27T11:47:52.384115+0000","last_clean":"2022-06-27T11:47:52.384115+0000","last_became_active":"2022-06-27T11:47:52.381985+0000","last_became_peered":"2022-06-27T11:47:52.381985+0000","last_unstale":"2022-06-27T11:47:52.384115+0000","last_undegraded":"2022-06-27T11:47:52.384115+0000","last_fullsized":"2022-06-27T11:47:52.384115+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:34:09.509759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:53.909338+0000","last_change":"2022-06-27T11:47:51.982183+0000","last_active":"2022-06-27T11:47:53.909338+0000","last_peered":"2022-06-27T11:47:53.909338+0000","last_clean":"2022-06-27T11:47:53.909338+0000","last_became_active":"2022-06-27T11:47:51.981339+0000","last_became_peered":"2022-06-27T11:47:51.981339+0000","last_unstale":"2022-06-27T11:47:53.909338+0000","last_undegraded":"2022-06-27T11:47:53.909338+0000","last_fullsized":"2022-06-27T11:47:53.909338+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:08:09.223947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384452+0000","last_change":"2022-06-27T11:47:52.377197+0000","last_active":"2022-06-27T11:47:52.384452+0000","last_peered":"2022-06-27T11:47:52.384452+0000","last_clean":"2022-06-27T11:47:52.384452+0000","last_became_active":"2022-06-27T11:47:52.376925+0000","last_became_peered":"2022-06-27T11:47:52.376925+0000","last_unstale":"2022-06-27T11:47:52.384452+0000","last_undegraded":"2022-06-27T11:47:52.384452+0000","last_fullsized":"2022-06-27T11:47:52.384452+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:39:27.103978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.364221+0000","last_change":"2022-06-27T11:47:51.981074+0000","last_active":"2022-06-27T11:47:52.364221+0000","last_peered":"2022-06-27T11:47:52.364221+0000","last_clean":"2022-06-27T11:47:52.364221+0000","last_became_active":"2022-06-27T11:47:51.980636+0000","last_became_peered":"2022-06-27T11:47:51.980636+0000","last_unstale":"2022-06-27T11:47:52.364221+0000","last_undegraded":"2022-06-27T11:47:52.364221+0000","last_fullsized":"2022-06-27T11:47:52.364221+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T23:32:52.850801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384380+0000","last_change":"2022-06-27T11:47:52.378415+0000","last_active":"2022-06-27T11:47:52.384380+0000","last_peered":"2022-06-27T11:47:52.384380+0000","last_clean":"2022-06-27T11:47:52.384380+0000","last_became_active":"2022-06-27T11:47:52.378123+0000","last_became_peered":"2022-06-27T11:47:52.378123+0000","last_unstale":"2022-06-27T11:47:52.384380+0000","last_undegraded":"2022-06-27T11:47:52.384380+0000","last_fullsized":"2022-06-27T11:47:52.384380+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T11:54:37.127828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.369137+0000","last_change":"2022-06-27T11:47:51.975684+0000","last_active":"2022-06-27T11:47:52.369137+0000","last_peered":"2022-06-27T11:47:52.369137+0000","last_clean":"2022-06-27T11:47:52.369137+0000","last_became_active":"2022-06-27T11:47:51.974902+0000","last_became_peered":"2022-06-27T11:47:51.974902+0000","last_unstale":"2022-06-27T11:47:52.369137+0000","last_undegraded":"2022-06-27T11:47:52.369137+0000","last_fullsized":"2022-06-27T11:47:52.369137+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:51:45.780798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.378008+0000","last_change":"2022-06-27T11:47:52.376089+0000","last_active":"2022-06-27T11:47:52.378008+0000","last_peered":"2022-06-27T11:47:52.378008+0000","last_clean":"2022-06-27T11:47:52.378008+0000","last_became_active":"2022-06-27T11:47:52.375780+0000","last_became_peered":"2022-06-27T11:47:52.375780+0000","last_unstale":"2022-06-27T11:47:52.378008+0000","last_undegraded":"2022-06-27T11:47:52.378008+0000","last_fullsized":"2022-06-27T11:47:52.378008+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T14:17:53.942880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.371313+0000","last_change":"2022-06-27T11:47:51.974071+0000","last_active":"2022-06-27T11:47:52.371313+0000","last_peered":"2022-06-27T11:47:52.371313+0000","last_clean":"2022-06-27T11:47:52.371313+0000","last_became_active":"2022-06-27T11:47:51.973682+0000","last_became_peered":"2022-06-27T11:47:51.973682+0000","last_unstale":"2022-06-27T11:47:52.371313+0000","last_undegraded":"2022-06-27T11:47:52.371313+0000","last_fullsized":"2022-06-27T11:47:52.371313+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T17:40:03.705083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384841+0000","last_change":"2022-06-27T11:47:52.377716+0000","last_active":"2022-06-27T11:47:52.384841+0000","last_peered":"2022-06-27T11:47:52.384841+0000","last_clean":"2022-06-27T11:47:52.384841+0000","last_became_active":"2022-06-27T11:47:52.377464+0000","last_became_peered":"2022-06-27T11:47:52.377464+0000","last_unstale":"2022-06-27T11:47:52.384841+0000","last_undegraded":"2022-06-27T11:47:52.384841+0000","last_fullsized":"2022-06-27T11:47:52.384841+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:39:36.234204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.368885+0000","last_change":"2022-06-27T11:47:51.975895+0000","last_active":"2022-06-27T11:47:52.368885+0000","last_peered":"2022-06-27T11:47:52.368885+0000","last_clean":"2022-06-27T11:47:52.368885+0000","last_became_active":"2022-06-27T11:47:51.975601+0000","last_became_peered":"2022-06-27T11:47:51.975601+0000","last_unstale":"2022-06-27T11:47:52.368885+0000","last_undegraded":"2022-06-27T11:47:52.368885+0000","last_fullsized":"2022-06-27T11:47:52.368885+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:43:55.833399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.385349+0000","last_change":"2022-06-27T11:47:52.383063+0000","last_active":"2022-06-27T11:47:52.385349+0000","last_peered":"2022-06-27T11:47:52.385349+0000","last_clean":"2022-06-27T11:47:52.385349+0000","last_became_active":"2022-06-27T11:47:52.382390+0000","last_became_peered":"2022-06-27T11:47:52.382390+0000","last_unstale":"2022-06-27T11:47:52.385349+0000","last_undegraded":"2022-06-27T11:47:52.385349+0000","last_fullsized":"2022-06-27T11:47:52.385349+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:40:10.929542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"21'3","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:57.367279+0000","last_change":"2022-06-27T11:47:51.980627+0000","last_active":"2022-06-27T11:47:57.367279+0000","last_peered":"2022-06-27T11:47:57.367279+0000","last_clean":"2022-06-27T11:47:57.367279+0000","last_became_active":"2022-06-27T11:47:51.980265+0000","last_became_peered":"2022-06-27T11:47:51.980265+0000","last_unstale":"2022-06-27T11:47:57.367279+0000","last_undegraded":"2022-06-27T11:47:57.367279+0000","last_fullsized":"2022-06-27T11:47:57.367279+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T14:18:32.599085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384584+0000","last_change":"2022-06-27T11:47:52.377475+0000","last_active":"2022-06-27T11:47:52.384584+0000","last_peered":"2022-06-27T11:47:52.384584+0000","last_clean":"2022-06-27T11:47:52.384584+0000","last_became_active":"2022-06-27T11:47:52.377220+0000","last_became_peered":"2022-06-27T11:47:52.377220+0000","last_unstale":"2022-06-27T11:47:52.384584+0000","last_undegraded":"2022-06-27T11:47:52.384584+0000","last_fullsized":"2022-06-27T11:47:52.384584+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T23:27:26.525688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"21'7","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:57.370253+0000","last_change":"2022-06-27T11:47:51.978195+0000","last_active":"2022-06-27T11:47:57.370253+0000","last_peered":"2022-06-27T11:47:57.370253+0000","last_clean":"2022-06-27T11:47:57.370253+0000","last_became_active":"2022-06-27T11:47:51.974673+0000","last_became_peered":"2022-06-27T11:47:51.974673+0000","last_unstale":"2022-06-27T11:47:57.370253+0000","last_undegraded":"2022-06-27T11:47:57.370253+0000","last_fullsized":"2022-06-27T11:47:57.370253+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T18:09:43.518644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1808,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.382204+0000","last_change":"2022-06-27T11:47:52.378740+0000","last_active":"2022-06-27T11:47:52.382204+0000","last_peered":"2022-06-27T11:47:52.382204+0000","last_clean":"2022-06-27T11:47:52.382204+0000","last_became_active":"2022-06-27T11:47:52.378451+0000","last_became_peered":"2022-06-27T11:47:52.378451+0000","last_unstale":"2022-06-27T11:47:52.382204+0000","last_undegraded":"2022-06-27T11:47:52.382204+0000","last_fullsized":"2022-06-27T11:47:52.382204+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:04:47.829409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.13","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:57.368186+0000","last_change":"2022-06-27T11:47:51.983039+0000","last_active":"2022-06-27T11:47:57.368186+0000","last_peered":"2022-06-27T11:47:57.368186+0000","last_clean":"2022-06-27T11:47:57.368186+0000","last_became_active":"2022-06-27T11:47:51.982683+0000","last_became_peered":"2022-06-27T11:47:51.982683+0000","last_unstale":"2022-06-27T11:47:57.368186+0000","last_undegraded":"2022-06-27T11:47:57.368186+0000","last_fullsized":"2022-06-27T11:47:57.368186+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T17:10:16.876753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384742+0000","last_change":"2022-06-27T11:47:52.374884+0000","last_active":"2022-06-27T11:47:52.384742+0000","last_peered":"2022-06-27T11:47:52.384742+0000","last_clean":"2022-06-27T11:47:52.384742+0000","last_became_active":"2022-06-27T11:47:52.374632+0000","last_became_peered":"2022-06-27T11:47:52.374632+0000","last_unstale":"2022-06-27T11:47:52.384742+0000","last_undegraded":"2022-06-27T11:47:52.384742+0000","last_fullsized":"2022-06-27T11:47:52.384742+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T21:33:54.724089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.374167+0000","last_change":"2022-06-27T11:47:51.976772+0000","last_active":"2022-06-27T11:47:52.374167+0000","last_peered":"2022-06-27T11:47:52.374167+0000","last_clean":"2022-06-27T11:47:52.374167+0000","last_became_active":"2022-06-27T11:47:51.976475+0000","last_became_peered":"2022-06-27T11:47:51.976475+0000","last_unstale":"2022-06-27T11:47:52.374167+0000","last_undegraded":"2022-06-27T11:47:52.374167+0000","last_fullsized":"2022-06-27T11:47:52.374167+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T23:39:10.919159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.377943+0000","last_change":"2022-06-27T11:47:52.373748+0000","last_active":"2022-06-27T11:47:52.377943+0000","last_peered":"2022-06-27T11:47:52.377943+0000","last_clean":"2022-06-27T11:47:52.377943+0000","last_became_active":"2022-06-27T11:47:52.373459+0000","last_became_peered":"2022-06-27T11:47:52.373459+0000","last_unstale":"2022-06-27T11:47:52.377943+0000","last_undegraded":"2022-06-27T11:47:52.377943+0000","last_fullsized":"2022-06-27T11:47:52.377943+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T18:27:22.254416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.367083+0000","last_change":"2022-06-27T11:47:51.982926+0000","last_active":"2022-06-27T11:47:52.367083+0000","last_peered":"2022-06-27T11:47:52.367083+0000","last_clean":"2022-06-27T11:47:52.367083+0000","last_became_active":"2022-06-27T11:47:51.982638+0000","last_became_peered":"2022-06-27T11:47:51.982638+0000","last_unstale":"2022-06-27T11:47:52.367083+0000","last_undegraded":"2022-06-27T11:47:52.367083+0000","last_fullsized":"2022-06-27T11:47:52.367083+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:34:04.834822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384791+0000","last_change":"2022-06-27T11:47:52.381623+0000","last_active":"2022-06-27T11:47:52.384791+0000","last_peered":"2022-06-27T11:47:52.384791+0000","last_clean":"2022-06-27T11:47:52.384791+0000","last_became_active":"2022-06-27T11:47:52.381258+0000","last_became_peered":"2022-06-27T11:47:52.381258+0000","last_unstale":"2022-06-27T11:47:52.384791+0000","last_undegraded":"2022-06-27T11:47:52.384791+0000","last_fullsized":"2022-06-27T11:47:52.384791+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:39:07.735224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"21'8","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:57.369384+0000","last_change":"2022-06-27T11:47:51.973125+0000","last_active":"2022-06-27T11:47:57.369384+0000","last_peered":"2022-06-27T11:47:57.369384+0000","last_clean":"2022-06-27T11:47:57.369384+0000","last_became_active":"2022-06-27T11:47:51.972806+0000","last_became_peered":"2022-06-27T11:47:51.972806+0000","last_unstale":"2022-06-27T11:47:57.369384+0000","last_undegraded":"2022-06-27T11:47:57.369384+0000","last_fullsized":"2022-06-27T11:47:57.369384+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:49:23.881991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384422+0000","last_change":"2022-06-27T11:47:52.376101+0000","last_active":"2022-06-27T11:47:52.384422+0000","last_peered":"2022-06-27T11:47:52.384422+0000","last_clean":"2022-06-27T11:47:52.384422+0000","last_became_active":"2022-06-27T11:47:52.375774+0000","last_became_peered":"2022-06-27T11:47:52.375774+0000","last_unstale":"2022-06-27T11:47:52.384422+0000","last_undegraded":"2022-06-27T11:47:52.384422+0000","last_fullsized":"2022-06-27T11:47:52.384422+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T18:21:13.049096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.f","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:53.492124+0000","last_change":"2022-06-27T11:47:51.981981+0000","last_active":"2022-06-27T11:47:53.492124+0000","last_peered":"2022-06-27T11:47:53.492124+0000","last_clean":"2022-06-27T11:47:53.492124+0000","last_became_active":"2022-06-27T11:47:51.981749+0000","last_became_peered":"2022-06-27T11:47:51.981749+0000","last_unstale":"2022-06-27T11:47:53.492124+0000","last_undegraded":"2022-06-27T11:47:53.492124+0000","last_fullsized":"2022-06-27T11:47:53.492124+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:28:52.457719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.378190+0000","last_change":"2022-06-27T11:47:52.374223+0000","last_active":"2022-06-27T11:47:52.378190+0000","last_peered":"2022-06-27T11:47:52.378190+0000","last_clean":"2022-06-27T11:47:52.378190+0000","last_became_active":"2022-06-27T11:47:52.373932+0000","last_became_peered":"2022-06-27T11:47:52.373932+0000","last_unstale":"2022-06-27T11:47:52.378190+0000","last_undegraded":"2022-06-27T11:47:52.378190+0000","last_fullsized":"2022-06-27T11:47:52.378190+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:48:33.254458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:57.368190+0000","last_change":"2022-06-27T11:47:51.982895+0000","last_active":"2022-06-27T11:47:57.368190+0000","last_peered":"2022-06-27T11:47:57.368190+0000","last_clean":"2022-06-27T11:47:57.368190+0000","last_became_active":"2022-06-27T11:47:51.982619+0000","last_became_peered":"2022-06-27T11:47:51.982619+0000","last_unstale":"2022-06-27T11:47:57.368190+0000","last_undegraded":"2022-06-27T11:47:57.368190+0000","last_fullsized":"2022-06-27T11:47:57.368190+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:22:01.241112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.377169+0000","last_change":"2022-06-27T11:47:52.375225+0000","last_active":"2022-06-27T11:47:52.377169+0000","last_peered":"2022-06-27T11:47:52.377169+0000","last_clean":"2022-06-27T11:47:52.377169+0000","last_became_active":"2022-06-27T11:47:52.374925+0000","last_became_peered":"2022-06-27T11:47:52.374925+0000","last_unstale":"2022-06-27T11:47:52.377169+0000","last_undegraded":"2022-06-27T11:47:52.377169+0000","last_fullsized":"2022-06-27T11:47:52.377169+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:54:31.945902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:53.492085+0000","last_change":"2022-06-27T11:47:51.975202+0000","last_active":"2022-06-27T11:47:53.492085+0000","last_peered":"2022-06-27T11:47:53.492085+0000","last_clean":"2022-06-27T11:47:53.492085+0000","last_became_active":"2022-06-27T11:47:51.974968+0000","last_became_peered":"2022-06-27T11:47:51.974968+0000","last_unstale":"2022-06-27T11:47:53.492085+0000","last_undegraded":"2022-06-27T11:47:53.492085+0000","last_fullsized":"2022-06-27T11:47:53.492085+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T21:16:30.362248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.378431+0000","last_change":"2022-06-27T11:47:52.377794+0000","last_active":"2022-06-27T11:47:52.378431+0000","last_peered":"2022-06-27T11:47:52.378431+0000","last_clean":"2022-06-27T11:47:52.378431+0000","last_became_active":"2022-06-27T11:47:52.377387+0000","last_became_peered":"2022-06-27T11:47:52.377387+0000","last_unstale":"2022-06-27T11:47:52.378431+0000","last_undegraded":"2022-06-27T11:47:52.378431+0000","last_fullsized":"2022-06-27T11:47:52.378431+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T17:00:00.924967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.368792+0000","last_change":"2022-06-27T11:47:51.976575+0000","last_active":"2022-06-27T11:47:52.368792+0000","last_peered":"2022-06-27T11:47:52.368792+0000","last_clean":"2022-06-27T11:47:52.368792+0000","last_became_active":"2022-06-27T11:47:51.976300+0000","last_became_peered":"2022-06-27T11:47:51.976300+0000","last_unstale":"2022-06-27T11:47:52.368792+0000","last_undegraded":"2022-06-27T11:47:52.368792+0000","last_fullsized":"2022-06-27T11:47:52.368792+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:08:46.067375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.387174+0000","last_change":"2022-06-27T11:47:52.386298+0000","last_active":"2022-06-27T11:47:52.387174+0000","last_peered":"2022-06-27T11:47:52.387174+0000","last_clean":"2022-06-27T11:47:52.387174+0000","last_became_active":"2022-06-27T11:47:52.385918+0000","last_became_peered":"2022-06-27T11:47:52.385918+0000","last_unstale":"2022-06-27T11:47:52.387174+0000","last_undegraded":"2022-06-27T11:47:52.387174+0000","last_fullsized":"2022-06-27T11:47:52.387174+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:21:51.780895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:57.364776+0000","last_change":"2022-06-27T11:47:51.973817+0000","last_active":"2022-06-27T11:47:57.364776+0000","last_peered":"2022-06-27T11:47:57.364776+0000","last_clean":"2022-06-27T11:47:57.364776+0000","last_became_active":"2022-06-27T11:47:51.973515+0000","last_became_peered":"2022-06-27T11:47:51.973515+0000","last_unstale":"2022-06-27T11:47:57.364776+0000","last_undegraded":"2022-06-27T11:47:57.364776+0000","last_fullsized":"2022-06-27T11:47:57.364776+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T21:01:45.170590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.374582+0000","last_change":"2022-06-27T11:47:52.371141+0000","last_active":"2022-06-27T11:47:52.374582+0000","last_peered":"2022-06-27T11:47:52.374582+0000","last_clean":"2022-06-27T11:47:52.374582+0000","last_became_active":"2022-06-27T11:47:52.370796+0000","last_became_peered":"2022-06-27T11:47:52.370796+0000","last_unstale":"2022-06-27T11:47:52.374582+0000","last_undegraded":"2022-06-27T11:47:52.374582+0000","last_fullsized":"2022-06-27T11:47:52.374582+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T23:16:39.266615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.369465+0000","last_change":"2022-06-27T11:47:51.983071+0000","last_active":"2022-06-27T11:47:52.369465+0000","last_peered":"2022-06-27T11:47:52.369465+0000","last_clean":"2022-06-27T11:47:52.369465+0000","last_became_active":"2022-06-27T11:47:51.982756+0000","last_became_peered":"2022-06-27T11:47:51.982756+0000","last_unstale":"2022-06-27T11:47:52.369465+0000","last_undegraded":"2022-06-27T11:47:52.369465+0000","last_fullsized":"2022-06-27T11:47:52.369465+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T17:14:00.737599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.387222+0000","last_change":"2022-06-27T11:47:52.382943+0000","last_active":"2022-06-27T11:47:52.387222+0000","last_peered":"2022-06-27T11:47:52.387222+0000","last_clean":"2022-06-27T11:47:52.387222+0000","last_became_active":"2022-06-27T11:47:52.382625+0000","last_became_peered":"2022-06-27T11:47:52.382625+0000","last_unstale":"2022-06-27T11:47:52.387222+0000","last_undegraded":"2022-06-27T11:47:52.387222+0000","last_fullsized":"2022-06-27T11:47:52.387222+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:01:18.570554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.362681+0000","last_change":"2022-06-27T11:47:51.978856+0000","last_active":"2022-06-27T11:47:52.362681+0000","last_peered":"2022-06-27T11:47:52.362681+0000","last_clean":"2022-06-27T11:47:52.362681+0000","last_became_active":"2022-06-27T11:47:51.978541+0000","last_became_peered":"2022-06-27T11:47:51.978541+0000","last_unstale":"2022-06-27T11:47:52.362681+0000","last_undegraded":"2022-06-27T11:47:52.362681+0000","last_fullsized":"2022-06-27T11:47:52.362681+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:36:24.190996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.378203+0000","last_change":"2022-06-27T11:47:52.376925+0000","last_active":"2022-06-27T11:47:52.378203+0000","last_peered":"2022-06-27T11:47:52.378203+0000","last_clean":"2022-06-27T11:47:52.378203+0000","last_became_active":"2022-06-27T11:47:52.376646+0000","last_became_peered":"2022-06-27T11:47:52.376646+0000","last_unstale":"2022-06-27T11:47:52.378203+0000","last_undegraded":"2022-06-27T11:47:52.378203+0000","last_fullsized":"2022-06-27T11:47:52.378203+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:17:51.913392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.369021+0000","last_change":"2022-06-27T11:47:51.979355+0000","last_active":"2022-06-27T11:47:52.369021+0000","last_peered":"2022-06-27T11:47:52.369021+0000","last_clean":"2022-06-27T11:47:52.369021+0000","last_became_active":"2022-06-27T11:47:51.977591+0000","last_became_peered":"2022-06-27T11:47:51.977591+0000","last_unstale":"2022-06-27T11:47:52.369021+0000","last_undegraded":"2022-06-27T11:47:52.369021+0000","last_fullsized":"2022-06-27T11:47:52.369021+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T11:50:43.086014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.385028+0000","last_change":"2022-06-27T11:47:52.375271+0000","last_active":"2022-06-27T11:47:52.385028+0000","last_peered":"2022-06-27T11:47:52.385028+0000","last_clean":"2022-06-27T11:47:52.385028+0000","last_became_active":"2022-06-27T11:47:52.374985+0000","last_became_peered":"2022-06-27T11:47:52.374985+0000","last_unstale":"2022-06-27T11:47:52.385028+0000","last_undegraded":"2022-06-27T11:47:52.385028+0000","last_fullsized":"2022-06-27T11:47:52.385028+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T17:58:40.711409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:57.367534+0000","last_change":"2022-06-27T11:47:51.979723+0000","last_active":"2022-06-27T11:47:57.367534+0000","last_peered":"2022-06-27T11:47:57.367534+0000","last_clean":"2022-06-27T11:47:57.367534+0000","last_became_active":"2022-06-27T11:47:51.977669+0000","last_became_peered":"2022-06-27T11:47:51.977669+0000","last_unstale":"2022-06-27T11:47:57.367534+0000","last_undegraded":"2022-06-27T11:47:57.367534+0000","last_fullsized":"2022-06-27T11:47:57.367534+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T14:35:14.770254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.363444+0000","last_change":"2022-06-27T11:47:48.388880+0000","last_active":"2022-06-27T11:47:52.363444+0000","last_peered":"2022-06-27T11:47:52.363444+0000","last_clean":"2022-06-27T11:47:52.363444+0000","last_became_active":"2022-06-27T11:47:48.388639+0000","last_became_peered":"2022-06-27T11:47:48.388639+0000","last_unstale":"2022-06-27T11:47:52.363444+0000","last_undegraded":"2022-06-27T11:47:52.363444+0000","last_fullsized":"2022-06-27T11:47:52.363444+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_clean_scrub_stamp":"2022-06-27T11:47:46.780512+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:31:10.549485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.380026+0000","last_change":"2022-06-27T11:47:52.373143+0000","last_active":"2022-06-27T11:47:52.380026+0000","last_peered":"2022-06-27T11:47:52.380026+0000","last_clean":"2022-06-27T11:47:52.380026+0000","last_became_active":"2022-06-27T11:47:52.372850+0000","last_became_peered":"2022-06-27T11:47:52.372850+0000","last_unstale":"2022-06-27T11:47:52.380026+0000","last_undegraded":"2022-06-27T11:47:52.380026+0000","last_fullsized":"2022-06-27T11:47:52.380026+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:48:44.251459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.369716+0000","last_change":"2022-06-27T11:47:47.793307+0000","last_active":"2022-06-27T11:47:52.369716+0000","last_peered":"2022-06-27T11:47:52.369716+0000","last_clean":"2022-06-27T11:47:52.369716+0000","last_became_active":"2022-06-27T11:47:47.792958+0000","last_became_peered":"2022-06-27T11:47:47.792958+0000","last_unstale":"2022-06-27T11:47:52.369716+0000","last_undegraded":"2022-06-27T11:47:52.369716+0000","last_fullsized":"2022-06-27T11:47:52.369716+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_clean_scrub_stamp":"2022-06-27T11:47:46.780512+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:08:35.128804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:53.910382+0000","last_change":"2022-06-27T11:47:51.976872+0000","last_active":"2022-06-27T11:47:53.910382+0000","last_peered":"2022-06-27T11:47:53.910382+0000","last_clean":"2022-06-27T11:47:53.910382+0000","last_became_active":"2022-06-27T11:47:51.976546+0000","last_became_peered":"2022-06-27T11:47:51.976546+0000","last_unstale":"2022-06-27T11:47:53.910382+0000","last_undegraded":"2022-06-27T11:47:53.910382+0000","last_fullsized":"2022-06-27T11:47:53.910382+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T14:44:43.349688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.375090+0000","last_change":"2022-06-27T11:47:52.372794+0000","last_active":"2022-06-27T11:47:52.375090+0000","last_peered":"2022-06-27T11:47:52.375090+0000","last_clean":"2022-06-27T11:47:52.375090+0000","last_became_active":"2022-06-27T11:47:52.372486+0000","last_became_peered":"2022-06-27T11:47:52.372486+0000","last_unstale":"2022-06-27T11:47:52.375090+0000","last_undegraded":"2022-06-27T11:47:52.375090+0000","last_fullsized":"2022-06-27T11:47:52.375090+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:23:56.307995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.363787+0000","last_change":"2022-06-27T11:47:47.792631+0000","last_active":"2022-06-27T11:47:52.363787+0000","last_peered":"2022-06-27T11:47:52.363787+0000","last_clean":"2022-06-27T11:47:52.363787+0000","last_became_active":"2022-06-27T11:47:47.792282+0000","last_became_peered":"2022-06-27T11:47:47.792282+0000","last_unstale":"2022-06-27T11:47:52.363787+0000","last_undegraded":"2022-06-27T11:47:52.363787+0000","last_fullsized":"2022-06-27T11:47:52.363787+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_clean_scrub_stamp":"2022-06-27T11:47:46.780512+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:41:25.449379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"17'117","reported_seq":163,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.368104+0000","last_change":"2022-06-27T11:47:47.791697+0000","last_active":"2022-06-27T11:47:52.368104+0000","last_peered":"2022-06-27T11:47:52.368104+0000","last_clean":"2022-06-27T11:47:52.368104+0000","last_became_active":"2022-06-27T11:47:47.790645+0000","last_became_peered":"2022-06-27T11:47:47.790645+0000","last_unstale":"2022-06-27T11:47:52.368104+0000","last_undegraded":"2022-06-27T11:47:52.368104+0000","last_fullsized":"2022-06-27T11:47:52.368104+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_clean_scrub_stamp":"2022-06-27T11:47:46.780512+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:11:09.344666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.367962+0000","last_change":"2022-06-27T11:47:51.982728+0000","last_active":"2022-06-27T11:47:52.367962+0000","last_peered":"2022-06-27T11:47:52.367962+0000","last_clean":"2022-06-27T11:47:52.367962+0000","last_became_active":"2022-06-27T11:47:51.982445+0000","last_became_peered":"2022-06-27T11:47:51.982445+0000","last_unstale":"2022-06-27T11:47:52.367962+0000","last_undegraded":"2022-06-27T11:47:52.367962+0000","last_fullsized":"2022-06-27T11:47:52.367962+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:42:04.682687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.386922+0000","last_change":"2022-06-27T11:47:52.386015+0000","last_active":"2022-06-27T11:47:52.386922+0000","last_peered":"2022-06-27T11:47:52.386922+0000","last_clean":"2022-06-27T11:47:52.386922+0000","last_became_active":"2022-06-27T11:47:52.385731+0000","last_became_peered":"2022-06-27T11:47:52.385731+0000","last_unstale":"2022-06-27T11:47:52.386922+0000","last_undegraded":"2022-06-27T11:47:52.386922+0000","last_fullsized":"2022-06-27T11:47:52.386922+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T11:58:50.483766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.366627+0000","last_change":"2022-06-27T11:47:47.807699+0000","last_active":"2022-06-27T11:47:52.366627+0000","last_peered":"2022-06-27T11:47:52.366627+0000","last_clean":"2022-06-27T11:47:52.366627+0000","last_became_active":"2022-06-27T11:47:47.807422+0000","last_became_peered":"2022-06-27T11:47:47.807422+0000","last_unstale":"2022-06-27T11:47:52.366627+0000","last_undegraded":"2022-06-27T11:47:52.366627+0000","last_fullsized":"2022-06-27T11:47:52.366627+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_clean_scrub_stamp":"2022-06-27T11:47:46.780512+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:15:00.549295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.363922+0000","last_change":"2022-06-27T11:47:51.974468+0000","last_active":"2022-06-27T11:47:52.363922+0000","last_peered":"2022-06-27T11:47:52.363922+0000","last_clean":"2022-06-27T11:47:52.363922+0000","last_became_active":"2022-06-27T11:47:51.974160+0000","last_became_peered":"2022-06-27T11:47:51.974160+0000","last_unstale":"2022-06-27T11:47:52.363922+0000","last_undegraded":"2022-06-27T11:47:52.363922+0000","last_fullsized":"2022-06-27T11:47:52.363922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T19:14:45.694282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.380162+0000","last_change":"2022-06-27T11:47:52.373931+0000","last_active":"2022-06-27T11:47:52.380162+0000","last_peered":"2022-06-27T11:47:52.380162+0000","last_clean":"2022-06-27T11:47:52.380162+0000","last_became_active":"2022-06-27T11:47:52.373562+0000","last_became_peered":"2022-06-27T11:47:52.373562+0000","last_unstale":"2022-06-27T11:47:52.380162+0000","last_undegraded":"2022-06-27T11:47:52.380162+0000","last_fullsized":"2022-06-27T11:47:52.380162+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T14:09:22.859955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.367866+0000","last_change":"2022-06-27T11:47:47.807650+0000","last_active":"2022-06-27T11:47:52.367866+0000","last_peered":"2022-06-27T11:47:52.367866+0000","last_clean":"2022-06-27T11:47:52.367866+0000","last_became_active":"2022-06-27T11:47:47.807375+0000","last_became_peered":"2022-06-27T11:47:47.807375+0000","last_unstale":"2022-06-27T11:47:52.367866+0000","last_undegraded":"2022-06-27T11:47:52.367866+0000","last_fullsized":"2022-06-27T11:47:52.367866+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_clean_scrub_stamp":"2022-06-27T11:47:46.780512+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T14:10:48.067284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.368898+0000","last_change":"2022-06-27T11:47:51.972154+0000","last_active":"2022-06-27T11:47:52.368898+0000","last_peered":"2022-06-27T11:47:52.368898+0000","last_clean":"2022-06-27T11:47:52.368898+0000","last_became_active":"2022-06-27T11:47:51.971762+0000","last_became_peered":"2022-06-27T11:47:51.971762+0000","last_unstale":"2022-06-27T11:47:52.368898+0000","last_undegraded":"2022-06-27T11:47:52.368898+0000","last_fullsized":"2022-06-27T11:47:52.368898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T18:15:14.271005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.379879+0000","last_change":"2022-06-27T11:47:52.378294+0000","last_active":"2022-06-27T11:47:52.379879+0000","last_peered":"2022-06-27T11:47:52.379879+0000","last_clean":"2022-06-27T11:47:52.379879+0000","last_became_active":"2022-06-27T11:47:52.377704+0000","last_became_peered":"2022-06-27T11:47:52.377704+0000","last_unstale":"2022-06-27T11:47:52.379879+0000","last_undegraded":"2022-06-27T11:47:52.379879+0000","last_fullsized":"2022-06-27T11:47:52.379879+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:26:34.474081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.364180+0000","last_change":"2022-06-27T11:47:47.807600+0000","last_active":"2022-06-27T11:47:52.364180+0000","last_peered":"2022-06-27T11:47:52.364180+0000","last_clean":"2022-06-27T11:47:52.364180+0000","last_became_active":"2022-06-27T11:47:47.807284+0000","last_became_peered":"2022-06-27T11:47:47.807284+0000","last_unstale":"2022-06-27T11:47:52.364180+0000","last_undegraded":"2022-06-27T11:47:52.364180+0000","last_fullsized":"2022-06-27T11:47:52.364180+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_clean_scrub_stamp":"2022-06-27T11:47:46.780512+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T20:57:06.487027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.362687+0000","last_change":"2022-06-27T11:47:51.980734+0000","last_active":"2022-06-27T11:47:52.362687+0000","last_peered":"2022-06-27T11:47:52.362687+0000","last_clean":"2022-06-27T11:47:52.362687+0000","last_became_active":"2022-06-27T11:47:51.980476+0000","last_became_peered":"2022-06-27T11:47:51.980476+0000","last_unstale":"2022-06-27T11:47:52.362687+0000","last_undegraded":"2022-06-27T11:47:52.362687+0000","last_fullsized":"2022-06-27T11:47:52.362687+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T21:32:16.435472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.385478+0000","last_change":"2022-06-27T11:47:52.379902+0000","last_active":"2022-06-27T11:47:52.385478+0000","last_peered":"2022-06-27T11:47:52.385478+0000","last_clean":"2022-06-27T11:47:52.385478+0000","last_became_active":"2022-06-27T11:47:52.379359+0000","last_became_peered":"2022-06-27T11:47:52.379359+0000","last_unstale":"2022-06-27T11:47:52.385478+0000","last_undegraded":"2022-06-27T11:47:52.385478+0000","last_fullsized":"2022-06-27T11:47:52.385478+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:41:27.884083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.370459+0000","last_change":"2022-06-27T11:47:47.807607+0000","last_active":"2022-06-27T11:47:52.370459+0000","last_peered":"2022-06-27T11:47:52.370459+0000","last_clean":"2022-06-27T11:47:52.370459+0000","last_became_active":"2022-06-27T11:47:47.807293+0000","last_became_peered":"2022-06-27T11:47:47.807293+0000","last_unstale":"2022-06-27T11:47:52.370459+0000","last_undegraded":"2022-06-27T11:47:52.370459+0000","last_fullsized":"2022-06-27T11:47:52.370459+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_clean_scrub_stamp":"2022-06-27T11:47:46.780512+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:00:35.318372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.364133+0000","last_change":"2022-06-27T11:47:51.976055+0000","last_active":"2022-06-27T11:47:52.364133+0000","last_peered":"2022-06-27T11:47:52.364133+0000","last_clean":"2022-06-27T11:47:52.364133+0000","last_became_active":"2022-06-27T11:47:51.975303+0000","last_became_peered":"2022-06-27T11:47:51.975303+0000","last_unstale":"2022-06-27T11:47:52.364133+0000","last_undegraded":"2022-06-27T11:47:52.364133+0000","last_fullsized":"2022-06-27T11:47:52.364133+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:27:32.125525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384750+0000","last_change":"2022-06-27T11:47:52.380775+0000","last_active":"2022-06-27T11:47:52.384750+0000","last_peered":"2022-06-27T11:47:52.384750+0000","last_clean":"2022-06-27T11:47:52.384750+0000","last_became_active":"2022-06-27T11:47:52.380439+0000","last_became_peered":"2022-06-27T11:47:52.380439+0000","last_unstale":"2022-06-27T11:47:52.384750+0000","last_undegraded":"2022-06-27T11:47:52.384750+0000","last_fullsized":"2022-06-27T11:47:52.384750+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:03:44.939329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.370743+0000","last_change":"2022-06-27T11:47:47.792314+0000","last_active":"2022-06-27T11:47:52.370743+0000","last_peered":"2022-06-27T11:47:52.370743+0000","last_clean":"2022-06-27T11:47:52.370743+0000","last_became_active":"2022-06-27T11:47:47.792032+0000","last_became_peered":"2022-06-27T11:47:47.792032+0000","last_unstale":"2022-06-27T11:47:52.370743+0000","last_undegraded":"2022-06-27T11:47:52.370743+0000","last_fullsized":"2022-06-27T11:47:52.370743+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_clean_scrub_stamp":"2022-06-27T11:47:46.780512+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T14:13:03.591775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.369208+0000","last_change":"2022-06-27T11:47:51.978639+0000","last_active":"2022-06-27T11:47:52.369208+0000","last_peered":"2022-06-27T11:47:52.369208+0000","last_clean":"2022-06-27T11:47:52.369208+0000","last_became_active":"2022-06-27T11:47:51.978341+0000","last_became_peered":"2022-06-27T11:47:51.978341+0000","last_unstale":"2022-06-27T11:47:52.369208+0000","last_undegraded":"2022-06-27T11:47:52.369208+0000","last_fullsized":"2022-06-27T11:47:52.369208+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T21:34:48.070558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.381133+0000","last_change":"2022-06-27T11:47:52.373588+0000","last_active":"2022-06-27T11:47:52.381133+0000","last_peered":"2022-06-27T11:47:52.381133+0000","last_clean":"2022-06-27T11:47:52.381133+0000","last_became_active":"2022-06-27T11:47:52.373208+0000","last_became_peered":"2022-06-27T11:47:52.373208+0000","last_unstale":"2022-06-27T11:47:52.381133+0000","last_undegraded":"2022-06-27T11:47:52.381133+0000","last_fullsized":"2022-06-27T11:47:52.381133+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T20:27:28.252147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:53.910019+0000","last_change":"2022-06-27T11:47:51.982051+0000","last_active":"2022-06-27T11:47:53.910019+0000","last_peered":"2022-06-27T11:47:53.910019+0000","last_clean":"2022-06-27T11:47:53.910019+0000","last_became_active":"2022-06-27T11:47:51.981742+0000","last_became_peered":"2022-06-27T11:47:51.981742+0000","last_unstale":"2022-06-27T11:47:53.910019+0000","last_undegraded":"2022-06-27T11:47:53.910019+0000","last_fullsized":"2022-06-27T11:47:53.910019+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:39:56.937086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.380333+0000","last_change":"2022-06-27T11:47:52.378192+0000","last_active":"2022-06-27T11:47:52.380333+0000","last_peered":"2022-06-27T11:47:52.380333+0000","last_clean":"2022-06-27T11:47:52.380333+0000","last_became_active":"2022-06-27T11:47:52.377919+0000","last_became_peered":"2022-06-27T11:47:52.377919+0000","last_unstale":"2022-06-27T11:47:52.380333+0000","last_undegraded":"2022-06-27T11:47:52.380333+0000","last_fullsized":"2022-06-27T11:47:52.380333+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T18:51:35.757778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:57.368777+0000","last_change":"2022-06-27T11:47:51.981101+0000","last_active":"2022-06-27T11:47:57.368777+0000","last_peered":"2022-06-27T11:47:57.368777+0000","last_clean":"2022-06-27T11:47:57.368777+0000","last_became_active":"2022-06-27T11:47:51.980793+0000","last_became_peered":"2022-06-27T11:47:51.980793+0000","last_unstale":"2022-06-27T11:47:57.368777+0000","last_undegraded":"2022-06-27T11:47:57.368777+0000","last_fullsized":"2022-06-27T11:47:57.368777+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:28:10.206148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.372181+0000","last_change":"2022-06-27T11:47:52.370334+0000","last_active":"2022-06-27T11:47:52.372181+0000","last_peered":"2022-06-27T11:47:52.372181+0000","last_clean":"2022-06-27T11:47:52.372181+0000","last_became_active":"2022-06-27T11:47:52.370028+0000","last_became_peered":"2022-06-27T11:47:52.370028+0000","last_unstale":"2022-06-27T11:47:52.372181+0000","last_undegraded":"2022-06-27T11:47:52.372181+0000","last_fullsized":"2022-06-27T11:47:52.372181+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T17:28:12.357757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:57.369040+0000","last_change":"2022-06-27T11:47:51.981186+0000","last_active":"2022-06-27T11:47:57.369040+0000","last_peered":"2022-06-27T11:47:57.369040+0000","last_clean":"2022-06-27T11:47:57.369040+0000","last_became_active":"2022-06-27T11:47:51.980921+0000","last_became_peered":"2022-06-27T11:47:51.980921+0000","last_unstale":"2022-06-27T11:47:57.369040+0000","last_undegraded":"2022-06-27T11:47:57.369040+0000","last_fullsized":"2022-06-27T11:47:57.369040+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:04:40.732299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.387201+0000","last_change":"2022-06-27T11:47:52.382091+0000","last_active":"2022-06-27T11:47:52.387201+0000","last_peered":"2022-06-27T11:47:52.387201+0000","last_clean":"2022-06-27T11:47:52.387201+0000","last_became_active":"2022-06-27T11:47:52.381633+0000","last_became_peered":"2022-06-27T11:47:52.381633+0000","last_unstale":"2022-06-27T11:47:52.387201+0000","last_undegraded":"2022-06-27T11:47:52.387201+0000","last_fullsized":"2022-06-27T11:47:52.387201+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:34:44.030534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:53.910420+0000","last_change":"2022-06-27T11:47:51.972963+0000","last_active":"2022-06-27T11:47:53.910420+0000","last_peered":"2022-06-27T11:47:53.910420+0000","last_clean":"2022-06-27T11:47:53.910420+0000","last_became_active":"2022-06-27T11:47:51.972686+0000","last_became_peered":"2022-06-27T11:47:51.972686+0000","last_unstale":"2022-06-27T11:47:53.910420+0000","last_undegraded":"2022-06-27T11:47:53.910420+0000","last_fullsized":"2022-06-27T11:47:53.910420+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T12:11:38.008538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.385365+0000","last_change":"2022-06-27T11:47:52.383414+0000","last_active":"2022-06-27T11:47:52.385365+0000","last_peered":"2022-06-27T11:47:52.385365+0000","last_clean":"2022-06-27T11:47:52.385365+0000","last_became_active":"2022-06-27T11:47:52.383070+0000","last_became_peered":"2022-06-27T11:47:52.383070+0000","last_unstale":"2022-06-27T11:47:52.385365+0000","last_undegraded":"2022-06-27T11:47:52.385365+0000","last_fullsized":"2022-06-27T11:47:52.385365+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:11:28.410045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"21'4","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:57.370018+0000","last_change":"2022-06-27T11:47:51.971664+0000","last_active":"2022-06-27T11:47:57.370018+0000","last_peered":"2022-06-27T11:47:57.370018+0000","last_clean":"2022-06-27T11:47:57.370018+0000","last_became_active":"2022-06-27T11:47:51.971385+0000","last_became_peered":"2022-06-27T11:47:51.971385+0000","last_unstale":"2022-06-27T11:47:57.370018+0000","last_undegraded":"2022-06-27T11:47:57.370018+0000","last_fullsized":"2022-06-27T11:47:57.370018+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T18:44:38.394388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.379723+0000","last_change":"2022-06-27T11:47:52.377305+0000","last_active":"2022-06-27T11:47:52.379723+0000","last_peered":"2022-06-27T11:47:52.379723+0000","last_clean":"2022-06-27T11:47:52.379723+0000","last_became_active":"2022-06-27T11:47:52.377012+0000","last_became_peered":"2022-06-27T11:47:52.377012+0000","last_unstale":"2022-06-27T11:47:52.379723+0000","last_undegraded":"2022-06-27T11:47:52.379723+0000","last_fullsized":"2022-06-27T11:47:52.379723+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:15:14.505656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.377592+0000","last_change":"2022-06-27T11:47:52.376074+0000","last_active":"2022-06-27T11:47:52.377592+0000","last_peered":"2022-06-27T11:47:52.377592+0000","last_clean":"2022-06-27T11:47:52.377592+0000","last_became_active":"2022-06-27T11:47:52.375836+0000","last_became_peered":"2022-06-27T11:47:52.375836+0000","last_unstale":"2022-06-27T11:47:52.377592+0000","last_undegraded":"2022-06-27T11:47:52.377592+0000","last_fullsized":"2022-06-27T11:47:52.377592+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T18:46:26.900833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384440+0000","last_change":"2022-06-27T11:47:52.376668+0000","last_active":"2022-06-27T11:47:52.384440+0000","last_peered":"2022-06-27T11:47:52.384440+0000","last_clean":"2022-06-27T11:47:52.384440+0000","last_became_active":"2022-06-27T11:47:52.376386+0000","last_became_peered":"2022-06-27T11:47:52.376386+0000","last_unstale":"2022-06-27T11:47:52.384440+0000","last_undegraded":"2022-06-27T11:47:52.384440+0000","last_fullsized":"2022-06-27T11:47:52.384440+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T11:48:35.874748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.379744+0000","last_change":"2022-06-27T11:47:52.377793+0000","last_active":"2022-06-27T11:47:52.379744+0000","last_peered":"2022-06-27T11:47:52.379744+0000","last_clean":"2022-06-27T11:47:52.379744+0000","last_became_active":"2022-06-27T11:47:52.377318+0000","last_became_peered":"2022-06-27T11:47:52.377318+0000","last_unstale":"2022-06-27T11:47:52.379744+0000","last_undegraded":"2022-06-27T11:47:52.379744+0000","last_fullsized":"2022-06-27T11:47:52.379744+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T17:54:22.511301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384925+0000","last_change":"2022-06-27T11:47:52.374638+0000","last_active":"2022-06-27T11:47:52.384925+0000","last_peered":"2022-06-27T11:47:52.384925+0000","last_clean":"2022-06-27T11:47:52.384925+0000","last_became_active":"2022-06-27T11:47:52.374352+0000","last_became_peered":"2022-06-27T11:47:52.374352+0000","last_unstale":"2022-06-27T11:47:52.384925+0000","last_undegraded":"2022-06-27T11:47:52.384925+0000","last_fullsized":"2022-06-27T11:47:52.384925+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:09:59.369369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.375769+0000","last_change":"2022-06-27T11:47:52.373288+0000","last_active":"2022-06-27T11:47:52.375769+0000","last_peered":"2022-06-27T11:47:52.375769+0000","last_clean":"2022-06-27T11:47:52.375769+0000","last_became_active":"2022-06-27T11:47:52.372891+0000","last_became_peered":"2022-06-27T11:47:52.372891+0000","last_unstale":"2022-06-27T11:47:52.375769+0000","last_undegraded":"2022-06-27T11:47:52.375769+0000","last_fullsized":"2022-06-27T11:47:52.375769+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T23:16:08.088450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2590,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":5180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40,"ondisk_log_size":40,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64756,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9016,"kb_used_data":304,"kb_used_omap":0,"kb_used_meta":8704,"kb_avail":94362824,"statfs":{"total":96636764160,"available":96627531776,"internally_reserved":0,"allocated":311296,"data_stored":671231,"data_compressed":32378,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":8912896},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38900000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57499999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8408,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":8192,"kb_avail":94363432,"statfs":{"total":96636764160,"available":96628154368,"internally_reserved":0,"allocated":212992,"data_stored":77836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8388608},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36299999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51900000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8988,"kb_used_data":276,"kb_used_omap":0,"kb_used_meta":8704,"kb_avail":94362852,"statfs":{"total":96636764160,"available":96627560448,"internally_reserved":0,"allocated":282624,"data_stored":669207,"data_compressed":32378,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":8912896},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36099999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52500000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8480,"kb_used_data":216,"kb_used_omap":0,"kb_used_meta":8256,"kb_avail":94363360,"statfs":{"total":96636764160,"available":96628080640,"internally_reserved":0,"allocated":221184,"data_stored":79405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8454144},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.253}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32300000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30299999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.442}]}]},{"osd":2,"up_from":13,"seq":55834574857,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8412,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":8192,"kb_avail":94363428,"statfs":{"total":96636764160,"available":96628150272,"internally_reserved":0,"allocated":217088,"data_stored":78839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8388608},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35999999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30599999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52400000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8476,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":8256,"kb_avail":94363364,"statfs":{"total":96636764160,"available":96628084736,"internally_reserved":0,"allocated":217088,"data_stored":77926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8454144},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24399999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33400000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42299999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]}]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8560,"kb_used_data":232,"kb_used_omap":0,"kb_used_meta":8320,"kb_avail":94363280,"statfs":{"total":96636764160,"available":96627998720,"internally_reserved":0,"allocated":237568,"data_stored":80703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8519680},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.436}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46500000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]}]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8420,"kb_used_data":220,"kb_used_omap":0,"kb_used_meta":8192,"kb_avail":94363420,"statfs":{"total":96636764160,"available":96628142080,"internally_reserved":0,"allocated":225280,"data_stored":79475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8388608},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68500000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.748}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72099999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76500000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84499999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32378,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32378,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-06-27T11:48:11.756 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2022-06-27T11:48:11.877 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.878+0000 7f1ec2792700 1 Processor -- start 2022-06-27T11:48:11.878 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.878+0000 7f1ec2792700 1 -- start start 2022-06-27T11:48:11.883 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.878+0000 7f1ec2792700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f1ebc150580 0x7f1ebc156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:11.883 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.879+0000 7f1ec2792700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1ebc150950 0x7f1ebc0c7b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:11.883 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.879+0000 7f1ec2792700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ebc0c76b0 con 0x7f1ebc06f620 2022-06-27T11:48:11.884 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.879+0000 7f1ec2792700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ebc157430 con 0x7f1ebc150580 2022-06-27T11:48:11.884 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.879+0000 7f1ec2792700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ebc0bad70 con 0x7f1ebc150950 2022-06-27T11:48:11.884 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.879+0000 7f1ebbfff700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f1ebc150580 0x7f1ebc156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:11.884 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.879+0000 7f1ebb7fe700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1ebc150950 0x7f1ebc0c7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:11.885 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.879+0000 7f1ec0d2f700 1 --1- >> v1:172.21.15.202:6789/0 conn(0x7f1ebc06f620 0x7f1ebc06d6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.202:6789/0 says I am v1:172.21.15.195:50134/0 (socket says 172.21.15.195:50134) 2022-06-27T11:48:11.885 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.879+0000 7f1ec0d2f700 1 -- 172.21.15.195:0/2624982871 learned_addr learned my addr 172.21.15.195:0/2624982871 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:48:11.885 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.879+0000 7f1ebbfff700 1 -- 172.21.15.195:0/2624982871 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1ebc150950 msgr2=0x7f1ebc0c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:11.885 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.879+0000 7f1ebbfff700 1 --2- 172.21.15.195:0/2624982871 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1ebc150950 0x7f1ebc0c7b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:11.885 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.879+0000 7f1ebbfff700 1 -- 172.21.15.195:0/2624982871 >> v1:172.21.15.202:6789/0 conn(0x7f1ebc06f620 legacy=0x7f1ebc06d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:11.886 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.879+0000 7f1ebaffd700 1 -- 172.21.15.195:0/2624982871 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2511876071 0 0) 0x7f1ebc0c76b0 con 0x7f1ebc06f620 2022-06-27T11:48:11.886 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.879+0000 7f1ebbfff700 1 -- 172.21.15.195:0/2624982871 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ebc0ba4d0 con 0x7f1ebc150580 2022-06-27T11:48:11.886 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.879+0000 7f1ebb7fe700 1 --2- 172.21.15.195:0/2624982871 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1ebc150950 0x7f1ebc0c7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:48:11.886 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.879+0000 7f1ebbfff700 1 --2- 172.21.15.195:0/2624982871 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1ebc150580 0x7f1ebc156ef0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1eac0097a0 tx=0x7f1eac009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47aa5518e0c972b0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:11.887 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.879+0000 7f1ebaffd700 1 -- 172.21.15.195:0/2624982871 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eac011630 con 0x7f1ebc150580 2022-06-27T11:48:11.887 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.880+0000 7f1ebaffd700 1 -- 172.21.15.195:0/2624982871 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eac011790 con 0x7f1ebc150580 2022-06-27T11:48:11.887 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.880+0000 7f1ebaffd700 1 -- 172.21.15.195:0/2624982871 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eac011b30 con 0x7f1ebc150580 2022-06-27T11:48:11.888 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.880+0000 7f1ec2792700 1 -- 172.21.15.195:0/2624982871 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1ebc150580 msgr2=0x7f1ebc156ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:11.888 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.880+0000 7f1ec2792700 1 --2- 172.21.15.195:0/2624982871 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1ebc150580 0x7f1ebc156ef0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1eac0097a0 tx=0x7f1eac009060 comp rx=0 tx=0).stop 2022-06-27T11:48:11.888 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.880+0000 7f1ec2792700 1 -- 172.21.15.195:0/2624982871 shutdown_connections 2022-06-27T11:48:11.889 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.880+0000 7f1ec2792700 1 --2- 172.21.15.195:0/2624982871 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1ebc150580 0x7f1ebc156ef0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:11.889 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.880+0000 7f1ec2792700 1 --2- 172.21.15.195:0/2624982871 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1ebc150950 0x7f1ebc0c7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:11.889 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.880+0000 7f1ec2792700 1 -- 172.21.15.195:0/2624982871 >> 172.21.15.195:0/2624982871 conn(0x7f1ebc063af0 msgr2=0x7f1ebc063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:11.889 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.880+0000 7f1ec2792700 1 -- 172.21.15.195:0/2624982871 shutdown_connections 2022-06-27T11:48:11.889 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.880+0000 7f1ec2792700 1 -- 172.21.15.195:0/2624982871 wait complete. 2022-06-27T11:48:11.890 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.881+0000 7f1ec2792700 1 Processor -- start 2022-06-27T11:48:11.890 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.881+0000 7f1ec2792700 1 -- start start 2022-06-27T11:48:11.890 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.882+0000 7f1ec2792700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1ebc06f620 0x7f1ebc0bc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:11.890 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.882+0000 7f1ec2792700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1ebc150580 0x7f1ebc0bca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:11.891 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.882+0000 7f1ec2792700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1ebc150950 0x7f1ebc0bd7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:11.891 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.882+0000 7f1ec2792700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ebc0c93e0 con 0x7f1ebc150950 2022-06-27T11:48:11.891 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.882+0000 7f1ec2792700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ebc0ca960 con 0x7f1ebc150580 2022-06-27T11:48:11.891 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.882+0000 7f1ec2792700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ebc0bc050 con 0x7f1ebc06f620 2022-06-27T11:48:11.891 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.882+0000 7f1ec0d2f700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1ebc150950 0x7f1ebc0bd7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:11.892 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.882+0000 7f1ec0d2f700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1ebc150950 0x7f1ebc0bd7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:42092/0 (socket says 172.21.15.195:42092) 2022-06-27T11:48:11.892 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.882+0000 7f1ec0d2f700 1 -- 172.21.15.195:0/3574096790 learned_addr learned my addr 172.21.15.195:0/3574096790 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:48:11.892 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.882+0000 7f1ebbfff700 1 --2- 172.21.15.195:0/3574096790 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1ebc06f620 0x7f1ebc0bc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:11.892 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.882+0000 7f1ebb7fe700 1 --2- 172.21.15.195:0/3574096790 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1ebc150580 0x7f1ebc0bca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:11.893 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.882+0000 7f1ec0d2f700 1 -- 172.21.15.195:0/3574096790 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1ebc06f620 msgr2=0x7f1ebc0bc4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:11.893 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.882+0000 7f1ec0d2f700 1 --2- 172.21.15.195:0/3574096790 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1ebc06f620 0x7f1ebc0bc4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:11.893 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.882+0000 7f1ec0d2f700 1 -- 172.21.15.195:0/3574096790 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1ebc150580 msgr2=0x7f1ebc0bca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:11.893 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.882+0000 7f1ec0d2f700 1 --2- 172.21.15.195:0/3574096790 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1ebc150580 0x7f1ebc0bca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:11.893 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.883+0000 7f1ec0d2f700 1 -- 172.21.15.195:0/3574096790 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eb00092e0 con 0x7f1ebc150950 2022-06-27T11:48:11.894 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.883+0000 7f1ec0d2f700 1 --2- 172.21.15.195:0/3574096790 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1ebc150950 0x7f1ebc0bd7e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f1eb400c840 tx=0x7f1eb4001540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:11.894 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.883+0000 7f1ea3fff700 1 -- 172.21.15.195:0/3574096790 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb4004c80 con 0x7f1ebc150950 2022-06-27T11:48:11.894 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.883+0000 7f1ea3fff700 1 -- 172.21.15.195:0/3574096790 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eb4004e00 con 0x7f1ebc150950 2022-06-27T11:48:11.895 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.883+0000 7f1ec2792700 1 -- 172.21.15.195:0/3574096790 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1eac0092e0 con 0x7f1ebc150950 2022-06-27T11:48:11.895 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.883+0000 7f1ea3fff700 1 -- 172.21.15.195:0/3574096790 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb40073a0 con 0x7f1ebc150950 2022-06-27T11:48:11.895 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.883+0000 7f1ec2792700 1 -- 172.21.15.195:0/3574096790 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ebc0bb680 con 0x7f1ebc150950 2022-06-27T11:48:11.895 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.884+0000 7f1ec2792700 1 -- 172.21.15.195:0/3574096790 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ebc0722c0 con 0x7f1ebc150950 2022-06-27T11:48:11.897 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.884+0000 7f1ea3fff700 1 -- 172.21.15.195:0/3574096790 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f1eb4010030 con 0x7f1ebc150950 2022-06-27T11:48:11.897 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.885+0000 7f1ea3fff700 1 --2- 172.21.15.195:0/3574096790 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1e9c063d00 0x7f1e9c0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:11.897 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.885+0000 7f1ea3fff700 1 -- 172.21.15.195:0/3574096790 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f1eb4080560 con 0x7f1ebc150950 2022-06-27T11:48:11.897 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.885+0000 7f1ebbfff700 1 --2- 172.21.15.195:0/3574096790 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1e9c063d00 0x7f1e9c0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:11.898 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.885+0000 7f1ebbfff700 1 --2- 172.21.15.195:0/3574096790 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1e9c063d00 0x7f1e9c0661b0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1eac009260 tx=0x7f1eac017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:11.898 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:11.889+0000 7f1ea3fff700 1 -- 172.21.15.195:0/3574096790 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f1eb404f0e0 con 0x7f1ebc150950 2022-06-27T11:48:12.064 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.064+0000 7f1ec2792700 1 -- 172.21.15.195:0/3574096790 --> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1ebc0bc190 con 0x7f1e9c063d00 2022-06-27T11:48:12.072 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.072+0000 7f1ea3fff700 1 -- 172.21.15.195:0/3574096790 <== mgr.4098 v2:172.21.15.195:6832/93955 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+284935 (secure 0 0 0) 0x7f1ebc0bc190 con 0x7f1e9c063d00 2022-06-27T11:48:12.079 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:48:12.083 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.075+0000 7f1ec2792700 1 -- 172.21.15.195:0/3574096790 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1e9c063d00 msgr2=0x7f1e9c0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:12.083 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.075+0000 7f1ec2792700 1 --2- 172.21.15.195:0/3574096790 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1e9c063d00 0x7f1e9c0661b0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1eac009260 tx=0x7f1eac017000 comp rx=0 tx=0).stop 2022-06-27T11:48:12.084 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.075+0000 7f1ec2792700 1 -- 172.21.15.195:0/3574096790 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1ebc150950 msgr2=0x7f1ebc0bd7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:12.084 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.075+0000 7f1ec2792700 1 --2- 172.21.15.195:0/3574096790 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1ebc150950 0x7f1ebc0bd7e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f1eb400c840 tx=0x7f1eb4001540 comp rx=0 tx=0).stop 2022-06-27T11:48:12.084 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.075+0000 7f1ec2792700 1 -- 172.21.15.195:0/3574096790 shutdown_connections 2022-06-27T11:48:12.085 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.075+0000 7f1ec2792700 1 --2- 172.21.15.195:0/3574096790 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1e9c063d00 0x7f1e9c0661b0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:12.085 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.075+0000 7f1ec2792700 1 --2- 172.21.15.195:0/3574096790 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1ebc150950 0x7f1ebc0bd7e0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:12.086 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.075+0000 7f1ec2792700 1 --2- 172.21.15.195:0/3574096790 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1ebc06f620 0x7f1ebc0bc4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:12.086 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.075+0000 7f1ec2792700 1 --2- 172.21.15.195:0/3574096790 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1ebc150580 0x7f1ebc0bca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:12.086 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.075+0000 7f1ec2792700 1 -- 172.21.15.195:0/3574096790 >> 172.21.15.195:0/3574096790 conn(0x7f1ebc063af0 msgr2=0x7f1ebc14e6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:12.087 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.075+0000 7f1ec2792700 1 -- 172.21.15.195:0/3574096790 shutdown_connections 2022-06-27T11:48:12.087 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.075+0000 7f1ec2792700 1 -- 172.21.15.195:0/3574096790 wait complete. 2022-06-27T11:48:12.087 INFO:teuthology.orchestra.run.smithi195.stderr:dumped all 2022-06-27T11:48:12.089 INFO:teuthology.orchestra.run.smithi195.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2022-06-27T11:48:10.343250+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592958,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":205,"num_write_kb":2431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":210,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":68760,"kb_used_data":1880,"kb_used_omap":0,"kb_used_meta":66816,"kb_avail":754905960,"statfs":{"total":773094113280,"available":773023703040,"internally_reserved":0,"allocated":1925120,"data_stored":1814622,"data_compressed":64756,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":68419584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2590,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002303"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384962+0000","last_change":"2022-06-27T11:47:52.381823+0000","last_active":"2022-06-27T11:47:52.384962+0000","last_peered":"2022-06-27T11:47:52.384962+0000","last_clean":"2022-06-27T11:47:52.384962+0000","last_became_active":"2022-06-27T11:47:52.381551+0000","last_became_peered":"2022-06-27T11:47:52.381551+0000","last_unstale":"2022-06-27T11:47:52.384962+0000","last_undegraded":"2022-06-27T11:47:52.384962+0000","last_fullsized":"2022-06-27T11:47:52.384962+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:17:34.164061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.378135+0000","last_change":"2022-06-27T11:47:52.374939+0000","last_active":"2022-06-27T11:47:52.378135+0000","last_peered":"2022-06-27T11:47:52.378135+0000","last_clean":"2022-06-27T11:47:52.378135+0000","last_became_active":"2022-06-27T11:47:52.374620+0000","last_became_peered":"2022-06-27T11:47:52.374620+0000","last_unstale":"2022-06-27T11:47:52.378135+0000","last_undegraded":"2022-06-27T11:47:52.378135+0000","last_fullsized":"2022-06-27T11:47:52.378135+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:56:27.254098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384127+0000","last_change":"2022-06-27T11:47:52.380802+0000","last_active":"2022-06-27T11:47:52.384127+0000","last_peered":"2022-06-27T11:47:52.384127+0000","last_clean":"2022-06-27T11:47:52.384127+0000","last_became_active":"2022-06-27T11:47:52.380500+0000","last_became_peered":"2022-06-27T11:47:52.380500+0000","last_unstale":"2022-06-27T11:47:52.384127+0000","last_undegraded":"2022-06-27T11:47:52.384127+0000","last_fullsized":"2022-06-27T11:47:52.384127+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:06:42.889598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384114+0000","last_change":"2022-06-27T11:47:52.380943+0000","last_active":"2022-06-27T11:47:52.384114+0000","last_peered":"2022-06-27T11:47:52.384114+0000","last_clean":"2022-06-27T11:47:52.384114+0000","last_became_active":"2022-06-27T11:47:52.380648+0000","last_became_peered":"2022-06-27T11:47:52.380648+0000","last_unstale":"2022-06-27T11:47:52.384114+0000","last_undegraded":"2022-06-27T11:47:52.384114+0000","last_fullsized":"2022-06-27T11:47:52.384114+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:14:19.729605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.386875+0000","last_change":"2022-06-27T11:47:52.386144+0000","last_active":"2022-06-27T11:47:52.386875+0000","last_peered":"2022-06-27T11:47:52.386875+0000","last_clean":"2022-06-27T11:47:52.386875+0000","last_became_active":"2022-06-27T11:47:52.385586+0000","last_became_peered":"2022-06-27T11:47:52.385586+0000","last_unstale":"2022-06-27T11:47:52.386875+0000","last_undegraded":"2022-06-27T11:47:52.386875+0000","last_fullsized":"2022-06-27T11:47:52.386875+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T14:30:03.660749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.387182+0000","last_change":"2022-06-27T11:47:52.381906+0000","last_active":"2022-06-27T11:47:52.387182+0000","last_peered":"2022-06-27T11:47:52.387182+0000","last_clean":"2022-06-27T11:47:52.387182+0000","last_became_active":"2022-06-27T11:47:52.381279+0000","last_became_peered":"2022-06-27T11:47:52.381279+0000","last_unstale":"2022-06-27T11:47:52.387182+0000","last_undegraded":"2022-06-27T11:47:52.387182+0000","last_fullsized":"2022-06-27T11:47:52.387182+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T18:41:03.991762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.383137+0000","last_change":"2022-06-27T11:47:52.377128+0000","last_active":"2022-06-27T11:47:52.383137+0000","last_peered":"2022-06-27T11:47:52.383137+0000","last_clean":"2022-06-27T11:47:52.383137+0000","last_became_active":"2022-06-27T11:47:52.376499+0000","last_became_peered":"2022-06-27T11:47:52.376499+0000","last_unstale":"2022-06-27T11:47:52.383137+0000","last_undegraded":"2022-06-27T11:47:52.383137+0000","last_fullsized":"2022-06-27T11:47:52.383137+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:00:13.208145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.378171+0000","last_change":"2022-06-27T11:47:52.374169+0000","last_active":"2022-06-27T11:47:52.378171+0000","last_peered":"2022-06-27T11:47:52.378171+0000","last_clean":"2022-06-27T11:47:52.378171+0000","last_became_active":"2022-06-27T11:47:52.373917+0000","last_became_peered":"2022-06-27T11:47:52.373917+0000","last_unstale":"2022-06-27T11:47:52.378171+0000","last_undegraded":"2022-06-27T11:47:52.378171+0000","last_fullsized":"2022-06-27T11:47:52.378171+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:43:52.741254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384769+0000","last_change":"2022-06-27T11:47:52.382039+0000","last_active":"2022-06-27T11:47:52.384769+0000","last_peered":"2022-06-27T11:47:52.384769+0000","last_clean":"2022-06-27T11:47:52.384769+0000","last_became_active":"2022-06-27T11:47:52.381202+0000","last_became_peered":"2022-06-27T11:47:52.381202+0000","last_unstale":"2022-06-27T11:47:52.384769+0000","last_undegraded":"2022-06-27T11:47:52.384769+0000","last_fullsized":"2022-06-27T11:47:52.384769+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:13:02.553823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.383894+0000","last_change":"2022-06-27T11:47:52.380839+0000","last_active":"2022-06-27T11:47:52.383894+0000","last_peered":"2022-06-27T11:47:52.383894+0000","last_clean":"2022-06-27T11:47:52.383894+0000","last_became_active":"2022-06-27T11:47:52.380527+0000","last_became_peered":"2022-06-27T11:47:52.380527+0000","last_unstale":"2022-06-27T11:47:52.383894+0000","last_undegraded":"2022-06-27T11:47:52.383894+0000","last_fullsized":"2022-06-27T11:47:52.383894+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T23:18:07.086598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.374783+0000","last_change":"2022-06-27T11:47:52.371267+0000","last_active":"2022-06-27T11:47:52.374783+0000","last_peered":"2022-06-27T11:47:52.374783+0000","last_clean":"2022-06-27T11:47:52.374783+0000","last_became_active":"2022-06-27T11:47:52.370921+0000","last_became_peered":"2022-06-27T11:47:52.370921+0000","last_unstale":"2022-06-27T11:47:52.374783+0000","last_undegraded":"2022-06-27T11:47:52.374783+0000","last_fullsized":"2022-06-27T11:47:52.374783+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T19:22:39.648715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.378994+0000","last_change":"2022-06-27T11:47:52.375381+0000","last_active":"2022-06-27T11:47:52.378994+0000","last_peered":"2022-06-27T11:47:52.378994+0000","last_clean":"2022-06-27T11:47:52.378994+0000","last_became_active":"2022-06-27T11:47:52.375089+0000","last_became_peered":"2022-06-27T11:47:52.375089+0000","last_unstale":"2022-06-27T11:47:52.378994+0000","last_undegraded":"2022-06-27T11:47:52.378994+0000","last_fullsized":"2022-06-27T11:47:52.378994+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:43:14.830424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384993+0000","last_change":"2022-06-27T11:47:52.381690+0000","last_active":"2022-06-27T11:47:52.384993+0000","last_peered":"2022-06-27T11:47:52.384993+0000","last_clean":"2022-06-27T11:47:52.384993+0000","last_became_active":"2022-06-27T11:47:52.381472+0000","last_became_peered":"2022-06-27T11:47:52.381472+0000","last_unstale":"2022-06-27T11:47:52.384993+0000","last_undegraded":"2022-06-27T11:47:52.384993+0000","last_fullsized":"2022-06-27T11:47:52.384993+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:14:56.919536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.378838+0000","last_change":"2022-06-27T11:47:52.376063+0000","last_active":"2022-06-27T11:47:52.378838+0000","last_peered":"2022-06-27T11:47:52.378838+0000","last_clean":"2022-06-27T11:47:52.378838+0000","last_became_active":"2022-06-27T11:47:52.375778+0000","last_became_peered":"2022-06-27T11:47:52.375778+0000","last_unstale":"2022-06-27T11:47:52.378838+0000","last_undegraded":"2022-06-27T11:47:52.378838+0000","last_fullsized":"2022-06-27T11:47:52.378838+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T23:16:52.221967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.383160+0000","last_change":"2022-06-27T11:47:52.377297+0000","last_active":"2022-06-27T11:47:52.383160+0000","last_peered":"2022-06-27T11:47:52.383160+0000","last_clean":"2022-06-27T11:47:52.383160+0000","last_became_active":"2022-06-27T11:47:52.376855+0000","last_became_peered":"2022-06-27T11:47:52.376855+0000","last_unstale":"2022-06-27T11:47:52.383160+0000","last_undegraded":"2022-06-27T11:47:52.383160+0000","last_fullsized":"2022-06-27T11:47:52.383160+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:33:39.990522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.387168+0000","last_change":"2022-06-27T11:47:52.381821+0000","last_active":"2022-06-27T11:47:52.387168+0000","last_peered":"2022-06-27T11:47:52.387168+0000","last_clean":"2022-06-27T11:47:52.387168+0000","last_became_active":"2022-06-27T11:47:52.381548+0000","last_became_peered":"2022-06-27T11:47:52.381548+0000","last_unstale":"2022-06-27T11:47:52.387168+0000","last_undegraded":"2022-06-27T11:47:52.387168+0000","last_fullsized":"2022-06-27T11:47:52.387168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T23:14:53.320016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384753+0000","last_change":"2022-06-27T11:47:52.381811+0000","last_active":"2022-06-27T11:47:52.384753+0000","last_peered":"2022-06-27T11:47:52.384753+0000","last_clean":"2022-06-27T11:47:52.384753+0000","last_became_active":"2022-06-27T11:47:52.381554+0000","last_became_peered":"2022-06-27T11:47:52.381554+0000","last_unstale":"2022-06-27T11:47:52.384753+0000","last_undegraded":"2022-06-27T11:47:52.384753+0000","last_fullsized":"2022-06-27T11:47:52.384753+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T14:19:38.519380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.387218+0000","last_change":"2022-06-27T11:47:52.382952+0000","last_active":"2022-06-27T11:47:52.387218+0000","last_peered":"2022-06-27T11:47:52.387218+0000","last_clean":"2022-06-27T11:47:52.387218+0000","last_became_active":"2022-06-27T11:47:52.382691+0000","last_became_peered":"2022-06-27T11:47:52.382691+0000","last_unstale":"2022-06-27T11:47:52.387218+0000","last_undegraded":"2022-06-27T11:47:52.387218+0000","last_fullsized":"2022-06-27T11:47:52.387218+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:28:54.743970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.383682+0000","last_change":"2022-06-27T11:47:52.379083+0000","last_active":"2022-06-27T11:47:52.383682+0000","last_peered":"2022-06-27T11:47:52.383682+0000","last_clean":"2022-06-27T11:47:52.383682+0000","last_became_active":"2022-06-27T11:47:52.378724+0000","last_became_peered":"2022-06-27T11:47:52.378724+0000","last_unstale":"2022-06-27T11:47:52.383682+0000","last_undegraded":"2022-06-27T11:47:52.383682+0000","last_fullsized":"2022-06-27T11:47:52.383682+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T21:25:07.786935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.383275+0000","last_change":"2022-06-27T11:47:52.377947+0000","last_active":"2022-06-27T11:47:52.383275+0000","last_peered":"2022-06-27T11:47:52.383275+0000","last_clean":"2022-06-27T11:47:52.383275+0000","last_became_active":"2022-06-27T11:47:52.377652+0000","last_became_peered":"2022-06-27T11:47:52.377652+0000","last_unstale":"2022-06-27T11:47:52.383275+0000","last_undegraded":"2022-06-27T11:47:52.383275+0000","last_fullsized":"2022-06-27T11:47:52.383275+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T20:48:28.642052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384444+0000","last_change":"2022-06-27T11:47:52.381920+0000","last_active":"2022-06-27T11:47:52.384444+0000","last_peered":"2022-06-27T11:47:52.384444+0000","last_clean":"2022-06-27T11:47:52.384444+0000","last_became_active":"2022-06-27T11:47:52.381477+0000","last_became_peered":"2022-06-27T11:47:52.381477+0000","last_unstale":"2022-06-27T11:47:52.384444+0000","last_undegraded":"2022-06-27T11:47:52.384444+0000","last_fullsized":"2022-06-27T11:47:52.384444+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T20:42:25.529336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.386345+0000","last_change":"2022-06-27T11:47:52.385231+0000","last_active":"2022-06-27T11:47:52.386345+0000","last_peered":"2022-06-27T11:47:52.386345+0000","last_clean":"2022-06-27T11:47:52.386345+0000","last_became_active":"2022-06-27T11:47:52.384972+0000","last_became_peered":"2022-06-27T11:47:52.384972+0000","last_unstale":"2022-06-27T11:47:52.386345+0000","last_undegraded":"2022-06-27T11:47:52.386345+0000","last_fullsized":"2022-06-27T11:47:52.386345+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T17:37:26.084788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.385330+0000","last_change":"2022-06-27T11:47:52.381358+0000","last_active":"2022-06-27T11:47:52.385330+0000","last_peered":"2022-06-27T11:47:52.385330+0000","last_clean":"2022-06-27T11:47:52.385330+0000","last_became_active":"2022-06-27T11:47:52.380997+0000","last_became_peered":"2022-06-27T11:47:52.380997+0000","last_unstale":"2022-06-27T11:47:52.385330+0000","last_undegraded":"2022-06-27T11:47:52.385330+0000","last_fullsized":"2022-06-27T11:47:52.385330+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T21:44:16.061234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384432+0000","last_change":"2022-06-27T11:47:52.381899+0000","last_active":"2022-06-27T11:47:52.384432+0000","last_peered":"2022-06-27T11:47:52.384432+0000","last_clean":"2022-06-27T11:47:52.384432+0000","last_became_active":"2022-06-27T11:47:52.381645+0000","last_became_peered":"2022-06-27T11:47:52.381645+0000","last_unstale":"2022-06-27T11:47:52.384432+0000","last_undegraded":"2022-06-27T11:47:52.384432+0000","last_fullsized":"2022-06-27T11:47:52.384432+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:05:59.742023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.385053+0000","last_change":"2022-06-27T11:47:52.378736+0000","last_active":"2022-06-27T11:47:52.385053+0000","last_peered":"2022-06-27T11:47:52.385053+0000","last_clean":"2022-06-27T11:47:52.385053+0000","last_became_active":"2022-06-27T11:47:52.378475+0000","last_became_peered":"2022-06-27T11:47:52.378475+0000","last_unstale":"2022-06-27T11:47:52.385053+0000","last_undegraded":"2022-06-27T11:47:52.385053+0000","last_fullsized":"2022-06-27T11:47:52.385053+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T12:42:12.528716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384922+0000","last_change":"2022-06-27T11:47:52.378044+0000","last_active":"2022-06-27T11:47:52.384922+0000","last_peered":"2022-06-27T11:47:52.384922+0000","last_clean":"2022-06-27T11:47:52.384922+0000","last_became_active":"2022-06-27T11:47:52.377791+0000","last_became_peered":"2022-06-27T11:47:52.377791+0000","last_unstale":"2022-06-27T11:47:52.384922+0000","last_undegraded":"2022-06-27T11:47:52.384922+0000","last_fullsized":"2022-06-27T11:47:52.384922+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:21:45.615162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384115+0000","last_change":"2022-06-27T11:47:52.382267+0000","last_active":"2022-06-27T11:47:52.384115+0000","last_peered":"2022-06-27T11:47:52.384115+0000","last_clean":"2022-06-27T11:47:52.384115+0000","last_became_active":"2022-06-27T11:47:52.381985+0000","last_became_peered":"2022-06-27T11:47:52.381985+0000","last_unstale":"2022-06-27T11:47:52.384115+0000","last_undegraded":"2022-06-27T11:47:52.384115+0000","last_fullsized":"2022-06-27T11:47:52.384115+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:34:09.509759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:53.909338+0000","last_change":"2022-06-27T11:47:51.982183+0000","last_active":"2022-06-27T11:47:53.909338+0000","last_peered":"2022-06-27T11:47:53.909338+0000","last_clean":"2022-06-27T11:47:53.909338+0000","last_became_active":"2022-06-27T11:47:51.981339+0000","last_became_peered":"2022-06-27T11:47:51.981339+0000","last_unstale":"2022-06-27T11:47:53.909338+0000","last_undegraded":"2022-06-27T11:47:53.909338+0000","last_fullsized":"2022-06-27T11:47:53.909338+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:08:09.223947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384452+0000","last_change":"2022-06-27T11:47:52.377197+0000","last_active":"2022-06-27T11:47:52.384452+0000","last_peered":"2022-06-27T11:47:52.384452+0000","last_clean":"2022-06-27T11:47:52.384452+0000","last_became_active":"2022-06-27T11:47:52.376925+0000","last_became_peered":"2022-06-27T11:47:52.376925+0000","last_unstale":"2022-06-27T11:47:52.384452+0000","last_undegraded":"2022-06-27T11:47:52.384452+0000","last_fullsized":"2022-06-27T11:47:52.384452+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:39:27.103978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.364221+0000","last_change":"2022-06-27T11:47:51.981074+0000","last_active":"2022-06-27T11:47:52.364221+0000","last_peered":"2022-06-27T11:47:52.364221+0000","last_clean":"2022-06-27T11:47:52.364221+0000","last_became_active":"2022-06-27T11:47:51.980636+0000","last_became_peered":"2022-06-27T11:47:51.980636+0000","last_unstale":"2022-06-27T11:47:52.364221+0000","last_undegraded":"2022-06-27T11:47:52.364221+0000","last_fullsized":"2022-06-27T11:47:52.364221+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T23:32:52.850801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384380+0000","last_change":"2022-06-27T11:47:52.378415+0000","last_active":"2022-06-27T11:47:52.384380+0000","last_peered":"2022-06-27T11:47:52.384380+0000","last_clean":"2022-06-27T11:47:52.384380+0000","last_became_active":"2022-06-27T11:47:52.378123+0000","last_became_peered":"2022-06-27T11:47:52.378123+0000","last_unstale":"2022-06-27T11:47:52.384380+0000","last_undegraded":"2022-06-27T11:47:52.384380+0000","last_fullsized":"2022-06-27T11:47:52.384380+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T11:54:37.127828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.369137+0000","last_change":"2022-06-27T11:47:51.975684+0000","last_active":"2022-06-27T11:47:52.369137+0000","last_peered":"2022-06-27T11:47:52.369137+0000","last_clean":"2022-06-27T11:47:52.369137+0000","last_became_active":"2022-06-27T11:47:51.974902+0000","last_became_peered":"2022-06-27T11:47:51.974902+0000","last_unstale":"2022-06-27T11:47:52.369137+0000","last_undegraded":"2022-06-27T11:47:52.369137+0000","last_fullsized":"2022-06-27T11:47:52.369137+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:51:45.780798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.378008+0000","last_change":"2022-06-27T11:47:52.376089+0000","last_active":"2022-06-27T11:47:52.378008+0000","last_peered":"2022-06-27T11:47:52.378008+0000","last_clean":"2022-06-27T11:47:52.378008+0000","last_became_active":"2022-06-27T11:47:52.375780+0000","last_became_peered":"2022-06-27T11:47:52.375780+0000","last_unstale":"2022-06-27T11:47:52.378008+0000","last_undegraded":"2022-06-27T11:47:52.378008+0000","last_fullsized":"2022-06-27T11:47:52.378008+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T14:17:53.942880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.371313+0000","last_change":"2022-06-27T11:47:51.974071+0000","last_active":"2022-06-27T11:47:52.371313+0000","last_peered":"2022-06-27T11:47:52.371313+0000","last_clean":"2022-06-27T11:47:52.371313+0000","last_became_active":"2022-06-27T11:47:51.973682+0000","last_became_peered":"2022-06-27T11:47:51.973682+0000","last_unstale":"2022-06-27T11:47:52.371313+0000","last_undegraded":"2022-06-27T11:47:52.371313+0000","last_fullsized":"2022-06-27T11:47:52.371313+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T17:40:03.705083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384841+0000","last_change":"2022-06-27T11:47:52.377716+0000","last_active":"2022-06-27T11:47:52.384841+0000","last_peered":"2022-06-27T11:47:52.384841+0000","last_clean":"2022-06-27T11:47:52.384841+0000","last_became_active":"2022-06-27T11:47:52.377464+0000","last_became_peered":"2022-06-27T11:47:52.377464+0000","last_unstale":"2022-06-27T11:47:52.384841+0000","last_undegraded":"2022-06-27T11:47:52.384841+0000","last_fullsized":"2022-06-27T11:47:52.384841+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:39:36.234204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.368885+0000","last_change":"2022-06-27T11:47:51.975895+0000","last_active":"2022-06-27T11:47:52.368885+0000","last_peered":"2022-06-27T11:47:52.368885+0000","last_clean":"2022-06-27T11:47:52.368885+0000","last_became_active":"2022-06-27T11:47:51.975601+0000","last_became_peered":"2022-06-27T11:47:51.975601+0000","last_unstale":"2022-06-27T11:47:52.368885+0000","last_undegraded":"2022-06-27T11:47:52.368885+0000","last_fullsized":"2022-06-27T11:47:52.368885+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:43:55.833399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.385349+0000","last_change":"2022-06-27T11:47:52.383063+0000","last_active":"2022-06-27T11:47:52.385349+0000","last_peered":"2022-06-27T11:47:52.385349+0000","last_clean":"2022-06-27T11:47:52.385349+0000","last_became_active":"2022-06-27T11:47:52.382390+0000","last_became_peered":"2022-06-27T11:47:52.382390+0000","last_unstale":"2022-06-27T11:47:52.385349+0000","last_undegraded":"2022-06-27T11:47:52.385349+0000","last_fullsized":"2022-06-27T11:47:52.385349+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:40:10.929542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"21'3","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:57.367279+0000","last_change":"2022-06-27T11:47:51.980627+0000","last_active":"2022-06-27T11:47:57.367279+0000","last_peered":"2022-06-27T11:47:57.367279+0000","last_clean":"2022-06-27T11:47:57.367279+0000","last_became_active":"2022-06-27T11:47:51.980265+0000","last_became_peered":"2022-06-27T11:47:51.980265+0000","last_unstale":"2022-06-27T11:47:57.367279+0000","last_undegraded":"2022-06-27T11:47:57.367279+0000","last_fullsized":"2022-06-27T11:47:57.367279+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T14:18:32.599085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384584+0000","last_change":"2022-06-27T11:47:52.377475+0000","last_active":"2022-06-27T11:47:52.384584+0000","last_peered":"2022-06-27T11:47:52.384584+0000","last_clean":"2022-06-27T11:47:52.384584+0000","last_became_active":"2022-06-27T11:47:52.377220+0000","last_became_peered":"2022-06-27T11:47:52.377220+0000","last_unstale":"2022-06-27T11:47:52.384584+0000","last_undegraded":"2022-06-27T11:47:52.384584+0000","last_fullsized":"2022-06-27T11:47:52.384584+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T23:27:26.525688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"21'7","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:57.370253+0000","last_change":"2022-06-27T11:47:51.978195+0000","last_active":"2022-06-27T11:47:57.370253+0000","last_peered":"2022-06-27T11:47:57.370253+0000","last_clean":"2022-06-27T11:47:57.370253+0000","last_became_active":"2022-06-27T11:47:51.974673+0000","last_became_peered":"2022-06-27T11:47:51.974673+0000","last_unstale":"2022-06-27T11:47:57.370253+0000","last_undegraded":"2022-06-27T11:47:57.370253+0000","last_fullsized":"2022-06-27T11:47:57.370253+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T18:09:43.518644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1808,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.382204+0000","last_change":"2022-06-27T11:47:52.378740+0000","last_active":"2022-06-27T11:47:52.382204+0000","last_peered":"2022-06-27T11:47:52.382204+0000","last_clean":"2022-06-27T11:47:52.382204+0000","last_became_active":"2022-06-27T11:47:52.378451+0000","last_became_peered":"2022-06-27T11:47:52.378451+0000","last_unstale":"2022-06-27T11:47:52.382204+0000","last_undegraded":"2022-06-27T11:47:52.382204+0000","last_fullsized":"2022-06-27T11:47:52.382204+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:04:47.829409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.13","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:57.368186+0000","last_change":"2022-06-27T11:47:51.983039+0000","last_active":"2022-06-27T11:47:57.368186+0000","last_peered":"2022-06-27T11:47:57.368186+0000","last_clean":"2022-06-27T11:47:57.368186+0000","last_became_active":"2022-06-27T11:47:51.982683+0000","last_became_peered":"2022-06-27T11:47:51.982683+0000","last_unstale":"2022-06-27T11:47:57.368186+0000","last_undegraded":"2022-06-27T11:47:57.368186+0000","last_fullsized":"2022-06-27T11:47:57.368186+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T17:10:16.876753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384742+0000","last_change":"2022-06-27T11:47:52.374884+0000","last_active":"2022-06-27T11:47:52.384742+0000","last_peered":"2022-06-27T11:47:52.384742+0000","last_clean":"2022-06-27T11:47:52.384742+0000","last_became_active":"2022-06-27T11:47:52.374632+0000","last_became_peered":"2022-06-27T11:47:52.374632+0000","last_unstale":"2022-06-27T11:47:52.384742+0000","last_undegraded":"2022-06-27T11:47:52.384742+0000","last_fullsized":"2022-06-27T11:47:52.384742+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T21:33:54.724089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.374167+0000","last_change":"2022-06-27T11:47:51.976772+0000","last_active":"2022-06-27T11:47:52.374167+0000","last_peered":"2022-06-27T11:47:52.374167+0000","last_clean":"2022-06-27T11:47:52.374167+0000","last_became_active":"2022-06-27T11:47:51.976475+0000","last_became_peered":"2022-06-27T11:47:51.976475+0000","last_unstale":"2022-06-27T11:47:52.374167+0000","last_undegraded":"2022-06-27T11:47:52.374167+0000","last_fullsized":"2022-06-27T11:47:52.374167+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T23:39:10.919159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.377943+0000","last_change":"2022-06-27T11:47:52.373748+0000","last_active":"2022-06-27T11:47:52.377943+0000","last_peered":"2022-06-27T11:47:52.377943+0000","last_clean":"2022-06-27T11:47:52.377943+0000","last_became_active":"2022-06-27T11:47:52.373459+0000","last_became_peered":"2022-06-27T11:47:52.373459+0000","last_unstale":"2022-06-27T11:47:52.377943+0000","last_undegraded":"2022-06-27T11:47:52.377943+0000","last_fullsized":"2022-06-27T11:47:52.377943+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T18:27:22.254416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.367083+0000","last_change":"2022-06-27T11:47:51.982926+0000","last_active":"2022-06-27T11:47:52.367083+0000","last_peered":"2022-06-27T11:47:52.367083+0000","last_clean":"2022-06-27T11:47:52.367083+0000","last_became_active":"2022-06-27T11:47:51.982638+0000","last_became_peered":"2022-06-27T11:47:51.982638+0000","last_unstale":"2022-06-27T11:47:52.367083+0000","last_undegraded":"2022-06-27T11:47:52.367083+0000","last_fullsized":"2022-06-27T11:47:52.367083+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:34:04.834822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384791+0000","last_change":"2022-06-27T11:47:52.381623+0000","last_active":"2022-06-27T11:47:52.384791+0000","last_peered":"2022-06-27T11:47:52.384791+0000","last_clean":"2022-06-27T11:47:52.384791+0000","last_became_active":"2022-06-27T11:47:52.381258+0000","last_became_peered":"2022-06-27T11:47:52.381258+0000","last_unstale":"2022-06-27T11:47:52.384791+0000","last_undegraded":"2022-06-27T11:47:52.384791+0000","last_fullsized":"2022-06-27T11:47:52.384791+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:39:07.735224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"21'8","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:57.369384+0000","last_change":"2022-06-27T11:47:51.973125+0000","last_active":"2022-06-27T11:47:57.369384+0000","last_peered":"2022-06-27T11:47:57.369384+0000","last_clean":"2022-06-27T11:47:57.369384+0000","last_became_active":"2022-06-27T11:47:51.972806+0000","last_became_peered":"2022-06-27T11:47:51.972806+0000","last_unstale":"2022-06-27T11:47:57.369384+0000","last_undegraded":"2022-06-27T11:47:57.369384+0000","last_fullsized":"2022-06-27T11:47:57.369384+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:49:23.881991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384422+0000","last_change":"2022-06-27T11:47:52.376101+0000","last_active":"2022-06-27T11:47:52.384422+0000","last_peered":"2022-06-27T11:47:52.384422+0000","last_clean":"2022-06-27T11:47:52.384422+0000","last_became_active":"2022-06-27T11:47:52.375774+0000","last_became_peered":"2022-06-27T11:47:52.375774+0000","last_unstale":"2022-06-27T11:47:52.384422+0000","last_undegraded":"2022-06-27T11:47:52.384422+0000","last_fullsized":"2022-06-27T11:47:52.384422+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T18:21:13.049096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.f","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:53.492124+0000","last_change":"2022-06-27T11:47:51.981981+0000","last_active":"2022-06-27T11:47:53.492124+0000","last_peered":"2022-06-27T11:47:53.492124+0000","last_clean":"2022-06-27T11:47:53.492124+0000","last_became_active":"2022-06-27T11:47:51.981749+0000","last_became_peered":"2022-06-27T11:47:51.981749+0000","last_unstale":"2022-06-27T11:47:53.492124+0000","last_undegraded":"2022-06-27T11:47:53.492124+0000","last_fullsized":"2022-06-27T11:47:53.492124+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:28:52.457719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.378190+0000","last_change":"2022-06-27T11:47:52.374223+0000","last_active":"2022-06-27T11:47:52.378190+0000","last_peered":"2022-06-27T11:47:52.378190+0000","last_clean":"2022-06-27T11:47:52.378190+0000","last_became_active":"2022-06-27T11:47:52.373932+0000","last_became_peered":"2022-06-27T11:47:52.373932+0000","last_unstale":"2022-06-27T11:47:52.378190+0000","last_undegraded":"2022-06-27T11:47:52.378190+0000","last_fullsized":"2022-06-27T11:47:52.378190+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:48:33.254458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:57.368190+0000","last_change":"2022-06-27T11:47:51.982895+0000","last_active":"2022-06-27T11:47:57.368190+0000","last_peered":"2022-06-27T11:47:57.368190+0000","last_clean":"2022-06-27T11:47:57.368190+0000","last_became_active":"2022-06-27T11:47:51.982619+0000","last_became_peered":"2022-06-27T11:47:51.982619+0000","last_unstale":"2022-06-27T11:47:57.368190+0000","last_undegraded":"2022-06-27T11:47:57.368190+0000","last_fullsized":"2022-06-27T11:47:57.368190+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:22:01.241112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.377169+0000","last_change":"2022-06-27T11:47:52.375225+0000","last_active":"2022-06-27T11:47:52.377169+0000","last_peered":"2022-06-27T11:47:52.377169+0000","last_clean":"2022-06-27T11:47:52.377169+0000","last_became_active":"2022-06-27T11:47:52.374925+0000","last_became_peered":"2022-06-27T11:47:52.374925+0000","last_unstale":"2022-06-27T11:47:52.377169+0000","last_undegraded":"2022-06-27T11:47:52.377169+0000","last_fullsized":"2022-06-27T11:47:52.377169+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:54:31.945902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:53.492085+0000","last_change":"2022-06-27T11:47:51.975202+0000","last_active":"2022-06-27T11:47:53.492085+0000","last_peered":"2022-06-27T11:47:53.492085+0000","last_clean":"2022-06-27T11:47:53.492085+0000","last_became_active":"2022-06-27T11:47:51.974968+0000","last_became_peered":"2022-06-27T11:47:51.974968+0000","last_unstale":"2022-06-27T11:47:53.492085+0000","last_undegraded":"2022-06-27T11:47:53.492085+0000","last_fullsized":"2022-06-27T11:47:53.492085+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T21:16:30.362248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.378431+0000","last_change":"2022-06-27T11:47:52.377794+0000","last_active":"2022-06-27T11:47:52.378431+0000","last_peered":"2022-06-27T11:47:52.378431+0000","last_clean":"2022-06-27T11:47:52.378431+0000","last_became_active":"2022-06-27T11:47:52.377387+0000","last_became_peered":"2022-06-27T11:47:52.377387+0000","last_unstale":"2022-06-27T11:47:52.378431+0000","last_undegraded":"2022-06-27T11:47:52.378431+0000","last_fullsized":"2022-06-27T11:47:52.378431+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T17:00:00.924967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.368792+0000","last_change":"2022-06-27T11:47:51.976575+0000","last_active":"2022-06-27T11:47:52.368792+0000","last_peered":"2022-06-27T11:47:52.368792+0000","last_clean":"2022-06-27T11:47:52.368792+0000","last_became_active":"2022-06-27T11:47:51.976300+0000","last_became_peered":"2022-06-27T11:47:51.976300+0000","last_unstale":"2022-06-27T11:47:52.368792+0000","last_undegraded":"2022-06-27T11:47:52.368792+0000","last_fullsized":"2022-06-27T11:47:52.368792+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:08:46.067375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.387174+0000","last_change":"2022-06-27T11:47:52.386298+0000","last_active":"2022-06-27T11:47:52.387174+0000","last_peered":"2022-06-27T11:47:52.387174+0000","last_clean":"2022-06-27T11:47:52.387174+0000","last_became_active":"2022-06-27T11:47:52.385918+0000","last_became_peered":"2022-06-27T11:47:52.385918+0000","last_unstale":"2022-06-27T11:47:52.387174+0000","last_undegraded":"2022-06-27T11:47:52.387174+0000","last_fullsized":"2022-06-27T11:47:52.387174+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:21:51.780895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:57.364776+0000","last_change":"2022-06-27T11:47:51.973817+0000","last_active":"2022-06-27T11:47:57.364776+0000","last_peered":"2022-06-27T11:47:57.364776+0000","last_clean":"2022-06-27T11:47:57.364776+0000","last_became_active":"2022-06-27T11:47:51.973515+0000","last_became_peered":"2022-06-27T11:47:51.973515+0000","last_unstale":"2022-06-27T11:47:57.364776+0000","last_undegraded":"2022-06-27T11:47:57.364776+0000","last_fullsized":"2022-06-27T11:47:57.364776+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T21:01:45.170590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.374582+0000","last_change":"2022-06-27T11:47:52.371141+0000","last_active":"2022-06-27T11:47:52.374582+0000","last_peered":"2022-06-27T11:47:52.374582+0000","last_clean":"2022-06-27T11:47:52.374582+0000","last_became_active":"2022-06-27T11:47:52.370796+0000","last_became_peered":"2022-06-27T11:47:52.370796+0000","last_unstale":"2022-06-27T11:47:52.374582+0000","last_undegraded":"2022-06-27T11:47:52.374582+0000","last_fullsized":"2022-06-27T11:47:52.374582+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T23:16:39.266615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.369465+0000","last_change":"2022-06-27T11:47:51.983071+0000","last_active":"2022-06-27T11:47:52.369465+0000","last_peered":"2022-06-27T11:47:52.369465+0000","last_clean":"2022-06-27T11:47:52.369465+0000","last_became_active":"2022-06-27T11:47:51.982756+0000","last_became_peered":"2022-06-27T11:47:51.982756+0000","last_unstale":"2022-06-27T11:47:52.369465+0000","last_undegraded":"2022-06-27T11:47:52.369465+0000","last_fullsized":"2022-06-27T11:47:52.369465+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T17:14:00.737599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.387222+0000","last_change":"2022-06-27T11:47:52.382943+0000","last_active":"2022-06-27T11:47:52.387222+0000","last_peered":"2022-06-27T11:47:52.387222+0000","last_clean":"2022-06-27T11:47:52.387222+0000","last_became_active":"2022-06-27T11:47:52.382625+0000","last_became_peered":"2022-06-27T11:47:52.382625+0000","last_unstale":"2022-06-27T11:47:52.387222+0000","last_undegraded":"2022-06-27T11:47:52.387222+0000","last_fullsized":"2022-06-27T11:47:52.387222+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:01:18.570554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.362681+0000","last_change":"2022-06-27T11:47:51.978856+0000","last_active":"2022-06-27T11:47:52.362681+0000","last_peered":"2022-06-27T11:47:52.362681+0000","last_clean":"2022-06-27T11:47:52.362681+0000","last_became_active":"2022-06-27T11:47:51.978541+0000","last_became_peered":"2022-06-27T11:47:51.978541+0000","last_unstale":"2022-06-27T11:47:52.362681+0000","last_undegraded":"2022-06-27T11:47:52.362681+0000","last_fullsized":"2022-06-27T11:47:52.362681+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:36:24.190996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.378203+0000","last_change":"2022-06-27T11:47:52.376925+0000","last_active":"2022-06-27T11:47:52.378203+0000","last_peered":"2022-06-27T11:47:52.378203+0000","last_clean":"2022-06-27T11:47:52.378203+0000","last_became_active":"2022-06-27T11:47:52.376646+0000","last_became_peered":"2022-06-27T11:47:52.376646+0000","last_unstale":"2022-06-27T11:47:52.378203+0000","last_undegraded":"2022-06-27T11:47:52.378203+0000","last_fullsized":"2022-06-27T11:47:52.378203+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:17:51.913392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.369021+0000","last_change":"2022-06-27T11:47:51.979355+0000","last_active":"2022-06-27T11:47:52.369021+0000","last_peered":"2022-06-27T11:47:52.369021+0000","last_clean":"2022-06-27T11:47:52.369021+0000","last_became_active":"2022-06-27T11:47:51.977591+0000","last_became_peered":"2022-06-27T11:47:51.977591+0000","last_unstale":"2022-06-27T11:47:52.369021+0000","last_undegraded":"2022-06-27T11:47:52.369021+0000","last_fullsized":"2022-06-27T11:47:52.369021+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T11:50:43.086014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.385028+0000","last_change":"2022-06-27T11:47:52.375271+0000","last_active":"2022-06-27T11:47:52.385028+0000","last_peered":"2022-06-27T11:47:52.385028+0000","last_clean":"2022-06-27T11:47:52.385028+0000","last_became_active":"2022-06-27T11:47:52.374985+0000","last_became_peered":"2022-06-27T11:47:52.374985+0000","last_unstale":"2022-06-27T11:47:52.385028+0000","last_undegraded":"2022-06-27T11:47:52.385028+0000","last_fullsized":"2022-06-27T11:47:52.385028+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T17:58:40.711409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:57.367534+0000","last_change":"2022-06-27T11:47:51.979723+0000","last_active":"2022-06-27T11:47:57.367534+0000","last_peered":"2022-06-27T11:47:57.367534+0000","last_clean":"2022-06-27T11:47:57.367534+0000","last_became_active":"2022-06-27T11:47:51.977669+0000","last_became_peered":"2022-06-27T11:47:51.977669+0000","last_unstale":"2022-06-27T11:47:57.367534+0000","last_undegraded":"2022-06-27T11:47:57.367534+0000","last_fullsized":"2022-06-27T11:47:57.367534+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T14:35:14.770254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.363444+0000","last_change":"2022-06-27T11:47:48.388880+0000","last_active":"2022-06-27T11:47:52.363444+0000","last_peered":"2022-06-27T11:47:52.363444+0000","last_clean":"2022-06-27T11:47:52.363444+0000","last_became_active":"2022-06-27T11:47:48.388639+0000","last_became_peered":"2022-06-27T11:47:48.388639+0000","last_unstale":"2022-06-27T11:47:52.363444+0000","last_undegraded":"2022-06-27T11:47:52.363444+0000","last_fullsized":"2022-06-27T11:47:52.363444+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_clean_scrub_stamp":"2022-06-27T11:47:46.780512+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:31:10.549485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.380026+0000","last_change":"2022-06-27T11:47:52.373143+0000","last_active":"2022-06-27T11:47:52.380026+0000","last_peered":"2022-06-27T11:47:52.380026+0000","last_clean":"2022-06-27T11:47:52.380026+0000","last_became_active":"2022-06-27T11:47:52.372850+0000","last_became_peered":"2022-06-27T11:47:52.372850+0000","last_unstale":"2022-06-27T11:47:52.380026+0000","last_undegraded":"2022-06-27T11:47:52.380026+0000","last_fullsized":"2022-06-27T11:47:52.380026+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:48:44.251459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.369716+0000","last_change":"2022-06-27T11:47:47.793307+0000","last_active":"2022-06-27T11:47:52.369716+0000","last_peered":"2022-06-27T11:47:52.369716+0000","last_clean":"2022-06-27T11:47:52.369716+0000","last_became_active":"2022-06-27T11:47:47.792958+0000","last_became_peered":"2022-06-27T11:47:47.792958+0000","last_unstale":"2022-06-27T11:47:52.369716+0000","last_undegraded":"2022-06-27T11:47:52.369716+0000","last_fullsized":"2022-06-27T11:47:52.369716+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_clean_scrub_stamp":"2022-06-27T11:47:46.780512+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:08:35.128804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:53.910382+0000","last_change":"2022-06-27T11:47:51.976872+0000","last_active":"2022-06-27T11:47:53.910382+0000","last_peered":"2022-06-27T11:47:53.910382+0000","last_clean":"2022-06-27T11:47:53.910382+0000","last_became_active":"2022-06-27T11:47:51.976546+0000","last_became_peered":"2022-06-27T11:47:51.976546+0000","last_unstale":"2022-06-27T11:47:53.910382+0000","last_undegraded":"2022-06-27T11:47:53.910382+0000","last_fullsized":"2022-06-27T11:47:53.910382+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T14:44:43.349688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.375090+0000","last_change":"2022-06-27T11:47:52.372794+0000","last_active":"2022-06-27T11:47:52.375090+0000","last_peered":"2022-06-27T11:47:52.375090+0000","last_clean":"2022-06-27T11:47:52.375090+0000","last_became_active":"2022-06-27T11:47:52.372486+0000","last_became_peered":"2022-06-27T11:47:52.372486+0000","last_unstale":"2022-06-27T11:47:52.375090+0000","last_undegraded":"2022-06-27T11:47:52.375090+0000","last_fullsized":"2022-06-27T11:47:52.375090+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:23:56.307995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.363787+0000","last_change":"2022-06-27T11:47:47.792631+0000","last_active":"2022-06-27T11:47:52.363787+0000","last_peered":"2022-06-27T11:47:52.363787+0000","last_clean":"2022-06-27T11:47:52.363787+0000","last_became_active":"2022-06-27T11:47:47.792282+0000","last_became_peered":"2022-06-27T11:47:47.792282+0000","last_unstale":"2022-06-27T11:47:52.363787+0000","last_undegraded":"2022-06-27T11:47:52.363787+0000","last_fullsized":"2022-06-27T11:47:52.363787+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_clean_scrub_stamp":"2022-06-27T11:47:46.780512+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:41:25.449379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"17'117","reported_seq":163,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.368104+0000","last_change":"2022-06-27T11:47:47.791697+0000","last_active":"2022-06-27T11:47:52.368104+0000","last_peered":"2022-06-27T11:47:52.368104+0000","last_clean":"2022-06-27T11:47:52.368104+0000","last_became_active":"2022-06-27T11:47:47.790645+0000","last_became_peered":"2022-06-27T11:47:47.790645+0000","last_unstale":"2022-06-27T11:47:52.368104+0000","last_undegraded":"2022-06-27T11:47:52.368104+0000","last_fullsized":"2022-06-27T11:47:52.368104+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_clean_scrub_stamp":"2022-06-27T11:47:46.780512+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:11:09.344666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.367962+0000","last_change":"2022-06-27T11:47:51.982728+0000","last_active":"2022-06-27T11:47:52.367962+0000","last_peered":"2022-06-27T11:47:52.367962+0000","last_clean":"2022-06-27T11:47:52.367962+0000","last_became_active":"2022-06-27T11:47:51.982445+0000","last_became_peered":"2022-06-27T11:47:51.982445+0000","last_unstale":"2022-06-27T11:47:52.367962+0000","last_undegraded":"2022-06-27T11:47:52.367962+0000","last_fullsized":"2022-06-27T11:47:52.367962+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:42:04.682687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.386922+0000","last_change":"2022-06-27T11:47:52.386015+0000","last_active":"2022-06-27T11:47:52.386922+0000","last_peered":"2022-06-27T11:47:52.386922+0000","last_clean":"2022-06-27T11:47:52.386922+0000","last_became_active":"2022-06-27T11:47:52.385731+0000","last_became_peered":"2022-06-27T11:47:52.385731+0000","last_unstale":"2022-06-27T11:47:52.386922+0000","last_undegraded":"2022-06-27T11:47:52.386922+0000","last_fullsized":"2022-06-27T11:47:52.386922+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T11:58:50.483766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.366627+0000","last_change":"2022-06-27T11:47:47.807699+0000","last_active":"2022-06-27T11:47:52.366627+0000","last_peered":"2022-06-27T11:47:52.366627+0000","last_clean":"2022-06-27T11:47:52.366627+0000","last_became_active":"2022-06-27T11:47:47.807422+0000","last_became_peered":"2022-06-27T11:47:47.807422+0000","last_unstale":"2022-06-27T11:47:52.366627+0000","last_undegraded":"2022-06-27T11:47:52.366627+0000","last_fullsized":"2022-06-27T11:47:52.366627+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_clean_scrub_stamp":"2022-06-27T11:47:46.780512+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:15:00.549295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.363922+0000","last_change":"2022-06-27T11:47:51.974468+0000","last_active":"2022-06-27T11:47:52.363922+0000","last_peered":"2022-06-27T11:47:52.363922+0000","last_clean":"2022-06-27T11:47:52.363922+0000","last_became_active":"2022-06-27T11:47:51.974160+0000","last_became_peered":"2022-06-27T11:47:51.974160+0000","last_unstale":"2022-06-27T11:47:52.363922+0000","last_undegraded":"2022-06-27T11:47:52.363922+0000","last_fullsized":"2022-06-27T11:47:52.363922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T19:14:45.694282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.380162+0000","last_change":"2022-06-27T11:47:52.373931+0000","last_active":"2022-06-27T11:47:52.380162+0000","last_peered":"2022-06-27T11:47:52.380162+0000","last_clean":"2022-06-27T11:47:52.380162+0000","last_became_active":"2022-06-27T11:47:52.373562+0000","last_became_peered":"2022-06-27T11:47:52.373562+0000","last_unstale":"2022-06-27T11:47:52.380162+0000","last_undegraded":"2022-06-27T11:47:52.380162+0000","last_fullsized":"2022-06-27T11:47:52.380162+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T14:09:22.859955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.367866+0000","last_change":"2022-06-27T11:47:47.807650+0000","last_active":"2022-06-27T11:47:52.367866+0000","last_peered":"2022-06-27T11:47:52.367866+0000","last_clean":"2022-06-27T11:47:52.367866+0000","last_became_active":"2022-06-27T11:47:47.807375+0000","last_became_peered":"2022-06-27T11:47:47.807375+0000","last_unstale":"2022-06-27T11:47:52.367866+0000","last_undegraded":"2022-06-27T11:47:52.367866+0000","last_fullsized":"2022-06-27T11:47:52.367866+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_clean_scrub_stamp":"2022-06-27T11:47:46.780512+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T14:10:48.067284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.368898+0000","last_change":"2022-06-27T11:47:51.972154+0000","last_active":"2022-06-27T11:47:52.368898+0000","last_peered":"2022-06-27T11:47:52.368898+0000","last_clean":"2022-06-27T11:47:52.368898+0000","last_became_active":"2022-06-27T11:47:51.971762+0000","last_became_peered":"2022-06-27T11:47:51.971762+0000","last_unstale":"2022-06-27T11:47:52.368898+0000","last_undegraded":"2022-06-27T11:47:52.368898+0000","last_fullsized":"2022-06-27T11:47:52.368898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T18:15:14.271005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.379879+0000","last_change":"2022-06-27T11:47:52.378294+0000","last_active":"2022-06-27T11:47:52.379879+0000","last_peered":"2022-06-27T11:47:52.379879+0000","last_clean":"2022-06-27T11:47:52.379879+0000","last_became_active":"2022-06-27T11:47:52.377704+0000","last_became_peered":"2022-06-27T11:47:52.377704+0000","last_unstale":"2022-06-27T11:47:52.379879+0000","last_undegraded":"2022-06-27T11:47:52.379879+0000","last_fullsized":"2022-06-27T11:47:52.379879+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:26:34.474081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.364180+0000","last_change":"2022-06-27T11:47:47.807600+0000","last_active":"2022-06-27T11:47:52.364180+0000","last_peered":"2022-06-27T11:47:52.364180+0000","last_clean":"2022-06-27T11:47:52.364180+0000","last_became_active":"2022-06-27T11:47:47.807284+0000","last_became_peered":"2022-06-27T11:47:47.807284+0000","last_unstale":"2022-06-27T11:47:52.364180+0000","last_undegraded":"2022-06-27T11:47:52.364180+0000","last_fullsized":"2022-06-27T11:47:52.364180+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_clean_scrub_stamp":"2022-06-27T11:47:46.780512+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T20:57:06.487027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.362687+0000","last_change":"2022-06-27T11:47:51.980734+0000","last_active":"2022-06-27T11:47:52.362687+0000","last_peered":"2022-06-27T11:47:52.362687+0000","last_clean":"2022-06-27T11:47:52.362687+0000","last_became_active":"2022-06-27T11:47:51.980476+0000","last_became_peered":"2022-06-27T11:47:51.980476+0000","last_unstale":"2022-06-27T11:47:52.362687+0000","last_undegraded":"2022-06-27T11:47:52.362687+0000","last_fullsized":"2022-06-27T11:47:52.362687+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T21:32:16.435472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.385478+0000","last_change":"2022-06-27T11:47:52.379902+0000","last_active":"2022-06-27T11:47:52.385478+0000","last_peered":"2022-06-27T11:47:52.385478+0000","last_clean":"2022-06-27T11:47:52.385478+0000","last_became_active":"2022-06-27T11:47:52.379359+0000","last_became_peered":"2022-06-27T11:47:52.379359+0000","last_unstale":"2022-06-27T11:47:52.385478+0000","last_undegraded":"2022-06-27T11:47:52.385478+0000","last_fullsized":"2022-06-27T11:47:52.385478+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:41:27.884083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.370459+0000","last_change":"2022-06-27T11:47:47.807607+0000","last_active":"2022-06-27T11:47:52.370459+0000","last_peered":"2022-06-27T11:47:52.370459+0000","last_clean":"2022-06-27T11:47:52.370459+0000","last_became_active":"2022-06-27T11:47:47.807293+0000","last_became_peered":"2022-06-27T11:47:47.807293+0000","last_unstale":"2022-06-27T11:47:52.370459+0000","last_undegraded":"2022-06-27T11:47:52.370459+0000","last_fullsized":"2022-06-27T11:47:52.370459+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_clean_scrub_stamp":"2022-06-27T11:47:46.780512+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:00:35.318372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.364133+0000","last_change":"2022-06-27T11:47:51.976055+0000","last_active":"2022-06-27T11:47:52.364133+0000","last_peered":"2022-06-27T11:47:52.364133+0000","last_clean":"2022-06-27T11:47:52.364133+0000","last_became_active":"2022-06-27T11:47:51.975303+0000","last_became_peered":"2022-06-27T11:47:51.975303+0000","last_unstale":"2022-06-27T11:47:52.364133+0000","last_undegraded":"2022-06-27T11:47:52.364133+0000","last_fullsized":"2022-06-27T11:47:52.364133+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:27:32.125525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384750+0000","last_change":"2022-06-27T11:47:52.380775+0000","last_active":"2022-06-27T11:47:52.384750+0000","last_peered":"2022-06-27T11:47:52.384750+0000","last_clean":"2022-06-27T11:47:52.384750+0000","last_became_active":"2022-06-27T11:47:52.380439+0000","last_became_peered":"2022-06-27T11:47:52.380439+0000","last_unstale":"2022-06-27T11:47:52.384750+0000","last_undegraded":"2022-06-27T11:47:52.384750+0000","last_fullsized":"2022-06-27T11:47:52.384750+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:03:44.939329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.370743+0000","last_change":"2022-06-27T11:47:47.792314+0000","last_active":"2022-06-27T11:47:52.370743+0000","last_peered":"2022-06-27T11:47:52.370743+0000","last_clean":"2022-06-27T11:47:52.370743+0000","last_became_active":"2022-06-27T11:47:47.792032+0000","last_became_peered":"2022-06-27T11:47:47.792032+0000","last_unstale":"2022-06-27T11:47:52.370743+0000","last_undegraded":"2022-06-27T11:47:52.370743+0000","last_fullsized":"2022-06-27T11:47:52.370743+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_clean_scrub_stamp":"2022-06-27T11:47:46.780512+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T14:13:03.591775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.369208+0000","last_change":"2022-06-27T11:47:51.978639+0000","last_active":"2022-06-27T11:47:52.369208+0000","last_peered":"2022-06-27T11:47:52.369208+0000","last_clean":"2022-06-27T11:47:52.369208+0000","last_became_active":"2022-06-27T11:47:51.978341+0000","last_became_peered":"2022-06-27T11:47:51.978341+0000","last_unstale":"2022-06-27T11:47:52.369208+0000","last_undegraded":"2022-06-27T11:47:52.369208+0000","last_fullsized":"2022-06-27T11:47:52.369208+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T21:34:48.070558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.381133+0000","last_change":"2022-06-27T11:47:52.373588+0000","last_active":"2022-06-27T11:47:52.381133+0000","last_peered":"2022-06-27T11:47:52.381133+0000","last_clean":"2022-06-27T11:47:52.381133+0000","last_became_active":"2022-06-27T11:47:52.373208+0000","last_became_peered":"2022-06-27T11:47:52.373208+0000","last_unstale":"2022-06-27T11:47:52.381133+0000","last_undegraded":"2022-06-27T11:47:52.381133+0000","last_fullsized":"2022-06-27T11:47:52.381133+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T20:27:28.252147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:53.910019+0000","last_change":"2022-06-27T11:47:51.982051+0000","last_active":"2022-06-27T11:47:53.910019+0000","last_peered":"2022-06-27T11:47:53.910019+0000","last_clean":"2022-06-27T11:47:53.910019+0000","last_became_active":"2022-06-27T11:47:51.981742+0000","last_became_peered":"2022-06-27T11:47:51.981742+0000","last_unstale":"2022-06-27T11:47:53.910019+0000","last_undegraded":"2022-06-27T11:47:53.910019+0000","last_fullsized":"2022-06-27T11:47:53.910019+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:39:56.937086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.380333+0000","last_change":"2022-06-27T11:47:52.378192+0000","last_active":"2022-06-27T11:47:52.380333+0000","last_peered":"2022-06-27T11:47:52.380333+0000","last_clean":"2022-06-27T11:47:52.380333+0000","last_became_active":"2022-06-27T11:47:52.377919+0000","last_became_peered":"2022-06-27T11:47:52.377919+0000","last_unstale":"2022-06-27T11:47:52.380333+0000","last_undegraded":"2022-06-27T11:47:52.380333+0000","last_fullsized":"2022-06-27T11:47:52.380333+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T18:51:35.757778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:57.368777+0000","last_change":"2022-06-27T11:47:51.981101+0000","last_active":"2022-06-27T11:47:57.368777+0000","last_peered":"2022-06-27T11:47:57.368777+0000","last_clean":"2022-06-27T11:47:57.368777+0000","last_became_active":"2022-06-27T11:47:51.980793+0000","last_became_peered":"2022-06-27T11:47:51.980793+0000","last_unstale":"2022-06-27T11:47:57.368777+0000","last_undegraded":"2022-06-27T11:47:57.368777+0000","last_fullsized":"2022-06-27T11:47:57.368777+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:28:10.206148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.372181+0000","last_change":"2022-06-27T11:47:52.370334+0000","last_active":"2022-06-27T11:47:52.372181+0000","last_peered":"2022-06-27T11:47:52.372181+0000","last_clean":"2022-06-27T11:47:52.372181+0000","last_became_active":"2022-06-27T11:47:52.370028+0000","last_became_peered":"2022-06-27T11:47:52.370028+0000","last_unstale":"2022-06-27T11:47:52.372181+0000","last_undegraded":"2022-06-27T11:47:52.372181+0000","last_fullsized":"2022-06-27T11:47:52.372181+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T17:28:12.357757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:57.369040+0000","last_change":"2022-06-27T11:47:51.981186+0000","last_active":"2022-06-27T11:47:57.369040+0000","last_peered":"2022-06-27T11:47:57.369040+0000","last_clean":"2022-06-27T11:47:57.369040+0000","last_became_active":"2022-06-27T11:47:51.980921+0000","last_became_peered":"2022-06-27T11:47:51.980921+0000","last_unstale":"2022-06-27T11:47:57.369040+0000","last_undegraded":"2022-06-27T11:47:57.369040+0000","last_fullsized":"2022-06-27T11:47:57.369040+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:04:40.732299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.387201+0000","last_change":"2022-06-27T11:47:52.382091+0000","last_active":"2022-06-27T11:47:52.387201+0000","last_peered":"2022-06-27T11:47:52.387201+0000","last_clean":"2022-06-27T11:47:52.387201+0000","last_became_active":"2022-06-27T11:47:52.381633+0000","last_became_peered":"2022-06-27T11:47:52.381633+0000","last_unstale":"2022-06-27T11:47:52.387201+0000","last_undegraded":"2022-06-27T11:47:52.387201+0000","last_fullsized":"2022-06-27T11:47:52.387201+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:34:44.030534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:53.910420+0000","last_change":"2022-06-27T11:47:51.972963+0000","last_active":"2022-06-27T11:47:53.910420+0000","last_peered":"2022-06-27T11:47:53.910420+0000","last_clean":"2022-06-27T11:47:53.910420+0000","last_became_active":"2022-06-27T11:47:51.972686+0000","last_became_peered":"2022-06-27T11:47:51.972686+0000","last_unstale":"2022-06-27T11:47:53.910420+0000","last_undegraded":"2022-06-27T11:47:53.910420+0000","last_fullsized":"2022-06-27T11:47:53.910420+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T12:11:38.008538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.385365+0000","last_change":"2022-06-27T11:47:52.383414+0000","last_active":"2022-06-27T11:47:52.385365+0000","last_peered":"2022-06-27T11:47:52.385365+0000","last_clean":"2022-06-27T11:47:52.385365+0000","last_became_active":"2022-06-27T11:47:52.383070+0000","last_became_peered":"2022-06-27T11:47:52.383070+0000","last_unstale":"2022-06-27T11:47:52.385365+0000","last_undegraded":"2022-06-27T11:47:52.385365+0000","last_fullsized":"2022-06-27T11:47:52.385365+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:11:28.410045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"21'4","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:57.370018+0000","last_change":"2022-06-27T11:47:51.971664+0000","last_active":"2022-06-27T11:47:57.370018+0000","last_peered":"2022-06-27T11:47:57.370018+0000","last_clean":"2022-06-27T11:47:57.370018+0000","last_became_active":"2022-06-27T11:47:51.971385+0000","last_became_peered":"2022-06-27T11:47:51.971385+0000","last_unstale":"2022-06-27T11:47:57.370018+0000","last_undegraded":"2022-06-27T11:47:57.370018+0000","last_fullsized":"2022-06-27T11:47:57.370018+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T18:44:38.394388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.379723+0000","last_change":"2022-06-27T11:47:52.377305+0000","last_active":"2022-06-27T11:47:52.379723+0000","last_peered":"2022-06-27T11:47:52.379723+0000","last_clean":"2022-06-27T11:47:52.379723+0000","last_became_active":"2022-06-27T11:47:52.377012+0000","last_became_peered":"2022-06-27T11:47:52.377012+0000","last_unstale":"2022-06-27T11:47:52.379723+0000","last_undegraded":"2022-06-27T11:47:52.379723+0000","last_fullsized":"2022-06-27T11:47:52.379723+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:15:14.505656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.377592+0000","last_change":"2022-06-27T11:47:52.376074+0000","last_active":"2022-06-27T11:47:52.377592+0000","last_peered":"2022-06-27T11:47:52.377592+0000","last_clean":"2022-06-27T11:47:52.377592+0000","last_became_active":"2022-06-27T11:47:52.375836+0000","last_became_peered":"2022-06-27T11:47:52.375836+0000","last_unstale":"2022-06-27T11:47:52.377592+0000","last_undegraded":"2022-06-27T11:47:52.377592+0000","last_fullsized":"2022-06-27T11:47:52.377592+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T18:46:26.900833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384440+0000","last_change":"2022-06-27T11:47:52.376668+0000","last_active":"2022-06-27T11:47:52.384440+0000","last_peered":"2022-06-27T11:47:52.384440+0000","last_clean":"2022-06-27T11:47:52.384440+0000","last_became_active":"2022-06-27T11:47:52.376386+0000","last_became_peered":"2022-06-27T11:47:52.376386+0000","last_unstale":"2022-06-27T11:47:52.384440+0000","last_undegraded":"2022-06-27T11:47:52.384440+0000","last_fullsized":"2022-06-27T11:47:52.384440+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T11:48:35.874748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.379744+0000","last_change":"2022-06-27T11:47:52.377793+0000","last_active":"2022-06-27T11:47:52.379744+0000","last_peered":"2022-06-27T11:47:52.379744+0000","last_clean":"2022-06-27T11:47:52.379744+0000","last_became_active":"2022-06-27T11:47:52.377318+0000","last_became_peered":"2022-06-27T11:47:52.377318+0000","last_unstale":"2022-06-27T11:47:52.379744+0000","last_undegraded":"2022-06-27T11:47:52.379744+0000","last_fullsized":"2022-06-27T11:47:52.379744+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T17:54:22.511301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384925+0000","last_change":"2022-06-27T11:47:52.374638+0000","last_active":"2022-06-27T11:47:52.384925+0000","last_peered":"2022-06-27T11:47:52.384925+0000","last_clean":"2022-06-27T11:47:52.384925+0000","last_became_active":"2022-06-27T11:47:52.374352+0000","last_became_peered":"2022-06-27T11:47:52.374352+0000","last_unstale":"2022-06-27T11:47:52.384925+0000","last_undegraded":"2022-06-27T11:47:52.384925+0000","last_fullsized":"2022-06-27T11:47:52.384925+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:09:59.369369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.375769+0000","last_change":"2022-06-27T11:47:52.373288+0000","last_active":"2022-06-27T11:47:52.375769+0000","last_peered":"2022-06-27T11:47:52.375769+0000","last_clean":"2022-06-27T11:47:52.375769+0000","last_became_active":"2022-06-27T11:47:52.372891+0000","last_became_peered":"2022-06-27T11:47:52.372891+0000","last_unstale":"2022-06-27T11:47:52.375769+0000","last_undegraded":"2022-06-27T11:47:52.375769+0000","last_fullsized":"2022-06-27T11:47:52.375769+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T23:16:08.088450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2590,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":5180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40,"ondisk_log_size":40,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64756,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9016,"kb_used_data":304,"kb_used_omap":0,"kb_used_meta":8704,"kb_avail":94362824,"statfs":{"total":96636764160,"available":96627531776,"internally_reserved":0,"allocated":311296,"data_stored":671231,"data_compressed":32378,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":8912896},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38900000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57499999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8408,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":8192,"kb_avail":94363432,"statfs":{"total":96636764160,"available":96628154368,"internally_reserved":0,"allocated":212992,"data_stored":77836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8388608},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36299999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51900000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8988,"kb_used_data":276,"kb_used_omap":0,"kb_used_meta":8704,"kb_avail":94362852,"statfs":{"total":96636764160,"available":96627560448,"internally_reserved":0,"allocated":282624,"data_stored":669207,"data_compressed":32378,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":8912896},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36099999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52500000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8480,"kb_used_data":216,"kb_used_omap":0,"kb_used_meta":8256,"kb_avail":94363360,"statfs":{"total":96636764160,"available":96628080640,"internally_reserved":0,"allocated":221184,"data_stored":79405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8454144},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.253}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32300000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30299999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.442}]}]},{"osd":2,"up_from":13,"seq":55834574857,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8412,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":8192,"kb_avail":94363428,"statfs":{"total":96636764160,"available":96628150272,"internally_reserved":0,"allocated":217088,"data_stored":78839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8388608},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35999999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30599999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52400000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8476,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":8256,"kb_avail":94363364,"statfs":{"total":96636764160,"available":96628084736,"internally_reserved":0,"allocated":217088,"data_stored":77926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8454144},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24399999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33400000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42299999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]}]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8560,"kb_used_data":232,"kb_used_omap":0,"kb_used_meta":8320,"kb_avail":94363280,"statfs":{"total":96636764160,"available":96627998720,"internally_reserved":0,"allocated":237568,"data_stored":80703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8519680},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.436}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46500000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]}]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8420,"kb_used_data":220,"kb_used_omap":0,"kb_used_meta":8192,"kb_avail":94363420,"statfs":{"total":96636764160,"available":96628142080,"internally_reserved":0,"allocated":225280,"data_stored":79475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8388608},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68500000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.748}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72099999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76500000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84499999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32378,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32378,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-06-27T11:48:12.099 INFO:tasks.mds_thrash.ceph_manager:clean! 2022-06-27T11:48:12.099 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2022-06-27T11:48:12.257 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.250+0000 7f3ebc363700 1 Processor -- start 2022-06-27T11:48:12.258 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.250+0000 7f3ebc363700 1 -- start start 2022-06-27T11:48:12.258 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.251+0000 7f3ebc363700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3eb406e850 0x7f3eb41570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:12.258 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.251+0000 7f3ebc363700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f3eb40c34a0 0x7f3eb40c2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:12.258 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.251+0000 7f3ebc363700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3eb406fba0 con 0x7f3eb406fef0 2022-06-27T11:48:12.259 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.251+0000 7f3ebc363700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3eb41575e0 con 0x7f3eb40c34a0 2022-06-27T11:48:12.259 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.251+0000 7f3ebc363700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb406f5e0 con 0x7f3eb406e850 2022-06-27T11:48:12.260 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.251+0000 7f3eb98fe700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f3eb406fef0 0x7f3eb4069de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:48286/0 (socket says 172.21.15.195:48286) 2022-06-27T11:48:12.260 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.251+0000 7f3eb98fe700 1 -- 172.21.15.195:0/529579005 learned_addr learned my addr 172.21.15.195:0/529579005 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:48:12.260 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.251+0000 7f3eba900700 1 --2- 172.21.15.195:0/529579005 >> v2:172.21.15.202:3300/0 conn(0x7f3eb40c34a0 0x7f3eb40c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:12.261 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.251+0000 7f3eba0ff700 1 --2- 172.21.15.195:0/529579005 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3eb406e850 0x7f3eb41570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:12.261 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.251+0000 7f3eb90fd700 1 -- 172.21.15.195:0/529579005 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1315614791 0 0) 0x7f3eb406fba0 con 0x7f3eb406fef0 2022-06-27T11:48:12.261 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.252+0000 7f3eb90fd700 1 -- 172.21.15.195:0/529579005 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e9c003430 con 0x7f3eb406fef0 2022-06-27T11:48:12.262 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.252+0000 7f3eb90fd700 1 -- 172.21.15.195:0/529579005 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4151785168 0 0) 0x7f3e9c003430 con 0x7f3eb406fef0 2022-06-27T11:48:12.262 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.252+0000 7f3eba0ff700 1 -- 172.21.15.195:0/529579005 >> v2:172.21.15.202:3300/0 conn(0x7f3eb40c34a0 msgr2=0x7f3eb40c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:12.263 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.252+0000 7f3eba0ff700 1 --2- 172.21.15.195:0/529579005 >> v2:172.21.15.202:3300/0 conn(0x7f3eb40c34a0 0x7f3eb40c2920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:12.263 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.252+0000 7f3eba0ff700 1 -- 172.21.15.195:0/529579005 >> v1:172.21.15.195:6789/0 conn(0x7f3eb406fef0 legacy=0x7f3eb4069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:12.264 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.252+0000 7f3eba0ff700 1 -- 172.21.15.195:0/529579005 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eb406f8c0 con 0x7f3eb406e850 2022-06-27T11:48:12.264 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.252+0000 7f3eba900700 1 --2- 172.21.15.195:0/529579005 >> v2:172.21.15.202:3300/0 conn(0x7f3eb40c34a0 0x7f3eb40c2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:48:12.264 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.252+0000 7f3eba0ff700 1 --2- 172.21.15.195:0/529579005 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3eb406e850 0x7f3eb41570a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3ea40097a0 tx=0x7f3ea4009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b2add6529b001e5 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:12.265 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.252+0000 7f3eb90fd700 1 -- 172.21.15.195:0/529579005 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb0001470 con 0x7f3eb406e850 2022-06-27T11:48:12.265 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.252+0000 7f3eb90fd700 1 -- 172.21.15.195:0/529579005 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ea4004030 con 0x7f3eb406e850 2022-06-27T11:48:12.266 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.252+0000 7f3eb90fd700 1 -- 172.21.15.195:0/529579005 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea40043b0 con 0x7f3eb406e850 2022-06-27T11:48:12.266 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.253+0000 7f3ebc363700 1 -- 172.21.15.195:0/529579005 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3eb406e850 msgr2=0x7f3eb41570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:12.267 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.253+0000 7f3ebc363700 1 --2- 172.21.15.195:0/529579005 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3eb406e850 0x7f3eb41570a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3ea40097a0 tx=0x7f3ea4009060 comp rx=0 tx=0).stop 2022-06-27T11:48:12.267 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.253+0000 7f3ebc363700 1 -- 172.21.15.195:0/529579005 shutdown_connections 2022-06-27T11:48:12.267 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.253+0000 7f3ebc363700 1 --2- 172.21.15.195:0/529579005 >> v2:172.21.15.202:3300/0 conn(0x7f3eb40c34a0 0x7f3eb40c2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:12.268 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.253+0000 7f3ebc363700 1 --2- 172.21.15.195:0/529579005 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3eb406e850 0x7f3eb41570a0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:12.268 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.253+0000 7f3ebc363700 1 -- 172.21.15.195:0/529579005 >> 172.21.15.195:0/529579005 conn(0x7f3eb40c6350 msgr2=0x7f3eb40c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:12.269 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.253+0000 7f3ebc363700 1 -- 172.21.15.195:0/529579005 shutdown_connections 2022-06-27T11:48:12.269 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.253+0000 7f3ebc363700 1 -- 172.21.15.195:0/529579005 wait complete. 2022-06-27T11:48:12.270 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.254+0000 7f3ebc363700 1 Processor -- start 2022-06-27T11:48:12.270 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.254+0000 7f3ebc363700 1 -- start start 2022-06-27T11:48:12.270 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.255+0000 7f3ebc363700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3eb406e850 0x7f3eb4151c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:12.271 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.255+0000 7f3ebc363700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3eb40c34a0 0x7f3eb416a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:12.271 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.255+0000 7f3ebc363700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3eb414f6f0 0x7f3eb41535e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:12.273 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.255+0000 7f3ebc363700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb4147e90 con 0x7f3eb406e850 2022-06-27T11:48:12.273 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.255+0000 7f3ebc363700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb4152470 con 0x7f3eb414f6f0 2022-06-27T11:48:12.274 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.255+0000 7f3ebc363700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb4169bf0 con 0x7f3eb40c34a0 2022-06-27T11:48:12.274 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.255+0000 7f3eba0ff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3eb406e850 0x7f3eb4151c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:12.275 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.255+0000 7f3eba0ff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3eb406e850 0x7f3eb4151c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:42102/0 (socket says 172.21.15.195:42102) 2022-06-27T11:48:12.275 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.255+0000 7f3eba0ff700 1 -- 172.21.15.195:0/2410068504 learned_addr learned my addr 172.21.15.195:0/2410068504 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:48:12.276 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.255+0000 7f3eb98fe700 1 --2- 172.21.15.195:0/2410068504 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3eb40c34a0 0x7f3eb416a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:12.276 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.255+0000 7f3eba900700 1 --2- 172.21.15.195:0/2410068504 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3eb414f6f0 0x7f3eb41535e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:12.277 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.255+0000 7f3eba0ff700 1 -- 172.21.15.195:0/2410068504 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3eb40c34a0 msgr2=0x7f3eb416a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:12.277 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.255+0000 7f3eba0ff700 1 --2- 172.21.15.195:0/2410068504 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3eb40c34a0 0x7f3eb416a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:12.277 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.255+0000 7f3eba0ff700 1 -- 172.21.15.195:0/2410068504 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3eb414f6f0 msgr2=0x7f3eb41535e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:12.278 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.255+0000 7f3eba0ff700 1 --2- 172.21.15.195:0/2410068504 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3eb414f6f0 0x7f3eb41535e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:12.278 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.255+0000 7f3eba0ff700 1 -- 172.21.15.195:0/2410068504 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eac009a20 con 0x7f3eb406e850 2022-06-27T11:48:12.279 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.255+0000 7f3eba0ff700 1 --2- 172.21.15.195:0/2410068504 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3eb406e850 0x7f3eb4151c90 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f3ea4003b00 tx=0x7f3ea4015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:12.279 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.255+0000 7f3ea27fc700 1 -- 172.21.15.195:0/2410068504 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea4011650 con 0x7f3eb406e850 2022-06-27T11:48:12.279 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.255+0000 7f3ea27fc700 1 -- 172.21.15.195:0/2410068504 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ea40117d0 con 0x7f3eb406e850 2022-06-27T11:48:12.280 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.255+0000 7f3ebc363700 1 -- 172.21.15.195:0/2410068504 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ea40092e0 con 0x7f3eb406e850 2022-06-27T11:48:12.280 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.255+0000 7f3ea27fc700 1 -- 172.21.15.195:0/2410068504 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea4011b90 con 0x7f3eb406e850 2022-06-27T11:48:12.280 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.255+0000 7f3ebc363700 1 -- 172.21.15.195:0/2410068504 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3eb4169ed0 con 0x7f3eb406e850 2022-06-27T11:48:12.281 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.256+0000 7f3ebc363700 1 -- 172.21.15.195:0/2410068504 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3eb4166b30 con 0x7f3eb406e850 2022-06-27T11:48:12.281 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.257+0000 7f3ea27fc700 1 -- 172.21.15.195:0/2410068504 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f3ea400f070 con 0x7f3eb406e850 2022-06-27T11:48:12.282 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.257+0000 7f3ea27fc700 1 --2- 172.21.15.195:0/2410068504 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3e94063d00 0x7f3e940661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:12.282 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.257+0000 7f3ea27fc700 1 -- 172.21.15.195:0/2410068504 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f3ea4082040 con 0x7f3eb406e850 2022-06-27T11:48:12.282 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.257+0000 7f3eb98fe700 1 --2- 172.21.15.195:0/2410068504 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3e94063d00 0x7f3e940661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:12.283 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.258+0000 7f3eb98fe700 1 --2- 172.21.15.195:0/2410068504 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3e94063d00 0x7f3e940661b0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f3eb4155800 tx=0x7f3eb0001520 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:12.284 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.261+0000 7f3ea27fc700 1 -- 172.21.15.195:0/2410068504 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f3ea4050ce0 con 0x7f3eb406e850 2022-06-27T11:48:12.435 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.435+0000 7f3ebc363700 1 -- 172.21.15.195:0/2410068504 --> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3eb40722c0 con 0x7f3e94063d00 2022-06-27T11:48:12.442 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.443+0000 7f3ea27fc700 1 -- 172.21.15.195:0/2410068504 <== mgr.4098 v2:172.21.15.195:6832/93955 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+284928 (secure 0 0 0) 0x7f3eb40722c0 con 0x7f3e94063d00 2022-06-27T11:48:12.447 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:48:12.450 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.446+0000 7f3ebc363700 1 -- 172.21.15.195:0/2410068504 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3e94063d00 msgr2=0x7f3e940661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:12.450 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.446+0000 7f3ebc363700 1 --2- 172.21.15.195:0/2410068504 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3e94063d00 0x7f3e940661b0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f3eb4155800 tx=0x7f3eb0001520 comp rx=0 tx=0).stop 2022-06-27T11:48:12.450 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.446+0000 7f3ebc363700 1 -- 172.21.15.195:0/2410068504 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3eb406e850 msgr2=0x7f3eb4151c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:12.451 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.446+0000 7f3ebc363700 1 --2- 172.21.15.195:0/2410068504 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3eb406e850 0x7f3eb4151c90 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f3ea4003b00 tx=0x7f3ea4015000 comp rx=0 tx=0).stop 2022-06-27T11:48:12.451 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.446+0000 7f3ebc363700 1 -- 172.21.15.195:0/2410068504 shutdown_connections 2022-06-27T11:48:12.451 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.446+0000 7f3ebc363700 1 --2- 172.21.15.195:0/2410068504 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3e94063d00 0x7f3e940661b0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:12.452 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.446+0000 7f3ebc363700 1 --2- 172.21.15.195:0/2410068504 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3eb406e850 0x7f3eb4151c90 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:12.452 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.446+0000 7f3ebc363700 1 --2- 172.21.15.195:0/2410068504 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3eb414f6f0 0x7f3eb41535e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:12.452 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.446+0000 7f3ebc363700 1 --2- 172.21.15.195:0/2410068504 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3eb40c34a0 0x7f3eb416a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:12.452 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.446+0000 7f3ebc363700 1 -- 172.21.15.195:0/2410068504 >> 172.21.15.195:0/2410068504 conn(0x7f3eb40c6350 msgr2=0x7f3eb40c26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:12.453 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.446+0000 7f3ebc363700 1 -- 172.21.15.195:0/2410068504 shutdown_connections 2022-06-27T11:48:12.453 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.446+0000 7f3ebc363700 1 -- 172.21.15.195:0/2410068504 wait complete. 2022-06-27T11:48:12.453 INFO:teuthology.orchestra.run.smithi195.stderr:dumped all 2022-06-27T11:48:12.458 INFO:teuthology.orchestra.run.smithi195.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2022-06-27T11:48:12.343938+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592958,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":205,"num_write_kb":2431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":210,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":68760,"kb_used_data":1880,"kb_used_omap":0,"kb_used_meta":66816,"kb_avail":754905960,"statfs":{"total":773094113280,"available":773023703040,"internally_reserved":0,"allocated":1925120,"data_stored":1814622,"data_compressed":64756,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":68419584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":260,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002670"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384962+0000","last_change":"2022-06-27T11:47:52.381823+0000","last_active":"2022-06-27T11:47:52.384962+0000","last_peered":"2022-06-27T11:47:52.384962+0000","last_clean":"2022-06-27T11:47:52.384962+0000","last_became_active":"2022-06-27T11:47:52.381551+0000","last_became_peered":"2022-06-27T11:47:52.381551+0000","last_unstale":"2022-06-27T11:47:52.384962+0000","last_undegraded":"2022-06-27T11:47:52.384962+0000","last_fullsized":"2022-06-27T11:47:52.384962+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:17:34.164061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.378135+0000","last_change":"2022-06-27T11:47:52.374939+0000","last_active":"2022-06-27T11:47:52.378135+0000","last_peered":"2022-06-27T11:47:52.378135+0000","last_clean":"2022-06-27T11:47:52.378135+0000","last_became_active":"2022-06-27T11:47:52.374620+0000","last_became_peered":"2022-06-27T11:47:52.374620+0000","last_unstale":"2022-06-27T11:47:52.378135+0000","last_undegraded":"2022-06-27T11:47:52.378135+0000","last_fullsized":"2022-06-27T11:47:52.378135+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:56:27.254098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384127+0000","last_change":"2022-06-27T11:47:52.380802+0000","last_active":"2022-06-27T11:47:52.384127+0000","last_peered":"2022-06-27T11:47:52.384127+0000","last_clean":"2022-06-27T11:47:52.384127+0000","last_became_active":"2022-06-27T11:47:52.380500+0000","last_became_peered":"2022-06-27T11:47:52.380500+0000","last_unstale":"2022-06-27T11:47:52.384127+0000","last_undegraded":"2022-06-27T11:47:52.384127+0000","last_fullsized":"2022-06-27T11:47:52.384127+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:06:42.889598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384114+0000","last_change":"2022-06-27T11:47:52.380943+0000","last_active":"2022-06-27T11:47:52.384114+0000","last_peered":"2022-06-27T11:47:52.384114+0000","last_clean":"2022-06-27T11:47:52.384114+0000","last_became_active":"2022-06-27T11:47:52.380648+0000","last_became_peered":"2022-06-27T11:47:52.380648+0000","last_unstale":"2022-06-27T11:47:52.384114+0000","last_undegraded":"2022-06-27T11:47:52.384114+0000","last_fullsized":"2022-06-27T11:47:52.384114+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:14:19.729605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.386875+0000","last_change":"2022-06-27T11:47:52.386144+0000","last_active":"2022-06-27T11:47:52.386875+0000","last_peered":"2022-06-27T11:47:52.386875+0000","last_clean":"2022-06-27T11:47:52.386875+0000","last_became_active":"2022-06-27T11:47:52.385586+0000","last_became_peered":"2022-06-27T11:47:52.385586+0000","last_unstale":"2022-06-27T11:47:52.386875+0000","last_undegraded":"2022-06-27T11:47:52.386875+0000","last_fullsized":"2022-06-27T11:47:52.386875+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T14:30:03.660749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.387182+0000","last_change":"2022-06-27T11:47:52.381906+0000","last_active":"2022-06-27T11:47:52.387182+0000","last_peered":"2022-06-27T11:47:52.387182+0000","last_clean":"2022-06-27T11:47:52.387182+0000","last_became_active":"2022-06-27T11:47:52.381279+0000","last_became_peered":"2022-06-27T11:47:52.381279+0000","last_unstale":"2022-06-27T11:47:52.387182+0000","last_undegraded":"2022-06-27T11:47:52.387182+0000","last_fullsized":"2022-06-27T11:47:52.387182+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T18:41:03.991762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.383137+0000","last_change":"2022-06-27T11:47:52.377128+0000","last_active":"2022-06-27T11:47:52.383137+0000","last_peered":"2022-06-27T11:47:52.383137+0000","last_clean":"2022-06-27T11:47:52.383137+0000","last_became_active":"2022-06-27T11:47:52.376499+0000","last_became_peered":"2022-06-27T11:47:52.376499+0000","last_unstale":"2022-06-27T11:47:52.383137+0000","last_undegraded":"2022-06-27T11:47:52.383137+0000","last_fullsized":"2022-06-27T11:47:52.383137+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:00:13.208145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.378171+0000","last_change":"2022-06-27T11:47:52.374169+0000","last_active":"2022-06-27T11:47:52.378171+0000","last_peered":"2022-06-27T11:47:52.378171+0000","last_clean":"2022-06-27T11:47:52.378171+0000","last_became_active":"2022-06-27T11:47:52.373917+0000","last_became_peered":"2022-06-27T11:47:52.373917+0000","last_unstale":"2022-06-27T11:47:52.378171+0000","last_undegraded":"2022-06-27T11:47:52.378171+0000","last_fullsized":"2022-06-27T11:47:52.378171+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:43:52.741254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384769+0000","last_change":"2022-06-27T11:47:52.382039+0000","last_active":"2022-06-27T11:47:52.384769+0000","last_peered":"2022-06-27T11:47:52.384769+0000","last_clean":"2022-06-27T11:47:52.384769+0000","last_became_active":"2022-06-27T11:47:52.381202+0000","last_became_peered":"2022-06-27T11:47:52.381202+0000","last_unstale":"2022-06-27T11:47:52.384769+0000","last_undegraded":"2022-06-27T11:47:52.384769+0000","last_fullsized":"2022-06-27T11:47:52.384769+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:13:02.553823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.383894+0000","last_change":"2022-06-27T11:47:52.380839+0000","last_active":"2022-06-27T11:47:52.383894+0000","last_peered":"2022-06-27T11:47:52.383894+0000","last_clean":"2022-06-27T11:47:52.383894+0000","last_became_active":"2022-06-27T11:47:52.380527+0000","last_became_peered":"2022-06-27T11:47:52.380527+0000","last_unstale":"2022-06-27T11:47:52.383894+0000","last_undegraded":"2022-06-27T11:47:52.383894+0000","last_fullsized":"2022-06-27T11:47:52.383894+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T23:18:07.086598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.374783+0000","last_change":"2022-06-27T11:47:52.371267+0000","last_active":"2022-06-27T11:47:52.374783+0000","last_peered":"2022-06-27T11:47:52.374783+0000","last_clean":"2022-06-27T11:47:52.374783+0000","last_became_active":"2022-06-27T11:47:52.370921+0000","last_became_peered":"2022-06-27T11:47:52.370921+0000","last_unstale":"2022-06-27T11:47:52.374783+0000","last_undegraded":"2022-06-27T11:47:52.374783+0000","last_fullsized":"2022-06-27T11:47:52.374783+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T19:22:39.648715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.378994+0000","last_change":"2022-06-27T11:47:52.375381+0000","last_active":"2022-06-27T11:47:52.378994+0000","last_peered":"2022-06-27T11:47:52.378994+0000","last_clean":"2022-06-27T11:47:52.378994+0000","last_became_active":"2022-06-27T11:47:52.375089+0000","last_became_peered":"2022-06-27T11:47:52.375089+0000","last_unstale":"2022-06-27T11:47:52.378994+0000","last_undegraded":"2022-06-27T11:47:52.378994+0000","last_fullsized":"2022-06-27T11:47:52.378994+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:43:14.830424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384993+0000","last_change":"2022-06-27T11:47:52.381690+0000","last_active":"2022-06-27T11:47:52.384993+0000","last_peered":"2022-06-27T11:47:52.384993+0000","last_clean":"2022-06-27T11:47:52.384993+0000","last_became_active":"2022-06-27T11:47:52.381472+0000","last_became_peered":"2022-06-27T11:47:52.381472+0000","last_unstale":"2022-06-27T11:47:52.384993+0000","last_undegraded":"2022-06-27T11:47:52.384993+0000","last_fullsized":"2022-06-27T11:47:52.384993+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:14:56.919536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.378838+0000","last_change":"2022-06-27T11:47:52.376063+0000","last_active":"2022-06-27T11:47:52.378838+0000","last_peered":"2022-06-27T11:47:52.378838+0000","last_clean":"2022-06-27T11:47:52.378838+0000","last_became_active":"2022-06-27T11:47:52.375778+0000","last_became_peered":"2022-06-27T11:47:52.375778+0000","last_unstale":"2022-06-27T11:47:52.378838+0000","last_undegraded":"2022-06-27T11:47:52.378838+0000","last_fullsized":"2022-06-27T11:47:52.378838+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T23:16:52.221967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.383160+0000","last_change":"2022-06-27T11:47:52.377297+0000","last_active":"2022-06-27T11:47:52.383160+0000","last_peered":"2022-06-27T11:47:52.383160+0000","last_clean":"2022-06-27T11:47:52.383160+0000","last_became_active":"2022-06-27T11:47:52.376855+0000","last_became_peered":"2022-06-27T11:47:52.376855+0000","last_unstale":"2022-06-27T11:47:52.383160+0000","last_undegraded":"2022-06-27T11:47:52.383160+0000","last_fullsized":"2022-06-27T11:47:52.383160+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:33:39.990522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.387168+0000","last_change":"2022-06-27T11:47:52.381821+0000","last_active":"2022-06-27T11:47:52.387168+0000","last_peered":"2022-06-27T11:47:52.387168+0000","last_clean":"2022-06-27T11:47:52.387168+0000","last_became_active":"2022-06-27T11:47:52.381548+0000","last_became_peered":"2022-06-27T11:47:52.381548+0000","last_unstale":"2022-06-27T11:47:52.387168+0000","last_undegraded":"2022-06-27T11:47:52.387168+0000","last_fullsized":"2022-06-27T11:47:52.387168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T23:14:53.320016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384753+0000","last_change":"2022-06-27T11:47:52.381811+0000","last_active":"2022-06-27T11:47:52.384753+0000","last_peered":"2022-06-27T11:47:52.384753+0000","last_clean":"2022-06-27T11:47:52.384753+0000","last_became_active":"2022-06-27T11:47:52.381554+0000","last_became_peered":"2022-06-27T11:47:52.381554+0000","last_unstale":"2022-06-27T11:47:52.384753+0000","last_undegraded":"2022-06-27T11:47:52.384753+0000","last_fullsized":"2022-06-27T11:47:52.384753+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T14:19:38.519380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.387218+0000","last_change":"2022-06-27T11:47:52.382952+0000","last_active":"2022-06-27T11:47:52.387218+0000","last_peered":"2022-06-27T11:47:52.387218+0000","last_clean":"2022-06-27T11:47:52.387218+0000","last_became_active":"2022-06-27T11:47:52.382691+0000","last_became_peered":"2022-06-27T11:47:52.382691+0000","last_unstale":"2022-06-27T11:47:52.387218+0000","last_undegraded":"2022-06-27T11:47:52.387218+0000","last_fullsized":"2022-06-27T11:47:52.387218+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:28:54.743970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.383682+0000","last_change":"2022-06-27T11:47:52.379083+0000","last_active":"2022-06-27T11:47:52.383682+0000","last_peered":"2022-06-27T11:47:52.383682+0000","last_clean":"2022-06-27T11:47:52.383682+0000","last_became_active":"2022-06-27T11:47:52.378724+0000","last_became_peered":"2022-06-27T11:47:52.378724+0000","last_unstale":"2022-06-27T11:47:52.383682+0000","last_undegraded":"2022-06-27T11:47:52.383682+0000","last_fullsized":"2022-06-27T11:47:52.383682+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T21:25:07.786935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.383275+0000","last_change":"2022-06-27T11:47:52.377947+0000","last_active":"2022-06-27T11:47:52.383275+0000","last_peered":"2022-06-27T11:47:52.383275+0000","last_clean":"2022-06-27T11:47:52.383275+0000","last_became_active":"2022-06-27T11:47:52.377652+0000","last_became_peered":"2022-06-27T11:47:52.377652+0000","last_unstale":"2022-06-27T11:47:52.383275+0000","last_undegraded":"2022-06-27T11:47:52.383275+0000","last_fullsized":"2022-06-27T11:47:52.383275+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T20:48:28.642052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384444+0000","last_change":"2022-06-27T11:47:52.381920+0000","last_active":"2022-06-27T11:47:52.384444+0000","last_peered":"2022-06-27T11:47:52.384444+0000","last_clean":"2022-06-27T11:47:52.384444+0000","last_became_active":"2022-06-27T11:47:52.381477+0000","last_became_peered":"2022-06-27T11:47:52.381477+0000","last_unstale":"2022-06-27T11:47:52.384444+0000","last_undegraded":"2022-06-27T11:47:52.384444+0000","last_fullsized":"2022-06-27T11:47:52.384444+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T20:42:25.529336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.386345+0000","last_change":"2022-06-27T11:47:52.385231+0000","last_active":"2022-06-27T11:47:52.386345+0000","last_peered":"2022-06-27T11:47:52.386345+0000","last_clean":"2022-06-27T11:47:52.386345+0000","last_became_active":"2022-06-27T11:47:52.384972+0000","last_became_peered":"2022-06-27T11:47:52.384972+0000","last_unstale":"2022-06-27T11:47:52.386345+0000","last_undegraded":"2022-06-27T11:47:52.386345+0000","last_fullsized":"2022-06-27T11:47:52.386345+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T17:37:26.084788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.385330+0000","last_change":"2022-06-27T11:47:52.381358+0000","last_active":"2022-06-27T11:47:52.385330+0000","last_peered":"2022-06-27T11:47:52.385330+0000","last_clean":"2022-06-27T11:47:52.385330+0000","last_became_active":"2022-06-27T11:47:52.380997+0000","last_became_peered":"2022-06-27T11:47:52.380997+0000","last_unstale":"2022-06-27T11:47:52.385330+0000","last_undegraded":"2022-06-27T11:47:52.385330+0000","last_fullsized":"2022-06-27T11:47:52.385330+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T21:44:16.061234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384432+0000","last_change":"2022-06-27T11:47:52.381899+0000","last_active":"2022-06-27T11:47:52.384432+0000","last_peered":"2022-06-27T11:47:52.384432+0000","last_clean":"2022-06-27T11:47:52.384432+0000","last_became_active":"2022-06-27T11:47:52.381645+0000","last_became_peered":"2022-06-27T11:47:52.381645+0000","last_unstale":"2022-06-27T11:47:52.384432+0000","last_undegraded":"2022-06-27T11:47:52.384432+0000","last_fullsized":"2022-06-27T11:47:52.384432+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:05:59.742023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.385053+0000","last_change":"2022-06-27T11:47:52.378736+0000","last_active":"2022-06-27T11:47:52.385053+0000","last_peered":"2022-06-27T11:47:52.385053+0000","last_clean":"2022-06-27T11:47:52.385053+0000","last_became_active":"2022-06-27T11:47:52.378475+0000","last_became_peered":"2022-06-27T11:47:52.378475+0000","last_unstale":"2022-06-27T11:47:52.385053+0000","last_undegraded":"2022-06-27T11:47:52.385053+0000","last_fullsized":"2022-06-27T11:47:52.385053+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T12:42:12.528716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384922+0000","last_change":"2022-06-27T11:47:52.378044+0000","last_active":"2022-06-27T11:47:52.384922+0000","last_peered":"2022-06-27T11:47:52.384922+0000","last_clean":"2022-06-27T11:47:52.384922+0000","last_became_active":"2022-06-27T11:47:52.377791+0000","last_became_peered":"2022-06-27T11:47:52.377791+0000","last_unstale":"2022-06-27T11:47:52.384922+0000","last_undegraded":"2022-06-27T11:47:52.384922+0000","last_fullsized":"2022-06-27T11:47:52.384922+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:21:45.615162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384115+0000","last_change":"2022-06-27T11:47:52.382267+0000","last_active":"2022-06-27T11:47:52.384115+0000","last_peered":"2022-06-27T11:47:52.384115+0000","last_clean":"2022-06-27T11:47:52.384115+0000","last_became_active":"2022-06-27T11:47:52.381985+0000","last_became_peered":"2022-06-27T11:47:52.381985+0000","last_unstale":"2022-06-27T11:47:52.384115+0000","last_undegraded":"2022-06-27T11:47:52.384115+0000","last_fullsized":"2022-06-27T11:47:52.384115+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:34:09.509759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:53.909338+0000","last_change":"2022-06-27T11:47:51.982183+0000","last_active":"2022-06-27T11:47:53.909338+0000","last_peered":"2022-06-27T11:47:53.909338+0000","last_clean":"2022-06-27T11:47:53.909338+0000","last_became_active":"2022-06-27T11:47:51.981339+0000","last_became_peered":"2022-06-27T11:47:51.981339+0000","last_unstale":"2022-06-27T11:47:53.909338+0000","last_undegraded":"2022-06-27T11:47:53.909338+0000","last_fullsized":"2022-06-27T11:47:53.909338+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:08:09.223947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384452+0000","last_change":"2022-06-27T11:47:52.377197+0000","last_active":"2022-06-27T11:47:52.384452+0000","last_peered":"2022-06-27T11:47:52.384452+0000","last_clean":"2022-06-27T11:47:52.384452+0000","last_became_active":"2022-06-27T11:47:52.376925+0000","last_became_peered":"2022-06-27T11:47:52.376925+0000","last_unstale":"2022-06-27T11:47:52.384452+0000","last_undegraded":"2022-06-27T11:47:52.384452+0000","last_fullsized":"2022-06-27T11:47:52.384452+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:39:27.103978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.364221+0000","last_change":"2022-06-27T11:47:51.981074+0000","last_active":"2022-06-27T11:47:52.364221+0000","last_peered":"2022-06-27T11:47:52.364221+0000","last_clean":"2022-06-27T11:47:52.364221+0000","last_became_active":"2022-06-27T11:47:51.980636+0000","last_became_peered":"2022-06-27T11:47:51.980636+0000","last_unstale":"2022-06-27T11:47:52.364221+0000","last_undegraded":"2022-06-27T11:47:52.364221+0000","last_fullsized":"2022-06-27T11:47:52.364221+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T23:32:52.850801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384380+0000","last_change":"2022-06-27T11:47:52.378415+0000","last_active":"2022-06-27T11:47:52.384380+0000","last_peered":"2022-06-27T11:47:52.384380+0000","last_clean":"2022-06-27T11:47:52.384380+0000","last_became_active":"2022-06-27T11:47:52.378123+0000","last_became_peered":"2022-06-27T11:47:52.378123+0000","last_unstale":"2022-06-27T11:47:52.384380+0000","last_undegraded":"2022-06-27T11:47:52.384380+0000","last_fullsized":"2022-06-27T11:47:52.384380+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T11:54:37.127828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.369137+0000","last_change":"2022-06-27T11:47:51.975684+0000","last_active":"2022-06-27T11:47:52.369137+0000","last_peered":"2022-06-27T11:47:52.369137+0000","last_clean":"2022-06-27T11:47:52.369137+0000","last_became_active":"2022-06-27T11:47:51.974902+0000","last_became_peered":"2022-06-27T11:47:51.974902+0000","last_unstale":"2022-06-27T11:47:52.369137+0000","last_undegraded":"2022-06-27T11:47:52.369137+0000","last_fullsized":"2022-06-27T11:47:52.369137+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:51:45.780798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.378008+0000","last_change":"2022-06-27T11:47:52.376089+0000","last_active":"2022-06-27T11:47:52.378008+0000","last_peered":"2022-06-27T11:47:52.378008+0000","last_clean":"2022-06-27T11:47:52.378008+0000","last_became_active":"2022-06-27T11:47:52.375780+0000","last_became_peered":"2022-06-27T11:47:52.375780+0000","last_unstale":"2022-06-27T11:47:52.378008+0000","last_undegraded":"2022-06-27T11:47:52.378008+0000","last_fullsized":"2022-06-27T11:47:52.378008+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T14:17:53.942880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.371313+0000","last_change":"2022-06-27T11:47:51.974071+0000","last_active":"2022-06-27T11:47:52.371313+0000","last_peered":"2022-06-27T11:47:52.371313+0000","last_clean":"2022-06-27T11:47:52.371313+0000","last_became_active":"2022-06-27T11:47:51.973682+0000","last_became_peered":"2022-06-27T11:47:51.973682+0000","last_unstale":"2022-06-27T11:47:52.371313+0000","last_undegraded":"2022-06-27T11:47:52.371313+0000","last_fullsized":"2022-06-27T11:47:52.371313+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T17:40:03.705083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384841+0000","last_change":"2022-06-27T11:47:52.377716+0000","last_active":"2022-06-27T11:47:52.384841+0000","last_peered":"2022-06-27T11:47:52.384841+0000","last_clean":"2022-06-27T11:47:52.384841+0000","last_became_active":"2022-06-27T11:47:52.377464+0000","last_became_peered":"2022-06-27T11:47:52.377464+0000","last_unstale":"2022-06-27T11:47:52.384841+0000","last_undegraded":"2022-06-27T11:47:52.384841+0000","last_fullsized":"2022-06-27T11:47:52.384841+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:39:36.234204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.368885+0000","last_change":"2022-06-27T11:47:51.975895+0000","last_active":"2022-06-27T11:47:52.368885+0000","last_peered":"2022-06-27T11:47:52.368885+0000","last_clean":"2022-06-27T11:47:52.368885+0000","last_became_active":"2022-06-27T11:47:51.975601+0000","last_became_peered":"2022-06-27T11:47:51.975601+0000","last_unstale":"2022-06-27T11:47:52.368885+0000","last_undegraded":"2022-06-27T11:47:52.368885+0000","last_fullsized":"2022-06-27T11:47:52.368885+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:43:55.833399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.385349+0000","last_change":"2022-06-27T11:47:52.383063+0000","last_active":"2022-06-27T11:47:52.385349+0000","last_peered":"2022-06-27T11:47:52.385349+0000","last_clean":"2022-06-27T11:47:52.385349+0000","last_became_active":"2022-06-27T11:47:52.382390+0000","last_became_peered":"2022-06-27T11:47:52.382390+0000","last_unstale":"2022-06-27T11:47:52.385349+0000","last_undegraded":"2022-06-27T11:47:52.385349+0000","last_fullsized":"2022-06-27T11:47:52.385349+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:40:10.929542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"21'3","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:57.367279+0000","last_change":"2022-06-27T11:47:51.980627+0000","last_active":"2022-06-27T11:47:57.367279+0000","last_peered":"2022-06-27T11:47:57.367279+0000","last_clean":"2022-06-27T11:47:57.367279+0000","last_became_active":"2022-06-27T11:47:51.980265+0000","last_became_peered":"2022-06-27T11:47:51.980265+0000","last_unstale":"2022-06-27T11:47:57.367279+0000","last_undegraded":"2022-06-27T11:47:57.367279+0000","last_fullsized":"2022-06-27T11:47:57.367279+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T14:18:32.599085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384584+0000","last_change":"2022-06-27T11:47:52.377475+0000","last_active":"2022-06-27T11:47:52.384584+0000","last_peered":"2022-06-27T11:47:52.384584+0000","last_clean":"2022-06-27T11:47:52.384584+0000","last_became_active":"2022-06-27T11:47:52.377220+0000","last_became_peered":"2022-06-27T11:47:52.377220+0000","last_unstale":"2022-06-27T11:47:52.384584+0000","last_undegraded":"2022-06-27T11:47:52.384584+0000","last_fullsized":"2022-06-27T11:47:52.384584+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T23:27:26.525688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"21'7","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:57.370253+0000","last_change":"2022-06-27T11:47:51.978195+0000","last_active":"2022-06-27T11:47:57.370253+0000","last_peered":"2022-06-27T11:47:57.370253+0000","last_clean":"2022-06-27T11:47:57.370253+0000","last_became_active":"2022-06-27T11:47:51.974673+0000","last_became_peered":"2022-06-27T11:47:51.974673+0000","last_unstale":"2022-06-27T11:47:57.370253+0000","last_undegraded":"2022-06-27T11:47:57.370253+0000","last_fullsized":"2022-06-27T11:47:57.370253+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T18:09:43.518644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1808,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.382204+0000","last_change":"2022-06-27T11:47:52.378740+0000","last_active":"2022-06-27T11:47:52.382204+0000","last_peered":"2022-06-27T11:47:52.382204+0000","last_clean":"2022-06-27T11:47:52.382204+0000","last_became_active":"2022-06-27T11:47:52.378451+0000","last_became_peered":"2022-06-27T11:47:52.378451+0000","last_unstale":"2022-06-27T11:47:52.382204+0000","last_undegraded":"2022-06-27T11:47:52.382204+0000","last_fullsized":"2022-06-27T11:47:52.382204+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:04:47.829409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.13","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:57.368186+0000","last_change":"2022-06-27T11:47:51.983039+0000","last_active":"2022-06-27T11:47:57.368186+0000","last_peered":"2022-06-27T11:47:57.368186+0000","last_clean":"2022-06-27T11:47:57.368186+0000","last_became_active":"2022-06-27T11:47:51.982683+0000","last_became_peered":"2022-06-27T11:47:51.982683+0000","last_unstale":"2022-06-27T11:47:57.368186+0000","last_undegraded":"2022-06-27T11:47:57.368186+0000","last_fullsized":"2022-06-27T11:47:57.368186+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T17:10:16.876753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384742+0000","last_change":"2022-06-27T11:47:52.374884+0000","last_active":"2022-06-27T11:47:52.384742+0000","last_peered":"2022-06-27T11:47:52.384742+0000","last_clean":"2022-06-27T11:47:52.384742+0000","last_became_active":"2022-06-27T11:47:52.374632+0000","last_became_peered":"2022-06-27T11:47:52.374632+0000","last_unstale":"2022-06-27T11:47:52.384742+0000","last_undegraded":"2022-06-27T11:47:52.384742+0000","last_fullsized":"2022-06-27T11:47:52.384742+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T21:33:54.724089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.374167+0000","last_change":"2022-06-27T11:47:51.976772+0000","last_active":"2022-06-27T11:47:52.374167+0000","last_peered":"2022-06-27T11:47:52.374167+0000","last_clean":"2022-06-27T11:47:52.374167+0000","last_became_active":"2022-06-27T11:47:51.976475+0000","last_became_peered":"2022-06-27T11:47:51.976475+0000","last_unstale":"2022-06-27T11:47:52.374167+0000","last_undegraded":"2022-06-27T11:47:52.374167+0000","last_fullsized":"2022-06-27T11:47:52.374167+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T23:39:10.919159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.377943+0000","last_change":"2022-06-27T11:47:52.373748+0000","last_active":"2022-06-27T11:47:52.377943+0000","last_peered":"2022-06-27T11:47:52.377943+0000","last_clean":"2022-06-27T11:47:52.377943+0000","last_became_active":"2022-06-27T11:47:52.373459+0000","last_became_peered":"2022-06-27T11:47:52.373459+0000","last_unstale":"2022-06-27T11:47:52.377943+0000","last_undegraded":"2022-06-27T11:47:52.377943+0000","last_fullsized":"2022-06-27T11:47:52.377943+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T18:27:22.254416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.367083+0000","last_change":"2022-06-27T11:47:51.982926+0000","last_active":"2022-06-27T11:47:52.367083+0000","last_peered":"2022-06-27T11:47:52.367083+0000","last_clean":"2022-06-27T11:47:52.367083+0000","last_became_active":"2022-06-27T11:47:51.982638+0000","last_became_peered":"2022-06-27T11:47:51.982638+0000","last_unstale":"2022-06-27T11:47:52.367083+0000","last_undegraded":"2022-06-27T11:47:52.367083+0000","last_fullsized":"2022-06-27T11:47:52.367083+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:34:04.834822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384791+0000","last_change":"2022-06-27T11:47:52.381623+0000","last_active":"2022-06-27T11:47:52.384791+0000","last_peered":"2022-06-27T11:47:52.384791+0000","last_clean":"2022-06-27T11:47:52.384791+0000","last_became_active":"2022-06-27T11:47:52.381258+0000","last_became_peered":"2022-06-27T11:47:52.381258+0000","last_unstale":"2022-06-27T11:47:52.384791+0000","last_undegraded":"2022-06-27T11:47:52.384791+0000","last_fullsized":"2022-06-27T11:47:52.384791+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:39:07.735224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"21'8","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:57.369384+0000","last_change":"2022-06-27T11:47:51.973125+0000","last_active":"2022-06-27T11:47:57.369384+0000","last_peered":"2022-06-27T11:47:57.369384+0000","last_clean":"2022-06-27T11:47:57.369384+0000","last_became_active":"2022-06-27T11:47:51.972806+0000","last_became_peered":"2022-06-27T11:47:51.972806+0000","last_unstale":"2022-06-27T11:47:57.369384+0000","last_undegraded":"2022-06-27T11:47:57.369384+0000","last_fullsized":"2022-06-27T11:47:57.369384+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:49:23.881991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384422+0000","last_change":"2022-06-27T11:47:52.376101+0000","last_active":"2022-06-27T11:47:52.384422+0000","last_peered":"2022-06-27T11:47:52.384422+0000","last_clean":"2022-06-27T11:47:52.384422+0000","last_became_active":"2022-06-27T11:47:52.375774+0000","last_became_peered":"2022-06-27T11:47:52.375774+0000","last_unstale":"2022-06-27T11:47:52.384422+0000","last_undegraded":"2022-06-27T11:47:52.384422+0000","last_fullsized":"2022-06-27T11:47:52.384422+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T18:21:13.049096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.f","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:53.492124+0000","last_change":"2022-06-27T11:47:51.981981+0000","last_active":"2022-06-27T11:47:53.492124+0000","last_peered":"2022-06-27T11:47:53.492124+0000","last_clean":"2022-06-27T11:47:53.492124+0000","last_became_active":"2022-06-27T11:47:51.981749+0000","last_became_peered":"2022-06-27T11:47:51.981749+0000","last_unstale":"2022-06-27T11:47:53.492124+0000","last_undegraded":"2022-06-27T11:47:53.492124+0000","last_fullsized":"2022-06-27T11:47:53.492124+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:28:52.457719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.378190+0000","last_change":"2022-06-27T11:47:52.374223+0000","last_active":"2022-06-27T11:47:52.378190+0000","last_peered":"2022-06-27T11:47:52.378190+0000","last_clean":"2022-06-27T11:47:52.378190+0000","last_became_active":"2022-06-27T11:47:52.373932+0000","last_became_peered":"2022-06-27T11:47:52.373932+0000","last_unstale":"2022-06-27T11:47:52.378190+0000","last_undegraded":"2022-06-27T11:47:52.378190+0000","last_fullsized":"2022-06-27T11:47:52.378190+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:48:33.254458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:57.368190+0000","last_change":"2022-06-27T11:47:51.982895+0000","last_active":"2022-06-27T11:47:57.368190+0000","last_peered":"2022-06-27T11:47:57.368190+0000","last_clean":"2022-06-27T11:47:57.368190+0000","last_became_active":"2022-06-27T11:47:51.982619+0000","last_became_peered":"2022-06-27T11:47:51.982619+0000","last_unstale":"2022-06-27T11:47:57.368190+0000","last_undegraded":"2022-06-27T11:47:57.368190+0000","last_fullsized":"2022-06-27T11:47:57.368190+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:22:01.241112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.377169+0000","last_change":"2022-06-27T11:47:52.375225+0000","last_active":"2022-06-27T11:47:52.377169+0000","last_peered":"2022-06-27T11:47:52.377169+0000","last_clean":"2022-06-27T11:47:52.377169+0000","last_became_active":"2022-06-27T11:47:52.374925+0000","last_became_peered":"2022-06-27T11:47:52.374925+0000","last_unstale":"2022-06-27T11:47:52.377169+0000","last_undegraded":"2022-06-27T11:47:52.377169+0000","last_fullsized":"2022-06-27T11:47:52.377169+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:54:31.945902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:53.492085+0000","last_change":"2022-06-27T11:47:51.975202+0000","last_active":"2022-06-27T11:47:53.492085+0000","last_peered":"2022-06-27T11:47:53.492085+0000","last_clean":"2022-06-27T11:47:53.492085+0000","last_became_active":"2022-06-27T11:47:51.974968+0000","last_became_peered":"2022-06-27T11:47:51.974968+0000","last_unstale":"2022-06-27T11:47:53.492085+0000","last_undegraded":"2022-06-27T11:47:53.492085+0000","last_fullsized":"2022-06-27T11:47:53.492085+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T21:16:30.362248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.378431+0000","last_change":"2022-06-27T11:47:52.377794+0000","last_active":"2022-06-27T11:47:52.378431+0000","last_peered":"2022-06-27T11:47:52.378431+0000","last_clean":"2022-06-27T11:47:52.378431+0000","last_became_active":"2022-06-27T11:47:52.377387+0000","last_became_peered":"2022-06-27T11:47:52.377387+0000","last_unstale":"2022-06-27T11:47:52.378431+0000","last_undegraded":"2022-06-27T11:47:52.378431+0000","last_fullsized":"2022-06-27T11:47:52.378431+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T17:00:00.924967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.368792+0000","last_change":"2022-06-27T11:47:51.976575+0000","last_active":"2022-06-27T11:47:52.368792+0000","last_peered":"2022-06-27T11:47:52.368792+0000","last_clean":"2022-06-27T11:47:52.368792+0000","last_became_active":"2022-06-27T11:47:51.976300+0000","last_became_peered":"2022-06-27T11:47:51.976300+0000","last_unstale":"2022-06-27T11:47:52.368792+0000","last_undegraded":"2022-06-27T11:47:52.368792+0000","last_fullsized":"2022-06-27T11:47:52.368792+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:08:46.067375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.387174+0000","last_change":"2022-06-27T11:47:52.386298+0000","last_active":"2022-06-27T11:47:52.387174+0000","last_peered":"2022-06-27T11:47:52.387174+0000","last_clean":"2022-06-27T11:47:52.387174+0000","last_became_active":"2022-06-27T11:47:52.385918+0000","last_became_peered":"2022-06-27T11:47:52.385918+0000","last_unstale":"2022-06-27T11:47:52.387174+0000","last_undegraded":"2022-06-27T11:47:52.387174+0000","last_fullsized":"2022-06-27T11:47:52.387174+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:21:51.780895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:57.364776+0000","last_change":"2022-06-27T11:47:51.973817+0000","last_active":"2022-06-27T11:47:57.364776+0000","last_peered":"2022-06-27T11:47:57.364776+0000","last_clean":"2022-06-27T11:47:57.364776+0000","last_became_active":"2022-06-27T11:47:51.973515+0000","last_became_peered":"2022-06-27T11:47:51.973515+0000","last_unstale":"2022-06-27T11:47:57.364776+0000","last_undegraded":"2022-06-27T11:47:57.364776+0000","last_fullsized":"2022-06-27T11:47:57.364776+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T21:01:45.170590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.374582+0000","last_change":"2022-06-27T11:47:52.371141+0000","last_active":"2022-06-27T11:47:52.374582+0000","last_peered":"2022-06-27T11:47:52.374582+0000","last_clean":"2022-06-27T11:47:52.374582+0000","last_became_active":"2022-06-27T11:47:52.370796+0000","last_became_peered":"2022-06-27T11:47:52.370796+0000","last_unstale":"2022-06-27T11:47:52.374582+0000","last_undegraded":"2022-06-27T11:47:52.374582+0000","last_fullsized":"2022-06-27T11:47:52.374582+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T23:16:39.266615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.369465+0000","last_change":"2022-06-27T11:47:51.983071+0000","last_active":"2022-06-27T11:47:52.369465+0000","last_peered":"2022-06-27T11:47:52.369465+0000","last_clean":"2022-06-27T11:47:52.369465+0000","last_became_active":"2022-06-27T11:47:51.982756+0000","last_became_peered":"2022-06-27T11:47:51.982756+0000","last_unstale":"2022-06-27T11:47:52.369465+0000","last_undegraded":"2022-06-27T11:47:52.369465+0000","last_fullsized":"2022-06-27T11:47:52.369465+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T17:14:00.737599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.387222+0000","last_change":"2022-06-27T11:47:52.382943+0000","last_active":"2022-06-27T11:47:52.387222+0000","last_peered":"2022-06-27T11:47:52.387222+0000","last_clean":"2022-06-27T11:47:52.387222+0000","last_became_active":"2022-06-27T11:47:52.382625+0000","last_became_peered":"2022-06-27T11:47:52.382625+0000","last_unstale":"2022-06-27T11:47:52.387222+0000","last_undegraded":"2022-06-27T11:47:52.387222+0000","last_fullsized":"2022-06-27T11:47:52.387222+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:01:18.570554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.362681+0000","last_change":"2022-06-27T11:47:51.978856+0000","last_active":"2022-06-27T11:47:52.362681+0000","last_peered":"2022-06-27T11:47:52.362681+0000","last_clean":"2022-06-27T11:47:52.362681+0000","last_became_active":"2022-06-27T11:47:51.978541+0000","last_became_peered":"2022-06-27T11:47:51.978541+0000","last_unstale":"2022-06-27T11:47:52.362681+0000","last_undegraded":"2022-06-27T11:47:52.362681+0000","last_fullsized":"2022-06-27T11:47:52.362681+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:36:24.190996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.378203+0000","last_change":"2022-06-27T11:47:52.376925+0000","last_active":"2022-06-27T11:47:52.378203+0000","last_peered":"2022-06-27T11:47:52.378203+0000","last_clean":"2022-06-27T11:47:52.378203+0000","last_became_active":"2022-06-27T11:47:52.376646+0000","last_became_peered":"2022-06-27T11:47:52.376646+0000","last_unstale":"2022-06-27T11:47:52.378203+0000","last_undegraded":"2022-06-27T11:47:52.378203+0000","last_fullsized":"2022-06-27T11:47:52.378203+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:17:51.913392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.369021+0000","last_change":"2022-06-27T11:47:51.979355+0000","last_active":"2022-06-27T11:47:52.369021+0000","last_peered":"2022-06-27T11:47:52.369021+0000","last_clean":"2022-06-27T11:47:52.369021+0000","last_became_active":"2022-06-27T11:47:51.977591+0000","last_became_peered":"2022-06-27T11:47:51.977591+0000","last_unstale":"2022-06-27T11:47:52.369021+0000","last_undegraded":"2022-06-27T11:47:52.369021+0000","last_fullsized":"2022-06-27T11:47:52.369021+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T11:50:43.086014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.385028+0000","last_change":"2022-06-27T11:47:52.375271+0000","last_active":"2022-06-27T11:47:52.385028+0000","last_peered":"2022-06-27T11:47:52.385028+0000","last_clean":"2022-06-27T11:47:52.385028+0000","last_became_active":"2022-06-27T11:47:52.374985+0000","last_became_peered":"2022-06-27T11:47:52.374985+0000","last_unstale":"2022-06-27T11:47:52.385028+0000","last_undegraded":"2022-06-27T11:47:52.385028+0000","last_fullsized":"2022-06-27T11:47:52.385028+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T17:58:40.711409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:57.367534+0000","last_change":"2022-06-27T11:47:51.979723+0000","last_active":"2022-06-27T11:47:57.367534+0000","last_peered":"2022-06-27T11:47:57.367534+0000","last_clean":"2022-06-27T11:47:57.367534+0000","last_became_active":"2022-06-27T11:47:51.977669+0000","last_became_peered":"2022-06-27T11:47:51.977669+0000","last_unstale":"2022-06-27T11:47:57.367534+0000","last_undegraded":"2022-06-27T11:47:57.367534+0000","last_fullsized":"2022-06-27T11:47:57.367534+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T14:35:14.770254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.363444+0000","last_change":"2022-06-27T11:47:48.388880+0000","last_active":"2022-06-27T11:47:52.363444+0000","last_peered":"2022-06-27T11:47:52.363444+0000","last_clean":"2022-06-27T11:47:52.363444+0000","last_became_active":"2022-06-27T11:47:48.388639+0000","last_became_peered":"2022-06-27T11:47:48.388639+0000","last_unstale":"2022-06-27T11:47:52.363444+0000","last_undegraded":"2022-06-27T11:47:52.363444+0000","last_fullsized":"2022-06-27T11:47:52.363444+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_clean_scrub_stamp":"2022-06-27T11:47:46.780512+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:31:10.549485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.380026+0000","last_change":"2022-06-27T11:47:52.373143+0000","last_active":"2022-06-27T11:47:52.380026+0000","last_peered":"2022-06-27T11:47:52.380026+0000","last_clean":"2022-06-27T11:47:52.380026+0000","last_became_active":"2022-06-27T11:47:52.372850+0000","last_became_peered":"2022-06-27T11:47:52.372850+0000","last_unstale":"2022-06-27T11:47:52.380026+0000","last_undegraded":"2022-06-27T11:47:52.380026+0000","last_fullsized":"2022-06-27T11:47:52.380026+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:48:44.251459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.369716+0000","last_change":"2022-06-27T11:47:47.793307+0000","last_active":"2022-06-27T11:47:52.369716+0000","last_peered":"2022-06-27T11:47:52.369716+0000","last_clean":"2022-06-27T11:47:52.369716+0000","last_became_active":"2022-06-27T11:47:47.792958+0000","last_became_peered":"2022-06-27T11:47:47.792958+0000","last_unstale":"2022-06-27T11:47:52.369716+0000","last_undegraded":"2022-06-27T11:47:52.369716+0000","last_fullsized":"2022-06-27T11:47:52.369716+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_clean_scrub_stamp":"2022-06-27T11:47:46.780512+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:08:35.128804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:53.910382+0000","last_change":"2022-06-27T11:47:51.976872+0000","last_active":"2022-06-27T11:47:53.910382+0000","last_peered":"2022-06-27T11:47:53.910382+0000","last_clean":"2022-06-27T11:47:53.910382+0000","last_became_active":"2022-06-27T11:47:51.976546+0000","last_became_peered":"2022-06-27T11:47:51.976546+0000","last_unstale":"2022-06-27T11:47:53.910382+0000","last_undegraded":"2022-06-27T11:47:53.910382+0000","last_fullsized":"2022-06-27T11:47:53.910382+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T14:44:43.349688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.375090+0000","last_change":"2022-06-27T11:47:52.372794+0000","last_active":"2022-06-27T11:47:52.375090+0000","last_peered":"2022-06-27T11:47:52.375090+0000","last_clean":"2022-06-27T11:47:52.375090+0000","last_became_active":"2022-06-27T11:47:52.372486+0000","last_became_peered":"2022-06-27T11:47:52.372486+0000","last_unstale":"2022-06-27T11:47:52.375090+0000","last_undegraded":"2022-06-27T11:47:52.375090+0000","last_fullsized":"2022-06-27T11:47:52.375090+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:23:56.307995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.363787+0000","last_change":"2022-06-27T11:47:47.792631+0000","last_active":"2022-06-27T11:47:52.363787+0000","last_peered":"2022-06-27T11:47:52.363787+0000","last_clean":"2022-06-27T11:47:52.363787+0000","last_became_active":"2022-06-27T11:47:47.792282+0000","last_became_peered":"2022-06-27T11:47:47.792282+0000","last_unstale":"2022-06-27T11:47:52.363787+0000","last_undegraded":"2022-06-27T11:47:52.363787+0000","last_fullsized":"2022-06-27T11:47:52.363787+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_clean_scrub_stamp":"2022-06-27T11:47:46.780512+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:41:25.449379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"17'117","reported_seq":163,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.368104+0000","last_change":"2022-06-27T11:47:47.791697+0000","last_active":"2022-06-27T11:47:52.368104+0000","last_peered":"2022-06-27T11:47:52.368104+0000","last_clean":"2022-06-27T11:47:52.368104+0000","last_became_active":"2022-06-27T11:47:47.790645+0000","last_became_peered":"2022-06-27T11:47:47.790645+0000","last_unstale":"2022-06-27T11:47:52.368104+0000","last_undegraded":"2022-06-27T11:47:52.368104+0000","last_fullsized":"2022-06-27T11:47:52.368104+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_clean_scrub_stamp":"2022-06-27T11:47:46.780512+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:11:09.344666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.367962+0000","last_change":"2022-06-27T11:47:51.982728+0000","last_active":"2022-06-27T11:47:52.367962+0000","last_peered":"2022-06-27T11:47:52.367962+0000","last_clean":"2022-06-27T11:47:52.367962+0000","last_became_active":"2022-06-27T11:47:51.982445+0000","last_became_peered":"2022-06-27T11:47:51.982445+0000","last_unstale":"2022-06-27T11:47:52.367962+0000","last_undegraded":"2022-06-27T11:47:52.367962+0000","last_fullsized":"2022-06-27T11:47:52.367962+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:42:04.682687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.386922+0000","last_change":"2022-06-27T11:47:52.386015+0000","last_active":"2022-06-27T11:47:52.386922+0000","last_peered":"2022-06-27T11:47:52.386922+0000","last_clean":"2022-06-27T11:47:52.386922+0000","last_became_active":"2022-06-27T11:47:52.385731+0000","last_became_peered":"2022-06-27T11:47:52.385731+0000","last_unstale":"2022-06-27T11:47:52.386922+0000","last_undegraded":"2022-06-27T11:47:52.386922+0000","last_fullsized":"2022-06-27T11:47:52.386922+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T11:58:50.483766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.366627+0000","last_change":"2022-06-27T11:47:47.807699+0000","last_active":"2022-06-27T11:47:52.366627+0000","last_peered":"2022-06-27T11:47:52.366627+0000","last_clean":"2022-06-27T11:47:52.366627+0000","last_became_active":"2022-06-27T11:47:47.807422+0000","last_became_peered":"2022-06-27T11:47:47.807422+0000","last_unstale":"2022-06-27T11:47:52.366627+0000","last_undegraded":"2022-06-27T11:47:52.366627+0000","last_fullsized":"2022-06-27T11:47:52.366627+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_clean_scrub_stamp":"2022-06-27T11:47:46.780512+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:15:00.549295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.363922+0000","last_change":"2022-06-27T11:47:51.974468+0000","last_active":"2022-06-27T11:47:52.363922+0000","last_peered":"2022-06-27T11:47:52.363922+0000","last_clean":"2022-06-27T11:47:52.363922+0000","last_became_active":"2022-06-27T11:47:51.974160+0000","last_became_peered":"2022-06-27T11:47:51.974160+0000","last_unstale":"2022-06-27T11:47:52.363922+0000","last_undegraded":"2022-06-27T11:47:52.363922+0000","last_fullsized":"2022-06-27T11:47:52.363922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T19:14:45.694282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.380162+0000","last_change":"2022-06-27T11:47:52.373931+0000","last_active":"2022-06-27T11:47:52.380162+0000","last_peered":"2022-06-27T11:47:52.380162+0000","last_clean":"2022-06-27T11:47:52.380162+0000","last_became_active":"2022-06-27T11:47:52.373562+0000","last_became_peered":"2022-06-27T11:47:52.373562+0000","last_unstale":"2022-06-27T11:47:52.380162+0000","last_undegraded":"2022-06-27T11:47:52.380162+0000","last_fullsized":"2022-06-27T11:47:52.380162+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T14:09:22.859955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.367866+0000","last_change":"2022-06-27T11:47:47.807650+0000","last_active":"2022-06-27T11:47:52.367866+0000","last_peered":"2022-06-27T11:47:52.367866+0000","last_clean":"2022-06-27T11:47:52.367866+0000","last_became_active":"2022-06-27T11:47:47.807375+0000","last_became_peered":"2022-06-27T11:47:47.807375+0000","last_unstale":"2022-06-27T11:47:52.367866+0000","last_undegraded":"2022-06-27T11:47:52.367866+0000","last_fullsized":"2022-06-27T11:47:52.367866+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_clean_scrub_stamp":"2022-06-27T11:47:46.780512+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T14:10:48.067284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.368898+0000","last_change":"2022-06-27T11:47:51.972154+0000","last_active":"2022-06-27T11:47:52.368898+0000","last_peered":"2022-06-27T11:47:52.368898+0000","last_clean":"2022-06-27T11:47:52.368898+0000","last_became_active":"2022-06-27T11:47:51.971762+0000","last_became_peered":"2022-06-27T11:47:51.971762+0000","last_unstale":"2022-06-27T11:47:52.368898+0000","last_undegraded":"2022-06-27T11:47:52.368898+0000","last_fullsized":"2022-06-27T11:47:52.368898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T18:15:14.271005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.379879+0000","last_change":"2022-06-27T11:47:52.378294+0000","last_active":"2022-06-27T11:47:52.379879+0000","last_peered":"2022-06-27T11:47:52.379879+0000","last_clean":"2022-06-27T11:47:52.379879+0000","last_became_active":"2022-06-27T11:47:52.377704+0000","last_became_peered":"2022-06-27T11:47:52.377704+0000","last_unstale":"2022-06-27T11:47:52.379879+0000","last_undegraded":"2022-06-27T11:47:52.379879+0000","last_fullsized":"2022-06-27T11:47:52.379879+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:26:34.474081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.364180+0000","last_change":"2022-06-27T11:47:47.807600+0000","last_active":"2022-06-27T11:47:52.364180+0000","last_peered":"2022-06-27T11:47:52.364180+0000","last_clean":"2022-06-27T11:47:52.364180+0000","last_became_active":"2022-06-27T11:47:47.807284+0000","last_became_peered":"2022-06-27T11:47:47.807284+0000","last_unstale":"2022-06-27T11:47:52.364180+0000","last_undegraded":"2022-06-27T11:47:52.364180+0000","last_fullsized":"2022-06-27T11:47:52.364180+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_clean_scrub_stamp":"2022-06-27T11:47:46.780512+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T20:57:06.487027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.362687+0000","last_change":"2022-06-27T11:47:51.980734+0000","last_active":"2022-06-27T11:47:52.362687+0000","last_peered":"2022-06-27T11:47:52.362687+0000","last_clean":"2022-06-27T11:47:52.362687+0000","last_became_active":"2022-06-27T11:47:51.980476+0000","last_became_peered":"2022-06-27T11:47:51.980476+0000","last_unstale":"2022-06-27T11:47:52.362687+0000","last_undegraded":"2022-06-27T11:47:52.362687+0000","last_fullsized":"2022-06-27T11:47:52.362687+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T21:32:16.435472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.385478+0000","last_change":"2022-06-27T11:47:52.379902+0000","last_active":"2022-06-27T11:47:52.385478+0000","last_peered":"2022-06-27T11:47:52.385478+0000","last_clean":"2022-06-27T11:47:52.385478+0000","last_became_active":"2022-06-27T11:47:52.379359+0000","last_became_peered":"2022-06-27T11:47:52.379359+0000","last_unstale":"2022-06-27T11:47:52.385478+0000","last_undegraded":"2022-06-27T11:47:52.385478+0000","last_fullsized":"2022-06-27T11:47:52.385478+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:41:27.884083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.370459+0000","last_change":"2022-06-27T11:47:47.807607+0000","last_active":"2022-06-27T11:47:52.370459+0000","last_peered":"2022-06-27T11:47:52.370459+0000","last_clean":"2022-06-27T11:47:52.370459+0000","last_became_active":"2022-06-27T11:47:47.807293+0000","last_became_peered":"2022-06-27T11:47:47.807293+0000","last_unstale":"2022-06-27T11:47:52.370459+0000","last_undegraded":"2022-06-27T11:47:52.370459+0000","last_fullsized":"2022-06-27T11:47:52.370459+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_clean_scrub_stamp":"2022-06-27T11:47:46.780512+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T15:00:35.318372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.364133+0000","last_change":"2022-06-27T11:47:51.976055+0000","last_active":"2022-06-27T11:47:52.364133+0000","last_peered":"2022-06-27T11:47:52.364133+0000","last_clean":"2022-06-27T11:47:52.364133+0000","last_became_active":"2022-06-27T11:47:51.975303+0000","last_became_peered":"2022-06-27T11:47:51.975303+0000","last_unstale":"2022-06-27T11:47:52.364133+0000","last_undegraded":"2022-06-27T11:47:52.364133+0000","last_fullsized":"2022-06-27T11:47:52.364133+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:27:32.125525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384750+0000","last_change":"2022-06-27T11:47:52.380775+0000","last_active":"2022-06-27T11:47:52.384750+0000","last_peered":"2022-06-27T11:47:52.384750+0000","last_clean":"2022-06-27T11:47:52.384750+0000","last_became_active":"2022-06-27T11:47:52.380439+0000","last_became_peered":"2022-06-27T11:47:52.380439+0000","last_unstale":"2022-06-27T11:47:52.384750+0000","last_undegraded":"2022-06-27T11:47:52.384750+0000","last_fullsized":"2022-06-27T11:47:52.384750+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:03:44.939329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.370743+0000","last_change":"2022-06-27T11:47:47.792314+0000","last_active":"2022-06-27T11:47:52.370743+0000","last_peered":"2022-06-27T11:47:52.370743+0000","last_clean":"2022-06-27T11:47:52.370743+0000","last_became_active":"2022-06-27T11:47:47.792032+0000","last_became_peered":"2022-06-27T11:47:47.792032+0000","last_unstale":"2022-06-27T11:47:52.370743+0000","last_undegraded":"2022-06-27T11:47:52.370743+0000","last_fullsized":"2022-06-27T11:47:52.370743+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:46.780512+0000","last_clean_scrub_stamp":"2022-06-27T11:47:46.780512+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T14:13:03.591775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.369208+0000","last_change":"2022-06-27T11:47:51.978639+0000","last_active":"2022-06-27T11:47:52.369208+0000","last_peered":"2022-06-27T11:47:52.369208+0000","last_clean":"2022-06-27T11:47:52.369208+0000","last_became_active":"2022-06-27T11:47:51.978341+0000","last_became_peered":"2022-06-27T11:47:51.978341+0000","last_unstale":"2022-06-27T11:47:52.369208+0000","last_undegraded":"2022-06-27T11:47:52.369208+0000","last_fullsized":"2022-06-27T11:47:52.369208+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T21:34:48.070558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.381133+0000","last_change":"2022-06-27T11:47:52.373588+0000","last_active":"2022-06-27T11:47:52.381133+0000","last_peered":"2022-06-27T11:47:52.381133+0000","last_clean":"2022-06-27T11:47:52.381133+0000","last_became_active":"2022-06-27T11:47:52.373208+0000","last_became_peered":"2022-06-27T11:47:52.373208+0000","last_unstale":"2022-06-27T11:47:52.381133+0000","last_undegraded":"2022-06-27T11:47:52.381133+0000","last_fullsized":"2022-06-27T11:47:52.381133+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T20:27:28.252147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:53.910019+0000","last_change":"2022-06-27T11:47:51.982051+0000","last_active":"2022-06-27T11:47:53.910019+0000","last_peered":"2022-06-27T11:47:53.910019+0000","last_clean":"2022-06-27T11:47:53.910019+0000","last_became_active":"2022-06-27T11:47:51.981742+0000","last_became_peered":"2022-06-27T11:47:51.981742+0000","last_unstale":"2022-06-27T11:47:53.910019+0000","last_undegraded":"2022-06-27T11:47:53.910019+0000","last_fullsized":"2022-06-27T11:47:53.910019+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T22:39:56.937086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.380333+0000","last_change":"2022-06-27T11:47:52.378192+0000","last_active":"2022-06-27T11:47:52.380333+0000","last_peered":"2022-06-27T11:47:52.380333+0000","last_clean":"2022-06-27T11:47:52.380333+0000","last_became_active":"2022-06-27T11:47:52.377919+0000","last_became_peered":"2022-06-27T11:47:52.377919+0000","last_unstale":"2022-06-27T11:47:52.380333+0000","last_undegraded":"2022-06-27T11:47:52.380333+0000","last_fullsized":"2022-06-27T11:47:52.380333+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T18:51:35.757778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:57.368777+0000","last_change":"2022-06-27T11:47:51.981101+0000","last_active":"2022-06-27T11:47:57.368777+0000","last_peered":"2022-06-27T11:47:57.368777+0000","last_clean":"2022-06-27T11:47:57.368777+0000","last_became_active":"2022-06-27T11:47:51.980793+0000","last_became_peered":"2022-06-27T11:47:51.980793+0000","last_unstale":"2022-06-27T11:47:57.368777+0000","last_undegraded":"2022-06-27T11:47:57.368777+0000","last_fullsized":"2022-06-27T11:47:57.368777+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:28:10.206148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.372181+0000","last_change":"2022-06-27T11:47:52.370334+0000","last_active":"2022-06-27T11:47:52.372181+0000","last_peered":"2022-06-27T11:47:52.372181+0000","last_clean":"2022-06-27T11:47:52.372181+0000","last_became_active":"2022-06-27T11:47:52.370028+0000","last_became_peered":"2022-06-27T11:47:52.370028+0000","last_unstale":"2022-06-27T11:47:52.372181+0000","last_undegraded":"2022-06-27T11:47:52.372181+0000","last_fullsized":"2022-06-27T11:47:52.372181+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T17:28:12.357757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:57.369040+0000","last_change":"2022-06-27T11:47:51.981186+0000","last_active":"2022-06-27T11:47:57.369040+0000","last_peered":"2022-06-27T11:47:57.369040+0000","last_clean":"2022-06-27T11:47:57.369040+0000","last_became_active":"2022-06-27T11:47:51.980921+0000","last_became_peered":"2022-06-27T11:47:51.980921+0000","last_unstale":"2022-06-27T11:47:57.369040+0000","last_undegraded":"2022-06-27T11:47:57.369040+0000","last_fullsized":"2022-06-27T11:47:57.369040+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:04:40.732299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.387201+0000","last_change":"2022-06-27T11:47:52.382091+0000","last_active":"2022-06-27T11:47:52.387201+0000","last_peered":"2022-06-27T11:47:52.387201+0000","last_clean":"2022-06-27T11:47:52.387201+0000","last_became_active":"2022-06-27T11:47:52.381633+0000","last_became_peered":"2022-06-27T11:47:52.381633+0000","last_unstale":"2022-06-27T11:47:52.387201+0000","last_undegraded":"2022-06-27T11:47:52.387201+0000","last_fullsized":"2022-06-27T11:47:52.387201+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:34:44.030534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:53.910420+0000","last_change":"2022-06-27T11:47:51.972963+0000","last_active":"2022-06-27T11:47:53.910420+0000","last_peered":"2022-06-27T11:47:53.910420+0000","last_clean":"2022-06-27T11:47:53.910420+0000","last_became_active":"2022-06-27T11:47:51.972686+0000","last_became_peered":"2022-06-27T11:47:51.972686+0000","last_unstale":"2022-06-27T11:47:53.910420+0000","last_undegraded":"2022-06-27T11:47:53.910420+0000","last_fullsized":"2022-06-27T11:47:53.910420+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T12:11:38.008538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.385365+0000","last_change":"2022-06-27T11:47:52.383414+0000","last_active":"2022-06-27T11:47:52.385365+0000","last_peered":"2022-06-27T11:47:52.385365+0000","last_clean":"2022-06-27T11:47:52.385365+0000","last_became_active":"2022-06-27T11:47:52.383070+0000","last_became_peered":"2022-06-27T11:47:52.383070+0000","last_unstale":"2022-06-27T11:47:52.385365+0000","last_undegraded":"2022-06-27T11:47:52.385365+0000","last_fullsized":"2022-06-27T11:47:52.385365+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:11:28.410045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"21'4","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-06-27T11:47:57.370018+0000","last_change":"2022-06-27T11:47:51.971664+0000","last_active":"2022-06-27T11:47:57.370018+0000","last_peered":"2022-06-27T11:47:57.370018+0000","last_clean":"2022-06-27T11:47:57.370018+0000","last_became_active":"2022-06-27T11:47:51.971385+0000","last_became_peered":"2022-06-27T11:47:51.971385+0000","last_unstale":"2022-06-27T11:47:57.370018+0000","last_undegraded":"2022-06-27T11:47:57.370018+0000","last_fullsized":"2022-06-27T11:47:57.370018+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:50.950859+0000","last_clean_scrub_stamp":"2022-06-27T11:47:50.950859+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T18:44:38.394388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.379723+0000","last_change":"2022-06-27T11:47:52.377305+0000","last_active":"2022-06-27T11:47:52.379723+0000","last_peered":"2022-06-27T11:47:52.379723+0000","last_clean":"2022-06-27T11:47:52.379723+0000","last_became_active":"2022-06-27T11:47:52.377012+0000","last_became_peered":"2022-06-27T11:47:52.377012+0000","last_unstale":"2022-06-27T11:47:52.379723+0000","last_undegraded":"2022-06-27T11:47:52.379723+0000","last_fullsized":"2022-06-27T11:47:52.379723+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T13:15:14.505656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.377592+0000","last_change":"2022-06-27T11:47:52.376074+0000","last_active":"2022-06-27T11:47:52.377592+0000","last_peered":"2022-06-27T11:47:52.377592+0000","last_clean":"2022-06-27T11:47:52.377592+0000","last_became_active":"2022-06-27T11:47:52.375836+0000","last_became_peered":"2022-06-27T11:47:52.375836+0000","last_unstale":"2022-06-27T11:47:52.377592+0000","last_undegraded":"2022-06-27T11:47:52.377592+0000","last_fullsized":"2022-06-27T11:47:52.377592+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T18:46:26.900833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384440+0000","last_change":"2022-06-27T11:47:52.376668+0000","last_active":"2022-06-27T11:47:52.384440+0000","last_peered":"2022-06-27T11:47:52.384440+0000","last_clean":"2022-06-27T11:47:52.384440+0000","last_became_active":"2022-06-27T11:47:52.376386+0000","last_became_peered":"2022-06-27T11:47:52.376386+0000","last_unstale":"2022-06-27T11:47:52.384440+0000","last_undegraded":"2022-06-27T11:47:52.384440+0000","last_fullsized":"2022-06-27T11:47:52.384440+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T11:48:35.874748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.379744+0000","last_change":"2022-06-27T11:47:52.377793+0000","last_active":"2022-06-27T11:47:52.379744+0000","last_peered":"2022-06-27T11:47:52.379744+0000","last_clean":"2022-06-27T11:47:52.379744+0000","last_became_active":"2022-06-27T11:47:52.377318+0000","last_became_peered":"2022-06-27T11:47:52.377318+0000","last_unstale":"2022-06-27T11:47:52.379744+0000","last_undegraded":"2022-06-27T11:47:52.379744+0000","last_fullsized":"2022-06-27T11:47:52.379744+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T17:54:22.511301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.384925+0000","last_change":"2022-06-27T11:47:52.374638+0000","last_active":"2022-06-27T11:47:52.384925+0000","last_peered":"2022-06-27T11:47:52.384925+0000","last_clean":"2022-06-27T11:47:52.384925+0000","last_became_active":"2022-06-27T11:47:52.374352+0000","last_became_peered":"2022-06-27T11:47:52.374352+0000","last_unstale":"2022-06-27T11:47:52.384925+0000","last_undegraded":"2022-06-27T11:47:52.384925+0000","last_fullsized":"2022-06-27T11:47:52.384925+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T16:09:59.369369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-06-27T11:47:52.375769+0000","last_change":"2022-06-27T11:47:52.373288+0000","last_active":"2022-06-27T11:47:52.375769+0000","last_peered":"2022-06-27T11:47:52.375769+0000","last_clean":"2022-06-27T11:47:52.375769+0000","last_became_active":"2022-06-27T11:47:52.372891+0000","last_became_peered":"2022-06-27T11:47:52.372891+0000","last_unstale":"2022-06-27T11:47:52.375769+0000","last_undegraded":"2022-06-27T11:47:52.375769+0000","last_fullsized":"2022-06-27T11:47:52.375769+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-27T11:47:51.954608+0000","last_clean_scrub_stamp":"2022-06-27T11:47:51.954608+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-06-28T23:16:08.088450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2590,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":5180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40,"ondisk_log_size":40,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64756,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9016,"kb_used_data":304,"kb_used_omap":0,"kb_used_meta":8704,"kb_avail":94362824,"statfs":{"total":96636764160,"available":96627531776,"internally_reserved":0,"allocated":311296,"data_stored":671231,"data_compressed":32378,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":8912896},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38900000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57499999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8408,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":8192,"kb_avail":94363432,"statfs":{"total":96636764160,"available":96628154368,"internally_reserved":0,"allocated":212992,"data_stored":77836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8388608},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36299999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51900000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8988,"kb_used_data":276,"kb_used_omap":0,"kb_used_meta":8704,"kb_avail":94362852,"statfs":{"total":96636764160,"available":96627560448,"internally_reserved":0,"allocated":282624,"data_stored":669207,"data_compressed":32378,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":8912896},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36099999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52500000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8480,"kb_used_data":216,"kb_used_omap":0,"kb_used_meta":8256,"kb_avail":94363360,"statfs":{"total":96636764160,"available":96628080640,"internally_reserved":0,"allocated":221184,"data_stored":79405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8454144},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.253}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32300000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30299999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.442}]}]},{"osd":2,"up_from":13,"seq":55834574857,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8412,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":8192,"kb_avail":94363428,"statfs":{"total":96636764160,"available":96628150272,"internally_reserved":0,"allocated":217088,"data_stored":78839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8388608},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35999999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30599999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52400000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8476,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":8256,"kb_avail":94363364,"statfs":{"total":96636764160,"available":96628084736,"internally_reserved":0,"allocated":217088,"data_stored":77926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8454144},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24399999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33400000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42299999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]}]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8560,"kb_used_data":232,"kb_used_omap":0,"kb_used_meta":8320,"kb_avail":94363280,"statfs":{"total":96636764160,"available":96627998720,"internally_reserved":0,"allocated":237568,"data_stored":80703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8519680},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.436}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46500000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]}]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8420,"kb_used_data":220,"kb_used_omap":0,"kb_used_meta":8192,"kb_avail":94363420,"statfs":{"total":96636764160,"available":96628142080,"internally_reserved":0,"allocated":225280,"data_stored":79475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8388608},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68500000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.748}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72099999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76500000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84499999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32378,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32378,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-06-27T11:48:12.465 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-27T11:48:12.618 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.618+0000 7f312346a700 1 Processor -- start 2022-06-27T11:48:12.619 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.618+0000 7f312346a700 1 -- start start 2022-06-27T11:48:12.629 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.619+0000 7f312346a700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f311c06e850 0x7f311c1570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:12.629 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.619+0000 7f312346a700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f311c06fef0 0x7f311c069de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:12.630 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.619+0000 7f312346a700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f311c06fba0 con 0x7f311c0c34a0 2022-06-27T11:48:12.630 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.619+0000 7f312346a700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f311c1575e0 con 0x7f311c06e850 2022-06-27T11:48:12.631 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.619+0000 7f312346a700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f311c06f5e0 con 0x7f311c06fef0 2022-06-27T11:48:12.631 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.619+0000 7f3121206700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f311c06e850 0x7f311c1570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:12.631 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.619+0000 7f3121a07700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f311c0c34a0 0x7f311c0c2920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:48302/0 (socket says 172.21.15.195:48302) 2022-06-27T11:48:12.632 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.619+0000 7f3121a07700 1 -- 172.21.15.195:0/2003517716 learned_addr learned my addr 172.21.15.195:0/2003517716 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:48:12.632 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.620+0000 7f3120a05700 1 --2- 172.21.15.195:0/2003517716 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f311c06fef0 0x7f311c069de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:12.632 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.620+0000 7f3121206700 1 -- 172.21.15.195:0/2003517716 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f311c06fef0 msgr2=0x7f311c069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:12.632 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.620+0000 7f3121206700 1 --2- 172.21.15.195:0/2003517716 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f311c06fef0 0x7f311c069de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:12.632 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.620+0000 7f3121206700 1 -- 172.21.15.195:0/2003517716 >> v1:172.21.15.195:6789/0 conn(0x7f311c0c34a0 legacy=0x7f311c0c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:12.633 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.620+0000 7f310bfff700 1 -- 172.21.15.195:0/2003517716 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2320752465 0 0) 0x7f311c06fba0 con 0x7f311c0c34a0 2022-06-27T11:48:12.633 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.620+0000 7f3121206700 1 -- 172.21.15.195:0/2003517716 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f311c06f8c0 con 0x7f311c06e850 2022-06-27T11:48:12.633 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.620+0000 7f3121206700 1 --2- 172.21.15.195:0/2003517716 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f311c06e850 0x7f311c1570a0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f310c0097a0 tx=0x7f310c009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cacf9eb0de588b5 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:12.634 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.620+0000 7f310bfff700 1 -- 172.21.15.195:0/2003517716 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f310c011630 con 0x7f311c06e850 2022-06-27T11:48:12.634 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.620+0000 7f310bfff700 1 -- 172.21.15.195:0/2003517716 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f310c011790 con 0x7f311c06e850 2022-06-27T11:48:12.634 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.620+0000 7f310bfff700 1 -- 172.21.15.195:0/2003517716 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f310c011b30 con 0x7f311c06e850 2022-06-27T11:48:12.634 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.620+0000 7f312346a700 1 -- 172.21.15.195:0/2003517716 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f311c06e850 msgr2=0x7f311c1570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:12.635 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.621+0000 7f312346a700 1 --2- 172.21.15.195:0/2003517716 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f311c06e850 0x7f311c1570a0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f310c0097a0 tx=0x7f310c009060 comp rx=0 tx=0).stop 2022-06-27T11:48:12.635 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.621+0000 7f312346a700 1 -- 172.21.15.195:0/2003517716 shutdown_connections 2022-06-27T11:48:12.635 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.621+0000 7f312346a700 1 --2- 172.21.15.195:0/2003517716 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f311c06e850 0x7f311c1570a0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:12.635 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.621+0000 7f312346a700 1 --2- 172.21.15.195:0/2003517716 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f311c06fef0 0x7f311c069de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:12.636 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.621+0000 7f312346a700 1 -- 172.21.15.195:0/2003517716 >> 172.21.15.195:0/2003517716 conn(0x7f311c0c6350 msgr2=0x7f311c0c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:12.636 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.621+0000 7f312346a700 1 -- 172.21.15.195:0/2003517716 shutdown_connections 2022-06-27T11:48:12.636 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.621+0000 7f312346a700 1 -- 172.21.15.195:0/2003517716 wait complete. 2022-06-27T11:48:12.636 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.622+0000 7f312346a700 1 Processor -- start 2022-06-27T11:48:12.636 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.622+0000 7f312346a700 1 -- start start 2022-06-27T11:48:12.637 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.623+0000 7f312346a700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f311c06e850 0x7f311c155d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:12.637 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.623+0000 7f312346a700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f311c0c34a0 0x7f311c151e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:12.637 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.623+0000 7f312346a700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f311c153b20 0x7f311c146360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:12.638 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.623+0000 7f312346a700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f311c0bfa00 con 0x7f311c153b20 2022-06-27T11:48:12.638 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.623+0000 7f312346a700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f311c153860 con 0x7f311c0c34a0 2022-06-27T11:48:12.638 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.623+0000 7f312346a700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f311c151980 con 0x7f311c06e850 2022-06-27T11:48:12.639 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.623+0000 7f3121a07700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f311c153b20 0x7f311c146360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:12.639 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.623+0000 7f3121a07700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f311c153b20 0x7f311c146360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:42120/0 (socket says 172.21.15.195:42120) 2022-06-27T11:48:12.639 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.623+0000 7f3121a07700 1 -- 172.21.15.195:0/3839711817 learned_addr learned my addr 172.21.15.195:0/3839711817 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:48:12.639 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.623+0000 7f3121206700 1 --2- 172.21.15.195:0/3839711817 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f311c06e850 0x7f311c155d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:12.639 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.623+0000 7f3120a05700 1 --2- 172.21.15.195:0/3839711817 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f311c0c34a0 0x7f311c151e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:12.640 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.623+0000 7f3121a07700 1 -- 172.21.15.195:0/3839711817 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f311c06e850 msgr2=0x7f311c155d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:12.640 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.623+0000 7f3121a07700 1 --2- 172.21.15.195:0/3839711817 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f311c06e850 0x7f311c155d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:12.640 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.623+0000 7f3121a07700 1 -- 172.21.15.195:0/3839711817 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f311c0c34a0 msgr2=0x7f311c151e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:12.640 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.623+0000 7f3121a07700 1 --2- 172.21.15.195:0/3839711817 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f311c0c34a0 0x7f311c151e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:12.641 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.623+0000 7f3121a07700 1 -- 172.21.15.195:0/3839711817 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f310c0092e0 con 0x7f311c153b20 2022-06-27T11:48:12.641 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.623+0000 7f3121206700 1 --2- 172.21.15.195:0/3839711817 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f311c06e850 0x7f311c155d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T11:48:12.641 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.623+0000 7f3121a07700 1 --2- 172.21.15.195:0/3839711817 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f311c153b20 0x7f311c146360 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f311400c860 tx=0x7f3114001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:12.641 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.623+0000 7f31097fa700 1 -- 172.21.15.195:0/3839711817 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3114004c80 con 0x7f311c153b20 2022-06-27T11:48:12.642 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.623+0000 7f31097fa700 1 -- 172.21.15.195:0/3839711817 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3114004de0 con 0x7f311c153b20 2022-06-27T11:48:12.642 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.623+0000 7f312346a700 1 -- 172.21.15.195:0/3839711817 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f311c0bfbe0 con 0x7f311c153b20 2022-06-27T11:48:12.642 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.623+0000 7f31097fa700 1 -- 172.21.15.195:0/3839711817 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3114007380 con 0x7f311c153b20 2022-06-27T11:48:12.642 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.623+0000 7f312346a700 1 -- 172.21.15.195:0/3839711817 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f311c151330 con 0x7f311c153b20 2022-06-27T11:48:12.643 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.624+0000 7f312346a700 1 -- 172.21.15.195:0/3839711817 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f311c0722c0 con 0x7f311c153b20 2022-06-27T11:48:12.643 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.625+0000 7f31097fa700 1 -- 172.21.15.195:0/3839711817 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f311401a030 con 0x7f311c153b20 2022-06-27T11:48:12.643 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.625+0000 7f31097fa700 1 --2- 172.21.15.195:0/3839711817 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f30fc063d00 0x7f30fc0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:12.643 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.625+0000 7f31097fa700 1 -- 172.21.15.195:0/3839711817 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f311407ff30 con 0x7f311c153b20 2022-06-27T11:48:12.643 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.625+0000 7f3121206700 1 --2- 172.21.15.195:0/3839711817 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f30fc063d00 0x7f30fc0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:12.644 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.626+0000 7f3121206700 1 --2- 172.21.15.195:0/3839711817 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f30fc063d00 0x7f30fc0661b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f311c154200 tx=0x7f310c017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:12.644 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.629+0000 7f31097fa700 1 -- 172.21.15.195:0/3839711817 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f311404eba0 con 0x7f311c153b20 2022-06-27T11:48:12.811 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.810+0000 7f312346a700 1 -- 172.21.15.195:0/3839711817 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f311c0c1000 con 0x7f311c153b20 2022-06-27T11:48:12.812 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.811+0000 7f31097fa700 1 -- 172.21.15.195:0/3839711817 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16914 (secure 0 0 0) 0x7f31140521c0 con 0x7f311c153b20 2022-06-27T11:48:12.813 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:48:12.814 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":21,"fsid":"81b3a1d3-6a2b-47e8-8d69-7027cfeae7af","created":"2022-06-27T11:47:35.094715+0000","modified":"2022-06-27T11:47:53.359270+0000","last_up_change":"2022-06-27T11:47:44.777459+0000","last_in_change":"2022-06-27T11:47:37.596818+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2022-06-27T11:47:46.368325+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2022-06-27T11:47:46.386132+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-06-27T11:47:50.901270+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-06-27T11:47:51.314719+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c64bc51f-d4f3-4bc2-806b-da0dc78d51b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6813","nonce":94217},{"type":"v1","addr":"172.21.15.195:6815","nonce":94217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6817","nonce":94217},{"type":"v1","addr":"172.21.15.195:6819","nonce":94217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":94217},{"type":"v1","addr":"172.21.15.195:6823","nonce":94217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6820","nonce":94217},{"type":"v1","addr":"172.21.15.195:6821","nonce":94217}]},"public_addr":"172.21.15.195:6815/94217","cluster_addr":"172.21.15.195:6819/94217","heartbeat_back_addr":"172.21.15.195:6823/94217","heartbeat_front_addr":"172.21.15.195:6821/94217","state":["exists","up"]},{"osd":1,"uuid":"bea71cdf-aa98-493f-9207-66d886a3e30e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":94223},{"type":"v1","addr":"172.21.15.195:6809","nonce":94223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":94223},{"type":"v1","addr":"172.21.15.195:6811","nonce":94223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6816","nonce":94223},{"type":"v1","addr":"172.21.15.195:6818","nonce":94223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":94223},{"type":"v1","addr":"172.21.15.195:6814","nonce":94223}]},"public_addr":"172.21.15.195:6809/94223","cluster_addr":"172.21.15.195:6811/94223","heartbeat_back_addr":"172.21.15.195:6818/94223","heartbeat_front_addr":"172.21.15.195:6814/94223","state":["exists","up"]},{"osd":2,"uuid":"bf6c87a0-a8ea-497a-9ce1-721285888065","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":94221},{"type":"v1","addr":"172.21.15.195:6801","nonce":94221}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":94221},{"type":"v1","addr":"172.21.15.195:6803","nonce":94221}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":94221},{"type":"v1","addr":"172.21.15.195:6807","nonce":94221}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":94221},{"type":"v1","addr":"172.21.15.195:6805","nonce":94221}]},"public_addr":"172.21.15.195:6801/94221","cluster_addr":"172.21.15.195:6803/94221","heartbeat_back_addr":"172.21.15.195:6807/94221","heartbeat_front_addr":"172.21.15.195:6805/94221","state":["exists","up"]},{"osd":3,"uuid":"e8aa47ce-af8c-47e7-9654-870cd53229f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":94232},{"type":"v1","addr":"172.21.15.195:6825","nonce":94232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":94232},{"type":"v1","addr":"172.21.15.195:6827","nonce":94232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":94232},{"type":"v1","addr":"172.21.15.195:6831","nonce":94232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":94232},{"type":"v1","addr":"172.21.15.195:6829","nonce":94232}]},"public_addr":"172.21.15.195:6825/94232","cluster_addr":"172.21.15.195:6827/94232","heartbeat_back_addr":"172.21.15.195:6831/94232","heartbeat_front_addr":"172.21.15.195:6829/94232","state":["exists","up"]},{"osd":4,"uuid":"99ffa553-f149-443a-98f5-68b10f77e085","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6800","nonce":94308},{"type":"v1","addr":"172.21.15.202:6801","nonce":94308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6802","nonce":94308},{"type":"v1","addr":"172.21.15.202:6803","nonce":94308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6806","nonce":94308},{"type":"v1","addr":"172.21.15.202:6807","nonce":94308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6804","nonce":94308},{"type":"v1","addr":"172.21.15.202:6805","nonce":94308}]},"public_addr":"172.21.15.202:6801/94308","cluster_addr":"172.21.15.202:6803/94308","heartbeat_back_addr":"172.21.15.202:6807/94308","heartbeat_front_addr":"172.21.15.202:6805/94308","state":["exists","up"]},{"osd":5,"uuid":"35409ef0-891b-4c01-82e4-26b8e64f502d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6824","nonce":94325},{"type":"v1","addr":"172.21.15.202:6825","nonce":94325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6826","nonce":94325},{"type":"v1","addr":"172.21.15.202:6827","nonce":94325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6830","nonce":94325},{"type":"v1","addr":"172.21.15.202:6831","nonce":94325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6828","nonce":94325},{"type":"v1","addr":"172.21.15.202:6829","nonce":94325}]},"public_addr":"172.21.15.202:6825/94325","cluster_addr":"172.21.15.202:6827/94325","heartbeat_back_addr":"172.21.15.202:6831/94325","heartbeat_front_addr":"172.21.15.202:6829/94325","state":["exists","up"]},{"osd":6,"uuid":"009124c4-8b5f-47d4-bd20-2c19271a2adc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6809","nonce":94309},{"type":"v1","addr":"172.21.15.202:6811","nonce":94309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6813","nonce":94309},{"type":"v1","addr":"172.21.15.202:6815","nonce":94309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6821","nonce":94309},{"type":"v1","addr":"172.21.15.202:6823","nonce":94309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6817","nonce":94309},{"type":"v1","addr":"172.21.15.202:6819","nonce":94309}]},"public_addr":"172.21.15.202:6811/94309","cluster_addr":"172.21.15.202:6815/94309","heartbeat_back_addr":"172.21.15.202:6823/94309","heartbeat_front_addr":"172.21.15.202:6819/94309","state":["exists","up"]},{"osd":7,"uuid":"7773115e-d3c3-4a0c-8801-d8ce63e0e5a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6808","nonce":94310},{"type":"v1","addr":"172.21.15.202:6810","nonce":94310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6812","nonce":94310},{"type":"v1","addr":"172.21.15.202:6814","nonce":94310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6820","nonce":94310},{"type":"v1","addr":"172.21.15.202:6822","nonce":94310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6816","nonce":94310},{"type":"v1","addr":"172.21.15.202:6818","nonce":94310}]},"public_addr":"172.21.15.202:6810/94310","cluster_addr":"172.21.15.202:6814/94310","heartbeat_back_addr":"172.21.15.202:6822/94310","heartbeat_front_addr":"172.21.15.202:6818/94310","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.776182+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.806922+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.237997+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.784371+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.723615+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.820332+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:43.841414+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.758071+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-06-27T11:48:12.822 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.813+0000 7f312346a700 1 -- 172.21.15.195:0/3839711817 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f30fc063d00 msgr2=0x7f30fc0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:12.822 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.813+0000 7f312346a700 1 --2- 172.21.15.195:0/3839711817 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f30fc063d00 0x7f30fc0661b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f311c154200 tx=0x7f310c017000 comp rx=0 tx=0).stop 2022-06-27T11:48:12.823 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.813+0000 7f312346a700 1 -- 172.21.15.195:0/3839711817 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f311c153b20 msgr2=0x7f311c146360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:12.823 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.813+0000 7f312346a700 1 --2- 172.21.15.195:0/3839711817 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f311c153b20 0x7f311c146360 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f311400c860 tx=0x7f3114001630 comp rx=0 tx=0).stop 2022-06-27T11:48:12.824 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.813+0000 7f312346a700 1 -- 172.21.15.195:0/3839711817 shutdown_connections 2022-06-27T11:48:12.824 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.813+0000 7f312346a700 1 --2- 172.21.15.195:0/3839711817 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f30fc063d00 0x7f30fc0661b0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:12.825 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.813+0000 7f312346a700 1 --2- 172.21.15.195:0/3839711817 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f311c153b20 0x7f311c146360 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:12.825 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.813+0000 7f312346a700 1 --2- 172.21.15.195:0/3839711817 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f311c06e850 0x7f311c155d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:12.825 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.813+0000 7f312346a700 1 --2- 172.21.15.195:0/3839711817 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f311c0c34a0 0x7f311c151e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:12.826 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.813+0000 7f312346a700 1 -- 172.21.15.195:0/3839711817 >> 172.21.15.195:0/3839711817 conn(0x7f311c0c6350 msgr2=0x7f311c068b10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:12.826 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.813+0000 7f312346a700 1 -- 172.21.15.195:0/3839711817 shutdown_connections 2022-06-27T11:48:12.827 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.813+0000 7f312346a700 1 -- 172.21.15.195:0/3839711817 wait complete. 2022-06-27T11:48:12.829 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2022-06-27T11:47:46.368325+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2022-06-27T11:47:46.386132+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2022-06-27T11:47:50.901270+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2022-06-27T11:47:51.314719+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-06-27T11:48:12.831 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-06-27T11:48:12.955 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.955+0000 7f4ba8639700 1 Processor -- start 2022-06-27T11:48:12.955 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.955+0000 7f4ba8639700 1 -- start start 2022-06-27T11:48:12.960 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.956+0000 7f4ba8639700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f4ba0150950 0x7f4ba0156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:12.961 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.956+0000 7f4ba8639700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4ba006f620 0x7f4ba006d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:12.961 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.956+0000 7f4ba8639700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ba00c76b0 con 0x7f4ba0150580 2022-06-27T11:48:12.961 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.956+0000 7f4ba8639700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ba0157430 con 0x7f4ba0150950 2022-06-27T11:48:12.962 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.956+0000 7f4ba8639700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba00c8250 con 0x7f4ba006f620 2022-06-27T11:48:12.962 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.956+0000 7f4ba6bd6700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4ba006f620 0x7f4ba006d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:12.962 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.956+0000 7f4ba5bd4700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f4ba0150950 0x7f4ba0156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:12.963 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.956+0000 7f4ba6bd6700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4ba006f620 0x7f4ba006d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3301/0 says I am v2:172.21.15.195:34880/0 (socket says 172.21.15.195:34880) 2022-06-27T11:48:12.964 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.956+0000 7f4ba6bd6700 1 -- 172.21.15.195:0/2820812119 learned_addr learned my addr 172.21.15.195:0/2820812119 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:48:12.964 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.956+0000 7f4ba63d5700 1 --1- >> v1:172.21.15.202:6789/0 conn(0x7f4ba0150580 0x7f4ba00c7b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.202:6789/0 says I am v1:172.21.15.195:50172/0 (socket says 172.21.15.195:50172) 2022-06-27T11:48:12.965 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.956+0000 7f4ba5bd4700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f4ba0150950 0x7f4ba0156ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3300/0 says I am v2:172.21.15.195:40250/0 (socket says 172.21.15.195:40250) 2022-06-27T11:48:12.965 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.957+0000 7f4ba6bd6700 1 -- 172.21.15.195:0/2820812119 >> v2:172.21.15.202:3300/0 conn(0x7f4ba0150950 msgr2=0x7f4ba0156ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:12.966 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.957+0000 7f4ba6bd6700 1 --2- 172.21.15.195:0/2820812119 >> v2:172.21.15.202:3300/0 conn(0x7f4ba0150950 0x7f4ba0156ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:12.966 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.957+0000 7f4ba6bd6700 1 -- 172.21.15.195:0/2820812119 >> v1:172.21.15.202:6789/0 conn(0x7f4ba0150580 legacy=0x7f4ba00c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:12.967 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.957+0000 7f4ba6bd6700 1 -- 172.21.15.195:0/2820812119 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ba00bad70 con 0x7f4ba006f620 2022-06-27T11:48:12.967 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.957+0000 7f4ba5bd4700 1 --2- 172.21.15.195:0/2820812119 >> v2:172.21.15.202:3300/0 conn(0x7f4ba0150950 0x7f4ba0156ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:48:12.968 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.957+0000 7f4ba6bd6700 1 --2- 172.21.15.195:0/2820812119 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4ba006f620 0x7f4ba006d6a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4b98009f20 tx=0x7f4b98009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=989b2b2a03f36038 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:12.968 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.957+0000 7f4ba53d3700 1 -- 172.21.15.195:0/2820812119 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b9800e070 con 0x7f4ba006f620 2022-06-27T11:48:12.968 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.957+0000 7f4ba53d3700 1 -- 172.21.15.195:0/2820812119 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b9800b9d0 con 0x7f4ba006f620 2022-06-27T11:48:12.969 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.957+0000 7f4ba53d3700 1 -- 172.21.15.195:0/2820812119 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b980092e0 con 0x7f4ba006f620 2022-06-27T11:48:12.969 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.957+0000 7f4ba8639700 1 -- 172.21.15.195:0/2820812119 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4ba006f620 msgr2=0x7f4ba006d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:12.970 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.957+0000 7f4ba8639700 1 --2- 172.21.15.195:0/2820812119 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4ba006f620 0x7f4ba006d6a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4b98009f20 tx=0x7f4b98009840 comp rx=0 tx=0).stop 2022-06-27T11:48:12.970 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.957+0000 7f4ba8639700 1 -- 172.21.15.195:0/2820812119 shutdown_connections 2022-06-27T11:48:12.971 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.957+0000 7f4ba8639700 1 --2- 172.21.15.195:0/2820812119 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4ba006f620 0x7f4ba006d6a0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:12.971 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.957+0000 7f4ba8639700 1 --2- 172.21.15.195:0/2820812119 >> v2:172.21.15.202:3300/0 conn(0x7f4ba0150950 0x7f4ba0156ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:12.973 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.958+0000 7f4ba8639700 1 -- 172.21.15.195:0/2820812119 >> 172.21.15.195:0/2820812119 conn(0x7f4ba0063af0 msgr2=0x7f4ba0063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:12.973 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.958+0000 7f4ba8639700 1 -- 172.21.15.195:0/2820812119 shutdown_connections 2022-06-27T11:48:12.973 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.958+0000 7f4ba8639700 1 -- 172.21.15.195:0/2820812119 wait complete. 2022-06-27T11:48:12.974 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.959+0000 7f4ba8639700 1 Processor -- start 2022-06-27T11:48:12.974 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.959+0000 7f4ba8639700 1 -- start start 2022-06-27T11:48:12.974 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.959+0000 7f4ba8639700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4ba006f620 0x7f4ba00bde30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:12.975 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.959+0000 7f4ba8639700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4ba0150580 0x7f4ba00bd080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:12.975 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.959+0000 7f4ba8639700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4ba0150950 0x7f4ba00c06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:12.975 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.959+0000 7f4ba8639700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba01566a0 con 0x7f4ba006f620 2022-06-27T11:48:12.975 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.959+0000 7f4ba8639700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba00bdc60 con 0x7f4ba0150580 2022-06-27T11:48:12.976 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.959+0000 7f4ba8639700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba01560e0 con 0x7f4ba0150950 2022-06-27T11:48:12.976 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.959+0000 7f4ba63d5700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4ba006f620 0x7f4ba00bde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:12.976 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.959+0000 7f4ba63d5700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4ba006f620 0x7f4ba00bde30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:42130/0 (socket says 172.21.15.195:42130) 2022-06-27T11:48:12.976 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.959+0000 7f4ba63d5700 1 -- 172.21.15.195:0/3639060460 learned_addr learned my addr 172.21.15.195:0/3639060460 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:48:12.976 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.959+0000 7f4ba5bd4700 1 --2- 172.21.15.195:0/3639060460 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4ba0150580 0x7f4ba00bd080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:12.977 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.959+0000 7f4ba6bd6700 1 --2- 172.21.15.195:0/3639060460 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4ba0150950 0x7f4ba00c06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:12.977 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.960+0000 7f4ba63d5700 1 -- 172.21.15.195:0/3639060460 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4ba0150950 msgr2=0x7f4ba00c06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:12.978 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.960+0000 7f4ba63d5700 1 --2- 172.21.15.195:0/3639060460 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4ba0150950 0x7f4ba00c06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:12.978 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.960+0000 7f4ba63d5700 1 -- 172.21.15.195:0/3639060460 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4ba0150580 msgr2=0x7f4ba00bd080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:12.979 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.960+0000 7f4ba63d5700 1 --2- 172.21.15.195:0/3639060460 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4ba0150580 0x7f4ba00bd080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:12.979 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.960+0000 7f4ba63d5700 1 -- 172.21.15.195:0/3639060460 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b9c0092e0 con 0x7f4ba006f620 2022-06-27T11:48:12.979 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.960+0000 7f4ba63d5700 1 --2- 172.21.15.195:0/3639060460 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4ba006f620 0x7f4ba00bde30 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f4b9000c510 tx=0x7f4b90007e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:12.979 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.960+0000 7f4b8e7fc700 1 -- 172.21.15.195:0/3639060460 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b9000ce80 con 0x7f4ba006f620 2022-06-27T11:48:12.980 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.960+0000 7f4b8e7fc700 1 -- 172.21.15.195:0/3639060460 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b900072e0 con 0x7f4ba006f620 2022-06-27T11:48:12.980 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.960+0000 7f4ba8639700 1 -- 172.21.15.195:0/3639060460 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b98009ae0 con 0x7f4ba006f620 2022-06-27T11:48:12.980 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.960+0000 7f4b8e7fc700 1 -- 172.21.15.195:0/3639060460 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b900046f0 con 0x7f4ba006f620 2022-06-27T11:48:12.981 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.960+0000 7f4ba8639700 1 -- 172.21.15.195:0/3639060460 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ba00bb7f0 con 0x7f4ba006f620 2022-06-27T11:48:12.982 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.961+0000 7f4ba8639700 1 -- 172.21.15.195:0/3639060460 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ba00be970 con 0x7f4ba006f620 2022-06-27T11:48:12.982 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.961+0000 7f4b8e7fc700 1 -- 172.21.15.195:0/3639060460 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f4b90010070 con 0x7f4ba006f620 2022-06-27T11:48:12.983 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.962+0000 7f4b8e7fc700 1 --2- 172.21.15.195:0/3639060460 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4b80063d30 0x7f4b800661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:12.983 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.962+0000 7f4b8e7fc700 1 -- 172.21.15.195:0/3639060460 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f4b9007fe50 con 0x7f4ba006f620 2022-06-27T11:48:12.983 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.962+0000 7f4ba5bd4700 1 --2- 172.21.15.195:0/3639060460 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4b80063d30 0x7f4b800661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:12.984 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.962+0000 7f4ba5bd4700 1 --2- 172.21.15.195:0/3639060460 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4b80063d30 0x7f4b800661e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4ba00bc240 tx=0x7f4b9c009060 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:12.984 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:12.966+0000 7f4b8e7fc700 1 -- 172.21.15.195:0/3639060460 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f4b9004ea40 con 0x7f4ba006f620 2022-06-27T11:48:13.148 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.147+0000 7f4ba8639700 1 -- 172.21.15.195:0/3639060460 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f4ba00722c0 con 0x7f4ba006f620 2022-06-27T11:48:13.149 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.148+0000 7f4b8e7fc700 1 -- 172.21.15.195:0/3639060460 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v21) v1 ==== 92+0+10 (secure 0 0 0) 0x7f4b90052060 con 0x7f4ba006f620 2022-06-27T11:48:13.150 INFO:teuthology.orchestra.run.smithi195.stdout:pg_num: 8 2022-06-27T11:48:13.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.149+0000 7f4ba8639700 1 -- 172.21.15.195:0/3639060460 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4b80063d30 msgr2=0x7f4b800661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:13.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.149+0000 7f4ba8639700 1 --2- 172.21.15.195:0/3639060460 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4b80063d30 0x7f4b800661e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4ba00bc240 tx=0x7f4b9c009060 comp rx=0 tx=0).stop 2022-06-27T11:48:13.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.150+0000 7f4ba8639700 1 -- 172.21.15.195:0/3639060460 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4ba006f620 msgr2=0x7f4ba00bde30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:13.155 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.150+0000 7f4ba8639700 1 --2- 172.21.15.195:0/3639060460 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4ba006f620 0x7f4ba00bde30 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f4b9000c510 tx=0x7f4b90007e30 comp rx=0 tx=0).stop 2022-06-27T11:48:13.155 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.150+0000 7f4ba8639700 1 -- 172.21.15.195:0/3639060460 shutdown_connections 2022-06-27T11:48:13.156 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.150+0000 7f4ba8639700 1 --2- 172.21.15.195:0/3639060460 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4b80063d30 0x7f4b800661e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:13.156 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.150+0000 7f4ba8639700 1 --2- 172.21.15.195:0/3639060460 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4ba006f620 0x7f4ba00bde30 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:13.156 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.150+0000 7f4ba8639700 1 --2- 172.21.15.195:0/3639060460 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4ba0150950 0x7f4ba00c06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:13.157 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.150+0000 7f4ba8639700 1 --2- 172.21.15.195:0/3639060460 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4ba0150580 0x7f4ba00bd080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:13.158 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.150+0000 7f4ba8639700 1 -- 172.21.15.195:0/3639060460 >> 172.21.15.195:0/3639060460 conn(0x7f4ba0063af0 msgr2=0x7f4ba014f420 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:13.158 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.150+0000 7f4ba8639700 1 -- 172.21.15.195:0/3639060460 shutdown_connections 2022-06-27T11:48:13.159 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.150+0000 7f4ba8639700 1 -- 172.21.15.195:0/3639060460 wait complete. 2022-06-27T11:48:13.162 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2022-06-27T11:48:13.285 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.286+0000 7fa853ffd700 1 Processor -- start 2022-06-27T11:48:13.287 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.286+0000 7fa853ffd700 1 -- start start 2022-06-27T11:48:13.293 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.287+0000 7fa853ffd700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fa84c06e850 0x7fa84c1570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:13.293 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.287+0000 7fa853ffd700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa84c06fef0 0x7fa84c069de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:13.294 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.287+0000 7fa853ffd700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa84c06fba0 con 0x7fa84c0c34a0 2022-06-27T11:48:13.294 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.287+0000 7fa853ffd700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa84c1575e0 con 0x7fa84c06e850 2022-06-27T11:48:13.294 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.287+0000 7fa853ffd700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa84c06f5e0 con 0x7fa84c06fef0 2022-06-27T11:48:13.295 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.287+0000 7fa851598700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa84c06fef0 0x7fa84c069de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:13.295 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.287+0000 7fa851d99700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fa84c06e850 0x7fa84c1570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:13.296 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.287+0000 7fa85259a700 1 --1- >> v1:172.21.15.202:6789/0 conn(0x7fa84c0c34a0 0x7fa84c0c2920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.202:6789/0 says I am v1:172.21.15.195:50190/0 (socket says 172.21.15.195:50190) 2022-06-27T11:48:13.296 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.287+0000 7fa85259a700 1 -- 172.21.15.195:0/945007578 learned_addr learned my addr 172.21.15.195:0/945007578 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:48:13.297 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.288+0000 7fa851d99700 1 -- 172.21.15.195:0/945007578 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa84c06fef0 msgr2=0x7fa84c069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:13.297 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.288+0000 7fa851d99700 1 --2- 172.21.15.195:0/945007578 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa84c06fef0 0x7fa84c069de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:13.297 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.288+0000 7fa851d99700 1 -- 172.21.15.195:0/945007578 >> v1:172.21.15.202:6789/0 conn(0x7fa84c0c34a0 legacy=0x7fa84c0c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:13.297 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.288+0000 7fa851d99700 1 -- 172.21.15.195:0/945007578 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa84c06f8c0 con 0x7fa84c06e850 2022-06-27T11:48:13.297 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.288+0000 7fa851d99700 1 --2- 172.21.15.195:0/945007578 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa84c06e850 0x7fa84c1570a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa83c0097a0 tx=0x7fa83c009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8085606766e4e225 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:13.298 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.288+0000 7fa850d97700 1 -- 172.21.15.195:0/945007578 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa83c00f6b0 con 0x7fa84c06e850 2022-06-27T11:48:13.298 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.288+0000 7fa850d97700 1 -- 172.21.15.195:0/945007578 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa83c00f810 con 0x7fa84c06e850 2022-06-27T11:48:13.298 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.288+0000 7fa850d97700 1 -- 172.21.15.195:0/945007578 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa83c00fbb0 con 0x7fa84c06e850 2022-06-27T11:48:13.299 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.288+0000 7fa853ffd700 1 -- 172.21.15.195:0/945007578 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa84c06e850 msgr2=0x7fa84c1570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:13.299 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.288+0000 7fa853ffd700 1 --2- 172.21.15.195:0/945007578 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa84c06e850 0x7fa84c1570a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa83c0097a0 tx=0x7fa83c009060 comp rx=0 tx=0).stop 2022-06-27T11:48:13.299 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.289+0000 7fa853ffd700 1 -- 172.21.15.195:0/945007578 shutdown_connections 2022-06-27T11:48:13.300 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.289+0000 7fa853ffd700 1 --2- 172.21.15.195:0/945007578 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa84c06e850 0x7fa84c1570a0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:13.300 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.289+0000 7fa853ffd700 1 --2- 172.21.15.195:0/945007578 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa84c06fef0 0x7fa84c069de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:13.301 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.289+0000 7fa853ffd700 1 -- 172.21.15.195:0/945007578 >> 172.21.15.195:0/945007578 conn(0x7fa84c0c6350 msgr2=0x7fa84c0c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:13.301 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.289+0000 7fa853ffd700 1 -- 172.21.15.195:0/945007578 shutdown_connections 2022-06-27T11:48:13.301 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.289+0000 7fa853ffd700 1 -- 172.21.15.195:0/945007578 wait complete. 2022-06-27T11:48:13.305 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.290+0000 7fa853ffd700 1 Processor -- start 2022-06-27T11:48:13.306 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.290+0000 7fa853ffd700 1 -- start start 2022-06-27T11:48:13.306 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.290+0000 7fa853ffd700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa84c06e850 0x7fa84c14f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:13.306 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.290+0000 7fa853ffd700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa84c0c34a0 0x7fa84c14db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:13.307 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.290+0000 7fa853ffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa84c157120 0x7fa84c147020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:13.307 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.290+0000 7fa853ffd700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa84c1563a0 con 0x7fa84c157120 2022-06-27T11:48:13.307 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.290+0000 7fa853ffd700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa84c165ea0 con 0x7fa84c06e850 2022-06-27T11:48:13.308 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.290+0000 7fa853ffd700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa84c0c8930 con 0x7fa84c0c34a0 2022-06-27T11:48:13.308 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.291+0000 7fa85259a700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa84c157120 0x7fa84c147020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:13.309 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.291+0000 7fa85259a700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa84c157120 0x7fa84c147020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:42148/0 (socket says 172.21.15.195:42148) 2022-06-27T11:48:13.310 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.291+0000 7fa85259a700 1 -- 172.21.15.195:0/3720948860 learned_addr learned my addr 172.21.15.195:0/3720948860 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:48:13.310 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.291+0000 7fa851d99700 1 --2- 172.21.15.195:0/3720948860 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa84c06e850 0x7fa84c14f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:13.311 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.291+0000 7fa851598700 1 --2- 172.21.15.195:0/3720948860 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa84c0c34a0 0x7fa84c14db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:13.311 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.291+0000 7fa85259a700 1 -- 172.21.15.195:0/3720948860 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa84c0c34a0 msgr2=0x7fa84c14db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:13.311 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.291+0000 7fa85259a700 1 --2- 172.21.15.195:0/3720948860 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa84c0c34a0 0x7fa84c14db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:13.312 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.291+0000 7fa85259a700 1 -- 172.21.15.195:0/3720948860 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa84c06e850 msgr2=0x7fa84c14f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:13.312 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.291+0000 7fa85259a700 1 --2- 172.21.15.195:0/3720948860 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa84c06e850 0x7fa84c14f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:13.312 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.291+0000 7fa85259a700 1 -- 172.21.15.195:0/3720948860 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa83c0092e0 con 0x7fa84c157120 2022-06-27T11:48:13.312 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.291+0000 7fa85259a700 1 --2- 172.21.15.195:0/3720948860 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa84c157120 0x7fa84c147020 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fa84400c870 tx=0x7fa8440015e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:13.313 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.291+0000 7fa839ffb700 1 -- 172.21.15.195:0/3720948860 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa844005d70 con 0x7fa84c157120 2022-06-27T11:48:13.313 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.291+0000 7fa839ffb700 1 -- 172.21.15.195:0/3720948860 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8440070f0 con 0x7fa84c157120 2022-06-27T11:48:13.313 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.291+0000 7fa853ffd700 1 -- 172.21.15.195:0/3720948860 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa84c0c8bd0 con 0x7fa84c157120 2022-06-27T11:48:13.314 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.291+0000 7fa839ffb700 1 -- 172.21.15.195:0/3720948860 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa844007470 con 0x7fa84c157120 2022-06-27T11:48:13.314 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.291+0000 7fa853ffd700 1 -- 172.21.15.195:0/3720948860 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa84c165570 con 0x7fa84c157120 2022-06-27T11:48:13.314 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.292+0000 7fa853ffd700 1 -- 172.21.15.195:0/3720948860 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa84c0722c0 con 0x7fa84c157120 2022-06-27T11:48:13.314 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.293+0000 7fa839ffb700 1 -- 172.21.15.195:0/3720948860 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fa844010070 con 0x7fa84c157120 2022-06-27T11:48:13.315 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.293+0000 7fa839ffb700 1 --2- 172.21.15.195:0/3720948860 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa82c063d00 0x7fa82c0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:13.315 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.293+0000 7fa839ffb700 1 -- 172.21.15.195:0/3720948860 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fa844080220 con 0x7fa84c157120 2022-06-27T11:48:13.315 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.293+0000 7fa851d99700 1 --2- 172.21.15.195:0/3720948860 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa82c063d00 0x7fa82c0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:13.315 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.293+0000 7fa851d99700 1 --2- 172.21.15.195:0/3720948860 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa82c063d00 0x7fa82c0661b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa83c0097a0 tx=0x7fa83c019000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:13.316 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.297+0000 7fa839ffb700 1 -- 172.21.15.195:0/3720948860 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fa84404ee90 con 0x7fa84c157120 2022-06-27T11:48:13.475 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.475+0000 7fa853ffd700 1 -- 172.21.15.195:0/3720948860 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa84c06f240 con 0x7fa84c157120 2022-06-27T11:48:13.476 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.476+0000 7fa839ffb700 1 -- 172.21.15.195:0/3720948860 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa8440524b0 con 0x7fa84c157120 2022-06-27T11:48:13.477 INFO:teuthology.orchestra.run.smithi195.stdout:pg_num: 1 2022-06-27T11:48:13.479 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.477+0000 7fa853ffd700 1 -- 172.21.15.195:0/3720948860 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa82c063d00 msgr2=0x7fa82c0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:13.480 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.477+0000 7fa853ffd700 1 --2- 172.21.15.195:0/3720948860 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa82c063d00 0x7fa82c0661b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa83c0097a0 tx=0x7fa83c019000 comp rx=0 tx=0).stop 2022-06-27T11:48:13.480 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.477+0000 7fa853ffd700 1 -- 172.21.15.195:0/3720948860 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa84c157120 msgr2=0x7fa84c147020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:13.481 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.477+0000 7fa853ffd700 1 --2- 172.21.15.195:0/3720948860 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa84c157120 0x7fa84c147020 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fa84400c870 tx=0x7fa8440015e0 comp rx=0 tx=0).stop 2022-06-27T11:48:13.481 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.477+0000 7fa853ffd700 1 -- 172.21.15.195:0/3720948860 shutdown_connections 2022-06-27T11:48:13.482 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.477+0000 7fa853ffd700 1 --2- 172.21.15.195:0/3720948860 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa82c063d00 0x7fa82c0661b0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:13.482 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.478+0000 7fa853ffd700 1 --2- 172.21.15.195:0/3720948860 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa84c157120 0x7fa84c147020 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:13.483 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.478+0000 7fa853ffd700 1 --2- 172.21.15.195:0/3720948860 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa84c06e850 0x7fa84c14f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:13.483 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.478+0000 7fa853ffd700 1 --2- 172.21.15.195:0/3720948860 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa84c0c34a0 0x7fa84c14db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:13.483 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.478+0000 7fa853ffd700 1 -- 172.21.15.195:0/3720948860 >> 172.21.15.195:0/3720948860 conn(0x7fa84c0c6350 msgr2=0x7fa84c150430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:13.483 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.478+0000 7fa853ffd700 1 -- 172.21.15.195:0/3720948860 shutdown_connections 2022-06-27T11:48:13.483 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.478+0000 7fa853ffd700 1 -- 172.21.15.195:0/3720948860 wait complete. 2022-06-27T11:48:13.490 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-06-27T11:48:13.614 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.614+0000 7f0651f07700 1 Processor -- start 2022-06-27T11:48:13.614 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.614+0000 7f0651f07700 1 -- start start 2022-06-27T11:48:13.617 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.615+0000 7f0651f07700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f064c06f640 0x7f064c156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:13.617 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.615+0000 7f0651f07700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f064c0c4160 0x7f064c0c1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:13.618 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.615+0000 7f0651f07700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f064c0c39e0 0x7f064c1535a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:13.618 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.615+0000 7f0651f07700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f064c0c1a20 con 0x7f064c0c39e0 2022-06-27T11:48:13.618 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.615+0000 7f0651f07700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f064c0c1130 con 0x7f064c0c4160 2022-06-27T11:48:13.618 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.615+0000 7f0651f07700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f064c06ea90 con 0x7f064c06f640 2022-06-27T11:48:13.619 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.615+0000 7f064bfff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f064c0c39e0 0x7f064c1535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:13.619 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.615+0000 7f064bfff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f064c0c39e0 0x7f064c1535a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:42158/0 (socket says 172.21.15.195:42158) 2022-06-27T11:48:13.619 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.615+0000 7f064bfff700 1 -- 172.21.15.195:0/1789217904 learned_addr learned my addr 172.21.15.195:0/1789217904 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:48:13.619 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.615+0000 7f064affd700 1 --2- 172.21.15.195:0/1789217904 >> v2:172.21.15.202:3300/0 conn(0x7f064c0c4160 0x7f064c0c1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:13.620 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.615+0000 7f064b7fe700 1 --2- 172.21.15.195:0/1789217904 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f064c06f640 0x7f064c156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:13.620 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.616+0000 7f064bfff700 1 -- 172.21.15.195:0/1789217904 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f064c06f640 msgr2=0x7f064c156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:13.620 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.616+0000 7f064bfff700 1 --2- 172.21.15.195:0/1789217904 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f064c06f640 0x7f064c156ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:13.620 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.616+0000 7f064bfff700 1 -- 172.21.15.195:0/1789217904 >> v2:172.21.15.202:3300/0 conn(0x7f064c0c4160 msgr2=0x7f064c0c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:13.621 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.616+0000 7f064bfff700 1 --2- 172.21.15.195:0/1789217904 >> v2:172.21.15.202:3300/0 conn(0x7f064c0c4160 0x7f064c0c1eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:13.621 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.616+0000 7f064bfff700 1 -- 172.21.15.195:0/1789217904 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f064c06ed70 con 0x7f064c0c39e0 2022-06-27T11:48:13.621 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.616+0000 7f064bfff700 1 --2- 172.21.15.195:0/1789217904 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f064c0c39e0 0x7f064c1535a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f064c06d100 tx=0x7f0644009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e92190a590d669c server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:13.621 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.616+0000 7f064a7fc700 1 -- 172.21.15.195:0/1789217904 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f064400e070 con 0x7f064c0c39e0 2022-06-27T11:48:13.622 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.616+0000 7f064a7fc700 1 -- 172.21.15.195:0/1789217904 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f064400ba00 con 0x7f064c0c39e0 2022-06-27T11:48:13.622 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.616+0000 7f064a7fc700 1 -- 172.21.15.195:0/1789217904 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06440092e0 con 0x7f064c0c39e0 2022-06-27T11:48:13.622 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.616+0000 7f0651f07700 1 -- 172.21.15.195:0/1789217904 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f064c0c39e0 msgr2=0x7f064c1535a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:13.623 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.616+0000 7f0651f07700 1 --2- 172.21.15.195:0/1789217904 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f064c0c39e0 0x7f064c1535a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f064c06d100 tx=0x7f0644009840 comp rx=0 tx=0).stop 2022-06-27T11:48:13.623 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.617+0000 7f0651f07700 1 -- 172.21.15.195:0/1789217904 shutdown_connections 2022-06-27T11:48:13.623 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.617+0000 7f0651f07700 1 --2- 172.21.15.195:0/1789217904 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f064c0c39e0 0x7f064c1535a0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:13.623 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.617+0000 7f0651f07700 1 --2- 172.21.15.195:0/1789217904 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f064c06f640 0x7f064c156ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:13.623 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.617+0000 7f0651f07700 1 --2- 172.21.15.195:0/1789217904 >> v2:172.21.15.202:3300/0 conn(0x7f064c0c4160 0x7f064c0c1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:13.624 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.617+0000 7f0651f07700 1 -- 172.21.15.195:0/1789217904 >> 172.21.15.195:0/1789217904 conn(0x7f064c0c6350 msgr2=0x7f064c0c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:13.624 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.617+0000 7f0651f07700 1 -- 172.21.15.195:0/1789217904 shutdown_connections 2022-06-27T11:48:13.624 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.617+0000 7f0651f07700 1 -- 172.21.15.195:0/1789217904 wait complete. 2022-06-27T11:48:13.625 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.618+0000 7f0651f07700 1 Processor -- start 2022-06-27T11:48:13.625 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.618+0000 7f0651f07700 1 -- start start 2022-06-27T11:48:13.629 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.619+0000 7f0651f07700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f064c06f640 0x7f064c166670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:13.630 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.619+0000 7f0651f07700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f064c0c39e0 0x7f064c0bb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:13.630 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.619+0000 7f0651f07700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f064c0c4160 0x7f064c1545c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:13.630 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.619+0000 7f0651f07700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f064c159570 con 0x7f064c06f640 2022-06-27T11:48:13.630 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.619+0000 7f0651f07700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f064c158cd0 con 0x7f064c0c39e0 2022-06-27T11:48:13.631 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.619+0000 7f0651f07700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f064c158fb0 con 0x7f064c0c4160 2022-06-27T11:48:13.631 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.619+0000 7f064b7fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f064c06f640 0x7f064c166670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:13.631 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.619+0000 7f064b7fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f064c06f640 0x7f064c166670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:42160/0 (socket says 172.21.15.195:42160) 2022-06-27T11:48:13.631 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.619+0000 7f064b7fe700 1 -- 172.21.15.195:0/1111072589 learned_addr learned my addr 172.21.15.195:0/1111072589 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:48:13.632 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.619+0000 7f064affd700 1 --2- 172.21.15.195:0/1111072589 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f064c0c39e0 0x7f064c0bb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:13.632 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.619+0000 7f064b7fe700 1 -- 172.21.15.195:0/1111072589 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f064c0c4160 msgr2=0x7f064c1545c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:13.632 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.619+0000 7f064b7fe700 1 --2- 172.21.15.195:0/1111072589 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f064c0c4160 0x7f064c1545c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:13.633 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.619+0000 7f064b7fe700 1 -- 172.21.15.195:0/1111072589 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f064c0c39e0 msgr2=0x7f064c0bb5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:13.633 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.619+0000 7f064b7fe700 1 --2- 172.21.15.195:0/1111072589 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f064c0c39e0 0x7f064c0bb5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:13.633 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.619+0000 7f064b7fe700 1 -- 172.21.15.195:0/1111072589 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0644009ae0 con 0x7f064c06f640 2022-06-27T11:48:13.634 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.619+0000 7f064b7fe700 1 --2- 172.21.15.195:0/1111072589 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f064c06f640 0x7f064c166670 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f064000e970 tx=0x7f0640009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:13.634 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.619+0000 7f06337fe700 1 -- 172.21.15.195:0/1111072589 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f064000c850 con 0x7f064c06f640 2022-06-27T11:48:13.634 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.619+0000 7f06337fe700 1 -- 172.21.15.195:0/1111072589 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f064000c9b0 con 0x7f064c06f640 2022-06-27T11:48:13.635 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.619+0000 7f0651f07700 1 -- 172.21.15.195:0/1111072589 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f064c159290 con 0x7f064c06f640 2022-06-27T11:48:13.635 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.619+0000 7f06337fe700 1 -- 172.21.15.195:0/1111072589 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f064000cd30 con 0x7f064c06f640 2022-06-27T11:48:13.635 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.619+0000 7f0651f07700 1 -- 172.21.15.195:0/1111072589 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f064c0bb0c0 con 0x7f064c06f640 2022-06-27T11:48:13.635 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.620+0000 7f0651f07700 1 -- 172.21.15.195:0/1111072589 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f064c0bbb20 con 0x7f064c06f640 2022-06-27T11:48:13.635 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.621+0000 7f06337fe700 1 -- 172.21.15.195:0/1111072589 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f0640019030 con 0x7f064c06f640 2022-06-27T11:48:13.636 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.621+0000 7f06337fe700 1 --2- 172.21.15.195:0/1111072589 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f062c063d00 0x7f062c0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:13.636 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.621+0000 7f064affd700 1 --2- 172.21.15.195:0/1111072589 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f062c063d00 0x7f062c0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:13.636 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.621+0000 7f06337fe700 1 -- 172.21.15.195:0/1111072589 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f06400808e0 con 0x7f064c06f640 2022-06-27T11:48:13.637 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.622+0000 7f064affd700 1 --2- 172.21.15.195:0/1111072589 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f062c063d00 0x7f062c0661b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f064c1677b0 tx=0x7f063c008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:13.637 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.625+0000 7f06337fe700 1 -- 172.21.15.195:0/1111072589 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f064004f4d0 con 0x7f064c06f640 2022-06-27T11:48:13.801 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.801+0000 7f0651f07700 1 -- 172.21.15.195:0/1111072589 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f064c0bb430 con 0x7f064c06f640 2022-06-27T11:48:13.801 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.802+0000 7f06337fe700 1 -- 172.21.15.195:0/1111072589 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f06400170a0 con 0x7f064c06f640 2022-06-27T11:48:13.802 INFO:teuthology.orchestra.run.smithi195.stdout:pg_num: 32 2022-06-27T11:48:13.803 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.803+0000 7f0651f07700 1 -- 172.21.15.195:0/1111072589 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f062c063d00 msgr2=0x7f062c0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:13.803 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.803+0000 7f0651f07700 1 --2- 172.21.15.195:0/1111072589 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f062c063d00 0x7f062c0661b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f064c1677b0 tx=0x7f063c008040 comp rx=0 tx=0).stop 2022-06-27T11:48:13.803 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.803+0000 7f0651f07700 1 -- 172.21.15.195:0/1111072589 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f064c06f640 msgr2=0x7f064c166670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:13.803 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.803+0000 7f0651f07700 1 --2- 172.21.15.195:0/1111072589 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f064c06f640 0x7f064c166670 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f064000e970 tx=0x7f0640009060 comp rx=0 tx=0).stop 2022-06-27T11:48:13.805 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.804+0000 7f0651f07700 1 -- 172.21.15.195:0/1111072589 shutdown_connections 2022-06-27T11:48:13.805 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.804+0000 7f0651f07700 1 --2- 172.21.15.195:0/1111072589 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f062c063d00 0x7f062c0661b0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:13.806 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.804+0000 7f0651f07700 1 --2- 172.21.15.195:0/1111072589 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f064c06f640 0x7f064c166670 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:13.806 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.804+0000 7f0651f07700 1 --2- 172.21.15.195:0/1111072589 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f064c0c4160 0x7f064c1545c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:13.806 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.804+0000 7f0651f07700 1 --2- 172.21.15.195:0/1111072589 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f064c0c39e0 0x7f064c0bb5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:13.806 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.804+0000 7f0651f07700 1 -- 172.21.15.195:0/1111072589 >> 172.21.15.195:0/1111072589 conn(0x7f064c0c6350 msgr2=0x7f064c152880 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:13.807 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.804+0000 7f0651f07700 1 -- 172.21.15.195:0/1111072589 shutdown_connections 2022-06-27T11:48:13.808 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.804+0000 7f0651f07700 1 -- 172.21.15.195:0/1111072589 wait complete. 2022-06-27T11:48:13.817 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-06-27T11:48:13.943 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.943+0000 7f2465a22700 1 Processor -- start 2022-06-27T11:48:13.943 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.943+0000 7f2465a22700 1 -- start start 2022-06-27T11:48:13.949 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.944+0000 7f2465a22700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f24600c40d0 0x7f2460157010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:13.949 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.944+0000 7f2465a22700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f24600c6190 0x7f2460067ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:13.950 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.944+0000 7f2465a22700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2460156b80 con 0x7f24600c44a0 2022-06-27T11:48:13.950 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.944+0000 7f2465a22700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f246006a040 con 0x7f24600c40d0 2022-06-27T11:48:13.950 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.944+0000 7f2465a22700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f246006a300 con 0x7f24600c6190 2022-06-27T11:48:13.951 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.944+0000 7f245effd700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f24600c40d0 0x7f2460157010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:13.951 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.944+0000 7f245effd700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f24600c40d0 0x7f2460157010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:42168/0 (socket says 172.21.15.195:42168) 2022-06-27T11:48:13.951 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.944+0000 7f245effd700 1 -- 172.21.15.195:0/1888826178 learned_addr learned my addr 172.21.15.195:0/1888826178 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:48:13.951 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.944+0000 7f245e7fc700 1 --2- 172.21.15.195:0/1888826178 >> v2:172.21.15.202:3300/0 conn(0x7f24600c6190 0x7f2460067ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:13.952 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.944+0000 7f245effd700 1 -- 172.21.15.195:0/1888826178 >> v2:172.21.15.202:3300/0 conn(0x7f24600c6190 msgr2=0x7f2460067ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:13.952 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.944+0000 7f245effd700 1 --2- 172.21.15.195:0/1888826178 >> v2:172.21.15.202:3300/0 conn(0x7f24600c6190 0x7f2460067ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:13.952 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.944+0000 7f245effd700 1 -- 172.21.15.195:0/1888826178 >> v1:172.21.15.202:6789/0 conn(0x7f24600c44a0 legacy=0x7f246014cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:13.952 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.944+0000 7f245effd700 1 -- 172.21.15.195:0/1888826178 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f246006a5e0 con 0x7f24600c40d0 2022-06-27T11:48:13.952 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.944+0000 7f245e7fc700 1 --2- 172.21.15.195:0/1888826178 >> v2:172.21.15.202:3300/0 conn(0x7f24600c6190 0x7f2460067ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T11:48:13.953 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.944+0000 7f245effd700 1 --2- 172.21.15.195:0/1888826178 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f24600c40d0 0x7f2460157010 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f24600c0680 tx=0x7f2454009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e4de97c6100f6b2 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:13.953 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.945+0000 7f245dffb700 1 -- 172.21.15.195:0/1888826178 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2454004030 con 0x7f24600c40d0 2022-06-27T11:48:13.953 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.945+0000 7f245dffb700 1 -- 172.21.15.195:0/1888826178 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2454004190 con 0x7f24600c40d0 2022-06-27T11:48:13.953 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.945+0000 7f245dffb700 1 -- 172.21.15.195:0/1888826178 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2454011630 con 0x7f24600c40d0 2022-06-27T11:48:13.954 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.945+0000 7f2465a22700 1 -- 172.21.15.195:0/1888826178 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f24600c40d0 msgr2=0x7f2460157010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:13.954 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.945+0000 7f2465a22700 1 --2- 172.21.15.195:0/1888826178 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f24600c40d0 0x7f2460157010 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f24600c0680 tx=0x7f2454009120 comp rx=0 tx=0).stop 2022-06-27T11:48:13.954 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.945+0000 7f2465a22700 1 -- 172.21.15.195:0/1888826178 shutdown_connections 2022-06-27T11:48:13.955 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.945+0000 7f2465a22700 1 --2- 172.21.15.195:0/1888826178 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f24600c40d0 0x7f2460157010 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:13.955 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.945+0000 7f2465a22700 1 --2- 172.21.15.195:0/1888826178 >> v2:172.21.15.202:3300/0 conn(0x7f24600c6190 0x7f2460067ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:13.955 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.945+0000 7f2465a22700 1 -- 172.21.15.195:0/1888826178 >> 172.21.15.195:0/1888826178 conn(0x7f24600ccb80 msgr2=0x7f24600ccf80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:13.955 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.945+0000 7f2465a22700 1 -- 172.21.15.195:0/1888826178 shutdown_connections 2022-06-27T11:48:13.956 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.945+0000 7f2465a22700 1 -- 172.21.15.195:0/1888826178 wait complete. 2022-06-27T11:48:13.956 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.946+0000 7f2465a22700 1 Processor -- start 2022-06-27T11:48:13.956 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.947+0000 7f2465a22700 1 -- start start 2022-06-27T11:48:13.956 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.947+0000 7f2465a22700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f24600c40d0 0x7f246014f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:13.957 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.947+0000 7f2465a22700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f24600c44a0 0x7f2460151b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:13.957 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.947+0000 7f2465a22700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f24600c6190 0x7f246015c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:13.957 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.947+0000 7f2465a22700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f246016a1f0 con 0x7f24600c44a0 2022-06-27T11:48:13.957 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.947+0000 7f2465a22700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2460156620 con 0x7f24600c40d0 2022-06-27T11:48:13.958 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.947+0000 7f2465a22700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2460153c70 con 0x7f24600c6190 2022-06-27T11:48:13.958 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.947+0000 7f245e7fc700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f24600c44a0 0x7f2460151b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:13.958 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.947+0000 7f245e7fc700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f24600c44a0 0x7f2460151b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:42174/0 (socket says 172.21.15.195:42174) 2022-06-27T11:48:13.959 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.947+0000 7f245e7fc700 1 -- 172.21.15.195:0/3240264903 learned_addr learned my addr 172.21.15.195:0/3240264903 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:48:13.959 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.947+0000 7f245effd700 1 --2- 172.21.15.195:0/3240264903 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f24600c40d0 0x7f246014f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:13.959 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.947+0000 7f245f7fe700 1 --2- 172.21.15.195:0/3240264903 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f24600c6190 0x7f246015c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:13.960 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.948+0000 7f245e7fc700 1 -- 172.21.15.195:0/3240264903 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f24600c6190 msgr2=0x7f246015c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:13.960 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.948+0000 7f245e7fc700 1 --2- 172.21.15.195:0/3240264903 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f24600c6190 0x7f246015c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:13.960 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.948+0000 7f245e7fc700 1 -- 172.21.15.195:0/3240264903 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f24600c40d0 msgr2=0x7f246014f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:13.960 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.948+0000 7f245e7fc700 1 --2- 172.21.15.195:0/3240264903 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f24600c40d0 0x7f246014f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:13.961 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.948+0000 7f245e7fc700 1 -- 172.21.15.195:0/3240264903 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24540093a0 con 0x7f24600c44a0 2022-06-27T11:48:13.961 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.948+0000 7f245e7fc700 1 --2- 172.21.15.195:0/3240264903 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f24600c44a0 0x7f2460151b90 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f245000c880 tx=0x7f2450008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:13.961 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.948+0000 7f2446ffd700 1 -- 172.21.15.195:0/3240264903 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2450011600 con 0x7f24600c44a0 2022-06-27T11:48:13.961 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.948+0000 7f2446ffd700 1 -- 172.21.15.195:0/3240264903 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2450011760 con 0x7f24600c44a0 2022-06-27T11:48:13.962 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.948+0000 7f2465a22700 1 -- 172.21.15.195:0/3240264903 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24601537e0 con 0x7f24600c44a0 2022-06-27T11:48:13.962 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.948+0000 7f2446ffd700 1 -- 172.21.15.195:0/3240264903 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2450011b20 con 0x7f24600c44a0 2022-06-27T11:48:13.962 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.948+0000 7f2465a22700 1 -- 172.21.15.195:0/3240264903 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24600c3a80 con 0x7f24600c44a0 2022-06-27T11:48:13.962 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.949+0000 7f2465a22700 1 -- 172.21.15.195:0/3240264903 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2460153970 con 0x7f24600c44a0 2022-06-27T11:48:13.963 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.949+0000 7f2446ffd700 1 -- 172.21.15.195:0/3240264903 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f2450021070 con 0x7f24600c44a0 2022-06-27T11:48:13.963 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.950+0000 7f2446ffd700 1 --2- 172.21.15.195:0/3240264903 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2440063db0 0x7f2440066260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:13.963 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.950+0000 7f2446ffd700 1 -- 172.21.15.195:0/3240264903 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f2450080550 con 0x7f24600c44a0 2022-06-27T11:48:13.963 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.950+0000 7f245effd700 1 --2- 172.21.15.195:0/3240264903 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2440063db0 0x7f2440066260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:13.964 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.950+0000 7f245effd700 1 --2- 172.21.15.195:0/3240264903 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2440063db0 0x7f2440066260 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2454009160 tx=0x7f2454016000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:13.965 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:13.954+0000 7f2446ffd700 1 -- 172.21.15.195:0/3240264903 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f245004f050 con 0x7f24600c44a0 2022-06-27T11:48:14.129 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.130+0000 7f2465a22700 1 -- 172.21.15.195:0/3240264903 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f24600722c0 con 0x7f24600c44a0 2022-06-27T11:48:14.130 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.130+0000 7f2446ffd700 1 -- 172.21.15.195:0/3240264903 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f245001f0a0 con 0x7f24600c44a0 2022-06-27T11:48:14.131 INFO:teuthology.orchestra.run.smithi195.stdout:pg_num: 64 2022-06-27T11:48:14.133 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.133+0000 7f2465a22700 1 -- 172.21.15.195:0/3240264903 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2440063db0 msgr2=0x7f2440066260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:14.134 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.133+0000 7f2465a22700 1 --2- 172.21.15.195:0/3240264903 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2440063db0 0x7f2440066260 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2454009160 tx=0x7f2454016000 comp rx=0 tx=0).stop 2022-06-27T11:48:14.134 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.133+0000 7f2465a22700 1 -- 172.21.15.195:0/3240264903 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f24600c44a0 msgr2=0x7f2460151b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:14.135 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.133+0000 7f2465a22700 1 --2- 172.21.15.195:0/3240264903 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f24600c44a0 0x7f2460151b90 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f245000c880 tx=0x7f2450008040 comp rx=0 tx=0).stop 2022-06-27T11:48:14.135 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.133+0000 7f2465a22700 1 -- 172.21.15.195:0/3240264903 shutdown_connections 2022-06-27T11:48:14.136 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.133+0000 7f2465a22700 1 --2- 172.21.15.195:0/3240264903 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2440063db0 0x7f2440066260 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:14.136 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.133+0000 7f2465a22700 1 --2- 172.21.15.195:0/3240264903 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f24600c6190 0x7f246015c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:14.137 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.133+0000 7f2465a22700 1 --2- 172.21.15.195:0/3240264903 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f24600c40d0 0x7f246014f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:14.137 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.133+0000 7f2465a22700 1 --2- 172.21.15.195:0/3240264903 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f24600c44a0 0x7f2460151b90 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:14.138 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.133+0000 7f2465a22700 1 -- 172.21.15.195:0/3240264903 >> 172.21.15.195:0/3240264903 conn(0x7f24600ccb80 msgr2=0x7f24600cb430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:14.138 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.133+0000 7f2465a22700 1 -- 172.21.15.195:0/3240264903 shutdown_connections 2022-06-27T11:48:14.139 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.133+0000 7f2465a22700 1 -- 172.21.15.195:0/3240264903 wait complete. 2022-06-27T11:48:14.144 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:48:14.272 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.265+0000 7fc6e8a0e700 1 Processor -- start 2022-06-27T11:48:14.273 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.265+0000 7fc6e8a0e700 1 -- start start 2022-06-27T11:48:14.273 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.266+0000 7fc6e8a0e700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc6e406a450 0x7fc6e4063af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:14.273 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.266+0000 7fc6e8a0e700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fc6e41508d0 0x7fc6e40bbf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:14.274 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.266+0000 7fc6e8a0e700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6e4064030 con 0x7fc6e4150500 2022-06-27T11:48:14.274 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.266+0000 7fc6e8a0e700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6e40c4160 con 0x7fc6e41508d0 2022-06-27T11:48:14.274 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.266+0000 7fc6e8a0e700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e40c4440 con 0x7fc6e406a450 2022-06-27T11:48:14.274 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.266+0000 7fc6e2d9d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fc6e41508d0 0x7fc6e40bbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:14.275 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.266+0000 7fc6e2d9d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fc6e41508d0 0x7fc6e40bbf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:42186/0 (socket says 172.21.15.195:42186) 2022-06-27T11:48:14.275 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.266+0000 7fc6e2d9d700 1 -- 172.21.15.195:0/1511354279 learned_addr learned my addr 172.21.15.195:0/1511354279 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:48:14.275 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.266+0000 7fc6e1d9b700 1 --2- 172.21.15.195:0/1511354279 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc6e406a450 0x7fc6e4063af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:14.275 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.266+0000 7fc6e2d9d700 1 -- 172.21.15.195:0/1511354279 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc6e406a450 msgr2=0x7fc6e4063af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:14.276 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.266+0000 7fc6e2d9d700 1 --2- 172.21.15.195:0/1511354279 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc6e406a450 0x7fc6e4063af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:14.276 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.266+0000 7fc6e2d9d700 1 -- 172.21.15.195:0/1511354279 >> v1:172.21.15.202:6789/0 conn(0x7fc6e4150500 legacy=0x7fc6e40c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:14.276 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.266+0000 7fc6e2d9d700 1 -- 172.21.15.195:0/1511354279 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6e415a800 con 0x7fc6e41508d0 2022-06-27T11:48:14.277 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.267+0000 7fc6e2d9d700 1 --2- 172.21.15.195:0/1511354279 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc6e41508d0 0x7fc6e40bbf50 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc009f20 tx=0x7fc6dc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15d8e3784d18c265 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:14.278 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.267+0000 7fc6e159a700 1 -- 172.21.15.195:0/1511354279 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6dc00e070 con 0x7fc6e41508d0 2022-06-27T11:48:14.278 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.267+0000 7fc6e159a700 1 -- 172.21.15.195:0/1511354279 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6dc00ba80 con 0x7fc6e41508d0 2022-06-27T11:48:14.278 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.267+0000 7fc6e159a700 1 -- 172.21.15.195:0/1511354279 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6dc0092e0 con 0x7fc6e41508d0 2022-06-27T11:48:14.278 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.267+0000 7fc6e8a0e700 1 -- 172.21.15.195:0/1511354279 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc6e41508d0 msgr2=0x7fc6e40bbf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:14.279 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.267+0000 7fc6e8a0e700 1 --2- 172.21.15.195:0/1511354279 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc6e41508d0 0x7fc6e40bbf50 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc009f20 tx=0x7fc6dc009840 comp rx=0 tx=0).stop 2022-06-27T11:48:14.279 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.267+0000 7fc6e8a0e700 1 -- 172.21.15.195:0/1511354279 shutdown_connections 2022-06-27T11:48:14.279 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.267+0000 7fc6e8a0e700 1 --2- 172.21.15.195:0/1511354279 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc6e41508d0 0x7fc6e40bbf50 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:14.279 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.267+0000 7fc6e8a0e700 1 --2- 172.21.15.195:0/1511354279 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc6e406a450 0x7fc6e4063af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:14.280 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.267+0000 7fc6e8a0e700 1 -- 172.21.15.195:0/1511354279 >> 172.21.15.195:0/1511354279 conn(0x7fc6e40c9b00 msgr2=0x7fc6e40c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:14.280 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.267+0000 7fc6e8a0e700 1 -- 172.21.15.195:0/1511354279 shutdown_connections 2022-06-27T11:48:14.280 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.267+0000 7fc6e8a0e700 1 -- 172.21.15.195:0/1511354279 wait complete. 2022-06-27T11:48:14.280 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.269+0000 7fc6e8a0e700 1 Processor -- start 2022-06-27T11:48:14.281 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.269+0000 7fc6e8a0e700 1 -- start start 2022-06-27T11:48:14.281 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.269+0000 7fc6e8a0e700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc6e406a450 0x7fc6e4166dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:14.281 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.269+0000 7fc6e8a0e700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc6e4150500 0x7fc6e4069870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:14.282 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.269+0000 7fc6e8a0e700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc6e41508d0 0x7fc6e414cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:14.282 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.269+0000 7fc6e8a0e700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e40c8de0 con 0x7fc6e41508d0 2022-06-27T11:48:14.282 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.269+0000 7fc6e8a0e700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e4166940 con 0x7fc6e4150500 2022-06-27T11:48:14.282 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.269+0000 7fc6e8a0e700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e4159450 con 0x7fc6e406a450 2022-06-27T11:48:14.283 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.269+0000 7fc6e2d9d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc6e41508d0 0x7fc6e414cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:14.283 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.269+0000 7fc6e2d9d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc6e41508d0 0x7fc6e414cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:42192/0 (socket says 172.21.15.195:42192) 2022-06-27T11:48:14.283 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.269+0000 7fc6e2d9d700 1 -- 172.21.15.195:0/813002779 learned_addr learned my addr 172.21.15.195:0/813002779 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:48:14.283 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.270+0000 7fc6e1d9b700 1 --2- 172.21.15.195:0/813002779 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc6e4150500 0x7fc6e4069870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:14.283 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.270+0000 7fc6e259c700 1 --2- 172.21.15.195:0/813002779 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc6e406a450 0x7fc6e4166dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:14.284 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.270+0000 7fc6e2d9d700 1 -- 172.21.15.195:0/813002779 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc6e406a450 msgr2=0x7fc6e4166dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:14.284 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.270+0000 7fc6e2d9d700 1 --2- 172.21.15.195:0/813002779 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc6e406a450 0x7fc6e4166dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:14.284 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.270+0000 7fc6e2d9d700 1 -- 172.21.15.195:0/813002779 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc6e4150500 msgr2=0x7fc6e4069870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:14.285 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.270+0000 7fc6e2d9d700 1 --2- 172.21.15.195:0/813002779 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc6e4150500 0x7fc6e4069870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:14.285 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.270+0000 7fc6e2d9d700 1 -- 172.21.15.195:0/813002779 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6dc009ae0 con 0x7fc6e41508d0 2022-06-27T11:48:14.285 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.270+0000 7fc6e1d9b700 1 --2- 172.21.15.195:0/813002779 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc6e4150500 0x7fc6e4069870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T11:48:14.285 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.270+0000 7fc6e2d9d700 1 --2- 172.21.15.195:0/813002779 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc6e41508d0 0x7fc6e414cf80 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc00b6c0 tx=0x7fc6dc018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:14.286 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.270+0000 7fc6d27fc700 1 -- 172.21.15.195:0/813002779 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6dc0092e0 con 0x7fc6e41508d0 2022-06-27T11:48:14.286 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.270+0000 7fc6d27fc700 1 -- 172.21.15.195:0/813002779 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6dc007820 con 0x7fc6e41508d0 2022-06-27T11:48:14.286 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.270+0000 7fc6e8a0e700 1 -- 172.21.15.195:0/813002779 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6e4169f80 con 0x7fc6e41508d0 2022-06-27T11:48:14.286 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.270+0000 7fc6d27fc700 1 -- 172.21.15.195:0/813002779 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6dc007be0 con 0x7fc6e41508d0 2022-06-27T11:48:14.287 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.270+0000 7fc6e8a0e700 1 -- 172.21.15.195:0/813002779 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6e416a520 con 0x7fc6e41508d0 2022-06-27T11:48:14.287 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.271+0000 7fc6e8a0e700 1 -- 172.21.15.195:0/813002779 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6e4166ad0 con 0x7fc6e41508d0 2022-06-27T11:48:14.287 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.272+0000 7fc6d27fc700 1 -- 172.21.15.195:0/813002779 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fc6dc004040 con 0x7fc6e41508d0 2022-06-27T11:48:14.287 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.272+0000 7fc6d27fc700 1 --2- 172.21.15.195:0/813002779 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc6c0063d30 0x7fc6c00661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:14.288 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.272+0000 7fc6d27fc700 1 -- 172.21.15.195:0/813002779 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fc6dc083010 con 0x7fc6e41508d0 2022-06-27T11:48:14.288 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.272+0000 7fc6e259c700 1 --2- 172.21.15.195:0/813002779 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc6c0063d30 0x7fc6c00661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:14.288 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.272+0000 7fc6e259c700 1 --2- 172.21.15.195:0/813002779 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc6c0063d30 0x7fc6c00661e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fc6d8001670 tx=0x7fc6d8009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:14.289 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.277+0000 7fc6d27fc700 1 -- 172.21.15.195:0/813002779 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fc6dc051c80 con 0x7fc6e41508d0 2022-06-27T11:48:14.503 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.503+0000 7fc6e8a0e700 1 -- 172.21.15.195:0/813002779 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc6e40722c0 con 0x7fc6e41508d0 2022-06-27T11:48:14.504 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.504+0000 7fc6d27fc700 1 -- 172.21.15.195:0/813002779 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6119 (secure 0 0 0) 0x7fc6dc0552a0 con 0x7fc6e41508d0 2022-06-27T11:48:14.504 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:48:14.505 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4287,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/864126677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":864126677},{"type":"v1","addr":"172.21.15.202:6833","nonce":864126677}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:47:57.373108+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.202:6835/2038169702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2038169702},{"type":"v1","addr":"172.21.15.202:6835","nonce":2038169702}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:48:14.506 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.506+0000 7fc6e8a0e700 1 -- 172.21.15.195:0/813002779 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc6c0063d30 msgr2=0x7fc6c00661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:14.506 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.506+0000 7fc6e8a0e700 1 --2- 172.21.15.195:0/813002779 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc6c0063d30 0x7fc6c00661e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fc6d8001670 tx=0x7fc6d8009040 comp rx=0 tx=0).stop 2022-06-27T11:48:14.507 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.506+0000 7fc6e8a0e700 1 -- 172.21.15.195:0/813002779 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc6e41508d0 msgr2=0x7fc6e414cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:14.507 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.506+0000 7fc6e8a0e700 1 --2- 172.21.15.195:0/813002779 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc6e41508d0 0x7fc6e414cf80 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc00b6c0 tx=0x7fc6dc018000 comp rx=0 tx=0).stop 2022-06-27T11:48:14.507 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.506+0000 7fc6e8a0e700 1 -- 172.21.15.195:0/813002779 shutdown_connections 2022-06-27T11:48:14.507 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.506+0000 7fc6e8a0e700 1 --2- 172.21.15.195:0/813002779 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc6c0063d30 0x7fc6c00661e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:14.508 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.506+0000 7fc6e8a0e700 1 --2- 172.21.15.195:0/813002779 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc6e41508d0 0x7fc6e414cf80 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:14.508 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.506+0000 7fc6e8a0e700 1 --2- 172.21.15.195:0/813002779 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc6e406a450 0x7fc6e4166dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:14.508 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.506+0000 7fc6e8a0e700 1 --2- 172.21.15.195:0/813002779 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc6e4150500 0x7fc6e4069870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:14.508 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.506+0000 7fc6e8a0e700 1 -- 172.21.15.195:0/813002779 >> 172.21.15.195:0/813002779 conn(0x7fc6e40c9b00 msgr2=0x7fc6e40c57a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:14.509 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.506+0000 7fc6e8a0e700 1 -- 172.21.15.195:0/813002779 shutdown_connections 2022-06-27T11:48:14.509 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.506+0000 7fc6e8a0e700 1 -- 172.21.15.195:0/813002779 wait complete. 2022-06-27T11:48:14.509 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 8 2022-06-27T11:48:14.518 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-27T11:48:14.649 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.638+0000 7fa3ad0d0700 1 Processor -- start 2022-06-27T11:48:14.649 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.638+0000 7fa3ad0d0700 1 -- start start 2022-06-27T11:48:14.650 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.639+0000 7fa3ad0d0700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fa3a806a450 0x7fa3a8063af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:14.650 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.639+0000 7fa3ad0d0700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fa3a81508d0 0x7fa3a80bbf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:14.650 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.639+0000 7fa3ad0d0700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3a8064030 con 0x7fa3a8150500 2022-06-27T11:48:14.651 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.639+0000 7fa3ad0d0700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3a80c4160 con 0x7fa3a806a450 2022-06-27T11:48:14.651 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.639+0000 7fa3ad0d0700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3a80c4440 con 0x7fa3a81508d0 2022-06-27T11:48:14.651 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.639+0000 7fa3a659c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fa3a806a450 0x7fa3a8063af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:14.652 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.639+0000 7fa3a659c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fa3a806a450 0x7fa3a8063af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:42198/0 (socket says 172.21.15.195:42198) 2022-06-27T11:48:14.652 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.639+0000 7fa3a659c700 1 -- 172.21.15.195:0/1308439019 learned_addr learned my addr 172.21.15.195:0/1308439019 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:48:14.652 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.639+0000 7fa3a759e700 1 --2- 172.21.15.195:0/1308439019 >> v2:172.21.15.202:3300/0 conn(0x7fa3a81508d0 0x7fa3a80bbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:14.653 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.640+0000 7fa3a659c700 1 -- 172.21.15.195:0/1308439019 >> v2:172.21.15.202:3300/0 conn(0x7fa3a81508d0 msgr2=0x7fa3a80bbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:48:14.653 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.640+0000 7fa3a659c700 1 --2- 172.21.15.195:0/1308439019 >> v2:172.21.15.202:3300/0 conn(0x7fa3a81508d0 0x7fa3a80bbf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:14.654 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.640+0000 7fa3a659c700 1 -- 172.21.15.195:0/1308439019 >> v1:172.21.15.202:6789/0 conn(0x7fa3a8150500 legacy=0x7fa3a80c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:14.654 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.640+0000 7fa3a659c700 1 -- 172.21.15.195:0/1308439019 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3a815a800 con 0x7fa3a806a450 2022-06-27T11:48:14.654 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.640+0000 7fa3a659c700 1 --2- 172.21.15.195:0/1308439019 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa3a806a450 0x7fa3a8063af0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fa3a8064800 tx=0x7fa398009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=304a202663e1b376 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:14.655 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.640+0000 7fa3a5d9b700 1 -- 172.21.15.195:0/1308439019 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa398011630 con 0x7fa3a806a450 2022-06-27T11:48:14.655 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.640+0000 7fa3a5d9b700 1 -- 172.21.15.195:0/1308439019 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa398011790 con 0x7fa3a806a450 2022-06-27T11:48:14.655 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.640+0000 7fa3a5d9b700 1 -- 172.21.15.195:0/1308439019 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa398011b30 con 0x7fa3a806a450 2022-06-27T11:48:14.656 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.640+0000 7fa3ad0d0700 1 -- 172.21.15.195:0/1308439019 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa3a806a450 msgr2=0x7fa3a8063af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:14.656 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.640+0000 7fa3ad0d0700 1 --2- 172.21.15.195:0/1308439019 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa3a806a450 0x7fa3a8063af0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fa3a8064800 tx=0x7fa398009120 comp rx=0 tx=0).stop 2022-06-27T11:48:14.657 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.640+0000 7fa3ad0d0700 1 -- 172.21.15.195:0/1308439019 shutdown_connections 2022-06-27T11:48:14.657 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.640+0000 7fa3ad0d0700 1 --2- 172.21.15.195:0/1308439019 >> v2:172.21.15.202:3300/0 conn(0x7fa3a81508d0 0x7fa3a80bbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:14.658 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.640+0000 7fa3ad0d0700 1 --2- 172.21.15.195:0/1308439019 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa3a806a450 0x7fa3a8063af0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:14.658 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.640+0000 7fa3ad0d0700 1 -- 172.21.15.195:0/1308439019 >> 172.21.15.195:0/1308439019 conn(0x7fa3a80c9b00 msgr2=0x7fa3a80c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:14.659 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.640+0000 7fa3ad0d0700 1 -- 172.21.15.195:0/1308439019 shutdown_connections 2022-06-27T11:48:14.660 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.640+0000 7fa3ad0d0700 1 -- 172.21.15.195:0/1308439019 wait complete. 2022-06-27T11:48:14.661 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.642+0000 7fa3ad0d0700 1 Processor -- start 2022-06-27T11:48:14.661 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.642+0000 7fa3ad0d0700 1 -- start start 2022-06-27T11:48:14.661 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.642+0000 7fa3ad0d0700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa3a806a450 0x7fa3a80c2990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:14.662 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.642+0000 7fa3ad0d0700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa3a8150500 0x7fa3a81662d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:14.662 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.642+0000 7fa3ad0d0700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa3a81508d0 0x7fa3a8166810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:14.663 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.642+0000 7fa3ad0d0700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a80c2570 con 0x7fa3a806a450 2022-06-27T11:48:14.663 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.642+0000 7fa3ad0d0700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a8159240 con 0x7fa3a81508d0 2022-06-27T11:48:14.663 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.642+0000 7fa3ad0d0700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a80c2290 con 0x7fa3a8150500 2022-06-27T11:48:14.664 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.642+0000 7fa3a6d9d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa3a806a450 0x7fa3a80c2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:14.664 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.643+0000 7fa3a6d9d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa3a806a450 0x7fa3a80c2990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:42202/0 (socket says 172.21.15.195:42202) 2022-06-27T11:48:14.665 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.643+0000 7fa3a6d9d700 1 -- 172.21.15.195:0/407953969 learned_addr learned my addr 172.21.15.195:0/407953969 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:48:14.665 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.643+0000 7fa3a759e700 1 --2- 172.21.15.195:0/407953969 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa3a81508d0 0x7fa3a8166810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:14.666 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.643+0000 7fa3a659c700 1 --2- 172.21.15.195:0/407953969 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa3a8150500 0x7fa3a81662d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:14.666 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.643+0000 7fa3a6d9d700 1 -- 172.21.15.195:0/407953969 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa3a8150500 msgr2=0x7fa3a81662d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:14.667 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.643+0000 7fa3a6d9d700 1 --2- 172.21.15.195:0/407953969 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa3a8150500 0x7fa3a81662d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:14.667 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.643+0000 7fa3a6d9d700 1 -- 172.21.15.195:0/407953969 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa3a81508d0 msgr2=0x7fa3a8166810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:14.668 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.643+0000 7fa3a6d9d700 1 --2- 172.21.15.195:0/407953969 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa3a81508d0 0x7fa3a8166810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:14.668 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.643+0000 7fa3a6d9d700 1 -- 172.21.15.195:0/407953969 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3980093a0 con 0x7fa3a806a450 2022-06-27T11:48:14.669 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.643+0000 7fa3a6d9d700 1 --2- 172.21.15.195:0/407953969 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa3a806a450 0x7fa3a80c2990 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa39c00c4e0 tx=0x7fa39c007e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:14.669 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.643+0000 7fa38effd700 1 -- 172.21.15.195:0/407953969 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa39c0050d0 con 0x7fa3a806a450 2022-06-27T11:48:14.670 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.643+0000 7fa38effd700 1 -- 172.21.15.195:0/407953969 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa39c005230 con 0x7fa3a806a450 2022-06-27T11:48:14.670 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.643+0000 7fa3ad0d0700 1 -- 172.21.15.195:0/407953969 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3a8158910 con 0x7fa3a806a450 2022-06-27T11:48:14.671 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.643+0000 7fa38effd700 1 -- 172.21.15.195:0/407953969 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa39c0117e0 con 0x7fa3a806a450 2022-06-27T11:48:14.671 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.643+0000 7fa3ad0d0700 1 -- 172.21.15.195:0/407953969 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3a8158ed0 con 0x7fa3a806a450 2022-06-27T11:48:14.672 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.644+0000 7fa3ad0d0700 1 -- 172.21.15.195:0/407953969 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3a81595d0 con 0x7fa3a806a450 2022-06-27T11:48:14.672 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.644+0000 7fa38effd700 1 -- 172.21.15.195:0/407953969 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fa39c019030 con 0x7fa3a806a450 2022-06-27T11:48:14.672 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.645+0000 7fa38effd700 1 --2- 172.21.15.195:0/407953969 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa388063d00 0x7fa3880661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:48:14.673 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.645+0000 7fa38effd700 1 -- 172.21.15.195:0/407953969 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fa39c07f830 con 0x7fa3a806a450 2022-06-27T11:48:14.673 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.645+0000 7fa3a659c700 1 --2- 172.21.15.195:0/407953969 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa388063d00 0x7fa3880661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:48:14.674 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.645+0000 7fa3a659c700 1 --2- 172.21.15.195:0/407953969 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa388063d00 0x7fa3880661b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa398009160 tx=0x7fa398017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:48:14.674 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.650+0000 7fa38effd700 1 -- 172.21.15.195:0/407953969 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fa39c04e420 con 0x7fa3a806a450 2022-06-27T11:48:14.829 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.829+0000 7fa3ad0d0700 1 -- 172.21.15.195:0/407953969 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa3a8064030 con 0x7fa3a806a450 2022-06-27T11:48:14.830 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.830+0000 7fa38effd700 1 -- 172.21.15.195:0/407953969 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16914 (secure 0 0 0) 0x7fa3a8064030 con 0x7fa3a806a450 2022-06-27T11:48:14.831 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:48:14.832 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":21,"fsid":"81b3a1d3-6a2b-47e8-8d69-7027cfeae7af","created":"2022-06-27T11:47:35.094715+0000","modified":"2022-06-27T11:47:53.359270+0000","last_up_change":"2022-06-27T11:47:44.777459+0000","last_in_change":"2022-06-27T11:47:37.596818+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2022-06-27T11:47:46.368325+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2022-06-27T11:47:46.386132+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-06-27T11:47:50.901270+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-06-27T11:47:51.314719+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c64bc51f-d4f3-4bc2-806b-da0dc78d51b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6813","nonce":94217},{"type":"v1","addr":"172.21.15.195:6815","nonce":94217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6817","nonce":94217},{"type":"v1","addr":"172.21.15.195:6819","nonce":94217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":94217},{"type":"v1","addr":"172.21.15.195:6823","nonce":94217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6820","nonce":94217},{"type":"v1","addr":"172.21.15.195:6821","nonce":94217}]},"public_addr":"172.21.15.195:6815/94217","cluster_addr":"172.21.15.195:6819/94217","heartbeat_back_addr":"172.21.15.195:6823/94217","heartbeat_front_addr":"172.21.15.195:6821/94217","state":["exists","up"]},{"osd":1,"uuid":"bea71cdf-aa98-493f-9207-66d886a3e30e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":94223},{"type":"v1","addr":"172.21.15.195:6809","nonce":94223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":94223},{"type":"v1","addr":"172.21.15.195:6811","nonce":94223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6816","nonce":94223},{"type":"v1","addr":"172.21.15.195:6818","nonce":94223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":94223},{"type":"v1","addr":"172.21.15.195:6814","nonce":94223}]},"public_addr":"172.21.15.195:6809/94223","cluster_addr":"172.21.15.195:6811/94223","heartbeat_back_addr":"172.21.15.195:6818/94223","heartbeat_front_addr":"172.21.15.195:6814/94223","state":["exists","up"]},{"osd":2,"uuid":"bf6c87a0-a8ea-497a-9ce1-721285888065","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":94221},{"type":"v1","addr":"172.21.15.195:6801","nonce":94221}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":94221},{"type":"v1","addr":"172.21.15.195:6803","nonce":94221}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":94221},{"type":"v1","addr":"172.21.15.195:6807","nonce":94221}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":94221},{"type":"v1","addr":"172.21.15.195:6805","nonce":94221}]},"public_addr":"172.21.15.195:6801/94221","cluster_addr":"172.21.15.195:6803/94221","heartbeat_back_addr":"172.21.15.195:6807/94221","heartbeat_front_addr":"172.21.15.195:6805/94221","state":["exists","up"]},{"osd":3,"uuid":"e8aa47ce-af8c-47e7-9654-870cd53229f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":94232},{"type":"v1","addr":"172.21.15.195:6825","nonce":94232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":94232},{"type":"v1","addr":"172.21.15.195:6827","nonce":94232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":94232},{"type":"v1","addr":"172.21.15.195:6831","nonce":94232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":94232},{"type":"v1","addr":"172.21.15.195:6829","nonce":94232}]},"public_addr":"172.21.15.195:6825/94232","cluster_addr":"172.21.15.195:6827/94232","heartbeat_back_addr":"172.21.15.195:6831/94232","heartbeat_front_addr":"172.21.15.195:6829/94232","state":["exists","up"]},{"osd":4,"uuid":"99ffa553-f149-443a-98f5-68b10f77e085","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6800","nonce":94308},{"type":"v1","addr":"172.21.15.202:6801","nonce":94308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6802","nonce":94308},{"type":"v1","addr":"172.21.15.202:6803","nonce":94308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6806","nonce":94308},{"type":"v1","addr":"172.21.15.202:6807","nonce":94308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6804","nonce":94308},{"type":"v1","addr":"172.21.15.202:6805","nonce":94308}]},"public_addr":"172.21.15.202:6801/94308","cluster_addr":"172.21.15.202:6803/94308","heartbeat_back_addr":"172.21.15.202:6807/94308","heartbeat_front_addr":"172.21.15.202:6805/94308","state":["exists","up"]},{"osd":5,"uuid":"35409ef0-891b-4c01-82e4-26b8e64f502d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6824","nonce":94325},{"type":"v1","addr":"172.21.15.202:6825","nonce":94325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6826","nonce":94325},{"type":"v1","addr":"172.21.15.202:6827","nonce":94325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6830","nonce":94325},{"type":"v1","addr":"172.21.15.202:6831","nonce":94325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6828","nonce":94325},{"type":"v1","addr":"172.21.15.202:6829","nonce":94325}]},"public_addr":"172.21.15.202:6825/94325","cluster_addr":"172.21.15.202:6827/94325","heartbeat_back_addr":"172.21.15.202:6831/94325","heartbeat_front_addr":"172.21.15.202:6829/94325","state":["exists","up"]},{"osd":6,"uuid":"009124c4-8b5f-47d4-bd20-2c19271a2adc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6809","nonce":94309},{"type":"v1","addr":"172.21.15.202:6811","nonce":94309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6813","nonce":94309},{"type":"v1","addr":"172.21.15.202:6815","nonce":94309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6821","nonce":94309},{"type":"v1","addr":"172.21.15.202:6823","nonce":94309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6817","nonce":94309},{"type":"v1","addr":"172.21.15.202:6819","nonce":94309}]},"public_addr":"172.21.15.202:6811/94309","cluster_addr":"172.21.15.202:6815/94309","heartbeat_back_addr":"172.21.15.202:6823/94309","heartbeat_front_addr":"172.21.15.202:6819/94309","state":["exists","up"]},{"osd":7,"uuid":"7773115e-d3c3-4a0c-8801-d8ce63e0e5a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6808","nonce":94310},{"type":"v1","addr":"172.21.15.202:6810","nonce":94310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6812","nonce":94310},{"type":"v1","addr":"172.21.15.202:6814","nonce":94310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6820","nonce":94310},{"type":"v1","addr":"172.21.15.202:6822","nonce":94310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6816","nonce":94310},{"type":"v1","addr":"172.21.15.202:6818","nonce":94310}]},"public_addr":"172.21.15.202:6810/94310","cluster_addr":"172.21.15.202:6814/94310","heartbeat_back_addr":"172.21.15.202:6822/94310","heartbeat_front_addr":"172.21.15.202:6818/94310","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.776182+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.806922+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.237997+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.784371+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.723615+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.820332+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:43.841414+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-27T11:47:42.758071+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-06-27T11:48:14.835 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.832+0000 7fa3ad0d0700 1 -- 172.21.15.195:0/407953969 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa388063d00 msgr2=0x7fa3880661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:14.835 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.832+0000 7fa3ad0d0700 1 --2- 172.21.15.195:0/407953969 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa388063d00 0x7fa3880661b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa398009160 tx=0x7fa398017000 comp rx=0 tx=0).stop 2022-06-27T11:48:14.836 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.832+0000 7fa3ad0d0700 1 -- 172.21.15.195:0/407953969 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa3a806a450 msgr2=0x7fa3a80c2990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:48:14.836 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.832+0000 7fa3ad0d0700 1 --2- 172.21.15.195:0/407953969 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa3a806a450 0x7fa3a80c2990 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa39c00c4e0 tx=0x7fa39c007e30 comp rx=0 tx=0).stop 2022-06-27T11:48:14.837 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.832+0000 7fa3ad0d0700 1 -- 172.21.15.195:0/407953969 shutdown_connections 2022-06-27T11:48:14.837 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.832+0000 7fa3ad0d0700 1 --2- 172.21.15.195:0/407953969 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa388063d00 0x7fa3880661b0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:14.837 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.832+0000 7fa3ad0d0700 1 --2- 172.21.15.195:0/407953969 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa3a806a450 0x7fa3a80c2990 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:14.838 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.832+0000 7fa3ad0d0700 1 --2- 172.21.15.195:0/407953969 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa3a81508d0 0x7fa3a8166810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:14.838 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.832+0000 7fa3ad0d0700 1 --2- 172.21.15.195:0/407953969 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa3a8150500 0x7fa3a81662d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:48:14.838 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.832+0000 7fa3ad0d0700 1 -- 172.21.15.195:0/407953969 >> 172.21.15.195:0/407953969 conn(0x7fa3a80c9b00 msgr2=0x7fa3a814fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:48:14.839 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.832+0000 7fa3ad0d0700 1 -- 172.21.15.195:0/407953969 shutdown_connections 2022-06-27T11:48:14.839 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:48:14.832+0000 7fa3ad0d0700 1 -- 172.21.15.195:0/407953969 wait complete. 2022-06-27T11:48:14.847 DEBUG:tasks.mds_thrash:Yielding 2022-06-27T11:48:14.848 INFO:teuthology.run_tasks:Running task workunit... 2022-06-27T11:48:14.877 INFO:tasks.workunit:Pulling workunits from ref 5292901bc999daa0d7ac128764a3f27c4233c252 2022-06-27T11:48:14.878 INFO:tasks.workunit:Making a separate scratch dir for every client... 2022-06-27T11:48:14.879 INFO:tasks.workunit:timeout=6h 2022-06-27T11:48:14.879 INFO:tasks.workunit:cleanup=True 2022-06-27T11:48:14.880 DEBUG:teuthology.orchestra.run.smithi195:> stat -- /home/ubuntu/cephtest/mnt.0 2022-06-27T11:48:14.881 INFO:tasks.mds_thrash.fs.[cephfs]:starting mds_do_thrash for fs cephfs 2022-06-27T11:48:14.882 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 88 secs before thrashing 2022-06-27T11:48:14.900 INFO:teuthology.orchestra.run.smithi195.stdout: File: /home/ubuntu/cephtest/mnt.0 2022-06-27T11:48:14.901 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2022-06-27T11:48:14.901 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 2fh/47d Inode: 1 Links: 2 2022-06-27T11:48:14.901 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2022-06-27T11:48:14.902 INFO:teuthology.orchestra.run.smithi195.stdout:Context: system_u:object_r:cephfs_t:s0 2022-06-27T11:48:14.902 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2022-06-27T11:48:14.902 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2022-06-27 11:47:57.360526179 +0000 2022-06-27T11:48:14.902 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2022-06-27 11:48:07.619178857 +0000 2022-06-27T11:48:14.903 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: 2022-06-27 11:47:57.360526179 +0000 2022-06-27T11:48:14.907 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2022-06-27T11:48:14.907 DEBUG:teuthology.orchestra.run.smithi195:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2022-06-27T11:48:14.974 DEBUG:teuthology.orchestra.run.smithi195:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 5292901bc999daa0d7ac128764a3f27c4233c252 2022-06-27T11:48:15.036 INFO:tasks.workunit.client.0.smithi195.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2022-06-27T11:48:26.253 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T11:48:26.258 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T11:48:49.643 INFO:tasks.workunit.client.0.smithi195.stderr:Note: switching to '5292901bc999daa0d7ac128764a3f27c4233c252'. 2022-06-27T11:48:49.644 INFO:tasks.workunit.client.0.smithi195.stderr: 2022-06-27T11:48:49.644 INFO:tasks.workunit.client.0.smithi195.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2022-06-27T11:48:49.645 INFO:tasks.workunit.client.0.smithi195.stderr:changes and commit them, and you can discard any commits you make in this 2022-06-27T11:48:49.645 INFO:tasks.workunit.client.0.smithi195.stderr:state without impacting any branches by switching back to a branch. 2022-06-27T11:48:49.645 INFO:tasks.workunit.client.0.smithi195.stderr: 2022-06-27T11:48:49.646 INFO:tasks.workunit.client.0.smithi195.stderr:If you want to create a new branch to retain commits you create, you may 2022-06-27T11:48:49.646 INFO:tasks.workunit.client.0.smithi195.stderr:do so (now or later) by using -c with the switch command. Example: 2022-06-27T11:48:49.647 INFO:tasks.workunit.client.0.smithi195.stderr: 2022-06-27T11:48:49.647 INFO:tasks.workunit.client.0.smithi195.stderr: git switch -c 2022-06-27T11:48:49.647 INFO:tasks.workunit.client.0.smithi195.stderr: 2022-06-27T11:48:49.648 INFO:tasks.workunit.client.0.smithi195.stderr:Or undo this operation with: 2022-06-27T11:48:49.648 INFO:tasks.workunit.client.0.smithi195.stderr: 2022-06-27T11:48:49.649 INFO:tasks.workunit.client.0.smithi195.stderr: git switch - 2022-06-27T11:48:49.649 INFO:tasks.workunit.client.0.smithi195.stderr: 2022-06-27T11:48:49.650 INFO:tasks.workunit.client.0.smithi195.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2022-06-27T11:48:49.650 INFO:tasks.workunit.client.0.smithi195.stderr: 2022-06-27T11:48:49.650 INFO:tasks.workunit.client.0.smithi195.stderr:HEAD is now at 5292901bc99 mds: replacing bootstrap session only if handle client session message 2022-06-27T11:48:49.653 DEBUG:teuthology.orchestra.run.smithi195:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2022-06-27T11:48:49.679 INFO:tasks.workunit.client.0.smithi195.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2022-06-27T11:48:49.680 INFO:tasks.workunit.client.0.smithi195.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2022-06-27T11:48:49.681 INFO:tasks.workunit.client.0.smithi195.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2022-06-27T11:48:50.166 INFO:tasks.workunit.client.0.smithi195.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2022-06-27T11:48:50.227 INFO:tasks.workunit.client.0.smithi195.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2022-06-27T11:48:50.271 INFO:tasks.workunit.client.0.smithi195.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2022-06-27T11:48:50.271 INFO:tasks.workunit.client.0.smithi195.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2022-06-27T11:48:50.272 INFO:tasks.workunit.client.0.smithi195.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2022-06-27T11:48:50.302 INFO:tasks.workunit.client.0.smithi195.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2022-06-27T11:48:50.305 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2022-06-27T11:48:50.306 DEBUG:teuthology.orchestra.run.smithi195:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2022-06-27T11:48:50.362 INFO:tasks.workunit:Running workunits matching fs/snaps on client.0... 2022-06-27T11:48:50.364 INFO:tasks.workunit:Running workunit fs/snaps/snap-rm-diff.sh... 2022-06-27T11:48:50.365 DEBUG:teuthology.orchestra.run.smithi195:workunit test fs/snaps/snap-rm-diff.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5292901bc999daa0d7ac128764a3f27c4233c252 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snap-rm-diff.sh 2022-06-27T11:48:50.440 INFO:tasks.workunit.client.0.smithi195.stderr:+ wget -q http://download.ceph.com/qa/linux-2.6.33.tar.bz2 2022-06-27T11:48:53.459 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir foo 2022-06-27T11:48:53.463 INFO:tasks.workunit.client.0.smithi195.stderr:+ cp linux-2.6.33.tar.bz2 foo 2022-06-27T11:48:53.512 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir foo/.snap/barsnap 2022-06-27T11:48:53.554 INFO:tasks.workunit.client.0.smithi195.stderr:+ rm foo/linux-2.6.33.tar.bz2 2022-06-27T11:48:53.567 INFO:tasks.workunit.client.0.smithi195.stderr:+ diff -q foo/.snap/barsnap/linux-2.6.33.tar.bz2 linux-2.6.33.tar.bz2 2022-06-27T11:48:53.716 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo 'passed: files are identical' 2022-06-27T11:48:53.716 INFO:tasks.workunit.client.0.smithi195.stdout:passed: files are identical 2022-06-27T11:48:53.717 INFO:tasks.workunit.client.0.smithi195.stderr:+ rmdir foo/.snap/barsnap 2022-06-27T11:48:53.737 INFO:tasks.workunit.client.0.smithi195.stdout:OK 2022-06-27T11:48:53.739 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo OK 2022-06-27T11:48:53.741 INFO:teuthology.orchestra.run:Running command with timeout 3600 2022-06-27T11:48:53.742 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2022-06-27T11:48:53.824 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-1.sh... 2022-06-27T11:48:53.825 DEBUG:teuthology.orchestra.run.smithi195:workunit test fs/snaps/snaptest-1.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5292901bc999daa0d7ac128764a3f27c4233c252 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-1.sh 2022-06-27T11:48:53.898 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo 1 2022-06-27T11:48:53.899 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo 2 2022-06-27T11:48:53.899 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo 3 2022-06-27T11:48:53.900 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' -e file4 ']' 2022-06-27T11:48:53.900 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/snap1 2022-06-27T11:48:53.919 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo 4 2022-06-27T11:48:53.942 INFO:tasks.workunit.client.0.smithi195.stderr:++ ls 2022-06-27T11:48:53.943 INFO:tasks.workunit.client.0.smithi195.stderr:+ now='file1 2022-06-27T11:48:53.943 INFO:tasks.workunit.client.0.smithi195.stderr:file2 2022-06-27T11:48:53.943 INFO:tasks.workunit.client.0.smithi195.stderr:file3 2022-06-27T11:48:53.944 INFO:tasks.workunit.client.0.smithi195.stderr:file4' 2022-06-27T11:48:53.944 INFO:tasks.workunit.client.0.smithi195.stderr:++ ls .snap/snap1 2022-06-27T11:48:53.945 INFO:tasks.workunit.client.0.smithi195.stderr:+ then='file1 2022-06-27T11:48:53.945 INFO:tasks.workunit.client.0.smithi195.stderr:file2 2022-06-27T11:48:53.945 INFO:tasks.workunit.client.0.smithi195.stderr:file3' 2022-06-27T11:48:53.946 INFO:tasks.workunit.client.0.smithi195.stderr:+ rmdir .snap/snap1 2022-06-27T11:48:54.226 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 'file1 2022-06-27T11:48:54.227 INFO:tasks.workunit.client.0.smithi195.stderr:file2 2022-06-27T11:48:54.227 INFO:tasks.workunit.client.0.smithi195.stderr:file3 2022-06-27T11:48:54.228 INFO:tasks.workunit.client.0.smithi195.stderr:file4' = 'file1 2022-06-27T11:48:54.228 INFO:tasks.workunit.client.0.smithi195.stderr:file2 2022-06-27T11:48:54.228 INFO:tasks.workunit.client.0.smithi195.stderr:file3' ']' 2022-06-27T11:48:54.229 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo 1 2022-06-27T11:48:54.231 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo 2 2022-06-27T11:48:54.237 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo 3 2022-06-27T11:48:54.243 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/snap1 2022-06-27T11:48:54.266 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo 4 2022-06-27T11:48:54.290 INFO:tasks.workunit.client.0.smithi195.stderr:+ rmdir .snap/snap1 2022-06-27T11:48:54.294 INFO:tasks.workunit.client.0.smithi195.stderr:+ rm file1 file2 file3 file4 2022-06-27T11:48:54.300 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo OK 2022-06-27T11:48:54.302 INFO:tasks.workunit.client.0.smithi195.stdout:OK 2022-06-27T11:48:54.302 INFO:teuthology.orchestra.run:Running command with timeout 3600 2022-06-27T11:48:54.304 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2022-06-27T11:48:54.379 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-2.sh... 2022-06-27T11:48:54.381 DEBUG:teuthology.orchestra.run.smithi195:workunit test fs/snaps/snaptest-2.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5292901bc999daa0d7ac128764a3f27c4233c252 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-2.sh 2022-06-27T11:48:54.446 INFO:tasks.workunit.client.0.smithi195.stdout:Create dir 100 to 199 ... 2022-06-27T11:48:54.448 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 100 2022-06-27T11:48:55.088 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 101 2022-06-27T11:48:55.106 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 102 2022-06-27T11:48:55.113 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 103 2022-06-27T11:48:55.127 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 104 2022-06-27T11:48:55.150 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 105 2022-06-27T11:48:55.155 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 106 2022-06-27T11:48:55.165 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 107 2022-06-27T11:48:55.178 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 108 2022-06-27T11:48:55.191 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 109 2022-06-27T11:48:55.208 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 110 2022-06-27T11:48:55.218 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 111 2022-06-27T11:48:55.239 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 112 2022-06-27T11:48:55.249 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 113 2022-06-27T11:48:55.260 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 114 2022-06-27T11:48:55.595 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 115 2022-06-27T11:48:55.612 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 116 2022-06-27T11:48:55.621 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 117 2022-06-27T11:48:55.632 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 118 2022-06-27T11:48:55.644 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 119 2022-06-27T11:48:55.656 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 120 2022-06-27T11:48:55.669 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 121 2022-06-27T11:48:55.681 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 122 2022-06-27T11:48:55.692 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 123 2022-06-27T11:48:55.703 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 124 2022-06-27T11:48:55.714 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 125 2022-06-27T11:48:55.724 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 126 2022-06-27T11:48:55.735 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 127 2022-06-27T11:48:55.747 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 128 2022-06-27T11:48:55.757 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 129 2022-06-27T11:48:56.050 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 130 2022-06-27T11:48:56.114 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 131 2022-06-27T11:48:56.130 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 132 2022-06-27T11:48:56.134 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 133 2022-06-27T11:48:56.151 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 134 2022-06-27T11:48:56.155 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 135 2022-06-27T11:48:56.189 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 136 2022-06-27T11:48:56.190 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 137 2022-06-27T11:48:56.190 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 138 2022-06-27T11:48:56.209 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 139 2022-06-27T11:48:56.227 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 140 2022-06-27T11:48:56.227 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 141 2022-06-27T11:48:56.228 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 142 2022-06-27T11:48:56.250 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 143 2022-06-27T11:48:56.251 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 144 2022-06-27T11:48:56.324 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T11:48:56.328 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T11:48:57.194 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 145 2022-06-27T11:48:57.205 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 146 2022-06-27T11:48:57.217 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 147 2022-06-27T11:48:57.229 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 148 2022-06-27T11:48:57.243 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 149 2022-06-27T11:48:58.100 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 150 2022-06-27T11:48:58.112 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 151 2022-06-27T11:48:58.125 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 152 2022-06-27T11:48:58.136 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 153 2022-06-27T11:48:58.148 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 154 2022-06-27T11:48:58.159 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 155 2022-06-27T11:48:58.171 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 156 2022-06-27T11:48:58.182 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 157 2022-06-27T11:48:58.197 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 158 2022-06-27T11:48:58.209 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 159 2022-06-27T11:48:58.221 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 160 2022-06-27T11:48:58.233 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 161 2022-06-27T11:48:58.245 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 162 2022-06-27T11:48:58.257 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 163 2022-06-27T11:48:58.268 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 164 2022-06-27T11:48:58.278 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 165 2022-06-27T11:48:58.288 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 166 2022-06-27T11:48:58.303 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 167 2022-06-27T11:48:58.310 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 168 2022-06-27T11:48:58.321 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 169 2022-06-27T11:48:58.336 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 170 2022-06-27T11:48:58.343 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 171 2022-06-27T11:48:58.356 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 172 2022-06-27T11:48:58.363 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 173 2022-06-27T11:48:58.373 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 174 2022-06-27T11:48:58.382 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 175 2022-06-27T11:48:58.392 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 176 2022-06-27T11:48:58.402 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 177 2022-06-27T11:48:58.411 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 178 2022-06-27T11:48:58.421 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 179 2022-06-27T11:48:58.770 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 180 2022-06-27T11:48:58.780 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 181 2022-06-27T11:48:58.791 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 182 2022-06-27T11:48:58.802 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 183 2022-06-27T11:48:58.815 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 184 2022-06-27T11:48:58.823 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 185 2022-06-27T11:48:58.834 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 186 2022-06-27T11:48:58.845 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 187 2022-06-27T11:48:58.857 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 188 2022-06-27T11:48:58.868 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 189 2022-06-27T11:48:58.885 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 190 2022-06-27T11:48:59.813 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 191 2022-06-27T11:48:59.828 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 192 2022-06-27T11:48:59.846 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 193 2022-06-27T11:48:59.856 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 194 2022-06-27T11:48:59.870 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 195 2022-06-27T11:49:00.216 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 196 2022-06-27T11:49:00.229 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 197 2022-06-27T11:49:00.241 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 198 2022-06-27T11:49:00.254 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 199 2022-06-27T11:49:00.266 INFO:tasks.workunit.client.0.smithi195.stdout:Take first snapshot .snap/test1 2022-06-27T11:49:02.357 INFO:tasks.workunit.client.0.smithi195.stdout:Create dir 200 to 299 ... 2022-06-27T11:49:02.360 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 200 2022-06-27T11:49:02.390 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 201 2022-06-27T11:49:02.407 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 202 2022-06-27T11:49:02.417 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 203 2022-06-27T11:49:02.606 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 204 2022-06-27T11:49:02.619 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 205 2022-06-27T11:49:02.630 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 206 2022-06-27T11:49:02.644 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 207 2022-06-27T11:49:02.657 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 208 2022-06-27T11:49:02.668 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 209 2022-06-27T11:49:02.682 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 210 2022-06-27T11:49:02.711 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 211 2022-06-27T11:49:02.722 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 212 2022-06-27T11:49:03.010 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 213 2022-06-27T11:49:03.022 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 214 2022-06-27T11:49:03.038 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 215 2022-06-27T11:49:03.047 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 216 2022-06-27T11:49:03.059 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 217 2022-06-27T11:49:03.073 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 218 2022-06-27T11:49:03.084 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 219 2022-06-27T11:49:03.097 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 220 2022-06-27T11:49:03.110 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 221 2022-06-27T11:49:03.123 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 222 2022-06-27T11:49:03.135 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 223 2022-06-27T11:49:03.148 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 224 2022-06-27T11:49:03.160 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 225 2022-06-27T11:49:03.172 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 226 2022-06-27T11:49:03.183 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 227 2022-06-27T11:49:03.194 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 228 2022-06-27T11:49:03.205 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 229 2022-06-27T11:49:03.215 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 230 2022-06-27T11:49:03.226 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 231 2022-06-27T11:49:03.236 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 232 2022-06-27T11:49:03.246 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 233 2022-06-27T11:49:03.256 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 234 2022-06-27T11:49:03.268 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 235 2022-06-27T11:49:03.278 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 236 2022-06-27T11:49:03.289 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 237 2022-06-27T11:49:03.300 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 238 2022-06-27T11:49:03.312 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 239 2022-06-27T11:49:03.324 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 240 2022-06-27T11:49:03.335 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 241 2022-06-27T11:49:03.346 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 242 2022-06-27T11:49:03.636 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 243 2022-06-27T11:49:03.648 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 244 2022-06-27T11:49:03.659 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 245 2022-06-27T11:49:03.670 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 246 2022-06-27T11:49:03.681 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 247 2022-06-27T11:49:03.692 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 248 2022-06-27T11:49:03.703 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 249 2022-06-27T11:49:03.715 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 250 2022-06-27T11:49:03.730 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 251 2022-06-27T11:49:07.415 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 252 2022-06-27T11:49:07.427 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 253 2022-06-27T11:49:07.440 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 254 2022-06-27T11:49:07.677 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 255 2022-06-27T11:49:07.690 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 256 2022-06-27T11:49:07.702 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 257 2022-06-27T11:49:07.856 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 258 2022-06-27T11:49:07.872 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 259 2022-06-27T11:49:07.886 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 260 2022-06-27T11:49:07.900 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 261 2022-06-27T11:49:07.909 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 262 2022-06-27T11:49:07.921 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 263 2022-06-27T11:49:07.932 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 264 2022-06-27T11:49:07.951 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 265 2022-06-27T11:49:07.955 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 266 2022-06-27T11:49:07.980 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 267 2022-06-27T11:49:07.984 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 268 2022-06-27T11:49:07.998 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 269 2022-06-27T11:49:08.010 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 270 2022-06-27T11:49:08.022 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 271 2022-06-27T11:49:08.034 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 272 2022-06-27T11:49:08.046 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 273 2022-06-27T11:49:08.057 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 274 2022-06-27T11:49:08.074 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 275 2022-06-27T11:49:08.088 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 276 2022-06-27T11:49:08.093 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 277 2022-06-27T11:49:08.107 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 278 2022-06-27T11:49:08.120 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 279 2022-06-27T11:49:08.133 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 280 2022-06-27T11:49:08.143 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 281 2022-06-27T11:49:08.158 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 282 2022-06-27T11:49:08.164 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 283 2022-06-27T11:49:08.174 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 284 2022-06-27T11:49:08.465 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 285 2022-06-27T11:49:08.479 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 286 2022-06-27T11:49:08.767 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 287 2022-06-27T11:49:08.779 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 288 2022-06-27T11:49:08.790 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 289 2022-06-27T11:49:08.802 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 290 2022-06-27T11:49:08.815 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 291 2022-06-27T11:49:08.827 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 292 2022-06-27T11:49:08.838 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 293 2022-06-27T11:49:08.850 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 294 2022-06-27T11:49:08.867 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 295 2022-06-27T11:49:08.872 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 296 2022-06-27T11:49:08.883 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 297 2022-06-27T11:49:08.900 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 298 2022-06-27T11:49:08.911 INFO:tasks.workunit.client.0.smithi195.stdout: create dir 299 2022-06-27T11:49:08.920 INFO:tasks.workunit.client.0.smithi195.stdout:Create a snapshot in every first level dir ... 2022-06-27T11:49:09.050 INFO:tasks.workunit.client.0.smithi195.stdout: create 100/.snap/snap-subdir-test 2022-06-27T11:49:09.081 INFO:tasks.workunit.client.0.smithi195.stdout: create 100/30 file after the snapshot 2022-06-27T11:49:09.096 INFO:tasks.workunit.client.0.smithi195.stdout: create 100/31 file after the snapshot 2022-06-27T11:49:09.097 INFO:tasks.workunit.client.0.smithi195.stdout: create 100/32 file after the snapshot 2022-06-27T11:49:09.098 INFO:tasks.workunit.client.0.smithi195.stdout: create 100/33 file after the snapshot 2022-06-27T11:49:09.103 INFO:tasks.workunit.client.0.smithi195.stdout: create 100/34 file after the snapshot 2022-06-27T11:49:09.103 INFO:tasks.workunit.client.0.smithi195.stdout: create 100/35 file after the snapshot 2022-06-27T11:49:09.104 INFO:tasks.workunit.client.0.smithi195.stdout: create 100/36 file after the snapshot 2022-06-27T11:49:09.105 INFO:tasks.workunit.client.0.smithi195.stdout: create 100/37 file after the snapshot 2022-06-27T11:49:09.106 INFO:tasks.workunit.client.0.smithi195.stdout: create 100/38 file after the snapshot 2022-06-27T11:49:09.111 INFO:tasks.workunit.client.0.smithi195.stdout: create 100/39 file after the snapshot 2022-06-27T11:49:09.111 INFO:tasks.workunit.client.0.smithi195.stdout: create 101/.snap/snap-subdir-test 2022-06-27T11:49:09.150 INFO:tasks.workunit.client.0.smithi195.stdout: create 101/30 file after the snapshot 2022-06-27T11:49:09.161 INFO:tasks.workunit.client.0.smithi195.stdout: create 101/31 file after the snapshot 2022-06-27T11:49:09.163 INFO:tasks.workunit.client.0.smithi195.stdout: create 101/32 file after the snapshot 2022-06-27T11:49:09.164 INFO:tasks.workunit.client.0.smithi195.stdout: create 101/33 file after the snapshot 2022-06-27T11:49:09.165 INFO:tasks.workunit.client.0.smithi195.stdout: create 101/34 file after the snapshot 2022-06-27T11:49:09.166 INFO:tasks.workunit.client.0.smithi195.stdout: create 101/35 file after the snapshot 2022-06-27T11:49:09.167 INFO:tasks.workunit.client.0.smithi195.stdout: create 101/36 file after the snapshot 2022-06-27T11:49:09.168 INFO:tasks.workunit.client.0.smithi195.stdout: create 101/37 file after the snapshot 2022-06-27T11:49:09.170 INFO:tasks.workunit.client.0.smithi195.stdout: create 101/38 file after the snapshot 2022-06-27T11:49:09.176 INFO:tasks.workunit.client.0.smithi195.stdout: create 101/39 file after the snapshot 2022-06-27T11:49:09.176 INFO:tasks.workunit.client.0.smithi195.stdout: create 102/.snap/snap-subdir-test 2022-06-27T11:49:09.218 INFO:tasks.workunit.client.0.smithi195.stdout: create 102/30 file after the snapshot 2022-06-27T11:49:09.225 INFO:tasks.workunit.client.0.smithi195.stdout: create 102/31 file after the snapshot 2022-06-27T11:49:09.227 INFO:tasks.workunit.client.0.smithi195.stdout: create 102/32 file after the snapshot 2022-06-27T11:49:09.234 INFO:tasks.workunit.client.0.smithi195.stdout: create 102/33 file after the snapshot 2022-06-27T11:49:09.235 INFO:tasks.workunit.client.0.smithi195.stdout: create 102/34 file after the snapshot 2022-06-27T11:49:09.235 INFO:tasks.workunit.client.0.smithi195.stdout: create 102/35 file after the snapshot 2022-06-27T11:49:09.235 INFO:tasks.workunit.client.0.smithi195.stdout: create 102/36 file after the snapshot 2022-06-27T11:49:09.611 INFO:tasks.workunit.client.0.smithi195.stdout: create 102/37 file after the snapshot 2022-06-27T11:49:09.612 INFO:tasks.workunit.client.0.smithi195.stdout: create 102/38 file after the snapshot 2022-06-27T11:49:09.619 INFO:tasks.workunit.client.0.smithi195.stdout: create 102/39 file after the snapshot 2022-06-27T11:49:09.619 INFO:tasks.workunit.client.0.smithi195.stdout: create 103/.snap/snap-subdir-test 2022-06-27T11:49:09.660 INFO:tasks.workunit.client.0.smithi195.stdout: create 103/30 file after the snapshot 2022-06-27T11:49:09.670 INFO:tasks.workunit.client.0.smithi195.stdout: create 103/31 file after the snapshot 2022-06-27T11:49:09.670 INFO:tasks.workunit.client.0.smithi195.stdout: create 103/32 file after the snapshot 2022-06-27T11:49:09.671 INFO:tasks.workunit.client.0.smithi195.stdout: create 103/33 file after the snapshot 2022-06-27T11:49:09.672 INFO:tasks.workunit.client.0.smithi195.stdout: create 103/34 file after the snapshot 2022-06-27T11:49:09.673 INFO:tasks.workunit.client.0.smithi195.stdout: create 103/35 file after the snapshot 2022-06-27T11:49:09.674 INFO:tasks.workunit.client.0.smithi195.stdout: create 103/36 file after the snapshot 2022-06-27T11:49:09.676 INFO:tasks.workunit.client.0.smithi195.stdout: create 103/37 file after the snapshot 2022-06-27T11:49:09.677 INFO:tasks.workunit.client.0.smithi195.stdout: create 103/38 file after the snapshot 2022-06-27T11:49:09.678 INFO:tasks.workunit.client.0.smithi195.stdout: create 103/39 file after the snapshot 2022-06-27T11:49:09.679 INFO:tasks.workunit.client.0.smithi195.stdout: create 104/.snap/snap-subdir-test 2022-06-27T11:49:09.719 INFO:tasks.workunit.client.0.smithi195.stdout: create 104/30 file after the snapshot 2022-06-27T11:49:09.741 INFO:tasks.workunit.client.0.smithi195.stdout: create 104/31 file after the snapshot 2022-06-27T11:49:09.742 INFO:tasks.workunit.client.0.smithi195.stdout: create 104/32 file after the snapshot 2022-06-27T11:49:09.742 INFO:tasks.workunit.client.0.smithi195.stdout: create 104/33 file after the snapshot 2022-06-27T11:49:09.743 INFO:tasks.workunit.client.0.smithi195.stdout: create 104/34 file after the snapshot 2022-06-27T11:49:09.743 INFO:tasks.workunit.client.0.smithi195.stdout: create 104/35 file after the snapshot 2022-06-27T11:49:09.744 INFO:tasks.workunit.client.0.smithi195.stdout: create 104/36 file after the snapshot 2022-06-27T11:49:09.744 INFO:tasks.workunit.client.0.smithi195.stdout: create 104/37 file after the snapshot 2022-06-27T11:49:09.745 INFO:tasks.workunit.client.0.smithi195.stdout: create 104/38 file after the snapshot 2022-06-27T11:49:09.746 INFO:tasks.workunit.client.0.smithi195.stdout: create 104/39 file after the snapshot 2022-06-27T11:49:09.746 INFO:tasks.workunit.client.0.smithi195.stdout: create 105/.snap/snap-subdir-test 2022-06-27T11:49:09.781 INFO:tasks.workunit.client.0.smithi195.stdout: create 105/30 file after the snapshot 2022-06-27T11:49:09.795 INFO:tasks.workunit.client.0.smithi195.stdout: create 105/31 file after the snapshot 2022-06-27T11:49:09.796 INFO:tasks.workunit.client.0.smithi195.stdout: create 105/32 file after the snapshot 2022-06-27T11:49:09.796 INFO:tasks.workunit.client.0.smithi195.stdout: create 105/33 file after the snapshot 2022-06-27T11:49:09.796 INFO:tasks.workunit.client.0.smithi195.stdout: create 105/34 file after the snapshot 2022-06-27T11:49:09.799 INFO:tasks.workunit.client.0.smithi195.stdout: create 105/35 file after the snapshot 2022-06-27T11:49:09.799 INFO:tasks.workunit.client.0.smithi195.stdout: create 105/36 file after the snapshot 2022-06-27T11:49:09.800 INFO:tasks.workunit.client.0.smithi195.stdout: create 105/37 file after the snapshot 2022-06-27T11:49:09.800 INFO:tasks.workunit.client.0.smithi195.stdout: create 105/38 file after the snapshot 2022-06-27T11:49:09.801 INFO:tasks.workunit.client.0.smithi195.stdout: create 105/39 file after the snapshot 2022-06-27T11:49:09.801 INFO:tasks.workunit.client.0.smithi195.stdout: create 106/.snap/snap-subdir-test 2022-06-27T11:49:09.836 INFO:tasks.workunit.client.0.smithi195.stdout: create 106/30 file after the snapshot 2022-06-27T11:49:09.852 INFO:tasks.workunit.client.0.smithi195.stdout: create 106/31 file after the snapshot 2022-06-27T11:49:09.853 INFO:tasks.workunit.client.0.smithi195.stdout: create 106/32 file after the snapshot 2022-06-27T11:49:09.853 INFO:tasks.workunit.client.0.smithi195.stdout: create 106/33 file after the snapshot 2022-06-27T11:49:09.854 INFO:tasks.workunit.client.0.smithi195.stdout: create 106/34 file after the snapshot 2022-06-27T11:49:09.854 INFO:tasks.workunit.client.0.smithi195.stdout: create 106/35 file after the snapshot 2022-06-27T11:49:09.855 INFO:tasks.workunit.client.0.smithi195.stdout: create 106/36 file after the snapshot 2022-06-27T11:49:09.856 INFO:tasks.workunit.client.0.smithi195.stdout: create 106/37 file after the snapshot 2022-06-27T11:49:09.857 INFO:tasks.workunit.client.0.smithi195.stdout: create 106/38 file after the snapshot 2022-06-27T11:49:09.858 INFO:tasks.workunit.client.0.smithi195.stdout: create 106/39 file after the snapshot 2022-06-27T11:49:09.858 INFO:tasks.workunit.client.0.smithi195.stdout: create 107/.snap/snap-subdir-test 2022-06-27T11:49:09.906 INFO:tasks.workunit.client.0.smithi195.stdout: create 107/30 file after the snapshot 2022-06-27T11:49:09.921 INFO:tasks.workunit.client.0.smithi195.stdout: create 107/31 file after the snapshot 2022-06-27T11:49:09.923 INFO:tasks.workunit.client.0.smithi195.stdout: create 107/32 file after the snapshot 2022-06-27T11:49:09.924 INFO:tasks.workunit.client.0.smithi195.stdout: create 107/33 file after the snapshot 2022-06-27T11:49:09.925 INFO:tasks.workunit.client.0.smithi195.stdout: create 107/34 file after the snapshot 2022-06-27T11:49:09.925 INFO:tasks.workunit.client.0.smithi195.stdout: create 107/35 file after the snapshot 2022-06-27T11:49:09.938 INFO:tasks.workunit.client.0.smithi195.stdout: create 107/36 file after the snapshot 2022-06-27T11:49:09.939 INFO:tasks.workunit.client.0.smithi195.stdout: create 107/37 file after the snapshot 2022-06-27T11:49:09.939 INFO:tasks.workunit.client.0.smithi195.stdout: create 107/38 file after the snapshot 2022-06-27T11:49:09.940 INFO:tasks.workunit.client.0.smithi195.stdout: create 107/39 file after the snapshot 2022-06-27T11:49:09.940 INFO:tasks.workunit.client.0.smithi195.stdout: create 108/.snap/snap-subdir-test 2022-06-27T11:49:10.311 INFO:tasks.workunit.client.0.smithi195.stdout: create 108/30 file after the snapshot 2022-06-27T11:49:10.323 INFO:tasks.workunit.client.0.smithi195.stdout: create 108/31 file after the snapshot 2022-06-27T11:49:10.324 INFO:tasks.workunit.client.0.smithi195.stdout: create 108/32 file after the snapshot 2022-06-27T11:49:10.325 INFO:tasks.workunit.client.0.smithi195.stdout: create 108/33 file after the snapshot 2022-06-27T11:49:10.326 INFO:tasks.workunit.client.0.smithi195.stdout: create 108/34 file after the snapshot 2022-06-27T11:49:10.328 INFO:tasks.workunit.client.0.smithi195.stdout: create 108/35 file after the snapshot 2022-06-27T11:49:10.328 INFO:tasks.workunit.client.0.smithi195.stdout: create 108/36 file after the snapshot 2022-06-27T11:49:10.329 INFO:tasks.workunit.client.0.smithi195.stdout: create 108/37 file after the snapshot 2022-06-27T11:49:10.330 INFO:tasks.workunit.client.0.smithi195.stdout: create 108/38 file after the snapshot 2022-06-27T11:49:10.330 INFO:tasks.workunit.client.0.smithi195.stdout: create 108/39 file after the snapshot 2022-06-27T11:49:10.340 INFO:tasks.workunit.client.0.smithi195.stdout: create 109/.snap/snap-subdir-test 2022-06-27T11:49:10.377 INFO:tasks.workunit.client.0.smithi195.stdout: create 109/30 file after the snapshot 2022-06-27T11:49:10.391 INFO:tasks.workunit.client.0.smithi195.stdout: create 109/31 file after the snapshot 2022-06-27T11:49:10.391 INFO:tasks.workunit.client.0.smithi195.stdout: create 109/32 file after the snapshot 2022-06-27T11:49:10.392 INFO:tasks.workunit.client.0.smithi195.stdout: create 109/33 file after the snapshot 2022-06-27T11:49:10.394 INFO:tasks.workunit.client.0.smithi195.stdout: create 109/34 file after the snapshot 2022-06-27T11:49:10.394 INFO:tasks.workunit.client.0.smithi195.stdout: create 109/35 file after the snapshot 2022-06-27T11:49:10.395 INFO:tasks.workunit.client.0.smithi195.stdout: create 109/36 file after the snapshot 2022-06-27T11:49:10.395 INFO:tasks.workunit.client.0.smithi195.stdout: create 109/37 file after the snapshot 2022-06-27T11:49:10.401 INFO:tasks.workunit.client.0.smithi195.stdout: create 109/38 file after the snapshot 2022-06-27T11:49:10.402 INFO:tasks.workunit.client.0.smithi195.stdout: create 109/39 file after the snapshot 2022-06-27T11:49:10.402 INFO:tasks.workunit.client.0.smithi195.stdout: create 110/.snap/snap-subdir-test 2022-06-27T11:49:10.428 INFO:tasks.workunit.client.0.smithi195.stdout: create 110/30 file after the snapshot 2022-06-27T11:49:10.443 INFO:tasks.workunit.client.0.smithi195.stdout: create 110/31 file after the snapshot 2022-06-27T11:49:10.444 INFO:tasks.workunit.client.0.smithi195.stdout: create 110/32 file after the snapshot 2022-06-27T11:49:10.445 INFO:tasks.workunit.client.0.smithi195.stdout: create 110/33 file after the snapshot 2022-06-27T11:49:10.446 INFO:tasks.workunit.client.0.smithi195.stdout: create 110/34 file after the snapshot 2022-06-27T11:49:10.448 INFO:tasks.workunit.client.0.smithi195.stdout: create 110/35 file after the snapshot 2022-06-27T11:49:10.448 INFO:tasks.workunit.client.0.smithi195.stdout: create 110/36 file after the snapshot 2022-06-27T11:49:10.449 INFO:tasks.workunit.client.0.smithi195.stdout: create 110/37 file after the snapshot 2022-06-27T11:49:10.460 INFO:tasks.workunit.client.0.smithi195.stdout: create 110/38 file after the snapshot 2022-06-27T11:49:10.461 INFO:tasks.workunit.client.0.smithi195.stdout: create 110/39 file after the snapshot 2022-06-27T11:49:10.461 INFO:tasks.workunit.client.0.smithi195.stdout: create 111/.snap/snap-subdir-test 2022-06-27T11:49:10.487 INFO:tasks.workunit.client.0.smithi195.stdout: create 111/30 file after the snapshot 2022-06-27T11:49:10.502 INFO:tasks.workunit.client.0.smithi195.stdout: create 111/31 file after the snapshot 2022-06-27T11:49:10.503 INFO:tasks.workunit.client.0.smithi195.stdout: create 111/32 file after the snapshot 2022-06-27T11:49:10.504 INFO:tasks.workunit.client.0.smithi195.stdout: create 111/33 file after the snapshot 2022-06-27T11:49:10.505 INFO:tasks.workunit.client.0.smithi195.stdout: create 111/34 file after the snapshot 2022-06-27T11:49:10.506 INFO:tasks.workunit.client.0.smithi195.stdout: create 111/35 file after the snapshot 2022-06-27T11:49:10.507 INFO:tasks.workunit.client.0.smithi195.stdout: create 111/36 file after the snapshot 2022-06-27T11:49:10.508 INFO:tasks.workunit.client.0.smithi195.stdout: create 111/37 file after the snapshot 2022-06-27T11:49:10.510 INFO:tasks.workunit.client.0.smithi195.stdout: create 111/38 file after the snapshot 2022-06-27T11:49:10.511 INFO:tasks.workunit.client.0.smithi195.stdout: create 111/39 file after the snapshot 2022-06-27T11:49:10.511 INFO:tasks.workunit.client.0.smithi195.stdout: create 112/.snap/snap-subdir-test 2022-06-27T11:49:10.542 INFO:tasks.workunit.client.0.smithi195.stdout: create 112/30 file after the snapshot 2022-06-27T11:49:10.558 INFO:tasks.workunit.client.0.smithi195.stdout: create 112/31 file after the snapshot 2022-06-27T11:49:10.560 INFO:tasks.workunit.client.0.smithi195.stdout: create 112/32 file after the snapshot 2022-06-27T11:49:10.560 INFO:tasks.workunit.client.0.smithi195.stdout: create 112/33 file after the snapshot 2022-06-27T11:49:10.562 INFO:tasks.workunit.client.0.smithi195.stdout: create 112/34 file after the snapshot 2022-06-27T11:49:10.562 INFO:tasks.workunit.client.0.smithi195.stdout: create 112/35 file after the snapshot 2022-06-27T11:49:10.564 INFO:tasks.workunit.client.0.smithi195.stdout: create 112/36 file after the snapshot 2022-06-27T11:49:10.565 INFO:tasks.workunit.client.0.smithi195.stdout: create 112/37 file after the snapshot 2022-06-27T11:49:10.565 INFO:tasks.workunit.client.0.smithi195.stdout: create 112/38 file after the snapshot 2022-06-27T11:49:10.566 INFO:tasks.workunit.client.0.smithi195.stdout: create 112/39 file after the snapshot 2022-06-27T11:49:10.567 INFO:tasks.workunit.client.0.smithi195.stdout: create 113/.snap/snap-subdir-test 2022-06-27T11:49:10.599 INFO:tasks.workunit.client.0.smithi195.stdout: create 113/30 file after the snapshot 2022-06-27T11:49:10.615 INFO:tasks.workunit.client.0.smithi195.stdout: create 113/31 file after the snapshot 2022-06-27T11:49:10.616 INFO:tasks.workunit.client.0.smithi195.stdout: create 113/32 file after the snapshot 2022-06-27T11:49:10.617 INFO:tasks.workunit.client.0.smithi195.stdout: create 113/33 file after the snapshot 2022-06-27T11:49:10.618 INFO:tasks.workunit.client.0.smithi195.stdout: create 113/34 file after the snapshot 2022-06-27T11:49:10.619 INFO:tasks.workunit.client.0.smithi195.stdout: create 113/35 file after the snapshot 2022-06-27T11:49:10.621 INFO:tasks.workunit.client.0.smithi195.stdout: create 113/36 file after the snapshot 2022-06-27T11:49:10.630 INFO:tasks.workunit.client.0.smithi195.stdout: create 113/37 file after the snapshot 2022-06-27T11:49:10.631 INFO:tasks.workunit.client.0.smithi195.stdout: create 113/38 file after the snapshot 2022-06-27T11:49:10.631 INFO:tasks.workunit.client.0.smithi195.stdout: create 113/39 file after the snapshot 2022-06-27T11:49:10.631 INFO:tasks.workunit.client.0.smithi195.stdout: create 114/.snap/snap-subdir-test 2022-06-27T11:49:10.659 INFO:tasks.workunit.client.0.smithi195.stdout: create 114/30 file after the snapshot 2022-06-27T11:49:10.674 INFO:tasks.workunit.client.0.smithi195.stdout: create 114/31 file after the snapshot 2022-06-27T11:49:10.683 INFO:tasks.workunit.client.0.smithi195.stdout: create 114/32 file after the snapshot 2022-06-27T11:49:10.685 INFO:tasks.workunit.client.0.smithi195.stdout: create 114/33 file after the snapshot 2022-06-27T11:49:10.686 INFO:tasks.workunit.client.0.smithi195.stdout: create 114/34 file after the snapshot 2022-06-27T11:49:10.686 INFO:tasks.workunit.client.0.smithi195.stdout: create 114/35 file after the snapshot 2022-06-27T11:49:10.687 INFO:tasks.workunit.client.0.smithi195.stdout: create 114/36 file after the snapshot 2022-06-27T11:49:10.687 INFO:tasks.workunit.client.0.smithi195.stdout: create 114/37 file after the snapshot 2022-06-27T11:49:10.688 INFO:tasks.workunit.client.0.smithi195.stdout: create 114/38 file after the snapshot 2022-06-27T11:49:10.688 INFO:tasks.workunit.client.0.smithi195.stdout: create 114/39 file after the snapshot 2022-06-27T11:49:10.689 INFO:tasks.workunit.client.0.smithi195.stdout: create 115/.snap/snap-subdir-test 2022-06-27T11:49:10.717 INFO:tasks.workunit.client.0.smithi195.stdout: create 115/30 file after the snapshot 2022-06-27T11:49:10.731 INFO:tasks.workunit.client.0.smithi195.stdout: create 115/31 file after the snapshot 2022-06-27T11:49:10.732 INFO:tasks.workunit.client.0.smithi195.stdout: create 115/32 file after the snapshot 2022-06-27T11:49:10.733 INFO:tasks.workunit.client.0.smithi195.stdout: create 115/33 file after the snapshot 2022-06-27T11:49:10.734 INFO:tasks.workunit.client.0.smithi195.stdout: create 115/34 file after the snapshot 2022-06-27T11:49:10.735 INFO:tasks.workunit.client.0.smithi195.stdout: create 115/35 file after the snapshot 2022-06-27T11:49:10.736 INFO:tasks.workunit.client.0.smithi195.stdout: create 115/36 file after the snapshot 2022-06-27T11:49:10.740 INFO:tasks.workunit.client.0.smithi195.stdout: create 115/37 file after the snapshot 2022-06-27T11:49:10.741 INFO:tasks.workunit.client.0.smithi195.stdout: create 115/38 file after the snapshot 2022-06-27T11:49:10.741 INFO:tasks.workunit.client.0.smithi195.stdout: create 115/39 file after the snapshot 2022-06-27T11:49:10.742 INFO:tasks.workunit.client.0.smithi195.stdout: create 116/.snap/snap-subdir-test 2022-06-27T11:49:10.773 INFO:tasks.workunit.client.0.smithi195.stdout: create 116/30 file after the snapshot 2022-06-27T11:49:10.785 INFO:tasks.workunit.client.0.smithi195.stdout: create 116/31 file after the snapshot 2022-06-27T11:49:10.787 INFO:tasks.workunit.client.0.smithi195.stdout: create 116/32 file after the snapshot 2022-06-27T11:49:10.788 INFO:tasks.workunit.client.0.smithi195.stdout: create 116/33 file after the snapshot 2022-06-27T11:49:10.789 INFO:tasks.workunit.client.0.smithi195.stdout: create 116/34 file after the snapshot 2022-06-27T11:49:10.790 INFO:tasks.workunit.client.0.smithi195.stdout: create 116/35 file after the snapshot 2022-06-27T11:49:10.792 INFO:tasks.workunit.client.0.smithi195.stdout: create 116/36 file after the snapshot 2022-06-27T11:49:10.792 INFO:tasks.workunit.client.0.smithi195.stdout: create 116/37 file after the snapshot 2022-06-27T11:49:10.793 INFO:tasks.workunit.client.0.smithi195.stdout: create 116/38 file after the snapshot 2022-06-27T11:49:10.795 INFO:tasks.workunit.client.0.smithi195.stdout: create 116/39 file after the snapshot 2022-06-27T11:49:10.796 INFO:tasks.workunit.client.0.smithi195.stdout: create 117/.snap/snap-subdir-test 2022-06-27T11:49:10.825 INFO:tasks.workunit.client.0.smithi195.stdout: create 117/30 file after the snapshot 2022-06-27T11:49:10.843 INFO:tasks.workunit.client.0.smithi195.stdout: create 117/31 file after the snapshot 2022-06-27T11:49:10.843 INFO:tasks.workunit.client.0.smithi195.stdout: create 117/32 file after the snapshot 2022-06-27T11:49:10.844 INFO:tasks.workunit.client.0.smithi195.stdout: create 117/33 file after the snapshot 2022-06-27T11:49:10.844 INFO:tasks.workunit.client.0.smithi195.stdout: create 117/34 file after the snapshot 2022-06-27T11:49:10.844 INFO:tasks.workunit.client.0.smithi195.stdout: create 117/35 file after the snapshot 2022-06-27T11:49:10.844 INFO:tasks.workunit.client.0.smithi195.stdout: create 117/36 file after the snapshot 2022-06-27T11:49:10.845 INFO:tasks.workunit.client.0.smithi195.stdout: create 117/37 file after the snapshot 2022-06-27T11:49:10.846 INFO:tasks.workunit.client.0.smithi195.stdout: create 117/38 file after the snapshot 2022-06-27T11:49:10.846 INFO:tasks.workunit.client.0.smithi195.stdout: create 117/39 file after the snapshot 2022-06-27T11:49:10.846 INFO:tasks.workunit.client.0.smithi195.stdout: create 118/.snap/snap-subdir-test 2022-06-27T11:49:10.888 INFO:tasks.workunit.client.0.smithi195.stdout: create 118/30 file after the snapshot 2022-06-27T11:49:10.891 INFO:tasks.workunit.client.0.smithi195.stdout: create 118/31 file after the snapshot 2022-06-27T11:49:10.893 INFO:tasks.workunit.client.0.smithi195.stdout: create 118/32 file after the snapshot 2022-06-27T11:49:10.894 INFO:tasks.workunit.client.0.smithi195.stdout: create 118/33 file after the snapshot 2022-06-27T11:49:10.894 INFO:tasks.workunit.client.0.smithi195.stdout: create 118/34 file after the snapshot 2022-06-27T11:49:10.895 INFO:tasks.workunit.client.0.smithi195.stdout: create 118/35 file after the snapshot 2022-06-27T11:49:10.895 INFO:tasks.workunit.client.0.smithi195.stdout: create 118/36 file after the snapshot 2022-06-27T11:49:10.896 INFO:tasks.workunit.client.0.smithi195.stdout: create 118/37 file after the snapshot 2022-06-27T11:49:10.897 INFO:tasks.workunit.client.0.smithi195.stdout: create 118/38 file after the snapshot 2022-06-27T11:49:10.897 INFO:tasks.workunit.client.0.smithi195.stdout: create 118/39 file after the snapshot 2022-06-27T11:49:10.898 INFO:tasks.workunit.client.0.smithi195.stdout: create 119/.snap/snap-subdir-test 2022-06-27T11:49:10.931 INFO:tasks.workunit.client.0.smithi195.stdout: create 119/30 file after the snapshot 2022-06-27T11:49:10.945 INFO:tasks.workunit.client.0.smithi195.stdout: create 119/31 file after the snapshot 2022-06-27T11:49:10.946 INFO:tasks.workunit.client.0.smithi195.stdout: create 119/32 file after the snapshot 2022-06-27T11:49:10.947 INFO:tasks.workunit.client.0.smithi195.stdout: create 119/33 file after the snapshot 2022-06-27T11:49:10.949 INFO:tasks.workunit.client.0.smithi195.stdout: create 119/34 file after the snapshot 2022-06-27T11:49:10.950 INFO:tasks.workunit.client.0.smithi195.stdout: create 119/35 file after the snapshot 2022-06-27T11:49:10.951 INFO:tasks.workunit.client.0.smithi195.stdout: create 119/36 file after the snapshot 2022-06-27T11:49:10.953 INFO:tasks.workunit.client.0.smithi195.stdout: create 119/37 file after the snapshot 2022-06-27T11:49:10.953 INFO:tasks.workunit.client.0.smithi195.stdout: create 119/38 file after the snapshot 2022-06-27T11:49:10.955 INFO:tasks.workunit.client.0.smithi195.stdout: create 119/39 file after the snapshot 2022-06-27T11:49:10.955 INFO:tasks.workunit.client.0.smithi195.stdout: create 120/.snap/snap-subdir-test 2022-06-27T11:49:10.984 INFO:tasks.workunit.client.0.smithi195.stdout: create 120/30 file after the snapshot 2022-06-27T11:49:11.287 INFO:tasks.workunit.client.0.smithi195.stdout: create 120/31 file after the snapshot 2022-06-27T11:49:11.288 INFO:tasks.workunit.client.0.smithi195.stdout: create 120/32 file after the snapshot 2022-06-27T11:49:11.289 INFO:tasks.workunit.client.0.smithi195.stdout: create 120/33 file after the snapshot 2022-06-27T11:49:11.290 INFO:tasks.workunit.client.0.smithi195.stdout: create 120/34 file after the snapshot 2022-06-27T11:49:11.290 INFO:tasks.workunit.client.0.smithi195.stdout: create 120/35 file after the snapshot 2022-06-27T11:49:11.291 INFO:tasks.workunit.client.0.smithi195.stdout: create 120/36 file after the snapshot 2022-06-27T11:49:11.294 INFO:tasks.workunit.client.0.smithi195.stdout: create 120/37 file after the snapshot 2022-06-27T11:49:11.295 INFO:tasks.workunit.client.0.smithi195.stdout: create 120/38 file after the snapshot 2022-06-27T11:49:11.295 INFO:tasks.workunit.client.0.smithi195.stdout: create 120/39 file after the snapshot 2022-06-27T11:49:11.296 INFO:tasks.workunit.client.0.smithi195.stdout: create 121/.snap/snap-subdir-test 2022-06-27T11:49:11.323 INFO:tasks.workunit.client.0.smithi195.stdout: create 121/30 file after the snapshot 2022-06-27T11:49:11.337 INFO:tasks.workunit.client.0.smithi195.stdout: create 121/31 file after the snapshot 2022-06-27T11:49:11.338 INFO:tasks.workunit.client.0.smithi195.stdout: create 121/32 file after the snapshot 2022-06-27T11:49:11.339 INFO:tasks.workunit.client.0.smithi195.stdout: create 121/33 file after the snapshot 2022-06-27T11:49:11.339 INFO:tasks.workunit.client.0.smithi195.stdout: create 121/34 file after the snapshot 2022-06-27T11:49:11.340 INFO:tasks.workunit.client.0.smithi195.stdout: create 121/35 file after the snapshot 2022-06-27T11:49:11.341 INFO:tasks.workunit.client.0.smithi195.stdout: create 121/36 file after the snapshot 2022-06-27T11:49:11.343 INFO:tasks.workunit.client.0.smithi195.stdout: create 121/37 file after the snapshot 2022-06-27T11:49:11.343 INFO:tasks.workunit.client.0.smithi195.stdout: create 121/38 file after the snapshot 2022-06-27T11:49:11.344 INFO:tasks.workunit.client.0.smithi195.stdout: create 121/39 file after the snapshot 2022-06-27T11:49:11.345 INFO:tasks.workunit.client.0.smithi195.stdout: create 122/.snap/snap-subdir-test 2022-06-27T11:49:11.376 INFO:tasks.workunit.client.0.smithi195.stdout: create 122/30 file after the snapshot 2022-06-27T11:49:11.399 INFO:tasks.workunit.client.0.smithi195.stdout: create 122/31 file after the snapshot 2022-06-27T11:49:11.400 INFO:tasks.workunit.client.0.smithi195.stdout: create 122/32 file after the snapshot 2022-06-27T11:49:11.400 INFO:tasks.workunit.client.0.smithi195.stdout: create 122/33 file after the snapshot 2022-06-27T11:49:11.400 INFO:tasks.workunit.client.0.smithi195.stdout: create 122/34 file after the snapshot 2022-06-27T11:49:11.401 INFO:tasks.workunit.client.0.smithi195.stdout: create 122/35 file after the snapshot 2022-06-27T11:49:11.401 INFO:tasks.workunit.client.0.smithi195.stdout: create 122/36 file after the snapshot 2022-06-27T11:49:11.401 INFO:tasks.workunit.client.0.smithi195.stdout: create 122/37 file after the snapshot 2022-06-27T11:49:11.401 INFO:tasks.workunit.client.0.smithi195.stdout: create 122/38 file after the snapshot 2022-06-27T11:49:11.402 INFO:tasks.workunit.client.0.smithi195.stdout: create 122/39 file after the snapshot 2022-06-27T11:49:11.402 INFO:tasks.workunit.client.0.smithi195.stdout: create 123/.snap/snap-subdir-test 2022-06-27T11:49:11.429 INFO:tasks.workunit.client.0.smithi195.stdout: create 123/30 file after the snapshot 2022-06-27T11:49:11.445 INFO:tasks.workunit.client.0.smithi195.stdout: create 123/31 file after the snapshot 2022-06-27T11:49:11.446 INFO:tasks.workunit.client.0.smithi195.stdout: create 123/32 file after the snapshot 2022-06-27T11:49:11.449 INFO:tasks.workunit.client.0.smithi195.stdout: create 123/33 file after the snapshot 2022-06-27T11:49:11.449 INFO:tasks.workunit.client.0.smithi195.stdout: create 123/34 file after the snapshot 2022-06-27T11:49:11.449 INFO:tasks.workunit.client.0.smithi195.stdout: create 123/35 file after the snapshot 2022-06-27T11:49:11.449 INFO:tasks.workunit.client.0.smithi195.stdout: create 123/36 file after the snapshot 2022-06-27T11:49:11.450 INFO:tasks.workunit.client.0.smithi195.stdout: create 123/37 file after the snapshot 2022-06-27T11:49:11.451 INFO:tasks.workunit.client.0.smithi195.stdout: create 123/38 file after the snapshot 2022-06-27T11:49:11.452 INFO:tasks.workunit.client.0.smithi195.stdout: create 123/39 file after the snapshot 2022-06-27T11:49:11.452 INFO:tasks.workunit.client.0.smithi195.stdout: create 124/.snap/snap-subdir-test 2022-06-27T11:49:11.482 INFO:tasks.workunit.client.0.smithi195.stdout: create 124/30 file after the snapshot 2022-06-27T11:49:11.503 INFO:tasks.workunit.client.0.smithi195.stdout: create 124/31 file after the snapshot 2022-06-27T11:49:11.504 INFO:tasks.workunit.client.0.smithi195.stdout: create 124/32 file after the snapshot 2022-06-27T11:49:11.504 INFO:tasks.workunit.client.0.smithi195.stdout: create 124/33 file after the snapshot 2022-06-27T11:49:11.505 INFO:tasks.workunit.client.0.smithi195.stdout: create 124/34 file after the snapshot 2022-06-27T11:49:11.505 INFO:tasks.workunit.client.0.smithi195.stdout: create 124/35 file after the snapshot 2022-06-27T11:49:11.506 INFO:tasks.workunit.client.0.smithi195.stdout: create 124/36 file after the snapshot 2022-06-27T11:49:11.509 INFO:tasks.workunit.client.0.smithi195.stdout: create 124/37 file after the snapshot 2022-06-27T11:49:11.509 INFO:tasks.workunit.client.0.smithi195.stdout: create 124/38 file after the snapshot 2022-06-27T11:49:11.509 INFO:tasks.workunit.client.0.smithi195.stdout: create 124/39 file after the snapshot 2022-06-27T11:49:11.510 INFO:tasks.workunit.client.0.smithi195.stdout: create 125/.snap/snap-subdir-test 2022-06-27T11:49:11.537 INFO:tasks.workunit.client.0.smithi195.stdout: create 125/30 file after the snapshot 2022-06-27T11:49:11.554 INFO:tasks.workunit.client.0.smithi195.stdout: create 125/31 file after the snapshot 2022-06-27T11:49:11.555 INFO:tasks.workunit.client.0.smithi195.stdout: create 125/32 file after the snapshot 2022-06-27T11:49:11.555 INFO:tasks.workunit.client.0.smithi195.stdout: create 125/33 file after the snapshot 2022-06-27T11:49:11.556 INFO:tasks.workunit.client.0.smithi195.stdout: create 125/34 file after the snapshot 2022-06-27T11:49:11.557 INFO:tasks.workunit.client.0.smithi195.stdout: create 125/35 file after the snapshot 2022-06-27T11:49:11.558 INFO:tasks.workunit.client.0.smithi195.stdout: create 125/36 file after the snapshot 2022-06-27T11:49:11.559 INFO:tasks.workunit.client.0.smithi195.stdout: create 125/37 file after the snapshot 2022-06-27T11:49:11.560 INFO:tasks.workunit.client.0.smithi195.stdout: create 125/38 file after the snapshot 2022-06-27T11:49:11.560 INFO:tasks.workunit.client.0.smithi195.stdout: create 125/39 file after the snapshot 2022-06-27T11:49:11.561 INFO:tasks.workunit.client.0.smithi195.stdout: create 126/.snap/snap-subdir-test 2022-06-27T11:49:11.595 INFO:tasks.workunit.client.0.smithi195.stdout: create 126/30 file after the snapshot 2022-06-27T11:49:11.617 INFO:tasks.workunit.client.0.smithi195.stdout: create 126/31 file after the snapshot 2022-06-27T11:49:11.618 INFO:tasks.workunit.client.0.smithi195.stdout: create 126/32 file after the snapshot 2022-06-27T11:49:11.618 INFO:tasks.workunit.client.0.smithi195.stdout: create 126/33 file after the snapshot 2022-06-27T11:49:11.619 INFO:tasks.workunit.client.0.smithi195.stdout: create 126/34 file after the snapshot 2022-06-27T11:49:11.619 INFO:tasks.workunit.client.0.smithi195.stdout: create 126/35 file after the snapshot 2022-06-27T11:49:11.620 INFO:tasks.workunit.client.0.smithi195.stdout: create 126/36 file after the snapshot 2022-06-27T11:49:11.620 INFO:tasks.workunit.client.0.smithi195.stdout: create 126/37 file after the snapshot 2022-06-27T11:49:11.621 INFO:tasks.workunit.client.0.smithi195.stdout: create 126/38 file after the snapshot 2022-06-27T11:49:11.622 INFO:tasks.workunit.client.0.smithi195.stdout: create 126/39 file after the snapshot 2022-06-27T11:49:11.622 INFO:tasks.workunit.client.0.smithi195.stdout: create 127/.snap/snap-subdir-test 2022-06-27T11:49:11.647 INFO:tasks.workunit.client.0.smithi195.stdout: create 127/30 file after the snapshot 2022-06-27T11:49:11.658 INFO:tasks.workunit.client.0.smithi195.stdout: create 127/31 file after the snapshot 2022-06-27T11:49:11.661 INFO:tasks.workunit.client.0.smithi195.stdout: create 127/32 file after the snapshot 2022-06-27T11:49:11.662 INFO:tasks.workunit.client.0.smithi195.stdout: create 127/33 file after the snapshot 2022-06-27T11:49:11.662 INFO:tasks.workunit.client.0.smithi195.stdout: create 127/34 file after the snapshot 2022-06-27T11:49:11.663 INFO:tasks.workunit.client.0.smithi195.stdout: create 127/35 file after the snapshot 2022-06-27T11:49:11.664 INFO:tasks.workunit.client.0.smithi195.stdout: create 127/36 file after the snapshot 2022-06-27T11:49:11.665 INFO:tasks.workunit.client.0.smithi195.stdout: create 127/37 file after the snapshot 2022-06-27T11:49:11.666 INFO:tasks.workunit.client.0.smithi195.stdout: create 127/38 file after the snapshot 2022-06-27T11:49:11.683 INFO:tasks.workunit.client.0.smithi195.stdout: create 127/39 file after the snapshot 2022-06-27T11:49:11.683 INFO:tasks.workunit.client.0.smithi195.stdout: create 128/.snap/snap-subdir-test 2022-06-27T11:49:11.707 INFO:tasks.workunit.client.0.smithi195.stdout: create 128/30 file after the snapshot 2022-06-27T11:49:11.747 INFO:tasks.workunit.client.0.smithi195.stdout: create 128/31 file after the snapshot 2022-06-27T11:49:11.747 INFO:tasks.workunit.client.0.smithi195.stdout: create 128/32 file after the snapshot 2022-06-27T11:49:11.748 INFO:tasks.workunit.client.0.smithi195.stdout: create 128/33 file after the snapshot 2022-06-27T11:49:11.748 INFO:tasks.workunit.client.0.smithi195.stdout: create 128/34 file after the snapshot 2022-06-27T11:49:11.748 INFO:tasks.workunit.client.0.smithi195.stdout: create 128/35 file after the snapshot 2022-06-27T11:49:11.749 INFO:tasks.workunit.client.0.smithi195.stdout: create 128/36 file after the snapshot 2022-06-27T11:49:11.749 INFO:tasks.workunit.client.0.smithi195.stdout: create 128/37 file after the snapshot 2022-06-27T11:49:11.760 INFO:tasks.workunit.client.0.smithi195.stdout: create 128/38 file after the snapshot 2022-06-27T11:49:11.761 INFO:tasks.workunit.client.0.smithi195.stdout: create 128/39 file after the snapshot 2022-06-27T11:49:11.761 INFO:tasks.workunit.client.0.smithi195.stdout: create 129/.snap/snap-subdir-test 2022-06-27T11:49:11.762 INFO:tasks.workunit.client.0.smithi195.stdout: create 129/30 file after the snapshot 2022-06-27T11:49:11.766 INFO:tasks.workunit.client.0.smithi195.stdout: create 129/31 file after the snapshot 2022-06-27T11:49:11.785 INFO:tasks.workunit.client.0.smithi195.stdout: create 129/32 file after the snapshot 2022-06-27T11:49:11.786 INFO:tasks.workunit.client.0.smithi195.stdout: create 129/33 file after the snapshot 2022-06-27T11:49:11.786 INFO:tasks.workunit.client.0.smithi195.stdout: create 129/34 file after the snapshot 2022-06-27T11:49:11.787 INFO:tasks.workunit.client.0.smithi195.stdout: create 129/35 file after the snapshot 2022-06-27T11:49:11.787 INFO:tasks.workunit.client.0.smithi195.stdout: create 129/36 file after the snapshot 2022-06-27T11:49:11.788 INFO:tasks.workunit.client.0.smithi195.stdout: create 129/37 file after the snapshot 2022-06-27T11:49:11.788 INFO:tasks.workunit.client.0.smithi195.stdout: create 129/38 file after the snapshot 2022-06-27T11:49:11.789 INFO:tasks.workunit.client.0.smithi195.stdout: create 129/39 file after the snapshot 2022-06-27T11:49:11.789 INFO:tasks.workunit.client.0.smithi195.stdout: create 130/.snap/snap-subdir-test 2022-06-27T11:49:11.813 INFO:tasks.workunit.client.0.smithi195.stdout: create 130/30 file after the snapshot 2022-06-27T11:49:11.813 INFO:tasks.workunit.client.0.smithi195.stdout: create 130/31 file after the snapshot 2022-06-27T11:49:11.814 INFO:tasks.workunit.client.0.smithi195.stdout: create 130/32 file after the snapshot 2022-06-27T11:49:11.833 INFO:tasks.workunit.client.0.smithi195.stdout: create 130/33 file after the snapshot 2022-06-27T11:49:11.833 INFO:tasks.workunit.client.0.smithi195.stdout: create 130/34 file after the snapshot 2022-06-27T11:49:11.852 INFO:tasks.workunit.client.0.smithi195.stdout: create 130/35 file after the snapshot 2022-06-27T11:49:11.853 INFO:tasks.workunit.client.0.smithi195.stdout: create 130/36 file after the snapshot 2022-06-27T11:49:11.853 INFO:tasks.workunit.client.0.smithi195.stdout: create 130/37 file after the snapshot 2022-06-27T11:49:11.854 INFO:tasks.workunit.client.0.smithi195.stdout: create 130/38 file after the snapshot 2022-06-27T11:49:11.854 INFO:tasks.workunit.client.0.smithi195.stdout: create 130/39 file after the snapshot 2022-06-27T11:49:11.855 INFO:tasks.workunit.client.0.smithi195.stdout: create 131/.snap/snap-subdir-test 2022-06-27T11:49:11.863 INFO:tasks.workunit.client.0.smithi195.stdout: create 131/30 file after the snapshot 2022-06-27T11:49:11.867 INFO:tasks.workunit.client.0.smithi195.stdout: create 131/31 file after the snapshot 2022-06-27T11:49:11.867 INFO:tasks.workunit.client.0.smithi195.stdout: create 131/32 file after the snapshot 2022-06-27T11:49:11.875 INFO:tasks.workunit.client.0.smithi195.stdout: create 131/33 file after the snapshot 2022-06-27T11:49:11.876 INFO:tasks.workunit.client.0.smithi195.stdout: create 131/34 file after the snapshot 2022-06-27T11:49:11.877 INFO:tasks.workunit.client.0.smithi195.stdout: create 131/35 file after the snapshot 2022-06-27T11:49:11.877 INFO:tasks.workunit.client.0.smithi195.stdout: create 131/36 file after the snapshot 2022-06-27T11:49:11.878 INFO:tasks.workunit.client.0.smithi195.stdout: create 131/37 file after the snapshot 2022-06-27T11:49:11.878 INFO:tasks.workunit.client.0.smithi195.stdout: create 131/38 file after the snapshot 2022-06-27T11:49:11.882 INFO:tasks.workunit.client.0.smithi195.stdout: create 131/39 file after the snapshot 2022-06-27T11:49:11.883 INFO:tasks.workunit.client.0.smithi195.stdout: create 132/.snap/snap-subdir-test 2022-06-27T11:49:11.913 INFO:tasks.workunit.client.0.smithi195.stdout: create 132/30 file after the snapshot 2022-06-27T11:49:11.933 INFO:tasks.workunit.client.0.smithi195.stdout: create 132/31 file after the snapshot 2022-06-27T11:49:11.934 INFO:tasks.workunit.client.0.smithi195.stdout: create 132/32 file after the snapshot 2022-06-27T11:49:11.935 INFO:tasks.workunit.client.0.smithi195.stdout: create 132/33 file after the snapshot 2022-06-27T11:49:11.938 INFO:tasks.workunit.client.0.smithi195.stdout: create 132/34 file after the snapshot 2022-06-27T11:49:11.939 INFO:tasks.workunit.client.0.smithi195.stdout: create 132/35 file after the snapshot 2022-06-27T11:49:11.940 INFO:tasks.workunit.client.0.smithi195.stdout: create 132/36 file after the snapshot 2022-06-27T11:49:11.940 INFO:tasks.workunit.client.0.smithi195.stdout: create 132/37 file after the snapshot 2022-06-27T11:49:11.943 INFO:tasks.workunit.client.0.smithi195.stdout: create 132/38 file after the snapshot 2022-06-27T11:49:11.944 INFO:tasks.workunit.client.0.smithi195.stdout: create 132/39 file after the snapshot 2022-06-27T11:49:12.218 INFO:tasks.workunit.client.0.smithi195.stdout: create 133/.snap/snap-subdir-test 2022-06-27T11:49:12.253 INFO:tasks.workunit.client.0.smithi195.stdout: create 133/30 file after the snapshot 2022-06-27T11:49:12.264 INFO:tasks.workunit.client.0.smithi195.stdout: create 133/31 file after the snapshot 2022-06-27T11:49:12.566 INFO:tasks.workunit.client.0.smithi195.stdout: create 133/32 file after the snapshot 2022-06-27T11:49:12.568 INFO:tasks.workunit.client.0.smithi195.stdout: create 133/33 file after the snapshot 2022-06-27T11:49:12.568 INFO:tasks.workunit.client.0.smithi195.stdout: create 133/34 file after the snapshot 2022-06-27T11:49:12.569 INFO:tasks.workunit.client.0.smithi195.stdout: create 133/35 file after the snapshot 2022-06-27T11:49:12.571 INFO:tasks.workunit.client.0.smithi195.stdout: create 133/36 file after the snapshot 2022-06-27T11:49:12.571 INFO:tasks.workunit.client.0.smithi195.stdout: create 133/37 file after the snapshot 2022-06-27T11:49:12.572 INFO:tasks.workunit.client.0.smithi195.stdout: create 133/38 file after the snapshot 2022-06-27T11:49:12.572 INFO:tasks.workunit.client.0.smithi195.stdout: create 133/39 file after the snapshot 2022-06-27T11:49:12.573 INFO:tasks.workunit.client.0.smithi195.stdout: create 134/.snap/snap-subdir-test 2022-06-27T11:49:12.583 INFO:tasks.workunit.client.0.smithi195.stdout: create 134/30 file after the snapshot 2022-06-27T11:49:12.601 INFO:tasks.workunit.client.0.smithi195.stdout: create 134/31 file after the snapshot 2022-06-27T11:49:12.602 INFO:tasks.workunit.client.0.smithi195.stdout: create 134/32 file after the snapshot 2022-06-27T11:49:12.603 INFO:tasks.workunit.client.0.smithi195.stdout: create 134/33 file after the snapshot 2022-06-27T11:49:12.603 INFO:tasks.workunit.client.0.smithi195.stdout: create 134/34 file after the snapshot 2022-06-27T11:49:12.604 INFO:tasks.workunit.client.0.smithi195.stdout: create 134/35 file after the snapshot 2022-06-27T11:49:12.612 INFO:tasks.workunit.client.0.smithi195.stdout: create 134/36 file after the snapshot 2022-06-27T11:49:12.612 INFO:tasks.workunit.client.0.smithi195.stdout: create 134/37 file after the snapshot 2022-06-27T11:49:12.613 INFO:tasks.workunit.client.0.smithi195.stdout: create 134/38 file after the snapshot 2022-06-27T11:49:12.613 INFO:tasks.workunit.client.0.smithi195.stdout: create 134/39 file after the snapshot 2022-06-27T11:49:12.614 INFO:tasks.workunit.client.0.smithi195.stdout: create 135/.snap/snap-subdir-test 2022-06-27T11:49:12.637 INFO:tasks.workunit.client.0.smithi195.stdout: create 135/30 file after the snapshot 2022-06-27T11:49:12.638 INFO:tasks.workunit.client.0.smithi195.stdout: create 135/31 file after the snapshot 2022-06-27T11:49:12.639 INFO:tasks.workunit.client.0.smithi195.stdout: create 135/32 file after the snapshot 2022-06-27T11:49:12.639 INFO:tasks.workunit.client.0.smithi195.stdout: create 135/33 file after the snapshot 2022-06-27T11:49:12.640 INFO:tasks.workunit.client.0.smithi195.stdout: create 135/34 file after the snapshot 2022-06-27T11:49:12.653 INFO:tasks.workunit.client.0.smithi195.stdout: create 135/35 file after the snapshot 2022-06-27T11:49:12.653 INFO:tasks.workunit.client.0.smithi195.stdout: create 135/36 file after the snapshot 2022-06-27T11:49:12.654 INFO:tasks.workunit.client.0.smithi195.stdout: create 135/37 file after the snapshot 2022-06-27T11:49:12.654 INFO:tasks.workunit.client.0.smithi195.stdout: create 135/38 file after the snapshot 2022-06-27T11:49:12.655 INFO:tasks.workunit.client.0.smithi195.stdout: create 135/39 file after the snapshot 2022-06-27T11:49:12.655 INFO:tasks.workunit.client.0.smithi195.stdout: create 136/.snap/snap-subdir-test 2022-06-27T11:49:12.683 INFO:tasks.workunit.client.0.smithi195.stdout: create 136/30 file after the snapshot 2022-06-27T11:49:12.712 INFO:tasks.workunit.client.0.smithi195.stdout: create 136/31 file after the snapshot 2022-06-27T11:49:12.713 INFO:tasks.workunit.client.0.smithi195.stdout: create 136/32 file after the snapshot 2022-06-27T11:49:12.713 INFO:tasks.workunit.client.0.smithi195.stdout: create 136/33 file after the snapshot 2022-06-27T11:49:12.714 INFO:tasks.workunit.client.0.smithi195.stdout: create 136/34 file after the snapshot 2022-06-27T11:49:12.714 INFO:tasks.workunit.client.0.smithi195.stdout: create 136/35 file after the snapshot 2022-06-27T11:49:12.715 INFO:tasks.workunit.client.0.smithi195.stdout: create 136/36 file after the snapshot 2022-06-27T11:49:12.717 INFO:tasks.workunit.client.0.smithi195.stdout: create 136/37 file after the snapshot 2022-06-27T11:49:12.718 INFO:tasks.workunit.client.0.smithi195.stdout: create 136/38 file after the snapshot 2022-06-27T11:49:12.718 INFO:tasks.workunit.client.0.smithi195.stdout: create 136/39 file after the snapshot 2022-06-27T11:49:12.719 INFO:tasks.workunit.client.0.smithi195.stdout: create 137/.snap/snap-subdir-test 2022-06-27T11:49:12.736 INFO:tasks.workunit.client.0.smithi195.stdout: create 137/30 file after the snapshot 2022-06-27T11:49:12.751 INFO:tasks.workunit.client.0.smithi195.stdout: create 137/31 file after the snapshot 2022-06-27T11:49:12.758 INFO:tasks.workunit.client.0.smithi195.stdout: create 137/32 file after the snapshot 2022-06-27T11:49:12.764 INFO:tasks.workunit.client.0.smithi195.stdout: create 137/33 file after the snapshot 2022-06-27T11:49:12.765 INFO:tasks.workunit.client.0.smithi195.stdout: create 137/34 file after the snapshot 2022-06-27T11:49:12.765 INFO:tasks.workunit.client.0.smithi195.stdout: create 137/35 file after the snapshot 2022-06-27T11:49:12.765 INFO:tasks.workunit.client.0.smithi195.stdout: create 137/36 file after the snapshot 2022-06-27T11:49:12.766 INFO:tasks.workunit.client.0.smithi195.stdout: create 137/37 file after the snapshot 2022-06-27T11:49:12.771 INFO:tasks.workunit.client.0.smithi195.stdout: create 137/38 file after the snapshot 2022-06-27T11:49:12.772 INFO:tasks.workunit.client.0.smithi195.stdout: create 137/39 file after the snapshot 2022-06-27T11:49:12.773 INFO:tasks.workunit.client.0.smithi195.stdout: create 138/.snap/snap-subdir-test 2022-06-27T11:49:12.789 INFO:tasks.workunit.client.0.smithi195.stdout: create 138/30 file after the snapshot 2022-06-27T11:49:12.803 INFO:tasks.workunit.client.0.smithi195.stdout: create 138/31 file after the snapshot 2022-06-27T11:49:12.805 INFO:tasks.workunit.client.0.smithi195.stdout: create 138/32 file after the snapshot 2022-06-27T11:49:12.806 INFO:tasks.workunit.client.0.smithi195.stdout: create 138/33 file after the snapshot 2022-06-27T11:49:12.807 INFO:tasks.workunit.client.0.smithi195.stdout: create 138/34 file after the snapshot 2022-06-27T11:49:12.808 INFO:tasks.workunit.client.0.smithi195.stdout: create 138/35 file after the snapshot 2022-06-27T11:49:12.808 INFO:tasks.workunit.client.0.smithi195.stdout: create 138/36 file after the snapshot 2022-06-27T11:49:12.809 INFO:tasks.workunit.client.0.smithi195.stdout: create 138/37 file after the snapshot 2022-06-27T11:49:12.811 INFO:tasks.workunit.client.0.smithi195.stdout: create 138/38 file after the snapshot 2022-06-27T11:49:12.811 INFO:tasks.workunit.client.0.smithi195.stdout: create 138/39 file after the snapshot 2022-06-27T11:49:12.812 INFO:tasks.workunit.client.0.smithi195.stdout: create 139/.snap/snap-subdir-test 2022-06-27T11:49:12.844 INFO:tasks.workunit.client.0.smithi195.stdout: create 139/30 file after the snapshot 2022-06-27T11:49:12.871 INFO:tasks.workunit.client.0.smithi195.stdout: create 139/31 file after the snapshot 2022-06-27T11:49:12.872 INFO:tasks.workunit.client.0.smithi195.stdout: create 139/32 file after the snapshot 2022-06-27T11:49:12.873 INFO:tasks.workunit.client.0.smithi195.stdout: create 139/33 file after the snapshot 2022-06-27T11:49:12.874 INFO:tasks.workunit.client.0.smithi195.stdout: create 139/34 file after the snapshot 2022-06-27T11:49:12.875 INFO:tasks.workunit.client.0.smithi195.stdout: create 139/35 file after the snapshot 2022-06-27T11:49:12.875 INFO:tasks.workunit.client.0.smithi195.stdout: create 139/36 file after the snapshot 2022-06-27T11:49:12.876 INFO:tasks.workunit.client.0.smithi195.stdout: create 139/37 file after the snapshot 2022-06-27T11:49:12.876 INFO:tasks.workunit.client.0.smithi195.stdout: create 139/38 file after the snapshot 2022-06-27T11:49:12.877 INFO:tasks.workunit.client.0.smithi195.stdout: create 139/39 file after the snapshot 2022-06-27T11:49:12.877 INFO:tasks.workunit.client.0.smithi195.stdout: create 140/.snap/snap-subdir-test 2022-06-27T11:49:12.893 INFO:tasks.workunit.client.0.smithi195.stdout: create 140/30 file after the snapshot 2022-06-27T11:49:12.909 INFO:tasks.workunit.client.0.smithi195.stdout: create 140/31 file after the snapshot 2022-06-27T11:49:12.909 INFO:tasks.workunit.client.0.smithi195.stdout: create 140/32 file after the snapshot 2022-06-27T11:49:12.910 INFO:tasks.workunit.client.0.smithi195.stdout: create 140/33 file after the snapshot 2022-06-27T11:49:12.911 INFO:tasks.workunit.client.0.smithi195.stdout: create 140/34 file after the snapshot 2022-06-27T11:49:12.911 INFO:tasks.workunit.client.0.smithi195.stdout: create 140/35 file after the snapshot 2022-06-27T11:49:12.913 INFO:tasks.workunit.client.0.smithi195.stdout: create 140/36 file after the snapshot 2022-06-27T11:49:12.914 INFO:tasks.workunit.client.0.smithi195.stdout: create 140/37 file after the snapshot 2022-06-27T11:49:12.915 INFO:tasks.workunit.client.0.smithi195.stdout: create 140/38 file after the snapshot 2022-06-27T11:49:12.916 INFO:tasks.workunit.client.0.smithi195.stdout: create 140/39 file after the snapshot 2022-06-27T11:49:12.917 INFO:tasks.workunit.client.0.smithi195.stdout: create 141/.snap/snap-subdir-test 2022-06-27T11:49:12.948 INFO:tasks.workunit.client.0.smithi195.stdout: create 141/30 file after the snapshot 2022-06-27T11:49:12.970 INFO:tasks.workunit.client.0.smithi195.stdout: create 141/31 file after the snapshot 2022-06-27T11:49:12.971 INFO:tasks.workunit.client.0.smithi195.stdout: create 141/32 file after the snapshot 2022-06-27T11:49:12.971 INFO:tasks.workunit.client.0.smithi195.stdout: create 141/33 file after the snapshot 2022-06-27T11:49:12.972 INFO:tasks.workunit.client.0.smithi195.stdout: create 141/34 file after the snapshot 2022-06-27T11:49:12.972 INFO:tasks.workunit.client.0.smithi195.stdout: create 141/35 file after the snapshot 2022-06-27T11:49:12.972 INFO:tasks.workunit.client.0.smithi195.stdout: create 141/36 file after the snapshot 2022-06-27T11:49:12.977 INFO:tasks.workunit.client.0.smithi195.stdout: create 141/37 file after the snapshot 2022-06-27T11:49:12.977 INFO:tasks.workunit.client.0.smithi195.stdout: create 141/38 file after the snapshot 2022-06-27T11:49:12.979 INFO:tasks.workunit.client.0.smithi195.stdout: create 141/39 file after the snapshot 2022-06-27T11:49:12.979 INFO:tasks.workunit.client.0.smithi195.stdout: create 142/.snap/snap-subdir-test 2022-06-27T11:49:13.015 INFO:tasks.workunit.client.0.smithi195.stdout: create 142/30 file after the snapshot 2022-06-27T11:49:13.019 INFO:tasks.workunit.client.0.smithi195.stdout: create 142/31 file after the snapshot 2022-06-27T11:49:13.020 INFO:tasks.workunit.client.0.smithi195.stdout: create 142/32 file after the snapshot 2022-06-27T11:49:13.024 INFO:tasks.workunit.client.0.smithi195.stdout: create 142/33 file after the snapshot 2022-06-27T11:49:13.024 INFO:tasks.workunit.client.0.smithi195.stdout: create 142/34 file after the snapshot 2022-06-27T11:49:13.025 INFO:tasks.workunit.client.0.smithi195.stdout: create 142/35 file after the snapshot 2022-06-27T11:49:13.025 INFO:tasks.workunit.client.0.smithi195.stdout: create 142/36 file after the snapshot 2022-06-27T11:49:13.027 INFO:tasks.workunit.client.0.smithi195.stdout: create 142/37 file after the snapshot 2022-06-27T11:49:13.027 INFO:tasks.workunit.client.0.smithi195.stdout: create 142/38 file after the snapshot 2022-06-27T11:49:13.028 INFO:tasks.workunit.client.0.smithi195.stdout: create 142/39 file after the snapshot 2022-06-27T11:49:13.028 INFO:tasks.workunit.client.0.smithi195.stdout: create 143/.snap/snap-subdir-test 2022-06-27T11:49:13.055 INFO:tasks.workunit.client.0.smithi195.stdout: create 143/30 file after the snapshot 2022-06-27T11:49:13.079 INFO:tasks.workunit.client.0.smithi195.stdout: create 143/31 file after the snapshot 2022-06-27T11:49:13.079 INFO:tasks.workunit.client.0.smithi195.stdout: create 143/32 file after the snapshot 2022-06-27T11:49:13.079 INFO:tasks.workunit.client.0.smithi195.stdout: create 143/33 file after the snapshot 2022-06-27T11:49:13.080 INFO:tasks.workunit.client.0.smithi195.stdout: create 143/34 file after the snapshot 2022-06-27T11:49:13.080 INFO:tasks.workunit.client.0.smithi195.stdout: create 143/35 file after the snapshot 2022-06-27T11:49:13.080 INFO:tasks.workunit.client.0.smithi195.stdout: create 143/36 file after the snapshot 2022-06-27T11:49:13.081 INFO:tasks.workunit.client.0.smithi195.stdout: create 143/37 file after the snapshot 2022-06-27T11:49:13.081 INFO:tasks.workunit.client.0.smithi195.stdout: create 143/38 file after the snapshot 2022-06-27T11:49:13.081 INFO:tasks.workunit.client.0.smithi195.stdout: create 143/39 file after the snapshot 2022-06-27T11:49:13.081 INFO:tasks.workunit.client.0.smithi195.stdout: create 144/.snap/snap-subdir-test 2022-06-27T11:49:13.110 INFO:tasks.workunit.client.0.smithi195.stdout: create 144/30 file after the snapshot 2022-06-27T11:49:13.127 INFO:tasks.workunit.client.0.smithi195.stdout: create 144/31 file after the snapshot 2022-06-27T11:49:13.127 INFO:tasks.workunit.client.0.smithi195.stdout: create 144/32 file after the snapshot 2022-06-27T11:49:13.128 INFO:tasks.workunit.client.0.smithi195.stdout: create 144/33 file after the snapshot 2022-06-27T11:49:13.128 INFO:tasks.workunit.client.0.smithi195.stdout: create 144/34 file after the snapshot 2022-06-27T11:49:13.129 INFO:tasks.workunit.client.0.smithi195.stdout: create 144/35 file after the snapshot 2022-06-27T11:49:13.131 INFO:tasks.workunit.client.0.smithi195.stdout: create 144/36 file after the snapshot 2022-06-27T11:49:13.135 INFO:tasks.workunit.client.0.smithi195.stdout: create 144/37 file after the snapshot 2022-06-27T11:49:13.136 INFO:tasks.workunit.client.0.smithi195.stdout: create 144/38 file after the snapshot 2022-06-27T11:49:13.136 INFO:tasks.workunit.client.0.smithi195.stdout: create 144/39 file after the snapshot 2022-06-27T11:49:13.136 INFO:tasks.workunit.client.0.smithi195.stdout: create 145/.snap/snap-subdir-test 2022-06-27T11:49:13.163 INFO:tasks.workunit.client.0.smithi195.stdout: create 145/30 file after the snapshot 2022-06-27T11:49:13.187 INFO:tasks.workunit.client.0.smithi195.stdout: create 145/31 file after the snapshot 2022-06-27T11:49:13.188 INFO:tasks.workunit.client.0.smithi195.stdout: create 145/32 file after the snapshot 2022-06-27T11:49:13.188 INFO:tasks.workunit.client.0.smithi195.stdout: create 145/33 file after the snapshot 2022-06-27T11:49:13.188 INFO:tasks.workunit.client.0.smithi195.stdout: create 145/34 file after the snapshot 2022-06-27T11:49:13.188 INFO:tasks.workunit.client.0.smithi195.stdout: create 145/35 file after the snapshot 2022-06-27T11:49:13.189 INFO:tasks.workunit.client.0.smithi195.stdout: create 145/36 file after the snapshot 2022-06-27T11:49:13.189 INFO:tasks.workunit.client.0.smithi195.stdout: create 145/37 file after the snapshot 2022-06-27T11:49:13.189 INFO:tasks.workunit.client.0.smithi195.stdout: create 145/38 file after the snapshot 2022-06-27T11:49:13.189 INFO:tasks.workunit.client.0.smithi195.stdout: create 145/39 file after the snapshot 2022-06-27T11:49:13.200 INFO:tasks.workunit.client.0.smithi195.stdout: create 146/.snap/snap-subdir-test 2022-06-27T11:49:13.217 INFO:tasks.workunit.client.0.smithi195.stdout: create 146/30 file after the snapshot 2022-06-27T11:49:13.231 INFO:tasks.workunit.client.0.smithi195.stdout: create 146/31 file after the snapshot 2022-06-27T11:49:13.232 INFO:tasks.workunit.client.0.smithi195.stdout: create 146/32 file after the snapshot 2022-06-27T11:49:13.233 INFO:tasks.workunit.client.0.smithi195.stdout: create 146/33 file after the snapshot 2022-06-27T11:49:13.245 INFO:tasks.workunit.client.0.smithi195.stdout: create 146/34 file after the snapshot 2022-06-27T11:49:13.246 INFO:tasks.workunit.client.0.smithi195.stdout: create 146/35 file after the snapshot 2022-06-27T11:49:13.247 INFO:tasks.workunit.client.0.smithi195.stdout: create 146/36 file after the snapshot 2022-06-27T11:49:13.247 INFO:tasks.workunit.client.0.smithi195.stdout: create 146/37 file after the snapshot 2022-06-27T11:49:13.247 INFO:tasks.workunit.client.0.smithi195.stdout: create 146/38 file after the snapshot 2022-06-27T11:49:13.248 INFO:tasks.workunit.client.0.smithi195.stdout: create 146/39 file after the snapshot 2022-06-27T11:49:13.248 INFO:tasks.workunit.client.0.smithi195.stdout: create 147/.snap/snap-subdir-test 2022-06-27T11:49:13.271 INFO:tasks.workunit.client.0.smithi195.stdout: create 147/30 file after the snapshot 2022-06-27T11:49:13.287 INFO:tasks.workunit.client.0.smithi195.stdout: create 147/31 file after the snapshot 2022-06-27T11:49:13.288 INFO:tasks.workunit.client.0.smithi195.stdout: create 147/32 file after the snapshot 2022-06-27T11:49:13.288 INFO:tasks.workunit.client.0.smithi195.stdout: create 147/33 file after the snapshot 2022-06-27T11:49:13.288 INFO:tasks.workunit.client.0.smithi195.stdout: create 147/34 file after the snapshot 2022-06-27T11:49:13.289 INFO:tasks.workunit.client.0.smithi195.stdout: create 147/35 file after the snapshot 2022-06-27T11:49:13.290 INFO:tasks.workunit.client.0.smithi195.stdout: create 147/36 file after the snapshot 2022-06-27T11:49:13.290 INFO:tasks.workunit.client.0.smithi195.stdout: create 147/37 file after the snapshot 2022-06-27T11:49:13.291 INFO:tasks.workunit.client.0.smithi195.stdout: create 147/38 file after the snapshot 2022-06-27T11:49:13.292 INFO:tasks.workunit.client.0.smithi195.stdout: create 147/39 file after the snapshot 2022-06-27T11:49:13.292 INFO:tasks.workunit.client.0.smithi195.stdout: create 148/.snap/snap-subdir-test 2022-06-27T11:49:13.322 INFO:tasks.workunit.client.0.smithi195.stdout: create 148/30 file after the snapshot 2022-06-27T11:49:13.341 INFO:tasks.workunit.client.0.smithi195.stdout: create 148/31 file after the snapshot 2022-06-27T11:49:13.342 INFO:tasks.workunit.client.0.smithi195.stdout: create 148/32 file after the snapshot 2022-06-27T11:49:13.344 INFO:tasks.workunit.client.0.smithi195.stdout: create 148/33 file after the snapshot 2022-06-27T11:49:13.345 INFO:tasks.workunit.client.0.smithi195.stdout: create 148/34 file after the snapshot 2022-06-27T11:49:13.345 INFO:tasks.workunit.client.0.smithi195.stdout: create 148/35 file after the snapshot 2022-06-27T11:49:13.346 INFO:tasks.workunit.client.0.smithi195.stdout: create 148/36 file after the snapshot 2022-06-27T11:49:13.347 INFO:tasks.workunit.client.0.smithi195.stdout: create 148/37 file after the snapshot 2022-06-27T11:49:13.348 INFO:tasks.workunit.client.0.smithi195.stdout: create 148/38 file after the snapshot 2022-06-27T11:49:13.348 INFO:tasks.workunit.client.0.smithi195.stdout: create 148/39 file after the snapshot 2022-06-27T11:49:13.350 INFO:tasks.workunit.client.0.smithi195.stdout: create 149/.snap/snap-subdir-test 2022-06-27T11:49:13.377 INFO:tasks.workunit.client.0.smithi195.stdout: create 149/30 file after the snapshot 2022-06-27T11:49:13.389 INFO:tasks.workunit.client.0.smithi195.stdout: create 149/31 file after the snapshot 2022-06-27T11:49:13.390 INFO:tasks.workunit.client.0.smithi195.stdout: create 149/32 file after the snapshot 2022-06-27T11:49:13.391 INFO:tasks.workunit.client.0.smithi195.stdout: create 149/33 file after the snapshot 2022-06-27T11:49:13.395 INFO:tasks.workunit.client.0.smithi195.stdout: create 149/34 file after the snapshot 2022-06-27T11:49:13.395 INFO:tasks.workunit.client.0.smithi195.stdout: create 149/35 file after the snapshot 2022-06-27T11:49:13.395 INFO:tasks.workunit.client.0.smithi195.stdout: create 149/36 file after the snapshot 2022-06-27T11:49:13.396 INFO:tasks.workunit.client.0.smithi195.stdout: create 149/37 file after the snapshot 2022-06-27T11:49:13.396 INFO:tasks.workunit.client.0.smithi195.stdout: create 149/38 file after the snapshot 2022-06-27T11:49:13.397 INFO:tasks.workunit.client.0.smithi195.stdout: create 149/39 file after the snapshot 2022-06-27T11:49:13.399 INFO:tasks.workunit.client.0.smithi195.stdout: create 150/.snap/snap-subdir-test 2022-06-27T11:49:13.425 INFO:tasks.workunit.client.0.smithi195.stdout: create 150/30 file after the snapshot 2022-06-27T11:49:13.439 INFO:tasks.workunit.client.0.smithi195.stdout: create 150/31 file after the snapshot 2022-06-27T11:49:13.440 INFO:tasks.workunit.client.0.smithi195.stdout: create 150/32 file after the snapshot 2022-06-27T11:49:13.441 INFO:tasks.workunit.client.0.smithi195.stdout: create 150/33 file after the snapshot 2022-06-27T11:49:13.442 INFO:tasks.workunit.client.0.smithi195.stdout: create 150/34 file after the snapshot 2022-06-27T11:49:13.443 INFO:tasks.workunit.client.0.smithi195.stdout: create 150/35 file after the snapshot 2022-06-27T11:49:13.443 INFO:tasks.workunit.client.0.smithi195.stdout: create 150/36 file after the snapshot 2022-06-27T11:49:13.444 INFO:tasks.workunit.client.0.smithi195.stdout: create 150/37 file after the snapshot 2022-06-27T11:49:13.445 INFO:tasks.workunit.client.0.smithi195.stdout: create 150/38 file after the snapshot 2022-06-27T11:49:13.446 INFO:tasks.workunit.client.0.smithi195.stdout: create 150/39 file after the snapshot 2022-06-27T11:49:13.449 INFO:tasks.workunit.client.0.smithi195.stdout: create 151/.snap/snap-subdir-test 2022-06-27T11:49:13.486 INFO:tasks.workunit.client.0.smithi195.stdout: create 151/30 file after the snapshot 2022-06-27T11:49:13.491 INFO:tasks.workunit.client.0.smithi195.stdout: create 151/31 file after the snapshot 2022-06-27T11:49:13.492 INFO:tasks.workunit.client.0.smithi195.stdout: create 151/32 file after the snapshot 2022-06-27T11:49:13.493 INFO:tasks.workunit.client.0.smithi195.stdout: create 151/33 file after the snapshot 2022-06-27T11:49:13.495 INFO:tasks.workunit.client.0.smithi195.stdout: create 151/34 file after the snapshot 2022-06-27T11:49:13.495 INFO:tasks.workunit.client.0.smithi195.stdout: create 151/35 file after the snapshot 2022-06-27T11:49:13.496 INFO:tasks.workunit.client.0.smithi195.stdout: create 151/36 file after the snapshot 2022-06-27T11:49:13.497 INFO:tasks.workunit.client.0.smithi195.stdout: create 151/37 file after the snapshot 2022-06-27T11:49:13.498 INFO:tasks.workunit.client.0.smithi195.stdout: create 151/38 file after the snapshot 2022-06-27T11:49:13.499 INFO:tasks.workunit.client.0.smithi195.stdout: create 151/39 file after the snapshot 2022-06-27T11:49:13.499 INFO:tasks.workunit.client.0.smithi195.stdout: create 152/.snap/snap-subdir-test 2022-06-27T11:49:13.528 INFO:tasks.workunit.client.0.smithi195.stdout: create 152/30 file after the snapshot 2022-06-27T11:49:13.543 INFO:tasks.workunit.client.0.smithi195.stdout: create 152/31 file after the snapshot 2022-06-27T11:49:13.544 INFO:tasks.workunit.client.0.smithi195.stdout: create 152/32 file after the snapshot 2022-06-27T11:49:13.545 INFO:tasks.workunit.client.0.smithi195.stdout: create 152/33 file after the snapshot 2022-06-27T11:49:13.546 INFO:tasks.workunit.client.0.smithi195.stdout: create 152/34 file after the snapshot 2022-06-27T11:49:13.547 INFO:tasks.workunit.client.0.smithi195.stdout: create 152/35 file after the snapshot 2022-06-27T11:49:13.548 INFO:tasks.workunit.client.0.smithi195.stdout: create 152/36 file after the snapshot 2022-06-27T11:49:13.548 INFO:tasks.workunit.client.0.smithi195.stdout: create 152/37 file after the snapshot 2022-06-27T11:49:13.549 INFO:tasks.workunit.client.0.smithi195.stdout: create 152/38 file after the snapshot 2022-06-27T11:49:13.551 INFO:tasks.workunit.client.0.smithi195.stdout: create 152/39 file after the snapshot 2022-06-27T11:49:13.551 INFO:tasks.workunit.client.0.smithi195.stdout: create 153/.snap/snap-subdir-test 2022-06-27T11:49:13.580 INFO:tasks.workunit.client.0.smithi195.stdout: create 153/30 file after the snapshot 2022-06-27T11:49:13.877 INFO:tasks.workunit.client.0.smithi195.stdout: create 153/31 file after the snapshot 2022-06-27T11:49:13.877 INFO:tasks.workunit.client.0.smithi195.stdout: create 153/32 file after the snapshot 2022-06-27T11:49:13.878 INFO:tasks.workunit.client.0.smithi195.stdout: create 153/33 file after the snapshot 2022-06-27T11:49:13.878 INFO:tasks.workunit.client.0.smithi195.stdout: create 153/34 file after the snapshot 2022-06-27T11:49:13.879 INFO:tasks.workunit.client.0.smithi195.stdout: create 153/35 file after the snapshot 2022-06-27T11:49:13.883 INFO:tasks.workunit.client.0.smithi195.stdout: create 153/36 file after the snapshot 2022-06-27T11:49:13.883 INFO:tasks.workunit.client.0.smithi195.stdout: create 153/37 file after the snapshot 2022-06-27T11:49:13.884 INFO:tasks.workunit.client.0.smithi195.stdout: create 153/38 file after the snapshot 2022-06-27T11:49:13.884 INFO:tasks.workunit.client.0.smithi195.stdout: create 153/39 file after the snapshot 2022-06-27T11:49:13.884 INFO:tasks.workunit.client.0.smithi195.stdout: create 154/.snap/snap-subdir-test 2022-06-27T11:49:13.916 INFO:tasks.workunit.client.0.smithi195.stdout: create 154/30 file after the snapshot 2022-06-27T11:49:13.925 INFO:tasks.workunit.client.0.smithi195.stdout: create 154/31 file after the snapshot 2022-06-27T11:49:13.925 INFO:tasks.workunit.client.0.smithi195.stdout: create 154/32 file after the snapshot 2022-06-27T11:49:13.926 INFO:tasks.workunit.client.0.smithi195.stdout: create 154/33 file after the snapshot 2022-06-27T11:49:13.927 INFO:tasks.workunit.client.0.smithi195.stdout: create 154/34 file after the snapshot 2022-06-27T11:49:13.928 INFO:tasks.workunit.client.0.smithi195.stdout: create 154/35 file after the snapshot 2022-06-27T11:49:13.929 INFO:tasks.workunit.client.0.smithi195.stdout: create 154/36 file after the snapshot 2022-06-27T11:49:13.930 INFO:tasks.workunit.client.0.smithi195.stdout: create 154/37 file after the snapshot 2022-06-27T11:49:13.931 INFO:tasks.workunit.client.0.smithi195.stdout: create 154/38 file after the snapshot 2022-06-27T11:49:13.932 INFO:tasks.workunit.client.0.smithi195.stdout: create 154/39 file after the snapshot 2022-06-27T11:49:13.932 INFO:tasks.workunit.client.0.smithi195.stdout: create 155/.snap/snap-subdir-test 2022-06-27T11:49:13.963 INFO:tasks.workunit.client.0.smithi195.stdout: create 155/30 file after the snapshot 2022-06-27T11:49:13.979 INFO:tasks.workunit.client.0.smithi195.stdout: create 155/31 file after the snapshot 2022-06-27T11:49:13.982 INFO:tasks.workunit.client.0.smithi195.stdout: create 155/32 file after the snapshot 2022-06-27T11:49:13.982 INFO:tasks.workunit.client.0.smithi195.stdout: create 155/33 file after the snapshot 2022-06-27T11:49:13.983 INFO:tasks.workunit.client.0.smithi195.stdout: create 155/34 file after the snapshot 2022-06-27T11:49:13.983 INFO:tasks.workunit.client.0.smithi195.stdout: create 155/35 file after the snapshot 2022-06-27T11:49:13.985 INFO:tasks.workunit.client.0.smithi195.stdout: create 155/36 file after the snapshot 2022-06-27T11:49:13.985 INFO:tasks.workunit.client.0.smithi195.stdout: create 155/37 file after the snapshot 2022-06-27T11:49:13.986 INFO:tasks.workunit.client.0.smithi195.stdout: create 155/38 file after the snapshot 2022-06-27T11:49:13.986 INFO:tasks.workunit.client.0.smithi195.stdout: create 155/39 file after the snapshot 2022-06-27T11:49:13.987 INFO:tasks.workunit.client.0.smithi195.stdout: create 156/.snap/snap-subdir-test 2022-06-27T11:49:14.023 INFO:tasks.workunit.client.0.smithi195.stdout: create 156/30 file after the snapshot 2022-06-27T11:49:14.057 INFO:tasks.workunit.client.0.smithi195.stdout: create 156/31 file after the snapshot 2022-06-27T11:49:14.057 INFO:tasks.workunit.client.0.smithi195.stdout: create 156/32 file after the snapshot 2022-06-27T11:49:14.058 INFO:tasks.workunit.client.0.smithi195.stdout: create 156/33 file after the snapshot 2022-06-27T11:49:14.058 INFO:tasks.workunit.client.0.smithi195.stdout: create 156/34 file after the snapshot 2022-06-27T11:49:14.059 INFO:tasks.workunit.client.0.smithi195.stdout: create 156/35 file after the snapshot 2022-06-27T11:49:14.059 INFO:tasks.workunit.client.0.smithi195.stdout: create 156/36 file after the snapshot 2022-06-27T11:49:14.059 INFO:tasks.workunit.client.0.smithi195.stdout: create 156/37 file after the snapshot 2022-06-27T11:49:14.060 INFO:tasks.workunit.client.0.smithi195.stdout: create 156/38 file after the snapshot 2022-06-27T11:49:14.060 INFO:tasks.workunit.client.0.smithi195.stdout: create 156/39 file after the snapshot 2022-06-27T11:49:14.060 INFO:tasks.workunit.client.0.smithi195.stdout: create 157/.snap/snap-subdir-test 2022-06-27T11:49:14.098 INFO:tasks.workunit.client.0.smithi195.stdout: create 157/30 file after the snapshot 2022-06-27T11:49:14.115 INFO:tasks.workunit.client.0.smithi195.stdout: create 157/31 file after the snapshot 2022-06-27T11:49:14.116 INFO:tasks.workunit.client.0.smithi195.stdout: create 157/32 file after the snapshot 2022-06-27T11:49:14.116 INFO:tasks.workunit.client.0.smithi195.stdout: create 157/33 file after the snapshot 2022-06-27T11:49:14.116 INFO:tasks.workunit.client.0.smithi195.stdout: create 157/34 file after the snapshot 2022-06-27T11:49:14.117 INFO:tasks.workunit.client.0.smithi195.stdout: create 157/35 file after the snapshot 2022-06-27T11:49:14.117 INFO:tasks.workunit.client.0.smithi195.stdout: create 157/36 file after the snapshot 2022-06-27T11:49:14.119 INFO:tasks.workunit.client.0.smithi195.stdout: create 157/37 file after the snapshot 2022-06-27T11:49:14.119 INFO:tasks.workunit.client.0.smithi195.stdout: create 157/38 file after the snapshot 2022-06-27T11:49:14.120 INFO:tasks.workunit.client.0.smithi195.stdout: create 157/39 file after the snapshot 2022-06-27T11:49:14.120 INFO:tasks.workunit.client.0.smithi195.stdout: create 158/.snap/snap-subdir-test 2022-06-27T11:49:14.163 INFO:tasks.workunit.client.0.smithi195.stdout: create 158/30 file after the snapshot 2022-06-27T11:49:14.179 INFO:tasks.workunit.client.0.smithi195.stdout: create 158/31 file after the snapshot 2022-06-27T11:49:14.180 INFO:tasks.workunit.client.0.smithi195.stdout: create 158/32 file after the snapshot 2022-06-27T11:49:14.181 INFO:tasks.workunit.client.0.smithi195.stdout: create 158/33 file after the snapshot 2022-06-27T11:49:14.182 INFO:tasks.workunit.client.0.smithi195.stdout: create 158/34 file after the snapshot 2022-06-27T11:49:14.182 INFO:tasks.workunit.client.0.smithi195.stdout: create 158/35 file after the snapshot 2022-06-27T11:49:14.184 INFO:tasks.workunit.client.0.smithi195.stdout: create 158/36 file after the snapshot 2022-06-27T11:49:14.184 INFO:tasks.workunit.client.0.smithi195.stdout: create 158/37 file after the snapshot 2022-06-27T11:49:14.187 INFO:tasks.workunit.client.0.smithi195.stdout: create 158/38 file after the snapshot 2022-06-27T11:49:14.187 INFO:tasks.workunit.client.0.smithi195.stdout: create 158/39 file after the snapshot 2022-06-27T11:49:14.187 INFO:tasks.workunit.client.0.smithi195.stdout: create 159/.snap/snap-subdir-test 2022-06-27T11:49:14.232 INFO:tasks.workunit.client.0.smithi195.stdout: create 159/30 file after the snapshot 2022-06-27T11:49:14.252 INFO:tasks.workunit.client.0.smithi195.stdout: create 159/31 file after the snapshot 2022-06-27T11:49:14.253 INFO:tasks.workunit.client.0.smithi195.stdout: create 159/32 file after the snapshot 2022-06-27T11:49:14.253 INFO:tasks.workunit.client.0.smithi195.stdout: create 159/33 file after the snapshot 2022-06-27T11:49:14.254 INFO:tasks.workunit.client.0.smithi195.stdout: create 159/34 file after the snapshot 2022-06-27T11:49:14.254 INFO:tasks.workunit.client.0.smithi195.stdout: create 159/35 file after the snapshot 2022-06-27T11:49:14.255 INFO:tasks.workunit.client.0.smithi195.stdout: create 159/36 file after the snapshot 2022-06-27T11:49:14.255 INFO:tasks.workunit.client.0.smithi195.stdout: create 159/37 file after the snapshot 2022-06-27T11:49:14.256 INFO:tasks.workunit.client.0.smithi195.stdout: create 159/38 file after the snapshot 2022-06-27T11:49:14.256 INFO:tasks.workunit.client.0.smithi195.stdout: create 159/39 file after the snapshot 2022-06-27T11:49:14.256 INFO:tasks.workunit.client.0.smithi195.stdout: create 160/.snap/snap-subdir-test 2022-06-27T11:49:14.890 INFO:tasks.workunit.client.0.smithi195.stdout: create 160/30 file after the snapshot 2022-06-27T11:49:14.907 INFO:tasks.workunit.client.0.smithi195.stdout: create 160/31 file after the snapshot 2022-06-27T11:49:14.908 INFO:tasks.workunit.client.0.smithi195.stdout: create 160/32 file after the snapshot 2022-06-27T11:49:14.908 INFO:tasks.workunit.client.0.smithi195.stdout: create 160/33 file after the snapshot 2022-06-27T11:49:14.909 INFO:tasks.workunit.client.0.smithi195.stdout: create 160/34 file after the snapshot 2022-06-27T11:49:14.909 INFO:tasks.workunit.client.0.smithi195.stdout: create 160/35 file after the snapshot 2022-06-27T11:49:14.910 INFO:tasks.workunit.client.0.smithi195.stdout: create 160/36 file after the snapshot 2022-06-27T11:49:14.911 INFO:tasks.workunit.client.0.smithi195.stdout: create 160/37 file after the snapshot 2022-06-27T11:49:14.912 INFO:tasks.workunit.client.0.smithi195.stdout: create 160/38 file after the snapshot 2022-06-27T11:49:14.913 INFO:tasks.workunit.client.0.smithi195.stdout: create 160/39 file after the snapshot 2022-06-27T11:49:14.913 INFO:tasks.workunit.client.0.smithi195.stdout: create 161/.snap/snap-subdir-test 2022-06-27T11:49:14.951 INFO:tasks.workunit.client.0.smithi195.stdout: create 161/30 file after the snapshot 2022-06-27T11:49:14.966 INFO:tasks.workunit.client.0.smithi195.stdout: create 161/31 file after the snapshot 2022-06-27T11:49:14.968 INFO:tasks.workunit.client.0.smithi195.stdout: create 161/32 file after the snapshot 2022-06-27T11:49:14.968 INFO:tasks.workunit.client.0.smithi195.stdout: create 161/33 file after the snapshot 2022-06-27T11:49:14.969 INFO:tasks.workunit.client.0.smithi195.stdout: create 161/34 file after the snapshot 2022-06-27T11:49:14.970 INFO:tasks.workunit.client.0.smithi195.stdout: create 161/35 file after the snapshot 2022-06-27T11:49:14.971 INFO:tasks.workunit.client.0.smithi195.stdout: create 161/36 file after the snapshot 2022-06-27T11:49:14.972 INFO:tasks.workunit.client.0.smithi195.stdout: create 161/37 file after the snapshot 2022-06-27T11:49:15.127 INFO:tasks.workunit.client.0.smithi195.stdout: create 161/38 file after the snapshot 2022-06-27T11:49:15.858 INFO:tasks.workunit.client.0.smithi195.stdout: create 161/39 file after the snapshot 2022-06-27T11:49:15.858 INFO:tasks.workunit.client.0.smithi195.stdout: create 162/.snap/snap-subdir-test 2022-06-27T11:49:15.906 INFO:tasks.workunit.client.0.smithi195.stdout: create 162/30 file after the snapshot 2022-06-27T11:49:15.915 INFO:tasks.workunit.client.0.smithi195.stdout: create 162/31 file after the snapshot 2022-06-27T11:49:15.916 INFO:tasks.workunit.client.0.smithi195.stdout: create 162/32 file after the snapshot 2022-06-27T11:49:15.916 INFO:tasks.workunit.client.0.smithi195.stdout: create 162/33 file after the snapshot 2022-06-27T11:49:15.917 INFO:tasks.workunit.client.0.smithi195.stdout: create 162/34 file after the snapshot 2022-06-27T11:49:15.917 INFO:tasks.workunit.client.0.smithi195.stdout: create 162/35 file after the snapshot 2022-06-27T11:49:15.917 INFO:tasks.workunit.client.0.smithi195.stdout: create 162/36 file after the snapshot 2022-06-27T11:49:15.921 INFO:tasks.workunit.client.0.smithi195.stdout: create 162/37 file after the snapshot 2022-06-27T11:49:15.922 INFO:tasks.workunit.client.0.smithi195.stdout: create 162/38 file after the snapshot 2022-06-27T11:49:15.922 INFO:tasks.workunit.client.0.smithi195.stdout: create 162/39 file after the snapshot 2022-06-27T11:49:15.923 INFO:tasks.workunit.client.0.smithi195.stdout: create 163/.snap/snap-subdir-test 2022-06-27T11:49:15.981 INFO:tasks.workunit.client.0.smithi195.stdout: create 163/30 file after the snapshot 2022-06-27T11:49:15.981 INFO:tasks.workunit.client.0.smithi195.stdout: create 163/31 file after the snapshot 2022-06-27T11:49:15.982 INFO:tasks.workunit.client.0.smithi195.stdout: create 163/32 file after the snapshot 2022-06-27T11:49:15.982 INFO:tasks.workunit.client.0.smithi195.stdout: create 163/33 file after the snapshot 2022-06-27T11:49:15.983 INFO:tasks.workunit.client.0.smithi195.stdout: create 163/34 file after the snapshot 2022-06-27T11:49:15.984 INFO:tasks.workunit.client.0.smithi195.stdout: create 163/35 file after the snapshot 2022-06-27T11:49:15.984 INFO:tasks.workunit.client.0.smithi195.stdout: create 163/36 file after the snapshot 2022-06-27T11:49:15.985 INFO:tasks.workunit.client.0.smithi195.stdout: create 163/37 file after the snapshot 2022-06-27T11:49:16.078 INFO:tasks.workunit.client.0.smithi195.stdout: create 163/38 file after the snapshot 2022-06-27T11:49:16.981 INFO:tasks.workunit.client.0.smithi195.stdout: create 163/39 file after the snapshot 2022-06-27T11:49:16.982 INFO:tasks.workunit.client.0.smithi195.stdout: create 164/.snap/snap-subdir-test 2022-06-27T11:49:17.022 INFO:tasks.workunit.client.0.smithi195.stdout: create 164/30 file after the snapshot 2022-06-27T11:49:17.045 INFO:tasks.workunit.client.0.smithi195.stdout: create 164/31 file after the snapshot 2022-06-27T11:49:17.046 INFO:tasks.workunit.client.0.smithi195.stdout: create 164/32 file after the snapshot 2022-06-27T11:49:17.046 INFO:tasks.workunit.client.0.smithi195.stdout: create 164/33 file after the snapshot 2022-06-27T11:49:17.046 INFO:tasks.workunit.client.0.smithi195.stdout: create 164/34 file after the snapshot 2022-06-27T11:49:17.047 INFO:tasks.workunit.client.0.smithi195.stdout: create 164/35 file after the snapshot 2022-06-27T11:49:17.047 INFO:tasks.workunit.client.0.smithi195.stdout: create 164/36 file after the snapshot 2022-06-27T11:49:17.047 INFO:tasks.workunit.client.0.smithi195.stdout: create 164/37 file after the snapshot 2022-06-27T11:49:17.047 INFO:tasks.workunit.client.0.smithi195.stdout: create 164/38 file after the snapshot 2022-06-27T11:49:17.048 INFO:tasks.workunit.client.0.smithi195.stdout: create 164/39 file after the snapshot 2022-06-27T11:49:17.048 INFO:tasks.workunit.client.0.smithi195.stdout: create 165/.snap/snap-subdir-test 2022-06-27T11:49:17.150 INFO:tasks.workunit.client.0.smithi195.stdout: create 165/30 file after the snapshot 2022-06-27T11:49:17.176 INFO:tasks.workunit.client.0.smithi195.stdout: create 165/31 file after the snapshot 2022-06-27T11:49:17.176 INFO:tasks.workunit.client.0.smithi195.stdout: create 165/32 file after the snapshot 2022-06-27T11:49:17.177 INFO:tasks.workunit.client.0.smithi195.stdout: create 165/33 file after the snapshot 2022-06-27T11:49:17.177 INFO:tasks.workunit.client.0.smithi195.stdout: create 165/34 file after the snapshot 2022-06-27T11:49:17.177 INFO:tasks.workunit.client.0.smithi195.stdout: create 165/35 file after the snapshot 2022-06-27T11:49:17.177 INFO:tasks.workunit.client.0.smithi195.stdout: create 165/36 file after the snapshot 2022-06-27T11:49:17.177 INFO:tasks.workunit.client.0.smithi195.stdout: create 165/37 file after the snapshot 2022-06-27T11:49:17.178 INFO:tasks.workunit.client.0.smithi195.stdout: create 165/38 file after the snapshot 2022-06-27T11:49:17.178 INFO:tasks.workunit.client.0.smithi195.stdout: create 165/39 file after the snapshot 2022-06-27T11:49:17.178 INFO:tasks.workunit.client.0.smithi195.stdout: create 166/.snap/snap-subdir-test 2022-06-27T11:49:17.217 INFO:tasks.workunit.client.0.smithi195.stdout: create 166/30 file after the snapshot 2022-06-27T11:49:17.218 INFO:tasks.workunit.client.0.smithi195.stdout: create 166/31 file after the snapshot 2022-06-27T11:49:17.218 INFO:tasks.workunit.client.0.smithi195.stdout: create 166/32 file after the snapshot 2022-06-27T11:49:17.223 INFO:tasks.workunit.client.0.smithi195.stdout: create 166/33 file after the snapshot 2022-06-27T11:49:17.224 INFO:tasks.workunit.client.0.smithi195.stdout: create 166/34 file after the snapshot 2022-06-27T11:49:17.224 INFO:tasks.workunit.client.0.smithi195.stdout: create 166/35 file after the snapshot 2022-06-27T11:49:17.225 INFO:tasks.workunit.client.0.smithi195.stdout: create 166/36 file after the snapshot 2022-06-27T11:49:17.227 INFO:tasks.workunit.client.0.smithi195.stdout: create 166/37 file after the snapshot 2022-06-27T11:49:17.227 INFO:tasks.workunit.client.0.smithi195.stdout: create 166/38 file after the snapshot 2022-06-27T11:49:17.228 INFO:tasks.workunit.client.0.smithi195.stdout: create 166/39 file after the snapshot 2022-06-27T11:49:17.229 INFO:tasks.workunit.client.0.smithi195.stdout: create 167/.snap/snap-subdir-test 2022-06-27T11:49:17.268 INFO:tasks.workunit.client.0.smithi195.stdout: create 167/30 file after the snapshot 2022-06-27T11:49:17.281 INFO:tasks.workunit.client.0.smithi195.stdout: create 167/31 file after the snapshot 2022-06-27T11:49:17.288 INFO:tasks.workunit.client.0.smithi195.stdout: create 167/32 file after the snapshot 2022-06-27T11:49:17.289 INFO:tasks.workunit.client.0.smithi195.stdout: create 167/33 file after the snapshot 2022-06-27T11:49:17.298 INFO:tasks.workunit.client.0.smithi195.stdout: create 167/34 file after the snapshot 2022-06-27T11:49:17.298 INFO:tasks.workunit.client.0.smithi195.stdout: create 167/35 file after the snapshot 2022-06-27T11:49:17.299 INFO:tasks.workunit.client.0.smithi195.stdout: create 167/36 file after the snapshot 2022-06-27T11:49:17.299 INFO:tasks.workunit.client.0.smithi195.stdout: create 167/37 file after the snapshot 2022-06-27T11:49:17.299 INFO:tasks.workunit.client.0.smithi195.stdout: create 167/38 file after the snapshot 2022-06-27T11:49:17.299 INFO:tasks.workunit.client.0.smithi195.stdout: create 167/39 file after the snapshot 2022-06-27T11:49:17.300 INFO:tasks.workunit.client.0.smithi195.stdout: create 168/.snap/snap-subdir-test 2022-06-27T11:49:17.326 INFO:tasks.workunit.client.0.smithi195.stdout: create 168/30 file after the snapshot 2022-06-27T11:49:17.349 INFO:tasks.workunit.client.0.smithi195.stdout: create 168/31 file after the snapshot 2022-06-27T11:49:17.350 INFO:tasks.workunit.client.0.smithi195.stdout: create 168/32 file after the snapshot 2022-06-27T11:49:17.350 INFO:tasks.workunit.client.0.smithi195.stdout: create 168/33 file after the snapshot 2022-06-27T11:49:17.351 INFO:tasks.workunit.client.0.smithi195.stdout: create 168/34 file after the snapshot 2022-06-27T11:49:17.351 INFO:tasks.workunit.client.0.smithi195.stdout: create 168/35 file after the snapshot 2022-06-27T11:49:17.352 INFO:tasks.workunit.client.0.smithi195.stdout: create 168/36 file after the snapshot 2022-06-27T11:49:17.352 INFO:tasks.workunit.client.0.smithi195.stdout: create 168/37 file after the snapshot 2022-06-27T11:49:17.361 INFO:tasks.workunit.client.0.smithi195.stdout: create 168/38 file after the snapshot 2022-06-27T11:49:17.361 INFO:tasks.workunit.client.0.smithi195.stdout: create 168/39 file after the snapshot 2022-06-27T11:49:17.362 INFO:tasks.workunit.client.0.smithi195.stdout: create 169/.snap/snap-subdir-test 2022-06-27T11:49:17.381 INFO:tasks.workunit.client.0.smithi195.stdout: create 169/30 file after the snapshot 2022-06-27T11:49:17.403 INFO:tasks.workunit.client.0.smithi195.stdout: create 169/31 file after the snapshot 2022-06-27T11:49:17.404 INFO:tasks.workunit.client.0.smithi195.stdout: create 169/32 file after the snapshot 2022-06-27T11:49:17.404 INFO:tasks.workunit.client.0.smithi195.stdout: create 169/33 file after the snapshot 2022-06-27T11:49:17.405 INFO:tasks.workunit.client.0.smithi195.stdout: create 169/34 file after the snapshot 2022-06-27T11:49:17.405 INFO:tasks.workunit.client.0.smithi195.stdout: create 169/35 file after the snapshot 2022-06-27T11:49:17.406 INFO:tasks.workunit.client.0.smithi195.stdout: create 169/36 file after the snapshot 2022-06-27T11:49:17.406 INFO:tasks.workunit.client.0.smithi195.stdout: create 169/37 file after the snapshot 2022-06-27T11:49:17.407 INFO:tasks.workunit.client.0.smithi195.stdout: create 169/38 file after the snapshot 2022-06-27T11:49:17.407 INFO:tasks.workunit.client.0.smithi195.stdout: create 169/39 file after the snapshot 2022-06-27T11:49:17.409 INFO:tasks.workunit.client.0.smithi195.stdout: create 170/.snap/snap-subdir-test 2022-06-27T11:49:17.431 INFO:tasks.workunit.client.0.smithi195.stdout: create 170/30 file after the snapshot 2022-06-27T11:49:17.449 INFO:tasks.workunit.client.0.smithi195.stdout: create 170/31 file after the snapshot 2022-06-27T11:49:17.450 INFO:tasks.workunit.client.0.smithi195.stdout: create 170/32 file after the snapshot 2022-06-27T11:49:17.450 INFO:tasks.workunit.client.0.smithi195.stdout: create 170/33 file after the snapshot 2022-06-27T11:49:17.450 INFO:tasks.workunit.client.0.smithi195.stdout: create 170/34 file after the snapshot 2022-06-27T11:49:17.452 INFO:tasks.workunit.client.0.smithi195.stdout: create 170/35 file after the snapshot 2022-06-27T11:49:17.453 INFO:tasks.workunit.client.0.smithi195.stdout: create 170/36 file after the snapshot 2022-06-27T11:49:17.453 INFO:tasks.workunit.client.0.smithi195.stdout: create 170/37 file after the snapshot 2022-06-27T11:49:17.453 INFO:tasks.workunit.client.0.smithi195.stdout: create 170/38 file after the snapshot 2022-06-27T11:49:17.454 INFO:tasks.workunit.client.0.smithi195.stdout: create 170/39 file after the snapshot 2022-06-27T11:49:17.455 INFO:tasks.workunit.client.0.smithi195.stdout: create 171/.snap/snap-subdir-test 2022-06-27T11:49:17.482 INFO:tasks.workunit.client.0.smithi195.stdout: create 171/30 file after the snapshot 2022-06-27T11:49:17.513 INFO:tasks.workunit.client.0.smithi195.stdout: create 171/31 file after the snapshot 2022-06-27T11:49:17.513 INFO:tasks.workunit.client.0.smithi195.stdout: create 171/32 file after the snapshot 2022-06-27T11:49:17.513 INFO:tasks.workunit.client.0.smithi195.stdout: create 171/33 file after the snapshot 2022-06-27T11:49:17.514 INFO:tasks.workunit.client.0.smithi195.stdout: create 171/34 file after the snapshot 2022-06-27T11:49:17.515 INFO:tasks.workunit.client.0.smithi195.stdout: create 171/35 file after the snapshot 2022-06-27T11:49:17.515 INFO:tasks.workunit.client.0.smithi195.stdout: create 171/36 file after the snapshot 2022-06-27T11:49:17.516 INFO:tasks.workunit.client.0.smithi195.stdout: create 171/37 file after the snapshot 2022-06-27T11:49:17.516 INFO:tasks.workunit.client.0.smithi195.stdout: create 171/38 file after the snapshot 2022-06-27T11:49:17.517 INFO:tasks.workunit.client.0.smithi195.stdout: create 171/39 file after the snapshot 2022-06-27T11:49:17.517 INFO:tasks.workunit.client.0.smithi195.stdout: create 172/.snap/snap-subdir-test 2022-06-27T11:49:17.537 INFO:tasks.workunit.client.0.smithi195.stdout: create 172/30 file after the snapshot 2022-06-27T11:49:17.553 INFO:tasks.workunit.client.0.smithi195.stdout: create 172/31 file after the snapshot 2022-06-27T11:49:17.554 INFO:tasks.workunit.client.0.smithi195.stdout: create 172/32 file after the snapshot 2022-06-27T11:49:17.554 INFO:tasks.workunit.client.0.smithi195.stdout: create 172/33 file after the snapshot 2022-06-27T11:49:17.555 INFO:tasks.workunit.client.0.smithi195.stdout: create 172/34 file after the snapshot 2022-06-27T11:49:17.556 INFO:tasks.workunit.client.0.smithi195.stdout: create 172/35 file after the snapshot 2022-06-27T11:49:17.557 INFO:tasks.workunit.client.0.smithi195.stdout: create 172/36 file after the snapshot 2022-06-27T11:49:17.558 INFO:tasks.workunit.client.0.smithi195.stdout: create 172/37 file after the snapshot 2022-06-27T11:49:17.559 INFO:tasks.workunit.client.0.smithi195.stdout: create 172/38 file after the snapshot 2022-06-27T11:49:17.559 INFO:tasks.workunit.client.0.smithi195.stdout: create 172/39 file after the snapshot 2022-06-27T11:49:17.562 INFO:tasks.workunit.client.0.smithi195.stdout: create 173/.snap/snap-subdir-test 2022-06-27T11:49:17.587 INFO:tasks.workunit.client.0.smithi195.stdout: create 173/30 file after the snapshot 2022-06-27T11:49:17.605 INFO:tasks.workunit.client.0.smithi195.stdout: create 173/31 file after the snapshot 2022-06-27T11:49:17.606 INFO:tasks.workunit.client.0.smithi195.stdout: create 173/32 file after the snapshot 2022-06-27T11:49:17.615 INFO:tasks.workunit.client.0.smithi195.stdout: create 173/33 file after the snapshot 2022-06-27T11:49:17.616 INFO:tasks.workunit.client.0.smithi195.stdout: create 173/34 file after the snapshot 2022-06-27T11:49:17.616 INFO:tasks.workunit.client.0.smithi195.stdout: create 173/35 file after the snapshot 2022-06-27T11:49:17.616 INFO:tasks.workunit.client.0.smithi195.stdout: create 173/36 file after the snapshot 2022-06-27T11:49:17.616 INFO:tasks.workunit.client.0.smithi195.stdout: create 173/37 file after the snapshot 2022-06-27T11:49:17.618 INFO:tasks.workunit.client.0.smithi195.stdout: create 173/38 file after the snapshot 2022-06-27T11:49:17.618 INFO:tasks.workunit.client.0.smithi195.stdout: create 173/39 file after the snapshot 2022-06-27T11:49:17.618 INFO:tasks.workunit.client.0.smithi195.stdout: create 174/.snap/snap-subdir-test 2022-06-27T11:49:17.638 INFO:tasks.workunit.client.0.smithi195.stdout: create 174/30 file after the snapshot 2022-06-27T11:49:17.653 INFO:tasks.workunit.client.0.smithi195.stdout: create 174/31 file after the snapshot 2022-06-27T11:49:17.654 INFO:tasks.workunit.client.0.smithi195.stdout: create 174/32 file after the snapshot 2022-06-27T11:49:17.656 INFO:tasks.workunit.client.0.smithi195.stdout: create 174/33 file after the snapshot 2022-06-27T11:49:17.656 INFO:tasks.workunit.client.0.smithi195.stdout: create 174/34 file after the snapshot 2022-06-27T11:49:17.657 INFO:tasks.workunit.client.0.smithi195.stdout: create 174/35 file after the snapshot 2022-06-27T11:49:17.658 INFO:tasks.workunit.client.0.smithi195.stdout: create 174/36 file after the snapshot 2022-06-27T11:49:17.659 INFO:tasks.workunit.client.0.smithi195.stdout: create 174/37 file after the snapshot 2022-06-27T11:49:17.660 INFO:tasks.workunit.client.0.smithi195.stdout: create 174/38 file after the snapshot 2022-06-27T11:49:17.660 INFO:tasks.workunit.client.0.smithi195.stdout: create 174/39 file after the snapshot 2022-06-27T11:49:17.661 INFO:tasks.workunit.client.0.smithi195.stdout: create 175/.snap/snap-subdir-test 2022-06-27T11:49:17.689 INFO:tasks.workunit.client.0.smithi195.stdout: create 175/30 file after the snapshot 2022-06-27T11:49:17.704 INFO:tasks.workunit.client.0.smithi195.stdout: create 175/31 file after the snapshot 2022-06-27T11:49:17.705 INFO:tasks.workunit.client.0.smithi195.stdout: create 175/32 file after the snapshot 2022-06-27T11:49:17.707 INFO:tasks.workunit.client.0.smithi195.stdout: create 175/33 file after the snapshot 2022-06-27T11:49:17.707 INFO:tasks.workunit.client.0.smithi195.stdout: create 175/34 file after the snapshot 2022-06-27T11:49:17.708 INFO:tasks.workunit.client.0.smithi195.stdout: create 175/35 file after the snapshot 2022-06-27T11:49:17.709 INFO:tasks.workunit.client.0.smithi195.stdout: create 175/36 file after the snapshot 2022-06-27T11:49:17.709 INFO:tasks.workunit.client.0.smithi195.stdout: create 175/37 file after the snapshot 2022-06-27T11:49:17.711 INFO:tasks.workunit.client.0.smithi195.stdout: create 175/38 file after the snapshot 2022-06-27T11:49:17.711 INFO:tasks.workunit.client.0.smithi195.stdout: create 175/39 file after the snapshot 2022-06-27T11:49:17.711 INFO:tasks.workunit.client.0.smithi195.stdout: create 176/.snap/snap-subdir-test 2022-06-27T11:49:17.739 INFO:tasks.workunit.client.0.smithi195.stdout: create 176/30 file after the snapshot 2022-06-27T11:49:17.760 INFO:tasks.workunit.client.0.smithi195.stdout: create 176/31 file after the snapshot 2022-06-27T11:49:17.761 INFO:tasks.workunit.client.0.smithi195.stdout: create 176/32 file after the snapshot 2022-06-27T11:49:17.761 INFO:tasks.workunit.client.0.smithi195.stdout: create 176/33 file after the snapshot 2022-06-27T11:49:17.767 INFO:tasks.workunit.client.0.smithi195.stdout: create 176/34 file after the snapshot 2022-06-27T11:49:17.767 INFO:tasks.workunit.client.0.smithi195.stdout: create 176/35 file after the snapshot 2022-06-27T11:49:18.062 INFO:tasks.workunit.client.0.smithi195.stdout: create 176/36 file after the snapshot 2022-06-27T11:49:18.522 INFO:tasks.workunit.client.0.smithi195.stdout: create 176/37 file after the snapshot 2022-06-27T11:49:18.522 INFO:tasks.workunit.client.0.smithi195.stdout: create 176/38 file after the snapshot 2022-06-27T11:49:18.523 INFO:tasks.workunit.client.0.smithi195.stdout: create 176/39 file after the snapshot 2022-06-27T11:49:18.523 INFO:tasks.workunit.client.0.smithi195.stdout: create 177/.snap/snap-subdir-test 2022-06-27T11:49:18.555 INFO:tasks.workunit.client.0.smithi195.stdout: create 177/30 file after the snapshot 2022-06-27T11:49:18.570 INFO:tasks.workunit.client.0.smithi195.stdout: create 177/31 file after the snapshot 2022-06-27T11:49:18.571 INFO:tasks.workunit.client.0.smithi195.stdout: create 177/32 file after the snapshot 2022-06-27T11:49:18.572 INFO:tasks.workunit.client.0.smithi195.stdout: create 177/33 file after the snapshot 2022-06-27T11:49:18.573 INFO:tasks.workunit.client.0.smithi195.stdout: create 177/34 file after the snapshot 2022-06-27T11:49:18.574 INFO:tasks.workunit.client.0.smithi195.stdout: create 177/35 file after the snapshot 2022-06-27T11:49:18.575 INFO:tasks.workunit.client.0.smithi195.stdout: create 177/36 file after the snapshot 2022-06-27T11:49:18.576 INFO:tasks.workunit.client.0.smithi195.stdout: create 177/37 file after the snapshot 2022-06-27T11:49:18.578 INFO:tasks.workunit.client.0.smithi195.stdout: create 177/38 file after the snapshot 2022-06-27T11:49:18.579 INFO:tasks.workunit.client.0.smithi195.stdout: create 177/39 file after the snapshot 2022-06-27T11:49:18.580 INFO:tasks.workunit.client.0.smithi195.stdout: create 178/.snap/snap-subdir-test 2022-06-27T11:49:19.300 INFO:tasks.workunit.client.0.smithi195.stdout: create 178/30 file after the snapshot 2022-06-27T11:49:19.317 INFO:tasks.workunit.client.0.smithi195.stdout: create 178/31 file after the snapshot 2022-06-27T11:49:19.317 INFO:tasks.workunit.client.0.smithi195.stdout: create 178/32 file after the snapshot 2022-06-27T11:49:19.318 INFO:tasks.workunit.client.0.smithi195.stdout: create 178/33 file after the snapshot 2022-06-27T11:49:19.318 INFO:tasks.workunit.client.0.smithi195.stdout: create 178/34 file after the snapshot 2022-06-27T11:49:19.319 INFO:tasks.workunit.client.0.smithi195.stdout: create 178/35 file after the snapshot 2022-06-27T11:49:19.321 INFO:tasks.workunit.client.0.smithi195.stdout: create 178/36 file after the snapshot 2022-06-27T11:49:19.322 INFO:tasks.workunit.client.0.smithi195.stdout: create 178/37 file after the snapshot 2022-06-27T11:49:19.323 INFO:tasks.workunit.client.0.smithi195.stdout: create 178/38 file after the snapshot 2022-06-27T11:49:19.326 INFO:tasks.workunit.client.0.smithi195.stdout: create 178/39 file after the snapshot 2022-06-27T11:49:19.327 INFO:tasks.workunit.client.0.smithi195.stdout: create 179/.snap/snap-subdir-test 2022-06-27T11:49:19.372 INFO:tasks.workunit.client.0.smithi195.stdout: create 179/30 file after the snapshot 2022-06-27T11:49:19.383 INFO:tasks.workunit.client.0.smithi195.stdout: create 179/31 file after the snapshot 2022-06-27T11:49:19.384 INFO:tasks.workunit.client.0.smithi195.stdout: create 179/32 file after the snapshot 2022-06-27T11:49:19.385 INFO:tasks.workunit.client.0.smithi195.stdout: create 179/33 file after the snapshot 2022-06-27T11:49:19.385 INFO:tasks.workunit.client.0.smithi195.stdout: create 179/34 file after the snapshot 2022-06-27T11:49:19.387 INFO:tasks.workunit.client.0.smithi195.stdout: create 179/35 file after the snapshot 2022-06-27T11:49:19.388 INFO:tasks.workunit.client.0.smithi195.stdout: create 179/36 file after the snapshot 2022-06-27T11:49:19.389 INFO:tasks.workunit.client.0.smithi195.stdout: create 179/37 file after the snapshot 2022-06-27T11:49:19.390 INFO:tasks.workunit.client.0.smithi195.stdout: create 179/38 file after the snapshot 2022-06-27T11:49:19.391 INFO:tasks.workunit.client.0.smithi195.stdout: create 179/39 file after the snapshot 2022-06-27T11:49:19.392 INFO:tasks.workunit.client.0.smithi195.stdout: create 180/.snap/snap-subdir-test 2022-06-27T11:49:19.430 INFO:tasks.workunit.client.0.smithi195.stdout: create 180/30 file after the snapshot 2022-06-27T11:49:19.449 INFO:tasks.workunit.client.0.smithi195.stdout: create 180/31 file after the snapshot 2022-06-27T11:49:19.450 INFO:tasks.workunit.client.0.smithi195.stdout: create 180/32 file after the snapshot 2022-06-27T11:49:19.450 INFO:tasks.workunit.client.0.smithi195.stdout: create 180/33 file after the snapshot 2022-06-27T11:49:19.450 INFO:tasks.workunit.client.0.smithi195.stdout: create 180/34 file after the snapshot 2022-06-27T11:49:19.450 INFO:tasks.workunit.client.0.smithi195.stdout: create 180/35 file after the snapshot 2022-06-27T11:49:19.451 INFO:tasks.workunit.client.0.smithi195.stdout: create 180/36 file after the snapshot 2022-06-27T11:49:19.452 INFO:tasks.workunit.client.0.smithi195.stdout: create 180/37 file after the snapshot 2022-06-27T11:49:19.452 INFO:tasks.workunit.client.0.smithi195.stdout: create 180/38 file after the snapshot 2022-06-27T11:49:19.453 INFO:tasks.workunit.client.0.smithi195.stdout: create 180/39 file after the snapshot 2022-06-27T11:49:19.454 INFO:tasks.workunit.client.0.smithi195.stdout: create 181/.snap/snap-subdir-test 2022-06-27T11:49:19.574 INFO:tasks.workunit.client.0.smithi195.stdout: create 181/30 file after the snapshot 2022-06-27T11:49:19.589 INFO:tasks.workunit.client.0.smithi195.stdout: create 181/31 file after the snapshot 2022-06-27T11:49:19.590 INFO:tasks.workunit.client.0.smithi195.stdout: create 181/32 file after the snapshot 2022-06-27T11:49:19.591 INFO:tasks.workunit.client.0.smithi195.stdout: create 181/33 file after the snapshot 2022-06-27T11:49:19.592 INFO:tasks.workunit.client.0.smithi195.stdout: create 181/34 file after the snapshot 2022-06-27T11:49:19.593 INFO:tasks.workunit.client.0.smithi195.stdout: create 181/35 file after the snapshot 2022-06-27T11:49:19.595 INFO:tasks.workunit.client.0.smithi195.stdout: create 181/36 file after the snapshot 2022-06-27T11:49:19.596 INFO:tasks.workunit.client.0.smithi195.stdout: create 181/37 file after the snapshot 2022-06-27T11:49:19.596 INFO:tasks.workunit.client.0.smithi195.stdout: create 181/38 file after the snapshot 2022-06-27T11:49:19.597 INFO:tasks.workunit.client.0.smithi195.stdout: create 181/39 file after the snapshot 2022-06-27T11:49:19.598 INFO:tasks.workunit.client.0.smithi195.stdout: create 182/.snap/snap-subdir-test 2022-06-27T11:49:19.636 INFO:tasks.workunit.client.0.smithi195.stdout: create 182/30 file after the snapshot 2022-06-27T11:49:19.651 INFO:tasks.workunit.client.0.smithi195.stdout: create 182/31 file after the snapshot 2022-06-27T11:49:19.653 INFO:tasks.workunit.client.0.smithi195.stdout: create 182/32 file after the snapshot 2022-06-27T11:49:19.655 INFO:tasks.workunit.client.0.smithi195.stdout: create 182/33 file after the snapshot 2022-06-27T11:49:19.657 INFO:tasks.workunit.client.0.smithi195.stdout: create 182/34 file after the snapshot 2022-06-27T11:49:19.657 INFO:tasks.workunit.client.0.smithi195.stdout: create 182/35 file after the snapshot 2022-06-27T11:49:19.658 INFO:tasks.workunit.client.0.smithi195.stdout: create 182/36 file after the snapshot 2022-06-27T11:49:19.659 INFO:tasks.workunit.client.0.smithi195.stdout: create 182/37 file after the snapshot 2022-06-27T11:49:19.659 INFO:tasks.workunit.client.0.smithi195.stdout: create 182/38 file after the snapshot 2022-06-27T11:49:19.660 INFO:tasks.workunit.client.0.smithi195.stdout: create 182/39 file after the snapshot 2022-06-27T11:49:19.661 INFO:tasks.workunit.client.0.smithi195.stdout: create 183/.snap/snap-subdir-test 2022-06-27T11:49:19.696 INFO:tasks.workunit.client.0.smithi195.stdout: create 183/30 file after the snapshot 2022-06-27T11:49:19.712 INFO:tasks.workunit.client.0.smithi195.stdout: create 183/31 file after the snapshot 2022-06-27T11:49:19.716 INFO:tasks.workunit.client.0.smithi195.stdout: create 183/32 file after the snapshot 2022-06-27T11:49:19.719 INFO:tasks.workunit.client.0.smithi195.stdout: create 183/33 file after the snapshot 2022-06-27T11:49:19.719 INFO:tasks.workunit.client.0.smithi195.stdout: create 183/34 file after the snapshot 2022-06-27T11:49:19.720 INFO:tasks.workunit.client.0.smithi195.stdout: create 183/35 file after the snapshot 2022-06-27T11:49:19.720 INFO:tasks.workunit.client.0.smithi195.stdout: create 183/36 file after the snapshot 2022-06-27T11:49:19.720 INFO:tasks.workunit.client.0.smithi195.stdout: create 183/37 file after the snapshot 2022-06-27T11:49:19.720 INFO:tasks.workunit.client.0.smithi195.stdout: create 183/38 file after the snapshot 2022-06-27T11:49:19.721 INFO:tasks.workunit.client.0.smithi195.stdout: create 183/39 file after the snapshot 2022-06-27T11:49:19.721 INFO:tasks.workunit.client.0.smithi195.stdout: create 184/.snap/snap-subdir-test 2022-06-27T11:49:19.755 INFO:tasks.workunit.client.0.smithi195.stdout: create 184/30 file after the snapshot 2022-06-27T11:49:19.770 INFO:tasks.workunit.client.0.smithi195.stdout: create 184/31 file after the snapshot 2022-06-27T11:49:19.772 INFO:tasks.workunit.client.0.smithi195.stdout: create 184/32 file after the snapshot 2022-06-27T11:49:19.776 INFO:tasks.workunit.client.0.smithi195.stdout: create 184/33 file after the snapshot 2022-06-27T11:49:19.777 INFO:tasks.workunit.client.0.smithi195.stdout: create 184/34 file after the snapshot 2022-06-27T11:49:19.777 INFO:tasks.workunit.client.0.smithi195.stdout: create 184/35 file after the snapshot 2022-06-27T11:49:19.777 INFO:tasks.workunit.client.0.smithi195.stdout: create 184/36 file after the snapshot 2022-06-27T11:49:19.778 INFO:tasks.workunit.client.0.smithi195.stdout: create 184/37 file after the snapshot 2022-06-27T11:49:19.870 INFO:tasks.workunit.client.0.smithi195.stdout: create 184/38 file after the snapshot 2022-06-27T11:49:19.876 INFO:tasks.workunit.client.0.smithi195.stdout: create 184/39 file after the snapshot 2022-06-27T11:49:19.876 INFO:tasks.workunit.client.0.smithi195.stdout: create 185/.snap/snap-subdir-test 2022-06-27T11:49:19.914 INFO:tasks.workunit.client.0.smithi195.stdout: create 185/30 file after the snapshot 2022-06-27T11:49:19.915 INFO:tasks.workunit.client.0.smithi195.stdout: create 185/31 file after the snapshot 2022-06-27T11:49:19.915 INFO:tasks.workunit.client.0.smithi195.stdout: create 185/32 file after the snapshot 2022-06-27T11:49:19.918 INFO:tasks.workunit.client.0.smithi195.stdout: create 185/33 file after the snapshot 2022-06-27T11:49:19.919 INFO:tasks.workunit.client.0.smithi195.stdout: create 185/34 file after the snapshot 2022-06-27T11:49:19.920 INFO:tasks.workunit.client.0.smithi195.stdout: create 185/35 file after the snapshot 2022-06-27T11:49:19.920 INFO:tasks.workunit.client.0.smithi195.stdout: create 185/36 file after the snapshot 2022-06-27T11:49:19.921 INFO:tasks.workunit.client.0.smithi195.stdout: create 185/37 file after the snapshot 2022-06-27T11:49:19.931 INFO:tasks.workunit.client.0.smithi195.stdout: create 185/38 file after the snapshot 2022-06-27T11:49:19.931 INFO:tasks.workunit.client.0.smithi195.stdout: create 185/39 file after the snapshot 2022-06-27T11:49:19.932 INFO:tasks.workunit.client.0.smithi195.stdout: create 186/.snap/snap-subdir-test 2022-06-27T11:49:19.959 INFO:tasks.workunit.client.0.smithi195.stdout: create 186/30 file after the snapshot 2022-06-27T11:49:19.974 INFO:tasks.workunit.client.0.smithi195.stdout: create 186/31 file after the snapshot 2022-06-27T11:49:19.976 INFO:tasks.workunit.client.0.smithi195.stdout: create 186/32 file after the snapshot 2022-06-27T11:49:19.977 INFO:tasks.workunit.client.0.smithi195.stdout: create 186/33 file after the snapshot 2022-06-27T11:49:19.979 INFO:tasks.workunit.client.0.smithi195.stdout: create 186/34 file after the snapshot 2022-06-27T11:49:19.979 INFO:tasks.workunit.client.0.smithi195.stdout: create 186/35 file after the snapshot 2022-06-27T11:49:19.980 INFO:tasks.workunit.client.0.smithi195.stdout: create 186/36 file after the snapshot 2022-06-27T11:49:19.980 INFO:tasks.workunit.client.0.smithi195.stdout: create 186/37 file after the snapshot 2022-06-27T11:49:19.983 INFO:tasks.workunit.client.0.smithi195.stdout: create 186/38 file after the snapshot 2022-06-27T11:49:19.983 INFO:tasks.workunit.client.0.smithi195.stdout: create 186/39 file after the snapshot 2022-06-27T11:49:19.984 INFO:tasks.workunit.client.0.smithi195.stdout: create 187/.snap/snap-subdir-test 2022-06-27T11:49:20.025 INFO:tasks.workunit.client.0.smithi195.stdout: create 187/30 file after the snapshot 2022-06-27T11:49:20.033 INFO:tasks.workunit.client.0.smithi195.stdout: create 187/31 file after the snapshot 2022-06-27T11:49:20.034 INFO:tasks.workunit.client.0.smithi195.stdout: create 187/32 file after the snapshot 2022-06-27T11:49:20.035 INFO:tasks.workunit.client.0.smithi195.stdout: create 187/33 file after the snapshot 2022-06-27T11:49:20.036 INFO:tasks.workunit.client.0.smithi195.stdout: create 187/34 file after the snapshot 2022-06-27T11:49:20.053 INFO:tasks.workunit.client.0.smithi195.stdout: create 187/35 file after the snapshot 2022-06-27T11:49:20.054 INFO:tasks.workunit.client.0.smithi195.stdout: create 187/36 file after the snapshot 2022-06-27T11:49:20.054 INFO:tasks.workunit.client.0.smithi195.stdout: create 187/37 file after the snapshot 2022-06-27T11:49:20.055 INFO:tasks.workunit.client.0.smithi195.stdout: create 187/38 file after the snapshot 2022-06-27T11:49:20.055 INFO:tasks.workunit.client.0.smithi195.stdout: create 187/39 file after the snapshot 2022-06-27T11:49:20.057 INFO:tasks.workunit.client.0.smithi195.stdout: create 188/.snap/snap-subdir-test 2022-06-27T11:49:20.069 INFO:tasks.workunit.client.0.smithi195.stdout: create 188/30 file after the snapshot 2022-06-27T11:49:20.103 INFO:tasks.workunit.client.0.smithi195.stdout: create 188/31 file after the snapshot 2022-06-27T11:49:20.104 INFO:tasks.workunit.client.0.smithi195.stdout: create 188/32 file after the snapshot 2022-06-27T11:49:20.104 INFO:tasks.workunit.client.0.smithi195.stdout: create 188/33 file after the snapshot 2022-06-27T11:49:20.104 INFO:tasks.workunit.client.0.smithi195.stdout: create 188/34 file after the snapshot 2022-06-27T11:49:20.104 INFO:tasks.workunit.client.0.smithi195.stdout: create 188/35 file after the snapshot 2022-06-27T11:49:20.105 INFO:tasks.workunit.client.0.smithi195.stdout: create 188/36 file after the snapshot 2022-06-27T11:49:20.105 INFO:tasks.workunit.client.0.smithi195.stdout: create 188/37 file after the snapshot 2022-06-27T11:49:20.105 INFO:tasks.workunit.client.0.smithi195.stdout: create 188/38 file after the snapshot 2022-06-27T11:49:20.105 INFO:tasks.workunit.client.0.smithi195.stdout: create 188/39 file after the snapshot 2022-06-27T11:49:20.105 INFO:tasks.workunit.client.0.smithi195.stdout: create 189/.snap/snap-subdir-test 2022-06-27T11:49:20.123 INFO:tasks.workunit.client.0.smithi195.stdout: create 189/30 file after the snapshot 2022-06-27T11:49:20.137 INFO:tasks.workunit.client.0.smithi195.stdout: create 189/31 file after the snapshot 2022-06-27T11:49:20.138 INFO:tasks.workunit.client.0.smithi195.stdout: create 189/32 file after the snapshot 2022-06-27T11:49:20.139 INFO:tasks.workunit.client.0.smithi195.stdout: create 189/33 file after the snapshot 2022-06-27T11:49:20.139 INFO:tasks.workunit.client.0.smithi195.stdout: create 189/34 file after the snapshot 2022-06-27T11:49:20.140 INFO:tasks.workunit.client.0.smithi195.stdout: create 189/35 file after the snapshot 2022-06-27T11:49:20.141 INFO:tasks.workunit.client.0.smithi195.stdout: create 189/36 file after the snapshot 2022-06-27T11:49:20.142 INFO:tasks.workunit.client.0.smithi195.stdout: create 189/37 file after the snapshot 2022-06-27T11:49:20.156 INFO:tasks.workunit.client.0.smithi195.stdout: create 189/38 file after the snapshot 2022-06-27T11:49:20.156 INFO:tasks.workunit.client.0.smithi195.stdout: create 189/39 file after the snapshot 2022-06-27T11:49:20.157 INFO:tasks.workunit.client.0.smithi195.stdout: create 190/.snap/snap-subdir-test 2022-06-27T11:49:20.173 INFO:tasks.workunit.client.0.smithi195.stdout: create 190/30 file after the snapshot 2022-06-27T11:49:20.191 INFO:tasks.workunit.client.0.smithi195.stdout: create 190/31 file after the snapshot 2022-06-27T11:49:20.192 INFO:tasks.workunit.client.0.smithi195.stdout: create 190/32 file after the snapshot 2022-06-27T11:49:20.192 INFO:tasks.workunit.client.0.smithi195.stdout: create 190/33 file after the snapshot 2022-06-27T11:49:20.193 INFO:tasks.workunit.client.0.smithi195.stdout: create 190/34 file after the snapshot 2022-06-27T11:49:20.193 INFO:tasks.workunit.client.0.smithi195.stdout: create 190/35 file after the snapshot 2022-06-27T11:49:20.194 INFO:tasks.workunit.client.0.smithi195.stdout: create 190/36 file after the snapshot 2022-06-27T11:49:20.196 INFO:tasks.workunit.client.0.smithi195.stdout: create 190/37 file after the snapshot 2022-06-27T11:49:20.197 INFO:tasks.workunit.client.0.smithi195.stdout: create 190/38 file after the snapshot 2022-06-27T11:49:20.197 INFO:tasks.workunit.client.0.smithi195.stdout: create 190/39 file after the snapshot 2022-06-27T11:49:20.198 INFO:tasks.workunit.client.0.smithi195.stdout: create 191/.snap/snap-subdir-test 2022-06-27T11:49:20.222 INFO:tasks.workunit.client.0.smithi195.stdout: create 191/30 file after the snapshot 2022-06-27T11:49:20.251 INFO:tasks.workunit.client.0.smithi195.stdout: create 191/31 file after the snapshot 2022-06-27T11:49:20.251 INFO:tasks.workunit.client.0.smithi195.stdout: create 191/32 file after the snapshot 2022-06-27T11:49:20.252 INFO:tasks.workunit.client.0.smithi195.stdout: create 191/33 file after the snapshot 2022-06-27T11:49:20.252 INFO:tasks.workunit.client.0.smithi195.stdout: create 191/34 file after the snapshot 2022-06-27T11:49:20.253 INFO:tasks.workunit.client.0.smithi195.stdout: create 191/35 file after the snapshot 2022-06-27T11:49:20.253 INFO:tasks.workunit.client.0.smithi195.stdout: create 191/36 file after the snapshot 2022-06-27T11:49:20.253 INFO:tasks.workunit.client.0.smithi195.stdout: create 191/37 file after the snapshot 2022-06-27T11:49:20.254 INFO:tasks.workunit.client.0.smithi195.stdout: create 191/38 file after the snapshot 2022-06-27T11:49:20.254 INFO:tasks.workunit.client.0.smithi195.stdout: create 191/39 file after the snapshot 2022-06-27T11:49:20.255 INFO:tasks.workunit.client.0.smithi195.stdout: create 192/.snap/snap-subdir-test 2022-06-27T11:49:20.273 INFO:tasks.workunit.client.0.smithi195.stdout: create 192/30 file after the snapshot 2022-06-27T11:49:20.296 INFO:tasks.workunit.client.0.smithi195.stdout: create 192/31 file after the snapshot 2022-06-27T11:49:20.297 INFO:tasks.workunit.client.0.smithi195.stdout: create 192/32 file after the snapshot 2022-06-27T11:49:20.298 INFO:tasks.workunit.client.0.smithi195.stdout: create 192/33 file after the snapshot 2022-06-27T11:49:20.298 INFO:tasks.workunit.client.0.smithi195.stdout: create 192/34 file after the snapshot 2022-06-27T11:49:20.299 INFO:tasks.workunit.client.0.smithi195.stdout: create 192/35 file after the snapshot 2022-06-27T11:49:20.299 INFO:tasks.workunit.client.0.smithi195.stdout: create 192/36 file after the snapshot 2022-06-27T11:49:20.299 INFO:tasks.workunit.client.0.smithi195.stdout: create 192/37 file after the snapshot 2022-06-27T11:49:20.300 INFO:tasks.workunit.client.0.smithi195.stdout: create 192/38 file after the snapshot 2022-06-27T11:49:20.300 INFO:tasks.workunit.client.0.smithi195.stdout: create 192/39 file after the snapshot 2022-06-27T11:49:20.301 INFO:tasks.workunit.client.0.smithi195.stdout: create 193/.snap/snap-subdir-test 2022-06-27T11:49:20.331 INFO:tasks.workunit.client.0.smithi195.stdout: create 193/30 file after the snapshot 2022-06-27T11:49:20.344 INFO:tasks.workunit.client.0.smithi195.stdout: create 193/31 file after the snapshot 2022-06-27T11:49:20.345 INFO:tasks.workunit.client.0.smithi195.stdout: create 193/32 file after the snapshot 2022-06-27T11:49:20.348 INFO:tasks.workunit.client.0.smithi195.stdout: create 193/33 file after the snapshot 2022-06-27T11:49:20.349 INFO:tasks.workunit.client.0.smithi195.stdout: create 193/34 file after the snapshot 2022-06-27T11:49:20.351 INFO:tasks.workunit.client.0.smithi195.stdout: create 193/35 file after the snapshot 2022-06-27T11:49:20.352 INFO:tasks.workunit.client.0.smithi195.stdout: create 193/36 file after the snapshot 2022-06-27T11:49:20.352 INFO:tasks.workunit.client.0.smithi195.stdout: create 193/37 file after the snapshot 2022-06-27T11:49:20.355 INFO:tasks.workunit.client.0.smithi195.stdout: create 193/38 file after the snapshot 2022-06-27T11:49:20.355 INFO:tasks.workunit.client.0.smithi195.stdout: create 193/39 file after the snapshot 2022-06-27T11:49:20.355 INFO:tasks.workunit.client.0.smithi195.stdout: create 194/.snap/snap-subdir-test 2022-06-27T11:49:20.385 INFO:tasks.workunit.client.0.smithi195.stdout: create 194/30 file after the snapshot 2022-06-27T11:49:20.398 INFO:tasks.workunit.client.0.smithi195.stdout: create 194/31 file after the snapshot 2022-06-27T11:49:20.399 INFO:tasks.workunit.client.0.smithi195.stdout: create 194/32 file after the snapshot 2022-06-27T11:49:20.400 INFO:tasks.workunit.client.0.smithi195.stdout: create 194/33 file after the snapshot 2022-06-27T11:49:20.401 INFO:tasks.workunit.client.0.smithi195.stdout: create 194/34 file after the snapshot 2022-06-27T11:49:20.402 INFO:tasks.workunit.client.0.smithi195.stdout: create 194/35 file after the snapshot 2022-06-27T11:49:20.403 INFO:tasks.workunit.client.0.smithi195.stdout: create 194/36 file after the snapshot 2022-06-27T11:49:20.405 INFO:tasks.workunit.client.0.smithi195.stdout: create 194/37 file after the snapshot 2022-06-27T11:49:20.405 INFO:tasks.workunit.client.0.smithi195.stdout: create 194/38 file after the snapshot 2022-06-27T11:49:20.406 INFO:tasks.workunit.client.0.smithi195.stdout: create 194/39 file after the snapshot 2022-06-27T11:49:20.407 INFO:tasks.workunit.client.0.smithi195.stdout: create 195/.snap/snap-subdir-test 2022-06-27T11:49:20.437 INFO:tasks.workunit.client.0.smithi195.stdout: create 195/30 file after the snapshot 2022-06-27T11:49:20.731 INFO:tasks.workunit.client.0.smithi195.stdout: create 195/31 file after the snapshot 2022-06-27T11:49:20.731 INFO:tasks.workunit.client.0.smithi195.stdout: create 195/32 file after the snapshot 2022-06-27T11:49:20.732 INFO:tasks.workunit.client.0.smithi195.stdout: create 195/33 file after the snapshot 2022-06-27T11:49:20.732 INFO:tasks.workunit.client.0.smithi195.stdout: create 195/34 file after the snapshot 2022-06-27T11:49:20.738 INFO:tasks.workunit.client.0.smithi195.stdout: create 195/35 file after the snapshot 2022-06-27T11:49:20.739 INFO:tasks.workunit.client.0.smithi195.stdout: create 195/36 file after the snapshot 2022-06-27T11:49:20.739 INFO:tasks.workunit.client.0.smithi195.stdout: create 195/37 file after the snapshot 2022-06-27T11:49:20.739 INFO:tasks.workunit.client.0.smithi195.stdout: create 195/38 file after the snapshot 2022-06-27T11:49:20.740 INFO:tasks.workunit.client.0.smithi195.stdout: create 195/39 file after the snapshot 2022-06-27T11:49:20.742 INFO:tasks.workunit.client.0.smithi195.stdout: create 196/.snap/snap-subdir-test 2022-06-27T11:49:20.767 INFO:tasks.workunit.client.0.smithi195.stdout: create 196/30 file after the snapshot 2022-06-27T11:49:20.782 INFO:tasks.workunit.client.0.smithi195.stdout: create 196/31 file after the snapshot 2022-06-27T11:49:20.783 INFO:tasks.workunit.client.0.smithi195.stdout: create 196/32 file after the snapshot 2022-06-27T11:49:20.783 INFO:tasks.workunit.client.0.smithi195.stdout: create 196/33 file after the snapshot 2022-06-27T11:49:20.784 INFO:tasks.workunit.client.0.smithi195.stdout: create 196/34 file after the snapshot 2022-06-27T11:49:20.787 INFO:tasks.workunit.client.0.smithi195.stdout: create 196/35 file after the snapshot 2022-06-27T11:49:20.787 INFO:tasks.workunit.client.0.smithi195.stdout: create 196/36 file after the snapshot 2022-06-27T11:49:20.788 INFO:tasks.workunit.client.0.smithi195.stdout: create 196/37 file after the snapshot 2022-06-27T11:49:20.789 INFO:tasks.workunit.client.0.smithi195.stdout: create 196/38 file after the snapshot 2022-06-27T11:49:20.802 INFO:tasks.workunit.client.0.smithi195.stdout: create 196/39 file after the snapshot 2022-06-27T11:49:20.803 INFO:tasks.workunit.client.0.smithi195.stdout: create 197/.snap/snap-subdir-test 2022-06-27T11:49:20.817 INFO:tasks.workunit.client.0.smithi195.stdout: create 197/30 file after the snapshot 2022-06-27T11:49:20.842 INFO:tasks.workunit.client.0.smithi195.stdout: create 197/31 file after the snapshot 2022-06-27T11:49:20.843 INFO:tasks.workunit.client.0.smithi195.stdout: create 197/32 file after the snapshot 2022-06-27T11:49:20.843 INFO:tasks.workunit.client.0.smithi195.stdout: create 197/33 file after the snapshot 2022-06-27T11:49:20.843 INFO:tasks.workunit.client.0.smithi195.stdout: create 197/34 file after the snapshot 2022-06-27T11:49:20.843 INFO:tasks.workunit.client.0.smithi195.stdout: create 197/35 file after the snapshot 2022-06-27T11:49:20.844 INFO:tasks.workunit.client.0.smithi195.stdout: create 197/36 file after the snapshot 2022-06-27T11:49:20.844 INFO:tasks.workunit.client.0.smithi195.stdout: create 197/37 file after the snapshot 2022-06-27T11:49:20.844 INFO:tasks.workunit.client.0.smithi195.stdout: create 197/38 file after the snapshot 2022-06-27T11:49:20.844 INFO:tasks.workunit.client.0.smithi195.stdout: create 197/39 file after the snapshot 2022-06-27T11:49:20.845 INFO:tasks.workunit.client.0.smithi195.stdout: create 198/.snap/snap-subdir-test 2022-06-27T11:49:20.867 INFO:tasks.workunit.client.0.smithi195.stdout: create 198/30 file after the snapshot 2022-06-27T11:49:20.887 INFO:tasks.workunit.client.0.smithi195.stdout: create 198/31 file after the snapshot 2022-06-27T11:49:20.887 INFO:tasks.workunit.client.0.smithi195.stdout: create 198/32 file after the snapshot 2022-06-27T11:49:20.888 INFO:tasks.workunit.client.0.smithi195.stdout: create 198/33 file after the snapshot 2022-06-27T11:49:20.888 INFO:tasks.workunit.client.0.smithi195.stdout: create 198/34 file after the snapshot 2022-06-27T11:49:20.888 INFO:tasks.workunit.client.0.smithi195.stdout: create 198/35 file after the snapshot 2022-06-27T11:49:20.888 INFO:tasks.workunit.client.0.smithi195.stdout: create 198/36 file after the snapshot 2022-06-27T11:49:20.889 INFO:tasks.workunit.client.0.smithi195.stdout: create 198/37 file after the snapshot 2022-06-27T11:49:20.892 INFO:tasks.workunit.client.0.smithi195.stdout: create 198/38 file after the snapshot 2022-06-27T11:49:20.892 INFO:tasks.workunit.client.0.smithi195.stdout: create 198/39 file after the snapshot 2022-06-27T11:49:20.898 INFO:tasks.workunit.client.0.smithi195.stdout: create 199/.snap/snap-subdir-test 2022-06-27T11:49:20.918 INFO:tasks.workunit.client.0.smithi195.stdout: create 199/30 file after the snapshot 2022-06-27T11:49:20.933 INFO:tasks.workunit.client.0.smithi195.stdout: create 199/31 file after the snapshot 2022-06-27T11:49:20.935 INFO:tasks.workunit.client.0.smithi195.stdout: create 199/32 file after the snapshot 2022-06-27T11:49:20.936 INFO:tasks.workunit.client.0.smithi195.stdout: create 199/33 file after the snapshot 2022-06-27T11:49:20.936 INFO:tasks.workunit.client.0.smithi195.stdout: create 199/34 file after the snapshot 2022-06-27T11:49:20.937 INFO:tasks.workunit.client.0.smithi195.stdout: create 199/35 file after the snapshot 2022-06-27T11:49:20.939 INFO:tasks.workunit.client.0.smithi195.stdout: create 199/36 file after the snapshot 2022-06-27T11:49:20.940 INFO:tasks.workunit.client.0.smithi195.stdout: create 199/37 file after the snapshot 2022-06-27T11:49:20.941 INFO:tasks.workunit.client.0.smithi195.stdout: create 199/38 file after the snapshot 2022-06-27T11:49:20.941 INFO:tasks.workunit.client.0.smithi195.stdout: create 199/39 file after the snapshot 2022-06-27T11:49:20.942 INFO:tasks.workunit.client.0.smithi195.stdout: create 200/.snap/snap-subdir-test 2022-06-27T11:49:20.968 INFO:tasks.workunit.client.0.smithi195.stdout: create 200/30 file after the snapshot 2022-06-27T11:49:20.969 INFO:tasks.workunit.client.0.smithi195.stdout: create 200/31 file after the snapshot 2022-06-27T11:49:20.971 INFO:tasks.workunit.client.0.smithi195.stdout: create 200/32 file after the snapshot 2022-06-27T11:49:20.972 INFO:tasks.workunit.client.0.smithi195.stdout: create 200/33 file after the snapshot 2022-06-27T11:49:20.975 INFO:tasks.workunit.client.0.smithi195.stdout: create 200/34 file after the snapshot 2022-06-27T11:49:20.976 INFO:tasks.workunit.client.0.smithi195.stdout: create 200/35 file after the snapshot 2022-06-27T11:49:20.977 INFO:tasks.workunit.client.0.smithi195.stdout: create 200/36 file after the snapshot 2022-06-27T11:49:20.979 INFO:tasks.workunit.client.0.smithi195.stdout: create 200/37 file after the snapshot 2022-06-27T11:49:20.981 INFO:tasks.workunit.client.0.smithi195.stdout: create 200/38 file after the snapshot 2022-06-27T11:49:20.982 INFO:tasks.workunit.client.0.smithi195.stdout: create 200/39 file after the snapshot 2022-06-27T11:49:20.983 INFO:tasks.workunit.client.0.smithi195.stdout: create 201/.snap/snap-subdir-test 2022-06-27T11:49:22.157 INFO:tasks.workunit.client.0.smithi195.stdout: create 201/30 file after the snapshot 2022-06-27T11:49:25.105 INFO:tasks.workunit.client.0.smithi195.stdout: create 201/31 file after the snapshot 2022-06-27T11:49:25.105 INFO:tasks.workunit.client.0.smithi195.stdout: create 201/32 file after the snapshot 2022-06-27T11:49:25.106 INFO:tasks.workunit.client.0.smithi195.stdout: create 201/33 file after the snapshot 2022-06-27T11:49:25.482 INFO:tasks.workunit.client.0.smithi195.stdout: create 201/34 file after the snapshot 2022-06-27T11:49:25.483 INFO:tasks.workunit.client.0.smithi195.stdout: create 201/35 file after the snapshot 2022-06-27T11:49:25.484 INFO:tasks.workunit.client.0.smithi195.stdout: create 201/36 file after the snapshot 2022-06-27T11:49:25.486 INFO:tasks.workunit.client.0.smithi195.stdout: create 201/37 file after the snapshot 2022-06-27T11:49:25.486 INFO:tasks.workunit.client.0.smithi195.stdout: create 201/38 file after the snapshot 2022-06-27T11:49:25.487 INFO:tasks.workunit.client.0.smithi195.stdout: create 201/39 file after the snapshot 2022-06-27T11:49:25.488 INFO:tasks.workunit.client.0.smithi195.stdout: create 202/.snap/snap-subdir-test 2022-06-27T11:49:25.530 INFO:tasks.workunit.client.0.smithi195.stdout: create 202/30 file after the snapshot 2022-06-27T11:49:25.546 INFO:tasks.workunit.client.0.smithi195.stdout: create 202/31 file after the snapshot 2022-06-27T11:49:25.551 INFO:tasks.workunit.client.0.smithi195.stdout: create 202/32 file after the snapshot 2022-06-27T11:49:25.553 INFO:tasks.workunit.client.0.smithi195.stdout: create 202/33 file after the snapshot 2022-06-27T11:49:25.553 INFO:tasks.workunit.client.0.smithi195.stdout: create 202/34 file after the snapshot 2022-06-27T11:49:25.554 INFO:tasks.workunit.client.0.smithi195.stdout: create 202/35 file after the snapshot 2022-06-27T11:49:25.554 INFO:tasks.workunit.client.0.smithi195.stdout: create 202/36 file after the snapshot 2022-06-27T11:49:25.556 INFO:tasks.workunit.client.0.smithi195.stdout: create 202/37 file after the snapshot 2022-06-27T11:49:25.557 INFO:tasks.workunit.client.0.smithi195.stdout: create 202/38 file after the snapshot 2022-06-27T11:49:25.557 INFO:tasks.workunit.client.0.smithi195.stdout: create 202/39 file after the snapshot 2022-06-27T11:49:25.558 INFO:tasks.workunit.client.0.smithi195.stdout: create 203/.snap/snap-subdir-test 2022-06-27T11:49:25.590 INFO:tasks.workunit.client.0.smithi195.stdout: create 203/30 file after the snapshot 2022-06-27T11:49:25.597 INFO:tasks.workunit.client.0.smithi195.stdout: create 203/31 file after the snapshot 2022-06-27T11:49:25.598 INFO:tasks.workunit.client.0.smithi195.stdout: create 203/32 file after the snapshot 2022-06-27T11:49:25.599 INFO:tasks.workunit.client.0.smithi195.stdout: create 203/33 file after the snapshot 2022-06-27T11:49:25.599 INFO:tasks.workunit.client.0.smithi195.stdout: create 203/34 file after the snapshot 2022-06-27T11:49:25.599 INFO:tasks.workunit.client.0.smithi195.stdout: create 203/35 file after the snapshot 2022-06-27T11:49:25.600 INFO:tasks.workunit.client.0.smithi195.stdout: create 203/36 file after the snapshot 2022-06-27T11:49:25.600 INFO:tasks.workunit.client.0.smithi195.stdout: create 203/37 file after the snapshot 2022-06-27T11:49:25.601 INFO:tasks.workunit.client.0.smithi195.stdout: create 203/38 file after the snapshot 2022-06-27T11:49:25.601 INFO:tasks.workunit.client.0.smithi195.stdout: create 203/39 file after the snapshot 2022-06-27T11:49:25.602 INFO:tasks.workunit.client.0.smithi195.stdout: create 204/.snap/snap-subdir-test 2022-06-27T11:49:25.637 INFO:tasks.workunit.client.0.smithi195.stdout: create 204/30 file after the snapshot 2022-06-27T11:49:25.653 INFO:tasks.workunit.client.0.smithi195.stdout: create 204/31 file after the snapshot 2022-06-27T11:49:25.653 INFO:tasks.workunit.client.0.smithi195.stdout: create 204/32 file after the snapshot 2022-06-27T11:49:25.653 INFO:tasks.workunit.client.0.smithi195.stdout: create 204/33 file after the snapshot 2022-06-27T11:49:25.660 INFO:tasks.workunit.client.0.smithi195.stdout: create 204/34 file after the snapshot 2022-06-27T11:49:25.661 INFO:tasks.workunit.client.0.smithi195.stdout: create 204/35 file after the snapshot 2022-06-27T11:49:25.661 INFO:tasks.workunit.client.0.smithi195.stdout: create 204/36 file after the snapshot 2022-06-27T11:49:25.661 INFO:tasks.workunit.client.0.smithi195.stdout: create 204/37 file after the snapshot 2022-06-27T11:49:25.663 INFO:tasks.workunit.client.0.smithi195.stdout: create 204/38 file after the snapshot 2022-06-27T11:49:25.664 INFO:tasks.workunit.client.0.smithi195.stdout: create 204/39 file after the snapshot 2022-06-27T11:49:25.664 INFO:tasks.workunit.client.0.smithi195.stdout: create 205/.snap/snap-subdir-test 2022-06-27T11:49:25.694 INFO:tasks.workunit.client.0.smithi195.stdout: create 205/30 file after the snapshot 2022-06-27T11:49:25.720 INFO:tasks.workunit.client.0.smithi195.stdout: create 205/31 file after the snapshot 2022-06-27T11:49:25.721 INFO:tasks.workunit.client.0.smithi195.stdout: create 205/32 file after the snapshot 2022-06-27T11:49:25.721 INFO:tasks.workunit.client.0.smithi195.stdout: create 205/33 file after the snapshot 2022-06-27T11:49:25.722 INFO:tasks.workunit.client.0.smithi195.stdout: create 205/34 file after the snapshot 2022-06-27T11:49:25.722 INFO:tasks.workunit.client.0.smithi195.stdout: create 205/35 file after the snapshot 2022-06-27T11:49:25.723 INFO:tasks.workunit.client.0.smithi195.stdout: create 205/36 file after the snapshot 2022-06-27T11:49:25.723 INFO:tasks.workunit.client.0.smithi195.stdout: create 205/37 file after the snapshot 2022-06-27T11:49:25.723 INFO:tasks.workunit.client.0.smithi195.stdout: create 205/38 file after the snapshot 2022-06-27T11:49:25.724 INFO:tasks.workunit.client.0.smithi195.stdout: create 205/39 file after the snapshot 2022-06-27T11:49:25.724 INFO:tasks.workunit.client.0.smithi195.stdout: create 206/.snap/snap-subdir-test 2022-06-27T11:49:25.750 INFO:tasks.workunit.client.0.smithi195.stdout: create 206/30 file after the snapshot 2022-06-27T11:49:25.771 INFO:tasks.workunit.client.0.smithi195.stdout: create 206/31 file after the snapshot 2022-06-27T11:49:25.771 INFO:tasks.workunit.client.0.smithi195.stdout: create 206/32 file after the snapshot 2022-06-27T11:49:25.772 INFO:tasks.workunit.client.0.smithi195.stdout: create 206/33 file after the snapshot 2022-06-27T11:49:25.772 INFO:tasks.workunit.client.0.smithi195.stdout: create 206/34 file after the snapshot 2022-06-27T11:49:25.772 INFO:tasks.workunit.client.0.smithi195.stdout: create 206/35 file after the snapshot 2022-06-27T11:49:25.773 INFO:tasks.workunit.client.0.smithi195.stdout: create 206/36 file after the snapshot 2022-06-27T11:49:25.773 INFO:tasks.workunit.client.0.smithi195.stdout: create 206/37 file after the snapshot 2022-06-27T11:49:25.774 INFO:tasks.workunit.client.0.smithi195.stdout: create 206/38 file after the snapshot 2022-06-27T11:49:25.775 INFO:tasks.workunit.client.0.smithi195.stdout: create 206/39 file after the snapshot 2022-06-27T11:49:25.775 INFO:tasks.workunit.client.0.smithi195.stdout: create 207/.snap/snap-subdir-test 2022-06-27T11:49:25.806 INFO:tasks.workunit.client.0.smithi195.stdout: create 207/30 file after the snapshot 2022-06-27T11:49:25.821 INFO:tasks.workunit.client.0.smithi195.stdout: create 207/31 file after the snapshot 2022-06-27T11:49:25.822 INFO:tasks.workunit.client.0.smithi195.stdout: create 207/32 file after the snapshot 2022-06-27T11:49:25.823 INFO:tasks.workunit.client.0.smithi195.stdout: create 207/33 file after the snapshot 2022-06-27T11:49:25.824 INFO:tasks.workunit.client.0.smithi195.stdout: create 207/34 file after the snapshot 2022-06-27T11:49:25.825 INFO:tasks.workunit.client.0.smithi195.stdout: create 207/35 file after the snapshot 2022-06-27T11:49:25.826 INFO:tasks.workunit.client.0.smithi195.stdout: create 207/36 file after the snapshot 2022-06-27T11:49:25.827 INFO:tasks.workunit.client.0.smithi195.stdout: create 207/37 file after the snapshot 2022-06-27T11:49:25.828 INFO:tasks.workunit.client.0.smithi195.stdout: create 207/38 file after the snapshot 2022-06-27T11:49:25.829 INFO:tasks.workunit.client.0.smithi195.stdout: create 207/39 file after the snapshot 2022-06-27T11:49:25.831 INFO:tasks.workunit.client.0.smithi195.stdout: create 208/.snap/snap-subdir-test 2022-06-27T11:49:25.862 INFO:tasks.workunit.client.0.smithi195.stdout: create 208/30 file after the snapshot 2022-06-27T11:49:25.863 INFO:tasks.workunit.client.0.smithi195.stdout: create 208/31 file after the snapshot 2022-06-27T11:49:25.864 INFO:tasks.workunit.client.0.smithi195.stdout: create 208/32 file after the snapshot 2022-06-27T11:49:25.865 INFO:tasks.workunit.client.0.smithi195.stdout: create 208/33 file after the snapshot 2022-06-27T11:49:25.865 INFO:tasks.workunit.client.0.smithi195.stdout: create 208/34 file after the snapshot 2022-06-27T11:49:25.866 INFO:tasks.workunit.client.0.smithi195.stdout: create 208/35 file after the snapshot 2022-06-27T11:49:25.867 INFO:tasks.workunit.client.0.smithi195.stdout: create 208/36 file after the snapshot 2022-06-27T11:49:25.868 INFO:tasks.workunit.client.0.smithi195.stdout: create 208/37 file after the snapshot 2022-06-27T11:49:25.869 INFO:tasks.workunit.client.0.smithi195.stdout: create 208/38 file after the snapshot 2022-06-27T11:49:25.870 INFO:tasks.workunit.client.0.smithi195.stdout: create 208/39 file after the snapshot 2022-06-27T11:49:25.871 INFO:tasks.workunit.client.0.smithi195.stdout: create 209/.snap/snap-subdir-test 2022-06-27T11:49:25.908 INFO:tasks.workunit.client.0.smithi195.stdout: create 209/30 file after the snapshot 2022-06-27T11:49:25.923 INFO:tasks.workunit.client.0.smithi195.stdout: create 209/31 file after the snapshot 2022-06-27T11:49:25.924 INFO:tasks.workunit.client.0.smithi195.stdout: create 209/32 file after the snapshot 2022-06-27T11:49:25.925 INFO:tasks.workunit.client.0.smithi195.stdout: create 209/33 file after the snapshot 2022-06-27T11:49:25.928 INFO:tasks.workunit.client.0.smithi195.stdout: create 209/34 file after the snapshot 2022-06-27T11:49:25.929 INFO:tasks.workunit.client.0.smithi195.stdout: create 209/35 file after the snapshot 2022-06-27T11:49:25.929 INFO:tasks.workunit.client.0.smithi195.stdout: create 209/36 file after the snapshot 2022-06-27T11:49:25.930 INFO:tasks.workunit.client.0.smithi195.stdout: create 209/37 file after the snapshot 2022-06-27T11:49:25.931 INFO:tasks.workunit.client.0.smithi195.stdout: create 209/38 file after the snapshot 2022-06-27T11:49:25.931 INFO:tasks.workunit.client.0.smithi195.stdout: create 209/39 file after the snapshot 2022-06-27T11:49:25.932 INFO:tasks.workunit.client.0.smithi195.stdout: create 210/.snap/snap-subdir-test 2022-06-27T11:49:25.968 INFO:tasks.workunit.client.0.smithi195.stdout: create 210/30 file after the snapshot 2022-06-27T11:49:25.984 INFO:tasks.workunit.client.0.smithi195.stdout: create 210/31 file after the snapshot 2022-06-27T11:49:25.984 INFO:tasks.workunit.client.0.smithi195.stdout: create 210/32 file after the snapshot 2022-06-27T11:49:25.984 INFO:tasks.workunit.client.0.smithi195.stdout: create 210/33 file after the snapshot 2022-06-27T11:49:25.985 INFO:tasks.workunit.client.0.smithi195.stdout: create 210/34 file after the snapshot 2022-06-27T11:49:25.985 INFO:tasks.workunit.client.0.smithi195.stdout: create 210/35 file after the snapshot 2022-06-27T11:49:25.986 INFO:tasks.workunit.client.0.smithi195.stdout: create 210/36 file after the snapshot 2022-06-27T11:49:25.987 INFO:tasks.workunit.client.0.smithi195.stdout: create 210/37 file after the snapshot 2022-06-27T11:49:25.987 INFO:tasks.workunit.client.0.smithi195.stdout: create 210/38 file after the snapshot 2022-06-27T11:49:25.988 INFO:tasks.workunit.client.0.smithi195.stdout: create 210/39 file after the snapshot 2022-06-27T11:49:25.988 INFO:tasks.workunit.client.0.smithi195.stdout: create 211/.snap/snap-subdir-test 2022-06-27T11:49:26.024 INFO:tasks.workunit.client.0.smithi195.stdout: create 211/30 file after the snapshot 2022-06-27T11:49:26.045 INFO:tasks.workunit.client.0.smithi195.stdout: create 211/31 file after the snapshot 2022-06-27T11:49:26.046 INFO:tasks.workunit.client.0.smithi195.stdout: create 211/32 file after the snapshot 2022-06-27T11:49:26.046 INFO:tasks.workunit.client.0.smithi195.stdout: create 211/33 file after the snapshot 2022-06-27T11:49:26.047 INFO:tasks.workunit.client.0.smithi195.stdout: create 211/34 file after the snapshot 2022-06-27T11:49:26.047 INFO:tasks.workunit.client.0.smithi195.stdout: create 211/35 file after the snapshot 2022-06-27T11:49:26.047 INFO:tasks.workunit.client.0.smithi195.stdout: create 211/36 file after the snapshot 2022-06-27T11:49:26.048 INFO:tasks.workunit.client.0.smithi195.stdout: create 211/37 file after the snapshot 2022-06-27T11:49:26.048 INFO:tasks.workunit.client.0.smithi195.stdout: create 211/38 file after the snapshot 2022-06-27T11:49:26.049 INFO:tasks.workunit.client.0.smithi195.stdout: create 211/39 file after the snapshot 2022-06-27T11:49:26.049 INFO:tasks.workunit.client.0.smithi195.stdout: create 212/.snap/snap-subdir-test 2022-06-27T11:49:26.072 INFO:tasks.workunit.client.0.smithi195.stdout: create 212/30 file after the snapshot 2022-06-27T11:49:26.089 INFO:tasks.workunit.client.0.smithi195.stdout: create 212/31 file after the snapshot 2022-06-27T11:49:26.089 INFO:tasks.workunit.client.0.smithi195.stdout: create 212/32 file after the snapshot 2022-06-27T11:49:26.089 INFO:tasks.workunit.client.0.smithi195.stdout: create 212/33 file after the snapshot 2022-06-27T11:49:26.091 INFO:tasks.workunit.client.0.smithi195.stdout: create 212/34 file after the snapshot 2022-06-27T11:49:26.091 INFO:tasks.workunit.client.0.smithi195.stdout: create 212/35 file after the snapshot 2022-06-27T11:49:26.092 INFO:tasks.workunit.client.0.smithi195.stdout: create 212/36 file after the snapshot 2022-06-27T11:49:26.092 INFO:tasks.workunit.client.0.smithi195.stdout: create 212/37 file after the snapshot 2022-06-27T11:49:26.093 INFO:tasks.workunit.client.0.smithi195.stdout: create 212/38 file after the snapshot 2022-06-27T11:49:26.093 INFO:tasks.workunit.client.0.smithi195.stdout: create 212/39 file after the snapshot 2022-06-27T11:49:26.095 INFO:tasks.workunit.client.0.smithi195.stdout: create 213/.snap/snap-subdir-test 2022-06-27T11:49:26.124 INFO:tasks.workunit.client.0.smithi195.stdout: create 213/30 file after the snapshot 2022-06-27T11:49:26.137 INFO:tasks.workunit.client.0.smithi195.stdout: create 213/31 file after the snapshot 2022-06-27T11:49:26.138 INFO:tasks.workunit.client.0.smithi195.stdout: create 213/32 file after the snapshot 2022-06-27T11:49:26.139 INFO:tasks.workunit.client.0.smithi195.stdout: create 213/33 file after the snapshot 2022-06-27T11:49:26.140 INFO:tasks.workunit.client.0.smithi195.stdout: create 213/34 file after the snapshot 2022-06-27T11:49:26.140 INFO:tasks.workunit.client.0.smithi195.stdout: create 213/35 file after the snapshot 2022-06-27T11:49:26.141 INFO:tasks.workunit.client.0.smithi195.stdout: create 213/36 file after the snapshot 2022-06-27T11:49:26.142 INFO:tasks.workunit.client.0.smithi195.stdout: create 213/37 file after the snapshot 2022-06-27T11:49:26.165 INFO:tasks.workunit.client.0.smithi195.stdout: create 213/38 file after the snapshot 2022-06-27T11:49:26.166 INFO:tasks.workunit.client.0.smithi195.stdout: create 213/39 file after the snapshot 2022-06-27T11:49:26.166 INFO:tasks.workunit.client.0.smithi195.stdout: create 214/.snap/snap-subdir-test 2022-06-27T11:49:26.171 INFO:tasks.workunit.client.0.smithi195.stdout: create 214/30 file after the snapshot 2022-06-27T11:49:26.187 INFO:tasks.workunit.client.0.smithi195.stdout: create 214/31 file after the snapshot 2022-06-27T11:49:26.188 INFO:tasks.workunit.client.0.smithi195.stdout: create 214/32 file after the snapshot 2022-06-27T11:49:26.189 INFO:tasks.workunit.client.0.smithi195.stdout: create 214/33 file after the snapshot 2022-06-27T11:49:26.191 INFO:tasks.workunit.client.0.smithi195.stdout: create 214/34 file after the snapshot 2022-06-27T11:49:26.192 INFO:tasks.workunit.client.0.smithi195.stdout: create 214/35 file after the snapshot 2022-06-27T11:49:26.192 INFO:tasks.workunit.client.0.smithi195.stdout: create 214/36 file after the snapshot 2022-06-27T11:49:26.194 INFO:tasks.workunit.client.0.smithi195.stdout: create 214/37 file after the snapshot 2022-06-27T11:49:26.195 INFO:tasks.workunit.client.0.smithi195.stdout: create 214/38 file after the snapshot 2022-06-27T11:49:26.195 INFO:tasks.workunit.client.0.smithi195.stdout: create 214/39 file after the snapshot 2022-06-27T11:49:26.202 INFO:tasks.workunit.client.0.smithi195.stdout: create 215/.snap/snap-subdir-test 2022-06-27T11:49:26.219 INFO:tasks.workunit.client.0.smithi195.stdout: create 215/30 file after the snapshot 2022-06-27T11:49:26.236 INFO:tasks.workunit.client.0.smithi195.stdout: create 215/31 file after the snapshot 2022-06-27T11:49:26.237 INFO:tasks.workunit.client.0.smithi195.stdout: create 215/32 file after the snapshot 2022-06-27T11:49:26.238 INFO:tasks.workunit.client.0.smithi195.stdout: create 215/33 file after the snapshot 2022-06-27T11:49:26.239 INFO:tasks.workunit.client.0.smithi195.stdout: create 215/34 file after the snapshot 2022-06-27T11:49:26.239 INFO:tasks.workunit.client.0.smithi195.stdout: create 215/35 file after the snapshot 2022-06-27T11:49:26.241 INFO:tasks.workunit.client.0.smithi195.stdout: create 215/36 file after the snapshot 2022-06-27T11:49:26.241 INFO:tasks.workunit.client.0.smithi195.stdout: create 215/37 file after the snapshot 2022-06-27T11:49:26.242 INFO:tasks.workunit.client.0.smithi195.stdout: create 215/38 file after the snapshot 2022-06-27T11:49:26.243 INFO:tasks.workunit.client.0.smithi195.stdout: create 215/39 file after the snapshot 2022-06-27T11:49:26.244 INFO:tasks.workunit.client.0.smithi195.stdout: create 216/.snap/snap-subdir-test 2022-06-27T11:49:26.269 INFO:tasks.workunit.client.0.smithi195.stdout: create 216/30 file after the snapshot 2022-06-27T11:49:26.285 INFO:tasks.workunit.client.0.smithi195.stdout: create 216/31 file after the snapshot 2022-06-27T11:49:26.286 INFO:tasks.workunit.client.0.smithi195.stdout: create 216/32 file after the snapshot 2022-06-27T11:49:26.286 INFO:tasks.workunit.client.0.smithi195.stdout: create 216/33 file after the snapshot 2022-06-27T11:49:26.287 INFO:tasks.workunit.client.0.smithi195.stdout: create 216/34 file after the snapshot 2022-06-27T11:49:26.288 INFO:tasks.workunit.client.0.smithi195.stdout: create 216/35 file after the snapshot 2022-06-27T11:49:26.289 INFO:tasks.workunit.client.0.smithi195.stdout: create 216/36 file after the snapshot 2022-06-27T11:49:26.290 INFO:tasks.workunit.client.0.smithi195.stdout: create 216/37 file after the snapshot 2022-06-27T11:49:26.291 INFO:tasks.workunit.client.0.smithi195.stdout: create 216/38 file after the snapshot 2022-06-27T11:49:26.292 INFO:tasks.workunit.client.0.smithi195.stdout: create 216/39 file after the snapshot 2022-06-27T11:49:26.292 INFO:tasks.workunit.client.0.smithi195.stdout: create 217/.snap/snap-subdir-test 2022-06-27T11:49:26.323 INFO:tasks.workunit.client.0.smithi195.stdout: create 217/30 file after the snapshot 2022-06-27T11:49:26.339 INFO:tasks.workunit.client.0.smithi195.stdout: create 217/31 file after the snapshot 2022-06-27T11:49:26.340 INFO:tasks.workunit.client.0.smithi195.stdout: create 217/32 file after the snapshot 2022-06-27T11:49:26.340 INFO:tasks.workunit.client.0.smithi195.stdout: create 217/33 file after the snapshot 2022-06-27T11:49:26.341 INFO:tasks.workunit.client.0.smithi195.stdout: create 217/34 file after the snapshot 2022-06-27T11:49:26.342 INFO:tasks.workunit.client.0.smithi195.stdout: create 217/35 file after the snapshot 2022-06-27T11:49:26.343 INFO:tasks.workunit.client.0.smithi195.stdout: create 217/36 file after the snapshot 2022-06-27T11:49:26.343 INFO:tasks.workunit.client.0.smithi195.stdout: create 217/37 file after the snapshot 2022-06-27T11:49:26.345 INFO:tasks.workunit.client.0.smithi195.stdout: create 217/38 file after the snapshot 2022-06-27T11:49:26.346 INFO:tasks.workunit.client.0.smithi195.stdout: create 217/39 file after the snapshot 2022-06-27T11:49:26.347 INFO:tasks.workunit.client.0.smithi195.stdout: create 218/.snap/snap-subdir-test 2022-06-27T11:49:26.381 INFO:tasks.workunit.client.0.smithi195.stdout: create 218/30 file after the snapshot 2022-06-27T11:49:26.393 INFO:tasks.workunit.client.0.smithi195.stdout: create 218/31 file after the snapshot 2022-06-27T11:49:26.394 INFO:tasks.workunit.client.0.smithi195.stdout: create 218/32 file after the snapshot 2022-06-27T11:49:26.395 INFO:tasks.workunit.client.0.smithi195.stdout: create 218/33 file after the snapshot 2022-06-27T11:49:26.396 INFO:tasks.workunit.client.0.smithi195.stdout: create 218/34 file after the snapshot 2022-06-27T11:49:26.397 INFO:tasks.workunit.client.0.smithi195.stdout: create 218/35 file after the snapshot 2022-06-27T11:49:26.398 INFO:tasks.workunit.client.0.smithi195.stdout: create 218/36 file after the snapshot 2022-06-27T11:49:26.399 INFO:tasks.workunit.client.0.smithi195.stdout: create 218/37 file after the snapshot 2022-06-27T11:49:26.400 INFO:tasks.workunit.client.0.smithi195.stdout: create 218/38 file after the snapshot 2022-06-27T11:49:26.401 INFO:tasks.workunit.client.0.smithi195.stdout: create 218/39 file after the snapshot 2022-06-27T11:49:26.401 INFO:tasks.workunit.client.0.smithi195.stdout: create 219/.snap/snap-subdir-test 2022-06-27T11:49:26.402 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T11:49:26.405 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T11:49:26.431 INFO:tasks.workunit.client.0.smithi195.stdout: create 219/30 file after the snapshot 2022-06-27T11:49:26.433 INFO:tasks.workunit.client.0.smithi195.stdout: create 219/31 file after the snapshot 2022-06-27T11:49:26.434 INFO:tasks.workunit.client.0.smithi195.stdout: create 219/32 file after the snapshot 2022-06-27T11:49:26.436 INFO:tasks.workunit.client.0.smithi195.stdout: create 219/33 file after the snapshot 2022-06-27T11:49:26.436 INFO:tasks.workunit.client.0.smithi195.stdout: create 219/34 file after the snapshot 2022-06-27T11:49:26.437 INFO:tasks.workunit.client.0.smithi195.stdout: create 219/35 file after the snapshot 2022-06-27T11:49:26.437 INFO:tasks.workunit.client.0.smithi195.stdout: create 219/36 file after the snapshot 2022-06-27T11:49:26.439 INFO:tasks.workunit.client.0.smithi195.stdout: create 219/37 file after the snapshot 2022-06-27T11:49:26.440 INFO:tasks.workunit.client.0.smithi195.stdout: create 219/38 file after the snapshot 2022-06-27T11:49:26.441 INFO:tasks.workunit.client.0.smithi195.stdout: create 219/39 file after the snapshot 2022-06-27T11:49:26.441 INFO:tasks.workunit.client.0.smithi195.stdout: create 220/.snap/snap-subdir-test 2022-06-27T11:49:26.467 INFO:tasks.workunit.client.0.smithi195.stdout: create 220/30 file after the snapshot 2022-06-27T11:49:26.482 INFO:tasks.workunit.client.0.smithi195.stdout: create 220/31 file after the snapshot 2022-06-27T11:49:26.483 INFO:tasks.workunit.client.0.smithi195.stdout: create 220/32 file after the snapshot 2022-06-27T11:49:26.484 INFO:tasks.workunit.client.0.smithi195.stdout: create 220/33 file after the snapshot 2022-06-27T11:49:26.485 INFO:tasks.workunit.client.0.smithi195.stdout: create 220/34 file after the snapshot 2022-06-27T11:49:26.486 INFO:tasks.workunit.client.0.smithi195.stdout: create 220/35 file after the snapshot 2022-06-27T11:49:26.496 INFO:tasks.workunit.client.0.smithi195.stdout: create 220/36 file after the snapshot 2022-06-27T11:49:26.497 INFO:tasks.workunit.client.0.smithi195.stdout: create 220/37 file after the snapshot 2022-06-27T11:49:26.498 INFO:tasks.workunit.client.0.smithi195.stdout: create 220/38 file after the snapshot 2022-06-27T11:49:26.498 INFO:tasks.workunit.client.0.smithi195.stdout: create 220/39 file after the snapshot 2022-06-27T11:49:26.499 INFO:tasks.workunit.client.0.smithi195.stdout: create 221/.snap/snap-subdir-test 2022-06-27T11:49:26.516 INFO:tasks.workunit.client.0.smithi195.stdout: create 221/30 file after the snapshot 2022-06-27T11:49:26.811 INFO:tasks.workunit.client.0.smithi195.stdout: create 221/31 file after the snapshot 2022-06-27T11:49:26.811 INFO:tasks.workunit.client.0.smithi195.stdout: create 221/32 file after the snapshot 2022-06-27T11:49:26.819 INFO:tasks.workunit.client.0.smithi195.stdout: create 221/33 file after the snapshot 2022-06-27T11:49:26.819 INFO:tasks.workunit.client.0.smithi195.stdout: create 221/34 file after the snapshot 2022-06-27T11:49:26.820 INFO:tasks.workunit.client.0.smithi195.stdout: create 221/35 file after the snapshot 2022-06-27T11:49:26.820 INFO:tasks.workunit.client.0.smithi195.stdout: create 221/36 file after the snapshot 2022-06-27T11:49:26.820 INFO:tasks.workunit.client.0.smithi195.stdout: create 221/37 file after the snapshot 2022-06-27T11:49:26.821 INFO:tasks.workunit.client.0.smithi195.stdout: create 221/38 file after the snapshot 2022-06-27T11:49:26.821 INFO:tasks.workunit.client.0.smithi195.stdout: create 221/39 file after the snapshot 2022-06-27T11:49:26.821 INFO:tasks.workunit.client.0.smithi195.stdout: create 222/.snap/snap-subdir-test 2022-06-27T11:49:26.850 INFO:tasks.workunit.client.0.smithi195.stdout: create 222/30 file after the snapshot 2022-06-27T11:49:26.851 INFO:tasks.workunit.client.0.smithi195.stdout: create 222/31 file after the snapshot 2022-06-27T11:49:26.851 INFO:tasks.workunit.client.0.smithi195.stdout: create 222/32 file after the snapshot 2022-06-27T11:49:26.851 INFO:tasks.workunit.client.0.smithi195.stdout: create 222/33 file after the snapshot 2022-06-27T11:49:26.851 INFO:tasks.workunit.client.0.smithi195.stdout: create 222/34 file after the snapshot 2022-06-27T11:49:26.861 INFO:tasks.workunit.client.0.smithi195.stdout: create 222/35 file after the snapshot 2022-06-27T11:49:26.861 INFO:tasks.workunit.client.0.smithi195.stdout: create 222/36 file after the snapshot 2022-06-27T11:49:26.861 INFO:tasks.workunit.client.0.smithi195.stdout: create 222/37 file after the snapshot 2022-06-27T11:49:26.862 INFO:tasks.workunit.client.0.smithi195.stdout: create 222/38 file after the snapshot 2022-06-27T11:49:26.862 INFO:tasks.workunit.client.0.smithi195.stdout: create 222/39 file after the snapshot 2022-06-27T11:49:26.862 INFO:tasks.workunit.client.0.smithi195.stdout: create 223/.snap/snap-subdir-test 2022-06-27T11:49:26.883 INFO:tasks.workunit.client.0.smithi195.stdout: create 223/30 file after the snapshot 2022-06-27T11:49:26.900 INFO:tasks.workunit.client.0.smithi195.stdout: create 223/31 file after the snapshot 2022-06-27T11:49:26.900 INFO:tasks.workunit.client.0.smithi195.stdout: create 223/32 file after the snapshot 2022-06-27T11:49:26.963 INFO:tasks.workunit.client.0.smithi195.stdout: create 223/33 file after the snapshot 2022-06-27T11:49:26.964 INFO:tasks.workunit.client.0.smithi195.stdout: create 223/34 file after the snapshot 2022-06-27T11:49:26.965 INFO:tasks.workunit.client.0.smithi195.stdout: create 223/35 file after the snapshot 2022-06-27T11:49:26.966 INFO:tasks.workunit.client.0.smithi195.stdout: create 223/36 file after the snapshot 2022-06-27T11:49:26.966 INFO:tasks.workunit.client.0.smithi195.stdout: create 223/37 file after the snapshot 2022-06-27T11:49:26.967 INFO:tasks.workunit.client.0.smithi195.stdout: create 223/38 file after the snapshot 2022-06-27T11:49:26.968 INFO:tasks.workunit.client.0.smithi195.stdout: create 223/39 file after the snapshot 2022-06-27T11:49:26.969 INFO:tasks.workunit.client.0.smithi195.stdout: create 224/.snap/snap-subdir-test 2022-06-27T11:49:26.996 INFO:tasks.workunit.client.0.smithi195.stdout: create 224/30 file after the snapshot 2022-06-27T11:49:27.012 INFO:tasks.workunit.client.0.smithi195.stdout: create 224/31 file after the snapshot 2022-06-27T11:49:27.014 INFO:tasks.workunit.client.0.smithi195.stdout: create 224/32 file after the snapshot 2022-06-27T11:49:27.014 INFO:tasks.workunit.client.0.smithi195.stdout: create 224/33 file after the snapshot 2022-06-27T11:49:27.015 INFO:tasks.workunit.client.0.smithi195.stdout: create 224/34 file after the snapshot 2022-06-27T11:49:27.016 INFO:tasks.workunit.client.0.smithi195.stdout: create 224/35 file after the snapshot 2022-06-27T11:49:27.017 INFO:tasks.workunit.client.0.smithi195.stdout: create 224/36 file after the snapshot 2022-06-27T11:49:27.017 INFO:tasks.workunit.client.0.smithi195.stdout: create 224/37 file after the snapshot 2022-06-27T11:49:27.018 INFO:tasks.workunit.client.0.smithi195.stdout: create 224/38 file after the snapshot 2022-06-27T11:49:27.019 INFO:tasks.workunit.client.0.smithi195.stdout: create 224/39 file after the snapshot 2022-06-27T11:49:27.020 INFO:tasks.workunit.client.0.smithi195.stdout: create 225/.snap/snap-subdir-test 2022-06-27T11:49:27.044 INFO:tasks.workunit.client.0.smithi195.stdout: create 225/30 file after the snapshot 2022-06-27T11:49:27.045 INFO:tasks.workunit.client.0.smithi195.stdout: create 225/31 file after the snapshot 2022-06-27T11:49:27.046 INFO:tasks.workunit.client.0.smithi195.stdout: create 225/32 file after the snapshot 2022-06-27T11:49:27.047 INFO:tasks.workunit.client.0.smithi195.stdout: create 225/33 file after the snapshot 2022-06-27T11:49:27.048 INFO:tasks.workunit.client.0.smithi195.stdout: create 225/34 file after the snapshot 2022-06-27T11:49:27.048 INFO:tasks.workunit.client.0.smithi195.stdout: create 225/35 file after the snapshot 2022-06-27T11:49:27.050 INFO:tasks.workunit.client.0.smithi195.stdout: create 225/36 file after the snapshot 2022-06-27T11:49:27.058 INFO:tasks.workunit.client.0.smithi195.stdout: create 225/37 file after the snapshot 2022-06-27T11:49:27.058 INFO:tasks.workunit.client.0.smithi195.stdout: create 225/38 file after the snapshot 2022-06-27T11:49:27.059 INFO:tasks.workunit.client.0.smithi195.stdout: create 225/39 file after the snapshot 2022-06-27T11:49:27.059 INFO:tasks.workunit.client.0.smithi195.stdout: create 226/.snap/snap-subdir-test 2022-06-27T11:49:27.922 INFO:tasks.workunit.client.0.smithi195.stdout: create 226/30 file after the snapshot 2022-06-27T11:49:27.937 INFO:tasks.workunit.client.0.smithi195.stdout: create 226/31 file after the snapshot 2022-06-27T11:49:27.938 INFO:tasks.workunit.client.0.smithi195.stdout: create 226/32 file after the snapshot 2022-06-27T11:49:27.939 INFO:tasks.workunit.client.0.smithi195.stdout: create 226/33 file after the snapshot 2022-06-27T11:49:27.940 INFO:tasks.workunit.client.0.smithi195.stdout: create 226/34 file after the snapshot 2022-06-27T11:49:27.941 INFO:tasks.workunit.client.0.smithi195.stdout: create 226/35 file after the snapshot 2022-06-27T11:49:27.942 INFO:tasks.workunit.client.0.smithi195.stdout: create 226/36 file after the snapshot 2022-06-27T11:49:27.943 INFO:tasks.workunit.client.0.smithi195.stdout: create 226/37 file after the snapshot 2022-06-27T11:49:27.945 INFO:tasks.workunit.client.0.smithi195.stdout: create 226/38 file after the snapshot 2022-06-27T11:49:27.946 INFO:tasks.workunit.client.0.smithi195.stdout: create 226/39 file after the snapshot 2022-06-27T11:49:27.946 INFO:tasks.workunit.client.0.smithi195.stdout: create 227/.snap/snap-subdir-test 2022-06-27T11:49:28.690 INFO:tasks.workunit.client.0.smithi195.stdout: create 227/30 file after the snapshot 2022-06-27T11:49:28.705 INFO:tasks.workunit.client.0.smithi195.stdout: create 227/31 file after the snapshot 2022-06-27T11:49:28.708 INFO:tasks.workunit.client.0.smithi195.stdout: create 227/32 file after the snapshot 2022-06-27T11:49:28.709 INFO:tasks.workunit.client.0.smithi195.stdout: create 227/33 file after the snapshot 2022-06-27T11:49:28.709 INFO:tasks.workunit.client.0.smithi195.stdout: create 227/34 file after the snapshot 2022-06-27T11:49:28.711 INFO:tasks.workunit.client.0.smithi195.stdout: create 227/35 file after the snapshot 2022-06-27T11:49:28.711 INFO:tasks.workunit.client.0.smithi195.stdout: create 227/36 file after the snapshot 2022-06-27T11:49:28.713 INFO:tasks.workunit.client.0.smithi195.stdout: create 227/37 file after the snapshot 2022-06-27T11:49:28.713 INFO:tasks.workunit.client.0.smithi195.stdout: create 227/38 file after the snapshot 2022-06-27T11:49:28.716 INFO:tasks.workunit.client.0.smithi195.stdout: create 227/39 file after the snapshot 2022-06-27T11:49:28.716 INFO:tasks.workunit.client.0.smithi195.stdout: create 228/.snap/snap-subdir-test 2022-06-27T11:49:28.744 INFO:tasks.workunit.client.0.smithi195.stdout: create 228/30 file after the snapshot 2022-06-27T11:49:28.745 INFO:tasks.workunit.client.0.smithi195.stdout: create 228/31 file after the snapshot 2022-06-27T11:49:28.746 INFO:tasks.workunit.client.0.smithi195.stdout: create 228/32 file after the snapshot 2022-06-27T11:49:28.752 INFO:tasks.workunit.client.0.smithi195.stdout: create 228/33 file after the snapshot 2022-06-27T11:49:28.753 INFO:tasks.workunit.client.0.smithi195.stdout: create 228/34 file after the snapshot 2022-06-27T11:49:28.754 INFO:tasks.workunit.client.0.smithi195.stdout: create 228/35 file after the snapshot 2022-06-27T11:49:28.754 INFO:tasks.workunit.client.0.smithi195.stdout: create 228/36 file after the snapshot 2022-06-27T11:49:28.755 INFO:tasks.workunit.client.0.smithi195.stdout: create 228/37 file after the snapshot 2022-06-27T11:49:28.755 INFO:tasks.workunit.client.0.smithi195.stdout: create 228/38 file after the snapshot 2022-06-27T11:49:28.756 INFO:tasks.workunit.client.0.smithi195.stdout: create 228/39 file after the snapshot 2022-06-27T11:49:28.756 INFO:tasks.workunit.client.0.smithi195.stdout: create 229/.snap/snap-subdir-test 2022-06-27T11:49:28.790 INFO:tasks.workunit.client.0.smithi195.stdout: create 229/30 file after the snapshot 2022-06-27T11:49:28.798 INFO:tasks.workunit.client.0.smithi195.stdout: create 229/31 file after the snapshot 2022-06-27T11:49:28.798 INFO:tasks.workunit.client.0.smithi195.stdout: create 229/32 file after the snapshot 2022-06-27T11:49:28.799 INFO:tasks.workunit.client.0.smithi195.stdout: create 229/33 file after the snapshot 2022-06-27T11:49:28.800 INFO:tasks.workunit.client.0.smithi195.stdout: create 229/34 file after the snapshot 2022-06-27T11:49:28.810 INFO:tasks.workunit.client.0.smithi195.stdout: create 229/35 file after the snapshot 2022-06-27T11:49:28.810 INFO:tasks.workunit.client.0.smithi195.stdout: create 229/36 file after the snapshot 2022-06-27T11:49:28.810 INFO:tasks.workunit.client.0.smithi195.stdout: create 229/37 file after the snapshot 2022-06-27T11:49:28.811 INFO:tasks.workunit.client.0.smithi195.stdout: create 229/38 file after the snapshot 2022-06-27T11:49:28.811 INFO:tasks.workunit.client.0.smithi195.stdout: create 229/39 file after the snapshot 2022-06-27T11:49:28.811 INFO:tasks.workunit.client.0.smithi195.stdout: create 230/.snap/snap-subdir-test 2022-06-27T11:49:28.834 INFO:tasks.workunit.client.0.smithi195.stdout: create 230/30 file after the snapshot 2022-06-27T11:49:28.850 INFO:tasks.workunit.client.0.smithi195.stdout: create 230/31 file after the snapshot 2022-06-27T11:49:28.851 INFO:tasks.workunit.client.0.smithi195.stdout: create 230/32 file after the snapshot 2022-06-27T11:49:28.852 INFO:tasks.workunit.client.0.smithi195.stdout: create 230/33 file after the snapshot 2022-06-27T11:49:28.852 INFO:tasks.workunit.client.0.smithi195.stdout: create 230/34 file after the snapshot 2022-06-27T11:49:28.853 INFO:tasks.workunit.client.0.smithi195.stdout: create 230/35 file after the snapshot 2022-06-27T11:49:28.854 INFO:tasks.workunit.client.0.smithi195.stdout: create 230/36 file after the snapshot 2022-06-27T11:49:28.855 INFO:tasks.workunit.client.0.smithi195.stdout: create 230/37 file after the snapshot 2022-06-27T11:49:28.856 INFO:tasks.workunit.client.0.smithi195.stdout: create 230/38 file after the snapshot 2022-06-27T11:49:28.856 INFO:tasks.workunit.client.0.smithi195.stdout: create 230/39 file after the snapshot 2022-06-27T11:49:28.858 INFO:tasks.workunit.client.0.smithi195.stdout: create 231/.snap/snap-subdir-test 2022-06-27T11:49:28.891 INFO:tasks.workunit.client.0.smithi195.stdout: create 231/30 file after the snapshot 2022-06-27T11:49:28.891 INFO:tasks.workunit.client.0.smithi195.stdout: create 231/31 file after the snapshot 2022-06-27T11:49:28.892 INFO:tasks.workunit.client.0.smithi195.stdout: create 231/32 file after the snapshot 2022-06-27T11:49:28.892 INFO:tasks.workunit.client.0.smithi195.stdout: create 231/33 file after the snapshot 2022-06-27T11:49:28.893 INFO:tasks.workunit.client.0.smithi195.stdout: create 231/34 file after the snapshot 2022-06-27T11:49:28.893 INFO:tasks.workunit.client.0.smithi195.stdout: create 231/35 file after the snapshot 2022-06-27T11:49:28.893 INFO:tasks.workunit.client.0.smithi195.stdout: create 231/36 file after the snapshot 2022-06-27T11:49:28.894 INFO:tasks.workunit.client.0.smithi195.stdout: create 231/37 file after the snapshot 2022-06-27T11:49:28.895 INFO:tasks.workunit.client.0.smithi195.stdout: create 231/38 file after the snapshot 2022-06-27T11:49:28.896 INFO:tasks.workunit.client.0.smithi195.stdout: create 231/39 file after the snapshot 2022-06-27T11:49:28.896 INFO:tasks.workunit.client.0.smithi195.stdout: create 232/.snap/snap-subdir-test 2022-06-27T11:49:28.926 INFO:tasks.workunit.client.0.smithi195.stdout: create 232/30 file after the snapshot 2022-06-27T11:49:28.945 INFO:tasks.workunit.client.0.smithi195.stdout: create 232/31 file after the snapshot 2022-06-27T11:49:28.946 INFO:tasks.workunit.client.0.smithi195.stdout: create 232/32 file after the snapshot 2022-06-27T11:49:28.946 INFO:tasks.workunit.client.0.smithi195.stdout: create 232/33 file after the snapshot 2022-06-27T11:49:28.947 INFO:tasks.workunit.client.0.smithi195.stdout: create 232/34 file after the snapshot 2022-06-27T11:49:28.947 INFO:tasks.workunit.client.0.smithi195.stdout: create 232/35 file after the snapshot 2022-06-27T11:49:28.948 INFO:tasks.workunit.client.0.smithi195.stdout: create 232/36 file after the snapshot 2022-06-27T11:49:28.950 INFO:tasks.workunit.client.0.smithi195.stdout: create 232/37 file after the snapshot 2022-06-27T11:49:28.950 INFO:tasks.workunit.client.0.smithi195.stdout: create 232/38 file after the snapshot 2022-06-27T11:49:28.950 INFO:tasks.workunit.client.0.smithi195.stdout: create 232/39 file after the snapshot 2022-06-27T11:49:28.951 INFO:tasks.workunit.client.0.smithi195.stdout: create 233/.snap/snap-subdir-test 2022-06-27T11:49:28.979 INFO:tasks.workunit.client.0.smithi195.stdout: create 233/30 file after the snapshot 2022-06-27T11:49:28.997 INFO:tasks.workunit.client.0.smithi195.stdout: create 233/31 file after the snapshot 2022-06-27T11:49:28.998 INFO:tasks.workunit.client.0.smithi195.stdout: create 233/32 file after the snapshot 2022-06-27T11:49:28.998 INFO:tasks.workunit.client.0.smithi195.stdout: create 233/33 file after the snapshot 2022-06-27T11:49:28.999 INFO:tasks.workunit.client.0.smithi195.stdout: create 233/34 file after the snapshot 2022-06-27T11:49:28.999 INFO:tasks.workunit.client.0.smithi195.stdout: create 233/35 file after the snapshot 2022-06-27T11:49:29.000 INFO:tasks.workunit.client.0.smithi195.stdout: create 233/36 file after the snapshot 2022-06-27T11:49:29.000 INFO:tasks.workunit.client.0.smithi195.stdout: create 233/37 file after the snapshot 2022-06-27T11:49:29.002 INFO:tasks.workunit.client.0.smithi195.stdout: create 233/38 file after the snapshot 2022-06-27T11:49:29.002 INFO:tasks.workunit.client.0.smithi195.stdout: create 233/39 file after the snapshot 2022-06-27T11:49:29.003 INFO:tasks.workunit.client.0.smithi195.stdout: create 234/.snap/snap-subdir-test 2022-06-27T11:49:29.029 INFO:tasks.workunit.client.0.smithi195.stdout: create 234/30 file after the snapshot 2022-06-27T11:49:29.030 INFO:tasks.workunit.client.0.smithi195.stdout: create 234/31 file after the snapshot 2022-06-27T11:49:29.032 INFO:tasks.workunit.client.0.smithi195.stdout: create 234/32 file after the snapshot 2022-06-27T11:49:29.033 INFO:tasks.workunit.client.0.smithi195.stdout: create 234/33 file after the snapshot 2022-06-27T11:49:29.034 INFO:tasks.workunit.client.0.smithi195.stdout: create 234/34 file after the snapshot 2022-06-27T11:49:29.036 INFO:tasks.workunit.client.0.smithi195.stdout: create 234/35 file after the snapshot 2022-06-27T11:49:29.037 INFO:tasks.workunit.client.0.smithi195.stdout: create 234/36 file after the snapshot 2022-06-27T11:49:29.038 INFO:tasks.workunit.client.0.smithi195.stdout: create 234/37 file after the snapshot 2022-06-27T11:49:29.040 INFO:tasks.workunit.client.0.smithi195.stdout: create 234/38 file after the snapshot 2022-06-27T11:49:29.041 INFO:tasks.workunit.client.0.smithi195.stdout: create 234/39 file after the snapshot 2022-06-27T11:49:29.043 INFO:tasks.workunit.client.0.smithi195.stdout: create 235/.snap/snap-subdir-test 2022-06-27T11:49:29.075 INFO:tasks.workunit.client.0.smithi195.stdout: create 235/30 file after the snapshot 2022-06-27T11:49:29.092 INFO:tasks.workunit.client.0.smithi195.stdout: create 235/31 file after the snapshot 2022-06-27T11:49:29.093 INFO:tasks.workunit.client.0.smithi195.stdout: create 235/32 file after the snapshot 2022-06-27T11:49:29.094 INFO:tasks.workunit.client.0.smithi195.stdout: create 235/33 file after the snapshot 2022-06-27T11:49:29.094 INFO:tasks.workunit.client.0.smithi195.stdout: create 235/34 file after the snapshot 2022-06-27T11:49:29.096 INFO:tasks.workunit.client.0.smithi195.stdout: create 235/35 file after the snapshot 2022-06-27T11:49:29.097 INFO:tasks.workunit.client.0.smithi195.stdout: create 235/36 file after the snapshot 2022-06-27T11:49:29.097 INFO:tasks.workunit.client.0.smithi195.stdout: create 235/37 file after the snapshot 2022-06-27T11:49:29.098 INFO:tasks.workunit.client.0.smithi195.stdout: create 235/38 file after the snapshot 2022-06-27T11:49:29.100 INFO:tasks.workunit.client.0.smithi195.stdout: create 235/39 file after the snapshot 2022-06-27T11:49:29.100 INFO:tasks.workunit.client.0.smithi195.stdout: create 236/.snap/snap-subdir-test 2022-06-27T11:49:29.127 INFO:tasks.workunit.client.0.smithi195.stdout: create 236/30 file after the snapshot 2022-06-27T11:49:29.152 INFO:tasks.workunit.client.0.smithi195.stdout: create 236/31 file after the snapshot 2022-06-27T11:49:29.153 INFO:tasks.workunit.client.0.smithi195.stdout: create 236/32 file after the snapshot 2022-06-27T11:49:29.154 INFO:tasks.workunit.client.0.smithi195.stdout: create 236/33 file after the snapshot 2022-06-27T11:49:29.155 INFO:tasks.workunit.client.0.smithi195.stdout: create 236/34 file after the snapshot 2022-06-27T11:49:29.155 INFO:tasks.workunit.client.0.smithi195.stdout: create 236/35 file after the snapshot 2022-06-27T11:49:29.156 INFO:tasks.workunit.client.0.smithi195.stdout: create 236/36 file after the snapshot 2022-06-27T11:49:29.156 INFO:tasks.workunit.client.0.smithi195.stdout: create 236/37 file after the snapshot 2022-06-27T11:49:29.157 INFO:tasks.workunit.client.0.smithi195.stdout: create 236/38 file after the snapshot 2022-06-27T11:49:29.157 INFO:tasks.workunit.client.0.smithi195.stdout: create 236/39 file after the snapshot 2022-06-27T11:49:29.158 INFO:tasks.workunit.client.0.smithi195.stdout: create 237/.snap/snap-subdir-test 2022-06-27T11:49:29.182 INFO:tasks.workunit.client.0.smithi195.stdout: create 237/30 file after the snapshot 2022-06-27T11:49:29.194 INFO:tasks.workunit.client.0.smithi195.stdout: create 237/31 file after the snapshot 2022-06-27T11:49:29.195 INFO:tasks.workunit.client.0.smithi195.stdout: create 237/32 file after the snapshot 2022-06-27T11:49:29.196 INFO:tasks.workunit.client.0.smithi195.stdout: create 237/33 file after the snapshot 2022-06-27T11:49:29.197 INFO:tasks.workunit.client.0.smithi195.stdout: create 237/34 file after the snapshot 2022-06-27T11:49:29.197 INFO:tasks.workunit.client.0.smithi195.stdout: create 237/35 file after the snapshot 2022-06-27T11:49:29.198 INFO:tasks.workunit.client.0.smithi195.stdout: create 237/36 file after the snapshot 2022-06-27T11:49:29.199 INFO:tasks.workunit.client.0.smithi195.stdout: create 237/37 file after the snapshot 2022-06-27T11:49:29.200 INFO:tasks.workunit.client.0.smithi195.stdout: create 237/38 file after the snapshot 2022-06-27T11:49:29.201 INFO:tasks.workunit.client.0.smithi195.stdout: create 237/39 file after the snapshot 2022-06-27T11:49:29.202 INFO:tasks.workunit.client.0.smithi195.stdout: create 238/.snap/snap-subdir-test 2022-06-27T11:49:29.230 INFO:tasks.workunit.client.0.smithi195.stdout: create 238/30 file after the snapshot 2022-06-27T11:49:29.232 INFO:tasks.workunit.client.0.smithi195.stdout: create 238/31 file after the snapshot 2022-06-27T11:49:29.233 INFO:tasks.workunit.client.0.smithi195.stdout: create 238/32 file after the snapshot 2022-06-27T11:49:29.233 INFO:tasks.workunit.client.0.smithi195.stdout: create 238/33 file after the snapshot 2022-06-27T11:49:29.234 INFO:tasks.workunit.client.0.smithi195.stdout: create 238/34 file after the snapshot 2022-06-27T11:49:29.235 INFO:tasks.workunit.client.0.smithi195.stdout: create 238/35 file after the snapshot 2022-06-27T11:49:29.235 INFO:tasks.workunit.client.0.smithi195.stdout: create 238/36 file after the snapshot 2022-06-27T11:49:29.236 INFO:tasks.workunit.client.0.smithi195.stdout: create 238/37 file after the snapshot 2022-06-27T11:49:29.239 INFO:tasks.workunit.client.0.smithi195.stdout: create 238/38 file after the snapshot 2022-06-27T11:49:29.239 INFO:tasks.workunit.client.0.smithi195.stdout: create 238/39 file after the snapshot 2022-06-27T11:49:29.240 INFO:tasks.workunit.client.0.smithi195.stdout: create 239/.snap/snap-subdir-test 2022-06-27T11:49:29.270 INFO:tasks.workunit.client.0.smithi195.stdout: create 239/30 file after the snapshot 2022-06-27T11:49:29.287 INFO:tasks.workunit.client.0.smithi195.stdout: create 239/31 file after the snapshot 2022-06-27T11:49:29.287 INFO:tasks.workunit.client.0.smithi195.stdout: create 239/32 file after the snapshot 2022-06-27T11:49:29.288 INFO:tasks.workunit.client.0.smithi195.stdout: create 239/33 file after the snapshot 2022-06-27T11:49:29.289 INFO:tasks.workunit.client.0.smithi195.stdout: create 239/34 file after the snapshot 2022-06-27T11:49:29.289 INFO:tasks.workunit.client.0.smithi195.stdout: create 239/35 file after the snapshot 2022-06-27T11:49:29.290 INFO:tasks.workunit.client.0.smithi195.stdout: create 239/36 file after the snapshot 2022-06-27T11:49:29.290 INFO:tasks.workunit.client.0.smithi195.stdout: create 239/37 file after the snapshot 2022-06-27T11:49:29.291 INFO:tasks.workunit.client.0.smithi195.stdout: create 239/38 file after the snapshot 2022-06-27T11:49:29.291 INFO:tasks.workunit.client.0.smithi195.stdout: create 239/39 file after the snapshot 2022-06-27T11:49:29.292 INFO:tasks.workunit.client.0.smithi195.stdout: create 240/.snap/snap-subdir-test 2022-06-27T11:49:29.324 INFO:tasks.workunit.client.0.smithi195.stdout: create 240/30 file after the snapshot 2022-06-27T11:49:29.325 INFO:tasks.workunit.client.0.smithi195.stdout: create 240/31 file after the snapshot 2022-06-27T11:49:29.331 INFO:tasks.workunit.client.0.smithi195.stdout: create 240/32 file after the snapshot 2022-06-27T11:49:29.331 INFO:tasks.workunit.client.0.smithi195.stdout: create 240/33 file after the snapshot 2022-06-27T11:49:29.332 INFO:tasks.workunit.client.0.smithi195.stdout: create 240/34 file after the snapshot 2022-06-27T11:49:29.333 INFO:tasks.workunit.client.0.smithi195.stdout: create 240/35 file after the snapshot 2022-06-27T11:49:29.333 INFO:tasks.workunit.client.0.smithi195.stdout: create 240/36 file after the snapshot 2022-06-27T11:49:29.334 INFO:tasks.workunit.client.0.smithi195.stdout: create 240/37 file after the snapshot 2022-06-27T11:49:29.334 INFO:tasks.workunit.client.0.smithi195.stdout: create 240/38 file after the snapshot 2022-06-27T11:49:29.335 INFO:tasks.workunit.client.0.smithi195.stdout: create 240/39 file after the snapshot 2022-06-27T11:49:29.336 INFO:tasks.workunit.client.0.smithi195.stdout: create 241/.snap/snap-subdir-test 2022-06-27T11:49:29.361 INFO:tasks.workunit.client.0.smithi195.stdout: create 241/30 file after the snapshot 2022-06-27T11:49:29.377 INFO:tasks.workunit.client.0.smithi195.stdout: create 241/31 file after the snapshot 2022-06-27T11:49:29.378 INFO:tasks.workunit.client.0.smithi195.stdout: create 241/32 file after the snapshot 2022-06-27T11:49:29.379 INFO:tasks.workunit.client.0.smithi195.stdout: create 241/33 file after the snapshot 2022-06-27T11:49:29.380 INFO:tasks.workunit.client.0.smithi195.stdout: create 241/34 file after the snapshot 2022-06-27T11:49:29.382 INFO:tasks.workunit.client.0.smithi195.stdout: create 241/35 file after the snapshot 2022-06-27T11:49:29.382 INFO:tasks.workunit.client.0.smithi195.stdout: create 241/36 file after the snapshot 2022-06-27T11:49:29.384 INFO:tasks.workunit.client.0.smithi195.stdout: create 241/37 file after the snapshot 2022-06-27T11:49:29.384 INFO:tasks.workunit.client.0.smithi195.stdout: create 241/38 file after the snapshot 2022-06-27T11:49:29.385 INFO:tasks.workunit.client.0.smithi195.stdout: create 241/39 file after the snapshot 2022-06-27T11:49:29.386 INFO:tasks.workunit.client.0.smithi195.stdout: create 242/.snap/snap-subdir-test 2022-06-27T11:49:29.413 INFO:tasks.workunit.client.0.smithi195.stdout: create 242/30 file after the snapshot 2022-06-27T11:49:29.426 INFO:tasks.workunit.client.0.smithi195.stdout: create 242/31 file after the snapshot 2022-06-27T11:49:29.427 INFO:tasks.workunit.client.0.smithi195.stdout: create 242/32 file after the snapshot 2022-06-27T11:49:29.428 INFO:tasks.workunit.client.0.smithi195.stdout: create 242/33 file after the snapshot 2022-06-27T11:49:29.429 INFO:tasks.workunit.client.0.smithi195.stdout: create 242/34 file after the snapshot 2022-06-27T11:49:29.430 INFO:tasks.workunit.client.0.smithi195.stdout: create 242/35 file after the snapshot 2022-06-27T11:49:29.431 INFO:tasks.workunit.client.0.smithi195.stdout: create 242/36 file after the snapshot 2022-06-27T11:49:29.432 INFO:tasks.workunit.client.0.smithi195.stdout: create 242/37 file after the snapshot 2022-06-27T11:49:29.433 INFO:tasks.workunit.client.0.smithi195.stdout: create 242/38 file after the snapshot 2022-06-27T11:49:29.434 INFO:tasks.workunit.client.0.smithi195.stdout: create 242/39 file after the snapshot 2022-06-27T11:49:29.434 INFO:tasks.workunit.client.0.smithi195.stdout: create 243/.snap/snap-subdir-test 2022-06-27T11:49:29.460 INFO:tasks.workunit.client.0.smithi195.stdout: create 243/30 file after the snapshot 2022-06-27T11:49:29.461 INFO:tasks.workunit.client.0.smithi195.stdout: create 243/31 file after the snapshot 2022-06-27T11:49:29.462 INFO:tasks.workunit.client.0.smithi195.stdout: create 243/32 file after the snapshot 2022-06-27T11:49:29.466 INFO:tasks.workunit.client.0.smithi195.stdout: create 243/33 file after the snapshot 2022-06-27T11:49:29.466 INFO:tasks.workunit.client.0.smithi195.stdout: create 243/34 file after the snapshot 2022-06-27T11:49:29.467 INFO:tasks.workunit.client.0.smithi195.stdout: create 243/35 file after the snapshot 2022-06-27T11:49:29.467 INFO:tasks.workunit.client.0.smithi195.stdout: create 243/36 file after the snapshot 2022-06-27T11:49:29.470 INFO:tasks.workunit.client.0.smithi195.stdout: create 243/37 file after the snapshot 2022-06-27T11:49:29.470 INFO:tasks.workunit.client.0.smithi195.stdout: create 243/38 file after the snapshot 2022-06-27T11:49:29.471 INFO:tasks.workunit.client.0.smithi195.stdout: create 243/39 file after the snapshot 2022-06-27T11:49:29.471 INFO:tasks.workunit.client.0.smithi195.stdout: create 244/.snap/snap-subdir-test 2022-06-27T11:49:29.499 INFO:tasks.workunit.client.0.smithi195.stdout: create 244/30 file after the snapshot 2022-06-27T11:49:29.500 INFO:tasks.workunit.client.0.smithi195.stdout: create 244/31 file after the snapshot 2022-06-27T11:49:29.501 INFO:tasks.workunit.client.0.smithi195.stdout: create 244/32 file after the snapshot 2022-06-27T11:49:29.502 INFO:tasks.workunit.client.0.smithi195.stdout: create 244/33 file after the snapshot 2022-06-27T11:49:29.503 INFO:tasks.workunit.client.0.smithi195.stdout: create 244/34 file after the snapshot 2022-06-27T11:49:29.504 INFO:tasks.workunit.client.0.smithi195.stdout: create 244/35 file after the snapshot 2022-06-27T11:49:29.505 INFO:tasks.workunit.client.0.smithi195.stdout: create 244/36 file after the snapshot 2022-06-27T11:49:29.506 INFO:tasks.workunit.client.0.smithi195.stdout: create 244/37 file after the snapshot 2022-06-27T11:49:29.507 INFO:tasks.workunit.client.0.smithi195.stdout: create 244/38 file after the snapshot 2022-06-27T11:49:29.507 INFO:tasks.workunit.client.0.smithi195.stdout: create 244/39 file after the snapshot 2022-06-27T11:49:29.508 INFO:tasks.workunit.client.0.smithi195.stdout: create 245/.snap/snap-subdir-test 2022-06-27T11:49:29.538 INFO:tasks.workunit.client.0.smithi195.stdout: create 245/30 file after the snapshot 2022-06-27T11:49:29.552 INFO:tasks.workunit.client.0.smithi195.stdout: create 245/31 file after the snapshot 2022-06-27T11:49:29.552 INFO:tasks.workunit.client.0.smithi195.stdout: create 245/32 file after the snapshot 2022-06-27T11:49:29.553 INFO:tasks.workunit.client.0.smithi195.stdout: create 245/33 file after the snapshot 2022-06-27T11:49:29.554 INFO:tasks.workunit.client.0.smithi195.stdout: create 245/34 file after the snapshot 2022-06-27T11:49:29.556 INFO:tasks.workunit.client.0.smithi195.stdout: create 245/35 file after the snapshot 2022-06-27T11:49:29.557 INFO:tasks.workunit.client.0.smithi195.stdout: create 245/36 file after the snapshot 2022-06-27T11:49:29.558 INFO:tasks.workunit.client.0.smithi195.stdout: create 245/37 file after the snapshot 2022-06-27T11:49:29.559 INFO:tasks.workunit.client.0.smithi195.stdout: create 245/38 file after the snapshot 2022-06-27T11:49:29.560 INFO:tasks.workunit.client.0.smithi195.stdout: create 245/39 file after the snapshot 2022-06-27T11:49:29.560 INFO:tasks.workunit.client.0.smithi195.stdout: create 246/.snap/snap-subdir-test 2022-06-27T11:49:29.588 INFO:tasks.workunit.client.0.smithi195.stdout: create 246/30 file after the snapshot 2022-06-27T11:49:29.603 INFO:tasks.workunit.client.0.smithi195.stdout: create 246/31 file after the snapshot 2022-06-27T11:49:29.604 INFO:tasks.workunit.client.0.smithi195.stdout: create 246/32 file after the snapshot 2022-06-27T11:49:29.605 INFO:tasks.workunit.client.0.smithi195.stdout: create 246/33 file after the snapshot 2022-06-27T11:49:29.606 INFO:tasks.workunit.client.0.smithi195.stdout: create 246/34 file after the snapshot 2022-06-27T11:49:29.607 INFO:tasks.workunit.client.0.smithi195.stdout: create 246/35 file after the snapshot 2022-06-27T11:49:29.610 INFO:tasks.workunit.client.0.smithi195.stdout: create 246/36 file after the snapshot 2022-06-27T11:49:29.611 INFO:tasks.workunit.client.0.smithi195.stdout: create 246/37 file after the snapshot 2022-06-27T11:49:29.611 INFO:tasks.workunit.client.0.smithi195.stdout: create 246/38 file after the snapshot 2022-06-27T11:49:29.612 INFO:tasks.workunit.client.0.smithi195.stdout: create 246/39 file after the snapshot 2022-06-27T11:49:29.613 INFO:tasks.workunit.client.0.smithi195.stdout: create 247/.snap/snap-subdir-test 2022-06-27T11:49:29.639 INFO:tasks.workunit.client.0.smithi195.stdout: create 247/30 file after the snapshot 2022-06-27T11:49:29.640 INFO:tasks.workunit.client.0.smithi195.stdout: create 247/31 file after the snapshot 2022-06-27T11:49:29.641 INFO:tasks.workunit.client.0.smithi195.stdout: create 247/32 file after the snapshot 2022-06-27T11:49:29.643 INFO:tasks.workunit.client.0.smithi195.stdout: create 247/33 file after the snapshot 2022-06-27T11:49:29.645 INFO:tasks.workunit.client.0.smithi195.stdout: create 247/34 file after the snapshot 2022-06-27T11:49:29.646 INFO:tasks.workunit.client.0.smithi195.stdout: create 247/35 file after the snapshot 2022-06-27T11:49:29.647 INFO:tasks.workunit.client.0.smithi195.stdout: create 247/36 file after the snapshot 2022-06-27T11:49:29.649 INFO:tasks.workunit.client.0.smithi195.stdout: create 247/37 file after the snapshot 2022-06-27T11:49:29.650 INFO:tasks.workunit.client.0.smithi195.stdout: create 247/38 file after the snapshot 2022-06-27T11:49:29.651 INFO:tasks.workunit.client.0.smithi195.stdout: create 247/39 file after the snapshot 2022-06-27T11:49:29.652 INFO:tasks.workunit.client.0.smithi195.stdout: create 248/.snap/snap-subdir-test 2022-06-27T11:49:29.680 INFO:tasks.workunit.client.0.smithi195.stdout: create 248/30 file after the snapshot 2022-06-27T11:49:29.696 INFO:tasks.workunit.client.0.smithi195.stdout: create 248/31 file after the snapshot 2022-06-27T11:49:29.697 INFO:tasks.workunit.client.0.smithi195.stdout: create 248/32 file after the snapshot 2022-06-27T11:49:29.698 INFO:tasks.workunit.client.0.smithi195.stdout: create 248/33 file after the snapshot 2022-06-27T11:49:29.699 INFO:tasks.workunit.client.0.smithi195.stdout: create 248/34 file after the snapshot 2022-06-27T11:49:29.700 INFO:tasks.workunit.client.0.smithi195.stdout: create 248/35 file after the snapshot 2022-06-27T11:49:29.701 INFO:tasks.workunit.client.0.smithi195.stdout: create 248/36 file after the snapshot 2022-06-27T11:49:29.702 INFO:tasks.workunit.client.0.smithi195.stdout: create 248/37 file after the snapshot 2022-06-27T11:49:29.702 INFO:tasks.workunit.client.0.smithi195.stdout: create 248/38 file after the snapshot 2022-06-27T11:49:29.704 INFO:tasks.workunit.client.0.smithi195.stdout: create 248/39 file after the snapshot 2022-06-27T11:49:29.704 INFO:tasks.workunit.client.0.smithi195.stdout: create 249/.snap/snap-subdir-test 2022-06-27T11:49:29.732 INFO:tasks.workunit.client.0.smithi195.stdout: create 249/30 file after the snapshot 2022-06-27T11:49:29.747 INFO:tasks.workunit.client.0.smithi195.stdout: create 249/31 file after the snapshot 2022-06-27T11:49:29.748 INFO:tasks.workunit.client.0.smithi195.stdout: create 249/32 file after the snapshot 2022-06-27T11:49:29.749 INFO:tasks.workunit.client.0.smithi195.stdout: create 249/33 file after the snapshot 2022-06-27T11:49:29.750 INFO:tasks.workunit.client.0.smithi195.stdout: create 249/34 file after the snapshot 2022-06-27T11:49:29.751 INFO:tasks.workunit.client.0.smithi195.stdout: create 249/35 file after the snapshot 2022-06-27T11:49:29.752 INFO:tasks.workunit.client.0.smithi195.stdout: create 249/36 file after the snapshot 2022-06-27T11:49:29.753 INFO:tasks.workunit.client.0.smithi195.stdout: create 249/37 file after the snapshot 2022-06-27T11:49:29.754 INFO:tasks.workunit.client.0.smithi195.stdout: create 249/38 file after the snapshot 2022-06-27T11:49:29.754 INFO:tasks.workunit.client.0.smithi195.stdout: create 249/39 file after the snapshot 2022-06-27T11:49:29.755 INFO:tasks.workunit.client.0.smithi195.stdout: create 250/.snap/snap-subdir-test 2022-06-27T11:49:29.782 INFO:tasks.workunit.client.0.smithi195.stdout: create 250/30 file after the snapshot 2022-06-27T11:49:29.796 INFO:tasks.workunit.client.0.smithi195.stdout: create 250/31 file after the snapshot 2022-06-27T11:49:29.798 INFO:tasks.workunit.client.0.smithi195.stdout: create 250/32 file after the snapshot 2022-06-27T11:49:29.799 INFO:tasks.workunit.client.0.smithi195.stdout: create 250/33 file after the snapshot 2022-06-27T11:49:29.800 INFO:tasks.workunit.client.0.smithi195.stdout: create 250/34 file after the snapshot 2022-06-27T11:49:29.800 INFO:tasks.workunit.client.0.smithi195.stdout: create 250/35 file after the snapshot 2022-06-27T11:49:29.802 INFO:tasks.workunit.client.0.smithi195.stdout: create 250/36 file after the snapshot 2022-06-27T11:49:29.802 INFO:tasks.workunit.client.0.smithi195.stdout: create 250/37 file after the snapshot 2022-06-27T11:49:29.803 INFO:tasks.workunit.client.0.smithi195.stdout: create 250/38 file after the snapshot 2022-06-27T11:49:29.804 INFO:tasks.workunit.client.0.smithi195.stdout: create 250/39 file after the snapshot 2022-06-27T11:49:29.804 INFO:tasks.workunit.client.0.smithi195.stdout: create 251/.snap/snap-subdir-test 2022-06-27T11:49:29.829 INFO:tasks.workunit.client.0.smithi195.stdout: create 251/30 file after the snapshot 2022-06-27T11:49:29.844 INFO:tasks.workunit.client.0.smithi195.stdout: create 251/31 file after the snapshot 2022-06-27T11:49:29.845 INFO:tasks.workunit.client.0.smithi195.stdout: create 251/32 file after the snapshot 2022-06-27T11:49:29.846 INFO:tasks.workunit.client.0.smithi195.stdout: create 251/33 file after the snapshot 2022-06-27T11:49:29.847 INFO:tasks.workunit.client.0.smithi195.stdout: create 251/34 file after the snapshot 2022-06-27T11:49:29.848 INFO:tasks.workunit.client.0.smithi195.stdout: create 251/35 file after the snapshot 2022-06-27T11:49:29.848 INFO:tasks.workunit.client.0.smithi195.stdout: create 251/36 file after the snapshot 2022-06-27T11:49:29.850 INFO:tasks.workunit.client.0.smithi195.stdout: create 251/37 file after the snapshot 2022-06-27T11:49:29.856 INFO:tasks.workunit.client.0.smithi195.stdout: create 251/38 file after the snapshot 2022-06-27T11:49:29.856 INFO:tasks.workunit.client.0.smithi195.stdout: create 251/39 file after the snapshot 2022-06-27T11:49:29.856 INFO:tasks.workunit.client.0.smithi195.stdout: create 252/.snap/snap-subdir-test 2022-06-27T11:49:29.879 INFO:tasks.workunit.client.0.smithi195.stdout: create 252/30 file after the snapshot 2022-06-27T11:49:29.880 INFO:tasks.workunit.client.0.smithi195.stdout: create 252/31 file after the snapshot 2022-06-27T11:49:29.880 INFO:tasks.workunit.client.0.smithi195.stdout: create 252/32 file after the snapshot 2022-06-27T11:49:29.882 INFO:tasks.workunit.client.0.smithi195.stdout: create 252/33 file after the snapshot 2022-06-27T11:49:29.884 INFO:tasks.workunit.client.0.smithi195.stdout: create 252/34 file after the snapshot 2022-06-27T11:49:29.884 INFO:tasks.workunit.client.0.smithi195.stdout: create 252/35 file after the snapshot 2022-06-27T11:49:29.885 INFO:tasks.workunit.client.0.smithi195.stdout: create 252/36 file after the snapshot 2022-06-27T11:49:29.885 INFO:tasks.workunit.client.0.smithi195.stdout: create 252/37 file after the snapshot 2022-06-27T11:49:29.886 INFO:tasks.workunit.client.0.smithi195.stdout: create 252/38 file after the snapshot 2022-06-27T11:49:29.886 INFO:tasks.workunit.client.0.smithi195.stdout: create 252/39 file after the snapshot 2022-06-27T11:49:29.887 INFO:tasks.workunit.client.0.smithi195.stdout: create 253/.snap/snap-subdir-test 2022-06-27T11:49:29.922 INFO:tasks.workunit.client.0.smithi195.stdout: create 253/30 file after the snapshot 2022-06-27T11:49:29.934 INFO:tasks.workunit.client.0.smithi195.stdout: create 253/31 file after the snapshot 2022-06-27T11:49:29.935 INFO:tasks.workunit.client.0.smithi195.stdout: create 253/32 file after the snapshot 2022-06-27T11:49:29.936 INFO:tasks.workunit.client.0.smithi195.stdout: create 253/33 file after the snapshot 2022-06-27T11:49:29.937 INFO:tasks.workunit.client.0.smithi195.stdout: create 253/34 file after the snapshot 2022-06-27T11:49:29.938 INFO:tasks.workunit.client.0.smithi195.stdout: create 253/35 file after the snapshot 2022-06-27T11:49:29.939 INFO:tasks.workunit.client.0.smithi195.stdout: create 253/36 file after the snapshot 2022-06-27T11:49:29.939 INFO:tasks.workunit.client.0.smithi195.stdout: create 253/37 file after the snapshot 2022-06-27T11:49:29.941 INFO:tasks.workunit.client.0.smithi195.stdout: create 253/38 file after the snapshot 2022-06-27T11:49:29.941 INFO:tasks.workunit.client.0.smithi195.stdout: create 253/39 file after the snapshot 2022-06-27T11:49:29.942 INFO:tasks.workunit.client.0.smithi195.stdout: create 254/.snap/snap-subdir-test 2022-06-27T11:49:29.972 INFO:tasks.workunit.client.0.smithi195.stdout: create 254/30 file after the snapshot 2022-06-27T11:49:29.988 INFO:tasks.workunit.client.0.smithi195.stdout: create 254/31 file after the snapshot 2022-06-27T11:49:29.989 INFO:tasks.workunit.client.0.smithi195.stdout: create 254/32 file after the snapshot 2022-06-27T11:49:29.990 INFO:tasks.workunit.client.0.smithi195.stdout: create 254/33 file after the snapshot 2022-06-27T11:49:29.990 INFO:tasks.workunit.client.0.smithi195.stdout: create 254/34 file after the snapshot 2022-06-27T11:49:29.991 INFO:tasks.workunit.client.0.smithi195.stdout: create 254/35 file after the snapshot 2022-06-27T11:49:29.992 INFO:tasks.workunit.client.0.smithi195.stdout: create 254/36 file after the snapshot 2022-06-27T11:49:29.993 INFO:tasks.workunit.client.0.smithi195.stdout: create 254/37 file after the snapshot 2022-06-27T11:49:29.994 INFO:tasks.workunit.client.0.smithi195.stdout: create 254/38 file after the snapshot 2022-06-27T11:49:29.994 INFO:tasks.workunit.client.0.smithi195.stdout: create 254/39 file after the snapshot 2022-06-27T11:49:29.995 INFO:tasks.workunit.client.0.smithi195.stdout: create 255/.snap/snap-subdir-test 2022-06-27T11:49:30.108 INFO:tasks.workunit.client.0.smithi195.stdout: create 255/30 file after the snapshot 2022-06-27T11:49:30.123 INFO:tasks.workunit.client.0.smithi195.stdout: create 255/31 file after the snapshot 2022-06-27T11:49:30.124 INFO:tasks.workunit.client.0.smithi195.stdout: create 255/32 file after the snapshot 2022-06-27T11:49:30.124 INFO:tasks.workunit.client.0.smithi195.stdout: create 255/33 file after the snapshot 2022-06-27T11:49:30.125 INFO:tasks.workunit.client.0.smithi195.stdout: create 255/34 file after the snapshot 2022-06-27T11:49:30.126 INFO:tasks.workunit.client.0.smithi195.stdout: create 255/35 file after the snapshot 2022-06-27T11:49:30.127 INFO:tasks.workunit.client.0.smithi195.stdout: create 255/36 file after the snapshot 2022-06-27T11:49:30.128 INFO:tasks.workunit.client.0.smithi195.stdout: create 255/37 file after the snapshot 2022-06-27T11:49:30.129 INFO:tasks.workunit.client.0.smithi195.stdout: create 255/38 file after the snapshot 2022-06-27T11:49:30.130 INFO:tasks.workunit.client.0.smithi195.stdout: create 255/39 file after the snapshot 2022-06-27T11:49:30.131 INFO:tasks.workunit.client.0.smithi195.stdout: create 256/.snap/snap-subdir-test 2022-06-27T11:49:30.161 INFO:tasks.workunit.client.0.smithi195.stdout: create 256/30 file after the snapshot 2022-06-27T11:49:30.175 INFO:tasks.workunit.client.0.smithi195.stdout: create 256/31 file after the snapshot 2022-06-27T11:49:30.175 INFO:tasks.workunit.client.0.smithi195.stdout: create 256/32 file after the snapshot 2022-06-27T11:49:30.176 INFO:tasks.workunit.client.0.smithi195.stdout: create 256/33 file after the snapshot 2022-06-27T11:49:30.178 INFO:tasks.workunit.client.0.smithi195.stdout: create 256/34 file after the snapshot 2022-06-27T11:49:31.115 INFO:tasks.workunit.client.0.smithi195.stdout: create 256/35 file after the snapshot 2022-06-27T11:49:31.160 INFO:tasks.workunit.client.0.smithi195.stdout: create 256/36 file after the snapshot 2022-06-27T11:49:31.161 INFO:tasks.workunit.client.0.smithi195.stdout: create 256/37 file after the snapshot 2022-06-27T11:49:31.162 INFO:tasks.workunit.client.0.smithi195.stdout: create 256/38 file after the snapshot 2022-06-27T11:49:31.163 INFO:tasks.workunit.client.0.smithi195.stdout: create 256/39 file after the snapshot 2022-06-27T11:49:31.164 INFO:tasks.workunit.client.0.smithi195.stdout: create 257/.snap/snap-subdir-test 2022-06-27T11:49:31.206 INFO:tasks.workunit.client.0.smithi195.stdout: create 257/30 file after the snapshot 2022-06-27T11:49:31.221 INFO:tasks.workunit.client.0.smithi195.stdout: create 257/31 file after the snapshot 2022-06-27T11:49:31.222 INFO:tasks.workunit.client.0.smithi195.stdout: create 257/32 file after the snapshot 2022-06-27T11:49:31.223 INFO:tasks.workunit.client.0.smithi195.stdout: create 257/33 file after the snapshot 2022-06-27T11:49:31.224 INFO:tasks.workunit.client.0.smithi195.stdout: create 257/34 file after the snapshot 2022-06-27T11:49:31.225 INFO:tasks.workunit.client.0.smithi195.stdout: create 257/35 file after the snapshot 2022-06-27T11:49:31.226 INFO:tasks.workunit.client.0.smithi195.stdout: create 257/36 file after the snapshot 2022-06-27T11:49:31.226 INFO:tasks.workunit.client.0.smithi195.stdout: create 257/37 file after the snapshot 2022-06-27T11:49:31.227 INFO:tasks.workunit.client.0.smithi195.stdout: create 257/38 file after the snapshot 2022-06-27T11:49:31.228 INFO:tasks.workunit.client.0.smithi195.stdout: create 257/39 file after the snapshot 2022-06-27T11:49:31.230 INFO:tasks.workunit.client.0.smithi195.stdout: create 258/.snap/snap-subdir-test 2022-06-27T11:49:31.267 INFO:tasks.workunit.client.0.smithi195.stdout: create 258/30 file after the snapshot 2022-06-27T11:49:31.281 INFO:tasks.workunit.client.0.smithi195.stdout: create 258/31 file after the snapshot 2022-06-27T11:49:31.282 INFO:tasks.workunit.client.0.smithi195.stdout: create 258/32 file after the snapshot 2022-06-27T11:49:31.283 INFO:tasks.workunit.client.0.smithi195.stdout: create 258/33 file after the snapshot 2022-06-27T11:49:31.287 INFO:tasks.workunit.client.0.smithi195.stdout: create 258/34 file after the snapshot 2022-06-27T11:49:31.288 INFO:tasks.workunit.client.0.smithi195.stdout: create 258/35 file after the snapshot 2022-06-27T11:49:31.288 INFO:tasks.workunit.client.0.smithi195.stdout: create 258/36 file after the snapshot 2022-06-27T11:49:31.288 INFO:tasks.workunit.client.0.smithi195.stdout: create 258/37 file after the snapshot 2022-06-27T11:49:31.289 INFO:tasks.workunit.client.0.smithi195.stdout: create 258/38 file after the snapshot 2022-06-27T11:49:31.289 INFO:tasks.workunit.client.0.smithi195.stdout: create 258/39 file after the snapshot 2022-06-27T11:49:31.290 INFO:tasks.workunit.client.0.smithi195.stdout: create 259/.snap/snap-subdir-test 2022-06-27T11:49:31.424 INFO:tasks.workunit.client.0.smithi195.stdout: create 259/30 file after the snapshot 2022-06-27T11:49:31.440 INFO:tasks.workunit.client.0.smithi195.stdout: create 259/31 file after the snapshot 2022-06-27T11:49:31.441 INFO:tasks.workunit.client.0.smithi195.stdout: create 259/32 file after the snapshot 2022-06-27T11:49:31.442 INFO:tasks.workunit.client.0.smithi195.stdout: create 259/33 file after the snapshot 2022-06-27T11:49:31.443 INFO:tasks.workunit.client.0.smithi195.stdout: create 259/34 file after the snapshot 2022-06-27T11:49:31.446 INFO:tasks.workunit.client.0.smithi195.stdout: create 259/35 file after the snapshot 2022-06-27T11:49:31.446 INFO:tasks.workunit.client.0.smithi195.stdout: create 259/36 file after the snapshot 2022-06-27T11:49:31.447 INFO:tasks.workunit.client.0.smithi195.stdout: create 259/37 file after the snapshot 2022-06-27T11:49:31.448 INFO:tasks.workunit.client.0.smithi195.stdout: create 259/38 file after the snapshot 2022-06-27T11:49:31.448 INFO:tasks.workunit.client.0.smithi195.stdout: create 259/39 file after the snapshot 2022-06-27T11:49:31.449 INFO:tasks.workunit.client.0.smithi195.stdout: create 260/.snap/snap-subdir-test 2022-06-27T11:49:31.487 INFO:tasks.workunit.client.0.smithi195.stdout: create 260/30 file after the snapshot 2022-06-27T11:49:31.500 INFO:tasks.workunit.client.0.smithi195.stdout: create 260/31 file after the snapshot 2022-06-27T11:49:31.513 INFO:tasks.workunit.client.0.smithi195.stdout: create 260/32 file after the snapshot 2022-06-27T11:49:31.514 INFO:tasks.workunit.client.0.smithi195.stdout: create 260/33 file after the snapshot 2022-06-27T11:49:31.515 INFO:tasks.workunit.client.0.smithi195.stdout: create 260/34 file after the snapshot 2022-06-27T11:49:31.515 INFO:tasks.workunit.client.0.smithi195.stdout: create 260/35 file after the snapshot 2022-06-27T11:49:31.516 INFO:tasks.workunit.client.0.smithi195.stdout: create 260/36 file after the snapshot 2022-06-27T11:49:31.516 INFO:tasks.workunit.client.0.smithi195.stdout: create 260/37 file after the snapshot 2022-06-27T11:49:31.517 INFO:tasks.workunit.client.0.smithi195.stdout: create 260/38 file after the snapshot 2022-06-27T11:49:31.517 INFO:tasks.workunit.client.0.smithi195.stdout: create 260/39 file after the snapshot 2022-06-27T11:49:31.518 INFO:tasks.workunit.client.0.smithi195.stdout: create 261/.snap/snap-subdir-test 2022-06-27T11:49:31.544 INFO:tasks.workunit.client.0.smithi195.stdout: create 261/30 file after the snapshot 2022-06-27T11:49:31.559 INFO:tasks.workunit.client.0.smithi195.stdout: create 261/31 file after the snapshot 2022-06-27T11:49:31.561 INFO:tasks.workunit.client.0.smithi195.stdout: create 261/32 file after the snapshot 2022-06-27T11:49:31.562 INFO:tasks.workunit.client.0.smithi195.stdout: create 261/33 file after the snapshot 2022-06-27T11:49:31.563 INFO:tasks.workunit.client.0.smithi195.stdout: create 261/34 file after the snapshot 2022-06-27T11:49:31.563 INFO:tasks.workunit.client.0.smithi195.stdout: create 261/35 file after the snapshot 2022-06-27T11:49:31.564 INFO:tasks.workunit.client.0.smithi195.stdout: create 261/36 file after the snapshot 2022-06-27T11:49:31.570 INFO:tasks.workunit.client.0.smithi195.stdout: create 261/37 file after the snapshot 2022-06-27T11:49:31.571 INFO:tasks.workunit.client.0.smithi195.stdout: create 261/38 file after the snapshot 2022-06-27T11:49:31.571 INFO:tasks.workunit.client.0.smithi195.stdout: create 261/39 file after the snapshot 2022-06-27T11:49:31.571 INFO:tasks.workunit.client.0.smithi195.stdout: create 262/.snap/snap-subdir-test 2022-06-27T11:49:31.602 INFO:tasks.workunit.client.0.smithi195.stdout: create 262/30 file after the snapshot 2022-06-27T11:49:31.603 INFO:tasks.workunit.client.0.smithi195.stdout: create 262/31 file after the snapshot 2022-06-27T11:49:31.604 INFO:tasks.workunit.client.0.smithi195.stdout: create 262/32 file after the snapshot 2022-06-27T11:49:31.605 INFO:tasks.workunit.client.0.smithi195.stdout: create 262/33 file after the snapshot 2022-06-27T11:49:31.606 INFO:tasks.workunit.client.0.smithi195.stdout: create 262/34 file after the snapshot 2022-06-27T11:49:31.607 INFO:tasks.workunit.client.0.smithi195.stdout: create 262/35 file after the snapshot 2022-06-27T11:49:31.608 INFO:tasks.workunit.client.0.smithi195.stdout: create 262/36 file after the snapshot 2022-06-27T11:49:31.610 INFO:tasks.workunit.client.0.smithi195.stdout: create 262/37 file after the snapshot 2022-06-27T11:49:31.612 INFO:tasks.workunit.client.0.smithi195.stdout: create 262/38 file after the snapshot 2022-06-27T11:49:31.613 INFO:tasks.workunit.client.0.smithi195.stdout: create 262/39 file after the snapshot 2022-06-27T11:49:31.613 INFO:tasks.workunit.client.0.smithi195.stdout: create 263/.snap/snap-subdir-test 2022-06-27T11:49:31.647 INFO:tasks.workunit.client.0.smithi195.stdout: create 263/30 file after the snapshot 2022-06-27T11:49:31.665 INFO:tasks.workunit.client.0.smithi195.stdout: create 263/31 file after the snapshot 2022-06-27T11:49:31.666 INFO:tasks.workunit.client.0.smithi195.stdout: create 263/32 file after the snapshot 2022-06-27T11:49:31.666 INFO:tasks.workunit.client.0.smithi195.stdout: create 263/33 file after the snapshot 2022-06-27T11:49:31.667 INFO:tasks.workunit.client.0.smithi195.stdout: create 263/34 file after the snapshot 2022-06-27T11:49:31.668 INFO:tasks.workunit.client.0.smithi195.stdout: create 263/35 file after the snapshot 2022-06-27T11:49:31.668 INFO:tasks.workunit.client.0.smithi195.stdout: create 263/36 file after the snapshot 2022-06-27T11:49:31.668 INFO:tasks.workunit.client.0.smithi195.stdout: create 263/37 file after the snapshot 2022-06-27T11:49:31.669 INFO:tasks.workunit.client.0.smithi195.stdout: create 263/38 file after the snapshot 2022-06-27T11:49:31.670 INFO:tasks.workunit.client.0.smithi195.stdout: create 263/39 file after the snapshot 2022-06-27T11:49:31.671 INFO:tasks.workunit.client.0.smithi195.stdout: create 264/.snap/snap-subdir-test 2022-06-27T11:49:31.709 INFO:tasks.workunit.client.0.smithi195.stdout: create 264/30 file after the snapshot 2022-06-27T11:49:31.719 INFO:tasks.workunit.client.0.smithi195.stdout: create 264/31 file after the snapshot 2022-06-27T11:49:31.720 INFO:tasks.workunit.client.0.smithi195.stdout: create 264/32 file after the snapshot 2022-06-27T11:49:31.725 INFO:tasks.workunit.client.0.smithi195.stdout: create 264/33 file after the snapshot 2022-06-27T11:49:31.725 INFO:tasks.workunit.client.0.smithi195.stdout: create 264/34 file after the snapshot 2022-06-27T11:49:31.725 INFO:tasks.workunit.client.0.smithi195.stdout: create 264/35 file after the snapshot 2022-06-27T11:49:31.726 INFO:tasks.workunit.client.0.smithi195.stdout: create 264/36 file after the snapshot 2022-06-27T11:49:31.726 INFO:tasks.workunit.client.0.smithi195.stdout: create 264/37 file after the snapshot 2022-06-27T11:49:31.727 INFO:tasks.workunit.client.0.smithi195.stdout: create 264/38 file after the snapshot 2022-06-27T11:49:31.728 INFO:tasks.workunit.client.0.smithi195.stdout: create 264/39 file after the snapshot 2022-06-27T11:49:31.729 INFO:tasks.workunit.client.0.smithi195.stdout: create 265/.snap/snap-subdir-test 2022-06-27T11:49:32.043 INFO:tasks.workunit.client.0.smithi195.stdout: create 265/30 file after the snapshot 2022-06-27T11:49:32.059 INFO:tasks.workunit.client.0.smithi195.stdout: create 265/31 file after the snapshot 2022-06-27T11:49:32.059 INFO:tasks.workunit.client.0.smithi195.stdout: create 265/32 file after the snapshot 2022-06-27T11:49:32.060 INFO:tasks.workunit.client.0.smithi195.stdout: create 265/33 file after the snapshot 2022-06-27T11:49:32.061 INFO:tasks.workunit.client.0.smithi195.stdout: create 265/34 file after the snapshot 2022-06-27T11:49:32.062 INFO:tasks.workunit.client.0.smithi195.stdout: create 265/35 file after the snapshot 2022-06-27T11:49:32.063 INFO:tasks.workunit.client.0.smithi195.stdout: create 265/36 file after the snapshot 2022-06-27T11:49:32.064 INFO:tasks.workunit.client.0.smithi195.stdout: create 265/37 file after the snapshot 2022-06-27T11:49:32.065 INFO:tasks.workunit.client.0.smithi195.stdout: create 265/38 file after the snapshot 2022-06-27T11:49:32.065 INFO:tasks.workunit.client.0.smithi195.stdout: create 265/39 file after the snapshot 2022-06-27T11:49:32.066 INFO:tasks.workunit.client.0.smithi195.stdout: create 266/.snap/snap-subdir-test 2022-06-27T11:49:32.100 INFO:tasks.workunit.client.0.smithi195.stdout: create 266/30 file after the snapshot 2022-06-27T11:49:32.102 INFO:tasks.workunit.client.0.smithi195.stdout: create 266/31 file after the snapshot 2022-06-27T11:49:32.103 INFO:tasks.workunit.client.0.smithi195.stdout: create 266/32 file after the snapshot 2022-06-27T11:49:32.105 INFO:tasks.workunit.client.0.smithi195.stdout: create 266/33 file after the snapshot 2022-06-27T11:49:32.105 INFO:tasks.workunit.client.0.smithi195.stdout: create 266/34 file after the snapshot 2022-06-27T11:49:32.106 INFO:tasks.workunit.client.0.smithi195.stdout: create 266/35 file after the snapshot 2022-06-27T11:49:32.107 INFO:tasks.workunit.client.0.smithi195.stdout: create 266/36 file after the snapshot 2022-06-27T11:49:32.108 INFO:tasks.workunit.client.0.smithi195.stdout: create 266/37 file after the snapshot 2022-06-27T11:49:32.108 INFO:tasks.workunit.client.0.smithi195.stdout: create 266/38 file after the snapshot 2022-06-27T11:49:32.109 INFO:tasks.workunit.client.0.smithi195.stdout: create 266/39 file after the snapshot 2022-06-27T11:49:32.386 INFO:tasks.workunit.client.0.smithi195.stdout: create 267/.snap/snap-subdir-test 2022-06-27T11:49:32.421 INFO:tasks.workunit.client.0.smithi195.stdout: create 267/30 file after the snapshot 2022-06-27T11:49:32.436 INFO:tasks.workunit.client.0.smithi195.stdout: create 267/31 file after the snapshot 2022-06-27T11:49:32.437 INFO:tasks.workunit.client.0.smithi195.stdout: create 267/32 file after the snapshot 2022-06-27T11:49:32.437 INFO:tasks.workunit.client.0.smithi195.stdout: create 267/33 file after the snapshot 2022-06-27T11:49:32.439 INFO:tasks.workunit.client.0.smithi195.stdout: create 267/34 file after the snapshot 2022-06-27T11:49:32.440 INFO:tasks.workunit.client.0.smithi195.stdout: create 267/35 file after the snapshot 2022-06-27T11:49:32.441 INFO:tasks.workunit.client.0.smithi195.stdout: create 267/36 file after the snapshot 2022-06-27T11:49:32.442 INFO:tasks.workunit.client.0.smithi195.stdout: create 267/37 file after the snapshot 2022-06-27T11:49:32.443 INFO:tasks.workunit.client.0.smithi195.stdout: create 267/38 file after the snapshot 2022-06-27T11:49:32.444 INFO:tasks.workunit.client.0.smithi195.stdout: create 267/39 file after the snapshot 2022-06-27T11:49:32.445 INFO:tasks.workunit.client.0.smithi195.stdout: create 268/.snap/snap-subdir-test 2022-06-27T11:49:32.476 INFO:tasks.workunit.client.0.smithi195.stdout: create 268/30 file after the snapshot 2022-06-27T11:49:32.491 INFO:tasks.workunit.client.0.smithi195.stdout: create 268/31 file after the snapshot 2022-06-27T11:49:32.492 INFO:tasks.workunit.client.0.smithi195.stdout: create 268/32 file after the snapshot 2022-06-27T11:49:32.493 INFO:tasks.workunit.client.0.smithi195.stdout: create 268/33 file after the snapshot 2022-06-27T11:49:32.494 INFO:tasks.workunit.client.0.smithi195.stdout: create 268/34 file after the snapshot 2022-06-27T11:49:32.495 INFO:tasks.workunit.client.0.smithi195.stdout: create 268/35 file after the snapshot 2022-06-27T11:49:32.495 INFO:tasks.workunit.client.0.smithi195.stdout: create 268/36 file after the snapshot 2022-06-27T11:49:32.496 INFO:tasks.workunit.client.0.smithi195.stdout: create 268/37 file after the snapshot 2022-06-27T11:49:32.497 INFO:tasks.workunit.client.0.smithi195.stdout: create 268/38 file after the snapshot 2022-06-27T11:49:32.498 INFO:tasks.workunit.client.0.smithi195.stdout: create 268/39 file after the snapshot 2022-06-27T11:49:32.500 INFO:tasks.workunit.client.0.smithi195.stdout: create 269/.snap/snap-subdir-test 2022-06-27T11:49:32.530 INFO:tasks.workunit.client.0.smithi195.stdout: create 269/30 file after the snapshot 2022-06-27T11:49:32.546 INFO:tasks.workunit.client.0.smithi195.stdout: create 269/31 file after the snapshot 2022-06-27T11:49:32.547 INFO:tasks.workunit.client.0.smithi195.stdout: create 269/32 file after the snapshot 2022-06-27T11:49:32.547 INFO:tasks.workunit.client.0.smithi195.stdout: create 269/33 file after the snapshot 2022-06-27T11:49:32.549 INFO:tasks.workunit.client.0.smithi195.stdout: create 269/34 file after the snapshot 2022-06-27T11:49:32.549 INFO:tasks.workunit.client.0.smithi195.stdout: create 269/35 file after the snapshot 2022-06-27T11:49:32.550 INFO:tasks.workunit.client.0.smithi195.stdout: create 269/36 file after the snapshot 2022-06-27T11:49:32.552 INFO:tasks.workunit.client.0.smithi195.stdout: create 269/37 file after the snapshot 2022-06-27T11:49:32.552 INFO:tasks.workunit.client.0.smithi195.stdout: create 269/38 file after the snapshot 2022-06-27T11:49:32.553 INFO:tasks.workunit.client.0.smithi195.stdout: create 269/39 file after the snapshot 2022-06-27T11:49:32.554 INFO:tasks.workunit.client.0.smithi195.stdout: create 270/.snap/snap-subdir-test 2022-06-27T11:49:32.587 INFO:tasks.workunit.client.0.smithi195.stdout: create 270/30 file after the snapshot 2022-06-27T11:49:32.587 INFO:tasks.workunit.client.0.smithi195.stdout: create 270/31 file after the snapshot 2022-06-27T11:49:32.588 INFO:tasks.workunit.client.0.smithi195.stdout: create 270/32 file after the snapshot 2022-06-27T11:49:32.588 INFO:tasks.workunit.client.0.smithi195.stdout: create 270/33 file after the snapshot 2022-06-27T11:49:32.589 INFO:tasks.workunit.client.0.smithi195.stdout: create 270/34 file after the snapshot 2022-06-27T11:49:32.589 INFO:tasks.workunit.client.0.smithi195.stdout: create 270/35 file after the snapshot 2022-06-27T11:49:32.590 INFO:tasks.workunit.client.0.smithi195.stdout: create 270/36 file after the snapshot 2022-06-27T11:49:32.591 INFO:tasks.workunit.client.0.smithi195.stdout: create 270/37 file after the snapshot 2022-06-27T11:49:32.591 INFO:tasks.workunit.client.0.smithi195.stdout: create 270/38 file after the snapshot 2022-06-27T11:49:32.592 INFO:tasks.workunit.client.0.smithi195.stdout: create 270/39 file after the snapshot 2022-06-27T11:49:32.593 INFO:tasks.workunit.client.0.smithi195.stdout: create 271/.snap/snap-subdir-test 2022-06-27T11:49:33.917 INFO:tasks.workunit.client.0.smithi195.stdout: create 271/30 file after the snapshot 2022-06-27T11:49:33.932 INFO:tasks.workunit.client.0.smithi195.stdout: create 271/31 file after the snapshot 2022-06-27T11:49:33.933 INFO:tasks.workunit.client.0.smithi195.stdout: create 271/32 file after the snapshot 2022-06-27T11:49:33.934 INFO:tasks.workunit.client.0.smithi195.stdout: create 271/33 file after the snapshot 2022-06-27T11:49:33.935 INFO:tasks.workunit.client.0.smithi195.stdout: create 271/34 file after the snapshot 2022-06-27T11:49:33.936 INFO:tasks.workunit.client.0.smithi195.stdout: create 271/35 file after the snapshot 2022-06-27T11:49:33.937 INFO:tasks.workunit.client.0.smithi195.stdout: create 271/36 file after the snapshot 2022-06-27T11:49:33.938 INFO:tasks.workunit.client.0.smithi195.stdout: create 271/37 file after the snapshot 2022-06-27T11:49:33.939 INFO:tasks.workunit.client.0.smithi195.stdout: create 271/38 file after the snapshot 2022-06-27T11:49:33.940 INFO:tasks.workunit.client.0.smithi195.stdout: create 271/39 file after the snapshot 2022-06-27T11:49:33.941 INFO:tasks.workunit.client.0.smithi195.stdout: create 272/.snap/snap-subdir-test 2022-06-27T11:49:34.013 INFO:tasks.workunit.client.0.smithi195.stdout: create 272/30 file after the snapshot 2022-06-27T11:49:34.029 INFO:tasks.workunit.client.0.smithi195.stdout: create 272/31 file after the snapshot 2022-06-27T11:49:34.030 INFO:tasks.workunit.client.0.smithi195.stdout: create 272/32 file after the snapshot 2022-06-27T11:49:34.031 INFO:tasks.workunit.client.0.smithi195.stdout: create 272/33 file after the snapshot 2022-06-27T11:49:34.032 INFO:tasks.workunit.client.0.smithi195.stdout: create 272/34 file after the snapshot 2022-06-27T11:49:34.033 INFO:tasks.workunit.client.0.smithi195.stdout: create 272/35 file after the snapshot 2022-06-27T11:49:34.034 INFO:tasks.workunit.client.0.smithi195.stdout: create 272/36 file after the snapshot 2022-06-27T11:49:34.037 INFO:tasks.workunit.client.0.smithi195.stdout: create 272/37 file after the snapshot 2022-06-27T11:49:34.038 INFO:tasks.workunit.client.0.smithi195.stdout: create 272/38 file after the snapshot 2022-06-27T11:49:34.038 INFO:tasks.workunit.client.0.smithi195.stdout: create 272/39 file after the snapshot 2022-06-27T11:49:34.039 INFO:tasks.workunit.client.0.smithi195.stdout: create 273/.snap/snap-subdir-test 2022-06-27T11:49:34.442 INFO:tasks.workunit.client.0.smithi195.stdout: create 273/30 file after the snapshot 2022-06-27T11:49:34.459 INFO:tasks.workunit.client.0.smithi195.stdout: create 273/31 file after the snapshot 2022-06-27T11:49:34.461 INFO:tasks.workunit.client.0.smithi195.stdout: create 273/32 file after the snapshot 2022-06-27T11:49:34.461 INFO:tasks.workunit.client.0.smithi195.stdout: create 273/33 file after the snapshot 2022-06-27T11:49:34.464 INFO:tasks.workunit.client.0.smithi195.stdout: create 273/34 file after the snapshot 2022-06-27T11:49:34.465 INFO:tasks.workunit.client.0.smithi195.stdout: create 273/35 file after the snapshot 2022-06-27T11:49:34.466 INFO:tasks.workunit.client.0.smithi195.stdout: create 273/36 file after the snapshot 2022-06-27T11:49:34.466 INFO:tasks.workunit.client.0.smithi195.stdout: create 273/37 file after the snapshot 2022-06-27T11:49:34.467 INFO:tasks.workunit.client.0.smithi195.stdout: create 273/38 file after the snapshot 2022-06-27T11:49:34.467 INFO:tasks.workunit.client.0.smithi195.stdout: create 273/39 file after the snapshot 2022-06-27T11:49:34.469 INFO:tasks.workunit.client.0.smithi195.stdout: create 274/.snap/snap-subdir-test 2022-06-27T11:49:34.493 INFO:tasks.workunit.client.0.smithi195.stdout: create 274/30 file after the snapshot 2022-06-27T11:49:34.494 INFO:tasks.workunit.client.0.smithi195.stdout: create 274/31 file after the snapshot 2022-06-27T11:49:34.502 INFO:tasks.workunit.client.0.smithi195.stdout: create 274/32 file after the snapshot 2022-06-27T11:49:34.503 INFO:tasks.workunit.client.0.smithi195.stdout: create 274/33 file after the snapshot 2022-06-27T11:49:34.503 INFO:tasks.workunit.client.0.smithi195.stdout: create 274/34 file after the snapshot 2022-06-27T11:49:34.503 INFO:tasks.workunit.client.0.smithi195.stdout: create 274/35 file after the snapshot 2022-06-27T11:49:34.503 INFO:tasks.workunit.client.0.smithi195.stdout: create 274/36 file after the snapshot 2022-06-27T11:49:34.504 INFO:tasks.workunit.client.0.smithi195.stdout: create 274/37 file after the snapshot 2022-06-27T11:49:34.504 INFO:tasks.workunit.client.0.smithi195.stdout: create 274/38 file after the snapshot 2022-06-27T11:49:34.504 INFO:tasks.workunit.client.0.smithi195.stdout: create 274/39 file after the snapshot 2022-06-27T11:49:34.504 INFO:tasks.workunit.client.0.smithi195.stdout: create 275/.snap/snap-subdir-test 2022-06-27T11:49:34.530 INFO:tasks.workunit.client.0.smithi195.stdout: create 275/30 file after the snapshot 2022-06-27T11:49:34.546 INFO:tasks.workunit.client.0.smithi195.stdout: create 275/31 file after the snapshot 2022-06-27T11:49:34.547 INFO:tasks.workunit.client.0.smithi195.stdout: create 275/32 file after the snapshot 2022-06-27T11:49:34.551 INFO:tasks.workunit.client.0.smithi195.stdout: create 275/33 file after the snapshot 2022-06-27T11:49:34.551 INFO:tasks.workunit.client.0.smithi195.stdout: create 275/34 file after the snapshot 2022-06-27T11:49:34.552 INFO:tasks.workunit.client.0.smithi195.stdout: create 275/35 file after the snapshot 2022-06-27T11:49:34.552 INFO:tasks.workunit.client.0.smithi195.stdout: create 275/36 file after the snapshot 2022-06-27T11:49:34.553 INFO:tasks.workunit.client.0.smithi195.stdout: create 275/37 file after the snapshot 2022-06-27T11:49:34.553 INFO:tasks.workunit.client.0.smithi195.stdout: create 275/38 file after the snapshot 2022-06-27T11:49:34.554 INFO:tasks.workunit.client.0.smithi195.stdout: create 275/39 file after the snapshot 2022-06-27T11:49:34.554 INFO:tasks.workunit.client.0.smithi195.stdout: create 276/.snap/snap-subdir-test 2022-06-27T11:49:34.578 INFO:tasks.workunit.client.0.smithi195.stdout: create 276/30 file after the snapshot 2022-06-27T11:49:34.594 INFO:tasks.workunit.client.0.smithi195.stdout: create 276/31 file after the snapshot 2022-06-27T11:49:34.595 INFO:tasks.workunit.client.0.smithi195.stdout: create 276/32 file after the snapshot 2022-06-27T11:49:34.596 INFO:tasks.workunit.client.0.smithi195.stdout: create 276/33 file after the snapshot 2022-06-27T11:49:34.597 INFO:tasks.workunit.client.0.smithi195.stdout: create 276/34 file after the snapshot 2022-06-27T11:49:34.598 INFO:tasks.workunit.client.0.smithi195.stdout: create 276/35 file after the snapshot 2022-06-27T11:49:34.599 INFO:tasks.workunit.client.0.smithi195.stdout: create 276/36 file after the snapshot 2022-06-27T11:49:34.599 INFO:tasks.workunit.client.0.smithi195.stdout: create 276/37 file after the snapshot 2022-06-27T11:49:34.600 INFO:tasks.workunit.client.0.smithi195.stdout: create 276/38 file after the snapshot 2022-06-27T11:49:34.601 INFO:tasks.workunit.client.0.smithi195.stdout: create 276/39 file after the snapshot 2022-06-27T11:49:34.602 INFO:tasks.workunit.client.0.smithi195.stdout: create 277/.snap/snap-subdir-test 2022-06-27T11:49:34.629 INFO:tasks.workunit.client.0.smithi195.stdout: create 277/30 file after the snapshot 2022-06-27T11:49:34.643 INFO:tasks.workunit.client.0.smithi195.stdout: create 277/31 file after the snapshot 2022-06-27T11:49:34.644 INFO:tasks.workunit.client.0.smithi195.stdout: create 277/32 file after the snapshot 2022-06-27T11:49:34.645 INFO:tasks.workunit.client.0.smithi195.stdout: create 277/33 file after the snapshot 2022-06-27T11:49:34.645 INFO:tasks.workunit.client.0.smithi195.stdout: create 277/34 file after the snapshot 2022-06-27T11:49:34.648 INFO:tasks.workunit.client.0.smithi195.stdout: create 277/35 file after the snapshot 2022-06-27T11:49:34.648 INFO:tasks.workunit.client.0.smithi195.stdout: create 277/36 file after the snapshot 2022-06-27T11:49:34.648 INFO:tasks.workunit.client.0.smithi195.stdout: create 277/37 file after the snapshot 2022-06-27T11:49:34.649 INFO:tasks.workunit.client.0.smithi195.stdout: create 277/38 file after the snapshot 2022-06-27T11:49:34.649 INFO:tasks.workunit.client.0.smithi195.stdout: create 277/39 file after the snapshot 2022-06-27T11:49:34.650 INFO:tasks.workunit.client.0.smithi195.stdout: create 278/.snap/snap-subdir-test 2022-06-27T11:49:34.676 INFO:tasks.workunit.client.0.smithi195.stdout: create 278/30 file after the snapshot 2022-06-27T11:49:34.693 INFO:tasks.workunit.client.0.smithi195.stdout: create 278/31 file after the snapshot 2022-06-27T11:49:34.693 INFO:tasks.workunit.client.0.smithi195.stdout: create 278/32 file after the snapshot 2022-06-27T11:49:34.694 INFO:tasks.workunit.client.0.smithi195.stdout: create 278/33 file after the snapshot 2022-06-27T11:49:34.694 INFO:tasks.workunit.client.0.smithi195.stdout: create 278/34 file after the snapshot 2022-06-27T11:49:34.695 INFO:tasks.workunit.client.0.smithi195.stdout: create 278/35 file after the snapshot 2022-06-27T11:49:34.695 INFO:tasks.workunit.client.0.smithi195.stdout: create 278/36 file after the snapshot 2022-06-27T11:49:34.696 INFO:tasks.workunit.client.0.smithi195.stdout: create 278/37 file after the snapshot 2022-06-27T11:49:34.697 INFO:tasks.workunit.client.0.smithi195.stdout: create 278/38 file after the snapshot 2022-06-27T11:49:34.699 INFO:tasks.workunit.client.0.smithi195.stdout: create 278/39 file after the snapshot 2022-06-27T11:49:34.700 INFO:tasks.workunit.client.0.smithi195.stdout: create 279/.snap/snap-subdir-test 2022-06-27T11:49:34.722 INFO:tasks.workunit.client.0.smithi195.stdout: create 279/30 file after the snapshot 2022-06-27T11:49:34.738 INFO:tasks.workunit.client.0.smithi195.stdout: create 279/31 file after the snapshot 2022-06-27T11:49:34.740 INFO:tasks.workunit.client.0.smithi195.stdout: create 279/32 file after the snapshot 2022-06-27T11:49:34.741 INFO:tasks.workunit.client.0.smithi195.stdout: create 279/33 file after the snapshot 2022-06-27T11:49:34.741 INFO:tasks.workunit.client.0.smithi195.stdout: create 279/34 file after the snapshot 2022-06-27T11:49:34.742 INFO:tasks.workunit.client.0.smithi195.stdout: create 279/35 file after the snapshot 2022-06-27T11:49:34.743 INFO:tasks.workunit.client.0.smithi195.stdout: create 279/36 file after the snapshot 2022-06-27T11:49:34.744 INFO:tasks.workunit.client.0.smithi195.stdout: create 279/37 file after the snapshot 2022-06-27T11:49:34.745 INFO:tasks.workunit.client.0.smithi195.stdout: create 279/38 file after the snapshot 2022-06-27T11:49:34.745 INFO:tasks.workunit.client.0.smithi195.stdout: create 279/39 file after the snapshot 2022-06-27T11:49:34.746 INFO:tasks.workunit.client.0.smithi195.stdout: create 280/.snap/snap-subdir-test 2022-06-27T11:49:34.770 INFO:tasks.workunit.client.0.smithi195.stdout: create 280/30 file after the snapshot 2022-06-27T11:49:34.786 INFO:tasks.workunit.client.0.smithi195.stdout: create 280/31 file after the snapshot 2022-06-27T11:49:34.811 INFO:tasks.workunit.client.0.smithi195.stdout: create 280/32 file after the snapshot 2022-06-27T11:49:34.812 INFO:tasks.workunit.client.0.smithi195.stdout: create 280/33 file after the snapshot 2022-06-27T11:49:34.812 INFO:tasks.workunit.client.0.smithi195.stdout: create 280/34 file after the snapshot 2022-06-27T11:49:34.813 INFO:tasks.workunit.client.0.smithi195.stdout: create 280/35 file after the snapshot 2022-06-27T11:49:34.814 INFO:tasks.workunit.client.0.smithi195.stdout: create 280/36 file after the snapshot 2022-06-27T11:49:34.815 INFO:tasks.workunit.client.0.smithi195.stdout: create 280/37 file after the snapshot 2022-06-27T11:49:34.815 INFO:tasks.workunit.client.0.smithi195.stdout: create 280/38 file after the snapshot 2022-06-27T11:49:34.816 INFO:tasks.workunit.client.0.smithi195.stdout: create 280/39 file after the snapshot 2022-06-27T11:49:34.816 INFO:tasks.workunit.client.0.smithi195.stdout: create 281/.snap/snap-subdir-test 2022-06-27T11:49:34.817 INFO:tasks.workunit.client.0.smithi195.stdout: create 281/30 file after the snapshot 2022-06-27T11:49:34.833 INFO:tasks.workunit.client.0.smithi195.stdout: create 281/31 file after the snapshot 2022-06-27T11:49:34.834 INFO:tasks.workunit.client.0.smithi195.stdout: create 281/32 file after the snapshot 2022-06-27T11:49:34.835 INFO:tasks.workunit.client.0.smithi195.stdout: create 281/33 file after the snapshot 2022-06-27T11:49:34.835 INFO:tasks.workunit.client.0.smithi195.stdout: create 281/34 file after the snapshot 2022-06-27T11:49:34.837 INFO:tasks.workunit.client.0.smithi195.stdout: create 281/35 file after the snapshot 2022-06-27T11:49:34.837 INFO:tasks.workunit.client.0.smithi195.stdout: create 281/36 file after the snapshot 2022-06-27T11:49:34.838 INFO:tasks.workunit.client.0.smithi195.stdout: create 281/37 file after the snapshot 2022-06-27T11:49:34.839 INFO:tasks.workunit.client.0.smithi195.stdout: create 281/38 file after the snapshot 2022-06-27T11:49:34.841 INFO:tasks.workunit.client.0.smithi195.stdout: create 281/39 file after the snapshot 2022-06-27T11:49:34.841 INFO:tasks.workunit.client.0.smithi195.stdout: create 282/.snap/snap-subdir-test 2022-06-27T11:49:34.864 INFO:tasks.workunit.client.0.smithi195.stdout: create 282/30 file after the snapshot 2022-06-27T11:49:34.880 INFO:tasks.workunit.client.0.smithi195.stdout: create 282/31 file after the snapshot 2022-06-27T11:49:34.881 INFO:tasks.workunit.client.0.smithi195.stdout: create 282/32 file after the snapshot 2022-06-27T11:49:34.881 INFO:tasks.workunit.client.0.smithi195.stdout: create 282/33 file after the snapshot 2022-06-27T11:49:34.882 INFO:tasks.workunit.client.0.smithi195.stdout: create 282/34 file after the snapshot 2022-06-27T11:49:34.883 INFO:tasks.workunit.client.0.smithi195.stdout: create 282/35 file after the snapshot 2022-06-27T11:49:34.884 INFO:tasks.workunit.client.0.smithi195.stdout: create 282/36 file after the snapshot 2022-06-27T11:49:34.885 INFO:tasks.workunit.client.0.smithi195.stdout: create 282/37 file after the snapshot 2022-06-27T11:49:34.886 INFO:tasks.workunit.client.0.smithi195.stdout: create 282/38 file after the snapshot 2022-06-27T11:49:34.887 INFO:tasks.workunit.client.0.smithi195.stdout: create 282/39 file after the snapshot 2022-06-27T11:49:34.888 INFO:tasks.workunit.client.0.smithi195.stdout: create 283/.snap/snap-subdir-test 2022-06-27T11:49:34.914 INFO:tasks.workunit.client.0.smithi195.stdout: create 283/30 file after the snapshot 2022-06-27T11:49:34.928 INFO:tasks.workunit.client.0.smithi195.stdout: create 283/31 file after the snapshot 2022-06-27T11:49:34.929 INFO:tasks.workunit.client.0.smithi195.stdout: create 283/32 file after the snapshot 2022-06-27T11:49:34.930 INFO:tasks.workunit.client.0.smithi195.stdout: create 283/33 file after the snapshot 2022-06-27T11:49:34.930 INFO:tasks.workunit.client.0.smithi195.stdout: create 283/34 file after the snapshot 2022-06-27T11:49:34.931 INFO:tasks.workunit.client.0.smithi195.stdout: create 283/35 file after the snapshot 2022-06-27T11:49:34.932 INFO:tasks.workunit.client.0.smithi195.stdout: create 283/36 file after the snapshot 2022-06-27T11:49:34.933 INFO:tasks.workunit.client.0.smithi195.stdout: create 283/37 file after the snapshot 2022-06-27T11:49:34.933 INFO:tasks.workunit.client.0.smithi195.stdout: create 283/38 file after the snapshot 2022-06-27T11:49:34.934 INFO:tasks.workunit.client.0.smithi195.stdout: create 283/39 file after the snapshot 2022-06-27T11:49:34.935 INFO:tasks.workunit.client.0.smithi195.stdout: create 284/.snap/snap-subdir-test 2022-06-27T11:49:34.959 INFO:tasks.workunit.client.0.smithi195.stdout: create 284/30 file after the snapshot 2022-06-27T11:49:34.974 INFO:tasks.workunit.client.0.smithi195.stdout: create 284/31 file after the snapshot 2022-06-27T11:49:34.975 INFO:tasks.workunit.client.0.smithi195.stdout: create 284/32 file after the snapshot 2022-06-27T11:49:34.976 INFO:tasks.workunit.client.0.smithi195.stdout: create 284/33 file after the snapshot 2022-06-27T11:49:34.977 INFO:tasks.workunit.client.0.smithi195.stdout: create 284/34 file after the snapshot 2022-06-27T11:49:34.978 INFO:tasks.workunit.client.0.smithi195.stdout: create 284/35 file after the snapshot 2022-06-27T11:49:34.979 INFO:tasks.workunit.client.0.smithi195.stdout: create 284/36 file after the snapshot 2022-06-27T11:49:34.980 INFO:tasks.workunit.client.0.smithi195.stdout: create 284/37 file after the snapshot 2022-06-27T11:49:34.981 INFO:tasks.workunit.client.0.smithi195.stdout: create 284/38 file after the snapshot 2022-06-27T11:49:34.982 INFO:tasks.workunit.client.0.smithi195.stdout: create 284/39 file after the snapshot 2022-06-27T11:49:34.983 INFO:tasks.workunit.client.0.smithi195.stdout: create 285/.snap/snap-subdir-test 2022-06-27T11:49:35.007 INFO:tasks.workunit.client.0.smithi195.stdout: create 285/30 file after the snapshot 2022-06-27T11:49:35.023 INFO:tasks.workunit.client.0.smithi195.stdout: create 285/31 file after the snapshot 2022-06-27T11:49:35.024 INFO:tasks.workunit.client.0.smithi195.stdout: create 285/32 file after the snapshot 2022-06-27T11:49:35.025 INFO:tasks.workunit.client.0.smithi195.stdout: create 285/33 file after the snapshot 2022-06-27T11:49:35.026 INFO:tasks.workunit.client.0.smithi195.stdout: create 285/34 file after the snapshot 2022-06-27T11:49:35.028 INFO:tasks.workunit.client.0.smithi195.stdout: create 285/35 file after the snapshot 2022-06-27T11:49:35.030 INFO:tasks.workunit.client.0.smithi195.stdout: create 285/36 file after the snapshot 2022-06-27T11:49:35.030 INFO:tasks.workunit.client.0.smithi195.stdout: create 285/37 file after the snapshot 2022-06-27T11:49:35.032 INFO:tasks.workunit.client.0.smithi195.stdout: create 285/38 file after the snapshot 2022-06-27T11:49:35.032 INFO:tasks.workunit.client.0.smithi195.stdout: create 285/39 file after the snapshot 2022-06-27T11:49:35.033 INFO:tasks.workunit.client.0.smithi195.stdout: create 286/.snap/snap-subdir-test 2022-06-27T11:49:35.058 INFO:tasks.workunit.client.0.smithi195.stdout: create 286/30 file after the snapshot 2022-06-27T11:49:35.070 INFO:tasks.workunit.client.0.smithi195.stdout: create 286/31 file after the snapshot 2022-06-27T11:49:35.071 INFO:tasks.workunit.client.0.smithi195.stdout: create 286/32 file after the snapshot 2022-06-27T11:49:35.072 INFO:tasks.workunit.client.0.smithi195.stdout: create 286/33 file after the snapshot 2022-06-27T11:49:35.073 INFO:tasks.workunit.client.0.smithi195.stdout: create 286/34 file after the snapshot 2022-06-27T11:49:35.073 INFO:tasks.workunit.client.0.smithi195.stdout: create 286/35 file after the snapshot 2022-06-27T11:49:35.074 INFO:tasks.workunit.client.0.smithi195.stdout: create 286/36 file after the snapshot 2022-06-27T11:49:35.075 INFO:tasks.workunit.client.0.smithi195.stdout: create 286/37 file after the snapshot 2022-06-27T11:49:35.076 INFO:tasks.workunit.client.0.smithi195.stdout: create 286/38 file after the snapshot 2022-06-27T11:49:35.077 INFO:tasks.workunit.client.0.smithi195.stdout: create 286/39 file after the snapshot 2022-06-27T11:49:35.078 INFO:tasks.workunit.client.0.smithi195.stdout: create 287/.snap/snap-subdir-test 2022-06-27T11:49:35.102 INFO:tasks.workunit.client.0.smithi195.stdout: create 287/30 file after the snapshot 2022-06-27T11:49:35.117 INFO:tasks.workunit.client.0.smithi195.stdout: create 287/31 file after the snapshot 2022-06-27T11:49:35.118 INFO:tasks.workunit.client.0.smithi195.stdout: create 287/32 file after the snapshot 2022-06-27T11:49:35.120 INFO:tasks.workunit.client.0.smithi195.stdout: create 287/33 file after the snapshot 2022-06-27T11:49:35.120 INFO:tasks.workunit.client.0.smithi195.stdout: create 287/34 file after the snapshot 2022-06-27T11:49:35.121 INFO:tasks.workunit.client.0.smithi195.stdout: create 287/35 file after the snapshot 2022-06-27T11:49:35.121 INFO:tasks.workunit.client.0.smithi195.stdout: create 287/36 file after the snapshot 2022-06-27T11:49:35.122 INFO:tasks.workunit.client.0.smithi195.stdout: create 287/37 file after the snapshot 2022-06-27T11:49:35.123 INFO:tasks.workunit.client.0.smithi195.stdout: create 287/38 file after the snapshot 2022-06-27T11:49:35.125 INFO:tasks.workunit.client.0.smithi195.stdout: create 287/39 file after the snapshot 2022-06-27T11:49:35.126 INFO:tasks.workunit.client.0.smithi195.stdout: create 288/.snap/snap-subdir-test 2022-06-27T11:49:35.150 INFO:tasks.workunit.client.0.smithi195.stdout: create 288/30 file after the snapshot 2022-06-27T11:49:35.442 INFO:tasks.workunit.client.0.smithi195.stdout: create 288/31 file after the snapshot 2022-06-27T11:49:35.443 INFO:tasks.workunit.client.0.smithi195.stdout: create 288/32 file after the snapshot 2022-06-27T11:49:35.443 INFO:tasks.workunit.client.0.smithi195.stdout: create 288/33 file after the snapshot 2022-06-27T11:49:35.444 INFO:tasks.workunit.client.0.smithi195.stdout: create 288/34 file after the snapshot 2022-06-27T11:49:35.445 INFO:tasks.workunit.client.0.smithi195.stdout: create 288/35 file after the snapshot 2022-06-27T11:49:35.446 INFO:tasks.workunit.client.0.smithi195.stdout: create 288/36 file after the snapshot 2022-06-27T11:49:35.446 INFO:tasks.workunit.client.0.smithi195.stdout: create 288/37 file after the snapshot 2022-06-27T11:49:35.447 INFO:tasks.workunit.client.0.smithi195.stdout: create 288/38 file after the snapshot 2022-06-27T11:49:35.448 INFO:tasks.workunit.client.0.smithi195.stdout: create 288/39 file after the snapshot 2022-06-27T11:49:35.449 INFO:tasks.workunit.client.0.smithi195.stdout: create 289/.snap/snap-subdir-test 2022-06-27T11:49:35.474 INFO:tasks.workunit.client.0.smithi195.stdout: create 289/30 file after the snapshot 2022-06-27T11:49:35.490 INFO:tasks.workunit.client.0.smithi195.stdout: create 289/31 file after the snapshot 2022-06-27T11:49:35.492 INFO:tasks.workunit.client.0.smithi195.stdout: create 289/32 file after the snapshot 2022-06-27T11:49:35.492 INFO:tasks.workunit.client.0.smithi195.stdout: create 289/33 file after the snapshot 2022-06-27T11:49:35.493 INFO:tasks.workunit.client.0.smithi195.stdout: create 289/34 file after the snapshot 2022-06-27T11:49:35.493 INFO:tasks.workunit.client.0.smithi195.stdout: create 289/35 file after the snapshot 2022-06-27T11:49:35.494 INFO:tasks.workunit.client.0.smithi195.stdout: create 289/36 file after the snapshot 2022-06-27T11:49:35.495 INFO:tasks.workunit.client.0.smithi195.stdout: create 289/37 file after the snapshot 2022-06-27T11:49:35.495 INFO:tasks.workunit.client.0.smithi195.stdout: create 289/38 file after the snapshot 2022-06-27T11:49:35.497 INFO:tasks.workunit.client.0.smithi195.stdout: create 289/39 file after the snapshot 2022-06-27T11:49:35.497 INFO:tasks.workunit.client.0.smithi195.stdout: create 290/.snap/snap-subdir-test 2022-06-27T11:49:35.522 INFO:tasks.workunit.client.0.smithi195.stdout: create 290/30 file after the snapshot 2022-06-27T11:49:35.537 INFO:tasks.workunit.client.0.smithi195.stdout: create 290/31 file after the snapshot 2022-06-27T11:49:35.538 INFO:tasks.workunit.client.0.smithi195.stdout: create 290/32 file after the snapshot 2022-06-27T11:49:35.539 INFO:tasks.workunit.client.0.smithi195.stdout: create 290/33 file after the snapshot 2022-06-27T11:49:35.540 INFO:tasks.workunit.client.0.smithi195.stdout: create 290/34 file after the snapshot 2022-06-27T11:49:35.541 INFO:tasks.workunit.client.0.smithi195.stdout: create 290/35 file after the snapshot 2022-06-27T11:49:35.541 INFO:tasks.workunit.client.0.smithi195.stdout: create 290/36 file after the snapshot 2022-06-27T11:49:35.542 INFO:tasks.workunit.client.0.smithi195.stdout: create 290/37 file after the snapshot 2022-06-27T11:49:35.543 INFO:tasks.workunit.client.0.smithi195.stdout: create 290/38 file after the snapshot 2022-06-27T11:49:35.544 INFO:tasks.workunit.client.0.smithi195.stdout: create 290/39 file after the snapshot 2022-06-27T11:49:35.545 INFO:tasks.workunit.client.0.smithi195.stdout: create 291/.snap/snap-subdir-test 2022-06-27T11:49:35.570 INFO:tasks.workunit.client.0.smithi195.stdout: create 291/30 file after the snapshot 2022-06-27T11:49:35.585 INFO:tasks.workunit.client.0.smithi195.stdout: create 291/31 file after the snapshot 2022-06-27T11:49:35.586 INFO:tasks.workunit.client.0.smithi195.stdout: create 291/32 file after the snapshot 2022-06-27T11:49:35.587 INFO:tasks.workunit.client.0.smithi195.stdout: create 291/33 file after the snapshot 2022-06-27T11:49:35.588 INFO:tasks.workunit.client.0.smithi195.stdout: create 291/34 file after the snapshot 2022-06-27T11:49:35.590 INFO:tasks.workunit.client.0.smithi195.stdout: create 291/35 file after the snapshot 2022-06-27T11:49:35.591 INFO:tasks.workunit.client.0.smithi195.stdout: create 291/36 file after the snapshot 2022-06-27T11:49:35.591 INFO:tasks.workunit.client.0.smithi195.stdout: create 291/37 file after the snapshot 2022-06-27T11:49:35.592 INFO:tasks.workunit.client.0.smithi195.stdout: create 291/38 file after the snapshot 2022-06-27T11:49:35.592 INFO:tasks.workunit.client.0.smithi195.stdout: create 291/39 file after the snapshot 2022-06-27T11:49:35.593 INFO:tasks.workunit.client.0.smithi195.stdout: create 292/.snap/snap-subdir-test 2022-06-27T11:49:35.616 INFO:tasks.workunit.client.0.smithi195.stdout: create 292/30 file after the snapshot 2022-06-27T11:49:35.633 INFO:tasks.workunit.client.0.smithi195.stdout: create 292/31 file after the snapshot 2022-06-27T11:49:35.633 INFO:tasks.workunit.client.0.smithi195.stdout: create 292/32 file after the snapshot 2022-06-27T11:49:35.634 INFO:tasks.workunit.client.0.smithi195.stdout: create 292/33 file after the snapshot 2022-06-27T11:49:35.635 INFO:tasks.workunit.client.0.smithi195.stdout: create 292/34 file after the snapshot 2022-06-27T11:49:35.635 INFO:tasks.workunit.client.0.smithi195.stdout: create 292/35 file after the snapshot 2022-06-27T11:49:35.636 INFO:tasks.workunit.client.0.smithi195.stdout: create 292/36 file after the snapshot 2022-06-27T11:49:35.637 INFO:tasks.workunit.client.0.smithi195.stdout: create 292/37 file after the snapshot 2022-06-27T11:49:35.637 INFO:tasks.workunit.client.0.smithi195.stdout: create 292/38 file after the snapshot 2022-06-27T11:49:35.639 INFO:tasks.workunit.client.0.smithi195.stdout: create 292/39 file after the snapshot 2022-06-27T11:49:35.639 INFO:tasks.workunit.client.0.smithi195.stdout: create 293/.snap/snap-subdir-test 2022-06-27T11:49:35.662 INFO:tasks.workunit.client.0.smithi195.stdout: create 293/30 file after the snapshot 2022-06-27T11:49:35.678 INFO:tasks.workunit.client.0.smithi195.stdout: create 293/31 file after the snapshot 2022-06-27T11:49:35.679 INFO:tasks.workunit.client.0.smithi195.stdout: create 293/32 file after the snapshot 2022-06-27T11:49:35.679 INFO:tasks.workunit.client.0.smithi195.stdout: create 293/33 file after the snapshot 2022-06-27T11:49:35.680 INFO:tasks.workunit.client.0.smithi195.stdout: create 293/34 file after the snapshot 2022-06-27T11:49:35.681 INFO:tasks.workunit.client.0.smithi195.stdout: create 293/35 file after the snapshot 2022-06-27T11:49:35.682 INFO:tasks.workunit.client.0.smithi195.stdout: create 293/36 file after the snapshot 2022-06-27T11:49:35.683 INFO:tasks.workunit.client.0.smithi195.stdout: create 293/37 file after the snapshot 2022-06-27T11:49:35.683 INFO:tasks.workunit.client.0.smithi195.stdout: create 293/38 file after the snapshot 2022-06-27T11:49:35.684 INFO:tasks.workunit.client.0.smithi195.stdout: create 293/39 file after the snapshot 2022-06-27T11:49:35.685 INFO:tasks.workunit.client.0.smithi195.stdout: create 294/.snap/snap-subdir-test 2022-06-27T11:49:35.709 INFO:tasks.workunit.client.0.smithi195.stdout: create 294/30 file after the snapshot 2022-06-27T11:49:35.726 INFO:tasks.workunit.client.0.smithi195.stdout: create 294/31 file after the snapshot 2022-06-27T11:49:35.727 INFO:tasks.workunit.client.0.smithi195.stdout: create 294/32 file after the snapshot 2022-06-27T11:49:36.106 INFO:tasks.workunit.client.0.smithi195.stdout: create 294/33 file after the snapshot 2022-06-27T11:49:36.306 INFO:tasks.workunit.client.0.smithi195.stdout: create 294/34 file after the snapshot 2022-06-27T11:49:36.306 INFO:tasks.workunit.client.0.smithi195.stdout: create 294/35 file after the snapshot 2022-06-27T11:49:36.307 INFO:tasks.workunit.client.0.smithi195.stdout: create 294/36 file after the snapshot 2022-06-27T11:49:36.308 INFO:tasks.workunit.client.0.smithi195.stdout: create 294/37 file after the snapshot 2022-06-27T11:49:36.309 INFO:tasks.workunit.client.0.smithi195.stdout: create 294/38 file after the snapshot 2022-06-27T11:49:36.310 INFO:tasks.workunit.client.0.smithi195.stdout: create 294/39 file after the snapshot 2022-06-27T11:49:36.311 INFO:tasks.workunit.client.0.smithi195.stdout: create 295/.snap/snap-subdir-test 2022-06-27T11:49:36.350 INFO:tasks.workunit.client.0.smithi195.stdout: create 295/30 file after the snapshot 2022-06-27T11:49:36.367 INFO:tasks.workunit.client.0.smithi195.stdout: create 295/31 file after the snapshot 2022-06-27T11:49:36.367 INFO:tasks.workunit.client.0.smithi195.stdout: create 295/32 file after the snapshot 2022-06-27T11:49:36.367 INFO:tasks.workunit.client.0.smithi195.stdout: create 295/33 file after the snapshot 2022-06-27T11:49:36.368 INFO:tasks.workunit.client.0.smithi195.stdout: create 295/34 file after the snapshot 2022-06-27T11:49:36.368 INFO:tasks.workunit.client.0.smithi195.stdout: create 295/35 file after the snapshot 2022-06-27T11:49:36.369 INFO:tasks.workunit.client.0.smithi195.stdout: create 295/36 file after the snapshot 2022-06-27T11:49:36.375 INFO:tasks.workunit.client.0.smithi195.stdout: create 295/37 file after the snapshot 2022-06-27T11:49:36.376 INFO:tasks.workunit.client.0.smithi195.stdout: create 295/38 file after the snapshot 2022-06-27T11:49:36.376 INFO:tasks.workunit.client.0.smithi195.stdout: create 295/39 file after the snapshot 2022-06-27T11:49:36.376 INFO:tasks.workunit.client.0.smithi195.stdout: create 296/.snap/snap-subdir-test 2022-06-27T11:49:36.405 INFO:tasks.workunit.client.0.smithi195.stdout: create 296/30 file after the snapshot 2022-06-27T11:49:36.421 INFO:tasks.workunit.client.0.smithi195.stdout: create 296/31 file after the snapshot 2022-06-27T11:49:36.421 INFO:tasks.workunit.client.0.smithi195.stdout: create 296/32 file after the snapshot 2022-06-27T11:49:36.422 INFO:tasks.workunit.client.0.smithi195.stdout: create 296/33 file after the snapshot 2022-06-27T11:49:36.422 INFO:tasks.workunit.client.0.smithi195.stdout: create 296/34 file after the snapshot 2022-06-27T11:49:36.423 INFO:tasks.workunit.client.0.smithi195.stdout: create 296/35 file after the snapshot 2022-06-27T11:49:36.427 INFO:tasks.workunit.client.0.smithi195.stdout: create 296/36 file after the snapshot 2022-06-27T11:49:36.427 INFO:tasks.workunit.client.0.smithi195.stdout: create 296/37 file after the snapshot 2022-06-27T11:49:36.427 INFO:tasks.workunit.client.0.smithi195.stdout: create 296/38 file after the snapshot 2022-06-27T11:49:36.428 INFO:tasks.workunit.client.0.smithi195.stdout: create 296/39 file after the snapshot 2022-06-27T11:49:36.429 INFO:tasks.workunit.client.0.smithi195.stdout: create 297/.snap/snap-subdir-test 2022-06-27T11:49:36.462 INFO:tasks.workunit.client.0.smithi195.stdout: create 297/30 file after the snapshot 2022-06-27T11:49:36.476 INFO:tasks.workunit.client.0.smithi195.stdout: create 297/31 file after the snapshot 2022-06-27T11:49:36.476 INFO:tasks.workunit.client.0.smithi195.stdout: create 297/32 file after the snapshot 2022-06-27T11:49:36.477 INFO:tasks.workunit.client.0.smithi195.stdout: create 297/33 file after the snapshot 2022-06-27T11:49:36.478 INFO:tasks.workunit.client.0.smithi195.stdout: create 297/34 file after the snapshot 2022-06-27T11:49:36.478 INFO:tasks.workunit.client.0.smithi195.stdout: create 297/35 file after the snapshot 2022-06-27T11:49:36.479 INFO:tasks.workunit.client.0.smithi195.stdout: create 297/36 file after the snapshot 2022-06-27T11:49:36.480 INFO:tasks.workunit.client.0.smithi195.stdout: create 297/37 file after the snapshot 2022-06-27T11:49:36.480 INFO:tasks.workunit.client.0.smithi195.stdout: create 297/38 file after the snapshot 2022-06-27T11:49:36.481 INFO:tasks.workunit.client.0.smithi195.stdout: create 297/39 file after the snapshot 2022-06-27T11:49:36.482 INFO:tasks.workunit.client.0.smithi195.stdout: create 298/.snap/snap-subdir-test 2022-06-27T11:49:36.513 INFO:tasks.workunit.client.0.smithi195.stdout: create 298/30 file after the snapshot 2022-06-27T11:49:36.528 INFO:tasks.workunit.client.0.smithi195.stdout: create 298/31 file after the snapshot 2022-06-27T11:49:36.529 INFO:tasks.workunit.client.0.smithi195.stdout: create 298/32 file after the snapshot 2022-06-27T11:49:36.530 INFO:tasks.workunit.client.0.smithi195.stdout: create 298/33 file after the snapshot 2022-06-27T11:49:36.531 INFO:tasks.workunit.client.0.smithi195.stdout: create 298/34 file after the snapshot 2022-06-27T11:49:36.532 INFO:tasks.workunit.client.0.smithi195.stdout: create 298/35 file after the snapshot 2022-06-27T11:49:36.532 INFO:tasks.workunit.client.0.smithi195.stdout: create 298/36 file after the snapshot 2022-06-27T11:49:36.534 INFO:tasks.workunit.client.0.smithi195.stdout: create 298/37 file after the snapshot 2022-06-27T11:49:36.535 INFO:tasks.workunit.client.0.smithi195.stdout: create 298/38 file after the snapshot 2022-06-27T11:49:36.535 INFO:tasks.workunit.client.0.smithi195.stdout: create 298/39 file after the snapshot 2022-06-27T11:49:36.536 INFO:tasks.workunit.client.0.smithi195.stdout: create 299/.snap/snap-subdir-test 2022-06-27T11:49:36.566 INFO:tasks.workunit.client.0.smithi195.stdout: create 299/30 file after the snapshot 2022-06-27T11:49:36.581 INFO:tasks.workunit.client.0.smithi195.stdout: create 299/31 file after the snapshot 2022-06-27T11:49:36.581 INFO:tasks.workunit.client.0.smithi195.stdout: create 299/32 file after the snapshot 2022-06-27T11:49:36.582 INFO:tasks.workunit.client.0.smithi195.stdout: create 299/33 file after the snapshot 2022-06-27T11:49:36.582 INFO:tasks.workunit.client.0.smithi195.stdout: create 299/34 file after the snapshot 2022-06-27T11:49:36.583 INFO:tasks.workunit.client.0.smithi195.stdout: create 299/35 file after the snapshot 2022-06-27T11:49:36.584 INFO:tasks.workunit.client.0.smithi195.stdout: create 299/36 file after the snapshot 2022-06-27T11:49:36.585 INFO:tasks.workunit.client.0.smithi195.stdout: create 299/37 file after the snapshot 2022-06-27T11:49:36.586 INFO:tasks.workunit.client.0.smithi195.stdout: create 299/38 file after the snapshot 2022-06-27T11:49:36.587 INFO:tasks.workunit.client.0.smithi195.stdout: create 299/39 file after the snapshot 2022-06-27T11:49:36.588 INFO:tasks.workunit.client.0.smithi195.stdout:Take second snapshot .snap/test2 2022-06-27T11:49:36.642 INFO:tasks.workunit.client.0.smithi195.stdout:Copy content of .snap/test1 to copyofsnap1 ... 2022-06-27T11:49:41.096 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1' -> 'copyofsnap1/test1' 2022-06-27T11:49:41.096 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/100' -> 'copyofsnap1/test1/100' 2022-06-27T11:49:41.097 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/100/10' -> 'copyofsnap1/test1/100/10' 2022-06-27T11:49:41.097 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/100/11' -> 'copyofsnap1/test1/100/11' 2022-06-27T11:49:41.097 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/100/12' -> 'copyofsnap1/test1/100/12' 2022-06-27T11:49:41.098 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/100/13' -> 'copyofsnap1/test1/100/13' 2022-06-27T11:49:41.098 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/100/14' -> 'copyofsnap1/test1/100/14' 2022-06-27T11:49:41.098 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/100/15' -> 'copyofsnap1/test1/100/15' 2022-06-27T11:49:41.098 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/100/16' -> 'copyofsnap1/test1/100/16' 2022-06-27T11:49:41.099 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/100/17' -> 'copyofsnap1/test1/100/17' 2022-06-27T11:49:41.100 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/100/18' -> 'copyofsnap1/test1/100/18' 2022-06-27T11:49:41.101 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/100/19' -> 'copyofsnap1/test1/100/19' 2022-06-27T11:49:41.101 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/100/20' -> 'copyofsnap1/test1/100/20' 2022-06-27T11:49:41.102 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/101' -> 'copyofsnap1/test1/101' 2022-06-27T11:49:41.102 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/101/10' -> 'copyofsnap1/test1/101/10' 2022-06-27T11:49:41.103 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/101/11' -> 'copyofsnap1/test1/101/11' 2022-06-27T11:49:41.103 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/101/12' -> 'copyofsnap1/test1/101/12' 2022-06-27T11:49:41.104 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/101/13' -> 'copyofsnap1/test1/101/13' 2022-06-27T11:49:41.104 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/101/14' -> 'copyofsnap1/test1/101/14' 2022-06-27T11:49:41.105 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/101/15' -> 'copyofsnap1/test1/101/15' 2022-06-27T11:49:41.107 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/101/16' -> 'copyofsnap1/test1/101/16' 2022-06-27T11:49:41.107 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/101/17' -> 'copyofsnap1/test1/101/17' 2022-06-27T11:49:41.108 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/101/18' -> 'copyofsnap1/test1/101/18' 2022-06-27T11:49:41.108 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/101/19' -> 'copyofsnap1/test1/101/19' 2022-06-27T11:49:41.108 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/101/20' -> 'copyofsnap1/test1/101/20' 2022-06-27T11:49:41.109 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/102' -> 'copyofsnap1/test1/102' 2022-06-27T11:49:41.109 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/102/10' -> 'copyofsnap1/test1/102/10' 2022-06-27T11:49:41.110 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/102/11' -> 'copyofsnap1/test1/102/11' 2022-06-27T11:49:41.110 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/102/12' -> 'copyofsnap1/test1/102/12' 2022-06-27T11:49:41.111 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/102/13' -> 'copyofsnap1/test1/102/13' 2022-06-27T11:49:41.111 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/102/14' -> 'copyofsnap1/test1/102/14' 2022-06-27T11:49:41.112 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/102/15' -> 'copyofsnap1/test1/102/15' 2022-06-27T11:49:41.112 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/102/16' -> 'copyofsnap1/test1/102/16' 2022-06-27T11:49:41.113 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/102/17' -> 'copyofsnap1/test1/102/17' 2022-06-27T11:49:41.113 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/102/18' -> 'copyofsnap1/test1/102/18' 2022-06-27T11:49:41.114 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/102/19' -> 'copyofsnap1/test1/102/19' 2022-06-27T11:49:41.114 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/102/20' -> 'copyofsnap1/test1/102/20' 2022-06-27T11:49:41.115 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/103' -> 'copyofsnap1/test1/103' 2022-06-27T11:49:41.115 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/103/10' -> 'copyofsnap1/test1/103/10' 2022-06-27T11:49:41.116 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/103/11' -> 'copyofsnap1/test1/103/11' 2022-06-27T11:49:41.116 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/103/12' -> 'copyofsnap1/test1/103/12' 2022-06-27T11:49:41.116 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/103/13' -> 'copyofsnap1/test1/103/13' 2022-06-27T11:49:41.117 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/103/14' -> 'copyofsnap1/test1/103/14' 2022-06-27T11:49:41.117 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/103/15' -> 'copyofsnap1/test1/103/15' 2022-06-27T11:49:41.118 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/103/16' -> 'copyofsnap1/test1/103/16' 2022-06-27T11:49:41.118 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/103/17' -> 'copyofsnap1/test1/103/17' 2022-06-27T11:49:41.119 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/103/18' -> 'copyofsnap1/test1/103/18' 2022-06-27T11:49:41.119 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/103/19' -> 'copyofsnap1/test1/103/19' 2022-06-27T11:49:41.120 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/103/20' -> 'copyofsnap1/test1/103/20' 2022-06-27T11:49:41.120 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/104' -> 'copyofsnap1/test1/104' 2022-06-27T11:49:41.121 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/104/10' -> 'copyofsnap1/test1/104/10' 2022-06-27T11:49:41.121 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/104/11' -> 'copyofsnap1/test1/104/11' 2022-06-27T11:49:41.122 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/104/12' -> 'copyofsnap1/test1/104/12' 2022-06-27T11:49:41.122 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/104/13' -> 'copyofsnap1/test1/104/13' 2022-06-27T11:49:41.123 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/104/14' -> 'copyofsnap1/test1/104/14' 2022-06-27T11:49:41.123 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/104/15' -> 'copyofsnap1/test1/104/15' 2022-06-27T11:49:41.123 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/104/16' -> 'copyofsnap1/test1/104/16' 2022-06-27T11:49:41.124 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/104/17' -> 'copyofsnap1/test1/104/17' 2022-06-27T11:49:41.124 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/104/18' -> 'copyofsnap1/test1/104/18' 2022-06-27T11:49:41.125 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/104/19' -> 'copyofsnap1/test1/104/19' 2022-06-27T11:49:41.125 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/104/20' -> 'copyofsnap1/test1/104/20' 2022-06-27T11:49:41.126 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/105' -> 'copyofsnap1/test1/105' 2022-06-27T11:49:41.126 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/105/10' -> 'copyofsnap1/test1/105/10' 2022-06-27T11:49:41.127 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/105/11' -> 'copyofsnap1/test1/105/11' 2022-06-27T11:49:41.127 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/105/12' -> 'copyofsnap1/test1/105/12' 2022-06-27T11:49:41.128 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/105/13' -> 'copyofsnap1/test1/105/13' 2022-06-27T11:49:41.128 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/105/14' -> 'copyofsnap1/test1/105/14' 2022-06-27T11:49:41.129 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/105/15' -> 'copyofsnap1/test1/105/15' 2022-06-27T11:49:41.129 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/105/16' -> 'copyofsnap1/test1/105/16' 2022-06-27T11:49:41.129 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/105/17' -> 'copyofsnap1/test1/105/17' 2022-06-27T11:49:41.130 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/105/18' -> 'copyofsnap1/test1/105/18' 2022-06-27T11:49:41.130 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/105/19' -> 'copyofsnap1/test1/105/19' 2022-06-27T11:49:41.131 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/105/20' -> 'copyofsnap1/test1/105/20' 2022-06-27T11:49:41.131 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/106' -> 'copyofsnap1/test1/106' 2022-06-27T11:49:41.132 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/106/10' -> 'copyofsnap1/test1/106/10' 2022-06-27T11:49:41.132 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/106/11' -> 'copyofsnap1/test1/106/11' 2022-06-27T11:49:41.133 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/106/12' -> 'copyofsnap1/test1/106/12' 2022-06-27T11:49:41.133 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/106/13' -> 'copyofsnap1/test1/106/13' 2022-06-27T11:49:41.134 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/106/14' -> 'copyofsnap1/test1/106/14' 2022-06-27T11:49:41.134 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/106/15' -> 'copyofsnap1/test1/106/15' 2022-06-27T11:49:41.135 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/106/16' -> 'copyofsnap1/test1/106/16' 2022-06-27T11:49:42.282 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/106/17' -> 'copyofsnap1/test1/106/17' 2022-06-27T11:49:42.283 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/106/18' -> 'copyofsnap1/test1/106/18' 2022-06-27T11:49:42.283 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/106/19' -> 'copyofsnap1/test1/106/19' 2022-06-27T11:49:42.284 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/106/20' -> 'copyofsnap1/test1/106/20' 2022-06-27T11:49:42.284 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/107' -> 'copyofsnap1/test1/107' 2022-06-27T11:49:42.285 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/107/10' -> 'copyofsnap1/test1/107/10' 2022-06-27T11:49:42.285 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/107/11' -> 'copyofsnap1/test1/107/11' 2022-06-27T11:49:42.286 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/107/12' -> 'copyofsnap1/test1/107/12' 2022-06-27T11:49:42.286 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/107/13' -> 'copyofsnap1/test1/107/13' 2022-06-27T11:49:42.286 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/107/14' -> 'copyofsnap1/test1/107/14' 2022-06-27T11:49:42.287 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/107/15' -> 'copyofsnap1/test1/107/15' 2022-06-27T11:49:42.287 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/107/16' -> 'copyofsnap1/test1/107/16' 2022-06-27T11:49:42.288 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/107/17' -> 'copyofsnap1/test1/107/17' 2022-06-27T11:49:42.288 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/107/18' -> 'copyofsnap1/test1/107/18' 2022-06-27T11:49:42.288 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/107/19' -> 'copyofsnap1/test1/107/19' 2022-06-27T11:49:42.289 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/107/20' -> 'copyofsnap1/test1/107/20' 2022-06-27T11:49:42.289 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/108' -> 'copyofsnap1/test1/108' 2022-06-27T11:49:42.290 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/108/10' -> 'copyofsnap1/test1/108/10' 2022-06-27T11:49:42.290 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/108/11' -> 'copyofsnap1/test1/108/11' 2022-06-27T11:49:42.290 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/108/12' -> 'copyofsnap1/test1/108/12' 2022-06-27T11:49:42.291 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/108/13' -> 'copyofsnap1/test1/108/13' 2022-06-27T11:49:42.291 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/108/14' -> 'copyofsnap1/test1/108/14' 2022-06-27T11:49:42.292 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/108/15' -> 'copyofsnap1/test1/108/15' 2022-06-27T11:49:42.292 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/108/16' -> 'copyofsnap1/test1/108/16' 2022-06-27T11:49:42.292 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/108/17' -> 'copyofsnap1/test1/108/17' 2022-06-27T11:49:42.293 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/108/18' -> 'copyofsnap1/test1/108/18' 2022-06-27T11:49:42.294 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/108/19' -> 'copyofsnap1/test1/108/19' 2022-06-27T11:49:42.295 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/108/20' -> 'copyofsnap1/test1/108/20' 2022-06-27T11:49:42.295 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/109' -> 'copyofsnap1/test1/109' 2022-06-27T11:49:42.295 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/109/10' -> 'copyofsnap1/test1/109/10' 2022-06-27T11:49:42.295 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/109/11' -> 'copyofsnap1/test1/109/11' 2022-06-27T11:49:42.296 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/109/12' -> 'copyofsnap1/test1/109/12' 2022-06-27T11:49:42.296 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/109/13' -> 'copyofsnap1/test1/109/13' 2022-06-27T11:49:42.296 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/109/14' -> 'copyofsnap1/test1/109/14' 2022-06-27T11:49:42.296 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/109/15' -> 'copyofsnap1/test1/109/15' 2022-06-27T11:49:42.297 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/109/16' -> 'copyofsnap1/test1/109/16' 2022-06-27T11:49:42.297 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/109/17' -> 'copyofsnap1/test1/109/17' 2022-06-27T11:49:42.297 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/109/18' -> 'copyofsnap1/test1/109/18' 2022-06-27T11:49:42.298 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/109/19' -> 'copyofsnap1/test1/109/19' 2022-06-27T11:49:42.298 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/109/20' -> 'copyofsnap1/test1/109/20' 2022-06-27T11:49:42.298 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/110' -> 'copyofsnap1/test1/110' 2022-06-27T11:49:42.298 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/110/10' -> 'copyofsnap1/test1/110/10' 2022-06-27T11:49:42.299 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/110/11' -> 'copyofsnap1/test1/110/11' 2022-06-27T11:49:42.299 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/110/12' -> 'copyofsnap1/test1/110/12' 2022-06-27T11:49:42.299 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/110/13' -> 'copyofsnap1/test1/110/13' 2022-06-27T11:49:42.299 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/110/14' -> 'copyofsnap1/test1/110/14' 2022-06-27T11:49:42.300 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/110/15' -> 'copyofsnap1/test1/110/15' 2022-06-27T11:49:42.300 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/110/16' -> 'copyofsnap1/test1/110/16' 2022-06-27T11:49:42.300 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/110/17' -> 'copyofsnap1/test1/110/17' 2022-06-27T11:49:42.300 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/110/18' -> 'copyofsnap1/test1/110/18' 2022-06-27T11:49:42.301 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/110/19' -> 'copyofsnap1/test1/110/19' 2022-06-27T11:49:42.301 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/110/20' -> 'copyofsnap1/test1/110/20' 2022-06-27T11:49:42.301 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/111' -> 'copyofsnap1/test1/111' 2022-06-27T11:49:42.301 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/111/10' -> 'copyofsnap1/test1/111/10' 2022-06-27T11:49:42.302 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/111/11' -> 'copyofsnap1/test1/111/11' 2022-06-27T11:49:42.302 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/111/12' -> 'copyofsnap1/test1/111/12' 2022-06-27T11:49:42.302 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/111/13' -> 'copyofsnap1/test1/111/13' 2022-06-27T11:49:42.302 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/111/14' -> 'copyofsnap1/test1/111/14' 2022-06-27T11:49:42.302 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/111/15' -> 'copyofsnap1/test1/111/15' 2022-06-27T11:49:42.303 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/111/16' -> 'copyofsnap1/test1/111/16' 2022-06-27T11:49:42.305 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/111/17' -> 'copyofsnap1/test1/111/17' 2022-06-27T11:49:42.305 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/111/18' -> 'copyofsnap1/test1/111/18' 2022-06-27T11:49:42.305 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/111/19' -> 'copyofsnap1/test1/111/19' 2022-06-27T11:49:42.306 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/111/20' -> 'copyofsnap1/test1/111/20' 2022-06-27T11:49:42.306 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/112' -> 'copyofsnap1/test1/112' 2022-06-27T11:49:42.306 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/112/10' -> 'copyofsnap1/test1/112/10' 2022-06-27T11:49:42.306 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/112/11' -> 'copyofsnap1/test1/112/11' 2022-06-27T11:49:42.307 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/112/12' -> 'copyofsnap1/test1/112/12' 2022-06-27T11:49:42.307 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/112/13' -> 'copyofsnap1/test1/112/13' 2022-06-27T11:49:42.307 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/112/14' -> 'copyofsnap1/test1/112/14' 2022-06-27T11:49:42.307 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/112/15' -> 'copyofsnap1/test1/112/15' 2022-06-27T11:49:42.307 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/112/16' -> 'copyofsnap1/test1/112/16' 2022-06-27T11:49:42.308 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/112/17' -> 'copyofsnap1/test1/112/17' 2022-06-27T11:49:42.308 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/112/18' -> 'copyofsnap1/test1/112/18' 2022-06-27T11:49:42.308 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/112/19' -> 'copyofsnap1/test1/112/19' 2022-06-27T11:49:42.308 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/112/20' -> 'copyofsnap1/test1/112/20' 2022-06-27T11:49:42.309 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/113' -> 'copyofsnap1/test1/113' 2022-06-27T11:49:42.309 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/113/10' -> 'copyofsnap1/test1/113/10' 2022-06-27T11:49:42.309 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/113/11' -> 'copyofsnap1/test1/113/11' 2022-06-27T11:49:42.309 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/113/12' -> 'copyofsnap1/test1/113/12' 2022-06-27T11:49:42.309 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/113/13' -> 'copyofsnap1/test1/113/13' 2022-06-27T11:49:42.884 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:49:43.029 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.019+0000 7fbf3eb90700 1 Processor -- start 2022-06-27T11:49:43.030 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.019+0000 7fbf3eb90700 1 -- start start 2022-06-27T11:49:43.031 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.020+0000 7fbf3eb90700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbf380c3f50 0x7fbf3815ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:49:43.031 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.020+0000 7fbf3eb90700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf380c33b0 con 0x7fbf3814e860 2022-06-27T11:49:43.032 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.020+0000 7fbf3eb90700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf3806f220 con 0x7fbf3814e490 2022-06-27T11:49:43.032 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.020+0000 7fbf3eb90700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf381542c0 con 0x7fbf380c3f50 2022-06-27T11:49:43.033 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.020+0000 7fbf37fff700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fbf3814e860 0x7fbf380c3840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:48636/0 (socket says 172.21.15.195:48636) 2022-06-27T11:49:43.033 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.020+0000 7fbf37fff700 1 -- 172.21.15.195:0/2112577304 learned_addr learned my addr 172.21.15.195:0/2112577304 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:49:43.034 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.020+0000 7fbf3d12d700 1 --2- 172.21.15.195:0/2112577304 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbf380c3f50 0x7fbf3815ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:49:43.034 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.021+0000 7fbf377fe700 1 -- 172.21.15.195:0/2112577304 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1684118826 0 0) 0x7fbf380c33b0 con 0x7fbf3814e860 2022-06-27T11:49:43.034 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.021+0000 7fbf377fe700 1 -- 172.21.15.195:0/2112577304 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf1c003430 con 0x7fbf3814e860 2022-06-27T11:49:43.035 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.021+0000 7fbf3d12d700 1 -- 172.21.15.195:0/2112577304 >> v1:172.21.15.202:6789/0 conn(0x7fbf3814e490 legacy=0x7fbf3806cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:49:43.036 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.021+0000 7fbf3d12d700 1 -- 172.21.15.195:0/2112577304 >> v1:172.21.15.195:6789/0 conn(0x7fbf3814e860 legacy=0x7fbf380c3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:49:43.036 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.021+0000 7fbf3d12d700 1 -- 172.21.15.195:0/2112577304 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf38154560 con 0x7fbf380c3f50 2022-06-27T11:49:43.037 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.021+0000 7fbf3d12d700 1 --2- 172.21.15.195:0/2112577304 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbf380c3f50 0x7fbf3815ec30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbf30009e90 tx=0x7fbf30009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2478838a36264c99 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:49:43.037 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.021+0000 7fbf377fe700 1 -- 172.21.15.195:0/2112577304 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf3000e070 con 0x7fbf380c3f50 2022-06-27T11:49:43.038 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.021+0000 7fbf377fe700 1 -- 172.21.15.195:0/2112577304 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf3000b920 con 0x7fbf380c3f50 2022-06-27T11:49:43.038 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.021+0000 7fbf377fe700 1 -- 172.21.15.195:0/2112577304 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf240013f0 con 0x7fbf380c3f50 2022-06-27T11:49:43.039 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.022+0000 7fbf3eb90700 1 -- 172.21.15.195:0/2112577304 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbf380c3f50 msgr2=0x7fbf3815ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:49:43.039 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.022+0000 7fbf3eb90700 1 --2- 172.21.15.195:0/2112577304 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbf380c3f50 0x7fbf3815ec30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbf30009e90 tx=0x7fbf30009780 comp rx=0 tx=0).stop 2022-06-27T11:49:43.039 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.022+0000 7fbf3eb90700 1 -- 172.21.15.195:0/2112577304 shutdown_connections 2022-06-27T11:49:43.040 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.022+0000 7fbf3eb90700 1 --2- 172.21.15.195:0/2112577304 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbf380c3f50 0x7fbf3815ec30 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:43.040 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.022+0000 7fbf3eb90700 1 -- 172.21.15.195:0/2112577304 >> 172.21.15.195:0/2112577304 conn(0x7fbf38063af0 msgr2=0x7fbf38063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:49:43.041 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.022+0000 7fbf3eb90700 1 -- 172.21.15.195:0/2112577304 shutdown_connections 2022-06-27T11:49:43.041 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.022+0000 7fbf3eb90700 1 -- 172.21.15.195:0/2112577304 wait complete. 2022-06-27T11:49:43.041 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.023+0000 7fbf3eb90700 1 Processor -- start 2022-06-27T11:49:43.042 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.023+0000 7fbf3eb90700 1 -- start start 2022-06-27T11:49:43.042 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.024+0000 7fbf3eb90700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbf380c3f50 0x7fbf3814ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:49:43.042 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.024+0000 7fbf3eb90700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbf3814e490 0x7fbf380c8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:49:43.043 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.024+0000 7fbf3eb90700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbf3814e860 0x7fbf38156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:49:43.043 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.024+0000 7fbf3eb90700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf38156a60 con 0x7fbf3814e490 2022-06-27T11:49:43.043 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.024+0000 7fbf3eb90700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf381685a0 con 0x7fbf3814e860 2022-06-27T11:49:43.043 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.024+0000 7fbf3eb90700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf3814bf80 con 0x7fbf380c3f50 2022-06-27T11:49:43.044 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.024+0000 7fbf37fff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbf3814e490 0x7fbf380c8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:49:43.044 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.024+0000 7fbf37fff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbf3814e490 0x7fbf380c8a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:42454/0 (socket says 172.21.15.195:42454) 2022-06-27T11:49:43.044 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.024+0000 7fbf3d12d700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbf3814e860 0x7fbf38156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:49:43.045 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.024+0000 7fbf37fff700 1 -- 172.21.15.195:0/4020862804 learned_addr learned my addr 172.21.15.195:0/4020862804 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:49:43.045 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.024+0000 7fbf3c92c700 1 --2- 172.21.15.195:0/4020862804 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbf380c3f50 0x7fbf3814ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:49:43.046 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.024+0000 7fbf37fff700 1 -- 172.21.15.195:0/4020862804 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbf380c3f50 msgr2=0x7fbf3814ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:49:43.046 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.024+0000 7fbf37fff700 1 --2- 172.21.15.195:0/4020862804 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbf380c3f50 0x7fbf3814ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:43.047 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.024+0000 7fbf37fff700 1 -- 172.21.15.195:0/4020862804 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbf3814e860 msgr2=0x7fbf38156ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:49:43.047 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.024+0000 7fbf37fff700 1 --2- 172.21.15.195:0/4020862804 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbf3814e860 0x7fbf38156ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:43.048 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.024+0000 7fbf37fff700 1 -- 172.21.15.195:0/4020862804 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf30009a20 con 0x7fbf3814e490 2022-06-27T11:49:43.048 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.024+0000 7fbf37fff700 1 --2- 172.21.15.195:0/4020862804 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbf3814e490 0x7fbf380c8a60 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fbf240011e0 tx=0x7fbf24001b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:49:43.048 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.024+0000 7fbf34ff9700 1 -- 172.21.15.195:0/4020862804 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf24007360 con 0x7fbf3814e490 2022-06-27T11:49:43.049 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.025+0000 7fbf34ff9700 1 -- 172.21.15.195:0/4020862804 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf240074e0 con 0x7fbf3814e490 2022-06-27T11:49:43.049 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.025+0000 7fbf34ff9700 1 -- 172.21.15.195:0/4020862804 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf24004940 con 0x7fbf3814e490 2022-06-27T11:49:43.049 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.025+0000 7fbf3eb90700 1 -- 172.21.15.195:0/4020862804 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf3814dc70 con 0x7fbf3814e490 2022-06-27T11:49:43.050 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.025+0000 7fbf3eb90700 1 -- 172.21.15.195:0/4020862804 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf38157e60 con 0x7fbf3814e490 2022-06-27T11:49:43.050 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.025+0000 7fbf3eb90700 1 -- 172.21.15.195:0/4020862804 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf38156c10 con 0x7fbf3814e490 2022-06-27T11:49:43.051 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.026+0000 7fbf34ff9700 1 -- 172.21.15.195:0/4020862804 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fbf2401a030 con 0x7fbf3814e490 2022-06-27T11:49:43.051 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.026+0000 7fbf34ff9700 1 --2- 172.21.15.195:0/4020862804 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbf20063d00 0x7fbf200661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:49:43.051 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.026+0000 7fbf34ff9700 1 -- 172.21.15.195:0/4020862804 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6144+0+0 (secure 0 0 0) 0x7fbf2407fe20 con 0x7fbf3814e490 2022-06-27T11:49:43.052 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.026+0000 7fbf3c92c700 1 --2- 172.21.15.195:0/4020862804 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbf20063d00 0x7fbf200661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:49:43.052 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.027+0000 7fbf3c92c700 1 --2- 172.21.15.195:0/4020862804 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbf20063d00 0x7fbf200661b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c001900 tx=0x7fbf2c009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:49:43.053 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.030+0000 7fbf34ff9700 1 -- 172.21.15.195:0/4020862804 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fbf38156c10 con 0x7fbf3814e490 2022-06-27T11:49:43.256 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.257+0000 7fbf3eb90700 1 -- 172.21.15.195:0/4020862804 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf380722c0 con 0x7fbf3814e490 2022-06-27T11:49:43.257 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.258+0000 7fbf34ff9700 1 -- 172.21.15.195:0/4020862804 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6119 (secure 0 0 0) 0x7fbf24018030 con 0x7fbf3814e490 2022-06-27T11:49:43.258 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:49:43.258 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4287,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/864126677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":864126677},{"type":"v1","addr":"172.21.15.202:6833","nonce":864126677}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:47:57.373108+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.202:6835/2038169702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2038169702},{"type":"v1","addr":"172.21.15.202:6835","nonce":2038169702}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:49:43.260 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.260+0000 7fbf3eb90700 1 -- 172.21.15.195:0/4020862804 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbf20063d00 msgr2=0x7fbf200661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:49:43.260 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.260+0000 7fbf3eb90700 1 --2- 172.21.15.195:0/4020862804 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbf20063d00 0x7fbf200661b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c001900 tx=0x7fbf2c009040 comp rx=0 tx=0).stop 2022-06-27T11:49:43.261 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.260+0000 7fbf3eb90700 1 -- 172.21.15.195:0/4020862804 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbf3814e490 msgr2=0x7fbf380c8a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:49:43.261 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.260+0000 7fbf3eb90700 1 --2- 172.21.15.195:0/4020862804 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbf3814e490 0x7fbf380c8a60 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fbf240011e0 tx=0x7fbf24001b90 comp rx=0 tx=0).stop 2022-06-27T11:49:43.261 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.260+0000 7fbf3eb90700 1 -- 172.21.15.195:0/4020862804 shutdown_connections 2022-06-27T11:49:43.262 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.260+0000 7fbf3eb90700 1 --2- 172.21.15.195:0/4020862804 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbf20063d00 0x7fbf200661b0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:43.262 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.260+0000 7fbf3eb90700 1 --2- 172.21.15.195:0/4020862804 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbf3814e860 0x7fbf38156ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:43.262 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.260+0000 7fbf3eb90700 1 --2- 172.21.15.195:0/4020862804 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbf380c3f50 0x7fbf3814ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:43.262 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.260+0000 7fbf3eb90700 1 --2- 172.21.15.195:0/4020862804 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbf3814e490 0x7fbf380c8a60 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:43.262 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.260+0000 7fbf3eb90700 1 -- 172.21.15.195:0/4020862804 >> 172.21.15.195:0/4020862804 conn(0x7fbf38063af0 msgr2=0x7fbf3814d350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:49:43.263 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.260+0000 7fbf3eb90700 1 -- 172.21.15.195:0/4020862804 shutdown_connections 2022-06-27T11:49:43.263 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.260+0000 7fbf3eb90700 1 -- 172.21.15.195:0/4020862804 wait complete. 2022-06-27T11:49:43.263 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 8 2022-06-27T11:49:43.271 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.d (rank=0) 2022-06-27T11:49:43.272 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2022-06-27T11:49:43.272 INFO:teuthology.orchestra.run:waiting for 300 2022-06-27T11:49:43.287 INFO:tasks.ceph.mds.d:Stopped 2022-06-27T11:49:43.288 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:49:43.411 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.412+0000 7fe408ce6700 1 Processor -- start 2022-06-27T11:49:43.412 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.412+0000 7fe408ce6700 1 -- start start 2022-06-27T11:49:43.420 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.413+0000 7fe408ce6700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fe40406fef0 0x7fe4041570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:49:43.420 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.413+0000 7fe408ce6700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe4040c34a0 0x7fe4040c2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:49:43.420 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.413+0000 7fe408ce6700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe40406fba0 con 0x7fe40406e850 2022-06-27T11:49:43.421 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.413+0000 7fe408ce6700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4041575e0 con 0x7fe40406fef0 2022-06-27T11:49:43.421 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.413+0000 7fe408ce6700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe40406f5e0 con 0x7fe4040c34a0 2022-06-27T11:49:43.421 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.413+0000 7fe401d9b700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fe40406fef0 0x7fe4041570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:49:43.422 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.413+0000 7fe402d9d700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe4040c34a0 0x7fe4040c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:49:43.422 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.413+0000 7fe40259c700 1 --1- >> v1:172.21.15.202:6789/0 conn(0x7fe40406e850 0x7fe404069de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.202:6789/0 says I am v1:172.21.15.195:50508/0 (socket says 172.21.15.195:50508) 2022-06-27T11:49:43.422 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.413+0000 7fe40259c700 1 -- 172.21.15.195:0/2424369811 learned_addr learned my addr 172.21.15.195:0/2424369811 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:49:43.423 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.413+0000 7fe401d9b700 1 -- 172.21.15.195:0/2424369811 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe4040c34a0 msgr2=0x7fe4040c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:49:43.424 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.413+0000 7fe401d9b700 1 --2- 172.21.15.195:0/2424369811 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe4040c34a0 0x7fe4040c2920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:43.424 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.413+0000 7fe401d9b700 1 -- 172.21.15.195:0/2424369811 >> v1:172.21.15.202:6789/0 conn(0x7fe40406e850 legacy=0x7fe404069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:49:43.424 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.413+0000 7fe401d9b700 1 -- 172.21.15.195:0/2424369811 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe40406f8c0 con 0x7fe40406fef0 2022-06-27T11:49:43.424 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.413+0000 7fe402d9d700 1 --2- 172.21.15.195:0/2424369811 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe4040c34a0 0x7fe4040c2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:49:43.425 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.414+0000 7fe401d9b700 1 --2- 172.21.15.195:0/2424369811 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe40406fef0 0x7fe4041570a0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe3f80097a0 tx=0x7fe3f8009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5017c0464e4300a2 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:49:43.425 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.414+0000 7fe40159a700 1 -- 172.21.15.195:0/2424369811 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3f8011630 con 0x7fe40406fef0 2022-06-27T11:49:43.425 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.414+0000 7fe40159a700 1 -- 172.21.15.195:0/2424369811 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3f8011790 con 0x7fe40406fef0 2022-06-27T11:49:43.425 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.414+0000 7fe40159a700 1 -- 172.21.15.195:0/2424369811 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3f8011b30 con 0x7fe40406fef0 2022-06-27T11:49:43.426 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.414+0000 7fe408ce6700 1 -- 172.21.15.195:0/2424369811 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe40406fef0 msgr2=0x7fe4041570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:49:43.426 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.414+0000 7fe408ce6700 1 --2- 172.21.15.195:0/2424369811 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe40406fef0 0x7fe4041570a0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe3f80097a0 tx=0x7fe3f8009060 comp rx=0 tx=0).stop 2022-06-27T11:49:43.426 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.414+0000 7fe408ce6700 1 -- 172.21.15.195:0/2424369811 shutdown_connections 2022-06-27T11:49:43.426 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.414+0000 7fe408ce6700 1 --2- 172.21.15.195:0/2424369811 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe4040c34a0 0x7fe4040c2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:43.427 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.414+0000 7fe408ce6700 1 --2- 172.21.15.195:0/2424369811 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe40406fef0 0x7fe4041570a0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:43.427 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.414+0000 7fe408ce6700 1 -- 172.21.15.195:0/2424369811 >> 172.21.15.195:0/2424369811 conn(0x7fe4040c6350 msgr2=0x7fe4040c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:49:43.427 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.414+0000 7fe408ce6700 1 -- 172.21.15.195:0/2424369811 shutdown_connections 2022-06-27T11:49:43.427 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.414+0000 7fe408ce6700 1 -- 172.21.15.195:0/2424369811 wait complete. 2022-06-27T11:49:43.428 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.415+0000 7fe408ce6700 1 Processor -- start 2022-06-27T11:49:43.428 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.416+0000 7fe408ce6700 1 -- start start 2022-06-27T11:49:43.428 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.416+0000 7fe408ce6700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe40406e850 0x7fe40414e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:49:43.428 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.416+0000 7fe408ce6700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe4040c34a0 0x7fe40415e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:49:43.428 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.416+0000 7fe408ce6700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe404153c00 0x7fe40414cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:49:43.429 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.416+0000 7fe408ce6700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4041490d0 con 0x7fe4040c34a0 2022-06-27T11:49:43.429 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.416+0000 7fe408ce6700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe40414ecd0 con 0x7fe404153c00 2022-06-27T11:49:43.429 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.416+0000 7fe408ce6700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe40414c4a0 con 0x7fe40406e850 2022-06-27T11:49:43.429 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.416+0000 7fe401d9b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe4040c34a0 0x7fe40415e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:49:43.430 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.416+0000 7fe401d9b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe4040c34a0 0x7fe40415e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:42468/0 (socket says 172.21.15.195:42468) 2022-06-27T11:49:43.430 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.416+0000 7fe401d9b700 1 -- 172.21.15.195:0/1823016633 learned_addr learned my addr 172.21.15.195:0/1823016633 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:49:43.430 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.416+0000 7fe40259c700 1 --2- 172.21.15.195:0/1823016633 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe40406e850 0x7fe40414e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:49:43.430 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.417+0000 7fe402d9d700 1 --2- 172.21.15.195:0/1823016633 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe404153c00 0x7fe40414cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:49:43.431 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.417+0000 7fe401d9b700 1 -- 172.21.15.195:0/1823016633 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe40406e850 msgr2=0x7fe40414e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:49:43.431 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.417+0000 7fe401d9b700 1 --2- 172.21.15.195:0/1823016633 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe40406e850 0x7fe40414e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:43.431 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.417+0000 7fe401d9b700 1 -- 172.21.15.195:0/1823016633 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe404153c00 msgr2=0x7fe40414cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:49:43.431 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.417+0000 7fe401d9b700 1 --2- 172.21.15.195:0/1823016633 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe404153c00 0x7fe40414cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:43.431 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.417+0000 7fe401d9b700 1 -- 172.21.15.195:0/1823016633 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3fc009a20 con 0x7fe4040c34a0 2022-06-27T11:49:43.432 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.417+0000 7fe401d9b700 1 --2- 172.21.15.195:0/1823016633 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe4040c34a0 0x7fe40415e2f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe3f8009e40 tx=0x7fe3f8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:49:43.432 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.417+0000 7fe3f27fc700 1 -- 172.21.15.195:0/1823016633 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3f8000c90 con 0x7fe4040c34a0 2022-06-27T11:49:43.432 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.417+0000 7fe3f27fc700 1 -- 172.21.15.195:0/1823016633 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3f8011900 con 0x7fe4040c34a0 2022-06-27T11:49:43.432 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.417+0000 7fe408ce6700 1 -- 172.21.15.195:0/1823016633 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3f80092e0 con 0x7fe4040c34a0 2022-06-27T11:49:43.433 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.417+0000 7fe3f27fc700 1 -- 172.21.15.195:0/1823016633 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3f8025420 con 0x7fe4040c34a0 2022-06-27T11:49:43.433 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.417+0000 7fe408ce6700 1 -- 172.21.15.195:0/1823016633 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe40414e300 con 0x7fe4040c34a0 2022-06-27T11:49:43.433 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.418+0000 7fe408ce6700 1 -- 172.21.15.195:0/1823016633 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe40414e490 con 0x7fe4040c34a0 2022-06-27T11:49:43.433 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.418+0000 7fe3f27fc700 1 -- 172.21.15.195:0/1823016633 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fe3f802c030 con 0x7fe4040c34a0 2022-06-27T11:49:43.433 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.419+0000 7fe3f27fc700 1 --2- 172.21.15.195:0/1823016633 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe3e0063d30 0x7fe3e00661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:49:43.438 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.419+0000 7fe3f27fc700 1 -- 172.21.15.195:0/1823016633 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6144+0+0 (secure 0 0 0) 0x7fe3f8081c50 con 0x7fe4040c34a0 2022-06-27T11:49:43.438 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.419+0000 7fe40259c700 1 --2- 172.21.15.195:0/1823016633 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe3e0063d30 0x7fe3e00661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:49:43.438 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.419+0000 7fe40259c700 1 --2- 172.21.15.195:0/1823016633 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe3e0063d30 0x7fe3e00661e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe3f4000c00 tx=0x7fe3f4009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:49:43.439 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.423+0000 7fe3f27fc700 1 -- 172.21.15.195:0/1823016633 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fe3f80508b0 con 0x7fe4040c34a0 2022-06-27T11:49:43.652 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.653+0000 7fe408ce6700 1 -- 172.21.15.195:0/1823016633 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe4040722c0 con 0x7fe4040c34a0 2022-06-27T11:49:43.654 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.654+0000 7fe3f27fc700 1 -- 172.21.15.195:0/1823016633 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6119 (secure 0 0 0) 0x7fe3f8015090 con 0x7fe4040c34a0 2022-06-27T11:49:43.654 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:49:43.654 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4287,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/864126677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":864126677},{"type":"v1","addr":"172.21.15.202:6833","nonce":864126677}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:47:57.373108+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.202:6835/2038169702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2038169702},{"type":"v1","addr":"172.21.15.202:6835","nonce":2038169702}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:49:43.656 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.656+0000 7fe408ce6700 1 -- 172.21.15.195:0/1823016633 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe3e0063d30 msgr2=0x7fe3e00661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:49:43.656 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.656+0000 7fe408ce6700 1 --2- 172.21.15.195:0/1823016633 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe3e0063d30 0x7fe3e00661e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe3f4000c00 tx=0x7fe3f4009040 comp rx=0 tx=0).stop 2022-06-27T11:49:43.656 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.656+0000 7fe408ce6700 1 -- 172.21.15.195:0/1823016633 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe4040c34a0 msgr2=0x7fe40415e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:49:43.657 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.656+0000 7fe408ce6700 1 --2- 172.21.15.195:0/1823016633 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe4040c34a0 0x7fe40415e2f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe3f8009e40 tx=0x7fe3f8017000 comp rx=0 tx=0).stop 2022-06-27T11:49:43.657 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.656+0000 7fe408ce6700 1 -- 172.21.15.195:0/1823016633 shutdown_connections 2022-06-27T11:49:43.657 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.656+0000 7fe408ce6700 1 --2- 172.21.15.195:0/1823016633 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe3e0063d30 0x7fe3e00661e0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:43.657 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.656+0000 7fe408ce6700 1 --2- 172.21.15.195:0/1823016633 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe404153c00 0x7fe40414cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:43.658 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.656+0000 7fe408ce6700 1 --2- 172.21.15.195:0/1823016633 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe40406e850 0x7fe40414e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:43.658 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.656+0000 7fe408ce6700 1 --2- 172.21.15.195:0/1823016633 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe4040c34a0 0x7fe40415e2f0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:43.658 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.656+0000 7fe408ce6700 1 -- 172.21.15.195:0/1823016633 >> 172.21.15.195:0/1823016633 conn(0x7fe4040c6350 msgr2=0x7fe4040c82d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:49:43.658 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.656+0000 7fe408ce6700 1 -- 172.21.15.195:0/1823016633 shutdown_connections 2022-06-27T11:49:43.659 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:43.656+0000 7fe408ce6700 1 -- 172.21.15.195:0/1823016633 wait complete. 2022-06-27T11:49:43.659 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 8 2022-06-27T11:49:43.668 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-06-27T11:49:45.670 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:49:45.816 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:45.816+0000 7f808f009700 1 Processor -- start 2022-06-27T11:49:45.816 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:45.816+0000 7f808f009700 1 -- start start 2022-06-27T11:49:45.832 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:45.817+0000 7f808f009700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f8088150580 0x7f8088156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:49:45.832 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:45.817+0000 7f808f009700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8088150950 0x7f80880c7b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:49:45.832 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:45.817+0000 7f808f009700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f808806f620 0x7f808806d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:49:45.833 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:45.817+0000 7f808f009700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8088157430 con 0x7f8088150580 2022-06-27T11:49:45.833 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:45.817+0000 7f808f009700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80880c76b0 con 0x7f808806f620 2022-06-27T11:49:45.833 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:45.817+0000 7f808f009700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80880bad70 con 0x7f8088150950 2022-06-27T11:49:45.834 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:45.817+0000 7f808cda5700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f8088150580 0x7f8088156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:49:45.834 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:45.817+0000 7f808cda5700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f8088150580 0x7f8088156ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:42480/0 (socket says 172.21.15.195:42480) 2022-06-27T11:49:45.835 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:45.817+0000 7f808cda5700 1 -- 172.21.15.195:0/2567048707 learned_addr learned my addr 172.21.15.195:0/2567048707 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:49:45.835 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:45.818+0000 7f808d5a6700 1 --2- 172.21.15.195:0/2567048707 >> v2:172.21.15.202:3300/0 conn(0x7f808806f620 0x7f808806d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:49:45.835 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:45.818+0000 7f8087fff700 1 --2- 172.21.15.195:0/2567048707 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8088150950 0x7f80880c7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:49:45.836 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:45.818+0000 7f808cda5700 1 -- 172.21.15.195:0/2567048707 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8088150950 msgr2=0x7f80880c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:49:45.836 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:45.818+0000 7f808cda5700 1 --2- 172.21.15.195:0/2567048707 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8088150950 0x7f80880c7b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:45.837 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:45.818+0000 7f808cda5700 1 -- 172.21.15.195:0/2567048707 >> v2:172.21.15.202:3300/0 conn(0x7f808806f620 msgr2=0x7f808806d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:49:45.837 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:45.818+0000 7f808cda5700 1 --2- 172.21.15.195:0/2567048707 >> v2:172.21.15.202:3300/0 conn(0x7f808806f620 0x7f808806d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:45.838 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:45.818+0000 7f808cda5700 1 -- 172.21.15.195:0/2567048707 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80880ba4d0 con 0x7f8088150580 2022-06-27T11:49:45.838 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:45.818+0000 7f808cda5700 1 --2- 172.21.15.195:0/2567048707 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8088150580 0x7f8088156ef0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8088069430 tx=0x7f807c009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4bf0eecc3e10239 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:49:45.839 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:45.818+0000 7f80877fe700 1 -- 172.21.15.195:0/2567048707 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f807c00b640 con 0x7f8088150580 2022-06-27T11:49:45.839 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:45.818+0000 7f80877fe700 1 -- 172.21.15.195:0/2567048707 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f807c00b7a0 con 0x7f8088150580 2022-06-27T11:49:45.839 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:45.818+0000 7f80877fe700 1 -- 172.21.15.195:0/2567048707 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f807c00bb40 con 0x7f8088150580 2022-06-27T11:49:45.840 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:45.819+0000 7f808f009700 1 -- 172.21.15.195:0/2567048707 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8088150580 msgr2=0x7f8088156ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:49:45.840 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:45.819+0000 7f808f009700 1 --2- 172.21.15.195:0/2567048707 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8088150580 0x7f8088156ef0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8088069430 tx=0x7f807c009120 comp rx=0 tx=0).stop 2022-06-27T11:49:45.841 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:45.819+0000 7f808f009700 1 -- 172.21.15.195:0/2567048707 shutdown_connections 2022-06-27T11:49:45.841 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:45.819+0000 7f808f009700 1 --2- 172.21.15.195:0/2567048707 >> v2:172.21.15.202:3300/0 conn(0x7f808806f620 0x7f808806d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:45.842 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:45.819+0000 7f808f009700 1 --2- 172.21.15.195:0/2567048707 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8088150580 0x7f8088156ef0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:45.843 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:45.819+0000 7f808f009700 1 --2- 172.21.15.195:0/2567048707 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8088150950 0x7f80880c7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:45.843 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:45.819+0000 7f808f009700 1 -- 172.21.15.195:0/2567048707 >> 172.21.15.195:0/2567048707 conn(0x7f8088063af0 msgr2=0x7f8088063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:49:45.843 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:45.819+0000 7f808f009700 1 -- 172.21.15.195:0/2567048707 shutdown_connections 2022-06-27T11:49:45.844 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:45.819+0000 7f808f009700 1 -- 172.21.15.195:0/2567048707 wait complete. 2022-06-27T11:49:45.844 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:45.820+0000 7f808f009700 1 Processor -- start 2022-06-27T11:49:45.845 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:45.820+0000 7f808f009700 1 -- start start 2022-06-27T11:49:45.845 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:45.821+0000 7f808f009700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f808806f620 0x7f80881666e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:49:45.846 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:45.821+0000 7f808f009700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8088150580 0x7f80880c5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:49:45.846 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:45.821+0000 7f808f009700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8088150950 0x7f80880bd7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:49:45.846 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:45.821+0000 7f808f009700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80880c47a0 con 0x7f8088150950 2022-06-27T11:49:45.847 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:45.821+0000 7f808f009700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f808816a3e0 con 0x7f808806f620 2022-06-27T11:49:45.847 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:45.821+0000 7f808f009700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8088169dd0 con 0x7f8088150580 2022-06-27T11:49:45.847 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:45.821+0000 7f808d5a6700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8088150950 0x7f80880bd7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:49:45.848 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:45.821+0000 7f808d5a6700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8088150950 0x7f80880bd7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:42490/0 (socket says 172.21.15.195:42490) 2022-06-27T11:49:45.848 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:45.821+0000 7f808d5a6700 1 -- 172.21.15.195:0/67473743 learned_addr learned my addr 172.21.15.195:0/67473743 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:49:45.849 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:45.821+0000 7f808cda5700 1 --2- 172.21.15.195:0/67473743 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f808806f620 0x7f80881666e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:49:45.849 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:45.821+0000 7f8087fff700 1 --2- 172.21.15.195:0/67473743 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8088150580 0x7f80880c5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:49:45.849 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:45.821+0000 7f808d5a6700 1 -- 172.21.15.195:0/67473743 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8088150580 msgr2=0x7f80880c5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:49:45.849 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:45.821+0000 7f808d5a6700 1 --2- 172.21.15.195:0/67473743 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8088150580 0x7f80880c5ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:45.850 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:45.821+0000 7f808d5a6700 1 -- 172.21.15.195:0/67473743 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f808806f620 msgr2=0x7f80881666e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:49:45.850 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:45.821+0000 7f808d5a6700 1 --2- 172.21.15.195:0/67473743 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f808806f620 0x7f80881666e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:45.850 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:45.821+0000 7f808d5a6700 1 -- 172.21.15.195:0/67473743 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f807c0093a0 con 0x7f8088150950 2022-06-27T11:49:45.851 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:45.821+0000 7f808cda5700 1 --2- 172.21.15.195:0/67473743 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f808806f620 0x7f80881666e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T11:49:45.851 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:45.821+0000 7f808d5a6700 1 --2- 172.21.15.195:0/67473743 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8088150950 0x7f80880bd7f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f8080007f20 tx=0x7f8080009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:49:45.851 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:45.821+0000 7f8084ff9700 1 -- 172.21.15.195:0/67473743 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f808000ec20 con 0x7f8088150950 2022-06-27T11:49:45.851 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:45.821+0000 7f8084ff9700 1 -- 172.21.15.195:0/67473743 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f808000ed80 con 0x7f8088150950 2022-06-27T11:49:45.852 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:45.821+0000 7f808f009700 1 -- 172.21.15.195:0/67473743 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80880c41e0 con 0x7f8088150950 2022-06-27T11:49:45.852 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:45.821+0000 7f8084ff9700 1 -- 172.21.15.195:0/67473743 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8080014670 con 0x7f8088150950 2022-06-27T11:49:45.852 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:45.821+0000 7f808f009700 1 -- 172.21.15.195:0/67473743 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f808816a070 con 0x7f8088150950 2022-06-27T11:49:45.852 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:45.822+0000 7f808f009700 1 -- 172.21.15.195:0/67473743 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8088169c30 con 0x7f8088150950 2022-06-27T11:49:45.853 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:45.823+0000 7f8084ff9700 1 -- 172.21.15.195:0/67473743 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f8080012050 con 0x7f8088150950 2022-06-27T11:49:45.853 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:45.823+0000 7f8084ff9700 1 --2- 172.21.15.195:0/67473743 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8070063d00 0x7f80700661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:49:45.853 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:45.823+0000 7f8084ff9700 1 -- 172.21.15.195:0/67473743 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6144+0+0 (secure 0 0 0) 0x7f8080080ee0 con 0x7f8088150950 2022-06-27T11:49:45.853 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:45.823+0000 7f808cda5700 1 --2- 172.21.15.195:0/67473743 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8070063d00 0x7f80700661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:49:45.853 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:45.824+0000 7f808cda5700 1 --2- 172.21.15.195:0/67473743 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8070063d00 0x7f80700661b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f807c009160 tx=0x7f807c017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:49:45.854 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:45.828+0000 7f8084ff9700 1 -- 172.21.15.195:0/67473743 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f8088169c30 con 0x7f8088150950 2022-06-27T11:49:46.057 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:46.055+0000 7f808f009700 1 -- 172.21.15.195:0/67473743 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f808816a590 con 0x7f8088150950 2022-06-27T11:49:46.059 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:46.055+0000 7f8084ff9700 1 -- 172.21.15.195:0/67473743 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6119 (secure 0 0 0) 0x7f808816a590 con 0x7f8088150950 2022-06-27T11:49:46.062 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:49:46.066 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4287,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/864126677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":864126677},{"type":"v1","addr":"172.21.15.202:6833","nonce":864126677}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:47:57.373108+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.202:6835/2038169702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2038169702},{"type":"v1","addr":"172.21.15.202:6835","nonce":2038169702}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:49:46.069 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:46.058+0000 7f808f009700 1 -- 172.21.15.195:0/67473743 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8070063d00 msgr2=0x7f80700661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:49:46.072 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:46.058+0000 7f808f009700 1 --2- 172.21.15.195:0/67473743 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8070063d00 0x7f80700661b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f807c009160 tx=0x7f807c017000 comp rx=0 tx=0).stop 2022-06-27T11:49:46.072 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:46.058+0000 7f808f009700 1 -- 172.21.15.195:0/67473743 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8088150950 msgr2=0x7f80880bd7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:49:46.073 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:46.058+0000 7f808f009700 1 --2- 172.21.15.195:0/67473743 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8088150950 0x7f80880bd7f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f8080007f20 tx=0x7f8080009840 comp rx=0 tx=0).stop 2022-06-27T11:49:46.073 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:46.058+0000 7f808f009700 1 -- 172.21.15.195:0/67473743 shutdown_connections 2022-06-27T11:49:46.074 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:46.058+0000 7f808f009700 1 --2- 172.21.15.195:0/67473743 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8070063d00 0x7f80700661b0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:46.074 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:46.058+0000 7f808f009700 1 --2- 172.21.15.195:0/67473743 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8088150950 0x7f80880bd7f0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:46.074 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:46.058+0000 7f808f009700 1 --2- 172.21.15.195:0/67473743 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f808806f620 0x7f80881666e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:46.075 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:46.058+0000 7f808f009700 1 --2- 172.21.15.195:0/67473743 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8088150580 0x7f80880c5ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:46.076 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:46.058+0000 7f808f009700 1 -- 172.21.15.195:0/67473743 >> 172.21.15.195:0/67473743 conn(0x7f8088063af0 msgr2=0x7f80880c5060 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:49:46.076 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:46.058+0000 7f808f009700 1 -- 172.21.15.195:0/67473743 shutdown_connections 2022-06-27T11:49:46.076 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:46.058+0000 7f808f009700 1 -- 172.21.15.195:0/67473743 wait complete. 2022-06-27T11:49:46.077 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 8 2022-06-27T11:49:46.079 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-06-27T11:49:48.081 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:49:48.221 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.222+0000 7f3f68b62700 1 Processor -- start 2022-06-27T11:49:48.222 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.222+0000 7f3f68b62700 1 -- start start 2022-06-27T11:49:48.246 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.223+0000 7f3f68b62700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3f640c61e0 0x7f3f640c2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:49:48.246 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.223+0000 7f3f68b62700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f640bc490 con 0x7f3f640c2bf0 2022-06-27T11:49:48.247 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.223+0000 7f3f68b62700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f6406a0e0 con 0x7f3f640c5000 2022-06-27T11:49:48.249 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.223+0000 7f3f68b62700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f640bc6e0 con 0x7f3f640c61e0 2022-06-27T11:49:48.249 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.223+0000 7f3f6259c700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f3f640c2bf0 0x7f3f640c48f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:48686/0 (socket says 172.21.15.195:48686) 2022-06-27T11:49:48.251 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.223+0000 7f3f6259c700 1 -- 172.21.15.195:0/678819694 learned_addr learned my addr 172.21.15.195:0/678819694 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:49:48.251 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.223+0000 7f3f62d9d700 1 --2- 172.21.15.195:0/678819694 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3f640c61e0 0x7f3f640c2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:49:48.252 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.223+0000 7f3f6159a700 1 -- 172.21.15.195:0/678819694 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2448427540 0 0) 0x7f3f640bc490 con 0x7f3f640c2bf0 2022-06-27T11:49:48.252 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.223+0000 7f3f6159a700 1 -- 172.21.15.195:0/678819694 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f48003430 con 0x7f3f640c2bf0 2022-06-27T11:49:48.253 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.223+0000 7f3f6159a700 1 -- 172.21.15.195:0/678819694 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1247894741 0 0) 0x7f3f48003430 con 0x7f3f640c2bf0 2022-06-27T11:49:48.253 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.223+0000 7f3f6159a700 1 -- 172.21.15.195:0/678819694 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3f640c61e0 msgr2=0x7f3f640c2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:49:48.255 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.223+0000 7f3f6159a700 1 --2- 172.21.15.195:0/678819694 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3f640c61e0 0x7f3f640c2fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:48.256 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.223+0000 7f3f6159a700 1 -- 172.21.15.195:0/678819694 >> v1:172.21.15.202:6789/0 conn(0x7f3f640c5000 legacy=0x7f3f640c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:49:48.256 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.223+0000 7f3f6159a700 1 -- 172.21.15.195:0/678819694 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f64158220 con 0x7f3f640c2bf0 2022-06-27T11:49:48.256 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.223+0000 7f3f6159a700 1 -- 172.21.15.195:0/678819694 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7f3f540015f0 con 0x7f3f640c2bf0 2022-06-27T11:49:48.257 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.223+0000 7f3f6159a700 1 -- 172.21.15.195:0/678819694 >> v1:172.21.15.195:6789/0 conn(0x7f3f640c2bf0 legacy=0x7f3f640c48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:49:48.257 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.223+0000 7f3f6159a700 1 --2- 172.21.15.195:0/678819694 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3f640c61e0 0x7f3f48003950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:49:48.257 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.223+0000 7f3f6159a700 1 --2- 172.21.15.195:0/678819694 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3f48003ed0 0x7f3f480046d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:49:48.258 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.223+0000 7f3f6159a700 1 --2- 172.21.15.195:0/678819694 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3f48004c10 0x7f3f48005080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:49:48.258 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.223+0000 7f3f6159a700 1 -- 172.21.15.195:0/678819694 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f640bc6e0 con 0x7f3f48003ed0 2022-06-27T11:49:48.258 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.223+0000 7f3f6159a700 1 -- 172.21.15.195:0/678819694 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4800bb30 con 0x7f3f48004c10 2022-06-27T11:49:48.258 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.223+0000 7f3f6159a700 1 -- 172.21.15.195:0/678819694 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4800bcf0 con 0x7f3f640c61e0 2022-06-27T11:49:48.259 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.223+0000 7f3f61d9b700 1 --2- 172.21.15.195:0/678819694 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3f48003ed0 0x7f3f480046d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:49:48.259 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.223+0000 7f3f6259c700 1 --2- 172.21.15.195:0/678819694 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3f640c61e0 0x7f3f48003950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:49:48.259 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.224+0000 7f3f62d9d700 1 --2- 172.21.15.195:0/678819694 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3f48004c10 0x7f3f48005080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:49:48.259 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.224+0000 7f3f61d9b700 1 -- 172.21.15.195:0/678819694 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3f640c61e0 msgr2=0x7f3f48003950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:49:48.260 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.224+0000 7f3f61d9b700 1 --2- 172.21.15.195:0/678819694 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3f640c61e0 0x7f3f48003950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:48.260 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.224+0000 7f3f61d9b700 1 -- 172.21.15.195:0/678819694 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3f48004c10 msgr2=0x7f3f48005080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:49:48.260 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.224+0000 7f3f61d9b700 1 --2- 172.21.15.195:0/678819694 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3f48004c10 0x7f3f48005080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:48.260 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.224+0000 7f3f61d9b700 1 -- 172.21.15.195:0/678819694 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f48003430 con 0x7f3f48003ed0 2022-06-27T11:49:48.261 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.224+0000 7f3f62d9d700 1 --2- 172.21.15.195:0/678819694 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3f48004c10 0x7f3f48005080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T11:49:48.261 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.224+0000 7f3f61d9b700 1 --2- 172.21.15.195:0/678819694 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3f48003ed0 0x7f3f480046d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f3f640c53f0 tx=0x7f3f58007bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cf756e710f80532 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:49:48.261 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.224+0000 7f3f6159a700 1 -- 172.21.15.195:0/678819694 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f580050d0 con 0x7f3f48003ed0 2022-06-27T11:49:48.261 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.224+0000 7f3f6159a700 1 -- 172.21.15.195:0/678819694 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f58005230 con 0x7f3f48003ed0 2022-06-27T11:49:48.262 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.224+0000 7f3f6159a700 1 -- 172.21.15.195:0/678819694 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f58003750 con 0x7f3f48003ed0 2022-06-27T11:49:48.262 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.224+0000 7f3f68b62700 1 -- 172.21.15.195:0/678819694 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3f48003ed0 msgr2=0x7f3f480046d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:49:48.262 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.224+0000 7f3f68b62700 1 --2- 172.21.15.195:0/678819694 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3f48003ed0 0x7f3f480046d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f3f640c53f0 tx=0x7f3f58007bb0 comp rx=0 tx=0).stop 2022-06-27T11:49:48.262 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.224+0000 7f3f62d9d700 1 -- 172.21.15.195:0/678819694 reap_dead start 2022-06-27T11:49:48.263 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.225+0000 7f3f68b62700 1 -- 172.21.15.195:0/678819694 shutdown_connections 2022-06-27T11:49:48.263 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.225+0000 7f3f68b62700 1 -- 172.21.15.195:0/678819694 >> 172.21.15.195:0/678819694 conn(0x7f3f640c9b00 msgr2=0x7f3f640c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:49:48.263 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.225+0000 7f3f68b62700 1 -- 172.21.15.195:0/678819694 shutdown_connections 2022-06-27T11:49:48.263 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.225+0000 7f3f68b62700 1 -- 172.21.15.195:0/678819694 wait complete. 2022-06-27T11:49:48.263 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.226+0000 7f3f68b62700 1 Processor -- start 2022-06-27T11:49:48.264 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.226+0000 7f3f68b62700 1 -- start start 2022-06-27T11:49:48.264 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.226+0000 7f3f68b62700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3f48003ed0 0x7f3f64159e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:49:48.264 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.226+0000 7f3f68b62700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3f640bb3b0 0x7f3f64169ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:49:48.264 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.226+0000 7f3f68b62700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3f6414b730 0x7f3f641591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:49:48.265 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.226+0000 7f3f68b62700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f64158740 con 0x7f3f48003ed0 2022-06-27T11:49:48.265 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.226+0000 7f3f68b62700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f64158f50 con 0x7f3f6414b730 2022-06-27T11:49:48.265 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.226+0000 7f3f68b62700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f6414b250 con 0x7f3f640bb3b0 2022-06-27T11:49:48.265 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.226+0000 7f3f6259c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3f48003ed0 0x7f3f64159e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:49:48.266 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.227+0000 7f3f6259c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3f48003ed0 0x7f3f64159e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:42510/0 (socket says 172.21.15.195:42510) 2022-06-27T11:49:48.266 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.227+0000 7f3f6259c700 1 -- 172.21.15.195:0/1601339241 learned_addr learned my addr 172.21.15.195:0/1601339241 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:49:48.266 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.227+0000 7f3f62d9d700 1 --2- 172.21.15.195:0/1601339241 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3f6414b730 0x7f3f641591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:49:48.266 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.227+0000 7f3f61d9b700 1 --2- 172.21.15.195:0/1601339241 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3f640bb3b0 0x7f3f64169ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:49:48.266 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.227+0000 7f3f6259c700 1 -- 172.21.15.195:0/1601339241 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3f640bb3b0 msgr2=0x7f3f64169ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:49:48.267 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.227+0000 7f3f6259c700 1 --2- 172.21.15.195:0/1601339241 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3f640bb3b0 0x7f3f64169ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:48.267 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.227+0000 7f3f6259c700 1 -- 172.21.15.195:0/1601339241 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3f6414b730 msgr2=0x7f3f641591d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:49:48.267 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.227+0000 7f3f6259c700 1 --2- 172.21.15.195:0/1601339241 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3f6414b730 0x7f3f641591d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:48.267 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.227+0000 7f3f6259c700 1 -- 172.21.15.195:0/1601339241 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f6415a4d0 con 0x7f3f48003ed0 2022-06-27T11:49:48.268 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.227+0000 7f3f6259c700 1 --2- 172.21.15.195:0/1601339241 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3f48003ed0 0x7f3f64159e90 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3f5400f930 tx=0x7f3f54009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:49:48.268 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.227+0000 7f3f527fc700 1 -- 172.21.15.195:0/1601339241 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f540072e0 con 0x7f3f48003ed0 2022-06-27T11:49:48.268 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.227+0000 7f3f527fc700 1 -- 172.21.15.195:0/1601339241 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f54007460 con 0x7f3f48003ed0 2022-06-27T11:49:48.268 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.227+0000 7f3f68b62700 1 -- 172.21.15.195:0/1601339241 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f64158990 con 0x7f3f48003ed0 2022-06-27T11:49:48.269 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.227+0000 7f3f527fc700 1 -- 172.21.15.195:0/1601339241 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f54005900 con 0x7f3f48003ed0 2022-06-27T11:49:48.269 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.227+0000 7f3f68b62700 1 -- 172.21.15.195:0/1601339241 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f6416a220 con 0x7f3f48003ed0 2022-06-27T11:49:48.269 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.228+0000 7f3f68b62700 1 -- 172.21.15.195:0/1601339241 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f640722c0 con 0x7f3f48003ed0 2022-06-27T11:49:48.269 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.228+0000 7f3f527fc700 1 -- 172.21.15.195:0/1601339241 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f3f54005d30 con 0x7f3f48003ed0 2022-06-27T11:49:48.269 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.229+0000 7f3f527fc700 1 --2- 172.21.15.195:0/1601339241 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3f40063d30 0x7f3f400661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:49:48.270 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.229+0000 7f3f527fc700 1 -- 172.21.15.195:0/1601339241 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6144+0+0 (secure 0 0 0) 0x7f3f540808c0 con 0x7f3f48003ed0 2022-06-27T11:49:48.270 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.229+0000 7f3f61d9b700 1 --2- 172.21.15.195:0/1601339241 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3f40063d30 0x7f3f400661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:49:48.270 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.229+0000 7f3f61d9b700 1 --2- 172.21.15.195:0/1601339241 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3f40063d30 0x7f3f400661e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f3f640bc0c0 tx=0x7f3f58015000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:49:48.270 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.233+0000 7f3f527fc700 1 -- 172.21.15.195:0/1601339241 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f3f5404f460 con 0x7f3f48003ed0 2022-06-27T11:49:48.459 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.460+0000 7f3f68b62700 1 -- 172.21.15.195:0/1601339241 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3f640bbdf0 con 0x7f3f48003ed0 2022-06-27T11:49:48.461 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.460+0000 7f3f527fc700 1 -- 172.21.15.195:0/1601339241 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6119 (secure 0 0 0) 0x7f3f54052a80 con 0x7f3f48003ed0 2022-06-27T11:49:48.462 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:49:48.463 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4287,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/864126677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":864126677},{"type":"v1","addr":"172.21.15.202:6833","nonce":864126677}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:47:57.373108+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.202:6835/2038169702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2038169702},{"type":"v1","addr":"172.21.15.202:6835","nonce":2038169702}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:49:48.464 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.462+0000 7f3f68b62700 1 -- 172.21.15.195:0/1601339241 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3f40063d30 msgr2=0x7f3f400661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:49:48.465 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.462+0000 7f3f68b62700 1 --2- 172.21.15.195:0/1601339241 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3f40063d30 0x7f3f400661e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f3f640bc0c0 tx=0x7f3f58015000 comp rx=0 tx=0).stop 2022-06-27T11:49:48.465 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.462+0000 7f3f68b62700 1 -- 172.21.15.195:0/1601339241 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3f48003ed0 msgr2=0x7f3f64159e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:49:48.465 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.462+0000 7f3f68b62700 1 --2- 172.21.15.195:0/1601339241 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3f48003ed0 0x7f3f64159e90 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3f5400f930 tx=0x7f3f54009040 comp rx=0 tx=0).stop 2022-06-27T11:49:48.466 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.462+0000 7f3f68b62700 1 -- 172.21.15.195:0/1601339241 shutdown_connections 2022-06-27T11:49:48.466 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.462+0000 7f3f68b62700 1 --2- 172.21.15.195:0/1601339241 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3f40063d30 0x7f3f400661e0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:48.467 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.462+0000 7f3f68b62700 1 --2- 172.21.15.195:0/1601339241 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3f48003ed0 0x7f3f64159e90 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:48.468 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.462+0000 7f3f68b62700 1 --2- 172.21.15.195:0/1601339241 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3f6414b730 0x7f3f641591d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:48.468 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.462+0000 7f3f68b62700 1 --2- 172.21.15.195:0/1601339241 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3f640bb3b0 0x7f3f64169ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:48.468 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.462+0000 7f3f68b62700 1 -- 172.21.15.195:0/1601339241 >> 172.21.15.195:0/1601339241 conn(0x7f3f640c9b00 msgr2=0x7f3f6414dac0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:49:48.469 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.463+0000 7f3f68b62700 1 -- 172.21.15.195:0/1601339241 shutdown_connections 2022-06-27T11:49:48.469 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:48.463+0000 7f3f68b62700 1 -- 172.21.15.195:0/1601339241 wait complete. 2022-06-27T11:49:48.469 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 8 2022-06-27T11:49:48.492 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-06-27T11:49:50.505 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:49:50.641 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.638+0000 7f35a0c18700 1 Processor -- start 2022-06-27T11:49:50.641 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.638+0000 7f35a0c18700 1 -- start start 2022-06-27T11:49:50.642 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.639+0000 7f35a0c18700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f359c0c4160 0x7f359c156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:49:50.642 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.639+0000 7f35a0c18700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f359c0c39e0 0x7f359c1535a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:49:50.642 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.639+0000 7f35a0c18700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f359c0c1a20 con 0x7f359c06f640 2022-06-27T11:49:50.642 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.639+0000 7f35a0c18700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f359c0c1130 con 0x7f359c0c4160 2022-06-27T11:49:50.643 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.639+0000 7f35a0c18700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f359c06ea90 con 0x7f359c0c39e0 2022-06-27T11:49:50.643 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.639+0000 7f3599d9b700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f359c0c4160 0x7f359c156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:49:50.643 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.639+0000 7f359ad9d700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f359c0c39e0 0x7f359c1535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:49:50.643 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.639+0000 7f359a59c700 1 --1- >> v1:172.21.15.202:6789/0 conn(0x7f359c06f640 0x7f359c0c1eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.202:6789/0 says I am v1:172.21.15.195:50566/0 (socket says 172.21.15.195:50566) 2022-06-27T11:49:50.644 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.639+0000 7f3599d9b700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f359c0c4160 0x7f359c156ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3300/0 says I am v2:172.21.15.195:40646/0 (socket says 172.21.15.195:40646) 2022-06-27T11:49:50.644 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.639+0000 7f359a59c700 1 -- 172.21.15.195:0/3644131509 learned_addr learned my addr 172.21.15.195:0/3644131509 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:49:50.644 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.639+0000 7f359ad9d700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f359c0c39e0 0x7f359c1535a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3301/0 says I am v2:172.21.15.195:35278/0 (socket says 172.21.15.195:35278) 2022-06-27T11:49:50.644 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.639+0000 7f359959a700 1 -- 172.21.15.195:0/3644131509 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3505800419 0 0) 0x7f359c0c1a20 con 0x7f359c06f640 2022-06-27T11:49:50.645 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.639+0000 7f359959a700 1 -- 172.21.15.195:0/3644131509 --> v1:172.21.15.202:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3580003430 con 0x7f359c06f640 2022-06-27T11:49:50.645 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.639+0000 7f3599d9b700 1 -- 172.21.15.195:0/3644131509 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f359c0c39e0 msgr2=0x7f359c1535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:49:50.645 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.639+0000 7f3599d9b700 1 --2- 172.21.15.195:0/3644131509 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f359c0c39e0 0x7f359c1535a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:50.645 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.639+0000 7f3599d9b700 1 -- 172.21.15.195:0/3644131509 >> v1:172.21.15.202:6789/0 conn(0x7f359c06f640 legacy=0x7f359c0c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:49:50.645 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.639+0000 7f3599d9b700 1 -- 172.21.15.195:0/3644131509 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f359c06ed70 con 0x7f359c0c4160 2022-06-27T11:49:50.646 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.640+0000 7f359ad9d700 1 --2- 172.21.15.195:0/3644131509 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f359c0c39e0 0x7f359c1535a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:49:50.646 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.640+0000 7f3599d9b700 1 --2- 172.21.15.195:0/3644131509 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f359c0c4160 0x7f359c156ed0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f35900097a0 tx=0x7f3590009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc9f5e8e937a233 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:49:50.646 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.640+0000 7f359959a700 1 -- 172.21.15.195:0/3644131509 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3590011630 con 0x7f359c0c4160 2022-06-27T11:49:50.646 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.640+0000 7f359959a700 1 -- 172.21.15.195:0/3644131509 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3590011790 con 0x7f359c0c4160 2022-06-27T11:49:50.647 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.640+0000 7f359959a700 1 -- 172.21.15.195:0/3644131509 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3590011b30 con 0x7f359c0c4160 2022-06-27T11:49:50.647 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.640+0000 7f35a0c18700 1 -- 172.21.15.195:0/3644131509 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f359c0c4160 msgr2=0x7f359c156ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:49:50.647 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.640+0000 7f35a0c18700 1 --2- 172.21.15.195:0/3644131509 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f359c0c4160 0x7f359c156ed0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f35900097a0 tx=0x7f3590009060 comp rx=0 tx=0).stop 2022-06-27T11:49:50.647 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.640+0000 7f35a0c18700 1 -- 172.21.15.195:0/3644131509 shutdown_connections 2022-06-27T11:49:50.647 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.640+0000 7f35a0c18700 1 --2- 172.21.15.195:0/3644131509 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f359c0c39e0 0x7f359c1535a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:50.648 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.640+0000 7f35a0c18700 1 --2- 172.21.15.195:0/3644131509 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f359c0c4160 0x7f359c156ed0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:50.648 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.640+0000 7f35a0c18700 1 -- 172.21.15.195:0/3644131509 >> 172.21.15.195:0/3644131509 conn(0x7f359c0c6350 msgr2=0x7f359c0c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:49:50.649 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.640+0000 7f35a0c18700 1 -- 172.21.15.195:0/3644131509 shutdown_connections 2022-06-27T11:49:50.649 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.640+0000 7f35a0c18700 1 -- 172.21.15.195:0/3644131509 wait complete. 2022-06-27T11:49:50.649 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.641+0000 7f35a0c18700 1 Processor -- start 2022-06-27T11:49:50.649 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.642+0000 7f35a0c18700 1 -- start start 2022-06-27T11:49:50.653 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.642+0000 7f35a0c18700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f359c06f640 0x7f359c157a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:49:50.653 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.642+0000 7f35a0c18700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f359c0c39e0 0x7f359c15ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:49:50.654 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.642+0000 7f35a0c18700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f359c0c4160 0x7f359c15b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:49:50.654 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.642+0000 7f35a0c18700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f359c0bd540 con 0x7f359c0c4160 2022-06-27T11:49:50.654 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.642+0000 7f35a0c18700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f359c15a6f0 con 0x7f359c0c39e0 2022-06-27T11:49:50.654 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.642+0000 7f35a0c18700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f359c154f80 con 0x7f359c06f640 2022-06-27T11:49:50.655 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.642+0000 7f359ad9d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f359c0c4160 0x7f359c15b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:49:50.655 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.642+0000 7f359ad9d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f359c0c4160 0x7f359c15b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:42528/0 (socket says 172.21.15.195:42528) 2022-06-27T11:49:50.655 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.642+0000 7f359ad9d700 1 -- 172.21.15.195:0/4105459952 learned_addr learned my addr 172.21.15.195:0/4105459952 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:49:50.655 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.643+0000 7f3599d9b700 1 --2- 172.21.15.195:0/4105459952 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f359c0c39e0 0x7f359c15ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:49:50.655 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.643+0000 7f359a59c700 1 --2- 172.21.15.195:0/4105459952 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f359c06f640 0x7f359c157a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:49:50.656 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.643+0000 7f359ad9d700 1 -- 172.21.15.195:0/4105459952 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f359c06f640 msgr2=0x7f359c157a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:49:50.656 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.643+0000 7f359ad9d700 1 --2- 172.21.15.195:0/4105459952 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f359c06f640 0x7f359c157a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:50.656 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.643+0000 7f359ad9d700 1 -- 172.21.15.195:0/4105459952 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f359c0c39e0 msgr2=0x7f359c15ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:49:50.656 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.643+0000 7f359ad9d700 1 --2- 172.21.15.195:0/4105459952 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f359c0c39e0 0x7f359c15ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:50.657 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.643+0000 7f359ad9d700 1 -- 172.21.15.195:0/4105459952 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3594009a20 con 0x7f359c0c4160 2022-06-27T11:49:50.657 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.643+0000 7f359ad9d700 1 --2- 172.21.15.195:0/4105459952 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f359c0c4160 0x7f359c15b0c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3594009fc0 tx=0x7f359400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:49:50.657 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.643+0000 7f358a7fc700 1 -- 172.21.15.195:0/4105459952 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f359400cc60 con 0x7f359c0c4160 2022-06-27T11:49:50.657 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.643+0000 7f358a7fc700 1 -- 172.21.15.195:0/4105459952 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f359400f040 con 0x7f359c0c4160 2022-06-27T11:49:50.658 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.643+0000 7f35a0c18700 1 -- 172.21.15.195:0/4105459952 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35900092e0 con 0x7f359c0c4160 2022-06-27T11:49:50.658 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.643+0000 7f358a7fc700 1 -- 172.21.15.195:0/4105459952 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35940038a0 con 0x7f359c0c4160 2022-06-27T11:49:50.658 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.643+0000 7f35a0c18700 1 -- 172.21.15.195:0/4105459952 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f359c159ae0 con 0x7f359c0c4160 2022-06-27T11:49:50.658 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.644+0000 7f35a0c18700 1 -- 172.21.15.195:0/4105459952 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f359c15a860 con 0x7f359c0c4160 2022-06-27T11:49:50.659 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.644+0000 7f358a7fc700 1 -- 172.21.15.195:0/4105459952 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f3594014480 con 0x7f359c0c4160 2022-06-27T11:49:50.659 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.645+0000 7f358a7fc700 1 --2- 172.21.15.195:0/4105459952 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3578063d00 0x7f35780661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:49:50.659 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.645+0000 7f359a59c700 1 --2- 172.21.15.195:0/4105459952 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3578063d00 0x7f35780661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:49:50.659 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.645+0000 7f358a7fc700 1 -- 172.21.15.195:0/4105459952 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6144+0+0 (secure 0 0 0) 0x7f3594081540 con 0x7f359c0c4160 2022-06-27T11:49:50.660 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.645+0000 7f359a59c700 1 --2- 172.21.15.195:0/4105459952 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3578063d00 0x7f35780661b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f358c007fd0 tx=0x7f358c009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:49:50.660 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.649+0000 7f358a7fc700 1 -- 172.21.15.195:0/4105459952 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f35940500e0 con 0x7f359c0c4160 2022-06-27T11:49:50.873 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.873+0000 7f35a0c18700 1 -- 172.21.15.195:0/4105459952 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f359c0722c0 con 0x7f359c0c4160 2022-06-27T11:49:50.874 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.874+0000 7f358a7fc700 1 -- 172.21.15.195:0/4105459952 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6119 (secure 0 0 0) 0x7f3594018090 con 0x7f359c0c4160 2022-06-27T11:49:50.874 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:49:50.875 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4287,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/864126677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":864126677},{"type":"v1","addr":"172.21.15.202:6833","nonce":864126677}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:47:57.373108+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.202:6835/2038169702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2038169702},{"type":"v1","addr":"172.21.15.202:6835","nonce":2038169702}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:49:50.876 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.876+0000 7f35a0c18700 1 -- 172.21.15.195:0/4105459952 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3578063d00 msgr2=0x7f35780661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:49:50.877 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.876+0000 7f35a0c18700 1 --2- 172.21.15.195:0/4105459952 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3578063d00 0x7f35780661b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f358c007fd0 tx=0x7f358c009040 comp rx=0 tx=0).stop 2022-06-27T11:49:50.877 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.876+0000 7f35a0c18700 1 -- 172.21.15.195:0/4105459952 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f359c0c4160 msgr2=0x7f359c15b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:49:50.877 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.876+0000 7f35a0c18700 1 --2- 172.21.15.195:0/4105459952 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f359c0c4160 0x7f359c15b0c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3594009fc0 tx=0x7f359400b040 comp rx=0 tx=0).stop 2022-06-27T11:49:50.879 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.876+0000 7f35a0c18700 1 -- 172.21.15.195:0/4105459952 shutdown_connections 2022-06-27T11:49:50.879 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.876+0000 7f35a0c18700 1 --2- 172.21.15.195:0/4105459952 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3578063d00 0x7f35780661b0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:50.879 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.876+0000 7f35a0c18700 1 --2- 172.21.15.195:0/4105459952 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f359c0c4160 0x7f359c15b0c0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:50.879 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.876+0000 7f35a0c18700 1 --2- 172.21.15.195:0/4105459952 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f359c06f640 0x7f359c157a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:50.880 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.876+0000 7f35a0c18700 1 --2- 172.21.15.195:0/4105459952 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f359c0c39e0 0x7f359c15ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:50.880 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.876+0000 7f35a0c18700 1 -- 172.21.15.195:0/4105459952 >> 172.21.15.195:0/4105459952 conn(0x7f359c0c6350 msgr2=0x7f359c153370 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:49:50.880 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.876+0000 7f35a0c18700 1 -- 172.21.15.195:0/4105459952 shutdown_connections 2022-06-27T11:49:50.881 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:50.876+0000 7f35a0c18700 1 -- 172.21.15.195:0/4105459952 wait complete. 2022-06-27T11:49:50.881 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 8 2022-06-27T11:49:50.887 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-06-27T11:49:52.889 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:49:53.022 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.023+0000 7fd729cba700 1 Processor -- start 2022-06-27T11:49:53.023 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.023+0000 7fd729cba700 1 -- start start 2022-06-27T11:49:53.028 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.024+0000 7fd729cba700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd72414e860 0x7fd7240c3840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:49:53.029 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.024+0000 7fd729cba700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fd7240c3d80 0x7fd724157890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:49:53.029 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.024+0000 7fd729cba700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7240c33b0 con 0x7fd72414e490 2022-06-27T11:49:53.029 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.024+0000 7fd729cba700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7240c4200 con 0x7fd7240c3d80 2022-06-27T11:49:53.029 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.024+0000 7fd729cba700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd72406f220 con 0x7fd72414e860 2022-06-27T11:49:53.030 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.024+0000 7fd7237fe700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fd72414e490 0x7fd72406cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:48734/0 (socket says 172.21.15.195:48734) 2022-06-27T11:49:53.030 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.024+0000 7fd7237fe700 1 -- 172.21.15.195:0/3392569230 learned_addr learned my addr 172.21.15.195:0/3392569230 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:49:53.030 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.024+0000 7fd71affd700 1 --2- 172.21.15.195:0/3392569230 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd72414e860 0x7fd7240c3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:49:53.030 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.024+0000 7fd723fff700 1 --2- 172.21.15.195:0/3392569230 >> v2:172.21.15.202:3300/0 conn(0x7fd7240c3d80 0x7fd724157890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:49:53.031 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.024+0000 7fd722ffd700 1 -- 172.21.15.195:0/3392569230 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1898930454 0 0) 0x7fd7240c33b0 con 0x7fd72414e490 2022-06-27T11:49:53.031 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.024+0000 7fd722ffd700 1 -- 172.21.15.195:0/3392569230 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd704003430 con 0x7fd72414e490 2022-06-27T11:49:53.031 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.024+0000 7fd71affd700 1 -- 172.21.15.195:0/3392569230 >> v2:172.21.15.202:3300/0 conn(0x7fd7240c3d80 msgr2=0x7fd724157890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:49:53.031 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.024+0000 7fd71affd700 1 --2- 172.21.15.195:0/3392569230 >> v2:172.21.15.202:3300/0 conn(0x7fd7240c3d80 0x7fd724157890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:53.032 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.024+0000 7fd71affd700 1 -- 172.21.15.195:0/3392569230 >> v1:172.21.15.195:6789/0 conn(0x7fd72414e490 legacy=0x7fd72406cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:49:53.032 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.024+0000 7fd71affd700 1 -- 172.21.15.195:0/3392569230 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7241542c0 con 0x7fd72414e860 2022-06-27T11:49:53.032 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.024+0000 7fd723fff700 1 --2- 172.21.15.195:0/3392569230 >> v2:172.21.15.202:3300/0 conn(0x7fd7240c3d80 0x7fd724157890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:49:53.032 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.025+0000 7fd71affd700 1 --2- 172.21.15.195:0/3392569230 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd72414e860 0x7fd7240c3840 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd70c0097a0 tx=0x7fd70c009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a2cf9f37ce8020bc server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:49:53.033 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.025+0000 7fd722ffd700 1 -- 172.21.15.195:0/3392569230 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd70c011630 con 0x7fd72414e860 2022-06-27T11:49:53.033 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.025+0000 7fd722ffd700 1 -- 172.21.15.195:0/3392569230 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd70c011790 con 0x7fd72414e860 2022-06-27T11:49:53.033 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.025+0000 7fd722ffd700 1 -- 172.21.15.195:0/3392569230 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd70c011b30 con 0x7fd72414e860 2022-06-27T11:49:53.033 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.025+0000 7fd729cba700 1 -- 172.21.15.195:0/3392569230 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd72414e860 msgr2=0x7fd7240c3840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:49:53.034 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.025+0000 7fd729cba700 1 --2- 172.21.15.195:0/3392569230 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd72414e860 0x7fd7240c3840 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd70c0097a0 tx=0x7fd70c009060 comp rx=0 tx=0).stop 2022-06-27T11:49:53.034 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.025+0000 7fd729cba700 1 -- 172.21.15.195:0/3392569230 shutdown_connections 2022-06-27T11:49:53.034 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.025+0000 7fd729cba700 1 --2- 172.21.15.195:0/3392569230 >> v2:172.21.15.202:3300/0 conn(0x7fd7240c3d80 0x7fd724157890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:53.034 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.025+0000 7fd729cba700 1 --2- 172.21.15.195:0/3392569230 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd72414e860 0x7fd7240c3840 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:53.035 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.025+0000 7fd729cba700 1 -- 172.21.15.195:0/3392569230 >> 172.21.15.195:0/3392569230 conn(0x7fd724063af0 msgr2=0x7fd724063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:49:53.035 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.025+0000 7fd729cba700 1 -- 172.21.15.195:0/3392569230 shutdown_connections 2022-06-27T11:49:53.035 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.025+0000 7fd729cba700 1 -- 172.21.15.195:0/3392569230 wait complete. 2022-06-27T11:49:53.035 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.026+0000 7fd729cba700 1 Processor -- start 2022-06-27T11:49:53.036 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.027+0000 7fd729cba700 1 -- start start 2022-06-27T11:49:53.036 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.027+0000 7fd729cba700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd7240c3d80 0x7fd724145900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:49:53.036 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.027+0000 7fd729cba700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd72414e490 0x7fd72414b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:49:53.036 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.027+0000 7fd729cba700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd72414e860 0x7fd724144640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:49:53.036 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.027+0000 7fd729cba700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7240c7a00 con 0x7fd72414e490 2022-06-27T11:49:53.037 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.027+0000 7fd729cba700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd724148800 con 0x7fd72414e860 2022-06-27T11:49:53.037 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.027+0000 7fd729cba700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7240c6b10 con 0x7fd7240c3d80 2022-06-27T11:49:53.037 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.028+0000 7fd71affd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd72414e490 0x7fd72414b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:49:53.037 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.028+0000 7fd71affd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd72414e490 0x7fd72414b010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:42554/0 (socket says 172.21.15.195:42554) 2022-06-27T11:49:53.038 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.028+0000 7fd71affd700 1 -- 172.21.15.195:0/1222935787 learned_addr learned my addr 172.21.15.195:0/1222935787 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:49:53.038 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.028+0000 7fd723fff700 1 --2- 172.21.15.195:0/1222935787 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd72414e860 0x7fd724144640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:49:53.038 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.028+0000 7fd7237fe700 1 --2- 172.21.15.195:0/1222935787 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd7240c3d80 0x7fd724145900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:49:53.038 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.028+0000 7fd71affd700 1 -- 172.21.15.195:0/1222935787 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd7240c3d80 msgr2=0x7fd724145900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:49:53.039 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.028+0000 7fd71affd700 1 --2- 172.21.15.195:0/1222935787 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd7240c3d80 0x7fd724145900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:53.039 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.028+0000 7fd71affd700 1 -- 172.21.15.195:0/1222935787 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd72414e860 msgr2=0x7fd724144640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:49:53.039 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.028+0000 7fd71affd700 1 --2- 172.21.15.195:0/1222935787 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd72414e860 0x7fd724144640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:53.039 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.028+0000 7fd71affd700 1 -- 172.21.15.195:0/1222935787 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd71c009a20 con 0x7fd72414e490 2022-06-27T11:49:53.040 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.028+0000 7fd71affd700 1 --2- 172.21.15.195:0/1222935787 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd72414e490 0x7fd72414b010 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd7240be550 tx=0x7fd70c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:49:53.040 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.028+0000 7fd71bfff700 1 -- 172.21.15.195:0/1222935787 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd70c0037b0 con 0x7fd72414e490 2022-06-27T11:49:53.040 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.028+0000 7fd71bfff700 1 -- 172.21.15.195:0/1222935787 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd70c003930 con 0x7fd72414e490 2022-06-27T11:49:53.040 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.028+0000 7fd729cba700 1 -- 172.21.15.195:0/1222935787 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd70c0092e0 con 0x7fd72414e490 2022-06-27T11:49:53.041 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.028+0000 7fd71bfff700 1 -- 172.21.15.195:0/1222935787 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd70c003cf0 con 0x7fd72414e490 2022-06-27T11:49:53.041 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.028+0000 7fd729cba700 1 -- 172.21.15.195:0/1222935787 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7240c70d0 con 0x7fd72414e490 2022-06-27T11:49:53.041 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.029+0000 7fd729cba700 1 -- 172.21.15.195:0/1222935787 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7240722c0 con 0x7fd72414e490 2022-06-27T11:49:53.042 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.030+0000 7fd71bfff700 1 -- 172.21.15.195:0/1222935787 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fd70c00f050 con 0x7fd72414e490 2022-06-27T11:49:53.043 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.030+0000 7fd71bfff700 1 --2- 172.21.15.195:0/1222935787 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd708063d00 0x7fd7080661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:49:53.043 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.030+0000 7fd71bfff700 1 -- 172.21.15.195:0/1222935787 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6144+0+0 (secure 0 0 0) 0x7fd70c081f30 con 0x7fd72414e490 2022-06-27T11:49:53.043 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.030+0000 7fd7237fe700 1 --2- 172.21.15.195:0/1222935787 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd708063d00 0x7fd7080661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:49:53.044 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.031+0000 7fd7237fe700 1 --2- 172.21.15.195:0/1222935787 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd708063d00 0x7fd7080661b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd714007fd0 tx=0x7fd714009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:49:53.044 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.034+0000 7fd71bfff700 1 -- 172.21.15.195:0/1222935787 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fd70c050b90 con 0x7fd72414e490 2022-06-27T11:49:53.262 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.263+0000 7fd729cba700 1 -- 172.21.15.195:0/1222935787 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd7240c83a0 con 0x7fd72414e490 2022-06-27T11:49:53.263 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.264+0000 7fd71bfff700 1 -- 172.21.15.195:0/1222935787 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6119 (secure 0 0 0) 0x7fd70c015030 con 0x7fd72414e490 2022-06-27T11:49:53.264 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:49:53.264 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4287,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/864126677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":864126677},{"type":"v1","addr":"172.21.15.202:6833","nonce":864126677}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:47:57.373108+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.202:6835/2038169702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2038169702},{"type":"v1","addr":"172.21.15.202:6835","nonce":2038169702}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:49:53.265 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.265+0000 7fd729cba700 1 -- 172.21.15.195:0/1222935787 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd708063d00 msgr2=0x7fd7080661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:49:53.266 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.265+0000 7fd729cba700 1 --2- 172.21.15.195:0/1222935787 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd708063d00 0x7fd7080661b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd714007fd0 tx=0x7fd714009040 comp rx=0 tx=0).stop 2022-06-27T11:49:53.266 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.265+0000 7fd729cba700 1 -- 172.21.15.195:0/1222935787 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd72414e490 msgr2=0x7fd72414b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:49:53.266 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.265+0000 7fd729cba700 1 --2- 172.21.15.195:0/1222935787 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd72414e490 0x7fd72414b010 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd7240be550 tx=0x7fd70c017000 comp rx=0 tx=0).stop 2022-06-27T11:49:53.266 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.266+0000 7fd729cba700 1 -- 172.21.15.195:0/1222935787 shutdown_connections 2022-06-27T11:49:53.267 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.266+0000 7fd729cba700 1 --2- 172.21.15.195:0/1222935787 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd708063d00 0x7fd7080661b0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:53.267 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.266+0000 7fd729cba700 1 --2- 172.21.15.195:0/1222935787 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd72414e860 0x7fd724144640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:53.268 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.266+0000 7fd729cba700 1 --2- 172.21.15.195:0/1222935787 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd7240c3d80 0x7fd724145900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:53.268 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.266+0000 7fd729cba700 1 --2- 172.21.15.195:0/1222935787 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd72414e490 0x7fd72414b010 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:53.268 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.266+0000 7fd729cba700 1 -- 172.21.15.195:0/1222935787 >> 172.21.15.195:0/1222935787 conn(0x7fd724063af0 msgr2=0x7fd72406cfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:49:53.268 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.266+0000 7fd729cba700 1 -- 172.21.15.195:0/1222935787 shutdown_connections 2022-06-27T11:49:53.269 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:53.266+0000 7fd729cba700 1 -- 172.21.15.195:0/1222935787 wait complete. 2022-06-27T11:49:53.269 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 8 2022-06-27T11:49:53.277 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-06-27T11:49:55.278 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:49:55.413 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.414+0000 7f4f74aee700 1 Processor -- start 2022-06-27T11:49:55.414 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.414+0000 7f4f74aee700 1 -- start start 2022-06-27T11:49:55.416 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.415+0000 7f4f74aee700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f4f701508d0 0x7f4f70063af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:49:55.417 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.415+0000 7f4f74aee700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f70064030 con 0x7f4f70150500 2022-06-27T11:49:55.417 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.415+0000 7f4f74aee700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f700c4160 con 0x7f4f7006a450 2022-06-27T11:49:55.417 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.415+0000 7f4f74aee700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f700c4440 con 0x7f4f701508d0 2022-06-27T11:49:55.418 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.415+0000 7f4f6ed9d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f4f701508d0 0x7f4f70063af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:49:55.418 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.415+0000 7f4f6e59c700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f4f70150500 0x7f4f700c48d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:48748/0 (socket says 172.21.15.195:48748) 2022-06-27T11:49:55.419 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.415+0000 7f4f6e59c700 1 -- 172.21.15.195:0/3806013298 learned_addr learned my addr 172.21.15.195:0/3806013298 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:49:55.419 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.415+0000 7f4f6ed9d700 1 -- 172.21.15.195:0/3806013298 >> v1:172.21.15.202:6789/0 conn(0x7f4f7006a450 legacy=0x7f4f700bbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:49:55.419 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.415+0000 7f4f6ed9d700 1 -- 172.21.15.195:0/3806013298 >> v1:172.21.15.195:6789/0 conn(0x7f4f70150500 legacy=0x7f4f700c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:49:55.420 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.415+0000 7f4f6ed9d700 1 -- 172.21.15.195:0/3806013298 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f7015a800 con 0x7f4f701508d0 2022-06-27T11:49:55.420 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.415+0000 7f4f6d59a700 1 -- 172.21.15.195:0/3806013298 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1813184635 0 0) 0x7f4f60001290 con 0x7f4f70150500 2022-06-27T11:49:55.421 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.415+0000 7f4f6ed9d700 1 --2- 172.21.15.195:0/3806013298 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4f701508d0 0x7f4f70063af0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f4f68009e60 tx=0x7f4f68009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eba4cc5272350b32 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:49:55.421 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.416+0000 7f4f6d59a700 1 -- 172.21.15.195:0/3806013298 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f6800e070 con 0x7f4f701508d0 2022-06-27T11:49:55.422 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.416+0000 7f4f6d59a700 1 -- 172.21.15.195:0/3806013298 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f6800bbf0 con 0x7f4f701508d0 2022-06-27T11:49:55.422 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.416+0000 7f4f6d59a700 1 -- 172.21.15.195:0/3806013298 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f680038a0 con 0x7f4f701508d0 2022-06-27T11:49:55.422 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.416+0000 7f4f74aee700 1 -- 172.21.15.195:0/3806013298 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4f701508d0 msgr2=0x7f4f70063af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:49:55.422 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.416+0000 7f4f74aee700 1 --2- 172.21.15.195:0/3806013298 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4f701508d0 0x7f4f70063af0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f4f68009e60 tx=0x7f4f68009780 comp rx=0 tx=0).stop 2022-06-27T11:49:55.423 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.416+0000 7f4f74aee700 1 -- 172.21.15.195:0/3806013298 shutdown_connections 2022-06-27T11:49:55.423 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.416+0000 7f4f74aee700 1 --2- 172.21.15.195:0/3806013298 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4f701508d0 0x7f4f70063af0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:55.423 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.416+0000 7f4f74aee700 1 -- 172.21.15.195:0/3806013298 >> 172.21.15.195:0/3806013298 conn(0x7f4f700c9b00 msgr2=0x7f4f700c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:49:55.423 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.416+0000 7f4f74aee700 1 -- 172.21.15.195:0/3806013298 shutdown_connections 2022-06-27T11:49:55.423 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.416+0000 7f4f74aee700 1 -- 172.21.15.195:0/3806013298 wait complete. 2022-06-27T11:49:55.424 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.417+0000 7f4f74aee700 1 Processor -- start 2022-06-27T11:49:55.424 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.417+0000 7f4f74aee700 1 -- start start 2022-06-27T11:49:55.427 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.418+0000 7f4f74aee700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4f7006a450 0x7f4f70158af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:49:55.427 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.418+0000 7f4f74aee700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4f70150500 0x7f4f700bef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:49:55.427 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.418+0000 7f4f74aee700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4f701508d0 0x7f4f700bf460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:49:55.427 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.418+0000 7f4f74aee700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f700bead0 con 0x7f4f701508d0 2022-06-27T11:49:55.428 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.418+0000 7f4f74aee700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f70159b90 con 0x7f4f7006a450 2022-06-27T11:49:55.428 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.418+0000 7f4f74aee700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f700c7b50 con 0x7f4f70150500 2022-06-27T11:49:55.428 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.418+0000 7f4f6ed9d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4f701508d0 0x7f4f700bf460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:49:55.428 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.418+0000 7f4f6ed9d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4f701508d0 0x7f4f700bf460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:42570/0 (socket says 172.21.15.195:42570) 2022-06-27T11:49:55.428 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.418+0000 7f4f6ed9d700 1 -- 172.21.15.195:0/52900604 learned_addr learned my addr 172.21.15.195:0/52900604 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:49:55.429 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.418+0000 7f4f6e59c700 1 --2- 172.21.15.195:0/52900604 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4f7006a450 0x7f4f70158af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:49:55.429 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.418+0000 7f4f6dd9b700 1 --2- 172.21.15.195:0/52900604 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4f70150500 0x7f4f700bef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:49:55.429 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.418+0000 7f4f6ed9d700 1 -- 172.21.15.195:0/52900604 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4f70150500 msgr2=0x7f4f700bef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:49:55.429 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.418+0000 7f4f6ed9d700 1 --2- 172.21.15.195:0/52900604 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4f70150500 0x7f4f700bef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:55.430 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.419+0000 7f4f6ed9d700 1 -- 172.21.15.195:0/52900604 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4f7006a450 msgr2=0x7f4f70158af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:49:55.430 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.419+0000 7f4f6ed9d700 1 --2- 172.21.15.195:0/52900604 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4f7006a450 0x7f4f70158af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:55.430 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.419+0000 7f4f6ed9d700 1 -- 172.21.15.195:0/52900604 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f68009a20 con 0x7f4f701508d0 2022-06-27T11:49:55.430 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.419+0000 7f4f6ed9d700 1 --2- 172.21.15.195:0/52900604 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4f701508d0 0x7f4f700bf460 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f4f680040b0 tx=0x7f4f68018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:49:55.431 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.419+0000 7f4f5e7fc700 1 -- 172.21.15.195:0/52900604 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f68004570 con 0x7f4f701508d0 2022-06-27T11:49:55.431 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.419+0000 7f4f5e7fc700 1 -- 172.21.15.195:0/52900604 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f680046f0 con 0x7f4f701508d0 2022-06-27T11:49:55.431 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.419+0000 7f4f5e7fc700 1 -- 172.21.15.195:0/52900604 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f680276c0 con 0x7f4f701508d0 2022-06-27T11:49:55.431 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.419+0000 7f4f74aee700 1 -- 172.21.15.195:0/52900604 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f70068bb0 con 0x7f4f701508d0 2022-06-27T11:49:55.431 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.419+0000 7f4f74aee700 1 -- 172.21.15.195:0/52900604 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f70166c60 con 0x7f4f701508d0 2022-06-27T11:49:55.432 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.420+0000 7f4f74aee700 1 -- 172.21.15.195:0/52900604 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f700bec40 con 0x7f4f701508d0 2022-06-27T11:49:55.432 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.420+0000 7f4f5e7fc700 1 -- 172.21.15.195:0/52900604 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f4f6801e430 con 0x7f4f701508d0 2022-06-27T11:49:55.432 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.421+0000 7f4f5e7fc700 1 --2- 172.21.15.195:0/52900604 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4f4c063d00 0x7f4f4c0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:49:55.432 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.421+0000 7f4f5e7fc700 1 -- 172.21.15.195:0/52900604 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6144+0+0 (secure 0 0 0) 0x7f4f68083390 con 0x7f4f701508d0 2022-06-27T11:49:55.432 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.421+0000 7f4f6e59c700 1 --2- 172.21.15.195:0/52900604 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4f4c063d00 0x7f4f4c0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:49:55.433 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.421+0000 7f4f6e59c700 1 --2- 172.21.15.195:0/52900604 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4f4c063d00 0x7f4f4c0661b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f4f600014f0 tx=0x7f4f600017d0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:49:55.433 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.425+0000 7f4f5e7fc700 1 -- 172.21.15.195:0/52900604 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f4f700bec40 con 0x7f4f701508d0 2022-06-27T11:49:55.651 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.650+0000 7f4f74aee700 1 -- 172.21.15.195:0/52900604 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4f700bec10 con 0x7f4f701508d0 2022-06-27T11:49:55.651 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.650+0000 7f4f5e7fc700 1 -- 172.21.15.195:0/52900604 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6119 (secure 0 0 0) 0x7f4f68051ed0 con 0x7f4f701508d0 2022-06-27T11:49:55.652 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:49:55.652 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4287,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/864126677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":864126677},{"type":"v1","addr":"172.21.15.202:6833","nonce":864126677}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:47:57.373108+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.202:6835/2038169702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2038169702},{"type":"v1","addr":"172.21.15.202:6835","nonce":2038169702}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:49:55.654 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.652+0000 7f4f74aee700 1 -- 172.21.15.195:0/52900604 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4f4c063d00 msgr2=0x7f4f4c0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:49:55.655 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.652+0000 7f4f74aee700 1 --2- 172.21.15.195:0/52900604 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4f4c063d00 0x7f4f4c0661b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f4f600014f0 tx=0x7f4f600017d0 comp rx=0 tx=0).stop 2022-06-27T11:49:55.655 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.652+0000 7f4f74aee700 1 -- 172.21.15.195:0/52900604 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4f701508d0 msgr2=0x7f4f700bf460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:49:55.655 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.652+0000 7f4f74aee700 1 --2- 172.21.15.195:0/52900604 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4f701508d0 0x7f4f700bf460 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f4f680040b0 tx=0x7f4f68018000 comp rx=0 tx=0).stop 2022-06-27T11:49:55.655 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.652+0000 7f4f74aee700 1 -- 172.21.15.195:0/52900604 shutdown_connections 2022-06-27T11:49:55.656 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.652+0000 7f4f74aee700 1 --2- 172.21.15.195:0/52900604 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4f4c063d00 0x7f4f4c0661b0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:55.656 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.652+0000 7f4f74aee700 1 --2- 172.21.15.195:0/52900604 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4f701508d0 0x7f4f700bf460 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:55.656 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.652+0000 7f4f74aee700 1 --2- 172.21.15.195:0/52900604 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4f7006a450 0x7f4f70158af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:55.656 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.652+0000 7f4f74aee700 1 --2- 172.21.15.195:0/52900604 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4f70150500 0x7f4f700bef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:55.657 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.652+0000 7f4f74aee700 1 -- 172.21.15.195:0/52900604 >> 172.21.15.195:0/52900604 conn(0x7f4f700c9b00 msgr2=0x7f4f700c88d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:49:55.657 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.652+0000 7f4f74aee700 1 -- 172.21.15.195:0/52900604 shutdown_connections 2022-06-27T11:49:55.657 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:55.652+0000 7f4f74aee700 1 -- 172.21.15.195:0/52900604 wait complete. 2022-06-27T11:49:55.657 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 8 2022-06-27T11:49:55.664 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-06-27T11:49:56.476 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T11:49:56.481 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T11:49:57.666 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:49:57.801 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:57.800+0000 7fa822b4d700 1 Processor -- start 2022-06-27T11:49:57.802 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:57.800+0000 7fa822b4d700 1 -- start start 2022-06-27T11:49:57.805 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:57.801+0000 7fa822b4d700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa81c06a450 0x7fa81c063af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:49:57.807 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:57.801+0000 7fa822b4d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fa81c1508d0 0x7fa81c0bbf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:49:57.807 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:57.801+0000 7fa822b4d700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa81c064030 con 0x7fa81c150500 2022-06-27T11:49:57.809 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:57.801+0000 7fa822b4d700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa81c0c4160 con 0x7fa81c1508d0 2022-06-27T11:49:57.810 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:57.801+0000 7fa822b4d700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa81c0c4440 con 0x7fa81c06a450 2022-06-27T11:49:57.811 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:57.801+0000 7fa8210ea700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fa81c1508d0 0x7fa81c0bbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:49:57.811 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:57.801+0000 7fa8208e9700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fa81c150500 0x7fa81c0c48d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:48762/0 (socket says 172.21.15.195:48762) 2022-06-27T11:49:57.811 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:57.801+0000 7fa8208e9700 1 -- 172.21.15.195:0/4108926416 learned_addr learned my addr 172.21.15.195:0/4108926416 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:49:57.812 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:57.801+0000 7fa81bfff700 1 --2- 172.21.15.195:0/4108926416 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa81c06a450 0x7fa81c063af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:49:57.812 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:57.801+0000 7fa8210ea700 1 -- 172.21.15.195:0/4108926416 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa81c06a450 msgr2=0x7fa81c063af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:49:57.812 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:57.801+0000 7fa8210ea700 1 --2- 172.21.15.195:0/4108926416 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa81c06a450 0x7fa81c063af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:57.817 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:57.801+0000 7fa8210ea700 1 -- 172.21.15.195:0/4108926416 >> v1:172.21.15.195:6789/0 conn(0x7fa81c150500 legacy=0x7fa81c0c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:49:57.818 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:57.801+0000 7fa8210ea700 1 -- 172.21.15.195:0/4108926416 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa81c15a800 con 0x7fa81c1508d0 2022-06-27T11:49:57.818 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:57.801+0000 7fa8210ea700 1 --2- 172.21.15.195:0/4108926416 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa81c1508d0 0x7fa81c0bbf50 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fa814009e60 tx=0x7fa814009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4ff67c723f22f15 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:49:57.825 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:57.801+0000 7fa81b7fe700 1 -- 172.21.15.195:0/4108926416 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa81400e070 con 0x7fa81c1508d0 2022-06-27T11:49:57.826 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:57.801+0000 7fa81b7fe700 1 -- 172.21.15.195:0/4108926416 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa81400bc30 con 0x7fa81c1508d0 2022-06-27T11:49:57.826 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:57.801+0000 7fa81b7fe700 1 -- 172.21.15.195:0/4108926416 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa814012430 con 0x7fa81c1508d0 2022-06-27T11:49:57.827 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:57.802+0000 7fa822b4d700 1 -- 172.21.15.195:0/4108926416 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa81c1508d0 msgr2=0x7fa81c0bbf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:49:57.831 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:57.802+0000 7fa822b4d700 1 --2- 172.21.15.195:0/4108926416 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa81c1508d0 0x7fa81c0bbf50 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fa814009e60 tx=0x7fa814009780 comp rx=0 tx=0).stop 2022-06-27T11:49:57.831 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:57.802+0000 7fa822b4d700 1 -- 172.21.15.195:0/4108926416 shutdown_connections 2022-06-27T11:49:57.831 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:57.802+0000 7fa822b4d700 1 --2- 172.21.15.195:0/4108926416 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa81c1508d0 0x7fa81c0bbf50 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:57.832 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:57.802+0000 7fa822b4d700 1 --2- 172.21.15.195:0/4108926416 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa81c06a450 0x7fa81c063af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:57.832 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:57.802+0000 7fa822b4d700 1 -- 172.21.15.195:0/4108926416 >> 172.21.15.195:0/4108926416 conn(0x7fa81c0c9b00 msgr2=0x7fa81c0c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:49:57.833 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:57.802+0000 7fa822b4d700 1 -- 172.21.15.195:0/4108926416 shutdown_connections 2022-06-27T11:49:57.833 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:57.802+0000 7fa822b4d700 1 -- 172.21.15.195:0/4108926416 wait complete. 2022-06-27T11:49:57.833 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:57.803+0000 7fa822b4d700 1 Processor -- start 2022-06-27T11:49:57.834 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:57.803+0000 7fa822b4d700 1 -- start start 2022-06-27T11:49:57.834 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:57.804+0000 7fa822b4d700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa81c06a450 0x7fa81c166720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:49:57.834 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:57.804+0000 7fa822b4d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa81c150500 0x7fa81c159360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:49:57.834 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:57.804+0000 7fa822b4d700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa81c1508d0 0x7fa81c0c5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:49:57.835 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:57.804+0000 7fa822b4d700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81c155410 con 0x7fa81c150500 2022-06-27T11:49:57.835 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:57.804+0000 7fa822b4d700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81c158910 con 0x7fa81c06a450 2022-06-27T11:49:57.835 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:57.804+0000 7fa822b4d700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa81c158ed0 con 0x7fa81c1508d0 2022-06-27T11:49:57.835 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:57.804+0000 7fa81bfff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa81c150500 0x7fa81c159360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:49:57.836 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:57.804+0000 7fa81bfff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa81c150500 0x7fa81c159360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:42580/0 (socket says 172.21.15.195:42580) 2022-06-27T11:49:57.836 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:57.804+0000 7fa81bfff700 1 -- 172.21.15.195:0/2990415123 learned_addr learned my addr 172.21.15.195:0/2990415123 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:49:57.836 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:57.804+0000 7fa8210ea700 1 --2- 172.21.15.195:0/2990415123 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa81c1508d0 0x7fa81c0c5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:49:57.836 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:57.804+0000 7fa8208e9700 1 --2- 172.21.15.195:0/2990415123 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa81c06a450 0x7fa81c166720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:49:57.837 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:57.804+0000 7fa81bfff700 1 -- 172.21.15.195:0/2990415123 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa81c1508d0 msgr2=0x7fa81c0c5250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:49:57.837 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:57.804+0000 7fa81bfff700 1 --2- 172.21.15.195:0/2990415123 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa81c1508d0 0x7fa81c0c5250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:57.837 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:57.804+0000 7fa81bfff700 1 -- 172.21.15.195:0/2990415123 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa81c06a450 msgr2=0x7fa81c166720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:49:57.837 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:57.804+0000 7fa81bfff700 1 --2- 172.21.15.195:0/2990415123 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa81c06a450 0x7fa81c166720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:57.838 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:57.804+0000 7fa81bfff700 1 -- 172.21.15.195:0/2990415123 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa814009a20 con 0x7fa81c150500 2022-06-27T11:49:57.838 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:57.805+0000 7fa81bfff700 1 --2- 172.21.15.195:0/2990415123 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa81c150500 0x7fa81c159360 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fa80800c8b0 tx=0x7fa808008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:49:57.838 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:57.805+0000 7fa818ff9700 1 -- 172.21.15.195:0/2990415123 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa808004020 con 0x7fa81c150500 2022-06-27T11:49:57.838 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:57.805+0000 7fa822b4d700 1 -- 172.21.15.195:0/2990415123 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa81c166c60 con 0x7fa81c150500 2022-06-27T11:49:57.839 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:57.805+0000 7fa818ff9700 1 -- 172.21.15.195:0/2990415123 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa808004390 con 0x7fa81c150500 2022-06-27T11:49:57.839 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:57.805+0000 7fa822b4d700 1 -- 172.21.15.195:0/2990415123 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa81c1671b0 con 0x7fa81c150500 2022-06-27T11:49:57.839 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:57.805+0000 7fa818ff9700 1 -- 172.21.15.195:0/2990415123 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa808011600 con 0x7fa81c150500 2022-06-27T11:49:57.839 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:57.806+0000 7fa822b4d700 1 -- 172.21.15.195:0/2990415123 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa81c0722c0 con 0x7fa81c150500 2022-06-27T11:49:57.839 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:57.806+0000 7fa818ff9700 1 -- 172.21.15.195:0/2990415123 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fa808011a30 con 0x7fa81c150500 2022-06-27T11:49:57.840 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:57.807+0000 7fa818ff9700 1 --2- 172.21.15.195:0/2990415123 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa804063d00 0x7fa8040661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:49:57.840 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:57.807+0000 7fa8208e9700 1 --2- 172.21.15.195:0/2990415123 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa804063d00 0x7fa8040661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:49:57.840 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:57.807+0000 7fa818ff9700 1 -- 172.21.15.195:0/2990415123 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6144+0+0 (secure 0 0 0) 0x7fa8080808c0 con 0x7fa81c150500 2022-06-27T11:49:57.841 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:57.807+0000 7fa8208e9700 1 --2- 172.21.15.195:0/2990415123 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa804063d00 0x7fa8040661b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fa8100010e0 tx=0x7fa810009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:49:57.841 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:57.811+0000 7fa818ff9700 1 -- 172.21.15.195:0/2990415123 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fa80804f400 con 0x7fa81c150500 2022-06-27T11:49:58.041 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:58.041+0000 7fa822b4d700 1 -- 172.21.15.195:0/2990415123 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa81c159010 con 0x7fa81c150500 2022-06-27T11:49:58.041 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:58.042+0000 7fa818ff9700 1 -- 172.21.15.195:0/2990415123 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6119 (secure 0 0 0) 0x7fa808015020 con 0x7fa81c150500 2022-06-27T11:49:58.042 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:49:58.042 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4287,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/864126677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":864126677},{"type":"v1","addr":"172.21.15.202:6833","nonce":864126677}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:47:57.373108+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.202:6835/2038169702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2038169702},{"type":"v1","addr":"172.21.15.202:6835","nonce":2038169702}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:49:58.045 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:58.044+0000 7fa822b4d700 1 -- 172.21.15.195:0/2990415123 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa804063d00 msgr2=0x7fa8040661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:49:58.046 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:58.044+0000 7fa822b4d700 1 --2- 172.21.15.195:0/2990415123 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa804063d00 0x7fa8040661b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fa8100010e0 tx=0x7fa810009040 comp rx=0 tx=0).stop 2022-06-27T11:49:58.046 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:58.044+0000 7fa822b4d700 1 -- 172.21.15.195:0/2990415123 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa81c150500 msgr2=0x7fa81c159360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:49:58.047 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:58.044+0000 7fa822b4d700 1 --2- 172.21.15.195:0/2990415123 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa81c150500 0x7fa81c159360 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fa80800c8b0 tx=0x7fa808008040 comp rx=0 tx=0).stop 2022-06-27T11:49:58.047 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:58.044+0000 7fa822b4d700 1 -- 172.21.15.195:0/2990415123 shutdown_connections 2022-06-27T11:49:58.047 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:58.044+0000 7fa822b4d700 1 --2- 172.21.15.195:0/2990415123 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa804063d00 0x7fa8040661b0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:58.048 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:58.044+0000 7fa822b4d700 1 --2- 172.21.15.195:0/2990415123 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa81c1508d0 0x7fa81c0c5250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:58.048 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:58.044+0000 7fa822b4d700 1 --2- 172.21.15.195:0/2990415123 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa81c06a450 0x7fa81c166720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:58.049 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:58.044+0000 7fa822b4d700 1 --2- 172.21.15.195:0/2990415123 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa81c150500 0x7fa81c159360 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:49:58.049 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:58.044+0000 7fa822b4d700 1 -- 172.21.15.195:0/2990415123 >> 172.21.15.195:0/2990415123 conn(0x7fa81c0c9b00 msgr2=0x7fa81c067ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:49:58.050 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:58.044+0000 7fa822b4d700 1 -- 172.21.15.195:0/2990415123 shutdown_connections 2022-06-27T11:49:58.050 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:49:58.044+0000 7fa822b4d700 1 -- 172.21.15.195:0/2990415123 wait complete. 2022-06-27T11:49:58.051 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 8 2022-06-27T11:49:58.056 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-06-27T11:50:00.058 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:50:00.194 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.194+0000 7fcc583b0700 1 Processor -- start 2022-06-27T11:50:00.194 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.194+0000 7fcc583b0700 1 -- start start 2022-06-27T11:50:00.197 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.195+0000 7fcc583b0700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcc50153de0 0x7fcc50068ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:00.197 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.195+0000 7fcc583b0700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fcc500c3010 0x7fcc500c3460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:00.197 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.195+0000 7fcc583b0700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc50068a10 con 0x7fcc500b8520 2022-06-27T11:50:00.198 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.195+0000 7fcc583b0700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc50154210 con 0x7fcc500c3010 2022-06-27T11:50:00.198 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.195+0000 7fcc583b0700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc50067ed0 con 0x7fcc50153de0 2022-06-27T11:50:00.198 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.196+0000 7fcc5694d700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fcc500c3010 0x7fcc500c3460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:00.198 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.196+0000 7fcc5614c700 1 --1- >> v1:172.21.15.202:6789/0 conn(0x7fcc500b8520 0x7fcc500c45f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.202:6789/0 says I am v1:172.21.15.195:50640/0 (socket says 172.21.15.195:50640) 2022-06-27T11:50:00.199 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.196+0000 7fcc5594b700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcc50153de0 0x7fcc50068ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:00.204 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.196+0000 7fcc5614c700 1 -- 172.21.15.195:0/3177368943 learned_addr learned my addr 172.21.15.195:0/3177368943 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:50:00.204 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.196+0000 7fcc5594b700 1 -- 172.21.15.195:0/3177368943 >> v2:172.21.15.202:3300/0 conn(0x7fcc500c3010 msgr2=0x7fcc500c3460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:50:00.204 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.196+0000 7fcc5594b700 1 --2- 172.21.15.195:0/3177368943 >> v2:172.21.15.202:3300/0 conn(0x7fcc500c3010 0x7fcc500c3460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:00.205 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.196+0000 7fcc5594b700 1 -- 172.21.15.195:0/3177368943 >> v1:172.21.15.202:6789/0 conn(0x7fcc500b8520 legacy=0x7fcc500c45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:00.205 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.196+0000 7fcc5594b700 1 -- 172.21.15.195:0/3177368943 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc500c3dd0 con 0x7fcc50153de0 2022-06-27T11:50:00.205 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.196+0000 7fcc5694d700 1 --2- 172.21.15.195:0/3177368943 >> v2:172.21.15.202:3300/0 conn(0x7fcc500c3010 0x7fcc500c3460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:50:00.205 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.196+0000 7fcc5594b700 1 --2- 172.21.15.195:0/3177368943 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcc50153de0 0x7fcc50068ea0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fcc4c0097a0 tx=0x7fcc4c009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5361867602009c3c server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:50:00.206 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.196+0000 7fcc5514a700 1 -- 172.21.15.195:0/3177368943 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc4c011630 con 0x7fcc50153de0 2022-06-27T11:50:00.206 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.196+0000 7fcc5514a700 1 -- 172.21.15.195:0/3177368943 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc4c011790 con 0x7fcc50153de0 2022-06-27T11:50:00.206 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.196+0000 7fcc5514a700 1 -- 172.21.15.195:0/3177368943 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc4c011b30 con 0x7fcc50153de0 2022-06-27T11:50:00.206 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.197+0000 7fcc583b0700 1 -- 172.21.15.195:0/3177368943 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcc50153de0 msgr2=0x7fcc50068ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:00.207 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.197+0000 7fcc583b0700 1 --2- 172.21.15.195:0/3177368943 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcc50153de0 0x7fcc50068ea0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fcc4c0097a0 tx=0x7fcc4c009060 comp rx=0 tx=0).stop 2022-06-27T11:50:00.207 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.197+0000 7fcc583b0700 1 -- 172.21.15.195:0/3177368943 shutdown_connections 2022-06-27T11:50:00.207 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.197+0000 7fcc583b0700 1 --2- 172.21.15.195:0/3177368943 >> v2:172.21.15.202:3300/0 conn(0x7fcc500c3010 0x7fcc500c3460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:00.207 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.197+0000 7fcc583b0700 1 --2- 172.21.15.195:0/3177368943 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcc50153de0 0x7fcc50068ea0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:00.208 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.197+0000 7fcc583b0700 1 -- 172.21.15.195:0/3177368943 >> 172.21.15.195:0/3177368943 conn(0x7fcc500c8740 msgr2=0x7fcc500c8b10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:50:00.208 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.197+0000 7fcc583b0700 1 -- 172.21.15.195:0/3177368943 shutdown_connections 2022-06-27T11:50:00.208 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.197+0000 7fcc583b0700 1 -- 172.21.15.195:0/3177368943 wait complete. 2022-06-27T11:50:00.212 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.199+0000 7fcc583b0700 1 Processor -- start 2022-06-27T11:50:00.213 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.199+0000 7fcc583b0700 1 -- start start 2022-06-27T11:50:00.213 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.199+0000 7fcc583b0700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcc500b8520 0x7fcc5016a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:00.213 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.199+0000 7fcc583b0700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcc500c3010 0x7fcc500bba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:00.213 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.199+0000 7fcc583b0700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcc50153de0 0x7fcc50150750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:00.214 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.199+0000 7fcc583b0700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc5016ae10 con 0x7fcc50153de0 2022-06-27T11:50:00.214 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.199+0000 7fcc583b0700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc500b7e70 con 0x7fcc500b8520 2022-06-27T11:50:00.214 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.199+0000 7fcc583b0700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc500b78f0 con 0x7fcc500c3010 2022-06-27T11:50:00.214 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.200+0000 7fcc5694d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcc50153de0 0x7fcc50150750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:00.215 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.200+0000 7fcc5694d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcc50153de0 0x7fcc50150750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:42602/0 (socket says 172.21.15.195:42602) 2022-06-27T11:50:00.215 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.200+0000 7fcc5694d700 1 -- 172.21.15.195:0/3904816783 learned_addr learned my addr 172.21.15.195:0/3904816783 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:50:00.215 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.200+0000 7fcc5594b700 1 --2- 172.21.15.195:0/3904816783 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcc500c3010 0x7fcc500bba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:00.215 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.200+0000 7fcc5614c700 1 --2- 172.21.15.195:0/3904816783 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcc500b8520 0x7fcc5016a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:00.216 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.200+0000 7fcc5694d700 1 -- 172.21.15.195:0/3904816783 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcc500c3010 msgr2=0x7fcc500bba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:00.216 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.200+0000 7fcc5694d700 1 --2- 172.21.15.195:0/3904816783 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcc500c3010 0x7fcc500bba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:00.216 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.200+0000 7fcc5694d700 1 -- 172.21.15.195:0/3904816783 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcc500b8520 msgr2=0x7fcc5016a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:00.216 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.200+0000 7fcc5694d700 1 --2- 172.21.15.195:0/3904816783 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcc500b8520 0x7fcc5016a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:00.217 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.200+0000 7fcc5694d700 1 -- 172.21.15.195:0/3904816783 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc48009a20 con 0x7fcc50153de0 2022-06-27T11:50:00.217 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.200+0000 7fcc5694d700 1 --2- 172.21.15.195:0/3904816783 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcc50153de0 0x7fcc50150750 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fcc48009fc0 tx=0x7fcc4800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:50:00.217 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.200+0000 7fcc3e7fc700 1 -- 172.21.15.195:0/3904816783 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc4800cc60 con 0x7fcc50153de0 2022-06-27T11:50:00.217 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.200+0000 7fcc3e7fc700 1 -- 172.21.15.195:0/3904816783 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc4800f040 con 0x7fcc50153de0 2022-06-27T11:50:00.218 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.200+0000 7fcc583b0700 1 -- 172.21.15.195:0/3904816783 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc4c0092e0 con 0x7fcc50153de0 2022-06-27T11:50:00.218 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.200+0000 7fcc3e7fc700 1 -- 172.21.15.195:0/3904816783 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc480038a0 con 0x7fcc50153de0 2022-06-27T11:50:00.218 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.200+0000 7fcc583b0700 1 -- 172.21.15.195:0/3904816783 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc50167850 con 0x7fcc50153de0 2022-06-27T11:50:00.218 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.201+0000 7fcc583b0700 1 -- 172.21.15.195:0/3904816783 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc500722c0 con 0x7fcc50153de0 2022-06-27T11:50:00.219 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.202+0000 7fcc3e7fc700 1 -- 172.21.15.195:0/3904816783 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fcc48014480 con 0x7fcc50153de0 2022-06-27T11:50:00.219 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.202+0000 7fcc3e7fc700 1 --2- 172.21.15.195:0/3904816783 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcc30063d00 0x7fcc300661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:00.219 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.202+0000 7fcc3e7fc700 1 -- 172.21.15.195:0/3904816783 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6202+0+0 (secure 0 0 0) 0x7fcc48081560 con 0x7fcc50153de0 2022-06-27T11:50:00.219 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.202+0000 7fcc5614c700 1 --2- 172.21.15.195:0/3904816783 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcc30063d00 0x7fcc300661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:00.220 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.203+0000 7fcc5614c700 1 --2- 172.21.15.195:0/3904816783 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcc30063d00 0x7fcc300661b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fcc40000c00 tx=0x7fcc40009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:50:00.220 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.206+0000 7fcc3e7fc700 1 -- 172.21.15.195:0/3904816783 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fcc48050060 con 0x7fcc50153de0 2022-06-27T11:50:00.442 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.439+0000 7fcc583b0700 1 -- 172.21.15.195:0/3904816783 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc500680e0 con 0x7fcc50153de0 2022-06-27T11:50:00.443 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.440+0000 7fcc3e7fc700 1 -- 172.21.15.195:0/3904816783 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+5361 (secure 0 0 0) 0x7fcc48053680 con 0x7fcc50153de0 2022-06-27T11:50:00.444 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:50:00.445 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:49:59.869673+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"b","rank":0,"incarnation":9,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6833/864126677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":864126677},{"type":"v1","addr":"172.21.15.202:6833","nonce":864126677}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:50:00.447 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.442+0000 7fcc583b0700 1 -- 172.21.15.195:0/3904816783 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcc30063d00 msgr2=0x7fcc300661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:00.448 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.442+0000 7fcc583b0700 1 --2- 172.21.15.195:0/3904816783 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcc30063d00 0x7fcc300661b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fcc40000c00 tx=0x7fcc40009040 comp rx=0 tx=0).stop 2022-06-27T11:50:00.448 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.442+0000 7fcc583b0700 1 -- 172.21.15.195:0/3904816783 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcc50153de0 msgr2=0x7fcc50150750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:00.449 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.442+0000 7fcc583b0700 1 --2- 172.21.15.195:0/3904816783 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcc50153de0 0x7fcc50150750 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fcc48009fc0 tx=0x7fcc4800b040 comp rx=0 tx=0).stop 2022-06-27T11:50:00.449 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.442+0000 7fcc583b0700 1 -- 172.21.15.195:0/3904816783 shutdown_connections 2022-06-27T11:50:00.450 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.442+0000 7fcc583b0700 1 --2- 172.21.15.195:0/3904816783 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcc30063d00 0x7fcc300661b0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:00.450 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.442+0000 7fcc583b0700 1 --2- 172.21.15.195:0/3904816783 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcc50153de0 0x7fcc50150750 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:00.451 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.442+0000 7fcc583b0700 1 --2- 172.21.15.195:0/3904816783 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcc500b8520 0x7fcc5016a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:00.453 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.442+0000 7fcc583b0700 1 --2- 172.21.15.195:0/3904816783 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcc500c3010 0x7fcc500bba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:00.453 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.442+0000 7fcc583b0700 1 -- 172.21.15.195:0/3904816783 >> 172.21.15.195:0/3904816783 conn(0x7fcc500c8740 msgr2=0x7fcc500c4c50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:50:00.454 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.442+0000 7fcc583b0700 1 -- 172.21.15.195:0/3904816783 shutdown_connections 2022-06-27T11:50:00.454 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.442+0000 7fcc583b0700 1 -- 172.21.15.195:0/3904816783 wait complete. 2022-06-27T11:50:00.455 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 9 2022-06-27T11:50:00.457 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d down, removed from mdsmap 2022-06-27T11:50:00.458 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2022-06-27T11:50:00.458 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:50:00.584 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.584+0000 7f79a027a700 1 Processor -- start 2022-06-27T11:50:00.584 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.584+0000 7f79a027a700 1 -- start start 2022-06-27T11:50:00.604 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.585+0000 7f79a027a700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f799806fef0 0x7f79981570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:00.605 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.585+0000 7f79a027a700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f79980c34a0 0x7f79980c2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:00.606 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.585+0000 7f79a027a700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f799806fba0 con 0x7f799806e850 2022-06-27T11:50:00.620 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.585+0000 7f79a027a700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79981575e0 con 0x7f79980c34a0 2022-06-27T11:50:00.620 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.585+0000 7f79a027a700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f799806f5e0 con 0x7f799806fef0 2022-06-27T11:50:00.620 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.586+0000 7f799e016700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f799806e850 0x7f7998069de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:48794/0 (socket says 172.21.15.195:48794) 2022-06-27T11:50:00.621 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.586+0000 7f799e016700 1 -- 172.21.15.195:0/628501917 learned_addr learned my addr 172.21.15.195:0/628501917 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:50:00.621 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.586+0000 7f799d815700 1 --2- 172.21.15.195:0/628501917 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f799806fef0 0x7f79981570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:00.637 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.586+0000 7f799e817700 1 --2- 172.21.15.195:0/628501917 >> v2:172.21.15.202:3300/0 conn(0x7f79980c34a0 0x7f79980c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:00.637 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.586+0000 7f799d014700 1 -- 172.21.15.195:0/628501917 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3971601151 0 0) 0x7f799806fba0 con 0x7f799806e850 2022-06-27T11:50:00.638 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.586+0000 7f799d014700 1 -- 172.21.15.195:0/628501917 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7980003430 con 0x7f799806e850 2022-06-27T11:50:00.638 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.586+0000 7f799d014700 1 -- 172.21.15.195:0/628501917 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 809379298 0 0) 0x7f7980003430 con 0x7f799806e850 2022-06-27T11:50:00.640 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.586+0000 7f799d014700 1 -- 172.21.15.195:0/628501917 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f799806fef0 msgr2=0x7f79981570a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:50:00.640 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.586+0000 7f799d014700 1 --2- 172.21.15.195:0/628501917 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f799806fef0 0x7f79981570a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:00.641 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.586+0000 7f799d014700 1 -- 172.21.15.195:0/628501917 >> v2:172.21.15.202:3300/0 conn(0x7f79980c34a0 msgr2=0x7f79980c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:50:00.641 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.586+0000 7f799d014700 1 --2- 172.21.15.195:0/628501917 >> v2:172.21.15.202:3300/0 conn(0x7f79980c34a0 0x7f79980c2920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:00.642 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.586+0000 7f799d014700 1 -- 172.21.15.195:0/628501917 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f799806f8c0 con 0x7f799806e850 2022-06-27T11:50:00.642 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.586+0000 7f799d014700 1 -- 172.21.15.195:0/628501917 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7f7988001470 con 0x7f799806e850 2022-06-27T11:50:00.643 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.586+0000 7f799d014700 1 -- 172.21.15.195:0/628501917 >> v1:172.21.15.195:6789/0 conn(0x7f799806e850 legacy=0x7f7998069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:00.643 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.586+0000 7f799d014700 1 --2- 172.21.15.195:0/628501917 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7980003920 0x7f7980003d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:00.644 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.586+0000 7f799d014700 1 --2- 172.21.15.195:0/628501917 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f79800046d0 0x7f7980004b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:00.644 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.586+0000 7f799d014700 1 --2- 172.21.15.195:0/628501917 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7980005040 0x7f798000b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:00.645 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.586+0000 7f799d014700 1 -- 172.21.15.195:0/628501917 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79981575e0 con 0x7f7980003920 2022-06-27T11:50:00.645 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.586+0000 7f799d014700 1 -- 172.21.15.195:0/628501917 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f799806f5e0 con 0x7f7980005040 2022-06-27T11:50:00.645 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.586+0000 7f799d014700 1 -- 172.21.15.195:0/628501917 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f798000be60 con 0x7f79800046d0 2022-06-27T11:50:00.647 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.586+0000 7f799e016700 1 --2- 172.21.15.195:0/628501917 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7980003920 0x7f7980003d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:00.653 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.586+0000 7f799d815700 1 --2- 172.21.15.195:0/628501917 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f79800046d0 0x7f7980004b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:00.653 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.586+0000 7f799e817700 1 --2- 172.21.15.195:0/628501917 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7980005040 0x7f798000b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:00.653 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.587+0000 7f799e016700 1 -- 172.21.15.195:0/628501917 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f79800046d0 msgr2=0x7f7980004b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:50:00.654 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.587+0000 7f799e016700 1 --2- 172.21.15.195:0/628501917 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f79800046d0 0x7f7980004b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:00.658 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.587+0000 7f799e016700 1 -- 172.21.15.195:0/628501917 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7980005040 msgr2=0x7f798000b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:50:00.658 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.587+0000 7f799e016700 1 --2- 172.21.15.195:0/628501917 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7980005040 0x7f798000b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:00.659 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.587+0000 7f799e016700 1 -- 172.21.15.195:0/628501917 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7980003430 con 0x7f7980003920 2022-06-27T11:50:00.659 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.587+0000 7f799e016700 1 --2- 172.21.15.195:0/628501917 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7980003920 0x7f7980003d30 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f798800c240 tx=0x7f7988001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=161708a02c12f835 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:50:00.660 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.587+0000 7f799d014700 1 -- 172.21.15.195:0/628501917 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f798800cb30 con 0x7f7980003920 2022-06-27T11:50:00.660 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.587+0000 7f799d014700 1 -- 172.21.15.195:0/628501917 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f798800ccb0 con 0x7f7980003920 2022-06-27T11:50:00.661 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.587+0000 7f799d014700 1 -- 172.21.15.195:0/628501917 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7988007320 con 0x7f7980003920 2022-06-27T11:50:00.661 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.587+0000 7f79a027a700 1 -- 172.21.15.195:0/628501917 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7980003920 msgr2=0x7f7980003d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:00.661 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.587+0000 7f79a027a700 1 --2- 172.21.15.195:0/628501917 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7980003920 0x7f7980003d30 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f798800c240 tx=0x7f7988001e70 comp rx=0 tx=0).stop 2022-06-27T11:50:00.662 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.587+0000 7f799e817700 1 -- 172.21.15.195:0/628501917 reap_dead start 2022-06-27T11:50:00.662 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.587+0000 7f79a027a700 1 -- 172.21.15.195:0/628501917 shutdown_connections 2022-06-27T11:50:00.663 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.587+0000 7f79a027a700 1 -- 172.21.15.195:0/628501917 >> 172.21.15.195:0/628501917 conn(0x7f79980c6350 msgr2=0x7f79980c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:50:00.663 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.587+0000 7f79a027a700 1 -- 172.21.15.195:0/628501917 shutdown_connections 2022-06-27T11:50:00.663 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.587+0000 7f79a027a700 1 -- 172.21.15.195:0/628501917 wait complete. 2022-06-27T11:50:00.664 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.589+0000 7f79a027a700 1 Processor -- start 2022-06-27T11:50:00.664 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.589+0000 7f79a027a700 1 -- start start 2022-06-27T11:50:00.665 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.589+0000 7f79a027a700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7980003920 0x7f79981557c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:00.665 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.589+0000 7f79a027a700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f799814cf00 0x7f79981664d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:00.667 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.589+0000 7f79a027a700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f79980c2950 0x7f7998166d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:00.668 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.589+0000 7f79a027a700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f799814a070 con 0x7f7980003920 2022-06-27T11:50:00.668 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.589+0000 7f79a027a700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79981548f0 con 0x7f79980c2950 2022-06-27T11:50:00.668 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.589+0000 7f79a027a700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7998163cc0 con 0x7f799814cf00 2022-06-27T11:50:00.669 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.589+0000 7f799e016700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7980003920 0x7f79981557c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:00.669 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.589+0000 7f799e016700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7980003920 0x7f79981557c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:42618/0 (socket says 172.21.15.195:42618) 2022-06-27T11:50:00.670 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.589+0000 7f799e016700 1 -- 172.21.15.195:0/1443349206 learned_addr learned my addr 172.21.15.195:0/1443349206 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:50:00.670 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.590+0000 7f799e817700 1 --2- 172.21.15.195:0/1443349206 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f79980c2950 0x7f7998166d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:00.671 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.590+0000 7f799d815700 1 --2- 172.21.15.195:0/1443349206 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f799814cf00 0x7f79981664d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:00.671 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.590+0000 7f799e016700 1 -- 172.21.15.195:0/1443349206 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f799814cf00 msgr2=0x7f79981664d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:00.671 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.590+0000 7f799e016700 1 --2- 172.21.15.195:0/1443349206 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f799814cf00 0x7f79981664d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:00.672 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.590+0000 7f799e016700 1 -- 172.21.15.195:0/1443349206 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f79980c2950 msgr2=0x7f7998166d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:00.672 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.590+0000 7f799e016700 1 --2- 172.21.15.195:0/1443349206 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f79980c2950 0x7f7998166d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:00.672 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.590+0000 7f799e016700 1 -- 172.21.15.195:0/1443349206 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7998154b00 con 0x7f7980003920 2022-06-27T11:50:00.673 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.590+0000 7f799d815700 1 --2- 172.21.15.195:0/1443349206 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f799814cf00 0x7f79981664d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T11:50:00.673 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.590+0000 7f799e016700 1 --2- 172.21.15.195:0/1443349206 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7980003920 0x7f79981557c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f79800044b0 tx=0x7f7988015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:50:00.674 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.590+0000 7f79867fc700 1 -- 172.21.15.195:0/1443349206 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79880072e0 con 0x7f7980003920 2022-06-27T11:50:00.674 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.590+0000 7f79867fc700 1 -- 172.21.15.195:0/1443349206 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7988000cb0 con 0x7f7980003920 2022-06-27T11:50:00.675 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.590+0000 7f79867fc700 1 -- 172.21.15.195:0/1443349206 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7988004920 con 0x7f7980003920 2022-06-27T11:50:00.675 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.590+0000 7f79a027a700 1 -- 172.21.15.195:0/1443349206 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7998154de0 con 0x7f7980003920 2022-06-27T11:50:00.675 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.590+0000 7f79a027a700 1 -- 172.21.15.195:0/1443349206 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79981542e0 con 0x7f7980003920 2022-06-27T11:50:00.676 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.591+0000 7f79a027a700 1 -- 172.21.15.195:0/1443349206 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79980722c0 con 0x7f7980003920 2022-06-27T11:50:00.676 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.591+0000 7f79867fc700 1 -- 172.21.15.195:0/1443349206 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f7988010070 con 0x7f7980003920 2022-06-27T11:50:00.677 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.592+0000 7f79867fc700 1 --2- 172.21.15.195:0/1443349206 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7978063d30 0x7f79780661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:00.677 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.592+0000 7f79867fc700 1 -- 172.21.15.195:0/1443349206 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6202+0+0 (secure 0 0 0) 0x7f7988081d20 con 0x7f7980003920 2022-06-27T11:50:00.678 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.592+0000 7f799d815700 1 --2- 172.21.15.195:0/1443349206 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7978063d30 0x7f79780661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:00.679 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.592+0000 7f799d815700 1 --2- 172.21.15.195:0/1443349206 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7978063d30 0x7f79780661e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f79940092b0 tx=0x7f7994008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:50:00.679 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.596+0000 7f79867fc700 1 -- 172.21.15.195:0/1443349206 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f7988050880 con 0x7f7980003920 2022-06-27T11:50:00.827 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.821+0000 7f79a027a700 1 -- 172.21.15.195:0/1443349206 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f799806f240 con 0x7f7980003920 2022-06-27T11:50:00.828 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.822+0000 7f79867fc700 1 -- 172.21.15.195:0/1443349206 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+5361 (secure 0 0 0) 0x7f7988022090 con 0x7f7980003920 2022-06-27T11:50:00.829 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:50:00.830 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:49:59.869673+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"b","rank":0,"incarnation":9,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6833/864126677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":864126677},{"type":"v1","addr":"172.21.15.202:6833","nonce":864126677}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:50:00.832 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.824+0000 7f79a027a700 1 -- 172.21.15.195:0/1443349206 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7978063d30 msgr2=0x7f79780661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:00.833 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.824+0000 7f79a027a700 1 --2- 172.21.15.195:0/1443349206 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7978063d30 0x7f79780661e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f79940092b0 tx=0x7f7994008040 comp rx=0 tx=0).stop 2022-06-27T11:50:00.833 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.824+0000 7f79a027a700 1 -- 172.21.15.195:0/1443349206 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7980003920 msgr2=0x7f79981557c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:00.834 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.824+0000 7f79a027a700 1 --2- 172.21.15.195:0/1443349206 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7980003920 0x7f79981557c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f79800044b0 tx=0x7f7988015000 comp rx=0 tx=0).stop 2022-06-27T11:50:00.834 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.824+0000 7f79a027a700 1 -- 172.21.15.195:0/1443349206 shutdown_connections 2022-06-27T11:50:00.835 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.824+0000 7f79a027a700 1 --2- 172.21.15.195:0/1443349206 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7978063d30 0x7f79780661e0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:00.835 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.824+0000 7f79a027a700 1 --2- 172.21.15.195:0/1443349206 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7980003920 0x7f79981557c0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:00.836 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.824+0000 7f79a027a700 1 --2- 172.21.15.195:0/1443349206 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f79980c2950 0x7f7998166d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:00.836 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.824+0000 7f79a027a700 1 --2- 172.21.15.195:0/1443349206 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f799814cf00 0x7f79981664d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:00.836 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.824+0000 7f79a027a700 1 -- 172.21.15.195:0/1443349206 >> 172.21.15.195:0/1443349206 conn(0x7f79980c6350 msgr2=0x7f79980c8170 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:50:00.837 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.824+0000 7f79a027a700 1 -- 172.21.15.195:0/1443349206 shutdown_connections 2022-06-27T11:50:00.837 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:00.824+0000 7f79a027a700 1 -- 172.21.15.195:0/1443349206 wait complete. 2022-06-27T11:50:00.837 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 9 2022-06-27T11:50:00.841 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 9, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-06-27T11:47:52.358095+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 9, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4287": { "addr": "172.21.15.202:6833/864126677", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6832", "nonce": 864126677, "type": "v2" }, { "addr": "172.21.15.202:6833", "nonce": 864126677, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4287, "incarnation": 9, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 26, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-06-27T11:49:59.869673+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4287 } } } ], "standbys": [ { "addr": "172.21.15.202:6837/3305006841", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6836", "nonce": 3305006841, "type": "v2" }, { "addr": "172.21.15.202:6837", "nonce": 3305006841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6835/2060709822", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6834", "nonce": 2060709822, "type": "v2" }, { "addr": "172.21.15.195:6835", "nonce": 2060709822, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6838/1598751652", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6836", "nonce": 1598751652, "type": "v2" }, { "addr": "172.21.15.195:6838", "nonce": 1598751652, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4275, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6839/1313800661", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6837", "nonce": 1313800661, "type": "v2" }, { "addr": "172.21.15.195:6839", "nonce": 1313800661, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4278, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-06-27T11:50:02.845 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:50:02.991 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:02.980+0000 7fa360500700 1 Processor -- start 2022-06-27T11:50:02.992 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:02.980+0000 7fa360500700 1 -- start start 2022-06-27T11:50:02.992 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:02.981+0000 7fa360500700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fa35806f620 0x7fa358156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:02.992 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:02.981+0000 7fa360500700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3580c76b0 con 0x7fa358150950 2022-06-27T11:50:02.993 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:02.981+0000 7fa360500700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3580c8250 con 0x7fa358150580 2022-06-27T11:50:02.994 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:02.981+0000 7fa360500700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa358157430 con 0x7fa35806f620 2022-06-27T11:50:02.994 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:02.981+0000 7fa35ea9d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fa35806f620 0x7fa358156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:02.994 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:02.981+0000 7fa35da9b700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fa358150950 0x7fa35806d6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:48814/0 (socket says 172.21.15.195:48814) 2022-06-27T11:50:02.995 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:02.981+0000 7fa35ea9d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fa35806f620 0x7fa358156ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:42630/0 (socket says 172.21.15.195:42630) 2022-06-27T11:50:02.995 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:02.981+0000 7fa35da9b700 1 -- 172.21.15.195:0/1330626574 learned_addr learned my addr 172.21.15.195:0/1330626574 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:50:02.996 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:02.981+0000 7fa35ea9d700 1 -- 172.21.15.195:0/1330626574 >> v1:172.21.15.202:6789/0 conn(0x7fa358150580 legacy=0x7fa3580c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:02.997 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:02.981+0000 7fa35ea9d700 1 -- 172.21.15.195:0/1330626574 >> v1:172.21.15.195:6789/0 conn(0x7fa358150950 legacy=0x7fa35806d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:03.002 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:02.981+0000 7fa35ea9d700 1 -- 172.21.15.195:0/1330626574 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3580bad70 con 0x7fa35806f620 2022-06-27T11:50:03.002 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:02.981+0000 7fa35d29a700 1 -- 172.21.15.195:0/1330626574 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3006358304 0 0) 0x7fa3580c76b0 con 0x7fa358150950 2022-06-27T11:50:03.003 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:02.981+0000 7fa35ea9d700 1 --2- 172.21.15.195:0/1330626574 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa35806f620 0x7fa358156ef0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fa350009e60 tx=0x7fa350009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ccd1f7d4a68b1f9 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:50:03.003 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:02.981+0000 7fa35d29a700 1 -- 172.21.15.195:0/1330626574 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa35000e070 con 0x7fa35806f620 2022-06-27T11:50:03.003 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:02.982+0000 7fa35d29a700 1 -- 172.21.15.195:0/1330626574 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa35000bc30 con 0x7fa35806f620 2022-06-27T11:50:03.003 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:02.982+0000 7fa35d29a700 1 -- 172.21.15.195:0/1330626574 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa350012430 con 0x7fa35806f620 2022-06-27T11:50:03.004 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:02.982+0000 7fa360500700 1 -- 172.21.15.195:0/1330626574 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa35806f620 msgr2=0x7fa358156ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:03.004 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:02.982+0000 7fa360500700 1 --2- 172.21.15.195:0/1330626574 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa35806f620 0x7fa358156ef0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fa350009e60 tx=0x7fa350009780 comp rx=0 tx=0).stop 2022-06-27T11:50:03.005 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:02.982+0000 7fa360500700 1 -- 172.21.15.195:0/1330626574 shutdown_connections 2022-06-27T11:50:03.005 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:02.982+0000 7fa360500700 1 --2- 172.21.15.195:0/1330626574 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa35806f620 0x7fa358156ef0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:03.006 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:02.982+0000 7fa360500700 1 -- 172.21.15.195:0/1330626574 >> 172.21.15.195:0/1330626574 conn(0x7fa358063af0 msgr2=0x7fa358063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:50:03.006 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:02.982+0000 7fa360500700 1 -- 172.21.15.195:0/1330626574 shutdown_connections 2022-06-27T11:50:03.006 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:02.982+0000 7fa360500700 1 -- 172.21.15.195:0/1330626574 wait complete. 2022-06-27T11:50:03.006 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:02.984+0000 7fa360500700 1 Processor -- start 2022-06-27T11:50:03.007 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:02.984+0000 7fa360500700 1 -- start start 2022-06-27T11:50:03.008 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:02.985+0000 7fa360500700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa35806f620 0x7fa3580c46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:03.008 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:02.985+0000 7fa360500700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa358150580 0x7fa35814b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:03.008 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:02.985+0000 7fa360500700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa358150950 0x7fa3580bbd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:03.008 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:02.985+0000 7fa360500700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3580bb290 con 0x7fa358150950 2022-06-27T11:50:03.009 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:02.985+0000 7fa360500700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3580c3940 con 0x7fa358150580 2022-06-27T11:50:03.009 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:02.985+0000 7fa360500700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3580c3be0 con 0x7fa35806f620 2022-06-27T11:50:03.009 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:02.985+0000 7fa35ea9d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa358150950 0x7fa3580bbd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:03.009 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:02.985+0000 7fa35ea9d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa358150950 0x7fa3580bbd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:42636/0 (socket says 172.21.15.195:42636) 2022-06-27T11:50:03.010 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:02.985+0000 7fa35ea9d700 1 -- 172.21.15.195:0/3510945265 learned_addr learned my addr 172.21.15.195:0/3510945265 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:50:03.010 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:02.985+0000 7fa35da9b700 1 --2- 172.21.15.195:0/3510945265 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa358150580 0x7fa35814b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:03.011 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:02.985+0000 7fa35e29c700 1 --2- 172.21.15.195:0/3510945265 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa35806f620 0x7fa3580c46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:03.011 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:02.985+0000 7fa35ea9d700 1 -- 172.21.15.195:0/3510945265 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa35806f620 msgr2=0x7fa3580c46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:03.011 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:02.985+0000 7fa35ea9d700 1 --2- 172.21.15.195:0/3510945265 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa35806f620 0x7fa3580c46b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:03.012 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:02.985+0000 7fa35ea9d700 1 -- 172.21.15.195:0/3510945265 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa358150580 msgr2=0x7fa35814b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:03.012 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:02.985+0000 7fa35ea9d700 1 --2- 172.21.15.195:0/3510945265 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa358150580 0x7fa35814b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:03.012 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:02.985+0000 7fa35ea9d700 1 -- 172.21.15.195:0/3510945265 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa350009a20 con 0x7fa358150950 2022-06-27T11:50:03.012 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:02.985+0000 7fa35ea9d700 1 --2- 172.21.15.195:0/3510945265 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa358150950 0x7fa3580bbd70 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fa350013000 tx=0x7fa350019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:50:03.013 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:02.985+0000 7fa3467fc700 1 -- 172.21.15.195:0/3510945265 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa35001b420 con 0x7fa358150950 2022-06-27T11:50:03.013 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:02.985+0000 7fa3467fc700 1 -- 172.21.15.195:0/3510945265 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa35001b5a0 con 0x7fa358150950 2022-06-27T11:50:03.013 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:02.986+0000 7fa3467fc700 1 -- 172.21.15.195:0/3510945265 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa35001b960 con 0x7fa358150950 2022-06-27T11:50:03.013 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:02.986+0000 7fa360500700 1 -- 172.21.15.195:0/3510945265 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3580c3680 con 0x7fa358150950 2022-06-27T11:50:03.013 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:02.986+0000 7fa360500700 1 -- 172.21.15.195:0/3510945265 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa35814ae50 con 0x7fa358150950 2022-06-27T11:50:03.014 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:02.986+0000 7fa360500700 1 -- 172.21.15.195:0/3510945265 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3580bba70 con 0x7fa358150950 2022-06-27T11:50:03.014 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:02.987+0000 7fa3467fc700 1 -- 172.21.15.195:0/3510945265 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fa35002c030 con 0x7fa358150950 2022-06-27T11:50:03.014 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:02.987+0000 7fa3467fc700 1 --2- 172.21.15.195:0/3510945265 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa338063d00 0x7fa3380661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:03.015 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:02.987+0000 7fa3467fc700 1 -- 172.21.15.195:0/3510945265 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6202+0+0 (secure 0 0 0) 0x7fa3500826b0 con 0x7fa358150950 2022-06-27T11:50:03.015 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:02.987+0000 7fa35e29c700 1 --2- 172.21.15.195:0/3510945265 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa338063d00 0x7fa3380661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:03.015 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:02.988+0000 7fa35e29c700 1 --2- 172.21.15.195:0/3510945265 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa338063d00 0x7fa3380661b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa354001880 tx=0x7fa354009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:50:03.016 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:02.992+0000 7fa3467fc700 1 -- 172.21.15.195:0/3510945265 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fa350051210 con 0x7fa358150950 2022-06-27T11:50:03.216 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:03.217+0000 7fa360500700 1 -- 172.21.15.195:0/3510945265 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa3580722c0 con 0x7fa358150950 2022-06-27T11:50:03.217 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:03.217+0000 7fa3467fc700 1 -- 172.21.15.195:0/3510945265 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+5361 (secure 0 0 0) 0x7fa35002a020 con 0x7fa358150950 2022-06-27T11:50:03.218 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:50:03.219 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:49:59.869673+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"b","rank":0,"incarnation":9,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6833/864126677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":864126677},{"type":"v1","addr":"172.21.15.202:6833","nonce":864126677}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:50:03.221 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:03.219+0000 7fa360500700 1 -- 172.21.15.195:0/3510945265 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa338063d00 msgr2=0x7fa3380661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:03.222 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:03.219+0000 7fa360500700 1 --2- 172.21.15.195:0/3510945265 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa338063d00 0x7fa3380661b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa354001880 tx=0x7fa354009040 comp rx=0 tx=0).stop 2022-06-27T11:50:03.222 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:03.219+0000 7fa360500700 1 -- 172.21.15.195:0/3510945265 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa358150950 msgr2=0x7fa3580bbd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:03.222 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:03.219+0000 7fa360500700 1 --2- 172.21.15.195:0/3510945265 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa358150950 0x7fa3580bbd70 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fa350013000 tx=0x7fa350019000 comp rx=0 tx=0).stop 2022-06-27T11:50:03.223 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:03.219+0000 7fa360500700 1 -- 172.21.15.195:0/3510945265 shutdown_connections 2022-06-27T11:50:03.223 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:03.219+0000 7fa360500700 1 --2- 172.21.15.195:0/3510945265 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa338063d00 0x7fa3380661b0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:03.225 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:03.219+0000 7fa360500700 1 --2- 172.21.15.195:0/3510945265 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa358150950 0x7fa3580bbd70 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:03.226 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:03.219+0000 7fa360500700 1 --2- 172.21.15.195:0/3510945265 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa35806f620 0x7fa3580c46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:03.226 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:03.219+0000 7fa360500700 1 --2- 172.21.15.195:0/3510945265 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa358150580 0x7fa35814b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:03.227 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:03.219+0000 7fa360500700 1 -- 172.21.15.195:0/3510945265 >> 172.21.15.195:0/3510945265 conn(0x7fa358063af0 msgr2=0x7fa3580c79e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:50:03.227 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:03.219+0000 7fa360500700 1 -- 172.21.15.195:0/3510945265 shutdown_connections 2022-06-27T11:50:03.228 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:03.219+0000 7fa360500700 1 -- 172.21.15.195:0/3510945265 wait complete. 2022-06-27T11:50:03.228 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 9 2022-06-27T11:50:03.231 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T11:50:05.232 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:50:05.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.362+0000 7fb5c1911700 1 Processor -- start 2022-06-27T11:50:05.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.362+0000 7fb5c1911700 1 -- start start 2022-06-27T11:50:05.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.363+0000 7fb5c1911700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fb5bc0c40d0 0x7fb5bc157010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:05.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.363+0000 7fb5c1911700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fb5bc067ed0 0x7fb5bc14cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:05.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.363+0000 7fb5c1911700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5bc156b80 con 0x7fb5bc0c6190 2022-06-27T11:50:05.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.363+0000 7fb5c1911700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5bc0682f0 con 0x7fb5bc067ed0 2022-06-27T11:50:05.372 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.363+0000 7fb5c1911700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5bc06a040 con 0x7fb5bc0c40d0 2022-06-27T11:50:05.372 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.363+0000 7fb5bb7fe700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fb5bc067ed0 0x7fb5bc14cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:05.373 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.363+0000 7fb5bb7fe700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fb5bc067ed0 0x7fb5bc14cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:42642/0 (socket says 172.21.15.195:42642) 2022-06-27T11:50:05.373 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.363+0000 7fb5bb7fe700 1 -- 172.21.15.195:0/3036046914 learned_addr learned my addr 172.21.15.195:0/3036046914 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:50:05.373 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.363+0000 7fb5baffd700 1 --2- 172.21.15.195:0/3036046914 >> v2:172.21.15.202:3300/0 conn(0x7fb5bc0c40d0 0x7fb5bc157010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:05.374 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.364+0000 7fb5bb7fe700 1 -- 172.21.15.195:0/3036046914 >> v2:172.21.15.202:3300/0 conn(0x7fb5bc0c40d0 msgr2=0x7fb5bc157010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:50:05.374 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.364+0000 7fb5bb7fe700 1 --2- 172.21.15.195:0/3036046914 >> v2:172.21.15.202:3300/0 conn(0x7fb5bc0c40d0 0x7fb5bc157010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:05.375 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.364+0000 7fb5bb7fe700 1 -- 172.21.15.195:0/3036046914 >> v1:172.21.15.202:6789/0 conn(0x7fb5bc0c6190 legacy=0x7fb5bc0c44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:05.375 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.364+0000 7fb5bb7fe700 1 -- 172.21.15.195:0/3036046914 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5bc06a300 con 0x7fb5bc067ed0 2022-06-27T11:50:05.376 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.364+0000 7fb5bb7fe700 1 --2- 172.21.15.195:0/3036046914 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb5bc067ed0 0x7fb5bc14cb80 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc0cdb00 tx=0x7fb5b4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d3c480b4c4b1577 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:50:05.377 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.364+0000 7fb5b9ffb700 1 -- 172.21.15.195:0/3036046914 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5b400e070 con 0x7fb5bc067ed0 2022-06-27T11:50:05.378 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.364+0000 7fb5b9ffb700 1 -- 172.21.15.195:0/3036046914 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5b400ba20 con 0x7fb5bc067ed0 2022-06-27T11:50:05.378 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.364+0000 7fb5b9ffb700 1 -- 172.21.15.195:0/3036046914 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5b40092e0 con 0x7fb5bc067ed0 2022-06-27T11:50:05.379 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.364+0000 7fb5c1911700 1 -- 172.21.15.195:0/3036046914 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb5bc067ed0 msgr2=0x7fb5bc14cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:05.379 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.364+0000 7fb5c1911700 1 --2- 172.21.15.195:0/3036046914 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb5bc067ed0 0x7fb5bc14cb80 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc0cdb00 tx=0x7fb5b4009840 comp rx=0 tx=0).stop 2022-06-27T11:50:05.380 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.364+0000 7fb5c1911700 1 -- 172.21.15.195:0/3036046914 shutdown_connections 2022-06-27T11:50:05.380 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.364+0000 7fb5c1911700 1 --2- 172.21.15.195:0/3036046914 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb5bc067ed0 0x7fb5bc14cb80 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:05.380 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.364+0000 7fb5c1911700 1 --2- 172.21.15.195:0/3036046914 >> v2:172.21.15.202:3300/0 conn(0x7fb5bc0c40d0 0x7fb5bc157010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:05.381 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.364+0000 7fb5c1911700 1 -- 172.21.15.195:0/3036046914 >> 172.21.15.195:0/3036046914 conn(0x7fb5bc0ccb80 msgr2=0x7fb5bc0ccf80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:50:05.381 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.364+0000 7fb5c1911700 1 -- 172.21.15.195:0/3036046914 shutdown_connections 2022-06-27T11:50:05.382 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.364+0000 7fb5c1911700 1 -- 172.21.15.195:0/3036046914 wait complete. 2022-06-27T11:50:05.382 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.366+0000 7fb5c1911700 1 Processor -- start 2022-06-27T11:50:05.382 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.366+0000 7fb5c1911700 1 -- start start 2022-06-27T11:50:05.383 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.366+0000 7fb5c1911700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb5bc067ed0 0x7fb5bc166fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:05.383 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.366+0000 7fb5c1911700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb5bc0c40d0 0x7fb5bc152570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:05.384 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.366+0000 7fb5c1911700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb5bc0c6190 0x7fb5bc152ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:05.384 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.366+0000 7fb5c1911700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5bc1674e0 con 0x7fb5bc067ed0 2022-06-27T11:50:05.384 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.366+0000 7fb5c1911700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5bc1520e0 con 0x7fb5bc0c40d0 2022-06-27T11:50:05.385 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.366+0000 7fb5c1911700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5bc0c0a40 con 0x7fb5bc0c6190 2022-06-27T11:50:05.385 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.367+0000 7fb5baffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb5bc067ed0 0x7fb5bc166fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:05.385 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.367+0000 7fb5baffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb5bc067ed0 0x7fb5bc166fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:42646/0 (socket says 172.21.15.195:42646) 2022-06-27T11:50:05.386 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.367+0000 7fb5baffd700 1 -- 172.21.15.195:0/2402120244 learned_addr learned my addr 172.21.15.195:0/2402120244 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:50:05.387 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.367+0000 7fb5ba7fc700 1 --2- 172.21.15.195:0/2402120244 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb5bc0c40d0 0x7fb5bc152570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:05.387 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.367+0000 7fb5bb7fe700 1 --2- 172.21.15.195:0/2402120244 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb5bc0c6190 0x7fb5bc152ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:05.388 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.367+0000 7fb5baffd700 1 -- 172.21.15.195:0/2402120244 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb5bc0c6190 msgr2=0x7fb5bc152ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:05.388 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.367+0000 7fb5baffd700 1 --2- 172.21.15.195:0/2402120244 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb5bc0c6190 0x7fb5bc152ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:05.389 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.367+0000 7fb5baffd700 1 -- 172.21.15.195:0/2402120244 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb5bc0c40d0 msgr2=0x7fb5bc152570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:05.389 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.367+0000 7fb5baffd700 1 --2- 172.21.15.195:0/2402120244 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb5bc0c40d0 0x7fb5bc152570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:05.389 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.367+0000 7fb5baffd700 1 -- 172.21.15.195:0/2402120244 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5b4009ae0 con 0x7fb5bc067ed0 2022-06-27T11:50:05.389 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.367+0000 7fb5baffd700 1 --2- 172.21.15.195:0/2402120244 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb5bc067ed0 0x7fb5bc166fa0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fb5b000e970 tx=0x7fb5b0009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:50:05.389 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.367+0000 7fb5a2ffd700 1 -- 172.21.15.195:0/2402120244 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5b0013680 con 0x7fb5bc067ed0 2022-06-27T11:50:05.390 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.367+0000 7fb5a2ffd700 1 -- 172.21.15.195:0/2402120244 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5b00137e0 con 0x7fb5bc067ed0 2022-06-27T11:50:05.390 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.367+0000 7fb5a2ffd700 1 -- 172.21.15.195:0/2402120244 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5b0013b60 con 0x7fb5bc067ed0 2022-06-27T11:50:05.391 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.367+0000 7fb5c1911700 1 -- 172.21.15.195:0/2402120244 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5bc0c0d20 con 0x7fb5bc067ed0 2022-06-27T11:50:05.391 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.367+0000 7fb5c1911700 1 -- 172.21.15.195:0/2402120244 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5bc153a50 con 0x7fb5bc067ed0 2022-06-27T11:50:05.391 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.368+0000 7fb5c1911700 1 -- 172.21.15.195:0/2402120244 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5bc1523c0 con 0x7fb5bc067ed0 2022-06-27T11:50:05.392 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.368+0000 7fb5a2ffd700 1 -- 172.21.15.195:0/2402120244 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fb5b0019030 con 0x7fb5bc067ed0 2022-06-27T11:50:05.392 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.369+0000 7fb5a2ffd700 1 --2- 172.21.15.195:0/2402120244 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb59c063d00 0x7fb59c0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:05.392 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.369+0000 7fb5ba7fc700 1 --2- 172.21.15.195:0/2402120244 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb59c063d00 0x7fb59c0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:05.393 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.369+0000 7fb5a2ffd700 1 -- 172.21.15.195:0/2402120244 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6202+0+0 (secure 0 0 0) 0x7fb5b0080850 con 0x7fb5bc067ed0 2022-06-27T11:50:05.393 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.369+0000 7fb5ba7fc700 1 --2- 172.21.15.195:0/2402120244 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb59c063d00 0x7fb59c0661b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc0c0650 tx=0x7fb5ac009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:50:05.394 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.373+0000 7fb5a2ffd700 1 -- 172.21.15.195:0/2402120244 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fb5b004f350 con 0x7fb5bc067ed0 2022-06-27T11:50:05.599 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.597+0000 7fb5c1911700 1 -- 172.21.15.195:0/2402120244 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb5bc0722c0 con 0x7fb5bc067ed0 2022-06-27T11:50:05.600 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.598+0000 7fb5a2ffd700 1 -- 172.21.15.195:0/2402120244 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+5361 (secure 0 0 0) 0x7fb5b0052970 con 0x7fb5bc067ed0 2022-06-27T11:50:05.600 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:50:05.601 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:49:59.869673+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"b","rank":0,"incarnation":9,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6833/864126677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":864126677},{"type":"v1","addr":"172.21.15.202:6833","nonce":864126677}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:50:05.604 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.600+0000 7fb5c1911700 1 -- 172.21.15.195:0/2402120244 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb59c063d00 msgr2=0x7fb59c0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:05.604 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.600+0000 7fb5c1911700 1 --2- 172.21.15.195:0/2402120244 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb59c063d00 0x7fb59c0661b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc0c0650 tx=0x7fb5ac009040 comp rx=0 tx=0).stop 2022-06-27T11:50:05.605 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.600+0000 7fb5c1911700 1 -- 172.21.15.195:0/2402120244 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb5bc067ed0 msgr2=0x7fb5bc166fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:05.605 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.600+0000 7fb5c1911700 1 --2- 172.21.15.195:0/2402120244 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb5bc067ed0 0x7fb5bc166fa0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fb5b000e970 tx=0x7fb5b0009060 comp rx=0 tx=0).stop 2022-06-27T11:50:05.606 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.600+0000 7fb5c1911700 1 -- 172.21.15.195:0/2402120244 shutdown_connections 2022-06-27T11:50:05.606 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.600+0000 7fb5c1911700 1 --2- 172.21.15.195:0/2402120244 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb59c063d00 0x7fb59c0661b0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:05.607 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.600+0000 7fb5c1911700 1 --2- 172.21.15.195:0/2402120244 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb5bc067ed0 0x7fb5bc166fa0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:05.607 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.600+0000 7fb5c1911700 1 --2- 172.21.15.195:0/2402120244 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb5bc0c6190 0x7fb5bc152ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:05.608 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.600+0000 7fb5c1911700 1 --2- 172.21.15.195:0/2402120244 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb5bc0c40d0 0x7fb5bc152570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:05.608 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.600+0000 7fb5c1911700 1 -- 172.21.15.195:0/2402120244 >> 172.21.15.195:0/2402120244 conn(0x7fb5bc0ccb80 msgr2=0x7fb5bc0c9600 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:50:05.608 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.600+0000 7fb5c1911700 1 -- 172.21.15.195:0/2402120244 shutdown_connections 2022-06-27T11:50:05.609 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:05.600+0000 7fb5c1911700 1 -- 172.21.15.195:0/2402120244 wait complete. 2022-06-27T11:50:05.609 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 9 2022-06-27T11:50:05.612 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T11:50:07.614 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:50:07.752 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.741+0000 7fa975b18700 1 Processor -- start 2022-06-27T11:50:07.753 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.741+0000 7fa975b18700 1 -- start start 2022-06-27T11:50:07.753 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.742+0000 7fa975b18700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fa97014e490 0x7fa97006cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:07.754 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.742+0000 7fa975b18700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fa97014e860 0x7fa9700c3840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:07.754 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.742+0000 7fa975b18700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9700c33b0 con 0x7fa9700c3d80 2022-06-27T11:50:07.755 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.742+0000 7fa975b18700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa97006d520 con 0x7fa97014e490 2022-06-27T11:50:07.755 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.742+0000 7fa975b18700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa97006f220 con 0x7fa97014e860 2022-06-27T11:50:07.756 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.742+0000 7fa96f7fe700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fa97014e490 0x7fa97006cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:07.756 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.742+0000 7fa96f7fe700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fa97014e490 0x7fa97006cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:42664/0 (socket says 172.21.15.195:42664) 2022-06-27T11:50:07.757 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.742+0000 7fa96f7fe700 1 -- 172.21.15.195:0/3854482440 learned_addr learned my addr 172.21.15.195:0/3854482440 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:50:07.757 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.742+0000 7fa96f7fe700 1 -- 172.21.15.195:0/3854482440 >> v2:172.21.15.202:3300/0 conn(0x7fa97014e860 msgr2=0x7fa9700c3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:50:07.758 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.742+0000 7fa96effd700 1 --2- 172.21.15.195:0/3854482440 >> v2:172.21.15.202:3300/0 conn(0x7fa97014e860 0x7fa9700c3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:07.758 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.742+0000 7fa96f7fe700 1 --2- 172.21.15.195:0/3854482440 >> v2:172.21.15.202:3300/0 conn(0x7fa97014e860 0x7fa9700c3840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:07.759 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.742+0000 7fa96f7fe700 1 -- 172.21.15.195:0/3854482440 >> v1:172.21.15.202:6789/0 conn(0x7fa9700c3d80 legacy=0x7fa97006f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:50:07.759 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.742+0000 7fa96f7fe700 1 -- 172.21.15.195:0/3854482440 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9701542c0 con 0x7fa97014e490 2022-06-27T11:50:07.760 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.742+0000 7fa96f7fe700 1 --2- 172.21.15.195:0/3854482440 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa97014e490 0x7fa97006cfe0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa970065b80 tx=0x7fa964009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=518db1a894dfee75 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:50:07.760 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.742+0000 7fa96e7fc700 1 -- 172.21.15.195:0/3854482440 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa964004030 con 0x7fa97014e490 2022-06-27T11:50:07.760 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.742+0000 7fa96e7fc700 1 -- 172.21.15.195:0/3854482440 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa964004190 con 0x7fa97014e490 2022-06-27T11:50:07.760 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.742+0000 7fa96e7fc700 1 -- 172.21.15.195:0/3854482440 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa964011630 con 0x7fa97014e490 2022-06-27T11:50:07.761 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.743+0000 7fa975b18700 1 -- 172.21.15.195:0/3854482440 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa97014e490 msgr2=0x7fa97006cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:07.761 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.743+0000 7fa975b18700 1 --2- 172.21.15.195:0/3854482440 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa97014e490 0x7fa97006cfe0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa970065b80 tx=0x7fa964009120 comp rx=0 tx=0).stop 2022-06-27T11:50:07.761 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.743+0000 7fa975b18700 1 -- 172.21.15.195:0/3854482440 shutdown_connections 2022-06-27T11:50:07.761 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.743+0000 7fa975b18700 1 --2- 172.21.15.195:0/3854482440 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa97014e490 0x7fa97006cfe0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:07.762 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.743+0000 7fa975b18700 1 --2- 172.21.15.195:0/3854482440 >> v2:172.21.15.202:3300/0 conn(0x7fa97014e860 0x7fa9700c3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:07.762 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.743+0000 7fa975b18700 1 -- 172.21.15.195:0/3854482440 >> 172.21.15.195:0/3854482440 conn(0x7fa970063af0 msgr2=0x7fa970063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:50:07.768 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.743+0000 7fa975b18700 1 -- 172.21.15.195:0/3854482440 shutdown_connections 2022-06-27T11:50:07.768 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.743+0000 7fa975b18700 1 -- 172.21.15.195:0/3854482440 wait complete. 2022-06-27T11:50:07.769 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.744+0000 7fa975b18700 1 Processor -- start 2022-06-27T11:50:07.770 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.744+0000 7fa975b18700 1 -- start start 2022-06-27T11:50:07.770 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.745+0000 7fa975b18700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa9700c3d80 0x7fa970165030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:07.771 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.745+0000 7fa975b18700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa97014e490 0x7fa970157ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:07.773 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.745+0000 7fa975b18700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa97014e860 0x7fa9700be990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:07.774 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.745+0000 7fa975b18700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa970156820 con 0x7fa97014e490 2022-06-27T11:50:07.774 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.745+0000 7fa975b18700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa970156b00 con 0x7fa9700c3d80 2022-06-27T11:50:07.775 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.745+0000 7fa975b18700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa970156de0 con 0x7fa97014e860 2022-06-27T11:50:07.775 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.745+0000 7fa96effd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa97014e490 0x7fa970157ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:07.776 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.745+0000 7fa96effd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa97014e490 0x7fa970157ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:42672/0 (socket says 172.21.15.195:42672) 2022-06-27T11:50:07.778 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.745+0000 7fa96effd700 1 -- 172.21.15.195:0/3660346930 learned_addr learned my addr 172.21.15.195:0/3660346930 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:50:07.778 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.745+0000 7fa96f7fe700 1 --2- 172.21.15.195:0/3660346930 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa9700c3d80 0x7fa970165030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:07.779 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.745+0000 7fa96ffff700 1 --2- 172.21.15.195:0/3660346930 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa97014e860 0x7fa9700be990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:07.779 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.745+0000 7fa96effd700 1 -- 172.21.15.195:0/3660346930 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa97014e860 msgr2=0x7fa9700be990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:07.780 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.745+0000 7fa96effd700 1 --2- 172.21.15.195:0/3660346930 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa97014e860 0x7fa9700be990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:07.780 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.745+0000 7fa96effd700 1 -- 172.21.15.195:0/3660346930 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa9700c3d80 msgr2=0x7fa970165030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:07.781 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.745+0000 7fa96effd700 1 --2- 172.21.15.195:0/3660346930 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa9700c3d80 0x7fa970165030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:07.781 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.745+0000 7fa96effd700 1 -- 172.21.15.195:0/3660346930 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9640093a0 con 0x7fa97014e490 2022-06-27T11:50:07.782 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.745+0000 7fa96ffff700 1 --2- 172.21.15.195:0/3660346930 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa97014e860 0x7fa9700be990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T11:50:07.786 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.745+0000 7fa96effd700 1 --2- 172.21.15.195:0/3660346930 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa97014e490 0x7fa970157ba0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fa96000c880 tx=0x7fa960008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:50:07.786 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.746+0000 7fa9577fe700 1 -- 172.21.15.195:0/3660346930 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa960004020 con 0x7fa97014e490 2022-06-27T11:50:07.787 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.746+0000 7fa9577fe700 1 -- 172.21.15.195:0/3660346930 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa960004180 con 0x7fa97014e490 2022-06-27T11:50:07.787 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.746+0000 7fa9577fe700 1 -- 172.21.15.195:0/3660346930 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa960011600 con 0x7fa97014e490 2022-06-27T11:50:07.788 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.746+0000 7fa975b18700 1 -- 172.21.15.195:0/3660346930 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9701570c0 con 0x7fa97014e490 2022-06-27T11:50:07.788 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.746+0000 7fa975b18700 1 -- 172.21.15.195:0/3660346930 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa970157680 con 0x7fa97014e490 2022-06-27T11:50:07.789 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.747+0000 7fa975b18700 1 -- 172.21.15.195:0/3660346930 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9700722c0 con 0x7fa97014e490 2022-06-27T11:50:07.789 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.747+0000 7fa9577fe700 1 -- 172.21.15.195:0/3660346930 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fa960011a30 con 0x7fa97014e490 2022-06-27T11:50:07.790 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.747+0000 7fa9577fe700 1 --2- 172.21.15.195:0/3660346930 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa950063d30 0x7fa9500661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:07.790 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.748+0000 7fa9577fe700 1 -- 172.21.15.195:0/3660346930 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6202+0+0 (secure 0 0 0) 0x7fa9600808e0 con 0x7fa97014e490 2022-06-27T11:50:07.791 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.748+0000 7fa96f7fe700 1 --2- 172.21.15.195:0/3660346930 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa950063d30 0x7fa9500661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:07.791 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.748+0000 7fa96f7fe700 1 --2- 172.21.15.195:0/3660346930 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa950063d30 0x7fa9500661e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa964009160 tx=0x7fa964016000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:50:07.791 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.752+0000 7fa9577fe700 1 -- 172.21.15.195:0/3660346930 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fa96004f440 con 0x7fa97014e490 2022-06-27T11:50:07.979 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.979+0000 7fa975b18700 1 -- 172.21.15.195:0/3660346930 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa9700c60d0 con 0x7fa97014e490 2022-06-27T11:50:07.980 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.980+0000 7fa9577fe700 1 -- 172.21.15.195:0/3660346930 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+5367 (secure 0 0 0) 0x7fa960015090 con 0x7fa97014e490 2022-06-27T11:50:07.981 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:50:07.981 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":10,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:50:05.983216+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"b","rank":0,"incarnation":9,"state":"up:reconnect","state_seq":36,"addr":"172.21.15.202:6833/864126677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":864126677},{"type":"v1","addr":"172.21.15.202:6833","nonce":864126677}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:50:07.984 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.981+0000 7fa975b18700 1 -- 172.21.15.195:0/3660346930 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa950063d30 msgr2=0x7fa9500661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:07.984 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.981+0000 7fa975b18700 1 --2- 172.21.15.195:0/3660346930 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa950063d30 0x7fa9500661e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa964009160 tx=0x7fa964016000 comp rx=0 tx=0).stop 2022-06-27T11:50:07.985 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.982+0000 7fa975b18700 1 -- 172.21.15.195:0/3660346930 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa97014e490 msgr2=0x7fa970157ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:07.989 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.982+0000 7fa975b18700 1 --2- 172.21.15.195:0/3660346930 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa97014e490 0x7fa970157ba0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fa96000c880 tx=0x7fa960008040 comp rx=0 tx=0).stop 2022-06-27T11:50:07.990 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.982+0000 7fa975b18700 1 -- 172.21.15.195:0/3660346930 shutdown_connections 2022-06-27T11:50:07.990 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.982+0000 7fa975b18700 1 --2- 172.21.15.195:0/3660346930 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa950063d30 0x7fa9500661e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:07.990 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.982+0000 7fa975b18700 1 --2- 172.21.15.195:0/3660346930 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa97014e860 0x7fa9700be990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:07.991 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.982+0000 7fa975b18700 1 --2- 172.21.15.195:0/3660346930 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa9700c3d80 0x7fa970165030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:07.992 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.982+0000 7fa975b18700 1 --2- 172.21.15.195:0/3660346930 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa97014e490 0x7fa970157ba0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:07.992 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.982+0000 7fa975b18700 1 -- 172.21.15.195:0/3660346930 >> 172.21.15.195:0/3660346930 conn(0x7fa970063af0 msgr2=0x7fa9700c9980 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:50:07.992 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.982+0000 7fa975b18700 1 -- 172.21.15.195:0/3660346930 shutdown_connections 2022-06-27T11:50:07.993 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:07.982+0000 7fa975b18700 1 -- 172.21.15.195:0/3660346930 wait complete. 2022-06-27T11:50:07.993 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 10 2022-06-27T11:50:07.996 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T11:50:09.998 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:50:10.131 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.131+0000 7f7810c73700 1 Processor -- start 2022-06-27T11:50:10.131 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.131+0000 7f7810c73700 1 -- start start 2022-06-27T11:50:10.134 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.132+0000 7f7810c73700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f780c150500 0x7f780c063af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:10.135 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.132+0000 7f7810c73700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f780c1508d0 0x7f780c0bbf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:10.135 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.132+0000 7f7810c73700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f780c064030 con 0x7f780c06a450 2022-06-27T11:50:10.135 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.132+0000 7f7810c73700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f780c0c4160 con 0x7f780c150500 2022-06-27T11:50:10.135 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.132+0000 7f7810c73700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f780c0c4440 con 0x7f780c1508d0 2022-06-27T11:50:10.136 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.132+0000 7f780a59c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f780c150500 0x7f780c063af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:10.136 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.132+0000 7f7809d9b700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f780c06a450 0x7f780c0c48d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:48866/0 (socket says 172.21.15.195:48866) 2022-06-27T11:50:10.136 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.132+0000 7f7809d9b700 1 -- 172.21.15.195:0/1469994691 learned_addr learned my addr 172.21.15.195:0/1469994691 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:50:10.137 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.132+0000 7f780ad9d700 1 --2- 172.21.15.195:0/1469994691 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f780c1508d0 0x7f780c0bbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:10.137 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.133+0000 7f780a59c700 1 -- 172.21.15.195:0/1469994691 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f780c1508d0 msgr2=0x7f780c0bbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:50:10.137 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.133+0000 7f780a59c700 1 --2- 172.21.15.195:0/1469994691 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f780c1508d0 0x7f780c0bbf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:10.137 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.133+0000 7f780a59c700 1 -- 172.21.15.195:0/1469994691 >> v1:172.21.15.195:6789/0 conn(0x7f780c06a450 legacy=0x7f780c0c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:10.137 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.133+0000 7f780959a700 1 -- 172.21.15.195:0/1469994691 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3818997131 0 0) 0x7f77fc001110 con 0x7f780c06a450 2022-06-27T11:50:10.138 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.133+0000 7f780a59c700 1 -- 172.21.15.195:0/1469994691 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f780c15a800 con 0x7f780c150500 2022-06-27T11:50:10.138 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.133+0000 7f780a59c700 1 --2- 172.21.15.195:0/1469994691 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f780c150500 0x7f780c063af0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f7800009720 tx=0x7f7800009ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=407de601911889af server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:50:10.138 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.133+0000 7f780959a700 1 -- 172.21.15.195:0/1469994691 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7800005ad0 con 0x7f780c150500 2022-06-27T11:50:10.139 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.133+0000 7f780959a700 1 -- 172.21.15.195:0/1469994691 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7800005c30 con 0x7f780c150500 2022-06-27T11:50:10.139 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.133+0000 7f780959a700 1 -- 172.21.15.195:0/1469994691 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7800006030 con 0x7f780c150500 2022-06-27T11:50:10.139 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.133+0000 7f7810c73700 1 -- 172.21.15.195:0/1469994691 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f780c150500 msgr2=0x7f780c063af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:10.140 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.133+0000 7f7810c73700 1 --2- 172.21.15.195:0/1469994691 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f780c150500 0x7f780c063af0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f7800009720 tx=0x7f7800009ae0 comp rx=0 tx=0).stop 2022-06-27T11:50:10.140 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.133+0000 7f7810c73700 1 -- 172.21.15.195:0/1469994691 shutdown_connections 2022-06-27T11:50:10.140 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.133+0000 7f7810c73700 1 --2- 172.21.15.195:0/1469994691 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f780c1508d0 0x7f780c0bbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:10.140 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.133+0000 7f7810c73700 1 --2- 172.21.15.195:0/1469994691 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f780c150500 0x7f780c063af0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:10.140 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.133+0000 7f7810c73700 1 -- 172.21.15.195:0/1469994691 >> 172.21.15.195:0/1469994691 conn(0x7f780c0c9b00 msgr2=0x7f780c0c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:50:10.141 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.134+0000 7f7810c73700 1 -- 172.21.15.195:0/1469994691 shutdown_connections 2022-06-27T11:50:10.141 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.134+0000 7f7810c73700 1 -- 172.21.15.195:0/1469994691 wait complete. 2022-06-27T11:50:10.141 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.135+0000 7f7810c73700 1 Processor -- start 2022-06-27T11:50:10.141 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.135+0000 7f7810c73700 1 -- start start 2022-06-27T11:50:10.145 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.136+0000 7f7810c73700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f780c06a450 0x7f780c0c5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:10.146 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.136+0000 7f7810c73700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f780c150500 0x7f780c16a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:10.146 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.136+0000 7f7810c73700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f780c1508d0 0x7f780c166be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:10.146 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.136+0000 7f7810c73700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f780c169fb0 con 0x7f780c1508d0 2022-06-27T11:50:10.147 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.136+0000 7f7810c73700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f780c1667b0 con 0x7f780c150500 2022-06-27T11:50:10.147 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.136+0000 7f7810c73700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f780c06ecf0 con 0x7f780c06a450 2022-06-27T11:50:10.147 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.136+0000 7f780ad9d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f780c1508d0 0x7f780c166be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:10.148 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.136+0000 7f780ad9d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f780c1508d0 0x7f780c166be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:42688/0 (socket says 172.21.15.195:42688) 2022-06-27T11:50:10.148 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.136+0000 7f780ad9d700 1 -- 172.21.15.195:0/2829514605 learned_addr learned my addr 172.21.15.195:0/2829514605 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:50:10.148 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.136+0000 7f7809d9b700 1 --2- 172.21.15.195:0/2829514605 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f780c150500 0x7f780c16a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:10.149 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.136+0000 7f780a59c700 1 --2- 172.21.15.195:0/2829514605 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f780c06a450 0x7f780c0c5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:10.149 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.136+0000 7f780ad9d700 1 -- 172.21.15.195:0/2829514605 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f780c06a450 msgr2=0x7f780c0c5250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:10.149 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.136+0000 7f780ad9d700 1 --2- 172.21.15.195:0/2829514605 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f780c06a450 0x7f780c0c5250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:10.149 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.136+0000 7f780ad9d700 1 -- 172.21.15.195:0/2829514605 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f780c150500 msgr2=0x7f780c16a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:10.150 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.136+0000 7f780ad9d700 1 --2- 172.21.15.195:0/2829514605 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f780c150500 0x7f780c16a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:10.150 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.136+0000 7f780ad9d700 1 -- 172.21.15.195:0/2829514605 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7800009290 con 0x7f780c1508d0 2022-06-27T11:50:10.150 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.136+0000 7f780a59c700 1 --2- 172.21.15.195:0/2829514605 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f780c06a450 0x7f780c0c5250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T11:50:10.150 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.136+0000 7f7809d9b700 1 --2- 172.21.15.195:0/2829514605 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f780c150500 0x7f780c16a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T11:50:10.150 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.137+0000 7f780ad9d700 1 --2- 172.21.15.195:0/2829514605 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f780c1508d0 0x7f780c166be0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f7804007f20 tx=0x7f7804009860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:50:10.151 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.137+0000 7f77fa7fc700 1 -- 172.21.15.195:0/2829514605 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f780400eca0 con 0x7f780c1508d0 2022-06-27T11:50:10.151 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.137+0000 7f77fa7fc700 1 -- 172.21.15.195:0/2829514605 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f780400ee20 con 0x7f780c1508d0 2022-06-27T11:50:10.151 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.137+0000 7f7810c73700 1 -- 172.21.15.195:0/2829514605 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f780c154310 con 0x7f780c1508d0 2022-06-27T11:50:10.151 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.137+0000 7f77fa7fc700 1 -- 172.21.15.195:0/2829514605 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7804014670 con 0x7f780c1508d0 2022-06-27T11:50:10.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.137+0000 7f7810c73700 1 -- 172.21.15.195:0/2829514605 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f780c1548d0 con 0x7f780c1508d0 2022-06-27T11:50:10.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.138+0000 7f7810c73700 1 -- 172.21.15.195:0/2829514605 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f780c169d80 con 0x7f780c1508d0 2022-06-27T11:50:10.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.138+0000 7f77fa7fc700 1 -- 172.21.15.195:0/2829514605 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f7804012050 con 0x7f780c1508d0 2022-06-27T11:50:10.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.138+0000 7f77fa7fc700 1 --2- 172.21.15.195:0/2829514605 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f77e8063d00 0x7f77e80661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:10.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.139+0000 7f77fa7fc700 1 -- 172.21.15.195:0/2829514605 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6202+0+0 (secure 0 0 0) 0x7f7804080fc0 con 0x7f780c1508d0 2022-06-27T11:50:10.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.139+0000 7f780a59c700 1 --2- 172.21.15.195:0/2829514605 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f77e8063d00 0x7f77e80661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:10.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.139+0000 7f780a59c700 1 --2- 172.21.15.195:0/2829514605 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f77e8063d00 0x7f77e80661b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7800009b20 tx=0x7f7800013000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:50:10.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.143+0000 7f77fa7fc700 1 -- 172.21.15.195:0/2829514605 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f780c169d80 con 0x7f780c1508d0 2022-06-27T11:50:10.366 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.367+0000 7f7810c73700 1 -- 172.21.15.195:0/2829514605 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f780c169d80 con 0x7f780c1508d0 2022-06-27T11:50:10.367 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.367+0000 7f77fa7fc700 1 -- 172.21.15.195:0/2829514605 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+5367 (secure 0 0 0) 0x7f780c169d80 con 0x7f780c1508d0 2022-06-27T11:50:10.367 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:50:10.368 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":10,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:50:05.983216+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"b","rank":0,"incarnation":9,"state":"up:reconnect","state_seq":36,"addr":"172.21.15.202:6833/864126677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":864126677},{"type":"v1","addr":"172.21.15.202:6833","nonce":864126677}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:50:10.369 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.369+0000 7f7810c73700 1 -- 172.21.15.195:0/2829514605 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f77e8063d00 msgr2=0x7f77e80661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:10.369 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.369+0000 7f7810c73700 1 --2- 172.21.15.195:0/2829514605 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f77e8063d00 0x7f77e80661b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7800009b20 tx=0x7f7800013000 comp rx=0 tx=0).stop 2022-06-27T11:50:10.369 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.369+0000 7f7810c73700 1 -- 172.21.15.195:0/2829514605 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f780c1508d0 msgr2=0x7f780c166be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:10.369 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.369+0000 7f7810c73700 1 --2- 172.21.15.195:0/2829514605 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f780c1508d0 0x7f780c166be0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f7804007f20 tx=0x7f7804009860 comp rx=0 tx=0).stop 2022-06-27T11:50:10.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.369+0000 7f7810c73700 1 -- 172.21.15.195:0/2829514605 shutdown_connections 2022-06-27T11:50:10.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.369+0000 7f7810c73700 1 --2- 172.21.15.195:0/2829514605 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f77e8063d00 0x7f77e80661b0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:10.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.369+0000 7f7810c73700 1 --2- 172.21.15.195:0/2829514605 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f780c1508d0 0x7f780c166be0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:10.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.369+0000 7f7810c73700 1 --2- 172.21.15.195:0/2829514605 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f780c06a450 0x7f780c0c5250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:10.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.369+0000 7f7810c73700 1 --2- 172.21.15.195:0/2829514605 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f780c150500 0x7f780c16a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:10.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.369+0000 7f7810c73700 1 -- 172.21.15.195:0/2829514605 >> 172.21.15.195:0/2829514605 conn(0x7f780c0c9b00 msgr2=0x7f780c067ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:50:10.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.369+0000 7f7810c73700 1 -- 172.21.15.195:0/2829514605 shutdown_connections 2022-06-27T11:50:10.374 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:10.369+0000 7f7810c73700 1 -- 172.21.15.195:0/2829514605 wait complete. 2022-06-27T11:50:10.374 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 10 2022-06-27T11:50:10.381 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T11:50:12.382 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:50:12.538 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.519+0000 7fa266e75700 1 Processor -- start 2022-06-27T11:50:12.539 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.519+0000 7fa266e75700 1 -- start start 2022-06-27T11:50:12.539 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.520+0000 7fa266e75700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fa26006f640 0x7fa260156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:12.540 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.520+0000 7fa266e75700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2600c1a20 con 0x7fa2600c4160 2022-06-27T11:50:12.540 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.520+0000 7fa266e75700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2600c1130 con 0x7fa2600c39e0 2022-06-27T11:50:12.541 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.520+0000 7fa266e75700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa26006ea90 con 0x7fa26006f640 2022-06-27T11:50:12.541 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.520+0000 7fa25ffff700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fa2600c4160 0x7fa2600c1eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:48884/0 (socket says 172.21.15.195:48884) 2022-06-27T11:50:12.542 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.520+0000 7fa25ffff700 1 -- 172.21.15.195:0/3402118413 learned_addr learned my addr 172.21.15.195:0/3402118413 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:50:12.542 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.520+0000 7fa264c11700 1 --2- 172.21.15.195:0/3402118413 >> v2:172.21.15.202:3300/0 conn(0x7fa26006f640 0x7fa260156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:12.543 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.520+0000 7fa25f7fe700 1 -- 172.21.15.195:0/3402118413 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3847981427 0 0) 0x7fa2600c1a20 con 0x7fa2600c4160 2022-06-27T11:50:12.543 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.520+0000 7fa25f7fe700 1 -- 172.21.15.195:0/3402118413 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa244003430 con 0x7fa2600c4160 2022-06-27T11:50:12.544 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.521+0000 7fa25f7fe700 1 -- 172.21.15.195:0/3402118413 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 150808867 0 0) 0x7fa2600c1130 con 0x7fa2600c39e0 2022-06-27T11:50:12.545 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.521+0000 7fa25f7fe700 1 -- 172.21.15.195:0/3402118413 --> v1:172.21.15.202:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2600c1a20 con 0x7fa2600c39e0 2022-06-27T11:50:12.546 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.521+0000 7fa25f7fe700 1 -- 172.21.15.195:0/3402118413 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4098304391 0 0) 0x7fa244003430 con 0x7fa2600c4160 2022-06-27T11:50:12.548 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.521+0000 7fa25f7fe700 1 -- 172.21.15.195:0/3402118413 >> v2:172.21.15.202:3300/0 conn(0x7fa26006f640 msgr2=0x7fa260156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:50:12.548 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.521+0000 7fa25f7fe700 1 --2- 172.21.15.195:0/3402118413 >> v2:172.21.15.202:3300/0 conn(0x7fa26006f640 0x7fa260156ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:12.549 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.521+0000 7fa25f7fe700 1 -- 172.21.15.195:0/3402118413 >> v1:172.21.15.202:6789/0 conn(0x7fa2600c39e0 legacy=0x7fa2601535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:12.549 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.521+0000 7fa25f7fe700 1 -- 172.21.15.195:0/3402118413 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa26006ed70 con 0x7fa2600c4160 2022-06-27T11:50:12.549 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.521+0000 7fa25f7fe700 1 -- 172.21.15.195:0/3402118413 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7fa24c001470 con 0x7fa2600c4160 2022-06-27T11:50:12.550 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.521+0000 7fa25f7fe700 1 -- 172.21.15.195:0/3402118413 >> v1:172.21.15.195:6789/0 conn(0x7fa2600c4160 legacy=0x7fa2600c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:12.550 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.521+0000 7fa25f7fe700 1 --2- 172.21.15.195:0/3402118413 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa244003b00 0x7fa244003ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:12.550 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.521+0000 7fa25f7fe700 1 --2- 172.21.15.195:0/3402118413 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa2440046d0 0x7fa244004b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:12.550 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.521+0000 7fa25f7fe700 1 --2- 172.21.15.195:0/3402118413 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa2440050d0 0x7fa24400b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:12.551 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.521+0000 7fa25f7fe700 1 -- 172.21.15.195:0/3402118413 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa26006ea90 con 0x7fa244003b00 2022-06-27T11:50:12.551 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.521+0000 7fa25f7fe700 1 -- 172.21.15.195:0/3402118413 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa24400bff0 con 0x7fa2440046d0 2022-06-27T11:50:12.551 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.521+0000 7fa25f7fe700 1 -- 172.21.15.195:0/3402118413 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa24400c180 con 0x7fa2440050d0 2022-06-27T11:50:12.551 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.521+0000 7fa264c11700 1 --2- 172.21.15.195:0/3402118413 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa244003b00 0x7fa244003ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:12.552 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.521+0000 7fa25ffff700 1 --2- 172.21.15.195:0/3402118413 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa2440046d0 0x7fa244004b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:12.552 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.521+0000 7fa265412700 1 --2- 172.21.15.195:0/3402118413 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa2440050d0 0x7fa24400b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:12.552 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.521+0000 7fa264c11700 1 -- 172.21.15.195:0/3402118413 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa2440050d0 msgr2=0x7fa24400b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:50:12.552 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.521+0000 7fa264c11700 1 --2- 172.21.15.195:0/3402118413 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa2440050d0 0x7fa24400b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:12.553 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.521+0000 7fa264c11700 1 -- 172.21.15.195:0/3402118413 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa2440046d0 msgr2=0x7fa244004b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:50:12.553 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.521+0000 7fa264c11700 1 --2- 172.21.15.195:0/3402118413 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa2440046d0 0x7fa244004b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:12.553 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.521+0000 7fa264c11700 1 -- 172.21.15.195:0/3402118413 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa244003430 con 0x7fa244003b00 2022-06-27T11:50:12.553 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.521+0000 7fa265412700 1 -- 172.21.15.195:0/3402118413 reap_dead start 2022-06-27T11:50:12.553 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.521+0000 7fa264c11700 1 --2- 172.21.15.195:0/3402118413 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa244003b00 0x7fa244003ef0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fa254009af0 tx=0x7fa254009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b5638f08610eb4e server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:50:12.554 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.521+0000 7fa25f7fe700 1 -- 172.21.15.195:0/3402118413 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa25400e640 con 0x7fa244003b00 2022-06-27T11:50:12.554 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.521+0000 7fa25f7fe700 1 -- 172.21.15.195:0/3402118413 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa25400e7a0 con 0x7fa244003b00 2022-06-27T11:50:12.554 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.521+0000 7fa25f7fe700 1 -- 172.21.15.195:0/3402118413 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa25400eb40 con 0x7fa244003b00 2022-06-27T11:50:12.554 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.522+0000 7fa266e75700 1 -- 172.21.15.195:0/3402118413 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa244003b00 msgr2=0x7fa244003ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:12.555 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.522+0000 7fa266e75700 1 --2- 172.21.15.195:0/3402118413 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa244003b00 0x7fa244003ef0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fa254009af0 tx=0x7fa254009060 comp rx=0 tx=0).stop 2022-06-27T11:50:12.555 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.522+0000 7fa266e75700 1 -- 172.21.15.195:0/3402118413 shutdown_connections 2022-06-27T11:50:12.555 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.522+0000 7fa266e75700 1 --2- 172.21.15.195:0/3402118413 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa244003b00 0x7fa244003ef0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:12.555 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.522+0000 7fa266e75700 1 -- 172.21.15.195:0/3402118413 >> 172.21.15.195:0/3402118413 conn(0x7fa2600c6350 msgr2=0x7fa2600c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:50:12.556 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.522+0000 7fa266e75700 1 -- 172.21.15.195:0/3402118413 shutdown_connections 2022-06-27T11:50:12.556 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.522+0000 7fa266e75700 1 -- 172.21.15.195:0/3402118413 wait complete. 2022-06-27T11:50:12.558 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.523+0000 7fa266e75700 1 Processor -- start 2022-06-27T11:50:12.558 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.523+0000 7fa266e75700 1 -- start start 2022-06-27T11:50:12.559 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.524+0000 7fa266e75700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa244003b00 0x7fa26015b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:12.559 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.524+0000 7fa266e75700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa26014f9c0 0x7fa26016a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:12.559 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.524+0000 7fa266e75700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa26014fe10 0x7fa260166de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:12.559 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.524+0000 7fa266e75700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2600bea80 con 0x7fa244003b00 2022-06-27T11:50:12.559 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.524+0000 7fa266e75700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2600c0250 con 0x7fa26014fe10 2022-06-27T11:50:12.560 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.524+0000 7fa266e75700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa260169fd0 con 0x7fa26014f9c0 2022-06-27T11:50:12.560 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.524+0000 7fa264c11700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa244003b00 0x7fa26015b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:12.560 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.524+0000 7fa264c11700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa244003b00 0x7fa26015b200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:42706/0 (socket says 172.21.15.195:42706) 2022-06-27T11:50:12.560 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.524+0000 7fa264c11700 1 -- 172.21.15.195:0/668474866 learned_addr learned my addr 172.21.15.195:0/668474866 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:50:12.561 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.524+0000 7fa25ffff700 1 --2- 172.21.15.195:0/668474866 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa26014f9c0 0x7fa26016a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:12.561 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.524+0000 7fa265412700 1 --2- 172.21.15.195:0/668474866 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa26014fe10 0x7fa260166de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:12.561 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.524+0000 7fa264c11700 1 -- 172.21.15.195:0/668474866 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa26014f9c0 msgr2=0x7fa26016a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:12.561 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.524+0000 7fa264c11700 1 --2- 172.21.15.195:0/668474866 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa26014f9c0 0x7fa26016a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:12.562 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.524+0000 7fa264c11700 1 -- 172.21.15.195:0/668474866 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa26014fe10 msgr2=0x7fa260166de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:12.562 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.524+0000 7fa264c11700 1 --2- 172.21.15.195:0/668474866 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa26014fe10 0x7fa260166de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:12.562 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.524+0000 7fa264c11700 1 -- 172.21.15.195:0/668474866 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2601668c0 con 0x7fa244003b00 2022-06-27T11:50:12.563 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.525+0000 7fa264c11700 1 --2- 172.21.15.195:0/668474866 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa244003b00 0x7fa26015b200 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa25400e4b0 tx=0x7fa25401a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:50:12.563 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.525+0000 7fa25cff9700 1 -- 172.21.15.195:0/668474866 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa254021070 con 0x7fa244003b00 2022-06-27T11:50:12.563 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.525+0000 7fa25cff9700 1 -- 172.21.15.195:0/668474866 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa254009e60 con 0x7fa244003b00 2022-06-27T11:50:12.563 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.525+0000 7fa266e75700 1 -- 172.21.15.195:0/668474866 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2600bf6d0 con 0x7fa244003b00 2022-06-27T11:50:12.563 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.525+0000 7fa25cff9700 1 -- 172.21.15.195:0/668474866 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa254003960 con 0x7fa244003b00 2022-06-27T11:50:12.564 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.525+0000 7fa266e75700 1 -- 172.21.15.195:0/668474866 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2600bfc90 con 0x7fa244003b00 2022-06-27T11:50:12.564 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.526+0000 7fa266e75700 1 -- 172.21.15.195:0/668474866 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2600722c0 con 0x7fa244003b00 2022-06-27T11:50:12.564 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.526+0000 7fa25cff9700 1 -- 172.21.15.195:0/668474866 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fa25402d030 con 0x7fa244003b00 2022-06-27T11:50:12.564 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.526+0000 7fa25cff9700 1 --2- 172.21.15.195:0/668474866 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa248063d30 0x7fa2480661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:12.565 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.527+0000 7fa25ffff700 1 --2- 172.21.15.195:0/668474866 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa248063d30 0x7fa2480661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:12.565 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.527+0000 7fa25cff9700 1 -- 172.21.15.195:0/668474866 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6202+0+0 (secure 0 0 0) 0x7fa254081fe0 con 0x7fa244003b00 2022-06-27T11:50:12.565 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.527+0000 7fa25ffff700 1 --2- 172.21.15.195:0/668474866 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa248063d30 0x7fa2480661e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa2600bd4d0 tx=0x7fa24c001d20 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:50:12.565 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.531+0000 7fa25cff9700 1 -- 172.21.15.195:0/668474866 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fa254050b40 con 0x7fa244003b00 2022-06-27T11:50:12.767 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.757+0000 7fa266e75700 1 -- 172.21.15.195:0/668474866 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa26006f240 con 0x7fa244003b00 2022-06-27T11:50:12.768 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.758+0000 7fa25cff9700 1 -- 172.21.15.195:0/668474866 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+5364 (secure 0 0 0) 0x7fa25402b020 con 0x7fa244003b00 2022-06-27T11:50:12.768 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:50:12.769 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":11,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":11,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:50:12.001975+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"b","rank":0,"incarnation":9,"state":"up:rejoin","state_seq":38,"addr":"172.21.15.202:6833/864126677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":864126677},{"type":"v1","addr":"172.21.15.202:6833","nonce":864126677}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:50:12.770 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.760+0000 7fa266e75700 1 -- 172.21.15.195:0/668474866 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa248063d30 msgr2=0x7fa2480661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:12.770 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.760+0000 7fa266e75700 1 --2- 172.21.15.195:0/668474866 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa248063d30 0x7fa2480661e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa2600bd4d0 tx=0x7fa24c001d20 comp rx=0 tx=0).stop 2022-06-27T11:50:12.771 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.760+0000 7fa266e75700 1 -- 172.21.15.195:0/668474866 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa244003b00 msgr2=0x7fa26015b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:12.771 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.760+0000 7fa266e75700 1 --2- 172.21.15.195:0/668474866 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa244003b00 0x7fa26015b200 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa25400e4b0 tx=0x7fa25401a000 comp rx=0 tx=0).stop 2022-06-27T11:50:12.771 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.760+0000 7fa266e75700 1 -- 172.21.15.195:0/668474866 shutdown_connections 2022-06-27T11:50:12.771 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.760+0000 7fa266e75700 1 --2- 172.21.15.195:0/668474866 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa248063d30 0x7fa2480661e0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:12.771 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.760+0000 7fa266e75700 1 --2- 172.21.15.195:0/668474866 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa244003b00 0x7fa26015b200 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:12.772 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.760+0000 7fa266e75700 1 --2- 172.21.15.195:0/668474866 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa26014fe10 0x7fa260166de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:12.772 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.760+0000 7fa266e75700 1 --2- 172.21.15.195:0/668474866 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa26014f9c0 0x7fa26016a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:12.772 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.760+0000 7fa266e75700 1 -- 172.21.15.195:0/668474866 >> 172.21.15.195:0/668474866 conn(0x7fa2600c6350 msgr2=0x7fa260150720 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:50:12.772 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.760+0000 7fa266e75700 1 -- 172.21.15.195:0/668474866 shutdown_connections 2022-06-27T11:50:12.773 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:12.760+0000 7fa266e75700 1 -- 172.21.15.195:0/668474866 wait complete. 2022-06-27T11:50:12.773 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 11 2022-06-27T11:50:12.776 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 11, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-06-27T11:47:52.358095+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 11, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4287": { "addr": "172.21.15.202:6833/864126677", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6832", "nonce": 864126677, "type": "v2" }, { "addr": "172.21.15.202:6833", "nonce": 864126677, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4287, "incarnation": 9, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:rejoin", "state_seq": 38 } }, "last_failure": 0, "last_failure_osd_epoch": 26, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-06-27T11:50:12.001975+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4287 } } } ], "standbys": [ { "addr": "172.21.15.202:6837/3305006841", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6836", "nonce": 3305006841, "type": "v2" }, { "addr": "172.21.15.202:6837", "nonce": 3305006841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6835/2060709822", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6834", "nonce": 2060709822, "type": "v2" }, { "addr": "172.21.15.195:6835", "nonce": 2060709822, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6838/1598751652", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6836", "nonce": 1598751652, "type": "v2" }, { "addr": "172.21.15.195:6838", "nonce": 1598751652, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4275, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6839/1313800661", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6837", "nonce": 1313800661, "type": "v2" }, { "addr": "172.21.15.195:6839", "nonce": 1313800661, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4278, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-06-27T11:50:14.778 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:50:14.948 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:14.914+0000 7f03f1281700 1 Processor -- start 2022-06-27T11:50:14.948 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:14.914+0000 7f03f1281700 1 -- start start 2022-06-27T11:50:14.948 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:14.915+0000 7f03f1281700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f03ec0c40d0 0x7f03ec157010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:14.949 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:14.915+0000 7f03f1281700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f03ec067ed0 0x7f03ec14cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:14.949 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:14.915+0000 7f03f1281700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03ec156b80 con 0x7f03ec0c6190 2022-06-27T11:50:14.949 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:14.915+0000 7f03f1281700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03ec0682f0 con 0x7f03ec0c40d0 2022-06-27T11:50:14.949 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:14.915+0000 7f03f1281700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03ec06a040 con 0x7f03ec067ed0 2022-06-27T11:50:14.950 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:14.915+0000 7f03eaffd700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f03ec0c40d0 0x7f03ec157010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:14.950 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:14.915+0000 7f03ea7fc700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f03ec0c6190 0x7f03ec0c44a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:48904/0 (socket says 172.21.15.195:48904) 2022-06-27T11:50:14.950 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:14.915+0000 7f03ea7fc700 1 -- 172.21.15.195:0/390028716 learned_addr learned my addr 172.21.15.195:0/390028716 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:50:14.950 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:14.915+0000 7f03eb7fe700 1 --2- 172.21.15.195:0/390028716 >> v2:172.21.15.202:3300/0 conn(0x7f03ec067ed0 0x7f03ec14cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:14.950 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:14.916+0000 7f03eaffd700 1 -- 172.21.15.195:0/390028716 >> v2:172.21.15.202:3300/0 conn(0x7f03ec067ed0 msgr2=0x7f03ec14cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:50:14.951 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:14.916+0000 7f03eaffd700 1 --2- 172.21.15.195:0/390028716 >> v2:172.21.15.202:3300/0 conn(0x7f03ec067ed0 0x7f03ec14cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:14.951 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:14.916+0000 7f03eaffd700 1 -- 172.21.15.195:0/390028716 >> v1:172.21.15.195:6789/0 conn(0x7f03ec0c6190 legacy=0x7f03ec0c44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:14.951 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:14.916+0000 7f03e9ffb700 1 -- 172.21.15.195:0/390028716 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1188709493 0 0) 0x7f03ec156b80 con 0x7f03ec0c6190 2022-06-27T11:50:14.951 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:14.916+0000 7f03eaffd700 1 -- 172.21.15.195:0/390028716 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03ec06a300 con 0x7f03ec0c40d0 2022-06-27T11:50:14.952 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:14.916+0000 7f03eaffd700 1 --2- 172.21.15.195:0/390028716 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f03ec0c40d0 0x7f03ec157010 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f03ec08afa0 tx=0x7f03e0009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24e0e824bfce712d server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:50:14.952 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:14.916+0000 7f03e9ffb700 1 -- 172.21.15.195:0/390028716 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e0011630 con 0x7f03ec0c40d0 2022-06-27T11:50:14.952 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:14.916+0000 7f03e9ffb700 1 -- 172.21.15.195:0/390028716 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03e0011790 con 0x7f03ec0c40d0 2022-06-27T11:50:14.952 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:14.916+0000 7f03e9ffb700 1 -- 172.21.15.195:0/390028716 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e0011b30 con 0x7f03ec0c40d0 2022-06-27T11:50:14.953 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:14.916+0000 7f03f1281700 1 -- 172.21.15.195:0/390028716 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f03ec0c40d0 msgr2=0x7f03ec157010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:14.953 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:14.916+0000 7f03f1281700 1 --2- 172.21.15.195:0/390028716 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f03ec0c40d0 0x7f03ec157010 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f03ec08afa0 tx=0x7f03e0009060 comp rx=0 tx=0).stop 2022-06-27T11:50:14.953 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:14.916+0000 7f03f1281700 1 -- 172.21.15.195:0/390028716 shutdown_connections 2022-06-27T11:50:14.953 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:14.916+0000 7f03f1281700 1 --2- 172.21.15.195:0/390028716 >> v2:172.21.15.202:3300/0 conn(0x7f03ec067ed0 0x7f03ec14cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:14.953 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:14.916+0000 7f03f1281700 1 --2- 172.21.15.195:0/390028716 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f03ec0c40d0 0x7f03ec157010 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:14.954 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:14.916+0000 7f03f1281700 1 -- 172.21.15.195:0/390028716 >> 172.21.15.195:0/390028716 conn(0x7f03ec0ccb80 msgr2=0x7f03ec0ccf80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:50:14.954 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:14.916+0000 7f03f1281700 1 -- 172.21.15.195:0/390028716 shutdown_connections 2022-06-27T11:50:14.954 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:14.916+0000 7f03f1281700 1 -- 172.21.15.195:0/390028716 wait complete. 2022-06-27T11:50:14.954 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:14.918+0000 7f03f1281700 1 Processor -- start 2022-06-27T11:50:14.955 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:14.918+0000 7f03f1281700 1 -- start start 2022-06-27T11:50:14.955 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:14.918+0000 7f03f1281700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f03ec067ed0 0x7f03ec0c06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:14.955 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:14.918+0000 7f03f1281700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f03ec0c40d0 0x7f03ec15d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:14.955 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:14.918+0000 7f03f1281700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f03ec0c6190 0x7f03ec157690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:14.956 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:14.918+0000 7f03f1281700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03ec169ca0 con 0x7f03ec0c40d0 2022-06-27T11:50:14.956 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:14.918+0000 7f03f1281700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03ec0c0d60 con 0x7f03ec0c6190 2022-06-27T11:50:14.956 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:14.919+0000 7f03f1281700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03ec169ef0 con 0x7f03ec067ed0 2022-06-27T11:50:14.956 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:14.919+0000 7f03ea7fc700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f03ec0c40d0 0x7f03ec15d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:14.957 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:14.919+0000 7f03ea7fc700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f03ec0c40d0 0x7f03ec15d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:42722/0 (socket says 172.21.15.195:42722) 2022-06-27T11:50:14.957 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:14.919+0000 7f03ea7fc700 1 -- 172.21.15.195:0/3904226663 learned_addr learned my addr 172.21.15.195:0/3904226663 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:50:14.957 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:14.919+0000 7f03eb7fe700 1 --2- 172.21.15.195:0/3904226663 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f03ec0c6190 0x7f03ec157690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:14.957 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:14.919+0000 7f03eaffd700 1 --2- 172.21.15.195:0/3904226663 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f03ec067ed0 0x7f03ec0c06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:14.958 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:14.919+0000 7f03ea7fc700 1 -- 172.21.15.195:0/3904226663 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f03ec067ed0 msgr2=0x7f03ec0c06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:14.958 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:14.919+0000 7f03ea7fc700 1 --2- 172.21.15.195:0/3904226663 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f03ec067ed0 0x7f03ec0c06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:14.958 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:14.919+0000 7f03ea7fc700 1 -- 172.21.15.195:0/3904226663 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f03ec0c6190 msgr2=0x7f03ec157690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:14.958 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:14.919+0000 7f03ea7fc700 1 --2- 172.21.15.195:0/3904226663 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f03ec0c6190 0x7f03ec157690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:14.959 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:14.919+0000 7f03ea7fc700 1 -- 172.21.15.195:0/3904226663 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03e00092e0 con 0x7f03ec0c40d0 2022-06-27T11:50:14.959 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:14.919+0000 7f03eaffd700 1 --2- 172.21.15.195:0/3904226663 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f03ec067ed0 0x7f03ec0c06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T11:50:14.959 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:14.919+0000 7f03eb7fe700 1 --2- 172.21.15.195:0/3904226663 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f03ec0c6190 0x7f03ec157690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T11:50:14.959 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:14.919+0000 7f03ea7fc700 1 --2- 172.21.15.195:0/3904226663 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f03ec0c40d0 0x7f03ec15d570 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f03dc00c490 tx=0x7f03dc007dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:50:14.960 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:14.919+0000 7f03d2ffd700 1 -- 172.21.15.195:0/3904226663 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03dc007430 con 0x7f03ec0c40d0 2022-06-27T11:50:14.960 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:14.919+0000 7f03d2ffd700 1 -- 172.21.15.195:0/3904226663 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03dc007590 con 0x7f03ec0c40d0 2022-06-27T11:50:14.960 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:14.919+0000 7f03f1281700 1 -- 172.21.15.195:0/3904226663 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03ec16a1d0 con 0x7f03ec0c40d0 2022-06-27T11:50:14.961 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:14.919+0000 7f03d2ffd700 1 -- 172.21.15.195:0/3904226663 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03dc004920 con 0x7f03ec0c40d0 2022-06-27T11:50:14.961 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:14.919+0000 7f03f1281700 1 -- 172.21.15.195:0/3904226663 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03ec16a790 con 0x7f03ec0c40d0 2022-06-27T11:50:14.962 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:14.920+0000 7f03f1281700 1 -- 172.21.15.195:0/3904226663 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03ec0722c0 con 0x7f03ec0c40d0 2022-06-27T11:50:14.962 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:14.921+0000 7f03d2ffd700 1 -- 172.21.15.195:0/3904226663 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f03dc010030 con 0x7f03ec0c40d0 2022-06-27T11:50:14.962 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:14.921+0000 7f03d2ffd700 1 --2- 172.21.15.195:0/3904226663 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f03cc063d00 0x7f03cc0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:14.962 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:14.921+0000 7f03d2ffd700 1 -- 172.21.15.195:0/3904226663 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6202+0+0 (secure 0 0 0) 0x7f03dc080510 con 0x7f03ec0c40d0 2022-06-27T11:50:14.963 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:14.921+0000 7f03eaffd700 1 --2- 172.21.15.195:0/3904226663 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f03cc063d00 0x7f03cc0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:14.963 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:14.921+0000 7f03eaffd700 1 --2- 172.21.15.195:0/3904226663 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f03cc063d00 0x7f03cc0661b0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f03e0009130 tx=0x7f03e0017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:50:14.963 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:14.925+0000 7f03d2ffd700 1 -- 172.21.15.195:0/3904226663 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f03dc04f090 con 0x7f03ec0c40d0 2022-06-27T11:50:15.147 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:15.147+0000 7f03f1281700 1 -- 172.21.15.195:0/3904226663 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f03ec1544d0 con 0x7f03ec0c40d0 2022-06-27T11:50:15.148 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:15.148+0000 7f03d2ffd700 1 -- 172.21.15.195:0/3904226663 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+5364 (secure 0 0 0) 0x7f03dc0526b0 con 0x7f03ec0c40d0 2022-06-27T11:50:15.149 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:50:15.150 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":11,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":11,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:50:12.001975+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"b","rank":0,"incarnation":9,"state":"up:rejoin","state_seq":38,"addr":"172.21.15.202:6833/864126677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":864126677},{"type":"v1","addr":"172.21.15.202:6833","nonce":864126677}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:50:15.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:15.150+0000 7f03f1281700 1 -- 172.21.15.195:0/3904226663 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f03cc063d00 msgr2=0x7f03cc0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:15.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:15.150+0000 7f03f1281700 1 --2- 172.21.15.195:0/3904226663 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f03cc063d00 0x7f03cc0661b0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f03e0009130 tx=0x7f03e0017000 comp rx=0 tx=0).stop 2022-06-27T11:50:15.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:15.150+0000 7f03f1281700 1 -- 172.21.15.195:0/3904226663 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f03ec0c40d0 msgr2=0x7f03ec15d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:15.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:15.150+0000 7f03f1281700 1 --2- 172.21.15.195:0/3904226663 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f03ec0c40d0 0x7f03ec15d570 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f03dc00c490 tx=0x7f03dc007dd0 comp rx=0 tx=0).stop 2022-06-27T11:50:15.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:15.150+0000 7f03f1281700 1 -- 172.21.15.195:0/3904226663 shutdown_connections 2022-06-27T11:50:15.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:15.150+0000 7f03f1281700 1 --2- 172.21.15.195:0/3904226663 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f03cc063d00 0x7f03cc0661b0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:15.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:15.150+0000 7f03f1281700 1 --2- 172.21.15.195:0/3904226663 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f03ec0c6190 0x7f03ec157690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:15.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:15.150+0000 7f03f1281700 1 --2- 172.21.15.195:0/3904226663 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f03ec067ed0 0x7f03ec0c06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:15.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:15.150+0000 7f03f1281700 1 --2- 172.21.15.195:0/3904226663 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f03ec0c40d0 0x7f03ec15d570 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:15.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:15.150+0000 7f03f1281700 1 -- 172.21.15.195:0/3904226663 >> 172.21.15.195:0/3904226663 conn(0x7f03ec0ccb80 msgr2=0x7f03ec0cbfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:50:15.155 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:15.150+0000 7f03f1281700 1 -- 172.21.15.195:0/3904226663 shutdown_connections 2022-06-27T11:50:15.156 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:15.150+0000 7f03f1281700 1 -- 172.21.15.195:0/3904226663 wait complete. 2022-06-27T11:50:15.156 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 11 2022-06-27T11:50:15.161 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T11:50:17.162 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:50:17.303 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.303+0000 7fa928488700 1 Processor -- start 2022-06-27T11:50:17.303 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.303+0000 7fa928488700 1 -- start start 2022-06-27T11:50:17.316 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.304+0000 7fa928488700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fa9200b8520 0x7fa920153de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:17.317 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.304+0000 7fa928488700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa9200c3010 0x7fa9200c3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:17.317 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.304+0000 7fa928488700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa920068a10 con 0x7fa920068ea0 2022-06-27T11:50:17.318 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.304+0000 7fa928488700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa920067ed0 con 0x7fa9200b8520 2022-06-27T11:50:17.318 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.304+0000 7fa928488700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9200692d0 con 0x7fa9200c3010 2022-06-27T11:50:17.319 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.304+0000 7fa926224700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fa9200b8520 0x7fa920153de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:17.319 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.304+0000 7fa925a23700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fa920068ea0 0x7fa9200c45f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:48916/0 (socket says 172.21.15.195:48916) 2022-06-27T11:50:17.320 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.304+0000 7fa925a23700 1 -- 172.21.15.195:0/2059538149 learned_addr learned my addr 172.21.15.195:0/2059538149 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:50:17.320 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.304+0000 7fa926a25700 1 --2- 172.21.15.195:0/2059538149 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa9200c3010 0x7fa9200c3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:17.321 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.304+0000 7fa926224700 1 -- 172.21.15.195:0/2059538149 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa9200c3010 msgr2=0x7fa9200c3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:50:17.321 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.304+0000 7fa926224700 1 --2- 172.21.15.195:0/2059538149 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa9200c3010 0x7fa9200c3490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:17.322 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.304+0000 7fa926224700 1 -- 172.21.15.195:0/2059538149 >> v1:172.21.15.195:6789/0 conn(0x7fa920068ea0 legacy=0x7fa9200c45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:17.322 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.305+0000 7fa926224700 1 -- 172.21.15.195:0/2059538149 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9200c3dd0 con 0x7fa9200b8520 2022-06-27T11:50:17.323 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.304+0000 7fa925222700 1 -- 172.21.15.195:0/2059538149 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2312264115 0 0) 0x7fa920068a10 con 0x7fa920068ea0 2022-06-27T11:50:17.324 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.305+0000 7fa926a25700 1 --2- 172.21.15.195:0/2059538149 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa9200c3010 0x7fa9200c3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T11:50:17.324 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.305+0000 7fa926224700 1 --2- 172.21.15.195:0/2059538149 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa9200b8520 0x7fa920153de0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fa9100097a0 tx=0x7fa910009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95ee73c85c2f76cb server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:50:17.324 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.305+0000 7fa925222700 1 -- 172.21.15.195:0/2059538149 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa910011630 con 0x7fa9200b8520 2022-06-27T11:50:17.325 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.305+0000 7fa925222700 1 -- 172.21.15.195:0/2059538149 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa910011790 con 0x7fa9200b8520 2022-06-27T11:50:17.325 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.305+0000 7fa925222700 1 -- 172.21.15.195:0/2059538149 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa910011b30 con 0x7fa9200b8520 2022-06-27T11:50:17.326 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.305+0000 7fa928488700 1 -- 172.21.15.195:0/2059538149 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa9200b8520 msgr2=0x7fa920153de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:17.326 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.305+0000 7fa928488700 1 --2- 172.21.15.195:0/2059538149 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa9200b8520 0x7fa920153de0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fa9100097a0 tx=0x7fa910009060 comp rx=0 tx=0).stop 2022-06-27T11:50:17.327 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.305+0000 7fa928488700 1 -- 172.21.15.195:0/2059538149 shutdown_connections 2022-06-27T11:50:17.327 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.305+0000 7fa928488700 1 --2- 172.21.15.195:0/2059538149 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa9200c3010 0x7fa9200c3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:17.328 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.305+0000 7fa928488700 1 --2- 172.21.15.195:0/2059538149 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa9200b8520 0x7fa920153de0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:17.328 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.305+0000 7fa928488700 1 -- 172.21.15.195:0/2059538149 >> 172.21.15.195:0/2059538149 conn(0x7fa9200c8740 msgr2=0x7fa9200c8b10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:50:17.329 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.305+0000 7fa928488700 1 -- 172.21.15.195:0/2059538149 shutdown_connections 2022-06-27T11:50:17.329 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.305+0000 7fa928488700 1 -- 172.21.15.195:0/2059538149 wait complete. 2022-06-27T11:50:17.330 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.307+0000 7fa928488700 1 Processor -- start 2022-06-27T11:50:17.330 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.307+0000 7fa928488700 1 -- start start 2022-06-27T11:50:17.331 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.308+0000 7fa928488700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa920068ea0 0x7fa920144590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:17.331 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.308+0000 7fa928488700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa9200b8520 0x7fa920140b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:17.331 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.308+0000 7fa928488700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa9200c3010 0x7fa920141060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:17.332 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.308+0000 7fa928488700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa920144100 con 0x7fa9200b8520 2022-06-27T11:50:17.332 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.308+0000 7fa928488700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa920165640 con 0x7fa920068ea0 2022-06-27T11:50:17.333 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.308+0000 7fa928488700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa920143070 con 0x7fa9200c3010 2022-06-27T11:50:17.333 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.308+0000 7fa925a23700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa9200b8520 0x7fa920140b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:17.334 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.308+0000 7fa925a23700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa9200b8520 0x7fa920140b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:42736/0 (socket says 172.21.15.195:42736) 2022-06-27T11:50:17.334 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.308+0000 7fa925a23700 1 -- 172.21.15.195:0/245834090 learned_addr learned my addr 172.21.15.195:0/245834090 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:50:17.335 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.308+0000 7fa926a25700 1 --2- 172.21.15.195:0/245834090 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa9200c3010 0x7fa920141060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:17.335 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.308+0000 7fa926224700 1 --2- 172.21.15.195:0/245834090 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa920068ea0 0x7fa920144590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:17.335 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.308+0000 7fa925a23700 1 -- 172.21.15.195:0/245834090 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa9200c3010 msgr2=0x7fa920141060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:17.336 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.308+0000 7fa925a23700 1 --2- 172.21.15.195:0/245834090 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa9200c3010 0x7fa920141060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:17.336 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.308+0000 7fa925a23700 1 -- 172.21.15.195:0/245834090 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa920068ea0 msgr2=0x7fa920144590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:17.336 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.308+0000 7fa925a23700 1 --2- 172.21.15.195:0/245834090 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa920068ea0 0x7fa920144590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:17.337 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.308+0000 7fa925a23700 1 -- 172.21.15.195:0/245834090 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9100092e0 con 0x7fa9200b8520 2022-06-27T11:50:17.337 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.308+0000 7fa925a23700 1 --2- 172.21.15.195:0/245834090 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa9200b8520 0x7fa920140b20 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fa91c00c590 tx=0x7fa91c007cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:50:17.337 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.309+0000 7fa90e7fc700 1 -- 172.21.15.195:0/245834090 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa91c007360 con 0x7fa9200b8520 2022-06-27T11:50:17.338 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.309+0000 7fa90e7fc700 1 -- 172.21.15.195:0/245834090 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa91c0074e0 con 0x7fa9200b8520 2022-06-27T11:50:17.338 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.309+0000 7fa928488700 1 -- 172.21.15.195:0/245834090 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa920143350 con 0x7fa9200b8520 2022-06-27T11:50:17.338 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.309+0000 7fa90e7fc700 1 -- 172.21.15.195:0/245834090 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa91c004940 con 0x7fa9200b8520 2022-06-27T11:50:17.339 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.309+0000 7fa928488700 1 -- 172.21.15.195:0/245834090 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa920143d90 con 0x7fa9200b8520 2022-06-27T11:50:17.340 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.310+0000 7fa928488700 1 -- 172.21.15.195:0/245834090 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9200722c0 con 0x7fa9200b8520 2022-06-27T11:50:17.340 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.310+0000 7fa90e7fc700 1 -- 172.21.15.195:0/245834090 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fa91c010030 con 0x7fa9200b8520 2022-06-27T11:50:17.341 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.310+0000 7fa90e7fc700 1 --2- 172.21.15.195:0/245834090 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa900063d30 0x7fa9000661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:17.341 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.310+0000 7fa90e7fc700 1 -- 172.21.15.195:0/245834090 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6202+0+0 (secure 0 0 0) 0x7fa91c080390 con 0x7fa9200b8520 2022-06-27T11:50:17.341 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.310+0000 7fa926224700 1 --2- 172.21.15.195:0/245834090 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa900063d30 0x7fa9000661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:17.342 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.311+0000 7fa926224700 1 --2- 172.21.15.195:0/245834090 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa900063d30 0x7fa9000661e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa910009260 tx=0x7fa910017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:50:17.342 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.315+0000 7fa90e7fc700 1 -- 172.21.15.195:0/245834090 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fa91c04ef10 con 0x7fa9200b8520 2022-06-27T11:50:17.552 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.545+0000 7fa928488700 1 -- 172.21.15.195:0/245834090 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa9200680e0 con 0x7fa9200b8520 2022-06-27T11:50:17.553 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.545+0000 7fa90e7fc700 1 -- 172.21.15.195:0/245834090 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+5364 (secure 0 0 0) 0x7fa91c052530 con 0x7fa9200b8520 2022-06-27T11:50:17.554 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:50:17.554 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":11,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":11,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:50:12.001975+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"b","rank":0,"incarnation":9,"state":"up:rejoin","state_seq":38,"addr":"172.21.15.202:6833/864126677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":864126677},{"type":"v1","addr":"172.21.15.202:6833","nonce":864126677}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:50:17.574 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.547+0000 7fa928488700 1 -- 172.21.15.195:0/245834090 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa900063d30 msgr2=0x7fa9000661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:17.581 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.547+0000 7fa928488700 1 --2- 172.21.15.195:0/245834090 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa900063d30 0x7fa9000661e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa910009260 tx=0x7fa910017000 comp rx=0 tx=0).stop 2022-06-27T11:50:17.582 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.547+0000 7fa928488700 1 -- 172.21.15.195:0/245834090 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa9200b8520 msgr2=0x7fa920140b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:17.585 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.547+0000 7fa928488700 1 --2- 172.21.15.195:0/245834090 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa9200b8520 0x7fa920140b20 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fa91c00c590 tx=0x7fa91c007cb0 comp rx=0 tx=0).stop 2022-06-27T11:50:17.585 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.547+0000 7fa928488700 1 -- 172.21.15.195:0/245834090 shutdown_connections 2022-06-27T11:50:17.586 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.547+0000 7fa928488700 1 --2- 172.21.15.195:0/245834090 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa900063d30 0x7fa9000661e0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:17.586 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.547+0000 7fa928488700 1 --2- 172.21.15.195:0/245834090 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa9200c3010 0x7fa920141060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:17.587 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.547+0000 7fa928488700 1 --2- 172.21.15.195:0/245834090 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa920068ea0 0x7fa920144590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:17.587 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.547+0000 7fa928488700 1 --2- 172.21.15.195:0/245834090 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa9200b8520 0x7fa920140b20 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:17.587 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.547+0000 7fa928488700 1 -- 172.21.15.195:0/245834090 >> 172.21.15.195:0/245834090 conn(0x7fa9200c8740 msgr2=0x7fa920152ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:50:17.588 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.547+0000 7fa928488700 1 -- 172.21.15.195:0/245834090 shutdown_connections 2022-06-27T11:50:17.588 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:17.547+0000 7fa928488700 1 -- 172.21.15.195:0/245834090 wait complete. 2022-06-27T11:50:17.589 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 11 2022-06-27T11:50:17.590 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T11:50:19.592 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:50:19.725 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.725+0000 7faa46d3d700 1 Processor -- start 2022-06-27T11:50:19.725 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.726+0000 7faa46d3d700 1 -- start start 2022-06-27T11:50:19.737 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.726+0000 7faa46d3d700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7faa4014e860 0x7faa400c3840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:19.738 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.726+0000 7faa46d3d700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa400c33b0 con 0x7faa400c3d80 2022-06-27T11:50:19.738 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.726+0000 7faa46d3d700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa4006f220 con 0x7faa4014e490 2022-06-27T11:50:19.739 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.726+0000 7faa46d3d700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa401542c0 con 0x7faa4014e860 2022-06-27T11:50:19.739 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.726+0000 7faa452da700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7faa400c3d80 0x7faa40157890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:48934/0 (socket says 172.21.15.195:48934) 2022-06-27T11:50:19.739 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.727+0000 7faa452da700 1 -- 172.21.15.195:0/3250542416 learned_addr learned my addr 172.21.15.195:0/3250542416 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:50:19.740 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.727+0000 7faa3ffff700 1 --2- 172.21.15.195:0/3250542416 >> v2:172.21.15.202:3300/0 conn(0x7faa4014e860 0x7faa400c3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:19.740 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.727+0000 7faa3f7fe700 1 -- 172.21.15.195:0/3250542416 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3846632831 0 0) 0x7faa400c33b0 con 0x7faa400c3d80 2022-06-27T11:50:19.740 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.727+0000 7faa3f7fe700 1 -- 172.21.15.195:0/3250542416 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa24003430 con 0x7faa400c3d80 2022-06-27T11:50:19.742 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.727+0000 7faa3f7fe700 1 -- 172.21.15.195:0/3250542416 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 305194439 0 0) 0x7faa4006f220 con 0x7faa4014e490 2022-06-27T11:50:19.742 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.727+0000 7faa3f7fe700 1 -- 172.21.15.195:0/3250542416 --> v1:172.21.15.202:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa400c33b0 con 0x7faa4014e490 2022-06-27T11:50:19.743 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.727+0000 7faa3f7fe700 1 -- 172.21.15.195:0/3250542416 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1022230122 0 0) 0x7faa24003430 con 0x7faa400c3d80 2022-06-27T11:50:19.743 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.727+0000 7faa3f7fe700 1 -- 172.21.15.195:0/3250542416 >> v2:172.21.15.202:3300/0 conn(0x7faa4014e860 msgr2=0x7faa400c3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:50:19.743 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.727+0000 7faa3f7fe700 1 --2- 172.21.15.195:0/3250542416 >> v2:172.21.15.202:3300/0 conn(0x7faa4014e860 0x7faa400c3840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:19.743 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.727+0000 7faa3f7fe700 1 -- 172.21.15.195:0/3250542416 >> v1:172.21.15.202:6789/0 conn(0x7faa4014e490 legacy=0x7faa4006cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:19.744 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.727+0000 7faa3f7fe700 1 -- 172.21.15.195:0/3250542416 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa40154560 con 0x7faa400c3d80 2022-06-27T11:50:19.744 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.727+0000 7faa3f7fe700 1 -- 172.21.15.195:0/3250542416 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7faa380017f0 con 0x7faa400c3d80 2022-06-27T11:50:19.744 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.727+0000 7faa3f7fe700 1 -- 172.21.15.195:0/3250542416 >> v1:172.21.15.195:6789/0 conn(0x7faa400c3d80 legacy=0x7faa40157890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:19.744 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.727+0000 7faa3f7fe700 1 --2- 172.21.15.195:0/3250542416 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7faa24003ab0 0x7faa24003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:19.745 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.727+0000 7faa3f7fe700 1 --2- 172.21.15.195:0/3250542416 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faa240046d0 0x7faa24004b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:19.745 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.727+0000 7faa3f7fe700 1 --2- 172.21.15.195:0/3250542416 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faa24005080 0x7faa2400b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:19.745 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.727+0000 7faa3f7fe700 1 -- 172.21.15.195:0/3250542416 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa401542c0 con 0x7faa24005080 2022-06-27T11:50:19.746 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.727+0000 7faa3f7fe700 1 -- 172.21.15.195:0/3250542416 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa2400bf70 con 0x7faa24003ab0 2022-06-27T11:50:19.747 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.727+0000 7faa3f7fe700 1 -- 172.21.15.195:0/3250542416 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa2400c130 con 0x7faa240046d0 2022-06-27T11:50:19.747 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.727+0000 7faa452da700 1 --2- 172.21.15.195:0/3250542416 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faa24005080 0x7faa2400b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:19.747 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.727+0000 7faa44ad9700 1 --2- 172.21.15.195:0/3250542416 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7faa24003ab0 0x7faa24003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:19.748 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.727+0000 7faa3ffff700 1 --2- 172.21.15.195:0/3250542416 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faa240046d0 0x7faa24004b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:19.748 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.728+0000 7faa452da700 1 -- 172.21.15.195:0/3250542416 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faa240046d0 msgr2=0x7faa24004b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:50:19.749 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.728+0000 7faa452da700 1 --2- 172.21.15.195:0/3250542416 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faa240046d0 0x7faa24004b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:19.749 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.728+0000 7faa452da700 1 -- 172.21.15.195:0/3250542416 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7faa24003ab0 msgr2=0x7faa24003ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:50:19.749 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.728+0000 7faa452da700 1 --2- 172.21.15.195:0/3250542416 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7faa24003ab0 0x7faa24003ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:19.750 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.728+0000 7faa452da700 1 -- 172.21.15.195:0/3250542416 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa24003430 con 0x7faa24005080 2022-06-27T11:50:19.750 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.728+0000 7faa452da700 1 -- 172.21.15.195:0/3250542416 reap_dead start 2022-06-27T11:50:19.750 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.728+0000 7faa452da700 1 --2- 172.21.15.195:0/3250542416 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faa24005080 0x7faa2400b920 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7faa3800c5f0 tx=0x7faa4006ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcf0c888543a9ff4 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:50:19.751 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.728+0000 7faa3f7fe700 1 -- 172.21.15.195:0/3250542416 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa3800cdf0 con 0x7faa24005080 2022-06-27T11:50:19.751 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.728+0000 7faa3f7fe700 1 -- 172.21.15.195:0/3250542416 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa38005b60 con 0x7faa24005080 2022-06-27T11:50:19.751 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.728+0000 7faa3f7fe700 1 -- 172.21.15.195:0/3250542416 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa380070d0 con 0x7faa24005080 2022-06-27T11:50:19.752 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.728+0000 7faa46d3d700 1 -- 172.21.15.195:0/3250542416 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faa24005080 msgr2=0x7faa2400b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:19.753 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.728+0000 7faa46d3d700 1 --2- 172.21.15.195:0/3250542416 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faa24005080 0x7faa2400b920 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7faa3800c5f0 tx=0x7faa4006ee20 comp rx=0 tx=0).stop 2022-06-27T11:50:19.753 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.728+0000 7faa46d3d700 1 -- 172.21.15.195:0/3250542416 shutdown_connections 2022-06-27T11:50:19.753 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.728+0000 7faa46d3d700 1 --2- 172.21.15.195:0/3250542416 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faa24005080 0x7faa2400b920 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:19.754 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.728+0000 7faa46d3d700 1 -- 172.21.15.195:0/3250542416 >> 172.21.15.195:0/3250542416 conn(0x7faa40063af0 msgr2=0x7faa40063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:50:19.754 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.728+0000 7faa46d3d700 1 -- 172.21.15.195:0/3250542416 shutdown_connections 2022-06-27T11:50:19.754 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.728+0000 7faa46d3d700 1 -- 172.21.15.195:0/3250542416 wait complete. 2022-06-27T11:50:19.755 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.730+0000 7faa46d3d700 1 Processor -- start 2022-06-27T11:50:19.755 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.730+0000 7faa46d3d700 1 -- start start 2022-06-27T11:50:19.755 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.730+0000 7faa46d3d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faa24005080 0x7faa40069f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:19.756 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.730+0000 7faa46d3d700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faa40145160 0x7faa40068fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:19.757 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.730+0000 7faa46d3d700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7faa4015a340 0x7faa40149e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:19.757 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.730+0000 7faa46d3d700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa40151d10 con 0x7faa24005080 2022-06-27T11:50:19.757 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.731+0000 7faa46d3d700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa401512c0 con 0x7faa4015a340 2022-06-27T11:50:19.758 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.731+0000 7faa46d3d700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa4006a4a0 con 0x7faa40145160 2022-06-27T11:50:19.758 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.731+0000 7faa44ad9700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faa24005080 0x7faa40069f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:19.758 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.731+0000 7faa44ad9700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faa24005080 0x7faa40069f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:42754/0 (socket says 172.21.15.195:42754) 2022-06-27T11:50:19.758 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.731+0000 7faa44ad9700 1 -- 172.21.15.195:0/464332992 learned_addr learned my addr 172.21.15.195:0/464332992 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:50:19.759 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.731+0000 7faa452da700 1 --2- 172.21.15.195:0/464332992 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7faa4015a340 0x7faa40149e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:19.759 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.731+0000 7faa3ffff700 1 --2- 172.21.15.195:0/464332992 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faa40145160 0x7faa40068fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:19.759 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.731+0000 7faa44ad9700 1 -- 172.21.15.195:0/464332992 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faa40145160 msgr2=0x7faa40068fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:19.760 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.731+0000 7faa44ad9700 1 --2- 172.21.15.195:0/464332992 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faa40145160 0x7faa40068fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:19.762 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.731+0000 7faa44ad9700 1 -- 172.21.15.195:0/464332992 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7faa4015a340 msgr2=0x7faa40149e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:19.762 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.731+0000 7faa44ad9700 1 --2- 172.21.15.195:0/464332992 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7faa4015a340 0x7faa40149e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:19.763 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.731+0000 7faa44ad9700 1 -- 172.21.15.195:0/464332992 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa4014bd40 con 0x7faa24005080 2022-06-27T11:50:19.763 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.731+0000 7faa452da700 1 --2- 172.21.15.195:0/464332992 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7faa4015a340 0x7faa40149e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T11:50:19.763 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.731+0000 7faa3ffff700 1 --2- 172.21.15.195:0/464332992 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faa40145160 0x7faa40068fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T11:50:19.764 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.731+0000 7faa44ad9700 1 --2- 172.21.15.195:0/464332992 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faa24005080 0x7faa40069f60 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7faa3400f8b0 tx=0x7faa34009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:50:19.764 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.731+0000 7faa3cff9700 1 -- 172.21.15.195:0/464332992 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa340052e0 con 0x7faa24005080 2022-06-27T11:50:19.765 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.731+0000 7faa3cff9700 1 -- 172.21.15.195:0/464332992 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa34003500 con 0x7faa24005080 2022-06-27T11:50:19.765 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.731+0000 7faa46d3d700 1 -- 172.21.15.195:0/464332992 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa401514f0 con 0x7faa24005080 2022-06-27T11:50:19.765 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.731+0000 7faa3cff9700 1 -- 172.21.15.195:0/464332992 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa340038c0 con 0x7faa24005080 2022-06-27T11:50:19.765 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.731+0000 7faa46d3d700 1 -- 172.21.15.195:0/464332992 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa40151a30 con 0x7faa24005080 2022-06-27T11:50:19.766 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.732+0000 7faa46d3d700 1 -- 172.21.15.195:0/464332992 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa400722c0 con 0x7faa24005080 2022-06-27T11:50:19.766 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.733+0000 7faa3cff9700 1 -- 172.21.15.195:0/464332992 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7faa34003c30 con 0x7faa24005080 2022-06-27T11:50:19.766 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.733+0000 7faa3cff9700 1 --2- 172.21.15.195:0/464332992 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7faa28063d30 0x7faa280661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:19.767 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.733+0000 7faa3cff9700 1 -- 172.21.15.195:0/464332992 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6202+0+0 (secure 0 0 0) 0x7faa34080b30 con 0x7faa24005080 2022-06-27T11:50:19.767 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.733+0000 7faa3ffff700 1 --2- 172.21.15.195:0/464332992 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7faa28063d30 0x7faa280661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:19.768 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.733+0000 7faa3ffff700 1 --2- 172.21.15.195:0/464332992 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7faa28063d30 0x7faa280661e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7faa4006b1c0 tx=0x7faa2c008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:50:19.768 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.737+0000 7faa3cff9700 1 -- 172.21.15.195:0/464332992 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7faa3404f690 con 0x7faa24005080 2022-06-27T11:50:19.961 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.962+0000 7faa46d3d700 1 -- 172.21.15.195:0/464332992 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faa401540a0 con 0x7faa24005080 2022-06-27T11:50:19.961 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.962+0000 7faa3cff9700 1 -- 172.21.15.195:0/464332992 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+5370 (secure 0 0 0) 0x7faa34052cb0 con 0x7faa24005080 2022-06-27T11:50:19.962 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:50:19.963 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:50:18.020149+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"b","rank":0,"incarnation":9,"state":"up:clientreplay","state_seq":40,"addr":"172.21.15.202:6833/864126677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":864126677},{"type":"v1","addr":"172.21.15.202:6833","nonce":864126677}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:50:19.965 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.964+0000 7faa46d3d700 1 -- 172.21.15.195:0/464332992 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7faa28063d30 msgr2=0x7faa280661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:19.965 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.964+0000 7faa46d3d700 1 --2- 172.21.15.195:0/464332992 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7faa28063d30 0x7faa280661e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7faa4006b1c0 tx=0x7faa2c008040 comp rx=0 tx=0).stop 2022-06-27T11:50:19.966 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.964+0000 7faa46d3d700 1 -- 172.21.15.195:0/464332992 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faa24005080 msgr2=0x7faa40069f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:19.966 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.964+0000 7faa46d3d700 1 --2- 172.21.15.195:0/464332992 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faa24005080 0x7faa40069f60 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7faa3400f8b0 tx=0x7faa34009040 comp rx=0 tx=0).stop 2022-06-27T11:50:19.966 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.964+0000 7faa46d3d700 1 -- 172.21.15.195:0/464332992 shutdown_connections 2022-06-27T11:50:19.966 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.964+0000 7faa46d3d700 1 --2- 172.21.15.195:0/464332992 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7faa28063d30 0x7faa280661e0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:19.966 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.964+0000 7faa46d3d700 1 --2- 172.21.15.195:0/464332992 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faa24005080 0x7faa40069f60 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:19.967 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.964+0000 7faa46d3d700 1 --2- 172.21.15.195:0/464332992 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7faa4015a340 0x7faa40149e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:19.967 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.964+0000 7faa46d3d700 1 --2- 172.21.15.195:0/464332992 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faa40145160 0x7faa40068fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:19.967 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.964+0000 7faa46d3d700 1 -- 172.21.15.195:0/464332992 >> 172.21.15.195:0/464332992 conn(0x7faa40063af0 msgr2=0x7faa4014d6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:50:19.967 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.964+0000 7faa46d3d700 1 -- 172.21.15.195:0/464332992 shutdown_connections 2022-06-27T11:50:19.969 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:19.964+0000 7faa46d3d700 1 -- 172.21.15.195:0/464332992 wait complete. 2022-06-27T11:50:19.969 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 12 2022-06-27T11:50:19.976 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T11:50:21.982 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:50:22.118 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.119+0000 7f407ae8a700 1 Processor -- start 2022-06-27T11:50:22.119 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.119+0000 7f407ae8a700 1 -- start start 2022-06-27T11:50:22.125 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.120+0000 7f407ae8a700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f407414e490 0x7f407406cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:22.125 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.120+0000 7f407ae8a700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f40740c3f50 0x7f4074157890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:22.126 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.120+0000 7f407ae8a700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40740c33b0 con 0x7f407414e860 2022-06-27T11:50:22.127 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.120+0000 7f407ae8a700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f407406d520 con 0x7f407414e490 2022-06-27T11:50:22.127 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.120+0000 7f407ae8a700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f407406f220 con 0x7f40740c3f50 2022-06-27T11:50:22.127 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.120+0000 7f4078c26700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f407414e490 0x7f407406cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:22.128 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.120+0000 7f4078c26700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f407414e490 0x7f407406cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:42762/0 (socket says 172.21.15.195:42762) 2022-06-27T11:50:22.128 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.120+0000 7f4078c26700 1 -- 172.21.15.195:0/2377915061 learned_addr learned my addr 172.21.15.195:0/2377915061 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:50:22.129 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.120+0000 7f4079427700 1 --2- 172.21.15.195:0/2377915061 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f40740c3f50 0x7f4074157890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:22.132 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.120+0000 7f4078c26700 1 -- 172.21.15.195:0/2377915061 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f40740c3f50 msgr2=0x7f4074157890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:50:22.132 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.120+0000 7f4078c26700 1 --2- 172.21.15.195:0/2377915061 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f40740c3f50 0x7f4074157890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:22.132 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.120+0000 7f4078c26700 1 -- 172.21.15.195:0/2377915061 >> v1:172.21.15.202:6789/0 conn(0x7f407414e860 legacy=0x7f40740c3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:50:22.132 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.120+0000 7f4078c26700 1 -- 172.21.15.195:0/2377915061 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40741542c0 con 0x7f407414e490 2022-06-27T11:50:22.133 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.120+0000 7f4079427700 1 --2- 172.21.15.195:0/2377915061 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f40740c3f50 0x7f4074157890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T11:50:22.133 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.120+0000 7f4078c26700 1 --2- 172.21.15.195:0/2377915061 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f407414e490 0x7f407406cfe0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f4068009860 tx=0x7f4068009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6e8751b732248f8 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:50:22.133 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.121+0000 7f40737fe700 1 -- 172.21.15.195:0/2377915061 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4068004030 con 0x7f407414e490 2022-06-27T11:50:22.134 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.121+0000 7f40737fe700 1 -- 172.21.15.195:0/2377915061 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4068004190 con 0x7f407414e490 2022-06-27T11:50:22.134 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.121+0000 7f40737fe700 1 -- 172.21.15.195:0/2377915061 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f406800b640 con 0x7f407414e490 2022-06-27T11:50:22.134 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.121+0000 7f407ae8a700 1 -- 172.21.15.195:0/2377915061 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f407414e490 msgr2=0x7f407406cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:22.134 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.121+0000 7f407ae8a700 1 --2- 172.21.15.195:0/2377915061 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f407414e490 0x7f407406cfe0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f4068009860 tx=0x7f4068009120 comp rx=0 tx=0).stop 2022-06-27T11:50:22.135 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.121+0000 7f407ae8a700 1 -- 172.21.15.195:0/2377915061 shutdown_connections 2022-06-27T11:50:22.135 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.121+0000 7f407ae8a700 1 --2- 172.21.15.195:0/2377915061 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f40740c3f50 0x7f4074157890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:22.136 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.121+0000 7f407ae8a700 1 --2- 172.21.15.195:0/2377915061 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f407414e490 0x7f407406cfe0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:22.136 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.121+0000 7f407ae8a700 1 -- 172.21.15.195:0/2377915061 >> 172.21.15.195:0/2377915061 conn(0x7f4074063af0 msgr2=0x7f4074063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:50:22.137 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.121+0000 7f407ae8a700 1 -- 172.21.15.195:0/2377915061 shutdown_connections 2022-06-27T11:50:22.137 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.121+0000 7f407ae8a700 1 -- 172.21.15.195:0/2377915061 wait complete. 2022-06-27T11:50:22.137 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.122+0000 7f407ae8a700 1 Processor -- start 2022-06-27T11:50:22.140 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.123+0000 7f407ae8a700 1 -- start start 2022-06-27T11:50:22.140 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.123+0000 7f407ae8a700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f40740c3f50 0x7f4074164f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:22.141 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.123+0000 7f407ae8a700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f407414e490 0x7f40740bef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:22.141 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.123+0000 7f407ae8a700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f407414e860 0x7f40740c0560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:22.141 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.123+0000 7f407ae8a700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4074158100 con 0x7f40740c3f50 2022-06-27T11:50:22.141 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.123+0000 7f407ae8a700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40740bfda0 con 0x7f407414e860 2022-06-27T11:50:22.142 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.123+0000 7f407ae8a700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40740c3b00 con 0x7f407414e490 2022-06-27T11:50:22.142 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.123+0000 7f4078c26700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f40740c3f50 0x7f4074164f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:22.145 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.123+0000 7f4078c26700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f40740c3f50 0x7f4074164f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:42768/0 (socket says 172.21.15.195:42768) 2022-06-27T11:50:22.145 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.123+0000 7f4078c26700 1 -- 172.21.15.195:0/1663192836 learned_addr learned my addr 172.21.15.195:0/1663192836 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:50:22.146 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.124+0000 7f4073fff700 1 --2- 172.21.15.195:0/1663192836 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f407414e490 0x7f40740bef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:22.147 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.124+0000 7f4079427700 1 --2- 172.21.15.195:0/1663192836 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f407414e860 0x7f40740c0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:22.148 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.124+0000 7f4078c26700 1 -- 172.21.15.195:0/1663192836 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f407414e490 msgr2=0x7f40740bef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:22.148 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.124+0000 7f4078c26700 1 --2- 172.21.15.195:0/1663192836 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f407414e490 0x7f40740bef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:22.148 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.124+0000 7f4078c26700 1 -- 172.21.15.195:0/1663192836 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f407414e860 msgr2=0x7f40740c0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:22.149 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.124+0000 7f4078c26700 1 --2- 172.21.15.195:0/1663192836 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f407414e860 0x7f40740c0560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:22.149 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.124+0000 7f4078c26700 1 -- 172.21.15.195:0/1663192836 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40680093a0 con 0x7f40740c3f50 2022-06-27T11:50:22.151 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.124+0000 7f4079427700 1 --2- 172.21.15.195:0/1663192836 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f407414e860 0x7f40740c0560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T11:50:22.151 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.124+0000 7f4078c26700 1 --2- 172.21.15.195:0/1663192836 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f40740c3f50 0x7f4074164f10 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f4068009160 tx=0x7f4068016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:50:22.151 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.124+0000 7f4070ff9700 1 -- 172.21.15.195:0/1663192836 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4068003730 con 0x7f40740c3f50 2022-06-27T11:50:22.151 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.124+0000 7f4070ff9700 1 -- 172.21.15.195:0/1663192836 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4068003890 con 0x7f40740c3f50 2022-06-27T11:50:22.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.124+0000 7f4070ff9700 1 -- 172.21.15.195:0/1663192836 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4068003c50 con 0x7f40740c3f50 2022-06-27T11:50:22.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.124+0000 7f407ae8a700 1 -- 172.21.15.195:0/1663192836 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4074157860 con 0x7f40740c3f50 2022-06-27T11:50:22.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.124+0000 7f407ae8a700 1 -- 172.21.15.195:0/1663192836 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4074157e20 con 0x7f40740c3f50 2022-06-27T11:50:22.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.125+0000 7f407ae8a700 1 -- 172.21.15.195:0/1663192836 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40740722c0 con 0x7f40740c3f50 2022-06-27T11:50:22.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.125+0000 7f4070ff9700 1 -- 172.21.15.195:0/1663192836 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f406802b030 con 0x7f40740c3f50 2022-06-27T11:50:22.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.126+0000 7f4070ff9700 1 --2- 172.21.15.195:0/1663192836 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f405c063d30 0x7f405c0661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:22.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.126+0000 7f4070ff9700 1 -- 172.21.15.195:0/1663192836 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6202+0+0 (secure 0 0 0) 0x7f4068082320 con 0x7f40740c3f50 2022-06-27T11:50:22.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.126+0000 7f4073fff700 1 --2- 172.21.15.195:0/1663192836 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f405c063d30 0x7f405c0661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:22.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.126+0000 7f4073fff700 1 --2- 172.21.15.195:0/1663192836 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f405c063d30 0x7f405c0661e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f4074153be0 tx=0x7f4060008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:50:22.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.130+0000 7f4070ff9700 1 -- 172.21.15.195:0/1663192836 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f4068050e80 con 0x7f40740c3f50 2022-06-27T11:50:22.357 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.357+0000 7f407ae8a700 1 -- 172.21.15.195:0/1663192836 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f40740c7250 con 0x7f40740c3f50 2022-06-27T11:50:22.358 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.357+0000 7f4070ff9700 1 -- 172.21.15.195:0/1663192836 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+5370 (secure 0 0 0) 0x7f4068029090 con 0x7f40740c3f50 2022-06-27T11:50:22.358 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:50:22.358 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:50:18.020149+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"b","rank":0,"incarnation":9,"state":"up:clientreplay","state_seq":40,"addr":"172.21.15.202:6833/864126677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":864126677},{"type":"v1","addr":"172.21.15.202:6833","nonce":864126677}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:50:22.360 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.359+0000 7f407ae8a700 1 -- 172.21.15.195:0/1663192836 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f405c063d30 msgr2=0x7f405c0661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:22.360 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.359+0000 7f407ae8a700 1 --2- 172.21.15.195:0/1663192836 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f405c063d30 0x7f405c0661e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f4074153be0 tx=0x7f4060008040 comp rx=0 tx=0).stop 2022-06-27T11:50:22.360 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.359+0000 7f407ae8a700 1 -- 172.21.15.195:0/1663192836 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f40740c3f50 msgr2=0x7f4074164f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:22.361 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.359+0000 7f407ae8a700 1 --2- 172.21.15.195:0/1663192836 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f40740c3f50 0x7f4074164f10 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f4068009160 tx=0x7f4068016000 comp rx=0 tx=0).stop 2022-06-27T11:50:22.361 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.359+0000 7f407ae8a700 1 -- 172.21.15.195:0/1663192836 shutdown_connections 2022-06-27T11:50:22.361 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.359+0000 7f407ae8a700 1 --2- 172.21.15.195:0/1663192836 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f405c063d30 0x7f405c0661e0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:22.361 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.359+0000 7f407ae8a700 1 --2- 172.21.15.195:0/1663192836 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f40740c3f50 0x7f4074164f10 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:22.362 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.359+0000 7f407ae8a700 1 --2- 172.21.15.195:0/1663192836 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f407414e860 0x7f40740c0560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:22.362 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.359+0000 7f407ae8a700 1 --2- 172.21.15.195:0/1663192836 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f407414e490 0x7f40740bef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:22.362 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.360+0000 7f407ae8a700 1 -- 172.21.15.195:0/1663192836 >> 172.21.15.195:0/1663192836 conn(0x7f4074063af0 msgr2=0x7f4074152750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:50:22.363 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.360+0000 7f407ae8a700 1 -- 172.21.15.195:0/1663192836 shutdown_connections 2022-06-27T11:50:22.363 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:22.360+0000 7f407ae8a700 1 -- 172.21.15.195:0/1663192836 wait complete. 2022-06-27T11:50:22.365 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 12 2022-06-27T11:50:22.371 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T11:50:24.374 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:50:24.511 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.500+0000 7fa542a90700 1 Processor -- start 2022-06-27T11:50:24.511 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.500+0000 7fa542a90700 1 -- start start 2022-06-27T11:50:24.512 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.501+0000 7fa542a90700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fa53c0c2bf0 0x7fa53c0c2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:24.512 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.501+0000 7fa542a90700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa53c0c61e0 0x7fa53c0c48f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:24.512 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.501+0000 7fa542a90700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fa53c0c4e30 0x7fa53c0c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:24.513 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.501+0000 7fa542a90700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa53c0c52b0 con 0x7fa53c0c2bf0 2022-06-27T11:50:24.513 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.501+0000 7fa542a90700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa53c0bc490 con 0x7fa53c0c4e30 2022-06-27T11:50:24.514 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.501+0000 7fa542a90700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa53c06a0e0 con 0x7fa53c0c61e0 2022-06-27T11:50:24.514 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.501+0000 7fa54082c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fa53c0c2bf0 0x7fa53c0c2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:24.515 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.501+0000 7fa54082c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fa53c0c2bf0 0x7fa53c0c2fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:42796/0 (socket says 172.21.15.195:42796) 2022-06-27T11:50:24.515 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.501+0000 7fa54082c700 1 -- 172.21.15.195:0/3538954431 learned_addr learned my addr 172.21.15.195:0/3538954431 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:50:24.515 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.501+0000 7fa53bfff700 1 --2- 172.21.15.195:0/3538954431 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa53c0c61e0 0x7fa53c0c48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:24.516 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.501+0000 7fa54102d700 1 --2- 172.21.15.195:0/3538954431 >> v2:172.21.15.202:3300/0 conn(0x7fa53c0c4e30 0x7fa53c0c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:24.516 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.502+0000 7fa54082c700 1 -- 172.21.15.195:0/3538954431 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa53c0c61e0 msgr2=0x7fa53c0c48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:50:24.517 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.502+0000 7fa54082c700 1 --2- 172.21.15.195:0/3538954431 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa53c0c61e0 0x7fa53c0c48f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:24.517 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.502+0000 7fa54082c700 1 -- 172.21.15.195:0/3538954431 >> v2:172.21.15.202:3300/0 conn(0x7fa53c0c4e30 msgr2=0x7fa53c0c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:50:24.518 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.502+0000 7fa54082c700 1 --2- 172.21.15.195:0/3538954431 >> v2:172.21.15.202:3300/0 conn(0x7fa53c0c4e30 0x7fa53c0c3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:24.518 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.502+0000 7fa54082c700 1 -- 172.21.15.195:0/3538954431 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa53c0bc6e0 con 0x7fa53c0c2bf0 2022-06-27T11:50:24.519 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.502+0000 7fa54082c700 1 --2- 172.21.15.195:0/3538954431 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa53c0c2bf0 0x7fa53c0c2fc0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fa53c06d020 tx=0x7fa5300090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a44c2975773dd22f server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:50:24.519 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.502+0000 7fa53b7fe700 1 -- 172.21.15.195:0/3538954431 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa530005ad0 con 0x7fa53c0c2bf0 2022-06-27T11:50:24.520 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.502+0000 7fa53b7fe700 1 -- 172.21.15.195:0/3538954431 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa530005c30 con 0x7fa53c0c2bf0 2022-06-27T11:50:24.520 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.502+0000 7fa53b7fe700 1 -- 172.21.15.195:0/3538954431 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa530004030 con 0x7fa53c0c2bf0 2022-06-27T11:50:24.521 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.502+0000 7fa542a90700 1 -- 172.21.15.195:0/3538954431 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa53c0c2bf0 msgr2=0x7fa53c0c2fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:24.521 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.502+0000 7fa542a90700 1 --2- 172.21.15.195:0/3538954431 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa53c0c2bf0 0x7fa53c0c2fc0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fa53c06d020 tx=0x7fa5300090f0 comp rx=0 tx=0).stop 2022-06-27T11:50:24.522 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.503+0000 7fa542a90700 1 -- 172.21.15.195:0/3538954431 shutdown_connections 2022-06-27T11:50:24.522 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.503+0000 7fa542a90700 1 --2- 172.21.15.195:0/3538954431 >> v2:172.21.15.202:3300/0 conn(0x7fa53c0c4e30 0x7fa53c0c3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:24.522 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.503+0000 7fa542a90700 1 --2- 172.21.15.195:0/3538954431 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa53c0c2bf0 0x7fa53c0c2fc0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:24.523 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.503+0000 7fa542a90700 1 --2- 172.21.15.195:0/3538954431 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa53c0c61e0 0x7fa53c0c48f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:24.523 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.503+0000 7fa542a90700 1 -- 172.21.15.195:0/3538954431 >> 172.21.15.195:0/3538954431 conn(0x7fa53c0c9b00 msgr2=0x7fa53c0c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:50:24.524 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.503+0000 7fa542a90700 1 -- 172.21.15.195:0/3538954431 shutdown_connections 2022-06-27T11:50:24.524 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.503+0000 7fa542a90700 1 -- 172.21.15.195:0/3538954431 wait complete. 2022-06-27T11:50:24.525 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.504+0000 7fa542a90700 1 Processor -- start 2022-06-27T11:50:24.525 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.504+0000 7fa542a90700 1 -- start start 2022-06-27T11:50:24.525 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.504+0000 7fa542a90700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa53c0c2bf0 0x7fa53c069690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:24.526 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.504+0000 7fa542a90700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa53c0c4e30 0x7fa53c0c7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:24.526 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.504+0000 7fa542a90700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa53c0c61e0 0x7fa53c0c8230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:24.527 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.504+0000 7fa542a90700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa53c16a830 con 0x7fa53c0c61e0 2022-06-27T11:50:24.527 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.504+0000 7fa542a90700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa53c1662f0 con 0x7fa53c0c4e30 2022-06-27T11:50:24.528 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.505+0000 7fa542a90700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa53c167110 con 0x7fa53c0c2bf0 2022-06-27T11:50:24.528 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.505+0000 7fa54102d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa53c0c61e0 0x7fa53c0c8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:24.528 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.505+0000 7fa54082c700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa53c0c2bf0 0x7fa53c069690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:24.529 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.505+0000 7fa53bfff700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa53c0c4e30 0x7fa53c0c7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:24.529 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.505+0000 7fa54082c700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa53c0c2bf0 0x7fa53c069690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3301/0 says I am v2:172.21.15.195:35556/0 (socket says 172.21.15.195:35556) 2022-06-27T11:50:24.530 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.505+0000 7fa54102d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa53c0c61e0 0x7fa53c0c8230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:42804/0 (socket says 172.21.15.195:42804) 2022-06-27T11:50:24.530 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.505+0000 7fa54082c700 1 -- 172.21.15.195:0/4107837487 learned_addr learned my addr 172.21.15.195:0/4107837487 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:50:24.530 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.505+0000 7fa53bfff700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa53c0c4e30 0x7fa53c0c7cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3300/0 says I am v2:172.21.15.195:40928/0 (socket says 172.21.15.195:40928) 2022-06-27T11:50:24.531 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.505+0000 7fa54102d700 1 -- 172.21.15.195:0/4107837487 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa53c0c2bf0 msgr2=0x7fa53c069690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:24.531 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.505+0000 7fa54102d700 1 --2- 172.21.15.195:0/4107837487 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa53c0c2bf0 0x7fa53c069690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:24.531 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.505+0000 7fa54102d700 1 -- 172.21.15.195:0/4107837487 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa53c0c4e30 msgr2=0x7fa53c0c7cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:24.531 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.505+0000 7fa54102d700 1 --2- 172.21.15.195:0/4107837487 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa53c0c4e30 0x7fa53c0c7cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:24.532 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.505+0000 7fa54102d700 1 -- 172.21.15.195:0/4107837487 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa530009350 con 0x7fa53c0c61e0 2022-06-27T11:50:24.532 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.505+0000 7fa53bfff700 1 --2- 172.21.15.195:0/4107837487 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa53c0c4e30 0x7fa53c0c7cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:50:24.532 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.505+0000 7fa54102d700 1 --2- 172.21.15.195:0/4107837487 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa53c0c61e0 0x7fa53c0c8230 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fa534009e50 tx=0x7fa53400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:50:24.532 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.505+0000 7fa538ff9700 1 -- 172.21.15.195:0/4107837487 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa53400caa0 con 0x7fa53c0c61e0 2022-06-27T11:50:24.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.505+0000 7fa538ff9700 1 -- 172.21.15.195:0/4107837487 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa53400f040 con 0x7fa53c0c61e0 2022-06-27T11:50:24.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.505+0000 7fa542a90700 1 -- 172.21.15.195:0/4107837487 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa53c166500 con 0x7fa53c0c61e0 2022-06-27T11:50:24.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.505+0000 7fa538ff9700 1 -- 172.21.15.195:0/4107837487 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa53400ce50 con 0x7fa53c0c61e0 2022-06-27T11:50:24.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.505+0000 7fa542a90700 1 -- 172.21.15.195:0/4107837487 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa53c166ac0 con 0x7fa53c0c61e0 2022-06-27T11:50:24.534 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.506+0000 7fa542a90700 1 -- 172.21.15.195:0/4107837487 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa53c1672a0 con 0x7fa53c0c61e0 2022-06-27T11:50:24.534 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.507+0000 7fa538ff9700 1 -- 172.21.15.195:0/4107837487 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fa534014690 con 0x7fa53c0c61e0 2022-06-27T11:50:24.534 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.507+0000 7fa538ff9700 1 --2- 172.21.15.195:0/4107837487 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa524063d00 0x7fa5240661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:24.535 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.507+0000 7fa538ff9700 1 -- 172.21.15.195:0/4107837487 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6202+0+0 (secure 0 0 0) 0x7fa534081720 con 0x7fa53c0c61e0 2022-06-27T11:50:24.535 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.507+0000 7fa54082c700 1 --2- 172.21.15.195:0/4107837487 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa524063d00 0x7fa5240661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:24.535 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.508+0000 7fa54082c700 1 --2- 172.21.15.195:0/4107837487 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa524063d00 0x7fa5240661b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa530004000 tx=0x7fa530015000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:50:24.536 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.511+0000 7fa538ff9700 1 -- 172.21.15.195:0/4107837487 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fa534050280 con 0x7fa53c0c61e0 2022-06-27T11:50:24.733 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.734+0000 7fa542a90700 1 -- 172.21.15.195:0/4107837487 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa53c0722c0 con 0x7fa53c0c61e0 2022-06-27T11:50:24.734 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.735+0000 7fa538ff9700 1 -- 172.21.15.195:0/4107837487 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+5370 (secure 0 0 0) 0x7fa534018020 con 0x7fa53c0c61e0 2022-06-27T11:50:24.735 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:50:24.735 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:50:18.020149+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"b","rank":0,"incarnation":9,"state":"up:clientreplay","state_seq":40,"addr":"172.21.15.202:6833/864126677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":864126677},{"type":"v1","addr":"172.21.15.202:6833","nonce":864126677}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:50:24.737 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.736+0000 7fa542a90700 1 -- 172.21.15.195:0/4107837487 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa524063d00 msgr2=0x7fa5240661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:24.738 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.736+0000 7fa542a90700 1 --2- 172.21.15.195:0/4107837487 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa524063d00 0x7fa5240661b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa530004000 tx=0x7fa530015000 comp rx=0 tx=0).stop 2022-06-27T11:50:24.738 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.736+0000 7fa542a90700 1 -- 172.21.15.195:0/4107837487 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa53c0c61e0 msgr2=0x7fa53c0c8230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:24.738 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.736+0000 7fa542a90700 1 --2- 172.21.15.195:0/4107837487 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa53c0c61e0 0x7fa53c0c8230 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fa534009e50 tx=0x7fa53400b040 comp rx=0 tx=0).stop 2022-06-27T11:50:24.739 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.737+0000 7fa542a90700 1 -- 172.21.15.195:0/4107837487 shutdown_connections 2022-06-27T11:50:24.739 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.737+0000 7fa542a90700 1 --2- 172.21.15.195:0/4107837487 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa524063d00 0x7fa5240661b0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:24.740 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.737+0000 7fa542a90700 1 --2- 172.21.15.195:0/4107837487 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa53c0c61e0 0x7fa53c0c8230 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:24.740 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.737+0000 7fa542a90700 1 --2- 172.21.15.195:0/4107837487 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa53c0c2bf0 0x7fa53c069690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:24.741 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.737+0000 7fa542a90700 1 --2- 172.21.15.195:0/4107837487 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa53c0c4e30 0x7fa53c0c7cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:24.741 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.737+0000 7fa542a90700 1 -- 172.21.15.195:0/4107837487 >> 172.21.15.195:0/4107837487 conn(0x7fa53c0c9b00 msgr2=0x7fa53c0c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:50:24.741 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.737+0000 7fa542a90700 1 -- 172.21.15.195:0/4107837487 shutdown_connections 2022-06-27T11:50:24.742 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:24.737+0000 7fa542a90700 1 -- 172.21.15.195:0/4107837487 wait complete. 2022-06-27T11:50:24.742 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 12 2022-06-27T11:50:24.751 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 12, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-06-27T11:47:52.358095+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 12, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4287": { "addr": "172.21.15.202:6833/864126677", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6832", "nonce": 864126677, "type": "v2" }, { "addr": "172.21.15.202:6833", "nonce": 864126677, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4287, "incarnation": 9, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:clientreplay", "state_seq": 40 } }, "last_failure": 0, "last_failure_osd_epoch": 26, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-06-27T11:50:18.020149+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4287 } } } ], "standbys": [ { "addr": "172.21.15.202:6837/3305006841", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6836", "nonce": 3305006841, "type": "v2" }, { "addr": "172.21.15.202:6837", "nonce": 3305006841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6835/2060709822", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6834", "nonce": 2060709822, "type": "v2" }, { "addr": "172.21.15.195:6835", "nonce": 2060709822, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6838/1598751652", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6836", "nonce": 1598751652, "type": "v2" }, { "addr": "172.21.15.195:6838", "nonce": 1598751652, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4275, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6839/1313800661", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6837", "nonce": 1313800661, "type": "v2" }, { "addr": "172.21.15.195:6839", "nonce": 1313800661, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4278, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-06-27T11:50:26.548 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T11:50:26.553 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T11:50:26.753 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:50:26.887 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:26.888+0000 7f84af7e9700 1 Processor -- start 2022-06-27T11:50:26.888 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:26.888+0000 7f84af7e9700 1 -- start start 2022-06-27T11:50:26.905 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:26.889+0000 7f84af7e9700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f84a806fef0 0x7f84a81570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:26.905 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:26.889+0000 7f84af7e9700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f84a80c34a0 0x7f84a80c2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:26.906 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:26.889+0000 7f84af7e9700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84a806fba0 con 0x7f84a806e850 2022-06-27T11:50:26.906 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:26.889+0000 7f84af7e9700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84a81575e0 con 0x7f84a806fef0 2022-06-27T11:50:26.907 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:26.889+0000 7f84af7e9700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84a806f5e0 con 0x7f84a80c34a0 2022-06-27T11:50:26.908 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:26.889+0000 7f84acd84700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f84a806fef0 0x7f84a81570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:26.908 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:26.889+0000 7f84acd84700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f84a806fef0 0x7f84a81570a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:42810/0 (socket says 172.21.15.195:42810) 2022-06-27T11:50:26.909 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:26.889+0000 7f84acd84700 1 -- 172.21.15.195:0/3465899049 learned_addr learned my addr 172.21.15.195:0/3465899049 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:50:26.909 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:26.889+0000 7f84add86700 1 --2- 172.21.15.195:0/3465899049 >> v2:172.21.15.202:3300/0 conn(0x7f84a80c34a0 0x7f84a80c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:26.910 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:26.889+0000 7f84acd84700 1 -- 172.21.15.195:0/3465899049 >> v2:172.21.15.202:3300/0 conn(0x7f84a80c34a0 msgr2=0x7f84a80c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:50:26.910 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:26.889+0000 7f84acd84700 1 --2- 172.21.15.195:0/3465899049 >> v2:172.21.15.202:3300/0 conn(0x7f84a80c34a0 0x7f84a80c2920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:26.911 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:26.889+0000 7f84acd84700 1 -- 172.21.15.195:0/3465899049 >> v1:172.21.15.202:6789/0 conn(0x7f84a806e850 legacy=0x7f84a8069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:26.911 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:26.889+0000 7f84acd84700 1 -- 172.21.15.195:0/3465899049 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84a806f8c0 con 0x7f84a806fef0 2022-06-27T11:50:26.911 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:26.889+0000 7f84add86700 1 --2- 172.21.15.195:0/3465899049 >> v2:172.21.15.202:3300/0 conn(0x7f84a80c34a0 0x7f84a80c2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T11:50:26.912 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:26.890+0000 7f84acd84700 1 --2- 172.21.15.195:0/3465899049 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f84a806fef0 0x7f84a81570a0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f84a80c1420 tx=0x7f84a4009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9dd6f6e0a0a4ba3 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:50:26.912 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:26.890+0000 7f8497fff700 1 -- 172.21.15.195:0/3465899049 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a400b6d0 con 0x7f84a806fef0 2022-06-27T11:50:26.915 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:26.890+0000 7f8497fff700 1 -- 172.21.15.195:0/3465899049 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84a400b830 con 0x7f84a806fef0 2022-06-27T11:50:26.915 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:26.890+0000 7f8497fff700 1 -- 172.21.15.195:0/3465899049 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a400bbd0 con 0x7f84a806fef0 2022-06-27T11:50:26.916 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:26.890+0000 7f84af7e9700 1 -- 172.21.15.195:0/3465899049 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f84a806fef0 msgr2=0x7f84a81570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:26.917 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:26.890+0000 7f84af7e9700 1 --2- 172.21.15.195:0/3465899049 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f84a806fef0 0x7f84a81570a0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f84a80c1420 tx=0x7f84a4009120 comp rx=0 tx=0).stop 2022-06-27T11:50:26.917 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:26.890+0000 7f84af7e9700 1 -- 172.21.15.195:0/3465899049 shutdown_connections 2022-06-27T11:50:26.918 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:26.890+0000 7f84af7e9700 1 --2- 172.21.15.195:0/3465899049 >> v2:172.21.15.202:3300/0 conn(0x7f84a80c34a0 0x7f84a80c2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:26.918 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:26.890+0000 7f84af7e9700 1 --2- 172.21.15.195:0/3465899049 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f84a806fef0 0x7f84a81570a0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:26.919 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:26.890+0000 7f84af7e9700 1 -- 172.21.15.195:0/3465899049 >> 172.21.15.195:0/3465899049 conn(0x7f84a80c6350 msgr2=0x7f84a80c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:50:26.919 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:26.890+0000 7f84af7e9700 1 -- 172.21.15.195:0/3465899049 shutdown_connections 2022-06-27T11:50:26.919 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:26.890+0000 7f84af7e9700 1 -- 172.21.15.195:0/3465899049 wait complete. 2022-06-27T11:50:26.920 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:26.891+0000 7f84af7e9700 1 Processor -- start 2022-06-27T11:50:26.921 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:26.892+0000 7f84af7e9700 1 -- start start 2022-06-27T11:50:26.921 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:26.892+0000 7f84af7e9700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f84a806e850 0x7f84a8167df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:26.922 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:26.892+0000 7f84af7e9700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f84a80c34a0 0x7f84a80bfbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:26.922 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:26.892+0000 7f84af7e9700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f84a8168330 0x7f84a80c0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:26.923 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:26.892+0000 7f84af7e9700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a80c0670 con 0x7f84a80c34a0 2022-06-27T11:50:26.923 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:26.892+0000 7f84af7e9700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a8156ff0 con 0x7f84a8168330 2022-06-27T11:50:26.924 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:26.892+0000 7f84af7e9700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84a8151350 con 0x7f84a806e850 2022-06-27T11:50:26.925 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:26.893+0000 7f84acd84700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f84a80c34a0 0x7f84a80bfbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:26.926 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:26.893+0000 7f84acd84700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f84a80c34a0 0x7f84a80bfbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:42816/0 (socket says 172.21.15.195:42816) 2022-06-27T11:50:26.927 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:26.893+0000 7f84acd84700 1 -- 172.21.15.195:0/4064516967 learned_addr learned my addr 172.21.15.195:0/4064516967 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:50:26.927 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:26.893+0000 7f84add86700 1 --2- 172.21.15.195:0/4064516967 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f84a8168330 0x7f84a80c0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:26.927 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:26.893+0000 7f84ad585700 1 --2- 172.21.15.195:0/4064516967 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f84a806e850 0x7f84a8167df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:26.928 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:26.893+0000 7f84acd84700 1 -- 172.21.15.195:0/4064516967 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f84a806e850 msgr2=0x7f84a8167df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:26.928 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:26.893+0000 7f84acd84700 1 --2- 172.21.15.195:0/4064516967 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f84a806e850 0x7f84a8167df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:26.929 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:26.893+0000 7f84acd84700 1 -- 172.21.15.195:0/4064516967 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f84a8168330 msgr2=0x7f84a80c0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:26.929 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:26.893+0000 7f84acd84700 1 --2- 172.21.15.195:0/4064516967 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f84a8168330 0x7f84a80c0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:26.930 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:26.893+0000 7f84acd84700 1 -- 172.21.15.195:0/4064516967 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84a40093a0 con 0x7f84a80c34a0 2022-06-27T11:50:26.930 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:26.893+0000 7f84add86700 1 --2- 172.21.15.195:0/4064516967 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f84a8168330 0x7f84a80c0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T11:50:26.931 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:26.893+0000 7f84acd84700 1 --2- 172.21.15.195:0/4064516967 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f84a80c34a0 0x7f84a80bfbf0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f84a4009f80 tx=0x7f84a4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:50:26.931 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:26.893+0000 7f84957fa700 1 -- 172.21.15.195:0/4064516967 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a4003800 con 0x7f84a80c34a0 2022-06-27T11:50:26.932 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:26.893+0000 7f84957fa700 1 -- 172.21.15.195:0/4064516967 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84a4003960 con 0x7f84a80c34a0 2022-06-27T11:50:26.932 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:26.893+0000 7f84957fa700 1 -- 172.21.15.195:0/4064516967 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a4003d20 con 0x7f84a80c34a0 2022-06-27T11:50:26.933 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:26.893+0000 7f84af7e9700 1 -- 172.21.15.195:0/4064516967 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84a8155f50 con 0x7f84a80c34a0 2022-06-27T11:50:26.933 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:26.893+0000 7f84af7e9700 1 -- 172.21.15.195:0/4064516967 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84a8156510 con 0x7f84a80c34a0 2022-06-27T11:50:26.934 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:26.894+0000 7f84af7e9700 1 -- 172.21.15.195:0/4064516967 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84a80c6fb0 con 0x7f84a80c34a0 2022-06-27T11:50:26.934 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:26.895+0000 7f84957fa700 1 -- 172.21.15.195:0/4064516967 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f84a400f050 con 0x7f84a80c34a0 2022-06-27T11:50:26.935 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:26.895+0000 7f84957fa700 1 --2- 172.21.15.195:0/4064516967 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8488063d00 0x7f84880661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:26.935 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:26.895+0000 7f84957fa700 1 -- 172.21.15.195:0/4064516967 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6202+0+0 (secure 0 0 0) 0x7f84a40820f0 con 0x7f84a80c34a0 2022-06-27T11:50:26.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:26.895+0000 7f84ad585700 1 --2- 172.21.15.195:0/4064516967 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8488063d00 0x7f84880661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:26.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:26.895+0000 7f84ad585700 1 --2- 172.21.15.195:0/4064516967 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8488063d00 0x7f84880661b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f8498001670 tx=0x7f8498009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:50:26.937 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:26.899+0000 7f84957fa700 1 -- 172.21.15.195:0/4064516967 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f84a4050c50 con 0x7f84a80c34a0 2022-06-27T11:50:27.126 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:27.125+0000 7f84af7e9700 1 -- 172.21.15.195:0/4064516967 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f84a80722c0 con 0x7f84a80c34a0 2022-06-27T11:50:27.126 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:27.126+0000 7f84957fa700 1 -- 172.21.15.195:0/4064516967 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+5364 (secure 0 0 0) 0x7f84a4015090 con 0x7f84a80c34a0 2022-06-27T11:50:27.127 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:50:27.128 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:50:26.047919+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"b","rank":0,"incarnation":9,"state":"up:active","state_seq":43,"addr":"172.21.15.202:6833/864126677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":864126677},{"type":"v1","addr":"172.21.15.202:6833","nonce":864126677}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:50:27.130 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:27.128+0000 7f84af7e9700 1 -- 172.21.15.195:0/4064516967 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8488063d00 msgr2=0x7f84880661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:27.131 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:27.128+0000 7f84af7e9700 1 --2- 172.21.15.195:0/4064516967 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8488063d00 0x7f84880661b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f8498001670 tx=0x7f8498009040 comp rx=0 tx=0).stop 2022-06-27T11:50:27.132 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:27.128+0000 7f84af7e9700 1 -- 172.21.15.195:0/4064516967 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f84a80c34a0 msgr2=0x7f84a80bfbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:27.132 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:27.128+0000 7f84af7e9700 1 --2- 172.21.15.195:0/4064516967 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f84a80c34a0 0x7f84a80bfbf0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f84a4009f80 tx=0x7f84a4017000 comp rx=0 tx=0).stop 2022-06-27T11:50:27.133 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:27.128+0000 7f84af7e9700 1 -- 172.21.15.195:0/4064516967 shutdown_connections 2022-06-27T11:50:27.133 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:27.128+0000 7f84af7e9700 1 --2- 172.21.15.195:0/4064516967 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8488063d00 0x7f84880661b0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:27.134 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:27.128+0000 7f84af7e9700 1 --2- 172.21.15.195:0/4064516967 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f84a8168330 0x7f84a80c0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:27.134 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:27.128+0000 7f84af7e9700 1 --2- 172.21.15.195:0/4064516967 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f84a806e850 0x7f84a8167df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:27.135 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:27.128+0000 7f84af7e9700 1 --2- 172.21.15.195:0/4064516967 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f84a80c34a0 0x7f84a80bfbf0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:27.136 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:27.128+0000 7f84af7e9700 1 -- 172.21.15.195:0/4064516967 >> 172.21.15.195:0/4064516967 conn(0x7f84a80c6350 msgr2=0x7f84a81501b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:50:27.136 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:27.128+0000 7f84af7e9700 1 -- 172.21.15.195:0/4064516967 shutdown_connections 2022-06-27T11:50:27.137 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:27.128+0000 7f84af7e9700 1 -- 172.21.15.195:0/4064516967 wait complete. 2022-06-27T11:50:27.137 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 13 2022-06-27T11:50:27.140 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=4293 2022-06-27T11:50:27.140 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 1 secs before reviving mds.d 2022-06-27T11:50:28.142 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.d 2022-06-27T11:50:28.143 INFO:tasks.ceph.mds.d:Restarting daemon 2022-06-27T11:50:28.143 DEBUG:teuthology.orchestra.run.smithi202:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2022-06-27T11:50:28.147 INFO:tasks.ceph.mds.d:Started 2022-06-27T11:50:28.147 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:50:28.235 INFO:tasks.ceph.mds.d.smithi202.stdout:starting mds.d at 2022-06-27T11:50:28.289 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.282+0000 7f2984fe9700 1 Processor -- start 2022-06-27T11:50:28.290 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.282+0000 7f2984fe9700 1 -- start start 2022-06-27T11:50:28.290 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.283+0000 7f2984fe9700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f298006a450 0x7f2980063af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:28.291 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.283+0000 7f2984fe9700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f29801508d0 0x7f29800bbf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:28.291 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.283+0000 7f2984fe9700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2980064030 con 0x7f2980150500 2022-06-27T11:50:28.293 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.283+0000 7f2984fe9700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29800c4160 con 0x7f29801508d0 2022-06-27T11:50:28.298 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.283+0000 7f2984fe9700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29800c4440 con 0x7f298006a450 2022-06-27T11:50:28.299 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.283+0000 7f297ed9d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f29801508d0 0x7f29800bbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:28.299 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.283+0000 7f297ed9d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f29801508d0 0x7f29800bbf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:42826/0 (socket says 172.21.15.195:42826) 2022-06-27T11:50:28.300 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.283+0000 7f297ed9d700 1 -- 172.21.15.195:0/1005319063 learned_addr learned my addr 172.21.15.195:0/1005319063 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:50:28.300 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.283+0000 7f297dd9b700 1 --2- 172.21.15.195:0/1005319063 >> v2:172.21.15.202:3300/0 conn(0x7f298006a450 0x7f2980063af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:28.301 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.283+0000 7f297ed9d700 1 -- 172.21.15.195:0/1005319063 >> v2:172.21.15.202:3300/0 conn(0x7f298006a450 msgr2=0x7f2980063af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:50:28.301 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.283+0000 7f297ed9d700 1 --2- 172.21.15.195:0/1005319063 >> v2:172.21.15.202:3300/0 conn(0x7f298006a450 0x7f2980063af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:28.302 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.283+0000 7f297ed9d700 1 -- 172.21.15.195:0/1005319063 >> v1:172.21.15.202:6789/0 conn(0x7f2980150500 legacy=0x7f29800c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:28.303 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.283+0000 7f297ed9d700 1 -- 172.21.15.195:0/1005319063 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f298015a800 con 0x7f29801508d0 2022-06-27T11:50:28.303 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.283+0000 7f297dd9b700 1 --2- 172.21.15.195:0/1005319063 >> v2:172.21.15.202:3300/0 conn(0x7f298006a450 0x7f2980063af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T11:50:28.304 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.283+0000 7f297ed9d700 1 --2- 172.21.15.195:0/1005319063 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f29801508d0 0x7f29800bbf50 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f298006ce70 tx=0x7f2978009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f1ea356cc727660 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:50:28.304 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.284+0000 7f297d59a700 1 -- 172.21.15.195:0/1005319063 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f297800e070 con 0x7f29801508d0 2022-06-27T11:50:28.305 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.284+0000 7f297d59a700 1 -- 172.21.15.195:0/1005319063 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f297800ba50 con 0x7f29801508d0 2022-06-27T11:50:28.305 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.284+0000 7f297d59a700 1 -- 172.21.15.195:0/1005319063 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29780092e0 con 0x7f29801508d0 2022-06-27T11:50:28.306 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.284+0000 7f2984fe9700 1 -- 172.21.15.195:0/1005319063 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f29801508d0 msgr2=0x7f29800bbf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:28.306 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.284+0000 7f2984fe9700 1 --2- 172.21.15.195:0/1005319063 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f29801508d0 0x7f29800bbf50 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f298006ce70 tx=0x7f2978009840 comp rx=0 tx=0).stop 2022-06-27T11:50:28.307 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.284+0000 7f2984fe9700 1 -- 172.21.15.195:0/1005319063 shutdown_connections 2022-06-27T11:50:28.307 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.284+0000 7f2984fe9700 1 --2- 172.21.15.195:0/1005319063 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f29801508d0 0x7f29800bbf50 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:28.308 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.284+0000 7f2984fe9700 1 --2- 172.21.15.195:0/1005319063 >> v2:172.21.15.202:3300/0 conn(0x7f298006a450 0x7f2980063af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:28.308 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.284+0000 7f2984fe9700 1 -- 172.21.15.195:0/1005319063 >> 172.21.15.195:0/1005319063 conn(0x7f29800c9b00 msgr2=0x7f29800c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:50:28.309 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.284+0000 7f2984fe9700 1 -- 172.21.15.195:0/1005319063 shutdown_connections 2022-06-27T11:50:28.309 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.284+0000 7f2984fe9700 1 -- 172.21.15.195:0/1005319063 wait complete. 2022-06-27T11:50:28.310 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.285+0000 7f2984fe9700 1 Processor -- start 2022-06-27T11:50:28.310 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.285+0000 7f2984fe9700 1 -- start start 2022-06-27T11:50:28.311 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.286+0000 7f2984fe9700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f298006a450 0x7f2980169e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:28.311 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.286+0000 7f2984fe9700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2980150500 0x7f2980167310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:28.312 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.286+0000 7f2984fe9700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f29801508d0 0x7f2980152f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:28.312 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.286+0000 7f2984fe9700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2980068450 con 0x7f2980150500 2022-06-27T11:50:28.313 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.286+0000 7f2984fe9700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2980166850 con 0x7f29801508d0 2022-06-27T11:50:28.313 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.286+0000 7f2984fe9700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2980158c50 con 0x7f298006a450 2022-06-27T11:50:28.314 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.286+0000 7f297dd9b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2980150500 0x7f2980167310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:28.314 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.286+0000 7f297dd9b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2980150500 0x7f2980167310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:42830/0 (socket says 172.21.15.195:42830) 2022-06-27T11:50:28.315 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.286+0000 7f297dd9b700 1 -- 172.21.15.195:0/1613444381 learned_addr learned my addr 172.21.15.195:0/1613444381 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:50:28.315 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.287+0000 7f297ed9d700 1 --2- 172.21.15.195:0/1613444381 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f29801508d0 0x7f2980152f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:28.316 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.287+0000 7f297e59c700 1 --2- 172.21.15.195:0/1613444381 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f298006a450 0x7f2980169e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:28.316 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.287+0000 7f297dd9b700 1 -- 172.21.15.195:0/1613444381 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f298006a450 msgr2=0x7f2980169e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:28.317 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.287+0000 7f297dd9b700 1 --2- 172.21.15.195:0/1613444381 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f298006a450 0x7f2980169e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:28.317 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.287+0000 7f297dd9b700 1 -- 172.21.15.195:0/1613444381 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f29801508d0 msgr2=0x7f2980152f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:28.318 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.287+0000 7f297dd9b700 1 --2- 172.21.15.195:0/1613444381 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f29801508d0 0x7f2980152f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:28.318 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.287+0000 7f297dd9b700 1 -- 172.21.15.195:0/1613444381 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2978009ae0 con 0x7f2980150500 2022-06-27T11:50:28.319 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.287+0000 7f297dd9b700 1 --2- 172.21.15.195:0/1613444381 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2980150500 0x7f2980167310 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f297400d810 tx=0x7f2974008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:50:28.319 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.287+0000 7f296e7fc700 1 -- 172.21.15.195:0/1613444381 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2974004030 con 0x7f2980150500 2022-06-27T11:50:28.320 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.287+0000 7f296e7fc700 1 -- 172.21.15.195:0/1613444381 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2974004190 con 0x7f2980150500 2022-06-27T11:50:28.320 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.287+0000 7f2984fe9700 1 -- 172.21.15.195:0/1613444381 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2980158f30 con 0x7f2980150500 2022-06-27T11:50:28.321 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.287+0000 7f296e7fc700 1 -- 172.21.15.195:0/1613444381 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29740125f0 con 0x7f2980150500 2022-06-27T11:50:28.322 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.287+0000 7f2984fe9700 1 -- 172.21.15.195:0/1613444381 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2980166b10 con 0x7f2980150500 2022-06-27T11:50:28.324 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.288+0000 7f2984fe9700 1 -- 172.21.15.195:0/1613444381 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2980159450 con 0x7f2980150500 2022-06-27T11:50:28.325 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.288+0000 7f296e7fc700 1 -- 172.21.15.195:0/1613444381 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f2974012a20 con 0x7f2980150500 2022-06-27T11:50:28.326 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.289+0000 7f296e7fc700 1 --2- 172.21.15.195:0/1613444381 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f295c063d00 0x7f295c0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:28.326 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.289+0000 7f296e7fc700 1 -- 172.21.15.195:0/1613444381 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6202+0+0 (secure 0 0 0) 0x7f2974080c80 con 0x7f2980150500 2022-06-27T11:50:28.327 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.289+0000 7f297e59c700 1 --2- 172.21.15.195:0/1613444381 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f295c063d00 0x7f295c0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:28.327 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.289+0000 7f297e59c700 1 --2- 172.21.15.195:0/1613444381 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f295c063d00 0x7f295c0661b0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f29700016f0 tx=0x7f2970009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:50:28.328 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.293+0000 7f296e7fc700 1 -- 172.21.15.195:0/1613444381 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f297404f7e0 con 0x7f2980150500 2022-06-27T11:50:28.524 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.525+0000 7f2984fe9700 1 -- 172.21.15.195:0/1613444381 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f29800c8290 con 0x7f2980150500 2022-06-27T11:50:28.525 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.526+0000 7f296e7fc700 1 -- 172.21.15.195:0/1613444381 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+5364 (secure 0 0 0) 0x7f2974052e00 con 0x7f2980150500 2022-06-27T11:50:28.526 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:50:28.526 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:50:26.047919+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"b","rank":0,"incarnation":9,"state":"up:active","state_seq":43,"addr":"172.21.15.202:6833/864126677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":864126677},{"type":"v1","addr":"172.21.15.202:6833","nonce":864126677}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:50:28.529 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.527+0000 7f2984fe9700 1 -- 172.21.15.195:0/1613444381 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f295c063d00 msgr2=0x7f295c0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:28.529 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.527+0000 7f2984fe9700 1 --2- 172.21.15.195:0/1613444381 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f295c063d00 0x7f295c0661b0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f29700016f0 tx=0x7f2970009040 comp rx=0 tx=0).stop 2022-06-27T11:50:28.530 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.527+0000 7f2984fe9700 1 -- 172.21.15.195:0/1613444381 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2980150500 msgr2=0x7f2980167310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:28.530 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.527+0000 7f2984fe9700 1 --2- 172.21.15.195:0/1613444381 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2980150500 0x7f2980167310 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f297400d810 tx=0x7f2974008040 comp rx=0 tx=0).stop 2022-06-27T11:50:28.531 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.527+0000 7f2984fe9700 1 -- 172.21.15.195:0/1613444381 shutdown_connections 2022-06-27T11:50:28.531 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.527+0000 7f2984fe9700 1 --2- 172.21.15.195:0/1613444381 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f295c063d00 0x7f295c0661b0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:28.532 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.527+0000 7f2984fe9700 1 --2- 172.21.15.195:0/1613444381 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f29801508d0 0x7f2980152f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:28.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.527+0000 7f2984fe9700 1 --2- 172.21.15.195:0/1613444381 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f298006a450 0x7f2980169e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:28.534 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.527+0000 7f2984fe9700 1 --2- 172.21.15.195:0/1613444381 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2980150500 0x7f2980167310 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:28.534 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.528+0000 7f2984fe9700 1 -- 172.21.15.195:0/1613444381 >> 172.21.15.195:0/1613444381 conn(0x7f29800c9b00 msgr2=0x7f298006ea40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:50:28.535 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.528+0000 7f2984fe9700 1 -- 172.21.15.195:0/1613444381 shutdown_connections 2022-06-27T11:50:28.535 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:28.528+0000 7f2984fe9700 1 -- 172.21.15.195:0/1613444381 wait complete. 2022-06-27T11:50:28.536 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 13 2022-06-27T11:50:28.539 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2022-06-27T11:50:30.541 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:50:30.679 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.676+0000 7feb9b18c700 1 Processor -- start 2022-06-27T11:50:30.679 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.676+0000 7feb9b18c700 1 -- start start 2022-06-27T11:50:30.680 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.677+0000 7feb9b18c700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7feb9414e860 0x7feb940c3840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:30.680 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.677+0000 7feb9b18c700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7feb940c3d80 0x7feb94157890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:30.680 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.677+0000 7feb9b18c700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb940c33b0 con 0x7feb9414e490 2022-06-27T11:50:30.681 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.677+0000 7feb9b18c700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb9406f220 con 0x7feb9414e860 2022-06-27T11:50:30.681 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.677+0000 7feb9b18c700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb941542c0 con 0x7feb940c3d80 2022-06-27T11:50:30.682 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.677+0000 7feb99729700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7feb940c3d80 0x7feb94157890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:30.682 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.677+0000 7feb99729700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7feb940c3d80 0x7feb94157890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3301/0 says I am v2:172.21.15.195:35600/0 (socket says 172.21.15.195:35600) 2022-06-27T11:50:30.682 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.677+0000 7feb93fff700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7feb9414e860 0x7feb940c3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:30.683 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.677+0000 7feb99729700 1 -- 172.21.15.195:0/4060625060 learned_addr learned my addr 172.21.15.195:0/4060625060 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:50:30.683 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.677+0000 7feb98f28700 1 --1- >> v1:172.21.15.202:6789/0 conn(0x7feb9414e490 0x7feb9406cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.202:6789/0 says I am v1:172.21.15.195:50888/0 (socket says 172.21.15.195:50888) 2022-06-27T11:50:30.683 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.677+0000 7feb937fe700 1 -- 172.21.15.195:0/4060625060 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1749192097 0 0) 0x7feb940c33b0 con 0x7feb9414e490 2022-06-27T11:50:30.683 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.677+0000 7feb937fe700 1 -- 172.21.15.195:0/4060625060 --> v1:172.21.15.202:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb78003430 con 0x7feb9414e490 2022-06-27T11:50:30.684 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.678+0000 7feb99729700 1 -- 172.21.15.195:0/4060625060 >> v2:172.21.15.202:3300/0 conn(0x7feb9414e860 msgr2=0x7feb940c3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:50:30.685 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.678+0000 7feb99729700 1 --2- 172.21.15.195:0/4060625060 >> v2:172.21.15.202:3300/0 conn(0x7feb9414e860 0x7feb940c3840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:30.687 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.678+0000 7feb99729700 1 -- 172.21.15.195:0/4060625060 >> v1:172.21.15.202:6789/0 conn(0x7feb9414e490 legacy=0x7feb9406cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:30.687 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.678+0000 7feb99729700 1 -- 172.21.15.195:0/4060625060 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb94154560 con 0x7feb940c3d80 2022-06-27T11:50:30.688 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.678+0000 7feb93fff700 1 --2- 172.21.15.195:0/4060625060 >> v2:172.21.15.202:3300/0 conn(0x7feb9414e860 0x7feb940c3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:50:30.688 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.678+0000 7feb99729700 1 --2- 172.21.15.195:0/4060625060 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7feb940c3d80 0x7feb94157890 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7feb8c009f20 tx=0x7feb8c009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f87197cdf19092f server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:50:30.689 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.678+0000 7feb937fe700 1 -- 172.21.15.195:0/4060625060 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb8c00e070 con 0x7feb940c3d80 2022-06-27T11:50:30.689 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.678+0000 7feb937fe700 1 -- 172.21.15.195:0/4060625060 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb8c00b970 con 0x7feb940c3d80 2022-06-27T11:50:30.690 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.678+0000 7feb937fe700 1 -- 172.21.15.195:0/4060625060 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb8c0092e0 con 0x7feb940c3d80 2022-06-27T11:50:30.691 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.678+0000 7feb9b18c700 1 -- 172.21.15.195:0/4060625060 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7feb940c3d80 msgr2=0x7feb94157890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:30.692 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.678+0000 7feb9b18c700 1 --2- 172.21.15.195:0/4060625060 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7feb940c3d80 0x7feb94157890 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7feb8c009f20 tx=0x7feb8c009840 comp rx=0 tx=0).stop 2022-06-27T11:50:30.692 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.678+0000 7feb9b18c700 1 -- 172.21.15.195:0/4060625060 shutdown_connections 2022-06-27T11:50:30.692 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.678+0000 7feb9b18c700 1 --2- 172.21.15.195:0/4060625060 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7feb940c3d80 0x7feb94157890 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:30.693 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.678+0000 7feb9b18c700 1 --2- 172.21.15.195:0/4060625060 >> v2:172.21.15.202:3300/0 conn(0x7feb9414e860 0x7feb940c3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:30.693 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.678+0000 7feb9b18c700 1 -- 172.21.15.195:0/4060625060 >> 172.21.15.195:0/4060625060 conn(0x7feb94063af0 msgr2=0x7feb94063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:50:30.693 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.678+0000 7feb9b18c700 1 -- 172.21.15.195:0/4060625060 shutdown_connections 2022-06-27T11:50:30.694 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.678+0000 7feb9b18c700 1 -- 172.21.15.195:0/4060625060 wait complete. 2022-06-27T11:50:30.694 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.680+0000 7feb9b18c700 1 Processor -- start 2022-06-27T11:50:30.694 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.680+0000 7feb9b18c700 1 -- start start 2022-06-27T11:50:30.697 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.680+0000 7feb9b18c700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7feb940c3d80 0x7feb94167560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:30.699 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.680+0000 7feb9b18c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7feb9414e490 0x7feb9414a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:30.700 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.680+0000 7feb9b18c700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7feb9414e860 0x7feb940c8880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:30.700 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.680+0000 7feb9b18c700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb940c7f50 con 0x7feb9414e490 2022-06-27T11:50:30.700 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.680+0000 7feb9b18c700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb94164750 con 0x7feb940c3d80 2022-06-27T11:50:30.700 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.680+0000 7feb9b18c700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb9414bc50 con 0x7feb9414e860 2022-06-27T11:50:30.701 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.681+0000 7feb93fff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7feb9414e490 0x7feb9414a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:30.701 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.681+0000 7feb93fff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7feb9414e490 0x7feb9414a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:42848/0 (socket says 172.21.15.195:42848) 2022-06-27T11:50:30.701 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.681+0000 7feb93fff700 1 -- 172.21.15.195:0/3189633576 learned_addr learned my addr 172.21.15.195:0/3189633576 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:50:30.701 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.681+0000 7feb99729700 1 --2- 172.21.15.195:0/3189633576 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7feb9414e860 0x7feb940c8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:30.702 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.681+0000 7feb98f28700 1 --2- 172.21.15.195:0/3189633576 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7feb940c3d80 0x7feb94167560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:30.702 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.681+0000 7feb93fff700 1 -- 172.21.15.195:0/3189633576 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7feb9414e860 msgr2=0x7feb940c8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:30.702 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.681+0000 7feb93fff700 1 --2- 172.21.15.195:0/3189633576 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7feb9414e860 0x7feb940c8880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:30.702 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.681+0000 7feb93fff700 1 -- 172.21.15.195:0/3189633576 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7feb940c3d80 msgr2=0x7feb94167560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:30.703 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.681+0000 7feb93fff700 1 --2- 172.21.15.195:0/3189633576 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7feb940c3d80 0x7feb94167560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:30.703 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.681+0000 7feb93fff700 1 -- 172.21.15.195:0/3189633576 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb800092e0 con 0x7feb9414e490 2022-06-27T11:50:30.703 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.681+0000 7feb93fff700 1 --2- 172.21.15.195:0/3189633576 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7feb9414e490 0x7feb9414a500 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7feb8000ea50 tx=0x7feb80009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:50:30.703 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.681+0000 7feb90ff9700 1 -- 172.21.15.195:0/3189633576 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb8000c850 con 0x7feb9414e490 2022-06-27T11:50:30.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.681+0000 7feb90ff9700 1 -- 172.21.15.195:0/3189633576 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb8000c9b0 con 0x7feb9414e490 2022-06-27T11:50:30.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.681+0000 7feb90ff9700 1 -- 172.21.15.195:0/3189633576 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb8000cd30 con 0x7feb9414e490 2022-06-27T11:50:30.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.681+0000 7feb9b18c700 1 -- 172.21.15.195:0/3189633576 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb8c009ae0 con 0x7feb9414e490 2022-06-27T11:50:30.705 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.681+0000 7feb9b18c700 1 -- 172.21.15.195:0/3189633576 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb94164100 con 0x7feb9414e490 2022-06-27T11:50:30.705 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.682+0000 7feb9b18c700 1 -- 172.21.15.195:0/3189633576 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb940722c0 con 0x7feb9414e490 2022-06-27T11:50:30.706 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.683+0000 7feb90ff9700 1 -- 172.21.15.195:0/3189633576 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7feb80019030 con 0x7feb9414e490 2022-06-27T11:50:30.706 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.683+0000 7feb90ff9700 1 --2- 172.21.15.195:0/3189633576 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7feb7c063d30 0x7feb7c0661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:30.707 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.683+0000 7feb90ff9700 1 -- 172.21.15.195:0/3189633576 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6202+0+0 (secure 0 0 0) 0x7feb80080a20 con 0x7feb9414e490 2022-06-27T11:50:30.708 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.683+0000 7feb98f28700 1 --2- 172.21.15.195:0/3189633576 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7feb7c063d30 0x7feb7c0661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:30.708 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.684+0000 7feb98f28700 1 --2- 172.21.15.195:0/3189633576 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7feb7c063d30 0x7feb7c0661e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7feb88007540 tx=0x7feb88000bc0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:50:30.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.687+0000 7feb90ff9700 1 -- 172.21.15.195:0/3189633576 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7feb8004f580 con 0x7feb9414e490 2022-06-27T11:50:30.913 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.913+0000 7feb9b18c700 1 -- 172.21.15.195:0/3189633576 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feb941644a0 con 0x7feb9414e490 2022-06-27T11:50:30.914 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.913+0000 7feb90ff9700 1 -- 172.21.15.195:0/3189633576 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+6124 (secure 0 0 0) 0x7feb80021090 con 0x7feb9414e490 2022-06-27T11:50:30.914 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:50:30.915 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4849,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/3271891211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":3271891211},{"type":"v1","addr":"172.21.15.202:6835","nonce":3271891211}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:50:26.047919+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"b","rank":0,"incarnation":9,"state":"up:active","state_seq":43,"addr":"172.21.15.202:6833/864126677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":864126677},{"type":"v1","addr":"172.21.15.202:6833","nonce":864126677}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:50:30.918 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.915+0000 7feb9b18c700 1 -- 172.21.15.195:0/3189633576 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7feb7c063d30 msgr2=0x7feb7c0661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:30.919 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.915+0000 7feb9b18c700 1 --2- 172.21.15.195:0/3189633576 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7feb7c063d30 0x7feb7c0661e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7feb88007540 tx=0x7feb88000bc0 comp rx=0 tx=0).stop 2022-06-27T11:50:30.919 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.915+0000 7feb9b18c700 1 -- 172.21.15.195:0/3189633576 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7feb9414e490 msgr2=0x7feb9414a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:30.920 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.915+0000 7feb9b18c700 1 --2- 172.21.15.195:0/3189633576 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7feb9414e490 0x7feb9414a500 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7feb8000ea50 tx=0x7feb80009060 comp rx=0 tx=0).stop 2022-06-27T11:50:30.920 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.915+0000 7feb9b18c700 1 -- 172.21.15.195:0/3189633576 shutdown_connections 2022-06-27T11:50:30.920 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.915+0000 7feb9b18c700 1 --2- 172.21.15.195:0/3189633576 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7feb7c063d30 0x7feb7c0661e0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:30.921 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.915+0000 7feb9b18c700 1 --2- 172.21.15.195:0/3189633576 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7feb9414e860 0x7feb940c8880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:30.921 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.915+0000 7feb9b18c700 1 --2- 172.21.15.195:0/3189633576 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7feb940c3d80 0x7feb94167560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:30.923 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.915+0000 7feb9b18c700 1 --2- 172.21.15.195:0/3189633576 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7feb9414e490 0x7feb9414a500 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:30.923 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.915+0000 7feb9b18c700 1 -- 172.21.15.195:0/3189633576 >> 172.21.15.195:0/3189633576 conn(0x7feb94063af0 msgr2=0x7feb9406cfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:50:30.924 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.915+0000 7feb9b18c700 1 -- 172.21.15.195:0/3189633576 shutdown_connections 2022-06-27T11:50:30.924 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:30.915+0000 7feb9b18c700 1 -- 172.21.15.195:0/3189633576 wait complete. 2022-06-27T11:50:30.924 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 14 2022-06-27T11:50:30.928 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d reported in up:standby state 2022-06-27T11:50:30.929 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 27 secs before thrashing 2022-06-27T11:50:31.794 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/113/14' -> 'copyofsnap1/test1/113/14' 2022-06-27T11:50:31.795 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/113/15' -> 'copyofsnap1/test1/113/15' 2022-06-27T11:50:31.795 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/113/16' -> 'copyofsnap1/test1/113/16' 2022-06-27T11:50:31.796 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/113/17' -> 'copyofsnap1/test1/113/17' 2022-06-27T11:50:31.796 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/113/18' -> 'copyofsnap1/test1/113/18' 2022-06-27T11:50:31.797 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/113/19' -> 'copyofsnap1/test1/113/19' 2022-06-27T11:50:31.797 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/113/20' -> 'copyofsnap1/test1/113/20' 2022-06-27T11:50:31.798 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/114' -> 'copyofsnap1/test1/114' 2022-06-27T11:50:31.806 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/114/10' -> 'copyofsnap1/test1/114/10' 2022-06-27T11:50:31.807 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/114/11' -> 'copyofsnap1/test1/114/11' 2022-06-27T11:50:31.807 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/114/12' -> 'copyofsnap1/test1/114/12' 2022-06-27T11:50:31.809 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/114/13' -> 'copyofsnap1/test1/114/13' 2022-06-27T11:50:31.810 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/114/14' -> 'copyofsnap1/test1/114/14' 2022-06-27T11:50:31.810 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/114/15' -> 'copyofsnap1/test1/114/15' 2022-06-27T11:50:31.810 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/114/16' -> 'copyofsnap1/test1/114/16' 2022-06-27T11:50:31.810 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/114/17' -> 'copyofsnap1/test1/114/17' 2022-06-27T11:50:31.811 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/114/18' -> 'copyofsnap1/test1/114/18' 2022-06-27T11:50:31.811 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/114/19' -> 'copyofsnap1/test1/114/19' 2022-06-27T11:50:31.811 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/114/20' -> 'copyofsnap1/test1/114/20' 2022-06-27T11:50:31.811 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/115' -> 'copyofsnap1/test1/115' 2022-06-27T11:50:31.812 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/115/10' -> 'copyofsnap1/test1/115/10' 2022-06-27T11:50:31.812 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/115/11' -> 'copyofsnap1/test1/115/11' 2022-06-27T11:50:31.812 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/115/12' -> 'copyofsnap1/test1/115/12' 2022-06-27T11:50:31.812 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/115/13' -> 'copyofsnap1/test1/115/13' 2022-06-27T11:50:31.813 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/115/14' -> 'copyofsnap1/test1/115/14' 2022-06-27T11:50:31.813 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/115/15' -> 'copyofsnap1/test1/115/15' 2022-06-27T11:50:31.813 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/115/16' -> 'copyofsnap1/test1/115/16' 2022-06-27T11:50:31.813 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/115/17' -> 'copyofsnap1/test1/115/17' 2022-06-27T11:50:31.814 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/115/18' -> 'copyofsnap1/test1/115/18' 2022-06-27T11:50:31.814 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/115/19' -> 'copyofsnap1/test1/115/19' 2022-06-27T11:50:31.814 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/115/20' -> 'copyofsnap1/test1/115/20' 2022-06-27T11:50:31.814 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/116' -> 'copyofsnap1/test1/116' 2022-06-27T11:50:31.814 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/116/10' -> 'copyofsnap1/test1/116/10' 2022-06-27T11:50:31.815 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/116/11' -> 'copyofsnap1/test1/116/11' 2022-06-27T11:50:31.815 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/116/12' -> 'copyofsnap1/test1/116/12' 2022-06-27T11:50:31.815 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/116/13' -> 'copyofsnap1/test1/116/13' 2022-06-27T11:50:31.815 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/116/14' -> 'copyofsnap1/test1/116/14' 2022-06-27T11:50:31.816 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/116/15' -> 'copyofsnap1/test1/116/15' 2022-06-27T11:50:31.816 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/116/16' -> 'copyofsnap1/test1/116/16' 2022-06-27T11:50:31.816 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/116/17' -> 'copyofsnap1/test1/116/17' 2022-06-27T11:50:31.816 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/116/18' -> 'copyofsnap1/test1/116/18' 2022-06-27T11:50:31.817 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/116/19' -> 'copyofsnap1/test1/116/19' 2022-06-27T11:50:31.817 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/116/20' -> 'copyofsnap1/test1/116/20' 2022-06-27T11:50:31.817 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/117' -> 'copyofsnap1/test1/117' 2022-06-27T11:50:31.817 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/117/10' -> 'copyofsnap1/test1/117/10' 2022-06-27T11:50:31.818 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/117/11' -> 'copyofsnap1/test1/117/11' 2022-06-27T11:50:31.819 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/117/12' -> 'copyofsnap1/test1/117/12' 2022-06-27T11:50:31.820 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/117/13' -> 'copyofsnap1/test1/117/13' 2022-06-27T11:50:31.820 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/117/14' -> 'copyofsnap1/test1/117/14' 2022-06-27T11:50:31.820 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/117/15' -> 'copyofsnap1/test1/117/15' 2022-06-27T11:50:31.821 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/117/16' -> 'copyofsnap1/test1/117/16' 2022-06-27T11:50:31.821 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/117/17' -> 'copyofsnap1/test1/117/17' 2022-06-27T11:50:31.821 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/117/18' -> 'copyofsnap1/test1/117/18' 2022-06-27T11:50:31.821 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/117/19' -> 'copyofsnap1/test1/117/19' 2022-06-27T11:50:31.822 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/117/20' -> 'copyofsnap1/test1/117/20' 2022-06-27T11:50:31.822 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/118' -> 'copyofsnap1/test1/118' 2022-06-27T11:50:31.822 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/118/10' -> 'copyofsnap1/test1/118/10' 2022-06-27T11:50:31.822 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/118/11' -> 'copyofsnap1/test1/118/11' 2022-06-27T11:50:31.823 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/118/12' -> 'copyofsnap1/test1/118/12' 2022-06-27T11:50:31.823 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/118/13' -> 'copyofsnap1/test1/118/13' 2022-06-27T11:50:31.823 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/118/14' -> 'copyofsnap1/test1/118/14' 2022-06-27T11:50:31.823 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/118/15' -> 'copyofsnap1/test1/118/15' 2022-06-27T11:50:31.824 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/118/16' -> 'copyofsnap1/test1/118/16' 2022-06-27T11:50:31.824 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/118/17' -> 'copyofsnap1/test1/118/17' 2022-06-27T11:50:31.824 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/118/18' -> 'copyofsnap1/test1/118/18' 2022-06-27T11:50:31.824 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/118/19' -> 'copyofsnap1/test1/118/19' 2022-06-27T11:50:31.824 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/118/20' -> 'copyofsnap1/test1/118/20' 2022-06-27T11:50:31.825 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/119' -> 'copyofsnap1/test1/119' 2022-06-27T11:50:31.825 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/119/10' -> 'copyofsnap1/test1/119/10' 2022-06-27T11:50:31.825 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/119/11' -> 'copyofsnap1/test1/119/11' 2022-06-27T11:50:31.825 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/119/12' -> 'copyofsnap1/test1/119/12' 2022-06-27T11:50:31.826 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/119/13' -> 'copyofsnap1/test1/119/13' 2022-06-27T11:50:31.826 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/119/14' -> 'copyofsnap1/test1/119/14' 2022-06-27T11:50:31.826 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/119/15' -> 'copyofsnap1/test1/119/15' 2022-06-27T11:50:31.826 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/119/16' -> 'copyofsnap1/test1/119/16' 2022-06-27T11:50:31.826 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/119/17' -> 'copyofsnap1/test1/119/17' 2022-06-27T11:50:31.827 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/119/18' -> 'copyofsnap1/test1/119/18' 2022-06-27T11:50:31.827 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/119/19' -> 'copyofsnap1/test1/119/19' 2022-06-27T11:50:31.827 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/119/20' -> 'copyofsnap1/test1/119/20' 2022-06-27T11:50:31.827 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/120' -> 'copyofsnap1/test1/120' 2022-06-27T11:50:31.828 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/120/10' -> 'copyofsnap1/test1/120/10' 2022-06-27T11:50:32.474 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/120/11' -> 'copyofsnap1/test1/120/11' 2022-06-27T11:50:32.474 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/120/12' -> 'copyofsnap1/test1/120/12' 2022-06-27T11:50:32.475 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/120/13' -> 'copyofsnap1/test1/120/13' 2022-06-27T11:50:32.476 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/120/14' -> 'copyofsnap1/test1/120/14' 2022-06-27T11:50:32.476 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/120/15' -> 'copyofsnap1/test1/120/15' 2022-06-27T11:50:32.477 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/120/16' -> 'copyofsnap1/test1/120/16' 2022-06-27T11:50:32.477 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/120/17' -> 'copyofsnap1/test1/120/17' 2022-06-27T11:50:32.477 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/120/18' -> 'copyofsnap1/test1/120/18' 2022-06-27T11:50:32.478 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/120/19' -> 'copyofsnap1/test1/120/19' 2022-06-27T11:50:32.478 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/120/20' -> 'copyofsnap1/test1/120/20' 2022-06-27T11:50:32.479 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/121' -> 'copyofsnap1/test1/121' 2022-06-27T11:50:32.480 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/121/10' -> 'copyofsnap1/test1/121/10' 2022-06-27T11:50:32.480 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/121/11' -> 'copyofsnap1/test1/121/11' 2022-06-27T11:50:32.481 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/121/12' -> 'copyofsnap1/test1/121/12' 2022-06-27T11:50:32.483 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/121/13' -> 'copyofsnap1/test1/121/13' 2022-06-27T11:50:32.484 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/121/14' -> 'copyofsnap1/test1/121/14' 2022-06-27T11:50:32.484 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/121/15' -> 'copyofsnap1/test1/121/15' 2022-06-27T11:50:32.484 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/121/16' -> 'copyofsnap1/test1/121/16' 2022-06-27T11:50:32.487 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/121/17' -> 'copyofsnap1/test1/121/17' 2022-06-27T11:50:32.487 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/121/18' -> 'copyofsnap1/test1/121/18' 2022-06-27T11:50:32.487 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/121/19' -> 'copyofsnap1/test1/121/19' 2022-06-27T11:50:32.488 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/121/20' -> 'copyofsnap1/test1/121/20' 2022-06-27T11:50:32.488 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/122' -> 'copyofsnap1/test1/122' 2022-06-27T11:50:32.488 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/122/10' -> 'copyofsnap1/test1/122/10' 2022-06-27T11:50:32.488 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/122/11' -> 'copyofsnap1/test1/122/11' 2022-06-27T11:50:32.489 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/122/12' -> 'copyofsnap1/test1/122/12' 2022-06-27T11:50:32.490 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/122/13' -> 'copyofsnap1/test1/122/13' 2022-06-27T11:50:32.490 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/122/14' -> 'copyofsnap1/test1/122/14' 2022-06-27T11:50:32.490 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/122/15' -> 'copyofsnap1/test1/122/15' 2022-06-27T11:50:32.491 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/122/16' -> 'copyofsnap1/test1/122/16' 2022-06-27T11:50:32.491 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/122/17' -> 'copyofsnap1/test1/122/17' 2022-06-27T11:50:32.492 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/122/18' -> 'copyofsnap1/test1/122/18' 2022-06-27T11:50:32.492 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/122/19' -> 'copyofsnap1/test1/122/19' 2022-06-27T11:50:32.492 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/122/20' -> 'copyofsnap1/test1/122/20' 2022-06-27T11:50:32.493 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/123' -> 'copyofsnap1/test1/123' 2022-06-27T11:50:32.493 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/123/10' -> 'copyofsnap1/test1/123/10' 2022-06-27T11:50:32.493 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/123/11' -> 'copyofsnap1/test1/123/11' 2022-06-27T11:50:32.494 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/123/12' -> 'copyofsnap1/test1/123/12' 2022-06-27T11:50:32.494 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/123/13' -> 'copyofsnap1/test1/123/13' 2022-06-27T11:50:32.495 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/123/14' -> 'copyofsnap1/test1/123/14' 2022-06-27T11:50:32.495 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/123/15' -> 'copyofsnap1/test1/123/15' 2022-06-27T11:50:32.496 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/123/16' -> 'copyofsnap1/test1/123/16' 2022-06-27T11:50:32.496 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/123/17' -> 'copyofsnap1/test1/123/17' 2022-06-27T11:50:32.496 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/123/18' -> 'copyofsnap1/test1/123/18' 2022-06-27T11:50:32.497 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/123/19' -> 'copyofsnap1/test1/123/19' 2022-06-27T11:50:32.497 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/123/20' -> 'copyofsnap1/test1/123/20' 2022-06-27T11:50:32.498 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/124' -> 'copyofsnap1/test1/124' 2022-06-27T11:50:32.498 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/124/10' -> 'copyofsnap1/test1/124/10' 2022-06-27T11:50:32.498 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/124/11' -> 'copyofsnap1/test1/124/11' 2022-06-27T11:50:32.499 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/124/12' -> 'copyofsnap1/test1/124/12' 2022-06-27T11:50:32.499 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/124/13' -> 'copyofsnap1/test1/124/13' 2022-06-27T11:50:32.499 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/124/14' -> 'copyofsnap1/test1/124/14' 2022-06-27T11:50:32.500 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/124/15' -> 'copyofsnap1/test1/124/15' 2022-06-27T11:50:32.500 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/124/16' -> 'copyofsnap1/test1/124/16' 2022-06-27T11:50:32.500 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/124/17' -> 'copyofsnap1/test1/124/17' 2022-06-27T11:50:32.500 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/124/18' -> 'copyofsnap1/test1/124/18' 2022-06-27T11:50:32.501 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/124/19' -> 'copyofsnap1/test1/124/19' 2022-06-27T11:50:32.501 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/124/20' -> 'copyofsnap1/test1/124/20' 2022-06-27T11:50:32.501 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/125' -> 'copyofsnap1/test1/125' 2022-06-27T11:50:32.501 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/125/10' -> 'copyofsnap1/test1/125/10' 2022-06-27T11:50:32.502 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/125/11' -> 'copyofsnap1/test1/125/11' 2022-06-27T11:50:32.502 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/125/12' -> 'copyofsnap1/test1/125/12' 2022-06-27T11:50:32.502 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/125/13' -> 'copyofsnap1/test1/125/13' 2022-06-27T11:50:32.502 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/125/14' -> 'copyofsnap1/test1/125/14' 2022-06-27T11:50:32.503 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/125/15' -> 'copyofsnap1/test1/125/15' 2022-06-27T11:50:32.503 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/125/16' -> 'copyofsnap1/test1/125/16' 2022-06-27T11:50:32.503 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/125/17' -> 'copyofsnap1/test1/125/17' 2022-06-27T11:50:32.504 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/125/18' -> 'copyofsnap1/test1/125/18' 2022-06-27T11:50:32.506 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/125/19' -> 'copyofsnap1/test1/125/19' 2022-06-27T11:50:32.506 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/125/20' -> 'copyofsnap1/test1/125/20' 2022-06-27T11:50:32.507 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/126' -> 'copyofsnap1/test1/126' 2022-06-27T11:50:32.507 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/126/10' -> 'copyofsnap1/test1/126/10' 2022-06-27T11:50:32.508 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/126/11' -> 'copyofsnap1/test1/126/11' 2022-06-27T11:50:32.508 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/126/12' -> 'copyofsnap1/test1/126/12' 2022-06-27T11:50:32.508 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/126/13' -> 'copyofsnap1/test1/126/13' 2022-06-27T11:50:32.509 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/126/14' -> 'copyofsnap1/test1/126/14' 2022-06-27T11:50:32.509 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/126/15' -> 'copyofsnap1/test1/126/15' 2022-06-27T11:50:32.510 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/126/16' -> 'copyofsnap1/test1/126/16' 2022-06-27T11:50:32.510 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/126/17' -> 'copyofsnap1/test1/126/17' 2022-06-27T11:50:32.510 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/126/18' -> 'copyofsnap1/test1/126/18' 2022-06-27T11:50:32.511 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/126/19' -> 'copyofsnap1/test1/126/19' 2022-06-27T11:50:32.622 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/126/20' -> 'copyofsnap1/test1/126/20' 2022-06-27T11:50:32.622 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/127' -> 'copyofsnap1/test1/127' 2022-06-27T11:50:32.623 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/127/10' -> 'copyofsnap1/test1/127/10' 2022-06-27T11:50:32.623 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/127/11' -> 'copyofsnap1/test1/127/11' 2022-06-27T11:50:32.624 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/127/12' -> 'copyofsnap1/test1/127/12' 2022-06-27T11:50:32.624 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/127/13' -> 'copyofsnap1/test1/127/13' 2022-06-27T11:50:32.625 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/127/14' -> 'copyofsnap1/test1/127/14' 2022-06-27T11:50:32.625 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/127/15' -> 'copyofsnap1/test1/127/15' 2022-06-27T11:50:32.625 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/127/16' -> 'copyofsnap1/test1/127/16' 2022-06-27T11:50:32.625 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/127/17' -> 'copyofsnap1/test1/127/17' 2022-06-27T11:50:32.626 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/127/18' -> 'copyofsnap1/test1/127/18' 2022-06-27T11:50:32.626 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/127/19' -> 'copyofsnap1/test1/127/19' 2022-06-27T11:50:32.627 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/127/20' -> 'copyofsnap1/test1/127/20' 2022-06-27T11:50:32.627 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/128' -> 'copyofsnap1/test1/128' 2022-06-27T11:50:32.627 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/128/10' -> 'copyofsnap1/test1/128/10' 2022-06-27T11:50:32.627 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/128/11' -> 'copyofsnap1/test1/128/11' 2022-06-27T11:50:32.628 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/128/12' -> 'copyofsnap1/test1/128/12' 2022-06-27T11:50:32.628 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/128/13' -> 'copyofsnap1/test1/128/13' 2022-06-27T11:50:32.628 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/128/14' -> 'copyofsnap1/test1/128/14' 2022-06-27T11:50:32.629 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/128/15' -> 'copyofsnap1/test1/128/15' 2022-06-27T11:50:32.629 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/128/16' -> 'copyofsnap1/test1/128/16' 2022-06-27T11:50:32.630 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/128/17' -> 'copyofsnap1/test1/128/17' 2022-06-27T11:50:32.630 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/128/18' -> 'copyofsnap1/test1/128/18' 2022-06-27T11:50:32.631 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/128/19' -> 'copyofsnap1/test1/128/19' 2022-06-27T11:50:32.632 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/128/20' -> 'copyofsnap1/test1/128/20' 2022-06-27T11:50:32.632 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/129' -> 'copyofsnap1/test1/129' 2022-06-27T11:50:32.633 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/129/10' -> 'copyofsnap1/test1/129/10' 2022-06-27T11:50:32.633 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/129/11' -> 'copyofsnap1/test1/129/11' 2022-06-27T11:50:32.634 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/129/12' -> 'copyofsnap1/test1/129/12' 2022-06-27T11:50:32.634 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/129/13' -> 'copyofsnap1/test1/129/13' 2022-06-27T11:50:32.634 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/129/14' -> 'copyofsnap1/test1/129/14' 2022-06-27T11:50:32.635 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/129/15' -> 'copyofsnap1/test1/129/15' 2022-06-27T11:50:32.635 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/129/16' -> 'copyofsnap1/test1/129/16' 2022-06-27T11:50:32.636 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/129/17' -> 'copyofsnap1/test1/129/17' 2022-06-27T11:50:32.636 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/129/18' -> 'copyofsnap1/test1/129/18' 2022-06-27T11:50:32.637 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/129/19' -> 'copyofsnap1/test1/129/19' 2022-06-27T11:50:32.637 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/129/20' -> 'copyofsnap1/test1/129/20' 2022-06-27T11:50:32.637 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/130' -> 'copyofsnap1/test1/130' 2022-06-27T11:50:32.638 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/130/10' -> 'copyofsnap1/test1/130/10' 2022-06-27T11:50:32.638 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/130/11' -> 'copyofsnap1/test1/130/11' 2022-06-27T11:50:32.639 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/130/12' -> 'copyofsnap1/test1/130/12' 2022-06-27T11:50:32.639 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/130/13' -> 'copyofsnap1/test1/130/13' 2022-06-27T11:50:32.640 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/130/14' -> 'copyofsnap1/test1/130/14' 2022-06-27T11:50:32.640 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/130/15' -> 'copyofsnap1/test1/130/15' 2022-06-27T11:50:32.641 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/130/16' -> 'copyofsnap1/test1/130/16' 2022-06-27T11:50:32.641 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/130/17' -> 'copyofsnap1/test1/130/17' 2022-06-27T11:50:32.641 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/130/18' -> 'copyofsnap1/test1/130/18' 2022-06-27T11:50:32.642 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/130/19' -> 'copyofsnap1/test1/130/19' 2022-06-27T11:50:32.642 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/130/20' -> 'copyofsnap1/test1/130/20' 2022-06-27T11:50:32.643 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/131' -> 'copyofsnap1/test1/131' 2022-06-27T11:50:32.643 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/131/10' -> 'copyofsnap1/test1/131/10' 2022-06-27T11:50:32.644 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/131/11' -> 'copyofsnap1/test1/131/11' 2022-06-27T11:50:32.644 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/131/12' -> 'copyofsnap1/test1/131/12' 2022-06-27T11:50:32.644 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/131/13' -> 'copyofsnap1/test1/131/13' 2022-06-27T11:50:32.645 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/131/14' -> 'copyofsnap1/test1/131/14' 2022-06-27T11:50:32.645 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/131/15' -> 'copyofsnap1/test1/131/15' 2022-06-27T11:50:32.646 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/131/16' -> 'copyofsnap1/test1/131/16' 2022-06-27T11:50:32.646 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/131/17' -> 'copyofsnap1/test1/131/17' 2022-06-27T11:50:32.646 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/131/18' -> 'copyofsnap1/test1/131/18' 2022-06-27T11:50:32.647 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/131/19' -> 'copyofsnap1/test1/131/19' 2022-06-27T11:50:32.647 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/131/20' -> 'copyofsnap1/test1/131/20' 2022-06-27T11:50:32.648 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/132' -> 'copyofsnap1/test1/132' 2022-06-27T11:50:32.649 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/132/10' -> 'copyofsnap1/test1/132/10' 2022-06-27T11:50:32.649 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/132/11' -> 'copyofsnap1/test1/132/11' 2022-06-27T11:50:32.649 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/132/12' -> 'copyofsnap1/test1/132/12' 2022-06-27T11:50:32.650 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/132/13' -> 'copyofsnap1/test1/132/13' 2022-06-27T11:50:32.650 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/132/14' -> 'copyofsnap1/test1/132/14' 2022-06-27T11:50:32.651 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/132/15' -> 'copyofsnap1/test1/132/15' 2022-06-27T11:50:32.651 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/132/16' -> 'copyofsnap1/test1/132/16' 2022-06-27T11:50:32.652 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/132/17' -> 'copyofsnap1/test1/132/17' 2022-06-27T11:50:32.652 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/132/18' -> 'copyofsnap1/test1/132/18' 2022-06-27T11:50:32.652 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/132/19' -> 'copyofsnap1/test1/132/19' 2022-06-27T11:50:32.653 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/132/20' -> 'copyofsnap1/test1/132/20' 2022-06-27T11:50:32.653 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/133' -> 'copyofsnap1/test1/133' 2022-06-27T11:50:32.654 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/133/10' -> 'copyofsnap1/test1/133/10' 2022-06-27T11:50:32.654 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/133/11' -> 'copyofsnap1/test1/133/11' 2022-06-27T11:50:32.654 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/133/12' -> 'copyofsnap1/test1/133/12' 2022-06-27T11:50:32.654 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/133/13' -> 'copyofsnap1/test1/133/13' 2022-06-27T11:50:32.655 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/133/14' -> 'copyofsnap1/test1/133/14' 2022-06-27T11:50:32.665 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/133/15' -> 'copyofsnap1/test1/133/15' 2022-06-27T11:50:32.666 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/133/16' -> 'copyofsnap1/test1/133/16' 2022-06-27T11:50:33.227 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/133/17' -> 'copyofsnap1/test1/133/17' 2022-06-27T11:50:33.228 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/133/18' -> 'copyofsnap1/test1/133/18' 2022-06-27T11:50:33.229 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/133/19' -> 'copyofsnap1/test1/133/19' 2022-06-27T11:50:33.229 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/133/20' -> 'copyofsnap1/test1/133/20' 2022-06-27T11:50:33.229 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/134' -> 'copyofsnap1/test1/134' 2022-06-27T11:50:33.230 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/134/10' -> 'copyofsnap1/test1/134/10' 2022-06-27T11:50:33.230 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/134/11' -> 'copyofsnap1/test1/134/11' 2022-06-27T11:50:33.231 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/134/12' -> 'copyofsnap1/test1/134/12' 2022-06-27T11:50:33.231 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/134/13' -> 'copyofsnap1/test1/134/13' 2022-06-27T11:50:33.233 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/134/14' -> 'copyofsnap1/test1/134/14' 2022-06-27T11:50:33.233 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/134/15' -> 'copyofsnap1/test1/134/15' 2022-06-27T11:50:33.235 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/134/16' -> 'copyofsnap1/test1/134/16' 2022-06-27T11:50:33.235 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/134/17' -> 'copyofsnap1/test1/134/17' 2022-06-27T11:50:33.236 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/134/18' -> 'copyofsnap1/test1/134/18' 2022-06-27T11:50:33.236 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/134/19' -> 'copyofsnap1/test1/134/19' 2022-06-27T11:50:33.236 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/134/20' -> 'copyofsnap1/test1/134/20' 2022-06-27T11:50:33.237 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/135' -> 'copyofsnap1/test1/135' 2022-06-27T11:50:33.237 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/135/10' -> 'copyofsnap1/test1/135/10' 2022-06-27T11:50:33.237 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/135/11' -> 'copyofsnap1/test1/135/11' 2022-06-27T11:50:33.238 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/135/12' -> 'copyofsnap1/test1/135/12' 2022-06-27T11:50:33.238 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/135/13' -> 'copyofsnap1/test1/135/13' 2022-06-27T11:50:33.239 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/135/14' -> 'copyofsnap1/test1/135/14' 2022-06-27T11:50:33.239 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/135/15' -> 'copyofsnap1/test1/135/15' 2022-06-27T11:50:33.240 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/135/16' -> 'copyofsnap1/test1/135/16' 2022-06-27T11:50:33.240 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/135/17' -> 'copyofsnap1/test1/135/17' 2022-06-27T11:50:33.241 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/135/18' -> 'copyofsnap1/test1/135/18' 2022-06-27T11:50:33.241 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/135/19' -> 'copyofsnap1/test1/135/19' 2022-06-27T11:50:33.242 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/135/20' -> 'copyofsnap1/test1/135/20' 2022-06-27T11:50:33.242 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/136' -> 'copyofsnap1/test1/136' 2022-06-27T11:50:33.243 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/136/10' -> 'copyofsnap1/test1/136/10' 2022-06-27T11:50:33.243 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/136/11' -> 'copyofsnap1/test1/136/11' 2022-06-27T11:50:33.244 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/136/12' -> 'copyofsnap1/test1/136/12' 2022-06-27T11:50:33.255 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/136/13' -> 'copyofsnap1/test1/136/13' 2022-06-27T11:50:33.255 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/136/14' -> 'copyofsnap1/test1/136/14' 2022-06-27T11:50:33.255 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/136/15' -> 'copyofsnap1/test1/136/15' 2022-06-27T11:50:33.256 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/136/16' -> 'copyofsnap1/test1/136/16' 2022-06-27T11:50:33.256 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/136/17' -> 'copyofsnap1/test1/136/17' 2022-06-27T11:50:33.257 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/136/18' -> 'copyofsnap1/test1/136/18' 2022-06-27T11:50:33.257 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/136/19' -> 'copyofsnap1/test1/136/19' 2022-06-27T11:50:33.258 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/136/20' -> 'copyofsnap1/test1/136/20' 2022-06-27T11:50:33.258 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/137' -> 'copyofsnap1/test1/137' 2022-06-27T11:50:33.259 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/137/10' -> 'copyofsnap1/test1/137/10' 2022-06-27T11:50:33.259 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/137/11' -> 'copyofsnap1/test1/137/11' 2022-06-27T11:50:33.259 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/137/12' -> 'copyofsnap1/test1/137/12' 2022-06-27T11:50:33.260 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/137/13' -> 'copyofsnap1/test1/137/13' 2022-06-27T11:50:33.260 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/137/14' -> 'copyofsnap1/test1/137/14' 2022-06-27T11:50:33.261 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/137/15' -> 'copyofsnap1/test1/137/15' 2022-06-27T11:50:33.262 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/137/16' -> 'copyofsnap1/test1/137/16' 2022-06-27T11:50:33.262 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/137/17' -> 'copyofsnap1/test1/137/17' 2022-06-27T11:50:33.262 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/137/18' -> 'copyofsnap1/test1/137/18' 2022-06-27T11:50:33.263 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/137/19' -> 'copyofsnap1/test1/137/19' 2022-06-27T11:50:33.263 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/137/20' -> 'copyofsnap1/test1/137/20' 2022-06-27T11:50:33.264 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/138' -> 'copyofsnap1/test1/138' 2022-06-27T11:50:33.265 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/138/10' -> 'copyofsnap1/test1/138/10' 2022-06-27T11:50:33.265 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/138/11' -> 'copyofsnap1/test1/138/11' 2022-06-27T11:50:33.265 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/138/12' -> 'copyofsnap1/test1/138/12' 2022-06-27T11:50:33.266 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/138/13' -> 'copyofsnap1/test1/138/13' 2022-06-27T11:50:33.266 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/138/14' -> 'copyofsnap1/test1/138/14' 2022-06-27T11:50:33.267 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/138/15' -> 'copyofsnap1/test1/138/15' 2022-06-27T11:50:33.268 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/138/16' -> 'copyofsnap1/test1/138/16' 2022-06-27T11:50:33.268 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/138/17' -> 'copyofsnap1/test1/138/17' 2022-06-27T11:50:33.269 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/138/18' -> 'copyofsnap1/test1/138/18' 2022-06-27T11:50:33.269 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/138/19' -> 'copyofsnap1/test1/138/19' 2022-06-27T11:50:33.270 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/138/20' -> 'copyofsnap1/test1/138/20' 2022-06-27T11:50:33.270 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/139' -> 'copyofsnap1/test1/139' 2022-06-27T11:50:33.270 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/139/10' -> 'copyofsnap1/test1/139/10' 2022-06-27T11:50:33.271 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/139/11' -> 'copyofsnap1/test1/139/11' 2022-06-27T11:50:33.271 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/139/12' -> 'copyofsnap1/test1/139/12' 2022-06-27T11:50:33.271 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/139/13' -> 'copyofsnap1/test1/139/13' 2022-06-27T11:50:33.272 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/139/14' -> 'copyofsnap1/test1/139/14' 2022-06-27T11:50:33.272 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/139/15' -> 'copyofsnap1/test1/139/15' 2022-06-27T11:50:33.273 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/139/16' -> 'copyofsnap1/test1/139/16' 2022-06-27T11:50:33.273 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/139/17' -> 'copyofsnap1/test1/139/17' 2022-06-27T11:50:33.274 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/139/18' -> 'copyofsnap1/test1/139/18' 2022-06-27T11:50:33.274 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/139/19' -> 'copyofsnap1/test1/139/19' 2022-06-27T11:50:33.274 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/139/20' -> 'copyofsnap1/test1/139/20' 2022-06-27T11:50:33.275 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/140' -> 'copyofsnap1/test1/140' 2022-06-27T11:50:33.275 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/140/10' -> 'copyofsnap1/test1/140/10' 2022-06-27T11:50:33.276 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/140/11' -> 'copyofsnap1/test1/140/11' 2022-06-27T11:50:33.276 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/140/12' -> 'copyofsnap1/test1/140/12' 2022-06-27T11:50:33.276 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/140/13' -> 'copyofsnap1/test1/140/13' 2022-06-27T11:50:33.909 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/140/14' -> 'copyofsnap1/test1/140/14' 2022-06-27T11:50:33.909 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/140/15' -> 'copyofsnap1/test1/140/15' 2022-06-27T11:50:33.910 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/140/16' -> 'copyofsnap1/test1/140/16' 2022-06-27T11:50:33.910 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/140/17' -> 'copyofsnap1/test1/140/17' 2022-06-27T11:50:33.911 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/140/18' -> 'copyofsnap1/test1/140/18' 2022-06-27T11:50:33.911 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/140/19' -> 'copyofsnap1/test1/140/19' 2022-06-27T11:50:33.912 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/140/20' -> 'copyofsnap1/test1/140/20' 2022-06-27T11:50:33.912 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/141' -> 'copyofsnap1/test1/141' 2022-06-27T11:50:33.913 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/141/10' -> 'copyofsnap1/test1/141/10' 2022-06-27T11:50:33.913 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/141/11' -> 'copyofsnap1/test1/141/11' 2022-06-27T11:50:33.914 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/141/12' -> 'copyofsnap1/test1/141/12' 2022-06-27T11:50:33.914 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/141/13' -> 'copyofsnap1/test1/141/13' 2022-06-27T11:50:33.915 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/141/14' -> 'copyofsnap1/test1/141/14' 2022-06-27T11:50:33.915 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/141/15' -> 'copyofsnap1/test1/141/15' 2022-06-27T11:50:33.916 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/141/16' -> 'copyofsnap1/test1/141/16' 2022-06-27T11:50:33.917 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/141/17' -> 'copyofsnap1/test1/141/17' 2022-06-27T11:50:33.917 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/141/18' -> 'copyofsnap1/test1/141/18' 2022-06-27T11:50:33.918 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/141/19' -> 'copyofsnap1/test1/141/19' 2022-06-27T11:50:33.918 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/141/20' -> 'copyofsnap1/test1/141/20' 2022-06-27T11:50:33.918 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/142' -> 'copyofsnap1/test1/142' 2022-06-27T11:50:33.918 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/142/10' -> 'copyofsnap1/test1/142/10' 2022-06-27T11:50:33.919 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/142/11' -> 'copyofsnap1/test1/142/11' 2022-06-27T11:50:33.919 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/142/12' -> 'copyofsnap1/test1/142/12' 2022-06-27T11:50:33.920 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/142/13' -> 'copyofsnap1/test1/142/13' 2022-06-27T11:50:33.920 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/142/14' -> 'copyofsnap1/test1/142/14' 2022-06-27T11:50:33.920 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/142/15' -> 'copyofsnap1/test1/142/15' 2022-06-27T11:50:33.921 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/142/16' -> 'copyofsnap1/test1/142/16' 2022-06-27T11:50:33.921 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/142/17' -> 'copyofsnap1/test1/142/17' 2022-06-27T11:50:33.921 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/142/18' -> 'copyofsnap1/test1/142/18' 2022-06-27T11:50:33.922 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/142/19' -> 'copyofsnap1/test1/142/19' 2022-06-27T11:50:33.922 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/142/20' -> 'copyofsnap1/test1/142/20' 2022-06-27T11:50:33.922 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/143' -> 'copyofsnap1/test1/143' 2022-06-27T11:50:33.922 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/143/10' -> 'copyofsnap1/test1/143/10' 2022-06-27T11:50:33.923 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/143/11' -> 'copyofsnap1/test1/143/11' 2022-06-27T11:50:33.923 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/143/12' -> 'copyofsnap1/test1/143/12' 2022-06-27T11:50:33.923 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/143/13' -> 'copyofsnap1/test1/143/13' 2022-06-27T11:50:33.923 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/143/14' -> 'copyofsnap1/test1/143/14' 2022-06-27T11:50:33.924 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/143/15' -> 'copyofsnap1/test1/143/15' 2022-06-27T11:50:33.924 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/143/16' -> 'copyofsnap1/test1/143/16' 2022-06-27T11:50:33.924 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/143/17' -> 'copyofsnap1/test1/143/17' 2022-06-27T11:50:33.924 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/143/18' -> 'copyofsnap1/test1/143/18' 2022-06-27T11:50:33.924 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/143/19' -> 'copyofsnap1/test1/143/19' 2022-06-27T11:50:33.925 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/143/20' -> 'copyofsnap1/test1/143/20' 2022-06-27T11:50:33.925 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/144' -> 'copyofsnap1/test1/144' 2022-06-27T11:50:33.925 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/144/10' -> 'copyofsnap1/test1/144/10' 2022-06-27T11:50:33.926 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/144/11' -> 'copyofsnap1/test1/144/11' 2022-06-27T11:50:33.926 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/144/12' -> 'copyofsnap1/test1/144/12' 2022-06-27T11:50:33.926 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/144/13' -> 'copyofsnap1/test1/144/13' 2022-06-27T11:50:33.927 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/144/14' -> 'copyofsnap1/test1/144/14' 2022-06-27T11:50:33.927 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/144/15' -> 'copyofsnap1/test1/144/15' 2022-06-27T11:50:33.927 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/144/16' -> 'copyofsnap1/test1/144/16' 2022-06-27T11:50:33.927 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/144/17' -> 'copyofsnap1/test1/144/17' 2022-06-27T11:50:33.927 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/144/18' -> 'copyofsnap1/test1/144/18' 2022-06-27T11:50:33.928 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/144/19' -> 'copyofsnap1/test1/144/19' 2022-06-27T11:50:33.928 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/144/20' -> 'copyofsnap1/test1/144/20' 2022-06-27T11:50:33.928 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/145' -> 'copyofsnap1/test1/145' 2022-06-27T11:50:33.928 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/145/10' -> 'copyofsnap1/test1/145/10' 2022-06-27T11:50:33.929 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/145/11' -> 'copyofsnap1/test1/145/11' 2022-06-27T11:50:33.929 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/145/12' -> 'copyofsnap1/test1/145/12' 2022-06-27T11:50:33.929 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/145/13' -> 'copyofsnap1/test1/145/13' 2022-06-27T11:50:33.929 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/145/14' -> 'copyofsnap1/test1/145/14' 2022-06-27T11:50:33.930 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/145/15' -> 'copyofsnap1/test1/145/15' 2022-06-27T11:50:33.930 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/145/16' -> 'copyofsnap1/test1/145/16' 2022-06-27T11:50:33.930 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/145/17' -> 'copyofsnap1/test1/145/17' 2022-06-27T11:50:33.930 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/145/18' -> 'copyofsnap1/test1/145/18' 2022-06-27T11:50:33.931 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/145/19' -> 'copyofsnap1/test1/145/19' 2022-06-27T11:50:33.931 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/145/20' -> 'copyofsnap1/test1/145/20' 2022-06-27T11:50:33.931 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/146' -> 'copyofsnap1/test1/146' 2022-06-27T11:50:33.931 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/146/10' -> 'copyofsnap1/test1/146/10' 2022-06-27T11:50:33.932 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/146/11' -> 'copyofsnap1/test1/146/11' 2022-06-27T11:50:33.932 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/146/12' -> 'copyofsnap1/test1/146/12' 2022-06-27T11:50:33.932 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/146/13' -> 'copyofsnap1/test1/146/13' 2022-06-27T11:50:33.932 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/146/14' -> 'copyofsnap1/test1/146/14' 2022-06-27T11:50:33.932 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/146/15' -> 'copyofsnap1/test1/146/15' 2022-06-27T11:50:33.933 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/146/16' -> 'copyofsnap1/test1/146/16' 2022-06-27T11:50:33.933 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/146/17' -> 'copyofsnap1/test1/146/17' 2022-06-27T11:50:33.933 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/146/18' -> 'copyofsnap1/test1/146/18' 2022-06-27T11:50:33.933 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/146/19' -> 'copyofsnap1/test1/146/19' 2022-06-27T11:50:33.934 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/146/20' -> 'copyofsnap1/test1/146/20' 2022-06-27T11:50:33.934 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/147' -> 'copyofsnap1/test1/147' 2022-06-27T11:50:33.934 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/147/10' -> 'copyofsnap1/test1/147/10' 2022-06-27T11:50:33.934 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/147/11' -> 'copyofsnap1/test1/147/11' 2022-06-27T11:50:33.935 INFO:tasks.ceph.mds.b.smithi202.stderr:2022-06-27T11:50:33.932+0000 7fe65a767700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2022-06-27T11:50:34.067 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/147/12' -> 'copyofsnap1/test1/147/12' 2022-06-27T11:50:34.067 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/147/13' -> 'copyofsnap1/test1/147/13' 2022-06-27T11:50:34.068 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/147/14' -> 'copyofsnap1/test1/147/14' 2022-06-27T11:50:34.068 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/147/15' -> 'copyofsnap1/test1/147/15' 2022-06-27T11:50:34.069 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/147/16' -> 'copyofsnap1/test1/147/16' 2022-06-27T11:50:34.069 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/147/17' -> 'copyofsnap1/test1/147/17' 2022-06-27T11:50:34.070 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/147/18' -> 'copyofsnap1/test1/147/18' 2022-06-27T11:50:34.070 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/147/19' -> 'copyofsnap1/test1/147/19' 2022-06-27T11:50:34.071 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/147/20' -> 'copyofsnap1/test1/147/20' 2022-06-27T11:50:34.071 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/148' -> 'copyofsnap1/test1/148' 2022-06-27T11:50:34.071 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/148/10' -> 'copyofsnap1/test1/148/10' 2022-06-27T11:50:34.072 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/148/11' -> 'copyofsnap1/test1/148/11' 2022-06-27T11:50:34.072 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/148/12' -> 'copyofsnap1/test1/148/12' 2022-06-27T11:50:34.072 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/148/13' -> 'copyofsnap1/test1/148/13' 2022-06-27T11:50:34.072 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/148/14' -> 'copyofsnap1/test1/148/14' 2022-06-27T11:50:34.072 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/148/15' -> 'copyofsnap1/test1/148/15' 2022-06-27T11:50:34.073 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/148/16' -> 'copyofsnap1/test1/148/16' 2022-06-27T11:50:34.073 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/148/17' -> 'copyofsnap1/test1/148/17' 2022-06-27T11:50:34.078 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/148/18' -> 'copyofsnap1/test1/148/18' 2022-06-27T11:50:34.078 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/148/19' -> 'copyofsnap1/test1/148/19' 2022-06-27T11:50:34.078 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/148/20' -> 'copyofsnap1/test1/148/20' 2022-06-27T11:50:34.079 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/149' -> 'copyofsnap1/test1/149' 2022-06-27T11:50:34.079 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/149/10' -> 'copyofsnap1/test1/149/10' 2022-06-27T11:50:34.080 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/149/11' -> 'copyofsnap1/test1/149/11' 2022-06-27T11:50:34.080 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/149/12' -> 'copyofsnap1/test1/149/12' 2022-06-27T11:50:34.080 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/149/13' -> 'copyofsnap1/test1/149/13' 2022-06-27T11:50:34.080 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/149/14' -> 'copyofsnap1/test1/149/14' 2022-06-27T11:50:34.081 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/149/15' -> 'copyofsnap1/test1/149/15' 2022-06-27T11:50:34.081 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/149/16' -> 'copyofsnap1/test1/149/16' 2022-06-27T11:50:34.081 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/149/17' -> 'copyofsnap1/test1/149/17' 2022-06-27T11:50:34.081 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/149/18' -> 'copyofsnap1/test1/149/18' 2022-06-27T11:50:34.082 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/149/19' -> 'copyofsnap1/test1/149/19' 2022-06-27T11:50:34.082 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/149/20' -> 'copyofsnap1/test1/149/20' 2022-06-27T11:50:34.082 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/150' -> 'copyofsnap1/test1/150' 2022-06-27T11:50:34.082 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/150/10' -> 'copyofsnap1/test1/150/10' 2022-06-27T11:50:34.083 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/150/11' -> 'copyofsnap1/test1/150/11' 2022-06-27T11:50:34.083 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/150/12' -> 'copyofsnap1/test1/150/12' 2022-06-27T11:50:34.083 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/150/13' -> 'copyofsnap1/test1/150/13' 2022-06-27T11:50:34.083 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/150/14' -> 'copyofsnap1/test1/150/14' 2022-06-27T11:50:34.083 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/150/15' -> 'copyofsnap1/test1/150/15' 2022-06-27T11:50:34.084 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/150/16' -> 'copyofsnap1/test1/150/16' 2022-06-27T11:50:34.084 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/150/17' -> 'copyofsnap1/test1/150/17' 2022-06-27T11:50:34.084 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/150/18' -> 'copyofsnap1/test1/150/18' 2022-06-27T11:50:34.084 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/150/19' -> 'copyofsnap1/test1/150/19' 2022-06-27T11:50:34.085 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/150/20' -> 'copyofsnap1/test1/150/20' 2022-06-27T11:50:34.085 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/151' -> 'copyofsnap1/test1/151' 2022-06-27T11:50:34.085 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/151/10' -> 'copyofsnap1/test1/151/10' 2022-06-27T11:50:34.085 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/151/11' -> 'copyofsnap1/test1/151/11' 2022-06-27T11:50:34.085 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/151/12' -> 'copyofsnap1/test1/151/12' 2022-06-27T11:50:34.086 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/151/13' -> 'copyofsnap1/test1/151/13' 2022-06-27T11:50:34.086 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/151/14' -> 'copyofsnap1/test1/151/14' 2022-06-27T11:50:34.086 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/151/15' -> 'copyofsnap1/test1/151/15' 2022-06-27T11:50:34.086 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/151/16' -> 'copyofsnap1/test1/151/16' 2022-06-27T11:50:34.087 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/151/17' -> 'copyofsnap1/test1/151/17' 2022-06-27T11:50:34.087 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/151/18' -> 'copyofsnap1/test1/151/18' 2022-06-27T11:50:34.087 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/151/19' -> 'copyofsnap1/test1/151/19' 2022-06-27T11:50:34.087 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/151/20' -> 'copyofsnap1/test1/151/20' 2022-06-27T11:50:34.087 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/152' -> 'copyofsnap1/test1/152' 2022-06-27T11:50:34.088 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/152/10' -> 'copyofsnap1/test1/152/10' 2022-06-27T11:50:34.088 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/152/11' -> 'copyofsnap1/test1/152/11' 2022-06-27T11:50:34.088 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/152/12' -> 'copyofsnap1/test1/152/12' 2022-06-27T11:50:34.088 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/152/13' -> 'copyofsnap1/test1/152/13' 2022-06-27T11:50:34.089 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/152/14' -> 'copyofsnap1/test1/152/14' 2022-06-27T11:50:34.089 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/152/15' -> 'copyofsnap1/test1/152/15' 2022-06-27T11:50:34.089 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/152/16' -> 'copyofsnap1/test1/152/16' 2022-06-27T11:50:34.090 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/152/17' -> 'copyofsnap1/test1/152/17' 2022-06-27T11:50:34.090 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/152/18' -> 'copyofsnap1/test1/152/18' 2022-06-27T11:50:34.090 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/152/19' -> 'copyofsnap1/test1/152/19' 2022-06-27T11:50:34.090 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/152/20' -> 'copyofsnap1/test1/152/20' 2022-06-27T11:50:34.091 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/153' -> 'copyofsnap1/test1/153' 2022-06-27T11:50:34.091 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/153/10' -> 'copyofsnap1/test1/153/10' 2022-06-27T11:50:34.091 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/153/11' -> 'copyofsnap1/test1/153/11' 2022-06-27T11:50:34.091 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/153/12' -> 'copyofsnap1/test1/153/12' 2022-06-27T11:50:34.093 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/153/13' -> 'copyofsnap1/test1/153/13' 2022-06-27T11:50:34.093 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/153/14' -> 'copyofsnap1/test1/153/14' 2022-06-27T11:50:34.093 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/153/15' -> 'copyofsnap1/test1/153/15' 2022-06-27T11:50:34.094 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/153/16' -> 'copyofsnap1/test1/153/16' 2022-06-27T11:50:34.094 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/153/17' -> 'copyofsnap1/test1/153/17' 2022-06-27T11:50:34.094 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/153/18' -> 'copyofsnap1/test1/153/18' 2022-06-27T11:50:34.095 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/153/19' -> 'copyofsnap1/test1/153/19' 2022-06-27T11:50:34.095 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/153/20' -> 'copyofsnap1/test1/153/20' 2022-06-27T11:50:34.200 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/154' -> 'copyofsnap1/test1/154' 2022-06-27T11:50:34.200 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/154/10' -> 'copyofsnap1/test1/154/10' 2022-06-27T11:50:34.201 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/154/11' -> 'copyofsnap1/test1/154/11' 2022-06-27T11:50:34.201 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/154/12' -> 'copyofsnap1/test1/154/12' 2022-06-27T11:50:34.201 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/154/13' -> 'copyofsnap1/test1/154/13' 2022-06-27T11:50:34.201 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/154/14' -> 'copyofsnap1/test1/154/14' 2022-06-27T11:50:34.202 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/154/15' -> 'copyofsnap1/test1/154/15' 2022-06-27T11:50:34.202 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/154/16' -> 'copyofsnap1/test1/154/16' 2022-06-27T11:50:34.202 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/154/17' -> 'copyofsnap1/test1/154/17' 2022-06-27T11:50:34.202 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/154/18' -> 'copyofsnap1/test1/154/18' 2022-06-27T11:50:34.203 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/154/19' -> 'copyofsnap1/test1/154/19' 2022-06-27T11:50:34.203 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/154/20' -> 'copyofsnap1/test1/154/20' 2022-06-27T11:50:34.203 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/155' -> 'copyofsnap1/test1/155' 2022-06-27T11:50:34.203 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/155/10' -> 'copyofsnap1/test1/155/10' 2022-06-27T11:50:34.204 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/155/11' -> 'copyofsnap1/test1/155/11' 2022-06-27T11:50:34.204 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/155/12' -> 'copyofsnap1/test1/155/12' 2022-06-27T11:50:34.204 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/155/13' -> 'copyofsnap1/test1/155/13' 2022-06-27T11:50:34.204 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/155/14' -> 'copyofsnap1/test1/155/14' 2022-06-27T11:50:34.205 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/155/15' -> 'copyofsnap1/test1/155/15' 2022-06-27T11:50:34.205 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/155/16' -> 'copyofsnap1/test1/155/16' 2022-06-27T11:50:34.205 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/155/17' -> 'copyofsnap1/test1/155/17' 2022-06-27T11:50:34.205 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/155/18' -> 'copyofsnap1/test1/155/18' 2022-06-27T11:50:34.205 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/155/19' -> 'copyofsnap1/test1/155/19' 2022-06-27T11:50:34.206 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/155/20' -> 'copyofsnap1/test1/155/20' 2022-06-27T11:50:34.206 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/156' -> 'copyofsnap1/test1/156' 2022-06-27T11:50:34.206 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/156/10' -> 'copyofsnap1/test1/156/10' 2022-06-27T11:50:34.207 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/156/11' -> 'copyofsnap1/test1/156/11' 2022-06-27T11:50:34.207 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/156/12' -> 'copyofsnap1/test1/156/12' 2022-06-27T11:50:34.207 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/156/13' -> 'copyofsnap1/test1/156/13' 2022-06-27T11:50:34.207 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/156/14' -> 'copyofsnap1/test1/156/14' 2022-06-27T11:50:34.208 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/156/15' -> 'copyofsnap1/test1/156/15' 2022-06-27T11:50:34.208 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/156/16' -> 'copyofsnap1/test1/156/16' 2022-06-27T11:50:34.208 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/156/17' -> 'copyofsnap1/test1/156/17' 2022-06-27T11:50:34.208 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/156/18' -> 'copyofsnap1/test1/156/18' 2022-06-27T11:50:34.208 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/156/19' -> 'copyofsnap1/test1/156/19' 2022-06-27T11:50:34.209 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/156/20' -> 'copyofsnap1/test1/156/20' 2022-06-27T11:50:34.209 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/157' -> 'copyofsnap1/test1/157' 2022-06-27T11:50:34.209 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/157/10' -> 'copyofsnap1/test1/157/10' 2022-06-27T11:50:34.209 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/157/11' -> 'copyofsnap1/test1/157/11' 2022-06-27T11:50:34.211 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/157/12' -> 'copyofsnap1/test1/157/12' 2022-06-27T11:50:34.211 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/157/13' -> 'copyofsnap1/test1/157/13' 2022-06-27T11:50:34.212 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/157/14' -> 'copyofsnap1/test1/157/14' 2022-06-27T11:50:34.212 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/157/15' -> 'copyofsnap1/test1/157/15' 2022-06-27T11:50:34.212 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/157/16' -> 'copyofsnap1/test1/157/16' 2022-06-27T11:50:34.212 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/157/17' -> 'copyofsnap1/test1/157/17' 2022-06-27T11:50:34.213 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/157/18' -> 'copyofsnap1/test1/157/18' 2022-06-27T11:50:34.213 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/157/19' -> 'copyofsnap1/test1/157/19' 2022-06-27T11:50:34.213 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/157/20' -> 'copyofsnap1/test1/157/20' 2022-06-27T11:50:34.213 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/158' -> 'copyofsnap1/test1/158' 2022-06-27T11:50:34.213 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/158/10' -> 'copyofsnap1/test1/158/10' 2022-06-27T11:50:34.214 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/158/11' -> 'copyofsnap1/test1/158/11' 2022-06-27T11:50:34.214 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/158/12' -> 'copyofsnap1/test1/158/12' 2022-06-27T11:50:34.215 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/158/13' -> 'copyofsnap1/test1/158/13' 2022-06-27T11:50:34.215 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/158/14' -> 'copyofsnap1/test1/158/14' 2022-06-27T11:50:34.215 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/158/15' -> 'copyofsnap1/test1/158/15' 2022-06-27T11:50:34.215 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/158/16' -> 'copyofsnap1/test1/158/16' 2022-06-27T11:50:34.216 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/158/17' -> 'copyofsnap1/test1/158/17' 2022-06-27T11:50:34.216 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/158/18' -> 'copyofsnap1/test1/158/18' 2022-06-27T11:50:34.216 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/158/19' -> 'copyofsnap1/test1/158/19' 2022-06-27T11:50:34.216 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/158/20' -> 'copyofsnap1/test1/158/20' 2022-06-27T11:50:34.217 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/159' -> 'copyofsnap1/test1/159' 2022-06-27T11:50:34.217 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/159/10' -> 'copyofsnap1/test1/159/10' 2022-06-27T11:50:34.217 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/159/11' -> 'copyofsnap1/test1/159/11' 2022-06-27T11:50:34.217 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/159/12' -> 'copyofsnap1/test1/159/12' 2022-06-27T11:50:34.218 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/159/13' -> 'copyofsnap1/test1/159/13' 2022-06-27T11:50:34.218 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/159/14' -> 'copyofsnap1/test1/159/14' 2022-06-27T11:50:34.218 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/159/15' -> 'copyofsnap1/test1/159/15' 2022-06-27T11:50:34.218 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/159/16' -> 'copyofsnap1/test1/159/16' 2022-06-27T11:50:34.219 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/159/17' -> 'copyofsnap1/test1/159/17' 2022-06-27T11:50:34.219 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/159/18' -> 'copyofsnap1/test1/159/18' 2022-06-27T11:50:34.219 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/159/19' -> 'copyofsnap1/test1/159/19' 2022-06-27T11:50:34.220 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/159/20' -> 'copyofsnap1/test1/159/20' 2022-06-27T11:50:34.220 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/160' -> 'copyofsnap1/test1/160' 2022-06-27T11:50:34.221 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/160/10' -> 'copyofsnap1/test1/160/10' 2022-06-27T11:50:34.221 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/160/11' -> 'copyofsnap1/test1/160/11' 2022-06-27T11:50:34.222 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/160/12' -> 'copyofsnap1/test1/160/12' 2022-06-27T11:50:34.222 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/160/13' -> 'copyofsnap1/test1/160/13' 2022-06-27T11:50:34.223 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/160/14' -> 'copyofsnap1/test1/160/14' 2022-06-27T11:50:34.223 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/160/15' -> 'copyofsnap1/test1/160/15' 2022-06-27T11:50:34.223 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/160/16' -> 'copyofsnap1/test1/160/16' 2022-06-27T11:50:34.224 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/160/17' -> 'copyofsnap1/test1/160/17' 2022-06-27T11:50:34.909 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/160/18' -> 'copyofsnap1/test1/160/18' 2022-06-27T11:50:34.909 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/160/19' -> 'copyofsnap1/test1/160/19' 2022-06-27T11:50:34.910 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/160/20' -> 'copyofsnap1/test1/160/20' 2022-06-27T11:50:34.910 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/161' -> 'copyofsnap1/test1/161' 2022-06-27T11:50:34.911 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/161/10' -> 'copyofsnap1/test1/161/10' 2022-06-27T11:50:34.911 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/161/11' -> 'copyofsnap1/test1/161/11' 2022-06-27T11:50:34.911 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/161/12' -> 'copyofsnap1/test1/161/12' 2022-06-27T11:50:34.912 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/161/13' -> 'copyofsnap1/test1/161/13' 2022-06-27T11:50:34.912 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/161/14' -> 'copyofsnap1/test1/161/14' 2022-06-27T11:50:34.913 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/161/15' -> 'copyofsnap1/test1/161/15' 2022-06-27T11:50:34.913 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/161/16' -> 'copyofsnap1/test1/161/16' 2022-06-27T11:50:34.913 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/161/17' -> 'copyofsnap1/test1/161/17' 2022-06-27T11:50:34.914 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/161/18' -> 'copyofsnap1/test1/161/18' 2022-06-27T11:50:34.914 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/161/19' -> 'copyofsnap1/test1/161/19' 2022-06-27T11:50:34.914 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/161/20' -> 'copyofsnap1/test1/161/20' 2022-06-27T11:50:34.915 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/162' -> 'copyofsnap1/test1/162' 2022-06-27T11:50:34.915 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/162/10' -> 'copyofsnap1/test1/162/10' 2022-06-27T11:50:34.916 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/162/11' -> 'copyofsnap1/test1/162/11' 2022-06-27T11:50:34.916 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/162/12' -> 'copyofsnap1/test1/162/12' 2022-06-27T11:50:34.916 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/162/13' -> 'copyofsnap1/test1/162/13' 2022-06-27T11:50:34.917 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/162/14' -> 'copyofsnap1/test1/162/14' 2022-06-27T11:50:34.917 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/162/15' -> 'copyofsnap1/test1/162/15' 2022-06-27T11:50:34.917 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/162/16' -> 'copyofsnap1/test1/162/16' 2022-06-27T11:50:34.918 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/162/17' -> 'copyofsnap1/test1/162/17' 2022-06-27T11:50:34.918 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/162/18' -> 'copyofsnap1/test1/162/18' 2022-06-27T11:50:34.919 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/162/19' -> 'copyofsnap1/test1/162/19' 2022-06-27T11:50:34.919 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/162/20' -> 'copyofsnap1/test1/162/20' 2022-06-27T11:50:34.920 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/163' -> 'copyofsnap1/test1/163' 2022-06-27T11:50:34.920 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/163/10' -> 'copyofsnap1/test1/163/10' 2022-06-27T11:50:34.921 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/163/11' -> 'copyofsnap1/test1/163/11' 2022-06-27T11:50:34.921 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/163/12' -> 'copyofsnap1/test1/163/12' 2022-06-27T11:50:34.921 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/163/13' -> 'copyofsnap1/test1/163/13' 2022-06-27T11:50:34.922 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/163/14' -> 'copyofsnap1/test1/163/14' 2022-06-27T11:50:34.922 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/163/15' -> 'copyofsnap1/test1/163/15' 2022-06-27T11:50:34.923 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/163/16' -> 'copyofsnap1/test1/163/16' 2022-06-27T11:50:34.923 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/163/17' -> 'copyofsnap1/test1/163/17' 2022-06-27T11:50:34.923 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/163/18' -> 'copyofsnap1/test1/163/18' 2022-06-27T11:50:34.924 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/163/19' -> 'copyofsnap1/test1/163/19' 2022-06-27T11:50:34.924 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/163/20' -> 'copyofsnap1/test1/163/20' 2022-06-27T11:50:34.924 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/164' -> 'copyofsnap1/test1/164' 2022-06-27T11:50:34.924 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/164/10' -> 'copyofsnap1/test1/164/10' 2022-06-27T11:50:34.925 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/164/11' -> 'copyofsnap1/test1/164/11' 2022-06-27T11:50:34.925 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/164/12' -> 'copyofsnap1/test1/164/12' 2022-06-27T11:50:34.925 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/164/13' -> 'copyofsnap1/test1/164/13' 2022-06-27T11:50:34.926 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/164/14' -> 'copyofsnap1/test1/164/14' 2022-06-27T11:50:34.926 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/164/15' -> 'copyofsnap1/test1/164/15' 2022-06-27T11:50:34.926 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/164/16' -> 'copyofsnap1/test1/164/16' 2022-06-27T11:50:34.927 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/164/17' -> 'copyofsnap1/test1/164/17' 2022-06-27T11:50:34.927 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/164/18' -> 'copyofsnap1/test1/164/18' 2022-06-27T11:50:34.927 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/164/19' -> 'copyofsnap1/test1/164/19' 2022-06-27T11:50:34.927 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/164/20' -> 'copyofsnap1/test1/164/20' 2022-06-27T11:50:34.927 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/165' -> 'copyofsnap1/test1/165' 2022-06-27T11:50:34.928 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/165/10' -> 'copyofsnap1/test1/165/10' 2022-06-27T11:50:34.928 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/165/11' -> 'copyofsnap1/test1/165/11' 2022-06-27T11:50:34.929 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/165/12' -> 'copyofsnap1/test1/165/12' 2022-06-27T11:50:34.929 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/165/13' -> 'copyofsnap1/test1/165/13' 2022-06-27T11:50:34.930 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/165/14' -> 'copyofsnap1/test1/165/14' 2022-06-27T11:50:34.930 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/165/15' -> 'copyofsnap1/test1/165/15' 2022-06-27T11:50:34.930 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/165/16' -> 'copyofsnap1/test1/165/16' 2022-06-27T11:50:34.930 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/165/17' -> 'copyofsnap1/test1/165/17' 2022-06-27T11:50:34.931 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/165/18' -> 'copyofsnap1/test1/165/18' 2022-06-27T11:50:34.931 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/165/19' -> 'copyofsnap1/test1/165/19' 2022-06-27T11:50:34.931 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/165/20' -> 'copyofsnap1/test1/165/20' 2022-06-27T11:50:34.932 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/166' -> 'copyofsnap1/test1/166' 2022-06-27T11:50:34.932 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/166/10' -> 'copyofsnap1/test1/166/10' 2022-06-27T11:50:34.932 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/166/11' -> 'copyofsnap1/test1/166/11' 2022-06-27T11:50:34.932 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/166/12' -> 'copyofsnap1/test1/166/12' 2022-06-27T11:50:34.933 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/166/13' -> 'copyofsnap1/test1/166/13' 2022-06-27T11:50:34.933 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/166/14' -> 'copyofsnap1/test1/166/14' 2022-06-27T11:50:34.933 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/166/15' -> 'copyofsnap1/test1/166/15' 2022-06-27T11:50:34.933 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/166/16' -> 'copyofsnap1/test1/166/16' 2022-06-27T11:50:34.934 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/166/17' -> 'copyofsnap1/test1/166/17' 2022-06-27T11:50:34.934 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/166/18' -> 'copyofsnap1/test1/166/18' 2022-06-27T11:50:34.935 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/166/19' -> 'copyofsnap1/test1/166/19' 2022-06-27T11:50:34.935 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/166/20' -> 'copyofsnap1/test1/166/20' 2022-06-27T11:50:34.936 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/167' -> 'copyofsnap1/test1/167' 2022-06-27T11:50:34.936 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/167/10' -> 'copyofsnap1/test1/167/10' 2022-06-27T11:50:34.937 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/167/11' -> 'copyofsnap1/test1/167/11' 2022-06-27T11:50:34.937 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/167/12' -> 'copyofsnap1/test1/167/12' 2022-06-27T11:50:34.938 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/167/13' -> 'copyofsnap1/test1/167/13' 2022-06-27T11:50:34.938 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/167/14' -> 'copyofsnap1/test1/167/14' 2022-06-27T11:50:35.058 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/167/15' -> 'copyofsnap1/test1/167/15' 2022-06-27T11:50:35.059 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/167/16' -> 'copyofsnap1/test1/167/16' 2022-06-27T11:50:35.059 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/167/17' -> 'copyofsnap1/test1/167/17' 2022-06-27T11:50:35.060 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/167/18' -> 'copyofsnap1/test1/167/18' 2022-06-27T11:50:35.060 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/167/19' -> 'copyofsnap1/test1/167/19' 2022-06-27T11:50:35.060 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/167/20' -> 'copyofsnap1/test1/167/20' 2022-06-27T11:50:35.061 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/168' -> 'copyofsnap1/test1/168' 2022-06-27T11:50:35.061 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/168/10' -> 'copyofsnap1/test1/168/10' 2022-06-27T11:50:35.061 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/168/11' -> 'copyofsnap1/test1/168/11' 2022-06-27T11:50:35.061 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/168/12' -> 'copyofsnap1/test1/168/12' 2022-06-27T11:50:35.062 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/168/13' -> 'copyofsnap1/test1/168/13' 2022-06-27T11:50:35.063 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/168/14' -> 'copyofsnap1/test1/168/14' 2022-06-27T11:50:35.063 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/168/15' -> 'copyofsnap1/test1/168/15' 2022-06-27T11:50:35.064 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/168/16' -> 'copyofsnap1/test1/168/16' 2022-06-27T11:50:35.064 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/168/17' -> 'copyofsnap1/test1/168/17' 2022-06-27T11:50:35.065 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/168/18' -> 'copyofsnap1/test1/168/18' 2022-06-27T11:50:35.065 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/168/19' -> 'copyofsnap1/test1/168/19' 2022-06-27T11:50:35.066 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/168/20' -> 'copyofsnap1/test1/168/20' 2022-06-27T11:50:35.066 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/169' -> 'copyofsnap1/test1/169' 2022-06-27T11:50:35.066 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/169/10' -> 'copyofsnap1/test1/169/10' 2022-06-27T11:50:35.067 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/169/11' -> 'copyofsnap1/test1/169/11' 2022-06-27T11:50:35.067 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/169/12' -> 'copyofsnap1/test1/169/12' 2022-06-27T11:50:35.067 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/169/13' -> 'copyofsnap1/test1/169/13' 2022-06-27T11:50:35.068 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/169/14' -> 'copyofsnap1/test1/169/14' 2022-06-27T11:50:35.068 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/169/15' -> 'copyofsnap1/test1/169/15' 2022-06-27T11:50:35.069 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/169/16' -> 'copyofsnap1/test1/169/16' 2022-06-27T11:50:35.069 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/169/17' -> 'copyofsnap1/test1/169/17' 2022-06-27T11:50:35.070 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/169/18' -> 'copyofsnap1/test1/169/18' 2022-06-27T11:50:35.070 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/169/19' -> 'copyofsnap1/test1/169/19' 2022-06-27T11:50:35.070 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/169/20' -> 'copyofsnap1/test1/169/20' 2022-06-27T11:50:35.071 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/170' -> 'copyofsnap1/test1/170' 2022-06-27T11:50:35.071 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/170/10' -> 'copyofsnap1/test1/170/10' 2022-06-27T11:50:35.071 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/170/11' -> 'copyofsnap1/test1/170/11' 2022-06-27T11:50:35.072 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/170/12' -> 'copyofsnap1/test1/170/12' 2022-06-27T11:50:35.072 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/170/13' -> 'copyofsnap1/test1/170/13' 2022-06-27T11:50:35.072 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/170/14' -> 'copyofsnap1/test1/170/14' 2022-06-27T11:50:35.072 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/170/15' -> 'copyofsnap1/test1/170/15' 2022-06-27T11:50:35.072 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/170/16' -> 'copyofsnap1/test1/170/16' 2022-06-27T11:50:35.073 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/170/17' -> 'copyofsnap1/test1/170/17' 2022-06-27T11:50:35.073 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/170/18' -> 'copyofsnap1/test1/170/18' 2022-06-27T11:50:35.073 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/170/19' -> 'copyofsnap1/test1/170/19' 2022-06-27T11:50:35.073 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/170/20' -> 'copyofsnap1/test1/170/20' 2022-06-27T11:50:35.074 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/171' -> 'copyofsnap1/test1/171' 2022-06-27T11:50:35.074 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/171/10' -> 'copyofsnap1/test1/171/10' 2022-06-27T11:50:35.074 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/171/11' -> 'copyofsnap1/test1/171/11' 2022-06-27T11:50:35.075 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/171/12' -> 'copyofsnap1/test1/171/12' 2022-06-27T11:50:35.075 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/171/13' -> 'copyofsnap1/test1/171/13' 2022-06-27T11:50:35.075 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/171/14' -> 'copyofsnap1/test1/171/14' 2022-06-27T11:50:35.075 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/171/15' -> 'copyofsnap1/test1/171/15' 2022-06-27T11:50:35.076 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/171/16' -> 'copyofsnap1/test1/171/16' 2022-06-27T11:50:35.076 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/171/17' -> 'copyofsnap1/test1/171/17' 2022-06-27T11:50:35.076 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/171/18' -> 'copyofsnap1/test1/171/18' 2022-06-27T11:50:35.077 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/171/19' -> 'copyofsnap1/test1/171/19' 2022-06-27T11:50:35.077 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/171/20' -> 'copyofsnap1/test1/171/20' 2022-06-27T11:50:35.078 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/172' -> 'copyofsnap1/test1/172' 2022-06-27T11:50:35.078 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/172/10' -> 'copyofsnap1/test1/172/10' 2022-06-27T11:50:35.079 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/172/11' -> 'copyofsnap1/test1/172/11' 2022-06-27T11:50:35.080 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/172/12' -> 'copyofsnap1/test1/172/12' 2022-06-27T11:50:35.080 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/172/13' -> 'copyofsnap1/test1/172/13' 2022-06-27T11:50:35.081 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/172/14' -> 'copyofsnap1/test1/172/14' 2022-06-27T11:50:35.081 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/172/15' -> 'copyofsnap1/test1/172/15' 2022-06-27T11:50:35.082 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/172/16' -> 'copyofsnap1/test1/172/16' 2022-06-27T11:50:35.082 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/172/17' -> 'copyofsnap1/test1/172/17' 2022-06-27T11:50:35.083 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/172/18' -> 'copyofsnap1/test1/172/18' 2022-06-27T11:50:35.083 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/172/19' -> 'copyofsnap1/test1/172/19' 2022-06-27T11:50:35.083 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/172/20' -> 'copyofsnap1/test1/172/20' 2022-06-27T11:50:35.084 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/173' -> 'copyofsnap1/test1/173' 2022-06-27T11:50:35.084 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/173/10' -> 'copyofsnap1/test1/173/10' 2022-06-27T11:50:35.084 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/173/11' -> 'copyofsnap1/test1/173/11' 2022-06-27T11:50:35.085 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/173/12' -> 'copyofsnap1/test1/173/12' 2022-06-27T11:50:35.085 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/173/13' -> 'copyofsnap1/test1/173/13' 2022-06-27T11:50:35.085 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/173/14' -> 'copyofsnap1/test1/173/14' 2022-06-27T11:50:35.086 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/173/15' -> 'copyofsnap1/test1/173/15' 2022-06-27T11:50:35.086 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/173/16' -> 'copyofsnap1/test1/173/16' 2022-06-27T11:50:35.086 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/173/17' -> 'copyofsnap1/test1/173/17' 2022-06-27T11:50:35.086 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/173/18' -> 'copyofsnap1/test1/173/18' 2022-06-27T11:50:35.087 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/173/19' -> 'copyofsnap1/test1/173/19' 2022-06-27T11:50:35.087 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/173/20' -> 'copyofsnap1/test1/173/20' 2022-06-27T11:50:35.087 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/174' -> 'copyofsnap1/test1/174' 2022-06-27T11:50:35.087 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/174/10' -> 'copyofsnap1/test1/174/10' 2022-06-27T11:50:35.087 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/174/11' -> 'copyofsnap1/test1/174/11' 2022-06-27T11:50:35.602 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/174/12' -> 'copyofsnap1/test1/174/12' 2022-06-27T11:50:35.602 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/174/13' -> 'copyofsnap1/test1/174/13' 2022-06-27T11:50:35.603 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/174/14' -> 'copyofsnap1/test1/174/14' 2022-06-27T11:50:35.603 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/174/15' -> 'copyofsnap1/test1/174/15' 2022-06-27T11:50:35.603 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/174/16' -> 'copyofsnap1/test1/174/16' 2022-06-27T11:50:35.604 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/174/17' -> 'copyofsnap1/test1/174/17' 2022-06-27T11:50:35.604 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/174/18' -> 'copyofsnap1/test1/174/18' 2022-06-27T11:50:35.605 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/174/19' -> 'copyofsnap1/test1/174/19' 2022-06-27T11:50:35.605 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/174/20' -> 'copyofsnap1/test1/174/20' 2022-06-27T11:50:35.607 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/175' -> 'copyofsnap1/test1/175' 2022-06-27T11:50:35.608 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/175/10' -> 'copyofsnap1/test1/175/10' 2022-06-27T11:50:35.608 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/175/11' -> 'copyofsnap1/test1/175/11' 2022-06-27T11:50:35.609 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/175/12' -> 'copyofsnap1/test1/175/12' 2022-06-27T11:50:35.609 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/175/13' -> 'copyofsnap1/test1/175/13' 2022-06-27T11:50:35.610 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/175/14' -> 'copyofsnap1/test1/175/14' 2022-06-27T11:50:35.611 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/175/15' -> 'copyofsnap1/test1/175/15' 2022-06-27T11:50:35.611 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/175/16' -> 'copyofsnap1/test1/175/16' 2022-06-27T11:50:35.612 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/175/17' -> 'copyofsnap1/test1/175/17' 2022-06-27T11:50:35.612 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/175/18' -> 'copyofsnap1/test1/175/18' 2022-06-27T11:50:35.612 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/175/19' -> 'copyofsnap1/test1/175/19' 2022-06-27T11:50:35.613 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/175/20' -> 'copyofsnap1/test1/175/20' 2022-06-27T11:50:35.613 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/176' -> 'copyofsnap1/test1/176' 2022-06-27T11:50:35.614 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/176/10' -> 'copyofsnap1/test1/176/10' 2022-06-27T11:50:35.614 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/176/11' -> 'copyofsnap1/test1/176/11' 2022-06-27T11:50:35.615 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/176/12' -> 'copyofsnap1/test1/176/12' 2022-06-27T11:50:35.615 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/176/13' -> 'copyofsnap1/test1/176/13' 2022-06-27T11:50:35.616 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/176/14' -> 'copyofsnap1/test1/176/14' 2022-06-27T11:50:35.616 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/176/15' -> 'copyofsnap1/test1/176/15' 2022-06-27T11:50:35.617 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/176/16' -> 'copyofsnap1/test1/176/16' 2022-06-27T11:50:35.618 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/176/17' -> 'copyofsnap1/test1/176/17' 2022-06-27T11:50:35.618 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/176/18' -> 'copyofsnap1/test1/176/18' 2022-06-27T11:50:35.619 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/176/19' -> 'copyofsnap1/test1/176/19' 2022-06-27T11:50:35.620 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/176/20' -> 'copyofsnap1/test1/176/20' 2022-06-27T11:50:35.620 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/177' -> 'copyofsnap1/test1/177' 2022-06-27T11:50:35.621 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/177/10' -> 'copyofsnap1/test1/177/10' 2022-06-27T11:50:35.621 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/177/11' -> 'copyofsnap1/test1/177/11' 2022-06-27T11:50:35.622 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/177/12' -> 'copyofsnap1/test1/177/12' 2022-06-27T11:50:35.622 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/177/13' -> 'copyofsnap1/test1/177/13' 2022-06-27T11:50:35.628 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/177/14' -> 'copyofsnap1/test1/177/14' 2022-06-27T11:50:35.629 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/177/15' -> 'copyofsnap1/test1/177/15' 2022-06-27T11:50:35.629 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/177/16' -> 'copyofsnap1/test1/177/16' 2022-06-27T11:50:35.629 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/177/17' -> 'copyofsnap1/test1/177/17' 2022-06-27T11:50:35.629 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/177/18' -> 'copyofsnap1/test1/177/18' 2022-06-27T11:50:35.630 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/177/19' -> 'copyofsnap1/test1/177/19' 2022-06-27T11:50:35.630 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/177/20' -> 'copyofsnap1/test1/177/20' 2022-06-27T11:50:35.630 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/178' -> 'copyofsnap1/test1/178' 2022-06-27T11:50:35.630 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/178/10' -> 'copyofsnap1/test1/178/10' 2022-06-27T11:50:35.631 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/178/11' -> 'copyofsnap1/test1/178/11' 2022-06-27T11:50:35.631 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/178/12' -> 'copyofsnap1/test1/178/12' 2022-06-27T11:50:35.631 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/178/13' -> 'copyofsnap1/test1/178/13' 2022-06-27T11:50:35.631 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/178/14' -> 'copyofsnap1/test1/178/14' 2022-06-27T11:50:35.631 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/178/15' -> 'copyofsnap1/test1/178/15' 2022-06-27T11:50:35.632 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/178/16' -> 'copyofsnap1/test1/178/16' 2022-06-27T11:50:35.632 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/178/17' -> 'copyofsnap1/test1/178/17' 2022-06-27T11:50:35.632 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/178/18' -> 'copyofsnap1/test1/178/18' 2022-06-27T11:50:35.632 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/178/19' -> 'copyofsnap1/test1/178/19' 2022-06-27T11:50:35.638 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/178/20' -> 'copyofsnap1/test1/178/20' 2022-06-27T11:50:35.638 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/179' -> 'copyofsnap1/test1/179' 2022-06-27T11:50:35.639 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/179/10' -> 'copyofsnap1/test1/179/10' 2022-06-27T11:50:35.639 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/179/11' -> 'copyofsnap1/test1/179/11' 2022-06-27T11:50:35.640 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/179/12' -> 'copyofsnap1/test1/179/12' 2022-06-27T11:50:35.640 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/179/13' -> 'copyofsnap1/test1/179/13' 2022-06-27T11:50:35.641 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/179/14' -> 'copyofsnap1/test1/179/14' 2022-06-27T11:50:35.641 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/179/15' -> 'copyofsnap1/test1/179/15' 2022-06-27T11:50:35.642 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/179/16' -> 'copyofsnap1/test1/179/16' 2022-06-27T11:50:35.642 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/179/17' -> 'copyofsnap1/test1/179/17' 2022-06-27T11:50:35.643 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/179/18' -> 'copyofsnap1/test1/179/18' 2022-06-27T11:50:35.643 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/179/19' -> 'copyofsnap1/test1/179/19' 2022-06-27T11:50:35.644 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/179/20' -> 'copyofsnap1/test1/179/20' 2022-06-27T11:50:35.644 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/180' -> 'copyofsnap1/test1/180' 2022-06-27T11:50:35.645 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/180/10' -> 'copyofsnap1/test1/180/10' 2022-06-27T11:50:35.645 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/180/11' -> 'copyofsnap1/test1/180/11' 2022-06-27T11:50:35.646 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/180/12' -> 'copyofsnap1/test1/180/12' 2022-06-27T11:50:35.647 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/180/13' -> 'copyofsnap1/test1/180/13' 2022-06-27T11:50:35.647 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/180/14' -> 'copyofsnap1/test1/180/14' 2022-06-27T11:50:35.648 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/180/15' -> 'copyofsnap1/test1/180/15' 2022-06-27T11:50:35.648 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/180/16' -> 'copyofsnap1/test1/180/16' 2022-06-27T11:50:35.649 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/180/17' -> 'copyofsnap1/test1/180/17' 2022-06-27T11:50:35.649 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/180/18' -> 'copyofsnap1/test1/180/18' 2022-06-27T11:50:35.650 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/180/19' -> 'copyofsnap1/test1/180/19' 2022-06-27T11:50:35.650 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/180/20' -> 'copyofsnap1/test1/180/20' 2022-06-27T11:50:35.746 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/181' -> 'copyofsnap1/test1/181' 2022-06-27T11:50:35.746 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/181/10' -> 'copyofsnap1/test1/181/10' 2022-06-27T11:50:35.747 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/181/11' -> 'copyofsnap1/test1/181/11' 2022-06-27T11:50:35.747 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/181/12' -> 'copyofsnap1/test1/181/12' 2022-06-27T11:50:35.747 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/181/13' -> 'copyofsnap1/test1/181/13' 2022-06-27T11:50:35.748 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/181/14' -> 'copyofsnap1/test1/181/14' 2022-06-27T11:50:35.748 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/181/15' -> 'copyofsnap1/test1/181/15' 2022-06-27T11:50:35.748 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/181/16' -> 'copyofsnap1/test1/181/16' 2022-06-27T11:50:35.749 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/181/17' -> 'copyofsnap1/test1/181/17' 2022-06-27T11:50:35.749 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/181/18' -> 'copyofsnap1/test1/181/18' 2022-06-27T11:50:35.749 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/181/19' -> 'copyofsnap1/test1/181/19' 2022-06-27T11:50:35.749 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/181/20' -> 'copyofsnap1/test1/181/20' 2022-06-27T11:50:35.750 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/182' -> 'copyofsnap1/test1/182' 2022-06-27T11:50:35.750 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/182/10' -> 'copyofsnap1/test1/182/10' 2022-06-27T11:50:35.751 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/182/11' -> 'copyofsnap1/test1/182/11' 2022-06-27T11:50:35.751 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/182/12' -> 'copyofsnap1/test1/182/12' 2022-06-27T11:50:35.751 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/182/13' -> 'copyofsnap1/test1/182/13' 2022-06-27T11:50:35.751 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/182/14' -> 'copyofsnap1/test1/182/14' 2022-06-27T11:50:35.751 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/182/15' -> 'copyofsnap1/test1/182/15' 2022-06-27T11:50:35.752 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/182/16' -> 'copyofsnap1/test1/182/16' 2022-06-27T11:50:35.752 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/182/17' -> 'copyofsnap1/test1/182/17' 2022-06-27T11:50:35.752 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/182/18' -> 'copyofsnap1/test1/182/18' 2022-06-27T11:50:35.753 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/182/19' -> 'copyofsnap1/test1/182/19' 2022-06-27T11:50:35.753 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/182/20' -> 'copyofsnap1/test1/182/20' 2022-06-27T11:50:35.753 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/183' -> 'copyofsnap1/test1/183' 2022-06-27T11:50:35.754 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/183/10' -> 'copyofsnap1/test1/183/10' 2022-06-27T11:50:35.755 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/183/11' -> 'copyofsnap1/test1/183/11' 2022-06-27T11:50:35.755 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/183/12' -> 'copyofsnap1/test1/183/12' 2022-06-27T11:50:35.755 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/183/13' -> 'copyofsnap1/test1/183/13' 2022-06-27T11:50:35.755 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/183/14' -> 'copyofsnap1/test1/183/14' 2022-06-27T11:50:35.756 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/183/15' -> 'copyofsnap1/test1/183/15' 2022-06-27T11:50:35.756 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/183/16' -> 'copyofsnap1/test1/183/16' 2022-06-27T11:50:35.756 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/183/17' -> 'copyofsnap1/test1/183/17' 2022-06-27T11:50:35.756 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/183/18' -> 'copyofsnap1/test1/183/18' 2022-06-27T11:50:35.757 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/183/19' -> 'copyofsnap1/test1/183/19' 2022-06-27T11:50:35.757 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/183/20' -> 'copyofsnap1/test1/183/20' 2022-06-27T11:50:35.757 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/184' -> 'copyofsnap1/test1/184' 2022-06-27T11:50:35.757 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/184/10' -> 'copyofsnap1/test1/184/10' 2022-06-27T11:50:35.758 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/184/11' -> 'copyofsnap1/test1/184/11' 2022-06-27T11:50:35.758 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/184/12' -> 'copyofsnap1/test1/184/12' 2022-06-27T11:50:35.758 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/184/13' -> 'copyofsnap1/test1/184/13' 2022-06-27T11:50:35.758 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/184/14' -> 'copyofsnap1/test1/184/14' 2022-06-27T11:50:35.759 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/184/15' -> 'copyofsnap1/test1/184/15' 2022-06-27T11:50:35.759 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/184/16' -> 'copyofsnap1/test1/184/16' 2022-06-27T11:50:35.759 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/184/17' -> 'copyofsnap1/test1/184/17' 2022-06-27T11:50:35.759 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/184/18' -> 'copyofsnap1/test1/184/18' 2022-06-27T11:50:35.760 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/184/19' -> 'copyofsnap1/test1/184/19' 2022-06-27T11:50:35.760 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/184/20' -> 'copyofsnap1/test1/184/20' 2022-06-27T11:50:35.760 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/185' -> 'copyofsnap1/test1/185' 2022-06-27T11:50:35.760 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/185/10' -> 'copyofsnap1/test1/185/10' 2022-06-27T11:50:35.761 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/185/11' -> 'copyofsnap1/test1/185/11' 2022-06-27T11:50:35.761 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/185/12' -> 'copyofsnap1/test1/185/12' 2022-06-27T11:50:35.761 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/185/13' -> 'copyofsnap1/test1/185/13' 2022-06-27T11:50:35.762 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/185/14' -> 'copyofsnap1/test1/185/14' 2022-06-27T11:50:35.762 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/185/15' -> 'copyofsnap1/test1/185/15' 2022-06-27T11:50:35.763 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/185/16' -> 'copyofsnap1/test1/185/16' 2022-06-27T11:50:35.763 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/185/17' -> 'copyofsnap1/test1/185/17' 2022-06-27T11:50:35.763 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/185/18' -> 'copyofsnap1/test1/185/18' 2022-06-27T11:50:35.763 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/185/19' -> 'copyofsnap1/test1/185/19' 2022-06-27T11:50:35.764 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/185/20' -> 'copyofsnap1/test1/185/20' 2022-06-27T11:50:35.764 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/186' -> 'copyofsnap1/test1/186' 2022-06-27T11:50:35.765 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/186/10' -> 'copyofsnap1/test1/186/10' 2022-06-27T11:50:35.765 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/186/11' -> 'copyofsnap1/test1/186/11' 2022-06-27T11:50:35.765 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/186/12' -> 'copyofsnap1/test1/186/12' 2022-06-27T11:50:35.766 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/186/13' -> 'copyofsnap1/test1/186/13' 2022-06-27T11:50:35.766 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/186/14' -> 'copyofsnap1/test1/186/14' 2022-06-27T11:50:35.766 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/186/15' -> 'copyofsnap1/test1/186/15' 2022-06-27T11:50:35.766 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/186/16' -> 'copyofsnap1/test1/186/16' 2022-06-27T11:50:35.767 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/186/17' -> 'copyofsnap1/test1/186/17' 2022-06-27T11:50:35.767 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/186/18' -> 'copyofsnap1/test1/186/18' 2022-06-27T11:50:35.767 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/186/19' -> 'copyofsnap1/test1/186/19' 2022-06-27T11:50:35.767 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/186/20' -> 'copyofsnap1/test1/186/20' 2022-06-27T11:50:35.768 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/187' -> 'copyofsnap1/test1/187' 2022-06-27T11:50:35.768 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/187/10' -> 'copyofsnap1/test1/187/10' 2022-06-27T11:50:35.768 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/187/11' -> 'copyofsnap1/test1/187/11' 2022-06-27T11:50:35.768 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/187/12' -> 'copyofsnap1/test1/187/12' 2022-06-27T11:50:35.768 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/187/13' -> 'copyofsnap1/test1/187/13' 2022-06-27T11:50:35.769 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/187/14' -> 'copyofsnap1/test1/187/14' 2022-06-27T11:50:35.770 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/187/15' -> 'copyofsnap1/test1/187/15' 2022-06-27T11:50:35.770 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/187/16' -> 'copyofsnap1/test1/187/16' 2022-06-27T11:50:35.770 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/187/17' -> 'copyofsnap1/test1/187/17' 2022-06-27T11:50:36.982 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/187/18' -> 'copyofsnap1/test1/187/18' 2022-06-27T11:50:36.982 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/187/19' -> 'copyofsnap1/test1/187/19' 2022-06-27T11:50:36.983 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/187/20' -> 'copyofsnap1/test1/187/20' 2022-06-27T11:50:36.983 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/188' -> 'copyofsnap1/test1/188' 2022-06-27T11:50:36.984 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/188/10' -> 'copyofsnap1/test1/188/10' 2022-06-27T11:50:36.984 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/188/11' -> 'copyofsnap1/test1/188/11' 2022-06-27T11:50:36.985 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/188/12' -> 'copyofsnap1/test1/188/12' 2022-06-27T11:50:36.985 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/188/13' -> 'copyofsnap1/test1/188/13' 2022-06-27T11:50:36.985 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/188/14' -> 'copyofsnap1/test1/188/14' 2022-06-27T11:50:36.986 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/188/15' -> 'copyofsnap1/test1/188/15' 2022-06-27T11:50:36.986 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/188/16' -> 'copyofsnap1/test1/188/16' 2022-06-27T11:50:36.987 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/188/17' -> 'copyofsnap1/test1/188/17' 2022-06-27T11:50:36.987 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/188/18' -> 'copyofsnap1/test1/188/18' 2022-06-27T11:50:36.988 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/188/19' -> 'copyofsnap1/test1/188/19' 2022-06-27T11:50:36.988 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/188/20' -> 'copyofsnap1/test1/188/20' 2022-06-27T11:50:36.988 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/189' -> 'copyofsnap1/test1/189' 2022-06-27T11:50:36.989 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/189/10' -> 'copyofsnap1/test1/189/10' 2022-06-27T11:50:36.989 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/189/11' -> 'copyofsnap1/test1/189/11' 2022-06-27T11:50:36.990 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/189/12' -> 'copyofsnap1/test1/189/12' 2022-06-27T11:50:36.990 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/189/13' -> 'copyofsnap1/test1/189/13' 2022-06-27T11:50:36.990 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/189/14' -> 'copyofsnap1/test1/189/14' 2022-06-27T11:50:36.991 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/189/15' -> 'copyofsnap1/test1/189/15' 2022-06-27T11:50:36.991 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/189/16' -> 'copyofsnap1/test1/189/16' 2022-06-27T11:50:36.991 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/189/17' -> 'copyofsnap1/test1/189/17' 2022-06-27T11:50:36.992 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/189/18' -> 'copyofsnap1/test1/189/18' 2022-06-27T11:50:36.992 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/189/19' -> 'copyofsnap1/test1/189/19' 2022-06-27T11:50:36.993 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/189/20' -> 'copyofsnap1/test1/189/20' 2022-06-27T11:50:36.993 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/190' -> 'copyofsnap1/test1/190' 2022-06-27T11:50:36.994 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/190/10' -> 'copyofsnap1/test1/190/10' 2022-06-27T11:50:36.994 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/190/11' -> 'copyofsnap1/test1/190/11' 2022-06-27T11:50:36.995 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/190/12' -> 'copyofsnap1/test1/190/12' 2022-06-27T11:50:36.995 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/190/13' -> 'copyofsnap1/test1/190/13' 2022-06-27T11:50:36.995 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/190/14' -> 'copyofsnap1/test1/190/14' 2022-06-27T11:50:36.996 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/190/15' -> 'copyofsnap1/test1/190/15' 2022-06-27T11:50:36.996 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/190/16' -> 'copyofsnap1/test1/190/16' 2022-06-27T11:50:36.996 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/190/17' -> 'copyofsnap1/test1/190/17' 2022-06-27T11:50:36.997 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/190/18' -> 'copyofsnap1/test1/190/18' 2022-06-27T11:50:36.997 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/190/19' -> 'copyofsnap1/test1/190/19' 2022-06-27T11:50:36.997 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/190/20' -> 'copyofsnap1/test1/190/20' 2022-06-27T11:50:36.998 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/191' -> 'copyofsnap1/test1/191' 2022-06-27T11:50:36.998 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/191/10' -> 'copyofsnap1/test1/191/10' 2022-06-27T11:50:36.999 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/191/11' -> 'copyofsnap1/test1/191/11' 2022-06-27T11:50:36.999 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/191/12' -> 'copyofsnap1/test1/191/12' 2022-06-27T11:50:37.000 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/191/13' -> 'copyofsnap1/test1/191/13' 2022-06-27T11:50:37.000 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/191/14' -> 'copyofsnap1/test1/191/14' 2022-06-27T11:50:37.001 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/191/15' -> 'copyofsnap1/test1/191/15' 2022-06-27T11:50:37.001 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/191/16' -> 'copyofsnap1/test1/191/16' 2022-06-27T11:50:37.001 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/191/17' -> 'copyofsnap1/test1/191/17' 2022-06-27T11:50:37.006 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/191/18' -> 'copyofsnap1/test1/191/18' 2022-06-27T11:50:37.007 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/191/19' -> 'copyofsnap1/test1/191/19' 2022-06-27T11:50:37.007 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/191/20' -> 'copyofsnap1/test1/191/20' 2022-06-27T11:50:37.008 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/192' -> 'copyofsnap1/test1/192' 2022-06-27T11:50:37.008 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/192/10' -> 'copyofsnap1/test1/192/10' 2022-06-27T11:50:37.009 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/192/11' -> 'copyofsnap1/test1/192/11' 2022-06-27T11:50:37.009 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/192/12' -> 'copyofsnap1/test1/192/12' 2022-06-27T11:50:37.010 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/192/13' -> 'copyofsnap1/test1/192/13' 2022-06-27T11:50:37.010 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/192/14' -> 'copyofsnap1/test1/192/14' 2022-06-27T11:50:37.011 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/192/15' -> 'copyofsnap1/test1/192/15' 2022-06-27T11:50:37.014 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/192/16' -> 'copyofsnap1/test1/192/16' 2022-06-27T11:50:37.015 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/192/17' -> 'copyofsnap1/test1/192/17' 2022-06-27T11:50:37.015 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/192/18' -> 'copyofsnap1/test1/192/18' 2022-06-27T11:50:37.016 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/192/19' -> 'copyofsnap1/test1/192/19' 2022-06-27T11:50:37.016 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/192/20' -> 'copyofsnap1/test1/192/20' 2022-06-27T11:50:37.017 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/193' -> 'copyofsnap1/test1/193' 2022-06-27T11:50:37.017 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/193/10' -> 'copyofsnap1/test1/193/10' 2022-06-27T11:50:37.018 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/193/11' -> 'copyofsnap1/test1/193/11' 2022-06-27T11:50:37.018 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/193/12' -> 'copyofsnap1/test1/193/12' 2022-06-27T11:50:37.019 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/193/13' -> 'copyofsnap1/test1/193/13' 2022-06-27T11:50:37.019 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/193/14' -> 'copyofsnap1/test1/193/14' 2022-06-27T11:50:37.019 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/193/15' -> 'copyofsnap1/test1/193/15' 2022-06-27T11:50:37.020 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/193/16' -> 'copyofsnap1/test1/193/16' 2022-06-27T11:50:37.020 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/193/17' -> 'copyofsnap1/test1/193/17' 2022-06-27T11:50:37.021 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/193/18' -> 'copyofsnap1/test1/193/18' 2022-06-27T11:50:37.021 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/193/19' -> 'copyofsnap1/test1/193/19' 2022-06-27T11:50:37.022 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/193/20' -> 'copyofsnap1/test1/193/20' 2022-06-27T11:50:37.023 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/194' -> 'copyofsnap1/test1/194' 2022-06-27T11:50:37.023 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/194/10' -> 'copyofsnap1/test1/194/10' 2022-06-27T11:50:37.024 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/194/11' -> 'copyofsnap1/test1/194/11' 2022-06-27T11:50:37.025 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/194/12' -> 'copyofsnap1/test1/194/12' 2022-06-27T11:50:37.030 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/194/13' -> 'copyofsnap1/test1/194/13' 2022-06-27T11:50:37.031 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/194/14' -> 'copyofsnap1/test1/194/14' 2022-06-27T11:50:37.562 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/194/15' -> 'copyofsnap1/test1/194/15' 2022-06-27T11:50:37.563 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/194/16' -> 'copyofsnap1/test1/194/16' 2022-06-27T11:50:37.563 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/194/17' -> 'copyofsnap1/test1/194/17' 2022-06-27T11:50:37.564 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/194/18' -> 'copyofsnap1/test1/194/18' 2022-06-27T11:50:37.564 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/194/19' -> 'copyofsnap1/test1/194/19' 2022-06-27T11:50:37.565 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/194/20' -> 'copyofsnap1/test1/194/20' 2022-06-27T11:50:37.565 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/195' -> 'copyofsnap1/test1/195' 2022-06-27T11:50:37.568 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/195/10' -> 'copyofsnap1/test1/195/10' 2022-06-27T11:50:37.568 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/195/11' -> 'copyofsnap1/test1/195/11' 2022-06-27T11:50:37.569 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/195/12' -> 'copyofsnap1/test1/195/12' 2022-06-27T11:50:37.569 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/195/13' -> 'copyofsnap1/test1/195/13' 2022-06-27T11:50:37.570 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/195/14' -> 'copyofsnap1/test1/195/14' 2022-06-27T11:50:37.570 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/195/15' -> 'copyofsnap1/test1/195/15' 2022-06-27T11:50:37.571 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/195/16' -> 'copyofsnap1/test1/195/16' 2022-06-27T11:50:37.571 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/195/17' -> 'copyofsnap1/test1/195/17' 2022-06-27T11:50:37.572 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/195/18' -> 'copyofsnap1/test1/195/18' 2022-06-27T11:50:37.572 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/195/19' -> 'copyofsnap1/test1/195/19' 2022-06-27T11:50:37.573 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/195/20' -> 'copyofsnap1/test1/195/20' 2022-06-27T11:50:37.573 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/196' -> 'copyofsnap1/test1/196' 2022-06-27T11:50:37.574 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/196/10' -> 'copyofsnap1/test1/196/10' 2022-06-27T11:50:37.574 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/196/11' -> 'copyofsnap1/test1/196/11' 2022-06-27T11:50:37.574 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/196/12' -> 'copyofsnap1/test1/196/12' 2022-06-27T11:50:37.575 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/196/13' -> 'copyofsnap1/test1/196/13' 2022-06-27T11:50:37.575 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/196/14' -> 'copyofsnap1/test1/196/14' 2022-06-27T11:50:37.576 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/196/15' -> 'copyofsnap1/test1/196/15' 2022-06-27T11:50:37.576 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/196/16' -> 'copyofsnap1/test1/196/16' 2022-06-27T11:50:37.577 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/196/17' -> 'copyofsnap1/test1/196/17' 2022-06-27T11:50:37.577 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/196/18' -> 'copyofsnap1/test1/196/18' 2022-06-27T11:50:37.578 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/196/19' -> 'copyofsnap1/test1/196/19' 2022-06-27T11:50:37.579 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/196/20' -> 'copyofsnap1/test1/196/20' 2022-06-27T11:50:37.579 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/197' -> 'copyofsnap1/test1/197' 2022-06-27T11:50:37.579 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/197/10' -> 'copyofsnap1/test1/197/10' 2022-06-27T11:50:37.580 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/197/11' -> 'copyofsnap1/test1/197/11' 2022-06-27T11:50:37.580 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/197/12' -> 'copyofsnap1/test1/197/12' 2022-06-27T11:50:37.581 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/197/13' -> 'copyofsnap1/test1/197/13' 2022-06-27T11:50:37.581 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/197/14' -> 'copyofsnap1/test1/197/14' 2022-06-27T11:50:37.581 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/197/15' -> 'copyofsnap1/test1/197/15' 2022-06-27T11:50:37.582 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/197/16' -> 'copyofsnap1/test1/197/16' 2022-06-27T11:50:37.582 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/197/17' -> 'copyofsnap1/test1/197/17' 2022-06-27T11:50:37.583 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/197/18' -> 'copyofsnap1/test1/197/18' 2022-06-27T11:50:37.583 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/197/19' -> 'copyofsnap1/test1/197/19' 2022-06-27T11:50:37.584 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/197/20' -> 'copyofsnap1/test1/197/20' 2022-06-27T11:50:37.584 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/198' -> 'copyofsnap1/test1/198' 2022-06-27T11:50:37.585 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/198/10' -> 'copyofsnap1/test1/198/10' 2022-06-27T11:50:37.585 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/198/11' -> 'copyofsnap1/test1/198/11' 2022-06-27T11:50:37.586 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/198/12' -> 'copyofsnap1/test1/198/12' 2022-06-27T11:50:37.586 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/198/13' -> 'copyofsnap1/test1/198/13' 2022-06-27T11:50:37.587 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/198/14' -> 'copyofsnap1/test1/198/14' 2022-06-27T11:50:37.587 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/198/15' -> 'copyofsnap1/test1/198/15' 2022-06-27T11:50:37.588 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/198/16' -> 'copyofsnap1/test1/198/16' 2022-06-27T11:50:37.588 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/198/17' -> 'copyofsnap1/test1/198/17' 2022-06-27T11:50:37.589 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/198/18' -> 'copyofsnap1/test1/198/18' 2022-06-27T11:50:37.589 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/198/19' -> 'copyofsnap1/test1/198/19' 2022-06-27T11:50:37.590 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/198/20' -> 'copyofsnap1/test1/198/20' 2022-06-27T11:50:37.590 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/199' -> 'copyofsnap1/test1/199' 2022-06-27T11:50:37.591 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/199/10' -> 'copyofsnap1/test1/199/10' 2022-06-27T11:50:37.591 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/199/11' -> 'copyofsnap1/test1/199/11' 2022-06-27T11:50:37.592 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/199/12' -> 'copyofsnap1/test1/199/12' 2022-06-27T11:50:37.592 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/199/13' -> 'copyofsnap1/test1/199/13' 2022-06-27T11:50:37.593 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/199/14' -> 'copyofsnap1/test1/199/14' 2022-06-27T11:50:37.593 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/199/15' -> 'copyofsnap1/test1/199/15' 2022-06-27T11:50:37.594 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/199/16' -> 'copyofsnap1/test1/199/16' 2022-06-27T11:50:37.594 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/199/17' -> 'copyofsnap1/test1/199/17' 2022-06-27T11:50:37.594 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/199/18' -> 'copyofsnap1/test1/199/18' 2022-06-27T11:50:37.595 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/199/19' -> 'copyofsnap1/test1/199/19' 2022-06-27T11:50:37.595 INFO:tasks.workunit.client.0.smithi195.stdout:'.snap/test1/199/20' -> 'copyofsnap1/test1/199/20' 2022-06-27T11:50:37.595 INFO:tasks.workunit.client.0.smithi195.stdout:Take third snapshot .snap/test3 2022-06-27T11:50:37.596 INFO:tasks.workunit.client.0.smithi195.stdout:Delete the snapshots... 2022-06-27T11:50:40.264 INFO:tasks.workunit.client.0.smithi195.stdout:.//.snap/test1 2022-06-27T11:50:40.265 INFO:tasks.workunit.client.0.smithi195.stdout:.//.snap/test2 2022-06-27T11:50:40.265 INFO:tasks.workunit.client.0.smithi195.stdout:.//.snap/test3 2022-06-27T11:50:40.424 INFO:tasks.workunit.client.0.smithi195.stdout:./274/.snap/snap-subdir-test 2022-06-27T11:50:40.432 INFO:tasks.workunit.client.0.smithi195.stdout:./166/.snap/snap-subdir-test 2022-06-27T11:50:40.439 INFO:tasks.workunit.client.0.smithi195.stdout:./256/.snap/snap-subdir-test 2022-06-27T11:50:40.964 INFO:tasks.workunit.client.0.smithi195.stdout:./247/.snap/snap-subdir-test 2022-06-27T11:50:40.969 INFO:tasks.workunit.client.0.smithi195.stdout:./248/.snap/snap-subdir-test 2022-06-27T11:50:40.974 INFO:tasks.workunit.client.0.smithi195.stdout:./147/.snap/snap-subdir-test 2022-06-27T11:50:40.979 INFO:tasks.workunit.client.0.smithi195.stdout:./151/.snap/snap-subdir-test 2022-06-27T11:50:40.984 INFO:tasks.workunit.client.0.smithi195.stdout:./170/.snap/snap-subdir-test 2022-06-27T11:50:40.989 INFO:tasks.workunit.client.0.smithi195.stdout:./167/.snap/snap-subdir-test 2022-06-27T11:50:42.282 INFO:tasks.workunit.client.0.smithi195.stdout:./140/.snap/snap-subdir-test 2022-06-27T11:50:42.286 INFO:tasks.workunit.client.0.smithi195.stdout:./142/.snap/snap-subdir-test 2022-06-27T11:50:42.292 INFO:tasks.workunit.client.0.smithi195.stdout:./150/.snap/snap-subdir-test 2022-06-27T11:50:42.303 INFO:tasks.workunit.client.0.smithi195.stdout:./273/.snap/snap-subdir-test 2022-06-27T11:50:42.304 INFO:tasks.workunit.client.0.smithi195.stdout:./269/.snap/snap-subdir-test 2022-06-27T11:50:42.311 INFO:tasks.workunit.client.0.smithi195.stdout:./105/.snap/snap-subdir-test 2022-06-27T11:50:42.319 INFO:tasks.workunit.client.0.smithi195.stdout:./210/.snap/snap-subdir-test 2022-06-27T11:50:43.117 INFO:tasks.workunit.client.0.smithi195.stdout:./265/.snap/snap-subdir-test 2022-06-27T11:50:43.122 INFO:tasks.workunit.client.0.smithi195.stdout:./218/.snap/snap-subdir-test 2022-06-27T11:50:43.133 INFO:tasks.workunit.client.0.smithi195.stdout:./290/.snap/snap-subdir-test 2022-06-27T11:50:43.133 INFO:tasks.workunit.client.0.smithi195.stdout:./149/.snap/snap-subdir-test 2022-06-27T11:50:43.138 INFO:tasks.workunit.client.0.smithi195.stdout:./109/.snap/snap-subdir-test 2022-06-27T11:50:43.143 INFO:tasks.workunit.client.0.smithi195.stdout:./292/.snap/snap-subdir-test 2022-06-27T11:50:43.146 INFO:tasks.workunit.client.0.smithi195.stdout:./242/.snap/snap-subdir-test 2022-06-27T11:50:43.164 INFO:tasks.workunit.client.0.smithi195.stdout:./158/.snap/snap-subdir-test 2022-06-27T11:50:43.165 INFO:tasks.workunit.client.0.smithi195.stdout:./243/.snap/snap-subdir-test 2022-06-27T11:50:43.165 INFO:tasks.workunit.client.0.smithi195.stdout:./106/.snap/snap-subdir-test 2022-06-27T11:50:43.166 INFO:tasks.workunit.client.0.smithi195.stdout:./250/.snap/snap-subdir-test 2022-06-27T11:50:43.178 INFO:tasks.workunit.client.0.smithi195.stdout:./221/.snap/snap-subdir-test 2022-06-27T11:50:43.179 INFO:tasks.workunit.client.0.smithi195.stdout:./244/.snap/snap-subdir-test 2022-06-27T11:50:43.180 INFO:tasks.workunit.client.0.smithi195.stdout:./258/.snap/snap-subdir-test 2022-06-27T11:50:43.190 INFO:tasks.workunit.client.0.smithi195.stdout:./268/.snap/snap-subdir-test 2022-06-27T11:50:43.191 INFO:tasks.workunit.client.0.smithi195.stdout:./189/.snap/snap-subdir-test 2022-06-27T11:50:43.193 INFO:tasks.workunit.client.0.smithi195.stdout:./285/.snap/snap-subdir-test 2022-06-27T11:50:43.224 INFO:tasks.workunit.client.0.smithi195.stdout:./208/.snap/snap-subdir-test 2022-06-27T11:50:43.225 INFO:tasks.workunit.client.0.smithi195.stdout:./288/.snap/snap-subdir-test 2022-06-27T11:50:43.226 INFO:tasks.workunit.client.0.smithi195.stdout:./209/.snap/snap-subdir-test 2022-06-27T11:50:43.227 INFO:tasks.workunit.client.0.smithi195.stdout:./253/.snap/snap-subdir-test 2022-06-27T11:50:43.227 INFO:tasks.workunit.client.0.smithi195.stdout:./249/.snap/snap-subdir-test 2022-06-27T11:50:43.228 INFO:tasks.workunit.client.0.smithi195.stdout:./133/.snap/snap-subdir-test 2022-06-27T11:50:43.239 INFO:tasks.workunit.client.0.smithi195.stdout:./291/.snap/snap-subdir-test 2022-06-27T11:50:43.240 INFO:tasks.workunit.client.0.smithi195.stdout:./226/.snap/snap-subdir-test 2022-06-27T11:50:43.240 INFO:tasks.workunit.client.0.smithi195.stdout:./173/.snap/snap-subdir-test 2022-06-27T11:50:43.241 INFO:tasks.workunit.client.0.smithi195.stdout:./174/.snap/snap-subdir-test 2022-06-27T11:50:43.241 INFO:tasks.workunit.client.0.smithi195.stdout:./299/.snap/snap-subdir-test 2022-06-27T11:50:43.246 INFO:tasks.workunit.client.0.smithi195.stdout:./262/.snap/snap-subdir-test 2022-06-27T11:50:43.250 INFO:tasks.workunit.client.0.smithi195.stdout:./231/.snap/snap-subdir-test 2022-06-27T11:50:43.251 INFO:tasks.workunit.client.0.smithi195.stdout:./214/.snap/snap-subdir-test 2022-06-27T11:50:43.252 INFO:tasks.workunit.client.0.smithi195.stdout:./281/.snap/snap-subdir-test 2022-06-27T11:50:43.256 INFO:tasks.workunit.client.0.smithi195.stdout:./289/.snap/snap-subdir-test 2022-06-27T11:50:43.268 INFO:tasks.workunit.client.0.smithi195.stdout:./283/.snap/snap-subdir-test 2022-06-27T11:50:43.277 INFO:tasks.workunit.client.0.smithi195.stdout:./232/.snap/snap-subdir-test 2022-06-27T11:50:43.278 INFO:tasks.workunit.client.0.smithi195.stdout:./298/.snap/snap-subdir-test 2022-06-27T11:50:43.278 INFO:tasks.workunit.client.0.smithi195.stdout:./233/.snap/snap-subdir-test 2022-06-27T11:50:43.279 INFO:tasks.workunit.client.0.smithi195.stdout:./155/.snap/snap-subdir-test 2022-06-27T11:50:43.289 INFO:tasks.workunit.client.0.smithi195.stdout:./114/.snap/snap-subdir-test 2022-06-27T11:50:43.289 INFO:tasks.workunit.client.0.smithi195.stdout:./159/.snap/snap-subdir-test 2022-06-27T11:50:43.295 INFO:tasks.workunit.client.0.smithi195.stdout:./187/.snap/snap-subdir-test 2022-06-27T11:50:43.297 INFO:tasks.workunit.client.0.smithi195.stdout:./185/.snap/snap-subdir-test 2022-06-27T11:50:43.298 INFO:tasks.workunit.client.0.smithi195.stdout:./293/.snap/snap-subdir-test 2022-06-27T11:50:43.315 INFO:tasks.workunit.client.0.smithi195.stdout:./228/.snap/snap-subdir-test 2022-06-27T11:50:43.315 INFO:tasks.workunit.client.0.smithi195.stdout:./120/.snap/snap-subdir-test 2022-06-27T11:50:43.316 INFO:tasks.workunit.client.0.smithi195.stdout:./110/.snap/snap-subdir-test 2022-06-27T11:50:43.329 INFO:tasks.workunit.client.0.smithi195.stdout:./267/.snap/snap-subdir-test 2022-06-27T11:50:43.333 INFO:tasks.workunit.client.0.smithi195.stdout:./202/.snap/snap-subdir-test 2022-06-27T11:50:43.333 INFO:tasks.workunit.client.0.smithi195.stdout:./137/.snap/snap-subdir-test 2022-06-27T11:50:43.347 INFO:tasks.workunit.client.0.smithi195.stdout:./246/.snap/snap-subdir-test 2022-06-27T11:50:43.348 INFO:tasks.workunit.client.0.smithi195.stdout:./219/.snap/snap-subdir-test 2022-06-27T11:50:43.350 INFO:tasks.workunit.client.0.smithi195.stdout:./198/.snap/snap-subdir-test 2022-06-27T11:50:43.351 INFO:tasks.workunit.client.0.smithi195.stdout:./136/.snap/snap-subdir-test 2022-06-27T11:50:43.351 INFO:tasks.workunit.client.0.smithi195.stdout:./270/.snap/snap-subdir-test 2022-06-27T11:50:43.352 INFO:tasks.workunit.client.0.smithi195.stdout:./125/.snap/snap-subdir-test 2022-06-27T11:50:43.630 INFO:tasks.workunit.client.0.smithi195.stdout:./194/.snap/snap-subdir-test 2022-06-27T11:50:43.636 INFO:tasks.workunit.client.0.smithi195.stdout:./119/.snap/snap-subdir-test 2022-06-27T11:50:43.641 INFO:tasks.workunit.client.0.smithi195.stdout:./297/.snap/snap-subdir-test 2022-06-27T11:50:43.642 INFO:tasks.workunit.client.0.smithi195.stdout:./230/.snap/snap-subdir-test 2022-06-27T11:50:43.653 INFO:tasks.workunit.client.0.smithi195.stdout:./260/.snap/snap-subdir-test 2022-06-27T11:50:43.690 INFO:tasks.workunit.client.0.smithi195.stdout:./157/.snap/snap-subdir-test 2022-06-27T11:50:43.690 INFO:tasks.workunit.client.0.smithi195.stdout:./172/.snap/snap-subdir-test 2022-06-27T11:50:43.691 INFO:tasks.workunit.client.0.smithi195.stdout:./277/.snap/snap-subdir-test 2022-06-27T11:50:43.703 INFO:tasks.workunit.client.0.smithi195.stdout:./186/.snap/snap-subdir-test 2022-06-27T11:50:43.703 INFO:tasks.workunit.client.0.smithi195.stdout:./237/.snap/snap-subdir-test 2022-06-27T11:50:44.547 INFO:tasks.workunit.client.0.smithi195.stdout:./190/.snap/snap-subdir-test 2022-06-27T11:50:44.551 INFO:tasks.workunit.client.0.smithi195.stdout:./213/.snap/snap-subdir-test 2022-06-27T11:50:44.558 INFO:tasks.workunit.client.0.smithi195.stdout:./104/.snap/snap-subdir-test 2022-06-27T11:50:44.563 INFO:tasks.workunit.client.0.smithi195.stdout:./164/.snap/snap-subdir-test 2022-06-27T11:50:44.568 INFO:tasks.workunit.client.0.smithi195.stdout:./112/.snap/snap-subdir-test 2022-06-27T11:50:44.577 INFO:tasks.workunit.client.0.smithi195.stdout:./295/.snap/snap-subdir-test 2022-06-27T11:50:44.586 INFO:tasks.workunit.client.0.smithi195.stdout:./272/.snap/snap-subdir-test 2022-06-27T11:50:44.586 INFO:tasks.workunit.client.0.smithi195.stdout:./100/.snap/snap-subdir-test 2022-06-27T11:50:44.589 INFO:tasks.workunit.client.0.smithi195.stdout:./134/.snap/snap-subdir-test 2022-06-27T11:50:44.594 INFO:tasks.workunit.client.0.smithi195.stdout:./200/.snap/snap-subdir-test 2022-06-27T11:50:44.876 INFO:tasks.workunit.client.0.smithi195.stdout:./115/.snap/snap-subdir-test 2022-06-27T11:50:44.883 INFO:tasks.workunit.client.0.smithi195.stdout:./284/.snap/snap-subdir-test 2022-06-27T11:50:44.887 INFO:tasks.workunit.client.0.smithi195.stdout:./280/.snap/snap-subdir-test 2022-06-27T11:50:44.892 INFO:tasks.workunit.client.0.smithi195.stdout:./116/.snap/snap-subdir-test 2022-06-27T11:50:44.896 INFO:tasks.workunit.client.0.smithi195.stdout:./245/.snap/snap-subdir-test 2022-06-27T11:50:44.902 INFO:tasks.workunit.client.0.smithi195.stdout:./195/.snap/snap-subdir-test 2022-06-27T11:50:44.907 INFO:tasks.workunit.client.0.smithi195.stdout:./138/.snap/snap-subdir-test 2022-06-27T11:50:44.928 INFO:tasks.workunit.client.0.smithi195.stdout:./135/.snap/snap-subdir-test 2022-06-27T11:50:44.928 INFO:tasks.workunit.client.0.smithi195.stdout:./117/.snap/snap-subdir-test 2022-06-27T11:50:44.929 INFO:tasks.workunit.client.0.smithi195.stdout:./241/.snap/snap-subdir-test 2022-06-27T11:50:44.929 INFO:tasks.workunit.client.0.smithi195.stdout:./143/.snap/snap-subdir-test 2022-06-27T11:50:44.940 INFO:tasks.workunit.client.0.smithi195.stdout:./180/.snap/snap-subdir-test 2022-06-27T11:50:45.789 INFO:tasks.workunit.client.0.smithi195.stdout:./126/.snap/snap-subdir-test 2022-06-27T11:50:45.795 INFO:tasks.workunit.client.0.smithi195.stdout:./161/.snap/snap-subdir-test 2022-06-27T11:50:45.800 INFO:tasks.workunit.client.0.smithi195.stdout:./204/.snap/snap-subdir-test 2022-06-27T11:50:45.804 INFO:tasks.workunit.client.0.smithi195.stdout:./176/.snap/snap-subdir-test 2022-06-27T11:50:45.809 INFO:tasks.workunit.client.0.smithi195.stdout:./255/.snap/snap-subdir-test 2022-06-27T11:50:45.816 INFO:tasks.workunit.client.0.smithi195.stdout:./215/.snap/snap-subdir-test 2022-06-27T11:50:45.824 INFO:tasks.workunit.client.0.smithi195.stdout:./111/.snap/snap-subdir-test 2022-06-27T11:50:46.202 INFO:tasks.workunit.client.0.smithi195.stdout:./238/.snap/snap-subdir-test 2022-06-27T11:50:46.203 INFO:tasks.workunit.client.0.smithi195.stdout:./191/.snap/snap-subdir-test 2022-06-27T11:50:46.208 INFO:tasks.workunit.client.0.smithi195.stdout:./184/.snap/snap-subdir-test 2022-06-27T11:50:46.208 INFO:tasks.workunit.client.0.smithi195.stdout:./192/.snap/snap-subdir-test 2022-06-27T11:50:46.213 INFO:tasks.workunit.client.0.smithi195.stdout:./103/.snap/snap-subdir-test 2022-06-27T11:50:46.225 INFO:tasks.workunit.client.0.smithi195.stdout:./128/.snap/snap-subdir-test 2022-06-27T11:50:46.226 INFO:tasks.workunit.client.0.smithi195.stdout:./229/.snap/snap-subdir-test 2022-06-27T11:50:46.227 INFO:tasks.workunit.client.0.smithi195.stdout:./144/.snap/snap-subdir-test 2022-06-27T11:50:46.242 INFO:tasks.workunit.client.0.smithi195.stdout:./146/.snap/snap-subdir-test 2022-06-27T11:50:46.242 INFO:tasks.workunit.client.0.smithi195.stdout:./227/.snap/snap-subdir-test 2022-06-27T11:50:46.244 INFO:tasks.workunit.client.0.smithi195.stdout:./278/.snap/snap-subdir-test 2022-06-27T11:50:46.247 INFO:tasks.workunit.client.0.smithi195.stdout:./141/.snap/snap-subdir-test 2022-06-27T11:50:46.534 INFO:tasks.workunit.client.0.smithi195.stdout:./234/.snap/snap-subdir-test 2022-06-27T11:50:46.538 INFO:tasks.workunit.client.0.smithi195.stdout:./225/.snap/snap-subdir-test 2022-06-27T11:50:46.551 INFO:tasks.workunit.client.0.smithi195.stdout:./182/.snap/snap-subdir-test 2022-06-27T11:50:46.551 INFO:tasks.workunit.client.0.smithi195.stdout:./107/.snap/snap-subdir-test 2022-06-27T11:50:46.556 INFO:tasks.workunit.client.0.smithi195.stdout:./178/.snap/snap-subdir-test 2022-06-27T11:50:46.558 INFO:tasks.workunit.client.0.smithi195.stdout:./266/.snap/snap-subdir-test 2022-06-27T11:50:46.562 INFO:tasks.workunit.client.0.smithi195.stdout:./240/.snap/snap-subdir-test 2022-06-27T11:50:46.567 INFO:tasks.workunit.client.0.smithi195.stdout:./156/.snap/snap-subdir-test 2022-06-27T11:50:46.572 INFO:tasks.workunit.client.0.smithi195.stdout:./131/.snap/snap-subdir-test 2022-06-27T11:50:46.579 INFO:tasks.workunit.client.0.smithi195.stdout:./123/.snap/snap-subdir-test 2022-06-27T11:50:46.583 INFO:tasks.workunit.client.0.smithi195.stdout:./160/.snap/snap-subdir-test 2022-06-27T11:50:46.586 INFO:tasks.workunit.client.0.smithi195.stdout:./165/.snap/snap-subdir-test 2022-06-27T11:50:46.592 INFO:tasks.workunit.client.0.smithi195.stdout:./101/.snap/snap-subdir-test 2022-06-27T11:50:46.596 INFO:tasks.workunit.client.0.smithi195.stdout:./148/.snap/snap-subdir-test 2022-06-27T11:50:46.601 INFO:tasks.workunit.client.0.smithi195.stdout:./168/.snap/snap-subdir-test 2022-06-27T11:50:46.605 INFO:tasks.workunit.client.0.smithi195.stdout:./179/.snap/snap-subdir-test 2022-06-27T11:50:46.610 INFO:tasks.workunit.client.0.smithi195.stdout:./257/.snap/snap-subdir-test 2022-06-27T11:50:46.676 INFO:tasks.workunit.client.0.smithi195.stdout:./252/.snap/snap-subdir-test 2022-06-27T11:50:46.681 INFO:tasks.workunit.client.0.smithi195.stdout:./132/.snap/snap-subdir-test 2022-06-27T11:50:46.687 INFO:tasks.workunit.client.0.smithi195.stdout:./236/.snap/snap-subdir-test 2022-06-27T11:50:46.690 INFO:tasks.workunit.client.0.smithi195.stdout:./211/.snap/snap-subdir-test 2022-06-27T11:50:46.695 INFO:tasks.workunit.client.0.smithi195.stdout:./199/.snap/snap-subdir-test 2022-06-27T11:50:46.699 INFO:tasks.workunit.client.0.smithi195.stdout:./205/.snap/snap-subdir-test 2022-06-27T11:50:46.705 INFO:tasks.workunit.client.0.smithi195.stdout:./163/.snap/snap-subdir-test 2022-06-27T11:50:46.718 INFO:tasks.workunit.client.0.smithi195.stdout:./261/.snap/snap-subdir-test 2022-06-27T11:50:46.719 INFO:tasks.workunit.client.0.smithi195.stdout:./201/.snap/snap-subdir-test 2022-06-27T11:50:46.719 INFO:tasks.workunit.client.0.smithi195.stdout:./282/.snap/snap-subdir-test 2022-06-27T11:50:46.726 INFO:tasks.workunit.client.0.smithi195.stdout:./296/.snap/snap-subdir-test 2022-06-27T11:50:46.727 INFO:tasks.workunit.client.0.smithi195.stdout:./217/.snap/snap-subdir-test 2022-06-27T11:50:46.748 INFO:tasks.workunit.client.0.smithi195.stdout:./153/.snap/snap-subdir-test 2022-06-27T11:50:46.749 INFO:tasks.workunit.client.0.smithi195.stdout:./206/.snap/snap-subdir-test 2022-06-27T11:50:46.749 INFO:tasks.workunit.client.0.smithi195.stdout:./122/.snap/snap-subdir-test 2022-06-27T11:50:46.752 INFO:tasks.workunit.client.0.smithi195.stdout:./162/.snap/snap-subdir-test 2022-06-27T11:50:46.753 INFO:tasks.workunit.client.0.smithi195.stdout:./197/.snap/snap-subdir-test 2022-06-27T11:50:46.757 INFO:tasks.workunit.client.0.smithi195.stdout:./216/.snap/snap-subdir-test 2022-06-27T11:50:48.480 INFO:tasks.workunit.client.0.smithi195.stdout:./254/.snap/snap-subdir-test 2022-06-27T11:50:48.481 INFO:tasks.workunit.client.0.smithi195.stdout:./113/.snap/snap-subdir-test 2022-06-27T11:50:48.486 INFO:tasks.workunit.client.0.smithi195.stdout:./152/.snap/snap-subdir-test 2022-06-27T11:50:48.491 INFO:tasks.workunit.client.0.smithi195.stdout:./121/.snap/snap-subdir-test 2022-06-27T11:50:49.102 INFO:tasks.workunit.client.0.smithi195.stdout:./118/.snap/snap-subdir-test 2022-06-27T11:50:49.106 INFO:tasks.workunit.client.0.smithi195.stdout:./196/.snap/snap-subdir-test 2022-06-27T11:50:49.743 INFO:tasks.workunit.client.0.smithi195.stdout:./124/.snap/snap-subdir-test 2022-06-27T11:50:49.748 INFO:tasks.workunit.client.0.smithi195.stdout:./239/.snap/snap-subdir-test 2022-06-27T11:50:49.752 INFO:tasks.workunit.client.0.smithi195.stdout:./188/.snap/snap-subdir-test 2022-06-27T11:50:49.757 INFO:tasks.workunit.client.0.smithi195.stdout:./287/.snap/snap-subdir-test 2022-06-27T11:50:49.762 INFO:tasks.workunit.client.0.smithi195.stdout:./275/.snap/snap-subdir-test 2022-06-27T11:50:49.766 INFO:tasks.workunit.client.0.smithi195.stdout:./259/.snap/snap-subdir-test 2022-06-27T11:50:49.771 INFO:tasks.workunit.client.0.smithi195.stdout:./224/.snap/snap-subdir-test 2022-06-27T11:50:49.776 INFO:tasks.workunit.client.0.smithi195.stdout:./264/.snap/snap-subdir-test 2022-06-27T11:50:49.784 INFO:tasks.workunit.client.0.smithi195.stdout:./145/.snap/snap-subdir-test 2022-06-27T11:50:49.785 INFO:tasks.workunit.client.0.smithi195.stdout:./279/.snap/snap-subdir-test 2022-06-27T11:50:49.794 INFO:tasks.workunit.client.0.smithi195.stdout:./263/.snap/snap-subdir-test 2022-06-27T11:50:49.796 INFO:tasks.workunit.client.0.smithi195.stdout:./130/.snap/snap-subdir-test 2022-06-27T11:50:49.801 INFO:tasks.workunit.client.0.smithi195.stdout:./193/.snap/snap-subdir-test 2022-06-27T11:50:49.817 INFO:tasks.workunit.client.0.smithi195.stdout:./223/.snap/snap-subdir-test 2022-06-27T11:50:49.818 INFO:tasks.workunit.client.0.smithi195.stdout:./207/.snap/snap-subdir-test 2022-06-27T11:50:49.818 INFO:tasks.workunit.client.0.smithi195.stdout:./276/.snap/snap-subdir-test 2022-06-27T11:50:49.822 INFO:tasks.workunit.client.0.smithi195.stdout:./271/.snap/snap-subdir-test 2022-06-27T11:50:49.825 INFO:tasks.workunit.client.0.smithi195.stdout:./251/.snap/snap-subdir-test 2022-06-27T11:50:49.830 INFO:tasks.workunit.client.0.smithi195.stdout:./175/.snap/snap-subdir-test 2022-06-27T11:50:49.835 INFO:tasks.workunit.client.0.smithi195.stdout:./222/.snap/snap-subdir-test 2022-06-27T11:50:49.844 INFO:tasks.workunit.client.0.smithi195.stdout:./220/.snap/snap-subdir-test 2022-06-27T11:50:49.851 INFO:tasks.workunit.client.0.smithi195.stdout:./203/.snap/snap-subdir-test 2022-06-27T11:50:49.851 INFO:tasks.workunit.client.0.smithi195.stdout:./181/.snap/snap-subdir-test 2022-06-27T11:50:50.415 INFO:tasks.workunit.client.0.smithi195.stdout:./286/.snap/snap-subdir-test 2022-06-27T11:50:50.421 INFO:tasks.workunit.client.0.smithi195.stdout:./127/.snap/snap-subdir-test 2022-06-27T11:50:50.427 INFO:tasks.workunit.client.0.smithi195.stdout:./108/.snap/snap-subdir-test 2022-06-27T11:50:50.432 INFO:tasks.workunit.client.0.smithi195.stdout:./183/.snap/snap-subdir-test 2022-06-27T11:50:50.437 INFO:tasks.workunit.client.0.smithi195.stdout:./294/.snap/snap-subdir-test 2022-06-27T11:50:50.441 INFO:tasks.workunit.client.0.smithi195.stdout:./235/.snap/snap-subdir-test 2022-06-27T11:50:50.447 INFO:tasks.workunit.client.0.smithi195.stdout:./177/.snap/snap-subdir-test 2022-06-27T11:50:50.451 INFO:tasks.workunit.client.0.smithi195.stdout:./169/.snap/snap-subdir-test 2022-06-27T11:50:50.456 INFO:tasks.workunit.client.0.smithi195.stdout:./212/.snap/snap-subdir-test 2022-06-27T11:50:50.462 INFO:tasks.workunit.client.0.smithi195.stdout:./154/.snap/snap-subdir-test 2022-06-27T11:50:50.468 INFO:tasks.workunit.client.0.smithi195.stdout:./171/.snap/snap-subdir-test 2022-06-27T11:50:50.472 INFO:tasks.workunit.client.0.smithi195.stdout:./139/.snap/snap-subdir-test 2022-06-27T11:50:50.479 INFO:tasks.workunit.client.0.smithi195.stdout:./102/.snap/snap-subdir-test 2022-06-27T11:50:50.482 INFO:tasks.workunit.client.0.smithi195.stdout:./129/.snap/snap-subdir-test 2022-06-27T11:50:56.744 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T11:50:56.754 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T11:50:56.766 INFO:tasks.workunit.client.0.smithi195.stdout:Delete all the files and directories ... 2022-06-27T11:50:57.931 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:50:58.071 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.072+0000 7f90af0dc700 1 Processor -- start 2022-06-27T11:50:58.072 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.072+0000 7f90af0dc700 1 -- start start 2022-06-27T11:50:58.076 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.073+0000 7f90af0dc700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f90a80c4160 0x7f90a8156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:58.076 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.073+0000 7f90af0dc700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90a80c1a20 con 0x7f90a80c39e0 2022-06-27T11:50:58.077 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.073+0000 7f90af0dc700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90a80c1130 con 0x7f90a806f640 2022-06-27T11:50:58.077 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.073+0000 7f90af0dc700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90a806ea90 con 0x7f90a80c4160 2022-06-27T11:50:58.077 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.073+0000 7f90ad679700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f90a80c39e0 0x7f90a81535a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:49222/0 (socket says 172.21.15.195:49222) 2022-06-27T11:50:58.078 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.073+0000 7f90ad679700 1 -- 172.21.15.195:0/3762630312 learned_addr learned my addr 172.21.15.195:0/3762630312 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:50:58.078 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.073+0000 7f90a7fff700 1 --2- 172.21.15.195:0/3762630312 >> v2:172.21.15.202:3300/0 conn(0x7f90a80c4160 0x7f90a8156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:58.078 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.074+0000 7f90a77fe700 1 -- 172.21.15.195:0/3762630312 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 473908017 0 0) 0x7f90a80c1a20 con 0x7f90a80c39e0 2022-06-27T11:50:58.078 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.074+0000 7f90a77fe700 1 -- 172.21.15.195:0/3762630312 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f908c003430 con 0x7f90a80c39e0 2022-06-27T11:50:58.079 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.074+0000 7f90a7fff700 1 -- 172.21.15.195:0/3762630312 >> v1:172.21.15.202:6789/0 conn(0x7f90a806f640 legacy=0x7f90a80c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:58.079 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.074+0000 7f90a7fff700 1 -- 172.21.15.195:0/3762630312 >> v1:172.21.15.195:6789/0 conn(0x7f90a80c39e0 legacy=0x7f90a81535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:58.079 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.074+0000 7f90a77fe700 1 -- 172.21.15.195:0/3762630312 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3509917819 0 0) 0x7f908c003430 con 0x7f90a80c39e0 2022-06-27T11:50:58.079 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.074+0000 7f90a7fff700 1 -- 172.21.15.195:0/3762630312 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90a806ed70 con 0x7f90a80c4160 2022-06-27T11:50:58.080 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.074+0000 7f90a7fff700 1 --2- 172.21.15.195:0/3762630312 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f90a80c4160 0x7f90a8156ed0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f90940097a0 tx=0x7f9094009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=92db18fd912153f5 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:50:58.080 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.075+0000 7f90a77fe700 1 -- 172.21.15.195:0/3762630312 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a00017f0 con 0x7f90a80c4160 2022-06-27T11:50:58.080 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.075+0000 7f90a77fe700 1 -- 172.21.15.195:0/3762630312 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9094004030 con 0x7f90a80c4160 2022-06-27T11:50:58.081 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.075+0000 7f90a77fe700 1 -- 172.21.15.195:0/3762630312 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90940043d0 con 0x7f90a80c4160 2022-06-27T11:50:58.081 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.075+0000 7f90af0dc700 1 -- 172.21.15.195:0/3762630312 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f90a80c4160 msgr2=0x7f90a8156ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:58.081 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.075+0000 7f90af0dc700 1 --2- 172.21.15.195:0/3762630312 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f90a80c4160 0x7f90a8156ed0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f90940097a0 tx=0x7f9094009060 comp rx=0 tx=0).stop 2022-06-27T11:50:58.081 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.075+0000 7f90af0dc700 1 -- 172.21.15.195:0/3762630312 shutdown_connections 2022-06-27T11:50:58.082 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.075+0000 7f90af0dc700 1 --2- 172.21.15.195:0/3762630312 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f90a80c4160 0x7f90a8156ed0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:58.082 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.075+0000 7f90af0dc700 1 -- 172.21.15.195:0/3762630312 >> 172.21.15.195:0/3762630312 conn(0x7f90a80c6350 msgr2=0x7f90a80c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:50:58.082 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.075+0000 7f90af0dc700 1 -- 172.21.15.195:0/3762630312 shutdown_connections 2022-06-27T11:50:58.083 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.075+0000 7f90af0dc700 1 -- 172.21.15.195:0/3762630312 wait complete. 2022-06-27T11:50:58.083 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.077+0000 7f90af0dc700 1 Processor -- start 2022-06-27T11:50:58.083 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.077+0000 7f90af0dc700 1 -- start start 2022-06-27T11:50:58.106 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.078+0000 7f90af0dc700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f90a806f640 0x7f90a81674a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:58.107 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.078+0000 7f90af0dc700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f90a80c39e0 0x7f90a80bc640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:58.107 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.078+0000 7f90af0dc700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f90a80c4160 0x7f90a8167ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:58.108 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.078+0000 7f90af0dc700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90a80bcb80 con 0x7f90a80c39e0 2022-06-27T11:50:58.111 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.078+0000 7f90af0dc700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90a80bb380 con 0x7f90a80c4160 2022-06-27T11:50:58.111 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.078+0000 7f90af0dc700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90a80bcda0 con 0x7f90a806f640 2022-06-27T11:50:58.112 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.078+0000 7f90a7fff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f90a80c39e0 0x7f90a80bc640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:58.112 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.078+0000 7f90a7fff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f90a80c39e0 0x7f90a80bc640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:43038/0 (socket says 172.21.15.195:43038) 2022-06-27T11:50:58.113 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.078+0000 7f90a7fff700 1 -- 172.21.15.195:0/1858601065 learned_addr learned my addr 172.21.15.195:0/1858601065 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:50:58.113 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.078+0000 7f90ad679700 1 --2- 172.21.15.195:0/1858601065 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f90a80c4160 0x7f90a8167ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:58.114 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.078+0000 7f90ace78700 1 --2- 172.21.15.195:0/1858601065 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f90a806f640 0x7f90a81674a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:58.115 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.079+0000 7f90a7fff700 1 -- 172.21.15.195:0/1858601065 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f90a806f640 msgr2=0x7f90a81674a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:58.115 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.079+0000 7f90a7fff700 1 --2- 172.21.15.195:0/1858601065 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f90a806f640 0x7f90a81674a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:58.116 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.079+0000 7f90a7fff700 1 -- 172.21.15.195:0/1858601065 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f90a80c4160 msgr2=0x7f90a8167ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:58.116 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.079+0000 7f90a7fff700 1 --2- 172.21.15.195:0/1858601065 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f90a80c4160 0x7f90a8167ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:58.116 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.079+0000 7f90a7fff700 1 -- 172.21.15.195:0/1858601065 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90940092e0 con 0x7f90a80c39e0 2022-06-27T11:50:58.117 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.079+0000 7f90a7fff700 1 --2- 172.21.15.195:0/1858601065 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f90a80c39e0 0x7f90a80bc640 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f9094009e40 tx=0x7f9094016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:50:58.117 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.079+0000 7f90a4ff9700 1 -- 172.21.15.195:0/1858601065 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9094003ea0 con 0x7f90a80c39e0 2022-06-27T11:50:58.117 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.079+0000 7f90a4ff9700 1 -- 172.21.15.195:0/1858601065 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f909401a670 con 0x7f90a80c39e0 2022-06-27T11:50:58.118 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.079+0000 7f90a4ff9700 1 -- 172.21.15.195:0/1858601065 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f909401aa30 con 0x7f90a80c39e0 2022-06-27T11:50:58.118 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.079+0000 7f90af0dc700 1 -- 172.21.15.195:0/1858601065 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90a80bd080 con 0x7f90a80c39e0 2022-06-27T11:50:58.119 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.079+0000 7f90af0dc700 1 -- 172.21.15.195:0/1858601065 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90a80bb8b0 con 0x7f90a80c39e0 2022-06-27T11:50:58.119 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.081+0000 7f90af0dc700 1 -- 172.21.15.195:0/1858601065 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90a80722c0 con 0x7f90a80c39e0 2022-06-27T11:50:58.119 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.081+0000 7f90a4ff9700 1 -- 172.21.15.195:0/1858601065 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f909402b030 con 0x7f90a80c39e0 2022-06-27T11:50:58.120 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.081+0000 7f90a4ff9700 1 --2- 172.21.15.195:0/1858601065 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9090063d30 0x7f90900661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:58.120 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.082+0000 7f90a4ff9700 1 -- 172.21.15.195:0/1858601065 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6466+0+0 (secure 0 0 0) 0x7f9094082060 con 0x7f90a80c39e0 2022-06-27T11:50:58.121 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.082+0000 7f90ace78700 1 --2- 172.21.15.195:0/1858601065 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9090063d30 0x7f90900661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:58.121 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.082+0000 7f90ace78700 1 --2- 172.21.15.195:0/1858601065 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9090063d30 0x7f90900661e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f909c0016b0 tx=0x7f909c009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:50:58.129 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.088+0000 7f90a4ff9700 1 -- 172.21.15.195:0/1858601065 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f9094050a50 con 0x7f90a80c39e0 2022-06-27T11:50:58.353 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.354+0000 7f90af0dc700 1 -- 172.21.15.195:0/1858601065 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90a806f240 con 0x7f90a80c39e0 2022-06-27T11:50:58.354 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.355+0000 7f90a4ff9700 1 -- 172.21.15.195:0/1858601065 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+6124 (secure 0 0 0) 0x7f9094029090 con 0x7f90a80c39e0 2022-06-27T11:50:58.355 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:50:58.355 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4849,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/3271891211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":3271891211},{"type":"v1","addr":"172.21.15.202:6835","nonce":3271891211}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:50:26.047919+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"b","rank":0,"incarnation":9,"state":"up:active","state_seq":43,"addr":"172.21.15.202:6833/864126677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":864126677},{"type":"v1","addr":"172.21.15.202:6833","nonce":864126677}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:50:58.358 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.357+0000 7f90af0dc700 1 -- 172.21.15.195:0/1858601065 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9090063d30 msgr2=0x7f90900661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:58.359 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.357+0000 7f90af0dc700 1 --2- 172.21.15.195:0/1858601065 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9090063d30 0x7f90900661e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f909c0016b0 tx=0x7f909c009040 comp rx=0 tx=0).stop 2022-06-27T11:50:58.360 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.357+0000 7f90af0dc700 1 -- 172.21.15.195:0/1858601065 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f90a80c39e0 msgr2=0x7f90a80bc640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:58.360 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.357+0000 7f90af0dc700 1 --2- 172.21.15.195:0/1858601065 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f90a80c39e0 0x7f90a80bc640 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f9094009e40 tx=0x7f9094016000 comp rx=0 tx=0).stop 2022-06-27T11:50:58.361 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.357+0000 7f90af0dc700 1 -- 172.21.15.195:0/1858601065 shutdown_connections 2022-06-27T11:50:58.362 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.357+0000 7f90af0dc700 1 --2- 172.21.15.195:0/1858601065 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9090063d30 0x7f90900661e0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:58.362 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.357+0000 7f90af0dc700 1 --2- 172.21.15.195:0/1858601065 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f90a80c4160 0x7f90a8167ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:58.363 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.357+0000 7f90af0dc700 1 --2- 172.21.15.195:0/1858601065 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f90a806f640 0x7f90a81674a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:58.363 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.357+0000 7f90af0dc700 1 --2- 172.21.15.195:0/1858601065 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f90a80c39e0 0x7f90a80bc640 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:58.364 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.357+0000 7f90af0dc700 1 -- 172.21.15.195:0/1858601065 >> 172.21.15.195:0/1858601065 conn(0x7f90a80c6350 msgr2=0x7f90a8152450 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:50:58.364 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.357+0000 7f90af0dc700 1 -- 172.21.15.195:0/1858601065 shutdown_connections 2022-06-27T11:50:58.364 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.357+0000 7f90af0dc700 1 -- 172.21.15.195:0/1858601065 wait complete. 2022-06-27T11:50:58.365 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 14 2022-06-27T11:50:58.371 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2022-06-27T11:50:58.372 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-06-27T11:50:58.372 INFO:teuthology.orchestra.run:waiting for 300 2022-06-27T11:50:58.388 INFO:tasks.ceph.mds.b:Stopped 2022-06-27T11:50:58.388 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:50:58.530 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.516+0000 7f0de092b700 1 Processor -- start 2022-06-27T11:50:58.531 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.516+0000 7f0de092b700 1 -- start start 2022-06-27T11:50:58.532 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.517+0000 7f0de092b700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f0ddc14e860 0x7f0ddc0c3840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:58.532 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.517+0000 7f0de092b700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ddc0c33b0 con 0x7f0ddc0c3d80 2022-06-27T11:50:58.532 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.517+0000 7f0de092b700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ddc06f220 con 0x7f0ddc14e490 2022-06-27T11:50:58.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.517+0000 7f0de092b700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ddc1542c0 con 0x7f0ddc14e860 2022-06-27T11:50:58.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.517+0000 7f0ddad9d700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f0ddc0c3d80 0x7f0ddc157890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:49236/0 (socket says 172.21.15.195:49236) 2022-06-27T11:50:58.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.517+0000 7f0ddad9d700 1 -- 172.21.15.195:0/1535935528 learned_addr learned my addr 172.21.15.195:0/1535935528 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:50:58.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.517+0000 7f0dd9d9b700 1 --2- 172.21.15.195:0/1535935528 >> v2:172.21.15.202:3300/0 conn(0x7f0ddc14e860 0x7f0ddc0c3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:58.534 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.517+0000 7f0dd959a700 1 -- 172.21.15.195:0/1535935528 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1261238188 0 0) 0x7f0ddc0c33b0 con 0x7f0ddc0c3d80 2022-06-27T11:50:58.534 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.517+0000 7f0dd959a700 1 -- 172.21.15.195:0/1535935528 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0dc0003430 con 0x7f0ddc0c3d80 2022-06-27T11:50:58.534 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.517+0000 7f0dd959a700 1 -- 172.21.15.195:0/1535935528 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1937431474 0 0) 0x7f0dc0003430 con 0x7f0ddc0c3d80 2022-06-27T11:50:58.534 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.517+0000 7f0dd959a700 1 -- 172.21.15.195:0/1535935528 >> v2:172.21.15.202:3300/0 conn(0x7f0ddc14e860 msgr2=0x7f0ddc0c3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:50:58.535 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.517+0000 7f0dd959a700 1 --2- 172.21.15.195:0/1535935528 >> v2:172.21.15.202:3300/0 conn(0x7f0ddc14e860 0x7f0ddc0c3840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:58.536 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.517+0000 7f0dd959a700 1 -- 172.21.15.195:0/1535935528 >> v1:172.21.15.202:6789/0 conn(0x7f0ddc14e490 legacy=0x7f0ddc06cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:58.536 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.517+0000 7f0dd959a700 1 -- 172.21.15.195:0/1535935528 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ddc154560 con 0x7f0ddc0c3d80 2022-06-27T11:50:58.537 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.517+0000 7f0dd959a700 1 -- 172.21.15.195:0/1535935528 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7f0dd40017f0 con 0x7f0ddc0c3d80 2022-06-27T11:50:58.537 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.517+0000 7f0dd959a700 1 -- 172.21.15.195:0/1535935528 >> v1:172.21.15.195:6789/0 conn(0x7f0ddc0c3d80 legacy=0x7f0ddc157890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:58.538 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.518+0000 7f0dd959a700 1 --2- 172.21.15.195:0/1535935528 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0dc0003920 0x7f0dc0003d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:58.538 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.518+0000 7f0dd959a700 1 --2- 172.21.15.195:0/1535935528 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0dc00046d0 0x7f0dc0004b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:58.539 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.518+0000 7f0dd959a700 1 --2- 172.21.15.195:0/1535935528 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0dc0005080 0x7f0dc000b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:58.540 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.518+0000 7f0dd959a700 1 -- 172.21.15.195:0/1535935528 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ddc1542c0 con 0x7f0dc0005080 2022-06-27T11:50:58.540 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.518+0000 7f0dd959a700 1 -- 172.21.15.195:0/1535935528 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc000bf90 con 0x7f0dc00046d0 2022-06-27T11:50:58.541 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.518+0000 7f0dd959a700 1 -- 172.21.15.195:0/1535935528 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc000c150 con 0x7f0dc0003920 2022-06-27T11:50:58.541 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.518+0000 7f0ddad9d700 1 --2- 172.21.15.195:0/1535935528 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0dc0005080 0x7f0dc000b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:58.542 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.518+0000 7f0dda59c700 1 --2- 172.21.15.195:0/1535935528 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0dc0003920 0x7f0dc0003d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:58.542 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.518+0000 7f0dd9d9b700 1 --2- 172.21.15.195:0/1535935528 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0dc00046d0 0x7f0dc0004b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:58.543 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.518+0000 7f0ddad9d700 1 -- 172.21.15.195:0/1535935528 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0dc0003920 msgr2=0x7f0dc0003d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:50:58.543 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.518+0000 7f0ddad9d700 1 --2- 172.21.15.195:0/1535935528 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0dc0003920 0x7f0dc0003d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:58.544 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.518+0000 7f0ddad9d700 1 -- 172.21.15.195:0/1535935528 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0dc00046d0 msgr2=0x7f0dc0004b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:50:58.544 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.518+0000 7f0ddad9d700 1 --2- 172.21.15.195:0/1535935528 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0dc00046d0 0x7f0dc0004b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:58.545 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.518+0000 7f0ddad9d700 1 -- 172.21.15.195:0/1535935528 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dc0003430 con 0x7f0dc0005080 2022-06-27T11:50:58.546 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.518+0000 7f0dd9d9b700 1 --2- 172.21.15.195:0/1535935528 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0dc00046d0 0x7f0dc0004b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T11:50:58.546 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.518+0000 7f0ddad9d700 1 -- 172.21.15.195:0/1535935528 reap_dead start 2022-06-27T11:50:58.546 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.518+0000 7f0ddad9d700 1 --2- 172.21.15.195:0/1535935528 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0dc0005080 0x7f0dc000b920 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f0dd400c5f0 tx=0x7f0ddc06ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e55004b23b9cda0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:50:58.546 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.518+0000 7f0dd959a700 1 -- 172.21.15.195:0/1535935528 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dd4005b60 con 0x7f0dc0005080 2022-06-27T11:50:58.547 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.518+0000 7f0dd959a700 1 -- 172.21.15.195:0/1535935528 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dd4005cc0 con 0x7f0dc0005080 2022-06-27T11:50:58.547 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.518+0000 7f0dd959a700 1 -- 172.21.15.195:0/1535935528 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dd4007320 con 0x7f0dc0005080 2022-06-27T11:50:58.548 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.519+0000 7f0de092b700 1 -- 172.21.15.195:0/1535935528 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0dc0005080 msgr2=0x7f0dc000b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:58.548 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.519+0000 7f0de092b700 1 --2- 172.21.15.195:0/1535935528 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0dc0005080 0x7f0dc000b920 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f0dd400c5f0 tx=0x7f0ddc06ee20 comp rx=0 tx=0).stop 2022-06-27T11:50:58.548 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.519+0000 7f0de092b700 1 -- 172.21.15.195:0/1535935528 shutdown_connections 2022-06-27T11:50:58.549 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.519+0000 7f0de092b700 1 --2- 172.21.15.195:0/1535935528 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0dc0005080 0x7f0dc000b920 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:58.549 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.519+0000 7f0de092b700 1 -- 172.21.15.195:0/1535935528 >> 172.21.15.195:0/1535935528 conn(0x7f0ddc063af0 msgr2=0x7f0ddc063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:50:58.549 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.519+0000 7f0de092b700 1 -- 172.21.15.195:0/1535935528 shutdown_connections 2022-06-27T11:50:58.550 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.519+0000 7f0de092b700 1 -- 172.21.15.195:0/1535935528 wait complete. 2022-06-27T11:50:58.550 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.520+0000 7f0de092b700 1 Processor -- start 2022-06-27T11:50:58.551 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.520+0000 7f0de092b700 1 -- start start 2022-06-27T11:50:58.551 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.521+0000 7f0de092b700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0dc0005080 0x7f0ddc06a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:58.551 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.521+0000 7f0de092b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0ddc0c75a0 0x7f0ddc069020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:58.551 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.521+0000 7f0de092b700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0ddc0bf130 0x7f0ddc14bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:58.552 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.521+0000 7f0de092b700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ddc15a870 con 0x7f0ddc0c75a0 2022-06-27T11:50:58.552 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.521+0000 7f0de092b700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ddc0c7110 con 0x7f0ddc0bf130 2022-06-27T11:50:58.552 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.521+0000 7f0de092b700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ddc0bf5a0 con 0x7f0dc0005080 2022-06-27T11:50:58.553 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.521+0000 7f0dd9d9b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0ddc0c75a0 0x7f0ddc069020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:58.553 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.521+0000 7f0dd9d9b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0ddc0c75a0 0x7f0ddc069020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:43058/0 (socket says 172.21.15.195:43058) 2022-06-27T11:50:58.554 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.521+0000 7f0dd9d9b700 1 -- 172.21.15.195:0/2086509178 learned_addr learned my addr 172.21.15.195:0/2086509178 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:50:58.554 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.521+0000 7f0ddad9d700 1 --2- 172.21.15.195:0/2086509178 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0ddc0bf130 0x7f0ddc14bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:58.555 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.521+0000 7f0dda59c700 1 --2- 172.21.15.195:0/2086509178 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0dc0005080 0x7f0ddc06a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:58.555 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.521+0000 7f0dd9d9b700 1 -- 172.21.15.195:0/2086509178 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0dc0005080 msgr2=0x7f0ddc06a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:58.556 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.521+0000 7f0dd9d9b700 1 --2- 172.21.15.195:0/2086509178 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0dc0005080 0x7f0ddc06a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:58.556 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.521+0000 7f0dd9d9b700 1 -- 172.21.15.195:0/2086509178 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0ddc0bf130 msgr2=0x7f0ddc14bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:58.557 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.521+0000 7f0dd9d9b700 1 --2- 172.21.15.195:0/2086509178 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0ddc0bf130 0x7f0ddc14bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:58.557 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.521+0000 7f0dd9d9b700 1 -- 172.21.15.195:0/2086509178 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ddc069640 con 0x7f0ddc0c75a0 2022-06-27T11:50:58.558 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.521+0000 7f0dd9d9b700 1 --2- 172.21.15.195:0/2086509178 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0ddc0c75a0 0x7f0ddc069020 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f0dd0011660 tx=0x7f0dd0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:50:58.558 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.521+0000 7f0dca7fc700 1 -- 172.21.15.195:0/2086509178 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dd0004020 con 0x7f0ddc0c75a0 2022-06-27T11:50:58.558 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.521+0000 7f0dca7fc700 1 -- 172.21.15.195:0/2086509178 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dd0004180 con 0x7f0ddc0c75a0 2022-06-27T11:50:58.559 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.521+0000 7f0de092b700 1 -- 172.21.15.195:0/2086509178 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ddc0c6500 con 0x7f0ddc0c75a0 2022-06-27T11:50:58.559 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.521+0000 7f0dca7fc700 1 -- 172.21.15.195:0/2086509178 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dd0015740 con 0x7f0ddc0c75a0 2022-06-27T11:50:58.560 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.521+0000 7f0de092b700 1 -- 172.21.15.195:0/2086509178 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ddc0c6ac0 con 0x7f0ddc0c75a0 2022-06-27T11:50:58.560 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.522+0000 7f0de092b700 1 -- 172.21.15.195:0/2086509178 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ddc0722c0 con 0x7f0ddc0c75a0 2022-06-27T11:50:58.561 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.523+0000 7f0dca7fc700 1 -- 172.21.15.195:0/2086509178 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f0dd0024070 con 0x7f0ddc0c75a0 2022-06-27T11:50:58.561 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.523+0000 7f0dca7fc700 1 --2- 172.21.15.195:0/2086509178 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0db8063d00 0x7f0db80661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:50:58.561 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.523+0000 7f0dca7fc700 1 -- 172.21.15.195:0/2086509178 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6466+0+0 (secure 0 0 0) 0x7f0dd0080a00 con 0x7f0ddc0c75a0 2022-06-27T11:50:58.562 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.523+0000 7f0dda59c700 1 --2- 172.21.15.195:0/2086509178 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0db8063d00 0x7f0db80661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:50:58.562 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.523+0000 7f0dda59c700 1 --2- 172.21.15.195:0/2086509178 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0db8063d00 0x7f0db80661b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc000c00 tx=0x7f0dcc009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:50:58.563 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.527+0000 7f0dca7fc700 1 -- 172.21.15.195:0/2086509178 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f0dd004f450 con 0x7f0ddc0c75a0 2022-06-27T11:50:58.751 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.753+0000 7f0de092b700 1 -- 172.21.15.195:0/2086509178 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ddc0c60d0 con 0x7f0ddc0c75a0 2022-06-27T11:50:58.752 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.753+0000 7f0dca7fc700 1 -- 172.21.15.195:0/2086509178 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+6124 (secure 0 0 0) 0x7f0dd0022070 con 0x7f0ddc0c75a0 2022-06-27T11:50:58.753 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:50:58.754 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4849,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/3271891211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":3271891211},{"type":"v1","addr":"172.21.15.202:6835","nonce":3271891211}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:50:26.047919+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"b","rank":0,"incarnation":9,"state":"up:active","state_seq":43,"addr":"172.21.15.202:6833/864126677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":864126677},{"type":"v1","addr":"172.21.15.202:6833","nonce":864126677}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:50:58.755 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.755+0000 7f0de092b700 1 -- 172.21.15.195:0/2086509178 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0db8063d00 msgr2=0x7f0db80661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:58.756 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.755+0000 7f0de092b700 1 --2- 172.21.15.195:0/2086509178 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0db8063d00 0x7f0db80661b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc000c00 tx=0x7f0dcc009040 comp rx=0 tx=0).stop 2022-06-27T11:50:58.756 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.755+0000 7f0de092b700 1 -- 172.21.15.195:0/2086509178 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0ddc0c75a0 msgr2=0x7f0ddc069020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:50:58.757 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.755+0000 7f0de092b700 1 --2- 172.21.15.195:0/2086509178 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0ddc0c75a0 0x7f0ddc069020 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f0dd0011660 tx=0x7f0dd0008040 comp rx=0 tx=0).stop 2022-06-27T11:50:58.757 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.755+0000 7f0de092b700 1 -- 172.21.15.195:0/2086509178 shutdown_connections 2022-06-27T11:50:58.758 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.755+0000 7f0de092b700 1 --2- 172.21.15.195:0/2086509178 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0db8063d00 0x7f0db80661b0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:58.758 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.755+0000 7f0de092b700 1 --2- 172.21.15.195:0/2086509178 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0ddc0bf130 0x7f0ddc14bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:58.758 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.755+0000 7f0de092b700 1 --2- 172.21.15.195:0/2086509178 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0dc0005080 0x7f0ddc06a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:58.759 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.755+0000 7f0de092b700 1 --2- 172.21.15.195:0/2086509178 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0ddc0c75a0 0x7f0ddc069020 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:50:58.760 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.755+0000 7f0de092b700 1 -- 172.21.15.195:0/2086509178 >> 172.21.15.195:0/2086509178 conn(0x7f0ddc063af0 msgr2=0x7f0ddc151f50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:50:58.761 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.755+0000 7f0de092b700 1 -- 172.21.15.195:0/2086509178 shutdown_connections 2022-06-27T11:50:58.766 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:50:58.755+0000 7f0de092b700 1 -- 172.21.15.195:0/2086509178 wait complete. 2022-06-27T11:50:58.767 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 14 2022-06-27T11:50:58.769 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2022-06-27T11:51:00.771 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:51:00.909 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.906+0000 7f0571f47700 1 Processor -- start 2022-06-27T11:51:00.910 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.906+0000 7f0571f47700 1 -- start start 2022-06-27T11:51:00.911 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.907+0000 7f0571f47700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f056c150500 0x7f056c063af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:00.911 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.907+0000 7f0571f47700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f056c064030 con 0x7f056c1508d0 2022-06-27T11:51:00.911 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.907+0000 7f0571f47700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f056c0c4160 con 0x7f056c06a450 2022-06-27T11:51:00.912 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.907+0000 7f0571f47700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f056c0c4440 con 0x7f056c150500 2022-06-27T11:51:00.912 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.907+0000 7f056bfff700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f056c1508d0 0x7f056c0bbf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:49258/0 (socket says 172.21.15.195:49258) 2022-06-27T11:51:00.913 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.907+0000 7f056bfff700 1 -- 172.21.15.195:0/808895653 learned_addr learned my addr 172.21.15.195:0/808895653 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:51:00.914 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.907+0000 7f056affd700 1 -- 172.21.15.195:0/808895653 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1955228234 0 0) 0x7f0564001490 con 0x7f056c1508d0 2022-06-27T11:51:00.914 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.907+0000 7f056b7fe700 1 --2- 172.21.15.195:0/808895653 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f056c150500 0x7f056c063af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:00.915 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.907+0000 7f056affd700 1 -- 172.21.15.195:0/808895653 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f054c003430 con 0x7f056c1508d0 2022-06-27T11:51:00.931 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.907+0000 7f056affd700 1 -- 172.21.15.195:0/808895653 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1343612297 0 0) 0x7f054c003430 con 0x7f056c1508d0 2022-06-27T11:51:00.931 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.907+0000 7f056affd700 1 -- 172.21.15.195:0/808895653 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f056c150500 msgr2=0x7f056c063af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:51:00.931 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.907+0000 7f056affd700 1 --2- 172.21.15.195:0/808895653 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f056c150500 0x7f056c063af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:00.932 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.907+0000 7f056affd700 1 -- 172.21.15.195:0/808895653 >> v1:172.21.15.202:6789/0 conn(0x7f056c06a450 legacy=0x7f056c0c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:00.932 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.907+0000 7f056affd700 1 -- 172.21.15.195:0/808895653 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f056c15a800 con 0x7f056c1508d0 2022-06-27T11:51:00.932 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.907+0000 7f056affd700 1 -- 172.21.15.195:0/808895653 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7f0564001980 con 0x7f056c1508d0 2022-06-27T11:51:00.933 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.907+0000 7f056affd700 1 -- 172.21.15.195:0/808895653 >> v1:172.21.15.195:6789/0 conn(0x7f056c1508d0 legacy=0x7f056c0bbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:00.933 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.907+0000 7f056affd700 1 --2- 172.21.15.195:0/808895653 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f054c003950 0x7f054c003d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:00.933 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.908+0000 7f056affd700 1 --2- 172.21.15.195:0/808895653 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f054c0046d0 0x7f054c004b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:00.934 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.908+0000 7f056affd700 1 --2- 172.21.15.195:0/808895653 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f056c150500 0x7f054c005080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:00.935 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.908+0000 7f056affd700 1 -- 172.21.15.195:0/808895653 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f056c0c4440 con 0x7f054c003950 2022-06-27T11:51:00.935 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.908+0000 7f056affd700 1 -- 172.21.15.195:0/808895653 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f054c00b9d0 con 0x7f054c0046d0 2022-06-27T11:51:00.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.908+0000 7f056affd700 1 -- 172.21.15.195:0/808895653 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f054c00bb90 con 0x7f056c150500 2022-06-27T11:51:00.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.908+0000 7f056b7fe700 1 --2- 172.21.15.195:0/808895653 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f054c003950 0x7f054c003d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:00.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.908+0000 7f0563fff700 1 --2- 172.21.15.195:0/808895653 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f054c0046d0 0x7f054c004b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:00.937 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.908+0000 7f056bfff700 1 --2- 172.21.15.195:0/808895653 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f056c150500 0x7f054c005080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:00.937 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.908+0000 7f056b7fe700 1 -- 172.21.15.195:0/808895653 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f056c150500 msgr2=0x7f054c005080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:51:00.937 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.908+0000 7f056b7fe700 1 --2- 172.21.15.195:0/808895653 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f056c150500 0x7f054c005080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:00.937 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.908+0000 7f056b7fe700 1 -- 172.21.15.195:0/808895653 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f054c0046d0 msgr2=0x7f054c004b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:51:00.938 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.908+0000 7f056b7fe700 1 --2- 172.21.15.195:0/808895653 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f054c0046d0 0x7f054c004b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:00.938 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.908+0000 7f056b7fe700 1 -- 172.21.15.195:0/808895653 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f054c003430 con 0x7f054c003950 2022-06-27T11:51:00.938 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.908+0000 7f056b7fe700 1 --2- 172.21.15.195:0/808895653 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f054c003950 0x7f054c003d20 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f055c009ae0 tx=0x7f055c006cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=713ab94547c9dafb server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:51:00.939 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.908+0000 7f056affd700 1 -- 172.21.15.195:0/808895653 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f055c009520 con 0x7f054c003950 2022-06-27T11:51:00.939 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.908+0000 7f056affd700 1 -- 172.21.15.195:0/808895653 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f055c0058c0 con 0x7f054c003950 2022-06-27T11:51:00.939 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.908+0000 7f056affd700 1 -- 172.21.15.195:0/808895653 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f055c005c60 con 0x7f054c003950 2022-06-27T11:51:00.939 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.909+0000 7f0571f47700 1 -- 172.21.15.195:0/808895653 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f054c003950 msgr2=0x7f054c003d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:00.940 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.909+0000 7f0571f47700 1 --2- 172.21.15.195:0/808895653 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f054c003950 0x7f054c003d20 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f055c009ae0 tx=0x7f055c006cb0 comp rx=0 tx=0).stop 2022-06-27T11:51:00.940 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.909+0000 7f056bfff700 1 -- 172.21.15.195:0/808895653 reap_dead start 2022-06-27T11:51:00.941 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.909+0000 7f0571f47700 1 -- 172.21.15.195:0/808895653 shutdown_connections 2022-06-27T11:51:00.941 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.909+0000 7f0571f47700 1 -- 172.21.15.195:0/808895653 >> 172.21.15.195:0/808895653 conn(0x7f056c0c9b00 msgr2=0x7f056c0c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:51:00.942 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.909+0000 7f0571f47700 1 -- 172.21.15.195:0/808895653 shutdown_connections 2022-06-27T11:51:00.942 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.909+0000 7f0571f47700 1 -- 172.21.15.195:0/808895653 wait complete. 2022-06-27T11:51:00.942 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.910+0000 7f0571f47700 1 Processor -- start 2022-06-27T11:51:00.943 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.910+0000 7f0571f47700 1 -- start start 2022-06-27T11:51:00.943 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.911+0000 7f0571f47700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f054c003950 0x7f056c14de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:00.943 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.911+0000 7f0571f47700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f056c0c11e0 0x7f056c0c17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:00.944 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.911+0000 7f0571f47700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f056c0bbdc0 0x7f056c0bc230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:00.944 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.911+0000 7f0571f47700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f056c0be480 con 0x7f056c0c11e0 2022-06-27T11:51:00.944 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.911+0000 7f0571f47700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f056c0c0ea0 con 0x7f056c0bbdc0 2022-06-27T11:51:00.945 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.911+0000 7f0571f47700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f056c0bd7f0 con 0x7f054c003950 2022-06-27T11:51:00.946 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.911+0000 7f0563fff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f056c0c11e0 0x7f056c0c17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:00.946 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.911+0000 7f0563fff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f056c0c11e0 0x7f056c0c17c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:43082/0 (socket says 172.21.15.195:43082) 2022-06-27T11:51:00.946 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.911+0000 7f0563fff700 1 -- 172.21.15.195:0/2090779056 learned_addr learned my addr 172.21.15.195:0/2090779056 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:51:00.947 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.911+0000 7f056b7fe700 1 --2- 172.21.15.195:0/2090779056 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f054c003950 0x7f056c14de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:00.947 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.911+0000 7f056bfff700 1 --2- 172.21.15.195:0/2090779056 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f056c0bbdc0 0x7f056c0bc230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:00.947 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.912+0000 7f0563fff700 1 -- 172.21.15.195:0/2090779056 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f054c003950 msgr2=0x7f056c14de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:00.948 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.912+0000 7f0563fff700 1 --2- 172.21.15.195:0/2090779056 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f054c003950 0x7f056c14de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:00.948 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.912+0000 7f0563fff700 1 -- 172.21.15.195:0/2090779056 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f056c0bbdc0 msgr2=0x7f056c0bc230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:00.948 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.912+0000 7f0563fff700 1 --2- 172.21.15.195:0/2090779056 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f056c0bbdc0 0x7f056c0bc230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:00.948 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.912+0000 7f0563fff700 1 -- 172.21.15.195:0/2090779056 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f056c0bfa00 con 0x7f056c0c11e0 2022-06-27T11:51:00.949 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.912+0000 7f0563fff700 1 --2- 172.21.15.195:0/2090779056 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f056c0c11e0 0x7f056c0c17c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f055400f9b0 tx=0x7f0554009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:51:00.949 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.912+0000 7f05637fe700 1 -- 172.21.15.195:0/2090779056 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05540052e0 con 0x7f056c0c11e0 2022-06-27T11:51:00.949 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.912+0000 7f05637fe700 1 -- 172.21.15.195:0/2090779056 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0554003500 con 0x7f056c0c11e0 2022-06-27T11:51:00.950 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.912+0000 7f0571f47700 1 -- 172.21.15.195:0/2090779056 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f056c0c0bc0 con 0x7f056c0c11e0 2022-06-27T11:51:00.950 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.912+0000 7f05637fe700 1 -- 172.21.15.195:0/2090779056 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05540038c0 con 0x7f056c0c11e0 2022-06-27T11:51:00.950 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.912+0000 7f0571f47700 1 -- 172.21.15.195:0/2090779056 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f056c0bd1a0 con 0x7f056c0c11e0 2022-06-27T11:51:00.951 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.913+0000 7f0571f47700 1 -- 172.21.15.195:0/2090779056 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f056c14e380 con 0x7f056c0c11e0 2022-06-27T11:51:00.951 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.913+0000 7f05637fe700 1 -- 172.21.15.195:0/2090779056 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f0554003cf0 con 0x7f056c0c11e0 2022-06-27T11:51:00.951 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.914+0000 7f05637fe700 1 --2- 172.21.15.195:0/2090779056 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0550063d30 0x7f05500661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:00.952 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.914+0000 7f05637fe700 1 -- 172.21.15.195:0/2090779056 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 7378+0+0 (secure 0 0 0) 0x7f0554080dc0 con 0x7f056c0c11e0 2022-06-27T11:51:00.952 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.914+0000 7f056b7fe700 1 --2- 172.21.15.195:0/2090779056 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0550063d30 0x7f05500661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:00.952 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.914+0000 7f056b7fe700 1 --2- 172.21.15.195:0/2090779056 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0550063d30 0x7f05500661e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f055c009ae0 tx=0x7f055c019000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:51:00.952 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:00.918+0000 7f05637fe700 1 -- 172.21.15.195:0/2090779056 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f055404f480 con 0x7f056c0c11e0 2022-06-27T11:51:01.160 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:01.156+0000 7f0571f47700 1 -- 172.21.15.195:0/2090779056 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f056c0722c0 con 0x7f056c0c11e0 2022-06-27T11:51:01.160 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:01.157+0000 7f05637fe700 1 -- 172.21.15.195:0/2090779056 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+6124 (secure 0 0 0) 0x7f0554052aa0 con 0x7f056c0c11e0 2022-06-27T11:51:01.161 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:51:01.161 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4849,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/3271891211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":3271891211},{"type":"v1","addr":"172.21.15.202:6835","nonce":3271891211}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:50:26.047919+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"b","rank":0,"incarnation":9,"state":"up:active","state_seq":43,"addr":"172.21.15.202:6833/864126677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":864126677},{"type":"v1","addr":"172.21.15.202:6833","nonce":864126677}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:51:01.163 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:01.159+0000 7f0571f47700 1 -- 172.21.15.195:0/2090779056 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0550063d30 msgr2=0x7f05500661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:01.163 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:01.159+0000 7f0571f47700 1 --2- 172.21.15.195:0/2090779056 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0550063d30 0x7f05500661e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f055c009ae0 tx=0x7f055c019000 comp rx=0 tx=0).stop 2022-06-27T11:51:01.164 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:01.159+0000 7f0571f47700 1 -- 172.21.15.195:0/2090779056 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f056c0c11e0 msgr2=0x7f056c0c17c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:01.164 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:01.159+0000 7f0571f47700 1 --2- 172.21.15.195:0/2090779056 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f056c0c11e0 0x7f056c0c17c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f055400f9b0 tx=0x7f0554009040 comp rx=0 tx=0).stop 2022-06-27T11:51:01.164 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:01.159+0000 7f0571f47700 1 -- 172.21.15.195:0/2090779056 shutdown_connections 2022-06-27T11:51:01.164 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:01.159+0000 7f0571f47700 1 --2- 172.21.15.195:0/2090779056 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0550063d30 0x7f05500661e0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:01.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:01.159+0000 7f0571f47700 1 --2- 172.21.15.195:0/2090779056 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f056c0bbdc0 0x7f056c0bc230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:01.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:01.159+0000 7f0571f47700 1 --2- 172.21.15.195:0/2090779056 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f054c003950 0x7f056c14de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:01.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:01.159+0000 7f0571f47700 1 --2- 172.21.15.195:0/2090779056 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f056c0c11e0 0x7f056c0c17c0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:01.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:01.159+0000 7f0571f47700 1 -- 172.21.15.195:0/2090779056 >> 172.21.15.195:0/2090779056 conn(0x7f056c0c9b00 msgr2=0x7f056c14fd60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:51:01.166 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:01.159+0000 7f0571f47700 1 -- 172.21.15.195:0/2090779056 shutdown_connections 2022-06-27T11:51:01.166 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:01.159+0000 7f0571f47700 1 -- 172.21.15.195:0/2090779056 wait complete. 2022-06-27T11:51:01.166 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 14 2022-06-27T11:51:01.182 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2022-06-27T11:51:03.183 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:51:03.321 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.322+0000 7f69a3bc7700 1 Processor -- start 2022-06-27T11:51:03.322 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.322+0000 7f69a3bc7700 1 -- start start 2022-06-27T11:51:03.331 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.323+0000 7f69a3bc7700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f699c06fef0 0x7f699c1570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:03.332 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.323+0000 7f69a3bc7700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f699c06fba0 con 0x7f699c06e850 2022-06-27T11:51:03.332 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.323+0000 7f69a3bc7700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f699c06f5e0 con 0x7f699c0c34a0 2022-06-27T11:51:03.332 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.323+0000 7f69a3bc7700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f699c1575e0 con 0x7f699c06fef0 2022-06-27T11:51:03.333 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.323+0000 7f69a1963700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f699c06e850 0x7f699c069de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:49280/0 (socket says 172.21.15.195:49280) 2022-06-27T11:51:03.334 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.323+0000 7f69a1963700 1 -- 172.21.15.195:0/4270332094 learned_addr learned my addr 172.21.15.195:0/4270332094 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:51:03.334 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.323+0000 7f69a1162700 1 --2- 172.21.15.195:0/4270332094 >> v2:172.21.15.202:3300/0 conn(0x7f699c06fef0 0x7f699c1570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:03.334 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.323+0000 7f69a0961700 1 -- 172.21.15.195:0/4270332094 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1024593358 0 0) 0x7f699c06fba0 con 0x7f699c06e850 2022-06-27T11:51:03.334 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.323+0000 7f69a0961700 1 -- 172.21.15.195:0/4270332094 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6984003430 con 0x7f699c06e850 2022-06-27T11:51:03.357 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.323+0000 7f69a0961700 1 -- 172.21.15.195:0/4270332094 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4085314192 0 0) 0x7f699c06f5e0 con 0x7f699c0c34a0 2022-06-27T11:51:03.357 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.323+0000 7f69a0961700 1 -- 172.21.15.195:0/4270332094 --> v1:172.21.15.202:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f699c06fba0 con 0x7f699c0c34a0 2022-06-27T11:51:03.357 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.323+0000 7f69a0961700 1 -- 172.21.15.195:0/4270332094 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1535879603 0 0) 0x7f6984003430 con 0x7f699c06e850 2022-06-27T11:51:03.360 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.323+0000 7f69a0961700 1 -- 172.21.15.195:0/4270332094 >> v2:172.21.15.202:3300/0 conn(0x7f699c06fef0 msgr2=0x7f699c1570a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:51:03.360 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.323+0000 7f69a0961700 1 --2- 172.21.15.195:0/4270332094 >> v2:172.21.15.202:3300/0 conn(0x7f699c06fef0 0x7f699c1570a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:03.360 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.323+0000 7f69a0961700 1 -- 172.21.15.195:0/4270332094 >> v1:172.21.15.202:6789/0 conn(0x7f699c0c34a0 legacy=0x7f699c0c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:03.361 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.323+0000 7f69a0961700 1 -- 172.21.15.195:0/4270332094 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f699c06f8c0 con 0x7f699c06e850 2022-06-27T11:51:03.361 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.323+0000 7f69a0961700 1 -- 172.21.15.195:0/4270332094 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7f698c001470 con 0x7f699c06e850 2022-06-27T11:51:03.361 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.323+0000 7f69a0961700 1 -- 172.21.15.195:0/4270332094 >> v1:172.21.15.195:6789/0 conn(0x7f699c06e850 legacy=0x7f699c069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:03.362 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.323+0000 7f69a0961700 1 --2- 172.21.15.195:0/4270332094 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6984003ab0 0x7f6984003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:03.362 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.323+0000 7f69a0961700 1 --2- 172.21.15.195:0/4270332094 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f69840046d0 0x7f6984004b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:03.363 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.323+0000 7f69a0961700 1 --2- 172.21.15.195:0/4270332094 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6984005080 0x7f698400db30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:03.363 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.323+0000 7f69a0961700 1 -- 172.21.15.195:0/4270332094 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f699c1575e0 con 0x7f6984005080 2022-06-27T11:51:03.364 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.323+0000 7f69a0961700 1 -- 172.21.15.195:0/4270332094 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f698400e180 con 0x7f69840046d0 2022-06-27T11:51:03.364 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.323+0000 7f69a0961700 1 -- 172.21.15.195:0/4270332094 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f698400e340 con 0x7f6984003ab0 2022-06-27T11:51:03.365 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.323+0000 7f69a2164700 1 --2- 172.21.15.195:0/4270332094 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6984005080 0x7f698400db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:03.365 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.323+0000 7f69a1162700 1 --2- 172.21.15.195:0/4270332094 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f69840046d0 0x7f6984004b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:03.365 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.324+0000 7f69a1963700 1 --2- 172.21.15.195:0/4270332094 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6984003ab0 0x7f6984003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:03.366 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.324+0000 7f69a2164700 1 -- 172.21.15.195:0/4270332094 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6984003ab0 msgr2=0x7f6984003ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:51:03.367 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.324+0000 7f69a2164700 1 --2- 172.21.15.195:0/4270332094 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6984003ab0 0x7f6984003ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:03.367 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.324+0000 7f69a2164700 1 -- 172.21.15.195:0/4270332094 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f69840046d0 msgr2=0x7f6984004b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:51:03.367 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.324+0000 7f69a2164700 1 --2- 172.21.15.195:0/4270332094 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f69840046d0 0x7f6984004b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:03.368 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.324+0000 7f69a2164700 1 -- 172.21.15.195:0/4270332094 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6984003430 con 0x7f6984005080 2022-06-27T11:51:03.368 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.324+0000 7f69a1162700 1 --2- 172.21.15.195:0/4270332094 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f69840046d0 0x7f6984004b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:51:03.373 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.324+0000 7f69a2164700 1 -- 172.21.15.195:0/4270332094 reap_dead start 2022-06-27T11:51:03.383 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.324+0000 7f69a2164700 1 --2- 172.21.15.195:0/4270332094 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6984005080 0x7f698400db30 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f6994002010 tx=0x7f6994007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea19ed2eefe32a7 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:51:03.384 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.324+0000 7f69a0961700 1 -- 172.21.15.195:0/4270332094 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69940019a0 con 0x7f6984005080 2022-06-27T11:51:03.384 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.324+0000 7f69a0961700 1 -- 172.21.15.195:0/4270332094 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f699400c8b0 con 0x7f6984005080 2022-06-27T11:51:03.385 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.324+0000 7f69a0961700 1 -- 172.21.15.195:0/4270332094 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f699400cc50 con 0x7f6984005080 2022-06-27T11:51:03.387 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.325+0000 7f69a3bc7700 1 -- 172.21.15.195:0/4270332094 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6984005080 msgr2=0x7f698400db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:03.388 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.325+0000 7f69a3bc7700 1 --2- 172.21.15.195:0/4270332094 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6984005080 0x7f698400db30 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f6994002010 tx=0x7f6994007f70 comp rx=0 tx=0).stop 2022-06-27T11:51:03.388 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.325+0000 7f69a3bc7700 1 -- 172.21.15.195:0/4270332094 shutdown_connections 2022-06-27T11:51:03.389 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.325+0000 7f69a3bc7700 1 --2- 172.21.15.195:0/4270332094 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6984005080 0x7f698400db30 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:03.389 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.325+0000 7f69a3bc7700 1 -- 172.21.15.195:0/4270332094 >> 172.21.15.195:0/4270332094 conn(0x7f699c0c6350 msgr2=0x7f699c0c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:51:03.390 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.325+0000 7f69a3bc7700 1 -- 172.21.15.195:0/4270332094 shutdown_connections 2022-06-27T11:51:03.390 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.325+0000 7f69a3bc7700 1 -- 172.21.15.195:0/4270332094 wait complete. 2022-06-27T11:51:03.391 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.326+0000 7f69a3bc7700 1 Processor -- start 2022-06-27T11:51:03.391 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.326+0000 7f69a3bc7700 1 -- start start 2022-06-27T11:51:03.392 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.327+0000 7f69a3bc7700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6984005080 0x7f699c0c7f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:03.392 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.327+0000 7f69a3bc7700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f699c16a570 0x7f699c1557b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:03.392 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.327+0000 7f69a3bc7700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f699c156e70 0x7f699c155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:03.393 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.327+0000 7f69a3bc7700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f699c16aa90 con 0x7f6984005080 2022-06-27T11:51:03.393 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.327+0000 7f69a3bc7700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f699c167370 con 0x7f699c156e70 2022-06-27T11:51:03.398 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.327+0000 7f69a3bc7700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f699c068860 con 0x7f699c16a570 2022-06-27T11:51:03.399 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.327+0000 7f69a1963700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6984005080 0x7f699c0c7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:03.399 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.327+0000 7f69a1963700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6984005080 0x7f699c0c7f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:43104/0 (socket says 172.21.15.195:43104) 2022-06-27T11:51:03.400 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.327+0000 7f69a1963700 1 -- 172.21.15.195:0/928960812 learned_addr learned my addr 172.21.15.195:0/928960812 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:51:03.406 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.327+0000 7f69a1162700 1 --2- 172.21.15.195:0/928960812 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f699c16a570 0x7f699c1557b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:03.407 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.327+0000 7f69a2164700 1 --2- 172.21.15.195:0/928960812 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f699c156e70 0x7f699c155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:03.407 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.327+0000 7f69a1963700 1 -- 172.21.15.195:0/928960812 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f699c16a570 msgr2=0x7f699c1557b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:03.407 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.327+0000 7f69a1963700 1 --2- 172.21.15.195:0/928960812 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f699c16a570 0x7f699c1557b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:03.407 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.327+0000 7f69a1963700 1 -- 172.21.15.195:0/928960812 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f699c156e70 msgr2=0x7f699c155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:03.408 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.327+0000 7f69a1963700 1 --2- 172.21.15.195:0/928960812 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f699c156e70 0x7f699c155cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:03.408 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.327+0000 7f69a1963700 1 -- 172.21.15.195:0/928960812 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f699c166760 con 0x7f6984005080 2022-06-27T11:51:03.408 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.327+0000 7f69a1963700 1 --2- 172.21.15.195:0/928960812 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6984005080 0x7f699c0c7f60 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f698c004a50 tx=0x7f698c001dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:51:03.408 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.327+0000 7f6989ffb700 1 -- 172.21.15.195:0/928960812 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f698c000cb0 con 0x7f6984005080 2022-06-27T11:51:03.409 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.327+0000 7f6989ffb700 1 -- 172.21.15.195:0/928960812 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f698c00f960 con 0x7f6984005080 2022-06-27T11:51:03.409 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.327+0000 7f69a3bc7700 1 -- 172.21.15.195:0/928960812 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f699c166a40 con 0x7f6984005080 2022-06-27T11:51:03.409 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.327+0000 7f6989ffb700 1 -- 172.21.15.195:0/928960812 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f698c00fce0 con 0x7f6984005080 2022-06-27T11:51:03.410 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.328+0000 7f69a3bc7700 1 -- 172.21.15.195:0/928960812 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f699c167000 con 0x7f6984005080 2022-06-27T11:51:03.410 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.329+0000 7f69a3bc7700 1 -- 172.21.15.195:0/928960812 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f699c0722c0 con 0x7f6984005080 2022-06-27T11:51:03.410 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.329+0000 7f6989ffb700 1 -- 172.21.15.195:0/928960812 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f698c01c030 con 0x7f6984005080 2022-06-27T11:51:03.410 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.329+0000 7f6989ffb700 1 --2- 172.21.15.195:0/928960812 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f697c063d00 0x7f697c0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:03.410 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.329+0000 7f6989ffb700 1 -- 172.21.15.195:0/928960812 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 7378+0+0 (secure 0 0 0) 0x7f698c0808f0 con 0x7f6984005080 2022-06-27T11:51:03.411 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.329+0000 7f69a1162700 1 --2- 172.21.15.195:0/928960812 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f697c063d00 0x7f697c0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:03.411 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.330+0000 7f69a1162700 1 --2- 172.21.15.195:0/928960812 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f697c063d00 0x7f697c0661b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f6998006d70 tx=0x7f6998008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:51:03.412 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.333+0000 7f6989ffb700 1 -- 172.21.15.195:0/928960812 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f698c04cc90 con 0x7f6984005080 2022-06-27T11:51:03.566 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.566+0000 7f69a3bc7700 1 -- 172.21.15.195:0/928960812 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f699c0690c0 con 0x7f6984005080 2022-06-27T11:51:03.567 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.566+0000 7f6989ffb700 1 -- 172.21.15.195:0/928960812 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+6124 (secure 0 0 0) 0x7f698c01a070 con 0x7f6984005080 2022-06-27T11:51:03.571 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:51:03.571 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4849,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/3271891211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":3271891211},{"type":"v1","addr":"172.21.15.202:6835","nonce":3271891211}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:50:26.047919+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"b","rank":0,"incarnation":9,"state":"up:active","state_seq":43,"addr":"172.21.15.202:6833/864126677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":864126677},{"type":"v1","addr":"172.21.15.202:6833","nonce":864126677}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:51:03.573 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.569+0000 7f69a3bc7700 1 -- 172.21.15.195:0/928960812 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f697c063d00 msgr2=0x7f697c0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:03.575 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.569+0000 7f69a3bc7700 1 --2- 172.21.15.195:0/928960812 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f697c063d00 0x7f697c0661b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f6998006d70 tx=0x7f6998008040 comp rx=0 tx=0).stop 2022-06-27T11:51:03.576 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.569+0000 7f69a3bc7700 1 -- 172.21.15.195:0/928960812 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6984005080 msgr2=0x7f699c0c7f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:03.576 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.569+0000 7f69a3bc7700 1 --2- 172.21.15.195:0/928960812 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6984005080 0x7f699c0c7f60 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f698c004a50 tx=0x7f698c001dd0 comp rx=0 tx=0).stop 2022-06-27T11:51:03.577 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.569+0000 7f69a3bc7700 1 -- 172.21.15.195:0/928960812 shutdown_connections 2022-06-27T11:51:03.578 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.569+0000 7f69a3bc7700 1 --2- 172.21.15.195:0/928960812 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f697c063d00 0x7f697c0661b0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:03.583 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.569+0000 7f69a3bc7700 1 --2- 172.21.15.195:0/928960812 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6984005080 0x7f699c0c7f60 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:03.583 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.569+0000 7f69a3bc7700 1 --2- 172.21.15.195:0/928960812 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f699c156e70 0x7f699c155cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:03.584 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.569+0000 7f69a3bc7700 1 --2- 172.21.15.195:0/928960812 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f699c16a570 0x7f699c1557b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:03.584 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.569+0000 7f69a3bc7700 1 -- 172.21.15.195:0/928960812 >> 172.21.15.195:0/928960812 conn(0x7f699c0c6350 msgr2=0x7f699c0c41e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:51:03.584 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.569+0000 7f69a3bc7700 1 -- 172.21.15.195:0/928960812 shutdown_connections 2022-06-27T11:51:03.585 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:03.569+0000 7f69a3bc7700 1 -- 172.21.15.195:0/928960812 wait complete. 2022-06-27T11:51:03.585 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 14 2022-06-27T11:51:03.600 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2022-06-27T11:51:05.601 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:51:05.736 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.734+0000 7f47de664700 1 Processor -- start 2022-06-27T11:51:05.736 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.734+0000 7f47de664700 1 -- start start 2022-06-27T11:51:05.737 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.735+0000 7f47de664700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f47d80c4b50 0x7f47d80683c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:05.737 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.735+0000 7f47de664700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47d81573b0 con 0x7f47d80c7480 2022-06-27T11:51:05.738 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.735+0000 7f47de664700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47d80c46c0 con 0x7f47d8068f80 2022-06-27T11:51:05.738 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.735+0000 7f47de664700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47d8156b70 con 0x7f47d80c4b50 2022-06-27T11:51:05.739 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.735+0000 7f47d77fe700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f47d80c7480 0x7f47d80c6510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:49300/0 (socket says 172.21.15.195:49300) 2022-06-27T11:51:05.739 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.735+0000 7f47d77fe700 1 -- 172.21.15.195:0/1173932461 learned_addr learned my addr 172.21.15.195:0/1173932461 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:51:05.740 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.735+0000 7f47dcc01700 1 --2- 172.21.15.195:0/1173932461 >> v2:172.21.15.202:3300/0 conn(0x7f47d80c4b50 0x7f47d80683c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:05.740 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.736+0000 7f47d6ffd700 1 -- 172.21.15.195:0/1173932461 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1830091617 0 0) 0x7f47d81573b0 con 0x7f47d80c7480 2022-06-27T11:51:05.740 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.736+0000 7f47d6ffd700 1 -- 172.21.15.195:0/1173932461 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47c4003430 con 0x7f47d80c7480 2022-06-27T11:51:05.741 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.736+0000 7f47dcc01700 1 -- 172.21.15.195:0/1173932461 >> v1:172.21.15.202:6789/0 conn(0x7f47d8068f80 legacy=0x7f47d8156460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:05.741 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.736+0000 7f47dcc01700 1 -- 172.21.15.195:0/1173932461 >> v1:172.21.15.195:6789/0 conn(0x7f47d80c7480 legacy=0x7f47d80c6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:05.747 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.736+0000 7f47d6ffd700 1 -- 172.21.15.195:0/1173932461 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1407819330 0 0) 0x7f47c4003430 con 0x7f47d80c7480 2022-06-27T11:51:05.748 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.736+0000 7f47dcc01700 1 -- 172.21.15.195:0/1173932461 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47d806a920 con 0x7f47d80c4b50 2022-06-27T11:51:05.748 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.736+0000 7f47dcc01700 1 --2- 172.21.15.195:0/1173932461 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f47d80c4b50 0x7f47d80683c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f47d0009e60 tx=0x7f47d0009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2964edf61cf258de server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:51:05.748 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.736+0000 7f47d6ffd700 1 -- 172.21.15.195:0/1173932461 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47d000e070 con 0x7f47d80c4b50 2022-06-27T11:51:05.749 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.736+0000 7f47d6ffd700 1 -- 172.21.15.195:0/1173932461 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47d00038a0 con 0x7f47d80c4b50 2022-06-27T11:51:05.749 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.736+0000 7f47d6ffd700 1 -- 172.21.15.195:0/1173932461 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47c8001470 con 0x7f47d80c4b50 2022-06-27T11:51:05.749 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.737+0000 7f47de664700 1 -- 172.21.15.195:0/1173932461 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f47d80c4b50 msgr2=0x7f47d80683c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:05.749 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.737+0000 7f47de664700 1 --2- 172.21.15.195:0/1173932461 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f47d80c4b50 0x7f47d80683c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f47d0009e60 tx=0x7f47d0009780 comp rx=0 tx=0).stop 2022-06-27T11:51:05.750 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.737+0000 7f47de664700 1 -- 172.21.15.195:0/1173932461 shutdown_connections 2022-06-27T11:51:05.750 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.737+0000 7f47de664700 1 --2- 172.21.15.195:0/1173932461 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f47d80c4b50 0x7f47d80683c0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:05.750 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.737+0000 7f47de664700 1 -- 172.21.15.195:0/1173932461 >> 172.21.15.195:0/1173932461 conn(0x7f47d80ccbd0 msgr2=0x7f47d80ccfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:51:05.750 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.737+0000 7f47de664700 1 -- 172.21.15.195:0/1173932461 shutdown_connections 2022-06-27T11:51:05.751 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.737+0000 7f47de664700 1 -- 172.21.15.195:0/1173932461 wait complete. 2022-06-27T11:51:05.751 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.738+0000 7f47de664700 1 Processor -- start 2022-06-27T11:51:05.751 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.738+0000 7f47de664700 1 -- start start 2022-06-27T11:51:05.752 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.739+0000 7f47de664700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f47d8068f80 0x7f47d814f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:05.752 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.739+0000 7f47de664700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f47d80c4b50 0x7f47d815bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:05.753 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.739+0000 7f47de664700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f47d80c7480 0x7f47d815c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:05.753 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.739+0000 7f47de664700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47d815b690 con 0x7f47d80c7480 2022-06-27T11:51:05.753 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.739+0000 7f47de664700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47d80686b0 con 0x7f47d8068f80 2022-06-27T11:51:05.754 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.739+0000 7f47de664700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47d8166430 con 0x7f47d80c4b50 2022-06-27T11:51:05.754 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.739+0000 7f47dcc01700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f47d80c7480 0x7f47d815c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:05.754 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.739+0000 7f47dcc01700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f47d80c7480 0x7f47d815c060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:43122/0 (socket says 172.21.15.195:43122) 2022-06-27T11:51:05.755 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.739+0000 7f47dcc01700 1 -- 172.21.15.195:0/3593830972 learned_addr learned my addr 172.21.15.195:0/3593830972 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:51:05.755 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.739+0000 7f47d7fff700 1 --2- 172.21.15.195:0/3593830972 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f47d8068f80 0x7f47d814f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:05.756 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.739+0000 7f47d77fe700 1 --2- 172.21.15.195:0/3593830972 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f47d80c4b50 0x7f47d815bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:05.756 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.739+0000 7f47dcc01700 1 -- 172.21.15.195:0/3593830972 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f47d80c4b50 msgr2=0x7f47d815bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:05.757 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.739+0000 7f47dcc01700 1 --2- 172.21.15.195:0/3593830972 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f47d80c4b50 0x7f47d815bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:05.757 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.739+0000 7f47dcc01700 1 -- 172.21.15.195:0/3593830972 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f47d8068f80 msgr2=0x7f47d814f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:05.758 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.739+0000 7f47dcc01700 1 --2- 172.21.15.195:0/3593830972 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f47d8068f80 0x7f47d814f4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:05.758 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.739+0000 7f47dcc01700 1 -- 172.21.15.195:0/3593830972 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47d0009a20 con 0x7f47d80c7480 2022-06-27T11:51:05.759 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.739+0000 7f47dcc01700 1 --2- 172.21.15.195:0/3593830972 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f47d80c7480 0x7f47d815c060 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f47d00097c0 tx=0x7f47d0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:51:05.759 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.740+0000 7f47bffff700 1 -- 172.21.15.195:0/3593830972 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47d0004240 con 0x7f47d80c7480 2022-06-27T11:51:05.760 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.740+0000 7f47bffff700 1 -- 172.21.15.195:0/3593830972 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47d00043a0 con 0x7f47d80c7480 2022-06-27T11:51:05.760 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.740+0000 7f47de664700 1 -- 172.21.15.195:0/3593830972 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47d81666d0 con 0x7f47d80c7480 2022-06-27T11:51:05.761 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.740+0000 7f47bffff700 1 -- 172.21.15.195:0/3593830972 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47d0004720 con 0x7f47d80c7480 2022-06-27T11:51:05.761 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.740+0000 7f47de664700 1 -- 172.21.15.195:0/3593830972 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47d815ad60 con 0x7f47d80c7480 2022-06-27T11:51:05.761 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.740+0000 7f47de664700 1 -- 172.21.15.195:0/3593830972 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47d815b840 con 0x7f47d80c7480 2022-06-27T11:51:05.762 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.741+0000 7f47bffff700 1 -- 172.21.15.195:0/3593830972 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f47d002d070 con 0x7f47d80c7480 2022-06-27T11:51:05.762 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.741+0000 7f47bffff700 1 --2- 172.21.15.195:0/3593830972 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f47b8063d00 0x7f47b80661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:05.763 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.742+0000 7f47d7fff700 1 --2- 172.21.15.195:0/3593830972 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f47b8063d00 0x7f47b80661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:05.763 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.742+0000 7f47bffff700 1 -- 172.21.15.195:0/3593830972 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 7378+0+0 (secure 0 0 0) 0x7f47d00826b0 con 0x7f47d80c7480 2022-06-27T11:51:05.764 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.742+0000 7f47d7fff700 1 --2- 172.21.15.195:0/3593830972 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f47b8063d00 0x7f47b80661b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f47cc001880 tx=0x7f47cc009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:51:05.765 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.746+0000 7f47bffff700 1 -- 172.21.15.195:0/3593830972 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f47d815b840 con 0x7f47d80c7480 2022-06-27T11:51:05.975 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.967+0000 7f47de664700 1 -- 172.21.15.195:0/3593830972 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f47d815b7d0 con 0x7f47d80c7480 2022-06-27T11:51:05.975 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.967+0000 7f47bffff700 1 -- 172.21.15.195:0/3593830972 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+6124 (secure 0 0 0) 0x7f47d0016090 con 0x7f47d80c7480 2022-06-27T11:51:05.976 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:51:05.977 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4849,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/3271891211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":3271891211},{"type":"v1","addr":"172.21.15.202:6835","nonce":3271891211}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:50:26.047919+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"b","rank":0,"incarnation":9,"state":"up:active","state_seq":43,"addr":"172.21.15.202:6833/864126677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":864126677},{"type":"v1","addr":"172.21.15.202:6833","nonce":864126677}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:51:05.998 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.969+0000 7f47de664700 1 -- 172.21.15.195:0/3593830972 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f47b8063d00 msgr2=0x7f47b80661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:05.998 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.969+0000 7f47de664700 1 --2- 172.21.15.195:0/3593830972 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f47b8063d00 0x7f47b80661b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f47cc001880 tx=0x7f47cc009040 comp rx=0 tx=0).stop 2022-06-27T11:51:05.999 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.969+0000 7f47de664700 1 -- 172.21.15.195:0/3593830972 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f47d80c7480 msgr2=0x7f47d815c060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:05.999 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.969+0000 7f47de664700 1 --2- 172.21.15.195:0/3593830972 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f47d80c7480 0x7f47d815c060 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f47d00097c0 tx=0x7f47d0017000 comp rx=0 tx=0).stop 2022-06-27T11:51:06.000 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.969+0000 7f47de664700 1 -- 172.21.15.195:0/3593830972 shutdown_connections 2022-06-27T11:51:06.001 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.969+0000 7f47de664700 1 --2- 172.21.15.195:0/3593830972 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f47b8063d00 0x7f47b80661b0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:06.001 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.969+0000 7f47de664700 1 --2- 172.21.15.195:0/3593830972 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f47d80c7480 0x7f47d815c060 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:06.002 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.969+0000 7f47de664700 1 --2- 172.21.15.195:0/3593830972 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f47d8068f80 0x7f47d814f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:06.002 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.969+0000 7f47de664700 1 --2- 172.21.15.195:0/3593830972 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f47d80c4b50 0x7f47d815bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:06.003 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.969+0000 7f47de664700 1 -- 172.21.15.195:0/3593830972 >> 172.21.15.195:0/3593830972 conn(0x7f47d80ccbd0 msgr2=0x7f47d8150240 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:51:06.003 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.969+0000 7f47de664700 1 -- 172.21.15.195:0/3593830972 shutdown_connections 2022-06-27T11:51:06.004 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:05.969+0000 7f47de664700 1 -- 172.21.15.195:0/3593830972 wait complete. 2022-06-27T11:51:06.004 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 14 2022-06-27T11:51:06.005 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2022-06-27T11:51:08.011 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:51:08.158 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.145+0000 7fc56effc700 1 Processor -- start 2022-06-27T11:51:08.159 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.145+0000 7fc56effc700 1 -- start start 2022-06-27T11:51:08.159 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.146+0000 7fc56effc700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fc56806f640 0x7fc568156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:08.160 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.146+0000 7fc56effc700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5680c1a20 con 0x7fc5680c39e0 2022-06-27T11:51:08.160 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.146+0000 7fc56effc700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5680c1130 con 0x7fc5680c4160 2022-06-27T11:51:08.160 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.146+0000 7fc56effc700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc56806ea90 con 0x7fc56806f640 2022-06-27T11:51:08.160 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.146+0000 7fc56d599700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fc5680c39e0 0x7fc5681535a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:49318/0 (socket says 172.21.15.195:49318) 2022-06-27T11:51:08.161 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.146+0000 7fc56d599700 1 -- 172.21.15.195:0/2171951142 learned_addr learned my addr 172.21.15.195:0/2171951142 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:51:08.161 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.146+0000 7fc56cd98700 1 --2- 172.21.15.195:0/2171951142 >> v2:172.21.15.202:3300/0 conn(0x7fc56806f640 0x7fc568156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:08.161 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.146+0000 7fc5677fe700 1 -- 172.21.15.195:0/2171951142 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3904393529 0 0) 0x7fc5680c1a20 con 0x7fc5680c39e0 2022-06-27T11:51:08.162 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.147+0000 7fc5677fe700 1 -- 172.21.15.195:0/2171951142 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc54c003430 con 0x7fc5680c39e0 2022-06-27T11:51:08.162 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.147+0000 7fc56cd98700 1 -- 172.21.15.195:0/2171951142 >> v1:172.21.15.202:6789/0 conn(0x7fc5680c4160 legacy=0x7fc5680c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:08.162 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.147+0000 7fc5677fe700 1 -- 172.21.15.195:0/2171951142 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 594717370 0 0) 0x7fc54c003430 con 0x7fc5680c39e0 2022-06-27T11:51:08.163 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.147+0000 7fc56cd98700 1 -- 172.21.15.195:0/2171951142 >> v1:172.21.15.195:6789/0 conn(0x7fc5680c39e0 legacy=0x7fc5681535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:08.163 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.147+0000 7fc56cd98700 1 -- 172.21.15.195:0/2171951142 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc56806ed70 con 0x7fc56806f640 2022-06-27T11:51:08.164 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.147+0000 7fc56cd98700 1 --2- 172.21.15.195:0/2171951142 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc56806f640 0x7fc568156ed0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc55c0097a0 tx=0x7fc55c009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e36a7812383ba69c server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:51:08.164 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.147+0000 7fc5677fe700 1 -- 172.21.15.195:0/2171951142 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5600017f0 con 0x7fc56806f640 2022-06-27T11:51:08.164 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.147+0000 7fc5677fe700 1 -- 172.21.15.195:0/2171951142 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc55c004030 con 0x7fc56806f640 2022-06-27T11:51:08.164 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.147+0000 7fc5677fe700 1 -- 172.21.15.195:0/2171951142 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc55c0043d0 con 0x7fc56806f640 2022-06-27T11:51:08.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.148+0000 7fc56effc700 1 -- 172.21.15.195:0/2171951142 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc56806f640 msgr2=0x7fc568156ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:08.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.148+0000 7fc56effc700 1 --2- 172.21.15.195:0/2171951142 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc56806f640 0x7fc568156ed0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc55c0097a0 tx=0x7fc55c009060 comp rx=0 tx=0).stop 2022-06-27T11:51:08.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.148+0000 7fc56effc700 1 -- 172.21.15.195:0/2171951142 shutdown_connections 2022-06-27T11:51:08.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.148+0000 7fc56effc700 1 --2- 172.21.15.195:0/2171951142 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc56806f640 0x7fc568156ed0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:08.166 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.148+0000 7fc56effc700 1 -- 172.21.15.195:0/2171951142 >> 172.21.15.195:0/2171951142 conn(0x7fc5680c6350 msgr2=0x7fc5680c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:51:08.166 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.148+0000 7fc56effc700 1 -- 172.21.15.195:0/2171951142 shutdown_connections 2022-06-27T11:51:08.166 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.148+0000 7fc56effc700 1 -- 172.21.15.195:0/2171951142 wait complete. 2022-06-27T11:51:08.166 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.149+0000 7fc56effc700 1 Processor -- start 2022-06-27T11:51:08.166 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.149+0000 7fc56effc700 1 -- start start 2022-06-27T11:51:08.167 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.150+0000 7fc56effc700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc56806f640 0x7fc56806a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:08.167 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.150+0000 7fc56effc700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc5680c39e0 0x7fc5680bece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:08.167 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.150+0000 7fc56effc700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc5680c4160 0x7fc5680bf220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:08.167 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.150+0000 7fc56effc700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc568167060 con 0x7fc56806f640 2022-06-27T11:51:08.168 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.150+0000 7fc56effc700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5680bc520 con 0x7fc5680c39e0 2022-06-27T11:51:08.168 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.150+0000 7fc56effc700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5680bcaa0 con 0x7fc5680c4160 2022-06-27T11:51:08.168 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.150+0000 7fc56cd98700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc56806f640 0x7fc56806a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:08.168 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.150+0000 7fc56cd98700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc56806f640 0x7fc56806a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:43132/0 (socket says 172.21.15.195:43132) 2022-06-27T11:51:08.169 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.150+0000 7fc56cd98700 1 -- 172.21.15.195:0/2477214966 learned_addr learned my addr 172.21.15.195:0/2477214966 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:51:08.169 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.150+0000 7fc567fff700 1 --2- 172.21.15.195:0/2477214966 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc5680c39e0 0x7fc5680bece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:08.169 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.151+0000 7fc56d599700 1 --2- 172.21.15.195:0/2477214966 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc5680c4160 0x7fc5680bf220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:08.169 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.151+0000 7fc56cd98700 1 -- 172.21.15.195:0/2477214966 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc5680c4160 msgr2=0x7fc5680bf220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:08.170 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.151+0000 7fc56cd98700 1 --2- 172.21.15.195:0/2477214966 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc5680c4160 0x7fc5680bf220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:08.170 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.151+0000 7fc56cd98700 1 -- 172.21.15.195:0/2477214966 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc5680c39e0 msgr2=0x7fc5680bece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:08.170 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.151+0000 7fc56cd98700 1 --2- 172.21.15.195:0/2477214966 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc5680c39e0 0x7fc5680bece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:08.170 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.151+0000 7fc56cd98700 1 -- 172.21.15.195:0/2477214966 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc55c0092e0 con 0x7fc56806f640 2022-06-27T11:51:08.171 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.151+0000 7fc56cd98700 1 --2- 172.21.15.195:0/2477214966 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc56806f640 0x7fc56806a790 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fc5680650f0 tx=0x7fc55c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:51:08.171 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.151+0000 7fc564ff9700 1 -- 172.21.15.195:0/2477214966 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc55c01a670 con 0x7fc56806f640 2022-06-27T11:51:08.171 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.151+0000 7fc564ff9700 1 -- 172.21.15.195:0/2477214966 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc55c01a7d0 con 0x7fc56806f640 2022-06-27T11:51:08.171 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.151+0000 7fc56effc700 1 -- 172.21.15.195:0/2477214966 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc568167250 con 0x7fc56806f640 2022-06-27T11:51:08.172 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.151+0000 7fc564ff9700 1 -- 172.21.15.195:0/2477214966 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc55c01ab90 con 0x7fc56806f640 2022-06-27T11:51:08.172 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.151+0000 7fc56effc700 1 -- 172.21.15.195:0/2477214966 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5680bc750 con 0x7fc56806f640 2022-06-27T11:51:08.172 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.152+0000 7fc56effc700 1 -- 172.21.15.195:0/2477214966 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5681675c0 con 0x7fc56806f640 2022-06-27T11:51:08.172 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.152+0000 7fc564ff9700 1 -- 172.21.15.195:0/2477214966 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fc55c02b030 con 0x7fc56806f640 2022-06-27T11:51:08.173 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.153+0000 7fc564ff9700 1 --2- 172.21.15.195:0/2477214966 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc550063d00 0x7fc5500661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:08.173 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.153+0000 7fc564ff9700 1 -- 172.21.15.195:0/2477214966 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 7378+0+0 (secure 0 0 0) 0x7fc55c082340 con 0x7fc56806f640 2022-06-27T11:51:08.173 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.153+0000 7fc567fff700 1 --2- 172.21.15.195:0/2477214966 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc550063d00 0x7fc5500661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:08.174 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.153+0000 7fc567fff700 1 --2- 172.21.15.195:0/2477214966 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc550063d00 0x7fc5500661b0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc5680c56b0 tx=0x7fc554009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:51:08.174 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.157+0000 7fc564ff9700 1 -- 172.21.15.195:0/2477214966 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fc5681675c0 con 0x7fc56806f640 2022-06-27T11:51:08.377 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.377+0000 7fc56effc700 1 -- 172.21.15.195:0/2477214966 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc5681675c0 con 0x7fc56806f640 2022-06-27T11:51:08.383 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.377+0000 7fc564ff9700 1 -- 172.21.15.195:0/2477214966 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+6124 (secure 0 0 0) 0x7fc5681675c0 con 0x7fc56806f640 2022-06-27T11:51:08.384 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:51:08.385 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4849,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/3271891211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":3271891211},{"type":"v1","addr":"172.21.15.202:6835","nonce":3271891211}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:50:26.047919+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"b","rank":0,"incarnation":9,"state":"up:active","state_seq":43,"addr":"172.21.15.202:6833/864126677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":864126677},{"type":"v1","addr":"172.21.15.202:6833","nonce":864126677}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:51:08.393 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.379+0000 7fc56effc700 1 -- 172.21.15.195:0/2477214966 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc550063d00 msgr2=0x7fc5500661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:08.394 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.379+0000 7fc56effc700 1 --2- 172.21.15.195:0/2477214966 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc550063d00 0x7fc5500661b0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc5680c56b0 tx=0x7fc554009040 comp rx=0 tx=0).stop 2022-06-27T11:51:08.395 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.379+0000 7fc56effc700 1 -- 172.21.15.195:0/2477214966 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc56806f640 msgr2=0x7fc56806a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:08.395 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.379+0000 7fc56effc700 1 --2- 172.21.15.195:0/2477214966 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc56806f640 0x7fc56806a790 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fc5680650f0 tx=0x7fc55c016000 comp rx=0 tx=0).stop 2022-06-27T11:51:08.395 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.379+0000 7fc56effc700 1 -- 172.21.15.195:0/2477214966 shutdown_connections 2022-06-27T11:51:08.396 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.379+0000 7fc56effc700 1 --2- 172.21.15.195:0/2477214966 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc550063d00 0x7fc5500661b0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:08.396 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.379+0000 7fc56effc700 1 --2- 172.21.15.195:0/2477214966 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc56806f640 0x7fc56806a790 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:08.396 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.379+0000 7fc56effc700 1 --2- 172.21.15.195:0/2477214966 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc5680c4160 0x7fc5680bf220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:08.397 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.379+0000 7fc56effc700 1 --2- 172.21.15.195:0/2477214966 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc5680c39e0 0x7fc5680bece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:08.397 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.379+0000 7fc56effc700 1 -- 172.21.15.195:0/2477214966 >> 172.21.15.195:0/2477214966 conn(0x7fc5680c6350 msgr2=0x7fc5680c4530 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:51:08.399 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.379+0000 7fc56effc700 1 -- 172.21.15.195:0/2477214966 shutdown_connections 2022-06-27T11:51:08.399 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:08.379+0000 7fc56effc700 1 -- 172.21.15.195:0/2477214966 wait complete. 2022-06-27T11:51:08.400 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 14 2022-06-27T11:51:08.401 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2022-06-27T11:51:10.406 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:51:10.573 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.548+0000 7fe54eaac700 1 Processor -- start 2022-06-27T11:51:10.573 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.548+0000 7fe54eaac700 1 -- start start 2022-06-27T11:51:10.574 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.549+0000 7fe54eaac700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fe548150500 0x7fe548063af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:10.574 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.549+0000 7fe54eaac700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe548064030 con 0x7fe5481508d0 2022-06-27T11:51:10.574 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.549+0000 7fe54eaac700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5480c4160 con 0x7fe54806a450 2022-06-27T11:51:10.575 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.549+0000 7fe54eaac700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5480c4440 con 0x7fe548150500 2022-06-27T11:51:10.575 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.549+0000 7fe54d049700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fe5481508d0 0x7fe5480bbf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:49328/0 (socket says 172.21.15.195:49328) 2022-06-27T11:51:10.575 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.549+0000 7fe54d049700 1 -- 172.21.15.195:0/2507883377 learned_addr learned my addr 172.21.15.195:0/2507883377 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:51:10.575 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.549+0000 7fe54c848700 1 --2- 172.21.15.195:0/2507883377 >> v2:172.21.15.202:3300/0 conn(0x7fe548150500 0x7fe548063af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:10.576 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.549+0000 7fe5477fe700 1 -- 172.21.15.195:0/2507883377 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3857087302 0 0) 0x7fe540001490 con 0x7fe5481508d0 2022-06-27T11:51:10.576 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.549+0000 7fe5477fe700 1 -- 172.21.15.195:0/2507883377 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe52c003430 con 0x7fe5481508d0 2022-06-27T11:51:10.576 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.549+0000 7fe5477fe700 1 -- 172.21.15.195:0/2507883377 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4196814279 0 0) 0x7fe5480c4160 con 0x7fe54806a450 2022-06-27T11:51:10.576 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.549+0000 7fe5477fe700 1 -- 172.21.15.195:0/2507883377 --> v1:172.21.15.202:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe540001490 con 0x7fe54806a450 2022-06-27T11:51:10.576 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.549+0000 7fe5477fe700 1 -- 172.21.15.195:0/2507883377 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3635606315 0 0) 0x7fe52c003430 con 0x7fe5481508d0 2022-06-27T11:51:10.577 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.550+0000 7fe5477fe700 1 -- 172.21.15.195:0/2507883377 >> v2:172.21.15.202:3300/0 conn(0x7fe548150500 msgr2=0x7fe548063af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:51:10.577 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.550+0000 7fe5477fe700 1 --2- 172.21.15.195:0/2507883377 >> v2:172.21.15.202:3300/0 conn(0x7fe548150500 0x7fe548063af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:10.577 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.550+0000 7fe5477fe700 1 -- 172.21.15.195:0/2507883377 >> v1:172.21.15.202:6789/0 conn(0x7fe54806a450 legacy=0x7fe5480c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:10.577 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.550+0000 7fe5477fe700 1 -- 172.21.15.195:0/2507883377 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe54815a800 con 0x7fe5481508d0 2022-06-27T11:51:10.578 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.550+0000 7fe5477fe700 1 -- 172.21.15.195:0/2507883377 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7fe540001980 con 0x7fe5481508d0 2022-06-27T11:51:10.578 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.550+0000 7fe5477fe700 1 -- 172.21.15.195:0/2507883377 >> v1:172.21.15.195:6789/0 conn(0x7fe5481508d0 legacy=0x7fe5480bbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:10.578 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.550+0000 7fe5477fe700 1 --2- 172.21.15.195:0/2507883377 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe52c003ab0 0x7fe52c003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:10.578 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.550+0000 7fe5477fe700 1 --2- 172.21.15.195:0/2507883377 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe52c0046d0 0x7fe52c004b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:10.579 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.550+0000 7fe5477fe700 1 --2- 172.21.15.195:0/2507883377 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe52c005080 0x7fe52c00b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:10.579 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.550+0000 7fe5477fe700 1 -- 172.21.15.195:0/2507883377 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5480c4440 con 0x7fe52c0046d0 2022-06-27T11:51:10.579 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.550+0000 7fe5477fe700 1 -- 172.21.15.195:0/2507883377 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe52c00bf70 con 0x7fe52c005080 2022-06-27T11:51:10.579 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.550+0000 7fe5477fe700 1 -- 172.21.15.195:0/2507883377 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe52c00c130 con 0x7fe52c003ab0 2022-06-27T11:51:10.580 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.550+0000 7fe54c848700 1 --2- 172.21.15.195:0/2507883377 >> v2:172.21.15.202:3300/0 conn(0x7fe548150500 0x7fe548063af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:51:10.580 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.550+0000 7fe547fff700 1 --2- 172.21.15.195:0/2507883377 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe52c0046d0 0x7fe52c004b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:10.580 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.550+0000 7fe54d049700 1 --2- 172.21.15.195:0/2507883377 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe52c005080 0x7fe52c00b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:10.580 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.550+0000 7fe54c848700 1 --2- 172.21.15.195:0/2507883377 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe52c003ab0 0x7fe52c003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:10.581 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.550+0000 7fe547fff700 1 -- 172.21.15.195:0/2507883377 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe52c003ab0 msgr2=0x7fe52c003ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:51:10.581 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.550+0000 7fe547fff700 1 --2- 172.21.15.195:0/2507883377 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe52c003ab0 0x7fe52c003ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:10.581 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.550+0000 7fe547fff700 1 -- 172.21.15.195:0/2507883377 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe52c005080 msgr2=0x7fe52c00b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:51:10.581 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.550+0000 7fe547fff700 1 --2- 172.21.15.195:0/2507883377 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe52c005080 0x7fe52c00b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:10.582 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.550+0000 7fe547fff700 1 -- 172.21.15.195:0/2507883377 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe52c003430 con 0x7fe52c0046d0 2022-06-27T11:51:10.582 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.550+0000 7fe54d049700 1 -- 172.21.15.195:0/2507883377 reap_dead start 2022-06-27T11:51:10.582 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.550+0000 7fe547fff700 1 --2- 172.21.15.195:0/2507883377 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe52c0046d0 0x7fe52c004b40 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fe5340018d0 tx=0x7fe534007bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2baf56fa5a320405 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:51:10.582 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.550+0000 7fe5477fe700 1 -- 172.21.15.195:0/2507883377 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5340050d0 con 0x7fe52c0046d0 2022-06-27T11:51:10.583 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.550+0000 7fe5477fe700 1 -- 172.21.15.195:0/2507883377 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe534005230 con 0x7fe52c0046d0 2022-06-27T11:51:10.583 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.550+0000 7fe5477fe700 1 -- 172.21.15.195:0/2507883377 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe534003750 con 0x7fe52c0046d0 2022-06-27T11:51:10.583 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.551+0000 7fe54eaac700 1 -- 172.21.15.195:0/2507883377 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe52c0046d0 msgr2=0x7fe52c004b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:10.585 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.551+0000 7fe54eaac700 1 --2- 172.21.15.195:0/2507883377 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe52c0046d0 0x7fe52c004b40 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fe5340018d0 tx=0x7fe534007bd0 comp rx=0 tx=0).stop 2022-06-27T11:51:10.585 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.551+0000 7fe54eaac700 1 -- 172.21.15.195:0/2507883377 shutdown_connections 2022-06-27T11:51:10.585 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.551+0000 7fe54eaac700 1 --2- 172.21.15.195:0/2507883377 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe52c0046d0 0x7fe52c004b40 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:10.586 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.551+0000 7fe54eaac700 1 -- 172.21.15.195:0/2507883377 >> 172.21.15.195:0/2507883377 conn(0x7fe5480c9b00 msgr2=0x7fe5480c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:51:10.587 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.551+0000 7fe54eaac700 1 -- 172.21.15.195:0/2507883377 shutdown_connections 2022-06-27T11:51:10.587 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.551+0000 7fe54eaac700 1 -- 172.21.15.195:0/2507883377 wait complete. 2022-06-27T11:51:10.587 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.552+0000 7fe54eaac700 1 Processor -- start 2022-06-27T11:51:10.587 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.552+0000 7fe54eaac700 1 -- start start 2022-06-27T11:51:10.588 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.553+0000 7fe54eaac700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe52c0046d0 0x7fe548152190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:10.588 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.553+0000 7fe54eaac700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe5480c2d40 0x7fe548169b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:10.588 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.553+0000 7fe54eaac700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe54816a060 0x7fe54814f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:10.588 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.553+0000 7fe54eaac700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5480c28b0 con 0x7fe54816a060 2022-06-27T11:51:10.589 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.553+0000 7fe54eaac700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5481526d0 con 0x7fe52c0046d0 2022-06-27T11:51:10.589 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.553+0000 7fe54eaac700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe54816a680 con 0x7fe5480c2d40 2022-06-27T11:51:10.589 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.553+0000 7fe54d049700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe54816a060 0x7fe54814f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:10.589 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.553+0000 7fe54d049700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe54816a060 0x7fe54814f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:43152/0 (socket says 172.21.15.195:43152) 2022-06-27T11:51:10.589 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.553+0000 7fe54d049700 1 -- 172.21.15.195:0/3852573722 learned_addr learned my addr 172.21.15.195:0/3852573722 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:51:10.590 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.553+0000 7fe54c848700 1 --2- 172.21.15.195:0/3852573722 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe52c0046d0 0x7fe548152190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:10.590 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.553+0000 7fe547fff700 1 --2- 172.21.15.195:0/3852573722 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe5480c2d40 0x7fe548169b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:10.590 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.553+0000 7fe54d049700 1 -- 172.21.15.195:0/3852573722 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe5480c2d40 msgr2=0x7fe548169b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:10.590 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.553+0000 7fe54d049700 1 --2- 172.21.15.195:0/3852573722 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe5480c2d40 0x7fe548169b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:10.591 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.553+0000 7fe54d049700 1 -- 172.21.15.195:0/3852573722 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe52c0046d0 msgr2=0x7fe548152190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:10.591 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.553+0000 7fe54d049700 1 --2- 172.21.15.195:0/3852573722 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe52c0046d0 0x7fe548152190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:10.591 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.553+0000 7fe54d049700 1 -- 172.21.15.195:0/3852573722 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5480c1f80 con 0x7fe54816a060 2022-06-27T11:51:10.591 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.553+0000 7fe547fff700 1 --2- 172.21.15.195:0/3852573722 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe5480c2d40 0x7fe548169b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T11:51:10.592 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.554+0000 7fe54d049700 1 --2- 172.21.15.195:0/3852573722 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe54816a060 0x7fe54814f6a0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fe548070690 tx=0x7fe5480bbe70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:51:10.592 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.554+0000 7fe544ff9700 1 -- 172.21.15.195:0/3852573722 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe540018030 con 0x7fe54816a060 2022-06-27T11:51:10.592 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.554+0000 7fe544ff9700 1 -- 172.21.15.195:0/3852573722 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe54000fbd0 con 0x7fe54816a060 2022-06-27T11:51:10.592 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.554+0000 7fe54eaac700 1 -- 172.21.15.195:0/3852573722 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5480c2260 con 0x7fe54816a060 2022-06-27T11:51:10.593 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.554+0000 7fe544ff9700 1 -- 172.21.15.195:0/3852573722 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe540001980 con 0x7fe54816a060 2022-06-27T11:51:10.593 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.554+0000 7fe54eaac700 1 -- 172.21.15.195:0/3852573722 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe548151910 con 0x7fe54816a060 2022-06-27T11:51:10.593 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.555+0000 7fe54eaac700 1 -- 172.21.15.195:0/3852573722 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5480c2a20 con 0x7fe54816a060 2022-06-27T11:51:10.593 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.555+0000 7fe544ff9700 1 -- 172.21.15.195:0/3852573722 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fe540013600 con 0x7fe54816a060 2022-06-27T11:51:10.594 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.555+0000 7fe544ff9700 1 --2- 172.21.15.195:0/3852573722 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe530063d30 0x7fe5300661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:10.594 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.556+0000 7fe544ff9700 1 -- 172.21.15.195:0/3852573722 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 6802+0+0 (secure 0 0 0) 0x7fe540019070 con 0x7fe54816a060 2022-06-27T11:51:10.594 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.556+0000 7fe54c848700 1 --2- 172.21.15.195:0/3852573722 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe530063d30 0x7fe5300661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:10.594 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.556+0000 7fe54c848700 1 --2- 172.21.15.195:0/3852573722 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe530063d30 0x7fe5300661e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe54804dbc0 tx=0x7fe53c008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:51:10.595 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.560+0000 7fe544ff9700 1 -- 172.21.15.195:0/3852573722 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fe54004f860 con 0x7fe54816a060 2022-06-27T11:51:10.786 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.786+0000 7fe54eaac700 1 -- 172.21.15.195:0/3852573722 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe5480722c0 con 0x7fe54816a060 2022-06-27T11:51:10.787 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.786+0000 7fe544ff9700 1 -- 172.21.15.195:0/3852573722 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+6124 (secure 0 0 0) 0x7fe540052e80 con 0x7fe54816a060 2022-06-27T11:51:10.788 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:51:10.788 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4849,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/3271891211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":3271891211},{"type":"v1","addr":"172.21.15.202:6835","nonce":3271891211}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:50:26.047919+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"b","rank":0,"incarnation":9,"state":"up:active","state_seq":43,"addr":"172.21.15.202:6833/864126677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":864126677},{"type":"v1","addr":"172.21.15.202:6833","nonce":864126677}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:51:10.792 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.788+0000 7fe54eaac700 1 -- 172.21.15.195:0/3852573722 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe530063d30 msgr2=0x7fe5300661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:10.792 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.788+0000 7fe54eaac700 1 --2- 172.21.15.195:0/3852573722 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe530063d30 0x7fe5300661e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe54804dbc0 tx=0x7fe53c008040 comp rx=0 tx=0).stop 2022-06-27T11:51:10.793 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.788+0000 7fe54eaac700 1 -- 172.21.15.195:0/3852573722 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe54816a060 msgr2=0x7fe54814f6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:10.793 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.788+0000 7fe54eaac700 1 --2- 172.21.15.195:0/3852573722 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe54816a060 0x7fe54814f6a0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fe548070690 tx=0x7fe5480bbe70 comp rx=0 tx=0).stop 2022-06-27T11:51:10.793 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.788+0000 7fe54eaac700 1 -- 172.21.15.195:0/3852573722 shutdown_connections 2022-06-27T11:51:10.794 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.788+0000 7fe54eaac700 1 --2- 172.21.15.195:0/3852573722 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe530063d30 0x7fe5300661e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:10.794 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.788+0000 7fe54eaac700 1 --2- 172.21.15.195:0/3852573722 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe54816a060 0x7fe54814f6a0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:10.795 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.788+0000 7fe54eaac700 1 --2- 172.21.15.195:0/3852573722 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe52c0046d0 0x7fe548152190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:10.795 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.788+0000 7fe54eaac700 1 --2- 172.21.15.195:0/3852573722 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe5480c2d40 0x7fe548169b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:10.795 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.788+0000 7fe54eaac700 1 -- 172.21.15.195:0/3852573722 >> 172.21.15.195:0/3852573722 conn(0x7fe5480c9b00 msgr2=0x7fe5480c8860 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:51:10.796 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.788+0000 7fe54eaac700 1 -- 172.21.15.195:0/3852573722 shutdown_connections 2022-06-27T11:51:10.796 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:10.789+0000 7fe54eaac700 1 -- 172.21.15.195:0/3852573722 wait complete. 2022-06-27T11:51:10.797 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 14 2022-06-27T11:51:10.803 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2022-06-27T11:51:12.807 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:51:12.971 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:12.952+0000 7f0b2717d700 1 Processor -- start 2022-06-27T11:51:12.971 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:12.952+0000 7f0b2717d700 1 -- start start 2022-06-27T11:51:12.972 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:12.953+0000 7f0b2717d700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0b2006fef0 0x7f0b201570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:12.972 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:12.953+0000 7f0b2717d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f0b200c34a0 0x7f0b200c2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:12.973 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:12.953+0000 7f0b2717d700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b2006fba0 con 0x7f0b2006e850 2022-06-27T11:51:12.973 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:12.953+0000 7f0b2717d700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b201575e0 con 0x7f0b200c34a0 2022-06-27T11:51:12.974 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:12.953+0000 7f0b2717d700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b2006f5e0 con 0x7f0b2006fef0 2022-06-27T11:51:12.974 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:12.953+0000 7f0b2571a700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f0b200c34a0 0x7f0b200c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:12.975 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:12.953+0000 7f0b24f19700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f0b2006e850 0x7f0b20069de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:49344/0 (socket says 172.21.15.195:49344) 2022-06-27T11:51:12.975 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:12.953+0000 7f0b24f19700 1 -- 172.21.15.195:0/2785933702 learned_addr learned my addr 172.21.15.195:0/2785933702 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:51:12.975 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:12.953+0000 7f0b1ffff700 1 --2- 172.21.15.195:0/2785933702 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0b2006fef0 0x7f0b201570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:12.976 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:12.953+0000 7f0b2571a700 1 -- 172.21.15.195:0/2785933702 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0b2006fef0 msgr2=0x7f0b201570a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:51:12.976 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:12.953+0000 7f0b2571a700 1 --2- 172.21.15.195:0/2785933702 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0b2006fef0 0x7f0b201570a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:12.977 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:12.953+0000 7f0b2571a700 1 -- 172.21.15.195:0/2785933702 >> v1:172.21.15.195:6789/0 conn(0x7f0b2006e850 legacy=0x7f0b20069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:12.977 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:12.953+0000 7f0b2571a700 1 -- 172.21.15.195:0/2785933702 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b2006f8c0 con 0x7f0b200c34a0 2022-06-27T11:51:12.978 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:12.953+0000 7f0b1f7fe700 1 -- 172.21.15.195:0/2785933702 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1354382478 0 0) 0x7f0b2006fba0 con 0x7f0b2006e850 2022-06-27T11:51:12.980 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:12.954+0000 7f0b2571a700 1 --2- 172.21.15.195:0/2785933702 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0b200c34a0 0x7f0b200c2920 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f0b18009e60 tx=0x7f0b18009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9f1073fb664f67c server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:51:12.980 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:12.954+0000 7f0b1f7fe700 1 -- 172.21.15.195:0/2785933702 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b1800e070 con 0x7f0b200c34a0 2022-06-27T11:51:12.981 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:12.954+0000 7f0b1f7fe700 1 -- 172.21.15.195:0/2785933702 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b1800bcc0 con 0x7f0b200c34a0 2022-06-27T11:51:12.981 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:12.954+0000 7f0b1f7fe700 1 -- 172.21.15.195:0/2785933702 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b18012680 con 0x7f0b200c34a0 2022-06-27T11:51:12.982 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:12.954+0000 7f0b2717d700 1 -- 172.21.15.195:0/2785933702 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0b200c34a0 msgr2=0x7f0b200c2920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:12.983 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:12.954+0000 7f0b2717d700 1 --2- 172.21.15.195:0/2785933702 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0b200c34a0 0x7f0b200c2920 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f0b18009e60 tx=0x7f0b18009780 comp rx=0 tx=0).stop 2022-06-27T11:51:12.983 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:12.954+0000 7f0b2717d700 1 -- 172.21.15.195:0/2785933702 shutdown_connections 2022-06-27T11:51:12.984 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:12.954+0000 7f0b2717d700 1 --2- 172.21.15.195:0/2785933702 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0b200c34a0 0x7f0b200c2920 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:12.984 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:12.954+0000 7f0b2717d700 1 --2- 172.21.15.195:0/2785933702 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0b2006fef0 0x7f0b201570a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:12.985 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:12.954+0000 7f0b2717d700 1 -- 172.21.15.195:0/2785933702 >> 172.21.15.195:0/2785933702 conn(0x7f0b200c6350 msgr2=0x7f0b200c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:51:12.986 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:12.954+0000 7f0b2717d700 1 -- 172.21.15.195:0/2785933702 shutdown_connections 2022-06-27T11:51:12.986 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:12.954+0000 7f0b2717d700 1 -- 172.21.15.195:0/2785933702 wait complete. 2022-06-27T11:51:12.987 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:12.955+0000 7f0b2717d700 1 Processor -- start 2022-06-27T11:51:12.987 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:12.956+0000 7f0b2717d700 1 -- start start 2022-06-27T11:51:12.988 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:12.956+0000 7f0b2717d700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0b2006e850 0x7f0b201549a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:12.989 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:12.956+0000 7f0b2717d700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0b200c34a0 0x7f0b201660d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:12.989 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:12.956+0000 7f0b2717d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0b201542e0 0x7f0b20149940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:12.989 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:12.956+0000 7f0b2717d700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b20155010 con 0x7f0b201542e0 2022-06-27T11:51:12.990 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:12.956+0000 7f0b2717d700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b20166610 con 0x7f0b200c34a0 2022-06-27T11:51:12.990 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:12.956+0000 7f0b2717d700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b200ba740 con 0x7f0b2006e850 2022-06-27T11:51:12.990 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:12.956+0000 7f0b2571a700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0b201542e0 0x7f0b20149940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:12.991 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:12.956+0000 7f0b2571a700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0b201542e0 0x7f0b20149940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:43164/0 (socket says 172.21.15.195:43164) 2022-06-27T11:51:12.991 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:12.956+0000 7f0b2571a700 1 -- 172.21.15.195:0/2008406547 learned_addr learned my addr 172.21.15.195:0/2008406547 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:51:12.992 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:12.957+0000 7f0b24f19700 1 --2- 172.21.15.195:0/2008406547 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0b2006e850 0x7f0b201549a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:12.993 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:12.957+0000 7f0b2571a700 1 -- 172.21.15.195:0/2008406547 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0b2006e850 msgr2=0x7f0b201549a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:12.993 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:12.957+0000 7f0b1ffff700 1 --2- 172.21.15.195:0/2008406547 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0b200c34a0 0x7f0b201660d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:12.994 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:12.957+0000 7f0b2571a700 1 --2- 172.21.15.195:0/2008406547 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0b2006e850 0x7f0b201549a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:12.994 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:12.957+0000 7f0b2571a700 1 -- 172.21.15.195:0/2008406547 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0b200c34a0 msgr2=0x7f0b201660d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:12.995 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:12.957+0000 7f0b2571a700 1 --2- 172.21.15.195:0/2008406547 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0b200c34a0 0x7f0b201660d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:12.995 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:12.957+0000 7f0b2571a700 1 -- 172.21.15.195:0/2008406547 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b18009a20 con 0x7f0b201542e0 2022-06-27T11:51:12.996 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:12.957+0000 7f0b24f19700 1 --2- 172.21.15.195:0/2008406547 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0b2006e850 0x7f0b201549a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T11:51:12.996 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:12.957+0000 7f0b1ffff700 1 --2- 172.21.15.195:0/2008406547 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0b200c34a0 0x7f0b201660d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T11:51:12.997 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:12.957+0000 7f0b2571a700 1 --2- 172.21.15.195:0/2008406547 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0b201542e0 0x7f0b20149940 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f0b200650f0 tx=0x7f0b18019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:51:12.997 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:12.957+0000 7f0b1cff9700 1 -- 172.21.15.195:0/2008406547 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b18003ab0 con 0x7f0b201542e0 2022-06-27T11:51:12.998 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:12.957+0000 7f0b1cff9700 1 -- 172.21.15.195:0/2008406547 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b18003c30 con 0x7f0b201542e0 2022-06-27T11:51:12.998 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:12.957+0000 7f0b2717d700 1 -- 172.21.15.195:0/2008406547 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b2006a170 con 0x7f0b201542e0 2022-06-27T11:51:12.999 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:12.957+0000 7f0b1cff9700 1 -- 172.21.15.195:0/2008406547 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b18004040 con 0x7f0b201542e0 2022-06-27T11:51:12.999 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:12.957+0000 7f0b2717d700 1 -- 172.21.15.195:0/2008406547 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b200688e0 con 0x7f0b201542e0 2022-06-27T11:51:13.000 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:12.958+0000 7f0b2717d700 1 -- 172.21.15.195:0/2008406547 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b200ba8f0 con 0x7f0b201542e0 2022-06-27T11:51:13.000 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:12.959+0000 7f0b1cff9700 1 -- 172.21.15.195:0/2008406547 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f0b180043d0 con 0x7f0b201542e0 2022-06-27T11:51:13.000 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:12.959+0000 7f0b1cff9700 1 --2- 172.21.15.195:0/2008406547 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0b08063d30 0x7f0b080661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:13.001 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:12.959+0000 7f0b1cff9700 1 -- 172.21.15.195:0/2008406547 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 6802+0+0 (secure 0 0 0) 0x7f0b180839d0 con 0x7f0b201542e0 2022-06-27T11:51:13.001 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:12.959+0000 7f0b24f19700 1 --2- 172.21.15.195:0/2008406547 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0b08063d30 0x7f0b080661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:13.002 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:12.959+0000 7f0b24f19700 1 --2- 172.21.15.195:0/2008406547 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0b08063d30 0x7f0b080661e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f0b140016d0 tx=0x7f0b14009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:51:13.002 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:12.965+0000 7f0b1cff9700 1 -- 172.21.15.195:0/2008406547 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f0b200ba8f0 con 0x7f0b201542e0 2022-06-27T11:51:13.197 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:13.194+0000 7f0b2717d700 1 -- 172.21.15.195:0/2008406547 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b200baef0 con 0x7f0b201542e0 2022-06-27T11:51:13.198 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:13.195+0000 7f0b1cff9700 1 -- 172.21.15.195:0/2008406547 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+6124 (secure 0 0 0) 0x7f0b180522d0 con 0x7f0b201542e0 2022-06-27T11:51:13.199 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:51:13.199 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4849,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/3271891211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":3271891211},{"type":"v1","addr":"172.21.15.202:6835","nonce":3271891211}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:50:26.047919+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"b","rank":0,"incarnation":9,"state":"up:active","state_seq":43,"addr":"172.21.15.202:6833/864126677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":864126677},{"type":"v1","addr":"172.21.15.202:6833","nonce":864126677}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:51:13.201 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:13.197+0000 7f0b2717d700 1 -- 172.21.15.195:0/2008406547 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0b08063d30 msgr2=0x7f0b080661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:13.201 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:13.197+0000 7f0b2717d700 1 --2- 172.21.15.195:0/2008406547 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0b08063d30 0x7f0b080661e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f0b140016d0 tx=0x7f0b14009040 comp rx=0 tx=0).stop 2022-06-27T11:51:13.202 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:13.197+0000 7f0b2717d700 1 -- 172.21.15.195:0/2008406547 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0b201542e0 msgr2=0x7f0b20149940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:13.202 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:13.197+0000 7f0b2717d700 1 --2- 172.21.15.195:0/2008406547 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0b201542e0 0x7f0b20149940 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f0b200650f0 tx=0x7f0b18019000 comp rx=0 tx=0).stop 2022-06-27T11:51:13.203 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:13.197+0000 7f0b2717d700 1 -- 172.21.15.195:0/2008406547 shutdown_connections 2022-06-27T11:51:13.203 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:13.197+0000 7f0b2717d700 1 --2- 172.21.15.195:0/2008406547 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0b08063d30 0x7f0b080661e0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:13.203 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:13.197+0000 7f0b2717d700 1 --2- 172.21.15.195:0/2008406547 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0b201542e0 0x7f0b20149940 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:13.203 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:13.197+0000 7f0b2717d700 1 --2- 172.21.15.195:0/2008406547 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0b2006e850 0x7f0b201549a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:13.203 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:13.197+0000 7f0b2717d700 1 --2- 172.21.15.195:0/2008406547 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0b200c34a0 0x7f0b201660d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:13.204 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:13.197+0000 7f0b2717d700 1 -- 172.21.15.195:0/2008406547 >> 172.21.15.195:0/2008406547 conn(0x7f0b200c6350 msgr2=0x7f0b200c76e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:51:13.204 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:13.197+0000 7f0b2717d700 1 -- 172.21.15.195:0/2008406547 shutdown_connections 2022-06-27T11:51:13.204 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:13.197+0000 7f0b2717d700 1 -- 172.21.15.195:0/2008406547 wait complete. 2022-06-27T11:51:13.205 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 14 2022-06-27T11:51:13.209 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2022-06-27T11:51:15.211 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:51:15.355 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.351+0000 7f1cc6e63700 1 Processor -- start 2022-06-27T11:51:15.356 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.351+0000 7f1cc6e63700 1 -- start start 2022-06-27T11:51:15.356 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.352+0000 7f1cc6e63700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1cc00c4160 0x7f1cc0156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:15.357 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.352+0000 7f1cc6e63700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cc00c1a20 con 0x7f1cc006f640 2022-06-27T11:51:15.357 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.352+0000 7f1cc6e63700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cc00c1130 con 0x7f1cc00c39e0 2022-06-27T11:51:15.361 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.352+0000 7f1cc6e63700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cc006ea90 con 0x7f1cc00c4160 2022-06-27T11:51:15.365 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.352+0000 7f1cbffff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1cc00c4160 0x7f1cc0156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:15.366 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.352+0000 7f1cc4bff700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f1cc006f640 0x7f1cc00c1eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:49356/0 (socket says 172.21.15.195:49356) 2022-06-27T11:51:15.366 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.352+0000 7f1cc4bff700 1 -- 172.21.15.195:0/3402954994 learned_addr learned my addr 172.21.15.195:0/3402954994 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:51:15.367 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.352+0000 7f1cbffff700 1 -- 172.21.15.195:0/3402954994 >> v1:172.21.15.202:6789/0 conn(0x7f1cc00c39e0 legacy=0x7f1cc01535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:51:15.367 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.352+0000 7f1cbffff700 1 -- 172.21.15.195:0/3402954994 >> v1:172.21.15.195:6789/0 conn(0x7f1cc006f640 legacy=0x7f1cc00c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:15.368 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.352+0000 7f1cbf7fe700 1 -- 172.21.15.195:0/3402954994 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 967629913 0 0) 0x7f1cc00c1a20 con 0x7f1cc006f640 2022-06-27T11:51:15.368 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.352+0000 7f1cbffff700 1 -- 172.21.15.195:0/3402954994 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cc006ed70 con 0x7f1cc00c4160 2022-06-27T11:51:15.369 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.352+0000 7f1cbffff700 1 --2- 172.21.15.195:0/3402954994 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1cc00c4160 0x7f1cc0156ed0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f1cac0097a0 tx=0x7f1cac009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99680bb196119eb5 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:51:15.369 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.352+0000 7f1cbf7fe700 1 -- 172.21.15.195:0/3402954994 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cac004030 con 0x7f1cc00c4160 2022-06-27T11:51:15.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.352+0000 7f1cbf7fe700 1 -- 172.21.15.195:0/3402954994 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cac004190 con 0x7f1cc00c4160 2022-06-27T11:51:15.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.352+0000 7f1cbf7fe700 1 -- 172.21.15.195:0/3402954994 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cac011630 con 0x7f1cc00c4160 2022-06-27T11:51:15.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.353+0000 7f1cc6e63700 1 -- 172.21.15.195:0/3402954994 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1cc00c4160 msgr2=0x7f1cc0156ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:15.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.353+0000 7f1cc6e63700 1 --2- 172.21.15.195:0/3402954994 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1cc00c4160 0x7f1cc0156ed0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f1cac0097a0 tx=0x7f1cac009060 comp rx=0 tx=0).stop 2022-06-27T11:51:15.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.353+0000 7f1cc6e63700 1 -- 172.21.15.195:0/3402954994 shutdown_connections 2022-06-27T11:51:15.372 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.353+0000 7f1cc6e63700 1 --2- 172.21.15.195:0/3402954994 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1cc00c4160 0x7f1cc0156ed0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:15.372 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.353+0000 7f1cc6e63700 1 -- 172.21.15.195:0/3402954994 >> 172.21.15.195:0/3402954994 conn(0x7f1cc00c6350 msgr2=0x7f1cc00c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:51:15.372 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.353+0000 7f1cc6e63700 1 -- 172.21.15.195:0/3402954994 shutdown_connections 2022-06-27T11:51:15.373 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.353+0000 7f1cc6e63700 1 -- 172.21.15.195:0/3402954994 wait complete. 2022-06-27T11:51:15.373 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.354+0000 7f1cc6e63700 1 Processor -- start 2022-06-27T11:51:15.374 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.354+0000 7f1cc6e63700 1 -- start start 2022-06-27T11:51:15.374 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.355+0000 7f1cc6e63700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1cc006f640 0x7f1cc015c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:15.375 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.355+0000 7f1cc6e63700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1cc00c39e0 0x7f1cc015b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:15.375 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.355+0000 7f1cc6e63700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1cc00c4160 0x7f1cc015b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:15.375 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.355+0000 7f1cc6e63700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc015b010 con 0x7f1cc00c4160 2022-06-27T11:51:15.376 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.355+0000 7f1cc6e63700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc0157fb0 con 0x7f1cc006f640 2022-06-27T11:51:15.376 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.355+0000 7f1cc6e63700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc00baf50 con 0x7f1cc00c39e0 2022-06-27T11:51:15.382 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.355+0000 7f1cc5400700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1cc00c4160 0x7f1cc015b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:15.383 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.355+0000 7f1cc5400700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1cc00c4160 0x7f1cc015b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:43178/0 (socket says 172.21.15.195:43178) 2022-06-27T11:51:15.383 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.355+0000 7f1cc5400700 1 -- 172.21.15.195:0/312151180 learned_addr learned my addr 172.21.15.195:0/312151180 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:51:15.383 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.355+0000 7f1cc4bff700 1 --2- 172.21.15.195:0/312151180 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1cc006f640 0x7f1cc015c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:15.384 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.355+0000 7f1cbffff700 1 --2- 172.21.15.195:0/312151180 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1cc00c39e0 0x7f1cc015b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:15.384 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.356+0000 7f1cc5400700 1 -- 172.21.15.195:0/312151180 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1cc00c39e0 msgr2=0x7f1cc015b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:15.385 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.356+0000 7f1cc5400700 1 --2- 172.21.15.195:0/312151180 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1cc00c39e0 0x7f1cc015b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:15.385 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.356+0000 7f1cc5400700 1 -- 172.21.15.195:0/312151180 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1cc006f640 msgr2=0x7f1cc015c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:15.386 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.356+0000 7f1cc5400700 1 --2- 172.21.15.195:0/312151180 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1cc006f640 0x7f1cc015c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:15.386 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.356+0000 7f1cc5400700 1 -- 172.21.15.195:0/312151180 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cac0092e0 con 0x7f1cc00c4160 2022-06-27T11:51:15.387 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.356+0000 7f1cc4bff700 1 --2- 172.21.15.195:0/312151180 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1cc006f640 0x7f1cc015c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T11:51:15.387 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.356+0000 7f1cc5400700 1 --2- 172.21.15.195:0/312151180 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1cc00c4160 0x7f1cc015b9e0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f1cb800c810 tx=0x7f1cb8003bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:51:15.387 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.356+0000 7f1cbcff9700 1 -- 172.21.15.195:0/312151180 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb8012630 con 0x7f1cc00c4160 2022-06-27T11:51:15.388 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.356+0000 7f1cbcff9700 1 -- 172.21.15.195:0/312151180 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cb80127d0 con 0x7f1cc00c4160 2022-06-27T11:51:15.389 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.356+0000 7f1cc6e63700 1 -- 172.21.15.195:0/312151180 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cc00bb230 con 0x7f1cc00c4160 2022-06-27T11:51:15.389 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.356+0000 7f1cbcff9700 1 -- 172.21.15.195:0/312151180 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb8012b90 con 0x7f1cc00c4160 2022-06-27T11:51:15.394 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.356+0000 7f1cc6e63700 1 -- 172.21.15.195:0/312151180 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cc00bb7f0 con 0x7f1cc00c4160 2022-06-27T11:51:15.395 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.357+0000 7f1cc6e63700 1 -- 172.21.15.195:0/312151180 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cc00bba10 con 0x7f1cc00c4160 2022-06-27T11:51:15.395 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.357+0000 7f1cbcff9700 1 -- 172.21.15.195:0/312151180 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f1cb8022070 con 0x7f1cc00c4160 2022-06-27T11:51:15.396 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.358+0000 7f1cbcff9700 1 --2- 172.21.15.195:0/312151180 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1ca8063d30 0x7f1ca80661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:15.396 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.358+0000 7f1cbcff9700 1 -- 172.21.15.195:0/312151180 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6540+0+0 (secure 0 0 0) 0x7f1cb8080cc0 con 0x7f1cc00c4160 2022-06-27T11:51:15.397 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.358+0000 7f1cc4bff700 1 --2- 172.21.15.195:0/312151180 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1ca8063d30 0x7f1ca80661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:15.397 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.358+0000 7f1cc4bff700 1 --2- 172.21.15.195:0/312151180 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1ca8063d30 0x7f1ca80661e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1cb40076e0 tx=0x7f1cb4007bd0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:51:15.398 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.362+0000 7f1cbcff9700 1 -- 172.21.15.195:0/312151180 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f1cb804f6d0 con 0x7f1cc00c4160 2022-06-27T11:51:15.593 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.584+0000 7f1cc6e63700 1 -- 172.21.15.195:0/312151180 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1cc00722c0 con 0x7f1cc00c4160 2022-06-27T11:51:15.593 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.585+0000 7f1cbcff9700 1 -- 172.21.15.195:0/312151180 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+5367 (secure 0 0 0) 0x7f1cb8052cf0 con 0x7f1cc00c4160 2022-06-27T11:51:15.594 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:51:15.594 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:51:14.886105+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4849},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4849":{"gid":4849,"name":"d","rank":0,"incarnation":15,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6835/3271891211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":3271891211},{"type":"v1","addr":"172.21.15.202:6835","nonce":3271891211}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:51:15.596 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.587+0000 7f1cc6e63700 1 -- 172.21.15.195:0/312151180 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1ca8063d30 msgr2=0x7f1ca80661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:15.596 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.587+0000 7f1cc6e63700 1 --2- 172.21.15.195:0/312151180 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1ca8063d30 0x7f1ca80661e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1cb40076e0 tx=0x7f1cb4007bd0 comp rx=0 tx=0).stop 2022-06-27T11:51:15.596 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.587+0000 7f1cc6e63700 1 -- 172.21.15.195:0/312151180 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1cc00c4160 msgr2=0x7f1cc015b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:15.597 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.587+0000 7f1cc6e63700 1 --2- 172.21.15.195:0/312151180 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1cc00c4160 0x7f1cc015b9e0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f1cb800c810 tx=0x7f1cb8003bd0 comp rx=0 tx=0).stop 2022-06-27T11:51:15.598 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.587+0000 7f1cc6e63700 1 -- 172.21.15.195:0/312151180 shutdown_connections 2022-06-27T11:51:15.598 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.587+0000 7f1cc6e63700 1 --2- 172.21.15.195:0/312151180 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1ca8063d30 0x7f1ca80661e0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:15.598 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.587+0000 7f1cc6e63700 1 --2- 172.21.15.195:0/312151180 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1cc00c4160 0x7f1cc015b9e0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:15.599 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.587+0000 7f1cc6e63700 1 --2- 172.21.15.195:0/312151180 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1cc006f640 0x7f1cc015c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:15.599 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.587+0000 7f1cc6e63700 1 --2- 172.21.15.195:0/312151180 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1cc00c39e0 0x7f1cc015b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:15.599 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.587+0000 7f1cc6e63700 1 -- 172.21.15.195:0/312151180 >> 172.21.15.195:0/312151180 conn(0x7f1cc00c6350 msgr2=0x7f1cc0152060 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:51:15.601 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.587+0000 7f1cc6e63700 1 -- 172.21.15.195:0/312151180 shutdown_connections 2022-06-27T11:51:15.601 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.587+0000 7f1cc6e63700 1 -- 172.21.15.195:0/312151180 wait complete. 2022-06-27T11:51:15.602 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 15 2022-06-27T11:51:15.604 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2022-06-27T11:51:15.605 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2022-06-27T11:51:15.605 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:51:15.738 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.725+0000 7f1224dca700 1 Processor -- start 2022-06-27T11:51:15.739 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.725+0000 7f1224dca700 1 -- start start 2022-06-27T11:51:15.739 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.726+0000 7f1224dca700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f1220150950 0x7f1220156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:15.739 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.726+0000 7f1224dca700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12200c76b0 con 0x7f122006f620 2022-06-27T11:51:15.740 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.726+0000 7f1224dca700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12200c8250 con 0x7f1220150580 2022-06-27T11:51:15.741 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.726+0000 7f1224dca700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1220157430 con 0x7f1220150950 2022-06-27T11:51:15.741 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.726+0000 7f121ed9d700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f122006f620 0x7f122006d6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:49370/0 (socket says 172.21.15.195:49370) 2022-06-27T11:51:15.741 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.726+0000 7f121ed9d700 1 -- 172.21.15.195:0/3575563483 learned_addr learned my addr 172.21.15.195:0/3575563483 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:51:15.742 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.727+0000 7f121dd9b700 1 --2- 172.21.15.195:0/3575563483 >> v2:172.21.15.202:3300/0 conn(0x7f1220150950 0x7f1220156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:15.743 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.727+0000 7f121d59a700 1 -- 172.21.15.195:0/3575563483 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4172785562 0 0) 0x7f12200c76b0 con 0x7f122006f620 2022-06-27T11:51:15.744 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.727+0000 7f121d59a700 1 -- 172.21.15.195:0/3575563483 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1204003430 con 0x7f122006f620 2022-06-27T11:51:15.745 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.727+0000 7f121dd9b700 1 -- 172.21.15.195:0/3575563483 >> v1:172.21.15.202:6789/0 conn(0x7f1220150580 legacy=0x7f12200c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:15.745 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.727+0000 7f121dd9b700 1 -- 172.21.15.195:0/3575563483 >> v1:172.21.15.195:6789/0 conn(0x7f122006f620 legacy=0x7f122006d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:15.746 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.727+0000 7f121dd9b700 1 -- 172.21.15.195:0/3575563483 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12200bad70 con 0x7f1220150950 2022-06-27T11:51:15.746 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.727+0000 7f121dd9b700 1 --2- 172.21.15.195:0/3575563483 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1220150950 0x7f1220156ef0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f12100097a0 tx=0x7f1210009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1734b5012ddc7064 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:51:15.746 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.727+0000 7f121d59a700 1 -- 172.21.15.195:0/3575563483 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1210011630 con 0x7f1220150950 2022-06-27T11:51:15.747 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.727+0000 7f121d59a700 1 -- 172.21.15.195:0/3575563483 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1210011790 con 0x7f1220150950 2022-06-27T11:51:15.747 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.727+0000 7f121d59a700 1 -- 172.21.15.195:0/3575563483 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1210011b10 con 0x7f1220150950 2022-06-27T11:51:15.748 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.728+0000 7f1224dca700 1 -- 172.21.15.195:0/3575563483 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1220150950 msgr2=0x7f1220156ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:15.749 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.728+0000 7f1224dca700 1 --2- 172.21.15.195:0/3575563483 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1220150950 0x7f1220156ef0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f12100097a0 tx=0x7f1210009060 comp rx=0 tx=0).stop 2022-06-27T11:51:15.749 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.728+0000 7f1224dca700 1 -- 172.21.15.195:0/3575563483 shutdown_connections 2022-06-27T11:51:15.750 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.728+0000 7f1224dca700 1 --2- 172.21.15.195:0/3575563483 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1220150950 0x7f1220156ef0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:15.750 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.728+0000 7f1224dca700 1 -- 172.21.15.195:0/3575563483 >> 172.21.15.195:0/3575563483 conn(0x7f1220063af0 msgr2=0x7f1220063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:51:15.751 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.728+0000 7f1224dca700 1 -- 172.21.15.195:0/3575563483 shutdown_connections 2022-06-27T11:51:15.751 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.728+0000 7f1224dca700 1 -- 172.21.15.195:0/3575563483 wait complete. 2022-06-27T11:51:15.752 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.729+0000 7f1224dca700 1 Processor -- start 2022-06-27T11:51:15.753 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.729+0000 7f1224dca700 1 -- start start 2022-06-27T11:51:15.753 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.730+0000 7f1224dca700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f122006f620 0x7f12200bd4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:15.754 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.730+0000 7f1224dca700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1220150580 0x7f12200cb360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:15.754 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.730+0000 7f1224dca700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1220150950 0x7f122014c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:15.755 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.730+0000 7f1224dca700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12200ca220 con 0x7f1220150950 2022-06-27T11:51:15.755 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.730+0000 7f1224dca700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12200bc410 con 0x7f122006f620 2022-06-27T11:51:15.756 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.730+0000 7f1224dca700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1220166320 con 0x7f1220150580 2022-06-27T11:51:15.756 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.730+0000 7f121ed9d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1220150950 0x7f122014c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:15.757 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.730+0000 7f121ed9d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1220150950 0x7f122014c060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:43192/0 (socket says 172.21.15.195:43192) 2022-06-27T11:51:15.757 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.730+0000 7f121ed9d700 1 -- 172.21.15.195:0/3341844560 learned_addr learned my addr 172.21.15.195:0/3341844560 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:51:15.758 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.730+0000 7f121e59c700 1 --2- 172.21.15.195:0/3341844560 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f122006f620 0x7f12200bd4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:15.758 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.730+0000 7f121dd9b700 1 --2- 172.21.15.195:0/3341844560 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1220150580 0x7f12200cb360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:15.758 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.730+0000 7f121ed9d700 1 -- 172.21.15.195:0/3341844560 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1220150580 msgr2=0x7f12200cb360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:15.759 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.730+0000 7f121ed9d700 1 --2- 172.21.15.195:0/3341844560 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1220150580 0x7f12200cb360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:15.759 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.730+0000 7f121ed9d700 1 -- 172.21.15.195:0/3341844560 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f122006f620 msgr2=0x7f12200bd4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:15.760 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.730+0000 7f121ed9d700 1 --2- 172.21.15.195:0/3341844560 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f122006f620 0x7f12200bd4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:15.761 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.730+0000 7f121ed9d700 1 -- 172.21.15.195:0/3341844560 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12100092e0 con 0x7f1220150950 2022-06-27T11:51:15.761 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.730+0000 7f121ed9d700 1 --2- 172.21.15.195:0/3341844560 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1220150950 0x7f122014c060 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f1218001460 tx=0x7f12180015b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:51:15.762 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.730+0000 7f120e7fc700 1 -- 172.21.15.195:0/3341844560 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1218005d70 con 0x7f1220150950 2022-06-27T11:51:15.762 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.730+0000 7f120e7fc700 1 -- 172.21.15.195:0/3341844560 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12180070f0 con 0x7f1220150950 2022-06-27T11:51:15.763 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.730+0000 7f120e7fc700 1 -- 172.21.15.195:0/3341844560 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1218007470 con 0x7f1220150950 2022-06-27T11:51:15.765 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.730+0000 7f1224dca700 1 -- 172.21.15.195:0/3341844560 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12200bc6d0 con 0x7f1220150950 2022-06-27T11:51:15.765 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.730+0000 7f1224dca700 1 -- 172.21.15.195:0/3341844560 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12201668c0 con 0x7f1220150950 2022-06-27T11:51:15.766 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.731+0000 7f1224dca700 1 -- 172.21.15.195:0/3341844560 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12200722c0 con 0x7f1220150950 2022-06-27T11:51:15.767 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.732+0000 7f120e7fc700 1 -- 172.21.15.195:0/3341844560 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f1218010070 con 0x7f1220150950 2022-06-27T11:51:15.767 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.732+0000 7f120e7fc700 1 --2- 172.21.15.195:0/3341844560 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f11fc063d00 0x7f11fc0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:15.768 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.732+0000 7f120e7fc700 1 -- 172.21.15.195:0/3341844560 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6540+0+0 (secure 0 0 0) 0x7f1218080520 con 0x7f1220150950 2022-06-27T11:51:15.768 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.732+0000 7f121e59c700 1 --2- 172.21.15.195:0/3341844560 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f11fc063d00 0x7f11fc0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:15.769 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.733+0000 7f121e59c700 1 --2- 172.21.15.195:0/3341844560 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f11fc063d00 0x7f11fc0661b0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f12140018a0 tx=0x7f1214009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:51:15.769 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.736+0000 7f120e7fc700 1 -- 172.21.15.195:0/3341844560 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f121804efb0 con 0x7f1220150950 2022-06-27T11:51:15.965 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.965+0000 7f1224dca700 1 -- 172.21.15.195:0/3341844560 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1220166f00 con 0x7f1220150950 2022-06-27T11:51:15.965 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.965+0000 7f120e7fc700 1 -- 172.21.15.195:0/3341844560 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+5367 (secure 0 0 0) 0x7f1220166f00 con 0x7f1220150950 2022-06-27T11:51:15.966 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:51:15.967 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:51:14.886105+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4849},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4849":{"gid":4849,"name":"d","rank":0,"incarnation":15,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6835/3271891211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":3271891211},{"type":"v1","addr":"172.21.15.202:6835","nonce":3271891211}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:51:15.969 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.968+0000 7f1224dca700 1 -- 172.21.15.195:0/3341844560 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f11fc063d00 msgr2=0x7f11fc0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:15.970 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.968+0000 7f1224dca700 1 --2- 172.21.15.195:0/3341844560 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f11fc063d00 0x7f11fc0661b0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f12140018a0 tx=0x7f1214009040 comp rx=0 tx=0).stop 2022-06-27T11:51:15.970 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.968+0000 7f1224dca700 1 -- 172.21.15.195:0/3341844560 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1220150950 msgr2=0x7f122014c060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:15.970 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.968+0000 7f1224dca700 1 --2- 172.21.15.195:0/3341844560 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1220150950 0x7f122014c060 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f1218001460 tx=0x7f12180015b0 comp rx=0 tx=0).stop 2022-06-27T11:51:15.970 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.968+0000 7f1224dca700 1 -- 172.21.15.195:0/3341844560 shutdown_connections 2022-06-27T11:51:15.971 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.968+0000 7f1224dca700 1 --2- 172.21.15.195:0/3341844560 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f11fc063d00 0x7f11fc0661b0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:15.971 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.968+0000 7f1224dca700 1 --2- 172.21.15.195:0/3341844560 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1220150950 0x7f122014c060 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:15.971 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.968+0000 7f1224dca700 1 --2- 172.21.15.195:0/3341844560 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f122006f620 0x7f12200bd4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:15.971 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.968+0000 7f1224dca700 1 --2- 172.21.15.195:0/3341844560 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1220150580 0x7f12200cb360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:15.971 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.968+0000 7f1224dca700 1 -- 172.21.15.195:0/3341844560 >> 172.21.15.195:0/3341844560 conn(0x7f1220063af0 msgr2=0x7f12200c5ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:51:15.972 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.968+0000 7f1224dca700 1 -- 172.21.15.195:0/3341844560 shutdown_connections 2022-06-27T11:51:15.972 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:15.968+0000 7f1224dca700 1 -- 172.21.15.195:0/3341844560 wait complete. 2022-06-27T11:51:15.972 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 15 2022-06-27T11:51:15.980 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 15, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-06-27T11:47:52.358095+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 15, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4849": { "addr": "172.21.15.202:6835/3271891211", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6834", "nonce": 3271891211, "type": "v2" }, { "addr": "172.21.15.202:6835", "nonce": 3271891211, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4849, "incarnation": 15, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 37, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-06-27T11:51:14.886105+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4849 } } } ], "standbys": [ { "addr": "172.21.15.202:6837/3305006841", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6836", "nonce": 3305006841, "type": "v2" }, { "addr": "172.21.15.202:6837", "nonce": 3305006841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6835/2060709822", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6834", "nonce": 2060709822, "type": "v2" }, { "addr": "172.21.15.195:6835", "nonce": 2060709822, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6838/1598751652", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6836", "nonce": 1598751652, "type": "v2" }, { "addr": "172.21.15.195:6838", "nonce": 1598751652, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4275, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6839/1313800661", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6837", "nonce": 1313800661, "type": "v2" }, { "addr": "172.21.15.195:6839", "nonce": 1313800661, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4278, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-06-27T11:51:17.981 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:51:18.123 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.115+0000 7fee73249700 1 Processor -- start 2022-06-27T11:51:18.124 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.116+0000 7fee73249700 1 -- start start 2022-06-27T11:51:18.124 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.116+0000 7fee73249700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fee6c0c2bf0 0x7fee6c0c2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:18.125 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.116+0000 7fee73249700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fee6c0c5000 0x7fee6c0c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:18.125 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.116+0000 7fee73249700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee6c0bc490 con 0x7fee6c0c61e0 2022-06-27T11:51:18.125 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.116+0000 7fee73249700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee6c06a0e0 con 0x7fee6c0c2bf0 2022-06-27T11:51:18.126 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.116+0000 7fee73249700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee6c0bc6e0 con 0x7fee6c0c5000 2022-06-27T11:51:18.126 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.117+0000 7fee70fe5700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fee6c0c2bf0 0x7fee6c0c2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:18.127 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.117+0000 7fee70fe5700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fee6c0c2bf0 0x7fee6c0c2fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:43196/0 (socket says 172.21.15.195:43196) 2022-06-27T11:51:18.127 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.117+0000 7fee70fe5700 1 -- 172.21.15.195:0/1747360839 learned_addr learned my addr 172.21.15.195:0/1747360839 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:51:18.127 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.117+0000 7fee717e6700 1 --2- 172.21.15.195:0/1747360839 >> v2:172.21.15.202:3300/0 conn(0x7fee6c0c5000 0x7fee6c0c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:18.128 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.117+0000 7fee70fe5700 1 -- 172.21.15.195:0/1747360839 >> v2:172.21.15.202:3300/0 conn(0x7fee6c0c5000 msgr2=0x7fee6c0c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:51:18.128 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.117+0000 7fee70fe5700 1 --2- 172.21.15.195:0/1747360839 >> v2:172.21.15.202:3300/0 conn(0x7fee6c0c5000 0x7fee6c0c3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:18.128 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.117+0000 7fee70fe5700 1 -- 172.21.15.195:0/1747360839 >> v1:172.21.15.202:6789/0 conn(0x7fee6c0c61e0 legacy=0x7fee6c0c48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:51:18.129 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.117+0000 7fee70fe5700 1 -- 172.21.15.195:0/1747360839 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee6c158220 con 0x7fee6c0c2bf0 2022-06-27T11:51:18.129 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.117+0000 7fee70fe5700 1 --2- 172.21.15.195:0/1747360839 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fee6c0c2bf0 0x7fee6c0c2fc0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fee6c0bc9a0 tx=0x7fee600090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3280a26c6c8afd4 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:51:18.130 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.117+0000 7fee6b7fe700 1 -- 172.21.15.195:0/1747360839 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee60005ad0 con 0x7fee6c0c2bf0 2022-06-27T11:51:18.130 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.117+0000 7fee6b7fe700 1 -- 172.21.15.195:0/1747360839 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee60005c30 con 0x7fee6c0c2bf0 2022-06-27T11:51:18.130 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.117+0000 7fee6b7fe700 1 -- 172.21.15.195:0/1747360839 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee60006030 con 0x7fee6c0c2bf0 2022-06-27T11:51:18.131 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.118+0000 7fee73249700 1 -- 172.21.15.195:0/1747360839 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fee6c0c2bf0 msgr2=0x7fee6c0c2fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:18.131 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.118+0000 7fee73249700 1 --2- 172.21.15.195:0/1747360839 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fee6c0c2bf0 0x7fee6c0c2fc0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fee6c0bc9a0 tx=0x7fee600090f0 comp rx=0 tx=0).stop 2022-06-27T11:51:18.131 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.118+0000 7fee73249700 1 -- 172.21.15.195:0/1747360839 shutdown_connections 2022-06-27T11:51:18.132 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.118+0000 7fee73249700 1 --2- 172.21.15.195:0/1747360839 >> v2:172.21.15.202:3300/0 conn(0x7fee6c0c5000 0x7fee6c0c3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:18.133 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.118+0000 7fee73249700 1 --2- 172.21.15.195:0/1747360839 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fee6c0c2bf0 0x7fee6c0c2fc0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:18.133 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.118+0000 7fee73249700 1 -- 172.21.15.195:0/1747360839 >> 172.21.15.195:0/1747360839 conn(0x7fee6c0c9b00 msgr2=0x7fee6c0c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:51:18.134 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.118+0000 7fee73249700 1 -- 172.21.15.195:0/1747360839 shutdown_connections 2022-06-27T11:51:18.134 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.118+0000 7fee73249700 1 -- 172.21.15.195:0/1747360839 wait complete. 2022-06-27T11:51:18.134 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.119+0000 7fee73249700 1 Processor -- start 2022-06-27T11:51:18.135 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.119+0000 7fee73249700 1 -- start start 2022-06-27T11:51:18.135 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.120+0000 7fee73249700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fee6c0c2bf0 0x7fee6c14f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:18.136 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.120+0000 7fee73249700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fee6c0c5000 0x7fee6c06e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:18.136 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.120+0000 7fee73249700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fee6c0c61e0 0x7fee6c167730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:18.136 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.120+0000 7fee73249700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee6c164c20 con 0x7fee6c0c5000 2022-06-27T11:51:18.137 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.120+0000 7fee73249700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee6c151730 con 0x7fee6c0c2bf0 2022-06-27T11:51:18.137 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.120+0000 7fee73249700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee6c1519b0 con 0x7fee6c0c61e0 2022-06-27T11:51:18.137 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.120+0000 7fee6bfff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fee6c0c5000 0x7fee6c06e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:18.138 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.120+0000 7fee6bfff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fee6c0c5000 0x7fee6c06e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:43204/0 (socket says 172.21.15.195:43204) 2022-06-27T11:51:18.139 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.120+0000 7fee6bfff700 1 -- 172.21.15.195:0/2121656424 learned_addr learned my addr 172.21.15.195:0/2121656424 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:51:18.140 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.120+0000 7fee70fe5700 1 --2- 172.21.15.195:0/2121656424 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fee6c0c2bf0 0x7fee6c14f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:18.141 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.120+0000 7fee717e6700 1 --2- 172.21.15.195:0/2121656424 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fee6c0c61e0 0x7fee6c167730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:18.141 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.120+0000 7fee6bfff700 1 -- 172.21.15.195:0/2121656424 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fee6c0c61e0 msgr2=0x7fee6c167730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:18.141 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.120+0000 7fee6bfff700 1 --2- 172.21.15.195:0/2121656424 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fee6c0c61e0 0x7fee6c167730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:18.142 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.120+0000 7fee6bfff700 1 -- 172.21.15.195:0/2121656424 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fee6c0c2bf0 msgr2=0x7fee6c14f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:18.142 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.120+0000 7fee6bfff700 1 --2- 172.21.15.195:0/2121656424 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fee6c0c2bf0 0x7fee6c14f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:18.143 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.120+0000 7fee6bfff700 1 -- 172.21.15.195:0/2121656424 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee60009350 con 0x7fee6c0c5000 2022-06-27T11:51:18.144 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.120+0000 7fee6bfff700 1 --2- 172.21.15.195:0/2121656424 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fee6c0c5000 0x7fee6c06e8f0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fee5800b470 tx=0x7fee58006f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:51:18.144 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.121+0000 7fee68ff9700 1 -- 172.21.15.195:0/2121656424 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee58011420 con 0x7fee6c0c5000 2022-06-27T11:51:18.145 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.121+0000 7fee68ff9700 1 -- 172.21.15.195:0/2121656424 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee58011580 con 0x7fee6c0c5000 2022-06-27T11:51:18.145 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.121+0000 7fee73249700 1 -- 172.21.15.195:0/2121656424 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee6c154570 con 0x7fee6c0c5000 2022-06-27T11:51:18.146 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.121+0000 7fee68ff9700 1 -- 172.21.15.195:0/2121656424 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee58011940 con 0x7fee6c0c5000 2022-06-27T11:51:18.146 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.121+0000 7fee73249700 1 -- 172.21.15.195:0/2121656424 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee6c150210 con 0x7fee6c0c5000 2022-06-27T11:51:18.148 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.122+0000 7fee73249700 1 -- 172.21.15.195:0/2121656424 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee6c164d90 con 0x7fee6c0c5000 2022-06-27T11:51:18.149 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.122+0000 7fee68ff9700 1 -- 172.21.15.195:0/2121656424 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fee5800f050 con 0x7fee6c0c5000 2022-06-27T11:51:18.149 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.122+0000 7fee68ff9700 1 --2- 172.21.15.195:0/2121656424 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fee54063d30 0x7fee540661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:18.150 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.122+0000 7fee68ff9700 1 -- 172.21.15.195:0/2121656424 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6540+0+0 (secure 0 0 0) 0x7fee580805a0 con 0x7fee6c0c5000 2022-06-27T11:51:18.150 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.122+0000 7fee70fe5700 1 --2- 172.21.15.195:0/2121656424 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fee54063d30 0x7fee540661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:18.150 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.123+0000 7fee70fe5700 1 --2- 172.21.15.195:0/2121656424 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fee54063d30 0x7fee540661e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fee6c0bc9a0 tx=0x7fee60016000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:51:18.151 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.127+0000 7fee68ff9700 1 -- 172.21.15.195:0/2121656424 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fee5804efb0 con 0x7fee6c0c5000 2022-06-27T11:51:18.351 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.352+0000 7fee73249700 1 -- 172.21.15.195:0/2121656424 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fee6c0722c0 con 0x7fee6c0c5000 2022-06-27T11:51:18.352 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.352+0000 7fee68ff9700 1 -- 172.21.15.195:0/2121656424 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+5367 (secure 0 0 0) 0x7fee58015090 con 0x7fee6c0c5000 2022-06-27T11:51:18.353 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:51:18.353 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:51:14.886105+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4849},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4849":{"gid":4849,"name":"d","rank":0,"incarnation":15,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6835/3271891211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":3271891211},{"type":"v1","addr":"172.21.15.202:6835","nonce":3271891211}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:51:18.356 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.354+0000 7fee73249700 1 -- 172.21.15.195:0/2121656424 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fee54063d30 msgr2=0x7fee540661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:18.356 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.354+0000 7fee73249700 1 --2- 172.21.15.195:0/2121656424 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fee54063d30 0x7fee540661e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fee6c0bc9a0 tx=0x7fee60016000 comp rx=0 tx=0).stop 2022-06-27T11:51:18.357 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.354+0000 7fee73249700 1 -- 172.21.15.195:0/2121656424 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fee6c0c5000 msgr2=0x7fee6c06e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:18.357 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.354+0000 7fee73249700 1 --2- 172.21.15.195:0/2121656424 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fee6c0c5000 0x7fee6c06e8f0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fee5800b470 tx=0x7fee58006f60 comp rx=0 tx=0).stop 2022-06-27T11:51:18.358 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.354+0000 7fee73249700 1 -- 172.21.15.195:0/2121656424 shutdown_connections 2022-06-27T11:51:18.358 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.354+0000 7fee73249700 1 --2- 172.21.15.195:0/2121656424 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fee54063d30 0x7fee540661e0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:18.359 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.354+0000 7fee73249700 1 --2- 172.21.15.195:0/2121656424 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fee6c0c61e0 0x7fee6c167730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:18.359 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.354+0000 7fee73249700 1 --2- 172.21.15.195:0/2121656424 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fee6c0c2bf0 0x7fee6c14f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:18.359 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.354+0000 7fee73249700 1 --2- 172.21.15.195:0/2121656424 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fee6c0c5000 0x7fee6c06e8f0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:18.360 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.354+0000 7fee73249700 1 -- 172.21.15.195:0/2121656424 >> 172.21.15.195:0/2121656424 conn(0x7fee6c0c9b00 msgr2=0x7fee6c14dd40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:51:18.360 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.354+0000 7fee73249700 1 -- 172.21.15.195:0/2121656424 shutdown_connections 2022-06-27T11:51:18.360 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:18.355+0000 7fee73249700 1 -- 172.21.15.195:0/2121656424 wait complete. 2022-06-27T11:51:18.361 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 15 2022-06-27T11:51:18.375 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T11:51:20.378 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:51:20.523 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.511+0000 7f4f3e7d0700 1 Processor -- start 2022-06-27T11:51:20.524 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.511+0000 7f4f3e7d0700 1 -- start start 2022-06-27T11:51:20.524 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.512+0000 7f4f3e7d0700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f4f38153cb0 0x7f4f38156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:20.525 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.512+0000 7f4f3e7d0700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f380c1a20 con 0x7f4f3806f640 2022-06-27T11:51:20.525 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.512+0000 7f4f3e7d0700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f380c1130 con 0x7f4f380c4160 2022-06-27T11:51:20.526 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.512+0000 7f4f3e7d0700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f3806ea90 con 0x7f4f38153cb0 2022-06-27T11:51:20.526 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.512+0000 7f4f37fff700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f4f3806f640 0x7f4f380c1eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:49400/0 (socket says 172.21.15.195:49400) 2022-06-27T11:51:20.526 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.512+0000 7f4f37fff700 1 -- 172.21.15.195:0/3690190561 learned_addr learned my addr 172.21.15.195:0/3690190561 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:51:20.527 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.512+0000 7f4f3cd6d700 1 --2- 172.21.15.195:0/3690190561 >> v2:172.21.15.195:3300/0 conn(0x7f4f38153cb0 0x7f4f38156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:20.527 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.512+0000 7f4f36ffd700 1 -- 172.21.15.195:0/3690190561 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3999708579 0 0) 0x7f4f380c1a20 con 0x7f4f3806f640 2022-06-27T11:51:20.528 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.512+0000 7f4f36ffd700 1 -- 172.21.15.195:0/3690190561 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f24003430 con 0x7f4f3806f640 2022-06-27T11:51:20.528 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.513+0000 7f4f3cd6d700 1 -- 172.21.15.195:0/3690190561 >> v1:172.21.15.202:6789/0 conn(0x7f4f380c4160 legacy=0x7f4f381535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:20.529 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.513+0000 7f4f36ffd700 1 -- 172.21.15.195:0/3690190561 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 302266229 0 0) 0x7f4f380c1130 con 0x7f4f380c4160 2022-06-27T11:51:20.529 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.513+0000 7f4f3cd6d700 1 -- 172.21.15.195:0/3690190561 >> v1:172.21.15.195:6789/0 conn(0x7f4f3806f640 legacy=0x7f4f380c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:20.529 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.513+0000 7f4f3cd6d700 1 -- 172.21.15.195:0/3690190561 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f3806ed70 con 0x7f4f38153cb0 2022-06-27T11:51:20.530 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.513+0000 7f4f3cd6d700 1 --2- 172.21.15.195:0/3690190561 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4f38153cb0 0x7f4f38156ed0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f4f30009e60 tx=0x7f4f30009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=867bc5b1b5f34f4c server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:51:20.530 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.513+0000 7f4f36ffd700 1 -- 172.21.15.195:0/3690190561 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f3000e070 con 0x7f4f38153cb0 2022-06-27T11:51:20.531 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.513+0000 7f4f36ffd700 1 -- 172.21.15.195:0/3690190561 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f3000bbf0 con 0x7f4f38153cb0 2022-06-27T11:51:20.531 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.513+0000 7f4f36ffd700 1 -- 172.21.15.195:0/3690190561 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f30012430 con 0x7f4f38153cb0 2022-06-27T11:51:20.532 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.513+0000 7f4f3e7d0700 1 -- 172.21.15.195:0/3690190561 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4f38153cb0 msgr2=0x7f4f38156ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:20.532 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.513+0000 7f4f3e7d0700 1 --2- 172.21.15.195:0/3690190561 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4f38153cb0 0x7f4f38156ed0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f4f30009e60 tx=0x7f4f30009780 comp rx=0 tx=0).stop 2022-06-27T11:51:20.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.513+0000 7f4f3e7d0700 1 -- 172.21.15.195:0/3690190561 shutdown_connections 2022-06-27T11:51:20.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.513+0000 7f4f3e7d0700 1 --2- 172.21.15.195:0/3690190561 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4f38153cb0 0x7f4f38156ed0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:20.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.513+0000 7f4f3e7d0700 1 -- 172.21.15.195:0/3690190561 >> 172.21.15.195:0/3690190561 conn(0x7f4f380c6350 msgr2=0x7f4f380c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:51:20.534 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.513+0000 7f4f3e7d0700 1 -- 172.21.15.195:0/3690190561 shutdown_connections 2022-06-27T11:51:20.534 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.513+0000 7f4f3e7d0700 1 -- 172.21.15.195:0/3690190561 wait complete. 2022-06-27T11:51:20.535 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.514+0000 7f4f3e7d0700 1 Processor -- start 2022-06-27T11:51:20.535 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.515+0000 7f4f3e7d0700 1 -- start start 2022-06-27T11:51:20.535 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.515+0000 7f4f3e7d0700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4f3806f640 0x7f4f38155cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:20.536 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.515+0000 7f4f3e7d0700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4f380c4160 0x7f4f3814f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:20.536 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.515+0000 7f4f3e7d0700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4f38153cb0 0x7f4f3814fec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:20.536 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.515+0000 7f4f3e7d0700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f38154f90 con 0x7f4f3806f640 2022-06-27T11:51:20.537 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.515+0000 7f4f3e7d0700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f38155830 con 0x7f4f380c4160 2022-06-27T11:51:20.537 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.515+0000 7f4f3e7d0700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f38154680 con 0x7f4f38153cb0 2022-06-27T11:51:20.538 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.516+0000 7f4f37fff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4f3806f640 0x7f4f38155cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:20.538 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.516+0000 7f4f37fff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4f3806f640 0x7f4f38155cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:43216/0 (socket says 172.21.15.195:43216) 2022-06-27T11:51:20.539 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.516+0000 7f4f37fff700 1 -- 172.21.15.195:0/3256096665 learned_addr learned my addr 172.21.15.195:0/3256096665 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:51:20.539 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.516+0000 7f4f377fe700 1 --2- 172.21.15.195:0/3256096665 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4f380c4160 0x7f4f3814f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:20.539 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.516+0000 7f4f3cd6d700 1 --2- 172.21.15.195:0/3256096665 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4f38153cb0 0x7f4f3814fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:20.540 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.516+0000 7f4f37fff700 1 -- 172.21.15.195:0/3256096665 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4f38153cb0 msgr2=0x7f4f3814fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:20.540 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.516+0000 7f4f37fff700 1 --2- 172.21.15.195:0/3256096665 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4f38153cb0 0x7f4f3814fec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:20.541 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.516+0000 7f4f37fff700 1 -- 172.21.15.195:0/3256096665 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4f380c4160 msgr2=0x7f4f3814f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:20.541 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.516+0000 7f4f37fff700 1 --2- 172.21.15.195:0/3256096665 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4f380c4160 0x7f4f3814f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:20.542 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.516+0000 7f4f37fff700 1 -- 172.21.15.195:0/3256096665 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f30009a20 con 0x7f4f3806f640 2022-06-27T11:51:20.542 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.516+0000 7f4f37fff700 1 --2- 172.21.15.195:0/3256096665 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4f3806f640 0x7f4f38155cc0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f4f2800c4e0 tx=0x7f4f28007c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:51:20.543 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.516+0000 7f4f1ffff700 1 -- 172.21.15.195:0/3256096665 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f2800ce50 con 0x7f4f3806f640 2022-06-27T11:51:20.543 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.516+0000 7f4f1ffff700 1 -- 172.21.15.195:0/3256096665 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f280072e0 con 0x7f4f3806f640 2022-06-27T11:51:20.544 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.516+0000 7f4f3e7d0700 1 -- 172.21.15.195:0/3256096665 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f38154960 con 0x7f4f3806f640 2022-06-27T11:51:20.546 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.516+0000 7f4f1ffff700 1 -- 172.21.15.195:0/3256096665 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f280076a0 con 0x7f4f3806f640 2022-06-27T11:51:20.546 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.516+0000 7f4f3e7d0700 1 -- 172.21.15.195:0/3256096665 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f381554c0 con 0x7f4f3806f640 2022-06-27T11:51:20.547 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.517+0000 7f4f3e7d0700 1 -- 172.21.15.195:0/3256096665 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f38155b10 con 0x7f4f3806f640 2022-06-27T11:51:20.547 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.518+0000 7f4f1ffff700 1 -- 172.21.15.195:0/3256096665 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f4f28019030 con 0x7f4f3806f640 2022-06-27T11:51:20.548 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.518+0000 7f4f1ffff700 1 --2- 172.21.15.195:0/3256096665 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4f18063d00 0x7f4f180661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:20.549 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.518+0000 7f4f1ffff700 1 -- 172.21.15.195:0/3256096665 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6540+0+0 (secure 0 0 0) 0x7f4f2807fbf0 con 0x7f4f3806f640 2022-06-27T11:51:20.550 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.518+0000 7f4f377fe700 1 --2- 172.21.15.195:0/3256096665 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4f18063d00 0x7f4f180661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:20.551 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.518+0000 7f4f377fe700 1 --2- 172.21.15.195:0/3256096665 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4f18063d00 0x7f4f180661b0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4f38150f90 tx=0x7f4f2c009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:51:20.551 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.522+0000 7f4f1ffff700 1 -- 172.21.15.195:0/3256096665 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f4f2804e600 con 0x7f4f3806f640 2022-06-27T11:51:20.752 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.753+0000 7f4f3e7d0700 1 -- 172.21.15.195:0/3256096665 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4f380722c0 con 0x7f4f3806f640 2022-06-27T11:51:20.753 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.753+0000 7f4f1ffff700 1 -- 172.21.15.195:0/3256096665 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+5367 (secure 0 0 0) 0x7f4f28017030 con 0x7f4f3806f640 2022-06-27T11:51:20.754 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:51:20.755 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:51:14.886105+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4849},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4849":{"gid":4849,"name":"d","rank":0,"incarnation":15,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6835/3271891211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":3271891211},{"type":"v1","addr":"172.21.15.202:6835","nonce":3271891211}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:51:20.758 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.755+0000 7f4f3e7d0700 1 -- 172.21.15.195:0/3256096665 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4f18063d00 msgr2=0x7f4f180661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:20.758 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.755+0000 7f4f3e7d0700 1 --2- 172.21.15.195:0/3256096665 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4f18063d00 0x7f4f180661b0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4f38150f90 tx=0x7f4f2c009040 comp rx=0 tx=0).stop 2022-06-27T11:51:20.759 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.755+0000 7f4f3e7d0700 1 -- 172.21.15.195:0/3256096665 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4f3806f640 msgr2=0x7f4f38155cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:20.759 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.755+0000 7f4f3e7d0700 1 --2- 172.21.15.195:0/3256096665 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4f3806f640 0x7f4f38155cc0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f4f2800c4e0 tx=0x7f4f28007c70 comp rx=0 tx=0).stop 2022-06-27T11:51:20.759 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.755+0000 7f4f3e7d0700 1 -- 172.21.15.195:0/3256096665 shutdown_connections 2022-06-27T11:51:20.760 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.755+0000 7f4f3e7d0700 1 --2- 172.21.15.195:0/3256096665 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4f18063d00 0x7f4f180661b0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:20.760 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.755+0000 7f4f3e7d0700 1 --2- 172.21.15.195:0/3256096665 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4f3806f640 0x7f4f38155cc0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:20.760 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.755+0000 7f4f3e7d0700 1 --2- 172.21.15.195:0/3256096665 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4f38153cb0 0x7f4f3814fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:20.761 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.755+0000 7f4f3e7d0700 1 --2- 172.21.15.195:0/3256096665 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4f380c4160 0x7f4f3814f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:20.762 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.755+0000 7f4f3e7d0700 1 -- 172.21.15.195:0/3256096665 >> 172.21.15.195:0/3256096665 conn(0x7f4f380c6350 msgr2=0x7f4f38156860 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:51:20.762 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.756+0000 7f4f3e7d0700 1 -- 172.21.15.195:0/3256096665 shutdown_connections 2022-06-27T11:51:20.763 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:20.756+0000 7f4f3e7d0700 1 -- 172.21.15.195:0/3256096665 wait complete. 2022-06-27T11:51:20.764 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 15 2022-06-27T11:51:20.769 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T11:51:22.771 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:51:22.925 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.908+0000 7fa9f0c2a700 1 Processor -- start 2022-06-27T11:51:22.926 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.908+0000 7fa9f0c2a700 1 -- start start 2022-06-27T11:51:22.926 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.909+0000 7fa9f0c2a700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa9ec068f80 0x7fa9ec0c4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:22.927 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.909+0000 7fa9f0c2a700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fa9ec0c7480 0x7fa9ec0683c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:22.927 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.909+0000 7fa9f0c2a700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9ec0697f0 con 0x7fa9ec069350 2022-06-27T11:51:22.928 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.909+0000 7fa9f0c2a700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9ec1573b0 con 0x7fa9ec0c7480 2022-06-27T11:51:22.928 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.909+0000 7fa9f0c2a700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9ec0c46c0 con 0x7fa9ec068f80 2022-06-27T11:51:22.930 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.909+0000 7fa9ead9d700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fa9ec069350 0x7fa9ec156460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:49418/0 (socket says 172.21.15.195:49418) 2022-06-27T11:51:22.930 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.909+0000 7fa9ead9d700 1 -- 172.21.15.195:0/1392820548 learned_addr learned my addr 172.21.15.195:0/1392820548 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:51:22.931 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.909+0000 7fa9e9d9b700 1 --2- 172.21.15.195:0/1392820548 >> v2:172.21.15.202:3300/0 conn(0x7fa9ec0c7480 0x7fa9ec0683c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:22.931 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.909+0000 7fa9e959a700 1 -- 172.21.15.195:0/1392820548 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 714411385 0 0) 0x7fa9ec0697f0 con 0x7fa9ec069350 2022-06-27T11:51:22.932 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.909+0000 7fa9ea59c700 1 --2- 172.21.15.195:0/1392820548 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa9ec068f80 0x7fa9ec0c4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:22.932 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.909+0000 7fa9e959a700 1 -- 172.21.15.195:0/1392820548 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9d0003430 con 0x7fa9ec069350 2022-06-27T11:51:22.933 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.909+0000 7fa9e959a700 1 -- 172.21.15.195:0/1392820548 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1076149177 0 0) 0x7fa9d0003430 con 0x7fa9ec069350 2022-06-27T11:51:22.933 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.909+0000 7fa9e959a700 1 -- 172.21.15.195:0/1392820548 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa9ec068f80 msgr2=0x7fa9ec0c4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:51:22.934 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.909+0000 7fa9e959a700 1 --2- 172.21.15.195:0/1392820548 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa9ec068f80 0x7fa9ec0c4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:22.934 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.909+0000 7fa9e959a700 1 -- 172.21.15.195:0/1392820548 >> v2:172.21.15.202:3300/0 conn(0x7fa9ec0c7480 msgr2=0x7fa9ec0683c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:51:22.934 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.909+0000 7fa9e959a700 1 --2- 172.21.15.195:0/1392820548 >> v2:172.21.15.202:3300/0 conn(0x7fa9ec0c7480 0x7fa9ec0683c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:22.935 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.909+0000 7fa9e959a700 1 -- 172.21.15.195:0/1392820548 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9ec156b70 con 0x7fa9ec069350 2022-06-27T11:51:22.935 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.909+0000 7fa9e959a700 1 -- 172.21.15.195:0/1392820548 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7fa9e40017f0 con 0x7fa9ec069350 2022-06-27T11:51:22.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.909+0000 7fa9ea59c700 1 --2- 172.21.15.195:0/1392820548 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa9ec068f80 0x7fa9ec0c4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T11:51:22.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.909+0000 7fa9e959a700 1 -- 172.21.15.195:0/1392820548 >> v1:172.21.15.195:6789/0 conn(0x7fa9ec069350 legacy=0x7fa9ec156460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:22.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.909+0000 7fa9e959a700 1 --2- 172.21.15.195:0/1392820548 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa9ec068f80 0x7fa9d0003920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:22.937 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.909+0000 7fa9e959a700 1 --2- 172.21.15.195:0/1392820548 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa9d0003e60 0x7fa9d00046d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:22.937 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.909+0000 7fa9e959a700 1 --2- 172.21.15.195:0/1392820548 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa9d0004c10 0x7fa9d0005080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:22.948 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.909+0000 7fa9e959a700 1 -- 172.21.15.195:0/1392820548 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9ec1573b0 con 0x7fa9d0003e60 2022-06-27T11:51:22.948 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.910+0000 7fa9e959a700 1 -- 172.21.15.195:0/1392820548 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9ec0c46c0 con 0x7fa9d0004c10 2022-06-27T11:51:22.949 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.910+0000 7fa9e959a700 1 -- 172.21.15.195:0/1392820548 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d000b9d0 con 0x7fa9ec068f80 2022-06-27T11:51:22.949 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.910+0000 7fa9e9d9b700 1 --2- 172.21.15.195:0/1392820548 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa9d0003e60 0x7fa9d00046d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:22.950 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.910+0000 7fa9ea59c700 1 --2- 172.21.15.195:0/1392820548 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa9ec068f80 0x7fa9d0003920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:22.950 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.910+0000 7fa9ead9d700 1 --2- 172.21.15.195:0/1392820548 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa9d0004c10 0x7fa9d0005080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:22.951 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.910+0000 7fa9e9d9b700 1 -- 172.21.15.195:0/1392820548 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa9ec068f80 msgr2=0x7fa9d0003920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:51:22.951 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.910+0000 7fa9e9d9b700 1 --2- 172.21.15.195:0/1392820548 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa9ec068f80 0x7fa9d0003920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:22.961 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.910+0000 7fa9e9d9b700 1 -- 172.21.15.195:0/1392820548 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa9d0004c10 msgr2=0x7fa9d0005080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:51:22.962 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.910+0000 7fa9e9d9b700 1 --2- 172.21.15.195:0/1392820548 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa9d0004c10 0x7fa9d0005080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:22.962 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.910+0000 7fa9e9d9b700 1 -- 172.21.15.195:0/1392820548 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9d0003430 con 0x7fa9d0003e60 2022-06-27T11:51:22.963 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.910+0000 7fa9e9d9b700 1 --2- 172.21.15.195:0/1392820548 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa9d0003e60 0x7fa9d00046d0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fa9e0006dc0 tx=0x7fa9e0009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1411924acaf100eb server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:51:22.963 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.910+0000 7fa9e959a700 1 -- 172.21.15.195:0/1392820548 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e0009470 con 0x7fa9d0003e60 2022-06-27T11:51:22.964 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.910+0000 7fa9e959a700 1 -- 172.21.15.195:0/1392820548 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9e000e610 con 0x7fa9d0003e60 2022-06-27T11:51:22.964 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.910+0000 7fa9e959a700 1 -- 172.21.15.195:0/1392820548 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e000e990 con 0x7fa9d0003e60 2022-06-27T11:51:22.965 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.911+0000 7fa9f0c2a700 1 -- 172.21.15.195:0/1392820548 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa9d0003e60 msgr2=0x7fa9d00046d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:22.969 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.911+0000 7fa9f0c2a700 1 --2- 172.21.15.195:0/1392820548 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa9d0003e60 0x7fa9d00046d0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fa9e0006dc0 tx=0x7fa9e0009060 comp rx=0 tx=0).stop 2022-06-27T11:51:22.970 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.911+0000 7fa9ead9d700 1 -- 172.21.15.195:0/1392820548 reap_dead start 2022-06-27T11:51:22.970 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.911+0000 7fa9f0c2a700 1 -- 172.21.15.195:0/1392820548 shutdown_connections 2022-06-27T11:51:22.970 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.911+0000 7fa9f0c2a700 1 -- 172.21.15.195:0/1392820548 >> 172.21.15.195:0/1392820548 conn(0x7fa9ec0ccbd0 msgr2=0x7fa9ec0ccfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:51:22.970 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.911+0000 7fa9f0c2a700 1 -- 172.21.15.195:0/1392820548 shutdown_connections 2022-06-27T11:51:22.971 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.911+0000 7fa9f0c2a700 1 -- 172.21.15.195:0/1392820548 wait complete. 2022-06-27T11:51:22.971 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.912+0000 7fa9f0c2a700 1 Processor -- start 2022-06-27T11:51:22.971 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.912+0000 7fa9f0c2a700 1 -- start start 2022-06-27T11:51:22.971 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.913+0000 7fa9f0c2a700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa9d0003e60 0x7fa9ec16a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:22.972 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.913+0000 7fa9f0c2a700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa9ec15a590 0x7fa9ec153200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:22.972 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.913+0000 7fa9f0c2a700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa9ec15a9e0 0x7fa9ec153740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:22.972 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.913+0000 7fa9f0c2a700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9ec1583c0 con 0x7fa9d0003e60 2022-06-27T11:51:22.973 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.913+0000 7fa9f0c2a700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9ec153c80 con 0x7fa9ec15a590 2022-06-27T11:51:22.976 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.913+0000 7fa9f0c2a700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9ec159830 con 0x7fa9ec15a9e0 2022-06-27T11:51:22.977 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.913+0000 7fa9ea59c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa9d0003e60 0x7fa9ec16a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:22.978 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.913+0000 7fa9ea59c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa9d0003e60 0x7fa9ec16a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:43238/0 (socket says 172.21.15.195:43238) 2022-06-27T11:51:22.978 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.913+0000 7fa9ea59c700 1 -- 172.21.15.195:0/1282741187 learned_addr learned my addr 172.21.15.195:0/1282741187 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:51:22.979 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.913+0000 7fa9e9d9b700 1 --2- 172.21.15.195:0/1282741187 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa9ec15a590 0x7fa9ec153200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:22.980 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.913+0000 7fa9ead9d700 1 --2- 172.21.15.195:0/1282741187 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa9ec15a9e0 0x7fa9ec153740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:22.980 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.913+0000 7fa9ea59c700 1 -- 172.21.15.195:0/1282741187 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa9ec15a9e0 msgr2=0x7fa9ec153740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:22.981 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.913+0000 7fa9ea59c700 1 --2- 172.21.15.195:0/1282741187 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa9ec15a9e0 0x7fa9ec153740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:22.981 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.913+0000 7fa9ea59c700 1 -- 172.21.15.195:0/1282741187 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa9ec15a590 msgr2=0x7fa9ec153200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:22.982 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.913+0000 7fa9ea59c700 1 --2- 172.21.15.195:0/1282741187 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa9ec15a590 0x7fa9ec153200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:22.982 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.913+0000 7fa9ea59c700 1 -- 172.21.15.195:0/1282741187 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9ec16a570 con 0x7fa9d0003e60 2022-06-27T11:51:22.983 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.913+0000 7fa9ea59c700 1 --2- 172.21.15.195:0/1282741187 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa9d0003e60 0x7fa9ec16a030 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc0109b0 tx=0x7fa9dc008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:51:22.983 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.913+0000 7fa9da7fc700 1 -- 172.21.15.195:0/1282741187 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9dc004020 con 0x7fa9d0003e60 2022-06-27T11:51:22.984 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.913+0000 7fa9da7fc700 1 -- 172.21.15.195:0/1282741187 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9dc004180 con 0x7fa9d0003e60 2022-06-27T11:51:22.985 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.913+0000 7fa9f0c2a700 1 -- 172.21.15.195:0/1282741187 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9ec157e70 con 0x7fa9d0003e60 2022-06-27T11:51:22.985 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.913+0000 7fa9da7fc700 1 -- 172.21.15.195:0/1282741187 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9dc016690 con 0x7fa9d0003e60 2022-06-27T11:51:22.986 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.913+0000 7fa9f0c2a700 1 -- 172.21.15.195:0/1282741187 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9ec0b2890 con 0x7fa9d0003e60 2022-06-27T11:51:22.986 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.914+0000 7fa9f0c2a700 1 -- 172.21.15.195:0/1282741187 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9ec0722c0 con 0x7fa9d0003e60 2022-06-27T11:51:22.987 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.915+0000 7fa9da7fc700 1 -- 172.21.15.195:0/1282741187 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fa9dc016a80 con 0x7fa9d0003e60 2022-06-27T11:51:22.987 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.915+0000 7fa9da7fc700 1 --2- 172.21.15.195:0/1282741187 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa9c8063d00 0x7fa9c80661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:22.988 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.915+0000 7fa9da7fc700 1 -- 172.21.15.195:0/1282741187 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6540+0+0 (secure 0 0 0) 0x7fa9dc0810e0 con 0x7fa9d0003e60 2022-06-27T11:51:22.988 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.915+0000 7fa9e9d9b700 1 --2- 172.21.15.195:0/1282741187 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa9c8063d00 0x7fa9c80661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:22.989 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.916+0000 7fa9e9d9b700 1 --2- 172.21.15.195:0/1282741187 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa9c8063d00 0x7fa9c80661b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa9ec0cc6c0 tx=0x7fa9e001a000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:51:22.990 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:22.919+0000 7fa9da7fc700 1 -- 172.21.15.195:0/1282741187 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fa9dc04fba0 con 0x7fa9d0003e60 2022-06-27T11:51:23.139 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:23.140+0000 7fa9f0c2a700 1 -- 172.21.15.195:0/1282741187 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa9ec0652c0 con 0x7fa9d0003e60 2022-06-27T11:51:23.140 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:23.140+0000 7fa9da7fc700 1 -- 172.21.15.195:0/1282741187 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+5371 (secure 0 0 0) 0x7fa9dc01a090 con 0x7fa9d0003e60 2022-06-27T11:51:23.140 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:51:23.140 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:51:21.983241+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4849},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4849":{"gid":4849,"name":"d","rank":0,"incarnation":15,"state":"up:reconnect","state_seq":15,"addr":"172.21.15.202:6835/3271891211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":3271891211},{"type":"v1","addr":"172.21.15.202:6835","nonce":3271891211}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:51:23.142 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:23.142+0000 7fa9f0c2a700 1 -- 172.21.15.195:0/1282741187 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa9c8063d00 msgr2=0x7fa9c80661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:23.142 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:23.142+0000 7fa9f0c2a700 1 --2- 172.21.15.195:0/1282741187 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa9c8063d00 0x7fa9c80661b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa9ec0cc6c0 tx=0x7fa9e001a000 comp rx=0 tx=0).stop 2022-06-27T11:51:23.143 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:23.142+0000 7fa9f0c2a700 1 -- 172.21.15.195:0/1282741187 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa9d0003e60 msgr2=0x7fa9ec16a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:23.143 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:23.142+0000 7fa9f0c2a700 1 --2- 172.21.15.195:0/1282741187 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa9d0003e60 0x7fa9ec16a030 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc0109b0 tx=0x7fa9dc008040 comp rx=0 tx=0).stop 2022-06-27T11:51:23.143 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:23.142+0000 7fa9f0c2a700 1 -- 172.21.15.195:0/1282741187 shutdown_connections 2022-06-27T11:51:23.143 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:23.142+0000 7fa9f0c2a700 1 --2- 172.21.15.195:0/1282741187 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa9c8063d00 0x7fa9c80661b0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:23.144 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:23.142+0000 7fa9f0c2a700 1 --2- 172.21.15.195:0/1282741187 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa9d0003e60 0x7fa9ec16a030 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:23.144 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:23.142+0000 7fa9f0c2a700 1 --2- 172.21.15.195:0/1282741187 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa9ec15a9e0 0x7fa9ec153740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:23.144 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:23.142+0000 7fa9f0c2a700 1 --2- 172.21.15.195:0/1282741187 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa9ec15a590 0x7fa9ec153200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:23.144 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:23.142+0000 7fa9f0c2a700 1 -- 172.21.15.195:0/1282741187 >> 172.21.15.195:0/1282741187 conn(0x7fa9ec0ccbd0 msgr2=0x7fa9ec1501b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:51:23.145 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:23.142+0000 7fa9f0c2a700 1 -- 172.21.15.195:0/1282741187 shutdown_connections 2022-06-27T11:51:23.145 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:23.142+0000 7fa9f0c2a700 1 -- 172.21.15.195:0/1282741187 wait complete. 2022-06-27T11:51:23.145 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 16 2022-06-27T11:51:23.154 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T11:51:25.155 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:51:25.304 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.289+0000 7fb9ffbf7700 1 Processor -- start 2022-06-27T11:51:25.304 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.290+0000 7fb9ffbf7700 1 -- start start 2022-06-27T11:51:25.306 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.290+0000 7fb9ffbf7700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb9f814e490 0x7fb9f806cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:25.307 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.290+0000 7fb9ffbf7700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fb9f814e860 0x7fb9f80c3840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:25.308 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.290+0000 7fb9ffbf7700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9f80c33b0 con 0x7fb9f80c3d80 2022-06-27T11:51:25.308 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.290+0000 7fb9ffbf7700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9f806d520 con 0x7fb9f814e860 2022-06-27T11:51:25.309 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.290+0000 7fb9ffbf7700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f806f220 con 0x7fb9f814e490 2022-06-27T11:51:25.309 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.291+0000 7fb9fd192700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fb9f814e860 0x7fb9f80c3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:25.310 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.291+0000 7fb9fe194700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fb9f80c3d80 0x7fb9f806f620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:49434/0 (socket says 172.21.15.195:49434) 2022-06-27T11:51:25.311 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.291+0000 7fb9fd192700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fb9f814e860 0x7fb9f80c3840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:43246/0 (socket says 172.21.15.195:43246) 2022-06-27T11:51:25.311 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.291+0000 7fb9fe194700 1 -- 172.21.15.195:0/515245268 learned_addr learned my addr 172.21.15.195:0/515245268 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:51:25.312 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.291+0000 7fb9fd993700 1 --2- 172.21.15.195:0/515245268 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb9f814e490 0x7fb9f806cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:25.312 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.291+0000 7fb9fd192700 1 -- 172.21.15.195:0/515245268 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb9f814e490 msgr2=0x7fb9f806cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:51:25.313 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.291+0000 7fb9fd192700 1 --2- 172.21.15.195:0/515245268 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb9f814e490 0x7fb9f806cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:25.313 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.291+0000 7fb9fd192700 1 -- 172.21.15.195:0/515245268 >> v1:172.21.15.195:6789/0 conn(0x7fb9f80c3d80 legacy=0x7fb9f806f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:25.314 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.291+0000 7fb9fd192700 1 -- 172.21.15.195:0/515245268 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9f81542c0 con 0x7fb9f814e860 2022-06-27T11:51:25.314 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.291+0000 7fb9fc991700 1 -- 172.21.15.195:0/515245268 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2567370361 0 0) 0x7fb9f80c33b0 con 0x7fb9f80c3d80 2022-06-27T11:51:25.315 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.291+0000 7fb9fd192700 1 --2- 172.21.15.195:0/515245268 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb9f814e860 0x7fb9f80c3840 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fb9f40097a0 tx=0x7fb9f4009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4bde2b169e828fd server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:51:25.315 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.291+0000 7fb9fc991700 1 -- 172.21.15.195:0/515245268 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9f4011630 con 0x7fb9f814e860 2022-06-27T11:51:25.316 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.291+0000 7fb9fc991700 1 -- 172.21.15.195:0/515245268 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9f4011790 con 0x7fb9f814e860 2022-06-27T11:51:25.317 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.291+0000 7fb9fc991700 1 -- 172.21.15.195:0/515245268 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9f4011b30 con 0x7fb9f814e860 2022-06-27T11:51:25.318 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.292+0000 7fb9ffbf7700 1 -- 172.21.15.195:0/515245268 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb9f814e860 msgr2=0x7fb9f80c3840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:25.318 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.292+0000 7fb9ffbf7700 1 --2- 172.21.15.195:0/515245268 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb9f814e860 0x7fb9f80c3840 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fb9f40097a0 tx=0x7fb9f4009060 comp rx=0 tx=0).stop 2022-06-27T11:51:25.318 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.292+0000 7fb9ffbf7700 1 -- 172.21.15.195:0/515245268 shutdown_connections 2022-06-27T11:51:25.318 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.292+0000 7fb9ffbf7700 1 --2- 172.21.15.195:0/515245268 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb9f814e490 0x7fb9f806cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:25.319 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.292+0000 7fb9ffbf7700 1 --2- 172.21.15.195:0/515245268 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb9f814e860 0x7fb9f80c3840 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:25.319 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.292+0000 7fb9ffbf7700 1 -- 172.21.15.195:0/515245268 >> 172.21.15.195:0/515245268 conn(0x7fb9f8063af0 msgr2=0x7fb9f8063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:51:25.319 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.292+0000 7fb9ffbf7700 1 -- 172.21.15.195:0/515245268 shutdown_connections 2022-06-27T11:51:25.319 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.292+0000 7fb9ffbf7700 1 -- 172.21.15.195:0/515245268 wait complete. 2022-06-27T11:51:25.320 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.293+0000 7fb9ffbf7700 1 Processor -- start 2022-06-27T11:51:25.320 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.293+0000 7fb9ffbf7700 1 -- start start 2022-06-27T11:51:25.320 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.294+0000 7fb9ffbf7700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb9f80c3d80 0x7fb9f8159cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:25.320 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.294+0000 7fb9ffbf7700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb9f814e490 0x7fb9f80c63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:25.320 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.294+0000 7fb9ffbf7700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb9f814e860 0x7fb9f80c6910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:25.321 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.294+0000 7fb9ffbf7700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f806a7e0 con 0x7fb9f80c3d80 2022-06-27T11:51:25.321 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.294+0000 7fb9ffbf7700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f80c7a70 con 0x7fb9f814e860 2022-06-27T11:51:25.321 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.294+0000 7fb9ffbf7700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f8069b60 con 0x7fb9f814e490 2022-06-27T11:51:25.321 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.294+0000 7fb9fd993700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb9f80c3d80 0x7fb9f8159cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:25.322 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.294+0000 7fb9fd993700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb9f80c3d80 0x7fb9f8159cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:43252/0 (socket says 172.21.15.195:43252) 2022-06-27T11:51:25.322 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.294+0000 7fb9fd993700 1 -- 172.21.15.195:0/484033365 learned_addr learned my addr 172.21.15.195:0/484033365 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:51:25.322 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.294+0000 7fb9fe194700 1 --2- 172.21.15.195:0/484033365 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb9f814e860 0x7fb9f80c6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:25.322 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.294+0000 7fb9fd192700 1 --2- 172.21.15.195:0/484033365 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb9f814e490 0x7fb9f80c63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:25.323 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.294+0000 7fb9fd993700 1 -- 172.21.15.195:0/484033365 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb9f814e490 msgr2=0x7fb9f80c63d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:25.323 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.294+0000 7fb9fd993700 1 --2- 172.21.15.195:0/484033365 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb9f814e490 0x7fb9f80c63d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:25.323 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.294+0000 7fb9fd993700 1 -- 172.21.15.195:0/484033365 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb9f814e860 msgr2=0x7fb9f80c6910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:25.323 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.294+0000 7fb9fd993700 1 --2- 172.21.15.195:0/484033365 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb9f814e860 0x7fb9f80c6910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:25.323 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.294+0000 7fb9fd993700 1 -- 172.21.15.195:0/484033365 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9f40092e0 con 0x7fb9f80c3d80 2022-06-27T11:51:25.324 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.294+0000 7fb9fe194700 1 --2- 172.21.15.195:0/484033365 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb9f814e860 0x7fb9f80c6910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T11:51:25.324 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.294+0000 7fb9fd993700 1 --2- 172.21.15.195:0/484033365 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb9f80c3d80 0x7fb9f8159cb0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fb9e800d860 tx=0x7fb9e8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:51:25.324 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.294+0000 7fb9e5ffb700 1 -- 172.21.15.195:0/484033365 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9e8014620 con 0x7fb9f80c3d80 2022-06-27T11:51:25.325 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.294+0000 7fb9e5ffb700 1 -- 172.21.15.195:0/484033365 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9e8014780 con 0x7fb9f80c3d80 2022-06-27T11:51:25.325 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.294+0000 7fb9ffbf7700 1 -- 172.21.15.195:0/484033365 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9f8069e40 con 0x7fb9f80c3d80 2022-06-27T11:51:25.325 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.295+0000 7fb9e5ffb700 1 -- 172.21.15.195:0/484033365 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9e8014b20 con 0x7fb9f80c3d80 2022-06-27T11:51:25.325 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.295+0000 7fb9ffbf7700 1 -- 172.21.15.195:0/484033365 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9f806a400 con 0x7fb9f80c3d80 2022-06-27T11:51:25.326 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.296+0000 7fb9ffbf7700 1 -- 172.21.15.195:0/484033365 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9f8069930 con 0x7fb9f80c3d80 2022-06-27T11:51:25.326 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.296+0000 7fb9e5ffb700 1 -- 172.21.15.195:0/484033365 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fb9e8024070 con 0x7fb9f80c3d80 2022-06-27T11:51:25.326 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.296+0000 7fb9e5ffb700 1 --2- 172.21.15.195:0/484033365 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb9d8063d00 0x7fb9d80661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:25.327 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.297+0000 7fb9e5ffb700 1 -- 172.21.15.195:0/484033365 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6540+0+0 (secure 0 0 0) 0x7fb9e8081200 con 0x7fb9f80c3d80 2022-06-27T11:51:25.328 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.297+0000 7fb9fd192700 1 --2- 172.21.15.195:0/484033365 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb9d8063d00 0x7fb9d80661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:25.328 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.297+0000 7fb9fd192700 1 --2- 172.21.15.195:0/484033365 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb9d8063d00 0x7fb9d80661b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb9f8065b00 tx=0x7fb9f4017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:51:25.328 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.301+0000 7fb9e5ffb700 1 -- 172.21.15.195:0/484033365 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fb9e804fb20 con 0x7fb9f80c3d80 2022-06-27T11:51:25.531 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.531+0000 7fb9ffbf7700 1 -- 172.21.15.195:0/484033365 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb9f80722c0 con 0x7fb9f80c3d80 2022-06-27T11:51:25.532 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.532+0000 7fb9e5ffb700 1 -- 172.21.15.195:0/484033365 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+5371 (secure 0 0 0) 0x7fb9e8053140 con 0x7fb9f80c3d80 2022-06-27T11:51:25.533 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:51:25.533 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:51:21.983241+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4849},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4849":{"gid":4849,"name":"d","rank":0,"incarnation":15,"state":"up:reconnect","state_seq":15,"addr":"172.21.15.202:6835/3271891211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":3271891211},{"type":"v1","addr":"172.21.15.202:6835","nonce":3271891211}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:51:25.537 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.534+0000 7fb9ffbf7700 1 -- 172.21.15.195:0/484033365 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb9d8063d00 msgr2=0x7fb9d80661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:25.537 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.534+0000 7fb9ffbf7700 1 --2- 172.21.15.195:0/484033365 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb9d8063d00 0x7fb9d80661b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb9f8065b00 tx=0x7fb9f4017000 comp rx=0 tx=0).stop 2022-06-27T11:51:25.537 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.534+0000 7fb9ffbf7700 1 -- 172.21.15.195:0/484033365 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb9f80c3d80 msgr2=0x7fb9f8159cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:25.538 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.534+0000 7fb9ffbf7700 1 --2- 172.21.15.195:0/484033365 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb9f80c3d80 0x7fb9f8159cb0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fb9e800d860 tx=0x7fb9e8008040 comp rx=0 tx=0).stop 2022-06-27T11:51:25.538 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.534+0000 7fb9ffbf7700 1 -- 172.21.15.195:0/484033365 shutdown_connections 2022-06-27T11:51:25.539 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.534+0000 7fb9ffbf7700 1 --2- 172.21.15.195:0/484033365 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb9d8063d00 0x7fb9d80661b0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:25.539 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.534+0000 7fb9ffbf7700 1 --2- 172.21.15.195:0/484033365 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb9f80c3d80 0x7fb9f8159cb0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:25.540 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.534+0000 7fb9ffbf7700 1 --2- 172.21.15.195:0/484033365 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb9f814e860 0x7fb9f80c6910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:25.541 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.534+0000 7fb9ffbf7700 1 --2- 172.21.15.195:0/484033365 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb9f814e490 0x7fb9f80c63d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:25.541 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.534+0000 7fb9ffbf7700 1 -- 172.21.15.195:0/484033365 >> 172.21.15.195:0/484033365 conn(0x7fb9f8063af0 msgr2=0x7fb9f814d500 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:51:25.542 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.534+0000 7fb9ffbf7700 1 -- 172.21.15.195:0/484033365 shutdown_connections 2022-06-27T11:51:25.542 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:25.534+0000 7fb9ffbf7700 1 -- 172.21.15.195:0/484033365 wait complete. 2022-06-27T11:51:25.543 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 16 2022-06-27T11:51:25.546 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T11:51:27.093 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T11:51:27.132 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T11:51:27.548 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:51:27.700 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.682+0000 7f70c552d700 1 Processor -- start 2022-06-27T11:51:27.701 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.682+0000 7f70c552d700 1 -- start start 2022-06-27T11:51:27.701 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.683+0000 7f70c552d700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f70c006f640 0x7f70c0156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:27.701 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.683+0000 7f70c552d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f70c00c4160 0x7f70c00c1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:27.702 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.683+0000 7f70c552d700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f70c00c39e0 0x7f70c01535a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:27.702 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.683+0000 7f70c552d700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70c00c1a20 con 0x7f70c00c4160 2022-06-27T11:51:27.702 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.683+0000 7f70c552d700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70c00c1130 con 0x7f70c006f640 2022-06-27T11:51:27.702 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.683+0000 7f70c552d700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70c006ea90 con 0x7f70c00c39e0 2022-06-27T11:51:27.703 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.683+0000 7f70be7fc700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f70c00c4160 0x7f70c00c1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:27.703 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.683+0000 7f70be7fc700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f70c00c4160 0x7f70c00c1eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:43260/0 (socket says 172.21.15.195:43260) 2022-06-27T11:51:27.703 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.683+0000 7f70be7fc700 1 -- 172.21.15.195:0/3310574457 learned_addr learned my addr 172.21.15.195:0/3310574457 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:51:27.703 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.684+0000 7f70beffd700 1 --2- 172.21.15.195:0/3310574457 >> v2:172.21.15.202:3300/0 conn(0x7f70c006f640 0x7f70c0156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:27.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.684+0000 7f70be7fc700 1 -- 172.21.15.195:0/3310574457 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f70c00c39e0 msgr2=0x7f70c01535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:51:27.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.684+0000 7f70be7fc700 1 --2- 172.21.15.195:0/3310574457 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f70c00c39e0 0x7f70c01535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:27.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.684+0000 7f70be7fc700 1 -- 172.21.15.195:0/3310574457 >> v2:172.21.15.202:3300/0 conn(0x7f70c006f640 msgr2=0x7f70c0156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:51:27.705 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.684+0000 7f70be7fc700 1 --2- 172.21.15.195:0/3310574457 >> v2:172.21.15.202:3300/0 conn(0x7f70c006f640 0x7f70c0156ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:27.705 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.684+0000 7f70be7fc700 1 -- 172.21.15.195:0/3310574457 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70c006ed70 con 0x7f70c00c4160 2022-06-27T11:51:27.705 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.684+0000 7f70be7fc700 1 --2- 172.21.15.195:0/3310574457 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f70c00c4160 0x7f70c00c1eb0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f70c006db30 tx=0x7f70b4009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34b2872d96c81501 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:51:27.706 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.684+0000 7f70bdffb700 1 -- 172.21.15.195:0/3310574457 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70b400b6d0 con 0x7f70c00c4160 2022-06-27T11:51:27.706 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.684+0000 7f70bdffb700 1 -- 172.21.15.195:0/3310574457 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70b400b830 con 0x7f70c00c4160 2022-06-27T11:51:27.706 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.684+0000 7f70bdffb700 1 -- 172.21.15.195:0/3310574457 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70b400bbd0 con 0x7f70c00c4160 2022-06-27T11:51:27.706 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.684+0000 7f70c552d700 1 -- 172.21.15.195:0/3310574457 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f70c00c4160 msgr2=0x7f70c00c1eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:27.707 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.684+0000 7f70c552d700 1 --2- 172.21.15.195:0/3310574457 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f70c00c4160 0x7f70c00c1eb0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f70c006db30 tx=0x7f70b4009120 comp rx=0 tx=0).stop 2022-06-27T11:51:27.707 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.684+0000 7f70c552d700 1 -- 172.21.15.195:0/3310574457 shutdown_connections 2022-06-27T11:51:27.707 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.684+0000 7f70c552d700 1 --2- 172.21.15.195:0/3310574457 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f70c00c39e0 0x7f70c01535a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:27.707 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.685+0000 7f70c552d700 1 --2- 172.21.15.195:0/3310574457 >> v2:172.21.15.202:3300/0 conn(0x7f70c006f640 0x7f70c0156ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:27.708 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.685+0000 7f70c552d700 1 --2- 172.21.15.195:0/3310574457 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f70c00c4160 0x7f70c00c1eb0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:27.708 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.685+0000 7f70c552d700 1 -- 172.21.15.195:0/3310574457 >> 172.21.15.195:0/3310574457 conn(0x7f70c00c6350 msgr2=0x7f70c00c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:51:27.708 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.685+0000 7f70c552d700 1 -- 172.21.15.195:0/3310574457 shutdown_connections 2022-06-27T11:51:27.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.685+0000 7f70c552d700 1 -- 172.21.15.195:0/3310574457 wait complete. 2022-06-27T11:51:27.710 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.686+0000 7f70c552d700 1 Processor -- start 2022-06-27T11:51:27.711 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.686+0000 7f70c552d700 1 -- start start 2022-06-27T11:51:27.711 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.687+0000 7f70c552d700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f70c006f640 0x7f70c006bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:27.711 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.687+0000 7f70c552d700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f70c00c39e0 0x7f70c016a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:27.712 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.687+0000 7f70c552d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f70c00c4160 0x7f70c016a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:27.712 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.687+0000 7f70c552d700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70c0169e90 con 0x7f70c00c4160 2022-06-27T11:51:27.712 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.687+0000 7f70c552d700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70c00c8890 con 0x7f70c006f640 2022-06-27T11:51:27.712 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.687+0000 7f70c552d700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70c00c8b00 con 0x7f70c00c39e0 2022-06-27T11:51:27.713 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.687+0000 7f70bf7fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f70c00c4160 0x7f70c016a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:27.713 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.687+0000 7f70bf7fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f70c00c4160 0x7f70c016a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:43270/0 (socket says 172.21.15.195:43270) 2022-06-27T11:51:27.713 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.687+0000 7f70bf7fe700 1 -- 172.21.15.195:0/4076036370 learned_addr learned my addr 172.21.15.195:0/4076036370 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:51:27.714 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.687+0000 7f70beffd700 1 --2- 172.21.15.195:0/4076036370 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f70c006f640 0x7f70c006bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:27.714 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.687+0000 7f70be7fc700 1 --2- 172.21.15.195:0/4076036370 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f70c00c39e0 0x7f70c016a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:27.714 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.687+0000 7f70bf7fe700 1 -- 172.21.15.195:0/4076036370 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f70c00c39e0 msgr2=0x7f70c016a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:27.714 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.687+0000 7f70bf7fe700 1 --2- 172.21.15.195:0/4076036370 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f70c00c39e0 0x7f70c016a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:27.715 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.687+0000 7f70bf7fe700 1 -- 172.21.15.195:0/4076036370 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f70c006f640 msgr2=0x7f70c006bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:27.715 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.687+0000 7f70bf7fe700 1 --2- 172.21.15.195:0/4076036370 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f70c006f640 0x7f70c006bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:27.715 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.687+0000 7f70bf7fe700 1 -- 172.21.15.195:0/4076036370 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70b40093a0 con 0x7f70c00c4160 2022-06-27T11:51:27.715 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.687+0000 7f70bf7fe700 1 --2- 172.21.15.195:0/4076036370 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f70c00c4160 0x7f70c016a860 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f70b800bc60 tx=0x7f70b8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:51:27.716 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.687+0000 7f70a6ffd700 1 -- 172.21.15.195:0/4076036370 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70b80092e0 con 0x7f70c00c4160 2022-06-27T11:51:27.716 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.687+0000 7f70a6ffd700 1 -- 172.21.15.195:0/4076036370 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70b800c7b0 con 0x7f70c00c4160 2022-06-27T11:51:27.716 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.687+0000 7f70c552d700 1 -- 172.21.15.195:0/4076036370 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70c00c8de0 con 0x7f70c00c4160 2022-06-27T11:51:27.716 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.687+0000 7f70a6ffd700 1 -- 172.21.15.195:0/4076036370 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70b800cb90 con 0x7f70c00c4160 2022-06-27T11:51:27.717 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.687+0000 7f70c552d700 1 -- 172.21.15.195:0/4076036370 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70c00c93a0 con 0x7f70c00c4160 2022-06-27T11:51:27.717 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.688+0000 7f70c552d700 1 -- 172.21.15.195:0/4076036370 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70c016a040 con 0x7f70c00c4160 2022-06-27T11:51:27.717 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.689+0000 7f70a6ffd700 1 -- 172.21.15.195:0/4076036370 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f70b8007400 con 0x7f70c00c4160 2022-06-27T11:51:27.717 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.689+0000 7f70a6ffd700 1 --2- 172.21.15.195:0/4076036370 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f70a0063d00 0x7f70a00661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:27.718 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.689+0000 7f70a6ffd700 1 -- 172.21.15.195:0/4076036370 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6540+0+0 (secure 0 0 0) 0x7f70b8080580 con 0x7f70c00c4160 2022-06-27T11:51:27.718 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.689+0000 7f70beffd700 1 --2- 172.21.15.195:0/4076036370 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f70a0063d00 0x7f70a00661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:27.718 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.690+0000 7f70beffd700 1 --2- 172.21.15.195:0/4076036370 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f70a0063d00 0x7f70a00661b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f70b000aa30 tx=0x7f70b0008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:51:27.718 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.693+0000 7f70a6ffd700 1 -- 172.21.15.195:0/4076036370 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f70b804ac90 con 0x7f70c00c4160 2022-06-27T11:51:27.923 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.921+0000 7f70c552d700 1 -- 172.21.15.195:0/4076036370 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f70c00722c0 con 0x7f70c00c4160 2022-06-27T11:51:27.923 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.921+0000 7f70a6ffd700 1 -- 172.21.15.195:0/4076036370 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+5368 (secure 0 0 0) 0x7f70c016a040 con 0x7f70c00c4160 2022-06-27T11:51:27.924 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:51:27.925 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:51:26.995446+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4849},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4849":{"gid":4849,"name":"d","rank":0,"incarnation":15,"state":"up:rejoin","state_seq":17,"addr":"172.21.15.202:6835/3271891211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":3271891211},{"type":"v1","addr":"172.21.15.202:6835","nonce":3271891211}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:51:27.928 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.923+0000 7f70c552d700 1 -- 172.21.15.195:0/4076036370 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f70a0063d00 msgr2=0x7f70a00661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:27.929 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.923+0000 7f70c552d700 1 --2- 172.21.15.195:0/4076036370 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f70a0063d00 0x7f70a00661b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f70b000aa30 tx=0x7f70b0008040 comp rx=0 tx=0).stop 2022-06-27T11:51:27.929 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.923+0000 7f70c552d700 1 -- 172.21.15.195:0/4076036370 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f70c00c4160 msgr2=0x7f70c016a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:27.929 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.923+0000 7f70c552d700 1 --2- 172.21.15.195:0/4076036370 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f70c00c4160 0x7f70c016a860 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f70b800bc60 tx=0x7f70b8009840 comp rx=0 tx=0).stop 2022-06-27T11:51:27.930 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.923+0000 7f70c552d700 1 -- 172.21.15.195:0/4076036370 shutdown_connections 2022-06-27T11:51:27.931 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.923+0000 7f70c552d700 1 --2- 172.21.15.195:0/4076036370 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f70a0063d00 0x7f70a00661b0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:27.931 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.923+0000 7f70c552d700 1 --2- 172.21.15.195:0/4076036370 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f70c00c4160 0x7f70c016a860 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:27.931 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.923+0000 7f70c552d700 1 --2- 172.21.15.195:0/4076036370 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f70c006f640 0x7f70c006bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:27.931 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.923+0000 7f70c552d700 1 --2- 172.21.15.195:0/4076036370 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f70c00c39e0 0x7f70c016a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:27.932 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.923+0000 7f70c552d700 1 -- 172.21.15.195:0/4076036370 >> 172.21.15.195:0/4076036370 conn(0x7f70c00c6350 msgr2=0x7f70c00c20d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:51:27.932 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.923+0000 7f70c552d700 1 -- 172.21.15.195:0/4076036370 shutdown_connections 2022-06-27T11:51:27.932 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:27.923+0000 7f70c552d700 1 -- 172.21.15.195:0/4076036370 wait complete. 2022-06-27T11:51:27.932 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 17 2022-06-27T11:51:27.939 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 17, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-06-27T11:47:52.358095+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 17, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4849": { "addr": "172.21.15.202:6835/3271891211", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6834", "nonce": 3271891211, "type": "v2" }, { "addr": "172.21.15.202:6835", "nonce": 3271891211, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4849, "incarnation": 15, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:rejoin", "state_seq": 17 } }, "last_failure": 0, "last_failure_osd_epoch": 37, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-06-27T11:51:26.995446+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4849 } } } ], "standbys": [ { "addr": "172.21.15.202:6837/3305006841", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6836", "nonce": 3305006841, "type": "v2" }, { "addr": "172.21.15.202:6837", "nonce": 3305006841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6835/2060709822", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6834", "nonce": 2060709822, "type": "v2" }, { "addr": "172.21.15.195:6835", "nonce": 2060709822, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6838/1598751652", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6836", "nonce": 1598751652, "type": "v2" }, { "addr": "172.21.15.195:6838", "nonce": 1598751652, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4275, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6839/1313800661", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6837", "nonce": 1313800661, "type": "v2" }, { "addr": "172.21.15.195:6839", "nonce": 1313800661, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4278, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-06-27T11:51:29.941 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:51:30.084 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.076+0000 7fcbbc4c2700 1 Processor -- start 2022-06-27T11:51:30.084 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.076+0000 7fcbbc4c2700 1 -- start start 2022-06-27T11:51:30.085 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.077+0000 7fcbbc4c2700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcbb406a440 0x7fcbb40688a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:30.085 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.077+0000 7fcbbc4c2700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fcbb41564d0 0x7fcbb4156950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:30.085 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.077+0000 7fcbbc4c2700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbb40c4b10 con 0x7fcbb4156100 2022-06-27T11:51:30.085 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.077+0000 7fcbbc4c2700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbb40c3dc0 con 0x7fcbb41564d0 2022-06-27T11:51:30.086 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.077+0000 7fcbbc4c2700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb4068f20 con 0x7fcbb406a440 2022-06-27T11:51:30.086 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.077+0000 7fcbbaa5f700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fcbb41564d0 0x7fcbb4156950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:30.086 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.077+0000 7fcbba25e700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fcbb4156100 0x7fcbb40c4fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:49462/0 (socket says 172.21.15.195:49462) 2022-06-27T11:51:30.087 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.077+0000 7fcbbaa5f700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fcbb41564d0 0x7fcbb4156950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:43274/0 (socket says 172.21.15.195:43274) 2022-06-27T11:51:30.087 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.077+0000 7fcbba25e700 1 -- 172.21.15.195:0/4148959277 learned_addr learned my addr 172.21.15.195:0/4148959277 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:51:30.087 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.077+0000 7fcbb9a5d700 1 --2- 172.21.15.195:0/4148959277 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcbb406a440 0x7fcbb40688a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:30.087 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.077+0000 7fcbbaa5f700 1 -- 172.21.15.195:0/4148959277 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcbb406a440 msgr2=0x7fcbb40688a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:51:30.088 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.077+0000 7fcbbaa5f700 1 --2- 172.21.15.195:0/4148959277 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcbb406a440 0x7fcbb40688a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:30.088 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.077+0000 7fcbbaa5f700 1 -- 172.21.15.195:0/4148959277 >> v1:172.21.15.195:6789/0 conn(0x7fcbb4156100 legacy=0x7fcbb40c4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:30.088 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.077+0000 7fcbb925c700 1 -- 172.21.15.195:0/4148959277 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2424864295 0 0) 0x7fcbb40c4b10 con 0x7fcbb4156100 2022-06-27T11:51:30.088 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.077+0000 7fcbbaa5f700 1 -- 172.21.15.195:0/4148959277 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbb40c56b0 con 0x7fcbb41564d0 2022-06-27T11:51:30.089 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.077+0000 7fcbbaa5f700 1 --2- 172.21.15.195:0/4148959277 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcbb41564d0 0x7fcbb4156950 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fcbac009e60 tx=0x7fcbac009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fb0e6a3cf2a5238 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:51:30.090 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.078+0000 7fcbb925c700 1 -- 172.21.15.195:0/4148959277 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbac00e070 con 0x7fcbb41564d0 2022-06-27T11:51:30.091 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.078+0000 7fcbb925c700 1 -- 172.21.15.195:0/4148959277 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbac00bc30 con 0x7fcbb41564d0 2022-06-27T11:51:30.091 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.078+0000 7fcbb925c700 1 -- 172.21.15.195:0/4148959277 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbac012430 con 0x7fcbb41564d0 2022-06-27T11:51:30.091 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.078+0000 7fcbbc4c2700 1 -- 172.21.15.195:0/4148959277 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcbb41564d0 msgr2=0x7fcbb4156950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:30.092 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.078+0000 7fcbbc4c2700 1 --2- 172.21.15.195:0/4148959277 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcbb41564d0 0x7fcbb4156950 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fcbac009e60 tx=0x7fcbac009780 comp rx=0 tx=0).stop 2022-06-27T11:51:30.092 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.078+0000 7fcbbc4c2700 1 -- 172.21.15.195:0/4148959277 shutdown_connections 2022-06-27T11:51:30.093 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.078+0000 7fcbbc4c2700 1 --2- 172.21.15.195:0/4148959277 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcbb41564d0 0x7fcbb4156950 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:30.093 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.078+0000 7fcbbc4c2700 1 --2- 172.21.15.195:0/4148959277 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcbb406a440 0x7fcbb40688a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:30.093 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.078+0000 7fcbbc4c2700 1 -- 172.21.15.195:0/4148959277 >> 172.21.15.195:0/4148959277 conn(0x7fcbb40c9b00 msgr2=0x7fcbb40c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:51:30.093 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.078+0000 7fcbbc4c2700 1 -- 172.21.15.195:0/4148959277 shutdown_connections 2022-06-27T11:51:30.094 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.078+0000 7fcbbc4c2700 1 -- 172.21.15.195:0/4148959277 wait complete. 2022-06-27T11:51:30.094 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.079+0000 7fcbbc4c2700 1 Processor -- start 2022-06-27T11:51:30.095 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.079+0000 7fcbbc4c2700 1 -- start start 2022-06-27T11:51:30.100 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.080+0000 7fcbbc4c2700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcbb406a440 0x7fcbb40be590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:30.100 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.080+0000 7fcbbc4c2700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcbb4156100 0x7fcbb40bdfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:30.101 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.080+0000 7fcbbc4c2700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcbb41564d0 0x7fcbb40bd000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:30.101 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.080+0000 7fcbbc4c2700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb406ec70 con 0x7fcbb4156100 2022-06-27T11:51:30.101 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.080+0000 7fcbbc4c2700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb40ba800 con 0x7fcbb406a440 2022-06-27T11:51:30.101 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.080+0000 7fcbbc4c2700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb40ba4b0 con 0x7fcbb41564d0 2022-06-27T11:51:30.102 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.080+0000 7fcbb9a5d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcbb4156100 0x7fcbb40bdfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:30.102 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.080+0000 7fcbb9a5d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcbb4156100 0x7fcbb40bdfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:43282/0 (socket says 172.21.15.195:43282) 2022-06-27T11:51:30.103 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.080+0000 7fcbb9a5d700 1 -- 172.21.15.195:0/2157879928 learned_addr learned my addr 172.21.15.195:0/2157879928 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:51:30.103 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.080+0000 7fcbba25e700 1 --2- 172.21.15.195:0/2157879928 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcbb406a440 0x7fcbb40be590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:30.104 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.080+0000 7fcbbaa5f700 1 --2- 172.21.15.195:0/2157879928 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcbb41564d0 0x7fcbb40bd000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:30.104 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.080+0000 7fcbb9a5d700 1 -- 172.21.15.195:0/2157879928 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcbb41564d0 msgr2=0x7fcbb40bd000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:30.105 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.080+0000 7fcbb9a5d700 1 --2- 172.21.15.195:0/2157879928 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcbb41564d0 0x7fcbb40bd000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:30.105 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.080+0000 7fcbb9a5d700 1 -- 172.21.15.195:0/2157879928 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcbb406a440 msgr2=0x7fcbb40be590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:30.105 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.080+0000 7fcbb9a5d700 1 --2- 172.21.15.195:0/2157879928 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcbb406a440 0x7fcbb40be590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:30.106 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.080+0000 7fcbb9a5d700 1 -- 172.21.15.195:0/2157879928 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbac009a20 con 0x7fcbb4156100 2022-06-27T11:51:30.106 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.080+0000 7fcbbaa5f700 1 --2- 172.21.15.195:0/2157879928 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcbb41564d0 0x7fcbb40bd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T11:51:30.106 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.080+0000 7fcbba25e700 1 --2- 172.21.15.195:0/2157879928 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcbb406a440 0x7fcbb40be590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T11:51:30.107 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.081+0000 7fcbb9a5d700 1 --2- 172.21.15.195:0/2157879928 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcbb4156100 0x7fcbb40bdfd0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fcba400d890 tx=0x7fcba4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:51:30.107 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.081+0000 7fcba27fc700 1 -- 172.21.15.195:0/2157879928 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba4014620 con 0x7fcbb4156100 2022-06-27T11:51:30.107 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.081+0000 7fcba27fc700 1 -- 172.21.15.195:0/2157879928 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcba4014780 con 0x7fcbb4156100 2022-06-27T11:51:30.108 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.081+0000 7fcba27fc700 1 -- 172.21.15.195:0/2157879928 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba4014b40 con 0x7fcbb4156100 2022-06-27T11:51:30.108 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.081+0000 7fcbbc4c2700 1 -- 172.21.15.195:0/2157879928 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbb406eec0 con 0x7fcbb4156100 2022-06-27T11:51:30.109 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.081+0000 7fcbbc4c2700 1 -- 172.21.15.195:0/2157879928 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbb40bf800 con 0x7fcbb4156100 2022-06-27T11:51:30.109 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.082+0000 7fcbbc4c2700 1 -- 172.21.15.195:0/2157879928 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbb40722c0 con 0x7fcbb4156100 2022-06-27T11:51:30.109 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.082+0000 7fcba27fc700 1 -- 172.21.15.195:0/2157879928 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fcba4024070 con 0x7fcbb4156100 2022-06-27T11:51:30.110 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.082+0000 7fcba27fc700 1 --2- 172.21.15.195:0/2157879928 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcb94063d30 0x7fcb940661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:30.110 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.083+0000 7fcba27fc700 1 -- 172.21.15.195:0/2157879928 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6540+0+0 (secure 0 0 0) 0x7fcba4081060 con 0x7fcbb4156100 2022-06-27T11:51:30.111 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.083+0000 7fcbba25e700 1 --2- 172.21.15.195:0/2157879928 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcb94063d30 0x7fcb940661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:30.111 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.083+0000 7fcbba25e700 1 --2- 172.21.15.195:0/2157879928 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcb94063d30 0x7fcb940661e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fcbb0007fd0 tx=0x7fcbb0009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:51:30.112 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.087+0000 7fcba27fc700 1 -- 172.21.15.195:0/2157879928 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fcba404fa70 con 0x7fcbb4156100 2022-06-27T11:51:30.317 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.310+0000 7fcbbc4c2700 1 -- 172.21.15.195:0/2157879928 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcbb40ba940 con 0x7fcbb4156100 2022-06-27T11:51:30.317 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.311+0000 7fcba27fc700 1 -- 172.21.15.195:0/2157879928 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+5368 (secure 0 0 0) 0x7fcba4053090 con 0x7fcbb4156100 2022-06-27T11:51:30.318 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:51:30.318 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:51:26.995446+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4849},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4849":{"gid":4849,"name":"d","rank":0,"incarnation":15,"state":"up:rejoin","state_seq":17,"addr":"172.21.15.202:6835/3271891211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":3271891211},{"type":"v1","addr":"172.21.15.202:6835","nonce":3271891211}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:51:30.320 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.313+0000 7fcbbc4c2700 1 -- 172.21.15.195:0/2157879928 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcb94063d30 msgr2=0x7fcb940661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:30.320 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.313+0000 7fcbbc4c2700 1 --2- 172.21.15.195:0/2157879928 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcb94063d30 0x7fcb940661e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fcbb0007fd0 tx=0x7fcbb0009040 comp rx=0 tx=0).stop 2022-06-27T11:51:30.320 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.313+0000 7fcbbc4c2700 1 -- 172.21.15.195:0/2157879928 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcbb4156100 msgr2=0x7fcbb40bdfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:30.320 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.313+0000 7fcbbc4c2700 1 --2- 172.21.15.195:0/2157879928 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcbb4156100 0x7fcbb40bdfd0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fcba400d890 tx=0x7fcba4008040 comp rx=0 tx=0).stop 2022-06-27T11:51:30.321 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.313+0000 7fcbbc4c2700 1 -- 172.21.15.195:0/2157879928 shutdown_connections 2022-06-27T11:51:30.321 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.313+0000 7fcbbc4c2700 1 --2- 172.21.15.195:0/2157879928 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcb94063d30 0x7fcb940661e0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:30.321 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.313+0000 7fcbbc4c2700 1 --2- 172.21.15.195:0/2157879928 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcbb41564d0 0x7fcbb40bd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:30.322 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.313+0000 7fcbbc4c2700 1 --2- 172.21.15.195:0/2157879928 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcbb406a440 0x7fcbb40be590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:30.322 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.313+0000 7fcbbc4c2700 1 --2- 172.21.15.195:0/2157879928 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcbb4156100 0x7fcbb40bdfd0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:30.323 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.313+0000 7fcbbc4c2700 1 -- 172.21.15.195:0/2157879928 >> 172.21.15.195:0/2157879928 conn(0x7fcbb40c9b00 msgr2=0x7fcbb40c8880 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:51:30.323 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.313+0000 7fcbbc4c2700 1 -- 172.21.15.195:0/2157879928 shutdown_connections 2022-06-27T11:51:30.324 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:30.313+0000 7fcbbc4c2700 1 -- 172.21.15.195:0/2157879928 wait complete. 2022-06-27T11:51:30.325 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 17 2022-06-27T11:51:30.327 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T11:51:32.328 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:51:32.508 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.485+0000 7f6c911f6700 1 Processor -- start 2022-06-27T11:51:32.508 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.485+0000 7f6c911f6700 1 -- start start 2022-06-27T11:51:32.509 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.486+0000 7f6c911f6700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f6c8c06f640 0x7f6c8c156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:32.509 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.486+0000 7f6c911f6700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6c8c0c4160 0x7f6c8c0c1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:32.510 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.486+0000 7f6c911f6700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f6c8c0c39e0 0x7f6c8c1535a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:32.510 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.486+0000 7f6c911f6700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c8c0c1a20 con 0x7f6c8c06f640 2022-06-27T11:51:32.511 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.486+0000 7f6c911f6700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c8c0c1130 con 0x7f6c8c0c39e0 2022-06-27T11:51:32.511 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.486+0000 7f6c911f6700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c8c06ea90 con 0x7f6c8c0c4160 2022-06-27T11:51:32.512 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.486+0000 7f6c8ad9d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f6c8c06f640 0x7f6c8c156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:32.513 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.486+0000 7f6c8ad9d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f6c8c06f640 0x7f6c8c156ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:43288/0 (socket says 172.21.15.195:43288) 2022-06-27T11:51:32.513 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.486+0000 7f6c8ad9d700 1 -- 172.21.15.195:0/1362561882 learned_addr learned my addr 172.21.15.195:0/1362561882 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:51:32.514 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.486+0000 7f6c8a59c700 1 --2- 172.21.15.195:0/1362561882 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6c8c0c4160 0x7f6c8c0c1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:32.515 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.486+0000 7f6c8b59e700 1 --2- 172.21.15.195:0/1362561882 >> v2:172.21.15.202:3300/0 conn(0x7f6c8c0c39e0 0x7f6c8c1535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:32.515 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.486+0000 7f6c8ad9d700 1 -- 172.21.15.195:0/1362561882 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6c8c0c4160 msgr2=0x7f6c8c0c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:51:32.515 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.486+0000 7f6c8ad9d700 1 --2- 172.21.15.195:0/1362561882 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6c8c0c4160 0x7f6c8c0c1eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:32.516 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.486+0000 7f6c8ad9d700 1 -- 172.21.15.195:0/1362561882 >> v2:172.21.15.202:3300/0 conn(0x7f6c8c0c39e0 msgr2=0x7f6c8c1535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:51:32.516 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.486+0000 7f6c8ad9d700 1 --2- 172.21.15.195:0/1362561882 >> v2:172.21.15.202:3300/0 conn(0x7f6c8c0c39e0 0x7f6c8c1535a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:32.517 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.486+0000 7f6c8ad9d700 1 -- 172.21.15.195:0/1362561882 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c8c06ed70 con 0x7f6c8c06f640 2022-06-27T11:51:32.518 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.486+0000 7f6c8ad9d700 1 --2- 172.21.15.195:0/1362561882 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6c8c06f640 0x7f6c8c156ed0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c06b530 tx=0x7f6c80009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec639082c7be51d5 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:51:32.518 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.487+0000 7f6c89d9b700 1 -- 172.21.15.195:0/1362561882 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c80011630 con 0x7f6c8c06f640 2022-06-27T11:51:32.519 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.487+0000 7f6c89d9b700 1 -- 172.21.15.195:0/1362561882 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c80011790 con 0x7f6c8c06f640 2022-06-27T11:51:32.520 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.487+0000 7f6c89d9b700 1 -- 172.21.15.195:0/1362561882 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c80011b30 con 0x7f6c8c06f640 2022-06-27T11:51:32.520 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.487+0000 7f6c911f6700 1 -- 172.21.15.195:0/1362561882 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6c8c06f640 msgr2=0x7f6c8c156ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:32.521 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.487+0000 7f6c911f6700 1 --2- 172.21.15.195:0/1362561882 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6c8c06f640 0x7f6c8c156ed0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c06b530 tx=0x7f6c80009120 comp rx=0 tx=0).stop 2022-06-27T11:51:32.521 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.487+0000 7f6c911f6700 1 -- 172.21.15.195:0/1362561882 shutdown_connections 2022-06-27T11:51:32.522 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.487+0000 7f6c911f6700 1 --2- 172.21.15.195:0/1362561882 >> v2:172.21.15.202:3300/0 conn(0x7f6c8c0c39e0 0x7f6c8c1535a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:32.522 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.487+0000 7f6c911f6700 1 --2- 172.21.15.195:0/1362561882 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6c8c06f640 0x7f6c8c156ed0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:32.523 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.487+0000 7f6c911f6700 1 --2- 172.21.15.195:0/1362561882 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6c8c0c4160 0x7f6c8c0c1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:32.523 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.487+0000 7f6c911f6700 1 -- 172.21.15.195:0/1362561882 >> 172.21.15.195:0/1362561882 conn(0x7f6c8c0c6350 msgr2=0x7f6c8c0c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:51:32.523 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.487+0000 7f6c911f6700 1 -- 172.21.15.195:0/1362561882 shutdown_connections 2022-06-27T11:51:32.524 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.487+0000 7f6c911f6700 1 -- 172.21.15.195:0/1362561882 wait complete. 2022-06-27T11:51:32.524 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.489+0000 7f6c911f6700 1 Processor -- start 2022-06-27T11:51:32.525 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.489+0000 7f6c911f6700 1 -- start start 2022-06-27T11:51:32.525 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.489+0000 7f6c911f6700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6c8c06f640 0x7f6c8c14f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:32.525 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.489+0000 7f6c911f6700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6c8c0c39e0 0x7f6c8c0ba6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:32.526 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.489+0000 7f6c911f6700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6c8c0c4160 0x7f6c8c0bac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:32.526 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.489+0000 7f6c911f6700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c8c0bb1e0 con 0x7f6c8c06f640 2022-06-27T11:51:32.527 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.489+0000 7f6c911f6700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c8c06b370 con 0x7f6c8c0c4160 2022-06-27T11:51:32.528 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.489+0000 7f6c911f6700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c8c157160 con 0x7f6c8c0c39e0 2022-06-27T11:51:32.528 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.489+0000 7f6c8ad9d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6c8c06f640 0x7f6c8c14f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:32.528 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.489+0000 7f6c8ad9d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6c8c06f640 0x7f6c8c14f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:43294/0 (socket says 172.21.15.195:43294) 2022-06-27T11:51:32.530 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.489+0000 7f6c8ad9d700 1 -- 172.21.15.195:0/1176009942 learned_addr learned my addr 172.21.15.195:0/1176009942 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:51:32.530 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.489+0000 7f6c8b59e700 1 --2- 172.21.15.195:0/1176009942 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6c8c0c4160 0x7f6c8c0bac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:32.531 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.489+0000 7f6c8a59c700 1 --2- 172.21.15.195:0/1176009942 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6c8c0c39e0 0x7f6c8c0ba6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:32.531 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.490+0000 7f6c8ad9d700 1 -- 172.21.15.195:0/1176009942 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6c8c0c39e0 msgr2=0x7f6c8c0ba6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:32.531 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.490+0000 7f6c8ad9d700 1 --2- 172.21.15.195:0/1176009942 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6c8c0c39e0 0x7f6c8c0ba6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:32.532 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.490+0000 7f6c8ad9d700 1 -- 172.21.15.195:0/1176009942 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6c8c0c4160 msgr2=0x7f6c8c0bac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:32.532 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.490+0000 7f6c8ad9d700 1 --2- 172.21.15.195:0/1176009942 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6c8c0c4160 0x7f6c8c0bac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:32.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.490+0000 7f6c8ad9d700 1 -- 172.21.15.195:0/1176009942 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c800093a0 con 0x7f6c8c06f640 2022-06-27T11:51:32.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.490+0000 7f6c8ad9d700 1 --2- 172.21.15.195:0/1176009942 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6c8c06f640 0x7f6c8c14f7f0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c0650f0 tx=0x7f6c80017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:51:32.534 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.490+0000 7f6c72ffd700 1 -- 172.21.15.195:0/1176009942 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c80011900 con 0x7f6c8c06f640 2022-06-27T11:51:32.534 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.490+0000 7f6c72ffd700 1 -- 172.21.15.195:0/1176009942 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c80011a80 con 0x7f6c8c06f640 2022-06-27T11:51:32.535 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.490+0000 7f6c911f6700 1 -- 172.21.15.195:0/1176009942 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c8c156550 con 0x7f6c8c06f640 2022-06-27T11:51:32.535 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.490+0000 7f6c72ffd700 1 -- 172.21.15.195:0/1176009942 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c800042a0 con 0x7f6c8c06f640 2022-06-27T11:51:32.535 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.490+0000 7f6c911f6700 1 -- 172.21.15.195:0/1176009942 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c8c156b10 con 0x7f6c8c06f640 2022-06-27T11:51:32.535 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.491+0000 7f6c911f6700 1 -- 172.21.15.195:0/1176009942 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c8c0722c0 con 0x7f6c8c06f640 2022-06-27T11:51:32.536 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.491+0000 7f6c72ffd700 1 -- 172.21.15.195:0/1176009942 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f6c8002b030 con 0x7f6c8c06f640 2022-06-27T11:51:32.536 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.492+0000 7f6c72ffd700 1 --2- 172.21.15.195:0/1176009942 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6c6c063d30 0x7f6c6c0661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:32.536 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.492+0000 7f6c72ffd700 1 -- 172.21.15.195:0/1176009942 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6540+0+0 (secure 0 0 0) 0x7f6c80081fe0 con 0x7f6c8c06f640 2022-06-27T11:51:32.536 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.492+0000 7f6c8a59c700 1 --2- 172.21.15.195:0/1176009942 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6c6c063d30 0x7f6c6c0661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:32.537 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.492+0000 7f6c8a59c700 1 --2- 172.21.15.195:0/1176009942 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6c6c063d30 0x7f6c6c0661e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c0c05d0 tx=0x7f6c7c009060 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:51:32.537 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.496+0000 7f6c72ffd700 1 -- 172.21.15.195:0/1176009942 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f6c800509f0 con 0x7f6c8c06f640 2022-06-27T11:51:32.725 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.724+0000 7f6c911f6700 1 -- 172.21.15.195:0/1176009942 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6c8c06f240 con 0x7f6c8c06f640 2022-06-27T11:51:32.726 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.725+0000 7f6c72ffd700 1 -- 172.21.15.195:0/1176009942 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+5368 (secure 0 0 0) 0x7f6c80029070 con 0x7f6c8c06f640 2022-06-27T11:51:32.727 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:51:32.727 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:51:26.995446+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4849},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4849":{"gid":4849,"name":"d","rank":0,"incarnation":15,"state":"up:rejoin","state_seq":17,"addr":"172.21.15.202:6835/3271891211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":3271891211},{"type":"v1","addr":"172.21.15.202:6835","nonce":3271891211}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:51:32.730 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.726+0000 7f6c911f6700 1 -- 172.21.15.195:0/1176009942 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6c6c063d30 msgr2=0x7f6c6c0661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:32.730 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.726+0000 7f6c911f6700 1 --2- 172.21.15.195:0/1176009942 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6c6c063d30 0x7f6c6c0661e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c0c05d0 tx=0x7f6c7c009060 comp rx=0 tx=0).stop 2022-06-27T11:51:32.731 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.726+0000 7f6c911f6700 1 -- 172.21.15.195:0/1176009942 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6c8c06f640 msgr2=0x7f6c8c14f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:32.731 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.726+0000 7f6c911f6700 1 --2- 172.21.15.195:0/1176009942 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6c8c06f640 0x7f6c8c14f7f0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c0650f0 tx=0x7f6c80017000 comp rx=0 tx=0).stop 2022-06-27T11:51:32.732 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.727+0000 7f6c911f6700 1 -- 172.21.15.195:0/1176009942 shutdown_connections 2022-06-27T11:51:32.732 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.727+0000 7f6c911f6700 1 --2- 172.21.15.195:0/1176009942 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6c6c063d30 0x7f6c6c0661e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:32.733 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.727+0000 7f6c911f6700 1 --2- 172.21.15.195:0/1176009942 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6c8c06f640 0x7f6c8c14f7f0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:32.733 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.727+0000 7f6c911f6700 1 --2- 172.21.15.195:0/1176009942 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6c8c0c4160 0x7f6c8c0bac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:32.734 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.727+0000 7f6c911f6700 1 --2- 172.21.15.195:0/1176009942 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6c8c0c39e0 0x7f6c8c0ba6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:32.734 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.727+0000 7f6c911f6700 1 -- 172.21.15.195:0/1176009942 >> 172.21.15.195:0/1176009942 conn(0x7f6c8c0c6350 msgr2=0x7f6c8c0c7a60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:51:32.735 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.727+0000 7f6c911f6700 1 -- 172.21.15.195:0/1176009942 shutdown_connections 2022-06-27T11:51:32.735 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:32.727+0000 7f6c911f6700 1 -- 172.21.15.195:0/1176009942 wait complete. 2022-06-27T11:51:32.736 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 17 2022-06-27T11:51:32.738 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T11:51:34.740 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:51:34.878 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:34.877+0000 7fef0afa5700 1 Processor -- start 2022-06-27T11:51:34.879 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:34.877+0000 7fef0afa5700 1 -- start start 2022-06-27T11:51:34.880 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:34.878+0000 7fef0afa5700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fef0406e850 0x7fef041570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:34.880 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:34.878+0000 7fef0afa5700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fef0406fef0 0x7fef04069de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:34.881 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:34.878+0000 7fef0afa5700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef0406fba0 con 0x7fef040c34a0 2022-06-27T11:51:34.881 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:34.878+0000 7fef0afa5700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef041575e0 con 0x7fef0406e850 2022-06-27T11:51:34.882 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:34.878+0000 7fef0afa5700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef0406f5e0 con 0x7fef0406fef0 2022-06-27T11:51:34.883 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:34.878+0000 7fef08d41700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fef0406e850 0x7fef041570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:34.884 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:34.878+0000 7fef09542700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fef040c34a0 0x7fef040c2920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:49492/0 (socket says 172.21.15.195:49492) 2022-06-27T11:51:34.884 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:34.878+0000 7fef08d41700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fef0406e850 0x7fef041570a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:43302/0 (socket says 172.21.15.195:43302) 2022-06-27T11:51:34.888 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:34.878+0000 7fef09542700 1 -- 172.21.15.195:0/1842994824 learned_addr learned my addr 172.21.15.195:0/1842994824 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:51:34.888 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:34.878+0000 7fef03fff700 1 --2- 172.21.15.195:0/1842994824 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fef0406fef0 0x7fef04069de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:34.889 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:34.878+0000 7fef08d41700 1 -- 172.21.15.195:0/1842994824 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fef0406fef0 msgr2=0x7fef04069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:51:34.889 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:34.878+0000 7fef08d41700 1 --2- 172.21.15.195:0/1842994824 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fef0406fef0 0x7fef04069de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:34.900 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:34.878+0000 7fef08d41700 1 -- 172.21.15.195:0/1842994824 >> v1:172.21.15.195:6789/0 conn(0x7fef040c34a0 legacy=0x7fef040c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:34.901 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:34.878+0000 7fef08d41700 1 -- 172.21.15.195:0/1842994824 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef0406f8c0 con 0x7fef0406e850 2022-06-27T11:51:34.901 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:34.878+0000 7fef037fe700 1 -- 172.21.15.195:0/1842994824 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3946445933 0 0) 0x7fef0406fba0 con 0x7fef040c34a0 2022-06-27T11:51:34.902 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:34.878+0000 7fef08d41700 1 --2- 172.21.15.195:0/1842994824 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fef0406e850 0x7fef041570a0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7feef80097a0 tx=0x7feef8009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5e3c876037260f server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:51:34.902 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:34.879+0000 7fef037fe700 1 -- 172.21.15.195:0/1842994824 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feef8011630 con 0x7fef0406e850 2022-06-27T11:51:34.903 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:34.879+0000 7fef037fe700 1 -- 172.21.15.195:0/1842994824 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feef8011790 con 0x7fef0406e850 2022-06-27T11:51:34.903 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:34.879+0000 7fef037fe700 1 -- 172.21.15.195:0/1842994824 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feef8011b30 con 0x7fef0406e850 2022-06-27T11:51:34.904 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:34.879+0000 7fef0afa5700 1 -- 172.21.15.195:0/1842994824 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fef0406e850 msgr2=0x7fef041570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:34.904 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:34.879+0000 7fef0afa5700 1 --2- 172.21.15.195:0/1842994824 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fef0406e850 0x7fef041570a0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7feef80097a0 tx=0x7feef8009060 comp rx=0 tx=0).stop 2022-06-27T11:51:34.905 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:34.879+0000 7fef0afa5700 1 -- 172.21.15.195:0/1842994824 shutdown_connections 2022-06-27T11:51:34.905 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:34.879+0000 7fef0afa5700 1 --2- 172.21.15.195:0/1842994824 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fef0406e850 0x7fef041570a0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:34.916 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:34.879+0000 7fef0afa5700 1 --2- 172.21.15.195:0/1842994824 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fef0406fef0 0x7fef04069de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:34.917 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:34.879+0000 7fef0afa5700 1 -- 172.21.15.195:0/1842994824 >> 172.21.15.195:0/1842994824 conn(0x7fef040c6350 msgr2=0x7fef040c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:51:34.917 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:34.879+0000 7fef0afa5700 1 -- 172.21.15.195:0/1842994824 shutdown_connections 2022-06-27T11:51:34.918 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:34.879+0000 7fef0afa5700 1 -- 172.21.15.195:0/1842994824 wait complete. 2022-06-27T11:51:34.919 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:34.880+0000 7fef0afa5700 1 Processor -- start 2022-06-27T11:51:34.919 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:34.881+0000 7fef0afa5700 1 -- start start 2022-06-27T11:51:34.920 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:34.881+0000 7fef0afa5700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fef0406e850 0x7fef0414e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:34.921 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:34.881+0000 7fef0afa5700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fef040c34a0 0x7fef040c28d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:34.921 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:34.881+0000 7fef0afa5700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fef040c03e0 0x7fef04162940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:34.922 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:34.881+0000 7fef0afa5700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef04162440 con 0x7fef040c34a0 2022-06-27T11:51:34.922 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:34.881+0000 7fef0afa5700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef040bd6e0 con 0x7fef040c03e0 2022-06-27T11:51:34.923 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:34.881+0000 7fef0afa5700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef040bd9c0 con 0x7fef0406e850 2022-06-27T11:51:34.923 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:34.881+0000 7fef03fff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fef040c34a0 0x7fef040c28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:34.924 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:34.882+0000 7fef03fff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fef040c34a0 0x7fef040c28d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:43308/0 (socket says 172.21.15.195:43308) 2022-06-27T11:51:34.924 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:34.882+0000 7fef03fff700 1 -- 172.21.15.195:0/984017218 learned_addr learned my addr 172.21.15.195:0/984017218 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:51:34.924 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:34.882+0000 7fef08d41700 1 --2- 172.21.15.195:0/984017218 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fef0406e850 0x7fef0414e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:34.925 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:34.882+0000 7fef09542700 1 --2- 172.21.15.195:0/984017218 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fef040c03e0 0x7fef04162940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:34.925 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:34.882+0000 7fef03fff700 1 -- 172.21.15.195:0/984017218 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fef0406e850 msgr2=0x7fef0414e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:34.927 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:34.882+0000 7fef03fff700 1 --2- 172.21.15.195:0/984017218 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fef0406e850 0x7fef0414e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:34.927 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:34.882+0000 7fef03fff700 1 -- 172.21.15.195:0/984017218 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fef040c03e0 msgr2=0x7fef04162940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:34.928 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:34.882+0000 7fef03fff700 1 --2- 172.21.15.195:0/984017218 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fef040c03e0 0x7fef04162940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:34.928 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:34.882+0000 7fef03fff700 1 -- 172.21.15.195:0/984017218 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feef80092e0 con 0x7fef040c34a0 2022-06-27T11:51:34.929 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:34.882+0000 7fef03fff700 1 --2- 172.21.15.195:0/984017218 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fef040c34a0 0x7fef040c28d0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7feef000e900 tx=0x7feef0009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:51:34.929 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:34.882+0000 7fef00ff9700 1 -- 172.21.15.195:0/984017218 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feef0013680 con 0x7fef040c34a0 2022-06-27T11:51:34.930 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:34.882+0000 7fef00ff9700 1 -- 172.21.15.195:0/984017218 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feef00137e0 con 0x7fef040c34a0 2022-06-27T11:51:34.930 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:34.882+0000 7fef00ff9700 1 -- 172.21.15.195:0/984017218 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feef0013b80 con 0x7fef040c34a0 2022-06-27T11:51:34.931 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:34.882+0000 7fef0afa5700 1 -- 172.21.15.195:0/984017218 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef040bdca0 con 0x7fef040c34a0 2022-06-27T11:51:34.931 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:34.882+0000 7fef0afa5700 1 -- 172.21.15.195:0/984017218 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef040be260 con 0x7fef040c34a0 2022-06-27T11:51:34.932 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:34.883+0000 7fef0afa5700 1 -- 172.21.15.195:0/984017218 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef04162660 con 0x7fef040c34a0 2022-06-27T11:51:34.933 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:34.883+0000 7fef00ff9700 1 -- 172.21.15.195:0/984017218 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7feef0019030 con 0x7fef040c34a0 2022-06-27T11:51:34.933 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:34.884+0000 7fef00ff9700 1 --2- 172.21.15.195:0/984017218 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7feeec063d00 0x7feeec0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:34.933 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:34.884+0000 7fef00ff9700 1 -- 172.21.15.195:0/984017218 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6540+0+0 (secure 0 0 0) 0x7feef0080790 con 0x7fef040c34a0 2022-06-27T11:51:34.934 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:34.884+0000 7fef08d41700 1 --2- 172.21.15.195:0/984017218 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7feeec063d00 0x7feeec0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:34.934 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:34.884+0000 7fef08d41700 1 --2- 172.21.15.195:0/984017218 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7feeec063d00 0x7feeec0661b0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7feef8009260 tx=0x7feef8017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:51:34.935 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:34.888+0000 7fef00ff9700 1 -- 172.21.15.195:0/984017218 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7feef004f1a0 con 0x7fef040c34a0 2022-06-27T11:51:35.116 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:35.114+0000 7fef0afa5700 1 -- 172.21.15.195:0/984017218 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef040be480 con 0x7fef040c34a0 2022-06-27T11:51:35.117 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:35.115+0000 7fef00ff9700 1 -- 172.21.15.195:0/984017218 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+5368 (secure 0 0 0) 0x7feef0017090 con 0x7fef040c34a0 2022-06-27T11:51:35.118 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:51:35.128 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:51:33.013206+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4849},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4849":{"gid":4849,"name":"d","rank":0,"incarnation":15,"state":"up:active","state_seq":19,"addr":"172.21.15.202:6835/3271891211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":3271891211},{"type":"v1","addr":"172.21.15.202:6835","nonce":3271891211}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:51:35.134 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:35.117+0000 7fef0afa5700 1 -- 172.21.15.195:0/984017218 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7feeec063d00 msgr2=0x7feeec0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:35.135 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:35.117+0000 7fef0afa5700 1 --2- 172.21.15.195:0/984017218 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7feeec063d00 0x7feeec0661b0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7feef8009260 tx=0x7feef8017000 comp rx=0 tx=0).stop 2022-06-27T11:51:35.135 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:35.117+0000 7fef0afa5700 1 -- 172.21.15.195:0/984017218 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fef040c34a0 msgr2=0x7fef040c28d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:35.136 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:35.117+0000 7fef0afa5700 1 --2- 172.21.15.195:0/984017218 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fef040c34a0 0x7fef040c28d0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7feef000e900 tx=0x7feef0009060 comp rx=0 tx=0).stop 2022-06-27T11:51:35.136 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:35.117+0000 7fef0afa5700 1 -- 172.21.15.195:0/984017218 shutdown_connections 2022-06-27T11:51:35.141 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:35.117+0000 7fef0afa5700 1 --2- 172.21.15.195:0/984017218 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7feeec063d00 0x7feeec0661b0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:35.141 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:35.117+0000 7fef0afa5700 1 --2- 172.21.15.195:0/984017218 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fef040c03e0 0x7fef04162940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:35.144 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:35.117+0000 7fef0afa5700 1 --2- 172.21.15.195:0/984017218 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fef0406e850 0x7fef0414e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:35.144 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:35.117+0000 7fef0afa5700 1 --2- 172.21.15.195:0/984017218 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fef040c34a0 0x7fef040c28d0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:35.145 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:35.117+0000 7fef0afa5700 1 -- 172.21.15.195:0/984017218 >> 172.21.15.195:0/984017218 conn(0x7fef040c6350 msgr2=0x7fef04150670 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:51:35.145 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:35.117+0000 7fef0afa5700 1 -- 172.21.15.195:0/984017218 shutdown_connections 2022-06-27T11:51:35.146 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:35.117+0000 7fef0afa5700 1 -- 172.21.15.195:0/984017218 wait complete. 2022-06-27T11:51:35.146 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 18 2022-06-27T11:51:35.147 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d has gained rank=0, replacing gid=4287 2022-06-27T11:51:35.148 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 7 secs before reviving mds.b 2022-06-27T11:51:39.067 INFO:tasks.ceph.mds.d.smithi202.stderr:2022-06-27T11:51:39.068+0000 7f640673f700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2022-06-27T11:51:39.121 INFO:tasks.workunit.client.0.smithi195.stdout:removed './100/19' 2022-06-27T11:51:39.121 INFO:tasks.workunit.client.0.smithi195.stdout:removed './100/15' 2022-06-27T11:51:39.122 INFO:tasks.workunit.client.0.smithi195.stdout:removed './100/31' 2022-06-27T11:51:39.122 INFO:tasks.workunit.client.0.smithi195.stdout:removed './100/36' 2022-06-27T11:51:39.123 INFO:tasks.workunit.client.0.smithi195.stdout:removed './100/20' 2022-06-27T11:51:39.123 INFO:tasks.workunit.client.0.smithi195.stdout:removed './100/12' 2022-06-27T11:51:39.123 INFO:tasks.workunit.client.0.smithi195.stdout:removed './100/39' 2022-06-27T11:51:39.124 INFO:tasks.workunit.client.0.smithi195.stdout:removed './100/10' 2022-06-27T11:51:39.124 INFO:tasks.workunit.client.0.smithi195.stdout:removed './100/34' 2022-06-27T11:51:39.124 INFO:tasks.workunit.client.0.smithi195.stdout:removed './100/38' 2022-06-27T11:51:39.124 INFO:tasks.workunit.client.0.smithi195.stdout:removed './100/11' 2022-06-27T11:51:39.125 INFO:tasks.workunit.client.0.smithi195.stdout:removed './100/16' 2022-06-27T11:51:39.125 INFO:tasks.workunit.client.0.smithi195.stdout:removed './100/32' 2022-06-27T11:51:39.125 INFO:tasks.workunit.client.0.smithi195.stdout:removed './100/30' 2022-06-27T11:51:39.125 INFO:tasks.workunit.client.0.smithi195.stdout:removed './100/14' 2022-06-27T11:51:39.126 INFO:tasks.workunit.client.0.smithi195.stdout:removed './100/18' 2022-06-27T11:51:39.126 INFO:tasks.workunit.client.0.smithi195.stdout:removed './100/13' 2022-06-27T11:51:39.126 INFO:tasks.workunit.client.0.smithi195.stdout:removed './100/33' 2022-06-27T11:51:39.126 INFO:tasks.workunit.client.0.smithi195.stdout:removed './100/17' 2022-06-27T11:51:39.127 INFO:tasks.workunit.client.0.smithi195.stdout:removed './100/35' 2022-06-27T11:51:39.127 INFO:tasks.workunit.client.0.smithi195.stdout:removed './100/37' 2022-06-27T11:51:39.127 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './100' 2022-06-27T11:51:39.127 INFO:tasks.workunit.client.0.smithi195.stdout:removed './101/19' 2022-06-27T11:51:39.128 INFO:tasks.workunit.client.0.smithi195.stdout:removed './101/15' 2022-06-27T11:51:39.128 INFO:tasks.workunit.client.0.smithi195.stdout:removed './101/31' 2022-06-27T11:51:39.128 INFO:tasks.workunit.client.0.smithi195.stdout:removed './101/36' 2022-06-27T11:51:39.129 INFO:tasks.workunit.client.0.smithi195.stdout:removed './101/20' 2022-06-27T11:51:39.129 INFO:tasks.workunit.client.0.smithi195.stdout:removed './101/12' 2022-06-27T11:51:39.129 INFO:tasks.workunit.client.0.smithi195.stdout:removed './101/39' 2022-06-27T11:51:39.130 INFO:tasks.workunit.client.0.smithi195.stdout:removed './101/10' 2022-06-27T11:51:39.131 INFO:tasks.workunit.client.0.smithi195.stdout:removed './101/34' 2022-06-27T11:51:39.131 INFO:tasks.workunit.client.0.smithi195.stdout:removed './101/38' 2022-06-27T11:51:39.131 INFO:tasks.workunit.client.0.smithi195.stdout:removed './101/11' 2022-06-27T11:51:39.131 INFO:tasks.workunit.client.0.smithi195.stdout:removed './101/16' 2022-06-27T11:51:39.132 INFO:tasks.workunit.client.0.smithi195.stdout:removed './101/32' 2022-06-27T11:51:39.132 INFO:tasks.workunit.client.0.smithi195.stdout:removed './101/30' 2022-06-27T11:51:39.132 INFO:tasks.workunit.client.0.smithi195.stdout:removed './101/14' 2022-06-27T11:51:39.132 INFO:tasks.workunit.client.0.smithi195.stdout:removed './101/18' 2022-06-27T11:51:39.133 INFO:tasks.workunit.client.0.smithi195.stdout:removed './101/13' 2022-06-27T11:51:39.133 INFO:tasks.workunit.client.0.smithi195.stdout:removed './101/33' 2022-06-27T11:51:39.133 INFO:tasks.workunit.client.0.smithi195.stdout:removed './101/17' 2022-06-27T11:51:39.133 INFO:tasks.workunit.client.0.smithi195.stdout:removed './101/35' 2022-06-27T11:51:39.134 INFO:tasks.workunit.client.0.smithi195.stdout:removed './101/37' 2022-06-27T11:51:39.134 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './101' 2022-06-27T11:51:39.134 INFO:tasks.workunit.client.0.smithi195.stdout:removed './102/19' 2022-06-27T11:51:39.135 INFO:tasks.workunit.client.0.smithi195.stdout:removed './102/15' 2022-06-27T11:51:39.135 INFO:tasks.workunit.client.0.smithi195.stdout:removed './102/31' 2022-06-27T11:51:39.135 INFO:tasks.workunit.client.0.smithi195.stdout:removed './102/36' 2022-06-27T11:51:39.136 INFO:tasks.workunit.client.0.smithi195.stdout:removed './102/20' 2022-06-27T11:51:39.136 INFO:tasks.workunit.client.0.smithi195.stdout:removed './102/12' 2022-06-27T11:51:39.137 INFO:tasks.workunit.client.0.smithi195.stdout:removed './102/39' 2022-06-27T11:51:39.137 INFO:tasks.workunit.client.0.smithi195.stdout:removed './102/10' 2022-06-27T11:51:39.137 INFO:tasks.workunit.client.0.smithi195.stdout:removed './102/34' 2022-06-27T11:51:39.138 INFO:tasks.workunit.client.0.smithi195.stdout:removed './102/38' 2022-06-27T11:51:39.138 INFO:tasks.workunit.client.0.smithi195.stdout:removed './102/11' 2022-06-27T11:51:39.138 INFO:tasks.workunit.client.0.smithi195.stdout:removed './102/16' 2022-06-27T11:51:39.139 INFO:tasks.workunit.client.0.smithi195.stdout:removed './102/32' 2022-06-27T11:51:39.139 INFO:tasks.workunit.client.0.smithi195.stdout:removed './102/30' 2022-06-27T11:51:39.140 INFO:tasks.workunit.client.0.smithi195.stdout:removed './102/14' 2022-06-27T11:51:39.140 INFO:tasks.workunit.client.0.smithi195.stdout:removed './102/18' 2022-06-27T11:51:39.141 INFO:tasks.workunit.client.0.smithi195.stdout:removed './102/13' 2022-06-27T11:51:39.141 INFO:tasks.workunit.client.0.smithi195.stdout:removed './102/33' 2022-06-27T11:51:39.142 INFO:tasks.workunit.client.0.smithi195.stdout:removed './102/17' 2022-06-27T11:51:39.142 INFO:tasks.workunit.client.0.smithi195.stdout:removed './102/35' 2022-06-27T11:51:39.142 INFO:tasks.workunit.client.0.smithi195.stdout:removed './102/37' 2022-06-27T11:51:39.142 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './102' 2022-06-27T11:51:39.143 INFO:tasks.workunit.client.0.smithi195.stdout:removed './103/19' 2022-06-27T11:51:39.143 INFO:tasks.workunit.client.0.smithi195.stdout:removed './103/15' 2022-06-27T11:51:39.143 INFO:tasks.workunit.client.0.smithi195.stdout:removed './103/31' 2022-06-27T11:51:39.143 INFO:tasks.workunit.client.0.smithi195.stdout:removed './103/36' 2022-06-27T11:51:39.144 INFO:tasks.workunit.client.0.smithi195.stdout:removed './103/20' 2022-06-27T11:51:39.144 INFO:tasks.workunit.client.0.smithi195.stdout:removed './103/12' 2022-06-27T11:51:39.144 INFO:tasks.workunit.client.0.smithi195.stdout:removed './103/39' 2022-06-27T11:51:39.144 INFO:tasks.workunit.client.0.smithi195.stdout:removed './103/10' 2022-06-27T11:51:39.144 INFO:tasks.workunit.client.0.smithi195.stdout:removed './103/34' 2022-06-27T11:51:39.145 INFO:tasks.workunit.client.0.smithi195.stdout:removed './103/38' 2022-06-27T11:51:39.145 INFO:tasks.workunit.client.0.smithi195.stdout:removed './103/11' 2022-06-27T11:51:39.145 INFO:tasks.workunit.client.0.smithi195.stdout:removed './103/16' 2022-06-27T11:51:39.145 INFO:tasks.workunit.client.0.smithi195.stdout:removed './103/32' 2022-06-27T11:51:39.146 INFO:tasks.workunit.client.0.smithi195.stdout:removed './103/30' 2022-06-27T11:51:39.146 INFO:tasks.workunit.client.0.smithi195.stdout:removed './103/14' 2022-06-27T11:51:39.146 INFO:tasks.workunit.client.0.smithi195.stdout:removed './103/18' 2022-06-27T11:51:39.146 INFO:tasks.workunit.client.0.smithi195.stdout:removed './103/13' 2022-06-27T11:51:39.146 INFO:tasks.workunit.client.0.smithi195.stdout:removed './103/33' 2022-06-27T11:51:39.147 INFO:tasks.workunit.client.0.smithi195.stdout:removed './103/17' 2022-06-27T11:51:39.147 INFO:tasks.workunit.client.0.smithi195.stdout:removed './103/35' 2022-06-27T11:51:39.147 INFO:tasks.workunit.client.0.smithi195.stdout:removed './103/37' 2022-06-27T11:51:39.147 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './103' 2022-06-27T11:51:39.148 INFO:tasks.workunit.client.0.smithi195.stdout:removed './104/19' 2022-06-27T11:51:39.148 INFO:tasks.workunit.client.0.smithi195.stdout:removed './104/15' 2022-06-27T11:51:39.148 INFO:tasks.workunit.client.0.smithi195.stdout:removed './104/31' 2022-06-27T11:51:39.148 INFO:tasks.workunit.client.0.smithi195.stdout:removed './104/36' 2022-06-27T11:51:39.148 INFO:tasks.workunit.client.0.smithi195.stdout:removed './104/20' 2022-06-27T11:51:39.149 INFO:tasks.workunit.client.0.smithi195.stdout:removed './104/12' 2022-06-27T11:51:39.149 INFO:tasks.workunit.client.0.smithi195.stdout:removed './104/39' 2022-06-27T11:51:39.149 INFO:tasks.workunit.client.0.smithi195.stdout:removed './104/10' 2022-06-27T11:51:39.149 INFO:tasks.workunit.client.0.smithi195.stdout:removed './104/34' 2022-06-27T11:51:39.150 INFO:tasks.workunit.client.0.smithi195.stdout:removed './104/38' 2022-06-27T11:51:39.150 INFO:tasks.workunit.client.0.smithi195.stdout:removed './104/11' 2022-06-27T11:51:39.150 INFO:tasks.workunit.client.0.smithi195.stdout:removed './104/16' 2022-06-27T11:51:39.152 INFO:tasks.workunit.client.0.smithi195.stdout:removed './104/32' 2022-06-27T11:51:39.152 INFO:tasks.workunit.client.0.smithi195.stdout:removed './104/30' 2022-06-27T11:51:39.153 INFO:tasks.workunit.client.0.smithi195.stdout:removed './104/14' 2022-06-27T11:51:39.153 INFO:tasks.workunit.client.0.smithi195.stdout:removed './104/18' 2022-06-27T11:51:39.156 INFO:tasks.workunit.client.0.smithi195.stdout:removed './104/13' 2022-06-27T11:51:39.156 INFO:tasks.workunit.client.0.smithi195.stdout:removed './104/33' 2022-06-27T11:51:39.156 INFO:tasks.workunit.client.0.smithi195.stdout:removed './104/17' 2022-06-27T11:51:39.157 INFO:tasks.workunit.client.0.smithi195.stdout:removed './104/35' 2022-06-27T11:51:39.157 INFO:tasks.workunit.client.0.smithi195.stdout:removed './104/37' 2022-06-27T11:51:39.158 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './104' 2022-06-27T11:51:39.158 INFO:tasks.workunit.client.0.smithi195.stdout:removed './105/19' 2022-06-27T11:51:39.159 INFO:tasks.workunit.client.0.smithi195.stdout:removed './105/15' 2022-06-27T11:51:39.159 INFO:tasks.workunit.client.0.smithi195.stdout:removed './105/31' 2022-06-27T11:51:39.159 INFO:tasks.workunit.client.0.smithi195.stdout:removed './105/36' 2022-06-27T11:51:39.161 INFO:tasks.workunit.client.0.smithi195.stdout:removed './105/20' 2022-06-27T11:51:39.161 INFO:tasks.workunit.client.0.smithi195.stdout:removed './105/12' 2022-06-27T11:51:39.161 INFO:tasks.workunit.client.0.smithi195.stdout:removed './105/39' 2022-06-27T11:51:39.162 INFO:tasks.workunit.client.0.smithi195.stdout:removed './105/10' 2022-06-27T11:51:39.162 INFO:tasks.workunit.client.0.smithi195.stdout:removed './105/34' 2022-06-27T11:51:39.163 INFO:tasks.workunit.client.0.smithi195.stdout:removed './105/38' 2022-06-27T11:51:39.163 INFO:tasks.workunit.client.0.smithi195.stdout:removed './105/11' 2022-06-27T11:51:39.163 INFO:tasks.workunit.client.0.smithi195.stdout:removed './105/16' 2022-06-27T11:51:39.164 INFO:tasks.workunit.client.0.smithi195.stdout:removed './105/32' 2022-06-27T11:51:39.164 INFO:tasks.workunit.client.0.smithi195.stdout:removed './105/30' 2022-06-27T11:51:39.165 INFO:tasks.workunit.client.0.smithi195.stdout:removed './105/14' 2022-06-27T11:51:39.165 INFO:tasks.workunit.client.0.smithi195.stdout:removed './105/18' 2022-06-27T11:51:39.166 INFO:tasks.workunit.client.0.smithi195.stdout:removed './105/13' 2022-06-27T11:51:39.166 INFO:tasks.workunit.client.0.smithi195.stdout:removed './105/33' 2022-06-27T11:51:39.167 INFO:tasks.workunit.client.0.smithi195.stdout:removed './105/17' 2022-06-27T11:51:39.167 INFO:tasks.workunit.client.0.smithi195.stdout:removed './105/35' 2022-06-27T11:51:39.168 INFO:tasks.workunit.client.0.smithi195.stdout:removed './105/37' 2022-06-27T11:51:39.168 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './105' 2022-06-27T11:51:39.169 INFO:tasks.workunit.client.0.smithi195.stdout:removed './106/19' 2022-06-27T11:51:39.169 INFO:tasks.workunit.client.0.smithi195.stdout:removed './106/15' 2022-06-27T11:51:39.169 INFO:tasks.workunit.client.0.smithi195.stdout:removed './106/31' 2022-06-27T11:51:39.170 INFO:tasks.workunit.client.0.smithi195.stdout:removed './106/36' 2022-06-27T11:51:39.170 INFO:tasks.workunit.client.0.smithi195.stdout:removed './106/20' 2022-06-27T11:51:39.171 INFO:tasks.workunit.client.0.smithi195.stdout:removed './106/12' 2022-06-27T11:51:39.171 INFO:tasks.workunit.client.0.smithi195.stdout:removed './106/39' 2022-06-27T11:51:39.172 INFO:tasks.workunit.client.0.smithi195.stdout:removed './106/10' 2022-06-27T11:51:39.172 INFO:tasks.workunit.client.0.smithi195.stdout:removed './106/34' 2022-06-27T11:51:39.173 INFO:tasks.workunit.client.0.smithi195.stdout:removed './106/38' 2022-06-27T11:51:39.173 INFO:tasks.workunit.client.0.smithi195.stdout:removed './106/11' 2022-06-27T11:51:39.173 INFO:tasks.workunit.client.0.smithi195.stdout:removed './106/16' 2022-06-27T11:51:39.174 INFO:tasks.workunit.client.0.smithi195.stdout:removed './106/32' 2022-06-27T11:51:39.175 INFO:tasks.workunit.client.0.smithi195.stdout:removed './106/30' 2022-06-27T11:51:39.176 INFO:tasks.workunit.client.0.smithi195.stdout:removed './106/14' 2022-06-27T11:51:39.176 INFO:tasks.workunit.client.0.smithi195.stdout:removed './106/18' 2022-06-27T11:51:39.177 INFO:tasks.workunit.client.0.smithi195.stdout:removed './106/13' 2022-06-27T11:51:39.177 INFO:tasks.workunit.client.0.smithi195.stdout:removed './106/33' 2022-06-27T11:51:39.177 INFO:tasks.workunit.client.0.smithi195.stdout:removed './106/17' 2022-06-27T11:51:39.178 INFO:tasks.workunit.client.0.smithi195.stdout:removed './106/35' 2022-06-27T11:51:39.178 INFO:tasks.workunit.client.0.smithi195.stdout:removed './106/37' 2022-06-27T11:51:39.178 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './106' 2022-06-27T11:51:39.179 INFO:tasks.workunit.client.0.smithi195.stdout:removed './107/19' 2022-06-27T11:51:39.179 INFO:tasks.workunit.client.0.smithi195.stdout:removed './107/15' 2022-06-27T11:51:39.180 INFO:tasks.workunit.client.0.smithi195.stdout:removed './107/31' 2022-06-27T11:51:39.180 INFO:tasks.workunit.client.0.smithi195.stdout:removed './107/36' 2022-06-27T11:51:39.180 INFO:tasks.workunit.client.0.smithi195.stdout:removed './107/20' 2022-06-27T11:51:39.181 INFO:tasks.workunit.client.0.smithi195.stdout:removed './107/12' 2022-06-27T11:51:39.181 INFO:tasks.workunit.client.0.smithi195.stdout:removed './107/39' 2022-06-27T11:51:39.182 INFO:tasks.workunit.client.0.smithi195.stdout:removed './107/10' 2022-06-27T11:51:39.182 INFO:tasks.workunit.client.0.smithi195.stdout:removed './107/34' 2022-06-27T11:51:39.182 INFO:tasks.workunit.client.0.smithi195.stdout:removed './107/38' 2022-06-27T11:51:39.183 INFO:tasks.workunit.client.0.smithi195.stdout:removed './107/11' 2022-06-27T11:51:39.183 INFO:tasks.workunit.client.0.smithi195.stdout:removed './107/16' 2022-06-27T11:51:39.184 INFO:tasks.workunit.client.0.smithi195.stdout:removed './107/32' 2022-06-27T11:51:39.184 INFO:tasks.workunit.client.0.smithi195.stdout:removed './107/30' 2022-06-27T11:51:39.184 INFO:tasks.workunit.client.0.smithi195.stdout:removed './107/14' 2022-06-27T11:51:39.185 INFO:tasks.workunit.client.0.smithi195.stdout:removed './107/18' 2022-06-27T11:51:39.185 INFO:tasks.workunit.client.0.smithi195.stdout:removed './107/13' 2022-06-27T11:51:39.185 INFO:tasks.workunit.client.0.smithi195.stdout:removed './107/33' 2022-06-27T11:51:39.186 INFO:tasks.workunit.client.0.smithi195.stdout:removed './107/17' 2022-06-27T11:51:39.186 INFO:tasks.workunit.client.0.smithi195.stdout:removed './107/35' 2022-06-27T11:51:39.187 INFO:tasks.workunit.client.0.smithi195.stdout:removed './107/37' 2022-06-27T11:51:39.187 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './107' 2022-06-27T11:51:39.188 INFO:tasks.workunit.client.0.smithi195.stdout:removed './108/19' 2022-06-27T11:51:39.188 INFO:tasks.workunit.client.0.smithi195.stdout:removed './108/15' 2022-06-27T11:51:39.189 INFO:tasks.workunit.client.0.smithi195.stdout:removed './108/31' 2022-06-27T11:51:39.189 INFO:tasks.workunit.client.0.smithi195.stdout:removed './108/36' 2022-06-27T11:51:39.189 INFO:tasks.workunit.client.0.smithi195.stdout:removed './108/20' 2022-06-27T11:51:39.190 INFO:tasks.workunit.client.0.smithi195.stdout:removed './108/12' 2022-06-27T11:51:39.190 INFO:tasks.workunit.client.0.smithi195.stdout:removed './108/39' 2022-06-27T11:51:39.191 INFO:tasks.workunit.client.0.smithi195.stdout:removed './108/10' 2022-06-27T11:51:39.191 INFO:tasks.workunit.client.0.smithi195.stdout:removed './108/34' 2022-06-27T11:51:39.192 INFO:tasks.workunit.client.0.smithi195.stdout:removed './108/38' 2022-06-27T11:51:39.192 INFO:tasks.workunit.client.0.smithi195.stdout:removed './108/11' 2022-06-27T11:51:39.193 INFO:tasks.workunit.client.0.smithi195.stdout:removed './108/16' 2022-06-27T11:51:39.193 INFO:tasks.workunit.client.0.smithi195.stdout:removed './108/32' 2022-06-27T11:51:39.194 INFO:tasks.workunit.client.0.smithi195.stdout:removed './108/30' 2022-06-27T11:51:39.194 INFO:tasks.workunit.client.0.smithi195.stdout:removed './108/14' 2022-06-27T11:51:39.194 INFO:tasks.workunit.client.0.smithi195.stdout:removed './108/18' 2022-06-27T11:51:39.195 INFO:tasks.workunit.client.0.smithi195.stdout:removed './108/13' 2022-06-27T11:51:39.195 INFO:tasks.workunit.client.0.smithi195.stdout:removed './108/33' 2022-06-27T11:51:39.196 INFO:tasks.workunit.client.0.smithi195.stdout:removed './108/17' 2022-06-27T11:51:39.196 INFO:tasks.workunit.client.0.smithi195.stdout:removed './108/35' 2022-06-27T11:51:39.196 INFO:tasks.workunit.client.0.smithi195.stdout:removed './108/37' 2022-06-27T11:51:39.197 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './108' 2022-06-27T11:51:39.197 INFO:tasks.workunit.client.0.smithi195.stdout:removed './109/19' 2022-06-27T11:51:39.198 INFO:tasks.workunit.client.0.smithi195.stdout:removed './109/15' 2022-06-27T11:51:39.198 INFO:tasks.workunit.client.0.smithi195.stdout:removed './109/31' 2022-06-27T11:51:39.198 INFO:tasks.workunit.client.0.smithi195.stdout:removed './109/36' 2022-06-27T11:51:39.199 INFO:tasks.workunit.client.0.smithi195.stdout:removed './109/20' 2022-06-27T11:51:39.199 INFO:tasks.workunit.client.0.smithi195.stdout:removed './109/12' 2022-06-27T11:51:39.200 INFO:tasks.workunit.client.0.smithi195.stdout:removed './109/39' 2022-06-27T11:51:39.200 INFO:tasks.workunit.client.0.smithi195.stdout:removed './109/10' 2022-06-27T11:51:39.200 INFO:tasks.workunit.client.0.smithi195.stdout:removed './109/34' 2022-06-27T11:51:39.201 INFO:tasks.workunit.client.0.smithi195.stdout:removed './109/38' 2022-06-27T11:51:39.201 INFO:tasks.workunit.client.0.smithi195.stdout:removed './109/11' 2022-06-27T11:51:39.201 INFO:tasks.workunit.client.0.smithi195.stdout:removed './109/16' 2022-06-27T11:51:39.201 INFO:tasks.workunit.client.0.smithi195.stdout:removed './109/32' 2022-06-27T11:51:39.201 INFO:tasks.workunit.client.0.smithi195.stdout:removed './109/30' 2022-06-27T11:51:42.150 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2022-06-27T11:51:42.150 INFO:tasks.ceph.mds.b:Restarting daemon 2022-06-27T11:51:42.151 DEBUG:teuthology.orchestra.run.smithi202:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-06-27T11:51:42.153 INFO:tasks.ceph.mds.b:Started 2022-06-27T11:51:42.153 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:51:42.245 INFO:tasks.workunit.client.0.smithi195.stdout:removed './109/14' 2022-06-27T11:51:42.246 INFO:tasks.workunit.client.0.smithi195.stdout:removed './109/18' 2022-06-27T11:51:42.246 INFO:tasks.workunit.client.0.smithi195.stdout:removed './109/13' 2022-06-27T11:51:42.247 INFO:tasks.workunit.client.0.smithi195.stdout:removed './109/33' 2022-06-27T11:51:42.247 INFO:tasks.workunit.client.0.smithi195.stdout:removed './109/17' 2022-06-27T11:51:42.248 INFO:tasks.workunit.client.0.smithi195.stdout:removed './109/35' 2022-06-27T11:51:42.248 INFO:tasks.workunit.client.0.smithi195.stdout:removed './109/37' 2022-06-27T11:51:42.249 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './109' 2022-06-27T11:51:42.249 INFO:tasks.workunit.client.0.smithi195.stdout:removed './110/19' 2022-06-27T11:51:42.250 INFO:tasks.workunit.client.0.smithi195.stdout:removed './110/15' 2022-06-27T11:51:42.250 INFO:tasks.workunit.client.0.smithi195.stdout:removed './110/31' 2022-06-27T11:51:42.250 INFO:tasks.workunit.client.0.smithi195.stdout:removed './110/36' 2022-06-27T11:51:42.251 INFO:tasks.workunit.client.0.smithi195.stdout:removed './110/20' 2022-06-27T11:51:42.251 INFO:tasks.workunit.client.0.smithi195.stdout:removed './110/12' 2022-06-27T11:51:42.252 INFO:tasks.workunit.client.0.smithi195.stdout:removed './110/39' 2022-06-27T11:51:42.252 INFO:tasks.workunit.client.0.smithi195.stdout:removed './110/10' 2022-06-27T11:51:42.254 INFO:tasks.workunit.client.0.smithi195.stdout:removed './110/34' 2022-06-27T11:51:42.254 INFO:tasks.workunit.client.0.smithi195.stdout:removed './110/38' 2022-06-27T11:51:42.255 INFO:tasks.workunit.client.0.smithi195.stdout:removed './110/11' 2022-06-27T11:51:42.255 INFO:tasks.workunit.client.0.smithi195.stdout:removed './110/16' 2022-06-27T11:51:42.256 INFO:tasks.workunit.client.0.smithi195.stdout:removed './110/32' 2022-06-27T11:51:42.256 INFO:tasks.workunit.client.0.smithi195.stdout:removed './110/30' 2022-06-27T11:51:42.257 INFO:tasks.workunit.client.0.smithi195.stdout:removed './110/14' 2022-06-27T11:51:42.257 INFO:tasks.workunit.client.0.smithi195.stdout:removed './110/18' 2022-06-27T11:51:42.258 INFO:tasks.workunit.client.0.smithi195.stdout:removed './110/13' 2022-06-27T11:51:42.258 INFO:tasks.workunit.client.0.smithi195.stdout:removed './110/33' 2022-06-27T11:51:42.259 INFO:tasks.workunit.client.0.smithi195.stdout:removed './110/17' 2022-06-27T11:51:42.259 INFO:tasks.workunit.client.0.smithi195.stdout:removed './110/35' 2022-06-27T11:51:42.260 INFO:tasks.workunit.client.0.smithi195.stdout:removed './110/37' 2022-06-27T11:51:42.260 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './110' 2022-06-27T11:51:42.261 INFO:tasks.workunit.client.0.smithi195.stdout:removed './111/19' 2022-06-27T11:51:42.262 INFO:tasks.workunit.client.0.smithi195.stdout:removed './111/15' 2022-06-27T11:51:42.262 INFO:tasks.workunit.client.0.smithi195.stdout:removed './111/31' 2022-06-27T11:51:42.263 INFO:tasks.workunit.client.0.smithi195.stdout:removed './111/36' 2022-06-27T11:51:42.265 INFO:tasks.workunit.client.0.smithi195.stdout:removed './111/20' 2022-06-27T11:51:42.265 INFO:tasks.workunit.client.0.smithi195.stdout:removed './111/12' 2022-06-27T11:51:42.266 INFO:tasks.workunit.client.0.smithi195.stdout:removed './111/39' 2022-06-27T11:51:42.266 INFO:tasks.workunit.client.0.smithi195.stdout:removed './111/10' 2022-06-27T11:51:42.267 INFO:tasks.workunit.client.0.smithi195.stdout:removed './111/34' 2022-06-27T11:51:42.267 INFO:tasks.workunit.client.0.smithi195.stdout:removed './111/38' 2022-06-27T11:51:42.268 INFO:tasks.workunit.client.0.smithi195.stdout:removed './111/11' 2022-06-27T11:51:42.268 INFO:tasks.workunit.client.0.smithi195.stdout:removed './111/16' 2022-06-27T11:51:42.269 INFO:tasks.workunit.client.0.smithi195.stdout:removed './111/32' 2022-06-27T11:51:42.270 INFO:tasks.workunit.client.0.smithi195.stdout:removed './111/30' 2022-06-27T11:51:42.270 INFO:tasks.workunit.client.0.smithi195.stdout:removed './111/14' 2022-06-27T11:51:42.271 INFO:tasks.workunit.client.0.smithi195.stdout:removed './111/18' 2022-06-27T11:51:42.271 INFO:tasks.workunit.client.0.smithi195.stdout:removed './111/13' 2022-06-27T11:51:42.271 INFO:tasks.workunit.client.0.smithi195.stdout:removed './111/33' 2022-06-27T11:51:42.272 INFO:tasks.workunit.client.0.smithi195.stdout:removed './111/17' 2022-06-27T11:51:42.272 INFO:tasks.workunit.client.0.smithi195.stdout:removed './111/35' 2022-06-27T11:51:42.273 INFO:tasks.workunit.client.0.smithi195.stdout:removed './111/37' 2022-06-27T11:51:42.273 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './111' 2022-06-27T11:51:42.274 INFO:tasks.workunit.client.0.smithi195.stdout:removed './112/19' 2022-06-27T11:51:42.275 INFO:tasks.workunit.client.0.smithi195.stdout:removed './112/15' 2022-06-27T11:51:42.276 INFO:tasks.workunit.client.0.smithi195.stdout:removed './112/31' 2022-06-27T11:51:42.277 INFO:tasks.workunit.client.0.smithi195.stdout:removed './112/36' 2022-06-27T11:51:42.277 INFO:tasks.workunit.client.0.smithi195.stdout:removed './112/20' 2022-06-27T11:51:42.277 INFO:tasks.workunit.client.0.smithi195.stdout:removed './112/12' 2022-06-27T11:51:42.278 INFO:tasks.workunit.client.0.smithi195.stdout:removed './112/39' 2022-06-27T11:51:42.278 INFO:tasks.workunit.client.0.smithi195.stdout:removed './112/10' 2022-06-27T11:51:42.279 INFO:tasks.workunit.client.0.smithi195.stdout:removed './112/34' 2022-06-27T11:51:42.279 INFO:tasks.workunit.client.0.smithi195.stdout:removed './112/38' 2022-06-27T11:51:42.280 INFO:tasks.workunit.client.0.smithi195.stdout:removed './112/11' 2022-06-27T11:51:42.280 INFO:tasks.workunit.client.0.smithi195.stdout:removed './112/16' 2022-06-27T11:51:42.281 INFO:tasks.workunit.client.0.smithi195.stdout:removed './112/32' 2022-06-27T11:51:42.281 INFO:tasks.workunit.client.0.smithi195.stdout:removed './112/30' 2022-06-27T11:51:42.282 INFO:tasks.workunit.client.0.smithi195.stdout:removed './112/14' 2022-06-27T11:51:42.282 INFO:tasks.workunit.client.0.smithi195.stdout:removed './112/18' 2022-06-27T11:51:42.282 INFO:tasks.workunit.client.0.smithi195.stdout:removed './112/13' 2022-06-27T11:51:42.283 INFO:tasks.workunit.client.0.smithi195.stdout:removed './112/33' 2022-06-27T11:51:42.283 INFO:tasks.workunit.client.0.smithi195.stdout:removed './112/17' 2022-06-27T11:51:42.284 INFO:tasks.workunit.client.0.smithi195.stdout:removed './112/35' 2022-06-27T11:51:42.284 INFO:tasks.workunit.client.0.smithi195.stdout:removed './112/37' 2022-06-27T11:51:42.285 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './112' 2022-06-27T11:51:42.285 INFO:tasks.workunit.client.0.smithi195.stdout:removed './113/19' 2022-06-27T11:51:42.287 INFO:tasks.workunit.client.0.smithi195.stdout:removed './113/15' 2022-06-27T11:51:42.287 INFO:tasks.workunit.client.0.smithi195.stdout:removed './113/31' 2022-06-27T11:51:42.289 INFO:tasks.workunit.client.0.smithi195.stdout:removed './113/36' 2022-06-27T11:51:42.290 INFO:tasks.workunit.client.0.smithi195.stdout:removed './113/20' 2022-06-27T11:51:42.290 INFO:tasks.workunit.client.0.smithi195.stdout:removed './113/12' 2022-06-27T11:51:42.291 INFO:tasks.workunit.client.0.smithi195.stdout:removed './113/39' 2022-06-27T11:51:42.291 INFO:tasks.workunit.client.0.smithi195.stdout:removed './113/10' 2022-06-27T11:51:42.292 INFO:tasks.workunit.client.0.smithi195.stdout:removed './113/34' 2022-06-27T11:51:42.292 INFO:tasks.workunit.client.0.smithi195.stdout:removed './113/38' 2022-06-27T11:51:42.293 INFO:tasks.workunit.client.0.smithi195.stdout:removed './113/11' 2022-06-27T11:51:42.293 INFO:tasks.workunit.client.0.smithi195.stdout:removed './113/16' 2022-06-27T11:51:42.294 INFO:tasks.workunit.client.0.smithi195.stdout:removed './113/32' 2022-06-27T11:51:42.294 INFO:tasks.workunit.client.0.smithi195.stdout:removed './113/30' 2022-06-27T11:51:42.295 INFO:tasks.workunit.client.0.smithi195.stdout:removed './113/14' 2022-06-27T11:51:42.295 INFO:tasks.workunit.client.0.smithi195.stdout:removed './113/18' 2022-06-27T11:51:42.296 INFO:tasks.workunit.client.0.smithi195.stdout:removed './113/13' 2022-06-27T11:51:42.296 INFO:tasks.workunit.client.0.smithi195.stdout:removed './113/33' 2022-06-27T11:51:42.297 INFO:tasks.workunit.client.0.smithi195.stdout:removed './113/17' 2022-06-27T11:51:42.298 INFO:tasks.workunit.client.0.smithi195.stdout:removed './113/35' 2022-06-27T11:51:42.298 INFO:tasks.workunit.client.0.smithi195.stdout:removed './113/37' 2022-06-27T11:51:42.313 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './113' 2022-06-27T11:51:42.313 INFO:tasks.workunit.client.0.smithi195.stdout:removed './114/19' 2022-06-27T11:51:42.314 INFO:tasks.workunit.client.0.smithi195.stdout:removed './114/15' 2022-06-27T11:51:42.314 INFO:tasks.workunit.client.0.smithi195.stdout:removed './114/31' 2022-06-27T11:51:42.314 INFO:tasks.workunit.client.0.smithi195.stdout:removed './114/36' 2022-06-27T11:51:42.314 INFO:tasks.workunit.client.0.smithi195.stdout:removed './114/20' 2022-06-27T11:51:42.314 INFO:tasks.workunit.client.0.smithi195.stdout:removed './114/12' 2022-06-27T11:51:42.315 INFO:tasks.workunit.client.0.smithi195.stdout:removed './114/39' 2022-06-27T11:51:42.315 INFO:tasks.workunit.client.0.smithi195.stdout:removed './114/10' 2022-06-27T11:51:42.315 INFO:tasks.workunit.client.0.smithi195.stdout:removed './114/34' 2022-06-27T11:51:42.315 INFO:tasks.workunit.client.0.smithi195.stdout:removed './114/38' 2022-06-27T11:51:42.316 INFO:tasks.workunit.client.0.smithi195.stdout:removed './114/11' 2022-06-27T11:51:42.316 INFO:tasks.workunit.client.0.smithi195.stdout:removed './114/16' 2022-06-27T11:51:42.316 INFO:tasks.workunit.client.0.smithi195.stdout:removed './114/32' 2022-06-27T11:51:42.317 INFO:tasks.workunit.client.0.smithi195.stdout:removed './114/30' 2022-06-27T11:51:42.317 INFO:tasks.workunit.client.0.smithi195.stdout:removed './114/14' 2022-06-27T11:51:42.317 INFO:tasks.workunit.client.0.smithi195.stdout:removed './114/18' 2022-06-27T11:51:42.318 INFO:tasks.workunit.client.0.smithi195.stdout:removed './114/13' 2022-06-27T11:51:42.318 INFO:tasks.workunit.client.0.smithi195.stdout:removed './114/33' 2022-06-27T11:51:42.318 INFO:tasks.workunit.client.0.smithi195.stdout:removed './114/17' 2022-06-27T11:51:42.318 INFO:tasks.workunit.client.0.smithi195.stdout:removed './114/35' 2022-06-27T11:51:42.319 INFO:tasks.workunit.client.0.smithi195.stdout:removed './114/37' 2022-06-27T11:51:42.319 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './114' 2022-06-27T11:51:42.320 INFO:tasks.workunit.client.0.smithi195.stdout:removed './115/19' 2022-06-27T11:51:42.320 INFO:tasks.workunit.client.0.smithi195.stdout:removed './115/15' 2022-06-27T11:51:42.323 INFO:tasks.workunit.client.0.smithi195.stdout:removed './115/31' 2022-06-27T11:51:42.324 INFO:tasks.workunit.client.0.smithi195.stdout:removed './115/36' 2022-06-27T11:51:42.324 INFO:tasks.workunit.client.0.smithi195.stdout:removed './115/20' 2022-06-27T11:51:42.325 INFO:tasks.workunit.client.0.smithi195.stdout:removed './115/12' 2022-06-27T11:51:42.326 INFO:tasks.workunit.client.0.smithi195.stdout:removed './115/39' 2022-06-27T11:51:42.327 INFO:tasks.workunit.client.0.smithi195.stdout:removed './115/10' 2022-06-27T11:51:42.327 INFO:tasks.workunit.client.0.smithi195.stdout:removed './115/34' 2022-06-27T11:51:42.327 INFO:tasks.workunit.client.0.smithi195.stdout:removed './115/38' 2022-06-27T11:51:42.328 INFO:tasks.workunit.client.0.smithi195.stdout:removed './115/11' 2022-06-27T11:51:42.329 INFO:tasks.workunit.client.0.smithi195.stdout:removed './115/16' 2022-06-27T11:51:42.329 INFO:tasks.workunit.client.0.smithi195.stdout:removed './115/32' 2022-06-27T11:51:42.330 INFO:tasks.workunit.client.0.smithi195.stdout:removed './115/30' 2022-06-27T11:51:42.330 INFO:tasks.workunit.client.0.smithi195.stdout:removed './115/14' 2022-06-27T11:51:42.330 INFO:tasks.workunit.client.0.smithi195.stdout:removed './115/18' 2022-06-27T11:51:42.331 INFO:tasks.workunit.client.0.smithi195.stdout:removed './115/13' 2022-06-27T11:51:42.331 INFO:tasks.workunit.client.0.smithi195.stdout:removed './115/33' 2022-06-27T11:51:42.331 INFO:tasks.workunit.client.0.smithi195.stdout:removed './115/17' 2022-06-27T11:51:42.332 INFO:tasks.workunit.client.0.smithi195.stdout:removed './115/35' 2022-06-27T11:51:42.332 INFO:tasks.workunit.client.0.smithi195.stdout:removed './115/37' 2022-06-27T11:51:42.334 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './115' 2022-06-27T11:51:42.334 INFO:tasks.workunit.client.0.smithi195.stdout:removed './116/19' 2022-06-27T11:51:42.334 INFO:tasks.workunit.client.0.smithi195.stdout:removed './116/15' 2022-06-27T11:51:42.334 INFO:tasks.workunit.client.0.smithi195.stdout:removed './116/31' 2022-06-27T11:51:42.335 INFO:tasks.workunit.client.0.smithi195.stdout:removed './116/36' 2022-06-27T11:51:42.335 INFO:tasks.workunit.client.0.smithi195.stdout:removed './116/20' 2022-06-27T11:51:42.335 INFO:tasks.workunit.client.0.smithi195.stdout:removed './116/12' 2022-06-27T11:51:42.335 INFO:tasks.workunit.client.0.smithi195.stdout:removed './116/39' 2022-06-27T11:51:42.336 INFO:tasks.workunit.client.0.smithi195.stdout:removed './116/10' 2022-06-27T11:51:42.336 INFO:tasks.workunit.client.0.smithi195.stdout:removed './116/34' 2022-06-27T11:51:42.336 INFO:tasks.workunit.client.0.smithi195.stdout:removed './116/38' 2022-06-27T11:51:42.337 INFO:tasks.workunit.client.0.smithi195.stdout:removed './116/11' 2022-06-27T11:51:42.337 INFO:tasks.workunit.client.0.smithi195.stdout:removed './116/16' 2022-06-27T11:51:42.337 INFO:tasks.workunit.client.0.smithi195.stdout:removed './116/32' 2022-06-27T11:51:42.337 INFO:tasks.workunit.client.0.smithi195.stdout:removed './116/30' 2022-06-27T11:51:42.338 INFO:tasks.workunit.client.0.smithi195.stdout:removed './116/14' 2022-06-27T11:51:42.338 INFO:tasks.workunit.client.0.smithi195.stdout:removed './116/18' 2022-06-27T11:51:42.338 INFO:tasks.workunit.client.0.smithi195.stdout:removed './116/13' 2022-06-27T11:51:42.338 INFO:tasks.workunit.client.0.smithi195.stdout:removed './116/33' 2022-06-27T11:51:42.338 INFO:tasks.workunit.client.0.smithi195.stdout:removed './116/17' 2022-06-27T11:51:42.339 INFO:tasks.workunit.client.0.smithi195.stdout:removed './116/35' 2022-06-27T11:51:42.339 INFO:tasks.workunit.client.0.smithi195.stdout:removed './116/37' 2022-06-27T11:51:42.339 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './116' 2022-06-27T11:51:42.339 INFO:tasks.workunit.client.0.smithi195.stdout:removed './117/19' 2022-06-27T11:51:42.340 INFO:tasks.workunit.client.0.smithi195.stdout:removed './117/15' 2022-06-27T11:51:42.340 INFO:tasks.workunit.client.0.smithi195.stdout:removed './117/31' 2022-06-27T11:51:42.340 INFO:tasks.workunit.client.0.smithi195.stdout:removed './117/36' 2022-06-27T11:51:42.340 INFO:tasks.workunit.client.0.smithi195.stdout:removed './117/20' 2022-06-27T11:51:42.340 INFO:tasks.workunit.client.0.smithi195.stdout:removed './117/12' 2022-06-27T11:51:42.341 INFO:tasks.workunit.client.0.smithi195.stdout:removed './117/39' 2022-06-27T11:51:42.341 INFO:tasks.workunit.client.0.smithi195.stdout:removed './117/10' 2022-06-27T11:51:42.341 INFO:tasks.workunit.client.0.smithi195.stdout:removed './117/34' 2022-06-27T11:51:42.341 INFO:tasks.workunit.client.0.smithi195.stdout:removed './117/38' 2022-06-27T11:51:42.342 INFO:tasks.workunit.client.0.smithi195.stdout:removed './117/11' 2022-06-27T11:51:42.342 INFO:tasks.workunit.client.0.smithi195.stdout:removed './117/16' 2022-06-27T11:51:42.342 INFO:tasks.workunit.client.0.smithi195.stdout:removed './117/32' 2022-06-27T11:51:42.342 INFO:tasks.workunit.client.0.smithi195.stdout:removed './117/30' 2022-06-27T11:51:42.343 INFO:tasks.workunit.client.0.smithi195.stdout:removed './117/14' 2022-06-27T11:51:42.343 INFO:tasks.workunit.client.0.smithi195.stdout:removed './117/18' 2022-06-27T11:51:42.343 INFO:tasks.workunit.client.0.smithi195.stdout:removed './117/13' 2022-06-27T11:51:42.343 INFO:tasks.workunit.client.0.smithi195.stdout:removed './117/33' 2022-06-27T11:51:42.344 INFO:tasks.workunit.client.0.smithi195.stdout:removed './117/17' 2022-06-27T11:51:42.344 INFO:tasks.workunit.client.0.smithi195.stdout:removed './117/35' 2022-06-27T11:51:42.344 INFO:tasks.workunit.client.0.smithi195.stdout:removed './117/37' 2022-06-27T11:51:42.345 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './117' 2022-06-27T11:51:42.345 INFO:tasks.workunit.client.0.smithi195.stdout:removed './118/19' 2022-06-27T11:51:42.345 INFO:tasks.workunit.client.0.smithi195.stdout:removed './118/15' 2022-06-27T11:51:42.345 INFO:tasks.workunit.client.0.smithi195.stdout:removed './118/31' 2022-06-27T11:51:42.345 INFO:tasks.workunit.client.0.smithi195.stdout:removed './118/36' 2022-06-27T11:51:42.346 INFO:tasks.workunit.client.0.smithi195.stdout:removed './118/20' 2022-06-27T11:51:42.346 INFO:tasks.workunit.client.0.smithi195.stdout:removed './118/12' 2022-06-27T11:51:42.346 INFO:tasks.workunit.client.0.smithi195.stdout:removed './118/39' 2022-06-27T11:51:42.346 INFO:tasks.workunit.client.0.smithi195.stdout:removed './118/10' 2022-06-27T11:51:42.347 INFO:tasks.workunit.client.0.smithi195.stdout:removed './118/34' 2022-06-27T11:51:42.347 INFO:tasks.workunit.client.0.smithi195.stdout:removed './118/38' 2022-06-27T11:51:42.347 INFO:tasks.workunit.client.0.smithi195.stdout:removed './118/11' 2022-06-27T11:51:42.348 INFO:tasks.workunit.client.0.smithi195.stdout:removed './118/16' 2022-06-27T11:51:42.348 INFO:tasks.workunit.client.0.smithi195.stdout:removed './118/32' 2022-06-27T11:51:42.348 INFO:tasks.workunit.client.0.smithi195.stdout:removed './118/30' 2022-06-27T11:51:42.349 INFO:tasks.workunit.client.0.smithi195.stdout:removed './118/14' 2022-06-27T11:51:42.349 INFO:tasks.workunit.client.0.smithi195.stdout:removed './118/18' 2022-06-27T11:51:42.349 INFO:tasks.workunit.client.0.smithi195.stdout:removed './118/13' 2022-06-27T11:51:42.350 INFO:tasks.workunit.client.0.smithi195.stdout:removed './118/33' 2022-06-27T11:51:42.350 INFO:tasks.workunit.client.0.smithi195.stdout:removed './118/17' 2022-06-27T11:51:42.351 INFO:tasks.workunit.client.0.smithi195.stdout:removed './118/35' 2022-06-27T11:51:42.351 INFO:tasks.workunit.client.0.smithi195.stdout:removed './118/37' 2022-06-27T11:51:42.351 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './118' 2022-06-27T11:51:42.352 INFO:tasks.workunit.client.0.smithi195.stdout:removed './119/19' 2022-06-27T11:51:42.352 INFO:tasks.workunit.client.0.smithi195.stdout:removed './119/15' 2022-06-27T11:51:42.352 INFO:tasks.workunit.client.0.smithi195.stdout:removed './119/31' 2022-06-27T11:51:42.353 INFO:tasks.workunit.client.0.smithi195.stdout:removed './119/36' 2022-06-27T11:51:42.353 INFO:tasks.workunit.client.0.smithi195.stdout:removed './119/20' 2022-06-27T11:51:42.354 INFO:tasks.workunit.client.0.smithi195.stdout:removed './119/12' 2022-06-27T11:51:42.356 INFO:tasks.ceph.mds.b.smithi202.stdout:starting mds.b at 2022-06-27T11:51:42.369 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.292+0000 7faac3356700 1 Processor -- start 2022-06-27T11:51:42.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.292+0000 7faac3356700 1 -- start start 2022-06-27T11:51:42.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.293+0000 7faac3356700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faabc14e490 0x7faabc06cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:42.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.293+0000 7faac3356700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7faabc14e860 0x7faabc0c3840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:42.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.293+0000 7faac3356700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faabc0c33b0 con 0x7faabc0c3d80 2022-06-27T11:51:42.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.293+0000 7faac3356700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faabc06d520 con 0x7faabc14e860 2022-06-27T11:51:42.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.293+0000 7faac3356700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faabc06f220 con 0x7faabc14e490 2022-06-27T11:51:42.372 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.293+0000 7faac10f2700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faabc14e490 0x7faabc06cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:42.372 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.293+0000 7faac08f1700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7faabc14e860 0x7faabc0c3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:42.372 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.293+0000 7faac18f3700 1 --1- >> v1:172.21.15.202:6789/0 conn(0x7faabc0c3d80 0x7faabc06f620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.202:6789/0 says I am v1:172.21.15.195:51384/0 (socket says 172.21.15.195:51384) 2022-06-27T11:51:42.372 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.293+0000 7faac18f3700 1 -- 172.21.15.195:0/3072791931 learned_addr learned my addr 172.21.15.195:0/3072791931 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:51:42.372 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.293+0000 7faaabfff700 1 -- 172.21.15.195:0/3072791931 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3645061045 0 0) 0x7faabc0c33b0 con 0x7faabc0c3d80 2022-06-27T11:51:42.373 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.293+0000 7faaabfff700 1 -- 172.21.15.195:0/3072791931 --> v1:172.21.15.202:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faaa4003430 con 0x7faabc0c3d80 2022-06-27T11:51:42.373 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.293+0000 7faac10f2700 1 -- 172.21.15.195:0/3072791931 >> v2:172.21.15.202:3300/0 conn(0x7faabc14e860 msgr2=0x7faabc0c3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:51:42.373 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.293+0000 7faac10f2700 1 --2- 172.21.15.195:0/3072791931 >> v2:172.21.15.202:3300/0 conn(0x7faabc14e860 0x7faabc0c3840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:42.373 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.293+0000 7faac10f2700 1 -- 172.21.15.195:0/3072791931 >> v1:172.21.15.202:6789/0 conn(0x7faabc0c3d80 legacy=0x7faabc06f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:42.374 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.293+0000 7faac10f2700 1 -- 172.21.15.195:0/3072791931 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faabc1542c0 con 0x7faabc14e490 2022-06-27T11:51:42.374 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.293+0000 7faac08f1700 1 --2- 172.21.15.195:0/3072791931 >> v2:172.21.15.202:3300/0 conn(0x7faabc14e860 0x7faabc0c3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:51:42.374 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.293+0000 7faac10f2700 1 --2- 172.21.15.195:0/3072791931 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faabc14e490 0x7faabc06cfe0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7faab80097a0 tx=0x7faab8009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa72ef6435bd7396 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:51:42.374 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.294+0000 7faaabfff700 1 -- 172.21.15.195:0/3072791931 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faab8011630 con 0x7faabc14e490 2022-06-27T11:51:42.374 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.294+0000 7faaabfff700 1 -- 172.21.15.195:0/3072791931 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faab8011790 con 0x7faabc14e490 2022-06-27T11:51:42.375 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.294+0000 7faaabfff700 1 -- 172.21.15.195:0/3072791931 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faab8011b10 con 0x7faabc14e490 2022-06-27T11:51:42.375 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.294+0000 7faac3356700 1 -- 172.21.15.195:0/3072791931 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faabc14e490 msgr2=0x7faabc06cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:42.375 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.294+0000 7faac3356700 1 --2- 172.21.15.195:0/3072791931 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faabc14e490 0x7faabc06cfe0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7faab80097a0 tx=0x7faab8009060 comp rx=0 tx=0).stop 2022-06-27T11:51:42.376 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.294+0000 7faac3356700 1 -- 172.21.15.195:0/3072791931 shutdown_connections 2022-06-27T11:51:42.376 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.294+0000 7faac3356700 1 --2- 172.21.15.195:0/3072791931 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faabc14e490 0x7faabc06cfe0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:42.376 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.294+0000 7faac3356700 1 --2- 172.21.15.195:0/3072791931 >> v2:172.21.15.202:3300/0 conn(0x7faabc14e860 0x7faabc0c3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:42.376 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.294+0000 7faac3356700 1 -- 172.21.15.195:0/3072791931 >> 172.21.15.195:0/3072791931 conn(0x7faabc063af0 msgr2=0x7faabc063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:51:42.376 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.294+0000 7faac3356700 1 -- 172.21.15.195:0/3072791931 shutdown_connections 2022-06-27T11:51:42.377 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.294+0000 7faac3356700 1 -- 172.21.15.195:0/3072791931 wait complete. 2022-06-27T11:51:42.377 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.296+0000 7faac3356700 1 Processor -- start 2022-06-27T11:51:42.377 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.296+0000 7faac3356700 1 -- start start 2022-06-27T11:51:42.378 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.297+0000 7faac3356700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faabc0c3d80 0x7faabc167a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:42.378 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.297+0000 7faac3356700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faabc14e490 0x7faabc0c7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:42.378 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.297+0000 7faac3356700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7faabc14e860 0x7faabc0c0d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:42.379 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.297+0000 7faac3356700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faabc164e20 con 0x7faabc14e490 2022-06-27T11:51:42.379 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.297+0000 7faac3356700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faabc0bebd0 con 0x7faabc14e860 2022-06-27T11:51:42.379 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.297+0000 7faac3356700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faabc0bee90 con 0x7faabc0c3d80 2022-06-27T11:51:42.380 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.297+0000 7faac10f2700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faabc0c3d80 0x7faabc167a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:42.380 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.297+0000 7faac08f1700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faabc14e490 0x7faabc0c7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:42.380 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.297+0000 7faac18f3700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7faabc14e860 0x7faabc0c0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:42.381 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.297+0000 7faac10f2700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faabc0c3d80 0x7faabc167a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3301/0 says I am v2:172.21.15.195:36094/0 (socket says 172.21.15.195:36094) 2022-06-27T11:51:42.381 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.297+0000 7faac10f2700 1 -- 172.21.15.195:0/3329335533 learned_addr learned my addr 172.21.15.195:0/3329335533 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:51:42.381 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.297+0000 7faac18f3700 1 -- 172.21.15.195:0/3329335533 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faabc0c3d80 msgr2=0x7faabc167a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:42.381 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.297+0000 7faac18f3700 1 --2- 172.21.15.195:0/3329335533 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faabc0c3d80 0x7faabc167a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:42.382 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.297+0000 7faac18f3700 1 -- 172.21.15.195:0/3329335533 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faabc14e490 msgr2=0x7faabc0c7cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:42.382 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.297+0000 7faac18f3700 1 --2- 172.21.15.195:0/3329335533 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faabc14e490 0x7faabc0c7cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:42.383 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.297+0000 7faac18f3700 1 -- 172.21.15.195:0/3329335533 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaac0092e0 con 0x7faabc14e860 2022-06-27T11:51:42.383 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.297+0000 7faac10f2700 1 --2- 172.21.15.195:0/3329335533 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faabc0c3d80 0x7faabc167a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:51:42.383 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.297+0000 7faac08f1700 1 --2- 172.21.15.195:0/3329335533 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faabc14e490 0x7faabc0c7cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:51:42.384 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.297+0000 7faac18f3700 1 --2- 172.21.15.195:0/3329335533 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7faabc14e860 0x7faabc0c0d40 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7faab400c790 tx=0x7faab4007f30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:51:42.384 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.298+0000 7faaa97fa700 1 -- 172.21.15.195:0/3329335533 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faab4005d70 con 0x7faabc14e860 2022-06-27T11:51:42.385 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.298+0000 7faaa97fa700 1 -- 172.21.15.195:0/3329335533 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faab40070d0 con 0x7faabc14e860 2022-06-27T11:51:42.385 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.298+0000 7faaa97fa700 1 -- 172.21.15.195:0/3329335533 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faab4007470 con 0x7faabc14e860 2022-06-27T11:51:42.386 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.298+0000 7faac3356700 1 -- 172.21.15.195:0/3329335533 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faab80092e0 con 0x7faabc14e860 2022-06-27T11:51:42.386 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.298+0000 7faac3356700 1 -- 172.21.15.195:0/3329335533 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faabc164ab0 con 0x7faabc14e860 2022-06-27T11:51:42.387 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.298+0000 7faac3356700 1 -- 172.21.15.195:0/3329335533 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faabc164f90 con 0x7faabc14e860 2022-06-27T11:51:42.387 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.299+0000 7faaa97fa700 1 -- 172.21.15.195:0/3329335533 <== mon.1 v2:172.21.15.202:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7faab4019030 con 0x7faabc14e860 2022-06-27T11:51:42.387 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.299+0000 7faaa97fa700 1 --2- 172.21.15.195:0/3329335533 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7faa9c063d80 0x7faa9c066230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:42.388 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.299+0000 7faaa97fa700 1 -- 172.21.15.195:0/3329335533 <== mon.1 v2:172.21.15.202:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6728+0+0 (secure 0 0 0) 0x7faab4080020 con 0x7faabc14e860 2022-06-27T11:51:42.388 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.299+0000 7faac10f2700 1 --2- 172.21.15.195:0/3329335533 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7faa9c063d80 0x7faa9c066230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:42.388 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.300+0000 7faac10f2700 1 --2- 172.21.15.195:0/3329335533 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7faa9c063d80 0x7faa9c066230 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7faab8009260 tx=0x7faab8017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:51:42.389 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.304+0000 7faaa97fa700 1 -- 172.21.15.195:0/3329335533 <== mon.1 v2:172.21.15.202:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7faab404e880 con 0x7faabc14e860 2022-06-27T11:51:42.529 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.530+0000 7faac3356700 1 -- 172.21.15.195:0/3329335533 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faabc0722c0 con 0x7faabc14e860 2022-06-27T11:51:42.531 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.532+0000 7faaa97fa700 1 -- 172.21.15.195:0/3329335533 <== mon.1 v2:172.21.15.202:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+5368 (secure 0 0 0) 0x7faab4051ea0 con 0x7faabc14e860 2022-06-27T11:51:42.532 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:51:42.533 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:51:33.013206+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4849},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4849":{"gid":4849,"name":"d","rank":0,"incarnation":15,"state":"up:active","state_seq":19,"addr":"172.21.15.202:6835/3271891211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":3271891211},{"type":"v1","addr":"172.21.15.202:6835","nonce":3271891211}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:51:42.535 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.533+0000 7faac3356700 1 -- 172.21.15.195:0/3329335533 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7faa9c063d80 msgr2=0x7faa9c066230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:42.535 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.533+0000 7faac3356700 1 --2- 172.21.15.195:0/3329335533 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7faa9c063d80 0x7faa9c066230 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7faab8009260 tx=0x7faab8017000 comp rx=0 tx=0).stop 2022-06-27T11:51:42.536 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.533+0000 7faac3356700 1 -- 172.21.15.195:0/3329335533 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7faabc14e860 msgr2=0x7faabc0c0d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:42.536 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.533+0000 7faac3356700 1 --2- 172.21.15.195:0/3329335533 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7faabc14e860 0x7faabc0c0d40 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7faab400c790 tx=0x7faab4007f30 comp rx=0 tx=0).stop 2022-06-27T11:51:42.536 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.533+0000 7faac3356700 1 -- 172.21.15.195:0/3329335533 shutdown_connections 2022-06-27T11:51:42.536 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.533+0000 7faac3356700 1 --2- 172.21.15.195:0/3329335533 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7faa9c063d80 0x7faa9c066230 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:42.537 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.533+0000 7faac3356700 1 --2- 172.21.15.195:0/3329335533 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7faabc14e860 0x7faabc0c0d40 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:42.537 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.533+0000 7faac3356700 1 --2- 172.21.15.195:0/3329335533 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faabc0c3d80 0x7faabc167a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:42.537 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.533+0000 7faac3356700 1 --2- 172.21.15.195:0/3329335533 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faabc14e490 0x7faabc0c7cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:42.537 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.533+0000 7faac3356700 1 -- 172.21.15.195:0/3329335533 >> 172.21.15.195:0/3329335533 conn(0x7faabc063af0 msgr2=0x7faabc0ca910 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:51:42.538 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.533+0000 7faac3356700 1 -- 172.21.15.195:0/3329335533 shutdown_connections 2022-06-27T11:51:42.538 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:42.533+0000 7faac3356700 1 -- 172.21.15.195:0/3329335533 wait complete. 2022-06-27T11:51:42.538 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 18 2022-06-27T11:51:42.545 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2022-06-27T11:51:44.547 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:51:44.691 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.679+0000 7f9b02308700 1 Processor -- start 2022-06-27T11:51:44.692 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.679+0000 7f9b02308700 1 -- start start 2022-06-27T11:51:44.692 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.679+0000 7f9b02308700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f9afc06e850 0x7f9afc1570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:44.693 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.679+0000 7f9b02308700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f9afc0c34a0 0x7f9afc0c2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:44.693 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.680+0000 7f9b02308700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9afc06fba0 con 0x7f9afc06fef0 2022-06-27T11:51:44.695 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.680+0000 7f9b02308700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9afc1575e0 con 0x7f9afc0c34a0 2022-06-27T11:51:44.695 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.680+0000 7f9b02308700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9afc06f5e0 con 0x7f9afc06e850 2022-06-27T11:51:44.696 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.680+0000 7f9b008a5700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f9afc0c34a0 0x7f9afc0c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:44.696 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.680+0000 7f9b008a5700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f9afc0c34a0 0x7f9afc0c2920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:43354/0 (socket says 172.21.15.195:43354) 2022-06-27T11:51:44.696 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.680+0000 7f9b008a5700 1 -- 172.21.15.195:0/2585233543 learned_addr learned my addr 172.21.15.195:0/2585233543 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:51:44.697 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.680+0000 7f9afbfff700 1 --2- 172.21.15.195:0/2585233543 >> v2:172.21.15.202:3300/0 conn(0x7f9afc06e850 0x7f9afc1570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:44.697 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.680+0000 7f9b008a5700 1 -- 172.21.15.195:0/2585233543 >> v2:172.21.15.202:3300/0 conn(0x7f9afc06e850 msgr2=0x7f9afc1570a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:51:44.698 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.680+0000 7f9b008a5700 1 --2- 172.21.15.195:0/2585233543 >> v2:172.21.15.202:3300/0 conn(0x7f9afc06e850 0x7f9afc1570a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:44.698 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.680+0000 7f9b008a5700 1 -- 172.21.15.195:0/2585233543 >> v1:172.21.15.202:6789/0 conn(0x7f9afc06fef0 legacy=0x7f9afc069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:44.699 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.680+0000 7f9b008a5700 1 -- 172.21.15.195:0/2585233543 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9afc06f8c0 con 0x7f9afc0c34a0 2022-06-27T11:51:44.699 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.680+0000 7f9b008a5700 1 --2- 172.21.15.195:0/2585233543 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9afc0c34a0 0x7f9afc0c2920 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f9afc06e120 tx=0x7f9af4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2e20b40411bb4c4 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:51:44.700 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.680+0000 7f9afaffd700 1 -- 172.21.15.195:0/2585233543 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af400e070 con 0x7f9afc0c34a0 2022-06-27T11:51:44.700 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.680+0000 7f9afaffd700 1 -- 172.21.15.195:0/2585233543 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9af400bab0 con 0x7f9afc0c34a0 2022-06-27T11:51:44.701 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.680+0000 7f9afaffd700 1 -- 172.21.15.195:0/2585233543 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af40092e0 con 0x7f9afc0c34a0 2022-06-27T11:51:44.701 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.681+0000 7f9b02308700 1 -- 172.21.15.195:0/2585233543 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9afc0c34a0 msgr2=0x7f9afc0c2920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:44.701 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.681+0000 7f9b02308700 1 --2- 172.21.15.195:0/2585233543 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9afc0c34a0 0x7f9afc0c2920 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f9afc06e120 tx=0x7f9af4009840 comp rx=0 tx=0).stop 2022-06-27T11:51:44.702 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.681+0000 7f9b02308700 1 -- 172.21.15.195:0/2585233543 shutdown_connections 2022-06-27T11:51:44.702 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.681+0000 7f9b02308700 1 --2- 172.21.15.195:0/2585233543 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9afc0c34a0 0x7f9afc0c2920 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:44.703 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.681+0000 7f9b02308700 1 --2- 172.21.15.195:0/2585233543 >> v2:172.21.15.202:3300/0 conn(0x7f9afc06e850 0x7f9afc1570a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:44.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.681+0000 7f9b02308700 1 -- 172.21.15.195:0/2585233543 >> 172.21.15.195:0/2585233543 conn(0x7f9afc0c6350 msgr2=0x7f9afc0c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:51:44.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.681+0000 7f9b02308700 1 -- 172.21.15.195:0/2585233543 shutdown_connections 2022-06-27T11:51:44.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.681+0000 7f9b02308700 1 -- 172.21.15.195:0/2585233543 wait complete. 2022-06-27T11:51:44.706 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.682+0000 7f9b02308700 1 Processor -- start 2022-06-27T11:51:44.706 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.682+0000 7f9b02308700 1 -- start start 2022-06-27T11:51:44.706 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.683+0000 7f9b02308700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9afc06e850 0x7f9afc148040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:44.707 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.683+0000 7f9b02308700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9afc0c34a0 0x7f9afc148580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:44.707 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.683+0000 7f9b02308700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9afc153250 0x7f9afc1431d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:44.707 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.683+0000 7f9b02308700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9afc147bb0 con 0x7f9afc153250 2022-06-27T11:51:44.707 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.683+0000 7f9b02308700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9afc144160 con 0x7f9afc0c34a0 2022-06-27T11:51:44.708 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.683+0000 7f9b02308700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9afc1443e0 con 0x7f9afc06e850 2022-06-27T11:51:44.708 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.683+0000 7f9b008a5700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9afc153250 0x7f9afc1431d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:44.708 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.683+0000 7f9afb7fe700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9afc0c34a0 0x7f9afc148580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:44.708 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.683+0000 7f9afbfff700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9afc06e850 0x7f9afc148040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:44.708 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.683+0000 7f9b008a5700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9afc153250 0x7f9afc1431d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:43360/0 (socket says 172.21.15.195:43360) 2022-06-27T11:51:44.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.683+0000 7f9b008a5700 1 -- 172.21.15.195:0/4024046035 learned_addr learned my addr 172.21.15.195:0/4024046035 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:51:44.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.683+0000 7f9afbfff700 1 -- 172.21.15.195:0/4024046035 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9afc0c34a0 msgr2=0x7f9afc148580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:44.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.683+0000 7f9afbfff700 1 --2- 172.21.15.195:0/4024046035 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9afc0c34a0 0x7f9afc148580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:44.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.683+0000 7f9afbfff700 1 -- 172.21.15.195:0/4024046035 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9afc153250 msgr2=0x7f9afc1431d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:44.710 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.683+0000 7f9afbfff700 1 --2- 172.21.15.195:0/4024046035 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9afc153250 0x7f9afc1431d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:44.710 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.683+0000 7f9afbfff700 1 -- 172.21.15.195:0/4024046035 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9af4009ae0 con 0x7f9afc06e850 2022-06-27T11:51:44.710 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.683+0000 7f9afb7fe700 1 --2- 172.21.15.195:0/4024046035 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9afc0c34a0 0x7f9afc148580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:51:44.710 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.683+0000 7f9b008a5700 1 --2- 172.21.15.195:0/4024046035 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9afc153250 0x7f9afc1431d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:51:44.711 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.683+0000 7f9afbfff700 1 --2- 172.21.15.195:0/4024046035 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9afc06e850 0x7f9afc148040 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9aec00e930 tx=0x7f9aec009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:51:44.711 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.684+0000 7f9ae3fff700 1 -- 172.21.15.195:0/4024046035 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aec013680 con 0x7f9afc06e850 2022-06-27T11:51:44.711 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.684+0000 7f9ae3fff700 1 -- 172.21.15.195:0/4024046035 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9aec0137e0 con 0x7f9afc06e850 2022-06-27T11:51:44.712 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.684+0000 7f9b02308700 1 -- 172.21.15.195:0/4024046035 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9afc1446c0 con 0x7f9afc06e850 2022-06-27T11:51:44.712 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.684+0000 7f9ae3fff700 1 -- 172.21.15.195:0/4024046035 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aec013b80 con 0x7f9afc06e850 2022-06-27T11:51:44.712 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.684+0000 7f9b02308700 1 -- 172.21.15.195:0/4024046035 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9afc144c80 con 0x7f9afc06e850 2022-06-27T11:51:44.713 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.684+0000 7f9b02308700 1 -- 172.21.15.195:0/4024046035 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9afc0caa10 con 0x7f9afc06e850 2022-06-27T11:51:44.713 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.685+0000 7f9ae3fff700 1 -- 172.21.15.195:0/4024046035 <== mon.2 v2:172.21.15.202:3301/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f9aec019030 con 0x7f9afc06e850 2022-06-27T11:51:44.713 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.685+0000 7f9ae3fff700 1 --2- 172.21.15.195:0/4024046035 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9adc063d30 0x7f9adc0661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:51:44.713 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.685+0000 7f9ae3fff700 1 -- 172.21.15.195:0/4024046035 <== mon.2 v2:172.21.15.202:3301/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6728+0+0 (secure 0 0 0) 0x7f9aec0807c0 con 0x7f9afc06e850 2022-06-27T11:51:44.713 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.685+0000 7f9afb7fe700 1 --2- 172.21.15.195:0/4024046035 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9adc063d30 0x7f9adc0661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:51:44.714 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.686+0000 7f9afb7fe700 1 --2- 172.21.15.195:0/4024046035 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9adc063d30 0x7f9adc0661e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f9afc0c5630 tx=0x7f9af0009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:51:44.714 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.690+0000 7f9ae3fff700 1 -- 172.21.15.195:0/4024046035 <== mon.2 v2:172.21.15.202:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f9aec04f0b0 con 0x7f9afc06e850 2022-06-27T11:51:44.802 INFO:tasks.workunit.client.0.smithi195.stdout:removed './119/39' 2022-06-27T11:51:44.802 INFO:tasks.workunit.client.0.smithi195.stdout:removed './119/10' 2022-06-27T11:51:44.803 INFO:tasks.workunit.client.0.smithi195.stdout:removed './119/34' 2022-06-27T11:51:44.803 INFO:tasks.workunit.client.0.smithi195.stdout:removed './119/38' 2022-06-27T11:51:44.803 INFO:tasks.workunit.client.0.smithi195.stdout:removed './119/11' 2022-06-27T11:51:44.804 INFO:tasks.workunit.client.0.smithi195.stdout:removed './119/16' 2022-06-27T11:51:44.804 INFO:tasks.workunit.client.0.smithi195.stdout:removed './119/32' 2022-06-27T11:51:44.804 INFO:tasks.workunit.client.0.smithi195.stdout:removed './119/30' 2022-06-27T11:51:44.804 INFO:tasks.workunit.client.0.smithi195.stdout:removed './119/14' 2022-06-27T11:51:44.805 INFO:tasks.workunit.client.0.smithi195.stdout:removed './119/18' 2022-06-27T11:51:44.805 INFO:tasks.workunit.client.0.smithi195.stdout:removed './119/13' 2022-06-27T11:51:44.805 INFO:tasks.workunit.client.0.smithi195.stdout:removed './119/33' 2022-06-27T11:51:44.805 INFO:tasks.workunit.client.0.smithi195.stdout:removed './119/17' 2022-06-27T11:51:44.806 INFO:tasks.workunit.client.0.smithi195.stdout:removed './119/35' 2022-06-27T11:51:44.806 INFO:tasks.workunit.client.0.smithi195.stdout:removed './119/37' 2022-06-27T11:51:44.806 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './119' 2022-06-27T11:51:44.806 INFO:tasks.workunit.client.0.smithi195.stdout:removed './120/19' 2022-06-27T11:51:44.807 INFO:tasks.workunit.client.0.smithi195.stdout:removed './120/15' 2022-06-27T11:51:44.807 INFO:tasks.workunit.client.0.smithi195.stdout:removed './120/31' 2022-06-27T11:51:44.807 INFO:tasks.workunit.client.0.smithi195.stdout:removed './120/36' 2022-06-27T11:51:44.808 INFO:tasks.workunit.client.0.smithi195.stdout:removed './120/20' 2022-06-27T11:51:44.808 INFO:tasks.workunit.client.0.smithi195.stdout:removed './120/12' 2022-06-27T11:51:44.809 INFO:tasks.workunit.client.0.smithi195.stdout:removed './120/39' 2022-06-27T11:51:44.809 INFO:tasks.workunit.client.0.smithi195.stdout:removed './120/10' 2022-06-27T11:51:44.810 INFO:tasks.workunit.client.0.smithi195.stdout:removed './120/34' 2022-06-27T11:51:44.810 INFO:tasks.workunit.client.0.smithi195.stdout:removed './120/38' 2022-06-27T11:51:44.811 INFO:tasks.workunit.client.0.smithi195.stdout:removed './120/11' 2022-06-27T11:51:44.812 INFO:tasks.workunit.client.0.smithi195.stdout:removed './120/16' 2022-06-27T11:51:44.812 INFO:tasks.workunit.client.0.smithi195.stdout:removed './120/32' 2022-06-27T11:51:44.813 INFO:tasks.workunit.client.0.smithi195.stdout:removed './120/30' 2022-06-27T11:51:44.813 INFO:tasks.workunit.client.0.smithi195.stdout:removed './120/14' 2022-06-27T11:51:44.814 INFO:tasks.workunit.client.0.smithi195.stdout:removed './120/18' 2022-06-27T11:51:44.815 INFO:tasks.workunit.client.0.smithi195.stdout:removed './120/13' 2022-06-27T11:51:44.815 INFO:tasks.workunit.client.0.smithi195.stdout:removed './120/33' 2022-06-27T11:51:44.816 INFO:tasks.workunit.client.0.smithi195.stdout:removed './120/17' 2022-06-27T11:51:44.816 INFO:tasks.workunit.client.0.smithi195.stdout:removed './120/35' 2022-06-27T11:51:44.816 INFO:tasks.workunit.client.0.smithi195.stdout:removed './120/37' 2022-06-27T11:51:44.817 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './120' 2022-06-27T11:51:44.818 INFO:tasks.workunit.client.0.smithi195.stdout:removed './121/19' 2022-06-27T11:51:44.818 INFO:tasks.workunit.client.0.smithi195.stdout:removed './121/15' 2022-06-27T11:51:44.818 INFO:tasks.workunit.client.0.smithi195.stdout:removed './121/31' 2022-06-27T11:51:44.818 INFO:tasks.workunit.client.0.smithi195.stdout:removed './121/36' 2022-06-27T11:51:44.819 INFO:tasks.workunit.client.0.smithi195.stdout:removed './121/20' 2022-06-27T11:51:44.819 INFO:tasks.workunit.client.0.smithi195.stdout:removed './121/12' 2022-06-27T11:51:44.819 INFO:tasks.workunit.client.0.smithi195.stdout:removed './121/39' 2022-06-27T11:51:44.819 INFO:tasks.workunit.client.0.smithi195.stdout:removed './121/10' 2022-06-27T11:51:44.820 INFO:tasks.workunit.client.0.smithi195.stdout:removed './121/34' 2022-06-27T11:51:44.820 INFO:tasks.workunit.client.0.smithi195.stdout:removed './121/38' 2022-06-27T11:51:44.820 INFO:tasks.workunit.client.0.smithi195.stdout:removed './121/11' 2022-06-27T11:51:44.820 INFO:tasks.workunit.client.0.smithi195.stdout:removed './121/16' 2022-06-27T11:51:44.821 INFO:tasks.workunit.client.0.smithi195.stdout:removed './121/32' 2022-06-27T11:51:44.821 INFO:tasks.workunit.client.0.smithi195.stdout:removed './121/30' 2022-06-27T11:51:44.821 INFO:tasks.workunit.client.0.smithi195.stdout:removed './121/14' 2022-06-27T11:51:44.821 INFO:tasks.workunit.client.0.smithi195.stdout:removed './121/18' 2022-06-27T11:51:44.821 INFO:tasks.workunit.client.0.smithi195.stdout:removed './121/13' 2022-06-27T11:51:44.822 INFO:tasks.workunit.client.0.smithi195.stdout:removed './121/33' 2022-06-27T11:51:44.822 INFO:tasks.workunit.client.0.smithi195.stdout:removed './121/17' 2022-06-27T11:51:44.822 INFO:tasks.workunit.client.0.smithi195.stdout:removed './121/35' 2022-06-27T11:51:44.822 INFO:tasks.workunit.client.0.smithi195.stdout:removed './121/37' 2022-06-27T11:51:44.823 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './121' 2022-06-27T11:51:44.823 INFO:tasks.workunit.client.0.smithi195.stdout:removed './122/19' 2022-06-27T11:51:44.823 INFO:tasks.workunit.client.0.smithi195.stdout:removed './122/15' 2022-06-27T11:51:44.823 INFO:tasks.workunit.client.0.smithi195.stdout:removed './122/31' 2022-06-27T11:51:44.824 INFO:tasks.workunit.client.0.smithi195.stdout:removed './122/36' 2022-06-27T11:51:44.824 INFO:tasks.workunit.client.0.smithi195.stdout:removed './122/20' 2022-06-27T11:51:44.824 INFO:tasks.workunit.client.0.smithi195.stdout:removed './122/12' 2022-06-27T11:51:44.824 INFO:tasks.workunit.client.0.smithi195.stdout:removed './122/39' 2022-06-27T11:51:44.825 INFO:tasks.workunit.client.0.smithi195.stdout:removed './122/10' 2022-06-27T11:51:44.825 INFO:tasks.workunit.client.0.smithi195.stdout:removed './122/34' 2022-06-27T11:51:44.825 INFO:tasks.workunit.client.0.smithi195.stdout:removed './122/38' 2022-06-27T11:51:44.825 INFO:tasks.workunit.client.0.smithi195.stdout:removed './122/11' 2022-06-27T11:51:44.826 INFO:tasks.workunit.client.0.smithi195.stdout:removed './122/16' 2022-06-27T11:51:44.826 INFO:tasks.workunit.client.0.smithi195.stdout:removed './122/32' 2022-06-27T11:51:44.826 INFO:tasks.workunit.client.0.smithi195.stdout:removed './122/30' 2022-06-27T11:51:44.827 INFO:tasks.workunit.client.0.smithi195.stdout:removed './122/14' 2022-06-27T11:51:44.827 INFO:tasks.workunit.client.0.smithi195.stdout:removed './122/18' 2022-06-27T11:51:44.827 INFO:tasks.workunit.client.0.smithi195.stdout:removed './122/13' 2022-06-27T11:51:44.827 INFO:tasks.workunit.client.0.smithi195.stdout:removed './122/33' 2022-06-27T11:51:44.827 INFO:tasks.workunit.client.0.smithi195.stdout:removed './122/17' 2022-06-27T11:51:44.828 INFO:tasks.workunit.client.0.smithi195.stdout:removed './122/35' 2022-06-27T11:51:44.828 INFO:tasks.workunit.client.0.smithi195.stdout:removed './122/37' 2022-06-27T11:51:44.828 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './122' 2022-06-27T11:51:44.829 INFO:tasks.workunit.client.0.smithi195.stdout:removed './123/19' 2022-06-27T11:51:44.829 INFO:tasks.workunit.client.0.smithi195.stdout:removed './123/15' 2022-06-27T11:51:44.829 INFO:tasks.workunit.client.0.smithi195.stdout:removed './123/31' 2022-06-27T11:51:44.830 INFO:tasks.workunit.client.0.smithi195.stdout:removed './123/36' 2022-06-27T11:51:44.832 INFO:tasks.workunit.client.0.smithi195.stdout:removed './123/20' 2022-06-27T11:51:44.832 INFO:tasks.workunit.client.0.smithi195.stdout:removed './123/12' 2022-06-27T11:51:44.833 INFO:tasks.workunit.client.0.smithi195.stdout:removed './123/39' 2022-06-27T11:51:44.833 INFO:tasks.workunit.client.0.smithi195.stdout:removed './123/10' 2022-06-27T11:51:44.834 INFO:tasks.workunit.client.0.smithi195.stdout:removed './123/34' 2022-06-27T11:51:44.834 INFO:tasks.workunit.client.0.smithi195.stdout:removed './123/38' 2022-06-27T11:51:44.834 INFO:tasks.workunit.client.0.smithi195.stdout:removed './123/11' 2022-06-27T11:51:44.835 INFO:tasks.workunit.client.0.smithi195.stdout:removed './123/16' 2022-06-27T11:51:44.835 INFO:tasks.workunit.client.0.smithi195.stdout:removed './123/32' 2022-06-27T11:51:44.836 INFO:tasks.workunit.client.0.smithi195.stdout:removed './123/30' 2022-06-27T11:51:44.836 INFO:tasks.workunit.client.0.smithi195.stdout:removed './123/14' 2022-06-27T11:51:44.837 INFO:tasks.workunit.client.0.smithi195.stdout:removed './123/18' 2022-06-27T11:51:44.837 INFO:tasks.workunit.client.0.smithi195.stdout:removed './123/13' 2022-06-27T11:51:44.838 INFO:tasks.workunit.client.0.smithi195.stdout:removed './123/33' 2022-06-27T11:51:44.838 INFO:tasks.workunit.client.0.smithi195.stdout:removed './123/17' 2022-06-27T11:51:44.838 INFO:tasks.workunit.client.0.smithi195.stdout:removed './123/35' 2022-06-27T11:51:44.839 INFO:tasks.workunit.client.0.smithi195.stdout:removed './123/37' 2022-06-27T11:51:44.839 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './123' 2022-06-27T11:51:44.840 INFO:tasks.workunit.client.0.smithi195.stdout:removed './124/19' 2022-06-27T11:51:44.840 INFO:tasks.workunit.client.0.smithi195.stdout:removed './124/15' 2022-06-27T11:51:44.841 INFO:tasks.workunit.client.0.smithi195.stdout:removed './124/31' 2022-06-27T11:51:44.841 INFO:tasks.workunit.client.0.smithi195.stdout:removed './124/36' 2022-06-27T11:51:44.842 INFO:tasks.workunit.client.0.smithi195.stdout:removed './124/20' 2022-06-27T11:51:44.842 INFO:tasks.workunit.client.0.smithi195.stdout:removed './124/12' 2022-06-27T11:51:44.843 INFO:tasks.workunit.client.0.smithi195.stdout:removed './124/39' 2022-06-27T11:51:44.843 INFO:tasks.workunit.client.0.smithi195.stdout:removed './124/10' 2022-06-27T11:51:44.843 INFO:tasks.workunit.client.0.smithi195.stdout:removed './124/34' 2022-06-27T11:51:44.844 INFO:tasks.workunit.client.0.smithi195.stdout:removed './124/38' 2022-06-27T11:51:44.845 INFO:tasks.workunit.client.0.smithi195.stdout:removed './124/11' 2022-06-27T11:51:44.845 INFO:tasks.workunit.client.0.smithi195.stdout:removed './124/16' 2022-06-27T11:51:44.846 INFO:tasks.workunit.client.0.smithi195.stdout:removed './124/32' 2022-06-27T11:51:44.846 INFO:tasks.workunit.client.0.smithi195.stdout:removed './124/30' 2022-06-27T11:51:44.847 INFO:tasks.workunit.client.0.smithi195.stdout:removed './124/14' 2022-06-27T11:51:44.847 INFO:tasks.workunit.client.0.smithi195.stdout:removed './124/18' 2022-06-27T11:51:44.848 INFO:tasks.workunit.client.0.smithi195.stdout:removed './124/13' 2022-06-27T11:51:44.848 INFO:tasks.workunit.client.0.smithi195.stdout:removed './124/33' 2022-06-27T11:51:44.848 INFO:tasks.workunit.client.0.smithi195.stdout:removed './124/17' 2022-06-27T11:51:44.849 INFO:tasks.workunit.client.0.smithi195.stdout:removed './124/35' 2022-06-27T11:51:44.849 INFO:tasks.workunit.client.0.smithi195.stdout:removed './124/37' 2022-06-27T11:51:44.850 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './124' 2022-06-27T11:51:44.850 INFO:tasks.workunit.client.0.smithi195.stdout:removed './125/19' 2022-06-27T11:51:44.851 INFO:tasks.workunit.client.0.smithi195.stdout:removed './125/15' 2022-06-27T11:51:44.851 INFO:tasks.workunit.client.0.smithi195.stdout:removed './125/31' 2022-06-27T11:51:44.852 INFO:tasks.workunit.client.0.smithi195.stdout:removed './125/36' 2022-06-27T11:51:44.852 INFO:tasks.workunit.client.0.smithi195.stdout:removed './125/20' 2022-06-27T11:51:44.853 INFO:tasks.workunit.client.0.smithi195.stdout:removed './125/12' 2022-06-27T11:51:44.853 INFO:tasks.workunit.client.0.smithi195.stdout:removed './125/39' 2022-06-27T11:51:44.853 INFO:tasks.workunit.client.0.smithi195.stdout:removed './125/10' 2022-06-27T11:51:44.854 INFO:tasks.workunit.client.0.smithi195.stdout:removed './125/34' 2022-06-27T11:51:44.854 INFO:tasks.workunit.client.0.smithi195.stdout:removed './125/38' 2022-06-27T11:51:44.855 INFO:tasks.workunit.client.0.smithi195.stdout:removed './125/11' 2022-06-27T11:51:44.855 INFO:tasks.workunit.client.0.smithi195.stdout:removed './125/16' 2022-06-27T11:51:44.856 INFO:tasks.workunit.client.0.smithi195.stdout:removed './125/32' 2022-06-27T11:51:44.857 INFO:tasks.workunit.client.0.smithi195.stdout:removed './125/30' 2022-06-27T11:51:44.857 INFO:tasks.workunit.client.0.smithi195.stdout:removed './125/14' 2022-06-27T11:51:44.858 INFO:tasks.workunit.client.0.smithi195.stdout:removed './125/18' 2022-06-27T11:51:44.858 INFO:tasks.workunit.client.0.smithi195.stdout:removed './125/13' 2022-06-27T11:51:44.859 INFO:tasks.workunit.client.0.smithi195.stdout:removed './125/33' 2022-06-27T11:51:44.859 INFO:tasks.workunit.client.0.smithi195.stdout:removed './125/17' 2022-06-27T11:51:44.860 INFO:tasks.workunit.client.0.smithi195.stdout:removed './125/35' 2022-06-27T11:51:44.860 INFO:tasks.workunit.client.0.smithi195.stdout:removed './125/37' 2022-06-27T11:51:44.861 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './125' 2022-06-27T11:51:44.861 INFO:tasks.workunit.client.0.smithi195.stdout:removed './126/19' 2022-06-27T11:51:44.862 INFO:tasks.workunit.client.0.smithi195.stdout:removed './126/15' 2022-06-27T11:51:44.862 INFO:tasks.workunit.client.0.smithi195.stdout:removed './126/31' 2022-06-27T11:51:44.863 INFO:tasks.workunit.client.0.smithi195.stdout:removed './126/36' 2022-06-27T11:51:44.863 INFO:tasks.workunit.client.0.smithi195.stdout:removed './126/20' 2022-06-27T11:51:44.864 INFO:tasks.workunit.client.0.smithi195.stdout:removed './126/12' 2022-06-27T11:51:44.864 INFO:tasks.workunit.client.0.smithi195.stdout:removed './126/39' 2022-06-27T11:51:44.865 INFO:tasks.workunit.client.0.smithi195.stdout:removed './126/10' 2022-06-27T11:51:44.865 INFO:tasks.workunit.client.0.smithi195.stdout:removed './126/34' 2022-06-27T11:51:44.866 INFO:tasks.workunit.client.0.smithi195.stdout:removed './126/38' 2022-06-27T11:51:44.866 INFO:tasks.workunit.client.0.smithi195.stdout:removed './126/11' 2022-06-27T11:51:44.867 INFO:tasks.workunit.client.0.smithi195.stdout:removed './126/16' 2022-06-27T11:51:44.867 INFO:tasks.workunit.client.0.smithi195.stdout:removed './126/32' 2022-06-27T11:51:44.868 INFO:tasks.workunit.client.0.smithi195.stdout:removed './126/30' 2022-06-27T11:51:44.868 INFO:tasks.workunit.client.0.smithi195.stdout:removed './126/14' 2022-06-27T11:51:44.869 INFO:tasks.workunit.client.0.smithi195.stdout:removed './126/18' 2022-06-27T11:51:44.869 INFO:tasks.workunit.client.0.smithi195.stdout:removed './126/13' 2022-06-27T11:51:44.869 INFO:tasks.workunit.client.0.smithi195.stdout:removed './126/33' 2022-06-27T11:51:44.870 INFO:tasks.workunit.client.0.smithi195.stdout:removed './126/17' 2022-06-27T11:51:44.870 INFO:tasks.workunit.client.0.smithi195.stdout:removed './126/35' 2022-06-27T11:51:44.871 INFO:tasks.workunit.client.0.smithi195.stdout:removed './126/37' 2022-06-27T11:51:44.871 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './126' 2022-06-27T11:51:44.871 INFO:tasks.workunit.client.0.smithi195.stdout:removed './127/19' 2022-06-27T11:51:44.872 INFO:tasks.workunit.client.0.smithi195.stdout:removed './127/15' 2022-06-27T11:51:44.872 INFO:tasks.workunit.client.0.smithi195.stdout:removed './127/31' 2022-06-27T11:51:44.873 INFO:tasks.workunit.client.0.smithi195.stdout:removed './127/36' 2022-06-27T11:51:44.873 INFO:tasks.workunit.client.0.smithi195.stdout:removed './127/20' 2022-06-27T11:51:44.874 INFO:tasks.workunit.client.0.smithi195.stdout:removed './127/12' 2022-06-27T11:51:44.874 INFO:tasks.workunit.client.0.smithi195.stdout:removed './127/39' 2022-06-27T11:51:44.874 INFO:tasks.workunit.client.0.smithi195.stdout:removed './127/10' 2022-06-27T11:51:44.875 INFO:tasks.workunit.client.0.smithi195.stdout:removed './127/34' 2022-06-27T11:51:44.875 INFO:tasks.workunit.client.0.smithi195.stdout:removed './127/38' 2022-06-27T11:51:44.876 INFO:tasks.workunit.client.0.smithi195.stdout:removed './127/11' 2022-06-27T11:51:44.876 INFO:tasks.workunit.client.0.smithi195.stdout:removed './127/16' 2022-06-27T11:51:44.877 INFO:tasks.workunit.client.0.smithi195.stdout:removed './127/32' 2022-06-27T11:51:44.877 INFO:tasks.workunit.client.0.smithi195.stdout:removed './127/30' 2022-06-27T11:51:44.877 INFO:tasks.workunit.client.0.smithi195.stdout:removed './127/14' 2022-06-27T11:51:44.877 INFO:tasks.workunit.client.0.smithi195.stdout:removed './127/18' 2022-06-27T11:51:44.878 INFO:tasks.workunit.client.0.smithi195.stdout:removed './127/13' 2022-06-27T11:51:44.878 INFO:tasks.workunit.client.0.smithi195.stdout:removed './127/33' 2022-06-27T11:51:44.878 INFO:tasks.workunit.client.0.smithi195.stdout:removed './127/17' 2022-06-27T11:51:44.878 INFO:tasks.workunit.client.0.smithi195.stdout:removed './127/35' 2022-06-27T11:51:44.879 INFO:tasks.workunit.client.0.smithi195.stdout:removed './127/37' 2022-06-27T11:51:44.879 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './127' 2022-06-27T11:51:44.879 INFO:tasks.workunit.client.0.smithi195.stdout:removed './128/19' 2022-06-27T11:51:44.879 INFO:tasks.workunit.client.0.smithi195.stdout:removed './128/15' 2022-06-27T11:51:44.879 INFO:tasks.workunit.client.0.smithi195.stdout:removed './128/31' 2022-06-27T11:51:44.880 INFO:tasks.workunit.client.0.smithi195.stdout:removed './128/36' 2022-06-27T11:51:44.880 INFO:tasks.workunit.client.0.smithi195.stdout:removed './128/20' 2022-06-27T11:51:44.880 INFO:tasks.workunit.client.0.smithi195.stdout:removed './128/12' 2022-06-27T11:51:44.881 INFO:tasks.workunit.client.0.smithi195.stdout:removed './128/39' 2022-06-27T11:51:44.881 INFO:tasks.workunit.client.0.smithi195.stdout:removed './128/10' 2022-06-27T11:51:44.881 INFO:tasks.workunit.client.0.smithi195.stdout:removed './128/34' 2022-06-27T11:51:44.881 INFO:tasks.workunit.client.0.smithi195.stdout:removed './128/38' 2022-06-27T11:51:44.882 INFO:tasks.workunit.client.0.smithi195.stdout:removed './128/11' 2022-06-27T11:51:44.882 INFO:tasks.workunit.client.0.smithi195.stdout:removed './128/16' 2022-06-27T11:51:44.882 INFO:tasks.workunit.client.0.smithi195.stdout:removed './128/32' 2022-06-27T11:51:44.882 INFO:tasks.workunit.client.0.smithi195.stdout:removed './128/30' 2022-06-27T11:51:44.883 INFO:tasks.workunit.client.0.smithi195.stdout:removed './128/14' 2022-06-27T11:51:44.883 INFO:tasks.workunit.client.0.smithi195.stdout:removed './128/18' 2022-06-27T11:51:44.883 INFO:tasks.workunit.client.0.smithi195.stdout:removed './128/13' 2022-06-27T11:51:44.883 INFO:tasks.workunit.client.0.smithi195.stdout:removed './128/33' 2022-06-27T11:51:44.884 INFO:tasks.workunit.client.0.smithi195.stdout:removed './128/17' 2022-06-27T11:51:44.884 INFO:tasks.workunit.client.0.smithi195.stdout:removed './128/35' 2022-06-27T11:51:44.932 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.929+0000 7f9b02308700 1 -- 172.21.15.195:0/4024046035 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9afc0722c0 con 0x7f9afc06e850 2022-06-27T11:51:44.933 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.930+0000 7f9ae3fff700 1 -- 172.21.15.195:0/4024046035 <== mon.2 v2:172.21.15.202:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+6128 (secure 0 0 0) 0x7f9aec0526d0 con 0x7f9afc06e850 2022-06-27T11:51:44.934 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:51:44.935 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4760,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/1026219246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1026219246},{"type":"v1","addr":"172.21.15.202:6833","nonce":1026219246}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:51:33.013206+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4849},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4849":{"gid":4849,"name":"d","rank":0,"incarnation":15,"state":"up:active","state_seq":19,"addr":"172.21.15.202:6835/3271891211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":3271891211},{"type":"v1","addr":"172.21.15.202:6835","nonce":3271891211}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:51:44.937 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.932+0000 7f9b02308700 1 -- 172.21.15.195:0/4024046035 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9adc063d30 msgr2=0x7f9adc0661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:44.938 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.932+0000 7f9b02308700 1 --2- 172.21.15.195:0/4024046035 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9adc063d30 0x7f9adc0661e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f9afc0c5630 tx=0x7f9af0009040 comp rx=0 tx=0).stop 2022-06-27T11:51:44.938 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.932+0000 7f9b02308700 1 -- 172.21.15.195:0/4024046035 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9afc06e850 msgr2=0x7f9afc148040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:51:44.938 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.932+0000 7f9b02308700 1 --2- 172.21.15.195:0/4024046035 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9afc06e850 0x7f9afc148040 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9aec00e930 tx=0x7f9aec009060 comp rx=0 tx=0).stop 2022-06-27T11:51:44.939 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.932+0000 7f9b02308700 1 -- 172.21.15.195:0/4024046035 shutdown_connections 2022-06-27T11:51:44.941 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.932+0000 7f9b02308700 1 --2- 172.21.15.195:0/4024046035 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9adc063d30 0x7f9adc0661e0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:44.941 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.932+0000 7f9b02308700 1 --2- 172.21.15.195:0/4024046035 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9afc153250 0x7f9afc1431d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:44.942 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.932+0000 7f9b02308700 1 --2- 172.21.15.195:0/4024046035 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9afc06e850 0x7f9afc148040 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:44.942 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.932+0000 7f9b02308700 1 --2- 172.21.15.195:0/4024046035 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9afc0c34a0 0x7f9afc148580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:51:44.943 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.932+0000 7f9b02308700 1 -- 172.21.15.195:0/4024046035 >> 172.21.15.195:0/4024046035 conn(0x7f9afc0c6350 msgr2=0x7f9afc0c6720 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:51:44.943 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.933+0000 7f9b02308700 1 -- 172.21.15.195:0/4024046035 shutdown_connections 2022-06-27T11:51:44.944 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:51:44.933+0000 7f9b02308700 1 -- 172.21.15.195:0/4024046035 wait complete. 2022-06-27T11:51:44.944 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 19 2022-06-27T11:51:44.946 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2022-06-27T11:51:44.947 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 27 secs before thrashing 2022-06-27T11:51:47.224 INFO:tasks.workunit.client.0.smithi195.stdout:removed './128/37' 2022-06-27T11:51:47.225 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './128' 2022-06-27T11:51:47.225 INFO:tasks.workunit.client.0.smithi195.stdout:removed './129/19' 2022-06-27T11:51:47.225 INFO:tasks.workunit.client.0.smithi195.stdout:removed './129/15' 2022-06-27T11:51:47.226 INFO:tasks.workunit.client.0.smithi195.stdout:removed './129/31' 2022-06-27T11:51:47.226 INFO:tasks.workunit.client.0.smithi195.stdout:removed './129/36' 2022-06-27T11:51:47.226 INFO:tasks.workunit.client.0.smithi195.stdout:removed './129/20' 2022-06-27T11:51:47.226 INFO:tasks.workunit.client.0.smithi195.stdout:removed './129/12' 2022-06-27T11:51:47.227 INFO:tasks.workunit.client.0.smithi195.stdout:removed './129/39' 2022-06-27T11:51:47.227 INFO:tasks.workunit.client.0.smithi195.stdout:removed './129/10' 2022-06-27T11:51:47.227 INFO:tasks.workunit.client.0.smithi195.stdout:removed './129/34' 2022-06-27T11:51:47.227 INFO:tasks.workunit.client.0.smithi195.stdout:removed './129/38' 2022-06-27T11:51:47.228 INFO:tasks.workunit.client.0.smithi195.stdout:removed './129/11' 2022-06-27T11:51:47.228 INFO:tasks.workunit.client.0.smithi195.stdout:removed './129/16' 2022-06-27T11:51:47.228 INFO:tasks.workunit.client.0.smithi195.stdout:removed './129/32' 2022-06-27T11:51:47.229 INFO:tasks.workunit.client.0.smithi195.stdout:removed './129/30' 2022-06-27T11:51:47.229 INFO:tasks.workunit.client.0.smithi195.stdout:removed './129/14' 2022-06-27T11:51:47.229 INFO:tasks.workunit.client.0.smithi195.stdout:removed './129/18' 2022-06-27T11:51:47.230 INFO:tasks.workunit.client.0.smithi195.stdout:removed './129/13' 2022-06-27T11:51:47.230 INFO:tasks.workunit.client.0.smithi195.stdout:removed './129/33' 2022-06-27T11:51:47.230 INFO:tasks.workunit.client.0.smithi195.stdout:removed './129/17' 2022-06-27T11:51:47.230 INFO:tasks.workunit.client.0.smithi195.stdout:removed './129/35' 2022-06-27T11:51:47.231 INFO:tasks.workunit.client.0.smithi195.stdout:removed './129/37' 2022-06-27T11:51:47.231 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './129' 2022-06-27T11:51:47.231 INFO:tasks.workunit.client.0.smithi195.stdout:removed './130/19' 2022-06-27T11:51:47.232 INFO:tasks.workunit.client.0.smithi195.stdout:removed './130/15' 2022-06-27T11:51:47.232 INFO:tasks.workunit.client.0.smithi195.stdout:removed './130/31' 2022-06-27T11:51:47.232 INFO:tasks.workunit.client.0.smithi195.stdout:removed './130/36' 2022-06-27T11:51:47.232 INFO:tasks.workunit.client.0.smithi195.stdout:removed './130/20' 2022-06-27T11:51:47.233 INFO:tasks.workunit.client.0.smithi195.stdout:removed './130/12' 2022-06-27T11:51:47.233 INFO:tasks.workunit.client.0.smithi195.stdout:removed './130/39' 2022-06-27T11:51:47.233 INFO:tasks.workunit.client.0.smithi195.stdout:removed './130/10' 2022-06-27T11:51:47.233 INFO:tasks.workunit.client.0.smithi195.stdout:removed './130/34' 2022-06-27T11:51:47.233 INFO:tasks.workunit.client.0.smithi195.stdout:removed './130/38' 2022-06-27T11:51:47.234 INFO:tasks.workunit.client.0.smithi195.stdout:removed './130/11' 2022-06-27T11:51:47.234 INFO:tasks.workunit.client.0.smithi195.stdout:removed './130/16' 2022-06-27T11:51:47.234 INFO:tasks.workunit.client.0.smithi195.stdout:removed './130/32' 2022-06-27T11:51:47.234 INFO:tasks.workunit.client.0.smithi195.stdout:removed './130/30' 2022-06-27T11:51:47.235 INFO:tasks.workunit.client.0.smithi195.stdout:removed './130/14' 2022-06-27T11:51:47.235 INFO:tasks.workunit.client.0.smithi195.stdout:removed './130/18' 2022-06-27T11:51:47.236 INFO:tasks.workunit.client.0.smithi195.stdout:removed './130/13' 2022-06-27T11:51:47.236 INFO:tasks.workunit.client.0.smithi195.stdout:removed './130/33' 2022-06-27T11:51:47.236 INFO:tasks.workunit.client.0.smithi195.stdout:removed './130/17' 2022-06-27T11:51:47.237 INFO:tasks.workunit.client.0.smithi195.stdout:removed './130/35' 2022-06-27T11:51:47.237 INFO:tasks.workunit.client.0.smithi195.stdout:removed './130/37' 2022-06-27T11:51:47.238 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './130' 2022-06-27T11:51:47.238 INFO:tasks.workunit.client.0.smithi195.stdout:removed './131/19' 2022-06-27T11:51:47.238 INFO:tasks.workunit.client.0.smithi195.stdout:removed './131/15' 2022-06-27T11:51:47.239 INFO:tasks.workunit.client.0.smithi195.stdout:removed './131/31' 2022-06-27T11:51:47.239 INFO:tasks.workunit.client.0.smithi195.stdout:removed './131/36' 2022-06-27T11:51:47.240 INFO:tasks.workunit.client.0.smithi195.stdout:removed './131/20' 2022-06-27T11:51:47.240 INFO:tasks.workunit.client.0.smithi195.stdout:removed './131/12' 2022-06-27T11:51:47.240 INFO:tasks.workunit.client.0.smithi195.stdout:removed './131/39' 2022-06-27T11:51:47.241 INFO:tasks.workunit.client.0.smithi195.stdout:removed './131/10' 2022-06-27T11:51:47.241 INFO:tasks.workunit.client.0.smithi195.stdout:removed './131/34' 2022-06-27T11:51:47.241 INFO:tasks.workunit.client.0.smithi195.stdout:removed './131/38' 2022-06-27T11:51:47.241 INFO:tasks.workunit.client.0.smithi195.stdout:removed './131/11' 2022-06-27T11:51:47.242 INFO:tasks.workunit.client.0.smithi195.stdout:removed './131/16' 2022-06-27T11:51:47.242 INFO:tasks.workunit.client.0.smithi195.stdout:removed './131/32' 2022-06-27T11:51:47.242 INFO:tasks.workunit.client.0.smithi195.stdout:removed './131/30' 2022-06-27T11:51:47.243 INFO:tasks.workunit.client.0.smithi195.stdout:removed './131/14' 2022-06-27T11:51:47.243 INFO:tasks.workunit.client.0.smithi195.stdout:removed './131/18' 2022-06-27T11:51:47.243 INFO:tasks.workunit.client.0.smithi195.stdout:removed './131/13' 2022-06-27T11:51:47.243 INFO:tasks.workunit.client.0.smithi195.stdout:removed './131/33' 2022-06-27T11:51:47.243 INFO:tasks.workunit.client.0.smithi195.stdout:removed './131/17' 2022-06-27T11:51:47.244 INFO:tasks.workunit.client.0.smithi195.stdout:removed './131/35' 2022-06-27T11:51:47.244 INFO:tasks.workunit.client.0.smithi195.stdout:removed './131/37' 2022-06-27T11:51:47.244 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './131' 2022-06-27T11:51:47.244 INFO:tasks.workunit.client.0.smithi195.stdout:removed './132/19' 2022-06-27T11:51:47.245 INFO:tasks.workunit.client.0.smithi195.stdout:removed './132/15' 2022-06-27T11:51:47.245 INFO:tasks.workunit.client.0.smithi195.stdout:removed './132/31' 2022-06-27T11:51:47.245 INFO:tasks.workunit.client.0.smithi195.stdout:removed './132/36' 2022-06-27T11:51:47.245 INFO:tasks.workunit.client.0.smithi195.stdout:removed './132/20' 2022-06-27T11:51:47.245 INFO:tasks.workunit.client.0.smithi195.stdout:removed './132/12' 2022-06-27T11:51:47.246 INFO:tasks.workunit.client.0.smithi195.stdout:removed './132/39' 2022-06-27T11:51:47.246 INFO:tasks.workunit.client.0.smithi195.stdout:removed './132/10' 2022-06-27T11:51:47.246 INFO:tasks.workunit.client.0.smithi195.stdout:removed './132/34' 2022-06-27T11:51:47.246 INFO:tasks.workunit.client.0.smithi195.stdout:removed './132/38' 2022-06-27T11:51:47.247 INFO:tasks.workunit.client.0.smithi195.stdout:removed './132/11' 2022-06-27T11:51:47.247 INFO:tasks.workunit.client.0.smithi195.stdout:removed './132/16' 2022-06-27T11:51:47.247 INFO:tasks.workunit.client.0.smithi195.stdout:removed './132/32' 2022-06-27T11:51:47.247 INFO:tasks.workunit.client.0.smithi195.stdout:removed './132/30' 2022-06-27T11:51:47.247 INFO:tasks.workunit.client.0.smithi195.stdout:removed './132/14' 2022-06-27T11:51:47.248 INFO:tasks.workunit.client.0.smithi195.stdout:removed './132/18' 2022-06-27T11:51:47.248 INFO:tasks.workunit.client.0.smithi195.stdout:removed './132/13' 2022-06-27T11:51:47.248 INFO:tasks.workunit.client.0.smithi195.stdout:removed './132/33' 2022-06-27T11:51:47.248 INFO:tasks.workunit.client.0.smithi195.stdout:removed './132/17' 2022-06-27T11:51:47.249 INFO:tasks.workunit.client.0.smithi195.stdout:removed './132/35' 2022-06-27T11:51:47.249 INFO:tasks.workunit.client.0.smithi195.stdout:removed './132/37' 2022-06-27T11:51:47.249 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './132' 2022-06-27T11:51:47.249 INFO:tasks.workunit.client.0.smithi195.stdout:removed './133/19' 2022-06-27T11:51:47.249 INFO:tasks.workunit.client.0.smithi195.stdout:removed './133/15' 2022-06-27T11:51:47.250 INFO:tasks.workunit.client.0.smithi195.stdout:removed './133/31' 2022-06-27T11:51:47.250 INFO:tasks.workunit.client.0.smithi195.stdout:removed './133/36' 2022-06-27T11:51:47.250 INFO:tasks.workunit.client.0.smithi195.stdout:removed './133/20' 2022-06-27T11:51:47.250 INFO:tasks.workunit.client.0.smithi195.stdout:removed './133/12' 2022-06-27T11:51:47.250 INFO:tasks.workunit.client.0.smithi195.stdout:removed './133/39' 2022-06-27T11:51:47.251 INFO:tasks.workunit.client.0.smithi195.stdout:removed './133/10' 2022-06-27T11:51:47.251 INFO:tasks.workunit.client.0.smithi195.stdout:removed './133/34' 2022-06-27T11:51:47.251 INFO:tasks.workunit.client.0.smithi195.stdout:removed './133/38' 2022-06-27T11:51:47.251 INFO:tasks.workunit.client.0.smithi195.stdout:removed './133/11' 2022-06-27T11:51:47.252 INFO:tasks.workunit.client.0.smithi195.stdout:removed './133/16' 2022-06-27T11:51:47.252 INFO:tasks.workunit.client.0.smithi195.stdout:removed './133/32' 2022-06-27T11:51:47.252 INFO:tasks.workunit.client.0.smithi195.stdout:removed './133/30' 2022-06-27T11:51:47.252 INFO:tasks.workunit.client.0.smithi195.stdout:removed './133/14' 2022-06-27T11:51:47.252 INFO:tasks.workunit.client.0.smithi195.stdout:removed './133/18' 2022-06-27T11:51:47.253 INFO:tasks.workunit.client.0.smithi195.stdout:removed './133/13' 2022-06-27T11:51:47.253 INFO:tasks.workunit.client.0.smithi195.stdout:removed './133/33' 2022-06-27T11:51:47.253 INFO:tasks.workunit.client.0.smithi195.stdout:removed './133/17' 2022-06-27T11:51:47.253 INFO:tasks.workunit.client.0.smithi195.stdout:removed './133/35' 2022-06-27T11:51:47.254 INFO:tasks.workunit.client.0.smithi195.stdout:removed './133/37' 2022-06-27T11:51:47.254 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './133' 2022-06-27T11:51:47.254 INFO:tasks.workunit.client.0.smithi195.stdout:removed './134/19' 2022-06-27T11:51:47.254 INFO:tasks.workunit.client.0.smithi195.stdout:removed './134/15' 2022-06-27T11:51:47.254 INFO:tasks.workunit.client.0.smithi195.stdout:removed './134/31' 2022-06-27T11:51:47.255 INFO:tasks.workunit.client.0.smithi195.stdout:removed './134/36' 2022-06-27T11:51:47.255 INFO:tasks.workunit.client.0.smithi195.stdout:removed './134/20' 2022-06-27T11:51:47.255 INFO:tasks.workunit.client.0.smithi195.stdout:removed './134/12' 2022-06-27T11:51:47.255 INFO:tasks.workunit.client.0.smithi195.stdout:removed './134/39' 2022-06-27T11:51:47.256 INFO:tasks.workunit.client.0.smithi195.stdout:removed './134/10' 2022-06-27T11:51:47.256 INFO:tasks.workunit.client.0.smithi195.stdout:removed './134/34' 2022-06-27T11:51:47.256 INFO:tasks.workunit.client.0.smithi195.stdout:removed './134/38' 2022-06-27T11:51:47.256 INFO:tasks.workunit.client.0.smithi195.stdout:removed './134/11' 2022-06-27T11:51:47.257 INFO:tasks.workunit.client.0.smithi195.stdout:removed './134/16' 2022-06-27T11:51:47.257 INFO:tasks.workunit.client.0.smithi195.stdout:removed './134/32' 2022-06-27T11:51:47.257 INFO:tasks.workunit.client.0.smithi195.stdout:removed './134/30' 2022-06-27T11:51:47.257 INFO:tasks.workunit.client.0.smithi195.stdout:removed './134/14' 2022-06-27T11:51:47.258 INFO:tasks.workunit.client.0.smithi195.stdout:removed './134/18' 2022-06-27T11:51:47.258 INFO:tasks.workunit.client.0.smithi195.stdout:removed './134/13' 2022-06-27T11:51:47.258 INFO:tasks.workunit.client.0.smithi195.stdout:removed './134/33' 2022-06-27T11:51:47.258 INFO:tasks.workunit.client.0.smithi195.stdout:removed './134/17' 2022-06-27T11:51:47.258 INFO:tasks.workunit.client.0.smithi195.stdout:removed './134/35' 2022-06-27T11:51:47.259 INFO:tasks.workunit.client.0.smithi195.stdout:removed './134/37' 2022-06-27T11:51:47.259 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './134' 2022-06-27T11:51:47.259 INFO:tasks.workunit.client.0.smithi195.stdout:removed './135/19' 2022-06-27T11:51:47.259 INFO:tasks.workunit.client.0.smithi195.stdout:removed './135/15' 2022-06-27T11:51:47.260 INFO:tasks.workunit.client.0.smithi195.stdout:removed './135/31' 2022-06-27T11:51:47.260 INFO:tasks.workunit.client.0.smithi195.stdout:removed './135/36' 2022-06-27T11:51:47.260 INFO:tasks.workunit.client.0.smithi195.stdout:removed './135/20' 2022-06-27T11:51:47.260 INFO:tasks.workunit.client.0.smithi195.stdout:removed './135/12' 2022-06-27T11:51:47.260 INFO:tasks.workunit.client.0.smithi195.stdout:removed './135/39' 2022-06-27T11:51:47.261 INFO:tasks.workunit.client.0.smithi195.stdout:removed './135/10' 2022-06-27T11:51:47.261 INFO:tasks.workunit.client.0.smithi195.stdout:removed './135/34' 2022-06-27T11:51:47.261 INFO:tasks.workunit.client.0.smithi195.stdout:removed './135/38' 2022-06-27T11:51:47.261 INFO:tasks.workunit.client.0.smithi195.stdout:removed './135/11' 2022-06-27T11:51:47.262 INFO:tasks.workunit.client.0.smithi195.stdout:removed './135/16' 2022-06-27T11:51:47.262 INFO:tasks.workunit.client.0.smithi195.stdout:removed './135/32' 2022-06-27T11:51:47.262 INFO:tasks.workunit.client.0.smithi195.stdout:removed './135/30' 2022-06-27T11:51:47.262 INFO:tasks.workunit.client.0.smithi195.stdout:removed './135/14' 2022-06-27T11:51:47.262 INFO:tasks.workunit.client.0.smithi195.stdout:removed './135/18' 2022-06-27T11:51:47.263 INFO:tasks.workunit.client.0.smithi195.stdout:removed './135/13' 2022-06-27T11:51:47.263 INFO:tasks.workunit.client.0.smithi195.stdout:removed './135/33' 2022-06-27T11:51:47.263 INFO:tasks.workunit.client.0.smithi195.stdout:removed './135/17' 2022-06-27T11:51:47.263 INFO:tasks.workunit.client.0.smithi195.stdout:removed './135/35' 2022-06-27T11:51:47.264 INFO:tasks.workunit.client.0.smithi195.stdout:removed './135/37' 2022-06-27T11:51:47.264 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './135' 2022-06-27T11:51:47.264 INFO:tasks.workunit.client.0.smithi195.stdout:removed './136/19' 2022-06-27T11:51:47.264 INFO:tasks.workunit.client.0.smithi195.stdout:removed './136/15' 2022-06-27T11:51:47.264 INFO:tasks.workunit.client.0.smithi195.stdout:removed './136/31' 2022-06-27T11:51:47.265 INFO:tasks.workunit.client.0.smithi195.stdout:removed './136/36' 2022-06-27T11:51:47.265 INFO:tasks.workunit.client.0.smithi195.stdout:removed './136/20' 2022-06-27T11:51:47.265 INFO:tasks.workunit.client.0.smithi195.stdout:removed './136/12' 2022-06-27T11:51:47.265 INFO:tasks.workunit.client.0.smithi195.stdout:removed './136/39' 2022-06-27T11:51:47.265 INFO:tasks.workunit.client.0.smithi195.stdout:removed './136/10' 2022-06-27T11:51:47.266 INFO:tasks.workunit.client.0.smithi195.stdout:removed './136/34' 2022-06-27T11:51:47.266 INFO:tasks.workunit.client.0.smithi195.stdout:removed './136/38' 2022-06-27T11:51:47.266 INFO:tasks.workunit.client.0.smithi195.stdout:removed './136/11' 2022-06-27T11:51:47.266 INFO:tasks.workunit.client.0.smithi195.stdout:removed './136/16' 2022-06-27T11:51:47.266 INFO:tasks.workunit.client.0.smithi195.stdout:removed './136/32' 2022-06-27T11:51:47.267 INFO:tasks.workunit.client.0.smithi195.stdout:removed './136/30' 2022-06-27T11:51:47.267 INFO:tasks.workunit.client.0.smithi195.stdout:removed './136/14' 2022-06-27T11:51:47.267 INFO:tasks.workunit.client.0.smithi195.stdout:removed './136/18' 2022-06-27T11:51:47.267 INFO:tasks.workunit.client.0.smithi195.stdout:removed './136/13' 2022-06-27T11:51:47.268 INFO:tasks.workunit.client.0.smithi195.stdout:removed './136/33' 2022-06-27T11:51:47.268 INFO:tasks.workunit.client.0.smithi195.stdout:removed './136/17' 2022-06-27T11:51:47.268 INFO:tasks.workunit.client.0.smithi195.stdout:removed './136/35' 2022-06-27T11:51:47.268 INFO:tasks.workunit.client.0.smithi195.stdout:removed './136/37' 2022-06-27T11:51:47.268 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './136' 2022-06-27T11:51:47.269 INFO:tasks.workunit.client.0.smithi195.stdout:removed './137/19' 2022-06-27T11:51:47.269 INFO:tasks.workunit.client.0.smithi195.stdout:removed './137/15' 2022-06-27T11:51:47.269 INFO:tasks.workunit.client.0.smithi195.stdout:removed './137/31' 2022-06-27T11:51:47.269 INFO:tasks.workunit.client.0.smithi195.stdout:removed './137/36' 2022-06-27T11:51:47.269 INFO:tasks.workunit.client.0.smithi195.stdout:removed './137/20' 2022-06-27T11:51:47.270 INFO:tasks.workunit.client.0.smithi195.stdout:removed './137/12' 2022-06-27T11:51:47.270 INFO:tasks.workunit.client.0.smithi195.stdout:removed './137/39' 2022-06-27T11:51:47.270 INFO:tasks.workunit.client.0.smithi195.stdout:removed './137/10' 2022-06-27T11:51:47.270 INFO:tasks.workunit.client.0.smithi195.stdout:removed './137/34' 2022-06-27T11:51:47.271 INFO:tasks.workunit.client.0.smithi195.stdout:removed './137/38' 2022-06-27T11:51:47.271 INFO:tasks.workunit.client.0.smithi195.stdout:removed './137/11' 2022-06-27T11:51:47.271 INFO:tasks.workunit.client.0.smithi195.stdout:removed './137/16' 2022-06-27T11:51:47.271 INFO:tasks.workunit.client.0.smithi195.stdout:removed './137/32' 2022-06-27T11:51:47.272 INFO:tasks.workunit.client.0.smithi195.stdout:removed './137/30' 2022-06-27T11:51:47.272 INFO:tasks.workunit.client.0.smithi195.stdout:removed './137/14' 2022-06-27T11:51:47.272 INFO:tasks.workunit.client.0.smithi195.stdout:removed './137/18' 2022-06-27T11:51:47.272 INFO:tasks.workunit.client.0.smithi195.stdout:removed './137/13' 2022-06-27T11:51:47.273 INFO:tasks.workunit.client.0.smithi195.stdout:removed './137/33' 2022-06-27T11:51:47.273 INFO:tasks.workunit.client.0.smithi195.stdout:removed './137/17' 2022-06-27T11:51:47.273 INFO:tasks.workunit.client.0.smithi195.stdout:removed './137/35' 2022-06-27T11:51:47.273 INFO:tasks.workunit.client.0.smithi195.stdout:removed './137/37' 2022-06-27T11:51:47.273 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './137' 2022-06-27T11:51:47.274 INFO:tasks.workunit.client.0.smithi195.stdout:removed './138/19' 2022-06-27T11:51:47.274 INFO:tasks.workunit.client.0.smithi195.stdout:removed './138/15' 2022-06-27T11:51:47.274 INFO:tasks.workunit.client.0.smithi195.stdout:removed './138/31' 2022-06-27T11:51:47.274 INFO:tasks.workunit.client.0.smithi195.stdout:removed './138/36' 2022-06-27T11:51:47.275 INFO:tasks.workunit.client.0.smithi195.stdout:removed './138/20' 2022-06-27T11:51:47.275 INFO:tasks.workunit.client.0.smithi195.stdout:removed './138/12' 2022-06-27T11:51:47.275 INFO:tasks.workunit.client.0.smithi195.stdout:removed './138/39' 2022-06-27T11:51:47.275 INFO:tasks.workunit.client.0.smithi195.stdout:removed './138/10' 2022-06-27T11:51:47.276 INFO:tasks.workunit.client.0.smithi195.stdout:removed './138/34' 2022-06-27T11:51:47.276 INFO:tasks.workunit.client.0.smithi195.stdout:removed './138/38' 2022-06-27T11:51:47.276 INFO:tasks.workunit.client.0.smithi195.stdout:removed './138/11' 2022-06-27T11:51:47.276 INFO:tasks.workunit.client.0.smithi195.stdout:removed './138/16' 2022-06-27T11:51:49.192 INFO:tasks.workunit.client.0.smithi195.stdout:removed './138/32' 2022-06-27T11:51:49.192 INFO:tasks.workunit.client.0.smithi195.stdout:removed './138/30' 2022-06-27T11:51:49.192 INFO:tasks.workunit.client.0.smithi195.stdout:removed './138/14' 2022-06-27T11:51:49.193 INFO:tasks.workunit.client.0.smithi195.stdout:removed './138/18' 2022-06-27T11:51:49.193 INFO:tasks.workunit.client.0.smithi195.stdout:removed './138/13' 2022-06-27T11:51:49.193 INFO:tasks.workunit.client.0.smithi195.stdout:removed './138/33' 2022-06-27T11:51:49.194 INFO:tasks.workunit.client.0.smithi195.stdout:removed './138/17' 2022-06-27T11:51:49.194 INFO:tasks.workunit.client.0.smithi195.stdout:removed './138/35' 2022-06-27T11:51:49.194 INFO:tasks.workunit.client.0.smithi195.stdout:removed './138/37' 2022-06-27T11:51:49.195 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './138' 2022-06-27T11:51:49.195 INFO:tasks.workunit.client.0.smithi195.stdout:removed './139/19' 2022-06-27T11:51:49.196 INFO:tasks.workunit.client.0.smithi195.stdout:removed './139/15' 2022-06-27T11:51:49.197 INFO:tasks.workunit.client.0.smithi195.stdout:removed './139/31' 2022-06-27T11:51:49.197 INFO:tasks.workunit.client.0.smithi195.stdout:removed './139/36' 2022-06-27T11:51:49.197 INFO:tasks.workunit.client.0.smithi195.stdout:removed './139/20' 2022-06-27T11:51:49.197 INFO:tasks.workunit.client.0.smithi195.stdout:removed './139/12' 2022-06-27T11:51:49.198 INFO:tasks.workunit.client.0.smithi195.stdout:removed './139/39' 2022-06-27T11:51:49.198 INFO:tasks.workunit.client.0.smithi195.stdout:removed './139/10' 2022-06-27T11:51:49.198 INFO:tasks.workunit.client.0.smithi195.stdout:removed './139/34' 2022-06-27T11:51:49.198 INFO:tasks.workunit.client.0.smithi195.stdout:removed './139/38' 2022-06-27T11:51:49.199 INFO:tasks.workunit.client.0.smithi195.stdout:removed './139/11' 2022-06-27T11:51:49.199 INFO:tasks.workunit.client.0.smithi195.stdout:removed './139/16' 2022-06-27T11:51:49.199 INFO:tasks.workunit.client.0.smithi195.stdout:removed './139/32' 2022-06-27T11:51:49.199 INFO:tasks.workunit.client.0.smithi195.stdout:removed './139/30' 2022-06-27T11:51:49.200 INFO:tasks.workunit.client.0.smithi195.stdout:removed './139/14' 2022-06-27T11:51:49.200 INFO:tasks.workunit.client.0.smithi195.stdout:removed './139/18' 2022-06-27T11:51:49.201 INFO:tasks.workunit.client.0.smithi195.stdout:removed './139/13' 2022-06-27T11:51:49.201 INFO:tasks.workunit.client.0.smithi195.stdout:removed './139/33' 2022-06-27T11:51:49.201 INFO:tasks.workunit.client.0.smithi195.stdout:removed './139/17' 2022-06-27T11:51:49.201 INFO:tasks.workunit.client.0.smithi195.stdout:removed './139/35' 2022-06-27T11:51:49.202 INFO:tasks.workunit.client.0.smithi195.stdout:removed './139/37' 2022-06-27T11:51:49.202 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './139' 2022-06-27T11:51:49.202 INFO:tasks.workunit.client.0.smithi195.stdout:removed './140/19' 2022-06-27T11:51:49.202 INFO:tasks.workunit.client.0.smithi195.stdout:removed './140/15' 2022-06-27T11:51:49.203 INFO:tasks.workunit.client.0.smithi195.stdout:removed './140/31' 2022-06-27T11:51:49.203 INFO:tasks.workunit.client.0.smithi195.stdout:removed './140/36' 2022-06-27T11:51:49.203 INFO:tasks.workunit.client.0.smithi195.stdout:removed './140/20' 2022-06-27T11:51:49.204 INFO:tasks.workunit.client.0.smithi195.stdout:removed './140/12' 2022-06-27T11:51:49.204 INFO:tasks.workunit.client.0.smithi195.stdout:removed './140/39' 2022-06-27T11:51:49.204 INFO:tasks.workunit.client.0.smithi195.stdout:removed './140/10' 2022-06-27T11:51:49.205 INFO:tasks.workunit.client.0.smithi195.stdout:removed './140/34' 2022-06-27T11:51:49.205 INFO:tasks.workunit.client.0.smithi195.stdout:removed './140/38' 2022-06-27T11:51:49.205 INFO:tasks.workunit.client.0.smithi195.stdout:removed './140/11' 2022-06-27T11:51:49.205 INFO:tasks.workunit.client.0.smithi195.stdout:removed './140/16' 2022-06-27T11:51:49.206 INFO:tasks.workunit.client.0.smithi195.stdout:removed './140/32' 2022-06-27T11:51:49.206 INFO:tasks.workunit.client.0.smithi195.stdout:removed './140/30' 2022-06-27T11:51:49.206 INFO:tasks.workunit.client.0.smithi195.stdout:removed './140/14' 2022-06-27T11:51:49.206 INFO:tasks.workunit.client.0.smithi195.stdout:removed './140/18' 2022-06-27T11:51:49.207 INFO:tasks.workunit.client.0.smithi195.stdout:removed './140/13' 2022-06-27T11:51:49.207 INFO:tasks.workunit.client.0.smithi195.stdout:removed './140/33' 2022-06-27T11:51:49.207 INFO:tasks.workunit.client.0.smithi195.stdout:removed './140/17' 2022-06-27T11:51:49.208 INFO:tasks.workunit.client.0.smithi195.stdout:removed './140/35' 2022-06-27T11:51:49.208 INFO:tasks.workunit.client.0.smithi195.stdout:removed './140/37' 2022-06-27T11:51:49.208 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './140' 2022-06-27T11:51:49.208 INFO:tasks.workunit.client.0.smithi195.stdout:removed './141/19' 2022-06-27T11:51:49.209 INFO:tasks.workunit.client.0.smithi195.stdout:removed './141/15' 2022-06-27T11:51:49.209 INFO:tasks.workunit.client.0.smithi195.stdout:removed './141/31' 2022-06-27T11:51:49.209 INFO:tasks.workunit.client.0.smithi195.stdout:removed './141/36' 2022-06-27T11:51:49.209 INFO:tasks.workunit.client.0.smithi195.stdout:removed './141/20' 2022-06-27T11:51:49.210 INFO:tasks.workunit.client.0.smithi195.stdout:removed './141/12' 2022-06-27T11:51:49.210 INFO:tasks.workunit.client.0.smithi195.stdout:removed './141/39' 2022-06-27T11:51:49.210 INFO:tasks.workunit.client.0.smithi195.stdout:removed './141/10' 2022-06-27T11:51:49.210 INFO:tasks.workunit.client.0.smithi195.stdout:removed './141/34' 2022-06-27T11:51:49.211 INFO:tasks.workunit.client.0.smithi195.stdout:removed './141/38' 2022-06-27T11:51:49.211 INFO:tasks.workunit.client.0.smithi195.stdout:removed './141/11' 2022-06-27T11:51:49.211 INFO:tasks.workunit.client.0.smithi195.stdout:removed './141/16' 2022-06-27T11:51:49.211 INFO:tasks.workunit.client.0.smithi195.stdout:removed './141/32' 2022-06-27T11:51:49.211 INFO:tasks.workunit.client.0.smithi195.stdout:removed './141/30' 2022-06-27T11:51:49.212 INFO:tasks.workunit.client.0.smithi195.stdout:removed './141/14' 2022-06-27T11:51:49.212 INFO:tasks.workunit.client.0.smithi195.stdout:removed './141/18' 2022-06-27T11:51:49.212 INFO:tasks.workunit.client.0.smithi195.stdout:removed './141/13' 2022-06-27T11:51:49.212 INFO:tasks.workunit.client.0.smithi195.stdout:removed './141/33' 2022-06-27T11:51:49.213 INFO:tasks.workunit.client.0.smithi195.stdout:removed './141/17' 2022-06-27T11:51:49.213 INFO:tasks.workunit.client.0.smithi195.stdout:removed './141/35' 2022-06-27T11:51:49.213 INFO:tasks.workunit.client.0.smithi195.stdout:removed './141/37' 2022-06-27T11:51:49.213 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './141' 2022-06-27T11:51:49.214 INFO:tasks.workunit.client.0.smithi195.stdout:removed './142/19' 2022-06-27T11:51:49.214 INFO:tasks.workunit.client.0.smithi195.stdout:removed './142/15' 2022-06-27T11:51:49.214 INFO:tasks.workunit.client.0.smithi195.stdout:removed './142/31' 2022-06-27T11:51:49.214 INFO:tasks.workunit.client.0.smithi195.stdout:removed './142/36' 2022-06-27T11:51:49.215 INFO:tasks.workunit.client.0.smithi195.stdout:removed './142/20' 2022-06-27T11:51:49.215 INFO:tasks.workunit.client.0.smithi195.stdout:removed './142/12' 2022-06-27T11:51:49.215 INFO:tasks.workunit.client.0.smithi195.stdout:removed './142/39' 2022-06-27T11:51:49.215 INFO:tasks.workunit.client.0.smithi195.stdout:removed './142/10' 2022-06-27T11:51:49.215 INFO:tasks.workunit.client.0.smithi195.stdout:removed './142/34' 2022-06-27T11:51:49.216 INFO:tasks.workunit.client.0.smithi195.stdout:removed './142/38' 2022-06-27T11:51:49.216 INFO:tasks.workunit.client.0.smithi195.stdout:removed './142/11' 2022-06-27T11:51:49.216 INFO:tasks.workunit.client.0.smithi195.stdout:removed './142/16' 2022-06-27T11:51:49.216 INFO:tasks.workunit.client.0.smithi195.stdout:removed './142/32' 2022-06-27T11:51:49.217 INFO:tasks.workunit.client.0.smithi195.stdout:removed './142/30' 2022-06-27T11:51:49.217 INFO:tasks.workunit.client.0.smithi195.stdout:removed './142/14' 2022-06-27T11:51:49.217 INFO:tasks.workunit.client.0.smithi195.stdout:removed './142/18' 2022-06-27T11:51:49.218 INFO:tasks.workunit.client.0.smithi195.stdout:removed './142/13' 2022-06-27T11:51:49.218 INFO:tasks.workunit.client.0.smithi195.stdout:removed './142/33' 2022-06-27T11:51:49.218 INFO:tasks.workunit.client.0.smithi195.stdout:removed './142/17' 2022-06-27T11:51:49.219 INFO:tasks.workunit.client.0.smithi195.stdout:removed './142/35' 2022-06-27T11:51:49.219 INFO:tasks.workunit.client.0.smithi195.stdout:removed './142/37' 2022-06-27T11:51:49.219 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './142' 2022-06-27T11:51:49.219 INFO:tasks.workunit.client.0.smithi195.stdout:removed './143/19' 2022-06-27T11:51:49.219 INFO:tasks.workunit.client.0.smithi195.stdout:removed './143/15' 2022-06-27T11:51:49.220 INFO:tasks.workunit.client.0.smithi195.stdout:removed './143/31' 2022-06-27T11:51:49.220 INFO:tasks.workunit.client.0.smithi195.stdout:removed './143/36' 2022-06-27T11:51:49.220 INFO:tasks.workunit.client.0.smithi195.stdout:removed './143/20' 2022-06-27T11:51:49.220 INFO:tasks.workunit.client.0.smithi195.stdout:removed './143/12' 2022-06-27T11:51:49.221 INFO:tasks.workunit.client.0.smithi195.stdout:removed './143/39' 2022-06-27T11:51:49.221 INFO:tasks.workunit.client.0.smithi195.stdout:removed './143/10' 2022-06-27T11:51:49.221 INFO:tasks.workunit.client.0.smithi195.stdout:removed './143/34' 2022-06-27T11:51:49.221 INFO:tasks.workunit.client.0.smithi195.stdout:removed './143/38' 2022-06-27T11:51:49.221 INFO:tasks.workunit.client.0.smithi195.stdout:removed './143/11' 2022-06-27T11:51:49.222 INFO:tasks.workunit.client.0.smithi195.stdout:removed './143/16' 2022-06-27T11:51:49.222 INFO:tasks.workunit.client.0.smithi195.stdout:removed './143/32' 2022-06-27T11:51:49.222 INFO:tasks.workunit.client.0.smithi195.stdout:removed './143/30' 2022-06-27T11:51:49.222 INFO:tasks.workunit.client.0.smithi195.stdout:removed './143/14' 2022-06-27T11:51:49.222 INFO:tasks.workunit.client.0.smithi195.stdout:removed './143/18' 2022-06-27T11:51:49.223 INFO:tasks.workunit.client.0.smithi195.stdout:removed './143/13' 2022-06-27T11:51:49.223 INFO:tasks.workunit.client.0.smithi195.stdout:removed './143/33' 2022-06-27T11:51:49.223 INFO:tasks.workunit.client.0.smithi195.stdout:removed './143/17' 2022-06-27T11:51:49.223 INFO:tasks.workunit.client.0.smithi195.stdout:removed './143/35' 2022-06-27T11:51:49.224 INFO:tasks.workunit.client.0.smithi195.stdout:removed './143/37' 2022-06-27T11:51:49.224 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './143' 2022-06-27T11:51:49.224 INFO:tasks.workunit.client.0.smithi195.stdout:removed './144/19' 2022-06-27T11:51:49.224 INFO:tasks.workunit.client.0.smithi195.stdout:removed './144/15' 2022-06-27T11:51:49.225 INFO:tasks.workunit.client.0.smithi195.stdout:removed './144/31' 2022-06-27T11:51:49.225 INFO:tasks.workunit.client.0.smithi195.stdout:removed './144/36' 2022-06-27T11:51:49.225 INFO:tasks.workunit.client.0.smithi195.stdout:removed './144/20' 2022-06-27T11:51:49.225 INFO:tasks.workunit.client.0.smithi195.stdout:removed './144/12' 2022-06-27T11:51:49.225 INFO:tasks.workunit.client.0.smithi195.stdout:removed './144/39' 2022-06-27T11:51:49.226 INFO:tasks.workunit.client.0.smithi195.stdout:removed './144/10' 2022-06-27T11:51:49.226 INFO:tasks.workunit.client.0.smithi195.stdout:removed './144/34' 2022-06-27T11:51:49.226 INFO:tasks.workunit.client.0.smithi195.stdout:removed './144/38' 2022-06-27T11:51:49.226 INFO:tasks.workunit.client.0.smithi195.stdout:removed './144/11' 2022-06-27T11:51:49.227 INFO:tasks.workunit.client.0.smithi195.stdout:removed './144/16' 2022-06-27T11:51:49.227 INFO:tasks.workunit.client.0.smithi195.stdout:removed './144/32' 2022-06-27T11:51:49.227 INFO:tasks.workunit.client.0.smithi195.stdout:removed './144/30' 2022-06-27T11:51:49.227 INFO:tasks.workunit.client.0.smithi195.stdout:removed './144/14' 2022-06-27T11:51:49.228 INFO:tasks.workunit.client.0.smithi195.stdout:removed './144/18' 2022-06-27T11:51:49.228 INFO:tasks.workunit.client.0.smithi195.stdout:removed './144/13' 2022-06-27T11:51:49.228 INFO:tasks.workunit.client.0.smithi195.stdout:removed './144/33' 2022-06-27T11:51:49.228 INFO:tasks.workunit.client.0.smithi195.stdout:removed './144/17' 2022-06-27T11:51:49.229 INFO:tasks.workunit.client.0.smithi195.stdout:removed './144/35' 2022-06-27T11:51:49.229 INFO:tasks.workunit.client.0.smithi195.stdout:removed './144/37' 2022-06-27T11:51:49.229 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './144' 2022-06-27T11:51:49.229 INFO:tasks.workunit.client.0.smithi195.stdout:removed './145/19' 2022-06-27T11:51:49.230 INFO:tasks.workunit.client.0.smithi195.stdout:removed './145/15' 2022-06-27T11:51:49.230 INFO:tasks.workunit.client.0.smithi195.stdout:removed './145/31' 2022-06-27T11:51:49.230 INFO:tasks.workunit.client.0.smithi195.stdout:removed './145/36' 2022-06-27T11:51:49.230 INFO:tasks.workunit.client.0.smithi195.stdout:removed './145/20' 2022-06-27T11:51:49.231 INFO:tasks.workunit.client.0.smithi195.stdout:removed './145/12' 2022-06-27T11:51:49.231 INFO:tasks.workunit.client.0.smithi195.stdout:removed './145/39' 2022-06-27T11:51:49.231 INFO:tasks.workunit.client.0.smithi195.stdout:removed './145/10' 2022-06-27T11:51:49.231 INFO:tasks.workunit.client.0.smithi195.stdout:removed './145/34' 2022-06-27T11:51:49.231 INFO:tasks.workunit.client.0.smithi195.stdout:removed './145/38' 2022-06-27T11:51:49.232 INFO:tasks.workunit.client.0.smithi195.stdout:removed './145/11' 2022-06-27T11:51:49.232 INFO:tasks.workunit.client.0.smithi195.stdout:removed './145/16' 2022-06-27T11:51:49.232 INFO:tasks.workunit.client.0.smithi195.stdout:removed './145/32' 2022-06-27T11:51:49.233 INFO:tasks.workunit.client.0.smithi195.stdout:removed './145/30' 2022-06-27T11:51:49.233 INFO:tasks.workunit.client.0.smithi195.stdout:removed './145/14' 2022-06-27T11:51:49.233 INFO:tasks.workunit.client.0.smithi195.stdout:removed './145/18' 2022-06-27T11:51:49.233 INFO:tasks.workunit.client.0.smithi195.stdout:removed './145/13' 2022-06-27T11:51:49.234 INFO:tasks.workunit.client.0.smithi195.stdout:removed './145/33' 2022-06-27T11:51:49.234 INFO:tasks.workunit.client.0.smithi195.stdout:removed './145/17' 2022-06-27T11:51:49.234 INFO:tasks.workunit.client.0.smithi195.stdout:removed './145/35' 2022-06-27T11:51:49.234 INFO:tasks.workunit.client.0.smithi195.stdout:removed './145/37' 2022-06-27T11:51:49.235 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './145' 2022-06-27T11:51:49.235 INFO:tasks.workunit.client.0.smithi195.stdout:removed './146/19' 2022-06-27T11:51:49.235 INFO:tasks.workunit.client.0.smithi195.stdout:removed './146/15' 2022-06-27T11:51:49.235 INFO:tasks.workunit.client.0.smithi195.stdout:removed './146/31' 2022-06-27T11:51:49.235 INFO:tasks.workunit.client.0.smithi195.stdout:removed './146/36' 2022-06-27T11:51:49.236 INFO:tasks.workunit.client.0.smithi195.stdout:removed './146/20' 2022-06-27T11:51:49.236 INFO:tasks.workunit.client.0.smithi195.stdout:removed './146/12' 2022-06-27T11:51:49.236 INFO:tasks.workunit.client.0.smithi195.stdout:removed './146/39' 2022-06-27T11:51:49.236 INFO:tasks.workunit.client.0.smithi195.stdout:removed './146/10' 2022-06-27T11:51:49.237 INFO:tasks.workunit.client.0.smithi195.stdout:removed './146/34' 2022-06-27T11:51:49.237 INFO:tasks.workunit.client.0.smithi195.stdout:removed './146/38' 2022-06-27T11:51:49.237 INFO:tasks.workunit.client.0.smithi195.stdout:removed './146/11' 2022-06-27T11:51:49.237 INFO:tasks.workunit.client.0.smithi195.stdout:removed './146/16' 2022-06-27T11:51:49.237 INFO:tasks.workunit.client.0.smithi195.stdout:removed './146/32' 2022-06-27T11:51:49.238 INFO:tasks.workunit.client.0.smithi195.stdout:removed './146/30' 2022-06-27T11:51:49.238 INFO:tasks.workunit.client.0.smithi195.stdout:removed './146/14' 2022-06-27T11:51:49.238 INFO:tasks.workunit.client.0.smithi195.stdout:removed './146/18' 2022-06-27T11:51:49.238 INFO:tasks.workunit.client.0.smithi195.stdout:removed './146/13' 2022-06-27T11:51:49.239 INFO:tasks.workunit.client.0.smithi195.stdout:removed './146/33' 2022-06-27T11:51:49.239 INFO:tasks.workunit.client.0.smithi195.stdout:removed './146/17' 2022-06-27T11:51:49.239 INFO:tasks.workunit.client.0.smithi195.stdout:removed './146/35' 2022-06-27T11:51:49.239 INFO:tasks.workunit.client.0.smithi195.stdout:removed './146/37' 2022-06-27T11:51:49.239 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './146' 2022-06-27T11:51:49.240 INFO:tasks.workunit.client.0.smithi195.stdout:removed './147/19' 2022-06-27T11:51:49.240 INFO:tasks.workunit.client.0.smithi195.stdout:removed './147/15' 2022-06-27T11:51:49.240 INFO:tasks.workunit.client.0.smithi195.stdout:removed './147/31' 2022-06-27T11:51:49.240 INFO:tasks.workunit.client.0.smithi195.stdout:removed './147/36' 2022-06-27T11:51:49.241 INFO:tasks.workunit.client.0.smithi195.stdout:removed './147/20' 2022-06-27T11:51:49.241 INFO:tasks.workunit.client.0.smithi195.stdout:removed './147/12' 2022-06-27T11:51:49.241 INFO:tasks.workunit.client.0.smithi195.stdout:removed './147/39' 2022-06-27T11:51:49.241 INFO:tasks.workunit.client.0.smithi195.stdout:removed './147/10' 2022-06-27T11:51:49.241 INFO:tasks.workunit.client.0.smithi195.stdout:removed './147/34' 2022-06-27T11:51:49.242 INFO:tasks.workunit.client.0.smithi195.stdout:removed './147/38' 2022-06-27T11:51:49.242 INFO:tasks.workunit.client.0.smithi195.stdout:removed './147/11' 2022-06-27T11:51:49.242 INFO:tasks.workunit.client.0.smithi195.stdout:removed './147/16' 2022-06-27T11:51:49.242 INFO:tasks.workunit.client.0.smithi195.stdout:removed './147/32' 2022-06-27T11:51:49.243 INFO:tasks.workunit.client.0.smithi195.stdout:removed './147/30' 2022-06-27T11:51:49.243 INFO:tasks.workunit.client.0.smithi195.stdout:removed './147/14' 2022-06-27T11:51:49.243 INFO:tasks.workunit.client.0.smithi195.stdout:removed './147/18' 2022-06-27T11:51:49.244 INFO:tasks.workunit.client.0.smithi195.stdout:removed './147/13' 2022-06-27T11:51:49.244 INFO:tasks.workunit.client.0.smithi195.stdout:removed './147/33' 2022-06-27T11:51:49.244 INFO:tasks.workunit.client.0.smithi195.stdout:removed './147/17' 2022-06-27T11:51:49.244 INFO:tasks.workunit.client.0.smithi195.stdout:removed './147/35' 2022-06-27T11:51:49.245 INFO:tasks.workunit.client.0.smithi195.stdout:removed './147/37' 2022-06-27T11:51:49.245 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './147' 2022-06-27T11:51:49.245 INFO:tasks.workunit.client.0.smithi195.stdout:removed './148/19' 2022-06-27T11:51:49.245 INFO:tasks.workunit.client.0.smithi195.stdout:removed './148/15' 2022-06-27T11:51:49.246 INFO:tasks.workunit.client.0.smithi195.stdout:removed './148/31' 2022-06-27T11:51:49.246 INFO:tasks.workunit.client.0.smithi195.stdout:removed './148/36' 2022-06-27T11:51:50.649 INFO:tasks.workunit.client.0.smithi195.stdout:removed './148/20' 2022-06-27T11:51:50.650 INFO:tasks.workunit.client.0.smithi195.stdout:removed './148/12' 2022-06-27T11:51:50.650 INFO:tasks.workunit.client.0.smithi195.stdout:removed './148/39' 2022-06-27T11:51:50.651 INFO:tasks.workunit.client.0.smithi195.stdout:removed './148/10' 2022-06-27T11:51:50.652 INFO:tasks.workunit.client.0.smithi195.stdout:removed './148/34' 2022-06-27T11:51:50.652 INFO:tasks.workunit.client.0.smithi195.stdout:removed './148/38' 2022-06-27T11:51:50.652 INFO:tasks.workunit.client.0.smithi195.stdout:removed './148/11' 2022-06-27T11:51:50.653 INFO:tasks.workunit.client.0.smithi195.stdout:removed './148/16' 2022-06-27T11:51:50.653 INFO:tasks.workunit.client.0.smithi195.stdout:removed './148/32' 2022-06-27T11:51:50.654 INFO:tasks.workunit.client.0.smithi195.stdout:removed './148/30' 2022-06-27T11:51:50.654 INFO:tasks.workunit.client.0.smithi195.stdout:removed './148/14' 2022-06-27T11:51:50.655 INFO:tasks.workunit.client.0.smithi195.stdout:removed './148/18' 2022-06-27T11:51:50.655 INFO:tasks.workunit.client.0.smithi195.stdout:removed './148/13' 2022-06-27T11:51:50.656 INFO:tasks.workunit.client.0.smithi195.stdout:removed './148/33' 2022-06-27T11:51:50.656 INFO:tasks.workunit.client.0.smithi195.stdout:removed './148/17' 2022-06-27T11:51:50.657 INFO:tasks.workunit.client.0.smithi195.stdout:removed './148/35' 2022-06-27T11:51:50.657 INFO:tasks.workunit.client.0.smithi195.stdout:removed './148/37' 2022-06-27T11:51:50.658 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './148' 2022-06-27T11:51:50.658 INFO:tasks.workunit.client.0.smithi195.stdout:removed './149/19' 2022-06-27T11:51:50.659 INFO:tasks.workunit.client.0.smithi195.stdout:removed './149/15' 2022-06-27T11:51:50.660 INFO:tasks.workunit.client.0.smithi195.stdout:removed './149/31' 2022-06-27T11:51:50.660 INFO:tasks.workunit.client.0.smithi195.stdout:removed './149/36' 2022-06-27T11:51:50.661 INFO:tasks.workunit.client.0.smithi195.stdout:removed './149/20' 2022-06-27T11:51:50.661 INFO:tasks.workunit.client.0.smithi195.stdout:removed './149/12' 2022-06-27T11:51:50.662 INFO:tasks.workunit.client.0.smithi195.stdout:removed './149/39' 2022-06-27T11:51:50.663 INFO:tasks.workunit.client.0.smithi195.stdout:removed './149/10' 2022-06-27T11:51:50.663 INFO:tasks.workunit.client.0.smithi195.stdout:removed './149/34' 2022-06-27T11:51:50.664 INFO:tasks.workunit.client.0.smithi195.stdout:removed './149/38' 2022-06-27T11:51:50.664 INFO:tasks.workunit.client.0.smithi195.stdout:removed './149/11' 2022-06-27T11:51:50.665 INFO:tasks.workunit.client.0.smithi195.stdout:removed './149/16' 2022-06-27T11:51:50.665 INFO:tasks.workunit.client.0.smithi195.stdout:removed './149/32' 2022-06-27T11:51:50.665 INFO:tasks.workunit.client.0.smithi195.stdout:removed './149/30' 2022-06-27T11:51:50.666 INFO:tasks.workunit.client.0.smithi195.stdout:removed './149/14' 2022-06-27T11:51:50.666 INFO:tasks.workunit.client.0.smithi195.stdout:removed './149/18' 2022-06-27T11:51:50.667 INFO:tasks.workunit.client.0.smithi195.stdout:removed './149/13' 2022-06-27T11:51:50.667 INFO:tasks.workunit.client.0.smithi195.stdout:removed './149/33' 2022-06-27T11:51:50.668 INFO:tasks.workunit.client.0.smithi195.stdout:removed './149/17' 2022-06-27T11:51:50.668 INFO:tasks.workunit.client.0.smithi195.stdout:removed './149/35' 2022-06-27T11:51:50.669 INFO:tasks.workunit.client.0.smithi195.stdout:removed './149/37' 2022-06-27T11:51:50.669 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './149' 2022-06-27T11:51:50.680 INFO:tasks.workunit.client.0.smithi195.stdout:removed './150/19' 2022-06-27T11:51:50.690 INFO:tasks.workunit.client.0.smithi195.stdout:removed './150/15' 2022-06-27T11:51:50.690 INFO:tasks.workunit.client.0.smithi195.stdout:removed './150/31' 2022-06-27T11:51:50.691 INFO:tasks.workunit.client.0.smithi195.stdout:removed './150/36' 2022-06-27T11:51:50.694 INFO:tasks.workunit.client.0.smithi195.stdout:removed './150/20' 2022-06-27T11:51:50.695 INFO:tasks.workunit.client.0.smithi195.stdout:removed './150/12' 2022-06-27T11:51:50.695 INFO:tasks.workunit.client.0.smithi195.stdout:removed './150/39' 2022-06-27T11:51:50.696 INFO:tasks.workunit.client.0.smithi195.stdout:removed './150/10' 2022-06-27T11:51:50.696 INFO:tasks.workunit.client.0.smithi195.stdout:removed './150/34' 2022-06-27T11:51:50.696 INFO:tasks.workunit.client.0.smithi195.stdout:removed './150/38' 2022-06-27T11:51:50.697 INFO:tasks.workunit.client.0.smithi195.stdout:removed './150/11' 2022-06-27T11:51:50.697 INFO:tasks.workunit.client.0.smithi195.stdout:removed './150/16' 2022-06-27T11:51:50.698 INFO:tasks.workunit.client.0.smithi195.stdout:removed './150/32' 2022-06-27T11:51:50.698 INFO:tasks.workunit.client.0.smithi195.stdout:removed './150/30' 2022-06-27T11:51:50.699 INFO:tasks.workunit.client.0.smithi195.stdout:removed './150/14' 2022-06-27T11:51:50.700 INFO:tasks.workunit.client.0.smithi195.stdout:removed './150/18' 2022-06-27T11:51:50.701 INFO:tasks.workunit.client.0.smithi195.stdout:removed './150/13' 2022-06-27T11:51:50.701 INFO:tasks.workunit.client.0.smithi195.stdout:removed './150/33' 2022-06-27T11:51:50.702 INFO:tasks.workunit.client.0.smithi195.stdout:removed './150/17' 2022-06-27T11:51:50.702 INFO:tasks.workunit.client.0.smithi195.stdout:removed './150/35' 2022-06-27T11:51:50.703 INFO:tasks.workunit.client.0.smithi195.stdout:removed './150/37' 2022-06-27T11:51:50.703 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './150' 2022-06-27T11:51:50.704 INFO:tasks.workunit.client.0.smithi195.stdout:removed './151/19' 2022-06-27T11:51:50.704 INFO:tasks.workunit.client.0.smithi195.stdout:removed './151/15' 2022-06-27T11:51:50.704 INFO:tasks.workunit.client.0.smithi195.stdout:removed './151/31' 2022-06-27T11:51:50.705 INFO:tasks.workunit.client.0.smithi195.stdout:removed './151/36' 2022-06-27T11:51:50.706 INFO:tasks.workunit.client.0.smithi195.stdout:removed './151/20' 2022-06-27T11:51:50.706 INFO:tasks.workunit.client.0.smithi195.stdout:removed './151/12' 2022-06-27T11:51:50.706 INFO:tasks.workunit.client.0.smithi195.stdout:removed './151/39' 2022-06-27T11:51:50.707 INFO:tasks.workunit.client.0.smithi195.stdout:removed './151/10' 2022-06-27T11:51:50.707 INFO:tasks.workunit.client.0.smithi195.stdout:removed './151/34' 2022-06-27T11:51:50.707 INFO:tasks.workunit.client.0.smithi195.stdout:removed './151/38' 2022-06-27T11:51:50.708 INFO:tasks.workunit.client.0.smithi195.stdout:removed './151/11' 2022-06-27T11:51:50.708 INFO:tasks.workunit.client.0.smithi195.stdout:removed './151/16' 2022-06-27T11:51:50.708 INFO:tasks.workunit.client.0.smithi195.stdout:removed './151/32' 2022-06-27T11:51:50.709 INFO:tasks.workunit.client.0.smithi195.stdout:removed './151/30' 2022-06-27T11:51:50.709 INFO:tasks.workunit.client.0.smithi195.stdout:removed './151/14' 2022-06-27T11:51:50.710 INFO:tasks.workunit.client.0.smithi195.stdout:removed './151/18' 2022-06-27T11:51:50.711 INFO:tasks.workunit.client.0.smithi195.stdout:removed './151/13' 2022-06-27T11:51:50.711 INFO:tasks.workunit.client.0.smithi195.stdout:removed './151/33' 2022-06-27T11:51:50.712 INFO:tasks.workunit.client.0.smithi195.stdout:removed './151/17' 2022-06-27T11:51:50.712 INFO:tasks.workunit.client.0.smithi195.stdout:removed './151/35' 2022-06-27T11:51:50.713 INFO:tasks.workunit.client.0.smithi195.stdout:removed './151/37' 2022-06-27T11:51:50.713 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './151' 2022-06-27T11:51:50.713 INFO:tasks.workunit.client.0.smithi195.stdout:removed './152/19' 2022-06-27T11:51:50.714 INFO:tasks.workunit.client.0.smithi195.stdout:removed './152/15' 2022-06-27T11:51:50.715 INFO:tasks.workunit.client.0.smithi195.stdout:removed './152/31' 2022-06-27T11:51:50.715 INFO:tasks.workunit.client.0.smithi195.stdout:removed './152/36' 2022-06-27T11:51:50.716 INFO:tasks.workunit.client.0.smithi195.stdout:removed './152/20' 2022-06-27T11:51:50.716 INFO:tasks.workunit.client.0.smithi195.stdout:removed './152/12' 2022-06-27T11:51:50.718 INFO:tasks.workunit.client.0.smithi195.stdout:removed './152/39' 2022-06-27T11:51:50.718 INFO:tasks.workunit.client.0.smithi195.stdout:removed './152/10' 2022-06-27T11:51:50.719 INFO:tasks.workunit.client.0.smithi195.stdout:removed './152/34' 2022-06-27T11:51:50.719 INFO:tasks.workunit.client.0.smithi195.stdout:removed './152/38' 2022-06-27T11:51:50.720 INFO:tasks.workunit.client.0.smithi195.stdout:removed './152/11' 2022-06-27T11:51:50.720 INFO:tasks.workunit.client.0.smithi195.stdout:removed './152/16' 2022-06-27T11:51:50.721 INFO:tasks.workunit.client.0.smithi195.stdout:removed './152/32' 2022-06-27T11:51:50.721 INFO:tasks.workunit.client.0.smithi195.stdout:removed './152/30' 2022-06-27T11:51:50.721 INFO:tasks.workunit.client.0.smithi195.stdout:removed './152/14' 2022-06-27T11:51:50.722 INFO:tasks.workunit.client.0.smithi195.stdout:removed './152/18' 2022-06-27T11:51:50.722 INFO:tasks.workunit.client.0.smithi195.stdout:removed './152/13' 2022-06-27T11:51:50.723 INFO:tasks.workunit.client.0.smithi195.stdout:removed './152/33' 2022-06-27T11:51:50.723 INFO:tasks.workunit.client.0.smithi195.stdout:removed './152/17' 2022-06-27T11:51:50.724 INFO:tasks.workunit.client.0.smithi195.stdout:removed './152/35' 2022-06-27T11:51:50.724 INFO:tasks.workunit.client.0.smithi195.stdout:removed './152/37' 2022-06-27T11:51:50.725 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './152' 2022-06-27T11:51:50.725 INFO:tasks.workunit.client.0.smithi195.stdout:removed './153/19' 2022-06-27T11:51:50.726 INFO:tasks.workunit.client.0.smithi195.stdout:removed './153/15' 2022-06-27T11:51:50.726 INFO:tasks.workunit.client.0.smithi195.stdout:removed './153/31' 2022-06-27T11:51:50.727 INFO:tasks.workunit.client.0.smithi195.stdout:removed './153/36' 2022-06-27T11:51:50.727 INFO:tasks.workunit.client.0.smithi195.stdout:removed './153/20' 2022-06-27T11:51:50.727 INFO:tasks.workunit.client.0.smithi195.stdout:removed './153/12' 2022-06-27T11:51:50.728 INFO:tasks.workunit.client.0.smithi195.stdout:removed './153/39' 2022-06-27T11:51:50.728 INFO:tasks.workunit.client.0.smithi195.stdout:removed './153/10' 2022-06-27T11:51:50.728 INFO:tasks.workunit.client.0.smithi195.stdout:removed './153/34' 2022-06-27T11:51:50.728 INFO:tasks.workunit.client.0.smithi195.stdout:removed './153/38' 2022-06-27T11:51:50.729 INFO:tasks.workunit.client.0.smithi195.stdout:removed './153/11' 2022-06-27T11:51:50.729 INFO:tasks.workunit.client.0.smithi195.stdout:removed './153/16' 2022-06-27T11:51:50.730 INFO:tasks.workunit.client.0.smithi195.stdout:removed './153/32' 2022-06-27T11:51:50.730 INFO:tasks.workunit.client.0.smithi195.stdout:removed './153/30' 2022-06-27T11:51:50.731 INFO:tasks.workunit.client.0.smithi195.stdout:removed './153/14' 2022-06-27T11:51:50.731 INFO:tasks.workunit.client.0.smithi195.stdout:removed './153/18' 2022-06-27T11:51:50.731 INFO:tasks.workunit.client.0.smithi195.stdout:removed './153/13' 2022-06-27T11:51:50.732 INFO:tasks.workunit.client.0.smithi195.stdout:removed './153/33' 2022-06-27T11:51:50.732 INFO:tasks.workunit.client.0.smithi195.stdout:removed './153/17' 2022-06-27T11:51:50.733 INFO:tasks.workunit.client.0.smithi195.stdout:removed './153/35' 2022-06-27T11:51:50.733 INFO:tasks.workunit.client.0.smithi195.stdout:removed './153/37' 2022-06-27T11:51:50.734 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './153' 2022-06-27T11:51:50.734 INFO:tasks.workunit.client.0.smithi195.stdout:removed './154/19' 2022-06-27T11:51:50.734 INFO:tasks.workunit.client.0.smithi195.stdout:removed './154/15' 2022-06-27T11:51:50.735 INFO:tasks.workunit.client.0.smithi195.stdout:removed './154/31' 2022-06-27T11:51:50.735 INFO:tasks.workunit.client.0.smithi195.stdout:removed './154/36' 2022-06-27T11:51:50.735 INFO:tasks.workunit.client.0.smithi195.stdout:removed './154/20' 2022-06-27T11:51:50.735 INFO:tasks.workunit.client.0.smithi195.stdout:removed './154/12' 2022-06-27T11:51:50.735 INFO:tasks.workunit.client.0.smithi195.stdout:removed './154/39' 2022-06-27T11:51:50.736 INFO:tasks.workunit.client.0.smithi195.stdout:removed './154/10' 2022-06-27T11:51:50.736 INFO:tasks.workunit.client.0.smithi195.stdout:removed './154/34' 2022-06-27T11:51:50.736 INFO:tasks.workunit.client.0.smithi195.stdout:removed './154/38' 2022-06-27T11:51:50.736 INFO:tasks.workunit.client.0.smithi195.stdout:removed './154/11' 2022-06-27T11:51:50.737 INFO:tasks.workunit.client.0.smithi195.stdout:removed './154/16' 2022-06-27T11:51:50.737 INFO:tasks.workunit.client.0.smithi195.stdout:removed './154/32' 2022-06-27T11:51:50.737 INFO:tasks.workunit.client.0.smithi195.stdout:removed './154/30' 2022-06-27T11:51:50.737 INFO:tasks.workunit.client.0.smithi195.stdout:removed './154/14' 2022-06-27T11:51:50.738 INFO:tasks.workunit.client.0.smithi195.stdout:removed './154/18' 2022-06-27T11:51:50.738 INFO:tasks.workunit.client.0.smithi195.stdout:removed './154/13' 2022-06-27T11:51:50.739 INFO:tasks.workunit.client.0.smithi195.stdout:removed './154/33' 2022-06-27T11:51:50.740 INFO:tasks.workunit.client.0.smithi195.stdout:removed './154/17' 2022-06-27T11:51:50.741 INFO:tasks.workunit.client.0.smithi195.stdout:removed './154/35' 2022-06-27T11:51:50.741 INFO:tasks.workunit.client.0.smithi195.stdout:removed './154/37' 2022-06-27T11:51:50.742 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './154' 2022-06-27T11:51:50.742 INFO:tasks.workunit.client.0.smithi195.stdout:removed './155/19' 2022-06-27T11:51:50.742 INFO:tasks.workunit.client.0.smithi195.stdout:removed './155/15' 2022-06-27T11:51:50.743 INFO:tasks.workunit.client.0.smithi195.stdout:removed './155/31' 2022-06-27T11:51:50.743 INFO:tasks.workunit.client.0.smithi195.stdout:removed './155/36' 2022-06-27T11:51:50.743 INFO:tasks.workunit.client.0.smithi195.stdout:removed './155/20' 2022-06-27T11:51:50.744 INFO:tasks.workunit.client.0.smithi195.stdout:removed './155/12' 2022-06-27T11:51:50.744 INFO:tasks.workunit.client.0.smithi195.stdout:removed './155/39' 2022-06-27T11:51:50.744 INFO:tasks.workunit.client.0.smithi195.stdout:removed './155/10' 2022-06-27T11:51:50.745 INFO:tasks.workunit.client.0.smithi195.stdout:removed './155/34' 2022-06-27T11:51:50.756 INFO:tasks.workunit.client.0.smithi195.stdout:removed './155/38' 2022-06-27T11:51:50.756 INFO:tasks.workunit.client.0.smithi195.stdout:removed './155/11' 2022-06-27T11:51:50.756 INFO:tasks.workunit.client.0.smithi195.stdout:removed './155/16' 2022-06-27T11:51:50.757 INFO:tasks.workunit.client.0.smithi195.stdout:removed './155/32' 2022-06-27T11:51:50.757 INFO:tasks.workunit.client.0.smithi195.stdout:removed './155/30' 2022-06-27T11:51:50.757 INFO:tasks.workunit.client.0.smithi195.stdout:removed './155/14' 2022-06-27T11:51:50.762 INFO:tasks.workunit.client.0.smithi195.stdout:removed './155/18' 2022-06-27T11:51:50.763 INFO:tasks.workunit.client.0.smithi195.stdout:removed './155/13' 2022-06-27T11:51:50.763 INFO:tasks.workunit.client.0.smithi195.stdout:removed './155/33' 2022-06-27T11:51:50.763 INFO:tasks.workunit.client.0.smithi195.stdout:removed './155/17' 2022-06-27T11:51:50.764 INFO:tasks.workunit.client.0.smithi195.stdout:removed './155/35' 2022-06-27T11:51:50.764 INFO:tasks.workunit.client.0.smithi195.stdout:removed './155/37' 2022-06-27T11:51:50.764 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './155' 2022-06-27T11:51:50.765 INFO:tasks.workunit.client.0.smithi195.stdout:removed './156/19' 2022-06-27T11:51:50.765 INFO:tasks.workunit.client.0.smithi195.stdout:removed './156/15' 2022-06-27T11:51:50.765 INFO:tasks.workunit.client.0.smithi195.stdout:removed './156/31' 2022-06-27T11:51:50.766 INFO:tasks.workunit.client.0.smithi195.stdout:removed './156/36' 2022-06-27T11:51:50.766 INFO:tasks.workunit.client.0.smithi195.stdout:removed './156/20' 2022-06-27T11:51:50.767 INFO:tasks.workunit.client.0.smithi195.stdout:removed './156/12' 2022-06-27T11:51:50.767 INFO:tasks.workunit.client.0.smithi195.stdout:removed './156/39' 2022-06-27T11:51:50.768 INFO:tasks.workunit.client.0.smithi195.stdout:removed './156/10' 2022-06-27T11:51:50.768 INFO:tasks.workunit.client.0.smithi195.stdout:removed './156/34' 2022-06-27T11:51:50.769 INFO:tasks.workunit.client.0.smithi195.stdout:removed './156/38' 2022-06-27T11:51:50.770 INFO:tasks.workunit.client.0.smithi195.stdout:removed './156/11' 2022-06-27T11:51:50.770 INFO:tasks.workunit.client.0.smithi195.stdout:removed './156/16' 2022-06-27T11:51:50.770 INFO:tasks.workunit.client.0.smithi195.stdout:removed './156/32' 2022-06-27T11:51:50.773 INFO:tasks.workunit.client.0.smithi195.stdout:removed './156/30' 2022-06-27T11:51:50.773 INFO:tasks.workunit.client.0.smithi195.stdout:removed './156/14' 2022-06-27T11:51:50.774 INFO:tasks.workunit.client.0.smithi195.stdout:removed './156/18' 2022-06-27T11:51:50.774 INFO:tasks.workunit.client.0.smithi195.stdout:removed './156/13' 2022-06-27T11:51:50.774 INFO:tasks.workunit.client.0.smithi195.stdout:removed './156/33' 2022-06-27T11:51:50.775 INFO:tasks.workunit.client.0.smithi195.stdout:removed './156/17' 2022-06-27T11:51:50.775 INFO:tasks.workunit.client.0.smithi195.stdout:removed './156/35' 2022-06-27T11:51:50.776 INFO:tasks.workunit.client.0.smithi195.stdout:removed './156/37' 2022-06-27T11:51:50.776 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './156' 2022-06-27T11:51:50.776 INFO:tasks.workunit.client.0.smithi195.stdout:removed './157/19' 2022-06-27T11:51:50.777 INFO:tasks.workunit.client.0.smithi195.stdout:removed './157/15' 2022-06-27T11:51:50.777 INFO:tasks.workunit.client.0.smithi195.stdout:removed './157/31' 2022-06-27T11:51:50.777 INFO:tasks.workunit.client.0.smithi195.stdout:removed './157/36' 2022-06-27T11:51:50.778 INFO:tasks.workunit.client.0.smithi195.stdout:removed './157/20' 2022-06-27T11:51:50.778 INFO:tasks.workunit.client.0.smithi195.stdout:removed './157/12' 2022-06-27T11:51:50.778 INFO:tasks.workunit.client.0.smithi195.stdout:removed './157/39' 2022-06-27T11:51:50.779 INFO:tasks.workunit.client.0.smithi195.stdout:removed './157/10' 2022-06-27T11:51:50.779 INFO:tasks.workunit.client.0.smithi195.stdout:removed './157/34' 2022-06-27T11:51:50.779 INFO:tasks.workunit.client.0.smithi195.stdout:removed './157/38' 2022-06-27T11:51:50.780 INFO:tasks.workunit.client.0.smithi195.stdout:removed './157/11' 2022-06-27T11:51:50.780 INFO:tasks.workunit.client.0.smithi195.stdout:removed './157/16' 2022-06-27T11:51:50.780 INFO:tasks.workunit.client.0.smithi195.stdout:removed './157/32' 2022-06-27T11:51:50.781 INFO:tasks.workunit.client.0.smithi195.stdout:removed './157/30' 2022-06-27T11:51:50.781 INFO:tasks.workunit.client.0.smithi195.stdout:removed './157/14' 2022-06-27T11:51:50.782 INFO:tasks.workunit.client.0.smithi195.stdout:removed './157/18' 2022-06-27T11:51:50.785 INFO:tasks.workunit.client.0.smithi195.stdout:removed './157/13' 2022-06-27T11:51:50.786 INFO:tasks.workunit.client.0.smithi195.stdout:removed './157/33' 2022-06-27T11:51:52.147 INFO:tasks.workunit.client.0.smithi195.stdout:removed './157/17' 2022-06-27T11:51:52.148 INFO:tasks.workunit.client.0.smithi195.stdout:removed './157/35' 2022-06-27T11:51:52.148 INFO:tasks.workunit.client.0.smithi195.stdout:removed './157/37' 2022-06-27T11:51:52.149 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './157' 2022-06-27T11:51:52.150 INFO:tasks.workunit.client.0.smithi195.stdout:removed './158/19' 2022-06-27T11:51:52.150 INFO:tasks.workunit.client.0.smithi195.stdout:removed './158/15' 2022-06-27T11:51:52.150 INFO:tasks.workunit.client.0.smithi195.stdout:removed './158/31' 2022-06-27T11:51:52.151 INFO:tasks.workunit.client.0.smithi195.stdout:removed './158/36' 2022-06-27T11:51:52.151 INFO:tasks.workunit.client.0.smithi195.stdout:removed './158/20' 2022-06-27T11:51:52.151 INFO:tasks.workunit.client.0.smithi195.stdout:removed './158/12' 2022-06-27T11:51:52.151 INFO:tasks.workunit.client.0.smithi195.stdout:removed './158/39' 2022-06-27T11:51:52.152 INFO:tasks.workunit.client.0.smithi195.stdout:removed './158/10' 2022-06-27T11:51:52.152 INFO:tasks.workunit.client.0.smithi195.stdout:removed './158/34' 2022-06-27T11:51:52.152 INFO:tasks.workunit.client.0.smithi195.stdout:removed './158/38' 2022-06-27T11:51:52.152 INFO:tasks.workunit.client.0.smithi195.stdout:removed './158/11' 2022-06-27T11:51:52.152 INFO:tasks.workunit.client.0.smithi195.stdout:removed './158/16' 2022-06-27T11:51:52.153 INFO:tasks.workunit.client.0.smithi195.stdout:removed './158/32' 2022-06-27T11:51:52.153 INFO:tasks.workunit.client.0.smithi195.stdout:removed './158/30' 2022-06-27T11:51:52.153 INFO:tasks.workunit.client.0.smithi195.stdout:removed './158/14' 2022-06-27T11:51:52.153 INFO:tasks.workunit.client.0.smithi195.stdout:removed './158/18' 2022-06-27T11:51:52.154 INFO:tasks.workunit.client.0.smithi195.stdout:removed './158/13' 2022-06-27T11:51:52.154 INFO:tasks.workunit.client.0.smithi195.stdout:removed './158/33' 2022-06-27T11:51:52.154 INFO:tasks.workunit.client.0.smithi195.stdout:removed './158/17' 2022-06-27T11:51:52.155 INFO:tasks.workunit.client.0.smithi195.stdout:removed './158/35' 2022-06-27T11:51:52.155 INFO:tasks.workunit.client.0.smithi195.stdout:removed './158/37' 2022-06-27T11:51:52.155 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './158' 2022-06-27T11:51:52.155 INFO:tasks.workunit.client.0.smithi195.stdout:removed './159/19' 2022-06-27T11:51:52.156 INFO:tasks.workunit.client.0.smithi195.stdout:removed './159/15' 2022-06-27T11:51:52.156 INFO:tasks.workunit.client.0.smithi195.stdout:removed './159/31' 2022-06-27T11:51:52.156 INFO:tasks.workunit.client.0.smithi195.stdout:removed './159/36' 2022-06-27T11:51:52.156 INFO:tasks.workunit.client.0.smithi195.stdout:removed './159/20' 2022-06-27T11:51:52.156 INFO:tasks.workunit.client.0.smithi195.stdout:removed './159/12' 2022-06-27T11:51:52.157 INFO:tasks.workunit.client.0.smithi195.stdout:removed './159/39' 2022-06-27T11:51:52.157 INFO:tasks.workunit.client.0.smithi195.stdout:removed './159/10' 2022-06-27T11:51:52.157 INFO:tasks.workunit.client.0.smithi195.stdout:removed './159/34' 2022-06-27T11:51:52.157 INFO:tasks.workunit.client.0.smithi195.stdout:removed './159/38' 2022-06-27T11:51:52.158 INFO:tasks.workunit.client.0.smithi195.stdout:removed './159/11' 2022-06-27T11:51:52.158 INFO:tasks.workunit.client.0.smithi195.stdout:removed './159/16' 2022-06-27T11:51:52.158 INFO:tasks.workunit.client.0.smithi195.stdout:removed './159/32' 2022-06-27T11:51:52.158 INFO:tasks.workunit.client.0.smithi195.stdout:removed './159/30' 2022-06-27T11:51:52.159 INFO:tasks.workunit.client.0.smithi195.stdout:removed './159/14' 2022-06-27T11:51:52.159 INFO:tasks.workunit.client.0.smithi195.stdout:removed './159/18' 2022-06-27T11:51:52.159 INFO:tasks.workunit.client.0.smithi195.stdout:removed './159/13' 2022-06-27T11:51:52.159 INFO:tasks.workunit.client.0.smithi195.stdout:removed './159/33' 2022-06-27T11:51:52.160 INFO:tasks.workunit.client.0.smithi195.stdout:removed './159/17' 2022-06-27T11:51:52.160 INFO:tasks.workunit.client.0.smithi195.stdout:removed './159/35' 2022-06-27T11:51:52.160 INFO:tasks.workunit.client.0.smithi195.stdout:removed './159/37' 2022-06-27T11:51:52.160 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './159' 2022-06-27T11:51:52.161 INFO:tasks.workunit.client.0.smithi195.stdout:removed './160/19' 2022-06-27T11:51:52.161 INFO:tasks.workunit.client.0.smithi195.stdout:removed './160/15' 2022-06-27T11:51:52.161 INFO:tasks.workunit.client.0.smithi195.stdout:removed './160/31' 2022-06-27T11:51:52.162 INFO:tasks.workunit.client.0.smithi195.stdout:removed './160/36' 2022-06-27T11:51:52.162 INFO:tasks.workunit.client.0.smithi195.stdout:removed './160/20' 2022-06-27T11:51:52.162 INFO:tasks.workunit.client.0.smithi195.stdout:removed './160/12' 2022-06-27T11:51:52.163 INFO:tasks.workunit.client.0.smithi195.stdout:removed './160/39' 2022-06-27T11:51:52.163 INFO:tasks.workunit.client.0.smithi195.stdout:removed './160/10' 2022-06-27T11:51:52.163 INFO:tasks.workunit.client.0.smithi195.stdout:removed './160/34' 2022-06-27T11:51:52.163 INFO:tasks.workunit.client.0.smithi195.stdout:removed './160/38' 2022-06-27T11:51:52.163 INFO:tasks.workunit.client.0.smithi195.stdout:removed './160/11' 2022-06-27T11:51:52.164 INFO:tasks.workunit.client.0.smithi195.stdout:removed './160/16' 2022-06-27T11:51:52.164 INFO:tasks.workunit.client.0.smithi195.stdout:removed './160/32' 2022-06-27T11:51:52.164 INFO:tasks.workunit.client.0.smithi195.stdout:removed './160/30' 2022-06-27T11:51:52.164 INFO:tasks.workunit.client.0.smithi195.stdout:removed './160/14' 2022-06-27T11:51:52.164 INFO:tasks.workunit.client.0.smithi195.stdout:removed './160/18' 2022-06-27T11:51:52.165 INFO:tasks.workunit.client.0.smithi195.stdout:removed './160/13' 2022-06-27T11:51:52.165 INFO:tasks.workunit.client.0.smithi195.stdout:removed './160/33' 2022-06-27T11:51:52.165 INFO:tasks.workunit.client.0.smithi195.stdout:removed './160/17' 2022-06-27T11:51:52.165 INFO:tasks.workunit.client.0.smithi195.stdout:removed './160/35' 2022-06-27T11:51:52.166 INFO:tasks.workunit.client.0.smithi195.stdout:removed './160/37' 2022-06-27T11:51:52.166 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './160' 2022-06-27T11:51:52.166 INFO:tasks.workunit.client.0.smithi195.stdout:removed './161/19' 2022-06-27T11:51:52.166 INFO:tasks.workunit.client.0.smithi195.stdout:removed './161/15' 2022-06-27T11:51:52.167 INFO:tasks.workunit.client.0.smithi195.stdout:removed './161/31' 2022-06-27T11:51:52.167 INFO:tasks.workunit.client.0.smithi195.stdout:removed './161/36' 2022-06-27T11:51:52.167 INFO:tasks.workunit.client.0.smithi195.stdout:removed './161/20' 2022-06-27T11:51:52.167 INFO:tasks.workunit.client.0.smithi195.stdout:removed './161/12' 2022-06-27T11:51:52.167 INFO:tasks.workunit.client.0.smithi195.stdout:removed './161/39' 2022-06-27T11:51:52.168 INFO:tasks.workunit.client.0.smithi195.stdout:removed './161/10' 2022-06-27T11:51:52.168 INFO:tasks.workunit.client.0.smithi195.stdout:removed './161/34' 2022-06-27T11:51:52.168 INFO:tasks.workunit.client.0.smithi195.stdout:removed './161/38' 2022-06-27T11:51:52.168 INFO:tasks.workunit.client.0.smithi195.stdout:removed './161/11' 2022-06-27T11:51:52.169 INFO:tasks.workunit.client.0.smithi195.stdout:removed './161/16' 2022-06-27T11:51:52.169 INFO:tasks.workunit.client.0.smithi195.stdout:removed './161/32' 2022-06-27T11:51:52.170 INFO:tasks.workunit.client.0.smithi195.stdout:removed './161/30' 2022-06-27T11:51:52.170 INFO:tasks.workunit.client.0.smithi195.stdout:removed './161/14' 2022-06-27T11:51:52.170 INFO:tasks.workunit.client.0.smithi195.stdout:removed './161/18' 2022-06-27T11:51:52.170 INFO:tasks.workunit.client.0.smithi195.stdout:removed './161/13' 2022-06-27T11:51:52.171 INFO:tasks.workunit.client.0.smithi195.stdout:removed './161/33' 2022-06-27T11:51:52.171 INFO:tasks.workunit.client.0.smithi195.stdout:removed './161/17' 2022-06-27T11:51:52.171 INFO:tasks.workunit.client.0.smithi195.stdout:removed './161/35' 2022-06-27T11:51:52.172 INFO:tasks.workunit.client.0.smithi195.stdout:removed './161/37' 2022-06-27T11:51:52.172 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './161' 2022-06-27T11:51:52.172 INFO:tasks.workunit.client.0.smithi195.stdout:removed './162/19' 2022-06-27T11:51:52.172 INFO:tasks.workunit.client.0.smithi195.stdout:removed './162/15' 2022-06-27T11:51:52.173 INFO:tasks.workunit.client.0.smithi195.stdout:removed './162/31' 2022-06-27T11:51:52.173 INFO:tasks.workunit.client.0.smithi195.stdout:removed './162/36' 2022-06-27T11:51:52.173 INFO:tasks.workunit.client.0.smithi195.stdout:removed './162/20' 2022-06-27T11:51:52.173 INFO:tasks.workunit.client.0.smithi195.stdout:removed './162/12' 2022-06-27T11:51:52.174 INFO:tasks.workunit.client.0.smithi195.stdout:removed './162/39' 2022-06-27T11:51:52.174 INFO:tasks.workunit.client.0.smithi195.stdout:removed './162/10' 2022-06-27T11:51:52.175 INFO:tasks.workunit.client.0.smithi195.stdout:removed './162/34' 2022-06-27T11:51:52.176 INFO:tasks.workunit.client.0.smithi195.stdout:removed './162/38' 2022-06-27T11:51:52.176 INFO:tasks.workunit.client.0.smithi195.stdout:removed './162/11' 2022-06-27T11:51:52.176 INFO:tasks.workunit.client.0.smithi195.stdout:removed './162/16' 2022-06-27T11:51:52.177 INFO:tasks.workunit.client.0.smithi195.stdout:removed './162/32' 2022-06-27T11:51:52.177 INFO:tasks.workunit.client.0.smithi195.stdout:removed './162/30' 2022-06-27T11:51:52.177 INFO:tasks.workunit.client.0.smithi195.stdout:removed './162/14' 2022-06-27T11:51:52.178 INFO:tasks.workunit.client.0.smithi195.stdout:removed './162/18' 2022-06-27T11:51:52.178 INFO:tasks.workunit.client.0.smithi195.stdout:removed './162/13' 2022-06-27T11:51:52.178 INFO:tasks.workunit.client.0.smithi195.stdout:removed './162/33' 2022-06-27T11:51:52.179 INFO:tasks.workunit.client.0.smithi195.stdout:removed './162/17' 2022-06-27T11:51:52.179 INFO:tasks.workunit.client.0.smithi195.stdout:removed './162/35' 2022-06-27T11:51:52.180 INFO:tasks.workunit.client.0.smithi195.stdout:removed './162/37' 2022-06-27T11:51:52.180 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './162' 2022-06-27T11:51:52.181 INFO:tasks.workunit.client.0.smithi195.stdout:removed './163/19' 2022-06-27T11:51:52.181 INFO:tasks.workunit.client.0.smithi195.stdout:removed './163/15' 2022-06-27T11:51:52.182 INFO:tasks.workunit.client.0.smithi195.stdout:removed './163/31' 2022-06-27T11:51:52.182 INFO:tasks.workunit.client.0.smithi195.stdout:removed './163/36' 2022-06-27T11:51:52.183 INFO:tasks.workunit.client.0.smithi195.stdout:removed './163/20' 2022-06-27T11:51:52.183 INFO:tasks.workunit.client.0.smithi195.stdout:removed './163/12' 2022-06-27T11:51:52.184 INFO:tasks.workunit.client.0.smithi195.stdout:removed './163/39' 2022-06-27T11:51:52.184 INFO:tasks.workunit.client.0.smithi195.stdout:removed './163/10' 2022-06-27T11:51:52.184 INFO:tasks.workunit.client.0.smithi195.stdout:removed './163/34' 2022-06-27T11:51:52.185 INFO:tasks.workunit.client.0.smithi195.stdout:removed './163/38' 2022-06-27T11:51:52.185 INFO:tasks.workunit.client.0.smithi195.stdout:removed './163/11' 2022-06-27T11:51:52.186 INFO:tasks.workunit.client.0.smithi195.stdout:removed './163/16' 2022-06-27T11:51:52.186 INFO:tasks.workunit.client.0.smithi195.stdout:removed './163/32' 2022-06-27T11:51:52.187 INFO:tasks.workunit.client.0.smithi195.stdout:removed './163/30' 2022-06-27T11:51:52.187 INFO:tasks.workunit.client.0.smithi195.stdout:removed './163/14' 2022-06-27T11:51:52.188 INFO:tasks.workunit.client.0.smithi195.stdout:removed './163/18' 2022-06-27T11:51:52.188 INFO:tasks.workunit.client.0.smithi195.stdout:removed './163/13' 2022-06-27T11:51:52.189 INFO:tasks.workunit.client.0.smithi195.stdout:removed './163/33' 2022-06-27T11:51:52.190 INFO:tasks.workunit.client.0.smithi195.stdout:removed './163/17' 2022-06-27T11:51:52.191 INFO:tasks.workunit.client.0.smithi195.stdout:removed './163/35' 2022-06-27T11:51:52.191 INFO:tasks.workunit.client.0.smithi195.stdout:removed './163/37' 2022-06-27T11:51:52.193 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './163' 2022-06-27T11:51:52.193 INFO:tasks.workunit.client.0.smithi195.stdout:removed './164/19' 2022-06-27T11:51:52.193 INFO:tasks.workunit.client.0.smithi195.stdout:removed './164/15' 2022-06-27T11:51:52.193 INFO:tasks.workunit.client.0.smithi195.stdout:removed './164/31' 2022-06-27T11:51:52.194 INFO:tasks.workunit.client.0.smithi195.stdout:removed './164/36' 2022-06-27T11:51:52.194 INFO:tasks.workunit.client.0.smithi195.stdout:removed './164/20' 2022-06-27T11:51:52.194 INFO:tasks.workunit.client.0.smithi195.stdout:removed './164/12' 2022-06-27T11:51:52.195 INFO:tasks.workunit.client.0.smithi195.stdout:removed './164/39' 2022-06-27T11:51:52.195 INFO:tasks.workunit.client.0.smithi195.stdout:removed './164/10' 2022-06-27T11:51:52.195 INFO:tasks.workunit.client.0.smithi195.stdout:removed './164/34' 2022-06-27T11:51:52.195 INFO:tasks.workunit.client.0.smithi195.stdout:removed './164/38' 2022-06-27T11:51:52.196 INFO:tasks.workunit.client.0.smithi195.stdout:removed './164/11' 2022-06-27T11:51:52.196 INFO:tasks.workunit.client.0.smithi195.stdout:removed './164/16' 2022-06-27T11:51:52.196 INFO:tasks.workunit.client.0.smithi195.stdout:removed './164/32' 2022-06-27T11:51:52.196 INFO:tasks.workunit.client.0.smithi195.stdout:removed './164/30' 2022-06-27T11:51:52.196 INFO:tasks.workunit.client.0.smithi195.stdout:removed './164/14' 2022-06-27T11:51:52.197 INFO:tasks.workunit.client.0.smithi195.stdout:removed './164/18' 2022-06-27T11:51:52.197 INFO:tasks.workunit.client.0.smithi195.stdout:removed './164/13' 2022-06-27T11:51:52.197 INFO:tasks.workunit.client.0.smithi195.stdout:removed './164/33' 2022-06-27T11:51:52.198 INFO:tasks.workunit.client.0.smithi195.stdout:removed './164/17' 2022-06-27T11:51:52.198 INFO:tasks.workunit.client.0.smithi195.stdout:removed './164/35' 2022-06-27T11:51:52.198 INFO:tasks.workunit.client.0.smithi195.stdout:removed './164/37' 2022-06-27T11:51:52.199 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './164' 2022-06-27T11:51:52.199 INFO:tasks.workunit.client.0.smithi195.stdout:removed './165/19' 2022-06-27T11:51:52.199 INFO:tasks.workunit.client.0.smithi195.stdout:removed './165/15' 2022-06-27T11:51:52.199 INFO:tasks.workunit.client.0.smithi195.stdout:removed './165/31' 2022-06-27T11:51:52.199 INFO:tasks.workunit.client.0.smithi195.stdout:removed './165/36' 2022-06-27T11:51:52.200 INFO:tasks.workunit.client.0.smithi195.stdout:removed './165/20' 2022-06-27T11:51:52.200 INFO:tasks.workunit.client.0.smithi195.stdout:removed './165/12' 2022-06-27T11:51:52.200 INFO:tasks.workunit.client.0.smithi195.stdout:removed './165/39' 2022-06-27T11:51:52.201 INFO:tasks.workunit.client.0.smithi195.stdout:removed './165/10' 2022-06-27T11:51:52.201 INFO:tasks.workunit.client.0.smithi195.stdout:removed './165/34' 2022-06-27T11:51:52.201 INFO:tasks.workunit.client.0.smithi195.stdout:removed './165/38' 2022-06-27T11:51:52.201 INFO:tasks.workunit.client.0.smithi195.stdout:removed './165/11' 2022-06-27T11:51:52.201 INFO:tasks.workunit.client.0.smithi195.stdout:removed './165/16' 2022-06-27T11:51:52.202 INFO:tasks.workunit.client.0.smithi195.stdout:removed './165/32' 2022-06-27T11:51:52.202 INFO:tasks.workunit.client.0.smithi195.stdout:removed './165/30' 2022-06-27T11:51:52.202 INFO:tasks.workunit.client.0.smithi195.stdout:removed './165/14' 2022-06-27T11:51:52.203 INFO:tasks.workunit.client.0.smithi195.stdout:removed './165/18' 2022-06-27T11:51:52.203 INFO:tasks.workunit.client.0.smithi195.stdout:removed './165/13' 2022-06-27T11:51:52.203 INFO:tasks.workunit.client.0.smithi195.stdout:removed './165/33' 2022-06-27T11:51:52.203 INFO:tasks.workunit.client.0.smithi195.stdout:removed './165/17' 2022-06-27T11:51:52.203 INFO:tasks.workunit.client.0.smithi195.stdout:removed './165/35' 2022-06-27T11:51:52.204 INFO:tasks.workunit.client.0.smithi195.stdout:removed './165/37' 2022-06-27T11:51:52.204 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './165' 2022-06-27T11:51:52.204 INFO:tasks.workunit.client.0.smithi195.stdout:removed './166/19' 2022-06-27T11:51:52.204 INFO:tasks.workunit.client.0.smithi195.stdout:removed './166/15' 2022-06-27T11:51:52.204 INFO:tasks.workunit.client.0.smithi195.stdout:removed './166/31' 2022-06-27T11:51:52.205 INFO:tasks.workunit.client.0.smithi195.stdout:removed './166/36' 2022-06-27T11:51:52.205 INFO:tasks.workunit.client.0.smithi195.stdout:removed './166/20' 2022-06-27T11:51:52.205 INFO:tasks.workunit.client.0.smithi195.stdout:removed './166/12' 2022-06-27T11:51:52.206 INFO:tasks.workunit.client.0.smithi195.stdout:removed './166/39' 2022-06-27T11:51:52.206 INFO:tasks.workunit.client.0.smithi195.stdout:removed './166/10' 2022-06-27T11:51:52.206 INFO:tasks.workunit.client.0.smithi195.stdout:removed './166/34' 2022-06-27T11:51:52.207 INFO:tasks.workunit.client.0.smithi195.stdout:removed './166/38' 2022-06-27T11:51:52.207 INFO:tasks.workunit.client.0.smithi195.stdout:removed './166/11' 2022-06-27T11:51:52.207 INFO:tasks.workunit.client.0.smithi195.stdout:removed './166/16' 2022-06-27T11:51:52.207 INFO:tasks.workunit.client.0.smithi195.stdout:removed './166/32' 2022-06-27T11:51:52.208 INFO:tasks.workunit.client.0.smithi195.stdout:removed './166/30' 2022-06-27T11:51:52.208 INFO:tasks.workunit.client.0.smithi195.stdout:removed './166/14' 2022-06-27T11:51:52.208 INFO:tasks.workunit.client.0.smithi195.stdout:removed './166/18' 2022-06-27T11:51:52.208 INFO:tasks.workunit.client.0.smithi195.stdout:removed './166/13' 2022-06-27T11:51:52.209 INFO:tasks.workunit.client.0.smithi195.stdout:removed './166/33' 2022-06-27T11:51:52.209 INFO:tasks.workunit.client.0.smithi195.stdout:removed './166/17' 2022-06-27T11:51:52.209 INFO:tasks.workunit.client.0.smithi195.stdout:removed './166/35' 2022-06-27T11:51:52.209 INFO:tasks.workunit.client.0.smithi195.stdout:removed './166/37' 2022-06-27T11:51:52.210 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './166' 2022-06-27T11:51:52.210 INFO:tasks.workunit.client.0.smithi195.stdout:removed './167/19' 2022-06-27T11:51:52.210 INFO:tasks.workunit.client.0.smithi195.stdout:removed './167/15' 2022-06-27T11:51:52.210 INFO:tasks.workunit.client.0.smithi195.stdout:removed './167/31' 2022-06-27T11:51:52.211 INFO:tasks.workunit.client.0.smithi195.stdout:removed './167/36' 2022-06-27T11:51:52.211 INFO:tasks.workunit.client.0.smithi195.stdout:removed './167/20' 2022-06-27T11:51:52.212 INFO:tasks.workunit.client.0.smithi195.stdout:removed './167/12' 2022-06-27T11:51:52.212 INFO:tasks.workunit.client.0.smithi195.stdout:removed './167/39' 2022-06-27T11:51:52.212 INFO:tasks.workunit.client.0.smithi195.stdout:removed './167/10' 2022-06-27T11:51:52.212 INFO:tasks.workunit.client.0.smithi195.stdout:removed './167/34' 2022-06-27T11:51:52.213 INFO:tasks.workunit.client.0.smithi195.stdout:removed './167/38' 2022-06-27T11:51:53.453 INFO:tasks.workunit.client.0.smithi195.stdout:removed './167/11' 2022-06-27T11:51:53.453 INFO:tasks.workunit.client.0.smithi195.stdout:removed './167/16' 2022-06-27T11:51:53.453 INFO:tasks.workunit.client.0.smithi195.stdout:removed './167/32' 2022-06-27T11:51:53.453 INFO:tasks.workunit.client.0.smithi195.stdout:removed './167/30' 2022-06-27T11:51:53.454 INFO:tasks.workunit.client.0.smithi195.stdout:removed './167/14' 2022-06-27T11:51:53.454 INFO:tasks.workunit.client.0.smithi195.stdout:removed './167/18' 2022-06-27T11:51:53.454 INFO:tasks.workunit.client.0.smithi195.stdout:removed './167/13' 2022-06-27T11:51:53.455 INFO:tasks.workunit.client.0.smithi195.stdout:removed './167/33' 2022-06-27T11:51:53.455 INFO:tasks.workunit.client.0.smithi195.stdout:removed './167/17' 2022-06-27T11:51:53.455 INFO:tasks.workunit.client.0.smithi195.stdout:removed './167/35' 2022-06-27T11:51:53.455 INFO:tasks.workunit.client.0.smithi195.stdout:removed './167/37' 2022-06-27T11:51:53.455 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './167' 2022-06-27T11:51:53.456 INFO:tasks.workunit.client.0.smithi195.stdout:removed './168/19' 2022-06-27T11:51:53.456 INFO:tasks.workunit.client.0.smithi195.stdout:removed './168/15' 2022-06-27T11:51:53.457 INFO:tasks.workunit.client.0.smithi195.stdout:removed './168/31' 2022-06-27T11:51:53.457 INFO:tasks.workunit.client.0.smithi195.stdout:removed './168/36' 2022-06-27T11:51:53.457 INFO:tasks.workunit.client.0.smithi195.stdout:removed './168/20' 2022-06-27T11:51:53.457 INFO:tasks.workunit.client.0.smithi195.stdout:removed './168/12' 2022-06-27T11:51:53.458 INFO:tasks.workunit.client.0.smithi195.stdout:removed './168/39' 2022-06-27T11:51:53.458 INFO:tasks.workunit.client.0.smithi195.stdout:removed './168/10' 2022-06-27T11:51:53.458 INFO:tasks.workunit.client.0.smithi195.stdout:removed './168/34' 2022-06-27T11:51:53.458 INFO:tasks.workunit.client.0.smithi195.stdout:removed './168/38' 2022-06-27T11:51:53.459 INFO:tasks.workunit.client.0.smithi195.stdout:removed './168/11' 2022-06-27T11:51:53.460 INFO:tasks.workunit.client.0.smithi195.stdout:removed './168/16' 2022-06-27T11:51:53.460 INFO:tasks.workunit.client.0.smithi195.stdout:removed './168/32' 2022-06-27T11:51:53.460 INFO:tasks.workunit.client.0.smithi195.stdout:removed './168/30' 2022-06-27T11:51:53.461 INFO:tasks.workunit.client.0.smithi195.stdout:removed './168/14' 2022-06-27T11:51:53.461 INFO:tasks.workunit.client.0.smithi195.stdout:removed './168/18' 2022-06-27T11:51:53.462 INFO:tasks.workunit.client.0.smithi195.stdout:removed './168/13' 2022-06-27T11:51:53.462 INFO:tasks.workunit.client.0.smithi195.stdout:removed './168/33' 2022-06-27T11:51:53.462 INFO:tasks.workunit.client.0.smithi195.stdout:removed './168/17' 2022-06-27T11:51:53.462 INFO:tasks.workunit.client.0.smithi195.stdout:removed './168/35' 2022-06-27T11:51:53.463 INFO:tasks.workunit.client.0.smithi195.stdout:removed './168/37' 2022-06-27T11:51:53.463 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './168' 2022-06-27T11:51:53.463 INFO:tasks.workunit.client.0.smithi195.stdout:removed './169/19' 2022-06-27T11:51:53.463 INFO:tasks.workunit.client.0.smithi195.stdout:removed './169/15' 2022-06-27T11:51:53.464 INFO:tasks.workunit.client.0.smithi195.stdout:removed './169/31' 2022-06-27T11:51:53.464 INFO:tasks.workunit.client.0.smithi195.stdout:removed './169/36' 2022-06-27T11:51:53.464 INFO:tasks.workunit.client.0.smithi195.stdout:removed './169/20' 2022-06-27T11:51:53.464 INFO:tasks.workunit.client.0.smithi195.stdout:removed './169/12' 2022-06-27T11:51:53.464 INFO:tasks.workunit.client.0.smithi195.stdout:removed './169/39' 2022-06-27T11:51:53.465 INFO:tasks.workunit.client.0.smithi195.stdout:removed './169/10' 2022-06-27T11:51:53.465 INFO:tasks.workunit.client.0.smithi195.stdout:removed './169/34' 2022-06-27T11:51:53.465 INFO:tasks.workunit.client.0.smithi195.stdout:removed './169/38' 2022-06-27T11:51:53.465 INFO:tasks.workunit.client.0.smithi195.stdout:removed './169/11' 2022-06-27T11:51:53.465 INFO:tasks.workunit.client.0.smithi195.stdout:removed './169/16' 2022-06-27T11:51:53.466 INFO:tasks.workunit.client.0.smithi195.stdout:removed './169/32' 2022-06-27T11:51:53.466 INFO:tasks.workunit.client.0.smithi195.stdout:removed './169/30' 2022-06-27T11:51:53.466 INFO:tasks.workunit.client.0.smithi195.stdout:removed './169/14' 2022-06-27T11:51:53.467 INFO:tasks.workunit.client.0.smithi195.stdout:removed './169/18' 2022-06-27T11:51:53.467 INFO:tasks.workunit.client.0.smithi195.stdout:removed './169/13' 2022-06-27T11:51:53.467 INFO:tasks.workunit.client.0.smithi195.stdout:removed './169/33' 2022-06-27T11:51:53.468 INFO:tasks.workunit.client.0.smithi195.stdout:removed './169/17' 2022-06-27T11:51:53.468 INFO:tasks.workunit.client.0.smithi195.stdout:removed './169/35' 2022-06-27T11:51:53.468 INFO:tasks.workunit.client.0.smithi195.stdout:removed './169/37' 2022-06-27T11:51:53.468 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './169' 2022-06-27T11:51:53.468 INFO:tasks.workunit.client.0.smithi195.stdout:removed './170/19' 2022-06-27T11:51:53.469 INFO:tasks.workunit.client.0.smithi195.stdout:removed './170/15' 2022-06-27T11:51:53.469 INFO:tasks.workunit.client.0.smithi195.stdout:removed './170/31' 2022-06-27T11:51:53.469 INFO:tasks.workunit.client.0.smithi195.stdout:removed './170/36' 2022-06-27T11:51:53.469 INFO:tasks.workunit.client.0.smithi195.stdout:removed './170/20' 2022-06-27T11:51:53.470 INFO:tasks.workunit.client.0.smithi195.stdout:removed './170/12' 2022-06-27T11:51:53.470 INFO:tasks.workunit.client.0.smithi195.stdout:removed './170/39' 2022-06-27T11:51:53.470 INFO:tasks.workunit.client.0.smithi195.stdout:removed './170/10' 2022-06-27T11:51:53.470 INFO:tasks.workunit.client.0.smithi195.stdout:removed './170/34' 2022-06-27T11:51:53.470 INFO:tasks.workunit.client.0.smithi195.stdout:removed './170/38' 2022-06-27T11:51:53.471 INFO:tasks.workunit.client.0.smithi195.stdout:removed './170/11' 2022-06-27T11:51:53.471 INFO:tasks.workunit.client.0.smithi195.stdout:removed './170/16' 2022-06-27T11:51:53.472 INFO:tasks.workunit.client.0.smithi195.stdout:removed './170/32' 2022-06-27T11:51:53.472 INFO:tasks.workunit.client.0.smithi195.stdout:removed './170/30' 2022-06-27T11:51:53.472 INFO:tasks.workunit.client.0.smithi195.stdout:removed './170/14' 2022-06-27T11:51:53.473 INFO:tasks.workunit.client.0.smithi195.stdout:removed './170/18' 2022-06-27T11:51:53.473 INFO:tasks.workunit.client.0.smithi195.stdout:removed './170/13' 2022-06-27T11:51:53.473 INFO:tasks.workunit.client.0.smithi195.stdout:removed './170/33' 2022-06-27T11:51:53.473 INFO:tasks.workunit.client.0.smithi195.stdout:removed './170/17' 2022-06-27T11:51:53.474 INFO:tasks.workunit.client.0.smithi195.stdout:removed './170/35' 2022-06-27T11:51:53.474 INFO:tasks.workunit.client.0.smithi195.stdout:removed './170/37' 2022-06-27T11:51:53.474 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './170' 2022-06-27T11:51:53.474 INFO:tasks.workunit.client.0.smithi195.stdout:removed './171/19' 2022-06-27T11:51:53.475 INFO:tasks.workunit.client.0.smithi195.stdout:removed './171/15' 2022-06-27T11:51:53.475 INFO:tasks.workunit.client.0.smithi195.stdout:removed './171/31' 2022-06-27T11:51:53.475 INFO:tasks.workunit.client.0.smithi195.stdout:removed './171/36' 2022-06-27T11:51:53.475 INFO:tasks.workunit.client.0.smithi195.stdout:removed './171/20' 2022-06-27T11:51:53.476 INFO:tasks.workunit.client.0.smithi195.stdout:removed './171/12' 2022-06-27T11:51:53.476 INFO:tasks.workunit.client.0.smithi195.stdout:removed './171/39' 2022-06-27T11:51:53.477 INFO:tasks.workunit.client.0.smithi195.stdout:removed './171/10' 2022-06-27T11:51:53.477 INFO:tasks.workunit.client.0.smithi195.stdout:removed './171/34' 2022-06-27T11:51:53.477 INFO:tasks.workunit.client.0.smithi195.stdout:removed './171/38' 2022-06-27T11:51:53.477 INFO:tasks.workunit.client.0.smithi195.stdout:removed './171/11' 2022-06-27T11:51:53.478 INFO:tasks.workunit.client.0.smithi195.stdout:removed './171/16' 2022-06-27T11:51:53.478 INFO:tasks.workunit.client.0.smithi195.stdout:removed './171/32' 2022-06-27T11:51:53.478 INFO:tasks.workunit.client.0.smithi195.stdout:removed './171/30' 2022-06-27T11:51:53.478 INFO:tasks.workunit.client.0.smithi195.stdout:removed './171/14' 2022-06-27T11:51:53.479 INFO:tasks.workunit.client.0.smithi195.stdout:removed './171/18' 2022-06-27T11:51:53.479 INFO:tasks.workunit.client.0.smithi195.stdout:removed './171/13' 2022-06-27T11:51:53.479 INFO:tasks.workunit.client.0.smithi195.stdout:removed './171/33' 2022-06-27T11:51:53.479 INFO:tasks.workunit.client.0.smithi195.stdout:removed './171/17' 2022-06-27T11:51:53.479 INFO:tasks.workunit.client.0.smithi195.stdout:removed './171/35' 2022-06-27T11:51:53.480 INFO:tasks.workunit.client.0.smithi195.stdout:removed './171/37' 2022-06-27T11:51:53.480 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './171' 2022-06-27T11:51:53.480 INFO:tasks.workunit.client.0.smithi195.stdout:removed './172/19' 2022-06-27T11:51:53.480 INFO:tasks.workunit.client.0.smithi195.stdout:removed './172/15' 2022-06-27T11:51:53.481 INFO:tasks.workunit.client.0.smithi195.stdout:removed './172/31' 2022-06-27T11:51:53.481 INFO:tasks.workunit.client.0.smithi195.stdout:removed './172/36' 2022-06-27T11:51:53.481 INFO:tasks.workunit.client.0.smithi195.stdout:removed './172/20' 2022-06-27T11:51:53.481 INFO:tasks.workunit.client.0.smithi195.stdout:removed './172/12' 2022-06-27T11:51:53.482 INFO:tasks.workunit.client.0.smithi195.stdout:removed './172/39' 2022-06-27T11:51:53.482 INFO:tasks.workunit.client.0.smithi195.stdout:removed './172/10' 2022-06-27T11:51:53.483 INFO:tasks.workunit.client.0.smithi195.stdout:removed './172/34' 2022-06-27T11:51:53.483 INFO:tasks.workunit.client.0.smithi195.stdout:removed './172/38' 2022-06-27T11:51:53.483 INFO:tasks.workunit.client.0.smithi195.stdout:removed './172/11' 2022-06-27T11:51:53.483 INFO:tasks.workunit.client.0.smithi195.stdout:removed './172/16' 2022-06-27T11:51:53.484 INFO:tasks.workunit.client.0.smithi195.stdout:removed './172/32' 2022-06-27T11:51:53.484 INFO:tasks.workunit.client.0.smithi195.stdout:removed './172/30' 2022-06-27T11:51:53.484 INFO:tasks.workunit.client.0.smithi195.stdout:removed './172/14' 2022-06-27T11:51:53.484 INFO:tasks.workunit.client.0.smithi195.stdout:removed './172/18' 2022-06-27T11:51:53.485 INFO:tasks.workunit.client.0.smithi195.stdout:removed './172/13' 2022-06-27T11:51:53.485 INFO:tasks.workunit.client.0.smithi195.stdout:removed './172/33' 2022-06-27T11:51:53.485 INFO:tasks.workunit.client.0.smithi195.stdout:removed './172/17' 2022-06-27T11:51:53.486 INFO:tasks.workunit.client.0.smithi195.stdout:removed './172/35' 2022-06-27T11:51:53.486 INFO:tasks.workunit.client.0.smithi195.stdout:removed './172/37' 2022-06-27T11:51:53.486 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './172' 2022-06-27T11:51:53.486 INFO:tasks.workunit.client.0.smithi195.stdout:removed './173/19' 2022-06-27T11:51:53.487 INFO:tasks.workunit.client.0.smithi195.stdout:removed './173/15' 2022-06-27T11:51:53.487 INFO:tasks.workunit.client.0.smithi195.stdout:removed './173/31' 2022-06-27T11:51:53.487 INFO:tasks.workunit.client.0.smithi195.stdout:removed './173/36' 2022-06-27T11:51:53.487 INFO:tasks.workunit.client.0.smithi195.stdout:removed './173/20' 2022-06-27T11:51:53.488 INFO:tasks.workunit.client.0.smithi195.stdout:removed './173/12' 2022-06-27T11:51:53.488 INFO:tasks.workunit.client.0.smithi195.stdout:removed './173/39' 2022-06-27T11:51:53.488 INFO:tasks.workunit.client.0.smithi195.stdout:removed './173/10' 2022-06-27T11:51:53.488 INFO:tasks.workunit.client.0.smithi195.stdout:removed './173/34' 2022-06-27T11:51:53.488 INFO:tasks.workunit.client.0.smithi195.stdout:removed './173/38' 2022-06-27T11:51:53.489 INFO:tasks.workunit.client.0.smithi195.stdout:removed './173/11' 2022-06-27T11:51:53.489 INFO:tasks.workunit.client.0.smithi195.stdout:removed './173/16' 2022-06-27T11:51:53.489 INFO:tasks.workunit.client.0.smithi195.stdout:removed './173/32' 2022-06-27T11:51:53.489 INFO:tasks.workunit.client.0.smithi195.stdout:removed './173/30' 2022-06-27T11:51:53.490 INFO:tasks.workunit.client.0.smithi195.stdout:removed './173/14' 2022-06-27T11:51:53.490 INFO:tasks.workunit.client.0.smithi195.stdout:removed './173/18' 2022-06-27T11:51:53.490 INFO:tasks.workunit.client.0.smithi195.stdout:removed './173/13' 2022-06-27T11:51:53.490 INFO:tasks.workunit.client.0.smithi195.stdout:removed './173/33' 2022-06-27T11:51:53.490 INFO:tasks.workunit.client.0.smithi195.stdout:removed './173/17' 2022-06-27T11:51:53.491 INFO:tasks.workunit.client.0.smithi195.stdout:removed './173/35' 2022-06-27T11:51:53.491 INFO:tasks.workunit.client.0.smithi195.stdout:removed './173/37' 2022-06-27T11:51:53.491 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './173' 2022-06-27T11:51:53.491 INFO:tasks.workunit.client.0.smithi195.stdout:removed './174/19' 2022-06-27T11:51:53.491 INFO:tasks.workunit.client.0.smithi195.stdout:removed './174/15' 2022-06-27T11:51:53.492 INFO:tasks.workunit.client.0.smithi195.stdout:removed './174/31' 2022-06-27T11:51:53.492 INFO:tasks.workunit.client.0.smithi195.stdout:removed './174/36' 2022-06-27T11:51:53.492 INFO:tasks.workunit.client.0.smithi195.stdout:removed './174/20' 2022-06-27T11:51:53.492 INFO:tasks.workunit.client.0.smithi195.stdout:removed './174/12' 2022-06-27T11:51:53.493 INFO:tasks.workunit.client.0.smithi195.stdout:removed './174/39' 2022-06-27T11:51:53.494 INFO:tasks.workunit.client.0.smithi195.stdout:removed './174/10' 2022-06-27T11:51:53.494 INFO:tasks.workunit.client.0.smithi195.stdout:removed './174/34' 2022-06-27T11:51:53.494 INFO:tasks.workunit.client.0.smithi195.stdout:removed './174/38' 2022-06-27T11:51:53.494 INFO:tasks.workunit.client.0.smithi195.stdout:removed './174/11' 2022-06-27T11:51:53.495 INFO:tasks.workunit.client.0.smithi195.stdout:removed './174/16' 2022-06-27T11:51:53.495 INFO:tasks.workunit.client.0.smithi195.stdout:removed './174/32' 2022-06-27T11:51:53.495 INFO:tasks.workunit.client.0.smithi195.stdout:removed './174/30' 2022-06-27T11:51:53.495 INFO:tasks.workunit.client.0.smithi195.stdout:removed './174/14' 2022-06-27T11:51:53.495 INFO:tasks.workunit.client.0.smithi195.stdout:removed './174/18' 2022-06-27T11:51:53.496 INFO:tasks.workunit.client.0.smithi195.stdout:removed './174/13' 2022-06-27T11:51:53.496 INFO:tasks.workunit.client.0.smithi195.stdout:removed './174/33' 2022-06-27T11:51:53.496 INFO:tasks.workunit.client.0.smithi195.stdout:removed './174/17' 2022-06-27T11:51:53.496 INFO:tasks.workunit.client.0.smithi195.stdout:removed './174/35' 2022-06-27T11:51:53.497 INFO:tasks.workunit.client.0.smithi195.stdout:removed './174/37' 2022-06-27T11:51:53.497 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './174' 2022-06-27T11:51:53.497 INFO:tasks.workunit.client.0.smithi195.stdout:removed './175/19' 2022-06-27T11:51:53.497 INFO:tasks.workunit.client.0.smithi195.stdout:removed './175/15' 2022-06-27T11:51:53.498 INFO:tasks.workunit.client.0.smithi195.stdout:removed './175/31' 2022-06-27T11:51:53.498 INFO:tasks.workunit.client.0.smithi195.stdout:removed './175/36' 2022-06-27T11:51:53.498 INFO:tasks.workunit.client.0.smithi195.stdout:removed './175/20' 2022-06-27T11:51:53.498 INFO:tasks.workunit.client.0.smithi195.stdout:removed './175/12' 2022-06-27T11:51:53.499 INFO:tasks.workunit.client.0.smithi195.stdout:removed './175/39' 2022-06-27T11:51:53.499 INFO:tasks.workunit.client.0.smithi195.stdout:removed './175/10' 2022-06-27T11:51:53.499 INFO:tasks.workunit.client.0.smithi195.stdout:removed './175/34' 2022-06-27T11:51:53.499 INFO:tasks.workunit.client.0.smithi195.stdout:removed './175/38' 2022-06-27T11:51:53.500 INFO:tasks.workunit.client.0.smithi195.stdout:removed './175/11' 2022-06-27T11:51:53.500 INFO:tasks.workunit.client.0.smithi195.stdout:removed './175/16' 2022-06-27T11:51:53.500 INFO:tasks.workunit.client.0.smithi195.stdout:removed './175/32' 2022-06-27T11:51:53.500 INFO:tasks.workunit.client.0.smithi195.stdout:removed './175/30' 2022-06-27T11:51:53.501 INFO:tasks.workunit.client.0.smithi195.stdout:removed './175/14' 2022-06-27T11:51:53.501 INFO:tasks.workunit.client.0.smithi195.stdout:removed './175/18' 2022-06-27T11:51:53.501 INFO:tasks.workunit.client.0.smithi195.stdout:removed './175/13' 2022-06-27T11:51:53.501 INFO:tasks.workunit.client.0.smithi195.stdout:removed './175/33' 2022-06-27T11:51:53.501 INFO:tasks.workunit.client.0.smithi195.stdout:removed './175/17' 2022-06-27T11:51:53.502 INFO:tasks.workunit.client.0.smithi195.stdout:removed './175/35' 2022-06-27T11:51:53.502 INFO:tasks.workunit.client.0.smithi195.stdout:removed './175/37' 2022-06-27T11:51:53.502 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './175' 2022-06-27T11:51:53.502 INFO:tasks.workunit.client.0.smithi195.stdout:removed './176/19' 2022-06-27T11:51:53.503 INFO:tasks.workunit.client.0.smithi195.stdout:removed './176/15' 2022-06-27T11:51:53.503 INFO:tasks.workunit.client.0.smithi195.stdout:removed './176/31' 2022-06-27T11:51:53.503 INFO:tasks.workunit.client.0.smithi195.stdout:removed './176/36' 2022-06-27T11:51:53.505 INFO:tasks.workunit.client.0.smithi195.stdout:removed './176/20' 2022-06-27T11:51:53.505 INFO:tasks.workunit.client.0.smithi195.stdout:removed './176/12' 2022-06-27T11:51:53.505 INFO:tasks.workunit.client.0.smithi195.stdout:removed './176/39' 2022-06-27T11:51:53.506 INFO:tasks.workunit.client.0.smithi195.stdout:removed './176/10' 2022-06-27T11:51:53.506 INFO:tasks.workunit.client.0.smithi195.stdout:removed './176/34' 2022-06-27T11:51:53.507 INFO:tasks.workunit.client.0.smithi195.stdout:removed './176/38' 2022-06-27T11:51:53.507 INFO:tasks.workunit.client.0.smithi195.stdout:removed './176/11' 2022-06-27T11:51:53.508 INFO:tasks.workunit.client.0.smithi195.stdout:removed './176/16' 2022-06-27T11:51:53.508 INFO:tasks.workunit.client.0.smithi195.stdout:removed './176/32' 2022-06-27T11:51:53.510 INFO:tasks.workunit.client.0.smithi195.stdout:removed './176/30' 2022-06-27T11:51:53.511 INFO:tasks.workunit.client.0.smithi195.stdout:removed './176/14' 2022-06-27T11:51:53.511 INFO:tasks.workunit.client.0.smithi195.stdout:removed './176/18' 2022-06-27T11:51:53.512 INFO:tasks.workunit.client.0.smithi195.stdout:removed './176/13' 2022-06-27T11:51:53.512 INFO:tasks.workunit.client.0.smithi195.stdout:removed './176/33' 2022-06-27T11:51:53.513 INFO:tasks.workunit.client.0.smithi195.stdout:removed './176/17' 2022-06-27T11:51:53.513 INFO:tasks.workunit.client.0.smithi195.stdout:removed './176/35' 2022-06-27T11:51:53.514 INFO:tasks.workunit.client.0.smithi195.stdout:removed './176/37' 2022-06-27T11:51:53.514 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './176' 2022-06-27T11:51:53.514 INFO:tasks.workunit.client.0.smithi195.stdout:removed './177/19' 2022-06-27T11:51:53.515 INFO:tasks.workunit.client.0.smithi195.stdout:removed './177/15' 2022-06-27T11:51:55.804 INFO:tasks.workunit.client.0.smithi195.stdout:removed './177/31' 2022-06-27T11:51:55.805 INFO:tasks.workunit.client.0.smithi195.stdout:removed './177/36' 2022-06-27T11:51:55.805 INFO:tasks.workunit.client.0.smithi195.stdout:removed './177/20' 2022-06-27T11:51:55.805 INFO:tasks.workunit.client.0.smithi195.stdout:removed './177/12' 2022-06-27T11:51:55.805 INFO:tasks.workunit.client.0.smithi195.stdout:removed './177/39' 2022-06-27T11:51:55.805 INFO:tasks.workunit.client.0.smithi195.stdout:removed './177/10' 2022-06-27T11:51:55.806 INFO:tasks.workunit.client.0.smithi195.stdout:removed './177/34' 2022-06-27T11:51:55.806 INFO:tasks.workunit.client.0.smithi195.stdout:removed './177/38' 2022-06-27T11:51:55.806 INFO:tasks.workunit.client.0.smithi195.stdout:removed './177/11' 2022-06-27T11:51:55.806 INFO:tasks.workunit.client.0.smithi195.stdout:removed './177/16' 2022-06-27T11:51:55.807 INFO:tasks.workunit.client.0.smithi195.stdout:removed './177/32' 2022-06-27T11:51:55.807 INFO:tasks.workunit.client.0.smithi195.stdout:removed './177/30' 2022-06-27T11:51:55.807 INFO:tasks.workunit.client.0.smithi195.stdout:removed './177/14' 2022-06-27T11:51:55.808 INFO:tasks.workunit.client.0.smithi195.stdout:removed './177/18' 2022-06-27T11:51:55.808 INFO:tasks.workunit.client.0.smithi195.stdout:removed './177/13' 2022-06-27T11:51:55.808 INFO:tasks.workunit.client.0.smithi195.stdout:removed './177/33' 2022-06-27T11:51:55.809 INFO:tasks.workunit.client.0.smithi195.stdout:removed './177/17' 2022-06-27T11:51:55.809 INFO:tasks.workunit.client.0.smithi195.stdout:removed './177/35' 2022-06-27T11:51:55.809 INFO:tasks.workunit.client.0.smithi195.stdout:removed './177/37' 2022-06-27T11:51:55.810 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './177' 2022-06-27T11:51:55.810 INFO:tasks.workunit.client.0.smithi195.stdout:removed './178/19' 2022-06-27T11:51:55.810 INFO:tasks.workunit.client.0.smithi195.stdout:removed './178/15' 2022-06-27T11:51:55.811 INFO:tasks.workunit.client.0.smithi195.stdout:removed './178/31' 2022-06-27T11:51:55.811 INFO:tasks.workunit.client.0.smithi195.stdout:removed './178/36' 2022-06-27T11:51:55.811 INFO:tasks.workunit.client.0.smithi195.stdout:removed './178/20' 2022-06-27T11:51:55.812 INFO:tasks.workunit.client.0.smithi195.stdout:removed './178/12' 2022-06-27T11:51:55.812 INFO:tasks.workunit.client.0.smithi195.stdout:removed './178/39' 2022-06-27T11:51:55.812 INFO:tasks.workunit.client.0.smithi195.stdout:removed './178/10' 2022-06-27T11:51:55.813 INFO:tasks.workunit.client.0.smithi195.stdout:removed './178/34' 2022-06-27T11:51:55.813 INFO:tasks.workunit.client.0.smithi195.stdout:removed './178/38' 2022-06-27T11:51:55.813 INFO:tasks.workunit.client.0.smithi195.stdout:removed './178/11' 2022-06-27T11:51:55.813 INFO:tasks.workunit.client.0.smithi195.stdout:removed './178/16' 2022-06-27T11:51:55.815 INFO:tasks.workunit.client.0.smithi195.stdout:removed './178/32' 2022-06-27T11:51:55.815 INFO:tasks.workunit.client.0.smithi195.stdout:removed './178/30' 2022-06-27T11:51:55.815 INFO:tasks.workunit.client.0.smithi195.stdout:removed './178/14' 2022-06-27T11:51:55.816 INFO:tasks.workunit.client.0.smithi195.stdout:removed './178/18' 2022-06-27T11:51:55.816 INFO:tasks.workunit.client.0.smithi195.stdout:removed './178/13' 2022-06-27T11:51:55.816 INFO:tasks.workunit.client.0.smithi195.stdout:removed './178/33' 2022-06-27T11:51:55.817 INFO:tasks.workunit.client.0.smithi195.stdout:removed './178/17' 2022-06-27T11:51:55.817 INFO:tasks.workunit.client.0.smithi195.stdout:removed './178/35' 2022-06-27T11:51:55.817 INFO:tasks.workunit.client.0.smithi195.stdout:removed './178/37' 2022-06-27T11:51:55.818 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './178' 2022-06-27T11:51:55.818 INFO:tasks.workunit.client.0.smithi195.stdout:removed './179/19' 2022-06-27T11:51:55.818 INFO:tasks.workunit.client.0.smithi195.stdout:removed './179/15' 2022-06-27T11:51:55.819 INFO:tasks.workunit.client.0.smithi195.stdout:removed './179/31' 2022-06-27T11:51:55.819 INFO:tasks.workunit.client.0.smithi195.stdout:removed './179/36' 2022-06-27T11:51:55.820 INFO:tasks.workunit.client.0.smithi195.stdout:removed './179/20' 2022-06-27T11:51:55.820 INFO:tasks.workunit.client.0.smithi195.stdout:removed './179/12' 2022-06-27T11:51:55.820 INFO:tasks.workunit.client.0.smithi195.stdout:removed './179/39' 2022-06-27T11:51:55.821 INFO:tasks.workunit.client.0.smithi195.stdout:removed './179/10' 2022-06-27T11:51:55.821 INFO:tasks.workunit.client.0.smithi195.stdout:removed './179/34' 2022-06-27T11:51:55.822 INFO:tasks.workunit.client.0.smithi195.stdout:removed './179/38' 2022-06-27T11:51:55.822 INFO:tasks.workunit.client.0.smithi195.stdout:removed './179/11' 2022-06-27T11:51:55.823 INFO:tasks.workunit.client.0.smithi195.stdout:removed './179/16' 2022-06-27T11:51:55.823 INFO:tasks.workunit.client.0.smithi195.stdout:removed './179/32' 2022-06-27T11:51:55.824 INFO:tasks.workunit.client.0.smithi195.stdout:removed './179/30' 2022-06-27T11:51:55.824 INFO:tasks.workunit.client.0.smithi195.stdout:removed './179/14' 2022-06-27T11:51:55.825 INFO:tasks.workunit.client.0.smithi195.stdout:removed './179/18' 2022-06-27T11:51:55.825 INFO:tasks.workunit.client.0.smithi195.stdout:removed './179/13' 2022-06-27T11:51:55.826 INFO:tasks.workunit.client.0.smithi195.stdout:removed './179/33' 2022-06-27T11:51:55.826 INFO:tasks.workunit.client.0.smithi195.stdout:removed './179/17' 2022-06-27T11:51:55.827 INFO:tasks.workunit.client.0.smithi195.stdout:removed './179/35' 2022-06-27T11:51:55.827 INFO:tasks.workunit.client.0.smithi195.stdout:removed './179/37' 2022-06-27T11:51:55.828 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './179' 2022-06-27T11:51:55.828 INFO:tasks.workunit.client.0.smithi195.stdout:removed './180/19' 2022-06-27T11:51:55.829 INFO:tasks.workunit.client.0.smithi195.stdout:removed './180/15' 2022-06-27T11:51:55.829 INFO:tasks.workunit.client.0.smithi195.stdout:removed './180/31' 2022-06-27T11:51:55.830 INFO:tasks.workunit.client.0.smithi195.stdout:removed './180/36' 2022-06-27T11:51:55.830 INFO:tasks.workunit.client.0.smithi195.stdout:removed './180/20' 2022-06-27T11:51:55.830 INFO:tasks.workunit.client.0.smithi195.stdout:removed './180/12' 2022-06-27T11:51:55.831 INFO:tasks.workunit.client.0.smithi195.stdout:removed './180/39' 2022-06-27T11:51:55.831 INFO:tasks.workunit.client.0.smithi195.stdout:removed './180/10' 2022-06-27T11:51:55.832 INFO:tasks.workunit.client.0.smithi195.stdout:removed './180/34' 2022-06-27T11:51:55.833 INFO:tasks.workunit.client.0.smithi195.stdout:removed './180/38' 2022-06-27T11:51:55.833 INFO:tasks.workunit.client.0.smithi195.stdout:removed './180/11' 2022-06-27T11:51:55.834 INFO:tasks.workunit.client.0.smithi195.stdout:removed './180/16' 2022-06-27T11:51:55.834 INFO:tasks.workunit.client.0.smithi195.stdout:removed './180/32' 2022-06-27T11:51:55.835 INFO:tasks.workunit.client.0.smithi195.stdout:removed './180/30' 2022-06-27T11:51:55.835 INFO:tasks.workunit.client.0.smithi195.stdout:removed './180/14' 2022-06-27T11:51:55.836 INFO:tasks.workunit.client.0.smithi195.stdout:removed './180/18' 2022-06-27T11:51:55.836 INFO:tasks.workunit.client.0.smithi195.stdout:removed './180/13' 2022-06-27T11:51:55.836 INFO:tasks.workunit.client.0.smithi195.stdout:removed './180/33' 2022-06-27T11:51:55.837 INFO:tasks.workunit.client.0.smithi195.stdout:removed './180/17' 2022-06-27T11:51:55.837 INFO:tasks.workunit.client.0.smithi195.stdout:removed './180/35' 2022-06-27T11:51:55.838 INFO:tasks.workunit.client.0.smithi195.stdout:removed './180/37' 2022-06-27T11:51:55.838 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './180' 2022-06-27T11:51:55.839 INFO:tasks.workunit.client.0.smithi195.stdout:removed './181/19' 2022-06-27T11:51:55.839 INFO:tasks.workunit.client.0.smithi195.stdout:removed './181/15' 2022-06-27T11:51:55.840 INFO:tasks.workunit.client.0.smithi195.stdout:removed './181/31' 2022-06-27T11:51:55.840 INFO:tasks.workunit.client.0.smithi195.stdout:removed './181/36' 2022-06-27T11:51:55.841 INFO:tasks.workunit.client.0.smithi195.stdout:removed './181/20' 2022-06-27T11:51:55.841 INFO:tasks.workunit.client.0.smithi195.stdout:removed './181/12' 2022-06-27T11:51:55.841 INFO:tasks.workunit.client.0.smithi195.stdout:removed './181/39' 2022-06-27T11:51:55.842 INFO:tasks.workunit.client.0.smithi195.stdout:removed './181/10' 2022-06-27T11:51:55.842 INFO:tasks.workunit.client.0.smithi195.stdout:removed './181/34' 2022-06-27T11:51:55.843 INFO:tasks.workunit.client.0.smithi195.stdout:removed './181/38' 2022-06-27T11:51:55.843 INFO:tasks.workunit.client.0.smithi195.stdout:removed './181/11' 2022-06-27T11:51:55.844 INFO:tasks.workunit.client.0.smithi195.stdout:removed './181/16' 2022-06-27T11:51:55.844 INFO:tasks.workunit.client.0.smithi195.stdout:removed './181/32' 2022-06-27T11:51:55.845 INFO:tasks.workunit.client.0.smithi195.stdout:removed './181/30' 2022-06-27T11:51:55.845 INFO:tasks.workunit.client.0.smithi195.stdout:removed './181/14' 2022-06-27T11:51:55.845 INFO:tasks.workunit.client.0.smithi195.stdout:removed './181/18' 2022-06-27T11:51:55.846 INFO:tasks.workunit.client.0.smithi195.stdout:removed './181/13' 2022-06-27T11:51:55.846 INFO:tasks.workunit.client.0.smithi195.stdout:removed './181/33' 2022-06-27T11:51:55.847 INFO:tasks.workunit.client.0.smithi195.stdout:removed './181/17' 2022-06-27T11:51:55.847 INFO:tasks.workunit.client.0.smithi195.stdout:removed './181/35' 2022-06-27T11:51:55.847 INFO:tasks.workunit.client.0.smithi195.stdout:removed './181/37' 2022-06-27T11:51:55.848 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './181' 2022-06-27T11:51:55.848 INFO:tasks.workunit.client.0.smithi195.stdout:removed './182/19' 2022-06-27T11:51:55.849 INFO:tasks.workunit.client.0.smithi195.stdout:removed './182/15' 2022-06-27T11:51:55.849 INFO:tasks.workunit.client.0.smithi195.stdout:removed './182/31' 2022-06-27T11:51:55.850 INFO:tasks.workunit.client.0.smithi195.stdout:removed './182/36' 2022-06-27T11:51:55.850 INFO:tasks.workunit.client.0.smithi195.stdout:removed './182/20' 2022-06-27T11:51:55.850 INFO:tasks.workunit.client.0.smithi195.stdout:removed './182/12' 2022-06-27T11:51:55.851 INFO:tasks.workunit.client.0.smithi195.stdout:removed './182/39' 2022-06-27T11:51:55.851 INFO:tasks.workunit.client.0.smithi195.stdout:removed './182/10' 2022-06-27T11:51:55.851 INFO:tasks.workunit.client.0.smithi195.stdout:removed './182/34' 2022-06-27T11:51:55.852 INFO:tasks.workunit.client.0.smithi195.stdout:removed './182/38' 2022-06-27T11:51:55.854 INFO:tasks.workunit.client.0.smithi195.stdout:removed './182/11' 2022-06-27T11:51:55.855 INFO:tasks.workunit.client.0.smithi195.stdout:removed './182/16' 2022-06-27T11:51:55.855 INFO:tasks.workunit.client.0.smithi195.stdout:removed './182/32' 2022-06-27T11:51:55.865 INFO:tasks.workunit.client.0.smithi195.stdout:removed './182/30' 2022-06-27T11:51:55.865 INFO:tasks.workunit.client.0.smithi195.stdout:removed './182/14' 2022-06-27T11:51:55.866 INFO:tasks.workunit.client.0.smithi195.stdout:removed './182/18' 2022-06-27T11:51:55.866 INFO:tasks.workunit.client.0.smithi195.stdout:removed './182/13' 2022-06-27T11:51:55.866 INFO:tasks.workunit.client.0.smithi195.stdout:removed './182/33' 2022-06-27T11:51:55.867 INFO:tasks.workunit.client.0.smithi195.stdout:removed './182/17' 2022-06-27T11:51:55.867 INFO:tasks.workunit.client.0.smithi195.stdout:removed './182/35' 2022-06-27T11:51:55.868 INFO:tasks.workunit.client.0.smithi195.stdout:removed './182/37' 2022-06-27T11:51:55.868 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './182' 2022-06-27T11:51:55.869 INFO:tasks.workunit.client.0.smithi195.stdout:removed './183/19' 2022-06-27T11:51:55.869 INFO:tasks.workunit.client.0.smithi195.stdout:removed './183/15' 2022-06-27T11:51:55.869 INFO:tasks.workunit.client.0.smithi195.stdout:removed './183/31' 2022-06-27T11:51:55.870 INFO:tasks.workunit.client.0.smithi195.stdout:removed './183/36' 2022-06-27T11:51:55.870 INFO:tasks.workunit.client.0.smithi195.stdout:removed './183/20' 2022-06-27T11:51:55.871 INFO:tasks.workunit.client.0.smithi195.stdout:removed './183/12' 2022-06-27T11:51:55.871 INFO:tasks.workunit.client.0.smithi195.stdout:removed './183/39' 2022-06-27T11:51:55.872 INFO:tasks.workunit.client.0.smithi195.stdout:removed './183/10' 2022-06-27T11:51:55.872 INFO:tasks.workunit.client.0.smithi195.stdout:removed './183/34' 2022-06-27T11:51:55.872 INFO:tasks.workunit.client.0.smithi195.stdout:removed './183/38' 2022-06-27T11:51:55.873 INFO:tasks.workunit.client.0.smithi195.stdout:removed './183/11' 2022-06-27T11:51:55.873 INFO:tasks.workunit.client.0.smithi195.stdout:removed './183/16' 2022-06-27T11:51:55.874 INFO:tasks.workunit.client.0.smithi195.stdout:removed './183/32' 2022-06-27T11:51:55.874 INFO:tasks.workunit.client.0.smithi195.stdout:removed './183/30' 2022-06-27T11:51:55.875 INFO:tasks.workunit.client.0.smithi195.stdout:removed './183/14' 2022-06-27T11:51:55.875 INFO:tasks.workunit.client.0.smithi195.stdout:removed './183/18' 2022-06-27T11:51:55.875 INFO:tasks.workunit.client.0.smithi195.stdout:removed './183/13' 2022-06-27T11:51:55.877 INFO:tasks.workunit.client.0.smithi195.stdout:removed './183/33' 2022-06-27T11:51:55.878 INFO:tasks.workunit.client.0.smithi195.stdout:removed './183/17' 2022-06-27T11:51:55.878 INFO:tasks.workunit.client.0.smithi195.stdout:removed './183/35' 2022-06-27T11:51:55.878 INFO:tasks.workunit.client.0.smithi195.stdout:removed './183/37' 2022-06-27T11:51:55.879 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './183' 2022-06-27T11:51:55.879 INFO:tasks.workunit.client.0.smithi195.stdout:removed './184/19' 2022-06-27T11:51:55.880 INFO:tasks.workunit.client.0.smithi195.stdout:removed './184/15' 2022-06-27T11:51:55.880 INFO:tasks.workunit.client.0.smithi195.stdout:removed './184/31' 2022-06-27T11:51:55.881 INFO:tasks.workunit.client.0.smithi195.stdout:removed './184/36' 2022-06-27T11:51:55.881 INFO:tasks.workunit.client.0.smithi195.stdout:removed './184/20' 2022-06-27T11:51:55.881 INFO:tasks.workunit.client.0.smithi195.stdout:removed './184/12' 2022-06-27T11:51:55.882 INFO:tasks.workunit.client.0.smithi195.stdout:removed './184/39' 2022-06-27T11:51:55.882 INFO:tasks.workunit.client.0.smithi195.stdout:removed './184/10' 2022-06-27T11:51:55.883 INFO:tasks.workunit.client.0.smithi195.stdout:removed './184/34' 2022-06-27T11:51:55.883 INFO:tasks.workunit.client.0.smithi195.stdout:removed './184/38' 2022-06-27T11:51:55.884 INFO:tasks.workunit.client.0.smithi195.stdout:removed './184/11' 2022-06-27T11:51:55.884 INFO:tasks.workunit.client.0.smithi195.stdout:removed './184/16' 2022-06-27T11:51:55.885 INFO:tasks.workunit.client.0.smithi195.stdout:removed './184/32' 2022-06-27T11:51:55.885 INFO:tasks.workunit.client.0.smithi195.stdout:removed './184/30' 2022-06-27T11:51:55.886 INFO:tasks.workunit.client.0.smithi195.stdout:removed './184/14' 2022-06-27T11:51:55.886 INFO:tasks.workunit.client.0.smithi195.stdout:removed './184/18' 2022-06-27T11:51:55.887 INFO:tasks.workunit.client.0.smithi195.stdout:removed './184/13' 2022-06-27T11:51:55.887 INFO:tasks.workunit.client.0.smithi195.stdout:removed './184/33' 2022-06-27T11:51:55.887 INFO:tasks.workunit.client.0.smithi195.stdout:removed './184/17' 2022-06-27T11:51:55.888 INFO:tasks.workunit.client.0.smithi195.stdout:removed './184/35' 2022-06-27T11:51:55.888 INFO:tasks.workunit.client.0.smithi195.stdout:removed './184/37' 2022-06-27T11:51:55.889 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './184' 2022-06-27T11:51:55.889 INFO:tasks.workunit.client.0.smithi195.stdout:removed './185/19' 2022-06-27T11:51:55.890 INFO:tasks.workunit.client.0.smithi195.stdout:removed './185/15' 2022-06-27T11:51:55.891 INFO:tasks.workunit.client.0.smithi195.stdout:removed './185/31' 2022-06-27T11:51:55.891 INFO:tasks.workunit.client.0.smithi195.stdout:removed './185/36' 2022-06-27T11:51:55.892 INFO:tasks.workunit.client.0.smithi195.stdout:removed './185/20' 2022-06-27T11:51:55.892 INFO:tasks.workunit.client.0.smithi195.stdout:removed './185/12' 2022-06-27T11:51:55.893 INFO:tasks.workunit.client.0.smithi195.stdout:removed './185/39' 2022-06-27T11:51:55.893 INFO:tasks.workunit.client.0.smithi195.stdout:removed './185/10' 2022-06-27T11:51:55.893 INFO:tasks.workunit.client.0.smithi195.stdout:removed './185/34' 2022-06-27T11:51:55.894 INFO:tasks.workunit.client.0.smithi195.stdout:removed './185/38' 2022-06-27T11:51:55.895 INFO:tasks.workunit.client.0.smithi195.stdout:removed './185/11' 2022-06-27T11:51:55.895 INFO:tasks.workunit.client.0.smithi195.stdout:removed './185/16' 2022-06-27T11:51:55.896 INFO:tasks.workunit.client.0.smithi195.stdout:removed './185/32' 2022-06-27T11:51:55.896 INFO:tasks.workunit.client.0.smithi195.stdout:removed './185/30' 2022-06-27T11:51:55.897 INFO:tasks.workunit.client.0.smithi195.stdout:removed './185/14' 2022-06-27T11:51:55.897 INFO:tasks.workunit.client.0.smithi195.stdout:removed './185/18' 2022-06-27T11:51:55.898 INFO:tasks.workunit.client.0.smithi195.stdout:removed './185/13' 2022-06-27T11:51:55.898 INFO:tasks.workunit.client.0.smithi195.stdout:removed './185/33' 2022-06-27T11:51:55.899 INFO:tasks.workunit.client.0.smithi195.stdout:removed './185/17' 2022-06-27T11:51:55.899 INFO:tasks.workunit.client.0.smithi195.stdout:removed './185/35' 2022-06-27T11:51:55.899 INFO:tasks.workunit.client.0.smithi195.stdout:removed './185/37' 2022-06-27T11:51:55.900 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './185' 2022-06-27T11:51:55.900 INFO:tasks.workunit.client.0.smithi195.stdout:removed './186/19' 2022-06-27T11:51:55.901 INFO:tasks.workunit.client.0.smithi195.stdout:removed './186/15' 2022-06-27T11:51:55.901 INFO:tasks.workunit.client.0.smithi195.stdout:removed './186/31' 2022-06-27T11:51:55.902 INFO:tasks.workunit.client.0.smithi195.stdout:removed './186/36' 2022-06-27T11:51:55.902 INFO:tasks.workunit.client.0.smithi195.stdout:removed './186/20' 2022-06-27T11:51:55.903 INFO:tasks.workunit.client.0.smithi195.stdout:removed './186/12' 2022-06-27T11:51:55.904 INFO:tasks.workunit.client.0.smithi195.stdout:removed './186/39' 2022-06-27T11:51:55.905 INFO:tasks.workunit.client.0.smithi195.stdout:removed './186/10' 2022-06-27T11:51:55.905 INFO:tasks.workunit.client.0.smithi195.stdout:removed './186/34' 2022-06-27T11:51:55.905 INFO:tasks.workunit.client.0.smithi195.stdout:removed './186/38' 2022-06-27T11:51:55.906 INFO:tasks.workunit.client.0.smithi195.stdout:removed './186/11' 2022-06-27T11:51:55.906 INFO:tasks.workunit.client.0.smithi195.stdout:removed './186/16' 2022-06-27T11:51:55.906 INFO:tasks.workunit.client.0.smithi195.stdout:removed './186/32' 2022-06-27T11:51:55.906 INFO:tasks.workunit.client.0.smithi195.stdout:removed './186/30' 2022-06-27T11:51:55.907 INFO:tasks.workunit.client.0.smithi195.stdout:removed './186/14' 2022-06-27T11:51:55.907 INFO:tasks.workunit.client.0.smithi195.stdout:removed './186/18' 2022-06-27T11:51:57.197 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T11:51:57.217 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T11:51:57.672 INFO:tasks.workunit.client.0.smithi195.stdout:removed './186/13' 2022-06-27T11:51:57.672 INFO:tasks.workunit.client.0.smithi195.stdout:removed './186/33' 2022-06-27T11:51:57.673 INFO:tasks.workunit.client.0.smithi195.stdout:removed './186/17' 2022-06-27T11:51:57.673 INFO:tasks.workunit.client.0.smithi195.stdout:removed './186/35' 2022-06-27T11:51:57.674 INFO:tasks.workunit.client.0.smithi195.stdout:removed './186/37' 2022-06-27T11:51:57.674 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './186' 2022-06-27T11:51:57.675 INFO:tasks.workunit.client.0.smithi195.stdout:removed './187/19' 2022-06-27T11:51:57.676 INFO:tasks.workunit.client.0.smithi195.stdout:removed './187/15' 2022-06-27T11:51:57.676 INFO:tasks.workunit.client.0.smithi195.stdout:removed './187/31' 2022-06-27T11:51:57.676 INFO:tasks.workunit.client.0.smithi195.stdout:removed './187/36' 2022-06-27T11:51:57.677 INFO:tasks.workunit.client.0.smithi195.stdout:removed './187/20' 2022-06-27T11:51:57.677 INFO:tasks.workunit.client.0.smithi195.stdout:removed './187/12' 2022-06-27T11:51:57.678 INFO:tasks.workunit.client.0.smithi195.stdout:removed './187/39' 2022-06-27T11:51:57.678 INFO:tasks.workunit.client.0.smithi195.stdout:removed './187/10' 2022-06-27T11:51:57.679 INFO:tasks.workunit.client.0.smithi195.stdout:removed './187/34' 2022-06-27T11:51:57.679 INFO:tasks.workunit.client.0.smithi195.stdout:removed './187/38' 2022-06-27T11:51:57.680 INFO:tasks.workunit.client.0.smithi195.stdout:removed './187/11' 2022-06-27T11:51:57.680 INFO:tasks.workunit.client.0.smithi195.stdout:removed './187/16' 2022-06-27T11:51:57.681 INFO:tasks.workunit.client.0.smithi195.stdout:removed './187/32' 2022-06-27T11:51:57.682 INFO:tasks.workunit.client.0.smithi195.stdout:removed './187/30' 2022-06-27T11:51:57.682 INFO:tasks.workunit.client.0.smithi195.stdout:removed './187/14' 2022-06-27T11:51:57.682 INFO:tasks.workunit.client.0.smithi195.stdout:removed './187/18' 2022-06-27T11:51:57.683 INFO:tasks.workunit.client.0.smithi195.stdout:removed './187/13' 2022-06-27T11:51:57.683 INFO:tasks.workunit.client.0.smithi195.stdout:removed './187/33' 2022-06-27T11:51:57.684 INFO:tasks.workunit.client.0.smithi195.stdout:removed './187/17' 2022-06-27T11:51:57.684 INFO:tasks.workunit.client.0.smithi195.stdout:removed './187/35' 2022-06-27T11:51:57.685 INFO:tasks.workunit.client.0.smithi195.stdout:removed './187/37' 2022-06-27T11:51:57.685 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './187' 2022-06-27T11:51:57.686 INFO:tasks.workunit.client.0.smithi195.stdout:removed './188/19' 2022-06-27T11:51:57.686 INFO:tasks.workunit.client.0.smithi195.stdout:removed './188/15' 2022-06-27T11:51:57.686 INFO:tasks.workunit.client.0.smithi195.stdout:removed './188/31' 2022-06-27T11:51:57.687 INFO:tasks.workunit.client.0.smithi195.stdout:removed './188/36' 2022-06-27T11:51:57.687 INFO:tasks.workunit.client.0.smithi195.stdout:removed './188/20' 2022-06-27T11:51:57.688 INFO:tasks.workunit.client.0.smithi195.stdout:removed './188/12' 2022-06-27T11:51:57.689 INFO:tasks.workunit.client.0.smithi195.stdout:removed './188/39' 2022-06-27T11:51:57.690 INFO:tasks.workunit.client.0.smithi195.stdout:removed './188/10' 2022-06-27T11:51:57.690 INFO:tasks.workunit.client.0.smithi195.stdout:removed './188/34' 2022-06-27T11:51:57.691 INFO:tasks.workunit.client.0.smithi195.stdout:removed './188/38' 2022-06-27T11:51:57.692 INFO:tasks.workunit.client.0.smithi195.stdout:removed './188/11' 2022-06-27T11:51:57.692 INFO:tasks.workunit.client.0.smithi195.stdout:removed './188/16' 2022-06-27T11:51:57.693 INFO:tasks.workunit.client.0.smithi195.stdout:removed './188/32' 2022-06-27T11:51:57.693 INFO:tasks.workunit.client.0.smithi195.stdout:removed './188/30' 2022-06-27T11:51:57.694 INFO:tasks.workunit.client.0.smithi195.stdout:removed './188/14' 2022-06-27T11:51:57.694 INFO:tasks.workunit.client.0.smithi195.stdout:removed './188/18' 2022-06-27T11:51:57.694 INFO:tasks.workunit.client.0.smithi195.stdout:removed './188/13' 2022-06-27T11:51:57.695 INFO:tasks.workunit.client.0.smithi195.stdout:removed './188/33' 2022-06-27T11:51:57.695 INFO:tasks.workunit.client.0.smithi195.stdout:removed './188/17' 2022-06-27T11:51:57.696 INFO:tasks.workunit.client.0.smithi195.stdout:removed './188/35' 2022-06-27T11:51:57.696 INFO:tasks.workunit.client.0.smithi195.stdout:removed './188/37' 2022-06-27T11:51:57.696 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './188' 2022-06-27T11:51:57.697 INFO:tasks.workunit.client.0.smithi195.stdout:removed './189/19' 2022-06-27T11:51:57.697 INFO:tasks.workunit.client.0.smithi195.stdout:removed './189/15' 2022-06-27T11:51:57.697 INFO:tasks.workunit.client.0.smithi195.stdout:removed './189/31' 2022-06-27T11:51:57.697 INFO:tasks.workunit.client.0.smithi195.stdout:removed './189/36' 2022-06-27T11:51:57.698 INFO:tasks.workunit.client.0.smithi195.stdout:removed './189/20' 2022-06-27T11:51:57.698 INFO:tasks.workunit.client.0.smithi195.stdout:removed './189/12' 2022-06-27T11:51:57.699 INFO:tasks.workunit.client.0.smithi195.stdout:removed './189/39' 2022-06-27T11:51:57.699 INFO:tasks.workunit.client.0.smithi195.stdout:removed './189/10' 2022-06-27T11:51:57.699 INFO:tasks.workunit.client.0.smithi195.stdout:removed './189/34' 2022-06-27T11:51:57.700 INFO:tasks.workunit.client.0.smithi195.stdout:removed './189/38' 2022-06-27T11:51:57.700 INFO:tasks.workunit.client.0.smithi195.stdout:removed './189/11' 2022-06-27T11:51:57.701 INFO:tasks.workunit.client.0.smithi195.stdout:removed './189/16' 2022-06-27T11:51:57.701 INFO:tasks.workunit.client.0.smithi195.stdout:removed './189/32' 2022-06-27T11:51:57.702 INFO:tasks.workunit.client.0.smithi195.stdout:removed './189/30' 2022-06-27T11:51:57.703 INFO:tasks.workunit.client.0.smithi195.stdout:removed './189/14' 2022-06-27T11:51:57.703 INFO:tasks.workunit.client.0.smithi195.stdout:removed './189/18' 2022-06-27T11:51:57.703 INFO:tasks.workunit.client.0.smithi195.stdout:removed './189/13' 2022-06-27T11:51:57.704 INFO:tasks.workunit.client.0.smithi195.stdout:removed './189/33' 2022-06-27T11:51:57.704 INFO:tasks.workunit.client.0.smithi195.stdout:removed './189/17' 2022-06-27T11:51:57.705 INFO:tasks.workunit.client.0.smithi195.stdout:removed './189/35' 2022-06-27T11:51:57.705 INFO:tasks.workunit.client.0.smithi195.stdout:removed './189/37' 2022-06-27T11:51:57.706 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './189' 2022-06-27T11:51:57.706 INFO:tasks.workunit.client.0.smithi195.stdout:removed './190/19' 2022-06-27T11:51:57.707 INFO:tasks.workunit.client.0.smithi195.stdout:removed './190/15' 2022-06-27T11:51:57.708 INFO:tasks.workunit.client.0.smithi195.stdout:removed './190/31' 2022-06-27T11:51:57.708 INFO:tasks.workunit.client.0.smithi195.stdout:removed './190/36' 2022-06-27T11:51:57.709 INFO:tasks.workunit.client.0.smithi195.stdout:removed './190/20' 2022-06-27T11:51:57.709 INFO:tasks.workunit.client.0.smithi195.stdout:removed './190/12' 2022-06-27T11:51:57.710 INFO:tasks.workunit.client.0.smithi195.stdout:removed './190/39' 2022-06-27T11:51:57.711 INFO:tasks.workunit.client.0.smithi195.stdout:removed './190/10' 2022-06-27T11:51:57.711 INFO:tasks.workunit.client.0.smithi195.stdout:removed './190/34' 2022-06-27T11:51:57.711 INFO:tasks.workunit.client.0.smithi195.stdout:removed './190/38' 2022-06-27T11:51:57.712 INFO:tasks.workunit.client.0.smithi195.stdout:removed './190/11' 2022-06-27T11:51:57.714 INFO:tasks.workunit.client.0.smithi195.stdout:removed './190/16' 2022-06-27T11:51:57.714 INFO:tasks.workunit.client.0.smithi195.stdout:removed './190/32' 2022-06-27T11:51:57.715 INFO:tasks.workunit.client.0.smithi195.stdout:removed './190/30' 2022-06-27T11:51:57.715 INFO:tasks.workunit.client.0.smithi195.stdout:removed './190/14' 2022-06-27T11:51:57.715 INFO:tasks.workunit.client.0.smithi195.stdout:removed './190/18' 2022-06-27T11:51:57.716 INFO:tasks.workunit.client.0.smithi195.stdout:removed './190/13' 2022-06-27T11:51:57.716 INFO:tasks.workunit.client.0.smithi195.stdout:removed './190/33' 2022-06-27T11:51:57.717 INFO:tasks.workunit.client.0.smithi195.stdout:removed './190/17' 2022-06-27T11:51:57.717 INFO:tasks.workunit.client.0.smithi195.stdout:removed './190/35' 2022-06-27T11:51:57.718 INFO:tasks.workunit.client.0.smithi195.stdout:removed './190/37' 2022-06-27T11:51:57.718 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './190' 2022-06-27T11:51:57.719 INFO:tasks.workunit.client.0.smithi195.stdout:removed './191/19' 2022-06-27T11:51:57.719 INFO:tasks.workunit.client.0.smithi195.stdout:removed './191/15' 2022-06-27T11:51:57.720 INFO:tasks.workunit.client.0.smithi195.stdout:removed './191/31' 2022-06-27T11:51:57.720 INFO:tasks.workunit.client.0.smithi195.stdout:removed './191/36' 2022-06-27T11:51:57.721 INFO:tasks.workunit.client.0.smithi195.stdout:removed './191/20' 2022-06-27T11:51:57.721 INFO:tasks.workunit.client.0.smithi195.stdout:removed './191/12' 2022-06-27T11:51:57.722 INFO:tasks.workunit.client.0.smithi195.stdout:removed './191/39' 2022-06-27T11:51:57.722 INFO:tasks.workunit.client.0.smithi195.stdout:removed './191/10' 2022-06-27T11:51:57.723 INFO:tasks.workunit.client.0.smithi195.stdout:removed './191/34' 2022-06-27T11:51:57.723 INFO:tasks.workunit.client.0.smithi195.stdout:removed './191/38' 2022-06-27T11:51:57.724 INFO:tasks.workunit.client.0.smithi195.stdout:removed './191/11' 2022-06-27T11:51:57.725 INFO:tasks.workunit.client.0.smithi195.stdout:removed './191/16' 2022-06-27T11:51:57.725 INFO:tasks.workunit.client.0.smithi195.stdout:removed './191/32' 2022-06-27T11:51:57.726 INFO:tasks.workunit.client.0.smithi195.stdout:removed './191/30' 2022-06-27T11:51:57.726 INFO:tasks.workunit.client.0.smithi195.stdout:removed './191/14' 2022-06-27T11:51:57.727 INFO:tasks.workunit.client.0.smithi195.stdout:removed './191/18' 2022-06-27T11:51:57.727 INFO:tasks.workunit.client.0.smithi195.stdout:removed './191/13' 2022-06-27T11:51:57.728 INFO:tasks.workunit.client.0.smithi195.stdout:removed './191/33' 2022-06-27T11:51:57.728 INFO:tasks.workunit.client.0.smithi195.stdout:removed './191/17' 2022-06-27T11:51:57.729 INFO:tasks.workunit.client.0.smithi195.stdout:removed './191/35' 2022-06-27T11:51:57.730 INFO:tasks.workunit.client.0.smithi195.stdout:removed './191/37' 2022-06-27T11:51:57.730 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './191' 2022-06-27T11:51:57.730 INFO:tasks.workunit.client.0.smithi195.stdout:removed './192/19' 2022-06-27T11:51:57.731 INFO:tasks.workunit.client.0.smithi195.stdout:removed './192/15' 2022-06-27T11:51:57.731 INFO:tasks.workunit.client.0.smithi195.stdout:removed './192/31' 2022-06-27T11:51:57.732 INFO:tasks.workunit.client.0.smithi195.stdout:removed './192/36' 2022-06-27T11:51:57.732 INFO:tasks.workunit.client.0.smithi195.stdout:removed './192/20' 2022-06-27T11:51:57.733 INFO:tasks.workunit.client.0.smithi195.stdout:removed './192/12' 2022-06-27T11:51:57.733 INFO:tasks.workunit.client.0.smithi195.stdout:removed './192/39' 2022-06-27T11:51:57.734 INFO:tasks.workunit.client.0.smithi195.stdout:removed './192/10' 2022-06-27T11:51:57.734 INFO:tasks.workunit.client.0.smithi195.stdout:removed './192/34' 2022-06-27T11:51:57.735 INFO:tasks.workunit.client.0.smithi195.stdout:removed './192/38' 2022-06-27T11:51:57.735 INFO:tasks.workunit.client.0.smithi195.stdout:removed './192/11' 2022-06-27T11:51:57.736 INFO:tasks.workunit.client.0.smithi195.stdout:removed './192/16' 2022-06-27T11:51:57.736 INFO:tasks.workunit.client.0.smithi195.stdout:removed './192/32' 2022-06-27T11:51:57.736 INFO:tasks.workunit.client.0.smithi195.stdout:removed './192/30' 2022-06-27T11:51:57.736 INFO:tasks.workunit.client.0.smithi195.stdout:removed './192/14' 2022-06-27T11:51:57.736 INFO:tasks.workunit.client.0.smithi195.stdout:removed './192/18' 2022-06-27T11:51:57.737 INFO:tasks.workunit.client.0.smithi195.stdout:removed './192/13' 2022-06-27T11:51:57.737 INFO:tasks.workunit.client.0.smithi195.stdout:removed './192/33' 2022-06-27T11:51:57.737 INFO:tasks.workunit.client.0.smithi195.stdout:removed './192/17' 2022-06-27T11:51:57.737 INFO:tasks.workunit.client.0.smithi195.stdout:removed './192/35' 2022-06-27T11:51:57.737 INFO:tasks.workunit.client.0.smithi195.stdout:removed './192/37' 2022-06-27T11:51:57.738 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './192' 2022-06-27T11:51:57.738 INFO:tasks.workunit.client.0.smithi195.stdout:removed './193/19' 2022-06-27T11:51:57.738 INFO:tasks.workunit.client.0.smithi195.stdout:removed './193/15' 2022-06-27T11:51:57.738 INFO:tasks.workunit.client.0.smithi195.stdout:removed './193/31' 2022-06-27T11:51:57.739 INFO:tasks.workunit.client.0.smithi195.stdout:removed './193/36' 2022-06-27T11:51:57.739 INFO:tasks.workunit.client.0.smithi195.stdout:removed './193/20' 2022-06-27T11:51:57.739 INFO:tasks.workunit.client.0.smithi195.stdout:removed './193/12' 2022-06-27T11:51:57.739 INFO:tasks.workunit.client.0.smithi195.stdout:removed './193/39' 2022-06-27T11:51:57.739 INFO:tasks.workunit.client.0.smithi195.stdout:removed './193/10' 2022-06-27T11:51:57.740 INFO:tasks.workunit.client.0.smithi195.stdout:removed './193/34' 2022-06-27T11:51:57.740 INFO:tasks.workunit.client.0.smithi195.stdout:removed './193/38' 2022-06-27T11:51:57.740 INFO:tasks.workunit.client.0.smithi195.stdout:removed './193/11' 2022-06-27T11:51:57.740 INFO:tasks.workunit.client.0.smithi195.stdout:removed './193/16' 2022-06-27T11:51:57.741 INFO:tasks.workunit.client.0.smithi195.stdout:removed './193/32' 2022-06-27T11:51:57.741 INFO:tasks.workunit.client.0.smithi195.stdout:removed './193/30' 2022-06-27T11:51:57.741 INFO:tasks.workunit.client.0.smithi195.stdout:removed './193/14' 2022-06-27T11:51:57.741 INFO:tasks.workunit.client.0.smithi195.stdout:removed './193/18' 2022-06-27T11:51:57.741 INFO:tasks.workunit.client.0.smithi195.stdout:removed './193/13' 2022-06-27T11:51:57.742 INFO:tasks.workunit.client.0.smithi195.stdout:removed './193/33' 2022-06-27T11:51:57.742 INFO:tasks.workunit.client.0.smithi195.stdout:removed './193/17' 2022-06-27T11:51:57.742 INFO:tasks.workunit.client.0.smithi195.stdout:removed './193/35' 2022-06-27T11:51:57.742 INFO:tasks.workunit.client.0.smithi195.stdout:removed './193/37' 2022-06-27T11:51:57.742 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './193' 2022-06-27T11:51:57.743 INFO:tasks.workunit.client.0.smithi195.stdout:removed './194/19' 2022-06-27T11:51:57.743 INFO:tasks.workunit.client.0.smithi195.stdout:removed './194/15' 2022-06-27T11:51:57.743 INFO:tasks.workunit.client.0.smithi195.stdout:removed './194/31' 2022-06-27T11:51:57.743 INFO:tasks.workunit.client.0.smithi195.stdout:removed './194/36' 2022-06-27T11:51:57.744 INFO:tasks.workunit.client.0.smithi195.stdout:removed './194/20' 2022-06-27T11:51:57.744 INFO:tasks.workunit.client.0.smithi195.stdout:removed './194/12' 2022-06-27T11:51:57.744 INFO:tasks.workunit.client.0.smithi195.stdout:removed './194/39' 2022-06-27T11:51:57.744 INFO:tasks.workunit.client.0.smithi195.stdout:removed './194/10' 2022-06-27T11:51:57.744 INFO:tasks.workunit.client.0.smithi195.stdout:removed './194/34' 2022-06-27T11:51:57.745 INFO:tasks.workunit.client.0.smithi195.stdout:removed './194/38' 2022-06-27T11:51:57.745 INFO:tasks.workunit.client.0.smithi195.stdout:removed './194/11' 2022-06-27T11:51:57.745 INFO:tasks.workunit.client.0.smithi195.stdout:removed './194/16' 2022-06-27T11:51:57.745 INFO:tasks.workunit.client.0.smithi195.stdout:removed './194/32' 2022-06-27T11:51:57.746 INFO:tasks.workunit.client.0.smithi195.stdout:removed './194/30' 2022-06-27T11:51:57.746 INFO:tasks.workunit.client.0.smithi195.stdout:removed './194/14' 2022-06-27T11:51:57.746 INFO:tasks.workunit.client.0.smithi195.stdout:removed './194/18' 2022-06-27T11:51:57.746 INFO:tasks.workunit.client.0.smithi195.stdout:removed './194/13' 2022-06-27T11:51:57.746 INFO:tasks.workunit.client.0.smithi195.stdout:removed './194/33' 2022-06-27T11:51:57.747 INFO:tasks.workunit.client.0.smithi195.stdout:removed './194/17' 2022-06-27T11:51:57.747 INFO:tasks.workunit.client.0.smithi195.stdout:removed './194/35' 2022-06-27T11:51:57.747 INFO:tasks.workunit.client.0.smithi195.stdout:removed './194/37' 2022-06-27T11:51:57.747 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './194' 2022-06-27T11:51:57.748 INFO:tasks.workunit.client.0.smithi195.stdout:removed './195/19' 2022-06-27T11:51:57.748 INFO:tasks.workunit.client.0.smithi195.stdout:removed './195/15' 2022-06-27T11:51:57.748 INFO:tasks.workunit.client.0.smithi195.stdout:removed './195/31' 2022-06-27T11:51:57.748 INFO:tasks.workunit.client.0.smithi195.stdout:removed './195/36' 2022-06-27T11:51:57.749 INFO:tasks.workunit.client.0.smithi195.stdout:removed './195/20' 2022-06-27T11:51:57.749 INFO:tasks.workunit.client.0.smithi195.stdout:removed './195/12' 2022-06-27T11:51:57.749 INFO:tasks.workunit.client.0.smithi195.stdout:removed './195/39' 2022-06-27T11:51:57.749 INFO:tasks.workunit.client.0.smithi195.stdout:removed './195/10' 2022-06-27T11:51:57.750 INFO:tasks.workunit.client.0.smithi195.stdout:removed './195/34' 2022-06-27T11:51:57.750 INFO:tasks.workunit.client.0.smithi195.stdout:removed './195/38' 2022-06-27T11:51:57.750 INFO:tasks.workunit.client.0.smithi195.stdout:removed './195/11' 2022-06-27T11:51:57.750 INFO:tasks.workunit.client.0.smithi195.stdout:removed './195/16' 2022-06-27T11:51:57.750 INFO:tasks.workunit.client.0.smithi195.stdout:removed './195/32' 2022-06-27T11:51:57.751 INFO:tasks.workunit.client.0.smithi195.stdout:removed './195/30' 2022-06-27T11:51:57.751 INFO:tasks.workunit.client.0.smithi195.stdout:removed './195/14' 2022-06-27T11:51:57.751 INFO:tasks.workunit.client.0.smithi195.stdout:removed './195/18' 2022-06-27T11:51:57.751 INFO:tasks.workunit.client.0.smithi195.stdout:removed './195/13' 2022-06-27T11:51:57.751 INFO:tasks.workunit.client.0.smithi195.stdout:removed './195/33' 2022-06-27T11:51:57.752 INFO:tasks.workunit.client.0.smithi195.stdout:removed './195/17' 2022-06-27T11:51:57.752 INFO:tasks.workunit.client.0.smithi195.stdout:removed './195/35' 2022-06-27T11:51:57.752 INFO:tasks.workunit.client.0.smithi195.stdout:removed './195/37' 2022-06-27T11:51:57.752 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './195' 2022-06-27T11:51:57.753 INFO:tasks.workunit.client.0.smithi195.stdout:removed './196/19' 2022-06-27T11:51:57.753 INFO:tasks.workunit.client.0.smithi195.stdout:removed './196/15' 2022-06-27T11:51:57.753 INFO:tasks.workunit.client.0.smithi195.stdout:removed './196/31' 2022-06-27T11:51:57.753 INFO:tasks.workunit.client.0.smithi195.stdout:removed './196/36' 2022-06-27T11:51:57.754 INFO:tasks.workunit.client.0.smithi195.stdout:removed './196/20' 2022-06-27T11:51:57.754 INFO:tasks.workunit.client.0.smithi195.stdout:removed './196/12' 2022-06-27T11:51:57.754 INFO:tasks.workunit.client.0.smithi195.stdout:removed './196/39' 2022-06-27T11:51:57.754 INFO:tasks.workunit.client.0.smithi195.stdout:removed './196/10' 2022-06-27T11:52:00.653 INFO:tasks.workunit.client.0.smithi195.stdout:removed './196/34' 2022-06-27T11:52:00.653 INFO:tasks.workunit.client.0.smithi195.stdout:removed './196/38' 2022-06-27T11:52:00.654 INFO:tasks.workunit.client.0.smithi195.stdout:removed './196/11' 2022-06-27T11:52:00.654 INFO:tasks.workunit.client.0.smithi195.stdout:removed './196/16' 2022-06-27T11:52:00.655 INFO:tasks.workunit.client.0.smithi195.stdout:removed './196/32' 2022-06-27T11:52:00.655 INFO:tasks.workunit.client.0.smithi195.stdout:removed './196/30' 2022-06-27T11:52:00.656 INFO:tasks.workunit.client.0.smithi195.stdout:removed './196/14' 2022-06-27T11:52:00.656 INFO:tasks.workunit.client.0.smithi195.stdout:removed './196/18' 2022-06-27T11:52:00.656 INFO:tasks.workunit.client.0.smithi195.stdout:removed './196/13' 2022-06-27T11:52:00.656 INFO:tasks.workunit.client.0.smithi195.stdout:removed './196/33' 2022-06-27T11:52:00.657 INFO:tasks.workunit.client.0.smithi195.stdout:removed './196/17' 2022-06-27T11:52:00.659 INFO:tasks.workunit.client.0.smithi195.stdout:removed './196/35' 2022-06-27T11:52:00.663 INFO:tasks.workunit.client.0.smithi195.stdout:removed './196/37' 2022-06-27T11:52:00.663 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './196' 2022-06-27T11:52:00.664 INFO:tasks.workunit.client.0.smithi195.stdout:removed './197/19' 2022-06-27T11:52:00.664 INFO:tasks.workunit.client.0.smithi195.stdout:removed './197/15' 2022-06-27T11:52:00.664 INFO:tasks.workunit.client.0.smithi195.stdout:removed './197/31' 2022-06-27T11:52:00.665 INFO:tasks.workunit.client.0.smithi195.stdout:removed './197/36' 2022-06-27T11:52:00.665 INFO:tasks.workunit.client.0.smithi195.stdout:removed './197/20' 2022-06-27T11:52:00.665 INFO:tasks.workunit.client.0.smithi195.stdout:removed './197/12' 2022-06-27T11:52:00.665 INFO:tasks.workunit.client.0.smithi195.stdout:removed './197/39' 2022-06-27T11:52:00.666 INFO:tasks.workunit.client.0.smithi195.stdout:removed './197/10' 2022-06-27T11:52:00.666 INFO:tasks.workunit.client.0.smithi195.stdout:removed './197/34' 2022-06-27T11:52:00.667 INFO:tasks.workunit.client.0.smithi195.stdout:removed './197/38' 2022-06-27T11:52:00.667 INFO:tasks.workunit.client.0.smithi195.stdout:removed './197/11' 2022-06-27T11:52:00.668 INFO:tasks.workunit.client.0.smithi195.stdout:removed './197/16' 2022-06-27T11:52:00.668 INFO:tasks.workunit.client.0.smithi195.stdout:removed './197/32' 2022-06-27T11:52:00.669 INFO:tasks.workunit.client.0.smithi195.stdout:removed './197/30' 2022-06-27T11:52:00.670 INFO:tasks.workunit.client.0.smithi195.stdout:removed './197/14' 2022-06-27T11:52:00.670 INFO:tasks.workunit.client.0.smithi195.stdout:removed './197/18' 2022-06-27T11:52:00.671 INFO:tasks.workunit.client.0.smithi195.stdout:removed './197/13' 2022-06-27T11:52:00.671 INFO:tasks.workunit.client.0.smithi195.stdout:removed './197/33' 2022-06-27T11:52:00.672 INFO:tasks.workunit.client.0.smithi195.stdout:removed './197/17' 2022-06-27T11:52:00.675 INFO:tasks.workunit.client.0.smithi195.stdout:removed './197/35' 2022-06-27T11:52:00.675 INFO:tasks.workunit.client.0.smithi195.stdout:removed './197/37' 2022-06-27T11:52:00.676 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './197' 2022-06-27T11:52:00.676 INFO:tasks.workunit.client.0.smithi195.stdout:removed './198/19' 2022-06-27T11:52:00.676 INFO:tasks.workunit.client.0.smithi195.stdout:removed './198/15' 2022-06-27T11:52:00.677 INFO:tasks.workunit.client.0.smithi195.stdout:removed './198/31' 2022-06-27T11:52:00.677 INFO:tasks.workunit.client.0.smithi195.stdout:removed './198/36' 2022-06-27T11:52:00.677 INFO:tasks.workunit.client.0.smithi195.stdout:removed './198/20' 2022-06-27T11:52:00.678 INFO:tasks.workunit.client.0.smithi195.stdout:removed './198/12' 2022-06-27T11:52:00.678 INFO:tasks.workunit.client.0.smithi195.stdout:removed './198/39' 2022-06-27T11:52:00.678 INFO:tasks.workunit.client.0.smithi195.stdout:removed './198/10' 2022-06-27T11:52:00.679 INFO:tasks.workunit.client.0.smithi195.stdout:removed './198/34' 2022-06-27T11:52:00.679 INFO:tasks.workunit.client.0.smithi195.stdout:removed './198/38' 2022-06-27T11:52:00.680 INFO:tasks.workunit.client.0.smithi195.stdout:removed './198/11' 2022-06-27T11:52:00.681 INFO:tasks.workunit.client.0.smithi195.stdout:removed './198/16' 2022-06-27T11:52:00.681 INFO:tasks.workunit.client.0.smithi195.stdout:removed './198/32' 2022-06-27T11:52:00.681 INFO:tasks.workunit.client.0.smithi195.stdout:removed './198/30' 2022-06-27T11:52:00.682 INFO:tasks.workunit.client.0.smithi195.stdout:removed './198/14' 2022-06-27T11:52:00.682 INFO:tasks.workunit.client.0.smithi195.stdout:removed './198/18' 2022-06-27T11:52:00.683 INFO:tasks.workunit.client.0.smithi195.stdout:removed './198/13' 2022-06-27T11:52:00.683 INFO:tasks.workunit.client.0.smithi195.stdout:removed './198/33' 2022-06-27T11:52:00.684 INFO:tasks.workunit.client.0.smithi195.stdout:removed './198/17' 2022-06-27T11:52:00.684 INFO:tasks.workunit.client.0.smithi195.stdout:removed './198/35' 2022-06-27T11:52:00.685 INFO:tasks.workunit.client.0.smithi195.stdout:removed './198/37' 2022-06-27T11:52:00.685 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './198' 2022-06-27T11:52:00.685 INFO:tasks.workunit.client.0.smithi195.stdout:removed './199/19' 2022-06-27T11:52:00.686 INFO:tasks.workunit.client.0.smithi195.stdout:removed './199/15' 2022-06-27T11:52:00.686 INFO:tasks.workunit.client.0.smithi195.stdout:removed './199/31' 2022-06-27T11:52:00.687 INFO:tasks.workunit.client.0.smithi195.stdout:removed './199/36' 2022-06-27T11:52:00.687 INFO:tasks.workunit.client.0.smithi195.stdout:removed './199/20' 2022-06-27T11:52:00.688 INFO:tasks.workunit.client.0.smithi195.stdout:removed './199/12' 2022-06-27T11:52:00.688 INFO:tasks.workunit.client.0.smithi195.stdout:removed './199/39' 2022-06-27T11:52:00.689 INFO:tasks.workunit.client.0.smithi195.stdout:removed './199/10' 2022-06-27T11:52:00.689 INFO:tasks.workunit.client.0.smithi195.stdout:removed './199/34' 2022-06-27T11:52:00.689 INFO:tasks.workunit.client.0.smithi195.stdout:removed './199/38' 2022-06-27T11:52:00.690 INFO:tasks.workunit.client.0.smithi195.stdout:removed './199/11' 2022-06-27T11:52:00.691 INFO:tasks.workunit.client.0.smithi195.stdout:removed './199/16' 2022-06-27T11:52:00.692 INFO:tasks.workunit.client.0.smithi195.stdout:removed './199/32' 2022-06-27T11:52:00.692 INFO:tasks.workunit.client.0.smithi195.stdout:removed './199/30' 2022-06-27T11:52:00.692 INFO:tasks.workunit.client.0.smithi195.stdout:removed './199/14' 2022-06-27T11:52:00.692 INFO:tasks.workunit.client.0.smithi195.stdout:removed './199/18' 2022-06-27T11:52:00.693 INFO:tasks.workunit.client.0.smithi195.stdout:removed './199/13' 2022-06-27T11:52:00.693 INFO:tasks.workunit.client.0.smithi195.stdout:removed './199/33' 2022-06-27T11:52:00.693 INFO:tasks.workunit.client.0.smithi195.stdout:removed './199/17' 2022-06-27T11:52:00.693 INFO:tasks.workunit.client.0.smithi195.stdout:removed './199/35' 2022-06-27T11:52:00.694 INFO:tasks.workunit.client.0.smithi195.stdout:removed './199/37' 2022-06-27T11:52:00.694 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './199' 2022-06-27T11:52:00.694 INFO:tasks.workunit.client.0.smithi195.stdout:removed './200/21' 2022-06-27T11:52:00.694 INFO:tasks.workunit.client.0.smithi195.stdout:removed './200/27' 2022-06-27T11:52:00.694 INFO:tasks.workunit.client.0.smithi195.stdout:removed './200/31' 2022-06-27T11:52:00.695 INFO:tasks.workunit.client.0.smithi195.stdout:removed './200/36' 2022-06-27T11:52:00.695 INFO:tasks.workunit.client.0.smithi195.stdout:removed './200/20' 2022-06-27T11:52:00.695 INFO:tasks.workunit.client.0.smithi195.stdout:removed './200/39' 2022-06-27T11:52:00.695 INFO:tasks.workunit.client.0.smithi195.stdout:removed './200/34' 2022-06-27T11:52:00.695 INFO:tasks.workunit.client.0.smithi195.stdout:removed './200/38' 2022-06-27T11:52:00.696 INFO:tasks.workunit.client.0.smithi195.stdout:removed './200/28' 2022-06-27T11:52:00.696 INFO:tasks.workunit.client.0.smithi195.stdout:removed './200/24' 2022-06-27T11:52:00.696 INFO:tasks.workunit.client.0.smithi195.stdout:removed './200/29' 2022-06-27T11:52:00.696 INFO:tasks.workunit.client.0.smithi195.stdout:removed './200/32' 2022-06-27T11:52:00.697 INFO:tasks.workunit.client.0.smithi195.stdout:removed './200/22' 2022-06-27T11:52:00.697 INFO:tasks.workunit.client.0.smithi195.stdout:removed './200/25' 2022-06-27T11:52:00.697 INFO:tasks.workunit.client.0.smithi195.stdout:removed './200/30' 2022-06-27T11:52:00.697 INFO:tasks.workunit.client.0.smithi195.stdout:removed './200/33' 2022-06-27T11:52:00.697 INFO:tasks.workunit.client.0.smithi195.stdout:removed './200/23' 2022-06-27T11:52:00.698 INFO:tasks.workunit.client.0.smithi195.stdout:removed './200/35' 2022-06-27T11:52:00.698 INFO:tasks.workunit.client.0.smithi195.stdout:removed './200/26' 2022-06-27T11:52:00.698 INFO:tasks.workunit.client.0.smithi195.stdout:removed './200/37' 2022-06-27T11:52:00.698 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './200' 2022-06-27T11:52:00.699 INFO:tasks.workunit.client.0.smithi195.stdout:removed './201/21' 2022-06-27T11:52:00.699 INFO:tasks.workunit.client.0.smithi195.stdout:removed './201/27' 2022-06-27T11:52:00.699 INFO:tasks.workunit.client.0.smithi195.stdout:removed './201/31' 2022-06-27T11:52:00.699 INFO:tasks.workunit.client.0.smithi195.stdout:removed './201/36' 2022-06-27T11:52:00.699 INFO:tasks.workunit.client.0.smithi195.stdout:removed './201/20' 2022-06-27T11:52:00.700 INFO:tasks.workunit.client.0.smithi195.stdout:removed './201/39' 2022-06-27T11:52:00.700 INFO:tasks.workunit.client.0.smithi195.stdout:removed './201/34' 2022-06-27T11:52:00.700 INFO:tasks.workunit.client.0.smithi195.stdout:removed './201/38' 2022-06-27T11:52:00.700 INFO:tasks.workunit.client.0.smithi195.stdout:removed './201/28' 2022-06-27T11:52:00.700 INFO:tasks.workunit.client.0.smithi195.stdout:removed './201/24' 2022-06-27T11:52:00.701 INFO:tasks.workunit.client.0.smithi195.stdout:removed './201/29' 2022-06-27T11:52:00.701 INFO:tasks.workunit.client.0.smithi195.stdout:removed './201/32' 2022-06-27T11:52:00.701 INFO:tasks.workunit.client.0.smithi195.stdout:removed './201/22' 2022-06-27T11:52:00.701 INFO:tasks.workunit.client.0.smithi195.stdout:removed './201/25' 2022-06-27T11:52:00.701 INFO:tasks.workunit.client.0.smithi195.stdout:removed './201/30' 2022-06-27T11:52:00.702 INFO:tasks.workunit.client.0.smithi195.stdout:removed './201/33' 2022-06-27T11:52:00.702 INFO:tasks.workunit.client.0.smithi195.stdout:removed './201/23' 2022-06-27T11:52:00.702 INFO:tasks.workunit.client.0.smithi195.stdout:removed './201/35' 2022-06-27T11:52:00.702 INFO:tasks.workunit.client.0.smithi195.stdout:removed './201/26' 2022-06-27T11:52:00.703 INFO:tasks.workunit.client.0.smithi195.stdout:removed './201/37' 2022-06-27T11:52:00.703 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './201' 2022-06-27T11:52:00.703 INFO:tasks.workunit.client.0.smithi195.stdout:removed './202/21' 2022-06-27T11:52:00.703 INFO:tasks.workunit.client.0.smithi195.stdout:removed './202/27' 2022-06-27T11:52:00.703 INFO:tasks.workunit.client.0.smithi195.stdout:removed './202/31' 2022-06-27T11:52:00.704 INFO:tasks.workunit.client.0.smithi195.stdout:removed './202/36' 2022-06-27T11:52:00.704 INFO:tasks.workunit.client.0.smithi195.stdout:removed './202/20' 2022-06-27T11:52:00.704 INFO:tasks.workunit.client.0.smithi195.stdout:removed './202/39' 2022-06-27T11:52:00.704 INFO:tasks.workunit.client.0.smithi195.stdout:removed './202/34' 2022-06-27T11:52:00.705 INFO:tasks.workunit.client.0.smithi195.stdout:removed './202/38' 2022-06-27T11:52:00.705 INFO:tasks.workunit.client.0.smithi195.stdout:removed './202/28' 2022-06-27T11:52:00.705 INFO:tasks.workunit.client.0.smithi195.stdout:removed './202/24' 2022-06-27T11:52:00.706 INFO:tasks.workunit.client.0.smithi195.stdout:removed './202/29' 2022-06-27T11:52:00.707 INFO:tasks.workunit.client.0.smithi195.stdout:removed './202/32' 2022-06-27T11:52:00.707 INFO:tasks.workunit.client.0.smithi195.stdout:removed './202/22' 2022-06-27T11:52:00.707 INFO:tasks.workunit.client.0.smithi195.stdout:removed './202/25' 2022-06-27T11:52:00.708 INFO:tasks.workunit.client.0.smithi195.stdout:removed './202/30' 2022-06-27T11:52:00.708 INFO:tasks.workunit.client.0.smithi195.stdout:removed './202/33' 2022-06-27T11:52:00.709 INFO:tasks.workunit.client.0.smithi195.stdout:removed './202/23' 2022-06-27T11:52:00.709 INFO:tasks.workunit.client.0.smithi195.stdout:removed './202/35' 2022-06-27T11:52:00.710 INFO:tasks.workunit.client.0.smithi195.stdout:removed './202/26' 2022-06-27T11:52:00.710 INFO:tasks.workunit.client.0.smithi195.stdout:removed './202/37' 2022-06-27T11:52:00.710 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './202' 2022-06-27T11:52:00.711 INFO:tasks.workunit.client.0.smithi195.stdout:removed './203/21' 2022-06-27T11:52:00.711 INFO:tasks.workunit.client.0.smithi195.stdout:removed './203/27' 2022-06-27T11:52:00.711 INFO:tasks.workunit.client.0.smithi195.stdout:removed './203/31' 2022-06-27T11:52:00.712 INFO:tasks.workunit.client.0.smithi195.stdout:removed './203/36' 2022-06-27T11:52:00.712 INFO:tasks.workunit.client.0.smithi195.stdout:removed './203/20' 2022-06-27T11:52:00.713 INFO:tasks.workunit.client.0.smithi195.stdout:removed './203/39' 2022-06-27T11:52:00.713 INFO:tasks.workunit.client.0.smithi195.stdout:removed './203/34' 2022-06-27T11:52:00.713 INFO:tasks.workunit.client.0.smithi195.stdout:removed './203/38' 2022-06-27T11:52:00.714 INFO:tasks.workunit.client.0.smithi195.stdout:removed './203/28' 2022-06-27T11:52:00.714 INFO:tasks.workunit.client.0.smithi195.stdout:removed './203/24' 2022-06-27T11:52:00.715 INFO:tasks.workunit.client.0.smithi195.stdout:removed './203/29' 2022-06-27T11:52:00.715 INFO:tasks.workunit.client.0.smithi195.stdout:removed './203/32' 2022-06-27T11:52:00.716 INFO:tasks.workunit.client.0.smithi195.stdout:removed './203/22' 2022-06-27T11:52:00.716 INFO:tasks.workunit.client.0.smithi195.stdout:removed './203/25' 2022-06-27T11:52:00.717 INFO:tasks.workunit.client.0.smithi195.stdout:removed './203/30' 2022-06-27T11:52:00.717 INFO:tasks.workunit.client.0.smithi195.stdout:removed './203/33' 2022-06-27T11:52:00.717 INFO:tasks.workunit.client.0.smithi195.stdout:removed './203/23' 2022-06-27T11:52:00.718 INFO:tasks.workunit.client.0.smithi195.stdout:removed './203/35' 2022-06-27T11:52:00.718 INFO:tasks.workunit.client.0.smithi195.stdout:removed './203/26' 2022-06-27T11:52:00.719 INFO:tasks.workunit.client.0.smithi195.stdout:removed './203/37' 2022-06-27T11:52:00.719 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './203' 2022-06-27T11:52:00.719 INFO:tasks.workunit.client.0.smithi195.stdout:removed './204/21' 2022-06-27T11:52:00.720 INFO:tasks.workunit.client.0.smithi195.stdout:removed './204/27' 2022-06-27T11:52:00.720 INFO:tasks.workunit.client.0.smithi195.stdout:removed './204/31' 2022-06-27T11:52:00.720 INFO:tasks.workunit.client.0.smithi195.stdout:removed './204/36' 2022-06-27T11:52:00.721 INFO:tasks.workunit.client.0.smithi195.stdout:removed './204/20' 2022-06-27T11:52:00.721 INFO:tasks.workunit.client.0.smithi195.stdout:removed './204/39' 2022-06-27T11:52:00.721 INFO:tasks.workunit.client.0.smithi195.stdout:removed './204/34' 2022-06-27T11:52:00.722 INFO:tasks.workunit.client.0.smithi195.stdout:removed './204/38' 2022-06-27T11:52:00.722 INFO:tasks.workunit.client.0.smithi195.stdout:removed './204/28' 2022-06-27T11:52:00.722 INFO:tasks.workunit.client.0.smithi195.stdout:removed './204/24' 2022-06-27T11:52:00.723 INFO:tasks.workunit.client.0.smithi195.stdout:removed './204/29' 2022-06-27T11:52:00.723 INFO:tasks.workunit.client.0.smithi195.stdout:removed './204/32' 2022-06-27T11:52:00.723 INFO:tasks.workunit.client.0.smithi195.stdout:removed './204/22' 2022-06-27T11:52:00.724 INFO:tasks.workunit.client.0.smithi195.stdout:removed './204/25' 2022-06-27T11:52:00.727 INFO:tasks.workunit.client.0.smithi195.stdout:removed './204/30' 2022-06-27T11:52:00.728 INFO:tasks.workunit.client.0.smithi195.stdout:removed './204/33' 2022-06-27T11:52:00.728 INFO:tasks.workunit.client.0.smithi195.stdout:removed './204/23' 2022-06-27T11:52:00.728 INFO:tasks.workunit.client.0.smithi195.stdout:removed './204/35' 2022-06-27T11:52:00.729 INFO:tasks.workunit.client.0.smithi195.stdout:removed './204/26' 2022-06-27T11:52:00.729 INFO:tasks.workunit.client.0.smithi195.stdout:removed './204/37' 2022-06-27T11:52:00.729 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './204' 2022-06-27T11:52:00.730 INFO:tasks.workunit.client.0.smithi195.stdout:removed './205/21' 2022-06-27T11:52:00.731 INFO:tasks.workunit.client.0.smithi195.stdout:removed './205/27' 2022-06-27T11:52:00.731 INFO:tasks.workunit.client.0.smithi195.stdout:removed './205/31' 2022-06-27T11:52:00.731 INFO:tasks.workunit.client.0.smithi195.stdout:removed './205/36' 2022-06-27T11:52:00.732 INFO:tasks.workunit.client.0.smithi195.stdout:removed './205/20' 2022-06-27T11:52:00.732 INFO:tasks.workunit.client.0.smithi195.stdout:removed './205/39' 2022-06-27T11:52:00.732 INFO:tasks.workunit.client.0.smithi195.stdout:removed './205/34' 2022-06-27T11:52:00.732 INFO:tasks.workunit.client.0.smithi195.stdout:removed './205/38' 2022-06-27T11:52:00.733 INFO:tasks.workunit.client.0.smithi195.stdout:removed './205/28' 2022-06-27T11:52:00.733 INFO:tasks.workunit.client.0.smithi195.stdout:removed './205/24' 2022-06-27T11:52:00.733 INFO:tasks.workunit.client.0.smithi195.stdout:removed './205/29' 2022-06-27T11:52:00.733 INFO:tasks.workunit.client.0.smithi195.stdout:removed './205/32' 2022-06-27T11:52:00.734 INFO:tasks.workunit.client.0.smithi195.stdout:removed './205/22' 2022-06-27T11:52:00.734 INFO:tasks.workunit.client.0.smithi195.stdout:removed './205/25' 2022-06-27T11:52:00.734 INFO:tasks.workunit.client.0.smithi195.stdout:removed './205/30' 2022-06-27T11:52:00.735 INFO:tasks.workunit.client.0.smithi195.stdout:removed './205/33' 2022-06-27T11:52:00.736 INFO:tasks.workunit.client.0.smithi195.stdout:removed './205/23' 2022-06-27T11:52:00.736 INFO:tasks.workunit.client.0.smithi195.stdout:removed './205/35' 2022-06-27T11:52:00.736 INFO:tasks.workunit.client.0.smithi195.stdout:removed './205/26' 2022-06-27T11:52:00.736 INFO:tasks.workunit.client.0.smithi195.stdout:removed './205/37' 2022-06-27T11:52:00.737 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './205' 2022-06-27T11:52:00.737 INFO:tasks.workunit.client.0.smithi195.stdout:removed './206/21' 2022-06-27T11:52:00.737 INFO:tasks.workunit.client.0.smithi195.stdout:removed './206/27' 2022-06-27T11:52:00.738 INFO:tasks.workunit.client.0.smithi195.stdout:removed './206/31' 2022-06-27T11:52:00.738 INFO:tasks.workunit.client.0.smithi195.stdout:removed './206/36' 2022-06-27T11:52:00.738 INFO:tasks.workunit.client.0.smithi195.stdout:removed './206/20' 2022-06-27T11:52:00.738 INFO:tasks.workunit.client.0.smithi195.stdout:removed './206/39' 2022-06-27T11:52:03.601 INFO:tasks.workunit.client.0.smithi195.stdout:removed './206/34' 2022-06-27T11:52:03.602 INFO:tasks.workunit.client.0.smithi195.stdout:removed './206/38' 2022-06-27T11:52:03.602 INFO:tasks.workunit.client.0.smithi195.stdout:removed './206/28' 2022-06-27T11:52:03.602 INFO:tasks.workunit.client.0.smithi195.stdout:removed './206/24' 2022-06-27T11:52:03.603 INFO:tasks.workunit.client.0.smithi195.stdout:removed './206/29' 2022-06-27T11:52:03.603 INFO:tasks.workunit.client.0.smithi195.stdout:removed './206/32' 2022-06-27T11:52:03.603 INFO:tasks.workunit.client.0.smithi195.stdout:removed './206/22' 2022-06-27T11:52:03.603 INFO:tasks.workunit.client.0.smithi195.stdout:removed './206/25' 2022-06-27T11:52:03.604 INFO:tasks.workunit.client.0.smithi195.stdout:removed './206/30' 2022-06-27T11:52:03.604 INFO:tasks.workunit.client.0.smithi195.stdout:removed './206/33' 2022-06-27T11:52:03.604 INFO:tasks.workunit.client.0.smithi195.stdout:removed './206/23' 2022-06-27T11:52:03.604 INFO:tasks.workunit.client.0.smithi195.stdout:removed './206/35' 2022-06-27T11:52:03.605 INFO:tasks.workunit.client.0.smithi195.stdout:removed './206/26' 2022-06-27T11:52:03.605 INFO:tasks.workunit.client.0.smithi195.stdout:removed './206/37' 2022-06-27T11:52:03.605 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './206' 2022-06-27T11:52:03.605 INFO:tasks.workunit.client.0.smithi195.stdout:removed './207/21' 2022-06-27T11:52:03.605 INFO:tasks.workunit.client.0.smithi195.stdout:removed './207/27' 2022-06-27T11:52:03.606 INFO:tasks.workunit.client.0.smithi195.stdout:removed './207/31' 2022-06-27T11:52:03.606 INFO:tasks.workunit.client.0.smithi195.stdout:removed './207/36' 2022-06-27T11:52:03.606 INFO:tasks.workunit.client.0.smithi195.stdout:removed './207/20' 2022-06-27T11:52:03.607 INFO:tasks.workunit.client.0.smithi195.stdout:removed './207/39' 2022-06-27T11:52:03.607 INFO:tasks.workunit.client.0.smithi195.stdout:removed './207/34' 2022-06-27T11:52:03.608 INFO:tasks.workunit.client.0.smithi195.stdout:removed './207/38' 2022-06-27T11:52:03.608 INFO:tasks.workunit.client.0.smithi195.stdout:removed './207/28' 2022-06-27T11:52:03.608 INFO:tasks.workunit.client.0.smithi195.stdout:removed './207/24' 2022-06-27T11:52:03.608 INFO:tasks.workunit.client.0.smithi195.stdout:removed './207/29' 2022-06-27T11:52:03.609 INFO:tasks.workunit.client.0.smithi195.stdout:removed './207/32' 2022-06-27T11:52:03.609 INFO:tasks.workunit.client.0.smithi195.stdout:removed './207/22' 2022-06-27T11:52:03.609 INFO:tasks.workunit.client.0.smithi195.stdout:removed './207/25' 2022-06-27T11:52:03.609 INFO:tasks.workunit.client.0.smithi195.stdout:removed './207/30' 2022-06-27T11:52:03.610 INFO:tasks.workunit.client.0.smithi195.stdout:removed './207/33' 2022-06-27T11:52:03.610 INFO:tasks.workunit.client.0.smithi195.stdout:removed './207/23' 2022-06-27T11:52:03.610 INFO:tasks.workunit.client.0.smithi195.stdout:removed './207/35' 2022-06-27T11:52:03.610 INFO:tasks.workunit.client.0.smithi195.stdout:removed './207/26' 2022-06-27T11:52:03.610 INFO:tasks.workunit.client.0.smithi195.stdout:removed './207/37' 2022-06-27T11:52:03.611 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './207' 2022-06-27T11:52:03.611 INFO:tasks.workunit.client.0.smithi195.stdout:removed './208/21' 2022-06-27T11:52:03.611 INFO:tasks.workunit.client.0.smithi195.stdout:removed './208/27' 2022-06-27T11:52:03.611 INFO:tasks.workunit.client.0.smithi195.stdout:removed './208/31' 2022-06-27T11:52:03.611 INFO:tasks.workunit.client.0.smithi195.stdout:removed './208/36' 2022-06-27T11:52:03.612 INFO:tasks.workunit.client.0.smithi195.stdout:removed './208/20' 2022-06-27T11:52:03.612 INFO:tasks.workunit.client.0.smithi195.stdout:removed './208/39' 2022-06-27T11:52:03.612 INFO:tasks.workunit.client.0.smithi195.stdout:removed './208/34' 2022-06-27T11:52:03.612 INFO:tasks.workunit.client.0.smithi195.stdout:removed './208/38' 2022-06-27T11:52:03.613 INFO:tasks.workunit.client.0.smithi195.stdout:removed './208/28' 2022-06-27T11:52:03.613 INFO:tasks.workunit.client.0.smithi195.stdout:removed './208/24' 2022-06-27T11:52:03.613 INFO:tasks.workunit.client.0.smithi195.stdout:removed './208/29' 2022-06-27T11:52:03.613 INFO:tasks.workunit.client.0.smithi195.stdout:removed './208/32' 2022-06-27T11:52:03.613 INFO:tasks.workunit.client.0.smithi195.stdout:removed './208/22' 2022-06-27T11:52:03.614 INFO:tasks.workunit.client.0.smithi195.stdout:removed './208/25' 2022-06-27T11:52:03.614 INFO:tasks.workunit.client.0.smithi195.stdout:removed './208/30' 2022-06-27T11:52:03.614 INFO:tasks.workunit.client.0.smithi195.stdout:removed './208/33' 2022-06-27T11:52:03.614 INFO:tasks.workunit.client.0.smithi195.stdout:removed './208/23' 2022-06-27T11:52:03.615 INFO:tasks.workunit.client.0.smithi195.stdout:removed './208/35' 2022-06-27T11:52:03.615 INFO:tasks.workunit.client.0.smithi195.stdout:removed './208/26' 2022-06-27T11:52:03.615 INFO:tasks.workunit.client.0.smithi195.stdout:removed './208/37' 2022-06-27T11:52:03.615 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './208' 2022-06-27T11:52:03.615 INFO:tasks.workunit.client.0.smithi195.stdout:removed './209/21' 2022-06-27T11:52:03.616 INFO:tasks.workunit.client.0.smithi195.stdout:removed './209/27' 2022-06-27T11:52:03.616 INFO:tasks.workunit.client.0.smithi195.stdout:removed './209/31' 2022-06-27T11:52:03.616 INFO:tasks.workunit.client.0.smithi195.stdout:removed './209/36' 2022-06-27T11:52:03.616 INFO:tasks.workunit.client.0.smithi195.stdout:removed './209/20' 2022-06-27T11:52:03.616 INFO:tasks.workunit.client.0.smithi195.stdout:removed './209/39' 2022-06-27T11:52:03.617 INFO:tasks.workunit.client.0.smithi195.stdout:removed './209/34' 2022-06-27T11:52:03.617 INFO:tasks.workunit.client.0.smithi195.stdout:removed './209/38' 2022-06-27T11:52:03.618 INFO:tasks.workunit.client.0.smithi195.stdout:removed './209/28' 2022-06-27T11:52:03.618 INFO:tasks.workunit.client.0.smithi195.stdout:removed './209/24' 2022-06-27T11:52:03.618 INFO:tasks.workunit.client.0.smithi195.stdout:removed './209/29' 2022-06-27T11:52:03.619 INFO:tasks.workunit.client.0.smithi195.stdout:removed './209/32' 2022-06-27T11:52:03.619 INFO:tasks.workunit.client.0.smithi195.stdout:removed './209/22' 2022-06-27T11:52:03.619 INFO:tasks.workunit.client.0.smithi195.stdout:removed './209/25' 2022-06-27T11:52:03.619 INFO:tasks.workunit.client.0.smithi195.stdout:removed './209/30' 2022-06-27T11:52:03.620 INFO:tasks.workunit.client.0.smithi195.stdout:removed './209/33' 2022-06-27T11:52:03.620 INFO:tasks.workunit.client.0.smithi195.stdout:removed './209/23' 2022-06-27T11:52:03.620 INFO:tasks.workunit.client.0.smithi195.stdout:removed './209/35' 2022-06-27T11:52:03.620 INFO:tasks.workunit.client.0.smithi195.stdout:removed './209/26' 2022-06-27T11:52:03.620 INFO:tasks.workunit.client.0.smithi195.stdout:removed './209/37' 2022-06-27T11:52:03.621 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './209' 2022-06-27T11:52:03.621 INFO:tasks.workunit.client.0.smithi195.stdout:removed './210/21' 2022-06-27T11:52:03.621 INFO:tasks.workunit.client.0.smithi195.stdout:removed './210/27' 2022-06-27T11:52:03.621 INFO:tasks.workunit.client.0.smithi195.stdout:removed './210/31' 2022-06-27T11:52:03.622 INFO:tasks.workunit.client.0.smithi195.stdout:removed './210/36' 2022-06-27T11:52:03.622 INFO:tasks.workunit.client.0.smithi195.stdout:removed './210/20' 2022-06-27T11:52:03.622 INFO:tasks.workunit.client.0.smithi195.stdout:removed './210/39' 2022-06-27T11:52:03.622 INFO:tasks.workunit.client.0.smithi195.stdout:removed './210/34' 2022-06-27T11:52:03.623 INFO:tasks.workunit.client.0.smithi195.stdout:removed './210/38' 2022-06-27T11:52:03.623 INFO:tasks.workunit.client.0.smithi195.stdout:removed './210/28' 2022-06-27T11:52:03.623 INFO:tasks.workunit.client.0.smithi195.stdout:removed './210/24' 2022-06-27T11:52:03.623 INFO:tasks.workunit.client.0.smithi195.stdout:removed './210/29' 2022-06-27T11:52:03.624 INFO:tasks.workunit.client.0.smithi195.stdout:removed './210/32' 2022-06-27T11:52:03.624 INFO:tasks.workunit.client.0.smithi195.stdout:removed './210/22' 2022-06-27T11:52:03.624 INFO:tasks.workunit.client.0.smithi195.stdout:removed './210/25' 2022-06-27T11:52:03.624 INFO:tasks.workunit.client.0.smithi195.stdout:removed './210/30' 2022-06-27T11:52:03.625 INFO:tasks.workunit.client.0.smithi195.stdout:removed './210/33' 2022-06-27T11:52:03.625 INFO:tasks.workunit.client.0.smithi195.stdout:removed './210/23' 2022-06-27T11:52:03.625 INFO:tasks.workunit.client.0.smithi195.stdout:removed './210/35' 2022-06-27T11:52:03.625 INFO:tasks.workunit.client.0.smithi195.stdout:removed './210/26' 2022-06-27T11:52:03.625 INFO:tasks.workunit.client.0.smithi195.stdout:removed './210/37' 2022-06-27T11:52:03.626 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './210' 2022-06-27T11:52:03.626 INFO:tasks.workunit.client.0.smithi195.stdout:removed './211/21' 2022-06-27T11:52:03.626 INFO:tasks.workunit.client.0.smithi195.stdout:removed './211/27' 2022-06-27T11:52:03.626 INFO:tasks.workunit.client.0.smithi195.stdout:removed './211/31' 2022-06-27T11:52:03.626 INFO:tasks.workunit.client.0.smithi195.stdout:removed './211/36' 2022-06-27T11:52:03.627 INFO:tasks.workunit.client.0.smithi195.stdout:removed './211/20' 2022-06-27T11:52:03.627 INFO:tasks.workunit.client.0.smithi195.stdout:removed './211/39' 2022-06-27T11:52:03.627 INFO:tasks.workunit.client.0.smithi195.stdout:removed './211/34' 2022-06-27T11:52:03.627 INFO:tasks.workunit.client.0.smithi195.stdout:removed './211/38' 2022-06-27T11:52:03.628 INFO:tasks.workunit.client.0.smithi195.stdout:removed './211/28' 2022-06-27T11:52:03.628 INFO:tasks.workunit.client.0.smithi195.stdout:removed './211/24' 2022-06-27T11:52:03.629 INFO:tasks.workunit.client.0.smithi195.stdout:removed './211/29' 2022-06-27T11:52:03.629 INFO:tasks.workunit.client.0.smithi195.stdout:removed './211/32' 2022-06-27T11:52:03.629 INFO:tasks.workunit.client.0.smithi195.stdout:removed './211/22' 2022-06-27T11:52:03.629 INFO:tasks.workunit.client.0.smithi195.stdout:removed './211/25' 2022-06-27T11:52:03.629 INFO:tasks.workunit.client.0.smithi195.stdout:removed './211/30' 2022-06-27T11:52:03.630 INFO:tasks.workunit.client.0.smithi195.stdout:removed './211/33' 2022-06-27T11:52:03.630 INFO:tasks.workunit.client.0.smithi195.stdout:removed './211/23' 2022-06-27T11:52:03.630 INFO:tasks.workunit.client.0.smithi195.stdout:removed './211/35' 2022-06-27T11:52:03.631 INFO:tasks.workunit.client.0.smithi195.stdout:removed './211/26' 2022-06-27T11:52:03.631 INFO:tasks.workunit.client.0.smithi195.stdout:removed './211/37' 2022-06-27T11:52:03.631 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './211' 2022-06-27T11:52:03.631 INFO:tasks.workunit.client.0.smithi195.stdout:removed './212/21' 2022-06-27T11:52:03.632 INFO:tasks.workunit.client.0.smithi195.stdout:removed './212/27' 2022-06-27T11:52:03.632 INFO:tasks.workunit.client.0.smithi195.stdout:removed './212/31' 2022-06-27T11:52:03.632 INFO:tasks.workunit.client.0.smithi195.stdout:removed './212/36' 2022-06-27T11:52:03.633 INFO:tasks.workunit.client.0.smithi195.stdout:removed './212/20' 2022-06-27T11:52:03.633 INFO:tasks.workunit.client.0.smithi195.stdout:removed './212/39' 2022-06-27T11:52:03.633 INFO:tasks.workunit.client.0.smithi195.stdout:removed './212/34' 2022-06-27T11:52:03.633 INFO:tasks.workunit.client.0.smithi195.stdout:removed './212/38' 2022-06-27T11:52:03.634 INFO:tasks.workunit.client.0.smithi195.stdout:removed './212/28' 2022-06-27T11:52:03.634 INFO:tasks.workunit.client.0.smithi195.stdout:removed './212/24' 2022-06-27T11:52:03.634 INFO:tasks.workunit.client.0.smithi195.stdout:removed './212/29' 2022-06-27T11:52:03.635 INFO:tasks.workunit.client.0.smithi195.stdout:removed './212/32' 2022-06-27T11:52:03.635 INFO:tasks.workunit.client.0.smithi195.stdout:removed './212/22' 2022-06-27T11:52:03.635 INFO:tasks.workunit.client.0.smithi195.stdout:removed './212/25' 2022-06-27T11:52:03.635 INFO:tasks.workunit.client.0.smithi195.stdout:removed './212/30' 2022-06-27T11:52:03.635 INFO:tasks.workunit.client.0.smithi195.stdout:removed './212/33' 2022-06-27T11:52:03.636 INFO:tasks.workunit.client.0.smithi195.stdout:removed './212/23' 2022-06-27T11:52:03.636 INFO:tasks.workunit.client.0.smithi195.stdout:removed './212/35' 2022-06-27T11:52:03.636 INFO:tasks.workunit.client.0.smithi195.stdout:removed './212/26' 2022-06-27T11:52:03.636 INFO:tasks.workunit.client.0.smithi195.stdout:removed './212/37' 2022-06-27T11:52:03.636 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './212' 2022-06-27T11:52:03.637 INFO:tasks.workunit.client.0.smithi195.stdout:removed './213/21' 2022-06-27T11:52:03.637 INFO:tasks.workunit.client.0.smithi195.stdout:removed './213/27' 2022-06-27T11:52:03.637 INFO:tasks.workunit.client.0.smithi195.stdout:removed './213/31' 2022-06-27T11:52:03.637 INFO:tasks.workunit.client.0.smithi195.stdout:removed './213/36' 2022-06-27T11:52:03.638 INFO:tasks.workunit.client.0.smithi195.stdout:removed './213/20' 2022-06-27T11:52:03.638 INFO:tasks.workunit.client.0.smithi195.stdout:removed './213/39' 2022-06-27T11:52:03.638 INFO:tasks.workunit.client.0.smithi195.stdout:removed './213/34' 2022-06-27T11:52:03.639 INFO:tasks.workunit.client.0.smithi195.stdout:removed './213/38' 2022-06-27T11:52:03.639 INFO:tasks.workunit.client.0.smithi195.stdout:removed './213/28' 2022-06-27T11:52:03.639 INFO:tasks.workunit.client.0.smithi195.stdout:removed './213/24' 2022-06-27T11:52:03.640 INFO:tasks.workunit.client.0.smithi195.stdout:removed './213/29' 2022-06-27T11:52:03.640 INFO:tasks.workunit.client.0.smithi195.stdout:removed './213/32' 2022-06-27T11:52:03.640 INFO:tasks.workunit.client.0.smithi195.stdout:removed './213/22' 2022-06-27T11:52:03.641 INFO:tasks.workunit.client.0.smithi195.stdout:removed './213/25' 2022-06-27T11:52:03.641 INFO:tasks.workunit.client.0.smithi195.stdout:removed './213/30' 2022-06-27T11:52:03.641 INFO:tasks.workunit.client.0.smithi195.stdout:removed './213/33' 2022-06-27T11:52:03.641 INFO:tasks.workunit.client.0.smithi195.stdout:removed './213/23' 2022-06-27T11:52:03.642 INFO:tasks.workunit.client.0.smithi195.stdout:removed './213/35' 2022-06-27T11:52:03.642 INFO:tasks.workunit.client.0.smithi195.stdout:removed './213/26' 2022-06-27T11:52:03.642 INFO:tasks.workunit.client.0.smithi195.stdout:removed './213/37' 2022-06-27T11:52:03.643 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './213' 2022-06-27T11:52:03.643 INFO:tasks.workunit.client.0.smithi195.stdout:removed './214/21' 2022-06-27T11:52:03.643 INFO:tasks.workunit.client.0.smithi195.stdout:removed './214/27' 2022-06-27T11:52:03.644 INFO:tasks.workunit.client.0.smithi195.stdout:removed './214/31' 2022-06-27T11:52:03.644 INFO:tasks.workunit.client.0.smithi195.stdout:removed './214/36' 2022-06-27T11:52:03.644 INFO:tasks.workunit.client.0.smithi195.stdout:removed './214/20' 2022-06-27T11:52:03.645 INFO:tasks.workunit.client.0.smithi195.stdout:removed './214/39' 2022-06-27T11:52:03.645 INFO:tasks.workunit.client.0.smithi195.stdout:removed './214/34' 2022-06-27T11:52:03.645 INFO:tasks.workunit.client.0.smithi195.stdout:removed './214/38' 2022-06-27T11:52:03.645 INFO:tasks.workunit.client.0.smithi195.stdout:removed './214/28' 2022-06-27T11:52:03.645 INFO:tasks.workunit.client.0.smithi195.stdout:removed './214/24' 2022-06-27T11:52:03.646 INFO:tasks.workunit.client.0.smithi195.stdout:removed './214/29' 2022-06-27T11:52:03.646 INFO:tasks.workunit.client.0.smithi195.stdout:removed './214/32' 2022-06-27T11:52:03.646 INFO:tasks.workunit.client.0.smithi195.stdout:removed './214/22' 2022-06-27T11:52:03.646 INFO:tasks.workunit.client.0.smithi195.stdout:removed './214/25' 2022-06-27T11:52:03.647 INFO:tasks.workunit.client.0.smithi195.stdout:removed './214/30' 2022-06-27T11:52:03.647 INFO:tasks.workunit.client.0.smithi195.stdout:removed './214/33' 2022-06-27T11:52:03.647 INFO:tasks.workunit.client.0.smithi195.stdout:removed './214/23' 2022-06-27T11:52:03.647 INFO:tasks.workunit.client.0.smithi195.stdout:removed './214/35' 2022-06-27T11:52:03.647 INFO:tasks.workunit.client.0.smithi195.stdout:removed './214/26' 2022-06-27T11:52:03.648 INFO:tasks.workunit.client.0.smithi195.stdout:removed './214/37' 2022-06-27T11:52:03.648 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './214' 2022-06-27T11:52:03.649 INFO:tasks.workunit.client.0.smithi195.stdout:removed './215/21' 2022-06-27T11:52:03.651 INFO:tasks.workunit.client.0.smithi195.stdout:removed './215/27' 2022-06-27T11:52:03.652 INFO:tasks.workunit.client.0.smithi195.stdout:removed './215/31' 2022-06-27T11:52:03.652 INFO:tasks.workunit.client.0.smithi195.stdout:removed './215/36' 2022-06-27T11:52:03.652 INFO:tasks.workunit.client.0.smithi195.stdout:removed './215/20' 2022-06-27T11:52:03.652 INFO:tasks.workunit.client.0.smithi195.stdout:removed './215/39' 2022-06-27T11:52:03.653 INFO:tasks.workunit.client.0.smithi195.stdout:removed './215/34' 2022-06-27T11:52:03.653 INFO:tasks.workunit.client.0.smithi195.stdout:removed './215/38' 2022-06-27T11:52:03.653 INFO:tasks.workunit.client.0.smithi195.stdout:removed './215/28' 2022-06-27T11:52:03.653 INFO:tasks.workunit.client.0.smithi195.stdout:removed './215/24' 2022-06-27T11:52:03.654 INFO:tasks.workunit.client.0.smithi195.stdout:removed './215/29' 2022-06-27T11:52:03.654 INFO:tasks.workunit.client.0.smithi195.stdout:removed './215/32' 2022-06-27T11:52:03.655 INFO:tasks.workunit.client.0.smithi195.stdout:removed './215/22' 2022-06-27T11:52:03.655 INFO:tasks.workunit.client.0.smithi195.stdout:removed './215/25' 2022-06-27T11:52:03.655 INFO:tasks.workunit.client.0.smithi195.stdout:removed './215/30' 2022-06-27T11:52:03.655 INFO:tasks.workunit.client.0.smithi195.stdout:removed './215/33' 2022-06-27T11:52:03.655 INFO:tasks.workunit.client.0.smithi195.stdout:removed './215/23' 2022-06-27T11:52:03.656 INFO:tasks.workunit.client.0.smithi195.stdout:removed './215/35' 2022-06-27T11:52:03.656 INFO:tasks.workunit.client.0.smithi195.stdout:removed './215/26' 2022-06-27T11:52:03.656 INFO:tasks.workunit.client.0.smithi195.stdout:removed './215/37' 2022-06-27T11:52:03.657 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './215' 2022-06-27T11:52:03.657 INFO:tasks.workunit.client.0.smithi195.stdout:removed './216/21' 2022-06-27T11:52:03.657 INFO:tasks.workunit.client.0.smithi195.stdout:removed './216/27' 2022-06-27T11:52:03.657 INFO:tasks.workunit.client.0.smithi195.stdout:removed './216/31' 2022-06-27T11:52:03.658 INFO:tasks.workunit.client.0.smithi195.stdout:removed './216/36' 2022-06-27T11:52:03.658 INFO:tasks.workunit.client.0.smithi195.stdout:removed './216/20' 2022-06-27T11:52:03.658 INFO:tasks.workunit.client.0.smithi195.stdout:removed './216/39' 2022-06-27T11:52:03.658 INFO:tasks.workunit.client.0.smithi195.stdout:removed './216/34' 2022-06-27T11:52:03.658 INFO:tasks.workunit.client.0.smithi195.stdout:removed './216/38' 2022-06-27T11:52:06.308 INFO:tasks.workunit.client.0.smithi195.stdout:removed './216/28' 2022-06-27T11:52:06.309 INFO:tasks.workunit.client.0.smithi195.stdout:removed './216/24' 2022-06-27T11:52:06.309 INFO:tasks.workunit.client.0.smithi195.stdout:removed './216/29' 2022-06-27T11:52:06.310 INFO:tasks.workunit.client.0.smithi195.stdout:removed './216/32' 2022-06-27T11:52:06.310 INFO:tasks.workunit.client.0.smithi195.stdout:removed './216/22' 2022-06-27T11:52:06.311 INFO:tasks.workunit.client.0.smithi195.stdout:removed './216/25' 2022-06-27T11:52:06.311 INFO:tasks.workunit.client.0.smithi195.stdout:removed './216/30' 2022-06-27T11:52:06.311 INFO:tasks.workunit.client.0.smithi195.stdout:removed './216/33' 2022-06-27T11:52:06.312 INFO:tasks.workunit.client.0.smithi195.stdout:removed './216/23' 2022-06-27T11:52:06.312 INFO:tasks.workunit.client.0.smithi195.stdout:removed './216/35' 2022-06-27T11:52:06.313 INFO:tasks.workunit.client.0.smithi195.stdout:removed './216/26' 2022-06-27T11:52:06.313 INFO:tasks.workunit.client.0.smithi195.stdout:removed './216/37' 2022-06-27T11:52:06.313 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './216' 2022-06-27T11:52:06.314 INFO:tasks.workunit.client.0.smithi195.stdout:removed './217/21' 2022-06-27T11:52:06.314 INFO:tasks.workunit.client.0.smithi195.stdout:removed './217/27' 2022-06-27T11:52:06.315 INFO:tasks.workunit.client.0.smithi195.stdout:removed './217/31' 2022-06-27T11:52:06.315 INFO:tasks.workunit.client.0.smithi195.stdout:removed './217/36' 2022-06-27T11:52:06.315 INFO:tasks.workunit.client.0.smithi195.stdout:removed './217/20' 2022-06-27T11:52:06.316 INFO:tasks.workunit.client.0.smithi195.stdout:removed './217/39' 2022-06-27T11:52:06.316 INFO:tasks.workunit.client.0.smithi195.stdout:removed './217/34' 2022-06-27T11:52:06.317 INFO:tasks.workunit.client.0.smithi195.stdout:removed './217/38' 2022-06-27T11:52:06.317 INFO:tasks.workunit.client.0.smithi195.stdout:removed './217/28' 2022-06-27T11:52:06.317 INFO:tasks.workunit.client.0.smithi195.stdout:removed './217/24' 2022-06-27T11:52:06.318 INFO:tasks.workunit.client.0.smithi195.stdout:removed './217/29' 2022-06-27T11:52:06.318 INFO:tasks.workunit.client.0.smithi195.stdout:removed './217/32' 2022-06-27T11:52:06.318 INFO:tasks.workunit.client.0.smithi195.stdout:removed './217/22' 2022-06-27T11:52:06.319 INFO:tasks.workunit.client.0.smithi195.stdout:removed './217/25' 2022-06-27T11:52:06.319 INFO:tasks.workunit.client.0.smithi195.stdout:removed './217/30' 2022-06-27T11:52:06.319 INFO:tasks.workunit.client.0.smithi195.stdout:removed './217/33' 2022-06-27T11:52:06.320 INFO:tasks.workunit.client.0.smithi195.stdout:removed './217/23' 2022-06-27T11:52:06.320 INFO:tasks.workunit.client.0.smithi195.stdout:removed './217/35' 2022-06-27T11:52:06.321 INFO:tasks.workunit.client.0.smithi195.stdout:removed './217/26' 2022-06-27T11:52:06.321 INFO:tasks.workunit.client.0.smithi195.stdout:removed './217/37' 2022-06-27T11:52:06.321 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './217' 2022-06-27T11:52:06.322 INFO:tasks.workunit.client.0.smithi195.stdout:removed './218/21' 2022-06-27T11:52:06.322 INFO:tasks.workunit.client.0.smithi195.stdout:removed './218/27' 2022-06-27T11:52:06.323 INFO:tasks.workunit.client.0.smithi195.stdout:removed './218/31' 2022-06-27T11:52:06.323 INFO:tasks.workunit.client.0.smithi195.stdout:removed './218/36' 2022-06-27T11:52:06.323 INFO:tasks.workunit.client.0.smithi195.stdout:removed './218/20' 2022-06-27T11:52:06.324 INFO:tasks.workunit.client.0.smithi195.stdout:removed './218/39' 2022-06-27T11:52:06.324 INFO:tasks.workunit.client.0.smithi195.stdout:removed './218/34' 2022-06-27T11:52:06.324 INFO:tasks.workunit.client.0.smithi195.stdout:removed './218/38' 2022-06-27T11:52:06.325 INFO:tasks.workunit.client.0.smithi195.stdout:removed './218/28' 2022-06-27T11:52:06.325 INFO:tasks.workunit.client.0.smithi195.stdout:removed './218/24' 2022-06-27T11:52:06.326 INFO:tasks.workunit.client.0.smithi195.stdout:removed './218/29' 2022-06-27T11:52:06.326 INFO:tasks.workunit.client.0.smithi195.stdout:removed './218/32' 2022-06-27T11:52:06.326 INFO:tasks.workunit.client.0.smithi195.stdout:removed './218/22' 2022-06-27T11:52:06.327 INFO:tasks.workunit.client.0.smithi195.stdout:removed './218/25' 2022-06-27T11:52:06.327 INFO:tasks.workunit.client.0.smithi195.stdout:removed './218/30' 2022-06-27T11:52:06.328 INFO:tasks.workunit.client.0.smithi195.stdout:removed './218/33' 2022-06-27T11:52:06.328 INFO:tasks.workunit.client.0.smithi195.stdout:removed './218/23' 2022-06-27T11:52:06.328 INFO:tasks.workunit.client.0.smithi195.stdout:removed './218/35' 2022-06-27T11:52:06.329 INFO:tasks.workunit.client.0.smithi195.stdout:removed './218/26' 2022-06-27T11:52:06.329 INFO:tasks.workunit.client.0.smithi195.stdout:removed './218/37' 2022-06-27T11:52:06.329 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './218' 2022-06-27T11:52:06.330 INFO:tasks.workunit.client.0.smithi195.stdout:removed './219/21' 2022-06-27T11:52:06.330 INFO:tasks.workunit.client.0.smithi195.stdout:removed './219/27' 2022-06-27T11:52:06.331 INFO:tasks.workunit.client.0.smithi195.stdout:removed './219/31' 2022-06-27T11:52:06.331 INFO:tasks.workunit.client.0.smithi195.stdout:removed './219/36' 2022-06-27T11:52:06.331 INFO:tasks.workunit.client.0.smithi195.stdout:removed './219/20' 2022-06-27T11:52:06.332 INFO:tasks.workunit.client.0.smithi195.stdout:removed './219/39' 2022-06-27T11:52:06.332 INFO:tasks.workunit.client.0.smithi195.stdout:removed './219/34' 2022-06-27T11:52:06.332 INFO:tasks.workunit.client.0.smithi195.stdout:removed './219/38' 2022-06-27T11:52:06.333 INFO:tasks.workunit.client.0.smithi195.stdout:removed './219/28' 2022-06-27T11:52:06.333 INFO:tasks.workunit.client.0.smithi195.stdout:removed './219/24' 2022-06-27T11:52:06.334 INFO:tasks.workunit.client.0.smithi195.stdout:removed './219/29' 2022-06-27T11:52:06.334 INFO:tasks.workunit.client.0.smithi195.stdout:removed './219/32' 2022-06-27T11:52:06.334 INFO:tasks.workunit.client.0.smithi195.stdout:removed './219/22' 2022-06-27T11:52:06.335 INFO:tasks.workunit.client.0.smithi195.stdout:removed './219/25' 2022-06-27T11:52:06.335 INFO:tasks.workunit.client.0.smithi195.stdout:removed './219/30' 2022-06-27T11:52:06.335 INFO:tasks.workunit.client.0.smithi195.stdout:removed './219/33' 2022-06-27T11:52:06.336 INFO:tasks.workunit.client.0.smithi195.stdout:removed './219/23' 2022-06-27T11:52:06.336 INFO:tasks.workunit.client.0.smithi195.stdout:removed './219/35' 2022-06-27T11:52:06.336 INFO:tasks.workunit.client.0.smithi195.stdout:removed './219/26' 2022-06-27T11:52:06.337 INFO:tasks.workunit.client.0.smithi195.stdout:removed './219/37' 2022-06-27T11:52:06.337 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './219' 2022-06-27T11:52:06.338 INFO:tasks.workunit.client.0.smithi195.stdout:removed './220/21' 2022-06-27T11:52:06.338 INFO:tasks.workunit.client.0.smithi195.stdout:removed './220/27' 2022-06-27T11:52:06.338 INFO:tasks.workunit.client.0.smithi195.stdout:removed './220/31' 2022-06-27T11:52:06.339 INFO:tasks.workunit.client.0.smithi195.stdout:removed './220/36' 2022-06-27T11:52:06.339 INFO:tasks.workunit.client.0.smithi195.stdout:removed './220/20' 2022-06-27T11:52:06.340 INFO:tasks.workunit.client.0.smithi195.stdout:removed './220/39' 2022-06-27T11:52:06.340 INFO:tasks.workunit.client.0.smithi195.stdout:removed './220/34' 2022-06-27T11:52:06.340 INFO:tasks.workunit.client.0.smithi195.stdout:removed './220/38' 2022-06-27T11:52:06.341 INFO:tasks.workunit.client.0.smithi195.stdout:removed './220/28' 2022-06-27T11:52:06.341 INFO:tasks.workunit.client.0.smithi195.stdout:removed './220/24' 2022-06-27T11:52:06.342 INFO:tasks.workunit.client.0.smithi195.stdout:removed './220/29' 2022-06-27T11:52:06.342 INFO:tasks.workunit.client.0.smithi195.stdout:removed './220/32' 2022-06-27T11:52:06.343 INFO:tasks.workunit.client.0.smithi195.stdout:removed './220/22' 2022-06-27T11:52:06.343 INFO:tasks.workunit.client.0.smithi195.stdout:removed './220/25' 2022-06-27T11:52:06.344 INFO:tasks.workunit.client.0.smithi195.stdout:removed './220/30' 2022-06-27T11:52:06.344 INFO:tasks.workunit.client.0.smithi195.stdout:removed './220/33' 2022-06-27T11:52:06.344 INFO:tasks.workunit.client.0.smithi195.stdout:removed './220/23' 2022-06-27T11:52:06.345 INFO:tasks.workunit.client.0.smithi195.stdout:removed './220/35' 2022-06-27T11:52:06.345 INFO:tasks.workunit.client.0.smithi195.stdout:removed './220/26' 2022-06-27T11:52:06.346 INFO:tasks.workunit.client.0.smithi195.stdout:removed './220/37' 2022-06-27T11:52:06.346 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './220' 2022-06-27T11:52:06.346 INFO:tasks.workunit.client.0.smithi195.stdout:removed './221/21' 2022-06-27T11:52:06.347 INFO:tasks.workunit.client.0.smithi195.stdout:removed './221/27' 2022-06-27T11:52:06.347 INFO:tasks.workunit.client.0.smithi195.stdout:removed './221/31' 2022-06-27T11:52:06.348 INFO:tasks.workunit.client.0.smithi195.stdout:removed './221/36' 2022-06-27T11:52:06.348 INFO:tasks.workunit.client.0.smithi195.stdout:removed './221/20' 2022-06-27T11:52:06.349 INFO:tasks.workunit.client.0.smithi195.stdout:removed './221/39' 2022-06-27T11:52:06.349 INFO:tasks.workunit.client.0.smithi195.stdout:removed './221/34' 2022-06-27T11:52:06.349 INFO:tasks.workunit.client.0.smithi195.stdout:removed './221/38' 2022-06-27T11:52:06.350 INFO:tasks.workunit.client.0.smithi195.stdout:removed './221/28' 2022-06-27T11:52:06.350 INFO:tasks.workunit.client.0.smithi195.stdout:removed './221/24' 2022-06-27T11:52:06.351 INFO:tasks.workunit.client.0.smithi195.stdout:removed './221/29' 2022-06-27T11:52:06.352 INFO:tasks.workunit.client.0.smithi195.stdout:removed './221/32' 2022-06-27T11:52:06.352 INFO:tasks.workunit.client.0.smithi195.stdout:removed './221/22' 2022-06-27T11:52:06.353 INFO:tasks.workunit.client.0.smithi195.stdout:removed './221/25' 2022-06-27T11:52:06.353 INFO:tasks.workunit.client.0.smithi195.stdout:removed './221/30' 2022-06-27T11:52:06.353 INFO:tasks.workunit.client.0.smithi195.stdout:removed './221/33' 2022-06-27T11:52:06.354 INFO:tasks.workunit.client.0.smithi195.stdout:removed './221/23' 2022-06-27T11:52:06.354 INFO:tasks.workunit.client.0.smithi195.stdout:removed './221/35' 2022-06-27T11:52:06.355 INFO:tasks.workunit.client.0.smithi195.stdout:removed './221/26' 2022-06-27T11:52:06.355 INFO:tasks.workunit.client.0.smithi195.stdout:removed './221/37' 2022-06-27T11:52:06.356 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './221' 2022-06-27T11:52:06.356 INFO:tasks.workunit.client.0.smithi195.stdout:removed './222/21' 2022-06-27T11:52:06.356 INFO:tasks.workunit.client.0.smithi195.stdout:removed './222/27' 2022-06-27T11:52:06.357 INFO:tasks.workunit.client.0.smithi195.stdout:removed './222/31' 2022-06-27T11:52:06.357 INFO:tasks.workunit.client.0.smithi195.stdout:removed './222/36' 2022-06-27T11:52:06.358 INFO:tasks.workunit.client.0.smithi195.stdout:removed './222/20' 2022-06-27T11:52:06.358 INFO:tasks.workunit.client.0.smithi195.stdout:removed './222/39' 2022-06-27T11:52:06.358 INFO:tasks.workunit.client.0.smithi195.stdout:removed './222/34' 2022-06-27T11:52:06.359 INFO:tasks.workunit.client.0.smithi195.stdout:removed './222/38' 2022-06-27T11:52:06.359 INFO:tasks.workunit.client.0.smithi195.stdout:removed './222/28' 2022-06-27T11:52:06.360 INFO:tasks.workunit.client.0.smithi195.stdout:removed './222/24' 2022-06-27T11:52:06.360 INFO:tasks.workunit.client.0.smithi195.stdout:removed './222/29' 2022-06-27T11:52:06.360 INFO:tasks.workunit.client.0.smithi195.stdout:removed './222/32' 2022-06-27T11:52:06.361 INFO:tasks.workunit.client.0.smithi195.stdout:removed './222/22' 2022-06-27T11:52:06.361 INFO:tasks.workunit.client.0.smithi195.stdout:removed './222/25' 2022-06-27T11:52:06.362 INFO:tasks.workunit.client.0.smithi195.stdout:removed './222/30' 2022-06-27T11:52:06.362 INFO:tasks.workunit.client.0.smithi195.stdout:removed './222/33' 2022-06-27T11:52:06.362 INFO:tasks.workunit.client.0.smithi195.stdout:removed './222/23' 2022-06-27T11:52:06.363 INFO:tasks.workunit.client.0.smithi195.stdout:removed './222/35' 2022-06-27T11:52:06.363 INFO:tasks.workunit.client.0.smithi195.stdout:removed './222/26' 2022-06-27T11:52:06.363 INFO:tasks.workunit.client.0.smithi195.stdout:removed './222/37' 2022-06-27T11:52:06.364 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './222' 2022-06-27T11:52:06.364 INFO:tasks.workunit.client.0.smithi195.stdout:removed './223/21' 2022-06-27T11:52:06.364 INFO:tasks.workunit.client.0.smithi195.stdout:removed './223/27' 2022-06-27T11:52:06.365 INFO:tasks.workunit.client.0.smithi195.stdout:removed './223/31' 2022-06-27T11:52:06.365 INFO:tasks.workunit.client.0.smithi195.stdout:removed './223/36' 2022-06-27T11:52:06.366 INFO:tasks.workunit.client.0.smithi195.stdout:removed './223/20' 2022-06-27T11:52:06.366 INFO:tasks.workunit.client.0.smithi195.stdout:removed './223/39' 2022-06-27T11:52:06.366 INFO:tasks.workunit.client.0.smithi195.stdout:removed './223/34' 2022-06-27T11:52:06.367 INFO:tasks.workunit.client.0.smithi195.stdout:removed './223/38' 2022-06-27T11:52:06.367 INFO:tasks.workunit.client.0.smithi195.stdout:removed './223/28' 2022-06-27T11:52:06.367 INFO:tasks.workunit.client.0.smithi195.stdout:removed './223/24' 2022-06-27T11:52:06.369 INFO:tasks.workunit.client.0.smithi195.stdout:removed './223/29' 2022-06-27T11:52:06.369 INFO:tasks.workunit.client.0.smithi195.stdout:removed './223/32' 2022-06-27T11:52:06.370 INFO:tasks.workunit.client.0.smithi195.stdout:removed './223/22' 2022-06-27T11:52:06.370 INFO:tasks.workunit.client.0.smithi195.stdout:removed './223/25' 2022-06-27T11:52:06.370 INFO:tasks.workunit.client.0.smithi195.stdout:removed './223/30' 2022-06-27T11:52:06.371 INFO:tasks.workunit.client.0.smithi195.stdout:removed './223/33' 2022-06-27T11:52:06.371 INFO:tasks.workunit.client.0.smithi195.stdout:removed './223/23' 2022-06-27T11:52:06.372 INFO:tasks.workunit.client.0.smithi195.stdout:removed './223/35' 2022-06-27T11:52:06.372 INFO:tasks.workunit.client.0.smithi195.stdout:removed './223/26' 2022-06-27T11:52:06.373 INFO:tasks.workunit.client.0.smithi195.stdout:removed './223/37' 2022-06-27T11:52:06.373 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './223' 2022-06-27T11:52:06.373 INFO:tasks.workunit.client.0.smithi195.stdout:removed './224/21' 2022-06-27T11:52:06.374 INFO:tasks.workunit.client.0.smithi195.stdout:removed './224/27' 2022-06-27T11:52:06.374 INFO:tasks.workunit.client.0.smithi195.stdout:removed './224/31' 2022-06-27T11:52:06.375 INFO:tasks.workunit.client.0.smithi195.stdout:removed './224/36' 2022-06-27T11:52:06.375 INFO:tasks.workunit.client.0.smithi195.stdout:removed './224/20' 2022-06-27T11:52:06.376 INFO:tasks.workunit.client.0.smithi195.stdout:removed './224/39' 2022-06-27T11:52:06.376 INFO:tasks.workunit.client.0.smithi195.stdout:removed './224/34' 2022-06-27T11:52:06.376 INFO:tasks.workunit.client.0.smithi195.stdout:removed './224/38' 2022-06-27T11:52:06.377 INFO:tasks.workunit.client.0.smithi195.stdout:removed './224/28' 2022-06-27T11:52:06.377 INFO:tasks.workunit.client.0.smithi195.stdout:removed './224/24' 2022-06-27T11:52:06.378 INFO:tasks.workunit.client.0.smithi195.stdout:removed './224/29' 2022-06-27T11:52:06.379 INFO:tasks.workunit.client.0.smithi195.stdout:removed './224/32' 2022-06-27T11:52:06.379 INFO:tasks.workunit.client.0.smithi195.stdout:removed './224/22' 2022-06-27T11:52:06.379 INFO:tasks.workunit.client.0.smithi195.stdout:removed './224/25' 2022-06-27T11:52:06.380 INFO:tasks.workunit.client.0.smithi195.stdout:removed './224/30' 2022-06-27T11:52:06.380 INFO:tasks.workunit.client.0.smithi195.stdout:removed './224/33' 2022-06-27T11:52:06.380 INFO:tasks.workunit.client.0.smithi195.stdout:removed './224/23' 2022-06-27T11:52:06.381 INFO:tasks.workunit.client.0.smithi195.stdout:removed './224/35' 2022-06-27T11:52:06.381 INFO:tasks.workunit.client.0.smithi195.stdout:removed './224/26' 2022-06-27T11:52:06.382 INFO:tasks.workunit.client.0.smithi195.stdout:removed './224/37' 2022-06-27T11:52:06.382 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './224' 2022-06-27T11:52:06.382 INFO:tasks.workunit.client.0.smithi195.stdout:removed './225/21' 2022-06-27T11:52:06.383 INFO:tasks.workunit.client.0.smithi195.stdout:removed './225/27' 2022-06-27T11:52:06.383 INFO:tasks.workunit.client.0.smithi195.stdout:removed './225/31' 2022-06-27T11:52:06.384 INFO:tasks.workunit.client.0.smithi195.stdout:removed './225/36' 2022-06-27T11:52:06.384 INFO:tasks.workunit.client.0.smithi195.stdout:removed './225/20' 2022-06-27T11:52:06.385 INFO:tasks.workunit.client.0.smithi195.stdout:removed './225/39' 2022-06-27T11:52:06.385 INFO:tasks.workunit.client.0.smithi195.stdout:removed './225/34' 2022-06-27T11:52:06.385 INFO:tasks.workunit.client.0.smithi195.stdout:removed './225/38' 2022-06-27T11:52:06.386 INFO:tasks.workunit.client.0.smithi195.stdout:removed './225/28' 2022-06-27T11:52:06.386 INFO:tasks.workunit.client.0.smithi195.stdout:removed './225/24' 2022-06-27T11:52:06.387 INFO:tasks.workunit.client.0.smithi195.stdout:removed './225/29' 2022-06-27T11:52:06.387 INFO:tasks.workunit.client.0.smithi195.stdout:removed './225/32' 2022-06-27T11:52:06.388 INFO:tasks.workunit.client.0.smithi195.stdout:removed './225/22' 2022-06-27T11:52:06.388 INFO:tasks.workunit.client.0.smithi195.stdout:removed './225/25' 2022-06-27T11:52:06.389 INFO:tasks.workunit.client.0.smithi195.stdout:removed './225/30' 2022-06-27T11:52:06.389 INFO:tasks.workunit.client.0.smithi195.stdout:removed './225/33' 2022-06-27T11:52:06.389 INFO:tasks.workunit.client.0.smithi195.stdout:removed './225/23' 2022-06-27T11:52:06.390 INFO:tasks.workunit.client.0.smithi195.stdout:removed './225/35' 2022-06-27T11:52:06.390 INFO:tasks.workunit.client.0.smithi195.stdout:removed './225/26' 2022-06-27T11:52:06.391 INFO:tasks.workunit.client.0.smithi195.stdout:removed './225/37' 2022-06-27T11:52:06.391 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './225' 2022-06-27T11:52:06.391 INFO:tasks.workunit.client.0.smithi195.stdout:removed './226/21' 2022-06-27T11:52:06.392 INFO:tasks.workunit.client.0.smithi195.stdout:removed './226/27' 2022-06-27T11:52:06.392 INFO:tasks.workunit.client.0.smithi195.stdout:removed './226/31' 2022-06-27T11:52:06.393 INFO:tasks.workunit.client.0.smithi195.stdout:removed './226/36' 2022-06-27T11:52:06.393 INFO:tasks.workunit.client.0.smithi195.stdout:removed './226/20' 2022-06-27T11:52:06.393 INFO:tasks.workunit.client.0.smithi195.stdout:removed './226/39' 2022-06-27T11:52:06.394 INFO:tasks.workunit.client.0.smithi195.stdout:removed './226/34' 2022-06-27T11:52:06.394 INFO:tasks.workunit.client.0.smithi195.stdout:removed './226/38' 2022-06-27T11:52:06.394 INFO:tasks.workunit.client.0.smithi195.stdout:removed './226/28' 2022-06-27T11:52:06.395 INFO:tasks.workunit.client.0.smithi195.stdout:removed './226/24' 2022-06-27T11:52:09.699 INFO:tasks.workunit.client.0.smithi195.stdout:removed './226/29' 2022-06-27T11:52:09.700 INFO:tasks.workunit.client.0.smithi195.stdout:removed './226/32' 2022-06-27T11:52:09.700 INFO:tasks.workunit.client.0.smithi195.stdout:removed './226/22' 2022-06-27T11:52:09.701 INFO:tasks.workunit.client.0.smithi195.stdout:removed './226/25' 2022-06-27T11:52:09.702 INFO:tasks.workunit.client.0.smithi195.stdout:removed './226/30' 2022-06-27T11:52:09.702 INFO:tasks.workunit.client.0.smithi195.stdout:removed './226/33' 2022-06-27T11:52:09.703 INFO:tasks.workunit.client.0.smithi195.stdout:removed './226/23' 2022-06-27T11:52:09.703 INFO:tasks.workunit.client.0.smithi195.stdout:removed './226/35' 2022-06-27T11:52:09.704 INFO:tasks.workunit.client.0.smithi195.stdout:removed './226/26' 2022-06-27T11:52:09.704 INFO:tasks.workunit.client.0.smithi195.stdout:removed './226/37' 2022-06-27T11:52:09.705 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './226' 2022-06-27T11:52:09.705 INFO:tasks.workunit.client.0.smithi195.stdout:removed './227/21' 2022-06-27T11:52:09.705 INFO:tasks.workunit.client.0.smithi195.stdout:removed './227/27' 2022-06-27T11:52:09.706 INFO:tasks.workunit.client.0.smithi195.stdout:removed './227/31' 2022-06-27T11:52:09.706 INFO:tasks.workunit.client.0.smithi195.stdout:removed './227/36' 2022-06-27T11:52:09.707 INFO:tasks.workunit.client.0.smithi195.stdout:removed './227/20' 2022-06-27T11:52:09.707 INFO:tasks.workunit.client.0.smithi195.stdout:removed './227/39' 2022-06-27T11:52:09.708 INFO:tasks.workunit.client.0.smithi195.stdout:removed './227/34' 2022-06-27T11:52:09.708 INFO:tasks.workunit.client.0.smithi195.stdout:removed './227/38' 2022-06-27T11:52:09.708 INFO:tasks.workunit.client.0.smithi195.stdout:removed './227/28' 2022-06-27T11:52:09.708 INFO:tasks.workunit.client.0.smithi195.stdout:removed './227/24' 2022-06-27T11:52:09.709 INFO:tasks.workunit.client.0.smithi195.stdout:removed './227/29' 2022-06-27T11:52:09.709 INFO:tasks.workunit.client.0.smithi195.stdout:removed './227/32' 2022-06-27T11:52:09.709 INFO:tasks.workunit.client.0.smithi195.stdout:removed './227/22' 2022-06-27T11:52:09.710 INFO:tasks.workunit.client.0.smithi195.stdout:removed './227/25' 2022-06-27T11:52:09.710 INFO:tasks.workunit.client.0.smithi195.stdout:removed './227/30' 2022-06-27T11:52:09.710 INFO:tasks.workunit.client.0.smithi195.stdout:removed './227/33' 2022-06-27T11:52:09.710 INFO:tasks.workunit.client.0.smithi195.stdout:removed './227/23' 2022-06-27T11:52:09.710 INFO:tasks.workunit.client.0.smithi195.stdout:removed './227/35' 2022-06-27T11:52:09.711 INFO:tasks.workunit.client.0.smithi195.stdout:removed './227/26' 2022-06-27T11:52:09.711 INFO:tasks.workunit.client.0.smithi195.stdout:removed './227/37' 2022-06-27T11:52:09.711 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './227' 2022-06-27T11:52:09.711 INFO:tasks.workunit.client.0.smithi195.stdout:removed './228/21' 2022-06-27T11:52:09.712 INFO:tasks.workunit.client.0.smithi195.stdout:removed './228/27' 2022-06-27T11:52:09.712 INFO:tasks.workunit.client.0.smithi195.stdout:removed './228/31' 2022-06-27T11:52:09.712 INFO:tasks.workunit.client.0.smithi195.stdout:removed './228/36' 2022-06-27T11:52:09.713 INFO:tasks.workunit.client.0.smithi195.stdout:removed './228/20' 2022-06-27T11:52:09.714 INFO:tasks.workunit.client.0.smithi195.stdout:removed './228/39' 2022-06-27T11:52:09.714 INFO:tasks.workunit.client.0.smithi195.stdout:removed './228/34' 2022-06-27T11:52:09.714 INFO:tasks.workunit.client.0.smithi195.stdout:removed './228/38' 2022-06-27T11:52:09.715 INFO:tasks.workunit.client.0.smithi195.stdout:removed './228/28' 2022-06-27T11:52:09.715 INFO:tasks.workunit.client.0.smithi195.stdout:removed './228/24' 2022-06-27T11:52:09.716 INFO:tasks.workunit.client.0.smithi195.stdout:removed './228/29' 2022-06-27T11:52:09.716 INFO:tasks.workunit.client.0.smithi195.stdout:removed './228/32' 2022-06-27T11:52:09.716 INFO:tasks.workunit.client.0.smithi195.stdout:removed './228/22' 2022-06-27T11:52:09.716 INFO:tasks.workunit.client.0.smithi195.stdout:removed './228/25' 2022-06-27T11:52:09.716 INFO:tasks.workunit.client.0.smithi195.stdout:removed './228/30' 2022-06-27T11:52:09.717 INFO:tasks.workunit.client.0.smithi195.stdout:removed './228/33' 2022-06-27T11:52:09.717 INFO:tasks.workunit.client.0.smithi195.stdout:removed './228/23' 2022-06-27T11:52:09.717 INFO:tasks.workunit.client.0.smithi195.stdout:removed './228/35' 2022-06-27T11:52:09.717 INFO:tasks.workunit.client.0.smithi195.stdout:removed './228/26' 2022-06-27T11:52:09.718 INFO:tasks.workunit.client.0.smithi195.stdout:removed './228/37' 2022-06-27T11:52:09.718 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './228' 2022-06-27T11:52:09.718 INFO:tasks.workunit.client.0.smithi195.stdout:removed './229/21' 2022-06-27T11:52:09.719 INFO:tasks.workunit.client.0.smithi195.stdout:removed './229/27' 2022-06-27T11:52:09.719 INFO:tasks.workunit.client.0.smithi195.stdout:removed './229/31' 2022-06-27T11:52:09.720 INFO:tasks.workunit.client.0.smithi195.stdout:removed './229/36' 2022-06-27T11:52:09.720 INFO:tasks.workunit.client.0.smithi195.stdout:removed './229/20' 2022-06-27T11:52:09.720 INFO:tasks.workunit.client.0.smithi195.stdout:removed './229/39' 2022-06-27T11:52:09.720 INFO:tasks.workunit.client.0.smithi195.stdout:removed './229/34' 2022-06-27T11:52:09.721 INFO:tasks.workunit.client.0.smithi195.stdout:removed './229/38' 2022-06-27T11:52:09.721 INFO:tasks.workunit.client.0.smithi195.stdout:removed './229/28' 2022-06-27T11:52:09.721 INFO:tasks.workunit.client.0.smithi195.stdout:removed './229/24' 2022-06-27T11:52:09.721 INFO:tasks.workunit.client.0.smithi195.stdout:removed './229/29' 2022-06-27T11:52:09.722 INFO:tasks.workunit.client.0.smithi195.stdout:removed './229/32' 2022-06-27T11:52:09.722 INFO:tasks.workunit.client.0.smithi195.stdout:removed './229/22' 2022-06-27T11:52:09.722 INFO:tasks.workunit.client.0.smithi195.stdout:removed './229/25' 2022-06-27T11:52:09.722 INFO:tasks.workunit.client.0.smithi195.stdout:removed './229/30' 2022-06-27T11:52:09.723 INFO:tasks.workunit.client.0.smithi195.stdout:removed './229/33' 2022-06-27T11:52:09.723 INFO:tasks.workunit.client.0.smithi195.stdout:removed './229/23' 2022-06-27T11:52:09.723 INFO:tasks.workunit.client.0.smithi195.stdout:removed './229/35' 2022-06-27T11:52:09.723 INFO:tasks.workunit.client.0.smithi195.stdout:removed './229/26' 2022-06-27T11:52:09.724 INFO:tasks.workunit.client.0.smithi195.stdout:removed './229/37' 2022-06-27T11:52:09.724 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './229' 2022-06-27T11:52:09.724 INFO:tasks.workunit.client.0.smithi195.stdout:removed './230/21' 2022-06-27T11:52:09.724 INFO:tasks.workunit.client.0.smithi195.stdout:removed './230/27' 2022-06-27T11:52:09.724 INFO:tasks.workunit.client.0.smithi195.stdout:removed './230/31' 2022-06-27T11:52:09.725 INFO:tasks.workunit.client.0.smithi195.stdout:removed './230/36' 2022-06-27T11:52:09.725 INFO:tasks.workunit.client.0.smithi195.stdout:removed './230/20' 2022-06-27T11:52:09.725 INFO:tasks.workunit.client.0.smithi195.stdout:removed './230/39' 2022-06-27T11:52:09.725 INFO:tasks.workunit.client.0.smithi195.stdout:removed './230/34' 2022-06-27T11:52:09.726 INFO:tasks.workunit.client.0.smithi195.stdout:removed './230/38' 2022-06-27T11:52:09.728 INFO:tasks.workunit.client.0.smithi195.stdout:removed './230/28' 2022-06-27T11:52:09.728 INFO:tasks.workunit.client.0.smithi195.stdout:removed './230/24' 2022-06-27T11:52:09.729 INFO:tasks.workunit.client.0.smithi195.stdout:removed './230/29' 2022-06-27T11:52:09.729 INFO:tasks.workunit.client.0.smithi195.stdout:removed './230/32' 2022-06-27T11:52:09.729 INFO:tasks.workunit.client.0.smithi195.stdout:removed './230/22' 2022-06-27T11:52:09.729 INFO:tasks.workunit.client.0.smithi195.stdout:removed './230/25' 2022-06-27T11:52:09.730 INFO:tasks.workunit.client.0.smithi195.stdout:removed './230/30' 2022-06-27T11:52:09.730 INFO:tasks.workunit.client.0.smithi195.stdout:removed './230/33' 2022-06-27T11:52:09.730 INFO:tasks.workunit.client.0.smithi195.stdout:removed './230/23' 2022-06-27T11:52:09.730 INFO:tasks.workunit.client.0.smithi195.stdout:removed './230/35' 2022-06-27T11:52:09.731 INFO:tasks.workunit.client.0.smithi195.stdout:removed './230/26' 2022-06-27T11:52:09.731 INFO:tasks.workunit.client.0.smithi195.stdout:removed './230/37' 2022-06-27T11:52:09.731 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './230' 2022-06-27T11:52:09.731 INFO:tasks.workunit.client.0.smithi195.stdout:removed './231/21' 2022-06-27T11:52:09.731 INFO:tasks.workunit.client.0.smithi195.stdout:removed './231/27' 2022-06-27T11:52:09.732 INFO:tasks.workunit.client.0.smithi195.stdout:removed './231/31' 2022-06-27T11:52:09.732 INFO:tasks.workunit.client.0.smithi195.stdout:removed './231/36' 2022-06-27T11:52:09.732 INFO:tasks.workunit.client.0.smithi195.stdout:removed './231/20' 2022-06-27T11:52:09.733 INFO:tasks.workunit.client.0.smithi195.stdout:removed './231/39' 2022-06-27T11:52:09.733 INFO:tasks.workunit.client.0.smithi195.stdout:removed './231/34' 2022-06-27T11:52:09.733 INFO:tasks.workunit.client.0.smithi195.stdout:removed './231/38' 2022-06-27T11:52:09.734 INFO:tasks.workunit.client.0.smithi195.stdout:removed './231/28' 2022-06-27T11:52:09.734 INFO:tasks.workunit.client.0.smithi195.stdout:removed './231/24' 2022-06-27T11:52:09.734 INFO:tasks.workunit.client.0.smithi195.stdout:removed './231/29' 2022-06-27T11:52:09.734 INFO:tasks.workunit.client.0.smithi195.stdout:removed './231/32' 2022-06-27T11:52:09.735 INFO:tasks.workunit.client.0.smithi195.stdout:removed './231/22' 2022-06-27T11:52:09.735 INFO:tasks.workunit.client.0.smithi195.stdout:removed './231/25' 2022-06-27T11:52:09.735 INFO:tasks.workunit.client.0.smithi195.stdout:removed './231/30' 2022-06-27T11:52:09.735 INFO:tasks.workunit.client.0.smithi195.stdout:removed './231/33' 2022-06-27T11:52:09.736 INFO:tasks.workunit.client.0.smithi195.stdout:removed './231/23' 2022-06-27T11:52:09.736 INFO:tasks.workunit.client.0.smithi195.stdout:removed './231/35' 2022-06-27T11:52:09.736 INFO:tasks.workunit.client.0.smithi195.stdout:removed './231/26' 2022-06-27T11:52:09.736 INFO:tasks.workunit.client.0.smithi195.stdout:removed './231/37' 2022-06-27T11:52:09.736 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './231' 2022-06-27T11:52:09.737 INFO:tasks.workunit.client.0.smithi195.stdout:removed './232/21' 2022-06-27T11:52:09.737 INFO:tasks.workunit.client.0.smithi195.stdout:removed './232/27' 2022-06-27T11:52:09.737 INFO:tasks.workunit.client.0.smithi195.stdout:removed './232/31' 2022-06-27T11:52:09.737 INFO:tasks.workunit.client.0.smithi195.stdout:removed './232/36' 2022-06-27T11:52:09.738 INFO:tasks.workunit.client.0.smithi195.stdout:removed './232/20' 2022-06-27T11:52:09.738 INFO:tasks.workunit.client.0.smithi195.stdout:removed './232/39' 2022-06-27T11:52:09.738 INFO:tasks.workunit.client.0.smithi195.stdout:removed './232/34' 2022-06-27T11:52:09.739 INFO:tasks.workunit.client.0.smithi195.stdout:removed './232/38' 2022-06-27T11:52:09.739 INFO:tasks.workunit.client.0.smithi195.stdout:removed './232/28' 2022-06-27T11:52:09.740 INFO:tasks.workunit.client.0.smithi195.stdout:removed './232/24' 2022-06-27T11:52:09.740 INFO:tasks.workunit.client.0.smithi195.stdout:removed './232/29' 2022-06-27T11:52:09.741 INFO:tasks.workunit.client.0.smithi195.stdout:removed './232/32' 2022-06-27T11:52:09.741 INFO:tasks.workunit.client.0.smithi195.stdout:removed './232/22' 2022-06-27T11:52:09.741 INFO:tasks.workunit.client.0.smithi195.stdout:removed './232/25' 2022-06-27T11:52:09.742 INFO:tasks.workunit.client.0.smithi195.stdout:removed './232/30' 2022-06-27T11:52:09.742 INFO:tasks.workunit.client.0.smithi195.stdout:removed './232/33' 2022-06-27T11:52:09.742 INFO:tasks.workunit.client.0.smithi195.stdout:removed './232/23' 2022-06-27T11:52:09.742 INFO:tasks.workunit.client.0.smithi195.stdout:removed './232/35' 2022-06-27T11:52:09.743 INFO:tasks.workunit.client.0.smithi195.stdout:removed './232/26' 2022-06-27T11:52:09.743 INFO:tasks.workunit.client.0.smithi195.stdout:removed './232/37' 2022-06-27T11:52:09.743 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './232' 2022-06-27T11:52:09.743 INFO:tasks.workunit.client.0.smithi195.stdout:removed './233/21' 2022-06-27T11:52:09.743 INFO:tasks.workunit.client.0.smithi195.stdout:removed './233/27' 2022-06-27T11:52:09.744 INFO:tasks.workunit.client.0.smithi195.stdout:removed './233/31' 2022-06-27T11:52:09.744 INFO:tasks.workunit.client.0.smithi195.stdout:removed './233/36' 2022-06-27T11:52:09.744 INFO:tasks.workunit.client.0.smithi195.stdout:removed './233/20' 2022-06-27T11:52:09.745 INFO:tasks.workunit.client.0.smithi195.stdout:removed './233/39' 2022-06-27T11:52:09.745 INFO:tasks.workunit.client.0.smithi195.stdout:removed './233/34' 2022-06-27T11:52:09.745 INFO:tasks.workunit.client.0.smithi195.stdout:removed './233/38' 2022-06-27T11:52:09.746 INFO:tasks.workunit.client.0.smithi195.stdout:removed './233/28' 2022-06-27T11:52:09.746 INFO:tasks.workunit.client.0.smithi195.stdout:removed './233/24' 2022-06-27T11:52:09.746 INFO:tasks.workunit.client.0.smithi195.stdout:removed './233/29' 2022-06-27T11:52:09.747 INFO:tasks.workunit.client.0.smithi195.stdout:removed './233/32' 2022-06-27T11:52:09.747 INFO:tasks.workunit.client.0.smithi195.stdout:removed './233/22' 2022-06-27T11:52:09.747 INFO:tasks.workunit.client.0.smithi195.stdout:removed './233/25' 2022-06-27T11:52:09.747 INFO:tasks.workunit.client.0.smithi195.stdout:removed './233/30' 2022-06-27T11:52:09.747 INFO:tasks.workunit.client.0.smithi195.stdout:removed './233/33' 2022-06-27T11:52:09.748 INFO:tasks.workunit.client.0.smithi195.stdout:removed './233/23' 2022-06-27T11:52:09.748 INFO:tasks.workunit.client.0.smithi195.stdout:removed './233/35' 2022-06-27T11:52:09.748 INFO:tasks.workunit.client.0.smithi195.stdout:removed './233/26' 2022-06-27T11:52:09.748 INFO:tasks.workunit.client.0.smithi195.stdout:removed './233/37' 2022-06-27T11:52:09.749 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './233' 2022-06-27T11:52:09.749 INFO:tasks.workunit.client.0.smithi195.stdout:removed './234/21' 2022-06-27T11:52:09.749 INFO:tasks.workunit.client.0.smithi195.stdout:removed './234/27' 2022-06-27T11:52:09.749 INFO:tasks.workunit.client.0.smithi195.stdout:removed './234/31' 2022-06-27T11:52:09.750 INFO:tasks.workunit.client.0.smithi195.stdout:removed './234/36' 2022-06-27T11:52:09.750 INFO:tasks.workunit.client.0.smithi195.stdout:removed './234/20' 2022-06-27T11:52:09.750 INFO:tasks.workunit.client.0.smithi195.stdout:removed './234/39' 2022-06-27T11:52:09.750 INFO:tasks.workunit.client.0.smithi195.stdout:removed './234/34' 2022-06-27T11:52:09.751 INFO:tasks.workunit.client.0.smithi195.stdout:removed './234/38' 2022-06-27T11:52:09.751 INFO:tasks.workunit.client.0.smithi195.stdout:removed './234/28' 2022-06-27T11:52:09.751 INFO:tasks.workunit.client.0.smithi195.stdout:removed './234/24' 2022-06-27T11:52:09.751 INFO:tasks.workunit.client.0.smithi195.stdout:removed './234/29' 2022-06-27T11:52:09.752 INFO:tasks.workunit.client.0.smithi195.stdout:removed './234/32' 2022-06-27T11:52:09.752 INFO:tasks.workunit.client.0.smithi195.stdout:removed './234/22' 2022-06-27T11:52:09.752 INFO:tasks.workunit.client.0.smithi195.stdout:removed './234/25' 2022-06-27T11:52:09.752 INFO:tasks.workunit.client.0.smithi195.stdout:removed './234/30' 2022-06-27T11:52:09.752 INFO:tasks.workunit.client.0.smithi195.stdout:removed './234/33' 2022-06-27T11:52:09.753 INFO:tasks.workunit.client.0.smithi195.stdout:removed './234/23' 2022-06-27T11:52:09.753 INFO:tasks.workunit.client.0.smithi195.stdout:removed './234/35' 2022-06-27T11:52:09.753 INFO:tasks.workunit.client.0.smithi195.stdout:removed './234/26' 2022-06-27T11:52:09.753 INFO:tasks.workunit.client.0.smithi195.stdout:removed './234/37' 2022-06-27T11:52:09.754 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './234' 2022-06-27T11:52:09.754 INFO:tasks.workunit.client.0.smithi195.stdout:removed './235/21' 2022-06-27T11:52:09.754 INFO:tasks.workunit.client.0.smithi195.stdout:removed './235/27' 2022-06-27T11:52:09.755 INFO:tasks.workunit.client.0.smithi195.stdout:removed './235/31' 2022-06-27T11:52:09.755 INFO:tasks.workunit.client.0.smithi195.stdout:removed './235/36' 2022-06-27T11:52:09.755 INFO:tasks.workunit.client.0.smithi195.stdout:removed './235/20' 2022-06-27T11:52:09.755 INFO:tasks.workunit.client.0.smithi195.stdout:removed './235/39' 2022-06-27T11:52:09.756 INFO:tasks.workunit.client.0.smithi195.stdout:removed './235/34' 2022-06-27T11:52:09.756 INFO:tasks.workunit.client.0.smithi195.stdout:removed './235/38' 2022-06-27T11:52:09.756 INFO:tasks.workunit.client.0.smithi195.stdout:removed './235/28' 2022-06-27T11:52:09.756 INFO:tasks.workunit.client.0.smithi195.stdout:removed './235/24' 2022-06-27T11:52:09.757 INFO:tasks.workunit.client.0.smithi195.stdout:removed './235/29' 2022-06-27T11:52:09.757 INFO:tasks.workunit.client.0.smithi195.stdout:removed './235/32' 2022-06-27T11:52:09.757 INFO:tasks.workunit.client.0.smithi195.stdout:removed './235/22' 2022-06-27T11:52:09.757 INFO:tasks.workunit.client.0.smithi195.stdout:removed './235/25' 2022-06-27T11:52:09.758 INFO:tasks.workunit.client.0.smithi195.stdout:removed './235/30' 2022-06-27T11:52:09.758 INFO:tasks.workunit.client.0.smithi195.stdout:removed './235/33' 2022-06-27T11:52:09.758 INFO:tasks.workunit.client.0.smithi195.stdout:removed './235/23' 2022-06-27T11:52:09.758 INFO:tasks.workunit.client.0.smithi195.stdout:removed './235/35' 2022-06-27T11:52:09.759 INFO:tasks.workunit.client.0.smithi195.stdout:removed './235/26' 2022-06-27T11:52:09.759 INFO:tasks.workunit.client.0.smithi195.stdout:removed './235/37' 2022-06-27T11:52:09.759 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './235' 2022-06-27T11:52:09.759 INFO:tasks.workunit.client.0.smithi195.stdout:removed './236/21' 2022-06-27T11:52:09.759 INFO:tasks.workunit.client.0.smithi195.stdout:removed './236/27' 2022-06-27T11:52:09.760 INFO:tasks.workunit.client.0.smithi195.stdout:removed './236/31' 2022-06-27T11:52:09.760 INFO:tasks.workunit.client.0.smithi195.stdout:removed './236/36' 2022-06-27T11:52:09.760 INFO:tasks.workunit.client.0.smithi195.stdout:removed './236/20' 2022-06-27T11:52:09.760 INFO:tasks.workunit.client.0.smithi195.stdout:removed './236/39' 2022-06-27T11:52:09.761 INFO:tasks.workunit.client.0.smithi195.stdout:removed './236/34' 2022-06-27T11:52:09.761 INFO:tasks.workunit.client.0.smithi195.stdout:removed './236/38' 2022-06-27T11:52:09.761 INFO:tasks.workunit.client.0.smithi195.stdout:removed './236/28' 2022-06-27T11:52:09.761 INFO:tasks.workunit.client.0.smithi195.stdout:removed './236/24' 2022-06-27T11:52:09.761 INFO:tasks.workunit.client.0.smithi195.stdout:removed './236/29' 2022-06-27T11:52:09.762 INFO:tasks.workunit.client.0.smithi195.stdout:removed './236/32' 2022-06-27T11:52:11.949 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:52:12.087 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.087+0000 7ff9c6cce700 1 Processor -- start 2022-06-27T11:52:12.088 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.088+0000 7ff9c6cce700 1 -- start start 2022-06-27T11:52:12.097 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.088+0000 7ff9c6cce700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7ff9c00688a0 0x7ff9c00c28c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:12.097 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.088+0000 7ff9c6cce700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9c00c4b10 con 0x7ff9c006a440 2022-06-27T11:52:12.099 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.088+0000 7ff9c6cce700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9c0068f20 con 0x7ff9c0156100 2022-06-27T11:52:12.099 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.088+0000 7ff9c6cce700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9c00c3dc0 con 0x7ff9c00688a0 2022-06-27T11:52:12.099 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.089+0000 7ff9c526b700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7ff9c00688a0 0x7ff9c00c28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:12.099 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.089+0000 7ff9bffff700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7ff9c006a440 0x7ff9c01564d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:49656/0 (socket says 172.21.15.195:49656) 2022-06-27T11:52:12.100 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.089+0000 7ff9bffff700 1 -- 172.21.15.195:0/3766753388 learned_addr learned my addr 172.21.15.195:0/3766753388 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:52:12.100 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.089+0000 7ff9c526b700 1 -- 172.21.15.195:0/3766753388 >> v1:172.21.15.202:6789/0 conn(0x7ff9c0156100 legacy=0x7ff9c00c4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:12.100 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.089+0000 7ff9c526b700 1 -- 172.21.15.195:0/3766753388 >> v1:172.21.15.195:6789/0 conn(0x7ff9c006a440 legacy=0x7ff9c01564d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:12.101 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.089+0000 7ff9c526b700 1 -- 172.21.15.195:0/3766753388 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9c00c56b0 con 0x7ff9c00688a0 2022-06-27T11:52:12.101 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.089+0000 7ff9bf7fe700 1 -- 172.21.15.195:0/3766753388 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1840290874 0 0) 0x7ff9c00c4b10 con 0x7ff9c006a440 2022-06-27T11:52:12.101 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.089+0000 7ff9c526b700 1 --2- 172.21.15.195:0/3766753388 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff9c00688a0 0x7ff9c00c28c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7ff9b800d0d0 tx=0x7ff9b800b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc64c92b0815f5dd server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:52:12.101 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.089+0000 7ff9bf7fe700 1 -- 172.21.15.195:0/3766753388 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9b8010070 con 0x7ff9c00688a0 2022-06-27T11:52:12.101 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.089+0000 7ff9bf7fe700 1 -- 172.21.15.195:0/3766753388 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9b800ddf0 con 0x7ff9c00688a0 2022-06-27T11:52:12.102 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.089+0000 7ff9bf7fe700 1 -- 172.21.15.195:0/3766753388 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9b800b320 con 0x7ff9c00688a0 2022-06-27T11:52:12.102 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.090+0000 7ff9c6cce700 1 -- 172.21.15.195:0/3766753388 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff9c00688a0 msgr2=0x7ff9c00c28c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:12.102 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.090+0000 7ff9c6cce700 1 --2- 172.21.15.195:0/3766753388 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff9c00688a0 0x7ff9c00c28c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7ff9b800d0d0 tx=0x7ff9b800b990 comp rx=0 tx=0).stop 2022-06-27T11:52:12.103 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.090+0000 7ff9c6cce700 1 -- 172.21.15.195:0/3766753388 shutdown_connections 2022-06-27T11:52:12.103 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.090+0000 7ff9c6cce700 1 --2- 172.21.15.195:0/3766753388 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff9c00688a0 0x7ff9c00c28c0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:12.103 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.090+0000 7ff9c6cce700 1 -- 172.21.15.195:0/3766753388 >> 172.21.15.195:0/3766753388 conn(0x7ff9c00c9b00 msgr2=0x7ff9c00c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:52:12.103 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.090+0000 7ff9c6cce700 1 -- 172.21.15.195:0/3766753388 shutdown_connections 2022-06-27T11:52:12.104 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.090+0000 7ff9c6cce700 1 -- 172.21.15.195:0/3766753388 wait complete. 2022-06-27T11:52:12.104 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.091+0000 7ff9c6cce700 1 Processor -- start 2022-06-27T11:52:12.104 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.091+0000 7ff9c6cce700 1 -- start start 2022-06-27T11:52:12.104 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.092+0000 7ff9c6cce700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff9c00688a0 0x7ff9c0151c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:12.105 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.092+0000 7ff9c6cce700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff9c006a440 0x7ff9c0153ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:12.105 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.092+0000 7ff9c6cce700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff9c0156100 0x7ff9c01652f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:12.105 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.092+0000 7ff9c6cce700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9c01628a0 con 0x7ff9c00688a0 2022-06-27T11:52:12.105 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.092+0000 7ff9c6cce700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9c01521b0 con 0x7ff9c006a440 2022-06-27T11:52:12.106 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.092+0000 7ff9c6cce700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9c00c6fc0 con 0x7ff9c0156100 2022-06-27T11:52:12.106 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.092+0000 7ff9c4a6a700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff9c00688a0 0x7ff9c0151c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:12.106 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.092+0000 7ff9c4a6a700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff9c00688a0 0x7ff9c0151c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:43476/0 (socket says 172.21.15.195:43476) 2022-06-27T11:52:12.106 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.092+0000 7ff9c4a6a700 1 -- 172.21.15.195:0/2366864220 learned_addr learned my addr 172.21.15.195:0/2366864220 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:52:12.106 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.092+0000 7ff9bffff700 1 --2- 172.21.15.195:0/2366864220 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff9c006a440 0x7ff9c0153ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:12.107 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.092+0000 7ff9c526b700 1 --2- 172.21.15.195:0/2366864220 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff9c0156100 0x7ff9c01652f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:12.107 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.092+0000 7ff9c4a6a700 1 -- 172.21.15.195:0/2366864220 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff9c0156100 msgr2=0x7ff9c01652f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:12.107 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.092+0000 7ff9c4a6a700 1 --2- 172.21.15.195:0/2366864220 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff9c0156100 0x7ff9c01652f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:12.107 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.092+0000 7ff9c4a6a700 1 -- 172.21.15.195:0/2366864220 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff9c006a440 msgr2=0x7ff9c0153ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:12.108 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.092+0000 7ff9c4a6a700 1 --2- 172.21.15.195:0/2366864220 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff9c006a440 0x7ff9c0153ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:12.108 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.092+0000 7ff9c4a6a700 1 -- 172.21.15.195:0/2366864220 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9b800bc30 con 0x7ff9c00688a0 2022-06-27T11:52:12.108 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.092+0000 7ff9c4a6a700 1 --2- 172.21.15.195:0/2366864220 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff9c00688a0 0x7ff9c0151c70 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7ff9b400c4c0 tx=0x7ff9b4007d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:52:12.108 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.093+0000 7ff9bcff9700 1 -- 172.21.15.195:0/2366864220 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9b400ce30 con 0x7ff9c00688a0 2022-06-27T11:52:12.109 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.093+0000 7ff9bcff9700 1 -- 172.21.15.195:0/2366864220 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9b40050d0 con 0x7ff9c00688a0 2022-06-27T11:52:12.109 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.093+0000 7ff9c6cce700 1 -- 172.21.15.195:0/2366864220 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9c0162af0 con 0x7ff9c00688a0 2022-06-27T11:52:12.109 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.093+0000 7ff9bcff9700 1 -- 172.21.15.195:0/2366864220 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9b4011590 con 0x7ff9c00688a0 2022-06-27T11:52:12.109 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.093+0000 7ff9c6cce700 1 -- 172.21.15.195:0/2366864220 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9c00c7560 con 0x7ff9c00688a0 2022-06-27T11:52:12.110 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.094+0000 7ff9c6cce700 1 -- 172.21.15.195:0/2366864220 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9c00722c0 con 0x7ff9c00688a0 2022-06-27T11:52:12.110 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.094+0000 7ff9bcff9700 1 -- 172.21.15.195:0/2366864220 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7ff9b4019030 con 0x7ff9c00688a0 2022-06-27T11:52:12.110 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.094+0000 7ff9bcff9700 1 --2- 172.21.15.195:0/2366864220 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff9a8063d00 0x7ff9a80661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:12.111 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.095+0000 7ff9bffff700 1 --2- 172.21.15.195:0/2366864220 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff9a8063d00 0x7ff9a80661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:12.111 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.095+0000 7ff9bcff9700 1 -- 172.21.15.195:0/2366864220 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6508+0+0 (secure 0 0 0) 0x7ff9b407fa60 con 0x7ff9c00688a0 2022-06-27T11:52:12.111 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.095+0000 7ff9bffff700 1 --2- 172.21.15.195:0/2366864220 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff9a8063d00 0x7ff9a80661b0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff9c00bc060 tx=0x7ff9ac007bf0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:52:12.112 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.099+0000 7ff9bcff9700 1 -- 172.21.15.195:0/2366864220 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7ff9b404e430 con 0x7ff9c00688a0 2022-06-27T11:52:12.301 INFO:tasks.workunit.client.0.smithi195.stdout:removed './236/22' 2022-06-27T11:52:12.302 INFO:tasks.workunit.client.0.smithi195.stdout:removed './236/25' 2022-06-27T11:52:12.302 INFO:tasks.workunit.client.0.smithi195.stdout:removed './236/30' 2022-06-27T11:52:12.303 INFO:tasks.workunit.client.0.smithi195.stdout:removed './236/33' 2022-06-27T11:52:12.303 INFO:tasks.workunit.client.0.smithi195.stdout:removed './236/23' 2022-06-27T11:52:12.303 INFO:tasks.workunit.client.0.smithi195.stdout:removed './236/35' 2022-06-27T11:52:12.303 INFO:tasks.workunit.client.0.smithi195.stdout:removed './236/26' 2022-06-27T11:52:12.304 INFO:tasks.workunit.client.0.smithi195.stdout:removed './236/37' 2022-06-27T11:52:12.304 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './236' 2022-06-27T11:52:12.304 INFO:tasks.workunit.client.0.smithi195.stdout:removed './237/21' 2022-06-27T11:52:12.304 INFO:tasks.workunit.client.0.smithi195.stdout:removed './237/27' 2022-06-27T11:52:12.305 INFO:tasks.workunit.client.0.smithi195.stdout:removed './237/31' 2022-06-27T11:52:12.305 INFO:tasks.workunit.client.0.smithi195.stdout:removed './237/36' 2022-06-27T11:52:12.305 INFO:tasks.workunit.client.0.smithi195.stdout:removed './237/20' 2022-06-27T11:52:12.306 INFO:tasks.workunit.client.0.smithi195.stdout:removed './237/39' 2022-06-27T11:52:12.306 INFO:tasks.workunit.client.0.smithi195.stdout:removed './237/34' 2022-06-27T11:52:12.307 INFO:tasks.workunit.client.0.smithi195.stdout:removed './237/38' 2022-06-27T11:52:12.307 INFO:tasks.workunit.client.0.smithi195.stdout:removed './237/28' 2022-06-27T11:52:12.308 INFO:tasks.workunit.client.0.smithi195.stdout:removed './237/24' 2022-06-27T11:52:12.308 INFO:tasks.workunit.client.0.smithi195.stdout:removed './237/29' 2022-06-27T11:52:12.308 INFO:tasks.workunit.client.0.smithi195.stdout:removed './237/32' 2022-06-27T11:52:12.309 INFO:tasks.workunit.client.0.smithi195.stdout:removed './237/22' 2022-06-27T11:52:12.309 INFO:tasks.workunit.client.0.smithi195.stdout:removed './237/25' 2022-06-27T11:52:12.310 INFO:tasks.workunit.client.0.smithi195.stdout:removed './237/30' 2022-06-27T11:52:12.310 INFO:tasks.workunit.client.0.smithi195.stdout:removed './237/33' 2022-06-27T11:52:12.310 INFO:tasks.workunit.client.0.smithi195.stdout:removed './237/23' 2022-06-27T11:52:12.310 INFO:tasks.workunit.client.0.smithi195.stdout:removed './237/35' 2022-06-27T11:52:12.310 INFO:tasks.workunit.client.0.smithi195.stdout:removed './237/26' 2022-06-27T11:52:12.311 INFO:tasks.workunit.client.0.smithi195.stdout:removed './237/37' 2022-06-27T11:52:12.311 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './237' 2022-06-27T11:52:12.311 INFO:tasks.workunit.client.0.smithi195.stdout:removed './238/21' 2022-06-27T11:52:12.312 INFO:tasks.workunit.client.0.smithi195.stdout:removed './238/27' 2022-06-27T11:52:12.312 INFO:tasks.workunit.client.0.smithi195.stdout:removed './238/31' 2022-06-27T11:52:12.312 INFO:tasks.workunit.client.0.smithi195.stdout:removed './238/36' 2022-06-27T11:52:12.313 INFO:tasks.workunit.client.0.smithi195.stdout:removed './238/20' 2022-06-27T11:52:12.313 INFO:tasks.workunit.client.0.smithi195.stdout:removed './238/39' 2022-06-27T11:52:12.313 INFO:tasks.workunit.client.0.smithi195.stdout:removed './238/34' 2022-06-27T11:52:12.313 INFO:tasks.workunit.client.0.smithi195.stdout:removed './238/38' 2022-06-27T11:52:12.314 INFO:tasks.workunit.client.0.smithi195.stdout:removed './238/28' 2022-06-27T11:52:12.314 INFO:tasks.workunit.client.0.smithi195.stdout:removed './238/24' 2022-06-27T11:52:12.314 INFO:tasks.workunit.client.0.smithi195.stdout:removed './238/29' 2022-06-27T11:52:12.314 INFO:tasks.workunit.client.0.smithi195.stdout:removed './238/32' 2022-06-27T11:52:12.315 INFO:tasks.workunit.client.0.smithi195.stdout:removed './238/22' 2022-06-27T11:52:12.315 INFO:tasks.workunit.client.0.smithi195.stdout:removed './238/25' 2022-06-27T11:52:12.315 INFO:tasks.workunit.client.0.smithi195.stdout:removed './238/30' 2022-06-27T11:52:12.316 INFO:tasks.workunit.client.0.smithi195.stdout:removed './238/33' 2022-06-27T11:52:12.316 INFO:tasks.workunit.client.0.smithi195.stdout:removed './238/23' 2022-06-27T11:52:12.316 INFO:tasks.workunit.client.0.smithi195.stdout:removed './238/35' 2022-06-27T11:52:12.316 INFO:tasks.workunit.client.0.smithi195.stdout:removed './238/26' 2022-06-27T11:52:12.317 INFO:tasks.workunit.client.0.smithi195.stdout:removed './238/37' 2022-06-27T11:52:12.317 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './238' 2022-06-27T11:52:12.317 INFO:tasks.workunit.client.0.smithi195.stdout:removed './239/21' 2022-06-27T11:52:12.317 INFO:tasks.workunit.client.0.smithi195.stdout:removed './239/27' 2022-06-27T11:52:12.317 INFO:tasks.workunit.client.0.smithi195.stdout:removed './239/31' 2022-06-27T11:52:12.318 INFO:tasks.workunit.client.0.smithi195.stdout:removed './239/36' 2022-06-27T11:52:12.318 INFO:tasks.workunit.client.0.smithi195.stdout:removed './239/20' 2022-06-27T11:52:12.318 INFO:tasks.workunit.client.0.smithi195.stdout:removed './239/39' 2022-06-27T11:52:12.318 INFO:tasks.workunit.client.0.smithi195.stdout:removed './239/34' 2022-06-27T11:52:12.319 INFO:tasks.workunit.client.0.smithi195.stdout:removed './239/38' 2022-06-27T11:52:12.320 INFO:tasks.workunit.client.0.smithi195.stdout:removed './239/28' 2022-06-27T11:52:12.320 INFO:tasks.workunit.client.0.smithi195.stdout:removed './239/24' 2022-06-27T11:52:12.320 INFO:tasks.workunit.client.0.smithi195.stdout:removed './239/29' 2022-06-27T11:52:12.321 INFO:tasks.workunit.client.0.smithi195.stdout:removed './239/32' 2022-06-27T11:52:12.321 INFO:tasks.workunit.client.0.smithi195.stdout:removed './239/22' 2022-06-27T11:52:12.321 INFO:tasks.workunit.client.0.smithi195.stdout:removed './239/25' 2022-06-27T11:52:12.321 INFO:tasks.workunit.client.0.smithi195.stdout:removed './239/30' 2022-06-27T11:52:12.322 INFO:tasks.workunit.client.0.smithi195.stdout:removed './239/33' 2022-06-27T11:52:12.322 INFO:tasks.workunit.client.0.smithi195.stdout:removed './239/23' 2022-06-27T11:52:12.322 INFO:tasks.workunit.client.0.smithi195.stdout:removed './239/35' 2022-06-27T11:52:12.322 INFO:tasks.workunit.client.0.smithi195.stdout:removed './239/26' 2022-06-27T11:52:12.323 INFO:tasks.workunit.client.0.smithi195.stdout:removed './239/37' 2022-06-27T11:52:12.323 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './239' 2022-06-27T11:52:12.323 INFO:tasks.workunit.client.0.smithi195.stdout:removed './240/21' 2022-06-27T11:52:12.323 INFO:tasks.workunit.client.0.smithi195.stdout:removed './240/27' 2022-06-27T11:52:12.324 INFO:tasks.workunit.client.0.smithi195.stdout:removed './240/31' 2022-06-27T11:52:12.324 INFO:tasks.workunit.client.0.smithi195.stdout:removed './240/36' 2022-06-27T11:52:12.324 INFO:tasks.workunit.client.0.smithi195.stdout:removed './240/20' 2022-06-27T11:52:12.324 INFO:tasks.workunit.client.0.smithi195.stdout:removed './240/39' 2022-06-27T11:52:12.325 INFO:tasks.workunit.client.0.smithi195.stdout:removed './240/34' 2022-06-27T11:52:12.325 INFO:tasks.workunit.client.0.smithi195.stdout:removed './240/38' 2022-06-27T11:52:12.325 INFO:tasks.workunit.client.0.smithi195.stdout:removed './240/28' 2022-06-27T11:52:12.326 INFO:tasks.workunit.client.0.smithi195.stdout:removed './240/24' 2022-06-27T11:52:12.326 INFO:tasks.workunit.client.0.smithi195.stdout:removed './240/29' 2022-06-27T11:52:12.326 INFO:tasks.workunit.client.0.smithi195.stdout:removed './240/32' 2022-06-27T11:52:12.326 INFO:tasks.workunit.client.0.smithi195.stdout:removed './240/22' 2022-06-27T11:52:12.327 INFO:tasks.workunit.client.0.smithi195.stdout:removed './240/25' 2022-06-27T11:52:12.327 INFO:tasks.workunit.client.0.smithi195.stdout:removed './240/30' 2022-06-27T11:52:12.327 INFO:tasks.workunit.client.0.smithi195.stdout:removed './240/33' 2022-06-27T11:52:12.328 INFO:tasks.workunit.client.0.smithi195.stdout:removed './240/23' 2022-06-27T11:52:12.328 INFO:tasks.workunit.client.0.smithi195.stdout:removed './240/35' 2022-06-27T11:52:12.328 INFO:tasks.workunit.client.0.smithi195.stdout:removed './240/26' 2022-06-27T11:52:12.329 INFO:tasks.workunit.client.0.smithi195.stdout:removed './240/37' 2022-06-27T11:52:12.329 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './240' 2022-06-27T11:52:12.330 INFO:tasks.workunit.client.0.smithi195.stdout:removed './241/21' 2022-06-27T11:52:12.331 INFO:tasks.workunit.client.0.smithi195.stdout:removed './241/27' 2022-06-27T11:52:12.331 INFO:tasks.workunit.client.0.smithi195.stdout:removed './241/31' 2022-06-27T11:52:12.331 INFO:tasks.workunit.client.0.smithi195.stdout:removed './241/36' 2022-06-27T11:52:12.331 INFO:tasks.workunit.client.0.smithi195.stdout:removed './241/20' 2022-06-27T11:52:12.332 INFO:tasks.workunit.client.0.smithi195.stdout:removed './241/39' 2022-06-27T11:52:12.332 INFO:tasks.workunit.client.0.smithi195.stdout:removed './241/34' 2022-06-27T11:52:12.332 INFO:tasks.workunit.client.0.smithi195.stdout:removed './241/38' 2022-06-27T11:52:12.332 INFO:tasks.workunit.client.0.smithi195.stdout:removed './241/28' 2022-06-27T11:52:12.333 INFO:tasks.workunit.client.0.smithi195.stdout:removed './241/24' 2022-06-27T11:52:12.333 INFO:tasks.workunit.client.0.smithi195.stdout:removed './241/29' 2022-06-27T11:52:12.333 INFO:tasks.workunit.client.0.smithi195.stdout:removed './241/32' 2022-06-27T11:52:12.334 INFO:tasks.workunit.client.0.smithi195.stdout:removed './241/22' 2022-06-27T11:52:12.334 INFO:tasks.workunit.client.0.smithi195.stdout:removed './241/25' 2022-06-27T11:52:12.334 INFO:tasks.workunit.client.0.smithi195.stdout:removed './241/30' 2022-06-27T11:52:12.334 INFO:tasks.workunit.client.0.smithi195.stdout:removed './241/33' 2022-06-27T11:52:12.335 INFO:tasks.workunit.client.0.smithi195.stdout:removed './241/23' 2022-06-27T11:52:12.335 INFO:tasks.workunit.client.0.smithi195.stdout:removed './241/35' 2022-06-27T11:52:12.335 INFO:tasks.workunit.client.0.smithi195.stdout:removed './241/26' 2022-06-27T11:52:12.335 INFO:tasks.workunit.client.0.smithi195.stdout:removed './241/37' 2022-06-27T11:52:12.336 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './241' 2022-06-27T11:52:12.336 INFO:tasks.workunit.client.0.smithi195.stdout:removed './242/21' 2022-06-27T11:52:12.336 INFO:tasks.workunit.client.0.smithi195.stdout:removed './242/27' 2022-06-27T11:52:12.336 INFO:tasks.workunit.client.0.smithi195.stdout:removed './242/31' 2022-06-27T11:52:12.337 INFO:tasks.workunit.client.0.smithi195.stdout:removed './242/36' 2022-06-27T11:52:12.337 INFO:tasks.workunit.client.0.smithi195.stdout:removed './242/20' 2022-06-27T11:52:12.337 INFO:tasks.workunit.client.0.smithi195.stdout:removed './242/39' 2022-06-27T11:52:12.337 INFO:tasks.workunit.client.0.smithi195.stdout:removed './242/34' 2022-06-27T11:52:12.338 INFO:tasks.workunit.client.0.smithi195.stdout:removed './242/38' 2022-06-27T11:52:12.338 INFO:tasks.workunit.client.0.smithi195.stdout:removed './242/28' 2022-06-27T11:52:12.338 INFO:tasks.workunit.client.0.smithi195.stdout:removed './242/24' 2022-06-27T11:52:12.338 INFO:tasks.workunit.client.0.smithi195.stdout:removed './242/29' 2022-06-27T11:52:12.339 INFO:tasks.workunit.client.0.smithi195.stdout:removed './242/32' 2022-06-27T11:52:12.339 INFO:tasks.workunit.client.0.smithi195.stdout:removed './242/22' 2022-06-27T11:52:12.339 INFO:tasks.workunit.client.0.smithi195.stdout:removed './242/25' 2022-06-27T11:52:12.339 INFO:tasks.workunit.client.0.smithi195.stdout:removed './242/30' 2022-06-27T11:52:12.340 INFO:tasks.workunit.client.0.smithi195.stdout:removed './242/33' 2022-06-27T11:52:12.340 INFO:tasks.workunit.client.0.smithi195.stdout:removed './242/23' 2022-06-27T11:52:12.340 INFO:tasks.workunit.client.0.smithi195.stdout:removed './242/35' 2022-06-27T11:52:12.340 INFO:tasks.workunit.client.0.smithi195.stdout:removed './242/26' 2022-06-27T11:52:12.341 INFO:tasks.workunit.client.0.smithi195.stdout:removed './242/37' 2022-06-27T11:52:12.342 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './242' 2022-06-27T11:52:12.342 INFO:tasks.workunit.client.0.smithi195.stdout:removed './243/21' 2022-06-27T11:52:12.342 INFO:tasks.workunit.client.0.smithi195.stdout:removed './243/27' 2022-06-27T11:52:12.342 INFO:tasks.workunit.client.0.smithi195.stdout:removed './243/31' 2022-06-27T11:52:12.343 INFO:tasks.workunit.client.0.smithi195.stdout:removed './243/36' 2022-06-27T11:52:12.343 INFO:tasks.workunit.client.0.smithi195.stdout:removed './243/20' 2022-06-27T11:52:12.343 INFO:tasks.workunit.client.0.smithi195.stdout:removed './243/39' 2022-06-27T11:52:12.343 INFO:tasks.workunit.client.0.smithi195.stdout:removed './243/34' 2022-06-27T11:52:12.344 INFO:tasks.workunit.client.0.smithi195.stdout:removed './243/38' 2022-06-27T11:52:12.344 INFO:tasks.workunit.client.0.smithi195.stdout:removed './243/28' 2022-06-27T11:52:12.344 INFO:tasks.workunit.client.0.smithi195.stdout:removed './243/24' 2022-06-27T11:52:12.344 INFO:tasks.workunit.client.0.smithi195.stdout:removed './243/29' 2022-06-27T11:52:12.345 INFO:tasks.workunit.client.0.smithi195.stdout:removed './243/32' 2022-06-27T11:52:12.345 INFO:tasks.workunit.client.0.smithi195.stdout:removed './243/22' 2022-06-27T11:52:12.345 INFO:tasks.workunit.client.0.smithi195.stdout:removed './243/25' 2022-06-27T11:52:12.346 INFO:tasks.workunit.client.0.smithi195.stdout:removed './243/30' 2022-06-27T11:52:12.346 INFO:tasks.workunit.client.0.smithi195.stdout:removed './243/33' 2022-06-27T11:52:12.346 INFO:tasks.workunit.client.0.smithi195.stdout:removed './243/23' 2022-06-27T11:52:12.346 INFO:tasks.workunit.client.0.smithi195.stdout:removed './243/35' 2022-06-27T11:52:12.347 INFO:tasks.workunit.client.0.smithi195.stdout:removed './243/26' 2022-06-27T11:52:12.347 INFO:tasks.workunit.client.0.smithi195.stdout:removed './243/37' 2022-06-27T11:52:12.347 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './243' 2022-06-27T11:52:12.347 INFO:tasks.workunit.client.0.smithi195.stdout:removed './244/21' 2022-06-27T11:52:12.348 INFO:tasks.workunit.client.0.smithi195.stdout:removed './244/27' 2022-06-27T11:52:12.348 INFO:tasks.workunit.client.0.smithi195.stdout:removed './244/31' 2022-06-27T11:52:12.348 INFO:tasks.workunit.client.0.smithi195.stdout:removed './244/36' 2022-06-27T11:52:12.348 INFO:tasks.workunit.client.0.smithi195.stdout:removed './244/20' 2022-06-27T11:52:12.349 INFO:tasks.workunit.client.0.smithi195.stdout:removed './244/39' 2022-06-27T11:52:12.349 INFO:tasks.workunit.client.0.smithi195.stdout:removed './244/34' 2022-06-27T11:52:12.349 INFO:tasks.workunit.client.0.smithi195.stdout:removed './244/38' 2022-06-27T11:52:12.349 INFO:tasks.workunit.client.0.smithi195.stdout:removed './244/28' 2022-06-27T11:52:12.349 INFO:tasks.workunit.client.0.smithi195.stdout:removed './244/24' 2022-06-27T11:52:12.350 INFO:tasks.workunit.client.0.smithi195.stdout:removed './244/29' 2022-06-27T11:52:12.350 INFO:tasks.workunit.client.0.smithi195.stdout:removed './244/32' 2022-06-27T11:52:12.350 INFO:tasks.workunit.client.0.smithi195.stdout:removed './244/22' 2022-06-27T11:52:12.350 INFO:tasks.workunit.client.0.smithi195.stdout:removed './244/25' 2022-06-27T11:52:12.351 INFO:tasks.workunit.client.0.smithi195.stdout:removed './244/30' 2022-06-27T11:52:12.351 INFO:tasks.workunit.client.0.smithi195.stdout:removed './244/33' 2022-06-27T11:52:12.352 INFO:tasks.workunit.client.0.smithi195.stdout:removed './244/23' 2022-06-27T11:52:12.352 INFO:tasks.workunit.client.0.smithi195.stdout:removed './244/35' 2022-06-27T11:52:12.352 INFO:tasks.workunit.client.0.smithi195.stdout:removed './244/26' 2022-06-27T11:52:12.353 INFO:tasks.workunit.client.0.smithi195.stdout:removed './244/37' 2022-06-27T11:52:12.353 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './244' 2022-06-27T11:52:12.353 INFO:tasks.workunit.client.0.smithi195.stdout:removed './245/21' 2022-06-27T11:52:12.354 INFO:tasks.workunit.client.0.smithi195.stdout:removed './245/27' 2022-06-27T11:52:12.354 INFO:tasks.workunit.client.0.smithi195.stdout:removed './245/31' 2022-06-27T11:52:12.354 INFO:tasks.workunit.client.0.smithi195.stdout:removed './245/36' 2022-06-27T11:52:12.354 INFO:tasks.workunit.client.0.smithi195.stdout:removed './245/20' 2022-06-27T11:52:12.354 INFO:tasks.workunit.client.0.smithi195.stdout:removed './245/39' 2022-06-27T11:52:12.355 INFO:tasks.workunit.client.0.smithi195.stdout:removed './245/34' 2022-06-27T11:52:12.355 INFO:tasks.workunit.client.0.smithi195.stdout:removed './245/38' 2022-06-27T11:52:12.355 INFO:tasks.workunit.client.0.smithi195.stdout:removed './245/28' 2022-06-27T11:52:12.355 INFO:tasks.workunit.client.0.smithi195.stdout:removed './245/24' 2022-06-27T11:52:12.356 INFO:tasks.workunit.client.0.smithi195.stdout:removed './245/29' 2022-06-27T11:52:12.356 INFO:tasks.workunit.client.0.smithi195.stdout:removed './245/32' 2022-06-27T11:52:12.356 INFO:tasks.workunit.client.0.smithi195.stdout:removed './245/22' 2022-06-27T11:52:12.357 INFO:tasks.workunit.client.0.smithi195.stdout:removed './245/25' 2022-06-27T11:52:12.357 INFO:tasks.workunit.client.0.smithi195.stdout:removed './245/30' 2022-06-27T11:52:12.358 INFO:tasks.workunit.client.0.smithi195.stdout:removed './245/33' 2022-06-27T11:52:12.359 INFO:tasks.workunit.client.0.smithi195.stdout:removed './245/23' 2022-06-27T11:52:12.359 INFO:tasks.workunit.client.0.smithi195.stdout:removed './245/35' 2022-06-27T11:52:12.360 INFO:tasks.workunit.client.0.smithi195.stdout:removed './245/26' 2022-06-27T11:52:12.360 INFO:tasks.workunit.client.0.smithi195.stdout:removed './245/37' 2022-06-27T11:52:12.361 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './245' 2022-06-27T11:52:12.361 INFO:tasks.workunit.client.0.smithi195.stdout:removed './246/21' 2022-06-27T11:52:12.362 INFO:tasks.workunit.client.0.smithi195.stdout:removed './246/27' 2022-06-27T11:52:12.362 INFO:tasks.workunit.client.0.smithi195.stdout:removed './246/31' 2022-06-27T11:52:12.362 INFO:tasks.workunit.client.0.smithi195.stdout:removed './246/36' 2022-06-27T11:52:12.363 INFO:tasks.workunit.client.0.smithi195.stdout:removed './246/20' 2022-06-27T11:52:12.363 INFO:tasks.workunit.client.0.smithi195.stdout:removed './246/39' 2022-06-27T11:52:12.363 INFO:tasks.workunit.client.0.smithi195.stdout:removed './246/34' 2022-06-27T11:52:12.363 INFO:tasks.workunit.client.0.smithi195.stdout:removed './246/38' 2022-06-27T11:52:12.364 INFO:tasks.workunit.client.0.smithi195.stdout:removed './246/28' 2022-06-27T11:52:12.364 INFO:tasks.workunit.client.0.smithi195.stdout:removed './246/24' 2022-06-27T11:52:12.364 INFO:tasks.workunit.client.0.smithi195.stdout:removed './246/29' 2022-06-27T11:52:12.365 INFO:tasks.workunit.client.0.smithi195.stdout:removed './246/32' 2022-06-27T11:52:12.365 INFO:tasks.workunit.client.0.smithi195.stdout:removed './246/22' 2022-06-27T11:52:12.366 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.330+0000 7ff9c6cce700 1 -- 172.21.15.195:0/2366864220 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff9c00c3ba0 con 0x7ff9c00688a0 2022-06-27T11:52:12.367 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.330+0000 7ff9bcff9700 1 -- 172.21.15.195:0/2366864220 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+6128 (secure 0 0 0) 0x7ff9b4017090 con 0x7ff9c00688a0 2022-06-27T11:52:12.367 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:52:12.368 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4760,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/1026219246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1026219246},{"type":"v1","addr":"172.21.15.202:6833","nonce":1026219246}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:51:33.013206+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4849},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4849":{"gid":4849,"name":"d","rank":0,"incarnation":15,"state":"up:active","state_seq":19,"addr":"172.21.15.202:6835/3271891211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":3271891211},{"type":"v1","addr":"172.21.15.202:6835","nonce":3271891211}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:52:12.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.332+0000 7ff9c6cce700 1 -- 172.21.15.195:0/2366864220 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff9a8063d00 msgr2=0x7ff9a80661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:12.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.332+0000 7ff9c6cce700 1 --2- 172.21.15.195:0/2366864220 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff9a8063d00 0x7ff9a80661b0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff9c00bc060 tx=0x7ff9ac007bf0 comp rx=0 tx=0).stop 2022-06-27T11:52:12.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.332+0000 7ff9c6cce700 1 -- 172.21.15.195:0/2366864220 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff9c00688a0 msgr2=0x7ff9c0151c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:12.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.332+0000 7ff9c6cce700 1 --2- 172.21.15.195:0/2366864220 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff9c00688a0 0x7ff9c0151c70 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7ff9b400c4c0 tx=0x7ff9b4007d60 comp rx=0 tx=0).stop 2022-06-27T11:52:12.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.332+0000 7ff9c6cce700 1 -- 172.21.15.195:0/2366864220 shutdown_connections 2022-06-27T11:52:12.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.332+0000 7ff9c6cce700 1 --2- 172.21.15.195:0/2366864220 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff9a8063d00 0x7ff9a80661b0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:12.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.332+0000 7ff9c6cce700 1 --2- 172.21.15.195:0/2366864220 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff9c00688a0 0x7ff9c0151c70 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:12.372 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.332+0000 7ff9c6cce700 1 --2- 172.21.15.195:0/2366864220 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff9c0156100 0x7ff9c01652f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:12.372 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.332+0000 7ff9c6cce700 1 --2- 172.21.15.195:0/2366864220 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff9c006a440 0x7ff9c0153ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:12.372 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.332+0000 7ff9c6cce700 1 -- 172.21.15.195:0/2366864220 >> 172.21.15.195:0/2366864220 conn(0x7ff9c00c9b00 msgr2=0x7ff9c014fd50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:52:12.372 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.332+0000 7ff9c6cce700 1 -- 172.21.15.195:0/2366864220 shutdown_connections 2022-06-27T11:52:12.373 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.332+0000 7ff9c6cce700 1 -- 172.21.15.195:0/2366864220 wait complete. 2022-06-27T11:52:12.373 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 19 2022-06-27T11:52:12.374 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.d (rank=0) 2022-06-27T11:52:12.374 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2022-06-27T11:52:12.375 INFO:teuthology.orchestra.run:waiting for 300 2022-06-27T11:52:12.386 INFO:tasks.ceph.mds.d:Stopped 2022-06-27T11:52:12.387 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:52:12.512 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.513+0000 7f54569b3700 1 Processor -- start 2022-06-27T11:52:12.513 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.513+0000 7f54569b3700 1 -- start start 2022-06-27T11:52:12.519 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.514+0000 7f54569b3700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f545006fef0 0x7f54501570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:12.519 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.514+0000 7f54569b3700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f54500c34a0 0x7f54500c2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:12.519 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.514+0000 7f54569b3700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f545006fba0 con 0x7f545006e850 2022-06-27T11:52:12.520 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.514+0000 7f54569b3700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54501575e0 con 0x7f545006fef0 2022-06-27T11:52:12.520 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.514+0000 7f54569b3700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f545006f5e0 con 0x7f54500c34a0 2022-06-27T11:52:12.520 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.514+0000 7f544f7fe700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f545006fef0 0x7f54501570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:12.520 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.514+0000 7f544ffff700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f545006e850 0x7f5450069de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:49670/0 (socket says 172.21.15.195:49670) 2022-06-27T11:52:12.521 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.514+0000 7f544ffff700 1 -- 172.21.15.195:0/45591760 learned_addr learned my addr 172.21.15.195:0/45591760 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:52:12.521 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.514+0000 7f5454f50700 1 --2- 172.21.15.195:0/45591760 >> v2:172.21.15.202:3300/0 conn(0x7f54500c34a0 0x7f54500c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:12.521 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.514+0000 7f544f7fe700 1 -- 172.21.15.195:0/45591760 >> v2:172.21.15.202:3300/0 conn(0x7f54500c34a0 msgr2=0x7f54500c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:52:12.522 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.514+0000 7f544f7fe700 1 --2- 172.21.15.195:0/45591760 >> v2:172.21.15.202:3300/0 conn(0x7f54500c34a0 0x7f54500c2920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:12.522 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.514+0000 7f544f7fe700 1 -- 172.21.15.195:0/45591760 >> v1:172.21.15.195:6789/0 conn(0x7f545006e850 legacy=0x7f5450069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:12.523 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.514+0000 7f544f7fe700 1 -- 172.21.15.195:0/45591760 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f545006f8c0 con 0x7f545006fef0 2022-06-27T11:52:12.523 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.514+0000 7f544effd700 1 -- 172.21.15.195:0/45591760 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 677486619 0 0) 0x7f545006fba0 con 0x7f545006e850 2022-06-27T11:52:12.523 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.514+0000 7f5454f50700 1 --2- 172.21.15.195:0/45591760 >> v2:172.21.15.202:3300/0 conn(0x7f54500c34a0 0x7f54500c2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T11:52:12.523 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.514+0000 7f544f7fe700 1 --2- 172.21.15.195:0/45591760 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f545006fef0 0x7f54501570a0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f54500c1420 tx=0x7f5440009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fc99f74dce36644 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:52:12.524 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.514+0000 7f544effd700 1 -- 172.21.15.195:0/45591760 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5440011630 con 0x7f545006fef0 2022-06-27T11:52:12.524 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.515+0000 7f544effd700 1 -- 172.21.15.195:0/45591760 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5440011790 con 0x7f545006fef0 2022-06-27T11:52:12.525 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.515+0000 7f544effd700 1 -- 172.21.15.195:0/45591760 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5440011b30 con 0x7f545006fef0 2022-06-27T11:52:12.525 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.515+0000 7f54569b3700 1 -- 172.21.15.195:0/45591760 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f545006fef0 msgr2=0x7f54501570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:12.526 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.515+0000 7f54569b3700 1 --2- 172.21.15.195:0/45591760 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f545006fef0 0x7f54501570a0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f54500c1420 tx=0x7f5440009060 comp rx=0 tx=0).stop 2022-06-27T11:52:12.526 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.515+0000 7f54569b3700 1 -- 172.21.15.195:0/45591760 shutdown_connections 2022-06-27T11:52:12.526 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.515+0000 7f54569b3700 1 --2- 172.21.15.195:0/45591760 >> v2:172.21.15.202:3300/0 conn(0x7f54500c34a0 0x7f54500c2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:12.526 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.515+0000 7f54569b3700 1 --2- 172.21.15.195:0/45591760 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f545006fef0 0x7f54501570a0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:12.527 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.515+0000 7f54569b3700 1 -- 172.21.15.195:0/45591760 >> 172.21.15.195:0/45591760 conn(0x7f54500c6350 msgr2=0x7f54500c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:52:12.527 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.515+0000 7f54569b3700 1 -- 172.21.15.195:0/45591760 shutdown_connections 2022-06-27T11:52:12.527 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.515+0000 7f54569b3700 1 -- 172.21.15.195:0/45591760 wait complete. 2022-06-27T11:52:12.527 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.516+0000 7f54569b3700 1 Processor -- start 2022-06-27T11:52:12.528 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.516+0000 7f54569b3700 1 -- start start 2022-06-27T11:52:12.528 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.517+0000 7f54569b3700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f545006e850 0x7f5450153590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:12.528 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.517+0000 7f54569b3700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f54500c34a0 0x7f5450156c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:12.528 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.517+0000 7f54569b3700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f54500702d0 0x7f5450167690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:12.529 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.517+0000 7f54569b3700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5450164cb0 con 0x7f545006e850 2022-06-27T11:52:12.529 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.517+0000 7f54569b3700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5450154290 con 0x7f54500c34a0 2022-06-27T11:52:12.529 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.517+0000 7f54569b3700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5450163e70 con 0x7f54500702d0 2022-06-27T11:52:12.530 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.517+0000 7f544ffff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f545006e850 0x7f5450153590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:12.530 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.517+0000 7f544ffff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f545006e850 0x7f5450153590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:43488/0 (socket says 172.21.15.195:43488) 2022-06-27T11:52:12.530 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.517+0000 7f544ffff700 1 -- 172.21.15.195:0/4231845663 learned_addr learned my addr 172.21.15.195:0/4231845663 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:52:12.531 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.517+0000 7f544f7fe700 1 --2- 172.21.15.195:0/4231845663 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f54500c34a0 0x7f5450156c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:12.531 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.517+0000 7f5454f50700 1 --2- 172.21.15.195:0/4231845663 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f54500702d0 0x7f5450167690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:12.531 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.517+0000 7f544ffff700 1 -- 172.21.15.195:0/4231845663 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f54500702d0 msgr2=0x7f5450167690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:12.532 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.517+0000 7f544ffff700 1 --2- 172.21.15.195:0/4231845663 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f54500702d0 0x7f5450167690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:12.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.517+0000 7f544ffff700 1 -- 172.21.15.195:0/4231845663 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f54500c34a0 msgr2=0x7f5450156c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:12.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.517+0000 7f544ffff700 1 --2- 172.21.15.195:0/4231845663 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f54500c34a0 0x7f5450156c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:12.534 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.517+0000 7f544ffff700 1 -- 172.21.15.195:0/4231845663 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54400092e0 con 0x7f545006e850 2022-06-27T11:52:12.534 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.518+0000 7f544ffff700 1 --2- 172.21.15.195:0/4231845663 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f545006e850 0x7f5450153590 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f543800c490 tx=0x7f5438007dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:52:12.535 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.518+0000 7f5447fff700 1 -- 172.21.15.195:0/4231845663 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5438007430 con 0x7f545006e850 2022-06-27T11:52:12.535 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.518+0000 7f5447fff700 1 -- 172.21.15.195:0/4231845663 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5438007590 con 0x7f545006e850 2022-06-27T11:52:12.536 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.518+0000 7f54569b3700 1 -- 172.21.15.195:0/4231845663 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5450164130 con 0x7f545006e850 2022-06-27T11:52:12.536 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.518+0000 7f5447fff700 1 -- 172.21.15.195:0/4231845663 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5438004920 con 0x7f545006e850 2022-06-27T11:52:12.537 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.518+0000 7f54569b3700 1 -- 172.21.15.195:0/4231845663 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54501646f0 con 0x7f545006e850 2022-06-27T11:52:12.537 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.519+0000 7f54569b3700 1 -- 172.21.15.195:0/4231845663 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54500722c0 con 0x7f545006e850 2022-06-27T11:52:12.538 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.519+0000 7f5447fff700 1 -- 172.21.15.195:0/4231845663 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f5438010030 con 0x7f545006e850 2022-06-27T11:52:12.538 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.519+0000 7f5447fff700 1 --2- 172.21.15.195:0/4231845663 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5434063d00 0x7f54340661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:12.539 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.520+0000 7f5447fff700 1 -- 172.21.15.195:0/4231845663 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6508+0+0 (secure 0 0 0) 0x7f5438080590 con 0x7f545006e850 2022-06-27T11:52:12.539 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.520+0000 7f544f7fe700 1 --2- 172.21.15.195:0/4231845663 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5434063d00 0x7f54340661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:12.541 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.520+0000 7f544f7fe700 1 --2- 172.21.15.195:0/4231845663 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5434063d00 0x7f54340661b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f5440011fd0 tx=0x7f5440017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:52:12.542 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.524+0000 7f5447fff700 1 -- 172.21.15.195:0/4231845663 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f543804f040 con 0x7f545006e850 2022-06-27T11:52:12.746 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.747+0000 7f54569b3700 1 -- 172.21.15.195:0/4231845663 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f545006f240 con 0x7f545006e850 2022-06-27T11:52:12.747 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.747+0000 7f5447fff700 1 -- 172.21.15.195:0/4231845663 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+6128 (secure 0 0 0) 0x7f5438052660 con 0x7f545006e850 2022-06-27T11:52:12.748 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:52:12.749 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4760,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/1026219246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1026219246},{"type":"v1","addr":"172.21.15.202:6833","nonce":1026219246}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:51:33.013206+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4849},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4849":{"gid":4849,"name":"d","rank":0,"incarnation":15,"state":"up:active","state_seq":19,"addr":"172.21.15.202:6835/3271891211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":3271891211},{"type":"v1","addr":"172.21.15.202:6835","nonce":3271891211}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:52:12.752 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.749+0000 7f54569b3700 1 -- 172.21.15.195:0/4231845663 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5434063d00 msgr2=0x7f54340661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:12.753 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.749+0000 7f54569b3700 1 --2- 172.21.15.195:0/4231845663 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5434063d00 0x7f54340661b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f5440011fd0 tx=0x7f5440017000 comp rx=0 tx=0).stop 2022-06-27T11:52:12.753 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.749+0000 7f54569b3700 1 -- 172.21.15.195:0/4231845663 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f545006e850 msgr2=0x7f5450153590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:12.754 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.749+0000 7f54569b3700 1 --2- 172.21.15.195:0/4231845663 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f545006e850 0x7f5450153590 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f543800c490 tx=0x7f5438007dd0 comp rx=0 tx=0).stop 2022-06-27T11:52:12.754 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.749+0000 7f54569b3700 1 -- 172.21.15.195:0/4231845663 shutdown_connections 2022-06-27T11:52:12.754 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.749+0000 7f54569b3700 1 --2- 172.21.15.195:0/4231845663 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5434063d00 0x7f54340661b0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:12.755 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.749+0000 7f54569b3700 1 --2- 172.21.15.195:0/4231845663 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f545006e850 0x7f5450153590 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:12.756 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.749+0000 7f54569b3700 1 --2- 172.21.15.195:0/4231845663 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f54500702d0 0x7f5450167690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:12.756 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.749+0000 7f54569b3700 1 --2- 172.21.15.195:0/4231845663 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f54500c34a0 0x7f5450156c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:12.757 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.749+0000 7f54569b3700 1 -- 172.21.15.195:0/4231845663 >> 172.21.15.195:0/4231845663 conn(0x7f54500c6350 msgr2=0x7f5450068040 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:52:12.757 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.750+0000 7f54569b3700 1 -- 172.21.15.195:0/4231845663 shutdown_connections 2022-06-27T11:52:12.757 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:12.750+0000 7f54569b3700 1 -- 172.21.15.195:0/4231845663 wait complete. 2022-06-27T11:52:12.758 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 19 2022-06-27T11:52:12.761 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-06-27T11:52:14.763 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:52:14.897 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:14.898+0000 7f7a964c6700 1 Processor -- start 2022-06-27T11:52:14.898 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:14.898+0000 7f7a964c6700 1 -- start start 2022-06-27T11:52:14.899 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:14.899+0000 7f7a964c6700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f7a900c7480 0x7f7a900c4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:14.900 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:14.899+0000 7f7a964c6700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f7a900683c0 0x7f7a90069350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:14.900 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:14.899+0000 7f7a964c6700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a901573b0 con 0x7f7a90068f80 2022-06-27T11:52:14.900 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:14.899+0000 7f7a964c6700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a90068840 con 0x7f7a900683c0 2022-06-27T11:52:14.900 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:14.899+0000 7f7a964c6700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a900c46c0 con 0x7f7a900c7480 2022-06-27T11:52:14.901 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:14.899+0000 7f7a94a63700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f7a900683c0 0x7f7a90069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:14.901 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:14.899+0000 7f7a8f7fe700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f7a900c7480 0x7f7a900c4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:14.901 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:14.899+0000 7f7a8ffff700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f7a90068f80 0x7f7a90156460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:49690/0 (socket says 172.21.15.195:49690) 2022-06-27T11:52:14.901 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:14.899+0000 7f7a94a63700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f7a900683c0 0x7f7a90069350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:43506/0 (socket says 172.21.15.195:43506) 2022-06-27T11:52:14.902 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:14.899+0000 7f7a8ffff700 1 -- 172.21.15.195:0/1321101513 learned_addr learned my addr 172.21.15.195:0/1321101513 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:52:14.902 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:14.899+0000 7f7a94a63700 1 -- 172.21.15.195:0/1321101513 >> v2:172.21.15.202:3300/0 conn(0x7f7a900c7480 msgr2=0x7f7a900c4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:52:14.902 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:14.899+0000 7f7a94a63700 1 --2- 172.21.15.195:0/1321101513 >> v2:172.21.15.202:3300/0 conn(0x7f7a900c7480 0x7f7a900c4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:14.902 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:14.899+0000 7f7a94a63700 1 -- 172.21.15.195:0/1321101513 >> v1:172.21.15.195:6789/0 conn(0x7f7a90068f80 legacy=0x7f7a90156460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:14.903 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:14.899+0000 7f7a94a63700 1 -- 172.21.15.195:0/1321101513 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a90156b70 con 0x7f7a900683c0 2022-06-27T11:52:14.903 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:14.899+0000 7f7a8f7fe700 1 --2- 172.21.15.195:0/1321101513 >> v2:172.21.15.202:3300/0 conn(0x7f7a900c7480 0x7f7a900c4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:52:14.903 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:14.899+0000 7f7a94a63700 1 --2- 172.21.15.195:0/1321101513 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7a900683c0 0x7f7a90069350 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f7a900c6430 tx=0x7f7a88009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f9f82b7c4005347 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:52:14.908 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:14.901+0000 7f7a8effd700 1 -- 172.21.15.195:0/1321101513 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a8800e070 con 0x7f7a900683c0 2022-06-27T11:52:14.908 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:14.901+0000 7f7a8effd700 1 -- 172.21.15.195:0/1321101513 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a8800bb60 con 0x7f7a900683c0 2022-06-27T11:52:14.908 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:14.901+0000 7f7a8effd700 1 -- 172.21.15.195:0/1321101513 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a88012430 con 0x7f7a900683c0 2022-06-27T11:52:14.909 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:14.901+0000 7f7a964c6700 1 -- 172.21.15.195:0/1321101513 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7a900683c0 msgr2=0x7f7a90069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:14.909 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:14.901+0000 7f7a964c6700 1 --2- 172.21.15.195:0/1321101513 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7a900683c0 0x7f7a90069350 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f7a900c6430 tx=0x7f7a88009780 comp rx=0 tx=0).stop 2022-06-27T11:52:14.909 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:14.902+0000 7f7a964c6700 1 -- 172.21.15.195:0/1321101513 shutdown_connections 2022-06-27T11:52:14.909 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:14.902+0000 7f7a964c6700 1 --2- 172.21.15.195:0/1321101513 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7a900683c0 0x7f7a90069350 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:14.910 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:14.902+0000 7f7a964c6700 1 --2- 172.21.15.195:0/1321101513 >> v2:172.21.15.202:3300/0 conn(0x7f7a900c7480 0x7f7a900c4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:14.910 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:14.902+0000 7f7a964c6700 1 -- 172.21.15.195:0/1321101513 >> 172.21.15.195:0/1321101513 conn(0x7f7a900ccbd0 msgr2=0x7f7a900ccfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:52:14.910 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:14.902+0000 7f7a964c6700 1 -- 172.21.15.195:0/1321101513 shutdown_connections 2022-06-27T11:52:14.910 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:14.902+0000 7f7a964c6700 1 -- 172.21.15.195:0/1321101513 wait complete. 2022-06-27T11:52:14.911 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:14.903+0000 7f7a964c6700 1 Processor -- start 2022-06-27T11:52:14.911 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:14.903+0000 7f7a964c6700 1 -- start start 2022-06-27T11:52:14.911 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:14.904+0000 7f7a964c6700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7a900683c0 0x7f7a900c96d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:14.911 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:14.904+0000 7f7a964c6700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7a90068f80 0x7f7a9015bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:14.912 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:14.904+0000 7f7a964c6700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7a900c7480 0x7f7a9015a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:14.912 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:14.904+0000 7f7a964c6700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a9016a630 con 0x7f7a900c7480 2022-06-27T11:52:14.912 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:14.904+0000 7f7a964c6700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a900b2ba0 con 0x7f7a900683c0 2022-06-27T11:52:14.912 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:14.904+0000 7f7a964c6700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a90169a20 con 0x7f7a90068f80 2022-06-27T11:52:14.913 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:14.904+0000 7f7a94a63700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7a900c7480 0x7f7a9015a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:14.913 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:14.904+0000 7f7a94a63700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7a900c7480 0x7f7a9015a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:43512/0 (socket says 172.21.15.195:43512) 2022-06-27T11:52:14.913 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:14.905+0000 7f7a94a63700 1 -- 172.21.15.195:0/1871532509 learned_addr learned my addr 172.21.15.195:0/1871532509 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:52:14.913 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:14.905+0000 7f7a8f7fe700 1 --2- 172.21.15.195:0/1871532509 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7a90068f80 0x7f7a9015bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:14.914 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:14.905+0000 7f7a8ffff700 1 --2- 172.21.15.195:0/1871532509 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7a900683c0 0x7f7a900c96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:14.914 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:14.905+0000 7f7a94a63700 1 -- 172.21.15.195:0/1871532509 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7a90068f80 msgr2=0x7f7a9015bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:14.914 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:14.905+0000 7f7a94a63700 1 --2- 172.21.15.195:0/1871532509 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7a90068f80 0x7f7a9015bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:14.915 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:14.905+0000 7f7a94a63700 1 -- 172.21.15.195:0/1871532509 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7a900683c0 msgr2=0x7f7a900c96d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:14.915 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:14.905+0000 7f7a94a63700 1 --2- 172.21.15.195:0/1871532509 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7a900683c0 0x7f7a900c96d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:14.915 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:14.905+0000 7f7a94a63700 1 -- 172.21.15.195:0/1871532509 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a840092e0 con 0x7f7a900c7480 2022-06-27T11:52:14.915 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:14.905+0000 7f7a94a63700 1 --2- 172.21.15.195:0/1871532509 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7a900c7480 0x7f7a9015a930 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f7a880097c0 tx=0x7f7a88019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:52:14.916 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:14.905+0000 7f7a77fff700 1 -- 172.21.15.195:0/1871532509 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a88003e20 con 0x7f7a900c7480 2022-06-27T11:52:14.916 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:14.905+0000 7f7a77fff700 1 -- 172.21.15.195:0/1871532509 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a88012a00 con 0x7f7a900c7480 2022-06-27T11:52:14.916 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:14.905+0000 7f7a964c6700 1 -- 172.21.15.195:0/1871532509 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a88009a20 con 0x7f7a900c7480 2022-06-27T11:52:14.916 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:14.905+0000 7f7a77fff700 1 -- 172.21.15.195:0/1871532509 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a88012dc0 con 0x7f7a900c7480 2022-06-27T11:52:14.917 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:14.905+0000 7f7a964c6700 1 -- 172.21.15.195:0/1871532509 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a90169fe0 con 0x7f7a900c7480 2022-06-27T11:52:14.917 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:14.906+0000 7f7a964c6700 1 -- 172.21.15.195:0/1871532509 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a900722c0 con 0x7f7a900c7480 2022-06-27T11:52:14.917 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:14.906+0000 7f7a77fff700 1 -- 172.21.15.195:0/1871532509 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f7a88004250 con 0x7f7a900c7480 2022-06-27T11:52:14.917 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:14.907+0000 7f7a77fff700 1 --2- 172.21.15.195:0/1871532509 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7a70063d00 0x7f7a700661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:14.918 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:14.907+0000 7f7a8ffff700 1 --2- 172.21.15.195:0/1871532509 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7a70063d00 0x7f7a700661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:14.918 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:14.907+0000 7f7a77fff700 1 -- 172.21.15.195:0/1871532509 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6508+0+0 (secure 0 0 0) 0x7f7a880832d0 con 0x7f7a900c7480 2022-06-27T11:52:14.918 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:14.907+0000 7f7a8ffff700 1 --2- 172.21.15.195:0/1871532509 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7a70063d00 0x7f7a700661b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f7a80007fd0 tx=0x7f7a80007d70 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:52:14.919 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:14.911+0000 7f7a77fff700 1 -- 172.21.15.195:0/1871532509 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f7a88051d00 con 0x7f7a900c7480 2022-06-27T11:52:15.146 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:15.146+0000 7f7a964c6700 1 -- 172.21.15.195:0/1871532509 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7a9006ac60 con 0x7f7a900c7480 2022-06-27T11:52:15.146 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:15.147+0000 7f7a77fff700 1 -- 172.21.15.195:0/1871532509 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+6128 (secure 0 0 0) 0x7f7a88055320 con 0x7f7a900c7480 2022-06-27T11:52:15.147 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:52:15.147 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4760,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/1026219246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1026219246},{"type":"v1","addr":"172.21.15.202:6833","nonce":1026219246}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:51:33.013206+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4849},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4849":{"gid":4849,"name":"d","rank":0,"incarnation":15,"state":"up:active","state_seq":19,"addr":"172.21.15.202:6835/3271891211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":3271891211},{"type":"v1","addr":"172.21.15.202:6835","nonce":3271891211}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:52:15.150 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:15.149+0000 7f7a964c6700 1 -- 172.21.15.195:0/1871532509 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7a70063d00 msgr2=0x7f7a700661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:15.150 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:15.149+0000 7f7a964c6700 1 --2- 172.21.15.195:0/1871532509 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7a70063d00 0x7f7a700661b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f7a80007fd0 tx=0x7f7a80007d70 comp rx=0 tx=0).stop 2022-06-27T11:52:15.151 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:15.149+0000 7f7a964c6700 1 -- 172.21.15.195:0/1871532509 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7a900c7480 msgr2=0x7f7a9015a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:15.151 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:15.149+0000 7f7a964c6700 1 --2- 172.21.15.195:0/1871532509 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7a900c7480 0x7f7a9015a930 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f7a880097c0 tx=0x7f7a88019000 comp rx=0 tx=0).stop 2022-06-27T11:52:15.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:15.149+0000 7f7a964c6700 1 -- 172.21.15.195:0/1871532509 shutdown_connections 2022-06-27T11:52:15.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:15.149+0000 7f7a964c6700 1 --2- 172.21.15.195:0/1871532509 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7a70063d00 0x7f7a700661b0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:15.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:15.149+0000 7f7a964c6700 1 --2- 172.21.15.195:0/1871532509 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7a900c7480 0x7f7a9015a930 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:15.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:15.149+0000 7f7a964c6700 1 --2- 172.21.15.195:0/1871532509 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7a900683c0 0x7f7a900c96d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:15.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:15.149+0000 7f7a964c6700 1 --2- 172.21.15.195:0/1871532509 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7a90068f80 0x7f7a9015bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:15.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:15.149+0000 7f7a964c6700 1 -- 172.21.15.195:0/1871532509 >> 172.21.15.195:0/1871532509 conn(0x7f7a900ccbd0 msgr2=0x7f7a90150040 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:52:15.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:15.149+0000 7f7a964c6700 1 -- 172.21.15.195:0/1871532509 shutdown_connections 2022-06-27T11:52:15.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:15.149+0000 7f7a964c6700 1 -- 172.21.15.195:0/1871532509 wait complete. 2022-06-27T11:52:15.154 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 19 2022-06-27T11:52:15.170 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-06-27T11:52:17.171 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:52:17.317 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.306+0000 7f1ece78c700 1 Processor -- start 2022-06-27T11:52:17.318 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.306+0000 7f1ece78c700 1 -- start start 2022-06-27T11:52:17.318 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.306+0000 7f1ece78c700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1ec8150950 0x7f1ec8156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:17.319 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.306+0000 7f1ece78c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1ec806f620 0x7f1ec806d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:17.321 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.306+0000 7f1ece78c700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ec80c76b0 con 0x7f1ec8150580 2022-06-27T11:52:17.321 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.306+0000 7f1ece78c700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ec8157430 con 0x7f1ec806f620 2022-06-27T11:52:17.322 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.306+0000 7f1ece78c700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec80c8250 con 0x7f1ec8150950 2022-06-27T11:52:17.323 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.307+0000 7f1eccd29700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1ec806f620 0x7f1ec806d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:17.324 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.307+0000 7f1eccd29700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1ec806f620 0x7f1ec806d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:43522/0 (socket says 172.21.15.195:43522) 2022-06-27T11:52:17.325 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.307+0000 7f1ec7fff700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f1ec8150580 0x7f1ec80c7b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:49708/0 (socket says 172.21.15.195:49708) 2022-06-27T11:52:17.325 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.307+0000 7f1eccd29700 1 -- 172.21.15.195:0/120278321 learned_addr learned my addr 172.21.15.195:0/120278321 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:52:17.326 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.307+0000 7f1ec77fe700 1 --2- 172.21.15.195:0/120278321 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1ec8150950 0x7f1ec8156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:17.326 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.307+0000 7f1eccd29700 1 -- 172.21.15.195:0/120278321 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1ec8150950 msgr2=0x7f1ec8156ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:52:17.327 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.307+0000 7f1eccd29700 1 --2- 172.21.15.195:0/120278321 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1ec8150950 0x7f1ec8156ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:17.327 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.307+0000 7f1eccd29700 1 -- 172.21.15.195:0/120278321 >> v1:172.21.15.195:6789/0 conn(0x7f1ec8150580 legacy=0x7f1ec80c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:17.328 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.307+0000 7f1eccd29700 1 -- 172.21.15.195:0/120278321 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ec80bad70 con 0x7f1ec806f620 2022-06-27T11:52:17.328 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.307+0000 7f1ec6ffd700 1 -- 172.21.15.195:0/120278321 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1947218952 0 0) 0x7f1ec80c76b0 con 0x7f1ec8150580 2022-06-27T11:52:17.329 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.307+0000 7f1eccd29700 1 --2- 172.21.15.195:0/120278321 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1ec806f620 0x7f1ec806d6a0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f1ec0009f20 tx=0x7f1ec0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17cee500c0fe0f5d server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:52:17.330 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.307+0000 7f1ec6ffd700 1 -- 172.21.15.195:0/120278321 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ec000e070 con 0x7f1ec806f620 2022-06-27T11:52:17.330 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.307+0000 7f1ec6ffd700 1 -- 172.21.15.195:0/120278321 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ec000ba80 con 0x7f1ec806f620 2022-06-27T11:52:17.331 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.307+0000 7f1ec6ffd700 1 -- 172.21.15.195:0/120278321 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ec00092e0 con 0x7f1ec806f620 2022-06-27T11:52:17.332 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.308+0000 7f1ece78c700 1 -- 172.21.15.195:0/120278321 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1ec806f620 msgr2=0x7f1ec806d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:17.333 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.308+0000 7f1ece78c700 1 --2- 172.21.15.195:0/120278321 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1ec806f620 0x7f1ec806d6a0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f1ec0009f20 tx=0x7f1ec0009840 comp rx=0 tx=0).stop 2022-06-27T11:52:17.334 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.308+0000 7f1ece78c700 1 -- 172.21.15.195:0/120278321 shutdown_connections 2022-06-27T11:52:17.335 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.308+0000 7f1ece78c700 1 --2- 172.21.15.195:0/120278321 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1ec806f620 0x7f1ec806d6a0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:17.336 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.308+0000 7f1ece78c700 1 --2- 172.21.15.195:0/120278321 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1ec8150950 0x7f1ec8156ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:17.337 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.308+0000 7f1ece78c700 1 -- 172.21.15.195:0/120278321 >> 172.21.15.195:0/120278321 conn(0x7f1ec8063af0 msgr2=0x7f1ec8063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:52:17.337 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.308+0000 7f1ece78c700 1 -- 172.21.15.195:0/120278321 shutdown_connections 2022-06-27T11:52:17.337 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.308+0000 7f1ece78c700 1 -- 172.21.15.195:0/120278321 wait complete. 2022-06-27T11:52:17.338 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.309+0000 7f1ece78c700 1 Processor -- start 2022-06-27T11:52:17.339 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.309+0000 7f1ece78c700 1 -- start start 2022-06-27T11:52:17.339 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.310+0000 7f1ece78c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1ec806f620 0x7f1ec8166f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:17.340 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.310+0000 7f1ece78c700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1ec8150580 0x7f1ec80c1d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:17.340 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.310+0000 7f1ece78c700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1ec8150950 0x7f1ec8069d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:17.341 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.310+0000 7f1ece78c700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec80bb850 con 0x7f1ec806f620 2022-06-27T11:52:17.341 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.310+0000 7f1ece78c700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec80c4370 con 0x7f1ec8150580 2022-06-27T11:52:17.341 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.310+0000 7f1ece78c700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec80c2250 con 0x7f1ec8150950 2022-06-27T11:52:17.342 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.310+0000 7f1ec7fff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1ec806f620 0x7f1ec8166f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:17.342 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.310+0000 7f1ec7fff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1ec806f620 0x7f1ec8166f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:43524/0 (socket says 172.21.15.195:43524) 2022-06-27T11:52:17.343 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.310+0000 7f1ec7fff700 1 -- 172.21.15.195:0/2106836062 learned_addr learned my addr 172.21.15.195:0/2106836062 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:52:17.343 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.310+0000 7f1eccd29700 1 --2- 172.21.15.195:0/2106836062 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1ec8150950 0x7f1ec8069d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:17.343 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.310+0000 7f1ec77fe700 1 --2- 172.21.15.195:0/2106836062 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1ec8150580 0x7f1ec80c1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:17.344 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.310+0000 7f1ec7fff700 1 -- 172.21.15.195:0/2106836062 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1ec8150950 msgr2=0x7f1ec8069d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:17.344 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.310+0000 7f1ec7fff700 1 --2- 172.21.15.195:0/2106836062 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1ec8150950 0x7f1ec8069d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:17.345 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.310+0000 7f1ec7fff700 1 -- 172.21.15.195:0/2106836062 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1ec8150580 msgr2=0x7f1ec80c1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:17.346 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.310+0000 7f1ec7fff700 1 --2- 172.21.15.195:0/2106836062 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1ec8150580 0x7f1ec80c1d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:17.346 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.310+0000 7f1ec7fff700 1 -- 172.21.15.195:0/2106836062 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ec0009ae0 con 0x7f1ec806f620 2022-06-27T11:52:17.347 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.310+0000 7f1ec7fff700 1 --2- 172.21.15.195:0/2106836062 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1ec806f620 0x7f1ec8166f70 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f1eb800c5c0 tx=0x7f1eb8007cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:52:17.347 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.311+0000 7f1eaffff700 1 -- 172.21.15.195:0/2106836062 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb8007360 con 0x7f1ec806f620 2022-06-27T11:52:17.347 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.311+0000 7f1eaffff700 1 -- 172.21.15.195:0/2106836062 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eb8007500 con 0x7f1ec806f620 2022-06-27T11:52:17.348 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.311+0000 7f1ece78c700 1 -- 172.21.15.195:0/2106836062 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ec80c3d60 con 0x7f1ec806f620 2022-06-27T11:52:17.348 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.311+0000 7f1eaffff700 1 -- 172.21.15.195:0/2106836062 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb8004940 con 0x7f1ec806f620 2022-06-27T11:52:17.349 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.311+0000 7f1ece78c700 1 -- 172.21.15.195:0/2106836062 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ec80bb570 con 0x7f1ec806f620 2022-06-27T11:52:17.349 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.312+0000 7f1ece78c700 1 -- 172.21.15.195:0/2106836062 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ec80c3bc0 con 0x7f1ec806f620 2022-06-27T11:52:17.350 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.312+0000 7f1eaffff700 1 -- 172.21.15.195:0/2106836062 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f1eb8010030 con 0x7f1ec806f620 2022-06-27T11:52:17.350 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.312+0000 7f1eaffff700 1 --2- 172.21.15.195:0/2106836062 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1ea8063d30 0x7f1ea80661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:17.351 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.312+0000 7f1eaffff700 1 -- 172.21.15.195:0/2106836062 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6508+0+0 (secure 0 0 0) 0x7f1eb8080470 con 0x7f1ec806f620 2022-06-27T11:52:17.351 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.313+0000 7f1ec77fe700 1 --2- 172.21.15.195:0/2106836062 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1ea8063d30 0x7f1ea80661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:17.352 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.313+0000 7f1ec77fe700 1 --2- 172.21.15.195:0/2106836062 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1ea8063d30 0x7f1ea80661e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc004560 tx=0x7f1ebc008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:52:17.352 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.317+0000 7f1eaffff700 1 -- 172.21.15.195:0/2106836062 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f1ec80c3bc0 con 0x7f1ec806f620 2022-06-27T11:52:17.541 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.540+0000 7f1ece78c700 1 -- 172.21.15.195:0/2106836062 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1ec80c3bc0 con 0x7f1ec806f620 2022-06-27T11:52:17.542 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.541+0000 7f1eaffff700 1 -- 172.21.15.195:0/2106836062 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+6128 (secure 0 0 0) 0x7f1ec80c3bc0 con 0x7f1ec806f620 2022-06-27T11:52:17.543 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:52:17.543 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4760,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/1026219246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1026219246},{"type":"v1","addr":"172.21.15.202:6833","nonce":1026219246}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:51:33.013206+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4849},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4849":{"gid":4849,"name":"d","rank":0,"incarnation":15,"state":"up:active","state_seq":19,"addr":"172.21.15.202:6835/3271891211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":3271891211},{"type":"v1","addr":"172.21.15.202:6835","nonce":3271891211}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:52:17.545 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.543+0000 7f1ece78c700 1 -- 172.21.15.195:0/2106836062 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1ea8063d30 msgr2=0x7f1ea80661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:17.546 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.543+0000 7f1ece78c700 1 --2- 172.21.15.195:0/2106836062 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1ea8063d30 0x7f1ea80661e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc004560 tx=0x7f1ebc008040 comp rx=0 tx=0).stop 2022-06-27T11:52:17.546 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.543+0000 7f1ece78c700 1 -- 172.21.15.195:0/2106836062 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1ec806f620 msgr2=0x7f1ec8166f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:17.547 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.543+0000 7f1ece78c700 1 --2- 172.21.15.195:0/2106836062 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1ec806f620 0x7f1ec8166f70 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f1eb800c5c0 tx=0x7f1eb8007cb0 comp rx=0 tx=0).stop 2022-06-27T11:52:17.548 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.543+0000 7f1ece78c700 1 -- 172.21.15.195:0/2106836062 shutdown_connections 2022-06-27T11:52:17.548 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.543+0000 7f1ece78c700 1 --2- 172.21.15.195:0/2106836062 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1ea8063d30 0x7f1ea80661e0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:17.548 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.543+0000 7f1ece78c700 1 --2- 172.21.15.195:0/2106836062 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1ec806f620 0x7f1ec8166f70 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:17.549 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.543+0000 7f1ece78c700 1 --2- 172.21.15.195:0/2106836062 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1ec8150950 0x7f1ec8069d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:17.549 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.543+0000 7f1ece78c700 1 --2- 172.21.15.195:0/2106836062 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1ec8150580 0x7f1ec80c1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:17.549 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.543+0000 7f1ece78c700 1 -- 172.21.15.195:0/2106836062 >> 172.21.15.195:0/2106836062 conn(0x7f1ec8063af0 msgr2=0x7f1ec814d620 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:52:17.549 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.543+0000 7f1ece78c700 1 -- 172.21.15.195:0/2106836062 shutdown_connections 2022-06-27T11:52:17.551 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:17.543+0000 7f1ece78c700 1 -- 172.21.15.195:0/2106836062 wait complete. 2022-06-27T11:52:17.551 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 19 2022-06-27T11:52:17.554 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-06-27T11:52:19.556 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:52:19.692 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.691+0000 7f9b250fc700 1 Processor -- start 2022-06-27T11:52:19.692 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.691+0000 7f9b250fc700 1 -- start start 2022-06-27T11:52:19.692 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.692+0000 7f9b250fc700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f9b2006a450 0x7f9b20063af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:19.693 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.692+0000 7f9b250fc700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b20064030 con 0x7f9b20150500 2022-06-27T11:52:19.693 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.692+0000 7f9b250fc700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b200c4160 con 0x7f9b201508d0 2022-06-27T11:52:19.693 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.692+0000 7f9b250fc700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b200c4440 con 0x7f9b2006a450 2022-06-27T11:52:19.693 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.692+0000 7f9b1e59c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f9b2006a450 0x7f9b20063af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:19.694 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.692+0000 7f9b1ed9d700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f9b20150500 0x7f9b200c48d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:49724/0 (socket says 172.21.15.195:49724) 2022-06-27T11:52:19.694 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.692+0000 7f9b1ed9d700 1 -- 172.21.15.195:0/42388032 learned_addr learned my addr 172.21.15.195:0/42388032 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:52:19.694 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.692+0000 7f9b1e59c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f9b2006a450 0x7f9b20063af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:43538/0 (socket says 172.21.15.195:43538) 2022-06-27T11:52:19.694 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.692+0000 7f9b1e59c700 1 -- 172.21.15.195:0/42388032 >> v1:172.21.15.202:6789/0 conn(0x7f9b201508d0 legacy=0x7f9b200bbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:19.695 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.692+0000 7f9b1e59c700 1 -- 172.21.15.195:0/42388032 >> v1:172.21.15.195:6789/0 conn(0x7f9b20150500 legacy=0x7f9b200c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:19.695 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.692+0000 7f9b1e59c700 1 -- 172.21.15.195:0/42388032 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b2015a800 con 0x7f9b2006a450 2022-06-27T11:52:19.695 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.692+0000 7f9b1dd9b700 1 -- 172.21.15.195:0/42388032 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1642498190 0 0) 0x7f9b10001290 con 0x7f9b20150500 2022-06-27T11:52:19.695 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.692+0000 7f9b1e59c700 1 --2- 172.21.15.195:0/42388032 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9b2006a450 0x7f9b20063af0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f9b140097a0 tx=0x7f9b14009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0661f029f66b4f server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:52:19.696 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.692+0000 7f9b1dd9b700 1 -- 172.21.15.195:0/42388032 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b14011630 con 0x7f9b2006a450 2022-06-27T11:52:19.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.692+0000 7f9b1dd9b700 1 -- 172.21.15.195:0/42388032 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b14011790 con 0x7f9b2006a450 2022-06-27T11:52:19.705 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.692+0000 7f9b1dd9b700 1 -- 172.21.15.195:0/42388032 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b14011b30 con 0x7f9b2006a450 2022-06-27T11:52:19.705 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.693+0000 7f9b250fc700 1 -- 172.21.15.195:0/42388032 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9b2006a450 msgr2=0x7f9b20063af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:19.705 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.693+0000 7f9b250fc700 1 --2- 172.21.15.195:0/42388032 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9b2006a450 0x7f9b20063af0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f9b140097a0 tx=0x7f9b14009060 comp rx=0 tx=0).stop 2022-06-27T11:52:19.705 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.693+0000 7f9b250fc700 1 -- 172.21.15.195:0/42388032 shutdown_connections 2022-06-27T11:52:19.706 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.693+0000 7f9b250fc700 1 --2- 172.21.15.195:0/42388032 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9b2006a450 0x7f9b20063af0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:19.706 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.693+0000 7f9b250fc700 1 -- 172.21.15.195:0/42388032 >> 172.21.15.195:0/42388032 conn(0x7f9b200c9b00 msgr2=0x7f9b200c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:52:19.706 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.693+0000 7f9b250fc700 1 -- 172.21.15.195:0/42388032 shutdown_connections 2022-06-27T11:52:19.706 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.693+0000 7f9b250fc700 1 -- 172.21.15.195:0/42388032 wait complete. 2022-06-27T11:52:19.707 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.694+0000 7f9b250fc700 1 Processor -- start 2022-06-27T11:52:19.707 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.694+0000 7f9b250fc700 1 -- start start 2022-06-27T11:52:19.707 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.695+0000 7f9b250fc700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9b2006a450 0x7f9b20159490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:19.708 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.695+0000 7f9b250fc700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9b20150500 0x7f9b200be8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:19.708 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.695+0000 7f9b250fc700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9b201508d0 0x7f9b2006de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:19.708 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.695+0000 7f9b250fc700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b2016a8e0 con 0x7f9b2006a450 2022-06-27T11:52:19.708 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.695+0000 7f9b250fc700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b20159000 con 0x7f9b20150500 2022-06-27T11:52:19.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.695+0000 7f9b250fc700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b2006e8a0 con 0x7f9b201508d0 2022-06-27T11:52:19.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.695+0000 7f9b1ed9d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9b2006a450 0x7f9b20159490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:19.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.695+0000 7f9b1ed9d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9b2006a450 0x7f9b20159490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:43542/0 (socket says 172.21.15.195:43542) 2022-06-27T11:52:19.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.695+0000 7f9b1ed9d700 1 -- 172.21.15.195:0/3910499130 learned_addr learned my addr 172.21.15.195:0/3910499130 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:52:19.710 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.695+0000 7f9b1e59c700 1 --2- 172.21.15.195:0/3910499130 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9b20150500 0x7f9b200be8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:19.710 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.695+0000 7f9b1f59e700 1 --2- 172.21.15.195:0/3910499130 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9b201508d0 0x7f9b2006de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:19.710 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.695+0000 7f9b1ed9d700 1 -- 172.21.15.195:0/3910499130 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9b201508d0 msgr2=0x7f9b2006de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:19.711 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.695+0000 7f9b1ed9d700 1 --2- 172.21.15.195:0/3910499130 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9b201508d0 0x7f9b2006de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:19.712 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.695+0000 7f9b1ed9d700 1 -- 172.21.15.195:0/3910499130 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9b20150500 msgr2=0x7f9b200be8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:19.712 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.695+0000 7f9b1ed9d700 1 --2- 172.21.15.195:0/3910499130 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9b20150500 0x7f9b200be8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:19.712 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.695+0000 7f9b1ed9d700 1 -- 172.21.15.195:0/3910499130 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b140092e0 con 0x7f9b2006a450 2022-06-27T11:52:19.712 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.696+0000 7f9b1ed9d700 1 --2- 172.21.15.195:0/3910499130 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9b2006a450 0x7f9b20159490 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f9b1000c540 tx=0x7f9b10001830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:52:19.713 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.696+0000 7f9b06ffd700 1 -- 172.21.15.195:0/3910499130 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b1000ce70 con 0x7f9b2006a450 2022-06-27T11:52:19.713 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.696+0000 7f9b06ffd700 1 -- 172.21.15.195:0/3910499130 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b100072e0 con 0x7f9b2006a450 2022-06-27T11:52:19.713 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.696+0000 7f9b250fc700 1 -- 172.21.15.195:0/3910499130 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b200bcf50 con 0x7f9b2006a450 2022-06-27T11:52:19.713 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.696+0000 7f9b06ffd700 1 -- 172.21.15.195:0/3910499130 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b100056b0 con 0x7f9b2006a450 2022-06-27T11:52:19.713 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.696+0000 7f9b250fc700 1 -- 172.21.15.195:0/3910499130 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b20158c90 con 0x7f9b2006a450 2022-06-27T11:52:19.714 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.697+0000 7f9b250fc700 1 -- 172.21.15.195:0/3910499130 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b201592e0 con 0x7f9b2006a450 2022-06-27T11:52:19.714 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.697+0000 7f9b06ffd700 1 -- 172.21.15.195:0/3910499130 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f9b10019030 con 0x7f9b2006a450 2022-06-27T11:52:19.714 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.698+0000 7f9b06ffd700 1 --2- 172.21.15.195:0/3910499130 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9b00063d00 0x7f9b000661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:19.714 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.698+0000 7f9b06ffd700 1 -- 172.21.15.195:0/3910499130 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6508+0+0 (secure 0 0 0) 0x7f9b1007fb50 con 0x7f9b2006a450 2022-06-27T11:52:19.715 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.698+0000 7f9b1e59c700 1 --2- 172.21.15.195:0/3910499130 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9b00063d00 0x7f9b000661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:19.715 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.698+0000 7f9b1e59c700 1 --2- 172.21.15.195:0/3910499130 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9b00063d00 0x7f9b000661b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9b14011fd0 tx=0x7f9b14017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:52:19.715 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.702+0000 7f9b06ffd700 1 -- 172.21.15.195:0/3910499130 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f9b1004e490 con 0x7f9b2006a450 2022-06-27T11:52:19.927 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.927+0000 7f9b250fc700 1 -- 172.21.15.195:0/3910499130 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9b20064030 con 0x7f9b2006a450 2022-06-27T11:52:19.927 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.927+0000 7f9b06ffd700 1 -- 172.21.15.195:0/3910499130 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+6128 (secure 0 0 0) 0x7f9b20064030 con 0x7f9b2006a450 2022-06-27T11:52:19.928 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:52:19.928 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4760,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/1026219246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1026219246},{"type":"v1","addr":"172.21.15.202:6833","nonce":1026219246}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:51:33.013206+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4849},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4849":{"gid":4849,"name":"d","rank":0,"incarnation":15,"state":"up:active","state_seq":19,"addr":"172.21.15.202:6835/3271891211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":3271891211},{"type":"v1","addr":"172.21.15.202:6835","nonce":3271891211}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:52:19.933 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.930+0000 7f9b250fc700 1 -- 172.21.15.195:0/3910499130 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9b00063d00 msgr2=0x7f9b000661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:19.933 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.930+0000 7f9b250fc700 1 --2- 172.21.15.195:0/3910499130 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9b00063d00 0x7f9b000661b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9b14011fd0 tx=0x7f9b14017000 comp rx=0 tx=0).stop 2022-06-27T11:52:19.934 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.930+0000 7f9b250fc700 1 -- 172.21.15.195:0/3910499130 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9b2006a450 msgr2=0x7f9b20159490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:19.934 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.930+0000 7f9b250fc700 1 --2- 172.21.15.195:0/3910499130 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9b2006a450 0x7f9b20159490 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f9b1000c540 tx=0x7f9b10001830 comp rx=0 tx=0).stop 2022-06-27T11:52:19.935 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.930+0000 7f9b250fc700 1 -- 172.21.15.195:0/3910499130 shutdown_connections 2022-06-27T11:52:19.935 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.930+0000 7f9b250fc700 1 --2- 172.21.15.195:0/3910499130 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9b00063d00 0x7f9b000661b0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:19.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.930+0000 7f9b250fc700 1 --2- 172.21.15.195:0/3910499130 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9b2006a450 0x7f9b20159490 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:19.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.930+0000 7f9b250fc700 1 --2- 172.21.15.195:0/3910499130 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9b201508d0 0x7f9b2006de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:19.937 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.930+0000 7f9b250fc700 1 --2- 172.21.15.195:0/3910499130 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9b20150500 0x7f9b200be8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:19.937 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.930+0000 7f9b250fc700 1 -- 172.21.15.195:0/3910499130 >> 172.21.15.195:0/3910499130 conn(0x7f9b200c9b00 msgr2=0x7f9b2015b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:52:19.938 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.930+0000 7f9b250fc700 1 -- 172.21.15.195:0/3910499130 shutdown_connections 2022-06-27T11:52:19.939 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:19.930+0000 7f9b250fc700 1 -- 172.21.15.195:0/3910499130 wait complete. 2022-06-27T11:52:19.939 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 19 2022-06-27T11:52:19.943 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-06-27T11:52:21.945 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:52:22.078 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.079+0000 7f89b0cbb700 1 Processor -- start 2022-06-27T11:52:22.079 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.079+0000 7f89b0cbb700 1 -- start start 2022-06-27T11:52:22.090 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.080+0000 7f89b0cbb700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f89ac0c4160 0x7f89ac156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:22.091 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.080+0000 7f89b0cbb700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f89ac0c39e0 0x7f89ac1535a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:22.091 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.080+0000 7f89b0cbb700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89ac0c1a20 con 0x7f89ac06f640 2022-06-27T11:52:22.091 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.080+0000 7f89b0cbb700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89ac0c1130 con 0x7f89ac0c39e0 2022-06-27T11:52:22.092 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.080+0000 7f89b0cbb700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89ac06ea90 con 0x7f89ac0c4160 2022-06-27T11:52:22.092 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.080+0000 7f89aad9d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f89ac0c39e0 0x7f89ac1535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:22.093 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.080+0000 7f89aad9d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f89ac0c39e0 0x7f89ac1535a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:43556/0 (socket says 172.21.15.195:43556) 2022-06-27T11:52:22.093 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.080+0000 7f89aad9d700 1 -- 172.21.15.195:0/1978402850 learned_addr learned my addr 172.21.15.195:0/1978402850 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:52:22.094 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.080+0000 7f89a9d9b700 1 --2- 172.21.15.195:0/1978402850 >> v2:172.21.15.202:3300/0 conn(0x7f89ac0c4160 0x7f89ac156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:22.094 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.080+0000 7f89aad9d700 1 -- 172.21.15.195:0/1978402850 >> v2:172.21.15.202:3300/0 conn(0x7f89ac0c4160 msgr2=0x7f89ac156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:52:22.095 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.080+0000 7f89aad9d700 1 --2- 172.21.15.195:0/1978402850 >> v2:172.21.15.202:3300/0 conn(0x7f89ac0c4160 0x7f89ac156ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:22.095 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.080+0000 7f89aad9d700 1 -- 172.21.15.195:0/1978402850 >> v1:172.21.15.202:6789/0 conn(0x7f89ac06f640 legacy=0x7f89ac0c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:52:22.095 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.080+0000 7f89aad9d700 1 -- 172.21.15.195:0/1978402850 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89ac06ed70 con 0x7f89ac0c39e0 2022-06-27T11:52:22.096 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.080+0000 7f89aad9d700 1 --2- 172.21.15.195:0/1978402850 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f89ac0c39e0 0x7f89ac1535a0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f89ac06dd70 tx=0x7f89a4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5151209805228df2 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:52:22.097 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.081+0000 7f89a959a700 1 -- 172.21.15.195:0/1978402850 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a400e070 con 0x7f89ac0c39e0 2022-06-27T11:52:22.097 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.081+0000 7f89a959a700 1 -- 172.21.15.195:0/1978402850 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89a400baa0 con 0x7f89ac0c39e0 2022-06-27T11:52:22.098 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.081+0000 7f89a959a700 1 -- 172.21.15.195:0/1978402850 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a40092e0 con 0x7f89ac0c39e0 2022-06-27T11:52:22.098 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.081+0000 7f89b0cbb700 1 -- 172.21.15.195:0/1978402850 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f89ac0c39e0 msgr2=0x7f89ac1535a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:22.098 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.081+0000 7f89b0cbb700 1 --2- 172.21.15.195:0/1978402850 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f89ac0c39e0 0x7f89ac1535a0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f89ac06dd70 tx=0x7f89a4009840 comp rx=0 tx=0).stop 2022-06-27T11:52:22.099 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.081+0000 7f89b0cbb700 1 -- 172.21.15.195:0/1978402850 shutdown_connections 2022-06-27T11:52:22.099 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.081+0000 7f89b0cbb700 1 --2- 172.21.15.195:0/1978402850 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f89ac0c39e0 0x7f89ac1535a0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:22.100 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.081+0000 7f89b0cbb700 1 --2- 172.21.15.195:0/1978402850 >> v2:172.21.15.202:3300/0 conn(0x7f89ac0c4160 0x7f89ac156ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:22.100 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.081+0000 7f89b0cbb700 1 -- 172.21.15.195:0/1978402850 >> 172.21.15.195:0/1978402850 conn(0x7f89ac0c6350 msgr2=0x7f89ac0c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:52:22.100 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.081+0000 7f89b0cbb700 1 -- 172.21.15.195:0/1978402850 shutdown_connections 2022-06-27T11:52:22.101 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.081+0000 7f89b0cbb700 1 -- 172.21.15.195:0/1978402850 wait complete. 2022-06-27T11:52:22.101 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.082+0000 7f89b0cbb700 1 Processor -- start 2022-06-27T11:52:22.102 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.082+0000 7f89b0cbb700 1 -- start start 2022-06-27T11:52:22.102 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.083+0000 7f89b0cbb700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f89ac06f640 0x7f89ac14afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:22.103 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.083+0000 7f89b0cbb700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f89ac0c39e0 0x7f89ac14e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:22.103 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.083+0000 7f89b0cbb700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f89ac0c4160 0x7f89ac14ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:22.104 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.083+0000 7f89b0cbb700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89ac169a30 con 0x7f89ac06f640 2022-06-27T11:52:22.104 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.083+0000 7f89b0cbb700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89ac169d10 con 0x7f89ac0c39e0 2022-06-27T11:52:22.104 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.083+0000 7f89b0cbb700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89ac169ff0 con 0x7f89ac0c4160 2022-06-27T11:52:22.105 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.083+0000 7f89aa59c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f89ac06f640 0x7f89ac14afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:22.105 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.083+0000 7f89aa59c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f89ac06f640 0x7f89ac14afe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:43558/0 (socket says 172.21.15.195:43558) 2022-06-27T11:52:22.106 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.083+0000 7f89aa59c700 1 -- 172.21.15.195:0/2341243200 learned_addr learned my addr 172.21.15.195:0/2341243200 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:52:22.106 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.083+0000 7f89a9d9b700 1 --2- 172.21.15.195:0/2341243200 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f89ac0c39e0 0x7f89ac14e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:22.107 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.083+0000 7f89aad9d700 1 --2- 172.21.15.195:0/2341243200 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f89ac0c4160 0x7f89ac14ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:22.107 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.083+0000 7f89aa59c700 1 -- 172.21.15.195:0/2341243200 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f89ac0c4160 msgr2=0x7f89ac14ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:22.107 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.083+0000 7f89aa59c700 1 --2- 172.21.15.195:0/2341243200 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f89ac0c4160 0x7f89ac14ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:22.108 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.083+0000 7f89aa59c700 1 -- 172.21.15.195:0/2341243200 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f89ac0c39e0 msgr2=0x7f89ac14e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:22.108 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.083+0000 7f89aa59c700 1 --2- 172.21.15.195:0/2341243200 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f89ac0c39e0 0x7f89ac14e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:22.109 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.083+0000 7f89aa59c700 1 -- 172.21.15.195:0/2341243200 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89a4009ae0 con 0x7f89ac06f640 2022-06-27T11:52:22.109 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.084+0000 7f89aad9d700 1 --2- 172.21.15.195:0/2341243200 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f89ac0c4160 0x7f89ac14ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T11:52:22.109 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.084+0000 7f89aa59c700 1 --2- 172.21.15.195:0/2341243200 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f89ac06f640 0x7f89ac14afe0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f899c00c490 tx=0x7f899c007e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:52:22.110 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.084+0000 7f899a7fc700 1 -- 172.21.15.195:0/2341243200 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f899c011590 con 0x7f89ac06f640 2022-06-27T11:52:22.110 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.084+0000 7f899a7fc700 1 -- 172.21.15.195:0/2341243200 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f899c0116f0 con 0x7f89ac06f640 2022-06-27T11:52:22.110 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.084+0000 7f89b0cbb700 1 -- 172.21.15.195:0/2341243200 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89ac16a2d0 con 0x7f89ac06f640 2022-06-27T11:52:22.110 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.084+0000 7f899a7fc700 1 -- 172.21.15.195:0/2341243200 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f899c011ab0 con 0x7f89ac06f640 2022-06-27T11:52:22.111 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.084+0000 7f89b0cbb700 1 -- 172.21.15.195:0/2341243200 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89ac154af0 con 0x7f89ac06f640 2022-06-27T11:52:22.111 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.085+0000 7f89b0cbb700 1 -- 172.21.15.195:0/2341243200 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89ac0722c0 con 0x7f89ac06f640 2022-06-27T11:52:22.111 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.085+0000 7f899a7fc700 1 -- 172.21.15.195:0/2341243200 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f899c010050 con 0x7f89ac06f640 2022-06-27T11:52:22.111 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.086+0000 7f899a7fc700 1 --2- 172.21.15.195:0/2341243200 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8988063d00 0x7f89880661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:22.112 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.086+0000 7f899a7fc700 1 -- 172.21.15.195:0/2341243200 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6508+0+0 (secure 0 0 0) 0x7f899c0804f0 con 0x7f89ac06f640 2022-06-27T11:52:22.112 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.086+0000 7f89a9d9b700 1 --2- 172.21.15.195:0/2341243200 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8988063d00 0x7f89880661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:22.112 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.086+0000 7f89a9d9b700 1 --2- 172.21.15.195:0/2341243200 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8988063d00 0x7f89880661b0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f89ac1674c0 tx=0x7f89a0008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:52:22.112 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.090+0000 7f899a7fc700 1 -- 172.21.15.195:0/2341243200 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f899c04efe0 con 0x7f89ac06f640 2022-06-27T11:52:22.317 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.317+0000 7f89b0cbb700 1 -- 172.21.15.195:0/2341243200 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f89ac1663c0 con 0x7f89ac06f640 2022-06-27T11:52:22.317 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.318+0000 7f899a7fc700 1 -- 172.21.15.195:0/2341243200 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+6128 (secure 0 0 0) 0x7f899c052600 con 0x7f89ac06f640 2022-06-27T11:52:22.318 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:52:22.318 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4760,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/1026219246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1026219246},{"type":"v1","addr":"172.21.15.202:6833","nonce":1026219246}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:51:33.013206+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4849},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4849":{"gid":4849,"name":"d","rank":0,"incarnation":15,"state":"up:active","state_seq":19,"addr":"172.21.15.202:6835/3271891211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":3271891211},{"type":"v1","addr":"172.21.15.202:6835","nonce":3271891211}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:52:22.320 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.320+0000 7f89b0cbb700 1 -- 172.21.15.195:0/2341243200 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8988063d00 msgr2=0x7f89880661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:22.320 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.320+0000 7f89b0cbb700 1 --2- 172.21.15.195:0/2341243200 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8988063d00 0x7f89880661b0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f89ac1674c0 tx=0x7f89a0008040 comp rx=0 tx=0).stop 2022-06-27T11:52:22.320 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.320+0000 7f89b0cbb700 1 -- 172.21.15.195:0/2341243200 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f89ac06f640 msgr2=0x7f89ac14afe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:22.321 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.320+0000 7f89b0cbb700 1 --2- 172.21.15.195:0/2341243200 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f89ac06f640 0x7f89ac14afe0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f899c00c490 tx=0x7f899c007e20 comp rx=0 tx=0).stop 2022-06-27T11:52:22.321 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.320+0000 7f89b0cbb700 1 -- 172.21.15.195:0/2341243200 shutdown_connections 2022-06-27T11:52:22.321 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.320+0000 7f89b0cbb700 1 --2- 172.21.15.195:0/2341243200 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8988063d00 0x7f89880661b0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:22.321 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.320+0000 7f89b0cbb700 1 --2- 172.21.15.195:0/2341243200 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f89ac06f640 0x7f89ac14afe0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:22.322 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.320+0000 7f89b0cbb700 1 --2- 172.21.15.195:0/2341243200 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f89ac0c4160 0x7f89ac14ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:22.322 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.320+0000 7f89b0cbb700 1 --2- 172.21.15.195:0/2341243200 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f89ac0c39e0 0x7f89ac14e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:22.323 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.320+0000 7f89b0cbb700 1 -- 172.21.15.195:0/2341243200 >> 172.21.15.195:0/2341243200 conn(0x7f89ac0c6350 msgr2=0x7f89ac0c1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:52:22.323 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.320+0000 7f89b0cbb700 1 -- 172.21.15.195:0/2341243200 shutdown_connections 2022-06-27T11:52:22.323 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:22.320+0000 7f89b0cbb700 1 -- 172.21.15.195:0/2341243200 wait complete. 2022-06-27T11:52:22.323 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 19 2022-06-27T11:52:22.331 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-06-27T11:52:24.333 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:52:24.469 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.470+0000 7ff6223db700 1 Processor -- start 2022-06-27T11:52:24.470 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.470+0000 7ff6223db700 1 -- start start 2022-06-27T11:52:24.475 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.471+0000 7ff6223db700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7ff61c0c5000 0x7ff61c0c2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:24.476 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.471+0000 7ff6223db700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff61c0c61e0 0x7ff61c0c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:24.476 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.471+0000 7ff6223db700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff61c0bc490 con 0x7ff61c0c2bf0 2022-06-27T11:52:24.476 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.471+0000 7ff6223db700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff61c06a0e0 con 0x7ff61c0c5000 2022-06-27T11:52:24.477 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.471+0000 7ff6223db700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff61c0bc6e0 con 0x7ff61c0c61e0 2022-06-27T11:52:24.477 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.471+0000 7ff61b7fe700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7ff61c0c5000 0x7ff61c0c2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:24.477 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.471+0000 7ff620978700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff61c0c61e0 0x7ff61c0c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:24.477 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.471+0000 7ff61bfff700 1 --1- >> v1:172.21.15.202:6789/0 conn(0x7ff61c0c2bf0 0x7ff61c0c48f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.202:6789/0 says I am v1:172.21.15.195:51618/0 (socket says 172.21.15.195:51618) 2022-06-27T11:52:24.478 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.471+0000 7ff61bfff700 1 -- 172.21.15.195:0/3293557907 learned_addr learned my addr 172.21.15.195:0/3293557907 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:52:24.478 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.472+0000 7ff620978700 1 -- 172.21.15.195:0/3293557907 >> v2:172.21.15.202:3300/0 conn(0x7ff61c0c5000 msgr2=0x7ff61c0c2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:52:24.478 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.472+0000 7ff620978700 1 --2- 172.21.15.195:0/3293557907 >> v2:172.21.15.202:3300/0 conn(0x7ff61c0c5000 0x7ff61c0c2fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:24.479 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.472+0000 7ff620978700 1 -- 172.21.15.195:0/3293557907 >> v1:172.21.15.202:6789/0 conn(0x7ff61c0c2bf0 legacy=0x7ff61c0c48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:24.479 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.472+0000 7ff620978700 1 -- 172.21.15.195:0/3293557907 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff61c158220 con 0x7ff61c0c61e0 2022-06-27T11:52:24.480 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.472+0000 7ff61b7fe700 1 --2- 172.21.15.195:0/3293557907 >> v2:172.21.15.202:3300/0 conn(0x7ff61c0c5000 0x7ff61c0c2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:52:24.480 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.472+0000 7ff620978700 1 --2- 172.21.15.195:0/3293557907 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff61c0c61e0 0x7ff61c0c3900 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff614009e60 tx=0x7ff614009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ccac29f743535f9 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:52:24.480 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.472+0000 7ff61affd700 1 -- 172.21.15.195:0/3293557907 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff61400e070 con 0x7ff61c0c61e0 2022-06-27T11:52:24.480 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.472+0000 7ff61affd700 1 -- 172.21.15.195:0/3293557907 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff61400bb60 con 0x7ff61c0c61e0 2022-06-27T11:52:24.481 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.472+0000 7ff61affd700 1 -- 172.21.15.195:0/3293557907 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff614012430 con 0x7ff61c0c61e0 2022-06-27T11:52:24.481 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.473+0000 7ff6223db700 1 -- 172.21.15.195:0/3293557907 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff61c0c61e0 msgr2=0x7ff61c0c3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:24.481 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.473+0000 7ff6223db700 1 --2- 172.21.15.195:0/3293557907 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff61c0c61e0 0x7ff61c0c3900 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff614009e60 tx=0x7ff614009780 comp rx=0 tx=0).stop 2022-06-27T11:52:24.481 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.473+0000 7ff6223db700 1 -- 172.21.15.195:0/3293557907 shutdown_connections 2022-06-27T11:52:24.482 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.473+0000 7ff6223db700 1 --2- 172.21.15.195:0/3293557907 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff61c0c61e0 0x7ff61c0c3900 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:24.482 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.473+0000 7ff6223db700 1 --2- 172.21.15.195:0/3293557907 >> v2:172.21.15.202:3300/0 conn(0x7ff61c0c5000 0x7ff61c0c2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:24.482 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.473+0000 7ff6223db700 1 -- 172.21.15.195:0/3293557907 >> 172.21.15.195:0/3293557907 conn(0x7ff61c0c9b00 msgr2=0x7ff61c0c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:52:24.483 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.473+0000 7ff6223db700 1 -- 172.21.15.195:0/3293557907 shutdown_connections 2022-06-27T11:52:24.483 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.473+0000 7ff6223db700 1 -- 172.21.15.195:0/3293557907 wait complete. 2022-06-27T11:52:24.483 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.474+0000 7ff6223db700 1 Processor -- start 2022-06-27T11:52:24.483 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.474+0000 7ff6223db700 1 -- start start 2022-06-27T11:52:24.484 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.475+0000 7ff6223db700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff61c0c2bf0 0x7ff61c15a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:24.484 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.475+0000 7ff6223db700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff61c0c5000 0x7ff61c155370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:24.484 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.475+0000 7ff6223db700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff61c0c61e0 0x7ff61c14b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:24.484 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.475+0000 7ff6223db700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff61c145820 con 0x7ff61c0c2bf0 2022-06-27T11:52:24.485 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.475+0000 7ff6223db700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff61c167af0 con 0x7ff61c0c5000 2022-06-27T11:52:24.485 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.475+0000 7ff6223db700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff61c1674e0 con 0x7ff61c0c61e0 2022-06-27T11:52:24.485 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.475+0000 7ff61bfff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff61c0c2bf0 0x7ff61c15a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:24.485 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.475+0000 7ff61bfff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff61c0c2bf0 0x7ff61c15a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:43576/0 (socket says 172.21.15.195:43576) 2022-06-27T11:52:24.486 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.475+0000 7ff61bfff700 1 -- 172.21.15.195:0/4146106015 learned_addr learned my addr 172.21.15.195:0/4146106015 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:52:24.486 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.475+0000 7ff620978700 1 --2- 172.21.15.195:0/4146106015 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff61c0c61e0 0x7ff61c14b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:24.486 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.475+0000 7ff61b7fe700 1 --2- 172.21.15.195:0/4146106015 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff61c0c5000 0x7ff61c155370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:24.486 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.475+0000 7ff61bfff700 1 -- 172.21.15.195:0/4146106015 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff61c0c61e0 msgr2=0x7ff61c14b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:24.487 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.475+0000 7ff61bfff700 1 --2- 172.21.15.195:0/4146106015 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff61c0c61e0 0x7ff61c14b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:24.487 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.475+0000 7ff61bfff700 1 -- 172.21.15.195:0/4146106015 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff61c0c5000 msgr2=0x7ff61c155370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:24.487 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.475+0000 7ff61bfff700 1 --2- 172.21.15.195:0/4146106015 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff61c0c5000 0x7ff61c155370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:24.487 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.475+0000 7ff61bfff700 1 -- 172.21.15.195:0/4146106015 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6100092e0 con 0x7ff61c0c2bf0 2022-06-27T11:52:24.488 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.475+0000 7ff61b7fe700 1 --2- 172.21.15.195:0/4146106015 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff61c0c5000 0x7ff61c155370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:52:24.488 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.475+0000 7ff61bfff700 1 --2- 172.21.15.195:0/4146106015 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff61c0c2bf0 0x7ff61c15a290 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7ff60c00c490 tx=0x7ff60c007e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:52:24.488 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.476+0000 7ff603fff700 1 -- 172.21.15.195:0/4146106015 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff60c00ce40 con 0x7ff61c0c2bf0 2022-06-27T11:52:24.488 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.476+0000 7ff603fff700 1 -- 172.21.15.195:0/4146106015 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff60c0072e0 con 0x7ff61c0c2bf0 2022-06-27T11:52:24.489 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.476+0000 7ff603fff700 1 -- 172.21.15.195:0/4146106015 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff60c0076a0 con 0x7ff61c0c2bf0 2022-06-27T11:52:24.489 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.476+0000 7ff6223db700 1 -- 172.21.15.195:0/4146106015 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff614009a20 con 0x7ff61c0c2bf0 2022-06-27T11:52:24.489 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.476+0000 7ff6223db700 1 -- 172.21.15.195:0/4146106015 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff61c0bd900 con 0x7ff61c0c2bf0 2022-06-27T11:52:24.489 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.477+0000 7ff6223db700 1 -- 172.21.15.195:0/4146106015 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff61c0722c0 con 0x7ff61c0c2bf0 2022-06-27T11:52:24.491 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.477+0000 7ff603fff700 1 -- 172.21.15.195:0/4146106015 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7ff60c019030 con 0x7ff61c0c2bf0 2022-06-27T11:52:24.491 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.477+0000 7ff603fff700 1 --2- 172.21.15.195:0/4146106015 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff5fc063d30 0x7ff5fc0661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:24.491 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.478+0000 7ff61b7fe700 1 --2- 172.21.15.195:0/4146106015 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff5fc063d30 0x7ff5fc0661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:24.492 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.478+0000 7ff603fff700 1 -- 172.21.15.195:0/4146106015 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6508+0+0 (secure 0 0 0) 0x7ff60c07fbc0 con 0x7ff61c0c2bf0 2022-06-27T11:52:24.492 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.478+0000 7ff61b7fe700 1 --2- 172.21.15.195:0/4146106015 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff5fc063d30 0x7ff5fc0661e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff61c06f120 tx=0x7ff610009060 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:52:24.492 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.482+0000 7ff603fff700 1 -- 172.21.15.195:0/4146106015 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7ff60c04e5f0 con 0x7ff61c0c2bf0 2022-06-27T11:52:24.707 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.708+0000 7ff6223db700 1 -- 172.21.15.195:0/4146106015 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff61c06c830 con 0x7ff61c0c2bf0 2022-06-27T11:52:24.708 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.709+0000 7ff603fff700 1 -- 172.21.15.195:0/4146106015 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+6128 (secure 0 0 0) 0x7ff60c017070 con 0x7ff61c0c2bf0 2022-06-27T11:52:24.709 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:52:24.709 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4760,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/1026219246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1026219246},{"type":"v1","addr":"172.21.15.202:6833","nonce":1026219246}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:51:33.013206+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4849},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4849":{"gid":4849,"name":"d","rank":0,"incarnation":15,"state":"up:active","state_seq":19,"addr":"172.21.15.202:6835/3271891211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":3271891211},{"type":"v1","addr":"172.21.15.202:6835","nonce":3271891211}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:52:24.711 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.711+0000 7ff6223db700 1 -- 172.21.15.195:0/4146106015 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff5fc063d30 msgr2=0x7ff5fc0661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:24.711 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.711+0000 7ff6223db700 1 --2- 172.21.15.195:0/4146106015 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff5fc063d30 0x7ff5fc0661e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff61c06f120 tx=0x7ff610009060 comp rx=0 tx=0).stop 2022-06-27T11:52:24.711 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.711+0000 7ff6223db700 1 -- 172.21.15.195:0/4146106015 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff61c0c2bf0 msgr2=0x7ff61c15a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:24.712 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.711+0000 7ff6223db700 1 --2- 172.21.15.195:0/4146106015 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff61c0c2bf0 0x7ff61c15a290 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7ff60c00c490 tx=0x7ff60c007e20 comp rx=0 tx=0).stop 2022-06-27T11:52:24.712 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.711+0000 7ff6223db700 1 -- 172.21.15.195:0/4146106015 shutdown_connections 2022-06-27T11:52:24.712 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.711+0000 7ff6223db700 1 --2- 172.21.15.195:0/4146106015 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff5fc063d30 0x7ff5fc0661e0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:24.713 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.711+0000 7ff6223db700 1 --2- 172.21.15.195:0/4146106015 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff61c0c2bf0 0x7ff61c15a290 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:24.713 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.711+0000 7ff6223db700 1 --2- 172.21.15.195:0/4146106015 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff61c0c61e0 0x7ff61c14b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:24.713 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.711+0000 7ff6223db700 1 --2- 172.21.15.195:0/4146106015 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff61c0c5000 0x7ff61c155370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:24.713 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.711+0000 7ff6223db700 1 -- 172.21.15.195:0/4146106015 >> 172.21.15.195:0/4146106015 conn(0x7ff61c0c9b00 msgr2=0x7ff61c0c6e40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:52:24.714 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.711+0000 7ff6223db700 1 -- 172.21.15.195:0/4146106015 shutdown_connections 2022-06-27T11:52:24.714 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:24.711+0000 7ff6223db700 1 -- 172.21.15.195:0/4146106015 wait complete. 2022-06-27T11:52:24.715 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 19 2022-06-27T11:52:24.723 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-06-27T11:52:26.724 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:52:26.858 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:26.859+0000 7f6cf02c4700 1 Processor -- start 2022-06-27T11:52:26.859 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:26.859+0000 7f6cf02c4700 1 -- start start 2022-06-27T11:52:26.864 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:26.860+0000 7f6cf02c4700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f6ce806e850 0x7f6ce81570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:26.865 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:26.860+0000 7f6cf02c4700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ce806fba0 con 0x7f6ce80c34a0 2022-06-27T11:52:26.865 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:26.860+0000 7f6cf02c4700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ce806f5e0 con 0x7f6ce806fef0 2022-06-27T11:52:26.866 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:26.860+0000 7f6cf02c4700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ce81575e0 con 0x7f6ce806e850 2022-06-27T11:52:26.866 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:26.860+0000 7f6cee861700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f6ce80c34a0 0x7f6ce80c2920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:49774/0 (socket says 172.21.15.195:49774) 2022-06-27T11:52:26.866 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:26.860+0000 7f6cee861700 1 -- 172.21.15.195:0/429897067 learned_addr learned my addr 172.21.15.195:0/429897067 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:52:26.866 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:26.860+0000 7f6cee060700 1 --2- 172.21.15.195:0/429897067 >> v2:172.21.15.202:3300/0 conn(0x7f6ce806e850 0x7f6ce81570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:26.867 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:26.860+0000 7f6ced05e700 1 -- 172.21.15.195:0/429897067 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4100566550 0 0) 0x7f6ce806fba0 con 0x7f6ce80c34a0 2022-06-27T11:52:26.867 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:26.860+0000 7f6ced05e700 1 -- 172.21.15.195:0/429897067 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cd0003430 con 0x7f6ce80c34a0 2022-06-27T11:52:26.867 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:26.861+0000 7f6cee060700 1 -- 172.21.15.195:0/429897067 >> v1:172.21.15.202:6789/0 conn(0x7f6ce806fef0 legacy=0x7f6ce8069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:26.868 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:26.861+0000 7f6cee060700 1 -- 172.21.15.195:0/429897067 >> v1:172.21.15.195:6789/0 conn(0x7f6ce80c34a0 legacy=0x7f6ce80c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:26.868 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:26.861+0000 7f6ced05e700 1 -- 172.21.15.195:0/429897067 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4116474660 0 0) 0x7f6cd0003430 con 0x7f6ce80c34a0 2022-06-27T11:52:26.868 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:26.861+0000 7f6cee060700 1 -- 172.21.15.195:0/429897067 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ce806f8c0 con 0x7f6ce806e850 2022-06-27T11:52:26.868 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:26.861+0000 7f6cee060700 1 --2- 172.21.15.195:0/429897067 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6ce806e850 0x7f6ce81570a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f6ce40097a0 tx=0x7f6ce4009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=23b0c49dc5b6daf server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:52:26.869 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:26.861+0000 7f6ced05e700 1 -- 172.21.15.195:0/429897067 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce00017f0 con 0x7f6ce806e850 2022-06-27T11:52:26.869 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:26.861+0000 7f6ced05e700 1 -- 172.21.15.195:0/429897067 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ce4004030 con 0x7f6ce806e850 2022-06-27T11:52:26.869 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:26.861+0000 7f6ced05e700 1 -- 172.21.15.195:0/429897067 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce40043d0 con 0x7f6ce806e850 2022-06-27T11:52:26.869 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:26.862+0000 7f6cf02c4700 1 -- 172.21.15.195:0/429897067 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6ce806e850 msgr2=0x7f6ce81570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:26.870 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:26.862+0000 7f6cf02c4700 1 --2- 172.21.15.195:0/429897067 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6ce806e850 0x7f6ce81570a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f6ce40097a0 tx=0x7f6ce4009060 comp rx=0 tx=0).stop 2022-06-27T11:52:26.870 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:26.862+0000 7f6cf02c4700 1 -- 172.21.15.195:0/429897067 shutdown_connections 2022-06-27T11:52:26.870 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:26.862+0000 7f6cf02c4700 1 --2- 172.21.15.195:0/429897067 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6ce806e850 0x7f6ce81570a0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:26.870 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:26.862+0000 7f6cf02c4700 1 -- 172.21.15.195:0/429897067 >> 172.21.15.195:0/429897067 conn(0x7f6ce80c6350 msgr2=0x7f6ce80c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:52:26.871 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:26.862+0000 7f6cf02c4700 1 -- 172.21.15.195:0/429897067 shutdown_connections 2022-06-27T11:52:26.871 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:26.862+0000 7f6cf02c4700 1 -- 172.21.15.195:0/429897067 wait complete. 2022-06-27T11:52:26.871 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:26.863+0000 7f6cf02c4700 1 Processor -- start 2022-06-27T11:52:26.871 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:26.863+0000 7f6cf02c4700 1 -- start start 2022-06-27T11:52:26.872 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:26.864+0000 7f6cf02c4700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6ce806e850 0x7f6ce80c43d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:26.872 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:26.864+0000 7f6cf02c4700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6ce80c34a0 0x7f6ce81667e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:26.872 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:26.864+0000 7f6cf02c4700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6ce80c0510 0x7f6ce8166d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:26.872 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:26.864+0000 7f6cf02c4700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce81570c0 con 0x7f6ce80c0510 2022-06-27T11:52:26.873 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:26.864+0000 7f6cf02c4700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce80c4e10 con 0x7f6ce806e850 2022-06-27T11:52:26.873 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:26.864+0000 7f6cf02c4700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce8156de0 con 0x7f6ce80c34a0 2022-06-27T11:52:26.873 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:26.864+0000 7f6cee861700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6ce80c0510 0x7f6ce8166d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:26.874 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:26.864+0000 7f6cee861700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6ce80c0510 0x7f6ce8166d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:43594/0 (socket says 172.21.15.195:43594) 2022-06-27T11:52:26.874 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:26.864+0000 7f6cee861700 1 -- 172.21.15.195:0/617622434 learned_addr learned my addr 172.21.15.195:0/617622434 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:52:26.874 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:26.864+0000 7f6cee060700 1 --2- 172.21.15.195:0/617622434 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6ce806e850 0x7f6ce80c43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:26.874 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:26.864+0000 7f6ced85f700 1 --2- 172.21.15.195:0/617622434 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6ce80c34a0 0x7f6ce81667e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:26.875 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:26.864+0000 7f6cee861700 1 -- 172.21.15.195:0/617622434 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6ce80c34a0 msgr2=0x7f6ce81667e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:26.875 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:26.864+0000 7f6cee861700 1 --2- 172.21.15.195:0/617622434 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6ce80c34a0 0x7f6ce81667e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:26.875 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:26.864+0000 7f6cee861700 1 -- 172.21.15.195:0/617622434 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6ce806e850 msgr2=0x7f6ce80c43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:26.875 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:26.864+0000 7f6cee861700 1 --2- 172.21.15.195:0/617622434 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6ce806e850 0x7f6ce80c43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:26.876 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:26.864+0000 7f6cee861700 1 -- 172.21.15.195:0/617622434 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ce40092e0 con 0x7f6ce80c0510 2022-06-27T11:52:26.876 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:26.864+0000 7f6ced85f700 1 --2- 172.21.15.195:0/617622434 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6ce80c34a0 0x7f6ce81667e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T11:52:26.876 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:26.864+0000 7f6cee060700 1 --2- 172.21.15.195:0/617622434 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6ce806e850 0x7f6ce80c43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T11:52:26.877 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:26.864+0000 7f6cee861700 1 --2- 172.21.15.195:0/617622434 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6ce80c0510 0x7f6ce8166d20 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f6ce0007fa0 tx=0x7f6ce0001e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:52:26.877 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:26.864+0000 7f6cd67fc700 1 -- 172.21.15.195:0/617622434 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce0004c80 con 0x7f6ce80c0510 2022-06-27T11:52:26.877 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:26.864+0000 7f6cd67fc700 1 -- 172.21.15.195:0/617622434 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ce0004e00 con 0x7f6ce80c0510 2022-06-27T11:52:26.877 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:26.864+0000 7f6cf02c4700 1 -- 172.21.15.195:0/617622434 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ce806cc80 con 0x7f6ce80c0510 2022-06-27T11:52:26.878 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:26.864+0000 7f6cd67fc700 1 -- 172.21.15.195:0/617622434 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce00073a0 con 0x7f6ce80c0510 2022-06-27T11:52:26.878 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:26.864+0000 7f6cf02c4700 1 -- 172.21.15.195:0/617622434 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ce8157ea0 con 0x7f6ce80c0510 2022-06-27T11:52:26.878 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:26.865+0000 7f6cf02c4700 1 -- 172.21.15.195:0/617622434 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ce8163cd0 con 0x7f6ce80c0510 2022-06-27T11:52:26.880 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:26.866+0000 7f6cd67fc700 1 -- 172.21.15.195:0/617622434 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f6ce0007710 con 0x7f6ce80c0510 2022-06-27T11:52:26.880 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:26.866+0000 7f6cd67fc700 1 --2- 172.21.15.195:0/617622434 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6cc8063d30 0x7f6cc80661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:26.880 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:26.866+0000 7f6cd67fc700 1 -- 172.21.15.195:0/617622434 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6508+0+0 (secure 0 0 0) 0x7f6ce0082dd0 con 0x7f6ce80c0510 2022-06-27T11:52:26.880 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:26.866+0000 7f6cee060700 1 --2- 172.21.15.195:0/617622434 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6cc8063d30 0x7f6cc80661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:26.881 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:26.867+0000 7f6cee060700 1 --2- 172.21.15.195:0/617622434 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6cc8063d30 0x7f6cc80661e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6ce40097a0 tx=0x7f6ce4016000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:52:26.881 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:26.871+0000 7f6cd67fc700 1 -- 172.21.15.195:0/617622434 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f6ce0051880 con 0x7f6ce80c0510 2022-06-27T11:52:27.093 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:27.094+0000 7f6cf02c4700 1 -- 172.21.15.195:0/617622434 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ce80722c0 con 0x7f6ce80c0510 2022-06-27T11:52:27.094 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:27.094+0000 7f6cd67fc700 1 -- 172.21.15.195:0/617622434 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+6128 (secure 0 0 0) 0x7f6ce0015070 con 0x7f6ce80c0510 2022-06-27T11:52:27.095 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:52:27.095 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4760,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/1026219246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1026219246},{"type":"v1","addr":"172.21.15.202:6833","nonce":1026219246}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:51:33.013206+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4849},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4849":{"gid":4849,"name":"d","rank":0,"incarnation":15,"state":"up:active","state_seq":19,"addr":"172.21.15.202:6835/3271891211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":3271891211},{"type":"v1","addr":"172.21.15.202:6835","nonce":3271891211}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:52:27.097 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:27.096+0000 7f6cf02c4700 1 -- 172.21.15.195:0/617622434 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6cc8063d30 msgr2=0x7f6cc80661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:27.097 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:27.096+0000 7f6cf02c4700 1 --2- 172.21.15.195:0/617622434 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6cc8063d30 0x7f6cc80661e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6ce40097a0 tx=0x7f6ce4016000 comp rx=0 tx=0).stop 2022-06-27T11:52:27.097 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:27.096+0000 7f6cf02c4700 1 -- 172.21.15.195:0/617622434 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6ce80c0510 msgr2=0x7f6ce8166d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:27.100 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:27.096+0000 7f6cf02c4700 1 --2- 172.21.15.195:0/617622434 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6ce80c0510 0x7f6ce8166d20 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f6ce0007fa0 tx=0x7f6ce0001e20 comp rx=0 tx=0).stop 2022-06-27T11:52:27.101 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:27.096+0000 7f6cf02c4700 1 -- 172.21.15.195:0/617622434 shutdown_connections 2022-06-27T11:52:27.102 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:27.096+0000 7f6cf02c4700 1 --2- 172.21.15.195:0/617622434 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6cc8063d30 0x7f6cc80661e0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:27.102 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:27.096+0000 7f6cf02c4700 1 --2- 172.21.15.195:0/617622434 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6ce80c0510 0x7f6ce8166d20 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:27.103 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:27.096+0000 7f6cf02c4700 1 --2- 172.21.15.195:0/617622434 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6ce806e850 0x7f6ce80c43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:27.104 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:27.096+0000 7f6cf02c4700 1 --2- 172.21.15.195:0/617622434 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6ce80c34a0 0x7f6ce81667e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:27.104 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:27.096+0000 7f6cf02c4700 1 -- 172.21.15.195:0/617622434 >> 172.21.15.195:0/617622434 conn(0x7f6ce80c6350 msgr2=0x7f6ce8153020 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:52:27.104 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:27.097+0000 7f6cf02c4700 1 -- 172.21.15.195:0/617622434 shutdown_connections 2022-06-27T11:52:27.104 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:27.097+0000 7f6cf02c4700 1 -- 172.21.15.195:0/617622434 wait complete. 2022-06-27T11:52:27.105 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 19 2022-06-27T11:52:27.112 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-06-27T11:52:27.304 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T11:52:27.308 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T11:52:29.113 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:52:29.264 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.261+0000 7f1c59443700 1 Processor -- start 2022-06-27T11:52:29.265 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.262+0000 7f1c59443700 1 -- start start 2022-06-27T11:52:29.278 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.262+0000 7f1c59443700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1c5406e850 0x7f1c541570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:29.278 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.262+0000 7f1c59443700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1c5406fef0 0x7f1c54069de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:29.279 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.262+0000 7f1c59443700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f1c540c34a0 0x7f1c5406a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:29.279 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.262+0000 7f1c59443700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c541575e0 con 0x7f1c5406fef0 2022-06-27T11:52:29.279 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.262+0000 7f1c59443700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c5406fba0 con 0x7f1c540c34a0 2022-06-27T11:52:29.280 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.262+0000 7f1c59443700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c5406f5e0 con 0x7f1c5406e850 2022-06-27T11:52:29.280 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.263+0000 7f1c527fc700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1c5406fef0 0x7f1c54069de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:29.280 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.263+0000 7f1c527fc700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1c5406fef0 0x7f1c54069de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:43600/0 (socket says 172.21.15.195:43600) 2022-06-27T11:52:29.280 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.263+0000 7f1c527fc700 1 -- 172.21.15.195:0/3266030479 learned_addr learned my addr 172.21.15.195:0/3266030479 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:52:29.281 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.263+0000 7f1c52ffd700 1 --2- 172.21.15.195:0/3266030479 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1c5406e850 0x7f1c541570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:29.281 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.263+0000 7f1c537fe700 1 --2- 172.21.15.195:0/3266030479 >> v2:172.21.15.202:3300/0 conn(0x7f1c540c34a0 0x7f1c5406a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:29.281 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.263+0000 7f1c527fc700 1 -- 172.21.15.195:0/3266030479 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1c5406e850 msgr2=0x7f1c541570a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:52:29.281 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.263+0000 7f1c527fc700 1 --2- 172.21.15.195:0/3266030479 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1c5406e850 0x7f1c541570a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:29.282 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.263+0000 7f1c527fc700 1 -- 172.21.15.195:0/3266030479 >> v2:172.21.15.202:3300/0 conn(0x7f1c540c34a0 msgr2=0x7f1c5406a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:52:29.282 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.263+0000 7f1c527fc700 1 --2- 172.21.15.195:0/3266030479 >> v2:172.21.15.202:3300/0 conn(0x7f1c540c34a0 0x7f1c5406a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:29.282 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.263+0000 7f1c527fc700 1 -- 172.21.15.195:0/3266030479 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c5406f8c0 con 0x7f1c5406fef0 2022-06-27T11:52:29.283 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.263+0000 7f1c527fc700 1 --2- 172.21.15.195:0/3266030479 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1c5406fef0 0x7f1c54069de0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f1c5406e1e0 tx=0x7f1c3c009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15411af5f59e0490 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:52:29.283 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.263+0000 7f1c51ffb700 1 -- 172.21.15.195:0/3266030479 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c3c0108a0 con 0x7f1c5406fef0 2022-06-27T11:52:29.283 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.263+0000 7f1c51ffb700 1 -- 172.21.15.195:0/3266030479 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c3c010a00 con 0x7f1c5406fef0 2022-06-27T11:52:29.283 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.263+0000 7f1c51ffb700 1 -- 172.21.15.195:0/3266030479 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c3c010da0 con 0x7f1c5406fef0 2022-06-27T11:52:29.284 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.264+0000 7f1c59443700 1 -- 172.21.15.195:0/3266030479 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1c5406fef0 msgr2=0x7f1c54069de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:29.284 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.264+0000 7f1c59443700 1 --2- 172.21.15.195:0/3266030479 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1c5406fef0 0x7f1c54069de0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f1c5406e1e0 tx=0x7f1c3c009120 comp rx=0 tx=0).stop 2022-06-27T11:52:29.284 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.264+0000 7f1c59443700 1 -- 172.21.15.195:0/3266030479 shutdown_connections 2022-06-27T11:52:29.284 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.264+0000 7f1c59443700 1 --2- 172.21.15.195:0/3266030479 >> v2:172.21.15.202:3300/0 conn(0x7f1c540c34a0 0x7f1c5406a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:29.285 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.264+0000 7f1c59443700 1 --2- 172.21.15.195:0/3266030479 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1c5406e850 0x7f1c541570a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:29.288 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.264+0000 7f1c59443700 1 --2- 172.21.15.195:0/3266030479 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1c5406fef0 0x7f1c54069de0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:29.289 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.264+0000 7f1c59443700 1 -- 172.21.15.195:0/3266030479 >> 172.21.15.195:0/3266030479 conn(0x7f1c540c6350 msgr2=0x7f1c540c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:52:29.290 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.264+0000 7f1c59443700 1 -- 172.21.15.195:0/3266030479 shutdown_connections 2022-06-27T11:52:29.290 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.264+0000 7f1c59443700 1 -- 172.21.15.195:0/3266030479 wait complete. 2022-06-27T11:52:29.290 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.265+0000 7f1c59443700 1 Processor -- start 2022-06-27T11:52:29.291 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.265+0000 7f1c59443700 1 -- start start 2022-06-27T11:52:29.291 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.266+0000 7f1c59443700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1c5406e850 0x7f1c54165fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:29.291 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.266+0000 7f1c59443700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1c540c34a0 0x7f1c54162000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:29.291 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.266+0000 7f1c59443700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1c5414f080 0x7f1c54150310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:29.291 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.266+0000 7f1c59443700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c54165550 con 0x7f1c540c34a0 2022-06-27T11:52:29.292 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.266+0000 7f1c59443700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c54154840 con 0x7f1c5414f080 2022-06-27T11:52:29.294 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.266+0000 7f1c59443700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c54154b20 con 0x7f1c5406e850 2022-06-27T11:52:29.294 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.266+0000 7f1c527fc700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1c540c34a0 0x7f1c54162000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:29.295 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.266+0000 7f1c527fc700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1c540c34a0 0x7f1c54162000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:43606/0 (socket says 172.21.15.195:43606) 2022-06-27T11:52:29.295 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.266+0000 7f1c527fc700 1 -- 172.21.15.195:0/154479586 learned_addr learned my addr 172.21.15.195:0/154479586 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:52:29.295 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.266+0000 7f1c52ffd700 1 --2- 172.21.15.195:0/154479586 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1c5406e850 0x7f1c54165fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:29.296 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.266+0000 7f1c537fe700 1 --2- 172.21.15.195:0/154479586 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1c5414f080 0x7f1c54150310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:29.296 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.266+0000 7f1c527fc700 1 -- 172.21.15.195:0/154479586 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1c5406e850 msgr2=0x7f1c54165fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:29.296 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.266+0000 7f1c527fc700 1 --2- 172.21.15.195:0/154479586 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1c5406e850 0x7f1c54165fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:29.296 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.266+0000 7f1c527fc700 1 -- 172.21.15.195:0/154479586 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1c5414f080 msgr2=0x7f1c54150310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:29.297 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.266+0000 7f1c527fc700 1 --2- 172.21.15.195:0/154479586 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1c5414f080 0x7f1c54150310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:29.297 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.266+0000 7f1c527fc700 1 -- 172.21.15.195:0/154479586 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c3c0093a0 con 0x7f1c540c34a0 2022-06-27T11:52:29.297 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.266+0000 7f1c537fe700 1 --2- 172.21.15.195:0/154479586 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1c5414f080 0x7f1c54150310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T11:52:29.297 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.266+0000 7f1c527fc700 1 --2- 172.21.15.195:0/154479586 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1c540c34a0 0x7f1c54162000 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f1c3c01b040 tx=0x7f1c3c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:52:29.297 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.267+0000 7f1c4affd700 1 -- 172.21.15.195:0/154479586 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c3c004050 con 0x7f1c540c34a0 2022-06-27T11:52:29.298 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.267+0000 7f1c4affd700 1 -- 172.21.15.195:0/154479586 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c3c0041b0 con 0x7f1c540c34a0 2022-06-27T11:52:29.298 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.267+0000 7f1c59443700 1 -- 172.21.15.195:0/154479586 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c54154e00 con 0x7f1c540c34a0 2022-06-27T11:52:29.299 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.267+0000 7f1c4affd700 1 -- 172.21.15.195:0/154479586 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c3c004570 con 0x7f1c540c34a0 2022-06-27T11:52:29.299 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.267+0000 7f1c59443700 1 -- 172.21.15.195:0/154479586 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c54165a80 con 0x7f1c540c34a0 2022-06-27T11:52:29.299 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.268+0000 7f1c59443700 1 -- 172.21.15.195:0/154479586 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c540c90e0 con 0x7f1c540c34a0 2022-06-27T11:52:29.299 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.268+0000 7f1c4affd700 1 -- 172.21.15.195:0/154479586 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f1c3c02a030 con 0x7f1c540c34a0 2022-06-27T11:52:29.299 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.268+0000 7f1c4affd700 1 --2- 172.21.15.195:0/154479586 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1c38063d00 0x7f1c380661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:29.300 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.268+0000 7f1c4affd700 1 -- 172.21.15.195:0/154479586 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6508+0+0 (secure 0 0 0) 0x7f1c3c085760 con 0x7f1c540c34a0 2022-06-27T11:52:29.300 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.268+0000 7f1c52ffd700 1 --2- 172.21.15.195:0/154479586 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1c38063d00 0x7f1c380661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:29.301 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.269+0000 7f1c52ffd700 1 --2- 172.21.15.195:0/154479586 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1c38063d00 0x7f1c380661b0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f1c4400fa20 tx=0x7f1c44008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:52:29.301 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.273+0000 7f1c4affd700 1 -- 172.21.15.195:0/154479586 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f1c540c90e0 con 0x7f1c540c34a0 2022-06-27T11:52:29.517 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.512+0000 7f1c59443700 1 -- 172.21.15.195:0/154479586 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1c540c90e0 con 0x7f1c540c34a0 2022-06-27T11:52:29.517 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.512+0000 7f1c4affd700 1 -- 172.21.15.195:0/154479586 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+6128 (secure 0 0 0) 0x7f1c540c90e0 con 0x7f1c540c34a0 2022-06-27T11:52:29.518 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:52:29.519 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4760,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/1026219246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1026219246},{"type":"v1","addr":"172.21.15.202:6833","nonce":1026219246}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:51:33.013206+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4849},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4849":{"gid":4849,"name":"d","rank":0,"incarnation":15,"state":"up:active","state_seq":19,"addr":"172.21.15.202:6835/3271891211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":3271891211},{"type":"v1","addr":"172.21.15.202:6835","nonce":3271891211}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:52:29.520 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.514+0000 7f1c59443700 1 -- 172.21.15.195:0/154479586 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1c38063d00 msgr2=0x7f1c380661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:29.520 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.514+0000 7f1c59443700 1 --2- 172.21.15.195:0/154479586 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1c38063d00 0x7f1c380661b0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f1c4400fa20 tx=0x7f1c44008040 comp rx=0 tx=0).stop 2022-06-27T11:52:29.520 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.514+0000 7f1c59443700 1 -- 172.21.15.195:0/154479586 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1c540c34a0 msgr2=0x7f1c54162000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:29.520 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.514+0000 7f1c59443700 1 --2- 172.21.15.195:0/154479586 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1c540c34a0 0x7f1c54162000 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f1c3c01b040 tx=0x7f1c3c018000 comp rx=0 tx=0).stop 2022-06-27T11:52:29.521 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.515+0000 7f1c59443700 1 -- 172.21.15.195:0/154479586 shutdown_connections 2022-06-27T11:52:29.521 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.515+0000 7f1c59443700 1 --2- 172.21.15.195:0/154479586 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1c38063d00 0x7f1c380661b0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:29.521 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.515+0000 7f1c59443700 1 --2- 172.21.15.195:0/154479586 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1c5414f080 0x7f1c54150310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:29.521 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.515+0000 7f1c59443700 1 --2- 172.21.15.195:0/154479586 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1c5406e850 0x7f1c54165fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:29.522 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.515+0000 7f1c59443700 1 --2- 172.21.15.195:0/154479586 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1c540c34a0 0x7f1c54162000 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:29.522 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.515+0000 7f1c59443700 1 -- 172.21.15.195:0/154479586 >> 172.21.15.195:0/154479586 conn(0x7f1c540c6350 msgr2=0x7f1c540c6720 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:52:29.523 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.515+0000 7f1c59443700 1 -- 172.21.15.195:0/154479586 shutdown_connections 2022-06-27T11:52:29.523 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:29.515+0000 7f1c59443700 1 -- 172.21.15.195:0/154479586 wait complete. 2022-06-27T11:52:29.523 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 19 2022-06-27T11:52:29.526 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-06-27T11:52:31.535 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:52:31.668 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.669+0000 7fd5add07700 1 Processor -- start 2022-06-27T11:52:31.668 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.669+0000 7fd5add07700 1 -- start start 2022-06-27T11:52:31.671 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.669+0000 7fd5add07700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd5a80c4160 0x7fd5a8156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:31.671 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.669+0000 7fd5add07700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fd5a80c39e0 0x7fd5a81535a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:31.672 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.669+0000 7fd5add07700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5a80c1a20 con 0x7fd5a806f640 2022-06-27T11:52:31.672 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.669+0000 7fd5add07700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5a80c1130 con 0x7fd5a80c39e0 2022-06-27T11:52:31.672 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.669+0000 7fd5add07700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a806ea90 con 0x7fd5a80c4160 2022-06-27T11:52:31.672 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.670+0000 7fd5a6ffd700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd5a80c4160 0x7fd5a8156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:31.672 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.670+0000 7fd5a7fff700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fd5a80c39e0 0x7fd5a81535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:31.673 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.670+0000 7fd5a7fff700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fd5a80c39e0 0x7fd5a81535a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3300/0 says I am v2:172.21.15.195:41744/0 (socket says 172.21.15.195:41744) 2022-06-27T11:52:31.673 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.670+0000 7fd5a77fe700 1 --1- >> v1:172.21.15.202:6789/0 conn(0x7fd5a806f640 0x7fd5a80c1eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.202:6789/0 says I am v1:172.21.15.195:51662/0 (socket says 172.21.15.195:51662) 2022-06-27T11:52:31.673 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.670+0000 7fd5a6ffd700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd5a80c4160 0x7fd5a8156ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3301/0 says I am v2:172.21.15.195:36372/0 (socket says 172.21.15.195:36372) 2022-06-27T11:52:31.673 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.670+0000 7fd5a7fff700 1 -- 172.21.15.195:0/1275740349 learned_addr learned my addr 172.21.15.195:0/1275740349 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:52:31.674 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.670+0000 7fd5a6ffd700 1 -- 172.21.15.195:0/1275740349 >> v2:172.21.15.202:3300/0 conn(0x7fd5a80c39e0 msgr2=0x7fd5a81535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:52:31.674 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.670+0000 7fd5a6ffd700 1 --2- 172.21.15.195:0/1275740349 >> v2:172.21.15.202:3300/0 conn(0x7fd5a80c39e0 0x7fd5a81535a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:31.674 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.670+0000 7fd5a6ffd700 1 -- 172.21.15.195:0/1275740349 >> v1:172.21.15.202:6789/0 conn(0x7fd5a806f640 legacy=0x7fd5a80c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:31.674 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.670+0000 7fd5a6ffd700 1 -- 172.21.15.195:0/1275740349 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5a806ed70 con 0x7fd5a80c4160 2022-06-27T11:52:31.675 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.670+0000 7fd5a7fff700 1 --2- 172.21.15.195:0/1275740349 >> v2:172.21.15.202:3300/0 conn(0x7fd5a80c39e0 0x7fd5a81535a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:52:31.675 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.670+0000 7fd5a6ffd700 1 --2- 172.21.15.195:0/1275740349 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd5a80c4160 0x7fd5a8156ed0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd59c0097a0 tx=0x7fd59c009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2cbc2085330626a1 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:52:31.675 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.670+0000 7fd5a67fc700 1 -- 172.21.15.195:0/1275740349 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd59c011630 con 0x7fd5a80c4160 2022-06-27T11:52:31.675 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.671+0000 7fd5a67fc700 1 -- 172.21.15.195:0/1275740349 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd59c011790 con 0x7fd5a80c4160 2022-06-27T11:52:31.675 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.671+0000 7fd5a67fc700 1 -- 172.21.15.195:0/1275740349 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd59c011b30 con 0x7fd5a80c4160 2022-06-27T11:52:31.676 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.671+0000 7fd5add07700 1 -- 172.21.15.195:0/1275740349 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd5a80c4160 msgr2=0x7fd5a8156ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:31.676 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.671+0000 7fd5add07700 1 --2- 172.21.15.195:0/1275740349 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd5a80c4160 0x7fd5a8156ed0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd59c0097a0 tx=0x7fd59c009060 comp rx=0 tx=0).stop 2022-06-27T11:52:31.676 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.671+0000 7fd5add07700 1 -- 172.21.15.195:0/1275740349 shutdown_connections 2022-06-27T11:52:31.677 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.671+0000 7fd5add07700 1 --2- 172.21.15.195:0/1275740349 >> v2:172.21.15.202:3300/0 conn(0x7fd5a80c39e0 0x7fd5a81535a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:31.677 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.671+0000 7fd5add07700 1 --2- 172.21.15.195:0/1275740349 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd5a80c4160 0x7fd5a8156ed0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:31.677 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.671+0000 7fd5add07700 1 -- 172.21.15.195:0/1275740349 >> 172.21.15.195:0/1275740349 conn(0x7fd5a80c6350 msgr2=0x7fd5a80c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:52:31.678 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.671+0000 7fd5add07700 1 -- 172.21.15.195:0/1275740349 shutdown_connections 2022-06-27T11:52:31.678 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.671+0000 7fd5add07700 1 -- 172.21.15.195:0/1275740349 wait complete. 2022-06-27T11:52:31.681 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.672+0000 7fd5add07700 1 Processor -- start 2022-06-27T11:52:31.682 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.672+0000 7fd5add07700 1 -- start start 2022-06-27T11:52:31.682 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.673+0000 7fd5add07700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd5a806f640 0x7fd5a8166ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:31.682 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.673+0000 7fd5add07700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd5a80c39e0 0x7fd5a8151b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:31.683 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.673+0000 7fd5add07700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd5a80c4160 0x7fd5a80c87d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:31.683 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.673+0000 7fd5add07700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a80bd7d0 con 0x7fd5a80c39e0 2022-06-27T11:52:31.683 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.673+0000 7fd5add07700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a8152430 con 0x7fd5a80c4160 2022-06-27T11:52:31.683 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.673+0000 7fd5add07700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a8152710 con 0x7fd5a806f640 2022-06-27T11:52:31.684 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.673+0000 7fd5a6ffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd5a80c39e0 0x7fd5a8151b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:31.684 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.673+0000 7fd5a6ffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd5a80c39e0 0x7fd5a8151b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:43622/0 (socket says 172.21.15.195:43622) 2022-06-27T11:52:31.684 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.673+0000 7fd5a6ffd700 1 -- 172.21.15.195:0/2526585746 learned_addr learned my addr 172.21.15.195:0/2526585746 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:52:31.684 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.673+0000 7fd5a77fe700 1 --2- 172.21.15.195:0/2526585746 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd5a806f640 0x7fd5a8166ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:31.685 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.673+0000 7fd5a7fff700 1 --2- 172.21.15.195:0/2526585746 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd5a80c4160 0x7fd5a80c87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:31.685 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.674+0000 7fd5a6ffd700 1 -- 172.21.15.195:0/2526585746 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd5a806f640 msgr2=0x7fd5a8166ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:31.685 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.674+0000 7fd5a6ffd700 1 --2- 172.21.15.195:0/2526585746 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd5a806f640 0x7fd5a8166ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:31.685 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.674+0000 7fd5a6ffd700 1 -- 172.21.15.195:0/2526585746 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd5a80c4160 msgr2=0x7fd5a80c87d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:31.685 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.674+0000 7fd5a6ffd700 1 --2- 172.21.15.195:0/2526585746 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd5a80c4160 0x7fd5a80c87d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:31.686 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.674+0000 7fd5a6ffd700 1 -- 172.21.15.195:0/2526585746 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5a0009ae0 con 0x7fd5a80c39e0 2022-06-27T11:52:31.686 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.674+0000 7fd5a6ffd700 1 --2- 172.21.15.195:0/2526585746 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd5a80c39e0 0x7fd5a8151b80 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fd59c009e40 tx=0x7fd59c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:52:31.686 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.674+0000 7fd58f7fe700 1 -- 172.21.15.195:0/2526585746 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd59c000c90 con 0x7fd5a80c39e0 2022-06-27T11:52:31.686 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.674+0000 7fd5add07700 1 -- 172.21.15.195:0/2526585746 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd59c0092e0 con 0x7fd5a80c39e0 2022-06-27T11:52:31.687 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.674+0000 7fd5add07700 1 -- 172.21.15.195:0/2526585746 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5a8169cd0 con 0x7fd5a80c39e0 2022-06-27T11:52:31.687 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.674+0000 7fd58f7fe700 1 -- 172.21.15.195:0/2526585746 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd59c011b10 con 0x7fd5a80c39e0 2022-06-27T11:52:31.687 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.674+0000 7fd58f7fe700 1 -- 172.21.15.195:0/2526585746 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd59c025420 con 0x7fd5a80c39e0 2022-06-27T11:52:31.688 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.675+0000 7fd5add07700 1 -- 172.21.15.195:0/2526585746 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd584005170 con 0x7fd5a80c39e0 2022-06-27T11:52:31.688 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.675+0000 7fd58f7fe700 1 -- 172.21.15.195:0/2526585746 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fd59c02c030 con 0x7fd5a80c39e0 2022-06-27T11:52:31.688 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.676+0000 7fd58f7fe700 1 --2- 172.21.15.195:0/2526585746 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd588063d00 0x7fd5880661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:31.689 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.676+0000 7fd58f7fe700 1 -- 172.21.15.195:0/2526585746 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6374+0+0 (secure 0 0 0) 0x7fd59c081e20 con 0x7fd5a80c39e0 2022-06-27T11:52:31.689 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.676+0000 7fd5a77fe700 1 --2- 172.21.15.195:0/2526585746 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd588063d00 0x7fd5880661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:31.689 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.676+0000 7fd5a77fe700 1 --2- 172.21.15.195:0/2526585746 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd588063d00 0x7fd5880661b0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd598007fd0 tx=0x7fd598007d70 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:52:31.689 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.680+0000 7fd58f7fe700 1 -- 172.21.15.195:0/2526585746 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fd59c050920 con 0x7fd5a80c39e0 2022-06-27T11:52:31.905 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.906+0000 7fd5add07700 1 -- 172.21.15.195:0/2526585746 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd584005dc0 con 0x7fd5a80c39e0 2022-06-27T11:52:31.907 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.907+0000 7fd58f7fe700 1 -- 172.21.15.195:0/2526585746 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+5367 (secure 0 0 0) 0x7fd59c015090 con 0x7fd5a80c39e0 2022-06-27T11:52:31.908 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:52:31.909 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:52:29.902078+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4760":{"gid":4760,"name":"b","rank":0,"incarnation":20,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6833/1026219246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1026219246},{"type":"v1","addr":"172.21.15.202:6833","nonce":1026219246}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:52:31.914 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.909+0000 7fd5add07700 1 -- 172.21.15.195:0/2526585746 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd588063d00 msgr2=0x7fd5880661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:31.915 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.909+0000 7fd5add07700 1 --2- 172.21.15.195:0/2526585746 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd588063d00 0x7fd5880661b0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd598007fd0 tx=0x7fd598007d70 comp rx=0 tx=0).stop 2022-06-27T11:52:31.916 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.909+0000 7fd5add07700 1 -- 172.21.15.195:0/2526585746 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd5a80c39e0 msgr2=0x7fd5a8151b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:31.916 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.909+0000 7fd5add07700 1 --2- 172.21.15.195:0/2526585746 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd5a80c39e0 0x7fd5a8151b80 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fd59c009e40 tx=0x7fd59c017000 comp rx=0 tx=0).stop 2022-06-27T11:52:31.917 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.909+0000 7fd5add07700 1 -- 172.21.15.195:0/2526585746 shutdown_connections 2022-06-27T11:52:31.917 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.909+0000 7fd5add07700 1 --2- 172.21.15.195:0/2526585746 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd588063d00 0x7fd5880661b0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:31.919 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.909+0000 7fd5add07700 1 --2- 172.21.15.195:0/2526585746 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd5a80c4160 0x7fd5a80c87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:31.919 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.909+0000 7fd5add07700 1 --2- 172.21.15.195:0/2526585746 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd5a806f640 0x7fd5a8166ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:31.919 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.909+0000 7fd5add07700 1 --2- 172.21.15.195:0/2526585746 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd5a80c39e0 0x7fd5a8151b80 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:31.920 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.909+0000 7fd5add07700 1 -- 172.21.15.195:0/2526585746 >> 172.21.15.195:0/2526585746 conn(0x7fd5a80c6350 msgr2=0x7fd5a81506e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:52:31.921 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.909+0000 7fd5add07700 1 -- 172.21.15.195:0/2526585746 shutdown_connections 2022-06-27T11:52:31.922 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:31.909+0000 7fd5add07700 1 -- 172.21.15.195:0/2526585746 wait complete. 2022-06-27T11:52:31.922 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 20 2022-06-27T11:52:31.926 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d down, removed from mdsmap 2022-06-27T11:52:31.926 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2022-06-27T11:52:31.930 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:52:32.050 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.051+0000 7f8934b79700 1 Processor -- start 2022-06-27T11:52:32.051 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.051+0000 7f8934b79700 1 -- start start 2022-06-27T11:52:32.054 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.052+0000 7f8934b79700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f893006f640 0x7f8930156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:32.055 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.052+0000 7f8934b79700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f89300c39e0 0x7f89301535a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:32.055 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.052+0000 7f8934b79700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89300c1a20 con 0x7f89300c4160 2022-06-27T11:52:32.055 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.052+0000 7f8934b79700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89300c1130 con 0x7f89300c39e0 2022-06-27T11:52:32.055 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.052+0000 7f8934b79700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f893006ea90 con 0x7f893006f640 2022-06-27T11:52:32.056 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.053+0000 7f892ed9d700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f89300c39e0 0x7f89301535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:32.056 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.053+0000 7f892e59c700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f893006f640 0x7f8930156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:32.056 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.053+0000 7f892ed9d700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f89300c39e0 0x7f89301535a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3300/0 says I am v2:172.21.15.195:41758/0 (socket says 172.21.15.195:41758) 2022-06-27T11:52:32.056 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.053+0000 7f892e59c700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f893006f640 0x7f8930156ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3301/0 says I am v2:172.21.15.195:36384/0 (socket says 172.21.15.195:36384) 2022-06-27T11:52:32.057 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.053+0000 7f892ed9d700 1 -- 172.21.15.195:0/3292848535 learned_addr learned my addr 172.21.15.195:0/3292848535 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:52:32.057 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.053+0000 7f892dd9b700 1 --1- 172.21.15.195:0/3292848535 >> v1:172.21.15.202:6789/0 conn(0x7f89300c4160 0x7f89300c1eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.202:6789/0 says I am v1:172.21.15.195:51676/0 (socket says 172.21.15.195:51676) 2022-06-27T11:52:32.058 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.053+0000 7f892e59c700 1 -- 172.21.15.195:0/3292848535 >> v2:172.21.15.202:3300/0 conn(0x7f89300c39e0 msgr2=0x7f89301535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:52:32.058 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.053+0000 7f892e59c700 1 --2- 172.21.15.195:0/3292848535 >> v2:172.21.15.202:3300/0 conn(0x7f89300c39e0 0x7f89301535a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:32.058 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.053+0000 7f892e59c700 1 -- 172.21.15.195:0/3292848535 >> v1:172.21.15.202:6789/0 conn(0x7f89300c4160 legacy=0x7f89300c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:32.058 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.053+0000 7f892e59c700 1 -- 172.21.15.195:0/3292848535 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f893006ed70 con 0x7f893006f640 2022-06-27T11:52:32.059 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.053+0000 7f892ed9d700 1 --2- 172.21.15.195:0/3292848535 >> v2:172.21.15.202:3300/0 conn(0x7f89300c39e0 0x7f89301535a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:52:32.059 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.053+0000 7f892e59c700 1 --2- 172.21.15.195:0/3292848535 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f893006f640 0x7f8930156ed0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f89240097a0 tx=0x7f8924009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5068acb0cea1d73e server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:52:32.059 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.053+0000 7f892d59a700 1 -- 172.21.15.195:0/3292848535 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8924011630 con 0x7f893006f640 2022-06-27T11:52:32.060 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.053+0000 7f892d59a700 1 -- 172.21.15.195:0/3292848535 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8924011790 con 0x7f893006f640 2022-06-27T11:52:32.060 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.053+0000 7f892d59a700 1 -- 172.21.15.195:0/3292848535 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8924011b30 con 0x7f893006f640 2022-06-27T11:52:32.060 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.054+0000 7f8934b79700 1 -- 172.21.15.195:0/3292848535 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f893006f640 msgr2=0x7f8930156ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:32.060 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.054+0000 7f8934b79700 1 --2- 172.21.15.195:0/3292848535 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f893006f640 0x7f8930156ed0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f89240097a0 tx=0x7f8924009060 comp rx=0 tx=0).stop 2022-06-27T11:52:32.061 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.054+0000 7f8934b79700 1 -- 172.21.15.195:0/3292848535 shutdown_connections 2022-06-27T11:52:32.061 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.054+0000 7f8934b79700 1 --2- 172.21.15.195:0/3292848535 >> v2:172.21.15.202:3300/0 conn(0x7f89300c39e0 0x7f89301535a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:32.061 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.054+0000 7f8934b79700 1 --2- 172.21.15.195:0/3292848535 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f893006f640 0x7f8930156ed0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:32.061 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.054+0000 7f8934b79700 1 -- 172.21.15.195:0/3292848535 >> 172.21.15.195:0/3292848535 conn(0x7f89300c6350 msgr2=0x7f89300c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:52:32.061 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.054+0000 7f8934b79700 1 -- 172.21.15.195:0/3292848535 shutdown_connections 2022-06-27T11:52:32.062 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.054+0000 7f8934b79700 1 -- 172.21.15.195:0/3292848535 wait complete. 2022-06-27T11:52:32.062 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.055+0000 7f8934b79700 1 Processor -- start 2022-06-27T11:52:32.062 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.055+0000 7f8934b79700 1 -- start start 2022-06-27T11:52:32.066 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.056+0000 7f8934b79700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f893006f640 0x7f89300bc6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:32.066 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.056+0000 7f8934b79700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f89300c39e0 0x7f8930166870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:32.066 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.056+0000 7f8934b79700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f89300c4160 0x7f8930166db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:32.067 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.056+0000 7f8934b79700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8930069f10 con 0x7f89300c39e0 2022-06-27T11:52:32.067 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.056+0000 7f8934b79700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8930166450 con 0x7f89300c4160 2022-06-27T11:52:32.067 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.056+0000 7f8934b79700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8930068ab0 con 0x7f893006f640 2022-06-27T11:52:32.068 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.056+0000 7f892dd9b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f89300c39e0 0x7f8930166870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:32.068 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.056+0000 7f892e59c700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f893006f640 0x7f89300bc6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:32.069 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.056+0000 7f892dd9b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f89300c39e0 0x7f8930166870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:43636/0 (socket says 172.21.15.195:43636) 2022-06-27T11:52:32.069 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.056+0000 7f892dd9b700 1 -- 172.21.15.195:0/1751169330 learned_addr learned my addr 172.21.15.195:0/1751169330 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:52:32.069 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.056+0000 7f892ed9d700 1 --2- 172.21.15.195:0/1751169330 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f89300c4160 0x7f8930166db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:32.069 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.056+0000 7f892e59c700 1 -- 172.21.15.195:0/1751169330 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f89300c4160 msgr2=0x7f8930166db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:32.070 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.056+0000 7f892e59c700 1 --2- 172.21.15.195:0/1751169330 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f89300c4160 0x7f8930166db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:32.070 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.056+0000 7f892e59c700 1 -- 172.21.15.195:0/1751169330 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f89300c39e0 msgr2=0x7f8930166870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:32.070 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.056+0000 7f892e59c700 1 --2- 172.21.15.195:0/1751169330 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f89300c39e0 0x7f8930166870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:32.071 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.056+0000 7f892e59c700 1 -- 172.21.15.195:0/1751169330 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8928009ae0 con 0x7f893006f640 2022-06-27T11:52:32.071 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.056+0000 7f892e59c700 1 --2- 172.21.15.195:0/1751169330 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f893006f640 0x7f89300bc6e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8924016000 tx=0x7f8924017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:52:32.071 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.057+0000 7f891e7fc700 1 -- 172.21.15.195:0/1751169330 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f892401be00 con 0x7f893006f640 2022-06-27T11:52:32.071 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.057+0000 7f891e7fc700 1 -- 172.21.15.195:0/1751169330 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8924011900 con 0x7f893006f640 2022-06-27T11:52:32.072 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.057+0000 7f891e7fc700 1 -- 172.21.15.195:0/1751169330 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8924025420 con 0x7f893006f640 2022-06-27T11:52:32.072 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.057+0000 7f8934b79700 1 -- 172.21.15.195:0/1751169330 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89240092e0 con 0x7f893006f640 2022-06-27T11:52:32.072 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.057+0000 7f8934b79700 1 -- 172.21.15.195:0/1751169330 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f893006ac10 con 0x7f893006f640 2022-06-27T11:52:32.073 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.058+0000 7f8934b79700 1 -- 172.21.15.195:0/1751169330 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89301666c0 con 0x7f893006f640 2022-06-27T11:52:32.073 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.058+0000 7f891e7fc700 1 -- 172.21.15.195:0/1751169330 <== mon.2 v2:172.21.15.202:3301/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f892400f050 con 0x7f893006f640 2022-06-27T11:52:32.073 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.059+0000 7f891e7fc700 1 --2- 172.21.15.195:0/1751169330 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f890c063d30 0x7f890c0661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:32.073 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.059+0000 7f891e7fc700 1 -- 172.21.15.195:0/1751169330 <== mon.2 v2:172.21.15.202:3301/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6374+0+0 (secure 0 0 0) 0x7f8924081f70 con 0x7f893006f640 2022-06-27T11:52:32.074 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.059+0000 7f892dd9b700 1 --2- 172.21.15.195:0/1751169330 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f890c063d30 0x7f890c0661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:32.074 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.059+0000 7f892dd9b700 1 --2- 172.21.15.195:0/1751169330 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f890c063d30 0x7f890c0661e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f89300beb70 tx=0x7f8920007d70 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:52:32.074 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.063+0000 7f891e7fc700 1 -- 172.21.15.195:0/1751169330 <== mon.2 v2:172.21.15.202:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f8924050aa0 con 0x7f893006f640 2022-06-27T11:52:32.291 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.291+0000 7f8934b79700 1 -- 172.21.15.195:0/1751169330 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f89300722c0 con 0x7f893006f640 2022-06-27T11:52:32.292 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.292+0000 7f891e7fc700 1 -- 172.21.15.195:0/1751169330 <== mon.2 v2:172.21.15.202:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+5367 (secure 0 0 0) 0x7f892402a070 con 0x7f893006f640 2022-06-27T11:52:32.292 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:52:32.292 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:52:29.902078+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4760":{"gid":4760,"name":"b","rank":0,"incarnation":20,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6833/1026219246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1026219246},{"type":"v1","addr":"172.21.15.202:6833","nonce":1026219246}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:52:32.297 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.294+0000 7f8934b79700 1 -- 172.21.15.195:0/1751169330 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f890c063d30 msgr2=0x7f890c0661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:32.297 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.294+0000 7f8934b79700 1 --2- 172.21.15.195:0/1751169330 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f890c063d30 0x7f890c0661e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f89300beb70 tx=0x7f8920007d70 comp rx=0 tx=0).stop 2022-06-27T11:52:32.298 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.294+0000 7f8934b79700 1 -- 172.21.15.195:0/1751169330 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f893006f640 msgr2=0x7f89300bc6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:32.298 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.294+0000 7f8934b79700 1 --2- 172.21.15.195:0/1751169330 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f893006f640 0x7f89300bc6e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8924016000 tx=0x7f8924017000 comp rx=0 tx=0).stop 2022-06-27T11:52:32.299 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.294+0000 7f8934b79700 1 -- 172.21.15.195:0/1751169330 shutdown_connections 2022-06-27T11:52:32.299 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.294+0000 7f8934b79700 1 --2- 172.21.15.195:0/1751169330 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f890c063d30 0x7f890c0661e0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:32.300 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.294+0000 7f8934b79700 1 --2- 172.21.15.195:0/1751169330 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f89300c4160 0x7f8930166db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:32.300 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.294+0000 7f8934b79700 1 --2- 172.21.15.195:0/1751169330 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f893006f640 0x7f89300bc6e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:32.300 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.294+0000 7f8934b79700 1 --2- 172.21.15.195:0/1751169330 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f89300c39e0 0x7f8930166870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:32.301 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.294+0000 7f8934b79700 1 -- 172.21.15.195:0/1751169330 >> 172.21.15.195:0/1751169330 conn(0x7f89300c6350 msgr2=0x7f89300c1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:52:32.301 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.294+0000 7f8934b79700 1 -- 172.21.15.195:0/1751169330 shutdown_connections 2022-06-27T11:52:32.302 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:32.294+0000 7f8934b79700 1 -- 172.21.15.195:0/1751169330 wait complete. 2022-06-27T11:52:32.302 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 20 2022-06-27T11:52:32.307 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 20, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-06-27T11:47:52.358095+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 20, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4760": { "addr": "172.21.15.202:6833/1026219246", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6832", "nonce": 1026219246, "type": "v2" }, { "addr": "172.21.15.202:6833", "nonce": 1026219246, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4760, "incarnation": 20, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 41, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-06-27T11:52:29.902078+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4760 } } } ], "standbys": [ { "addr": "172.21.15.202:6837/3305006841", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6836", "nonce": 3305006841, "type": "v2" }, { "addr": "172.21.15.202:6837", "nonce": 3305006841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6835/2060709822", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6834", "nonce": 2060709822, "type": "v2" }, { "addr": "172.21.15.195:6835", "nonce": 2060709822, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6838/1598751652", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6836", "nonce": 1598751652, "type": "v2" }, { "addr": "172.21.15.195:6838", "nonce": 1598751652, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4275, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6839/1313800661", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6837", "nonce": 1313800661, "type": "v2" }, { "addr": "172.21.15.195:6839", "nonce": 1313800661, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4278, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-06-27T11:52:34.307 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:52:34.480 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.454+0000 7f7aefe0a700 1 Processor -- start 2022-06-27T11:52:34.481 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.454+0000 7f7aefe0a700 1 -- start start 2022-06-27T11:52:34.482 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.455+0000 7f7aefe0a700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7ae806f640 0x7f7ae8156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:34.482 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.455+0000 7f7aefe0a700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f7ae80c39e0 0x7f7ae81535a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:34.482 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.455+0000 7f7aefe0a700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ae80c1a20 con 0x7f7ae80c4160 2022-06-27T11:52:34.483 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.455+0000 7f7aefe0a700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ae80c1130 con 0x7f7ae80c39e0 2022-06-27T11:52:34.484 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.455+0000 7f7aefe0a700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae806ea90 con 0x7f7ae806f640 2022-06-27T11:52:34.484 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.455+0000 7f7aee3a7700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f7ae80c39e0 0x7f7ae81535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:34.485 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.455+0000 7f7aedba6700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7ae806f640 0x7f7ae8156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:34.485 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.456+0000 7f7aee3a7700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f7ae80c39e0 0x7f7ae81535a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3300/0 says I am v2:172.21.15.195:41772/0 (socket says 172.21.15.195:41772) 2022-06-27T11:52:34.486 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.456+0000 7f7aedba6700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7ae806f640 0x7f7ae8156ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3301/0 says I am v2:172.21.15.195:36398/0 (socket says 172.21.15.195:36398) 2022-06-27T11:52:34.486 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.456+0000 7f7aee3a7700 1 -- 172.21.15.195:0/1107952363 learned_addr learned my addr 172.21.15.195:0/1107952363 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:52:34.487 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.456+0000 7f7aed3a5700 1 --1- >> v1:172.21.15.202:6789/0 conn(0x7f7ae80c4160 0x7f7ae80c1eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.202:6789/0 says I am v1:172.21.15.195:51692/0 (socket says 172.21.15.195:51692) 2022-06-27T11:52:34.487 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.456+0000 7f7aee3a7700 1 -- 172.21.15.195:0/1107952363 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7ae806f640 msgr2=0x7f7ae8156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:52:34.488 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.456+0000 7f7aee3a7700 1 --2- 172.21.15.195:0/1107952363 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7ae806f640 0x7f7ae8156ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:34.488 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.456+0000 7f7aee3a7700 1 -- 172.21.15.195:0/1107952363 >> v1:172.21.15.202:6789/0 conn(0x7f7ae80c4160 legacy=0x7f7ae80c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:34.489 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.456+0000 7f7aee3a7700 1 -- 172.21.15.195:0/1107952363 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ae806ed70 con 0x7f7ae80c39e0 2022-06-27T11:52:34.490 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.456+0000 7f7aedba6700 1 --2- 172.21.15.195:0/1107952363 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7ae806f640 0x7f7ae8156ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:52:34.490 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.456+0000 7f7aee3a7700 1 --2- 172.21.15.195:0/1107952363 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7ae80c39e0 0x7f7ae81535a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7ae0009f20 tx=0x7f7ae0009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c7a2e34d0f697886 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:52:34.490 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.456+0000 7f7aecba4700 1 -- 172.21.15.195:0/1107952363 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae000e070 con 0x7f7ae80c39e0 2022-06-27T11:52:34.491 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.456+0000 7f7aecba4700 1 -- 172.21.15.195:0/1107952363 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ae000ba60 con 0x7f7ae80c39e0 2022-06-27T11:52:34.492 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.456+0000 7f7aecba4700 1 -- 172.21.15.195:0/1107952363 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae00092e0 con 0x7f7ae80c39e0 2022-06-27T11:52:34.492 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.457+0000 7f7aefe0a700 1 -- 172.21.15.195:0/1107952363 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7ae80c39e0 msgr2=0x7f7ae81535a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:34.493 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.457+0000 7f7aefe0a700 1 --2- 172.21.15.195:0/1107952363 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7ae80c39e0 0x7f7ae81535a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7ae0009f20 tx=0x7f7ae0009840 comp rx=0 tx=0).stop 2022-06-27T11:52:34.493 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.457+0000 7f7aefe0a700 1 -- 172.21.15.195:0/1107952363 shutdown_connections 2022-06-27T11:52:34.494 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.457+0000 7f7aefe0a700 1 --2- 172.21.15.195:0/1107952363 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7ae80c39e0 0x7f7ae81535a0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:34.499 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.457+0000 7f7aefe0a700 1 --2- 172.21.15.195:0/1107952363 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7ae806f640 0x7f7ae8156ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:34.500 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.457+0000 7f7aefe0a700 1 -- 172.21.15.195:0/1107952363 >> 172.21.15.195:0/1107952363 conn(0x7f7ae80c6350 msgr2=0x7f7ae80c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:52:34.500 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.457+0000 7f7aefe0a700 1 -- 172.21.15.195:0/1107952363 shutdown_connections 2022-06-27T11:52:34.501 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.457+0000 7f7aefe0a700 1 -- 172.21.15.195:0/1107952363 wait complete. 2022-06-27T11:52:34.501 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.458+0000 7f7aefe0a700 1 Processor -- start 2022-06-27T11:52:34.501 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.458+0000 7f7aefe0a700 1 -- start start 2022-06-27T11:52:34.502 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.459+0000 7f7aefe0a700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7ae806f640 0x7f7ae814f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:34.502 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.459+0000 7f7aefe0a700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7ae80c39e0 0x7f7ae80ba330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:34.503 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.459+0000 7f7aefe0a700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7ae80c4160 0x7f7ae81546e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:34.503 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.459+0000 7f7aefe0a700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae80bc020 con 0x7f7ae80c39e0 2022-06-27T11:52:34.504 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.459+0000 7f7aefe0a700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae80bd030 con 0x7f7ae806f640 2022-06-27T11:52:34.504 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.459+0000 7f7aefe0a700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae8156230 con 0x7f7ae80c4160 2022-06-27T11:52:34.505 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.459+0000 7f7aed3a5700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7ae80c39e0 0x7f7ae80ba330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:34.505 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.459+0000 7f7aed3a5700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7ae80c39e0 0x7f7ae80ba330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:43650/0 (socket says 172.21.15.195:43650) 2022-06-27T11:52:34.505 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.459+0000 7f7aed3a5700 1 -- 172.21.15.195:0/2774811990 learned_addr learned my addr 172.21.15.195:0/2774811990 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:52:34.506 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.459+0000 7f7aedba6700 1 --2- 172.21.15.195:0/2774811990 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7ae806f640 0x7f7ae814f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:34.506 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.459+0000 7f7aee3a7700 1 --2- 172.21.15.195:0/2774811990 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7ae80c4160 0x7f7ae81546e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:34.507 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.459+0000 7f7aed3a5700 1 -- 172.21.15.195:0/2774811990 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7ae80c4160 msgr2=0x7f7ae81546e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:34.507 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.459+0000 7f7aed3a5700 1 --2- 172.21.15.195:0/2774811990 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7ae80c4160 0x7f7ae81546e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:34.508 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.459+0000 7f7aed3a5700 1 -- 172.21.15.195:0/2774811990 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7ae806f640 msgr2=0x7f7ae814f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:34.508 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.459+0000 7f7aed3a5700 1 --2- 172.21.15.195:0/2774811990 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7ae806f640 0x7f7ae814f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:34.509 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.459+0000 7f7aed3a5700 1 -- 172.21.15.195:0/2774811990 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ad80092e0 con 0x7f7ae80c39e0 2022-06-27T11:52:34.509 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.459+0000 7f7aedba6700 1 --2- 172.21.15.195:0/2774811990 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7ae806f640 0x7f7ae814f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:52:34.509 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.459+0000 7f7aed3a5700 1 --2- 172.21.15.195:0/2774811990 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7ae80c39e0 0x7f7ae80ba330 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f7ae400c440 tx=0x7f7ae4007d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:52:34.511 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.460+0000 7f7ad5ffb700 1 -- 172.21.15.195:0/2774811990 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae400ce60 con 0x7f7ae80c39e0 2022-06-27T11:52:34.511 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.460+0000 7f7ad5ffb700 1 -- 172.21.15.195:0/2774811990 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ae40072e0 con 0x7f7ae80c39e0 2022-06-27T11:52:34.511 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.460+0000 7f7aefe0a700 1 -- 172.21.15.195:0/2774811990 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ae0009ae0 con 0x7f7ae80c39e0 2022-06-27T11:52:34.512 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.460+0000 7f7ad5ffb700 1 -- 172.21.15.195:0/2774811990 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae40046f0 con 0x7f7ae80c39e0 2022-06-27T11:52:34.512 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.460+0000 7f7aefe0a700 1 -- 172.21.15.195:0/2774811990 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ae80bcd00 con 0x7f7ae80c39e0 2022-06-27T11:52:34.513 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.461+0000 7f7aefe0a700 1 -- 172.21.15.195:0/2774811990 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ae80722c0 con 0x7f7ae80c39e0 2022-06-27T11:52:34.513 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.461+0000 7f7ad5ffb700 1 -- 172.21.15.195:0/2774811990 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f7ae4010070 con 0x7f7ae80c39e0 2022-06-27T11:52:34.513 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.461+0000 7f7ad5ffb700 1 --2- 172.21.15.195:0/2774811990 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7ac8063d00 0x7f7ac80661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:34.514 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.462+0000 7f7aedba6700 1 --2- 172.21.15.195:0/2774811990 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7ac8063d00 0x7f7ac80661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:34.514 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.462+0000 7f7ad5ffb700 1 -- 172.21.15.195:0/2774811990 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6374+0+0 (secure 0 0 0) 0x7f7ae407f9c0 con 0x7f7ae80c39e0 2022-06-27T11:52:34.515 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.462+0000 7f7aedba6700 1 --2- 172.21.15.195:0/2774811990 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7ac8063d00 0x7f7ac80661b0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7ad80114c0 tx=0x7f7ad8009060 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:52:34.515 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.466+0000 7f7ad5ffb700 1 -- 172.21.15.195:0/2774811990 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f7ae4049c90 con 0x7f7ae80c39e0 2022-06-27T11:52:34.691 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.691+0000 7f7aefe0a700 1 -- 172.21.15.195:0/2774811990 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7ae8157b70 con 0x7f7ae80c39e0 2022-06-27T11:52:34.692 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.692+0000 7f7ad5ffb700 1 -- 172.21.15.195:0/2774811990 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+5367 (secure 0 0 0) 0x7f7ae404e380 con 0x7f7ae80c39e0 2022-06-27T11:52:34.692 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:52:34.693 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:52:29.902078+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4760":{"gid":4760,"name":"b","rank":0,"incarnation":20,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6833/1026219246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1026219246},{"type":"v1","addr":"172.21.15.202:6833","nonce":1026219246}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:52:34.694 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.694+0000 7f7aefe0a700 1 -- 172.21.15.195:0/2774811990 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7ac8063d00 msgr2=0x7f7ac80661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:34.695 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.694+0000 7f7aefe0a700 1 --2- 172.21.15.195:0/2774811990 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7ac8063d00 0x7f7ac80661b0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7ad80114c0 tx=0x7f7ad8009060 comp rx=0 tx=0).stop 2022-06-27T11:52:34.695 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.694+0000 7f7aefe0a700 1 -- 172.21.15.195:0/2774811990 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7ae80c39e0 msgr2=0x7f7ae80ba330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:34.696 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.694+0000 7f7aefe0a700 1 --2- 172.21.15.195:0/2774811990 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7ae80c39e0 0x7f7ae80ba330 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f7ae400c440 tx=0x7f7ae4007d70 comp rx=0 tx=0).stop 2022-06-27T11:52:34.696 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.694+0000 7f7aefe0a700 1 -- 172.21.15.195:0/2774811990 shutdown_connections 2022-06-27T11:52:34.696 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.694+0000 7f7aefe0a700 1 --2- 172.21.15.195:0/2774811990 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7ac8063d00 0x7f7ac80661b0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:34.697 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.694+0000 7f7aefe0a700 1 --2- 172.21.15.195:0/2774811990 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7ae80c4160 0x7f7ae81546e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:34.697 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.694+0000 7f7aefe0a700 1 --2- 172.21.15.195:0/2774811990 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7ae806f640 0x7f7ae814f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:34.706 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.694+0000 7f7aefe0a700 1 --2- 172.21.15.195:0/2774811990 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7ae80c39e0 0x7f7ae80ba330 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:34.707 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.694+0000 7f7aefe0a700 1 -- 172.21.15.195:0/2774811990 >> 172.21.15.195:0/2774811990 conn(0x7f7ae80c6350 msgr2=0x7f7ae8153370 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:52:34.707 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.694+0000 7f7aefe0a700 1 -- 172.21.15.195:0/2774811990 shutdown_connections 2022-06-27T11:52:34.707 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:34.694+0000 7f7aefe0a700 1 -- 172.21.15.195:0/2774811990 wait complete. 2022-06-27T11:52:34.708 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 20 2022-06-27T11:52:34.708 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T11:52:36.709 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:52:36.857 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:36.852+0000 7f71f57d3700 1 Processor -- start 2022-06-27T11:52:36.858 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:36.852+0000 7f71f57d3700 1 -- start start 2022-06-27T11:52:36.858 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:36.853+0000 7f71f57d3700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f71f0068f80 0x7f71f00c4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:36.858 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:36.853+0000 7f71f57d3700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71f01573b0 con 0x7f71f00c7480 2022-06-27T11:52:36.859 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:36.853+0000 7f71f57d3700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71f00c46c0 con 0x7f71f00683c0 2022-06-27T11:52:36.859 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:36.853+0000 7f71f57d3700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71f0068860 con 0x7f71f0068f80 2022-06-27T11:52:36.859 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:36.853+0000 7f71ee7fc700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f71f00c7480 0x7f71f0156460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:49844/0 (socket says 172.21.15.195:49844) 2022-06-27T11:52:36.859 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:36.853+0000 7f71ee7fc700 1 -- 172.21.15.195:0/2719382396 learned_addr learned my addr 172.21.15.195:0/2719382396 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:52:36.860 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:36.853+0000 7f71eeffd700 1 --2- 172.21.15.195:0/2719382396 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f71f0068f80 0x7f71f00c4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:36.860 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:36.853+0000 7f71edffb700 1 -- 172.21.15.195:0/2719382396 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3165956541 0 0) 0x7f71f01573b0 con 0x7f71f00c7480 2022-06-27T11:52:36.860 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:36.853+0000 7f71edffb700 1 -- 172.21.15.195:0/2719382396 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71dc003430 con 0x7f71f00c7480 2022-06-27T11:52:36.860 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:36.853+0000 7f71edffb700 1 -- 172.21.15.195:0/2719382396 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 909304201 0 0) 0x7f71f00c46c0 con 0x7f71f00683c0 2022-06-27T11:52:36.861 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:36.853+0000 7f71edffb700 1 -- 172.21.15.195:0/2719382396 --> v1:172.21.15.202:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71f01573b0 con 0x7f71f00683c0 2022-06-27T11:52:36.861 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:36.854+0000 7f71eeffd700 1 -- 172.21.15.195:0/2719382396 >> v1:172.21.15.202:6789/0 conn(0x7f71f00683c0 legacy=0x7f71f00c6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:36.862 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:36.854+0000 7f71edffb700 1 -- 172.21.15.195:0/2719382396 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 669177930 0 0) 0x7f71dc003430 con 0x7f71f00c7480 2022-06-27T11:52:36.862 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:36.854+0000 7f71eeffd700 1 -- 172.21.15.195:0/2719382396 >> v1:172.21.15.195:6789/0 conn(0x7f71f00c7480 legacy=0x7f71f0156460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:36.863 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:36.854+0000 7f71eeffd700 1 -- 172.21.15.195:0/2719382396 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71f0156b70 con 0x7f71f0068f80 2022-06-27T11:52:36.863 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:36.854+0000 7f71eeffd700 1 --2- 172.21.15.195:0/2719382396 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f71f0068f80 0x7f71f00c4b50 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f71e40097d0 tx=0x7f71e4009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4cb90f02e0661c9 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:52:36.864 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:36.854+0000 7f71edffb700 1 -- 172.21.15.195:0/2719382396 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e0001470 con 0x7f71f0068f80 2022-06-27T11:52:36.864 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:36.854+0000 7f71edffb700 1 -- 172.21.15.195:0/2719382396 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71e4004030 con 0x7f71f0068f80 2022-06-27T11:52:36.864 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:36.854+0000 7f71edffb700 1 -- 172.21.15.195:0/2719382396 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e40043b0 con 0x7f71f0068f80 2022-06-27T11:52:36.865 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:36.854+0000 7f71f57d3700 1 -- 172.21.15.195:0/2719382396 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f71f0068f80 msgr2=0x7f71f00c4b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:36.865 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:36.854+0000 7f71f57d3700 1 --2- 172.21.15.195:0/2719382396 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f71f0068f80 0x7f71f00c4b50 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f71e40097d0 tx=0x7f71e4009060 comp rx=0 tx=0).stop 2022-06-27T11:52:36.866 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:36.854+0000 7f71f57d3700 1 -- 172.21.15.195:0/2719382396 shutdown_connections 2022-06-27T11:52:36.866 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:36.854+0000 7f71f57d3700 1 --2- 172.21.15.195:0/2719382396 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f71f0068f80 0x7f71f00c4b50 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:36.867 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:36.854+0000 7f71f57d3700 1 -- 172.21.15.195:0/2719382396 >> 172.21.15.195:0/2719382396 conn(0x7f71f00ccbd0 msgr2=0x7f71f00ccfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:52:36.867 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:36.854+0000 7f71f57d3700 1 -- 172.21.15.195:0/2719382396 shutdown_connections 2022-06-27T11:52:36.868 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:36.854+0000 7f71f57d3700 1 -- 172.21.15.195:0/2719382396 wait complete. 2022-06-27T11:52:36.868 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:36.856+0000 7f71f57d3700 1 Processor -- start 2022-06-27T11:52:36.868 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:36.856+0000 7f71f57d3700 1 -- start start 2022-06-27T11:52:36.869 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:36.856+0000 7f71f57d3700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f71f00683c0 0x7f71f0166280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:36.869 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:36.856+0000 7f71f57d3700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f71f0068f80 0x7f71f00c6680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:36.870 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:36.857+0000 7f71f57d3700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f71f00c7480 0x7f71f0157630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:36.870 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:36.857+0000 7f71f57d3700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71f0151200 con 0x7f71f00683c0 2022-06-27T11:52:36.871 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:36.857+0000 7f71f57d3700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71f01514e0 con 0x7f71f00c7480 2022-06-27T11:52:36.871 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:36.857+0000 7f71f57d3700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71f0169970 con 0x7f71f0068f80 2022-06-27T11:52:36.872 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:36.857+0000 7f71eeffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f71f00683c0 0x7f71f0166280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:36.872 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:36.857+0000 7f71eeffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f71f00683c0 0x7f71f0166280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:43662/0 (socket says 172.21.15.195:43662) 2022-06-27T11:52:36.873 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:36.857+0000 7f71eeffd700 1 -- 172.21.15.195:0/2125731349 learned_addr learned my addr 172.21.15.195:0/2125731349 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:52:36.873 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:36.857+0000 7f71ee7fc700 1 --2- 172.21.15.195:0/2125731349 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f71f0068f80 0x7f71f00c6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:36.873 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:36.857+0000 7f71ef7fe700 1 --2- 172.21.15.195:0/2125731349 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f71f00c7480 0x7f71f0157630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:36.875 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:36.857+0000 7f71eeffd700 1 -- 172.21.15.195:0/2125731349 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f71f0068f80 msgr2=0x7f71f00c6680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:36.875 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:36.857+0000 7f71eeffd700 1 --2- 172.21.15.195:0/2125731349 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f71f0068f80 0x7f71f00c6680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:36.876 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:36.857+0000 7f71eeffd700 1 -- 172.21.15.195:0/2125731349 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f71f00c7480 msgr2=0x7f71f0157630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:36.876 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:36.857+0000 7f71eeffd700 1 --2- 172.21.15.195:0/2125731349 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f71f00c7480 0x7f71f0157630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:36.877 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:36.857+0000 7f71eeffd700 1 -- 172.21.15.195:0/2125731349 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71e40092e0 con 0x7f71f00683c0 2022-06-27T11:52:36.877 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:36.857+0000 7f71ee7fc700 1 --2- 172.21.15.195:0/2125731349 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f71f0068f80 0x7f71f00c6680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:52:36.878 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:36.857+0000 7f71eeffd700 1 --2- 172.21.15.195:0/2125731349 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f71f00683c0 0x7f71f0166280 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f71f00cdc40 tx=0x7f71e4015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:52:36.878 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:36.857+0000 7f71d6ffd700 1 -- 172.21.15.195:0/2125731349 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e4000c60 con 0x7f71f00683c0 2022-06-27T11:52:36.879 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:36.857+0000 7f71d6ffd700 1 -- 172.21.15.195:0/2125731349 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71e4011650 con 0x7f71f00683c0 2022-06-27T11:52:36.879 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:36.857+0000 7f71f57d3700 1 -- 172.21.15.195:0/2125731349 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71f0169c50 con 0x7f71f00683c0 2022-06-27T11:52:36.880 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:36.857+0000 7f71d6ffd700 1 -- 172.21.15.195:0/2125731349 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e40119f0 con 0x7f71f00683c0 2022-06-27T11:52:36.880 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:36.857+0000 7f71f57d3700 1 -- 172.21.15.195:0/2125731349 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71f0150ec0 con 0x7f71f00683c0 2022-06-27T11:52:36.881 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:36.858+0000 7f71f57d3700 1 -- 172.21.15.195:0/2125731349 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71f01697d0 con 0x7f71f00683c0 2022-06-27T11:52:36.881 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:36.859+0000 7f71d6ffd700 1 -- 172.21.15.195:0/2125731349 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f71e400f070 con 0x7f71f00683c0 2022-06-27T11:52:36.884 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:36.859+0000 7f71d6ffd700 1 --2- 172.21.15.195:0/2125731349 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f71d0063d00 0x7f71d00661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:36.885 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:36.859+0000 7f71d6ffd700 1 -- 172.21.15.195:0/2125731349 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6374+0+0 (secure 0 0 0) 0x7f71e4082980 con 0x7f71f00683c0 2022-06-27T11:52:36.885 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:36.859+0000 7f71ee7fc700 1 --2- 172.21.15.195:0/2125731349 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f71d0063d00 0x7f71d00661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:36.886 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:36.859+0000 7f71ee7fc700 1 --2- 172.21.15.195:0/2125731349 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f71d0063d00 0x7f71d00661b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f71f00c3820 tx=0x7f71e0000bc0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:52:36.886 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:36.863+0000 7f71d6ffd700 1 -- 172.21.15.195:0/2125731349 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f71f01697d0 con 0x7f71f00683c0 2022-06-27T11:52:37.092 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:37.085+0000 7f71f57d3700 1 -- 172.21.15.195:0/2125731349 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f71f01697d0 con 0x7f71f00683c0 2022-06-27T11:52:37.094 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:37.085+0000 7f71d6ffd700 1 -- 172.21.15.195:0/2125731349 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+5367 (secure 0 0 0) 0x7f71f01697d0 con 0x7f71f00683c0 2022-06-27T11:52:37.095 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:52:37.097 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:52:29.902078+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4760":{"gid":4760,"name":"b","rank":0,"incarnation":20,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6833/1026219246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1026219246},{"type":"v1","addr":"172.21.15.202:6833","nonce":1026219246}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:52:37.100 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:37.087+0000 7f71f57d3700 1 -- 172.21.15.195:0/2125731349 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f71d0063d00 msgr2=0x7f71d00661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:37.100 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:37.087+0000 7f71f57d3700 1 --2- 172.21.15.195:0/2125731349 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f71d0063d00 0x7f71d00661b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f71f00c3820 tx=0x7f71e0000bc0 comp rx=0 tx=0).stop 2022-06-27T11:52:37.100 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:37.087+0000 7f71f57d3700 1 -- 172.21.15.195:0/2125731349 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f71f00683c0 msgr2=0x7f71f0166280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:37.101 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:37.087+0000 7f71f57d3700 1 --2- 172.21.15.195:0/2125731349 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f71f00683c0 0x7f71f0166280 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f71f00cdc40 tx=0x7f71e4015000 comp rx=0 tx=0).stop 2022-06-27T11:52:37.101 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:37.087+0000 7f71f57d3700 1 -- 172.21.15.195:0/2125731349 shutdown_connections 2022-06-27T11:52:37.101 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:37.087+0000 7f71f57d3700 1 --2- 172.21.15.195:0/2125731349 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f71d0063d00 0x7f71d00661b0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:37.101 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:37.087+0000 7f71f57d3700 1 --2- 172.21.15.195:0/2125731349 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f71f00683c0 0x7f71f0166280 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:37.102 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:37.087+0000 7f71f57d3700 1 --2- 172.21.15.195:0/2125731349 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f71f00c7480 0x7f71f0157630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:37.102 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:37.087+0000 7f71f57d3700 1 --2- 172.21.15.195:0/2125731349 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f71f0068f80 0x7f71f00c6680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:37.103 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:37.087+0000 7f71f57d3700 1 -- 172.21.15.195:0/2125731349 >> 172.21.15.195:0/2125731349 conn(0x7f71f00ccbd0 msgr2=0x7f71f0069f50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:52:37.103 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:37.087+0000 7f71f57d3700 1 -- 172.21.15.195:0/2125731349 shutdown_connections 2022-06-27T11:52:37.103 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:37.087+0000 7f71f57d3700 1 -- 172.21.15.195:0/2125731349 wait complete. 2022-06-27T11:52:37.103 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 20 2022-06-27T11:52:37.104 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T11:52:39.105 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:52:39.238 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.239+0000 7f93db87c700 1 Processor -- start 2022-06-27T11:52:39.239 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.240+0000 7f93db87c700 1 -- start start 2022-06-27T11:52:39.241 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.240+0000 7f93db87c700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f93d41508d0 0x7f93d4063af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:39.241 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.240+0000 7f93db87c700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93d4064030 con 0x7f93d4150500 2022-06-27T11:52:39.241 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.240+0000 7f93db87c700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93d40c4160 con 0x7f93d406a450 2022-06-27T11:52:39.242 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.240+0000 7f93db87c700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93d40c4440 con 0x7f93d41508d0 2022-06-27T11:52:39.254 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.241+0000 7f93d9618700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f93d4150500 0x7f93d40c48d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:49860/0 (socket says 172.21.15.195:49860) 2022-06-27T11:52:39.255 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.241+0000 7f93d9618700 1 -- 172.21.15.195:0/23663112 learned_addr learned my addr 172.21.15.195:0/23663112 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:52:39.255 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.241+0000 7f93d9e19700 1 --2- 172.21.15.195:0/23663112 >> v2:172.21.15.202:3300/0 conn(0x7f93d41508d0 0x7f93d4063af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:39.256 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.241+0000 7f93c3fff700 1 -- 172.21.15.195:0/23663112 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3186322377 0 0) 0x7f93c4001290 con 0x7f93d4150500 2022-06-27T11:52:39.256 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.241+0000 7f93c3fff700 1 -- 172.21.15.195:0/23663112 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93bc003430 con 0x7f93d4150500 2022-06-27T11:52:39.257 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.241+0000 7f93c3fff700 1 -- 172.21.15.195:0/23663112 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 657375121 0 0) 0x7f93bc003430 con 0x7f93d4150500 2022-06-27T11:52:39.257 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.241+0000 7f93d9e19700 1 -- 172.21.15.195:0/23663112 >> v1:172.21.15.202:6789/0 conn(0x7f93d406a450 legacy=0x7f93d40bbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:39.257 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.241+0000 7f93d9e19700 1 -- 172.21.15.195:0/23663112 >> v1:172.21.15.195:6789/0 conn(0x7f93d4150500 legacy=0x7f93d40c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:39.258 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.241+0000 7f93d9e19700 1 -- 172.21.15.195:0/23663112 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93d415a800 con 0x7f93d41508d0 2022-06-27T11:52:39.258 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.241+0000 7f93d9e19700 1 --2- 172.21.15.195:0/23663112 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f93d41508d0 0x7f93d4063af0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f93cc009e60 tx=0x7f93cc009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d6ce8639c676d47 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:52:39.259 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.241+0000 7f93c3fff700 1 -- 172.21.15.195:0/23663112 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93cc00e070 con 0x7f93d41508d0 2022-06-27T11:52:39.259 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.241+0000 7f93c3fff700 1 -- 172.21.15.195:0/23663112 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93cc0038a0 con 0x7f93d41508d0 2022-06-27T11:52:39.260 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.241+0000 7f93c3fff700 1 -- 172.21.15.195:0/23663112 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c4001780 con 0x7f93d41508d0 2022-06-27T11:52:39.260 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.242+0000 7f93db87c700 1 -- 172.21.15.195:0/23663112 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f93d41508d0 msgr2=0x7f93d4063af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:39.261 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.242+0000 7f93db87c700 1 --2- 172.21.15.195:0/23663112 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f93d41508d0 0x7f93d4063af0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f93cc009e60 tx=0x7f93cc009780 comp rx=0 tx=0).stop 2022-06-27T11:52:39.261 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.242+0000 7f93db87c700 1 -- 172.21.15.195:0/23663112 shutdown_connections 2022-06-27T11:52:39.262 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.242+0000 7f93db87c700 1 --2- 172.21.15.195:0/23663112 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f93d41508d0 0x7f93d4063af0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:39.262 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.242+0000 7f93db87c700 1 -- 172.21.15.195:0/23663112 >> 172.21.15.195:0/23663112 conn(0x7f93d40c9b00 msgr2=0x7f93d40c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:52:39.262 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.242+0000 7f93db87c700 1 -- 172.21.15.195:0/23663112 shutdown_connections 2022-06-27T11:52:39.263 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.242+0000 7f93db87c700 1 -- 172.21.15.195:0/23663112 wait complete. 2022-06-27T11:52:39.263 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.243+0000 7f93db87c700 1 Processor -- start 2022-06-27T11:52:39.263 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.243+0000 7f93db87c700 1 -- start start 2022-06-27T11:52:39.264 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.244+0000 7f93db87c700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f93d406a450 0x7f93d4153590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:39.264 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.244+0000 7f93db87c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f93d4150500 0x7f93d415ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:39.265 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.244+0000 7f93db87c700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f93d41508d0 0x7f93d4166af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:39.265 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.244+0000 7f93db87c700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93d406a1f0 con 0x7f93d4150500 2022-06-27T11:52:39.265 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.244+0000 7f93db87c700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93d4153c40 con 0x7f93d406a450 2022-06-27T11:52:39.266 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.244+0000 7f93db87c700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93d4166660 con 0x7f93d41508d0 2022-06-27T11:52:39.266 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.244+0000 7f93d8e17700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f93d4150500 0x7f93d415ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:39.266 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.244+0000 7f93d8e17700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f93d4150500 0x7f93d415ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:43680/0 (socket says 172.21.15.195:43680) 2022-06-27T11:52:39.267 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.244+0000 7f93d8e17700 1 -- 172.21.15.195:0/3985056450 learned_addr learned my addr 172.21.15.195:0/3985056450 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:52:39.267 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.245+0000 7f93d9618700 1 --2- 172.21.15.195:0/3985056450 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f93d406a450 0x7f93d4153590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:39.267 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.245+0000 7f93d9e19700 1 --2- 172.21.15.195:0/3985056450 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f93d41508d0 0x7f93d4166af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:39.267 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.245+0000 7f93d8e17700 1 -- 172.21.15.195:0/3985056450 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f93d41508d0 msgr2=0x7f93d4166af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:39.267 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.245+0000 7f93d8e17700 1 --2- 172.21.15.195:0/3985056450 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f93d41508d0 0x7f93d4166af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:39.268 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.245+0000 7f93d8e17700 1 -- 172.21.15.195:0/3985056450 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f93d406a450 msgr2=0x7f93d4153590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:39.268 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.245+0000 7f93d8e17700 1 --2- 172.21.15.195:0/3985056450 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f93d406a450 0x7f93d4153590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:39.268 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.245+0000 7f93d8e17700 1 -- 172.21.15.195:0/3985056450 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93cc009a20 con 0x7f93d4150500 2022-06-27T11:52:39.268 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.245+0000 7f93d8e17700 1 --2- 172.21.15.195:0/3985056450 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f93d4150500 0x7f93d415ac20 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f93d000c4c0 tx=0x7f93d0007e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:52:39.269 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.245+0000 7f93c17fa700 1 -- 172.21.15.195:0/3985056450 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93d00050d0 con 0x7f93d4150500 2022-06-27T11:52:39.269 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.245+0000 7f93c17fa700 1 -- 172.21.15.195:0/3985056450 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93d0005230 con 0x7f93d4150500 2022-06-27T11:52:39.269 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.245+0000 7f93db87c700 1 -- 172.21.15.195:0/3985056450 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93d40bf740 con 0x7f93d4150500 2022-06-27T11:52:39.269 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.245+0000 7f93c17fa700 1 -- 172.21.15.195:0/3985056450 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93d00117a0 con 0x7f93d4150500 2022-06-27T11:52:39.270 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.245+0000 7f93db87c700 1 -- 172.21.15.195:0/3985056450 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93d40bfd00 con 0x7f93d4150500 2022-06-27T11:52:39.270 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.246+0000 7f93db87c700 1 -- 172.21.15.195:0/3985056450 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93d40722c0 con 0x7f93d4150500 2022-06-27T11:52:39.270 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.246+0000 7f93c17fa700 1 -- 172.21.15.195:0/3985056450 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f93d0019030 con 0x7f93d4150500 2022-06-27T11:52:39.270 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.247+0000 7f93c17fa700 1 --2- 172.21.15.195:0/3985056450 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f93b4063d00 0x7f93b40661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:39.271 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.247+0000 7f93c17fa700 1 -- 172.21.15.195:0/3985056450 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6374+0+0 (secure 0 0 0) 0x7f93d007fb40 con 0x7f93d4150500 2022-06-27T11:52:39.271 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.247+0000 7f93d9618700 1 --2- 172.21.15.195:0/3985056450 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f93b4063d00 0x7f93b40661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:39.271 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.247+0000 7f93d9618700 1 --2- 172.21.15.195:0/3985056450 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f93b4063d00 0x7f93b40661b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f93c4001510 tx=0x7f93c4009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:52:39.271 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.251+0000 7f93c17fa700 1 -- 172.21.15.195:0/3985056450 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f93d0049c90 con 0x7f93d4150500 2022-06-27T11:52:39.473 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.472+0000 7f93db87c700 1 -- 172.21.15.195:0/3985056450 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f93d41667a0 con 0x7f93d4150500 2022-06-27T11:52:39.474 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.473+0000 7f93c17fa700 1 -- 172.21.15.195:0/3985056450 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+5371 (secure 0 0 0) 0x7f93d0017020 con 0x7f93d4150500 2022-06-27T11:52:39.474 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:52:39.475 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:52:37.988984+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4760":{"gid":4760,"name":"b","rank":0,"incarnation":20,"state":"up:reconnect","state_seq":15,"addr":"172.21.15.202:6833/1026219246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1026219246},{"type":"v1","addr":"172.21.15.202:6833","nonce":1026219246}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:52:39.477 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.475+0000 7f93db87c700 1 -- 172.21.15.195:0/3985056450 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f93b4063d00 msgr2=0x7f93b40661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:39.478 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.475+0000 7f93db87c700 1 --2- 172.21.15.195:0/3985056450 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f93b4063d00 0x7f93b40661b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f93c4001510 tx=0x7f93c4009040 comp rx=0 tx=0).stop 2022-06-27T11:52:39.478 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.475+0000 7f93db87c700 1 -- 172.21.15.195:0/3985056450 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f93d4150500 msgr2=0x7f93d415ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:39.479 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.475+0000 7f93db87c700 1 --2- 172.21.15.195:0/3985056450 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f93d4150500 0x7f93d415ac20 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f93d000c4c0 tx=0x7f93d0007e40 comp rx=0 tx=0).stop 2022-06-27T11:52:39.479 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.475+0000 7f93db87c700 1 -- 172.21.15.195:0/3985056450 shutdown_connections 2022-06-27T11:52:39.479 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.475+0000 7f93db87c700 1 --2- 172.21.15.195:0/3985056450 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f93b4063d00 0x7f93b40661b0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:39.480 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.475+0000 7f93db87c700 1 --2- 172.21.15.195:0/3985056450 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f93d41508d0 0x7f93d4166af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:39.480 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.475+0000 7f93db87c700 1 --2- 172.21.15.195:0/3985056450 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f93d406a450 0x7f93d4153590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:39.481 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.475+0000 7f93db87c700 1 --2- 172.21.15.195:0/3985056450 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f93d4150500 0x7f93d415ac20 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:39.481 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.475+0000 7f93db87c700 1 -- 172.21.15.195:0/3985056450 >> 172.21.15.195:0/3985056450 conn(0x7f93d40c9b00 msgr2=0x7f93d40c5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:52:39.482 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.475+0000 7f93db87c700 1 -- 172.21.15.195:0/3985056450 shutdown_connections 2022-06-27T11:52:39.482 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:39.475+0000 7f93db87c700 1 -- 172.21.15.195:0/3985056450 wait complete. 2022-06-27T11:52:39.482 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 21 2022-06-27T11:52:39.487 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T11:52:41.488 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:52:41.633 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.624+0000 7f181715a700 1 Processor -- start 2022-06-27T11:52:41.634 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.624+0000 7f181715a700 1 -- start start 2022-06-27T11:52:41.634 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.625+0000 7f181715a700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f181014e490 0x7f181006cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:41.635 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.625+0000 7f181715a700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f181014e860 0x7f18100c3840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:41.635 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.625+0000 7f181715a700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18100c33b0 con 0x7f18100c3d80 2022-06-27T11:52:41.637 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.625+0000 7f181715a700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f181006d520 con 0x7f181014e860 2022-06-27T11:52:41.637 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.625+0000 7f181715a700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f181006f220 con 0x7f181014e490 2022-06-27T11:52:41.637 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.625+0000 7f180ffff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f181014e860 0x7f18100c3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:41.638 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.625+0000 7f180ffff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f181014e860 0x7f18100c3840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:43690/0 (socket says 172.21.15.195:43690) 2022-06-27T11:52:41.638 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.625+0000 7f180ffff700 1 -- 172.21.15.195:0/3666778696 learned_addr learned my addr 172.21.15.195:0/3666778696 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:52:41.639 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.625+0000 7f1814ef6700 1 --2- 172.21.15.195:0/3666778696 >> v2:172.21.15.202:3300/0 conn(0x7f181014e490 0x7f181006cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:41.639 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.625+0000 7f180ffff700 1 -- 172.21.15.195:0/3666778696 >> v2:172.21.15.202:3300/0 conn(0x7f181014e490 msgr2=0x7f181006cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:52:41.640 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.625+0000 7f180ffff700 1 --2- 172.21.15.195:0/3666778696 >> v2:172.21.15.202:3300/0 conn(0x7f181014e490 0x7f181006cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:41.640 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.625+0000 7f180ffff700 1 -- 172.21.15.195:0/3666778696 >> v1:172.21.15.202:6789/0 conn(0x7f18100c3d80 legacy=0x7f181006f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:41.640 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.625+0000 7f180ffff700 1 -- 172.21.15.195:0/3666778696 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18101542c0 con 0x7f181014e860 2022-06-27T11:52:41.641 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.625+0000 7f180ffff700 1 --2- 172.21.15.195:0/3666778696 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f181014e860 0x7f18100c3840 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f1810064800 tx=0x7f17fc009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=170523afc8bfba24 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:52:41.641 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.625+0000 7f180f7fe700 1 -- 172.21.15.195:0/3666778696 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17fc00b640 con 0x7f181014e860 2022-06-27T11:52:41.642 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.625+0000 7f180f7fe700 1 -- 172.21.15.195:0/3666778696 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17fc00b7a0 con 0x7f181014e860 2022-06-27T11:52:41.642 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.625+0000 7f180f7fe700 1 -- 172.21.15.195:0/3666778696 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17fc00bb40 con 0x7f181014e860 2022-06-27T11:52:41.643 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.626+0000 7f181715a700 1 -- 172.21.15.195:0/3666778696 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f181014e860 msgr2=0x7f18100c3840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:41.643 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.626+0000 7f181715a700 1 --2- 172.21.15.195:0/3666778696 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f181014e860 0x7f18100c3840 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f1810064800 tx=0x7f17fc009120 comp rx=0 tx=0).stop 2022-06-27T11:52:41.643 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.626+0000 7f181715a700 1 -- 172.21.15.195:0/3666778696 shutdown_connections 2022-06-27T11:52:41.644 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.626+0000 7f181715a700 1 --2- 172.21.15.195:0/3666778696 >> v2:172.21.15.202:3300/0 conn(0x7f181014e490 0x7f181006cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:41.644 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.626+0000 7f181715a700 1 --2- 172.21.15.195:0/3666778696 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f181014e860 0x7f18100c3840 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:41.644 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.626+0000 7f181715a700 1 -- 172.21.15.195:0/3666778696 >> 172.21.15.195:0/3666778696 conn(0x7f1810063af0 msgr2=0x7f1810063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:52:41.645 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.626+0000 7f181715a700 1 -- 172.21.15.195:0/3666778696 shutdown_connections 2022-06-27T11:52:41.645 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.626+0000 7f181715a700 1 -- 172.21.15.195:0/3666778696 wait complete. 2022-06-27T11:52:41.645 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.627+0000 7f181715a700 1 Processor -- start 2022-06-27T11:52:41.646 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.627+0000 7f181715a700 1 -- start start 2022-06-27T11:52:41.646 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.628+0000 7f181715a700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f18100c3d80 0x7f18101497e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:41.647 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.628+0000 7f181715a700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f181014e490 0x7f18100bf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:41.648 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.628+0000 7f181715a700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f181014e860 0x7f18100bfe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:41.648 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.628+0000 7f181715a700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18101522c0 con 0x7f181014e860 2022-06-27T11:52:41.649 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.628+0000 7f181715a700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1810167b30 con 0x7f18100c3d80 2022-06-27T11:52:41.649 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.628+0000 7f181715a700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18100bf460 con 0x7f181014e490 2022-06-27T11:52:41.650 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.628+0000 7f18156f7700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f181014e860 0x7f18100bfe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:41.650 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.628+0000 7f18156f7700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f181014e860 0x7f18100bfe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:43696/0 (socket says 172.21.15.195:43696) 2022-06-27T11:52:41.651 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.628+0000 7f18156f7700 1 -- 172.21.15.195:0/3596785929 learned_addr learned my addr 172.21.15.195:0/3596785929 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:52:41.651 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.628+0000 7f1814ef6700 1 --2- 172.21.15.195:0/3596785929 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f18100c3d80 0x7f18101497e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:41.651 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.628+0000 7f180ffff700 1 --2- 172.21.15.195:0/3596785929 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f181014e490 0x7f18100bf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:41.652 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.628+0000 7f18156f7700 1 -- 172.21.15.195:0/3596785929 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f181014e490 msgr2=0x7f18100bf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:41.652 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.628+0000 7f18156f7700 1 --2- 172.21.15.195:0/3596785929 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f181014e490 0x7f18100bf8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:41.653 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.628+0000 7f18156f7700 1 -- 172.21.15.195:0/3596785929 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f18100c3d80 msgr2=0x7f18101497e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:41.653 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.628+0000 7f18156f7700 1 --2- 172.21.15.195:0/3596785929 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f18100c3d80 0x7f18101497e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:41.653 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.629+0000 7f18156f7700 1 -- 172.21.15.195:0/3596785929 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17fc0093a0 con 0x7f181014e860 2022-06-27T11:52:41.654 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.629+0000 7f180ffff700 1 --2- 172.21.15.195:0/3596785929 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f181014e490 0x7f18100bf8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T11:52:41.654 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.629+0000 7f1814ef6700 1 --2- 172.21.15.195:0/3596785929 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f18100c3d80 0x7f18101497e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T11:52:41.654 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.629+0000 7f18156f7700 1 --2- 172.21.15.195:0/3596785929 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f181014e860 0x7f18100bfe30 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f180800c710 tx=0x7f1808001540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:52:41.655 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.629+0000 7f180cff9700 1 -- 172.21.15.195:0/3596785929 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1808003880 con 0x7f181014e860 2022-06-27T11:52:41.655 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.629+0000 7f180cff9700 1 -- 172.21.15.195:0/3596785929 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18080039e0 con 0x7f181014e860 2022-06-27T11:52:41.656 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.629+0000 7f181715a700 1 -- 172.21.15.195:0/3596785929 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18100be850 con 0x7f181014e860 2022-06-27T11:52:41.656 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.629+0000 7f180cff9700 1 -- 172.21.15.195:0/3596785929 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1808003d60 con 0x7f181014e860 2022-06-27T11:52:41.656 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.629+0000 7f181715a700 1 -- 172.21.15.195:0/3596785929 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18100bee10 con 0x7f181014e860 2022-06-27T11:52:41.656 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.630+0000 7f181715a700 1 -- 172.21.15.195:0/3596785929 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18100bf5f0 con 0x7f181014e860 2022-06-27T11:52:41.656 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.630+0000 7f180cff9700 1 -- 172.21.15.195:0/3596785929 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f1808010050 con 0x7f181014e860 2022-06-27T11:52:41.657 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.631+0000 7f180cff9700 1 --2- 172.21.15.195:0/3596785929 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f17f8063d00 0x7f17f80661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:41.657 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.631+0000 7f180cff9700 1 -- 172.21.15.195:0/3596785929 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6374+0+0 (secure 0 0 0) 0x7f1808080440 con 0x7f181014e860 2022-06-27T11:52:41.657 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.631+0000 7f1814ef6700 1 --2- 172.21.15.195:0/3596785929 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f17f8063d00 0x7f17f80661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:41.657 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.631+0000 7f1814ef6700 1 --2- 172.21.15.195:0/3596785929 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f17f8063d00 0x7f17f80661b0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f1804006fd0 tx=0x7f1804008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:52:41.658 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.635+0000 7f180cff9700 1 -- 172.21.15.195:0/3596785929 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f180804ee00 con 0x7f181014e860 2022-06-27T11:52:41.864 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.865+0000 7f181715a700 1 -- 172.21.15.195:0/3596785929 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f18100722c0 con 0x7f181014e860 2022-06-27T11:52:41.865 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.866+0000 7f180cff9700 1 -- 172.21.15.195:0/3596785929 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+5371 (secure 0 0 0) 0x7f1808052420 con 0x7f181014e860 2022-06-27T11:52:41.865 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:52:41.865 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:52:37.988984+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4760":{"gid":4760,"name":"b","rank":0,"incarnation":20,"state":"up:reconnect","state_seq":15,"addr":"172.21.15.202:6833/1026219246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1026219246},{"type":"v1","addr":"172.21.15.202:6833","nonce":1026219246}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:52:41.867 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.868+0000 7f181715a700 1 -- 172.21.15.195:0/3596785929 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f17f8063d00 msgr2=0x7f17f80661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:41.868 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.868+0000 7f181715a700 1 --2- 172.21.15.195:0/3596785929 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f17f8063d00 0x7f17f80661b0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f1804006fd0 tx=0x7f1804008040 comp rx=0 tx=0).stop 2022-06-27T11:52:41.868 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.868+0000 7f181715a700 1 -- 172.21.15.195:0/3596785929 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f181014e860 msgr2=0x7f18100bfe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:41.868 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.868+0000 7f181715a700 1 --2- 172.21.15.195:0/3596785929 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f181014e860 0x7f18100bfe30 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f180800c710 tx=0x7f1808001540 comp rx=0 tx=0).stop 2022-06-27T11:52:41.868 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.868+0000 7f181715a700 1 -- 172.21.15.195:0/3596785929 shutdown_connections 2022-06-27T11:52:41.869 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.868+0000 7f181715a700 1 --2- 172.21.15.195:0/3596785929 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f17f8063d00 0x7f17f80661b0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:41.869 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.868+0000 7f181715a700 1 --2- 172.21.15.195:0/3596785929 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f181014e860 0x7f18100bfe30 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:41.869 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.868+0000 7f181715a700 1 --2- 172.21.15.195:0/3596785929 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f18100c3d80 0x7f18101497e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:41.870 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.868+0000 7f181715a700 1 --2- 172.21.15.195:0/3596785929 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f181014e490 0x7f18100bf8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:41.870 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.868+0000 7f181715a700 1 -- 172.21.15.195:0/3596785929 >> 172.21.15.195:0/3596785929 conn(0x7f1810063af0 msgr2=0x7f1810152b60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:52:41.870 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.868+0000 7f181715a700 1 -- 172.21.15.195:0/3596785929 shutdown_connections 2022-06-27T11:52:41.870 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:41.868+0000 7f181715a700 1 -- 172.21.15.195:0/3596785929 wait complete. 2022-06-27T11:52:41.871 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 21 2022-06-27T11:52:41.879 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T11:52:43.881 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:52:44.027 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.027+0000 7f545baac700 1 Processor -- start 2022-06-27T11:52:44.027 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.027+0000 7f545baac700 1 -- start start 2022-06-27T11:52:44.030 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.028+0000 7f545baac700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f54540c2bf0 0x7f54540c2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:44.031 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.028+0000 7f545baac700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f54540c5000 0x7f54540c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:44.031 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.028+0000 7f545baac700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54540bc490 con 0x7f54540c61e0 2022-06-27T11:52:44.032 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.028+0000 7f545baac700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f545406a0e0 con 0x7f54540c2bf0 2022-06-27T11:52:44.032 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.028+0000 7f545baac700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54540bc6e0 con 0x7f54540c5000 2022-06-27T11:52:44.033 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.028+0000 7f5459047700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f54540c61e0 0x7f54540c48f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:49892/0 (socket says 172.21.15.195:49892) 2022-06-27T11:52:44.033 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.028+0000 7f5459047700 1 -- 172.21.15.195:0/2313716385 learned_addr learned my addr 172.21.15.195:0/2313716385 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:52:44.033 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.029+0000 7f5458846700 1 -- 172.21.15.195:0/2313716385 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3927088259 0 0) 0x7f54540bc490 con 0x7f54540c61e0 2022-06-27T11:52:44.034 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.029+0000 7f5458846700 1 -- 172.21.15.195:0/2313716385 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f543c003430 con 0x7f54540c61e0 2022-06-27T11:52:44.034 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.029+0000 7f5459848700 1 --2- 172.21.15.195:0/2313716385 >> v2:172.21.15.202:3300/0 conn(0x7f54540c2bf0 0x7f54540c2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:44.035 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.029+0000 7f545a049700 1 --2- 172.21.15.195:0/2313716385 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f54540c5000 0x7f54540c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:44.035 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.029+0000 7f5458846700 1 -- 172.21.15.195:0/2313716385 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4292027974 0 0) 0x7f543c003430 con 0x7f54540c61e0 2022-06-27T11:52:44.036 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.029+0000 7f5458846700 1 -- 172.21.15.195:0/2313716385 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f54540c5000 msgr2=0x7f54540c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:52:44.036 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.029+0000 7f5458846700 1 --2- 172.21.15.195:0/2313716385 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f54540c5000 0x7f54540c3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:44.036 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.029+0000 7f5458846700 1 -- 172.21.15.195:0/2313716385 >> v2:172.21.15.202:3300/0 conn(0x7f54540c2bf0 msgr2=0x7f54540c2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:52:44.050 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.029+0000 7f5458846700 1 --2- 172.21.15.195:0/2313716385 >> v2:172.21.15.202:3300/0 conn(0x7f54540c2bf0 0x7f54540c2fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:44.052 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.029+0000 7f5458846700 1 -- 172.21.15.195:0/2313716385 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5454158220 con 0x7f54540c61e0 2022-06-27T11:52:44.053 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.029+0000 7f5458846700 1 -- 172.21.15.195:0/2313716385 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7f5450001470 con 0x7f54540c61e0 2022-06-27T11:52:44.054 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.029+0000 7f5458846700 1 -- 172.21.15.195:0/2313716385 >> v1:172.21.15.195:6789/0 conn(0x7f54540c61e0 legacy=0x7f54540c48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:44.054 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.029+0000 7f5458846700 1 --2- 172.21.15.195:0/2313716385 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f54540c5000 0x7f543c003920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:44.055 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.029+0000 7f5458846700 1 --2- 172.21.15.195:0/2313716385 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f543c003ea0 0x7f543c0046d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:44.055 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.029+0000 7f5458846700 1 --2- 172.21.15.195:0/2313716385 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f543c004c10 0x7f543c005080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:44.055 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.029+0000 7f5458846700 1 -- 172.21.15.195:0/2313716385 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f545406a0e0 con 0x7f543c003ea0 2022-06-27T11:52:44.056 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.029+0000 7f5458846700 1 -- 172.21.15.195:0/2313716385 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54540bc6e0 con 0x7f543c004c10 2022-06-27T11:52:44.056 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.029+0000 7f5458846700 1 -- 172.21.15.195:0/2313716385 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f543c00ba00 con 0x7f54540c5000 2022-06-27T11:52:44.057 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.029+0000 7f5459047700 1 --2- 172.21.15.195:0/2313716385 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f543c003ea0 0x7f543c0046d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:44.057 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.029+0000 7f5459848700 1 --2- 172.21.15.195:0/2313716385 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f54540c5000 0x7f543c003920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:44.058 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.029+0000 7f545a049700 1 --2- 172.21.15.195:0/2313716385 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f543c004c10 0x7f543c005080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:44.058 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.029+0000 7f5459047700 1 -- 172.21.15.195:0/2313716385 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f54540c5000 msgr2=0x7f543c003920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:52:44.059 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.029+0000 7f5459047700 1 --2- 172.21.15.195:0/2313716385 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f54540c5000 0x7f543c003920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:44.059 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.029+0000 7f5459047700 1 -- 172.21.15.195:0/2313716385 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f543c004c10 msgr2=0x7f543c005080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:52:44.060 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.029+0000 7f5459047700 1 --2- 172.21.15.195:0/2313716385 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f543c004c10 0x7f543c005080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:44.060 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.029+0000 7f5459047700 1 -- 172.21.15.195:0/2313716385 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f543c003430 con 0x7f543c003ea0 2022-06-27T11:52:44.061 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.029+0000 7f5459047700 1 --2- 172.21.15.195:0/2313716385 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f543c003ea0 0x7f543c0046d0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f545000c1e0 tx=0x7f54540c1820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c58f6dff80c2e57e server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:52:44.061 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.030+0000 7f5458846700 1 -- 172.21.15.195:0/2313716385 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f545000cb10 con 0x7f543c003ea0 2022-06-27T11:52:44.062 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.030+0000 7f5458846700 1 -- 172.21.15.195:0/2313716385 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f545000cc70 con 0x7f543c003ea0 2022-06-27T11:52:44.072 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.030+0000 7f5458846700 1 -- 172.21.15.195:0/2313716385 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54500070d0 con 0x7f543c003ea0 2022-06-27T11:52:44.073 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.030+0000 7f545baac700 1 -- 172.21.15.195:0/2313716385 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f543c003ea0 msgr2=0x7f543c0046d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:44.073 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.030+0000 7f545baac700 1 --2- 172.21.15.195:0/2313716385 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f543c003ea0 0x7f543c0046d0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f545000c1e0 tx=0x7f54540c1820 comp rx=0 tx=0).stop 2022-06-27T11:52:44.074 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.030+0000 7f545a049700 1 -- 172.21.15.195:0/2313716385 reap_dead start 2022-06-27T11:52:44.074 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.030+0000 7f545baac700 1 -- 172.21.15.195:0/2313716385 shutdown_connections 2022-06-27T11:52:44.075 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.030+0000 7f545baac700 1 -- 172.21.15.195:0/2313716385 >> 172.21.15.195:0/2313716385 conn(0x7f54540c9b00 msgr2=0x7f54540c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:52:44.075 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.030+0000 7f545baac700 1 -- 172.21.15.195:0/2313716385 shutdown_connections 2022-06-27T11:52:44.076 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.030+0000 7f545baac700 1 -- 172.21.15.195:0/2313716385 wait complete. 2022-06-27T11:52:44.076 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.031+0000 7f545baac700 1 Processor -- start 2022-06-27T11:52:44.077 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.031+0000 7f545baac700 1 -- start start 2022-06-27T11:52:44.077 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.032+0000 7f545baac700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f543c003ea0 0x7f545414bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:44.083 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.032+0000 7f545baac700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f545414a5e0 0x7f5454164870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:44.083 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.032+0000 7f545baac700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f545414aa30 0x7f54540bd3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:44.084 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.032+0000 7f545baac700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54541643e0 con 0x7f545414aa30 2022-06-27T11:52:44.084 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.032+0000 7f545baac700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f545414c2a0 con 0x7f545414a5e0 2022-06-27T11:52:44.085 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.032+0000 7f545baac700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5454164090 con 0x7f543c003ea0 2022-06-27T11:52:44.085 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.032+0000 7f545a049700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f545414aa30 0x7f54540bd3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:44.086 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.032+0000 7f545a049700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f545414aa30 0x7f54540bd3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:43718/0 (socket says 172.21.15.195:43718) 2022-06-27T11:52:44.087 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.032+0000 7f545a049700 1 -- 172.21.15.195:0/47718407 learned_addr learned my addr 172.21.15.195:0/47718407 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:52:44.087 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.032+0000 7f5459848700 1 --2- 172.21.15.195:0/47718407 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f543c003ea0 0x7f545414bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:44.088 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.032+0000 7f5459047700 1 --2- 172.21.15.195:0/47718407 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f545414a5e0 0x7f5454164870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:44.088 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.033+0000 7f545a049700 1 -- 172.21.15.195:0/47718407 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f543c003ea0 msgr2=0x7f545414bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:44.089 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.033+0000 7f545a049700 1 --2- 172.21.15.195:0/47718407 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f543c003ea0 0x7f545414bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:44.089 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.033+0000 7f545a049700 1 -- 172.21.15.195:0/47718407 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f545414a5e0 msgr2=0x7f5454164870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:44.090 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.033+0000 7f545a049700 1 --2- 172.21.15.195:0/47718407 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f545414a5e0 0x7f5454164870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:44.090 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.033+0000 7f545a049700 1 -- 172.21.15.195:0/47718407 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5454167a20 con 0x7f545414aa30 2022-06-27T11:52:44.091 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.033+0000 7f5459848700 1 --2- 172.21.15.195:0/47718407 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f543c003ea0 0x7f545414bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T11:52:44.091 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.033+0000 7f5459047700 1 --2- 172.21.15.195:0/47718407 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f545414a5e0 0x7f5454164870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T11:52:44.092 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.033+0000 7f545a049700 1 --2- 172.21.15.195:0/47718407 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f545414aa30 0x7f54540bd3e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f54540c53f0 tx=0x7f544c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:52:44.092 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.033+0000 7f5441ffb700 1 -- 172.21.15.195:0/47718407 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f544c00e040 con 0x7f545414aa30 2022-06-27T11:52:44.092 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.033+0000 7f5441ffb700 1 -- 172.21.15.195:0/47718407 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f544c00f040 con 0x7f545414aa30 2022-06-27T11:52:44.093 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.033+0000 7f545baac700 1 -- 172.21.15.195:0/47718407 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5454167d00 con 0x7f545414aa30 2022-06-27T11:52:44.093 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.033+0000 7f5441ffb700 1 -- 172.21.15.195:0/47718407 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f544c00fe20 con 0x7f545414aa30 2022-06-27T11:52:44.095 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.033+0000 7f545baac700 1 -- 172.21.15.195:0/47718407 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54541682c0 con 0x7f545414aa30 2022-06-27T11:52:44.097 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.034+0000 7f545baac700 1 -- 172.21.15.195:0/47718407 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5454164200 con 0x7f545414aa30 2022-06-27T11:52:44.097 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.034+0000 7f5441ffb700 1 -- 172.21.15.195:0/47718407 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f544c003ab0 con 0x7f545414aa30 2022-06-27T11:52:44.098 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.035+0000 7f5441ffb700 1 --2- 172.21.15.195:0/47718407 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5434063d30 0x7f54340661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:44.098 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.035+0000 7f5441ffb700 1 -- 172.21.15.195:0/47718407 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6374+0+0 (secure 0 0 0) 0x7f544c080fa0 con 0x7f545414aa30 2022-06-27T11:52:44.099 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.035+0000 7f5459848700 1 --2- 172.21.15.195:0/47718407 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5434063d30 0x7f54340661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:44.099 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.035+0000 7f5459848700 1 --2- 172.21.15.195:0/47718407 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5434063d30 0x7f54340661e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f5444009260 tx=0x7f5444008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:52:44.100 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.039+0000 7f5441ffb700 1 -- 172.21.15.195:0/47718407 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f544c04fa50 con 0x7f545414aa30 2022-06-27T11:52:44.268 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.266+0000 7f545baac700 1 -- 172.21.15.195:0/47718407 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f54540722c0 con 0x7f545414aa30 2022-06-27T11:52:44.269 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.266+0000 7f5441ffb700 1 -- 172.21.15.195:0/47718407 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5368 (secure 0 0 0) 0x7f544c053070 con 0x7f545414aa30 2022-06-27T11:52:44.269 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:52:44.270 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:52:43.008118+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4760":{"gid":4760,"name":"b","rank":0,"incarnation":20,"state":"up:rejoin","state_seq":17,"addr":"172.21.15.202:6833/1026219246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1026219246},{"type":"v1","addr":"172.21.15.202:6833","nonce":1026219246}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:52:44.271 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.268+0000 7f545baac700 1 -- 172.21.15.195:0/47718407 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5434063d30 msgr2=0x7f54340661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:44.271 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.268+0000 7f545baac700 1 --2- 172.21.15.195:0/47718407 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5434063d30 0x7f54340661e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f5444009260 tx=0x7f5444008040 comp rx=0 tx=0).stop 2022-06-27T11:52:44.272 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.269+0000 7f545baac700 1 -- 172.21.15.195:0/47718407 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f545414aa30 msgr2=0x7f54540bd3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:44.272 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.269+0000 7f545baac700 1 --2- 172.21.15.195:0/47718407 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f545414aa30 0x7f54540bd3e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f54540c53f0 tx=0x7f544c00b040 comp rx=0 tx=0).stop 2022-06-27T11:52:44.272 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.269+0000 7f545baac700 1 -- 172.21.15.195:0/47718407 shutdown_connections 2022-06-27T11:52:44.272 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.269+0000 7f545baac700 1 --2- 172.21.15.195:0/47718407 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5434063d30 0x7f54340661e0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:44.272 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.269+0000 7f545baac700 1 --2- 172.21.15.195:0/47718407 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f545414aa30 0x7f54540bd3e0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:44.273 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.269+0000 7f545baac700 1 --2- 172.21.15.195:0/47718407 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f543c003ea0 0x7f545414bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:44.273 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.269+0000 7f545baac700 1 --2- 172.21.15.195:0/47718407 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f545414a5e0 0x7f5454164870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:44.273 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.269+0000 7f545baac700 1 -- 172.21.15.195:0/47718407 >> 172.21.15.195:0/47718407 conn(0x7f54540c9b00 msgr2=0x7f5454063af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:52:44.273 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.269+0000 7f545baac700 1 -- 172.21.15.195:0/47718407 shutdown_connections 2022-06-27T11:52:44.274 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:44.269+0000 7f545baac700 1 -- 172.21.15.195:0/47718407 wait complete. 2022-06-27T11:52:44.274 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 22 2022-06-27T11:52:44.280 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 22, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-06-27T11:47:52.358095+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 22, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4760": { "addr": "172.21.15.202:6833/1026219246", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6832", "nonce": 1026219246, "type": "v2" }, { "addr": "172.21.15.202:6833", "nonce": 1026219246, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4760, "incarnation": 20, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:rejoin", "state_seq": 17 } }, "last_failure": 0, "last_failure_osd_epoch": 41, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-06-27T11:52:43.008118+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4760 } } } ], "standbys": [ { "addr": "172.21.15.202:6837/3305006841", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6836", "nonce": 3305006841, "type": "v2" }, { "addr": "172.21.15.202:6837", "nonce": 3305006841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6835/2060709822", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6834", "nonce": 2060709822, "type": "v2" }, { "addr": "172.21.15.195:6835", "nonce": 2060709822, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6838/1598751652", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6836", "nonce": 1598751652, "type": "v2" }, { "addr": "172.21.15.195:6838", "nonce": 1598751652, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4275, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6839/1313800661", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6837", "nonce": 1313800661, "type": "v2" }, { "addr": "172.21.15.195:6839", "nonce": 1313800661, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4278, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-06-27T11:52:46.282 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:52:46.423 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.421+0000 7fafafe85700 1 Processor -- start 2022-06-27T11:52:46.423 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.421+0000 7fafafe85700 1 -- start start 2022-06-27T11:52:46.424 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.422+0000 7fafafe85700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fafa8068f80 0x7fafa80c4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:46.424 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.422+0000 7fafafe85700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafa81573b0 con 0x7fafa80c7480 2022-06-27T11:52:46.424 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.422+0000 7fafafe85700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafa80c46c0 con 0x7fafa80683c0 2022-06-27T11:52:46.424 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.422+0000 7fafafe85700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafa8068840 con 0x7fafa8068f80 2022-06-27T11:52:46.425 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.422+0000 7fafadc21700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fafa8068f80 0x7fafa80c4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:46.425 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.422+0000 7fafad420700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fafa80c7480 0x7fafa8156460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:49916/0 (socket says 172.21.15.195:49916) 2022-06-27T11:52:46.425 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.422+0000 7fafad420700 1 -- 172.21.15.195:0/2515061777 learned_addr learned my addr 172.21.15.195:0/2515061777 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:52:46.425 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.422+0000 7fafadc21700 1 -- 172.21.15.195:0/2515061777 >> v1:172.21.15.202:6789/0 conn(0x7fafa80683c0 legacy=0x7fafa80c6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:52:46.434 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.422+0000 7fafadc21700 1 -- 172.21.15.195:0/2515061777 >> v1:172.21.15.195:6789/0 conn(0x7fafa80c7480 legacy=0x7fafa8156460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:46.434 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.422+0000 7fafadc21700 1 -- 172.21.15.195:0/2515061777 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafa8156b70 con 0x7fafa8068f80 2022-06-27T11:52:46.435 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.422+0000 7fafadc21700 1 --2- 172.21.15.195:0/2515061777 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fafa8068f80 0x7fafa80c4b50 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fafa40097a0 tx=0x7fafa4009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=137400dcfbf9ac7d server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:52:46.436 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.423+0000 7fafacc1f700 1 -- 172.21.15.195:0/2515061777 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa4004030 con 0x7fafa8068f80 2022-06-27T11:52:46.437 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.423+0000 7fafacc1f700 1 -- 172.21.15.195:0/2515061777 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafa4004190 con 0x7fafa8068f80 2022-06-27T11:52:46.437 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.423+0000 7fafacc1f700 1 -- 172.21.15.195:0/2515061777 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa4011630 con 0x7fafa8068f80 2022-06-27T11:52:46.448 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.423+0000 7fafafe85700 1 -- 172.21.15.195:0/2515061777 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fafa8068f80 msgr2=0x7fafa80c4b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:46.448 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.423+0000 7fafafe85700 1 --2- 172.21.15.195:0/2515061777 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fafa8068f80 0x7fafa80c4b50 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fafa40097a0 tx=0x7fafa4009060 comp rx=0 tx=0).stop 2022-06-27T11:52:46.449 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.423+0000 7fafafe85700 1 -- 172.21.15.195:0/2515061777 shutdown_connections 2022-06-27T11:52:46.450 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.423+0000 7fafafe85700 1 --2- 172.21.15.195:0/2515061777 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fafa8068f80 0x7fafa80c4b50 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:46.450 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.423+0000 7fafafe85700 1 -- 172.21.15.195:0/2515061777 >> 172.21.15.195:0/2515061777 conn(0x7fafa80ccbd0 msgr2=0x7fafa80ccfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:52:46.451 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.423+0000 7fafafe85700 1 -- 172.21.15.195:0/2515061777 shutdown_connections 2022-06-27T11:52:46.451 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.423+0000 7fafafe85700 1 -- 172.21.15.195:0/2515061777 wait complete. 2022-06-27T11:52:46.452 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.424+0000 7fafafe85700 1 Processor -- start 2022-06-27T11:52:46.452 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.424+0000 7fafafe85700 1 -- start start 2022-06-27T11:52:46.452 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.425+0000 7fafafe85700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fafa80683c0 0x7fafa8158890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:46.453 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.425+0000 7fafafe85700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fafa8068f80 0x7fafa8151da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:46.453 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.425+0000 7fafafe85700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fafa80c7480 0x7fafa81522e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:46.454 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.425+0000 7fafafe85700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafa81672a0 con 0x7fafa80c7480 2022-06-27T11:52:46.454 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.425+0000 7fafafe85700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafa8166d50 con 0x7fafa80683c0 2022-06-27T11:52:46.455 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.425+0000 7fafafe85700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafa8166fc0 con 0x7fafa8068f80 2022-06-27T11:52:46.455 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.425+0000 7fafae422700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fafa80c7480 0x7fafa81522e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:46.456 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.425+0000 7fafae422700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fafa80c7480 0x7fafa81522e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:43736/0 (socket says 172.21.15.195:43736) 2022-06-27T11:52:46.456 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.425+0000 7fafae422700 1 -- 172.21.15.195:0/1754206120 learned_addr learned my addr 172.21.15.195:0/1754206120 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:52:46.456 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.425+0000 7fafadc21700 1 --2- 172.21.15.195:0/1754206120 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fafa80683c0 0x7fafa8158890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:46.457 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.425+0000 7fafad420700 1 --2- 172.21.15.195:0/1754206120 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fafa8068f80 0x7fafa8151da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:46.457 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.425+0000 7fafae422700 1 -- 172.21.15.195:0/1754206120 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fafa8068f80 msgr2=0x7fafa8151da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:46.458 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.425+0000 7fafae422700 1 --2- 172.21.15.195:0/1754206120 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fafa8068f80 0x7fafa8151da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:46.458 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.425+0000 7fafae422700 1 -- 172.21.15.195:0/1754206120 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fafa80683c0 msgr2=0x7fafa8158890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:46.459 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.425+0000 7fafae422700 1 --2- 172.21.15.195:0/1754206120 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fafa80683c0 0x7fafa8158890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:46.459 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.425+0000 7fafae422700 1 -- 172.21.15.195:0/1754206120 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafa40092e0 con 0x7fafa80c7480 2022-06-27T11:52:46.460 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.425+0000 7fafad420700 1 --2- 172.21.15.195:0/1754206120 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fafa8068f80 0x7fafa8151da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T11:52:46.460 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.426+0000 7fafae422700 1 --2- 172.21.15.195:0/1754206120 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fafa80c7480 0x7fafa81522e0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fafa000c840 tx=0x7fafa0003bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:52:46.460 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.426+0000 7faf95ffb700 1 -- 172.21.15.195:0/1754206120 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa0012630 con 0x7fafa80c7480 2022-06-27T11:52:46.461 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.426+0000 7faf95ffb700 1 -- 172.21.15.195:0/1754206120 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafa00127d0 con 0x7fafa80c7480 2022-06-27T11:52:46.461 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.426+0000 7fafafe85700 1 -- 172.21.15.195:0/1754206120 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafa8152e90 con 0x7fafa80c7480 2022-06-27T11:52:46.462 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.426+0000 7faf95ffb700 1 -- 172.21.15.195:0/1754206120 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa0012b90 con 0x7fafa80c7480 2022-06-27T11:52:46.462 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.426+0000 7fafafe85700 1 -- 172.21.15.195:0/1754206120 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafa8153450 con 0x7fafa80c7480 2022-06-27T11:52:46.463 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.427+0000 7fafafe85700 1 -- 172.21.15.195:0/1754206120 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafa8152c10 con 0x7fafa80c7480 2022-06-27T11:52:46.463 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.427+0000 7faf95ffb700 1 -- 172.21.15.195:0/1754206120 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fafa0022070 con 0x7fafa80c7480 2022-06-27T11:52:46.464 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.427+0000 7faf95ffb700 1 --2- 172.21.15.195:0/1754206120 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7faf88063d00 0x7faf880661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:46.464 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.427+0000 7faf95ffb700 1 -- 172.21.15.195:0/1754206120 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6374+0+0 (secure 0 0 0) 0x7fafa0080b50 con 0x7fafa80c7480 2022-06-27T11:52:46.465 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.427+0000 7fafadc21700 1 --2- 172.21.15.195:0/1754206120 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7faf88063d00 0x7faf880661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:46.465 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.428+0000 7fafadc21700 1 --2- 172.21.15.195:0/1754206120 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7faf88063d00 0x7faf880661b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fafa40097a0 tx=0x7fafa4016000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:52:46.466 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.433+0000 7faf95ffb700 1 -- 172.21.15.195:0/1754206120 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fafa004f600 con 0x7fafa80c7480 2022-06-27T11:52:46.661 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.662+0000 7fafafe85700 1 -- 172.21.15.195:0/1754206120 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fafa806ac60 con 0x7fafa80c7480 2022-06-27T11:52:46.662 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.662+0000 7faf95ffb700 1 -- 172.21.15.195:0/1754206120 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5368 (secure 0 0 0) 0x7fafa0052c20 con 0x7fafa80c7480 2022-06-27T11:52:46.662 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:52:46.662 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:52:43.008118+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4760":{"gid":4760,"name":"b","rank":0,"incarnation":20,"state":"up:rejoin","state_seq":17,"addr":"172.21.15.202:6833/1026219246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1026219246},{"type":"v1","addr":"172.21.15.202:6833","nonce":1026219246}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:52:46.664 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.664+0000 7fafafe85700 1 -- 172.21.15.195:0/1754206120 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7faf88063d00 msgr2=0x7faf880661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:46.664 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.664+0000 7fafafe85700 1 --2- 172.21.15.195:0/1754206120 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7faf88063d00 0x7faf880661b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fafa40097a0 tx=0x7fafa4016000 comp rx=0 tx=0).stop 2022-06-27T11:52:46.664 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.664+0000 7fafafe85700 1 -- 172.21.15.195:0/1754206120 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fafa80c7480 msgr2=0x7fafa81522e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:46.665 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.664+0000 7fafafe85700 1 --2- 172.21.15.195:0/1754206120 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fafa80c7480 0x7fafa81522e0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fafa000c840 tx=0x7fafa0003bd0 comp rx=0 tx=0).stop 2022-06-27T11:52:46.665 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.664+0000 7fafafe85700 1 -- 172.21.15.195:0/1754206120 shutdown_connections 2022-06-27T11:52:46.665 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.664+0000 7fafafe85700 1 --2- 172.21.15.195:0/1754206120 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7faf88063d00 0x7faf880661b0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:46.665 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.664+0000 7fafafe85700 1 --2- 172.21.15.195:0/1754206120 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fafa80c7480 0x7fafa81522e0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:46.666 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.664+0000 7fafafe85700 1 --2- 172.21.15.195:0/1754206120 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fafa80683c0 0x7fafa8158890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:46.666 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.664+0000 7fafafe85700 1 --2- 172.21.15.195:0/1754206120 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fafa8068f80 0x7fafa8151da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:46.666 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.664+0000 7fafafe85700 1 -- 172.21.15.195:0/1754206120 >> 172.21.15.195:0/1754206120 conn(0x7fafa80ccbd0 msgr2=0x7fafa8155d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:52:46.666 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.664+0000 7fafafe85700 1 -- 172.21.15.195:0/1754206120 shutdown_connections 2022-06-27T11:52:46.667 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:46.664+0000 7fafafe85700 1 -- 172.21.15.195:0/1754206120 wait complete. 2022-06-27T11:52:46.667 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 22 2022-06-27T11:52:46.677 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T11:52:48.682 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:52:48.841 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:48.830+0000 7f524f591700 1 Processor -- start 2022-06-27T11:52:48.842 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:48.831+0000 7f524f591700 1 -- start start 2022-06-27T11:52:48.842 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:48.831+0000 7f524f591700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f524814e490 0x7f524806cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:48.842 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:48.831+0000 7f524f591700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f52480c3f50 0x7f5248157890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:48.843 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:48.831+0000 7f524f591700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52480c33b0 con 0x7f524814e860 2022-06-27T11:52:48.843 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:48.831+0000 7f524f591700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f524806d520 con 0x7f524814e490 2022-06-27T11:52:48.843 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:48.831+0000 7f524f591700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f524806f220 con 0x7f52480c3f50 2022-06-27T11:52:48.843 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:48.832+0000 7f524d32d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f524814e490 0x7f524806cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:48.844 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:48.832+0000 7f524cb2c700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f524814e860 0x7f52480c3840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:49930/0 (socket says 172.21.15.195:49930) 2022-06-27T11:52:48.844 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:48.832+0000 7f524cb2c700 1 -- 172.21.15.195:0/760185842 learned_addr learned my addr 172.21.15.195:0/760185842 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:52:48.844 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:48.832+0000 7f524db2e700 1 --2- 172.21.15.195:0/760185842 >> v2:172.21.15.202:3300/0 conn(0x7f52480c3f50 0x7f5248157890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:48.844 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:48.832+0000 7f524d32d700 1 -- 172.21.15.195:0/760185842 >> v2:172.21.15.202:3300/0 conn(0x7f52480c3f50 msgr2=0x7f5248157890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:52:48.845 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:48.832+0000 7f524d32d700 1 --2- 172.21.15.195:0/760185842 >> v2:172.21.15.202:3300/0 conn(0x7f52480c3f50 0x7f5248157890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:48.845 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:48.832+0000 7f524d32d700 1 -- 172.21.15.195:0/760185842 >> v1:172.21.15.195:6789/0 conn(0x7f524814e860 legacy=0x7f52480c3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:48.845 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:48.832+0000 7f5237fff700 1 -- 172.21.15.195:0/760185842 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3942043714 0 0) 0x7f52480c33b0 con 0x7f524814e860 2022-06-27T11:52:48.846 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:48.832+0000 7f524d32d700 1 -- 172.21.15.195:0/760185842 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52481542c0 con 0x7f524814e490 2022-06-27T11:52:48.846 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:48.832+0000 7f524d32d700 1 --2- 172.21.15.195:0/760185842 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f524814e490 0x7f524806cfe0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f524806cd00 tx=0x7f5238009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56127912554cd5bc server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:52:48.846 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:48.832+0000 7f5237fff700 1 -- 172.21.15.195:0/760185842 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5238011630 con 0x7f524814e490 2022-06-27T11:52:48.846 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:48.832+0000 7f5237fff700 1 -- 172.21.15.195:0/760185842 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5238011790 con 0x7f524814e490 2022-06-27T11:52:48.847 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:48.832+0000 7f5237fff700 1 -- 172.21.15.195:0/760185842 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5238011b30 con 0x7f524814e490 2022-06-27T11:52:48.847 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:48.833+0000 7f524f591700 1 -- 172.21.15.195:0/760185842 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f524814e490 msgr2=0x7f524806cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:48.847 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:48.833+0000 7f524f591700 1 --2- 172.21.15.195:0/760185842 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f524814e490 0x7f524806cfe0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f524806cd00 tx=0x7f5238009060 comp rx=0 tx=0).stop 2022-06-27T11:52:48.848 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:48.833+0000 7f524f591700 1 -- 172.21.15.195:0/760185842 shutdown_connections 2022-06-27T11:52:48.848 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:48.833+0000 7f524f591700 1 --2- 172.21.15.195:0/760185842 >> v2:172.21.15.202:3300/0 conn(0x7f52480c3f50 0x7f5248157890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:48.848 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:48.833+0000 7f524f591700 1 --2- 172.21.15.195:0/760185842 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f524814e490 0x7f524806cfe0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:48.848 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:48.833+0000 7f524f591700 1 -- 172.21.15.195:0/760185842 >> 172.21.15.195:0/760185842 conn(0x7f5248063af0 msgr2=0x7f5248063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:52:48.849 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:48.833+0000 7f524f591700 1 -- 172.21.15.195:0/760185842 shutdown_connections 2022-06-27T11:52:48.849 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:48.833+0000 7f524f591700 1 -- 172.21.15.195:0/760185842 wait complete. 2022-06-27T11:52:48.849 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:48.834+0000 7f524f591700 1 Processor -- start 2022-06-27T11:52:48.849 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:48.834+0000 7f524f591700 1 -- start start 2022-06-27T11:52:48.851 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:48.835+0000 7f524f591700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f52480c3f50 0x7f524814b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:48.851 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:48.835+0000 7f524f591700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f524814e490 0x7f52480c7980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:48.851 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:48.835+0000 7f524f591700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f524814e860 0x7f52480c7ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:48.852 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:48.835+0000 7f524f591700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52480c0000 con 0x7f524814e490 2022-06-27T11:52:48.852 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:48.835+0000 7f524f591700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5248164d40 con 0x7f524814e860 2022-06-27T11:52:48.852 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:48.835+0000 7f524f591700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5248165020 con 0x7f52480c3f50 2022-06-27T11:52:48.852 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:48.835+0000 7f524cb2c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f524814e490 0x7f52480c7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:48.853 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:48.835+0000 7f524cb2c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f524814e490 0x7f52480c7980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:43750/0 (socket says 172.21.15.195:43750) 2022-06-27T11:52:48.853 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:48.835+0000 7f524cb2c700 1 -- 172.21.15.195:0/3791554913 learned_addr learned my addr 172.21.15.195:0/3791554913 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:52:48.853 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:48.835+0000 7f524db2e700 1 --2- 172.21.15.195:0/3791554913 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f524814e860 0x7f52480c7ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:48.853 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:48.835+0000 7f524d32d700 1 --2- 172.21.15.195:0/3791554913 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f52480c3f50 0x7f524814b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:48.854 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:48.835+0000 7f524cb2c700 1 -- 172.21.15.195:0/3791554913 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f52480c3f50 msgr2=0x7f524814b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:48.854 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:48.835+0000 7f524cb2c700 1 --2- 172.21.15.195:0/3791554913 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f52480c3f50 0x7f524814b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:48.854 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:48.835+0000 7f524cb2c700 1 -- 172.21.15.195:0/3791554913 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f524814e860 msgr2=0x7f52480c7ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:48.855 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:48.835+0000 7f524cb2c700 1 --2- 172.21.15.195:0/3791554913 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f524814e860 0x7f52480c7ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:48.858 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:48.835+0000 7f524cb2c700 1 -- 172.21.15.195:0/3791554913 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52380092e0 con 0x7f524814e490 2022-06-27T11:52:48.859 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:48.835+0000 7f524cb2c700 1 --2- 172.21.15.195:0/3791554913 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f524814e490 0x7f52480c7980 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f524400c490 tx=0x7f5244007dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:52:48.859 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:48.836+0000 7f52357fa700 1 -- 172.21.15.195:0/3791554913 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5244007360 con 0x7f524814e490 2022-06-27T11:52:48.859 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:48.836+0000 7f52357fa700 1 -- 172.21.15.195:0/3791554913 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52440074c0 con 0x7f524814e490 2022-06-27T11:52:48.860 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:48.836+0000 7f524f591700 1 -- 172.21.15.195:0/3791554913 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52480bee40 con 0x7f524814e490 2022-06-27T11:52:48.860 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:48.836+0000 7f52357fa700 1 -- 172.21.15.195:0/3791554913 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5244004940 con 0x7f524814e490 2022-06-27T11:52:48.861 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:48.836+0000 7f524f591700 1 -- 172.21.15.195:0/3791554913 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52480bf400 con 0x7f524814e490 2022-06-27T11:52:48.862 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:48.837+0000 7f524f591700 1 -- 172.21.15.195:0/3791554913 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52481551c0 con 0x7f524814e490 2022-06-27T11:52:48.862 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:48.837+0000 7f52357fa700 1 -- 172.21.15.195:0/3791554913 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f5244010030 con 0x7f524814e490 2022-06-27T11:52:48.862 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:48.837+0000 7f52357fa700 1 --2- 172.21.15.195:0/3791554913 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5228063d00 0x7f52280661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:48.863 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:48.837+0000 7f52357fa700 1 -- 172.21.15.195:0/3791554913 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6374+0+0 (secure 0 0 0) 0x7f52440803a0 con 0x7f524814e490 2022-06-27T11:52:48.864 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:48.837+0000 7f524d32d700 1 --2- 172.21.15.195:0/3791554913 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5228063d00 0x7f52280661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:48.864 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:48.838+0000 7f524d32d700 1 --2- 172.21.15.195:0/3791554913 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5228063d00 0x7f52280661b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5238011fd0 tx=0x7f5238017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:52:48.864 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:48.842+0000 7f52357fa700 1 -- 172.21.15.195:0/3791554913 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f524404ee70 con 0x7f524814e490 2022-06-27T11:52:49.066 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:49.066+0000 7f524f591700 1 -- 172.21.15.195:0/3791554913 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f52480722c0 con 0x7f524814e490 2022-06-27T11:52:49.067 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:49.066+0000 7f52357fa700 1 -- 172.21.15.195:0/3791554913 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+5374 (secure 0 0 0) 0x7f5244052490 con 0x7f524814e490 2022-06-27T11:52:49.067 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:52:49.068 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:52:49.030997+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4760":{"gid":4760,"name":"b","rank":0,"incarnation":20,"state":"up:clientreplay","state_seq":19,"addr":"172.21.15.202:6833/1026219246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1026219246},{"type":"v1","addr":"172.21.15.202:6833","nonce":1026219246}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:52:49.069 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:49.068+0000 7f524f591700 1 -- 172.21.15.195:0/3791554913 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5228063d00 msgr2=0x7f52280661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:49.069 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:49.068+0000 7f524f591700 1 --2- 172.21.15.195:0/3791554913 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5228063d00 0x7f52280661b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5238011fd0 tx=0x7f5238017000 comp rx=0 tx=0).stop 2022-06-27T11:52:49.069 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:49.068+0000 7f524f591700 1 -- 172.21.15.195:0/3791554913 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f524814e490 msgr2=0x7f52480c7980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:49.070 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:49.068+0000 7f524f591700 1 --2- 172.21.15.195:0/3791554913 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f524814e490 0x7f52480c7980 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f524400c490 tx=0x7f5244007dd0 comp rx=0 tx=0).stop 2022-06-27T11:52:49.070 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:49.068+0000 7f524f591700 1 -- 172.21.15.195:0/3791554913 shutdown_connections 2022-06-27T11:52:49.070 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:49.068+0000 7f524f591700 1 --2- 172.21.15.195:0/3791554913 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5228063d00 0x7f52280661b0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:49.070 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:49.068+0000 7f524f591700 1 --2- 172.21.15.195:0/3791554913 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f524814e860 0x7f52480c7ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:49.071 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:49.068+0000 7f524f591700 1 --2- 172.21.15.195:0/3791554913 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f52480c3f50 0x7f524814b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:49.071 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:49.068+0000 7f524f591700 1 --2- 172.21.15.195:0/3791554913 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f524814e490 0x7f52480c7980 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:49.071 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:49.068+0000 7f524f591700 1 -- 172.21.15.195:0/3791554913 >> 172.21.15.195:0/3791554913 conn(0x7f5248063af0 msgr2=0x7f524814d510 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:52:49.071 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:49.069+0000 7f524f591700 1 -- 172.21.15.195:0/3791554913 shutdown_connections 2022-06-27T11:52:49.072 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:49.069+0000 7f524f591700 1 -- 172.21.15.195:0/3791554913 wait complete. 2022-06-27T11:52:49.072 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 23 2022-06-27T11:52:49.079 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T11:52:51.081 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:52:51.219 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.215+0000 7fd34c398700 1 Processor -- start 2022-06-27T11:52:51.220 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.215+0000 7fd34c398700 1 -- start start 2022-06-27T11:52:51.220 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.216+0000 7fd34c398700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd34406e850 0x7fd3441570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:51.221 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.216+0000 7fd34c398700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd3440c34a0 0x7fd3440c2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:51.221 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.216+0000 7fd34c398700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd34406fba0 con 0x7fd34406fef0 2022-06-27T11:52:51.222 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.216+0000 7fd34c398700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3441575e0 con 0x7fd34406e850 2022-06-27T11:52:51.223 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.216+0000 7fd34c398700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd34406f5e0 con 0x7fd3440c34a0 2022-06-27T11:52:51.223 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.216+0000 7fd34a134700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd34406e850 0x7fd3441570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:51.224 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.216+0000 7fd34a134700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd34406e850 0x7fd3441570a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:43756/0 (socket says 172.21.15.195:43756) 2022-06-27T11:52:51.224 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.216+0000 7fd34a134700 1 -- 172.21.15.195:0/2018699250 learned_addr learned my addr 172.21.15.195:0/2018699250 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:52:51.225 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.217+0000 7fd34a935700 1 --2- 172.21.15.195:0/2018699250 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd3440c34a0 0x7fd3440c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:51.225 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.217+0000 7fd34a134700 1 -- 172.21.15.195:0/2018699250 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd3440c34a0 msgr2=0x7fd3440c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:52:51.226 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.217+0000 7fd34a134700 1 --2- 172.21.15.195:0/2018699250 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd3440c34a0 0x7fd3440c2920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:51.226 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.217+0000 7fd34a134700 1 -- 172.21.15.195:0/2018699250 >> v1:172.21.15.202:6789/0 conn(0x7fd34406fef0 legacy=0x7fd344069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:51.227 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.217+0000 7fd34a134700 1 -- 172.21.15.195:0/2018699250 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd34406f8c0 con 0x7fd34406e850 2022-06-27T11:52:51.238 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.217+0000 7fd34a935700 1 --2- 172.21.15.195:0/2018699250 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd3440c34a0 0x7fd3440c2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T11:52:51.239 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.217+0000 7fd34a134700 1 --2- 172.21.15.195:0/2018699250 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd34406e850 0x7fd3441570a0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fd338009860 tx=0x7fd338009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afa74c13624cb981 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:52:51.239 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.217+0000 7fd349132700 1 -- 172.21.15.195:0/2018699250 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd33800b710 con 0x7fd34406e850 2022-06-27T11:52:51.239 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.217+0000 7fd349132700 1 -- 172.21.15.195:0/2018699250 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd33800b870 con 0x7fd34406e850 2022-06-27T11:52:51.240 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.217+0000 7fd349132700 1 -- 172.21.15.195:0/2018699250 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd33800bc30 con 0x7fd34406e850 2022-06-27T11:52:51.240 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.218+0000 7fd34c398700 1 -- 172.21.15.195:0/2018699250 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd34406e850 msgr2=0x7fd3441570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:51.241 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.218+0000 7fd34c398700 1 --2- 172.21.15.195:0/2018699250 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd34406e850 0x7fd3441570a0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fd338009860 tx=0x7fd338009120 comp rx=0 tx=0).stop 2022-06-27T11:52:51.241 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.218+0000 7fd34c398700 1 -- 172.21.15.195:0/2018699250 shutdown_connections 2022-06-27T11:52:51.242 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.218+0000 7fd34c398700 1 --2- 172.21.15.195:0/2018699250 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd3440c34a0 0x7fd3440c2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:51.242 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.218+0000 7fd34c398700 1 --2- 172.21.15.195:0/2018699250 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd34406e850 0x7fd3441570a0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:51.243 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.218+0000 7fd34c398700 1 -- 172.21.15.195:0/2018699250 >> 172.21.15.195:0/2018699250 conn(0x7fd3440c6350 msgr2=0x7fd3440c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:52:51.243 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.218+0000 7fd34c398700 1 -- 172.21.15.195:0/2018699250 shutdown_connections 2022-06-27T11:52:51.244 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.218+0000 7fd34c398700 1 -- 172.21.15.195:0/2018699250 wait complete. 2022-06-27T11:52:51.244 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.219+0000 7fd34c398700 1 Processor -- start 2022-06-27T11:52:51.245 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.219+0000 7fd34c398700 1 -- start start 2022-06-27T11:52:51.245 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.219+0000 7fd34c398700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd34406e850 0x7fd3440c28d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:51.245 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.219+0000 7fd34c398700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd3440c34a0 0x7fd34414c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:51.246 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.220+0000 7fd34c398700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd34414e860 0x7fd34414ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:51.246 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.220+0000 7fd34c398700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd34414c070 con 0x7fd34414e860 2022-06-27T11:52:51.247 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.220+0000 7fd34c398700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd34414b1a0 con 0x7fd3440c34a0 2022-06-27T11:52:51.247 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.220+0000 7fd34c398700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd34414b460 con 0x7fd34406e850 2022-06-27T11:52:51.247 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.220+0000 7fd34a935700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd34414e860 0x7fd34414ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:51.248 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.220+0000 7fd34a935700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd34414e860 0x7fd34414ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:43766/0 (socket says 172.21.15.195:43766) 2022-06-27T11:52:51.248 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.220+0000 7fd34a935700 1 -- 172.21.15.195:0/4273956829 learned_addr learned my addr 172.21.15.195:0/4273956829 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:52:51.248 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.220+0000 7fd34a134700 1 --2- 172.21.15.195:0/4273956829 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd34406e850 0x7fd3440c28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:51.249 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.220+0000 7fd349933700 1 --2- 172.21.15.195:0/4273956829 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd3440c34a0 0x7fd34414c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:51.249 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.220+0000 7fd34a935700 1 -- 172.21.15.195:0/4273956829 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd34406e850 msgr2=0x7fd3440c28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:51.250 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.220+0000 7fd34a935700 1 --2- 172.21.15.195:0/4273956829 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd34406e850 0x7fd3440c28d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:51.250 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.220+0000 7fd34a935700 1 -- 172.21.15.195:0/4273956829 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd3440c34a0 msgr2=0x7fd34414c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:51.250 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.220+0000 7fd34a935700 1 --2- 172.21.15.195:0/4273956829 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd3440c34a0 0x7fd34414c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:51.251 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.220+0000 7fd34a935700 1 -- 172.21.15.195:0/4273956829 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3380093a0 con 0x7fd34414e860 2022-06-27T11:52:51.252 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.220+0000 7fd34a935700 1 --2- 172.21.15.195:0/4273956829 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd34414e860 0x7fd34414ca40 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fd33c00d180 tx=0x7fd33c009a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:52:51.252 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.220+0000 7fd3367fc700 1 -- 172.21.15.195:0/4273956829 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd33c00de20 con 0x7fd34414e860 2022-06-27T11:52:51.253 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.220+0000 7fd3367fc700 1 -- 172.21.15.195:0/4273956829 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd33c00f040 con 0x7fd34414e860 2022-06-27T11:52:51.253 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.220+0000 7fd34c398700 1 -- 172.21.15.195:0/4273956829 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd34414b740 con 0x7fd34414e860 2022-06-27T11:52:51.254 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.220+0000 7fd3367fc700 1 -- 172.21.15.195:0/4273956829 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd33c003af0 con 0x7fd34414e860 2022-06-27T11:52:51.254 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.220+0000 7fd34c398700 1 -- 172.21.15.195:0/4273956829 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd34414bd00 con 0x7fd34414e860 2022-06-27T11:52:51.255 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.221+0000 7fd34c398700 1 -- 172.21.15.195:0/4273956829 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3440722c0 con 0x7fd34414e860 2022-06-27T11:52:51.255 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.222+0000 7fd3367fc700 1 -- 172.21.15.195:0/4273956829 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fd33c01c030 con 0x7fd34414e860 2022-06-27T11:52:51.256 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.222+0000 7fd3367fc700 1 --2- 172.21.15.195:0/4273956829 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd324063d00 0x7fd3240661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:51.256 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.222+0000 7fd3367fc700 1 -- 172.21.15.195:0/4273956829 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6374+0+0 (secure 0 0 0) 0x7fd33c080b70 con 0x7fd34414e860 2022-06-27T11:52:51.256 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.222+0000 7fd34a134700 1 --2- 172.21.15.195:0/4273956829 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd324063d00 0x7fd3240661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:51.257 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.223+0000 7fd34a134700 1 --2- 172.21.15.195:0/4273956829 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd324063d00 0x7fd3240661b0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fd338009160 tx=0x7fd338017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:52:51.257 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.226+0000 7fd3367fc700 1 -- 172.21.15.195:0/4273956829 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fd33c04f620 con 0x7fd34414e860 2022-06-27T11:52:51.455 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.454+0000 7fd34c398700 1 -- 172.21.15.195:0/4273956829 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd344070350 con 0x7fd34414e860 2022-06-27T11:52:51.455 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.455+0000 7fd3367fc700 1 -- 172.21.15.195:0/4273956829 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+5374 (secure 0 0 0) 0x7fd33c01a070 con 0x7fd34414e860 2022-06-27T11:52:51.456 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:52:51.457 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:52:49.030997+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4760":{"gid":4760,"name":"b","rank":0,"incarnation":20,"state":"up:clientreplay","state_seq":19,"addr":"172.21.15.202:6833/1026219246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1026219246},{"type":"v1","addr":"172.21.15.202:6833","nonce":1026219246}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:52:51.459 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.456+0000 7fd34c398700 1 -- 172.21.15.195:0/4273956829 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd324063d00 msgr2=0x7fd3240661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:51.460 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.456+0000 7fd34c398700 1 --2- 172.21.15.195:0/4273956829 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd324063d00 0x7fd3240661b0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fd338009160 tx=0x7fd338017000 comp rx=0 tx=0).stop 2022-06-27T11:52:51.460 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.456+0000 7fd34c398700 1 -- 172.21.15.195:0/4273956829 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd34414e860 msgr2=0x7fd34414ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:51.460 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.456+0000 7fd34c398700 1 --2- 172.21.15.195:0/4273956829 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd34414e860 0x7fd34414ca40 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fd33c00d180 tx=0x7fd33c009a50 comp rx=0 tx=0).stop 2022-06-27T11:52:51.461 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.457+0000 7fd34c398700 1 -- 172.21.15.195:0/4273956829 shutdown_connections 2022-06-27T11:52:51.461 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.457+0000 7fd34c398700 1 --2- 172.21.15.195:0/4273956829 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd324063d00 0x7fd3240661b0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:51.461 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.457+0000 7fd34c398700 1 --2- 172.21.15.195:0/4273956829 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd34414e860 0x7fd34414ca40 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:51.461 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.457+0000 7fd34c398700 1 --2- 172.21.15.195:0/4273956829 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd34406e850 0x7fd3440c28d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:51.462 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.457+0000 7fd34c398700 1 --2- 172.21.15.195:0/4273956829 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd3440c34a0 0x7fd34414c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:51.462 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.457+0000 7fd34c398700 1 -- 172.21.15.195:0/4273956829 >> 172.21.15.195:0/4273956829 conn(0x7fd3440c6350 msgr2=0x7fd3440c9000 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:52:51.462 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.457+0000 7fd34c398700 1 -- 172.21.15.195:0/4273956829 shutdown_connections 2022-06-27T11:52:51.463 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:51.457+0000 7fd34c398700 1 -- 172.21.15.195:0/4273956829 wait complete. 2022-06-27T11:52:51.463 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 23 2022-06-27T11:52:51.468 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T11:52:53.469 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:52:53.617 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.604+0000 7f7e216fd700 1 Processor -- start 2022-06-27T11:52:53.617 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.604+0000 7f7e216fd700 1 -- start start 2022-06-27T11:52:53.617 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.605+0000 7f7e216fd700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f7e1c0c6190 0x7f7e1c157010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:53.618 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.605+0000 7f7e216fd700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e1c156b80 con 0x7f7e1c067ed0 2022-06-27T11:52:53.618 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.605+0000 7f7e216fd700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e1c0682f0 con 0x7f7e1c0c40d0 2022-06-27T11:52:53.618 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.605+0000 7f7e216fd700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e1c06a040 con 0x7f7e1c0c6190 2022-06-27T11:52:53.618 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.605+0000 7f7e1a7fc700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f7e1c0c6190 0x7f7e1c157010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:53.619 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.605+0000 7f7e1a7fc700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f7e1c0c6190 0x7f7e1c157010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:43774/0 (socket says 172.21.15.195:43774) 2022-06-27T11:52:53.619 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.605+0000 7f7e1a7fc700 1 -- 172.21.15.195:0/819090846 learned_addr learned my addr 172.21.15.195:0/819090846 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:52:53.619 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.605+0000 7f7e1b7fe700 1 --1- 172.21.15.195:0/819090846 >> v1:172.21.15.195:6789/0 conn(0x7f7e1c067ed0 0x7f7e1c14cb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:49964/0 (socket says 172.21.15.195:49964) 2022-06-27T11:52:53.619 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.605+0000 7f7e1a7fc700 1 -- 172.21.15.195:0/819090846 >> v1:172.21.15.202:6789/0 conn(0x7f7e1c0c40d0 legacy=0x7f7e1c0c44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:53.620 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.605+0000 7f7e1a7fc700 1 -- 172.21.15.195:0/819090846 >> v1:172.21.15.195:6789/0 conn(0x7f7e1c067ed0 legacy=0x7f7e1c14cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:53.620 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.605+0000 7f7e1a7fc700 1 -- 172.21.15.195:0/819090846 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e1c06a300 con 0x7f7e1c0c6190 2022-06-27T11:52:53.620 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.605+0000 7f7e1a7fc700 1 --2- 172.21.15.195:0/819090846 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7e1c0c6190 0x7f7e1c157010 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c009860 tx=0x7f7e0c009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c4273510c3a23d4 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:52:53.620 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.606+0000 7f7e19ffb700 1 -- 172.21.15.195:0/819090846 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e0c00b680 con 0x7f7e1c0c6190 2022-06-27T11:52:53.620 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.606+0000 7f7e19ffb700 1 -- 172.21.15.195:0/819090846 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e0c00b7e0 con 0x7f7e1c0c6190 2022-06-27T11:52:53.621 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.606+0000 7f7e19ffb700 1 -- 172.21.15.195:0/819090846 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e0c00bba0 con 0x7f7e1c0c6190 2022-06-27T11:52:53.621 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.606+0000 7f7e216fd700 1 -- 172.21.15.195:0/819090846 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7e1c0c6190 msgr2=0x7f7e1c157010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:53.621 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.606+0000 7f7e216fd700 1 --2- 172.21.15.195:0/819090846 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7e1c0c6190 0x7f7e1c157010 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c009860 tx=0x7f7e0c009120 comp rx=0 tx=0).stop 2022-06-27T11:52:53.621 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.606+0000 7f7e216fd700 1 -- 172.21.15.195:0/819090846 shutdown_connections 2022-06-27T11:52:53.622 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.606+0000 7f7e216fd700 1 --2- 172.21.15.195:0/819090846 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7e1c0c6190 0x7f7e1c157010 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:53.622 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.606+0000 7f7e216fd700 1 -- 172.21.15.195:0/819090846 >> 172.21.15.195:0/819090846 conn(0x7f7e1c0ccb80 msgr2=0x7f7e1c0ccf80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:52:53.622 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.606+0000 7f7e216fd700 1 -- 172.21.15.195:0/819090846 shutdown_connections 2022-06-27T11:52:53.622 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.606+0000 7f7e216fd700 1 -- 172.21.15.195:0/819090846 wait complete. 2022-06-27T11:52:53.623 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.607+0000 7f7e216fd700 1 Processor -- start 2022-06-27T11:52:53.623 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.607+0000 7f7e216fd700 1 -- start start 2022-06-27T11:52:53.623 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.608+0000 7f7e216fd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7e1c067ed0 0x7f7e1c1667b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:53.623 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.608+0000 7f7e216fd700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7e1c0c40d0 0x7f7e1c159a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:53.624 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.608+0000 7f7e216fd700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7e1c0c6190 0x7f7e1c159fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:53.624 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.608+0000 7f7e216fd700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1c158d60 con 0x7f7e1c067ed0 2022-06-27T11:52:53.624 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.608+0000 7f7e216fd700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1c15a4e0 con 0x7f7e1c0c6190 2022-06-27T11:52:53.624 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.608+0000 7f7e216fd700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1c158590 con 0x7f7e1c0c40d0 2022-06-27T11:52:53.625 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.608+0000 7f7e1affd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7e1c067ed0 0x7f7e1c1667b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:53.625 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.608+0000 7f7e1affd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7e1c067ed0 0x7f7e1c1667b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:43778/0 (socket says 172.21.15.195:43778) 2022-06-27T11:52:53.625 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.608+0000 7f7e1affd700 1 -- 172.21.15.195:0/1666368195 learned_addr learned my addr 172.21.15.195:0/1666368195 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:52:53.626 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.608+0000 7f7e1a7fc700 1 --2- 172.21.15.195:0/1666368195 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7e1c0c40d0 0x7f7e1c159a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:53.627 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.608+0000 7f7e1b7fe700 1 --2- 172.21.15.195:0/1666368195 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7e1c0c6190 0x7f7e1c159fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:53.627 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.609+0000 7f7e1affd700 1 -- 172.21.15.195:0/1666368195 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7e1c0c40d0 msgr2=0x7f7e1c159a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:53.627 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.609+0000 7f7e1affd700 1 --2- 172.21.15.195:0/1666368195 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7e1c0c40d0 0x7f7e1c159a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:53.628 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.609+0000 7f7e1affd700 1 -- 172.21.15.195:0/1666368195 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7e1c0c6190 msgr2=0x7f7e1c159fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:53.628 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.609+0000 7f7e1affd700 1 --2- 172.21.15.195:0/1666368195 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7e1c0c6190 0x7f7e1c159fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:53.628 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.609+0000 7f7e1affd700 1 -- 172.21.15.195:0/1666368195 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e0c0093a0 con 0x7f7e1c067ed0 2022-06-27T11:52:53.629 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.609+0000 7f7e1a7fc700 1 --2- 172.21.15.195:0/1666368195 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7e1c0c40d0 0x7f7e1c159a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T11:52:53.629 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.609+0000 7f7e1b7fe700 1 --2- 172.21.15.195:0/1666368195 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7e1c0c6190 0x7f7e1c159fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T11:52:53.629 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.609+0000 7f7e1affd700 1 --2- 172.21.15.195:0/1666368195 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7e1c067ed0 0x7f7e1c1667b0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f7e1000c490 tx=0x7f7e10007e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:52:53.629 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.609+0000 7f7e02ffd700 1 -- 172.21.15.195:0/1666368195 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e100050d0 con 0x7f7e1c067ed0 2022-06-27T11:52:53.630 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.609+0000 7f7e02ffd700 1 -- 172.21.15.195:0/1666368195 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e10005230 con 0x7f7e1c067ed0 2022-06-27T11:52:53.630 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.609+0000 7f7e216fd700 1 -- 172.21.15.195:0/1666368195 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e1c1672c0 con 0x7f7e1c067ed0 2022-06-27T11:52:53.630 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.609+0000 7f7e02ffd700 1 -- 172.21.15.195:0/1666368195 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e100117a0 con 0x7f7e1c067ed0 2022-06-27T11:52:53.630 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.609+0000 7f7e216fd700 1 -- 172.21.15.195:0/1666368195 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e1c158a80 con 0x7f7e1c067ed0 2022-06-27T11:52:53.631 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.610+0000 7f7e216fd700 1 -- 172.21.15.195:0/1666368195 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e1c0722c0 con 0x7f7e1c067ed0 2022-06-27T11:52:53.631 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.610+0000 7f7e02ffd700 1 -- 172.21.15.195:0/1666368195 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f7e10019030 con 0x7f7e1c067ed0 2022-06-27T11:52:53.631 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.610+0000 7f7e02ffd700 1 --2- 172.21.15.195:0/1666368195 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7dfc063d00 0x7f7dfc0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:53.631 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.611+0000 7f7e02ffd700 1 -- 172.21.15.195:0/1666368195 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6374+0+0 (secure 0 0 0) 0x7f7e1007fb40 con 0x7f7e1c067ed0 2022-06-27T11:52:53.632 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.611+0000 7f7e1a7fc700 1 --2- 172.21.15.195:0/1666368195 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7dfc063d00 0x7f7dfc0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:53.632 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.611+0000 7f7e1a7fc700 1 --2- 172.21.15.195:0/1666368195 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7dfc063d00 0x7f7dfc0661b0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c157430 tx=0x7f7e0c017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:52:53.632 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.615+0000 7f7e02ffd700 1 -- 172.21.15.195:0/1666368195 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f7e10049c90 con 0x7f7e1c067ed0 2022-06-27T11:52:53.841 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.842+0000 7f7e216fd700 1 -- 172.21.15.195:0/1666368195 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e1c1598d0 con 0x7f7e1c067ed0 2022-06-27T11:52:53.842 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.842+0000 7f7e02ffd700 1 -- 172.21.15.195:0/1666368195 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+5374 (secure 0 0 0) 0x7f7e10017020 con 0x7f7e1c067ed0 2022-06-27T11:52:53.843 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:52:53.843 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:52:49.030997+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4760":{"gid":4760,"name":"b","rank":0,"incarnation":20,"state":"up:clientreplay","state_seq":19,"addr":"172.21.15.202:6833/1026219246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1026219246},{"type":"v1","addr":"172.21.15.202:6833","nonce":1026219246}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:52:53.845 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.844+0000 7f7e216fd700 1 -- 172.21.15.195:0/1666368195 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7dfc063d00 msgr2=0x7f7dfc0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:53.846 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.844+0000 7f7e216fd700 1 --2- 172.21.15.195:0/1666368195 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7dfc063d00 0x7f7dfc0661b0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c157430 tx=0x7f7e0c017000 comp rx=0 tx=0).stop 2022-06-27T11:52:53.846 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.844+0000 7f7e216fd700 1 -- 172.21.15.195:0/1666368195 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7e1c067ed0 msgr2=0x7f7e1c1667b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:53.846 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.844+0000 7f7e216fd700 1 --2- 172.21.15.195:0/1666368195 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7e1c067ed0 0x7f7e1c1667b0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f7e1000c490 tx=0x7f7e10007e40 comp rx=0 tx=0).stop 2022-06-27T11:52:53.846 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.844+0000 7f7e216fd700 1 -- 172.21.15.195:0/1666368195 shutdown_connections 2022-06-27T11:52:53.847 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.844+0000 7f7e216fd700 1 --2- 172.21.15.195:0/1666368195 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7dfc063d00 0x7f7dfc0661b0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:53.847 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.844+0000 7f7e216fd700 1 --2- 172.21.15.195:0/1666368195 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7e1c067ed0 0x7f7e1c1667b0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:53.847 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.844+0000 7f7e216fd700 1 --2- 172.21.15.195:0/1666368195 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7e1c0c6190 0x7f7e1c159fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:53.847 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.844+0000 7f7e216fd700 1 --2- 172.21.15.195:0/1666368195 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7e1c0c40d0 0x7f7e1c159a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:53.848 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.844+0000 7f7e216fd700 1 -- 172.21.15.195:0/1666368195 >> 172.21.15.195:0/1666368195 conn(0x7f7e1c0ccb80 msgr2=0x7f7e1c0c44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:52:53.848 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.844+0000 7f7e216fd700 1 -- 172.21.15.195:0/1666368195 shutdown_connections 2022-06-27T11:52:53.848 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:53.844+0000 7f7e216fd700 1 -- 172.21.15.195:0/1666368195 wait complete. 2022-06-27T11:52:53.848 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 23 2022-06-27T11:52:53.856 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T11:52:55.858 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:52:55.992 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:55.993+0000 7f0429e5d700 1 Processor -- start 2022-06-27T11:52:55.993 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:55.993+0000 7f0429e5d700 1 -- start start 2022-06-27T11:52:55.994 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:55.994+0000 7f0429e5d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f04241509c0 0x7f04240c7440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:55.994 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:55.994+0000 7f0429e5d700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04240c43d0 con 0x7f0424067ed0 2022-06-27T11:52:55.995 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:55.994+0000 7f0429e5d700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0424068430 con 0x7f0424156670 2022-06-27T11:52:55.995 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:55.994+0000 7f0429e5d700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0424068710 con 0x7f04241509c0 2022-06-27T11:52:55.995 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:55.994+0000 7f0423fff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f04241509c0 0x7f04240c7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:55.995 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:55.994+0000 7f0422ffd700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f0424067ed0 0x7f04241502b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:49976/0 (socket says 172.21.15.195:49976) 2022-06-27T11:52:55.996 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:55.994+0000 7f0422ffd700 1 -- 172.21.15.195:0/1066621888 learned_addr learned my addr 172.21.15.195:0/1066621888 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:52:55.996 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:55.994+0000 7f0423fff700 1 -- 172.21.15.195:0/1066621888 >> v1:172.21.15.202:6789/0 conn(0x7f0424156670 legacy=0x7f04240c4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:52:55.996 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:55.994+0000 7f0423fff700 1 -- 172.21.15.195:0/1066621888 >> v1:172.21.15.195:6789/0 conn(0x7f0424067ed0 legacy=0x7f04241502b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:55.996 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:55.994+0000 7f0423fff700 1 -- 172.21.15.195:0/1066621888 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04240689f0 con 0x7f04241509c0 2022-06-27T11:52:55.997 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:55.994+0000 7f0423fff700 1 --2- 172.21.15.195:0/1066621888 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f04241509c0 0x7f04240c7440 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f041c009e60 tx=0x7f041c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92b15ec82ac81fd3 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:52:55.997 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:55.994+0000 7f04227fc700 1 -- 172.21.15.195:0/1066621888 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f041c00e070 con 0x7f04241509c0 2022-06-27T11:52:55.998 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:55.994+0000 7f04227fc700 1 -- 172.21.15.195:0/1066621888 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f041c00bbf0 con 0x7f04241509c0 2022-06-27T11:52:56.004 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:55.994+0000 7f04227fc700 1 -- 172.21.15.195:0/1066621888 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f041c0038a0 con 0x7f04241509c0 2022-06-27T11:52:56.005 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:55.995+0000 7f0429e5d700 1 -- 172.21.15.195:0/1066621888 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f04241509c0 msgr2=0x7f04240c7440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:56.005 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:55.995+0000 7f0429e5d700 1 --2- 172.21.15.195:0/1066621888 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f04241509c0 0x7f04240c7440 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f041c009e60 tx=0x7f041c009780 comp rx=0 tx=0).stop 2022-06-27T11:52:56.006 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:55.995+0000 7f0429e5d700 1 -- 172.21.15.195:0/1066621888 shutdown_connections 2022-06-27T11:52:56.006 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:55.995+0000 7f0429e5d700 1 --2- 172.21.15.195:0/1066621888 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f04241509c0 0x7f04240c7440 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:56.007 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:55.995+0000 7f0429e5d700 1 -- 172.21.15.195:0/1066621888 >> 172.21.15.195:0/1066621888 conn(0x7f0424069e00 msgr2=0x7f042406a200 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:52:56.007 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:55.995+0000 7f0429e5d700 1 -- 172.21.15.195:0/1066621888 shutdown_connections 2022-06-27T11:52:56.008 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:55.995+0000 7f0429e5d700 1 -- 172.21.15.195:0/1066621888 wait complete. 2022-06-27T11:52:56.008 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:55.996+0000 7f0429e5d700 1 Processor -- start 2022-06-27T11:52:56.009 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:55.996+0000 7f0429e5d700 1 -- start start 2022-06-27T11:52:56.009 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:55.997+0000 7f0429e5d700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0424067ed0 0x7f04241526a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:56.009 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:55.997+0000 7f0429e5d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f04241509c0 0x7f04240c7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:56.010 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:55.997+0000 7f0429e5d700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0424156670 0x7f04240c8200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:56.010 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:55.997+0000 7f0429e5d700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04241518f0 con 0x7f04241509c0 2022-06-27T11:52:56.011 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:55.997+0000 7f0429e5d700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04241595d0 con 0x7f0424156670 2022-06-27T11:52:56.011 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:55.997+0000 7f0429e5d700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0424153000 con 0x7f0424067ed0 2022-06-27T11:52:56.011 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:55.997+0000 7f0422ffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f04241509c0 0x7f04240c7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:56.012 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:55.997+0000 7f0422ffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f04241509c0 0x7f04240c7cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:43794/0 (socket says 172.21.15.195:43794) 2022-06-27T11:52:56.012 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:55.997+0000 7f0422ffd700 1 -- 172.21.15.195:0/1785211753 learned_addr learned my addr 172.21.15.195:0/1785211753 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:52:56.013 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:55.997+0000 7f0423fff700 1 --2- 172.21.15.195:0/1785211753 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0424156670 0x7f04240c8200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:56.013 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:55.997+0000 7f04237fe700 1 --2- 172.21.15.195:0/1785211753 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0424067ed0 0x7f04241526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:56.013 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:55.997+0000 7f0422ffd700 1 -- 172.21.15.195:0/1785211753 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0424067ed0 msgr2=0x7f04241526a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:56.014 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:55.997+0000 7f0422ffd700 1 --2- 172.21.15.195:0/1785211753 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0424067ed0 0x7f04241526a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:56.014 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:55.997+0000 7f0422ffd700 1 -- 172.21.15.195:0/1785211753 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0424156670 msgr2=0x7f04240c8200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:56.014 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:55.997+0000 7f0422ffd700 1 --2- 172.21.15.195:0/1785211753 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0424156670 0x7f04240c8200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:56.015 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:55.997+0000 7f0422ffd700 1 -- 172.21.15.195:0/1785211753 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f041c009a20 con 0x7f04241509c0 2022-06-27T11:52:56.015 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:55.997+0000 7f0422ffd700 1 --2- 172.21.15.195:0/1785211753 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f04241509c0 0x7f04240c7cc0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f041800c4e0 tx=0x7f0418007e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:52:56.015 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:55.998+0000 7f040b7fe700 1 -- 172.21.15.195:0/1785211753 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f041800ce30 con 0x7f04241509c0 2022-06-27T11:52:56.016 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:55.998+0000 7f040b7fe700 1 -- 172.21.15.195:0/1785211753 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04180072e0 con 0x7f04241509c0 2022-06-27T11:52:56.016 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:55.998+0000 7f040b7fe700 1 -- 172.21.15.195:0/1785211753 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0418007680 con 0x7f04241509c0 2022-06-27T11:52:56.016 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:55.998+0000 7f0429e5d700 1 -- 172.21.15.195:0/1785211753 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04240c9900 con 0x7f04241509c0 2022-06-27T11:52:56.017 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:55.998+0000 7f0429e5d700 1 -- 172.21.15.195:0/1785211753 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0424159260 con 0x7f04241509c0 2022-06-27T11:52:56.017 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:55.999+0000 7f0429e5d700 1 -- 172.21.15.195:0/1785211753 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04240722c0 con 0x7f04241509c0 2022-06-27T11:52:56.018 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:55.999+0000 7f040b7fe700 1 -- 172.21.15.195:0/1785211753 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f0418010070 con 0x7f04241509c0 2022-06-27T11:52:56.018 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:55.999+0000 7f040b7fe700 1 --2- 172.21.15.195:0/1785211753 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0404063d00 0x7f04040661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:56.019 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:55.999+0000 7f040b7fe700 1 -- 172.21.15.195:0/1785211753 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6374+0+0 (secure 0 0 0) 0x7f041807ff00 con 0x7f04241509c0 2022-06-27T11:52:56.019 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:56.000+0000 7f04237fe700 1 --2- 172.21.15.195:0/1785211753 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0404063d00 0x7f04040661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:56.019 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:56.000+0000 7f04237fe700 1 --2- 172.21.15.195:0/1785211753 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0404063d00 0x7f04040661b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f041400b990 tx=0x7f0414008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:52:56.020 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:56.004+0000 7f040b7fe700 1 -- 172.21.15.195:0/1785211753 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f041804e9b0 con 0x7f04241509c0 2022-06-27T11:52:56.229 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:56.230+0000 7f0429e5d700 1 -- 172.21.15.195:0/1785211753 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0424068cd0 con 0x7f04241509c0 2022-06-27T11:52:56.230 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:56.230+0000 7f040b7fe700 1 -- 172.21.15.195:0/1785211753 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5368 (secure 0 0 0) 0x7f0418051fd0 con 0x7f04241509c0 2022-06-27T11:52:56.231 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:52:56.231 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:52:55.042048+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4760":{"gid":4760,"name":"b","rank":0,"incarnation":20,"state":"up:active","state_seq":21,"addr":"172.21.15.202:6833/1026219246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1026219246},{"type":"v1","addr":"172.21.15.202:6833","nonce":1026219246}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:52:56.233 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:56.232+0000 7f0429e5d700 1 -- 172.21.15.195:0/1785211753 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0404063d00 msgr2=0x7f04040661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:56.235 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:56.232+0000 7f0429e5d700 1 --2- 172.21.15.195:0/1785211753 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0404063d00 0x7f04040661b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f041400b990 tx=0x7f0414008040 comp rx=0 tx=0).stop 2022-06-27T11:52:56.235 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:56.232+0000 7f0429e5d700 1 -- 172.21.15.195:0/1785211753 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f04241509c0 msgr2=0x7f04240c7cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:56.236 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:56.232+0000 7f0429e5d700 1 --2- 172.21.15.195:0/1785211753 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f04241509c0 0x7f04240c7cc0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f041800c4e0 tx=0x7f0418007e30 comp rx=0 tx=0).stop 2022-06-27T11:52:56.236 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:56.232+0000 7f0429e5d700 1 -- 172.21.15.195:0/1785211753 shutdown_connections 2022-06-27T11:52:56.237 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:56.232+0000 7f0429e5d700 1 --2- 172.21.15.195:0/1785211753 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0404063d00 0x7f04040661b0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:56.237 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:56.232+0000 7f0429e5d700 1 --2- 172.21.15.195:0/1785211753 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0424156670 0x7f04240c8200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:56.238 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:56.232+0000 7f0429e5d700 1 --2- 172.21.15.195:0/1785211753 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0424067ed0 0x7f04241526a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:56.238 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:56.232+0000 7f0429e5d700 1 --2- 172.21.15.195:0/1785211753 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f04241509c0 0x7f04240c7cc0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:56.238 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:56.232+0000 7f0429e5d700 1 -- 172.21.15.195:0/1785211753 >> 172.21.15.195:0/1785211753 conn(0x7f0424069e00 msgr2=0x7f0424154390 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:52:56.239 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:56.232+0000 7f0429e5d700 1 -- 172.21.15.195:0/1785211753 shutdown_connections 2022-06-27T11:52:56.239 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:56.232+0000 7f0429e5d700 1 -- 172.21.15.195:0/1785211753 wait complete. 2022-06-27T11:52:56.240 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 24 2022-06-27T11:52:56.248 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=4849 2022-06-27T11:52:56.249 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 2 secs before reviving mds.d 2022-06-27T11:52:57.409 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T11:52:57.436 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T11:52:58.254 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.d 2022-06-27T11:52:58.254 INFO:tasks.ceph.mds.d:Restarting daemon 2022-06-27T11:52:58.256 DEBUG:teuthology.orchestra.run.smithi202:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2022-06-27T11:52:58.269 INFO:tasks.ceph.mds.d:Started 2022-06-27T11:52:58.270 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:52:58.353 INFO:tasks.ceph.mds.d.smithi202.stdout:starting mds.d at 2022-06-27T11:52:58.414 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.415+0000 7f1cfe3ed700 1 Processor -- start 2022-06-27T11:52:58.415 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.415+0000 7f1cfe3ed700 1 -- start start 2022-06-27T11:52:58.428 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.415+0000 7f1cfe3ed700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1cf806a450 0x7f1cf8063af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:58.429 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.415+0000 7f1cfe3ed700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f1cf81508d0 0x7f1cf80bbf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:58.429 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.415+0000 7f1cfe3ed700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cf8064030 con 0x7f1cf8150500 2022-06-27T11:52:58.429 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.415+0000 7f1cfe3ed700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cf80c4160 con 0x7f1cf806a450 2022-06-27T11:52:58.429 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.415+0000 7f1cfe3ed700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cf80c4440 con 0x7f1cf81508d0 2022-06-27T11:52:58.430 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.416+0000 7f1cf77fe700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1cf806a450 0x7f1cf8063af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:58.431 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.416+0000 7f1cf7fff700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f1cf8150500 0x7f1cf80c48d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:49988/0 (socket says 172.21.15.195:49988) 2022-06-27T11:52:58.431 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.416+0000 7f1cf7fff700 1 -- 172.21.15.195:0/1149931581 learned_addr learned my addr 172.21.15.195:0/1149931581 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:52:58.432 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.416+0000 7f1cfc98a700 1 --2- 172.21.15.195:0/1149931581 >> v2:172.21.15.202:3300/0 conn(0x7f1cf81508d0 0x7f1cf80bbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:58.432 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.416+0000 7f1cf77fe700 1 -- 172.21.15.195:0/1149931581 >> v2:172.21.15.202:3300/0 conn(0x7f1cf81508d0 msgr2=0x7f1cf80bbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:52:58.433 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.416+0000 7f1cf77fe700 1 --2- 172.21.15.195:0/1149931581 >> v2:172.21.15.202:3300/0 conn(0x7f1cf81508d0 0x7f1cf80bbf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:58.441 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.416+0000 7f1cf77fe700 1 -- 172.21.15.195:0/1149931581 >> v1:172.21.15.195:6789/0 conn(0x7f1cf8150500 legacy=0x7f1cf80c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:58.442 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.416+0000 7f1cf6ffd700 1 -- 172.21.15.195:0/1149931581 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1702937336 0 0) 0x7f1ce8001290 con 0x7f1cf8150500 2022-06-27T11:52:58.443 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.416+0000 7f1cf77fe700 1 -- 172.21.15.195:0/1149931581 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cf815a800 con 0x7f1cf806a450 2022-06-27T11:52:58.443 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.416+0000 7f1cf77fe700 1 --2- 172.21.15.195:0/1149931581 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1cf806a450 0x7f1cf8063af0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f1cf8064800 tx=0x7f1cec009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49f1b0bf8d50684d server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:52:58.445 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.416+0000 7f1cf6ffd700 1 -- 172.21.15.195:0/1149931581 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cec011630 con 0x7f1cf806a450 2022-06-27T11:52:58.449 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.416+0000 7f1cf6ffd700 1 -- 172.21.15.195:0/1149931581 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cec011790 con 0x7f1cf806a450 2022-06-27T11:52:58.450 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.417+0000 7f1cf6ffd700 1 -- 172.21.15.195:0/1149931581 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cec011b30 con 0x7f1cf806a450 2022-06-27T11:52:58.450 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.417+0000 7f1cfe3ed700 1 -- 172.21.15.195:0/1149931581 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1cf806a450 msgr2=0x7f1cf8063af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:58.452 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.417+0000 7f1cfe3ed700 1 --2- 172.21.15.195:0/1149931581 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1cf806a450 0x7f1cf8063af0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f1cf8064800 tx=0x7f1cec009060 comp rx=0 tx=0).stop 2022-06-27T11:52:58.452 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.417+0000 7f1cfe3ed700 1 -- 172.21.15.195:0/1149931581 shutdown_connections 2022-06-27T11:52:58.452 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.417+0000 7f1cfe3ed700 1 --2- 172.21.15.195:0/1149931581 >> v2:172.21.15.202:3300/0 conn(0x7f1cf81508d0 0x7f1cf80bbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:58.453 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.417+0000 7f1cfe3ed700 1 --2- 172.21.15.195:0/1149931581 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1cf806a450 0x7f1cf8063af0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:58.453 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.417+0000 7f1cfe3ed700 1 -- 172.21.15.195:0/1149931581 >> 172.21.15.195:0/1149931581 conn(0x7f1cf80c9b00 msgr2=0x7f1cf80c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:52:58.453 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.417+0000 7f1cfe3ed700 1 -- 172.21.15.195:0/1149931581 shutdown_connections 2022-06-27T11:52:58.453 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.417+0000 7f1cfe3ed700 1 -- 172.21.15.195:0/1149931581 wait complete. 2022-06-27T11:52:58.454 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.418+0000 7f1cfe3ed700 1 Processor -- start 2022-06-27T11:52:58.454 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.418+0000 7f1cfe3ed700 1 -- start start 2022-06-27T11:52:58.454 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.419+0000 7f1cfe3ed700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1cf806a450 0x7f1cf80bde90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:58.454 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.419+0000 7f1cfe3ed700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1cf8150500 0x7f1cf806eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:58.454 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.419+0000 7f1cfe3ed700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1cf81508d0 0x7f1cf8169ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:58.455 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.419+0000 7f1cfe3ed700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf80c2950 con 0x7f1cf81508d0 2022-06-27T11:52:58.455 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.419+0000 7f1cfe3ed700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf80c2f10 con 0x7f1cf8150500 2022-06-27T11:52:58.455 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.419+0000 7f1cfe3ed700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf8159b10 con 0x7f1cf806a450 2022-06-27T11:52:58.455 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.419+0000 7f1cfc98a700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1cf81508d0 0x7f1cf8169ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:58.456 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.419+0000 7f1cfc98a700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1cf81508d0 0x7f1cf8169ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:43810/0 (socket says 172.21.15.195:43810) 2022-06-27T11:52:58.456 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.419+0000 7f1cfc98a700 1 -- 172.21.15.195:0/494172540 learned_addr learned my addr 172.21.15.195:0/494172540 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:52:58.456 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.419+0000 7f1cf7fff700 1 --2- 172.21.15.195:0/494172540 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1cf806a450 0x7f1cf80bde90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:58.457 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.419+0000 7f1cf77fe700 1 --2- 172.21.15.195:0/494172540 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1cf8150500 0x7f1cf806eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:58.457 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.420+0000 7f1cfc98a700 1 -- 172.21.15.195:0/494172540 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1cf806a450 msgr2=0x7f1cf80bde90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:58.457 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.420+0000 7f1cfc98a700 1 --2- 172.21.15.195:0/494172540 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1cf806a450 0x7f1cf80bde90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:58.457 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.420+0000 7f1cfc98a700 1 -- 172.21.15.195:0/494172540 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1cf8150500 msgr2=0x7f1cf806eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:58.458 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.420+0000 7f1cfc98a700 1 --2- 172.21.15.195:0/494172540 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1cf8150500 0x7f1cf806eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:58.458 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.420+0000 7f1cfc98a700 1 -- 172.21.15.195:0/494172540 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cec0092e0 con 0x7f1cf81508d0 2022-06-27T11:52:58.458 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.420+0000 7f1cf77fe700 1 --2- 172.21.15.195:0/494172540 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1cf8150500 0x7f1cf806eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T11:52:58.460 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.420+0000 7f1cfc98a700 1 --2- 172.21.15.195:0/494172540 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1cf81508d0 0x7f1cf8169ce0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f1cf0009e20 tx=0x7f1cf000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:52:58.460 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.420+0000 7f1cdffff700 1 -- 172.21.15.195:0/494172540 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cf000cb50 con 0x7f1cf81508d0 2022-06-27T11:52:58.460 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.420+0000 7f1cdffff700 1 -- 172.21.15.195:0/494172540 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cf000f040 con 0x7f1cf81508d0 2022-06-27T11:52:58.460 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.420+0000 7f1cfe3ed700 1 -- 172.21.15.195:0/494172540 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cf80bcfc0 con 0x7f1cf81508d0 2022-06-27T11:52:58.461 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.420+0000 7f1cdffff700 1 -- 172.21.15.195:0/494172540 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cf0014480 con 0x7f1cf81508d0 2022-06-27T11:52:58.461 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.420+0000 7f1cfe3ed700 1 -- 172.21.15.195:0/494172540 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cf80c2ba0 con 0x7f1cf81508d0 2022-06-27T11:52:58.461 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.421+0000 7f1cfe3ed700 1 -- 172.21.15.195:0/494172540 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cf80bd4e0 con 0x7f1cf81508d0 2022-06-27T11:52:58.462 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.421+0000 7f1cdffff700 1 -- 172.21.15.195:0/494172540 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f1cf0014830 con 0x7f1cf81508d0 2022-06-27T11:52:58.462 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.422+0000 7f1cdffff700 1 --2- 172.21.15.195:0/494172540 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1cd8063d00 0x7f1cd80661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:52:58.462 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.422+0000 7f1cf7fff700 1 --2- 172.21.15.195:0/494172540 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1cd8063d00 0x7f1cd80661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:52:58.462 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.422+0000 7f1cdffff700 1 -- 172.21.15.195:0/494172540 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6374+0+0 (secure 0 0 0) 0x7f1cf0081590 con 0x7f1cf81508d0 2022-06-27T11:52:58.463 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.422+0000 7f1cf7fff700 1 --2- 172.21.15.195:0/494172540 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1cd8063d00 0x7f1cd80661b0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f1ce8001eb0 tx=0x7f1ce80017d0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:52:58.463 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.426+0000 7f1cdffff700 1 -- 172.21.15.195:0/494172540 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f1cf004ff50 con 0x7f1cf81508d0 2022-06-27T11:52:58.645 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.646+0000 7f1cfe3ed700 1 -- 172.21.15.195:0/494172540 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1cf816a640 con 0x7f1cf81508d0 2022-06-27T11:52:58.646 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.646+0000 7f1cdffff700 1 -- 172.21.15.195:0/494172540 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5368 (secure 0 0 0) 0x7f1cf0018090 con 0x7f1cf81508d0 2022-06-27T11:52:58.646 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:52:58.646 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:52:55.042048+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4760":{"gid":4760,"name":"b","rank":0,"incarnation":20,"state":"up:active","state_seq":21,"addr":"172.21.15.202:6833/1026219246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1026219246},{"type":"v1","addr":"172.21.15.202:6833","nonce":1026219246}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:52:58.653 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.648+0000 7f1cfe3ed700 1 -- 172.21.15.195:0/494172540 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1cd8063d00 msgr2=0x7f1cd80661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:58.654 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.648+0000 7f1cfe3ed700 1 --2- 172.21.15.195:0/494172540 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1cd8063d00 0x7f1cd80661b0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f1ce8001eb0 tx=0x7f1ce80017d0 comp rx=0 tx=0).stop 2022-06-27T11:52:58.654 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.648+0000 7f1cfe3ed700 1 -- 172.21.15.195:0/494172540 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1cf81508d0 msgr2=0x7f1cf8169ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:52:58.655 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.648+0000 7f1cfe3ed700 1 --2- 172.21.15.195:0/494172540 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1cf81508d0 0x7f1cf8169ce0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f1cf0009e20 tx=0x7f1cf000b040 comp rx=0 tx=0).stop 2022-06-27T11:52:58.655 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.648+0000 7f1cfe3ed700 1 -- 172.21.15.195:0/494172540 shutdown_connections 2022-06-27T11:52:58.656 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.648+0000 7f1cfe3ed700 1 --2- 172.21.15.195:0/494172540 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1cd8063d00 0x7f1cd80661b0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:58.656 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.648+0000 7f1cfe3ed700 1 --2- 172.21.15.195:0/494172540 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1cf81508d0 0x7f1cf8169ce0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:58.657 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.648+0000 7f1cfe3ed700 1 --2- 172.21.15.195:0/494172540 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1cf806a450 0x7f1cf80bde90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:58.657 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.648+0000 7f1cfe3ed700 1 --2- 172.21.15.195:0/494172540 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1cf8150500 0x7f1cf806eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:52:58.658 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.648+0000 7f1cfe3ed700 1 -- 172.21.15.195:0/494172540 >> 172.21.15.195:0/494172540 conn(0x7f1cf80c9b00 msgr2=0x7f1cf80c78e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:52:58.662 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.648+0000 7f1cfe3ed700 1 -- 172.21.15.195:0/494172540 shutdown_connections 2022-06-27T11:52:58.662 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:52:58.649+0000 7f1cfe3ed700 1 -- 172.21.15.195:0/494172540 wait complete. 2022-06-27T11:52:58.663 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 24 2022-06-27T11:52:58.666 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2022-06-27T11:53:00.667 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:53:00.800 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:00.801+0000 7fe58878c700 1 Processor -- start 2022-06-27T11:53:00.801 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:00.801+0000 7fe58878c700 1 -- start start 2022-06-27T11:53:00.807 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:00.802+0000 7fe58878c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fe58006f640 0x7fe580156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:53:00.807 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:00.802+0000 7fe58878c700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fe5800c4160 0x7fe5800c1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:53:00.808 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:00.802+0000 7fe58878c700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe5800c39e0 0x7fe5801535a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:53:00.808 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:00.802+0000 7fe58878c700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5800c1a20 con 0x7fe58006f640 2022-06-27T11:53:00.808 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:00.802+0000 7fe58878c700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5800c1130 con 0x7fe5800c4160 2022-06-27T11:53:00.809 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:00.802+0000 7fe58878c700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe58006ea90 con 0x7fe5800c39e0 2022-06-27T11:53:00.809 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:00.802+0000 7fe586528700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fe58006f640 0x7fe580156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:53:00.809 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:00.802+0000 7fe586528700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fe58006f640 0x7fe580156ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:43818/0 (socket says 172.21.15.195:43818) 2022-06-27T11:53:00.810 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:00.802+0000 7fe586528700 1 -- 172.21.15.195:0/3104250214 learned_addr learned my addr 172.21.15.195:0/3104250214 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:53:00.810 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:00.802+0000 7fe586d29700 1 --2- 172.21.15.195:0/3104250214 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe5800c39e0 0x7fe5801535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:53:00.810 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:00.802+0000 7fe585d27700 1 --2- 172.21.15.195:0/3104250214 >> v2:172.21.15.202:3300/0 conn(0x7fe5800c4160 0x7fe5800c1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:53:00.810 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:00.802+0000 7fe586528700 1 -- 172.21.15.195:0/3104250214 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe5800c39e0 msgr2=0x7fe5801535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:53:00.812 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:00.802+0000 7fe586528700 1 --2- 172.21.15.195:0/3104250214 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe5800c39e0 0x7fe5801535a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:53:00.812 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:00.802+0000 7fe586528700 1 -- 172.21.15.195:0/3104250214 >> v2:172.21.15.202:3300/0 conn(0x7fe5800c4160 msgr2=0x7fe5800c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:53:00.812 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:00.802+0000 7fe586528700 1 --2- 172.21.15.195:0/3104250214 >> v2:172.21.15.202:3300/0 conn(0x7fe5800c4160 0x7fe5800c1eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:53:00.813 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:00.803+0000 7fe586528700 1 -- 172.21.15.195:0/3104250214 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe58006ed70 con 0x7fe58006f640 2022-06-27T11:53:00.814 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:00.803+0000 7fe586528700 1 --2- 172.21.15.195:0/3104250214 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe58006f640 0x7fe580156ed0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fe58006dd70 tx=0x7fe570009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c201cd56009bb45a server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:53:00.814 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:00.803+0000 7fe585526700 1 -- 172.21.15.195:0/3104250214 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe570011630 con 0x7fe58006f640 2022-06-27T11:53:00.815 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:00.803+0000 7fe585526700 1 -- 172.21.15.195:0/3104250214 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe570011790 con 0x7fe58006f640 2022-06-27T11:53:00.815 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:00.803+0000 7fe585526700 1 -- 172.21.15.195:0/3104250214 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe570011b30 con 0x7fe58006f640 2022-06-27T11:53:00.816 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:00.803+0000 7fe58878c700 1 -- 172.21.15.195:0/3104250214 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe58006f640 msgr2=0x7fe580156ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:53:00.816 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:00.803+0000 7fe58878c700 1 --2- 172.21.15.195:0/3104250214 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe58006f640 0x7fe580156ed0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fe58006dd70 tx=0x7fe570009120 comp rx=0 tx=0).stop 2022-06-27T11:53:00.817 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:00.803+0000 7fe58878c700 1 -- 172.21.15.195:0/3104250214 shutdown_connections 2022-06-27T11:53:00.817 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:00.803+0000 7fe58878c700 1 --2- 172.21.15.195:0/3104250214 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe5800c39e0 0x7fe5801535a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:53:00.817 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:00.803+0000 7fe58878c700 1 --2- 172.21.15.195:0/3104250214 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe58006f640 0x7fe580156ed0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:53:00.818 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:00.803+0000 7fe58878c700 1 --2- 172.21.15.195:0/3104250214 >> v2:172.21.15.202:3300/0 conn(0x7fe5800c4160 0x7fe5800c1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:53:00.818 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:00.803+0000 7fe58878c700 1 -- 172.21.15.195:0/3104250214 >> 172.21.15.195:0/3104250214 conn(0x7fe5800c6350 msgr2=0x7fe5800c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:53:00.819 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:00.803+0000 7fe58878c700 1 -- 172.21.15.195:0/3104250214 shutdown_connections 2022-06-27T11:53:00.819 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:00.803+0000 7fe58878c700 1 -- 172.21.15.195:0/3104250214 wait complete. 2022-06-27T11:53:00.819 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:00.805+0000 7fe58878c700 1 Processor -- start 2022-06-27T11:53:00.820 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:00.805+0000 7fe58878c700 1 -- start start 2022-06-27T11:53:00.820 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:00.805+0000 7fe58878c700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe58006f640 0x7fe58016a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:53:00.821 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:00.805+0000 7fe58878c700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe5800c39e0 0x7fe58016aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:53:00.822 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:00.805+0000 7fe58878c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe5800c4160 0x7fe58006bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:53:00.822 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:00.805+0000 7fe58878c700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5800c55d0 con 0x7fe5800c4160 2022-06-27T11:53:00.823 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:00.805+0000 7fe58878c700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5800c5330 con 0x7fe5800c39e0 2022-06-27T11:53:00.823 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:00.805+0000 7fe58878c700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5800bce80 con 0x7fe58006f640 2022-06-27T11:53:00.823 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:00.805+0000 7fe586d29700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe5800c4160 0x7fe58006bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:53:00.823 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:00.805+0000 7fe586d29700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe5800c4160 0x7fe58006bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:43828/0 (socket says 172.21.15.195:43828) 2022-06-27T11:53:00.824 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:00.805+0000 7fe586d29700 1 -- 172.21.15.195:0/3987136002 learned_addr learned my addr 172.21.15.195:0/3987136002 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:53:00.824 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:00.806+0000 7fe586528700 1 --2- 172.21.15.195:0/3987136002 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe58006f640 0x7fe58016a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:53:00.824 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:00.806+0000 7fe585d27700 1 --2- 172.21.15.195:0/3987136002 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe5800c39e0 0x7fe58016aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:53:00.824 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:00.806+0000 7fe586d29700 1 -- 172.21.15.195:0/3987136002 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe58006f640 msgr2=0x7fe58016a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:53:00.825 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:00.806+0000 7fe586d29700 1 --2- 172.21.15.195:0/3987136002 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe58006f640 0x7fe58016a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:53:00.825 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:00.806+0000 7fe586d29700 1 -- 172.21.15.195:0/3987136002 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe5800c39e0 msgr2=0x7fe58016aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:53:00.825 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:00.806+0000 7fe586d29700 1 --2- 172.21.15.195:0/3987136002 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe5800c39e0 0x7fe58016aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:53:00.825 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:00.806+0000 7fe586d29700 1 -- 172.21.15.195:0/3987136002 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5700093a0 con 0x7fe5800c4160 2022-06-27T11:53:00.826 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:00.806+0000 7fe586d29700 1 --2- 172.21.15.195:0/3987136002 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe5800c4160 0x7fe58006bfe0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fe578009e90 tx=0x7fe57800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:53:00.826 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:00.806+0000 7fe56e7fc700 1 -- 172.21.15.195:0/3987136002 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe57800cc20 con 0x7fe5800c4160 2022-06-27T11:53:00.826 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:00.806+0000 7fe56e7fc700 1 -- 172.21.15.195:0/3987136002 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe57800f040 con 0x7fe5800c4160 2022-06-27T11:53:00.827 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:00.806+0000 7fe58878c700 1 -- 172.21.15.195:0/3987136002 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5800bd160 con 0x7fe5800c4160 2022-06-27T11:53:00.827 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:00.806+0000 7fe56e7fc700 1 -- 172.21.15.195:0/3987136002 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe578014480 con 0x7fe5800c4160 2022-06-27T11:53:00.827 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:00.806+0000 7fe58878c700 1 -- 172.21.15.195:0/3987136002 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe580157860 con 0x7fe5800c4160 2022-06-27T11:53:00.827 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:00.807+0000 7fe58878c700 1 -- 172.21.15.195:0/3987136002 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5800722c0 con 0x7fe5800c4160 2022-06-27T11:53:00.827 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:00.807+0000 7fe56e7fc700 1 -- 172.21.15.195:0/3987136002 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fe578014830 con 0x7fe5800c4160 2022-06-27T11:53:00.828 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:00.808+0000 7fe56e7fc700 1 --2- 172.21.15.195:0/3987136002 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe560063d00 0x7fe5600661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:53:00.828 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:00.808+0000 7fe56e7fc700 1 -- 172.21.15.195:0/3987136002 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6374+0+0 (secure 0 0 0) 0x7fe5780819b0 con 0x7fe5800c4160 2022-06-27T11:53:00.828 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:00.808+0000 7fe586528700 1 --2- 172.21.15.195:0/3987136002 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe560063d00 0x7fe5600661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:53:00.829 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:00.808+0000 7fe586528700 1 --2- 172.21.15.195:0/3987136002 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe560063d00 0x7fe5600661b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe570009160 tx=0x7fe570017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:53:00.829 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:00.812+0000 7fe56e7fc700 1 -- 172.21.15.195:0/3987136002 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fe578050400 con 0x7fe5800c4160 2022-06-27T11:53:01.040 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:01.040+0000 7fe58878c700 1 -- 172.21.15.195:0/3987136002 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe5800c1680 con 0x7fe5800c4160 2022-06-27T11:53:01.041 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:01.041+0000 7fe56e7fc700 1 -- 172.21.15.195:0/3987136002 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6128 (secure 0 0 0) 0x7fe578018090 con 0x7fe5800c4160 2022-06-27T11:53:01.041 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:53:01.042 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4856,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/2970948376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2970948376},{"type":"v1","addr":"172.21.15.202:6835","nonce":2970948376}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:52:55.042048+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4760":{"gid":4760,"name":"b","rank":0,"incarnation":20,"state":"up:active","state_seq":21,"addr":"172.21.15.202:6833/1026219246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1026219246},{"type":"v1","addr":"172.21.15.202:6833","nonce":1026219246}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:53:01.046 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:01.043+0000 7fe58878c700 1 -- 172.21.15.195:0/3987136002 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe560063d00 msgr2=0x7fe5600661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:53:01.047 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:01.043+0000 7fe58878c700 1 --2- 172.21.15.195:0/3987136002 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe560063d00 0x7fe5600661b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe570009160 tx=0x7fe570017000 comp rx=0 tx=0).stop 2022-06-27T11:53:01.047 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:01.043+0000 7fe58878c700 1 -- 172.21.15.195:0/3987136002 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe5800c4160 msgr2=0x7fe58006bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:53:01.048 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:01.043+0000 7fe58878c700 1 --2- 172.21.15.195:0/3987136002 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe5800c4160 0x7fe58006bfe0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fe578009e90 tx=0x7fe57800b040 comp rx=0 tx=0).stop 2022-06-27T11:53:01.048 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:01.043+0000 7fe58878c700 1 -- 172.21.15.195:0/3987136002 shutdown_connections 2022-06-27T11:53:01.049 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:01.043+0000 7fe58878c700 1 --2- 172.21.15.195:0/3987136002 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe560063d00 0x7fe5600661b0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:53:01.049 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:01.043+0000 7fe58878c700 1 --2- 172.21.15.195:0/3987136002 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe5800c4160 0x7fe58006bfe0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:53:01.049 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:01.043+0000 7fe58878c700 1 --2- 172.21.15.195:0/3987136002 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe58006f640 0x7fe58016a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:53:01.050 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:01.043+0000 7fe58878c700 1 --2- 172.21.15.195:0/3987136002 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe5800c39e0 0x7fe58016aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:53:01.050 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:01.043+0000 7fe58878c700 1 -- 172.21.15.195:0/3987136002 >> 172.21.15.195:0/3987136002 conn(0x7fe5800c6350 msgr2=0x7fe580154c20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:53:01.051 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:01.043+0000 7fe58878c700 1 -- 172.21.15.195:0/3987136002 shutdown_connections 2022-06-27T11:53:01.051 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:53:01.043+0000 7fe58878c700 1 -- 172.21.15.195:0/3987136002 wait complete. 2022-06-27T11:53:01.052 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 25 2022-06-27T11:53:01.055 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d reported in up:standby state 2022-06-27T11:53:01.055 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 111 secs before thrashing 2022-06-27T11:53:02.360 INFO:tasks.workunit.client.0.smithi195.stdout:removed './246/25' 2022-06-27T11:53:02.360 INFO:tasks.workunit.client.0.smithi195.stdout:removed './246/30' 2022-06-27T11:53:02.361 INFO:tasks.workunit.client.0.smithi195.stdout:removed './246/33' 2022-06-27T11:53:02.361 INFO:tasks.workunit.client.0.smithi195.stdout:removed './246/23' 2022-06-27T11:53:02.362 INFO:tasks.workunit.client.0.smithi195.stdout:removed './246/35' 2022-06-27T11:53:02.362 INFO:tasks.workunit.client.0.smithi195.stdout:removed './246/26' 2022-06-27T11:53:02.362 INFO:tasks.workunit.client.0.smithi195.stdout:removed './246/37' 2022-06-27T11:53:02.363 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './246' 2022-06-27T11:53:02.363 INFO:tasks.workunit.client.0.smithi195.stdout:removed './247/21' 2022-06-27T11:53:02.364 INFO:tasks.workunit.client.0.smithi195.stdout:removed './247/27' 2022-06-27T11:53:02.364 INFO:tasks.workunit.client.0.smithi195.stdout:removed './247/31' 2022-06-27T11:53:02.365 INFO:tasks.workunit.client.0.smithi195.stdout:removed './247/36' 2022-06-27T11:53:02.365 INFO:tasks.workunit.client.0.smithi195.stdout:removed './247/20' 2022-06-27T11:53:02.366 INFO:tasks.workunit.client.0.smithi195.stdout:removed './247/39' 2022-06-27T11:53:02.367 INFO:tasks.workunit.client.0.smithi195.stdout:removed './247/34' 2022-06-27T11:53:02.367 INFO:tasks.workunit.client.0.smithi195.stdout:removed './247/38' 2022-06-27T11:53:02.367 INFO:tasks.workunit.client.0.smithi195.stdout:removed './247/28' 2022-06-27T11:53:02.368 INFO:tasks.workunit.client.0.smithi195.stdout:removed './247/24' 2022-06-27T11:53:02.368 INFO:tasks.workunit.client.0.smithi195.stdout:removed './247/29' 2022-06-27T11:53:02.368 INFO:tasks.workunit.client.0.smithi195.stdout:removed './247/32' 2022-06-27T11:53:02.368 INFO:tasks.workunit.client.0.smithi195.stdout:removed './247/22' 2022-06-27T11:53:02.369 INFO:tasks.workunit.client.0.smithi195.stdout:removed './247/25' 2022-06-27T11:53:02.369 INFO:tasks.workunit.client.0.smithi195.stdout:removed './247/30' 2022-06-27T11:53:02.369 INFO:tasks.workunit.client.0.smithi195.stdout:removed './247/33' 2022-06-27T11:53:02.369 INFO:tasks.workunit.client.0.smithi195.stdout:removed './247/23' 2022-06-27T11:53:02.370 INFO:tasks.workunit.client.0.smithi195.stdout:removed './247/35' 2022-06-27T11:53:02.370 INFO:tasks.workunit.client.0.smithi195.stdout:removed './247/26' 2022-06-27T11:53:02.370 INFO:tasks.workunit.client.0.smithi195.stdout:removed './247/37' 2022-06-27T11:53:02.370 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './247' 2022-06-27T11:53:02.370 INFO:tasks.workunit.client.0.smithi195.stdout:removed './248/21' 2022-06-27T11:53:02.371 INFO:tasks.workunit.client.0.smithi195.stdout:removed './248/27' 2022-06-27T11:53:02.371 INFO:tasks.workunit.client.0.smithi195.stdout:removed './248/31' 2022-06-27T11:53:02.371 INFO:tasks.workunit.client.0.smithi195.stdout:removed './248/36' 2022-06-27T11:53:02.371 INFO:tasks.workunit.client.0.smithi195.stdout:removed './248/20' 2022-06-27T11:53:02.372 INFO:tasks.workunit.client.0.smithi195.stdout:removed './248/39' 2022-06-27T11:53:02.372 INFO:tasks.workunit.client.0.smithi195.stdout:removed './248/34' 2022-06-27T11:53:02.372 INFO:tasks.workunit.client.0.smithi195.stdout:removed './248/38' 2022-06-27T11:53:02.372 INFO:tasks.workunit.client.0.smithi195.stdout:removed './248/28' 2022-06-27T11:53:02.372 INFO:tasks.workunit.client.0.smithi195.stdout:removed './248/24' 2022-06-27T11:53:02.373 INFO:tasks.workunit.client.0.smithi195.stdout:removed './248/29' 2022-06-27T11:53:02.373 INFO:tasks.workunit.client.0.smithi195.stdout:removed './248/32' 2022-06-27T11:53:02.373 INFO:tasks.workunit.client.0.smithi195.stdout:removed './248/22' 2022-06-27T11:53:02.373 INFO:tasks.workunit.client.0.smithi195.stdout:removed './248/25' 2022-06-27T11:53:02.373 INFO:tasks.workunit.client.0.smithi195.stdout:removed './248/30' 2022-06-27T11:53:02.374 INFO:tasks.workunit.client.0.smithi195.stdout:removed './248/33' 2022-06-27T11:53:02.374 INFO:tasks.workunit.client.0.smithi195.stdout:removed './248/23' 2022-06-27T11:53:02.374 INFO:tasks.workunit.client.0.smithi195.stdout:removed './248/35' 2022-06-27T11:53:02.374 INFO:tasks.workunit.client.0.smithi195.stdout:removed './248/26' 2022-06-27T11:53:02.375 INFO:tasks.workunit.client.0.smithi195.stdout:removed './248/37' 2022-06-27T11:53:02.375 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './248' 2022-06-27T11:53:02.375 INFO:tasks.workunit.client.0.smithi195.stdout:removed './249/21' 2022-06-27T11:53:02.375 INFO:tasks.workunit.client.0.smithi195.stdout:removed './249/27' 2022-06-27T11:53:02.375 INFO:tasks.workunit.client.0.smithi195.stdout:removed './249/31' 2022-06-27T11:53:02.376 INFO:tasks.workunit.client.0.smithi195.stdout:removed './249/36' 2022-06-27T11:53:02.376 INFO:tasks.workunit.client.0.smithi195.stdout:removed './249/20' 2022-06-27T11:53:02.376 INFO:tasks.workunit.client.0.smithi195.stdout:removed './249/39' 2022-06-27T11:53:02.376 INFO:tasks.workunit.client.0.smithi195.stdout:removed './249/34' 2022-06-27T11:53:02.377 INFO:tasks.workunit.client.0.smithi195.stdout:removed './249/38' 2022-06-27T11:53:02.377 INFO:tasks.workunit.client.0.smithi195.stdout:removed './249/28' 2022-06-27T11:53:02.377 INFO:tasks.workunit.client.0.smithi195.stdout:removed './249/24' 2022-06-27T11:53:02.377 INFO:tasks.workunit.client.0.smithi195.stdout:removed './249/29' 2022-06-27T11:53:02.377 INFO:tasks.workunit.client.0.smithi195.stdout:removed './249/32' 2022-06-27T11:53:02.378 INFO:tasks.workunit.client.0.smithi195.stdout:removed './249/22' 2022-06-27T11:53:02.378 INFO:tasks.workunit.client.0.smithi195.stdout:removed './249/25' 2022-06-27T11:53:02.378 INFO:tasks.workunit.client.0.smithi195.stdout:removed './249/30' 2022-06-27T11:53:02.378 INFO:tasks.workunit.client.0.smithi195.stdout:removed './249/33' 2022-06-27T11:53:02.379 INFO:tasks.workunit.client.0.smithi195.stdout:removed './249/23' 2022-06-27T11:53:02.379 INFO:tasks.workunit.client.0.smithi195.stdout:removed './249/35' 2022-06-27T11:53:02.379 INFO:tasks.workunit.client.0.smithi195.stdout:removed './249/26' 2022-06-27T11:53:02.379 INFO:tasks.workunit.client.0.smithi195.stdout:removed './249/37' 2022-06-27T11:53:02.379 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './249' 2022-06-27T11:53:02.380 INFO:tasks.workunit.client.0.smithi195.stdout:removed './250/21' 2022-06-27T11:53:02.380 INFO:tasks.workunit.client.0.smithi195.stdout:removed './250/27' 2022-06-27T11:53:02.380 INFO:tasks.workunit.client.0.smithi195.stdout:removed './250/31' 2022-06-27T11:53:02.380 INFO:tasks.workunit.client.0.smithi195.stdout:removed './250/36' 2022-06-27T11:53:02.380 INFO:tasks.workunit.client.0.smithi195.stdout:removed './250/20' 2022-06-27T11:53:02.381 INFO:tasks.workunit.client.0.smithi195.stdout:removed './250/39' 2022-06-27T11:53:02.381 INFO:tasks.workunit.client.0.smithi195.stdout:removed './250/34' 2022-06-27T11:53:02.381 INFO:tasks.workunit.client.0.smithi195.stdout:removed './250/38' 2022-06-27T11:53:02.381 INFO:tasks.workunit.client.0.smithi195.stdout:removed './250/28' 2022-06-27T11:53:02.382 INFO:tasks.workunit.client.0.smithi195.stdout:removed './250/24' 2022-06-27T11:53:02.382 INFO:tasks.workunit.client.0.smithi195.stdout:removed './250/29' 2022-06-27T11:53:02.382 INFO:tasks.workunit.client.0.smithi195.stdout:removed './250/32' 2022-06-27T11:53:02.382 INFO:tasks.workunit.client.0.smithi195.stdout:removed './250/22' 2022-06-27T11:53:02.382 INFO:tasks.workunit.client.0.smithi195.stdout:removed './250/25' 2022-06-27T11:53:02.383 INFO:tasks.workunit.client.0.smithi195.stdout:removed './250/30' 2022-06-27T11:53:02.383 INFO:tasks.workunit.client.0.smithi195.stdout:removed './250/33' 2022-06-27T11:53:02.383 INFO:tasks.workunit.client.0.smithi195.stdout:removed './250/23' 2022-06-27T11:53:02.383 INFO:tasks.workunit.client.0.smithi195.stdout:removed './250/35' 2022-06-27T11:53:02.383 INFO:tasks.workunit.client.0.smithi195.stdout:removed './250/26' 2022-06-27T11:53:02.384 INFO:tasks.workunit.client.0.smithi195.stdout:removed './250/37' 2022-06-27T11:53:02.384 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './250' 2022-06-27T11:53:02.384 INFO:tasks.workunit.client.0.smithi195.stdout:removed './251/21' 2022-06-27T11:53:02.384 INFO:tasks.workunit.client.0.smithi195.stdout:removed './251/27' 2022-06-27T11:53:02.385 INFO:tasks.workunit.client.0.smithi195.stdout:removed './251/31' 2022-06-27T11:53:02.385 INFO:tasks.workunit.client.0.smithi195.stdout:removed './251/36' 2022-06-27T11:53:02.385 INFO:tasks.workunit.client.0.smithi195.stdout:removed './251/20' 2022-06-27T11:53:02.386 INFO:tasks.workunit.client.0.smithi195.stdout:removed './251/39' 2022-06-27T11:53:02.386 INFO:tasks.workunit.client.0.smithi195.stdout:removed './251/34' 2022-06-27T11:53:02.386 INFO:tasks.workunit.client.0.smithi195.stdout:removed './251/38' 2022-06-27T11:53:02.386 INFO:tasks.workunit.client.0.smithi195.stdout:removed './251/28' 2022-06-27T11:53:02.387 INFO:tasks.workunit.client.0.smithi195.stdout:removed './251/24' 2022-06-27T11:53:02.387 INFO:tasks.workunit.client.0.smithi195.stdout:removed './251/29' 2022-06-27T11:53:02.387 INFO:tasks.workunit.client.0.smithi195.stdout:removed './251/32' 2022-06-27T11:53:02.387 INFO:tasks.workunit.client.0.smithi195.stdout:removed './251/22' 2022-06-27T11:53:02.387 INFO:tasks.workunit.client.0.smithi195.stdout:removed './251/25' 2022-06-27T11:53:02.388 INFO:tasks.workunit.client.0.smithi195.stdout:removed './251/30' 2022-06-27T11:53:02.388 INFO:tasks.workunit.client.0.smithi195.stdout:removed './251/33' 2022-06-27T11:53:02.388 INFO:tasks.workunit.client.0.smithi195.stdout:removed './251/23' 2022-06-27T11:53:02.389 INFO:tasks.workunit.client.0.smithi195.stdout:removed './251/35' 2022-06-27T11:53:02.389 INFO:tasks.workunit.client.0.smithi195.stdout:removed './251/26' 2022-06-27T11:53:02.389 INFO:tasks.workunit.client.0.smithi195.stdout:removed './251/37' 2022-06-27T11:53:02.389 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './251' 2022-06-27T11:53:02.389 INFO:tasks.workunit.client.0.smithi195.stdout:removed './252/21' 2022-06-27T11:53:02.390 INFO:tasks.workunit.client.0.smithi195.stdout:removed './252/27' 2022-06-27T11:53:02.390 INFO:tasks.workunit.client.0.smithi195.stdout:removed './252/31' 2022-06-27T11:53:02.390 INFO:tasks.workunit.client.0.smithi195.stdout:removed './252/36' 2022-06-27T11:53:02.390 INFO:tasks.workunit.client.0.smithi195.stdout:removed './252/20' 2022-06-27T11:53:02.391 INFO:tasks.workunit.client.0.smithi195.stdout:removed './252/39' 2022-06-27T11:53:02.391 INFO:tasks.workunit.client.0.smithi195.stdout:removed './252/34' 2022-06-27T11:53:02.391 INFO:tasks.workunit.client.0.smithi195.stdout:removed './252/38' 2022-06-27T11:53:02.391 INFO:tasks.workunit.client.0.smithi195.stdout:removed './252/28' 2022-06-27T11:53:02.392 INFO:tasks.workunit.client.0.smithi195.stdout:removed './252/24' 2022-06-27T11:53:02.392 INFO:tasks.workunit.client.0.smithi195.stdout:removed './252/29' 2022-06-27T11:53:02.393 INFO:tasks.workunit.client.0.smithi195.stdout:removed './252/32' 2022-06-27T11:53:02.393 INFO:tasks.workunit.client.0.smithi195.stdout:removed './252/22' 2022-06-27T11:53:02.393 INFO:tasks.workunit.client.0.smithi195.stdout:removed './252/25' 2022-06-27T11:53:02.393 INFO:tasks.workunit.client.0.smithi195.stdout:removed './252/30' 2022-06-27T11:53:02.394 INFO:tasks.workunit.client.0.smithi195.stdout:removed './252/33' 2022-06-27T11:53:02.394 INFO:tasks.workunit.client.0.smithi195.stdout:removed './252/23' 2022-06-27T11:53:02.394 INFO:tasks.workunit.client.0.smithi195.stdout:removed './252/35' 2022-06-27T11:53:02.394 INFO:tasks.workunit.client.0.smithi195.stdout:removed './252/26' 2022-06-27T11:53:02.394 INFO:tasks.workunit.client.0.smithi195.stdout:removed './252/37' 2022-06-27T11:53:02.395 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './252' 2022-06-27T11:53:02.395 INFO:tasks.workunit.client.0.smithi195.stdout:removed './253/21' 2022-06-27T11:53:02.395 INFO:tasks.workunit.client.0.smithi195.stdout:removed './253/27' 2022-06-27T11:53:02.395 INFO:tasks.workunit.client.0.smithi195.stdout:removed './253/31' 2022-06-27T11:53:02.396 INFO:tasks.workunit.client.0.smithi195.stdout:removed './253/36' 2022-06-27T11:53:02.396 INFO:tasks.workunit.client.0.smithi195.stdout:removed './253/20' 2022-06-27T11:53:02.396 INFO:tasks.workunit.client.0.smithi195.stdout:removed './253/39' 2022-06-27T11:53:02.396 INFO:tasks.workunit.client.0.smithi195.stdout:removed './253/34' 2022-06-27T11:53:02.396 INFO:tasks.workunit.client.0.smithi195.stdout:removed './253/38' 2022-06-27T11:53:02.397 INFO:tasks.workunit.client.0.smithi195.stdout:removed './253/28' 2022-06-27T11:53:02.397 INFO:tasks.workunit.client.0.smithi195.stdout:removed './253/24' 2022-06-27T11:53:02.397 INFO:tasks.workunit.client.0.smithi195.stdout:removed './253/29' 2022-06-27T11:53:02.397 INFO:tasks.workunit.client.0.smithi195.stdout:removed './253/32' 2022-06-27T11:53:02.397 INFO:tasks.workunit.client.0.smithi195.stdout:removed './253/22' 2022-06-27T11:53:02.398 INFO:tasks.workunit.client.0.smithi195.stdout:removed './253/25' 2022-06-27T11:53:02.398 INFO:tasks.workunit.client.0.smithi195.stdout:removed './253/30' 2022-06-27T11:53:02.398 INFO:tasks.workunit.client.0.smithi195.stdout:removed './253/33' 2022-06-27T11:53:02.399 INFO:tasks.workunit.client.0.smithi195.stdout:removed './253/23' 2022-06-27T11:53:02.399 INFO:tasks.workunit.client.0.smithi195.stdout:removed './253/35' 2022-06-27T11:53:02.399 INFO:tasks.workunit.client.0.smithi195.stdout:removed './253/26' 2022-06-27T11:53:02.399 INFO:tasks.workunit.client.0.smithi195.stdout:removed './253/37' 2022-06-27T11:53:02.399 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './253' 2022-06-27T11:53:02.400 INFO:tasks.workunit.client.0.smithi195.stdout:removed './254/21' 2022-06-27T11:53:02.400 INFO:tasks.workunit.client.0.smithi195.stdout:removed './254/27' 2022-06-27T11:53:02.400 INFO:tasks.workunit.client.0.smithi195.stdout:removed './254/31' 2022-06-27T11:53:02.400 INFO:tasks.workunit.client.0.smithi195.stdout:removed './254/36' 2022-06-27T11:53:02.401 INFO:tasks.workunit.client.0.smithi195.stdout:removed './254/20' 2022-06-27T11:53:02.401 INFO:tasks.workunit.client.0.smithi195.stdout:removed './254/39' 2022-06-27T11:53:02.401 INFO:tasks.workunit.client.0.smithi195.stdout:removed './254/34' 2022-06-27T11:53:02.401 INFO:tasks.workunit.client.0.smithi195.stdout:removed './254/38' 2022-06-27T11:53:02.402 INFO:tasks.workunit.client.0.smithi195.stdout:removed './254/28' 2022-06-27T11:53:02.402 INFO:tasks.workunit.client.0.smithi195.stdout:removed './254/24' 2022-06-27T11:53:02.402 INFO:tasks.workunit.client.0.smithi195.stdout:removed './254/29' 2022-06-27T11:53:02.403 INFO:tasks.workunit.client.0.smithi195.stdout:removed './254/32' 2022-06-27T11:53:02.403 INFO:tasks.workunit.client.0.smithi195.stdout:removed './254/22' 2022-06-27T11:53:02.403 INFO:tasks.workunit.client.0.smithi195.stdout:removed './254/25' 2022-06-27T11:53:02.404 INFO:tasks.workunit.client.0.smithi195.stdout:removed './254/30' 2022-06-27T11:53:02.404 INFO:tasks.workunit.client.0.smithi195.stdout:removed './254/33' 2022-06-27T11:53:02.404 INFO:tasks.workunit.client.0.smithi195.stdout:removed './254/23' 2022-06-27T11:53:02.405 INFO:tasks.workunit.client.0.smithi195.stdout:removed './254/35' 2022-06-27T11:53:02.405 INFO:tasks.workunit.client.0.smithi195.stdout:removed './254/26' 2022-06-27T11:53:02.405 INFO:tasks.workunit.client.0.smithi195.stdout:removed './254/37' 2022-06-27T11:53:02.406 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './254' 2022-06-27T11:53:02.406 INFO:tasks.workunit.client.0.smithi195.stdout:removed './255/21' 2022-06-27T11:53:02.407 INFO:tasks.workunit.client.0.smithi195.stdout:removed './255/27' 2022-06-27T11:53:02.407 INFO:tasks.workunit.client.0.smithi195.stdout:removed './255/31' 2022-06-27T11:53:02.408 INFO:tasks.workunit.client.0.smithi195.stdout:removed './255/36' 2022-06-27T11:53:02.408 INFO:tasks.workunit.client.0.smithi195.stdout:removed './255/20' 2022-06-27T11:53:02.409 INFO:tasks.workunit.client.0.smithi195.stdout:removed './255/39' 2022-06-27T11:53:02.409 INFO:tasks.workunit.client.0.smithi195.stdout:removed './255/34' 2022-06-27T11:53:02.409 INFO:tasks.workunit.client.0.smithi195.stdout:removed './255/38' 2022-06-27T11:53:02.409 INFO:tasks.workunit.client.0.smithi195.stdout:removed './255/28' 2022-06-27T11:53:02.409 INFO:tasks.workunit.client.0.smithi195.stdout:removed './255/24' 2022-06-27T11:53:02.410 INFO:tasks.workunit.client.0.smithi195.stdout:removed './255/29' 2022-06-27T11:53:02.410 INFO:tasks.workunit.client.0.smithi195.stdout:removed './255/32' 2022-06-27T11:53:02.410 INFO:tasks.workunit.client.0.smithi195.stdout:removed './255/22' 2022-06-27T11:53:02.410 INFO:tasks.workunit.client.0.smithi195.stdout:removed './255/25' 2022-06-27T11:53:02.411 INFO:tasks.workunit.client.0.smithi195.stdout:removed './255/30' 2022-06-27T11:53:02.411 INFO:tasks.workunit.client.0.smithi195.stdout:removed './255/33' 2022-06-27T11:53:02.411 INFO:tasks.workunit.client.0.smithi195.stdout:removed './255/23' 2022-06-27T11:53:02.411 INFO:tasks.workunit.client.0.smithi195.stdout:removed './255/35' 2022-06-27T11:53:02.411 INFO:tasks.workunit.client.0.smithi195.stdout:removed './255/26' 2022-06-27T11:53:02.412 INFO:tasks.workunit.client.0.smithi195.stdout:removed './255/37' 2022-06-27T11:53:02.412 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './255' 2022-06-27T11:53:02.412 INFO:tasks.workunit.client.0.smithi195.stdout:removed './256/21' 2022-06-27T11:53:02.412 INFO:tasks.workunit.client.0.smithi195.stdout:removed './256/27' 2022-06-27T11:53:02.413 INFO:tasks.workunit.client.0.smithi195.stdout:removed './256/31' 2022-06-27T11:53:02.413 INFO:tasks.workunit.client.0.smithi195.stdout:removed './256/36' 2022-06-27T11:53:02.413 INFO:tasks.workunit.client.0.smithi195.stdout:removed './256/20' 2022-06-27T11:53:02.413 INFO:tasks.workunit.client.0.smithi195.stdout:removed './256/39' 2022-06-27T11:53:02.413 INFO:tasks.workunit.client.0.smithi195.stdout:removed './256/34' 2022-06-27T11:53:02.414 INFO:tasks.workunit.client.0.smithi195.stdout:removed './256/38' 2022-06-27T11:53:02.414 INFO:tasks.workunit.client.0.smithi195.stdout:removed './256/28' 2022-06-27T11:53:02.414 INFO:tasks.workunit.client.0.smithi195.stdout:removed './256/24' 2022-06-27T11:53:02.414 INFO:tasks.workunit.client.0.smithi195.stdout:removed './256/29' 2022-06-27T11:53:02.415 INFO:tasks.workunit.client.0.smithi195.stdout:removed './256/32' 2022-06-27T11:53:02.415 INFO:tasks.workunit.client.0.smithi195.stdout:removed './256/22' 2022-06-27T11:53:02.415 INFO:tasks.workunit.client.0.smithi195.stdout:removed './256/25' 2022-06-27T11:53:02.415 INFO:tasks.workunit.client.0.smithi195.stdout:removed './256/30' 2022-06-27T11:53:03.044 INFO:tasks.ceph.mds.b.smithi202.stderr:2022-06-27T11:53:03.044+0000 7f6f58146700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2022-06-27T11:53:05.577 INFO:tasks.workunit.client.0.smithi195.stdout:removed './256/33' 2022-06-27T11:53:05.577 INFO:tasks.workunit.client.0.smithi195.stdout:removed './256/23' 2022-06-27T11:53:05.578 INFO:tasks.workunit.client.0.smithi195.stdout:removed './256/35' 2022-06-27T11:53:05.578 INFO:tasks.workunit.client.0.smithi195.stdout:removed './256/26' 2022-06-27T11:53:05.578 INFO:tasks.workunit.client.0.smithi195.stdout:removed './256/37' 2022-06-27T11:53:05.579 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './256' 2022-06-27T11:53:05.579 INFO:tasks.workunit.client.0.smithi195.stdout:removed './257/21' 2022-06-27T11:53:05.579 INFO:tasks.workunit.client.0.smithi195.stdout:removed './257/27' 2022-06-27T11:53:05.579 INFO:tasks.workunit.client.0.smithi195.stdout:removed './257/31' 2022-06-27T11:53:05.579 INFO:tasks.workunit.client.0.smithi195.stdout:removed './257/36' 2022-06-27T11:53:05.580 INFO:tasks.workunit.client.0.smithi195.stdout:removed './257/20' 2022-06-27T11:53:05.580 INFO:tasks.workunit.client.0.smithi195.stdout:removed './257/39' 2022-06-27T11:53:05.580 INFO:tasks.workunit.client.0.smithi195.stdout:removed './257/34' 2022-06-27T11:53:05.580 INFO:tasks.workunit.client.0.smithi195.stdout:removed './257/38' 2022-06-27T11:53:05.581 INFO:tasks.workunit.client.0.smithi195.stdout:removed './257/28' 2022-06-27T11:53:05.581 INFO:tasks.workunit.client.0.smithi195.stdout:removed './257/24' 2022-06-27T11:53:05.581 INFO:tasks.workunit.client.0.smithi195.stdout:removed './257/29' 2022-06-27T11:53:05.581 INFO:tasks.workunit.client.0.smithi195.stdout:removed './257/32' 2022-06-27T11:53:05.582 INFO:tasks.workunit.client.0.smithi195.stdout:removed './257/22' 2022-06-27T11:53:05.582 INFO:tasks.workunit.client.0.smithi195.stdout:removed './257/25' 2022-06-27T11:53:05.582 INFO:tasks.workunit.client.0.smithi195.stdout:removed './257/30' 2022-06-27T11:53:05.582 INFO:tasks.workunit.client.0.smithi195.stdout:removed './257/33' 2022-06-27T11:53:05.585 INFO:tasks.workunit.client.0.smithi195.stdout:removed './257/23' 2022-06-27T11:53:05.585 INFO:tasks.workunit.client.0.smithi195.stdout:removed './257/35' 2022-06-27T11:53:05.586 INFO:tasks.workunit.client.0.smithi195.stdout:removed './257/26' 2022-06-27T11:53:05.586 INFO:tasks.workunit.client.0.smithi195.stdout:removed './257/37' 2022-06-27T11:53:05.587 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './257' 2022-06-27T11:53:05.587 INFO:tasks.workunit.client.0.smithi195.stdout:removed './258/21' 2022-06-27T11:53:05.588 INFO:tasks.workunit.client.0.smithi195.stdout:removed './258/27' 2022-06-27T11:53:05.588 INFO:tasks.workunit.client.0.smithi195.stdout:removed './258/31' 2022-06-27T11:53:05.588 INFO:tasks.workunit.client.0.smithi195.stdout:removed './258/36' 2022-06-27T11:53:05.589 INFO:tasks.workunit.client.0.smithi195.stdout:removed './258/20' 2022-06-27T11:53:05.589 INFO:tasks.workunit.client.0.smithi195.stdout:removed './258/39' 2022-06-27T11:53:05.589 INFO:tasks.workunit.client.0.smithi195.stdout:removed './258/34' 2022-06-27T11:53:05.590 INFO:tasks.workunit.client.0.smithi195.stdout:removed './258/38' 2022-06-27T11:53:05.590 INFO:tasks.workunit.client.0.smithi195.stdout:removed './258/28' 2022-06-27T11:53:05.591 INFO:tasks.workunit.client.0.smithi195.stdout:removed './258/24' 2022-06-27T11:53:05.591 INFO:tasks.workunit.client.0.smithi195.stdout:removed './258/29' 2022-06-27T11:53:05.592 INFO:tasks.workunit.client.0.smithi195.stdout:removed './258/32' 2022-06-27T11:53:05.592 INFO:tasks.workunit.client.0.smithi195.stdout:removed './258/22' 2022-06-27T11:53:05.592 INFO:tasks.workunit.client.0.smithi195.stdout:removed './258/25' 2022-06-27T11:53:05.593 INFO:tasks.workunit.client.0.smithi195.stdout:removed './258/30' 2022-06-27T11:53:05.593 INFO:tasks.workunit.client.0.smithi195.stdout:removed './258/33' 2022-06-27T11:53:05.593 INFO:tasks.workunit.client.0.smithi195.stdout:removed './258/23' 2022-06-27T11:53:05.594 INFO:tasks.workunit.client.0.smithi195.stdout:removed './258/35' 2022-06-27T11:53:05.594 INFO:tasks.workunit.client.0.smithi195.stdout:removed './258/26' 2022-06-27T11:53:05.594 INFO:tasks.workunit.client.0.smithi195.stdout:removed './258/37' 2022-06-27T11:53:05.594 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './258' 2022-06-27T11:53:05.595 INFO:tasks.workunit.client.0.smithi195.stdout:removed './259/21' 2022-06-27T11:53:05.595 INFO:tasks.workunit.client.0.smithi195.stdout:removed './259/27' 2022-06-27T11:53:05.595 INFO:tasks.workunit.client.0.smithi195.stdout:removed './259/31' 2022-06-27T11:53:05.595 INFO:tasks.workunit.client.0.smithi195.stdout:removed './259/36' 2022-06-27T11:53:05.596 INFO:tasks.workunit.client.0.smithi195.stdout:removed './259/20' 2022-06-27T11:53:05.596 INFO:tasks.workunit.client.0.smithi195.stdout:removed './259/39' 2022-06-27T11:53:05.596 INFO:tasks.workunit.client.0.smithi195.stdout:removed './259/34' 2022-06-27T11:53:05.596 INFO:tasks.workunit.client.0.smithi195.stdout:removed './259/38' 2022-06-27T11:53:05.597 INFO:tasks.workunit.client.0.smithi195.stdout:removed './259/28' 2022-06-27T11:53:05.597 INFO:tasks.workunit.client.0.smithi195.stdout:removed './259/24' 2022-06-27T11:53:05.597 INFO:tasks.workunit.client.0.smithi195.stdout:removed './259/29' 2022-06-27T11:53:05.597 INFO:tasks.workunit.client.0.smithi195.stdout:removed './259/32' 2022-06-27T11:53:05.598 INFO:tasks.workunit.client.0.smithi195.stdout:removed './259/22' 2022-06-27T11:53:05.598 INFO:tasks.workunit.client.0.smithi195.stdout:removed './259/25' 2022-06-27T11:53:05.598 INFO:tasks.workunit.client.0.smithi195.stdout:removed './259/30' 2022-06-27T11:53:05.598 INFO:tasks.workunit.client.0.smithi195.stdout:removed './259/33' 2022-06-27T11:53:05.599 INFO:tasks.workunit.client.0.smithi195.stdout:removed './259/23' 2022-06-27T11:53:05.599 INFO:tasks.workunit.client.0.smithi195.stdout:removed './259/35' 2022-06-27T11:53:05.599 INFO:tasks.workunit.client.0.smithi195.stdout:removed './259/26' 2022-06-27T11:53:05.599 INFO:tasks.workunit.client.0.smithi195.stdout:removed './259/37' 2022-06-27T11:53:05.600 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './259' 2022-06-27T11:53:05.600 INFO:tasks.workunit.client.0.smithi195.stdout:removed './260/21' 2022-06-27T11:53:05.600 INFO:tasks.workunit.client.0.smithi195.stdout:removed './260/27' 2022-06-27T11:53:05.600 INFO:tasks.workunit.client.0.smithi195.stdout:removed './260/31' 2022-06-27T11:53:05.600 INFO:tasks.workunit.client.0.smithi195.stdout:removed './260/36' 2022-06-27T11:53:05.601 INFO:tasks.workunit.client.0.smithi195.stdout:removed './260/20' 2022-06-27T11:53:05.601 INFO:tasks.workunit.client.0.smithi195.stdout:removed './260/39' 2022-06-27T11:53:05.601 INFO:tasks.workunit.client.0.smithi195.stdout:removed './260/34' 2022-06-27T11:53:05.602 INFO:tasks.workunit.client.0.smithi195.stdout:removed './260/38' 2022-06-27T11:53:05.602 INFO:tasks.workunit.client.0.smithi195.stdout:removed './260/28' 2022-06-27T11:53:05.602 INFO:tasks.workunit.client.0.smithi195.stdout:removed './260/24' 2022-06-27T11:53:05.602 INFO:tasks.workunit.client.0.smithi195.stdout:removed './260/29' 2022-06-27T11:53:05.603 INFO:tasks.workunit.client.0.smithi195.stdout:removed './260/32' 2022-06-27T11:53:05.603 INFO:tasks.workunit.client.0.smithi195.stdout:removed './260/22' 2022-06-27T11:53:05.603 INFO:tasks.workunit.client.0.smithi195.stdout:removed './260/25' 2022-06-27T11:53:05.603 INFO:tasks.workunit.client.0.smithi195.stdout:removed './260/30' 2022-06-27T11:53:05.603 INFO:tasks.workunit.client.0.smithi195.stdout:removed './260/33' 2022-06-27T11:53:05.604 INFO:tasks.workunit.client.0.smithi195.stdout:removed './260/23' 2022-06-27T11:53:05.604 INFO:tasks.workunit.client.0.smithi195.stdout:removed './260/35' 2022-06-27T11:53:05.604 INFO:tasks.workunit.client.0.smithi195.stdout:removed './260/26' 2022-06-27T11:53:05.604 INFO:tasks.workunit.client.0.smithi195.stdout:removed './260/37' 2022-06-27T11:53:05.605 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './260' 2022-06-27T11:53:05.605 INFO:tasks.workunit.client.0.smithi195.stdout:removed './261/21' 2022-06-27T11:53:05.605 INFO:tasks.workunit.client.0.smithi195.stdout:removed './261/27' 2022-06-27T11:53:05.605 INFO:tasks.workunit.client.0.smithi195.stdout:removed './261/31' 2022-06-27T11:53:05.606 INFO:tasks.workunit.client.0.smithi195.stdout:removed './261/36' 2022-06-27T11:53:05.606 INFO:tasks.workunit.client.0.smithi195.stdout:removed './261/20' 2022-06-27T11:53:05.606 INFO:tasks.workunit.client.0.smithi195.stdout:removed './261/39' 2022-06-27T11:53:05.606 INFO:tasks.workunit.client.0.smithi195.stdout:removed './261/34' 2022-06-27T11:53:05.607 INFO:tasks.workunit.client.0.smithi195.stdout:removed './261/38' 2022-06-27T11:53:05.607 INFO:tasks.workunit.client.0.smithi195.stdout:removed './261/28' 2022-06-27T11:53:05.607 INFO:tasks.workunit.client.0.smithi195.stdout:removed './261/24' 2022-06-27T11:53:05.607 INFO:tasks.workunit.client.0.smithi195.stdout:removed './261/29' 2022-06-27T11:53:05.608 INFO:tasks.workunit.client.0.smithi195.stdout:removed './261/32' 2022-06-27T11:53:05.608 INFO:tasks.workunit.client.0.smithi195.stdout:removed './261/22' 2022-06-27T11:53:05.608 INFO:tasks.workunit.client.0.smithi195.stdout:removed './261/25' 2022-06-27T11:53:05.608 INFO:tasks.workunit.client.0.smithi195.stdout:removed './261/30' 2022-06-27T11:53:05.609 INFO:tasks.workunit.client.0.smithi195.stdout:removed './261/33' 2022-06-27T11:53:05.609 INFO:tasks.workunit.client.0.smithi195.stdout:removed './261/23' 2022-06-27T11:53:05.609 INFO:tasks.workunit.client.0.smithi195.stdout:removed './261/35' 2022-06-27T11:53:05.609 INFO:tasks.workunit.client.0.smithi195.stdout:removed './261/26' 2022-06-27T11:53:05.610 INFO:tasks.workunit.client.0.smithi195.stdout:removed './261/37' 2022-06-27T11:53:05.610 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './261' 2022-06-27T11:53:05.610 INFO:tasks.workunit.client.0.smithi195.stdout:removed './262/21' 2022-06-27T11:53:05.610 INFO:tasks.workunit.client.0.smithi195.stdout:removed './262/27' 2022-06-27T11:53:05.611 INFO:tasks.workunit.client.0.smithi195.stdout:removed './262/31' 2022-06-27T11:53:05.611 INFO:tasks.workunit.client.0.smithi195.stdout:removed './262/36' 2022-06-27T11:53:05.611 INFO:tasks.workunit.client.0.smithi195.stdout:removed './262/20' 2022-06-27T11:53:05.611 INFO:tasks.workunit.client.0.smithi195.stdout:removed './262/39' 2022-06-27T11:53:05.611 INFO:tasks.workunit.client.0.smithi195.stdout:removed './262/34' 2022-06-27T11:53:05.612 INFO:tasks.workunit.client.0.smithi195.stdout:removed './262/38' 2022-06-27T11:53:05.612 INFO:tasks.workunit.client.0.smithi195.stdout:removed './262/28' 2022-06-27T11:53:05.612 INFO:tasks.workunit.client.0.smithi195.stdout:removed './262/24' 2022-06-27T11:53:05.612 INFO:tasks.workunit.client.0.smithi195.stdout:removed './262/29' 2022-06-27T11:53:05.613 INFO:tasks.workunit.client.0.smithi195.stdout:removed './262/32' 2022-06-27T11:53:05.613 INFO:tasks.workunit.client.0.smithi195.stdout:removed './262/22' 2022-06-27T11:53:05.613 INFO:tasks.workunit.client.0.smithi195.stdout:removed './262/25' 2022-06-27T11:53:05.613 INFO:tasks.workunit.client.0.smithi195.stdout:removed './262/30' 2022-06-27T11:53:05.613 INFO:tasks.workunit.client.0.smithi195.stdout:removed './262/33' 2022-06-27T11:53:05.614 INFO:tasks.workunit.client.0.smithi195.stdout:removed './262/23' 2022-06-27T11:53:05.614 INFO:tasks.workunit.client.0.smithi195.stdout:removed './262/35' 2022-06-27T11:53:05.614 INFO:tasks.workunit.client.0.smithi195.stdout:removed './262/26' 2022-06-27T11:53:05.615 INFO:tasks.workunit.client.0.smithi195.stdout:removed './262/37' 2022-06-27T11:53:05.615 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './262' 2022-06-27T11:53:05.615 INFO:tasks.workunit.client.0.smithi195.stdout:removed './263/21' 2022-06-27T11:53:05.615 INFO:tasks.workunit.client.0.smithi195.stdout:removed './263/27' 2022-06-27T11:53:05.616 INFO:tasks.workunit.client.0.smithi195.stdout:removed './263/31' 2022-06-27T11:53:05.616 INFO:tasks.workunit.client.0.smithi195.stdout:removed './263/36' 2022-06-27T11:53:05.616 INFO:tasks.workunit.client.0.smithi195.stdout:removed './263/20' 2022-06-27T11:53:05.616 INFO:tasks.workunit.client.0.smithi195.stdout:removed './263/39' 2022-06-27T11:53:05.617 INFO:tasks.workunit.client.0.smithi195.stdout:removed './263/34' 2022-06-27T11:53:05.617 INFO:tasks.workunit.client.0.smithi195.stdout:removed './263/38' 2022-06-27T11:53:05.617 INFO:tasks.workunit.client.0.smithi195.stdout:removed './263/28' 2022-06-27T11:53:05.617 INFO:tasks.workunit.client.0.smithi195.stdout:removed './263/24' 2022-06-27T11:53:05.617 INFO:tasks.workunit.client.0.smithi195.stdout:removed './263/29' 2022-06-27T11:53:05.618 INFO:tasks.workunit.client.0.smithi195.stdout:removed './263/32' 2022-06-27T11:53:05.618 INFO:tasks.workunit.client.0.smithi195.stdout:removed './263/22' 2022-06-27T11:53:05.618 INFO:tasks.workunit.client.0.smithi195.stdout:removed './263/25' 2022-06-27T11:53:05.618 INFO:tasks.workunit.client.0.smithi195.stdout:removed './263/30' 2022-06-27T11:53:05.619 INFO:tasks.workunit.client.0.smithi195.stdout:removed './263/33' 2022-06-27T11:53:05.619 INFO:tasks.workunit.client.0.smithi195.stdout:removed './263/23' 2022-06-27T11:53:05.619 INFO:tasks.workunit.client.0.smithi195.stdout:removed './263/35' 2022-06-27T11:53:05.619 INFO:tasks.workunit.client.0.smithi195.stdout:removed './263/26' 2022-06-27T11:53:05.619 INFO:tasks.workunit.client.0.smithi195.stdout:removed './263/37' 2022-06-27T11:53:05.620 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './263' 2022-06-27T11:53:05.620 INFO:tasks.workunit.client.0.smithi195.stdout:removed './264/21' 2022-06-27T11:53:05.620 INFO:tasks.workunit.client.0.smithi195.stdout:removed './264/27' 2022-06-27T11:53:05.620 INFO:tasks.workunit.client.0.smithi195.stdout:removed './264/31' 2022-06-27T11:53:05.621 INFO:tasks.workunit.client.0.smithi195.stdout:removed './264/36' 2022-06-27T11:53:05.621 INFO:tasks.workunit.client.0.smithi195.stdout:removed './264/20' 2022-06-27T11:53:05.621 INFO:tasks.workunit.client.0.smithi195.stdout:removed './264/39' 2022-06-27T11:53:05.621 INFO:tasks.workunit.client.0.smithi195.stdout:removed './264/34' 2022-06-27T11:53:05.622 INFO:tasks.workunit.client.0.smithi195.stdout:removed './264/38' 2022-06-27T11:53:05.623 INFO:tasks.workunit.client.0.smithi195.stdout:removed './264/28' 2022-06-27T11:53:05.623 INFO:tasks.workunit.client.0.smithi195.stdout:removed './264/24' 2022-06-27T11:53:05.624 INFO:tasks.workunit.client.0.smithi195.stdout:removed './264/29' 2022-06-27T11:53:05.626 INFO:tasks.workunit.client.0.smithi195.stdout:removed './264/32' 2022-06-27T11:53:05.627 INFO:tasks.workunit.client.0.smithi195.stdout:removed './264/22' 2022-06-27T11:53:05.627 INFO:tasks.workunit.client.0.smithi195.stdout:removed './264/25' 2022-06-27T11:53:05.627 INFO:tasks.workunit.client.0.smithi195.stdout:removed './264/30' 2022-06-27T11:53:05.628 INFO:tasks.workunit.client.0.smithi195.stdout:removed './264/33' 2022-06-27T11:53:05.628 INFO:tasks.workunit.client.0.smithi195.stdout:removed './264/23' 2022-06-27T11:53:05.629 INFO:tasks.workunit.client.0.smithi195.stdout:removed './264/35' 2022-06-27T11:53:05.629 INFO:tasks.workunit.client.0.smithi195.stdout:removed './264/26' 2022-06-27T11:53:05.630 INFO:tasks.workunit.client.0.smithi195.stdout:removed './264/37' 2022-06-27T11:53:05.630 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './264' 2022-06-27T11:53:05.631 INFO:tasks.workunit.client.0.smithi195.stdout:removed './265/21' 2022-06-27T11:53:05.631 INFO:tasks.workunit.client.0.smithi195.stdout:removed './265/27' 2022-06-27T11:53:05.631 INFO:tasks.workunit.client.0.smithi195.stdout:removed './265/31' 2022-06-27T11:53:05.632 INFO:tasks.workunit.client.0.smithi195.stdout:removed './265/36' 2022-06-27T11:53:05.632 INFO:tasks.workunit.client.0.smithi195.stdout:removed './265/20' 2022-06-27T11:53:05.633 INFO:tasks.workunit.client.0.smithi195.stdout:removed './265/39' 2022-06-27T11:53:05.633 INFO:tasks.workunit.client.0.smithi195.stdout:removed './265/34' 2022-06-27T11:53:05.633 INFO:tasks.workunit.client.0.smithi195.stdout:removed './265/38' 2022-06-27T11:53:05.634 INFO:tasks.workunit.client.0.smithi195.stdout:removed './265/28' 2022-06-27T11:53:05.634 INFO:tasks.workunit.client.0.smithi195.stdout:removed './265/24' 2022-06-27T11:53:05.635 INFO:tasks.workunit.client.0.smithi195.stdout:removed './265/29' 2022-06-27T11:53:05.635 INFO:tasks.workunit.client.0.smithi195.stdout:removed './265/32' 2022-06-27T11:53:05.635 INFO:tasks.workunit.client.0.smithi195.stdout:removed './265/22' 2022-06-27T11:53:05.636 INFO:tasks.workunit.client.0.smithi195.stdout:removed './265/25' 2022-06-27T11:53:05.636 INFO:tasks.workunit.client.0.smithi195.stdout:removed './265/30' 2022-06-27T11:53:05.637 INFO:tasks.workunit.client.0.smithi195.stdout:removed './265/33' 2022-06-27T11:53:05.637 INFO:tasks.workunit.client.0.smithi195.stdout:removed './265/23' 2022-06-27T11:53:05.637 INFO:tasks.workunit.client.0.smithi195.stdout:removed './265/35' 2022-06-27T11:53:05.638 INFO:tasks.workunit.client.0.smithi195.stdout:removed './265/26' 2022-06-27T11:53:05.638 INFO:tasks.workunit.client.0.smithi195.stdout:removed './265/37' 2022-06-27T11:53:05.639 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './265' 2022-06-27T11:53:05.639 INFO:tasks.workunit.client.0.smithi195.stdout:removed './266/21' 2022-06-27T11:53:05.640 INFO:tasks.workunit.client.0.smithi195.stdout:removed './266/27' 2022-06-27T11:53:05.640 INFO:tasks.workunit.client.0.smithi195.stdout:removed './266/31' 2022-06-27T11:53:05.641 INFO:tasks.workunit.client.0.smithi195.stdout:removed './266/36' 2022-06-27T11:53:05.641 INFO:tasks.workunit.client.0.smithi195.stdout:removed './266/20' 2022-06-27T11:53:05.641 INFO:tasks.workunit.client.0.smithi195.stdout:removed './266/39' 2022-06-27T11:53:05.642 INFO:tasks.workunit.client.0.smithi195.stdout:removed './266/34' 2022-06-27T11:53:05.642 INFO:tasks.workunit.client.0.smithi195.stdout:removed './266/38' 2022-06-27T11:53:05.643 INFO:tasks.workunit.client.0.smithi195.stdout:removed './266/28' 2022-06-27T11:53:05.643 INFO:tasks.workunit.client.0.smithi195.stdout:removed './266/24' 2022-06-27T11:53:05.644 INFO:tasks.workunit.client.0.smithi195.stdout:removed './266/29' 2022-06-27T11:53:05.644 INFO:tasks.workunit.client.0.smithi195.stdout:removed './266/32' 2022-06-27T11:53:05.645 INFO:tasks.workunit.client.0.smithi195.stdout:removed './266/22' 2022-06-27T11:53:05.645 INFO:tasks.workunit.client.0.smithi195.stdout:removed './266/25' 2022-06-27T11:53:05.646 INFO:tasks.workunit.client.0.smithi195.stdout:removed './266/30' 2022-06-27T11:53:05.646 INFO:tasks.workunit.client.0.smithi195.stdout:removed './266/33' 2022-06-27T11:53:05.647 INFO:tasks.workunit.client.0.smithi195.stdout:removed './266/23' 2022-06-27T11:53:08.579 INFO:tasks.workunit.client.0.smithi195.stdout:removed './266/35' 2022-06-27T11:53:08.580 INFO:tasks.workunit.client.0.smithi195.stdout:removed './266/26' 2022-06-27T11:53:08.580 INFO:tasks.workunit.client.0.smithi195.stdout:removed './266/37' 2022-06-27T11:53:08.580 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './266' 2022-06-27T11:53:08.581 INFO:tasks.workunit.client.0.smithi195.stdout:removed './267/21' 2022-06-27T11:53:08.581 INFO:tasks.workunit.client.0.smithi195.stdout:removed './267/27' 2022-06-27T11:53:08.581 INFO:tasks.workunit.client.0.smithi195.stdout:removed './267/31' 2022-06-27T11:53:08.581 INFO:tasks.workunit.client.0.smithi195.stdout:removed './267/36' 2022-06-27T11:53:08.582 INFO:tasks.workunit.client.0.smithi195.stdout:removed './267/20' 2022-06-27T11:53:08.582 INFO:tasks.workunit.client.0.smithi195.stdout:removed './267/39' 2022-06-27T11:53:08.582 INFO:tasks.workunit.client.0.smithi195.stdout:removed './267/34' 2022-06-27T11:53:08.583 INFO:tasks.workunit.client.0.smithi195.stdout:removed './267/38' 2022-06-27T11:53:08.583 INFO:tasks.workunit.client.0.smithi195.stdout:removed './267/28' 2022-06-27T11:53:08.583 INFO:tasks.workunit.client.0.smithi195.stdout:removed './267/24' 2022-06-27T11:53:08.583 INFO:tasks.workunit.client.0.smithi195.stdout:removed './267/29' 2022-06-27T11:53:08.584 INFO:tasks.workunit.client.0.smithi195.stdout:removed './267/32' 2022-06-27T11:53:08.584 INFO:tasks.workunit.client.0.smithi195.stdout:removed './267/22' 2022-06-27T11:53:08.584 INFO:tasks.workunit.client.0.smithi195.stdout:removed './267/25' 2022-06-27T11:53:08.584 INFO:tasks.workunit.client.0.smithi195.stdout:removed './267/30' 2022-06-27T11:53:08.585 INFO:tasks.workunit.client.0.smithi195.stdout:removed './267/33' 2022-06-27T11:53:08.586 INFO:tasks.workunit.client.0.smithi195.stdout:removed './267/23' 2022-06-27T11:53:08.587 INFO:tasks.workunit.client.0.smithi195.stdout:removed './267/35' 2022-06-27T11:53:08.588 INFO:tasks.workunit.client.0.smithi195.stdout:removed './267/26' 2022-06-27T11:53:08.588 INFO:tasks.workunit.client.0.smithi195.stdout:removed './267/37' 2022-06-27T11:53:08.589 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './267' 2022-06-27T11:53:08.589 INFO:tasks.workunit.client.0.smithi195.stdout:removed './268/21' 2022-06-27T11:53:08.589 INFO:tasks.workunit.client.0.smithi195.stdout:removed './268/27' 2022-06-27T11:53:08.590 INFO:tasks.workunit.client.0.smithi195.stdout:removed './268/31' 2022-06-27T11:53:08.590 INFO:tasks.workunit.client.0.smithi195.stdout:removed './268/36' 2022-06-27T11:53:08.591 INFO:tasks.workunit.client.0.smithi195.stdout:removed './268/20' 2022-06-27T11:53:08.592 INFO:tasks.workunit.client.0.smithi195.stdout:removed './268/39' 2022-06-27T11:53:08.592 INFO:tasks.workunit.client.0.smithi195.stdout:removed './268/34' 2022-06-27T11:53:08.593 INFO:tasks.workunit.client.0.smithi195.stdout:removed './268/38' 2022-06-27T11:53:08.594 INFO:tasks.workunit.client.0.smithi195.stdout:removed './268/28' 2022-06-27T11:53:08.595 INFO:tasks.workunit.client.0.smithi195.stdout:removed './268/24' 2022-06-27T11:53:08.595 INFO:tasks.workunit.client.0.smithi195.stdout:removed './268/29' 2022-06-27T11:53:08.595 INFO:tasks.workunit.client.0.smithi195.stdout:removed './268/32' 2022-06-27T11:53:08.595 INFO:tasks.workunit.client.0.smithi195.stdout:removed './268/22' 2022-06-27T11:53:08.596 INFO:tasks.workunit.client.0.smithi195.stdout:removed './268/25' 2022-06-27T11:53:08.596 INFO:tasks.workunit.client.0.smithi195.stdout:removed './268/30' 2022-06-27T11:53:08.596 INFO:tasks.workunit.client.0.smithi195.stdout:removed './268/33' 2022-06-27T11:53:08.599 INFO:tasks.workunit.client.0.smithi195.stdout:removed './268/23' 2022-06-27T11:53:08.599 INFO:tasks.workunit.client.0.smithi195.stdout:removed './268/35' 2022-06-27T11:53:08.599 INFO:tasks.workunit.client.0.smithi195.stdout:removed './268/26' 2022-06-27T11:53:08.600 INFO:tasks.workunit.client.0.smithi195.stdout:removed './268/37' 2022-06-27T11:53:08.600 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './268' 2022-06-27T11:53:08.600 INFO:tasks.workunit.client.0.smithi195.stdout:removed './269/21' 2022-06-27T11:53:08.600 INFO:tasks.workunit.client.0.smithi195.stdout:removed './269/27' 2022-06-27T11:53:08.601 INFO:tasks.workunit.client.0.smithi195.stdout:removed './269/31' 2022-06-27T11:53:08.601 INFO:tasks.workunit.client.0.smithi195.stdout:removed './269/36' 2022-06-27T11:53:08.601 INFO:tasks.workunit.client.0.smithi195.stdout:removed './269/20' 2022-06-27T11:53:08.602 INFO:tasks.workunit.client.0.smithi195.stdout:removed './269/39' 2022-06-27T11:53:08.602 INFO:tasks.workunit.client.0.smithi195.stdout:removed './269/34' 2022-06-27T11:53:08.602 INFO:tasks.workunit.client.0.smithi195.stdout:removed './269/38' 2022-06-27T11:53:08.602 INFO:tasks.workunit.client.0.smithi195.stdout:removed './269/28' 2022-06-27T11:53:08.603 INFO:tasks.workunit.client.0.smithi195.stdout:removed './269/24' 2022-06-27T11:53:08.603 INFO:tasks.workunit.client.0.smithi195.stdout:removed './269/29' 2022-06-27T11:53:08.603 INFO:tasks.workunit.client.0.smithi195.stdout:removed './269/32' 2022-06-27T11:53:08.603 INFO:tasks.workunit.client.0.smithi195.stdout:removed './269/22' 2022-06-27T11:53:08.604 INFO:tasks.workunit.client.0.smithi195.stdout:removed './269/25' 2022-06-27T11:53:08.604 INFO:tasks.workunit.client.0.smithi195.stdout:removed './269/30' 2022-06-27T11:53:08.604 INFO:tasks.workunit.client.0.smithi195.stdout:removed './269/33' 2022-06-27T11:53:08.604 INFO:tasks.workunit.client.0.smithi195.stdout:removed './269/23' 2022-06-27T11:53:08.604 INFO:tasks.workunit.client.0.smithi195.stdout:removed './269/35' 2022-06-27T11:53:08.605 INFO:tasks.workunit.client.0.smithi195.stdout:removed './269/26' 2022-06-27T11:53:08.605 INFO:tasks.workunit.client.0.smithi195.stdout:removed './269/37' 2022-06-27T11:53:08.605 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './269' 2022-06-27T11:53:08.605 INFO:tasks.workunit.client.0.smithi195.stdout:removed './270/21' 2022-06-27T11:53:08.606 INFO:tasks.workunit.client.0.smithi195.stdout:removed './270/27' 2022-06-27T11:53:08.606 INFO:tasks.workunit.client.0.smithi195.stdout:removed './270/31' 2022-06-27T11:53:08.606 INFO:tasks.workunit.client.0.smithi195.stdout:removed './270/36' 2022-06-27T11:53:08.606 INFO:tasks.workunit.client.0.smithi195.stdout:removed './270/20' 2022-06-27T11:53:08.607 INFO:tasks.workunit.client.0.smithi195.stdout:removed './270/39' 2022-06-27T11:53:08.607 INFO:tasks.workunit.client.0.smithi195.stdout:removed './270/34' 2022-06-27T11:53:08.607 INFO:tasks.workunit.client.0.smithi195.stdout:removed './270/38' 2022-06-27T11:53:08.607 INFO:tasks.workunit.client.0.smithi195.stdout:removed './270/28' 2022-06-27T11:53:08.607 INFO:tasks.workunit.client.0.smithi195.stdout:removed './270/24' 2022-06-27T11:53:08.608 INFO:tasks.workunit.client.0.smithi195.stdout:removed './270/29' 2022-06-27T11:53:08.608 INFO:tasks.workunit.client.0.smithi195.stdout:removed './270/32' 2022-06-27T11:53:08.608 INFO:tasks.workunit.client.0.smithi195.stdout:removed './270/22' 2022-06-27T11:53:08.608 INFO:tasks.workunit.client.0.smithi195.stdout:removed './270/25' 2022-06-27T11:53:08.609 INFO:tasks.workunit.client.0.smithi195.stdout:removed './270/30' 2022-06-27T11:53:08.609 INFO:tasks.workunit.client.0.smithi195.stdout:removed './270/33' 2022-06-27T11:53:08.609 INFO:tasks.workunit.client.0.smithi195.stdout:removed './270/23' 2022-06-27T11:53:08.609 INFO:tasks.workunit.client.0.smithi195.stdout:removed './270/35' 2022-06-27T11:53:08.609 INFO:tasks.workunit.client.0.smithi195.stdout:removed './270/26' 2022-06-27T11:53:08.610 INFO:tasks.workunit.client.0.smithi195.stdout:removed './270/37' 2022-06-27T11:53:08.610 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './270' 2022-06-27T11:53:08.610 INFO:tasks.workunit.client.0.smithi195.stdout:removed './271/21' 2022-06-27T11:53:08.610 INFO:tasks.workunit.client.0.smithi195.stdout:removed './271/27' 2022-06-27T11:53:08.611 INFO:tasks.workunit.client.0.smithi195.stdout:removed './271/31' 2022-06-27T11:53:08.611 INFO:tasks.workunit.client.0.smithi195.stdout:removed './271/36' 2022-06-27T11:53:08.611 INFO:tasks.workunit.client.0.smithi195.stdout:removed './271/20' 2022-06-27T11:53:08.611 INFO:tasks.workunit.client.0.smithi195.stdout:removed './271/39' 2022-06-27T11:53:08.612 INFO:tasks.workunit.client.0.smithi195.stdout:removed './271/34' 2022-06-27T11:53:08.612 INFO:tasks.workunit.client.0.smithi195.stdout:removed './271/38' 2022-06-27T11:53:08.612 INFO:tasks.workunit.client.0.smithi195.stdout:removed './271/28' 2022-06-27T11:53:08.612 INFO:tasks.workunit.client.0.smithi195.stdout:removed './271/24' 2022-06-27T11:53:08.612 INFO:tasks.workunit.client.0.smithi195.stdout:removed './271/29' 2022-06-27T11:53:08.613 INFO:tasks.workunit.client.0.smithi195.stdout:removed './271/32' 2022-06-27T11:53:08.613 INFO:tasks.workunit.client.0.smithi195.stdout:removed './271/22' 2022-06-27T11:53:08.613 INFO:tasks.workunit.client.0.smithi195.stdout:removed './271/25' 2022-06-27T11:53:08.613 INFO:tasks.workunit.client.0.smithi195.stdout:removed './271/30' 2022-06-27T11:53:08.614 INFO:tasks.workunit.client.0.smithi195.stdout:removed './271/33' 2022-06-27T11:53:08.614 INFO:tasks.workunit.client.0.smithi195.stdout:removed './271/23' 2022-06-27T11:53:08.614 INFO:tasks.workunit.client.0.smithi195.stdout:removed './271/35' 2022-06-27T11:53:08.614 INFO:tasks.workunit.client.0.smithi195.stdout:removed './271/26' 2022-06-27T11:53:08.615 INFO:tasks.workunit.client.0.smithi195.stdout:removed './271/37' 2022-06-27T11:53:08.615 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './271' 2022-06-27T11:53:08.615 INFO:tasks.workunit.client.0.smithi195.stdout:removed './272/21' 2022-06-27T11:53:08.615 INFO:tasks.workunit.client.0.smithi195.stdout:removed './272/27' 2022-06-27T11:53:08.616 INFO:tasks.workunit.client.0.smithi195.stdout:removed './272/31' 2022-06-27T11:53:08.616 INFO:tasks.workunit.client.0.smithi195.stdout:removed './272/36' 2022-06-27T11:53:08.616 INFO:tasks.workunit.client.0.smithi195.stdout:removed './272/20' 2022-06-27T11:53:08.616 INFO:tasks.workunit.client.0.smithi195.stdout:removed './272/39' 2022-06-27T11:53:08.617 INFO:tasks.workunit.client.0.smithi195.stdout:removed './272/34' 2022-06-27T11:53:08.617 INFO:tasks.workunit.client.0.smithi195.stdout:removed './272/38' 2022-06-27T11:53:08.617 INFO:tasks.workunit.client.0.smithi195.stdout:removed './272/28' 2022-06-27T11:53:08.617 INFO:tasks.workunit.client.0.smithi195.stdout:removed './272/24' 2022-06-27T11:53:08.618 INFO:tasks.workunit.client.0.smithi195.stdout:removed './272/29' 2022-06-27T11:53:08.618 INFO:tasks.workunit.client.0.smithi195.stdout:removed './272/32' 2022-06-27T11:53:08.618 INFO:tasks.workunit.client.0.smithi195.stdout:removed './272/22' 2022-06-27T11:53:08.618 INFO:tasks.workunit.client.0.smithi195.stdout:removed './272/25' 2022-06-27T11:53:08.619 INFO:tasks.workunit.client.0.smithi195.stdout:removed './272/30' 2022-06-27T11:53:08.619 INFO:tasks.workunit.client.0.smithi195.stdout:removed './272/33' 2022-06-27T11:53:08.619 INFO:tasks.workunit.client.0.smithi195.stdout:removed './272/23' 2022-06-27T11:53:08.619 INFO:tasks.workunit.client.0.smithi195.stdout:removed './272/35' 2022-06-27T11:53:08.619 INFO:tasks.workunit.client.0.smithi195.stdout:removed './272/26' 2022-06-27T11:53:08.620 INFO:tasks.workunit.client.0.smithi195.stdout:removed './272/37' 2022-06-27T11:53:08.620 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './272' 2022-06-27T11:53:08.620 INFO:tasks.workunit.client.0.smithi195.stdout:removed './273/21' 2022-06-27T11:53:08.620 INFO:tasks.workunit.client.0.smithi195.stdout:removed './273/27' 2022-06-27T11:53:08.621 INFO:tasks.workunit.client.0.smithi195.stdout:removed './273/31' 2022-06-27T11:53:08.621 INFO:tasks.workunit.client.0.smithi195.stdout:removed './273/36' 2022-06-27T11:53:08.621 INFO:tasks.workunit.client.0.smithi195.stdout:removed './273/20' 2022-06-27T11:53:08.621 INFO:tasks.workunit.client.0.smithi195.stdout:removed './273/39' 2022-06-27T11:53:08.622 INFO:tasks.workunit.client.0.smithi195.stdout:removed './273/34' 2022-06-27T11:53:08.622 INFO:tasks.workunit.client.0.smithi195.stdout:removed './273/38' 2022-06-27T11:53:08.622 INFO:tasks.workunit.client.0.smithi195.stdout:removed './273/28' 2022-06-27T11:53:08.622 INFO:tasks.workunit.client.0.smithi195.stdout:removed './273/24' 2022-06-27T11:53:08.623 INFO:tasks.workunit.client.0.smithi195.stdout:removed './273/29' 2022-06-27T11:53:08.623 INFO:tasks.workunit.client.0.smithi195.stdout:removed './273/32' 2022-06-27T11:53:08.623 INFO:tasks.workunit.client.0.smithi195.stdout:removed './273/22' 2022-06-27T11:53:08.623 INFO:tasks.workunit.client.0.smithi195.stdout:removed './273/25' 2022-06-27T11:53:08.624 INFO:tasks.workunit.client.0.smithi195.stdout:removed './273/30' 2022-06-27T11:53:08.624 INFO:tasks.workunit.client.0.smithi195.stdout:removed './273/33' 2022-06-27T11:53:08.624 INFO:tasks.workunit.client.0.smithi195.stdout:removed './273/23' 2022-06-27T11:53:08.624 INFO:tasks.workunit.client.0.smithi195.stdout:removed './273/35' 2022-06-27T11:53:08.624 INFO:tasks.workunit.client.0.smithi195.stdout:removed './273/26' 2022-06-27T11:53:08.625 INFO:tasks.workunit.client.0.smithi195.stdout:removed './273/37' 2022-06-27T11:53:08.625 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './273' 2022-06-27T11:53:08.625 INFO:tasks.workunit.client.0.smithi195.stdout:removed './274/21' 2022-06-27T11:53:08.625 INFO:tasks.workunit.client.0.smithi195.stdout:removed './274/27' 2022-06-27T11:53:08.626 INFO:tasks.workunit.client.0.smithi195.stdout:removed './274/31' 2022-06-27T11:53:08.626 INFO:tasks.workunit.client.0.smithi195.stdout:removed './274/36' 2022-06-27T11:53:08.626 INFO:tasks.workunit.client.0.smithi195.stdout:removed './274/20' 2022-06-27T11:53:08.626 INFO:tasks.workunit.client.0.smithi195.stdout:removed './274/39' 2022-06-27T11:53:08.627 INFO:tasks.workunit.client.0.smithi195.stdout:removed './274/34' 2022-06-27T11:53:08.627 INFO:tasks.workunit.client.0.smithi195.stdout:removed './274/38' 2022-06-27T11:53:08.627 INFO:tasks.workunit.client.0.smithi195.stdout:removed './274/28' 2022-06-27T11:53:08.627 INFO:tasks.workunit.client.0.smithi195.stdout:removed './274/24' 2022-06-27T11:53:08.628 INFO:tasks.workunit.client.0.smithi195.stdout:removed './274/29' 2022-06-27T11:53:08.628 INFO:tasks.workunit.client.0.smithi195.stdout:removed './274/32' 2022-06-27T11:53:08.628 INFO:tasks.workunit.client.0.smithi195.stdout:removed './274/22' 2022-06-27T11:53:08.628 INFO:tasks.workunit.client.0.smithi195.stdout:removed './274/25' 2022-06-27T11:53:08.628 INFO:tasks.workunit.client.0.smithi195.stdout:removed './274/30' 2022-06-27T11:53:08.629 INFO:tasks.workunit.client.0.smithi195.stdout:removed './274/33' 2022-06-27T11:53:08.629 INFO:tasks.workunit.client.0.smithi195.stdout:removed './274/23' 2022-06-27T11:53:08.629 INFO:tasks.workunit.client.0.smithi195.stdout:removed './274/35' 2022-06-27T11:53:08.629 INFO:tasks.workunit.client.0.smithi195.stdout:removed './274/26' 2022-06-27T11:53:08.630 INFO:tasks.workunit.client.0.smithi195.stdout:removed './274/37' 2022-06-27T11:53:08.630 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './274' 2022-06-27T11:53:08.630 INFO:tasks.workunit.client.0.smithi195.stdout:removed './275/21' 2022-06-27T11:53:08.630 INFO:tasks.workunit.client.0.smithi195.stdout:removed './275/27' 2022-06-27T11:53:08.631 INFO:tasks.workunit.client.0.smithi195.stdout:removed './275/31' 2022-06-27T11:53:08.631 INFO:tasks.workunit.client.0.smithi195.stdout:removed './275/36' 2022-06-27T11:53:08.631 INFO:tasks.workunit.client.0.smithi195.stdout:removed './275/20' 2022-06-27T11:53:08.631 INFO:tasks.workunit.client.0.smithi195.stdout:removed './275/39' 2022-06-27T11:53:08.631 INFO:tasks.workunit.client.0.smithi195.stdout:removed './275/34' 2022-06-27T11:53:08.632 INFO:tasks.workunit.client.0.smithi195.stdout:removed './275/38' 2022-06-27T11:53:08.632 INFO:tasks.workunit.client.0.smithi195.stdout:removed './275/28' 2022-06-27T11:53:08.632 INFO:tasks.workunit.client.0.smithi195.stdout:removed './275/24' 2022-06-27T11:53:08.632 INFO:tasks.workunit.client.0.smithi195.stdout:removed './275/29' 2022-06-27T11:53:08.633 INFO:tasks.workunit.client.0.smithi195.stdout:removed './275/32' 2022-06-27T11:53:08.633 INFO:tasks.workunit.client.0.smithi195.stdout:removed './275/22' 2022-06-27T11:53:08.633 INFO:tasks.workunit.client.0.smithi195.stdout:removed './275/25' 2022-06-27T11:53:08.633 INFO:tasks.workunit.client.0.smithi195.stdout:removed './275/30' 2022-06-27T11:53:08.633 INFO:tasks.workunit.client.0.smithi195.stdout:removed './275/33' 2022-06-27T11:53:08.634 INFO:tasks.workunit.client.0.smithi195.stdout:removed './275/23' 2022-06-27T11:53:08.634 INFO:tasks.workunit.client.0.smithi195.stdout:removed './275/35' 2022-06-27T11:53:08.634 INFO:tasks.workunit.client.0.smithi195.stdout:removed './275/26' 2022-06-27T11:53:08.634 INFO:tasks.workunit.client.0.smithi195.stdout:removed './275/37' 2022-06-27T11:53:08.635 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './275' 2022-06-27T11:53:08.635 INFO:tasks.workunit.client.0.smithi195.stdout:removed './276/21' 2022-06-27T11:53:08.635 INFO:tasks.workunit.client.0.smithi195.stdout:removed './276/27' 2022-06-27T11:53:08.635 INFO:tasks.workunit.client.0.smithi195.stdout:removed './276/31' 2022-06-27T11:53:08.635 INFO:tasks.workunit.client.0.smithi195.stdout:removed './276/36' 2022-06-27T11:53:08.636 INFO:tasks.workunit.client.0.smithi195.stdout:removed './276/20' 2022-06-27T11:53:08.636 INFO:tasks.workunit.client.0.smithi195.stdout:removed './276/39' 2022-06-27T11:53:08.636 INFO:tasks.workunit.client.0.smithi195.stdout:removed './276/34' 2022-06-27T11:53:08.636 INFO:tasks.workunit.client.0.smithi195.stdout:removed './276/38' 2022-06-27T11:53:08.637 INFO:tasks.workunit.client.0.smithi195.stdout:removed './276/28' 2022-06-27T11:53:08.637 INFO:tasks.workunit.client.0.smithi195.stdout:removed './276/24' 2022-06-27T11:53:08.637 INFO:tasks.workunit.client.0.smithi195.stdout:removed './276/29' 2022-06-27T11:53:08.637 INFO:tasks.workunit.client.0.smithi195.stdout:removed './276/32' 2022-06-27T11:53:08.638 INFO:tasks.workunit.client.0.smithi195.stdout:removed './276/22' 2022-06-27T11:53:08.638 INFO:tasks.workunit.client.0.smithi195.stdout:removed './276/25' 2022-06-27T11:53:08.639 INFO:tasks.workunit.client.0.smithi195.stdout:removed './276/30' 2022-06-27T11:53:08.639 INFO:tasks.workunit.client.0.smithi195.stdout:removed './276/33' 2022-06-27T11:53:08.639 INFO:tasks.workunit.client.0.smithi195.stdout:removed './276/23' 2022-06-27T11:53:08.639 INFO:tasks.workunit.client.0.smithi195.stdout:removed './276/35' 2022-06-27T11:53:08.639 INFO:tasks.workunit.client.0.smithi195.stdout:removed './276/26' 2022-06-27T11:53:09.453 INFO:tasks.workunit.client.0.smithi195.stdout:removed './276/37' 2022-06-27T11:53:09.453 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './276' 2022-06-27T11:53:09.454 INFO:tasks.workunit.client.0.smithi195.stdout:removed './277/21' 2022-06-27T11:53:09.454 INFO:tasks.workunit.client.0.smithi195.stdout:removed './277/27' 2022-06-27T11:53:09.455 INFO:tasks.workunit.client.0.smithi195.stdout:removed './277/31' 2022-06-27T11:53:09.455 INFO:tasks.workunit.client.0.smithi195.stdout:removed './277/36' 2022-06-27T11:53:09.456 INFO:tasks.workunit.client.0.smithi195.stdout:removed './277/20' 2022-06-27T11:53:09.456 INFO:tasks.workunit.client.0.smithi195.stdout:removed './277/39' 2022-06-27T11:53:09.457 INFO:tasks.workunit.client.0.smithi195.stdout:removed './277/34' 2022-06-27T11:53:09.457 INFO:tasks.workunit.client.0.smithi195.stdout:removed './277/38' 2022-06-27T11:53:09.458 INFO:tasks.workunit.client.0.smithi195.stdout:removed './277/28' 2022-06-27T11:53:09.458 INFO:tasks.workunit.client.0.smithi195.stdout:removed './277/24' 2022-06-27T11:53:09.459 INFO:tasks.workunit.client.0.smithi195.stdout:removed './277/29' 2022-06-27T11:53:09.459 INFO:tasks.workunit.client.0.smithi195.stdout:removed './277/32' 2022-06-27T11:53:09.459 INFO:tasks.workunit.client.0.smithi195.stdout:removed './277/22' 2022-06-27T11:53:09.460 INFO:tasks.workunit.client.0.smithi195.stdout:removed './277/25' 2022-06-27T11:53:09.460 INFO:tasks.workunit.client.0.smithi195.stdout:removed './277/30' 2022-06-27T11:53:09.461 INFO:tasks.workunit.client.0.smithi195.stdout:removed './277/33' 2022-06-27T11:53:09.461 INFO:tasks.workunit.client.0.smithi195.stdout:removed './277/23' 2022-06-27T11:53:09.462 INFO:tasks.workunit.client.0.smithi195.stdout:removed './277/35' 2022-06-27T11:53:09.462 INFO:tasks.workunit.client.0.smithi195.stdout:removed './277/26' 2022-06-27T11:53:09.462 INFO:tasks.workunit.client.0.smithi195.stdout:removed './277/37' 2022-06-27T11:53:09.463 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './277' 2022-06-27T11:53:09.463 INFO:tasks.workunit.client.0.smithi195.stdout:removed './278/21' 2022-06-27T11:53:09.464 INFO:tasks.workunit.client.0.smithi195.stdout:removed './278/27' 2022-06-27T11:53:09.464 INFO:tasks.workunit.client.0.smithi195.stdout:removed './278/31' 2022-06-27T11:53:09.465 INFO:tasks.workunit.client.0.smithi195.stdout:removed './278/36' 2022-06-27T11:53:09.465 INFO:tasks.workunit.client.0.smithi195.stdout:removed './278/20' 2022-06-27T11:53:09.466 INFO:tasks.workunit.client.0.smithi195.stdout:removed './278/39' 2022-06-27T11:53:09.466 INFO:tasks.workunit.client.0.smithi195.stdout:removed './278/34' 2022-06-27T11:53:09.466 INFO:tasks.workunit.client.0.smithi195.stdout:removed './278/38' 2022-06-27T11:53:09.467 INFO:tasks.workunit.client.0.smithi195.stdout:removed './278/28' 2022-06-27T11:53:09.467 INFO:tasks.workunit.client.0.smithi195.stdout:removed './278/24' 2022-06-27T11:53:09.468 INFO:tasks.workunit.client.0.smithi195.stdout:removed './278/29' 2022-06-27T11:53:09.468 INFO:tasks.workunit.client.0.smithi195.stdout:removed './278/32' 2022-06-27T11:53:09.469 INFO:tasks.workunit.client.0.smithi195.stdout:removed './278/22' 2022-06-27T11:53:09.469 INFO:tasks.workunit.client.0.smithi195.stdout:removed './278/25' 2022-06-27T11:53:09.470 INFO:tasks.workunit.client.0.smithi195.stdout:removed './278/30' 2022-06-27T11:53:09.470 INFO:tasks.workunit.client.0.smithi195.stdout:removed './278/33' 2022-06-27T11:53:09.471 INFO:tasks.workunit.client.0.smithi195.stdout:removed './278/23' 2022-06-27T11:53:09.471 INFO:tasks.workunit.client.0.smithi195.stdout:removed './278/35' 2022-06-27T11:53:09.472 INFO:tasks.workunit.client.0.smithi195.stdout:removed './278/26' 2022-06-27T11:53:09.472 INFO:tasks.workunit.client.0.smithi195.stdout:removed './278/37' 2022-06-27T11:53:09.473 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './278' 2022-06-27T11:53:09.473 INFO:tasks.workunit.client.0.smithi195.stdout:removed './279/21' 2022-06-27T11:53:09.474 INFO:tasks.workunit.client.0.smithi195.stdout:removed './279/27' 2022-06-27T11:53:09.474 INFO:tasks.workunit.client.0.smithi195.stdout:removed './279/31' 2022-06-27T11:53:09.474 INFO:tasks.workunit.client.0.smithi195.stdout:removed './279/36' 2022-06-27T11:53:09.475 INFO:tasks.workunit.client.0.smithi195.stdout:removed './279/20' 2022-06-27T11:53:09.475 INFO:tasks.workunit.client.0.smithi195.stdout:removed './279/39' 2022-06-27T11:53:09.476 INFO:tasks.workunit.client.0.smithi195.stdout:removed './279/34' 2022-06-27T11:53:09.476 INFO:tasks.workunit.client.0.smithi195.stdout:removed './279/38' 2022-06-27T11:53:09.476 INFO:tasks.workunit.client.0.smithi195.stdout:removed './279/28' 2022-06-27T11:53:09.477 INFO:tasks.workunit.client.0.smithi195.stdout:removed './279/24' 2022-06-27T11:53:09.478 INFO:tasks.workunit.client.0.smithi195.stdout:removed './279/29' 2022-06-27T11:53:09.479 INFO:tasks.workunit.client.0.smithi195.stdout:removed './279/32' 2022-06-27T11:53:09.479 INFO:tasks.workunit.client.0.smithi195.stdout:removed './279/22' 2022-06-27T11:53:09.480 INFO:tasks.workunit.client.0.smithi195.stdout:removed './279/25' 2022-06-27T11:53:09.480 INFO:tasks.workunit.client.0.smithi195.stdout:removed './279/30' 2022-06-27T11:53:09.480 INFO:tasks.workunit.client.0.smithi195.stdout:removed './279/33' 2022-06-27T11:53:09.481 INFO:tasks.workunit.client.0.smithi195.stdout:removed './279/23' 2022-06-27T11:53:09.481 INFO:tasks.workunit.client.0.smithi195.stdout:removed './279/35' 2022-06-27T11:53:09.491 INFO:tasks.workunit.client.0.smithi195.stdout:removed './279/26' 2022-06-27T11:53:09.491 INFO:tasks.workunit.client.0.smithi195.stdout:removed './279/37' 2022-06-27T11:53:09.492 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './279' 2022-06-27T11:53:09.492 INFO:tasks.workunit.client.0.smithi195.stdout:removed './280/21' 2022-06-27T11:53:09.493 INFO:tasks.workunit.client.0.smithi195.stdout:removed './280/27' 2022-06-27T11:53:09.493 INFO:tasks.workunit.client.0.smithi195.stdout:removed './280/31' 2022-06-27T11:53:09.494 INFO:tasks.workunit.client.0.smithi195.stdout:removed './280/36' 2022-06-27T11:53:09.495 INFO:tasks.workunit.client.0.smithi195.stdout:removed './280/20' 2022-06-27T11:53:09.495 INFO:tasks.workunit.client.0.smithi195.stdout:removed './280/39' 2022-06-27T11:53:09.495 INFO:tasks.workunit.client.0.smithi195.stdout:removed './280/34' 2022-06-27T11:53:09.496 INFO:tasks.workunit.client.0.smithi195.stdout:removed './280/38' 2022-06-27T11:53:09.496 INFO:tasks.workunit.client.0.smithi195.stdout:removed './280/28' 2022-06-27T11:53:09.497 INFO:tasks.workunit.client.0.smithi195.stdout:removed './280/24' 2022-06-27T11:53:09.497 INFO:tasks.workunit.client.0.smithi195.stdout:removed './280/29' 2022-06-27T11:53:09.498 INFO:tasks.workunit.client.0.smithi195.stdout:removed './280/32' 2022-06-27T11:53:09.498 INFO:tasks.workunit.client.0.smithi195.stdout:removed './280/22' 2022-06-27T11:53:09.499 INFO:tasks.workunit.client.0.smithi195.stdout:removed './280/25' 2022-06-27T11:53:09.499 INFO:tasks.workunit.client.0.smithi195.stdout:removed './280/30' 2022-06-27T11:53:09.500 INFO:tasks.workunit.client.0.smithi195.stdout:removed './280/33' 2022-06-27T11:53:09.500 INFO:tasks.workunit.client.0.smithi195.stdout:removed './280/23' 2022-06-27T11:53:09.500 INFO:tasks.workunit.client.0.smithi195.stdout:removed './280/35' 2022-06-27T11:53:09.501 INFO:tasks.workunit.client.0.smithi195.stdout:removed './280/26' 2022-06-27T11:53:09.501 INFO:tasks.workunit.client.0.smithi195.stdout:removed './280/37' 2022-06-27T11:53:09.502 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './280' 2022-06-27T11:53:09.502 INFO:tasks.workunit.client.0.smithi195.stdout:removed './281/21' 2022-06-27T11:53:09.503 INFO:tasks.workunit.client.0.smithi195.stdout:removed './281/27' 2022-06-27T11:53:09.503 INFO:tasks.workunit.client.0.smithi195.stdout:removed './281/31' 2022-06-27T11:53:09.504 INFO:tasks.workunit.client.0.smithi195.stdout:removed './281/36' 2022-06-27T11:53:09.504 INFO:tasks.workunit.client.0.smithi195.stdout:removed './281/20' 2022-06-27T11:53:09.505 INFO:tasks.workunit.client.0.smithi195.stdout:removed './281/39' 2022-06-27T11:53:09.505 INFO:tasks.workunit.client.0.smithi195.stdout:removed './281/34' 2022-06-27T11:53:09.506 INFO:tasks.workunit.client.0.smithi195.stdout:removed './281/38' 2022-06-27T11:53:09.506 INFO:tasks.workunit.client.0.smithi195.stdout:removed './281/28' 2022-06-27T11:53:09.507 INFO:tasks.workunit.client.0.smithi195.stdout:removed './281/24' 2022-06-27T11:53:09.507 INFO:tasks.workunit.client.0.smithi195.stdout:removed './281/29' 2022-06-27T11:53:09.508 INFO:tasks.workunit.client.0.smithi195.stdout:removed './281/32' 2022-06-27T11:53:09.508 INFO:tasks.workunit.client.0.smithi195.stdout:removed './281/22' 2022-06-27T11:53:09.508 INFO:tasks.workunit.client.0.smithi195.stdout:removed './281/25' 2022-06-27T11:53:09.509 INFO:tasks.workunit.client.0.smithi195.stdout:removed './281/30' 2022-06-27T11:53:09.509 INFO:tasks.workunit.client.0.smithi195.stdout:removed './281/33' 2022-06-27T11:53:09.510 INFO:tasks.workunit.client.0.smithi195.stdout:removed './281/23' 2022-06-27T11:53:09.510 INFO:tasks.workunit.client.0.smithi195.stdout:removed './281/35' 2022-06-27T11:53:09.511 INFO:tasks.workunit.client.0.smithi195.stdout:removed './281/26' 2022-06-27T11:53:09.511 INFO:tasks.workunit.client.0.smithi195.stdout:removed './281/37' 2022-06-27T11:53:09.512 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './281' 2022-06-27T11:53:09.512 INFO:tasks.workunit.client.0.smithi195.stdout:removed './282/21' 2022-06-27T11:53:09.513 INFO:tasks.workunit.client.0.smithi195.stdout:removed './282/27' 2022-06-27T11:53:09.513 INFO:tasks.workunit.client.0.smithi195.stdout:removed './282/31' 2022-06-27T11:53:09.513 INFO:tasks.workunit.client.0.smithi195.stdout:removed './282/36' 2022-06-27T11:53:09.514 INFO:tasks.workunit.client.0.smithi195.stdout:removed './282/20' 2022-06-27T11:53:09.514 INFO:tasks.workunit.client.0.smithi195.stdout:removed './282/39' 2022-06-27T11:53:09.515 INFO:tasks.workunit.client.0.smithi195.stdout:removed './282/34' 2022-06-27T11:53:09.515 INFO:tasks.workunit.client.0.smithi195.stdout:removed './282/38' 2022-06-27T11:53:09.516 INFO:tasks.workunit.client.0.smithi195.stdout:removed './282/28' 2022-06-27T11:53:09.516 INFO:tasks.workunit.client.0.smithi195.stdout:removed './282/24' 2022-06-27T11:53:09.517 INFO:tasks.workunit.client.0.smithi195.stdout:removed './282/29' 2022-06-27T11:53:09.517 INFO:tasks.workunit.client.0.smithi195.stdout:removed './282/32' 2022-06-27T11:53:09.517 INFO:tasks.workunit.client.0.smithi195.stdout:removed './282/22' 2022-06-27T11:53:09.518 INFO:tasks.workunit.client.0.smithi195.stdout:removed './282/25' 2022-06-27T11:53:09.518 INFO:tasks.workunit.client.0.smithi195.stdout:removed './282/30' 2022-06-27T11:53:09.519 INFO:tasks.workunit.client.0.smithi195.stdout:removed './282/33' 2022-06-27T11:53:09.519 INFO:tasks.workunit.client.0.smithi195.stdout:removed './282/23' 2022-06-27T11:53:09.520 INFO:tasks.workunit.client.0.smithi195.stdout:removed './282/35' 2022-06-27T11:53:09.520 INFO:tasks.workunit.client.0.smithi195.stdout:removed './282/26' 2022-06-27T11:53:09.521 INFO:tasks.workunit.client.0.smithi195.stdout:removed './282/37' 2022-06-27T11:53:09.521 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './282' 2022-06-27T11:53:09.521 INFO:tasks.workunit.client.0.smithi195.stdout:removed './283/21' 2022-06-27T11:53:09.522 INFO:tasks.workunit.client.0.smithi195.stdout:removed './283/27' 2022-06-27T11:53:09.522 INFO:tasks.workunit.client.0.smithi195.stdout:removed './283/31' 2022-06-27T11:53:09.523 INFO:tasks.workunit.client.0.smithi195.stdout:removed './283/36' 2022-06-27T11:53:09.523 INFO:tasks.workunit.client.0.smithi195.stdout:removed './283/20' 2022-06-27T11:53:09.524 INFO:tasks.workunit.client.0.smithi195.stdout:removed './283/39' 2022-06-27T11:53:09.525 INFO:tasks.workunit.client.0.smithi195.stdout:removed './283/34' 2022-06-27T11:53:09.525 INFO:tasks.workunit.client.0.smithi195.stdout:removed './283/38' 2022-06-27T11:53:09.526 INFO:tasks.workunit.client.0.smithi195.stdout:removed './283/28' 2022-06-27T11:53:09.526 INFO:tasks.workunit.client.0.smithi195.stdout:removed './283/24' 2022-06-27T11:53:09.527 INFO:tasks.workunit.client.0.smithi195.stdout:removed './283/29' 2022-06-27T11:53:09.527 INFO:tasks.workunit.client.0.smithi195.stdout:removed './283/32' 2022-06-27T11:53:09.528 INFO:tasks.workunit.client.0.smithi195.stdout:removed './283/22' 2022-06-27T11:53:09.528 INFO:tasks.workunit.client.0.smithi195.stdout:removed './283/25' 2022-06-27T11:53:09.529 INFO:tasks.workunit.client.0.smithi195.stdout:removed './283/30' 2022-06-27T11:53:09.529 INFO:tasks.workunit.client.0.smithi195.stdout:removed './283/33' 2022-06-27T11:53:09.529 INFO:tasks.workunit.client.0.smithi195.stdout:removed './283/23' 2022-06-27T11:53:09.530 INFO:tasks.workunit.client.0.smithi195.stdout:removed './283/35' 2022-06-27T11:53:09.530 INFO:tasks.workunit.client.0.smithi195.stdout:removed './283/26' 2022-06-27T11:53:09.531 INFO:tasks.workunit.client.0.smithi195.stdout:removed './283/37' 2022-06-27T11:53:09.531 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './283' 2022-06-27T11:53:09.532 INFO:tasks.workunit.client.0.smithi195.stdout:removed './284/21' 2022-06-27T11:53:09.532 INFO:tasks.workunit.client.0.smithi195.stdout:removed './284/27' 2022-06-27T11:53:09.533 INFO:tasks.workunit.client.0.smithi195.stdout:removed './284/31' 2022-06-27T11:53:09.533 INFO:tasks.workunit.client.0.smithi195.stdout:removed './284/36' 2022-06-27T11:53:09.534 INFO:tasks.workunit.client.0.smithi195.stdout:removed './284/20' 2022-06-27T11:53:09.534 INFO:tasks.workunit.client.0.smithi195.stdout:removed './284/39' 2022-06-27T11:53:09.535 INFO:tasks.workunit.client.0.smithi195.stdout:removed './284/34' 2022-06-27T11:53:09.535 INFO:tasks.workunit.client.0.smithi195.stdout:removed './284/38' 2022-06-27T11:53:09.535 INFO:tasks.workunit.client.0.smithi195.stdout:removed './284/28' 2022-06-27T11:53:09.536 INFO:tasks.workunit.client.0.smithi195.stdout:removed './284/24' 2022-06-27T11:53:09.536 INFO:tasks.workunit.client.0.smithi195.stdout:removed './284/29' 2022-06-27T11:53:09.537 INFO:tasks.workunit.client.0.smithi195.stdout:removed './284/32' 2022-06-27T11:53:09.537 INFO:tasks.workunit.client.0.smithi195.stdout:removed './284/22' 2022-06-27T11:53:09.538 INFO:tasks.workunit.client.0.smithi195.stdout:removed './284/25' 2022-06-27T11:53:09.538 INFO:tasks.workunit.client.0.smithi195.stdout:removed './284/30' 2022-06-27T11:53:09.539 INFO:tasks.workunit.client.0.smithi195.stdout:removed './284/33' 2022-06-27T11:53:09.539 INFO:tasks.workunit.client.0.smithi195.stdout:removed './284/23' 2022-06-27T11:53:09.540 INFO:tasks.workunit.client.0.smithi195.stdout:removed './284/35' 2022-06-27T11:53:09.540 INFO:tasks.workunit.client.0.smithi195.stdout:removed './284/26' 2022-06-27T11:53:09.540 INFO:tasks.workunit.client.0.smithi195.stdout:removed './284/37' 2022-06-27T11:53:09.541 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './284' 2022-06-27T11:53:09.541 INFO:tasks.workunit.client.0.smithi195.stdout:removed './285/21' 2022-06-27T11:53:09.542 INFO:tasks.workunit.client.0.smithi195.stdout:removed './285/27' 2022-06-27T11:53:09.542 INFO:tasks.workunit.client.0.smithi195.stdout:removed './285/31' 2022-06-27T11:53:09.543 INFO:tasks.workunit.client.0.smithi195.stdout:removed './285/36' 2022-06-27T11:53:09.543 INFO:tasks.workunit.client.0.smithi195.stdout:removed './285/20' 2022-06-27T11:53:09.544 INFO:tasks.workunit.client.0.smithi195.stdout:removed './285/39' 2022-06-27T11:53:09.544 INFO:tasks.workunit.client.0.smithi195.stdout:removed './285/34' 2022-06-27T11:53:09.545 INFO:tasks.workunit.client.0.smithi195.stdout:removed './285/38' 2022-06-27T11:53:09.545 INFO:tasks.workunit.client.0.smithi195.stdout:removed './285/28' 2022-06-27T11:53:09.546 INFO:tasks.workunit.client.0.smithi195.stdout:removed './285/24' 2022-06-27T11:53:09.546 INFO:tasks.workunit.client.0.smithi195.stdout:removed './285/29' 2022-06-27T11:53:09.546 INFO:tasks.workunit.client.0.smithi195.stdout:removed './285/32' 2022-06-27T11:53:09.547 INFO:tasks.workunit.client.0.smithi195.stdout:removed './285/22' 2022-06-27T11:53:09.547 INFO:tasks.workunit.client.0.smithi195.stdout:removed './285/25' 2022-06-27T11:53:09.548 INFO:tasks.workunit.client.0.smithi195.stdout:removed './285/30' 2022-06-27T11:53:09.549 INFO:tasks.workunit.client.0.smithi195.stdout:removed './285/33' 2022-06-27T11:53:09.549 INFO:tasks.workunit.client.0.smithi195.stdout:removed './285/23' 2022-06-27T11:53:09.549 INFO:tasks.workunit.client.0.smithi195.stdout:removed './285/35' 2022-06-27T11:53:09.550 INFO:tasks.workunit.client.0.smithi195.stdout:removed './285/26' 2022-06-27T11:53:09.550 INFO:tasks.workunit.client.0.smithi195.stdout:removed './285/37' 2022-06-27T11:53:09.551 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './285' 2022-06-27T11:53:09.552 INFO:tasks.workunit.client.0.smithi195.stdout:removed './286/21' 2022-06-27T11:53:09.552 INFO:tasks.workunit.client.0.smithi195.stdout:removed './286/27' 2022-06-27T11:53:09.552 INFO:tasks.workunit.client.0.smithi195.stdout:removed './286/31' 2022-06-27T11:53:09.553 INFO:tasks.workunit.client.0.smithi195.stdout:removed './286/36' 2022-06-27T11:53:09.553 INFO:tasks.workunit.client.0.smithi195.stdout:removed './286/20' 2022-06-27T11:53:09.553 INFO:tasks.workunit.client.0.smithi195.stdout:removed './286/39' 2022-06-27T11:53:09.553 INFO:tasks.workunit.client.0.smithi195.stdout:removed './286/34' 2022-06-27T11:53:09.553 INFO:tasks.workunit.client.0.smithi195.stdout:removed './286/38' 2022-06-27T11:53:09.554 INFO:tasks.workunit.client.0.smithi195.stdout:removed './286/28' 2022-06-27T11:53:09.554 INFO:tasks.workunit.client.0.smithi195.stdout:removed './286/24' 2022-06-27T11:53:09.555 INFO:tasks.workunit.client.0.smithi195.stdout:removed './286/29' 2022-06-27T11:53:09.555 INFO:tasks.workunit.client.0.smithi195.stdout:removed './286/32' 2022-06-27T11:53:09.555 INFO:tasks.workunit.client.0.smithi195.stdout:removed './286/22' 2022-06-27T11:53:09.556 INFO:tasks.workunit.client.0.smithi195.stdout:removed './286/25' 2022-06-27T11:53:09.556 INFO:tasks.workunit.client.0.smithi195.stdout:removed './286/30' 2022-06-27T11:53:09.557 INFO:tasks.workunit.client.0.smithi195.stdout:removed './286/33' 2022-06-27T11:53:09.557 INFO:tasks.workunit.client.0.smithi195.stdout:removed './286/23' 2022-06-27T11:53:09.558 INFO:tasks.workunit.client.0.smithi195.stdout:removed './286/35' 2022-06-27T11:53:09.558 INFO:tasks.workunit.client.0.smithi195.stdout:removed './286/26' 2022-06-27T11:53:09.558 INFO:tasks.workunit.client.0.smithi195.stdout:removed './286/37' 2022-06-27T11:53:09.559 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './286' 2022-06-27T11:53:12.070 INFO:tasks.workunit.client.0.smithi195.stdout:removed './287/21' 2022-06-27T11:53:12.079 INFO:tasks.workunit.client.0.smithi195.stdout:removed './287/27' 2022-06-27T11:53:12.080 INFO:tasks.workunit.client.0.smithi195.stdout:removed './287/31' 2022-06-27T11:53:12.080 INFO:tasks.workunit.client.0.smithi195.stdout:removed './287/36' 2022-06-27T11:53:12.080 INFO:tasks.workunit.client.0.smithi195.stdout:removed './287/20' 2022-06-27T11:53:12.081 INFO:tasks.workunit.client.0.smithi195.stdout:removed './287/39' 2022-06-27T11:53:12.081 INFO:tasks.workunit.client.0.smithi195.stdout:removed './287/34' 2022-06-27T11:53:12.081 INFO:tasks.workunit.client.0.smithi195.stdout:removed './287/38' 2022-06-27T11:53:12.081 INFO:tasks.workunit.client.0.smithi195.stdout:removed './287/28' 2022-06-27T11:53:12.082 INFO:tasks.workunit.client.0.smithi195.stdout:removed './287/24' 2022-06-27T11:53:12.082 INFO:tasks.workunit.client.0.smithi195.stdout:removed './287/29' 2022-06-27T11:53:12.082 INFO:tasks.workunit.client.0.smithi195.stdout:removed './287/32' 2022-06-27T11:53:12.083 INFO:tasks.workunit.client.0.smithi195.stdout:removed './287/22' 2022-06-27T11:53:12.083 INFO:tasks.workunit.client.0.smithi195.stdout:removed './287/25' 2022-06-27T11:53:12.083 INFO:tasks.workunit.client.0.smithi195.stdout:removed './287/30' 2022-06-27T11:53:12.083 INFO:tasks.workunit.client.0.smithi195.stdout:removed './287/33' 2022-06-27T11:53:12.083 INFO:tasks.workunit.client.0.smithi195.stdout:removed './287/23' 2022-06-27T11:53:12.084 INFO:tasks.workunit.client.0.smithi195.stdout:removed './287/35' 2022-06-27T11:53:12.084 INFO:tasks.workunit.client.0.smithi195.stdout:removed './287/26' 2022-06-27T11:53:12.084 INFO:tasks.workunit.client.0.smithi195.stdout:removed './287/37' 2022-06-27T11:53:12.084 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './287' 2022-06-27T11:53:12.085 INFO:tasks.workunit.client.0.smithi195.stdout:removed './288/21' 2022-06-27T11:53:12.085 INFO:tasks.workunit.client.0.smithi195.stdout:removed './288/27' 2022-06-27T11:53:12.085 INFO:tasks.workunit.client.0.smithi195.stdout:removed './288/31' 2022-06-27T11:53:12.085 INFO:tasks.workunit.client.0.smithi195.stdout:removed './288/36' 2022-06-27T11:53:12.085 INFO:tasks.workunit.client.0.smithi195.stdout:removed './288/20' 2022-06-27T11:53:12.086 INFO:tasks.workunit.client.0.smithi195.stdout:removed './288/39' 2022-06-27T11:53:12.086 INFO:tasks.workunit.client.0.smithi195.stdout:removed './288/34' 2022-06-27T11:53:12.086 INFO:tasks.workunit.client.0.smithi195.stdout:removed './288/38' 2022-06-27T11:53:12.086 INFO:tasks.workunit.client.0.smithi195.stdout:removed './288/28' 2022-06-27T11:53:12.087 INFO:tasks.workunit.client.0.smithi195.stdout:removed './288/24' 2022-06-27T11:53:12.087 INFO:tasks.workunit.client.0.smithi195.stdout:removed './288/29' 2022-06-27T11:53:12.087 INFO:tasks.workunit.client.0.smithi195.stdout:removed './288/32' 2022-06-27T11:53:12.087 INFO:tasks.workunit.client.0.smithi195.stdout:removed './288/22' 2022-06-27T11:53:12.088 INFO:tasks.workunit.client.0.smithi195.stdout:removed './288/25' 2022-06-27T11:53:12.088 INFO:tasks.workunit.client.0.smithi195.stdout:removed './288/30' 2022-06-27T11:53:12.088 INFO:tasks.workunit.client.0.smithi195.stdout:removed './288/33' 2022-06-27T11:53:12.088 INFO:tasks.workunit.client.0.smithi195.stdout:removed './288/23' 2022-06-27T11:53:12.089 INFO:tasks.workunit.client.0.smithi195.stdout:removed './288/35' 2022-06-27T11:53:12.089 INFO:tasks.workunit.client.0.smithi195.stdout:removed './288/26' 2022-06-27T11:53:12.089 INFO:tasks.workunit.client.0.smithi195.stdout:removed './288/37' 2022-06-27T11:53:12.089 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './288' 2022-06-27T11:53:12.090 INFO:tasks.workunit.client.0.smithi195.stdout:removed './289/21' 2022-06-27T11:53:12.092 INFO:tasks.workunit.client.0.smithi195.stdout:removed './289/27' 2022-06-27T11:53:12.092 INFO:tasks.workunit.client.0.smithi195.stdout:removed './289/31' 2022-06-27T11:53:12.092 INFO:tasks.workunit.client.0.smithi195.stdout:removed './289/36' 2022-06-27T11:53:12.092 INFO:tasks.workunit.client.0.smithi195.stdout:removed './289/20' 2022-06-27T11:53:12.093 INFO:tasks.workunit.client.0.smithi195.stdout:removed './289/39' 2022-06-27T11:53:12.093 INFO:tasks.workunit.client.0.smithi195.stdout:removed './289/34' 2022-06-27T11:53:12.093 INFO:tasks.workunit.client.0.smithi195.stdout:removed './289/38' 2022-06-27T11:53:12.093 INFO:tasks.workunit.client.0.smithi195.stdout:removed './289/28' 2022-06-27T11:53:12.094 INFO:tasks.workunit.client.0.smithi195.stdout:removed './289/24' 2022-06-27T11:53:12.094 INFO:tasks.workunit.client.0.smithi195.stdout:removed './289/29' 2022-06-27T11:53:12.094 INFO:tasks.workunit.client.0.smithi195.stdout:removed './289/32' 2022-06-27T11:53:12.094 INFO:tasks.workunit.client.0.smithi195.stdout:removed './289/22' 2022-06-27T11:53:12.095 INFO:tasks.workunit.client.0.smithi195.stdout:removed './289/25' 2022-06-27T11:53:12.095 INFO:tasks.workunit.client.0.smithi195.stdout:removed './289/30' 2022-06-27T11:53:12.095 INFO:tasks.workunit.client.0.smithi195.stdout:removed './289/33' 2022-06-27T11:53:12.095 INFO:tasks.workunit.client.0.smithi195.stdout:removed './289/23' 2022-06-27T11:53:12.096 INFO:tasks.workunit.client.0.smithi195.stdout:removed './289/35' 2022-06-27T11:53:12.096 INFO:tasks.workunit.client.0.smithi195.stdout:removed './289/26' 2022-06-27T11:53:12.096 INFO:tasks.workunit.client.0.smithi195.stdout:removed './289/37' 2022-06-27T11:53:12.096 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './289' 2022-06-27T11:53:12.096 INFO:tasks.workunit.client.0.smithi195.stdout:removed './290/21' 2022-06-27T11:53:12.097 INFO:tasks.workunit.client.0.smithi195.stdout:removed './290/27' 2022-06-27T11:53:12.097 INFO:tasks.workunit.client.0.smithi195.stdout:removed './290/31' 2022-06-27T11:53:12.097 INFO:tasks.workunit.client.0.smithi195.stdout:removed './290/36' 2022-06-27T11:53:12.097 INFO:tasks.workunit.client.0.smithi195.stdout:removed './290/20' 2022-06-27T11:53:12.098 INFO:tasks.workunit.client.0.smithi195.stdout:removed './290/39' 2022-06-27T11:53:12.098 INFO:tasks.workunit.client.0.smithi195.stdout:removed './290/34' 2022-06-27T11:53:12.098 INFO:tasks.workunit.client.0.smithi195.stdout:removed './290/38' 2022-06-27T11:53:12.099 INFO:tasks.workunit.client.0.smithi195.stdout:removed './290/28' 2022-06-27T11:53:12.099 INFO:tasks.workunit.client.0.smithi195.stdout:removed './290/24' 2022-06-27T11:53:12.099 INFO:tasks.workunit.client.0.smithi195.stdout:removed './290/29' 2022-06-27T11:53:12.099 INFO:tasks.workunit.client.0.smithi195.stdout:removed './290/32' 2022-06-27T11:53:12.100 INFO:tasks.workunit.client.0.smithi195.stdout:removed './290/22' 2022-06-27T11:53:12.100 INFO:tasks.workunit.client.0.smithi195.stdout:removed './290/25' 2022-06-27T11:53:12.101 INFO:tasks.workunit.client.0.smithi195.stdout:removed './290/30' 2022-06-27T11:53:12.101 INFO:tasks.workunit.client.0.smithi195.stdout:removed './290/33' 2022-06-27T11:53:12.101 INFO:tasks.workunit.client.0.smithi195.stdout:removed './290/23' 2022-06-27T11:53:12.101 INFO:tasks.workunit.client.0.smithi195.stdout:removed './290/35' 2022-06-27T11:53:12.102 INFO:tasks.workunit.client.0.smithi195.stdout:removed './290/26' 2022-06-27T11:53:12.103 INFO:tasks.workunit.client.0.smithi195.stdout:removed './290/37' 2022-06-27T11:53:12.103 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './290' 2022-06-27T11:53:12.103 INFO:tasks.workunit.client.0.smithi195.stdout:removed './291/21' 2022-06-27T11:53:12.103 INFO:tasks.workunit.client.0.smithi195.stdout:removed './291/27' 2022-06-27T11:53:12.103 INFO:tasks.workunit.client.0.smithi195.stdout:removed './291/31' 2022-06-27T11:53:12.104 INFO:tasks.workunit.client.0.smithi195.stdout:removed './291/36' 2022-06-27T11:53:12.104 INFO:tasks.workunit.client.0.smithi195.stdout:removed './291/20' 2022-06-27T11:53:12.104 INFO:tasks.workunit.client.0.smithi195.stdout:removed './291/39' 2022-06-27T11:53:12.104 INFO:tasks.workunit.client.0.smithi195.stdout:removed './291/34' 2022-06-27T11:53:12.105 INFO:tasks.workunit.client.0.smithi195.stdout:removed './291/38' 2022-06-27T11:53:12.105 INFO:tasks.workunit.client.0.smithi195.stdout:removed './291/28' 2022-06-27T11:53:12.105 INFO:tasks.workunit.client.0.smithi195.stdout:removed './291/24' 2022-06-27T11:53:12.105 INFO:tasks.workunit.client.0.smithi195.stdout:removed './291/29' 2022-06-27T11:53:12.106 INFO:tasks.workunit.client.0.smithi195.stdout:removed './291/32' 2022-06-27T11:53:12.106 INFO:tasks.workunit.client.0.smithi195.stdout:removed './291/22' 2022-06-27T11:53:12.106 INFO:tasks.workunit.client.0.smithi195.stdout:removed './291/25' 2022-06-27T11:53:12.106 INFO:tasks.workunit.client.0.smithi195.stdout:removed './291/30' 2022-06-27T11:53:12.107 INFO:tasks.workunit.client.0.smithi195.stdout:removed './291/33' 2022-06-27T11:53:12.107 INFO:tasks.workunit.client.0.smithi195.stdout:removed './291/23' 2022-06-27T11:53:12.107 INFO:tasks.workunit.client.0.smithi195.stdout:removed './291/35' 2022-06-27T11:53:12.107 INFO:tasks.workunit.client.0.smithi195.stdout:removed './291/26' 2022-06-27T11:53:12.107 INFO:tasks.workunit.client.0.smithi195.stdout:removed './291/37' 2022-06-27T11:53:12.108 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './291' 2022-06-27T11:53:12.108 INFO:tasks.workunit.client.0.smithi195.stdout:removed './292/21' 2022-06-27T11:53:12.108 INFO:tasks.workunit.client.0.smithi195.stdout:removed './292/27' 2022-06-27T11:53:12.108 INFO:tasks.workunit.client.0.smithi195.stdout:removed './292/31' 2022-06-27T11:53:12.109 INFO:tasks.workunit.client.0.smithi195.stdout:removed './292/36' 2022-06-27T11:53:12.109 INFO:tasks.workunit.client.0.smithi195.stdout:removed './292/20' 2022-06-27T11:53:12.109 INFO:tasks.workunit.client.0.smithi195.stdout:removed './292/39' 2022-06-27T11:53:12.109 INFO:tasks.workunit.client.0.smithi195.stdout:removed './292/34' 2022-06-27T11:53:12.109 INFO:tasks.workunit.client.0.smithi195.stdout:removed './292/38' 2022-06-27T11:53:12.110 INFO:tasks.workunit.client.0.smithi195.stdout:removed './292/28' 2022-06-27T11:53:12.110 INFO:tasks.workunit.client.0.smithi195.stdout:removed './292/24' 2022-06-27T11:53:12.110 INFO:tasks.workunit.client.0.smithi195.stdout:removed './292/29' 2022-06-27T11:53:12.110 INFO:tasks.workunit.client.0.smithi195.stdout:removed './292/32' 2022-06-27T11:53:12.111 INFO:tasks.workunit.client.0.smithi195.stdout:removed './292/22' 2022-06-27T11:53:12.111 INFO:tasks.workunit.client.0.smithi195.stdout:removed './292/25' 2022-06-27T11:53:12.111 INFO:tasks.workunit.client.0.smithi195.stdout:removed './292/30' 2022-06-27T11:53:12.111 INFO:tasks.workunit.client.0.smithi195.stdout:removed './292/33' 2022-06-27T11:53:12.112 INFO:tasks.workunit.client.0.smithi195.stdout:removed './292/23' 2022-06-27T11:53:12.112 INFO:tasks.workunit.client.0.smithi195.stdout:removed './292/35' 2022-06-27T11:53:12.112 INFO:tasks.workunit.client.0.smithi195.stdout:removed './292/26' 2022-06-27T11:53:12.112 INFO:tasks.workunit.client.0.smithi195.stdout:removed './292/37' 2022-06-27T11:53:12.113 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './292' 2022-06-27T11:53:12.113 INFO:tasks.workunit.client.0.smithi195.stdout:removed './293/21' 2022-06-27T11:53:12.114 INFO:tasks.workunit.client.0.smithi195.stdout:removed './293/27' 2022-06-27T11:53:12.114 INFO:tasks.workunit.client.0.smithi195.stdout:removed './293/31' 2022-06-27T11:53:12.114 INFO:tasks.workunit.client.0.smithi195.stdout:removed './293/36' 2022-06-27T11:53:12.114 INFO:tasks.workunit.client.0.smithi195.stdout:removed './293/20' 2022-06-27T11:53:12.115 INFO:tasks.workunit.client.0.smithi195.stdout:removed './293/39' 2022-06-27T11:53:12.115 INFO:tasks.workunit.client.0.smithi195.stdout:removed './293/34' 2022-06-27T11:53:12.115 INFO:tasks.workunit.client.0.smithi195.stdout:removed './293/38' 2022-06-27T11:53:12.115 INFO:tasks.workunit.client.0.smithi195.stdout:removed './293/28' 2022-06-27T11:53:12.116 INFO:tasks.workunit.client.0.smithi195.stdout:removed './293/24' 2022-06-27T11:53:12.116 INFO:tasks.workunit.client.0.smithi195.stdout:removed './293/29' 2022-06-27T11:53:12.116 INFO:tasks.workunit.client.0.smithi195.stdout:removed './293/32' 2022-06-27T11:53:12.116 INFO:tasks.workunit.client.0.smithi195.stdout:removed './293/22' 2022-06-27T11:53:12.117 INFO:tasks.workunit.client.0.smithi195.stdout:removed './293/25' 2022-06-27T11:53:12.117 INFO:tasks.workunit.client.0.smithi195.stdout:removed './293/30' 2022-06-27T11:53:12.117 INFO:tasks.workunit.client.0.smithi195.stdout:removed './293/33' 2022-06-27T11:53:12.117 INFO:tasks.workunit.client.0.smithi195.stdout:removed './293/23' 2022-06-27T11:53:12.117 INFO:tasks.workunit.client.0.smithi195.stdout:removed './293/35' 2022-06-27T11:53:12.118 INFO:tasks.workunit.client.0.smithi195.stdout:removed './293/26' 2022-06-27T11:53:12.118 INFO:tasks.workunit.client.0.smithi195.stdout:removed './293/37' 2022-06-27T11:53:12.118 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './293' 2022-06-27T11:53:12.118 INFO:tasks.workunit.client.0.smithi195.stdout:removed './294/21' 2022-06-27T11:53:12.119 INFO:tasks.workunit.client.0.smithi195.stdout:removed './294/27' 2022-06-27T11:53:12.119 INFO:tasks.workunit.client.0.smithi195.stdout:removed './294/31' 2022-06-27T11:53:12.119 INFO:tasks.workunit.client.0.smithi195.stdout:removed './294/36' 2022-06-27T11:53:12.119 INFO:tasks.workunit.client.0.smithi195.stdout:removed './294/20' 2022-06-27T11:53:12.120 INFO:tasks.workunit.client.0.smithi195.stdout:removed './294/39' 2022-06-27T11:53:12.120 INFO:tasks.workunit.client.0.smithi195.stdout:removed './294/34' 2022-06-27T11:53:12.120 INFO:tasks.workunit.client.0.smithi195.stdout:removed './294/38' 2022-06-27T11:53:12.120 INFO:tasks.workunit.client.0.smithi195.stdout:removed './294/28' 2022-06-27T11:53:12.121 INFO:tasks.workunit.client.0.smithi195.stdout:removed './294/24' 2022-06-27T11:53:12.121 INFO:tasks.workunit.client.0.smithi195.stdout:removed './294/29' 2022-06-27T11:53:12.121 INFO:tasks.workunit.client.0.smithi195.stdout:removed './294/32' 2022-06-27T11:53:12.121 INFO:tasks.workunit.client.0.smithi195.stdout:removed './294/22' 2022-06-27T11:53:12.121 INFO:tasks.workunit.client.0.smithi195.stdout:removed './294/25' 2022-06-27T11:53:12.122 INFO:tasks.workunit.client.0.smithi195.stdout:removed './294/30' 2022-06-27T11:53:12.122 INFO:tasks.workunit.client.0.smithi195.stdout:removed './294/33' 2022-06-27T11:53:12.122 INFO:tasks.workunit.client.0.smithi195.stdout:removed './294/23' 2022-06-27T11:53:12.122 INFO:tasks.workunit.client.0.smithi195.stdout:removed './294/35' 2022-06-27T11:53:12.123 INFO:tasks.workunit.client.0.smithi195.stdout:removed './294/26' 2022-06-27T11:53:12.123 INFO:tasks.workunit.client.0.smithi195.stdout:removed './294/37' 2022-06-27T11:53:12.124 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './294' 2022-06-27T11:53:12.124 INFO:tasks.workunit.client.0.smithi195.stdout:removed './295/21' 2022-06-27T11:53:12.124 INFO:tasks.workunit.client.0.smithi195.stdout:removed './295/27' 2022-06-27T11:53:12.124 INFO:tasks.workunit.client.0.smithi195.stdout:removed './295/31' 2022-06-27T11:53:12.125 INFO:tasks.workunit.client.0.smithi195.stdout:removed './295/36' 2022-06-27T11:53:12.125 INFO:tasks.workunit.client.0.smithi195.stdout:removed './295/20' 2022-06-27T11:53:12.125 INFO:tasks.workunit.client.0.smithi195.stdout:removed './295/39' 2022-06-27T11:53:12.125 INFO:tasks.workunit.client.0.smithi195.stdout:removed './295/34' 2022-06-27T11:53:12.126 INFO:tasks.workunit.client.0.smithi195.stdout:removed './295/38' 2022-06-27T11:53:12.126 INFO:tasks.workunit.client.0.smithi195.stdout:removed './295/28' 2022-06-27T11:53:12.126 INFO:tasks.workunit.client.0.smithi195.stdout:removed './295/24' 2022-06-27T11:53:12.127 INFO:tasks.workunit.client.0.smithi195.stdout:removed './295/29' 2022-06-27T11:53:12.127 INFO:tasks.workunit.client.0.smithi195.stdout:removed './295/32' 2022-06-27T11:53:12.127 INFO:tasks.workunit.client.0.smithi195.stdout:removed './295/22' 2022-06-27T11:53:12.127 INFO:tasks.workunit.client.0.smithi195.stdout:removed './295/25' 2022-06-27T11:53:12.128 INFO:tasks.workunit.client.0.smithi195.stdout:removed './295/30' 2022-06-27T11:53:12.128 INFO:tasks.workunit.client.0.smithi195.stdout:removed './295/33' 2022-06-27T11:53:12.128 INFO:tasks.workunit.client.0.smithi195.stdout:removed './295/23' 2022-06-27T11:53:12.128 INFO:tasks.workunit.client.0.smithi195.stdout:removed './295/35' 2022-06-27T11:53:12.129 INFO:tasks.workunit.client.0.smithi195.stdout:removed './295/26' 2022-06-27T11:53:12.129 INFO:tasks.workunit.client.0.smithi195.stdout:removed './295/37' 2022-06-27T11:53:12.129 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './295' 2022-06-27T11:53:12.129 INFO:tasks.workunit.client.0.smithi195.stdout:removed './296/21' 2022-06-27T11:53:12.130 INFO:tasks.workunit.client.0.smithi195.stdout:removed './296/27' 2022-06-27T11:53:12.130 INFO:tasks.workunit.client.0.smithi195.stdout:removed './296/31' 2022-06-27T11:53:12.130 INFO:tasks.workunit.client.0.smithi195.stdout:removed './296/36' 2022-06-27T11:53:12.130 INFO:tasks.workunit.client.0.smithi195.stdout:removed './296/20' 2022-06-27T11:53:12.131 INFO:tasks.workunit.client.0.smithi195.stdout:removed './296/39' 2022-06-27T11:53:12.131 INFO:tasks.workunit.client.0.smithi195.stdout:removed './296/34' 2022-06-27T11:53:12.131 INFO:tasks.workunit.client.0.smithi195.stdout:removed './296/38' 2022-06-27T11:53:12.131 INFO:tasks.workunit.client.0.smithi195.stdout:removed './296/28' 2022-06-27T11:53:12.132 INFO:tasks.workunit.client.0.smithi195.stdout:removed './296/24' 2022-06-27T11:53:12.132 INFO:tasks.workunit.client.0.smithi195.stdout:removed './296/29' 2022-06-27T11:53:12.132 INFO:tasks.workunit.client.0.smithi195.stdout:removed './296/32' 2022-06-27T11:53:12.132 INFO:tasks.workunit.client.0.smithi195.stdout:removed './296/22' 2022-06-27T11:53:12.132 INFO:tasks.workunit.client.0.smithi195.stdout:removed './296/25' 2022-06-27T11:53:12.133 INFO:tasks.workunit.client.0.smithi195.stdout:removed './296/30' 2022-06-27T11:53:12.133 INFO:tasks.workunit.client.0.smithi195.stdout:removed './296/33' 2022-06-27T11:53:12.133 INFO:tasks.workunit.client.0.smithi195.stdout:removed './296/23' 2022-06-27T11:53:12.134 INFO:tasks.workunit.client.0.smithi195.stdout:removed './296/35' 2022-06-27T11:53:12.135 INFO:tasks.workunit.client.0.smithi195.stdout:removed './296/26' 2022-06-27T11:53:12.135 INFO:tasks.workunit.client.0.smithi195.stdout:removed './296/37' 2022-06-27T11:53:12.136 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './296' 2022-06-27T11:53:12.136 INFO:tasks.workunit.client.0.smithi195.stdout:removed './297/21' 2022-06-27T11:53:12.137 INFO:tasks.workunit.client.0.smithi195.stdout:removed './297/27' 2022-06-27T11:53:12.894 INFO:tasks.workunit.client.0.smithi195.stdout:removed './297/31' 2022-06-27T11:53:12.894 INFO:tasks.workunit.client.0.smithi195.stdout:removed './297/36' 2022-06-27T11:53:12.895 INFO:tasks.workunit.client.0.smithi195.stdout:removed './297/20' 2022-06-27T11:53:12.895 INFO:tasks.workunit.client.0.smithi195.stdout:removed './297/39' 2022-06-27T11:53:12.895 INFO:tasks.workunit.client.0.smithi195.stdout:removed './297/34' 2022-06-27T11:53:12.895 INFO:tasks.workunit.client.0.smithi195.stdout:removed './297/38' 2022-06-27T11:53:12.896 INFO:tasks.workunit.client.0.smithi195.stdout:removed './297/28' 2022-06-27T11:53:12.896 INFO:tasks.workunit.client.0.smithi195.stdout:removed './297/24' 2022-06-27T11:53:12.896 INFO:tasks.workunit.client.0.smithi195.stdout:removed './297/29' 2022-06-27T11:53:12.897 INFO:tasks.workunit.client.0.smithi195.stdout:removed './297/32' 2022-06-27T11:53:12.897 INFO:tasks.workunit.client.0.smithi195.stdout:removed './297/22' 2022-06-27T11:53:12.897 INFO:tasks.workunit.client.0.smithi195.stdout:removed './297/25' 2022-06-27T11:53:12.897 INFO:tasks.workunit.client.0.smithi195.stdout:removed './297/30' 2022-06-27T11:53:12.898 INFO:tasks.workunit.client.0.smithi195.stdout:removed './297/33' 2022-06-27T11:53:12.898 INFO:tasks.workunit.client.0.smithi195.stdout:removed './297/23' 2022-06-27T11:53:12.898 INFO:tasks.workunit.client.0.smithi195.stdout:removed './297/35' 2022-06-27T11:53:12.898 INFO:tasks.workunit.client.0.smithi195.stdout:removed './297/26' 2022-06-27T11:53:12.899 INFO:tasks.workunit.client.0.smithi195.stdout:removed './297/37' 2022-06-27T11:53:12.899 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './297' 2022-06-27T11:53:12.899 INFO:tasks.workunit.client.0.smithi195.stdout:removed './298/21' 2022-06-27T11:53:12.899 INFO:tasks.workunit.client.0.smithi195.stdout:removed './298/27' 2022-06-27T11:53:12.899 INFO:tasks.workunit.client.0.smithi195.stdout:removed './298/31' 2022-06-27T11:53:12.900 INFO:tasks.workunit.client.0.smithi195.stdout:removed './298/36' 2022-06-27T11:53:12.900 INFO:tasks.workunit.client.0.smithi195.stdout:removed './298/20' 2022-06-27T11:53:12.900 INFO:tasks.workunit.client.0.smithi195.stdout:removed './298/39' 2022-06-27T11:53:12.900 INFO:tasks.workunit.client.0.smithi195.stdout:removed './298/34' 2022-06-27T11:53:12.901 INFO:tasks.workunit.client.0.smithi195.stdout:removed './298/38' 2022-06-27T11:53:12.901 INFO:tasks.workunit.client.0.smithi195.stdout:removed './298/28' 2022-06-27T11:53:12.901 INFO:tasks.workunit.client.0.smithi195.stdout:removed './298/24' 2022-06-27T11:53:12.901 INFO:tasks.workunit.client.0.smithi195.stdout:removed './298/29' 2022-06-27T11:53:12.901 INFO:tasks.workunit.client.0.smithi195.stdout:removed './298/32' 2022-06-27T11:53:12.902 INFO:tasks.workunit.client.0.smithi195.stdout:removed './298/22' 2022-06-27T11:53:12.902 INFO:tasks.workunit.client.0.smithi195.stdout:removed './298/25' 2022-06-27T11:53:12.902 INFO:tasks.workunit.client.0.smithi195.stdout:removed './298/30' 2022-06-27T11:53:12.902 INFO:tasks.workunit.client.0.smithi195.stdout:removed './298/33' 2022-06-27T11:53:12.903 INFO:tasks.workunit.client.0.smithi195.stdout:removed './298/23' 2022-06-27T11:53:12.903 INFO:tasks.workunit.client.0.smithi195.stdout:removed './298/35' 2022-06-27T11:53:12.904 INFO:tasks.workunit.client.0.smithi195.stdout:removed './298/26' 2022-06-27T11:53:12.904 INFO:tasks.workunit.client.0.smithi195.stdout:removed './298/37' 2022-06-27T11:53:12.904 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './298' 2022-06-27T11:53:12.904 INFO:tasks.workunit.client.0.smithi195.stdout:removed './299/21' 2022-06-27T11:53:12.905 INFO:tasks.workunit.client.0.smithi195.stdout:removed './299/27' 2022-06-27T11:53:12.905 INFO:tasks.workunit.client.0.smithi195.stdout:removed './299/31' 2022-06-27T11:53:12.905 INFO:tasks.workunit.client.0.smithi195.stdout:removed './299/36' 2022-06-27T11:53:12.905 INFO:tasks.workunit.client.0.smithi195.stdout:removed './299/20' 2022-06-27T11:53:12.906 INFO:tasks.workunit.client.0.smithi195.stdout:removed './299/39' 2022-06-27T11:53:12.906 INFO:tasks.workunit.client.0.smithi195.stdout:removed './299/34' 2022-06-27T11:53:12.906 INFO:tasks.workunit.client.0.smithi195.stdout:removed './299/38' 2022-06-27T11:53:12.907 INFO:tasks.workunit.client.0.smithi195.stdout:removed './299/28' 2022-06-27T11:53:12.907 INFO:tasks.workunit.client.0.smithi195.stdout:removed './299/24' 2022-06-27T11:53:12.907 INFO:tasks.workunit.client.0.smithi195.stdout:removed './299/29' 2022-06-27T11:53:12.907 INFO:tasks.workunit.client.0.smithi195.stdout:removed './299/32' 2022-06-27T11:53:12.908 INFO:tasks.workunit.client.0.smithi195.stdout:removed './299/22' 2022-06-27T11:53:12.908 INFO:tasks.workunit.client.0.smithi195.stdout:removed './299/25' 2022-06-27T11:53:12.908 INFO:tasks.workunit.client.0.smithi195.stdout:removed './299/30' 2022-06-27T11:53:12.909 INFO:tasks.workunit.client.0.smithi195.stdout:removed './299/33' 2022-06-27T11:53:12.909 INFO:tasks.workunit.client.0.smithi195.stdout:removed './299/23' 2022-06-27T11:53:12.909 INFO:tasks.workunit.client.0.smithi195.stdout:removed './299/35' 2022-06-27T11:53:12.909 INFO:tasks.workunit.client.0.smithi195.stdout:removed './299/26' 2022-06-27T11:53:12.910 INFO:tasks.workunit.client.0.smithi195.stdout:removed './299/37' 2022-06-27T11:53:12.910 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './299' 2022-06-27T11:53:12.918 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/166/19' 2022-06-27T11:53:12.918 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/166/15' 2022-06-27T11:53:12.918 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/166/20' 2022-06-27T11:53:12.918 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/166/12' 2022-06-27T11:53:12.919 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/166/10' 2022-06-27T11:53:12.919 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/166/11' 2022-06-27T11:53:12.919 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/166/16' 2022-06-27T11:53:12.919 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/166/14' 2022-06-27T11:53:12.920 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/166/18' 2022-06-27T11:53:12.920 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/166/13' 2022-06-27T11:53:12.920 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/166/17' 2022-06-27T11:53:12.920 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/166' 2022-06-27T11:53:12.921 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/147/19' 2022-06-27T11:53:12.921 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/147/15' 2022-06-27T11:53:12.921 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/147/20' 2022-06-27T11:53:12.922 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/147/12' 2022-06-27T11:53:12.922 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/147/10' 2022-06-27T11:53:12.922 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/147/11' 2022-06-27T11:53:12.922 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/147/16' 2022-06-27T11:53:12.923 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/147/14' 2022-06-27T11:53:12.923 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/147/18' 2022-06-27T11:53:12.923 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/147/13' 2022-06-27T11:53:12.923 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/147/17' 2022-06-27T11:53:12.923 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/147' 2022-06-27T11:53:12.924 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/151/19' 2022-06-27T11:53:12.924 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/151/15' 2022-06-27T11:53:12.925 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/151/20' 2022-06-27T11:53:12.925 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/151/12' 2022-06-27T11:53:12.925 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/151/10' 2022-06-27T11:53:12.925 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/151/11' 2022-06-27T11:53:12.926 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/151/16' 2022-06-27T11:53:12.926 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/151/14' 2022-06-27T11:53:12.926 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/151/18' 2022-06-27T11:53:12.926 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/151/13' 2022-06-27T11:53:12.927 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/151/17' 2022-06-27T11:53:12.927 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/151' 2022-06-27T11:53:12.927 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/170/19' 2022-06-27T11:53:12.927 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/170/15' 2022-06-27T11:53:12.928 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/170/20' 2022-06-27T11:53:12.929 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/170/12' 2022-06-27T11:53:12.929 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/170/10' 2022-06-27T11:53:12.929 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/170/11' 2022-06-27T11:53:12.930 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/170/16' 2022-06-27T11:53:12.930 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/170/14' 2022-06-27T11:53:12.930 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/170/18' 2022-06-27T11:53:12.930 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/170/13' 2022-06-27T11:53:12.931 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/170/17' 2022-06-27T11:53:12.931 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/170' 2022-06-27T11:53:12.931 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/167/19' 2022-06-27T11:53:12.931 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/167/15' 2022-06-27T11:53:12.932 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/167/20' 2022-06-27T11:53:12.932 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/167/12' 2022-06-27T11:53:12.932 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/167/10' 2022-06-27T11:53:12.932 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/167/11' 2022-06-27T11:53:12.932 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/167/16' 2022-06-27T11:53:12.933 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/167/14' 2022-06-27T11:53:12.933 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/167/18' 2022-06-27T11:53:12.933 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/167/13' 2022-06-27T11:53:12.933 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/167/17' 2022-06-27T11:53:12.934 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/167' 2022-06-27T11:53:12.934 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/140/19' 2022-06-27T11:53:12.934 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/140/15' 2022-06-27T11:53:12.934 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/140/20' 2022-06-27T11:53:12.935 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/140/12' 2022-06-27T11:53:12.935 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/140/10' 2022-06-27T11:53:12.935 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/140/11' 2022-06-27T11:53:12.935 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/140/16' 2022-06-27T11:53:12.936 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/140/14' 2022-06-27T11:53:12.936 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/140/18' 2022-06-27T11:53:12.936 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/140/13' 2022-06-27T11:53:12.937 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/140/17' 2022-06-27T11:53:12.937 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/140' 2022-06-27T11:53:12.937 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/142/19' 2022-06-27T11:53:12.937 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/142/15' 2022-06-27T11:53:12.937 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/142/20' 2022-06-27T11:53:12.938 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/142/12' 2022-06-27T11:53:12.938 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/142/10' 2022-06-27T11:53:13.342 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/142/11' 2022-06-27T11:53:13.342 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/142/16' 2022-06-27T11:53:13.343 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/142/14' 2022-06-27T11:53:13.343 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/142/18' 2022-06-27T11:53:13.344 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/142/13' 2022-06-27T11:53:13.344 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/142/17' 2022-06-27T11:53:13.344 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/142' 2022-06-27T11:53:13.345 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/150/19' 2022-06-27T11:53:13.345 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/150/15' 2022-06-27T11:53:13.346 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/150/20' 2022-06-27T11:53:13.346 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/150/12' 2022-06-27T11:53:13.347 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/150/10' 2022-06-27T11:53:13.348 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/150/11' 2022-06-27T11:53:13.348 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/150/16' 2022-06-27T11:53:13.348 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/150/14' 2022-06-27T11:53:13.349 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/150/18' 2022-06-27T11:53:13.349 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/150/13' 2022-06-27T11:53:13.350 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/150/17' 2022-06-27T11:53:13.350 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/150' 2022-06-27T11:53:13.350 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/105/19' 2022-06-27T11:53:13.351 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/105/15' 2022-06-27T11:53:13.351 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/105/20' 2022-06-27T11:53:13.352 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/105/12' 2022-06-27T11:53:13.352 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/105/10' 2022-06-27T11:53:13.353 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/105/11' 2022-06-27T11:53:13.353 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/105/16' 2022-06-27T11:53:13.353 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/105/14' 2022-06-27T11:53:13.354 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/105/18' 2022-06-27T11:53:13.354 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/105/13' 2022-06-27T11:53:13.355 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/105/17' 2022-06-27T11:53:13.355 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/105' 2022-06-27T11:53:13.356 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/149/19' 2022-06-27T11:53:13.356 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/149/15' 2022-06-27T11:53:13.357 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/149/20' 2022-06-27T11:53:13.357 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/149/12' 2022-06-27T11:53:13.357 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/149/10' 2022-06-27T11:53:13.358 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/149/11' 2022-06-27T11:53:13.358 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/149/16' 2022-06-27T11:53:13.361 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/149/14' 2022-06-27T11:53:13.361 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/149/18' 2022-06-27T11:53:13.362 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/149/13' 2022-06-27T11:53:13.362 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/149/17' 2022-06-27T11:53:13.362 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/149' 2022-06-27T11:53:13.363 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/109/19' 2022-06-27T11:53:13.363 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/109/15' 2022-06-27T11:53:13.363 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/109/20' 2022-06-27T11:53:13.363 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/109/12' 2022-06-27T11:53:13.363 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/109/10' 2022-06-27T11:53:13.364 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/109/11' 2022-06-27T11:53:13.364 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/109/16' 2022-06-27T11:53:13.364 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/109/14' 2022-06-27T11:53:13.365 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/109/18' 2022-06-27T11:53:13.365 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/109/13' 2022-06-27T11:53:13.366 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/109/17' 2022-06-27T11:53:13.366 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/109' 2022-06-27T11:53:13.366 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/158/19' 2022-06-27T11:53:13.366 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/158/15' 2022-06-27T11:53:13.367 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/158/20' 2022-06-27T11:53:13.367 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/158/12' 2022-06-27T11:53:13.367 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/158/10' 2022-06-27T11:53:13.367 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/158/11' 2022-06-27T11:53:13.368 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/158/16' 2022-06-27T11:53:13.368 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/158/14' 2022-06-27T11:53:13.368 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/158/18' 2022-06-27T11:53:13.368 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/158/13' 2022-06-27T11:53:13.368 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/158/17' 2022-06-27T11:53:13.369 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/158' 2022-06-27T11:53:13.369 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/106/19' 2022-06-27T11:53:13.369 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/106/15' 2022-06-27T11:53:13.369 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/106/20' 2022-06-27T11:53:13.369 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/106/12' 2022-06-27T11:53:13.370 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/106/10' 2022-06-27T11:53:13.370 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/106/11' 2022-06-27T11:53:13.371 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/106/16' 2022-06-27T11:53:13.371 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/106/14' 2022-06-27T11:53:13.371 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/106/18' 2022-06-27T11:53:13.372 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/106/13' 2022-06-27T11:53:13.372 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/106/17' 2022-06-27T11:53:13.372 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/106' 2022-06-27T11:53:13.372 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/189/19' 2022-06-27T11:53:13.373 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/189/15' 2022-06-27T11:53:13.373 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/189/20' 2022-06-27T11:53:13.373 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/189/12' 2022-06-27T11:53:13.374 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/189/10' 2022-06-27T11:53:13.374 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/189/11' 2022-06-27T11:53:13.374 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/189/16' 2022-06-27T11:53:13.375 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/189/14' 2022-06-27T11:53:13.375 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/189/18' 2022-06-27T11:53:13.376 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/189/13' 2022-06-27T11:53:13.376 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/189/17' 2022-06-27T11:53:13.376 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/189' 2022-06-27T11:53:13.376 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/133/19' 2022-06-27T11:53:13.377 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/133/15' 2022-06-27T11:53:13.377 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/133/20' 2022-06-27T11:53:13.377 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/133/12' 2022-06-27T11:53:13.377 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/133/10' 2022-06-27T11:53:13.377 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/133/11' 2022-06-27T11:53:13.378 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/133/16' 2022-06-27T11:53:13.378 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/133/14' 2022-06-27T11:53:13.378 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/133/18' 2022-06-27T11:53:13.378 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/133/13' 2022-06-27T11:53:13.379 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/133/17' 2022-06-27T11:53:13.379 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/133' 2022-06-27T11:53:13.379 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/173/19' 2022-06-27T11:53:13.379 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/173/15' 2022-06-27T11:53:13.380 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/173/20' 2022-06-27T11:53:13.380 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/173/12' 2022-06-27T11:53:13.380 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/173/10' 2022-06-27T11:53:13.380 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/173/11' 2022-06-27T11:53:14.247 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/173/16' 2022-06-27T11:53:14.247 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/173/14' 2022-06-27T11:53:14.248 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/173/18' 2022-06-27T11:53:14.248 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/173/13' 2022-06-27T11:53:14.248 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/173/17' 2022-06-27T11:53:14.248 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/173' 2022-06-27T11:53:14.249 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/174/19' 2022-06-27T11:53:14.249 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/174/15' 2022-06-27T11:53:14.249 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/174/20' 2022-06-27T11:53:14.249 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/174/12' 2022-06-27T11:53:14.250 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/174/10' 2022-06-27T11:53:14.250 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/174/11' 2022-06-27T11:53:14.250 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/174/16' 2022-06-27T11:53:14.250 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/174/14' 2022-06-27T11:53:14.251 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/174/18' 2022-06-27T11:53:14.251 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/174/13' 2022-06-27T11:53:14.251 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/174/17' 2022-06-27T11:53:14.252 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/174' 2022-06-27T11:53:14.252 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/155/19' 2022-06-27T11:53:14.252 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/155/15' 2022-06-27T11:53:14.252 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/155/20' 2022-06-27T11:53:14.253 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/155/12' 2022-06-27T11:53:14.253 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/155/10' 2022-06-27T11:53:14.253 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/155/11' 2022-06-27T11:53:14.253 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/155/16' 2022-06-27T11:53:14.254 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/155/14' 2022-06-27T11:53:14.254 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/155/18' 2022-06-27T11:53:14.254 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/155/13' 2022-06-27T11:53:14.255 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/155/17' 2022-06-27T11:53:14.255 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/155' 2022-06-27T11:53:14.255 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/114/19' 2022-06-27T11:53:14.255 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/114/15' 2022-06-27T11:53:14.256 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/114/20' 2022-06-27T11:53:14.256 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/114/12' 2022-06-27T11:53:14.256 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/114/10' 2022-06-27T11:53:14.256 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/114/11' 2022-06-27T11:53:14.257 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/114/16' 2022-06-27T11:53:14.257 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/114/14' 2022-06-27T11:53:14.257 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/114/18' 2022-06-27T11:53:14.257 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/114/13' 2022-06-27T11:53:14.258 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/114/17' 2022-06-27T11:53:14.258 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/114' 2022-06-27T11:53:14.258 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/159/19' 2022-06-27T11:53:14.258 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/159/15' 2022-06-27T11:53:14.259 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/159/20' 2022-06-27T11:53:14.259 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/159/12' 2022-06-27T11:53:14.259 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/159/10' 2022-06-27T11:53:14.259 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/159/11' 2022-06-27T11:53:14.260 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/159/16' 2022-06-27T11:53:14.260 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/159/14' 2022-06-27T11:53:14.260 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/159/18' 2022-06-27T11:53:14.260 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/159/13' 2022-06-27T11:53:14.261 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/159/17' 2022-06-27T11:53:14.261 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/159' 2022-06-27T11:53:14.261 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/187/19' 2022-06-27T11:53:14.261 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/187/15' 2022-06-27T11:53:14.262 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/187/20' 2022-06-27T11:53:14.262 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/187/12' 2022-06-27T11:53:14.262 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/187/10' 2022-06-27T11:53:14.263 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/187/11' 2022-06-27T11:53:14.263 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/187/16' 2022-06-27T11:53:14.263 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/187/14' 2022-06-27T11:53:14.263 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/187/18' 2022-06-27T11:53:14.264 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/187/13' 2022-06-27T11:53:14.264 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/187/17' 2022-06-27T11:53:14.264 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/187' 2022-06-27T11:53:14.264 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/185/19' 2022-06-27T11:53:14.265 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/185/15' 2022-06-27T11:53:14.265 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/185/20' 2022-06-27T11:53:14.265 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/185/12' 2022-06-27T11:53:14.266 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/185/10' 2022-06-27T11:53:14.266 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/185/11' 2022-06-27T11:53:14.266 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/185/16' 2022-06-27T11:53:14.266 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/185/14' 2022-06-27T11:53:14.267 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/185/18' 2022-06-27T11:53:14.267 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/185/13' 2022-06-27T11:53:14.267 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/185/17' 2022-06-27T11:53:14.267 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/185' 2022-06-27T11:53:14.268 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/120/19' 2022-06-27T11:53:14.268 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/120/15' 2022-06-27T11:53:14.268 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/120/20' 2022-06-27T11:53:14.268 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/120/12' 2022-06-27T11:53:14.269 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/120/10' 2022-06-27T11:53:14.269 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/120/11' 2022-06-27T11:53:14.269 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/120/16' 2022-06-27T11:53:14.269 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/120/14' 2022-06-27T11:53:14.270 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/120/18' 2022-06-27T11:53:14.270 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/120/13' 2022-06-27T11:53:14.270 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/120/17' 2022-06-27T11:53:14.270 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/120' 2022-06-27T11:53:14.271 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/110/19' 2022-06-27T11:53:14.271 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/110/15' 2022-06-27T11:53:14.271 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/110/20' 2022-06-27T11:53:14.271 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/110/12' 2022-06-27T11:53:14.272 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/110/10' 2022-06-27T11:53:14.272 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/110/11' 2022-06-27T11:53:14.272 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/110/16' 2022-06-27T11:53:14.272 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/110/14' 2022-06-27T11:53:14.273 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/110/18' 2022-06-27T11:53:14.273 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/110/13' 2022-06-27T11:53:14.273 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/110/17' 2022-06-27T11:53:14.273 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/110' 2022-06-27T11:53:14.274 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/137/19' 2022-06-27T11:53:14.274 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/137/15' 2022-06-27T11:53:14.274 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/137/20' 2022-06-27T11:53:14.275 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/137/12' 2022-06-27T11:53:14.275 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/137/10' 2022-06-27T11:53:14.275 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/137/11' 2022-06-27T11:53:14.275 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/137/16' 2022-06-27T11:53:14.821 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/137/14' 2022-06-27T11:53:14.821 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/137/18' 2022-06-27T11:53:14.822 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/137/13' 2022-06-27T11:53:14.822 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/137/17' 2022-06-27T11:53:14.823 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/137' 2022-06-27T11:53:14.823 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/198/19' 2022-06-27T11:53:14.824 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/198/15' 2022-06-27T11:53:14.824 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/198/20' 2022-06-27T11:53:14.825 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/198/12' 2022-06-27T11:53:14.827 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/198/10' 2022-06-27T11:53:14.828 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/198/11' 2022-06-27T11:53:14.828 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/198/16' 2022-06-27T11:53:14.828 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/198/14' 2022-06-27T11:53:14.828 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/198/18' 2022-06-27T11:53:14.829 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/198/13' 2022-06-27T11:53:14.829 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/198/17' 2022-06-27T11:53:14.829 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/198' 2022-06-27T11:53:14.829 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/136/19' 2022-06-27T11:53:14.830 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/136/15' 2022-06-27T11:53:14.830 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/136/20' 2022-06-27T11:53:14.830 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/136/12' 2022-06-27T11:53:14.830 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/136/10' 2022-06-27T11:53:14.831 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/136/11' 2022-06-27T11:53:14.831 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/136/16' 2022-06-27T11:53:14.831 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/136/14' 2022-06-27T11:53:14.831 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/136/18' 2022-06-27T11:53:14.831 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/136/13' 2022-06-27T11:53:14.832 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/136/17' 2022-06-27T11:53:14.832 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/136' 2022-06-27T11:53:14.832 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/125/19' 2022-06-27T11:53:14.832 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/125/15' 2022-06-27T11:53:14.834 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/125/20' 2022-06-27T11:53:14.834 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/125/12' 2022-06-27T11:53:14.834 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/125/10' 2022-06-27T11:53:14.835 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/125/11' 2022-06-27T11:53:14.835 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/125/16' 2022-06-27T11:53:14.835 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/125/14' 2022-06-27T11:53:14.835 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/125/18' 2022-06-27T11:53:14.836 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/125/13' 2022-06-27T11:53:14.836 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/125/17' 2022-06-27T11:53:14.837 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/125' 2022-06-27T11:53:14.837 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/194/19' 2022-06-27T11:53:14.837 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/194/15' 2022-06-27T11:53:14.838 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/194/20' 2022-06-27T11:53:14.839 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/194/12' 2022-06-27T11:53:14.839 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/194/10' 2022-06-27T11:53:14.839 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/194/11' 2022-06-27T11:53:14.840 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/194/16' 2022-06-27T11:53:14.840 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/194/14' 2022-06-27T11:53:14.840 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/194/18' 2022-06-27T11:53:14.840 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/194/13' 2022-06-27T11:53:14.841 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/194/17' 2022-06-27T11:53:14.841 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/194' 2022-06-27T11:53:14.841 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/119/19' 2022-06-27T11:53:14.842 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/119/15' 2022-06-27T11:53:14.842 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/119/20' 2022-06-27T11:53:14.843 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/119/12' 2022-06-27T11:53:14.843 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/119/10' 2022-06-27T11:53:14.843 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/119/11' 2022-06-27T11:53:14.844 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/119/16' 2022-06-27T11:53:14.845 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/119/14' 2022-06-27T11:53:14.845 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/119/18' 2022-06-27T11:53:14.845 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/119/13' 2022-06-27T11:53:14.846 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/119/17' 2022-06-27T11:53:14.846 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/119' 2022-06-27T11:53:14.846 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/157/19' 2022-06-27T11:53:14.849 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/157/15' 2022-06-27T11:53:14.850 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/157/20' 2022-06-27T11:53:14.850 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/157/12' 2022-06-27T11:53:14.851 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/157/10' 2022-06-27T11:53:14.851 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/157/11' 2022-06-27T11:53:14.851 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/157/16' 2022-06-27T11:53:14.852 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/157/14' 2022-06-27T11:53:14.853 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/157/18' 2022-06-27T11:53:14.854 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/157/13' 2022-06-27T11:53:14.854 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/157/17' 2022-06-27T11:53:14.855 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/157' 2022-06-27T11:53:14.855 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/172/19' 2022-06-27T11:53:14.855 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/172/15' 2022-06-27T11:53:14.856 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/172/20' 2022-06-27T11:53:14.856 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/172/12' 2022-06-27T11:53:14.857 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/172/10' 2022-06-27T11:53:14.857 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/172/11' 2022-06-27T11:53:14.857 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/172/16' 2022-06-27T11:53:14.858 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/172/14' 2022-06-27T11:53:14.859 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/172/18' 2022-06-27T11:53:14.860 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/172/13' 2022-06-27T11:53:14.860 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/172/17' 2022-06-27T11:53:14.860 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/172' 2022-06-27T11:53:14.861 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/186/19' 2022-06-27T11:53:14.861 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/186/15' 2022-06-27T11:53:14.862 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/186/20' 2022-06-27T11:53:14.862 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/186/12' 2022-06-27T11:53:14.863 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/186/10' 2022-06-27T11:53:14.863 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/186/11' 2022-06-27T11:53:14.864 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/186/16' 2022-06-27T11:53:14.864 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/186/14' 2022-06-27T11:53:14.864 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/186/18' 2022-06-27T11:53:14.865 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/186/13' 2022-06-27T11:53:14.865 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/186/17' 2022-06-27T11:53:14.866 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/186' 2022-06-27T11:53:14.866 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/190/19' 2022-06-27T11:53:14.867 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/190/15' 2022-06-27T11:53:14.867 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/190/20' 2022-06-27T11:53:14.867 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/190/12' 2022-06-27T11:53:14.868 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/190/10' 2022-06-27T11:53:14.868 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/190/11' 2022-06-27T11:53:14.869 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/190/16' 2022-06-27T11:53:14.869 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/190/14' 2022-06-27T11:53:16.778 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/190/18' 2022-06-27T11:53:16.779 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/190/13' 2022-06-27T11:53:16.779 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/190/17' 2022-06-27T11:53:16.780 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/190' 2022-06-27T11:53:16.781 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/104/19' 2022-06-27T11:53:16.781 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/104/15' 2022-06-27T11:53:16.782 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/104/20' 2022-06-27T11:53:16.782 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/104/12' 2022-06-27T11:53:16.783 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/104/10' 2022-06-27T11:53:16.783 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/104/11' 2022-06-27T11:53:16.783 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/104/16' 2022-06-27T11:53:16.783 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/104/14' 2022-06-27T11:53:16.784 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/104/18' 2022-06-27T11:53:16.784 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/104/13' 2022-06-27T11:53:16.784 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/104/17' 2022-06-27T11:53:16.784 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/104' 2022-06-27T11:53:16.785 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/164/19' 2022-06-27T11:53:16.785 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/164/15' 2022-06-27T11:53:16.785 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/164/20' 2022-06-27T11:53:16.785 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/164/12' 2022-06-27T11:53:16.786 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/164/10' 2022-06-27T11:53:16.786 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/164/11' 2022-06-27T11:53:16.786 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/164/16' 2022-06-27T11:53:16.786 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/164/14' 2022-06-27T11:53:16.787 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/164/18' 2022-06-27T11:53:16.787 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/164/13' 2022-06-27T11:53:16.787 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/164/17' 2022-06-27T11:53:16.787 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/164' 2022-06-27T11:53:16.788 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/112/19' 2022-06-27T11:53:16.788 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/112/15' 2022-06-27T11:53:16.788 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/112/20' 2022-06-27T11:53:16.789 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/112/12' 2022-06-27T11:53:16.789 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/112/10' 2022-06-27T11:53:16.789 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/112/11' 2022-06-27T11:53:16.789 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/112/16' 2022-06-27T11:53:16.790 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/112/14' 2022-06-27T11:53:16.790 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/112/18' 2022-06-27T11:53:16.790 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/112/13' 2022-06-27T11:53:16.790 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/112/17' 2022-06-27T11:53:16.791 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/112' 2022-06-27T11:53:16.791 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/100/19' 2022-06-27T11:53:16.791 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/100/15' 2022-06-27T11:53:16.792 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/100/20' 2022-06-27T11:53:16.792 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/100/12' 2022-06-27T11:53:16.792 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/100/10' 2022-06-27T11:53:16.792 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/100/11' 2022-06-27T11:53:16.793 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/100/16' 2022-06-27T11:53:16.793 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/100/14' 2022-06-27T11:53:16.793 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/100/18' 2022-06-27T11:53:16.793 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/100/13' 2022-06-27T11:53:16.794 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/100/17' 2022-06-27T11:53:16.794 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/100' 2022-06-27T11:53:16.794 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/134/19' 2022-06-27T11:53:16.795 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/134/15' 2022-06-27T11:53:16.795 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/134/20' 2022-06-27T11:53:16.795 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/134/12' 2022-06-27T11:53:16.796 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/134/10' 2022-06-27T11:53:16.796 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/134/11' 2022-06-27T11:53:16.796 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/134/16' 2022-06-27T11:53:16.796 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/134/14' 2022-06-27T11:53:16.797 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/134/18' 2022-06-27T11:53:16.797 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/134/13' 2022-06-27T11:53:16.797 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/134/17' 2022-06-27T11:53:16.797 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/134' 2022-06-27T11:53:16.798 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/115/19' 2022-06-27T11:53:16.798 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/115/15' 2022-06-27T11:53:16.798 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/115/20' 2022-06-27T11:53:16.799 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/115/12' 2022-06-27T11:53:16.799 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/115/10' 2022-06-27T11:53:16.799 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/115/11' 2022-06-27T11:53:16.799 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/115/16' 2022-06-27T11:53:16.800 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/115/14' 2022-06-27T11:53:16.800 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/115/18' 2022-06-27T11:53:16.800 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/115/13' 2022-06-27T11:53:16.801 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/115/17' 2022-06-27T11:53:16.801 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/115' 2022-06-27T11:53:16.801 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/116/19' 2022-06-27T11:53:16.802 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/116/15' 2022-06-27T11:53:16.802 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/116/20' 2022-06-27T11:53:16.802 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/116/12' 2022-06-27T11:53:16.802 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/116/10' 2022-06-27T11:53:16.803 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/116/11' 2022-06-27T11:53:16.803 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/116/16' 2022-06-27T11:53:16.803 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/116/14' 2022-06-27T11:53:16.804 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/116/18' 2022-06-27T11:53:16.804 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/116/13' 2022-06-27T11:53:16.804 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/116/17' 2022-06-27T11:53:16.805 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/116' 2022-06-27T11:53:16.805 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/195/19' 2022-06-27T11:53:16.805 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/195/15' 2022-06-27T11:53:16.805 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/195/20' 2022-06-27T11:53:16.806 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/195/12' 2022-06-27T11:53:16.806 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/195/10' 2022-06-27T11:53:16.806 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/195/11' 2022-06-27T11:53:16.806 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/195/16' 2022-06-27T11:53:16.807 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/195/14' 2022-06-27T11:53:16.807 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/195/18' 2022-06-27T11:53:16.807 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/195/13' 2022-06-27T11:53:16.807 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/195/17' 2022-06-27T11:53:16.808 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/195' 2022-06-27T11:53:16.808 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/138/19' 2022-06-27T11:53:16.808 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/138/15' 2022-06-27T11:53:16.808 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/138/20' 2022-06-27T11:53:16.809 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/138/12' 2022-06-27T11:53:16.809 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/138/10' 2022-06-27T11:53:16.809 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/138/11' 2022-06-27T11:53:16.810 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/138/16' 2022-06-27T11:53:16.810 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/138/14' 2022-06-27T11:53:16.810 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/138/18' 2022-06-27T11:53:20.872 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/138/13' 2022-06-27T11:53:20.872 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/138/17' 2022-06-27T11:53:20.873 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/138' 2022-06-27T11:53:20.873 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/135/19' 2022-06-27T11:53:20.874 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/135/15' 2022-06-27T11:53:20.874 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/135/20' 2022-06-27T11:53:20.875 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/135/12' 2022-06-27T11:53:20.875 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/135/10' 2022-06-27T11:53:20.876 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/135/11' 2022-06-27T11:53:20.876 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/135/16' 2022-06-27T11:53:20.877 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/135/14' 2022-06-27T11:53:20.877 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/135/18' 2022-06-27T11:53:20.878 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/135/13' 2022-06-27T11:53:20.878 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/135/17' 2022-06-27T11:53:20.878 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/135' 2022-06-27T11:53:20.879 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/117/19' 2022-06-27T11:53:20.879 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/117/15' 2022-06-27T11:53:20.880 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/117/20' 2022-06-27T11:53:20.880 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/117/12' 2022-06-27T11:53:20.881 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/117/10' 2022-06-27T11:53:20.881 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/117/11' 2022-06-27T11:53:20.882 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/117/16' 2022-06-27T11:53:20.882 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/117/14' 2022-06-27T11:53:20.883 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/117/18' 2022-06-27T11:53:20.883 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/117/13' 2022-06-27T11:53:20.884 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/117/17' 2022-06-27T11:53:20.884 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/117' 2022-06-27T11:53:20.884 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/143/19' 2022-06-27T11:53:20.885 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/143/15' 2022-06-27T11:53:20.885 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/143/20' 2022-06-27T11:53:20.886 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/143/12' 2022-06-27T11:53:20.886 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/143/10' 2022-06-27T11:53:20.887 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/143/11' 2022-06-27T11:53:20.887 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/143/16' 2022-06-27T11:53:20.888 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/143/14' 2022-06-27T11:53:20.888 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/143/18' 2022-06-27T11:53:20.889 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/143/13' 2022-06-27T11:53:20.889 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/143/17' 2022-06-27T11:53:20.889 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/143' 2022-06-27T11:53:20.890 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/180/19' 2022-06-27T11:53:20.890 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/180/15' 2022-06-27T11:53:20.891 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/180/20' 2022-06-27T11:53:20.891 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/180/12' 2022-06-27T11:53:20.891 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/180/10' 2022-06-27T11:53:20.892 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/180/11' 2022-06-27T11:53:20.892 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/180/16' 2022-06-27T11:53:20.893 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/180/14' 2022-06-27T11:53:20.893 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/180/18' 2022-06-27T11:53:20.894 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/180/13' 2022-06-27T11:53:20.894 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/180/17' 2022-06-27T11:53:20.894 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/180' 2022-06-27T11:53:20.895 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/126/19' 2022-06-27T11:53:20.895 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/126/15' 2022-06-27T11:53:20.896 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/126/20' 2022-06-27T11:53:20.896 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/126/12' 2022-06-27T11:53:20.897 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/126/10' 2022-06-27T11:53:20.897 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/126/11' 2022-06-27T11:53:20.898 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/126/16' 2022-06-27T11:53:20.898 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/126/14' 2022-06-27T11:53:20.899 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/126/18' 2022-06-27T11:53:20.899 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/126/13' 2022-06-27T11:53:20.899 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/126/17' 2022-06-27T11:53:20.900 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/126' 2022-06-27T11:53:20.900 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/161/19' 2022-06-27T11:53:20.901 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/161/15' 2022-06-27T11:53:20.901 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/161/20' 2022-06-27T11:53:20.902 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/161/12' 2022-06-27T11:53:20.902 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/161/10' 2022-06-27T11:53:20.902 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/161/11' 2022-06-27T11:53:20.903 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/161/16' 2022-06-27T11:53:20.903 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/161/14' 2022-06-27T11:53:20.903 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/161/18' 2022-06-27T11:53:20.903 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/161/13' 2022-06-27T11:53:20.903 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/161/17' 2022-06-27T11:53:20.904 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/161' 2022-06-27T11:53:20.904 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/176/19' 2022-06-27T11:53:20.905 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/176/15' 2022-06-27T11:53:20.905 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/176/20' 2022-06-27T11:53:20.905 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/176/12' 2022-06-27T11:53:20.906 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/176/10' 2022-06-27T11:53:20.906 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/176/11' 2022-06-27T11:53:20.906 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/176/16' 2022-06-27T11:53:20.907 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/176/14' 2022-06-27T11:53:20.907 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/176/18' 2022-06-27T11:53:20.907 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/176/13' 2022-06-27T11:53:20.907 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/176/17' 2022-06-27T11:53:20.908 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/176' 2022-06-27T11:53:20.908 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/111/19' 2022-06-27T11:53:20.908 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/111/15' 2022-06-27T11:53:20.908 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/111/20' 2022-06-27T11:53:20.909 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/111/12' 2022-06-27T11:53:20.909 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/111/10' 2022-06-27T11:53:20.909 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/111/11' 2022-06-27T11:53:20.909 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/111/16' 2022-06-27T11:53:20.910 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/111/14' 2022-06-27T11:53:20.910 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/111/18' 2022-06-27T11:53:20.910 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/111/13' 2022-06-27T11:53:20.910 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/111/17' 2022-06-27T11:53:20.911 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/111' 2022-06-27T11:53:20.911 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/191/19' 2022-06-27T11:53:20.911 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/191/15' 2022-06-27T11:53:20.911 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/191/20' 2022-06-27T11:53:20.911 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/191/12' 2022-06-27T11:53:20.912 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/191/10' 2022-06-27T11:53:20.912 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/191/11' 2022-06-27T11:53:20.912 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/191/16' 2022-06-27T11:53:20.912 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/191/14' 2022-06-27T11:53:20.913 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/191/18' 2022-06-27T11:53:20.913 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/191/13' 2022-06-27T11:53:22.873 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/191/17' 2022-06-27T11:53:22.873 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/191' 2022-06-27T11:53:22.874 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/184/19' 2022-06-27T11:53:22.874 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/184/15' 2022-06-27T11:53:22.875 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/184/20' 2022-06-27T11:53:22.875 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/184/12' 2022-06-27T11:53:22.875 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/184/10' 2022-06-27T11:53:22.876 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/184/11' 2022-06-27T11:53:22.876 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/184/16' 2022-06-27T11:53:22.876 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/184/14' 2022-06-27T11:53:22.880 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/184/18' 2022-06-27T11:53:22.880 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/184/13' 2022-06-27T11:53:22.881 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/184/17' 2022-06-27T11:53:22.881 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/184' 2022-06-27T11:53:22.882 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/192/19' 2022-06-27T11:53:22.882 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/192/15' 2022-06-27T11:53:22.883 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/192/20' 2022-06-27T11:53:22.883 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/192/12' 2022-06-27T11:53:22.884 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/192/10' 2022-06-27T11:53:22.884 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/192/11' 2022-06-27T11:53:22.885 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/192/16' 2022-06-27T11:53:22.885 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/192/14' 2022-06-27T11:53:22.885 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/192/18' 2022-06-27T11:53:22.886 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/192/13' 2022-06-27T11:53:22.886 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/192/17' 2022-06-27T11:53:22.887 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/192' 2022-06-27T11:53:22.887 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/103/19' 2022-06-27T11:53:22.888 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/103/15' 2022-06-27T11:53:22.888 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/103/20' 2022-06-27T11:53:22.888 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/103/12' 2022-06-27T11:53:22.889 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/103/10' 2022-06-27T11:53:22.890 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/103/11' 2022-06-27T11:53:22.890 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/103/16' 2022-06-27T11:53:22.890 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/103/14' 2022-06-27T11:53:22.891 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/103/18' 2022-06-27T11:53:22.891 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/103/13' 2022-06-27T11:53:22.892 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/103/17' 2022-06-27T11:53:22.893 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/103' 2022-06-27T11:53:22.893 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/128/19' 2022-06-27T11:53:22.894 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/128/15' 2022-06-27T11:53:22.894 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/128/20' 2022-06-27T11:53:22.895 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/128/12' 2022-06-27T11:53:22.895 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/128/10' 2022-06-27T11:53:22.896 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/128/11' 2022-06-27T11:53:22.896 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/128/16' 2022-06-27T11:53:22.897 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/128/14' 2022-06-27T11:53:22.897 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/128/18' 2022-06-27T11:53:22.898 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/128/13' 2022-06-27T11:53:22.898 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/128/17' 2022-06-27T11:53:22.898 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/128' 2022-06-27T11:53:22.899 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/144/19' 2022-06-27T11:53:22.899 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/144/15' 2022-06-27T11:53:22.900 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/144/20' 2022-06-27T11:53:22.900 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/144/12' 2022-06-27T11:53:22.901 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/144/10' 2022-06-27T11:53:22.901 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/144/11' 2022-06-27T11:53:22.901 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/144/16' 2022-06-27T11:53:22.903 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/144/14' 2022-06-27T11:53:22.904 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/144/18' 2022-06-27T11:53:22.904 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/144/13' 2022-06-27T11:53:22.904 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/144/17' 2022-06-27T11:53:22.905 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/144' 2022-06-27T11:53:22.906 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/146/19' 2022-06-27T11:53:22.906 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/146/15' 2022-06-27T11:53:22.906 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/146/20' 2022-06-27T11:53:22.907 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/146/12' 2022-06-27T11:53:22.907 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/146/10' 2022-06-27T11:53:22.907 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/146/11' 2022-06-27T11:53:22.907 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/146/16' 2022-06-27T11:53:22.908 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/146/14' 2022-06-27T11:53:22.908 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/146/18' 2022-06-27T11:53:22.908 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/146/13' 2022-06-27T11:53:22.909 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/146/17' 2022-06-27T11:53:22.909 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/146' 2022-06-27T11:53:22.909 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/141/19' 2022-06-27T11:53:22.909 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/141/15' 2022-06-27T11:53:22.910 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/141/20' 2022-06-27T11:53:22.910 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/141/12' 2022-06-27T11:53:22.910 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/141/10' 2022-06-27T11:53:22.910 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/141/11' 2022-06-27T11:53:22.911 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/141/16' 2022-06-27T11:53:22.911 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/141/14' 2022-06-27T11:53:22.911 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/141/18' 2022-06-27T11:53:22.912 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/141/13' 2022-06-27T11:53:22.912 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/141/17' 2022-06-27T11:53:22.914 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/141' 2022-06-27T11:53:22.914 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/182/19' 2022-06-27T11:53:22.914 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/182/15' 2022-06-27T11:53:22.915 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/182/20' 2022-06-27T11:53:22.915 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/182/12' 2022-06-27T11:53:22.915 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/182/10' 2022-06-27T11:53:22.915 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/182/11' 2022-06-27T11:53:22.916 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/182/16' 2022-06-27T11:53:22.916 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/182/14' 2022-06-27T11:53:22.916 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/182/18' 2022-06-27T11:53:22.917 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/182/13' 2022-06-27T11:53:22.917 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/182/17' 2022-06-27T11:53:22.917 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/182' 2022-06-27T11:53:22.918 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/107/19' 2022-06-27T11:53:22.918 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/107/15' 2022-06-27T11:53:22.918 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/107/20' 2022-06-27T11:53:22.918 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/107/12' 2022-06-27T11:53:22.919 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/107/10' 2022-06-27T11:53:22.919 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/107/11' 2022-06-27T11:53:22.919 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/107/16' 2022-06-27T11:53:22.919 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/107/14' 2022-06-27T11:53:22.920 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/107/18' 2022-06-27T11:53:22.920 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/107/13' 2022-06-27T11:53:22.920 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/107/17' 2022-06-27T11:53:23.345 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/107' 2022-06-27T11:53:23.346 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/178/19' 2022-06-27T11:53:23.346 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/178/15' 2022-06-27T11:53:23.346 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/178/20' 2022-06-27T11:53:23.347 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/178/12' 2022-06-27T11:53:23.349 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/178/10' 2022-06-27T11:53:23.350 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/178/11' 2022-06-27T11:53:23.350 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/178/16' 2022-06-27T11:53:23.351 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/178/14' 2022-06-27T11:53:23.351 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/178/18' 2022-06-27T11:53:23.351 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/178/13' 2022-06-27T11:53:23.352 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/178/17' 2022-06-27T11:53:23.352 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/178' 2022-06-27T11:53:23.352 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/156/19' 2022-06-27T11:53:23.353 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/156/15' 2022-06-27T11:53:23.354 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/156/20' 2022-06-27T11:53:23.354 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/156/12' 2022-06-27T11:53:23.355 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/156/10' 2022-06-27T11:53:23.355 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/156/11' 2022-06-27T11:53:23.356 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/156/16' 2022-06-27T11:53:23.356 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/156/14' 2022-06-27T11:53:23.357 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/156/18' 2022-06-27T11:53:23.357 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/156/13' 2022-06-27T11:53:23.358 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/156/17' 2022-06-27T11:53:23.358 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/156' 2022-06-27T11:53:23.359 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/131/19' 2022-06-27T11:53:23.359 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/131/15' 2022-06-27T11:53:23.359 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/131/20' 2022-06-27T11:53:23.361 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/131/12' 2022-06-27T11:53:23.361 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/131/10' 2022-06-27T11:53:23.361 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/131/11' 2022-06-27T11:53:23.362 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/131/16' 2022-06-27T11:53:23.362 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/131/14' 2022-06-27T11:53:23.363 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/131/18' 2022-06-27T11:53:23.363 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/131/13' 2022-06-27T11:53:23.364 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/131/17' 2022-06-27T11:53:23.364 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/131' 2022-06-27T11:53:23.365 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/123/19' 2022-06-27T11:53:23.365 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/123/15' 2022-06-27T11:53:23.366 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/123/20' 2022-06-27T11:53:23.366 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/123/12' 2022-06-27T11:53:23.367 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/123/10' 2022-06-27T11:53:23.370 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/123/11' 2022-06-27T11:53:23.371 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/123/16' 2022-06-27T11:53:23.371 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/123/14' 2022-06-27T11:53:23.371 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/123/18' 2022-06-27T11:53:23.372 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/123/13' 2022-06-27T11:53:23.372 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/123/17' 2022-06-27T11:53:23.373 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/123' 2022-06-27T11:53:23.373 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/160/19' 2022-06-27T11:53:23.374 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/160/15' 2022-06-27T11:53:23.375 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/160/20' 2022-06-27T11:53:23.376 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/160/12' 2022-06-27T11:53:23.376 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/160/10' 2022-06-27T11:53:23.377 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/160/11' 2022-06-27T11:53:23.377 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/160/16' 2022-06-27T11:53:23.378 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/160/14' 2022-06-27T11:53:23.378 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/160/18' 2022-06-27T11:53:23.379 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/160/13' 2022-06-27T11:53:23.380 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/160/17' 2022-06-27T11:53:23.380 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/160' 2022-06-27T11:53:23.381 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/165/19' 2022-06-27T11:53:23.381 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/165/15' 2022-06-27T11:53:23.382 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/165/20' 2022-06-27T11:53:23.382 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/165/12' 2022-06-27T11:53:23.383 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/165/10' 2022-06-27T11:53:23.383 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/165/11' 2022-06-27T11:53:23.384 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/165/16' 2022-06-27T11:53:23.384 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/165/14' 2022-06-27T11:53:23.385 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/165/18' 2022-06-27T11:53:23.385 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/165/13' 2022-06-27T11:53:23.386 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/165/17' 2022-06-27T11:53:23.386 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/165' 2022-06-27T11:53:23.387 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/101/19' 2022-06-27T11:53:23.387 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/101/15' 2022-06-27T11:53:23.388 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/101/20' 2022-06-27T11:53:23.388 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/101/12' 2022-06-27T11:53:23.389 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/101/10' 2022-06-27T11:53:23.389 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/101/11' 2022-06-27T11:53:23.389 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/101/16' 2022-06-27T11:53:23.389 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/101/14' 2022-06-27T11:53:23.390 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/101/18' 2022-06-27T11:53:23.390 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/101/13' 2022-06-27T11:53:23.390 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/101/17' 2022-06-27T11:53:23.390 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/101' 2022-06-27T11:53:23.391 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/148/19' 2022-06-27T11:53:23.391 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/148/15' 2022-06-27T11:53:23.391 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/148/20' 2022-06-27T11:53:23.391 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/148/12' 2022-06-27T11:53:23.391 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/148/10' 2022-06-27T11:53:23.392 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/148/11' 2022-06-27T11:53:23.392 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/148/16' 2022-06-27T11:53:23.392 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/148/14' 2022-06-27T11:53:23.392 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/148/18' 2022-06-27T11:53:23.393 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/148/13' 2022-06-27T11:53:23.393 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/148/17' 2022-06-27T11:53:23.393 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/148' 2022-06-27T11:53:23.393 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/168/19' 2022-06-27T11:53:23.393 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/168/15' 2022-06-27T11:53:23.394 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/168/20' 2022-06-27T11:53:23.394 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/168/12' 2022-06-27T11:53:23.394 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/168/10' 2022-06-27T11:53:23.394 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/168/11' 2022-06-27T11:53:23.395 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/168/16' 2022-06-27T11:53:23.395 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/168/14' 2022-06-27T11:53:23.395 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/168/18' 2022-06-27T11:53:23.395 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/168/13' 2022-06-27T11:53:23.395 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/168/17' 2022-06-27T11:53:23.396 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/168' 2022-06-27T11:53:23.943 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/179/19' 2022-06-27T11:53:23.944 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/179/15' 2022-06-27T11:53:23.944 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/179/20' 2022-06-27T11:53:23.944 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/179/12' 2022-06-27T11:53:23.944 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/179/10' 2022-06-27T11:53:23.945 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/179/11' 2022-06-27T11:53:23.945 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/179/16' 2022-06-27T11:53:23.945 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/179/14' 2022-06-27T11:53:23.946 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/179/18' 2022-06-27T11:53:23.946 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/179/13' 2022-06-27T11:53:23.946 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/179/17' 2022-06-27T11:53:23.946 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/179' 2022-06-27T11:53:23.947 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/132/19' 2022-06-27T11:53:23.947 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/132/15' 2022-06-27T11:53:23.947 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/132/20' 2022-06-27T11:53:23.947 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/132/12' 2022-06-27T11:53:23.948 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/132/10' 2022-06-27T11:53:23.948 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/132/11' 2022-06-27T11:53:23.948 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/132/16' 2022-06-27T11:53:23.948 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/132/14' 2022-06-27T11:53:23.949 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/132/18' 2022-06-27T11:53:23.949 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/132/13' 2022-06-27T11:53:23.949 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/132/17' 2022-06-27T11:53:23.951 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/132' 2022-06-27T11:53:23.951 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/199/19' 2022-06-27T11:53:23.951 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/199/15' 2022-06-27T11:53:23.952 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/199/20' 2022-06-27T11:53:23.952 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/199/12' 2022-06-27T11:53:23.952 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/199/10' 2022-06-27T11:53:23.952 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/199/11' 2022-06-27T11:53:23.953 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/199/16' 2022-06-27T11:53:23.953 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/199/14' 2022-06-27T11:53:23.953 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/199/18' 2022-06-27T11:53:23.954 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/199/13' 2022-06-27T11:53:23.954 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/199/17' 2022-06-27T11:53:23.954 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/199' 2022-06-27T11:53:23.955 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/163/19' 2022-06-27T11:53:23.955 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/163/15' 2022-06-27T11:53:23.955 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/163/20' 2022-06-27T11:53:23.955 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/163/12' 2022-06-27T11:53:23.955 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/163/10' 2022-06-27T11:53:23.956 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/163/11' 2022-06-27T11:53:23.956 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/163/16' 2022-06-27T11:53:23.956 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/163/14' 2022-06-27T11:53:23.956 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/163/18' 2022-06-27T11:53:23.962 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/163/13' 2022-06-27T11:53:23.963 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/163/17' 2022-06-27T11:53:23.963 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/163' 2022-06-27T11:53:23.964 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/153/19' 2022-06-27T11:53:23.964 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/153/15' 2022-06-27T11:53:23.964 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/153/20' 2022-06-27T11:53:23.964 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/153/12' 2022-06-27T11:53:23.965 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/153/10' 2022-06-27T11:53:23.965 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/153/11' 2022-06-27T11:53:23.965 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/153/16' 2022-06-27T11:53:23.965 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/153/14' 2022-06-27T11:53:23.965 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/153/18' 2022-06-27T11:53:23.966 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/153/13' 2022-06-27T11:53:23.966 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/153/17' 2022-06-27T11:53:23.966 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/153' 2022-06-27T11:53:23.966 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/122/19' 2022-06-27T11:53:23.967 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/122/15' 2022-06-27T11:53:23.967 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/122/20' 2022-06-27T11:53:23.967 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/122/12' 2022-06-27T11:53:23.967 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/122/10' 2022-06-27T11:53:23.968 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/122/11' 2022-06-27T11:53:23.968 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/122/16' 2022-06-27T11:53:23.968 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/122/14' 2022-06-27T11:53:23.968 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/122/18' 2022-06-27T11:53:23.968 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/122/13' 2022-06-27T11:53:23.969 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/122/17' 2022-06-27T11:53:23.969 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/122' 2022-06-27T11:53:23.969 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/162/19' 2022-06-27T11:53:23.969 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/162/15' 2022-06-27T11:53:23.970 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/162/20' 2022-06-27T11:53:23.970 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/162/12' 2022-06-27T11:53:23.970 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/162/10' 2022-06-27T11:53:23.970 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/162/11' 2022-06-27T11:53:23.970 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/162/16' 2022-06-27T11:53:23.971 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/162/14' 2022-06-27T11:53:23.971 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/162/18' 2022-06-27T11:53:23.971 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/162/13' 2022-06-27T11:53:23.971 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/162/17' 2022-06-27T11:53:23.972 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/162' 2022-06-27T11:53:23.972 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/197/19' 2022-06-27T11:53:23.972 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/197/15' 2022-06-27T11:53:23.972 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/197/20' 2022-06-27T11:53:23.972 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/197/12' 2022-06-27T11:53:23.973 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/197/10' 2022-06-27T11:53:23.973 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/197/11' 2022-06-27T11:53:23.973 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/197/16' 2022-06-27T11:53:23.973 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/197/14' 2022-06-27T11:53:23.974 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/197/18' 2022-06-27T11:53:23.974 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/197/13' 2022-06-27T11:53:23.974 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/197/17' 2022-06-27T11:53:23.994 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/197' 2022-06-27T11:53:23.995 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/113/19' 2022-06-27T11:53:23.995 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/113/15' 2022-06-27T11:53:23.995 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/113/20' 2022-06-27T11:53:23.996 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/113/12' 2022-06-27T11:53:23.996 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/113/10' 2022-06-27T11:53:23.997 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/113/11' 2022-06-27T11:53:23.997 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/113/16' 2022-06-27T11:53:23.998 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/113/14' 2022-06-27T11:53:23.998 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/113/18' 2022-06-27T11:53:23.999 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/113/13' 2022-06-27T11:53:23.999 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/113/17' 2022-06-27T11:53:24.001 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/113' 2022-06-27T11:53:24.001 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/152/19' 2022-06-27T11:53:24.764 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/152/15' 2022-06-27T11:53:24.764 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/152/20' 2022-06-27T11:53:24.765 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/152/12' 2022-06-27T11:53:24.765 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/152/10' 2022-06-27T11:53:24.766 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/152/11' 2022-06-27T11:53:24.766 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/152/16' 2022-06-27T11:53:24.766 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/152/14' 2022-06-27T11:53:24.767 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/152/18' 2022-06-27T11:53:24.767 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/152/13' 2022-06-27T11:53:24.768 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/152/17' 2022-06-27T11:53:24.769 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/152' 2022-06-27T11:53:24.769 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/121/19' 2022-06-27T11:53:24.770 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/121/15' 2022-06-27T11:53:24.770 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/121/20' 2022-06-27T11:53:24.770 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/121/12' 2022-06-27T11:53:24.771 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/121/10' 2022-06-27T11:53:24.771 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/121/11' 2022-06-27T11:53:24.772 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/121/16' 2022-06-27T11:53:24.772 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/121/14' 2022-06-27T11:53:24.772 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/121/18' 2022-06-27T11:53:24.773 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/121/13' 2022-06-27T11:53:24.773 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/121/17' 2022-06-27T11:53:24.773 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/121' 2022-06-27T11:53:24.774 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/118/19' 2022-06-27T11:53:24.775 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/118/15' 2022-06-27T11:53:24.775 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/118/20' 2022-06-27T11:53:24.776 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/118/12' 2022-06-27T11:53:24.776 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/118/10' 2022-06-27T11:53:24.776 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/118/11' 2022-06-27T11:53:24.777 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/118/16' 2022-06-27T11:53:24.777 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/118/14' 2022-06-27T11:53:24.781 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/118/18' 2022-06-27T11:53:24.782 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/118/13' 2022-06-27T11:53:24.782 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/118/17' 2022-06-27T11:53:24.783 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/118' 2022-06-27T11:53:24.785 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/196/19' 2022-06-27T11:53:24.786 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/196/15' 2022-06-27T11:53:24.786 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/196/20' 2022-06-27T11:53:24.787 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/196/12' 2022-06-27T11:53:24.787 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/196/10' 2022-06-27T11:53:24.787 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/196/11' 2022-06-27T11:53:24.788 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/196/16' 2022-06-27T11:53:24.788 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/196/14' 2022-06-27T11:53:24.788 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/196/18' 2022-06-27T11:53:24.789 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/196/13' 2022-06-27T11:53:24.789 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/196/17' 2022-06-27T11:53:24.790 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/196' 2022-06-27T11:53:24.790 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/124/19' 2022-06-27T11:53:24.793 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/124/15' 2022-06-27T11:53:24.794 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/124/20' 2022-06-27T11:53:24.794 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/124/12' 2022-06-27T11:53:24.795 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/124/10' 2022-06-27T11:53:24.795 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/124/11' 2022-06-27T11:53:24.795 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/124/16' 2022-06-27T11:53:24.796 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/124/14' 2022-06-27T11:53:24.796 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/124/18' 2022-06-27T11:53:24.797 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/124/13' 2022-06-27T11:53:24.797 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/124/17' 2022-06-27T11:53:24.798 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/124' 2022-06-27T11:53:24.798 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/188/19' 2022-06-27T11:53:24.799 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/188/15' 2022-06-27T11:53:24.799 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/188/20' 2022-06-27T11:53:24.799 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/188/12' 2022-06-27T11:53:24.800 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/188/10' 2022-06-27T11:53:24.800 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/188/11' 2022-06-27T11:53:24.800 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/188/16' 2022-06-27T11:53:24.801 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/188/14' 2022-06-27T11:53:24.801 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/188/18' 2022-06-27T11:53:24.802 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/188/13' 2022-06-27T11:53:24.802 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/188/17' 2022-06-27T11:53:24.803 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/188' 2022-06-27T11:53:24.803 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/145/19' 2022-06-27T11:53:24.804 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/145/15' 2022-06-27T11:53:24.804 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/145/20' 2022-06-27T11:53:24.804 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/145/12' 2022-06-27T11:53:24.805 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/145/10' 2022-06-27T11:53:24.805 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/145/11' 2022-06-27T11:53:24.806 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/145/16' 2022-06-27T11:53:24.806 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/145/14' 2022-06-27T11:53:24.806 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/145/18' 2022-06-27T11:53:24.807 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/145/13' 2022-06-27T11:53:24.807 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/145/17' 2022-06-27T11:53:24.808 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/145' 2022-06-27T11:53:24.808 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/130/19' 2022-06-27T11:53:24.808 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/130/15' 2022-06-27T11:53:24.809 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/130/20' 2022-06-27T11:53:24.809 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/130/12' 2022-06-27T11:53:24.809 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/130/10' 2022-06-27T11:53:24.810 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/130/11' 2022-06-27T11:53:24.810 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/130/16' 2022-06-27T11:53:24.810 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/130/14' 2022-06-27T11:53:24.811 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/130/18' 2022-06-27T11:53:24.811 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/130/13' 2022-06-27T11:53:24.812 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/130/17' 2022-06-27T11:53:24.812 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/130' 2022-06-27T11:53:24.812 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/193/19' 2022-06-27T11:53:24.813 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/193/15' 2022-06-27T11:53:24.813 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/193/20' 2022-06-27T11:53:24.814 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/193/12' 2022-06-27T11:53:24.814 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/193/10' 2022-06-27T11:53:24.815 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/193/11' 2022-06-27T11:53:24.815 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/193/16' 2022-06-27T11:53:24.815 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/193/14' 2022-06-27T11:53:24.816 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/193/18' 2022-06-27T11:53:24.816 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/193/13' 2022-06-27T11:53:24.817 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/193/17' 2022-06-27T11:53:24.817 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/193' 2022-06-27T11:53:24.818 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/175/19' 2022-06-27T11:53:24.819 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/175/15' 2022-06-27T11:53:25.978 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/175/20' 2022-06-27T11:53:25.979 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/175/12' 2022-06-27T11:53:25.979 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/175/10' 2022-06-27T11:53:25.979 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/175/11' 2022-06-27T11:53:25.980 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/175/16' 2022-06-27T11:53:25.980 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/175/14' 2022-06-27T11:53:25.980 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/175/18' 2022-06-27T11:53:25.980 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/175/13' 2022-06-27T11:53:25.981 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/175/17' 2022-06-27T11:53:25.982 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/175' 2022-06-27T11:53:25.983 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/181/19' 2022-06-27T11:53:25.983 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/181/15' 2022-06-27T11:53:25.984 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/181/20' 2022-06-27T11:53:25.984 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/181/12' 2022-06-27T11:53:25.984 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/181/10' 2022-06-27T11:53:25.985 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/181/11' 2022-06-27T11:53:25.986 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/181/16' 2022-06-27T11:53:25.986 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/181/14' 2022-06-27T11:53:25.986 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/181/18' 2022-06-27T11:53:25.987 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/181/13' 2022-06-27T11:53:25.987 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/181/17' 2022-06-27T11:53:25.988 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/181' 2022-06-27T11:53:25.988 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/127/19' 2022-06-27T11:53:25.988 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/127/15' 2022-06-27T11:53:25.989 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/127/20' 2022-06-27T11:53:25.989 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/127/12' 2022-06-27T11:53:25.990 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/127/10' 2022-06-27T11:53:25.990 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/127/11' 2022-06-27T11:53:25.991 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/127/16' 2022-06-27T11:53:25.991 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/127/14' 2022-06-27T11:53:25.992 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/127/18' 2022-06-27T11:53:25.992 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/127/13' 2022-06-27T11:53:25.993 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/127/17' 2022-06-27T11:53:25.993 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/127' 2022-06-27T11:53:25.994 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/108/19' 2022-06-27T11:53:25.994 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/108/15' 2022-06-27T11:53:25.994 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/108/20' 2022-06-27T11:53:25.995 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/108/12' 2022-06-27T11:53:25.995 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/108/10' 2022-06-27T11:53:25.996 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/108/11' 2022-06-27T11:53:25.996 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/108/16' 2022-06-27T11:53:25.997 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/108/14' 2022-06-27T11:53:25.997 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/108/18' 2022-06-27T11:53:25.998 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/108/13' 2022-06-27T11:53:25.998 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/108/17' 2022-06-27T11:53:25.998 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/108' 2022-06-27T11:53:25.999 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/183/19' 2022-06-27T11:53:25.999 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/183/15' 2022-06-27T11:53:25.999 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/183/20' 2022-06-27T11:53:26.000 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/183/12' 2022-06-27T11:53:26.000 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/183/10' 2022-06-27T11:53:26.000 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/183/11' 2022-06-27T11:53:26.000 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/183/16' 2022-06-27T11:53:26.001 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/183/14' 2022-06-27T11:53:26.001 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/183/18' 2022-06-27T11:53:26.002 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/183/13' 2022-06-27T11:53:26.002 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/183/17' 2022-06-27T11:53:26.002 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/183' 2022-06-27T11:53:26.003 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/177/19' 2022-06-27T11:53:26.003 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/177/15' 2022-06-27T11:53:26.003 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/177/20' 2022-06-27T11:53:26.004 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/177/12' 2022-06-27T11:53:26.004 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/177/10' 2022-06-27T11:53:26.005 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/177/11' 2022-06-27T11:53:26.005 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/177/16' 2022-06-27T11:53:26.006 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/177/14' 2022-06-27T11:53:26.006 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/177/18' 2022-06-27T11:53:26.007 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/177/13' 2022-06-27T11:53:26.007 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/177/17' 2022-06-27T11:53:26.008 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/177' 2022-06-27T11:53:26.008 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/169/19' 2022-06-27T11:53:26.009 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/169/15' 2022-06-27T11:53:26.009 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/169/20' 2022-06-27T11:53:26.010 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/169/12' 2022-06-27T11:53:26.010 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/169/10' 2022-06-27T11:53:26.011 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/169/11' 2022-06-27T11:53:26.011 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/169/16' 2022-06-27T11:53:26.012 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/169/14' 2022-06-27T11:53:26.012 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/169/18' 2022-06-27T11:53:26.012 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/169/13' 2022-06-27T11:53:26.013 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/169/17' 2022-06-27T11:53:26.014 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/169' 2022-06-27T11:53:26.014 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/154/19' 2022-06-27T11:53:26.014 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/154/15' 2022-06-27T11:53:26.015 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/154/20' 2022-06-27T11:53:26.015 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/154/12' 2022-06-27T11:53:26.015 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/154/10' 2022-06-27T11:53:26.016 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/154/11' 2022-06-27T11:53:26.016 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/154/16' 2022-06-27T11:53:26.016 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/154/14' 2022-06-27T11:53:26.017 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/154/18' 2022-06-27T11:53:26.017 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/154/13' 2022-06-27T11:53:26.018 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/154/17' 2022-06-27T11:53:26.018 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/154' 2022-06-27T11:53:26.019 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/171/19' 2022-06-27T11:53:26.019 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/171/15' 2022-06-27T11:53:26.019 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/171/20' 2022-06-27T11:53:26.020 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/171/12' 2022-06-27T11:53:26.020 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/171/10' 2022-06-27T11:53:26.020 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/171/11' 2022-06-27T11:53:26.020 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/171/16' 2022-06-27T11:53:26.021 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/171/14' 2022-06-27T11:53:26.021 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/171/18' 2022-06-27T11:53:26.021 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/171/13' 2022-06-27T11:53:26.021 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/171/17' 2022-06-27T11:53:26.021 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/171' 2022-06-27T11:53:26.022 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/139/19' 2022-06-27T11:53:26.022 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/139/15' 2022-06-27T11:53:26.022 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/139/20' 2022-06-27T11:53:27.103 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/139/12' 2022-06-27T11:53:27.104 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/139/10' 2022-06-27T11:53:27.104 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/139/11' 2022-06-27T11:53:27.104 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/139/16' 2022-06-27T11:53:27.105 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/139/14' 2022-06-27T11:53:27.107 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/139/18' 2022-06-27T11:53:27.108 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/139/13' 2022-06-27T11:53:27.108 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/139/17' 2022-06-27T11:53:27.109 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/139' 2022-06-27T11:53:27.109 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/102/19' 2022-06-27T11:53:27.110 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/102/15' 2022-06-27T11:53:27.110 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/102/20' 2022-06-27T11:53:27.110 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/102/12' 2022-06-27T11:53:27.111 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/102/10' 2022-06-27T11:53:27.111 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/102/11' 2022-06-27T11:53:27.111 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/102/16' 2022-06-27T11:53:27.112 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/102/14' 2022-06-27T11:53:27.112 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/102/18' 2022-06-27T11:53:27.113 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/102/13' 2022-06-27T11:53:27.113 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/102/17' 2022-06-27T11:53:27.113 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/102' 2022-06-27T11:53:27.114 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/129/19' 2022-06-27T11:53:27.114 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/129/15' 2022-06-27T11:53:27.115 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/129/20' 2022-06-27T11:53:27.115 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/129/12' 2022-06-27T11:53:27.115 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/129/10' 2022-06-27T11:53:27.116 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/129/11' 2022-06-27T11:53:27.116 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/129/16' 2022-06-27T11:53:27.117 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/129/14' 2022-06-27T11:53:27.117 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/129/18' 2022-06-27T11:53:27.118 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/129/13' 2022-06-27T11:53:27.119 INFO:tasks.workunit.client.0.smithi195.stdout:removed './copyofsnap1/test1/129/17' 2022-06-27T11:53:27.119 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1/129' 2022-06-27T11:53:27.120 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1/test1' 2022-06-27T11:53:27.120 INFO:tasks.workunit.client.0.smithi195.stdout:removed directory './copyofsnap1' 2022-06-27T11:53:27.120 INFO:tasks.workunit.client.0.smithi195.stdout:OK 2022-06-27T11:53:27.121 INFO:teuthology.orchestra.run:Running command with timeout 3600 2022-06-27T11:53:27.122 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2022-06-27T11:53:27.171 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-authwb.sh... 2022-06-27T11:53:27.172 DEBUG:teuthology.orchestra.run.smithi195:workunit test fs/snaps/snaptest-authwb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5292901bc999daa0d7ac128764a3f27c4233c252 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-authwb.sh 2022-06-27T11:53:27.253 INFO:tasks.workunit.client.0.smithi195.stderr:+ set -e 2022-06-27T11:53:27.253 INFO:tasks.workunit.client.0.smithi195.stderr:+ touch foo 2022-06-27T11:53:27.256 INFO:tasks.workunit.client.0.smithi195.stderr:+ chmod +x foo 2022-06-27T11:53:27.256 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/s 2022-06-27T11:53:27.294 INFO:tasks.workunit.client.0.smithi195.stderr:+ find .snap/s/foo -executable 2022-06-27T11:53:27.294 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep foo 2022-06-27T11:53:27.301 INFO:tasks.workunit.client.0.smithi195.stdout:.snap/s/foo 2022-06-27T11:53:27.301 INFO:tasks.workunit.client.0.smithi195.stderr:+ rmdir .snap/s 2022-06-27T11:53:27.312 INFO:tasks.workunit.client.0.smithi195.stderr:+ rm foo 2022-06-27T11:53:27.313 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo OK 2022-06-27T11:53:27.316 INFO:tasks.workunit.client.0.smithi195.stdout:OK 2022-06-27T11:53:27.316 INFO:teuthology.orchestra.run:Running command with timeout 3600 2022-06-27T11:53:27.317 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2022-06-27T11:53:27.400 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-capwb.sh... 2022-06-27T11:53:27.401 DEBUG:teuthology.orchestra.run.smithi195:workunit test fs/snaps/snaptest-capwb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5292901bc999daa0d7ac128764a3f27c4233c252 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-capwb.sh 2022-06-27T11:53:27.475 INFO:tasks.workunit.client.0.smithi195.stderr:+ set -e 2022-06-27T11:53:27.475 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir foo 2022-06-27T11:53:27.478 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo x 2022-06-27T11:53:27.480 INFO:tasks.workunit.client.0.smithi195.stderr:+ sync 2022-06-27T11:53:27.540 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T11:53:27.574 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T11:53:28.054 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir foo/.snap/ss 2022-06-27T11:53:28.086 INFO:tasks.workunit.client.0.smithi195.stderr:+ ln foo/x foo/xx 2022-06-27T11:53:28.096 INFO:tasks.workunit.client.0.smithi195.stderr:+ cat foo/.snap/ss/x 2022-06-27T11:53:28.103 INFO:tasks.workunit.client.0.smithi195.stdout:x 2022-06-27T11:53:28.104 INFO:tasks.workunit.client.0.smithi195.stderr:+ rmdir foo/.snap/ss 2022-06-27T11:53:28.116 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo a 2022-06-27T11:53:28.117 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo b 2022-06-27T11:53:28.117 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir foo/.snap/s 2022-06-27T11:53:28.138 INFO:tasks.workunit.client.0.smithi195.stderr:++ cat foo/.snap/s/a 2022-06-27T11:53:28.161 INFO:tasks.workunit.client.0.smithi195.stderr:+ r=a 2022-06-27T11:53:28.161 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' -z a ']' 2022-06-27T11:53:28.162 INFO:tasks.workunit.client.0.smithi195.stderr:+ ln foo/b foo/b2 2022-06-27T11:53:28.162 INFO:tasks.workunit.client.0.smithi195.stderr:+ cat foo/.snap/s/b 2022-06-27T11:53:28.171 INFO:tasks.workunit.client.0.smithi195.stdout:b 2022-06-27T11:53:28.172 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo 'this used to hang:' 2022-06-27T11:53:28.173 INFO:tasks.workunit.client.0.smithi195.stdout:this used to hang: 2022-06-27T11:53:28.173 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo more 2022-06-27T11:53:28.178 INFO:tasks.workunit.client.0.smithi195.stdout:oh, it didn't hang! good job. 2022-06-27T11:53:28.179 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo 'oh, it didn'\''t hang! good job.' 2022-06-27T11:53:28.180 INFO:tasks.workunit.client.0.smithi195.stderr:+ cat foo/b 2022-06-27T11:53:28.180 INFO:tasks.workunit.client.0.smithi195.stdout:b 2022-06-27T11:53:28.181 INFO:tasks.workunit.client.0.smithi195.stdout:more 2022-06-27T11:53:28.181 INFO:tasks.workunit.client.0.smithi195.stderr:+ rmdir foo/.snap/s 2022-06-27T11:53:28.182 INFO:tasks.workunit.client.0.smithi195.stderr:+ rm -r foo 2022-06-27T11:53:28.193 INFO:tasks.workunit.client.0.smithi195.stdout:OK 2022-06-27T11:53:28.194 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo OK 2022-06-27T11:53:28.196 INFO:teuthology.orchestra.run:Running command with timeout 3600 2022-06-27T11:53:28.196 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2022-06-27T11:53:28.284 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-dir-rename.sh... 2022-06-27T11:53:28.285 DEBUG:teuthology.orchestra.run.smithi195:workunit test fs/snaps/snaptest-dir-rename.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5292901bc999daa0d7ac128764a3f27c4233c252 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-dir-rename.sh 2022-06-27T11:53:28.353 INFO:tasks.workunit.client.0.smithi195.stderr:+ set -e 2022-06-27T11:53:28.354 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir a 2022-06-27T11:53:28.355 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/bar 2022-06-27T11:53:28.382 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir a/.snap/foo 2022-06-27T11:53:28.429 INFO:tasks.workunit.client.0.smithi195.stderr:+ rmdir a/.snap/foo 2022-06-27T11:53:28.446 INFO:tasks.workunit.client.0.smithi195.stderr:+ rmdir a 2022-06-27T11:53:28.451 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/bar/a 2022-06-27T11:53:28.452 INFO:tasks.workunit.client.0.smithi195.stdout: File: .snap/bar/a 2022-06-27T11:53:28.453 INFO:tasks.workunit.client.0.smithi195.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2022-06-27T11:53:28.455 INFO:tasks.workunit.client.0.smithi195.stdout:Device: 100001h/1048577d Inode: 1099511633797 Links: 2 2022-06-27T11:53:28.456 INFO:tasks.workunit.client.0.smithi195.stdout:Access: (0775/drwxrwxr-x) Uid: ( 1000/ ubuntu) Gid: ( 1264/ ubuntu) 2022-06-27T11:53:28.456 INFO:tasks.workunit.client.0.smithi195.stdout:Context: unconfined_u:object_r:cephfs_t:s0 2022-06-27T11:53:28.457 INFO:tasks.workunit.client.0.smithi195.stdout:Access: 2022-06-27 11:53:28.355283516 +0000 2022-06-27T11:53:28.457 INFO:tasks.workunit.client.0.smithi195.stdout:Modify: 2022-06-27 11:53:28.355283516 +0000 2022-06-27T11:53:28.458 INFO:tasks.workunit.client.0.smithi195.stdout:Change: 2022-06-27 11:53:28.355283516 +0000 2022-06-27T11:53:28.460 INFO:tasks.workunit.client.0.smithi195.stdout: Birth: 2022-06-27 11:53:28.355283516 +0000 2022-06-27T11:53:28.461 INFO:tasks.workunit.client.0.smithi195.stderr:+ rmdir .snap/bar 2022-06-27T11:53:28.461 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo OK 2022-06-27T11:53:28.463 INFO:tasks.workunit.client.0.smithi195.stdout:OK 2022-06-27T11:53:28.464 INFO:teuthology.orchestra.run:Running command with timeout 3600 2022-06-27T11:53:28.464 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2022-06-27T11:53:28.542 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-double-null.sh... 2022-06-27T11:53:28.542 DEBUG:teuthology.orchestra.run.smithi195:workunit test fs/snaps/snaptest-double-null.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5292901bc999daa0d7ac128764a3f27c4233c252 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-double-null.sh 2022-06-27T11:53:28.606 INFO:tasks.workunit.client.0.smithi195.stderr:+ set -e 2022-06-27T11:53:28.607 INFO:tasks.workunit.client.0.smithi195.stderr:++ seq 1 20 2022-06-27T11:53:28.607 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 20` 2022-06-27T11:53:28.607 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir a 2022-06-27T11:53:28.609 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir a/.snap/one 2022-06-27T11:53:28.609 INFO:tasks.workunit.client.0.smithi195.stderr:+ cat 2022-06-27T11:53:28.637 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir a/.snap/two 2022-06-27T11:53:28.678 INFO:tasks.workunit.client.0.smithi195.stderr:+ chmod 777 a/foo 2022-06-27T11:53:28.679 INFO:tasks.workunit.client.0.smithi195.stderr:+ sync 2022-06-27T11:53:28.797 INFO:tasks.workunit.client.0.smithi195.stderr:+ ps 2022-06-27T11:53:28.818 INFO:tasks.workunit.client.0.smithi195.stdout: PID TTY TIME CMD 2022-06-27T11:53:28.818 INFO:tasks.workunit.client.0.smithi195.stdout: 1282 ? 00:00:00 systemd 2022-06-27T11:53:28.819 INFO:tasks.workunit.client.0.smithi195.stdout: 1285 ? 00:00:00 (sd-pam) 2022-06-27T11:53:28.819 INFO:tasks.workunit.client.0.smithi195.stdout: 81346 ? 00:00:00 sshd 2022-06-27T11:53:28.820 INFO:tasks.workunit.client.0.smithi195.stdout: 108414 ? 00:00:00 timeout 2022-06-27T11:53:28.820 INFO:tasks.workunit.client.0.smithi195.stdout: 108435 ? 00:00:00 snaptest-double 2022-06-27T11:53:28.821 INFO:tasks.workunit.client.0.smithi195.stdout: 108443 ? 00:00:00 ps 2022-06-27T11:53:28.822 INFO:tasks.workunit.client.0.smithi195.stderr:+ rmdir a/.snap/one a/.snap/two 2022-06-27T11:53:28.823 INFO:tasks.workunit.client.0.smithi195.stderr:+ rm a/foo 2022-06-27T11:53:28.834 INFO:tasks.workunit.client.0.smithi195.stderr:+ rmdir a 2022-06-27T11:53:28.834 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 20` 2022-06-27T11:53:28.835 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir a 2022-06-27T11:53:28.854 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir a/.snap/one 2022-06-27T11:53:28.855 INFO:tasks.workunit.client.0.smithi195.stderr:+ cat 2022-06-27T11:53:29.139 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir a/.snap/two 2022-06-27T11:53:29.208 INFO:tasks.workunit.client.0.smithi195.stderr:+ chmod 777 a/foo 2022-06-27T11:53:29.209 INFO:tasks.workunit.client.0.smithi195.stderr:+ sync 2022-06-27T11:53:29.316 INFO:tasks.workunit.client.0.smithi195.stderr:+ ps 2022-06-27T11:53:29.323 INFO:tasks.workunit.client.0.smithi195.stdout: PID TTY TIME CMD 2022-06-27T11:53:29.324 INFO:tasks.workunit.client.0.smithi195.stdout: 1282 ? 00:00:00 systemd 2022-06-27T11:53:29.324 INFO:tasks.workunit.client.0.smithi195.stdout: 1285 ? 00:00:00 (sd-pam) 2022-06-27T11:53:29.325 INFO:tasks.workunit.client.0.smithi195.stdout: 81346 ? 00:00:00 sshd 2022-06-27T11:53:29.325 INFO:tasks.workunit.client.0.smithi195.stdout: 108414 ? 00:00:00 timeout 2022-06-27T11:53:29.333 INFO:tasks.workunit.client.0.smithi195.stdout: 108435 ? 00:00:00 snaptest-double 2022-06-27T11:53:29.334 INFO:tasks.workunit.client.0.smithi195.stdout: 108453 ? 00:00:00 ps 2022-06-27T11:53:29.335 INFO:tasks.workunit.client.0.smithi195.stderr:+ rmdir a/.snap/one a/.snap/two 2022-06-27T11:53:29.336 INFO:tasks.workunit.client.0.smithi195.stderr:+ rm a/foo 2022-06-27T11:53:29.346 INFO:tasks.workunit.client.0.smithi195.stderr:+ rmdir a 2022-06-27T11:53:30.075 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 20` 2022-06-27T11:53:30.076 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir a 2022-06-27T11:53:30.076 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir a/.snap/one 2022-06-27T11:53:30.080 INFO:tasks.workunit.client.0.smithi195.stderr:+ cat 2022-06-27T11:53:30.303 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir a/.snap/two 2022-06-27T11:53:30.334 INFO:tasks.workunit.client.0.smithi195.stderr:+ chmod 777 a/foo 2022-06-27T11:53:30.334 INFO:tasks.workunit.client.0.smithi195.stderr:+ sync 2022-06-27T11:53:30.428 INFO:tasks.workunit.client.0.smithi195.stderr:+ ps 2022-06-27T11:53:30.437 INFO:tasks.workunit.client.0.smithi195.stdout: PID TTY TIME CMD 2022-06-27T11:53:30.438 INFO:tasks.workunit.client.0.smithi195.stdout: 1282 ? 00:00:00 systemd 2022-06-27T11:53:30.438 INFO:tasks.workunit.client.0.smithi195.stdout: 1285 ? 00:00:00 (sd-pam) 2022-06-27T11:53:30.439 INFO:tasks.workunit.client.0.smithi195.stdout: 81346 ? 00:00:00 sshd 2022-06-27T11:53:30.439 INFO:tasks.workunit.client.0.smithi195.stdout: 108414 ? 00:00:00 timeout 2022-06-27T11:53:30.440 INFO:tasks.workunit.client.0.smithi195.stdout: 108435 ? 00:00:00 snaptest-double 2022-06-27T11:53:30.440 INFO:tasks.workunit.client.0.smithi195.stdout: 108463 ? 00:00:00 ps 2022-06-27T11:53:30.441 INFO:tasks.workunit.client.0.smithi195.stderr:+ rmdir a/.snap/one a/.snap/two 2022-06-27T11:53:30.450 INFO:tasks.workunit.client.0.smithi195.stderr:+ rm a/foo 2022-06-27T11:53:30.453 INFO:tasks.workunit.client.0.smithi195.stderr:+ rmdir a 2022-06-27T11:53:30.456 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 20` 2022-06-27T11:53:30.456 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir a 2022-06-27T11:53:30.471 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir a/.snap/one 2022-06-27T11:53:30.478 INFO:tasks.workunit.client.0.smithi195.stderr:+ cat 2022-06-27T11:53:30.480 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir a/.snap/two 2022-06-27T11:53:30.504 INFO:tasks.workunit.client.0.smithi195.stderr:+ chmod 777 a/foo 2022-06-27T11:53:30.505 INFO:tasks.workunit.client.0.smithi195.stderr:+ sync 2022-06-27T11:53:30.599 INFO:tasks.workunit.client.0.smithi195.stderr:+ ps 2022-06-27T11:53:30.603 INFO:tasks.workunit.client.0.smithi195.stdout: PID TTY TIME CMD 2022-06-27T11:53:30.603 INFO:tasks.workunit.client.0.smithi195.stdout: 1282 ? 00:00:00 systemd 2022-06-27T11:53:30.604 INFO:tasks.workunit.client.0.smithi195.stdout: 1285 ? 00:00:00 (sd-pam) 2022-06-27T11:53:30.604 INFO:tasks.workunit.client.0.smithi195.stdout: 81346 ? 00:00:00 sshd 2022-06-27T11:53:30.604 INFO:tasks.workunit.client.0.smithi195.stdout: 108414 ? 00:00:00 timeout 2022-06-27T11:53:30.604 INFO:tasks.workunit.client.0.smithi195.stdout: 108435 ? 00:00:00 snaptest-double 2022-06-27T11:53:30.605 INFO:tasks.workunit.client.0.smithi195.stdout: 108473 ? 00:00:00 ps 2022-06-27T11:53:30.605 INFO:tasks.workunit.client.0.smithi195.stderr:+ rmdir a/.snap/one a/.snap/two 2022-06-27T11:53:31.207 INFO:tasks.workunit.client.0.smithi195.stderr:+ rm a/foo 2022-06-27T11:53:31.209 INFO:tasks.workunit.client.0.smithi195.stderr:+ rmdir a 2022-06-27T11:53:31.214 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 20` 2022-06-27T11:53:31.215 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir a 2022-06-27T11:53:31.217 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir a/.snap/one 2022-06-27T11:53:31.217 INFO:tasks.workunit.client.0.smithi195.stderr:+ cat 2022-06-27T11:53:31.238 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir a/.snap/two 2022-06-27T11:53:31.269 INFO:tasks.workunit.client.0.smithi195.stderr:+ chmod 777 a/foo 2022-06-27T11:53:31.270 INFO:tasks.workunit.client.0.smithi195.stderr:+ sync 2022-06-27T11:53:31.489 INFO:tasks.workunit.client.0.smithi195.stderr:+ ps 2022-06-27T11:53:31.495 INFO:tasks.workunit.client.0.smithi195.stdout: PID TTY TIME CMD 2022-06-27T11:53:31.495 INFO:tasks.workunit.client.0.smithi195.stdout: 1282 ? 00:00:00 systemd 2022-06-27T11:53:31.496 INFO:tasks.workunit.client.0.smithi195.stdout: 1285 ? 00:00:00 (sd-pam) 2022-06-27T11:53:31.496 INFO:tasks.workunit.client.0.smithi195.stdout: 81346 ? 00:00:00 sshd 2022-06-27T11:53:31.497 INFO:tasks.workunit.client.0.smithi195.stdout: 108414 ? 00:00:00 timeout 2022-06-27T11:53:31.497 INFO:tasks.workunit.client.0.smithi195.stdout: 108435 ? 00:00:00 snaptest-double 2022-06-27T11:53:31.497 INFO:tasks.workunit.client.0.smithi195.stdout: 108483 ? 00:00:00 ps 2022-06-27T11:53:31.499 INFO:tasks.workunit.client.0.smithi195.stderr:+ rmdir a/.snap/one a/.snap/two 2022-06-27T11:53:31.505 INFO:tasks.workunit.client.0.smithi195.stderr:+ rm a/foo 2022-06-27T11:53:31.513 INFO:tasks.workunit.client.0.smithi195.stderr:+ rmdir a 2022-06-27T11:53:31.513 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 20` 2022-06-27T11:53:31.514 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir a 2022-06-27T11:53:31.516 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir a/.snap/one 2022-06-27T11:53:31.517 INFO:tasks.workunit.client.0.smithi195.stderr:+ cat 2022-06-27T11:53:31.534 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir a/.snap/two 2022-06-27T11:53:31.565 INFO:tasks.workunit.client.0.smithi195.stderr:+ chmod 777 a/foo 2022-06-27T11:53:31.568 INFO:tasks.workunit.client.0.smithi195.stderr:+ sync 2022-06-27T11:53:31.662 INFO:tasks.workunit.client.0.smithi195.stderr:+ ps 2022-06-27T11:53:31.670 INFO:tasks.workunit.client.0.smithi195.stdout: PID TTY TIME CMD 2022-06-27T11:53:31.671 INFO:tasks.workunit.client.0.smithi195.stdout: 1282 ? 00:00:00 systemd 2022-06-27T11:53:31.671 INFO:tasks.workunit.client.0.smithi195.stdout: 1285 ? 00:00:00 (sd-pam) 2022-06-27T11:53:31.671 INFO:tasks.workunit.client.0.smithi195.stdout: 81346 ? 00:00:00 sshd 2022-06-27T11:53:31.672 INFO:tasks.workunit.client.0.smithi195.stdout: 108414 ? 00:00:00 timeout 2022-06-27T11:53:31.672 INFO:tasks.workunit.client.0.smithi195.stdout: 108435 ? 00:00:00 snaptest-double 2022-06-27T11:53:31.672 INFO:tasks.workunit.client.0.smithi195.stdout: 108493 ? 00:00:00 ps 2022-06-27T11:53:31.673 INFO:tasks.workunit.client.0.smithi195.stderr:+ rmdir a/.snap/one a/.snap/two 2022-06-27T11:53:31.681 INFO:tasks.workunit.client.0.smithi195.stderr:+ rm a/foo 2022-06-27T11:53:31.682 INFO:tasks.workunit.client.0.smithi195.stderr:+ rmdir a 2022-06-27T11:53:31.688 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 20` 2022-06-27T11:53:31.689 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir a 2022-06-27T11:53:31.692 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir a/.snap/one 2022-06-27T11:53:31.693 INFO:tasks.workunit.client.0.smithi195.stderr:+ cat 2022-06-27T11:53:31.710 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir a/.snap/two 2022-06-27T11:53:31.742 INFO:tasks.workunit.client.0.smithi195.stderr:+ chmod 777 a/foo 2022-06-27T11:53:31.743 INFO:tasks.workunit.client.0.smithi195.stderr:+ sync 2022-06-27T11:53:31.805 INFO:tasks.workunit.client.0.smithi195.stderr:+ ps 2022-06-27T11:53:31.814 INFO:tasks.workunit.client.0.smithi195.stdout: PID TTY TIME CMD 2022-06-27T11:53:31.815 INFO:tasks.workunit.client.0.smithi195.stdout: 1282 ? 00:00:00 systemd 2022-06-27T11:53:31.815 INFO:tasks.workunit.client.0.smithi195.stdout: 1285 ? 00:00:00 (sd-pam) 2022-06-27T11:53:31.816 INFO:tasks.workunit.client.0.smithi195.stdout: 81346 ? 00:00:00 sshd 2022-06-27T11:53:31.816 INFO:tasks.workunit.client.0.smithi195.stdout: 108414 ? 00:00:00 timeout 2022-06-27T11:53:31.817 INFO:tasks.workunit.client.0.smithi195.stdout: 108435 ? 00:00:00 snaptest-double 2022-06-27T11:53:31.818 INFO:tasks.workunit.client.0.smithi195.stdout: 108503 ? 00:00:00 ps 2022-06-27T11:53:31.819 INFO:tasks.workunit.client.0.smithi195.stderr:+ rmdir a/.snap/one a/.snap/two 2022-06-27T11:53:31.854 INFO:tasks.workunit.client.0.smithi195.stderr:+ rm a/foo 2022-06-27T11:53:31.855 INFO:tasks.workunit.client.0.smithi195.stderr:+ rmdir a 2022-06-27T11:53:31.856 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 20` 2022-06-27T11:53:31.857 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir a 2022-06-27T11:53:31.857 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir a/.snap/one 2022-06-27T11:53:31.858 INFO:tasks.workunit.client.0.smithi195.stderr:+ cat 2022-06-27T11:53:31.858 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir a/.snap/two 2022-06-27T11:53:31.883 INFO:tasks.workunit.client.0.smithi195.stderr:+ chmod 777 a/foo 2022-06-27T11:53:31.883 INFO:tasks.workunit.client.0.smithi195.stderr:+ sync 2022-06-27T11:53:31.962 INFO:tasks.workunit.client.0.smithi195.stderr:+ ps 2022-06-27T11:53:31.978 INFO:tasks.workunit.client.0.smithi195.stdout: PID TTY TIME CMD 2022-06-27T11:53:31.979 INFO:tasks.workunit.client.0.smithi195.stdout: 1282 ? 00:00:00 systemd 2022-06-27T11:53:31.979 INFO:tasks.workunit.client.0.smithi195.stdout: 1285 ? 00:00:00 (sd-pam) 2022-06-27T11:53:31.979 INFO:tasks.workunit.client.0.smithi195.stdout: 81346 ? 00:00:00 sshd 2022-06-27T11:53:31.980 INFO:tasks.workunit.client.0.smithi195.stdout: 108414 ? 00:00:00 timeout 2022-06-27T11:53:31.980 INFO:tasks.workunit.client.0.smithi195.stdout: 108435 ? 00:00:00 snaptest-double 2022-06-27T11:53:31.980 INFO:tasks.workunit.client.0.smithi195.stdout: 108513 ? 00:00:00 ps 2022-06-27T11:53:31.981 INFO:tasks.workunit.client.0.smithi195.stderr:+ rmdir a/.snap/one a/.snap/two 2022-06-27T11:53:31.991 INFO:tasks.workunit.client.0.smithi195.stderr:+ rm a/foo 2022-06-27T11:53:31.991 INFO:tasks.workunit.client.0.smithi195.stderr:+ rmdir a 2022-06-27T11:53:31.994 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 20` 2022-06-27T11:53:31.994 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir a 2022-06-27T11:53:31.995 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir a/.snap/one 2022-06-27T11:53:31.995 INFO:tasks.workunit.client.0.smithi195.stderr:+ cat 2022-06-27T11:53:32.015 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir a/.snap/two 2022-06-27T11:53:32.044 INFO:tasks.workunit.client.0.smithi195.stderr:+ chmod 777 a/foo 2022-06-27T11:53:32.045 INFO:tasks.workunit.client.0.smithi195.stderr:+ sync 2022-06-27T11:53:32.143 INFO:tasks.workunit.client.0.smithi195.stderr:+ ps 2022-06-27T11:53:32.146 INFO:tasks.workunit.client.0.smithi195.stdout: PID TTY TIME CMD 2022-06-27T11:53:32.146 INFO:tasks.workunit.client.0.smithi195.stdout: 1282 ? 00:00:00 systemd 2022-06-27T11:53:32.147 INFO:tasks.workunit.client.0.smithi195.stdout: 1285 ? 00:00:00 (sd-pam) 2022-06-27T11:53:32.147 INFO:tasks.workunit.client.0.smithi195.stdout: 81346 ? 00:00:00 sshd 2022-06-27T11:53:32.147 INFO:tasks.workunit.client.0.smithi195.stdout: 108414 ? 00:00:00 timeout 2022-06-27T11:53:32.147 INFO:tasks.workunit.client.0.smithi195.stdout: 108435 ? 00:00:00 snaptest-double 2022-06-27T11:53:32.148 INFO:tasks.workunit.client.0.smithi195.stdout: 108523 ? 00:00:00 ps 2022-06-27T11:53:32.148 INFO:tasks.workunit.client.0.smithi195.stderr:+ rmdir a/.snap/one a/.snap/two 2022-06-27T11:53:32.157 INFO:tasks.workunit.client.0.smithi195.stderr:+ rm a/foo 2022-06-27T11:53:32.159 INFO:tasks.workunit.client.0.smithi195.stderr:+ rmdir a 2022-06-27T11:53:32.177 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 20` 2022-06-27T11:53:32.177 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir a 2022-06-27T11:53:32.178 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir a/.snap/one 2022-06-27T11:53:32.178 INFO:tasks.workunit.client.0.smithi195.stderr:+ cat 2022-06-27T11:53:32.190 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir a/.snap/two 2022-06-27T11:53:32.210 INFO:tasks.workunit.client.0.smithi195.stderr:+ chmod 777 a/foo 2022-06-27T11:53:32.211 INFO:tasks.workunit.client.0.smithi195.stderr:+ sync 2022-06-27T11:53:32.306 INFO:tasks.workunit.client.0.smithi195.stderr:+ ps 2022-06-27T11:53:32.308 INFO:tasks.workunit.client.0.smithi195.stdout: PID TTY TIME CMD 2022-06-27T11:53:32.308 INFO:tasks.workunit.client.0.smithi195.stdout: 1282 ? 00:00:00 systemd 2022-06-27T11:53:32.309 INFO:tasks.workunit.client.0.smithi195.stdout: 1285 ? 00:00:00 (sd-pam) 2022-06-27T11:53:32.309 INFO:tasks.workunit.client.0.smithi195.stdout: 81346 ? 00:00:00 sshd 2022-06-27T11:53:32.309 INFO:tasks.workunit.client.0.smithi195.stdout: 108414 ? 00:00:00 timeout 2022-06-27T11:53:32.309 INFO:tasks.workunit.client.0.smithi195.stdout: 108435 ? 00:00:00 snaptest-double 2022-06-27T11:53:32.310 INFO:tasks.workunit.client.0.smithi195.stdout: 108533 ? 00:00:00 ps 2022-06-27T11:53:32.310 INFO:tasks.workunit.client.0.smithi195.stderr:+ rmdir a/.snap/one a/.snap/two 2022-06-27T11:53:32.318 INFO:tasks.workunit.client.0.smithi195.stderr:+ rm a/foo 2022-06-27T11:53:32.319 INFO:tasks.workunit.client.0.smithi195.stderr:+ rmdir a 2022-06-27T11:53:32.325 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 20` 2022-06-27T11:53:32.326 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir a 2022-06-27T11:53:32.327 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir a/.snap/one 2022-06-27T11:53:32.327 INFO:tasks.workunit.client.0.smithi195.stderr:+ cat 2022-06-27T11:53:32.374 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir a/.snap/two 2022-06-27T11:53:32.396 INFO:tasks.workunit.client.0.smithi195.stderr:+ chmod 777 a/foo 2022-06-27T11:53:32.397 INFO:tasks.workunit.client.0.smithi195.stderr:+ sync 2022-06-27T11:53:32.458 INFO:tasks.workunit.client.0.smithi195.stderr:+ ps 2022-06-27T11:53:32.465 INFO:tasks.workunit.client.0.smithi195.stdout: PID TTY TIME CMD 2022-06-27T11:53:32.466 INFO:tasks.workunit.client.0.smithi195.stdout: 1282 ? 00:00:00 systemd 2022-06-27T11:53:32.467 INFO:tasks.workunit.client.0.smithi195.stdout: 1285 ? 00:00:00 (sd-pam) 2022-06-27T11:53:32.468 INFO:tasks.workunit.client.0.smithi195.stdout: 81346 ? 00:00:00 sshd 2022-06-27T11:53:32.468 INFO:tasks.workunit.client.0.smithi195.stdout: 108414 ? 00:00:00 timeout 2022-06-27T11:53:32.469 INFO:tasks.workunit.client.0.smithi195.stdout: 108435 ? 00:00:00 snaptest-double 2022-06-27T11:53:32.470 INFO:tasks.workunit.client.0.smithi195.stdout: 108543 ? 00:00:00 ps 2022-06-27T11:53:32.471 INFO:tasks.workunit.client.0.smithi195.stderr:+ rmdir a/.snap/one a/.snap/two 2022-06-27T11:53:32.488 INFO:tasks.workunit.client.0.smithi195.stderr:+ rm a/foo 2022-06-27T11:53:32.488 INFO:tasks.workunit.client.0.smithi195.stderr:+ rmdir a 2022-06-27T11:53:32.489 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 20` 2022-06-27T11:53:32.489 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir a 2022-06-27T11:53:32.489 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir a/.snap/one 2022-06-27T11:53:32.490 INFO:tasks.workunit.client.0.smithi195.stderr:+ cat 2022-06-27T11:53:32.515 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir a/.snap/two 2022-06-27T11:53:32.548 INFO:tasks.workunit.client.0.smithi195.stderr:+ chmod 777 a/foo 2022-06-27T11:53:32.549 INFO:tasks.workunit.client.0.smithi195.stderr:+ sync 2022-06-27T11:53:32.606 INFO:tasks.workunit.client.0.smithi195.stderr:+ ps 2022-06-27T11:53:32.614 INFO:tasks.workunit.client.0.smithi195.stdout: PID TTY TIME CMD 2022-06-27T11:53:32.615 INFO:tasks.workunit.client.0.smithi195.stdout: 1282 ? 00:00:00 systemd 2022-06-27T11:53:32.615 INFO:tasks.workunit.client.0.smithi195.stdout: 1285 ? 00:00:00 (sd-pam) 2022-06-27T11:53:32.616 INFO:tasks.workunit.client.0.smithi195.stdout: 81346 ? 00:00:00 sshd 2022-06-27T11:53:32.616 INFO:tasks.workunit.client.0.smithi195.stdout: 108414 ? 00:00:00 timeout 2022-06-27T11:53:32.617 INFO:tasks.workunit.client.0.smithi195.stdout: 108435 ? 00:00:00 snaptest-double 2022-06-27T11:53:32.617 INFO:tasks.workunit.client.0.smithi195.stdout: 108553 ? 00:00:00 ps 2022-06-27T11:53:32.619 INFO:tasks.workunit.client.0.smithi195.stderr:+ rmdir a/.snap/one a/.snap/two 2022-06-27T11:53:32.628 INFO:tasks.workunit.client.0.smithi195.stderr:+ rm a/foo 2022-06-27T11:53:32.629 INFO:tasks.workunit.client.0.smithi195.stderr:+ rmdir a 2022-06-27T11:53:32.634 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 20` 2022-06-27T11:53:32.634 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir a 2022-06-27T11:53:32.635 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir a/.snap/one 2022-06-27T11:53:32.636 INFO:tasks.workunit.client.0.smithi195.stderr:+ cat 2022-06-27T11:53:32.659 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir a/.snap/two 2022-06-27T11:53:32.703 INFO:tasks.workunit.client.0.smithi195.stderr:+ chmod 777 a/foo 2022-06-27T11:53:32.703 INFO:tasks.workunit.client.0.smithi195.stderr:+ sync 2022-06-27T11:53:32.775 INFO:tasks.workunit.client.0.smithi195.stderr:+ ps 2022-06-27T11:53:32.781 INFO:tasks.workunit.client.0.smithi195.stdout: PID TTY TIME CMD 2022-06-27T11:53:32.782 INFO:tasks.workunit.client.0.smithi195.stdout: 1282 ? 00:00:00 systemd 2022-06-27T11:53:32.782 INFO:tasks.workunit.client.0.smithi195.stdout: 1285 ? 00:00:00 (sd-pam) 2022-06-27T11:53:32.783 INFO:tasks.workunit.client.0.smithi195.stdout: 81346 ? 00:00:00 sshd 2022-06-27T11:53:32.783 INFO:tasks.workunit.client.0.smithi195.stdout: 108414 ? 00:00:00 timeout 2022-06-27T11:53:32.784 INFO:tasks.workunit.client.0.smithi195.stdout: 108435 ? 00:00:00 snaptest-double 2022-06-27T11:53:32.784 INFO:tasks.workunit.client.0.smithi195.stdout: 108563 ? 00:00:00 ps 2022-06-27T11:53:32.785 INFO:tasks.workunit.client.0.smithi195.stderr:+ rmdir a/.snap/one a/.snap/two 2022-06-27T11:53:32.791 INFO:tasks.workunit.client.0.smithi195.stderr:+ rm a/foo 2022-06-27T11:53:32.793 INFO:tasks.workunit.client.0.smithi195.stderr:+ rmdir a 2022-06-27T11:53:32.798 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 20` 2022-06-27T11:53:32.798 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir a 2022-06-27T11:53:32.801 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir a/.snap/one 2022-06-27T11:53:32.801 INFO:tasks.workunit.client.0.smithi195.stderr:+ cat 2022-06-27T11:53:32.829 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir a/.snap/two 2022-06-27T11:53:32.867 INFO:tasks.workunit.client.0.smithi195.stderr:+ chmod 777 a/foo 2022-06-27T11:53:32.868 INFO:tasks.workunit.client.0.smithi195.stderr:+ sync 2022-06-27T11:53:32.921 INFO:tasks.workunit.client.0.smithi195.stderr:+ ps 2022-06-27T11:53:32.940 INFO:tasks.workunit.client.0.smithi195.stdout: PID TTY TIME CMD 2022-06-27T11:53:32.941 INFO:tasks.workunit.client.0.smithi195.stdout: 1282 ? 00:00:00 systemd 2022-06-27T11:53:32.941 INFO:tasks.workunit.client.0.smithi195.stdout: 1285 ? 00:00:00 (sd-pam) 2022-06-27T11:53:32.942 INFO:tasks.workunit.client.0.smithi195.stdout: 81346 ? 00:00:00 sshd 2022-06-27T11:53:32.942 INFO:tasks.workunit.client.0.smithi195.stdout: 108414 ? 00:00:00 timeout 2022-06-27T11:53:32.943 INFO:tasks.workunit.client.0.smithi195.stdout: 108435 ? 00:00:00 snaptest-double 2022-06-27T11:53:32.943 INFO:tasks.workunit.client.0.smithi195.stdout: 108573 ? 00:00:00 ps 2022-06-27T11:53:32.944 INFO:tasks.workunit.client.0.smithi195.stderr:+ rmdir a/.snap/one a/.snap/two 2022-06-27T11:53:32.945 INFO:tasks.workunit.client.0.smithi195.stderr:+ rm a/foo 2022-06-27T11:53:32.945 INFO:tasks.workunit.client.0.smithi195.stderr:+ rmdir a 2022-06-27T11:53:32.954 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 20` 2022-06-27T11:53:32.955 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir a 2022-06-27T11:53:32.955 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir a/.snap/one 2022-06-27T11:53:32.955 INFO:tasks.workunit.client.0.smithi195.stderr:+ cat 2022-06-27T11:53:32.974 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir a/.snap/two 2022-06-27T11:53:33.012 INFO:tasks.workunit.client.0.smithi195.stderr:+ chmod 777 a/foo 2022-06-27T11:53:33.014 INFO:tasks.workunit.client.0.smithi195.stderr:+ sync 2022-06-27T11:53:33.138 INFO:tasks.workunit.client.0.smithi195.stderr:+ ps 2022-06-27T11:53:33.146 INFO:tasks.workunit.client.0.smithi195.stdout: PID TTY TIME CMD 2022-06-27T11:53:33.147 INFO:tasks.workunit.client.0.smithi195.stdout: 1282 ? 00:00:00 systemd 2022-06-27T11:53:33.147 INFO:tasks.workunit.client.0.smithi195.stdout: 1285 ? 00:00:00 (sd-pam) 2022-06-27T11:53:33.148 INFO:tasks.workunit.client.0.smithi195.stdout: 81346 ? 00:00:00 sshd 2022-06-27T11:53:33.148 INFO:tasks.workunit.client.0.smithi195.stdout: 108414 ? 00:00:00 timeout 2022-06-27T11:53:33.149 INFO:tasks.workunit.client.0.smithi195.stdout: 108435 ? 00:00:00 snaptest-double 2022-06-27T11:53:33.149 INFO:tasks.workunit.client.0.smithi195.stdout: 108583 ? 00:00:00 ps 2022-06-27T11:53:33.150 INFO:tasks.workunit.client.0.smithi195.stderr:+ rmdir a/.snap/one a/.snap/two 2022-06-27T11:53:33.171 INFO:tasks.workunit.client.0.smithi195.stderr:+ rm a/foo 2022-06-27T11:53:33.172 INFO:tasks.workunit.client.0.smithi195.stderr:+ rmdir a 2022-06-27T11:53:33.185 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 20` 2022-06-27T11:53:33.186 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir a 2022-06-27T11:53:33.188 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir a/.snap/one 2022-06-27T11:53:33.189 INFO:tasks.workunit.client.0.smithi195.stderr:+ cat 2022-06-27T11:53:33.225 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir a/.snap/two 2022-06-27T11:53:33.250 INFO:tasks.workunit.client.0.smithi195.stderr:+ chmod 777 a/foo 2022-06-27T11:53:33.251 INFO:tasks.workunit.client.0.smithi195.stderr:+ sync 2022-06-27T11:53:33.412 INFO:tasks.workunit.client.0.smithi195.stderr:+ ps 2022-06-27T11:53:33.422 INFO:tasks.workunit.client.0.smithi195.stdout: PID TTY TIME CMD 2022-06-27T11:53:33.422 INFO:tasks.workunit.client.0.smithi195.stdout: 1282 ? 00:00:00 systemd 2022-06-27T11:53:33.422 INFO:tasks.workunit.client.0.smithi195.stdout: 1285 ? 00:00:00 (sd-pam) 2022-06-27T11:53:33.423 INFO:tasks.workunit.client.0.smithi195.stdout: 81346 ? 00:00:00 sshd 2022-06-27T11:53:33.424 INFO:tasks.workunit.client.0.smithi195.stdout: 108414 ? 00:00:00 timeout 2022-06-27T11:53:33.424 INFO:tasks.workunit.client.0.smithi195.stdout: 108435 ? 00:00:00 snaptest-double 2022-06-27T11:53:33.425 INFO:tasks.workunit.client.0.smithi195.stdout: 108593 ? 00:00:00 ps 2022-06-27T11:53:33.425 INFO:tasks.workunit.client.0.smithi195.stderr:+ rmdir a/.snap/one a/.snap/two 2022-06-27T11:53:33.742 INFO:tasks.workunit.client.0.smithi195.stderr:+ rm a/foo 2022-06-27T11:53:33.745 INFO:tasks.workunit.client.0.smithi195.stderr:+ rmdir a 2022-06-27T11:53:33.762 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 20` 2022-06-27T11:53:33.763 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir a 2022-06-27T11:53:33.763 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir a/.snap/one 2022-06-27T11:53:33.764 INFO:tasks.workunit.client.0.smithi195.stderr:+ cat 2022-06-27T11:53:33.778 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir a/.snap/two 2022-06-27T11:53:33.805 INFO:tasks.workunit.client.0.smithi195.stderr:+ chmod 777 a/foo 2022-06-27T11:53:33.806 INFO:tasks.workunit.client.0.smithi195.stderr:+ sync 2022-06-27T11:53:33.877 INFO:tasks.workunit.client.0.smithi195.stderr:+ ps 2022-06-27T11:53:33.882 INFO:tasks.workunit.client.0.smithi195.stdout: PID TTY TIME CMD 2022-06-27T11:53:33.882 INFO:tasks.workunit.client.0.smithi195.stdout: 1282 ? 00:00:00 systemd 2022-06-27T11:53:33.883 INFO:tasks.workunit.client.0.smithi195.stdout: 1285 ? 00:00:00 (sd-pam) 2022-06-27T11:53:33.883 INFO:tasks.workunit.client.0.smithi195.stdout: 81346 ? 00:00:00 sshd 2022-06-27T11:53:33.884 INFO:tasks.workunit.client.0.smithi195.stdout: 108414 ? 00:00:00 timeout 2022-06-27T11:53:33.884 INFO:tasks.workunit.client.0.smithi195.stdout: 108435 ? 00:00:00 snaptest-double 2022-06-27T11:53:33.885 INFO:tasks.workunit.client.0.smithi195.stdout: 108603 ? 00:00:00 ps 2022-06-27T11:53:33.886 INFO:tasks.workunit.client.0.smithi195.stderr:+ rmdir a/.snap/one a/.snap/two 2022-06-27T11:53:33.894 INFO:tasks.workunit.client.0.smithi195.stderr:+ rm a/foo 2022-06-27T11:53:33.895 INFO:tasks.workunit.client.0.smithi195.stderr:+ rmdir a 2022-06-27T11:53:33.903 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 20` 2022-06-27T11:53:33.904 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir a 2022-06-27T11:53:33.904 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir a/.snap/one 2022-06-27T11:53:33.904 INFO:tasks.workunit.client.0.smithi195.stderr:+ cat 2022-06-27T11:53:33.922 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir a/.snap/two 2022-06-27T11:53:33.952 INFO:tasks.workunit.client.0.smithi195.stderr:+ chmod 777 a/foo 2022-06-27T11:53:33.953 INFO:tasks.workunit.client.0.smithi195.stderr:+ sync 2022-06-27T11:53:34.029 INFO:tasks.workunit.client.0.smithi195.stderr:+ ps 2022-06-27T11:53:34.030 INFO:tasks.workunit.client.0.smithi195.stdout: PID TTY TIME CMD 2022-06-27T11:53:34.031 INFO:tasks.workunit.client.0.smithi195.stdout: 1282 ? 00:00:00 systemd 2022-06-27T11:53:34.031 INFO:tasks.workunit.client.0.smithi195.stdout: 1285 ? 00:00:00 (sd-pam) 2022-06-27T11:53:34.032 INFO:tasks.workunit.client.0.smithi195.stdout: 81346 ? 00:00:00 sshd 2022-06-27T11:53:34.032 INFO:tasks.workunit.client.0.smithi195.stdout: 108414 ? 00:00:00 timeout 2022-06-27T11:53:34.033 INFO:tasks.workunit.client.0.smithi195.stdout: 108435 ? 00:00:00 snaptest-double 2022-06-27T11:53:34.034 INFO:tasks.workunit.client.0.smithi195.stdout: 108613 ? 00:00:00 ps 2022-06-27T11:53:34.036 INFO:tasks.workunit.client.0.smithi195.stderr:+ rmdir a/.snap/one a/.snap/two 2022-06-27T11:53:34.037 INFO:tasks.workunit.client.0.smithi195.stderr:+ rm a/foo 2022-06-27T11:53:34.037 INFO:tasks.workunit.client.0.smithi195.stderr:+ rmdir a 2022-06-27T11:53:34.055 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 20` 2022-06-27T11:53:34.056 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir a 2022-06-27T11:53:34.057 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir a/.snap/one 2022-06-27T11:53:34.058 INFO:tasks.workunit.client.0.smithi195.stderr:+ cat 2022-06-27T11:53:34.070 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir a/.snap/two 2022-06-27T11:53:34.096 INFO:tasks.workunit.client.0.smithi195.stderr:+ chmod 777 a/foo 2022-06-27T11:53:34.098 INFO:tasks.workunit.client.0.smithi195.stderr:+ sync 2022-06-27T11:53:34.360 INFO:tasks.workunit.client.0.smithi195.stderr:+ ps 2022-06-27T11:53:34.367 INFO:tasks.workunit.client.0.smithi195.stdout: PID TTY TIME CMD 2022-06-27T11:53:34.368 INFO:tasks.workunit.client.0.smithi195.stdout: 1282 ? 00:00:00 systemd 2022-06-27T11:53:34.368 INFO:tasks.workunit.client.0.smithi195.stdout: 1285 ? 00:00:00 (sd-pam) 2022-06-27T11:53:34.369 INFO:tasks.workunit.client.0.smithi195.stdout: 81346 ? 00:00:00 sshd 2022-06-27T11:53:34.369 INFO:tasks.workunit.client.0.smithi195.stdout: 108414 ? 00:00:00 timeout 2022-06-27T11:53:34.370 INFO:tasks.workunit.client.0.smithi195.stdout: 108435 ? 00:00:00 snaptest-double 2022-06-27T11:53:34.370 INFO:tasks.workunit.client.0.smithi195.stdout: 108623 ? 00:00:00 ps 2022-06-27T11:53:34.371 INFO:tasks.workunit.client.0.smithi195.stderr:+ rmdir a/.snap/one a/.snap/two 2022-06-27T11:53:34.382 INFO:tasks.workunit.client.0.smithi195.stderr:+ rm a/foo 2022-06-27T11:53:34.383 INFO:tasks.workunit.client.0.smithi195.stderr:+ rmdir a 2022-06-27T11:53:34.402 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 20` 2022-06-27T11:53:34.403 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir a 2022-06-27T11:53:34.403 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir a/.snap/one 2022-06-27T11:53:34.404 INFO:tasks.workunit.client.0.smithi195.stderr:+ cat 2022-06-27T11:53:34.418 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir a/.snap/two 2022-06-27T11:53:34.443 INFO:tasks.workunit.client.0.smithi195.stderr:+ chmod 777 a/foo 2022-06-27T11:53:34.444 INFO:tasks.workunit.client.0.smithi195.stderr:+ sync 2022-06-27T11:53:34.600 INFO:tasks.workunit.client.0.smithi195.stderr:+ ps 2022-06-27T11:53:34.601 INFO:tasks.workunit.client.0.smithi195.stdout: PID TTY TIME CMD 2022-06-27T11:53:34.601 INFO:tasks.workunit.client.0.smithi195.stdout: 1282 ? 00:00:00 systemd 2022-06-27T11:53:34.602 INFO:tasks.workunit.client.0.smithi195.stdout: 1285 ? 00:00:00 (sd-pam) 2022-06-27T11:53:34.602 INFO:tasks.workunit.client.0.smithi195.stdout: 81346 ? 00:00:00 sshd 2022-06-27T11:53:34.603 INFO:tasks.workunit.client.0.smithi195.stdout: 108414 ? 00:00:00 timeout 2022-06-27T11:53:34.603 INFO:tasks.workunit.client.0.smithi195.stdout: 108435 ? 00:00:00 snaptest-double 2022-06-27T11:53:34.603 INFO:tasks.workunit.client.0.smithi195.stdout: 108633 ? 00:00:00 ps 2022-06-27T11:53:34.604 INFO:tasks.workunit.client.0.smithi195.stderr:+ rmdir a/.snap/one a/.snap/two 2022-06-27T11:53:34.605 INFO:tasks.workunit.client.0.smithi195.stderr:+ rm a/foo 2022-06-27T11:53:34.607 INFO:tasks.workunit.client.0.smithi195.stderr:+ rmdir a 2022-06-27T11:53:34.615 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo OK 2022-06-27T11:53:34.616 INFO:tasks.workunit.client.0.smithi195.stdout:OK 2022-06-27T11:53:34.617 INFO:teuthology.orchestra.run:Running command with timeout 3600 2022-06-27T11:53:34.617 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2022-06-27T11:53:34.704 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-estale.sh... 2022-06-27T11:53:34.705 DEBUG:teuthology.orchestra.run.smithi195:workunit test fs/snaps/snaptest-estale.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5292901bc999daa0d7ac128764a3f27c4233c252 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-estale.sh 2022-06-27T11:53:34.802 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/foo 2022-06-27T11:53:34.821 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo 'We want ENOENT, not ESTALE, here.' 2022-06-27T11:53:34.822 INFO:tasks.workunit.client.0.smithi195.stdout:We want ENOENT, not ESTALE, here. 2022-06-27T11:53:34.823 INFO:tasks.workunit.client.0.smithi195.stderr:++ seq 1 100 2022-06-27T11:53:34.823 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:34.824 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/1 2022-06-27T11:53:34.824 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:34.838 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/1': No such file or directory 2022-06-27T11:53:34.839 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:34.840 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/2 2022-06-27T11:53:34.840 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:34.841 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/2': No such file or directory 2022-06-27T11:53:34.842 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:34.843 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/3 2022-06-27T11:53:34.843 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:34.844 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/3': No such file or directory 2022-06-27T11:53:34.845 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:34.846 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/4 2022-06-27T11:53:34.846 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:34.847 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/4': No such file or directory 2022-06-27T11:53:34.848 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:34.849 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/5 2022-06-27T11:53:34.849 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:34.850 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/5': No such file or directory 2022-06-27T11:53:34.851 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:34.851 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/6 2022-06-27T11:53:34.852 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:34.852 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/6': No such file or directory 2022-06-27T11:53:34.855 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:34.855 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/7 2022-06-27T11:53:34.855 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:34.856 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/7': No such file or directory 2022-06-27T11:53:34.857 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:34.858 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/8 2022-06-27T11:53:34.858 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:34.859 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/8': No such file or directory 2022-06-27T11:53:34.861 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:34.861 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/9 2022-06-27T11:53:34.862 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:34.862 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/9': No such file or directory 2022-06-27T11:53:34.863 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:34.863 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/10 2022-06-27T11:53:34.864 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:34.864 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/10': No such file or directory 2022-06-27T11:53:34.865 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:34.865 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/11 2022-06-27T11:53:34.866 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:34.866 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/11': No such file or directory 2022-06-27T11:53:34.867 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:34.867 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/12 2022-06-27T11:53:34.868 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:34.868 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/12': No such file or directory 2022-06-27T11:53:34.869 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:34.869 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/13 2022-06-27T11:53:34.870 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:34.871 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/13': No such file or directory 2022-06-27T11:53:34.872 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:34.873 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/14 2022-06-27T11:53:34.873 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:34.873 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/14': No such file or directory 2022-06-27T11:53:34.874 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:34.875 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/15 2022-06-27T11:53:34.875 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:34.876 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/15': No such file or directory 2022-06-27T11:53:34.876 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:34.877 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/16 2022-06-27T11:53:34.878 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:34.880 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/16': No such file or directory 2022-06-27T11:53:34.881 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:34.882 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/17 2022-06-27T11:53:34.882 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:34.884 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/17': No such file or directory 2022-06-27T11:53:34.885 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:34.886 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/18 2022-06-27T11:53:34.886 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:34.887 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/18': No such file or directory 2022-06-27T11:53:34.889 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:34.889 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/19 2022-06-27T11:53:34.890 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:34.891 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/19': No such file or directory 2022-06-27T11:53:34.892 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:34.892 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/20 2022-06-27T11:53:34.892 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:34.893 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/20': No such file or directory 2022-06-27T11:53:34.893 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:34.894 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/21 2022-06-27T11:53:34.894 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:34.902 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/21': No such file or directory 2022-06-27T11:53:34.904 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:34.904 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/22 2022-06-27T11:53:34.905 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:34.909 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/22': No such file or directory 2022-06-27T11:53:34.911 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:34.911 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/23 2022-06-27T11:53:34.912 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:34.913 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/23': No such file or directory 2022-06-27T11:53:34.914 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:34.915 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/24 2022-06-27T11:53:34.915 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:34.916 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/24': No such file or directory 2022-06-27T11:53:34.917 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:34.918 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/25 2022-06-27T11:53:34.918 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:34.919 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/25': No such file or directory 2022-06-27T11:53:34.920 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:34.921 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/26 2022-06-27T11:53:34.921 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:34.922 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/26': No such file or directory 2022-06-27T11:53:34.923 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:34.923 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/27 2022-06-27T11:53:34.924 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:34.924 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/27': No such file or directory 2022-06-27T11:53:34.925 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:34.926 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/28 2022-06-27T11:53:34.926 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:34.927 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/28': No such file or directory 2022-06-27T11:53:34.927 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:34.928 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/29 2022-06-27T11:53:34.928 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:34.928 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/29': No such file or directory 2022-06-27T11:53:34.929 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:34.929 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/30 2022-06-27T11:53:34.929 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:34.930 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/30': No such file or directory 2022-06-27T11:53:34.931 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:34.931 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/31 2022-06-27T11:53:34.932 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:34.932 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/31': No such file or directory 2022-06-27T11:53:34.933 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:34.933 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/32 2022-06-27T11:53:34.934 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:34.934 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/32': No such file or directory 2022-06-27T11:53:34.935 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:34.936 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/33 2022-06-27T11:53:34.936 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:34.936 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/33': No such file or directory 2022-06-27T11:53:34.937 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:34.937 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/34 2022-06-27T11:53:34.938 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:34.938 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/34': No such file or directory 2022-06-27T11:53:34.939 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:34.939 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/35 2022-06-27T11:53:34.939 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:34.940 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/35': No such file or directory 2022-06-27T11:53:34.941 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:34.941 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/36 2022-06-27T11:53:34.942 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:34.942 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/36': No such file or directory 2022-06-27T11:53:34.943 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:34.943 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/37 2022-06-27T11:53:34.944 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:34.944 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/37': No such file or directory 2022-06-27T11:53:34.945 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:34.945 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/38 2022-06-27T11:53:34.945 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:34.946 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/38': No such file or directory 2022-06-27T11:53:34.947 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:34.947 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/39 2022-06-27T11:53:34.947 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:34.948 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/39': No such file or directory 2022-06-27T11:53:34.949 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:34.949 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/40 2022-06-27T11:53:34.950 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:34.950 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/40': No such file or directory 2022-06-27T11:53:34.951 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:34.951 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/41 2022-06-27T11:53:34.951 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:34.952 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/41': No such file or directory 2022-06-27T11:53:34.953 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:34.953 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/42 2022-06-27T11:53:34.953 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:34.953 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/42': No such file or directory 2022-06-27T11:53:34.954 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:34.955 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/43 2022-06-27T11:53:34.955 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:34.955 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/43': No such file or directory 2022-06-27T11:53:34.956 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:34.956 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/44 2022-06-27T11:53:34.957 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:34.957 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/44': No such file or directory 2022-06-27T11:53:34.958 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:34.958 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/45 2022-06-27T11:53:34.958 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:34.959 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/45': No such file or directory 2022-06-27T11:53:34.960 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:34.960 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/46 2022-06-27T11:53:34.961 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:34.961 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/46': No such file or directory 2022-06-27T11:53:34.962 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:34.963 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/47 2022-06-27T11:53:34.964 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:34.964 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/47': No such file or directory 2022-06-27T11:53:34.965 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:34.965 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/48 2022-06-27T11:53:34.965 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:34.966 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/48': No such file or directory 2022-06-27T11:53:34.967 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:34.968 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/49 2022-06-27T11:53:34.971 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:34.972 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/49': No such file or directory 2022-06-27T11:53:34.973 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:34.974 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/50 2022-06-27T11:53:34.974 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:34.975 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/50': No such file or directory 2022-06-27T11:53:34.982 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:34.983 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/51 2022-06-27T11:53:34.983 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:34.984 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/51': No such file or directory 2022-06-27T11:53:34.985 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:34.985 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/52 2022-06-27T11:53:34.986 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:34.986 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/52': No such file or directory 2022-06-27T11:53:34.988 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:34.988 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/53 2022-06-27T11:53:34.989 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:34.990 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/53': No such file or directory 2022-06-27T11:53:34.991 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:34.992 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/54 2022-06-27T11:53:34.992 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:34.993 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/54': No such file or directory 2022-06-27T11:53:34.994 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:34.994 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/55 2022-06-27T11:53:34.995 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:34.996 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/55': No such file or directory 2022-06-27T11:53:34.997 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:34.997 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/56 2022-06-27T11:53:34.997 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:34.998 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/56': No such file or directory 2022-06-27T11:53:34.999 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:34.999 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/57 2022-06-27T11:53:34.999 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:35.000 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/57': No such file or directory 2022-06-27T11:53:35.000 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:35.001 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/58 2022-06-27T11:53:35.001 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:35.002 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/58': No such file or directory 2022-06-27T11:53:35.002 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:35.003 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/59 2022-06-27T11:53:35.003 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:35.003 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/59': No such file or directory 2022-06-27T11:53:35.004 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:35.004 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/60 2022-06-27T11:53:35.004 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:35.005 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/60': No such file or directory 2022-06-27T11:53:35.007 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:35.007 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/61 2022-06-27T11:53:35.007 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:35.008 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/61': No such file or directory 2022-06-27T11:53:35.010 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:35.010 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/62 2022-06-27T11:53:35.011 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:35.012 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/62': No such file or directory 2022-06-27T11:53:35.013 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:35.013 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/63 2022-06-27T11:53:35.013 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:35.014 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/63': No such file or directory 2022-06-27T11:53:35.015 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:35.015 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/64 2022-06-27T11:53:35.015 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:35.016 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/64': No such file or directory 2022-06-27T11:53:35.017 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:35.017 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/65 2022-06-27T11:53:35.018 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:35.018 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/65': No such file or directory 2022-06-27T11:53:35.019 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:35.019 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/66 2022-06-27T11:53:35.020 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:35.020 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/66': No such file or directory 2022-06-27T11:53:35.022 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:35.022 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/67 2022-06-27T11:53:35.023 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:35.023 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/67': No such file or directory 2022-06-27T11:53:35.025 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:35.025 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/68 2022-06-27T11:53:35.026 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:35.028 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/68': No such file or directory 2022-06-27T11:53:35.029 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:35.030 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/69 2022-06-27T11:53:35.030 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:35.031 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/69': No such file or directory 2022-06-27T11:53:35.032 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:35.033 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/70 2022-06-27T11:53:35.033 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:35.034 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/70': No such file or directory 2022-06-27T11:53:35.035 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:35.036 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/71 2022-06-27T11:53:35.036 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:35.037 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/71': No such file or directory 2022-06-27T11:53:35.039 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:35.039 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/72 2022-06-27T11:53:35.040 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:35.041 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/72': No such file or directory 2022-06-27T11:53:35.042 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:35.043 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/73 2022-06-27T11:53:35.043 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:35.044 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/73': No such file or directory 2022-06-27T11:53:35.045 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:35.045 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/74 2022-06-27T11:53:35.045 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:35.046 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/74': No such file or directory 2022-06-27T11:53:35.047 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:35.048 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/75 2022-06-27T11:53:35.048 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:35.049 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/75': No such file or directory 2022-06-27T11:53:35.051 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:35.051 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/76 2022-06-27T11:53:35.052 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:35.053 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/76': No such file or directory 2022-06-27T11:53:35.054 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:35.055 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/77 2022-06-27T11:53:35.055 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:35.056 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/77': No such file or directory 2022-06-27T11:53:35.058 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:35.058 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/78 2022-06-27T11:53:35.060 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:35.061 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/78': No such file or directory 2022-06-27T11:53:35.063 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:35.063 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/79 2022-06-27T11:53:35.064 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:35.064 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/79': No such file or directory 2022-06-27T11:53:35.066 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:35.067 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/80 2022-06-27T11:53:35.067 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:35.068 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/80': No such file or directory 2022-06-27T11:53:35.070 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:35.070 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/81 2022-06-27T11:53:35.071 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:35.072 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/81': No such file or directory 2022-06-27T11:53:35.073 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:35.074 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/82 2022-06-27T11:53:35.075 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:35.076 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/82': No such file or directory 2022-06-27T11:53:35.077 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:35.078 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/83 2022-06-27T11:53:35.078 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:35.080 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/83': No such file or directory 2022-06-27T11:53:35.082 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:35.082 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/84 2022-06-27T11:53:35.083 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:35.084 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/84': No such file or directory 2022-06-27T11:53:35.085 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:35.086 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/85 2022-06-27T11:53:35.087 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:35.087 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/85': No such file or directory 2022-06-27T11:53:35.089 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:35.089 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/86 2022-06-27T11:53:35.090 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:35.091 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/86': No such file or directory 2022-06-27T11:53:35.092 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:35.093 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/87 2022-06-27T11:53:35.093 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:35.094 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/87': No such file or directory 2022-06-27T11:53:35.095 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:35.096 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/88 2022-06-27T11:53:35.096 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:35.097 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/88': No such file or directory 2022-06-27T11:53:35.099 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:35.099 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/89 2022-06-27T11:53:35.099 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:35.100 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/89': No such file or directory 2022-06-27T11:53:35.102 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:35.102 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/90 2022-06-27T11:53:35.103 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:35.103 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/90': No such file or directory 2022-06-27T11:53:35.105 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:35.105 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/91 2022-06-27T11:53:35.105 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:35.106 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/91': No such file or directory 2022-06-27T11:53:35.109 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:35.109 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/92 2022-06-27T11:53:35.110 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:35.111 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/92': No such file or directory 2022-06-27T11:53:35.112 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:35.113 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/93 2022-06-27T11:53:35.114 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:35.115 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/93': No such file or directory 2022-06-27T11:53:35.117 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:35.118 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/94 2022-06-27T11:53:35.118 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:35.119 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/94': No such file or directory 2022-06-27T11:53:35.121 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:35.121 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/95 2022-06-27T11:53:35.122 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:35.123 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/95': No such file or directory 2022-06-27T11:53:35.124 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:35.125 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/96 2022-06-27T11:53:35.126 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:35.127 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/96': No such file or directory 2022-06-27T11:53:35.128 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:35.129 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/97 2022-06-27T11:53:35.129 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:35.131 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/97': No such file or directory 2022-06-27T11:53:35.132 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:35.132 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/98 2022-06-27T11:53:35.133 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:35.134 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/98': No such file or directory 2022-06-27T11:53:35.135 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:35.136 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/99 2022-06-27T11:53:35.137 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:35.138 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/99': No such file or directory 2022-06-27T11:53:35.140 INFO:tasks.workunit.client.0.smithi195.stderr:+ for f in `seq 1 100` 2022-06-27T11:53:35.140 INFO:tasks.workunit.client.0.smithi195.stderr:+ stat .snap/foo/100 2022-06-27T11:53:35.141 INFO:tasks.workunit.client.0.smithi195.stderr:+ grep 'No such file' 2022-06-27T11:53:35.141 INFO:tasks.workunit.client.0.smithi195.stdout:stat: cannot statx '.snap/foo/100': No such file or directory 2022-06-27T11:53:35.143 INFO:tasks.workunit.client.0.smithi195.stderr:+ rmdir .snap/foo 2022-06-27T11:53:35.144 INFO:tasks.workunit.client.0.smithi195.stdout:OK 2022-06-27T11:53:35.146 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo OK 2022-06-27T11:53:35.147 INFO:teuthology.orchestra.run:Running command with timeout 3600 2022-06-27T11:53:35.147 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2022-06-27T11:53:35.223 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-git-ceph.sh... 2022-06-27T11:53:35.224 DEBUG:teuthology.orchestra.run.smithi195:workunit test fs/snaps/snaptest-git-ceph.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5292901bc999daa0d7ac128764a3f27c4233c252 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh 2022-06-27T11:53:35.305 INFO:tasks.workunit.client.0.smithi195.stderr:+ set -e 2022-06-27T11:53:35.306 INFO:tasks.workunit.client.0.smithi195.stderr:+ retried=false 2022-06-27T11:53:35.306 INFO:tasks.workunit.client.0.smithi195.stderr:+ trap -- retry EXIT 2022-06-27T11:53:35.306 INFO:tasks.workunit.client.0.smithi195.stderr:+ rm -rf ceph 2022-06-27T11:53:35.307 INFO:tasks.workunit.client.0.smithi195.stderr:+ timeout 1800 git clone git://git.ceph.com/ceph.git 2022-06-27T11:53:35.308 INFO:tasks.workunit.client.0.smithi195.stderr:Cloning into 'ceph'... 2022-06-27T11:53:58.048 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T11:53:58.069 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T11:54:28.300 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T11:54:28.305 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T11:54:52.058 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:54:52.199 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.200+0000 7fda50c74700 1 Processor -- start 2022-06-27T11:54:52.227 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.200+0000 7fda50c74700 1 -- start start 2022-06-27T11:54:52.227 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.202+0000 7fda50c74700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fda4c156670 0x7fda4c0c7440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:54:52.228 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.202+0000 7fda50c74700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fda4c067ed0 0x7fda4c0c4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:54:52.228 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.202+0000 7fda50c74700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda4c0c43d0 con 0x7fda4c069390 2022-06-27T11:54:52.229 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.202+0000 7fda50c74700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda4c068430 con 0x7fda4c067ed0 2022-06-27T11:54:52.229 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.202+0000 7fda50c74700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda4c068710 con 0x7fda4c156670 2022-06-27T11:54:52.230 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.202+0000 7fda49d9b700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fda4c067ed0 0x7fda4c0c4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:54:52.230 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.202+0000 7fda49d9b700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fda4c067ed0 0x7fda4c0c4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:44180/0 (socket says 172.21.15.195:44180) 2022-06-27T11:54:52.230 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.202+0000 7fda49d9b700 1 -- 172.21.15.195:0/3998346593 learned_addr learned my addr 172.21.15.195:0/3998346593 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:54:52.231 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.202+0000 7fda4a59c700 1 --2- 172.21.15.195:0/3998346593 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fda4c156670 0x7fda4c0c7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:54:52.231 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.203+0000 7fda4a59c700 1 -- 172.21.15.195:0/3998346593 >> v2:172.21.15.195:3300/0 conn(0x7fda4c067ed0 msgr2=0x7fda4c0c4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:54:52.231 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.203+0000 7fda4a59c700 1 --2- 172.21.15.195:0/3998346593 >> v2:172.21.15.195:3300/0 conn(0x7fda4c067ed0 0x7fda4c0c4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:54:52.231 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.203+0000 7fda4a59c700 1 -- 172.21.15.195:0/3998346593 >> v1:172.21.15.202:6789/0 conn(0x7fda4c069390 legacy=0x7fda4c1502b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:54:52.231 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.203+0000 7fda4a59c700 1 -- 172.21.15.195:0/3998346593 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda4c0689f0 con 0x7fda4c156670 2022-06-27T11:54:52.232 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.203+0000 7fda4a59c700 1 --2- 172.21.15.195:0/3998346593 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fda4c156670 0x7fda4c0c7440 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fda3c0097a0 tx=0x7fda3c009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2553c6a928e09170 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:54:52.232 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.203+0000 7fda4959a700 1 -- 172.21.15.195:0/3998346593 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda3c011630 con 0x7fda4c156670 2022-06-27T11:54:52.232 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.203+0000 7fda4959a700 1 -- 172.21.15.195:0/3998346593 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda3c011790 con 0x7fda4c156670 2022-06-27T11:54:52.232 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.203+0000 7fda4959a700 1 -- 172.21.15.195:0/3998346593 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda3c011b30 con 0x7fda4c156670 2022-06-27T11:54:52.233 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.204+0000 7fda50c74700 1 -- 172.21.15.195:0/3998346593 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fda4c156670 msgr2=0x7fda4c0c7440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:54:52.233 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.204+0000 7fda50c74700 1 --2- 172.21.15.195:0/3998346593 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fda4c156670 0x7fda4c0c7440 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fda3c0097a0 tx=0x7fda3c009060 comp rx=0 tx=0).stop 2022-06-27T11:54:52.233 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.204+0000 7fda50c74700 1 -- 172.21.15.195:0/3998346593 shutdown_connections 2022-06-27T11:54:52.233 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.204+0000 7fda50c74700 1 --2- 172.21.15.195:0/3998346593 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fda4c156670 0x7fda4c0c7440 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:54:52.234 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.204+0000 7fda50c74700 1 --2- 172.21.15.195:0/3998346593 >> v2:172.21.15.195:3300/0 conn(0x7fda4c067ed0 0x7fda4c0c4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:54:52.234 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.204+0000 7fda50c74700 1 -- 172.21.15.195:0/3998346593 >> 172.21.15.195:0/3998346593 conn(0x7fda4c069e00 msgr2=0x7fda4c06a200 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:54:52.234 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.204+0000 7fda50c74700 1 -- 172.21.15.195:0/3998346593 shutdown_connections 2022-06-27T11:54:52.234 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.205+0000 7fda50c74700 1 -- 172.21.15.195:0/3998346593 wait complete. 2022-06-27T11:54:52.235 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.207+0000 7fda50c74700 1 Processor -- start 2022-06-27T11:54:52.235 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.208+0000 7fda50c74700 1 -- start start 2022-06-27T11:54:52.235 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.209+0000 7fda50c74700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fda4c067ed0 0x7fda4c151f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:54:52.235 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.209+0000 7fda50c74700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fda4c069390 0x7fda4c152440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:54:52.236 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.209+0000 7fda50c74700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fda4c156670 0x7fda4c14bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:54:52.236 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.209+0000 7fda50c74700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda4c1674a0 con 0x7fda4c067ed0 2022-06-27T11:54:52.236 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.209+0000 7fda50c74700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda4c166920 con 0x7fda4c069390 2022-06-27T11:54:52.236 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.209+0000 7fda50c74700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda4c166c00 con 0x7fda4c156670 2022-06-27T11:54:52.237 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.209+0000 7fda4ad9d700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fda4c156670 0x7fda4c14bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:54:52.237 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.209+0000 7fda49d9b700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fda4c069390 0x7fda4c152440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:54:52.237 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.209+0000 7fda4a59c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fda4c067ed0 0x7fda4c151f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:54:52.237 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.209+0000 7fda4ad9d700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fda4c156670 0x7fda4c14bdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3301/0 says I am v2:172.21.15.195:36946/0 (socket says 172.21.15.195:36946) 2022-06-27T11:54:52.237 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.209+0000 7fda4ad9d700 1 -- 172.21.15.195:0/1302096233 learned_addr learned my addr 172.21.15.195:0/1302096233 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:54:52.238 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.210+0000 7fda4a59c700 1 -- 172.21.15.195:0/1302096233 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fda4c156670 msgr2=0x7fda4c14bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:54:52.238 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.210+0000 7fda4a59c700 1 --2- 172.21.15.195:0/1302096233 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fda4c156670 0x7fda4c14bdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:54:52.238 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.210+0000 7fda4a59c700 1 -- 172.21.15.195:0/1302096233 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fda4c069390 msgr2=0x7fda4c152440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:54:52.239 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.210+0000 7fda4a59c700 1 --2- 172.21.15.195:0/1302096233 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fda4c069390 0x7fda4c152440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:54:52.239 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.210+0000 7fda4a59c700 1 -- 172.21.15.195:0/1302096233 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda3c0092e0 con 0x7fda4c067ed0 2022-06-27T11:54:52.239 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.210+0000 7fda4a59c700 1 --2- 172.21.15.195:0/1302096233 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fda4c067ed0 0x7fda4c151f00 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fda3c016000 tx=0x7fda3c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:54:52.239 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.210+0000 7fda3a7fc700 1 -- 172.21.15.195:0/1302096233 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda3c011b10 con 0x7fda4c067ed0 2022-06-27T11:54:52.239 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.210+0000 7fda3a7fc700 1 -- 172.21.15.195:0/1302096233 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda3c01be00 con 0x7fda4c067ed0 2022-06-27T11:54:52.240 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.210+0000 7fda50c74700 1 -- 172.21.15.195:0/1302096233 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda4c166ee0 con 0x7fda4c067ed0 2022-06-27T11:54:52.240 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.210+0000 7fda3a7fc700 1 -- 172.21.15.195:0/1302096233 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda3c025620 con 0x7fda4c067ed0 2022-06-27T11:54:52.240 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.210+0000 7fda50c74700 1 -- 172.21.15.195:0/1302096233 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda4c1519e0 con 0x7fda4c067ed0 2022-06-27T11:54:52.241 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.211+0000 7fda50c74700 1 -- 172.21.15.195:0/1302096233 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda4c0722c0 con 0x7fda4c067ed0 2022-06-27T11:54:52.241 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.211+0000 7fda3a7fc700 1 -- 172.21.15.195:0/1302096233 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fda3c00f050 con 0x7fda4c067ed0 2022-06-27T11:54:52.241 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.212+0000 7fda3a7fc700 1 --2- 172.21.15.195:0/1302096233 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fda28063d00 0x7fda280661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:54:52.241 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.212+0000 7fda3a7fc700 1 -- 172.21.15.195:0/1302096233 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6402+0+0 (secure 0 0 0) 0x7fda3c081fc0 con 0x7fda4c067ed0 2022-06-27T11:54:52.242 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.212+0000 7fda49d9b700 1 --2- 172.21.15.195:0/1302096233 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fda28063d00 0x7fda280661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:54:52.242 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.212+0000 7fda49d9b700 1 --2- 172.21.15.195:0/1302096233 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fda28063d00 0x7fda280661b0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fda4c0c3d70 tx=0x7fda40009120 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:54:52.242 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.216+0000 7fda3a7fc700 1 -- 172.21.15.195:0/1302096233 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fda3c050960 con 0x7fda4c067ed0 2022-06-27T11:54:52.449 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.442+0000 7fda50c74700 1 -- 172.21.15.195:0/1302096233 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fda4c068cd0 con 0x7fda4c067ed0 2022-06-27T11:54:52.453 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.442+0000 7fda3a7fc700 1 -- 172.21.15.195:0/1302096233 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6128 (secure 0 0 0) 0x7fda3c015090 con 0x7fda4c067ed0 2022-06-27T11:54:52.454 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:54:52.455 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4856,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/2970948376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2970948376},{"type":"v1","addr":"172.21.15.202:6835","nonce":2970948376}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:52:55.042048+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4760":{"gid":4760,"name":"b","rank":0,"incarnation":20,"state":"up:active","state_seq":21,"addr":"172.21.15.202:6833/1026219246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1026219246},{"type":"v1","addr":"172.21.15.202:6833","nonce":1026219246}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:54:52.458 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.444+0000 7fda50c74700 1 -- 172.21.15.195:0/1302096233 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fda28063d00 msgr2=0x7fda280661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:54:52.459 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.444+0000 7fda50c74700 1 --2- 172.21.15.195:0/1302096233 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fda28063d00 0x7fda280661b0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fda4c0c3d70 tx=0x7fda40009120 comp rx=0 tx=0).stop 2022-06-27T11:54:52.459 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.444+0000 7fda50c74700 1 -- 172.21.15.195:0/1302096233 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fda4c067ed0 msgr2=0x7fda4c151f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:54:52.460 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.444+0000 7fda50c74700 1 --2- 172.21.15.195:0/1302096233 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fda4c067ed0 0x7fda4c151f00 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fda3c016000 tx=0x7fda3c017000 comp rx=0 tx=0).stop 2022-06-27T11:54:52.461 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.444+0000 7fda50c74700 1 -- 172.21.15.195:0/1302096233 shutdown_connections 2022-06-27T11:54:52.461 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.444+0000 7fda50c74700 1 --2- 172.21.15.195:0/1302096233 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fda28063d00 0x7fda280661b0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:54:52.462 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.444+0000 7fda50c74700 1 --2- 172.21.15.195:0/1302096233 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fda4c067ed0 0x7fda4c151f00 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:54:52.462 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.444+0000 7fda50c74700 1 --2- 172.21.15.195:0/1302096233 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fda4c156670 0x7fda4c14bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:54:52.464 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.444+0000 7fda50c74700 1 --2- 172.21.15.195:0/1302096233 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fda4c069390 0x7fda4c152440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:54:52.464 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.444+0000 7fda50c74700 1 -- 172.21.15.195:0/1302096233 >> 172.21.15.195:0/1302096233 conn(0x7fda4c069e00 msgr2=0x7fda4c0c4860 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:54:52.464 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.444+0000 7fda50c74700 1 -- 172.21.15.195:0/1302096233 shutdown_connections 2022-06-27T11:54:52.465 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.444+0000 7fda50c74700 1 -- 172.21.15.195:0/1302096233 wait complete. 2022-06-27T11:54:52.465 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 25 2022-06-27T11:54:52.467 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2022-06-27T11:54:52.470 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-06-27T11:54:52.470 INFO:teuthology.orchestra.run:waiting for 300 2022-06-27T11:54:52.485 INFO:tasks.ceph.mds.b:Stopped 2022-06-27T11:54:52.486 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:54:52.614 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.615+0000 7fc0ed68c700 1 Processor -- start 2022-06-27T11:54:52.615 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.615+0000 7fc0ed68c700 1 -- start start 2022-06-27T11:54:52.617 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.616+0000 7fc0ed68c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fc0e80c6190 0x7fc0e8157010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:54:52.618 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.616+0000 7fc0ed68c700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc0e8067ed0 0x7fc0e814cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:54:52.618 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.616+0000 7fc0ed68c700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0e8068330 con 0x7fc0e80c40d0 2022-06-27T11:54:52.618 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.616+0000 7fc0ed68c700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0e8156b80 con 0x7fc0e80c6190 2022-06-27T11:54:52.619 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.616+0000 7fc0ed68c700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e806a040 con 0x7fc0e8067ed0 2022-06-27T11:54:52.619 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.616+0000 7fc0e67fc700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fc0e80c6190 0x7fc0e8157010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:54:52.619 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.616+0000 7fc0e67fc700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fc0e80c6190 0x7fc0e8157010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:44196/0 (socket says 172.21.15.195:44196) 2022-06-27T11:54:52.620 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.616+0000 7fc0e67fc700 1 -- 172.21.15.195:0/2774753387 learned_addr learned my addr 172.21.15.195:0/2774753387 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:54:52.620 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.616+0000 7fc0e77fe700 1 --2- 172.21.15.195:0/2774753387 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc0e8067ed0 0x7fc0e814cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:54:52.620 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.616+0000 7fc0e67fc700 1 -- 172.21.15.195:0/2774753387 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc0e8067ed0 msgr2=0x7fc0e814cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:54:52.620 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.616+0000 7fc0e67fc700 1 --2- 172.21.15.195:0/2774753387 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc0e8067ed0 0x7fc0e814cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:54:52.621 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.616+0000 7fc0e67fc700 1 -- 172.21.15.195:0/2774753387 >> v1:172.21.15.202:6789/0 conn(0x7fc0e80c40d0 legacy=0x7fc0e80c44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:54:52.621 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.616+0000 7fc0e67fc700 1 -- 172.21.15.195:0/2774753387 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0e806a300 con 0x7fc0e80c6190 2022-06-27T11:54:52.621 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.616+0000 7fc0e67fc700 1 --2- 172.21.15.195:0/2774753387 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc0e80c6190 0x7fc0e8157010 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc009860 tx=0x7fc0dc009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a7b548283e6fb1a server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:54:52.621 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.617+0000 7fc0e5ffb700 1 -- 172.21.15.195:0/2774753387 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0dc00b680 con 0x7fc0e80c6190 2022-06-27T11:54:52.622 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.617+0000 7fc0e5ffb700 1 -- 172.21.15.195:0/2774753387 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0dc00b7e0 con 0x7fc0e80c6190 2022-06-27T11:54:52.622 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.617+0000 7fc0e5ffb700 1 -- 172.21.15.195:0/2774753387 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0dc00bba0 con 0x7fc0e80c6190 2022-06-27T11:54:52.623 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.617+0000 7fc0ed68c700 1 -- 172.21.15.195:0/2774753387 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc0e80c6190 msgr2=0x7fc0e8157010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:54:52.623 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.617+0000 7fc0ed68c700 1 --2- 172.21.15.195:0/2774753387 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc0e80c6190 0x7fc0e8157010 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc009860 tx=0x7fc0dc009120 comp rx=0 tx=0).stop 2022-06-27T11:54:52.623 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.617+0000 7fc0ed68c700 1 -- 172.21.15.195:0/2774753387 shutdown_connections 2022-06-27T11:54:52.623 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.617+0000 7fc0ed68c700 1 --2- 172.21.15.195:0/2774753387 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc0e8067ed0 0x7fc0e814cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:54:52.624 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.617+0000 7fc0ed68c700 1 --2- 172.21.15.195:0/2774753387 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc0e80c6190 0x7fc0e8157010 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:54:52.624 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.617+0000 7fc0ed68c700 1 -- 172.21.15.195:0/2774753387 >> 172.21.15.195:0/2774753387 conn(0x7fc0e80ccb80 msgr2=0x7fc0e80ccf80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:54:52.625 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.617+0000 7fc0ed68c700 1 -- 172.21.15.195:0/2774753387 shutdown_connections 2022-06-27T11:54:52.625 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.617+0000 7fc0ed68c700 1 -- 172.21.15.195:0/2774753387 wait complete. 2022-06-27T11:54:52.629 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.618+0000 7fc0ed68c700 1 Processor -- start 2022-06-27T11:54:52.629 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.618+0000 7fc0ed68c700 1 -- start start 2022-06-27T11:54:52.629 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.619+0000 7fc0ed68c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc0e8067ed0 0x7fc0e814c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:54:52.630 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.619+0000 7fc0ed68c700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc0e80c40d0 0x7fc0e814ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:54:52.630 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.619+0000 7fc0ed68c700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc0e80c6190 0x7fc0e814f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:54:52.630 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.619+0000 7fc0ed68c700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e815b760 con 0x7fc0e8067ed0 2022-06-27T11:54:52.631 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.619+0000 7fc0ed68c700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e8157f70 con 0x7fc0e80c6190 2022-06-27T11:54:52.631 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.619+0000 7fc0ed68c700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e8157810 con 0x7fc0e80c40d0 2022-06-27T11:54:52.631 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.619+0000 7fc0e6ffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc0e8067ed0 0x7fc0e814c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:54:52.631 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.619+0000 7fc0e6ffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc0e8067ed0 0x7fc0e814c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:44200/0 (socket says 172.21.15.195:44200) 2022-06-27T11:54:52.632 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.619+0000 7fc0e6ffd700 1 -- 172.21.15.195:0/944552429 learned_addr learned my addr 172.21.15.195:0/944552429 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:54:52.632 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.619+0000 7fc0e77fe700 1 --2- 172.21.15.195:0/944552429 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc0e80c6190 0x7fc0e814f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:54:52.642 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.619+0000 7fc0e67fc700 1 --2- 172.21.15.195:0/944552429 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc0e80c40d0 0x7fc0e814ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:54:52.643 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.620+0000 7fc0e6ffd700 1 -- 172.21.15.195:0/944552429 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc0e80c40d0 msgr2=0x7fc0e814ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:54:52.644 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.620+0000 7fc0e6ffd700 1 --2- 172.21.15.195:0/944552429 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc0e80c40d0 0x7fc0e814ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:54:52.644 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.620+0000 7fc0e6ffd700 1 -- 172.21.15.195:0/944552429 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc0e80c6190 msgr2=0x7fc0e814f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:54:52.645 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.620+0000 7fc0e6ffd700 1 --2- 172.21.15.195:0/944552429 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc0e80c6190 0x7fc0e814f990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:54:52.645 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.620+0000 7fc0e6ffd700 1 -- 172.21.15.195:0/944552429 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0dc0093a0 con 0x7fc0e8067ed0 2022-06-27T11:54:52.645 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.620+0000 7fc0e6ffd700 1 --2- 172.21.15.195:0/944552429 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc0e8067ed0 0x7fc0e814c500 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fc0d800c490 tx=0x7fc0d8007e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:54:52.645 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.620+0000 7fc0ceffd700 1 -- 172.21.15.195:0/944552429 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0d80050d0 con 0x7fc0e8067ed0 2022-06-27T11:54:52.646 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.620+0000 7fc0ceffd700 1 -- 172.21.15.195:0/944552429 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0d8005230 con 0x7fc0e8067ed0 2022-06-27T11:54:52.646 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.620+0000 7fc0ceffd700 1 -- 172.21.15.195:0/944552429 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0d80117a0 con 0x7fc0e8067ed0 2022-06-27T11:54:52.646 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.620+0000 7fc0ed68c700 1 -- 172.21.15.195:0/944552429 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0e8157af0 con 0x7fc0e8067ed0 2022-06-27T11:54:52.647 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.620+0000 7fc0ed68c700 1 -- 172.21.15.195:0/944552429 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0e8158460 con 0x7fc0e8067ed0 2022-06-27T11:54:52.647 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.621+0000 7fc0ed68c700 1 -- 172.21.15.195:0/944552429 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0e80722c0 con 0x7fc0e8067ed0 2022-06-27T11:54:52.647 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.621+0000 7fc0ceffd700 1 -- 172.21.15.195:0/944552429 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fc0d8019030 con 0x7fc0e8067ed0 2022-06-27T11:54:52.648 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.622+0000 7fc0ceffd700 1 --2- 172.21.15.195:0/944552429 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc0c8063d00 0x7fc0c80661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:54:52.648 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.622+0000 7fc0e67fc700 1 --2- 172.21.15.195:0/944552429 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc0c8063d00 0x7fc0c80661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:54:52.648 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.622+0000 7fc0ceffd700 1 -- 172.21.15.195:0/944552429 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6402+0+0 (secure 0 0 0) 0x7fc0d807f9c0 con 0x7fc0e8067ed0 2022-06-27T11:54:52.648 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.622+0000 7fc0e67fc700 1 --2- 172.21.15.195:0/944552429 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc0c8063d00 0x7fc0c80661b0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc009160 tx=0x7fc0dc017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:54:52.649 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.626+0000 7fc0ceffd700 1 -- 172.21.15.195:0/944552429 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fc0d8049c90 con 0x7fc0e8067ed0 2022-06-27T11:54:52.851 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.852+0000 7fc0ed68c700 1 -- 172.21.15.195:0/944552429 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc0e8158620 con 0x7fc0e8067ed0 2022-06-27T11:54:52.852 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.852+0000 7fc0ceffd700 1 -- 172.21.15.195:0/944552429 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6128 (secure 0 0 0) 0x7fc0d8017020 con 0x7fc0e8067ed0 2022-06-27T11:54:52.853 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:54:52.853 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4856,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/2970948376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2970948376},{"type":"v1","addr":"172.21.15.202:6835","nonce":2970948376}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:52:55.042048+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4760":{"gid":4760,"name":"b","rank":0,"incarnation":20,"state":"up:active","state_seq":21,"addr":"172.21.15.202:6833/1026219246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1026219246},{"type":"v1","addr":"172.21.15.202:6833","nonce":1026219246}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:54:52.855 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.854+0000 7fc0ed68c700 1 -- 172.21.15.195:0/944552429 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc0c8063d00 msgr2=0x7fc0c80661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:54:52.855 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.854+0000 7fc0ed68c700 1 --2- 172.21.15.195:0/944552429 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc0c8063d00 0x7fc0c80661b0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc009160 tx=0x7fc0dc017000 comp rx=0 tx=0).stop 2022-06-27T11:54:52.855 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.854+0000 7fc0ed68c700 1 -- 172.21.15.195:0/944552429 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc0e8067ed0 msgr2=0x7fc0e814c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:54:52.856 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.854+0000 7fc0ed68c700 1 --2- 172.21.15.195:0/944552429 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc0e8067ed0 0x7fc0e814c500 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fc0d800c490 tx=0x7fc0d8007e40 comp rx=0 tx=0).stop 2022-06-27T11:54:52.856 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.854+0000 7fc0ed68c700 1 -- 172.21.15.195:0/944552429 shutdown_connections 2022-06-27T11:54:52.856 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.854+0000 7fc0ed68c700 1 --2- 172.21.15.195:0/944552429 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc0c8063d00 0x7fc0c80661b0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:54:52.857 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.854+0000 7fc0ed68c700 1 --2- 172.21.15.195:0/944552429 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc0e8067ed0 0x7fc0e814c500 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:54:52.857 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.854+0000 7fc0ed68c700 1 --2- 172.21.15.195:0/944552429 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc0e80c6190 0x7fc0e814f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:54:52.857 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.854+0000 7fc0ed68c700 1 --2- 172.21.15.195:0/944552429 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc0e80c40d0 0x7fc0e814ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:54:52.857 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.854+0000 7fc0ed68c700 1 -- 172.21.15.195:0/944552429 >> 172.21.15.195:0/944552429 conn(0x7fc0e80ccb80 msgr2=0x7fc0e80cc260 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:54:52.858 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.854+0000 7fc0ed68c700 1 -- 172.21.15.195:0/944552429 shutdown_connections 2022-06-27T11:54:52.858 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:52.854+0000 7fc0ed68c700 1 -- 172.21.15.195:0/944552429 wait complete. 2022-06-27T11:54:52.858 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 25 2022-06-27T11:54:52.868 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2022-06-27T11:54:54.869 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:54:55.004 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.004+0000 7f5c899df700 1 Processor -- start 2022-06-27T11:54:55.004 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.004+0000 7f5c899df700 1 -- start start 2022-06-27T11:54:55.007 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.005+0000 7f5c899df700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5c84150500 0x7f5c84063af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:54:55.008 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.005+0000 7f5c899df700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f5c8406a450 0x7f5c840c48d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:54:55.008 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.005+0000 7f5c899df700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c84064030 con 0x7f5c841508d0 2022-06-27T11:54:55.008 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.005+0000 7f5c899df700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c840c4160 con 0x7f5c8406a450 2022-06-27T11:54:55.008 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.005+0000 7f5c899df700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c840c4440 con 0x7f5c84150500 2022-06-27T11:54:55.009 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.005+0000 7f5c827fc700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f5c8406a450 0x7f5c840c48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:54:55.009 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.005+0000 7f5c837fe700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f5c841508d0 0x7f5c840bbf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:50402/0 (socket says 172.21.15.195:50402) 2022-06-27T11:54:55.009 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.005+0000 7f5c837fe700 1 -- 172.21.15.195:0/2912565643 learned_addr learned my addr 172.21.15.195:0/2912565643 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:54:55.009 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.006+0000 7f5c82ffd700 1 --2- 172.21.15.195:0/2912565643 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5c84150500 0x7f5c84063af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:54:55.010 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.006+0000 7f5c827fc700 1 -- 172.21.15.195:0/2912565643 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5c84150500 msgr2=0x7f5c84063af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:54:55.010 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.006+0000 7f5c827fc700 1 --2- 172.21.15.195:0/2912565643 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5c84150500 0x7f5c84063af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:54:55.010 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.006+0000 7f5c827fc700 1 -- 172.21.15.195:0/2912565643 >> v1:172.21.15.195:6789/0 conn(0x7f5c841508d0 legacy=0x7f5c840bbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:54:55.010 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.006+0000 7f5c827fc700 1 -- 172.21.15.195:0/2912565643 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c8415a800 con 0x7f5c8406a450 2022-06-27T11:54:55.011 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.006+0000 7f5c81ffb700 1 -- 172.21.15.195:0/2912565643 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1286217452 0 0) 0x7f5c7c001490 con 0x7f5c841508d0 2022-06-27T11:54:55.011 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.006+0000 7f5c82ffd700 1 --2- 172.21.15.195:0/2912565643 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5c84150500 0x7f5c84063af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T11:54:55.011 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.006+0000 7f5c827fc700 1 --2- 172.21.15.195:0/2912565643 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5c8406a450 0x7f5c840c48d0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f5c740097a0 tx=0x7f5c74009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4ab75f6d6f0f972 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:54:55.011 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.006+0000 7f5c81ffb700 1 -- 172.21.15.195:0/2912565643 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c74011630 con 0x7f5c8406a450 2022-06-27T11:54:55.012 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.006+0000 7f5c81ffb700 1 -- 172.21.15.195:0/2912565643 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c74011790 con 0x7f5c8406a450 2022-06-27T11:54:55.012 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.006+0000 7f5c81ffb700 1 -- 172.21.15.195:0/2912565643 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c74011b30 con 0x7f5c8406a450 2022-06-27T11:54:55.012 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.006+0000 7f5c899df700 1 -- 172.21.15.195:0/2912565643 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5c8406a450 msgr2=0x7f5c840c48d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:54:55.013 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.006+0000 7f5c899df700 1 --2- 172.21.15.195:0/2912565643 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5c8406a450 0x7f5c840c48d0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f5c740097a0 tx=0x7f5c74009060 comp rx=0 tx=0).stop 2022-06-27T11:54:55.013 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.007+0000 7f5c899df700 1 -- 172.21.15.195:0/2912565643 shutdown_connections 2022-06-27T11:54:55.014 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.007+0000 7f5c899df700 1 --2- 172.21.15.195:0/2912565643 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5c84150500 0x7f5c84063af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:54:55.014 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.007+0000 7f5c899df700 1 --2- 172.21.15.195:0/2912565643 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5c8406a450 0x7f5c840c48d0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:54:55.014 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.007+0000 7f5c899df700 1 -- 172.21.15.195:0/2912565643 >> 172.21.15.195:0/2912565643 conn(0x7f5c840c9b00 msgr2=0x7f5c840c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:54:55.016 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.007+0000 7f5c899df700 1 -- 172.21.15.195:0/2912565643 shutdown_connections 2022-06-27T11:54:55.016 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.007+0000 7f5c899df700 1 -- 172.21.15.195:0/2912565643 wait complete. 2022-06-27T11:54:55.016 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.008+0000 7f5c899df700 1 Processor -- start 2022-06-27T11:54:55.016 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.008+0000 7f5c899df700 1 -- start start 2022-06-27T11:54:55.027 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.009+0000 7f5c899df700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5c8406a450 0x7f5c84166ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:54:55.028 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.009+0000 7f5c899df700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5c84150500 0x7f5c84153060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:54:55.031 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.009+0000 7f5c899df700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5c841508d0 0x7f5c840c2950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:54:55.032 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.009+0000 7f5c899df700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c84152540 con 0x7f5c8406a450 2022-06-27T11:54:55.034 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.009+0000 7f5c899df700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c841587e0 con 0x7f5c84150500 2022-06-27T11:54:55.035 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.009+0000 7f5c899df700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c84166a40 con 0x7f5c841508d0 2022-06-27T11:54:55.035 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.009+0000 7f5c82ffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5c8406a450 0x7f5c84166ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:54:55.036 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.009+0000 7f5c82ffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5c8406a450 0x7f5c84166ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:44220/0 (socket says 172.21.15.195:44220) 2022-06-27T11:54:55.036 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.009+0000 7f5c82ffd700 1 -- 172.21.15.195:0/2155029838 learned_addr learned my addr 172.21.15.195:0/2155029838 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:54:55.036 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.009+0000 7f5c827fc700 1 --2- 172.21.15.195:0/2155029838 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5c84150500 0x7f5c84153060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:54:55.037 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.009+0000 7f5c837fe700 1 --2- 172.21.15.195:0/2155029838 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5c841508d0 0x7f5c840c2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:54:55.037 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.009+0000 7f5c82ffd700 1 -- 172.21.15.195:0/2155029838 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5c841508d0 msgr2=0x7f5c840c2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:54:55.037 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.009+0000 7f5c82ffd700 1 --2- 172.21.15.195:0/2155029838 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5c841508d0 0x7f5c840c2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:54:55.037 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.009+0000 7f5c82ffd700 1 -- 172.21.15.195:0/2155029838 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5c84150500 msgr2=0x7f5c84153060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:54:55.037 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.009+0000 7f5c82ffd700 1 --2- 172.21.15.195:0/2155029838 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5c84150500 0x7f5c84153060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:54:55.038 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.009+0000 7f5c82ffd700 1 -- 172.21.15.195:0/2155029838 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c740092e0 con 0x7f5c8406a450 2022-06-27T11:54:55.038 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.009+0000 7f5c827fc700 1 --2- 172.21.15.195:0/2155029838 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5c84150500 0x7f5c84153060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T11:54:55.039 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.009+0000 7f5c82ffd700 1 --2- 172.21.15.195:0/2155029838 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5c8406a450 0x7f5c84166ed0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f5c7800c8b0 tx=0x7f5c78008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:54:55.040 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.009+0000 7f5c6affd700 1 -- 172.21.15.195:0/2155029838 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c780058c0 con 0x7f5c8406a450 2022-06-27T11:54:55.040 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.009+0000 7f5c6affd700 1 -- 172.21.15.195:0/2155029838 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c78005a20 con 0x7f5c8406a450 2022-06-27T11:54:55.040 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.009+0000 7f5c6affd700 1 -- 172.21.15.195:0/2155029838 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c78005dc0 con 0x7f5c8406a450 2022-06-27T11:54:55.040 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.009+0000 7f5c899df700 1 -- 172.21.15.195:0/2155029838 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c84158a10 con 0x7f5c8406a450 2022-06-27T11:54:55.041 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.009+0000 7f5c899df700 1 -- 172.21.15.195:0/2155029838 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c84158fd0 con 0x7f5c8406a450 2022-06-27T11:54:55.041 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.011+0000 7f5c899df700 1 -- 172.21.15.195:0/2155029838 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c840722c0 con 0x7f5c8406a450 2022-06-27T11:54:55.041 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.011+0000 7f5c6affd700 1 -- 172.21.15.195:0/2155029838 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f5c78020070 con 0x7f5c8406a450 2022-06-27T11:54:55.041 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.011+0000 7f5c6affd700 1 --2- 172.21.15.195:0/2155029838 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5c64063d30 0x7f5c640661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:54:55.041 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.011+0000 7f5c6affd700 1 -- 172.21.15.195:0/2155029838 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6402+0+0 (secure 0 0 0) 0x7f5c780806b0 con 0x7f5c8406a450 2022-06-27T11:54:55.042 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.011+0000 7f5c827fc700 1 --2- 172.21.15.195:0/2155029838 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5c64063d30 0x7f5c640661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:54:55.042 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.012+0000 7f5c827fc700 1 --2- 172.21.15.195:0/2155029838 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5c64063d30 0x7f5c640661e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5c740097a0 tx=0x7f5c74017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:54:55.042 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.016+0000 7f5c6affd700 1 -- 172.21.15.195:0/2155029838 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f5c7804f140 con 0x7f5c8406a450 2022-06-27T11:54:55.258 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.250+0000 7f5c899df700 1 -- 172.21.15.195:0/2155029838 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5c840ba8e0 con 0x7f5c8406a450 2022-06-27T11:54:55.259 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.250+0000 7f5c6affd700 1 -- 172.21.15.195:0/2155029838 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6128 (secure 0 0 0) 0x7f5c7801e070 con 0x7f5c8406a450 2022-06-27T11:54:55.259 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:54:55.260 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4856,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/2970948376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2970948376},{"type":"v1","addr":"172.21.15.202:6835","nonce":2970948376}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:52:55.042048+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4760":{"gid":4760,"name":"b","rank":0,"incarnation":20,"state":"up:active","state_seq":21,"addr":"172.21.15.202:6833/1026219246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1026219246},{"type":"v1","addr":"172.21.15.202:6833","nonce":1026219246}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:54:55.261 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.253+0000 7f5c899df700 1 -- 172.21.15.195:0/2155029838 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5c64063d30 msgr2=0x7f5c640661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:54:55.261 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.253+0000 7f5c899df700 1 --2- 172.21.15.195:0/2155029838 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5c64063d30 0x7f5c640661e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5c740097a0 tx=0x7f5c74017000 comp rx=0 tx=0).stop 2022-06-27T11:54:55.261 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.253+0000 7f5c899df700 1 -- 172.21.15.195:0/2155029838 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5c8406a450 msgr2=0x7f5c84166ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:54:55.275 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.253+0000 7f5c899df700 1 --2- 172.21.15.195:0/2155029838 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5c8406a450 0x7f5c84166ed0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f5c7800c8b0 tx=0x7f5c78008040 comp rx=0 tx=0).stop 2022-06-27T11:54:55.275 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.253+0000 7f5c899df700 1 -- 172.21.15.195:0/2155029838 shutdown_connections 2022-06-27T11:54:55.276 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.253+0000 7f5c899df700 1 --2- 172.21.15.195:0/2155029838 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5c64063d30 0x7f5c640661e0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:54:55.276 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.253+0000 7f5c899df700 1 --2- 172.21.15.195:0/2155029838 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5c8406a450 0x7f5c84166ed0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:54:55.277 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.253+0000 7f5c899df700 1 --2- 172.21.15.195:0/2155029838 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5c841508d0 0x7f5c840c2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:54:55.277 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.253+0000 7f5c899df700 1 --2- 172.21.15.195:0/2155029838 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5c84150500 0x7f5c84153060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:54:55.277 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.253+0000 7f5c899df700 1 -- 172.21.15.195:0/2155029838 >> 172.21.15.195:0/2155029838 conn(0x7f5c840c9b00 msgr2=0x7f5c840c36d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:54:55.278 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.253+0000 7f5c899df700 1 -- 172.21.15.195:0/2155029838 shutdown_connections 2022-06-27T11:54:55.278 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:55.253+0000 7f5c899df700 1 -- 172.21.15.195:0/2155029838 wait complete. 2022-06-27T11:54:55.279 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 25 2022-06-27T11:54:55.281 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2022-06-27T11:54:57.282 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:54:57.432 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.416+0000 7f6dbf644700 1 Processor -- start 2022-06-27T11:54:57.433 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.416+0000 7f6dbf644700 1 -- start start 2022-06-27T11:54:57.433 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.417+0000 7f6dbf644700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f6db806f640 0x7f6db8156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:54:57.434 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.417+0000 7f6dbf644700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f6db80c4160 0x7f6db80c1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:54:57.434 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.417+0000 7f6dbf644700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6db80c39e0 0x7f6db81535a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:54:57.435 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.417+0000 7f6dbf644700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6db80c1a20 con 0x7f6db806f640 2022-06-27T11:54:57.435 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.417+0000 7f6dbf644700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6db80c1130 con 0x7f6db80c4160 2022-06-27T11:54:57.436 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.417+0000 7f6dbf644700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6db806ea90 con 0x7f6db80c39e0 2022-06-27T11:54:57.436 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.417+0000 7f6dbd3e0700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f6db806f640 0x7f6db8156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:54:57.437 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.417+0000 7f6dbd3e0700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f6db806f640 0x7f6db8156ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:44232/0 (socket says 172.21.15.195:44232) 2022-06-27T11:54:57.437 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.417+0000 7f6dbd3e0700 1 -- 172.21.15.195:0/875634826 learned_addr learned my addr 172.21.15.195:0/875634826 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:54:57.438 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.417+0000 7f6dbcbdf700 1 --2- 172.21.15.195:0/875634826 >> v2:172.21.15.202:3300/0 conn(0x7f6db80c4160 0x7f6db80c1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:54:57.438 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.417+0000 7f6dbdbe1700 1 --2- 172.21.15.195:0/875634826 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6db80c39e0 0x7f6db81535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:54:57.439 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.417+0000 7f6dbd3e0700 1 -- 172.21.15.195:0/875634826 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6db80c39e0 msgr2=0x7f6db81535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:54:57.439 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.417+0000 7f6dbd3e0700 1 --2- 172.21.15.195:0/875634826 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6db80c39e0 0x7f6db81535a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:54:57.440 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.417+0000 7f6dbd3e0700 1 -- 172.21.15.195:0/875634826 >> v2:172.21.15.202:3300/0 conn(0x7f6db80c4160 msgr2=0x7f6db80c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:54:57.440 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.417+0000 7f6dbd3e0700 1 --2- 172.21.15.195:0/875634826 >> v2:172.21.15.202:3300/0 conn(0x7f6db80c4160 0x7f6db80c1eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:54:57.441 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.417+0000 7f6dbd3e0700 1 -- 172.21.15.195:0/875634826 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6db806ed70 con 0x7f6db806f640 2022-06-27T11:54:57.441 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.417+0000 7f6dbcbdf700 1 --2- 172.21.15.195:0/875634826 >> v2:172.21.15.202:3300/0 conn(0x7f6db80c4160 0x7f6db80c1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T11:54:57.458 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.418+0000 7f6dbd3e0700 1 --2- 172.21.15.195:0/875634826 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6db806f640 0x7f6db8156ed0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f6db806dd70 tx=0x7f6da8009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3b56954e75a0929 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:54:57.458 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.418+0000 7f6da7fff700 1 -- 172.21.15.195:0/875634826 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da800b6d0 con 0x7f6db806f640 2022-06-27T11:54:57.459 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.418+0000 7f6da7fff700 1 -- 172.21.15.195:0/875634826 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6da800b830 con 0x7f6db806f640 2022-06-27T11:54:57.459 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.418+0000 7f6da7fff700 1 -- 172.21.15.195:0/875634826 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da800bbd0 con 0x7f6db806f640 2022-06-27T11:54:57.459 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.418+0000 7f6dbf644700 1 -- 172.21.15.195:0/875634826 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6db806f640 msgr2=0x7f6db8156ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:54:57.460 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.418+0000 7f6dbf644700 1 --2- 172.21.15.195:0/875634826 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6db806f640 0x7f6db8156ed0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f6db806dd70 tx=0x7f6da8009120 comp rx=0 tx=0).stop 2022-06-27T11:54:57.460 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.418+0000 7f6dbf644700 1 -- 172.21.15.195:0/875634826 shutdown_connections 2022-06-27T11:54:57.460 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.418+0000 7f6dbf644700 1 --2- 172.21.15.195:0/875634826 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6db80c39e0 0x7f6db81535a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:54:57.460 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.418+0000 7f6dbf644700 1 --2- 172.21.15.195:0/875634826 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6db806f640 0x7f6db8156ed0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:54:57.461 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.418+0000 7f6dbf644700 1 --2- 172.21.15.195:0/875634826 >> v2:172.21.15.202:3300/0 conn(0x7f6db80c4160 0x7f6db80c1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:54:57.461 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.418+0000 7f6dbf644700 1 -- 172.21.15.195:0/875634826 >> 172.21.15.195:0/875634826 conn(0x7f6db80c6350 msgr2=0x7f6db80c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:54:57.461 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.418+0000 7f6dbf644700 1 -- 172.21.15.195:0/875634826 shutdown_connections 2022-06-27T11:54:57.461 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.418+0000 7f6dbf644700 1 -- 172.21.15.195:0/875634826 wait complete. 2022-06-27T11:54:57.462 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.420+0000 7f6dbf644700 1 Processor -- start 2022-06-27T11:54:57.462 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.420+0000 7f6dbf644700 1 -- start start 2022-06-27T11:54:57.463 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.420+0000 7f6dbf644700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6db806f640 0x7f6db806bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:54:57.463 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.421+0000 7f6dbf644700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6db80c39e0 0x7f6db80ba610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:54:57.463 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.421+0000 7f6dbf644700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6db80c4160 0x7f6db80bab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:54:57.463 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.421+0000 7f6dbf644700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6db815a160 con 0x7f6db806f640 2022-06-27T11:54:57.464 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.421+0000 7f6dbf644700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6db8159ba0 con 0x7f6db80c4160 2022-06-27T11:54:57.464 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.421+0000 7f6dbf644700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6db8159e80 con 0x7f6db80c39e0 2022-06-27T11:54:57.465 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.421+0000 7f6dbd3e0700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6db806f640 0x7f6db806bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:54:57.466 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.421+0000 7f6dbd3e0700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6db806f640 0x7f6db806bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:44238/0 (socket says 172.21.15.195:44238) 2022-06-27T11:54:57.466 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.421+0000 7f6dbd3e0700 1 -- 172.21.15.195:0/1198523276 learned_addr learned my addr 172.21.15.195:0/1198523276 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:54:57.467 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.421+0000 7f6dbdbe1700 1 --2- 172.21.15.195:0/1198523276 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6db80c4160 0x7f6db80bab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:54:57.467 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.421+0000 7f6dbcbdf700 1 --2- 172.21.15.195:0/1198523276 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6db80c39e0 0x7f6db80ba610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:54:57.468 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.421+0000 7f6dbd3e0700 1 -- 172.21.15.195:0/1198523276 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6db80c39e0 msgr2=0x7f6db80ba610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:54:57.468 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.421+0000 7f6dbd3e0700 1 --2- 172.21.15.195:0/1198523276 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6db80c39e0 0x7f6db80ba610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:54:57.469 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.421+0000 7f6dbd3e0700 1 -- 172.21.15.195:0/1198523276 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6db80c4160 msgr2=0x7f6db80bab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:54:57.469 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.421+0000 7f6dbd3e0700 1 --2- 172.21.15.195:0/1198523276 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6db80c4160 0x7f6db80bab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:54:57.470 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.421+0000 7f6dbd3e0700 1 -- 172.21.15.195:0/1198523276 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6da80093a0 con 0x7f6db806f640 2022-06-27T11:54:57.470 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.421+0000 7f6dbd3e0700 1 --2- 172.21.15.195:0/1198523276 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6db806f640 0x7f6db806bfe0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f6da800a010 tx=0x7f6da8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:54:57.471 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.421+0000 7f6da57fa700 1 -- 172.21.15.195:0/1198523276 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da8003880 con 0x7f6db806f640 2022-06-27T11:54:57.471 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.421+0000 7f6da57fa700 1 -- 172.21.15.195:0/1198523276 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6da8003a00 con 0x7f6db806f640 2022-06-27T11:54:57.472 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.421+0000 7f6dbf644700 1 -- 172.21.15.195:0/1198523276 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6db815b0c0 con 0x7f6db806f640 2022-06-27T11:54:57.472 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.421+0000 7f6da57fa700 1 -- 172.21.15.195:0/1198523276 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da8003dc0 con 0x7f6db806f640 2022-06-27T11:54:57.473 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.421+0000 7f6dbf644700 1 -- 172.21.15.195:0/1198523276 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6db815b990 con 0x7f6db806f640 2022-06-27T11:54:57.474 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.422+0000 7f6dbf644700 1 -- 172.21.15.195:0/1198523276 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6db80ba460 con 0x7f6db806f640 2022-06-27T11:54:57.474 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.423+0000 7f6da57fa700 1 -- 172.21.15.195:0/1198523276 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f6da802b030 con 0x7f6db806f640 2022-06-27T11:54:57.474 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.423+0000 7f6da57fa700 1 --2- 172.21.15.195:0/1198523276 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6d98063d00 0x7f6d980661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:54:57.475 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.423+0000 7f6da57fa700 1 -- 172.21.15.195:0/1198523276 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6402+0+0 (secure 0 0 0) 0x7f6da80822d0 con 0x7f6db806f640 2022-06-27T11:54:57.475 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.423+0000 7f6dbcbdf700 1 --2- 172.21.15.195:0/1198523276 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6d98063d00 0x7f6d980661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:54:57.476 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.424+0000 7f6dbcbdf700 1 --2- 172.21.15.195:0/1198523276 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6d98063d00 0x7f6d980661b0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f6db814f2c0 tx=0x7f6db4008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:54:57.476 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.427+0000 7f6da57fa700 1 -- 172.21.15.195:0/1198523276 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f6da8050d60 con 0x7f6db806f640 2022-06-27T11:54:57.676 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.651+0000 7f6dbf644700 1 -- 172.21.15.195:0/1198523276 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6db8159fc0 con 0x7f6db806f640 2022-06-27T11:54:57.676 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.651+0000 7f6da57fa700 1 -- 172.21.15.195:0/1198523276 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6128 (secure 0 0 0) 0x7f6db8159fc0 con 0x7f6db806f640 2022-06-27T11:54:57.678 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:54:57.679 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4856,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/2970948376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2970948376},{"type":"v1","addr":"172.21.15.202:6835","nonce":2970948376}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:52:55.042048+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4760":{"gid":4760,"name":"b","rank":0,"incarnation":20,"state":"up:active","state_seq":21,"addr":"172.21.15.202:6833/1026219246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1026219246},{"type":"v1","addr":"172.21.15.202:6833","nonce":1026219246}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:54:57.682 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.653+0000 7f6dbf644700 1 -- 172.21.15.195:0/1198523276 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6d98063d00 msgr2=0x7f6d980661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:54:57.683 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.653+0000 7f6dbf644700 1 --2- 172.21.15.195:0/1198523276 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6d98063d00 0x7f6d980661b0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f6db814f2c0 tx=0x7f6db4008040 comp rx=0 tx=0).stop 2022-06-27T11:54:57.683 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.653+0000 7f6dbf644700 1 -- 172.21.15.195:0/1198523276 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6db806f640 msgr2=0x7f6db806bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:54:57.684 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.653+0000 7f6dbf644700 1 --2- 172.21.15.195:0/1198523276 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6db806f640 0x7f6db806bfe0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f6da800a010 tx=0x7f6da8017000 comp rx=0 tx=0).stop 2022-06-27T11:54:57.685 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.653+0000 7f6dbf644700 1 -- 172.21.15.195:0/1198523276 shutdown_connections 2022-06-27T11:54:57.685 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.653+0000 7f6dbf644700 1 --2- 172.21.15.195:0/1198523276 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6d98063d00 0x7f6d980661b0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:54:57.686 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.653+0000 7f6dbf644700 1 --2- 172.21.15.195:0/1198523276 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6db806f640 0x7f6db806bfe0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:54:57.686 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.653+0000 7f6dbf644700 1 --2- 172.21.15.195:0/1198523276 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6db80c4160 0x7f6db80bab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:54:57.686 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.653+0000 7f6dbf644700 1 --2- 172.21.15.195:0/1198523276 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6db80c39e0 0x7f6db80ba610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:54:57.687 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.653+0000 7f6dbf644700 1 -- 172.21.15.195:0/1198523276 >> 172.21.15.195:0/1198523276 conn(0x7f6db80c6350 msgr2=0x7f6db814bf40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:54:57.688 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.653+0000 7f6dbf644700 1 -- 172.21.15.195:0/1198523276 shutdown_connections 2022-06-27T11:54:57.689 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:57.653+0000 7f6dbf644700 1 -- 172.21.15.195:0/1198523276 wait complete. 2022-06-27T11:54:57.689 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 25 2022-06-27T11:54:57.707 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2022-06-27T11:54:58.369 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T11:54:58.385 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T11:54:59.710 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:54:59.860 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:59.860+0000 7f4357a32700 1 Processor -- start 2022-06-27T11:54:59.861 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:59.860+0000 7f4357a32700 1 -- start start 2022-06-27T11:54:59.873 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:59.861+0000 7f4357a32700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f435006a690 0x7f435015a890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:54:59.876 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:59.861+0000 7f4357a32700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f435006d050 0x7f435015e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:54:59.877 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:59.861+0000 7f4357a32700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f435006aa60 0x7f43500c7c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:54:59.877 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:59.861+0000 7f4357a32700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4350069be0 con 0x7f435006d050 2022-06-27T11:54:59.878 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:59.861+0000 7f4357a32700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f435015edb0 con 0x7f435006a690 2022-06-27T11:54:59.878 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:59.861+0000 7f4357a32700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f435015ae90 con 0x7f435006aa60 2022-06-27T11:54:59.878 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:59.861+0000 7f4354fcd700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f435006d050 0x7f435015e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:54:59.879 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:59.861+0000 7f4354fcd700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f435006d050 0x7f435015e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:44246/0 (socket says 172.21.15.195:44246) 2022-06-27T11:54:59.879 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:59.861+0000 7f4354fcd700 1 -- 172.21.15.195:0/3924772805 learned_addr learned my addr 172.21.15.195:0/3924772805 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:54:59.880 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:59.861+0000 7f4355fcf700 1 --2- 172.21.15.195:0/3924772805 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f435006aa60 0x7f43500c7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:54:59.880 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:59.861+0000 7f43557ce700 1 --2- 172.21.15.195:0/3924772805 >> v2:172.21.15.202:3300/0 conn(0x7f435006a690 0x7f435015a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:54:59.885 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:59.862+0000 7f4354fcd700 1 -- 172.21.15.195:0/3924772805 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f435006aa60 msgr2=0x7f43500c7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:54:59.885 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:59.862+0000 7f4354fcd700 1 --2- 172.21.15.195:0/3924772805 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f435006aa60 0x7f43500c7c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:54:59.885 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:59.862+0000 7f4354fcd700 1 -- 172.21.15.195:0/3924772805 >> v2:172.21.15.202:3300/0 conn(0x7f435006a690 msgr2=0x7f435015a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:54:59.886 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:59.862+0000 7f4354fcd700 1 --2- 172.21.15.195:0/3924772805 >> v2:172.21.15.202:3300/0 conn(0x7f435006a690 0x7f435015a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:54:59.886 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:59.862+0000 7f4354fcd700 1 -- 172.21.15.195:0/3924772805 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f435015b170 con 0x7f435006d050 2022-06-27T11:54:59.887 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:59.862+0000 7f4354fcd700 1 --2- 172.21.15.195:0/3924772805 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f435006d050 0x7f435015e870 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f4350065b00 tx=0x7f4340009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a634e3f91bfcba2 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:54:59.887 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:59.862+0000 7f433ffff700 1 -- 172.21.15.195:0/3924772805 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4340011630 con 0x7f435006d050 2022-06-27T11:54:59.888 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:59.862+0000 7f433ffff700 1 -- 172.21.15.195:0/3924772805 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4340011790 con 0x7f435006d050 2022-06-27T11:54:59.888 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:59.862+0000 7f433ffff700 1 -- 172.21.15.195:0/3924772805 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4340011b30 con 0x7f435006d050 2022-06-27T11:54:59.889 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:59.862+0000 7f4357a32700 1 -- 172.21.15.195:0/3924772805 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f435006d050 msgr2=0x7f435015e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:54:59.889 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:59.862+0000 7f4357a32700 1 --2- 172.21.15.195:0/3924772805 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f435006d050 0x7f435015e870 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f4350065b00 tx=0x7f4340009120 comp rx=0 tx=0).stop 2022-06-27T11:54:59.891 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:59.862+0000 7f4357a32700 1 -- 172.21.15.195:0/3924772805 shutdown_connections 2022-06-27T11:54:59.893 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:59.862+0000 7f4357a32700 1 --2- 172.21.15.195:0/3924772805 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f435006aa60 0x7f43500c7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:54:59.897 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:59.862+0000 7f4357a32700 1 --2- 172.21.15.195:0/3924772805 >> v2:172.21.15.202:3300/0 conn(0x7f435006a690 0x7f435015a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:54:59.897 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:59.862+0000 7f4357a32700 1 --2- 172.21.15.195:0/3924772805 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f435006d050 0x7f435015e870 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:54:59.898 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:59.863+0000 7f4357a32700 1 -- 172.21.15.195:0/3924772805 >> 172.21.15.195:0/3924772805 conn(0x7f4350063af0 msgr2=0x7f4350063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:54:59.898 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:59.863+0000 7f4357a32700 1 -- 172.21.15.195:0/3924772805 shutdown_connections 2022-06-27T11:54:59.899 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:59.863+0000 7f4357a32700 1 -- 172.21.15.195:0/3924772805 wait complete. 2022-06-27T11:54:59.899 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:59.864+0000 7f4357a32700 1 Processor -- start 2022-06-27T11:54:59.899 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:59.864+0000 7f4357a32700 1 -- start start 2022-06-27T11:54:59.900 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:59.865+0000 7f4357a32700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f435006a690 0x7f435015ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:54:59.900 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:59.865+0000 7f4357a32700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f435006aa60 0x7f435016e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:54:59.901 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:59.865+0000 7f4357a32700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f435006d050 0x7f435016ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:54:59.901 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:59.865+0000 7f4357a32700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43500c21d0 con 0x7f435006d050 2022-06-27T11:54:59.902 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:59.865+0000 7f4357a32700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f435015daa0 con 0x7f435006a690 2022-06-27T11:54:59.903 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:59.865+0000 7f4357a32700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f435015c840 con 0x7f435006aa60 2022-06-27T11:54:59.903 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:59.865+0000 7f4355fcf700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f435006d050 0x7f435016ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:54:59.904 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:59.865+0000 7f4355fcf700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f435006d050 0x7f435016ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:44256/0 (socket says 172.21.15.195:44256) 2022-06-27T11:54:59.905 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:59.865+0000 7f4355fcf700 1 -- 172.21.15.195:0/1415602730 learned_addr learned my addr 172.21.15.195:0/1415602730 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:54:59.905 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:59.865+0000 7f4354fcd700 1 --2- 172.21.15.195:0/1415602730 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f435006aa60 0x7f435016e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:54:59.906 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:59.865+0000 7f43557ce700 1 --2- 172.21.15.195:0/1415602730 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f435006a690 0x7f435015ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:54:59.916 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:59.865+0000 7f4355fcf700 1 -- 172.21.15.195:0/1415602730 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f435006aa60 msgr2=0x7f435016e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:54:59.917 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:59.865+0000 7f4355fcf700 1 --2- 172.21.15.195:0/1415602730 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f435006aa60 0x7f435016e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:54:59.917 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:59.865+0000 7f4355fcf700 1 -- 172.21.15.195:0/1415602730 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f435006a690 msgr2=0x7f435015ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:54:59.917 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:59.865+0000 7f4355fcf700 1 --2- 172.21.15.195:0/1415602730 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f435006a690 0x7f435015ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:54:59.918 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:59.865+0000 7f4355fcf700 1 -- 172.21.15.195:0/1415602730 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43400093a0 con 0x7f435006d050 2022-06-27T11:54:59.918 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:59.865+0000 7f4355fcf700 1 --2- 172.21.15.195:0/1415602730 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f435006d050 0x7f435016ebb0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f4348009ec0 tx=0x7f434800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:54:59.918 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:59.866+0000 7f433d7fa700 1 -- 172.21.15.195:0/1415602730 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f434800cb70 con 0x7f435006d050 2022-06-27T11:54:59.919 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:59.866+0000 7f433d7fa700 1 -- 172.21.15.195:0/1415602730 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f434800f040 con 0x7f435006d050 2022-06-27T11:54:59.919 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:59.866+0000 7f433d7fa700 1 -- 172.21.15.195:0/1415602730 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4348014480 con 0x7f435006d050 2022-06-27T11:54:59.920 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:59.866+0000 7f4357a32700 1 -- 172.21.15.195:0/1415602730 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f435015c510 con 0x7f435006d050 2022-06-27T11:54:59.920 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:59.866+0000 7f4357a32700 1 -- 172.21.15.195:0/1415602730 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f435015dcf0 con 0x7f435006d050 2022-06-27T11:54:59.920 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:59.867+0000 7f4357a32700 1 -- 172.21.15.195:0/1415602730 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f435015e030 con 0x7f435006d050 2022-06-27T11:54:59.921 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:59.867+0000 7f433d7fa700 1 -- 172.21.15.195:0/1415602730 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f43480147f0 con 0x7f435006d050 2022-06-27T11:54:59.921 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:59.867+0000 7f433d7fa700 1 --2- 172.21.15.195:0/1415602730 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4330063d00 0x7f43300661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:54:59.921 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:59.868+0000 7f433d7fa700 1 -- 172.21.15.195:0/1415602730 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6402+0+0 (secure 0 0 0) 0x7f4348081840 con 0x7f435006d050 2022-06-27T11:54:59.922 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:59.868+0000 7f43557ce700 1 --2- 172.21.15.195:0/1415602730 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4330063d00 0x7f43300661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:54:59.922 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:59.868+0000 7f43557ce700 1 --2- 172.21.15.195:0/1415602730 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4330063d00 0x7f43300661b0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f434c009bd0 tx=0x7f434c009060 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:54:59.923 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:54:59.872+0000 7f433d7fa700 1 -- 172.21.15.195:0/1415602730 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f43480502d0 con 0x7f435006d050 2022-06-27T11:55:00.095 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:00.092+0000 7f4357a32700 1 -- 172.21.15.195:0/1415602730 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f43500722c0 con 0x7f435006d050 2022-06-27T11:55:00.095 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:00.092+0000 7f433d7fa700 1 -- 172.21.15.195:0/1415602730 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6128 (secure 0 0 0) 0x7f4348018090 con 0x7f435006d050 2022-06-27T11:55:00.096 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:55:00.097 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4856,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/2970948376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2970948376},{"type":"v1","addr":"172.21.15.202:6835","nonce":2970948376}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:52:55.042048+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4760":{"gid":4760,"name":"b","rank":0,"incarnation":20,"state":"up:active","state_seq":21,"addr":"172.21.15.202:6833/1026219246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1026219246},{"type":"v1","addr":"172.21.15.202:6833","nonce":1026219246}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:55:00.108 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:00.094+0000 7f4357a32700 1 -- 172.21.15.195:0/1415602730 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4330063d00 msgr2=0x7f43300661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:00.109 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:00.094+0000 7f4357a32700 1 --2- 172.21.15.195:0/1415602730 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4330063d00 0x7f43300661b0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f434c009bd0 tx=0x7f434c009060 comp rx=0 tx=0).stop 2022-06-27T11:55:00.110 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:00.094+0000 7f4357a32700 1 -- 172.21.15.195:0/1415602730 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f435006d050 msgr2=0x7f435016ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:00.111 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:00.094+0000 7f4357a32700 1 --2- 172.21.15.195:0/1415602730 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f435006d050 0x7f435016ebb0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f4348009ec0 tx=0x7f434800b040 comp rx=0 tx=0).stop 2022-06-27T11:55:00.112 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:00.094+0000 7f4357a32700 1 -- 172.21.15.195:0/1415602730 shutdown_connections 2022-06-27T11:55:00.112 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:00.094+0000 7f4357a32700 1 --2- 172.21.15.195:0/1415602730 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4330063d00 0x7f43300661b0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:00.113 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:00.094+0000 7f4357a32700 1 --2- 172.21.15.195:0/1415602730 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f435006d050 0x7f435016ebb0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:00.113 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:00.094+0000 7f4357a32700 1 --2- 172.21.15.195:0/1415602730 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f435006a690 0x7f435015ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:00.115 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:00.094+0000 7f4357a32700 1 --2- 172.21.15.195:0/1415602730 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f435006aa60 0x7f435016e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:00.116 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:00.094+0000 7f4357a32700 1 -- 172.21.15.195:0/1415602730 >> 172.21.15.195:0/1415602730 conn(0x7f4350063af0 msgr2=0x7f435014df40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:55:00.117 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:00.094+0000 7f4357a32700 1 -- 172.21.15.195:0/1415602730 shutdown_connections 2022-06-27T11:55:00.117 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:00.094+0000 7f4357a32700 1 -- 172.21.15.195:0/1415602730 wait complete. 2022-06-27T11:55:00.118 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 25 2022-06-27T11:55:00.120 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2022-06-27T11:55:02.122 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:55:02.277 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.277+0000 7f4d75ee5700 1 Processor -- start 2022-06-27T11:55:02.277 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.277+0000 7f4d75ee5700 1 -- start start 2022-06-27T11:55:02.283 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.278+0000 7f4d75ee5700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4d7006e850 0x7f4d701570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:02.284 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.278+0000 7f4d75ee5700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f4d700c34a0 0x7f4d700c2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:02.284 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.278+0000 7f4d75ee5700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d7006fba0 con 0x7f4d7006fef0 2022-06-27T11:55:02.284 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.278+0000 7f4d75ee5700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d701575e0 con 0x7f4d700c34a0 2022-06-27T11:55:02.285 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.278+0000 7f4d75ee5700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d7006f5e0 con 0x7f4d7006e850 2022-06-27T11:55:02.285 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.278+0000 7f4d6ffff700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f4d700c34a0 0x7f4d700c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:02.286 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.278+0000 7f4d6f7fe700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4d7006e850 0x7f4d701570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:02.286 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.278+0000 7f4d6ffff700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f4d700c34a0 0x7f4d700c2920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3300/0 says I am v2:172.21.15.195:42392/0 (socket says 172.21.15.195:42392) 2022-06-27T11:55:02.286 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.278+0000 7f4d6ffff700 1 -- 172.21.15.195:0/1579492103 learned_addr learned my addr 172.21.15.195:0/1579492103 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:55:02.287 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.279+0000 7f4d6e7fc700 1 -- 172.21.15.195:0/1579492103 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4285122971 0 0) 0x7f4d7006fba0 con 0x7f4d7006fef0 2022-06-27T11:55:02.287 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.279+0000 7f4d6e7fc700 1 -- 172.21.15.195:0/1579492103 --> v1:172.21.15.202:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d50003430 con 0x7f4d7006fef0 2022-06-27T11:55:02.287 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.279+0000 7f4d6ffff700 1 -- 172.21.15.195:0/1579492103 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4d7006e850 msgr2=0x7f4d701570a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:55:02.288 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.279+0000 7f4d6ffff700 1 --2- 172.21.15.195:0/1579492103 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4d7006e850 0x7f4d701570a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:02.288 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.279+0000 7f4d6ffff700 1 -- 172.21.15.195:0/1579492103 >> v1:172.21.15.202:6789/0 conn(0x7f4d7006fef0 legacy=0x7f4d70069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:02.288 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.279+0000 7f4d6ffff700 1 -- 172.21.15.195:0/1579492103 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d7006f8c0 con 0x7f4d700c34a0 2022-06-27T11:55:02.289 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.279+0000 7f4d6f7fe700 1 --2- 172.21.15.195:0/1579492103 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4d7006e850 0x7f4d701570a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:55:02.289 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.279+0000 7f4d6ffff700 1 --2- 172.21.15.195:0/1579492103 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4d700c34a0 0x7f4d700c2920 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4d6800d1a0 tx=0x7f4d6800ba50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2078ac642319a583 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:55:02.289 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.279+0000 7f4d6e7fc700 1 -- 172.21.15.195:0/1579492103 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d68010070 con 0x7f4d700c34a0 2022-06-27T11:55:02.290 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.279+0000 7f4d6e7fc700 1 -- 172.21.15.195:0/1579492103 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d6800ddf0 con 0x7f4d700c34a0 2022-06-27T11:55:02.290 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.279+0000 7f4d6e7fc700 1 -- 172.21.15.195:0/1579492103 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d6800b2e0 con 0x7f4d700c34a0 2022-06-27T11:55:02.291 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.280+0000 7f4d75ee5700 1 -- 172.21.15.195:0/1579492103 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4d700c34a0 msgr2=0x7f4d700c2920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:02.291 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.280+0000 7f4d75ee5700 1 --2- 172.21.15.195:0/1579492103 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4d700c34a0 0x7f4d700c2920 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4d6800d1a0 tx=0x7f4d6800ba50 comp rx=0 tx=0).stop 2022-06-27T11:55:02.291 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.280+0000 7f4d75ee5700 1 -- 172.21.15.195:0/1579492103 shutdown_connections 2022-06-27T11:55:02.291 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.280+0000 7f4d75ee5700 1 --2- 172.21.15.195:0/1579492103 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4d700c34a0 0x7f4d700c2920 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:02.292 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.280+0000 7f4d75ee5700 1 --2- 172.21.15.195:0/1579492103 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4d7006e850 0x7f4d701570a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:02.292 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.280+0000 7f4d75ee5700 1 -- 172.21.15.195:0/1579492103 >> 172.21.15.195:0/1579492103 conn(0x7f4d700c6350 msgr2=0x7f4d700c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:55:02.292 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.280+0000 7f4d75ee5700 1 -- 172.21.15.195:0/1579492103 shutdown_connections 2022-06-27T11:55:02.292 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.280+0000 7f4d75ee5700 1 -- 172.21.15.195:0/1579492103 wait complete. 2022-06-27T11:55:02.293 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.281+0000 7f4d75ee5700 1 Processor -- start 2022-06-27T11:55:02.293 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.281+0000 7f4d75ee5700 1 -- start start 2022-06-27T11:55:02.293 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.282+0000 7f4d75ee5700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4d7006e850 0x7f4d7014dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:02.293 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.282+0000 7f4d75ee5700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4d700c34a0 0x7f4d701654e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:02.293 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.282+0000 7f4d75ee5700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4d701548a0 0x7f4d70165a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:02.294 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.282+0000 7f4d75ee5700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d701619d0 con 0x7f4d701548a0 2022-06-27T11:55:02.294 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.282+0000 7f4d75ee5700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d70156080 con 0x7f4d700c34a0 2022-06-27T11:55:02.294 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.282+0000 7f4d75ee5700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d701567a0 con 0x7f4d7006e850 2022-06-27T11:55:02.295 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.282+0000 7f4d6ffff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4d701548a0 0x7f4d70165a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:02.300 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.282+0000 7f4d6ffff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4d701548a0 0x7f4d70165a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:44272/0 (socket says 172.21.15.195:44272) 2022-06-27T11:55:02.300 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.282+0000 7f4d6ffff700 1 -- 172.21.15.195:0/2271491761 learned_addr learned my addr 172.21.15.195:0/2271491761 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:55:02.301 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.282+0000 7f4d6f7fe700 1 --2- 172.21.15.195:0/2271491761 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4d7006e850 0x7f4d7014dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:02.301 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.282+0000 7f4d6effd700 1 --2- 172.21.15.195:0/2271491761 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4d700c34a0 0x7f4d701654e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:02.302 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.282+0000 7f4d6ffff700 1 -- 172.21.15.195:0/2271491761 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4d7006e850 msgr2=0x7f4d7014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:02.302 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.282+0000 7f4d6ffff700 1 --2- 172.21.15.195:0/2271491761 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4d7006e850 0x7f4d7014dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:02.302 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.282+0000 7f4d6ffff700 1 -- 172.21.15.195:0/2271491761 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4d700c34a0 msgr2=0x7f4d701654e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:02.302 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.282+0000 7f4d6ffff700 1 --2- 172.21.15.195:0/2271491761 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4d700c34a0 0x7f4d701654e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:02.303 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.282+0000 7f4d6ffff700 1 -- 172.21.15.195:0/2271491761 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d580092e0 con 0x7f4d701548a0 2022-06-27T11:55:02.303 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.282+0000 7f4d6ffff700 1 --2- 172.21.15.195:0/2271491761 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4d701548a0 0x7f4d70165a20 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f4d700650f0 tx=0x7f4d6801a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:55:02.303 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.283+0000 7f4d677fe700 1 -- 172.21.15.195:0/2271491761 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d68009610 con 0x7f4d701548a0 2022-06-27T11:55:02.303 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.283+0000 7f4d677fe700 1 -- 172.21.15.195:0/2271491761 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d68009770 con 0x7f4d701548a0 2022-06-27T11:55:02.304 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.283+0000 7f4d75ee5700 1 -- 172.21.15.195:0/2271491761 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d6800bcf0 con 0x7f4d701548a0 2022-06-27T11:55:02.304 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.283+0000 7f4d677fe700 1 -- 172.21.15.195:0/2271491761 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d68009b10 con 0x7f4d701548a0 2022-06-27T11:55:02.304 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.283+0000 7f4d75ee5700 1 -- 172.21.15.195:0/2271491761 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d70156d60 con 0x7f4d701548a0 2022-06-27T11:55:02.304 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.283+0000 7f4d75ee5700 1 -- 172.21.15.195:0/2271491761 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d700722c0 con 0x7f4d701548a0 2022-06-27T11:55:02.305 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.284+0000 7f4d677fe700 1 -- 172.21.15.195:0/2271491761 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f4d68004040 con 0x7f4d701548a0 2022-06-27T11:55:02.306 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.284+0000 7f4d677fe700 1 --2- 172.21.15.195:0/2271491761 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4d54063d00 0x7f4d540661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:02.306 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.284+0000 7f4d677fe700 1 -- 172.21.15.195:0/2271491761 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6402+0+0 (secure 0 0 0) 0x7f4d68084bb0 con 0x7f4d701548a0 2022-06-27T11:55:02.307 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.284+0000 7f4d6f7fe700 1 --2- 172.21.15.195:0/2271491761 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4d54063d00 0x7f4d540661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:02.307 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.285+0000 7f4d6f7fe700 1 --2- 172.21.15.195:0/2271491761 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4d54063d00 0x7f4d540661b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4d580092b0 tx=0x7f4d58008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:55:02.307 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.289+0000 7f4d677fe700 1 -- 172.21.15.195:0/2271491761 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f4d6808b080 con 0x7f4d701548a0 2022-06-27T11:55:02.513 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.513+0000 7f4d75ee5700 1 -- 172.21.15.195:0/2271491761 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4d7006f240 con 0x7f4d701548a0 2022-06-27T11:55:02.514 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.513+0000 7f4d677fe700 1 -- 172.21.15.195:0/2271491761 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6128 (secure 0 0 0) 0x7f4d68053640 con 0x7f4d701548a0 2022-06-27T11:55:02.515 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:55:02.515 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4856,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/2970948376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2970948376},{"type":"v1","addr":"172.21.15.202:6835","nonce":2970948376}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:52:55.042048+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4760":{"gid":4760,"name":"b","rank":0,"incarnation":20,"state":"up:active","state_seq":21,"addr":"172.21.15.202:6833/1026219246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1026219246},{"type":"v1","addr":"172.21.15.202:6833","nonce":1026219246}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:55:02.517 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.515+0000 7f4d75ee5700 1 -- 172.21.15.195:0/2271491761 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4d54063d00 msgr2=0x7f4d540661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:02.530 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.515+0000 7f4d75ee5700 1 --2- 172.21.15.195:0/2271491761 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4d54063d00 0x7f4d540661b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4d580092b0 tx=0x7f4d58008040 comp rx=0 tx=0).stop 2022-06-27T11:55:02.530 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.515+0000 7f4d75ee5700 1 -- 172.21.15.195:0/2271491761 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4d701548a0 msgr2=0x7f4d70165a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:02.532 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.515+0000 7f4d75ee5700 1 --2- 172.21.15.195:0/2271491761 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4d701548a0 0x7f4d70165a20 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f4d700650f0 tx=0x7f4d6801a000 comp rx=0 tx=0).stop 2022-06-27T11:55:02.532 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.515+0000 7f4d75ee5700 1 -- 172.21.15.195:0/2271491761 shutdown_connections 2022-06-27T11:55:02.532 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.515+0000 7f4d75ee5700 1 --2- 172.21.15.195:0/2271491761 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4d54063d00 0x7f4d540661b0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:02.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.515+0000 7f4d75ee5700 1 --2- 172.21.15.195:0/2271491761 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4d701548a0 0x7f4d70165a20 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:02.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.515+0000 7f4d75ee5700 1 --2- 172.21.15.195:0/2271491761 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4d7006e850 0x7f4d7014dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:02.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.515+0000 7f4d75ee5700 1 --2- 172.21.15.195:0/2271491761 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4d700c34a0 0x7f4d701654e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:02.534 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.516+0000 7f4d75ee5700 1 -- 172.21.15.195:0/2271491761 >> 172.21.15.195:0/2271491761 conn(0x7f4d700c6350 msgr2=0x7f4d7006c810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:55:02.534 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.516+0000 7f4d75ee5700 1 -- 172.21.15.195:0/2271491761 shutdown_connections 2022-06-27T11:55:02.535 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:02.516+0000 7f4d75ee5700 1 -- 172.21.15.195:0/2271491761 wait complete. 2022-06-27T11:55:02.535 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 25 2022-06-27T11:55:02.539 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2022-06-27T11:55:04.540 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:55:04.684 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.682+0000 7fd87a2ed700 1 Processor -- start 2022-06-27T11:55:04.684 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.682+0000 7fd87a2ed700 1 -- start start 2022-06-27T11:55:04.684 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.683+0000 7fd87a2ed700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd874150950 0x7fd874156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:04.685 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.683+0000 7fd87a2ed700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8740c76b0 con 0x7fd87406f620 2022-06-27T11:55:04.685 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.683+0000 7fd87a2ed700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8740c8250 con 0x7fd874150580 2022-06-27T11:55:04.685 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.683+0000 7fd87a2ed700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd874157430 con 0x7fd874150950 2022-06-27T11:55:04.685 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.683+0000 7fd8737fe700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd874150950 0x7fd874156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:04.686 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.683+0000 7fd87888a700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fd87406f620 0x7fd87406d6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:50468/0 (socket says 172.21.15.195:50468) 2022-06-27T11:55:04.686 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.683+0000 7fd87888a700 1 -- 172.21.15.195:0/2255047201 learned_addr learned my addr 172.21.15.195:0/2255047201 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:55:04.686 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.684+0000 7fd8737fe700 1 -- 172.21.15.195:0/2255047201 >> v1:172.21.15.202:6789/0 conn(0x7fd874150580 legacy=0x7fd8740c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:04.686 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.684+0000 7fd8737fe700 1 -- 172.21.15.195:0/2255047201 >> v1:172.21.15.195:6789/0 conn(0x7fd87406f620 legacy=0x7fd87406d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:04.686 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.684+0000 7fd8737fe700 1 -- 172.21.15.195:0/2255047201 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8740bad70 con 0x7fd874150950 2022-06-27T11:55:04.687 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.684+0000 7fd872ffd700 1 -- 172.21.15.195:0/2255047201 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2983792807 0 0) 0x7fd8740c76b0 con 0x7fd87406f620 2022-06-27T11:55:04.687 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.684+0000 7fd8737fe700 1 --2- 172.21.15.195:0/2255047201 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd874150950 0x7fd874156ef0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fd8680097a0 tx=0x7fd868009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66b92e88f4594037 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:55:04.687 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.684+0000 7fd872ffd700 1 -- 172.21.15.195:0/2255047201 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd868011630 con 0x7fd874150950 2022-06-27T11:55:04.687 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.684+0000 7fd872ffd700 1 -- 172.21.15.195:0/2255047201 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd868011790 con 0x7fd874150950 2022-06-27T11:55:04.688 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.684+0000 7fd872ffd700 1 -- 172.21.15.195:0/2255047201 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd868011b30 con 0x7fd874150950 2022-06-27T11:55:04.691 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.684+0000 7fd87a2ed700 1 -- 172.21.15.195:0/2255047201 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd874150950 msgr2=0x7fd874156ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:04.691 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.684+0000 7fd87a2ed700 1 --2- 172.21.15.195:0/2255047201 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd874150950 0x7fd874156ef0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fd8680097a0 tx=0x7fd868009060 comp rx=0 tx=0).stop 2022-06-27T11:55:04.692 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.685+0000 7fd87a2ed700 1 -- 172.21.15.195:0/2255047201 shutdown_connections 2022-06-27T11:55:04.692 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.685+0000 7fd87a2ed700 1 --2- 172.21.15.195:0/2255047201 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd874150950 0x7fd874156ef0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:04.692 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.685+0000 7fd87a2ed700 1 -- 172.21.15.195:0/2255047201 >> 172.21.15.195:0/2255047201 conn(0x7fd874063af0 msgr2=0x7fd874063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:55:04.692 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.685+0000 7fd87a2ed700 1 -- 172.21.15.195:0/2255047201 shutdown_connections 2022-06-27T11:55:04.693 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.685+0000 7fd87a2ed700 1 -- 172.21.15.195:0/2255047201 wait complete. 2022-06-27T11:55:04.693 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.686+0000 7fd87a2ed700 1 Processor -- start 2022-06-27T11:55:04.693 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.686+0000 7fd87a2ed700 1 -- start start 2022-06-27T11:55:04.693 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.687+0000 7fd87a2ed700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd87406f620 0x7fd87414e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:04.694 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.687+0000 7fd87a2ed700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd874150580 0x7fd8740bcaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:04.694 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.687+0000 7fd87a2ed700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd874150950 0x7fd874166c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:04.695 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.687+0000 7fd87a2ed700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd874166760 con 0x7fd874150580 2022-06-27T11:55:04.695 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.687+0000 7fd87a2ed700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd874169ed0 con 0x7fd874150950 2022-06-27T11:55:04.695 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.687+0000 7fd87a2ed700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd87416a1b0 con 0x7fd87406f620 2022-06-27T11:55:04.695 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.687+0000 7fd8737fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd874150580 0x7fd8740bcaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:04.696 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.687+0000 7fd8737fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd874150580 0x7fd8740bcaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:44284/0 (socket says 172.21.15.195:44284) 2022-06-27T11:55:04.696 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.687+0000 7fd8737fe700 1 -- 172.21.15.195:0/956231982 learned_addr learned my addr 172.21.15.195:0/956231982 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:55:04.696 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.687+0000 7fd87888a700 1 --2- 172.21.15.195:0/956231982 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd874150950 0x7fd874166c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:04.697 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.687+0000 7fd873fff700 1 --2- 172.21.15.195:0/956231982 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd87406f620 0x7fd87414e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:04.697 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.687+0000 7fd8737fe700 1 -- 172.21.15.195:0/956231982 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd87406f620 msgr2=0x7fd87414e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:04.698 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.687+0000 7fd8737fe700 1 --2- 172.21.15.195:0/956231982 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd87406f620 0x7fd87414e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:04.698 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.687+0000 7fd8737fe700 1 -- 172.21.15.195:0/956231982 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd874150950 msgr2=0x7fd874166c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:04.699 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.687+0000 7fd8737fe700 1 --2- 172.21.15.195:0/956231982 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd874150950 0x7fd874166c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:04.699 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.687+0000 7fd8737fe700 1 -- 172.21.15.195:0/956231982 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8680092e0 con 0x7fd874150580 2022-06-27T11:55:04.700 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.687+0000 7fd873fff700 1 --2- 172.21.15.195:0/956231982 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd87406f620 0x7fd87414e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T11:55:04.700 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.687+0000 7fd8737fe700 1 --2- 172.21.15.195:0/956231982 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd874150580 0x7fd8740bcaf0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fd874065b00 tx=0x7fd868017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:55:04.701 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.687+0000 7fd85bfff700 1 -- 172.21.15.195:0/956231982 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd868011b10 con 0x7fd874150580 2022-06-27T11:55:04.701 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.687+0000 7fd85bfff700 1 -- 172.21.15.195:0/956231982 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd868000cc0 con 0x7fd874150580 2022-06-27T11:55:04.702 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.687+0000 7fd87a2ed700 1 -- 172.21.15.195:0/956231982 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd87416a490 con 0x7fd874150580 2022-06-27T11:55:04.702 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.687+0000 7fd85bfff700 1 -- 172.21.15.195:0/956231982 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd868025620 con 0x7fd874150580 2022-06-27T11:55:04.702 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.687+0000 7fd87a2ed700 1 -- 172.21.15.195:0/956231982 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd87416aa50 con 0x7fd874150580 2022-06-27T11:55:04.703 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.688+0000 7fd87a2ed700 1 -- 172.21.15.195:0/956231982 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd874166980 con 0x7fd874150580 2022-06-27T11:55:04.703 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.689+0000 7fd85bfff700 1 -- 172.21.15.195:0/956231982 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fd86800f050 con 0x7fd874150580 2022-06-27T11:55:04.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.689+0000 7fd85bfff700 1 --2- 172.21.15.195:0/956231982 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd854063d00 0x7fd8540661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:04.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.689+0000 7fd85bfff700 1 -- 172.21.15.195:0/956231982 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6402+0+0 (secure 0 0 0) 0x7fd868082160 con 0x7fd874150580 2022-06-27T11:55:04.705 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.689+0000 7fd873fff700 1 --2- 172.21.15.195:0/956231982 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd854063d00 0x7fd8540661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:04.705 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.690+0000 7fd873fff700 1 --2- 172.21.15.195:0/956231982 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd854063d00 0x7fd8540661b0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fd8640016f0 tx=0x7fd864009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:55:04.706 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.693+0000 7fd85bfff700 1 -- 172.21.15.195:0/956231982 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fd874166980 con 0x7fd874150580 2022-06-27T11:55:04.917 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.918+0000 7fd87a2ed700 1 -- 172.21.15.195:0/956231982 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd874166980 con 0x7fd874150580 2022-06-27T11:55:04.918 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.918+0000 7fd85bfff700 1 -- 172.21.15.195:0/956231982 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6128 (secure 0 0 0) 0x7fd874166980 con 0x7fd874150580 2022-06-27T11:55:04.918 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:55:04.919 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4856,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/2970948376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2970948376},{"type":"v1","addr":"172.21.15.202:6835","nonce":2970948376}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:52:55.042048+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4760":{"gid":4760,"name":"b","rank":0,"incarnation":20,"state":"up:active","state_seq":21,"addr":"172.21.15.202:6833/1026219246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1026219246},{"type":"v1","addr":"172.21.15.202:6833","nonce":1026219246}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:55:04.920 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.921+0000 7fd87a2ed700 1 -- 172.21.15.195:0/956231982 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd854063d00 msgr2=0x7fd8540661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:04.920 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.921+0000 7fd87a2ed700 1 --2- 172.21.15.195:0/956231982 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd854063d00 0x7fd8540661b0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fd8640016f0 tx=0x7fd864009040 comp rx=0 tx=0).stop 2022-06-27T11:55:04.921 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.921+0000 7fd87a2ed700 1 -- 172.21.15.195:0/956231982 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd874150580 msgr2=0x7fd8740bcaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:04.921 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.921+0000 7fd87a2ed700 1 --2- 172.21.15.195:0/956231982 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd874150580 0x7fd8740bcaf0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fd874065b00 tx=0x7fd868017000 comp rx=0 tx=0).stop 2022-06-27T11:55:04.921 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.921+0000 7fd87a2ed700 1 -- 172.21.15.195:0/956231982 shutdown_connections 2022-06-27T11:55:04.921 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.921+0000 7fd87a2ed700 1 --2- 172.21.15.195:0/956231982 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd854063d00 0x7fd8540661b0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:04.922 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.921+0000 7fd87a2ed700 1 --2- 172.21.15.195:0/956231982 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd874150950 0x7fd874166c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:04.922 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.921+0000 7fd87a2ed700 1 --2- 172.21.15.195:0/956231982 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd87406f620 0x7fd87414e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:04.922 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.921+0000 7fd87a2ed700 1 --2- 172.21.15.195:0/956231982 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd874150580 0x7fd8740bcaf0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:04.923 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.921+0000 7fd87a2ed700 1 -- 172.21.15.195:0/956231982 >> 172.21.15.195:0/956231982 conn(0x7fd874063af0 msgr2=0x7fd8740c5060 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:55:04.923 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.921+0000 7fd87a2ed700 1 -- 172.21.15.195:0/956231982 shutdown_connections 2022-06-27T11:55:04.924 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:04.921+0000 7fd87a2ed700 1 -- 172.21.15.195:0/956231982 wait complete. 2022-06-27T11:55:04.925 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 25 2022-06-27T11:55:04.933 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2022-06-27T11:55:06.938 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:55:07.124 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.074+0000 7efcb1506700 1 Processor -- start 2022-06-27T11:55:07.124 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.074+0000 7efcb1506700 1 -- start start 2022-06-27T11:55:07.125 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.075+0000 7efcb1506700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7efcac157010 0x7efcac067ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:07.125 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.075+0000 7efcb1506700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcac156b80 con 0x7efcac0c40d0 2022-06-27T11:55:07.126 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.075+0000 7efcb1506700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcac06a040 con 0x7efcac0c6190 2022-06-27T11:55:07.126 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.075+0000 7efcb1506700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcac06a300 con 0x7efcac157010 2022-06-27T11:55:07.127 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.075+0000 7efcaaffd700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7efcac0c40d0 0x7efcac0c44a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:50484/0 (socket says 172.21.15.195:50484) 2022-06-27T11:55:07.128 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.075+0000 7efcaaffd700 1 -- 172.21.15.195:0/1167669326 learned_addr learned my addr 172.21.15.195:0/1167669326 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:55:07.130 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.075+0000 7efcab7fe700 1 --2- 172.21.15.195:0/1167669326 >> v2:172.21.15.202:3300/0 conn(0x7efcac157010 0x7efcac067ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:07.131 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.075+0000 7efca9ffb700 1 -- 172.21.15.195:0/1167669326 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1035896635 0 0) 0x7efcac156b80 con 0x7efcac0c40d0 2022-06-27T11:55:07.131 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.075+0000 7efca9ffb700 1 -- 172.21.15.195:0/1167669326 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc8c003430 con 0x7efcac0c40d0 2022-06-27T11:55:07.131 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.075+0000 7efca9ffb700 1 -- 172.21.15.195:0/1167669326 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 737091116 0 0) 0x7efc8c003430 con 0x7efcac0c40d0 2022-06-27T11:55:07.132 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.075+0000 7efca9ffb700 1 -- 172.21.15.195:0/1167669326 >> v2:172.21.15.202:3300/0 conn(0x7efcac157010 msgr2=0x7efcac067ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:55:07.141 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.075+0000 7efca9ffb700 1 --2- 172.21.15.195:0/1167669326 >> v2:172.21.15.202:3300/0 conn(0x7efcac157010 0x7efcac067ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:07.142 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.075+0000 7efca9ffb700 1 -- 172.21.15.195:0/1167669326 >> v1:172.21.15.202:6789/0 conn(0x7efcac0c6190 legacy=0x7efcac14cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:07.142 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.075+0000 7efca9ffb700 1 -- 172.21.15.195:0/1167669326 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcac06a5e0 con 0x7efcac0c40d0 2022-06-27T11:55:07.143 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.075+0000 7efca9ffb700 1 -- 172.21.15.195:0/1167669326 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7efc9c0017f0 con 0x7efcac0c40d0 2022-06-27T11:55:07.143 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.076+0000 7efca9ffb700 1 -- 172.21.15.195:0/1167669326 >> v1:172.21.15.195:6789/0 conn(0x7efcac0c40d0 legacy=0x7efcac0c44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:07.144 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.076+0000 7efca9ffb700 1 --2- 172.21.15.195:0/1167669326 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7efc8c003920 0x7efc8c003d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:07.145 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.076+0000 7efca9ffb700 1 --2- 172.21.15.195:0/1167669326 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efc8c0046d0 0x7efc8c004b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:07.145 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.076+0000 7efca9ffb700 1 --2- 172.21.15.195:0/1167669326 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7efc8c005080 0x7efc8c00b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:07.146 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.076+0000 7efca9ffb700 1 -- 172.21.15.195:0/1167669326 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcac06a300 con 0x7efc8c0046d0 2022-06-27T11:55:07.146 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.076+0000 7efca9ffb700 1 -- 172.21.15.195:0/1167669326 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc8c00c020 con 0x7efc8c003920 2022-06-27T11:55:07.146 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.076+0000 7efca9ffb700 1 -- 172.21.15.195:0/1167669326 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc8c00c1e0 con 0x7efc8c005080 2022-06-27T11:55:07.147 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.076+0000 7efcab7fe700 1 --2- 172.21.15.195:0/1167669326 >> v2:172.21.15.202:3300/0 conn(0x7efcac157010 0x7efcac067ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:55:07.147 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.076+0000 7efcaa7fc700 1 --2- 172.21.15.195:0/1167669326 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efc8c0046d0 0x7efc8c004b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:07.148 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.076+0000 7efcaaffd700 1 --2- 172.21.15.195:0/1167669326 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7efc8c003920 0x7efc8c003d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:07.148 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.076+0000 7efcab7fe700 1 --2- 172.21.15.195:0/1167669326 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7efc8c005080 0x7efc8c00b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:07.149 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.076+0000 7efcaa7fc700 1 -- 172.21.15.195:0/1167669326 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7efc8c005080 msgr2=0x7efc8c00b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:55:07.150 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.076+0000 7efcaa7fc700 1 --2- 172.21.15.195:0/1167669326 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7efc8c005080 0x7efc8c00b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:07.151 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.076+0000 7efcaa7fc700 1 -- 172.21.15.195:0/1167669326 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7efc8c003920 msgr2=0x7efc8c003d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:55:07.151 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.076+0000 7efcaa7fc700 1 --2- 172.21.15.195:0/1167669326 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7efc8c003920 0x7efc8c003d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:07.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.076+0000 7efcaa7fc700 1 -- 172.21.15.195:0/1167669326 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc8c003430 con 0x7efc8c0046d0 2022-06-27T11:55:07.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.076+0000 7efcab7fe700 1 --2- 172.21.15.195:0/1167669326 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7efc8c005080 0x7efc8c00b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T11:55:07.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.076+0000 7efcaaffd700 1 --2- 172.21.15.195:0/1167669326 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7efc8c003920 0x7efc8c003d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:55:07.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.076+0000 7efcab7fe700 1 -- 172.21.15.195:0/1167669326 reap_dead start 2022-06-27T11:55:07.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.076+0000 7efcaa7fc700 1 --2- 172.21.15.195:0/1167669326 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efc8c0046d0 0x7efc8c004b40 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7efc94001760 tx=0x7efc94007bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce196d6696587706 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:55:07.155 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.076+0000 7efca9ffb700 1 -- 172.21.15.195:0/1167669326 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc940050d0 con 0x7efc8c0046d0 2022-06-27T11:55:07.155 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.076+0000 7efca9ffb700 1 -- 172.21.15.195:0/1167669326 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc94005230 con 0x7efc8c0046d0 2022-06-27T11:55:07.157 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.076+0000 7efca9ffb700 1 -- 172.21.15.195:0/1167669326 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc94003750 con 0x7efc8c0046d0 2022-06-27T11:55:07.160 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.077+0000 7efcb1506700 1 -- 172.21.15.195:0/1167669326 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efc8c0046d0 msgr2=0x7efc8c004b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:07.162 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.077+0000 7efcb1506700 1 --2- 172.21.15.195:0/1167669326 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efc8c0046d0 0x7efc8c004b40 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7efc94001760 tx=0x7efc94007bb0 comp rx=0 tx=0).stop 2022-06-27T11:55:07.162 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.077+0000 7efcb1506700 1 -- 172.21.15.195:0/1167669326 shutdown_connections 2022-06-27T11:55:07.162 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.077+0000 7efcb1506700 1 --2- 172.21.15.195:0/1167669326 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efc8c0046d0 0x7efc8c004b40 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:07.163 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.077+0000 7efcb1506700 1 -- 172.21.15.195:0/1167669326 >> 172.21.15.195:0/1167669326 conn(0x7efcac0ccb80 msgr2=0x7efcac0ccf80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:55:07.163 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.077+0000 7efcb1506700 1 -- 172.21.15.195:0/1167669326 shutdown_connections 2022-06-27T11:55:07.164 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.077+0000 7efcb1506700 1 -- 172.21.15.195:0/1167669326 wait complete. 2022-06-27T11:55:07.164 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.078+0000 7efcb1506700 1 Processor -- start 2022-06-27T11:55:07.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.078+0000 7efcb1506700 1 -- start start 2022-06-27T11:55:07.166 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.079+0000 7efcb1506700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7efc8c0046d0 0x7efcac16a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:07.166 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.079+0000 7efcb1506700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efcac1678e0 0x7efcac151d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:07.167 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.079+0000 7efcb1506700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7efcac15a0a0 0x7efcac1690f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:07.167 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.079+0000 7efcb1506700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcac152270 con 0x7efcac1678e0 2022-06-27T11:55:07.169 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.079+0000 7efcb1506700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcac14cf00 con 0x7efc8c0046d0 2022-06-27T11:55:07.169 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.079+0000 7efcb1506700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcac153870 con 0x7efcac15a0a0 2022-06-27T11:55:07.170 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.079+0000 7efcaa7fc700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efcac1678e0 0x7efcac151d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:07.171 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.079+0000 7efcaa7fc700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efcac1678e0 0x7efcac151d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:44310/0 (socket says 172.21.15.195:44310) 2022-06-27T11:55:07.171 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.079+0000 7efcaa7fc700 1 -- 172.21.15.195:0/1002054743 learned_addr learned my addr 172.21.15.195:0/1002054743 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:55:07.172 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.079+0000 7efcaaffd700 1 --2- 172.21.15.195:0/1002054743 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7efc8c0046d0 0x7efcac16a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:07.173 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.079+0000 7efcab7fe700 1 --2- 172.21.15.195:0/1002054743 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7efcac15a0a0 0x7efcac1690f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:07.173 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.079+0000 7efcaa7fc700 1 -- 172.21.15.195:0/1002054743 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7efcac15a0a0 msgr2=0x7efcac1690f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:07.174 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.079+0000 7efcaa7fc700 1 --2- 172.21.15.195:0/1002054743 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7efcac15a0a0 0x7efcac1690f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:07.174 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.079+0000 7efcaa7fc700 1 -- 172.21.15.195:0/1002054743 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7efc8c0046d0 msgr2=0x7efcac16a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:07.175 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.079+0000 7efcaa7fc700 1 --2- 172.21.15.195:0/1002054743 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7efc8c0046d0 0x7efcac16a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:07.175 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.079+0000 7efcaa7fc700 1 -- 172.21.15.195:0/1002054743 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcac1518a0 con 0x7efcac1678e0 2022-06-27T11:55:07.178 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.080+0000 7efcaa7fc700 1 --2- 172.21.15.195:0/1002054743 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efcac1678e0 0x7efcac151d30 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7efc94005150 tx=0x7efc94015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:55:07.179 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.080+0000 7efca2ffd700 1 -- 172.21.15.195:0/1002054743 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc94012420 con 0x7efcac1678e0 2022-06-27T11:55:07.179 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.080+0000 7efca2ffd700 1 -- 172.21.15.195:0/1002054743 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc94012580 con 0x7efcac1678e0 2022-06-27T11:55:07.180 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.080+0000 7efcb1506700 1 -- 172.21.15.195:0/1002054743 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcac0c8130 con 0x7efcac1678e0 2022-06-27T11:55:07.180 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.080+0000 7efca2ffd700 1 -- 172.21.15.195:0/1002054743 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc94012940 con 0x7efcac1678e0 2022-06-27T11:55:07.180 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.080+0000 7efcb1506700 1 -- 172.21.15.195:0/1002054743 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcac153da0 con 0x7efcac1678e0 2022-06-27T11:55:07.181 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.081+0000 7efcb1506700 1 -- 172.21.15.195:0/1002054743 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcac151a30 con 0x7efcac1678e0 2022-06-27T11:55:07.182 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.081+0000 7efca2ffd700 1 -- 172.21.15.195:0/1002054743 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7efc94027030 con 0x7efcac1678e0 2022-06-27T11:55:07.183 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.081+0000 7efca2ffd700 1 --2- 172.21.15.195:0/1002054743 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7efc90063d30 0x7efc900661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:07.184 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.082+0000 7efca2ffd700 1 -- 172.21.15.195:0/1002054743 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6402+0+0 (secure 0 0 0) 0x7efc940811b0 con 0x7efcac1678e0 2022-06-27T11:55:07.184 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.082+0000 7efcaaffd700 1 --2- 172.21.15.195:0/1002054743 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7efc90063d30 0x7efc900661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:07.185 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.082+0000 7efcaaffd700 1 --2- 172.21.15.195:0/1002054743 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7efc90063d30 0x7efc900661e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7efc9c001580 tx=0x7efcac150b60 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:55:07.185 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.086+0000 7efca2ffd700 1 -- 172.21.15.195:0/1002054743 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7efc9404fc40 con 0x7efcac1678e0 2022-06-27T11:55:07.316 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.314+0000 7efcb1506700 1 -- 172.21.15.195:0/1002054743 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efcac0722c0 con 0x7efcac1678e0 2022-06-27T11:55:07.317 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.315+0000 7efca2ffd700 1 -- 172.21.15.195:0/1002054743 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6128 (secure 0 0 0) 0x7efc94017090 con 0x7efcac1678e0 2022-06-27T11:55:07.319 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:55:07.320 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4856,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/2970948376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2970948376},{"type":"v1","addr":"172.21.15.202:6835","nonce":2970948376}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:52:55.042048+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4760":{"gid":4760,"name":"b","rank":0,"incarnation":20,"state":"up:active","state_seq":21,"addr":"172.21.15.202:6833/1026219246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1026219246},{"type":"v1","addr":"172.21.15.202:6833","nonce":1026219246}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:55:07.321 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.317+0000 7efcb1506700 1 -- 172.21.15.195:0/1002054743 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7efc90063d30 msgr2=0x7efc900661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:07.322 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.317+0000 7efcb1506700 1 --2- 172.21.15.195:0/1002054743 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7efc90063d30 0x7efc900661e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7efc9c001580 tx=0x7efcac150b60 comp rx=0 tx=0).stop 2022-06-27T11:55:07.323 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.317+0000 7efcb1506700 1 -- 172.21.15.195:0/1002054743 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efcac1678e0 msgr2=0x7efcac151d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:07.323 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.317+0000 7efcb1506700 1 --2- 172.21.15.195:0/1002054743 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efcac1678e0 0x7efcac151d30 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7efc94005150 tx=0x7efc94015000 comp rx=0 tx=0).stop 2022-06-27T11:55:07.323 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.317+0000 7efcb1506700 1 -- 172.21.15.195:0/1002054743 shutdown_connections 2022-06-27T11:55:07.324 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.317+0000 7efcb1506700 1 --2- 172.21.15.195:0/1002054743 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7efc90063d30 0x7efc900661e0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:07.324 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.317+0000 7efcb1506700 1 --2- 172.21.15.195:0/1002054743 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7efcac15a0a0 0x7efcac1690f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:07.325 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.317+0000 7efcb1506700 1 --2- 172.21.15.195:0/1002054743 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7efc8c0046d0 0x7efcac16a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:07.325 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.317+0000 7efcb1506700 1 --2- 172.21.15.195:0/1002054743 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efcac1678e0 0x7efcac151d30 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:07.326 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.317+0000 7efcb1506700 1 -- 172.21.15.195:0/1002054743 >> 172.21.15.195:0/1002054743 conn(0x7efcac0ccb80 msgr2=0x7efcac0c9600 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:55:07.326 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.317+0000 7efcb1506700 1 -- 172.21.15.195:0/1002054743 shutdown_connections 2022-06-27T11:55:07.326 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:07.317+0000 7efcb1506700 1 -- 172.21.15.195:0/1002054743 wait complete. 2022-06-27T11:55:07.330 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 25 2022-06-27T11:55:07.331 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2022-06-27T11:55:09.340 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:55:09.487 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.474+0000 7f264b44d700 1 Processor -- start 2022-06-27T11:55:09.487 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.474+0000 7f264b44d700 1 -- start start 2022-06-27T11:55:09.487 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.475+0000 7f264b44d700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f264406a440 0x7f26440688a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:09.488 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.475+0000 7f264b44d700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26440c4b10 con 0x7f2644156100 2022-06-27T11:55:09.488 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.475+0000 7f264b44d700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2644068f20 con 0x7f26441564d0 2022-06-27T11:55:09.488 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.475+0000 7f264b44d700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26440c3dc0 con 0x7f264406a440 2022-06-27T11:55:09.489 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.475+0000 7f26491e9700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f2644156100 0x7f26440c4fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:50504/0 (socket says 172.21.15.195:50504) 2022-06-27T11:55:09.490 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.475+0000 7f26491e9700 1 -- 172.21.15.195:0/3931206836 learned_addr learned my addr 172.21.15.195:0/3931206836 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:55:09.491 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.475+0000 7f26489e8700 1 --2- 172.21.15.195:0/3931206836 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f264406a440 0x7f26440688a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:09.491 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.475+0000 7f2633fff700 1 -- 172.21.15.195:0/3931206836 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3853268195 0 0) 0x7f26440c4b10 con 0x7f2644156100 2022-06-27T11:55:09.491 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.475+0000 7f2633fff700 1 -- 172.21.15.195:0/3931206836 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f262c003430 con 0x7f2644156100 2022-06-27T11:55:09.491 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.475+0000 7f2633fff700 1 -- 172.21.15.195:0/3931206836 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3764689338 0 0) 0x7f262c003430 con 0x7f2644156100 2022-06-27T11:55:09.492 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.475+0000 7f2633fff700 1 -- 172.21.15.195:0/3931206836 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f264406a440 msgr2=0x7f26440688a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:55:09.492 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.475+0000 7f2633fff700 1 --2- 172.21.15.195:0/3931206836 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f264406a440 0x7f26440688a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:09.492 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.475+0000 7f2633fff700 1 -- 172.21.15.195:0/3931206836 >> v1:172.21.15.202:6789/0 conn(0x7f26441564d0 legacy=0x7f26440c28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:09.492 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.475+0000 7f2633fff700 1 -- 172.21.15.195:0/3931206836 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26440c56b0 con 0x7f2644156100 2022-06-27T11:55:09.493 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.475+0000 7f2633fff700 1 -- 172.21.15.195:0/3931206836 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7f26340015f0 con 0x7f2644156100 2022-06-27T11:55:09.493 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.475+0000 7f2633fff700 1 -- 172.21.15.195:0/3931206836 >> v1:172.21.15.195:6789/0 conn(0x7f2644156100 legacy=0x7f26440c4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:09.493 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.475+0000 7f2633fff700 1 --2- 172.21.15.195:0/3931206836 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f264406a440 0x7f262c003950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:09.493 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.475+0000 7f2633fff700 1 --2- 172.21.15.195:0/3931206836 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f262c003ed0 0x7f262c0046d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:09.493 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.475+0000 7f2633fff700 1 --2- 172.21.15.195:0/3931206836 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f262c004c10 0x7f262c005080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:09.494 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.475+0000 7f2633fff700 1 -- 172.21.15.195:0/3931206836 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26440c3dc0 con 0x7f262c003ed0 2022-06-27T11:55:09.494 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.475+0000 7f2633fff700 1 -- 172.21.15.195:0/3931206836 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f262c00cbc0 con 0x7f262c004c10 2022-06-27T11:55:09.494 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.475+0000 7f2633fff700 1 -- 172.21.15.195:0/3931206836 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f262c00bb30 con 0x7f264406a440 2022-06-27T11:55:09.495 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.476+0000 7f26489e8700 1 --2- 172.21.15.195:0/3931206836 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f262c003ed0 0x7f262c0046d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:09.495 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.476+0000 7f26491e9700 1 --2- 172.21.15.195:0/3931206836 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f264406a440 0x7f262c003950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:09.495 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.476+0000 7f26499ea700 1 --2- 172.21.15.195:0/3931206836 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f262c004c10 0x7f262c005080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:09.496 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.476+0000 7f26489e8700 1 -- 172.21.15.195:0/3931206836 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f264406a440 msgr2=0x7f262c003950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:55:09.496 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.476+0000 7f26489e8700 1 --2- 172.21.15.195:0/3931206836 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f264406a440 0x7f262c003950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:09.496 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.476+0000 7f26489e8700 1 -- 172.21.15.195:0/3931206836 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f262c004c10 msgr2=0x7f262c005080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:55:09.496 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.476+0000 7f26489e8700 1 --2- 172.21.15.195:0/3931206836 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f262c004c10 0x7f262c005080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:09.497 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.476+0000 7f26489e8700 1 -- 172.21.15.195:0/3931206836 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f262c003430 con 0x7f262c003ed0 2022-06-27T11:55:09.497 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.476+0000 7f26499ea700 1 --2- 172.21.15.195:0/3931206836 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f262c004c10 0x7f262c005080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T11:55:09.497 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.476+0000 7f26489e8700 1 --2- 172.21.15.195:0/3931206836 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f262c003ed0 0x7f262c0046d0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f2640006dc0 tx=0x7f2640009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42eda27972d62d39 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:55:09.497 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.476+0000 7f2633fff700 1 -- 172.21.15.195:0/3931206836 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2640009470 con 0x7f262c003ed0 2022-06-27T11:55:09.498 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.476+0000 7f2633fff700 1 -- 172.21.15.195:0/3931206836 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f264000e640 con 0x7f262c003ed0 2022-06-27T11:55:09.498 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.476+0000 7f2633fff700 1 -- 172.21.15.195:0/3931206836 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f264000e9e0 con 0x7f262c003ed0 2022-06-27T11:55:09.498 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.477+0000 7f264b44d700 1 -- 172.21.15.195:0/3931206836 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f262c003ed0 msgr2=0x7f262c0046d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:09.498 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.477+0000 7f264b44d700 1 --2- 172.21.15.195:0/3931206836 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f262c003ed0 0x7f262c0046d0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f2640006dc0 tx=0x7f2640009060 comp rx=0 tx=0).stop 2022-06-27T11:55:09.499 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.477+0000 7f26499ea700 1 -- 172.21.15.195:0/3931206836 reap_dead start 2022-06-27T11:55:09.499 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.477+0000 7f264b44d700 1 -- 172.21.15.195:0/3931206836 shutdown_connections 2022-06-27T11:55:09.499 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.477+0000 7f264b44d700 1 -- 172.21.15.195:0/3931206836 >> 172.21.15.195:0/3931206836 conn(0x7f26440c9b00 msgr2=0x7f26440c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:55:09.500 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.477+0000 7f264b44d700 1 -- 172.21.15.195:0/3931206836 shutdown_connections 2022-06-27T11:55:09.500 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.477+0000 7f264b44d700 1 -- 172.21.15.195:0/3931206836 wait complete. 2022-06-27T11:55:09.500 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.478+0000 7f264b44d700 1 Processor -- start 2022-06-27T11:55:09.501 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.478+0000 7f264b44d700 1 -- start start 2022-06-27T11:55:09.502 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.479+0000 7f264b44d700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f262c003ed0 0x7f26441467c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:09.503 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.479+0000 7f264b44d700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f26440c74b0 0x7f2644163100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:09.504 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.479+0000 7f264b44d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f26440c80f0 0x7f264414d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:09.504 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.479+0000 7f264b44d700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2644147710 con 0x7f26440c80f0 2022-06-27T11:55:09.505 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.479+0000 7f264b44d700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26440c0fe0 con 0x7f262c003ed0 2022-06-27T11:55:09.505 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.479+0000 7f264b44d700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26441626f0 con 0x7f26440c74b0 2022-06-27T11:55:09.506 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.479+0000 7f26499ea700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f26440c80f0 0x7f264414d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:09.506 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.479+0000 7f26499ea700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f26440c80f0 0x7f264414d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:44332/0 (socket says 172.21.15.195:44332) 2022-06-27T11:55:09.511 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.479+0000 7f26491e9700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f262c003ed0 0x7f26441467c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:09.512 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.479+0000 7f26499ea700 1 -- 172.21.15.195:0/3574016632 learned_addr learned my addr 172.21.15.195:0/3574016632 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:55:09.512 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.479+0000 7f26489e8700 1 --2- 172.21.15.195:0/3574016632 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f26440c74b0 0x7f2644163100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:09.512 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.479+0000 7f26499ea700 1 -- 172.21.15.195:0/3574016632 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f26440c74b0 msgr2=0x7f2644163100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:09.512 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.479+0000 7f26499ea700 1 --2- 172.21.15.195:0/3574016632 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f26440c74b0 0x7f2644163100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:09.513 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.479+0000 7f26499ea700 1 -- 172.21.15.195:0/3574016632 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f262c003ed0 msgr2=0x7f26441467c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:09.513 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.479+0000 7f26499ea700 1 --2- 172.21.15.195:0/3574016632 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f262c003ed0 0x7f26441467c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:09.513 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.479+0000 7f26499ea700 1 -- 172.21.15.195:0/3574016632 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2644162c70 con 0x7f26440c80f0 2022-06-27T11:55:09.513 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.479+0000 7f26499ea700 1 --2- 172.21.15.195:0/3574016632 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f26440c80f0 0x7f264414d5b0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f26441568a0 tx=0x7f263c009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:55:09.514 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.479+0000 7f26317fa700 1 -- 172.21.15.195:0/3574016632 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f263c006c70 con 0x7f26440c80f0 2022-06-27T11:55:09.514 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.480+0000 7f26317fa700 1 -- 172.21.15.195:0/3574016632 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f263c006dd0 con 0x7f26440c80f0 2022-06-27T11:55:09.515 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.480+0000 7f264b44d700 1 -- 172.21.15.195:0/3574016632 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2644146e70 con 0x7f26440c80f0 2022-06-27T11:55:09.515 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.480+0000 7f26317fa700 1 -- 172.21.15.195:0/3574016632 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f263c003ad0 con 0x7f26440c80f0 2022-06-27T11:55:09.515 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.480+0000 7f264b44d700 1 -- 172.21.15.195:0/3574016632 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2644147430 con 0x7f26440c80f0 2022-06-27T11:55:09.516 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.480+0000 7f264b44d700 1 -- 172.21.15.195:0/3574016632 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2644162e00 con 0x7f26440c80f0 2022-06-27T11:55:09.516 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.481+0000 7f26317fa700 1 -- 172.21.15.195:0/3574016632 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f263c01b030 con 0x7f26440c80f0 2022-06-27T11:55:09.517 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.481+0000 7f26317fa700 1 --2- 172.21.15.195:0/3574016632 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2624063d30 0x7f26240661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:09.517 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.481+0000 7f26317fa700 1 -- 172.21.15.195:0/3574016632 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6402+0+0 (secure 0 0 0) 0x7f263c07fcb0 con 0x7f26440c80f0 2022-06-27T11:55:09.518 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.481+0000 7f26491e9700 1 --2- 172.21.15.195:0/3574016632 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2624063d30 0x7f26240661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:09.518 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.482+0000 7f26491e9700 1 --2- 172.21.15.195:0/3574016632 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2624063d30 0x7f26240661e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f26340011d0 tx=0x7f2634009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:55:09.519 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.485+0000 7f26317fa700 1 -- 172.21.15.195:0/3574016632 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f263c04e6e0 con 0x7f26440c80f0 2022-06-27T11:55:09.719 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.719+0000 7f264b44d700 1 -- 172.21.15.195:0/3574016632 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f26440722c0 con 0x7f26440c80f0 2022-06-27T11:55:09.720 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.719+0000 7f26317fa700 1 -- 172.21.15.195:0/3574016632 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6128 (secure 0 0 0) 0x7f263c019030 con 0x7f26440c80f0 2022-06-27T11:55:09.720 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:55:09.721 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4856,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/2970948376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2970948376},{"type":"v1","addr":"172.21.15.202:6835","nonce":2970948376}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:52:55.042048+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4760":{"gid":4760,"name":"b","rank":0,"incarnation":20,"state":"up:active","state_seq":21,"addr":"172.21.15.202:6833/1026219246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1026219246},{"type":"v1","addr":"172.21.15.202:6833","nonce":1026219246}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:55:09.722 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.721+0000 7f264b44d700 1 -- 172.21.15.195:0/3574016632 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2624063d30 msgr2=0x7f26240661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:09.722 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.721+0000 7f264b44d700 1 --2- 172.21.15.195:0/3574016632 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2624063d30 0x7f26240661e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f26340011d0 tx=0x7f2634009040 comp rx=0 tx=0).stop 2022-06-27T11:55:09.722 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.721+0000 7f264b44d700 1 -- 172.21.15.195:0/3574016632 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f26440c80f0 msgr2=0x7f264414d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:09.723 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.721+0000 7f264b44d700 1 --2- 172.21.15.195:0/3574016632 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f26440c80f0 0x7f264414d5b0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f26441568a0 tx=0x7f263c009040 comp rx=0 tx=0).stop 2022-06-27T11:55:09.723 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.721+0000 7f264b44d700 1 -- 172.21.15.195:0/3574016632 shutdown_connections 2022-06-27T11:55:09.723 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.721+0000 7f264b44d700 1 --2- 172.21.15.195:0/3574016632 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2624063d30 0x7f26240661e0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:09.723 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.721+0000 7f264b44d700 1 --2- 172.21.15.195:0/3574016632 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f26440c80f0 0x7f264414d5b0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:09.724 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.721+0000 7f264b44d700 1 --2- 172.21.15.195:0/3574016632 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f262c003ed0 0x7f26441467c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:09.724 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.721+0000 7f264b44d700 1 --2- 172.21.15.195:0/3574016632 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f26440c74b0 0x7f2644163100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:09.724 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.721+0000 7f264b44d700 1 -- 172.21.15.195:0/3574016632 >> 172.21.15.195:0/3574016632 conn(0x7f26440c9b00 msgr2=0x7f2644067ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:55:09.725 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.721+0000 7f264b44d700 1 -- 172.21.15.195:0/3574016632 shutdown_connections 2022-06-27T11:55:09.725 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:09.721+0000 7f264b44d700 1 -- 172.21.15.195:0/3574016632 wait complete. 2022-06-27T11:55:09.725 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 25 2022-06-27T11:55:09.746 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2022-06-27T11:55:11.749 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:55:11.886 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:11.886+0000 7f9bb98e7700 1 Processor -- start 2022-06-27T11:55:11.886 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:11.886+0000 7f9bb98e7700 1 -- start start 2022-06-27T11:55:11.889 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:11.887+0000 7f9bb98e7700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9bb406e850 0x7f9bb41570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:11.890 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:11.887+0000 7f9bb98e7700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f9bb406fef0 0x7f9bb4069de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:11.890 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:11.887+0000 7f9bb98e7700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f9bb40c34a0 0x7f9bb406a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:11.890 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:11.887+0000 7f9bb98e7700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bb41575e0 con 0x7f9bb406fef0 2022-06-27T11:55:11.891 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:11.887+0000 7f9bb98e7700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bb406fba0 con 0x7f9bb40c34a0 2022-06-27T11:55:11.891 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:11.887+0000 7f9bb98e7700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb406f5e0 con 0x7f9bb406e850 2022-06-27T11:55:11.891 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:11.887+0000 7f9bb27fc700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f9bb406fef0 0x7f9bb4069de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:11.891 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:11.887+0000 7f9bb27fc700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f9bb406fef0 0x7f9bb4069de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:44338/0 (socket says 172.21.15.195:44338) 2022-06-27T11:55:11.892 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:11.887+0000 7f9bb27fc700 1 -- 172.21.15.195:0/1245595944 learned_addr learned my addr 172.21.15.195:0/1245595944 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:55:11.892 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:11.888+0000 7f9bb37fe700 1 --2- 172.21.15.195:0/1245595944 >> v2:172.21.15.202:3300/0 conn(0x7f9bb40c34a0 0x7f9bb406a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:11.892 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:11.888+0000 7f9bb2ffd700 1 --2- 172.21.15.195:0/1245595944 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9bb406e850 0x7f9bb41570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:11.892 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:11.888+0000 7f9bb27fc700 1 -- 172.21.15.195:0/1245595944 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9bb406e850 msgr2=0x7f9bb41570a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:55:11.893 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:11.888+0000 7f9bb27fc700 1 --2- 172.21.15.195:0/1245595944 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9bb406e850 0x7f9bb41570a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:11.893 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:11.888+0000 7f9bb27fc700 1 -- 172.21.15.195:0/1245595944 >> v2:172.21.15.202:3300/0 conn(0x7f9bb40c34a0 msgr2=0x7f9bb406a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:55:11.893 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:11.888+0000 7f9bb27fc700 1 --2- 172.21.15.195:0/1245595944 >> v2:172.21.15.202:3300/0 conn(0x7f9bb40c34a0 0x7f9bb406a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:11.894 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:11.888+0000 7f9bb27fc700 1 -- 172.21.15.195:0/1245595944 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bb406f8c0 con 0x7f9bb406fef0 2022-06-27T11:55:11.894 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:11.888+0000 7f9bb37fe700 1 --2- 172.21.15.195:0/1245595944 >> v2:172.21.15.202:3300/0 conn(0x7f9bb40c34a0 0x7f9bb406a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T11:55:11.902 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:11.888+0000 7f9bb2ffd700 1 --2- 172.21.15.195:0/1245595944 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9bb406e850 0x7f9bb41570a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T11:55:11.902 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:11.888+0000 7f9bb27fc700 1 --2- 172.21.15.195:0/1245595944 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9bb406fef0 0x7f9bb4069de0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f9bb406e1e0 tx=0x7f9ba4009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3c334c09a6f2ad9 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:55:11.903 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:11.888+0000 7f9bb1ffb700 1 -- 172.21.15.195:0/1245595944 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ba400b6d0 con 0x7f9bb406fef0 2022-06-27T11:55:11.903 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:11.888+0000 7f9bb1ffb700 1 -- 172.21.15.195:0/1245595944 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ba400b830 con 0x7f9bb406fef0 2022-06-27T11:55:11.905 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:11.888+0000 7f9bb1ffb700 1 -- 172.21.15.195:0/1245595944 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ba400bbd0 con 0x7f9bb406fef0 2022-06-27T11:55:11.905 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:11.889+0000 7f9bb98e7700 1 -- 172.21.15.195:0/1245595944 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9bb406fef0 msgr2=0x7f9bb4069de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:11.905 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:11.889+0000 7f9bb98e7700 1 --2- 172.21.15.195:0/1245595944 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9bb406fef0 0x7f9bb4069de0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f9bb406e1e0 tx=0x7f9ba4009120 comp rx=0 tx=0).stop 2022-06-27T11:55:11.905 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:11.889+0000 7f9bb98e7700 1 -- 172.21.15.195:0/1245595944 shutdown_connections 2022-06-27T11:55:11.906 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:11.889+0000 7f9bb98e7700 1 --2- 172.21.15.195:0/1245595944 >> v2:172.21.15.202:3300/0 conn(0x7f9bb40c34a0 0x7f9bb406a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:11.906 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:11.889+0000 7f9bb98e7700 1 --2- 172.21.15.195:0/1245595944 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9bb406e850 0x7f9bb41570a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:11.906 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:11.889+0000 7f9bb98e7700 1 --2- 172.21.15.195:0/1245595944 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9bb406fef0 0x7f9bb4069de0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:11.906 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:11.889+0000 7f9bb98e7700 1 -- 172.21.15.195:0/1245595944 >> 172.21.15.195:0/1245595944 conn(0x7f9bb40c6350 msgr2=0x7f9bb40c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:55:11.907 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:11.889+0000 7f9bb98e7700 1 -- 172.21.15.195:0/1245595944 shutdown_connections 2022-06-27T11:55:11.907 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:11.889+0000 7f9bb98e7700 1 -- 172.21.15.195:0/1245595944 wait complete. 2022-06-27T11:55:11.907 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:11.890+0000 7f9bb98e7700 1 Processor -- start 2022-06-27T11:55:11.907 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:11.890+0000 7f9bb98e7700 1 -- start start 2022-06-27T11:55:11.910 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:11.891+0000 7f9bb98e7700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9bb406e850 0x7f9bb414c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:11.911 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:11.891+0000 7f9bb98e7700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9bb40c34a0 0x7f9bb4151dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:11.911 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:11.891+0000 7f9bb98e7700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9bb41567e0 0x7f9bb4152310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:11.911 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:11.891+0000 7f9bb98e7700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb4154320 con 0x7f9bb406e850 2022-06-27T11:55:11.912 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:11.891+0000 7f9bb98e7700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb4163dd0 con 0x7f9bb40c34a0 2022-06-27T11:55:11.912 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:11.891+0000 7f9bb98e7700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb41640b0 con 0x7f9bb41567e0 2022-06-27T11:55:11.912 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:11.891+0000 7f9bb2ffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9bb406e850 0x7f9bb414c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:11.912 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:11.891+0000 7f9bb2ffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9bb406e850 0x7f9bb414c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:44342/0 (socket says 172.21.15.195:44342) 2022-06-27T11:55:11.912 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:11.891+0000 7f9bb2ffd700 1 -- 172.21.15.195:0/2270565895 learned_addr learned my addr 172.21.15.195:0/2270565895 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:55:11.913 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:11.891+0000 7f9bb27fc700 1 --2- 172.21.15.195:0/2270565895 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9bb40c34a0 0x7f9bb4151dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:11.913 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:11.891+0000 7f9bb37fe700 1 --2- 172.21.15.195:0/2270565895 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9bb41567e0 0x7f9bb4152310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:11.913 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:11.891+0000 7f9bb2ffd700 1 -- 172.21.15.195:0/2270565895 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9bb41567e0 msgr2=0x7f9bb4152310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:11.913 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:11.891+0000 7f9bb2ffd700 1 --2- 172.21.15.195:0/2270565895 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9bb41567e0 0x7f9bb4152310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:11.923 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:11.891+0000 7f9bb2ffd700 1 -- 172.21.15.195:0/2270565895 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9bb40c34a0 msgr2=0x7f9bb4151dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:11.923 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:11.891+0000 7f9bb2ffd700 1 --2- 172.21.15.195:0/2270565895 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9bb40c34a0 0x7f9bb4151dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:11.924 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:11.891+0000 7f9bb2ffd700 1 -- 172.21.15.195:0/2270565895 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ba40093a0 con 0x7f9bb406e850 2022-06-27T11:55:11.924 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:11.891+0000 7f9bb2ffd700 1 --2- 172.21.15.195:0/2270565895 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9bb406e850 0x7f9bb414c510 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f9ba800c850 tx=0x7f9ba8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:55:11.925 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:11.892+0000 7f9b9affd700 1 -- 172.21.15.195:0/2270565895 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ba8011600 con 0x7f9bb406e850 2022-06-27T11:55:11.925 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:11.892+0000 7f9b9affd700 1 -- 172.21.15.195:0/2270565895 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ba8011760 con 0x7f9bb406e850 2022-06-27T11:55:11.925 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:11.892+0000 7f9bb98e7700 1 -- 172.21.15.195:0/2270565895 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bb4154040 con 0x7f9bb406e850 2022-06-27T11:55:11.925 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:11.892+0000 7f9b9affd700 1 -- 172.21.15.195:0/2270565895 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ba8011b20 con 0x7f9bb406e850 2022-06-27T11:55:11.926 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:11.892+0000 7f9bb98e7700 1 -- 172.21.15.195:0/2270565895 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bb40c87d0 con 0x7f9bb406e850 2022-06-27T11:55:11.926 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:11.893+0000 7f9bb98e7700 1 -- 172.21.15.195:0/2270565895 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bb40722c0 con 0x7f9bb406e850 2022-06-27T11:55:11.926 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:11.893+0000 7f9b9affd700 1 -- 172.21.15.195:0/2270565895 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f9ba8021070 con 0x7f9bb406e850 2022-06-27T11:55:11.926 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:11.893+0000 7f9b9affd700 1 --2- 172.21.15.195:0/2270565895 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9b94063d00 0x7f9b940661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:11.927 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:11.893+0000 7f9b9affd700 1 -- 172.21.15.195:0/2270565895 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f9ba8080700 con 0x7f9bb406e850 2022-06-27T11:55:11.927 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:11.893+0000 7f9bb27fc700 1 --2- 172.21.15.195:0/2270565895 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9b94063d00 0x7f9b940661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:11.927 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:11.894+0000 7f9bb27fc700 1 --2- 172.21.15.195:0/2270565895 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9b94063d00 0x7f9b940661b0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f9bb40c73e0 tx=0x7f9ba4017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:55:11.927 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:11.897+0000 7f9b9affd700 1 -- 172.21.15.195:0/2270565895 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f9ba804f160 con 0x7f9bb406e850 2022-06-27T11:55:12.186 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.175+0000 7f9bb98e7700 1 -- 172.21.15.195:0/2270565895 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9bb40bcf90 con 0x7f9bb406e850 2022-06-27T11:55:12.195 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.188+0000 7f9b9affd700 1 -- 172.21.15.195:0/2270565895 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5367 (secure 0 0 0) 0x7f9ba801f090 con 0x7f9bb406e850 2022-06-27T11:55:12.195 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:55:12.195 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:55:09.933730+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4856},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4856":{"gid":4856,"name":"d","rank":0,"incarnation":26,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6835/2970948376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2970948376},{"type":"v1","addr":"172.21.15.202:6835","nonce":2970948376}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:55:12.196 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.192+0000 7f9bb98e7700 1 -- 172.21.15.195:0/2270565895 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9b94063d00 msgr2=0x7f9b940661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:12.196 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.192+0000 7f9bb98e7700 1 --2- 172.21.15.195:0/2270565895 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9b94063d00 0x7f9b940661b0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f9bb40c73e0 tx=0x7f9ba4017000 comp rx=0 tx=0).stop 2022-06-27T11:55:12.196 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.192+0000 7f9bb98e7700 1 -- 172.21.15.195:0/2270565895 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9bb406e850 msgr2=0x7f9bb414c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:12.197 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.192+0000 7f9bb98e7700 1 --2- 172.21.15.195:0/2270565895 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9bb406e850 0x7f9bb414c510 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f9ba800c850 tx=0x7f9ba8008040 comp rx=0 tx=0).stop 2022-06-27T11:55:12.197 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.192+0000 7f9bb98e7700 1 -- 172.21.15.195:0/2270565895 shutdown_connections 2022-06-27T11:55:12.197 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.192+0000 7f9bb98e7700 1 --2- 172.21.15.195:0/2270565895 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9b94063d00 0x7f9b940661b0 secure :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f9bb40c73e0 tx=0x7f9ba4017000 comp rx=0 tx=0).stop 2022-06-27T11:55:12.197 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.192+0000 7f9bb98e7700 1 --2- 172.21.15.195:0/2270565895 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9bb406e850 0x7f9bb414c510 secure :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f9ba800c850 tx=0x7f9ba8008040 comp rx=0 tx=0).stop 2022-06-27T11:55:12.198 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.192+0000 7f9bb98e7700 1 --2- 172.21.15.195:0/2270565895 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9bb41567e0 0x7f9bb4152310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:12.198 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.192+0000 7f9bb98e7700 1 --2- 172.21.15.195:0/2270565895 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9bb40c34a0 0x7f9bb4151dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:12.198 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.192+0000 7f9bb98e7700 1 -- 172.21.15.195:0/2270565895 >> 172.21.15.195:0/2270565895 conn(0x7f9bb40c6350 msgr2=0x7f9bb40c6720 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:55:12.198 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.193+0000 7f9bb98e7700 1 -- 172.21.15.195:0/2270565895 shutdown_connections 2022-06-27T11:55:12.199 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.193+0000 7f9bb98e7700 1 -- 172.21.15.195:0/2270565895 wait complete. 2022-06-27T11:55:12.199 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 26 2022-06-27T11:55:12.214 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2022-06-27T11:55:12.214 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2022-06-27T11:55:12.214 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:55:12.336 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.334+0000 7fe53244e700 1 Processor -- start 2022-06-27T11:55:12.337 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.334+0000 7fe53244e700 1 -- start start 2022-06-27T11:55:12.343 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.335+0000 7fe53244e700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fe52c0c6190 0x7fe52c157010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:12.344 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.335+0000 7fe53244e700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe52c067ed0 0x7fe52c14cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:12.344 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.335+0000 7fe53244e700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe52c068330 con 0x7fe52c0c40d0 2022-06-27T11:55:12.344 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.335+0000 7fe53244e700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe52c156b80 con 0x7fe52c0c6190 2022-06-27T11:55:12.345 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.335+0000 7fe53244e700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe52c06a040 con 0x7fe52c067ed0 2022-06-27T11:55:12.346 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.335+0000 7fe52b7fe700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fe52c0c6190 0x7fe52c157010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:12.346 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.335+0000 7fe52b7fe700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fe52c0c6190 0x7fe52c157010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:44360/0 (socket says 172.21.15.195:44360) 2022-06-27T11:55:12.346 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.335+0000 7fe52b7fe700 1 -- 172.21.15.195:0/1268092250 learned_addr learned my addr 172.21.15.195:0/1268092250 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:55:12.346 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.335+0000 7fe52bfff700 1 --1- 172.21.15.195:0/1268092250 >> v1:172.21.15.195:6789/0 conn(0x7fe52c0c40d0 0x7fe52c0c44a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:50546/0 (socket says 172.21.15.195:50546) 2022-06-27T11:55:12.347 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.335+0000 7fe5309eb700 1 --2- 172.21.15.195:0/1268092250 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe52c067ed0 0x7fe52c14cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:12.347 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.335+0000 7fe52b7fe700 1 -- 172.21.15.195:0/1268092250 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe52c067ed0 msgr2=0x7fe52c14cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:55:12.347 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.335+0000 7fe52b7fe700 1 --2- 172.21.15.195:0/1268092250 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe52c067ed0 0x7fe52c14cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:12.347 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.335+0000 7fe52b7fe700 1 -- 172.21.15.195:0/1268092250 >> v1:172.21.15.195:6789/0 conn(0x7fe52c0c40d0 legacy=0x7fe52c0c44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:12.352 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.335+0000 7fe52b7fe700 1 -- 172.21.15.195:0/1268092250 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe52c06a300 con 0x7fe52c0c6190 2022-06-27T11:55:12.353 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.335+0000 7fe52affd700 1 -- 172.21.15.195:0/1268092250 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1051697536 0 0) 0x7fe52c068330 con 0x7fe52c0c40d0 2022-06-27T11:55:12.353 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.336+0000 7fe52b7fe700 1 --2- 172.21.15.195:0/1268092250 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe52c0c6190 0x7fe52c157010 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fe51c009860 tx=0x7fe51c009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=617e4ff7d173842 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:55:12.353 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.336+0000 7fe52affd700 1 -- 172.21.15.195:0/1268092250 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe51c011630 con 0x7fe52c0c6190 2022-06-27T11:55:12.354 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.336+0000 7fe52affd700 1 -- 172.21.15.195:0/1268092250 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe51c011790 con 0x7fe52c0c6190 2022-06-27T11:55:12.354 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.336+0000 7fe52affd700 1 -- 172.21.15.195:0/1268092250 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe51c011b30 con 0x7fe52c0c6190 2022-06-27T11:55:12.354 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.336+0000 7fe53244e700 1 -- 172.21.15.195:0/1268092250 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe52c0c6190 msgr2=0x7fe52c157010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:12.354 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.336+0000 7fe53244e700 1 --2- 172.21.15.195:0/1268092250 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe52c0c6190 0x7fe52c157010 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fe51c009860 tx=0x7fe51c009120 comp rx=0 tx=0).stop 2022-06-27T11:55:12.355 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.336+0000 7fe53244e700 1 -- 172.21.15.195:0/1268092250 shutdown_connections 2022-06-27T11:55:12.355 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.336+0000 7fe53244e700 1 --2- 172.21.15.195:0/1268092250 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe52c067ed0 0x7fe52c14cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:12.355 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.336+0000 7fe53244e700 1 --2- 172.21.15.195:0/1268092250 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe52c0c6190 0x7fe52c157010 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:12.356 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.336+0000 7fe53244e700 1 -- 172.21.15.195:0/1268092250 >> 172.21.15.195:0/1268092250 conn(0x7fe52c0ccb80 msgr2=0x7fe52c0ccf80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:55:12.356 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.336+0000 7fe53244e700 1 -- 172.21.15.195:0/1268092250 shutdown_connections 2022-06-27T11:55:12.356 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.336+0000 7fe53244e700 1 -- 172.21.15.195:0/1268092250 wait complete. 2022-06-27T11:55:12.356 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.338+0000 7fe53244e700 1 Processor -- start 2022-06-27T11:55:12.356 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.338+0000 7fe53244e700 1 -- start start 2022-06-27T11:55:12.357 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.339+0000 7fe53244e700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe52c067ed0 0x7fe52c1661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:12.357 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.339+0000 7fe53244e700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe52c0c40d0 0x7fe52c157690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:12.357 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.339+0000 7fe53244e700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe52c0c6190 0x7fe52c14f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:12.357 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.339+0000 7fe53244e700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe52c1593f0 con 0x7fe52c0c40d0 2022-06-27T11:55:12.358 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.339+0000 7fe53244e700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe52c157d20 con 0x7fe52c0c6190 2022-06-27T11:55:12.358 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.339+0000 7fe53244e700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe52c158000 con 0x7fe52c067ed0 2022-06-27T11:55:12.358 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.339+0000 7fe52b7fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe52c0c40d0 0x7fe52c157690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:12.358 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.339+0000 7fe52b7fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe52c0c40d0 0x7fe52c157690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:44366/0 (socket says 172.21.15.195:44366) 2022-06-27T11:55:12.359 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.339+0000 7fe52b7fe700 1 -- 172.21.15.195:0/3759701353 learned_addr learned my addr 172.21.15.195:0/3759701353 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:55:12.359 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.339+0000 7fe52bfff700 1 --2- 172.21.15.195:0/3759701353 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe52c067ed0 0x7fe52c1661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:12.359 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.339+0000 7fe5309eb700 1 --2- 172.21.15.195:0/3759701353 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe52c0c6190 0x7fe52c14f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:12.360 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.339+0000 7fe52b7fe700 1 -- 172.21.15.195:0/3759701353 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe52c067ed0 msgr2=0x7fe52c1661e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:12.360 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.339+0000 7fe52b7fe700 1 --2- 172.21.15.195:0/3759701353 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe52c067ed0 0x7fe52c1661e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:12.360 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.339+0000 7fe52b7fe700 1 -- 172.21.15.195:0/3759701353 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe52c0c6190 msgr2=0x7fe52c14f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:12.360 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.339+0000 7fe52b7fe700 1 --2- 172.21.15.195:0/3759701353 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe52c0c6190 0x7fe52c14f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:12.361 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.339+0000 7fe52b7fe700 1 -- 172.21.15.195:0/3759701353 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe51c0093a0 con 0x7fe52c0c40d0 2022-06-27T11:55:12.361 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.340+0000 7fe52b7fe700 1 --2- 172.21.15.195:0/3759701353 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe52c0c40d0 0x7fe52c157690 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fe52c0697c0 tx=0x7fe51c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:55:12.361 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.340+0000 7fe513fff700 1 -- 172.21.15.195:0/3759701353 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe51c003730 con 0x7fe52c0c40d0 2022-06-27T11:55:12.361 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.340+0000 7fe513fff700 1 -- 172.21.15.195:0/3759701353 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe51c0038b0 con 0x7fe52c0c40d0 2022-06-27T11:55:12.362 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.340+0000 7fe53244e700 1 -- 172.21.15.195:0/3759701353 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe52c1582e0 con 0x7fe52c0c40d0 2022-06-27T11:55:12.362 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.340+0000 7fe513fff700 1 -- 172.21.15.195:0/3759701353 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe51c003c70 con 0x7fe52c0c40d0 2022-06-27T11:55:12.363 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.340+0000 7fe53244e700 1 -- 172.21.15.195:0/3759701353 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe52c1588a0 con 0x7fe52c0c40d0 2022-06-27T11:55:12.363 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.341+0000 7fe53244e700 1 -- 172.21.15.195:0/3759701353 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe52c0722c0 con 0x7fe52c0c40d0 2022-06-27T11:55:12.363 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.341+0000 7fe513fff700 1 -- 172.21.15.195:0/3759701353 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fe51c02b030 con 0x7fe52c0c40d0 2022-06-27T11:55:12.363 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.342+0000 7fe513fff700 1 --2- 172.21.15.195:0/3759701353 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe50c063d00 0x7fe50c0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:12.364 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.342+0000 7fe52bfff700 1 --2- 172.21.15.195:0/3759701353 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe50c063d00 0x7fe50c0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:12.364 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.342+0000 7fe513fff700 1 -- 172.21.15.195:0/3759701353 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fe51c081e70 con 0x7fe52c0c40d0 2022-06-27T11:55:12.364 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.342+0000 7fe52bfff700 1 --2- 172.21.15.195:0/3759701353 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe50c063d00 0x7fe50c0661b0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe520004810 tx=0x7fe520007bf0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:55:12.364 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.346+0000 7fe513fff700 1 -- 172.21.15.195:0/3759701353 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fe51c0508d0 con 0x7fe52c0c40d0 2022-06-27T11:55:12.573 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.573+0000 7fe53244e700 1 -- 172.21.15.195:0/3759701353 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe52c158e80 con 0x7fe52c0c40d0 2022-06-27T11:55:12.573 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.573+0000 7fe513fff700 1 -- 172.21.15.195:0/3759701353 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5367 (secure 0 0 0) 0x7fe51c029020 con 0x7fe52c0c40d0 2022-06-27T11:55:12.573 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:55:12.574 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:55:09.933730+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4856},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4856":{"gid":4856,"name":"d","rank":0,"incarnation":26,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6835/2970948376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2970948376},{"type":"v1","addr":"172.21.15.202:6835","nonce":2970948376}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:55:12.575 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.575+0000 7fe53244e700 1 -- 172.21.15.195:0/3759701353 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe50c063d00 msgr2=0x7fe50c0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:12.575 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.575+0000 7fe53244e700 1 --2- 172.21.15.195:0/3759701353 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe50c063d00 0x7fe50c0661b0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe520004810 tx=0x7fe520007bf0 comp rx=0 tx=0).stop 2022-06-27T11:55:12.576 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.575+0000 7fe53244e700 1 -- 172.21.15.195:0/3759701353 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe52c0c40d0 msgr2=0x7fe52c157690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:12.577 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.575+0000 7fe53244e700 1 --2- 172.21.15.195:0/3759701353 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe52c0c40d0 0x7fe52c157690 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fe52c0697c0 tx=0x7fe51c017000 comp rx=0 tx=0).stop 2022-06-27T11:55:12.578 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.575+0000 7fe53244e700 1 -- 172.21.15.195:0/3759701353 shutdown_connections 2022-06-27T11:55:12.578 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.575+0000 7fe53244e700 1 --2- 172.21.15.195:0/3759701353 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe50c063d00 0x7fe50c0661b0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:12.579 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.575+0000 7fe53244e700 1 --2- 172.21.15.195:0/3759701353 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe52c0c6190 0x7fe52c14f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:12.579 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.575+0000 7fe53244e700 1 --2- 172.21.15.195:0/3759701353 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe52c067ed0 0x7fe52c1661e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:12.579 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.575+0000 7fe53244e700 1 --2- 172.21.15.195:0/3759701353 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe52c0c40d0 0x7fe52c157690 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:12.580 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.575+0000 7fe53244e700 1 -- 172.21.15.195:0/3759701353 >> 172.21.15.195:0/3759701353 conn(0x7fe52c0ccb80 msgr2=0x7fe52c14fff0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:55:12.581 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.575+0000 7fe53244e700 1 -- 172.21.15.195:0/3759701353 shutdown_connections 2022-06-27T11:55:12.582 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:12.575+0000 7fe53244e700 1 -- 172.21.15.195:0/3759701353 wait complete. 2022-06-27T11:55:12.582 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 26 2022-06-27T11:55:12.596 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 26, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-06-27T11:47:52.358095+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 26, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4856": { "addr": "172.21.15.202:6835/2970948376", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6834", "nonce": 2970948376, "type": "v2" }, { "addr": "172.21.15.202:6835", "nonce": 2970948376, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4856, "incarnation": 26, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 51, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-06-27T11:55:09.933730+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4856 } } } ], "standbys": [ { "addr": "172.21.15.202:6837/3305006841", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6836", "nonce": 3305006841, "type": "v2" }, { "addr": "172.21.15.202:6837", "nonce": 3305006841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6835/2060709822", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6834", "nonce": 2060709822, "type": "v2" }, { "addr": "172.21.15.195:6835", "nonce": 2060709822, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6838/1598751652", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6836", "nonce": 1598751652, "type": "v2" }, { "addr": "172.21.15.195:6838", "nonce": 1598751652, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4275, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6839/1313800661", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6837", "nonce": 1313800661, "type": "v2" }, { "addr": "172.21.15.195:6839", "nonce": 1313800661, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4278, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-06-27T11:55:14.598 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:55:14.743 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.732+0000 7f5bad3f6700 1 Processor -- start 2022-06-27T11:55:14.743 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.732+0000 7f5bad3f6700 1 -- start start 2022-06-27T11:55:14.743 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.733+0000 7f5bad3f6700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f5ba8068f80 0x7f5ba80c4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:14.744 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.733+0000 7f5bad3f6700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f5ba80c7480 0x7f5ba80683c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:14.744 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.733+0000 7f5bad3f6700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5ba8069350 0x7f5ba8156460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:14.744 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.733+0000 7f5bad3f6700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ba81573b0 con 0x7f5ba80c7480 2022-06-27T11:55:14.744 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.733+0000 7f5bad3f6700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ba80c46c0 con 0x7f5ba8068f80 2022-06-27T11:55:14.745 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.733+0000 7f5bad3f6700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba80697d0 con 0x7f5ba8069350 2022-06-27T11:55:14.745 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.733+0000 7f5ba67fc700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f5ba80c7480 0x7f5ba80683c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:14.745 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.733+0000 7f5ba67fc700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f5ba80c7480 0x7f5ba80683c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:44378/0 (socket says 172.21.15.195:44378) 2022-06-27T11:55:14.745 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.733+0000 7f5ba67fc700 1 -- 172.21.15.195:0/2387523162 learned_addr learned my addr 172.21.15.195:0/2387523162 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:55:14.745 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.733+0000 7f5ba77fe700 1 --2- 172.21.15.195:0/2387523162 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5ba8069350 0x7f5ba8156460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:14.746 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.733+0000 7f5ba6ffd700 1 --2- 172.21.15.195:0/2387523162 >> v2:172.21.15.202:3300/0 conn(0x7f5ba8068f80 0x7f5ba80c4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:14.746 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.733+0000 7f5ba67fc700 1 -- 172.21.15.195:0/2387523162 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5ba8069350 msgr2=0x7f5ba8156460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:55:14.746 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.733+0000 7f5ba67fc700 1 --2- 172.21.15.195:0/2387523162 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5ba8069350 0x7f5ba8156460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:14.746 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.733+0000 7f5ba67fc700 1 -- 172.21.15.195:0/2387523162 >> v2:172.21.15.202:3300/0 conn(0x7f5ba8068f80 msgr2=0x7f5ba80c4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:55:14.747 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.733+0000 7f5ba67fc700 1 --2- 172.21.15.195:0/2387523162 >> v2:172.21.15.202:3300/0 conn(0x7f5ba8068f80 0x7f5ba80c4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:14.747 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.733+0000 7f5ba67fc700 1 -- 172.21.15.195:0/2387523162 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ba806a920 con 0x7f5ba80c7480 2022-06-27T11:55:14.747 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.733+0000 7f5ba6ffd700 1 --2- 172.21.15.195:0/2387523162 >> v2:172.21.15.202:3300/0 conn(0x7f5ba8068f80 0x7f5ba80c4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T11:55:14.747 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.734+0000 7f5ba67fc700 1 --2- 172.21.15.195:0/2387523162 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5ba80c7480 0x7f5ba80683c0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f5ba8067b00 tx=0x7f5b98009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28c2cd5bf7661db4 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:55:14.748 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.734+0000 7f5ba5ffb700 1 -- 172.21.15.195:0/2387523162 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b9800b640 con 0x7f5ba80c7480 2022-06-27T11:55:14.749 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.734+0000 7f5ba5ffb700 1 -- 172.21.15.195:0/2387523162 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b9800b7a0 con 0x7f5ba80c7480 2022-06-27T11:55:14.749 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.734+0000 7f5ba5ffb700 1 -- 172.21.15.195:0/2387523162 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b9800bb40 con 0x7f5ba80c7480 2022-06-27T11:55:14.750 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.734+0000 7f5bad3f6700 1 -- 172.21.15.195:0/2387523162 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5ba80c7480 msgr2=0x7f5ba80683c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:14.753 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.734+0000 7f5bad3f6700 1 --2- 172.21.15.195:0/2387523162 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5ba80c7480 0x7f5ba80683c0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f5ba8067b00 tx=0x7f5b98009120 comp rx=0 tx=0).stop 2022-06-27T11:55:14.762 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.734+0000 7f5bad3f6700 1 -- 172.21.15.195:0/2387523162 shutdown_connections 2022-06-27T11:55:14.764 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.734+0000 7f5bad3f6700 1 --2- 172.21.15.195:0/2387523162 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5ba8069350 0x7f5ba8156460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:14.764 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.734+0000 7f5bad3f6700 1 --2- 172.21.15.195:0/2387523162 >> v2:172.21.15.202:3300/0 conn(0x7f5ba8068f80 0x7f5ba80c4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:14.765 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.734+0000 7f5bad3f6700 1 --2- 172.21.15.195:0/2387523162 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5ba80c7480 0x7f5ba80683c0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:14.765 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.734+0000 7f5bad3f6700 1 -- 172.21.15.195:0/2387523162 >> 172.21.15.195:0/2387523162 conn(0x7f5ba80ccbd0 msgr2=0x7f5ba80ccfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:55:14.766 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.734+0000 7f5bad3f6700 1 -- 172.21.15.195:0/2387523162 shutdown_connections 2022-06-27T11:55:14.766 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.734+0000 7f5bad3f6700 1 -- 172.21.15.195:0/2387523162 wait complete. 2022-06-27T11:55:14.767 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.735+0000 7f5bad3f6700 1 Processor -- start 2022-06-27T11:55:14.767 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.736+0000 7f5bad3f6700 1 -- start start 2022-06-27T11:55:14.768 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.736+0000 7f5bad3f6700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5ba8068f80 0x7f5ba8155710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:14.768 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.736+0000 7f5bad3f6700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5ba8069350 0x7f5ba814edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:14.769 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.736+0000 7f5bad3f6700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5ba80c7480 0x7f5ba814f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:14.769 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.736+0000 7f5bad3f6700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba80a9370 con 0x7f5ba8068f80 2022-06-27T11:55:14.770 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.736+0000 7f5bad3f6700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba816ab50 con 0x7f5ba80c7480 2022-06-27T11:55:14.770 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.736+0000 7f5bad3f6700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba8169ed0 con 0x7f5ba8069350 2022-06-27T11:55:14.771 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.736+0000 7f5ba6ffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5ba8068f80 0x7f5ba8155710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:14.771 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.736+0000 7f5ba6ffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5ba8068f80 0x7f5ba8155710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:44382/0 (socket says 172.21.15.195:44382) 2022-06-27T11:55:14.776 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.736+0000 7f5ba6ffd700 1 -- 172.21.15.195:0/3730253570 learned_addr learned my addr 172.21.15.195:0/3730253570 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:55:14.777 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.737+0000 7f5ba77fe700 1 --2- 172.21.15.195:0/3730253570 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5ba80c7480 0x7f5ba814f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:14.777 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.737+0000 7f5ba67fc700 1 --2- 172.21.15.195:0/3730253570 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5ba8069350 0x7f5ba814edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:14.777 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.737+0000 7f5ba6ffd700 1 -- 172.21.15.195:0/3730253570 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5ba8069350 msgr2=0x7f5ba814edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:14.777 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.737+0000 7f5ba6ffd700 1 --2- 172.21.15.195:0/3730253570 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5ba8069350 0x7f5ba814edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:14.779 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.737+0000 7f5ba6ffd700 1 -- 172.21.15.195:0/3730253570 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5ba80c7480 msgr2=0x7f5ba814f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:14.780 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.737+0000 7f5ba6ffd700 1 --2- 172.21.15.195:0/3730253570 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5ba80c7480 0x7f5ba814f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:14.790 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.737+0000 7f5ba6ffd700 1 -- 172.21.15.195:0/3730253570 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b980093a0 con 0x7f5ba8068f80 2022-06-27T11:55:14.791 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.737+0000 7f5ba77fe700 1 --2- 172.21.15.195:0/3730253570 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5ba80c7480 0x7f5ba814f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T11:55:14.791 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.737+0000 7f5ba6ffd700 1 --2- 172.21.15.195:0/3730253570 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5ba8068f80 0x7f5ba8155710 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c00c8b0 tx=0x7f5b9c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:55:14.791 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.737+0000 7f5b8effd700 1 -- 172.21.15.195:0/3730253570 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b9c011600 con 0x7f5ba8068f80 2022-06-27T11:55:14.791 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.737+0000 7f5b8effd700 1 -- 172.21.15.195:0/3730253570 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b9c011760 con 0x7f5ba8068f80 2022-06-27T11:55:14.792 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.737+0000 7f5b8effd700 1 -- 172.21.15.195:0/3730253570 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b9c011b00 con 0x7f5ba8068f80 2022-06-27T11:55:14.792 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.737+0000 7f5bad3f6700 1 -- 172.21.15.195:0/3730253570 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ba816a1b0 con 0x7f5ba8068f80 2022-06-27T11:55:14.792 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.737+0000 7f5bad3f6700 1 -- 172.21.15.195:0/3730253570 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ba816a770 con 0x7f5ba8068f80 2022-06-27T11:55:14.792 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.738+0000 7f5bad3f6700 1 -- 172.21.15.195:0/3730253570 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ba80722c0 con 0x7f5ba8068f80 2022-06-27T11:55:14.793 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.738+0000 7f5b8effd700 1 -- 172.21.15.195:0/3730253570 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f5b9c021070 con 0x7f5ba8068f80 2022-06-27T11:55:14.793 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.739+0000 7f5b8effd700 1 --2- 172.21.15.195:0/3730253570 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5b88063d00 0x7f5b880661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:14.793 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.739+0000 7f5b8effd700 1 -- 172.21.15.195:0/3730253570 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f5b9c080820 con 0x7f5ba8068f80 2022-06-27T11:55:14.793 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.739+0000 7f5ba67fc700 1 --2- 172.21.15.195:0/3730253570 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5b88063d00 0x7f5b880661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:14.794 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.739+0000 7f5ba67fc700 1 --2- 172.21.15.195:0/3730253570 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5b88063d00 0x7f5b880661b0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f5b98009160 tx=0x7f5b98017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:55:14.794 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.743+0000 7f5b8effd700 1 -- 172.21.15.195:0/3730253570 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f5b9c04f280 con 0x7f5ba8068f80 2022-06-27T11:55:14.980 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.979+0000 7f5bad3f6700 1 -- 172.21.15.195:0/3730253570 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ba816a900 con 0x7f5ba8068f80 2022-06-27T11:55:14.980 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.979+0000 7f5b8effd700 1 -- 172.21.15.195:0/3730253570 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5367 (secure 0 0 0) 0x7f5b9c01f090 con 0x7f5ba8068f80 2022-06-27T11:55:14.980 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:55:14.981 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:55:09.933730+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4856},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4856":{"gid":4856,"name":"d","rank":0,"incarnation":26,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6835/2970948376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2970948376},{"type":"v1","addr":"172.21.15.202:6835","nonce":2970948376}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:55:14.982 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.981+0000 7f5bad3f6700 1 -- 172.21.15.195:0/3730253570 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5b88063d00 msgr2=0x7f5b880661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:14.982 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.981+0000 7f5bad3f6700 1 --2- 172.21.15.195:0/3730253570 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5b88063d00 0x7f5b880661b0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f5b98009160 tx=0x7f5b98017000 comp rx=0 tx=0).stop 2022-06-27T11:55:14.982 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.981+0000 7f5bad3f6700 1 -- 172.21.15.195:0/3730253570 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5ba8068f80 msgr2=0x7f5ba8155710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:14.983 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.981+0000 7f5bad3f6700 1 --2- 172.21.15.195:0/3730253570 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5ba8068f80 0x7f5ba8155710 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c00c8b0 tx=0x7f5b9c008040 comp rx=0 tx=0).stop 2022-06-27T11:55:14.983 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.981+0000 7f5bad3f6700 1 -- 172.21.15.195:0/3730253570 shutdown_connections 2022-06-27T11:55:14.984 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.981+0000 7f5bad3f6700 1 --2- 172.21.15.195:0/3730253570 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5b88063d00 0x7f5b880661b0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:14.986 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.981+0000 7f5bad3f6700 1 --2- 172.21.15.195:0/3730253570 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5ba8068f80 0x7f5ba8155710 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:14.986 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.981+0000 7f5bad3f6700 1 --2- 172.21.15.195:0/3730253570 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5ba80c7480 0x7f5ba814f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:14.986 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.981+0000 7f5bad3f6700 1 --2- 172.21.15.195:0/3730253570 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5ba8069350 0x7f5ba814edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:14.993 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.981+0000 7f5bad3f6700 1 -- 172.21.15.195:0/3730253570 >> 172.21.15.195:0/3730253570 conn(0x7f5ba80ccbd0 msgr2=0x7f5ba8150240 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:55:14.993 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.981+0000 7f5bad3f6700 1 -- 172.21.15.195:0/3730253570 shutdown_connections 2022-06-27T11:55:14.994 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:14.981+0000 7f5bad3f6700 1 -- 172.21.15.195:0/3730253570 wait complete. 2022-06-27T11:55:14.994 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 26 2022-06-27T11:55:14.999 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T11:55:17.001 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:55:17.148 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.134+0000 7eff6e4e5700 1 Processor -- start 2022-06-27T11:55:17.148 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.134+0000 7eff6e4e5700 1 -- start start 2022-06-27T11:55:17.149 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.135+0000 7eff6e4e5700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7eff6806e850 0x7eff681570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:17.149 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.135+0000 7eff6e4e5700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7eff680c34a0 0x7eff680c2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:17.151 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.135+0000 7eff6e4e5700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff6806fba0 con 0x7eff6806fef0 2022-06-27T11:55:17.151 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.135+0000 7eff6e4e5700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff681575e0 con 0x7eff6806e850 2022-06-27T11:55:17.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.135+0000 7eff6e4e5700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff6806f5e0 con 0x7eff680c34a0 2022-06-27T11:55:17.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.135+0000 7eff67fff700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7eff6806e850 0x7eff681570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:17.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.135+0000 7eff6ca82700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7eff680c34a0 0x7eff680c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:17.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.135+0000 7eff677fe700 1 --1- >> v1:172.21.15.202:6789/0 conn(0x7eff6806fef0 0x7eff68069de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.202:6789/0 says I am v1:172.21.15.195:52442/0 (socket says 172.21.15.195:52442) 2022-06-27T11:55:17.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.135+0000 7eff677fe700 1 -- 172.21.15.195:0/737839406 learned_addr learned my addr 172.21.15.195:0/737839406 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:55:17.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.135+0000 7eff67fff700 1 -- 172.21.15.195:0/737839406 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7eff680c34a0 msgr2=0x7eff680c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:55:17.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.135+0000 7eff67fff700 1 --2- 172.21.15.195:0/737839406 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7eff680c34a0 0x7eff680c2920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:17.155 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.135+0000 7eff67fff700 1 -- 172.21.15.195:0/737839406 >> v1:172.21.15.202:6789/0 conn(0x7eff6806fef0 legacy=0x7eff68069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:17.156 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.135+0000 7eff67fff700 1 -- 172.21.15.195:0/737839406 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff6806f8c0 con 0x7eff6806e850 2022-06-27T11:55:17.156 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.135+0000 7eff66ffd700 1 -- 172.21.15.195:0/737839406 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2124119450 0 0) 0x7eff6806fba0 con 0x7eff6806fef0 2022-06-27T11:55:17.157 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.135+0000 7eff6ca82700 1 --2- 172.21.15.195:0/737839406 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7eff680c34a0 0x7eff680c2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:55:17.157 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.136+0000 7eff67fff700 1 --2- 172.21.15.195:0/737839406 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7eff6806e850 0x7eff681570a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7eff5c0097a0 tx=0x7eff5c009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=94cf3b518f443e3e server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:55:17.158 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.136+0000 7eff66ffd700 1 -- 172.21.15.195:0/737839406 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff5c011630 con 0x7eff6806e850 2022-06-27T11:55:17.158 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.136+0000 7eff66ffd700 1 -- 172.21.15.195:0/737839406 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff5c011790 con 0x7eff6806e850 2022-06-27T11:55:17.159 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.136+0000 7eff66ffd700 1 -- 172.21.15.195:0/737839406 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff5c011b30 con 0x7eff6806e850 2022-06-27T11:55:17.159 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.136+0000 7eff6e4e5700 1 -- 172.21.15.195:0/737839406 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7eff6806e850 msgr2=0x7eff681570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:17.159 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.136+0000 7eff6e4e5700 1 --2- 172.21.15.195:0/737839406 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7eff6806e850 0x7eff681570a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7eff5c0097a0 tx=0x7eff5c009060 comp rx=0 tx=0).stop 2022-06-27T11:55:17.160 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.136+0000 7eff6e4e5700 1 -- 172.21.15.195:0/737839406 shutdown_connections 2022-06-27T11:55:17.160 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.136+0000 7eff6e4e5700 1 --2- 172.21.15.195:0/737839406 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7eff680c34a0 0x7eff680c2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:17.160 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.136+0000 7eff6e4e5700 1 --2- 172.21.15.195:0/737839406 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7eff6806e850 0x7eff681570a0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:17.161 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.136+0000 7eff6e4e5700 1 -- 172.21.15.195:0/737839406 >> 172.21.15.195:0/737839406 conn(0x7eff680c6350 msgr2=0x7eff680c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:55:17.161 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.136+0000 7eff6e4e5700 1 -- 172.21.15.195:0/737839406 shutdown_connections 2022-06-27T11:55:17.162 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.136+0000 7eff6e4e5700 1 -- 172.21.15.195:0/737839406 wait complete. 2022-06-27T11:55:17.162 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.138+0000 7eff6e4e5700 1 Processor -- start 2022-06-27T11:55:17.163 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.138+0000 7eff6e4e5700 1 -- start start 2022-06-27T11:55:17.163 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.138+0000 7eff6e4e5700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7eff6806e850 0x7eff68154770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:17.163 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.138+0000 7eff6e4e5700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7eff680c34a0 0x7eff68154cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:17.164 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.138+0000 7eff6e4e5700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7eff680c05c0 0x7eff68155a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:17.164 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.138+0000 7eff6e4e5700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff68156a40 con 0x7eff680c05c0 2022-06-27T11:55:17.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.138+0000 7eff6e4e5700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff681542e0 con 0x7eff6806e850 2022-06-27T11:55:17.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.138+0000 7eff6e4e5700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff68156c70 con 0x7eff680c34a0 2022-06-27T11:55:17.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.139+0000 7eff67fff700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7eff6806e850 0x7eff68154770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:17.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.139+0000 7eff6ca82700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7eff680c05c0 0x7eff68155a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:17.166 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.139+0000 7eff67fff700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7eff6806e850 0x7eff68154770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3300/0 says I am v2:172.21.15.195:42526/0 (socket says 172.21.15.195:42526) 2022-06-27T11:55:17.166 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.139+0000 7eff67fff700 1 -- 172.21.15.195:0/2884535720 learned_addr learned my addr 172.21.15.195:0/2884535720 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:55:17.167 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.139+0000 7eff677fe700 1 --2- 172.21.15.195:0/2884535720 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7eff680c34a0 0x7eff68154cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:17.167 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.139+0000 7eff67fff700 1 -- 172.21.15.195:0/2884535720 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7eff680c34a0 msgr2=0x7eff68154cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:17.168 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.139+0000 7eff67fff700 1 --2- 172.21.15.195:0/2884535720 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7eff680c34a0 0x7eff68154cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:17.168 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.139+0000 7eff67fff700 1 -- 172.21.15.195:0/2884535720 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7eff680c05c0 msgr2=0x7eff68155a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:17.168 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.139+0000 7eff67fff700 1 --2- 172.21.15.195:0/2884535720 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7eff680c05c0 0x7eff68155a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:17.169 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.139+0000 7eff67fff700 1 -- 172.21.15.195:0/2884535720 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff60009a20 con 0x7eff6806e850 2022-06-27T11:55:17.169 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.139+0000 7eff67fff700 1 --2- 172.21.15.195:0/2884535720 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7eff6806e850 0x7eff68154770 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7eff5c009e40 tx=0x7eff5c017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:55:17.170 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.139+0000 7eff4ffff700 1 -- 172.21.15.195:0/2884535720 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff5c000c90 con 0x7eff6806e850 2022-06-27T11:55:17.170 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.139+0000 7eff4ffff700 1 -- 172.21.15.195:0/2884535720 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff5c011900 con 0x7eff6806e850 2022-06-27T11:55:17.171 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.139+0000 7eff6e4e5700 1 -- 172.21.15.195:0/2884535720 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff5c0092e0 con 0x7eff6806e850 2022-06-27T11:55:17.171 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.139+0000 7eff4ffff700 1 -- 172.21.15.195:0/2884535720 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff5c025420 con 0x7eff6806e850 2022-06-27T11:55:17.172 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.139+0000 7eff6e4e5700 1 -- 172.21.15.195:0/2884535720 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff6814b250 con 0x7eff6806e850 2022-06-27T11:55:17.172 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.140+0000 7eff6e4e5700 1 -- 172.21.15.195:0/2884535720 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff680722c0 con 0x7eff6806e850 2022-06-27T11:55:17.173 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.140+0000 7eff4ffff700 1 -- 172.21.15.195:0/2884535720 <== mon.1 v2:172.21.15.202:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7eff5c02c030 con 0x7eff6806e850 2022-06-27T11:55:17.173 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.141+0000 7eff4ffff700 1 --2- 172.21.15.195:0/2884535720 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7eff48063d30 0x7eff480661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:17.173 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.141+0000 7eff4ffff700 1 -- 172.21.15.195:0/2884535720 <== mon.1 v2:172.21.15.202:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6460+0+0 (secure 0 0 0) 0x7eff5c081ed0 con 0x7eff6806e850 2022-06-27T11:55:17.174 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.141+0000 7eff677fe700 1 --2- 172.21.15.195:0/2884535720 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7eff48063d30 0x7eff480661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:17.174 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.141+0000 7eff677fe700 1 --2- 172.21.15.195:0/2884535720 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7eff48063d30 0x7eff480661e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7eff580079d0 tx=0x7eff58007bf0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:55:17.175 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.146+0000 7eff4ffff700 1 -- 172.21.15.195:0/2884535720 <== mon.1 v2:172.21.15.202:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7eff5c0509f0 con 0x7eff6806e850 2022-06-27T11:55:17.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.371+0000 7eff6e4e5700 1 -- 172.21.15.195:0/2884535720 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7eff68154420 con 0x7eff6806e850 2022-06-27T11:55:17.373 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.373+0000 7eff4ffff700 1 -- 172.21.15.195:0/2884535720 <== mon.1 v2:172.21.15.202:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5367 (secure 0 0 0) 0x7eff5c02a020 con 0x7eff6806e850 2022-06-27T11:55:17.373 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:55:17.374 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:55:09.933730+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4856},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4856":{"gid":4856,"name":"d","rank":0,"incarnation":26,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6835/2970948376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2970948376},{"type":"v1","addr":"172.21.15.202:6835","nonce":2970948376}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:55:17.377 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.374+0000 7eff6e4e5700 1 -- 172.21.15.195:0/2884535720 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7eff48063d30 msgr2=0x7eff480661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:17.377 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.374+0000 7eff6e4e5700 1 --2- 172.21.15.195:0/2884535720 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7eff48063d30 0x7eff480661e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7eff580079d0 tx=0x7eff58007bf0 comp rx=0 tx=0).stop 2022-06-27T11:55:17.377 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.374+0000 7eff6e4e5700 1 -- 172.21.15.195:0/2884535720 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7eff6806e850 msgr2=0x7eff68154770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:17.378 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.374+0000 7eff6e4e5700 1 --2- 172.21.15.195:0/2884535720 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7eff6806e850 0x7eff68154770 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7eff5c009e40 tx=0x7eff5c017000 comp rx=0 tx=0).stop 2022-06-27T11:55:17.378 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.374+0000 7eff6e4e5700 1 -- 172.21.15.195:0/2884535720 shutdown_connections 2022-06-27T11:55:17.379 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.374+0000 7eff6e4e5700 1 --2- 172.21.15.195:0/2884535720 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7eff48063d30 0x7eff480661e0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:17.379 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.374+0000 7eff6e4e5700 1 --2- 172.21.15.195:0/2884535720 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7eff680c05c0 0x7eff68155a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:17.379 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.374+0000 7eff6e4e5700 1 --2- 172.21.15.195:0/2884535720 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7eff6806e850 0x7eff68154770 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:17.380 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.374+0000 7eff6e4e5700 1 --2- 172.21.15.195:0/2884535720 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7eff680c34a0 0x7eff68154cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:17.380 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.374+0000 7eff6e4e5700 1 -- 172.21.15.195:0/2884535720 >> 172.21.15.195:0/2884535720 conn(0x7eff680c6350 msgr2=0x7eff680c2930 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:55:17.380 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.374+0000 7eff6e4e5700 1 -- 172.21.15.195:0/2884535720 shutdown_connections 2022-06-27T11:55:17.380 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:17.374+0000 7eff6e4e5700 1 -- 172.21.15.195:0/2884535720 wait complete. 2022-06-27T11:55:17.381 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 26 2022-06-27T11:55:17.388 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T11:55:19.394 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:55:19.639 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.538+0000 7f533dff6700 1 Processor -- start 2022-06-27T11:55:19.640 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.538+0000 7f533dff6700 1 -- start start 2022-06-27T11:55:19.640 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.539+0000 7f533dff6700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f5338150500 0x7f5338063af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:19.641 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.539+0000 7f533dff6700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5338064030 con 0x7f53381508d0 2022-06-27T11:55:19.641 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.539+0000 7f533dff6700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53380c4160 con 0x7f533806a450 2022-06-27T11:55:19.642 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.539+0000 7f533dff6700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53380c4440 con 0x7f5338150500 2022-06-27T11:55:19.642 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.539+0000 7f53377fe700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f5338150500 0x7f5338063af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:19.642 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.539+0000 7f5337fff700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f53381508d0 0x7f53380bbf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:50602/0 (socket says 172.21.15.195:50602) 2022-06-27T11:55:19.643 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.539+0000 7f5337fff700 1 -- 172.21.15.195:0/424417398 learned_addr learned my addr 172.21.15.195:0/424417398 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:55:19.643 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.539+0000 7f53377fe700 1 -- 172.21.15.195:0/424417398 >> v1:172.21.15.202:6789/0 conn(0x7f533806a450 legacy=0x7f53380c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:55:19.644 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.539+0000 7f53377fe700 1 -- 172.21.15.195:0/424417398 >> v1:172.21.15.195:6789/0 conn(0x7f53381508d0 legacy=0x7f53380bbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:19.644 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.539+0000 7f53367fc700 1 -- 172.21.15.195:0/424417398 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3600556743 0 0) 0x7f5330001490 con 0x7f53381508d0 2022-06-27T11:55:19.645 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.539+0000 7f53377fe700 1 -- 172.21.15.195:0/424417398 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f533815a800 con 0x7f5338150500 2022-06-27T11:55:19.645 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.539+0000 7f53377fe700 1 --2- 172.21.15.195:0/424417398 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5338150500 0x7f5338063af0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f5328009720 tx=0x7f5328009ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4afe79d5e5e90d62 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:55:19.645 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.539+0000 7f53367fc700 1 -- 172.21.15.195:0/424417398 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5328005ad0 con 0x7f5338150500 2022-06-27T11:55:19.646 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.540+0000 7f53367fc700 1 -- 172.21.15.195:0/424417398 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5328005c30 con 0x7f5338150500 2022-06-27T11:55:19.646 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.540+0000 7f53367fc700 1 -- 172.21.15.195:0/424417398 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5328006030 con 0x7f5338150500 2022-06-27T11:55:19.647 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.540+0000 7f533dff6700 1 -- 172.21.15.195:0/424417398 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5338150500 msgr2=0x7f5338063af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:19.647 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.540+0000 7f533dff6700 1 --2- 172.21.15.195:0/424417398 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5338150500 0x7f5338063af0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f5328009720 tx=0x7f5328009ae0 comp rx=0 tx=0).stop 2022-06-27T11:55:19.648 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.540+0000 7f533dff6700 1 -- 172.21.15.195:0/424417398 shutdown_connections 2022-06-27T11:55:19.648 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.540+0000 7f533dff6700 1 --2- 172.21.15.195:0/424417398 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5338150500 0x7f5338063af0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:19.648 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.540+0000 7f533dff6700 1 -- 172.21.15.195:0/424417398 >> 172.21.15.195:0/424417398 conn(0x7f53380c9b00 msgr2=0x7f53380c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:55:19.649 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.540+0000 7f533dff6700 1 -- 172.21.15.195:0/424417398 shutdown_connections 2022-06-27T11:55:19.649 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.540+0000 7f533dff6700 1 -- 172.21.15.195:0/424417398 wait complete. 2022-06-27T11:55:19.667 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.542+0000 7f533dff6700 1 Processor -- start 2022-06-27T11:55:19.667 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.542+0000 7f533dff6700 1 -- start start 2022-06-27T11:55:19.668 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.543+0000 7f533dff6700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f533806a450 0x7f53380bf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:19.668 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.543+0000 7f533dff6700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5338150500 0x7f5338166bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:19.668 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.543+0000 7f533dff6700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f53381508d0 0x7f5338167110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:19.669 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.543+0000 7f533dff6700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5338167650 con 0x7f5338150500 2022-06-27T11:55:19.669 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.543+0000 7f533dff6700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5338158880 con 0x7f53381508d0 2022-06-27T11:55:19.669 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.543+0000 7f533dff6700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5338158b60 con 0x7f533806a450 2022-06-27T11:55:19.670 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.543+0000 7f5336ffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5338150500 0x7f5338166bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:19.670 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.543+0000 7f5336ffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5338150500 0x7f5338166bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:44418/0 (socket says 172.21.15.195:44418) 2022-06-27T11:55:19.671 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.543+0000 7f5336ffd700 1 -- 172.21.15.195:0/391901047 learned_addr learned my addr 172.21.15.195:0/391901047 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:55:19.671 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.543+0000 7f5337fff700 1 --2- 172.21.15.195:0/391901047 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f53381508d0 0x7f5338167110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:19.671 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.543+0000 7f5336ffd700 1 -- 172.21.15.195:0/391901047 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f533806a450 msgr2=0x7f53380bf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:19.671 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.543+0000 7f53377fe700 1 --2- 172.21.15.195:0/391901047 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f533806a450 0x7f53380bf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:19.672 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.543+0000 7f5336ffd700 1 --2- 172.21.15.195:0/391901047 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f533806a450 0x7f53380bf0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:19.672 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.543+0000 7f5336ffd700 1 -- 172.21.15.195:0/391901047 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f53381508d0 msgr2=0x7f5338167110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:19.672 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.543+0000 7f5336ffd700 1 --2- 172.21.15.195:0/391901047 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f53381508d0 0x7f5338167110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:19.672 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.543+0000 7f5336ffd700 1 -- 172.21.15.195:0/391901047 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5328009290 con 0x7f5338150500 2022-06-27T11:55:19.673 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.543+0000 7f5337fff700 1 --2- 172.21.15.195:0/391901047 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f53381508d0 0x7f5338167110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T11:55:19.673 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.543+0000 7f53377fe700 1 --2- 172.21.15.195:0/391901047 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f533806a450 0x7f53380bf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T11:55:19.673 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.543+0000 7f5336ffd700 1 --2- 172.21.15.195:0/391901047 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5338150500 0x7f5338166bd0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f532000c490 tx=0x7f5320007e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:55:19.673 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.543+0000 7f532f7fe700 1 -- 172.21.15.195:0/391901047 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53200050d0 con 0x7f5338150500 2022-06-27T11:55:19.674 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.543+0000 7f532f7fe700 1 -- 172.21.15.195:0/391901047 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5320005230 con 0x7f5338150500 2022-06-27T11:55:19.674 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.543+0000 7f533dff6700 1 -- 172.21.15.195:0/391901047 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5338158e40 con 0x7f5338150500 2022-06-27T11:55:19.674 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.543+0000 7f532f7fe700 1 -- 172.21.15.195:0/391901047 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53200117a0 con 0x7f5338150500 2022-06-27T11:55:19.674 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.544+0000 7f533dff6700 1 -- 172.21.15.195:0/391901047 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5338159400 con 0x7f5338150500 2022-06-27T11:55:19.674 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.545+0000 7f533dff6700 1 -- 172.21.15.195:0/391901047 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53380722c0 con 0x7f5338150500 2022-06-27T11:55:19.675 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.545+0000 7f532f7fe700 1 -- 172.21.15.195:0/391901047 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f5320019030 con 0x7f5338150500 2022-06-27T11:55:19.676 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.545+0000 7f532f7fe700 1 --2- 172.21.15.195:0/391901047 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f531c063d30 0x7f531c0661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:19.676 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.545+0000 7f53377fe700 1 --2- 172.21.15.195:0/391901047 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f531c063d30 0x7f531c0661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:19.676 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.545+0000 7f532f7fe700 1 -- 172.21.15.195:0/391901047 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f532007fa40 con 0x7f5338150500 2022-06-27T11:55:19.676 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.546+0000 7f53377fe700 1 --2- 172.21.15.195:0/391901047 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f531c063d30 0x7f531c0661e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f5328009720 tx=0x7f5328013000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:55:19.677 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.550+0000 7f532f7fe700 1 -- 172.21.15.195:0/391901047 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f5320049c90 con 0x7f5338150500 2022-06-27T11:55:19.779 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.776+0000 7f533dff6700 1 -- 172.21.15.195:0/391901047 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f53380ba8e0 con 0x7f5338150500 2022-06-27T11:55:19.780 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.777+0000 7f532f7fe700 1 -- 172.21.15.195:0/391901047 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+5371 (secure 0 0 0) 0x7f5320017020 con 0x7f5338150500 2022-06-27T11:55:19.780 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:55:19.781 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:55:18.940402+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4856},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4856":{"gid":4856,"name":"d","rank":0,"incarnation":26,"state":"up:reconnect","state_seq":37,"addr":"172.21.15.202:6835/2970948376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2970948376},{"type":"v1","addr":"172.21.15.202:6835","nonce":2970948376}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:55:19.847 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.779+0000 7f533dff6700 1 -- 172.21.15.195:0/391901047 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f531c063d30 msgr2=0x7f531c0661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:19.848 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.779+0000 7f533dff6700 1 --2- 172.21.15.195:0/391901047 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f531c063d30 0x7f531c0661e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f5328009720 tx=0x7f5328013000 comp rx=0 tx=0).stop 2022-06-27T11:55:19.848 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.779+0000 7f533dff6700 1 -- 172.21.15.195:0/391901047 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5338150500 msgr2=0x7f5338166bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:19.849 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.779+0000 7f533dff6700 1 --2- 172.21.15.195:0/391901047 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5338150500 0x7f5338166bd0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f532000c490 tx=0x7f5320007e40 comp rx=0 tx=0).stop 2022-06-27T11:55:19.849 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.779+0000 7f533dff6700 1 -- 172.21.15.195:0/391901047 shutdown_connections 2022-06-27T11:55:19.849 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.779+0000 7f533dff6700 1 --2- 172.21.15.195:0/391901047 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f531c063d30 0x7f531c0661e0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:19.850 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.779+0000 7f533dff6700 1 --2- 172.21.15.195:0/391901047 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f53381508d0 0x7f5338167110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:19.850 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.779+0000 7f533dff6700 1 --2- 172.21.15.195:0/391901047 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f533806a450 0x7f53380bf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:19.851 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.779+0000 7f533dff6700 1 --2- 172.21.15.195:0/391901047 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5338150500 0x7f5338166bd0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:19.851 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.779+0000 7f533dff6700 1 -- 172.21.15.195:0/391901047 >> 172.21.15.195:0/391901047 conn(0x7f53380c9b00 msgr2=0x7f533806eca0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:55:19.852 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.779+0000 7f533dff6700 1 -- 172.21.15.195:0/391901047 shutdown_connections 2022-06-27T11:55:19.852 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:19.779+0000 7f533dff6700 1 -- 172.21.15.195:0/391901047 wait complete. 2022-06-27T11:55:19.853 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 27 2022-06-27T11:55:19.854 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T11:55:21.855 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:55:22.007 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:21.992+0000 7fc680216700 1 Processor -- start 2022-06-27T11:55:22.009 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:21.992+0000 7fc680216700 1 -- start start 2022-06-27T11:55:22.009 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:21.993+0000 7fc680216700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fc678156100 0x7fc6780688a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:22.010 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:21.993+0000 7fc680216700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc67806a440 0x7fc6780c4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:22.010 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:21.993+0000 7fc680216700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6780c4b10 con 0x7fc6781564d0 2022-06-27T11:55:22.011 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:21.993+0000 7fc680216700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6780c3dc0 con 0x7fc678156100 2022-06-27T11:55:22.011 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:21.993+0000 7fc680216700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc678068f20 con 0x7fc67806a440 2022-06-27T11:55:22.011 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:21.993+0000 7fc67e7b3700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fc6781564d0 0x7fc6780c28c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:50618/0 (socket says 172.21.15.195:50618) 2022-06-27T11:55:22.012 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:21.993+0000 7fc67e7b3700 1 -- 172.21.15.195:0/2294623014 learned_addr learned my addr 172.21.15.195:0/2294623014 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:55:22.012 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:21.993+0000 7fc67d7b1700 1 --2- 172.21.15.195:0/2294623014 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc67806a440 0x7fc6780c4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:22.013 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:21.993+0000 7fc67dfb2700 1 --2- 172.21.15.195:0/2294623014 >> v2:172.21.15.202:3300/0 conn(0x7fc678156100 0x7fc6780688a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:22.013 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:21.993+0000 7fc67cfb0700 1 -- 172.21.15.195:0/2294623014 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1805330964 0 0) 0x7fc6780c4b10 con 0x7fc6781564d0 2022-06-27T11:55:22.015 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:21.994+0000 7fc67cfb0700 1 -- 172.21.15.195:0/2294623014 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc660003430 con 0x7fc6781564d0 2022-06-27T11:55:22.016 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:21.994+0000 7fc67cfb0700 1 -- 172.21.15.195:0/2294623014 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3119686695 0 0) 0x7fc660003430 con 0x7fc6781564d0 2022-06-27T11:55:22.017 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:21.994+0000 7fc67d7b1700 1 -- 172.21.15.195:0/2294623014 >> v2:172.21.15.202:3300/0 conn(0x7fc678156100 msgr2=0x7fc6780688a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:55:22.017 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:21.994+0000 7fc67d7b1700 1 --2- 172.21.15.195:0/2294623014 >> v2:172.21.15.202:3300/0 conn(0x7fc678156100 0x7fc6780688a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:22.017 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:21.994+0000 7fc67d7b1700 1 -- 172.21.15.195:0/2294623014 >> v1:172.21.15.195:6789/0 conn(0x7fc6781564d0 legacy=0x7fc6780c28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:22.018 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:21.994+0000 7fc67d7b1700 1 -- 172.21.15.195:0/2294623014 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6780c6360 con 0x7fc67806a440 2022-06-27T11:55:22.018 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:21.994+0000 7fc67dfb2700 1 --2- 172.21.15.195:0/2294623014 >> v2:172.21.15.202:3300/0 conn(0x7fc678156100 0x7fc6780688a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:55:22.019 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:21.994+0000 7fc67d7b1700 1 --2- 172.21.15.195:0/2294623014 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc67806a440 0x7fc6780c4fa0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc6680097a0 tx=0x7fc668009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19b8ecc1109b8ed3 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:55:22.019 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:21.994+0000 7fc67cfb0700 1 -- 172.21.15.195:0/2294623014 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6700017f0 con 0x7fc67806a440 2022-06-27T11:55:22.020 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:21.994+0000 7fc67cfb0700 1 -- 172.21.15.195:0/2294623014 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc668004030 con 0x7fc67806a440 2022-06-27T11:55:22.021 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:21.994+0000 7fc67cfb0700 1 -- 172.21.15.195:0/2294623014 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6680043b0 con 0x7fc67806a440 2022-06-27T11:55:22.022 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:21.995+0000 7fc680216700 1 -- 172.21.15.195:0/2294623014 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc67806a440 msgr2=0x7fc6780c4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:22.022 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:21.995+0000 7fc680216700 1 --2- 172.21.15.195:0/2294623014 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc67806a440 0x7fc6780c4fa0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc6680097a0 tx=0x7fc668009060 comp rx=0 tx=0).stop 2022-06-27T11:55:22.023 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:21.995+0000 7fc680216700 1 -- 172.21.15.195:0/2294623014 shutdown_connections 2022-06-27T11:55:22.026 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:21.995+0000 7fc680216700 1 --2- 172.21.15.195:0/2294623014 >> v2:172.21.15.202:3300/0 conn(0x7fc678156100 0x7fc6780688a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:22.026 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:21.995+0000 7fc680216700 1 --2- 172.21.15.195:0/2294623014 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc67806a440 0x7fc6780c4fa0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:22.027 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:21.995+0000 7fc680216700 1 -- 172.21.15.195:0/2294623014 >> 172.21.15.195:0/2294623014 conn(0x7fc6780c9b00 msgr2=0x7fc6780c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:55:22.028 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:21.995+0000 7fc680216700 1 -- 172.21.15.195:0/2294623014 shutdown_connections 2022-06-27T11:55:22.028 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:21.995+0000 7fc680216700 1 -- 172.21.15.195:0/2294623014 wait complete. 2022-06-27T11:55:22.029 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:21.996+0000 7fc680216700 1 Processor -- start 2022-06-27T11:55:22.029 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:21.996+0000 7fc680216700 1 -- start start 2022-06-27T11:55:22.030 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:21.997+0000 7fc680216700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc67806a440 0x7fc678153ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:22.030 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:21.997+0000 7fc680216700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc678156100 0x7fc678154430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:22.031 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:21.997+0000 7fc680216700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc6781564d0 0x7fc678151230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:22.031 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:21.997+0000 7fc680216700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc678151f80 con 0x7fc678156100 2022-06-27T11:55:22.032 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:21.997+0000 7fc680216700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc678149640 con 0x7fc67806a440 2022-06-27T11:55:22.032 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:21.997+0000 7fc680216700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6780bf650 con 0x7fc6781564d0 2022-06-27T11:55:22.033 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:21.997+0000 7fc67d7b1700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc678156100 0x7fc678154430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:22.033 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:21.997+0000 7fc67e7b3700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc6781564d0 0x7fc678151230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:22.035 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:21.997+0000 7fc67d7b1700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc678156100 0x7fc678154430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:44434/0 (socket says 172.21.15.195:44434) 2022-06-27T11:55:22.037 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:21.997+0000 7fc67d7b1700 1 -- 172.21.15.195:0/544505256 learned_addr learned my addr 172.21.15.195:0/544505256 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:55:22.037 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:21.997+0000 7fc67dfb2700 1 --2- 172.21.15.195:0/544505256 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc67806a440 0x7fc678153ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:22.038 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:21.998+0000 7fc67d7b1700 1 -- 172.21.15.195:0/544505256 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc6781564d0 msgr2=0x7fc678151230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:22.039 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:21.998+0000 7fc67d7b1700 1 --2- 172.21.15.195:0/544505256 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc6781564d0 0x7fc678151230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:22.039 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:21.998+0000 7fc67d7b1700 1 -- 172.21.15.195:0/544505256 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc67806a440 msgr2=0x7fc678153ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:22.039 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:21.998+0000 7fc67d7b1700 1 --2- 172.21.15.195:0/544505256 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc67806a440 0x7fc678153ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:22.040 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:21.998+0000 7fc67d7b1700 1 -- 172.21.15.195:0/544505256 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc674009290 con 0x7fc678156100 2022-06-27T11:55:22.040 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:21.998+0000 7fc67d7b1700 1 --2- 172.21.15.195:0/544505256 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc678156100 0x7fc678154430 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fc6680046a0 tx=0x7fc668015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:55:22.041 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:21.998+0000 7fc665ffb700 1 -- 172.21.15.195:0/544505256 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc668003e70 con 0x7fc678156100 2022-06-27T11:55:22.041 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:21.998+0000 7fc665ffb700 1 -- 172.21.15.195:0/544505256 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc668011650 con 0x7fc678156100 2022-06-27T11:55:22.042 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:21.998+0000 7fc680216700 1 -- 172.21.15.195:0/544505256 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6680092e0 con 0x7fc678156100 2022-06-27T11:55:22.042 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:21.998+0000 7fc665ffb700 1 -- 172.21.15.195:0/544505256 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc668011a10 con 0x7fc678156100 2022-06-27T11:55:22.042 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:21.998+0000 7fc680216700 1 -- 172.21.15.195:0/544505256 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc678149360 con 0x7fc678156100 2022-06-27T11:55:22.043 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:21.999+0000 7fc680216700 1 -- 172.21.15.195:0/544505256 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6780bf800 con 0x7fc678156100 2022-06-27T11:55:22.043 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:22.000+0000 7fc665ffb700 1 -- 172.21.15.195:0/544505256 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fc66800f070 con 0x7fc678156100 2022-06-27T11:55:22.044 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:22.000+0000 7fc665ffb700 1 --2- 172.21.15.195:0/544505256 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc658063d00 0x7fc6580661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:22.045 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:22.000+0000 7fc665ffb700 1 -- 172.21.15.195:0/544505256 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fc6680827d0 con 0x7fc678156100 2022-06-27T11:55:22.045 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:22.000+0000 7fc67dfb2700 1 --2- 172.21.15.195:0/544505256 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc658063d00 0x7fc6580661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:22.047 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:22.000+0000 7fc67dfb2700 1 --2- 172.21.15.195:0/544505256 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc658063d00 0x7fc6580661b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc674009740 tx=0x7fc6740096d0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:55:22.047 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:22.004+0000 7fc665ffb700 1 -- 172.21.15.195:0/544505256 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fc6780bf800 con 0x7fc678156100 2022-06-27T11:55:22.234 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:22.230+0000 7fc680216700 1 -- 172.21.15.195:0/544505256 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc6780bf800 con 0x7fc678156100 2022-06-27T11:55:22.235 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:22.231+0000 7fc665ffb700 1 -- 172.21.15.195:0/544505256 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+5371 (secure 0 0 0) 0x7fc6780bf800 con 0x7fc678156100 2022-06-27T11:55:22.236 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:55:22.236 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:55:18.940402+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4856},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4856":{"gid":4856,"name":"d","rank":0,"incarnation":26,"state":"up:reconnect","state_seq":37,"addr":"172.21.15.202:6835/2970948376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2970948376},{"type":"v1","addr":"172.21.15.202:6835","nonce":2970948376}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:55:22.241 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:22.233+0000 7fc680216700 1 -- 172.21.15.195:0/544505256 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc658063d00 msgr2=0x7fc6580661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:22.242 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:22.233+0000 7fc680216700 1 --2- 172.21.15.195:0/544505256 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc658063d00 0x7fc6580661b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc674009740 tx=0x7fc6740096d0 comp rx=0 tx=0).stop 2022-06-27T11:55:22.242 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:22.233+0000 7fc680216700 1 -- 172.21.15.195:0/544505256 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc678156100 msgr2=0x7fc678154430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:22.242 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:22.233+0000 7fc680216700 1 --2- 172.21.15.195:0/544505256 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc678156100 0x7fc678154430 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fc6680046a0 tx=0x7fc668015000 comp rx=0 tx=0).stop 2022-06-27T11:55:22.243 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:22.233+0000 7fc680216700 1 -- 172.21.15.195:0/544505256 shutdown_connections 2022-06-27T11:55:22.243 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:22.233+0000 7fc680216700 1 --2- 172.21.15.195:0/544505256 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc658063d00 0x7fc6580661b0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:22.243 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:22.233+0000 7fc680216700 1 --2- 172.21.15.195:0/544505256 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc6781564d0 0x7fc678151230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:22.243 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:22.233+0000 7fc680216700 1 --2- 172.21.15.195:0/544505256 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc67806a440 0x7fc678153ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:22.244 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:22.233+0000 7fc680216700 1 --2- 172.21.15.195:0/544505256 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc678156100 0x7fc678154430 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:22.244 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:22.233+0000 7fc680216700 1 -- 172.21.15.195:0/544505256 >> 172.21.15.195:0/544505256 conn(0x7fc6780c9b00 msgr2=0x7fc678071200 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:55:22.244 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:22.233+0000 7fc680216700 1 -- 172.21.15.195:0/544505256 shutdown_connections 2022-06-27T11:55:22.244 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:22.233+0000 7fc680216700 1 -- 172.21.15.195:0/544505256 wait complete. 2022-06-27T11:55:22.245 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 27 2022-06-27T11:55:22.249 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T11:55:24.252 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:55:24.386 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.386+0000 7f8ce083d700 1 Processor -- start 2022-06-27T11:55:24.387 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.386+0000 7f8ce083d700 1 -- start start 2022-06-27T11:55:24.393 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.387+0000 7f8ce083d700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8cdc06f640 0x7f8cdc156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:24.394 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.387+0000 7f8ce083d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f8cdc0c4160 0x7f8cdc0c1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:24.394 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.387+0000 7f8ce083d700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cdc0c1a20 con 0x7f8cdc0c39e0 2022-06-27T11:55:24.395 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.387+0000 7f8ce083d700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cdc0c1130 con 0x7f8cdc0c4160 2022-06-27T11:55:24.395 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.387+0000 7f8ce083d700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cdc06ea90 con 0x7f8cdc06f640 2022-06-27T11:55:24.395 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.387+0000 7f8cd9d9b700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f8cdc0c4160 0x7f8cdc0c1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:24.396 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.387+0000 7f8cd9d9b700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f8cdc0c4160 0x7f8cdc0c1eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:44442/0 (socket says 172.21.15.195:44442) 2022-06-27T11:55:24.396 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.387+0000 7f8cd9d9b700 1 -- 172.21.15.195:0/4066026605 learned_addr learned my addr 172.21.15.195:0/4066026605 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:55:24.397 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.388+0000 7f8cda59c700 1 --2- 172.21.15.195:0/4066026605 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8cdc06f640 0x7f8cdc156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:24.397 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.388+0000 7f8cd9d9b700 1 -- 172.21.15.195:0/4066026605 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8cdc06f640 msgr2=0x7f8cdc156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:55:24.408 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.388+0000 7f8cd9d9b700 1 --2- 172.21.15.195:0/4066026605 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8cdc06f640 0x7f8cdc156ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:24.409 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.388+0000 7f8cd9d9b700 1 -- 172.21.15.195:0/4066026605 >> v1:172.21.15.202:6789/0 conn(0x7f8cdc0c39e0 legacy=0x7f8cdc1535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:55:24.409 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.388+0000 7f8cd9d9b700 1 -- 172.21.15.195:0/4066026605 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cdc06ed70 con 0x7f8cdc0c4160 2022-06-27T11:55:24.410 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.388+0000 7f8cd9d9b700 1 --2- 172.21.15.195:0/4066026605 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8cdc0c4160 0x7f8cdc0c1eb0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f8ccc009860 tx=0x7f8ccc009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21307387e3206102 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:55:24.410 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.388+0000 7f8cd959a700 1 -- 172.21.15.195:0/4066026605 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ccc00b710 con 0x7f8cdc0c4160 2022-06-27T11:55:24.410 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.388+0000 7f8cd959a700 1 -- 172.21.15.195:0/4066026605 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ccc00b870 con 0x7f8cdc0c4160 2022-06-27T11:55:24.411 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.388+0000 7f8cd959a700 1 -- 172.21.15.195:0/4066026605 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ccc00bc30 con 0x7f8cdc0c4160 2022-06-27T11:55:24.411 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.389+0000 7f8ce083d700 1 -- 172.21.15.195:0/4066026605 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8cdc0c4160 msgr2=0x7f8cdc0c1eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:24.412 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.389+0000 7f8ce083d700 1 --2- 172.21.15.195:0/4066026605 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8cdc0c4160 0x7f8cdc0c1eb0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f8ccc009860 tx=0x7f8ccc009120 comp rx=0 tx=0).stop 2022-06-27T11:55:24.412 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.389+0000 7f8ce083d700 1 -- 172.21.15.195:0/4066026605 shutdown_connections 2022-06-27T11:55:24.413 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.389+0000 7f8ce083d700 1 --2- 172.21.15.195:0/4066026605 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8cdc06f640 0x7f8cdc156ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:24.413 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.389+0000 7f8ce083d700 1 --2- 172.21.15.195:0/4066026605 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8cdc0c4160 0x7f8cdc0c1eb0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:24.414 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.389+0000 7f8ce083d700 1 -- 172.21.15.195:0/4066026605 >> 172.21.15.195:0/4066026605 conn(0x7f8cdc0c6350 msgr2=0x7f8cdc0c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:55:24.414 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.389+0000 7f8ce083d700 1 -- 172.21.15.195:0/4066026605 shutdown_connections 2022-06-27T11:55:24.415 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.389+0000 7f8ce083d700 1 -- 172.21.15.195:0/4066026605 wait complete. 2022-06-27T11:55:24.432 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.390+0000 7f8ce083d700 1 Processor -- start 2022-06-27T11:55:24.433 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.390+0000 7f8ce083d700 1 -- start start 2022-06-27T11:55:24.433 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.390+0000 7f8ce083d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8cdc06f640 0x7f8cdc1668e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:24.442 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.391+0000 7f8ce083d700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8cdc0c39e0 0x7f8cdc0bb950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:24.442 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.391+0000 7f8ce083d700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8cdc0c4160 0x7f8cdc0bc230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:24.443 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.391+0000 7f8ce083d700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cdc0bbe90 con 0x7f8cdc06f640 2022-06-27T11:55:24.443 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.391+0000 7f8ce083d700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cdc0bb4c0 con 0x7f8cdc0c4160 2022-06-27T11:55:24.444 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.391+0000 7f8ce083d700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cdc0ba330 con 0x7f8cdc0c39e0 2022-06-27T11:55:24.444 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.391+0000 7f8cda59c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8cdc06f640 0x7f8cdc1668e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:24.470 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.391+0000 7f8cda59c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8cdc06f640 0x7f8cdc1668e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:44446/0 (socket says 172.21.15.195:44446) 2022-06-27T11:55:24.471 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.391+0000 7f8cda59c700 1 -- 172.21.15.195:0/3988603497 learned_addr learned my addr 172.21.15.195:0/3988603497 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:55:24.472 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.391+0000 7f8cd9d9b700 1 --2- 172.21.15.195:0/3988603497 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8cdc0c39e0 0x7f8cdc0bb950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:24.472 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.391+0000 7f8cdad9d700 1 --2- 172.21.15.195:0/3988603497 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8cdc0c4160 0x7f8cdc0bc230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:24.472 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.391+0000 7f8cda59c700 1 -- 172.21.15.195:0/3988603497 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8cdc0c39e0 msgr2=0x7f8cdc0bb950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:24.473 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.391+0000 7f8cda59c700 1 --2- 172.21.15.195:0/3988603497 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8cdc0c39e0 0x7f8cdc0bb950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:24.473 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.391+0000 7f8cda59c700 1 -- 172.21.15.195:0/3988603497 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8cdc0c4160 msgr2=0x7f8cdc0bc230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:24.473 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.391+0000 7f8cda59c700 1 --2- 172.21.15.195:0/3988603497 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8cdc0c4160 0x7f8cdc0bc230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:24.494 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.391+0000 7f8cda59c700 1 -- 172.21.15.195:0/3988603497 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ccc0093a0 con 0x7f8cdc06f640 2022-06-27T11:55:24.495 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.391+0000 7f8cda59c700 1 --2- 172.21.15.195:0/3988603497 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8cdc06f640 0x7f8cdc1668e0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f8cd000d860 tx=0x7f8cd0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:55:24.495 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.391+0000 7f8cca7fc700 1 -- 172.21.15.195:0/3988603497 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cd00125f0 con 0x7f8cdc06f640 2022-06-27T11:55:24.495 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.391+0000 7f8cca7fc700 1 -- 172.21.15.195:0/3988603497 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cd0012750 con 0x7f8cdc06f640 2022-06-27T11:55:24.496 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.391+0000 7f8ce083d700 1 -- 172.21.15.195:0/3988603497 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cdc0ba5d0 con 0x7f8cdc06f640 2022-06-27T11:55:24.496 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.391+0000 7f8cca7fc700 1 -- 172.21.15.195:0/3988603497 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cd0012af0 con 0x7f8cdc06f640 2022-06-27T11:55:24.497 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.391+0000 7f8ce083d700 1 -- 172.21.15.195:0/3988603497 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cdc0bab90 con 0x7f8cdc06f640 2022-06-27T11:55:24.497 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.392+0000 7f8ce083d700 1 -- 172.21.15.195:0/3988603497 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cdc0bb7a0 con 0x7f8cdc06f640 2022-06-27T11:55:24.498 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.393+0000 7f8cca7fc700 1 -- 172.21.15.195:0/3988603497 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f8cd001e460 con 0x7f8cdc06f640 2022-06-27T11:55:24.498 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.393+0000 7f8cca7fc700 1 --2- 172.21.15.195:0/3988603497 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8cb8063d00 0x7f8cb80661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:24.498 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.393+0000 7f8cca7fc700 1 -- 172.21.15.195:0/3988603497 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f8cd0080f60 con 0x7f8cdc06f640 2022-06-27T11:55:24.499 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.393+0000 7f8cd9d9b700 1 --2- 172.21.15.195:0/3988603497 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8cb8063d00 0x7f8cb80661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:24.499 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.394+0000 7f8cd9d9b700 1 --2- 172.21.15.195:0/3988603497 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8cb8063d00 0x7f8cb80661b0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f8cdc0c0b70 tx=0x7f8ccc017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:55:24.500 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.397+0000 7f8cca7fc700 1 -- 172.21.15.195:0/3988603497 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f8cd004f960 con 0x7f8cdc06f640 2022-06-27T11:55:24.629 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.628+0000 7f8ce083d700 1 -- 172.21.15.195:0/3988603497 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8cdc0722c0 con 0x7f8cdc06f640 2022-06-27T11:55:24.629 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.628+0000 7f8cca7fc700 1 -- 172.21.15.195:0/3988603497 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+5371 (secure 0 0 0) 0x7f8cd0052f80 con 0x7f8cdc06f640 2022-06-27T11:55:24.630 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:55:24.630 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:55:18.940402+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4856},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4856":{"gid":4856,"name":"d","rank":0,"incarnation":26,"state":"up:reconnect","state_seq":37,"addr":"172.21.15.202:6835/2970948376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2970948376},{"type":"v1","addr":"172.21.15.202:6835","nonce":2970948376}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:55:24.637 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.630+0000 7f8ce083d700 1 -- 172.21.15.195:0/3988603497 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8cb8063d00 msgr2=0x7f8cb80661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:24.638 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.630+0000 7f8ce083d700 1 --2- 172.21.15.195:0/3988603497 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8cb8063d00 0x7f8cb80661b0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f8cdc0c0b70 tx=0x7f8ccc017000 comp rx=0 tx=0).stop 2022-06-27T11:55:24.638 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.630+0000 7f8ce083d700 1 -- 172.21.15.195:0/3988603497 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8cdc06f640 msgr2=0x7f8cdc1668e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:24.639 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.630+0000 7f8ce083d700 1 --2- 172.21.15.195:0/3988603497 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8cdc06f640 0x7f8cdc1668e0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f8cd000d860 tx=0x7f8cd0008040 comp rx=0 tx=0).stop 2022-06-27T11:55:24.639 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.630+0000 7f8ce083d700 1 -- 172.21.15.195:0/3988603497 shutdown_connections 2022-06-27T11:55:24.639 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.630+0000 7f8ce083d700 1 --2- 172.21.15.195:0/3988603497 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8cb8063d00 0x7f8cb80661b0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:24.640 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.630+0000 7f8ce083d700 1 --2- 172.21.15.195:0/3988603497 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8cdc06f640 0x7f8cdc1668e0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:24.640 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.630+0000 7f8ce083d700 1 --2- 172.21.15.195:0/3988603497 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8cdc0c4160 0x7f8cdc0bc230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:24.641 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.630+0000 7f8ce083d700 1 --2- 172.21.15.195:0/3988603497 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8cdc0c39e0 0x7f8cdc0bb950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:24.641 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.630+0000 7f8ce083d700 1 -- 172.21.15.195:0/3988603497 >> 172.21.15.195:0/3988603497 conn(0x7f8cdc0c6350 msgr2=0x7f8cdc151c10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:55:24.643 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.630+0000 7f8ce083d700 1 -- 172.21.15.195:0/3988603497 shutdown_connections 2022-06-27T11:55:24.643 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:24.630+0000 7f8ce083d700 1 -- 172.21.15.195:0/3988603497 wait complete. 2022-06-27T11:55:24.644 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 27 2022-06-27T11:55:24.649 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 27, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-06-27T11:47:52.358095+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 27, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4856": { "addr": "172.21.15.202:6835/2970948376", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6834", "nonce": 2970948376, "type": "v2" }, { "addr": "172.21.15.202:6835", "nonce": 2970948376, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4856, "incarnation": 26, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:reconnect", "state_seq": 37 } }, "last_failure": 0, "last_failure_osd_epoch": 51, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-06-27T11:55:18.940402+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4856 } } } ], "standbys": [ { "addr": "172.21.15.202:6837/3305006841", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6836", "nonce": 3305006841, "type": "v2" }, { "addr": "172.21.15.202:6837", "nonce": 3305006841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6835/2060709822", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6834", "nonce": 2060709822, "type": "v2" }, { "addr": "172.21.15.195:6835", "nonce": 2060709822, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6838/1598751652", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6836", "nonce": 1598751652, "type": "v2" }, { "addr": "172.21.15.195:6838", "nonce": 1598751652, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4275, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6839/1313800661", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6837", "nonce": 1313800661, "type": "v2" }, { "addr": "172.21.15.195:6839", "nonce": 1313800661, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4278, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-06-27T11:55:26.651 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:55:26.808 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:26.804+0000 7fb70f2ac700 1 Processor -- start 2022-06-27T11:55:26.811 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:26.804+0000 7fb70f2ac700 1 -- start start 2022-06-27T11:55:26.812 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:26.805+0000 7fb70f2ac700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fb70806e850 0x7fb7081570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:26.812 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:26.805+0000 7fb70f2ac700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb70806fef0 0x7fb708069de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:26.813 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:26.805+0000 7fb70f2ac700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb70806fba0 con 0x7fb7080c34a0 2022-06-27T11:55:26.813 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:26.805+0000 7fb70f2ac700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7081575e0 con 0x7fb70806e850 2022-06-27T11:55:26.814 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:26.805+0000 7fb70f2ac700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb70806f5e0 con 0x7fb70806fef0 2022-06-27T11:55:26.815 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:26.805+0000 7fb70d048700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fb70806e850 0x7fb7081570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:26.815 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:26.805+0000 7fb70d048700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fb70806e850 0x7fb7081570a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:44454/0 (socket says 172.21.15.195:44454) 2022-06-27T11:55:26.816 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:26.805+0000 7fb70d048700 1 -- 172.21.15.195:0/3193174063 learned_addr learned my addr 172.21.15.195:0/3193174063 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:55:26.816 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:26.806+0000 7fb70c847700 1 --2- 172.21.15.195:0/3193174063 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb70806fef0 0x7fb708069de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:26.817 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:26.806+0000 7fb70d048700 1 -- 172.21.15.195:0/3193174063 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb70806fef0 msgr2=0x7fb708069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:55:26.817 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:26.806+0000 7fb70d048700 1 --2- 172.21.15.195:0/3193174063 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb70806fef0 0x7fb708069de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:26.845 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:26.806+0000 7fb70d048700 1 -- 172.21.15.195:0/3193174063 >> v1:172.21.15.202:6789/0 conn(0x7fb7080c34a0 legacy=0x7fb7080c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:26.846 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:26.806+0000 7fb70d048700 1 -- 172.21.15.195:0/3193174063 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb70806f8c0 con 0x7fb70806e850 2022-06-27T11:55:26.846 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:26.806+0000 7fb70d048700 1 --2- 172.21.15.195:0/3193174063 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb70806e850 0x7fb7081570a0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fb70400bb10 tx=0x7fb70400b3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=450e90ca6a50931 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:55:26.846 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:26.806+0000 7fb6fbfff700 1 -- 172.21.15.195:0/3193174063 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb70400dbb0 con 0x7fb70806e850 2022-06-27T11:55:26.847 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:26.806+0000 7fb6fbfff700 1 -- 172.21.15.195:0/3193174063 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb70400dd10 con 0x7fb70806e850 2022-06-27T11:55:26.847 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:26.806+0000 7fb6fbfff700 1 -- 172.21.15.195:0/3193174063 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7040196b0 con 0x7fb70806e850 2022-06-27T11:55:26.847 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:26.807+0000 7fb70f2ac700 1 -- 172.21.15.195:0/3193174063 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb70806e850 msgr2=0x7fb7081570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:26.847 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:26.807+0000 7fb70f2ac700 1 --2- 172.21.15.195:0/3193174063 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb70806e850 0x7fb7081570a0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fb70400bb10 tx=0x7fb70400b3b0 comp rx=0 tx=0).stop 2022-06-27T11:55:26.848 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:26.807+0000 7fb70f2ac700 1 -- 172.21.15.195:0/3193174063 shutdown_connections 2022-06-27T11:55:26.848 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:26.807+0000 7fb70f2ac700 1 --2- 172.21.15.195:0/3193174063 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb70806e850 0x7fb7081570a0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:26.848 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:26.807+0000 7fb70f2ac700 1 --2- 172.21.15.195:0/3193174063 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb70806fef0 0x7fb708069de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:26.849 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:26.807+0000 7fb70f2ac700 1 -- 172.21.15.195:0/3193174063 >> 172.21.15.195:0/3193174063 conn(0x7fb7080c6350 msgr2=0x7fb7080c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:55:26.849 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:26.807+0000 7fb70f2ac700 1 -- 172.21.15.195:0/3193174063 shutdown_connections 2022-06-27T11:55:26.851 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:26.807+0000 7fb70f2ac700 1 -- 172.21.15.195:0/3193174063 wait complete. 2022-06-27T11:55:26.851 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:26.808+0000 7fb70f2ac700 1 Processor -- start 2022-06-27T11:55:26.851 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:26.808+0000 7fb70f2ac700 1 -- start start 2022-06-27T11:55:26.853 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:26.809+0000 7fb70f2ac700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb70806e850 0x7fb70816a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:26.854 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:26.809+0000 7fb70f2ac700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb7080c34a0 0x7fb7080c8360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:26.854 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:26.809+0000 7fb70f2ac700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb708155990 0x7fb7080bb400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:26.856 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:26.809+0000 7fb70f2ac700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7080bcb50 con 0x7fb7080c34a0 2022-06-27T11:55:26.856 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:26.809+0000 7fb70f2ac700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb70816a030 con 0x7fb70806e850 2022-06-27T11:55:26.857 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:26.809+0000 7fb70f2ac700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb708169d00 con 0x7fb708155990 2022-06-27T11:55:26.857 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:26.809+0000 7fb70c847700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb7080c34a0 0x7fb7080c8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:26.858 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:26.809+0000 7fb70c847700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb7080c34a0 0x7fb7080c8360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:44462/0 (socket says 172.21.15.195:44462) 2022-06-27T11:55:26.858 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:26.809+0000 7fb70c847700 1 -- 172.21.15.195:0/3073311876 learned_addr learned my addr 172.21.15.195:0/3073311876 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:55:26.859 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:26.809+0000 7fb70d048700 1 --2- 172.21.15.195:0/3073311876 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb70806e850 0x7fb70816a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:26.859 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:26.809+0000 7fb70d849700 1 --2- 172.21.15.195:0/3073311876 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb708155990 0x7fb7080bb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:26.859 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:26.809+0000 7fb70c847700 1 -- 172.21.15.195:0/3073311876 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb708155990 msgr2=0x7fb7080bb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:26.860 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:26.809+0000 7fb70c847700 1 --2- 172.21.15.195:0/3073311876 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb708155990 0x7fb7080bb400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:26.860 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:26.809+0000 7fb70c847700 1 -- 172.21.15.195:0/3073311876 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb70806e850 msgr2=0x7fb70816a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:26.872 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:26.809+0000 7fb70c847700 1 --2- 172.21.15.195:0/3073311876 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb70806e850 0x7fb70816a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:26.873 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:26.809+0000 7fb70c847700 1 -- 172.21.15.195:0/3073311876 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb70400b650 con 0x7fb7080c34a0 2022-06-27T11:55:26.874 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:26.809+0000 7fb70c847700 1 --2- 172.21.15.195:0/3073311876 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb7080c34a0 0x7fb7080c8360 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc00d8c0 tx=0x7fb6fc008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:55:26.874 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:26.809+0000 7fb6f97fa700 1 -- 172.21.15.195:0/3073311876 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6fc0125f0 con 0x7fb7080c34a0 2022-06-27T11:55:26.874 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:26.809+0000 7fb6f97fa700 1 -- 172.21.15.195:0/3073311876 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6fc012750 con 0x7fb7080c34a0 2022-06-27T11:55:26.875 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:26.809+0000 7fb70f2ac700 1 -- 172.21.15.195:0/3073311876 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb708166c80 con 0x7fb7080c34a0 2022-06-27T11:55:26.875 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:26.809+0000 7fb6f97fa700 1 -- 172.21.15.195:0/3073311876 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6fc012af0 con 0x7fb7080c34a0 2022-06-27T11:55:26.875 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:26.810+0000 7fb70f2ac700 1 -- 172.21.15.195:0/3073311876 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb708167240 con 0x7fb7080c34a0 2022-06-27T11:55:26.875 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:26.810+0000 7fb70f2ac700 1 -- 172.21.15.195:0/3073311876 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7080722c0 con 0x7fb7080c34a0 2022-06-27T11:55:26.876 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:26.811+0000 7fb6f97fa700 1 -- 172.21.15.195:0/3073311876 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fb6fc01e460 con 0x7fb7080c34a0 2022-06-27T11:55:26.876 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:26.811+0000 7fb6f97fa700 1 --2- 172.21.15.195:0/3073311876 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb6e8063d30 0x7fb6e80661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:26.876 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:26.811+0000 7fb6f97fa700 1 -- 172.21.15.195:0/3073311876 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fb6fc080f50 con 0x7fb7080c34a0 2022-06-27T11:55:26.876 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:26.811+0000 7fb70d048700 1 --2- 172.21.15.195:0/3073311876 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb6e8063d30 0x7fb6e80661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:26.877 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:26.812+0000 7fb70d048700 1 --2- 172.21.15.195:0/3073311876 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb6e8063d30 0x7fb6e80661e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb70400b3f0 tx=0x7fb70401b000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:55:26.877 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:26.816+0000 7fb6f97fa700 1 -- 172.21.15.195:0/3073311876 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fb6fc04f950 con 0x7fb7080c34a0 2022-06-27T11:55:27.043 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:27.042+0000 7fb70f2ac700 1 -- 172.21.15.195:0/3073311876 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7080c2f60 con 0x7fb7080c34a0 2022-06-27T11:55:27.045 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:27.042+0000 7fb6f97fa700 1 -- 172.21.15.195:0/3073311876 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5368 (secure 0 0 0) 0x7fb6fc052f70 con 0x7fb7080c34a0 2022-06-27T11:55:27.071 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:55:27.072 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:55:24.953586+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4856},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4856":{"gid":4856,"name":"d","rank":0,"incarnation":26,"state":"up:rejoin","state_seq":39,"addr":"172.21.15.202:6835/2970948376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2970948376},{"type":"v1","addr":"172.21.15.202:6835","nonce":2970948376}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:55:27.083 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:27.044+0000 7fb70f2ac700 1 -- 172.21.15.195:0/3073311876 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb6e8063d30 msgr2=0x7fb6e80661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:27.085 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:27.044+0000 7fb70f2ac700 1 --2- 172.21.15.195:0/3073311876 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb6e8063d30 0x7fb6e80661e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb70400b3f0 tx=0x7fb70401b000 comp rx=0 tx=0).stop 2022-06-27T11:55:27.086 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:27.044+0000 7fb70f2ac700 1 -- 172.21.15.195:0/3073311876 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb7080c34a0 msgr2=0x7fb7080c8360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:27.086 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:27.044+0000 7fb70f2ac700 1 --2- 172.21.15.195:0/3073311876 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb7080c34a0 0x7fb7080c8360 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc00d8c0 tx=0x7fb6fc008040 comp rx=0 tx=0).stop 2022-06-27T11:55:27.088 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:27.044+0000 7fb70f2ac700 1 -- 172.21.15.195:0/3073311876 shutdown_connections 2022-06-27T11:55:27.089 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:27.044+0000 7fb70f2ac700 1 --2- 172.21.15.195:0/3073311876 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb6e8063d30 0x7fb6e80661e0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:27.089 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:27.044+0000 7fb70f2ac700 1 --2- 172.21.15.195:0/3073311876 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb708155990 0x7fb7080bb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:27.089 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:27.044+0000 7fb70f2ac700 1 --2- 172.21.15.195:0/3073311876 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb70806e850 0x7fb70816a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:27.090 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:27.044+0000 7fb70f2ac700 1 --2- 172.21.15.195:0/3073311876 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb7080c34a0 0x7fb7080c8360 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:27.090 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:27.044+0000 7fb70f2ac700 1 -- 172.21.15.195:0/3073311876 >> 172.21.15.195:0/3073311876 conn(0x7fb7080c6350 msgr2=0x7fb708068b40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:55:27.091 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:27.044+0000 7fb70f2ac700 1 -- 172.21.15.195:0/3073311876 shutdown_connections 2022-06-27T11:55:27.091 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:27.044+0000 7fb70f2ac700 1 -- 172.21.15.195:0/3073311876 wait complete. 2022-06-27T11:55:27.094 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 28 2022-06-27T11:55:27.095 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T11:55:28.478 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T11:55:28.485 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T11:55:29.099 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:55:29.258 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.235+0000 7fd3a03f6700 1 Processor -- start 2022-06-27T11:55:29.258 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.235+0000 7fd3a03f6700 1 -- start start 2022-06-27T11:55:29.259 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.235+0000 7fd3a03f6700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fd39806e850 0x7fd3981570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:29.259 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.235+0000 7fd3a03f6700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd39806fef0 0x7fd398069de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:29.260 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.235+0000 7fd3a03f6700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd3980c34a0 0x7fd39806a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:29.260 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.235+0000 7fd3a03f6700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3981575e0 con 0x7fd3980c34a0 2022-06-27T11:55:29.261 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.235+0000 7fd3a03f6700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd39806fba0 con 0x7fd39806e850 2022-06-27T11:55:29.261 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.235+0000 7fd3a03f6700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd39806f5e0 con 0x7fd39806fef0 2022-06-27T11:55:29.261 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.235+0000 7fd39e993700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd3980c34a0 0x7fd39806a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:29.262 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.235+0000 7fd39e993700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd3980c34a0 0x7fd39806a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:44476/0 (socket says 172.21.15.195:44476) 2022-06-27T11:55:29.262 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.235+0000 7fd39e993700 1 -- 172.21.15.195:0/3719761492 learned_addr learned my addr 172.21.15.195:0/3719761492 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:55:29.263 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.235+0000 7fd39d991700 1 --2- 172.21.15.195:0/3719761492 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd39806fef0 0x7fd398069de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:29.263 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.235+0000 7fd39e192700 1 --2- 172.21.15.195:0/3719761492 >> v2:172.21.15.202:3300/0 conn(0x7fd39806e850 0x7fd3981570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:29.264 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.235+0000 7fd39e993700 1 -- 172.21.15.195:0/3719761492 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd39806fef0 msgr2=0x7fd398069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:55:29.264 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.235+0000 7fd39e993700 1 --2- 172.21.15.195:0/3719761492 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd39806fef0 0x7fd398069de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:29.264 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.235+0000 7fd39e993700 1 -- 172.21.15.195:0/3719761492 >> v2:172.21.15.202:3300/0 conn(0x7fd39806e850 msgr2=0x7fd3981570a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:55:29.265 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.235+0000 7fd39e993700 1 --2- 172.21.15.195:0/3719761492 >> v2:172.21.15.202:3300/0 conn(0x7fd39806e850 0x7fd3981570a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:29.265 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.235+0000 7fd39e993700 1 -- 172.21.15.195:0/3719761492 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd39806f8c0 con 0x7fd3980c34a0 2022-06-27T11:55:29.265 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.235+0000 7fd39e192700 1 --2- 172.21.15.195:0/3719761492 >> v2:172.21.15.202:3300/0 conn(0x7fd39806e850 0x7fd3981570a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T11:55:29.266 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.235+0000 7fd39e993700 1 --2- 172.21.15.195:0/3719761492 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd3980c34a0 0x7fd39806a320 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fd39806e120 tx=0x7fd39000ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a88e911bad9ff43 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:55:29.266 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.235+0000 7fd39d190700 1 -- 172.21.15.195:0/3719761492 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd390010070 con 0x7fd3980c34a0 2022-06-27T11:55:29.279 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.235+0000 7fd39d190700 1 -- 172.21.15.195:0/3719761492 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd39000b0d0 con 0x7fd3980c34a0 2022-06-27T11:55:29.280 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.235+0000 7fd39d190700 1 -- 172.21.15.195:0/3719761492 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd39000b490 con 0x7fd3980c34a0 2022-06-27T11:55:29.280 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.236+0000 7fd3a03f6700 1 -- 172.21.15.195:0/3719761492 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd3980c34a0 msgr2=0x7fd39806a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:29.281 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.236+0000 7fd3a03f6700 1 --2- 172.21.15.195:0/3719761492 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd3980c34a0 0x7fd39806a320 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fd39806e120 tx=0x7fd39000ba50 comp rx=0 tx=0).stop 2022-06-27T11:55:29.281 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.236+0000 7fd3a03f6700 1 -- 172.21.15.195:0/3719761492 shutdown_connections 2022-06-27T11:55:29.281 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.236+0000 7fd3a03f6700 1 --2- 172.21.15.195:0/3719761492 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd3980c34a0 0x7fd39806a320 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:29.282 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.236+0000 7fd3a03f6700 1 --2- 172.21.15.195:0/3719761492 >> v2:172.21.15.202:3300/0 conn(0x7fd39806e850 0x7fd3981570a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:29.282 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.236+0000 7fd3a03f6700 1 --2- 172.21.15.195:0/3719761492 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd39806fef0 0x7fd398069de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:29.283 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.236+0000 7fd3a03f6700 1 -- 172.21.15.195:0/3719761492 >> 172.21.15.195:0/3719761492 conn(0x7fd3980c6350 msgr2=0x7fd3980c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:55:29.283 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.236+0000 7fd3a03f6700 1 -- 172.21.15.195:0/3719761492 shutdown_connections 2022-06-27T11:55:29.294 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.236+0000 7fd3a03f6700 1 -- 172.21.15.195:0/3719761492 wait complete. 2022-06-27T11:55:29.295 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.238+0000 7fd3a03f6700 1 Processor -- start 2022-06-27T11:55:29.295 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.238+0000 7fd3a03f6700 1 -- start start 2022-06-27T11:55:29.296 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.238+0000 7fd3a03f6700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd39806e850 0x7fd3980c6900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:29.296 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.238+0000 7fd3a03f6700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd3980c34a0 0x7fd3980b6940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:29.297 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.239+0000 7fd3a03f6700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd398152440 0x7fd3980b6e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:29.297 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.239+0000 7fd3a03f6700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3980b8010 con 0x7fd398152440 2022-06-27T11:55:29.298 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.239+0000 7fd3a03f6700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3981656e0 con 0x7fd3980c34a0 2022-06-27T11:55:29.298 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.239+0000 7fd3a03f6700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd398162f30 con 0x7fd39806e850 2022-06-27T11:55:29.299 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.239+0000 7fd39e993700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd398152440 0x7fd3980b6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:29.299 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.239+0000 7fd39e993700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd398152440 0x7fd3980b6e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:44482/0 (socket says 172.21.15.195:44482) 2022-06-27T11:55:29.300 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.239+0000 7fd39e993700 1 -- 172.21.15.195:0/777604394 learned_addr learned my addr 172.21.15.195:0/777604394 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:55:29.300 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.239+0000 7fd39e192700 1 --2- 172.21.15.195:0/777604394 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd39806e850 0x7fd3980c6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:29.301 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.239+0000 7fd39d991700 1 --2- 172.21.15.195:0/777604394 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd3980c34a0 0x7fd3980b6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:29.301 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.239+0000 7fd39e993700 1 -- 172.21.15.195:0/777604394 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd39806e850 msgr2=0x7fd3980c6900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:29.301 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.239+0000 7fd39e993700 1 --2- 172.21.15.195:0/777604394 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd39806e850 0x7fd3980c6900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:29.311 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.239+0000 7fd39e993700 1 -- 172.21.15.195:0/777604394 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd3980c34a0 msgr2=0x7fd3980b6940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:29.312 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.239+0000 7fd39e993700 1 --2- 172.21.15.195:0/777604394 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd3980c34a0 0x7fd3980b6940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:29.312 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.239+0000 7fd39e993700 1 -- 172.21.15.195:0/777604394 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd39000bcf0 con 0x7fd398152440 2022-06-27T11:55:29.312 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.239+0000 7fd39e192700 1 --2- 172.21.15.195:0/777604394 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd39806e850 0x7fd3980c6900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T11:55:29.313 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.239+0000 7fd39d991700 1 --2- 172.21.15.195:0/777604394 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd3980c34a0 0x7fd3980b6940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T11:55:29.313 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.239+0000 7fd39e993700 1 --2- 172.21.15.195:0/777604394 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd398152440 0x7fd3980b6e80 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fd39000d8c0 tx=0x7fd39001a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:55:29.313 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.239+0000 7fd3867fc700 1 -- 172.21.15.195:0/777604394 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd39000bb50 con 0x7fd398152440 2022-06-27T11:55:29.314 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.239+0000 7fd3867fc700 1 -- 172.21.15.195:0/777604394 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd390009a30 con 0x7fd398152440 2022-06-27T11:55:29.314 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.239+0000 7fd3a03f6700 1 -- 172.21.15.195:0/777604394 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd398165980 con 0x7fd398152440 2022-06-27T11:55:29.315 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.239+0000 7fd3867fc700 1 -- 172.21.15.195:0/777604394 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd390009dd0 con 0x7fd398152440 2022-06-27T11:55:29.315 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.239+0000 7fd3a03f6700 1 -- 172.21.15.195:0/777604394 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd398165f40 con 0x7fd398152440 2022-06-27T11:55:29.328 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.240+0000 7fd3a03f6700 1 -- 172.21.15.195:0/777604394 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3980722c0 con 0x7fd398152440 2022-06-27T11:55:29.329 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.241+0000 7fd3867fc700 1 -- 172.21.15.195:0/777604394 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fd390004250 con 0x7fd398152440 2022-06-27T11:55:29.329 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.241+0000 7fd3867fc700 1 --2- 172.21.15.195:0/777604394 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd378063d30 0x7fd3780661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:29.330 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.241+0000 7fd3867fc700 1 -- 172.21.15.195:0/777604394 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fd390084f40 con 0x7fd398152440 2022-06-27T11:55:29.330 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.241+0000 7fd39e192700 1 --2- 172.21.15.195:0/777604394 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd378063d30 0x7fd3780661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:29.331 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.241+0000 7fd39e192700 1 --2- 172.21.15.195:0/777604394 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd378063d30 0x7fd3780661e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd394006fd0 tx=0x7fd394008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:55:29.331 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.246+0000 7fd3867fc700 1 -- 172.21.15.195:0/777604394 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fd3900b4c90 con 0x7fd398152440 2022-06-27T11:55:29.470 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.470+0000 7fd3a03f6700 1 -- 172.21.15.195:0/777604394 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd39806f240 con 0x7fd398152440 2022-06-27T11:55:29.471 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.471+0000 7fd3867fc700 1 -- 172.21.15.195:0/777604394 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5368 (secure 0 0 0) 0x7fd3900539a0 con 0x7fd398152440 2022-06-27T11:55:29.472 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:55:29.472 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:55:24.953586+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4856},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4856":{"gid":4856,"name":"d","rank":0,"incarnation":26,"state":"up:rejoin","state_seq":39,"addr":"172.21.15.202:6835/2970948376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2970948376},{"type":"v1","addr":"172.21.15.202:6835","nonce":2970948376}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:55:29.475 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.472+0000 7fd3a03f6700 1 -- 172.21.15.195:0/777604394 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd378063d30 msgr2=0x7fd3780661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:29.475 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.472+0000 7fd3a03f6700 1 --2- 172.21.15.195:0/777604394 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd378063d30 0x7fd3780661e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd394006fd0 tx=0x7fd394008040 comp rx=0 tx=0).stop 2022-06-27T11:55:29.476 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.472+0000 7fd3a03f6700 1 -- 172.21.15.195:0/777604394 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd398152440 msgr2=0x7fd3980b6e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:29.476 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.472+0000 7fd3a03f6700 1 --2- 172.21.15.195:0/777604394 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd398152440 0x7fd3980b6e80 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fd39000d8c0 tx=0x7fd39001a000 comp rx=0 tx=0).stop 2022-06-27T11:55:29.477 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.473+0000 7fd3a03f6700 1 -- 172.21.15.195:0/777604394 shutdown_connections 2022-06-27T11:55:29.477 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.473+0000 7fd3a03f6700 1 --2- 172.21.15.195:0/777604394 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd378063d30 0x7fd3780661e0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:29.478 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.473+0000 7fd3a03f6700 1 --2- 172.21.15.195:0/777604394 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd398152440 0x7fd3980b6e80 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:29.478 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.473+0000 7fd3a03f6700 1 --2- 172.21.15.195:0/777604394 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd39806e850 0x7fd3980c6900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:29.479 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.473+0000 7fd3a03f6700 1 --2- 172.21.15.195:0/777604394 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd3980c34a0 0x7fd3980b6940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:29.479 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.473+0000 7fd3a03f6700 1 -- 172.21.15.195:0/777604394 >> 172.21.15.195:0/777604394 conn(0x7fd3980c6350 msgr2=0x7fd3980c26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:55:29.479 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.473+0000 7fd3a03f6700 1 -- 172.21.15.195:0/777604394 shutdown_connections 2022-06-27T11:55:29.480 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:29.473+0000 7fd3a03f6700 1 -- 172.21.15.195:0/777604394 wait complete. 2022-06-27T11:55:29.480 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 28 2022-06-27T11:55:29.487 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T11:55:31.490 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:55:31.654 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.631+0000 7fd8e07f0700 1 Processor -- start 2022-06-27T11:55:31.655 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.631+0000 7fd8e07f0700 1 -- start start 2022-06-27T11:55:31.656 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.632+0000 7fd8e07f0700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd8d8150580 0x7fd8d8156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:31.656 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.632+0000 7fd8e07f0700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fd8d8150950 0x7fd8d80c7b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:31.656 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.632+0000 7fd8e07f0700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8d80c76b0 con 0x7fd8d806f620 2022-06-27T11:55:31.657 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.632+0000 7fd8e07f0700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8d8157430 con 0x7fd8d8150950 2022-06-27T11:55:31.657 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.632+0000 7fd8e07f0700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d80bad70 con 0x7fd8d8150580 2022-06-27T11:55:31.657 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.632+0000 7fd8ded8d700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fd8d806f620 0x7fd8d806d6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:50678/0 (socket says 172.21.15.195:50678) 2022-06-27T11:55:31.658 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.632+0000 7fd8ded8d700 1 -- 172.21.15.195:0/4055320666 learned_addr learned my addr 172.21.15.195:0/4055320666 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:55:31.658 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.633+0000 7fd8dd58a700 1 -- 172.21.15.195:0/4055320666 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1729249108 0 0) 0x7fd8d80c76b0 con 0x7fd8d806f620 2022-06-27T11:55:31.658 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.633+0000 7fd8de58c700 1 --2- 172.21.15.195:0/4055320666 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd8d8150580 0x7fd8d8156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:31.659 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.633+0000 7fd8dd58a700 1 -- 172.21.15.195:0/4055320666 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8c0003430 con 0x7fd8d806f620 2022-06-27T11:55:31.659 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.633+0000 7fd8ddd8b700 1 --2- 172.21.15.195:0/4055320666 >> v2:172.21.15.202:3300/0 conn(0x7fd8d8150950 0x7fd8d80c7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:31.659 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.633+0000 7fd8dd58a700 1 -- 172.21.15.195:0/4055320666 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1169615965 0 0) 0x7fd8c0003430 con 0x7fd8d806f620 2022-06-27T11:55:31.660 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.633+0000 7fd8dd58a700 1 -- 172.21.15.195:0/4055320666 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd8d8150580 msgr2=0x7fd8d8156ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:55:31.660 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.633+0000 7fd8dd58a700 1 --2- 172.21.15.195:0/4055320666 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd8d8150580 0x7fd8d8156ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:31.661 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.633+0000 7fd8dd58a700 1 -- 172.21.15.195:0/4055320666 >> v2:172.21.15.202:3300/0 conn(0x7fd8d8150950 msgr2=0x7fd8d80c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:55:31.661 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.633+0000 7fd8dd58a700 1 --2- 172.21.15.195:0/4055320666 >> v2:172.21.15.202:3300/0 conn(0x7fd8d8150950 0x7fd8d80c7b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:31.662 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.633+0000 7fd8dd58a700 1 -- 172.21.15.195:0/4055320666 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8d80ba4d0 con 0x7fd8d806f620 2022-06-27T11:55:31.662 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.633+0000 7fd8dd58a700 1 -- 172.21.15.195:0/4055320666 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7fd8d00017f0 con 0x7fd8d806f620 2022-06-27T11:55:31.663 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.633+0000 7fd8dd58a700 1 -- 172.21.15.195:0/4055320666 >> v1:172.21.15.195:6789/0 conn(0x7fd8d806f620 legacy=0x7fd8d806d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:31.663 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.633+0000 7fd8dd58a700 1 --2- 172.21.15.195:0/4055320666 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd8c0003920 0x7fd8c0003d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:31.663 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.633+0000 7fd8dd58a700 1 --2- 172.21.15.195:0/4055320666 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd8c00046d0 0x7fd8c0004b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:31.664 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.633+0000 7fd8dd58a700 1 --2- 172.21.15.195:0/4055320666 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd8d8150580 0x7fd8c0005080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:31.664 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.633+0000 7fd8dd58a700 1 -- 172.21.15.195:0/4055320666 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d80bad70 con 0x7fd8c0003920 2022-06-27T11:55:31.664 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.633+0000 7fd8dd58a700 1 -- 172.21.15.195:0/4055320666 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c000b920 con 0x7fd8c00046d0 2022-06-27T11:55:31.665 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.633+0000 7fd8dd58a700 1 -- 172.21.15.195:0/4055320666 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c000bae0 con 0x7fd8d8150580 2022-06-27T11:55:31.665 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.633+0000 7fd8de58c700 1 --2- 172.21.15.195:0/4055320666 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd8c0003920 0x7fd8c0003d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:31.665 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.633+0000 7fd8ddd8b700 1 --2- 172.21.15.195:0/4055320666 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd8c00046d0 0x7fd8c0004b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:31.666 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.633+0000 7fd8ded8d700 1 --2- 172.21.15.195:0/4055320666 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd8d8150580 0x7fd8c0005080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:31.667 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.633+0000 7fd8de58c700 1 -- 172.21.15.195:0/4055320666 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd8d8150580 msgr2=0x7fd8c0005080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:55:31.667 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.633+0000 7fd8de58c700 1 --2- 172.21.15.195:0/4055320666 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd8d8150580 0x7fd8c0005080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:31.668 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.633+0000 7fd8de58c700 1 -- 172.21.15.195:0/4055320666 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd8c00046d0 msgr2=0x7fd8c0004b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:55:31.668 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.633+0000 7fd8de58c700 1 --2- 172.21.15.195:0/4055320666 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd8c00046d0 0x7fd8c0004b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:31.668 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.633+0000 7fd8de58c700 1 -- 172.21.15.195:0/4055320666 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8c0003430 con 0x7fd8c0003920 2022-06-27T11:55:31.669 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.633+0000 7fd8de58c700 1 --2- 172.21.15.195:0/4055320666 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd8c0003920 0x7fd8c0003d30 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fd8c800ba40 tx=0x7fd8c8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6295472d6294475f server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:55:31.669 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.634+0000 7fd8dd58a700 1 -- 172.21.15.195:0/4055320666 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8c800be80 con 0x7fd8c0003920 2022-06-27T11:55:31.669 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.634+0000 7fd8dd58a700 1 -- 172.21.15.195:0/4055320666 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8c800d5d0 con 0x7fd8c0003920 2022-06-27T11:55:31.670 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.634+0000 7fd8dd58a700 1 -- 172.21.15.195:0/4055320666 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8c800d970 con 0x7fd8c0003920 2022-06-27T11:55:31.670 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.634+0000 7fd8e07f0700 1 -- 172.21.15.195:0/4055320666 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd8c0003920 msgr2=0x7fd8c0003d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:31.670 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.634+0000 7fd8e07f0700 1 --2- 172.21.15.195:0/4055320666 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd8c0003920 0x7fd8c0003d30 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fd8c800ba40 tx=0x7fd8c8008040 comp rx=0 tx=0).stop 2022-06-27T11:55:31.671 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.634+0000 7fd8ded8d700 1 -- 172.21.15.195:0/4055320666 reap_dead start 2022-06-27T11:55:31.671 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.634+0000 7fd8e07f0700 1 -- 172.21.15.195:0/4055320666 shutdown_connections 2022-06-27T11:55:31.672 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.634+0000 7fd8e07f0700 1 -- 172.21.15.195:0/4055320666 >> 172.21.15.195:0/4055320666 conn(0x7fd8d8063af0 msgr2=0x7fd8d8063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:55:31.672 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.634+0000 7fd8e07f0700 1 -- 172.21.15.195:0/4055320666 shutdown_connections 2022-06-27T11:55:31.672 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.634+0000 7fd8e07f0700 1 -- 172.21.15.195:0/4055320666 wait complete. 2022-06-27T11:55:31.673 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.635+0000 7fd8e07f0700 1 Processor -- start 2022-06-27T11:55:31.673 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.635+0000 7fd8e07f0700 1 -- start start 2022-06-27T11:55:31.673 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.636+0000 7fd8e07f0700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd8c0003920 0x7fd8d816a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:31.674 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.636+0000 7fd8e07f0700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd8d815aaf0 0x7fd8d80c59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:31.674 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.636+0000 7fd8e07f0700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd8d8157c10 0x7fd8d8166950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:31.674 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.636+0000 7fd8e07f0700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d8166530 con 0x7fd8d815aaf0 2022-06-27T11:55:31.675 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.636+0000 7fd8e07f0700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d815b270 con 0x7fd8c0003920 2022-06-27T11:55:31.675 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.636+0000 7fd8e07f0700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d80ca3f0 con 0x7fd8d8157c10 2022-06-27T11:55:31.676 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.636+0000 7fd8ddd8b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd8d815aaf0 0x7fd8d80c59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:31.676 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.636+0000 7fd8ddd8b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd8d815aaf0 0x7fd8d80c59b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:44502/0 (socket says 172.21.15.195:44502) 2022-06-27T11:55:31.679 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.636+0000 7fd8ddd8b700 1 -- 172.21.15.195:0/269397934 learned_addr learned my addr 172.21.15.195:0/269397934 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:55:31.679 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.636+0000 7fd8de58c700 1 --2- 172.21.15.195:0/269397934 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd8c0003920 0x7fd8d816a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:31.680 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.636+0000 7fd8ded8d700 1 --2- 172.21.15.195:0/269397934 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd8d8157c10 0x7fd8d8166950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:31.680 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.636+0000 7fd8ddd8b700 1 -- 172.21.15.195:0/269397934 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd8d8157c10 msgr2=0x7fd8d8166950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:31.681 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.637+0000 7fd8ddd8b700 1 --2- 172.21.15.195:0/269397934 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd8d8157c10 0x7fd8d8166950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:31.681 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.637+0000 7fd8ddd8b700 1 -- 172.21.15.195:0/269397934 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd8c0003920 msgr2=0x7fd8d816a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:31.682 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.637+0000 7fd8ddd8b700 1 --2- 172.21.15.195:0/269397934 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd8c0003920 0x7fd8d816a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:31.682 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.637+0000 7fd8ddd8b700 1 -- 172.21.15.195:0/269397934 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8d80c9b50 con 0x7fd8d815aaf0 2022-06-27T11:55:31.682 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.637+0000 7fd8ddd8b700 1 --2- 172.21.15.195:0/269397934 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd8d815aaf0 0x7fd8d80c59b0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fd8d40119f0 tx=0x7fd8d4009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:55:31.683 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.637+0000 7fd8c67fc700 1 -- 172.21.15.195:0/269397934 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8d4009560 con 0x7fd8d815aaf0 2022-06-27T11:55:31.683 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.637+0000 7fd8c67fc700 1 -- 172.21.15.195:0/269397934 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8d400f940 con 0x7fd8d815aaf0 2022-06-27T11:55:31.683 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.637+0000 7fd8c67fc700 1 -- 172.21.15.195:0/269397934 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8d400fd00 con 0x7fd8d815aaf0 2022-06-27T11:55:31.684 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.637+0000 7fd8e07f0700 1 -- 172.21.15.195:0/269397934 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8d80c9e30 con 0x7fd8d815aaf0 2022-06-27T11:55:31.684 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.637+0000 7fd8e07f0700 1 -- 172.21.15.195:0/269397934 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8d80c9850 con 0x7fd8d815aaf0 2022-06-27T11:55:31.685 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.638+0000 7fd8e07f0700 1 -- 172.21.15.195:0/269397934 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8d8166670 con 0x7fd8d815aaf0 2022-06-27T11:55:31.685 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.638+0000 7fd8c67fc700 1 -- 172.21.15.195:0/269397934 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fd8d401d690 con 0x7fd8d815aaf0 2022-06-27T11:55:31.685 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.639+0000 7fd8c67fc700 1 --2- 172.21.15.195:0/269397934 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd8b8063d30 0x7fd8b80661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:31.686 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.639+0000 7fd8c67fc700 1 -- 172.21.15.195:0/269397934 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fd8d4018070 con 0x7fd8d815aaf0 2022-06-27T11:55:31.686 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.639+0000 7fd8de58c700 1 --2- 172.21.15.195:0/269397934 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd8b8063d30 0x7fd8b80661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:31.687 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.639+0000 7fd8de58c700 1 --2- 172.21.15.195:0/269397934 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd8b8063d30 0x7fd8b80661e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd8c800ba40 tx=0x7fd8c801a000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:55:31.687 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.643+0000 7fd8c67fc700 1 -- 172.21.15.195:0/269397934 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fd8d8166670 con 0x7fd8d815aaf0 2022-06-27T11:55:31.868 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.868+0000 7fd8e07f0700 1 -- 172.21.15.195:0/269397934 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd8d8166670 con 0x7fd8d815aaf0 2022-06-27T11:55:31.868 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.868+0000 7fd8c67fc700 1 -- 172.21.15.195:0/269397934 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+5374 (secure 0 0 0) 0x7fd8d8166670 con 0x7fd8d815aaf0 2022-06-27T11:55:31.869 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:55:31.870 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:55:30.964053+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4856},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4856":{"gid":4856,"name":"d","rank":0,"incarnation":26,"state":"up:clientreplay","state_seq":41,"addr":"172.21.15.202:6835/2970948376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2970948376},{"type":"v1","addr":"172.21.15.202:6835","nonce":2970948376}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:55:31.872 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.870+0000 7fd8e07f0700 1 -- 172.21.15.195:0/269397934 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd8b8063d30 msgr2=0x7fd8b80661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:31.873 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.870+0000 7fd8e07f0700 1 --2- 172.21.15.195:0/269397934 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd8b8063d30 0x7fd8b80661e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd8c800ba40 tx=0x7fd8c801a000 comp rx=0 tx=0).stop 2022-06-27T11:55:31.873 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.870+0000 7fd8e07f0700 1 -- 172.21.15.195:0/269397934 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd8d815aaf0 msgr2=0x7fd8d80c59b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:31.874 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.870+0000 7fd8e07f0700 1 --2- 172.21.15.195:0/269397934 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd8d815aaf0 0x7fd8d80c59b0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fd8d40119f0 tx=0x7fd8d4009060 comp rx=0 tx=0).stop 2022-06-27T11:55:31.874 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.870+0000 7fd8e07f0700 1 -- 172.21.15.195:0/269397934 shutdown_connections 2022-06-27T11:55:31.875 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.870+0000 7fd8e07f0700 1 --2- 172.21.15.195:0/269397934 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd8b8063d30 0x7fd8b80661e0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:31.875 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.870+0000 7fd8e07f0700 1 --2- 172.21.15.195:0/269397934 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd8d8157c10 0x7fd8d8166950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:31.875 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.870+0000 7fd8e07f0700 1 --2- 172.21.15.195:0/269397934 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd8c0003920 0x7fd8d816a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:31.876 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.870+0000 7fd8e07f0700 1 --2- 172.21.15.195:0/269397934 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd8d815aaf0 0x7fd8d80c59b0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:31.876 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.870+0000 7fd8e07f0700 1 -- 172.21.15.195:0/269397934 >> 172.21.15.195:0/269397934 conn(0x7fd8d8063af0 msgr2=0x7fd8d814c090 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:55:31.877 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.870+0000 7fd8e07f0700 1 -- 172.21.15.195:0/269397934 shutdown_connections 2022-06-27T11:55:31.877 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:31.871+0000 7fd8e07f0700 1 -- 172.21.15.195:0/269397934 wait complete. 2022-06-27T11:55:31.886 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 29 2022-06-27T11:55:31.888 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T11:55:33.894 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:55:34.075 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.030+0000 7f8101fec700 1 Processor -- start 2022-06-27T11:55:34.075 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.030+0000 7f8101fec700 1 -- start start 2022-06-27T11:55:34.076 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.031+0000 7f8101fec700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f80fc0c2bf0 0x7f80fc0c2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:34.076 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.031+0000 7f8101fec700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80fc0bc490 con 0x7f80fc0c61e0 2022-06-27T11:55:34.077 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.031+0000 7f8101fec700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80fc06a0e0 con 0x7f80fc0c5000 2022-06-27T11:55:34.077 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.031+0000 7f8101fec700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80fc0bc6e0 con 0x7f80fc0c2bf0 2022-06-27T11:55:34.078 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.031+0000 7f80faffd700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f80fc0c61e0 0x7f80fc0c48f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:50706/0 (socket says 172.21.15.195:50706) 2022-06-27T11:55:34.078 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.031+0000 7f80faffd700 1 -- 172.21.15.195:0/766447920 learned_addr learned my addr 172.21.15.195:0/766447920 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:55:34.079 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.031+0000 7f80fb7fe700 1 --2- 172.21.15.195:0/766447920 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f80fc0c2bf0 0x7f80fc0c2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:34.080 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.031+0000 7f80fa7fc700 1 -- 172.21.15.195:0/766447920 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 49533687 0 0) 0x7f80fc0bc490 con 0x7f80fc0c61e0 2022-06-27T11:55:34.082 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.031+0000 7f80fa7fc700 1 -- 172.21.15.195:0/766447920 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80e8003430 con 0x7f80fc0c61e0 2022-06-27T11:55:34.082 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.031+0000 7f80fa7fc700 1 -- 172.21.15.195:0/766447920 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3922051216 0 0) 0x7f80fc06a0e0 con 0x7f80fc0c5000 2022-06-27T11:55:34.083 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.031+0000 7f80fa7fc700 1 -- 172.21.15.195:0/766447920 --> v1:172.21.15.202:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80fc0bc490 con 0x7f80fc0c5000 2022-06-27T11:55:34.083 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.031+0000 7f80fa7fc700 1 -- 172.21.15.195:0/766447920 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1350736507 0 0) 0x7f80e8003430 con 0x7f80fc0c61e0 2022-06-27T11:55:34.084 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.031+0000 7f80fa7fc700 1 -- 172.21.15.195:0/766447920 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f80fc0c2bf0 msgr2=0x7f80fc0c2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:55:34.084 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.031+0000 7f80fa7fc700 1 --2- 172.21.15.195:0/766447920 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f80fc0c2bf0 0x7f80fc0c2fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:34.085 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.031+0000 7f80fa7fc700 1 -- 172.21.15.195:0/766447920 >> v1:172.21.15.202:6789/0 conn(0x7f80fc0c5000 legacy=0x7f80fc0c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:34.085 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.031+0000 7f80fa7fc700 1 -- 172.21.15.195:0/766447920 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80fc158220 con 0x7f80fc0c61e0 2022-06-27T11:55:34.086 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.031+0000 7f80fa7fc700 1 -- 172.21.15.195:0/766447920 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7f80f0001470 con 0x7f80fc0c61e0 2022-06-27T11:55:34.086 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.031+0000 7f80fa7fc700 1 -- 172.21.15.195:0/766447920 >> v1:172.21.15.195:6789/0 conn(0x7f80fc0c61e0 legacy=0x7f80fc0c48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:34.086 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.031+0000 7f80fa7fc700 1 --2- 172.21.15.195:0/766447920 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f80e8003ae0 0x7f80e8003ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:34.087 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.031+0000 7f80fa7fc700 1 --2- 172.21.15.195:0/766447920 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f80e80046d0 0x7f80e8004b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:34.087 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.032+0000 7f80fa7fc700 1 --2- 172.21.15.195:0/766447920 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f80fc0c2bf0 0x7f80e80051a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:34.088 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.032+0000 7f80fa7fc700 1 -- 172.21.15.195:0/766447920 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80fc0bc6e0 con 0x7f80e80046d0 2022-06-27T11:55:34.089 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.032+0000 7f80fa7fc700 1 -- 172.21.15.195:0/766447920 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80e800bb00 con 0x7f80e8003ae0 2022-06-27T11:55:34.090 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.032+0000 7f80fa7fc700 1 -- 172.21.15.195:0/766447920 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80e800bcc0 con 0x7f80fc0c2bf0 2022-06-27T11:55:34.093 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.032+0000 7f80faffd700 1 --2- 172.21.15.195:0/766447920 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f80e80046d0 0x7f80e8004b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:34.093 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.032+0000 7f80fb7fe700 1 --2- 172.21.15.195:0/766447920 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f80e8003ae0 0x7f80e8003ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:34.094 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.032+0000 7f80fbfff700 1 --2- 172.21.15.195:0/766447920 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f80fc0c2bf0 0x7f80e80051a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:34.094 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.032+0000 7f80faffd700 1 -- 172.21.15.195:0/766447920 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f80fc0c2bf0 msgr2=0x7f80e80051a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:55:34.095 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.032+0000 7f80faffd700 1 --2- 172.21.15.195:0/766447920 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f80fc0c2bf0 0x7f80e80051a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:34.095 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.032+0000 7f80faffd700 1 -- 172.21.15.195:0/766447920 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f80e8003ae0 msgr2=0x7f80e8003ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:55:34.096 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.032+0000 7f80faffd700 1 --2- 172.21.15.195:0/766447920 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f80e8003ae0 0x7f80e8003ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:34.096 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.032+0000 7f80faffd700 1 -- 172.21.15.195:0/766447920 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80e8003430 con 0x7f80e80046d0 2022-06-27T11:55:34.097 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.032+0000 7f80fbfff700 1 --2- 172.21.15.195:0/766447920 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f80fc0c2bf0 0x7f80e80051a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T11:55:34.097 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.032+0000 7f80faffd700 1 --2- 172.21.15.195:0/766447920 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f80e80046d0 0x7f80e8004b40 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f80f0007f90 tx=0x7f80fc0c6610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddc92ef08f3058a7 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:55:34.098 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.032+0000 7f80fa7fc700 1 -- 172.21.15.195:0/766447920 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f000c9a0 con 0x7f80e80046d0 2022-06-27T11:55:34.098 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.032+0000 7f80fa7fc700 1 -- 172.21.15.195:0/766447920 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80f000cb00 con 0x7f80e80046d0 2022-06-27T11:55:34.099 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.032+0000 7f80fa7fc700 1 -- 172.21.15.195:0/766447920 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f00070d0 con 0x7f80e80046d0 2022-06-27T11:55:34.099 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.033+0000 7f8101fec700 1 -- 172.21.15.195:0/766447920 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f80e80046d0 msgr2=0x7f80e8004b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:34.100 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.033+0000 7f8101fec700 1 --2- 172.21.15.195:0/766447920 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f80e80046d0 0x7f80e8004b40 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f80f0007f90 tx=0x7f80fc0c6610 comp rx=0 tx=0).stop 2022-06-27T11:55:34.100 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.033+0000 7f80fbfff700 1 -- 172.21.15.195:0/766447920 reap_dead start 2022-06-27T11:55:34.101 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.033+0000 7f8101fec700 1 -- 172.21.15.195:0/766447920 shutdown_connections 2022-06-27T11:55:34.101 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.033+0000 7f8101fec700 1 -- 172.21.15.195:0/766447920 >> 172.21.15.195:0/766447920 conn(0x7f80fc0c9b00 msgr2=0x7f80fc0c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:55:34.102 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.033+0000 7f8101fec700 1 -- 172.21.15.195:0/766447920 shutdown_connections 2022-06-27T11:55:34.102 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.033+0000 7f8101fec700 1 -- 172.21.15.195:0/766447920 wait complete. 2022-06-27T11:55:34.103 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.034+0000 7f8101fec700 1 Processor -- start 2022-06-27T11:55:34.103 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.034+0000 7f8101fec700 1 -- start start 2022-06-27T11:55:34.104 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.035+0000 7f8101fec700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f80e80046d0 0x7f80fc15a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:34.104 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.035+0000 7f8101fec700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f80fc14a570 0x7f80fc152260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:34.104 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.035+0000 7f8101fec700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f80fc14b0d0 0x7f80fc14b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:34.105 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.035+0000 7f8101fec700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80fc1533d0 con 0x7f80fc14b0d0 2022-06-27T11:55:34.107 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.035+0000 7f8101fec700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80fc15a630 con 0x7f80fc14a570 2022-06-27T11:55:34.108 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.035+0000 7f8101fec700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80fc14ba80 con 0x7f80e80046d0 2022-06-27T11:55:34.108 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.035+0000 7f80fbfff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f80fc14b0d0 0x7f80fc14b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:34.109 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.036+0000 7f80fbfff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f80fc14b0d0 0x7f80fc14b540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:44532/0 (socket says 172.21.15.195:44532) 2022-06-27T11:55:34.110 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.036+0000 7f80fbfff700 1 -- 172.21.15.195:0/211019627 learned_addr learned my addr 172.21.15.195:0/211019627 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:55:34.111 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.036+0000 7f80fb7fe700 1 --2- 172.21.15.195:0/211019627 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f80e80046d0 0x7f80fc15a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:34.111 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.036+0000 7f80faffd700 1 --2- 172.21.15.195:0/211019627 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f80fc14a570 0x7f80fc152260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:34.120 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.036+0000 7f80fbfff700 1 -- 172.21.15.195:0/211019627 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f80e80046d0 msgr2=0x7f80fc15a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:34.121 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.036+0000 7f80fbfff700 1 --2- 172.21.15.195:0/211019627 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f80e80046d0 0x7f80fc15a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:34.121 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.036+0000 7f80fbfff700 1 -- 172.21.15.195:0/211019627 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f80fc14a570 msgr2=0x7f80fc152260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:34.122 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.036+0000 7f80fbfff700 1 --2- 172.21.15.195:0/211019627 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f80fc14a570 0x7f80fc152260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:34.131 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.036+0000 7f80fbfff700 1 -- 172.21.15.195:0/211019627 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80fc164080 con 0x7f80fc14b0d0 2022-06-27T11:55:34.132 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.036+0000 7f80fb7fe700 1 --2- 172.21.15.195:0/211019627 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f80e80046d0 0x7f80fc15a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T11:55:34.133 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.036+0000 7f80faffd700 1 --2- 172.21.15.195:0/211019627 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f80fc14a570 0x7f80fc152260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T11:55:34.133 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.036+0000 7f80fbfff700 1 --2- 172.21.15.195:0/211019627 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f80fc14b0d0 0x7f80fc14b540 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f80e8004650 tx=0x7f80f4009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:55:34.134 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.036+0000 7f80e37fe700 1 -- 172.21.15.195:0/211019627 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f4003a90 con 0x7f80fc14b0d0 2022-06-27T11:55:34.134 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.036+0000 7f80e37fe700 1 -- 172.21.15.195:0/211019627 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80f4003c10 con 0x7f80fc14b0d0 2022-06-27T11:55:34.134 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.036+0000 7f8101fec700 1 -- 172.21.15.195:0/211019627 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80fc164320 con 0x7f80fc14b0d0 2022-06-27T11:55:34.135 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.036+0000 7f80e37fe700 1 -- 172.21.15.195:0/211019627 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f4006a60 con 0x7f80fc14b0d0 2022-06-27T11:55:34.135 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.036+0000 7f8101fec700 1 -- 172.21.15.195:0/211019627 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80fc1648e0 con 0x7f80fc14b0d0 2022-06-27T11:55:34.136 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.037+0000 7f8101fec700 1 -- 172.21.15.195:0/211019627 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80fc0722c0 con 0x7f80fc14b0d0 2022-06-27T11:55:34.136 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.038+0000 7f80e37fe700 1 -- 172.21.15.195:0/211019627 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f80f401b030 con 0x7f80fc14b0d0 2022-06-27T11:55:34.136 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.038+0000 7f80e37fe700 1 --2- 172.21.15.195:0/211019627 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f80dc063d30 0x7f80dc0661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:34.137 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.038+0000 7f80e37fe700 1 -- 172.21.15.195:0/211019627 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f80f407fdb0 con 0x7f80fc14b0d0 2022-06-27T11:55:34.137 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.038+0000 7f80fb7fe700 1 --2- 172.21.15.195:0/211019627 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f80dc063d30 0x7f80dc0661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:34.138 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.038+0000 7f80fb7fe700 1 --2- 172.21.15.195:0/211019627 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f80dc063d30 0x7f80dc0661e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f80ec006c20 tx=0x7f80ec008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:55:34.139 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.042+0000 7f80e37fe700 1 -- 172.21.15.195:0/211019627 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f80f404e810 con 0x7f80fc14b0d0 2022-06-27T11:55:34.274 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.266+0000 7f8101fec700 1 -- 172.21.15.195:0/211019627 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f80fc0c6650 con 0x7f80fc14b0d0 2022-06-27T11:55:34.275 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.266+0000 7f80e37fe700 1 -- 172.21.15.195:0/211019627 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+5374 (secure 0 0 0) 0x7f80f4019030 con 0x7f80fc14b0d0 2022-06-27T11:55:34.276 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:55:34.276 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:55:30.964053+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4856},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4856":{"gid":4856,"name":"d","rank":0,"incarnation":26,"state":"up:clientreplay","state_seq":41,"addr":"172.21.15.202:6835/2970948376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2970948376},{"type":"v1","addr":"172.21.15.202:6835","nonce":2970948376}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:55:34.289 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.268+0000 7f8101fec700 1 -- 172.21.15.195:0/211019627 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f80dc063d30 msgr2=0x7f80dc0661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:34.290 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.268+0000 7f8101fec700 1 --2- 172.21.15.195:0/211019627 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f80dc063d30 0x7f80dc0661e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f80ec006c20 tx=0x7f80ec008040 comp rx=0 tx=0).stop 2022-06-27T11:55:34.290 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.268+0000 7f8101fec700 1 -- 172.21.15.195:0/211019627 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f80fc14b0d0 msgr2=0x7f80fc14b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:34.291 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.268+0000 7f8101fec700 1 --2- 172.21.15.195:0/211019627 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f80fc14b0d0 0x7f80fc14b540 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f80e8004650 tx=0x7f80f4009040 comp rx=0 tx=0).stop 2022-06-27T11:55:34.291 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.269+0000 7f8101fec700 1 -- 172.21.15.195:0/211019627 shutdown_connections 2022-06-27T11:55:34.291 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.269+0000 7f8101fec700 1 --2- 172.21.15.195:0/211019627 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f80dc063d30 0x7f80dc0661e0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:34.292 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.269+0000 7f8101fec700 1 --2- 172.21.15.195:0/211019627 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f80fc14b0d0 0x7f80fc14b540 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:34.293 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.269+0000 7f8101fec700 1 --2- 172.21.15.195:0/211019627 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f80e80046d0 0x7f80fc15a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:34.306 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.269+0000 7f8101fec700 1 --2- 172.21.15.195:0/211019627 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f80fc14a570 0x7f80fc152260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:34.306 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.269+0000 7f8101fec700 1 -- 172.21.15.195:0/211019627 >> 172.21.15.195:0/211019627 conn(0x7f80fc0c9b00 msgr2=0x7f80fc0c48f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:55:34.334 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.269+0000 7f8101fec700 1 -- 172.21.15.195:0/211019627 shutdown_connections 2022-06-27T11:55:34.334 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:34.269+0000 7f8101fec700 1 -- 172.21.15.195:0/211019627 wait complete. 2022-06-27T11:55:34.335 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 29 2022-06-27T11:55:34.336 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T11:55:36.338 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:55:36.515 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.470+0000 7f19c4f27700 1 Processor -- start 2022-06-27T11:55:36.516 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.470+0000 7f19c4f27700 1 -- start start 2022-06-27T11:55:36.516 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.471+0000 7f19c4f27700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f19c006f640 0x7f19c0156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:36.517 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.471+0000 7f19c4f27700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f19c00c4160 0x7f19c00c1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:36.517 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.471+0000 7f19c4f27700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f19c00c39e0 0x7f19c01535a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:36.518 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.471+0000 7f19c4f27700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19c00c1a20 con 0x7f19c00c4160 2022-06-27T11:55:36.519 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.471+0000 7f19c4f27700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19c00c1130 con 0x7f19c006f640 2022-06-27T11:55:36.519 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.471+0000 7f19c4f27700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19c006ea90 con 0x7f19c00c39e0 2022-06-27T11:55:36.530 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.471+0000 7f19bdd9b700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f19c00c4160 0x7f19c00c1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:36.530 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.471+0000 7f19bdd9b700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f19c00c4160 0x7f19c00c1eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:44548/0 (socket says 172.21.15.195:44548) 2022-06-27T11:55:36.531 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.471+0000 7f19bdd9b700 1 -- 172.21.15.195:0/855515162 learned_addr learned my addr 172.21.15.195:0/855515162 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:55:36.532 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.471+0000 7f19bed9d700 1 --2- 172.21.15.195:0/855515162 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f19c00c39e0 0x7f19c01535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:36.532 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.471+0000 7f19be59c700 1 --2- 172.21.15.195:0/855515162 >> v2:172.21.15.202:3300/0 conn(0x7f19c006f640 0x7f19c0156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:36.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.472+0000 7f19bdd9b700 1 -- 172.21.15.195:0/855515162 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f19c00c39e0 msgr2=0x7f19c01535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:55:36.534 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.472+0000 7f19bdd9b700 1 --2- 172.21.15.195:0/855515162 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f19c00c39e0 0x7f19c01535a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:36.534 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.472+0000 7f19bdd9b700 1 -- 172.21.15.195:0/855515162 >> v2:172.21.15.202:3300/0 conn(0x7f19c006f640 msgr2=0x7f19c0156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:55:36.535 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.472+0000 7f19bdd9b700 1 --2- 172.21.15.195:0/855515162 >> v2:172.21.15.202:3300/0 conn(0x7f19c006f640 0x7f19c0156ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:36.535 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.472+0000 7f19bdd9b700 1 -- 172.21.15.195:0/855515162 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19c006ed70 con 0x7f19c00c4160 2022-06-27T11:55:36.536 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.472+0000 7f19bdd9b700 1 --2- 172.21.15.195:0/855515162 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f19c00c4160 0x7f19c00c1eb0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f19c006db30 tx=0x7f19b0009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd72b0a7b3907c34 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:55:36.536 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.472+0000 7f19bd59a700 1 -- 172.21.15.195:0/855515162 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19b0011630 con 0x7f19c00c4160 2022-06-27T11:55:36.537 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.472+0000 7f19bd59a700 1 -- 172.21.15.195:0/855515162 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19b0011790 con 0x7f19c00c4160 2022-06-27T11:55:36.537 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.472+0000 7f19bd59a700 1 -- 172.21.15.195:0/855515162 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19b0011b30 con 0x7f19c00c4160 2022-06-27T11:55:36.537 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.472+0000 7f19c4f27700 1 -- 172.21.15.195:0/855515162 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f19c00c4160 msgr2=0x7f19c00c1eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:36.538 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.472+0000 7f19c4f27700 1 --2- 172.21.15.195:0/855515162 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f19c00c4160 0x7f19c00c1eb0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f19c006db30 tx=0x7f19b0009120 comp rx=0 tx=0).stop 2022-06-27T11:55:36.538 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.473+0000 7f19c4f27700 1 -- 172.21.15.195:0/855515162 shutdown_connections 2022-06-27T11:55:36.539 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.473+0000 7f19c4f27700 1 --2- 172.21.15.195:0/855515162 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f19c00c39e0 0x7f19c01535a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:36.539 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.473+0000 7f19c4f27700 1 --2- 172.21.15.195:0/855515162 >> v2:172.21.15.202:3300/0 conn(0x7f19c006f640 0x7f19c0156ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:36.540 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.473+0000 7f19c4f27700 1 --2- 172.21.15.195:0/855515162 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f19c00c4160 0x7f19c00c1eb0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:36.540 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.473+0000 7f19c4f27700 1 -- 172.21.15.195:0/855515162 >> 172.21.15.195:0/855515162 conn(0x7f19c00c6350 msgr2=0x7f19c00c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:55:36.541 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.473+0000 7f19c4f27700 1 -- 172.21.15.195:0/855515162 shutdown_connections 2022-06-27T11:55:36.541 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.473+0000 7f19c4f27700 1 -- 172.21.15.195:0/855515162 wait complete. 2022-06-27T11:55:36.542 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.474+0000 7f19c4f27700 1 Processor -- start 2022-06-27T11:55:36.542 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.474+0000 7f19c4f27700 1 -- start start 2022-06-27T11:55:36.543 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.475+0000 7f19c4f27700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f19c006f640 0x7f19c01671e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:36.543 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.475+0000 7f19c4f27700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f19c00c39e0 0x7f19c0153550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:36.544 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.475+0000 7f19c4f27700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f19c00c4160 0x7f19c00bb4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:36.544 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.475+0000 7f19c4f27700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19c0152f20 con 0x7f19c00c4160 2022-06-27T11:55:36.545 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.475+0000 7f19c4f27700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19c0152030 con 0x7f19c00c39e0 2022-06-27T11:55:36.545 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.475+0000 7f19c4f27700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19c0152310 con 0x7f19c006f640 2022-06-27T11:55:36.561 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.475+0000 7f19bed9d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f19c00c4160 0x7f19c00bb4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:36.561 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.475+0000 7f19bed9d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f19c00c4160 0x7f19c00bb4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:44558/0 (socket says 172.21.15.195:44558) 2022-06-27T11:55:36.562 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.475+0000 7f19be59c700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f19c006f640 0x7f19c01671e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:36.562 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.475+0000 7f19bed9d700 1 -- 172.21.15.195:0/149664207 learned_addr learned my addr 172.21.15.195:0/149664207 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:55:36.564 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.475+0000 7f19bdd9b700 1 --2- 172.21.15.195:0/149664207 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f19c00c39e0 0x7f19c0153550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:36.564 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.475+0000 7f19bed9d700 1 -- 172.21.15.195:0/149664207 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f19c006f640 msgr2=0x7f19c01671e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:36.565 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.475+0000 7f19bed9d700 1 --2- 172.21.15.195:0/149664207 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f19c006f640 0x7f19c01671e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:36.565 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.475+0000 7f19bed9d700 1 -- 172.21.15.195:0/149664207 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f19c00c39e0 msgr2=0x7f19c0153550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:36.566 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.475+0000 7f19bed9d700 1 --2- 172.21.15.195:0/149664207 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f19c00c39e0 0x7f19c0153550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:36.566 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.475+0000 7f19bed9d700 1 -- 172.21.15.195:0/149664207 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19b00093a0 con 0x7f19c00c4160 2022-06-27T11:55:36.567 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.475+0000 7f19bed9d700 1 --2- 172.21.15.195:0/149664207 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f19c00c4160 0x7f19c00bb4b0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f19b8009e90 tx=0x7f19b800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:55:36.567 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.475+0000 7f19ae7fc700 1 -- 172.21.15.195:0/149664207 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19b800cb70 con 0x7f19c00c4160 2022-06-27T11:55:36.572 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.475+0000 7f19ae7fc700 1 -- 172.21.15.195:0/149664207 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19b800f040 con 0x7f19c00c4160 2022-06-27T11:55:36.573 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.475+0000 7f19c4f27700 1 -- 172.21.15.195:0/149664207 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19c01525f0 con 0x7f19c00c4160 2022-06-27T11:55:36.579 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.475+0000 7f19ae7fc700 1 -- 172.21.15.195:0/149664207 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19b8014480 con 0x7f19c00c4160 2022-06-27T11:55:36.580 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.475+0000 7f19c4f27700 1 -- 172.21.15.195:0/149664207 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19c0152bb0 con 0x7f19c00c4160 2022-06-27T11:55:36.580 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.476+0000 7f19c4f27700 1 -- 172.21.15.195:0/149664207 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19c00bb320 con 0x7f19c00c4160 2022-06-27T11:55:36.581 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.477+0000 7f19ae7fc700 1 -- 172.21.15.195:0/149664207 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f19b80147f0 con 0x7f19c00c4160 2022-06-27T11:55:36.581 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.477+0000 7f19ae7fc700 1 --2- 172.21.15.195:0/149664207 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f199c063d00 0x7f199c0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:36.581 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.477+0000 7f19ae7fc700 1 -- 172.21.15.195:0/149664207 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f19b80819a0 con 0x7f19c00c4160 2022-06-27T11:55:36.582 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.477+0000 7f19be59c700 1 --2- 172.21.15.195:0/149664207 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f199c063d00 0x7f199c0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:36.583 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.477+0000 7f19be59c700 1 --2- 172.21.15.195:0/149664207 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f199c063d00 0x7f199c0661b0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f19b4009be0 tx=0x7f19b4009060 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:55:36.583 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.482+0000 7f19ae7fc700 1 -- 172.21.15.195:0/149664207 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f19b8050400 con 0x7f19c00c4160 2022-06-27T11:55:36.743 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.734+0000 7f19c4f27700 1 -- 172.21.15.195:0/149664207 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f19c00722c0 con 0x7f19c00c4160 2022-06-27T11:55:36.743 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.734+0000 7f19ae7fc700 1 -- 172.21.15.195:0/149664207 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+5374 (secure 0 0 0) 0x7f19b8018020 con 0x7f19c00c4160 2022-06-27T11:55:36.744 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:55:36.745 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:55:30.964053+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4856},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4856":{"gid":4856,"name":"d","rank":0,"incarnation":26,"state":"up:clientreplay","state_seq":41,"addr":"172.21.15.202:6835/2970948376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2970948376},{"type":"v1","addr":"172.21.15.202:6835","nonce":2970948376}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:55:36.749 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.737+0000 7f19c4f27700 1 -- 172.21.15.195:0/149664207 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f199c063d00 msgr2=0x7f199c0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:36.749 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.737+0000 7f19c4f27700 1 --2- 172.21.15.195:0/149664207 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f199c063d00 0x7f199c0661b0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f19b4009be0 tx=0x7f19b4009060 comp rx=0 tx=0).stop 2022-06-27T11:55:36.749 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.737+0000 7f19c4f27700 1 -- 172.21.15.195:0/149664207 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f19c00c4160 msgr2=0x7f19c00bb4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:36.750 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.737+0000 7f19c4f27700 1 --2- 172.21.15.195:0/149664207 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f19c00c4160 0x7f19c00bb4b0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f19b8009e90 tx=0x7f19b800b040 comp rx=0 tx=0).stop 2022-06-27T11:55:36.750 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.737+0000 7f19c4f27700 1 -- 172.21.15.195:0/149664207 shutdown_connections 2022-06-27T11:55:36.750 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.737+0000 7f19c4f27700 1 --2- 172.21.15.195:0/149664207 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f199c063d00 0x7f199c0661b0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:36.751 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.737+0000 7f19c4f27700 1 --2- 172.21.15.195:0/149664207 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f19c00c4160 0x7f19c00bb4b0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:36.751 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.737+0000 7f19c4f27700 1 --2- 172.21.15.195:0/149664207 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f19c006f640 0x7f19c01671e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:36.751 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.737+0000 7f19c4f27700 1 --2- 172.21.15.195:0/149664207 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f19c00c39e0 0x7f19c0153550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:36.751 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.737+0000 7f19c4f27700 1 -- 172.21.15.195:0/149664207 >> 172.21.15.195:0/149664207 conn(0x7f19c00c6350 msgr2=0x7f19c01506e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:55:36.752 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.737+0000 7f19c4f27700 1 -- 172.21.15.195:0/149664207 shutdown_connections 2022-06-27T11:55:36.752 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:36.737+0000 7f19c4f27700 1 -- 172.21.15.195:0/149664207 wait complete. 2022-06-27T11:55:36.752 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 29 2022-06-27T11:55:36.754 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 29, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-06-27T11:47:52.358095+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 29, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4856": { "addr": "172.21.15.202:6835/2970948376", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6834", "nonce": 2970948376, "type": "v2" }, { "addr": "172.21.15.202:6835", "nonce": 2970948376, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4856, "incarnation": 26, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:clientreplay", "state_seq": 41 } }, "last_failure": 0, "last_failure_osd_epoch": 51, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-06-27T11:55:30.964053+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4856 } } } ], "standbys": [ { "addr": "172.21.15.202:6837/3305006841", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6836", "nonce": 3305006841, "type": "v2" }, { "addr": "172.21.15.202:6837", "nonce": 3305006841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6835/2060709822", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6834", "nonce": 2060709822, "type": "v2" }, { "addr": "172.21.15.195:6835", "nonce": 2060709822, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6838/1598751652", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6836", "nonce": 1598751652, "type": "v2" }, { "addr": "172.21.15.195:6838", "nonce": 1598751652, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4275, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6839/1313800661", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6837", "nonce": 1313800661, "type": "v2" }, { "addr": "172.21.15.195:6839", "nonce": 1313800661, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4278, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-06-27T11:55:38.756 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:55:38.917 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.890+0000 7fbc7fa8a700 1 Processor -- start 2022-06-27T11:55:38.917 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.890+0000 7fbc7fa8a700 1 -- start start 2022-06-27T11:55:38.918 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.891+0000 7fbc7fa8a700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fbc7806e850 0x7fbc781570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:38.918 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.891+0000 7fbc7fa8a700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbc780c34a0 0x7fbc780c2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:38.919 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.891+0000 7fbc7fa8a700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc7806fba0 con 0x7fbc7806fef0 2022-06-27T11:55:38.919 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.891+0000 7fbc7fa8a700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc781575e0 con 0x7fbc7806e850 2022-06-27T11:55:38.920 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.891+0000 7fbc7fa8a700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc7806f5e0 con 0x7fbc780c34a0 2022-06-27T11:55:38.920 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.891+0000 7fbc7d025700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fbc7806fef0 0x7fbc78069de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:50752/0 (socket says 172.21.15.195:50752) 2022-06-27T11:55:38.921 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.891+0000 7fbc7d025700 1 -- 172.21.15.195:0/3071295915 learned_addr learned my addr 172.21.15.195:0/3071295915 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:55:38.921 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.892+0000 7fbc7e027700 1 --2- 172.21.15.195:0/3071295915 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbc780c34a0 0x7fbc780c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:38.922 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.892+0000 7fbc7d826700 1 --2- 172.21.15.195:0/3071295915 >> v2:172.21.15.202:3300/0 conn(0x7fbc7806e850 0x7fbc781570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:38.922 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.892+0000 7fbc7c824700 1 -- 172.21.15.195:0/3071295915 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 368391203 0 0) 0x7fbc7806fba0 con 0x7fbc7806fef0 2022-06-27T11:55:38.923 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.892+0000 7fbc7c824700 1 -- 172.21.15.195:0/3071295915 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc60003430 con 0x7fbc7806fef0 2022-06-27T11:55:38.923 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.892+0000 7fbc7c824700 1 -- 172.21.15.195:0/3071295915 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1284829317 0 0) 0x7fbc60003430 con 0x7fbc7806fef0 2022-06-27T11:55:38.924 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.892+0000 7fbc7c824700 1 -- 172.21.15.195:0/3071295915 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbc780c34a0 msgr2=0x7fbc780c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:55:38.924 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.892+0000 7fbc7c824700 1 --2- 172.21.15.195:0/3071295915 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbc780c34a0 0x7fbc780c2920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:38.925 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.892+0000 7fbc7c824700 1 -- 172.21.15.195:0/3071295915 >> v2:172.21.15.202:3300/0 conn(0x7fbc7806e850 msgr2=0x7fbc781570a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:55:38.926 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.892+0000 7fbc7c824700 1 --2- 172.21.15.195:0/3071295915 >> v2:172.21.15.202:3300/0 conn(0x7fbc7806e850 0x7fbc781570a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:38.927 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.892+0000 7fbc7c824700 1 -- 172.21.15.195:0/3071295915 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc7806f8c0 con 0x7fbc7806fef0 2022-06-27T11:55:38.927 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.892+0000 7fbc7c824700 1 -- 172.21.15.195:0/3071295915 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7fbc74001470 con 0x7fbc7806fef0 2022-06-27T11:55:38.928 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.892+0000 7fbc7c824700 1 -- 172.21.15.195:0/3071295915 >> v1:172.21.15.195:6789/0 conn(0x7fbc7806fef0 legacy=0x7fbc78069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:38.928 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.892+0000 7fbc7c824700 1 --2- 172.21.15.195:0/3071295915 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbc60003920 0x7fbc60003d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:38.929 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.892+0000 7fbc7c824700 1 --2- 172.21.15.195:0/3071295915 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbc780c34a0 0x7fbc600046d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:38.929 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.892+0000 7fbc7c824700 1 --2- 172.21.15.195:0/3071295915 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbc60004c10 0x7fbc60005080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:38.930 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.892+0000 7fbc7c824700 1 -- 172.21.15.195:0/3071295915 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc781575e0 con 0x7fbc60003920 2022-06-27T11:55:38.931 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.892+0000 7fbc7c824700 1 -- 172.21.15.195:0/3071295915 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc7806f5e0 con 0x7fbc60004c10 2022-06-27T11:55:38.932 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.892+0000 7fbc7c824700 1 -- 172.21.15.195:0/3071295915 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc6000b920 con 0x7fbc780c34a0 2022-06-27T11:55:38.932 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.892+0000 7fbc7d826700 1 --2- 172.21.15.195:0/3071295915 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbc60003920 0x7fbc60003d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:38.933 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.892+0000 7fbc7e027700 1 --2- 172.21.15.195:0/3071295915 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbc60004c10 0x7fbc60005080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:38.933 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.892+0000 7fbc7d025700 1 --2- 172.21.15.195:0/3071295915 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbc780c34a0 0x7fbc600046d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:38.934 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.892+0000 7fbc7d826700 1 -- 172.21.15.195:0/3071295915 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbc780c34a0 msgr2=0x7fbc600046d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:55:38.934 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.892+0000 7fbc7d826700 1 --2- 172.21.15.195:0/3071295915 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbc780c34a0 0x7fbc600046d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:38.935 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.892+0000 7fbc7d826700 1 -- 172.21.15.195:0/3071295915 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbc60004c10 msgr2=0x7fbc60005080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:55:38.935 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.892+0000 7fbc7d826700 1 --2- 172.21.15.195:0/3071295915 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbc60004c10 0x7fbc60005080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:38.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.892+0000 7fbc7d826700 1 -- 172.21.15.195:0/3071295915 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc60003430 con 0x7fbc60003920 2022-06-27T11:55:38.937 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.893+0000 7fbc7d826700 1 --2- 172.21.15.195:0/3071295915 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbc60003920 0x7fbc60003d30 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fbc68009ac0 tx=0x7fbc68009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79739ef545c5584a server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:55:38.937 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.893+0000 7fbc7c824700 1 -- 172.21.15.195:0/3071295915 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc68009470 con 0x7fbc60003920 2022-06-27T11:55:38.939 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.893+0000 7fbc7c824700 1 -- 172.21.15.195:0/3071295915 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc6800e5e0 con 0x7fbc60003920 2022-06-27T11:55:38.940 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.893+0000 7fbc7c824700 1 -- 172.21.15.195:0/3071295915 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc6800e980 con 0x7fbc60003920 2022-06-27T11:55:38.941 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.893+0000 7fbc7fa8a700 1 -- 172.21.15.195:0/3071295915 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbc60003920 msgr2=0x7fbc60003d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:38.941 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.893+0000 7fbc7fa8a700 1 --2- 172.21.15.195:0/3071295915 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbc60003920 0x7fbc60003d30 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fbc68009ac0 tx=0x7fbc68009060 comp rx=0 tx=0).stop 2022-06-27T11:55:38.942 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.893+0000 7fbc7e027700 1 -- 172.21.15.195:0/3071295915 reap_dead start 2022-06-27T11:55:38.942 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.893+0000 7fbc7fa8a700 1 -- 172.21.15.195:0/3071295915 shutdown_connections 2022-06-27T11:55:38.942 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.893+0000 7fbc7fa8a700 1 -- 172.21.15.195:0/3071295915 >> 172.21.15.195:0/3071295915 conn(0x7fbc780c6350 msgr2=0x7fbc780c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:55:38.943 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.893+0000 7fbc7fa8a700 1 -- 172.21.15.195:0/3071295915 shutdown_connections 2022-06-27T11:55:38.943 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.893+0000 7fbc7fa8a700 1 -- 172.21.15.195:0/3071295915 wait complete. 2022-06-27T11:55:38.944 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.894+0000 7fbc7fa8a700 1 Processor -- start 2022-06-27T11:55:38.944 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.895+0000 7fbc7fa8a700 1 -- start start 2022-06-27T11:55:38.945 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.895+0000 7fbc7fa8a700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbc60003920 0x7fbc780c82d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:38.945 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.895+0000 7fbc7fa8a700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbc7814fe40 0x7fbc781510a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:38.945 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.895+0000 7fbc7fa8a700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbc781515e0 0x7fbc78165a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:38.946 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.895+0000 7fbc7fa8a700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc780c9540 con 0x7fbc60003920 2022-06-27T11:55:38.947 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.895+0000 7fbc7fa8a700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc78149dd0 con 0x7fbc7814fe40 2022-06-27T11:55:38.947 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.895+0000 7fbc7fa8a700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc78149200 con 0x7fbc781515e0 2022-06-27T11:55:38.947 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.895+0000 7fbc7d826700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbc60003920 0x7fbc780c82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:38.948 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.895+0000 7fbc7d826700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbc60003920 0x7fbc780c82d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:44574/0 (socket says 172.21.15.195:44574) 2022-06-27T11:55:38.949 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.895+0000 7fbc7d826700 1 -- 172.21.15.195:0/1785396168 learned_addr learned my addr 172.21.15.195:0/1785396168 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:55:38.949 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.896+0000 7fbc7e027700 1 --2- 172.21.15.195:0/1785396168 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbc781515e0 0x7fbc78165a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:38.950 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.896+0000 7fbc7d025700 1 --2- 172.21.15.195:0/1785396168 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbc7814fe40 0x7fbc781510a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:38.951 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.896+0000 7fbc7d826700 1 -- 172.21.15.195:0/1785396168 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbc781515e0 msgr2=0x7fbc78165a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:38.953 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.896+0000 7fbc7d826700 1 --2- 172.21.15.195:0/1785396168 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbc781515e0 0x7fbc78165a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:38.953 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.896+0000 7fbc7d826700 1 -- 172.21.15.195:0/1785396168 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbc7814fe40 msgr2=0x7fbc781510a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:38.953 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.896+0000 7fbc7d826700 1 --2- 172.21.15.195:0/1785396168 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbc7814fe40 0x7fbc781510a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:38.953 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.896+0000 7fbc7d826700 1 -- 172.21.15.195:0/1785396168 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc78151a10 con 0x7fbc60003920 2022-06-27T11:55:38.954 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.896+0000 7fbc7d025700 1 --2- 172.21.15.195:0/1785396168 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbc7814fe40 0x7fbc781510a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T11:55:38.954 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.896+0000 7fbc7d826700 1 --2- 172.21.15.195:0/1785396168 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbc60003920 0x7fbc780c82d0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fbc68004590 tx=0x7fbc6801a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:55:38.954 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.896+0000 7fbc65ffb700 1 -- 172.21.15.195:0/1785396168 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc68021070 con 0x7fbc60003920 2022-06-27T11:55:38.954 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.896+0000 7fbc65ffb700 1 -- 172.21.15.195:0/1785396168 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc6800e820 con 0x7fbc60003920 2022-06-27T11:55:38.955 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.896+0000 7fbc7fa8a700 1 -- 172.21.15.195:0/1785396168 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc781494a0 con 0x7fbc60003920 2022-06-27T11:55:38.955 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.896+0000 7fbc65ffb700 1 -- 172.21.15.195:0/1785396168 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc68003770 con 0x7fbc60003920 2022-06-27T11:55:38.955 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.896+0000 7fbc7fa8a700 1 -- 172.21.15.195:0/1785396168 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc78149a60 con 0x7fbc60003920 2022-06-27T11:55:38.956 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.897+0000 7fbc7fa8a700 1 -- 172.21.15.195:0/1785396168 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc780722c0 con 0x7fbc60003920 2022-06-27T11:55:38.958 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.898+0000 7fbc65ffb700 1 -- 172.21.15.195:0/1785396168 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fbc6802e030 con 0x7fbc60003920 2022-06-27T11:55:38.959 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.898+0000 7fbc65ffb700 1 --2- 172.21.15.195:0/1785396168 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbc58063d30 0x7fbc580661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:38.959 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.898+0000 7fbc65ffb700 1 -- 172.21.15.195:0/1785396168 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fbc68082170 con 0x7fbc60003920 2022-06-27T11:55:38.960 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.898+0000 7fbc7d025700 1 --2- 172.21.15.195:0/1785396168 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbc58063d30 0x7fbc580661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:38.960 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.898+0000 7fbc7d025700 1 --2- 172.21.15.195:0/1785396168 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbc58063d30 0x7fbc580661e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fbc74001d00 tx=0x7fbc74009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:55:38.960 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:38.902+0000 7fbc65ffb700 1 -- 172.21.15.195:0/1785396168 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fbc68050bd0 con 0x7fbc60003920 2022-06-27T11:55:39.133 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:39.131+0000 7fbc7fa8a700 1 -- 172.21.15.195:0/1785396168 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbc7806f240 con 0x7fbc60003920 2022-06-27T11:55:39.133 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:39.131+0000 7fbc65ffb700 1 -- 172.21.15.195:0/1785396168 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5368 (secure 0 0 0) 0x7fbc68017090 con 0x7fbc60003920 2022-06-27T11:55:39.134 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:55:39.135 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:55:37.983652+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4856},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4856":{"gid":4856,"name":"d","rank":0,"incarnation":26,"state":"up:active","state_seq":43,"addr":"172.21.15.202:6835/2970948376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2970948376},{"type":"v1","addr":"172.21.15.202:6835","nonce":2970948376}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:55:39.137 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:39.133+0000 7fbc7fa8a700 1 -- 172.21.15.195:0/1785396168 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbc58063d30 msgr2=0x7fbc580661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:39.137 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:39.133+0000 7fbc7fa8a700 1 --2- 172.21.15.195:0/1785396168 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbc58063d30 0x7fbc580661e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fbc74001d00 tx=0x7fbc74009040 comp rx=0 tx=0).stop 2022-06-27T11:55:39.138 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:39.133+0000 7fbc7fa8a700 1 -- 172.21.15.195:0/1785396168 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbc60003920 msgr2=0x7fbc780c82d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:39.138 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:39.133+0000 7fbc7fa8a700 1 --2- 172.21.15.195:0/1785396168 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbc60003920 0x7fbc780c82d0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fbc68004590 tx=0x7fbc6801a000 comp rx=0 tx=0).stop 2022-06-27T11:55:39.139 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:39.133+0000 7fbc7fa8a700 1 -- 172.21.15.195:0/1785396168 shutdown_connections 2022-06-27T11:55:39.139 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:39.133+0000 7fbc7fa8a700 1 --2- 172.21.15.195:0/1785396168 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbc58063d30 0x7fbc580661e0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:39.140 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:39.133+0000 7fbc7fa8a700 1 --2- 172.21.15.195:0/1785396168 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbc60003920 0x7fbc780c82d0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:39.140 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:39.133+0000 7fbc7fa8a700 1 --2- 172.21.15.195:0/1785396168 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbc781515e0 0x7fbc78165a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:39.141 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:39.133+0000 7fbc7fa8a700 1 --2- 172.21.15.195:0/1785396168 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbc7814fe40 0x7fbc781510a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:39.141 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:39.133+0000 7fbc7fa8a700 1 -- 172.21.15.195:0/1785396168 >> 172.21.15.195:0/1785396168 conn(0x7fbc780c6350 msgr2=0x7fbc781474c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:55:39.142 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:39.133+0000 7fbc7fa8a700 1 -- 172.21.15.195:0/1785396168 shutdown_connections 2022-06-27T11:55:39.142 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:39.133+0000 7fbc7fa8a700 1 -- 172.21.15.195:0/1785396168 wait complete. 2022-06-27T11:55:39.143 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 30 2022-06-27T11:55:39.145 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d has gained rank=0, replacing gid=4760 2022-06-27T11:55:39.146 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 7 secs before reviving mds.b 2022-06-27T11:55:44.064 INFO:tasks.ceph.mds.d.smithi202.stderr:2022-06-27T11:55:44.064+0000 7fcb91f79700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2022-06-27T11:55:46.150 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2022-06-27T11:55:46.151 INFO:tasks.ceph.mds.b:Restarting daemon 2022-06-27T11:55:46.152 DEBUG:teuthology.orchestra.run.smithi202:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-06-27T11:55:46.155 INFO:tasks.ceph.mds.b:Started 2022-06-27T11:55:46.156 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:55:46.277 INFO:tasks.ceph.mds.b.smithi202.stdout:starting mds.b at 2022-06-27T11:55:46.320 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.297+0000 7f2ffaae0700 1 Processor -- start 2022-06-27T11:55:46.321 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.297+0000 7f2ffaae0700 1 -- start start 2022-06-27T11:55:46.321 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.298+0000 7f2ffaae0700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2ff414e490 0x7f2ff406cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:46.322 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.298+0000 7f2ffaae0700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f2ff40c3f50 0x7f2ff4157890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:46.322 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.298+0000 7f2ffaae0700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ff40c33b0 con 0x7f2ff414e860 2022-06-27T11:55:46.322 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.298+0000 7f2ffaae0700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ff406d520 con 0x7f2ff40c3f50 2022-06-27T11:55:46.323 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.298+0000 7f2ffaae0700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff406f220 con 0x7f2ff414e490 2022-06-27T11:55:46.323 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.298+0000 7f2ff3fff700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f2ff414e860 0x7f2ff40c3840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:50780/0 (socket says 172.21.15.195:50780) 2022-06-27T11:55:46.325 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.298+0000 7f2ff3fff700 1 -- 172.21.15.195:0/3697575163 learned_addr learned my addr 172.21.15.195:0/3697575163 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:55:46.326 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.298+0000 7f2ff887c700 1 --2- 172.21.15.195:0/3697575163 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2ff414e490 0x7f2ff406cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:46.326 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.298+0000 7f2ff907d700 1 --2- 172.21.15.195:0/3697575163 >> v2:172.21.15.202:3300/0 conn(0x7f2ff40c3f50 0x7f2ff4157890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:46.327 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.299+0000 7f2ff37fe700 1 -- 172.21.15.195:0/3697575163 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2304945983 0 0) 0x7f2ff40c33b0 con 0x7f2ff414e860 2022-06-27T11:55:46.327 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.299+0000 7f2ff37fe700 1 -- 172.21.15.195:0/3697575163 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fd8003430 con 0x7f2ff414e860 2022-06-27T11:55:46.328 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.299+0000 7f2ff887c700 1 -- 172.21.15.195:0/3697575163 >> v2:172.21.15.202:3300/0 conn(0x7f2ff40c3f50 msgr2=0x7f2ff4157890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:55:46.328 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.299+0000 7f2ff887c700 1 --2- 172.21.15.195:0/3697575163 >> v2:172.21.15.202:3300/0 conn(0x7f2ff40c3f50 0x7f2ff4157890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:46.329 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.299+0000 7f2ff887c700 1 -- 172.21.15.195:0/3697575163 >> v1:172.21.15.195:6789/0 conn(0x7f2ff414e860 legacy=0x7f2ff40c3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:46.329 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.299+0000 7f2ff887c700 1 -- 172.21.15.195:0/3697575163 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ff41542c0 con 0x7f2ff414e490 2022-06-27T11:55:46.330 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.299+0000 7f2ff887c700 1 --2- 172.21.15.195:0/3697575163 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2ff414e490 0x7f2ff406cfe0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2fe80097a0 tx=0x7f2fe8009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f2b0cecca10ce32 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:55:46.330 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.299+0000 7f2ff37fe700 1 -- 172.21.15.195:0/3697575163 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe8011630 con 0x7f2ff414e490 2022-06-27T11:55:46.331 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.299+0000 7f2ff37fe700 1 -- 172.21.15.195:0/3697575163 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fe8011790 con 0x7f2ff414e490 2022-06-27T11:55:46.331 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.299+0000 7f2ff37fe700 1 -- 172.21.15.195:0/3697575163 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe8011b30 con 0x7f2ff414e490 2022-06-27T11:55:46.332 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.299+0000 7f2ffaae0700 1 -- 172.21.15.195:0/3697575163 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2ff414e490 msgr2=0x7f2ff406cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:46.332 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.300+0000 7f2ffaae0700 1 --2- 172.21.15.195:0/3697575163 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2ff414e490 0x7f2ff406cfe0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2fe80097a0 tx=0x7f2fe8009060 comp rx=0 tx=0).stop 2022-06-27T11:55:46.333 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.300+0000 7f2ffaae0700 1 -- 172.21.15.195:0/3697575163 shutdown_connections 2022-06-27T11:55:46.334 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.300+0000 7f2ffaae0700 1 --2- 172.21.15.195:0/3697575163 >> v2:172.21.15.202:3300/0 conn(0x7f2ff40c3f50 0x7f2ff4157890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:46.334 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.300+0000 7f2ffaae0700 1 --2- 172.21.15.195:0/3697575163 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2ff414e490 0x7f2ff406cfe0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:46.335 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.300+0000 7f2ffaae0700 1 -- 172.21.15.195:0/3697575163 >> 172.21.15.195:0/3697575163 conn(0x7f2ff4063af0 msgr2=0x7f2ff4063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:55:46.335 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.300+0000 7f2ffaae0700 1 -- 172.21.15.195:0/3697575163 shutdown_connections 2022-06-27T11:55:46.336 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.300+0000 7f2ffaae0700 1 -- 172.21.15.195:0/3697575163 wait complete. 2022-06-27T11:55:46.336 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.301+0000 7f2ffaae0700 1 Processor -- start 2022-06-27T11:55:46.337 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.301+0000 7f2ffaae0700 1 -- start start 2022-06-27T11:55:46.337 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.302+0000 7f2ffaae0700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2ff40c3f50 0x7f2ff41649d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:46.338 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.302+0000 7f2ffaae0700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2ff414e490 0x7f2ff40c7650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:46.338 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.302+0000 7f2ffaae0700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2ff414e860 0x7f2ff40c0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:46.339 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.302+0000 7f2ffaae0700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff4152080 con 0x7f2ff414e860 2022-06-27T11:55:46.340 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.302+0000 7f2ffaae0700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff4164540 con 0x7f2ff40c3f50 2022-06-27T11:55:46.340 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.302+0000 7f2ffaae0700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff41522d0 con 0x7f2ff414e490 2022-06-27T11:55:46.341 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.302+0000 7f2ff907d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2ff414e860 0x7f2ff40c0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:46.341 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.302+0000 7f2ff907d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2ff414e860 0x7f2ff40c0280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:44600/0 (socket says 172.21.15.195:44600) 2022-06-27T11:55:46.341 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.302+0000 7f2ff907d700 1 -- 172.21.15.195:0/2292028549 learned_addr learned my addr 172.21.15.195:0/2292028549 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:55:46.342 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.302+0000 7f2ff3fff700 1 --2- 172.21.15.195:0/2292028549 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2ff414e490 0x7f2ff40c7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:46.342 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.302+0000 7f2ff887c700 1 --2- 172.21.15.195:0/2292028549 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2ff40c3f50 0x7f2ff41649d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:46.344 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.303+0000 7f2ff907d700 1 -- 172.21.15.195:0/2292028549 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2ff414e490 msgr2=0x7f2ff40c7650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:46.344 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.303+0000 7f2ff907d700 1 --2- 172.21.15.195:0/2292028549 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2ff414e490 0x7f2ff40c7650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:46.344 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.303+0000 7f2ff907d700 1 -- 172.21.15.195:0/2292028549 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2ff40c3f50 msgr2=0x7f2ff41649d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:46.345 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.303+0000 7f2ff907d700 1 --2- 172.21.15.195:0/2292028549 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2ff40c3f50 0x7f2ff41649d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:46.345 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.303+0000 7f2ff907d700 1 -- 172.21.15.195:0/2292028549 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fe80092e0 con 0x7f2ff414e860 2022-06-27T11:55:46.345 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.303+0000 7f2ff907d700 1 --2- 172.21.15.195:0/2292028549 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2ff414e860 0x7f2ff40c0280 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f2fec009ec0 tx=0x7f2fec00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:55:46.345 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.303+0000 7f2ff0ff9700 1 -- 172.21.15.195:0/2292028549 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fec00cc00 con 0x7f2ff414e860 2022-06-27T11:55:46.346 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.303+0000 7f2ff0ff9700 1 -- 172.21.15.195:0/2292028549 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fec00f040 con 0x7f2ff414e860 2022-06-27T11:55:46.346 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.303+0000 7f2ffaae0700 1 -- 172.21.15.195:0/2292028549 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ff41525b0 con 0x7f2ff414e860 2022-06-27T11:55:46.346 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.303+0000 7f2ff0ff9700 1 -- 172.21.15.195:0/2292028549 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fec014480 con 0x7f2ff414e860 2022-06-27T11:55:46.346 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.303+0000 7f2ffaae0700 1 -- 172.21.15.195:0/2292028549 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ff41641d0 con 0x7f2ff414e860 2022-06-27T11:55:46.347 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.304+0000 7f2ffaae0700 1 -- 172.21.15.195:0/2292028549 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ff40722c0 con 0x7f2ff414e860 2022-06-27T11:55:46.347 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.304+0000 7f2ff0ff9700 1 -- 172.21.15.195:0/2292028549 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f2fec014830 con 0x7f2ff414e860 2022-06-27T11:55:46.347 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.305+0000 7f2ff0ff9700 1 --2- 172.21.15.195:0/2292028549 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2fdc063db0 0x7f2fdc066260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:46.347 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.305+0000 7f2ff887c700 1 --2- 172.21.15.195:0/2292028549 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2fdc063db0 0x7f2fdc066260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:46.348 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.305+0000 7f2ff0ff9700 1 -- 172.21.15.195:0/2292028549 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f2fec0816b0 con 0x7f2ff414e860 2022-06-27T11:55:46.348 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.305+0000 7f2ff887c700 1 --2- 172.21.15.195:0/2292028549 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2fdc063db0 0x7f2fdc066260 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f2fe8009260 tx=0x7f2fe8017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:55:46.348 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.309+0000 7f2ff0ff9700 1 -- 172.21.15.195:0/2292028549 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f2fec050020 con 0x7f2ff414e860 2022-06-27T11:55:46.557 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.556+0000 7f2ffaae0700 1 -- 172.21.15.195:0/2292028549 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2ff4164390 con 0x7f2ff414e860 2022-06-27T11:55:46.557 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.556+0000 7f2ff0ff9700 1 -- 172.21.15.195:0/2292028549 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5368 (secure 0 0 0) 0x7f2fec053640 con 0x7f2ff414e860 2022-06-27T11:55:46.560 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:55:46.561 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:55:37.983652+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4856},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4856":{"gid":4856,"name":"d","rank":0,"incarnation":26,"state":"up:active","state_seq":43,"addr":"172.21.15.202:6835/2970948376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2970948376},{"type":"v1","addr":"172.21.15.202:6835","nonce":2970948376}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:55:46.576 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.558+0000 7f2ffaae0700 1 -- 172.21.15.195:0/2292028549 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2fdc063db0 msgr2=0x7f2fdc066260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:46.576 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.558+0000 7f2ffaae0700 1 --2- 172.21.15.195:0/2292028549 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2fdc063db0 0x7f2fdc066260 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f2fe8009260 tx=0x7f2fe8017000 comp rx=0 tx=0).stop 2022-06-27T11:55:46.577 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.558+0000 7f2ffaae0700 1 -- 172.21.15.195:0/2292028549 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2ff414e860 msgr2=0x7f2ff40c0280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:46.577 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.558+0000 7f2ffaae0700 1 --2- 172.21.15.195:0/2292028549 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2ff414e860 0x7f2ff40c0280 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f2fec009ec0 tx=0x7f2fec00b040 comp rx=0 tx=0).stop 2022-06-27T11:55:46.577 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.558+0000 7f2ffaae0700 1 -- 172.21.15.195:0/2292028549 shutdown_connections 2022-06-27T11:55:46.577 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.558+0000 7f2ffaae0700 1 --2- 172.21.15.195:0/2292028549 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2fdc063db0 0x7f2fdc066260 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:46.578 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.558+0000 7f2ffaae0700 1 --2- 172.21.15.195:0/2292028549 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2ff414e860 0x7f2ff40c0280 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:46.579 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.558+0000 7f2ffaae0700 1 --2- 172.21.15.195:0/2292028549 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2ff40c3f50 0x7f2ff41649d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:46.579 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.558+0000 7f2ffaae0700 1 --2- 172.21.15.195:0/2292028549 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2ff414e490 0x7f2ff40c7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:46.580 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.558+0000 7f2ffaae0700 1 -- 172.21.15.195:0/2292028549 >> 172.21.15.195:0/2292028549 conn(0x7f2ff4063af0 msgr2=0x7f2ff40c3840 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:55:46.580 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.558+0000 7f2ffaae0700 1 -- 172.21.15.195:0/2292028549 shutdown_connections 2022-06-27T11:55:46.582 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:46.558+0000 7f2ffaae0700 1 -- 172.21.15.195:0/2292028549 wait complete. 2022-06-27T11:55:46.583 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 30 2022-06-27T11:55:46.584 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2022-06-27T11:55:48.586 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:55:48.763 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.735+0000 7f0cdb79f700 1 Processor -- start 2022-06-27T11:55:48.764 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.735+0000 7f0cdb79f700 1 -- start start 2022-06-27T11:55:48.765 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.736+0000 7f0cdb79f700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f0cd40c40d0 0x7f0cd4157010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:48.765 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.736+0000 7f0cdb79f700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f0cd4067ed0 0x7f0cd414cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:48.765 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.736+0000 7f0cdb79f700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cd4156b80 con 0x7f0cd40c6190 2022-06-27T11:55:48.766 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.736+0000 7f0cdb79f700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cd40682f0 con 0x7f0cd4067ed0 2022-06-27T11:55:48.766 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.736+0000 7f0cdb79f700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cd406a040 con 0x7f0cd40c40d0 2022-06-27T11:55:48.766 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.736+0000 7f0cd9d3c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f0cd4067ed0 0x7f0cd414cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:48.767 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.736+0000 7f0cd9d3c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f0cd4067ed0 0x7f0cd414cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:44612/0 (socket says 172.21.15.195:44612) 2022-06-27T11:55:48.767 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.736+0000 7f0cd9d3c700 1 -- 172.21.15.195:0/3824331946 learned_addr learned my addr 172.21.15.195:0/3824331946 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:55:48.767 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.736+0000 7f0cd953b700 1 --2- 172.21.15.195:0/3824331946 >> v2:172.21.15.202:3300/0 conn(0x7f0cd40c40d0 0x7f0cd4157010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:48.768 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.736+0000 7f0cd9d3c700 1 -- 172.21.15.195:0/3824331946 >> v2:172.21.15.202:3300/0 conn(0x7f0cd40c40d0 msgr2=0x7f0cd4157010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:55:48.768 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.736+0000 7f0cd9d3c700 1 --2- 172.21.15.195:0/3824331946 >> v2:172.21.15.202:3300/0 conn(0x7f0cd40c40d0 0x7f0cd4157010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:48.769 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.736+0000 7f0cd9d3c700 1 -- 172.21.15.195:0/3824331946 >> v1:172.21.15.202:6789/0 conn(0x7f0cd40c6190 legacy=0x7f0cd40c44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:48.770 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.736+0000 7f0cd9d3c700 1 -- 172.21.15.195:0/3824331946 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cd406a300 con 0x7f0cd4067ed0 2022-06-27T11:55:48.770 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.737+0000 7f0cd9d3c700 1 --2- 172.21.15.195:0/3824331946 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0cd4067ed0 0x7f0cd414cb80 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f0cd40cdb00 tx=0x7f0ccc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a4f8cb530009e24 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:55:48.771 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.737+0000 7f0cc3fff700 1 -- 172.21.15.195:0/3824331946 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ccc00e070 con 0x7f0cd4067ed0 2022-06-27T11:55:48.771 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.737+0000 7f0cc3fff700 1 -- 172.21.15.195:0/3824331946 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ccc00ba20 con 0x7f0cd4067ed0 2022-06-27T11:55:48.771 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.737+0000 7f0cc3fff700 1 -- 172.21.15.195:0/3824331946 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ccc0092e0 con 0x7f0cd4067ed0 2022-06-27T11:55:48.772 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.737+0000 7f0cdb79f700 1 -- 172.21.15.195:0/3824331946 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0cd4067ed0 msgr2=0x7f0cd414cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:48.772 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.737+0000 7f0cdb79f700 1 --2- 172.21.15.195:0/3824331946 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0cd4067ed0 0x7f0cd414cb80 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f0cd40cdb00 tx=0x7f0ccc009840 comp rx=0 tx=0).stop 2022-06-27T11:55:48.772 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.737+0000 7f0cdb79f700 1 -- 172.21.15.195:0/3824331946 shutdown_connections 2022-06-27T11:55:48.773 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.737+0000 7f0cdb79f700 1 --2- 172.21.15.195:0/3824331946 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0cd4067ed0 0x7f0cd414cb80 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:48.773 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.737+0000 7f0cdb79f700 1 --2- 172.21.15.195:0/3824331946 >> v2:172.21.15.202:3300/0 conn(0x7f0cd40c40d0 0x7f0cd4157010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:48.773 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.737+0000 7f0cdb79f700 1 -- 172.21.15.195:0/3824331946 >> 172.21.15.195:0/3824331946 conn(0x7f0cd40ccb80 msgr2=0x7f0cd40ccf80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:55:48.774 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.737+0000 7f0cdb79f700 1 -- 172.21.15.195:0/3824331946 shutdown_connections 2022-06-27T11:55:48.774 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.737+0000 7f0cdb79f700 1 -- 172.21.15.195:0/3824331946 wait complete. 2022-06-27T11:55:48.774 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.738+0000 7f0cdb79f700 1 Processor -- start 2022-06-27T11:55:48.775 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.739+0000 7f0cdb79f700 1 -- start start 2022-06-27T11:55:48.775 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.739+0000 7f0cdb79f700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0cd4067ed0 0x7f0cd4166fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:48.775 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.739+0000 7f0cdb79f700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0cd40c40d0 0x7f0cd4152570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:48.775 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.739+0000 7f0cdb79f700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0cd40c6190 0x7f0cd4152ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:48.776 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.739+0000 7f0cdb79f700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd41674e0 con 0x7f0cd40c6190 2022-06-27T11:55:48.776 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.739+0000 7f0cdb79f700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd41520e0 con 0x7f0cd40c40d0 2022-06-27T11:55:48.776 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.739+0000 7f0cdb79f700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd40c0a40 con 0x7f0cd4067ed0 2022-06-27T11:55:48.777 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.739+0000 7f0cd9d3c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0cd40c6190 0x7f0cd4152ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:48.777 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.740+0000 7f0cd9d3c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0cd40c6190 0x7f0cd4152ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:44618/0 (socket says 172.21.15.195:44618) 2022-06-27T11:55:48.777 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.740+0000 7f0cd9d3c700 1 -- 172.21.15.195:0/2562475921 learned_addr learned my addr 172.21.15.195:0/2562475921 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:55:48.777 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.740+0000 7f0cd8d3a700 1 --2- 172.21.15.195:0/2562475921 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0cd40c40d0 0x7f0cd4152570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:48.778 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.740+0000 7f0cd953b700 1 --2- 172.21.15.195:0/2562475921 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0cd4067ed0 0x7f0cd4166fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:48.778 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.740+0000 7f0cd9d3c700 1 -- 172.21.15.195:0/2562475921 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0cd4067ed0 msgr2=0x7f0cd4166fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:48.778 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.740+0000 7f0cd9d3c700 1 --2- 172.21.15.195:0/2562475921 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0cd4067ed0 0x7f0cd4166fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:48.779 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.740+0000 7f0cd9d3c700 1 -- 172.21.15.195:0/2562475921 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0cd40c40d0 msgr2=0x7f0cd4152570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:48.780 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.740+0000 7f0cd9d3c700 1 --2- 172.21.15.195:0/2562475921 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0cd40c40d0 0x7f0cd4152570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:48.780 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.740+0000 7f0cd9d3c700 1 -- 172.21.15.195:0/2562475921 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ccc009ae0 con 0x7f0cd40c6190 2022-06-27T11:55:48.781 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.740+0000 7f0cd953b700 1 --2- 172.21.15.195:0/2562475921 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0cd4067ed0 0x7f0cd4166fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T11:55:48.781 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.740+0000 7f0cd8d3a700 1 --2- 172.21.15.195:0/2562475921 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0cd40c40d0 0x7f0cd4152570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T11:55:48.782 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.740+0000 7f0cd9d3c700 1 --2- 172.21.15.195:0/2562475921 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0cd40c6190 0x7f0cd4152ab0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f0ccc00b6c0 tx=0x7f0ccc018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:55:48.782 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.740+0000 7f0cc17fa700 1 -- 172.21.15.195:0/2562475921 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ccc003e80 con 0x7f0cd40c6190 2022-06-27T11:55:48.788 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.740+0000 7f0cc17fa700 1 -- 172.21.15.195:0/2562475921 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ccc007610 con 0x7f0cd40c6190 2022-06-27T11:55:48.788 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.740+0000 7f0cc17fa700 1 -- 172.21.15.195:0/2562475921 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ccc007990 con 0x7f0cd40c6190 2022-06-27T11:55:48.789 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.740+0000 7f0cdb79f700 1 -- 172.21.15.195:0/2562475921 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cd40c0d20 con 0x7f0cd40c6190 2022-06-27T11:55:48.790 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.740+0000 7f0cdb79f700 1 -- 172.21.15.195:0/2562475921 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cd4153a50 con 0x7f0cd40c6190 2022-06-27T11:55:48.791 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.741+0000 7f0cdb79f700 1 -- 172.21.15.195:0/2562475921 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cd40722c0 con 0x7f0cd40c6190 2022-06-27T11:55:48.791 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.741+0000 7f0cc17fa700 1 -- 172.21.15.195:0/2562475921 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f0ccc004040 con 0x7f0cd40c6190 2022-06-27T11:55:48.792 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.742+0000 7f0cc17fa700 1 --2- 172.21.15.195:0/2562475921 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0cb4063d00 0x7f0cb40661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:55:48.793 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.742+0000 7f0cc17fa700 1 -- 172.21.15.195:0/2562475921 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f0ccc082810 con 0x7f0cd40c6190 2022-06-27T11:55:48.794 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.742+0000 7f0cd953b700 1 --2- 172.21.15.195:0/2562475921 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0cb4063d00 0x7f0cb40661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:55:48.794 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.742+0000 7f0cd953b700 1 --2- 172.21.15.195:0/2562475921 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0cb4063d00 0x7f0cb40661b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f0cc4009c40 tx=0x7f0cc4008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:55:48.794 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.746+0000 7f0cc17fa700 1 -- 172.21.15.195:0/2562475921 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f0ccc051270 con 0x7f0cd40c6190 2022-06-27T11:55:48.975 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.972+0000 7f0cdb79f700 1 -- 172.21.15.195:0/2562475921 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0cd40652c0 con 0x7f0cd40c6190 2022-06-27T11:55:48.976 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.972+0000 7f0cc17fa700 1 -- 172.21.15.195:0/2562475921 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+6128 (secure 0 0 0) 0x7f0ccc02a090 con 0x7f0cd40c6190 2022-06-27T11:55:48.977 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:55:48.980 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/4154570751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":4154570751},{"type":"v1","addr":"172.21.15.202:6833","nonce":4154570751}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:55:37.983652+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4856},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4856":{"gid":4856,"name":"d","rank":0,"incarnation":26,"state":"up:active","state_seq":43,"addr":"172.21.15.202:6835/2970948376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2970948376},{"type":"v1","addr":"172.21.15.202:6835","nonce":2970948376}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:55:48.987 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.974+0000 7f0cdb79f700 1 -- 172.21.15.195:0/2562475921 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0cb4063d00 msgr2=0x7f0cb40661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:48.987 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.974+0000 7f0cdb79f700 1 --2- 172.21.15.195:0/2562475921 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0cb4063d00 0x7f0cb40661b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f0cc4009c40 tx=0x7f0cc4008040 comp rx=0 tx=0).stop 2022-06-27T11:55:48.988 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.974+0000 7f0cdb79f700 1 -- 172.21.15.195:0/2562475921 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0cd40c6190 msgr2=0x7f0cd4152ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:55:48.990 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.974+0000 7f0cdb79f700 1 --2- 172.21.15.195:0/2562475921 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0cd40c6190 0x7f0cd4152ab0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f0ccc00b6c0 tx=0x7f0ccc018000 comp rx=0 tx=0).stop 2022-06-27T11:55:48.991 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.974+0000 7f0cdb79f700 1 -- 172.21.15.195:0/2562475921 shutdown_connections 2022-06-27T11:55:48.991 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.974+0000 7f0cdb79f700 1 --2- 172.21.15.195:0/2562475921 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0cb4063d00 0x7f0cb40661b0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:48.992 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.974+0000 7f0cdb79f700 1 --2- 172.21.15.195:0/2562475921 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0cd40c6190 0x7f0cd4152ab0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:48.992 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.974+0000 7f0cdb79f700 1 --2- 172.21.15.195:0/2562475921 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0cd4067ed0 0x7f0cd4166fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:48.993 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.974+0000 7f0cdb79f700 1 --2- 172.21.15.195:0/2562475921 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0cd40c40d0 0x7f0cd4152570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:55:48.993 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.974+0000 7f0cdb79f700 1 -- 172.21.15.195:0/2562475921 >> 172.21.15.195:0/2562475921 conn(0x7f0cd40ccb80 msgr2=0x7f0cd40c9600 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:55:48.993 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.974+0000 7f0cdb79f700 1 -- 172.21.15.195:0/2562475921 shutdown_connections 2022-06-27T11:55:48.994 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:55:48.974+0000 7f0cdb79f700 1 -- 172.21.15.195:0/2562475921 wait complete. 2022-06-27T11:55:48.994 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 31 2022-06-27T11:55:48.997 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2022-06-27T11:55:49.000 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 55 secs before thrashing 2022-06-27T11:55:58.555 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T11:55:58.564 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T11:56:28.627 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T11:56:28.636 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T11:56:44.001 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:56:44.135 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.135+0000 7fe03426d700 1 Processor -- start 2022-06-27T11:56:44.136 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.135+0000 7fe03426d700 1 -- start start 2022-06-27T11:56:44.139 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.136+0000 7fe03426d700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fe02c06a450 0x7fe02c063af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:56:44.139 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.136+0000 7fe03426d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fe02c1508d0 0x7fe02c0bbf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:56:44.139 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.136+0000 7fe03426d700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe02c064030 con 0x7fe02c150500 2022-06-27T11:56:44.140 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.136+0000 7fe03426d700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe02c0c4160 con 0x7fe02c1508d0 2022-06-27T11:56:44.140 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.136+0000 7fe03426d700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe02c0c4440 con 0x7fe02c06a450 2022-06-27T11:56:44.140 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.136+0000 7fe03280a700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fe02c1508d0 0x7fe02c0bbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:56:44.140 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.137+0000 7fe03280a700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fe02c1508d0 0x7fe02c0bbf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:44864/0 (socket says 172.21.15.195:44864) 2022-06-27T11:56:44.141 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.137+0000 7fe03280a700 1 -- 172.21.15.195:0/1914743054 learned_addr learned my addr 172.21.15.195:0/1914743054 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:56:44.141 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.137+0000 7fe031808700 1 --2- 172.21.15.195:0/1914743054 >> v2:172.21.15.202:3300/0 conn(0x7fe02c06a450 0x7fe02c063af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:56:44.141 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.137+0000 7fe03280a700 1 -- 172.21.15.195:0/1914743054 >> v2:172.21.15.202:3300/0 conn(0x7fe02c06a450 msgr2=0x7fe02c063af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:56:44.141 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.137+0000 7fe03280a700 1 --2- 172.21.15.195:0/1914743054 >> v2:172.21.15.202:3300/0 conn(0x7fe02c06a450 0x7fe02c063af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:44.142 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.137+0000 7fe03280a700 1 -- 172.21.15.195:0/1914743054 >> v1:172.21.15.202:6789/0 conn(0x7fe02c150500 legacy=0x7fe02c0c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:56:44.142 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.137+0000 7fe03280a700 1 -- 172.21.15.195:0/1914743054 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe02c15a800 con 0x7fe02c1508d0 2022-06-27T11:56:44.142 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.137+0000 7fe031808700 1 --2- 172.21.15.195:0/1914743054 >> v2:172.21.15.202:3300/0 conn(0x7fe02c06a450 0x7fe02c063af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T11:56:44.142 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.137+0000 7fe03280a700 1 --2- 172.21.15.195:0/1914743054 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe02c1508d0 0x7fe02c0bbf50 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fe02c06ce70 tx=0x7fe024009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed20e9846306132c server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:56:44.143 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.137+0000 7fe031007700 1 -- 172.21.15.195:0/1914743054 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe02400e070 con 0x7fe02c1508d0 2022-06-27T11:56:44.143 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.137+0000 7fe031007700 1 -- 172.21.15.195:0/1914743054 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe02400ba10 con 0x7fe02c1508d0 2022-06-27T11:56:44.146 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.137+0000 7fe031007700 1 -- 172.21.15.195:0/1914743054 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0240092e0 con 0x7fe02c1508d0 2022-06-27T11:56:44.146 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.138+0000 7fe03426d700 1 -- 172.21.15.195:0/1914743054 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe02c1508d0 msgr2=0x7fe02c0bbf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:56:44.146 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.138+0000 7fe03426d700 1 --2- 172.21.15.195:0/1914743054 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe02c1508d0 0x7fe02c0bbf50 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fe02c06ce70 tx=0x7fe024009840 comp rx=0 tx=0).stop 2022-06-27T11:56:44.147 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.138+0000 7fe03426d700 1 -- 172.21.15.195:0/1914743054 shutdown_connections 2022-06-27T11:56:44.147 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.138+0000 7fe03426d700 1 --2- 172.21.15.195:0/1914743054 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe02c1508d0 0x7fe02c0bbf50 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:44.147 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.138+0000 7fe03426d700 1 --2- 172.21.15.195:0/1914743054 >> v2:172.21.15.202:3300/0 conn(0x7fe02c06a450 0x7fe02c063af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:44.147 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.138+0000 7fe03426d700 1 -- 172.21.15.195:0/1914743054 >> 172.21.15.195:0/1914743054 conn(0x7fe02c0c9b00 msgr2=0x7fe02c0c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:56:44.148 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.138+0000 7fe03426d700 1 -- 172.21.15.195:0/1914743054 shutdown_connections 2022-06-27T11:56:44.148 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.138+0000 7fe03426d700 1 -- 172.21.15.195:0/1914743054 wait complete. 2022-06-27T11:56:44.148 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.139+0000 7fe03426d700 1 Processor -- start 2022-06-27T11:56:44.149 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.139+0000 7fe03426d700 1 -- start start 2022-06-27T11:56:44.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.140+0000 7fe03426d700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe02c06a450 0x7fe02c169c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:56:44.155 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.140+0000 7fe03426d700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe02c150500 0x7fe02c0bf080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:56:44.155 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.140+0000 7fe03426d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe02c1508d0 0x7fe02c159040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:56:44.156 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.140+0000 7fe03426d700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe02c14f9b0 con 0x7fe02c1508d0 2022-06-27T11:56:44.156 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.140+0000 7fe03426d700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe02c157590 con 0x7fe02c150500 2022-06-27T11:56:44.158 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.140+0000 7fe03426d700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe02c158880 con 0x7fe02c06a450 2022-06-27T11:56:44.158 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.140+0000 7fe03280a700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe02c1508d0 0x7fe02c159040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:56:44.159 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.140+0000 7fe03280a700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe02c1508d0 0x7fe02c159040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:44872/0 (socket says 172.21.15.195:44872) 2022-06-27T11:56:44.159 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.140+0000 7fe03280a700 1 -- 172.21.15.195:0/1664253930 learned_addr learned my addr 172.21.15.195:0/1664253930 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:56:44.159 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.140+0000 7fe031808700 1 --2- 172.21.15.195:0/1664253930 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe02c150500 0x7fe02c0bf080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:56:44.160 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.140+0000 7fe032009700 1 --2- 172.21.15.195:0/1664253930 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe02c06a450 0x7fe02c169c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:56:44.161 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.140+0000 7fe03280a700 1 -- 172.21.15.195:0/1664253930 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe02c06a450 msgr2=0x7fe02c169c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:56:44.161 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.140+0000 7fe03280a700 1 --2- 172.21.15.195:0/1664253930 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe02c06a450 0x7fe02c169c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:44.161 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.140+0000 7fe03280a700 1 -- 172.21.15.195:0/1664253930 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe02c150500 msgr2=0x7fe02c0bf080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:56:44.162 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.140+0000 7fe03280a700 1 --2- 172.21.15.195:0/1664253930 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe02c150500 0x7fe02c0bf080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:44.162 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.140+0000 7fe03280a700 1 -- 172.21.15.195:0/1664253930 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe024009ae0 con 0x7fe02c1508d0 2022-06-27T11:56:44.162 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.140+0000 7fe031808700 1 --2- 172.21.15.195:0/1664253930 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe02c150500 0x7fe02c0bf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T11:56:44.162 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.140+0000 7fe03280a700 1 --2- 172.21.15.195:0/1664253930 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe02c1508d0 0x7fe02c159040 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fe024009880 tx=0x7fe024017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:56:44.163 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.140+0000 7fe01e7fc700 1 -- 172.21.15.195:0/1664253930 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe024004240 con 0x7fe02c1508d0 2022-06-27T11:56:44.163 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.140+0000 7fe01e7fc700 1 -- 172.21.15.195:0/1664253930 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0240043a0 con 0x7fe02c1508d0 2022-06-27T11:56:44.163 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.140+0000 7fe03426d700 1 -- 172.21.15.195:0/1664253930 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe02c0bcfa0 con 0x7fe02c1508d0 2022-06-27T11:56:44.163 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.140+0000 7fe01e7fc700 1 -- 172.21.15.195:0/1664253930 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe024004760 con 0x7fe02c1508d0 2022-06-27T11:56:44.164 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.140+0000 7fe03426d700 1 -- 172.21.15.195:0/1664253930 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe02c0bd540 con 0x7fe02c1508d0 2022-06-27T11:56:44.164 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.141+0000 7fe03426d700 1 -- 172.21.15.195:0/1664253930 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe02c0722c0 con 0x7fe02c1508d0 2022-06-27T11:56:44.164 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.142+0000 7fe01e7fc700 1 -- 172.21.15.195:0/1664253930 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fe02402c070 con 0x7fe02c1508d0 2022-06-27T11:56:44.164 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.142+0000 7fe01e7fc700 1 --2- 172.21.15.195:0/1664253930 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe00c063d00 0x7fe00c0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:56:44.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.142+0000 7fe01e7fc700 1 -- 172.21.15.195:0/1664253930 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fe024081df0 con 0x7fe02c1508d0 2022-06-27T11:56:44.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.142+0000 7fe032009700 1 --2- 172.21.15.195:0/1664253930 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe00c063d00 0x7fe00c0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:56:44.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.142+0000 7fe032009700 1 --2- 172.21.15.195:0/1664253930 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe00c063d00 0x7fe00c0661b0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fe020007fd0 tx=0x7fe020007db0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:56:44.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.146+0000 7fe01e7fc700 1 -- 172.21.15.195:0/1664253930 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fe024050850 con 0x7fe02c1508d0 2022-06-27T11:56:44.372 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.371+0000 7fe03426d700 1 -- 172.21.15.195:0/1664253930 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe02c064030 con 0x7fe02c1508d0 2022-06-27T11:56:44.372 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.371+0000 7fe01e7fc700 1 -- 172.21.15.195:0/1664253930 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+6128 (secure 0 0 0) 0x7fe02c064030 con 0x7fe02c1508d0 2022-06-27T11:56:44.373 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:56:44.373 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/4154570751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":4154570751},{"type":"v1","addr":"172.21.15.202:6833","nonce":4154570751}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:55:37.983652+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4856},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4856":{"gid":4856,"name":"d","rank":0,"incarnation":26,"state":"up:active","state_seq":43,"addr":"172.21.15.202:6835/2970948376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2970948376},{"type":"v1","addr":"172.21.15.202:6835","nonce":2970948376}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:56:44.374 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.373+0000 7fe03426d700 1 -- 172.21.15.195:0/1664253930 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe00c063d00 msgr2=0x7fe00c0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:56:44.374 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.373+0000 7fe03426d700 1 --2- 172.21.15.195:0/1664253930 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe00c063d00 0x7fe00c0661b0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fe020007fd0 tx=0x7fe020007db0 comp rx=0 tx=0).stop 2022-06-27T11:56:44.375 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.373+0000 7fe03426d700 1 -- 172.21.15.195:0/1664253930 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe02c1508d0 msgr2=0x7fe02c159040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:56:44.375 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.373+0000 7fe03426d700 1 --2- 172.21.15.195:0/1664253930 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe02c1508d0 0x7fe02c159040 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fe024009880 tx=0x7fe024017000 comp rx=0 tx=0).stop 2022-06-27T11:56:44.375 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.373+0000 7fe03426d700 1 -- 172.21.15.195:0/1664253930 shutdown_connections 2022-06-27T11:56:44.388 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.373+0000 7fe03426d700 1 --2- 172.21.15.195:0/1664253930 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe00c063d00 0x7fe00c0661b0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:44.388 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.373+0000 7fe03426d700 1 --2- 172.21.15.195:0/1664253930 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe02c1508d0 0x7fe02c159040 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:44.389 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.373+0000 7fe03426d700 1 --2- 172.21.15.195:0/1664253930 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe02c06a450 0x7fe02c169c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:44.389 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.373+0000 7fe03426d700 1 --2- 172.21.15.195:0/1664253930 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe02c150500 0x7fe02c0bf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:44.390 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.373+0000 7fe03426d700 1 -- 172.21.15.195:0/1664253930 >> 172.21.15.195:0/1664253930 conn(0x7fe02c0c9b00 msgr2=0x7fe02c0c58c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:56:44.390 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.373+0000 7fe03426d700 1 -- 172.21.15.195:0/1664253930 shutdown_connections 2022-06-27T11:56:44.390 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.373+0000 7fe03426d700 1 -- 172.21.15.195:0/1664253930 wait complete. 2022-06-27T11:56:44.391 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 31 2022-06-27T11:56:44.393 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.d (rank=0) 2022-06-27T11:56:44.393 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2022-06-27T11:56:44.395 INFO:teuthology.orchestra.run:waiting for 300 2022-06-27T11:56:44.410 INFO:tasks.ceph.mds.d:Stopped 2022-06-27T11:56:44.411 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:56:44.541 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.542+0000 7fe5ecac2700 1 Processor -- start 2022-06-27T11:56:44.542 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.542+0000 7fe5ecac2700 1 -- start start 2022-06-27T11:56:44.553 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.543+0000 7fe5ecac2700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe5e806f640 0x7fe5e8156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:56:44.554 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.543+0000 7fe5ecac2700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5e80c1a20 con 0x7fe5e80c39e0 2022-06-27T11:56:44.554 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.543+0000 7fe5ecac2700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5e80c1130 con 0x7fe5e80c4160 2022-06-27T11:56:44.555 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.543+0000 7fe5ecac2700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e806ea90 con 0x7fe5e806f640 2022-06-27T11:56:44.555 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.543+0000 7fe5e6d9d700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fe5e80c39e0 0x7fe5e81535a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:51064/0 (socket says 172.21.15.195:51064) 2022-06-27T11:56:44.555 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.543+0000 7fe5e6d9d700 1 -- 172.21.15.195:0/1316222877 learned_addr learned my addr 172.21.15.195:0/1316222877 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:56:44.555 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.543+0000 7fe5e659c700 1 --2- 172.21.15.195:0/1316222877 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe5e806f640 0x7fe5e8156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:56:44.556 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.543+0000 7fe5e559a700 1 -- 172.21.15.195:0/1316222877 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 892019505 0 0) 0x7fe5e80c1a20 con 0x7fe5e80c39e0 2022-06-27T11:56:44.556 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.543+0000 7fe5e559a700 1 -- 172.21.15.195:0/1316222877 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5cc003430 con 0x7fe5e80c39e0 2022-06-27T11:56:44.556 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.543+0000 7fe5e559a700 1 -- 172.21.15.195:0/1316222877 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3026324731 0 0) 0x7fe5cc003430 con 0x7fe5e80c39e0 2022-06-27T11:56:44.556 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.543+0000 7fe5e559a700 1 -- 172.21.15.195:0/1316222877 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe5e806f640 msgr2=0x7fe5e8156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:56:44.556 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.543+0000 7fe5e559a700 1 --2- 172.21.15.195:0/1316222877 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe5e806f640 0x7fe5e8156ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:44.557 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.543+0000 7fe5e559a700 1 -- 172.21.15.195:0/1316222877 >> v1:172.21.15.202:6789/0 conn(0x7fe5e80c4160 legacy=0x7fe5e80c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:56:44.557 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.543+0000 7fe5e559a700 1 -- 172.21.15.195:0/1316222877 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5e806ed70 con 0x7fe5e80c39e0 2022-06-27T11:56:44.557 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.544+0000 7fe5e559a700 1 -- 172.21.15.195:0/1316222877 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7fe5e00017f0 con 0x7fe5e80c39e0 2022-06-27T11:56:44.558 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.544+0000 7fe5e559a700 1 -- 172.21.15.195:0/1316222877 >> v1:172.21.15.195:6789/0 conn(0x7fe5e80c39e0 legacy=0x7fe5e81535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:56:44.558 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.544+0000 7fe5e559a700 1 --2- 172.21.15.195:0/1316222877 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe5cc003950 0x7fe5cc003d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:56:44.558 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.544+0000 7fe5e559a700 1 --2- 172.21.15.195:0/1316222877 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe5e806f640 0x7fe5cc0046d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:56:44.559 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.544+0000 7fe5e559a700 1 --2- 172.21.15.195:0/1316222877 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe5cc004c10 0x7fe5cc005080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:56:44.559 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.544+0000 7fe5e559a700 1 -- 172.21.15.195:0/1316222877 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e806ea90 con 0x7fe5cc004c10 2022-06-27T11:56:44.559 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.544+0000 7fe5e559a700 1 -- 172.21.15.195:0/1316222877 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5cc00ba30 con 0x7fe5cc003950 2022-06-27T11:56:44.560 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.544+0000 7fe5e559a700 1 -- 172.21.15.195:0/1316222877 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5cc00bbf0 con 0x7fe5e806f640 2022-06-27T11:56:44.560 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.544+0000 7fe5e6d9d700 1 --2- 172.21.15.195:0/1316222877 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe5cc004c10 0x7fe5cc005080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:56:44.560 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.544+0000 7fe5e659c700 1 --2- 172.21.15.195:0/1316222877 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe5cc003950 0x7fe5cc003d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:56:44.561 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.544+0000 7fe5e5d9b700 1 --2- 172.21.15.195:0/1316222877 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe5e806f640 0x7fe5cc0046d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:56:44.561 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.544+0000 7fe5e6d9d700 1 -- 172.21.15.195:0/1316222877 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe5e806f640 msgr2=0x7fe5cc0046d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:56:44.561 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.544+0000 7fe5e6d9d700 1 --2- 172.21.15.195:0/1316222877 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe5e806f640 0x7fe5cc0046d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:44.561 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.544+0000 7fe5e6d9d700 1 -- 172.21.15.195:0/1316222877 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe5cc003950 msgr2=0x7fe5cc003d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:56:44.562 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.544+0000 7fe5e6d9d700 1 --2- 172.21.15.195:0/1316222877 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe5cc003950 0x7fe5cc003d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:44.562 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.544+0000 7fe5e6d9d700 1 -- 172.21.15.195:0/1316222877 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5cc003430 con 0x7fe5cc004c10 2022-06-27T11:56:44.562 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.544+0000 7fe5e6d9d700 1 --2- 172.21.15.195:0/1316222877 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe5cc004c10 0x7fe5cc005080 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fe5e000c530 tx=0x7fe5e0001d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc8910d1a33d2149 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:56:44.563 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.544+0000 7fe5e559a700 1 -- 172.21.15.195:0/1316222877 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e0001fc0 con 0x7fe5cc004c10 2022-06-27T11:56:44.563 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.544+0000 7fe5e559a700 1 -- 172.21.15.195:0/1316222877 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5e0005b60 con 0x7fe5cc004c10 2022-06-27T11:56:44.563 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.544+0000 7fe5e559a700 1 -- 172.21.15.195:0/1316222877 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e00070d0 con 0x7fe5cc004c10 2022-06-27T11:56:44.564 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.545+0000 7fe5ecac2700 1 -- 172.21.15.195:0/1316222877 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe5cc004c10 msgr2=0x7fe5cc005080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:56:44.564 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.545+0000 7fe5ecac2700 1 --2- 172.21.15.195:0/1316222877 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe5cc004c10 0x7fe5cc005080 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fe5e000c530 tx=0x7fe5e0001d90 comp rx=0 tx=0).stop 2022-06-27T11:56:44.564 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.545+0000 7fe5e6d9d700 1 -- 172.21.15.195:0/1316222877 reap_dead start 2022-06-27T11:56:44.565 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.545+0000 7fe5ecac2700 1 -- 172.21.15.195:0/1316222877 shutdown_connections 2022-06-27T11:56:44.565 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.545+0000 7fe5ecac2700 1 -- 172.21.15.195:0/1316222877 >> 172.21.15.195:0/1316222877 conn(0x7fe5e80c6350 msgr2=0x7fe5e80c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:56:44.565 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.545+0000 7fe5ecac2700 1 -- 172.21.15.195:0/1316222877 shutdown_connections 2022-06-27T11:56:44.566 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.545+0000 7fe5ecac2700 1 -- 172.21.15.195:0/1316222877 wait complete. 2022-06-27T11:56:44.566 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.546+0000 7fe5ecac2700 1 Processor -- start 2022-06-27T11:56:44.566 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.546+0000 7fe5ecac2700 1 -- start start 2022-06-27T11:56:44.567 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.547+0000 7fe5ecac2700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe5cc004c10 0x7fe5e8154930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:56:44.567 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.547+0000 7fe5ecac2700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe5e80bcfa0 0x7fe5e814e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:56:44.567 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.547+0000 7fe5ecac2700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe5e8166b90 0x7fe5e8167000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:56:44.567 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.547+0000 7fe5ecac2700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e8155070 con 0x7fe5e8166b90 2022-06-27T11:56:44.568 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.547+0000 7fe5ecac2700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e8154e70 con 0x7fe5e80bcfa0 2022-06-27T11:56:44.568 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.547+0000 7fe5ecac2700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e8167540 con 0x7fe5cc004c10 2022-06-27T11:56:44.568 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.547+0000 7fe5e6d9d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe5e8166b90 0x7fe5e8167000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:56:44.569 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.547+0000 7fe5e6d9d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe5e8166b90 0x7fe5e8167000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:44892/0 (socket says 172.21.15.195:44892) 2022-06-27T11:56:44.569 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.547+0000 7fe5e6d9d700 1 -- 172.21.15.195:0/1025604013 learned_addr learned my addr 172.21.15.195:0/1025604013 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:56:44.569 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.547+0000 7fe5e659c700 1 --2- 172.21.15.195:0/1025604013 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe5cc004c10 0x7fe5e8154930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:56:44.570 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.547+0000 7fe5e5d9b700 1 --2- 172.21.15.195:0/1025604013 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe5e80bcfa0 0x7fe5e814e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:56:44.570 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.547+0000 7fe5e6d9d700 1 -- 172.21.15.195:0/1025604013 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe5cc004c10 msgr2=0x7fe5e8154930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:56:44.572 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.547+0000 7fe5e6d9d700 1 --2- 172.21.15.195:0/1025604013 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe5cc004c10 0x7fe5e8154930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:44.572 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.547+0000 7fe5e6d9d700 1 -- 172.21.15.195:0/1025604013 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe5e80bcfa0 msgr2=0x7fe5e814e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:56:44.572 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.547+0000 7fe5e6d9d700 1 --2- 172.21.15.195:0/1025604013 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe5e80bcfa0 0x7fe5e814e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:44.573 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.547+0000 7fe5e6d9d700 1 -- 172.21.15.195:0/1025604013 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5e816a120 con 0x7fe5e8166b90 2022-06-27T11:56:44.573 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.547+0000 7fe5e6d9d700 1 --2- 172.21.15.195:0/1025604013 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe5e8166b90 0x7fe5e8167000 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fe5cc00ba00 tx=0x7fe5e0015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:56:44.573 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.547+0000 7fe5d67fc700 1 -- 172.21.15.195:0/1025604013 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e00070d0 con 0x7fe5e8166b90 2022-06-27T11:56:44.574 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.547+0000 7fe5d67fc700 1 -- 172.21.15.195:0/1025604013 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5e0001fc0 con 0x7fe5e8166b90 2022-06-27T11:56:44.574 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.547+0000 7fe5ecac2700 1 -- 172.21.15.195:0/1025604013 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5e816a400 con 0x7fe5e8166b90 2022-06-27T11:56:44.575 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.547+0000 7fe5d67fc700 1 -- 172.21.15.195:0/1025604013 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e000ce60 con 0x7fe5e8166b90 2022-06-27T11:56:44.575 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.548+0000 7fe5ecac2700 1 -- 172.21.15.195:0/1025604013 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5e816a9c0 con 0x7fe5e8166b90 2022-06-27T11:56:44.575 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.548+0000 7fe5ecac2700 1 -- 172.21.15.195:0/1025604013 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5e80722c0 con 0x7fe5e8166b90 2022-06-27T11:56:44.576 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.549+0000 7fe5d67fc700 1 -- 172.21.15.195:0/1025604013 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fe5e0010070 con 0x7fe5e8166b90 2022-06-27T11:56:44.576 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.549+0000 7fe5d67fc700 1 --2- 172.21.15.195:0/1025604013 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe5c4063d00 0x7fe5c40661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:56:44.576 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.549+0000 7fe5d67fc700 1 -- 172.21.15.195:0/1025604013 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fe5e0081a20 con 0x7fe5e8166b90 2022-06-27T11:56:44.577 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.549+0000 7fe5e659c700 1 --2- 172.21.15.195:0/1025604013 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe5c4063d00 0x7fe5c40661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:56:44.577 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.549+0000 7fe5e659c700 1 --2- 172.21.15.195:0/1025604013 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe5c4063d00 0x7fe5c40661b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fe5d8006dc0 tx=0x7fe5d8009060 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:56:44.578 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.553+0000 7fe5d67fc700 1 -- 172.21.15.195:0/1025604013 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fe5e0050480 con 0x7fe5e8166b90 2022-06-27T11:56:44.773 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.772+0000 7fe5ecac2700 1 -- 172.21.15.195:0/1025604013 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe5e8155500 con 0x7fe5e8166b90 2022-06-27T11:56:44.773 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.773+0000 7fe5d67fc700 1 -- 172.21.15.195:0/1025604013 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+6128 (secure 0 0 0) 0x7fe5e0017020 con 0x7fe5e8166b90 2022-06-27T11:56:44.775 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:56:44.775 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/4154570751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":4154570751},{"type":"v1","addr":"172.21.15.202:6833","nonce":4154570751}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:55:37.983652+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4856},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4856":{"gid":4856,"name":"d","rank":0,"incarnation":26,"state":"up:active","state_seq":43,"addr":"172.21.15.202:6835/2970948376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2970948376},{"type":"v1","addr":"172.21.15.202:6835","nonce":2970948376}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:56:44.777 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.775+0000 7fe5ecac2700 1 -- 172.21.15.195:0/1025604013 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe5c4063d00 msgr2=0x7fe5c40661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:56:44.777 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.775+0000 7fe5ecac2700 1 --2- 172.21.15.195:0/1025604013 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe5c4063d00 0x7fe5c40661b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fe5d8006dc0 tx=0x7fe5d8009060 comp rx=0 tx=0).stop 2022-06-27T11:56:44.777 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.775+0000 7fe5ecac2700 1 -- 172.21.15.195:0/1025604013 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe5e8166b90 msgr2=0x7fe5e8167000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:56:44.778 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.775+0000 7fe5ecac2700 1 --2- 172.21.15.195:0/1025604013 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe5e8166b90 0x7fe5e8167000 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fe5cc00ba00 tx=0x7fe5e0015000 comp rx=0 tx=0).stop 2022-06-27T11:56:44.778 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.775+0000 7fe5ecac2700 1 -- 172.21.15.195:0/1025604013 shutdown_connections 2022-06-27T11:56:44.779 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.775+0000 7fe5ecac2700 1 --2- 172.21.15.195:0/1025604013 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe5c4063d00 0x7fe5c40661b0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:44.779 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.775+0000 7fe5ecac2700 1 --2- 172.21.15.195:0/1025604013 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe5e8166b90 0x7fe5e8167000 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:44.779 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.775+0000 7fe5ecac2700 1 --2- 172.21.15.195:0/1025604013 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe5cc004c10 0x7fe5e8154930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:44.780 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.775+0000 7fe5ecac2700 1 --2- 172.21.15.195:0/1025604013 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe5e80bcfa0 0x7fe5e814e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:44.781 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.775+0000 7fe5ecac2700 1 -- 172.21.15.195:0/1025604013 >> 172.21.15.195:0/1025604013 conn(0x7fe5e80c6350 msgr2=0x7fe5e81504b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:56:44.781 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.775+0000 7fe5ecac2700 1 -- 172.21.15.195:0/1025604013 shutdown_connections 2022-06-27T11:56:44.781 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:44.775+0000 7fe5ecac2700 1 -- 172.21.15.195:0/1025604013 wait complete. 2022-06-27T11:56:44.782 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 31 2022-06-27T11:56:44.789 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-06-27T11:56:46.791 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:56:46.945 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:46.942+0000 7fa5a22c4700 1 Processor -- start 2022-06-27T11:56:46.945 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:46.943+0000 7fa5a22c4700 1 -- start start 2022-06-27T11:56:46.946 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:46.943+0000 7fa5a22c4700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fa59c150580 0x7fa59c156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:56:46.946 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:46.943+0000 7fa5a22c4700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa59c150950 0x7fa59c0c7b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:56:46.947 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:46.943+0000 7fa5a22c4700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fa59c06f620 0x7fa59c06d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:56:46.947 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:46.943+0000 7fa5a22c4700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa59c157430 con 0x7fa59c150580 2022-06-27T11:56:46.948 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:46.943+0000 7fa5a22c4700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa59c0c76b0 con 0x7fa59c06f620 2022-06-27T11:56:46.948 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:46.943+0000 7fa5a22c4700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa59c0bad70 con 0x7fa59c150950 2022-06-27T11:56:46.953 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:46.943+0000 7fa59bfff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fa59c150580 0x7fa59c156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:56:46.953 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:46.944+0000 7fa59bfff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fa59c150580 0x7fa59c156ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:44902/0 (socket says 172.21.15.195:44902) 2022-06-27T11:56:46.954 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:46.944+0000 7fa59bfff700 1 -- 172.21.15.195:0/2116760167 learned_addr learned my addr 172.21.15.195:0/2116760167 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:56:46.954 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:46.944+0000 7fa5a0861700 1 --2- 172.21.15.195:0/2116760167 >> v2:172.21.15.202:3300/0 conn(0x7fa59c06f620 0x7fa59c06d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:56:46.966 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:46.944+0000 7fa59b7fe700 1 --2- 172.21.15.195:0/2116760167 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa59c150950 0x7fa59c0c7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:56:46.967 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:46.944+0000 7fa59bfff700 1 -- 172.21.15.195:0/2116760167 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa59c150950 msgr2=0x7fa59c0c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:56:46.967 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:46.944+0000 7fa59bfff700 1 --2- 172.21.15.195:0/2116760167 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa59c150950 0x7fa59c0c7b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:46.970 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:46.944+0000 7fa59bfff700 1 -- 172.21.15.195:0/2116760167 >> v2:172.21.15.202:3300/0 conn(0x7fa59c06f620 msgr2=0x7fa59c06d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:56:46.971 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:46.944+0000 7fa59bfff700 1 --2- 172.21.15.195:0/2116760167 >> v2:172.21.15.202:3300/0 conn(0x7fa59c06f620 0x7fa59c06d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:46.971 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:46.944+0000 7fa59bfff700 1 -- 172.21.15.195:0/2116760167 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa59c0ba4d0 con 0x7fa59c150580 2022-06-27T11:56:46.971 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:46.944+0000 7fa5a0861700 1 --2- 172.21.15.195:0/2116760167 >> v2:172.21.15.202:3300/0 conn(0x7fa59c06f620 0x7fa59c06d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T11:56:46.972 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:46.944+0000 7fa59b7fe700 1 --2- 172.21.15.195:0/2116760167 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa59c150950 0x7fa59c0c7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T11:56:46.972 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:46.944+0000 7fa59bfff700 1 --2- 172.21.15.195:0/2116760167 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa59c150580 0x7fa59c156ef0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fa59c069430 tx=0x7fa590009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f80434e3d8608c0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:56:46.973 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:46.944+0000 7fa59affd700 1 -- 172.21.15.195:0/2116760167 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa59000b640 con 0x7fa59c150580 2022-06-27T11:56:46.973 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:46.944+0000 7fa59affd700 1 -- 172.21.15.195:0/2116760167 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa59000b7a0 con 0x7fa59c150580 2022-06-27T11:56:46.974 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:46.944+0000 7fa59affd700 1 -- 172.21.15.195:0/2116760167 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa59000bb40 con 0x7fa59c150580 2022-06-27T11:56:46.974 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:46.945+0000 7fa5a22c4700 1 -- 172.21.15.195:0/2116760167 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa59c150580 msgr2=0x7fa59c156ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:56:46.975 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:46.945+0000 7fa5a22c4700 1 --2- 172.21.15.195:0/2116760167 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa59c150580 0x7fa59c156ef0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fa59c069430 tx=0x7fa590009120 comp rx=0 tx=0).stop 2022-06-27T11:56:46.975 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:46.945+0000 7fa5a22c4700 1 -- 172.21.15.195:0/2116760167 shutdown_connections 2022-06-27T11:56:46.976 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:46.945+0000 7fa5a22c4700 1 --2- 172.21.15.195:0/2116760167 >> v2:172.21.15.202:3300/0 conn(0x7fa59c06f620 0x7fa59c06d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:46.976 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:46.945+0000 7fa5a22c4700 1 --2- 172.21.15.195:0/2116760167 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa59c150580 0x7fa59c156ef0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:46.977 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:46.945+0000 7fa5a22c4700 1 --2- 172.21.15.195:0/2116760167 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa59c150950 0x7fa59c0c7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:46.979 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:46.945+0000 7fa5a22c4700 1 -- 172.21.15.195:0/2116760167 >> 172.21.15.195:0/2116760167 conn(0x7fa59c063af0 msgr2=0x7fa59c063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:56:46.979 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:46.945+0000 7fa5a22c4700 1 -- 172.21.15.195:0/2116760167 shutdown_connections 2022-06-27T11:56:46.979 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:46.945+0000 7fa5a22c4700 1 -- 172.21.15.195:0/2116760167 wait complete. 2022-06-27T11:56:46.979 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:46.946+0000 7fa5a22c4700 1 Processor -- start 2022-06-27T11:56:46.980 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:46.946+0000 7fa5a22c4700 1 -- start start 2022-06-27T11:56:46.980 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:46.947+0000 7fa5a22c4700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa59c06f620 0x7fa59c14d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:56:46.980 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:46.947+0000 7fa5a22c4700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa59c150580 0x7fa59c166b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:56:46.980 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:46.947+0000 7fa5a22c4700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa59c150950 0x7fa59c142da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:56:46.981 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:46.947+0000 7fa5a22c4700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa59c1666c0 con 0x7fa59c150950 2022-06-27T11:56:46.981 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:46.947+0000 7fa5a22c4700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa59c16a450 con 0x7fa59c150580 2022-06-27T11:56:46.981 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:46.947+0000 7fa5a22c4700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa59c16a6f0 con 0x7fa59c06f620 2022-06-27T11:56:46.982 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:46.947+0000 7fa5a0861700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa59c150950 0x7fa59c142da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:56:46.982 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:46.947+0000 7fa5a0861700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa59c150950 0x7fa59c142da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:44910/0 (socket says 172.21.15.195:44910) 2022-06-27T11:56:46.982 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:46.947+0000 7fa59bfff700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa59c06f620 0x7fa59c14d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:56:46.982 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:46.947+0000 7fa5a0861700 1 -- 172.21.15.195:0/1468854756 learned_addr learned my addr 172.21.15.195:0/1468854756 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:56:46.983 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:46.947+0000 7fa59b7fe700 1 --2- 172.21.15.195:0/1468854756 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa59c150580 0x7fa59c166b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:56:46.983 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:46.947+0000 7fa5a0861700 1 -- 172.21.15.195:0/1468854756 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa59c06f620 msgr2=0x7fa59c14d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:56:46.983 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:46.947+0000 7fa5a0861700 1 --2- 172.21.15.195:0/1468854756 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa59c06f620 0x7fa59c14d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:46.983 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:46.947+0000 7fa5a0861700 1 -- 172.21.15.195:0/1468854756 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa59c150580 msgr2=0x7fa59c166b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:56:46.983 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:46.947+0000 7fa5a0861700 1 --2- 172.21.15.195:0/1468854756 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa59c150580 0x7fa59c166b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:46.984 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:46.947+0000 7fa5a0861700 1 -- 172.21.15.195:0/1468854756 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5900093a0 con 0x7fa59c150950 2022-06-27T11:56:46.984 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:46.948+0000 7fa5a0861700 1 --2- 172.21.15.195:0/1468854756 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa59c150950 0x7fa59c142da0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fa59400cef0 tx=0x7fa594007850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:56:46.984 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:46.948+0000 7fa583fff700 1 -- 172.21.15.195:0/1468854756 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa59400ad80 con 0x7fa59c150950 2022-06-27T11:56:46.984 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:46.948+0000 7fa583fff700 1 -- 172.21.15.195:0/1468854756 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5940038a0 con 0x7fa59c150950 2022-06-27T11:56:46.985 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:46.948+0000 7fa5a22c4700 1 -- 172.21.15.195:0/1468854756 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa59c166370 con 0x7fa59c150950 2022-06-27T11:56:46.985 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:46.948+0000 7fa583fff700 1 -- 172.21.15.195:0/1468854756 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa594003c60 con 0x7fa59c150950 2022-06-27T11:56:46.985 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:46.948+0000 7fa5a22c4700 1 -- 172.21.15.195:0/1468854756 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa59c14b320 con 0x7fa59c150950 2022-06-27T11:56:46.986 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:46.949+0000 7fa5a22c4700 1 -- 172.21.15.195:0/1468854756 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa59c166870 con 0x7fa59c150950 2022-06-27T11:56:46.986 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:46.949+0000 7fa583fff700 1 -- 172.21.15.195:0/1468854756 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fa594004230 con 0x7fa59c150950 2022-06-27T11:56:46.986 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:46.949+0000 7fa583fff700 1 --2- 172.21.15.195:0/1468854756 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa57c063d00 0x7fa57c0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:56:46.986 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:46.949+0000 7fa583fff700 1 -- 172.21.15.195:0/1468854756 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fa594012070 con 0x7fa59c150950 2022-06-27T11:56:46.987 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:46.950+0000 7fa59bfff700 1 --2- 172.21.15.195:0/1468854756 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa57c063d00 0x7fa57c0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:56:46.987 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:46.950+0000 7fa59bfff700 1 --2- 172.21.15.195:0/1468854756 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa57c063d00 0x7fa57c0661b0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fa590009160 tx=0x7fa590017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:56:46.987 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:46.954+0000 7fa583fff700 1 -- 172.21.15.195:0/1468854756 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fa59c166870 con 0x7fa59c150950 2022-06-27T11:56:47.185 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:47.184+0000 7fa5a22c4700 1 -- 172.21.15.195:0/1468854756 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa59c166870 con 0x7fa59c150950 2022-06-27T11:56:47.186 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:47.185+0000 7fa583fff700 1 -- 172.21.15.195:0/1468854756 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+6128 (secure 0 0 0) 0x7fa59c166870 con 0x7fa59c150950 2022-06-27T11:56:47.187 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:56:47.187 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/4154570751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":4154570751},{"type":"v1","addr":"172.21.15.202:6833","nonce":4154570751}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:55:37.983652+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4856},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4856":{"gid":4856,"name":"d","rank":0,"incarnation":26,"state":"up:active","state_seq":43,"addr":"172.21.15.202:6835/2970948376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2970948376},{"type":"v1","addr":"172.21.15.202:6835","nonce":2970948376}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:56:47.189 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:47.187+0000 7fa5a22c4700 1 -- 172.21.15.195:0/1468854756 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa57c063d00 msgr2=0x7fa57c0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:56:47.190 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:47.187+0000 7fa5a22c4700 1 --2- 172.21.15.195:0/1468854756 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa57c063d00 0x7fa57c0661b0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fa590009160 tx=0x7fa590017000 comp rx=0 tx=0).stop 2022-06-27T11:56:47.190 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:47.187+0000 7fa5a22c4700 1 -- 172.21.15.195:0/1468854756 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa59c150950 msgr2=0x7fa59c142da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:56:47.190 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:47.187+0000 7fa5a22c4700 1 --2- 172.21.15.195:0/1468854756 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa59c150950 0x7fa59c142da0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fa59400cef0 tx=0x7fa594007850 comp rx=0 tx=0).stop 2022-06-27T11:56:47.191 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:47.187+0000 7fa5a22c4700 1 -- 172.21.15.195:0/1468854756 shutdown_connections 2022-06-27T11:56:47.192 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:47.187+0000 7fa5a22c4700 1 --2- 172.21.15.195:0/1468854756 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa57c063d00 0x7fa57c0661b0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:47.194 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:47.187+0000 7fa5a22c4700 1 --2- 172.21.15.195:0/1468854756 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa59c150950 0x7fa59c142da0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:47.195 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:47.187+0000 7fa5a22c4700 1 --2- 172.21.15.195:0/1468854756 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa59c06f620 0x7fa59c14d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:47.195 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:47.187+0000 7fa5a22c4700 1 --2- 172.21.15.195:0/1468854756 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa59c150580 0x7fa59c166b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:47.196 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:47.187+0000 7fa5a22c4700 1 -- 172.21.15.195:0/1468854756 >> 172.21.15.195:0/1468854756 conn(0x7fa59c063af0 msgr2=0x7fa59c156a30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:56:47.196 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:47.187+0000 7fa5a22c4700 1 -- 172.21.15.195:0/1468854756 shutdown_connections 2022-06-27T11:56:47.197 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:47.187+0000 7fa5a22c4700 1 -- 172.21.15.195:0/1468854756 wait complete. 2022-06-27T11:56:47.197 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 31 2022-06-27T11:56:47.206 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-06-27T11:56:49.209 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:56:49.344 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.344+0000 7fe35b85b700 1 Processor -- start 2022-06-27T11:56:49.345 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.345+0000 7fe35b85b700 1 -- start start 2022-06-27T11:56:49.348 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.345+0000 7fe35b85b700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe354067ed0 0x7fe3540c7440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:56:49.349 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.346+0000 7fe35b85b700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3540c43d0 con 0x7fe354156670 2022-06-27T11:56:49.349 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.346+0000 7fe35b85b700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe354068430 con 0x7fe354069390 2022-06-27T11:56:49.349 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.346+0000 7fe35b85b700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe354068710 con 0x7fe354067ed0 2022-06-27T11:56:49.350 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.346+0000 7fe3595f7700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fe354156670 0x7fe3540c4860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:51110/0 (socket says 172.21.15.195:51110) 2022-06-27T11:56:49.350 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.346+0000 7fe3595f7700 1 -- 172.21.15.195:0/3935619091 learned_addr learned my addr 172.21.15.195:0/3935619091 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:56:49.350 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.346+0000 7fe343fff700 1 -- 172.21.15.195:0/3935619091 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1347144032 0 0) 0x7fe3540c43d0 con 0x7fe354156670 2022-06-27T11:56:49.350 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.346+0000 7fe343fff700 1 -- 172.21.15.195:0/3935619091 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe33c003430 con 0x7fe354156670 2022-06-27T11:56:49.350 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.346+0000 7fe358df6700 1 --2- 172.21.15.195:0/3935619091 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe354067ed0 0x7fe3540c7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:56:49.351 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.346+0000 7fe343fff700 1 -- 172.21.15.195:0/3935619091 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3756409719 0 0) 0x7fe33c003430 con 0x7fe354156670 2022-06-27T11:56:49.351 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.346+0000 7fe343fff700 1 -- 172.21.15.195:0/3935619091 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe354067ed0 msgr2=0x7fe3540c7440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:56:49.351 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.346+0000 7fe343fff700 1 --2- 172.21.15.195:0/3935619091 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe354067ed0 0x7fe3540c7440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:49.351 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.346+0000 7fe343fff700 1 -- 172.21.15.195:0/3935619091 >> v1:172.21.15.202:6789/0 conn(0x7fe354069390 legacy=0x7fe3541502b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:56:49.352 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.346+0000 7fe343fff700 1 -- 172.21.15.195:0/3935619091 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3540689f0 con 0x7fe354156670 2022-06-27T11:56:49.352 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.346+0000 7fe343fff700 1 -- 172.21.15.195:0/3935619091 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7fe344001470 con 0x7fe354156670 2022-06-27T11:56:49.352 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.346+0000 7fe343fff700 1 -- 172.21.15.195:0/3935619091 >> v1:172.21.15.195:6789/0 conn(0x7fe354156670 legacy=0x7fe3540c4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:56:49.352 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.346+0000 7fe343fff700 1 --2- 172.21.15.195:0/3935619091 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe33c003950 0x7fe33c003d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:56:49.353 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.346+0000 7fe343fff700 1 --2- 172.21.15.195:0/3935619091 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe354067ed0 0x7fe33c0046d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:56:49.353 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.346+0000 7fe343fff700 1 --2- 172.21.15.195:0/3935619091 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe33c004c10 0x7fe33c005080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:56:49.353 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.346+0000 7fe343fff700 1 -- 172.21.15.195:0/3935619091 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe354068710 con 0x7fe33c003950 2022-06-27T11:56:49.353 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.346+0000 7fe343fff700 1 -- 172.21.15.195:0/3935619091 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe33c00ba30 con 0x7fe33c004c10 2022-06-27T11:56:49.353 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.346+0000 7fe343fff700 1 -- 172.21.15.195:0/3935619091 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe33c00bbf0 con 0x7fe354067ed0 2022-06-27T11:56:49.354 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.346+0000 7fe3595f7700 1 --2- 172.21.15.195:0/3935619091 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe33c003950 0x7fe33c003d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:56:49.354 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.346+0000 7fe358df6700 1 --2- 172.21.15.195:0/3935619091 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe354067ed0 0x7fe33c0046d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:56:49.354 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.346+0000 7fe359df8700 1 --2- 172.21.15.195:0/3935619091 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe33c004c10 0x7fe33c005080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:56:49.354 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.347+0000 7fe3595f7700 1 -- 172.21.15.195:0/3935619091 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe354067ed0 msgr2=0x7fe33c0046d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:56:49.355 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.347+0000 7fe3595f7700 1 --2- 172.21.15.195:0/3935619091 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe354067ed0 0x7fe33c0046d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:49.355 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.347+0000 7fe3595f7700 1 -- 172.21.15.195:0/3935619091 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe33c004c10 msgr2=0x7fe33c005080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:56:49.355 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.347+0000 7fe3595f7700 1 --2- 172.21.15.195:0/3935619091 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe33c004c10 0x7fe33c005080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:49.355 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.347+0000 7fe3595f7700 1 -- 172.21.15.195:0/3935619091 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe33c003430 con 0x7fe33c003950 2022-06-27T11:56:49.356 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.347+0000 7fe3595f7700 1 --2- 172.21.15.195:0/3935619091 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe33c003950 0x7fe33c003d60 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fe34400c1e0 tx=0x7fe354156420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3290810c8fcf6a47 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:56:49.356 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.347+0000 7fe343fff700 1 -- 172.21.15.195:0/3935619091 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe34400cab0 con 0x7fe33c003950 2022-06-27T11:56:49.356 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.347+0000 7fe343fff700 1 -- 172.21.15.195:0/3935619091 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe34400cc10 con 0x7fe33c003950 2022-06-27T11:56:49.356 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.347+0000 7fe343fff700 1 -- 172.21.15.195:0/3935619091 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3440070d0 con 0x7fe33c003950 2022-06-27T11:56:49.356 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.348+0000 7fe35b85b700 1 -- 172.21.15.195:0/3935619091 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe33c003950 msgr2=0x7fe33c003d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:56:49.357 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.348+0000 7fe35b85b700 1 --2- 172.21.15.195:0/3935619091 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe33c003950 0x7fe33c003d60 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fe34400c1e0 tx=0x7fe354156420 comp rx=0 tx=0).stop 2022-06-27T11:56:49.357 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.348+0000 7fe359df8700 1 -- 172.21.15.195:0/3935619091 reap_dead start 2022-06-27T11:56:49.357 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.348+0000 7fe35b85b700 1 -- 172.21.15.195:0/3935619091 shutdown_connections 2022-06-27T11:56:49.357 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.348+0000 7fe35b85b700 1 -- 172.21.15.195:0/3935619091 >> 172.21.15.195:0/3935619091 conn(0x7fe354069e00 msgr2=0x7fe35406a200 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:56:49.358 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.348+0000 7fe35b85b700 1 -- 172.21.15.195:0/3935619091 shutdown_connections 2022-06-27T11:56:49.358 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.348+0000 7fe35b85b700 1 -- 172.21.15.195:0/3935619091 wait complete. 2022-06-27T11:56:49.358 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.349+0000 7fe35b85b700 1 Processor -- start 2022-06-27T11:56:49.358 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.349+0000 7fe35b85b700 1 -- start start 2022-06-27T11:56:49.362 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.350+0000 7fe35b85b700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe33c003950 0x7fe3540c2db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:56:49.362 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.350+0000 7fe35b85b700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe3540b19f0 0x7fe3540b1e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:56:49.362 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.350+0000 7fe35b85b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe3540c2100 0x7fe3540c2570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:56:49.363 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.350+0000 7fe35b85b700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3540b2360 con 0x7fe3540c2100 2022-06-27T11:56:49.363 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.350+0000 7fe35b85b700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3540c2ab0 con 0x7fe33c003950 2022-06-27T11:56:49.363 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.350+0000 7fe35b85b700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3540c36b0 con 0x7fe3540b19f0 2022-06-27T11:56:49.363 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.350+0000 7fe359df8700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe3540c2100 0x7fe3540c2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:56:49.364 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.350+0000 7fe359df8700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe3540c2100 0x7fe3540c2570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:44938/0 (socket says 172.21.15.195:44938) 2022-06-27T11:56:49.364 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.350+0000 7fe359df8700 1 -- 172.21.15.195:0/2191208366 learned_addr learned my addr 172.21.15.195:0/2191208366 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:56:49.364 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.350+0000 7fe3595f7700 1 --2- 172.21.15.195:0/2191208366 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe33c003950 0x7fe3540c2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:56:49.364 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.350+0000 7fe358df6700 1 --2- 172.21.15.195:0/2191208366 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe3540b19f0 0x7fe3540b1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:56:49.365 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.350+0000 7fe359df8700 1 -- 172.21.15.195:0/2191208366 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe3540b19f0 msgr2=0x7fe3540b1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:56:49.365 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.350+0000 7fe359df8700 1 --2- 172.21.15.195:0/2191208366 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe3540b19f0 0x7fe3540b1e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:49.365 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.350+0000 7fe359df8700 1 -- 172.21.15.195:0/2191208366 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe33c003950 msgr2=0x7fe3540c2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:56:49.365 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.350+0000 7fe359df8700 1 --2- 172.21.15.195:0/2191208366 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe33c003950 0x7fe3540c2db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:49.365 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.350+0000 7fe359df8700 1 -- 172.21.15.195:0/2191208366 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3540c3990 con 0x7fe3540c2100 2022-06-27T11:56:49.366 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.350+0000 7fe359df8700 1 --2- 172.21.15.195:0/2191208366 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe3540c2100 0x7fe3540c2570 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fe35408ad00 tx=0x7fe34c009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:56:49.366 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.350+0000 7fe3417fa700 1 -- 172.21.15.195:0/2191208366 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe34c003a90 con 0x7fe3540c2100 2022-06-27T11:56:49.366 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.350+0000 7fe3417fa700 1 -- 172.21.15.195:0/2191208366 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe34c003bf0 con 0x7fe3540c2100 2022-06-27T11:56:49.366 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.350+0000 7fe3417fa700 1 -- 172.21.15.195:0/2191208366 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe34c0050d0 con 0x7fe3540c2100 2022-06-27T11:56:49.367 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.350+0000 7fe35b85b700 1 -- 172.21.15.195:0/2191208366 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3540c3c70 con 0x7fe3540c2100 2022-06-27T11:56:49.367 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.351+0000 7fe35b85b700 1 -- 172.21.15.195:0/2191208366 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3540c1900 con 0x7fe3540c2100 2022-06-27T11:56:49.367 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.351+0000 7fe35b85b700 1 -- 172.21.15.195:0/2191208366 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3540722c0 con 0x7fe3540c2100 2022-06-27T11:56:49.367 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.352+0000 7fe3417fa700 1 -- 172.21.15.195:0/2191208366 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fe34c005520 con 0x7fe3540c2100 2022-06-27T11:56:49.368 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.352+0000 7fe3417fa700 1 --2- 172.21.15.195:0/2191208366 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe334063d00 0x7fe3340661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:56:49.368 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.352+0000 7fe3417fa700 1 -- 172.21.15.195:0/2191208366 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fe34c080900 con 0x7fe3540c2100 2022-06-27T11:56:49.368 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.352+0000 7fe3595f7700 1 --2- 172.21.15.195:0/2191208366 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe334063d00 0x7fe3340661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:56:49.368 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.353+0000 7fe3595f7700 1 --2- 172.21.15.195:0/2191208366 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe334063d00 0x7fe3340661b0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe344007eb0 tx=0x7fe344015000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:56:49.368 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.356+0000 7fe3417fa700 1 -- 172.21.15.195:0/2191208366 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fe34c04f360 con 0x7fe3540c2100 2022-06-27T11:56:49.580 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.580+0000 7fe35b85b700 1 -- 172.21.15.195:0/2191208366 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe354068cd0 con 0x7fe3540c2100 2022-06-27T11:56:49.581 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.580+0000 7fe3417fa700 1 -- 172.21.15.195:0/2191208366 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+6128 (secure 0 0 0) 0x7fe34c052980 con 0x7fe3540c2100 2022-06-27T11:56:49.581 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:56:49.583 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/4154570751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":4154570751},{"type":"v1","addr":"172.21.15.202:6833","nonce":4154570751}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:55:37.983652+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4856},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4856":{"gid":4856,"name":"d","rank":0,"incarnation":26,"state":"up:active","state_seq":43,"addr":"172.21.15.202:6835/2970948376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2970948376},{"type":"v1","addr":"172.21.15.202:6835","nonce":2970948376}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:56:49.585 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.582+0000 7fe35b85b700 1 -- 172.21.15.195:0/2191208366 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe334063d00 msgr2=0x7fe3340661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:56:49.587 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.582+0000 7fe35b85b700 1 --2- 172.21.15.195:0/2191208366 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe334063d00 0x7fe3340661b0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe344007eb0 tx=0x7fe344015000 comp rx=0 tx=0).stop 2022-06-27T11:56:49.588 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.582+0000 7fe35b85b700 1 -- 172.21.15.195:0/2191208366 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe3540c2100 msgr2=0x7fe3540c2570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:56:49.589 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.582+0000 7fe35b85b700 1 --2- 172.21.15.195:0/2191208366 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe3540c2100 0x7fe3540c2570 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fe35408ad00 tx=0x7fe34c009040 comp rx=0 tx=0).stop 2022-06-27T11:56:49.589 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.582+0000 7fe35b85b700 1 -- 172.21.15.195:0/2191208366 shutdown_connections 2022-06-27T11:56:49.590 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.582+0000 7fe35b85b700 1 --2- 172.21.15.195:0/2191208366 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe334063d00 0x7fe3340661b0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:49.590 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.582+0000 7fe35b85b700 1 --2- 172.21.15.195:0/2191208366 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe3540c2100 0x7fe3540c2570 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:49.591 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.582+0000 7fe35b85b700 1 --2- 172.21.15.195:0/2191208366 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe33c003950 0x7fe3540c2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:49.591 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.582+0000 7fe35b85b700 1 --2- 172.21.15.195:0/2191208366 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe3540b19f0 0x7fe3540b1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:49.592 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.582+0000 7fe35b85b700 1 -- 172.21.15.195:0/2191208366 >> 172.21.15.195:0/2191208366 conn(0x7fe354069e00 msgr2=0x7fe35414f590 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:56:49.592 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.582+0000 7fe35b85b700 1 -- 172.21.15.195:0/2191208366 shutdown_connections 2022-06-27T11:56:49.593 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:49.582+0000 7fe35b85b700 1 -- 172.21.15.195:0/2191208366 wait complete. 2022-06-27T11:56:49.593 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 31 2022-06-27T11:56:49.599 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-06-27T11:56:51.600 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:56:51.743 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.734+0000 7fae243cb700 1 Processor -- start 2022-06-27T11:56:51.744 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.734+0000 7fae243cb700 1 -- start start 2022-06-27T11:56:51.744 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.735+0000 7fae243cb700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fae1c150500 0x7fae1c063af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:56:51.744 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.735+0000 7fae243cb700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae1c064030 con 0x7fae1c06a450 2022-06-27T11:56:51.745 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.735+0000 7fae243cb700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae1c0c4160 con 0x7fae1c1508d0 2022-06-27T11:56:51.745 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.735+0000 7fae243cb700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae1c0c4440 con 0x7fae1c150500 2022-06-27T11:56:51.746 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.735+0000 7fae21966700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fae1c06a450 0x7fae1c0c48d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:51130/0 (socket says 172.21.15.195:51130) 2022-06-27T11:56:51.747 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.735+0000 7fae21966700 1 -- 172.21.15.195:0/1892430552 learned_addr learned my addr 172.21.15.195:0/1892430552 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:56:51.748 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.736+0000 7fae22167700 1 --2- 172.21.15.195:0/1892430552 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fae1c150500 0x7fae1c063af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:56:51.748 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.736+0000 7fae21165700 1 -- 172.21.15.195:0/1892430552 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3415181413 0 0) 0x7fae18001110 con 0x7fae1c06a450 2022-06-27T11:56:51.748 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.736+0000 7fae21165700 1 -- 172.21.15.195:0/1892430552 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae04003430 con 0x7fae1c06a450 2022-06-27T11:56:51.749 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.736+0000 7fae21165700 1 -- 172.21.15.195:0/1892430552 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2352727504 0 0) 0x7fae1c0c4160 con 0x7fae1c1508d0 2022-06-27T11:56:51.749 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.736+0000 7fae21165700 1 -- 172.21.15.195:0/1892430552 --> v1:172.21.15.202:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae18001110 con 0x7fae1c1508d0 2022-06-27T11:56:51.749 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.736+0000 7fae22167700 1 -- 172.21.15.195:0/1892430552 >> v1:172.21.15.202:6789/0 conn(0x7fae1c1508d0 legacy=0x7fae1c0bbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:56:51.749 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.736+0000 7fae22167700 1 -- 172.21.15.195:0/1892430552 >> v1:172.21.15.195:6789/0 conn(0x7fae1c06a450 legacy=0x7fae1c0c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:56:51.750 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.736+0000 7fae22167700 1 -- 172.21.15.195:0/1892430552 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae1c15a800 con 0x7fae1c150500 2022-06-27T11:56:51.750 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.736+0000 7fae22167700 1 --2- 172.21.15.195:0/1892430552 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fae1c150500 0x7fae1c063af0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fae0c009750 tx=0x7fae0c009a60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e5b968d24844e32 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:56:51.750 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.736+0000 7fae21165700 1 -- 172.21.15.195:0/1892430552 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae0c005ad0 con 0x7fae1c150500 2022-06-27T11:56:51.750 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.736+0000 7fae21165700 1 -- 172.21.15.195:0/1892430552 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae0c005c30 con 0x7fae1c150500 2022-06-27T11:56:51.751 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.736+0000 7fae21165700 1 -- 172.21.15.195:0/1892430552 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae0c006030 con 0x7fae1c150500 2022-06-27T11:56:51.751 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.737+0000 7fae243cb700 1 -- 172.21.15.195:0/1892430552 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fae1c150500 msgr2=0x7fae1c063af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:56:51.751 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.737+0000 7fae243cb700 1 --2- 172.21.15.195:0/1892430552 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fae1c150500 0x7fae1c063af0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fae0c009750 tx=0x7fae0c009a60 comp rx=0 tx=0).stop 2022-06-27T11:56:51.751 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.737+0000 7fae243cb700 1 -- 172.21.15.195:0/1892430552 shutdown_connections 2022-06-27T11:56:51.751 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.737+0000 7fae243cb700 1 --2- 172.21.15.195:0/1892430552 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fae1c150500 0x7fae1c063af0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:51.752 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.737+0000 7fae243cb700 1 -- 172.21.15.195:0/1892430552 >> 172.21.15.195:0/1892430552 conn(0x7fae1c0c9b00 msgr2=0x7fae1c0c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:56:51.752 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.737+0000 7fae243cb700 1 -- 172.21.15.195:0/1892430552 shutdown_connections 2022-06-27T11:56:51.752 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.737+0000 7fae243cb700 1 -- 172.21.15.195:0/1892430552 wait complete. 2022-06-27T11:56:51.752 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.738+0000 7fae243cb700 1 Processor -- start 2022-06-27T11:56:51.753 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.738+0000 7fae243cb700 1 -- start start 2022-06-27T11:56:51.753 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.739+0000 7fae243cb700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fae1c06a450 0x7fae1c166200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:56:51.753 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.739+0000 7fae243cb700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fae1c150500 0x7fae1c14ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:56:51.754 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.739+0000 7fae243cb700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fae1c1508d0 0x7fae1c1697d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:56:51.760 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.739+0000 7fae243cb700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae1c166740 con 0x7fae1c1508d0 2022-06-27T11:56:51.761 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.739+0000 7fae243cb700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae1c1592f0 con 0x7fae1c06a450 2022-06-27T11:56:51.761 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.739+0000 7fae243cb700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae1c14cb10 con 0x7fae1c150500 2022-06-27T11:56:51.761 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.739+0000 7fae22968700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fae1c1508d0 0x7fae1c1697d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:56:51.761 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.739+0000 7fae22167700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fae1c06a450 0x7fae1c166200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:56:51.762 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.739+0000 7fae21966700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fae1c150500 0x7fae1c14ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:56:51.762 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.739+0000 7fae22968700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fae1c1508d0 0x7fae1c1697d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:44952/0 (socket says 172.21.15.195:44952) 2022-06-27T11:56:51.762 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.739+0000 7fae22167700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fae1c06a450 0x7fae1c166200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3300/0 says I am v2:172.21.15.195:43074/0 (socket says 172.21.15.195:43074) 2022-06-27T11:56:51.762 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.739+0000 7fae22968700 1 -- 172.21.15.195:0/1556249463 learned_addr learned my addr 172.21.15.195:0/1556249463 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:56:51.763 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.739+0000 7fae22968700 1 -- 172.21.15.195:0/1556249463 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fae1c150500 msgr2=0x7fae1c14ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:56:51.763 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.739+0000 7fae22968700 1 --2- 172.21.15.195:0/1556249463 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fae1c150500 0x7fae1c14ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:51.763 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.739+0000 7fae22968700 1 -- 172.21.15.195:0/1556249463 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fae1c06a450 msgr2=0x7fae1c166200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:56:51.763 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.739+0000 7fae22968700 1 --2- 172.21.15.195:0/1556249463 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fae1c06a450 0x7fae1c166200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:51.764 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.739+0000 7fae22968700 1 -- 172.21.15.195:0/1556249463 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae0c009290 con 0x7fae1c1508d0 2022-06-27T11:56:51.764 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.739+0000 7fae22968700 1 --2- 172.21.15.195:0/1556249463 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fae1c1508d0 0x7fae1c1697d0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fae1400c790 tx=0x7fae140015b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:56:51.764 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.739+0000 7fae0a7fc700 1 -- 172.21.15.195:0/1556249463 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae14003880 con 0x7fae1c1508d0 2022-06-27T11:56:51.766 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.739+0000 7fae0a7fc700 1 -- 172.21.15.195:0/1556249463 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae140039e0 con 0x7fae1c1508d0 2022-06-27T11:56:51.767 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.739+0000 7fae0a7fc700 1 -- 172.21.15.195:0/1556249463 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae14003da0 con 0x7fae1c1508d0 2022-06-27T11:56:51.769 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.740+0000 7fae243cb700 1 -- 172.21.15.195:0/1556249463 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae1c0c04d0 con 0x7fae1c1508d0 2022-06-27T11:56:51.769 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.740+0000 7fae243cb700 1 -- 172.21.15.195:0/1556249463 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae1c158e80 con 0x7fae1c1508d0 2022-06-27T11:56:51.769 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.740+0000 7fae243cb700 1 -- 172.21.15.195:0/1556249463 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae1c0722c0 con 0x7fae1c1508d0 2022-06-27T11:56:51.769 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.741+0000 7fae0a7fc700 1 -- 172.21.15.195:0/1556249463 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fae14019030 con 0x7fae1c1508d0 2022-06-27T11:56:51.770 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.741+0000 7fae0a7fc700 1 --2- 172.21.15.195:0/1556249463 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fadfc063d00 0x7fadfc0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:56:51.770 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.741+0000 7fae0a7fc700 1 -- 172.21.15.195:0/1556249463 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fae1407fdf0 con 0x7fae1c1508d0 2022-06-27T11:56:51.770 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.741+0000 7fae22167700 1 --2- 172.21.15.195:0/1556249463 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fadfc063d00 0x7fadfc0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:56:51.770 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.742+0000 7fae22167700 1 --2- 172.21.15.195:0/1556249463 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fadfc063d00 0x7fadfc0661b0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fae0c009aa0 tx=0x7fae0c016000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:56:51.771 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.746+0000 7fae0a7fc700 1 -- 172.21.15.195:0/1556249463 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fae1404e7f0 con 0x7fae1c1508d0 2022-06-27T11:56:51.973 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.970+0000 7fae243cb700 1 -- 172.21.15.195:0/1556249463 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae1c0ba8e0 con 0x7fae1c1508d0 2022-06-27T11:56:51.974 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.971+0000 7fae0a7fc700 1 -- 172.21.15.195:0/1556249463 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+6128 (secure 0 0 0) 0x7fae14017020 con 0x7fae1c1508d0 2022-06-27T11:56:51.974 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:56:51.975 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/4154570751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":4154570751},{"type":"v1","addr":"172.21.15.202:6833","nonce":4154570751}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:55:37.983652+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4856},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4856":{"gid":4856,"name":"d","rank":0,"incarnation":26,"state":"up:active","state_seq":43,"addr":"172.21.15.202:6835/2970948376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2970948376},{"type":"v1","addr":"172.21.15.202:6835","nonce":2970948376}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:56:51.975 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.973+0000 7fae243cb700 1 -- 172.21.15.195:0/1556249463 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fadfc063d00 msgr2=0x7fadfc0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:56:51.976 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.973+0000 7fae243cb700 1 --2- 172.21.15.195:0/1556249463 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fadfc063d00 0x7fadfc0661b0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fae0c009aa0 tx=0x7fae0c016000 comp rx=0 tx=0).stop 2022-06-27T11:56:51.976 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.973+0000 7fae243cb700 1 -- 172.21.15.195:0/1556249463 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fae1c1508d0 msgr2=0x7fae1c1697d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:56:51.976 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.973+0000 7fae243cb700 1 --2- 172.21.15.195:0/1556249463 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fae1c1508d0 0x7fae1c1697d0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fae1400c790 tx=0x7fae140015b0 comp rx=0 tx=0).stop 2022-06-27T11:56:51.976 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.973+0000 7fae243cb700 1 -- 172.21.15.195:0/1556249463 shutdown_connections 2022-06-27T11:56:51.977 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.973+0000 7fae243cb700 1 --2- 172.21.15.195:0/1556249463 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fadfc063d00 0x7fadfc0661b0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:51.979 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.973+0000 7fae243cb700 1 --2- 172.21.15.195:0/1556249463 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fae1c1508d0 0x7fae1c1697d0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:51.979 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.973+0000 7fae243cb700 1 --2- 172.21.15.195:0/1556249463 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fae1c06a450 0x7fae1c166200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:51.979 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.973+0000 7fae243cb700 1 --2- 172.21.15.195:0/1556249463 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fae1c150500 0x7fae1c14ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:51.980 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.973+0000 7fae243cb700 1 -- 172.21.15.195:0/1556249463 >> 172.21.15.195:0/1556249463 conn(0x7fae1c0c9b00 msgr2=0x7fae1c0c7f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:56:51.980 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.973+0000 7fae243cb700 1 -- 172.21.15.195:0/1556249463 shutdown_connections 2022-06-27T11:56:51.980 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:51.973+0000 7fae243cb700 1 -- 172.21.15.195:0/1556249463 wait complete. 2022-06-27T11:56:51.980 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 31 2022-06-27T11:56:51.986 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-06-27T11:56:53.987 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:56:54.122 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.122+0000 7fd34aee3700 1 Processor -- start 2022-06-27T11:56:54.123 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.122+0000 7fd34aee3700 1 -- start start 2022-06-27T11:56:54.123 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.123+0000 7fd34aee3700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fd34406fef0 0x7fd3441570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:56:54.123 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.123+0000 7fd34aee3700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd3440c34a0 0x7fd3440c2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:56:54.125 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.123+0000 7fd34aee3700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd34406fba0 con 0x7fd34406e850 2022-06-27T11:56:54.125 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.123+0000 7fd34aee3700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3441575e0 con 0x7fd34406fef0 2022-06-27T11:56:54.125 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.123+0000 7fd34aee3700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd34406f5e0 con 0x7fd3440c34a0 2022-06-27T11:56:54.134 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.123+0000 7fd343fff700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fd34406fef0 0x7fd3441570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:56:54.135 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.123+0000 7fd343fff700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fd34406fef0 0x7fd3441570a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3300/0 says I am v2:172.21.15.195:43086/0 (socket says 172.21.15.195:43086) 2022-06-27T11:56:54.135 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.123+0000 7fd343fff700 1 -- 172.21.15.195:0/1034896235 learned_addr learned my addr 172.21.15.195:0/1034896235 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:56:54.136 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.123+0000 7fd348c7f700 1 --1- 172.21.15.195:0/1034896235 >> v1:172.21.15.202:6789/0 conn(0x7fd34406e850 0x7fd344069de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.202:6789/0 says I am v1:172.21.15.195:53006/0 (socket says 172.21.15.195:53006) 2022-06-27T11:56:54.136 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.123+0000 7fd349480700 1 --2- 172.21.15.195:0/1034896235 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd3440c34a0 0x7fd3440c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:56:54.136 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.124+0000 7fd349480700 1 -- 172.21.15.195:0/1034896235 >> v2:172.21.15.202:3300/0 conn(0x7fd34406fef0 msgr2=0x7fd3441570a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:56:54.137 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.124+0000 7fd349480700 1 --2- 172.21.15.195:0/1034896235 >> v2:172.21.15.202:3300/0 conn(0x7fd34406fef0 0x7fd3441570a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:54.137 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.124+0000 7fd349480700 1 -- 172.21.15.195:0/1034896235 >> v1:172.21.15.202:6789/0 conn(0x7fd34406e850 legacy=0x7fd344069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:56:54.138 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.124+0000 7fd349480700 1 -- 172.21.15.195:0/1034896235 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd34406f8c0 con 0x7fd3440c34a0 2022-06-27T11:56:54.138 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.124+0000 7fd3437fe700 1 -- 172.21.15.195:0/1034896235 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4255922253 0 0) 0x7fd34406fba0 con 0x7fd34406e850 2022-06-27T11:56:54.138 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.124+0000 7fd343fff700 1 --2- 172.21.15.195:0/1034896235 >> v2:172.21.15.202:3300/0 conn(0x7fd34406fef0 0x7fd3441570a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:56:54.139 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.124+0000 7fd349480700 1 --2- 172.21.15.195:0/1034896235 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd3440c34a0 0x7fd3440c2920 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd33c009e60 tx=0x7fd33c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=349d2776a09fc505 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:56:54.139 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.124+0000 7fd3437fe700 1 -- 172.21.15.195:0/1034896235 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd33c00e070 con 0x7fd3440c34a0 2022-06-27T11:56:54.139 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.124+0000 7fd3437fe700 1 -- 172.21.15.195:0/1034896235 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd33c00bbf0 con 0x7fd3440c34a0 2022-06-27T11:56:54.140 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.124+0000 7fd3437fe700 1 -- 172.21.15.195:0/1034896235 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd33c012430 con 0x7fd3440c34a0 2022-06-27T11:56:54.140 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.125+0000 7fd34aee3700 1 -- 172.21.15.195:0/1034896235 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd3440c34a0 msgr2=0x7fd3440c2920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:56:54.140 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.125+0000 7fd34aee3700 1 --2- 172.21.15.195:0/1034896235 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd3440c34a0 0x7fd3440c2920 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd33c009e60 tx=0x7fd33c009780 comp rx=0 tx=0).stop 2022-06-27T11:56:54.140 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.125+0000 7fd34aee3700 1 -- 172.21.15.195:0/1034896235 shutdown_connections 2022-06-27T11:56:54.140 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.125+0000 7fd34aee3700 1 --2- 172.21.15.195:0/1034896235 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd3440c34a0 0x7fd3440c2920 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:54.141 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.125+0000 7fd34aee3700 1 --2- 172.21.15.195:0/1034896235 >> v2:172.21.15.202:3300/0 conn(0x7fd34406fef0 0x7fd3441570a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:54.141 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.125+0000 7fd34aee3700 1 -- 172.21.15.195:0/1034896235 >> 172.21.15.195:0/1034896235 conn(0x7fd3440c6350 msgr2=0x7fd3440c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:56:54.141 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.125+0000 7fd34aee3700 1 -- 172.21.15.195:0/1034896235 shutdown_connections 2022-06-27T11:56:54.141 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.125+0000 7fd34aee3700 1 -- 172.21.15.195:0/1034896235 wait complete. 2022-06-27T11:56:54.142 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.126+0000 7fd34aee3700 1 Processor -- start 2022-06-27T11:56:54.142 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.126+0000 7fd34aee3700 1 -- start start 2022-06-27T11:56:54.142 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.127+0000 7fd34aee3700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd34406e850 0x7fd344162b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:56:54.143 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.127+0000 7fd34aee3700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd3440c34a0 0x7fd3440b7e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:56:54.143 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.127+0000 7fd34aee3700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd344165590 0x7fd344069ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:56:54.143 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.127+0000 7fd34aee3700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3440ba470 con 0x7fd3440c34a0 2022-06-27T11:56:54.143 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.127+0000 7fd34aee3700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd344162770 con 0x7fd34406e850 2022-06-27T11:56:54.144 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.127+0000 7fd34aee3700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd344165aa0 con 0x7fd344165590 2022-06-27T11:56:54.144 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.127+0000 7fd343fff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd3440c34a0 0x7fd3440b7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:56:54.144 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.127+0000 7fd343fff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd3440c34a0 0x7fd3440b7e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:44966/0 (socket says 172.21.15.195:44966) 2022-06-27T11:56:54.144 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.127+0000 7fd343fff700 1 -- 172.21.15.195:0/564214048 learned_addr learned my addr 172.21.15.195:0/564214048 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:56:54.145 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.127+0000 7fd348c7f700 1 --2- 172.21.15.195:0/564214048 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd34406e850 0x7fd344162b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:56:54.145 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.127+0000 7fd349480700 1 --2- 172.21.15.195:0/564214048 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd344165590 0x7fd344069ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:56:54.145 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.127+0000 7fd343fff700 1 -- 172.21.15.195:0/564214048 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd344165590 msgr2=0x7fd344069ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:56:54.146 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.127+0000 7fd343fff700 1 --2- 172.21.15.195:0/564214048 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd344165590 0x7fd344069ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:54.146 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.127+0000 7fd343fff700 1 -- 172.21.15.195:0/564214048 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd34406e850 msgr2=0x7fd344162b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:56:54.146 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.127+0000 7fd343fff700 1 --2- 172.21.15.195:0/564214048 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd34406e850 0x7fd344162b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:54.146 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.127+0000 7fd343fff700 1 -- 172.21.15.195:0/564214048 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd33000b650 con 0x7fd3440c34a0 2022-06-27T11:56:54.147 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.127+0000 7fd343fff700 1 --2- 172.21.15.195:0/564214048 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd3440c34a0 0x7fd3440b7e30 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fd330010ee0 tx=0x7fd33000b3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:56:54.149 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.128+0000 7fd340ff9700 1 -- 172.21.15.195:0/564214048 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd33000ee10 con 0x7fd3440c34a0 2022-06-27T11:56:54.150 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.128+0000 7fd340ff9700 1 -- 172.21.15.195:0/564214048 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd330011450 con 0x7fd3440c34a0 2022-06-27T11:56:54.150 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.128+0000 7fd34aee3700 1 -- 172.21.15.195:0/564214048 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd33c009a20 con 0x7fd3440c34a0 2022-06-27T11:56:54.150 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.128+0000 7fd340ff9700 1 -- 172.21.15.195:0/564214048 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3300117f0 con 0x7fd3440c34a0 2022-06-27T11:56:54.151 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.128+0000 7fd34aee3700 1 -- 172.21.15.195:0/564214048 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd344166040 con 0x7fd3440c34a0 2022-06-27T11:56:54.151 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.129+0000 7fd34aee3700 1 -- 172.21.15.195:0/564214048 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3440722c0 con 0x7fd3440c34a0 2022-06-27T11:56:54.151 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.129+0000 7fd340ff9700 1 -- 172.21.15.195:0/564214048 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fd330020030 con 0x7fd3440c34a0 2022-06-27T11:56:54.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.129+0000 7fd340ff9700 1 --2- 172.21.15.195:0/564214048 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd32c063d30 0x7fd32c0661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:56:54.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.129+0000 7fd340ff9700 1 -- 172.21.15.195:0/564214048 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fd3300834d0 con 0x7fd3440c34a0 2022-06-27T11:56:54.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.129+0000 7fd348c7f700 1 --2- 172.21.15.195:0/564214048 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd32c063d30 0x7fd32c0661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:56:54.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.130+0000 7fd348c7f700 1 --2- 172.21.15.195:0/564214048 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd32c063d30 0x7fd32c0661e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd338007fd0 tx=0x7fd338007d10 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:56:54.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.134+0000 7fd340ff9700 1 -- 172.21.15.195:0/564214048 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fd330051f30 con 0x7fd3440c34a0 2022-06-27T11:56:54.365 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.358+0000 7fd34aee3700 1 -- 172.21.15.195:0/564214048 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd3440c7860 con 0x7fd3440c34a0 2022-06-27T11:56:54.365 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.359+0000 7fd340ff9700 1 -- 172.21.15.195:0/564214048 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+6128 (secure 0 0 0) 0x7fd33001e020 con 0x7fd3440c34a0 2022-06-27T11:56:54.366 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:56:54.366 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/4154570751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":4154570751},{"type":"v1","addr":"172.21.15.202:6833","nonce":4154570751}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:55:37.983652+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4856},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4856":{"gid":4856,"name":"d","rank":0,"incarnation":26,"state":"up:active","state_seq":43,"addr":"172.21.15.202:6835/2970948376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2970948376},{"type":"v1","addr":"172.21.15.202:6835","nonce":2970948376}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:56:54.367 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.361+0000 7fd34aee3700 1 -- 172.21.15.195:0/564214048 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd32c063d30 msgr2=0x7fd32c0661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:56:54.368 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.361+0000 7fd34aee3700 1 --2- 172.21.15.195:0/564214048 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd32c063d30 0x7fd32c0661e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd338007fd0 tx=0x7fd338007d10 comp rx=0 tx=0).stop 2022-06-27T11:56:54.368 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.361+0000 7fd34aee3700 1 -- 172.21.15.195:0/564214048 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd3440c34a0 msgr2=0x7fd3440b7e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:56:54.368 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.361+0000 7fd34aee3700 1 --2- 172.21.15.195:0/564214048 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd3440c34a0 0x7fd3440b7e30 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fd330010ee0 tx=0x7fd33000b3b0 comp rx=0 tx=0).stop 2022-06-27T11:56:54.369 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.361+0000 7fd34aee3700 1 -- 172.21.15.195:0/564214048 shutdown_connections 2022-06-27T11:56:54.369 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.361+0000 7fd34aee3700 1 --2- 172.21.15.195:0/564214048 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd32c063d30 0x7fd32c0661e0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:54.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.361+0000 7fd34aee3700 1 --2- 172.21.15.195:0/564214048 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd344165590 0x7fd344069ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:54.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.361+0000 7fd34aee3700 1 --2- 172.21.15.195:0/564214048 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd34406e850 0x7fd344162b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:54.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.361+0000 7fd34aee3700 1 --2- 172.21.15.195:0/564214048 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd3440c34a0 0x7fd3440b7e30 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:54.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.361+0000 7fd34aee3700 1 -- 172.21.15.195:0/564214048 >> 172.21.15.195:0/564214048 conn(0x7fd3440c6350 msgr2=0x7fd3440c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:56:54.372 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.361+0000 7fd34aee3700 1 -- 172.21.15.195:0/564214048 shutdown_connections 2022-06-27T11:56:54.372 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:54.361+0000 7fd34aee3700 1 -- 172.21.15.195:0/564214048 wait complete. 2022-06-27T11:56:54.372 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 31 2022-06-27T11:56:54.374 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-06-27T11:56:56.375 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:56:56.510 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.510+0000 7fbfc6bc0700 1 Processor -- start 2022-06-27T11:56:56.510 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.510+0000 7fbfc6bc0700 1 -- start start 2022-06-27T11:56:56.523 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.511+0000 7fbfc6bc0700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbfc01508d0 0x7fbfc0063af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:56:56.523 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.511+0000 7fbfc6bc0700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfc0064030 con 0x7fbfc006a450 2022-06-27T11:56:56.524 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.511+0000 7fbfc6bc0700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfc00c4160 con 0x7fbfc0150500 2022-06-27T11:56:56.524 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.511+0000 7fbfc6bc0700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc00c4440 con 0x7fbfc01508d0 2022-06-27T11:56:56.524 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.511+0000 7fbfbffff700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fbfc006a450 0x7fbfc00bbf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:51162/0 (socket says 172.21.15.195:51162) 2022-06-27T11:56:56.524 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.511+0000 7fbfbffff700 1 -- 172.21.15.195:0/1417219921 learned_addr learned my addr 172.21.15.195:0/1417219921 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:56:56.525 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.511+0000 7fbfc515d700 1 --2- 172.21.15.195:0/1417219921 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbfc01508d0 0x7fbfc0063af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:56:56.525 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.511+0000 7fbfbf7fe700 1 -- 172.21.15.195:0/1417219921 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1586993155 0 0) 0x7fbfac001110 con 0x7fbfc006a450 2022-06-27T11:56:56.525 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.511+0000 7fbfbf7fe700 1 -- 172.21.15.195:0/1417219921 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfa4003430 con 0x7fbfc006a450 2022-06-27T11:56:56.525 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.511+0000 7fbfbf7fe700 1 -- 172.21.15.195:0/1417219921 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 269337278 0 0) 0x7fbfa4003430 con 0x7fbfc006a450 2022-06-27T11:56:56.526 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.511+0000 7fbfbf7fe700 1 -- 172.21.15.195:0/1417219921 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbfc01508d0 msgr2=0x7fbfc0063af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:56:56.526 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.511+0000 7fbfbf7fe700 1 --2- 172.21.15.195:0/1417219921 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbfc01508d0 0x7fbfc0063af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:56.527 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.511+0000 7fbfbf7fe700 1 -- 172.21.15.195:0/1417219921 >> v1:172.21.15.202:6789/0 conn(0x7fbfc0150500 legacy=0x7fbfc00c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:56:56.527 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.511+0000 7fbfbf7fe700 1 -- 172.21.15.195:0/1417219921 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfc015a800 con 0x7fbfc006a450 2022-06-27T11:56:56.528 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.512+0000 7fbfbf7fe700 1 -- 172.21.15.195:0/1417219921 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7fbfac001600 con 0x7fbfc006a450 2022-06-27T11:56:56.528 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.512+0000 7fbfbf7fe700 1 -- 172.21.15.195:0/1417219921 >> v1:172.21.15.195:6789/0 conn(0x7fbfc006a450 legacy=0x7fbfc00bbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:56:56.528 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.512+0000 7fbfbf7fe700 1 --2- 172.21.15.195:0/1417219921 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbfc01508d0 0x7fbfa4003950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:56:56.529 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.512+0000 7fbfbf7fe700 1 --2- 172.21.15.195:0/1417219921 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbfa4003ed0 0x7fbfa40046d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:56:56.529 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.512+0000 7fbfbf7fe700 1 --2- 172.21.15.195:0/1417219921 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbfa4004c10 0x7fbfa4005080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:56:56.529 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.512+0000 7fbfbf7fe700 1 -- 172.21.15.195:0/1417219921 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc00c4440 con 0x7fbfa4003ed0 2022-06-27T11:56:56.529 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.512+0000 7fbfbf7fe700 1 -- 172.21.15.195:0/1417219921 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa400bab0 con 0x7fbfa4004c10 2022-06-27T11:56:56.530 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.512+0000 7fbfbf7fe700 1 -- 172.21.15.195:0/1417219921 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa400bc70 con 0x7fbfc01508d0 2022-06-27T11:56:56.531 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.512+0000 7fbfbffff700 1 --2- 172.21.15.195:0/1417219921 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbfa4003ed0 0x7fbfa40046d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:56:56.531 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.512+0000 7fbfc495c700 1 --2- 172.21.15.195:0/1417219921 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbfc01508d0 0x7fbfa4003950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:56:56.532 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.512+0000 7fbfc515d700 1 --2- 172.21.15.195:0/1417219921 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbfa4004c10 0x7fbfa4005080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:56:56.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.512+0000 7fbfbffff700 1 -- 172.21.15.195:0/1417219921 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbfc01508d0 msgr2=0x7fbfa4003950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:56:56.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.512+0000 7fbfbffff700 1 --2- 172.21.15.195:0/1417219921 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbfc01508d0 0x7fbfa4003950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:56.534 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.512+0000 7fbfbffff700 1 -- 172.21.15.195:0/1417219921 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbfa4004c10 msgr2=0x7fbfa4005080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:56:56.534 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.512+0000 7fbfbffff700 1 --2- 172.21.15.195:0/1417219921 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbfa4004c10 0x7fbfa4005080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:56.535 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.512+0000 7fbfbffff700 1 -- 172.21.15.195:0/1417219921 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfa4003430 con 0x7fbfa4003ed0 2022-06-27T11:56:56.535 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.512+0000 7fbfbffff700 1 --2- 172.21.15.195:0/1417219921 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbfa4003ed0 0x7fbfa40046d0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fbfac001eb0 tx=0x7fbfac007910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ade78affc113fad server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:56:56.536 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.512+0000 7fbfbf7fe700 1 -- 172.21.15.195:0/1417219921 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfac00cc30 con 0x7fbfa4003ed0 2022-06-27T11:56:56.536 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.512+0000 7fbfbf7fe700 1 -- 172.21.15.195:0/1417219921 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfac00cd90 con 0x7fbfa4003ed0 2022-06-27T11:56:56.536 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.512+0000 7fbfbf7fe700 1 -- 172.21.15.195:0/1417219921 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfac007320 con 0x7fbfa4003ed0 2022-06-27T11:56:56.537 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.513+0000 7fbfc6bc0700 1 -- 172.21.15.195:0/1417219921 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbfa4003ed0 msgr2=0x7fbfa40046d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:56:56.537 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.513+0000 7fbfc6bc0700 1 --2- 172.21.15.195:0/1417219921 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbfa4003ed0 0x7fbfa40046d0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fbfac001eb0 tx=0x7fbfac007910 comp rx=0 tx=0).stop 2022-06-27T11:56:56.538 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.513+0000 7fbfc515d700 1 -- 172.21.15.195:0/1417219921 reap_dead start 2022-06-27T11:56:56.538 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.513+0000 7fbfc6bc0700 1 -- 172.21.15.195:0/1417219921 shutdown_connections 2022-06-27T11:56:56.539 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.513+0000 7fbfc6bc0700 1 -- 172.21.15.195:0/1417219921 >> 172.21.15.195:0/1417219921 conn(0x7fbfc00c9b00 msgr2=0x7fbfc00c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:56:56.539 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.513+0000 7fbfc6bc0700 1 -- 172.21.15.195:0/1417219921 shutdown_connections 2022-06-27T11:56:56.539 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.513+0000 7fbfc6bc0700 1 -- 172.21.15.195:0/1417219921 wait complete. 2022-06-27T11:56:56.540 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.514+0000 7fbfc6bc0700 1 Processor -- start 2022-06-27T11:56:56.540 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.514+0000 7fbfc6bc0700 1 -- start start 2022-06-27T11:56:56.541 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.515+0000 7fbfc6bc0700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbfc00bf490 0x7fbfc00bf860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:56:56.541 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.515+0000 7fbfc6bc0700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbfc01592f0 0x7fbfc016b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:56:56.542 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.515+0000 7fbfc6bc0700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbfc01587e0 0x7fbfc0158c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:56:56.542 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.515+0000 7fbfc6bc0700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc0154cb0 con 0x7fbfc00bf490 2022-06-27T11:56:56.542 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.515+0000 7fbfc6bc0700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc016b1c0 con 0x7fbfc01587e0 2022-06-27T11:56:56.542 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.515+0000 7fbfc6bc0700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc0154410 con 0x7fbfc01592f0 2022-06-27T11:56:56.543 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.515+0000 7fbfc495c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbfc00bf490 0x7fbfc00bf860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:56:56.543 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.515+0000 7fbfc495c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbfc00bf490 0x7fbfc00bf860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:44984/0 (socket says 172.21.15.195:44984) 2022-06-27T11:56:56.543 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.515+0000 7fbfc495c700 1 -- 172.21.15.195:0/3686726910 learned_addr learned my addr 172.21.15.195:0/3686726910 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:56:56.543 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.515+0000 7fbfc515d700 1 --2- 172.21.15.195:0/3686726910 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbfc01587e0 0x7fbfc0158c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:56:56.544 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.515+0000 7fbfbffff700 1 --2- 172.21.15.195:0/3686726910 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbfc01592f0 0x7fbfc016b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:56:56.544 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.515+0000 7fbfc495c700 1 -- 172.21.15.195:0/3686726910 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbfc01592f0 msgr2=0x7fbfc016b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:56:56.544 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.515+0000 7fbfc495c700 1 --2- 172.21.15.195:0/3686726910 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbfc01592f0 0x7fbfc016b650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:56.544 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.515+0000 7fbfc495c700 1 -- 172.21.15.195:0/3686726910 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbfc01587e0 msgr2=0x7fbfc0158c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:56:56.544 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.515+0000 7fbfc495c700 1 --2- 172.21.15.195:0/3686726910 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbfc01587e0 0x7fbfc0158c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:56.545 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.515+0000 7fbfc495c700 1 -- 172.21.15.195:0/3686726910 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfc01546f0 con 0x7fbfc00bf490 2022-06-27T11:56:56.545 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.516+0000 7fbfc495c700 1 --2- 172.21.15.195:0/3686726910 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbfc00bf490 0x7fbfc00bf860 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fbfb400f8e0 tx=0x7fbfb4009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:56:56.545 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.516+0000 7fbfbcff9700 1 -- 172.21.15.195:0/3686726910 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfb40053f0 con 0x7fbfc00bf490 2022-06-27T11:56:56.545 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.516+0000 7fbfbcff9700 1 -- 172.21.15.195:0/3686726910 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfb4007050 con 0x7fbfc00bf490 2022-06-27T11:56:56.546 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.516+0000 7fbfc6bc0700 1 -- 172.21.15.195:0/3686726910 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfc01549d0 con 0x7fbfc00bf490 2022-06-27T11:56:56.546 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.516+0000 7fbfbcff9700 1 -- 172.21.15.195:0/3686726910 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfb4007410 con 0x7fbfc00bf490 2022-06-27T11:56:56.546 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.516+0000 7fbfc6bc0700 1 -- 172.21.15.195:0/3686726910 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfc014c1f0 con 0x7fbfc00bf490 2022-06-27T11:56:56.546 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.517+0000 7fbfc6bc0700 1 -- 172.21.15.195:0/3686726910 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfc016acb0 con 0x7fbfc00bf490 2022-06-27T11:56:56.546 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.517+0000 7fbfbcff9700 1 -- 172.21.15.195:0/3686726910 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fbfb401c030 con 0x7fbfc00bf490 2022-06-27T11:56:56.547 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.517+0000 7fbfbcff9700 1 --2- 172.21.15.195:0/3686726910 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbfa8063d00 0x7fbfa80661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:56:56.547 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.517+0000 7fbfbcff9700 1 -- 172.21.15.195:0/3686726910 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fbfb40800f0 con 0x7fbfc00bf490 2022-06-27T11:56:56.548 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.517+0000 7fbfbffff700 1 --2- 172.21.15.195:0/3686726910 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbfa8063d00 0x7fbfa80661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:56:56.548 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.518+0000 7fbfbffff700 1 --2- 172.21.15.195:0/3686726910 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbfa8063d00 0x7fbfa80661b0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fbfac007d30 tx=0x7fbfac012000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:56:56.548 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.522+0000 7fbfbcff9700 1 -- 172.21.15.195:0/3686726910 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fbfb404cc90 con 0x7fbfc00bf490 2022-06-27T11:56:56.752 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.747+0000 7fbfc6bc0700 1 -- 172.21.15.195:0/3686726910 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbfc016b4a0 con 0x7fbfc00bf490 2022-06-27T11:56:56.752 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.747+0000 7fbfbcff9700 1 -- 172.21.15.195:0/3686726910 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+6128 (secure 0 0 0) 0x7fbfb401a090 con 0x7fbfc00bf490 2022-06-27T11:56:56.753 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:56:56.754 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/4154570751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":4154570751},{"type":"v1","addr":"172.21.15.202:6833","nonce":4154570751}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:55:37.983652+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4856},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4856":{"gid":4856,"name":"d","rank":0,"incarnation":26,"state":"up:active","state_seq":43,"addr":"172.21.15.202:6835/2970948376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2970948376},{"type":"v1","addr":"172.21.15.202:6835","nonce":2970948376}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:56:56.756 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.749+0000 7fbfc6bc0700 1 -- 172.21.15.195:0/3686726910 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbfa8063d00 msgr2=0x7fbfa80661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:56:56.756 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.749+0000 7fbfc6bc0700 1 --2- 172.21.15.195:0/3686726910 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbfa8063d00 0x7fbfa80661b0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fbfac007d30 tx=0x7fbfac012000 comp rx=0 tx=0).stop 2022-06-27T11:56:56.757 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.749+0000 7fbfc6bc0700 1 -- 172.21.15.195:0/3686726910 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbfc00bf490 msgr2=0x7fbfc00bf860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:56:56.757 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.749+0000 7fbfc6bc0700 1 --2- 172.21.15.195:0/3686726910 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbfc00bf490 0x7fbfc00bf860 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fbfb400f8e0 tx=0x7fbfb4009040 comp rx=0 tx=0).stop 2022-06-27T11:56:56.757 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.749+0000 7fbfc6bc0700 1 -- 172.21.15.195:0/3686726910 shutdown_connections 2022-06-27T11:56:56.758 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.749+0000 7fbfc6bc0700 1 --2- 172.21.15.195:0/3686726910 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbfa8063d00 0x7fbfa80661b0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:56.759 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.749+0000 7fbfc6bc0700 1 --2- 172.21.15.195:0/3686726910 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbfc00bf490 0x7fbfc00bf860 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:56.759 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.749+0000 7fbfc6bc0700 1 --2- 172.21.15.195:0/3686726910 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbfc01587e0 0x7fbfc0158c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:56.760 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.749+0000 7fbfc6bc0700 1 --2- 172.21.15.195:0/3686726910 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbfc01592f0 0x7fbfc016b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:56.760 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.749+0000 7fbfc6bc0700 1 -- 172.21.15.195:0/3686726910 >> 172.21.15.195:0/3686726910 conn(0x7fbfc00c9b00 msgr2=0x7fbfc00bbe30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:56:56.760 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.749+0000 7fbfc6bc0700 1 -- 172.21.15.195:0/3686726910 shutdown_connections 2022-06-27T11:56:56.761 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:56.749+0000 7fbfc6bc0700 1 -- 172.21.15.195:0/3686726910 wait complete. 2022-06-27T11:56:56.761 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 31 2022-06-27T11:56:56.766 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-06-27T11:56:58.768 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:56:58.866 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T11:56:58.876 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T11:56:58.915 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:58.915+0000 7f0100785700 1 Processor -- start 2022-06-27T11:56:58.927 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:58.915+0000 7f0100785700 1 -- start start 2022-06-27T11:56:58.927 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:58.916+0000 7f0100785700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f00f8150500 0x7f00f8063af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:56:58.928 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:58.916+0000 7f0100785700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f00f806a450 0x7f00f80c48d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:56:58.928 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:58.916+0000 7f0100785700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00f8064030 con 0x7f00f81508d0 2022-06-27T11:56:58.930 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:58.916+0000 7f0100785700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00f80c4160 con 0x7f00f8150500 2022-06-27T11:56:58.930 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:58.916+0000 7f0100785700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00f80c4440 con 0x7f00f806a450 2022-06-27T11:56:58.931 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:58.916+0000 7f00fe521700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f00f8150500 0x7f00f8063af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:56:58.931 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:58.916+0000 7f00fe521700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f00f8150500 0x7f00f8063af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:44996/0 (socket says 172.21.15.195:44996) 2022-06-27T11:56:58.932 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:58.916+0000 7f00fe521700 1 -- 172.21.15.195:0/1503708252 learned_addr learned my addr 172.21.15.195:0/1503708252 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:56:58.932 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:58.916+0000 7f00fdd20700 1 --2- 172.21.15.195:0/1503708252 >> v2:172.21.15.202:3300/0 conn(0x7f00f806a450 0x7f00f80c48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:56:58.932 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:58.916+0000 7f00fe521700 1 -- 172.21.15.195:0/1503708252 >> v2:172.21.15.202:3300/0 conn(0x7f00f806a450 msgr2=0x7f00f80c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:56:58.932 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:58.916+0000 7f00fe521700 1 --2- 172.21.15.195:0/1503708252 >> v2:172.21.15.202:3300/0 conn(0x7f00f806a450 0x7f00f80c48d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:58.933 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:58.916+0000 7f00fe521700 1 -- 172.21.15.195:0/1503708252 >> v1:172.21.15.202:6789/0 conn(0x7f00f81508d0 legacy=0x7f00f80bbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:56:58.933 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:58.916+0000 7f00fe521700 1 -- 172.21.15.195:0/1503708252 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00f815a800 con 0x7f00f8150500 2022-06-27T11:56:58.933 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:58.917+0000 7f00fe521700 1 --2- 172.21.15.195:0/1503708252 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f00f8150500 0x7f00f8063af0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f00f806ce70 tx=0x7f00e80090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c026e66af32a6fdf server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:56:58.933 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:58.917+0000 7f00fd51f700 1 -- 172.21.15.195:0/1503708252 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00e8005ad0 con 0x7f00f8150500 2022-06-27T11:56:58.934 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:58.917+0000 7f00fd51f700 1 -- 172.21.15.195:0/1503708252 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00e8005c50 con 0x7f00f8150500 2022-06-27T11:56:58.934 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:58.917+0000 7f00fd51f700 1 -- 172.21.15.195:0/1503708252 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00e8004030 con 0x7f00f8150500 2022-06-27T11:56:58.934 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:58.917+0000 7f0100785700 1 -- 172.21.15.195:0/1503708252 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f00f8150500 msgr2=0x7f00f8063af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:56:58.934 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:58.917+0000 7f0100785700 1 --2- 172.21.15.195:0/1503708252 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f00f8150500 0x7f00f8063af0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f00f806ce70 tx=0x7f00e80090f0 comp rx=0 tx=0).stop 2022-06-27T11:56:58.935 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:58.917+0000 7f0100785700 1 -- 172.21.15.195:0/1503708252 shutdown_connections 2022-06-27T11:56:58.935 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:58.917+0000 7f0100785700 1 --2- 172.21.15.195:0/1503708252 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f00f8150500 0x7f00f8063af0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:58.935 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:58.917+0000 7f0100785700 1 --2- 172.21.15.195:0/1503708252 >> v2:172.21.15.202:3300/0 conn(0x7f00f806a450 0x7f00f80c48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:58.935 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:58.917+0000 7f0100785700 1 -- 172.21.15.195:0/1503708252 >> 172.21.15.195:0/1503708252 conn(0x7f00f80c9b00 msgr2=0x7f00f80c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:56:58.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:58.917+0000 7f0100785700 1 -- 172.21.15.195:0/1503708252 shutdown_connections 2022-06-27T11:56:58.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:58.917+0000 7f0100785700 1 -- 172.21.15.195:0/1503708252 wait complete. 2022-06-27T11:56:58.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:58.918+0000 7f0100785700 1 Processor -- start 2022-06-27T11:56:58.940 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:58.919+0000 7f0100785700 1 -- start start 2022-06-27T11:56:58.941 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:58.919+0000 7f0100785700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f00f806a450 0x7f00f8156500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:56:58.941 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:58.919+0000 7f0100785700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f00f8150500 0x7f00f80c4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:56:58.941 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:58.919+0000 7f0100785700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f00f81508d0 0x7f00f8167070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:56:58.942 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:58.919+0000 7f0100785700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00f8166be0 con 0x7f00f806a450 2022-06-27T11:56:58.942 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:58.919+0000 7f0100785700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00f8156df0 con 0x7f00f81508d0 2022-06-27T11:56:58.942 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:58.919+0000 7f0100785700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00f8154270 con 0x7f00f8150500 2022-06-27T11:56:58.943 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:58.919+0000 7f00fe521700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f00f806a450 0x7f00f8156500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:56:58.943 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:58.919+0000 7f00fe521700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f00f806a450 0x7f00f8156500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45000/0 (socket says 172.21.15.195:45000) 2022-06-27T11:56:58.943 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:58.919+0000 7f00fe521700 1 -- 172.21.15.195:0/1944998254 learned_addr learned my addr 172.21.15.195:0/1944998254 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:56:58.943 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:58.919+0000 7f00fdd20700 1 --2- 172.21.15.195:0/1944998254 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f00f8150500 0x7f00f80c4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:56:58.944 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:58.919+0000 7f00fed22700 1 --2- 172.21.15.195:0/1944998254 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f00f81508d0 0x7f00f8167070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:56:58.944 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:58.920+0000 7f00fe521700 1 -- 172.21.15.195:0/1944998254 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f00f8150500 msgr2=0x7f00f80c4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:56:58.944 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:58.920+0000 7f00fe521700 1 --2- 172.21.15.195:0/1944998254 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f00f8150500 0x7f00f80c4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:58.945 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:58.920+0000 7f00fe521700 1 -- 172.21.15.195:0/1944998254 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f00f81508d0 msgr2=0x7f00f8167070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:56:58.946 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:58.920+0000 7f00fe521700 1 --2- 172.21.15.195:0/1944998254 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f00f81508d0 0x7f00f8167070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:58.953 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:58.920+0000 7f00fe521700 1 -- 172.21.15.195:0/1944998254 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00e8009350 con 0x7f00f806a450 2022-06-27T11:56:58.954 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:58.920+0000 7f00fe521700 1 --2- 172.21.15.195:0/1944998254 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f00f806a450 0x7f00f8156500 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f00e8004000 tx=0x7f00e8015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:56:58.954 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:58.920+0000 7f00e67fc700 1 -- 172.21.15.195:0/1944998254 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00e8000d10 con 0x7f00f806a450 2022-06-27T11:56:58.955 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:58.920+0000 7f00e67fc700 1 -- 172.21.15.195:0/1944998254 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00e8003520 con 0x7f00f806a450 2022-06-27T11:56:58.955 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:58.920+0000 7f0100785700 1 -- 172.21.15.195:0/1944998254 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00f8154550 con 0x7f00f806a450 2022-06-27T11:56:58.960 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:58.920+0000 7f00e67fc700 1 -- 172.21.15.195:0/1944998254 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00e80036d0 con 0x7f00f806a450 2022-06-27T11:56:58.960 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:58.920+0000 7f0100785700 1 -- 172.21.15.195:0/1944998254 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00f8154b10 con 0x7f00f806a450 2022-06-27T11:56:58.964 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:58.921+0000 7f0100785700 1 -- 172.21.15.195:0/1944998254 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00f8166ec0 con 0x7f00f806a450 2022-06-27T11:56:58.964 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:58.921+0000 7f00e67fc700 1 -- 172.21.15.195:0/1944998254 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f00e802e070 con 0x7f00f806a450 2022-06-27T11:56:58.964 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:58.922+0000 7f00e67fc700 1 --2- 172.21.15.195:0/1944998254 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f00d8063d00 0x7f00d80661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:56:58.964 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:58.922+0000 7f00e67fc700 1 -- 172.21.15.195:0/1944998254 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f00e80828d0 con 0x7f00f806a450 2022-06-27T11:56:58.965 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:58.922+0000 7f00fdd20700 1 --2- 172.21.15.195:0/1944998254 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f00d8063d00 0x7f00d80661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:56:58.965 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:58.922+0000 7f00fdd20700 1 --2- 172.21.15.195:0/1944998254 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f00d8063d00 0x7f00d80661b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f00f80c34e0 tx=0x7f00f4009060 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:56:58.965 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:58.926+0000 7f00e67fc700 1 -- 172.21.15.195:0/1944998254 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f00e8051330 con 0x7f00f806a450 2022-06-27T11:56:59.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:59.153+0000 7f0100785700 1 -- 172.21.15.195:0/1944998254 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f00f80c67a0 con 0x7f00f806a450 2022-06-27T11:56:59.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:59.154+0000 7f00e67fc700 1 -- 172.21.15.195:0/1944998254 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+6128 (secure 0 0 0) 0x7f00e802c020 con 0x7f00f806a450 2022-06-27T11:56:59.155 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:56:59.156 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/4154570751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":4154570751},{"type":"v1","addr":"172.21.15.202:6833","nonce":4154570751}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:55:37.983652+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4856},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4856":{"gid":4856,"name":"d","rank":0,"incarnation":26,"state":"up:active","state_seq":43,"addr":"172.21.15.202:6835/2970948376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2970948376},{"type":"v1","addr":"172.21.15.202:6835","nonce":2970948376}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:56:59.157 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:59.156+0000 7f0100785700 1 -- 172.21.15.195:0/1944998254 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f00d8063d00 msgr2=0x7f00d80661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:56:59.157 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:59.156+0000 7f0100785700 1 --2- 172.21.15.195:0/1944998254 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f00d8063d00 0x7f00d80661b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f00f80c34e0 tx=0x7f00f4009060 comp rx=0 tx=0).stop 2022-06-27T11:56:59.158 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:59.156+0000 7f0100785700 1 -- 172.21.15.195:0/1944998254 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f00f806a450 msgr2=0x7f00f8156500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:56:59.158 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:59.156+0000 7f0100785700 1 --2- 172.21.15.195:0/1944998254 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f00f806a450 0x7f00f8156500 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f00e8004000 tx=0x7f00e8015000 comp rx=0 tx=0).stop 2022-06-27T11:56:59.158 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:59.156+0000 7f0100785700 1 -- 172.21.15.195:0/1944998254 shutdown_connections 2022-06-27T11:56:59.158 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:59.156+0000 7f0100785700 1 --2- 172.21.15.195:0/1944998254 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f00d8063d00 0x7f00d80661b0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:59.159 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:59.156+0000 7f0100785700 1 --2- 172.21.15.195:0/1944998254 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f00f806a450 0x7f00f8156500 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:59.159 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:59.156+0000 7f0100785700 1 --2- 172.21.15.195:0/1944998254 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f00f81508d0 0x7f00f8167070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:59.159 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:59.156+0000 7f0100785700 1 --2- 172.21.15.195:0/1944998254 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f00f8150500 0x7f00f80c4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:56:59.160 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:59.156+0000 7f0100785700 1 -- 172.21.15.195:0/1944998254 >> 172.21.15.195:0/1944998254 conn(0x7f00f80c9b00 msgr2=0x7f00f80c7120 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:56:59.160 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:59.156+0000 7f0100785700 1 -- 172.21.15.195:0/1944998254 shutdown_connections 2022-06-27T11:56:59.160 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:56:59.156+0000 7f0100785700 1 -- 172.21.15.195:0/1944998254 wait complete. 2022-06-27T11:56:59.160 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 31 2022-06-27T11:56:59.169 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-06-27T11:57:01.171 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:57:01.324 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.304+0000 7fedefd2e700 1 Processor -- start 2022-06-27T11:57:01.325 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.304+0000 7fedefd2e700 1 -- start start 2022-06-27T11:57:01.326 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.305+0000 7fedefd2e700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fede81508d0 0x7fede8063af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:01.326 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.305+0000 7fedefd2e700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fede8064030 con 0x7fede8150500 2022-06-27T11:57:01.327 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.305+0000 7fedefd2e700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fede80c4160 con 0x7fede806a450 2022-06-27T11:57:01.328 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.305+0000 7fedefd2e700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fede80c4440 con 0x7fede81508d0 2022-06-27T11:57:01.328 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.305+0000 7fededaca700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fede8150500 0x7fede80c48d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:51196/0 (socket says 172.21.15.195:51196) 2022-06-27T11:57:01.329 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.305+0000 7fededaca700 1 -- 172.21.15.195:0/1536126060 learned_addr learned my addr 172.21.15.195:0/1536126060 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:57:01.329 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.305+0000 7fedee2cb700 1 --2- 172.21.15.195:0/1536126060 >> v2:172.21.15.202:3300/0 conn(0x7fede81508d0 0x7fede8063af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:01.330 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.305+0000 7fedecac8700 1 -- 172.21.15.195:0/1536126060 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 862669856 0 0) 0x7fedd8001290 con 0x7fede8150500 2022-06-27T11:57:01.330 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.305+0000 7fedecac8700 1 -- 172.21.15.195:0/1536126060 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedd0003430 con 0x7fede8150500 2022-06-27T11:57:01.331 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.306+0000 7fedecac8700 1 -- 172.21.15.195:0/1536126060 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2485173117 0 0) 0x7fedd0003430 con 0x7fede8150500 2022-06-27T11:57:01.331 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.306+0000 7fedecac8700 1 -- 172.21.15.195:0/1536126060 >> v2:172.21.15.202:3300/0 conn(0x7fede81508d0 msgr2=0x7fede8063af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:57:01.332 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.306+0000 7fedecac8700 1 --2- 172.21.15.195:0/1536126060 >> v2:172.21.15.202:3300/0 conn(0x7fede81508d0 0x7fede8063af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:01.332 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.306+0000 7fedecac8700 1 -- 172.21.15.195:0/1536126060 >> v1:172.21.15.202:6789/0 conn(0x7fede806a450 legacy=0x7fede80bbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:01.333 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.306+0000 7fedecac8700 1 -- 172.21.15.195:0/1536126060 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fede815a800 con 0x7fede8150500 2022-06-27T11:57:01.333 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.306+0000 7fedecac8700 1 -- 172.21.15.195:0/1536126060 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7fedd8001780 con 0x7fede8150500 2022-06-27T11:57:01.334 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.306+0000 7fedecac8700 1 -- 172.21.15.195:0/1536126060 >> v1:172.21.15.195:6789/0 conn(0x7fede8150500 legacy=0x7fede80c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:01.334 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.306+0000 7fedecac8700 1 --2- 172.21.15.195:0/1536126060 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fedd0003920 0x7fedd0003cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:01.335 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.306+0000 7fedecac8700 1 --2- 172.21.15.195:0/1536126060 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fedd00046d0 0x7fedd0004b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:01.336 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.306+0000 7fedecac8700 1 --2- 172.21.15.195:0/1536126060 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fedd0005080 0x7fedd000b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:01.336 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.306+0000 7fedecac8700 1 -- 172.21.15.195:0/1536126060 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede80c4440 con 0x7fedd0003920 2022-06-27T11:57:01.337 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.306+0000 7fedecac8700 1 -- 172.21.15.195:0/1536126060 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd000bf10 con 0x7fedd00046d0 2022-06-27T11:57:01.338 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.306+0000 7fedecac8700 1 -- 172.21.15.195:0/1536126060 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedd000c0d0 con 0x7fedd0005080 2022-06-27T11:57:01.338 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.306+0000 7fededaca700 1 --2- 172.21.15.195:0/1536126060 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fedd0003920 0x7fedd0003cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:01.339 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.306+0000 7feded2c9700 1 --2- 172.21.15.195:0/1536126060 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fedd00046d0 0x7fedd0004b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:01.339 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.306+0000 7fedee2cb700 1 --2- 172.21.15.195:0/1536126060 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fedd0005080 0x7fedd000b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:01.340 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.306+0000 7fededaca700 1 -- 172.21.15.195:0/1536126060 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fedd0005080 msgr2=0x7fedd000b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:57:01.341 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.306+0000 7fededaca700 1 --2- 172.21.15.195:0/1536126060 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fedd0005080 0x7fedd000b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:01.341 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.306+0000 7fededaca700 1 -- 172.21.15.195:0/1536126060 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fedd00046d0 msgr2=0x7fedd0004b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:57:01.342 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.306+0000 7fededaca700 1 --2- 172.21.15.195:0/1536126060 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fedd00046d0 0x7fedd0004b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:01.343 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.306+0000 7fedee2cb700 1 -- 172.21.15.195:0/1536126060 reap_dead start 2022-06-27T11:57:01.343 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.306+0000 7fededaca700 1 -- 172.21.15.195:0/1536126060 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedd0003430 con 0x7fedd0003920 2022-06-27T11:57:01.344 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.306+0000 7fededaca700 1 --2- 172.21.15.195:0/1536126060 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fedd0003920 0x7fedd0003cf0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fedd800c340 tx=0x7fede80bbe70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e06d4683caaf64da server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:57:01.344 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.306+0000 7fedecac8700 1 -- 172.21.15.195:0/1536126060 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd800cbf0 con 0x7fedd0003920 2022-06-27T11:57:01.345 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.306+0000 7fedecac8700 1 -- 172.21.15.195:0/1536126060 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedd800cd70 con 0x7fedd0003920 2022-06-27T11:57:01.346 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.306+0000 7fedecac8700 1 -- 172.21.15.195:0/1536126060 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd8007320 con 0x7fedd0003920 2022-06-27T11:57:01.346 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.307+0000 7fedefd2e700 1 -- 172.21.15.195:0/1536126060 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fedd0003920 msgr2=0x7fedd0003cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:01.347 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.307+0000 7fedefd2e700 1 --2- 172.21.15.195:0/1536126060 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fedd0003920 0x7fedd0003cf0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fedd800c340 tx=0x7fede80bbe70 comp rx=0 tx=0).stop 2022-06-27T11:57:01.347 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.307+0000 7fedefd2e700 1 -- 172.21.15.195:0/1536126060 shutdown_connections 2022-06-27T11:57:01.348 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.307+0000 7fedefd2e700 1 --2- 172.21.15.195:0/1536126060 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fedd0003920 0x7fedd0003cf0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:01.348 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.307+0000 7fedefd2e700 1 -- 172.21.15.195:0/1536126060 >> 172.21.15.195:0/1536126060 conn(0x7fede80c9b00 msgr2=0x7fede80c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:57:01.349 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.307+0000 7fedefd2e700 1 -- 172.21.15.195:0/1536126060 shutdown_connections 2022-06-27T11:57:01.349 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.307+0000 7fedefd2e700 1 -- 172.21.15.195:0/1536126060 wait complete. 2022-06-27T11:57:01.350 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.308+0000 7fedefd2e700 1 Processor -- start 2022-06-27T11:57:01.350 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.308+0000 7fedefd2e700 1 -- start start 2022-06-27T11:57:01.351 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.309+0000 7fedefd2e700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fedd0003920 0x7fede80c11e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:01.351 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.309+0000 7fedefd2e700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fede806dde0 0x7fede81697a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:01.352 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.309+0000 7fedefd2e700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fede806e210 0x7fede8169ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:01.352 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.309+0000 7fedefd2e700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede80bdd50 con 0x7fedd0003920 2022-06-27T11:57:01.353 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.309+0000 7fedefd2e700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede80be5f0 con 0x7fede806dde0 2022-06-27T11:57:01.353 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.309+0000 7fedefd2e700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fede80c5440 con 0x7fede806e210 2022-06-27T11:57:01.354 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.309+0000 7fededaca700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fedd0003920 0x7fede80c11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:01.354 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.309+0000 7fededaca700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fedd0003920 0x7fede80c11e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45020/0 (socket says 172.21.15.195:45020) 2022-06-27T11:57:01.355 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.309+0000 7fededaca700 1 -- 172.21.15.195:0/4267643365 learned_addr learned my addr 172.21.15.195:0/4267643365 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:57:01.355 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.309+0000 7feded2c9700 1 --2- 172.21.15.195:0/4267643365 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fede806dde0 0x7fede81697a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:01.356 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.309+0000 7fedee2cb700 1 --2- 172.21.15.195:0/4267643365 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fede806e210 0x7fede8169ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:01.356 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.309+0000 7fededaca700 1 -- 172.21.15.195:0/4267643365 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fede806e210 msgr2=0x7fede8169ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:01.357 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.309+0000 7fededaca700 1 --2- 172.21.15.195:0/4267643365 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fede806e210 0x7fede8169ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:01.357 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.309+0000 7fededaca700 1 -- 172.21.15.195:0/4267643365 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fede806dde0 msgr2=0x7fede81697a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:01.358 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.309+0000 7fededaca700 1 --2- 172.21.15.195:0/4267643365 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fede806dde0 0x7fede81697a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:01.358 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.309+0000 7fededaca700 1 -- 172.21.15.195:0/4267643365 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fede80bf8d0 con 0x7fedd0003920 2022-06-27T11:57:01.359 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.309+0000 7fededaca700 1 --2- 172.21.15.195:0/4267643365 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fedd0003920 0x7fede80c11e0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fedd800cee0 tx=0x7fedd8015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:57:01.359 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.310+0000 7fedd5ffb700 1 -- 172.21.15.195:0/4267643365 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd8000f70 con 0x7fedd0003920 2022-06-27T11:57:01.360 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.310+0000 7fedd5ffb700 1 -- 172.21.15.195:0/4267643365 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedd80072e0 con 0x7fedd0003920 2022-06-27T11:57:01.360 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.310+0000 7fedefd2e700 1 -- 172.21.15.195:0/4267643365 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fede80bfbb0 con 0x7fedd0003920 2022-06-27T11:57:01.361 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.310+0000 7fedd5ffb700 1 -- 172.21.15.195:0/4267643365 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd8005900 con 0x7fedd0003920 2022-06-27T11:57:01.361 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.310+0000 7fedefd2e700 1 -- 172.21.15.195:0/4267643365 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fede80be280 con 0x7fedd0003920 2022-06-27T11:57:01.362 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.311+0000 7fedefd2e700 1 -- 172.21.15.195:0/4267643365 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fede80be8d0 con 0x7fedd0003920 2022-06-27T11:57:01.363 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.311+0000 7fedd5ffb700 1 -- 172.21.15.195:0/4267643365 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fedd8010070 con 0x7fedd0003920 2022-06-27T11:57:01.363 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.311+0000 7fedd5ffb700 1 --2- 172.21.15.195:0/4267643365 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fedc8063d30 0x7fedc80661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:01.364 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.312+0000 7fedd5ffb700 1 -- 172.21.15.195:0/4267643365 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6546+0+0 (secure 0 0 0) 0x7fedd80819e0 con 0x7fedd0003920 2022-06-27T11:57:01.364 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.312+0000 7feded2c9700 1 --2- 172.21.15.195:0/4267643365 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fedc8063d30 0x7fedc80661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:01.365 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.312+0000 7feded2c9700 1 --2- 172.21.15.195:0/4267643365 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fedc8063d30 0x7fedc80661e0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fede40010e0 tx=0x7fede4009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:57:01.365 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.316+0000 7fedd5ffb700 1 -- 172.21.15.195:0/4267643365 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fedd80502f0 con 0x7fedd0003920 2022-06-27T11:57:01.534 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.534+0000 7fedefd2e700 1 -- 172.21.15.195:0/4267643365 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fede8064030 con 0x7fedd0003920 2022-06-27T11:57:01.535 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.534+0000 7fedd5ffb700 1 -- 172.21.15.195:0/4267643365 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+5367 (secure 0 0 0) 0x7fede8064030 con 0x7fedd0003920 2022-06-27T11:57:01.536 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:57:01.538 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:56:59.959866+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4976},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4976":{"gid":4976,"name":"b","rank":0,"incarnation":32,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6833/4154570751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":4154570751},{"type":"v1","addr":"172.21.15.202:6833","nonce":4154570751}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:57:01.540 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.536+0000 7fedefd2e700 1 -- 172.21.15.195:0/4267643365 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fedc8063d30 msgr2=0x7fedc80661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:01.541 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.536+0000 7fedefd2e700 1 --2- 172.21.15.195:0/4267643365 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fedc8063d30 0x7fedc80661e0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fede40010e0 tx=0x7fede4009040 comp rx=0 tx=0).stop 2022-06-27T11:57:01.541 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.536+0000 7fedefd2e700 1 -- 172.21.15.195:0/4267643365 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fedd0003920 msgr2=0x7fede80c11e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:01.542 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.536+0000 7fedefd2e700 1 --2- 172.21.15.195:0/4267643365 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fedd0003920 0x7fede80c11e0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fedd800cee0 tx=0x7fedd8015000 comp rx=0 tx=0).stop 2022-06-27T11:57:01.542 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.536+0000 7fedefd2e700 1 -- 172.21.15.195:0/4267643365 shutdown_connections 2022-06-27T11:57:01.542 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.536+0000 7fedefd2e700 1 --2- 172.21.15.195:0/4267643365 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fedc8063d30 0x7fedc80661e0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:01.542 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.536+0000 7fedefd2e700 1 --2- 172.21.15.195:0/4267643365 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fedd0003920 0x7fede80c11e0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:01.543 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.536+0000 7fedefd2e700 1 --2- 172.21.15.195:0/4267643365 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fede806e210 0x7fede8169ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:01.543 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.536+0000 7fedefd2e700 1 --2- 172.21.15.195:0/4267643365 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fede806dde0 0x7fede81697a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:01.544 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.537+0000 7fedefd2e700 1 -- 172.21.15.195:0/4267643365 >> 172.21.15.195:0/4267643365 conn(0x7fede80c9b00 msgr2=0x7fede80c3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:57:01.544 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.537+0000 7fedefd2e700 1 -- 172.21.15.195:0/4267643365 shutdown_connections 2022-06-27T11:57:01.545 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.537+0000 7fedefd2e700 1 -- 172.21.15.195:0/4267643365 wait complete. 2022-06-27T11:57:01.545 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 32 2022-06-27T11:57:01.552 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d down, removed from mdsmap 2022-06-27T11:57:01.553 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2022-06-27T11:57:01.563 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:57:01.697 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.689+0000 7f3888689700 1 Processor -- start 2022-06-27T11:57:01.697 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.690+0000 7f3888689700 1 -- start start 2022-06-27T11:57:01.698 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.690+0000 7f3888689700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f388014e490 0x7f388006cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:01.698 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.690+0000 7f3888689700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f388014e860 0x7f38800c3840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:01.698 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.690+0000 7f3888689700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38800c33b0 con 0x7f38800c3d80 2022-06-27T11:57:01.698 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.690+0000 7f3888689700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f388006d520 con 0x7f388014e490 2022-06-27T11:57:01.699 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.690+0000 7f3888689700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f388006f220 con 0x7f388014e860 2022-06-27T11:57:01.699 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.691+0000 7f3886425700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f388014e490 0x7f388006cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:01.700 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.691+0000 7f3886425700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f388014e490 0x7f388006cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45028/0 (socket says 172.21.15.195:45028) 2022-06-27T11:57:01.700 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.691+0000 7f3886425700 1 -- 172.21.15.195:0/88730189 learned_addr learned my addr 172.21.15.195:0/88730189 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:57:01.700 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.691+0000 7f3885c24700 1 --2- 172.21.15.195:0/88730189 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f388014e860 0x7f38800c3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:01.700 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.691+0000 7f3886425700 1 -- 172.21.15.195:0/88730189 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f388014e860 msgr2=0x7f38800c3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:57:01.701 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.691+0000 7f3886425700 1 --2- 172.21.15.195:0/88730189 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f388014e860 0x7f38800c3840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:01.701 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.691+0000 7f3886425700 1 -- 172.21.15.195:0/88730189 >> v1:172.21.15.202:6789/0 conn(0x7f38800c3d80 legacy=0x7f388006f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:01.701 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.691+0000 7f3886425700 1 -- 172.21.15.195:0/88730189 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38801542c0 con 0x7f388014e490 2022-06-27T11:57:01.701 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.691+0000 7f3886425700 1 --2- 172.21.15.195:0/88730189 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f388014e490 0x7f388006cfe0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f3870009860 tx=0x7f3870009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff67ad4661a701ff server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:57:01.702 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.691+0000 7f3885423700 1 -- 172.21.15.195:0/88730189 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3870011630 con 0x7f388014e490 2022-06-27T11:57:01.702 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.691+0000 7f3885423700 1 -- 172.21.15.195:0/88730189 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3870011790 con 0x7f388014e490 2022-06-27T11:57:01.702 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.691+0000 7f3885423700 1 -- 172.21.15.195:0/88730189 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3870011b30 con 0x7f388014e490 2022-06-27T11:57:01.702 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.692+0000 7f3888689700 1 -- 172.21.15.195:0/88730189 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f388014e490 msgr2=0x7f388006cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:01.703 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.692+0000 7f3888689700 1 --2- 172.21.15.195:0/88730189 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f388014e490 0x7f388006cfe0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f3870009860 tx=0x7f3870009120 comp rx=0 tx=0).stop 2022-06-27T11:57:01.703 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.692+0000 7f3888689700 1 -- 172.21.15.195:0/88730189 shutdown_connections 2022-06-27T11:57:01.703 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.692+0000 7f3888689700 1 --2- 172.21.15.195:0/88730189 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f388014e490 0x7f388006cfe0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:01.703 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.692+0000 7f3888689700 1 --2- 172.21.15.195:0/88730189 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f388014e860 0x7f38800c3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:01.703 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.692+0000 7f3888689700 1 -- 172.21.15.195:0/88730189 >> 172.21.15.195:0/88730189 conn(0x7f3880063af0 msgr2=0x7f3880063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:57:01.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.692+0000 7f3888689700 1 -- 172.21.15.195:0/88730189 shutdown_connections 2022-06-27T11:57:01.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.692+0000 7f3888689700 1 -- 172.21.15.195:0/88730189 wait complete. 2022-06-27T11:57:01.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.693+0000 7f3888689700 1 Processor -- start 2022-06-27T11:57:01.705 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.693+0000 7f3888689700 1 -- start start 2022-06-27T11:57:01.705 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.694+0000 7f3888689700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f38800c3d80 0x7f3880151690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:01.705 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.694+0000 7f3888689700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f388014e490 0x7f388014b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:01.706 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.694+0000 7f3888689700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f388014e860 0x7f3880146a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:01.706 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.694+0000 7f3888689700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3880151bd0 con 0x7f388014e860 2022-06-27T11:57:01.707 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.694+0000 7f3888689700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3880168010 con 0x7f38800c3d80 2022-06-27T11:57:01.707 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.694+0000 7f3888689700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38801682d0 con 0x7f388014e490 2022-06-27T11:57:01.708 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.694+0000 7f3886c26700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f388014e860 0x7f3880146a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:01.708 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.694+0000 7f3886c26700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f388014e860 0x7f3880146a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45038/0 (socket says 172.21.15.195:45038) 2022-06-27T11:57:01.708 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.694+0000 7f3886c26700 1 -- 172.21.15.195:0/1152255014 learned_addr learned my addr 172.21.15.195:0/1152255014 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:57:01.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.694+0000 7f3885c24700 1 --2- 172.21.15.195:0/1152255014 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f388014e490 0x7f388014b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:01.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.694+0000 7f3886425700 1 --2- 172.21.15.195:0/1152255014 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f38800c3d80 0x7f3880151690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:01.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.694+0000 7f3886c26700 1 -- 172.21.15.195:0/1152255014 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f388014e490 msgr2=0x7f388014b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:01.710 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.694+0000 7f3886c26700 1 --2- 172.21.15.195:0/1152255014 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f388014e490 0x7f388014b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:01.710 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.694+0000 7f3886c26700 1 -- 172.21.15.195:0/1152255014 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f38800c3d80 msgr2=0x7f3880151690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:01.710 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.694+0000 7f3886c26700 1 --2- 172.21.15.195:0/1152255014 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f38800c3d80 0x7f3880151690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:01.711 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.694+0000 7f3886c26700 1 -- 172.21.15.195:0/1152255014 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38700093a0 con 0x7f388014e860 2022-06-27T11:57:01.711 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.694+0000 7f3885c24700 1 --2- 172.21.15.195:0/1152255014 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f388014e490 0x7f388014b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:57:01.711 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.694+0000 7f3886c26700 1 --2- 172.21.15.195:0/1152255014 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f388014e860 0x7f3880146a30 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f387800c7c0 tx=0x7f38780015e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:57:01.711 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.694+0000 7f387e7fc700 1 -- 172.21.15.195:0/1152255014 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3878003880 con 0x7f388014e860 2022-06-27T11:57:01.712 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.694+0000 7f387e7fc700 1 -- 172.21.15.195:0/1152255014 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38780039e0 con 0x7f388014e860 2022-06-27T11:57:01.712 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.694+0000 7f3888689700 1 -- 172.21.15.195:0/1152255014 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38801685b0 con 0x7f388014e860 2022-06-27T11:57:01.712 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.694+0000 7f387e7fc700 1 -- 172.21.15.195:0/1152255014 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3878003d60 con 0x7f388014e860 2022-06-27T11:57:01.712 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.694+0000 7f3888689700 1 -- 172.21.15.195:0/1152255014 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3880168b70 con 0x7f388014e860 2022-06-27T11:57:01.713 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.695+0000 7f3888689700 1 -- 172.21.15.195:0/1152255014 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f388014dd70 con 0x7f388014e860 2022-06-27T11:57:01.713 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.696+0000 7f387e7fc700 1 -- 172.21.15.195:0/1152255014 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f3878010050 con 0x7f388014e860 2022-06-27T11:57:01.713 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.696+0000 7f387e7fc700 1 --2- 172.21.15.195:0/1152255014 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3860063d00 0x7f38600661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:01.713 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.696+0000 7f387e7fc700 1 -- 172.21.15.195:0/1152255014 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6546+0+0 (secure 0 0 0) 0x7f38780803c0 con 0x7f388014e860 2022-06-27T11:57:01.714 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.696+0000 7f3886425700 1 --2- 172.21.15.195:0/1152255014 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3860063d00 0x7f38600661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:01.714 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.697+0000 7f3886425700 1 --2- 172.21.15.195:0/1152255014 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3860063d00 0x7f38600661b0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3870009160 tx=0x7f3870017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:57:01.715 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.700+0000 7f387e7fc700 1 -- 172.21.15.195:0/1152255014 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f387804edc0 con 0x7f388014e860 2022-06-27T11:57:01.927 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.927+0000 7f3888689700 1 -- 172.21.15.195:0/1152255014 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f38800722c0 con 0x7f388014e860 2022-06-27T11:57:01.928 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.927+0000 7f387e7fc700 1 -- 172.21.15.195:0/1152255014 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+5367 (secure 0 0 0) 0x7f38780523e0 con 0x7f388014e860 2022-06-27T11:57:01.929 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:57:01.930 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:56:59.959866+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4976},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4976":{"gid":4976,"name":"b","rank":0,"incarnation":32,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6833/4154570751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":4154570751},{"type":"v1","addr":"172.21.15.202:6833","nonce":4154570751}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:57:01.932 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.929+0000 7f3888689700 1 -- 172.21.15.195:0/1152255014 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3860063d00 msgr2=0x7f38600661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:01.933 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.929+0000 7f3888689700 1 --2- 172.21.15.195:0/1152255014 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3860063d00 0x7f38600661b0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3870009160 tx=0x7f3870017000 comp rx=0 tx=0).stop 2022-06-27T11:57:01.934 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.929+0000 7f3888689700 1 -- 172.21.15.195:0/1152255014 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f388014e860 msgr2=0x7f3880146a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:01.934 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.929+0000 7f3888689700 1 --2- 172.21.15.195:0/1152255014 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f388014e860 0x7f3880146a30 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f387800c7c0 tx=0x7f38780015e0 comp rx=0 tx=0).stop 2022-06-27T11:57:01.935 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.929+0000 7f3888689700 1 -- 172.21.15.195:0/1152255014 shutdown_connections 2022-06-27T11:57:01.935 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.929+0000 7f3888689700 1 --2- 172.21.15.195:0/1152255014 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3860063d00 0x7f38600661b0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:01.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.929+0000 7f3888689700 1 --2- 172.21.15.195:0/1152255014 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f388014e860 0x7f3880146a30 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:01.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.929+0000 7f3888689700 1 --2- 172.21.15.195:0/1152255014 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f38800c3d80 0x7f3880151690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:01.937 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.929+0000 7f3888689700 1 --2- 172.21.15.195:0/1152255014 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f388014e490 0x7f388014b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:01.937 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.929+0000 7f3888689700 1 -- 172.21.15.195:0/1152255014 >> 172.21.15.195:0/1152255014 conn(0x7f3880063af0 msgr2=0x7f38800c9350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:57:01.938 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.929+0000 7f3888689700 1 -- 172.21.15.195:0/1152255014 shutdown_connections 2022-06-27T11:57:01.939 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:01.929+0000 7f3888689700 1 -- 172.21.15.195:0/1152255014 wait complete. 2022-06-27T11:57:01.939 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 32 2022-06-27T11:57:01.946 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 32, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-06-27T11:47:52.358095+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 32, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4976": { "addr": "172.21.15.202:6833/4154570751", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6832", "nonce": 4154570751, "type": "v2" }, { "addr": "172.21.15.202:6833", "nonce": 4154570751, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4976, "incarnation": 32, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 52, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-06-27T11:56:59.959866+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4976 } } } ], "standbys": [ { "addr": "172.21.15.202:6837/3305006841", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6836", "nonce": 3305006841, "type": "v2" }, { "addr": "172.21.15.202:6837", "nonce": 3305006841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6835/2060709822", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6834", "nonce": 2060709822, "type": "v2" }, { "addr": "172.21.15.195:6835", "nonce": 2060709822, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6838/1598751652", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6836", "nonce": 1598751652, "type": "v2" }, { "addr": "172.21.15.195:6838", "nonce": 1598751652, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4275, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6839/1313800661", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6837", "nonce": 1313800661, "type": "v2" }, { "addr": "172.21.15.195:6839", "nonce": 1313800661, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4278, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-06-27T11:57:03.948 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:57:04.107 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.082+0000 7f9132e0d700 1 Processor -- start 2022-06-27T11:57:04.108 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.082+0000 7f9132e0d700 1 -- start start 2022-06-27T11:57:04.109 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.082+0000 7f9132e0d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f912c150500 0x7f912c063af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:04.109 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.083+0000 7f9132e0d700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f912c06a450 0x7f912c0c48d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:04.111 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.083+0000 7f9132e0d700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f912c064030 con 0x7f912c1508d0 2022-06-27T11:57:04.111 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.083+0000 7f9132e0d700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f912c0c4160 con 0x7f912c150500 2022-06-27T11:57:04.112 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.083+0000 7f9132e0d700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f912c0c4440 con 0x7f912c06a450 2022-06-27T11:57:04.112 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.083+0000 7f9130ba9700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f912c150500 0x7f912c063af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:04.113 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.083+0000 7f91313aa700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f912c1508d0 0x7f912c0bbf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:51250/0 (socket says 172.21.15.195:51250) 2022-06-27T11:57:04.113 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.083+0000 7f9130ba9700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f912c150500 0x7f912c063af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45060/0 (socket says 172.21.15.195:45060) 2022-06-27T11:57:04.114 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.083+0000 7f91313aa700 1 -- 172.21.15.195:0/2227158856 learned_addr learned my addr 172.21.15.195:0/2227158856 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:57:04.114 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.083+0000 7f912bfff700 1 --2- 172.21.15.195:0/2227158856 >> v2:172.21.15.202:3300/0 conn(0x7f912c06a450 0x7f912c0c48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:04.115 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.083+0000 7f9130ba9700 1 -- 172.21.15.195:0/2227158856 >> v2:172.21.15.202:3300/0 conn(0x7f912c06a450 msgr2=0x7f912c0c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:57:04.115 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.083+0000 7f9130ba9700 1 --2- 172.21.15.195:0/2227158856 >> v2:172.21.15.202:3300/0 conn(0x7f912c06a450 0x7f912c0c48d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:04.116 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.083+0000 7f9130ba9700 1 -- 172.21.15.195:0/2227158856 >> v1:172.21.15.195:6789/0 conn(0x7f912c1508d0 legacy=0x7f912c0bbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:04.116 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.083+0000 7f9130ba9700 1 -- 172.21.15.195:0/2227158856 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f912c15a800 con 0x7f912c150500 2022-06-27T11:57:04.117 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.083+0000 7f912b7fe700 1 -- 172.21.15.195:0/2227158856 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2062893529 0 0) 0x7f9124001490 con 0x7f912c1508d0 2022-06-27T11:57:04.117 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.083+0000 7f9130ba9700 1 --2- 172.21.15.195:0/2227158856 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f912c150500 0x7f912c063af0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f912c06ce70 tx=0x7f9120009a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b02c11cd79f5f65 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:57:04.117 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.083+0000 7f912b7fe700 1 -- 172.21.15.195:0/2227158856 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9120005ad0 con 0x7f912c150500 2022-06-27T11:57:04.118 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.083+0000 7f912b7fe700 1 -- 172.21.15.195:0/2227158856 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9120005c30 con 0x7f912c150500 2022-06-27T11:57:04.118 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.083+0000 7f912b7fe700 1 -- 172.21.15.195:0/2227158856 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9120006030 con 0x7f912c150500 2022-06-27T11:57:04.119 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.084+0000 7f9132e0d700 1 -- 172.21.15.195:0/2227158856 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f912c150500 msgr2=0x7f912c063af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:04.119 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.084+0000 7f9132e0d700 1 --2- 172.21.15.195:0/2227158856 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f912c150500 0x7f912c063af0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f912c06ce70 tx=0x7f9120009a00 comp rx=0 tx=0).stop 2022-06-27T11:57:04.119 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.084+0000 7f9132e0d700 1 -- 172.21.15.195:0/2227158856 shutdown_connections 2022-06-27T11:57:04.120 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.084+0000 7f9132e0d700 1 --2- 172.21.15.195:0/2227158856 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f912c150500 0x7f912c063af0 secure :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f912c06ce70 tx=0x7f9120009a00 comp rx=0 tx=0).stop 2022-06-27T11:57:04.120 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.084+0000 7f9132e0d700 1 --2- 172.21.15.195:0/2227158856 >> v2:172.21.15.202:3300/0 conn(0x7f912c06a450 0x7f912c0c48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:04.120 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.084+0000 7f9132e0d700 1 -- 172.21.15.195:0/2227158856 >> 172.21.15.195:0/2227158856 conn(0x7f912c0c9b00 msgr2=0x7f912c0c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:57:04.121 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.084+0000 7f9132e0d700 1 -- 172.21.15.195:0/2227158856 shutdown_connections 2022-06-27T11:57:04.121 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.084+0000 7f9132e0d700 1 -- 172.21.15.195:0/2227158856 wait complete. 2022-06-27T11:57:04.121 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.085+0000 7f9132e0d700 1 Processor -- start 2022-06-27T11:57:04.122 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.086+0000 7f9132e0d700 1 -- start start 2022-06-27T11:57:04.122 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.087+0000 7f9132e0d700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f912c06a450 0x7f912c14bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:04.122 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.087+0000 7f9132e0d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f912c1508d0 0x7f912c0c55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:04.122 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.087+0000 7f9132e0d700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f912c0c5b30 0x7f912c169820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:04.122 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.087+0000 7f9132e0d700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f912c0c2ae0 con 0x7f912c1508d0 2022-06-27T11:57:04.123 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.087+0000 7f9132e0d700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f912c0c3420 con 0x7f912c06a450 2022-06-27T11:57:04.123 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.087+0000 7f9132e0d700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f912c14c320 con 0x7f912c0c5b30 2022-06-27T11:57:04.123 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.087+0000 7f912bfff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f912c1508d0 0x7f912c0c55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:04.124 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.087+0000 7f9130ba9700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f912c06a450 0x7f912c14bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:04.124 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.087+0000 7f912bfff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f912c1508d0 0x7f912c0c55f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45068/0 (socket says 172.21.15.195:45068) 2022-06-27T11:57:04.124 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.087+0000 7f912bfff700 1 -- 172.21.15.195:0/3555681166 learned_addr learned my addr 172.21.15.195:0/3555681166 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:57:04.124 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.087+0000 7f91313aa700 1 --2- 172.21.15.195:0/3555681166 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f912c0c5b30 0x7f912c169820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:04.125 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.087+0000 7f912bfff700 1 -- 172.21.15.195:0/3555681166 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f912c0c5b30 msgr2=0x7f912c169820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:04.125 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.087+0000 7f912bfff700 1 --2- 172.21.15.195:0/3555681166 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f912c0c5b30 0x7f912c169820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:04.125 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.087+0000 7f912bfff700 1 -- 172.21.15.195:0/3555681166 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f912c06a450 msgr2=0x7f912c14bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:04.125 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.087+0000 7f912bfff700 1 --2- 172.21.15.195:0/3555681166 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f912c06a450 0x7f912c14bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:04.125 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.087+0000 7f912bfff700 1 -- 172.21.15.195:0/3555681166 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9120009290 con 0x7f912c1508d0 2022-06-27T11:57:04.126 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.087+0000 7f9130ba9700 1 --2- 172.21.15.195:0/3555681166 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f912c06a450 0x7f912c14bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:57:04.126 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.087+0000 7f912bfff700 1 --2- 172.21.15.195:0/3555681166 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f912c1508d0 0x7f912c0c55f0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f911800e9d0 tx=0x7f9118009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:57:04.126 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.087+0000 7f9128ff9700 1 -- 172.21.15.195:0/3555681166 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9118013680 con 0x7f912c1508d0 2022-06-27T11:57:04.126 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.088+0000 7f9128ff9700 1 -- 172.21.15.195:0/3555681166 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91180137e0 con 0x7f912c1508d0 2022-06-27T11:57:04.127 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.088+0000 7f9128ff9700 1 -- 172.21.15.195:0/3555681166 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9118013b60 con 0x7f912c1508d0 2022-06-27T11:57:04.127 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.088+0000 7f9132e0d700 1 -- 172.21.15.195:0/3555681166 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f912c151260 con 0x7f912c1508d0 2022-06-27T11:57:04.127 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.088+0000 7f9132e0d700 1 -- 172.21.15.195:0/3555681166 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f912c0c2820 con 0x7f912c1508d0 2022-06-27T11:57:04.127 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.088+0000 7f9132e0d700 1 -- 172.21.15.195:0/3555681166 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f912c0c5f00 con 0x7f912c1508d0 2022-06-27T11:57:04.128 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.089+0000 7f9128ff9700 1 -- 172.21.15.195:0/3555681166 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f9118023070 con 0x7f912c1508d0 2022-06-27T11:57:04.128 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.089+0000 7f9128ff9700 1 --2- 172.21.15.195:0/3555681166 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9114063d30 0x7f91140661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:04.128 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.089+0000 7f9128ff9700 1 -- 172.21.15.195:0/3555681166 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6546+0+0 (secure 0 0 0) 0x7f9118080c90 con 0x7f912c1508d0 2022-06-27T11:57:04.128 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.089+0000 7f9130ba9700 1 --2- 172.21.15.195:0/3555681166 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9114063d30 0x7f91140661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:04.129 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.090+0000 7f9130ba9700 1 --2- 172.21.15.195:0/3555681166 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9114063d30 0x7f91140661e0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f912c06cb90 tx=0x7f9120016000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:57:04.129 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.093+0000 7f9128ff9700 1 -- 172.21.15.195:0/3555681166 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f911804f630 con 0x7f912c1508d0 2022-06-27T11:57:04.322 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.319+0000 7f9132e0d700 1 -- 172.21.15.195:0/3555681166 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f912c150d90 con 0x7f912c1508d0 2022-06-27T11:57:04.322 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.319+0000 7f9128ff9700 1 -- 172.21.15.195:0/3555681166 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+5367 (secure 0 0 0) 0x7f912c150d90 con 0x7f912c1508d0 2022-06-27T11:57:04.323 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:57:04.323 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:56:59.959866+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4976},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4976":{"gid":4976,"name":"b","rank":0,"incarnation":32,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6833/4154570751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":4154570751},{"type":"v1","addr":"172.21.15.202:6833","nonce":4154570751}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:57:04.324 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.321+0000 7f9132e0d700 1 -- 172.21.15.195:0/3555681166 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9114063d30 msgr2=0x7f91140661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:04.325 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.321+0000 7f9132e0d700 1 --2- 172.21.15.195:0/3555681166 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9114063d30 0x7f91140661e0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f912c06cb90 tx=0x7f9120016000 comp rx=0 tx=0).stop 2022-06-27T11:57:04.325 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.321+0000 7f9132e0d700 1 -- 172.21.15.195:0/3555681166 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f912c1508d0 msgr2=0x7f912c0c55f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:04.325 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.321+0000 7f9132e0d700 1 --2- 172.21.15.195:0/3555681166 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f912c1508d0 0x7f912c0c55f0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f911800e9d0 tx=0x7f9118009060 comp rx=0 tx=0).stop 2022-06-27T11:57:04.325 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.321+0000 7f9132e0d700 1 -- 172.21.15.195:0/3555681166 shutdown_connections 2022-06-27T11:57:04.326 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.321+0000 7f9132e0d700 1 --2- 172.21.15.195:0/3555681166 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9114063d30 0x7f91140661e0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:04.326 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.321+0000 7f9132e0d700 1 --2- 172.21.15.195:0/3555681166 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f912c0c5b30 0x7f912c169820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:04.326 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.321+0000 7f9132e0d700 1 --2- 172.21.15.195:0/3555681166 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f912c06a450 0x7f912c14bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:04.326 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.321+0000 7f9132e0d700 1 --2- 172.21.15.195:0/3555681166 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f912c1508d0 0x7f912c0c55f0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:04.327 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.321+0000 7f9132e0d700 1 -- 172.21.15.195:0/3555681166 >> 172.21.15.195:0/3555681166 conn(0x7f912c0c9b00 msgr2=0x7f912c14fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:57:04.327 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.321+0000 7f9132e0d700 1 -- 172.21.15.195:0/3555681166 shutdown_connections 2022-06-27T11:57:04.328 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:04.321+0000 7f9132e0d700 1 -- 172.21.15.195:0/3555681166 wait complete. 2022-06-27T11:57:04.328 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 32 2022-06-27T11:57:04.340 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T11:57:06.342 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:57:06.513 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.485+0000 7f7e307c5700 1 Processor -- start 2022-06-27T11:57:06.514 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.486+0000 7f7e307c5700 1 -- start start 2022-06-27T11:57:06.515 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.487+0000 7f7e307c5700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f7e2814e860 0x7f7e280c3840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:06.515 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.487+0000 7f7e307c5700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7e280c3d80 0x7f7e28157890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:06.516 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.487+0000 7f7e307c5700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e280c33b0 con 0x7f7e2814e490 2022-06-27T11:57:06.516 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.487+0000 7f7e307c5700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e2806f220 con 0x7f7e2814e860 2022-06-27T11:57:06.516 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.487+0000 7f7e307c5700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e281542c0 con 0x7f7e280c3d80 2022-06-27T11:57:06.517 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.487+0000 7f7e2e561700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f7e2814e490 0x7f7e2806cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:51262/0 (socket says 172.21.15.195:51262) 2022-06-27T11:57:06.517 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.487+0000 7f7e2e561700 1 -- 172.21.15.195:0/1402984286 learned_addr learned my addr 172.21.15.195:0/1402984286 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:57:06.518 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.487+0000 7f7e2dd60700 1 --2- 172.21.15.195:0/1402984286 >> v2:172.21.15.202:3300/0 conn(0x7f7e2814e860 0x7f7e280c3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:06.518 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.487+0000 7f7e2ed62700 1 --2- 172.21.15.195:0/1402984286 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7e280c3d80 0x7f7e28157890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:06.519 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.487+0000 7f7e2dd60700 1 -- 172.21.15.195:0/1402984286 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7e280c3d80 msgr2=0x7f7e28157890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:57:06.519 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.487+0000 7f7e2dd60700 1 --2- 172.21.15.195:0/1402984286 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7e280c3d80 0x7f7e28157890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:06.519 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.487+0000 7f7e2dd60700 1 -- 172.21.15.195:0/1402984286 >> v1:172.21.15.195:6789/0 conn(0x7f7e2814e490 legacy=0x7f7e2806cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:06.520 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.487+0000 7f7e2dd60700 1 -- 172.21.15.195:0/1402984286 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e28154560 con 0x7f7e2814e860 2022-06-27T11:57:06.520 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.487+0000 7f7e2ed62700 1 --2- 172.21.15.195:0/1402984286 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7e280c3d80 0x7f7e28157890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:57:06.521 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.487+0000 7f7e2dd60700 1 --2- 172.21.15.195:0/1402984286 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7e2814e860 0x7f7e280c3840 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7e240097a0 tx=0x7f7e24009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=45fa35a3c3c77a23 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:57:06.521 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.487+0000 7f7e2d55f700 1 -- 172.21.15.195:0/1402984286 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e24011630 con 0x7f7e2814e860 2022-06-27T11:57:06.521 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.488+0000 7f7e2d55f700 1 -- 172.21.15.195:0/1402984286 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e24011790 con 0x7f7e2814e860 2022-06-27T11:57:06.522 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.488+0000 7f7e2d55f700 1 -- 172.21.15.195:0/1402984286 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e24011b30 con 0x7f7e2814e860 2022-06-27T11:57:06.522 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.488+0000 7f7e307c5700 1 -- 172.21.15.195:0/1402984286 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7e2814e860 msgr2=0x7f7e280c3840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:06.523 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.488+0000 7f7e307c5700 1 --2- 172.21.15.195:0/1402984286 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7e2814e860 0x7f7e280c3840 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7e240097a0 tx=0x7f7e24009060 comp rx=0 tx=0).stop 2022-06-27T11:57:06.523 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.488+0000 7f7e307c5700 1 -- 172.21.15.195:0/1402984286 shutdown_connections 2022-06-27T11:57:06.524 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.488+0000 7f7e307c5700 1 --2- 172.21.15.195:0/1402984286 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7e280c3d80 0x7f7e28157890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:06.524 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.488+0000 7f7e307c5700 1 --2- 172.21.15.195:0/1402984286 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7e2814e860 0x7f7e280c3840 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:06.524 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.488+0000 7f7e307c5700 1 -- 172.21.15.195:0/1402984286 >> 172.21.15.195:0/1402984286 conn(0x7f7e28063af0 msgr2=0x7f7e28063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:57:06.529 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.488+0000 7f7e307c5700 1 -- 172.21.15.195:0/1402984286 shutdown_connections 2022-06-27T11:57:06.530 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.488+0000 7f7e307c5700 1 -- 172.21.15.195:0/1402984286 wait complete. 2022-06-27T11:57:06.530 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.489+0000 7f7e307c5700 1 Processor -- start 2022-06-27T11:57:06.531 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.489+0000 7f7e307c5700 1 -- start start 2022-06-27T11:57:06.531 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.490+0000 7f7e307c5700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7e280c3d80 0x7f7e2814fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:06.531 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.490+0000 7f7e307c5700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7e2814e490 0x7f7e28156a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:06.531 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.490+0000 7f7e307c5700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7e2814e860 0x7f7e28149490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:06.532 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.490+0000 7f7e307c5700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e281581f0 con 0x7f7e2814e490 2022-06-27T11:57:06.532 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.490+0000 7f7e307c5700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e281565f0 con 0x7f7e280c3d80 2022-06-27T11:57:06.532 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.490+0000 7f7e307c5700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e28157f10 con 0x7f7e2814e860 2022-06-27T11:57:06.532 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.490+0000 7f7e2dd60700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7e2814e490 0x7f7e28156a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:06.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.490+0000 7f7e2dd60700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7e2814e490 0x7f7e28156a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45082/0 (socket says 172.21.15.195:45082) 2022-06-27T11:57:06.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.490+0000 7f7e2dd60700 1 -- 172.21.15.195:0/2032352765 learned_addr learned my addr 172.21.15.195:0/2032352765 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:57:06.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.490+0000 7f7e2e561700 1 --2- 172.21.15.195:0/2032352765 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7e280c3d80 0x7f7e2814fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:06.534 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.490+0000 7f7e2ed62700 1 --2- 172.21.15.195:0/2032352765 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7e2814e860 0x7f7e28149490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:06.534 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.490+0000 7f7e2dd60700 1 -- 172.21.15.195:0/2032352765 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7e2814e860 msgr2=0x7f7e28149490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:06.534 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.490+0000 7f7e2dd60700 1 --2- 172.21.15.195:0/2032352765 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7e2814e860 0x7f7e28149490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:06.535 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.490+0000 7f7e2dd60700 1 -- 172.21.15.195:0/2032352765 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7e280c3d80 msgr2=0x7f7e2814fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:06.535 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.490+0000 7f7e2dd60700 1 --2- 172.21.15.195:0/2032352765 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7e280c3d80 0x7f7e2814fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:06.535 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.490+0000 7f7e2dd60700 1 -- 172.21.15.195:0/2032352765 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e20009a20 con 0x7f7e2814e490 2022-06-27T11:57:06.536 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.491+0000 7f7e2dd60700 1 --2- 172.21.15.195:0/2032352765 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7e2814e490 0x7f7e28156a80 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f7e24009e40 tx=0x7f7e24017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:57:06.536 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.491+0000 7f7e167fc700 1 -- 172.21.15.195:0/2032352765 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e24000c90 con 0x7f7e2814e490 2022-06-27T11:57:06.536 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.491+0000 7f7e167fc700 1 -- 172.21.15.195:0/2032352765 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e24011900 con 0x7f7e2814e490 2022-06-27T11:57:06.536 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.491+0000 7f7e307c5700 1 -- 172.21.15.195:0/2032352765 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e240092e0 con 0x7f7e2814e490 2022-06-27T11:57:06.537 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.491+0000 7f7e167fc700 1 -- 172.21.15.195:0/2032352765 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e24025420 con 0x7f7e2814e490 2022-06-27T11:57:06.537 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.491+0000 7f7e307c5700 1 -- 172.21.15.195:0/2032352765 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e28155fa0 con 0x7f7e2814e490 2022-06-27T11:57:06.537 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.492+0000 7f7e307c5700 1 -- 172.21.15.195:0/2032352765 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e28156760 con 0x7f7e2814e490 2022-06-27T11:57:06.537 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.492+0000 7f7e167fc700 1 -- 172.21.15.195:0/2032352765 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f7e2402c030 con 0x7f7e2814e490 2022-06-27T11:57:06.538 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.492+0000 7f7e167fc700 1 --2- 172.21.15.195:0/2032352765 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7e08063d00 0x7f7e080661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:06.538 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.493+0000 7f7e167fc700 1 -- 172.21.15.195:0/2032352765 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6546+0+0 (secure 0 0 0) 0x7f7e24081fd0 con 0x7f7e2814e490 2022-06-27T11:57:06.539 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.493+0000 7f7e2e561700 1 --2- 172.21.15.195:0/2032352765 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7e08063d00 0x7f7e080661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:06.539 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.493+0000 7f7e2e561700 1 --2- 172.21.15.195:0/2032352765 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7e08063d00 0x7f7e080661b0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f7e18007fd0 tx=0x7f7e18007d70 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:57:06.539 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.497+0000 7f7e167fc700 1 -- 172.21.15.195:0/2032352765 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f7e240509a0 con 0x7f7e2814e490 2022-06-27T11:57:06.728 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.723+0000 7f7e307c5700 1 -- 172.21.15.195:0/2032352765 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e280722c0 con 0x7f7e2814e490 2022-06-27T11:57:06.728 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.723+0000 7f7e167fc700 1 -- 172.21.15.195:0/2032352765 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+5367 (secure 0 0 0) 0x7f7e2402a070 con 0x7f7e2814e490 2022-06-27T11:57:06.729 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:57:06.730 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:56:59.959866+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4976},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4976":{"gid":4976,"name":"b","rank":0,"incarnation":32,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6833/4154570751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":4154570751},{"type":"v1","addr":"172.21.15.202:6833","nonce":4154570751}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:57:06.733 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.725+0000 7f7e307c5700 1 -- 172.21.15.195:0/2032352765 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7e08063d00 msgr2=0x7f7e080661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:06.733 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.725+0000 7f7e307c5700 1 --2- 172.21.15.195:0/2032352765 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7e08063d00 0x7f7e080661b0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f7e18007fd0 tx=0x7f7e18007d70 comp rx=0 tx=0).stop 2022-06-27T11:57:06.734 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.725+0000 7f7e307c5700 1 -- 172.21.15.195:0/2032352765 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7e2814e490 msgr2=0x7f7e28156a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:06.734 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.725+0000 7f7e307c5700 1 --2- 172.21.15.195:0/2032352765 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7e2814e490 0x7f7e28156a80 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f7e24009e40 tx=0x7f7e24017000 comp rx=0 tx=0).stop 2022-06-27T11:57:06.735 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.725+0000 7f7e307c5700 1 -- 172.21.15.195:0/2032352765 shutdown_connections 2022-06-27T11:57:06.735 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.725+0000 7f7e307c5700 1 --2- 172.21.15.195:0/2032352765 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7e08063d00 0x7f7e080661b0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:06.735 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.725+0000 7f7e307c5700 1 --2- 172.21.15.195:0/2032352765 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7e2814e860 0x7f7e28149490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:06.736 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.725+0000 7f7e307c5700 1 --2- 172.21.15.195:0/2032352765 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7e280c3d80 0x7f7e2814fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:06.736 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.725+0000 7f7e307c5700 1 --2- 172.21.15.195:0/2032352765 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7e2814e490 0x7f7e28156a80 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:06.737 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.725+0000 7f7e307c5700 1 -- 172.21.15.195:0/2032352765 >> 172.21.15.195:0/2032352765 conn(0x7f7e28063af0 msgr2=0x7f7e280c9d50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:57:06.738 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.725+0000 7f7e307c5700 1 -- 172.21.15.195:0/2032352765 shutdown_connections 2022-06-27T11:57:06.739 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:06.725+0000 7f7e307c5700 1 -- 172.21.15.195:0/2032352765 wait complete. 2022-06-27T11:57:06.739 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 32 2022-06-27T11:57:06.741 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T11:57:08.745 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:57:08.922 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:08.879+0000 7ffb03c95700 1 Processor -- start 2022-06-27T11:57:08.923 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:08.879+0000 7ffb03c95700 1 -- start start 2022-06-27T11:57:08.923 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:08.880+0000 7ffb03c95700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ffafc14e860 0x7ffafc0c3840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:08.923 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:08.880+0000 7ffb03c95700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffafc0c33b0 con 0x7ffafc0c3d80 2022-06-27T11:57:08.924 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:08.880+0000 7ffb03c95700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffafc0c4200 con 0x7ffafc14e490 2022-06-27T11:57:08.924 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:08.880+0000 7ffb03c95700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffafc06f220 con 0x7ffafc14e860 2022-06-27T11:57:08.924 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:08.880+0000 7ffb02232700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7ffafc0c3d80 0x7ffafc157890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:51280/0 (socket says 172.21.15.195:51280) 2022-06-27T11:57:08.924 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:08.880+0000 7ffb02232700 1 -- 172.21.15.195:0/2861955344 learned_addr learned my addr 172.21.15.195:0/2861955344 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:57:08.925 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:08.880+0000 7ffb01230700 1 --2- 172.21.15.195:0/2861955344 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ffafc14e860 0x7ffafc0c3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:08.925 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:08.880+0000 7ffb00a2f700 1 -- 172.21.15.195:0/2861955344 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2668407980 0 0) 0x7ffafc0c33b0 con 0x7ffafc0c3d80 2022-06-27T11:57:08.929 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:08.880+0000 7ffb00a2f700 1 -- 172.21.15.195:0/2861955344 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffae4003430 con 0x7ffafc0c3d80 2022-06-27T11:57:08.941 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:08.880+0000 7ffb00a2f700 1 -- 172.21.15.195:0/2861955344 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3173611074 0 0) 0x7ffafc0c4200 con 0x7ffafc14e490 2022-06-27T11:57:08.941 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:08.880+0000 7ffb00a2f700 1 -- 172.21.15.195:0/2861955344 --> v1:172.21.15.202:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffafc0c33b0 con 0x7ffafc14e490 2022-06-27T11:57:08.942 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:08.880+0000 7ffb01230700 1 -- 172.21.15.195:0/2861955344 >> v1:172.21.15.202:6789/0 conn(0x7ffafc14e490 legacy=0x7ffafc06cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:08.943 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:08.880+0000 7ffb01230700 1 -- 172.21.15.195:0/2861955344 >> v1:172.21.15.195:6789/0 conn(0x7ffafc0c3d80 legacy=0x7ffafc157890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:08.943 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:08.880+0000 7ffb01230700 1 -- 172.21.15.195:0/2861955344 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffafc1542c0 con 0x7ffafc14e860 2022-06-27T11:57:08.943 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:08.881+0000 7ffb01230700 1 --2- 172.21.15.195:0/2861955344 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ffafc14e860 0x7ffafc0c3840 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ffaec0097d0 tx=0x7ffaec009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c49e665ae42d3fd4 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:57:08.944 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:08.881+0000 7ffb00a2f700 1 -- 172.21.15.195:0/2861955344 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaec011630 con 0x7ffafc14e860 2022-06-27T11:57:08.944 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:08.881+0000 7ffb00a2f700 1 -- 172.21.15.195:0/2861955344 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaec011790 con 0x7ffafc14e860 2022-06-27T11:57:08.945 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:08.881+0000 7ffb00a2f700 1 -- 172.21.15.195:0/2861955344 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaec011b30 con 0x7ffafc14e860 2022-06-27T11:57:08.945 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:08.881+0000 7ffb03c95700 1 -- 172.21.15.195:0/2861955344 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ffafc14e860 msgr2=0x7ffafc0c3840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:08.946 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:08.881+0000 7ffb03c95700 1 --2- 172.21.15.195:0/2861955344 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ffafc14e860 0x7ffafc0c3840 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ffaec0097d0 tx=0x7ffaec009060 comp rx=0 tx=0).stop 2022-06-27T11:57:08.946 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:08.881+0000 7ffb03c95700 1 -- 172.21.15.195:0/2861955344 shutdown_connections 2022-06-27T11:57:08.946 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:08.881+0000 7ffb03c95700 1 --2- 172.21.15.195:0/2861955344 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ffafc14e860 0x7ffafc0c3840 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:08.947 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:08.881+0000 7ffb03c95700 1 -- 172.21.15.195:0/2861955344 >> 172.21.15.195:0/2861955344 conn(0x7ffafc063af0 msgr2=0x7ffafc063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:57:08.947 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:08.881+0000 7ffb03c95700 1 -- 172.21.15.195:0/2861955344 shutdown_connections 2022-06-27T11:57:08.948 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:08.881+0000 7ffb03c95700 1 -- 172.21.15.195:0/2861955344 wait complete. 2022-06-27T11:57:08.948 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:08.882+0000 7ffb03c95700 1 Processor -- start 2022-06-27T11:57:08.949 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:08.882+0000 7ffb03c95700 1 -- start start 2022-06-27T11:57:08.949 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:08.883+0000 7ffb03c95700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ffafc0c3d80 0x7ffafc157830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:08.950 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:08.883+0000 7ffb03c95700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ffafc14e490 0x7ffafc0bfbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:08.950 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:08.883+0000 7ffb03c95700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffafc14e860 0x7ffafc1497e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:08.951 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:08.883+0000 7ffb03c95700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffafc0bf020 con 0x7ffafc14e860 2022-06-27T11:57:08.951 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:08.883+0000 7ffb03c95700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffafc1573a0 con 0x7ffafc0c3d80 2022-06-27T11:57:08.952 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:08.883+0000 7ffb03c95700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffafc149d20 con 0x7ffafc14e490 2022-06-27T11:57:08.952 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:08.883+0000 7ffb02232700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffafc14e860 0x7ffafc1497e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:08.953 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:08.883+0000 7ffb02232700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffafc14e860 0x7ffafc1497e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45098/0 (socket says 172.21.15.195:45098) 2022-06-27T11:57:08.953 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:08.883+0000 7ffb02232700 1 -- 172.21.15.195:0/499166790 learned_addr learned my addr 172.21.15.195:0/499166790 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:57:08.954 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:08.883+0000 7ffb01230700 1 --2- 172.21.15.195:0/499166790 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ffafc14e490 0x7ffafc0bfbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:08.954 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:08.883+0000 7ffb01a31700 1 --2- 172.21.15.195:0/499166790 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ffafc0c3d80 0x7ffafc157830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:08.955 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:08.884+0000 7ffb02232700 1 -- 172.21.15.195:0/499166790 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ffafc14e490 msgr2=0x7ffafc0bfbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:08.955 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:08.884+0000 7ffb02232700 1 --2- 172.21.15.195:0/499166790 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ffafc14e490 0x7ffafc0bfbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:08.956 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:08.884+0000 7ffb02232700 1 -- 172.21.15.195:0/499166790 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ffafc0c3d80 msgr2=0x7ffafc157830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:08.956 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:08.884+0000 7ffb02232700 1 --2- 172.21.15.195:0/499166790 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ffafc0c3d80 0x7ffafc157830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:08.956 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:08.884+0000 7ffb02232700 1 -- 172.21.15.195:0/499166790 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaec0092e0 con 0x7ffafc14e860 2022-06-27T11:57:08.957 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:08.884+0000 7ffb02232700 1 --2- 172.21.15.195:0/499166790 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffafc14e860 0x7ffafc1497e0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7ffaf4001560 tx=0x7ffaf40015b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:57:08.957 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:08.884+0000 7ffae9ffb700 1 -- 172.21.15.195:0/499166790 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf4005d70 con 0x7ffafc14e860 2022-06-27T11:57:08.958 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:08.884+0000 7ffae9ffb700 1 -- 172.21.15.195:0/499166790 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaf40070d0 con 0x7ffafc14e860 2022-06-27T11:57:08.958 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:08.884+0000 7ffb03c95700 1 -- 172.21.15.195:0/499166790 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffafc155140 con 0x7ffafc14e860 2022-06-27T11:57:08.959 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:08.884+0000 7ffae9ffb700 1 -- 172.21.15.195:0/499166790 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf4007490 con 0x7ffafc14e860 2022-06-27T11:57:08.959 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:08.884+0000 7ffb03c95700 1 -- 172.21.15.195:0/499166790 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffafc156a70 con 0x7ffafc14e860 2022-06-27T11:57:08.960 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:08.885+0000 7ffb03c95700 1 -- 172.21.15.195:0/499166790 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffafc0722c0 con 0x7ffafc14e860 2022-06-27T11:57:08.960 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:08.885+0000 7ffae9ffb700 1 -- 172.21.15.195:0/499166790 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7ffaf4019030 con 0x7ffafc14e860 2022-06-27T11:57:08.960 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:08.886+0000 7ffae9ffb700 1 --2- 172.21.15.195:0/499166790 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ffadc063d00 0x7ffadc0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:08.961 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:08.886+0000 7ffae9ffb700 1 -- 172.21.15.195:0/499166790 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6546+0+0 (secure 0 0 0) 0x7ffaf407feb0 con 0x7ffafc14e860 2022-06-27T11:57:08.961 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:08.886+0000 7ffb01a31700 1 --2- 172.21.15.195:0/499166790 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ffadc063d00 0x7ffadc0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:08.962 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:08.886+0000 7ffb01a31700 1 --2- 172.21.15.195:0/499166790 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ffadc063d00 0x7ffadc0661b0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ffaf80018b0 tx=0x7ffaf8009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:57:08.962 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:08.890+0000 7ffae9ffb700 1 -- 172.21.15.195:0/499166790 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7ffaf404e8b0 con 0x7ffafc14e860 2022-06-27T11:57:09.119 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:09.118+0000 7ffb03c95700 1 -- 172.21.15.195:0/499166790 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffafc1574e0 con 0x7ffafc14e860 2022-06-27T11:57:09.119 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:09.118+0000 7ffae9ffb700 1 -- 172.21.15.195:0/499166790 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+5367 (secure 0 0 0) 0x7ffaf4017070 con 0x7ffafc14e860 2022-06-27T11:57:09.120 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:57:09.121 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:56:59.959866+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4976},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4976":{"gid":4976,"name":"b","rank":0,"incarnation":32,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6833/4154570751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":4154570751},{"type":"v1","addr":"172.21.15.202:6833","nonce":4154570751}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:57:09.123 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:09.120+0000 7ffb03c95700 1 -- 172.21.15.195:0/499166790 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ffadc063d00 msgr2=0x7ffadc0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:09.123 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:09.120+0000 7ffb03c95700 1 --2- 172.21.15.195:0/499166790 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ffadc063d00 0x7ffadc0661b0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ffaf80018b0 tx=0x7ffaf8009040 comp rx=0 tx=0).stop 2022-06-27T11:57:09.124 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:09.120+0000 7ffb03c95700 1 -- 172.21.15.195:0/499166790 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffafc14e860 msgr2=0x7ffafc1497e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:09.124 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:09.120+0000 7ffb03c95700 1 --2- 172.21.15.195:0/499166790 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffafc14e860 0x7ffafc1497e0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7ffaf4001560 tx=0x7ffaf40015b0 comp rx=0 tx=0).stop 2022-06-27T11:57:09.125 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:09.120+0000 7ffb03c95700 1 -- 172.21.15.195:0/499166790 shutdown_connections 2022-06-27T11:57:09.125 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:09.120+0000 7ffb03c95700 1 --2- 172.21.15.195:0/499166790 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ffadc063d00 0x7ffadc0661b0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:09.126 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:09.120+0000 7ffb03c95700 1 --2- 172.21.15.195:0/499166790 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffafc14e860 0x7ffafc1497e0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:09.126 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:09.120+0000 7ffb03c95700 1 --2- 172.21.15.195:0/499166790 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ffafc0c3d80 0x7ffafc157830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:09.127 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:09.120+0000 7ffb03c95700 1 --2- 172.21.15.195:0/499166790 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ffafc14e490 0x7ffafc0bfbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:09.127 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:09.120+0000 7ffb03c95700 1 -- 172.21.15.195:0/499166790 >> 172.21.15.195:0/499166790 conn(0x7ffafc063af0 msgr2=0x7ffafc1528f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:57:09.128 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:09.120+0000 7ffb03c95700 1 -- 172.21.15.195:0/499166790 shutdown_connections 2022-06-27T11:57:09.129 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:09.120+0000 7ffb03c95700 1 -- 172.21.15.195:0/499166790 wait complete. 2022-06-27T11:57:09.129 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 32 2022-06-27T11:57:09.132 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T11:57:11.134 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:57:11.305 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.278+0000 7f62e29d8700 1 Processor -- start 2022-06-27T11:57:11.306 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.278+0000 7f62e29d8700 1 -- start start 2022-06-27T11:57:11.306 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.280+0000 7f62e29d8700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f62dc06f620 0x7f62dc156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:11.306 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.280+0000 7f62e29d8700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62dc0c76b0 con 0x7f62dc150950 2022-06-27T11:57:11.307 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.280+0000 7f62e29d8700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62dc0c8250 con 0x7f62dc150580 2022-06-27T11:57:11.307 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.280+0000 7f62e29d8700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62dc157430 con 0x7f62dc06f620 2022-06-27T11:57:11.307 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.280+0000 7f62db7fe700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f62dc150950 0x7f62dc06d6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:51300/0 (socket says 172.21.15.195:51300) 2022-06-27T11:57:11.307 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.280+0000 7f62db7fe700 1 -- 172.21.15.195:0/1694441533 learned_addr learned my addr 172.21.15.195:0/1694441533 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:57:11.308 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.280+0000 7f62e0f75700 1 --2- 172.21.15.195:0/1694441533 >> v2:172.21.15.202:3300/0 conn(0x7f62dc06f620 0x7f62dc156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:11.308 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.280+0000 7f62daffd700 1 -- 172.21.15.195:0/1694441533 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3582176970 0 0) 0x7f62dc0c76b0 con 0x7f62dc150950 2022-06-27T11:57:11.308 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.281+0000 7f62daffd700 1 -- 172.21.15.195:0/1694441533 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62c8003430 con 0x7f62dc150950 2022-06-27T11:57:11.308 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.281+0000 7f62daffd700 1 -- 172.21.15.195:0/1694441533 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4210320847 0 0) 0x7f62dc0c8250 con 0x7f62dc150580 2022-06-27T11:57:11.309 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.281+0000 7f62daffd700 1 -- 172.21.15.195:0/1694441533 --> v1:172.21.15.202:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62dc0c76b0 con 0x7f62dc150580 2022-06-27T11:57:11.309 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.281+0000 7f62e0f75700 1 -- 172.21.15.195:0/1694441533 >> v1:172.21.15.202:6789/0 conn(0x7f62dc150580 legacy=0x7f62dc0c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:11.309 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.281+0000 7f62e0f75700 1 -- 172.21.15.195:0/1694441533 >> v1:172.21.15.195:6789/0 conn(0x7f62dc150950 legacy=0x7f62dc06d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:11.309 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.281+0000 7f62e0f75700 1 -- 172.21.15.195:0/1694441533 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62dc0bad70 con 0x7f62dc06f620 2022-06-27T11:57:11.310 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.281+0000 7f62e0f75700 1 --2- 172.21.15.195:0/1694441533 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f62dc06f620 0x7f62dc156ef0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f62d4009e60 tx=0x7f62d4009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=59ea0034bbc1148 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:57:11.310 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.281+0000 7f62daffd700 1 -- 172.21.15.195:0/1694441533 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62d400e070 con 0x7f62dc06f620 2022-06-27T11:57:11.310 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.281+0000 7f62daffd700 1 -- 172.21.15.195:0/1694441533 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62d400bab0 con 0x7f62dc06f620 2022-06-27T11:57:11.311 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.281+0000 7f62daffd700 1 -- 172.21.15.195:0/1694441533 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62d400be50 con 0x7f62dc06f620 2022-06-27T11:57:11.311 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.282+0000 7f62e29d8700 1 -- 172.21.15.195:0/1694441533 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f62dc06f620 msgr2=0x7f62dc156ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:11.311 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.282+0000 7f62e29d8700 1 --2- 172.21.15.195:0/1694441533 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f62dc06f620 0x7f62dc156ef0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f62d4009e60 tx=0x7f62d4009780 comp rx=0 tx=0).stop 2022-06-27T11:57:11.311 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.282+0000 7f62e29d8700 1 -- 172.21.15.195:0/1694441533 shutdown_connections 2022-06-27T11:57:11.312 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.282+0000 7f62e29d8700 1 --2- 172.21.15.195:0/1694441533 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f62dc06f620 0x7f62dc156ef0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:11.312 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.282+0000 7f62e29d8700 1 -- 172.21.15.195:0/1694441533 >> 172.21.15.195:0/1694441533 conn(0x7f62dc063af0 msgr2=0x7f62dc063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:57:11.312 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.282+0000 7f62e29d8700 1 -- 172.21.15.195:0/1694441533 shutdown_connections 2022-06-27T11:57:11.312 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.282+0000 7f62e29d8700 1 -- 172.21.15.195:0/1694441533 wait complete. 2022-06-27T11:57:11.314 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.284+0000 7f62e29d8700 1 Processor -- start 2022-06-27T11:57:11.315 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.284+0000 7f62e29d8700 1 -- start start 2022-06-27T11:57:11.315 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.285+0000 7f62e29d8700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f62dc06f620 0x7f62dc069b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:11.315 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.285+0000 7f62e29d8700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f62dc150580 0x7f62dc1453c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:11.315 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.285+0000 7f62e29d8700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f62dc150950 0x7f62dc145900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:11.316 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.285+0000 7f62e29d8700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62dc16a8f0 con 0x7f62dc150580 2022-06-27T11:57:11.316 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.285+0000 7f62e29d8700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62dc1583a0 con 0x7f62dc06f620 2022-06-27T11:57:11.316 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.285+0000 7f62e29d8700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62dc169c70 con 0x7f62dc150950 2022-06-27T11:57:11.316 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.285+0000 7f62db7fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f62dc150580 0x7f62dc1453c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:11.317 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.285+0000 7f62db7fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f62dc150580 0x7f62dc1453c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45118/0 (socket says 172.21.15.195:45118) 2022-06-27T11:57:11.317 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.285+0000 7f62db7fe700 1 -- 172.21.15.195:0/368191415 learned_addr learned my addr 172.21.15.195:0/368191415 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:57:11.318 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.285+0000 7f62e0f75700 1 --2- 172.21.15.195:0/368191415 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f62dc150950 0x7f62dc145900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:11.318 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.285+0000 7f62dbfff700 1 --2- 172.21.15.195:0/368191415 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f62dc06f620 0x7f62dc069b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:11.318 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.286+0000 7f62db7fe700 1 -- 172.21.15.195:0/368191415 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f62dc150950 msgr2=0x7f62dc145900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:11.318 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.286+0000 7f62db7fe700 1 --2- 172.21.15.195:0/368191415 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f62dc150950 0x7f62dc145900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:11.319 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.286+0000 7f62db7fe700 1 -- 172.21.15.195:0/368191415 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f62dc06f620 msgr2=0x7f62dc069b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:11.319 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.286+0000 7f62db7fe700 1 --2- 172.21.15.195:0/368191415 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f62dc06f620 0x7f62dc069b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:11.319 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.286+0000 7f62db7fe700 1 -- 172.21.15.195:0/368191415 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62d4009a20 con 0x7f62dc150580 2022-06-27T11:57:11.320 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.286+0000 7f62db7fe700 1 --2- 172.21.15.195:0/368191415 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f62dc150580 0x7f62dc1453c0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f62d000c590 tx=0x7f62d0007ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:57:11.320 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.286+0000 7f62c3fff700 1 -- 172.21.15.195:0/368191415 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62d00072e0 con 0x7f62dc150580 2022-06-27T11:57:11.321 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.286+0000 7f62c3fff700 1 -- 172.21.15.195:0/368191415 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62d0007460 con 0x7f62dc150580 2022-06-27T11:57:11.321 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.286+0000 7f62e29d8700 1 -- 172.21.15.195:0/368191415 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62dc169f50 con 0x7f62dc150580 2022-06-27T11:57:11.321 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.286+0000 7f62c3fff700 1 -- 172.21.15.195:0/368191415 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62d00057c0 con 0x7f62dc150580 2022-06-27T11:57:11.321 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.286+0000 7f62e29d8700 1 -- 172.21.15.195:0/368191415 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62dc16a510 con 0x7f62dc150580 2022-06-27T11:57:11.322 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.288+0000 7f62e29d8700 1 -- 172.21.15.195:0/368191415 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62dc16aa30 con 0x7f62dc150580 2022-06-27T11:57:11.322 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.288+0000 7f62c3fff700 1 -- 172.21.15.195:0/368191415 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f62d0019030 con 0x7f62dc150580 2022-06-27T11:57:11.322 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.289+0000 7f62c3fff700 1 --2- 172.21.15.195:0/368191415 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f62bc063d00 0x7f62bc0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:11.323 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.289+0000 7f62c3fff700 1 -- 172.21.15.195:0/368191415 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6546+0+0 (secure 0 0 0) 0x7f62d007fd10 con 0x7f62dc150580 2022-06-27T11:57:11.323 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.289+0000 7f62dbfff700 1 --2- 172.21.15.195:0/368191415 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f62bc063d00 0x7f62bc0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:11.325 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.290+0000 7f62dbfff700 1 --2- 172.21.15.195:0/368191415 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f62bc063d00 0x7f62bc0661b0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f62cc0018b0 tx=0x7f62cc009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:57:11.325 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.294+0000 7f62c3fff700 1 -- 172.21.15.195:0/368191415 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f62d0085020 con 0x7f62dc150580 2022-06-27T11:57:11.520 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.520+0000 7f62e29d8700 1 -- 172.21.15.195:0/368191415 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f62dc0722c0 con 0x7f62dc150580 2022-06-27T11:57:11.521 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.520+0000 7f62c3fff700 1 -- 172.21.15.195:0/368191415 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+5371 (secure 0 0 0) 0x7f62d004e620 con 0x7f62dc150580 2022-06-27T11:57:11.521 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:57:11.521 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:57:10.963352+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4976},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4976":{"gid":4976,"name":"b","rank":0,"incarnation":32,"state":"up:reconnect","state_seq":22,"addr":"172.21.15.202:6833/4154570751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":4154570751},{"type":"v1","addr":"172.21.15.202:6833","nonce":4154570751}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:57:11.522 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.522+0000 7f62e29d8700 1 -- 172.21.15.195:0/368191415 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f62bc063d00 msgr2=0x7f62bc0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:11.523 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.522+0000 7f62e29d8700 1 --2- 172.21.15.195:0/368191415 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f62bc063d00 0x7f62bc0661b0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f62cc0018b0 tx=0x7f62cc009040 comp rx=0 tx=0).stop 2022-06-27T11:57:11.523 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.522+0000 7f62e29d8700 1 -- 172.21.15.195:0/368191415 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f62dc150580 msgr2=0x7f62dc1453c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:11.523 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.522+0000 7f62e29d8700 1 --2- 172.21.15.195:0/368191415 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f62dc150580 0x7f62dc1453c0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f62d000c590 tx=0x7f62d0007ce0 comp rx=0 tx=0).stop 2022-06-27T11:57:11.523 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.522+0000 7f62e29d8700 1 -- 172.21.15.195:0/368191415 shutdown_connections 2022-06-27T11:57:11.524 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.522+0000 7f62e29d8700 1 --2- 172.21.15.195:0/368191415 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f62bc063d00 0x7f62bc0661b0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:11.524 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.522+0000 7f62e29d8700 1 --2- 172.21.15.195:0/368191415 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f62dc150950 0x7f62dc145900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:11.525 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.522+0000 7f62e29d8700 1 --2- 172.21.15.195:0/368191415 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f62dc06f620 0x7f62dc069b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:11.525 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.522+0000 7f62e29d8700 1 --2- 172.21.15.195:0/368191415 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f62dc150580 0x7f62dc1453c0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:11.526 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.522+0000 7f62e29d8700 1 -- 172.21.15.195:0/368191415 >> 172.21.15.195:0/368191415 conn(0x7f62dc063af0 msgr2=0x7f62dc14b5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:57:11.527 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.523+0000 7f62e29d8700 1 -- 172.21.15.195:0/368191415 shutdown_connections 2022-06-27T11:57:11.527 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:11.523+0000 7f62e29d8700 1 -- 172.21.15.195:0/368191415 wait complete. 2022-06-27T11:57:11.527 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 33 2022-06-27T11:57:11.546 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T11:57:13.548 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:57:13.689 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.687+0000 7fccc6a94700 1 Processor -- start 2022-06-27T11:57:13.689 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.687+0000 7fccc6a94700 1 -- start start 2022-06-27T11:57:13.690 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.688+0000 7fccc6a94700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fccc0153cb0 0x7fccc0156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:13.690 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.688+0000 7fccc6a94700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccc00c1a20 con 0x7fccc006f640 2022-06-27T11:57:13.690 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.688+0000 7fccc6a94700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccc00c1130 con 0x7fccc00c4160 2022-06-27T11:57:13.690 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.688+0000 7fccc6a94700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccc006ea90 con 0x7fccc0153cb0 2022-06-27T11:57:13.691 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.688+0000 7fccc5031700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fccc0153cb0 0x7fccc0156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:13.691 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.688+0000 7fccc5031700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fccc0153cb0 0x7fccc0156ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45128/0 (socket says 172.21.15.195:45128) 2022-06-27T11:57:13.692 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.688+0000 7fccc5031700 1 -- 172.21.15.195:0/348710702 learned_addr learned my addr 172.21.15.195:0/348710702 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:57:13.692 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.688+0000 7fccc5031700 1 -- 172.21.15.195:0/348710702 >> v1:172.21.15.202:6789/0 conn(0x7fccc00c4160 legacy=0x7fccc01535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:57:13.692 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.688+0000 7fccc5031700 1 -- 172.21.15.195:0/348710702 >> v1:172.21.15.195:6789/0 conn(0x7fccc006f640 legacy=0x7fccc00c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:13.693 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.688+0000 7fccc5031700 1 -- 172.21.15.195:0/348710702 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccc006ed70 con 0x7fccc0153cb0 2022-06-27T11:57:13.693 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.688+0000 7fccbf7fe700 1 -- 172.21.15.195:0/348710702 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1482394914 0 0) 0x7fccc00c1a20 con 0x7fccc006f640 2022-06-27T11:57:13.694 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.688+0000 7fccc5031700 1 --2- 172.21.15.195:0/348710702 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fccc0153cb0 0x7fccc0156ed0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fccb8009f20 tx=0x7fccb8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75bb59ebe72d833a server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:57:13.694 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.688+0000 7fccbf7fe700 1 -- 172.21.15.195:0/348710702 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb800e070 con 0x7fccc0153cb0 2022-06-27T11:57:13.694 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.689+0000 7fccbf7fe700 1 -- 172.21.15.195:0/348710702 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccb800bad0 con 0x7fccc0153cb0 2022-06-27T11:57:13.694 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.689+0000 7fccbf7fe700 1 -- 172.21.15.195:0/348710702 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb80092e0 con 0x7fccc0153cb0 2022-06-27T11:57:13.695 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.689+0000 7fccc6a94700 1 -- 172.21.15.195:0/348710702 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fccc0153cb0 msgr2=0x7fccc0156ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:13.696 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.689+0000 7fccc6a94700 1 --2- 172.21.15.195:0/348710702 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fccc0153cb0 0x7fccc0156ed0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fccb8009f20 tx=0x7fccb8009840 comp rx=0 tx=0).stop 2022-06-27T11:57:13.697 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.689+0000 7fccc6a94700 1 -- 172.21.15.195:0/348710702 shutdown_connections 2022-06-27T11:57:13.698 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.689+0000 7fccc6a94700 1 --2- 172.21.15.195:0/348710702 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fccc0153cb0 0x7fccc0156ed0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:13.698 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.689+0000 7fccc6a94700 1 -- 172.21.15.195:0/348710702 >> 172.21.15.195:0/348710702 conn(0x7fccc00c6350 msgr2=0x7fccc00c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:57:13.699 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.689+0000 7fccc6a94700 1 -- 172.21.15.195:0/348710702 shutdown_connections 2022-06-27T11:57:13.699 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.689+0000 7fccc6a94700 1 -- 172.21.15.195:0/348710702 wait complete. 2022-06-27T11:57:13.707 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.691+0000 7fccc6a94700 1 Processor -- start 2022-06-27T11:57:13.708 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.691+0000 7fccc6a94700 1 -- start start 2022-06-27T11:57:13.708 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.692+0000 7fccc6a94700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fccc006f640 0x7fccc016afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:13.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.692+0000 7fccc6a94700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fccc00c4160 0x7fccc00bdf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:13.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.692+0000 7fccc6a94700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fccc0153cb0 0x7fccc00bfd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:13.710 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.692+0000 7fccc6a94700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccc00bea40 con 0x7fccc0153cb0 2022-06-27T11:57:13.710 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.692+0000 7fccc6a94700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccc0154120 con 0x7fccc006f640 2022-06-27T11:57:13.710 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.692+0000 7fccc6a94700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccc00bd540 con 0x7fccc00c4160 2022-06-27T11:57:13.711 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.692+0000 7fccc5031700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fccc0153cb0 0x7fccc00bfd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:13.711 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.692+0000 7fccc5031700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fccc0153cb0 0x7fccc00bfd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45134/0 (socket says 172.21.15.195:45134) 2022-06-27T11:57:13.712 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.692+0000 7fccc5031700 1 -- 172.21.15.195:0/1201602166 learned_addr learned my addr 172.21.15.195:0/1201602166 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:57:13.712 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.692+0000 7fccbffff700 1 --2- 172.21.15.195:0/1201602166 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fccc00c4160 0x7fccc00bdf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:13.712 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.692+0000 7fccc4830700 1 --2- 172.21.15.195:0/1201602166 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fccc006f640 0x7fccc016afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:13.713 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.692+0000 7fccc5031700 1 -- 172.21.15.195:0/1201602166 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fccc00c4160 msgr2=0x7fccc00bdf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:13.713 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.692+0000 7fccc5031700 1 --2- 172.21.15.195:0/1201602166 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fccc00c4160 0x7fccc00bdf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:13.713 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.692+0000 7fccc5031700 1 -- 172.21.15.195:0/1201602166 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fccc006f640 msgr2=0x7fccc016afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:13.714 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.692+0000 7fccc5031700 1 --2- 172.21.15.195:0/1201602166 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fccc006f640 0x7fccc016afa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:13.714 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.692+0000 7fccc5031700 1 -- 172.21.15.195:0/1201602166 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccb8009ae0 con 0x7fccc0153cb0 2022-06-27T11:57:13.715 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.692+0000 7fccc5031700 1 --2- 172.21.15.195:0/1201602166 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fccc0153cb0 0x7fccc00bfd90 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fccc006e350 tx=0x7fccb8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:57:13.716 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.692+0000 7fccbcff9700 1 -- 172.21.15.195:0/1201602166 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb8004240 con 0x7fccc0153cb0 2022-06-27T11:57:13.718 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.692+0000 7fccbcff9700 1 -- 172.21.15.195:0/1201602166 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccb80043a0 con 0x7fccc0153cb0 2022-06-27T11:57:13.718 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.692+0000 7fccbcff9700 1 -- 172.21.15.195:0/1201602166 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb8004760 con 0x7fccc0153cb0 2022-06-27T11:57:13.718 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.692+0000 7fccc6a94700 1 -- 172.21.15.195:0/1201602166 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccc00bdb00 con 0x7fccc0153cb0 2022-06-27T11:57:13.719 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.692+0000 7fccc6a94700 1 -- 172.21.15.195:0/1201602166 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccc0154650 con 0x7fccc0153cb0 2022-06-27T11:57:13.719 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.693+0000 7fccc6a94700 1 -- 172.21.15.195:0/1201602166 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccc0065f30 con 0x7fccc0153cb0 2022-06-27T11:57:13.720 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.694+0000 7fccbcff9700 1 -- 172.21.15.195:0/1201602166 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fccb801e640 con 0x7fccc0153cb0 2022-06-27T11:57:13.720 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.694+0000 7fccbcff9700 1 --2- 172.21.15.195:0/1201602166 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcca8063d00 0x7fcca80661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:13.721 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.694+0000 7fccbcff9700 1 -- 172.21.15.195:0/1201602166 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6546+0+0 (secure 0 0 0) 0x7fccb80825b0 con 0x7fccc0153cb0 2022-06-27T11:57:13.721 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.694+0000 7fccc4830700 1 --2- 172.21.15.195:0/1201602166 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcca8063d00 0x7fcca80661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:13.722 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.695+0000 7fccc4830700 1 --2- 172.21.15.195:0/1201602166 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcca8063d00 0x7fcca80661b0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fccb4001690 tx=0x7fccb4009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:57:13.722 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.698+0000 7fccbcff9700 1 -- 172.21.15.195:0/1201602166 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fccc0065f30 con 0x7fccc0153cb0 2022-06-27T11:57:13.924 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.923+0000 7fccc6a94700 1 -- 172.21.15.195:0/1201602166 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fccc0065f30 con 0x7fccc0153cb0 2022-06-27T11:57:13.925 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.924+0000 7fccbcff9700 1 -- 172.21.15.195:0/1201602166 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+5371 (secure 0 0 0) 0x7fccc0065f30 con 0x7fccc0153cb0 2022-06-27T11:57:13.926 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:57:13.927 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:57:10.963352+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4976},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4976":{"gid":4976,"name":"b","rank":0,"incarnation":32,"state":"up:reconnect","state_seq":22,"addr":"172.21.15.202:6833/4154570751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":4154570751},{"type":"v1","addr":"172.21.15.202:6833","nonce":4154570751}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:57:13.929 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.926+0000 7fccc6a94700 1 -- 172.21.15.195:0/1201602166 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcca8063d00 msgr2=0x7fcca80661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:13.930 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.926+0000 7fccc6a94700 1 --2- 172.21.15.195:0/1201602166 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcca8063d00 0x7fcca80661b0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fccb4001690 tx=0x7fccb4009040 comp rx=0 tx=0).stop 2022-06-27T11:57:13.930 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.926+0000 7fccc6a94700 1 -- 172.21.15.195:0/1201602166 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fccc0153cb0 msgr2=0x7fccc00bfd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:13.931 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.926+0000 7fccc6a94700 1 --2- 172.21.15.195:0/1201602166 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fccc0153cb0 0x7fccc00bfd90 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fccc006e350 tx=0x7fccb8017000 comp rx=0 tx=0).stop 2022-06-27T11:57:13.931 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.926+0000 7fccc6a94700 1 -- 172.21.15.195:0/1201602166 shutdown_connections 2022-06-27T11:57:13.932 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.926+0000 7fccc6a94700 1 --2- 172.21.15.195:0/1201602166 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcca8063d00 0x7fcca80661b0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:13.932 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.926+0000 7fccc6a94700 1 --2- 172.21.15.195:0/1201602166 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fccc0153cb0 0x7fccc00bfd90 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:13.938 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.926+0000 7fccc6a94700 1 --2- 172.21.15.195:0/1201602166 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fccc006f640 0x7fccc016afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:13.938 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.926+0000 7fccc6a94700 1 --2- 172.21.15.195:0/1201602166 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fccc00c4160 0x7fccc00bdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:13.938 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.926+0000 7fccc6a94700 1 -- 172.21.15.195:0/1201602166 >> 172.21.15.195:0/1201602166 conn(0x7fccc00c6350 msgr2=0x7fccc0151050 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:57:13.939 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.926+0000 7fccc6a94700 1 -- 172.21.15.195:0/1201602166 shutdown_connections 2022-06-27T11:57:13.939 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:13.926+0000 7fccc6a94700 1 -- 172.21.15.195:0/1201602166 wait complete. 2022-06-27T11:57:13.939 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 33 2022-06-27T11:57:13.942 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 33, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-06-27T11:47:52.358095+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 33, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4976": { "addr": "172.21.15.202:6833/4154570751", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6832", "nonce": 4154570751, "type": "v2" }, { "addr": "172.21.15.202:6833", "nonce": 4154570751, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4976, "incarnation": 32, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:reconnect", "state_seq": 22 } }, "last_failure": 0, "last_failure_osd_epoch": 52, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-06-27T11:57:10.963352+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4976 } } } ], "standbys": [ { "addr": "172.21.15.202:6837/3305006841", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6836", "nonce": 3305006841, "type": "v2" }, { "addr": "172.21.15.202:6837", "nonce": 3305006841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6835/2060709822", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6834", "nonce": 2060709822, "type": "v2" }, { "addr": "172.21.15.195:6835", "nonce": 2060709822, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6838/1598751652", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6836", "nonce": 1598751652, "type": "v2" }, { "addr": "172.21.15.195:6838", "nonce": 1598751652, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4275, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6839/1313800661", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6837", "nonce": 1313800661, "type": "v2" }, { "addr": "172.21.15.195:6839", "nonce": 1313800661, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4278, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-06-27T11:57:15.944 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:57:16.097 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.088+0000 7ff0bba33700 1 Processor -- start 2022-06-27T11:57:16.098 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.089+0000 7ff0bba33700 1 -- start start 2022-06-27T11:57:16.099 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.089+0000 7ff0bba33700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff0b406fef0 0x7ff0b41570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:16.100 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.089+0000 7ff0bba33700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7ff0b40c34a0 0x7ff0b40c2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:16.100 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.089+0000 7ff0bba33700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0b406fba0 con 0x7ff0b406e850 2022-06-27T11:57:16.101 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.089+0000 7ff0bba33700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0b41575e0 con 0x7ff0b40c34a0 2022-06-27T11:57:16.101 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.089+0000 7ff0bba33700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b406f5e0 con 0x7ff0b406fef0 2022-06-27T11:57:16.101 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.089+0000 7ff0b9fd0700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7ff0b40c34a0 0x7ff0b40c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:16.102 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.090+0000 7ff0b9fd0700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7ff0b40c34a0 0x7ff0b40c2920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45150/0 (socket says 172.21.15.195:45150) 2022-06-27T11:57:16.102 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.090+0000 7ff0b9fd0700 1 -- 172.21.15.195:0/3932469966 learned_addr learned my addr 172.21.15.195:0/3932469966 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:57:16.103 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.090+0000 7ff0b8fce700 1 --2- 172.21.15.195:0/3932469966 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff0b406fef0 0x7ff0b41570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:16.103 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.090+0000 7ff0b9fd0700 1 -- 172.21.15.195:0/3932469966 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff0b406fef0 msgr2=0x7ff0b41570a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:57:16.103 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.090+0000 7ff0b9fd0700 1 --2- 172.21.15.195:0/3932469966 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff0b406fef0 0x7ff0b41570a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:16.104 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.090+0000 7ff0b9fd0700 1 -- 172.21.15.195:0/3932469966 >> v1:172.21.15.202:6789/0 conn(0x7ff0b406e850 legacy=0x7ff0b4069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:16.104 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.090+0000 7ff0b9fd0700 1 -- 172.21.15.195:0/3932469966 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0b406f8c0 con 0x7ff0b40c34a0 2022-06-27T11:57:16.104 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.090+0000 7ff0b9fd0700 1 --2- 172.21.15.195:0/3932469966 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff0b40c34a0 0x7ff0b40c2920 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac009f20 tx=0x7ff0ac009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4767288e6d93b2f server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:57:16.105 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.090+0000 7ff0a3fff700 1 -- 172.21.15.195:0/3932469966 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0ac00e070 con 0x7ff0b40c34a0 2022-06-27T11:57:16.105 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.090+0000 7ff0a3fff700 1 -- 172.21.15.195:0/3932469966 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0ac00bb10 con 0x7ff0b40c34a0 2022-06-27T11:57:16.105 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.090+0000 7ff0a3fff700 1 -- 172.21.15.195:0/3932469966 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0ac0092e0 con 0x7ff0b40c34a0 2022-06-27T11:57:16.105 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.091+0000 7ff0bba33700 1 -- 172.21.15.195:0/3932469966 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff0b40c34a0 msgr2=0x7ff0b40c2920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:16.106 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.091+0000 7ff0bba33700 1 --2- 172.21.15.195:0/3932469966 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff0b40c34a0 0x7ff0b40c2920 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac009f20 tx=0x7ff0ac009840 comp rx=0 tx=0).stop 2022-06-27T11:57:16.106 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.091+0000 7ff0bba33700 1 -- 172.21.15.195:0/3932469966 shutdown_connections 2022-06-27T11:57:16.106 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.091+0000 7ff0bba33700 1 --2- 172.21.15.195:0/3932469966 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff0b40c34a0 0x7ff0b40c2920 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:16.106 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.091+0000 7ff0bba33700 1 --2- 172.21.15.195:0/3932469966 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff0b406fef0 0x7ff0b41570a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:16.107 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.091+0000 7ff0bba33700 1 -- 172.21.15.195:0/3932469966 >> 172.21.15.195:0/3932469966 conn(0x7ff0b40c6350 msgr2=0x7ff0b40c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:57:16.107 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.091+0000 7ff0bba33700 1 -- 172.21.15.195:0/3932469966 shutdown_connections 2022-06-27T11:57:16.107 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.091+0000 7ff0bba33700 1 -- 172.21.15.195:0/3932469966 wait complete. 2022-06-27T11:57:16.107 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.092+0000 7ff0bba33700 1 Processor -- start 2022-06-27T11:57:16.108 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.092+0000 7ff0bba33700 1 -- start start 2022-06-27T11:57:16.108 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.093+0000 7ff0bba33700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff0b406e850 0x7ff0b414d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:16.108 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.093+0000 7ff0bba33700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff0b40c34a0 0x7ff0b414dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:16.108 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.093+0000 7ff0bba33700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff0b40c3080 0x7ff0b414e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:16.108 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.093+0000 7ff0bba33700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b4164c20 con 0x7ff0b406e850 2022-06-27T11:57:16.109 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.093+0000 7ff0bba33700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b4068b20 con 0x7ff0b40c3080 2022-06-27T11:57:16.109 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.093+0000 7ff0bba33700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b4068050 con 0x7ff0b40c34a0 2022-06-27T11:57:16.109 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.093+0000 7ff0b97cf700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff0b406e850 0x7ff0b414d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:16.109 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.093+0000 7ff0b97cf700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff0b406e850 0x7ff0b414d0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45152/0 (socket says 172.21.15.195:45152) 2022-06-27T11:57:16.110 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.093+0000 7ff0b97cf700 1 -- 172.21.15.195:0/1319537423 learned_addr learned my addr 172.21.15.195:0/1319537423 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:57:16.110 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.094+0000 7ff0b8fce700 1 --2- 172.21.15.195:0/1319537423 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff0b40c34a0 0x7ff0b414dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:16.110 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.094+0000 7ff0b9fd0700 1 --2- 172.21.15.195:0/1319537423 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff0b40c3080 0x7ff0b414e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:16.110 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.094+0000 7ff0b97cf700 1 -- 172.21.15.195:0/1319537423 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff0b40c34a0 msgr2=0x7ff0b414dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:16.111 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.094+0000 7ff0b97cf700 1 --2- 172.21.15.195:0/1319537423 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff0b40c34a0 0x7ff0b414dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:16.111 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.094+0000 7ff0b97cf700 1 -- 172.21.15.195:0/1319537423 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff0b40c3080 msgr2=0x7ff0b414e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:16.111 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.094+0000 7ff0b97cf700 1 --2- 172.21.15.195:0/1319537423 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff0b40c3080 0x7ff0b414e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:16.111 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.094+0000 7ff0b97cf700 1 -- 172.21.15.195:0/1319537423 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0ac009ae0 con 0x7ff0b406e850 2022-06-27T11:57:16.112 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.094+0000 7ff0b8fce700 1 --2- 172.21.15.195:0/1319537423 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff0b40c34a0 0x7ff0b414dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T11:57:16.112 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.094+0000 7ff0b9fd0700 1 --2- 172.21.15.195:0/1319537423 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff0b40c3080 0x7ff0b414e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T11:57:16.112 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.094+0000 7ff0b97cf700 1 --2- 172.21.15.195:0/1319537423 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff0b406e850 0x7ff0b414d0e0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7ff0a400c440 tx=0x7ff0a4007e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:57:16.112 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.094+0000 7ff0a17fa700 1 -- 172.21.15.195:0/1319537423 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a40050d0 con 0x7ff0b406e850 2022-06-27T11:57:16.112 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.094+0000 7ff0a17fa700 1 -- 172.21.15.195:0/1319537423 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0a4005250 con 0x7ff0b406e850 2022-06-27T11:57:16.113 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.094+0000 7ff0bba33700 1 -- 172.21.15.195:0/1319537423 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0b4164940 con 0x7ff0b406e850 2022-06-27T11:57:16.113 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.094+0000 7ff0a17fa700 1 -- 172.21.15.195:0/1319537423 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a40117e0 con 0x7ff0b406e850 2022-06-27T11:57:16.113 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.094+0000 7ff0bba33700 1 -- 172.21.15.195:0/1319537423 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0b40c2d60 con 0x7ff0b406e850 2022-06-27T11:57:16.113 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.095+0000 7ff0bba33700 1 -- 172.21.15.195:0/1319537423 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0b40722c0 con 0x7ff0b406e850 2022-06-27T11:57:16.114 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.096+0000 7ff0a17fa700 1 -- 172.21.15.195:0/1319537423 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7ff0a4019030 con 0x7ff0b406e850 2022-06-27T11:57:16.114 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.096+0000 7ff0a17fa700 1 --2- 172.21.15.195:0/1319537423 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff094063d00 0x7ff0940661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:16.114 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.096+0000 7ff0b8fce700 1 --2- 172.21.15.195:0/1319537423 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff094063d00 0x7ff0940661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:16.114 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.096+0000 7ff0a17fa700 1 -- 172.21.15.195:0/1319537423 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6546+0+0 (secure 0 0 0) 0x7ff0a407fb00 con 0x7ff0b406e850 2022-06-27T11:57:16.115 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.096+0000 7ff0b8fce700 1 --2- 172.21.15.195:0/1319537423 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff094063d00 0x7ff0940661b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff0b406eef0 tx=0x7ff0b0008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:57:16.115 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.100+0000 7ff0a17fa700 1 -- 172.21.15.195:0/1319537423 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7ff0a404e500 con 0x7ff0b406e850 2022-06-27T11:57:16.327 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.327+0000 7ff0bba33700 1 -- 172.21.15.195:0/1319537423 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff0b406f240 con 0x7ff0b406e850 2022-06-27T11:57:16.328 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.327+0000 7ff0a17fa700 1 -- 172.21.15.195:0/1319537423 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+5371 (secure 0 0 0) 0x7ff0a4017020 con 0x7ff0b406e850 2022-06-27T11:57:16.329 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:57:16.330 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:57:10.963352+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4976},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4976":{"gid":4976,"name":"b","rank":0,"incarnation":32,"state":"up:reconnect","state_seq":22,"addr":"172.21.15.202:6833/4154570751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":4154570751},{"type":"v1","addr":"172.21.15.202:6833","nonce":4154570751}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:57:16.333 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.329+0000 7ff0bba33700 1 -- 172.21.15.195:0/1319537423 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff094063d00 msgr2=0x7ff0940661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:16.334 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.329+0000 7ff0bba33700 1 --2- 172.21.15.195:0/1319537423 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff094063d00 0x7ff0940661b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff0b406eef0 tx=0x7ff0b0008040 comp rx=0 tx=0).stop 2022-06-27T11:57:16.334 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.329+0000 7ff0bba33700 1 -- 172.21.15.195:0/1319537423 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff0b406e850 msgr2=0x7ff0b414d0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:16.335 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.329+0000 7ff0bba33700 1 --2- 172.21.15.195:0/1319537423 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff0b406e850 0x7ff0b414d0e0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7ff0a400c440 tx=0x7ff0a4007e40 comp rx=0 tx=0).stop 2022-06-27T11:57:16.335 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.329+0000 7ff0bba33700 1 -- 172.21.15.195:0/1319537423 shutdown_connections 2022-06-27T11:57:16.335 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.329+0000 7ff0bba33700 1 --2- 172.21.15.195:0/1319537423 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff094063d00 0x7ff0940661b0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:16.336 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.329+0000 7ff0bba33700 1 --2- 172.21.15.195:0/1319537423 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff0b406e850 0x7ff0b414d0e0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:16.336 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.329+0000 7ff0bba33700 1 --2- 172.21.15.195:0/1319537423 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff0b40c3080 0x7ff0b414e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:16.337 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.329+0000 7ff0bba33700 1 --2- 172.21.15.195:0/1319537423 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff0b40c34a0 0x7ff0b414dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:16.337 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.329+0000 7ff0bba33700 1 -- 172.21.15.195:0/1319537423 >> 172.21.15.195:0/1319537423 conn(0x7ff0b40c6350 msgr2=0x7ff0b40c9030 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:57:16.338 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.329+0000 7ff0bba33700 1 -- 172.21.15.195:0/1319537423 shutdown_connections 2022-06-27T11:57:16.339 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:16.329+0000 7ff0bba33700 1 -- 172.21.15.195:0/1319537423 wait complete. 2022-06-27T11:57:16.339 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 33 2022-06-27T11:57:16.351 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T11:57:18.353 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:57:18.489 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.489+0000 7fe40cd8a700 1 Processor -- start 2022-06-27T11:57:18.515 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.489+0000 7fe40cd8a700 1 -- start start 2022-06-27T11:57:18.523 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.490+0000 7fe40cd8a700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fe40806a440 0x7fe4080688a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:18.523 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.490+0000 7fe40cd8a700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fe4081564d0 0x7fe408156950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:18.524 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.490+0000 7fe40cd8a700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4080c4b10 con 0x7fe408156100 2022-06-27T11:57:18.524 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.490+0000 7fe40cd8a700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4080c3dc0 con 0x7fe4081564d0 2022-06-27T11:57:18.524 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.490+0000 7fe40cd8a700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe408068f20 con 0x7fe40806a440 2022-06-27T11:57:18.525 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.491+0000 7fe406d9d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fe4081564d0 0x7fe408156950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:18.525 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.491+0000 7fe406d9d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fe4081564d0 0x7fe408156950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45164/0 (socket says 172.21.15.195:45164) 2022-06-27T11:57:18.526 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.491+0000 7fe406d9d700 1 -- 172.21.15.195:0/1878305014 learned_addr learned my addr 172.21.15.195:0/1878305014 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:57:18.532 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.491+0000 7fe405d9b700 1 --2- 172.21.15.195:0/1878305014 >> v2:172.21.15.202:3300/0 conn(0x7fe40806a440 0x7fe4080688a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:18.532 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.491+0000 7fe406d9d700 1 -- 172.21.15.195:0/1878305014 >> v2:172.21.15.202:3300/0 conn(0x7fe40806a440 msgr2=0x7fe4080688a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:57:18.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.491+0000 7fe406d9d700 1 --2- 172.21.15.195:0/1878305014 >> v2:172.21.15.202:3300/0 conn(0x7fe40806a440 0x7fe4080688a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:18.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.491+0000 7fe406d9d700 1 -- 172.21.15.195:0/1878305014 >> v1:172.21.15.202:6789/0 conn(0x7fe408156100 legacy=0x7fe4080c4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:18.534 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.491+0000 7fe406d9d700 1 -- 172.21.15.195:0/1878305014 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4080c56b0 con 0x7fe4081564d0 2022-06-27T11:57:18.534 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.491+0000 7fe406d9d700 1 --2- 172.21.15.195:0/1878305014 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe4081564d0 0x7fe408156950 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fe408072140 tx=0x7fe400009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4689bb08dec0ef2 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:57:18.534 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.491+0000 7fe40559a700 1 -- 172.21.15.195:0/1878305014 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4000092e0 con 0x7fe4081564d0 2022-06-27T11:57:18.535 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.491+0000 7fe40559a700 1 -- 172.21.15.195:0/1878305014 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe40000ba20 con 0x7fe4081564d0 2022-06-27T11:57:18.547 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.491+0000 7fe40559a700 1 -- 172.21.15.195:0/1878305014 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe40000bde0 con 0x7fe4081564d0 2022-06-27T11:57:18.547 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.492+0000 7fe40cd8a700 1 -- 172.21.15.195:0/1878305014 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe4081564d0 msgr2=0x7fe408156950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:18.559 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.492+0000 7fe40cd8a700 1 --2- 172.21.15.195:0/1878305014 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe4081564d0 0x7fe408156950 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fe408072140 tx=0x7fe400009840 comp rx=0 tx=0).stop 2022-06-27T11:57:18.560 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.492+0000 7fe40cd8a700 1 -- 172.21.15.195:0/1878305014 shutdown_connections 2022-06-27T11:57:18.560 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.492+0000 7fe40cd8a700 1 --2- 172.21.15.195:0/1878305014 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe4081564d0 0x7fe408156950 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:18.561 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.492+0000 7fe40cd8a700 1 --2- 172.21.15.195:0/1878305014 >> v2:172.21.15.202:3300/0 conn(0x7fe40806a440 0x7fe4080688a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:18.561 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.492+0000 7fe40cd8a700 1 -- 172.21.15.195:0/1878305014 >> 172.21.15.195:0/1878305014 conn(0x7fe4080c9b00 msgr2=0x7fe4080c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:57:18.561 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.492+0000 7fe40cd8a700 1 -- 172.21.15.195:0/1878305014 shutdown_connections 2022-06-27T11:57:18.562 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.492+0000 7fe40cd8a700 1 -- 172.21.15.195:0/1878305014 wait complete. 2022-06-27T11:57:18.562 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.493+0000 7fe40cd8a700 1 Processor -- start 2022-06-27T11:57:18.563 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.493+0000 7fe40cd8a700 1 -- start start 2022-06-27T11:57:18.563 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.494+0000 7fe40cd8a700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe40806a440 0x7fe4081498b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:18.563 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.494+0000 7fe40cd8a700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe408156100 0x7fe408147930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:18.564 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.494+0000 7fe40cd8a700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe4081564d0 0x7fe408147e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:18.564 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.494+0000 7fe40cd8a700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe408155c80 con 0x7fe40806a440 2022-06-27T11:57:18.565 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.494+0000 7fe40cd8a700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe408155750 con 0x7fe408156100 2022-06-27T11:57:18.578 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.494+0000 7fe40cd8a700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4080c1c70 con 0x7fe4081564d0 2022-06-27T11:57:18.579 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.494+0000 7fe40659c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe40806a440 0x7fe4081498b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:18.579 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.494+0000 7fe40659c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe40806a440 0x7fe4081498b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45166/0 (socket says 172.21.15.195:45166) 2022-06-27T11:57:18.580 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.494+0000 7fe40659c700 1 -- 172.21.15.195:0/1563162207 learned_addr learned my addr 172.21.15.195:0/1563162207 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:57:18.580 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.494+0000 7fe405d9b700 1 --2- 172.21.15.195:0/1563162207 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe408156100 0x7fe408147930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:18.581 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.494+0000 7fe406d9d700 1 --2- 172.21.15.195:0/1563162207 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe4081564d0 0x7fe408147e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:18.581 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.494+0000 7fe40659c700 1 -- 172.21.15.195:0/1563162207 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe4081564d0 msgr2=0x7fe408147e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:18.581 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.494+0000 7fe40659c700 1 --2- 172.21.15.195:0/1563162207 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe4081564d0 0x7fe408147e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:18.582 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.494+0000 7fe40659c700 1 -- 172.21.15.195:0/1563162207 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe408156100 msgr2=0x7fe408147930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:18.582 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.494+0000 7fe40659c700 1 --2- 172.21.15.195:0/1563162207 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe408156100 0x7fe408147930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:18.583 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.494+0000 7fe40659c700 1 -- 172.21.15.195:0/1563162207 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe400009ae0 con 0x7fe40806a440 2022-06-27T11:57:18.583 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.494+0000 7fe40659c700 1 --2- 172.21.15.195:0/1563162207 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe40806a440 0x7fe4081498b0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fe3f000c4f0 tx=0x7fe3f0007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:57:18.584 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.495+0000 7fe3fe7fc700 1 -- 172.21.15.195:0/1563162207 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3f000ce60 con 0x7fe40806a440 2022-06-27T11:57:18.584 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.495+0000 7fe3fe7fc700 1 -- 172.21.15.195:0/1563162207 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3f00050d0 con 0x7fe40806a440 2022-06-27T11:57:18.585 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.495+0000 7fe40cd8a700 1 -- 172.21.15.195:0/1563162207 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4081559a0 con 0x7fe40806a440 2022-06-27T11:57:18.585 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.495+0000 7fe3fe7fc700 1 -- 172.21.15.195:0/1563162207 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3f0011590 con 0x7fe40806a440 2022-06-27T11:57:18.586 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.495+0000 7fe40cd8a700 1 -- 172.21.15.195:0/1563162207 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4080c8140 con 0x7fe40806a440 2022-06-27T11:57:18.586 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.495+0000 7fe40cd8a700 1 -- 172.21.15.195:0/1563162207 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4080722c0 con 0x7fe40806a440 2022-06-27T11:57:18.587 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.496+0000 7fe3fe7fc700 1 -- 172.21.15.195:0/1563162207 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fe3f0019030 con 0x7fe40806a440 2022-06-27T11:57:18.587 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.496+0000 7fe3fe7fc700 1 --2- 172.21.15.195:0/1563162207 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe3ec063d00 0x7fe3ec0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:18.588 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.496+0000 7fe3fe7fc700 1 -- 172.21.15.195:0/1563162207 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6546+0+0 (secure 0 0 0) 0x7fe3f007fb00 con 0x7fe40806a440 2022-06-27T11:57:18.588 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.497+0000 7fe405d9b700 1 --2- 172.21.15.195:0/1563162207 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe3ec063d00 0x7fe3ec0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:18.589 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.497+0000 7fe405d9b700 1 --2- 172.21.15.195:0/1563162207 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe3ec063d00 0x7fe3ec0661b0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe4080c0f50 tx=0x7fe3f8009060 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:57:18.589 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.501+0000 7fe3fe7fc700 1 -- 172.21.15.195:0/1563162207 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fe3f004e500 con 0x7fe40806a440 2022-06-27T11:57:18.729 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.726+0000 7fe40cd8a700 1 -- 172.21.15.195:0/1563162207 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe4080c6360 con 0x7fe40806a440 2022-06-27T11:57:18.729 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.727+0000 7fe3fe7fc700 1 -- 172.21.15.195:0/1563162207 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+5368 (secure 0 0 0) 0x7fe3f0017020 con 0x7fe40806a440 2022-06-27T11:57:18.730 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:57:18.731 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:57:16.971234+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4976},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4976":{"gid":4976,"name":"b","rank":0,"incarnation":32,"state":"up:rejoin","state_seq":24,"addr":"172.21.15.202:6833/4154570751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":4154570751},{"type":"v1","addr":"172.21.15.202:6833","nonce":4154570751}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:57:18.733 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.729+0000 7fe40cd8a700 1 -- 172.21.15.195:0/1563162207 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe3ec063d00 msgr2=0x7fe3ec0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:18.734 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.729+0000 7fe40cd8a700 1 --2- 172.21.15.195:0/1563162207 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe3ec063d00 0x7fe3ec0661b0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe4080c0f50 tx=0x7fe3f8009060 comp rx=0 tx=0).stop 2022-06-27T11:57:18.734 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.729+0000 7fe40cd8a700 1 -- 172.21.15.195:0/1563162207 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe40806a440 msgr2=0x7fe4081498b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:18.735 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.729+0000 7fe40cd8a700 1 --2- 172.21.15.195:0/1563162207 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe40806a440 0x7fe4081498b0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fe3f000c4f0 tx=0x7fe3f0007df0 comp rx=0 tx=0).stop 2022-06-27T11:57:18.735 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.729+0000 7fe40cd8a700 1 -- 172.21.15.195:0/1563162207 shutdown_connections 2022-06-27T11:57:18.735 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.729+0000 7fe40cd8a700 1 --2- 172.21.15.195:0/1563162207 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe3ec063d00 0x7fe3ec0661b0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:18.736 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.729+0000 7fe40cd8a700 1 --2- 172.21.15.195:0/1563162207 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe40806a440 0x7fe4081498b0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:18.736 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.729+0000 7fe40cd8a700 1 --2- 172.21.15.195:0/1563162207 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe4081564d0 0x7fe408147e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:18.736 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.729+0000 7fe40cd8a700 1 --2- 172.21.15.195:0/1563162207 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe408156100 0x7fe408147930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:18.737 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.729+0000 7fe40cd8a700 1 -- 172.21.15.195:0/1563162207 >> 172.21.15.195:0/1563162207 conn(0x7fe4080c9b00 msgr2=0x7fe40814f980 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:57:18.737 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.729+0000 7fe40cd8a700 1 -- 172.21.15.195:0/1563162207 shutdown_connections 2022-06-27T11:57:18.738 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:18.729+0000 7fe40cd8a700 1 -- 172.21.15.195:0/1563162207 wait complete. 2022-06-27T11:57:18.738 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 34 2022-06-27T11:57:18.751 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T11:57:20.753 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:57:20.897 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:20.888+0000 7f34292ab700 1 Processor -- start 2022-06-27T11:57:20.898 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:20.888+0000 7f34292ab700 1 -- start start 2022-06-27T11:57:20.899 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:20.888+0000 7f34292ab700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f34240c7480 0x7f34240c4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:20.899 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:20.888+0000 7f34292ab700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f34240683c0 0x7f3424069350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:20.900 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:20.888+0000 7f34292ab700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34241573b0 con 0x7f3424068f80 2022-06-27T11:57:20.900 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:20.888+0000 7f34292ab700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3424068840 con 0x7f34240683c0 2022-06-27T11:57:20.900 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:20.888+0000 7f34292ab700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34240c46c0 con 0x7f34240c7480 2022-06-27T11:57:20.901 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:20.889+0000 7f34237fe700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f34240683c0 0x7f3424069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:20.901 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:20.889+0000 7f3422ffd700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f3424068f80 0x7f3424156460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:51366/0 (socket says 172.21.15.195:51366) 2022-06-27T11:57:20.903 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:20.889+0000 7f3422ffd700 1 -- 172.21.15.195:0/1184674050 learned_addr learned my addr 172.21.15.195:0/1184674050 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:57:20.903 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:20.889+0000 7f34227fc700 1 --2- 172.21.15.195:0/1184674050 >> v2:172.21.15.202:3300/0 conn(0x7f34240c7480 0x7f34240c4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:20.904 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:20.889+0000 7f34237fe700 1 -- 172.21.15.195:0/1184674050 >> v2:172.21.15.202:3300/0 conn(0x7f34240c7480 msgr2=0x7f34240c4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:57:20.904 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:20.889+0000 7f34237fe700 1 --2- 172.21.15.195:0/1184674050 >> v2:172.21.15.202:3300/0 conn(0x7f34240c7480 0x7f34240c4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:20.905 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:20.889+0000 7f34237fe700 1 -- 172.21.15.195:0/1184674050 >> v1:172.21.15.195:6789/0 conn(0x7f3424068f80 legacy=0x7f3424156460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:20.905 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:20.889+0000 7f34237fe700 1 -- 172.21.15.195:0/1184674050 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3424156b70 con 0x7f34240683c0 2022-06-27T11:57:20.906 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:20.889+0000 7f3421ffb700 1 -- 172.21.15.195:0/1184674050 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3876419166 0 0) 0x7f34241573b0 con 0x7f3424068f80 2022-06-27T11:57:20.906 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:20.889+0000 7f34227fc700 1 --2- 172.21.15.195:0/1184674050 >> v2:172.21.15.202:3300/0 conn(0x7f34240c7480 0x7f34240c4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T11:57:20.907 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:20.889+0000 7f34237fe700 1 --2- 172.21.15.195:0/1184674050 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f34240683c0 0x7f3424069350 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f34240c6430 tx=0x7f341c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28b304fedf4e9bc2 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:57:20.907 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:20.889+0000 7f3421ffb700 1 -- 172.21.15.195:0/1184674050 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f341c00e070 con 0x7f34240683c0 2022-06-27T11:57:20.908 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:20.889+0000 7f3421ffb700 1 -- 172.21.15.195:0/1184674050 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f341c00bc30 con 0x7f34240683c0 2022-06-27T11:57:20.908 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:20.889+0000 7f3421ffb700 1 -- 172.21.15.195:0/1184674050 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f341c012430 con 0x7f34240683c0 2022-06-27T11:57:20.909 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:20.890+0000 7f34292ab700 1 -- 172.21.15.195:0/1184674050 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f34240683c0 msgr2=0x7f3424069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:20.909 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:20.890+0000 7f34292ab700 1 --2- 172.21.15.195:0/1184674050 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f34240683c0 0x7f3424069350 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f34240c6430 tx=0x7f341c009780 comp rx=0 tx=0).stop 2022-06-27T11:57:20.910 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:20.890+0000 7f34292ab700 1 -- 172.21.15.195:0/1184674050 shutdown_connections 2022-06-27T11:57:20.910 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:20.890+0000 7f34292ab700 1 --2- 172.21.15.195:0/1184674050 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f34240683c0 0x7f3424069350 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:20.911 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:20.890+0000 7f34292ab700 1 --2- 172.21.15.195:0/1184674050 >> v2:172.21.15.202:3300/0 conn(0x7f34240c7480 0x7f34240c4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:20.911 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:20.890+0000 7f34292ab700 1 -- 172.21.15.195:0/1184674050 >> 172.21.15.195:0/1184674050 conn(0x7f34240ccbd0 msgr2=0x7f34240ccfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:57:20.911 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:20.890+0000 7f34292ab700 1 -- 172.21.15.195:0/1184674050 shutdown_connections 2022-06-27T11:57:20.912 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:20.890+0000 7f34292ab700 1 -- 172.21.15.195:0/1184674050 wait complete. 2022-06-27T11:57:20.912 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:20.891+0000 7f34292ab700 1 Processor -- start 2022-06-27T11:57:20.913 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:20.891+0000 7f34292ab700 1 -- start start 2022-06-27T11:57:20.913 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:20.892+0000 7f34292ab700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f34240683c0 0x7f34241667f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:20.914 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:20.892+0000 7f34292ab700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3424068f80 0x7f342414f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:20.914 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:20.892+0000 7f34292ab700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f34240c7480 0x7f342415b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:20.915 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:20.892+0000 7f34292ab700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34240a8640 con 0x7f3424068f80 2022-06-27T11:57:20.915 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:20.892+0000 7f34292ab700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3424166d30 con 0x7f34240683c0 2022-06-27T11:57:20.915 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:20.892+0000 7f34292ab700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f342415a8b0 con 0x7f34240c7480 2022-06-27T11:57:20.916 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:20.892+0000 7f34227fc700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3424068f80 0x7f342414f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:20.916 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:20.892+0000 7f34227fc700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3424068f80 0x7f342414f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45186/0 (socket says 172.21.15.195:45186) 2022-06-27T11:57:20.917 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:20.892+0000 7f34227fc700 1 -- 172.21.15.195:0/3056890822 learned_addr learned my addr 172.21.15.195:0/3056890822 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:57:20.917 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:20.892+0000 7f3422ffd700 1 --2- 172.21.15.195:0/3056890822 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f34240683c0 0x7f34241667f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:20.918 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:20.892+0000 7f34237fe700 1 --2- 172.21.15.195:0/3056890822 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f34240c7480 0x7f342415b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:20.918 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:20.892+0000 7f34227fc700 1 -- 172.21.15.195:0/3056890822 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f34240c7480 msgr2=0x7f342415b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:20.919 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:20.892+0000 7f34227fc700 1 --2- 172.21.15.195:0/3056890822 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f34240c7480 0x7f342415b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:20.919 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:20.892+0000 7f34227fc700 1 -- 172.21.15.195:0/3056890822 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f34240683c0 msgr2=0x7f34241667f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:20.920 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:20.892+0000 7f34227fc700 1 --2- 172.21.15.195:0/3056890822 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f34240683c0 0x7f34241667f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:20.922 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:20.892+0000 7f34227fc700 1 -- 172.21.15.195:0/3056890822 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f341c009a20 con 0x7f3424068f80 2022-06-27T11:57:20.923 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:20.893+0000 7f34227fc700 1 --2- 172.21.15.195:0/3056890822 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3424068f80 0x7f342414f490 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f340c00d840 tx=0x7f340c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:57:20.923 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:20.893+0000 7f341affd700 1 -- 172.21.15.195:0/3056890822 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f340c004030 con 0x7f3424068f80 2022-06-27T11:57:20.924 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:20.893+0000 7f341affd700 1 -- 172.21.15.195:0/3056890822 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f340c004190 con 0x7f3424068f80 2022-06-27T11:57:20.924 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:20.893+0000 7f34292ab700 1 -- 172.21.15.195:0/3056890822 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3424158ac0 con 0x7f3424068f80 2022-06-27T11:57:20.924 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:20.893+0000 7f341affd700 1 -- 172.21.15.195:0/3056890822 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f340c0125f0 con 0x7f3424068f80 2022-06-27T11:57:20.925 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:20.893+0000 7f34292ab700 1 -- 172.21.15.195:0/3056890822 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3424158fc0 con 0x7f3424068f80 2022-06-27T11:57:20.925 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:20.894+0000 7f34292ab700 1 -- 172.21.15.195:0/3056890822 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34240722c0 con 0x7f3424068f80 2022-06-27T11:57:20.926 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:20.894+0000 7f341affd700 1 -- 172.21.15.195:0/3056890822 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f340c012960 con 0x7f3424068f80 2022-06-27T11:57:20.926 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:20.894+0000 7f341affd700 1 --2- 172.21.15.195:0/3056890822 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3408063d00 0x7f34080661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:20.927 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:20.894+0000 7f341affd700 1 -- 172.21.15.195:0/3056890822 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6546+0+0 (secure 0 0 0) 0x7f340c080d10 con 0x7f3424068f80 2022-06-27T11:57:20.927 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:20.895+0000 7f3422ffd700 1 --2- 172.21.15.195:0/3056890822 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3408063d00 0x7f34080661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:20.928 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:20.895+0000 7f3422ffd700 1 --2- 172.21.15.195:0/3056890822 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3408063d00 0x7f34080661b0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3414004810 tx=0x7f3414007bf0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:57:20.929 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:20.899+0000 7f341affd700 1 -- 172.21.15.195:0/3056890822 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f340c04f710 con 0x7f3424068f80 2022-06-27T11:57:21.124 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:21.124+0000 7f34292ab700 1 -- 172.21.15.195:0/3056890822 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f342406ac60 con 0x7f3424068f80 2022-06-27T11:57:21.124 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:21.124+0000 7f341affd700 1 -- 172.21.15.195:0/3056890822 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+5368 (secure 0 0 0) 0x7f340c052d30 con 0x7f3424068f80 2022-06-27T11:57:21.125 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:57:21.125 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:57:16.971234+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4976},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4976":{"gid":4976,"name":"b","rank":0,"incarnation":32,"state":"up:rejoin","state_seq":24,"addr":"172.21.15.202:6833/4154570751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":4154570751},{"type":"v1","addr":"172.21.15.202:6833","nonce":4154570751}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:57:21.140 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:21.126+0000 7f34292ab700 1 -- 172.21.15.195:0/3056890822 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3408063d00 msgr2=0x7f34080661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:21.140 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:21.126+0000 7f34292ab700 1 --2- 172.21.15.195:0/3056890822 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3408063d00 0x7f34080661b0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3414004810 tx=0x7f3414007bf0 comp rx=0 tx=0).stop 2022-06-27T11:57:21.140 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:21.126+0000 7f34292ab700 1 -- 172.21.15.195:0/3056890822 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3424068f80 msgr2=0x7f342414f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:21.141 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:21.126+0000 7f34292ab700 1 --2- 172.21.15.195:0/3056890822 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3424068f80 0x7f342414f490 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f340c00d840 tx=0x7f340c008040 comp rx=0 tx=0).stop 2022-06-27T11:57:21.141 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:21.127+0000 7f34292ab700 1 -- 172.21.15.195:0/3056890822 shutdown_connections 2022-06-27T11:57:21.143 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:21.127+0000 7f34292ab700 1 --2- 172.21.15.195:0/3056890822 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3408063d00 0x7f34080661b0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:21.143 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:21.127+0000 7f34292ab700 1 --2- 172.21.15.195:0/3056890822 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f34240c7480 0x7f342415b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:21.144 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:21.127+0000 7f34292ab700 1 --2- 172.21.15.195:0/3056890822 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f34240683c0 0x7f34241667f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:21.144 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:21.127+0000 7f34292ab700 1 --2- 172.21.15.195:0/3056890822 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3424068f80 0x7f342414f490 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:21.144 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:21.127+0000 7f34292ab700 1 -- 172.21.15.195:0/3056890822 >> 172.21.15.195:0/3056890822 conn(0x7f34240ccbd0 msgr2=0x7f3424156360 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:57:21.145 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:21.127+0000 7f34292ab700 1 -- 172.21.15.195:0/3056890822 shutdown_connections 2022-06-27T11:57:21.145 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:21.127+0000 7f34292ab700 1 -- 172.21.15.195:0/3056890822 wait complete. 2022-06-27T11:57:21.146 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 34 2022-06-27T11:57:21.147 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T11:57:23.148 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:57:23.282 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.282+0000 7f8f5f4c4700 1 Processor -- start 2022-06-27T11:57:23.282 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.282+0000 7f8f5f4c4700 1 -- start start 2022-06-27T11:57:23.285 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.283+0000 7f8f5f4c4700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8f5806f640 0x7f8f58156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:23.286 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.283+0000 7f8f5f4c4700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f8f580c39e0 0x7f8f581535a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:23.286 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.283+0000 7f8f5f4c4700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f580c1a20 con 0x7f8f580c4160 2022-06-27T11:57:23.286 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.283+0000 7f8f5f4c4700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f580c1130 con 0x7f8f580c39e0 2022-06-27T11:57:23.286 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.283+0000 7f8f5f4c4700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f5806ea90 con 0x7f8f5806f640 2022-06-27T11:57:23.287 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.283+0000 7f8f5da61700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f8f580c39e0 0x7f8f581535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:23.287 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.283+0000 7f8f5da61700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f8f580c39e0 0x7f8f581535a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45196/0 (socket says 172.21.15.195:45196) 2022-06-27T11:57:23.287 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.283+0000 7f8f5da61700 1 -- 172.21.15.195:0/1370375759 learned_addr learned my addr 172.21.15.195:0/1370375759 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:57:23.287 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.284+0000 7f8f5d260700 1 --2- 172.21.15.195:0/1370375759 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8f5806f640 0x7f8f58156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:23.288 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.284+0000 7f8f5da61700 1 -- 172.21.15.195:0/1370375759 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8f5806f640 msgr2=0x7f8f58156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:57:23.288 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.284+0000 7f8f5da61700 1 --2- 172.21.15.195:0/1370375759 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8f5806f640 0x7f8f58156ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:23.288 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.284+0000 7f8f5da61700 1 -- 172.21.15.195:0/1370375759 >> v1:172.21.15.202:6789/0 conn(0x7f8f580c4160 legacy=0x7f8f580c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:23.288 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.284+0000 7f8f5da61700 1 -- 172.21.15.195:0/1370375759 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f5806ed70 con 0x7f8f580c39e0 2022-06-27T11:57:23.289 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.284+0000 7f8f5da61700 1 --2- 172.21.15.195:0/1370375759 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8f580c39e0 0x7f8f581535a0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f8f50009f20 tx=0x7f8f50009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef9929022245a87b server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:57:23.289 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.284+0000 7f8f4bfff700 1 -- 172.21.15.195:0/1370375759 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f5000e070 con 0x7f8f580c39e0 2022-06-27T11:57:23.289 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.284+0000 7f8f4bfff700 1 -- 172.21.15.195:0/1370375759 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f5000ba60 con 0x7f8f580c39e0 2022-06-27T11:57:23.289 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.284+0000 7f8f4bfff700 1 -- 172.21.15.195:0/1370375759 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f500092e0 con 0x7f8f580c39e0 2022-06-27T11:57:23.290 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.285+0000 7f8f5f4c4700 1 -- 172.21.15.195:0/1370375759 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8f580c39e0 msgr2=0x7f8f581535a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:23.292 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.285+0000 7f8f5f4c4700 1 --2- 172.21.15.195:0/1370375759 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8f580c39e0 0x7f8f581535a0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f8f50009f20 tx=0x7f8f50009840 comp rx=0 tx=0).stop 2022-06-27T11:57:23.292 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.285+0000 7f8f5f4c4700 1 -- 172.21.15.195:0/1370375759 shutdown_connections 2022-06-27T11:57:23.293 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.285+0000 7f8f5f4c4700 1 --2- 172.21.15.195:0/1370375759 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8f580c39e0 0x7f8f581535a0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:23.293 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.285+0000 7f8f5f4c4700 1 --2- 172.21.15.195:0/1370375759 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8f5806f640 0x7f8f58156ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:23.293 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.285+0000 7f8f5f4c4700 1 -- 172.21.15.195:0/1370375759 >> 172.21.15.195:0/1370375759 conn(0x7f8f580c6350 msgr2=0x7f8f580c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:57:23.293 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.285+0000 7f8f5f4c4700 1 -- 172.21.15.195:0/1370375759 shutdown_connections 2022-06-27T11:57:23.294 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.285+0000 7f8f5f4c4700 1 -- 172.21.15.195:0/1370375759 wait complete. 2022-06-27T11:57:23.294 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.286+0000 7f8f5f4c4700 1 Processor -- start 2022-06-27T11:57:23.294 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.286+0000 7f8f5f4c4700 1 -- start start 2022-06-27T11:57:23.297 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.287+0000 7f8f5f4c4700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8f5806f640 0x7f8f5814f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:23.298 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.287+0000 7f8f5f4c4700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8f580c39e0 0x7f8f580ba330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:23.298 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.287+0000 7f8f5f4c4700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8f580c4160 0x7f8f581546e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:23.298 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.287+0000 7f8f5f4c4700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f580bc020 con 0x7f8f580c4160 2022-06-27T11:57:23.298 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.287+0000 7f8f5f4c4700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f580bd030 con 0x7f8f580c39e0 2022-06-27T11:57:23.299 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.287+0000 7f8f5f4c4700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f58156230 con 0x7f8f5806f640 2022-06-27T11:57:23.299 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.287+0000 7f8f5da61700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8f580c4160 0x7f8f581546e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:23.299 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.287+0000 7f8f5da61700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8f580c4160 0x7f8f581546e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45202/0 (socket says 172.21.15.195:45202) 2022-06-27T11:57:23.299 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.287+0000 7f8f5da61700 1 -- 172.21.15.195:0/1327619382 learned_addr learned my addr 172.21.15.195:0/1327619382 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:57:23.299 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.287+0000 7f8f5d260700 1 --2- 172.21.15.195:0/1327619382 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8f5806f640 0x7f8f5814f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:23.300 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.287+0000 7f8f5ca5f700 1 --2- 172.21.15.195:0/1327619382 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8f580c39e0 0x7f8f580ba330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:23.300 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.287+0000 7f8f5da61700 1 -- 172.21.15.195:0/1327619382 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8f5806f640 msgr2=0x7f8f5814f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:23.300 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.287+0000 7f8f5da61700 1 --2- 172.21.15.195:0/1327619382 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8f5806f640 0x7f8f5814f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:23.300 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.287+0000 7f8f5da61700 1 -- 172.21.15.195:0/1327619382 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8f580c39e0 msgr2=0x7f8f580ba330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:23.301 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.287+0000 7f8f5da61700 1 --2- 172.21.15.195:0/1327619382 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8f580c39e0 0x7f8f580ba330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:23.301 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.287+0000 7f8f5da61700 1 -- 172.21.15.195:0/1327619382 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f50009ae0 con 0x7f8f580c4160 2022-06-27T11:57:23.301 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.287+0000 7f8f5da61700 1 --2- 172.21.15.195:0/1327619382 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8f580c4160 0x7f8f581546e0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f8f580650f0 tx=0x7f8f50018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:57:23.301 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.287+0000 7f8f497fa700 1 -- 172.21.15.195:0/1327619382 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f500092e0 con 0x7f8f580c4160 2022-06-27T11:57:23.315 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.287+0000 7f8f497fa700 1 -- 172.21.15.195:0/1327619382 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f50007610 con 0x7f8f580c4160 2022-06-27T11:57:23.315 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.288+0000 7f8f5f4c4700 1 -- 172.21.15.195:0/1327619382 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f58154120 con 0x7f8f580c4160 2022-06-27T11:57:23.316 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.288+0000 7f8f497fa700 1 -- 172.21.15.195:0/1327619382 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f500079d0 con 0x7f8f580c4160 2022-06-27T11:57:23.317 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.288+0000 7f8f5f4c4700 1 -- 172.21.15.195:0/1327619382 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f58157b70 con 0x7f8f580c4160 2022-06-27T11:57:23.317 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.288+0000 7f8f5f4c4700 1 -- 172.21.15.195:0/1327619382 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f580bd1e0 con 0x7f8f580c4160 2022-06-27T11:57:23.317 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.289+0000 7f8f497fa700 1 -- 172.21.15.195:0/1327619382 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f8f50004040 con 0x7f8f580c4160 2022-06-27T11:57:23.318 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.289+0000 7f8f497fa700 1 --2- 172.21.15.195:0/1327619382 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8f38063d00 0x7f8f380661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:23.318 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.289+0000 7f8f497fa700 1 -- 172.21.15.195:0/1327619382 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6546+0+0 (secure 0 0 0) 0x7f8f50082650 con 0x7f8f580c4160 2022-06-27T11:57:23.319 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.289+0000 7f8f5d260700 1 --2- 172.21.15.195:0/1327619382 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8f38063d00 0x7f8f380661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:23.319 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.290+0000 7f8f5d260700 1 --2- 172.21.15.195:0/1327619382 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8f38063d00 0x7f8f380661b0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f8f4c009c70 tx=0x7f8f4c009060 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:57:23.327 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.293+0000 7f8f497fa700 1 -- 172.21.15.195:0/1327619382 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f8f580bd1e0 con 0x7f8f580c4160 2022-06-27T11:57:23.518 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.517+0000 7f8f5f4c4700 1 -- 172.21.15.195:0/1327619382 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8f580bd1e0 con 0x7f8f580c4160 2022-06-27T11:57:23.519 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.517+0000 7f8f497fa700 1 -- 172.21.15.195:0/1327619382 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+5374 (secure 0 0 0) 0x7f8f580bd1e0 con 0x7f8f580c4160 2022-06-27T11:57:23.519 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:57:23.520 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:57:22.984765+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4976},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4976":{"gid":4976,"name":"b","rank":0,"incarnation":32,"state":"up:clientreplay","state_seq":26,"addr":"172.21.15.202:6833/4154570751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":4154570751},{"type":"v1","addr":"172.21.15.202:6833","nonce":4154570751}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:57:23.521 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.519+0000 7f8f5f4c4700 1 -- 172.21.15.195:0/1327619382 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8f38063d00 msgr2=0x7f8f380661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:23.522 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.519+0000 7f8f5f4c4700 1 --2- 172.21.15.195:0/1327619382 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8f38063d00 0x7f8f380661b0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f8f4c009c70 tx=0x7f8f4c009060 comp rx=0 tx=0).stop 2022-06-27T11:57:23.532 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.519+0000 7f8f5f4c4700 1 -- 172.21.15.195:0/1327619382 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8f580c4160 msgr2=0x7f8f581546e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:23.532 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.519+0000 7f8f5f4c4700 1 --2- 172.21.15.195:0/1327619382 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8f580c4160 0x7f8f581546e0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f8f580650f0 tx=0x7f8f50018000 comp rx=0 tx=0).stop 2022-06-27T11:57:23.532 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.519+0000 7f8f5f4c4700 1 -- 172.21.15.195:0/1327619382 shutdown_connections 2022-06-27T11:57:23.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.519+0000 7f8f5f4c4700 1 --2- 172.21.15.195:0/1327619382 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8f38063d00 0x7f8f380661b0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:23.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.519+0000 7f8f5f4c4700 1 --2- 172.21.15.195:0/1327619382 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8f580c4160 0x7f8f581546e0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:23.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.519+0000 7f8f5f4c4700 1 --2- 172.21.15.195:0/1327619382 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8f5806f640 0x7f8f5814f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:23.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.519+0000 7f8f5f4c4700 1 --2- 172.21.15.195:0/1327619382 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8f580c39e0 0x7f8f580ba330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:23.534 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.519+0000 7f8f5f4c4700 1 -- 172.21.15.195:0/1327619382 >> 172.21.15.195:0/1327619382 conn(0x7f8f580c6350 msgr2=0x7f8f58153370 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:57:23.534 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.519+0000 7f8f5f4c4700 1 -- 172.21.15.195:0/1327619382 shutdown_connections 2022-06-27T11:57:23.540 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:23.519+0000 7f8f5f4c4700 1 -- 172.21.15.195:0/1327619382 wait complete. 2022-06-27T11:57:23.542 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 35 2022-06-27T11:57:23.543 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T11:57:25.547 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:57:25.697 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.682+0000 7fee1f8d1700 1 Processor -- start 2022-06-27T11:57:25.698 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.682+0000 7fee1f8d1700 1 -- start start 2022-06-27T11:57:25.698 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.683+0000 7fee1f8d1700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fee1806f640 0x7fee18156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:25.698 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.683+0000 7fee1f8d1700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fee180c4160 0x7fee180c1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:25.698 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.683+0000 7fee1f8d1700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee180c1a20 con 0x7fee180c39e0 2022-06-27T11:57:25.699 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.683+0000 7fee1f8d1700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee180c1130 con 0x7fee1806f640 2022-06-27T11:57:25.699 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.683+0000 7fee1f8d1700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee1806ea90 con 0x7fee180c4160 2022-06-27T11:57:25.699 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.683+0000 7fee1de6e700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fee180c39e0 0x7fee181535a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:51400/0 (socket says 172.21.15.195:51400) 2022-06-27T11:57:25.699 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.683+0000 7fee1de6e700 1 -- 172.21.15.195:0/2177833348 learned_addr learned my addr 172.21.15.195:0/2177833348 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:57:25.700 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.683+0000 7fee1ce6c700 1 --2- 172.21.15.195:0/2177833348 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fee180c4160 0x7fee180c1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:25.700 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.683+0000 7fee1d66d700 1 --2- 172.21.15.195:0/2177833348 >> v2:172.21.15.202:3300/0 conn(0x7fee1806f640 0x7fee18156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:25.701 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.683+0000 7fee07fff700 1 -- 172.21.15.195:0/2177833348 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3807928688 0 0) 0x7fee180c1a20 con 0x7fee180c39e0 2022-06-27T11:57:25.701 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.684+0000 7fee07fff700 1 -- 172.21.15.195:0/2177833348 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee00003430 con 0x7fee180c39e0 2022-06-27T11:57:25.701 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.684+0000 7fee07fff700 1 -- 172.21.15.195:0/2177833348 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1046143879 0 0) 0x7fee00003430 con 0x7fee180c39e0 2022-06-27T11:57:25.702 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.684+0000 7fee07fff700 1 -- 172.21.15.195:0/2177833348 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fee180c4160 msgr2=0x7fee180c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:57:25.702 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.684+0000 7fee07fff700 1 --2- 172.21.15.195:0/2177833348 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fee180c4160 0x7fee180c1eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:25.702 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.684+0000 7fee07fff700 1 -- 172.21.15.195:0/2177833348 >> v2:172.21.15.202:3300/0 conn(0x7fee1806f640 msgr2=0x7fee18156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:57:25.702 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.684+0000 7fee07fff700 1 --2- 172.21.15.195:0/2177833348 >> v2:172.21.15.202:3300/0 conn(0x7fee1806f640 0x7fee18156ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:25.703 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.684+0000 7fee07fff700 1 -- 172.21.15.195:0/2177833348 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee1806ed70 con 0x7fee180c39e0 2022-06-27T11:57:25.703 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.684+0000 7fee07fff700 1 -- 172.21.15.195:0/2177833348 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7fee100017f0 con 0x7fee180c39e0 2022-06-27T11:57:25.703 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.684+0000 7fee07fff700 1 -- 172.21.15.195:0/2177833348 >> v1:172.21.15.195:6789/0 conn(0x7fee180c39e0 legacy=0x7fee181535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:25.703 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.684+0000 7fee07fff700 1 --2- 172.21.15.195:0/2177833348 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fee00003920 0x7fee00003d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:25.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.684+0000 7fee07fff700 1 --2- 172.21.15.195:0/2177833348 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fee180c4160 0x7fee000046d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:25.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.684+0000 7fee07fff700 1 --2- 172.21.15.195:0/2177833348 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fee00004c10 0x7fee00005080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:25.705 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.684+0000 7fee07fff700 1 -- 172.21.15.195:0/2177833348 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee180c1130 con 0x7fee00003920 2022-06-27T11:57:25.705 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.684+0000 7fee07fff700 1 -- 172.21.15.195:0/2177833348 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee1806ea90 con 0x7fee00004c10 2022-06-27T11:57:25.705 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.684+0000 7fee07fff700 1 -- 172.21.15.195:0/2177833348 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee0000b920 con 0x7fee180c4160 2022-06-27T11:57:25.706 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.684+0000 7fee1d66d700 1 --2- 172.21.15.195:0/2177833348 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fee00003920 0x7fee00003d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:25.706 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.684+0000 7fee1ce6c700 1 --2- 172.21.15.195:0/2177833348 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fee180c4160 0x7fee000046d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:25.707 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.684+0000 7fee1de6e700 1 --2- 172.21.15.195:0/2177833348 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fee00004c10 0x7fee00005080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:25.707 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.684+0000 7fee1d66d700 1 -- 172.21.15.195:0/2177833348 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fee180c4160 msgr2=0x7fee000046d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:57:25.707 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.684+0000 7fee1d66d700 1 --2- 172.21.15.195:0/2177833348 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fee180c4160 0x7fee000046d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:25.708 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.684+0000 7fee1d66d700 1 -- 172.21.15.195:0/2177833348 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fee00004c10 msgr2=0x7fee00005080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:57:25.708 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.684+0000 7fee1d66d700 1 --2- 172.21.15.195:0/2177833348 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fee00004c10 0x7fee00005080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:25.708 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.684+0000 7fee1d66d700 1 -- 172.21.15.195:0/2177833348 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee00003430 con 0x7fee00003920 2022-06-27T11:57:25.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.684+0000 7fee1d66d700 1 --2- 172.21.15.195:0/2177833348 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fee00003920 0x7fee00003d30 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fee08009ac0 tx=0x7fee08009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12bb2ab530f54c2f server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:57:25.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.685+0000 7fee07fff700 1 -- 172.21.15.195:0/2177833348 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee08009470 con 0x7fee00003920 2022-06-27T11:57:25.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.685+0000 7fee07fff700 1 -- 172.21.15.195:0/2177833348 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee0800e5e0 con 0x7fee00003920 2022-06-27T11:57:25.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.685+0000 7fee07fff700 1 -- 172.21.15.195:0/2177833348 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee0800e980 con 0x7fee00003920 2022-06-27T11:57:25.710 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.685+0000 7fee1f8d1700 1 -- 172.21.15.195:0/2177833348 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fee00003920 msgr2=0x7fee00003d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:25.710 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.685+0000 7fee1f8d1700 1 --2- 172.21.15.195:0/2177833348 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fee00003920 0x7fee00003d30 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fee08009ac0 tx=0x7fee08009060 comp rx=0 tx=0).stop 2022-06-27T11:57:25.712 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.685+0000 7fee1de6e700 1 -- 172.21.15.195:0/2177833348 reap_dead start 2022-06-27T11:57:25.712 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.685+0000 7fee1f8d1700 1 -- 172.21.15.195:0/2177833348 shutdown_connections 2022-06-27T11:57:25.713 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.685+0000 7fee1f8d1700 1 -- 172.21.15.195:0/2177833348 >> 172.21.15.195:0/2177833348 conn(0x7fee180c6350 msgr2=0x7fee180c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:57:25.713 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.685+0000 7fee1f8d1700 1 -- 172.21.15.195:0/2177833348 shutdown_connections 2022-06-27T11:57:25.713 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.685+0000 7fee1f8d1700 1 -- 172.21.15.195:0/2177833348 wait complete. 2022-06-27T11:57:25.714 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.686+0000 7fee1f8d1700 1 Processor -- start 2022-06-27T11:57:25.714 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.686+0000 7fee1f8d1700 1 -- start start 2022-06-27T11:57:25.714 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.687+0000 7fee1f8d1700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fee00003920 0x7fee18168d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:25.715 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.687+0000 7fee1f8d1700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fee180bcb30 0x7fee180c7990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:25.716 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.687+0000 7fee1f8d1700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fee18169a80 0x7fee18169ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:25.717 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.687+0000 7fee1f8d1700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee18169240 con 0x7fee18169a80 2022-06-27T11:57:25.720 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.687+0000 7fee1f8d1700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee1806aff0 con 0x7fee180bcb30 2022-06-27T11:57:25.720 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.687+0000 7fee1f8d1700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee181695f0 con 0x7fee00003920 2022-06-27T11:57:25.721 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.687+0000 7fee1de6e700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fee18169a80 0x7fee18169ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:25.721 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.687+0000 7fee1de6e700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fee18169a80 0x7fee18169ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45224/0 (socket says 172.21.15.195:45224) 2022-06-27T11:57:25.722 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.687+0000 7fee1de6e700 1 -- 172.21.15.195:0/673001338 learned_addr learned my addr 172.21.15.195:0/673001338 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:57:25.722 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.687+0000 7fee1ce6c700 1 --2- 172.21.15.195:0/673001338 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fee180bcb30 0x7fee180c7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:25.723 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.688+0000 7fee1d66d700 1 --2- 172.21.15.195:0/673001338 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fee00003920 0x7fee18168d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:25.723 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.688+0000 7fee1de6e700 1 -- 172.21.15.195:0/673001338 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fee00003920 msgr2=0x7fee18168d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:25.724 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.688+0000 7fee1de6e700 1 --2- 172.21.15.195:0/673001338 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fee00003920 0x7fee18168d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:25.724 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.688+0000 7fee1de6e700 1 -- 172.21.15.195:0/673001338 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fee180bcb30 msgr2=0x7fee180c7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:25.724 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.688+0000 7fee1de6e700 1 --2- 172.21.15.195:0/673001338 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fee180bcb30 0x7fee180c7990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:25.725 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.688+0000 7fee1de6e700 1 -- 172.21.15.195:0/673001338 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee1806a7c0 con 0x7fee18169a80 2022-06-27T11:57:25.725 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.688+0000 7fee1d66d700 1 --2- 172.21.15.195:0/673001338 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fee00003920 0x7fee18168d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T11:57:25.726 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.688+0000 7fee1de6e700 1 --2- 172.21.15.195:0/673001338 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fee18169a80 0x7fee18169ef0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fee1806dda0 tx=0x7fee100021f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:57:25.727 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.688+0000 7fee057fa700 1 -- 172.21.15.195:0/673001338 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee1000fc00 con 0x7fee18169a80 2022-06-27T11:57:25.727 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.688+0000 7fee057fa700 1 -- 172.21.15.195:0/673001338 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee1000fd80 con 0x7fee18169a80 2022-06-27T11:57:25.727 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.688+0000 7fee1f8d1700 1 -- 172.21.15.195:0/673001338 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee180c8cf0 con 0x7fee18169a80 2022-06-27T11:57:25.728 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.688+0000 7fee057fa700 1 -- 172.21.15.195:0/673001338 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee10013600 con 0x7fee18169a80 2022-06-27T11:57:25.728 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.688+0000 7fee1f8d1700 1 -- 172.21.15.195:0/673001338 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee1806aa30 con 0x7fee18169a80 2022-06-27T11:57:25.729 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.689+0000 7fee1f8d1700 1 -- 172.21.15.195:0/673001338 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee180722c0 con 0x7fee18169a80 2022-06-27T11:57:25.730 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.689+0000 7fee057fa700 1 -- 172.21.15.195:0/673001338 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fee10013970 con 0x7fee18169a80 2022-06-27T11:57:25.731 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.690+0000 7fee057fa700 1 --2- 172.21.15.195:0/673001338 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fedf8063d00 0x7fedf80661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:25.731 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.690+0000 7fee057fa700 1 -- 172.21.15.195:0/673001338 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6546+0+0 (secure 0 0 0) 0x7fee10080350 con 0x7fee18169a80 2022-06-27T11:57:25.732 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.690+0000 7fee1d66d700 1 --2- 172.21.15.195:0/673001338 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fedf8063d00 0x7fedf80661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:25.732 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.690+0000 7fee1d66d700 1 --2- 172.21.15.195:0/673001338 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fedf8063d00 0x7fedf80661b0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fee08009ac0 tx=0x7fee0801a000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:57:25.732 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.694+0000 7fee057fa700 1 -- 172.21.15.195:0/673001338 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fee1004ed50 con 0x7fee18169a80 2022-06-27T11:57:25.923 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.922+0000 7fee1f8d1700 1 -- 172.21.15.195:0/673001338 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fee1806f240 con 0x7fee18169a80 2022-06-27T11:57:25.924 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.923+0000 7fee057fa700 1 -- 172.21.15.195:0/673001338 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+5374 (secure 0 0 0) 0x7fee1001a020 con 0x7fee18169a80 2022-06-27T11:57:25.925 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:57:25.925 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:57:22.984765+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4976},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4976":{"gid":4976,"name":"b","rank":0,"incarnation":32,"state":"up:clientreplay","state_seq":26,"addr":"172.21.15.202:6833/4154570751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":4154570751},{"type":"v1","addr":"172.21.15.202:6833","nonce":4154570751}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:57:25.928 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.925+0000 7fee1f8d1700 1 -- 172.21.15.195:0/673001338 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fedf8063d00 msgr2=0x7fedf80661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:25.928 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.925+0000 7fee1f8d1700 1 --2- 172.21.15.195:0/673001338 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fedf8063d00 0x7fedf80661b0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fee08009ac0 tx=0x7fee0801a000 comp rx=0 tx=0).stop 2022-06-27T11:57:25.929 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.925+0000 7fee1f8d1700 1 -- 172.21.15.195:0/673001338 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fee18169a80 msgr2=0x7fee18169ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:25.929 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.925+0000 7fee1f8d1700 1 --2- 172.21.15.195:0/673001338 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fee18169a80 0x7fee18169ef0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fee1806dda0 tx=0x7fee100021f0 comp rx=0 tx=0).stop 2022-06-27T11:57:25.931 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.925+0000 7fee1f8d1700 1 -- 172.21.15.195:0/673001338 shutdown_connections 2022-06-27T11:57:25.931 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.925+0000 7fee1f8d1700 1 --2- 172.21.15.195:0/673001338 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fedf8063d00 0x7fedf80661b0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:25.931 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.925+0000 7fee1f8d1700 1 --2- 172.21.15.195:0/673001338 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fee18169a80 0x7fee18169ef0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:25.932 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.925+0000 7fee1f8d1700 1 --2- 172.21.15.195:0/673001338 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fee00003920 0x7fee18168d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:25.932 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.925+0000 7fee1f8d1700 1 --2- 172.21.15.195:0/673001338 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fee180bcb30 0x7fee180c7990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:25.933 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.925+0000 7fee1f8d1700 1 -- 172.21.15.195:0/673001338 >> 172.21.15.195:0/673001338 conn(0x7fee180c6350 msgr2=0x7fee181504b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:57:25.933 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.925+0000 7fee1f8d1700 1 -- 172.21.15.195:0/673001338 shutdown_connections 2022-06-27T11:57:25.934 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:25.925+0000 7fee1f8d1700 1 -- 172.21.15.195:0/673001338 wait complete. 2022-06-27T11:57:25.934 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 35 2022-06-27T11:57:25.939 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 35, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-06-27T11:47:52.358095+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 35, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4976": { "addr": "172.21.15.202:6833/4154570751", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6832", "nonce": 4154570751, "type": "v2" }, { "addr": "172.21.15.202:6833", "nonce": 4154570751, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4976, "incarnation": 32, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:clientreplay", "state_seq": 26 } }, "last_failure": 0, "last_failure_osd_epoch": 52, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-06-27T11:57:22.984765+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4976 } } } ], "standbys": [ { "addr": "172.21.15.202:6837/3305006841", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6836", "nonce": 3305006841, "type": "v2" }, { "addr": "172.21.15.202:6837", "nonce": 3305006841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6835/2060709822", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6834", "nonce": 2060709822, "type": "v2" }, { "addr": "172.21.15.195:6835", "nonce": 2060709822, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6838/1598751652", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6836", "nonce": 1598751652, "type": "v2" }, { "addr": "172.21.15.195:6838", "nonce": 1598751652, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4275, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6839/1313800661", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6837", "nonce": 1313800661, "type": "v2" }, { "addr": "172.21.15.195:6839", "nonce": 1313800661, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4278, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-06-27T11:57:27.942 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:57:28.084 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.084+0000 7f2d507af700 1 Processor -- start 2022-06-27T11:57:28.084 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.084+0000 7f2d507af700 1 -- start start 2022-06-27T11:57:28.086 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.085+0000 7f2d507af700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f2d480c7050 0x7f2d4806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:28.086 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.085+0000 7f2d507af700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2d4806a690 0x7f2d4806ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:28.086 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.085+0000 7f2d507af700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d48156a70 con 0x7f2d480c7db0 2022-06-27T11:57:28.087 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.085+0000 7f2d507af700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d4806a420 con 0x7f2d480c7050 2022-06-27T11:57:28.087 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.085+0000 7f2d507af700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d48069b80 con 0x7f2d4806a690 2022-06-27T11:57:28.087 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.086+0000 7f2d4e54b700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f2d480c7db0 0x7f2d48156f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:51416/0 (socket says 172.21.15.195:51416) 2022-06-27T11:57:28.087 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.086+0000 7f2d4e54b700 1 -- 172.21.15.195:0/2511805854 learned_addr learned my addr 172.21.15.195:0/2511805854 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:57:28.088 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.086+0000 7f2d4ed4c700 1 --2- 172.21.15.195:0/2511805854 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2d4806a690 0x7f2d4806ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:28.088 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.086+0000 7f2d4dd4a700 1 --2- 172.21.15.195:0/2511805854 >> v2:172.21.15.202:3300/0 conn(0x7f2d480c7050 0x7f2d4806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:28.088 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.086+0000 7f2d4d549700 1 -- 172.21.15.195:0/2511805854 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3543830103 0 0) 0x7f2d48156a70 con 0x7f2d480c7db0 2022-06-27T11:57:28.089 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.086+0000 7f2d4d549700 1 -- 172.21.15.195:0/2511805854 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d30003430 con 0x7f2d480c7db0 2022-06-27T11:57:28.089 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.086+0000 7f2d4ed4c700 1 -- 172.21.15.195:0/2511805854 >> v2:172.21.15.202:3300/0 conn(0x7f2d480c7050 msgr2=0x7f2d4806b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:57:28.089 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.086+0000 7f2d4ed4c700 1 --2- 172.21.15.195:0/2511805854 >> v2:172.21.15.202:3300/0 conn(0x7f2d480c7050 0x7f2d4806b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:28.090 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.086+0000 7f2d4ed4c700 1 -- 172.21.15.195:0/2511805854 >> v1:172.21.15.195:6789/0 conn(0x7f2d480c7db0 legacy=0x7f2d48156f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:28.090 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.086+0000 7f2d4ed4c700 1 -- 172.21.15.195:0/2511805854 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d48069e60 con 0x7f2d4806a690 2022-06-27T11:57:28.097 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.086+0000 7f2d4ed4c700 1 --2- 172.21.15.195:0/2511805854 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2d4806a690 0x7f2d4806ab10 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2d40009e60 tx=0x7f2d40009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3fe1fbbc9b254076 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:57:28.097 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.086+0000 7f2d4d549700 1 -- 172.21.15.195:0/2511805854 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d4000e070 con 0x7f2d4806a690 2022-06-27T11:57:28.097 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.086+0000 7f2d4d549700 1 -- 172.21.15.195:0/2511805854 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d4000bb40 con 0x7f2d4806a690 2022-06-27T11:57:28.098 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.086+0000 7f2d4d549700 1 -- 172.21.15.195:0/2511805854 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d40012430 con 0x7f2d4806a690 2022-06-27T11:57:28.098 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.087+0000 7f2d507af700 1 -- 172.21.15.195:0/2511805854 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2d4806a690 msgr2=0x7f2d4806ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:28.098 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.087+0000 7f2d507af700 1 --2- 172.21.15.195:0/2511805854 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2d4806a690 0x7f2d4806ab10 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2d40009e60 tx=0x7f2d40009780 comp rx=0 tx=0).stop 2022-06-27T11:57:28.098 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.087+0000 7f2d507af700 1 -- 172.21.15.195:0/2511805854 shutdown_connections 2022-06-27T11:57:28.099 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.087+0000 7f2d507af700 1 --2- 172.21.15.195:0/2511805854 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2d4806a690 0x7f2d4806ab10 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:28.099 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.087+0000 7f2d507af700 1 --2- 172.21.15.195:0/2511805854 >> v2:172.21.15.202:3300/0 conn(0x7f2d480c7050 0x7f2d4806b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:28.099 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.087+0000 7f2d507af700 1 -- 172.21.15.195:0/2511805854 >> 172.21.15.195:0/2511805854 conn(0x7f2d48063af0 msgr2=0x7f2d48063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:57:28.100 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.087+0000 7f2d507af700 1 -- 172.21.15.195:0/2511805854 shutdown_connections 2022-06-27T11:57:28.100 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.087+0000 7f2d507af700 1 -- 172.21.15.195:0/2511805854 wait complete. 2022-06-27T11:57:28.100 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.088+0000 7f2d507af700 1 Processor -- start 2022-06-27T11:57:28.100 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.088+0000 7f2d507af700 1 -- start start 2022-06-27T11:57:28.100 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.089+0000 7f2d507af700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2d4806a690 0x7f2d48155c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:28.101 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.089+0000 7f2d507af700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2d480c7050 0x7f2d48166140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:28.101 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.089+0000 7f2d507af700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2d480c7db0 0x7f2d48166680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:28.101 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.089+0000 7f2d507af700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d480c3490 con 0x7f2d480c7db0 2022-06-27T11:57:28.101 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.089+0000 7f2d507af700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d4814fa00 con 0x7f2d4806a690 2022-06-27T11:57:28.102 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.089+0000 7f2d507af700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d481695c0 con 0x7f2d480c7050 2022-06-27T11:57:28.102 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.089+0000 7f2d4ed4c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2d480c7db0 0x7f2d48166680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:28.102 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.089+0000 7f2d4ed4c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2d480c7db0 0x7f2d48166680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45236/0 (socket says 172.21.15.195:45236) 2022-06-27T11:57:28.102 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.089+0000 7f2d4ed4c700 1 -- 172.21.15.195:0/2200238915 learned_addr learned my addr 172.21.15.195:0/2200238915 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:57:28.102 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.089+0000 7f2d4e54b700 1 --2- 172.21.15.195:0/2200238915 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2d4806a690 0x7f2d48155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:28.103 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.089+0000 7f2d4dd4a700 1 --2- 172.21.15.195:0/2200238915 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2d480c7050 0x7f2d48166140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:28.103 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.089+0000 7f2d4ed4c700 1 -- 172.21.15.195:0/2200238915 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2d480c7050 msgr2=0x7f2d48166140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:28.103 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.089+0000 7f2d4ed4c700 1 --2- 172.21.15.195:0/2200238915 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2d480c7050 0x7f2d48166140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:28.103 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.089+0000 7f2d4ed4c700 1 -- 172.21.15.195:0/2200238915 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2d4806a690 msgr2=0x7f2d48155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:28.104 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.089+0000 7f2d4ed4c700 1 --2- 172.21.15.195:0/2200238915 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2d4806a690 0x7f2d48155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:28.104 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.089+0000 7f2d4ed4c700 1 -- 172.21.15.195:0/2200238915 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d40009a20 con 0x7f2d480c7db0 2022-06-27T11:57:28.104 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.090+0000 7f2d4ed4c700 1 --2- 172.21.15.195:0/2200238915 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2d480c7db0 0x7f2d48166680 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f2d40013000 tx=0x7f2d40019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:57:28.104 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.090+0000 7f2d367fc700 1 -- 172.21.15.195:0/2200238915 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d40003e40 con 0x7f2d480c7db0 2022-06-27T11:57:28.104 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.090+0000 7f2d367fc700 1 -- 172.21.15.195:0/2200238915 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d40021070 con 0x7f2d480c7db0 2022-06-27T11:57:28.105 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.090+0000 7f2d507af700 1 -- 172.21.15.195:0/2200238915 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d481698a0 con 0x7f2d480c7db0 2022-06-27T11:57:28.105 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.090+0000 7f2d367fc700 1 -- 172.21.15.195:0/2200238915 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d400129f0 con 0x7f2d480c7db0 2022-06-27T11:57:28.105 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.090+0000 7f2d507af700 1 -- 172.21.15.195:0/2200238915 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d48169e60 con 0x7f2d480c7db0 2022-06-27T11:57:28.105 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.091+0000 7f2d507af700 1 -- 172.21.15.195:0/2200238915 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d480722c0 con 0x7f2d480c7db0 2022-06-27T11:57:28.105 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.091+0000 7f2d367fc700 1 -- 172.21.15.195:0/2200238915 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f2d40004040 con 0x7f2d480c7db0 2022-06-27T11:57:28.106 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.092+0000 7f2d367fc700 1 --2- 172.21.15.195:0/2200238915 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2d28063d00 0x7f2d280661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:28.106 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.092+0000 7f2d367fc700 1 -- 172.21.15.195:0/2200238915 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6546+0+0 (secure 0 0 0) 0x7f2d40082bb0 con 0x7f2d480c7db0 2022-06-27T11:57:28.106 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.092+0000 7f2d4e54b700 1 --2- 172.21.15.195:0/2200238915 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2d28063d00 0x7f2d280661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:28.106 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.092+0000 7f2d4e54b700 1 --2- 172.21.15.195:0/2200238915 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2d28063d00 0x7f2d280661b0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f2d44005790 tx=0x7f2d44009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:57:28.107 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.096+0000 7f2d367fc700 1 -- 172.21.15.195:0/2200238915 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f2d40089080 con 0x7f2d480c7db0 2022-06-27T11:57:28.323 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.323+0000 7f2d507af700 1 -- 172.21.15.195:0/2200238915 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2d4806a140 con 0x7f2d480c7db0 2022-06-27T11:57:28.324 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.324+0000 7f2d367fc700 1 -- 172.21.15.195:0/2200238915 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+5374 (secure 0 0 0) 0x7f2d400515b0 con 0x7f2d480c7db0 2022-06-27T11:57:28.324 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:57:28.325 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:57:22.984765+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4976},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4976":{"gid":4976,"name":"b","rank":0,"incarnation":32,"state":"up:clientreplay","state_seq":26,"addr":"172.21.15.202:6833/4154570751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":4154570751},{"type":"v1","addr":"172.21.15.202:6833","nonce":4154570751}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:57:28.326 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.326+0000 7f2d507af700 1 -- 172.21.15.195:0/2200238915 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2d28063d00 msgr2=0x7f2d280661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:28.326 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.326+0000 7f2d507af700 1 --2- 172.21.15.195:0/2200238915 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2d28063d00 0x7f2d280661b0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f2d44005790 tx=0x7f2d44009040 comp rx=0 tx=0).stop 2022-06-27T11:57:28.327 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.326+0000 7f2d507af700 1 -- 172.21.15.195:0/2200238915 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2d480c7db0 msgr2=0x7f2d48166680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:28.327 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.326+0000 7f2d507af700 1 --2- 172.21.15.195:0/2200238915 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2d480c7db0 0x7f2d48166680 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f2d40013000 tx=0x7f2d40019000 comp rx=0 tx=0).stop 2022-06-27T11:57:28.327 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.326+0000 7f2d507af700 1 -- 172.21.15.195:0/2200238915 shutdown_connections 2022-06-27T11:57:28.327 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.326+0000 7f2d507af700 1 --2- 172.21.15.195:0/2200238915 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2d28063d00 0x7f2d280661b0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:28.328 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.326+0000 7f2d507af700 1 --2- 172.21.15.195:0/2200238915 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2d480c7db0 0x7f2d48166680 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:28.328 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.326+0000 7f2d507af700 1 --2- 172.21.15.195:0/2200238915 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2d4806a690 0x7f2d48155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:28.329 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.326+0000 7f2d507af700 1 --2- 172.21.15.195:0/2200238915 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2d480c7050 0x7f2d48166140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:28.329 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.326+0000 7f2d507af700 1 -- 172.21.15.195:0/2200238915 >> 172.21.15.195:0/2200238915 conn(0x7f2d48063af0 msgr2=0x7f2d480ba330 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:57:28.330 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.326+0000 7f2d507af700 1 -- 172.21.15.195:0/2200238915 shutdown_connections 2022-06-27T11:57:28.330 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:28.326+0000 7f2d507af700 1 -- 172.21.15.195:0/2200238915 wait complete. 2022-06-27T11:57:28.330 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 35 2022-06-27T11:57:28.338 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T11:57:28.967 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T11:57:28.972 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T11:57:30.340 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:57:30.464 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.461+0000 7feee4224700 1 Processor -- start 2022-06-27T11:57:30.464 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.461+0000 7feee4224700 1 -- start start 2022-06-27T11:57:30.465 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.462+0000 7feee4224700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7feedc154900 0x7feedc0c2120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:30.465 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.462+0000 7feee4224700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7feedc151930 0x7feedc0bf4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:30.465 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.462+0000 7feee4224700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feedc154da0 con 0x7feedc151560 2022-06-27T11:57:30.465 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.462+0000 7feee4224700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feedc0c12a0 con 0x7feedc151930 2022-06-27T11:57:30.466 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.462+0000 7feee4224700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feedc0c1580 con 0x7feedc154900 2022-06-27T11:57:30.466 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.462+0000 7feee27c1700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7feedc151930 0x7feedc0bf4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:30.466 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.462+0000 7feee27c1700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7feedc151930 0x7feedc0bf4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45262/0 (socket says 172.21.15.195:45262) 2022-06-27T11:57:30.466 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.462+0000 7feee1fc0700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7feedc151560 0x7feedc1541f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:51446/0 (socket says 172.21.15.195:51446) 2022-06-27T11:57:30.466 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.462+0000 7feee27c1700 1 -- 172.21.15.195:0/869771617 learned_addr learned my addr 172.21.15.195:0/869771617 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:57:30.467 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.463+0000 7feee17bf700 1 --2- 172.21.15.195:0/869771617 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7feedc154900 0x7feedc0c2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:30.467 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.463+0000 7feee27c1700 1 -- 172.21.15.195:0/869771617 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7feedc154900 msgr2=0x7feedc0c2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:57:30.467 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.463+0000 7feee27c1700 1 --2- 172.21.15.195:0/869771617 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7feedc154900 0x7feedc0c2120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:30.467 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.463+0000 7feee27c1700 1 -- 172.21.15.195:0/869771617 >> v1:172.21.15.195:6789/0 conn(0x7feedc151560 legacy=0x7feedc1541f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:30.468 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.463+0000 7feee0fbe700 1 -- 172.21.15.195:0/869771617 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 373611273 0 0) 0x7feedc154da0 con 0x7feedc151560 2022-06-27T11:57:30.468 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.463+0000 7feee27c1700 1 -- 172.21.15.195:0/869771617 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feedc0c29f0 con 0x7feedc151930 2022-06-27T11:57:30.468 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.463+0000 7feee27c1700 1 --2- 172.21.15.195:0/869771617 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7feedc151930 0x7feedc0bf4a0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7feed4009f20 tx=0x7feed4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bb67b86c3f7e33b server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:57:30.468 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.463+0000 7feee0fbe700 1 -- 172.21.15.195:0/869771617 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed400e070 con 0x7feedc151930 2022-06-27T11:57:30.468 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.463+0000 7feee0fbe700 1 -- 172.21.15.195:0/869771617 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feed400ba60 con 0x7feedc151930 2022-06-27T11:57:30.469 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.463+0000 7feee0fbe700 1 -- 172.21.15.195:0/869771617 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed40092e0 con 0x7feedc151930 2022-06-27T11:57:30.469 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.464+0000 7feee4224700 1 -- 172.21.15.195:0/869771617 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7feedc151930 msgr2=0x7feedc0bf4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:30.469 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.464+0000 7feee4224700 1 --2- 172.21.15.195:0/869771617 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7feedc151930 0x7feedc0bf4a0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7feed4009f20 tx=0x7feed4009840 comp rx=0 tx=0).stop 2022-06-27T11:57:30.470 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.464+0000 7feee4224700 1 -- 172.21.15.195:0/869771617 shutdown_connections 2022-06-27T11:57:30.470 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.464+0000 7feee4224700 1 --2- 172.21.15.195:0/869771617 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7feedc151930 0x7feedc0bf4a0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:30.470 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.464+0000 7feee4224700 1 --2- 172.21.15.195:0/869771617 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7feedc154900 0x7feedc0c2120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:30.470 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.464+0000 7feee4224700 1 -- 172.21.15.195:0/869771617 >> 172.21.15.195:0/869771617 conn(0x7feedc0c6350 msgr2=0x7feedc0c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:57:30.471 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.464+0000 7feee4224700 1 -- 172.21.15.195:0/869771617 shutdown_connections 2022-06-27T11:57:30.471 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.464+0000 7feee4224700 1 -- 172.21.15.195:0/869771617 wait complete. 2022-06-27T11:57:30.474 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.465+0000 7feee4224700 1 Processor -- start 2022-06-27T11:57:30.475 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.465+0000 7feee4224700 1 -- start start 2022-06-27T11:57:30.475 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.466+0000 7feee4224700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7feedc151560 0x7feedc0c9340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:30.475 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.466+0000 7feee4224700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7feedc151930 0x7feedc14e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:30.475 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.466+0000 7feee4224700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7feedc154900 0x7feedc14e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:30.476 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.466+0000 7feee4224700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feedc165650 con 0x7feedc154900 2022-06-27T11:57:30.476 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.466+0000 7feee4224700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feedc06aad0 con 0x7feedc151560 2022-06-27T11:57:30.476 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.466+0000 7feee4224700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feedc168200 con 0x7feedc151930 2022-06-27T11:57:30.476 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.466+0000 7feee27c1700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7feedc154900 0x7feedc14e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:30.476 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.466+0000 7feee27c1700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7feedc154900 0x7feedc14e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45268/0 (socket says 172.21.15.195:45268) 2022-06-27T11:57:30.477 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.466+0000 7feee27c1700 1 -- 172.21.15.195:0/517399563 learned_addr learned my addr 172.21.15.195:0/517399563 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:57:30.477 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.466+0000 7feee17bf700 1 --2- 172.21.15.195:0/517399563 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7feedc151930 0x7feedc14e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:30.477 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.466+0000 7feee1fc0700 1 --2- 172.21.15.195:0/517399563 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7feedc151560 0x7feedc0c9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:30.477 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.467+0000 7feee27c1700 1 -- 172.21.15.195:0/517399563 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7feedc151930 msgr2=0x7feedc14e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:30.478 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.467+0000 7feee27c1700 1 --2- 172.21.15.195:0/517399563 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7feedc151930 0x7feedc14e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:30.478 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.467+0000 7feee27c1700 1 -- 172.21.15.195:0/517399563 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7feedc151560 msgr2=0x7feedc0c9340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:30.478 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.467+0000 7feee27c1700 1 --2- 172.21.15.195:0/517399563 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7feedc151560 0x7feedc0c9340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:30.478 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.467+0000 7feee27c1700 1 -- 172.21.15.195:0/517399563 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feed4009ae0 con 0x7feedc154900 2022-06-27T11:57:30.478 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.467+0000 7feee27c1700 1 --2- 172.21.15.195:0/517399563 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7feedc154900 0x7feedc14e740 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7feedc06b560 tx=0x7feed4018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:57:30.479 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.467+0000 7feec9ffb700 1 -- 172.21.15.195:0/517399563 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed40094e0 con 0x7feedc154900 2022-06-27T11:57:30.479 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.467+0000 7feec9ffb700 1 -- 172.21.15.195:0/517399563 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feed40092e0 con 0x7feedc154900 2022-06-27T11:57:30.479 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.467+0000 7feee4224700 1 -- 172.21.15.195:0/517399563 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feedc1684e0 con 0x7feedc154900 2022-06-27T11:57:30.480 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.467+0000 7feec9ffb700 1 -- 172.21.15.195:0/517399563 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed4007a70 con 0x7feedc154900 2022-06-27T11:57:30.480 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.467+0000 7feee4224700 1 -- 172.21.15.195:0/517399563 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feedc06b020 con 0x7feedc154900 2022-06-27T11:57:30.480 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.468+0000 7feee4224700 1 -- 172.21.15.195:0/517399563 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feedc165800 con 0x7feedc154900 2022-06-27T11:57:30.480 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.468+0000 7feec9ffb700 1 -- 172.21.15.195:0/517399563 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7feed402d070 con 0x7feedc154900 2022-06-27T11:57:30.480 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.469+0000 7feec9ffb700 1 --2- 172.21.15.195:0/517399563 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7feebc063d30 0x7feebc0661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:30.481 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.469+0000 7feec9ffb700 1 -- 172.21.15.195:0/517399563 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6546+0+0 (secure 0 0 0) 0x7feed40823c0 con 0x7feedc154900 2022-06-27T11:57:30.481 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.469+0000 7feee1fc0700 1 --2- 172.21.15.195:0/517399563 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7feebc063d30 0x7feebc0661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:30.481 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.469+0000 7feee1fc0700 1 --2- 172.21.15.195:0/517399563 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7feebc063d30 0x7feebc0661e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7feecc007670 tx=0x7feecc007b70 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:57:30.481 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.473+0000 7feec9ffb700 1 -- 172.21.15.195:0/517399563 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7feedc165800 con 0x7feedc154900 2022-06-27T11:57:30.697 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.697+0000 7feee4224700 1 -- 172.21.15.195:0/517399563 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feedc165800 con 0x7feedc154900 2022-06-27T11:57:30.722 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.698+0000 7feec9ffb700 1 -- 172.21.15.195:0/517399563 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+5374 (secure 0 0 0) 0x7feedc165800 con 0x7feedc154900 2022-06-27T11:57:30.722 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:57:30.725 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:57:22.984765+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4976},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4976":{"gid":4976,"name":"b","rank":0,"incarnation":32,"state":"up:clientreplay","state_seq":26,"addr":"172.21.15.202:6833/4154570751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":4154570751},{"type":"v1","addr":"172.21.15.202:6833","nonce":4154570751}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:57:30.728 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.700+0000 7feee4224700 1 -- 172.21.15.195:0/517399563 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7feebc063d30 msgr2=0x7feebc0661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:30.729 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.700+0000 7feee4224700 1 --2- 172.21.15.195:0/517399563 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7feebc063d30 0x7feebc0661e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7feecc007670 tx=0x7feecc007b70 comp rx=0 tx=0).stop 2022-06-27T11:57:30.729 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.700+0000 7feee4224700 1 -- 172.21.15.195:0/517399563 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7feedc154900 msgr2=0x7feedc14e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:30.730 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.700+0000 7feee4224700 1 --2- 172.21.15.195:0/517399563 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7feedc154900 0x7feedc14e740 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7feedc06b560 tx=0x7feed4018000 comp rx=0 tx=0).stop 2022-06-27T11:57:30.731 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.700+0000 7feee4224700 1 -- 172.21.15.195:0/517399563 shutdown_connections 2022-06-27T11:57:30.731 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.700+0000 7feee4224700 1 --2- 172.21.15.195:0/517399563 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7feebc063d30 0x7feebc0661e0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:30.731 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.700+0000 7feee4224700 1 --2- 172.21.15.195:0/517399563 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7feedc154900 0x7feedc14e740 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:30.732 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.700+0000 7feee4224700 1 --2- 172.21.15.195:0/517399563 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7feedc151560 0x7feedc0c9340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:30.732 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.700+0000 7feee4224700 1 --2- 172.21.15.195:0/517399563 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7feedc151930 0x7feedc14e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:30.732 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.700+0000 7feee4224700 1 -- 172.21.15.195:0/517399563 >> 172.21.15.195:0/517399563 conn(0x7feedc0c6350 msgr2=0x7feedc1504d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:57:30.733 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.700+0000 7feee4224700 1 -- 172.21.15.195:0/517399563 shutdown_connections 2022-06-27T11:57:30.733 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:30.700+0000 7feee4224700 1 -- 172.21.15.195:0/517399563 wait complete. 2022-06-27T11:57:30.734 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 35 2022-06-27T11:57:30.735 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T11:57:32.737 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:57:32.873 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.873+0000 7fd40a286700 1 Processor -- start 2022-06-27T11:57:32.874 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.873+0000 7fd40a286700 1 -- start start 2022-06-27T11:57:32.886 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.874+0000 7fd40a286700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fd40406fef0 0x7fd4041570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:32.886 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.874+0000 7fd40a286700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd40406fba0 con 0x7fd40406e850 2022-06-27T11:57:32.887 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.874+0000 7fd40a286700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd40406f5e0 con 0x7fd4040c34a0 2022-06-27T11:57:32.887 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.874+0000 7fd40a286700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4041575e0 con 0x7fd40406fef0 2022-06-27T11:57:32.888 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.874+0000 7fd403fff700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fd40406e850 0x7fd404069de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:51462/0 (socket says 172.21.15.195:51462) 2022-06-27T11:57:32.888 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.874+0000 7fd403fff700 1 -- 172.21.15.195:0/2554741083 learned_addr learned my addr 172.21.15.195:0/2554741083 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:57:32.889 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.875+0000 7fd4037fe700 1 --2- 172.21.15.195:0/2554741083 >> v2:172.21.15.202:3300/0 conn(0x7fd40406fef0 0x7fd4041570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:32.889 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.875+0000 7fd402ffd700 1 -- 172.21.15.195:0/2554741083 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 679420347 0 0) 0x7fd40406fba0 con 0x7fd40406e850 2022-06-27T11:57:32.889 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.875+0000 7fd402ffd700 1 -- 172.21.15.195:0/2554741083 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3f0003430 con 0x7fd40406e850 2022-06-27T11:57:32.890 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.875+0000 7fd402ffd700 1 -- 172.21.15.195:0/2554741083 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 540580190 0 0) 0x7fd40406f5e0 con 0x7fd4040c34a0 2022-06-27T11:57:32.890 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.875+0000 7fd402ffd700 1 -- 172.21.15.195:0/2554741083 --> v1:172.21.15.202:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd40406fba0 con 0x7fd4040c34a0 2022-06-27T11:57:32.891 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.875+0000 7fd402ffd700 1 -- 172.21.15.195:0/2554741083 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2339109865 0 0) 0x7fd3f0003430 con 0x7fd40406e850 2022-06-27T11:57:32.891 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.875+0000 7fd402ffd700 1 -- 172.21.15.195:0/2554741083 >> v2:172.21.15.202:3300/0 conn(0x7fd40406fef0 msgr2=0x7fd4041570a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:57:32.892 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.875+0000 7fd402ffd700 1 --2- 172.21.15.195:0/2554741083 >> v2:172.21.15.202:3300/0 conn(0x7fd40406fef0 0x7fd4041570a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:32.892 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.875+0000 7fd402ffd700 1 -- 172.21.15.195:0/2554741083 >> v1:172.21.15.202:6789/0 conn(0x7fd4040c34a0 legacy=0x7fd4040c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:32.892 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.875+0000 7fd402ffd700 1 -- 172.21.15.195:0/2554741083 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd40406f8c0 con 0x7fd40406e850 2022-06-27T11:57:32.893 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.875+0000 7fd402ffd700 1 -- 172.21.15.195:0/2554741083 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7fd3f4001470 con 0x7fd40406e850 2022-06-27T11:57:32.896 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.875+0000 7fd402ffd700 1 -- 172.21.15.195:0/2554741083 >> v1:172.21.15.195:6789/0 conn(0x7fd40406e850 legacy=0x7fd404069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:32.897 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.875+0000 7fd402ffd700 1 --2- 172.21.15.195:0/2554741083 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd3f0003ab0 0x7fd3f0003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:32.897 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.875+0000 7fd402ffd700 1 --2- 172.21.15.195:0/2554741083 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd3f00046d0 0x7fd3f0004b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:32.898 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.875+0000 7fd402ffd700 1 --2- 172.21.15.195:0/2554741083 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd3f0005080 0x7fd3f000b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:32.898 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.875+0000 7fd402ffd700 1 -- 172.21.15.195:0/2554741083 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4041575e0 con 0x7fd3f0005080 2022-06-27T11:57:32.899 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.875+0000 7fd402ffd700 1 -- 172.21.15.195:0/2554741083 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f000bf70 con 0x7fd3f00046d0 2022-06-27T11:57:32.899 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.875+0000 7fd402ffd700 1 -- 172.21.15.195:0/2554741083 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f000c130 con 0x7fd3f0003ab0 2022-06-27T11:57:32.900 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.875+0000 7fd408823700 1 --2- 172.21.15.195:0/2554741083 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd3f0005080 0x7fd3f000b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:32.900 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.875+0000 7fd403fff700 1 --2- 172.21.15.195:0/2554741083 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd3f0003ab0 0x7fd3f0003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:32.900 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.875+0000 7fd4037fe700 1 --2- 172.21.15.195:0/2554741083 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd3f00046d0 0x7fd3f0004b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:32.901 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.875+0000 7fd408823700 1 -- 172.21.15.195:0/2554741083 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd3f0003ab0 msgr2=0x7fd3f0003ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:57:32.901 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.875+0000 7fd408823700 1 --2- 172.21.15.195:0/2554741083 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd3f0003ab0 0x7fd3f0003ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:32.902 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.875+0000 7fd408823700 1 -- 172.21.15.195:0/2554741083 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd3f00046d0 msgr2=0x7fd3f0004b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:57:32.902 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.875+0000 7fd408823700 1 --2- 172.21.15.195:0/2554741083 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd3f00046d0 0x7fd3f0004b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:32.903 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.875+0000 7fd408823700 1 -- 172.21.15.195:0/2554741083 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3f0003430 con 0x7fd3f0005080 2022-06-27T11:57:32.903 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.876+0000 7fd408823700 1 -- 172.21.15.195:0/2554741083 reap_dead start 2022-06-27T11:57:32.903 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.876+0000 7fd408823700 1 --2- 172.21.15.195:0/2554741083 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd3f0005080 0x7fd3f000b920 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc002010 tx=0x7fd3fc007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=969084d979c4195b server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:57:32.904 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.876+0000 7fd402ffd700 1 -- 172.21.15.195:0/2554741083 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3fc0019a0 con 0x7fd3f0005080 2022-06-27T11:57:32.904 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.876+0000 7fd402ffd700 1 -- 172.21.15.195:0/2554741083 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3fc00c8b0 con 0x7fd3f0005080 2022-06-27T11:57:32.905 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.876+0000 7fd402ffd700 1 -- 172.21.15.195:0/2554741083 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3fc00cc50 con 0x7fd3f0005080 2022-06-27T11:57:32.905 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.876+0000 7fd40a286700 1 -- 172.21.15.195:0/2554741083 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd3f0005080 msgr2=0x7fd3f000b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:32.906 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.876+0000 7fd40a286700 1 --2- 172.21.15.195:0/2554741083 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd3f0005080 0x7fd3f000b920 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc002010 tx=0x7fd3fc007f70 comp rx=0 tx=0).stop 2022-06-27T11:57:32.907 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.876+0000 7fd40a286700 1 -- 172.21.15.195:0/2554741083 shutdown_connections 2022-06-27T11:57:32.907 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.876+0000 7fd40a286700 1 --2- 172.21.15.195:0/2554741083 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd3f0005080 0x7fd3f000b920 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:32.908 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.876+0000 7fd40a286700 1 -- 172.21.15.195:0/2554741083 >> 172.21.15.195:0/2554741083 conn(0x7fd4040c6350 msgr2=0x7fd4040c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:57:32.909 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.876+0000 7fd40a286700 1 -- 172.21.15.195:0/2554741083 shutdown_connections 2022-06-27T11:57:32.909 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.876+0000 7fd40a286700 1 -- 172.21.15.195:0/2554741083 wait complete. 2022-06-27T11:57:32.909 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.877+0000 7fd40a286700 1 Processor -- start 2022-06-27T11:57:32.910 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.878+0000 7fd40a286700 1 -- start start 2022-06-27T11:57:32.910 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.878+0000 7fd40a286700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd3f0005080 0x7fd404152db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:32.911 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.878+0000 7fd40a286700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd404151aa0 0x7fd4040bac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:32.911 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.878+0000 7fd40a286700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd40416a580 0x7fd404144700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:32.912 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.878+0000 7fd40a286700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd404152920 con 0x7fd40416a580 2022-06-27T11:57:32.912 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.878+0000 7fd40a286700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4040ba330 con 0x7fd3f0005080 2022-06-27T11:57:32.913 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.878+0000 7fd40a286700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd404144c40 con 0x7fd404151aa0 2022-06-27T11:57:32.913 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.878+0000 7fd408823700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd40416a580 0x7fd404144700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:32.913 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.879+0000 7fd408823700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd40416a580 0x7fd404144700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45288/0 (socket says 172.21.15.195:45288) 2022-06-27T11:57:32.914 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.879+0000 7fd408823700 1 -- 172.21.15.195:0/3937292265 learned_addr learned my addr 172.21.15.195:0/3937292265 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:57:32.914 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.879+0000 7fd403fff700 1 --2- 172.21.15.195:0/3937292265 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd3f0005080 0x7fd404152db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:32.915 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.879+0000 7fd4037fe700 1 --2- 172.21.15.195:0/3937292265 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd404151aa0 0x7fd4040bac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:32.915 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.879+0000 7fd408823700 1 -- 172.21.15.195:0/3937292265 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd404151aa0 msgr2=0x7fd4040bac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:32.915 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.879+0000 7fd408823700 1 --2- 172.21.15.195:0/3937292265 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd404151aa0 0x7fd4040bac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:32.916 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.879+0000 7fd408823700 1 -- 172.21.15.195:0/3937292265 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd3f0005080 msgr2=0x7fd404152db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:32.916 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.879+0000 7fd408823700 1 --2- 172.21.15.195:0/3937292265 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd3f0005080 0x7fd404152db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:32.916 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.879+0000 7fd408823700 1 -- 172.21.15.195:0/3937292265 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd40416a9f0 con 0x7fd40416a580 2022-06-27T11:57:32.917 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.879+0000 7fd403fff700 1 --2- 172.21.15.195:0/3937292265 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd3f0005080 0x7fd404152db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T11:57:32.917 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.879+0000 7fd408823700 1 --2- 172.21.15.195:0/3937292265 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd40416a580 0x7fd404144700 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fd40406e1e0 tx=0x7fd3fc014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:57:32.917 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.879+0000 7fd3ebfff700 1 -- 172.21.15.195:0/3937292265 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3fc003e90 con 0x7fd40416a580 2022-06-27T11:57:32.921 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.879+0000 7fd3ebfff700 1 -- 172.21.15.195:0/3937292265 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3fc0052e0 con 0x7fd40416a580 2022-06-27T11:57:32.921 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.879+0000 7fd40a286700 1 -- 172.21.15.195:0/3937292265 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4040ba5d0 con 0x7fd40416a580 2022-06-27T11:57:32.922 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.879+0000 7fd3ebfff700 1 -- 172.21.15.195:0/3937292265 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3fc0056a0 con 0x7fd40416a580 2022-06-27T11:57:32.922 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.879+0000 7fd40a286700 1 -- 172.21.15.195:0/3937292265 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd404151ff0 con 0x7fd40416a580 2022-06-27T11:57:32.923 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.880+0000 7fd40a286700 1 -- 172.21.15.195:0/3937292265 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4040722c0 con 0x7fd40416a580 2022-06-27T11:57:32.923 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.881+0000 7fd3ebfff700 1 -- 172.21.15.195:0/3937292265 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fd3fc010050 con 0x7fd40416a580 2022-06-27T11:57:32.923 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.881+0000 7fd3ebfff700 1 --2- 172.21.15.195:0/3937292265 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd3e4063d30 0x7fd3e40661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:32.924 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.881+0000 7fd3ebfff700 1 -- 172.21.15.195:0/3937292265 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6546+0+0 (secure 0 0 0) 0x7fd3fc082040 con 0x7fd40416a580 2022-06-27T11:57:32.924 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.881+0000 7fd403fff700 1 --2- 172.21.15.195:0/3937292265 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd3e4063d30 0x7fd3e40661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:32.925 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.881+0000 7fd403fff700 1 --2- 172.21.15.195:0/3937292265 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd3e4063d30 0x7fd3e40661e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd3f40015a0 tx=0x7fd3f4009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:57:32.925 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:32.885+0000 7fd3ebfff700 1 -- 172.21.15.195:0/3937292265 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fd3fc050a40 con 0x7fd40416a580 2022-06-27T11:57:33.120 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:33.113+0000 7fd40a286700 1 -- 172.21.15.195:0/3937292265 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd4040c0ad0 con 0x7fd40416a580 2022-06-27T11:57:33.120 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:33.114+0000 7fd3ebfff700 1 -- 172.21.15.195:0/3937292265 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+5368 (secure 0 0 0) 0x7fd3fc054060 con 0x7fd40416a580 2022-06-27T11:57:33.121 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:57:33.122 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:57:31.004761+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4976},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4976":{"gid":4976,"name":"b","rank":0,"incarnation":32,"state":"up:active","state_seq":28,"addr":"172.21.15.202:6833/4154570751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":4154570751},{"type":"v1","addr":"172.21.15.202:6833","nonce":4154570751}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:57:33.125 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:33.116+0000 7fd40a286700 1 -- 172.21.15.195:0/3937292265 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd3e4063d30 msgr2=0x7fd3e40661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:33.133 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:33.116+0000 7fd40a286700 1 --2- 172.21.15.195:0/3937292265 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd3e4063d30 0x7fd3e40661e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd3f40015a0 tx=0x7fd3f4009040 comp rx=0 tx=0).stop 2022-06-27T11:57:33.133 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:33.116+0000 7fd40a286700 1 -- 172.21.15.195:0/3937292265 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd40416a580 msgr2=0x7fd404144700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:33.134 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:33.116+0000 7fd40a286700 1 --2- 172.21.15.195:0/3937292265 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd40416a580 0x7fd404144700 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fd40406e1e0 tx=0x7fd3fc014000 comp rx=0 tx=0).stop 2022-06-27T11:57:33.134 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:33.116+0000 7fd40a286700 1 -- 172.21.15.195:0/3937292265 shutdown_connections 2022-06-27T11:57:33.136 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:33.116+0000 7fd40a286700 1 --2- 172.21.15.195:0/3937292265 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd3e4063d30 0x7fd3e40661e0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:33.136 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:33.116+0000 7fd40a286700 1 --2- 172.21.15.195:0/3937292265 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd40416a580 0x7fd404144700 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:33.137 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:33.116+0000 7fd40a286700 1 --2- 172.21.15.195:0/3937292265 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd3f0005080 0x7fd404152db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:33.137 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:33.116+0000 7fd40a286700 1 --2- 172.21.15.195:0/3937292265 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd404151aa0 0x7fd4040bac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:33.138 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:33.116+0000 7fd40a286700 1 -- 172.21.15.195:0/3937292265 >> 172.21.15.195:0/3937292265 conn(0x7fd4040c6350 msgr2=0x7fd4041505f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:57:33.138 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:33.116+0000 7fd40a286700 1 -- 172.21.15.195:0/3937292265 shutdown_connections 2022-06-27T11:57:33.138 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:33.116+0000 7fd40a286700 1 -- 172.21.15.195:0/3937292265 wait complete. 2022-06-27T11:57:33.138 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 36 2022-06-27T11:57:33.140 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=4856 2022-06-27T11:57:33.140 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 3 secs before reviving mds.d 2022-06-27T11:57:36.141 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.d 2022-06-27T11:57:36.142 INFO:tasks.ceph.mds.d:Restarting daemon 2022-06-27T11:57:36.142 DEBUG:teuthology.orchestra.run.smithi202:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2022-06-27T11:57:36.145 INFO:tasks.ceph.mds.d:Started 2022-06-27T11:57:36.146 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:57:36.223 INFO:tasks.ceph.mds.d.smithi202.stdout:starting mds.d at 2022-06-27T11:57:36.276 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.276+0000 7f354b468700 1 Processor -- start 2022-06-27T11:57:36.277 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.277+0000 7f354b468700 1 -- start start 2022-06-27T11:57:36.278 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.277+0000 7f354b468700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f354406fef0 0x7f35441570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:36.279 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.277+0000 7f354b468700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f354406fba0 con 0x7f35440c34a0 2022-06-27T11:57:36.279 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.278+0000 7f354b468700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f354406f5e0 con 0x7f354406e850 2022-06-27T11:57:36.279 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.278+0000 7f354b468700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35441575e0 con 0x7f354406fef0 2022-06-27T11:57:36.293 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.278+0000 7f3549a05700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f35440c34a0 0x7f35440c2920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:51492/0 (socket says 172.21.15.195:51492) 2022-06-27T11:57:36.293 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.278+0000 7f3549a05700 1 -- 172.21.15.195:0/1515035007 learned_addr learned my addr 172.21.15.195:0/1515035007 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:57:36.294 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.278+0000 7f3548a03700 1 --2- 172.21.15.195:0/1515035007 >> v2:172.21.15.202:3300/0 conn(0x7f354406fef0 0x7f35441570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:36.294 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.278+0000 7f3533fff700 1 -- 172.21.15.195:0/1515035007 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3566430239 0 0) 0x7f354406fba0 con 0x7f35440c34a0 2022-06-27T11:57:36.294 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.278+0000 7f3533fff700 1 -- 172.21.15.195:0/1515035007 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f352c003430 con 0x7f35440c34a0 2022-06-27T11:57:36.294 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.278+0000 7f3533fff700 1 -- 172.21.15.195:0/1515035007 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 656898986 0 0) 0x7f354406f5e0 con 0x7f354406e850 2022-06-27T11:57:36.295 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.278+0000 7f3533fff700 1 -- 172.21.15.195:0/1515035007 --> v1:172.21.15.202:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f354406fba0 con 0x7f354406e850 2022-06-27T11:57:36.295 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.278+0000 7f3548a03700 1 -- 172.21.15.195:0/1515035007 >> v1:172.21.15.202:6789/0 conn(0x7f354406e850 legacy=0x7f3544069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:36.295 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.278+0000 7f3548a03700 1 -- 172.21.15.195:0/1515035007 >> v1:172.21.15.195:6789/0 conn(0x7f35440c34a0 legacy=0x7f35440c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:36.295 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.278+0000 7f3548a03700 1 -- 172.21.15.195:0/1515035007 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f354406f8c0 con 0x7f354406fef0 2022-06-27T11:57:36.296 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.278+0000 7f3548a03700 1 --2- 172.21.15.195:0/1515035007 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f354406fef0 0x7f35441570a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f35400097a0 tx=0x7f3540009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4dafda637197618c server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:57:36.296 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.279+0000 7f3533fff700 1 -- 172.21.15.195:0/1515035007 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3540011630 con 0x7f354406fef0 2022-06-27T11:57:36.296 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.279+0000 7f3533fff700 1 -- 172.21.15.195:0/1515035007 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3540011790 con 0x7f354406fef0 2022-06-27T11:57:36.296 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.279+0000 7f3533fff700 1 -- 172.21.15.195:0/1515035007 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3540011b10 con 0x7f354406fef0 2022-06-27T11:57:36.297 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.279+0000 7f354b468700 1 -- 172.21.15.195:0/1515035007 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f354406fef0 msgr2=0x7f35441570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:36.297 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.279+0000 7f354b468700 1 --2- 172.21.15.195:0/1515035007 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f354406fef0 0x7f35441570a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f35400097a0 tx=0x7f3540009060 comp rx=0 tx=0).stop 2022-06-27T11:57:36.297 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.279+0000 7f354b468700 1 -- 172.21.15.195:0/1515035007 shutdown_connections 2022-06-27T11:57:36.297 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.279+0000 7f354b468700 1 --2- 172.21.15.195:0/1515035007 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f354406fef0 0x7f35441570a0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:36.298 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.279+0000 7f354b468700 1 -- 172.21.15.195:0/1515035007 >> 172.21.15.195:0/1515035007 conn(0x7f35440c6350 msgr2=0x7f35440c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:57:36.298 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.279+0000 7f354b468700 1 -- 172.21.15.195:0/1515035007 shutdown_connections 2022-06-27T11:57:36.298 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.279+0000 7f354b468700 1 -- 172.21.15.195:0/1515035007 wait complete. 2022-06-27T11:57:36.298 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.281+0000 7f354b468700 1 Processor -- start 2022-06-27T11:57:36.298 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.281+0000 7f354b468700 1 -- start start 2022-06-27T11:57:36.299 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.281+0000 7f354b468700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f354406e850 0x7f3544163e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:36.299 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.281+0000 7f354b468700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f35440c34a0 0x7f35441649b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:36.299 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.281+0000 7f354b468700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f35440bf740 0x7f35440bfb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:36.299 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.281+0000 7f354b468700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3544069b10 con 0x7f35440bf740 2022-06-27T11:57:36.300 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.281+0000 7f354b468700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35440b7de0 con 0x7f354406e850 2022-06-27T11:57:36.300 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.281+0000 7f354b468700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35441571d0 con 0x7f35440c34a0 2022-06-27T11:57:36.300 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.282+0000 7f3549a05700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f35440bf740 0x7f35440bfb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:36.300 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.282+0000 7f3549204700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f354406e850 0x7f3544163e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:36.300 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.282+0000 7f3549a05700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f35440bf740 0x7f35440bfb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45310/0 (socket says 172.21.15.195:45310) 2022-06-27T11:57:36.301 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.282+0000 7f3549a05700 1 -- 172.21.15.195:0/1284797106 learned_addr learned my addr 172.21.15.195:0/1284797106 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:57:36.301 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.282+0000 7f3548a03700 1 --2- 172.21.15.195:0/1284797106 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f35440c34a0 0x7f35441649b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:36.301 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.282+0000 7f3549204700 1 -- 172.21.15.195:0/1284797106 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f35440c34a0 msgr2=0x7f35441649b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:36.301 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.282+0000 7f3549204700 1 --2- 172.21.15.195:0/1284797106 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f35440c34a0 0x7f35441649b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:36.302 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.282+0000 7f3549204700 1 -- 172.21.15.195:0/1284797106 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f35440bf740 msgr2=0x7f35440bfb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:36.302 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.282+0000 7f3549204700 1 --2- 172.21.15.195:0/1284797106 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f35440bf740 0x7f35440bfb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:36.302 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.282+0000 7f3549204700 1 -- 172.21.15.195:0/1284797106 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35400092e0 con 0x7f354406e850 2022-06-27T11:57:36.302 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.282+0000 7f3549204700 1 --2- 172.21.15.195:0/1284797106 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f354406e850 0x7f3544163e70 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f3534000c00 tx=0x7f3534007ba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:57:36.303 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.282+0000 7f35317fa700 1 -- 172.21.15.195:0/1284797106 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35340050d0 con 0x7f354406e850 2022-06-27T11:57:36.303 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.282+0000 7f35317fa700 1 -- 172.21.15.195:0/1284797106 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3534005250 con 0x7f354406e850 2022-06-27T11:57:36.303 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.282+0000 7f354b468700 1 -- 172.21.15.195:0/1284797106 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35440b7020 con 0x7f354406e850 2022-06-27T11:57:36.303 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.282+0000 7f35317fa700 1 -- 172.21.15.195:0/1284797106 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3534003750 con 0x7f354406e850 2022-06-27T11:57:36.304 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.282+0000 7f354b468700 1 -- 172.21.15.195:0/1284797106 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35440b75e0 con 0x7f354406e850 2022-06-27T11:57:36.304 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.283+0000 7f354b468700 1 -- 172.21.15.195:0/1284797106 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35440c1000 con 0x7f354406e850 2022-06-27T11:57:36.304 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.284+0000 7f35317fa700 1 -- 172.21.15.195:0/1284797106 <== mon.1 v2:172.21.15.202:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f3534019030 con 0x7f354406e850 2022-06-27T11:57:36.304 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.284+0000 7f35317fa700 1 --2- 172.21.15.195:0/1284797106 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3524063d30 0x7f35240661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:36.305 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.284+0000 7f35317fa700 1 -- 172.21.15.195:0/1284797106 <== mon.1 v2:172.21.15.202:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6546+0+0 (secure 0 0 0) 0x7f353407f980 con 0x7f354406e850 2022-06-27T11:57:36.305 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.284+0000 7f3548a03700 1 --2- 172.21.15.195:0/1284797106 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3524063d30 0x7f35240661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:36.306 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.284+0000 7f3548a03700 1 --2- 172.21.15.195:0/1284797106 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3524063d30 0x7f35240661e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f35440c0a10 tx=0x7f3540017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:57:36.306 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.289+0000 7f35317fa700 1 -- 172.21.15.195:0/1284797106 <== mon.1 v2:172.21.15.202:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f353404e380 con 0x7f354406e850 2022-06-27T11:57:36.563 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.559+0000 7f354b468700 1 -- 172.21.15.195:0/1284797106 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f35440722c0 con 0x7f354406e850 2022-06-27T11:57:36.563 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.560+0000 7f35317fa700 1 -- 172.21.15.195:0/1284797106 <== mon.1 v2:172.21.15.202:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+5368 (secure 0 0 0) 0x7f3534017090 con 0x7f354406e850 2022-06-27T11:57:36.569 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:57:36.569 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:57:31.004761+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4976},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4976":{"gid":4976,"name":"b","rank":0,"incarnation":32,"state":"up:active","state_seq":28,"addr":"172.21.15.202:6833/4154570751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":4154570751},{"type":"v1","addr":"172.21.15.202:6833","nonce":4154570751}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:57:36.573 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.562+0000 7f354b468700 1 -- 172.21.15.195:0/1284797106 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3524063d30 msgr2=0x7f35240661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:36.573 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.562+0000 7f354b468700 1 --2- 172.21.15.195:0/1284797106 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3524063d30 0x7f35240661e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f35440c0a10 tx=0x7f3540017000 comp rx=0 tx=0).stop 2022-06-27T11:57:36.573 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.562+0000 7f354b468700 1 -- 172.21.15.195:0/1284797106 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f354406e850 msgr2=0x7f3544163e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:36.574 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.562+0000 7f354b468700 1 --2- 172.21.15.195:0/1284797106 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f354406e850 0x7f3544163e70 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f3534000c00 tx=0x7f3534007ba0 comp rx=0 tx=0).stop 2022-06-27T11:57:36.574 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.562+0000 7f354b468700 1 -- 172.21.15.195:0/1284797106 shutdown_connections 2022-06-27T11:57:36.575 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.562+0000 7f354b468700 1 --2- 172.21.15.195:0/1284797106 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3524063d30 0x7f35240661e0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:36.575 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.562+0000 7f354b468700 1 --2- 172.21.15.195:0/1284797106 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f35440bf740 0x7f35440bfb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:36.575 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.562+0000 7f354b468700 1 --2- 172.21.15.195:0/1284797106 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f354406e850 0x7f3544163e70 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:36.576 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.562+0000 7f354b468700 1 --2- 172.21.15.195:0/1284797106 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f35440c34a0 0x7f35441649b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:36.576 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.562+0000 7f354b468700 1 -- 172.21.15.195:0/1284797106 >> 172.21.15.195:0/1284797106 conn(0x7f35440c6350 msgr2=0x7f354406be00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:57:36.577 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.562+0000 7f354b468700 1 -- 172.21.15.195:0/1284797106 shutdown_connections 2022-06-27T11:57:36.579 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:36.562+0000 7f354b468700 1 -- 172.21.15.195:0/1284797106 wait complete. 2022-06-27T11:57:36.579 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 36 2022-06-27T11:57:36.582 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2022-06-27T11:57:37.012 INFO:tasks.ceph.mds.b.smithi202.stderr:2022-06-27T11:57:37.012+0000 7ff6aa27d700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2022-06-27T11:57:38.003 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 1% (201/11436) Updating files: 2% (229/11436) Updating files: 3% (344/11436) Updating files: 4% (458/11436) Updating files: 4% (505/11436) Updating files: 4% (523/11436) Updating files: 5% (572/11436) Updating files: 6% (687/11436) Updating files: 6% (740/11436) Updating files: 6% (757/11436) Updating files: 7% (801/11436) Updating files: 7% (896/11436) Updating files: 8% (915/11436) Updating files: 8% (973/11436) Updating files: 9% (1030/11436) Updating files: 9% (1142/11436) Updating files: 10% (1144/11436) Updating files: 10% (1179/11436) Updating files: 11% (1258/11436) Updating files: 11% (1261/11436) Updating files: 12% (1373/11436) Updating files: 13% (1487/11436) Updating files: 14% (1602/11436) Updating files: 14% (1645/11436) Updating files: 15% (1716/11436) Updating files: 15% (1731/11436) Updating files: 16% (1830/11436) Updating files: 16% (1854/11436) Updating files: 17% (1945/11436) Updating files: 17% (1987/11436) Updating files: 18% (2059/11436) Updating files: 18% (2084/11436) Updating files: 19% (2173/11436) Updating files: 19% (2209/11436) Updating files: 19% (2213/11436) Updating files: 20% (2288/11436) Updating files: 20% (2358/11436) Updating files: 21% (2402/11436) Updating files: 21% (2450/11436) Updating files: 22% (2516/11436) Updating files: 22% (2572/11436) Updating files: 22% (2622/11436) Updating files: 23% (2631/11436) Updating files: 24% (2745/11436) Updating files: 24% (2749/11436) Updating files: 25% (2859/11436) Updating files: 26% (2974/11436) Updating files: 27% (3088/11436) Updating files: 27% (3091/11436) Updating files: 28% (3203/11436) Updating files: 28% (3259/11436) Updating files: 28% (3290/11436) Updating files: 29% (3317/11436) Updating files: 30% (3431/11436) Updating files: 31% (3546/11436) Updating files: 31% (3556/11436) Updating files: 31% (3595/11436) Updating files: 32% (3660/11436) Updating files: 32% (3674/11436) Updating files: 33% (3774/11436) Updating files: 33% (3865/11436) Updating files: 34% (3889/11436) Updating files: 34% (3987/11436) Updating files: 35% (4003/11436) Updating files: 36% (4117/11436) Updating files: 36% (4128/11436) Updating files: 36% (4182/11436) Updating files: 37% (4232/11436) Updating files: 38% (4346/11436) Updating files: 38% (4401/11436) Updating files: 38% (4414/11436) Updating files: 39% (4461/11436) Updating files: 39% (4569/11436) Updating files: 40% (4575/11436) Updating files: 40% (4616/11436) Updating files: 41% (4689/11436) Updating files: 41% (4739/11436) Updating files: 41% (4755/11436) Updating files: 42% (4804/11436) Updating files: 43% (4918/11436) Updating files: 43% (5012/11436) Updating files: 44% (5032/11436) Updating files: 44% (5140/11436) Updating files: 45% (5147/11436) Updating files: 45% (5233/11436) Updating files: 46% (5261/11436) Updating files: 47% (5375/11436) Updating files: 47% (5434/11436) Updating files: 47% (5437/11436) Updating files: 48% (5490/11436) Updating files: 49% (5604/11436) Updating files: 50% (5718/11436) Updating files: 50% (5758/11436) Updating files: 51% (5833/11436) Updating files: 51% (5929/11436) Updating files: 52% (5947/11436) Updating files: 52% (6016/11436) Updating files: 53% (6062/11436) Updating files: 54% (6176/11436) Updating files: 55% (6290/11436) Updating files: 56% (6405/11436) Updating files: 56% (6456/11436) Updating files: 56% (6491/11436) Updating files: 57% (6519/11436) Updating files: 57% (6626/11436) Updating files: 58% (6633/11436) Updating files: 59% (6748/11436) Updating files: 60% (6862/11436) Updating files: 61% (6976/11436) Updating files: 61% (7052/11436) Updating files: 61% (7090/11436) Updating files: 62% (7091/11436) Updating files: 62% (7146/11436) Updating files: 63% (7205/11436) Updating files: 64% (7320/11436) Updating files: 65% (7434/11436) Updating files: 66% (7548/11436) Updating files: 66% (7592/11436) Updating files: 66% (7612/11436) Updating files: 67% (7663/11436) Updating files: 67% (7736/11436) Updating files: 68% (7777/11436) Updating files: 69% (7891/11436) Updating files: 69% (7906/11436) Updating files: 70% (8006/11436) Updating files: 70% (8023/11436) Updating files: 70% (8067/11436) Updating files: 70% (8068/11436) Updating files: 71% (8120/11436) Updating files: 71% (8142/11436) Updating files: 71% (8220/11436) Updating files: 72% (8234/11436) Updating files: 72% (8324/11436) Updating files: 73% (8349/11436) Updating files: 73% (8384/11436) Updating files: 74% (8463/11436) Updating files: 75% (8577/11436) Updating files: 76% (8692/11436) Updating files: 76% (8783/11436) Updating files: 76% (8798/11436) Updating files: 77% (8806/11436) Updating files: 77% (8875/11436) Updating files: 77% (8883/11436) Updating files: 78% (8921/11436) Updating files: 78% (9018/11436) Updating files: 79% (9035/11436) Updating files: 79% (9077/11436) Updating files: 79% (9100/11436) Updating files: 80% (9149/11436) Updating files: 80% (9207/11436) Updating files: 80% (9214/11436) Updating files: 81% (9264/11436) Updating files: 82% (9378/11436) Updating files: 82% (9384/11436) Updating files: 82% (9452/11436) Updating files: 83% (9492/11436) Updating files: 83% (9511/11436) Updating files: 83% (9541/11436) Updating files: 83% (9576/11436) Updating files: 84% (9607/11436) Updating files: 85% (9721/11436) Updating files: 86% (9835/11436) Updating files: 86% (9894/11436) Updating files: 87% (9950/11436) Updating files: 87% (10037/11436) Updating files: 87% (10046/11436) Updating files: 87% (10063/11436) Updating files: 88% (10064/11436) Updating files: 88% (10170/11436) Updating files: 88% (10172/11436) Updating files: 89% (10179/11436) Updating files: 89% (10285/11436) Updating files: 90% (10293/11436) Updating files: 91% (10407/11436) Updating files: 91% (10487/11436) Updating files: 92% (10522/11436) Updating files: 93% (10636/11436) Updating files: 94% (10750/11436) Updating files: 94% (10778/11436) Updating files: 95% (10865/11436) Updating files: 95% (10867/11436) Updating files: 95% (10959/11436) Updating files: 96% (10979/11436) Updating files: 96% (10985/11436) Updating files: 97% (11093/11436) Updating files: 97% (11186/11436) Updating files: 97% (11187/11436) Updating files: 98% (11208/11436) Updating files: 98% (11254/11436) Updating files: 99% (11322/11436) Updating files: 100% (11436/11436) Updating files: 100% (11436/11436), done. 2022-06-27T11:57:38.104 INFO:tasks.workunit.client.0.smithi195.stderr:+ trap - EXIT 2022-06-27T11:57:38.105 INFO:tasks.workunit.client.0.smithi195.stderr:+ cd ceph 2022-06-27T11:57:38.105 INFO:tasks.workunit.client.0.smithi195.stderr:++ seq 1 90 2022-06-27T11:57:38.106 INFO:tasks.workunit.client.0.smithi195.stderr:+ versions='1 2022-06-27T11:57:38.106 INFO:tasks.workunit.client.0.smithi195.stderr:2 2022-06-27T11:57:38.107 INFO:tasks.workunit.client.0.smithi195.stderr:3 2022-06-27T11:57:38.107 INFO:tasks.workunit.client.0.smithi195.stderr:4 2022-06-27T11:57:38.108 INFO:tasks.workunit.client.0.smithi195.stderr:5 2022-06-27T11:57:38.108 INFO:tasks.workunit.client.0.smithi195.stderr:6 2022-06-27T11:57:38.108 INFO:tasks.workunit.client.0.smithi195.stderr:7 2022-06-27T11:57:38.109 INFO:tasks.workunit.client.0.smithi195.stderr:8 2022-06-27T11:57:38.109 INFO:tasks.workunit.client.0.smithi195.stderr:9 2022-06-27T11:57:38.113 INFO:tasks.workunit.client.0.smithi195.stderr:10 2022-06-27T11:57:38.113 INFO:tasks.workunit.client.0.smithi195.stderr:11 2022-06-27T11:57:38.114 INFO:tasks.workunit.client.0.smithi195.stderr:12 2022-06-27T11:57:38.115 INFO:tasks.workunit.client.0.smithi195.stderr:13 2022-06-27T11:57:38.115 INFO:tasks.workunit.client.0.smithi195.stderr:14 2022-06-27T11:57:38.115 INFO:tasks.workunit.client.0.smithi195.stderr:15 2022-06-27T11:57:38.116 INFO:tasks.workunit.client.0.smithi195.stderr:16 2022-06-27T11:57:38.116 INFO:tasks.workunit.client.0.smithi195.stderr:17 2022-06-27T11:57:38.117 INFO:tasks.workunit.client.0.smithi195.stderr:18 2022-06-27T11:57:38.117 INFO:tasks.workunit.client.0.smithi195.stderr:19 2022-06-27T11:57:38.117 INFO:tasks.workunit.client.0.smithi195.stderr:20 2022-06-27T11:57:38.118 INFO:tasks.workunit.client.0.smithi195.stderr:21 2022-06-27T11:57:38.119 INFO:tasks.workunit.client.0.smithi195.stderr:22 2022-06-27T11:57:38.119 INFO:tasks.workunit.client.0.smithi195.stderr:23 2022-06-27T11:57:38.119 INFO:tasks.workunit.client.0.smithi195.stderr:24 2022-06-27T11:57:38.120 INFO:tasks.workunit.client.0.smithi195.stderr:25 2022-06-27T11:57:38.120 INFO:tasks.workunit.client.0.smithi195.stderr:26 2022-06-27T11:57:38.121 INFO:tasks.workunit.client.0.smithi195.stderr:27 2022-06-27T11:57:38.122 INFO:tasks.workunit.client.0.smithi195.stderr:28 2022-06-27T11:57:38.122 INFO:tasks.workunit.client.0.smithi195.stderr:29 2022-06-27T11:57:38.122 INFO:tasks.workunit.client.0.smithi195.stderr:30 2022-06-27T11:57:38.130 INFO:tasks.workunit.client.0.smithi195.stderr:31 2022-06-27T11:57:38.130 INFO:tasks.workunit.client.0.smithi195.stderr:32 2022-06-27T11:57:38.131 INFO:tasks.workunit.client.0.smithi195.stderr:33 2022-06-27T11:57:38.131 INFO:tasks.workunit.client.0.smithi195.stderr:34 2022-06-27T11:57:38.131 INFO:tasks.workunit.client.0.smithi195.stderr:35 2022-06-27T11:57:38.131 INFO:tasks.workunit.client.0.smithi195.stderr:36 2022-06-27T11:57:38.132 INFO:tasks.workunit.client.0.smithi195.stderr:37 2022-06-27T11:57:38.132 INFO:tasks.workunit.client.0.smithi195.stderr:38 2022-06-27T11:57:38.132 INFO:tasks.workunit.client.0.smithi195.stderr:39 2022-06-27T11:57:38.132 INFO:tasks.workunit.client.0.smithi195.stderr:40 2022-06-27T11:57:38.132 INFO:tasks.workunit.client.0.smithi195.stderr:41 2022-06-27T11:57:38.133 INFO:tasks.workunit.client.0.smithi195.stderr:42 2022-06-27T11:57:38.133 INFO:tasks.workunit.client.0.smithi195.stderr:43 2022-06-27T11:57:38.135 INFO:tasks.workunit.client.0.smithi195.stderr:44 2022-06-27T11:57:38.136 INFO:tasks.workunit.client.0.smithi195.stderr:45 2022-06-27T11:57:38.136 INFO:tasks.workunit.client.0.smithi195.stderr:46 2022-06-27T11:57:38.137 INFO:tasks.workunit.client.0.smithi195.stderr:47 2022-06-27T11:57:38.137 INFO:tasks.workunit.client.0.smithi195.stderr:48 2022-06-27T11:57:38.138 INFO:tasks.workunit.client.0.smithi195.stderr:49 2022-06-27T11:57:38.138 INFO:tasks.workunit.client.0.smithi195.stderr:50 2022-06-27T11:57:38.138 INFO:tasks.workunit.client.0.smithi195.stderr:51 2022-06-27T11:57:38.139 INFO:tasks.workunit.client.0.smithi195.stderr:52 2022-06-27T11:57:38.139 INFO:tasks.workunit.client.0.smithi195.stderr:53 2022-06-27T11:57:38.139 INFO:tasks.workunit.client.0.smithi195.stderr:54 2022-06-27T11:57:38.140 INFO:tasks.workunit.client.0.smithi195.stderr:55 2022-06-27T11:57:38.140 INFO:tasks.workunit.client.0.smithi195.stderr:56 2022-06-27T11:57:38.140 INFO:tasks.workunit.client.0.smithi195.stderr:57 2022-06-27T11:57:38.141 INFO:tasks.workunit.client.0.smithi195.stderr:58 2022-06-27T11:57:38.142 INFO:tasks.workunit.client.0.smithi195.stderr:59 2022-06-27T11:57:38.142 INFO:tasks.workunit.client.0.smithi195.stderr:60 2022-06-27T11:57:38.142 INFO:tasks.workunit.client.0.smithi195.stderr:61 2022-06-27T11:57:38.143 INFO:tasks.workunit.client.0.smithi195.stderr:62 2022-06-27T11:57:38.146 INFO:tasks.workunit.client.0.smithi195.stderr:63 2022-06-27T11:57:38.146 INFO:tasks.workunit.client.0.smithi195.stderr:64 2022-06-27T11:57:38.147 INFO:tasks.workunit.client.0.smithi195.stderr:65 2022-06-27T11:57:38.147 INFO:tasks.workunit.client.0.smithi195.stderr:66 2022-06-27T11:57:38.148 INFO:tasks.workunit.client.0.smithi195.stderr:67 2022-06-27T11:57:38.148 INFO:tasks.workunit.client.0.smithi195.stderr:68 2022-06-27T11:57:38.148 INFO:tasks.workunit.client.0.smithi195.stderr:69 2022-06-27T11:57:38.149 INFO:tasks.workunit.client.0.smithi195.stderr:70 2022-06-27T11:57:38.149 INFO:tasks.workunit.client.0.smithi195.stderr:71 2022-06-27T11:57:38.149 INFO:tasks.workunit.client.0.smithi195.stderr:72 2022-06-27T11:57:38.150 INFO:tasks.workunit.client.0.smithi195.stderr:73 2022-06-27T11:57:38.150 INFO:tasks.workunit.client.0.smithi195.stderr:74 2022-06-27T11:57:38.151 INFO:tasks.workunit.client.0.smithi195.stderr:75 2022-06-27T11:57:38.151 INFO:tasks.workunit.client.0.smithi195.stderr:76 2022-06-27T11:57:38.151 INFO:tasks.workunit.client.0.smithi195.stderr:77 2022-06-27T11:57:38.152 INFO:tasks.workunit.client.0.smithi195.stderr:78 2022-06-27T11:57:38.152 INFO:tasks.workunit.client.0.smithi195.stderr:79 2022-06-27T11:57:38.153 INFO:tasks.workunit.client.0.smithi195.stderr:80 2022-06-27T11:57:38.153 INFO:tasks.workunit.client.0.smithi195.stderr:81 2022-06-27T11:57:38.154 INFO:tasks.workunit.client.0.smithi195.stderr:82 2022-06-27T11:57:38.154 INFO:tasks.workunit.client.0.smithi195.stderr:83 2022-06-27T11:57:38.154 INFO:tasks.workunit.client.0.smithi195.stderr:84 2022-06-27T11:57:38.155 INFO:tasks.workunit.client.0.smithi195.stderr:85 2022-06-27T11:57:38.155 INFO:tasks.workunit.client.0.smithi195.stderr:86 2022-06-27T11:57:38.156 INFO:tasks.workunit.client.0.smithi195.stderr:87 2022-06-27T11:57:38.156 INFO:tasks.workunit.client.0.smithi195.stderr:88 2022-06-27T11:57:38.157 INFO:tasks.workunit.client.0.smithi195.stderr:89 2022-06-27T11:57:38.157 INFO:tasks.workunit.client.0.smithi195.stderr:90' 2022-06-27T11:57:38.157 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T11:57:38.158 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 1 -eq 48 ']' 2022-06-27T11:57:38.158 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.1 2022-06-27T11:57:38.159 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.1 2022-06-27T11:57:38.160 INFO:tasks.workunit.client.0.smithi195.stdout:v0.1 2022-06-27T11:57:38.160 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.1 2022-06-27T11:57:38.582 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:57:38.714 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.714+0000 7f5811d1d700 1 Processor -- start 2022-06-27T11:57:38.715 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.714+0000 7f5811d1d700 1 -- start start 2022-06-27T11:57:38.729 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.715+0000 7f5811d1d700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f580c06f640 0x7f580c156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:38.729 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.715+0000 7f5811d1d700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f580c0c4160 0x7f580c0c1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:38.730 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.715+0000 7f5811d1d700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f580c0c1a20 con 0x7f580c0c39e0 2022-06-27T11:57:38.730 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.715+0000 7f5811d1d700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f580c0c1130 con 0x7f580c06f640 2022-06-27T11:57:38.731 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.715+0000 7f5811d1d700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f580c06ea90 con 0x7f580c0c4160 2022-06-27T11:57:38.731 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.715+0000 7f580bfff700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f580c0c39e0 0x7f580c1535a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:51518/0 (socket says 172.21.15.195:51518) 2022-06-27T11:57:38.741 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.715+0000 7f580bfff700 1 -- 172.21.15.195:0/1842244808 learned_addr learned my addr 172.21.15.195:0/1842244808 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:57:38.741 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.715+0000 7f580b7fe700 1 --2- 172.21.15.195:0/1842244808 >> v2:172.21.15.202:3300/0 conn(0x7f580c06f640 0x7f580c156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:38.742 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.715+0000 7f580affd700 1 --2- 172.21.15.195:0/1842244808 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f580c0c4160 0x7f580c0c1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:38.742 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.716+0000 7f580a7fc700 1 -- 172.21.15.195:0/1842244808 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2055513238 0 0) 0x7f580c0c1a20 con 0x7f580c0c39e0 2022-06-27T11:57:38.743 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.716+0000 7f580a7fc700 1 -- 172.21.15.195:0/1842244808 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57f8003430 con 0x7f580c0c39e0 2022-06-27T11:57:38.743 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.716+0000 7f580a7fc700 1 -- 172.21.15.195:0/1842244808 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2332192664 0 0) 0x7f57f8003430 con 0x7f580c0c39e0 2022-06-27T11:57:38.743 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.716+0000 7f580a7fc700 1 -- 172.21.15.195:0/1842244808 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f580c0c4160 msgr2=0x7f580c0c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:57:38.744 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.716+0000 7f580a7fc700 1 --2- 172.21.15.195:0/1842244808 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f580c0c4160 0x7f580c0c1eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:38.745 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.716+0000 7f580a7fc700 1 -- 172.21.15.195:0/1842244808 >> v2:172.21.15.202:3300/0 conn(0x7f580c06f640 msgr2=0x7f580c156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:57:38.745 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.716+0000 7f580a7fc700 1 --2- 172.21.15.195:0/1842244808 >> v2:172.21.15.202:3300/0 conn(0x7f580c06f640 0x7f580c156ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:38.745 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.716+0000 7f580a7fc700 1 -- 172.21.15.195:0/1842244808 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f580c06ed70 con 0x7f580c0c39e0 2022-06-27T11:57:38.746 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.716+0000 7f580a7fc700 1 -- 172.21.15.195:0/1842244808 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7f58040017f0 con 0x7f580c0c39e0 2022-06-27T11:57:38.761 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.716+0000 7f580a7fc700 1 -- 172.21.15.195:0/1842244808 >> v1:172.21.15.195:6789/0 conn(0x7f580c0c39e0 legacy=0x7f580c1535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:38.762 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.716+0000 7f580a7fc700 1 --2- 172.21.15.195:0/1842244808 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f57f8003920 0x7f57f8003d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:38.762 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.716+0000 7f580a7fc700 1 --2- 172.21.15.195:0/1842244808 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f57f80046d0 0x7f57f8004b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:38.763 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.716+0000 7f580a7fc700 1 --2- 172.21.15.195:0/1842244808 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f580c0c4160 0x7f57f8005080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:38.763 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.716+0000 7f580a7fc700 1 -- 172.21.15.195:0/1842244808 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f580c0c1130 con 0x7f57f8003920 2022-06-27T11:57:38.764 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.716+0000 7f580a7fc700 1 -- 172.21.15.195:0/1842244808 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f580c06ea90 con 0x7f57f80046d0 2022-06-27T11:57:38.764 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.716+0000 7f580a7fc700 1 -- 172.21.15.195:0/1842244808 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57f800b920 con 0x7f580c0c4160 2022-06-27T11:57:38.764 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.716+0000 7f580b7fe700 1 --2- 172.21.15.195:0/1842244808 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f57f8003920 0x7f57f8003d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:38.764 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.716+0000 7f580affd700 1 --2- 172.21.15.195:0/1842244808 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f57f80046d0 0x7f57f8004b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:38.765 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.716+0000 7f580bfff700 1 --2- 172.21.15.195:0/1842244808 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f580c0c4160 0x7f57f8005080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:38.765 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.716+0000 7f580b7fe700 1 -- 172.21.15.195:0/1842244808 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f580c0c4160 msgr2=0x7f57f8005080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:57:38.765 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.716+0000 7f580b7fe700 1 --2- 172.21.15.195:0/1842244808 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f580c0c4160 0x7f57f8005080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:38.765 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.716+0000 7f580b7fe700 1 -- 172.21.15.195:0/1842244808 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f57f80046d0 msgr2=0x7f57f8004b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:57:38.765 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.716+0000 7f580b7fe700 1 --2- 172.21.15.195:0/1842244808 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f57f80046d0 0x7f57f8004b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:38.766 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.716+0000 7f580b7fe700 1 -- 172.21.15.195:0/1842244808 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57f8003430 con 0x7f57f8003920 2022-06-27T11:57:38.766 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.717+0000 7f580b7fe700 1 --2- 172.21.15.195:0/1842244808 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f57f8003920 0x7f57f8003d30 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f57fc009af0 tx=0x7f57fc009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=753ffbf9a273506 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:57:38.767 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.717+0000 7f580a7fc700 1 -- 172.21.15.195:0/1842244808 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57fc009470 con 0x7f57f8003920 2022-06-27T11:57:38.768 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.717+0000 7f580a7fc700 1 -- 172.21.15.195:0/1842244808 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57fc00e640 con 0x7f57f8003920 2022-06-27T11:57:38.768 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.717+0000 7f580a7fc700 1 -- 172.21.15.195:0/1842244808 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57fc00e9e0 con 0x7f57f8003920 2022-06-27T11:57:38.768 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.717+0000 7f5811d1d700 1 -- 172.21.15.195:0/1842244808 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f57f8003920 msgr2=0x7f57f8003d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:38.769 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.717+0000 7f5811d1d700 1 --2- 172.21.15.195:0/1842244808 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f57f8003920 0x7f57f8003d30 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f57fc009af0 tx=0x7f57fc009060 comp rx=0 tx=0).stop 2022-06-27T11:57:38.769 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.717+0000 7f580bfff700 1 -- 172.21.15.195:0/1842244808 reap_dead start 2022-06-27T11:57:38.769 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.717+0000 7f5811d1d700 1 -- 172.21.15.195:0/1842244808 shutdown_connections 2022-06-27T11:57:38.770 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.717+0000 7f5811d1d700 1 -- 172.21.15.195:0/1842244808 >> 172.21.15.195:0/1842244808 conn(0x7f580c0c6350 msgr2=0x7f580c0c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:57:38.770 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.717+0000 7f5811d1d700 1 -- 172.21.15.195:0/1842244808 shutdown_connections 2022-06-27T11:57:38.771 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.717+0000 7f5811d1d700 1 -- 172.21.15.195:0/1842244808 wait complete. 2022-06-27T11:57:38.771 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.718+0000 7f5811d1d700 1 Processor -- start 2022-06-27T11:57:38.772 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.719+0000 7f5811d1d700 1 -- start start 2022-06-27T11:57:38.772 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.719+0000 7f5811d1d700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f57f8003920 0x7f580c067ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:38.772 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.719+0000 7f5811d1d700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f580c06a750 0x7f580c14de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:38.773 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.719+0000 7f5811d1d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f580c14e3d0 0x7f580c166160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:38.773 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.719+0000 7f5811d1d700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f580c06b400 con 0x7f580c14e3d0 2022-06-27T11:57:38.773 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.719+0000 7f5811d1d700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f580c14e840 con 0x7f580c06a750 2022-06-27T11:57:38.773 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.719+0000 7f5811d1d700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f580c06a0c0 con 0x7f57f8003920 2022-06-27T11:57:38.774 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.719+0000 7f580bfff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f580c14e3d0 0x7f580c166160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:38.774 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.719+0000 7f580bfff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f580c14e3d0 0x7f580c166160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45342/0 (socket says 172.21.15.195:45342) 2022-06-27T11:57:38.774 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.719+0000 7f580bfff700 1 -- 172.21.15.195:0/2956842954 learned_addr learned my addr 172.21.15.195:0/2956842954 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:57:38.774 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.719+0000 7f580b7fe700 1 --2- 172.21.15.195:0/2956842954 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f57f8003920 0x7f580c067ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:38.775 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.719+0000 7f580affd700 1 --2- 172.21.15.195:0/2956842954 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f580c06a750 0x7f580c14de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:38.775 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.720+0000 7f580bfff700 1 -- 172.21.15.195:0/2956842954 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f57f8003920 msgr2=0x7f580c067ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:38.775 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.720+0000 7f580bfff700 1 --2- 172.21.15.195:0/2956842954 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f57f8003920 0x7f580c067ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:38.776 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.720+0000 7f580bfff700 1 -- 172.21.15.195:0/2956842954 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f580c06a750 msgr2=0x7f580c14de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:38.776 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.720+0000 7f580bfff700 1 --2- 172.21.15.195:0/2956842954 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f580c06a750 0x7f580c14de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:38.776 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.720+0000 7f580bfff700 1 -- 172.21.15.195:0/2956842954 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f580c06a3a0 con 0x7f580c14e3d0 2022-06-27T11:57:38.776 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.720+0000 7f580bfff700 1 --2- 172.21.15.195:0/2956842954 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f580c14e3d0 0x7f580c166160 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f580c06dd70 tx=0x7f5804001d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:57:38.777 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.720+0000 7f57f37fe700 1 -- 172.21.15.195:0/2956842954 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5804018030 con 0x7f580c14e3d0 2022-06-27T11:57:38.777 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.720+0000 7f57f37fe700 1 -- 172.21.15.195:0/2956842954 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f580400fb20 con 0x7f580c14e3d0 2022-06-27T11:57:38.777 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.720+0000 7f57f37fe700 1 -- 172.21.15.195:0/2956842954 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f580400fec0 con 0x7f580c14e3d0 2022-06-27T11:57:38.777 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.720+0000 7f5811d1d700 1 -- 172.21.15.195:0/2956842954 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f580c15b6a0 con 0x7f580c14e3d0 2022-06-27T11:57:38.778 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.720+0000 7f5811d1d700 1 -- 172.21.15.195:0/2956842954 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f580c15bc60 con 0x7f580c14e3d0 2022-06-27T11:57:38.778 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.721+0000 7f5811d1d700 1 -- 172.21.15.195:0/2956842954 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f580c0722c0 con 0x7f580c14e3d0 2022-06-27T11:57:38.778 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.721+0000 7f57f37fe700 1 -- 172.21.15.195:0/2956842954 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f5804013600 con 0x7f580c14e3d0 2022-06-27T11:57:38.778 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.722+0000 7f57f37fe700 1 --2- 172.21.15.195:0/2956842954 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f57ec063d00 0x7f57ec0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:38.779 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.722+0000 7f57f37fe700 1 -- 172.21.15.195:0/2956842954 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6546+0+0 (secure 0 0 0) 0x7f5804019070 con 0x7f580c14e3d0 2022-06-27T11:57:38.779 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.722+0000 7f580b7fe700 1 --2- 172.21.15.195:0/2956842954 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f57ec063d00 0x7f57ec0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:38.779 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.722+0000 7f580b7fe700 1 --2- 172.21.15.195:0/2956842954 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f57ec063d00 0x7f57ec0661b0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f57fc006d90 tx=0x7f57fc01a000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:57:38.779 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.726+0000 7f57f37fe700 1 -- 172.21.15.195:0/2956842954 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f580404f7d0 con 0x7f580c14e3d0 2022-06-27T11:57:38.959 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.952+0000 7f5811d1d700 1 -- 172.21.15.195:0/2956842954 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f580c15c240 con 0x7f580c14e3d0 2022-06-27T11:57:38.959 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.952+0000 7f57f37fe700 1 -- 172.21.15.195:0/2956842954 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+6128 (secure 0 0 0) 0x7f58040017b0 con 0x7f580c14e3d0 2022-06-27T11:57:38.960 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:57:38.960 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":5392,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/2089155519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2089155519},{"type":"v1","addr":"172.21.15.202:6835","nonce":2089155519}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:57:31.004761+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4976},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4976":{"gid":4976,"name":"b","rank":0,"incarnation":32,"state":"up:active","state_seq":28,"addr":"172.21.15.202:6833/4154570751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":4154570751},{"type":"v1","addr":"172.21.15.202:6833","nonce":4154570751}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:57:38.963 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.955+0000 7f5811d1d700 1 -- 172.21.15.195:0/2956842954 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f57ec063d00 msgr2=0x7f57ec0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:38.963 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.955+0000 7f5811d1d700 1 --2- 172.21.15.195:0/2956842954 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f57ec063d00 0x7f57ec0661b0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f57fc006d90 tx=0x7f57fc01a000 comp rx=0 tx=0).stop 2022-06-27T11:57:38.963 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.955+0000 7f5811d1d700 1 -- 172.21.15.195:0/2956842954 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f580c14e3d0 msgr2=0x7f580c166160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:38.964 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.955+0000 7f5811d1d700 1 --2- 172.21.15.195:0/2956842954 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f580c14e3d0 0x7f580c166160 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f580c06dd70 tx=0x7f5804001d90 comp rx=0 tx=0).stop 2022-06-27T11:57:38.964 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.955+0000 7f5811d1d700 1 -- 172.21.15.195:0/2956842954 shutdown_connections 2022-06-27T11:57:38.964 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.955+0000 7f5811d1d700 1 --2- 172.21.15.195:0/2956842954 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f57ec063d00 0x7f57ec0661b0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:38.964 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.955+0000 7f5811d1d700 1 --2- 172.21.15.195:0/2956842954 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f580c14e3d0 0x7f580c166160 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:38.965 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.955+0000 7f5811d1d700 1 --2- 172.21.15.195:0/2956842954 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f57f8003920 0x7f580c067ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:38.965 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.955+0000 7f5811d1d700 1 --2- 172.21.15.195:0/2956842954 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f580c06a750 0x7f580c14de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:38.965 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.955+0000 7f5811d1d700 1 -- 172.21.15.195:0/2956842954 >> 172.21.15.195:0/2956842954 conn(0x7f580c0c6350 msgr2=0x7f580c1504d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:57:38.965 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.955+0000 7f5811d1d700 1 -- 172.21.15.195:0/2956842954 shutdown_connections 2022-06-27T11:57:38.966 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:38.955+0000 7f5811d1d700 1 -- 172.21.15.195:0/2956842954 wait complete. 2022-06-27T11:57:38.966 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 37 2022-06-27T11:57:38.967 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d reported in up:standby state 2022-06-27T11:57:38.967 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 11 secs before thrashing 2022-06-27T11:57:49.969 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:57:50.103 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.103+0000 7fd4a9da5700 1 Processor -- start 2022-06-27T11:57:50.104 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.104+0000 7fd4a9da5700 1 -- start start 2022-06-27T11:57:50.125 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.104+0000 7fd4a9da5700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd4a406f640 0x7fd4a4156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:50.126 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.104+0000 7fd4a9da5700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd4a40c4160 0x7fd4a40c1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:50.126 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.104+0000 7fd4a9da5700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fd4a40c39e0 0x7fd4a41535a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:50.127 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.104+0000 7fd4a9da5700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4a40c1a20 con 0x7fd4a406f640 2022-06-27T11:57:50.127 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.104+0000 7fd4a9da5700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4a40c1130 con 0x7fd4a40c39e0 2022-06-27T11:57:50.128 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.104+0000 7fd4a9da5700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a406ea90 con 0x7fd4a40c4160 2022-06-27T11:57:50.128 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.105+0000 7fd4a37fe700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd4a406f640 0x7fd4a4156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:50.130 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.105+0000 7fd4a37fe700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd4a406f640 0x7fd4a4156ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45370/0 (socket says 172.21.15.195:45370) 2022-06-27T11:57:50.130 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.105+0000 7fd4a37fe700 1 -- 172.21.15.195:0/521511217 learned_addr learned my addr 172.21.15.195:0/521511217 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:57:50.131 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.105+0000 7fd4a2ffd700 1 --2- 172.21.15.195:0/521511217 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd4a40c4160 0x7fd4a40c1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:50.132 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.105+0000 7fd4a3fff700 1 --2- 172.21.15.195:0/521511217 >> v2:172.21.15.202:3300/0 conn(0x7fd4a40c39e0 0x7fd4a41535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:50.133 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.105+0000 7fd4a37fe700 1 -- 172.21.15.195:0/521511217 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd4a40c4160 msgr2=0x7fd4a40c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:57:50.133 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.105+0000 7fd4a37fe700 1 --2- 172.21.15.195:0/521511217 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd4a40c4160 0x7fd4a40c1eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:50.134 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.105+0000 7fd4a37fe700 1 -- 172.21.15.195:0/521511217 >> v2:172.21.15.202:3300/0 conn(0x7fd4a40c39e0 msgr2=0x7fd4a41535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:57:50.134 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.105+0000 7fd4a37fe700 1 --2- 172.21.15.195:0/521511217 >> v2:172.21.15.202:3300/0 conn(0x7fd4a40c39e0 0x7fd4a41535a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:50.135 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.105+0000 7fd4a37fe700 1 -- 172.21.15.195:0/521511217 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4a406ed70 con 0x7fd4a406f640 2022-06-27T11:57:50.135 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.105+0000 7fd4a37fe700 1 --2- 172.21.15.195:0/521511217 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd4a406f640 0x7fd4a4156ed0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fd4a406b530 tx=0x7fd494009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c3321e1311f42cc server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:57:50.136 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.105+0000 7fd4a27fc700 1 -- 172.21.15.195:0/521511217 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd49400b6d0 con 0x7fd4a406f640 2022-06-27T11:57:50.136 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.105+0000 7fd4a27fc700 1 -- 172.21.15.195:0/521511217 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd49400b830 con 0x7fd4a406f640 2022-06-27T11:57:50.136 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.105+0000 7fd4a27fc700 1 -- 172.21.15.195:0/521511217 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd49400bbd0 con 0x7fd4a406f640 2022-06-27T11:57:50.137 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.106+0000 7fd4a9da5700 1 -- 172.21.15.195:0/521511217 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd4a406f640 msgr2=0x7fd4a4156ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:50.137 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.106+0000 7fd4a9da5700 1 --2- 172.21.15.195:0/521511217 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd4a406f640 0x7fd4a4156ed0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fd4a406b530 tx=0x7fd494009120 comp rx=0 tx=0).stop 2022-06-27T11:57:50.138 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.106+0000 7fd4a9da5700 1 -- 172.21.15.195:0/521511217 shutdown_connections 2022-06-27T11:57:50.138 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.106+0000 7fd4a9da5700 1 --2- 172.21.15.195:0/521511217 >> v2:172.21.15.202:3300/0 conn(0x7fd4a40c39e0 0x7fd4a41535a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:50.140 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.106+0000 7fd4a9da5700 1 --2- 172.21.15.195:0/521511217 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd4a406f640 0x7fd4a4156ed0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:50.140 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.106+0000 7fd4a9da5700 1 --2- 172.21.15.195:0/521511217 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd4a40c4160 0x7fd4a40c1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:50.141 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.106+0000 7fd4a9da5700 1 -- 172.21.15.195:0/521511217 >> 172.21.15.195:0/521511217 conn(0x7fd4a40c6350 msgr2=0x7fd4a40c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:57:50.141 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.106+0000 7fd4a9da5700 1 -- 172.21.15.195:0/521511217 shutdown_connections 2022-06-27T11:57:50.141 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.106+0000 7fd4a9da5700 1 -- 172.21.15.195:0/521511217 wait complete. 2022-06-27T11:57:50.141 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.107+0000 7fd4a9da5700 1 Processor -- start 2022-06-27T11:57:50.142 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.107+0000 7fd4a9da5700 1 -- start start 2022-06-27T11:57:50.142 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.108+0000 7fd4a9da5700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd4a406f640 0x7fd4a4166180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:50.142 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.108+0000 7fd4a9da5700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd4a40c39e0 0x7fd4a414e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:50.142 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.108+0000 7fd4a9da5700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd4a40c4160 0x7fd4a4154120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:50.143 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.108+0000 7fd4a9da5700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a40bd300 con 0x7fd4a40c4160 2022-06-27T11:57:50.143 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.108+0000 7fd4a9da5700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a4152bf0 con 0x7fd4a40c39e0 2022-06-27T11:57:50.143 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.108+0000 7fd4a9da5700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a40bb260 con 0x7fd4a406f640 2022-06-27T11:57:50.143 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.108+0000 7fd4a3fff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd4a40c4160 0x7fd4a4154120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:50.144 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.108+0000 7fd4a3fff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd4a40c4160 0x7fd4a4154120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45380/0 (socket says 172.21.15.195:45380) 2022-06-27T11:57:50.144 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.108+0000 7fd4a2ffd700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd4a40c39e0 0x7fd4a414e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:50.144 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.108+0000 7fd4a37fe700 1 --2- 172.21.15.195:0/3518433446 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd4a406f640 0x7fd4a4166180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:50.144 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.108+0000 7fd4a3fff700 1 -- 172.21.15.195:0/3518433446 learned_addr learned my addr 172.21.15.195:0/3518433446 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:57:50.145 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.108+0000 7fd4a3fff700 1 -- 172.21.15.195:0/3518433446 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd4a406f640 msgr2=0x7fd4a4166180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:50.145 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.108+0000 7fd4a3fff700 1 --2- 172.21.15.195:0/3518433446 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd4a406f640 0x7fd4a4166180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:50.145 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.108+0000 7fd4a3fff700 1 -- 172.21.15.195:0/3518433446 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd4a40c39e0 msgr2=0x7fd4a414e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:50.146 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.108+0000 7fd4a3fff700 1 --2- 172.21.15.195:0/3518433446 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd4a40c39e0 0x7fd4a414e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:50.146 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.108+0000 7fd4a3fff700 1 -- 172.21.15.195:0/3518433446 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4940093a0 con 0x7fd4a40c4160 2022-06-27T11:57:50.147 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.108+0000 7fd4a3fff700 1 --2- 172.21.15.195:0/3518433446 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd4a40c4160 0x7fd4a4154120 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fd49c009fd0 tx=0x7fd49c009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:57:50.147 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.109+0000 7fd49b7fe700 1 -- 172.21.15.195:0/3518433446 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd49c00eca0 con 0x7fd4a40c4160 2022-06-27T11:57:50.148 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.109+0000 7fd49b7fe700 1 -- 172.21.15.195:0/3518433446 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd49c00ee00 con 0x7fd4a40c4160 2022-06-27T11:57:50.148 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.109+0000 7fd49b7fe700 1 -- 172.21.15.195:0/3518433446 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd49c014670 con 0x7fd4a40c4160 2022-06-27T11:57:50.149 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.109+0000 7fd4a9da5700 1 -- 172.21.15.195:0/3518433446 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4a40bca60 con 0x7fd4a40c4160 2022-06-27T11:57:50.149 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.109+0000 7fd4a9da5700 1 -- 172.21.15.195:0/3518433446 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4a40bd020 con 0x7fd4a40c4160 2022-06-27T11:57:50.149 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.110+0000 7fd4a9da5700 1 -- 172.21.15.195:0/3518433446 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4a40bb0c0 con 0x7fd4a40c4160 2022-06-27T11:57:50.150 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.110+0000 7fd49b7fe700 1 -- 172.21.15.195:0/3518433446 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fd49c012050 con 0x7fd4a40c4160 2022-06-27T11:57:50.151 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.110+0000 7fd49b7fe700 1 --2- 172.21.15.195:0/3518433446 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd488063d30 0x7fd4880661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:50.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.110+0000 7fd49b7fe700 1 -- 172.21.15.195:0/3518433446 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6546+0+0 (secure 0 0 0) 0x7fd49c080ff0 con 0x7fd4a40c4160 2022-06-27T11:57:50.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.110+0000 7fd4a37fe700 1 --2- 172.21.15.195:0/3518433446 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd488063d30 0x7fd4880661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:50.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.111+0000 7fd4a37fe700 1 --2- 172.21.15.195:0/3518433446 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd488063d30 0x7fd4880661e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fd494009160 tx=0x7fd494017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:57:50.155 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.115+0000 7fd49b7fe700 1 -- 172.21.15.195:0/3518433446 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fd4a40bb0c0 con 0x7fd4a40c4160 2022-06-27T11:57:50.360 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.358+0000 7fd4a9da5700 1 -- 172.21.15.195:0/3518433446 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd4a40bb0c0 con 0x7fd4a40c4160 2022-06-27T11:57:50.366 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.359+0000 7fd49b7fe700 1 -- 172.21.15.195:0/3518433446 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+6128 (secure 0 0 0) 0x7fd4a40bb0c0 con 0x7fd4a40c4160 2022-06-27T11:57:50.367 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:57:50.376 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":5392,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/2089155519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2089155519},{"type":"v1","addr":"172.21.15.202:6835","nonce":2089155519}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:57:31.004761+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4976},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4976":{"gid":4976,"name":"b","rank":0,"incarnation":32,"state":"up:active","state_seq":28,"addr":"172.21.15.202:6833/4154570751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":4154570751},{"type":"v1","addr":"172.21.15.202:6833","nonce":4154570751}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:57:50.378 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.361+0000 7fd4a9da5700 1 -- 172.21.15.195:0/3518433446 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd488063d30 msgr2=0x7fd4880661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:50.378 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.361+0000 7fd4a9da5700 1 --2- 172.21.15.195:0/3518433446 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd488063d30 0x7fd4880661e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fd494009160 tx=0x7fd494017000 comp rx=0 tx=0).stop 2022-06-27T11:57:50.379 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.361+0000 7fd4a9da5700 1 -- 172.21.15.195:0/3518433446 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd4a40c4160 msgr2=0x7fd4a4154120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:50.379 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.361+0000 7fd4a9da5700 1 --2- 172.21.15.195:0/3518433446 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd4a40c4160 0x7fd4a4154120 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fd49c009fd0 tx=0x7fd49c009900 comp rx=0 tx=0).stop 2022-06-27T11:57:50.379 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.361+0000 7fd4a9da5700 1 -- 172.21.15.195:0/3518433446 shutdown_connections 2022-06-27T11:57:50.379 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.361+0000 7fd4a9da5700 1 --2- 172.21.15.195:0/3518433446 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd488063d30 0x7fd4880661e0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:50.380 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.361+0000 7fd4a9da5700 1 --2- 172.21.15.195:0/3518433446 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd4a40c4160 0x7fd4a4154120 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:50.380 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.361+0000 7fd4a9da5700 1 --2- 172.21.15.195:0/3518433446 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd4a406f640 0x7fd4a4166180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:50.380 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.361+0000 7fd4a9da5700 1 --2- 172.21.15.195:0/3518433446 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd4a40c39e0 0x7fd4a414e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:50.381 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.361+0000 7fd4a9da5700 1 -- 172.21.15.195:0/3518433446 >> 172.21.15.195:0/3518433446 conn(0x7fd4a40c6350 msgr2=0x7fd4a4157600 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:57:50.381 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.361+0000 7fd4a9da5700 1 -- 172.21.15.195:0/3518433446 shutdown_connections 2022-06-27T11:57:50.382 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.361+0000 7fd4a9da5700 1 -- 172.21.15.195:0/3518433446 wait complete. 2022-06-27T11:57:50.382 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 37 2022-06-27T11:57:50.383 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2022-06-27T11:57:50.383 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-06-27T11:57:50.384 INFO:teuthology.orchestra.run:waiting for 300 2022-06-27T11:57:50.397 INFO:tasks.ceph.mds.b:Stopped 2022-06-27T11:57:50.398 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:57:50.547 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.533+0000 7f8754ccd700 1 Processor -- start 2022-06-27T11:57:50.548 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.533+0000 7f8754ccd700 1 -- start start 2022-06-27T11:57:50.548 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.534+0000 7f8754ccd700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f8750150580 0x7f8750156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:50.551 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.534+0000 7f8754ccd700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8750150950 0x7f87500c7b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:50.552 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.534+0000 7f8754ccd700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87500c76b0 con 0x7f875006f620 2022-06-27T11:57:50.552 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.534+0000 7f8754ccd700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8750157430 con 0x7f8750150580 2022-06-27T11:57:50.552 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.534+0000 7f8754ccd700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87500bad70 con 0x7f8750150950 2022-06-27T11:57:50.553 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.534+0000 7f874e59c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f8750150580 0x7f8750156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:50.553 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.534+0000 7f874ed9d700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f875006f620 0x7f875006d6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:51576/0 (socket says 172.21.15.195:51576) 2022-06-27T11:57:50.553 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.534+0000 7f874ed9d700 1 -- 172.21.15.195:0/1152247864 learned_addr learned my addr 172.21.15.195:0/1152247864 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:57:50.554 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.534+0000 7f874dd9b700 1 --2- 172.21.15.195:0/1152247864 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8750150950 0x7f87500c7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:50.554 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.534+0000 7f874e59c700 1 -- 172.21.15.195:0/1152247864 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8750150950 msgr2=0x7f87500c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:57:50.555 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.534+0000 7f874e59c700 1 --2- 172.21.15.195:0/1152247864 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8750150950 0x7f87500c7b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:50.555 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.534+0000 7f874e59c700 1 -- 172.21.15.195:0/1152247864 >> v1:172.21.15.195:6789/0 conn(0x7f875006f620 legacy=0x7f875006d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:50.555 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.534+0000 7f874d59a700 1 -- 172.21.15.195:0/1152247864 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1461966189 0 0) 0x7f87500c76b0 con 0x7f875006f620 2022-06-27T11:57:50.555 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.534+0000 7f874e59c700 1 -- 172.21.15.195:0/1152247864 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87500ba4d0 con 0x7f8750150580 2022-06-27T11:57:50.556 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.534+0000 7f874e59c700 1 --2- 172.21.15.195:0/1152247864 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8750150580 0x7f8750156ef0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f87400097a0 tx=0x7f8740009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=120595ef1156f839 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:57:50.556 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.534+0000 7f874d59a700 1 -- 172.21.15.195:0/1152247864 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8740011630 con 0x7f8750150580 2022-06-27T11:57:50.556 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.534+0000 7f874d59a700 1 -- 172.21.15.195:0/1152247864 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8740011790 con 0x7f8750150580 2022-06-27T11:57:50.557 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.534+0000 7f874d59a700 1 -- 172.21.15.195:0/1152247864 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8740011b30 con 0x7f8750150580 2022-06-27T11:57:50.557 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.535+0000 7f8754ccd700 1 -- 172.21.15.195:0/1152247864 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8750150580 msgr2=0x7f8750156ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:50.557 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.535+0000 7f8754ccd700 1 --2- 172.21.15.195:0/1152247864 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8750150580 0x7f8750156ef0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f87400097a0 tx=0x7f8740009060 comp rx=0 tx=0).stop 2022-06-27T11:57:50.557 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.535+0000 7f8754ccd700 1 -- 172.21.15.195:0/1152247864 shutdown_connections 2022-06-27T11:57:50.558 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.535+0000 7f8754ccd700 1 --2- 172.21.15.195:0/1152247864 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8750150580 0x7f8750156ef0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:50.558 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.535+0000 7f8754ccd700 1 --2- 172.21.15.195:0/1152247864 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8750150950 0x7f87500c7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:50.558 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.535+0000 7f8754ccd700 1 -- 172.21.15.195:0/1152247864 >> 172.21.15.195:0/1152247864 conn(0x7f8750063af0 msgr2=0x7f8750063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:57:50.558 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.535+0000 7f8754ccd700 1 -- 172.21.15.195:0/1152247864 shutdown_connections 2022-06-27T11:57:50.559 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.535+0000 7f8754ccd700 1 -- 172.21.15.195:0/1152247864 wait complete. 2022-06-27T11:57:50.559 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.536+0000 7f8754ccd700 1 Processor -- start 2022-06-27T11:57:50.559 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.536+0000 7f8754ccd700 1 -- start start 2022-06-27T11:57:50.559 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.537+0000 7f8754ccd700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f875006f620 0x7f875006a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:50.560 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.537+0000 7f8754ccd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8750150580 0x7f87500c2d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:50.560 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.537+0000 7f8754ccd700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8750150950 0x7f87500c32a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:50.560 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.537+0000 7f8754ccd700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87500cb520 con 0x7f8750150580 2022-06-27T11:57:50.560 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.537+0000 7f8754ccd700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87500ca680 con 0x7f875006f620 2022-06-27T11:57:50.561 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.537+0000 7f8754ccd700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87500ca910 con 0x7f8750150950 2022-06-27T11:57:50.561 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.537+0000 7f874dd9b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8750150580 0x7f87500c2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:50.561 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.537+0000 7f874dd9b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8750150580 0x7f87500c2d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45396/0 (socket says 172.21.15.195:45396) 2022-06-27T11:57:50.561 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.537+0000 7f874dd9b700 1 -- 172.21.15.195:0/32315728 learned_addr learned my addr 172.21.15.195:0/32315728 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:57:50.562 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.537+0000 7f874ed9d700 1 --2- 172.21.15.195:0/32315728 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8750150950 0x7f87500c32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:50.562 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.537+0000 7f874e59c700 1 --2- 172.21.15.195:0/32315728 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f875006f620 0x7f875006a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:50.563 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.537+0000 7f874dd9b700 1 -- 172.21.15.195:0/32315728 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8750150950 msgr2=0x7f87500c32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:50.563 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.537+0000 7f874dd9b700 1 --2- 172.21.15.195:0/32315728 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8750150950 0x7f87500c32a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:50.563 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.537+0000 7f874dd9b700 1 -- 172.21.15.195:0/32315728 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f875006f620 msgr2=0x7f875006a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:50.563 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.537+0000 7f874dd9b700 1 --2- 172.21.15.195:0/32315728 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f875006f620 0x7f875006a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:50.564 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.537+0000 7f874dd9b700 1 -- 172.21.15.195:0/32315728 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87400092e0 con 0x7f8750150580 2022-06-27T11:57:50.564 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.537+0000 7f874dd9b700 1 --2- 172.21.15.195:0/32315728 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8750150580 0x7f87500c2d60 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f874400d860 tx=0x7f8744008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:57:50.564 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.538+0000 7f873e7fc700 1 -- 172.21.15.195:0/32315728 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87440125f0 con 0x7f8750150580 2022-06-27T11:57:50.564 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.538+0000 7f873e7fc700 1 -- 172.21.15.195:0/32315728 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8744012750 con 0x7f8750150580 2022-06-27T11:57:50.565 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.538+0000 7f8754ccd700 1 -- 172.21.15.195:0/32315728 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87500cabf0 con 0x7f8750150580 2022-06-27T11:57:50.565 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.538+0000 7f873e7fc700 1 -- 172.21.15.195:0/32315728 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8744012af0 con 0x7f8750150580 2022-06-27T11:57:50.565 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.538+0000 7f8754ccd700 1 -- 172.21.15.195:0/32315728 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87500cb1b0 con 0x7f8750150580 2022-06-27T11:57:50.565 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.539+0000 7f8754ccd700 1 -- 172.21.15.195:0/32315728 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87500cb370 con 0x7f8750150580 2022-06-27T11:57:50.566 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.539+0000 7f873e7fc700 1 -- 172.21.15.195:0/32315728 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f874401e460 con 0x7f8750150580 2022-06-27T11:57:50.566 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.539+0000 7f873e7fc700 1 --2- 172.21.15.195:0/32315728 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f872c063d00 0x7f872c0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:50.566 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.539+0000 7f873e7fc700 1 -- 172.21.15.195:0/32315728 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6546+0+0 (secure 0 0 0) 0x7f8744080fe0 con 0x7f8750150580 2022-06-27T11:57:50.566 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.539+0000 7f874e59c700 1 --2- 172.21.15.195:0/32315728 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f872c063d00 0x7f872c0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:50.567 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.540+0000 7f874e59c700 1 --2- 172.21.15.195:0/32315728 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f872c063d00 0x7f872c0661b0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f8740009260 tx=0x7f8740017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:57:50.567 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.544+0000 7f873e7fc700 1 -- 172.21.15.195:0/32315728 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f874404f980 con 0x7f8750150580 2022-06-27T11:57:50.773 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.773+0000 7f8754ccd700 1 -- 172.21.15.195:0/32315728 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87500722c0 con 0x7f8750150580 2022-06-27T11:57:50.774 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.774+0000 7f873e7fc700 1 -- 172.21.15.195:0/32315728 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+6128 (secure 0 0 0) 0x7f8744052fa0 con 0x7f8750150580 2022-06-27T11:57:50.775 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:57:50.776 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":5392,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/2089155519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2089155519},{"type":"v1","addr":"172.21.15.202:6835","nonce":2089155519}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:57:31.004761+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4976},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4976":{"gid":4976,"name":"b","rank":0,"incarnation":32,"state":"up:active","state_seq":28,"addr":"172.21.15.202:6833/4154570751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":4154570751},{"type":"v1","addr":"172.21.15.202:6833","nonce":4154570751}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:57:50.779 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.776+0000 7f8754ccd700 1 -- 172.21.15.195:0/32315728 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f872c063d00 msgr2=0x7f872c0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:50.780 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.776+0000 7f8754ccd700 1 --2- 172.21.15.195:0/32315728 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f872c063d00 0x7f872c0661b0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f8740009260 tx=0x7f8740017000 comp rx=0 tx=0).stop 2022-06-27T11:57:50.780 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.776+0000 7f8754ccd700 1 -- 172.21.15.195:0/32315728 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8750150580 msgr2=0x7f87500c2d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:50.780 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.776+0000 7f8754ccd700 1 --2- 172.21.15.195:0/32315728 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8750150580 0x7f87500c2d60 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f874400d860 tx=0x7f8744008040 comp rx=0 tx=0).stop 2022-06-27T11:57:50.781 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.776+0000 7f8754ccd700 1 -- 172.21.15.195:0/32315728 shutdown_connections 2022-06-27T11:57:50.781 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.776+0000 7f8754ccd700 1 --2- 172.21.15.195:0/32315728 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f872c063d00 0x7f872c0661b0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:50.782 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.776+0000 7f8754ccd700 1 --2- 172.21.15.195:0/32315728 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8750150950 0x7f87500c32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:50.782 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.776+0000 7f8754ccd700 1 --2- 172.21.15.195:0/32315728 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f875006f620 0x7f875006a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:50.783 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.776+0000 7f8754ccd700 1 --2- 172.21.15.195:0/32315728 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8750150580 0x7f87500c2d60 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:50.784 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.776+0000 7f8754ccd700 1 -- 172.21.15.195:0/32315728 >> 172.21.15.195:0/32315728 conn(0x7f8750063af0 msgr2=0x7f875014f5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:57:50.784 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.776+0000 7f8754ccd700 1 -- 172.21.15.195:0/32315728 shutdown_connections 2022-06-27T11:57:50.785 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:50.776+0000 7f8754ccd700 1 -- 172.21.15.195:0/32315728 wait complete. 2022-06-27T11:57:50.785 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 37 2022-06-27T11:57:50.807 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2022-06-27T11:57:52.809 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:57:52.943 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:52.943+0000 7f59386cf700 1 Processor -- start 2022-06-27T11:57:52.943 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:52.943+0000 7f59386cf700 1 -- start start 2022-06-27T11:57:52.945 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:52.944+0000 7f59386cf700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f593006a450 0x7f5930063af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:52.946 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:52.944+0000 7f59386cf700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f59301508d0 0x7f59300bbf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:52.946 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:52.944+0000 7f59386cf700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5930064030 con 0x7f5930150500 2022-06-27T11:57:52.946 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:52.944+0000 7f59386cf700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59300c4160 con 0x7f59301508d0 2022-06-27T11:57:52.946 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:52.944+0000 7f59386cf700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59300c4440 con 0x7f593006a450 2022-06-27T11:57:52.947 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:52.945+0000 7f593646b700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f5930150500 0x7f59300c48d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:51600/0 (socket says 172.21.15.195:51600) 2022-06-27T11:57:52.947 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:52.945+0000 7f593646b700 1 -- 172.21.15.195:0/465259533 learned_addr learned my addr 172.21.15.195:0/465259533 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:57:52.947 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:52.945+0000 7f5936c6c700 1 --2- 172.21.15.195:0/465259533 >> v2:172.21.15.195:3300/0 conn(0x7f59301508d0 0x7f59300bbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:52.948 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:52.945+0000 7f5935c6a700 1 --2- 172.21.15.195:0/465259533 >> v2:172.21.15.202:3300/0 conn(0x7f593006a450 0x7f5930063af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:52.948 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:52.945+0000 7f5935469700 1 -- 172.21.15.195:0/465259533 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4081150773 0 0) 0x7f592c001290 con 0x7f5930150500 2022-06-27T11:57:52.948 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:52.945+0000 7f5935469700 1 -- 172.21.15.195:0/465259533 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5918003430 con 0x7f5930150500 2022-06-27T11:57:52.949 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:52.945+0000 7f5936c6c700 1 -- 172.21.15.195:0/465259533 >> v2:172.21.15.202:3300/0 conn(0x7f593006a450 msgr2=0x7f5930063af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:57:52.949 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:52.945+0000 7f5936c6c700 1 --2- 172.21.15.195:0/465259533 >> v2:172.21.15.202:3300/0 conn(0x7f593006a450 0x7f5930063af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:52.949 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:52.945+0000 7f5936c6c700 1 -- 172.21.15.195:0/465259533 >> v1:172.21.15.195:6789/0 conn(0x7f5930150500 legacy=0x7f59300c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:52.949 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:52.945+0000 7f5936c6c700 1 -- 172.21.15.195:0/465259533 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f593015a800 con 0x7f59301508d0 2022-06-27T11:57:52.950 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:52.945+0000 7f5936c6c700 1 --2- 172.21.15.195:0/465259533 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f59301508d0 0x7f59300bbf50 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f593006ce70 tx=0x7f5928009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=321abd5faaa352cc server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:57:52.950 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:52.945+0000 7f5935469700 1 -- 172.21.15.195:0/465259533 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f592800e070 con 0x7f59301508d0 2022-06-27T11:57:52.950 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:52.945+0000 7f5935469700 1 -- 172.21.15.195:0/465259533 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f592800bb60 con 0x7f59301508d0 2022-06-27T11:57:52.950 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:52.945+0000 7f5935469700 1 -- 172.21.15.195:0/465259533 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5928012430 con 0x7f59301508d0 2022-06-27T11:57:52.955 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:52.946+0000 7f59386cf700 1 -- 172.21.15.195:0/465259533 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f59301508d0 msgr2=0x7f59300bbf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:52.956 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:52.946+0000 7f59386cf700 1 --2- 172.21.15.195:0/465259533 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f59301508d0 0x7f59300bbf50 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f593006ce70 tx=0x7f5928009780 comp rx=0 tx=0).stop 2022-06-27T11:57:52.956 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:52.946+0000 7f59386cf700 1 -- 172.21.15.195:0/465259533 shutdown_connections 2022-06-27T11:57:52.956 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:52.946+0000 7f59386cf700 1 --2- 172.21.15.195:0/465259533 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f59301508d0 0x7f59300bbf50 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:52.956 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:52.946+0000 7f59386cf700 1 --2- 172.21.15.195:0/465259533 >> v2:172.21.15.202:3300/0 conn(0x7f593006a450 0x7f5930063af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:52.957 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:52.946+0000 7f59386cf700 1 -- 172.21.15.195:0/465259533 >> 172.21.15.195:0/465259533 conn(0x7f59300c9b00 msgr2=0x7f59300c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:57:52.957 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:52.946+0000 7f59386cf700 1 -- 172.21.15.195:0/465259533 shutdown_connections 2022-06-27T11:57:52.957 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:52.946+0000 7f59386cf700 1 -- 172.21.15.195:0/465259533 wait complete. 2022-06-27T11:57:52.957 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:52.947+0000 7f59386cf700 1 Processor -- start 2022-06-27T11:57:52.957 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:52.947+0000 7f59386cf700 1 -- start start 2022-06-27T11:57:52.958 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:52.948+0000 7f59386cf700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f593006a450 0x7f5930165e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:52.958 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:52.948+0000 7f59386cf700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5930150500 0x7f593014c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:52.958 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:52.948+0000 7f59386cf700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f59301508d0 0x7f593014cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:52.959 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:52.948+0000 7f59386cf700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5930068000 con 0x7f59301508d0 2022-06-27T11:57:52.959 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:52.948+0000 7f59386cf700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59301693b0 con 0x7f593006a450 2022-06-27T11:57:52.960 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:52.948+0000 7f59386cf700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5930169650 con 0x7f5930150500 2022-06-27T11:57:52.960 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:52.948+0000 7f5936c6c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f59301508d0 0x7f593014cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:52.960 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:52.948+0000 7f5936c6c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f59301508d0 0x7f593014cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45422/0 (socket says 172.21.15.195:45422) 2022-06-27T11:57:52.960 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:52.948+0000 7f5936c6c700 1 -- 172.21.15.195:0/1676588755 learned_addr learned my addr 172.21.15.195:0/1676588755 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:57:52.960 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:52.948+0000 7f593646b700 1 --2- 172.21.15.195:0/1676588755 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f593006a450 0x7f5930165e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:52.961 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:52.948+0000 7f5935c6a700 1 --2- 172.21.15.195:0/1676588755 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5930150500 0x7f593014c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:52.961 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:52.948+0000 7f5936c6c700 1 -- 172.21.15.195:0/1676588755 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5930150500 msgr2=0x7f593014c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:52.961 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:52.948+0000 7f5936c6c700 1 --2- 172.21.15.195:0/1676588755 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5930150500 0x7f593014c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:52.961 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:52.948+0000 7f5936c6c700 1 -- 172.21.15.195:0/1676588755 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f593006a450 msgr2=0x7f5930165e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:52.962 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:52.948+0000 7f5936c6c700 1 --2- 172.21.15.195:0/1676588755 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f593006a450 0x7f5930165e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:52.962 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:52.948+0000 7f5936c6c700 1 -- 172.21.15.195:0/1676588755 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5928009a20 con 0x7f59301508d0 2022-06-27T11:57:52.962 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:52.948+0000 7f593646b700 1 --2- 172.21.15.195:0/1676588755 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f593006a450 0x7f5930165e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T11:57:52.962 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:52.948+0000 7f5936c6c700 1 --2- 172.21.15.195:0/1676588755 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f59301508d0 0x7f593014cee0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f59280097c0 tx=0x7f5928019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:57:52.963 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:52.948+0000 7f591e7fc700 1 -- 172.21.15.195:0/1676588755 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5928009420 con 0x7f59301508d0 2022-06-27T11:57:52.963 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:52.949+0000 7f591e7fc700 1 -- 172.21.15.195:0/1676588755 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5928012a00 con 0x7f59301508d0 2022-06-27T11:57:52.963 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:52.949+0000 7f59386cf700 1 -- 172.21.15.195:0/1676588755 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5930169930 con 0x7f59301508d0 2022-06-27T11:57:52.963 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:52.949+0000 7f591e7fc700 1 -- 172.21.15.195:0/1676588755 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5928012da0 con 0x7f59301508d0 2022-06-27T11:57:52.964 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:52.949+0000 7f59386cf700 1 -- 172.21.15.195:0/1676588755 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5930169ef0 con 0x7f59301508d0 2022-06-27T11:57:52.964 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:52.949+0000 7f59386cf700 1 -- 172.21.15.195:0/1676588755 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59300681b0 con 0x7f59301508d0 2022-06-27T11:57:52.964 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:52.950+0000 7f591e7fc700 1 -- 172.21.15.195:0/1676588755 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f5928004250 con 0x7f59301508d0 2022-06-27T11:57:52.964 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:52.950+0000 7f591e7fc700 1 --2- 172.21.15.195:0/1676588755 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5910063d00 0x7f59100661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:52.964 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:52.950+0000 7f591e7fc700 1 -- 172.21.15.195:0/1676588755 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6546+0+0 (secure 0 0 0) 0x7f5928083320 con 0x7f59301508d0 2022-06-27T11:57:52.965 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:52.950+0000 7f593646b700 1 --2- 172.21.15.195:0/1676588755 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5910063d00 0x7f59100661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:52.965 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:52.951+0000 7f593646b700 1 --2- 172.21.15.195:0/1676588755 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5910063d00 0x7f59100661b0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f592c0013f0 tx=0x7f592c009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:57:52.965 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:52.955+0000 7f591e7fc700 1 -- 172.21.15.195:0/1676588755 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f5928051d20 con 0x7f59301508d0 2022-06-27T11:57:53.184 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:53.184+0000 7f59386cf700 1 -- 172.21.15.195:0/1676588755 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f59300722c0 con 0x7f59301508d0 2022-06-27T11:57:53.185 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:53.185+0000 7f591e7fc700 1 -- 172.21.15.195:0/1676588755 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+6128 (secure 0 0 0) 0x7f5928055340 con 0x7f59301508d0 2022-06-27T11:57:53.187 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:57:53.187 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":5392,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/2089155519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2089155519},{"type":"v1","addr":"172.21.15.202:6835","nonce":2089155519}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:57:31.004761+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4976},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4976":{"gid":4976,"name":"b","rank":0,"incarnation":32,"state":"up:active","state_seq":28,"addr":"172.21.15.202:6833/4154570751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":4154570751},{"type":"v1","addr":"172.21.15.202:6833","nonce":4154570751}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:57:53.190 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:53.187+0000 7f59386cf700 1 -- 172.21.15.195:0/1676588755 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5910063d00 msgr2=0x7f59100661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:53.190 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:53.187+0000 7f59386cf700 1 --2- 172.21.15.195:0/1676588755 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5910063d00 0x7f59100661b0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f592c0013f0 tx=0x7f592c009040 comp rx=0 tx=0).stop 2022-06-27T11:57:53.190 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:53.187+0000 7f59386cf700 1 -- 172.21.15.195:0/1676588755 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f59301508d0 msgr2=0x7f593014cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:53.191 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:53.187+0000 7f59386cf700 1 --2- 172.21.15.195:0/1676588755 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f59301508d0 0x7f593014cee0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f59280097c0 tx=0x7f5928019000 comp rx=0 tx=0).stop 2022-06-27T11:57:53.191 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:53.187+0000 7f59386cf700 1 -- 172.21.15.195:0/1676588755 shutdown_connections 2022-06-27T11:57:53.191 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:53.187+0000 7f59386cf700 1 --2- 172.21.15.195:0/1676588755 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5910063d00 0x7f59100661b0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:53.192 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:53.187+0000 7f59386cf700 1 --2- 172.21.15.195:0/1676588755 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f59301508d0 0x7f593014cee0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:53.192 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:53.187+0000 7f59386cf700 1 --2- 172.21.15.195:0/1676588755 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f593006a450 0x7f5930165e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:53.192 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:53.187+0000 7f59386cf700 1 --2- 172.21.15.195:0/1676588755 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5930150500 0x7f593014c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:53.192 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:53.187+0000 7f59386cf700 1 -- 172.21.15.195:0/1676588755 >> 172.21.15.195:0/1676588755 conn(0x7f59300c9b00 msgr2=0x7f593014fd50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:57:53.192 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:53.187+0000 7f59386cf700 1 -- 172.21.15.195:0/1676588755 shutdown_connections 2022-06-27T11:57:53.193 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:53.187+0000 7f59386cf700 1 -- 172.21.15.195:0/1676588755 wait complete. 2022-06-27T11:57:53.193 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 37 2022-06-27T11:57:53.202 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2022-06-27T11:57:55.204 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:57:55.344 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.344+0000 7fe336f8c700 1 Processor -- start 2022-06-27T11:57:55.345 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.344+0000 7fe336f8c700 1 -- start start 2022-06-27T11:57:55.350 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.345+0000 7fe336f8c700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fe33006f640 0x7fe330156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:55.351 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.345+0000 7fe336f8c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fe3300c4160 0x7fe3300c1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:55.351 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.345+0000 7fe336f8c700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe3300c39e0 0x7fe3301535a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:55.351 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.345+0000 7fe336f8c700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3300c1a20 con 0x7fe3300c4160 2022-06-27T11:57:55.352 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.345+0000 7fe336f8c700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3300c1130 con 0x7fe33006f640 2022-06-27T11:57:55.352 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.345+0000 7fe336f8c700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe33006ea90 con 0x7fe3300c39e0 2022-06-27T11:57:55.352 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.345+0000 7fe32ffff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fe3300c4160 0x7fe3300c1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:55.352 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.345+0000 7fe32ffff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fe3300c4160 0x7fe3300c1eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45432/0 (socket says 172.21.15.195:45432) 2022-06-27T11:57:55.352 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.345+0000 7fe32ffff700 1 -- 172.21.15.195:0/2298584227 learned_addr learned my addr 172.21.15.195:0/2298584227 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:57:55.353 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.346+0000 7fe334d28700 1 --2- 172.21.15.195:0/2298584227 >> v2:172.21.15.202:3300/0 conn(0x7fe33006f640 0x7fe330156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:55.353 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.346+0000 7fe335529700 1 --2- 172.21.15.195:0/2298584227 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe3300c39e0 0x7fe3301535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:55.353 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.346+0000 7fe32ffff700 1 -- 172.21.15.195:0/2298584227 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe3300c39e0 msgr2=0x7fe3301535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:57:55.354 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.346+0000 7fe32ffff700 1 --2- 172.21.15.195:0/2298584227 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe3300c39e0 0x7fe3301535a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:55.354 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.346+0000 7fe32ffff700 1 -- 172.21.15.195:0/2298584227 >> v2:172.21.15.202:3300/0 conn(0x7fe33006f640 msgr2=0x7fe330156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:57:55.354 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.346+0000 7fe32ffff700 1 --2- 172.21.15.195:0/2298584227 >> v2:172.21.15.202:3300/0 conn(0x7fe33006f640 0x7fe330156ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:55.355 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.346+0000 7fe32ffff700 1 -- 172.21.15.195:0/2298584227 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe33006ed70 con 0x7fe3300c4160 2022-06-27T11:57:55.355 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.346+0000 7fe334d28700 1 --2- 172.21.15.195:0/2298584227 >> v2:172.21.15.202:3300/0 conn(0x7fe33006f640 0x7fe330156ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T11:57:55.355 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.346+0000 7fe32ffff700 1 --2- 172.21.15.195:0/2298584227 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe3300c4160 0x7fe3300c1eb0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fe33006db30 tx=0x7fe31c009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56b15e9252abad5f server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:57:55.355 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.346+0000 7fe32f7fe700 1 -- 172.21.15.195:0/2298584227 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe31c00b6d0 con 0x7fe3300c4160 2022-06-27T11:57:55.356 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.346+0000 7fe32f7fe700 1 -- 172.21.15.195:0/2298584227 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe31c00b830 con 0x7fe3300c4160 2022-06-27T11:57:55.356 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.346+0000 7fe32f7fe700 1 -- 172.21.15.195:0/2298584227 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe31c00bbd0 con 0x7fe3300c4160 2022-06-27T11:57:55.356 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.347+0000 7fe336f8c700 1 -- 172.21.15.195:0/2298584227 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe3300c4160 msgr2=0x7fe3300c1eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:55.357 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.347+0000 7fe336f8c700 1 --2- 172.21.15.195:0/2298584227 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe3300c4160 0x7fe3300c1eb0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fe33006db30 tx=0x7fe31c009120 comp rx=0 tx=0).stop 2022-06-27T11:57:55.357 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.347+0000 7fe336f8c700 1 -- 172.21.15.195:0/2298584227 shutdown_connections 2022-06-27T11:57:55.357 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.347+0000 7fe336f8c700 1 --2- 172.21.15.195:0/2298584227 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe3300c39e0 0x7fe3301535a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:55.357 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.347+0000 7fe336f8c700 1 --2- 172.21.15.195:0/2298584227 >> v2:172.21.15.202:3300/0 conn(0x7fe33006f640 0x7fe330156ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:55.358 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.347+0000 7fe336f8c700 1 --2- 172.21.15.195:0/2298584227 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe3300c4160 0x7fe3300c1eb0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:55.358 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.347+0000 7fe336f8c700 1 -- 172.21.15.195:0/2298584227 >> 172.21.15.195:0/2298584227 conn(0x7fe3300c6350 msgr2=0x7fe3300c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:57:55.358 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.347+0000 7fe336f8c700 1 -- 172.21.15.195:0/2298584227 shutdown_connections 2022-06-27T11:57:55.358 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.347+0000 7fe336f8c700 1 -- 172.21.15.195:0/2298584227 wait complete. 2022-06-27T11:57:55.359 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.348+0000 7fe336f8c700 1 Processor -- start 2022-06-27T11:57:55.359 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.348+0000 7fe336f8c700 1 -- start start 2022-06-27T11:57:55.359 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.349+0000 7fe336f8c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe33006f640 0x7fe330157600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:55.359 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.349+0000 7fe336f8c700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe3300c39e0 0x7fe33016aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:55.360 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.349+0000 7fe336f8c700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe33015c150 0x7fe33014a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:55.361 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.349+0000 7fe336f8c700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3301679b0 con 0x7fe33006f640 2022-06-27T11:57:55.361 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.349+0000 7fe336f8c700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe330166da0 con 0x7fe3300c39e0 2022-06-27T11:57:55.361 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.349+0000 7fe336f8c700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe330167080 con 0x7fe33015c150 2022-06-27T11:57:55.362 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.349+0000 7fe334d28700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe33006f640 0x7fe330157600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:55.362 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.349+0000 7fe334d28700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe33006f640 0x7fe330157600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45436/0 (socket says 172.21.15.195:45436) 2022-06-27T11:57:55.362 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.349+0000 7fe334d28700 1 -- 172.21.15.195:0/360649326 learned_addr learned my addr 172.21.15.195:0/360649326 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:57:55.363 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.349+0000 7fe32ffff700 1 --2- 172.21.15.195:0/360649326 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe3300c39e0 0x7fe33016aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:55.363 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.349+0000 7fe335529700 1 --2- 172.21.15.195:0/360649326 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe33015c150 0x7fe33014a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:55.365 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.349+0000 7fe334d28700 1 -- 172.21.15.195:0/360649326 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe33015c150 msgr2=0x7fe33014a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:55.365 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.349+0000 7fe334d28700 1 --2- 172.21.15.195:0/360649326 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe33015c150 0x7fe33014a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:55.366 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.349+0000 7fe334d28700 1 -- 172.21.15.195:0/360649326 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe3300c39e0 msgr2=0x7fe33016aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:55.367 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.349+0000 7fe334d28700 1 --2- 172.21.15.195:0/360649326 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe3300c39e0 0x7fe33016aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:55.367 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.349+0000 7fe334d28700 1 -- 172.21.15.195:0/360649326 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe31c0093a0 con 0x7fe33006f640 2022-06-27T11:57:55.367 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.349+0000 7fe334d28700 1 --2- 172.21.15.195:0/360649326 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe33006f640 0x7fe330157600 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fe32400d810 tx=0x7fe324008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:57:55.367 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.349+0000 7fe32cff9700 1 -- 172.21.15.195:0/360649326 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe324004030 con 0x7fe33006f640 2022-06-27T11:57:55.368 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.349+0000 7fe32cff9700 1 -- 172.21.15.195:0/360649326 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe324004190 con 0x7fe33006f640 2022-06-27T11:57:55.368 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.349+0000 7fe32cff9700 1 -- 172.21.15.195:0/360649326 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3240125f0 con 0x7fe33006f640 2022-06-27T11:57:55.368 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.349+0000 7fe336f8c700 1 -- 172.21.15.195:0/360649326 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe330167360 con 0x7fe33006f640 2022-06-27T11:57:55.368 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.349+0000 7fe336f8c700 1 -- 172.21.15.195:0/360649326 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe330166a70 con 0x7fe33006f640 2022-06-27T11:57:55.368 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.350+0000 7fe336f8c700 1 -- 172.21.15.195:0/360649326 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3300722c0 con 0x7fe33006f640 2022-06-27T11:57:55.369 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.351+0000 7fe32cff9700 1 -- 172.21.15.195:0/360649326 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fe324012960 con 0x7fe33006f640 2022-06-27T11:57:55.369 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.351+0000 7fe32cff9700 1 --2- 172.21.15.195:0/360649326 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe318063d00 0x7fe3180661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:55.369 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.351+0000 7fe32cff9700 1 -- 172.21.15.195:0/360649326 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6546+0+0 (secure 0 0 0) 0x7fe324080d20 con 0x7fe33006f640 2022-06-27T11:57:55.369 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.351+0000 7fe32ffff700 1 --2- 172.21.15.195:0/360649326 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe318063d00 0x7fe3180661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:55.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.352+0000 7fe32ffff700 1 --2- 172.21.15.195:0/360649326 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe318063d00 0x7fe3180661b0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fe3300c73e0 tx=0x7fe31c017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:57:55.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.356+0000 7fe32cff9700 1 -- 172.21.15.195:0/360649326 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fe32404f6c0 con 0x7fe33006f640 2022-06-27T11:57:55.587 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.582+0000 7fe336f8c700 1 -- 172.21.15.195:0/360649326 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe33006f240 con 0x7fe33006f640 2022-06-27T11:57:55.588 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.582+0000 7fe32cff9700 1 -- 172.21.15.195:0/360649326 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+6128 (secure 0 0 0) 0x7fe324052ce0 con 0x7fe33006f640 2022-06-27T11:57:55.589 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:57:55.602 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":5392,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/2089155519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2089155519},{"type":"v1","addr":"172.21.15.202:6835","nonce":2089155519}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:57:31.004761+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4976},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4976":{"gid":4976,"name":"b","rank":0,"incarnation":32,"state":"up:active","state_seq":28,"addr":"172.21.15.202:6833/4154570751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":4154570751},{"type":"v1","addr":"172.21.15.202:6833","nonce":4154570751}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:57:55.604 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.584+0000 7fe336f8c700 1 -- 172.21.15.195:0/360649326 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe318063d00 msgr2=0x7fe3180661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:55.604 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.584+0000 7fe336f8c700 1 --2- 172.21.15.195:0/360649326 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe318063d00 0x7fe3180661b0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fe3300c73e0 tx=0x7fe31c017000 comp rx=0 tx=0).stop 2022-06-27T11:57:55.605 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.584+0000 7fe336f8c700 1 -- 172.21.15.195:0/360649326 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe33006f640 msgr2=0x7fe330157600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:55.605 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.584+0000 7fe336f8c700 1 --2- 172.21.15.195:0/360649326 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe33006f640 0x7fe330157600 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fe32400d810 tx=0x7fe324008040 comp rx=0 tx=0).stop 2022-06-27T11:57:55.605 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.584+0000 7fe336f8c700 1 -- 172.21.15.195:0/360649326 shutdown_connections 2022-06-27T11:57:55.605 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.584+0000 7fe336f8c700 1 --2- 172.21.15.195:0/360649326 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe318063d00 0x7fe3180661b0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:55.620 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.584+0000 7fe336f8c700 1 --2- 172.21.15.195:0/360649326 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe33006f640 0x7fe330157600 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:55.620 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.584+0000 7fe336f8c700 1 --2- 172.21.15.195:0/360649326 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe33015c150 0x7fe33014a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:55.621 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.584+0000 7fe336f8c700 1 --2- 172.21.15.195:0/360649326 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe3300c39e0 0x7fe33016aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:55.621 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.584+0000 7fe336f8c700 1 -- 172.21.15.195:0/360649326 >> 172.21.15.195:0/360649326 conn(0x7fe3300c6350 msgr2=0x7fe330152a50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:57:55.621 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.584+0000 7fe336f8c700 1 -- 172.21.15.195:0/360649326 shutdown_connections 2022-06-27T11:57:55.622 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:55.584+0000 7fe336f8c700 1 -- 172.21.15.195:0/360649326 wait complete. 2022-06-27T11:57:55.624 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 37 2022-06-27T11:57:55.625 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2022-06-27T11:57:57.626 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:57:57.760 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.760+0000 7f0028466700 1 Processor -- start 2022-06-27T11:57:57.761 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.760+0000 7f0028466700 1 -- start start 2022-06-27T11:57:57.768 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.761+0000 7f0028466700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f0020157e40 0x7f00200c1b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:57.768 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.761+0000 7f0028466700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00200c6140 con 0x7f00200c20c0 2022-06-27T11:57:57.769 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.761+0000 7f0028466700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f002015ecd0 con 0x7f00200c3a00 2022-06-27T11:57:57.769 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.761+0000 7f0028466700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00200c2540 con 0x7f0020157e40 2022-06-27T11:57:57.770 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.762+0000 7f0026a03700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f00200c20c0 0x7f0020165150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:51636/0 (socket says 172.21.15.195:51636) 2022-06-27T11:57:57.770 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.762+0000 7f0026a03700 1 -- 172.21.15.195:0/2443165031 learned_addr learned my addr 172.21.15.195:0/2443165031 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:57:57.770 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.762+0000 7f0026202700 1 --2- 172.21.15.195:0/2443165031 >> v2:172.21.15.202:3300/0 conn(0x7f0020157e40 0x7f00200c1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:57.771 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.762+0000 7f0025200700 1 -- 172.21.15.195:0/2443165031 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3928474782 0 0) 0x7f00200c6140 con 0x7f00200c20c0 2022-06-27T11:57:57.771 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.762+0000 7f0025200700 1 -- 172.21.15.195:0/2443165031 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0008003430 con 0x7f00200c20c0 2022-06-27T11:57:57.772 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.762+0000 7f0025200700 1 -- 172.21.15.195:0/2443165031 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1265619697 0 0) 0x7f001c001110 con 0x7f00200c3a00 2022-06-27T11:57:57.772 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.762+0000 7f0025200700 1 -- 172.21.15.195:0/2443165031 --> v1:172.21.15.202:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00200c6140 con 0x7f00200c3a00 2022-06-27T11:57:57.773 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.762+0000 7f0025200700 1 -- 172.21.15.195:0/2443165031 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 447612317 0 0) 0x7f0008003430 con 0x7f00200c20c0 2022-06-27T11:57:57.773 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.762+0000 7f0025200700 1 -- 172.21.15.195:0/2443165031 >> v2:172.21.15.202:3300/0 conn(0x7f0020157e40 msgr2=0x7f00200c1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:57:57.775 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.762+0000 7f0025200700 1 --2- 172.21.15.195:0/2443165031 >> v2:172.21.15.202:3300/0 conn(0x7f0020157e40 0x7f00200c1b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:57.776 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.762+0000 7f0025200700 1 -- 172.21.15.195:0/2443165031 >> v1:172.21.15.202:6789/0 conn(0x7f00200c3a00 legacy=0x7f002006f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:57.776 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.762+0000 7f0025200700 1 -- 172.21.15.195:0/2443165031 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f002015e9f0 con 0x7f00200c20c0 2022-06-27T11:57:57.777 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.762+0000 7f0025200700 1 -- 172.21.15.195:0/2443165031 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7f00180017f0 con 0x7f00200c20c0 2022-06-27T11:57:57.777 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.762+0000 7f0025200700 1 -- 172.21.15.195:0/2443165031 >> v1:172.21.15.195:6789/0 conn(0x7f00200c20c0 legacy=0x7f0020165150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:57.794 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.762+0000 7f0025200700 1 --2- 172.21.15.195:0/2443165031 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0008003ab0 0x7f0008003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:57.803 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.762+0000 7f0025200700 1 --2- 172.21.15.195:0/2443165031 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f00080046d0 0x7f0008004b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:57.803 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.762+0000 7f0025200700 1 --2- 172.21.15.195:0/2443165031 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f00080050f0 0x7f000800b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:57.804 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.762+0000 7f0025200700 1 -- 172.21.15.195:0/2443165031 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00200c2540 con 0x7f00080050f0 2022-06-27T11:57:57.805 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.762+0000 7f0025200700 1 -- 172.21.15.195:0/2443165031 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f000800c0c0 con 0x7f0008003ab0 2022-06-27T11:57:57.805 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.762+0000 7f0025200700 1 -- 172.21.15.195:0/2443165031 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f000800c280 con 0x7f00080046d0 2022-06-27T11:57:57.805 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.763+0000 7f0026a03700 1 --2- 172.21.15.195:0/2443165031 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f00080050f0 0x7f000800b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:57.805 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.763+0000 7f0026202700 1 --2- 172.21.15.195:0/2443165031 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0008003ab0 0x7f0008003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:57.806 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.763+0000 7f0025a01700 1 --2- 172.21.15.195:0/2443165031 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f00080046d0 0x7f0008004b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:57.806 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.763+0000 7f0026a03700 1 -- 172.21.15.195:0/2443165031 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f00080046d0 msgr2=0x7f0008004b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:57:57.806 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.763+0000 7f0026a03700 1 --2- 172.21.15.195:0/2443165031 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f00080046d0 0x7f0008004b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:57.806 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.763+0000 7f0026a03700 1 -- 172.21.15.195:0/2443165031 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0008003ab0 msgr2=0x7f0008003ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:57:57.808 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.763+0000 7f0026a03700 1 --2- 172.21.15.195:0/2443165031 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0008003ab0 0x7f0008003ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:57.809 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.763+0000 7f0026a03700 1 -- 172.21.15.195:0/2443165031 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0008003430 con 0x7f00080050f0 2022-06-27T11:57:57.809 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.763+0000 7f0026a03700 1 -- 172.21.15.195:0/2443165031 reap_dead start 2022-06-27T11:57:57.809 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.763+0000 7f0026a03700 1 --2- 172.21.15.195:0/2443165031 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f00080050f0 0x7f000800b920 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f001800c5c0 tx=0x7f0018007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cf0c79353baecf2 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:57:57.810 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.763+0000 7f0025200700 1 -- 172.21.15.195:0/2443165031 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f001800cdc0 con 0x7f00080050f0 2022-06-27T11:57:57.810 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.763+0000 7f0025200700 1 -- 172.21.15.195:0/2443165031 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0018005b60 con 0x7f00080050f0 2022-06-27T11:57:57.811 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.763+0000 7f0025200700 1 -- 172.21.15.195:0/2443165031 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00180070d0 con 0x7f00080050f0 2022-06-27T11:57:57.811 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.764+0000 7f0028466700 1 -- 172.21.15.195:0/2443165031 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f00080050f0 msgr2=0x7f000800b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:57.812 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.764+0000 7f0028466700 1 --2- 172.21.15.195:0/2443165031 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f00080050f0 0x7f000800b920 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f001800c5c0 tx=0x7f0018007f60 comp rx=0 tx=0).stop 2022-06-27T11:57:57.812 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.764+0000 7f0028466700 1 -- 172.21.15.195:0/2443165031 shutdown_connections 2022-06-27T11:57:57.813 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.764+0000 7f0028466700 1 --2- 172.21.15.195:0/2443165031 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f00080050f0 0x7f000800b920 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:57.813 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.764+0000 7f0028466700 1 -- 172.21.15.195:0/2443165031 >> 172.21.15.195:0/2443165031 conn(0x7f00200c6350 msgr2=0x7f00200c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:57:57.813 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.764+0000 7f0028466700 1 -- 172.21.15.195:0/2443165031 shutdown_connections 2022-06-27T11:57:57.813 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.764+0000 7f0028466700 1 -- 172.21.15.195:0/2443165031 wait complete. 2022-06-27T11:57:57.813 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.765+0000 7f0028466700 1 Processor -- start 2022-06-27T11:57:57.814 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.765+0000 7f0028466700 1 -- start start 2022-06-27T11:57:57.814 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.766+0000 7f0028466700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f00080050f0 0x7f002016b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:57.833 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.766+0000 7f0028466700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f002015b2c0 0x7f0020153b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:57.833 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.766+0000 7f0028466700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f00200bf670 0x7f00200bfac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:57.834 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.766+0000 7f0028466700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f002015ba90 con 0x7f00080050f0 2022-06-27T11:57:57.834 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.766+0000 7f0028466700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f002006af40 con 0x7f00200bf670 2022-06-27T11:57:57.834 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.766+0000 7f0028466700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f002016bac0 con 0x7f002015b2c0 2022-06-27T11:57:57.834 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.766+0000 7f0026202700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f00080050f0 0x7f002016b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:57.835 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.766+0000 7f0026202700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f00080050f0 0x7f002016b580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45456/0 (socket says 172.21.15.195:45456) 2022-06-27T11:57:57.835 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.766+0000 7f0026202700 1 -- 172.21.15.195:0/1094325588 learned_addr learned my addr 172.21.15.195:0/1094325588 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:57:57.835 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.766+0000 7f0025a01700 1 --2- 172.21.15.195:0/1094325588 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f002015b2c0 0x7f0020153b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:57.835 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.766+0000 7f0026a03700 1 --2- 172.21.15.195:0/1094325588 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f00200bf670 0x7f00200bfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:57.835 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.766+0000 7f0026202700 1 -- 172.21.15.195:0/1094325588 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f002015b2c0 msgr2=0x7f0020153b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:57.836 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.766+0000 7f0026202700 1 --2- 172.21.15.195:0/1094325588 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f002015b2c0 0x7f0020153b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:57.836 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.766+0000 7f0026202700 1 -- 172.21.15.195:0/1094325588 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f00200bf670 msgr2=0x7f00200bfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:57.836 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.766+0000 7f0026202700 1 --2- 172.21.15.195:0/1094325588 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f00200bf670 0x7f00200bfac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:57.836 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.766+0000 7f0026202700 1 -- 172.21.15.195:0/1094325588 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00200c0000 con 0x7f00080050f0 2022-06-27T11:57:57.837 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.767+0000 7f0026202700 1 --2- 172.21.15.195:0/1094325588 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f00080050f0 0x7f002016b580 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f0010011910 tx=0x7f0010009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:57:57.837 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.767+0000 7f000e7fc700 1 -- 172.21.15.195:0/1094325588 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0010004240 con 0x7f00080050f0 2022-06-27T11:57:57.837 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.767+0000 7f000e7fc700 1 -- 172.21.15.195:0/1094325588 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00100043a0 con 0x7f00080050f0 2022-06-27T11:57:57.837 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.767+0000 7f0028466700 1 -- 172.21.15.195:0/1094325588 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f002015b7b0 con 0x7f00080050f0 2022-06-27T11:57:57.838 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.767+0000 7f000e7fc700 1 -- 172.21.15.195:0/1094325588 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0010003750 con 0x7f00080050f0 2022-06-27T11:57:57.838 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.767+0000 7f0028466700 1 -- 172.21.15.195:0/1094325588 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00200beb90 con 0x7f00080050f0 2022-06-27T11:57:57.838 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.768+0000 7f0028466700 1 -- 172.21.15.195:0/1094325588 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00200722c0 con 0x7f00080050f0 2022-06-27T11:57:57.838 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.768+0000 7f000e7fc700 1 -- 172.21.15.195:0/1094325588 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f0010024070 con 0x7f00080050f0 2022-06-27T11:57:57.839 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.768+0000 7f000e7fc700 1 --2- 172.21.15.195:0/1094325588 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0000063d30 0x7f00000661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:57:57.839 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.768+0000 7f000e7fc700 1 -- 172.21.15.195:0/1094325588 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6546+0+0 (secure 0 0 0) 0x7f0010018030 con 0x7f00080050f0 2022-06-27T11:57:57.839 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.768+0000 7f0025a01700 1 --2- 172.21.15.195:0/1094325588 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0000063d30 0x7f00000661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:57:57.839 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.769+0000 7f0025a01700 1 --2- 172.21.15.195:0/1094325588 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0000063d30 0x7f00000661e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f00200c5b00 tx=0x7f001c009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:57:57.840 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.773+0000 7f000e7fc700 1 -- 172.21.15.195:0/1094325588 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f001004fab0 con 0x7f00080050f0 2022-06-27T11:57:57.999 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:57.999+0000 7f0028466700 1 -- 172.21.15.195:0/1094325588 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0020156ce0 con 0x7f00080050f0 2022-06-27T11:57:57.999 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:58.000+0000 7f000e7fc700 1 -- 172.21.15.195:0/1094325588 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+6128 (secure 0 0 0) 0x7f0010022090 con 0x7f00080050f0 2022-06-27T11:57:58.000 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:57:58.001 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":5392,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/2089155519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2089155519},{"type":"v1","addr":"172.21.15.202:6835","nonce":2089155519}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:57:31.004761+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4976},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4976":{"gid":4976,"name":"b","rank":0,"incarnation":32,"state":"up:active","state_seq":28,"addr":"172.21.15.202:6833/4154570751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":4154570751},{"type":"v1","addr":"172.21.15.202:6833","nonce":4154570751}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:57:58.001 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:58.001+0000 7f0028466700 1 -- 172.21.15.195:0/1094325588 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0000063d30 msgr2=0x7f00000661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:58.002 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:58.001+0000 7f0028466700 1 --2- 172.21.15.195:0/1094325588 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0000063d30 0x7f00000661e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f00200c5b00 tx=0x7f001c009040 comp rx=0 tx=0).stop 2022-06-27T11:57:58.002 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:58.001+0000 7f0028466700 1 -- 172.21.15.195:0/1094325588 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f00080050f0 msgr2=0x7f002016b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:57:58.002 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:58.001+0000 7f0028466700 1 --2- 172.21.15.195:0/1094325588 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f00080050f0 0x7f002016b580 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f0010011910 tx=0x7f0010009060 comp rx=0 tx=0).stop 2022-06-27T11:57:58.002 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:58.002+0000 7f0028466700 1 -- 172.21.15.195:0/1094325588 shutdown_connections 2022-06-27T11:57:58.021 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:58.002+0000 7f0028466700 1 --2- 172.21.15.195:0/1094325588 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0000063d30 0x7f00000661e0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:58.021 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:58.002+0000 7f0028466700 1 --2- 172.21.15.195:0/1094325588 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f00080050f0 0x7f002016b580 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:58.022 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:58.002+0000 7f0028466700 1 --2- 172.21.15.195:0/1094325588 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f00200bf670 0x7f00200bfac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:58.022 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:58.002+0000 7f0028466700 1 --2- 172.21.15.195:0/1094325588 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f002015b2c0 0x7f0020153b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:57:58.024 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:58.002+0000 7f0028466700 1 -- 172.21.15.195:0/1094325588 >> 172.21.15.195:0/1094325588 conn(0x7f00200c6350 msgr2=0x7f00200c8c70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:57:58.024 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:58.002+0000 7f0028466700 1 -- 172.21.15.195:0/1094325588 shutdown_connections 2022-06-27T11:57:58.025 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:57:58.002+0000 7f0028466700 1 -- 172.21.15.195:0/1094325588 wait complete. 2022-06-27T11:57:58.025 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 37 2022-06-27T11:57:58.026 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2022-06-27T11:57:59.051 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T11:57:59.055 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T11:58:00.029 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:58:00.170 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.170+0000 7f6122817700 1 Processor -- start 2022-06-27T11:58:00.171 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.170+0000 7f6122817700 1 -- start start 2022-06-27T11:58:00.174 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.171+0000 7f6122817700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f611c14e490 0x7f611c06cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:00.175 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.171+0000 7f6122817700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f611c14e860 0x7f611c0c3840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:00.175 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.171+0000 7f6122817700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f611c0c33b0 con 0x7f611c0c3d80 2022-06-27T11:58:00.175 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.171+0000 7f6122817700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f611c06d520 con 0x7f611c14e860 2022-06-27T11:58:00.176 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.171+0000 7f6122817700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f611c06f220 con 0x7f611c14e490 2022-06-27T11:58:00.176 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.171+0000 7f611b7fe700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f611c14e860 0x7f611c0c3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:00.177 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.171+0000 7f611b7fe700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f611c14e860 0x7f611c0c3840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45468/0 (socket says 172.21.15.195:45468) 2022-06-27T11:58:00.177 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.172+0000 7f611b7fe700 1 -- 172.21.15.195:0/1897206221 learned_addr learned my addr 172.21.15.195:0/1897206221 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:58:00.177 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.172+0000 7f611bfff700 1 --2- 172.21.15.195:0/1897206221 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f611c14e490 0x7f611c06cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:00.182 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.172+0000 7f611b7fe700 1 -- 172.21.15.195:0/1897206221 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f611c14e490 msgr2=0x7f611c06cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:58:00.214 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.172+0000 7f611b7fe700 1 --2- 172.21.15.195:0/1897206221 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f611c14e490 0x7f611c06cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:00.216 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.172+0000 7f611b7fe700 1 -- 172.21.15.195:0/1897206221 >> v1:172.21.15.202:6789/0 conn(0x7f611c0c3d80 legacy=0x7f611c06f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:00.216 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.172+0000 7f611b7fe700 1 -- 172.21.15.195:0/1897206221 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f611c1542c0 con 0x7f611c14e860 2022-06-27T11:58:00.217 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.172+0000 7f611bfff700 1 --2- 172.21.15.195:0/1897206221 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f611c14e490 0x7f611c06cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T11:58:00.217 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.172+0000 7f611b7fe700 1 --2- 172.21.15.195:0/1897206221 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f611c14e860 0x7f611c0c3840 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f6104009860 tx=0x7f6104009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9febf5a3f8ea9eb server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:00.218 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.172+0000 7f611affd700 1 -- 172.21.15.195:0/1897206221 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f610400b680 con 0x7f611c14e860 2022-06-27T11:58:00.219 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.172+0000 7f611affd700 1 -- 172.21.15.195:0/1897206221 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f610400b7e0 con 0x7f611c14e860 2022-06-27T11:58:00.221 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.172+0000 7f611affd700 1 -- 172.21.15.195:0/1897206221 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f610400bba0 con 0x7f611c14e860 2022-06-27T11:58:00.221 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.173+0000 7f6122817700 1 -- 172.21.15.195:0/1897206221 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f611c14e860 msgr2=0x7f611c0c3840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:00.222 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.173+0000 7f6122817700 1 --2- 172.21.15.195:0/1897206221 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f611c14e860 0x7f611c0c3840 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f6104009860 tx=0x7f6104009120 comp rx=0 tx=0).stop 2022-06-27T11:58:00.222 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.173+0000 7f6122817700 1 -- 172.21.15.195:0/1897206221 shutdown_connections 2022-06-27T11:58:00.223 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.173+0000 7f6122817700 1 --2- 172.21.15.195:0/1897206221 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f611c14e490 0x7f611c06cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:00.223 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.173+0000 7f6122817700 1 --2- 172.21.15.195:0/1897206221 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f611c14e860 0x7f611c0c3840 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:00.224 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.173+0000 7f6122817700 1 -- 172.21.15.195:0/1897206221 >> 172.21.15.195:0/1897206221 conn(0x7f611c063af0 msgr2=0x7f611c063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:58:00.224 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.173+0000 7f6122817700 1 -- 172.21.15.195:0/1897206221 shutdown_connections 2022-06-27T11:58:00.225 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.173+0000 7f6122817700 1 -- 172.21.15.195:0/1897206221 wait complete. 2022-06-27T11:58:00.226 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.174+0000 7f6122817700 1 Processor -- start 2022-06-27T11:58:00.226 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.174+0000 7f6122817700 1 -- start start 2022-06-27T11:58:00.226 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.175+0000 7f6122817700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f611c0c3d80 0x7f611c167110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:00.227 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.175+0000 7f6122817700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f611c14e490 0x7f611c068fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:00.227 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.175+0000 7f6122817700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f611c14e860 0x7f611c155670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:00.228 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.175+0000 7f6122817700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f611c0bf2c0 con 0x7f611c0c3d80 2022-06-27T11:58:00.228 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.175+0000 7f6122817700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f611c0be9a0 con 0x7f611c14e860 2022-06-27T11:58:00.228 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.175+0000 7f6122817700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f611c167650 con 0x7f611c14e490 2022-06-27T11:58:00.229 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.175+0000 7f611bfff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f611c0c3d80 0x7f611c167110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:00.229 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.175+0000 7f611bfff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f611c0c3d80 0x7f611c167110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45472/0 (socket says 172.21.15.195:45472) 2022-06-27T11:58:00.229 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.175+0000 7f611bfff700 1 -- 172.21.15.195:0/2289769957 learned_addr learned my addr 172.21.15.195:0/2289769957 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:58:00.230 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.175+0000 7f611b7fe700 1 --2- 172.21.15.195:0/2289769957 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f611c14e490 0x7f611c068fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:00.231 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.175+0000 7f6120db4700 1 --2- 172.21.15.195:0/2289769957 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f611c14e860 0x7f611c155670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:00.231 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.175+0000 7f611bfff700 1 -- 172.21.15.195:0/2289769957 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f611c14e490 msgr2=0x7f611c068fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:00.231 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.175+0000 7f611bfff700 1 --2- 172.21.15.195:0/2289769957 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f611c14e490 0x7f611c068fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:00.232 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.175+0000 7f611bfff700 1 -- 172.21.15.195:0/2289769957 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f611c14e860 msgr2=0x7f611c155670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:00.232 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.175+0000 7f611bfff700 1 --2- 172.21.15.195:0/2289769957 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f611c14e860 0x7f611c155670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:00.233 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.175+0000 7f611bfff700 1 -- 172.21.15.195:0/2289769957 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61040093a0 con 0x7f611c0c3d80 2022-06-27T11:58:00.234 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.175+0000 7f611bfff700 1 --2- 172.21.15.195:0/2289769957 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f611c0c3d80 0x7f611c167110 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f610c00c880 tx=0x7f610c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:00.234 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.176+0000 7f6113fff700 1 -- 172.21.15.195:0/2289769957 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f610c011600 con 0x7f611c0c3d80 2022-06-27T11:58:00.235 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.176+0000 7f6113fff700 1 -- 172.21.15.195:0/2289769957 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f610c011760 con 0x7f611c0c3d80 2022-06-27T11:58:00.235 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.176+0000 7f6122817700 1 -- 172.21.15.195:0/2289769957 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f611c0c1210 con 0x7f611c0c3d80 2022-06-27T11:58:00.236 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.176+0000 7f6113fff700 1 -- 172.21.15.195:0/2289769957 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f610c011ae0 con 0x7f611c0c3d80 2022-06-27T11:58:00.236 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.176+0000 7f6122817700 1 -- 172.21.15.195:0/2289769957 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f611c0c08e0 con 0x7f611c0c3d80 2022-06-27T11:58:00.237 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.177+0000 7f6122817700 1 -- 172.21.15.195:0/2289769957 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f611c0727f0 con 0x7f611c0c3d80 2022-06-27T11:58:00.237 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.177+0000 7f6113fff700 1 -- 172.21.15.195:0/2289769957 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f610c021070 con 0x7f611c0c3d80 2022-06-27T11:58:00.237 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.178+0000 7f6113fff700 1 --2- 172.21.15.195:0/2289769957 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6100063d30 0x7f61000661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:00.238 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.178+0000 7f6113fff700 1 -- 172.21.15.195:0/2289769957 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6546+0+0 (secure 0 0 0) 0x7f610c080710 con 0x7f611c0c3d80 2022-06-27T11:58:00.238 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.178+0000 7f611b7fe700 1 --2- 172.21.15.195:0/2289769957 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6100063d30 0x7f61000661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:00.239 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.178+0000 7f611b7fe700 1 --2- 172.21.15.195:0/2289769957 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6100063d30 0x7f61000661e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f611c065b00 tx=0x7f6104017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:00.239 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.182+0000 7f6113fff700 1 -- 172.21.15.195:0/2289769957 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f610c04f110 con 0x7f611c0c3d80 2022-06-27T11:58:00.409 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.410+0000 7f6122817700 1 -- 172.21.15.195:0/2289769957 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f611c0722c0 con 0x7f611c0c3d80 2022-06-27T11:58:00.411 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.410+0000 7f6113fff700 1 -- 172.21.15.195:0/2289769957 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+6128 (secure 0 0 0) 0x7f610c01f090 con 0x7f611c0c3d80 2022-06-27T11:58:00.412 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:58:00.416 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":5392,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/2089155519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2089155519},{"type":"v1","addr":"172.21.15.202:6835","nonce":2089155519}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:57:31.004761+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4976},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4976":{"gid":4976,"name":"b","rank":0,"incarnation":32,"state":"up:active","state_seq":28,"addr":"172.21.15.202:6833/4154570751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":4154570751},{"type":"v1","addr":"172.21.15.202:6833","nonce":4154570751}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:58:00.423 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.412+0000 7f6122817700 1 -- 172.21.15.195:0/2289769957 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6100063d30 msgr2=0x7f61000661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:00.423 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.412+0000 7f6122817700 1 --2- 172.21.15.195:0/2289769957 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6100063d30 0x7f61000661e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f611c065b00 tx=0x7f6104017000 comp rx=0 tx=0).stop 2022-06-27T11:58:00.424 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.412+0000 7f6122817700 1 -- 172.21.15.195:0/2289769957 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f611c0c3d80 msgr2=0x7f611c167110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:00.424 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.412+0000 7f6122817700 1 --2- 172.21.15.195:0/2289769957 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f611c0c3d80 0x7f611c167110 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f610c00c880 tx=0x7f610c008040 comp rx=0 tx=0).stop 2022-06-27T11:58:00.424 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.412+0000 7f6122817700 1 -- 172.21.15.195:0/2289769957 shutdown_connections 2022-06-27T11:58:00.424 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.412+0000 7f6122817700 1 --2- 172.21.15.195:0/2289769957 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6100063d30 0x7f61000661e0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:00.425 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.412+0000 7f6122817700 1 --2- 172.21.15.195:0/2289769957 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f611c0c3d80 0x7f611c167110 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:00.425 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.412+0000 7f6122817700 1 --2- 172.21.15.195:0/2289769957 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f611c14e860 0x7f611c155670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:00.426 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.412+0000 7f6122817700 1 --2- 172.21.15.195:0/2289769957 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f611c14e490 0x7f611c068fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:00.426 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.412+0000 7f6122817700 1 -- 172.21.15.195:0/2289769957 >> 172.21.15.195:0/2289769957 conn(0x7f611c063af0 msgr2=0x7f611c152ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:58:00.426 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.412+0000 7f6122817700 1 -- 172.21.15.195:0/2289769957 shutdown_connections 2022-06-27T11:58:00.426 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:00.413+0000 7f6122817700 1 -- 172.21.15.195:0/2289769957 wait complete. 2022-06-27T11:58:00.427 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 37 2022-06-27T11:58:00.428 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2022-06-27T11:58:02.430 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:58:02.575 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.573+0000 7f28e328d700 1 Processor -- start 2022-06-27T11:58:02.576 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.573+0000 7f28e328d700 1 -- start start 2022-06-27T11:58:02.576 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.574+0000 7f28e328d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f28dc06fef0 0x7f28dc1570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:02.576 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.574+0000 7f28e328d700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f28dc0c34a0 0x7f28dc0c2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:02.577 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.574+0000 7f28e328d700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28dc06fba0 con 0x7f28dc06e850 2022-06-27T11:58:02.577 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.574+0000 7f28e328d700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28dc1575e0 con 0x7f28dc06fef0 2022-06-27T11:58:02.578 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.574+0000 7f28e328d700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28dc06f5e0 con 0x7f28dc0c34a0 2022-06-27T11:58:02.587 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.574+0000 7f28e0828700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f28dc06fef0 0x7f28dc1570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:02.588 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.574+0000 7f28e0828700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f28dc06fef0 0x7f28dc1570a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45486/0 (socket says 172.21.15.195:45486) 2022-06-27T11:58:02.589 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.574+0000 7f28e0828700 1 -- 172.21.15.195:0/115934040 learned_addr learned my addr 172.21.15.195:0/115934040 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:58:02.589 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.575+0000 7f28e182a700 1 --2- 172.21.15.195:0/115934040 >> v2:172.21.15.202:3300/0 conn(0x7f28dc0c34a0 0x7f28dc0c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:02.590 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.575+0000 7f28e0828700 1 -- 172.21.15.195:0/115934040 >> v2:172.21.15.202:3300/0 conn(0x7f28dc0c34a0 msgr2=0x7f28dc0c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:58:02.591 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.575+0000 7f28e0828700 1 --2- 172.21.15.195:0/115934040 >> v2:172.21.15.202:3300/0 conn(0x7f28dc0c34a0 0x7f28dc0c2920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:02.591 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.575+0000 7f28e0828700 1 -- 172.21.15.195:0/115934040 >> v1:172.21.15.202:6789/0 conn(0x7f28dc06e850 legacy=0x7f28dc069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:02.591 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.575+0000 7f28e0828700 1 -- 172.21.15.195:0/115934040 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28dc06f8c0 con 0x7f28dc06fef0 2022-06-27T11:58:02.591 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.575+0000 7f28e0828700 1 --2- 172.21.15.195:0/115934040 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f28dc06fef0 0x7f28dc1570a0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f28dc0c1420 tx=0x7f28d800b3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=590464547a9d9fd5 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:02.592 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.575+0000 7f28cbfff700 1 -- 172.21.15.195:0/115934040 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28d800db00 con 0x7f28dc06fef0 2022-06-27T11:58:02.592 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.575+0000 7f28cbfff700 1 -- 172.21.15.195:0/115934040 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28d800dc60 con 0x7f28dc06fef0 2022-06-27T11:58:02.592 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.575+0000 7f28cbfff700 1 -- 172.21.15.195:0/115934040 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28d8019480 con 0x7f28dc06fef0 2022-06-27T11:58:02.592 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.576+0000 7f28e328d700 1 -- 172.21.15.195:0/115934040 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f28dc06fef0 msgr2=0x7f28dc1570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:02.593 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.576+0000 7f28e328d700 1 --2- 172.21.15.195:0/115934040 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f28dc06fef0 0x7f28dc1570a0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f28dc0c1420 tx=0x7f28d800b3b0 comp rx=0 tx=0).stop 2022-06-27T11:58:02.593 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.576+0000 7f28e328d700 1 -- 172.21.15.195:0/115934040 shutdown_connections 2022-06-27T11:58:02.594 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.576+0000 7f28e328d700 1 --2- 172.21.15.195:0/115934040 >> v2:172.21.15.202:3300/0 conn(0x7f28dc0c34a0 0x7f28dc0c2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:02.594 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.576+0000 7f28e328d700 1 --2- 172.21.15.195:0/115934040 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f28dc06fef0 0x7f28dc1570a0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:02.594 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.576+0000 7f28e328d700 1 -- 172.21.15.195:0/115934040 >> 172.21.15.195:0/115934040 conn(0x7f28dc0c6350 msgr2=0x7f28dc0c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:58:02.595 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.576+0000 7f28e328d700 1 -- 172.21.15.195:0/115934040 shutdown_connections 2022-06-27T11:58:02.595 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.576+0000 7f28e328d700 1 -- 172.21.15.195:0/115934040 wait complete. 2022-06-27T11:58:02.595 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.577+0000 7f28e328d700 1 Processor -- start 2022-06-27T11:58:02.596 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.577+0000 7f28e328d700 1 -- start start 2022-06-27T11:58:02.596 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.578+0000 7f28e328d700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f28dc06e850 0x7f28dc165d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:02.596 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.578+0000 7f28e328d700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f28dc0c34a0 0x7f28dc0c6720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:02.596 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.578+0000 7f28e328d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f28dc0be7a0 0x7f28dc0c6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:02.597 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.578+0000 7f28e328d700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28dc1658a0 con 0x7f28dc0be7a0 2022-06-27T11:58:02.597 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.578+0000 7f28e328d700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28dc14fd30 con 0x7f28dc06e850 2022-06-27T11:58:02.597 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.578+0000 7f28e328d700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28dc153c80 con 0x7f28dc0c34a0 2022-06-27T11:58:02.597 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.578+0000 7f28e182a700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f28dc0be7a0 0x7f28dc0c6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:02.598 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.578+0000 7f28e182a700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f28dc0be7a0 0x7f28dc0c6c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45492/0 (socket says 172.21.15.195:45492) 2022-06-27T11:58:02.598 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.578+0000 7f28e182a700 1 -- 172.21.15.195:0/148385463 learned_addr learned my addr 172.21.15.195:0/148385463 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:58:02.598 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.578+0000 7f28e1029700 1 --2- 172.21.15.195:0/148385463 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f28dc06e850 0x7f28dc165d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:02.598 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.578+0000 7f28e0828700 1 --2- 172.21.15.195:0/148385463 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f28dc0c34a0 0x7f28dc0c6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:02.599 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.578+0000 7f28e182a700 1 -- 172.21.15.195:0/148385463 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f28dc0c34a0 msgr2=0x7f28dc0c6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:02.599 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.578+0000 7f28e182a700 1 --2- 172.21.15.195:0/148385463 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f28dc0c34a0 0x7f28dc0c6720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:02.599 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.578+0000 7f28e182a700 1 -- 172.21.15.195:0/148385463 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f28dc06e850 msgr2=0x7f28dc165d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:02.599 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.578+0000 7f28e182a700 1 --2- 172.21.15.195:0/148385463 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f28dc06e850 0x7f28dc165d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:02.599 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.578+0000 7f28e182a700 1 -- 172.21.15.195:0/148385463 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28d800b650 con 0x7f28dc0be7a0 2022-06-27T11:58:02.600 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.578+0000 7f28e182a700 1 --2- 172.21.15.195:0/148385463 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f28dc0be7a0 0x7f28dc0c6c60 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f28d4009e90 tx=0x7f28d400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:02.600 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.578+0000 7f28c97fa700 1 -- 172.21.15.195:0/148385463 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28d400cc00 con 0x7f28dc0be7a0 2022-06-27T11:58:02.600 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.578+0000 7f28c97fa700 1 -- 172.21.15.195:0/148385463 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28d400f040 con 0x7f28dc0be7a0 2022-06-27T11:58:02.600 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.579+0000 7f28e328d700 1 -- 172.21.15.195:0/148385463 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28dc165530 con 0x7f28dc0be7a0 2022-06-27T11:58:02.601 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.579+0000 7f28c97fa700 1 -- 172.21.15.195:0/148385463 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28d4014480 con 0x7f28dc0be7a0 2022-06-27T11:58:02.601 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.579+0000 7f28e328d700 1 -- 172.21.15.195:0/148385463 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28dc166270 con 0x7f28dc0be7a0 2022-06-27T11:58:02.601 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.579+0000 7f28e328d700 1 -- 172.21.15.195:0/148385463 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28dc0722c0 con 0x7f28dc0be7a0 2022-06-27T11:58:02.601 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.580+0000 7f28c97fa700 1 -- 172.21.15.195:0/148385463 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f28d4014830 con 0x7f28dc0be7a0 2022-06-27T11:58:02.601 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.580+0000 7f28c97fa700 1 --2- 172.21.15.195:0/148385463 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f28bc063d00 0x7f28bc0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:02.603 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.580+0000 7f28c97fa700 1 -- 172.21.15.195:0/148385463 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6546+0+0 (secure 0 0 0) 0x7f28d40816d0 con 0x7f28dc0be7a0 2022-06-27T11:58:02.603 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.580+0000 7f28e1029700 1 --2- 172.21.15.195:0/148385463 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f28bc063d00 0x7f28bc0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:02.603 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.581+0000 7f28e1029700 1 --2- 172.21.15.195:0/148385463 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f28bc063d00 0x7f28bc0661b0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f28cc0017e0 tx=0x7f28cc009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:02.603 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.585+0000 7f28c97fa700 1 -- 172.21.15.195:0/148385463 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f28d40500d0 con 0x7f28dc0be7a0 2022-06-27T11:58:02.806 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.805+0000 7f28e328d700 1 -- 172.21.15.195:0/148385463 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f28dc06f240 con 0x7f28dc0be7a0 2022-06-27T11:58:02.806 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.806+0000 7f28c97fa700 1 -- 172.21.15.195:0/148385463 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+6128 (secure 0 0 0) 0x7f28d40536f0 con 0x7f28dc0be7a0 2022-06-27T11:58:02.807 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:58:02.808 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":5392,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/2089155519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2089155519},{"type":"v1","addr":"172.21.15.202:6835","nonce":2089155519}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:57:31.004761+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4976},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4976":{"gid":4976,"name":"b","rank":0,"incarnation":32,"state":"up:active","state_seq":28,"addr":"172.21.15.202:6833/4154570751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":4154570751},{"type":"v1","addr":"172.21.15.202:6833","nonce":4154570751}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:58:02.809 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.807+0000 7f28e328d700 1 -- 172.21.15.195:0/148385463 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f28bc063d00 msgr2=0x7f28bc0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:02.809 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.807+0000 7f28e328d700 1 --2- 172.21.15.195:0/148385463 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f28bc063d00 0x7f28bc0661b0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f28cc0017e0 tx=0x7f28cc009040 comp rx=0 tx=0).stop 2022-06-27T11:58:02.810 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.807+0000 7f28e328d700 1 -- 172.21.15.195:0/148385463 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f28dc0be7a0 msgr2=0x7f28dc0c6c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:02.810 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.807+0000 7f28e328d700 1 --2- 172.21.15.195:0/148385463 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f28dc0be7a0 0x7f28dc0c6c60 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f28d4009e90 tx=0x7f28d400b040 comp rx=0 tx=0).stop 2022-06-27T11:58:02.812 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.808+0000 7f28e328d700 1 -- 172.21.15.195:0/148385463 shutdown_connections 2022-06-27T11:58:02.812 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.808+0000 7f28e328d700 1 --2- 172.21.15.195:0/148385463 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f28bc063d00 0x7f28bc0661b0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:02.812 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.808+0000 7f28e328d700 1 --2- 172.21.15.195:0/148385463 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f28dc0be7a0 0x7f28dc0c6c60 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:02.813 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.808+0000 7f28e328d700 1 --2- 172.21.15.195:0/148385463 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f28dc06e850 0x7f28dc165d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:02.813 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.808+0000 7f28e328d700 1 --2- 172.21.15.195:0/148385463 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f28dc0c34a0 0x7f28dc0c6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:02.813 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.808+0000 7f28e328d700 1 -- 172.21.15.195:0/148385463 >> 172.21.15.195:0/148385463 conn(0x7f28dc0c6350 msgr2=0x7f28dc0c8860 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:58:02.814 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.808+0000 7f28e328d700 1 -- 172.21.15.195:0/148385463 shutdown_connections 2022-06-27T11:58:02.814 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:02.808+0000 7f28e328d700 1 -- 172.21.15.195:0/148385463 wait complete. 2022-06-27T11:58:02.814 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 37 2022-06-27T11:58:02.820 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2022-06-27T11:58:04.822 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:58:04.984 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:04.957+0000 7f2f11c70700 1 Processor -- start 2022-06-27T11:58:04.986 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:04.957+0000 7f2f11c70700 1 -- start start 2022-06-27T11:58:04.987 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:04.958+0000 7f2f11c70700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f2f0c06a440 0x7f2f0c0688a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:04.987 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:04.958+0000 7f2f11c70700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2f0c1564d0 0x7f2f0c156950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:04.989 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:04.958+0000 7f2f11c70700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f0c0c4b10 con 0x7f2f0c156100 2022-06-27T11:58:04.989 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:04.958+0000 7f2f11c70700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f0c0c3dc0 con 0x7f2f0c06a440 2022-06-27T11:58:04.990 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:04.958+0000 7f2f11c70700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0c068f20 con 0x7f2f0c1564d0 2022-06-27T11:58:04.991 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:04.958+0000 7f2f0bfff700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2f0c1564d0 0x7f2f0c156950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:04.991 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:04.958+0000 7f2f0affd700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f2f0c06a440 0x7f2f0c0688a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:04.992 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:04.959+0000 7f2f0bfff700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2f0c1564d0 0x7f2f0c156950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3301/0 says I am v2:172.21.15.195:38256/0 (socket says 172.21.15.195:38256) 2022-06-27T11:58:04.992 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:04.959+0000 7f2f0bfff700 1 -- 172.21.15.195:0/2406521479 learned_addr learned my addr 172.21.15.195:0/2406521479 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:58:04.992 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:04.959+0000 7f2f0bfff700 1 -- 172.21.15.195:0/2406521479 >> v2:172.21.15.202:3300/0 conn(0x7f2f0c06a440 msgr2=0x7f2f0c0688a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:58:04.993 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:04.959+0000 7f2f0bfff700 1 --2- 172.21.15.195:0/2406521479 >> v2:172.21.15.202:3300/0 conn(0x7f2f0c06a440 0x7f2f0c0688a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:04.993 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:04.959+0000 7f2f0bfff700 1 -- 172.21.15.195:0/2406521479 >> v1:172.21.15.202:6789/0 conn(0x7f2f0c156100 legacy=0x7f2f0c0c4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:04.997 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:04.959+0000 7f2f0bfff700 1 -- 172.21.15.195:0/2406521479 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f0c0c56b0 con 0x7f2f0c1564d0 2022-06-27T11:58:04.997 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:04.959+0000 7f2f0bfff700 1 --2- 172.21.15.195:0/2406521479 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2f0c1564d0 0x7f2f0c156950 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2f04009f20 tx=0x7f2f04009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd782ac362a1ca43 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:04.998 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:04.959+0000 7f2f0a7fc700 1 -- 172.21.15.195:0/2406521479 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f0400e070 con 0x7f2f0c1564d0 2022-06-27T11:58:04.998 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:04.959+0000 7f2f0a7fc700 1 -- 172.21.15.195:0/2406521479 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f0400b9d0 con 0x7f2f0c1564d0 2022-06-27T11:58:04.998 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:04.959+0000 7f2f0a7fc700 1 -- 172.21.15.195:0/2406521479 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f040092e0 con 0x7f2f0c1564d0 2022-06-27T11:58:04.999 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:04.960+0000 7f2f11c70700 1 -- 172.21.15.195:0/2406521479 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2f0c1564d0 msgr2=0x7f2f0c156950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:04.999 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:04.960+0000 7f2f11c70700 1 --2- 172.21.15.195:0/2406521479 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2f0c1564d0 0x7f2f0c156950 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2f04009f20 tx=0x7f2f04009840 comp rx=0 tx=0).stop 2022-06-27T11:58:05.000 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:04.960+0000 7f2f11c70700 1 -- 172.21.15.195:0/2406521479 shutdown_connections 2022-06-27T11:58:05.000 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:04.960+0000 7f2f11c70700 1 --2- 172.21.15.195:0/2406521479 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2f0c1564d0 0x7f2f0c156950 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:05.001 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:04.960+0000 7f2f11c70700 1 --2- 172.21.15.195:0/2406521479 >> v2:172.21.15.202:3300/0 conn(0x7f2f0c06a440 0x7f2f0c0688a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:05.001 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:04.960+0000 7f2f11c70700 1 -- 172.21.15.195:0/2406521479 >> 172.21.15.195:0/2406521479 conn(0x7f2f0c0c9b00 msgr2=0x7f2f0c0c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:58:05.001 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:04.960+0000 7f2f11c70700 1 -- 172.21.15.195:0/2406521479 shutdown_connections 2022-06-27T11:58:05.007 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:04.960+0000 7f2f11c70700 1 -- 172.21.15.195:0/2406521479 wait complete. 2022-06-27T11:58:05.008 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:04.961+0000 7f2f11c70700 1 Processor -- start 2022-06-27T11:58:05.008 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:04.961+0000 7f2f11c70700 1 -- start start 2022-06-27T11:58:05.008 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:04.962+0000 7f2f11c70700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2f0c06a440 0x7f2f0c148eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:05.009 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:04.962+0000 7f2f11c70700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2f0c156100 0x7f2f0c0ba570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:05.009 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:04.962+0000 7f2f11c70700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2f0c1564d0 0x7f2f0c0baab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:05.010 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:04.962+0000 7f2f11c70700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0c147b60 con 0x7f2f0c1564d0 2022-06-27T11:58:05.010 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:04.962+0000 7f2f11c70700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0c167280 con 0x7f2f0c06a440 2022-06-27T11:58:05.011 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:04.962+0000 7f2f11c70700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0c1663f0 con 0x7f2f0c156100 2022-06-27T11:58:05.011 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:04.962+0000 7f2f0bfff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2f0c1564d0 0x7f2f0c0baab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:05.012 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:04.962+0000 7f2f0bfff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2f0c1564d0 0x7f2f0c0baab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45506/0 (socket says 172.21.15.195:45506) 2022-06-27T11:58:05.012 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:04.962+0000 7f2f0affd700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2f0c156100 0x7f2f0c0ba570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:05.012 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:04.962+0000 7f2f0bfff700 1 -- 172.21.15.195:0/2571406255 learned_addr learned my addr 172.21.15.195:0/2571406255 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:58:05.013 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:04.962+0000 7f2f0b7fe700 1 --2- 172.21.15.195:0/2571406255 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2f0c06a440 0x7f2f0c148eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:05.013 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:04.962+0000 7f2f0bfff700 1 -- 172.21.15.195:0/2571406255 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2f0c156100 msgr2=0x7f2f0c0ba570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:05.013 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:04.962+0000 7f2f0bfff700 1 --2- 172.21.15.195:0/2571406255 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2f0c156100 0x7f2f0c0ba570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:05.013 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:04.962+0000 7f2f0bfff700 1 -- 172.21.15.195:0/2571406255 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2f0c06a440 msgr2=0x7f2f0c148eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:05.014 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:04.962+0000 7f2f0bfff700 1 --2- 172.21.15.195:0/2571406255 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2f0c06a440 0x7f2f0c148eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:05.014 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:04.962+0000 7f2f0bfff700 1 -- 172.21.15.195:0/2571406255 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f04009ae0 con 0x7f2f0c1564d0 2022-06-27T11:58:05.014 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:04.962+0000 7f2f0bfff700 1 --2- 172.21.15.195:0/2571406255 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2f0c1564d0 0x7f2f0c0baab0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f2f04003bd0 tx=0x7f2f04018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:05.014 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:04.963+0000 7f2ef37fe700 1 -- 172.21.15.195:0/2571406255 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f04004040 con 0x7f2f0c1564d0 2022-06-27T11:58:05.015 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:04.963+0000 7f2ef37fe700 1 -- 172.21.15.195:0/2571406255 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f040041a0 con 0x7f2f0c1564d0 2022-06-27T11:58:05.015 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:04.963+0000 7f2f11c70700 1 -- 172.21.15.195:0/2571406255 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f0c0bbe80 con 0x7f2f0c1564d0 2022-06-27T11:58:05.015 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:04.963+0000 7f2ef37fe700 1 -- 172.21.15.195:0/2571406255 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f04004560 con 0x7f2f0c1564d0 2022-06-27T11:58:05.016 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:04.963+0000 7f2f11c70700 1 -- 172.21.15.195:0/2571406255 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f0c166950 con 0x7f2f0c1564d0 2022-06-27T11:58:05.016 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:04.963+0000 7f2f11c70700 1 -- 172.21.15.195:0/2571406255 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f0c0722c0 con 0x7f2f0c1564d0 2022-06-27T11:58:05.016 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:04.964+0000 7f2ef37fe700 1 -- 172.21.15.195:0/2571406255 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f2f04028460 con 0x7f2f0c1564d0 2022-06-27T11:58:05.017 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:04.964+0000 7f2ef37fe700 1 --2- 172.21.15.195:0/2571406255 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2eec063d00 0x7f2eec0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:05.017 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:04.964+0000 7f2ef37fe700 1 -- 172.21.15.195:0/2571406255 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6546+0+0 (secure 0 0 0) 0x7f2f04082690 con 0x7f2f0c1564d0 2022-06-27T11:58:05.018 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:04.964+0000 7f2f0b7fe700 1 --2- 172.21.15.195:0/2571406255 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2eec063d00 0x7f2eec0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:05.018 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:04.965+0000 7f2f0b7fe700 1 --2- 172.21.15.195:0/2571406255 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2eec063d00 0x7f2eec0661b0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f2f00000c00 tx=0x7f2f00009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:05.019 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:04.969+0000 7f2ef37fe700 1 -- 172.21.15.195:0/2571406255 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f2f04050f40 con 0x7f2f0c1564d0 2022-06-27T11:58:05.197 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:05.198+0000 7f2f11c70700 1 -- 172.21.15.195:0/2571406255 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2f0c0c6360 con 0x7f2f0c1564d0 2022-06-27T11:58:05.203 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:05.198+0000 7f2ef37fe700 1 -- 172.21.15.195:0/2571406255 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+6128 (secure 0 0 0) 0x7f2f04054560 con 0x7f2f0c1564d0 2022-06-27T11:58:05.205 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:58:05.206 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":5392,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/2089155519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2089155519},{"type":"v1","addr":"172.21.15.202:6835","nonce":2089155519}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:57:31.004761+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4976},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4976":{"gid":4976,"name":"b","rank":0,"incarnation":32,"state":"up:active","state_seq":28,"addr":"172.21.15.202:6833/4154570751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":4154570751},{"type":"v1","addr":"172.21.15.202:6833","nonce":4154570751}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:58:05.209 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:05.200+0000 7f2f11c70700 1 -- 172.21.15.195:0/2571406255 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2eec063d00 msgr2=0x7f2eec0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:05.209 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:05.200+0000 7f2f11c70700 1 --2- 172.21.15.195:0/2571406255 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2eec063d00 0x7f2eec0661b0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f2f00000c00 tx=0x7f2f00009040 comp rx=0 tx=0).stop 2022-06-27T11:58:05.209 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:05.200+0000 7f2f11c70700 1 -- 172.21.15.195:0/2571406255 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2f0c1564d0 msgr2=0x7f2f0c0baab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:05.209 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:05.200+0000 7f2f11c70700 1 --2- 172.21.15.195:0/2571406255 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2f0c1564d0 0x7f2f0c0baab0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f2f04003bd0 tx=0x7f2f04018000 comp rx=0 tx=0).stop 2022-06-27T11:58:05.210 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:05.200+0000 7f2f11c70700 1 -- 172.21.15.195:0/2571406255 shutdown_connections 2022-06-27T11:58:05.210 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:05.200+0000 7f2f11c70700 1 --2- 172.21.15.195:0/2571406255 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2eec063d00 0x7f2eec0661b0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:05.210 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:05.200+0000 7f2f11c70700 1 --2- 172.21.15.195:0/2571406255 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2f0c1564d0 0x7f2f0c0baab0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:05.210 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:05.200+0000 7f2f11c70700 1 --2- 172.21.15.195:0/2571406255 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2f0c06a440 0x7f2f0c148eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:05.211 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:05.200+0000 7f2f11c70700 1 --2- 172.21.15.195:0/2571406255 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2f0c156100 0x7f2f0c0ba570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:05.211 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:05.200+0000 7f2f11c70700 1 -- 172.21.15.195:0/2571406255 >> 172.21.15.195:0/2571406255 conn(0x7f2f0c0c9b00 msgr2=0x7f2f0c0c0350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:58:05.212 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:05.200+0000 7f2f11c70700 1 -- 172.21.15.195:0/2571406255 shutdown_connections 2022-06-27T11:58:05.212 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:05.200+0000 7f2f11c70700 1 -- 172.21.15.195:0/2571406255 wait complete. 2022-06-27T11:58:05.212 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 37 2022-06-27T11:58:05.216 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2022-06-27T11:58:07.217 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:58:07.356 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.356+0000 7fad31234700 1 Processor -- start 2022-06-27T11:58:07.357 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.356+0000 7fad31234700 1 -- start start 2022-06-27T11:58:07.358 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.357+0000 7fad31234700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fad2c0c3f50 0x7fad2c15ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:07.359 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.357+0000 7fad31234700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad2c0c33b0 con 0x7fad2c14e860 2022-06-27T11:58:07.359 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.357+0000 7fad31234700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad2c06f220 con 0x7fad2c14e490 2022-06-27T11:58:07.359 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.357+0000 7fad31234700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad2c1542c0 con 0x7fad2c0c3f50 2022-06-27T11:58:07.360 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.357+0000 7fad2a59c700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fad2c14e860 0x7fad2c0c3840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:51704/0 (socket says 172.21.15.195:51704) 2022-06-27T11:58:07.360 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.357+0000 7fad2a59c700 1 -- 172.21.15.195:0/3603801769 learned_addr learned my addr 172.21.15.195:0/3603801769 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:58:07.360 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.358+0000 7fad29d9b700 1 -- 172.21.15.195:0/3603801769 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 829864297 0 0) 0x7fad2c0c33b0 con 0x7fad2c14e860 2022-06-27T11:58:07.361 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.358+0000 7fad29d9b700 1 -- 172.21.15.195:0/3603801769 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad0c003430 con 0x7fad2c14e860 2022-06-27T11:58:07.361 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.358+0000 7fad2b59e700 1 --2- 172.21.15.195:0/3603801769 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fad2c0c3f50 0x7fad2c15ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:07.362 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.358+0000 7fad29d9b700 1 -- 172.21.15.195:0/3603801769 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 786622054 0 0) 0x7fad0c003430 con 0x7fad2c14e860 2022-06-27T11:58:07.362 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.358+0000 7fad29d9b700 1 -- 172.21.15.195:0/3603801769 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fad2c0c3f50 msgr2=0x7fad2c15ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:58:07.373 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.358+0000 7fad29d9b700 1 --2- 172.21.15.195:0/3603801769 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fad2c0c3f50 0x7fad2c15ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:07.374 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.358+0000 7fad29d9b700 1 -- 172.21.15.195:0/3603801769 >> v1:172.21.15.202:6789/0 conn(0x7fad2c14e490 legacy=0x7fad2c06cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:07.375 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.358+0000 7fad29d9b700 1 -- 172.21.15.195:0/3603801769 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad2c154560 con 0x7fad2c14e860 2022-06-27T11:58:07.375 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.358+0000 7fad29d9b700 1 -- 172.21.15.195:0/3603801769 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7fad1c001470 con 0x7fad2c14e860 2022-06-27T11:58:07.376 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.358+0000 7fad29d9b700 1 -- 172.21.15.195:0/3603801769 >> v1:172.21.15.195:6789/0 conn(0x7fad2c14e860 legacy=0x7fad2c0c3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:07.376 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.358+0000 7fad29d9b700 1 --2- 172.21.15.195:0/3603801769 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fad0c003950 0x7fad0c003d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:07.376 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.358+0000 7fad29d9b700 1 --2- 172.21.15.195:0/3603801769 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fad0c0046d0 0x7fad0c004b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:07.377 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.358+0000 7fad29d9b700 1 --2- 172.21.15.195:0/3603801769 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fad2c0c3f50 0x7fad0c0050a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:07.377 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.358+0000 7fad29d9b700 1 -- 172.21.15.195:0/3603801769 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad2c1542c0 con 0x7fad0c003950 2022-06-27T11:58:07.377 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.358+0000 7fad29d9b700 1 -- 172.21.15.195:0/3603801769 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad0c00ba20 con 0x7fad0c0046d0 2022-06-27T11:58:07.377 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.358+0000 7fad29d9b700 1 -- 172.21.15.195:0/3603801769 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad0c00bbe0 con 0x7fad2c0c3f50 2022-06-27T11:58:07.378 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.358+0000 7fad2ad9d700 1 --2- 172.21.15.195:0/3603801769 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fad0c003950 0x7fad0c003d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:07.378 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.358+0000 7fad2b59e700 1 --2- 172.21.15.195:0/3603801769 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fad2c0c3f50 0x7fad0c0050a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:07.378 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.358+0000 7fad2a59c700 1 --2- 172.21.15.195:0/3603801769 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fad0c0046d0 0x7fad0c004b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:07.378 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.358+0000 7fad2ad9d700 1 -- 172.21.15.195:0/3603801769 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fad2c0c3f50 msgr2=0x7fad0c0050a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:58:07.379 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.358+0000 7fad2ad9d700 1 --2- 172.21.15.195:0/3603801769 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fad2c0c3f50 0x7fad0c0050a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:07.379 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.358+0000 7fad2ad9d700 1 -- 172.21.15.195:0/3603801769 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fad0c0046d0 msgr2=0x7fad0c004b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:58:07.380 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.358+0000 7fad2ad9d700 1 --2- 172.21.15.195:0/3603801769 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fad0c0046d0 0x7fad0c004b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:07.381 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.358+0000 7fad2ad9d700 1 -- 172.21.15.195:0/3603801769 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad0c003430 con 0x7fad0c003950 2022-06-27T11:58:07.381 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.359+0000 7fad2ad9d700 1 --2- 172.21.15.195:0/3603801769 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fad0c003950 0x7fad0c003d60 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fad14001760 tx=0x7fad14007bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f30211b44dc5ebf server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:07.382 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.359+0000 7fad29d9b700 1 -- 172.21.15.195:0/3603801769 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad140050d0 con 0x7fad0c003950 2022-06-27T11:58:07.383 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.359+0000 7fad29d9b700 1 -- 172.21.15.195:0/3603801769 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad14005230 con 0x7fad0c003950 2022-06-27T11:58:07.383 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.359+0000 7fad29d9b700 1 -- 172.21.15.195:0/3603801769 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad14003750 con 0x7fad0c003950 2022-06-27T11:58:07.384 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.359+0000 7fad31234700 1 -- 172.21.15.195:0/3603801769 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fad0c003950 msgr2=0x7fad0c003d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:07.384 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.359+0000 7fad31234700 1 --2- 172.21.15.195:0/3603801769 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fad0c003950 0x7fad0c003d60 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fad14001760 tx=0x7fad14007bb0 comp rx=0 tx=0).stop 2022-06-27T11:58:07.385 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.359+0000 7fad2b59e700 1 -- 172.21.15.195:0/3603801769 reap_dead start 2022-06-27T11:58:07.385 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.359+0000 7fad31234700 1 -- 172.21.15.195:0/3603801769 shutdown_connections 2022-06-27T11:58:07.387 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.359+0000 7fad31234700 1 -- 172.21.15.195:0/3603801769 >> 172.21.15.195:0/3603801769 conn(0x7fad2c063af0 msgr2=0x7fad2c063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:58:07.387 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.359+0000 7fad31234700 1 -- 172.21.15.195:0/3603801769 shutdown_connections 2022-06-27T11:58:07.389 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.359+0000 7fad31234700 1 -- 172.21.15.195:0/3603801769 wait complete. 2022-06-27T11:58:07.389 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.361+0000 7fad31234700 1 Processor -- start 2022-06-27T11:58:07.389 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.361+0000 7fad31234700 1 -- start start 2022-06-27T11:58:07.390 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.361+0000 7fad31234700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fad0c003950 0x7fad2c166ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:07.390 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.361+0000 7fad31234700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fad2c06ac40 0x7fad2c150fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:07.390 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.361+0000 7fad31234700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fad2c157000 0x7fad2c0c7850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:07.390 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.361+0000 7fad31234700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad2c06a9f0 con 0x7fad2c157000 2022-06-27T11:58:07.391 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.361+0000 7fad31234700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad2c069f40 con 0x7fad2c06ac40 2022-06-27T11:58:07.391 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.361+0000 7fad31234700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad2c0c88b0 con 0x7fad0c003950 2022-06-27T11:58:07.391 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.361+0000 7fad2b59e700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fad2c157000 0x7fad2c0c7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:07.391 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.361+0000 7fad2b59e700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fad2c157000 0x7fad2c0c7850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45532/0 (socket says 172.21.15.195:45532) 2022-06-27T11:58:07.392 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.361+0000 7fad2b59e700 1 -- 172.21.15.195:0/1605100426 learned_addr learned my addr 172.21.15.195:0/1605100426 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:58:07.392 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.362+0000 7fad2a59c700 1 --2- 172.21.15.195:0/1605100426 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fad2c06ac40 0x7fad2c150fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:07.392 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.362+0000 7fad2ad9d700 1 --2- 172.21.15.195:0/1605100426 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fad0c003950 0x7fad2c166ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:07.392 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.362+0000 7fad2b59e700 1 -- 172.21.15.195:0/1605100426 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fad0c003950 msgr2=0x7fad2c166ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:07.393 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.362+0000 7fad2b59e700 1 --2- 172.21.15.195:0/1605100426 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fad0c003950 0x7fad2c166ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:07.393 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.362+0000 7fad2b59e700 1 -- 172.21.15.195:0/1605100426 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fad2c06ac40 msgr2=0x7fad2c150fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:07.393 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.362+0000 7fad2b59e700 1 --2- 172.21.15.195:0/1605100426 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fad2c06ac40 0x7fad2c150fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:07.393 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.362+0000 7fad2b59e700 1 -- 172.21.15.195:0/1605100426 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad2c06a710 con 0x7fad2c157000 2022-06-27T11:58:07.394 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.362+0000 7fad2b59e700 1 --2- 172.21.15.195:0/1605100426 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fad2c157000 0x7fad2c0c7850 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fad240094d0 tx=0x7fad2400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:07.394 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.362+0000 7fad22ffd700 1 -- 172.21.15.195:0/1605100426 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad2400e040 con 0x7fad2c157000 2022-06-27T11:58:07.395 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.362+0000 7fad22ffd700 1 -- 172.21.15.195:0/1605100426 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad2400f040 con 0x7fad2c157000 2022-06-27T11:58:07.395 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.362+0000 7fad31234700 1 -- 172.21.15.195:0/1605100426 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad2c069180 con 0x7fad2c157000 2022-06-27T11:58:07.395 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.362+0000 7fad22ffd700 1 -- 172.21.15.195:0/1605100426 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad2400fe50 con 0x7fad2c157000 2022-06-27T11:58:07.395 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.362+0000 7fad31234700 1 -- 172.21.15.195:0/1605100426 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad2c069740 con 0x7fad2c157000 2022-06-27T11:58:07.396 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.363+0000 7fad31234700 1 -- 172.21.15.195:0/1605100426 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad2c0722c0 con 0x7fad2c157000 2022-06-27T11:58:07.396 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.363+0000 7fad22ffd700 1 -- 172.21.15.195:0/1605100426 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fad24003ab0 con 0x7fad2c157000 2022-06-27T11:58:07.396 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.364+0000 7fad22ffd700 1 --2- 172.21.15.195:0/1605100426 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fad10063db0 0x7fad10066260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:07.396 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.364+0000 7fad22ffd700 1 -- 172.21.15.195:0/1605100426 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6546+0+0 (secure 0 0 0) 0x7fad240811e0 con 0x7fad2c157000 2022-06-27T11:58:07.397 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.364+0000 7fad2ad9d700 1 --2- 172.21.15.195:0/1605100426 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fad10063db0 0x7fad10066260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:07.397 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.364+0000 7fad2ad9d700 1 --2- 172.21.15.195:0/1605100426 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fad10063db0 0x7fad10066260 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fad14000ff0 tx=0x7fad14015000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:07.397 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.368+0000 7fad22ffd700 1 -- 172.21.15.195:0/1605100426 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fad2404faf0 con 0x7fad2c157000 2022-06-27T11:58:07.597 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.592+0000 7fad31234700 1 -- 172.21.15.195:0/1605100426 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fad2c1586d0 con 0x7fad2c157000 2022-06-27T11:58:07.598 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.592+0000 7fad22ffd700 1 -- 172.21.15.195:0/1605100426 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+6128 (secure 0 0 0) 0x7fad24053110 con 0x7fad2c157000 2022-06-27T11:58:07.599 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:58:07.599 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":5392,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/2089155519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2089155519},{"type":"v1","addr":"172.21.15.202:6835","nonce":2089155519}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:57:31.004761+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4976},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4976":{"gid":4976,"name":"b","rank":0,"incarnation":32,"state":"up:active","state_seq":28,"addr":"172.21.15.202:6833/4154570751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":4154570751},{"type":"v1","addr":"172.21.15.202:6833","nonce":4154570751}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:58:07.609 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.594+0000 7fad31234700 1 -- 172.21.15.195:0/1605100426 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fad10063db0 msgr2=0x7fad10066260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:07.610 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.594+0000 7fad31234700 1 --2- 172.21.15.195:0/1605100426 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fad10063db0 0x7fad10066260 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fad14000ff0 tx=0x7fad14015000 comp rx=0 tx=0).stop 2022-06-27T11:58:07.611 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.594+0000 7fad31234700 1 -- 172.21.15.195:0/1605100426 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fad2c157000 msgr2=0x7fad2c0c7850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:07.611 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.594+0000 7fad31234700 1 --2- 172.21.15.195:0/1605100426 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fad2c157000 0x7fad2c0c7850 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fad240094d0 tx=0x7fad2400b040 comp rx=0 tx=0).stop 2022-06-27T11:58:07.611 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.594+0000 7fad31234700 1 -- 172.21.15.195:0/1605100426 shutdown_connections 2022-06-27T11:58:07.612 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.594+0000 7fad31234700 1 --2- 172.21.15.195:0/1605100426 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fad10063db0 0x7fad10066260 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:07.612 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.594+0000 7fad31234700 1 --2- 172.21.15.195:0/1605100426 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fad2c157000 0x7fad2c0c7850 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:07.612 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.595+0000 7fad31234700 1 --2- 172.21.15.195:0/1605100426 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fad0c003950 0x7fad2c166ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:07.614 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.595+0000 7fad31234700 1 --2- 172.21.15.195:0/1605100426 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fad2c06ac40 0x7fad2c150fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:07.615 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.595+0000 7fad31234700 1 -- 172.21.15.195:0/1605100426 >> 172.21.15.195:0/1605100426 conn(0x7fad2c063af0 msgr2=0x7fad2c06cfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:58:07.615 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.595+0000 7fad31234700 1 -- 172.21.15.195:0/1605100426 shutdown_connections 2022-06-27T11:58:07.616 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:07.595+0000 7fad31234700 1 -- 172.21.15.195:0/1605100426 wait complete. 2022-06-27T11:58:07.616 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 37 2022-06-27T11:58:07.617 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2022-06-27T11:58:09.619 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:58:09.771 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.753+0000 7f271190f700 1 Processor -- start 2022-06-27T11:58:09.772 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.753+0000 7f271190f700 1 -- start start 2022-06-27T11:58:09.772 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.754+0000 7f271190f700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f270c06f640 0x7f270c156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:09.772 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.754+0000 7f271190f700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f270c0c1a20 con 0x7f270c0c39e0 2022-06-27T11:58:09.773 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.754+0000 7f271190f700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f270c0c1130 con 0x7f270c0c4160 2022-06-27T11:58:09.773 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.754+0000 7f271190f700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f270c06ea90 con 0x7f270c06f640 2022-06-27T11:58:09.773 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.754+0000 7f270b7fe700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f270c0c39e0 0x7f270c1535a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:51726/0 (socket says 172.21.15.195:51726) 2022-06-27T11:58:09.773 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.754+0000 7f270b7fe700 1 -- 172.21.15.195:0/1195187937 learned_addr learned my addr 172.21.15.195:0/1195187937 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:58:09.774 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.755+0000 7f270affd700 1 --2- 172.21.15.195:0/1195187937 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f270c06f640 0x7f270c156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:09.774 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.755+0000 7f2709ffb700 1 -- 172.21.15.195:0/1195187937 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3196145632 0 0) 0x7f270c0c1a20 con 0x7f270c0c39e0 2022-06-27T11:58:09.774 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.755+0000 7f2709ffb700 1 -- 172.21.15.195:0/1195187937 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26f8003430 con 0x7f270c0c39e0 2022-06-27T11:58:09.774 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.755+0000 7f2709ffb700 1 -- 172.21.15.195:0/1195187937 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2850340876 0 0) 0x7f270c0c1130 con 0x7f270c0c4160 2022-06-27T11:58:09.775 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.755+0000 7f2709ffb700 1 -- 172.21.15.195:0/1195187937 --> v1:172.21.15.202:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f270c0c1a20 con 0x7f270c0c4160 2022-06-27T11:58:09.775 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.755+0000 7f2709ffb700 1 -- 172.21.15.195:0/1195187937 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1752726111 0 0) 0x7f26f8003430 con 0x7f270c0c39e0 2022-06-27T11:58:09.775 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.755+0000 7f2709ffb700 1 -- 172.21.15.195:0/1195187937 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f270c06f640 msgr2=0x7f270c156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:58:09.775 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.755+0000 7f2709ffb700 1 --2- 172.21.15.195:0/1195187937 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f270c06f640 0x7f270c156ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:09.775 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.755+0000 7f2709ffb700 1 -- 172.21.15.195:0/1195187937 >> v1:172.21.15.202:6789/0 conn(0x7f270c0c4160 legacy=0x7f270c0c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:09.776 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.755+0000 7f2709ffb700 1 -- 172.21.15.195:0/1195187937 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f270c06ed70 con 0x7f270c0c39e0 2022-06-27T11:58:09.776 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.755+0000 7f2709ffb700 1 -- 172.21.15.195:0/1195187937 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7f27040017f0 con 0x7f270c0c39e0 2022-06-27T11:58:09.776 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.755+0000 7f2709ffb700 1 -- 172.21.15.195:0/1195187937 >> v1:172.21.15.195:6789/0 conn(0x7f270c0c39e0 legacy=0x7f270c1535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:09.776 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.755+0000 7f2709ffb700 1 --2- 172.21.15.195:0/1195187937 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f26f8003ae0 0x7f26f8003ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:09.777 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.755+0000 7f2709ffb700 1 --2- 172.21.15.195:0/1195187937 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f26f80046d0 0x7f26f8004b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:09.777 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.755+0000 7f2709ffb700 1 --2- 172.21.15.195:0/1195187937 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f26f8005080 0x7f26f800b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:09.777 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.755+0000 7f2709ffb700 1 -- 172.21.15.195:0/1195187937 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f270c06ea90 con 0x7f26f8005080 2022-06-27T11:58:09.777 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.755+0000 7f2709ffb700 1 -- 172.21.15.195:0/1195187937 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f800bee0 con 0x7f26f8003ae0 2022-06-27T11:58:09.778 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.755+0000 7f2709ffb700 1 -- 172.21.15.195:0/1195187937 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26f800c0a0 con 0x7f26f80046d0 2022-06-27T11:58:09.778 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.755+0000 7f270affd700 1 --2- 172.21.15.195:0/1195187937 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f270c06f640 0x7f270c156ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:58:09.778 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.755+0000 7f270b7fe700 1 --2- 172.21.15.195:0/1195187937 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f26f8005080 0x7f26f800b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:09.778 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.755+0000 7f270a7fc700 1 --2- 172.21.15.195:0/1195187937 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f26f80046d0 0x7f26f8004b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:09.780 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.755+0000 7f270affd700 1 --2- 172.21.15.195:0/1195187937 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f26f8003ae0 0x7f26f8003ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:09.780 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.756+0000 7f270b7fe700 1 -- 172.21.15.195:0/1195187937 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f26f80046d0 msgr2=0x7f26f8004b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:58:09.781 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.756+0000 7f270b7fe700 1 --2- 172.21.15.195:0/1195187937 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f26f80046d0 0x7f26f8004b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:09.781 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.756+0000 7f270b7fe700 1 -- 172.21.15.195:0/1195187937 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f26f8003ae0 msgr2=0x7f26f8003ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:58:09.782 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.756+0000 7f270b7fe700 1 --2- 172.21.15.195:0/1195187937 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f26f8003ae0 0x7f26f8003ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:09.782 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.756+0000 7f270b7fe700 1 -- 172.21.15.195:0/1195187937 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26f8003430 con 0x7f26f8005080 2022-06-27T11:58:09.783 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.756+0000 7f270b7fe700 1 --2- 172.21.15.195:0/1195187937 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f26f8005080 0x7f26f800b920 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f270400c5c0 tx=0x7f2704007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70d1144999d335b6 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:09.783 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.756+0000 7f2709ffb700 1 -- 172.21.15.195:0/1195187937 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f270400ce90 con 0x7f26f8005080 2022-06-27T11:58:09.784 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.756+0000 7f2709ffb700 1 -- 172.21.15.195:0/1195187937 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2704005b60 con 0x7f26f8005080 2022-06-27T11:58:09.784 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.756+0000 7f2709ffb700 1 -- 172.21.15.195:0/1195187937 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27040070d0 con 0x7f26f8005080 2022-06-27T11:58:09.784 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.756+0000 7f271190f700 1 -- 172.21.15.195:0/1195187937 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f26f8005080 msgr2=0x7f26f800b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:09.785 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.756+0000 7f271190f700 1 --2- 172.21.15.195:0/1195187937 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f26f8005080 0x7f26f800b920 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f270400c5c0 tx=0x7f2704007f60 comp rx=0 tx=0).stop 2022-06-27T11:58:09.785 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.756+0000 7f270b7fe700 1 -- 172.21.15.195:0/1195187937 reap_dead start 2022-06-27T11:58:09.786 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.756+0000 7f271190f700 1 -- 172.21.15.195:0/1195187937 shutdown_connections 2022-06-27T11:58:09.786 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.756+0000 7f271190f700 1 -- 172.21.15.195:0/1195187937 >> 172.21.15.195:0/1195187937 conn(0x7f270c0c6350 msgr2=0x7f270c0c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:58:09.787 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.756+0000 7f271190f700 1 -- 172.21.15.195:0/1195187937 shutdown_connections 2022-06-27T11:58:09.787 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.756+0000 7f271190f700 1 -- 172.21.15.195:0/1195187937 wait complete. 2022-06-27T11:58:09.788 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.758+0000 7f271190f700 1 Processor -- start 2022-06-27T11:58:09.788 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.758+0000 7f271190f700 1 -- start start 2022-06-27T11:58:09.789 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.758+0000 7f271190f700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f26f8005080 0x7f270c14fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:09.790 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.758+0000 7f271190f700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f270c157fd0 0x7f270c156830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:09.790 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.758+0000 7f271190f700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f270c14b080 0x7f270c14b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:09.791 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.758+0000 7f271190f700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f270c1554e0 con 0x7f270c14b080 2022-06-27T11:58:09.791 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.759+0000 7f271190f700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f270c154050 con 0x7f26f8005080 2022-06-27T11:58:09.792 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.759+0000 7f271190f700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f270c14ba30 con 0x7f270c157fd0 2022-06-27T11:58:09.792 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.759+0000 7f270b7fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f270c14b080 0x7f270c14b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:09.793 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.759+0000 7f270b7fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f270c14b080 0x7f270c14b4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45552/0 (socket says 172.21.15.195:45552) 2022-06-27T11:58:09.793 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.759+0000 7f270b7fe700 1 -- 172.21.15.195:0/983259035 learned_addr learned my addr 172.21.15.195:0/983259035 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:58:09.794 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.759+0000 7f270affd700 1 --2- 172.21.15.195:0/983259035 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f26f8005080 0x7f270c14fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:09.794 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.759+0000 7f270a7fc700 1 --2- 172.21.15.195:0/983259035 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f270c157fd0 0x7f270c156830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:09.795 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.759+0000 7f270b7fe700 1 -- 172.21.15.195:0/983259035 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f270c157fd0 msgr2=0x7f270c156830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:09.795 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.759+0000 7f270b7fe700 1 --2- 172.21.15.195:0/983259035 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f270c157fd0 0x7f270c156830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:09.796 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.759+0000 7f270b7fe700 1 -- 172.21.15.195:0/983259035 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f26f8005080 msgr2=0x7f270c14fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:09.796 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.759+0000 7f270b7fe700 1 --2- 172.21.15.195:0/983259035 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f26f8005080 0x7f270c14fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:09.797 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.759+0000 7f270b7fe700 1 -- 172.21.15.195:0/983259035 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f270c06fcb0 con 0x7f270c14b080 2022-06-27T11:58:09.797 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.759+0000 7f270b7fe700 1 --2- 172.21.15.195:0/983259035 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f270c14b080 0x7f270c14b4f0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f270400cbb0 tx=0x7f2704015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:09.798 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.759+0000 7f26f2ffd700 1 -- 172.21.15.195:0/983259035 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f270400ce90 con 0x7f270c14b080 2022-06-27T11:58:09.798 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.759+0000 7f26f2ffd700 1 -- 172.21.15.195:0/983259035 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2704001a10 con 0x7f270c14b080 2022-06-27T11:58:09.799 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.759+0000 7f271190f700 1 -- 172.21.15.195:0/983259035 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f270c169d30 con 0x7f270c14b080 2022-06-27T11:58:09.799 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.759+0000 7f26f2ffd700 1 -- 172.21.15.195:0/983259035 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2704012400 con 0x7f270c14b080 2022-06-27T11:58:09.801 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.759+0000 7f271190f700 1 -- 172.21.15.195:0/983259035 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f270c16a2f0 con 0x7f270c14b080 2022-06-27T11:58:09.801 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.760+0000 7f271190f700 1 -- 172.21.15.195:0/983259035 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f270c154200 con 0x7f270c14b080 2022-06-27T11:58:09.802 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.761+0000 7f26f2ffd700 1 -- 172.21.15.195:0/983259035 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f2704026030 con 0x7f270c14b080 2022-06-27T11:58:09.802 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.761+0000 7f26f2ffd700 1 --2- 172.21.15.195:0/983259035 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f26ec063d30 0x7f26ec0661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:09.803 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.761+0000 7f26f2ffd700 1 -- 172.21.15.195:0/983259035 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6546+0+0 (secure 0 0 0) 0x7f2704081880 con 0x7f270c14b080 2022-06-27T11:58:09.803 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.761+0000 7f270affd700 1 --2- 172.21.15.195:0/983259035 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f26ec063d30 0x7f26ec0661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:09.804 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.762+0000 7f270affd700 1 --2- 172.21.15.195:0/983259035 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f26ec063d30 0x7f26ec0661e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f270c06db30 tx=0x7f2700008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:09.804 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.766+0000 7f26f2ffd700 1 -- 172.21.15.195:0/983259035 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f270c154200 con 0x7f270c14b080 2022-06-27T11:58:09.992 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.992+0000 7f271190f700 1 -- 172.21.15.195:0/983259035 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f270c154200 con 0x7f270c14b080 2022-06-27T11:58:09.993 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.993+0000 7f26f2ffd700 1 -- 172.21.15.195:0/983259035 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+5367 (secure 0 0 0) 0x7f270c154200 con 0x7f270c14b080 2022-06-27T11:58:09.993 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:58:09.994 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:58:09.974633+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5392},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5392":{"gid":5392,"name":"d","rank":0,"incarnation":38,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6835/2089155519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2089155519},{"type":"v1","addr":"172.21.15.202:6835","nonce":2089155519}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:58:09.995 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.996+0000 7f271190f700 1 -- 172.21.15.195:0/983259035 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f26ec063d30 msgr2=0x7f26ec0661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:09.995 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.996+0000 7f271190f700 1 --2- 172.21.15.195:0/983259035 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f26ec063d30 0x7f26ec0661e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f270c06db30 tx=0x7f2700008040 comp rx=0 tx=0).stop 2022-06-27T11:58:09.996 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.996+0000 7f271190f700 1 -- 172.21.15.195:0/983259035 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f270c14b080 msgr2=0x7f270c14b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:09.996 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.996+0000 7f271190f700 1 --2- 172.21.15.195:0/983259035 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f270c14b080 0x7f270c14b4f0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f270400cbb0 tx=0x7f2704015000 comp rx=0 tx=0).stop 2022-06-27T11:58:09.996 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.996+0000 7f271190f700 1 -- 172.21.15.195:0/983259035 shutdown_connections 2022-06-27T11:58:09.997 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.996+0000 7f271190f700 1 --2- 172.21.15.195:0/983259035 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f26ec063d30 0x7f26ec0661e0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:09.997 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.996+0000 7f271190f700 1 --2- 172.21.15.195:0/983259035 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f270c14b080 0x7f270c14b4f0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:09.997 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.996+0000 7f271190f700 1 --2- 172.21.15.195:0/983259035 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f26f8005080 0x7f270c14fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:09.997 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.996+0000 7f271190f700 1 --2- 172.21.15.195:0/983259035 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f270c157fd0 0x7f270c156830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:10.001 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.996+0000 7f271190f700 1 -- 172.21.15.195:0/983259035 >> 172.21.15.195:0/983259035 conn(0x7f270c0c6350 msgr2=0x7f270c151e50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:58:10.001 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.996+0000 7f271190f700 1 -- 172.21.15.195:0/983259035 shutdown_connections 2022-06-27T11:58:10.002 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:09.996+0000 7f271190f700 1 -- 172.21.15.195:0/983259035 wait complete. 2022-06-27T11:58:10.002 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 38 2022-06-27T11:58:10.008 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2022-06-27T11:58:10.009 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2022-06-27T11:58:10.009 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:58:10.144 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.131+0000 7ff2c3f9c700 1 Processor -- start 2022-06-27T11:58:10.145 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.131+0000 7ff2c3f9c700 1 -- start start 2022-06-27T11:58:10.145 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.131+0000 7ff2c3f9c700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7ff2bc06f640 0x7ff2bc156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:10.146 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.131+0000 7ff2c3f9c700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff2bc0c4160 0x7ff2bc0c1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:10.146 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.131+0000 7ff2c3f9c700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2bc0c1a20 con 0x7ff2bc0c39e0 2022-06-27T11:58:10.147 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.131+0000 7ff2c3f9c700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2bc0c1130 con 0x7ff2bc06f640 2022-06-27T11:58:10.147 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.131+0000 7ff2c3f9c700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2bc06ea90 con 0x7ff2bc0c4160 2022-06-27T11:58:10.147 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.131+0000 7ff2c1d38700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7ff2bc06f640 0x7ff2bc156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:10.148 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.131+0000 7ff2c1537700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff2bc0c4160 0x7ff2bc0c1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:10.148 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.131+0000 7ff2c2539700 1 --1- >> v1:172.21.15.202:6789/0 conn(0x7ff2bc0c39e0 0x7ff2bc1535a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.202:6789/0 says I am v1:172.21.15.195:53608/0 (socket says 172.21.15.195:53608) 2022-06-27T11:58:10.150 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.131+0000 7ff2c2539700 1 -- 172.21.15.195:0/3607393876 learned_addr learned my addr 172.21.15.195:0/3607393876 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:58:10.150 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.132+0000 7ff2c0d36700 1 -- 172.21.15.195:0/3607393876 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1110492990 0 0) 0x7ff2bc0c1a20 con 0x7ff2bc0c39e0 2022-06-27T11:58:10.150 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.132+0000 7ff2c0d36700 1 -- 172.21.15.195:0/3607393876 --> v1:172.21.15.202:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2a4003430 con 0x7ff2bc0c39e0 2022-06-27T11:58:10.151 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.132+0000 7ff2c1537700 1 -- 172.21.15.195:0/3607393876 >> v2:172.21.15.202:3300/0 conn(0x7ff2bc06f640 msgr2=0x7ff2bc156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:58:10.151 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.132+0000 7ff2c1537700 1 --2- 172.21.15.195:0/3607393876 >> v2:172.21.15.202:3300/0 conn(0x7ff2bc06f640 0x7ff2bc156ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:10.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.132+0000 7ff2c1537700 1 -- 172.21.15.195:0/3607393876 >> v1:172.21.15.202:6789/0 conn(0x7ff2bc0c39e0 legacy=0x7ff2bc1535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:10.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.132+0000 7ff2c1537700 1 -- 172.21.15.195:0/3607393876 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2bc06ed70 con 0x7ff2bc0c4160 2022-06-27T11:58:10.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.132+0000 7ff2c1d38700 1 --2- 172.21.15.195:0/3607393876 >> v2:172.21.15.202:3300/0 conn(0x7ff2bc06f640 0x7ff2bc156ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:58:10.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.132+0000 7ff2c1537700 1 --2- 172.21.15.195:0/3607393876 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff2bc0c4160 0x7ff2bc0c1eb0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff2ac0097a0 tx=0x7ff2ac009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21b8bc14c7a58150 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:10.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.132+0000 7ff2c0d36700 1 -- 172.21.15.195:0/3607393876 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2ac011630 con 0x7ff2bc0c4160 2022-06-27T11:58:10.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.132+0000 7ff2c0d36700 1 -- 172.21.15.195:0/3607393876 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2ac011790 con 0x7ff2bc0c4160 2022-06-27T11:58:10.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.132+0000 7ff2c0d36700 1 -- 172.21.15.195:0/3607393876 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2ac011b30 con 0x7ff2bc0c4160 2022-06-27T11:58:10.155 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.134+0000 7ff2c3f9c700 1 -- 172.21.15.195:0/3607393876 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff2bc0c4160 msgr2=0x7ff2bc0c1eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:10.155 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.134+0000 7ff2c3f9c700 1 --2- 172.21.15.195:0/3607393876 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff2bc0c4160 0x7ff2bc0c1eb0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff2ac0097a0 tx=0x7ff2ac009060 comp rx=0 tx=0).stop 2022-06-27T11:58:10.155 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.134+0000 7ff2c3f9c700 1 -- 172.21.15.195:0/3607393876 shutdown_connections 2022-06-27T11:58:10.156 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.134+0000 7ff2c3f9c700 1 --2- 172.21.15.195:0/3607393876 >> v2:172.21.15.202:3300/0 conn(0x7ff2bc06f640 0x7ff2bc156ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:10.156 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.134+0000 7ff2c3f9c700 1 --2- 172.21.15.195:0/3607393876 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff2bc0c4160 0x7ff2bc0c1eb0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:10.157 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.134+0000 7ff2c3f9c700 1 -- 172.21.15.195:0/3607393876 >> 172.21.15.195:0/3607393876 conn(0x7ff2bc0c6350 msgr2=0x7ff2bc0c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:58:10.157 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.134+0000 7ff2c3f9c700 1 -- 172.21.15.195:0/3607393876 shutdown_connections 2022-06-27T11:58:10.157 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.134+0000 7ff2c3f9c700 1 -- 172.21.15.195:0/3607393876 wait complete. 2022-06-27T11:58:10.158 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.134+0000 7ff2c3f9c700 1 Processor -- start 2022-06-27T11:58:10.158 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.134+0000 7ff2c3f9c700 1 -- start start 2022-06-27T11:58:10.159 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.134+0000 7ff2c3f9c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff2bc06f640 0x7ff2bc0c4710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:10.159 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.135+0000 7ff2c3f9c700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff2bc0c39e0 0x7ff2bc14e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:10.160 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.135+0000 7ff2c3f9c700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff2bc0c4160 0x7ff2bc14e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:10.160 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.135+0000 7ff2c3f9c700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2bc16aa70 con 0x7ff2bc06f640 2022-06-27T11:58:10.160 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.135+0000 7ff2c3f9c700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2bc0c5230 con 0x7ff2bc0c4160 2022-06-27T11:58:10.161 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.135+0000 7ff2c3f9c700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2bc169df0 con 0x7ff2bc0c39e0 2022-06-27T11:58:10.161 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.135+0000 7ff2c1d38700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff2bc06f640 0x7ff2bc0c4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:10.162 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.135+0000 7ff2c1d38700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff2bc06f640 0x7ff2bc0c4710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45562/0 (socket says 172.21.15.195:45562) 2022-06-27T11:58:10.162 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.135+0000 7ff2c1d38700 1 -- 172.21.15.195:0/4015117077 learned_addr learned my addr 172.21.15.195:0/4015117077 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:58:10.162 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.135+0000 7ff2c2539700 1 --2- 172.21.15.195:0/4015117077 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff2bc0c4160 0x7ff2bc14e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:10.163 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.135+0000 7ff2c1537700 1 --2- 172.21.15.195:0/4015117077 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff2bc0c39e0 0x7ff2bc14e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:10.163 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.135+0000 7ff2c1d38700 1 -- 172.21.15.195:0/4015117077 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff2bc0c39e0 msgr2=0x7ff2bc14e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:10.163 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.135+0000 7ff2c1d38700 1 --2- 172.21.15.195:0/4015117077 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff2bc0c39e0 0x7ff2bc14e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:10.164 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.135+0000 7ff2c1d38700 1 -- 172.21.15.195:0/4015117077 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff2bc0c4160 msgr2=0x7ff2bc14e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:10.164 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.135+0000 7ff2c1d38700 1 --2- 172.21.15.195:0/4015117077 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff2bc0c4160 0x7ff2bc14e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:10.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.135+0000 7ff2c1d38700 1 -- 172.21.15.195:0/4015117077 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2b80092e0 con 0x7ff2bc06f640 2022-06-27T11:58:10.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.135+0000 7ff2c1537700 1 --2- 172.21.15.195:0/4015117077 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff2bc0c39e0 0x7ff2bc14e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T11:58:10.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.135+0000 7ff2c2539700 1 --2- 172.21.15.195:0/4015117077 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff2bc0c4160 0x7ff2bc14e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T11:58:10.166 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.135+0000 7ff2c1d38700 1 --2- 172.21.15.195:0/4015117077 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff2bc06f640 0x7ff2bc0c4710 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7ff2b800ea70 tx=0x7ff2b8009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:10.166 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.135+0000 7ff2a9ffb700 1 -- 172.21.15.195:0/4015117077 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2b800c920 con 0x7ff2bc06f640 2022-06-27T11:58:10.167 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.135+0000 7ff2a9ffb700 1 -- 172.21.15.195:0/4015117077 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2b800ca80 con 0x7ff2bc06f640 2022-06-27T11:58:10.167 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.135+0000 7ff2c3f9c700 1 -- 172.21.15.195:0/4015117077 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2ac0092e0 con 0x7ff2bc06f640 2022-06-27T11:58:10.167 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.135+0000 7ff2a9ffb700 1 -- 172.21.15.195:0/4015117077 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2b800ce00 con 0x7ff2bc06f640 2022-06-27T11:58:10.168 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.136+0000 7ff2c3f9c700 1 -- 172.21.15.195:0/4015117077 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2bc16a3b0 con 0x7ff2bc06f640 2022-06-27T11:58:10.168 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.136+0000 7ff2c3f9c700 1 -- 172.21.15.195:0/4015117077 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2bc0722c0 con 0x7ff2bc06f640 2022-06-27T11:58:10.169 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.137+0000 7ff2a9ffb700 1 -- 172.21.15.195:0/4015117077 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7ff2b8019030 con 0x7ff2bc06f640 2022-06-27T11:58:10.169 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.137+0000 7ff2a9ffb700 1 --2- 172.21.15.195:0/4015117077 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff29c063d00 0x7ff29c0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:10.170 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.137+0000 7ff2a9ffb700 1 -- 172.21.15.195:0/4015117077 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6632+0+0 (secure 0 0 0) 0x7ff2b8080d00 con 0x7ff2bc06f640 2022-06-27T11:58:10.170 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.137+0000 7ff2c1537700 1 --2- 172.21.15.195:0/4015117077 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff29c063d00 0x7ff29c0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:10.170 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.138+0000 7ff2c1537700 1 --2- 172.21.15.195:0/4015117077 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff29c063d00 0x7ff29c0661b0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff2bc0c57f0 tx=0x7ff2ac017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:10.171 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.142+0000 7ff2a9ffb700 1 -- 172.21.15.195:0/4015117077 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7ff2b804f5c0 con 0x7ff2bc06f640 2022-06-27T11:58:10.369 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.368+0000 7ff2c3f9c700 1 -- 172.21.15.195:0/4015117077 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff2bc06f240 con 0x7ff2bc06f640 2022-06-27T11:58:10.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.368+0000 7ff2a9ffb700 1 -- 172.21.15.195:0/4015117077 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+5367 (secure 0 0 0) 0x7ff2b8021090 con 0x7ff2bc06f640 2022-06-27T11:58:10.370 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:58:10.371 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:58:09.974633+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5392},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5392":{"gid":5392,"name":"d","rank":0,"incarnation":38,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6835/2089155519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2089155519},{"type":"v1","addr":"172.21.15.202:6835","nonce":2089155519}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:58:10.373 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.372+0000 7ff2c3f9c700 1 -- 172.21.15.195:0/4015117077 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff29c063d00 msgr2=0x7ff29c0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:10.373 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.372+0000 7ff2c3f9c700 1 --2- 172.21.15.195:0/4015117077 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff29c063d00 0x7ff29c0661b0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff2bc0c57f0 tx=0x7ff2ac017000 comp rx=0 tx=0).stop 2022-06-27T11:58:10.374 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.372+0000 7ff2c3f9c700 1 -- 172.21.15.195:0/4015117077 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff2bc06f640 msgr2=0x7ff2bc0c4710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:10.374 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.372+0000 7ff2c3f9c700 1 --2- 172.21.15.195:0/4015117077 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff2bc06f640 0x7ff2bc0c4710 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7ff2b800ea70 tx=0x7ff2b8009060 comp rx=0 tx=0).stop 2022-06-27T11:58:10.374 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.372+0000 7ff2c3f9c700 1 -- 172.21.15.195:0/4015117077 shutdown_connections 2022-06-27T11:58:10.376 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.372+0000 7ff2c3f9c700 1 --2- 172.21.15.195:0/4015117077 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff29c063d00 0x7ff29c0661b0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:10.376 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.372+0000 7ff2c3f9c700 1 --2- 172.21.15.195:0/4015117077 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff2bc06f640 0x7ff2bc0c4710 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:10.376 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.372+0000 7ff2c3f9c700 1 --2- 172.21.15.195:0/4015117077 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff2bc0c4160 0x7ff2bc14e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:10.377 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.372+0000 7ff2c3f9c700 1 --2- 172.21.15.195:0/4015117077 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff2bc0c39e0 0x7ff2bc14e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:10.377 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.372+0000 7ff2c3f9c700 1 -- 172.21.15.195:0/4015117077 >> 172.21.15.195:0/4015117077 conn(0x7ff2bc0c6350 msgr2=0x7ff2bc1506e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:58:10.378 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.372+0000 7ff2c3f9c700 1 -- 172.21.15.195:0/4015117077 shutdown_connections 2022-06-27T11:58:10.378 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:10.372+0000 7ff2c3f9c700 1 -- 172.21.15.195:0/4015117077 wait complete. 2022-06-27T11:58:10.379 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 38 2022-06-27T11:58:10.385 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 38, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-06-27T11:47:52.358095+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 38, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5392": { "addr": "172.21.15.202:6835/2089155519", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6834", "nonce": 2089155519, "type": "v2" }, { "addr": "172.21.15.202:6835", "nonce": 2089155519, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5392, "incarnation": 38, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 53, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-06-27T11:58:09.974633+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5392 } } } ], "standbys": [ { "addr": "172.21.15.202:6837/3305006841", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6836", "nonce": 3305006841, "type": "v2" }, { "addr": "172.21.15.202:6837", "nonce": 3305006841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6835/2060709822", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6834", "nonce": 2060709822, "type": "v2" }, { "addr": "172.21.15.195:6835", "nonce": 2060709822, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6838/1598751652", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6836", "nonce": 1598751652, "type": "v2" }, { "addr": "172.21.15.195:6838", "nonce": 1598751652, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4275, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6839/1313800661", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6837", "nonce": 1313800661, "type": "v2" }, { "addr": "172.21.15.195:6839", "nonce": 1313800661, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4278, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-06-27T11:58:12.387 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:58:12.536 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.521+0000 7f63dd720700 1 Processor -- start 2022-06-27T11:58:12.537 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.522+0000 7f63dd720700 1 -- start start 2022-06-27T11:58:12.537 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.522+0000 7f63dd720700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f63d80c34b0 0x7f63d815fc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:12.538 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.522+0000 7f63dd720700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63d80c3980 con 0x7f63d8154310 2022-06-27T11:58:12.538 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.522+0000 7f63dd720700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63d80c3c40 con 0x7f63d8154dd0 2022-06-27T11:58:12.538 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.522+0000 7f63dd720700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63d8151dd0 con 0x7f63d80c34b0 2022-06-27T11:58:12.539 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.522+0000 7f63d77fe700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f63d80c34b0 0x7f63d815fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:12.539 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.522+0000 7f63d67fc700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f63d8154310 0x7f63d80c1dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:51762/0 (socket says 172.21.15.195:51762) 2022-06-27T11:58:12.540 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.522+0000 7f63d67fc700 1 -- 172.21.15.195:0/1113472317 learned_addr learned my addr 172.21.15.195:0/1113472317 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:58:12.553 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.523+0000 7f63d77fe700 1 -- 172.21.15.195:0/1113472317 >> v1:172.21.15.202:6789/0 conn(0x7f63d8154dd0 legacy=0x7f63d8158100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:12.554 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.523+0000 7f63d77fe700 1 -- 172.21.15.195:0/1113472317 >> v1:172.21.15.195:6789/0 conn(0x7f63d8154310 legacy=0x7f63d80c1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:12.554 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.523+0000 7f63d77fe700 1 -- 172.21.15.195:0/1113472317 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63d80c3f20 con 0x7f63d80c34b0 2022-06-27T11:58:12.555 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.523+0000 7f63d5ffb700 1 -- 172.21.15.195:0/1113472317 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 237088232 0 0) 0x7f63d80c3980 con 0x7f63d8154310 2022-06-27T11:58:12.555 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.523+0000 7f63d77fe700 1 --2- 172.21.15.195:0/1113472317 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d80c34b0 0x7f63d815fc10 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f63d0009e60 tx=0x7f63d0009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ffaf97a4c36816d server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:12.556 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.523+0000 7f63d5ffb700 1 -- 172.21.15.195:0/1113472317 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d000e070 con 0x7f63d80c34b0 2022-06-27T11:58:12.556 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.523+0000 7f63d5ffb700 1 -- 172.21.15.195:0/1113472317 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63d000bcc0 con 0x7f63d80c34b0 2022-06-27T11:58:12.556 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.523+0000 7f63d5ffb700 1 -- 172.21.15.195:0/1113472317 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d0012680 con 0x7f63d80c34b0 2022-06-27T11:58:12.578 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.524+0000 7f63dd720700 1 -- 172.21.15.195:0/1113472317 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d80c34b0 msgr2=0x7f63d815fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:12.585 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.524+0000 7f63dd720700 1 --2- 172.21.15.195:0/1113472317 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d80c34b0 0x7f63d815fc10 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f63d0009e60 tx=0x7f63d0009780 comp rx=0 tx=0).stop 2022-06-27T11:58:12.585 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.524+0000 7f63dd720700 1 -- 172.21.15.195:0/1113472317 shutdown_connections 2022-06-27T11:58:12.586 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.524+0000 7f63dd720700 1 --2- 172.21.15.195:0/1113472317 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d80c34b0 0x7f63d815fc10 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:12.587 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.524+0000 7f63dd720700 1 -- 172.21.15.195:0/1113472317 >> 172.21.15.195:0/1113472317 conn(0x7f63d80c8740 msgr2=0x7f63d80c8b10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:58:12.587 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.524+0000 7f63dd720700 1 -- 172.21.15.195:0/1113472317 shutdown_connections 2022-06-27T11:58:12.587 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.524+0000 7f63dd720700 1 -- 172.21.15.195:0/1113472317 wait complete. 2022-06-27T11:58:12.588 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.525+0000 7f63dd720700 1 Processor -- start 2022-06-27T11:58:12.588 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.525+0000 7f63dd720700 1 -- start start 2022-06-27T11:58:12.589 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.526+0000 7f63dd720700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f63d80c34b0 0x7f63d8155460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:12.589 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.526+0000 7f63dd720700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f63d8154310 0x7f63d80b87a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:12.592 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.526+0000 7f63dd720700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8154dd0 0x7f63d80c6830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:12.592 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.526+0000 7f63dd720700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d80c7ff0 con 0x7f63d8154dd0 2022-06-27T11:58:12.592 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.526+0000 7f63dd720700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d814f210 con 0x7f63d80c34b0 2022-06-27T11:58:12.593 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.526+0000 7f63dd720700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63d814e600 con 0x7f63d8154310 2022-06-27T11:58:12.593 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.526+0000 7f63d77fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8154dd0 0x7f63d80c6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:12.594 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.526+0000 7f63d77fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8154dd0 0x7f63d80c6830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45584/0 (socket says 172.21.15.195:45584) 2022-06-27T11:58:12.594 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.526+0000 7f63d77fe700 1 -- 172.21.15.195:0/3382757503 learned_addr learned my addr 172.21.15.195:0/3382757503 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:58:12.595 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.526+0000 7f63d6ffd700 1 --2- 172.21.15.195:0/3382757503 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f63d80c34b0 0x7f63d8155460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:12.595 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.526+0000 7f63d67fc700 1 --2- 172.21.15.195:0/3382757503 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f63d8154310 0x7f63d80b87a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:12.596 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.526+0000 7f63d77fe700 1 -- 172.21.15.195:0/3382757503 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f63d8154310 msgr2=0x7f63d80b87a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:12.596 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.526+0000 7f63d77fe700 1 --2- 172.21.15.195:0/3382757503 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f63d8154310 0x7f63d80b87a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:12.596 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.526+0000 7f63d77fe700 1 -- 172.21.15.195:0/3382757503 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f63d80c34b0 msgr2=0x7f63d8155460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:12.597 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.526+0000 7f63d77fe700 1 --2- 172.21.15.195:0/3382757503 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f63d80c34b0 0x7f63d8155460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:12.597 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.526+0000 7f63d77fe700 1 -- 172.21.15.195:0/3382757503 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63d0009a20 con 0x7f63d8154dd0 2022-06-27T11:58:12.598 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.526+0000 7f63d77fe700 1 --2- 172.21.15.195:0/3382757503 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8154dd0 0x7f63d80c6830 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f63d000b6c0 tx=0x7f63d0019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:12.598 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.526+0000 7f63beffd700 1 -- 172.21.15.195:0/3382757503 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d001b420 con 0x7f63d8154dd0 2022-06-27T11:58:12.599 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.526+0000 7f63beffd700 1 -- 172.21.15.195:0/3382757503 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63d001b580 con 0x7f63d8154dd0 2022-06-27T11:58:12.599 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.526+0000 7f63beffd700 1 -- 172.21.15.195:0/3382757503 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d001b940 con 0x7f63d8154dd0 2022-06-27T11:58:12.600 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.526+0000 7f63dd720700 1 -- 172.21.15.195:0/3382757503 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63d814e8e0 con 0x7f63d8154dd0 2022-06-27T11:58:12.614 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.527+0000 7f63dd720700 1 -- 172.21.15.195:0/3382757503 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63d814eea0 con 0x7f63d8154dd0 2022-06-27T11:58:12.615 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.527+0000 7f63dd720700 1 -- 172.21.15.195:0/3382757503 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63d814f380 con 0x7f63d8154dd0 2022-06-27T11:58:12.616 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.528+0000 7f63beffd700 1 -- 172.21.15.195:0/3382757503 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f63d002c030 con 0x7f63d8154dd0 2022-06-27T11:58:12.616 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.528+0000 7f63beffd700 1 --2- 172.21.15.195:0/3382757503 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f63b8063d00 0x7f63b80661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:12.617 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.528+0000 7f63beffd700 1 -- 172.21.15.195:0/3382757503 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6632+0+0 (secure 0 0 0) 0x7f63d0082880 con 0x7f63d8154dd0 2022-06-27T11:58:12.618 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.528+0000 7f63d6ffd700 1 --2- 172.21.15.195:0/3382757503 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f63b8063d00 0x7f63b80661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:12.619 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.529+0000 7f63d6ffd700 1 --2- 172.21.15.195:0/3382757503 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f63b8063d00 0x7f63b80661b0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f63cc0017e0 tx=0x7f63cc009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:12.620 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.532+0000 7f63beffd700 1 -- 172.21.15.195:0/3382757503 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f63d0051230 con 0x7f63d8154dd0 2022-06-27T11:58:12.760 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.756+0000 7f63dd720700 1 -- 172.21.15.195:0/3382757503 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f63d80722c0 con 0x7f63d8154dd0 2022-06-27T11:58:12.760 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.757+0000 7f63beffd700 1 -- 172.21.15.195:0/3382757503 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+5367 (secure 0 0 0) 0x7f63d002a070 con 0x7f63d8154dd0 2022-06-27T11:58:12.761 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:58:12.764 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:58:09.974633+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5392},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5392":{"gid":5392,"name":"d","rank":0,"incarnation":38,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6835/2089155519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2089155519},{"type":"v1","addr":"172.21.15.202:6835","nonce":2089155519}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:58:12.766 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.759+0000 7f63dd720700 1 -- 172.21.15.195:0/3382757503 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f63b8063d00 msgr2=0x7f63b80661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:12.767 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.759+0000 7f63dd720700 1 --2- 172.21.15.195:0/3382757503 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f63b8063d00 0x7f63b80661b0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f63cc0017e0 tx=0x7f63cc009040 comp rx=0 tx=0).stop 2022-06-27T11:58:12.767 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.759+0000 7f63dd720700 1 -- 172.21.15.195:0/3382757503 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8154dd0 msgr2=0x7f63d80c6830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:12.767 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.759+0000 7f63dd720700 1 --2- 172.21.15.195:0/3382757503 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8154dd0 0x7f63d80c6830 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f63d000b6c0 tx=0x7f63d0019000 comp rx=0 tx=0).stop 2022-06-27T11:58:12.768 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.759+0000 7f63dd720700 1 -- 172.21.15.195:0/3382757503 shutdown_connections 2022-06-27T11:58:12.768 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.759+0000 7f63dd720700 1 --2- 172.21.15.195:0/3382757503 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f63b8063d00 0x7f63b80661b0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:12.768 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.759+0000 7f63dd720700 1 --2- 172.21.15.195:0/3382757503 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8154dd0 0x7f63d80c6830 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:12.769 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.759+0000 7f63dd720700 1 --2- 172.21.15.195:0/3382757503 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f63d80c34b0 0x7f63d8155460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:12.769 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.759+0000 7f63dd720700 1 --2- 172.21.15.195:0/3382757503 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f63d8154310 0x7f63d80b87a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:12.769 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.759+0000 7f63dd720700 1 -- 172.21.15.195:0/3382757503 >> 172.21.15.195:0/3382757503 conn(0x7f63d80c8740 msgr2=0x7f63d8158000 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:58:12.770 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.759+0000 7f63dd720700 1 -- 172.21.15.195:0/3382757503 shutdown_connections 2022-06-27T11:58:12.770 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:12.759+0000 7f63dd720700 1 -- 172.21.15.195:0/3382757503 wait complete. 2022-06-27T11:58:12.778 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 38 2022-06-27T11:58:12.781 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T11:58:14.782 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:58:14.933 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:14.915+0000 7f78625fc700 1 Processor -- start 2022-06-27T11:58:14.934 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:14.915+0000 7f78625fc700 1 -- start start 2022-06-27T11:58:14.934 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:14.916+0000 7f78625fc700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f785c14e490 0x7f785c06cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:14.935 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:14.916+0000 7f78625fc700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f785c14e860 0x7f785c0c3840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:14.935 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:14.916+0000 7f78625fc700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f785c0c33b0 con 0x7f785c0c3d80 2022-06-27T11:58:14.935 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:14.916+0000 7f78625fc700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f785c06d520 con 0x7f785c14e490 2022-06-27T11:58:14.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:14.916+0000 7f78625fc700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f785c06f220 con 0x7f785c14e860 2022-06-27T11:58:14.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:14.916+0000 7f785bfff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f785c14e490 0x7f785c06cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:14.937 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:14.916+0000 7f785bfff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f785c14e490 0x7f785c06cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45588/0 (socket says 172.21.15.195:45588) 2022-06-27T11:58:14.937 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:14.916+0000 7f785bfff700 1 -- 172.21.15.195:0/2604936286 learned_addr learned my addr 172.21.15.195:0/2604936286 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:58:14.938 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:14.916+0000 7f785b7fe700 1 --2- 172.21.15.195:0/2604936286 >> v2:172.21.15.202:3300/0 conn(0x7f785c14e860 0x7f785c0c3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:14.938 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:14.916+0000 7f785bfff700 1 -- 172.21.15.195:0/2604936286 >> v2:172.21.15.202:3300/0 conn(0x7f785c14e860 msgr2=0x7f785c0c3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:58:14.938 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:14.916+0000 7f785bfff700 1 --2- 172.21.15.195:0/2604936286 >> v2:172.21.15.202:3300/0 conn(0x7f785c14e860 0x7f785c0c3840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:14.939 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:14.916+0000 7f785bfff700 1 -- 172.21.15.195:0/2604936286 >> v1:172.21.15.202:6789/0 conn(0x7f785c0c3d80 legacy=0x7f785c06f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:14.939 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:14.916+0000 7f785bfff700 1 -- 172.21.15.195:0/2604936286 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f785c1542c0 con 0x7f785c14e490 2022-06-27T11:58:14.940 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:14.917+0000 7f785bfff700 1 --2- 172.21.15.195:0/2604936286 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f785c14e490 0x7f785c06cfe0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f785c065b80 tx=0x7f784c009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e126e2e77b5d84db server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:14.941 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:14.917+0000 7f785affd700 1 -- 172.21.15.195:0/2604936286 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f784c00b640 con 0x7f785c14e490 2022-06-27T11:58:14.942 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:14.917+0000 7f785affd700 1 -- 172.21.15.195:0/2604936286 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f784c00b7a0 con 0x7f785c14e490 2022-06-27T11:58:14.942 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:14.917+0000 7f785affd700 1 -- 172.21.15.195:0/2604936286 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f784c00bb40 con 0x7f785c14e490 2022-06-27T11:58:14.943 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:14.917+0000 7f78625fc700 1 -- 172.21.15.195:0/2604936286 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f785c14e490 msgr2=0x7f785c06cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:14.943 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:14.917+0000 7f78625fc700 1 --2- 172.21.15.195:0/2604936286 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f785c14e490 0x7f785c06cfe0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f785c065b80 tx=0x7f784c009120 comp rx=0 tx=0).stop 2022-06-27T11:58:14.944 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:14.917+0000 7f78625fc700 1 -- 172.21.15.195:0/2604936286 shutdown_connections 2022-06-27T11:58:14.944 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:14.917+0000 7f78625fc700 1 --2- 172.21.15.195:0/2604936286 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f785c14e490 0x7f785c06cfe0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:14.946 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:14.917+0000 7f78625fc700 1 --2- 172.21.15.195:0/2604936286 >> v2:172.21.15.202:3300/0 conn(0x7f785c14e860 0x7f785c0c3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:14.947 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:14.917+0000 7f78625fc700 1 -- 172.21.15.195:0/2604936286 >> 172.21.15.195:0/2604936286 conn(0x7f785c063af0 msgr2=0x7f785c063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:58:14.947 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:14.917+0000 7f78625fc700 1 -- 172.21.15.195:0/2604936286 shutdown_connections 2022-06-27T11:58:14.948 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:14.917+0000 7f78625fc700 1 -- 172.21.15.195:0/2604936286 wait complete. 2022-06-27T11:58:14.948 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:14.919+0000 7f78625fc700 1 Processor -- start 2022-06-27T11:58:14.948 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:14.919+0000 7f78625fc700 1 -- start start 2022-06-27T11:58:14.949 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:14.920+0000 7f78625fc700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f785c0c3d80 0x7f785c166710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:14.949 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:14.920+0000 7f78625fc700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f785c14e490 0x7f785c156510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:14.950 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:14.920+0000 7f78625fc700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f785c14e860 0x7f785c156a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:14.950 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:14.920+0000 7f78625fc700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f785c1576b0 con 0x7f785c14e860 2022-06-27T11:58:14.951 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:14.920+0000 7f78625fc700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f785c145f70 con 0x7f785c0c3d80 2022-06-27T11:58:14.951 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:14.920+0000 7f78625fc700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f785c064170 con 0x7f785c14e490 2022-06-27T11:58:14.953 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:14.920+0000 7f7860b99700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f785c14e860 0x7f785c156a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:14.953 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:14.920+0000 7f7860b99700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f785c14e860 0x7f785c156a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45598/0 (socket says 172.21.15.195:45598) 2022-06-27T11:58:14.954 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:14.920+0000 7f7860b99700 1 -- 172.21.15.195:0/840366125 learned_addr learned my addr 172.21.15.195:0/840366125 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:58:14.954 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:14.920+0000 7f785bfff700 1 --2- 172.21.15.195:0/840366125 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f785c0c3d80 0x7f785c166710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:14.955 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:14.920+0000 7f785b7fe700 1 --2- 172.21.15.195:0/840366125 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f785c14e490 0x7f785c156510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:14.955 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:14.921+0000 7f7860b99700 1 -- 172.21.15.195:0/840366125 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f785c14e490 msgr2=0x7f785c156510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:14.956 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:14.921+0000 7f7860b99700 1 --2- 172.21.15.195:0/840366125 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f785c14e490 0x7f785c156510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:14.956 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:14.921+0000 7f7860b99700 1 -- 172.21.15.195:0/840366125 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f785c0c3d80 msgr2=0x7f785c166710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:14.957 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:14.921+0000 7f7860b99700 1 --2- 172.21.15.195:0/840366125 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f785c0c3d80 0x7f785c166710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:14.957 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:14.921+0000 7f7860b99700 1 -- 172.21.15.195:0/840366125 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f784c0093a0 con 0x7f785c14e860 2022-06-27T11:58:14.958 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:14.921+0000 7f7860b99700 1 --2- 172.21.15.195:0/840366125 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f785c14e860 0x7f785c156a50 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f785400c710 tx=0x7f7854001540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:14.958 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:14.921+0000 7f7843fff700 1 -- 172.21.15.195:0/840366125 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7854003880 con 0x7f785c14e860 2022-06-27T11:58:14.959 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:14.921+0000 7f7843fff700 1 -- 172.21.15.195:0/840366125 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78540039e0 con 0x7f785c14e860 2022-06-27T11:58:14.959 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:14.921+0000 7f78625fc700 1 -- 172.21.15.195:0/840366125 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f785c166c50 con 0x7f785c14e860 2022-06-27T11:58:14.959 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:14.921+0000 7f7843fff700 1 -- 172.21.15.195:0/840366125 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7854003da0 con 0x7f785c14e860 2022-06-27T11:58:14.960 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:14.921+0000 7f78625fc700 1 -- 172.21.15.195:0/840366125 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f785c155f30 con 0x7f785c14e860 2022-06-27T11:58:14.960 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:14.922+0000 7f78625fc700 1 -- 172.21.15.195:0/840366125 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f785c0722c0 con 0x7f785c14e860 2022-06-27T11:58:14.961 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:14.922+0000 7f7843fff700 1 -- 172.21.15.195:0/840366125 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f7854019030 con 0x7f785c14e860 2022-06-27T11:58:14.961 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:14.923+0000 7f7843fff700 1 --2- 172.21.15.195:0/840366125 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f783c063d00 0x7f783c0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:14.962 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:14.923+0000 7f7843fff700 1 -- 172.21.15.195:0/840366125 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6632+0+0 (secure 0 0 0) 0x7f785407fc80 con 0x7f785c14e860 2022-06-27T11:58:14.962 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:14.923+0000 7f785bfff700 1 --2- 172.21.15.195:0/840366125 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f783c063d00 0x7f783c0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:14.968 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:14.923+0000 7f785bfff700 1 --2- 172.21.15.195:0/840366125 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f783c063d00 0x7f783c0661b0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f784c009160 tx=0x7f784c017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:14.968 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:14.927+0000 7f7843fff700 1 -- 172.21.15.195:0/840366125 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f785404e630 con 0x7f785c14e860 2022-06-27T11:58:15.173 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:15.172+0000 7f78625fc700 1 -- 172.21.15.195:0/840366125 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f785c0c60d0 con 0x7f785c14e860 2022-06-27T11:58:15.174 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:15.173+0000 7f7843fff700 1 -- 172.21.15.195:0/840366125 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+5367 (secure 0 0 0) 0x7f7854017020 con 0x7f785c14e860 2022-06-27T11:58:15.175 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:58:15.176 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:58:09.974633+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5392},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5392":{"gid":5392,"name":"d","rank":0,"incarnation":38,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6835/2089155519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2089155519},{"type":"v1","addr":"172.21.15.202:6835","nonce":2089155519}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:58:15.178 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:15.175+0000 7f78625fc700 1 -- 172.21.15.195:0/840366125 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f783c063d00 msgr2=0x7f783c0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:15.179 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:15.175+0000 7f78625fc700 1 --2- 172.21.15.195:0/840366125 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f783c063d00 0x7f783c0661b0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f784c009160 tx=0x7f784c017000 comp rx=0 tx=0).stop 2022-06-27T11:58:15.179 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:15.175+0000 7f78625fc700 1 -- 172.21.15.195:0/840366125 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f785c14e860 msgr2=0x7f785c156a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:15.180 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:15.175+0000 7f78625fc700 1 --2- 172.21.15.195:0/840366125 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f785c14e860 0x7f785c156a50 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f785400c710 tx=0x7f7854001540 comp rx=0 tx=0).stop 2022-06-27T11:58:15.183 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:15.175+0000 7f78625fc700 1 -- 172.21.15.195:0/840366125 shutdown_connections 2022-06-27T11:58:15.184 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:15.175+0000 7f78625fc700 1 --2- 172.21.15.195:0/840366125 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f783c063d00 0x7f783c0661b0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:15.184 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:15.175+0000 7f78625fc700 1 --2- 172.21.15.195:0/840366125 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f785c14e860 0x7f785c156a50 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:15.185 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:15.175+0000 7f78625fc700 1 --2- 172.21.15.195:0/840366125 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f785c0c3d80 0x7f785c166710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:15.185 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:15.175+0000 7f78625fc700 1 --2- 172.21.15.195:0/840366125 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f785c14e490 0x7f785c156510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:15.186 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:15.175+0000 7f78625fc700 1 -- 172.21.15.195:0/840366125 >> 172.21.15.195:0/840366125 conn(0x7f785c063af0 msgr2=0x7f785c153a90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:58:15.186 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:15.175+0000 7f78625fc700 1 -- 172.21.15.195:0/840366125 shutdown_connections 2022-06-27T11:58:15.187 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:15.175+0000 7f78625fc700 1 -- 172.21.15.195:0/840366125 wait complete. 2022-06-27T11:58:15.187 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 38 2022-06-27T11:58:15.190 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T11:58:17.194 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:58:17.364 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.334+0000 7f9dbdd4e700 1 Processor -- start 2022-06-27T11:58:17.365 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.334+0000 7f9dbdd4e700 1 -- start start 2022-06-27T11:58:17.366 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.335+0000 7f9dbdd4e700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f9db8156100 0x7f9db80688a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:17.367 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.335+0000 7f9dbdd4e700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f9db806a440 0x7f9db80c4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:17.367 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.335+0000 7f9dbdd4e700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9db80c4b10 con 0x7f9db81564d0 2022-06-27T11:58:17.368 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.335+0000 7f9dbdd4e700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9db80c3dc0 con 0x7f9db8156100 2022-06-27T11:58:17.368 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.335+0000 7f9dbdd4e700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9db8068f20 con 0x7f9db806a440 2022-06-27T11:58:17.369 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.335+0000 7f9db77fe700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f9db8156100 0x7f9db80688a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:17.369 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.335+0000 7f9db7fff700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f9db81564d0 0x7f9db80c28c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:51796/0 (socket says 172.21.15.195:51796) 2022-06-27T11:58:17.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.335+0000 7f9db7fff700 1 -- 172.21.15.195:0/434504212 learned_addr learned my addr 172.21.15.195:0/434504212 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:58:17.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.335+0000 7f9db6ffd700 1 --2- 172.21.15.195:0/434504212 >> v2:172.21.15.202:3300/0 conn(0x7f9db806a440 0x7f9db80c4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:17.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.335+0000 7f9db67fc700 1 -- 172.21.15.195:0/434504212 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2971527692 0 0) 0x7f9db80c4b10 con 0x7f9db81564d0 2022-06-27T11:58:17.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.335+0000 7f9db67fc700 1 -- 172.21.15.195:0/434504212 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9da4003430 con 0x7f9db81564d0 2022-06-27T11:58:17.372 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.335+0000 7f9db77fe700 1 -- 172.21.15.195:0/434504212 >> v2:172.21.15.202:3300/0 conn(0x7f9db806a440 msgr2=0x7f9db80c4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:58:17.372 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.335+0000 7f9db77fe700 1 --2- 172.21.15.195:0/434504212 >> v2:172.21.15.202:3300/0 conn(0x7f9db806a440 0x7f9db80c4fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:17.372 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.335+0000 7f9db77fe700 1 -- 172.21.15.195:0/434504212 >> v1:172.21.15.195:6789/0 conn(0x7f9db81564d0 legacy=0x7f9db80c28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:17.373 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.335+0000 7f9db77fe700 1 -- 172.21.15.195:0/434504212 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9db80c6360 con 0x7f9db8156100 2022-06-27T11:58:17.373 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.336+0000 7f9db77fe700 1 --2- 172.21.15.195:0/434504212 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9db8156100 0x7f9db80688a0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f9db8072140 tx=0x7f9dac00a200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f90aa80af11605c5 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:17.374 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.336+0000 7f9db67fc700 1 -- 172.21.15.195:0/434504212 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dac005ad0 con 0x7f9db8156100 2022-06-27T11:58:17.374 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.336+0000 7f9db67fc700 1 -- 172.21.15.195:0/434504212 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dac005c30 con 0x7f9db8156100 2022-06-27T11:58:17.375 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.336+0000 7f9db67fc700 1 -- 172.21.15.195:0/434504212 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dac006030 con 0x7f9db8156100 2022-06-27T11:58:17.376 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.336+0000 7f9dbdd4e700 1 -- 172.21.15.195:0/434504212 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9db8156100 msgr2=0x7f9db80688a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:17.376 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.336+0000 7f9dbdd4e700 1 --2- 172.21.15.195:0/434504212 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9db8156100 0x7f9db80688a0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f9db8072140 tx=0x7f9dac00a200 comp rx=0 tx=0).stop 2022-06-27T11:58:17.378 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.336+0000 7f9dbdd4e700 1 -- 172.21.15.195:0/434504212 shutdown_connections 2022-06-27T11:58:17.379 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.336+0000 7f9dbdd4e700 1 --2- 172.21.15.195:0/434504212 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9db8156100 0x7f9db80688a0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:17.379 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.336+0000 7f9dbdd4e700 1 --2- 172.21.15.195:0/434504212 >> v2:172.21.15.202:3300/0 conn(0x7f9db806a440 0x7f9db80c4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:17.380 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.336+0000 7f9dbdd4e700 1 -- 172.21.15.195:0/434504212 >> 172.21.15.195:0/434504212 conn(0x7f9db80c9b00 msgr2=0x7f9db80c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:58:17.380 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.336+0000 7f9dbdd4e700 1 -- 172.21.15.195:0/434504212 shutdown_connections 2022-06-27T11:58:17.381 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.336+0000 7f9dbdd4e700 1 -- 172.21.15.195:0/434504212 wait complete. 2022-06-27T11:58:17.381 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.338+0000 7f9dbdd4e700 1 Processor -- start 2022-06-27T11:58:17.382 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.338+0000 7f9dbdd4e700 1 -- start start 2022-06-27T11:58:17.382 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.338+0000 7f9dbdd4e700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9db806a440 0x7f9db8148e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:17.383 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.338+0000 7f9dbdd4e700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9db8156100 0x7f9db8149380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:17.384 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.338+0000 7f9dbdd4e700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9db81564d0 0x7f9db814bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:17.384 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.338+0000 7f9dbdd4e700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9db8149dd0 con 0x7f9db8156100 2022-06-27T11:58:17.385 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.338+0000 7f9dbdd4e700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9db80c8ab0 con 0x7f9db806a440 2022-06-27T11:58:17.385 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.338+0000 7f9dbdd4e700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9db80c7bc0 con 0x7f9db81564d0 2022-06-27T11:58:17.386 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.338+0000 7f9db6ffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9db8156100 0x7f9db8149380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:17.386 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.339+0000 7f9db6ffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9db8156100 0x7f9db8149380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45610/0 (socket says 172.21.15.195:45610) 2022-06-27T11:58:17.387 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.339+0000 7f9db77fe700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9db806a440 0x7f9db8148e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:17.387 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.339+0000 7f9db6ffd700 1 -- 172.21.15.195:0/4029294304 learned_addr learned my addr 172.21.15.195:0/4029294304 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:58:17.388 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.339+0000 7f9db7fff700 1 --2- 172.21.15.195:0/4029294304 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9db81564d0 0x7f9db814bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:17.389 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.339+0000 7f9db6ffd700 1 -- 172.21.15.195:0/4029294304 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9db81564d0 msgr2=0x7f9db814bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:17.389 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.339+0000 7f9db6ffd700 1 --2- 172.21.15.195:0/4029294304 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9db81564d0 0x7f9db814bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:17.390 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.339+0000 7f9db6ffd700 1 -- 172.21.15.195:0/4029294304 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9db806a440 msgr2=0x7f9db8148e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:17.390 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.339+0000 7f9db6ffd700 1 --2- 172.21.15.195:0/4029294304 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9db806a440 0x7f9db8148e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:17.391 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.339+0000 7f9db6ffd700 1 -- 172.21.15.195:0/4029294304 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dac00a420 con 0x7f9db8156100 2022-06-27T11:58:17.392 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.339+0000 7f9db7fff700 1 --2- 172.21.15.195:0/4029294304 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9db81564d0 0x7f9db814bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T11:58:17.392 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.339+0000 7f9db6ffd700 1 --2- 172.21.15.195:0/4029294304 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9db8156100 0x7f9db8149380 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f9da800e970 tx=0x7f9da8009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:17.393 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.339+0000 7f9d9f7fe700 1 -- 172.21.15.195:0/4029294304 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9da800ee60 con 0x7f9db8156100 2022-06-27T11:58:17.394 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.339+0000 7f9d9f7fe700 1 -- 172.21.15.195:0/4029294304 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9da800c850 con 0x7f9db8156100 2022-06-27T11:58:17.394 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.339+0000 7f9dbdd4e700 1 -- 172.21.15.195:0/4029294304 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9db80c7ea0 con 0x7f9db8156100 2022-06-27T11:58:17.395 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.339+0000 7f9d9f7fe700 1 -- 172.21.15.195:0/4029294304 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9da800cbf0 con 0x7f9db8156100 2022-06-27T11:58:17.395 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.339+0000 7f9dbdd4e700 1 -- 172.21.15.195:0/4029294304 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9db80c8460 con 0x7f9db8156100 2022-06-27T11:58:17.396 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.340+0000 7f9dbdd4e700 1 -- 172.21.15.195:0/4029294304 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9db80c8bf0 con 0x7f9db8156100 2022-06-27T11:58:17.397 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.340+0000 7f9d9f7fe700 1 -- 172.21.15.195:0/4029294304 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f9da8019030 con 0x7f9db8156100 2022-06-27T11:58:17.397 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.341+0000 7f9d9f7fe700 1 --2- 172.21.15.195:0/4029294304 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9d98063d30 0x7f9d980661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:17.402 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.341+0000 7f9d9f7fe700 1 -- 172.21.15.195:0/4029294304 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6632+0+0 (secure 0 0 0) 0x7f9da8080aa0 con 0x7f9db8156100 2022-06-27T11:58:17.403 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.341+0000 7f9db77fe700 1 --2- 172.21.15.195:0/4029294304 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9d98063d30 0x7f9d980661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:17.403 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.341+0000 7f9db77fe700 1 --2- 172.21.15.195:0/4029294304 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9d98063d30 0x7f9d980661e0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f9dac00a240 tx=0x7f9dac019000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:17.403 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.345+0000 7f9d9f7fe700 1 -- 172.21.15.195:0/4029294304 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f9db80c8bf0 con 0x7f9db8156100 2022-06-27T11:58:17.570 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.570+0000 7f9dbdd4e700 1 -- 172.21.15.195:0/4029294304 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9db80c8bf0 con 0x7f9db8156100 2022-06-27T11:58:17.571 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.571+0000 7f9d9f7fe700 1 -- 172.21.15.195:0/4029294304 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+5367 (secure 0 0 0) 0x7f9db80c8bf0 con 0x7f9db8156100 2022-06-27T11:58:17.572 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:58:17.573 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:58:09.974633+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5392},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5392":{"gid":5392,"name":"d","rank":0,"incarnation":38,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6835/2089155519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2089155519},{"type":"v1","addr":"172.21.15.202:6835","nonce":2089155519}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:58:17.576 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.573+0000 7f9dbdd4e700 1 -- 172.21.15.195:0/4029294304 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9d98063d30 msgr2=0x7f9d980661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:17.577 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.573+0000 7f9dbdd4e700 1 --2- 172.21.15.195:0/4029294304 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9d98063d30 0x7f9d980661e0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f9dac00a240 tx=0x7f9dac019000 comp rx=0 tx=0).stop 2022-06-27T11:58:17.578 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.573+0000 7f9dbdd4e700 1 -- 172.21.15.195:0/4029294304 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9db8156100 msgr2=0x7f9db8149380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:17.578 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.573+0000 7f9dbdd4e700 1 --2- 172.21.15.195:0/4029294304 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9db8156100 0x7f9db8149380 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f9da800e970 tx=0x7f9da8009060 comp rx=0 tx=0).stop 2022-06-27T11:58:17.579 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.573+0000 7f9dbdd4e700 1 -- 172.21.15.195:0/4029294304 shutdown_connections 2022-06-27T11:58:17.579 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.573+0000 7f9dbdd4e700 1 --2- 172.21.15.195:0/4029294304 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9d98063d30 0x7f9d980661e0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:17.580 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.573+0000 7f9dbdd4e700 1 --2- 172.21.15.195:0/4029294304 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9db81564d0 0x7f9db814bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:17.580 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.573+0000 7f9dbdd4e700 1 --2- 172.21.15.195:0/4029294304 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9db806a440 0x7f9db8148e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:17.580 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.573+0000 7f9dbdd4e700 1 --2- 172.21.15.195:0/4029294304 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9db8156100 0x7f9db8149380 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:17.580 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.573+0000 7f9dbdd4e700 1 -- 172.21.15.195:0/4029294304 >> 172.21.15.195:0/4029294304 conn(0x7f9db80c9b00 msgr2=0x7f9db814f6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:58:17.581 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.573+0000 7f9dbdd4e700 1 -- 172.21.15.195:0/4029294304 shutdown_connections 2022-06-27T11:58:17.581 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:17.573+0000 7f9dbdd4e700 1 -- 172.21.15.195:0/4029294304 wait complete. 2022-06-27T11:58:17.581 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 38 2022-06-27T11:58:17.587 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T11:58:19.596 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:58:19.733 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.729+0000 7fb860e1b700 1 Processor -- start 2022-06-27T11:58:19.733 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.730+0000 7fb860e1b700 1 -- start start 2022-06-27T11:58:19.733 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.730+0000 7fb860e1b700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb85c06f640 0x7fb85c156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:19.734 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.730+0000 7fb860e1b700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fb85c0c39e0 0x7fb85c1535a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:19.734 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.730+0000 7fb860e1b700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb85c0c1a20 con 0x7fb85c0c4160 2022-06-27T11:58:19.735 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.730+0000 7fb860e1b700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb85c0c1130 con 0x7fb85c0c39e0 2022-06-27T11:58:19.735 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.730+0000 7fb860e1b700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb85c06ea90 con 0x7fb85c06f640 2022-06-27T11:58:19.735 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.731+0000 7fb859d9b700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fb85c0c4160 0x7fb85c0c1eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:51804/0 (socket says 172.21.15.195:51804) 2022-06-27T11:58:19.735 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.731+0000 7fb859d9b700 1 -- 172.21.15.195:0/4053340717 learned_addr learned my addr 172.21.15.195:0/4053340717 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:58:19.735 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.731+0000 7fb85ad9d700 1 --2- 172.21.15.195:0/4053340717 >> v2:172.21.15.202:3300/0 conn(0x7fb85c0c39e0 0x7fb85c1535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:19.736 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.731+0000 7fb85a59c700 1 --2- 172.21.15.195:0/4053340717 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb85c06f640 0x7fb85c156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:19.736 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.731+0000 7fb85959a700 1 -- 172.21.15.195:0/4053340717 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 740815036 0 0) 0x7fb85c0c1a20 con 0x7fb85c0c4160 2022-06-27T11:58:19.736 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.731+0000 7fb85959a700 1 -- 172.21.15.195:0/4053340717 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb83c003430 con 0x7fb85c0c4160 2022-06-27T11:58:19.736 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.731+0000 7fb85a59c700 1 -- 172.21.15.195:0/4053340717 >> v2:172.21.15.202:3300/0 conn(0x7fb85c0c39e0 msgr2=0x7fb85c1535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:58:19.737 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.731+0000 7fb85a59c700 1 --2- 172.21.15.195:0/4053340717 >> v2:172.21.15.202:3300/0 conn(0x7fb85c0c39e0 0x7fb85c1535a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:19.737 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.731+0000 7fb85a59c700 1 -- 172.21.15.195:0/4053340717 >> v1:172.21.15.195:6789/0 conn(0x7fb85c0c4160 legacy=0x7fb85c0c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:19.737 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.731+0000 7fb85a59c700 1 -- 172.21.15.195:0/4053340717 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb85c06ed70 con 0x7fb85c06f640 2022-06-27T11:58:19.738 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.731+0000 7fb85ad9d700 1 --2- 172.21.15.195:0/4053340717 >> v2:172.21.15.202:3300/0 conn(0x7fb85c0c39e0 0x7fb85c1535a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:58:19.738 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.731+0000 7fb85a59c700 1 --2- 172.21.15.195:0/4053340717 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb85c06f640 0x7fb85c156ed0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb8440097a0 tx=0x7fb844009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=808fb502332ca7fe server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:19.738 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.731+0000 7fb85959a700 1 -- 172.21.15.195:0/4053340717 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb844011630 con 0x7fb85c06f640 2022-06-27T11:58:19.738 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.731+0000 7fb85959a700 1 -- 172.21.15.195:0/4053340717 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb844011790 con 0x7fb85c06f640 2022-06-27T11:58:19.739 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.731+0000 7fb85959a700 1 -- 172.21.15.195:0/4053340717 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb844011b30 con 0x7fb85c06f640 2022-06-27T11:58:19.739 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.732+0000 7fb860e1b700 1 -- 172.21.15.195:0/4053340717 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb85c06f640 msgr2=0x7fb85c156ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:19.739 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.732+0000 7fb860e1b700 1 --2- 172.21.15.195:0/4053340717 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb85c06f640 0x7fb85c156ed0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb8440097a0 tx=0x7fb844009060 comp rx=0 tx=0).stop 2022-06-27T11:58:19.739 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.732+0000 7fb860e1b700 1 -- 172.21.15.195:0/4053340717 shutdown_connections 2022-06-27T11:58:19.740 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.732+0000 7fb860e1b700 1 --2- 172.21.15.195:0/4053340717 >> v2:172.21.15.202:3300/0 conn(0x7fb85c0c39e0 0x7fb85c1535a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:19.740 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.732+0000 7fb860e1b700 1 --2- 172.21.15.195:0/4053340717 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb85c06f640 0x7fb85c156ed0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:19.740 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.732+0000 7fb860e1b700 1 -- 172.21.15.195:0/4053340717 >> 172.21.15.195:0/4053340717 conn(0x7fb85c0c6350 msgr2=0x7fb85c0c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:58:19.740 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.732+0000 7fb860e1b700 1 -- 172.21.15.195:0/4053340717 shutdown_connections 2022-06-27T11:58:19.741 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.732+0000 7fb860e1b700 1 -- 172.21.15.195:0/4053340717 wait complete. 2022-06-27T11:58:19.744 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.733+0000 7fb860e1b700 1 Processor -- start 2022-06-27T11:58:19.744 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.733+0000 7fb860e1b700 1 -- start start 2022-06-27T11:58:19.744 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.734+0000 7fb860e1b700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb85c06f640 0x7fb85c0bfb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:19.745 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.734+0000 7fb860e1b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb85c0c39e0 0x7fb85c0c00b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:19.745 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.734+0000 7fb860e1b700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb85c0c4160 0x7fb85c151c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:19.745 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.734+0000 7fb860e1b700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb85c0bf6e0 con 0x7fb85c0c39e0 2022-06-27T11:58:19.745 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.734+0000 7fb860e1b700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb85c166530 con 0x7fb85c06f640 2022-06-27T11:58:19.746 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.734+0000 7fb860e1b700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb85c1667f0 con 0x7fb85c0c4160 2022-06-27T11:58:19.746 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.734+0000 7fb859d9b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb85c0c39e0 0x7fb85c0c00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:19.747 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.734+0000 7fb859d9b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb85c0c39e0 0x7fb85c0c00b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45624/0 (socket says 172.21.15.195:45624) 2022-06-27T11:58:19.747 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.734+0000 7fb859d9b700 1 -- 172.21.15.195:0/837925398 learned_addr learned my addr 172.21.15.195:0/837925398 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:58:19.747 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.734+0000 7fb85a59c700 1 --2- 172.21.15.195:0/837925398 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb85c06f640 0x7fb85c0bfb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:19.748 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.734+0000 7fb85ad9d700 1 --2- 172.21.15.195:0/837925398 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb85c0c4160 0x7fb85c151c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:19.749 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.734+0000 7fb859d9b700 1 -- 172.21.15.195:0/837925398 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb85c0c4160 msgr2=0x7fb85c151c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:19.749 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.734+0000 7fb859d9b700 1 --2- 172.21.15.195:0/837925398 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb85c0c4160 0x7fb85c151c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:19.749 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.734+0000 7fb859d9b700 1 -- 172.21.15.195:0/837925398 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb85c06f640 msgr2=0x7fb85c0bfb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:19.750 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.734+0000 7fb859d9b700 1 --2- 172.21.15.195:0/837925398 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb85c06f640 0x7fb85c0bfb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:19.750 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.734+0000 7fb859d9b700 1 -- 172.21.15.195:0/837925398 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb854009a20 con 0x7fb85c0c39e0 2022-06-27T11:58:19.751 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.734+0000 7fb859d9b700 1 --2- 172.21.15.195:0/837925398 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb85c0c39e0 0x7fb85c0c00b0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fb84c00c410 tx=0x7fb84c007bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:19.751 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.735+0000 7fb8527fc700 1 -- 172.21.15.195:0/837925398 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb84c00ce10 con 0x7fb85c0c39e0 2022-06-27T11:58:19.751 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.735+0000 7fb8527fc700 1 -- 172.21.15.195:0/837925398 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb84c0072e0 con 0x7fb85c0c39e0 2022-06-27T11:58:19.752 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.735+0000 7fb860e1b700 1 -- 172.21.15.195:0/837925398 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8440092e0 con 0x7fb85c0c39e0 2022-06-27T11:58:19.752 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.735+0000 7fb8527fc700 1 -- 172.21.15.195:0/837925398 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb84c0076a0 con 0x7fb85c0c39e0 2022-06-27T11:58:19.753 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.735+0000 7fb860e1b700 1 -- 172.21.15.195:0/837925398 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb85c166db0 con 0x7fb85c0c39e0 2022-06-27T11:58:19.753 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.735+0000 7fb860e1b700 1 -- 172.21.15.195:0/837925398 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb85c0bf850 con 0x7fb85c0c39e0 2022-06-27T11:58:19.754 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.736+0000 7fb8527fc700 1 -- 172.21.15.195:0/837925398 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fb84c019030 con 0x7fb85c0c39e0 2022-06-27T11:58:19.754 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.736+0000 7fb8527fc700 1 --2- 172.21.15.195:0/837925398 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb840063d00 0x7fb8400661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:19.755 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.736+0000 7fb8527fc700 1 -- 172.21.15.195:0/837925398 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6632+0+0 (secure 0 0 0) 0x7fb84c07fa30 con 0x7fb85c0c39e0 2022-06-27T11:58:19.755 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.736+0000 7fb85a59c700 1 --2- 172.21.15.195:0/837925398 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb840063d00 0x7fb8400661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:19.755 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.737+0000 7fb85a59c700 1 --2- 172.21.15.195:0/837925398 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb840063d00 0x7fb8400661b0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb844009260 tx=0x7fb844017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:19.756 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.740+0000 7fb8527fc700 1 -- 172.21.15.195:0/837925398 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fb84c049c90 con 0x7fb85c0c39e0 2022-06-27T11:58:19.971 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.967+0000 7fb860e1b700 1 -- 172.21.15.195:0/837925398 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb85c0722c0 con 0x7fb85c0c39e0 2022-06-27T11:58:19.971 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.968+0000 7fb8527fc700 1 -- 172.21.15.195:0/837925398 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+5367 (secure 0 0 0) 0x7fb84c017020 con 0x7fb85c0c39e0 2022-06-27T11:58:19.972 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:58:19.972 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:58:09.974633+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5392},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5392":{"gid":5392,"name":"d","rank":0,"incarnation":38,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6835/2089155519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2089155519},{"type":"v1","addr":"172.21.15.202:6835","nonce":2089155519}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:58:19.973 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.970+0000 7fb860e1b700 1 -- 172.21.15.195:0/837925398 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb840063d00 msgr2=0x7fb8400661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:19.974 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.970+0000 7fb860e1b700 1 --2- 172.21.15.195:0/837925398 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb840063d00 0x7fb8400661b0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb844009260 tx=0x7fb844017000 comp rx=0 tx=0).stop 2022-06-27T11:58:19.974 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.970+0000 7fb860e1b700 1 -- 172.21.15.195:0/837925398 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb85c0c39e0 msgr2=0x7fb85c0c00b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:19.974 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.970+0000 7fb860e1b700 1 --2- 172.21.15.195:0/837925398 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb85c0c39e0 0x7fb85c0c00b0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fb84c00c410 tx=0x7fb84c007bf0 comp rx=0 tx=0).stop 2022-06-27T11:58:19.974 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.970+0000 7fb860e1b700 1 -- 172.21.15.195:0/837925398 shutdown_connections 2022-06-27T11:58:19.975 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.970+0000 7fb860e1b700 1 --2- 172.21.15.195:0/837925398 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb840063d00 0x7fb8400661b0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:19.975 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.970+0000 7fb860e1b700 1 --2- 172.21.15.195:0/837925398 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb85c0c4160 0x7fb85c151c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:19.975 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.970+0000 7fb860e1b700 1 --2- 172.21.15.195:0/837925398 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb85c06f640 0x7fb85c0bfb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:19.975 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.970+0000 7fb860e1b700 1 --2- 172.21.15.195:0/837925398 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb85c0c39e0 0x7fb85c0c00b0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:19.976 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.970+0000 7fb860e1b700 1 -- 172.21.15.195:0/837925398 >> 172.21.15.195:0/837925398 conn(0x7fb85c0c6350 msgr2=0x7fb85c152910 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:58:19.976 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.970+0000 7fb860e1b700 1 -- 172.21.15.195:0/837925398 shutdown_connections 2022-06-27T11:58:19.976 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:19.970+0000 7fb860e1b700 1 -- 172.21.15.195:0/837925398 wait complete. 2022-06-27T11:58:19.976 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 38 2022-06-27T11:58:19.985 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T11:58:21.988 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:58:22.146 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.123+0000 7f9b4d8a7700 1 Processor -- start 2022-06-27T11:58:22.147 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.124+0000 7f9b4d8a7700 1 -- start start 2022-06-27T11:58:22.147 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.124+0000 7f9b4d8a7700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9b480b8520 0x7f9b48153de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:22.148 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.124+0000 7f9b4d8a7700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f9b48068ea0 0x7f9b480c45f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:22.148 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.124+0000 7f9b4d8a7700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b48068a10 con 0x7f9b480c4b30 2022-06-27T11:58:22.148 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.124+0000 7f9b4d8a7700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b48067ed0 con 0x7f9b48068ea0 2022-06-27T11:58:22.149 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.124+0000 7f9b4d8a7700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b480692d0 con 0x7f9b480b8520 2022-06-27T11:58:22.149 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.125+0000 7f9b477fe700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f9b480c4b30 0x7f9b480c3010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:51820/0 (socket says 172.21.15.195:51820) 2022-06-27T11:58:22.150 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.125+0000 7f9b477fe700 1 -- 172.21.15.195:0/2583581757 learned_addr learned my addr 172.21.15.195:0/2583581757 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:58:22.150 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.125+0000 7f9b46ffd700 1 --2- 172.21.15.195:0/2583581757 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9b480b8520 0x7f9b48153de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:22.151 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.125+0000 7f9b467fc700 1 --2- 172.21.15.195:0/2583581757 >> v2:172.21.15.202:3300/0 conn(0x7f9b48068ea0 0x7f9b480c45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:22.151 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.125+0000 7f9b45ffb700 1 -- 172.21.15.195:0/2583581757 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1667617954 0 0) 0x7f9b48068a10 con 0x7f9b480c4b30 2022-06-27T11:58:22.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.125+0000 7f9b45ffb700 1 -- 172.21.15.195:0/2583581757 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b34003430 con 0x7f9b480c4b30 2022-06-27T11:58:22.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.125+0000 7f9b46ffd700 1 -- 172.21.15.195:0/2583581757 >> v2:172.21.15.202:3300/0 conn(0x7f9b48068ea0 msgr2=0x7f9b480c45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:58:22.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.125+0000 7f9b45ffb700 1 -- 172.21.15.195:0/2583581757 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2325829611 0 0) 0x7f9b34003430 con 0x7f9b480c4b30 2022-06-27T11:58:22.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.125+0000 7f9b46ffd700 1 --2- 172.21.15.195:0/2583581757 >> v2:172.21.15.202:3300/0 conn(0x7f9b48068ea0 0x7f9b480c45f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:22.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.125+0000 7f9b46ffd700 1 -- 172.21.15.195:0/2583581757 >> v1:172.21.15.195:6789/0 conn(0x7f9b480c4b30 legacy=0x7f9b480c3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:22.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.125+0000 7f9b46ffd700 1 -- 172.21.15.195:0/2583581757 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b480c3dd0 con 0x7f9b480b8520 2022-06-27T11:58:22.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.125+0000 7f9b467fc700 1 --2- 172.21.15.195:0/2583581757 >> v2:172.21.15.202:3300/0 conn(0x7f9b48068ea0 0x7f9b480c45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:58:22.155 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.125+0000 7f9b46ffd700 1 --2- 172.21.15.195:0/2583581757 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9b480b8520 0x7f9b48153de0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c00a940 tx=0x7f9b3c00a200 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a988701552cde028 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:22.155 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.125+0000 7f9b45ffb700 1 -- 172.21.15.195:0/2583581757 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b400017f0 con 0x7f9b480b8520 2022-06-27T11:58:22.156 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.125+0000 7f9b45ffb700 1 -- 172.21.15.195:0/2583581757 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b3c004030 con 0x7f9b480b8520 2022-06-27T11:58:22.156 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.125+0000 7f9b45ffb700 1 -- 172.21.15.195:0/2583581757 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c0043f0 con 0x7f9b480b8520 2022-06-27T11:58:22.157 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.126+0000 7f9b4d8a7700 1 -- 172.21.15.195:0/2583581757 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9b480b8520 msgr2=0x7f9b48153de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:22.157 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.126+0000 7f9b4d8a7700 1 --2- 172.21.15.195:0/2583581757 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9b480b8520 0x7f9b48153de0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c00a940 tx=0x7f9b3c00a200 comp rx=0 tx=0).stop 2022-06-27T11:58:22.158 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.126+0000 7f9b4d8a7700 1 -- 172.21.15.195:0/2583581757 shutdown_connections 2022-06-27T11:58:22.158 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.126+0000 7f9b4d8a7700 1 --2- 172.21.15.195:0/2583581757 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9b480b8520 0x7f9b48153de0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:22.159 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.126+0000 7f9b4d8a7700 1 --2- 172.21.15.195:0/2583581757 >> v2:172.21.15.202:3300/0 conn(0x7f9b48068ea0 0x7f9b480c45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:22.159 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.126+0000 7f9b4d8a7700 1 -- 172.21.15.195:0/2583581757 >> 172.21.15.195:0/2583581757 conn(0x7f9b480c8740 msgr2=0x7f9b480c8b10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:58:22.159 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.126+0000 7f9b4d8a7700 1 -- 172.21.15.195:0/2583581757 shutdown_connections 2022-06-27T11:58:22.160 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.126+0000 7f9b4d8a7700 1 -- 172.21.15.195:0/2583581757 wait complete. 2022-06-27T11:58:22.160 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.128+0000 7f9b4d8a7700 1 Processor -- start 2022-06-27T11:58:22.161 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.128+0000 7f9b4d8a7700 1 -- start start 2022-06-27T11:58:22.161 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.129+0000 7f9b4d8a7700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9b48068ea0 0x7f9b4814c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:22.162 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.129+0000 7f9b4d8a7700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9b480b8520 0x7f9b480bac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:22.162 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.129+0000 7f9b4d8a7700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9b480c4b30 0x7f9b480bb170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:22.163 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.129+0000 7f9b4d8a7700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b480c1bd0 con 0x7f9b480c4b30 2022-06-27T11:58:22.163 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.129+0000 7f9b4d8a7700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b480ba7a0 con 0x7f9b48068ea0 2022-06-27T11:58:22.164 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.129+0000 7f9b4d8a7700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b4806f990 con 0x7f9b480b8520 2022-06-27T11:58:22.164 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.129+0000 7f9b477fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9b480c4b30 0x7f9b480bb170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:22.164 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.129+0000 7f9b477fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9b480c4b30 0x7f9b480bb170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45640/0 (socket says 172.21.15.195:45640) 2022-06-27T11:58:22.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.129+0000 7f9b477fe700 1 -- 172.21.15.195:0/2918734848 learned_addr learned my addr 172.21.15.195:0/2918734848 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:58:22.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.129+0000 7f9b46ffd700 1 --2- 172.21.15.195:0/2918734848 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9b48068ea0 0x7f9b4814c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:22.166 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.129+0000 7f9b467fc700 1 --2- 172.21.15.195:0/2918734848 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9b480b8520 0x7f9b480bac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:22.166 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.129+0000 7f9b477fe700 1 -- 172.21.15.195:0/2918734848 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9b480b8520 msgr2=0x7f9b480bac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:22.167 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.129+0000 7f9b477fe700 1 --2- 172.21.15.195:0/2918734848 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9b480b8520 0x7f9b480bac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:22.167 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.129+0000 7f9b477fe700 1 -- 172.21.15.195:0/2918734848 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9b48068ea0 msgr2=0x7f9b4814c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:22.168 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.129+0000 7f9b477fe700 1 --2- 172.21.15.195:0/2918734848 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9b48068ea0 0x7f9b4814c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:22.168 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.129+0000 7f9b477fe700 1 -- 172.21.15.195:0/2918734848 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b380092e0 con 0x7f9b480c4b30 2022-06-27T11:58:22.169 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.129+0000 7f9b477fe700 1 --2- 172.21.15.195:0/2918734848 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9b480c4b30 0x7f9b480bb170 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f9b4000cb90 tx=0x7f9b40007b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:22.170 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.129+0000 7f9b2effd700 1 -- 172.21.15.195:0/2918734848 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b4000cea0 con 0x7f9b480c4b30 2022-06-27T11:58:22.170 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.130+0000 7f9b2effd700 1 -- 172.21.15.195:0/2918734848 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b40004c80 con 0x7f9b480c4b30 2022-06-27T11:58:22.171 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.130+0000 7f9b4d8a7700 1 -- 172.21.15.195:0/2918734848 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b3c00a480 con 0x7f9b480c4b30 2022-06-27T11:58:22.171 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.130+0000 7f9b2effd700 1 -- 172.21.15.195:0/2918734848 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b400071b0 con 0x7f9b480c4b30 2022-06-27T11:58:22.172 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.130+0000 7f9b4d8a7700 1 -- 172.21.15.195:0/2918734848 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b4806f6b0 con 0x7f9b480c4b30 2022-06-27T11:58:22.172 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.130+0000 7f9b4d8a7700 1 -- 172.21.15.195:0/2918734848 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b480722c0 con 0x7f9b480c4b30 2022-06-27T11:58:22.173 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.131+0000 7f9b2effd700 1 -- 172.21.15.195:0/2918734848 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f9b4001a030 con 0x7f9b480c4b30 2022-06-27T11:58:22.173 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.131+0000 7f9b2effd700 1 --2- 172.21.15.195:0/2918734848 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9b28063d00 0x7f9b280661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:22.174 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.131+0000 7f9b2effd700 1 -- 172.21.15.195:0/2918734848 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6632+0+0 (secure 0 0 0) 0x7f9b40080130 con 0x7f9b480c4b30 2022-06-27T11:58:22.174 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.131+0000 7f9b46ffd700 1 --2- 172.21.15.195:0/2918734848 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9b28063d00 0x7f9b280661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:22.175 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.132+0000 7f9b46ffd700 1 --2- 172.21.15.195:0/2918734848 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9b28063d00 0x7f9b280661b0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c00a240 tx=0x7f9b3c019000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:22.175 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.136+0000 7f9b2effd700 1 -- 172.21.15.195:0/2918734848 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f9b4004eb60 con 0x7f9b480c4b30 2022-06-27T11:58:22.363 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.360+0000 7f9b4d8a7700 1 -- 172.21.15.195:0/2918734848 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9b480ba8e0 con 0x7f9b480c4b30 2022-06-27T11:58:22.363 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.360+0000 7f9b2effd700 1 -- 172.21.15.195:0/2918734848 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+5371 (secure 0 0 0) 0x7f9b40018020 con 0x7f9b480c4b30 2022-06-27T11:58:22.364 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:58:22.365 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:58:21.043163+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5392},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5392":{"gid":5392,"name":"d","rank":0,"incarnation":38,"state":"up:reconnect","state_seq":13,"addr":"172.21.15.202:6835/2089155519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2089155519},{"type":"v1","addr":"172.21.15.202:6835","nonce":2089155519}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:58:22.367 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.362+0000 7f9b4d8a7700 1 -- 172.21.15.195:0/2918734848 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9b28063d00 msgr2=0x7f9b280661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:22.367 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.362+0000 7f9b4d8a7700 1 --2- 172.21.15.195:0/2918734848 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9b28063d00 0x7f9b280661b0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c00a240 tx=0x7f9b3c019000 comp rx=0 tx=0).stop 2022-06-27T11:58:22.368 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.362+0000 7f9b4d8a7700 1 -- 172.21.15.195:0/2918734848 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9b480c4b30 msgr2=0x7f9b480bb170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:22.368 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.362+0000 7f9b4d8a7700 1 --2- 172.21.15.195:0/2918734848 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9b480c4b30 0x7f9b480bb170 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f9b4000cb90 tx=0x7f9b40007b30 comp rx=0 tx=0).stop 2022-06-27T11:58:22.369 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.362+0000 7f9b4d8a7700 1 -- 172.21.15.195:0/2918734848 shutdown_connections 2022-06-27T11:58:22.369 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.362+0000 7f9b4d8a7700 1 --2- 172.21.15.195:0/2918734848 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9b28063d00 0x7f9b280661b0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:22.369 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.362+0000 7f9b4d8a7700 1 --2- 172.21.15.195:0/2918734848 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9b480c4b30 0x7f9b480bb170 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:22.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.362+0000 7f9b4d8a7700 1 --2- 172.21.15.195:0/2918734848 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9b48068ea0 0x7f9b4814c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:22.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.362+0000 7f9b4d8a7700 1 --2- 172.21.15.195:0/2918734848 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9b480b8520 0x7f9b480bac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:22.373 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.362+0000 7f9b4d8a7700 1 -- 172.21.15.195:0/2918734848 >> 172.21.15.195:0/2918734848 conn(0x7f9b480c8740 msgr2=0x7f9b48151ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:58:22.374 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.362+0000 7f9b4d8a7700 1 -- 172.21.15.195:0/2918734848 shutdown_connections 2022-06-27T11:58:22.374 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:22.362+0000 7f9b4d8a7700 1 -- 172.21.15.195:0/2918734848 wait complete. 2022-06-27T11:58:22.375 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 39 2022-06-27T11:58:22.378 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 39, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-06-27T11:47:52.358095+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 39, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5392": { "addr": "172.21.15.202:6835/2089155519", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6834", "nonce": 2089155519, "type": "v2" }, { "addr": "172.21.15.202:6835", "nonce": 2089155519, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5392, "incarnation": 38, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:reconnect", "state_seq": 13 } }, "last_failure": 0, "last_failure_osd_epoch": 53, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-06-27T11:58:21.043163+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5392 } } } ], "standbys": [ { "addr": "172.21.15.202:6837/3305006841", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6836", "nonce": 3305006841, "type": "v2" }, { "addr": "172.21.15.202:6837", "nonce": 3305006841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6835/2060709822", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6834", "nonce": 2060709822, "type": "v2" }, { "addr": "172.21.15.195:6835", "nonce": 2060709822, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6838/1598751652", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6836", "nonce": 1598751652, "type": "v2" }, { "addr": "172.21.15.195:6838", "nonce": 1598751652, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4275, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6839/1313800661", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6837", "nonce": 1313800661, "type": "v2" }, { "addr": "172.21.15.195:6839", "nonce": 1313800661, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4278, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-06-27T11:58:24.380 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:58:24.516 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.516+0000 7f720bee4700 1 Processor -- start 2022-06-27T11:58:24.516 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.516+0000 7f720bee4700 1 -- start start 2022-06-27T11:58:24.519 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.517+0000 7f720bee4700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f720406e850 0x7f72041570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:24.520 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.517+0000 7f720bee4700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f72040c34a0 0x7f72040c2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:24.520 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.517+0000 7f720bee4700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f720406fba0 con 0x7f720406fef0 2022-06-27T11:58:24.520 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.517+0000 7f720bee4700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72041575e0 con 0x7f720406e850 2022-06-27T11:58:24.520 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.517+0000 7f720bee4700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f720406f5e0 con 0x7f72040c34a0 2022-06-27T11:58:24.521 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.517+0000 7f7209c80700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f720406e850 0x7f72041570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:24.521 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.517+0000 7f720947f700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f720406fef0 0x7f7204069de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:51832/0 (socket says 172.21.15.195:51832) 2022-06-27T11:58:24.521 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.517+0000 7f720947f700 1 -- 172.21.15.195:0/163842279 learned_addr learned my addr 172.21.15.195:0/163842279 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:58:24.521 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.517+0000 7f720a481700 1 --2- 172.21.15.195:0/163842279 >> v2:172.21.15.202:3300/0 conn(0x7f72040c34a0 0x7f72040c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:24.522 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.517+0000 7f7209c80700 1 -- 172.21.15.195:0/163842279 >> v2:172.21.15.202:3300/0 conn(0x7f72040c34a0 msgr2=0x7f72040c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:58:24.522 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.517+0000 7f7209c80700 1 --2- 172.21.15.195:0/163842279 >> v2:172.21.15.202:3300/0 conn(0x7f72040c34a0 0x7f72040c2920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:24.522 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.517+0000 7f7209c80700 1 -- 172.21.15.195:0/163842279 >> v1:172.21.15.195:6789/0 conn(0x7f720406fef0 legacy=0x7f7204069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:24.523 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.517+0000 7f7209c80700 1 -- 172.21.15.195:0/163842279 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f720406f8c0 con 0x7f720406e850 2022-06-27T11:58:24.523 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.517+0000 7f7208c7e700 1 -- 172.21.15.195:0/163842279 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1634668368 0 0) 0x7f720406fba0 con 0x7f720406fef0 2022-06-27T11:58:24.523 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.518+0000 7f7209c80700 1 --2- 172.21.15.195:0/163842279 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f720406e850 0x7f72041570a0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f72040c1420 tx=0x7f71f4009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bab11e0699b6734 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:24.523 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.518+0000 7f7208c7e700 1 -- 172.21.15.195:0/163842279 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f4011630 con 0x7f720406e850 2022-06-27T11:58:24.524 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.518+0000 7f7208c7e700 1 -- 172.21.15.195:0/163842279 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71f4011790 con 0x7f720406e850 2022-06-27T11:58:24.524 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.518+0000 7f7208c7e700 1 -- 172.21.15.195:0/163842279 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f4011b30 con 0x7f720406e850 2022-06-27T11:58:24.524 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.518+0000 7f720bee4700 1 -- 172.21.15.195:0/163842279 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f720406e850 msgr2=0x7f72041570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:24.525 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.518+0000 7f720bee4700 1 --2- 172.21.15.195:0/163842279 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f720406e850 0x7f72041570a0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f72040c1420 tx=0x7f71f4009060 comp rx=0 tx=0).stop 2022-06-27T11:58:24.525 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.518+0000 7f720bee4700 1 -- 172.21.15.195:0/163842279 shutdown_connections 2022-06-27T11:58:24.525 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.518+0000 7f720bee4700 1 --2- 172.21.15.195:0/163842279 >> v2:172.21.15.202:3300/0 conn(0x7f72040c34a0 0x7f72040c2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:24.525 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.518+0000 7f720bee4700 1 --2- 172.21.15.195:0/163842279 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f720406e850 0x7f72041570a0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:24.526 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.518+0000 7f720bee4700 1 -- 172.21.15.195:0/163842279 >> 172.21.15.195:0/163842279 conn(0x7f72040c6350 msgr2=0x7f72040c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:58:24.526 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.518+0000 7f720bee4700 1 -- 172.21.15.195:0/163842279 shutdown_connections 2022-06-27T11:58:24.527 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.518+0000 7f720bee4700 1 -- 172.21.15.195:0/163842279 wait complete. 2022-06-27T11:58:24.527 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.520+0000 7f720bee4700 1 Processor -- start 2022-06-27T11:58:24.527 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.520+0000 7f720bee4700 1 -- start start 2022-06-27T11:58:24.536 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.520+0000 7f720bee4700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f720406e850 0x7f7204152d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:24.536 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.520+0000 7f720bee4700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f72040c34a0 0x7f72041523e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:24.537 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.520+0000 7f720bee4700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7204150ed0 0x7f7204151320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:24.537 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.520+0000 7f720bee4700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72041537a0 con 0x7f720406e850 2022-06-27T11:58:24.537 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.520+0000 7f720bee4700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7204150200 con 0x7f7204150ed0 2022-06-27T11:58:24.538 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.520+0000 7f720bee4700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f720414fd70 con 0x7f72040c34a0 2022-06-27T11:58:24.538 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.520+0000 7f7209c80700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f720406e850 0x7f7204152d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:24.538 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.520+0000 7f7209c80700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f720406e850 0x7f7204152d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45650/0 (socket says 172.21.15.195:45650) 2022-06-27T11:58:24.539 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.520+0000 7f7209c80700 1 -- 172.21.15.195:0/2767267613 learned_addr learned my addr 172.21.15.195:0/2767267613 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:58:24.539 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.521+0000 7f720a481700 1 --2- 172.21.15.195:0/2767267613 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7204150ed0 0x7f7204151320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:24.539 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.521+0000 7f720947f700 1 --2- 172.21.15.195:0/2767267613 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f72040c34a0 0x7f72041523e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:24.539 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.521+0000 7f7209c80700 1 -- 172.21.15.195:0/2767267613 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f72040c34a0 msgr2=0x7f72041523e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:24.539 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.521+0000 7f7209c80700 1 --2- 172.21.15.195:0/2767267613 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f72040c34a0 0x7f72041523e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:24.540 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.521+0000 7f7209c80700 1 -- 172.21.15.195:0/2767267613 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7204150ed0 msgr2=0x7f7204151320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:24.540 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.521+0000 7f7209c80700 1 --2- 172.21.15.195:0/2767267613 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7204150ed0 0x7f7204151320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:24.540 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.521+0000 7f7209c80700 1 -- 172.21.15.195:0/2767267613 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71f40092e0 con 0x7f720406e850 2022-06-27T11:58:24.540 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.521+0000 7f720947f700 1 --2- 172.21.15.195:0/2767267613 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f72040c34a0 0x7f72041523e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T11:58:24.541 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.521+0000 7f7209c80700 1 --2- 172.21.15.195:0/2767267613 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f720406e850 0x7f7204152d10 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f71f4009ec0 tx=0x7f71f4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:24.541 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.521+0000 7f71f1ffb700 1 -- 172.21.15.195:0/2767267613 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f4003900 con 0x7f720406e850 2022-06-27T11:58:24.541 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.521+0000 7f71f1ffb700 1 -- 172.21.15.195:0/2767267613 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71f4003a60 con 0x7f720406e850 2022-06-27T11:58:24.541 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.521+0000 7f720bee4700 1 -- 172.21.15.195:0/2767267613 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7204153250 con 0x7f720406e850 2022-06-27T11:58:24.542 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.521+0000 7f71f1ffb700 1 -- 172.21.15.195:0/2767267613 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f4003e20 con 0x7f720406e850 2022-06-27T11:58:24.542 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.521+0000 7f720bee4700 1 -- 172.21.15.195:0/2767267613 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72041534c0 con 0x7f720406e850 2022-06-27T11:58:24.542 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.522+0000 7f720bee4700 1 -- 172.21.15.195:0/2767267613 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7204150050 con 0x7f720406e850 2022-06-27T11:58:24.542 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.522+0000 7f71f1ffb700 1 -- 172.21.15.195:0/2767267613 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f71f400f050 con 0x7f720406e850 2022-06-27T11:58:24.543 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.523+0000 7f71f1ffb700 1 --2- 172.21.15.195:0/2767267613 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f71e4063d00 0x7f71e40661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:24.543 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.523+0000 7f71f1ffb700 1 -- 172.21.15.195:0/2767267613 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6632+0+0 (secure 0 0 0) 0x7f71f40821a0 con 0x7f720406e850 2022-06-27T11:58:24.543 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.523+0000 7f720947f700 1 --2- 172.21.15.195:0/2767267613 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f71e4063d00 0x7f71e40661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:24.543 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.523+0000 7f720947f700 1 --2- 172.21.15.195:0/2767267613 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f71e4063d00 0x7f71e40661b0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f72040c94d0 tx=0x7f7200009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:24.544 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.527+0000 7f71f1ffb700 1 -- 172.21.15.195:0/2767267613 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f71f4050af0 con 0x7f720406e850 2022-06-27T11:58:24.752 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.750+0000 7f720bee4700 1 -- 172.21.15.195:0/2767267613 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f72040722c0 con 0x7f720406e850 2022-06-27T11:58:24.753 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.750+0000 7f71f1ffb700 1 -- 172.21.15.195:0/2767267613 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+5371 (secure 0 0 0) 0x7f71f4054110 con 0x7f720406e850 2022-06-27T11:58:24.754 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:58:24.754 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:58:21.043163+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5392},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5392":{"gid":5392,"name":"d","rank":0,"incarnation":38,"state":"up:reconnect","state_seq":13,"addr":"172.21.15.202:6835/2089155519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2089155519},{"type":"v1","addr":"172.21.15.202:6835","nonce":2089155519}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:58:24.756 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.752+0000 7f720bee4700 1 -- 172.21.15.195:0/2767267613 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f71e4063d00 msgr2=0x7f71e40661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:24.757 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.752+0000 7f720bee4700 1 --2- 172.21.15.195:0/2767267613 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f71e4063d00 0x7f71e40661b0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f72040c94d0 tx=0x7f7200009040 comp rx=0 tx=0).stop 2022-06-27T11:58:24.757 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.752+0000 7f720bee4700 1 -- 172.21.15.195:0/2767267613 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f720406e850 msgr2=0x7f7204152d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:24.758 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.752+0000 7f720bee4700 1 --2- 172.21.15.195:0/2767267613 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f720406e850 0x7f7204152d10 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f71f4009ec0 tx=0x7f71f4017000 comp rx=0 tx=0).stop 2022-06-27T11:58:24.758 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.752+0000 7f720bee4700 1 -- 172.21.15.195:0/2767267613 shutdown_connections 2022-06-27T11:58:24.759 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.752+0000 7f720bee4700 1 --2- 172.21.15.195:0/2767267613 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f71e4063d00 0x7f71e40661b0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:24.759 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.752+0000 7f720bee4700 1 --2- 172.21.15.195:0/2767267613 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f720406e850 0x7f7204152d10 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:24.759 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.752+0000 7f720bee4700 1 --2- 172.21.15.195:0/2767267613 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7204150ed0 0x7f7204151320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:24.760 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.752+0000 7f720bee4700 1 --2- 172.21.15.195:0/2767267613 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f72040c34a0 0x7f72041523e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:24.760 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.752+0000 7f720bee4700 1 -- 172.21.15.195:0/2767267613 >> 172.21.15.195:0/2767267613 conn(0x7f72040c6350 msgr2=0x7f72040c6720 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:58:24.761 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.752+0000 7f720bee4700 1 -- 172.21.15.195:0/2767267613 shutdown_connections 2022-06-27T11:58:24.761 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:24.752+0000 7f720bee4700 1 -- 172.21.15.195:0/2767267613 wait complete. 2022-06-27T11:58:24.761 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 39 2022-06-27T11:58:24.765 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T11:58:26.777 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:58:26.924 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:26.922+0000 7f79ed07b700 1 Processor -- start 2022-06-27T11:58:26.924 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:26.923+0000 7f79ed07b700 1 -- start start 2022-06-27T11:58:26.925 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:26.923+0000 7f79ed07b700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f79e806cbc0 0x7f79e8150190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:26.925 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:26.923+0000 7f79ed07b700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79e806f860 con 0x7f79e806dcc0 2022-06-27T11:58:26.925 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:26.923+0000 7f79ed07b700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79e8156ef0 con 0x7f79e8069ca0 2022-06-27T11:58:26.926 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:26.923+0000 7f79ed07b700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79e80694d0 con 0x7f79e806cbc0 2022-06-27T11:58:26.926 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:26.923+0000 7f79e759e700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f79e806dcc0 0x7f79e8160c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:51850/0 (socket says 172.21.15.195:51850) 2022-06-27T11:58:26.926 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:26.923+0000 7f79e759e700 1 -- 172.21.15.195:0/3043785668 learned_addr learned my addr 172.21.15.195:0/3043785668 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:58:26.926 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:26.924+0000 7f79e659c700 1 --2- 172.21.15.195:0/3043785668 >> v2:172.21.15.202:3300/0 conn(0x7f79e806cbc0 0x7f79e8150190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:26.926 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:26.924+0000 7f79e5d9b700 1 -- 172.21.15.195:0/3043785668 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4290601372 0 0) 0x7f79e806f860 con 0x7f79e806dcc0 2022-06-27T11:58:26.927 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:26.924+0000 7f79e5d9b700 1 -- 172.21.15.195:0/3043785668 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79d4003430 con 0x7f79e806dcc0 2022-06-27T11:58:26.927 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:26.924+0000 7f79e659c700 1 -- 172.21.15.195:0/3043785668 >> v1:172.21.15.202:6789/0 conn(0x7f79e8069ca0 legacy=0x7f79e806fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:26.927 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:26.924+0000 7f79e5d9b700 1 -- 172.21.15.195:0/3043785668 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2652375987 0 0) 0x7f79d4003430 con 0x7f79e806dcc0 2022-06-27T11:58:26.927 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:26.924+0000 7f79e659c700 1 -- 172.21.15.195:0/3043785668 >> v1:172.21.15.195:6789/0 conn(0x7f79e806dcc0 legacy=0x7f79e8160c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:26.928 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:26.924+0000 7f79e659c700 1 -- 172.21.15.195:0/3043785668 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79e806f210 con 0x7f79e806cbc0 2022-06-27T11:58:26.928 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:26.924+0000 7f79e659c700 1 --2- 172.21.15.195:0/3043785668 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f79e806cbc0 0x7f79e8150190 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f79dc0097a0 tx=0x7f79dc009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9db2489ed0b4c843 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:26.928 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:26.924+0000 7f79e5d9b700 1 -- 172.21.15.195:0/3043785668 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79e00017f0 con 0x7f79e806cbc0 2022-06-27T11:58:26.928 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:26.924+0000 7f79e5d9b700 1 -- 172.21.15.195:0/3043785668 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79dc004030 con 0x7f79e806cbc0 2022-06-27T11:58:26.929 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:26.924+0000 7f79e5d9b700 1 -- 172.21.15.195:0/3043785668 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79dc0043b0 con 0x7f79e806cbc0 2022-06-27T11:58:26.933 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:26.925+0000 7f79ed07b700 1 -- 172.21.15.195:0/3043785668 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f79e806cbc0 msgr2=0x7f79e8150190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:26.933 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:26.925+0000 7f79ed07b700 1 --2- 172.21.15.195:0/3043785668 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f79e806cbc0 0x7f79e8150190 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f79dc0097a0 tx=0x7f79dc009060 comp rx=0 tx=0).stop 2022-06-27T11:58:26.933 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:26.925+0000 7f79ed07b700 1 -- 172.21.15.195:0/3043785668 shutdown_connections 2022-06-27T11:58:26.934 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:26.925+0000 7f79ed07b700 1 --2- 172.21.15.195:0/3043785668 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f79e806cbc0 0x7f79e8150190 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:26.934 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:26.925+0000 7f79ed07b700 1 -- 172.21.15.195:0/3043785668 >> 172.21.15.195:0/3043785668 conn(0x7f79e80ca2a0 msgr2=0x7f79e80c6270 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:58:26.934 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:26.925+0000 7f79ed07b700 1 -- 172.21.15.195:0/3043785668 shutdown_connections 2022-06-27T11:58:26.934 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:26.925+0000 7f79ed07b700 1 -- 172.21.15.195:0/3043785668 wait complete. 2022-06-27T11:58:26.934 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:26.926+0000 7f79ed07b700 1 Processor -- start 2022-06-27T11:58:26.935 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:26.926+0000 7f79ed07b700 1 -- start start 2022-06-27T11:58:26.935 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:26.927+0000 7f79ed07b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f79e8069ca0 0x7f79e8166c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:26.935 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:26.927+0000 7f79ed07b700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f79e806cbc0 0x7f79e8157a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:26.935 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:26.927+0000 7f79ed07b700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f79e806dcc0 0x7f79e8146050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:26.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:26.927+0000 7f79ed07b700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79e8150170 con 0x7f79e8069ca0 2022-06-27T11:58:26.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:26.927+0000 7f79ed07b700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79e814d1d0 con 0x7f79e806dcc0 2022-06-27T11:58:26.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:26.927+0000 7f79ed07b700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79e81599b0 con 0x7f79e806cbc0 2022-06-27T11:58:26.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:26.927+0000 7f79e6d9d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f79e8069ca0 0x7f79e8166c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:26.937 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:26.927+0000 7f79e6d9d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f79e8069ca0 0x7f79e8166c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45664/0 (socket says 172.21.15.195:45664) 2022-06-27T11:58:26.937 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:26.927+0000 7f79e6d9d700 1 -- 172.21.15.195:0/2206935889 learned_addr learned my addr 172.21.15.195:0/2206935889 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:58:26.937 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:26.927+0000 7f79e759e700 1 --2- 172.21.15.195:0/2206935889 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f79e806dcc0 0x7f79e8146050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:26.937 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:26.927+0000 7f79e659c700 1 --2- 172.21.15.195:0/2206935889 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f79e806cbc0 0x7f79e8157a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:26.938 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:26.927+0000 7f79e6d9d700 1 -- 172.21.15.195:0/2206935889 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f79e806cbc0 msgr2=0x7f79e8157a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:26.938 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:26.927+0000 7f79e6d9d700 1 --2- 172.21.15.195:0/2206935889 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f79e806cbc0 0x7f79e8157a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:26.938 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:26.927+0000 7f79e6d9d700 1 -- 172.21.15.195:0/2206935889 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f79e806dcc0 msgr2=0x7f79e8146050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:26.938 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:26.927+0000 7f79e6d9d700 1 --2- 172.21.15.195:0/2206935889 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f79e806dcc0 0x7f79e8146050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:26.939 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:26.927+0000 7f79e6d9d700 1 -- 172.21.15.195:0/2206935889 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79dc0092e0 con 0x7f79e8069ca0 2022-06-27T11:58:26.939 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:26.928+0000 7f79e6d9d700 1 --2- 172.21.15.195:0/2206935889 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f79e8069ca0 0x7f79e8166c50 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f79d800c590 tx=0x7f79d8007e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:26.939 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:26.928+0000 7f79ceffd700 1 -- 172.21.15.195:0/2206935889 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79d80050d0 con 0x7f79e8069ca0 2022-06-27T11:58:26.939 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:26.928+0000 7f79ceffd700 1 -- 172.21.15.195:0/2206935889 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79d8005230 con 0x7f79e8069ca0 2022-06-27T11:58:26.939 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:26.928+0000 7f79ed07b700 1 -- 172.21.15.195:0/2206935889 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79e80bc730 con 0x7f79e8069ca0 2022-06-27T11:58:26.940 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:26.928+0000 7f79ceffd700 1 -- 172.21.15.195:0/2206935889 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79d80117e0 con 0x7f79e8069ca0 2022-06-27T11:58:26.940 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:26.928+0000 7f79ed07b700 1 -- 172.21.15.195:0/2206935889 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79e814d3e0 con 0x7f79e8069ca0 2022-06-27T11:58:26.940 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:26.929+0000 7f79ed07b700 1 -- 172.21.15.195:0/2206935889 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79e814e340 con 0x7f79e8069ca0 2022-06-27T11:58:26.940 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:26.929+0000 7f79ceffd700 1 -- 172.21.15.195:0/2206935889 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f79d8019030 con 0x7f79e8069ca0 2022-06-27T11:58:26.941 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:26.929+0000 7f79ceffd700 1 --2- 172.21.15.195:0/2206935889 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f79c8063d00 0x7f79c80661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:26.941 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:26.930+0000 7f79ceffd700 1 -- 172.21.15.195:0/2206935889 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6632+0+0 (secure 0 0 0) 0x7f79d807fc90 con 0x7f79e8069ca0 2022-06-27T11:58:26.941 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:26.930+0000 7f79e659c700 1 --2- 172.21.15.195:0/2206935889 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f79c8063d00 0x7f79c80661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:26.941 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:26.930+0000 7f79e659c700 1 --2- 172.21.15.195:0/2206935889 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f79c8063d00 0x7f79c80661b0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f79e806cfc0 tx=0x7f79dc016000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:26.942 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:26.934+0000 7f79ceffd700 1 -- 172.21.15.195:0/2206935889 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f79d804e640 con 0x7f79e8069ca0 2022-06-27T11:58:27.161 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:27.161+0000 7f79ed07b700 1 -- 172.21.15.195:0/2206935889 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f79e815a0b0 con 0x7f79e8069ca0 2022-06-27T11:58:27.162 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:27.162+0000 7f79ceffd700 1 -- 172.21.15.195:0/2206935889 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+5368 (secure 0 0 0) 0x7f79d8017020 con 0x7f79e8069ca0 2022-06-27T11:58:27.163 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:58:27.163 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:58:26.057859+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5392},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5392":{"gid":5392,"name":"d","rank":0,"incarnation":38,"state":"up:rejoin","state_seq":15,"addr":"172.21.15.202:6835/2089155519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2089155519},{"type":"v1","addr":"172.21.15.202:6835","nonce":2089155519}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:58:27.166 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:27.164+0000 7f79ed07b700 1 -- 172.21.15.195:0/2206935889 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f79c8063d00 msgr2=0x7f79c80661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:27.166 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:27.164+0000 7f79ed07b700 1 --2- 172.21.15.195:0/2206935889 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f79c8063d00 0x7f79c80661b0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f79e806cfc0 tx=0x7f79dc016000 comp rx=0 tx=0).stop 2022-06-27T11:58:27.166 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:27.164+0000 7f79ed07b700 1 -- 172.21.15.195:0/2206935889 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f79e8069ca0 msgr2=0x7f79e8166c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:27.167 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:27.164+0000 7f79ed07b700 1 --2- 172.21.15.195:0/2206935889 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f79e8069ca0 0x7f79e8166c50 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f79d800c590 tx=0x7f79d8007e20 comp rx=0 tx=0).stop 2022-06-27T11:58:27.167 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:27.164+0000 7f79ed07b700 1 -- 172.21.15.195:0/2206935889 shutdown_connections 2022-06-27T11:58:27.167 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:27.164+0000 7f79ed07b700 1 --2- 172.21.15.195:0/2206935889 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f79c8063d00 0x7f79c80661b0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:27.167 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:27.164+0000 7f79ed07b700 1 --2- 172.21.15.195:0/2206935889 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f79e8069ca0 0x7f79e8166c50 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:27.168 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:27.164+0000 7f79ed07b700 1 --2- 172.21.15.195:0/2206935889 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f79e806dcc0 0x7f79e8146050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:27.168 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:27.164+0000 7f79ed07b700 1 --2- 172.21.15.195:0/2206935889 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f79e806cbc0 0x7f79e8157a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:27.169 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:27.164+0000 7f79ed07b700 1 -- 172.21.15.195:0/2206935889 >> 172.21.15.195:0/2206935889 conn(0x7f79e80ca2a0 msgr2=0x7f79e81558a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:58:27.169 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:27.164+0000 7f79ed07b700 1 -- 172.21.15.195:0/2206935889 shutdown_connections 2022-06-27T11:58:27.169 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:27.164+0000 7f79ed07b700 1 -- 172.21.15.195:0/2206935889 wait complete. 2022-06-27T11:58:27.170 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 40 2022-06-27T11:58:27.176 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T11:58:29.147 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T11:58:29.152 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T11:58:29.177 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:58:29.337 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.337+0000 7f563d35a700 1 Processor -- start 2022-06-27T11:58:29.337 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.337+0000 7f563d35a700 1 -- start start 2022-06-27T11:58:29.349 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.338+0000 7f563d35a700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f56380c7480 0x7f56380c4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:29.351 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.338+0000 7f563d35a700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56381573b0 con 0x7f5638068f80 2022-06-27T11:58:29.351 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.338+0000 7f563d35a700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56380c46c0 con 0x7f56380683c0 2022-06-27T11:58:29.352 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.338+0000 7f563d35a700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5638068840 con 0x7f56380c7480 2022-06-27T11:58:29.352 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.338+0000 7f5636ffd700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f5638068f80 0x7f5638156460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:51860/0 (socket says 172.21.15.195:51860) 2022-06-27T11:58:29.353 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.338+0000 7f5636ffd700 1 -- 172.21.15.195:0/3840096513 learned_addr learned my addr 172.21.15.195:0/3840096513 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:58:29.353 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.338+0000 7f5635ffb700 1 -- 172.21.15.195:0/3840096513 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2885994145 0 0) 0x7f56381573b0 con 0x7f5638068f80 2022-06-27T11:58:29.353 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.338+0000 7f5635ffb700 1 -- 172.21.15.195:0/3840096513 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5624003430 con 0x7f5638068f80 2022-06-27T11:58:29.354 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.338+0000 7f56367fc700 1 --2- 172.21.15.195:0/3840096513 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f56380c7480 0x7f56380c4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:29.354 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.338+0000 7f5635ffb700 1 -- 172.21.15.195:0/3840096513 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 790983131 0 0) 0x7f5624003430 con 0x7f5638068f80 2022-06-27T11:58:29.355 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.338+0000 7f5635ffb700 1 -- 172.21.15.195:0/3840096513 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f56380c7480 msgr2=0x7f56380c4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:58:29.355 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.338+0000 7f5635ffb700 1 --2- 172.21.15.195:0/3840096513 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f56380c7480 0x7f56380c4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:29.356 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.338+0000 7f5635ffb700 1 -- 172.21.15.195:0/3840096513 >> v1:172.21.15.202:6789/0 conn(0x7f56380683c0 legacy=0x7f56380c6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:29.356 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.338+0000 7f5635ffb700 1 -- 172.21.15.195:0/3840096513 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5638156b70 con 0x7f5638068f80 2022-06-27T11:58:29.357 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.338+0000 7f5635ffb700 1 -- 172.21.15.195:0/3840096513 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7f5628001470 con 0x7f5638068f80 2022-06-27T11:58:29.357 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.338+0000 7f5635ffb700 1 -- 172.21.15.195:0/3840096513 >> v1:172.21.15.195:6789/0 conn(0x7f5638068f80 legacy=0x7f5638156460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:29.357 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.338+0000 7f5635ffb700 1 --2- 172.21.15.195:0/3840096513 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f56380c7480 0x7f5624003950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:29.358 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.338+0000 7f5635ffb700 1 --2- 172.21.15.195:0/3840096513 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5624003f80 0x7f56240046d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:29.358 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.338+0000 7f5635ffb700 1 --2- 172.21.15.195:0/3840096513 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5624004c60 0x7f56240050b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:29.359 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.338+0000 7f5635ffb700 1 -- 172.21.15.195:0/3840096513 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5638068840 con 0x7f5624003f80 2022-06-27T11:58:29.359 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.338+0000 7f5635ffb700 1 -- 172.21.15.195:0/3840096513 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f562400bb30 con 0x7f5624004c60 2022-06-27T11:58:29.360 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.338+0000 7f5635ffb700 1 -- 172.21.15.195:0/3840096513 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f562400bcf0 con 0x7f56380c7480 2022-06-27T11:58:29.360 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.339+0000 7f56367fc700 1 --2- 172.21.15.195:0/3840096513 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5624003f80 0x7f56240046d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:29.361 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.339+0000 7f5636ffd700 1 --2- 172.21.15.195:0/3840096513 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f56380c7480 0x7f5624003950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:29.362 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.339+0000 7f56377fe700 1 --2- 172.21.15.195:0/3840096513 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5624004c60 0x7f56240050b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:29.362 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.339+0000 7f56367fc700 1 -- 172.21.15.195:0/3840096513 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f56380c7480 msgr2=0x7f5624003950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:58:29.363 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.339+0000 7f56367fc700 1 --2- 172.21.15.195:0/3840096513 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f56380c7480 0x7f5624003950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:29.363 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.339+0000 7f56367fc700 1 -- 172.21.15.195:0/3840096513 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5624004c60 msgr2=0x7f56240050b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:58:29.363 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.339+0000 7f56367fc700 1 --2- 172.21.15.195:0/3840096513 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5624004c60 0x7f56240050b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:29.364 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.339+0000 7f56367fc700 1 -- 172.21.15.195:0/3840096513 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5624003430 con 0x7f5624003f80 2022-06-27T11:58:29.365 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.339+0000 7f56367fc700 1 --2- 172.21.15.195:0/3840096513 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5624003f80 0x7f56240046d0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f562c00ba40 tx=0x7f562c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fe511b65049e166 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:29.365 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.339+0000 7f5635ffb700 1 -- 172.21.15.195:0/3840096513 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f562c00be80 con 0x7f5624003f80 2022-06-27T11:58:29.366 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.339+0000 7f5635ffb700 1 -- 172.21.15.195:0/3840096513 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f562c00d5d0 con 0x7f5624003f80 2022-06-27T11:58:29.367 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.339+0000 7f5635ffb700 1 -- 172.21.15.195:0/3840096513 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f562c00d970 con 0x7f5624003f80 2022-06-27T11:58:29.367 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.340+0000 7f563d35a700 1 -- 172.21.15.195:0/3840096513 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5624003f80 msgr2=0x7f56240046d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:29.367 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.340+0000 7f563d35a700 1 --2- 172.21.15.195:0/3840096513 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5624003f80 0x7f56240046d0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f562c00ba40 tx=0x7f562c008040 comp rx=0 tx=0).stop 2022-06-27T11:58:29.368 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.340+0000 7f56377fe700 1 -- 172.21.15.195:0/3840096513 reap_dead start 2022-06-27T11:58:29.368 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.340+0000 7f563d35a700 1 -- 172.21.15.195:0/3840096513 shutdown_connections 2022-06-27T11:58:29.369 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.340+0000 7f563d35a700 1 -- 172.21.15.195:0/3840096513 >> 172.21.15.195:0/3840096513 conn(0x7f56380ccbd0 msgr2=0x7f56380ccfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:58:29.369 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.340+0000 7f563d35a700 1 -- 172.21.15.195:0/3840096513 shutdown_connections 2022-06-27T11:58:29.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.340+0000 7f563d35a700 1 -- 172.21.15.195:0/3840096513 wait complete. 2022-06-27T11:58:29.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.341+0000 7f563d35a700 1 Processor -- start 2022-06-27T11:58:29.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.341+0000 7f563d35a700 1 -- start start 2022-06-27T11:58:29.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.342+0000 7f563d35a700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5638157810 0x7f5638166730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:29.372 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.342+0000 7f563d35a700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f56381522b0 0x7f5638153100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:29.373 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.342+0000 7f563d35a700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5638169c80 0x7f5638153640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:29.373 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.342+0000 7f563d35a700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56381526b0 con 0x7f5638157810 2022-06-27T11:58:29.374 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.342+0000 7f563d35a700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5638152c70 con 0x7f5638169c80 2022-06-27T11:58:29.374 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.342+0000 7f563d35a700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5638166c70 con 0x7f56381522b0 2022-06-27T11:58:29.383 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.342+0000 7f5636ffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5638157810 0x7f5638166730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:29.384 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.342+0000 7f5636ffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5638157810 0x7f5638166730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45684/0 (socket says 172.21.15.195:45684) 2022-06-27T11:58:29.384 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.342+0000 7f5636ffd700 1 -- 172.21.15.195:0/1880558846 learned_addr learned my addr 172.21.15.195:0/1880558846 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:58:29.385 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.342+0000 7f56377fe700 1 --2- 172.21.15.195:0/1880558846 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5638169c80 0x7f5638153640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:29.385 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.342+0000 7f56367fc700 1 --2- 172.21.15.195:0/1880558846 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f56381522b0 0x7f5638153100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:29.386 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.342+0000 7f5636ffd700 1 -- 172.21.15.195:0/1880558846 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f56381522b0 msgr2=0x7f5638153100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:29.386 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.342+0000 7f5636ffd700 1 --2- 172.21.15.195:0/1880558846 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f56381522b0 0x7f5638153100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:29.387 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.342+0000 7f5636ffd700 1 -- 172.21.15.195:0/1880558846 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5638169c80 msgr2=0x7f5638153640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:29.387 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.342+0000 7f5636ffd700 1 --2- 172.21.15.195:0/1880558846 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5638169c80 0x7f5638153640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:29.387 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.342+0000 7f5636ffd700 1 -- 172.21.15.195:0/1880558846 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5638166ee0 con 0x7f5638157810 2022-06-27T11:58:29.388 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.342+0000 7f5636ffd700 1 --2- 172.21.15.195:0/1880558846 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5638157810 0x7f5638166730 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f5628004ae0 tx=0x7f5628001e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:29.388 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.342+0000 7f561effd700 1 -- 172.21.15.195:0/1880558846 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5628004ea0 con 0x7f5638157810 2022-06-27T11:58:29.389 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.342+0000 7f561effd700 1 -- 172.21.15.195:0/1880558846 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f562800f960 con 0x7f5638157810 2022-06-27T11:58:29.389 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.342+0000 7f563d35a700 1 -- 172.21.15.195:0/1880558846 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56381671c0 con 0x7f5638157810 2022-06-27T11:58:29.392 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.342+0000 7f561effd700 1 -- 172.21.15.195:0/1880558846 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f562800fd20 con 0x7f5638157810 2022-06-27T11:58:29.392 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.342+0000 7f563d35a700 1 -- 172.21.15.195:0/1880558846 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5638152900 con 0x7f5638157810 2022-06-27T11:58:29.394 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.343+0000 7f563d35a700 1 -- 172.21.15.195:0/1880558846 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5638152a90 con 0x7f5638157810 2022-06-27T11:58:29.394 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.344+0000 7f561effd700 1 -- 172.21.15.195:0/1880558846 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f562801c030 con 0x7f5638157810 2022-06-27T11:58:29.395 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.344+0000 7f561effd700 1 --2- 172.21.15.195:0/1880558846 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5618063d00 0x7f56180661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:29.395 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.344+0000 7f561effd700 1 -- 172.21.15.195:0/1880558846 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6632+0+0 (secure 0 0 0) 0x7f5628080690 con 0x7f5638157810 2022-06-27T11:58:29.395 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.344+0000 7f56367fc700 1 --2- 172.21.15.195:0/1880558846 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5618063d00 0x7f56180661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:29.395 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.344+0000 7f56367fc700 1 --2- 172.21.15.195:0/1880558846 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5618063d00 0x7f56180661b0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f56380665c0 tx=0x7f562c01a000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:29.396 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.348+0000 7f561effd700 1 -- 172.21.15.195:0/1880558846 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f562804f0a0 con 0x7f5638157810 2022-06-27T11:58:29.576 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.573+0000 7f563d35a700 1 -- 172.21.15.195:0/1880558846 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f56380722c0 con 0x7f5638157810 2022-06-27T11:58:29.577 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.574+0000 7f561effd700 1 -- 172.21.15.195:0/1880558846 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+5368 (secure 0 0 0) 0x7f56280526c0 con 0x7f5638157810 2022-06-27T11:58:29.577 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:58:29.578 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:58:26.057859+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5392},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5392":{"gid":5392,"name":"d","rank":0,"incarnation":38,"state":"up:rejoin","state_seq":15,"addr":"172.21.15.202:6835/2089155519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2089155519},{"type":"v1","addr":"172.21.15.202:6835","nonce":2089155519}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:58:29.581 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.576+0000 7f563d35a700 1 -- 172.21.15.195:0/1880558846 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5618063d00 msgr2=0x7f56180661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:29.581 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.576+0000 7f563d35a700 1 --2- 172.21.15.195:0/1880558846 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5618063d00 0x7f56180661b0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f56380665c0 tx=0x7f562c01a000 comp rx=0 tx=0).stop 2022-06-27T11:58:29.582 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.576+0000 7f563d35a700 1 -- 172.21.15.195:0/1880558846 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5638157810 msgr2=0x7f5638166730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:29.582 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.576+0000 7f563d35a700 1 --2- 172.21.15.195:0/1880558846 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5638157810 0x7f5638166730 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f5628004ae0 tx=0x7f5628001e30 comp rx=0 tx=0).stop 2022-06-27T11:58:29.583 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.576+0000 7f563d35a700 1 -- 172.21.15.195:0/1880558846 shutdown_connections 2022-06-27T11:58:29.583 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.576+0000 7f563d35a700 1 --2- 172.21.15.195:0/1880558846 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5618063d00 0x7f56180661b0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:29.584 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.576+0000 7f563d35a700 1 --2- 172.21.15.195:0/1880558846 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5638157810 0x7f5638166730 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:29.584 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.576+0000 7f563d35a700 1 --2- 172.21.15.195:0/1880558846 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5638169c80 0x7f5638153640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:29.585 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.576+0000 7f563d35a700 1 --2- 172.21.15.195:0/1880558846 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f56381522b0 0x7f5638153100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:29.585 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.576+0000 7f563d35a700 1 -- 172.21.15.195:0/1880558846 >> 172.21.15.195:0/1880558846 conn(0x7f56380ccbd0 msgr2=0x7f5638156300 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:58:29.586 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.576+0000 7f563d35a700 1 -- 172.21.15.195:0/1880558846 shutdown_connections 2022-06-27T11:58:29.586 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:29.576+0000 7f563d35a700 1 -- 172.21.15.195:0/1880558846 wait complete. 2022-06-27T11:58:29.587 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 40 2022-06-27T11:58:29.606 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T11:58:31.609 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:58:31.759 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.742+0000 7f1b57aa2700 1 Processor -- start 2022-06-27T11:58:31.759 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.742+0000 7f1b57aa2700 1 -- start start 2022-06-27T11:58:31.759 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.743+0000 7f1b57aa2700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f1b50156100 0x7f1b500688a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:31.760 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.743+0000 7f1b57aa2700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1b5006a440 0x7f1b500c4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:31.760 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.743+0000 7f1b57aa2700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b500c4b10 con 0x7f1b501564d0 2022-06-27T11:58:31.760 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.743+0000 7f1b57aa2700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b500c3dc0 con 0x7f1b50156100 2022-06-27T11:58:31.760 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.743+0000 7f1b57aa2700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b50068f20 con 0x7f1b5006a440 2022-06-27T11:58:31.761 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.743+0000 7f1b5503d700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1b5006a440 0x7f1b500c4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:31.761 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.743+0000 7f1b5583e700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f1b50156100 0x7f1b500688a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:31.761 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.743+0000 7f1b5583e700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f1b50156100 0x7f1b500688a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3300/0 says I am v2:172.21.15.195:43818/0 (socket says 172.21.15.195:43818) 2022-06-27T11:58:31.761 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.743+0000 7f1b5583e700 1 -- 172.21.15.195:0/1572975075 learned_addr learned my addr 172.21.15.195:0/1572975075 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:58:31.762 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.743+0000 7f1b5603f700 1 --1- 172.21.15.195:0/1572975075 >> v1:172.21.15.202:6789/0 conn(0x7f1b501564d0 0x7f1b500c28c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.202:6789/0 says I am v1:172.21.15.195:53740/0 (socket says 172.21.15.195:53740) 2022-06-27T11:58:31.762 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.743+0000 7f1b5583e700 1 -- 172.21.15.195:0/1572975075 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1b5006a440 msgr2=0x7f1b500c4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:58:31.762 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.743+0000 7f1b5583e700 1 --2- 172.21.15.195:0/1572975075 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1b5006a440 0x7f1b500c4fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:31.762 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.743+0000 7f1b5583e700 1 -- 172.21.15.195:0/1572975075 >> v1:172.21.15.202:6789/0 conn(0x7f1b501564d0 legacy=0x7f1b500c28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:31.763 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.743+0000 7f1b5483c700 1 -- 172.21.15.195:0/1572975075 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3623794537 0 0) 0x7f1b500c4b10 con 0x7f1b501564d0 2022-06-27T11:58:31.763 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.743+0000 7f1b5583e700 1 -- 172.21.15.195:0/1572975075 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b500c6360 con 0x7f1b50156100 2022-06-27T11:58:31.763 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.744+0000 7f1b5583e700 1 --2- 172.21.15.195:0/1572975075 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1b50156100 0x7f1b500688a0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c006fd0 tx=0x7f1b4c0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca1e8bf00e752d2 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:31.763 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.744+0000 7f1b5483c700 1 -- 172.21.15.195:0/1572975075 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b4c005ad0 con 0x7f1b50156100 2022-06-27T11:58:31.763 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.744+0000 7f1b5483c700 1 -- 172.21.15.195:0/1572975075 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b4c005c30 con 0x7f1b50156100 2022-06-27T11:58:31.764 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.744+0000 7f1b5483c700 1 -- 172.21.15.195:0/1572975075 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b4c004030 con 0x7f1b50156100 2022-06-27T11:58:31.764 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.744+0000 7f1b57aa2700 1 -- 172.21.15.195:0/1572975075 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1b50156100 msgr2=0x7f1b500688a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:31.764 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.744+0000 7f1b57aa2700 1 --2- 172.21.15.195:0/1572975075 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1b50156100 0x7f1b500688a0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c006fd0 tx=0x7f1b4c0090f0 comp rx=0 tx=0).stop 2022-06-27T11:58:31.764 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.744+0000 7f1b57aa2700 1 -- 172.21.15.195:0/1572975075 shutdown_connections 2022-06-27T11:58:31.765 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.744+0000 7f1b57aa2700 1 --2- 172.21.15.195:0/1572975075 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1b50156100 0x7f1b500688a0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:31.765 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.744+0000 7f1b57aa2700 1 --2- 172.21.15.195:0/1572975075 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1b5006a440 0x7f1b500c4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:31.765 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.744+0000 7f1b57aa2700 1 -- 172.21.15.195:0/1572975075 >> 172.21.15.195:0/1572975075 conn(0x7f1b500c9b00 msgr2=0x7f1b500c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:58:31.765 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.744+0000 7f1b57aa2700 1 -- 172.21.15.195:0/1572975075 shutdown_connections 2022-06-27T11:58:31.766 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.744+0000 7f1b57aa2700 1 -- 172.21.15.195:0/1572975075 wait complete. 2022-06-27T11:58:31.766 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.745+0000 7f1b57aa2700 1 Processor -- start 2022-06-27T11:58:31.766 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.745+0000 7f1b57aa2700 1 -- start start 2022-06-27T11:58:31.767 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.746+0000 7f1b57aa2700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1b5006a440 0x7f1b500bf890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:31.767 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.746+0000 7f1b57aa2700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1b50156100 0x7f1b500c2870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:31.767 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.746+0000 7f1b57aa2700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1b501564d0 0x7f1b500bd130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:31.768 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.746+0000 7f1b57aa2700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b50155dd0 con 0x7f1b5006a440 2022-06-27T11:58:31.768 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.746+0000 7f1b57aa2700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b500c8e70 con 0x7f1b501564d0 2022-06-27T11:58:31.768 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.746+0000 7f1b57aa2700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b50155940 con 0x7f1b50156100 2022-06-27T11:58:31.769 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.746+0000 7f1b5583e700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1b5006a440 0x7f1b500bf890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:31.769 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.746+0000 7f1b5583e700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1b5006a440 0x7f1b500bf890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45696/0 (socket says 172.21.15.195:45696) 2022-06-27T11:58:31.769 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.746+0000 7f1b5583e700 1 -- 172.21.15.195:0/3327920315 learned_addr learned my addr 172.21.15.195:0/3327920315 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:58:31.769 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.746+0000 7f1b5503d700 1 --2- 172.21.15.195:0/3327920315 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1b50156100 0x7f1b500c2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:31.770 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.746+0000 7f1b5603f700 1 --2- 172.21.15.195:0/3327920315 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1b501564d0 0x7f1b500bd130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:31.770 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.747+0000 7f1b5583e700 1 -- 172.21.15.195:0/3327920315 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1b50156100 msgr2=0x7f1b500c2870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:31.771 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.747+0000 7f1b5583e700 1 --2- 172.21.15.195:0/3327920315 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1b50156100 0x7f1b500c2870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:31.771 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.747+0000 7f1b5583e700 1 -- 172.21.15.195:0/3327920315 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1b501564d0 msgr2=0x7f1b500bd130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:31.772 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.747+0000 7f1b5583e700 1 --2- 172.21.15.195:0/3327920315 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1b501564d0 0x7f1b500bd130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:31.773 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.747+0000 7f1b5583e700 1 -- 172.21.15.195:0/3327920315 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b4c009350 con 0x7f1b5006a440 2022-06-27T11:58:31.773 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.747+0000 7f1b5603f700 1 --2- 172.21.15.195:0/3327920315 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1b501564d0 0x7f1b500bd130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T11:58:31.773 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.747+0000 7f1b5583e700 1 --2- 172.21.15.195:0/3327920315 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1b5006a440 0x7f1b500bf890 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c009ed0 tx=0x7f1b4c015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:31.774 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.747+0000 7f1b41ffb700 1 -- 172.21.15.195:0/3327920315 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b4c000c90 con 0x7f1b5006a440 2022-06-27T11:58:31.774 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.747+0000 7f1b41ffb700 1 -- 172.21.15.195:0/3327920315 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b4c005fb0 con 0x7f1b5006a440 2022-06-27T11:58:31.775 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.747+0000 7f1b57aa2700 1 -- 172.21.15.195:0/3327920315 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b50154d30 con 0x7f1b5006a440 2022-06-27T11:58:31.775 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.747+0000 7f1b41ffb700 1 -- 172.21.15.195:0/3327920315 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b4c006160 con 0x7f1b5006a440 2022-06-27T11:58:31.776 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.747+0000 7f1b57aa2700 1 -- 172.21.15.195:0/3327920315 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b501552f0 con 0x7f1b5006a440 2022-06-27T11:58:31.777 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.748+0000 7f1b57aa2700 1 -- 172.21.15.195:0/3327920315 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b50155c20 con 0x7f1b5006a440 2022-06-27T11:58:31.777 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.749+0000 7f1b41ffb700 1 -- 172.21.15.195:0/3327920315 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f1b4c02e070 con 0x7f1b5006a440 2022-06-27T11:58:31.778 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.749+0000 7f1b41ffb700 1 --2- 172.21.15.195:0/3327920315 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1b30063d00 0x7f1b300661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:31.778 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.749+0000 7f1b41ffb700 1 -- 172.21.15.195:0/3327920315 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6632+0+0 (secure 0 0 0) 0x7f1b4c082ae0 con 0x7f1b5006a440 2022-06-27T11:58:31.779 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.749+0000 7f1b5503d700 1 --2- 172.21.15.195:0/3327920315 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1b30063d00 0x7f1b300661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:31.779 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.749+0000 7f1b5503d700 1 --2- 172.21.15.195:0/3327920315 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1b30063d00 0x7f1b300661b0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f1b5015ec80 tx=0x7f1b44008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:31.780 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.753+0000 7f1b41ffb700 1 -- 172.21.15.195:0/3327920315 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f1b4c051490 con 0x7f1b5006a440 2022-06-27T11:58:31.980 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.980+0000 7f1b57aa2700 1 -- 172.21.15.195:0/3327920315 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1b500722c0 con 0x7f1b5006a440 2022-06-27T11:58:31.982 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.981+0000 7f1b41ffb700 1 -- 172.21.15.195:0/3327920315 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+5368 (secure 0 0 0) 0x7f1b4c02c070 con 0x7f1b5006a440 2022-06-27T11:58:31.983 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:58:31.984 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:58:26.057859+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5392},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5392":{"gid":5392,"name":"d","rank":0,"incarnation":38,"state":"up:rejoin","state_seq":15,"addr":"172.21.15.202:6835/2089155519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2089155519},{"type":"v1","addr":"172.21.15.202:6835","nonce":2089155519}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:58:31.986 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.983+0000 7f1b57aa2700 1 -- 172.21.15.195:0/3327920315 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1b30063d00 msgr2=0x7f1b300661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:31.987 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.983+0000 7f1b57aa2700 1 --2- 172.21.15.195:0/3327920315 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1b30063d00 0x7f1b300661b0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f1b5015ec80 tx=0x7f1b44008040 comp rx=0 tx=0).stop 2022-06-27T11:58:31.987 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.983+0000 7f1b57aa2700 1 -- 172.21.15.195:0/3327920315 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1b5006a440 msgr2=0x7f1b500bf890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:31.987 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.983+0000 7f1b57aa2700 1 --2- 172.21.15.195:0/3327920315 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1b5006a440 0x7f1b500bf890 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c009ed0 tx=0x7f1b4c015000 comp rx=0 tx=0).stop 2022-06-27T11:58:31.987 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.983+0000 7f1b57aa2700 1 -- 172.21.15.195:0/3327920315 shutdown_connections 2022-06-27T11:58:31.987 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.983+0000 7f1b57aa2700 1 --2- 172.21.15.195:0/3327920315 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1b30063d00 0x7f1b300661b0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:31.988 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.983+0000 7f1b57aa2700 1 --2- 172.21.15.195:0/3327920315 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1b5006a440 0x7f1b500bf890 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:31.988 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.983+0000 7f1b57aa2700 1 --2- 172.21.15.195:0/3327920315 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1b501564d0 0x7f1b500bd130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:31.988 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.983+0000 7f1b57aa2700 1 --2- 172.21.15.195:0/3327920315 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1b50156100 0x7f1b500c2870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:31.989 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.983+0000 7f1b57aa2700 1 -- 172.21.15.195:0/3327920315 >> 172.21.15.195:0/3327920315 conn(0x7f1b500c9b00 msgr2=0x7f1b5006ebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:58:31.989 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.983+0000 7f1b57aa2700 1 -- 172.21.15.195:0/3327920315 shutdown_connections 2022-06-27T11:58:31.989 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:31.983+0000 7f1b57aa2700 1 -- 172.21.15.195:0/3327920315 wait complete. 2022-06-27T11:58:31.989 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 40 2022-06-27T11:58:31.998 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T11:58:34.002 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:58:34.150 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.142+0000 7f58e24d7700 1 Processor -- start 2022-06-27T11:58:34.150 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.142+0000 7f58e24d7700 1 -- start start 2022-06-27T11:58:34.150 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.143+0000 7f58e24d7700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f58dc14e860 0x7f58dc0c3840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:34.151 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.143+0000 7f58e24d7700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f58dc0c3d80 0x7f58dc157890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:34.151 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.143+0000 7f58e24d7700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58dc0c33b0 con 0x7f58dc14e490 2022-06-27T11:58:34.151 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.143+0000 7f58e24d7700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58dc06f220 con 0x7f58dc14e860 2022-06-27T11:58:34.151 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.143+0000 7f58e24d7700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58dc1542c0 con 0x7f58dc0c3d80 2022-06-27T11:58:34.151 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.143+0000 7f58e0a74700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f58dc0c3d80 0x7f58dc157890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:34.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.143+0000 7f58db7fe700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f58dc14e860 0x7f58dc0c3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:34.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.143+0000 7f58dbfff700 1 --1- >> v1:172.21.15.202:6789/0 conn(0x7f58dc14e490 0x7f58dc06cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.202:6789/0 says I am v1:172.21.15.195:53752/0 (socket says 172.21.15.195:53752) 2022-06-27T11:58:34.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.143+0000 7f58dbfff700 1 -- 172.21.15.195:0/3152762112 learned_addr learned my addr 172.21.15.195:0/3152762112 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:58:34.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.143+0000 7f58e0a74700 1 -- 172.21.15.195:0/3152762112 >> v2:172.21.15.202:3300/0 conn(0x7f58dc14e860 msgr2=0x7f58dc0c3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:58:34.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.143+0000 7f58e0a74700 1 --2- 172.21.15.195:0/3152762112 >> v2:172.21.15.202:3300/0 conn(0x7f58dc14e860 0x7f58dc0c3840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:34.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.143+0000 7f58e0a74700 1 -- 172.21.15.195:0/3152762112 >> v1:172.21.15.202:6789/0 conn(0x7f58dc14e490 legacy=0x7f58dc06cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:34.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.143+0000 7f58e0a74700 1 -- 172.21.15.195:0/3152762112 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58dc154560 con 0x7f58dc0c3d80 2022-06-27T11:58:34.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.143+0000 7f58db7fe700 1 --2- 172.21.15.195:0/3152762112 >> v2:172.21.15.202:3300/0 conn(0x7f58dc14e860 0x7f58dc0c3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:58:34.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.144+0000 7f58e0a74700 1 --2- 172.21.15.195:0/3152762112 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f58dc0c3d80 0x7f58dc157890 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f58d4009e60 tx=0x7f58d4009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74ad5c9670831e1 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:34.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.144+0000 7f58daffd700 1 -- 172.21.15.195:0/3152762112 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d400e070 con 0x7f58dc0c3d80 2022-06-27T11:58:34.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.144+0000 7f58daffd700 1 -- 172.21.15.195:0/3152762112 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58d400bb60 con 0x7f58dc0c3d80 2022-06-27T11:58:34.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.144+0000 7f58daffd700 1 -- 172.21.15.195:0/3152762112 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d4012430 con 0x7f58dc0c3d80 2022-06-27T11:58:34.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.144+0000 7f58e24d7700 1 -- 172.21.15.195:0/3152762112 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f58dc0c3d80 msgr2=0x7f58dc157890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:34.155 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.144+0000 7f58e24d7700 1 --2- 172.21.15.195:0/3152762112 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f58dc0c3d80 0x7f58dc157890 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f58d4009e60 tx=0x7f58d4009780 comp rx=0 tx=0).stop 2022-06-27T11:58:34.155 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.144+0000 7f58e24d7700 1 -- 172.21.15.195:0/3152762112 shutdown_connections 2022-06-27T11:58:34.155 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.144+0000 7f58e24d7700 1 --2- 172.21.15.195:0/3152762112 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f58dc0c3d80 0x7f58dc157890 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:34.155 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.144+0000 7f58e24d7700 1 --2- 172.21.15.195:0/3152762112 >> v2:172.21.15.202:3300/0 conn(0x7f58dc14e860 0x7f58dc0c3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:34.156 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.144+0000 7f58e24d7700 1 -- 172.21.15.195:0/3152762112 >> 172.21.15.195:0/3152762112 conn(0x7f58dc063af0 msgr2=0x7f58dc063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:58:34.164 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.144+0000 7f58e24d7700 1 -- 172.21.15.195:0/3152762112 shutdown_connections 2022-06-27T11:58:34.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.144+0000 7f58e24d7700 1 -- 172.21.15.195:0/3152762112 wait complete. 2022-06-27T11:58:34.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.145+0000 7f58e24d7700 1 Processor -- start 2022-06-27T11:58:34.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.146+0000 7f58e24d7700 1 -- start start 2022-06-27T11:58:34.166 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.146+0000 7f58e24d7700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f58dc0c3d80 0x7f58dc0c0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:34.166 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.146+0000 7f58e24d7700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f58dc14e490 0x7f58dc1529a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:34.166 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.146+0000 7f58e24d7700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f58dc14e860 0x7f58dc152ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:34.167 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.146+0000 7f58e24d7700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58dc155490 con 0x7f58dc0c3d80 2022-06-27T11:58:34.167 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.146+0000 7f58e24d7700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58dc155c60 con 0x7f58dc14e860 2022-06-27T11:58:34.168 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.146+0000 7f58e24d7700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58dc0c1120 con 0x7f58dc14e490 2022-06-27T11:58:34.168 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.147+0000 7f58dbfff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f58dc0c3d80 0x7f58dc0c0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:34.168 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.147+0000 7f58dbfff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f58dc0c3d80 0x7f58dc0c0be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45712/0 (socket says 172.21.15.195:45712) 2022-06-27T11:58:34.169 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.147+0000 7f58dbfff700 1 -- 172.21.15.195:0/2952338120 learned_addr learned my addr 172.21.15.195:0/2952338120 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:58:34.174 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.147+0000 7f58db7fe700 1 --2- 172.21.15.195:0/2952338120 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f58dc14e490 0x7f58dc1529a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:34.174 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.147+0000 7f58e0a74700 1 --2- 172.21.15.195:0/2952338120 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f58dc14e860 0x7f58dc152ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:34.175 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.147+0000 7f58dbfff700 1 -- 172.21.15.195:0/2952338120 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f58dc14e490 msgr2=0x7f58dc1529a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:34.175 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.147+0000 7f58dbfff700 1 --2- 172.21.15.195:0/2952338120 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f58dc14e490 0x7f58dc1529a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:34.175 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.147+0000 7f58dbfff700 1 -- 172.21.15.195:0/2952338120 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f58dc14e860 msgr2=0x7f58dc152ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:34.176 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.147+0000 7f58dbfff700 1 --2- 172.21.15.195:0/2952338120 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f58dc14e860 0x7f58dc152ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:34.176 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.147+0000 7f58dbfff700 1 -- 172.21.15.195:0/2952338120 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58cc0092e0 con 0x7f58dc0c3d80 2022-06-27T11:58:34.182 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.147+0000 7f58dbfff700 1 --2- 172.21.15.195:0/2952338120 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f58dc0c3d80 0x7f58dc0c0be0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f58d000c490 tx=0x7f58d0007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:34.183 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.147+0000 7f58c3fff700 1 -- 172.21.15.195:0/2952338120 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d000ce00 con 0x7f58dc0c3d80 2022-06-27T11:58:34.183 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.147+0000 7f58c3fff700 1 -- 172.21.15.195:0/2952338120 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58d00072e0 con 0x7f58dc0c3d80 2022-06-27T11:58:34.183 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.147+0000 7f58e24d7700 1 -- 172.21.15.195:0/2952338120 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58d4009a20 con 0x7f58dc0c3d80 2022-06-27T11:58:34.188 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.147+0000 7f58c3fff700 1 -- 172.21.15.195:0/2952338120 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d0007660 con 0x7f58dc0c3d80 2022-06-27T11:58:34.197 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.147+0000 7f58e24d7700 1 -- 172.21.15.195:0/2952338120 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58dc1556a0 con 0x7f58dc0c3d80 2022-06-27T11:58:34.198 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.148+0000 7f58e24d7700 1 -- 172.21.15.195:0/2952338120 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58dc0722c0 con 0x7f58dc0c3d80 2022-06-27T11:58:34.199 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.149+0000 7f58c3fff700 1 -- 172.21.15.195:0/2952338120 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f58d0019030 con 0x7f58dc0c3d80 2022-06-27T11:58:34.199 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.149+0000 7f58c3fff700 1 --2- 172.21.15.195:0/2952338120 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f58bc063d30 0x7f58bc0661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:34.200 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.149+0000 7f58c3fff700 1 -- 172.21.15.195:0/2952338120 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6632+0+0 (secure 0 0 0) 0x7f58d007fb80 con 0x7f58dc0c3d80 2022-06-27T11:58:34.200 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.149+0000 7f58db7fe700 1 --2- 172.21.15.195:0/2952338120 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f58bc063d30 0x7f58bc0661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:34.201 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.150+0000 7f58db7fe700 1 --2- 172.21.15.195:0/2952338120 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f58bc063d30 0x7f58bc0661e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f58dc0be4f0 tx=0x7f58cc009060 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:34.201 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.153+0000 7f58c3fff700 1 -- 172.21.15.195:0/2952338120 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f58d004e5e0 con 0x7f58dc0c3d80 2022-06-27T11:58:34.384 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.377+0000 7f58e24d7700 1 -- 172.21.15.195:0/2952338120 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f58dc0c60d0 con 0x7f58dc0c3d80 2022-06-27T11:58:34.384 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.377+0000 7f58c3fff700 1 -- 172.21.15.195:0/2952338120 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+5368 (secure 0 0 0) 0x7f58d0017070 con 0x7f58dc0c3d80 2022-06-27T11:58:34.385 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:58:34.386 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:58:32.073997+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5392},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5392":{"gid":5392,"name":"d","rank":0,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.202:6835/2089155519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2089155519},{"type":"v1","addr":"172.21.15.202:6835","nonce":2089155519}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:58:34.388 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.379+0000 7f58e24d7700 1 -- 172.21.15.195:0/2952338120 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f58bc063d30 msgr2=0x7f58bc0661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:34.388 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.379+0000 7f58e24d7700 1 --2- 172.21.15.195:0/2952338120 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f58bc063d30 0x7f58bc0661e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f58dc0be4f0 tx=0x7f58cc009060 comp rx=0 tx=0).stop 2022-06-27T11:58:34.389 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.379+0000 7f58e24d7700 1 -- 172.21.15.195:0/2952338120 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f58dc0c3d80 msgr2=0x7f58dc0c0be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:34.389 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.379+0000 7f58e24d7700 1 --2- 172.21.15.195:0/2952338120 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f58dc0c3d80 0x7f58dc0c0be0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f58d000c490 tx=0x7f58d0007df0 comp rx=0 tx=0).stop 2022-06-27T11:58:34.390 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.379+0000 7f58e24d7700 1 -- 172.21.15.195:0/2952338120 shutdown_connections 2022-06-27T11:58:34.390 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.379+0000 7f58e24d7700 1 --2- 172.21.15.195:0/2952338120 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f58bc063d30 0x7f58bc0661e0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:34.391 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.379+0000 7f58e24d7700 1 --2- 172.21.15.195:0/2952338120 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f58dc0c3d80 0x7f58dc0c0be0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:34.391 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.379+0000 7f58e24d7700 1 --2- 172.21.15.195:0/2952338120 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f58dc14e860 0x7f58dc152ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:34.391 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.379+0000 7f58e24d7700 1 --2- 172.21.15.195:0/2952338120 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f58dc14e490 0x7f58dc1529a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:34.392 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.379+0000 7f58e24d7700 1 -- 172.21.15.195:0/2952338120 >> 172.21.15.195:0/2952338120 conn(0x7f58dc063af0 msgr2=0x7f58dc151df0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:58:34.392 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.379+0000 7f58e24d7700 1 -- 172.21.15.195:0/2952338120 shutdown_connections 2022-06-27T11:58:34.394 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:34.379+0000 7f58e24d7700 1 -- 172.21.15.195:0/2952338120 wait complete. 2022-06-27T11:58:34.396 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 41 2022-06-27T11:58:34.397 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d has gained rank=0, replacing gid=4976 2022-06-27T11:58:34.398 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 4 secs before reviving mds.b 2022-06-27T11:58:38.398 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2022-06-27T11:58:38.399 INFO:tasks.ceph.mds.b:Restarting daemon 2022-06-27T11:58:38.399 DEBUG:teuthology.orchestra.run.smithi202:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-06-27T11:58:38.402 INFO:tasks.ceph.mds.b:Started 2022-06-27T11:58:38.403 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:58:38.478 INFO:tasks.ceph.mds.b.smithi202.stdout:starting mds.b at 2022-06-27T11:58:38.561 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.536+0000 7f04a71d0700 1 Processor -- start 2022-06-27T11:58:38.561 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.536+0000 7f04a71d0700 1 -- start start 2022-06-27T11:58:38.561 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.537+0000 7f04a71d0700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f04a006f640 0x7f04a0156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:38.562 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.537+0000 7f04a71d0700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f04a00c4160 0x7f04a00c1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:38.562 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.537+0000 7f04a71d0700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04a00c1a20 con 0x7f04a00c39e0 2022-06-27T11:58:38.563 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.537+0000 7f04a71d0700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04a00c1130 con 0x7f04a006f640 2022-06-27T11:58:38.563 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.537+0000 7f04a71d0700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04a006ea90 con 0x7f04a00c4160 2022-06-27T11:58:38.563 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.537+0000 7f04a576d700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f04a00c39e0 0x7f04a01535a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:51910/0 (socket says 172.21.15.195:51910) 2022-06-27T11:58:38.564 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.537+0000 7f04a576d700 1 -- 172.21.15.195:0/770880865 learned_addr learned my addr 172.21.15.195:0/770880865 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:58:38.564 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.537+0000 7f04a4f6c700 1 --2- 172.21.15.195:0/770880865 >> v2:172.21.15.202:3300/0 conn(0x7f04a006f640 0x7f04a0156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:38.564 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.537+0000 7f049ffff700 1 --2- 172.21.15.195:0/770880865 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f04a00c4160 0x7f04a00c1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:38.564 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.537+0000 7f049f7fe700 1 -- 172.21.15.195:0/770880865 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 596131740 0 0) 0x7f04a00c1a20 con 0x7f04a00c39e0 2022-06-27T11:58:38.565 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.537+0000 7f049f7fe700 1 -- 172.21.15.195:0/770880865 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0484003430 con 0x7f04a00c39e0 2022-06-27T11:58:38.565 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.538+0000 7f049ffff700 1 -- 172.21.15.195:0/770880865 >> v2:172.21.15.202:3300/0 conn(0x7f04a006f640 msgr2=0x7f04a0156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:58:38.565 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.538+0000 7f049ffff700 1 --2- 172.21.15.195:0/770880865 >> v2:172.21.15.202:3300/0 conn(0x7f04a006f640 0x7f04a0156ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:38.565 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.538+0000 7f049ffff700 1 -- 172.21.15.195:0/770880865 >> v1:172.21.15.195:6789/0 conn(0x7f04a00c39e0 legacy=0x7f04a01535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:38.566 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.538+0000 7f049ffff700 1 -- 172.21.15.195:0/770880865 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04a006ed70 con 0x7f04a00c4160 2022-06-27T11:58:38.566 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.538+0000 7f04a4f6c700 1 --2- 172.21.15.195:0/770880865 >> v2:172.21.15.202:3300/0 conn(0x7f04a006f640 0x7f04a0156ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:58:38.566 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.538+0000 7f049ffff700 1 --2- 172.21.15.195:0/770880865 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f04a00c4160 0x7f04a00c1eb0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f048c0097a0 tx=0x7f048c009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46eb10c1c9872bc6 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:38.567 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.538+0000 7f049f7fe700 1 -- 172.21.15.195:0/770880865 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f048c011630 con 0x7f04a00c4160 2022-06-27T11:58:38.567 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.538+0000 7f049f7fe700 1 -- 172.21.15.195:0/770880865 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f048c011790 con 0x7f04a00c4160 2022-06-27T11:58:38.567 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.538+0000 7f049f7fe700 1 -- 172.21.15.195:0/770880865 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f048c011b30 con 0x7f04a00c4160 2022-06-27T11:58:38.568 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.538+0000 7f04a71d0700 1 -- 172.21.15.195:0/770880865 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f04a00c4160 msgr2=0x7f04a00c1eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:38.568 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.538+0000 7f04a71d0700 1 --2- 172.21.15.195:0/770880865 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f04a00c4160 0x7f04a00c1eb0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f048c0097a0 tx=0x7f048c009060 comp rx=0 tx=0).stop 2022-06-27T11:58:38.568 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.538+0000 7f04a71d0700 1 -- 172.21.15.195:0/770880865 shutdown_connections 2022-06-27T11:58:38.569 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.538+0000 7f04a71d0700 1 --2- 172.21.15.195:0/770880865 >> v2:172.21.15.202:3300/0 conn(0x7f04a006f640 0x7f04a0156ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:38.569 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.538+0000 7f04a71d0700 1 --2- 172.21.15.195:0/770880865 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f04a00c4160 0x7f04a00c1eb0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:38.570 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.538+0000 7f04a71d0700 1 -- 172.21.15.195:0/770880865 >> 172.21.15.195:0/770880865 conn(0x7f04a00c6350 msgr2=0x7f04a00c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:58:38.570 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.538+0000 7f04a71d0700 1 -- 172.21.15.195:0/770880865 shutdown_connections 2022-06-27T11:58:38.570 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.538+0000 7f04a71d0700 1 -- 172.21.15.195:0/770880865 wait complete. 2022-06-27T11:58:38.571 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.540+0000 7f04a71d0700 1 Processor -- start 2022-06-27T11:58:38.571 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.540+0000 7f04a71d0700 1 -- start start 2022-06-27T11:58:38.572 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.540+0000 7f04a71d0700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f04a006f640 0x7f04a00c4710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:38.573 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.540+0000 7f04a71d0700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f04a00c39e0 0x7f04a014e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:38.573 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.540+0000 7f04a71d0700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f04a00c4160 0x7f04a014e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:38.574 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.540+0000 7f04a71d0700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04a016aa70 con 0x7f04a00c39e0 2022-06-27T11:58:38.574 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.540+0000 7f04a71d0700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04a00c5230 con 0x7f04a00c4160 2022-06-27T11:58:38.575 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.540+0000 7f04a71d0700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04a0169df0 con 0x7f04a006f640 2022-06-27T11:58:38.575 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.541+0000 7f049ffff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f04a00c39e0 0x7f04a014e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:38.576 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.541+0000 7f049ffff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f04a00c39e0 0x7f04a014e2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45726/0 (socket says 172.21.15.195:45726) 2022-06-27T11:58:38.576 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.541+0000 7f049ffff700 1 -- 172.21.15.195:0/3852234342 learned_addr learned my addr 172.21.15.195:0/3852234342 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:58:38.577 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.541+0000 7f04a576d700 1 --2- 172.21.15.195:0/3852234342 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f04a00c4160 0x7f04a014e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:38.577 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.541+0000 7f04a4f6c700 1 --2- 172.21.15.195:0/3852234342 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f04a006f640 0x7f04a00c4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:38.578 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.541+0000 7f049ffff700 1 -- 172.21.15.195:0/3852234342 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f04a006f640 msgr2=0x7f04a00c4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:38.579 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.541+0000 7f049ffff700 1 --2- 172.21.15.195:0/3852234342 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f04a006f640 0x7f04a00c4710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:38.579 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.541+0000 7f049ffff700 1 -- 172.21.15.195:0/3852234342 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f04a00c4160 msgr2=0x7f04a014e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:38.580 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.541+0000 7f049ffff700 1 --2- 172.21.15.195:0/3852234342 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f04a00c4160 0x7f04a014e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:38.580 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.541+0000 7f049ffff700 1 -- 172.21.15.195:0/3852234342 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04940092e0 con 0x7f04a00c39e0 2022-06-27T11:58:38.581 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.541+0000 7f049ffff700 1 --2- 172.21.15.195:0/3852234342 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f04a00c39e0 0x7f04a014e2e0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f048c009d90 tx=0x7f048c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:38.581 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.541+0000 7f049cff9700 1 -- 172.21.15.195:0/3852234342 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f048c003730 con 0x7f04a00c39e0 2022-06-27T11:58:38.582 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.541+0000 7f049cff9700 1 -- 172.21.15.195:0/3852234342 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f048c003890 con 0x7f04a00c39e0 2022-06-27T11:58:38.582 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.541+0000 7f04a71d0700 1 -- 172.21.15.195:0/3852234342 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f048c0092e0 con 0x7f04a00c39e0 2022-06-27T11:58:38.582 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.541+0000 7f049cff9700 1 -- 172.21.15.195:0/3852234342 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f048c003c50 con 0x7f04a00c39e0 2022-06-27T11:58:38.583 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.541+0000 7f04a71d0700 1 -- 172.21.15.195:0/3852234342 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04a016a3b0 con 0x7f04a00c39e0 2022-06-27T11:58:38.583 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.542+0000 7f04a71d0700 1 -- 172.21.15.195:0/3852234342 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04a00722c0 con 0x7f04a00c39e0 2022-06-27T11:58:38.584 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.543+0000 7f049cff9700 1 -- 172.21.15.195:0/3852234342 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f048c00f050 con 0x7f04a00c39e0 2022-06-27T11:58:38.584 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.543+0000 7f049cff9700 1 --2- 172.21.15.195:0/3852234342 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0488063d00 0x7f04880661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:38.585 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.543+0000 7f04a4f6c700 1 --2- 172.21.15.195:0/3852234342 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0488063d00 0x7f04880661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:38.585 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.543+0000 7f049cff9700 1 -- 172.21.15.195:0/3852234342 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6632+0+0 (secure 0 0 0) 0x7f048c081e90 con 0x7f04a00c39e0 2022-06-27T11:58:38.586 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.544+0000 7f04a4f6c700 1 --2- 172.21.15.195:0/3852234342 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0488063d00 0x7f04880661b0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f0494009e00 tx=0x7f0494009060 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:38.586 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.548+0000 7f049cff9700 1 -- 172.21.15.195:0/3852234342 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f048c050840 con 0x7f04a00c39e0 2022-06-27T11:58:38.780 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.776+0000 7f04a71d0700 1 -- 172.21.15.195:0/3852234342 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f04a006f240 con 0x7f04a00c39e0 2022-06-27T11:58:38.780 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.777+0000 7f049cff9700 1 -- 172.21.15.195:0/3852234342 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+5368 (secure 0 0 0) 0x7f048c015030 con 0x7f04a00c39e0 2022-06-27T11:58:38.781 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:58:38.781 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:58:32.073997+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5392},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5392":{"gid":5392,"name":"d","rank":0,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.202:6835/2089155519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2089155519},{"type":"v1","addr":"172.21.15.202:6835","nonce":2089155519}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:58:38.795 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.779+0000 7f04a71d0700 1 -- 172.21.15.195:0/3852234342 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0488063d00 msgr2=0x7f04880661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:38.795 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.779+0000 7f04a71d0700 1 --2- 172.21.15.195:0/3852234342 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0488063d00 0x7f04880661b0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f0494009e00 tx=0x7f0494009060 comp rx=0 tx=0).stop 2022-06-27T11:58:38.796 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.779+0000 7f04a71d0700 1 -- 172.21.15.195:0/3852234342 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f04a00c39e0 msgr2=0x7f04a014e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:38.796 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.779+0000 7f04a71d0700 1 --2- 172.21.15.195:0/3852234342 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f04a00c39e0 0x7f04a014e2e0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f048c009d90 tx=0x7f048c017000 comp rx=0 tx=0).stop 2022-06-27T11:58:38.797 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.779+0000 7f04a71d0700 1 -- 172.21.15.195:0/3852234342 shutdown_connections 2022-06-27T11:58:38.797 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.779+0000 7f04a71d0700 1 --2- 172.21.15.195:0/3852234342 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0488063d00 0x7f04880661b0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:38.799 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.779+0000 7f04a71d0700 1 --2- 172.21.15.195:0/3852234342 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f04a00c4160 0x7f04a014e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:38.799 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.779+0000 7f04a71d0700 1 --2- 172.21.15.195:0/3852234342 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f04a006f640 0x7f04a00c4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:38.800 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.779+0000 7f04a71d0700 1 --2- 172.21.15.195:0/3852234342 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f04a00c39e0 0x7f04a014e2e0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:38.800 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.779+0000 7f04a71d0700 1 -- 172.21.15.195:0/3852234342 >> 172.21.15.195:0/3852234342 conn(0x7f04a00c6350 msgr2=0x7f04a01506e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:58:38.800 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.779+0000 7f04a71d0700 1 -- 172.21.15.195:0/3852234342 shutdown_connections 2022-06-27T11:58:38.800 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:38.779+0000 7f04a71d0700 1 -- 172.21.15.195:0/3852234342 wait complete. 2022-06-27T11:58:38.801 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 41 2022-06-27T11:58:38.801 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2022-06-27T11:58:40.803 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:58:40.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:40.935+0000 7f88e3995700 1 Processor -- start 2022-06-27T11:58:40.938 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:40.936+0000 7f88e3995700 1 -- start start 2022-06-27T11:58:40.949 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:40.936+0000 7f88e3995700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f88dc0c31d0 0x7f88dc0c61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:40.949 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:40.936+0000 7f88e3995700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f88dc067ed0 0x7f88dc0c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:40.949 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:40.936+0000 7f88e3995700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88dc14d8a0 con 0x7f88dc06eff0 2022-06-27T11:58:40.950 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:40.936+0000 7f88e3995700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88dc0c47e0 con 0x7f88dc067ed0 2022-06-27T11:58:40.950 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:40.936+0000 7f88e3995700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88dc0c3e10 con 0x7f88dc0c31d0 2022-06-27T11:58:40.950 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:40.937+0000 7f88e1f32700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f88dc067ed0 0x7f88dc0c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:40.950 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:40.937+0000 7f88e1731700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f88dc06eff0 0x7f88dc0c2ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:51932/0 (socket says 172.21.15.195:51932) 2022-06-27T11:58:40.951 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:40.937+0000 7f88e1731700 1 -- 172.21.15.195:0/4019018293 learned_addr learned my addr 172.21.15.195:0/4019018293 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:58:40.951 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:40.937+0000 7f88e0f30700 1 --2- 172.21.15.195:0/4019018293 >> v2:172.21.15.202:3300/0 conn(0x7f88dc0c31d0 0x7f88dc0c61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:40.951 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:40.937+0000 7f88e1f32700 1 -- 172.21.15.195:0/4019018293 >> v2:172.21.15.202:3300/0 conn(0x7f88dc0c31d0 msgr2=0x7f88dc0c61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:58:40.951 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:40.937+0000 7f88e1f32700 1 --2- 172.21.15.195:0/4019018293 >> v2:172.21.15.202:3300/0 conn(0x7f88dc0c31d0 0x7f88dc0c61e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:40.952 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:40.937+0000 7f88e1f32700 1 -- 172.21.15.195:0/4019018293 >> v1:172.21.15.195:6789/0 conn(0x7f88dc06eff0 legacy=0x7f88dc0c2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:40.952 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:40.937+0000 7f88e1f32700 1 -- 172.21.15.195:0/4019018293 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88dc06d970 con 0x7f88dc067ed0 2022-06-27T11:58:40.953 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:40.937+0000 7f88cbfff700 1 -- 172.21.15.195:0/4019018293 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 598183437 0 0) 0x7f88dc14d8a0 con 0x7f88dc06eff0 2022-06-27T11:58:40.953 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:40.937+0000 7f88e1f32700 1 --2- 172.21.15.195:0/4019018293 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f88dc067ed0 0x7f88dc0c42a0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f88dc06d430 tx=0x7f88d4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccf33d5a36b82cbb server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:40.954 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:40.937+0000 7f88cbfff700 1 -- 172.21.15.195:0/4019018293 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88d400e070 con 0x7f88dc067ed0 2022-06-27T11:58:40.954 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:40.937+0000 7f88cbfff700 1 -- 172.21.15.195:0/4019018293 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88d400bbf0 con 0x7f88dc067ed0 2022-06-27T11:58:40.954 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:40.937+0000 7f88cbfff700 1 -- 172.21.15.195:0/4019018293 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88d4012430 con 0x7f88dc067ed0 2022-06-27T11:58:40.955 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:40.938+0000 7f88e3995700 1 -- 172.21.15.195:0/4019018293 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f88dc067ed0 msgr2=0x7f88dc0c42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:40.955 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:40.938+0000 7f88e3995700 1 --2- 172.21.15.195:0/4019018293 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f88dc067ed0 0x7f88dc0c42a0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f88dc06d430 tx=0x7f88d4009780 comp rx=0 tx=0).stop 2022-06-27T11:58:40.956 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:40.938+0000 7f88e3995700 1 -- 172.21.15.195:0/4019018293 shutdown_connections 2022-06-27T11:58:40.956 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:40.938+0000 7f88e3995700 1 --2- 172.21.15.195:0/4019018293 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f88dc067ed0 0x7f88dc0c42a0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:40.957 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:40.938+0000 7f88e3995700 1 --2- 172.21.15.195:0/4019018293 >> v2:172.21.15.202:3300/0 conn(0x7f88dc0c31d0 0x7f88dc0c61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:40.957 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:40.938+0000 7f88e3995700 1 -- 172.21.15.195:0/4019018293 >> 172.21.15.195:0/4019018293 conn(0x7f88dc0c9b00 msgr2=0x7f88dc0c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:58:40.958 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:40.938+0000 7f88e3995700 1 -- 172.21.15.195:0/4019018293 shutdown_connections 2022-06-27T11:58:40.958 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:40.938+0000 7f88e3995700 1 -- 172.21.15.195:0/4019018293 wait complete. 2022-06-27T11:58:40.959 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:40.939+0000 7f88e3995700 1 Processor -- start 2022-06-27T11:58:40.959 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:40.939+0000 7f88e3995700 1 -- start start 2022-06-27T11:58:40.959 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:40.940+0000 7f88e3995700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f88dc067ed0 0x7f88dc1542a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:40.959 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:40.940+0000 7f88e3995700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f88dc06eff0 0x7f88dc146bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:40.960 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:40.940+0000 7f88e3995700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f88dc14d140 0x7f88dc1696f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:40.960 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:40.940+0000 7f88e3995700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88dc156e30 con 0x7f88dc06eff0 2022-06-27T11:58:40.960 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:40.940+0000 7f88e3995700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88dc1547e0 con 0x7f88dc067ed0 2022-06-27T11:58:40.960 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:40.940+0000 7f88e3995700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88dc0be640 con 0x7f88dc14d140 2022-06-27T11:58:40.960 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:40.940+0000 7f88e0f30700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f88dc06eff0 0x7f88dc146bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:40.961 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:40.940+0000 7f88e0f30700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f88dc06eff0 0x7f88dc146bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45752/0 (socket says 172.21.15.195:45752) 2022-06-27T11:58:40.961 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:40.940+0000 7f88e0f30700 1 -- 172.21.15.195:0/4219817590 learned_addr learned my addr 172.21.15.195:0/4219817590 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:58:40.961 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:40.940+0000 7f88e1731700 1 --2- 172.21.15.195:0/4219817590 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f88dc067ed0 0x7f88dc1542a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:40.961 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:40.940+0000 7f88e1f32700 1 --2- 172.21.15.195:0/4219817590 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f88dc14d140 0x7f88dc1696f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:40.962 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:40.940+0000 7f88e0f30700 1 -- 172.21.15.195:0/4219817590 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f88dc14d140 msgr2=0x7f88dc1696f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:40.962 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:40.940+0000 7f88e0f30700 1 --2- 172.21.15.195:0/4219817590 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f88dc14d140 0x7f88dc1696f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:40.962 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:40.940+0000 7f88e0f30700 1 -- 172.21.15.195:0/4219817590 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f88dc067ed0 msgr2=0x7f88dc1542a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:40.962 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:40.940+0000 7f88e0f30700 1 --2- 172.21.15.195:0/4219817590 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f88dc067ed0 0x7f88dc1542a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:40.963 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:40.940+0000 7f88e0f30700 1 -- 172.21.15.195:0/4219817590 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88d4009a20 con 0x7f88dc06eff0 2022-06-27T11:58:40.963 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:40.940+0000 7f88e1f32700 1 --2- 172.21.15.195:0/4219817590 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f88dc14d140 0x7f88dc1696f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T11:58:40.963 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:40.940+0000 7f88e0f30700 1 --2- 172.21.15.195:0/4219817590 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f88dc06eff0 0x7f88dc146bd0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f88d800e930 tx=0x7f88d8009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:40.963 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:40.941+0000 7f88c97fa700 1 -- 172.21.15.195:0/4219817590 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88d8013680 con 0x7f88dc06eff0 2022-06-27T11:58:40.964 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:40.941+0000 7f88c97fa700 1 -- 172.21.15.195:0/4219817590 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88d80137e0 con 0x7f88dc06eff0 2022-06-27T11:58:40.964 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:40.941+0000 7f88e3995700 1 -- 172.21.15.195:0/4219817590 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88dc156490 con 0x7f88dc06eff0 2022-06-27T11:58:40.964 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:40.941+0000 7f88c97fa700 1 -- 172.21.15.195:0/4219817590 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88d8013b80 con 0x7f88dc06eff0 2022-06-27T11:58:40.964 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:40.941+0000 7f88e3995700 1 -- 172.21.15.195:0/4219817590 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88dc156a50 con 0x7f88dc06eff0 2022-06-27T11:58:40.965 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:40.942+0000 7f88e3995700 1 -- 172.21.15.195:0/4219817590 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88dc0722c0 con 0x7f88dc06eff0 2022-06-27T11:58:40.965 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:40.942+0000 7f88c97fa700 1 -- 172.21.15.195:0/4219817590 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f88d8019030 con 0x7f88dc06eff0 2022-06-27T11:58:40.965 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:40.942+0000 7f88c97fa700 1 --2- 172.21.15.195:0/4219817590 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f88bc063d00 0x7f88bc0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:40.965 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:40.942+0000 7f88c97fa700 1 -- 172.21.15.195:0/4219817590 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6632+0+0 (secure 0 0 0) 0x7f88d80808c0 con 0x7f88dc06eff0 2022-06-27T11:58:40.965 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:40.942+0000 7f88e1731700 1 --2- 172.21.15.195:0/4219817590 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f88bc063d00 0x7f88bc0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:40.966 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:40.943+0000 7f88e1731700 1 --2- 172.21.15.195:0/4219817590 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f88bc063d00 0x7f88bc0661b0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f88cc0010e0 tx=0x7f88cc009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:40.966 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:40.947+0000 7f88c97fa700 1 -- 172.21.15.195:0/4219817590 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f88d804f270 con 0x7f88dc06eff0 2022-06-27T11:58:41.169 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:41.169+0000 7f88e3995700 1 -- 172.21.15.195:0/4219817590 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f88dc153f20 con 0x7f88dc06eff0 2022-06-27T11:58:41.170 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:41.169+0000 7f88c97fa700 1 -- 172.21.15.195:0/4219817590 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+6128 (secure 0 0 0) 0x7f88d8017090 con 0x7f88dc06eff0 2022-06-27T11:58:41.171 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:58:41.171 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":5192,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/2651813808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":2651813808},{"type":"v1","addr":"172.21.15.202:6833","nonce":2651813808}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:58:32.073997+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5392},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5392":{"gid":5392,"name":"d","rank":0,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.202:6835/2089155519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2089155519},{"type":"v1","addr":"172.21.15.202:6835","nonce":2089155519}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:58:41.174 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:41.171+0000 7f88e3995700 1 -- 172.21.15.195:0/4219817590 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f88bc063d00 msgr2=0x7f88bc0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:41.175 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:41.171+0000 7f88e3995700 1 --2- 172.21.15.195:0/4219817590 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f88bc063d00 0x7f88bc0661b0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f88cc0010e0 tx=0x7f88cc009040 comp rx=0 tx=0).stop 2022-06-27T11:58:41.175 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:41.171+0000 7f88e3995700 1 -- 172.21.15.195:0/4219817590 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f88dc06eff0 msgr2=0x7f88dc146bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:41.175 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:41.171+0000 7f88e3995700 1 --2- 172.21.15.195:0/4219817590 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f88dc06eff0 0x7f88dc146bd0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f88d800e930 tx=0x7f88d8009060 comp rx=0 tx=0).stop 2022-06-27T11:58:41.176 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:41.171+0000 7f88e3995700 1 -- 172.21.15.195:0/4219817590 shutdown_connections 2022-06-27T11:58:41.176 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:41.171+0000 7f88e3995700 1 --2- 172.21.15.195:0/4219817590 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f88bc063d00 0x7f88bc0661b0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:41.177 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:41.171+0000 7f88e3995700 1 --2- 172.21.15.195:0/4219817590 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f88dc14d140 0x7f88dc1696f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:41.177 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:41.171+0000 7f88e3995700 1 --2- 172.21.15.195:0/4219817590 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f88dc067ed0 0x7f88dc1542a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:41.178 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:41.171+0000 7f88e3995700 1 --2- 172.21.15.195:0/4219817590 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f88dc06eff0 0x7f88dc146bd0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:41.179 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:41.171+0000 7f88e3995700 1 -- 172.21.15.195:0/4219817590 >> 172.21.15.195:0/4219817590 conn(0x7f88dc0c9b00 msgr2=0x7f88dc0c7d40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:58:41.179 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:41.171+0000 7f88e3995700 1 -- 172.21.15.195:0/4219817590 shutdown_connections 2022-06-27T11:58:41.180 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:41.171+0000 7f88e3995700 1 -- 172.21.15.195:0/4219817590 wait complete. 2022-06-27T11:58:41.180 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 42 2022-06-27T11:58:41.183 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2022-06-27T11:58:41.184 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 1 secs before thrashing 2022-06-27T11:58:42.020 INFO:tasks.ceph.mds.d.smithi202.stderr:2022-06-27T11:58:42.020+0000 7fd9137c2700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2022-06-27T11:58:42.186 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:58:42.347 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.328+0000 7f4d4c77a700 1 Processor -- start 2022-06-27T11:58:42.348 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.328+0000 7f4d4c77a700 1 -- start start 2022-06-27T11:58:42.348 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.328+0000 7f4d4c77a700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f4d440c40d0 0x7f4d44157010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:42.349 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.329+0000 7f4d4c77a700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f4d440c6190 0x7f4d44067ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:42.349 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.329+0000 7f4d4c77a700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d44156b80 con 0x7f4d440c44a0 2022-06-27T11:58:42.349 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.329+0000 7f4d4c77a700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d4406a040 con 0x7f4d440c40d0 2022-06-27T11:58:42.354 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.329+0000 7f4d4c77a700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d4406a300 con 0x7f4d440c6190 2022-06-27T11:58:42.354 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.329+0000 7f4d4a516700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f4d440c40d0 0x7f4d44157010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:42.355 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.329+0000 7f4d4ad17700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f4d440c44a0 0x7f4d4414cb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:51950/0 (socket says 172.21.15.195:51950) 2022-06-27T11:58:42.362 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.329+0000 7f4d4ad17700 1 -- 172.21.15.195:0/615543815 learned_addr learned my addr 172.21.15.195:0/615543815 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:58:42.363 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.329+0000 7f4d4a516700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f4d440c40d0 0x7f4d44157010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45758/0 (socket says 172.21.15.195:45758) 2022-06-27T11:58:42.368 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.329+0000 7f4d49d15700 1 --2- 172.21.15.195:0/615543815 >> v2:172.21.15.202:3300/0 conn(0x7f4d440c6190 0x7f4d44067ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:42.369 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.329+0000 7f4d4a516700 1 -- 172.21.15.195:0/615543815 >> v2:172.21.15.202:3300/0 conn(0x7f4d440c6190 msgr2=0x7f4d44067ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:58:42.369 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.329+0000 7f4d4a516700 1 --2- 172.21.15.195:0/615543815 >> v2:172.21.15.202:3300/0 conn(0x7f4d440c6190 0x7f4d44067ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:42.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.329+0000 7f4d4a516700 1 -- 172.21.15.195:0/615543815 >> v1:172.21.15.195:6789/0 conn(0x7f4d440c44a0 legacy=0x7f4d4414cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:42.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.329+0000 7f4d4a516700 1 -- 172.21.15.195:0/615543815 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d4406a5e0 con 0x7f4d440c40d0 2022-06-27T11:58:42.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.329+0000 7f4d49514700 1 -- 172.21.15.195:0/615543815 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3849326181 0 0) 0x7f4d44156b80 con 0x7f4d440c44a0 2022-06-27T11:58:42.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.329+0000 7f4d4a516700 1 --2- 172.21.15.195:0/615543815 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4d440c40d0 0x7f4d44157010 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f4d440c0680 tx=0x7f4d34009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=775fc5b883de5386 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:42.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.329+0000 7f4d49514700 1 -- 172.21.15.195:0/615543815 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d34011630 con 0x7f4d440c40d0 2022-06-27T11:58:42.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.329+0000 7f4d49514700 1 -- 172.21.15.195:0/615543815 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d34011790 con 0x7f4d440c40d0 2022-06-27T11:58:42.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.329+0000 7f4d49514700 1 -- 172.21.15.195:0/615543815 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d34011b30 con 0x7f4d440c40d0 2022-06-27T11:58:42.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.330+0000 7f4d4c77a700 1 -- 172.21.15.195:0/615543815 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4d440c40d0 msgr2=0x7f4d44157010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:42.372 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.330+0000 7f4d4c77a700 1 --2- 172.21.15.195:0/615543815 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4d440c40d0 0x7f4d44157010 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f4d440c0680 tx=0x7f4d34009060 comp rx=0 tx=0).stop 2022-06-27T11:58:42.372 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.330+0000 7f4d4c77a700 1 -- 172.21.15.195:0/615543815 shutdown_connections 2022-06-27T11:58:42.372 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.330+0000 7f4d4c77a700 1 --2- 172.21.15.195:0/615543815 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4d440c40d0 0x7f4d44157010 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:42.372 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.330+0000 7f4d4c77a700 1 --2- 172.21.15.195:0/615543815 >> v2:172.21.15.202:3300/0 conn(0x7f4d440c6190 0x7f4d44067ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:42.372 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.330+0000 7f4d4c77a700 1 -- 172.21.15.195:0/615543815 >> 172.21.15.195:0/615543815 conn(0x7f4d440ccb80 msgr2=0x7f4d440ccf80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:58:42.373 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.330+0000 7f4d4c77a700 1 -- 172.21.15.195:0/615543815 shutdown_connections 2022-06-27T11:58:42.373 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.330+0000 7f4d4c77a700 1 -- 172.21.15.195:0/615543815 wait complete. 2022-06-27T11:58:42.373 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.331+0000 7f4d4c77a700 1 Processor -- start 2022-06-27T11:58:42.374 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.331+0000 7f4d4c77a700 1 -- start start 2022-06-27T11:58:42.374 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.332+0000 7f4d4c77a700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4d440c40d0 0x7f4d441517a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:42.375 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.332+0000 7f4d4c77a700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4d440c44a0 0x7f4d441537c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:42.375 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.332+0000 7f4d4c77a700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4d440c6190 0x7f4d44166c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:42.375 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.332+0000 7f4d4c77a700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d44166770 con 0x7f4d440c6190 2022-06-27T11:58:42.376 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.332+0000 7f4d4c77a700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d44166420 con 0x7f4d440c44a0 2022-06-27T11:58:42.376 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.332+0000 7f4d4c77a700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d441523b0 con 0x7f4d440c40d0 2022-06-27T11:58:42.377 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.332+0000 7f4d4ad17700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4d440c6190 0x7f4d44166c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:42.378 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.332+0000 7f4d4ad17700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4d440c6190 0x7f4d44166c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45768/0 (socket says 172.21.15.195:45768) 2022-06-27T11:58:42.378 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.332+0000 7f4d4ad17700 1 -- 172.21.15.195:0/315732243 learned_addr learned my addr 172.21.15.195:0/315732243 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:58:42.379 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.332+0000 7f4d49d15700 1 --2- 172.21.15.195:0/315732243 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4d440c44a0 0x7f4d441537c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:42.379 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.332+0000 7f4d4a516700 1 --2- 172.21.15.195:0/315732243 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4d440c40d0 0x7f4d441517a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:42.380 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.332+0000 7f4d4ad17700 1 -- 172.21.15.195:0/315732243 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4d440c40d0 msgr2=0x7f4d441517a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:42.380 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.332+0000 7f4d4ad17700 1 --2- 172.21.15.195:0/315732243 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4d440c40d0 0x7f4d441517a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:42.381 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.332+0000 7f4d4ad17700 1 -- 172.21.15.195:0/315732243 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4d440c44a0 msgr2=0x7f4d441537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:42.381 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.332+0000 7f4d4ad17700 1 --2- 172.21.15.195:0/315732243 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4d440c44a0 0x7f4d441537c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:42.382 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.332+0000 7f4d4ad17700 1 -- 172.21.15.195:0/315732243 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d340092e0 con 0x7f4d440c6190 2022-06-27T11:58:42.382 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.332+0000 7f4d4ad17700 1 --2- 172.21.15.195:0/315732243 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4d440c6190 0x7f4d44166c00 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f4d3c00c860 tx=0x7f4d3c001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:42.383 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.333+0000 7f4d327fc700 1 -- 172.21.15.195:0/315732243 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d3c004c80 con 0x7f4d440c6190 2022-06-27T11:58:42.383 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.333+0000 7f4d327fc700 1 -- 172.21.15.195:0/315732243 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d3c004de0 con 0x7f4d440c6190 2022-06-27T11:58:42.383 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.333+0000 7f4d4c77a700 1 -- 172.21.15.195:0/315732243 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d44152690 con 0x7f4d440c6190 2022-06-27T11:58:42.384 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.333+0000 7f4d327fc700 1 -- 172.21.15.195:0/315732243 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d3c0073a0 con 0x7f4d440c6190 2022-06-27T11:58:42.384 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.333+0000 7f4d4c77a700 1 -- 172.21.15.195:0/315732243 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d44152c50 con 0x7f4d440c6190 2022-06-27T11:58:42.384 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.334+0000 7f4d4c77a700 1 -- 172.21.15.195:0/315732243 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d44166920 con 0x7f4d440c6190 2022-06-27T11:58:42.385 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.334+0000 7f4d327fc700 1 -- 172.21.15.195:0/315732243 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f4d3c01a030 con 0x7f4d440c6190 2022-06-27T11:58:42.385 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.334+0000 7f4d327fc700 1 --2- 172.21.15.195:0/315732243 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4d24063d00 0x7f4d240661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:42.386 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.334+0000 7f4d327fc700 1 -- 172.21.15.195:0/315732243 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6632+0+0 (secure 0 0 0) 0x7f4d3c07ff50 con 0x7f4d440c6190 2022-06-27T11:58:42.386 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.334+0000 7f4d4a516700 1 --2- 172.21.15.195:0/315732243 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4d24063d00 0x7f4d240661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:42.387 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.335+0000 7f4d4a516700 1 --2- 172.21.15.195:0/315732243 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4d24063d00 0x7f4d240661b0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f4d34011fd0 tx=0x7f4d34017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:42.387 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.339+0000 7f4d327fc700 1 -- 172.21.15.195:0/315732243 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f4d44166920 con 0x7f4d440c6190 2022-06-27T11:58:42.606 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.603+0000 7f4d4c77a700 1 -- 172.21.15.195:0/315732243 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4d441668b0 con 0x7f4d440c6190 2022-06-27T11:58:42.606 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.603+0000 7f4d327fc700 1 -- 172.21.15.195:0/315732243 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+6128 (secure 0 0 0) 0x7f4d3c018070 con 0x7f4d440c6190 2022-06-27T11:58:42.607 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:58:42.608 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":5192,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/2651813808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":2651813808},{"type":"v1","addr":"172.21.15.202:6833","nonce":2651813808}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:58:32.073997+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5392},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5392":{"gid":5392,"name":"d","rank":0,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.202:6835/2089155519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2089155519},{"type":"v1","addr":"172.21.15.202:6835","nonce":2089155519}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:58:42.611 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.605+0000 7f4d4c77a700 1 -- 172.21.15.195:0/315732243 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4d24063d00 msgr2=0x7f4d240661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:42.611 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.605+0000 7f4d4c77a700 1 --2- 172.21.15.195:0/315732243 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4d24063d00 0x7f4d240661b0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f4d34011fd0 tx=0x7f4d34017000 comp rx=0 tx=0).stop 2022-06-27T11:58:42.612 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.605+0000 7f4d4c77a700 1 -- 172.21.15.195:0/315732243 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4d440c6190 msgr2=0x7f4d44166c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:42.612 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.605+0000 7f4d4c77a700 1 --2- 172.21.15.195:0/315732243 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4d440c6190 0x7f4d44166c00 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f4d3c00c860 tx=0x7f4d3c001630 comp rx=0 tx=0).stop 2022-06-27T11:58:42.613 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.605+0000 7f4d4c77a700 1 -- 172.21.15.195:0/315732243 shutdown_connections 2022-06-27T11:58:42.614 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.605+0000 7f4d4c77a700 1 --2- 172.21.15.195:0/315732243 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4d24063d00 0x7f4d240661b0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:42.614 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.605+0000 7f4d4c77a700 1 --2- 172.21.15.195:0/315732243 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4d440c6190 0x7f4d44166c00 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:42.615 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.605+0000 7f4d4c77a700 1 --2- 172.21.15.195:0/315732243 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4d440c40d0 0x7f4d441517a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:42.620 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.605+0000 7f4d4c77a700 1 --2- 172.21.15.195:0/315732243 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4d440c44a0 0x7f4d441537c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:42.621 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.605+0000 7f4d4c77a700 1 -- 172.21.15.195:0/315732243 >> 172.21.15.195:0/315732243 conn(0x7f4d440ccb80 msgr2=0x7f4d440c9600 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:58:42.621 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.605+0000 7f4d4c77a700 1 -- 172.21.15.195:0/315732243 shutdown_connections 2022-06-27T11:58:42.621 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.605+0000 7f4d4c77a700 1 -- 172.21.15.195:0/315732243 wait complete. 2022-06-27T11:58:42.622 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 42 2022-06-27T11:58:42.623 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.d (rank=0) 2022-06-27T11:58:42.624 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2022-06-27T11:58:42.624 INFO:teuthology.orchestra.run:waiting for 300 2022-06-27T11:58:42.637 INFO:tasks.ceph.mds.d:Stopped 2022-06-27T11:58:42.638 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:58:42.782 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.763+0000 7f3c0be9d700 1 Processor -- start 2022-06-27T11:58:42.783 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.763+0000 7f3c0be9d700 1 -- start start 2022-06-27T11:58:42.783 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.764+0000 7f3c0be9d700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3c04150580 0x7f3c04156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:42.783 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.764+0000 7f3c0be9d700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f3c04150950 0x7f3c040c7b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:42.783 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.764+0000 7f3c0be9d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f3c0406f620 0x7f3c0406d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:42.784 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.764+0000 7f3c0be9d700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c04157430 con 0x7f3c0406f620 2022-06-27T11:58:42.784 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.764+0000 7f3c0be9d700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c040c76b0 con 0x7f3c04150950 2022-06-27T11:58:42.784 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.764+0000 7f3c0be9d700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c040bad70 con 0x7f3c04150580 2022-06-27T11:58:42.784 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.764+0000 7f3c0a43a700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f3c0406f620 0x7f3c0406d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:42.785 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.764+0000 7f3c0a43a700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f3c0406f620 0x7f3c0406d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45778/0 (socket says 172.21.15.195:45778) 2022-06-27T11:58:42.785 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.765+0000 7f3c0a43a700 1 -- 172.21.15.195:0/119190658 learned_addr learned my addr 172.21.15.195:0/119190658 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:58:42.785 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.765+0000 7f3c09c39700 1 --2- 172.21.15.195:0/119190658 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3c04150580 0x7f3c04156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:42.785 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.765+0000 7f3c09438700 1 --2- 172.21.15.195:0/119190658 >> v2:172.21.15.202:3300/0 conn(0x7f3c04150950 0x7f3c040c7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:42.785 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.765+0000 7f3c0a43a700 1 -- 172.21.15.195:0/119190658 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3c04150580 msgr2=0x7f3c04156ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:58:42.787 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.765+0000 7f3c0a43a700 1 --2- 172.21.15.195:0/119190658 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3c04150580 0x7f3c04156ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:42.787 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.765+0000 7f3c0a43a700 1 -- 172.21.15.195:0/119190658 >> v2:172.21.15.202:3300/0 conn(0x7f3c04150950 msgr2=0x7f3c040c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:58:42.788 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.765+0000 7f3c0a43a700 1 --2- 172.21.15.195:0/119190658 >> v2:172.21.15.202:3300/0 conn(0x7f3c04150950 0x7f3c040c7b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:42.788 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.765+0000 7f3c0a43a700 1 -- 172.21.15.195:0/119190658 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c040ba4d0 con 0x7f3c0406f620 2022-06-27T11:58:42.789 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.765+0000 7f3c09c39700 1 --2- 172.21.15.195:0/119190658 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3c04150580 0x7f3c04156ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T11:58:42.789 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.765+0000 7f3c09438700 1 --2- 172.21.15.195:0/119190658 >> v2:172.21.15.202:3300/0 conn(0x7f3c04150950 0x7f3c040c7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T11:58:42.789 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.765+0000 7f3c0a43a700 1 --2- 172.21.15.195:0/119190658 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3c0406f620 0x7f3c0406d6a0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc009f20 tx=0x7f3bfc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e25c43ff257adf09 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:42.790 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.765+0000 7f3c08c37700 1 -- 172.21.15.195:0/119190658 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bfc00e070 con 0x7f3c0406f620 2022-06-27T11:58:42.790 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.765+0000 7f3c08c37700 1 -- 172.21.15.195:0/119190658 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bfc00ba40 con 0x7f3c0406f620 2022-06-27T11:58:42.791 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.765+0000 7f3c08c37700 1 -- 172.21.15.195:0/119190658 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bfc0092e0 con 0x7f3c0406f620 2022-06-27T11:58:42.791 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.766+0000 7f3c0be9d700 1 -- 172.21.15.195:0/119190658 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3c0406f620 msgr2=0x7f3c0406d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:42.792 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.766+0000 7f3c0be9d700 1 --2- 172.21.15.195:0/119190658 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3c0406f620 0x7f3c0406d6a0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc009f20 tx=0x7f3bfc009840 comp rx=0 tx=0).stop 2022-06-27T11:58:42.792 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.766+0000 7f3c0be9d700 1 -- 172.21.15.195:0/119190658 shutdown_connections 2022-06-27T11:58:42.793 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.766+0000 7f3c0be9d700 1 --2- 172.21.15.195:0/119190658 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3c0406f620 0x7f3c0406d6a0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:42.793 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.766+0000 7f3c0be9d700 1 --2- 172.21.15.195:0/119190658 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3c04150580 0x7f3c04156ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:42.794 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.766+0000 7f3c0be9d700 1 --2- 172.21.15.195:0/119190658 >> v2:172.21.15.202:3300/0 conn(0x7f3c04150950 0x7f3c040c7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:42.795 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.766+0000 7f3c0be9d700 1 -- 172.21.15.195:0/119190658 >> 172.21.15.195:0/119190658 conn(0x7f3c04063af0 msgr2=0x7f3c04063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:58:42.795 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.766+0000 7f3c0be9d700 1 -- 172.21.15.195:0/119190658 shutdown_connections 2022-06-27T11:58:42.796 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.766+0000 7f3c0be9d700 1 -- 172.21.15.195:0/119190658 wait complete. 2022-06-27T11:58:42.796 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.768+0000 7f3c0be9d700 1 Processor -- start 2022-06-27T11:58:42.797 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.768+0000 7f3c0be9d700 1 -- start start 2022-06-27T11:58:42.797 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.769+0000 7f3c0be9d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3c0406f620 0x7f3c0414f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:42.798 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.769+0000 7f3c0be9d700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3c04150580 0x7f3c0414a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:42.798 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.769+0000 7f3c0be9d700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3c04150950 0x7f3c0414ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:42.799 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.769+0000 7f3c0be9d700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0414a500 con 0x7f3c0406f620 2022-06-27T11:58:42.799 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.769+0000 7f3c0be9d700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c041673d0 con 0x7f3c04150580 2022-06-27T11:58:42.800 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.769+0000 7f3c0be9d700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c04147d40 con 0x7f3c04150950 2022-06-27T11:58:42.800 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.769+0000 7f3c09c39700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3c0406f620 0x7f3c0414f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:42.801 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.769+0000 7f3c09c39700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3c0406f620 0x7f3c0414f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45780/0 (socket says 172.21.15.195:45780) 2022-06-27T11:58:42.801 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.769+0000 7f3c09c39700 1 -- 172.21.15.195:0/1589739448 learned_addr learned my addr 172.21.15.195:0/1589739448 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:58:42.801 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.769+0000 7f3c0a43a700 1 --2- 172.21.15.195:0/1589739448 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3c04150950 0x7f3c0414ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:42.802 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.769+0000 7f3c09438700 1 --2- 172.21.15.195:0/1589739448 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3c04150580 0x7f3c0414a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:42.802 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.769+0000 7f3c09c39700 1 -- 172.21.15.195:0/1589739448 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3c04150950 msgr2=0x7f3c0414ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:42.803 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.769+0000 7f3c09c39700 1 --2- 172.21.15.195:0/1589739448 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3c04150950 0x7f3c0414ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:42.803 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.769+0000 7f3c09c39700 1 -- 172.21.15.195:0/1589739448 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3c04150580 msgr2=0x7f3c0414a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:42.804 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.769+0000 7f3c09c39700 1 --2- 172.21.15.195:0/1589739448 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3c04150580 0x7f3c0414a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:42.804 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.769+0000 7f3c09c39700 1 -- 172.21.15.195:0/1589739448 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bfc009ae0 con 0x7f3c0406f620 2022-06-27T11:58:42.805 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.769+0000 7f3c09c39700 1 --2- 172.21.15.195:0/1589739448 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3c0406f620 0x7f3c0414f5e0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f3c0000c880 tx=0x7f3c00008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:42.805 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.770+0000 7f3bf1ffb700 1 -- 172.21.15.195:0/1589739448 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c00011600 con 0x7f3c0406f620 2022-06-27T11:58:42.806 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.770+0000 7f3bf1ffb700 1 -- 172.21.15.195:0/1589739448 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c00011760 con 0x7f3c0406f620 2022-06-27T11:58:42.806 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.770+0000 7f3c0be9d700 1 -- 172.21.15.195:0/1589739448 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c04148020 con 0x7f3c0406f620 2022-06-27T11:58:42.807 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.770+0000 7f3bf1ffb700 1 -- 172.21.15.195:0/1589739448 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c00011ae0 con 0x7f3c0406f620 2022-06-27T11:58:42.807 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.770+0000 7f3c0be9d700 1 -- 172.21.15.195:0/1589739448 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c0414a120 con 0x7f3c0406f620 2022-06-27T11:58:42.813 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.771+0000 7f3c0be9d700 1 -- 172.21.15.195:0/1589739448 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c040722c0 con 0x7f3c0406f620 2022-06-27T11:58:42.814 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.771+0000 7f3bf1ffb700 1 -- 172.21.15.195:0/1589739448 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f3c00021070 con 0x7f3c0406f620 2022-06-27T11:58:42.814 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.771+0000 7f3bf1ffb700 1 --2- 172.21.15.195:0/1589739448 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3be4063db0 0x7f3be4066260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:42.815 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.772+0000 7f3bf1ffb700 1 -- 172.21.15.195:0/1589739448 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6632+0+0 (secure 0 0 0) 0x7f3c00080aa0 con 0x7f3c0406f620 2022-06-27T11:58:42.815 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.772+0000 7f3c09438700 1 --2- 172.21.15.195:0/1589739448 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3be4063db0 0x7f3be4066260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:42.815 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.772+0000 7f3c09438700 1 --2- 172.21.15.195:0/1589739448 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3be4063db0 0x7f3be4066260 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f3c04071f60 tx=0x7f3bf4008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:42.816 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:42.776+0000 7f3bf1ffb700 1 -- 172.21.15.195:0/1589739448 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f3c0004f360 con 0x7f3c0406f620 2022-06-27T11:58:43.008 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:43.007+0000 7f3c0be9d700 1 -- 172.21.15.195:0/1589739448 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c0414e360 con 0x7f3c0406f620 2022-06-27T11:58:43.009 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:43.008+0000 7f3bf1ffb700 1 -- 172.21.15.195:0/1589739448 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+6128 (secure 0 0 0) 0x7f3c0001f090 con 0x7f3c0406f620 2022-06-27T11:58:43.009 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:58:43.010 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":5192,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/2651813808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":2651813808},{"type":"v1","addr":"172.21.15.202:6833","nonce":2651813808}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:58:32.073997+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5392},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5392":{"gid":5392,"name":"d","rank":0,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.202:6835/2089155519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2089155519},{"type":"v1","addr":"172.21.15.202:6835","nonce":2089155519}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:58:43.017 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:43.010+0000 7f3c0be9d700 1 -- 172.21.15.195:0/1589739448 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3be4063db0 msgr2=0x7f3be4066260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:43.017 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:43.010+0000 7f3c0be9d700 1 --2- 172.21.15.195:0/1589739448 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3be4063db0 0x7f3be4066260 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f3c04071f60 tx=0x7f3bf4008040 comp rx=0 tx=0).stop 2022-06-27T11:58:43.018 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:43.010+0000 7f3c0be9d700 1 -- 172.21.15.195:0/1589739448 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3c0406f620 msgr2=0x7f3c0414f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:43.019 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:43.010+0000 7f3c0be9d700 1 --2- 172.21.15.195:0/1589739448 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3c0406f620 0x7f3c0414f5e0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f3c0000c880 tx=0x7f3c00008040 comp rx=0 tx=0).stop 2022-06-27T11:58:43.019 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:43.010+0000 7f3c0be9d700 1 -- 172.21.15.195:0/1589739448 shutdown_connections 2022-06-27T11:58:43.020 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:43.010+0000 7f3c0be9d700 1 --2- 172.21.15.195:0/1589739448 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3be4063db0 0x7f3be4066260 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:43.020 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:43.010+0000 7f3c0be9d700 1 --2- 172.21.15.195:0/1589739448 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3c0406f620 0x7f3c0414f5e0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:43.020 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:43.010+0000 7f3c0be9d700 1 --2- 172.21.15.195:0/1589739448 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3c04150950 0x7f3c0414ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:43.021 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:43.010+0000 7f3c0be9d700 1 --2- 172.21.15.195:0/1589739448 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3c04150580 0x7f3c0414a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:43.021 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:43.010+0000 7f3c0be9d700 1 -- 172.21.15.195:0/1589739448 >> 172.21.15.195:0/1589739448 conn(0x7f3c04063af0 msgr2=0x7f3c04069e70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:58:43.022 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:43.010+0000 7f3c0be9d700 1 -- 172.21.15.195:0/1589739448 shutdown_connections 2022-06-27T11:58:43.022 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:43.010+0000 7f3c0be9d700 1 -- 172.21.15.195:0/1589739448 wait complete. 2022-06-27T11:58:43.023 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 42 2022-06-27T11:58:43.028 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-06-27T11:58:45.032 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:58:45.179 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.170+0000 7ffa6c447700 1 Processor -- start 2022-06-27T11:58:45.179 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.170+0000 7ffa6c447700 1 -- start start 2022-06-27T11:58:45.180 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.171+0000 7ffa6c447700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7ffa64151930 0x7ffa640c2120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:45.180 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.171+0000 7ffa6c447700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa64154da0 con 0x7ffa64151560 2022-06-27T11:58:45.181 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.171+0000 7ffa6c447700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa640c12a0 con 0x7ffa64154900 2022-06-27T11:58:45.181 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.171+0000 7ffa6c447700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa640c1580 con 0x7ffa64151930 2022-06-27T11:58:45.182 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.172+0000 7ffa6a9e4700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7ffa64151930 0x7ffa640c2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:45.186 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.172+0000 7ffa6a1e3700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7ffa64151560 0x7ffa641541f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:51986/0 (socket says 172.21.15.195:51986) 2022-06-27T11:58:45.188 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.172+0000 7ffa6a1e3700 1 -- 172.21.15.195:0/3932911211 learned_addr learned my addr 172.21.15.195:0/3932911211 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:58:45.188 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.172+0000 7ffa6a9e4700 1 -- 172.21.15.195:0/3932911211 >> v1:172.21.15.202:6789/0 conn(0x7ffa64154900 legacy=0x7ffa6415cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:45.188 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.172+0000 7ffa6a9e4700 1 -- 172.21.15.195:0/3932911211 >> v1:172.21.15.195:6789/0 conn(0x7ffa64151560 legacy=0x7ffa641541f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:45.189 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.172+0000 7ffa6a9e4700 1 -- 172.21.15.195:0/3932911211 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa640c29f0 con 0x7ffa64151930 2022-06-27T11:58:45.189 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.172+0000 7ffa691e1700 1 -- 172.21.15.195:0/3932911211 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1189144327 0 0) 0x7ffa64154da0 con 0x7ffa64151560 2022-06-27T11:58:45.194 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.172+0000 7ffa6a9e4700 1 --2- 172.21.15.195:0/3932911211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffa64151930 0x7ffa640c2120 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c009e60 tx=0x7ffa5c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8678ece2c056b7a server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:45.195 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.172+0000 7ffa691e1700 1 -- 172.21.15.195:0/3932911211 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa5c00e070 con 0x7ffa64151930 2022-06-27T11:58:45.195 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.172+0000 7ffa691e1700 1 -- 172.21.15.195:0/3932911211 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa5c00bcc0 con 0x7ffa64151930 2022-06-27T11:58:45.195 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.172+0000 7ffa691e1700 1 -- 172.21.15.195:0/3932911211 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa5c012680 con 0x7ffa64151930 2022-06-27T11:58:45.196 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.173+0000 7ffa6c447700 1 -- 172.21.15.195:0/3932911211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffa64151930 msgr2=0x7ffa640c2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:45.196 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.173+0000 7ffa6c447700 1 --2- 172.21.15.195:0/3932911211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffa64151930 0x7ffa640c2120 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c009e60 tx=0x7ffa5c009780 comp rx=0 tx=0).stop 2022-06-27T11:58:45.202 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.173+0000 7ffa6c447700 1 -- 172.21.15.195:0/3932911211 shutdown_connections 2022-06-27T11:58:45.203 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.173+0000 7ffa6c447700 1 --2- 172.21.15.195:0/3932911211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffa64151930 0x7ffa640c2120 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:45.203 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.173+0000 7ffa6c447700 1 -- 172.21.15.195:0/3932911211 >> 172.21.15.195:0/3932911211 conn(0x7ffa640c6350 msgr2=0x7ffa640c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:58:45.204 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.173+0000 7ffa6c447700 1 -- 172.21.15.195:0/3932911211 shutdown_connections 2022-06-27T11:58:45.204 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.173+0000 7ffa6c447700 1 -- 172.21.15.195:0/3932911211 wait complete. 2022-06-27T11:58:45.206 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.174+0000 7ffa6c447700 1 Processor -- start 2022-06-27T11:58:45.207 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.174+0000 7ffa6c447700 1 -- start start 2022-06-27T11:58:45.207 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.175+0000 7ffa6c447700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ffa64151560 0x7ffa640c8db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:45.208 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.175+0000 7ffa6c447700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffa64151930 0x7ffa64155a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:45.208 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.175+0000 7ffa6c447700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ffa64154900 0x7ffa64155f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:45.209 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.175+0000 7ffa6c447700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa64157970 con 0x7ffa64151930 2022-06-27T11:58:45.209 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.175+0000 7ffa6c447700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa64157160 con 0x7ffa64154900 2022-06-27T11:58:45.209 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.175+0000 7ffa6c447700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa641555b0 con 0x7ffa64151560 2022-06-27T11:58:45.210 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.175+0000 7ffa699e2700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffa64151930 0x7ffa64155a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:45.210 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.175+0000 7ffa699e2700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffa64151930 0x7ffa64155a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45806/0 (socket says 172.21.15.195:45806) 2022-06-27T11:58:45.211 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.175+0000 7ffa699e2700 1 -- 172.21.15.195:0/3570037286 learned_addr learned my addr 172.21.15.195:0/3570037286 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:58:45.211 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.175+0000 7ffa6a1e3700 1 --2- 172.21.15.195:0/3570037286 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ffa64151560 0x7ffa640c8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:45.212 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.175+0000 7ffa6a9e4700 1 --2- 172.21.15.195:0/3570037286 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ffa64154900 0x7ffa64155f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:45.212 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.175+0000 7ffa699e2700 1 -- 172.21.15.195:0/3570037286 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ffa64151560 msgr2=0x7ffa640c8db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:45.213 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.175+0000 7ffa699e2700 1 --2- 172.21.15.195:0/3570037286 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ffa64151560 0x7ffa640c8db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:45.213 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.175+0000 7ffa699e2700 1 -- 172.21.15.195:0/3570037286 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ffa64154900 msgr2=0x7ffa64155f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:45.213 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.175+0000 7ffa699e2700 1 --2- 172.21.15.195:0/3570037286 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ffa64154900 0x7ffa64155f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:45.214 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.175+0000 7ffa699e2700 1 -- 172.21.15.195:0/3570037286 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa5c009a20 con 0x7ffa64151930 2022-06-27T11:58:45.214 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.175+0000 7ffa6a9e4700 1 --2- 172.21.15.195:0/3570037286 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ffa64154900 0x7ffa64155f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T11:58:45.215 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.175+0000 7ffa6a1e3700 1 --2- 172.21.15.195:0/3570037286 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ffa64151560 0x7ffa640c8db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T11:58:45.215 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.175+0000 7ffa699e2700 1 --2- 172.21.15.195:0/3570037286 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffa64151930 0x7ffa64155a40 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7ffa6000c440 tx=0x7ffa60007e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:45.216 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.175+0000 7ffa567fc700 1 -- 172.21.15.195:0/3570037286 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa600050d0 con 0x7ffa64151930 2022-06-27T11:58:45.216 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.175+0000 7ffa567fc700 1 -- 172.21.15.195:0/3570037286 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa60005250 con 0x7ffa64151930 2022-06-27T11:58:45.217 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.176+0000 7ffa6c447700 1 -- 172.21.15.195:0/3570037286 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa64168700 con 0x7ffa64151930 2022-06-27T11:58:45.217 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.176+0000 7ffa567fc700 1 -- 172.21.15.195:0/3570037286 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa600117e0 con 0x7ffa64151930 2022-06-27T11:58:45.217 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.176+0000 7ffa6c447700 1 -- 172.21.15.195:0/3570037286 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa64157690 con 0x7ffa64151930 2022-06-27T11:58:45.218 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.176+0000 7ffa6c447700 1 -- 172.21.15.195:0/3570037286 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa64155380 con 0x7ffa64151930 2022-06-27T11:58:45.218 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.177+0000 7ffa567fc700 1 -- 172.21.15.195:0/3570037286 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7ffa60019030 con 0x7ffa64151930 2022-06-27T11:58:45.219 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.177+0000 7ffa567fc700 1 --2- 172.21.15.195:0/3570037286 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ffa44063d00 0x7ffa440661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:45.219 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.177+0000 7ffa567fc700 1 -- 172.21.15.195:0/3570037286 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6632+0+0 (secure 0 0 0) 0x7ffa6007fbb0 con 0x7ffa64151930 2022-06-27T11:58:45.220 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.177+0000 7ffa6a1e3700 1 --2- 172.21.15.195:0/3570037286 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ffa44063d00 0x7ffa440661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:45.220 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.178+0000 7ffa6a1e3700 1 --2- 172.21.15.195:0/3570037286 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ffa44063d00 0x7ffa440661b0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ffa58001790 tx=0x7ffa58009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:45.221 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.181+0000 7ffa567fc700 1 -- 172.21.15.195:0/3570037286 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7ffa64155380 con 0x7ffa64151930 2022-06-27T11:58:45.418 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.408+0000 7ffa6c447700 1 -- 172.21.15.195:0/3570037286 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffa64155380 con 0x7ffa64151930 2022-06-27T11:58:45.419 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.408+0000 7ffa567fc700 1 -- 172.21.15.195:0/3570037286 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+6128 (secure 0 0 0) 0x7ffa64155380 con 0x7ffa64151930 2022-06-27T11:58:45.419 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:58:45.420 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":5192,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/2651813808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":2651813808},{"type":"v1","addr":"172.21.15.202:6833","nonce":2651813808}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:58:32.073997+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5392},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5392":{"gid":5392,"name":"d","rank":0,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.202:6835/2089155519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2089155519},{"type":"v1","addr":"172.21.15.202:6835","nonce":2089155519}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:58:45.435 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.410+0000 7ffa6c447700 1 -- 172.21.15.195:0/3570037286 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ffa44063d00 msgr2=0x7ffa440661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:45.435 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.410+0000 7ffa6c447700 1 --2- 172.21.15.195:0/3570037286 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ffa44063d00 0x7ffa440661b0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ffa58001790 tx=0x7ffa58009040 comp rx=0 tx=0).stop 2022-06-27T11:58:45.435 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.410+0000 7ffa6c447700 1 -- 172.21.15.195:0/3570037286 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffa64151930 msgr2=0x7ffa64155a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:45.436 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.410+0000 7ffa6c447700 1 --2- 172.21.15.195:0/3570037286 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffa64151930 0x7ffa64155a40 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7ffa6000c440 tx=0x7ffa60007e40 comp rx=0 tx=0).stop 2022-06-27T11:58:45.436 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.410+0000 7ffa6c447700 1 -- 172.21.15.195:0/3570037286 shutdown_connections 2022-06-27T11:58:45.437 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.410+0000 7ffa6c447700 1 --2- 172.21.15.195:0/3570037286 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ffa44063d00 0x7ffa440661b0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:45.437 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.410+0000 7ffa6c447700 1 --2- 172.21.15.195:0/3570037286 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ffa64154900 0x7ffa64155f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:45.437 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.410+0000 7ffa6c447700 1 --2- 172.21.15.195:0/3570037286 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ffa64151560 0x7ffa640c8db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:45.438 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.410+0000 7ffa6c447700 1 --2- 172.21.15.195:0/3570037286 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffa64151930 0x7ffa64155a40 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:45.438 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.410+0000 7ffa6c447700 1 -- 172.21.15.195:0/3570037286 >> 172.21.15.195:0/3570037286 conn(0x7ffa640c6350 msgr2=0x7ffa64150550 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:58:45.439 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.410+0000 7ffa6c447700 1 -- 172.21.15.195:0/3570037286 shutdown_connections 2022-06-27T11:58:45.439 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:45.410+0000 7ffa6c447700 1 -- 172.21.15.195:0/3570037286 wait complete. 2022-06-27T11:58:45.440 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 42 2022-06-27T11:58:45.455 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-06-27T11:58:47.457 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:58:47.615 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.600+0000 7f9c66bac700 1 Processor -- start 2022-06-27T11:58:47.616 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.600+0000 7f9c66bac700 1 -- start start 2022-06-27T11:58:47.616 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.601+0000 7f9c66bac700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9c6006e850 0x7f9c601570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:47.616 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.601+0000 7f9c66bac700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f9c6006fef0 0x7f9c60069de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:47.617 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.601+0000 7f9c66bac700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c6006fba0 con 0x7f9c600c34a0 2022-06-27T11:58:47.617 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.601+0000 7f9c66bac700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c601575e0 con 0x7f9c6006fef0 2022-06-27T11:58:47.618 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.601+0000 7f9c66bac700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c6006f5e0 con 0x7f9c6006e850 2022-06-27T11:58:47.620 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.601+0000 7f9c5ffff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f9c6006fef0 0x7f9c60069de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:47.621 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.601+0000 7f9c5ffff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f9c6006fef0 0x7f9c60069de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45814/0 (socket says 172.21.15.195:45814) 2022-06-27T11:58:47.621 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.601+0000 7f9c5ffff700 1 -- 172.21.15.195:0/3053578880 learned_addr learned my addr 172.21.15.195:0/3053578880 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:58:47.622 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.601+0000 7f9c64948700 1 --2- 172.21.15.195:0/3053578880 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9c6006e850 0x7f9c601570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:47.622 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.602+0000 7f9c5ffff700 1 -- 172.21.15.195:0/3053578880 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9c6006e850 msgr2=0x7f9c601570a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:58:47.623 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.602+0000 7f9c5ffff700 1 --2- 172.21.15.195:0/3053578880 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9c6006e850 0x7f9c601570a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:47.624 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.602+0000 7f9c5ffff700 1 -- 172.21.15.195:0/3053578880 >> v1:172.21.15.202:6789/0 conn(0x7f9c600c34a0 legacy=0x7f9c600c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:58:47.624 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.602+0000 7f9c5ffff700 1 -- 172.21.15.195:0/3053578880 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c6006f8c0 con 0x7f9c6006fef0 2022-06-27T11:58:47.625 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.602+0000 7f9c64948700 1 --2- 172.21.15.195:0/3053578880 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9c6006e850 0x7f9c601570a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T11:58:47.625 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.602+0000 7f9c5ffff700 1 --2- 172.21.15.195:0/3053578880 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9c6006fef0 0x7f9c60069de0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f9c4c009860 tx=0x7f9c4c009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1819303e5af847f0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:47.625 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.602+0000 7f9c5f7fe700 1 -- 172.21.15.195:0/3053578880 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c4c004030 con 0x7f9c6006fef0 2022-06-27T11:58:47.629 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.602+0000 7f9c5f7fe700 1 -- 172.21.15.195:0/3053578880 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c4c004190 con 0x7f9c6006fef0 2022-06-27T11:58:47.630 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.602+0000 7f9c5f7fe700 1 -- 172.21.15.195:0/3053578880 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c4c00b6d0 con 0x7f9c6006fef0 2022-06-27T11:58:47.630 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.602+0000 7f9c66bac700 1 -- 172.21.15.195:0/3053578880 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9c6006fef0 msgr2=0x7f9c60069de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:47.631 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.602+0000 7f9c66bac700 1 --2- 172.21.15.195:0/3053578880 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9c6006fef0 0x7f9c60069de0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f9c4c009860 tx=0x7f9c4c009120 comp rx=0 tx=0).stop 2022-06-27T11:58:47.631 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.602+0000 7f9c66bac700 1 -- 172.21.15.195:0/3053578880 shutdown_connections 2022-06-27T11:58:47.632 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.602+0000 7f9c66bac700 1 --2- 172.21.15.195:0/3053578880 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9c6006e850 0x7f9c601570a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:47.632 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.602+0000 7f9c66bac700 1 --2- 172.21.15.195:0/3053578880 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9c6006fef0 0x7f9c60069de0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:47.633 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.602+0000 7f9c66bac700 1 -- 172.21.15.195:0/3053578880 >> 172.21.15.195:0/3053578880 conn(0x7f9c600c6350 msgr2=0x7f9c600c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:58:47.634 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.602+0000 7f9c66bac700 1 -- 172.21.15.195:0/3053578880 shutdown_connections 2022-06-27T11:58:47.634 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.602+0000 7f9c66bac700 1 -- 172.21.15.195:0/3053578880 wait complete. 2022-06-27T11:58:47.635 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.604+0000 7f9c66bac700 1 Processor -- start 2022-06-27T11:58:47.635 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.604+0000 7f9c66bac700 1 -- start start 2022-06-27T11:58:47.636 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.604+0000 7f9c66bac700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9c6006e850 0x7f9c60152d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:47.636 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.604+0000 7f9c66bac700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9c600c34a0 0x7f9c601539e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:47.637 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.604+0000 7f9c66bac700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9c601532d0 0x7f9c60153f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:47.637 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.604+0000 7f9c66bac700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c60152900 con 0x7f9c601532d0 2022-06-27T11:58:47.637 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.604+0000 7f9c66bac700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c600c7040 con 0x7f9c6006e850 2022-06-27T11:58:47.639 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.604+0000 7f9c66bac700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c6014c280 con 0x7f9c600c34a0 2022-06-27T11:58:47.639 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.605+0000 7f9c65149700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9c601532d0 0x7f9c60153f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:47.643 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.605+0000 7f9c65149700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9c601532d0 0x7f9c60153f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45820/0 (socket says 172.21.15.195:45820) 2022-06-27T11:58:47.643 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.605+0000 7f9c65149700 1 -- 172.21.15.195:0/437083229 learned_addr learned my addr 172.21.15.195:0/437083229 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:58:47.643 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.605+0000 7f9c64948700 1 --2- 172.21.15.195:0/437083229 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9c6006e850 0x7f9c60152d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:47.644 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.605+0000 7f9c5ffff700 1 --2- 172.21.15.195:0/437083229 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9c600c34a0 0x7f9c601539e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:47.644 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.605+0000 7f9c65149700 1 -- 172.21.15.195:0/437083229 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9c600c34a0 msgr2=0x7f9c601539e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:47.645 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.605+0000 7f9c65149700 1 --2- 172.21.15.195:0/437083229 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9c600c34a0 0x7f9c601539e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:47.647 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.605+0000 7f9c65149700 1 -- 172.21.15.195:0/437083229 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9c6006e850 msgr2=0x7f9c60152d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:47.648 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.605+0000 7f9c65149700 1 --2- 172.21.15.195:0/437083229 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9c6006e850 0x7f9c60152d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:47.649 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.605+0000 7f9c65149700 1 -- 172.21.15.195:0/437083229 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c4c0093a0 con 0x7f9c601532d0 2022-06-27T11:58:47.649 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.605+0000 7f9c65149700 1 --2- 172.21.15.195:0/437083229 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9c601532d0 0x7f9c60153f20 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f9c5800c840 tx=0x7f9c58003bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:47.650 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.605+0000 7f9c5cff9700 1 -- 172.21.15.195:0/437083229 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c58012630 con 0x7f9c601532d0 2022-06-27T11:58:47.650 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.605+0000 7f9c5cff9700 1 -- 172.21.15.195:0/437083229 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c580127b0 con 0x7f9c601532d0 2022-06-27T11:58:47.651 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.605+0000 7f9c66bac700 1 -- 172.21.15.195:0/437083229 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c600c0610 con 0x7f9c601532d0 2022-06-27T11:58:47.653 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.605+0000 7f9c5cff9700 1 -- 172.21.15.195:0/437083229 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c58012b70 con 0x7f9c601532d0 2022-06-27T11:58:47.653 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.605+0000 7f9c66bac700 1 -- 172.21.15.195:0/437083229 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c601525d0 con 0x7f9c601532d0 2022-06-27T11:58:47.654 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.606+0000 7f9c66bac700 1 -- 172.21.15.195:0/437083229 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c600722c0 con 0x7f9c601532d0 2022-06-27T11:58:47.654 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.607+0000 7f9c5cff9700 1 -- 172.21.15.195:0/437083229 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f9c58022070 con 0x7f9c601532d0 2022-06-27T11:58:47.655 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.607+0000 7f9c5cff9700 1 --2- 172.21.15.195:0/437083229 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9c48063d00 0x7f9c480661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:47.655 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.607+0000 7f9c5cff9700 1 -- 172.21.15.195:0/437083229 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6632+0+0 (secure 0 0 0) 0x7f9c58080bc0 con 0x7f9c601532d0 2022-06-27T11:58:47.655 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.607+0000 7f9c64948700 1 --2- 172.21.15.195:0/437083229 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9c48063d00 0x7f9c480661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:47.656 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.607+0000 7f9c64948700 1 --2- 172.21.15.195:0/437083229 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9c48063d00 0x7f9c480661b0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f9c5400aa10 tx=0x7f9c54008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:47.656 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.611+0000 7f9c5cff9700 1 -- 172.21.15.195:0/437083229 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f9c5804f570 con 0x7f9c601532d0 2022-06-27T11:58:47.831 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.831+0000 7f9c66bac700 1 -- 172.21.15.195:0/437083229 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9c60152a40 con 0x7f9c601532d0 2022-06-27T11:58:47.832 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.832+0000 7f9c5cff9700 1 -- 172.21.15.195:0/437083229 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+6128 (secure 0 0 0) 0x7f9c58016070 con 0x7f9c601532d0 2022-06-27T11:58:47.833 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:58:47.833 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":5192,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/2651813808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":2651813808},{"type":"v1","addr":"172.21.15.202:6833","nonce":2651813808}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:58:32.073997+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5392},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5392":{"gid":5392,"name":"d","rank":0,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.202:6835/2089155519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2089155519},{"type":"v1","addr":"172.21.15.202:6835","nonce":2089155519}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:58:47.835 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.834+0000 7f9c66bac700 1 -- 172.21.15.195:0/437083229 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9c48063d00 msgr2=0x7f9c480661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:47.836 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.834+0000 7f9c66bac700 1 --2- 172.21.15.195:0/437083229 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9c48063d00 0x7f9c480661b0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f9c5400aa10 tx=0x7f9c54008040 comp rx=0 tx=0).stop 2022-06-27T11:58:47.836 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.834+0000 7f9c66bac700 1 -- 172.21.15.195:0/437083229 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9c601532d0 msgr2=0x7f9c60153f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:47.837 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.834+0000 7f9c66bac700 1 --2- 172.21.15.195:0/437083229 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9c601532d0 0x7f9c60153f20 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f9c5800c840 tx=0x7f9c58003bd0 comp rx=0 tx=0).stop 2022-06-27T11:58:47.837 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.834+0000 7f9c66bac700 1 -- 172.21.15.195:0/437083229 shutdown_connections 2022-06-27T11:58:47.837 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.834+0000 7f9c66bac700 1 --2- 172.21.15.195:0/437083229 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9c48063d00 0x7f9c480661b0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:47.838 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.834+0000 7f9c66bac700 1 --2- 172.21.15.195:0/437083229 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9c601532d0 0x7f9c60153f20 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:47.838 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.834+0000 7f9c66bac700 1 --2- 172.21.15.195:0/437083229 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9c6006e850 0x7f9c60152d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:47.838 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.834+0000 7f9c66bac700 1 --2- 172.21.15.195:0/437083229 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9c600c34a0 0x7f9c601539e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:47.838 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.834+0000 7f9c66bac700 1 -- 172.21.15.195:0/437083229 >> 172.21.15.195:0/437083229 conn(0x7f9c600c6350 msgr2=0x7f9c601506c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:58:47.839 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.834+0000 7f9c66bac700 1 -- 172.21.15.195:0/437083229 shutdown_connections 2022-06-27T11:58:47.839 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:47.834+0000 7f9c66bac700 1 -- 172.21.15.195:0/437083229 wait complete. 2022-06-27T11:58:47.839 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 42 2022-06-27T11:58:47.847 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-06-27T11:58:49.849 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:58:49.983 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:49.983+0000 7f60d941d700 1 Processor -- start 2022-06-27T11:58:49.984 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:49.983+0000 7f60d941d700 1 -- start start 2022-06-27T11:58:49.997 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:49.984+0000 7f60d941d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f60d406e850 0x7f60d41570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:49.998 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:49.984+0000 7f60d941d700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f60d40c34a0 0x7f60d40c2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:49.998 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:49.984+0000 7f60d941d700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60d406fba0 con 0x7f60d406fef0 2022-06-27T11:58:49.998 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:49.984+0000 7f60d941d700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60d41575e0 con 0x7f60d406e850 2022-06-27T11:58:50.001 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:49.984+0000 7f60d941d700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60d406f5e0 con 0x7f60d40c34a0 2022-06-27T11:58:50.001 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:49.984+0000 7f60d2ffd700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f60d406e850 0x7f60d41570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:50.001 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:49.984+0000 7f60d2ffd700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f60d406e850 0x7f60d41570a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45826/0 (socket says 172.21.15.195:45826) 2022-06-27T11:58:50.001 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:49.984+0000 7f60d2ffd700 1 -- 172.21.15.195:0/1823041407 learned_addr learned my addr 172.21.15.195:0/1823041407 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:58:50.002 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:49.984+0000 7f60d37fe700 1 --2- 172.21.15.195:0/1823041407 >> v2:172.21.15.202:3300/0 conn(0x7f60d40c34a0 0x7f60d40c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:50.002 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:49.985+0000 7f60d2ffd700 1 -- 172.21.15.195:0/1823041407 >> v2:172.21.15.202:3300/0 conn(0x7f60d40c34a0 msgr2=0x7f60d40c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:58:50.003 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:49.985+0000 7f60d2ffd700 1 --2- 172.21.15.195:0/1823041407 >> v2:172.21.15.202:3300/0 conn(0x7f60d40c34a0 0x7f60d40c2920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:50.003 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:49.985+0000 7f60d2ffd700 1 -- 172.21.15.195:0/1823041407 >> v1:172.21.15.202:6789/0 conn(0x7f60d406fef0 legacy=0x7f60d4069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:50.003 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:49.985+0000 7f60d2ffd700 1 -- 172.21.15.195:0/1823041407 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60d406f8c0 con 0x7f60d406e850 2022-06-27T11:58:50.003 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:49.985+0000 7f60d2ffd700 1 --2- 172.21.15.195:0/1823041407 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f60d406e850 0x7f60d41570a0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f60d40c1420 tx=0x7f60c4009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c369ed2fb1136c4 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:50.004 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:49.985+0000 7f60d1ffb700 1 -- 172.21.15.195:0/1823041407 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60c4011630 con 0x7f60d406e850 2022-06-27T11:58:50.004 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:49.985+0000 7f60d1ffb700 1 -- 172.21.15.195:0/1823041407 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60c4011790 con 0x7f60d406e850 2022-06-27T11:58:50.004 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:49.985+0000 7f60d1ffb700 1 -- 172.21.15.195:0/1823041407 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60c4011b30 con 0x7f60d406e850 2022-06-27T11:58:50.004 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:49.985+0000 7f60d941d700 1 -- 172.21.15.195:0/1823041407 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f60d406e850 msgr2=0x7f60d41570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:50.005 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:49.985+0000 7f60d941d700 1 --2- 172.21.15.195:0/1823041407 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f60d406e850 0x7f60d41570a0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f60d40c1420 tx=0x7f60c4009120 comp rx=0 tx=0).stop 2022-06-27T11:58:50.005 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:49.985+0000 7f60d941d700 1 -- 172.21.15.195:0/1823041407 shutdown_connections 2022-06-27T11:58:50.005 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:49.985+0000 7f60d941d700 1 --2- 172.21.15.195:0/1823041407 >> v2:172.21.15.202:3300/0 conn(0x7f60d40c34a0 0x7f60d40c2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:50.005 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:49.985+0000 7f60d941d700 1 --2- 172.21.15.195:0/1823041407 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f60d406e850 0x7f60d41570a0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:50.006 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:49.985+0000 7f60d941d700 1 -- 172.21.15.195:0/1823041407 >> 172.21.15.195:0/1823041407 conn(0x7f60d40c6350 msgr2=0x7f60d40c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:58:50.006 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:49.986+0000 7f60d941d700 1 -- 172.21.15.195:0/1823041407 shutdown_connections 2022-06-27T11:58:50.006 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:49.986+0000 7f60d941d700 1 -- 172.21.15.195:0/1823041407 wait complete. 2022-06-27T11:58:50.007 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:49.987+0000 7f60d941d700 1 Processor -- start 2022-06-27T11:58:50.007 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:49.987+0000 7f60d941d700 1 -- start start 2022-06-27T11:58:50.007 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:49.988+0000 7f60d941d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f60d406e850 0x7f60d41501e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:50.007 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:49.988+0000 7f60d941d700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f60d40c34a0 0x7f60d4151090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:50.008 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:49.988+0000 7f60d941d700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f60d416a920 0x7f60d41542e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:50.008 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:49.988+0000 7f60d941d700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60d416a490 con 0x7f60d406e850 2022-06-27T11:58:50.008 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:49.988+0000 7f60d941d700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60d4150de0 con 0x7f60d40c34a0 2022-06-27T11:58:50.008 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:49.988+0000 7f60d941d700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60d41515d0 con 0x7f60d416a920 2022-06-27T11:58:50.009 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:49.988+0000 7f60d2ffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f60d406e850 0x7f60d41501e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:50.010 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:49.988+0000 7f60d2ffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f60d406e850 0x7f60d41501e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45832/0 (socket says 172.21.15.195:45832) 2022-06-27T11:58:50.010 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:49.988+0000 7f60d2ffd700 1 -- 172.21.15.195:0/381156393 learned_addr learned my addr 172.21.15.195:0/381156393 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:58:50.010 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:49.988+0000 7f60d27fc700 1 --2- 172.21.15.195:0/381156393 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f60d40c34a0 0x7f60d4151090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:50.011 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:49.988+0000 7f60d37fe700 1 --2- 172.21.15.195:0/381156393 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f60d416a920 0x7f60d41542e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:50.011 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:49.988+0000 7f60d2ffd700 1 -- 172.21.15.195:0/381156393 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f60d416a920 msgr2=0x7f60d41542e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:50.011 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:49.988+0000 7f60d2ffd700 1 --2- 172.21.15.195:0/381156393 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f60d416a920 0x7f60d41542e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:50.011 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:49.988+0000 7f60d2ffd700 1 -- 172.21.15.195:0/381156393 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f60d40c34a0 msgr2=0x7f60d4151090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:50.012 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:49.988+0000 7f60d2ffd700 1 --2- 172.21.15.195:0/381156393 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f60d40c34a0 0x7f60d4151090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:50.012 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:49.988+0000 7f60d2ffd700 1 -- 172.21.15.195:0/381156393 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60c40093a0 con 0x7f60d406e850 2022-06-27T11:58:50.012 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:49.988+0000 7f60d2ffd700 1 --2- 172.21.15.195:0/381156393 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f60d406e850 0x7f60d41501e0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f60c4011c30 tx=0x7f60c4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:50.012 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:49.988+0000 7f60baffd700 1 -- 172.21.15.195:0/381156393 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60c4011630 con 0x7f60d406e850 2022-06-27T11:58:50.021 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:49.988+0000 7f60baffd700 1 -- 172.21.15.195:0/381156393 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60c4003730 con 0x7f60d406e850 2022-06-27T11:58:50.021 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:49.988+0000 7f60baffd700 1 -- 172.21.15.195:0/381156393 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60c4003af0 con 0x7f60d406e850 2022-06-27T11:58:50.021 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:49.989+0000 7f60d941d700 1 -- 172.21.15.195:0/381156393 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60d40c3000 con 0x7f60d406e850 2022-06-27T11:58:50.021 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:49.989+0000 7f60d941d700 1 -- 172.21.15.195:0/381156393 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60d4150b00 con 0x7f60d406e850 2022-06-27T11:58:50.022 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:49.990+0000 7f60d941d700 1 -- 172.21.15.195:0/381156393 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60d40722c0 con 0x7f60d406e850 2022-06-27T11:58:50.022 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:49.990+0000 7f60baffd700 1 -- 172.21.15.195:0/381156393 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f60c402b030 con 0x7f60d406e850 2022-06-27T11:58:50.022 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:49.990+0000 7f60baffd700 1 --2- 172.21.15.195:0/381156393 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f60b4063d30 0x7f60b40661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:50.023 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:49.990+0000 7f60baffd700 1 -- 172.21.15.195:0/381156393 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6632+0+0 (secure 0 0 0) 0x7f60c4081f30 con 0x7f60d406e850 2022-06-27T11:58:50.023 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:49.991+0000 7f60d27fc700 1 --2- 172.21.15.195:0/381156393 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f60b4063d30 0x7f60b40661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:50.023 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:49.991+0000 7f60d27fc700 1 --2- 172.21.15.195:0/381156393 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f60b4063d30 0x7f60b40661e0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f60d4159af0 tx=0x7f60c8009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:50.024 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:49.995+0000 7f60baffd700 1 -- 172.21.15.195:0/381156393 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f60c40b1c90 con 0x7f60d406e850 2022-06-27T11:58:50.216 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:50.216+0000 7f60d941d700 1 -- 172.21.15.195:0/381156393 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f60d406f240 con 0x7f60d406e850 2022-06-27T11:58:50.217 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:50.217+0000 7f60baffd700 1 -- 172.21.15.195:0/381156393 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+6128 (secure 0 0 0) 0x7f60c4029070 con 0x7f60d406e850 2022-06-27T11:58:50.217 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:58:50.218 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":5192,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/2651813808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":2651813808},{"type":"v1","addr":"172.21.15.202:6833","nonce":2651813808}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:58:32.073997+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5392},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5392":{"gid":5392,"name":"d","rank":0,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.202:6835/2089155519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2089155519},{"type":"v1","addr":"172.21.15.202:6835","nonce":2089155519}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:58:50.220 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:50.219+0000 7f60d941d700 1 -- 172.21.15.195:0/381156393 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f60b4063d30 msgr2=0x7f60b40661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:50.220 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:50.219+0000 7f60d941d700 1 --2- 172.21.15.195:0/381156393 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f60b4063d30 0x7f60b40661e0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f60d4159af0 tx=0x7f60c8009040 comp rx=0 tx=0).stop 2022-06-27T11:58:50.221 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:50.219+0000 7f60d941d700 1 -- 172.21.15.195:0/381156393 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f60d406e850 msgr2=0x7f60d41501e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:50.221 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:50.219+0000 7f60d941d700 1 --2- 172.21.15.195:0/381156393 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f60d406e850 0x7f60d41501e0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f60c4011c30 tx=0x7f60c4017000 comp rx=0 tx=0).stop 2022-06-27T11:58:50.221 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:50.219+0000 7f60d941d700 1 -- 172.21.15.195:0/381156393 shutdown_connections 2022-06-27T11:58:50.222 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:50.219+0000 7f60d941d700 1 --2- 172.21.15.195:0/381156393 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f60b4063d30 0x7f60b40661e0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:50.222 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:50.219+0000 7f60d941d700 1 --2- 172.21.15.195:0/381156393 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f60d406e850 0x7f60d41501e0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:50.223 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:50.219+0000 7f60d941d700 1 --2- 172.21.15.195:0/381156393 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f60d416a920 0x7f60d41542e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:50.223 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:50.219+0000 7f60d941d700 1 --2- 172.21.15.195:0/381156393 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f60d40c34a0 0x7f60d4151090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:50.223 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:50.219+0000 7f60d941d700 1 -- 172.21.15.195:0/381156393 >> 172.21.15.195:0/381156393 conn(0x7f60d40c6350 msgr2=0x7f60d40c76e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:58:50.223 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:50.219+0000 7f60d941d700 1 -- 172.21.15.195:0/381156393 shutdown_connections 2022-06-27T11:58:50.223 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:50.219+0000 7f60d941d700 1 -- 172.21.15.195:0/381156393 wait complete. 2022-06-27T11:58:50.224 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 42 2022-06-27T11:58:50.234 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-06-27T11:58:52.238 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:58:52.407 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.373+0000 7f1d6f8c8700 1 Processor -- start 2022-06-27T11:58:52.408 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.374+0000 7f1d6f8c8700 1 -- start start 2022-06-27T11:58:52.409 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.374+0000 7f1d6f8c8700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1d6806e850 0x7f1d681570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:52.409 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.374+0000 7f1d6f8c8700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1d680c34a0 0x7f1d680c2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:52.429 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.374+0000 7f1d6f8c8700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d6806fba0 con 0x7f1d6806fef0 2022-06-27T11:58:52.430 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.374+0000 7f1d6f8c8700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d681575e0 con 0x7f1d6806e850 2022-06-27T11:58:52.430 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.374+0000 7f1d6f8c8700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d6806f5e0 con 0x7f1d680c34a0 2022-06-27T11:58:52.430 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.375+0000 7f1d6d664700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1d6806e850 0x7f1d681570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:52.431 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.375+0000 7f1d6ce63700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f1d6806fef0 0x7f1d68069de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:52030/0 (socket says 172.21.15.195:52030) 2022-06-27T11:58:52.431 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.375+0000 7f1d6d664700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1d6806e850 0x7f1d681570a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45842/0 (socket says 172.21.15.195:45842) 2022-06-27T11:58:52.431 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.375+0000 7f1d6ce63700 1 -- 172.21.15.195:0/901093595 learned_addr learned my addr 172.21.15.195:0/901093595 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:58:52.431 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.375+0000 7f1d6de65700 1 --2- 172.21.15.195:0/901093595 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1d680c34a0 0x7f1d680c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:52.432 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.375+0000 7f1d6d664700 1 -- 172.21.15.195:0/901093595 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1d680c34a0 msgr2=0x7f1d680c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:58:52.432 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.375+0000 7f1d6d664700 1 --2- 172.21.15.195:0/901093595 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1d680c34a0 0x7f1d680c2920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:52.433 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.375+0000 7f1d6d664700 1 -- 172.21.15.195:0/901093595 >> v1:172.21.15.195:6789/0 conn(0x7f1d6806fef0 legacy=0x7f1d68069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:52.433 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.375+0000 7f1d6d664700 1 -- 172.21.15.195:0/901093595 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d6806f8c0 con 0x7f1d6806e850 2022-06-27T11:58:52.433 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.375+0000 7f1d5bfff700 1 -- 172.21.15.195:0/901093595 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1260234001 0 0) 0x7f1d6806fba0 con 0x7f1d6806fef0 2022-06-27T11:58:52.434 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.375+0000 7f1d6d664700 1 --2- 172.21.15.195:0/901093595 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1d6806e850 0x7f1d681570a0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f1d5c0097a0 tx=0x7f1d5c009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=609dc522a97d4ed2 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:52.434 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.375+0000 7f1d5bfff700 1 -- 172.21.15.195:0/901093595 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d5c011630 con 0x7f1d6806e850 2022-06-27T11:58:52.434 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.375+0000 7f1d5bfff700 1 -- 172.21.15.195:0/901093595 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d5c011790 con 0x7f1d6806e850 2022-06-27T11:58:52.435 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.375+0000 7f1d5bfff700 1 -- 172.21.15.195:0/901093595 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d5c011b30 con 0x7f1d6806e850 2022-06-27T11:58:52.435 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.376+0000 7f1d6f8c8700 1 -- 172.21.15.195:0/901093595 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1d6806e850 msgr2=0x7f1d681570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:52.436 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.376+0000 7f1d6f8c8700 1 --2- 172.21.15.195:0/901093595 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1d6806e850 0x7f1d681570a0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f1d5c0097a0 tx=0x7f1d5c009060 comp rx=0 tx=0).stop 2022-06-27T11:58:52.436 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.376+0000 7f1d6f8c8700 1 -- 172.21.15.195:0/901093595 shutdown_connections 2022-06-27T11:58:52.437 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.376+0000 7f1d6f8c8700 1 --2- 172.21.15.195:0/901093595 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1d680c34a0 0x7f1d680c2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:52.437 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.376+0000 7f1d6f8c8700 1 --2- 172.21.15.195:0/901093595 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1d6806e850 0x7f1d681570a0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:52.438 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.376+0000 7f1d6f8c8700 1 -- 172.21.15.195:0/901093595 >> 172.21.15.195:0/901093595 conn(0x7f1d680c6350 msgr2=0x7f1d680c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:58:52.438 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.376+0000 7f1d6f8c8700 1 -- 172.21.15.195:0/901093595 shutdown_connections 2022-06-27T11:58:52.439 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.376+0000 7f1d6f8c8700 1 -- 172.21.15.195:0/901093595 wait complete. 2022-06-27T11:58:52.439 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.377+0000 7f1d6f8c8700 1 Processor -- start 2022-06-27T11:58:52.440 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.377+0000 7f1d6f8c8700 1 -- start start 2022-06-27T11:58:52.440 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.378+0000 7f1d6f8c8700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1d6806e850 0x7f1d680b8110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:52.441 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.378+0000 7f1d6f8c8700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1d680c34a0 0x7f1d680c8380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:52.446 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.378+0000 7f1d6f8c8700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1d68153c40 0x7f1d680b6450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:52.446 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.378+0000 7f1d6f8c8700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d6814ef90 con 0x7f1d680c34a0 2022-06-27T11:58:52.447 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.378+0000 7f1d6f8c8700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d68166100 con 0x7f1d6806e850 2022-06-27T11:58:52.447 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.378+0000 7f1d6f8c8700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d6814f1e0 con 0x7f1d68153c40 2022-06-27T11:58:52.448 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.379+0000 7f1d6ce63700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1d680c34a0 0x7f1d680c8380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:52.448 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.379+0000 7f1d6ce63700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1d680c34a0 0x7f1d680c8380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45850/0 (socket says 172.21.15.195:45850) 2022-06-27T11:58:52.449 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.379+0000 7f1d6ce63700 1 -- 172.21.15.195:0/1287161481 learned_addr learned my addr 172.21.15.195:0/1287161481 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:58:52.449 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.379+0000 7f1d6d664700 1 --2- 172.21.15.195:0/1287161481 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1d6806e850 0x7f1d680b8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:52.449 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.379+0000 7f1d6de65700 1 --2- 172.21.15.195:0/1287161481 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1d68153c40 0x7f1d680b6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:52.451 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.379+0000 7f1d6ce63700 1 -- 172.21.15.195:0/1287161481 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1d68153c40 msgr2=0x7f1d680b6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:52.451 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.379+0000 7f1d6ce63700 1 --2- 172.21.15.195:0/1287161481 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1d68153c40 0x7f1d680b6450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:52.452 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.379+0000 7f1d6ce63700 1 -- 172.21.15.195:0/1287161481 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1d6806e850 msgr2=0x7f1d680b8110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:52.470 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.379+0000 7f1d6ce63700 1 --2- 172.21.15.195:0/1287161481 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1d6806e850 0x7f1d680b8110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:52.470 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.379+0000 7f1d6ce63700 1 -- 172.21.15.195:0/1287161481 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d5c0092e0 con 0x7f1d680c34a0 2022-06-27T11:58:52.471 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.379+0000 7f1d6d664700 1 --2- 172.21.15.195:0/1287161481 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1d6806e850 0x7f1d680b8110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T11:58:52.471 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.379+0000 7f1d6ce63700 1 --2- 172.21.15.195:0/1287161481 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1d680c34a0 0x7f1d680c8380 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f1d6400c490 tx=0x7f1d64007dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:52.472 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.379+0000 7f1d597fa700 1 -- 172.21.15.195:0/1287161481 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d64007430 con 0x7f1d680c34a0 2022-06-27T11:58:52.472 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.379+0000 7f1d597fa700 1 -- 172.21.15.195:0/1287161481 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d64007590 con 0x7f1d680c34a0 2022-06-27T11:58:52.473 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.379+0000 7f1d6f8c8700 1 -- 172.21.15.195:0/1287161481 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d6814f4c0 con 0x7f1d680c34a0 2022-06-27T11:58:52.473 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.379+0000 7f1d597fa700 1 -- 172.21.15.195:0/1287161481 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d64004920 con 0x7f1d680c34a0 2022-06-27T11:58:52.473 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.379+0000 7f1d6f8c8700 1 -- 172.21.15.195:0/1287161481 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d6814fa80 con 0x7f1d680c34a0 2022-06-27T11:58:52.474 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.380+0000 7f1d6f8c8700 1 -- 172.21.15.195:0/1287161481 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d6814fca0 con 0x7f1d680c34a0 2022-06-27T11:58:52.474 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.381+0000 7f1d597fa700 1 -- 172.21.15.195:0/1287161481 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f1d64010030 con 0x7f1d680c34a0 2022-06-27T11:58:52.475 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.381+0000 7f1d597fa700 1 --2- 172.21.15.195:0/1287161481 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1d48063d30 0x7f1d480661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:52.475 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.381+0000 7f1d597fa700 1 -- 172.21.15.195:0/1287161481 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6632+0+0 (secure 0 0 0) 0x7f1d640805a0 con 0x7f1d680c34a0 2022-06-27T11:58:52.475 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.381+0000 7f1d6d664700 1 --2- 172.21.15.195:0/1287161481 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1d48063d30 0x7f1d480661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:52.476 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.381+0000 7f1d6d664700 1 --2- 172.21.15.195:0/1287161481 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1d48063d30 0x7f1d480661e0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f1d5c009260 tx=0x7f1d5c017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:52.476 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.385+0000 7f1d597fa700 1 -- 172.21.15.195:0/1287161481 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f1d6404efd0 con 0x7f1d680c34a0 2022-06-27T11:58:52.613 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.613+0000 7f1d6f8c8700 1 -- 172.21.15.195:0/1287161481 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1d680b6db0 con 0x7f1d680c34a0 2022-06-27T11:58:52.640 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.613+0000 7f1d597fa700 1 -- 172.21.15.195:0/1287161481 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+6128 (secure 0 0 0) 0x7f1d680b6db0 con 0x7f1d680c34a0 2022-06-27T11:58:52.641 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:58:52.641 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":5192,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/2651813808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":2651813808},{"type":"v1","addr":"172.21.15.202:6833","nonce":2651813808}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:58:32.073997+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5392},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5392":{"gid":5392,"name":"d","rank":0,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.202:6835/2089155519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2089155519},{"type":"v1","addr":"172.21.15.202:6835","nonce":2089155519}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:58:52.643 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.615+0000 7f1d6f8c8700 1 -- 172.21.15.195:0/1287161481 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1d48063d30 msgr2=0x7f1d480661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:52.644 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.615+0000 7f1d6f8c8700 1 --2- 172.21.15.195:0/1287161481 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1d48063d30 0x7f1d480661e0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f1d5c009260 tx=0x7f1d5c017000 comp rx=0 tx=0).stop 2022-06-27T11:58:52.644 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.615+0000 7f1d6f8c8700 1 -- 172.21.15.195:0/1287161481 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1d680c34a0 msgr2=0x7f1d680c8380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:52.644 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.615+0000 7f1d6f8c8700 1 --2- 172.21.15.195:0/1287161481 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1d680c34a0 0x7f1d680c8380 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f1d6400c490 tx=0x7f1d64007dd0 comp rx=0 tx=0).stop 2022-06-27T11:58:52.644 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.615+0000 7f1d6f8c8700 1 -- 172.21.15.195:0/1287161481 shutdown_connections 2022-06-27T11:58:52.645 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.615+0000 7f1d6f8c8700 1 --2- 172.21.15.195:0/1287161481 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1d48063d30 0x7f1d480661e0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:52.645 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.615+0000 7f1d6f8c8700 1 --2- 172.21.15.195:0/1287161481 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1d68153c40 0x7f1d680b6450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:52.645 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.615+0000 7f1d6f8c8700 1 --2- 172.21.15.195:0/1287161481 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1d6806e850 0x7f1d680b8110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:52.645 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.615+0000 7f1d6f8c8700 1 --2- 172.21.15.195:0/1287161481 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1d680c34a0 0x7f1d680c8380 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:52.646 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.615+0000 7f1d6f8c8700 1 -- 172.21.15.195:0/1287161481 >> 172.21.15.195:0/1287161481 conn(0x7f1d680c6350 msgr2=0x7f1d68068a50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:58:52.646 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.616+0000 7f1d6f8c8700 1 -- 172.21.15.195:0/1287161481 shutdown_connections 2022-06-27T11:58:52.646 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:52.616+0000 7f1d6f8c8700 1 -- 172.21.15.195:0/1287161481 wait complete. 2022-06-27T11:58:52.647 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 42 2022-06-27T11:58:52.647 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-06-27T11:58:54.651 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:58:54.784 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:54.784+0000 7fc4f01f5700 1 Processor -- start 2022-06-27T11:58:54.853 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:54.784+0000 7fc4f01f5700 1 -- start start 2022-06-27T11:58:54.882 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:54.785+0000 7fc4f01f5700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fc4e8150950 0x7fc4e8156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:54.882 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:54.785+0000 7fc4f01f5700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fc4e806f620 0x7fc4e806d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:54.883 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:54.785+0000 7fc4f01f5700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4e80c76b0 con 0x7fc4e8150580 2022-06-27T11:58:54.883 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:54.785+0000 7fc4f01f5700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4e8157430 con 0x7fc4e8150950 2022-06-27T11:58:54.883 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:54.785+0000 7fc4f01f5700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4e80c8250 con 0x7fc4e806f620 2022-06-27T11:58:54.883 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:54.785+0000 7fc4ed790700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fc4e8150950 0x7fc4e8156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:54.883 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:54.785+0000 7fc4ed790700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fc4e8150950 0x7fc4e8156ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45858/0 (socket says 172.21.15.195:45858) 2022-06-27T11:58:54.884 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:54.785+0000 7fc4ed790700 1 -- 172.21.15.195:0/1030739669 learned_addr learned my addr 172.21.15.195:0/1030739669 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:58:54.884 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:54.785+0000 7fc4ee792700 1 --2- 172.21.15.195:0/1030739669 >> v2:172.21.15.202:3300/0 conn(0x7fc4e806f620 0x7fc4e806d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:54.884 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:54.785+0000 7fc4ed790700 1 -- 172.21.15.195:0/1030739669 >> v2:172.21.15.202:3300/0 conn(0x7fc4e806f620 msgr2=0x7fc4e806d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:58:54.884 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:54.785+0000 7fc4ed790700 1 --2- 172.21.15.195:0/1030739669 >> v2:172.21.15.202:3300/0 conn(0x7fc4e806f620 0x7fc4e806d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:54.885 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:54.785+0000 7fc4ed790700 1 -- 172.21.15.195:0/1030739669 >> v1:172.21.15.202:6789/0 conn(0x7fc4e8150580 legacy=0x7fc4e80c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:58:54.885 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:54.785+0000 7fc4ed790700 1 -- 172.21.15.195:0/1030739669 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4e80bad70 con 0x7fc4e8150950 2022-06-27T11:58:54.885 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:54.785+0000 7fc4ee792700 1 --2- 172.21.15.195:0/1030739669 >> v2:172.21.15.202:3300/0 conn(0x7fc4e806f620 0x7fc4e806d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T11:58:54.885 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:54.785+0000 7fc4ed790700 1 --2- 172.21.15.195:0/1030739669 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc4e8150950 0x7fc4e8156ef0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fc4e80c4f70 tx=0x7fc4d8009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30051ef10050f938 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:54.885 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:54.786+0000 7fc4ecf8f700 1 -- 172.21.15.195:0/1030739669 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d8004030 con 0x7fc4e8150950 2022-06-27T11:58:54.886 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:54.786+0000 7fc4ecf8f700 1 -- 172.21.15.195:0/1030739669 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4d8004190 con 0x7fc4e8150950 2022-06-27T11:58:54.886 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:54.786+0000 7fc4ecf8f700 1 -- 172.21.15.195:0/1030739669 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d8011630 con 0x7fc4e8150950 2022-06-27T11:58:54.886 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:54.786+0000 7fc4f01f5700 1 -- 172.21.15.195:0/1030739669 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc4e8150950 msgr2=0x7fc4e8156ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:54.886 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:54.786+0000 7fc4f01f5700 1 --2- 172.21.15.195:0/1030739669 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc4e8150950 0x7fc4e8156ef0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fc4e80c4f70 tx=0x7fc4d8009120 comp rx=0 tx=0).stop 2022-06-27T11:58:54.887 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:54.786+0000 7fc4f01f5700 1 -- 172.21.15.195:0/1030739669 shutdown_connections 2022-06-27T11:58:54.887 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:54.786+0000 7fc4f01f5700 1 --2- 172.21.15.195:0/1030739669 >> v2:172.21.15.202:3300/0 conn(0x7fc4e806f620 0x7fc4e806d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:54.887 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:54.786+0000 7fc4f01f5700 1 --2- 172.21.15.195:0/1030739669 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc4e8150950 0x7fc4e8156ef0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:54.887 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:54.786+0000 7fc4f01f5700 1 -- 172.21.15.195:0/1030739669 >> 172.21.15.195:0/1030739669 conn(0x7fc4e8063af0 msgr2=0x7fc4e8063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:58:54.888 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:54.786+0000 7fc4f01f5700 1 -- 172.21.15.195:0/1030739669 shutdown_connections 2022-06-27T11:58:54.888 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:54.786+0000 7fc4f01f5700 1 -- 172.21.15.195:0/1030739669 wait complete. 2022-06-27T11:58:54.888 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:54.787+0000 7fc4f01f5700 1 Processor -- start 2022-06-27T11:58:54.888 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:54.787+0000 7fc4f01f5700 1 -- start start 2022-06-27T11:58:54.888 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:54.788+0000 7fc4f01f5700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc4e806f620 0x7fc4e814b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:54.889 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:54.788+0000 7fc4f01f5700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc4e8150580 0x7fc4e80c33d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:54.889 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:54.788+0000 7fc4f01f5700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc4e8150950 0x7fc4e80c3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:54.889 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:54.788+0000 7fc4f01f5700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4e81562f0 con 0x7fc4e8150580 2022-06-27T11:58:54.889 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:54.788+0000 7fc4f01f5700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4e80c2b30 con 0x7fc4e8150950 2022-06-27T11:58:54.889 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:54.788+0000 7fc4f01f5700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4e8166e50 con 0x7fc4e806f620 2022-06-27T11:58:54.890 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:54.788+0000 7fc4ed790700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc4e8150580 0x7fc4e80c33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:54.890 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:54.788+0000 7fc4ed790700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc4e8150580 0x7fc4e80c33d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45864/0 (socket says 172.21.15.195:45864) 2022-06-27T11:58:54.890 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:54.788+0000 7fc4ed790700 1 -- 172.21.15.195:0/1870625156 learned_addr learned my addr 172.21.15.195:0/1870625156 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:58:54.890 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:54.788+0000 7fc4ee792700 1 --2- 172.21.15.195:0/1870625156 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc4e8150950 0x7fc4e80c3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:54.891 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:54.788+0000 7fc4edf91700 1 --2- 172.21.15.195:0/1870625156 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc4e806f620 0x7fc4e814b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:54.891 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:54.788+0000 7fc4ed790700 1 -- 172.21.15.195:0/1870625156 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc4e806f620 msgr2=0x7fc4e814b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:54.891 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:54.788+0000 7fc4ed790700 1 --2- 172.21.15.195:0/1870625156 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc4e806f620 0x7fc4e814b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:54.902 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:54.788+0000 7fc4ed790700 1 -- 172.21.15.195:0/1870625156 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc4e8150950 msgr2=0x7fc4e80c3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:54.903 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:54.788+0000 7fc4ed790700 1 --2- 172.21.15.195:0/1870625156 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc4e8150950 0x7fc4e80c3910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:54.903 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:54.788+0000 7fc4ed790700 1 -- 172.21.15.195:0/1870625156 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4d80093a0 con 0x7fc4e8150580 2022-06-27T11:58:54.904 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:54.788+0000 7fc4edf91700 1 --2- 172.21.15.195:0/1870625156 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc4e806f620 0x7fc4e814b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T11:58:54.904 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:54.789+0000 7fc4ed790700 1 --2- 172.21.15.195:0/1870625156 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc4e8150580 0x7fc4e80c33d0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fc4d80046e0 tx=0x7fc4d8016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:54.905 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:54.789+0000 7fc4d5ffb700 1 -- 172.21.15.195:0/1870625156 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d8003730 con 0x7fc4e8150580 2022-06-27T11:58:54.905 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:54.789+0000 7fc4d5ffb700 1 -- 172.21.15.195:0/1870625156 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4d80038b0 con 0x7fc4e8150580 2022-06-27T11:58:54.906 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:54.789+0000 7fc4f01f5700 1 -- 172.21.15.195:0/1870625156 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4e81670f0 con 0x7fc4e8150580 2022-06-27T11:58:54.907 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:54.789+0000 7fc4d5ffb700 1 -- 172.21.15.195:0/1870625156 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d8003c70 con 0x7fc4e8150580 2022-06-27T11:58:54.908 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:54.789+0000 7fc4f01f5700 1 -- 172.21.15.195:0/1870625156 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4e80c2200 con 0x7fc4e8150580 2022-06-27T11:58:54.908 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:54.790+0000 7fc4f01f5700 1 -- 172.21.15.195:0/1870625156 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4e80722c0 con 0x7fc4e8150580 2022-06-27T11:58:54.909 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:54.790+0000 7fc4d5ffb700 1 -- 172.21.15.195:0/1870625156 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fc4d802b030 con 0x7fc4e8150580 2022-06-27T11:58:54.909 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:54.791+0000 7fc4d5ffb700 1 --2- 172.21.15.195:0/1870625156 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc4c8063d00 0x7fc4c80661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:54.909 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:54.791+0000 7fc4d5ffb700 1 -- 172.21.15.195:0/1870625156 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6632+0+0 (secure 0 0 0) 0x7fc4d8082570 con 0x7fc4e8150580 2022-06-27T11:58:54.910 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:54.791+0000 7fc4edf91700 1 --2- 172.21.15.195:0/1870625156 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc4c8063d00 0x7fc4c80661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:54.910 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:54.791+0000 7fc4edf91700 1 --2- 172.21.15.195:0/1870625156 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc4c8063d00 0x7fc4c80661b0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fc4e4007fd0 tx=0x7fc4e4009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:54.911 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:54.795+0000 7fc4d5ffb700 1 -- 172.21.15.195:0/1870625156 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fc4d8050ec0 con 0x7fc4e8150580 2022-06-27T11:58:55.024 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.018+0000 7fc4f01f5700 1 -- 172.21.15.195:0/1870625156 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc4e80c2c70 con 0x7fc4e8150580 2022-06-27T11:58:55.025 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.019+0000 7fc4d5ffb700 1 -- 172.21.15.195:0/1870625156 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+5367 (secure 0 0 0) 0x7fc4d80544e0 con 0x7fc4e8150580 2022-06-27T11:58:55.025 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:58:55.026 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:58:54.982841+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5192},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5192":{"gid":5192,"name":"b","rank":0,"incarnation":43,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6833/2651813808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":2651813808},{"type":"v1","addr":"172.21.15.202:6833","nonce":2651813808}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:58:55.028 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.021+0000 7fc4f01f5700 1 -- 172.21.15.195:0/1870625156 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc4c8063d00 msgr2=0x7fc4c80661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:55.030 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.021+0000 7fc4f01f5700 1 --2- 172.21.15.195:0/1870625156 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc4c8063d00 0x7fc4c80661b0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fc4e4007fd0 tx=0x7fc4e4009040 comp rx=0 tx=0).stop 2022-06-27T11:58:55.030 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.021+0000 7fc4f01f5700 1 -- 172.21.15.195:0/1870625156 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc4e8150580 msgr2=0x7fc4e80c33d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:55.030 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.021+0000 7fc4f01f5700 1 --2- 172.21.15.195:0/1870625156 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc4e8150580 0x7fc4e80c33d0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fc4d80046e0 tx=0x7fc4d8016000 comp rx=0 tx=0).stop 2022-06-27T11:58:55.031 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.021+0000 7fc4f01f5700 1 -- 172.21.15.195:0/1870625156 shutdown_connections 2022-06-27T11:58:55.031 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.021+0000 7fc4f01f5700 1 --2- 172.21.15.195:0/1870625156 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc4c8063d00 0x7fc4c80661b0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:55.032 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.021+0000 7fc4f01f5700 1 --2- 172.21.15.195:0/1870625156 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc4e8150950 0x7fc4e80c3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:55.032 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.021+0000 7fc4f01f5700 1 --2- 172.21.15.195:0/1870625156 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc4e806f620 0x7fc4e814b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:55.032 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.021+0000 7fc4f01f5700 1 --2- 172.21.15.195:0/1870625156 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc4e8150580 0x7fc4e80c33d0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:55.033 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.021+0000 7fc4f01f5700 1 -- 172.21.15.195:0/1870625156 >> 172.21.15.195:0/1870625156 conn(0x7fc4e8063af0 msgr2=0x7fc4e814d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:58:55.035 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.021+0000 7fc4f01f5700 1 -- 172.21.15.195:0/1870625156 shutdown_connections 2022-06-27T11:58:55.035 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.021+0000 7fc4f01f5700 1 -- 172.21.15.195:0/1870625156 wait complete. 2022-06-27T11:58:55.036 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 43 2022-06-27T11:58:55.038 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d down, removed from mdsmap 2022-06-27T11:58:55.039 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2022-06-27T11:58:55.039 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:58:55.172 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.160+0000 7fe1cba10700 1 Processor -- start 2022-06-27T11:58:55.173 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.160+0000 7fe1cba10700 1 -- start start 2022-06-27T11:58:55.173 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.161+0000 7fe1cba10700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe1c414e490 0x7fe1c406cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:55.174 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.161+0000 7fe1cba10700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fe1c40c3f50 0x7fe1c4157890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:55.174 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.161+0000 7fe1cba10700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1c40c33b0 con 0x7fe1c414e860 2022-06-27T11:58:55.175 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.161+0000 7fe1cba10700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1c406d520 con 0x7fe1c40c3f50 2022-06-27T11:58:55.175 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.161+0000 7fe1cba10700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c406f220 con 0x7fe1c414e490 2022-06-27T11:58:55.176 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.161+0000 7fe1c9fad700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fe1c40c3f50 0x7fe1c4157890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:55.176 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.161+0000 7fe1c9fad700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fe1c40c3f50 0x7fe1c4157890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45874/0 (socket says 172.21.15.195:45874) 2022-06-27T11:58:55.176 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.161+0000 7fe1c9fad700 1 -- 172.21.15.195:0/2437916273 learned_addr learned my addr 172.21.15.195:0/2437916273 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:58:55.177 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.161+0000 7fe1c97ac700 1 --2- 172.21.15.195:0/2437916273 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe1c414e490 0x7fe1c406cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:55.177 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.161+0000 7fe1c9fad700 1 -- 172.21.15.195:0/2437916273 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe1c414e490 msgr2=0x7fe1c406cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:58:55.177 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.161+0000 7fe1c9fad700 1 --2- 172.21.15.195:0/2437916273 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe1c414e490 0x7fe1c406cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:55.178 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.161+0000 7fe1c9fad700 1 -- 172.21.15.195:0/2437916273 >> v1:172.21.15.202:6789/0 conn(0x7fe1c414e860 legacy=0x7fe1c40c3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:58:55.178 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.161+0000 7fe1c9fad700 1 -- 172.21.15.195:0/2437916273 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1c41542c0 con 0x7fe1c40c3f50 2022-06-27T11:58:55.179 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.162+0000 7fe1c9fad700 1 --2- 172.21.15.195:0/2437916273 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe1c40c3f50 0x7fe1c4157890 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fe1bc009f20 tx=0x7fe1bc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef7ebcd69548214d server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:55.179 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.162+0000 7fe1b3fff700 1 -- 172.21.15.195:0/2437916273 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1bc00e070 con 0x7fe1c40c3f50 2022-06-27T11:58:55.180 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.162+0000 7fe1b3fff700 1 -- 172.21.15.195:0/2437916273 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1bc00ba40 con 0x7fe1c40c3f50 2022-06-27T11:58:55.180 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.162+0000 7fe1b3fff700 1 -- 172.21.15.195:0/2437916273 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1bc0092e0 con 0x7fe1c40c3f50 2022-06-27T11:58:55.180 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.162+0000 7fe1cba10700 1 -- 172.21.15.195:0/2437916273 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe1c40c3f50 msgr2=0x7fe1c4157890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:55.181 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.162+0000 7fe1cba10700 1 --2- 172.21.15.195:0/2437916273 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe1c40c3f50 0x7fe1c4157890 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fe1bc009f20 tx=0x7fe1bc009840 comp rx=0 tx=0).stop 2022-06-27T11:58:55.183 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.162+0000 7fe1cba10700 1 -- 172.21.15.195:0/2437916273 shutdown_connections 2022-06-27T11:58:55.183 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.162+0000 7fe1cba10700 1 --2- 172.21.15.195:0/2437916273 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe1c40c3f50 0x7fe1c4157890 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:55.183 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.162+0000 7fe1cba10700 1 --2- 172.21.15.195:0/2437916273 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe1c414e490 0x7fe1c406cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:55.184 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.162+0000 7fe1cba10700 1 -- 172.21.15.195:0/2437916273 >> 172.21.15.195:0/2437916273 conn(0x7fe1c4063af0 msgr2=0x7fe1c4063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:58:55.193 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.162+0000 7fe1cba10700 1 -- 172.21.15.195:0/2437916273 shutdown_connections 2022-06-27T11:58:55.212 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.162+0000 7fe1cba10700 1 -- 172.21.15.195:0/2437916273 wait complete. 2022-06-27T11:58:55.212 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.164+0000 7fe1cba10700 1 Processor -- start 2022-06-27T11:58:55.213 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.164+0000 7fe1cba10700 1 -- start start 2022-06-27T11:58:55.225 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.165+0000 7fe1cba10700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe1c40c3f50 0x7fe1c414b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:55.226 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.165+0000 7fe1cba10700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe1c414e490 0x7fe1c414ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:55.227 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.165+0000 7fe1cba10700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe1c414e860 0x7fe1c414a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:55.227 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.165+0000 7fe1cba10700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c4164d90 con 0x7fe1c414e860 2022-06-27T11:58:55.228 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.165+0000 7fe1cba10700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c414d630 con 0x7fe1c414e490 2022-06-27T11:58:55.233 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.165+0000 7fe1cba10700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c4164430 con 0x7fe1c40c3f50 2022-06-27T11:58:55.234 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.165+0000 7fe1c9fad700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe1c414e860 0x7fe1c414a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:55.235 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.165+0000 7fe1c9fad700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe1c414e860 0x7fe1c414a3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45880/0 (socket says 172.21.15.195:45880) 2022-06-27T11:58:55.235 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.165+0000 7fe1c9fad700 1 -- 172.21.15.195:0/3596672660 learned_addr learned my addr 172.21.15.195:0/3596672660 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:58:55.236 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.165+0000 7fe1c97ac700 1 --2- 172.21.15.195:0/3596672660 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe1c40c3f50 0x7fe1c414b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:55.237 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.165+0000 7fe1c8fab700 1 --2- 172.21.15.195:0/3596672660 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe1c414e490 0x7fe1c414ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:55.237 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.165+0000 7fe1c9fad700 1 -- 172.21.15.195:0/3596672660 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe1c40c3f50 msgr2=0x7fe1c414b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:55.238 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.165+0000 7fe1c9fad700 1 --2- 172.21.15.195:0/3596672660 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe1c40c3f50 0x7fe1c414b5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:55.238 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.165+0000 7fe1c9fad700 1 -- 172.21.15.195:0/3596672660 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe1c414e490 msgr2=0x7fe1c414ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:55.239 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.165+0000 7fe1c9fad700 1 --2- 172.21.15.195:0/3596672660 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe1c414e490 0x7fe1c414ff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:55.239 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.165+0000 7fe1c9fad700 1 -- 172.21.15.195:0/3596672660 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1bc009ae0 con 0x7fe1c414e860 2022-06-27T11:58:55.240 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.166+0000 7fe1c9fad700 1 --2- 172.21.15.195:0/3596672660 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe1c414e860 0x7fe1c414a3a0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fe1c4070770 tx=0x7fe1bc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:55.241 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.166+0000 7fe1b17fa700 1 -- 172.21.15.195:0/3596672660 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1bc004240 con 0x7fe1c414e860 2022-06-27T11:58:55.241 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.166+0000 7fe1b17fa700 1 -- 172.21.15.195:0/3596672660 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1bc0043c0 con 0x7fe1c414e860 2022-06-27T11:58:55.242 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.166+0000 7fe1cba10700 1 -- 172.21.15.195:0/3596672660 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1c41651b0 con 0x7fe1c414e860 2022-06-27T11:58:55.242 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.166+0000 7fe1b17fa700 1 -- 172.21.15.195:0/3596672660 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1bc004780 con 0x7fe1c414e860 2022-06-27T11:58:55.243 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.166+0000 7fe1cba10700 1 -- 172.21.15.195:0/3596672660 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1c41646d0 con 0x7fe1c414e860 2022-06-27T11:58:55.243 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.167+0000 7fe1cba10700 1 -- 172.21.15.195:0/3596672660 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1c4164ed0 con 0x7fe1c414e860 2022-06-27T11:58:55.243 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.167+0000 7fe1b17fa700 1 -- 172.21.15.195:0/3596672660 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fe1bc02c070 con 0x7fe1c414e860 2022-06-27T11:58:55.244 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.167+0000 7fe1b17fa700 1 --2- 172.21.15.195:0/3596672660 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe1a4063d00 0x7fe1a40661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:55.244 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.168+0000 7fe1b17fa700 1 -- 172.21.15.195:0/3596672660 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6718+0+0 (secure 0 0 0) 0x7fe1bc082160 con 0x7fe1c414e860 2022-06-27T11:58:55.245 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.168+0000 7fe1c97ac700 1 --2- 172.21.15.195:0/3596672660 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe1a4063d00 0x7fe1a40661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:55.245 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.168+0000 7fe1c97ac700 1 --2- 172.21.15.195:0/3596672660 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe1a4063d00 0x7fe1a40661b0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fe1c000aa30 tx=0x7fe1c0008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:55.245 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.172+0000 7fe1b17fa700 1 -- 172.21.15.195:0/3596672660 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fe1c4164ed0 con 0x7fe1c414e860 2022-06-27T11:58:55.393 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.392+0000 7fe1cba10700 1 -- 172.21.15.195:0/3596672660 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe1c4164ed0 con 0x7fe1c414e860 2022-06-27T11:58:55.394 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.393+0000 7fe1b17fa700 1 -- 172.21.15.195:0/3596672660 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+5367 (secure 0 0 0) 0x7fe1c4164ed0 con 0x7fe1c414e860 2022-06-27T11:58:55.394 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:58:55.395 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:58:54.982841+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5192},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5192":{"gid":5192,"name":"b","rank":0,"incarnation":43,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6833/2651813808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":2651813808},{"type":"v1","addr":"172.21.15.202:6833","nonce":2651813808}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:58:55.397 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.395+0000 7fe1cba10700 1 -- 172.21.15.195:0/3596672660 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe1a4063d00 msgr2=0x7fe1a40661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:55.397 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.395+0000 7fe1cba10700 1 --2- 172.21.15.195:0/3596672660 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe1a4063d00 0x7fe1a40661b0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fe1c000aa30 tx=0x7fe1c0008040 comp rx=0 tx=0).stop 2022-06-27T11:58:55.398 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.395+0000 7fe1cba10700 1 -- 172.21.15.195:0/3596672660 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe1c414e860 msgr2=0x7fe1c414a3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:55.398 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.395+0000 7fe1cba10700 1 --2- 172.21.15.195:0/3596672660 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe1c414e860 0x7fe1c414a3a0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fe1c4070770 tx=0x7fe1bc017000 comp rx=0 tx=0).stop 2022-06-27T11:58:55.399 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.395+0000 7fe1cba10700 1 -- 172.21.15.195:0/3596672660 shutdown_connections 2022-06-27T11:58:55.400 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.395+0000 7fe1cba10700 1 --2- 172.21.15.195:0/3596672660 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe1a4063d00 0x7fe1a40661b0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:55.400 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.395+0000 7fe1cba10700 1 --2- 172.21.15.195:0/3596672660 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe1c414e860 0x7fe1c414a3a0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:55.400 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.395+0000 7fe1cba10700 1 --2- 172.21.15.195:0/3596672660 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe1c40c3f50 0x7fe1c414b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:55.401 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.395+0000 7fe1cba10700 1 --2- 172.21.15.195:0/3596672660 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe1c414e490 0x7fe1c414ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:55.401 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.395+0000 7fe1cba10700 1 -- 172.21.15.195:0/3596672660 >> 172.21.15.195:0/3596672660 conn(0x7fe1c4063af0 msgr2=0x7fe1c4154820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:58:55.402 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.395+0000 7fe1cba10700 1 -- 172.21.15.195:0/3596672660 shutdown_connections 2022-06-27T11:58:55.402 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:55.395+0000 7fe1cba10700 1 -- 172.21.15.195:0/3596672660 wait complete. 2022-06-27T11:58:55.404 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 43 2022-06-27T11:58:55.420 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 43, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-06-27T11:47:52.358095+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 43, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5192": { "addr": "172.21.15.202:6833/2651813808", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6832", "nonce": 2651813808, "type": "v2" }, { "addr": "172.21.15.202:6833", "nonce": 2651813808, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5192, "incarnation": 43, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 54, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-06-27T11:58:54.982841+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5192 } } } ], "standbys": [ { "addr": "172.21.15.202:6837/3305006841", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6836", "nonce": 3305006841, "type": "v2" }, { "addr": "172.21.15.202:6837", "nonce": 3305006841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6835/2060709822", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6834", "nonce": 2060709822, "type": "v2" }, { "addr": "172.21.15.195:6835", "nonce": 2060709822, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6838/1598751652", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6836", "nonce": 1598751652, "type": "v2" }, { "addr": "172.21.15.195:6838", "nonce": 1598751652, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4275, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6839/1313800661", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6837", "nonce": 1313800661, "type": "v2" }, { "addr": "172.21.15.195:6839", "nonce": 1313800661, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4278, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-06-27T11:58:57.422 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:58:57.566 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.563+0000 7f9af2072700 1 Processor -- start 2022-06-27T11:58:57.566 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.563+0000 7f9af2072700 1 -- start start 2022-06-27T11:58:57.566 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.563+0000 7f9af2072700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f9aec06eff0 0x7f9aec0c61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:57.567 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.563+0000 7f9af2072700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f9aec067ed0 0x7f9aec0c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:57.567 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.563+0000 7f9af2072700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9aec14d8a0 con 0x7f9aec06f3c0 2022-06-27T11:58:57.567 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.563+0000 7f9af2072700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9aec0c47e0 con 0x7f9aec067ed0 2022-06-27T11:58:57.567 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.563+0000 7f9af2072700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9aec0c3e10 con 0x7f9aec06eff0 2022-06-27T11:58:57.568 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.564+0000 7f9aeaffd700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f9aec067ed0 0x7f9aec0c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:57.568 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.564+0000 7f9aeaffd700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f9aec067ed0 0x7f9aec0c42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45886/0 (socket says 172.21.15.195:45886) 2022-06-27T11:58:57.568 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.564+0000 7f9aeaffd700 1 -- 172.21.15.195:0/3411052933 learned_addr learned my addr 172.21.15.195:0/3411052933 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:58:57.568 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.564+0000 7f9aeb7fe700 1 --2- 172.21.15.195:0/3411052933 >> v2:172.21.15.202:3300/0 conn(0x7f9aec06eff0 0x7f9aec0c61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:57.569 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.564+0000 7f9aeaffd700 1 -- 172.21.15.195:0/3411052933 >> v2:172.21.15.202:3300/0 conn(0x7f9aec06eff0 msgr2=0x7f9aec0c61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:58:57.569 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.564+0000 7f9aeaffd700 1 --2- 172.21.15.195:0/3411052933 >> v2:172.21.15.202:3300/0 conn(0x7f9aec06eff0 0x7f9aec0c61e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:57.569 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.564+0000 7f9aeaffd700 1 -- 172.21.15.195:0/3411052933 >> v1:172.21.15.202:6789/0 conn(0x7f9aec06f3c0 legacy=0x7f9aec0c2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:57.569 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.564+0000 7f9aeaffd700 1 -- 172.21.15.195:0/3411052933 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aec06d970 con 0x7f9aec067ed0 2022-06-27T11:58:57.570 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.564+0000 7f9aeb7fe700 1 --2- 172.21.15.195:0/3411052933 >> v2:172.21.15.202:3300/0 conn(0x7f9aec06eff0 0x7f9aec0c61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T11:58:57.570 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.564+0000 7f9aeaffd700 1 --2- 172.21.15.195:0/3411052933 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9aec067ed0 0x7f9aec0c42a0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f9aec06c7f0 tx=0x7f9ae0009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b2ed2b2715bf549 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:57.570 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.564+0000 7f9aea7fc700 1 -- 172.21.15.195:0/3411052933 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ae000b640 con 0x7f9aec067ed0 2022-06-27T11:58:57.570 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.564+0000 7f9aea7fc700 1 -- 172.21.15.195:0/3411052933 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ae000b7a0 con 0x7f9aec067ed0 2022-06-27T11:58:57.571 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.564+0000 7f9aea7fc700 1 -- 172.21.15.195:0/3411052933 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ae000bb40 con 0x7f9aec067ed0 2022-06-27T11:58:57.571 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.565+0000 7f9af2072700 1 -- 172.21.15.195:0/3411052933 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9aec067ed0 msgr2=0x7f9aec0c42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:57.571 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.565+0000 7f9af2072700 1 --2- 172.21.15.195:0/3411052933 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9aec067ed0 0x7f9aec0c42a0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f9aec06c7f0 tx=0x7f9ae0009120 comp rx=0 tx=0).stop 2022-06-27T11:58:57.571 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.565+0000 7f9af2072700 1 -- 172.21.15.195:0/3411052933 shutdown_connections 2022-06-27T11:58:57.572 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.565+0000 7f9af2072700 1 --2- 172.21.15.195:0/3411052933 >> v2:172.21.15.202:3300/0 conn(0x7f9aec06eff0 0x7f9aec0c61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:57.572 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.565+0000 7f9af2072700 1 --2- 172.21.15.195:0/3411052933 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9aec067ed0 0x7f9aec0c42a0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:57.572 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.565+0000 7f9af2072700 1 -- 172.21.15.195:0/3411052933 >> 172.21.15.195:0/3411052933 conn(0x7f9aec0c9b00 msgr2=0x7f9aec0c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:58:57.572 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.565+0000 7f9af2072700 1 -- 172.21.15.195:0/3411052933 shutdown_connections 2022-06-27T11:58:57.573 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.565+0000 7f9af2072700 1 -- 172.21.15.195:0/3411052933 wait complete. 2022-06-27T11:58:57.573 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.566+0000 7f9af2072700 1 Processor -- start 2022-06-27T11:58:57.573 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.566+0000 7f9af2072700 1 -- start start 2022-06-27T11:58:57.577 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.567+0000 7f9af2072700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9aec067ed0 0x7f9aec14ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:57.577 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.567+0000 7f9af2072700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9aec06eff0 0x7f9aec166210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:57.577 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.567+0000 7f9af2072700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9aec06f3c0 0x7f9aec06de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:57.578 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.567+0000 7f9af2072700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aec0bd4d0 con 0x7f9aec067ed0 2022-06-27T11:58:57.578 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.567+0000 7f9af2072700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aec162cc0 con 0x7f9aec06eff0 2022-06-27T11:58:57.578 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.567+0000 7f9af2072700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9aec150960 con 0x7f9aec06f3c0 2022-06-27T11:58:57.579 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.567+0000 7f9aeb7fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9aec067ed0 0x7f9aec14ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:57.579 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.567+0000 7f9aeb7fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9aec067ed0 0x7f9aec14ffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45890/0 (socket says 172.21.15.195:45890) 2022-06-27T11:58:57.579 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.567+0000 7f9aeb7fe700 1 -- 172.21.15.195:0/1997473671 learned_addr learned my addr 172.21.15.195:0/1997473671 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:58:57.580 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.567+0000 7f9aeaffd700 1 --2- 172.21.15.195:0/1997473671 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9aec06eff0 0x7f9aec166210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:57.580 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.567+0000 7f9aebfff700 1 --2- 172.21.15.195:0/1997473671 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9aec06f3c0 0x7f9aec06de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:57.580 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.567+0000 7f9aeb7fe700 1 -- 172.21.15.195:0/1997473671 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9aec06f3c0 msgr2=0x7f9aec06de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:57.581 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.567+0000 7f9aeb7fe700 1 --2- 172.21.15.195:0/1997473671 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9aec06f3c0 0x7f9aec06de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:57.581 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.567+0000 7f9aeb7fe700 1 -- 172.21.15.195:0/1997473671 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9aec06eff0 msgr2=0x7f9aec166210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:57.585 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.567+0000 7f9aeb7fe700 1 --2- 172.21.15.195:0/1997473671 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9aec06eff0 0x7f9aec166210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:57.585 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.567+0000 7f9aeb7fe700 1 -- 172.21.15.195:0/1997473671 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ae00093a0 con 0x7f9aec067ed0 2022-06-27T11:58:57.586 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.567+0000 7f9aeb7fe700 1 --2- 172.21.15.195:0/1997473671 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9aec067ed0 0x7f9aec14ffd0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f9adc00fe20 tx=0x7f9adc009570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:57.586 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.568+0000 7f9ad37fe700 1 -- 172.21.15.195:0/1997473671 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9adc00fe50 con 0x7f9aec067ed0 2022-06-27T11:58:57.586 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.568+0000 7f9ad37fe700 1 -- 172.21.15.195:0/1997473671 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9adc00dd60 con 0x7f9aec067ed0 2022-06-27T11:58:57.586 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.568+0000 7f9af2072700 1 -- 172.21.15.195:0/1997473671 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9aec1623d0 con 0x7f9aec067ed0 2022-06-27T11:58:57.587 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.568+0000 7f9ad37fe700 1 -- 172.21.15.195:0/1997473671 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9adc004260 con 0x7f9aec067ed0 2022-06-27T11:58:57.587 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.568+0000 7f9af2072700 1 -- 172.21.15.195:0/1997473671 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9aec150680 con 0x7f9aec067ed0 2022-06-27T11:58:57.587 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.569+0000 7f9af2072700 1 -- 172.21.15.195:0/1997473671 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9aec162fa0 con 0x7f9aec067ed0 2022-06-27T11:58:57.588 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.569+0000 7f9ad37fe700 1 -- 172.21.15.195:0/1997473671 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f9adc022070 con 0x7f9aec067ed0 2022-06-27T11:58:57.588 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.569+0000 7f9ad37fe700 1 --2- 172.21.15.195:0/1997473671 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9acc063d00 0x7f9acc0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:57.588 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.569+0000 7f9ad37fe700 1 -- 172.21.15.195:0/1997473671 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6718+0+0 (secure 0 0 0) 0x7f9adc0830f0 con 0x7f9aec067ed0 2022-06-27T11:58:57.588 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.569+0000 7f9aeaffd700 1 --2- 172.21.15.195:0/1997473671 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9acc063d00 0x7f9acc0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:58:57.588 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.570+0000 7f9aeaffd700 1 --2- 172.21.15.195:0/1997473671 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9acc063d00 0x7f9acc0661b0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f9ae0009160 tx=0x7f9ae0017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:58:57.589 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.574+0000 7f9ad37fe700 1 -- 172.21.15.195:0/1997473671 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f9adc051b00 con 0x7f9aec067ed0 2022-06-27T11:58:57.795 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.793+0000 7f9af2072700 1 -- 172.21.15.195:0/1997473671 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9aec0722c0 con 0x7f9aec067ed0 2022-06-27T11:58:57.796 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.794+0000 7f9ad37fe700 1 -- 172.21.15.195:0/1997473671 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+5367 (secure 0 0 0) 0x7f9adc020070 con 0x7f9aec067ed0 2022-06-27T11:58:57.797 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:58:57.797 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:58:54.982841+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5192},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5192":{"gid":5192,"name":"b","rank":0,"incarnation":43,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6833/2651813808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":2651813808},{"type":"v1","addr":"172.21.15.202:6833","nonce":2651813808}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:58:57.801 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.796+0000 7f9af2072700 1 -- 172.21.15.195:0/1997473671 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9acc063d00 msgr2=0x7f9acc0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:57.801 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.796+0000 7f9af2072700 1 --2- 172.21.15.195:0/1997473671 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9acc063d00 0x7f9acc0661b0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f9ae0009160 tx=0x7f9ae0017000 comp rx=0 tx=0).stop 2022-06-27T11:58:57.802 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.796+0000 7f9af2072700 1 -- 172.21.15.195:0/1997473671 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9aec067ed0 msgr2=0x7f9aec14ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:58:57.802 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.796+0000 7f9af2072700 1 --2- 172.21.15.195:0/1997473671 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9aec067ed0 0x7f9aec14ffd0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f9adc00fe20 tx=0x7f9adc009570 comp rx=0 tx=0).stop 2022-06-27T11:58:57.803 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.796+0000 7f9af2072700 1 -- 172.21.15.195:0/1997473671 shutdown_connections 2022-06-27T11:58:57.803 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.796+0000 7f9af2072700 1 --2- 172.21.15.195:0/1997473671 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9acc063d00 0x7f9acc0661b0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:57.804 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.796+0000 7f9af2072700 1 --2- 172.21.15.195:0/1997473671 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9aec067ed0 0x7f9aec14ffd0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:57.809 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.796+0000 7f9af2072700 1 --2- 172.21.15.195:0/1997473671 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9aec06f3c0 0x7f9aec06de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:57.810 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.796+0000 7f9af2072700 1 --2- 172.21.15.195:0/1997473671 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9aec06eff0 0x7f9aec166210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:58:57.810 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.796+0000 7f9af2072700 1 -- 172.21.15.195:0/1997473671 >> 172.21.15.195:0/1997473671 conn(0x7f9aec0c9b00 msgr2=0x7f9aec063c20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:58:57.810 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.796+0000 7f9af2072700 1 -- 172.21.15.195:0/1997473671 shutdown_connections 2022-06-27T11:58:57.810 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:57.796+0000 7f9af2072700 1 -- 172.21.15.195:0/1997473671 wait complete. 2022-06-27T11:58:57.811 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 43 2022-06-27T11:58:57.812 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T11:58:59.252 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T11:58:59.256 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T11:58:59.815 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:58:59.949 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.949+0000 7ff4f9659700 1 Processor -- start 2022-06-27T11:58:59.950 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.949+0000 7ff4f9659700 1 -- start start 2022-06-27T11:58:59.999 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.950+0000 7ff4f9659700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7ff4f4156670 0x7ff4f40c7440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:58:59.999 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.950+0000 7ff4f9659700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff4f4069390 0x7ff4f41502b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:00.000 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.950+0000 7ff4f9659700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4f40c43d0 con 0x7ff4f4067ed0 2022-06-27T11:59:00.000 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.950+0000 7ff4f9659700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4f4068430 con 0x7ff4f4156670 2022-06-27T11:59:00.001 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.950+0000 7ff4f9659700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f4068710 con 0x7ff4f4069390 2022-06-27T11:59:00.001 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.950+0000 7ff4f27fc700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7ff4f4067ed0 0x7ff4f40c4860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:52088/0 (socket says 172.21.15.195:52088) 2022-06-27T11:59:00.006 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.950+0000 7ff4f27fc700 1 -- 172.21.15.195:0/2777431819 learned_addr learned my addr 172.21.15.195:0/2777431819 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:59:00.006 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.950+0000 7ff4f2ffd700 1 --2- 172.21.15.195:0/2777431819 >> v2:172.21.15.202:3300/0 conn(0x7ff4f4156670 0x7ff4f40c7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:00.007 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.950+0000 7ff4f37fe700 1 --2- 172.21.15.195:0/2777431819 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff4f4069390 0x7ff4f41502b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:00.007 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.950+0000 7ff4f1ffb700 1 -- 172.21.15.195:0/2777431819 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1639370092 0 0) 0x7ff4f40c43d0 con 0x7ff4f4067ed0 2022-06-27T11:59:00.007 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.950+0000 7ff4f1ffb700 1 -- 172.21.15.195:0/2777431819 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4e0003430 con 0x7ff4f4067ed0 2022-06-27T11:59:00.008 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.950+0000 7ff4f1ffb700 1 -- 172.21.15.195:0/2777431819 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2239088391 0 0) 0x7ff4e0003430 con 0x7ff4f4067ed0 2022-06-27T11:59:00.008 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.950+0000 7ff4f1ffb700 1 -- 172.21.15.195:0/2777431819 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff4f4069390 msgr2=0x7ff4f41502b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:59:00.009 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.950+0000 7ff4f1ffb700 1 --2- 172.21.15.195:0/2777431819 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff4f4069390 0x7ff4f41502b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:00.009 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.950+0000 7ff4f1ffb700 1 -- 172.21.15.195:0/2777431819 >> v2:172.21.15.202:3300/0 conn(0x7ff4f4156670 msgr2=0x7ff4f40c7440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:59:00.009 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.950+0000 7ff4f1ffb700 1 --2- 172.21.15.195:0/2777431819 >> v2:172.21.15.202:3300/0 conn(0x7ff4f4156670 0x7ff4f40c7440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:00.010 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.951+0000 7ff4f1ffb700 1 -- 172.21.15.195:0/2777431819 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4f40689f0 con 0x7ff4f4067ed0 2022-06-27T11:59:00.010 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.951+0000 7ff4f1ffb700 1 -- 172.21.15.195:0/2777431819 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7ff4e4001470 con 0x7ff4f4067ed0 2022-06-27T11:59:00.017 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.951+0000 7ff4f1ffb700 1 -- 172.21.15.195:0/2777431819 >> v1:172.21.15.195:6789/0 conn(0x7ff4f4067ed0 legacy=0x7ff4f40c4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:00.017 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.951+0000 7ff4f1ffb700 1 --2- 172.21.15.195:0/2777431819 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff4f4069390 0x7ff4e0003920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:00.017 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.951+0000 7ff4f1ffb700 1 --2- 172.21.15.195:0/2777431819 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff4e0003ea0 0x7ff4e00046d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:00.018 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.951+0000 7ff4f1ffb700 1 --2- 172.21.15.195:0/2777431819 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff4e0004c10 0x7ff4e0005080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:00.018 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.951+0000 7ff4f1ffb700 1 -- 172.21.15.195:0/2777431819 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f4068430 con 0x7ff4e0003ea0 2022-06-27T11:59:00.019 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.951+0000 7ff4f1ffb700 1 -- 172.21.15.195:0/2777431819 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f4068710 con 0x7ff4e0004c10 2022-06-27T11:59:00.019 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.951+0000 7ff4f1ffb700 1 -- 172.21.15.195:0/2777431819 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e000b950 con 0x7ff4f4069390 2022-06-27T11:59:00.020 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.951+0000 7ff4f27fc700 1 --2- 172.21.15.195:0/2777431819 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff4e0003ea0 0x7ff4e00046d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:00.020 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.951+0000 7ff4f2ffd700 1 --2- 172.21.15.195:0/2777431819 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff4f4069390 0x7ff4e0003920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:00.020 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.951+0000 7ff4f37fe700 1 --2- 172.21.15.195:0/2777431819 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff4e0004c10 0x7ff4e0005080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:00.021 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.951+0000 7ff4f27fc700 1 -- 172.21.15.195:0/2777431819 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff4f4069390 msgr2=0x7ff4e0003920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:59:00.021 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.951+0000 7ff4f27fc700 1 --2- 172.21.15.195:0/2777431819 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff4f4069390 0x7ff4e0003920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:00.022 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.951+0000 7ff4f27fc700 1 -- 172.21.15.195:0/2777431819 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff4e0004c10 msgr2=0x7ff4e0005080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:59:00.022 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.951+0000 7ff4f27fc700 1 --2- 172.21.15.195:0/2777431819 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff4e0004c10 0x7ff4e0005080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:00.023 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.951+0000 7ff4f27fc700 1 -- 172.21.15.195:0/2777431819 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4e0003430 con 0x7ff4e0003ea0 2022-06-27T11:59:00.023 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.951+0000 7ff4f27fc700 1 --2- 172.21.15.195:0/2777431819 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff4e0003ea0 0x7ff4e00046d0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7ff4e400c210 tx=0x7ff4f4156420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcf75ac332f51609 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:59:00.024 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.951+0000 7ff4f1ffb700 1 -- 172.21.15.195:0/2777431819 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4e400cae0 con 0x7ff4e0003ea0 2022-06-27T11:59:00.024 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.951+0000 7ff4f1ffb700 1 -- 172.21.15.195:0/2777431819 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4e400cc40 con 0x7ff4e0003ea0 2022-06-27T11:59:00.025 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.951+0000 7ff4f1ffb700 1 -- 172.21.15.195:0/2777431819 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4e40070d0 con 0x7ff4e0003ea0 2022-06-27T11:59:00.025 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.952+0000 7ff4f9659700 1 -- 172.21.15.195:0/2777431819 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff4e0003ea0 msgr2=0x7ff4e00046d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:00.025 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.952+0000 7ff4f9659700 1 --2- 172.21.15.195:0/2777431819 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff4e0003ea0 0x7ff4e00046d0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7ff4e400c210 tx=0x7ff4f4156420 comp rx=0 tx=0).stop 2022-06-27T11:59:00.040 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.952+0000 7ff4f37fe700 1 -- 172.21.15.195:0/2777431819 reap_dead start 2022-06-27T11:59:00.040 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.952+0000 7ff4f9659700 1 -- 172.21.15.195:0/2777431819 shutdown_connections 2022-06-27T11:59:00.043 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.952+0000 7ff4f9659700 1 -- 172.21.15.195:0/2777431819 >> 172.21.15.195:0/2777431819 conn(0x7ff4f4069e00 msgr2=0x7ff4f406a200 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:59:00.043 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.952+0000 7ff4f9659700 1 -- 172.21.15.195:0/2777431819 shutdown_connections 2022-06-27T11:59:00.043 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.952+0000 7ff4f9659700 1 -- 172.21.15.195:0/2777431819 wait complete. 2022-06-27T11:59:00.044 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.953+0000 7ff4f9659700 1 Processor -- start 2022-06-27T11:59:00.044 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.953+0000 7ff4f9659700 1 -- start start 2022-06-27T11:59:00.044 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.954+0000 7ff4f9659700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff4e0003ea0 0x7ff4f4152930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:00.044 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.954+0000 7ff4f9659700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff4f4157590 0x7ff4f41685f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:00.045 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.954+0000 7ff4f9659700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff4f4168b30 0x7ff4f4169e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:00.045 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.954+0000 7ff4f9659700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f4168fa0 con 0x7ff4f4168b30 2022-06-27T11:59:00.045 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.954+0000 7ff4f9659700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f4167e30 con 0x7ff4e0003ea0 2022-06-27T11:59:00.046 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.954+0000 7ff4f9659700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f40c3770 con 0x7ff4f4157590 2022-06-27T11:59:00.046 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.954+0000 7ff4f37fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff4f4168b30 0x7ff4f4169e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:00.046 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.954+0000 7ff4f37fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff4f4168b30 0x7ff4f4169e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45914/0 (socket says 172.21.15.195:45914) 2022-06-27T11:59:00.046 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.954+0000 7ff4f37fe700 1 -- 172.21.15.195:0/2488018010 learned_addr learned my addr 172.21.15.195:0/2488018010 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:59:00.047 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.954+0000 7ff4f27fc700 1 --2- 172.21.15.195:0/2488018010 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff4f4157590 0x7ff4f41685f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:00.047 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.954+0000 7ff4f2ffd700 1 --2- 172.21.15.195:0/2488018010 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff4e0003ea0 0x7ff4f4152930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:00.047 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.954+0000 7ff4f37fe700 1 -- 172.21.15.195:0/2488018010 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff4f4157590 msgr2=0x7ff4f41685f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:00.047 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.954+0000 7ff4f37fe700 1 --2- 172.21.15.195:0/2488018010 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff4f4157590 0x7ff4f41685f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:00.048 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.954+0000 7ff4f37fe700 1 -- 172.21.15.195:0/2488018010 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff4e0003ea0 msgr2=0x7ff4f4152930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:00.048 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.954+0000 7ff4f37fe700 1 --2- 172.21.15.195:0/2488018010 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff4e0003ea0 0x7ff4f4152930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:00.048 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.954+0000 7ff4f37fe700 1 -- 172.21.15.195:0/2488018010 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4f40c3a50 con 0x7ff4f4168b30 2022-06-27T11:59:00.048 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.954+0000 7ff4f37fe700 1 --2- 172.21.15.195:0/2488018010 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff4f4168b30 0x7ff4f4169e20 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7ff4f408ad00 tx=0x7ff4ec00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:59:00.049 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.955+0000 7ff4daffd700 1 -- 172.21.15.195:0/2488018010 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4ec00e040 con 0x7ff4f4168b30 2022-06-27T11:59:00.049 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.955+0000 7ff4daffd700 1 -- 172.21.15.195:0/2488018010 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4ec00f040 con 0x7ff4f4168b30 2022-06-27T11:59:00.049 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.955+0000 7ff4f9659700 1 -- 172.21.15.195:0/2488018010 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4f40c3d30 con 0x7ff4f4168b30 2022-06-27T11:59:00.049 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.955+0000 7ff4daffd700 1 -- 172.21.15.195:0/2488018010 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4ec00fd70 con 0x7ff4f4168b30 2022-06-27T11:59:00.050 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.955+0000 7ff4f9659700 1 -- 172.21.15.195:0/2488018010 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4f4158450 con 0x7ff4f4168b30 2022-06-27T11:59:00.050 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.956+0000 7ff4f9659700 1 -- 172.21.15.195:0/2488018010 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4f4168310 con 0x7ff4f4168b30 2022-06-27T11:59:00.050 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.956+0000 7ff4daffd700 1 -- 172.21.15.195:0/2488018010 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7ff4ec025070 con 0x7ff4f4168b30 2022-06-27T11:59:00.050 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.956+0000 7ff4daffd700 1 --2- 172.21.15.195:0/2488018010 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff4d4063d30 0x7ff4d40661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:00.051 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.956+0000 7ff4daffd700 1 -- 172.21.15.195:0/2488018010 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6718+0+0 (secure 0 0 0) 0x7ff4ec080ea0 con 0x7ff4f4168b30 2022-06-27T11:59:00.051 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.956+0000 7ff4f2ffd700 1 --2- 172.21.15.195:0/2488018010 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff4d4063d30 0x7ff4d40661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:00.051 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.957+0000 7ff4f2ffd700 1 --2- 172.21.15.195:0/2488018010 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff4d4063d30 0x7ff4d40661e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff4e8006dc0 tx=0x7ff4e8009060 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:59:00.051 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:58:59.961+0000 7ff4daffd700 1 -- 172.21.15.195:0/2488018010 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7ff4f4168310 con 0x7ff4f4168b30 2022-06-27T11:59:00.207 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:00.203+0000 7ff4f9659700 1 -- 172.21.15.195:0/2488018010 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff4f4168310 con 0x7ff4f4168b30 2022-06-27T11:59:00.208 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:00.204+0000 7ff4daffd700 1 -- 172.21.15.195:0/2488018010 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+5367 (secure 0 0 0) 0x7ff4f4168310 con 0x7ff4f4168b30 2022-06-27T11:59:00.209 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:59:00.209 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:58:54.982841+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5192},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5192":{"gid":5192,"name":"b","rank":0,"incarnation":43,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6833/2651813808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":2651813808},{"type":"v1","addr":"172.21.15.202:6833","nonce":2651813808}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:59:00.213 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:00.206+0000 7ff4f9659700 1 -- 172.21.15.195:0/2488018010 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff4d4063d30 msgr2=0x7ff4d40661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:00.214 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:00.206+0000 7ff4f9659700 1 --2- 172.21.15.195:0/2488018010 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff4d4063d30 0x7ff4d40661e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff4e8006dc0 tx=0x7ff4e8009060 comp rx=0 tx=0).stop 2022-06-27T11:59:00.215 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:00.206+0000 7ff4f9659700 1 -- 172.21.15.195:0/2488018010 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff4f4168b30 msgr2=0x7ff4f4169e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:00.215 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:00.206+0000 7ff4f9659700 1 --2- 172.21.15.195:0/2488018010 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff4f4168b30 0x7ff4f4169e20 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7ff4f408ad00 tx=0x7ff4ec00b040 comp rx=0 tx=0).stop 2022-06-27T11:59:00.215 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:00.206+0000 7ff4f9659700 1 -- 172.21.15.195:0/2488018010 shutdown_connections 2022-06-27T11:59:00.216 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:00.206+0000 7ff4f9659700 1 --2- 172.21.15.195:0/2488018010 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff4d4063d30 0x7ff4d40661e0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:00.221 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:00.206+0000 7ff4f9659700 1 --2- 172.21.15.195:0/2488018010 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff4f4168b30 0x7ff4f4169e20 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:00.221 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:00.206+0000 7ff4f9659700 1 --2- 172.21.15.195:0/2488018010 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff4e0003ea0 0x7ff4f4152930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:00.221 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:00.206+0000 7ff4f9659700 1 --2- 172.21.15.195:0/2488018010 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff4f4157590 0x7ff4f41685f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:00.222 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:00.206+0000 7ff4f9659700 1 -- 172.21.15.195:0/2488018010 >> 172.21.15.195:0/2488018010 conn(0x7ff4f4069e00 msgr2=0x7ff4f40cb2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:59:00.223 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:00.206+0000 7ff4f9659700 1 -- 172.21.15.195:0/2488018010 shutdown_connections 2022-06-27T11:59:00.223 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:00.206+0000 7ff4f9659700 1 -- 172.21.15.195:0/2488018010 wait complete. 2022-06-27T11:59:00.223 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 43 2022-06-27T11:59:00.226 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T11:59:02.228 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:59:02.361 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.361+0000 7f4c227bc700 1 Processor -- start 2022-06-27T11:59:02.362 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.361+0000 7f4c227bc700 1 -- start start 2022-06-27T11:59:02.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.362+0000 7f4c227bc700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f4c1c06fef0 0x7f4c1c1570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:02.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.362+0000 7f4c227bc700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f4c1c0c34a0 0x7f4c1c0c2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:02.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.362+0000 7f4c227bc700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c1c06fba0 con 0x7f4c1c06e850 2022-06-27T11:59:02.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.362+0000 7f4c227bc700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c1c1575e0 con 0x7f4c1c0c34a0 2022-06-27T11:59:02.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.362+0000 7f4c227bc700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c1c06f5e0 con 0x7f4c1c06fef0 2022-06-27T11:59:02.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.362+0000 7f4c20d59700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f4c1c0c34a0 0x7f4c1c0c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:02.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.362+0000 7f4c20d59700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f4c1c0c34a0 0x7f4c1c0c2920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45922/0 (socket says 172.21.15.195:45922) 2022-06-27T11:59:02.372 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.362+0000 7f4c20d59700 1 -- 172.21.15.195:0/219734337 learned_addr learned my addr 172.21.15.195:0/219734337 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:59:02.372 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.363+0000 7f4c1b7fe700 1 --2- 172.21.15.195:0/219734337 >> v2:172.21.15.202:3300/0 conn(0x7f4c1c06fef0 0x7f4c1c1570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:02.372 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.363+0000 7f4c20d59700 1 -- 172.21.15.195:0/219734337 >> v2:172.21.15.202:3300/0 conn(0x7f4c1c06fef0 msgr2=0x7f4c1c1570a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:59:02.373 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.363+0000 7f4c20d59700 1 --2- 172.21.15.195:0/219734337 >> v2:172.21.15.202:3300/0 conn(0x7f4c1c06fef0 0x7f4c1c1570a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:02.373 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.363+0000 7f4c20d59700 1 -- 172.21.15.195:0/219734337 >> v1:172.21.15.202:6789/0 conn(0x7f4c1c06e850 legacy=0x7f4c1c069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:02.373 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.363+0000 7f4c20d59700 1 -- 172.21.15.195:0/219734337 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c1c06f8c0 con 0x7f4c1c0c34a0 2022-06-27T11:59:02.373 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.363+0000 7f4c20d59700 1 --2- 172.21.15.195:0/219734337 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4c1c0c34a0 0x7f4c1c0c2920 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f4c1c06e600 tx=0x7f4c14009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cad80f71b25eebdd server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:59:02.374 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.363+0000 7f4c1affd700 1 -- 172.21.15.195:0/219734337 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c1400e070 con 0x7f4c1c0c34a0 2022-06-27T11:59:02.374 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.363+0000 7f4c1affd700 1 -- 172.21.15.195:0/219734337 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c1400bab0 con 0x7f4c1c0c34a0 2022-06-27T11:59:02.374 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.363+0000 7f4c1affd700 1 -- 172.21.15.195:0/219734337 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c140092e0 con 0x7f4c1c0c34a0 2022-06-27T11:59:02.374 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.364+0000 7f4c227bc700 1 -- 172.21.15.195:0/219734337 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4c1c0c34a0 msgr2=0x7f4c1c0c2920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:02.375 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.364+0000 7f4c227bc700 1 --2- 172.21.15.195:0/219734337 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4c1c0c34a0 0x7f4c1c0c2920 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f4c1c06e600 tx=0x7f4c14009840 comp rx=0 tx=0).stop 2022-06-27T11:59:02.375 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.364+0000 7f4c227bc700 1 -- 172.21.15.195:0/219734337 shutdown_connections 2022-06-27T11:59:02.375 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.364+0000 7f4c227bc700 1 --2- 172.21.15.195:0/219734337 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4c1c0c34a0 0x7f4c1c0c2920 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:02.375 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.364+0000 7f4c227bc700 1 --2- 172.21.15.195:0/219734337 >> v2:172.21.15.202:3300/0 conn(0x7f4c1c06fef0 0x7f4c1c1570a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:02.376 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.364+0000 7f4c227bc700 1 -- 172.21.15.195:0/219734337 >> 172.21.15.195:0/219734337 conn(0x7f4c1c0c6350 msgr2=0x7f4c1c0c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:59:02.376 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.364+0000 7f4c227bc700 1 -- 172.21.15.195:0/219734337 shutdown_connections 2022-06-27T11:59:02.376 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.364+0000 7f4c227bc700 1 -- 172.21.15.195:0/219734337 wait complete. 2022-06-27T11:59:02.376 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.365+0000 7f4c227bc700 1 Processor -- start 2022-06-27T11:59:02.377 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.365+0000 7f4c227bc700 1 -- start start 2022-06-27T11:59:02.377 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.366+0000 7f4c227bc700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4c1c06e850 0x7f4c1c167550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:02.377 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.366+0000 7f4c227bc700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4c1c0c34a0 0x7f4c1c155850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:02.378 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.366+0000 7f4c227bc700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4c1c158130 0x7f4c1c151e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:02.378 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.366+0000 7f4c227bc700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c1c151970 con 0x7f4c1c0c34a0 2022-06-27T11:59:02.379 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.366+0000 7f4c227bc700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c1c1510b0 con 0x7f4c1c06e850 2022-06-27T11:59:02.379 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.366+0000 7f4c227bc700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c1c158630 con 0x7f4c1c158130 2022-06-27T11:59:02.379 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.366+0000 7f4c1b7fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4c1c0c34a0 0x7f4c1c155850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:02.379 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.366+0000 7f4c1bfff700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4c1c06e850 0x7f4c1c167550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:02.380 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.366+0000 7f4c1b7fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4c1c0c34a0 0x7f4c1c155850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45926/0 (socket says 172.21.15.195:45926) 2022-06-27T11:59:02.380 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.366+0000 7f4c1b7fe700 1 -- 172.21.15.195:0/1515584678 learned_addr learned my addr 172.21.15.195:0/1515584678 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:59:02.380 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.366+0000 7f4c20d59700 1 --2- 172.21.15.195:0/1515584678 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4c1c158130 0x7f4c1c151e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:02.380 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.366+0000 7f4c1b7fe700 1 -- 172.21.15.195:0/1515584678 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4c1c158130 msgr2=0x7f4c1c151e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:02.381 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.366+0000 7f4c1b7fe700 1 --2- 172.21.15.195:0/1515584678 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4c1c158130 0x7f4c1c151e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:02.381 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.366+0000 7f4c1b7fe700 1 -- 172.21.15.195:0/1515584678 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4c1c06e850 msgr2=0x7f4c1c167550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:02.381 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.366+0000 7f4c1b7fe700 1 --2- 172.21.15.195:0/1515584678 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4c1c06e850 0x7f4c1c167550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:02.382 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.366+0000 7f4c1b7fe700 1 -- 172.21.15.195:0/1515584678 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c14009ae0 con 0x7f4c1c0c34a0 2022-06-27T11:59:02.382 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.366+0000 7f4c1b7fe700 1 --2- 172.21.15.195:0/1515584678 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4c1c0c34a0 0x7f4c1c155850 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f4c0c00e930 tx=0x7f4c0c009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:59:02.382 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.366+0000 7f4c03fff700 1 -- 172.21.15.195:0/1515584678 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c0c013680 con 0x7f4c1c0c34a0 2022-06-27T11:59:02.383 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.366+0000 7f4c03fff700 1 -- 172.21.15.195:0/1515584678 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c0c0137e0 con 0x7f4c1c0c34a0 2022-06-27T11:59:02.383 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.366+0000 7f4c03fff700 1 -- 172.21.15.195:0/1515584678 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c0c013b80 con 0x7f4c1c0c34a0 2022-06-27T11:59:02.384 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.366+0000 7f4c227bc700 1 -- 172.21.15.195:0/1515584678 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c1c151320 con 0x7f4c1c0c34a0 2022-06-27T11:59:02.384 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.366+0000 7f4c227bc700 1 -- 172.21.15.195:0/1515584678 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c1c1567f0 con 0x7f4c1c0c34a0 2022-06-27T11:59:02.384 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.367+0000 7f4c227bc700 1 -- 172.21.15.195:0/1515584678 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c1c151b20 con 0x7f4c1c0c34a0 2022-06-27T11:59:02.384 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.368+0000 7f4c03fff700 1 -- 172.21.15.195:0/1515584678 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f4c0c019030 con 0x7f4c1c0c34a0 2022-06-27T11:59:02.385 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.368+0000 7f4c03fff700 1 --2- 172.21.15.195:0/1515584678 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4bfc063d30 0x7f4bfc0661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:02.385 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.368+0000 7f4c03fff700 1 -- 172.21.15.195:0/1515584678 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6718+0+0 (secure 0 0 0) 0x7f4c0c0809a0 con 0x7f4c1c0c34a0 2022-06-27T11:59:02.385 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.368+0000 7f4c1bfff700 1 --2- 172.21.15.195:0/1515584678 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4bfc063d30 0x7f4bfc0661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:02.385 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.369+0000 7f4c1bfff700 1 --2- 172.21.15.195:0/1515584678 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4bfc063d30 0x7f4bfc0661e0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f4c100017e0 tx=0x7f4c10009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:59:02.386 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.373+0000 7f4c03fff700 1 -- 172.21.15.195:0/1515584678 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f4c1c151b20 con 0x7f4c1c0c34a0 2022-06-27T11:59:02.609 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.604+0000 7f4c227bc700 1 -- 172.21.15.195:0/1515584678 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4c1c151b20 con 0x7f4c1c0c34a0 2022-06-27T11:59:02.609 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.605+0000 7f4c03fff700 1 -- 172.21.15.195:0/1515584678 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+5367 (secure 0 0 0) 0x7f4c1c151b20 con 0x7f4c1c0c34a0 2022-06-27T11:59:02.610 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:59:02.610 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:58:54.982841+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5192},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5192":{"gid":5192,"name":"b","rank":0,"incarnation":43,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6833/2651813808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":2651813808},{"type":"v1","addr":"172.21.15.202:6833","nonce":2651813808}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:59:02.611 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.607+0000 7f4c227bc700 1 -- 172.21.15.195:0/1515584678 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4bfc063d30 msgr2=0x7f4bfc0661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:02.612 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.607+0000 7f4c227bc700 1 --2- 172.21.15.195:0/1515584678 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4bfc063d30 0x7f4bfc0661e0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f4c100017e0 tx=0x7f4c10009040 comp rx=0 tx=0).stop 2022-06-27T11:59:02.612 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.607+0000 7f4c227bc700 1 -- 172.21.15.195:0/1515584678 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4c1c0c34a0 msgr2=0x7f4c1c155850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:02.612 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.607+0000 7f4c227bc700 1 --2- 172.21.15.195:0/1515584678 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4c1c0c34a0 0x7f4c1c155850 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f4c0c00e930 tx=0x7f4c0c009060 comp rx=0 tx=0).stop 2022-06-27T11:59:02.613 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.607+0000 7f4c227bc700 1 -- 172.21.15.195:0/1515584678 shutdown_connections 2022-06-27T11:59:02.613 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.607+0000 7f4c227bc700 1 --2- 172.21.15.195:0/1515584678 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4bfc063d30 0x7f4bfc0661e0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:02.613 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.607+0000 7f4c227bc700 1 --2- 172.21.15.195:0/1515584678 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4c1c158130 0x7f4c1c151e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:02.613 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.607+0000 7f4c227bc700 1 --2- 172.21.15.195:0/1515584678 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4c1c06e850 0x7f4c1c167550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:02.613 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.607+0000 7f4c227bc700 1 --2- 172.21.15.195:0/1515584678 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4c1c0c34a0 0x7f4c1c155850 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:02.614 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.607+0000 7f4c227bc700 1 -- 172.21.15.195:0/1515584678 >> 172.21.15.195:0/1515584678 conn(0x7f4c1c0c6350 msgr2=0x7f4c1c068530 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:59:02.615 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.607+0000 7f4c227bc700 1 -- 172.21.15.195:0/1515584678 shutdown_connections 2022-06-27T11:59:02.615 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:02.607+0000 7f4c227bc700 1 -- 172.21.15.195:0/1515584678 wait complete. 2022-06-27T11:59:02.615 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 43 2022-06-27T11:59:02.621 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T11:59:04.622 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:59:04.775 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:04.763+0000 7f7100320700 1 Processor -- start 2022-06-27T11:59:04.776 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:04.763+0000 7f7100320700 1 -- start start 2022-06-27T11:59:04.776 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:04.764+0000 7f7100320700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f70f8150950 0x7f70f8156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:04.776 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:04.764+0000 7f7100320700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70f80c76b0 con 0x7f70f8150580 2022-06-27T11:59:04.777 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:04.764+0000 7f7100320700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70f80c8250 con 0x7f70f806f620 2022-06-27T11:59:04.777 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:04.764+0000 7f7100320700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70f8157430 con 0x7f70f8150950 2022-06-27T11:59:04.777 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:04.764+0000 7f70fe0bc700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f70f8150580 0x7f70f80c7b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:52124/0 (socket says 172.21.15.195:52124) 2022-06-27T11:59:04.777 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:04.764+0000 7f70fe0bc700 1 -- 172.21.15.195:0/3626995518 learned_addr learned my addr 172.21.15.195:0/3626995518 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:59:04.778 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:04.764+0000 7f70fd8bb700 1 --2- 172.21.15.195:0/3626995518 >> v2:172.21.15.202:3300/0 conn(0x7f70f8150950 0x7f70f8156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:04.778 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:04.764+0000 7f70fd0ba700 1 -- 172.21.15.195:0/3626995518 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1193994468 0 0) 0x7f70f80c76b0 con 0x7f70f8150580 2022-06-27T11:59:04.778 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:04.764+0000 7f70fd0ba700 1 -- 172.21.15.195:0/3626995518 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70e0003430 con 0x7f70f8150580 2022-06-27T11:59:04.778 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:04.764+0000 7f70fd8bb700 1 -- 172.21.15.195:0/3626995518 >> v1:172.21.15.202:6789/0 conn(0x7f70f806f620 legacy=0x7f70f806d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:04.779 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:04.764+0000 7f70fd8bb700 1 -- 172.21.15.195:0/3626995518 >> v1:172.21.15.195:6789/0 conn(0x7f70f8150580 legacy=0x7f70f80c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:04.779 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:04.764+0000 7f70fd8bb700 1 -- 172.21.15.195:0/3626995518 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70f80bad70 con 0x7f70f8150950 2022-06-27T11:59:04.779 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:04.765+0000 7f70fd8bb700 1 --2- 172.21.15.195:0/3626995518 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f70f8150950 0x7f70f8156ef0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f70e80097a0 tx=0x7f70e8009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1954e6b044975c28 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:59:04.791 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:04.765+0000 7f70fd0ba700 1 -- 172.21.15.195:0/3626995518 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e8011630 con 0x7f70f8150950 2022-06-27T11:59:04.791 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:04.765+0000 7f70fd0ba700 1 -- 172.21.15.195:0/3626995518 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70e8011790 con 0x7f70f8150950 2022-06-27T11:59:04.792 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:04.765+0000 7f70fd0ba700 1 -- 172.21.15.195:0/3626995518 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e8011b30 con 0x7f70f8150950 2022-06-27T11:59:04.792 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:04.765+0000 7f7100320700 1 -- 172.21.15.195:0/3626995518 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f70f8150950 msgr2=0x7f70f8156ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:04.793 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:04.765+0000 7f7100320700 1 --2- 172.21.15.195:0/3626995518 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f70f8150950 0x7f70f8156ef0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f70e80097a0 tx=0x7f70e8009060 comp rx=0 tx=0).stop 2022-06-27T11:59:04.793 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:04.765+0000 7f7100320700 1 -- 172.21.15.195:0/3626995518 shutdown_connections 2022-06-27T11:59:04.793 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:04.765+0000 7f7100320700 1 --2- 172.21.15.195:0/3626995518 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f70f8150950 0x7f70f8156ef0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:04.794 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:04.765+0000 7f7100320700 1 -- 172.21.15.195:0/3626995518 >> 172.21.15.195:0/3626995518 conn(0x7f70f8063af0 msgr2=0x7f70f8063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:59:04.794 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:04.765+0000 7f7100320700 1 -- 172.21.15.195:0/3626995518 shutdown_connections 2022-06-27T11:59:04.794 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:04.765+0000 7f7100320700 1 -- 172.21.15.195:0/3626995518 wait complete. 2022-06-27T11:59:04.795 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:04.766+0000 7f7100320700 1 Processor -- start 2022-06-27T11:59:04.796 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:04.767+0000 7f7100320700 1 -- start start 2022-06-27T11:59:04.796 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:04.767+0000 7f7100320700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f70f806f620 0x7f70f8158ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:04.797 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:04.767+0000 7f7100320700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f70f8150580 0x7f70f814d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:04.797 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:04.767+0000 7f7100320700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f70f8150950 0x7f70f8169ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:04.798 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:04.767+0000 7f7100320700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70f81675f0 con 0x7f70f8150580 2022-06-27T11:59:04.798 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:04.767+0000 7f7100320700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70f81669e0 con 0x7f70f806f620 2022-06-27T11:59:04.798 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:04.767+0000 7f7100320700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70f8166cc0 con 0x7f70f8150950 2022-06-27T11:59:04.799 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:04.767+0000 7f70fd8bb700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f70f8150580 0x7f70f814d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:04.800 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:04.767+0000 7f70fd8bb700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f70f8150580 0x7f70f814d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45944/0 (socket says 172.21.15.195:45944) 2022-06-27T11:59:04.800 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:04.767+0000 7f70fd8bb700 1 -- 172.21.15.195:0/2004652071 learned_addr learned my addr 172.21.15.195:0/2004652071 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:59:04.801 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:04.767+0000 7f70fe0bc700 1 --2- 172.21.15.195:0/2004652071 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f70f806f620 0x7f70f8158ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:04.801 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:04.767+0000 7f70fe8bd700 1 --2- 172.21.15.195:0/2004652071 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f70f8150950 0x7f70f8169ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:04.802 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:04.768+0000 7f70fd8bb700 1 -- 172.21.15.195:0/2004652071 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f70f8150950 msgr2=0x7f70f8169ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:04.803 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:04.768+0000 7f70fd8bb700 1 --2- 172.21.15.195:0/2004652071 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f70f8150950 0x7f70f8169ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:04.803 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:04.768+0000 7f70fd8bb700 1 -- 172.21.15.195:0/2004652071 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f70f806f620 msgr2=0x7f70f8158ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:04.806 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:04.768+0000 7f70fd8bb700 1 --2- 172.21.15.195:0/2004652071 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f70f806f620 0x7f70f8158ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:04.806 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:04.768+0000 7f70fd8bb700 1 -- 172.21.15.195:0/2004652071 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70e80092e0 con 0x7f70f8150580 2022-06-27T11:59:04.806 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:04.768+0000 7f70fd8bb700 1 --2- 172.21.15.195:0/2004652071 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f70f8150580 0x7f70f814d690 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f70e8009e40 tx=0x7f70e8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:59:04.807 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:04.768+0000 7f70e67fc700 1 -- 172.21.15.195:0/2004652071 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e8000bc0 con 0x7f70f8150580 2022-06-27T11:59:04.807 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:04.768+0000 7f70e67fc700 1 -- 172.21.15.195:0/2004652071 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70e8011900 con 0x7f70f8150580 2022-06-27T11:59:04.807 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:04.768+0000 7f7100320700 1 -- 172.21.15.195:0/2004652071 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70f8166fa0 con 0x7f70f8150580 2022-06-27T11:59:04.807 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:04.768+0000 7f70e67fc700 1 -- 172.21.15.195:0/2004652071 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e8000d20 con 0x7f70f8150580 2022-06-27T11:59:04.808 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:04.768+0000 7f7100320700 1 -- 172.21.15.195:0/2004652071 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70f8166690 con 0x7f70f8150580 2022-06-27T11:59:04.808 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:04.769+0000 7f7100320700 1 -- 172.21.15.195:0/2004652071 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70f8169be0 con 0x7f70f8150580 2022-06-27T11:59:04.816 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:04.769+0000 7f70e67fc700 1 -- 172.21.15.195:0/2004652071 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f70e802c030 con 0x7f70f8150580 2022-06-27T11:59:04.817 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:04.770+0000 7f70e67fc700 1 --2- 172.21.15.195:0/2004652071 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f70d8063d00 0x7f70d80661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:04.817 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:04.770+0000 7f70e67fc700 1 -- 172.21.15.195:0/2004652071 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6718+0+0 (secure 0 0 0) 0x7f70e8082130 con 0x7f70f8150580 2022-06-27T11:59:04.817 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:04.770+0000 7f70fe0bc700 1 --2- 172.21.15.195:0/2004652071 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f70d8063d00 0x7f70d80661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:04.817 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:04.770+0000 7f70fe0bc700 1 --2- 172.21.15.195:0/2004652071 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f70d8063d00 0x7f70d80661b0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f70f4007fd0 tx=0x7f70f4009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:59:04.818 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:04.774+0000 7f70e67fc700 1 -- 172.21.15.195:0/2004652071 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f70e80509a0 con 0x7f70f8150580 2022-06-27T11:59:05.036 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:05.003+0000 7f7100320700 1 -- 172.21.15.195:0/2004652071 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f70f80722c0 con 0x7f70f8150580 2022-06-27T11:59:05.047 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:05.003+0000 7f70e67fc700 1 -- 172.21.15.195:0/2004652071 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+5367 (secure 0 0 0) 0x7f70e802a020 con 0x7f70f8150580 2022-06-27T11:59:05.048 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:59:05.048 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:58:54.982841+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5192},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5192":{"gid":5192,"name":"b","rank":0,"incarnation":43,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6833/2651813808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":2651813808},{"type":"v1","addr":"172.21.15.202:6833","nonce":2651813808}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:59:05.051 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:05.005+0000 7f7100320700 1 -- 172.21.15.195:0/2004652071 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f70d8063d00 msgr2=0x7f70d80661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:05.052 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:05.005+0000 7f7100320700 1 --2- 172.21.15.195:0/2004652071 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f70d8063d00 0x7f70d80661b0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f70f4007fd0 tx=0x7f70f4009040 comp rx=0 tx=0).stop 2022-06-27T11:59:05.052 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:05.005+0000 7f7100320700 1 -- 172.21.15.195:0/2004652071 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f70f8150580 msgr2=0x7f70f814d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:05.053 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:05.005+0000 7f7100320700 1 --2- 172.21.15.195:0/2004652071 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f70f8150580 0x7f70f814d690 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f70e8009e40 tx=0x7f70e8017000 comp rx=0 tx=0).stop 2022-06-27T11:59:05.054 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:05.005+0000 7f7100320700 1 -- 172.21.15.195:0/2004652071 shutdown_connections 2022-06-27T11:59:05.054 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:05.005+0000 7f7100320700 1 --2- 172.21.15.195:0/2004652071 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f70d8063d00 0x7f70d80661b0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:05.054 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:05.005+0000 7f7100320700 1 --2- 172.21.15.195:0/2004652071 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f70f8150950 0x7f70f8169ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:05.055 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:05.005+0000 7f7100320700 1 --2- 172.21.15.195:0/2004652071 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f70f806f620 0x7f70f8158ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:05.055 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:05.005+0000 7f7100320700 1 --2- 172.21.15.195:0/2004652071 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f70f8150580 0x7f70f814d690 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:05.063 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:05.005+0000 7f7100320700 1 -- 172.21.15.195:0/2004652071 >> 172.21.15.195:0/2004652071 conn(0x7f70f8063af0 msgr2=0x7f70f806a380 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:59:05.064 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:05.005+0000 7f7100320700 1 -- 172.21.15.195:0/2004652071 shutdown_connections 2022-06-27T11:59:05.064 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:05.005+0000 7f7100320700 1 -- 172.21.15.195:0/2004652071 wait complete. 2022-06-27T11:59:05.064 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 43 2022-06-27T11:59:05.066 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T11:59:07.068 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:59:07.227 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.207+0000 7fc30dca4700 1 Processor -- start 2022-06-27T11:59:07.228 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.207+0000 7fc30dca4700 1 -- start start 2022-06-27T11:59:07.228 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.208+0000 7fc30dca4700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc3080c4160 0x7fc308156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:07.229 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.208+0000 7fc30dca4700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fc3080c39e0 0x7fc3081535a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:07.229 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.208+0000 7fc30dca4700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3080c1a20 con 0x7fc30806f640 2022-06-27T11:59:07.230 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.208+0000 7fc30dca4700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3080c1130 con 0x7fc3080c39e0 2022-06-27T11:59:07.231 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.208+0000 7fc30dca4700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc30806ea90 con 0x7fc3080c4160 2022-06-27T11:59:07.231 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.208+0000 7fc307fff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fc3080c39e0 0x7fc3081535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:07.232 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.208+0000 7fc307fff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fc3080c39e0 0x7fc3081535a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45956/0 (socket says 172.21.15.195:45956) 2022-06-27T11:59:07.232 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.208+0000 7fc307fff700 1 -- 172.21.15.195:0/958481099 learned_addr learned my addr 172.21.15.195:0/958481099 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:59:07.233 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.208+0000 7fc306ffd700 1 --2- 172.21.15.195:0/958481099 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc3080c4160 0x7fc308156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:07.233 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.208+0000 7fc307fff700 1 -- 172.21.15.195:0/958481099 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc3080c4160 msgr2=0x7fc308156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:59:07.234 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.208+0000 7fc307fff700 1 --2- 172.21.15.195:0/958481099 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc3080c4160 0x7fc308156ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:07.235 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.208+0000 7fc307fff700 1 -- 172.21.15.195:0/958481099 >> v1:172.21.15.202:6789/0 conn(0x7fc30806f640 legacy=0x7fc3080c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:59:07.236 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.208+0000 7fc307fff700 1 -- 172.21.15.195:0/958481099 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc30806ed70 con 0x7fc3080c39e0 2022-06-27T11:59:07.236 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.208+0000 7fc306ffd700 1 --2- 172.21.15.195:0/958481099 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc3080c4160 0x7fc308156ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T11:59:07.236 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.209+0000 7fc307fff700 1 --2- 172.21.15.195:0/958481099 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc3080c39e0 0x7fc3081535a0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fc300009f20 tx=0x7fc300009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d253ee7fbf51d93c server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:59:07.242 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.209+0000 7fc3067fc700 1 -- 172.21.15.195:0/958481099 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc30000e070 con 0x7fc3080c39e0 2022-06-27T11:59:07.243 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.209+0000 7fc3067fc700 1 -- 172.21.15.195:0/958481099 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc30000bad0 con 0x7fc3080c39e0 2022-06-27T11:59:07.243 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.209+0000 7fc3067fc700 1 -- 172.21.15.195:0/958481099 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3000092e0 con 0x7fc3080c39e0 2022-06-27T11:59:07.243 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.209+0000 7fc30dca4700 1 -- 172.21.15.195:0/958481099 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc3080c39e0 msgr2=0x7fc3081535a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:07.243 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.209+0000 7fc30dca4700 1 --2- 172.21.15.195:0/958481099 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc3080c39e0 0x7fc3081535a0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fc300009f20 tx=0x7fc300009840 comp rx=0 tx=0).stop 2022-06-27T11:59:07.244 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.209+0000 7fc30dca4700 1 -- 172.21.15.195:0/958481099 shutdown_connections 2022-06-27T11:59:07.244 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.209+0000 7fc30dca4700 1 --2- 172.21.15.195:0/958481099 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc3080c39e0 0x7fc3081535a0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:07.244 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.209+0000 7fc30dca4700 1 --2- 172.21.15.195:0/958481099 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc3080c4160 0x7fc308156ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:07.244 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.209+0000 7fc30dca4700 1 -- 172.21.15.195:0/958481099 >> 172.21.15.195:0/958481099 conn(0x7fc3080c6350 msgr2=0x7fc3080c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:59:07.245 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.209+0000 7fc30dca4700 1 -- 172.21.15.195:0/958481099 shutdown_connections 2022-06-27T11:59:07.246 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.209+0000 7fc30dca4700 1 -- 172.21.15.195:0/958481099 wait complete. 2022-06-27T11:59:07.246 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.211+0000 7fc30dca4700 1 Processor -- start 2022-06-27T11:59:07.247 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.211+0000 7fc30dca4700 1 -- start start 2022-06-27T11:59:07.248 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.211+0000 7fc30dca4700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc30806f640 0x7fc308169770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:07.249 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.211+0000 7fc30dca4700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc3080c39e0 0x7fc308156d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:07.249 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.211+0000 7fc30dca4700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc3080c4160 0x7fc30815c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:07.250 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.211+0000 7fc30dca4700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc308154810 con 0x7fc3080c4160 2022-06-27T11:59:07.250 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.211+0000 7fc30dca4700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3080ba880 con 0x7fc3080c39e0 2022-06-27T11:59:07.250 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.211+0000 7fc30dca4700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc308156900 con 0x7fc30806f640 2022-06-27T11:59:07.250 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.211+0000 7fc307fff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc3080c4160 0x7fc30815c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:07.251 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.211+0000 7fc307fff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc3080c4160 0x7fc30815c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45962/0 (socket says 172.21.15.195:45962) 2022-06-27T11:59:07.251 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.211+0000 7fc307fff700 1 -- 172.21.15.195:0/2932539816 learned_addr learned my addr 172.21.15.195:0/2932539816 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:59:07.251 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.212+0000 7fc306ffd700 1 --2- 172.21.15.195:0/2932539816 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc3080c39e0 0x7fc308156d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:07.251 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.212+0000 7fc3077fe700 1 --2- 172.21.15.195:0/2932539816 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc30806f640 0x7fc308169770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:07.252 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.212+0000 7fc307fff700 1 -- 172.21.15.195:0/2932539816 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc30806f640 msgr2=0x7fc308169770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:07.252 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.212+0000 7fc307fff700 1 --2- 172.21.15.195:0/2932539816 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc30806f640 0x7fc308169770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:07.252 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.212+0000 7fc307fff700 1 -- 172.21.15.195:0/2932539816 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc3080c39e0 msgr2=0x7fc308156d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:07.252 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.212+0000 7fc307fff700 1 --2- 172.21.15.195:0/2932539816 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc3080c39e0 0x7fc308156d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:07.253 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.212+0000 7fc307fff700 1 -- 172.21.15.195:0/2932539816 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc300009ae0 con 0x7fc3080c4160 2022-06-27T11:59:07.253 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.212+0000 7fc306ffd700 1 --2- 172.21.15.195:0/2932539816 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc3080c39e0 0x7fc308156d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T11:59:07.253 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.212+0000 7fc307fff700 1 --2- 172.21.15.195:0/2932539816 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc3080c4160 0x7fc30815c2d0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fc30000b6c0 tx=0x7fc300017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:59:07.253 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.212+0000 7fc2ef7fe700 1 -- 172.21.15.195:0/2932539816 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3000094e0 con 0x7fc3080c4160 2022-06-27T11:59:07.253 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.212+0000 7fc2ef7fe700 1 -- 172.21.15.195:0/2932539816 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc300004240 con 0x7fc3080c4160 2022-06-27T11:59:07.254 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.212+0000 7fc30dca4700 1 -- 172.21.15.195:0/2932539816 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc308155eb0 con 0x7fc3080c4160 2022-06-27T11:59:07.254 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.212+0000 7fc2ef7fe700 1 -- 172.21.15.195:0/2932539816 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3000045e0 con 0x7fc3080c4160 2022-06-27T11:59:07.254 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.212+0000 7fc30dca4700 1 -- 172.21.15.195:0/2932539816 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc308154250 con 0x7fc3080c4160 2022-06-27T11:59:07.255 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.213+0000 7fc30dca4700 1 -- 172.21.15.195:0/2932539816 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3080722c0 con 0x7fc3080c4160 2022-06-27T11:59:07.255 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.213+0000 7fc2ef7fe700 1 -- 172.21.15.195:0/2932539816 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fc30002c070 con 0x7fc3080c4160 2022-06-27T11:59:07.255 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.214+0000 7fc2ef7fe700 1 --2- 172.21.15.195:0/2932539816 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc2e8063d00 0x7fc2e80661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:07.255 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.214+0000 7fc2ef7fe700 1 -- 172.21.15.195:0/2932539816 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6718+0+0 (secure 0 0 0) 0x7fc3000820e0 con 0x7fc3080c4160 2022-06-27T11:59:07.258 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.214+0000 7fc3077fe700 1 --2- 172.21.15.195:0/2932539816 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc2e8063d00 0x7fc2e80661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:07.259 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.214+0000 7fc3077fe700 1 --2- 172.21.15.195:0/2932539816 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc2e8063d00 0x7fc2e80661b0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc003c60 tx=0x7fc2fc009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:59:07.260 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.218+0000 7fc2ef7fe700 1 -- 172.21.15.195:0/2932539816 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fc300050a40 con 0x7fc3080c4160 2022-06-27T11:59:07.443 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.440+0000 7fc30dca4700 1 -- 172.21.15.195:0/2932539816 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc308156a40 con 0x7fc3080c4160 2022-06-27T11:59:07.444 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.441+0000 7fc2ef7fe700 1 -- 172.21.15.195:0/2932539816 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+5370 (secure 0 0 0) 0x7fc30002a020 con 0x7fc3080c4160 2022-06-27T11:59:07.445 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:59:07.445 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:59:06.036658+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5192},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5192":{"gid":5192,"name":"b","rank":0,"incarnation":43,"state":"up:reconnect","state_seq":8,"addr":"172.21.15.202:6833/2651813808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":2651813808},{"type":"v1","addr":"172.21.15.202:6833","nonce":2651813808}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:59:07.458 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.443+0000 7fc30dca4700 1 -- 172.21.15.195:0/2932539816 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc2e8063d00 msgr2=0x7fc2e80661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:07.459 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.443+0000 7fc30dca4700 1 --2- 172.21.15.195:0/2932539816 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc2e8063d00 0x7fc2e80661b0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc003c60 tx=0x7fc2fc009040 comp rx=0 tx=0).stop 2022-06-27T11:59:07.459 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.443+0000 7fc30dca4700 1 -- 172.21.15.195:0/2932539816 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc3080c4160 msgr2=0x7fc30815c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:07.460 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.443+0000 7fc30dca4700 1 --2- 172.21.15.195:0/2932539816 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc3080c4160 0x7fc30815c2d0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fc30000b6c0 tx=0x7fc300017000 comp rx=0 tx=0).stop 2022-06-27T11:59:07.461 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.443+0000 7fc30dca4700 1 -- 172.21.15.195:0/2932539816 shutdown_connections 2022-06-27T11:59:07.461 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.443+0000 7fc30dca4700 1 --2- 172.21.15.195:0/2932539816 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc2e8063d00 0x7fc2e80661b0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:07.462 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.443+0000 7fc30dca4700 1 --2- 172.21.15.195:0/2932539816 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc3080c4160 0x7fc30815c2d0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:07.462 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.443+0000 7fc30dca4700 1 --2- 172.21.15.195:0/2932539816 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc30806f640 0x7fc308169770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:07.463 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.443+0000 7fc30dca4700 1 --2- 172.21.15.195:0/2932539816 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc3080c39e0 0x7fc308156d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:07.463 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.443+0000 7fc30dca4700 1 -- 172.21.15.195:0/2932539816 >> 172.21.15.195:0/2932539816 conn(0x7fc3080c6350 msgr2=0x7fc308150550 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:59:07.464 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.443+0000 7fc30dca4700 1 -- 172.21.15.195:0/2932539816 shutdown_connections 2022-06-27T11:59:07.464 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:07.443+0000 7fc30dca4700 1 -- 172.21.15.195:0/2932539816 wait complete. 2022-06-27T11:59:07.464 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 44 2022-06-27T11:59:07.467 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 44, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-06-27T11:47:52.358095+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 44, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5192": { "addr": "172.21.15.202:6833/2651813808", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6832", "nonce": 2651813808, "type": "v2" }, { "addr": "172.21.15.202:6833", "nonce": 2651813808, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5192, "incarnation": 43, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:reconnect", "state_seq": 8 } }, "last_failure": 0, "last_failure_osd_epoch": 54, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-06-27T11:59:06.036658+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5192 } } } ], "standbys": [ { "addr": "172.21.15.202:6837/3305006841", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6836", "nonce": 3305006841, "type": "v2" }, { "addr": "172.21.15.202:6837", "nonce": 3305006841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6835/2060709822", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6834", "nonce": 2060709822, "type": "v2" }, { "addr": "172.21.15.195:6835", "nonce": 2060709822, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6838/1598751652", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6836", "nonce": 1598751652, "type": "v2" }, { "addr": "172.21.15.195:6838", "nonce": 1598751652, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4275, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6839/1313800661", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6837", "nonce": 1313800661, "type": "v2" }, { "addr": "172.21.15.195:6839", "nonce": 1313800661, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4278, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-06-27T11:59:09.474 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:59:09.614 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.614+0000 7f3a0d378700 1 Processor -- start 2022-06-27T11:59:09.615 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.614+0000 7f3a0d378700 1 -- start start 2022-06-27T11:59:09.618 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.615+0000 7f3a0d378700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f3a080c40d0 0x7f3a08157010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:09.618 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.615+0000 7f3a0d378700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f3a08067ed0 0x7f3a0814cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:09.618 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.615+0000 7f3a0d378700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a08156b80 con 0x7f3a080c6190 2022-06-27T11:59:09.619 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.615+0000 7f3a0d378700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a080682f0 con 0x7f3a08067ed0 2022-06-27T11:59:09.619 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.615+0000 7f3a0d378700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a0806a040 con 0x7f3a080c40d0 2022-06-27T11:59:09.619 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.615+0000 7f3a077fe700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f3a08067ed0 0x7f3a0814cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:09.619 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.615+0000 7f3a077fe700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f3a08067ed0 0x7f3a0814cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45968/0 (socket says 172.21.15.195:45968) 2022-06-27T11:59:09.620 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.615+0000 7f3a077fe700 1 -- 172.21.15.195:0/3983137407 learned_addr learned my addr 172.21.15.195:0/3983137407 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:59:09.620 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.615+0000 7f3a06ffd700 1 --2- 172.21.15.195:0/3983137407 >> v2:172.21.15.202:3300/0 conn(0x7f3a080c40d0 0x7f3a08157010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:09.620 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.616+0000 7f3a077fe700 1 -- 172.21.15.195:0/3983137407 >> v2:172.21.15.202:3300/0 conn(0x7f3a080c40d0 msgr2=0x7f3a08157010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:59:09.620 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.616+0000 7f3a077fe700 1 --2- 172.21.15.195:0/3983137407 >> v2:172.21.15.202:3300/0 conn(0x7f3a080c40d0 0x7f3a08157010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:09.620 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.616+0000 7f3a077fe700 1 -- 172.21.15.195:0/3983137407 >> v1:172.21.15.202:6789/0 conn(0x7f3a080c6190 legacy=0x7f3a080c44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:09.621 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.616+0000 7f3a077fe700 1 -- 172.21.15.195:0/3983137407 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a0806a300 con 0x7f3a08067ed0 2022-06-27T11:59:09.621 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.616+0000 7f3a077fe700 1 --2- 172.21.15.195:0/3983137407 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3a08067ed0 0x7f3a0814cb80 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f3a080cdb00 tx=0x7f3a00009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46500ee1cac30067 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:59:09.621 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.616+0000 7f3a05ffb700 1 -- 172.21.15.195:0/3983137407 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a0000e070 con 0x7f3a08067ed0 2022-06-27T11:59:09.621 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.616+0000 7f3a05ffb700 1 -- 172.21.15.195:0/3983137407 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a0000ba20 con 0x7f3a08067ed0 2022-06-27T11:59:09.622 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.616+0000 7f3a05ffb700 1 -- 172.21.15.195:0/3983137407 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a000092e0 con 0x7f3a08067ed0 2022-06-27T11:59:09.622 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.616+0000 7f3a0d378700 1 -- 172.21.15.195:0/3983137407 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3a08067ed0 msgr2=0x7f3a0814cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:09.622 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.616+0000 7f3a0d378700 1 --2- 172.21.15.195:0/3983137407 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3a08067ed0 0x7f3a0814cb80 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f3a080cdb00 tx=0x7f3a00009840 comp rx=0 tx=0).stop 2022-06-27T11:59:09.622 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.616+0000 7f3a0d378700 1 -- 172.21.15.195:0/3983137407 shutdown_connections 2022-06-27T11:59:09.623 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.616+0000 7f3a0d378700 1 --2- 172.21.15.195:0/3983137407 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3a08067ed0 0x7f3a0814cb80 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:09.623 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.616+0000 7f3a0d378700 1 --2- 172.21.15.195:0/3983137407 >> v2:172.21.15.202:3300/0 conn(0x7f3a080c40d0 0x7f3a08157010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:09.623 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.616+0000 7f3a0d378700 1 -- 172.21.15.195:0/3983137407 >> 172.21.15.195:0/3983137407 conn(0x7f3a080ccb80 msgr2=0x7f3a080ccf80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:59:09.623 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.616+0000 7f3a0d378700 1 -- 172.21.15.195:0/3983137407 shutdown_connections 2022-06-27T11:59:09.623 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.617+0000 7f3a0d378700 1 -- 172.21.15.195:0/3983137407 wait complete. 2022-06-27T11:59:09.624 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.618+0000 7f3a0d378700 1 Processor -- start 2022-06-27T11:59:09.625 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.618+0000 7f3a0d378700 1 -- start start 2022-06-27T11:59:09.638 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.619+0000 7f3a0d378700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3a08067ed0 0x7f3a08166fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:09.638 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.619+0000 7f3a0d378700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3a080c40d0 0x7f3a08152570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:09.639 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.619+0000 7f3a0d378700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3a080c6190 0x7f3a08152ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:09.640 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.619+0000 7f3a0d378700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a081674e0 con 0x7f3a080c6190 2022-06-27T11:59:09.640 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.619+0000 7f3a0d378700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a081520e0 con 0x7f3a080c40d0 2022-06-27T11:59:09.640 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.619+0000 7f3a0d378700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a080c0a40 con 0x7f3a08067ed0 2022-06-27T11:59:09.641 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.619+0000 7f3a077fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3a080c6190 0x7f3a08152ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:09.641 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.619+0000 7f3a077fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3a080c6190 0x7f3a08152ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45974/0 (socket says 172.21.15.195:45974) 2022-06-27T11:59:09.642 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.619+0000 7f3a077fe700 1 -- 172.21.15.195:0/969944604 learned_addr learned my addr 172.21.15.195:0/969944604 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:59:09.642 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.619+0000 7f3a067fc700 1 --2- 172.21.15.195:0/969944604 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3a080c40d0 0x7f3a08152570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:09.643 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.619+0000 7f3a06ffd700 1 --2- 172.21.15.195:0/969944604 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3a08067ed0 0x7f3a08166fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:09.644 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.619+0000 7f3a077fe700 1 -- 172.21.15.195:0/969944604 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3a08067ed0 msgr2=0x7f3a08166fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:09.644 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.619+0000 7f3a077fe700 1 --2- 172.21.15.195:0/969944604 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3a08067ed0 0x7f3a08166fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:09.645 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.619+0000 7f3a077fe700 1 -- 172.21.15.195:0/969944604 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3a080c40d0 msgr2=0x7f3a08152570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:09.645 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.619+0000 7f3a077fe700 1 --2- 172.21.15.195:0/969944604 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3a080c40d0 0x7f3a08152570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:09.645 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.619+0000 7f3a077fe700 1 -- 172.21.15.195:0/969944604 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a00009ae0 con 0x7f3a080c6190 2022-06-27T11:59:09.646 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.619+0000 7f3a077fe700 1 --2- 172.21.15.195:0/969944604 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3a080c6190 0x7f3a08152ab0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f3a0000b6c0 tx=0x7f3a00018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:59:09.647 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.619+0000 7f39eeffd700 1 -- 172.21.15.195:0/969944604 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a00003d20 con 0x7f3a080c6190 2022-06-27T11:59:09.647 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.619+0000 7f39eeffd700 1 -- 172.21.15.195:0/969944604 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a00003e80 con 0x7f3a080c6190 2022-06-27T11:59:09.648 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.619+0000 7f3a0d378700 1 -- 172.21.15.195:0/969944604 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a080c0d20 con 0x7f3a080c6190 2022-06-27T11:59:09.648 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.619+0000 7f39eeffd700 1 -- 172.21.15.195:0/969944604 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a00007820 con 0x7f3a080c6190 2022-06-27T11:59:09.649 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.619+0000 7f3a0d378700 1 -- 172.21.15.195:0/969944604 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a08153a50 con 0x7f3a080c6190 2022-06-27T11:59:09.649 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.620+0000 7f3a0d378700 1 -- 172.21.15.195:0/969944604 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a080722c0 con 0x7f3a080c6190 2022-06-27T11:59:09.650 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.621+0000 7f39eeffd700 1 -- 172.21.15.195:0/969944604 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f3a00004040 con 0x7f3a080c6190 2022-06-27T11:59:09.650 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.621+0000 7f39eeffd700 1 --2- 172.21.15.195:0/969944604 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f39e8063d00 0x7f39e80661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:09.650 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.621+0000 7f39eeffd700 1 -- 172.21.15.195:0/969944604 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6718+0+0 (secure 0 0 0) 0x7f3a00082800 con 0x7f3a080c6190 2022-06-27T11:59:09.652 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.621+0000 7f3a06ffd700 1 --2- 172.21.15.195:0/969944604 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f39e8063d00 0x7f39e80661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:09.652 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.621+0000 7f3a06ffd700 1 --2- 172.21.15.195:0/969944604 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f39e8063d00 0x7f39e80661b0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f39fc009c70 tx=0x7f39fc009060 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:59:09.652 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.625+0000 7f39eeffd700 1 -- 172.21.15.195:0/969944604 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f3a00051160 con 0x7f3a080c6190 2022-06-27T11:59:09.864 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.856+0000 7f3a0d378700 1 -- 172.21.15.195:0/969944604 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3a080652c0 con 0x7f3a080c6190 2022-06-27T11:59:09.865 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.856+0000 7f39eeffd700 1 -- 172.21.15.195:0/969944604 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+5370 (secure 0 0 0) 0x7f3a00054780 con 0x7f3a080c6190 2022-06-27T11:59:09.866 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:59:09.866 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:59:06.036658+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5192},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5192":{"gid":5192,"name":"b","rank":0,"incarnation":43,"state":"up:reconnect","state_seq":8,"addr":"172.21.15.202:6833/2651813808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":2651813808},{"type":"v1","addr":"172.21.15.202:6833","nonce":2651813808}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:59:09.869 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.858+0000 7f3a0d378700 1 -- 172.21.15.195:0/969944604 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f39e8063d00 msgr2=0x7f39e80661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:09.869 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.858+0000 7f3a0d378700 1 --2- 172.21.15.195:0/969944604 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f39e8063d00 0x7f39e80661b0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f39fc009c70 tx=0x7f39fc009060 comp rx=0 tx=0).stop 2022-06-27T11:59:09.871 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.858+0000 7f3a0d378700 1 -- 172.21.15.195:0/969944604 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3a080c6190 msgr2=0x7f3a08152ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:09.879 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.858+0000 7f3a0d378700 1 --2- 172.21.15.195:0/969944604 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3a080c6190 0x7f3a08152ab0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f3a0000b6c0 tx=0x7f3a00018000 comp rx=0 tx=0).stop 2022-06-27T11:59:09.880 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.858+0000 7f3a0d378700 1 -- 172.21.15.195:0/969944604 shutdown_connections 2022-06-27T11:59:09.890 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.858+0000 7f3a0d378700 1 --2- 172.21.15.195:0/969944604 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f39e8063d00 0x7f39e80661b0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:09.890 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.858+0000 7f3a0d378700 1 --2- 172.21.15.195:0/969944604 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3a080c6190 0x7f3a08152ab0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:09.891 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.858+0000 7f3a0d378700 1 --2- 172.21.15.195:0/969944604 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3a08067ed0 0x7f3a08166fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:09.891 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.858+0000 7f3a0d378700 1 --2- 172.21.15.195:0/969944604 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3a080c40d0 0x7f3a08152570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:09.892 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.858+0000 7f3a0d378700 1 -- 172.21.15.195:0/969944604 >> 172.21.15.195:0/969944604 conn(0x7f3a080ccb80 msgr2=0x7f3a080c9600 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:59:09.892 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.858+0000 7f3a0d378700 1 -- 172.21.15.195:0/969944604 shutdown_connections 2022-06-27T11:59:09.892 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:09.858+0000 7f3a0d378700 1 -- 172.21.15.195:0/969944604 wait complete. 2022-06-27T11:59:09.897 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 44 2022-06-27T11:59:09.899 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T11:59:11.900 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:59:12.103 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.056+0000 7ff29f174700 1 Processor -- start 2022-06-27T11:59:12.104 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.056+0000 7ff29f174700 1 -- start start 2022-06-27T11:59:12.104 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.057+0000 7ff29f174700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7ff2980c5000 0x7ff2980c2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:12.104 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.057+0000 7ff29f174700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2980bc490 con 0x7ff2980c61e0 2022-06-27T11:59:12.105 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.057+0000 7ff29f174700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff29806a0e0 con 0x7ff2980c2bf0 2022-06-27T11:59:12.105 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.057+0000 7ff29f174700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2980bc6e0 con 0x7ff2980c5000 2022-06-27T11:59:12.106 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.058+0000 7ff29d711700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7ff2980c61e0 0x7ff2980c3900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:52170/0 (socket says 172.21.15.195:52170) 2022-06-27T11:59:12.108 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.058+0000 7ff29d711700 1 -- 172.21.15.195:0/3120577422 learned_addr learned my addr 172.21.15.195:0/3120577422 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:59:12.108 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.058+0000 7ff28ffff700 1 --2- 172.21.15.195:0/3120577422 >> v2:172.21.15.195:3300/0 conn(0x7ff2980c5000 0x7ff2980c2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:12.109 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.058+0000 7ff28f7fe700 1 -- 172.21.15.195:0/3120577422 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3744089090 0 0) 0x7ff2980bc490 con 0x7ff2980c61e0 2022-06-27T11:59:12.109 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.058+0000 7ff28f7fe700 1 -- 172.21.15.195:0/3120577422 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff284003430 con 0x7ff2980c61e0 2022-06-27T11:59:12.110 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.058+0000 7ff28ffff700 1 -- 172.21.15.195:0/3120577422 >> v1:172.21.15.202:6789/0 conn(0x7ff2980c2bf0 legacy=0x7ff2980c48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:12.110 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.058+0000 7ff28ffff700 1 -- 172.21.15.195:0/3120577422 >> v1:172.21.15.195:6789/0 conn(0x7ff2980c61e0 legacy=0x7ff2980c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:12.112 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.058+0000 7ff28ffff700 1 -- 172.21.15.195:0/3120577422 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff298158220 con 0x7ff2980c5000 2022-06-27T11:59:12.112 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.058+0000 7ff28ffff700 1 --2- 172.21.15.195:0/3120577422 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff2980c5000 0x7ff2980c2fc0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7ff2880097a0 tx=0x7ff288009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19284641e1987089 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:59:12.112 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.058+0000 7ff28f7fe700 1 -- 172.21.15.195:0/3120577422 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff288011630 con 0x7ff2980c5000 2022-06-27T11:59:12.113 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.058+0000 7ff28f7fe700 1 -- 172.21.15.195:0/3120577422 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff288011790 con 0x7ff2980c5000 2022-06-27T11:59:12.113 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.058+0000 7ff28f7fe700 1 -- 172.21.15.195:0/3120577422 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff288011b50 con 0x7ff2980c5000 2022-06-27T11:59:12.113 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.059+0000 7ff29f174700 1 -- 172.21.15.195:0/3120577422 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff2980c5000 msgr2=0x7ff2980c2fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:12.113 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.059+0000 7ff29f174700 1 --2- 172.21.15.195:0/3120577422 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff2980c5000 0x7ff2980c2fc0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7ff2880097a0 tx=0x7ff288009060 comp rx=0 tx=0).stop 2022-06-27T11:59:12.114 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.059+0000 7ff29f174700 1 -- 172.21.15.195:0/3120577422 shutdown_connections 2022-06-27T11:59:12.115 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.059+0000 7ff29f174700 1 --2- 172.21.15.195:0/3120577422 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff2980c5000 0x7ff2980c2fc0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:12.115 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.059+0000 7ff29f174700 1 -- 172.21.15.195:0/3120577422 >> 172.21.15.195:0/3120577422 conn(0x7ff2980c9b00 msgr2=0x7ff2980c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:59:12.115 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.059+0000 7ff29f174700 1 -- 172.21.15.195:0/3120577422 shutdown_connections 2022-06-27T11:59:12.116 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.059+0000 7ff29f174700 1 -- 172.21.15.195:0/3120577422 wait complete. 2022-06-27T11:59:12.116 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.060+0000 7ff29f174700 1 Processor -- start 2022-06-27T11:59:12.116 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.061+0000 7ff29f174700 1 -- start start 2022-06-27T11:59:12.117 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.061+0000 7ff29f174700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff2980c2bf0 0x7ff29815a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:12.117 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.061+0000 7ff29f174700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff2980c5000 0x7ff29814a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:12.118 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.061+0000 7ff29f174700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff2980c61e0 0x7ff298163ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:12.118 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.061+0000 7ff29f174700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff298159330 con 0x7ff2980c5000 2022-06-27T11:59:12.119 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.061+0000 7ff29f174700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2981471b0 con 0x7ff2980c2bf0 2022-06-27T11:59:12.119 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.061+0000 7ff29f174700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff298158fe0 con 0x7ff2980c61e0 2022-06-27T11:59:12.119 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.062+0000 7ff28ffff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff2980c5000 0x7ff29814a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:12.120 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.062+0000 7ff28ffff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff2980c5000 0x7ff29814a4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45986/0 (socket says 172.21.15.195:45986) 2022-06-27T11:59:12.120 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.062+0000 7ff28ffff700 1 -- 172.21.15.195:0/1117460988 learned_addr learned my addr 172.21.15.195:0/1117460988 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:59:12.121 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.062+0000 7ff29cf10700 1 --2- 172.21.15.195:0/1117460988 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff2980c2bf0 0x7ff29815a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:12.121 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.062+0000 7ff29d711700 1 --2- 172.21.15.195:0/1117460988 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff2980c61e0 0x7ff298163ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:12.121 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.062+0000 7ff28ffff700 1 -- 172.21.15.195:0/1117460988 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff2980c61e0 msgr2=0x7ff298163ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:12.122 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.062+0000 7ff28ffff700 1 --2- 172.21.15.195:0/1117460988 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff2980c61e0 0x7ff298163ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:12.122 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.062+0000 7ff28ffff700 1 -- 172.21.15.195:0/1117460988 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff2980c2bf0 msgr2=0x7ff29815a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:12.122 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.062+0000 7ff28ffff700 1 --2- 172.21.15.195:0/1117460988 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff2980c2bf0 0x7ff29815a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:12.123 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.062+0000 7ff28ffff700 1 -- 172.21.15.195:0/1117460988 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2880092e0 con 0x7ff2980c5000 2022-06-27T11:59:12.123 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.062+0000 7ff29d711700 1 --2- 172.21.15.195:0/1117460988 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff2980c61e0 0x7ff298163ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T11:59:12.124 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.062+0000 7ff28ffff700 1 --2- 172.21.15.195:0/1117460988 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff2980c5000 0x7ff29814a4a0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7ff29806f810 tx=0x7ff288017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:59:12.124 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.062+0000 7ff28cff9700 1 -- 172.21.15.195:0/1117460988 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff288011b10 con 0x7ff2980c5000 2022-06-27T11:59:12.125 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.062+0000 7ff28cff9700 1 -- 172.21.15.195:0/1117460988 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff28801be00 con 0x7ff2980c5000 2022-06-27T11:59:12.125 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.062+0000 7ff29f174700 1 -- 172.21.15.195:0/1117460988 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff29815a570 con 0x7ff2980c5000 2022-06-27T11:59:12.126 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.062+0000 7ff28cff9700 1 -- 172.21.15.195:0/1117460988 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff288023640 con 0x7ff2980c5000 2022-06-27T11:59:12.126 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.062+0000 7ff29f174700 1 -- 172.21.15.195:0/1117460988 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff29814add0 con 0x7ff2980c5000 2022-06-27T11:59:12.127 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.063+0000 7ff29f174700 1 -- 172.21.15.195:0/1117460988 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff298159470 con 0x7ff2980c5000 2022-06-27T11:59:12.127 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.064+0000 7ff28cff9700 1 -- 172.21.15.195:0/1117460988 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7ff28802a030 con 0x7ff2980c5000 2022-06-27T11:59:12.127 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.064+0000 7ff28cff9700 1 --2- 172.21.15.195:0/1117460988 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff278063d00 0x7ff2780661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:12.128 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.064+0000 7ff28cff9700 1 -- 172.21.15.195:0/1117460988 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6718+0+0 (secure 0 0 0) 0x7ff288081e10 con 0x7ff2980c5000 2022-06-27T11:59:12.128 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.064+0000 7ff29cf10700 1 --2- 172.21.15.195:0/1117460988 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff278063d00 0x7ff2780661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:12.129 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.064+0000 7ff29cf10700 1 --2- 172.21.15.195:0/1117460988 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff278063d00 0x7ff2780661b0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ff294001870 tx=0x7ff294009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:59:12.129 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.068+0000 7ff28cff9700 1 -- 172.21.15.195:0/1117460988 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7ff298159470 con 0x7ff2980c5000 2022-06-27T11:59:12.289 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.289+0000 7ff29f174700 1 -- 172.21.15.195:0/1117460988 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff298159470 con 0x7ff2980c5000 2022-06-27T11:59:12.290 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.289+0000 7ff28cff9700 1 -- 172.21.15.195:0/1117460988 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+5368 (secure 0 0 0) 0x7ff298159470 con 0x7ff2980c5000 2022-06-27T11:59:12.291 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:59:12.292 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:59:11.050208+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5192},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5192":{"gid":5192,"name":"b","rank":0,"incarnation":43,"state":"up:rejoin","state_seq":10,"addr":"172.21.15.202:6833/2651813808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":2651813808},{"type":"v1","addr":"172.21.15.202:6833","nonce":2651813808}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:59:12.295 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.291+0000 7ff29f174700 1 -- 172.21.15.195:0/1117460988 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff278063d00 msgr2=0x7ff2780661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:12.295 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.291+0000 7ff29f174700 1 --2- 172.21.15.195:0/1117460988 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff278063d00 0x7ff2780661b0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ff294001870 tx=0x7ff294009040 comp rx=0 tx=0).stop 2022-06-27T11:59:12.296 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.291+0000 7ff29f174700 1 -- 172.21.15.195:0/1117460988 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff2980c5000 msgr2=0x7ff29814a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:12.297 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.291+0000 7ff29f174700 1 --2- 172.21.15.195:0/1117460988 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff2980c5000 0x7ff29814a4a0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7ff29806f810 tx=0x7ff288017000 comp rx=0 tx=0).stop 2022-06-27T11:59:12.297 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.291+0000 7ff29f174700 1 -- 172.21.15.195:0/1117460988 shutdown_connections 2022-06-27T11:59:12.298 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.291+0000 7ff29f174700 1 --2- 172.21.15.195:0/1117460988 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff278063d00 0x7ff2780661b0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:12.298 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.291+0000 7ff29f174700 1 --2- 172.21.15.195:0/1117460988 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff2980c61e0 0x7ff298163ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:12.299 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.291+0000 7ff29f174700 1 --2- 172.21.15.195:0/1117460988 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff2980c2bf0 0x7ff29815a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:12.299 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.291+0000 7ff29f174700 1 --2- 172.21.15.195:0/1117460988 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff2980c5000 0x7ff29814a4a0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:12.300 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.291+0000 7ff29f174700 1 -- 172.21.15.195:0/1117460988 >> 172.21.15.195:0/1117460988 conn(0x7ff2980c9b00 msgr2=0x7ff298063af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:59:12.300 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.291+0000 7ff29f174700 1 -- 172.21.15.195:0/1117460988 shutdown_connections 2022-06-27T11:59:12.301 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:12.291+0000 7ff29f174700 1 -- 172.21.15.195:0/1117460988 wait complete. 2022-06-27T11:59:12.301 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 45 2022-06-27T11:59:12.304 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T11:59:14.306 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:59:14.439 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.439+0000 7fc1482b0700 1 Processor -- start 2022-06-27T11:59:14.440 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.439+0000 7fc1482b0700 1 -- start start 2022-06-27T11:59:14.444 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.440+0000 7fc1482b0700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fc14014e860 0x7fc1400c3840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:14.444 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.440+0000 7fc1482b0700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc1400c3d80 0x7fc140157890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:14.444 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.440+0000 7fc1482b0700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1400c33b0 con 0x7fc14014e490 2022-06-27T11:59:14.445 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.440+0000 7fc1482b0700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc14006f220 con 0x7fc14014e860 2022-06-27T11:59:14.445 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.440+0000 7fc1482b0700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1401542c0 con 0x7fc1400c3d80 2022-06-27T11:59:14.445 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.440+0000 7fc14684d700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc1400c3d80 0x7fc140157890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:14.446 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.440+0000 7fc14584b700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fc14014e860 0x7fc1400c3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:14.447 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.440+0000 7fc14604c700 1 --1- >> v1:172.21.15.202:6789/0 conn(0x7fc14014e490 0x7fc14006cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.202:6789/0 says I am v1:172.21.15.195:54044/0 (socket says 172.21.15.195:54044) 2022-06-27T11:59:14.447 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.440+0000 7fc14684d700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc1400c3d80 0x7fc140157890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3301/0 says I am v2:172.21.15.195:38756/0 (socket says 172.21.15.195:38756) 2022-06-27T11:59:14.447 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.440+0000 7fc14584b700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fc14014e860 0x7fc1400c3840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3300/0 says I am v2:172.21.15.195:44124/0 (socket says 172.21.15.195:44124) 2022-06-27T11:59:14.447 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.440+0000 7fc14604c700 1 -- 172.21.15.195:0/327555527 learned_addr learned my addr 172.21.15.195:0/327555527 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:59:14.448 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.441+0000 7fc14504a700 1 -- 172.21.15.195:0/327555527 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1397897935 0 0) 0x7fc1400c33b0 con 0x7fc14014e490 2022-06-27T11:59:14.448 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.441+0000 7fc14504a700 1 -- 172.21.15.195:0/327555527 --> v1:172.21.15.202:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc128003430 con 0x7fc14014e490 2022-06-27T11:59:14.448 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.441+0000 7fc14684d700 1 -- 172.21.15.195:0/327555527 >> v2:172.21.15.202:3300/0 conn(0x7fc14014e860 msgr2=0x7fc1400c3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:59:14.448 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.441+0000 7fc14684d700 1 --2- 172.21.15.195:0/327555527 >> v2:172.21.15.202:3300/0 conn(0x7fc14014e860 0x7fc1400c3840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:14.449 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.441+0000 7fc14684d700 1 -- 172.21.15.195:0/327555527 >> v1:172.21.15.202:6789/0 conn(0x7fc14014e490 legacy=0x7fc14006cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:14.449 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.441+0000 7fc14684d700 1 -- 172.21.15.195:0/327555527 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc140154560 con 0x7fc1400c3d80 2022-06-27T11:59:14.449 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.441+0000 7fc14584b700 1 --2- 172.21.15.195:0/327555527 >> v2:172.21.15.202:3300/0 conn(0x7fc14014e860 0x7fc1400c3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:59:14.449 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.441+0000 7fc14684d700 1 --2- 172.21.15.195:0/327555527 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc1400c3d80 0x7fc140157890 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc138009e60 tx=0x7fc138009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ebd1fb619e5952b6 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:59:14.451 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.441+0000 7fc14504a700 1 -- 172.21.15.195:0/327555527 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc13800e070 con 0x7fc1400c3d80 2022-06-27T11:59:14.451 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.441+0000 7fc14504a700 1 -- 172.21.15.195:0/327555527 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc13800bab0 con 0x7fc1400c3d80 2022-06-27T11:59:14.451 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.441+0000 7fc14504a700 1 -- 172.21.15.195:0/327555527 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc13800be30 con 0x7fc1400c3d80 2022-06-27T11:59:14.451 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.442+0000 7fc1482b0700 1 -- 172.21.15.195:0/327555527 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc1400c3d80 msgr2=0x7fc140157890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:14.452 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.442+0000 7fc1482b0700 1 --2- 172.21.15.195:0/327555527 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc1400c3d80 0x7fc140157890 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc138009e60 tx=0x7fc138009780 comp rx=0 tx=0).stop 2022-06-27T11:59:14.452 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.442+0000 7fc1482b0700 1 -- 172.21.15.195:0/327555527 shutdown_connections 2022-06-27T11:59:14.452 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.442+0000 7fc1482b0700 1 --2- 172.21.15.195:0/327555527 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc1400c3d80 0x7fc140157890 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:14.452 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.442+0000 7fc1482b0700 1 --2- 172.21.15.195:0/327555527 >> v2:172.21.15.202:3300/0 conn(0x7fc14014e860 0x7fc1400c3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:14.452 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.442+0000 7fc1482b0700 1 -- 172.21.15.195:0/327555527 >> 172.21.15.195:0/327555527 conn(0x7fc140063af0 msgr2=0x7fc140063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:59:14.453 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.442+0000 7fc1482b0700 1 -- 172.21.15.195:0/327555527 shutdown_connections 2022-06-27T11:59:14.453 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.442+0000 7fc1482b0700 1 -- 172.21.15.195:0/327555527 wait complete. 2022-06-27T11:59:14.456 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.443+0000 7fc1482b0700 1 Processor -- start 2022-06-27T11:59:14.457 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.443+0000 7fc1482b0700 1 -- start start 2022-06-27T11:59:14.457 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.444+0000 7fc1482b0700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc1400c3d80 0x7fc14014baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:14.457 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.444+0000 7fc1482b0700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc14014e490 0x7fc140068fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:14.458 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.444+0000 7fc1482b0700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc14014e860 0x7fc1400c8880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:14.458 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.444+0000 7fc1482b0700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc14014a450 con 0x7fc14014e860 2022-06-27T11:59:14.460 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.444+0000 7fc1482b0700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1400c8360 con 0x7fc14014e490 2022-06-27T11:59:14.460 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.444+0000 7fc1482b0700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc14014a710 con 0x7fc1400c3d80 2022-06-27T11:59:14.461 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.444+0000 7fc14684d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc14014e860 0x7fc1400c8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:14.461 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.444+0000 7fc14584b700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc14014e490 0x7fc140068fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:14.461 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.444+0000 7fc14684d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc14014e860 0x7fc1400c8880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:46006/0 (socket says 172.21.15.195:46006) 2022-06-27T11:59:14.462 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.444+0000 7fc14604c700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc1400c3d80 0x7fc14014baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:14.462 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.444+0000 7fc14684d700 1 -- 172.21.15.195:0/1535775105 learned_addr learned my addr 172.21.15.195:0/1535775105 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:59:14.464 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.444+0000 7fc14684d700 1 -- 172.21.15.195:0/1535775105 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc1400c3d80 msgr2=0x7fc14014baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:14.464 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.444+0000 7fc14684d700 1 --2- 172.21.15.195:0/1535775105 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc1400c3d80 0x7fc14014baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:14.464 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.444+0000 7fc14684d700 1 -- 172.21.15.195:0/1535775105 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc14014e490 msgr2=0x7fc140068fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:14.465 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.444+0000 7fc14684d700 1 --2- 172.21.15.195:0/1535775105 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc14014e490 0x7fc140068fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:14.465 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.444+0000 7fc14684d700 1 -- 172.21.15.195:0/1535775105 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc13c0092e0 con 0x7fc14014e860 2022-06-27T11:59:14.466 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.444+0000 7fc14684d700 1 --2- 172.21.15.195:0/1535775105 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc14014e860 0x7fc1400c8880 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fc1400be550 tx=0x7fc138019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:59:14.466 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.444+0000 7fc12e7fc700 1 -- 172.21.15.195:0/1535775105 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc138003e40 con 0x7fc14014e860 2022-06-27T11:59:14.467 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.444+0000 7fc12e7fc700 1 -- 172.21.15.195:0/1535775105 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc138012850 con 0x7fc14014e860 2022-06-27T11:59:14.467 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.444+0000 7fc1482b0700 1 -- 172.21.15.195:0/1535775105 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc138009a20 con 0x7fc14014e860 2022-06-27T11:59:14.467 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.445+0000 7fc12e7fc700 1 -- 172.21.15.195:0/1535775105 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc138012c10 con 0x7fc14014e860 2022-06-27T11:59:14.468 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.445+0000 7fc1482b0700 1 -- 172.21.15.195:0/1535775105 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1400c7b60 con 0x7fc14014e860 2022-06-27T11:59:14.468 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.445+0000 7fc1482b0700 1 -- 172.21.15.195:0/1535775105 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc140149e80 con 0x7fc14014e860 2022-06-27T11:59:14.469 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.446+0000 7fc12e7fc700 1 -- 172.21.15.195:0/1535775105 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fc138004040 con 0x7fc14014e860 2022-06-27T11:59:14.469 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.446+0000 7fc12e7fc700 1 --2- 172.21.15.195:0/1535775105 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc120063d30 0x7fc1200661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:14.470 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.446+0000 7fc12e7fc700 1 -- 172.21.15.195:0/1535775105 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6718+0+0 (secure 0 0 0) 0x7fc138083320 con 0x7fc14014e860 2022-06-27T11:59:14.471 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.446+0000 7fc14604c700 1 --2- 172.21.15.195:0/1535775105 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc120063d30 0x7fc1200661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:14.471 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.447+0000 7fc14604c700 1 --2- 172.21.15.195:0/1535775105 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc120063d30 0x7fc1200661e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fc130005790 tx=0x7fc130009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:59:14.472 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.450+0000 7fc12e7fc700 1 -- 172.21.15.195:0/1535775105 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fc140149e80 con 0x7fc14014e860 2022-06-27T11:59:14.679 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.679+0000 7fc1482b0700 1 -- 172.21.15.195:0/1535775105 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc140149e80 con 0x7fc14014e860 2022-06-27T11:59:14.679 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.679+0000 7fc12e7fc700 1 -- 172.21.15.195:0/1535775105 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+5368 (secure 0 0 0) 0x7fc140149e80 con 0x7fc14014e860 2022-06-27T11:59:14.680 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:59:14.680 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:59:11.050208+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5192},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5192":{"gid":5192,"name":"b","rank":0,"incarnation":43,"state":"up:rejoin","state_seq":10,"addr":"172.21.15.202:6833/2651813808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":2651813808},{"type":"v1","addr":"172.21.15.202:6833","nonce":2651813808}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:59:14.682 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.681+0000 7fc1482b0700 1 -- 172.21.15.195:0/1535775105 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc120063d30 msgr2=0x7fc1200661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:14.682 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.681+0000 7fc1482b0700 1 --2- 172.21.15.195:0/1535775105 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc120063d30 0x7fc1200661e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fc130005790 tx=0x7fc130009040 comp rx=0 tx=0).stop 2022-06-27T11:59:14.682 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.681+0000 7fc1482b0700 1 -- 172.21.15.195:0/1535775105 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc14014e860 msgr2=0x7fc1400c8880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:14.683 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.681+0000 7fc1482b0700 1 --2- 172.21.15.195:0/1535775105 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc14014e860 0x7fc1400c8880 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fc1400be550 tx=0x7fc138019000 comp rx=0 tx=0).stop 2022-06-27T11:59:14.683 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.681+0000 7fc1482b0700 1 -- 172.21.15.195:0/1535775105 shutdown_connections 2022-06-27T11:59:14.684 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.681+0000 7fc1482b0700 1 --2- 172.21.15.195:0/1535775105 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc120063d30 0x7fc1200661e0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:14.684 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.681+0000 7fc1482b0700 1 --2- 172.21.15.195:0/1535775105 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc14014e860 0x7fc1400c8880 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:14.684 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.681+0000 7fc1482b0700 1 --2- 172.21.15.195:0/1535775105 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc1400c3d80 0x7fc14014baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:14.685 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.681+0000 7fc1482b0700 1 --2- 172.21.15.195:0/1535775105 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc14014e490 0x7fc140068fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:14.685 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.681+0000 7fc1482b0700 1 -- 172.21.15.195:0/1535775105 >> 172.21.15.195:0/1535775105 conn(0x7fc140063af0 msgr2=0x7fc14006cfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:59:14.685 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.681+0000 7fc1482b0700 1 -- 172.21.15.195:0/1535775105 shutdown_connections 2022-06-27T11:59:14.686 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:14.681+0000 7fc1482b0700 1 -- 172.21.15.195:0/1535775105 wait complete. 2022-06-27T11:59:14.686 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 45 2022-06-27T11:59:14.700 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T11:59:16.708 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:59:16.845 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:16.842+0000 7fe292ec7700 1 Processor -- start 2022-06-27T11:59:16.845 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:16.843+0000 7fe292ec7700 1 -- start start 2022-06-27T11:59:16.846 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:16.843+0000 7fe292ec7700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe28c06f640 0x7fe28c156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:16.846 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:16.843+0000 7fe292ec7700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fe28c0c39e0 0x7fe28c1535a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:16.846 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:16.843+0000 7fe292ec7700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe28c0c1a20 con 0x7fe28c0c4160 2022-06-27T11:59:16.846 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:16.843+0000 7fe292ec7700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe28c0c1130 con 0x7fe28c0c39e0 2022-06-27T11:59:16.847 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:16.843+0000 7fe292ec7700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe28c06ea90 con 0x7fe28c06f640 2022-06-27T11:59:16.848 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:16.844+0000 7fe291464700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fe28c0c39e0 0x7fe28c1535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:16.848 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:16.844+0000 7fe291464700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fe28c0c39e0 0x7fe28c1535a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:46014/0 (socket says 172.21.15.195:46014) 2022-06-27T11:59:16.849 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:16.844+0000 7fe291464700 1 -- 172.21.15.195:0/2474335174 learned_addr learned my addr 172.21.15.195:0/2474335174 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:59:16.850 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:16.844+0000 7fe290c63700 1 --2- 172.21.15.195:0/2474335174 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe28c06f640 0x7fe28c156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:16.850 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:16.844+0000 7fe291464700 1 -- 172.21.15.195:0/2474335174 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe28c06f640 msgr2=0x7fe28c156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:59:16.851 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:16.844+0000 7fe291464700 1 --2- 172.21.15.195:0/2474335174 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe28c06f640 0x7fe28c156ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:16.852 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:16.844+0000 7fe291464700 1 -- 172.21.15.195:0/2474335174 >> v1:172.21.15.202:6789/0 conn(0x7fe28c0c4160 legacy=0x7fe28c0c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:16.852 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:16.844+0000 7fe291464700 1 -- 172.21.15.195:0/2474335174 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe28c06ed70 con 0x7fe28c0c39e0 2022-06-27T11:59:16.853 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:16.844+0000 7fe291464700 1 --2- 172.21.15.195:0/2474335174 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe28c0c39e0 0x7fe28c1535a0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fe284009f20 tx=0x7fe284009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9eec4828d96f4d8 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:59:16.853 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:16.844+0000 7fe28b7fe700 1 -- 172.21.15.195:0/2474335174 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe28400e070 con 0x7fe28c0c39e0 2022-06-27T11:59:16.854 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:16.844+0000 7fe28b7fe700 1 -- 172.21.15.195:0/2474335174 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe28400ba60 con 0x7fe28c0c39e0 2022-06-27T11:59:16.854 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:16.844+0000 7fe28b7fe700 1 -- 172.21.15.195:0/2474335174 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2840092e0 con 0x7fe28c0c39e0 2022-06-27T11:59:16.855 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:16.845+0000 7fe292ec7700 1 -- 172.21.15.195:0/2474335174 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe28c0c39e0 msgr2=0x7fe28c1535a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:16.857 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:16.845+0000 7fe292ec7700 1 --2- 172.21.15.195:0/2474335174 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe28c0c39e0 0x7fe28c1535a0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fe284009f20 tx=0x7fe284009840 comp rx=0 tx=0).stop 2022-06-27T11:59:16.857 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:16.845+0000 7fe292ec7700 1 -- 172.21.15.195:0/2474335174 shutdown_connections 2022-06-27T11:59:16.857 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:16.845+0000 7fe292ec7700 1 --2- 172.21.15.195:0/2474335174 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe28c0c39e0 0x7fe28c1535a0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:16.857 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:16.845+0000 7fe292ec7700 1 --2- 172.21.15.195:0/2474335174 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe28c06f640 0x7fe28c156ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:16.858 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:16.845+0000 7fe292ec7700 1 -- 172.21.15.195:0/2474335174 >> 172.21.15.195:0/2474335174 conn(0x7fe28c0c6350 msgr2=0x7fe28c0c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:59:16.858 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:16.845+0000 7fe292ec7700 1 -- 172.21.15.195:0/2474335174 shutdown_connections 2022-06-27T11:59:16.858 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:16.845+0000 7fe292ec7700 1 -- 172.21.15.195:0/2474335174 wait complete. 2022-06-27T11:59:16.865 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:16.846+0000 7fe292ec7700 1 Processor -- start 2022-06-27T11:59:16.865 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:16.846+0000 7fe292ec7700 1 -- start start 2022-06-27T11:59:16.866 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:16.847+0000 7fe292ec7700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe28c06f640 0x7fe28c14f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:16.867 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:16.847+0000 7fe292ec7700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe28c0c39e0 0x7fe28c0ba330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:16.868 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:16.847+0000 7fe292ec7700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe28c0c4160 0x7fe28c1546e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:16.868 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:16.847+0000 7fe292ec7700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe28c0bc020 con 0x7fe28c0c4160 2022-06-27T11:59:16.869 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:16.847+0000 7fe292ec7700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe28c0bd030 con 0x7fe28c0c39e0 2022-06-27T11:59:16.869 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:16.847+0000 7fe292ec7700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe28c156230 con 0x7fe28c06f640 2022-06-27T11:59:16.869 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:16.847+0000 7fe291464700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe28c0c4160 0x7fe28c1546e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:16.870 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:16.847+0000 7fe291464700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe28c0c4160 0x7fe28c1546e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:46020/0 (socket says 172.21.15.195:46020) 2022-06-27T11:59:16.870 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:16.847+0000 7fe291464700 1 -- 172.21.15.195:0/1793453084 learned_addr learned my addr 172.21.15.195:0/1793453084 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:59:16.871 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:16.847+0000 7fe290c63700 1 --2- 172.21.15.195:0/1793453084 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe28c06f640 0x7fe28c14f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:16.871 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:16.847+0000 7fe28bfff700 1 --2- 172.21.15.195:0/1793453084 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe28c0c39e0 0x7fe28c0ba330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:16.872 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:16.847+0000 7fe291464700 1 -- 172.21.15.195:0/1793453084 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe28c06f640 msgr2=0x7fe28c14f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:16.872 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:16.847+0000 7fe291464700 1 --2- 172.21.15.195:0/1793453084 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe28c06f640 0x7fe28c14f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:16.873 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:16.847+0000 7fe291464700 1 -- 172.21.15.195:0/1793453084 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe28c0c39e0 msgr2=0x7fe28c0ba330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:16.874 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:16.847+0000 7fe291464700 1 --2- 172.21.15.195:0/1793453084 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe28c0c39e0 0x7fe28c0ba330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:16.874 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:16.847+0000 7fe291464700 1 -- 172.21.15.195:0/1793453084 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe284009ae0 con 0x7fe28c0c4160 2022-06-27T11:59:16.874 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:16.847+0000 7fe291464700 1 --2- 172.21.15.195:0/1793453084 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe28c0c4160 0x7fe28c1546e0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fe28c0650f0 tx=0x7fe284018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:59:16.874 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:16.848+0000 7fe288ff9700 1 -- 172.21.15.195:0/1793453084 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2840092e0 con 0x7fe28c0c4160 2022-06-27T11:59:16.875 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:16.848+0000 7fe288ff9700 1 -- 172.21.15.195:0/1793453084 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe284007610 con 0x7fe28c0c4160 2022-06-27T11:59:16.875 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:16.848+0000 7fe288ff9700 1 -- 172.21.15.195:0/1793453084 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2840079d0 con 0x7fe28c0c4160 2022-06-27T11:59:16.875 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:16.848+0000 7fe292ec7700 1 -- 172.21.15.195:0/1793453084 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe28c154120 con 0x7fe28c0c4160 2022-06-27T11:59:16.876 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:16.848+0000 7fe292ec7700 1 -- 172.21.15.195:0/1793453084 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe28c157b70 con 0x7fe28c0c4160 2022-06-27T11:59:16.876 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:16.848+0000 7fe292ec7700 1 -- 172.21.15.195:0/1793453084 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe28c0bd1e0 con 0x7fe28c0c4160 2022-06-27T11:59:16.876 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:16.849+0000 7fe288ff9700 1 -- 172.21.15.195:0/1793453084 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fe284004040 con 0x7fe28c0c4160 2022-06-27T11:59:16.876 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:16.849+0000 7fe288ff9700 1 --2- 172.21.15.195:0/1793453084 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe274063d00 0x7fe2740661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:16.877 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:16.849+0000 7fe288ff9700 1 -- 172.21.15.195:0/1793453084 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6718+0+0 (secure 0 0 0) 0x7fe2840827a0 con 0x7fe28c0c4160 2022-06-27T11:59:16.877 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:16.849+0000 7fe290c63700 1 --2- 172.21.15.195:0/1793453084 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe274063d00 0x7fe2740661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:16.878 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:16.850+0000 7fe290c63700 1 --2- 172.21.15.195:0/1793453084 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe274063d00 0x7fe2740661b0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fe280009c70 tx=0x7fe280009060 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:59:16.878 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:16.854+0000 7fe288ff9700 1 -- 172.21.15.195:0/1793453084 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fe28c0bd1e0 con 0x7fe28c0c4160 2022-06-27T11:59:17.082 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:17.080+0000 7fe292ec7700 1 -- 172.21.15.195:0/1793453084 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe28c0bd1e0 con 0x7fe28c0c4160 2022-06-27T11:59:17.083 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:17.081+0000 7fe288ff9700 1 -- 172.21.15.195:0/1793453084 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+5374 (secure 0 0 0) 0x7fe28c0bd1e0 con 0x7fe28c0c4160 2022-06-27T11:59:17.084 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:59:17.084 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":46,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:59:17.069976+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5192},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5192":{"gid":5192,"name":"b","rank":0,"incarnation":43,"state":"up:clientreplay","state_seq":12,"addr":"172.21.15.202:6833/2651813808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":2651813808},{"type":"v1","addr":"172.21.15.202:6833","nonce":2651813808}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:59:17.092 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:17.083+0000 7fe292ec7700 1 -- 172.21.15.195:0/1793453084 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe274063d00 msgr2=0x7fe2740661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:17.093 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:17.083+0000 7fe292ec7700 1 --2- 172.21.15.195:0/1793453084 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe274063d00 0x7fe2740661b0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fe280009c70 tx=0x7fe280009060 comp rx=0 tx=0).stop 2022-06-27T11:59:17.093 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:17.083+0000 7fe292ec7700 1 -- 172.21.15.195:0/1793453084 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe28c0c4160 msgr2=0x7fe28c1546e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:17.096 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:17.083+0000 7fe292ec7700 1 --2- 172.21.15.195:0/1793453084 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe28c0c4160 0x7fe28c1546e0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fe28c0650f0 tx=0x7fe284018000 comp rx=0 tx=0).stop 2022-06-27T11:59:17.096 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:17.083+0000 7fe292ec7700 1 -- 172.21.15.195:0/1793453084 shutdown_connections 2022-06-27T11:59:17.097 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:17.083+0000 7fe292ec7700 1 --2- 172.21.15.195:0/1793453084 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe274063d00 0x7fe2740661b0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:17.097 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:17.083+0000 7fe292ec7700 1 --2- 172.21.15.195:0/1793453084 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe28c0c4160 0x7fe28c1546e0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:17.098 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:17.083+0000 7fe292ec7700 1 --2- 172.21.15.195:0/1793453084 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe28c06f640 0x7fe28c14f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:17.099 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:17.083+0000 7fe292ec7700 1 --2- 172.21.15.195:0/1793453084 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe28c0c39e0 0x7fe28c0ba330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:17.099 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:17.083+0000 7fe292ec7700 1 -- 172.21.15.195:0/1793453084 >> 172.21.15.195:0/1793453084 conn(0x7fe28c0c6350 msgr2=0x7fe28c153370 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:59:17.100 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:17.083+0000 7fe292ec7700 1 -- 172.21.15.195:0/1793453084 shutdown_connections 2022-06-27T11:59:17.100 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:17.083+0000 7fe292ec7700 1 -- 172.21.15.195:0/1793453084 wait complete. 2022-06-27T11:59:17.101 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 46 2022-06-27T11:59:17.104 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T11:59:19.105 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:59:19.245 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.245+0000 7f9fcbbac700 1 Processor -- start 2022-06-27T11:59:19.246 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.246+0000 7f9fcbbac700 1 -- start start 2022-06-27T11:59:19.258 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.246+0000 7f9fcbbac700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f9fc414e490 0x7f9fc406cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:19.259 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.246+0000 7f9fcbbac700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9fc414e860 0x7f9fc40c3840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:19.259 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.246+0000 7f9fcbbac700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f9fc40c3d80 0x7f9fc406f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:19.260 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.246+0000 7f9fcbbac700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fc40c33b0 con 0x7f9fc40c3d80 2022-06-27T11:59:19.260 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.246+0000 7f9fcbbac700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fc406d520 con 0x7f9fc414e490 2022-06-27T11:59:19.261 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.246+0000 7f9fcbbac700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fc406f220 con 0x7f9fc414e860 2022-06-27T11:59:19.262 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.247+0000 7f9fca149700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f9fc40c3d80 0x7f9fc406f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:19.262 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.247+0000 7f9fca149700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f9fc40c3d80 0x7f9fc406f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:46028/0 (socket says 172.21.15.195:46028) 2022-06-27T11:59:19.262 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.247+0000 7f9fca149700 1 -- 172.21.15.195:0/946201654 learned_addr learned my addr 172.21.15.195:0/946201654 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:59:19.287 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.247+0000 7f9fc9147700 1 --2- 172.21.15.195:0/946201654 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9fc414e860 0x7f9fc40c3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:19.287 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.247+0000 7f9fc9948700 1 --2- 172.21.15.195:0/946201654 >> v2:172.21.15.202:3300/0 conn(0x7f9fc414e490 0x7f9fc406cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:19.288 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.247+0000 7f9fca149700 1 -- 172.21.15.195:0/946201654 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9fc414e860 msgr2=0x7f9fc40c3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:59:19.288 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.247+0000 7f9fca149700 1 --2- 172.21.15.195:0/946201654 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9fc414e860 0x7f9fc40c3840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:19.288 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.247+0000 7f9fca149700 1 -- 172.21.15.195:0/946201654 >> v2:172.21.15.202:3300/0 conn(0x7f9fc414e490 msgr2=0x7f9fc406cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:59:19.289 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.247+0000 7f9fca149700 1 --2- 172.21.15.195:0/946201654 >> v2:172.21.15.202:3300/0 conn(0x7f9fc414e490 0x7f9fc406cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:19.289 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.247+0000 7f9fca149700 1 -- 172.21.15.195:0/946201654 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fc41542c0 con 0x7f9fc40c3d80 2022-06-27T11:59:19.291 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.247+0000 7f9fca149700 1 --2- 172.21.15.195:0/946201654 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9fc40c3d80 0x7f9fc406f620 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f9fc4064800 tx=0x7f9fbc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=892125c619eb9570 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:59:19.291 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.247+0000 7f9fc8946700 1 -- 172.21.15.195:0/946201654 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fbc00e070 con 0x7f9fc40c3d80 2022-06-27T11:59:19.292 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.247+0000 7f9fc8946700 1 -- 172.21.15.195:0/946201654 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fbc00ba50 con 0x7f9fc40c3d80 2022-06-27T11:59:19.292 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.247+0000 7f9fc8946700 1 -- 172.21.15.195:0/946201654 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fbc0092e0 con 0x7f9fc40c3d80 2022-06-27T11:59:19.300 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.248+0000 7f9fcbbac700 1 -- 172.21.15.195:0/946201654 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9fc40c3d80 msgr2=0x7f9fc406f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:19.301 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.248+0000 7f9fcbbac700 1 --2- 172.21.15.195:0/946201654 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9fc40c3d80 0x7f9fc406f620 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f9fc4064800 tx=0x7f9fbc009840 comp rx=0 tx=0).stop 2022-06-27T11:59:19.301 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.248+0000 7f9fcbbac700 1 -- 172.21.15.195:0/946201654 shutdown_connections 2022-06-27T11:59:19.302 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.248+0000 7f9fcbbac700 1 --2- 172.21.15.195:0/946201654 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9fc40c3d80 0x7f9fc406f620 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:19.302 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.248+0000 7f9fcbbac700 1 --2- 172.21.15.195:0/946201654 >> v2:172.21.15.202:3300/0 conn(0x7f9fc414e490 0x7f9fc406cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:19.303 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.248+0000 7f9fcbbac700 1 --2- 172.21.15.195:0/946201654 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9fc414e860 0x7f9fc40c3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:19.303 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.248+0000 7f9fcbbac700 1 -- 172.21.15.195:0/946201654 >> 172.21.15.195:0/946201654 conn(0x7f9fc4063af0 msgr2=0x7f9fc4063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:59:19.304 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.248+0000 7f9fcbbac700 1 -- 172.21.15.195:0/946201654 shutdown_connections 2022-06-27T11:59:19.307 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.248+0000 7f9fcbbac700 1 -- 172.21.15.195:0/946201654 wait complete. 2022-06-27T11:59:19.307 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.249+0000 7f9fcbbac700 1 Processor -- start 2022-06-27T11:59:19.308 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.249+0000 7f9fcbbac700 1 -- start start 2022-06-27T11:59:19.308 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.250+0000 7f9fcbbac700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9fc40c3d80 0x7f9fc4149da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:19.308 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.250+0000 7f9fcbbac700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9fc414e490 0x7f9fc414cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:19.309 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.250+0000 7f9fcbbac700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9fc414e860 0x7f9fc40692f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:19.309 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.250+0000 7f9fcbbac700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fc40c7e00 con 0x7f9fc414e490 2022-06-27T11:59:19.310 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.250+0000 7f9fcbbac700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fc414a2e0 con 0x7f9fc40c3d80 2022-06-27T11:59:19.310 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.250+0000 7f9fcbbac700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fc40c6f10 con 0x7f9fc414e860 2022-06-27T11:59:19.311 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.250+0000 7f9fc9147700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9fc414e490 0x7f9fc414cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:19.311 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.250+0000 7f9fc9147700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9fc414e490 0x7f9fc414cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:46032/0 (socket says 172.21.15.195:46032) 2022-06-27T11:59:19.311 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.250+0000 7f9fc9147700 1 -- 172.21.15.195:0/784534592 learned_addr learned my addr 172.21.15.195:0/784534592 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:59:19.312 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.250+0000 7f9fc9948700 1 --2- 172.21.15.195:0/784534592 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9fc40c3d80 0x7f9fc4149da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:19.312 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.250+0000 7f9fca149700 1 --2- 172.21.15.195:0/784534592 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9fc414e860 0x7f9fc40692f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:19.313 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.250+0000 7f9fc9147700 1 -- 172.21.15.195:0/784534592 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9fc414e860 msgr2=0x7f9fc40692f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:19.313 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.250+0000 7f9fc9147700 1 --2- 172.21.15.195:0/784534592 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9fc414e860 0x7f9fc40692f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:19.314 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.250+0000 7f9fc9147700 1 -- 172.21.15.195:0/784534592 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9fc40c3d80 msgr2=0x7f9fc4149da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:19.314 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.250+0000 7f9fc9147700 1 --2- 172.21.15.195:0/784534592 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9fc40c3d80 0x7f9fc4149da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:19.314 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.250+0000 7f9fc9147700 1 -- 172.21.15.195:0/784534592 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fbc009ae0 con 0x7f9fc414e490 2022-06-27T11:59:19.326 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.250+0000 7f9fc9147700 1 --2- 172.21.15.195:0/784534592 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9fc414e490 0x7f9fc414cdd0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f9fc000d860 tx=0x7f9fc0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:59:19.326 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.250+0000 7f9fb1ffb700 1 -- 172.21.15.195:0/784534592 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fc00125f0 con 0x7f9fc414e490 2022-06-27T11:59:19.326 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.250+0000 7f9fb1ffb700 1 -- 172.21.15.195:0/784534592 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fc0012750 con 0x7f9fc414e490 2022-06-27T11:59:19.327 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.250+0000 7f9fcbbac700 1 -- 172.21.15.195:0/784534592 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fc40c71f0 con 0x7f9fc414e490 2022-06-27T11:59:19.327 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.251+0000 7f9fb1ffb700 1 -- 172.21.15.195:0/784534592 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fc0012b10 con 0x7f9fc414e490 2022-06-27T11:59:19.327 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.251+0000 7f9fcbbac700 1 -- 172.21.15.195:0/784534592 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fc40c77b0 con 0x7f9fc414e490 2022-06-27T11:59:19.327 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.252+0000 7f9fcbbac700 1 -- 172.21.15.195:0/784534592 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fc40c7fb0 con 0x7f9fc414e490 2022-06-27T11:59:19.328 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.252+0000 7f9fb1ffb700 1 -- 172.21.15.195:0/784534592 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f9fc001e460 con 0x7f9fc414e490 2022-06-27T11:59:19.328 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.252+0000 7f9fb1ffb700 1 --2- 172.21.15.195:0/784534592 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9fa4063d00 0x7f9fa40661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:19.328 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.252+0000 7f9fb1ffb700 1 -- 172.21.15.195:0/784534592 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6718+0+0 (secure 0 0 0) 0x7f9fc0080fd0 con 0x7f9fc414e490 2022-06-27T11:59:19.328 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.252+0000 7f9fc9948700 1 --2- 172.21.15.195:0/784534592 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9fa4063d00 0x7f9fa40661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:19.328 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.253+0000 7f9fc9948700 1 --2- 172.21.15.195:0/784534592 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9fa4063d00 0x7f9fa40661b0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f9fb4004560 tx=0x7f9fb4008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:59:19.329 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.257+0000 7f9fb1ffb700 1 -- 172.21.15.195:0/784534592 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f9fc40c7fb0 con 0x7f9fc414e490 2022-06-27T11:59:19.484 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.481+0000 7f9fcbbac700 1 -- 172.21.15.195:0/784534592 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9fc40722c0 con 0x7f9fc414e490 2022-06-27T11:59:19.485 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.481+0000 7f9fb1ffb700 1 -- 172.21.15.195:0/784534592 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+5374 (secure 0 0 0) 0x7f9fc004f980 con 0x7f9fc414e490 2022-06-27T11:59:19.485 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:59:19.486 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":46,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:59:17.069976+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5192},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5192":{"gid":5192,"name":"b","rank":0,"incarnation":43,"state":"up:clientreplay","state_seq":12,"addr":"172.21.15.202:6833/2651813808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":2651813808},{"type":"v1","addr":"172.21.15.202:6833","nonce":2651813808}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:59:19.487 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.483+0000 7f9fcbbac700 1 -- 172.21.15.195:0/784534592 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9fa4063d00 msgr2=0x7f9fa40661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:19.487 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.483+0000 7f9fcbbac700 1 --2- 172.21.15.195:0/784534592 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9fa4063d00 0x7f9fa40661b0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f9fb4004560 tx=0x7f9fb4008040 comp rx=0 tx=0).stop 2022-06-27T11:59:19.487 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.483+0000 7f9fcbbac700 1 -- 172.21.15.195:0/784534592 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9fc414e490 msgr2=0x7f9fc414cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:19.488 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.483+0000 7f9fcbbac700 1 --2- 172.21.15.195:0/784534592 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9fc414e490 0x7f9fc414cdd0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f9fc000d860 tx=0x7f9fc0008040 comp rx=0 tx=0).stop 2022-06-27T11:59:19.488 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.483+0000 7f9fcbbac700 1 -- 172.21.15.195:0/784534592 shutdown_connections 2022-06-27T11:59:19.488 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.483+0000 7f9fcbbac700 1 --2- 172.21.15.195:0/784534592 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9fa4063d00 0x7f9fa40661b0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:19.488 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.483+0000 7f9fcbbac700 1 --2- 172.21.15.195:0/784534592 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9fc414e860 0x7f9fc40692f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:19.489 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.483+0000 7f9fcbbac700 1 --2- 172.21.15.195:0/784534592 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9fc40c3d80 0x7f9fc4149da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:19.490 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.483+0000 7f9fcbbac700 1 --2- 172.21.15.195:0/784534592 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9fc414e490 0x7f9fc414cdd0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:19.490 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.483+0000 7f9fcbbac700 1 -- 172.21.15.195:0/784534592 >> 172.21.15.195:0/784534592 conn(0x7f9fc4063af0 msgr2=0x7f9fc40c59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:59:19.490 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.483+0000 7f9fcbbac700 1 -- 172.21.15.195:0/784534592 shutdown_connections 2022-06-27T11:59:19.490 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:19.483+0000 7f9fcbbac700 1 -- 172.21.15.195:0/784534592 wait complete. 2022-06-27T11:59:19.491 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 46 2022-06-27T11:59:19.496 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 46, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-06-27T11:47:52.358095+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 46, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5192": { "addr": "172.21.15.202:6833/2651813808", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6832", "nonce": 2651813808, "type": "v2" }, { "addr": "172.21.15.202:6833", "nonce": 2651813808, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5192, "incarnation": 43, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:clientreplay", "state_seq": 12 } }, "last_failure": 0, "last_failure_osd_epoch": 54, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-06-27T11:59:17.069976+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5192 } } } ], "standbys": [ { "addr": "172.21.15.202:6837/3305006841", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6836", "nonce": 3305006841, "type": "v2" }, { "addr": "172.21.15.202:6837", "nonce": 3305006841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6835/2060709822", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6834", "nonce": 2060709822, "type": "v2" }, { "addr": "172.21.15.195:6835", "nonce": 2060709822, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6838/1598751652", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6836", "nonce": 1598751652, "type": "v2" }, { "addr": "172.21.15.195:6838", "nonce": 1598751652, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4275, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6839/1313800661", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6837", "nonce": 1313800661, "type": "v2" }, { "addr": "172.21.15.195:6839", "nonce": 1313800661, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4278, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-06-27T11:59:21.498 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:59:21.640 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.637+0000 7feddee7a700 1 Processor -- start 2022-06-27T11:59:21.641 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.637+0000 7feddee7a700 1 -- start start 2022-06-27T11:59:21.641 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.638+0000 7feddee7a700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fedd80c4160 0x7fedd8156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:21.641 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.638+0000 7feddee7a700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fedd80c39e0 0x7fedd81535a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:21.641 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.638+0000 7feddee7a700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedd80c1a20 con 0x7fedd806f640 2022-06-27T11:59:21.642 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.638+0000 7feddee7a700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedd80c1130 con 0x7fedd80c4160 2022-06-27T11:59:21.642 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.638+0000 7feddee7a700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedd806ea90 con 0x7fedd80c39e0 2022-06-27T11:59:21.642 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.638+0000 7fedd7fff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fedd80c4160 0x7fedd8156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:21.642 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.638+0000 7fedd7fff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fedd80c4160 0x7fedd8156ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:46040/0 (socket says 172.21.15.195:46040) 2022-06-27T11:59:21.643 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.638+0000 7fedd7fff700 1 -- 172.21.15.195:0/1268758209 learned_addr learned my addr 172.21.15.195:0/1268758209 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:59:21.643 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.638+0000 7feddd417700 1 --2- 172.21.15.195:0/1268758209 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fedd80c39e0 0x7fedd81535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:21.643 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.638+0000 7fedd7fff700 1 -- 172.21.15.195:0/1268758209 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fedd80c39e0 msgr2=0x7fedd81535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:59:21.643 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.638+0000 7fedd7fff700 1 --2- 172.21.15.195:0/1268758209 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fedd80c39e0 0x7fedd81535a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:21.644 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.638+0000 7fedd7fff700 1 -- 172.21.15.195:0/1268758209 >> v1:172.21.15.202:6789/0 conn(0x7fedd806f640 legacy=0x7fedd80c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:59:21.644 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.638+0000 7fedd7fff700 1 -- 172.21.15.195:0/1268758209 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedd806ed70 con 0x7fedd80c4160 2022-06-27T11:59:21.644 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.638+0000 7fedd7fff700 1 --2- 172.21.15.195:0/1268758209 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fedd80c4160 0x7fedd8156ed0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fedc4009860 tx=0x7fedc4009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13ccd0ed881e431d server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:59:21.644 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.638+0000 7fedd77fe700 1 -- 172.21.15.195:0/1268758209 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedc4004030 con 0x7fedd80c4160 2022-06-27T11:59:21.644 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.639+0000 7fedd77fe700 1 -- 172.21.15.195:0/1268758209 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedc4004190 con 0x7fedd80c4160 2022-06-27T11:59:21.645 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.639+0000 7fedd77fe700 1 -- 172.21.15.195:0/1268758209 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedc400b6d0 con 0x7fedd80c4160 2022-06-27T11:59:21.645 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.639+0000 7feddee7a700 1 -- 172.21.15.195:0/1268758209 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fedd80c4160 msgr2=0x7fedd8156ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:21.646 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.639+0000 7feddee7a700 1 --2- 172.21.15.195:0/1268758209 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fedd80c4160 0x7fedd8156ed0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fedc4009860 tx=0x7fedc4009120 comp rx=0 tx=0).stop 2022-06-27T11:59:21.646 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.639+0000 7feddee7a700 1 -- 172.21.15.195:0/1268758209 shutdown_connections 2022-06-27T11:59:21.646 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.639+0000 7feddee7a700 1 --2- 172.21.15.195:0/1268758209 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fedd80c39e0 0x7fedd81535a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:21.646 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.639+0000 7feddee7a700 1 --2- 172.21.15.195:0/1268758209 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fedd80c4160 0x7fedd8156ed0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:21.647 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.639+0000 7feddee7a700 1 -- 172.21.15.195:0/1268758209 >> 172.21.15.195:0/1268758209 conn(0x7fedd80c6350 msgr2=0x7fedd80c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:59:21.647 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.639+0000 7feddee7a700 1 -- 172.21.15.195:0/1268758209 shutdown_connections 2022-06-27T11:59:21.647 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.639+0000 7feddee7a700 1 -- 172.21.15.195:0/1268758209 wait complete. 2022-06-27T11:59:21.647 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.640+0000 7feddee7a700 1 Processor -- start 2022-06-27T11:59:21.648 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.640+0000 7feddee7a700 1 -- start start 2022-06-27T11:59:21.653 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.641+0000 7feddee7a700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fedd806f640 0x7fedd8169f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:21.653 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.641+0000 7feddee7a700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fedd80c39e0 0x7fedd816a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:21.654 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.641+0000 7feddee7a700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fedd80c4160 0x7fedd8147ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:21.654 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.641+0000 7feddee7a700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd8169ab0 con 0x7fedd806f640 2022-06-27T11:59:21.654 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.641+0000 7feddee7a700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd81541c0 con 0x7fedd80c39e0 2022-06-27T11:59:21.655 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.641+0000 7feddee7a700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedd81544a0 con 0x7fedd80c4160 2022-06-27T11:59:21.655 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.641+0000 7feddcc16700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fedd806f640 0x7fedd8169f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:21.656 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.641+0000 7feddcc16700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fedd806f640 0x7fedd8169f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:46044/0 (socket says 172.21.15.195:46044) 2022-06-27T11:59:21.657 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.641+0000 7feddcc16700 1 -- 172.21.15.195:0/1022565404 learned_addr learned my addr 172.21.15.195:0/1022565404 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:59:21.657 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.642+0000 7feddd417700 1 --2- 172.21.15.195:0/1022565404 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fedd80c4160 0x7fedd8147ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:21.657 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.642+0000 7fedd7fff700 1 --2- 172.21.15.195:0/1022565404 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fedd80c39e0 0x7fedd816a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:21.658 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.642+0000 7feddcc16700 1 -- 172.21.15.195:0/1022565404 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fedd80c4160 msgr2=0x7fedd8147ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:21.658 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.642+0000 7feddcc16700 1 --2- 172.21.15.195:0/1022565404 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fedd80c4160 0x7fedd8147ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:21.659 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.642+0000 7feddcc16700 1 -- 172.21.15.195:0/1022565404 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fedd80c39e0 msgr2=0x7fedd816a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:21.659 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.642+0000 7feddcc16700 1 --2- 172.21.15.195:0/1022565404 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fedd80c39e0 0x7fedd816a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:21.660 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.642+0000 7feddcc16700 1 -- 172.21.15.195:0/1022565404 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedc40093a0 con 0x7fedd806f640 2022-06-27T11:59:21.660 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.642+0000 7fedd7fff700 1 --2- 172.21.15.195:0/1022565404 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fedd80c39e0 0x7fedd816a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T11:59:21.660 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.642+0000 7feddd417700 1 --2- 172.21.15.195:0/1022565404 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fedd80c4160 0x7fedd8147ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T11:59:21.661 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.642+0000 7feddcc16700 1 --2- 172.21.15.195:0/1022565404 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fedd806f640 0x7fedd8169f40 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fedcc00c490 tx=0x7fedcc007e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:59:21.661 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.642+0000 7fedd4ff9700 1 -- 172.21.15.195:0/1022565404 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedcc011590 con 0x7fedd806f640 2022-06-27T11:59:21.661 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.642+0000 7fedd4ff9700 1 -- 172.21.15.195:0/1022565404 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedcc0116f0 con 0x7fedd806f640 2022-06-27T11:59:21.662 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.642+0000 7feddee7a700 1 -- 172.21.15.195:0/1022565404 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedd8154780 con 0x7fedd806f640 2022-06-27T11:59:21.662 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.642+0000 7fedd4ff9700 1 -- 172.21.15.195:0/1022565404 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedcc011ab0 con 0x7fedd806f640 2022-06-27T11:59:21.663 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.642+0000 7feddee7a700 1 -- 172.21.15.195:0/1022565404 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedd8154d40 con 0x7fedd806f640 2022-06-27T11:59:21.663 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.643+0000 7feddee7a700 1 -- 172.21.15.195:0/1022565404 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedd8169d90 con 0x7fedd806f640 2022-06-27T11:59:21.664 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.643+0000 7fedd4ff9700 1 -- 172.21.15.195:0/1022565404 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fedcc010050 con 0x7fedd806f640 2022-06-27T11:59:21.664 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.644+0000 7fedd4ff9700 1 --2- 172.21.15.195:0/1022565404 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fedc0063d00 0x7fedc00661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:21.665 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.644+0000 7fedd7fff700 1 --2- 172.21.15.195:0/1022565404 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fedc0063d00 0x7fedc00661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:21.665 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.644+0000 7fedd4ff9700 1 -- 172.21.15.195:0/1022565404 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6718+0+0 (secure 0 0 0) 0x7fedcc0805b0 con 0x7fedd806f640 2022-06-27T11:59:21.665 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.644+0000 7fedd7fff700 1 --2- 172.21.15.195:0/1022565404 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fedc0063d00 0x7fedc00661b0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fedc4009160 tx=0x7fedc4016000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:59:21.665 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.648+0000 7fedd4ff9700 1 -- 172.21.15.195:0/1022565404 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fedcc04ef10 con 0x7fedd806f640 2022-06-27T11:59:21.874 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.875+0000 7feddee7a700 1 -- 172.21.15.195:0/1022565404 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fedd80722c0 con 0x7fedd806f640 2022-06-27T11:59:21.876 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.875+0000 7fedd4ff9700 1 -- 172.21.15.195:0/1022565404 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+5374 (secure 0 0 0) 0x7fedcc052530 con 0x7fedd806f640 2022-06-27T11:59:21.876 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:59:21.877 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":46,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:59:17.069976+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5192},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5192":{"gid":5192,"name":"b","rank":0,"incarnation":43,"state":"up:clientreplay","state_seq":12,"addr":"172.21.15.202:6833/2651813808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":2651813808},{"type":"v1","addr":"172.21.15.202:6833","nonce":2651813808}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:59:21.879 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.877+0000 7feddee7a700 1 -- 172.21.15.195:0/1022565404 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fedc0063d00 msgr2=0x7fedc00661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:21.879 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.877+0000 7feddee7a700 1 --2- 172.21.15.195:0/1022565404 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fedc0063d00 0x7fedc00661b0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fedc4009160 tx=0x7fedc4016000 comp rx=0 tx=0).stop 2022-06-27T11:59:21.879 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.877+0000 7feddee7a700 1 -- 172.21.15.195:0/1022565404 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fedd806f640 msgr2=0x7fedd8169f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:21.879 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.877+0000 7feddee7a700 1 --2- 172.21.15.195:0/1022565404 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fedd806f640 0x7fedd8169f40 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fedcc00c490 tx=0x7fedcc007e20 comp rx=0 tx=0).stop 2022-06-27T11:59:21.880 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.877+0000 7feddee7a700 1 -- 172.21.15.195:0/1022565404 shutdown_connections 2022-06-27T11:59:21.880 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.877+0000 7feddee7a700 1 --2- 172.21.15.195:0/1022565404 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fedc0063d00 0x7fedc00661b0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:21.880 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.877+0000 7feddee7a700 1 --2- 172.21.15.195:0/1022565404 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fedd806f640 0x7fedd8169f40 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:21.880 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.877+0000 7feddee7a700 1 --2- 172.21.15.195:0/1022565404 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fedd80c4160 0x7fedd8147ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:21.881 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.877+0000 7feddee7a700 1 --2- 172.21.15.195:0/1022565404 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fedd80c39e0 0x7fedd816a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:21.881 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.877+0000 7feddee7a700 1 -- 172.21.15.195:0/1022565404 >> 172.21.15.195:0/1022565404 conn(0x7fedd80c6350 msgr2=0x7fedd81528e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:59:21.881 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.877+0000 7feddee7a700 1 -- 172.21.15.195:0/1022565404 shutdown_connections 2022-06-27T11:59:21.882 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:21.877+0000 7feddee7a700 1 -- 172.21.15.195:0/1022565404 wait complete. 2022-06-27T11:59:21.882 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 46 2022-06-27T11:59:21.889 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T11:59:23.893 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:59:24.060 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.029+0000 7fe573122700 1 Processor -- start 2022-06-27T11:59:24.061 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.029+0000 7fe573122700 1 -- start start 2022-06-27T11:59:24.062 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.029+0000 7fe573122700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fe56c0c5000 0x7fe56c0c2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:24.062 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.029+0000 7fe573122700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fe56c0c61e0 0x7fe56c0c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:24.063 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.029+0000 7fe573122700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe56c0bc490 con 0x7fe56c0c2bf0 2022-06-27T11:59:24.063 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.029+0000 7fe573122700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe56c06a0e0 con 0x7fe56c0c5000 2022-06-27T11:59:24.064 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.029+0000 7fe573122700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe56c0bc6e0 con 0x7fe56c0c61e0 2022-06-27T11:59:24.064 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.030+0000 7fe56bfff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fe56c0c5000 0x7fe56c0c2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:24.065 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.030+0000 7fe56bfff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fe56c0c5000 0x7fe56c0c2fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:46072/0 (socket says 172.21.15.195:46072) 2022-06-27T11:59:24.065 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.030+0000 7fe56bfff700 1 -- 172.21.15.195:0/3119271346 learned_addr learned my addr 172.21.15.195:0/3119271346 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:59:24.066 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.030+0000 7fe5716bf700 1 --2- 172.21.15.195:0/3119271346 >> v2:172.21.15.202:3300/0 conn(0x7fe56c0c61e0 0x7fe56c0c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:24.066 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.030+0000 7fe56bfff700 1 -- 172.21.15.195:0/3119271346 >> v2:172.21.15.202:3300/0 conn(0x7fe56c0c61e0 msgr2=0x7fe56c0c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:59:24.066 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.030+0000 7fe56bfff700 1 --2- 172.21.15.195:0/3119271346 >> v2:172.21.15.202:3300/0 conn(0x7fe56c0c61e0 0x7fe56c0c3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:24.067 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.030+0000 7fe56bfff700 1 -- 172.21.15.195:0/3119271346 >> v1:172.21.15.202:6789/0 conn(0x7fe56c0c2bf0 legacy=0x7fe56c0c48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:24.067 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.030+0000 7fe56bfff700 1 -- 172.21.15.195:0/3119271346 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe56c158220 con 0x7fe56c0c5000 2022-06-27T11:59:24.068 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.030+0000 7fe56bfff700 1 --2- 172.21.15.195:0/3119271346 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe56c0c5000 0x7fe56c0c2fc0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fe56c0bc9a0 tx=0x7fe558009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df8a92a4871f5c5 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:59:24.068 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.030+0000 7fe56b7fe700 1 -- 172.21.15.195:0/3119271346 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe55800b640 con 0x7fe56c0c5000 2022-06-27T11:59:24.069 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.030+0000 7fe56b7fe700 1 -- 172.21.15.195:0/3119271346 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe55800b7a0 con 0x7fe56c0c5000 2022-06-27T11:59:24.069 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.030+0000 7fe56b7fe700 1 -- 172.21.15.195:0/3119271346 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe55800bb40 con 0x7fe56c0c5000 2022-06-27T11:59:24.070 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.031+0000 7fe573122700 1 -- 172.21.15.195:0/3119271346 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe56c0c5000 msgr2=0x7fe56c0c2fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:24.070 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.031+0000 7fe573122700 1 --2- 172.21.15.195:0/3119271346 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe56c0c5000 0x7fe56c0c2fc0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fe56c0bc9a0 tx=0x7fe558009120 comp rx=0 tx=0).stop 2022-06-27T11:59:24.071 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.031+0000 7fe573122700 1 -- 172.21.15.195:0/3119271346 shutdown_connections 2022-06-27T11:59:24.071 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.031+0000 7fe573122700 1 --2- 172.21.15.195:0/3119271346 >> v2:172.21.15.202:3300/0 conn(0x7fe56c0c61e0 0x7fe56c0c3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:24.071 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.031+0000 7fe573122700 1 --2- 172.21.15.195:0/3119271346 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe56c0c5000 0x7fe56c0c2fc0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:24.072 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.031+0000 7fe573122700 1 -- 172.21.15.195:0/3119271346 >> 172.21.15.195:0/3119271346 conn(0x7fe56c0c9b00 msgr2=0x7fe56c0c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:59:24.072 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.031+0000 7fe573122700 1 -- 172.21.15.195:0/3119271346 shutdown_connections 2022-06-27T11:59:24.073 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.031+0000 7fe573122700 1 -- 172.21.15.195:0/3119271346 wait complete. 2022-06-27T11:59:24.073 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.032+0000 7fe573122700 1 Processor -- start 2022-06-27T11:59:24.074 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.032+0000 7fe573122700 1 -- start start 2022-06-27T11:59:24.074 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.033+0000 7fe573122700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe56c0c2bf0 0x7fe56c1545b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:24.075 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.033+0000 7fe573122700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe56c0c5000 0x7fe56c0bd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:24.075 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.033+0000 7fe573122700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe56c0c61e0 0x7fe56c0bdc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:24.076 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.033+0000 7fe573122700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe56c153820 con 0x7fe56c0c5000 2022-06-27T11:59:24.076 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.033+0000 7fe573122700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe56c0be790 con 0x7fe56c0c2bf0 2022-06-27T11:59:24.076 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.033+0000 7fe573122700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe56c16a710 con 0x7fe56c0c61e0 2022-06-27T11:59:24.077 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.033+0000 7fe56bfff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe56c0c5000 0x7fe56c0bd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:24.077 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.033+0000 7fe56bfff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe56c0c5000 0x7fe56c0bd6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:46078/0 (socket says 172.21.15.195:46078) 2022-06-27T11:59:24.078 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.033+0000 7fe56bfff700 1 -- 172.21.15.195:0/3977164952 learned_addr learned my addr 172.21.15.195:0/3977164952 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:59:24.078 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.033+0000 7fe570ebe700 1 --2- 172.21.15.195:0/3977164952 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe56c0c2bf0 0x7fe56c1545b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:24.079 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.034+0000 7fe5716bf700 1 --2- 172.21.15.195:0/3977164952 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe56c0c61e0 0x7fe56c0bdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:24.079 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.034+0000 7fe56bfff700 1 -- 172.21.15.195:0/3977164952 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe56c0c61e0 msgr2=0x7fe56c0bdc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:24.079 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.034+0000 7fe56bfff700 1 --2- 172.21.15.195:0/3977164952 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe56c0c61e0 0x7fe56c0bdc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:24.080 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.034+0000 7fe56bfff700 1 -- 172.21.15.195:0/3977164952 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe56c0c2bf0 msgr2=0x7fe56c1545b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:24.080 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.034+0000 7fe56bfff700 1 --2- 172.21.15.195:0/3977164952 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe56c0c2bf0 0x7fe56c1545b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:24.081 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.034+0000 7fe56bfff700 1 -- 172.21.15.195:0/3977164952 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5580093a0 con 0x7fe56c0c5000 2022-06-27T11:59:24.081 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.034+0000 7fe56bfff700 1 --2- 172.21.15.195:0/3977164952 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe56c0c5000 0x7fe56c0bd6e0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fe56c06f810 tx=0x7fe558017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:59:24.082 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.034+0000 7fe568ff9700 1 -- 172.21.15.195:0/3977164952 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe558004530 con 0x7fe56c0c5000 2022-06-27T11:59:24.082 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.034+0000 7fe568ff9700 1 -- 172.21.15.195:0/3977164952 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe558003730 con 0x7fe56c0c5000 2022-06-27T11:59:24.082 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.034+0000 7fe568ff9700 1 -- 172.21.15.195:0/3977164952 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe558003af0 con 0x7fe56c0c5000 2022-06-27T11:59:24.083 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.034+0000 7fe573122700 1 -- 172.21.15.195:0/3977164952 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe56c16a9e0 con 0x7fe56c0c5000 2022-06-27T11:59:24.084 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.034+0000 7fe573122700 1 -- 172.21.15.195:0/3977164952 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe56c16afa0 con 0x7fe56c0c5000 2022-06-27T11:59:24.084 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.035+0000 7fe573122700 1 -- 172.21.15.195:0/3977164952 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe56c0be920 con 0x7fe56c0c5000 2022-06-27T11:59:24.084 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.035+0000 7fe568ff9700 1 -- 172.21.15.195:0/3977164952 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fe55800f050 con 0x7fe56c0c5000 2022-06-27T11:59:24.085 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.036+0000 7fe568ff9700 1 --2- 172.21.15.195:0/3977164952 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe554063d30 0x7fe5540661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:24.085 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.036+0000 7fe568ff9700 1 -- 172.21.15.195:0/3977164952 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6718+0+0 (secure 0 0 0) 0x7fe558081e20 con 0x7fe56c0c5000 2022-06-27T11:59:24.086 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.036+0000 7fe570ebe700 1 --2- 172.21.15.195:0/3977164952 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe554063d30 0x7fe5540661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:24.087 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.036+0000 7fe570ebe700 1 --2- 172.21.15.195:0/3977164952 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe554063d30 0x7fe5540661e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fe560007fd0 tx=0x7fe560009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:59:24.087 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.040+0000 7fe568ff9700 1 -- 172.21.15.195:0/3977164952 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fe558050780 con 0x7fe56c0c5000 2022-06-27T11:59:24.269 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.268+0000 7fe573122700 1 -- 172.21.15.195:0/3977164952 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe56c0722c0 con 0x7fe56c0c5000 2022-06-27T11:59:24.269 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.269+0000 7fe568ff9700 1 -- 172.21.15.195:0/3977164952 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+5368 (secure 0 0 0) 0x7fe558053da0 con 0x7fe56c0c5000 2022-06-27T11:59:24.270 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:59:24.270 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:59:23.081414+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5192},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5192":{"gid":5192,"name":"b","rank":0,"incarnation":43,"state":"up:active","state_seq":14,"addr":"172.21.15.202:6833/2651813808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":2651813808},{"type":"v1","addr":"172.21.15.202:6833","nonce":2651813808}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:59:24.272 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.270+0000 7fe573122700 1 -- 172.21.15.195:0/3977164952 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe554063d30 msgr2=0x7fe5540661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:24.272 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.270+0000 7fe573122700 1 --2- 172.21.15.195:0/3977164952 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe554063d30 0x7fe5540661e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fe560007fd0 tx=0x7fe560009040 comp rx=0 tx=0).stop 2022-06-27T11:59:24.274 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.270+0000 7fe573122700 1 -- 172.21.15.195:0/3977164952 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe56c0c5000 msgr2=0x7fe56c0bd6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:24.274 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.270+0000 7fe573122700 1 --2- 172.21.15.195:0/3977164952 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe56c0c5000 0x7fe56c0bd6e0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fe56c06f810 tx=0x7fe558017000 comp rx=0 tx=0).stop 2022-06-27T11:59:24.275 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.271+0000 7fe573122700 1 -- 172.21.15.195:0/3977164952 shutdown_connections 2022-06-27T11:59:24.275 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.271+0000 7fe573122700 1 --2- 172.21.15.195:0/3977164952 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe554063d30 0x7fe5540661e0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:24.275 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.271+0000 7fe573122700 1 --2- 172.21.15.195:0/3977164952 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe56c0c61e0 0x7fe56c0bdc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:24.275 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.271+0000 7fe573122700 1 --2- 172.21.15.195:0/3977164952 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe56c0c2bf0 0x7fe56c1545b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:24.276 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.271+0000 7fe573122700 1 --2- 172.21.15.195:0/3977164952 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe56c0c5000 0x7fe56c0bd6e0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:24.276 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.271+0000 7fe573122700 1 -- 172.21.15.195:0/3977164952 >> 172.21.15.195:0/3977164952 conn(0x7fe56c0c9b00 msgr2=0x7fe56c0c2450 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:59:24.276 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.271+0000 7fe573122700 1 -- 172.21.15.195:0/3977164952 shutdown_connections 2022-06-27T11:59:24.276 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:24.271+0000 7fe573122700 1 -- 172.21.15.195:0/3977164952 wait complete. 2022-06-27T11:59:24.276 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 47 2022-06-27T11:59:24.283 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=5392 2022-06-27T11:59:24.284 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 2 secs before reviving mds.d 2022-06-27T11:59:26.287 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.d 2022-06-27T11:59:26.369 INFO:tasks.ceph.mds.d:Restarting daemon 2022-06-27T11:59:26.370 DEBUG:teuthology.orchestra.run.smithi202:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2022-06-27T11:59:26.380 INFO:tasks.ceph.mds.d:Started 2022-06-27T11:59:26.381 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:59:26.464 INFO:tasks.ceph.mds.d.smithi202.stdout:starting mds.d at 2022-06-27T11:59:26.534 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.534+0000 7f2574179700 1 Processor -- start 2022-06-27T11:59:26.535 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.534+0000 7f2574179700 1 -- start start 2022-06-27T11:59:26.556 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.535+0000 7f2574179700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f256c150580 0x7f256c156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:26.557 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.535+0000 7f2574179700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f256c06f620 0x7f256c06d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:26.557 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.535+0000 7f2574179700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f256c0c76b0 con 0x7f256c150950 2022-06-27T11:59:26.558 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.535+0000 7f2574179700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f256c157430 con 0x7f256c06f620 2022-06-27T11:59:26.559 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.535+0000 7f2574179700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f256c0c8250 con 0x7f256c150580 2022-06-27T11:59:26.559 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.535+0000 7f2572716700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f256c06f620 0x7f256c06d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:26.560 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.535+0000 7f2571f15700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f256c150580 0x7f256c156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:26.560 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.535+0000 7f2571714700 1 --1- >> v1:172.21.15.202:6789/0 conn(0x7f256c150950 0x7f256c0c7b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.202:6789/0 says I am v1:172.21.15.195:54136/0 (socket says 172.21.15.195:54136) 2022-06-27T11:59:26.561 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.535+0000 7f2571714700 1 -- 172.21.15.195:0/3751233655 learned_addr learned my addr 172.21.15.195:0/3751233655 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T11:59:26.561 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.535+0000 7f2572716700 1 -- 172.21.15.195:0/3751233655 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f256c150580 msgr2=0x7f256c156ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:59:26.561 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.535+0000 7f2572716700 1 --2- 172.21.15.195:0/3751233655 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f256c150580 0x7f256c156ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:26.562 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.535+0000 7f2572716700 1 -- 172.21.15.195:0/3751233655 >> v1:172.21.15.202:6789/0 conn(0x7f256c150950 legacy=0x7f256c0c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:26.562 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.535+0000 7f2572716700 1 -- 172.21.15.195:0/3751233655 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f256c0bad70 con 0x7f256c06f620 2022-06-27T11:59:26.563 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.535+0000 7f2570f13700 1 -- 172.21.15.195:0/3751233655 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 292004874 0 0) 0x7f256c0c76b0 con 0x7f256c150950 2022-06-27T11:59:26.564 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.535+0000 7f2571f15700 1 --2- 172.21.15.195:0/3751233655 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f256c150580 0x7f256c156ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:59:26.564 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.535+0000 7f2572716700 1 --2- 172.21.15.195:0/3751233655 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f256c06f620 0x7f256c06d6a0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f2564009e90 tx=0x7f2564009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=afc02b4fa3b8f4f2 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:59:26.565 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.536+0000 7f2570f13700 1 -- 172.21.15.195:0/3751233655 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f256400e070 con 0x7f256c06f620 2022-06-27T11:59:26.565 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.536+0000 7f2570f13700 1 -- 172.21.15.195:0/3751233655 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f256400bbf0 con 0x7f256c06f620 2022-06-27T11:59:26.566 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.536+0000 7f2570f13700 1 -- 172.21.15.195:0/3751233655 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2564012430 con 0x7f256c06f620 2022-06-27T11:59:26.566 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.536+0000 7f2574179700 1 -- 172.21.15.195:0/3751233655 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f256c06f620 msgr2=0x7f256c06d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:26.567 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.536+0000 7f2574179700 1 --2- 172.21.15.195:0/3751233655 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f256c06f620 0x7f256c06d6a0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f2564009e90 tx=0x7f2564009780 comp rx=0 tx=0).stop 2022-06-27T11:59:26.567 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.536+0000 7f2574179700 1 -- 172.21.15.195:0/3751233655 shutdown_connections 2022-06-27T11:59:26.568 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.536+0000 7f2574179700 1 --2- 172.21.15.195:0/3751233655 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f256c06f620 0x7f256c06d6a0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:26.568 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.536+0000 7f2574179700 1 --2- 172.21.15.195:0/3751233655 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f256c150580 0x7f256c156ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:26.569 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.536+0000 7f2574179700 1 -- 172.21.15.195:0/3751233655 >> 172.21.15.195:0/3751233655 conn(0x7f256c063af0 msgr2=0x7f256c063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:59:26.569 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.536+0000 7f2574179700 1 -- 172.21.15.195:0/3751233655 shutdown_connections 2022-06-27T11:59:26.570 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.536+0000 7f2574179700 1 -- 172.21.15.195:0/3751233655 wait complete. 2022-06-27T11:59:26.570 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.537+0000 7f2574179700 1 Processor -- start 2022-06-27T11:59:26.571 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.537+0000 7f2574179700 1 -- start start 2022-06-27T11:59:26.571 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.538+0000 7f2574179700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f256c06f620 0x7f256c0bd740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:26.571 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.538+0000 7f2574179700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f256c150580 0x7f256c0c1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:26.572 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.538+0000 7f2574179700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f256c150950 0x7f256c0c3820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:26.572 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.538+0000 7f2574179700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f256c1671b0 con 0x7f256c150950 2022-06-27T11:59:26.573 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.538+0000 7f2574179700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f256c14e1b0 con 0x7f256c06f620 2022-06-27T11:59:26.573 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.538+0000 7f2574179700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f256c0be240 con 0x7f256c150580 2022-06-27T11:59:26.573 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.538+0000 7f2572716700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f256c150950 0x7f256c0c3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:26.574 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.538+0000 7f2571f15700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f256c06f620 0x7f256c0bd740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:26.574 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.538+0000 7f2572716700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f256c150950 0x7f256c0c3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:46098/0 (socket says 172.21.15.195:46098) 2022-06-27T11:59:26.575 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.538+0000 7f2571714700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f256c150580 0x7f256c0c1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:26.575 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.538+0000 7f2571f15700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f256c06f620 0x7f256c0bd740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3300/0 says I am v2:172.21.15.195:44220/0 (socket says 172.21.15.195:44220) 2022-06-27T11:59:26.576 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.538+0000 7f2572716700 1 -- 172.21.15.195:0/388796092 learned_addr learned my addr 172.21.15.195:0/388796092 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:59:26.577 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.539+0000 7f2572716700 1 -- 172.21.15.195:0/388796092 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f256c150580 msgr2=0x7f256c0c1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:26.578 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.539+0000 7f2572716700 1 --2- 172.21.15.195:0/388796092 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f256c150580 0x7f256c0c1ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:26.578 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.539+0000 7f2572716700 1 -- 172.21.15.195:0/388796092 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f256c06f620 msgr2=0x7f256c0bd740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:26.579 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.539+0000 7f2572716700 1 --2- 172.21.15.195:0/388796092 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f256c06f620 0x7f256c0bd740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:26.579 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.539+0000 7f2572716700 1 -- 172.21.15.195:0/388796092 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f255c0092e0 con 0x7f256c150950 2022-06-27T11:59:26.580 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.539+0000 7f2571714700 1 --2- 172.21.15.195:0/388796092 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f256c150580 0x7f256c0c1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:59:26.580 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.539+0000 7f2571f15700 1 --2- 172.21.15.195:0/388796092 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f256c06f620 0x7f256c0bd740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T11:59:26.581 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.539+0000 7f2572716700 1 --2- 172.21.15.195:0/388796092 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f256c150950 0x7f256c0c3820 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f256400b6c0 tx=0x7f2564019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:59:26.581 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.539+0000 7f2559ffb700 1 -- 172.21.15.195:0/388796092 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25640129b0 con 0x7f256c150950 2022-06-27T11:59:26.581 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.539+0000 7f2559ffb700 1 -- 172.21.15.195:0/388796092 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2564012b30 con 0x7f256c150950 2022-06-27T11:59:26.582 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.539+0000 7f2559ffb700 1 -- 172.21.15.195:0/388796092 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2564004040 con 0x7f256c150950 2022-06-27T11:59:26.582 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.539+0000 7f2574179700 1 -- 172.21.15.195:0/388796092 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2564009a20 con 0x7f256c150950 2022-06-27T11:59:26.582 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.539+0000 7f2574179700 1 -- 172.21.15.195:0/388796092 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f256c1663f0 con 0x7f256c150950 2022-06-27T11:59:26.583 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.540+0000 7f2574179700 1 -- 172.21.15.195:0/388796092 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f256c06eba0 con 0x7f256c150950 2022-06-27T11:59:26.583 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.540+0000 7f2559ffb700 1 -- 172.21.15.195:0/388796092 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f25640043b0 con 0x7f256c150950 2022-06-27T11:59:26.583 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.541+0000 7f2559ffb700 1 --2- 172.21.15.195:0/388796092 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f254c063d00 0x7f254c0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:26.584 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.541+0000 7f2559ffb700 1 -- 172.21.15.195:0/388796092 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6718+0+0 (secure 0 0 0) 0x7f2564083490 con 0x7f256c150950 2022-06-27T11:59:26.584 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.541+0000 7f2571f15700 1 --2- 172.21.15.195:0/388796092 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f254c063d00 0x7f254c0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:26.585 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.541+0000 7f2571f15700 1 --2- 172.21.15.195:0/388796092 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f254c063d00 0x7f254c0661b0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f255c0114c0 tx=0x7f255c009060 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:59:26.585 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.545+0000 7f2559ffb700 1 -- 172.21.15.195:0/388796092 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f2564051df0 con 0x7f256c150950 2022-06-27T11:59:26.772 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.772+0000 7f2574179700 1 -- 172.21.15.195:0/388796092 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f256c0722c0 con 0x7f256c150950 2022-06-27T11:59:26.773 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.772+0000 7f2559ffb700 1 -- 172.21.15.195:0/388796092 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+5368 (secure 0 0 0) 0x7f2564055410 con 0x7f256c150950 2022-06-27T11:59:26.773 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:59:26.774 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:59:23.081414+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5192},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5192":{"gid":5192,"name":"b","rank":0,"incarnation":43,"state":"up:active","state_seq":14,"addr":"172.21.15.202:6833/2651813808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":2651813808},{"type":"v1","addr":"172.21.15.202:6833","nonce":2651813808}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:59:26.779 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.774+0000 7f2574179700 1 -- 172.21.15.195:0/388796092 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f254c063d00 msgr2=0x7f254c0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:26.780 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.774+0000 7f2574179700 1 --2- 172.21.15.195:0/388796092 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f254c063d00 0x7f254c0661b0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f255c0114c0 tx=0x7f255c009060 comp rx=0 tx=0).stop 2022-06-27T11:59:26.780 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.774+0000 7f2574179700 1 -- 172.21.15.195:0/388796092 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f256c150950 msgr2=0x7f256c0c3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:26.781 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.774+0000 7f2574179700 1 --2- 172.21.15.195:0/388796092 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f256c150950 0x7f256c0c3820 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f256400b6c0 tx=0x7f2564019000 comp rx=0 tx=0).stop 2022-06-27T11:59:26.781 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.774+0000 7f2574179700 1 -- 172.21.15.195:0/388796092 shutdown_connections 2022-06-27T11:59:26.782 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.774+0000 7f2574179700 1 --2- 172.21.15.195:0/388796092 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f254c063d00 0x7f254c0661b0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:26.782 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.774+0000 7f2574179700 1 --2- 172.21.15.195:0/388796092 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f256c150950 0x7f256c0c3820 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:26.783 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.774+0000 7f2574179700 1 --2- 172.21.15.195:0/388796092 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f256c06f620 0x7f256c0bd740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:26.783 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.774+0000 7f2574179700 1 --2- 172.21.15.195:0/388796092 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f256c150580 0x7f256c0c1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:26.784 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.774+0000 7f2574179700 1 -- 172.21.15.195:0/388796092 >> 172.21.15.195:0/388796092 conn(0x7f256c063af0 msgr2=0x7f256c0c5e50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:59:26.784 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.774+0000 7f2574179700 1 -- 172.21.15.195:0/388796092 shutdown_connections 2022-06-27T11:59:26.785 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:26.774+0000 7f2574179700 1 -- 172.21.15.195:0/388796092 wait complete. 2022-06-27T11:59:26.785 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 47 2022-06-27T11:59:26.789 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2022-06-27T11:59:28.792 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T11:59:28.919 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:28.916+0000 7faaafb0a700 1 Processor -- start 2022-06-27T11:59:28.920 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:28.916+0000 7faaafb0a700 1 -- start start 2022-06-27T11:59:28.920 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:28.917+0000 7faaafb0a700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7faaa806eff0 0x7faaa80c61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:28.920 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:28.917+0000 7faaafb0a700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaa814d8a0 con 0x7faaa8067ed0 2022-06-27T11:59:28.921 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:28.917+0000 7faaafb0a700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaa80c3e10 con 0x7faaa80c31d0 2022-06-27T11:59:28.921 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:28.917+0000 7faaafb0a700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faaa806d970 con 0x7faaa806eff0 2022-06-27T11:59:28.921 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:28.917+0000 7faaad8a6700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7faaa806eff0 0x7faaa80c61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:28.922 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:28.917+0000 7faaad8a6700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7faaa806eff0 0x7faaa80c61e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:46116/0 (socket says 172.21.15.195:46116) 2022-06-27T11:59:28.922 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:28.917+0000 7faaad8a6700 1 -- 172.21.15.195:0/451171288 learned_addr learned my addr 172.21.15.195:0/451171288 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:59:28.922 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:28.917+0000 7faaad0a5700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7faaa8067ed0 0x7faaa80c2ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:52304/0 (socket says 172.21.15.195:52304) 2022-06-27T11:59:28.922 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:28.917+0000 7faaad8a6700 1 -- 172.21.15.195:0/451171288 >> v1:172.21.15.202:6789/0 conn(0x7faaa80c31d0 legacy=0x7faaa815da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T11:59:28.923 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:28.917+0000 7faaad8a6700 1 -- 172.21.15.195:0/451171288 >> v1:172.21.15.195:6789/0 conn(0x7faaa8067ed0 legacy=0x7faaa80c2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:28.923 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:28.917+0000 7faaad8a6700 1 -- 172.21.15.195:0/451171288 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaa8153f20 con 0x7faaa806eff0 2022-06-27T11:59:28.923 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:28.917+0000 7faaac8a4700 1 -- 172.21.15.195:0/451171288 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 567319279 0 0) 0x7faaa814d8a0 con 0x7faaa8067ed0 2022-06-27T11:59:28.923 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:28.917+0000 7faaad8a6700 1 --2- 172.21.15.195:0/451171288 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faaa806eff0 0x7faaa80c61e0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7faaa4006fd0 tx=0x7faaa40090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6245ebde2fea33a server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:59:28.924 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:28.917+0000 7faaac8a4700 1 -- 172.21.15.195:0/451171288 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaa4005ad0 con 0x7faaa806eff0 2022-06-27T11:59:28.924 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:28.917+0000 7faaac8a4700 1 -- 172.21.15.195:0/451171288 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaa4005c30 con 0x7faaa806eff0 2022-06-27T11:59:28.924 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:28.917+0000 7faaac8a4700 1 -- 172.21.15.195:0/451171288 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaa4004030 con 0x7faaa806eff0 2022-06-27T11:59:28.924 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:28.918+0000 7faaafb0a700 1 -- 172.21.15.195:0/451171288 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faaa806eff0 msgr2=0x7faaa80c61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:28.924 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:28.918+0000 7faaafb0a700 1 --2- 172.21.15.195:0/451171288 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faaa806eff0 0x7faaa80c61e0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7faaa4006fd0 tx=0x7faaa40090f0 comp rx=0 tx=0).stop 2022-06-27T11:59:28.925 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:28.918+0000 7faaafb0a700 1 -- 172.21.15.195:0/451171288 shutdown_connections 2022-06-27T11:59:28.925 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:28.918+0000 7faaafb0a700 1 --2- 172.21.15.195:0/451171288 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faaa806eff0 0x7faaa80c61e0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:28.925 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:28.918+0000 7faaafb0a700 1 -- 172.21.15.195:0/451171288 >> 172.21.15.195:0/451171288 conn(0x7faaa80c9b00 msgr2=0x7faaa80c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:59:28.925 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:28.918+0000 7faaafb0a700 1 -- 172.21.15.195:0/451171288 shutdown_connections 2022-06-27T11:59:28.925 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:28.918+0000 7faaafb0a700 1 -- 172.21.15.195:0/451171288 wait complete. 2022-06-27T11:59:28.929 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:28.919+0000 7faaafb0a700 1 Processor -- start 2022-06-27T11:59:28.930 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:28.920+0000 7faaafb0a700 1 -- start start 2022-06-27T11:59:28.931 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:28.920+0000 7faaafb0a700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faaa8067ed0 0x7faaa80c07e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:28.931 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:28.920+0000 7faaafb0a700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faaa806eff0 0x7faaa80c0d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:28.931 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:28.920+0000 7faaafb0a700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7faaa8160c30 0x7faaa8140170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:28.932 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:28.920+0000 7faaafb0a700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaa8151f90 con 0x7faaa806eff0 2022-06-27T11:59:28.932 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:28.920+0000 7faaafb0a700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaa8152d30 con 0x7faaa8160c30 2022-06-27T11:59:28.932 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:28.920+0000 7faaafb0a700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaa8156d50 con 0x7faaa8067ed0 2022-06-27T11:59:28.932 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:28.921+0000 7faaad0a5700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faaa806eff0 0x7faaa80c0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:28.933 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:28.921+0000 7faaad0a5700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faaa806eff0 0x7faaa80c0d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:46124/0 (socket says 172.21.15.195:46124) 2022-06-27T11:59:28.933 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:28.921+0000 7faaad0a5700 1 -- 172.21.15.195:0/1982508561 learned_addr learned my addr 172.21.15.195:0/1982508561 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T11:59:28.934 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:28.921+0000 7faaad8a6700 1 --2- 172.21.15.195:0/1982508561 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faaa8067ed0 0x7faaa80c07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:28.934 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:28.921+0000 7faaae0a7700 1 --2- 172.21.15.195:0/1982508561 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7faaa8160c30 0x7faaa8140170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:28.934 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:28.921+0000 7faaad0a5700 1 -- 172.21.15.195:0/1982508561 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faaa8067ed0 msgr2=0x7faaa80c07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:28.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:28.921+0000 7faaad0a5700 1 --2- 172.21.15.195:0/1982508561 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faaa8067ed0 0x7faaa80c07e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:28.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:28.921+0000 7faaad0a5700 1 -- 172.21.15.195:0/1982508561 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7faaa8160c30 msgr2=0x7faaa8140170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:28.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:28.921+0000 7faaad0a5700 1 --2- 172.21.15.195:0/1982508561 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7faaa8160c30 0x7faaa8140170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:28.937 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:28.921+0000 7faaad0a5700 1 -- 172.21.15.195:0/1982508561 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaa4009350 con 0x7faaa806eff0 2022-06-27T11:59:28.937 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:28.921+0000 7faaad0a5700 1 --2- 172.21.15.195:0/1982508561 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faaa806eff0 0x7faaa80c0d20 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7faa9800c560 tx=0x7faa98007c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:59:28.937 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:28.921+0000 7faa95ffb700 1 -- 172.21.15.195:0/1982508561 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa98007360 con 0x7faaa806eff0 2022-06-27T11:59:28.937 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:28.921+0000 7faa95ffb700 1 -- 172.21.15.195:0/1982508561 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa980074c0 con 0x7faaa806eff0 2022-06-27T11:59:28.937 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:28.921+0000 7faaafb0a700 1 -- 172.21.15.195:0/1982508561 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaa8157030 con 0x7faaa806eff0 2022-06-27T11:59:28.938 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:28.921+0000 7faa95ffb700 1 -- 172.21.15.195:0/1982508561 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa98004900 con 0x7faaa806eff0 2022-06-27T11:59:28.938 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:28.921+0000 7faaafb0a700 1 -- 172.21.15.195:0/1982508561 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faaa81516b0 con 0x7faaa806eff0 2022-06-27T11:59:28.938 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:28.922+0000 7faaafb0a700 1 -- 172.21.15.195:0/1982508561 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faaa80722c0 con 0x7faaa806eff0 2022-06-27T11:59:28.938 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:28.922+0000 7faa95ffb700 1 -- 172.21.15.195:0/1982508561 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7faa98010030 con 0x7faaa806eff0 2022-06-27T11:59:28.939 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:28.923+0000 7faa95ffb700 1 --2- 172.21.15.195:0/1982508561 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7faa88063d00 0x7faa880661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T11:59:28.939 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:28.923+0000 7faa95ffb700 1 -- 172.21.15.195:0/1982508561 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6718+0+0 (secure 0 0 0) 0x7faa98080560 con 0x7faaa806eff0 2022-06-27T11:59:28.939 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:28.923+0000 7faaad8a6700 1 --2- 172.21.15.195:0/1982508561 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7faa88063d00 0x7faa880661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T11:59:28.939 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:28.923+0000 7faaad8a6700 1 --2- 172.21.15.195:0/1982508561 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7faa88063d00 0x7faa880661b0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7faaa4009130 tx=0x7faaa4015000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T11:59:28.940 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:28.927+0000 7faa95ffb700 1 -- 172.21.15.195:0/1982508561 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7faa9804ef40 con 0x7faaa806eff0 2022-06-27T11:59:29.104 INFO:tasks.ceph.mds.b.smithi202.stderr:2022-06-27T11:59:29.104+0000 7f42cfe70700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2022-06-27T11:59:29.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:29.154+0000 7faaafb0a700 1 -- 172.21.15.195:0/1982508561 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faaa80c5120 con 0x7faaa806eff0 2022-06-27T11:59:29.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:29.154+0000 7faa95ffb700 1 -- 172.21.15.195:0/1982508561 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+6125 (secure 0 0 0) 0x7faa98052560 con 0x7faaa806eff0 2022-06-27T11:59:29.155 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T11:59:29.155 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/417105338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":417105338},{"type":"v1","addr":"172.21.15.202:6835","nonce":417105338}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:59:23.081414+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5192},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5192":{"gid":5192,"name":"b","rank":0,"incarnation":43,"state":"up:active","state_seq":14,"addr":"172.21.15.202:6833/2651813808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":2651813808},{"type":"v1","addr":"172.21.15.202:6833","nonce":2651813808}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T11:59:29.157 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:29.156+0000 7faaafb0a700 1 -- 172.21.15.195:0/1982508561 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7faa88063d00 msgr2=0x7faa880661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:29.158 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:29.156+0000 7faaafb0a700 1 --2- 172.21.15.195:0/1982508561 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7faa88063d00 0x7faa880661b0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7faaa4009130 tx=0x7faaa4015000 comp rx=0 tx=0).stop 2022-06-27T11:59:29.158 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:29.156+0000 7faaafb0a700 1 -- 172.21.15.195:0/1982508561 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faaa806eff0 msgr2=0x7faaa80c0d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T11:59:29.158 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:29.156+0000 7faaafb0a700 1 --2- 172.21.15.195:0/1982508561 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faaa806eff0 0x7faaa80c0d20 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7faa9800c560 tx=0x7faa98007c80 comp rx=0 tx=0).stop 2022-06-27T11:59:29.159 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:29.156+0000 7faaafb0a700 1 -- 172.21.15.195:0/1982508561 shutdown_connections 2022-06-27T11:59:29.159 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:29.156+0000 7faaafb0a700 1 --2- 172.21.15.195:0/1982508561 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7faa88063d00 0x7faa880661b0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:29.159 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:29.156+0000 7faaafb0a700 1 --2- 172.21.15.195:0/1982508561 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7faaa8160c30 0x7faaa8140170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:29.159 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:29.156+0000 7faaafb0a700 1 --2- 172.21.15.195:0/1982508561 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faaa8067ed0 0x7faaa80c07e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:29.160 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:29.157+0000 7faaafb0a700 1 --2- 172.21.15.195:0/1982508561 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faaa806eff0 0x7faaa80c0d20 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T11:59:29.160 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:29.157+0000 7faaafb0a700 1 -- 172.21.15.195:0/1982508561 >> 172.21.15.195:0/1982508561 conn(0x7faaa80c9b00 msgr2=0x7faaa80c5e30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T11:59:29.160 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:29.157+0000 7faaafb0a700 1 -- 172.21.15.195:0/1982508561 shutdown_connections 2022-06-27T11:59:29.160 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T11:59:29.157+0000 7faaafb0a700 1 -- 172.21.15.195:0/1982508561 wait complete. 2022-06-27T11:59:29.161 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 48 2022-06-27T11:59:29.169 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d reported in up:standby state 2022-06-27T11:59:29.169 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 107 secs before thrashing 2022-06-27T11:59:29.340 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T11:59:29.343 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T11:59:59.436 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T11:59:59.442 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:00:29.504 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:00:29.518 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:00:59.609 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:00:59.615 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:01:16.172 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:01:16.321 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.319+0000 7faa88ed4700 1 Processor -- start 2022-06-27T12:01:16.332 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.319+0000 7faa88ed4700 1 -- start start 2022-06-27T12:01:16.341 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.320+0000 7faa88ed4700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faa840c0610 0x7faa8406b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:16.341 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.320+0000 7faa88ed4700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa840c1ba0 con 0x7faa840c3fc0 2022-06-27T12:01:16.341 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.320+0000 7faa88ed4700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa840c3720 con 0x7faa8406aca0 2022-06-27T12:01:16.341 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.320+0000 7faa88ed4700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa840c0ee0 con 0x7faa840c0610 2022-06-27T12:01:16.342 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.320+0000 7faa8259c700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7faa840c3fc0 0x7faa840c3010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:52646/0 (socket says 172.21.15.195:52646) 2022-06-27T12:01:16.342 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.320+0000 7faa8259c700 1 -- 172.21.15.195:0/1474543339 learned_addr learned my addr 172.21.15.195:0/1474543339 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:01:16.342 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.320+0000 7faa82d9d700 1 --2- 172.21.15.195:0/1474543339 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faa840c0610 0x7faa8406b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:16.342 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.320+0000 7faa8159a700 1 -- 172.21.15.195:0/1474543339 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2237678736 0 0) 0x7faa840c1ba0 con 0x7faa840c3fc0 2022-06-27T12:01:16.343 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.320+0000 7faa8159a700 1 -- 172.21.15.195:0/1474543339 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa68003430 con 0x7faa840c3fc0 2022-06-27T12:01:16.343 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.321+0000 7faa82d9d700 1 -- 172.21.15.195:0/1474543339 >> v1:172.21.15.202:6789/0 conn(0x7faa8406aca0 legacy=0x7faa840bff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:16.343 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.321+0000 7faa82d9d700 1 -- 172.21.15.195:0/1474543339 >> v1:172.21.15.195:6789/0 conn(0x7faa840c3fc0 legacy=0x7faa840c3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:16.343 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.321+0000 7faa8159a700 1 -- 172.21.15.195:0/1474543339 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 987787885 0 0) 0x7faa840c3720 con 0x7faa8406aca0 2022-06-27T12:01:16.343 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.321+0000 7faa82d9d700 1 -- 172.21.15.195:0/1474543339 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa840c0c00 con 0x7faa840c0610 2022-06-27T12:01:16.344 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.321+0000 7faa82d9d700 1 --2- 172.21.15.195:0/1474543339 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faa840c0610 0x7faa8406b0d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7faa7c009e90 tx=0x7faa7c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e68e5be9fe330e45 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:01:16.344 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.321+0000 7faa8159a700 1 -- 172.21.15.195:0/1474543339 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa7c00e070 con 0x7faa840c0610 2022-06-27T12:01:16.344 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.321+0000 7faa8159a700 1 -- 172.21.15.195:0/1474543339 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa7c00bcc0 con 0x7faa840c0610 2022-06-27T12:01:16.344 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.321+0000 7faa8159a700 1 -- 172.21.15.195:0/1474543339 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa7c012430 con 0x7faa840c0610 2022-06-27T12:01:16.345 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.322+0000 7faa88ed4700 1 -- 172.21.15.195:0/1474543339 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faa840c0610 msgr2=0x7faa8406b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:16.345 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.322+0000 7faa88ed4700 1 --2- 172.21.15.195:0/1474543339 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faa840c0610 0x7faa8406b0d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7faa7c009e90 tx=0x7faa7c009780 comp rx=0 tx=0).stop 2022-06-27T12:01:16.345 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.322+0000 7faa88ed4700 1 -- 172.21.15.195:0/1474543339 shutdown_connections 2022-06-27T12:01:16.345 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.322+0000 7faa88ed4700 1 --2- 172.21.15.195:0/1474543339 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faa840c0610 0x7faa8406b0d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:16.345 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.322+0000 7faa88ed4700 1 -- 172.21.15.195:0/1474543339 >> 172.21.15.195:0/1474543339 conn(0x7faa840c8740 msgr2=0x7faa840c8b10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:01:16.346 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.322+0000 7faa88ed4700 1 -- 172.21.15.195:0/1474543339 shutdown_connections 2022-06-27T12:01:16.346 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.322+0000 7faa88ed4700 1 -- 172.21.15.195:0/1474543339 wait complete. 2022-06-27T12:01:16.346 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.323+0000 7faa88ed4700 1 Processor -- start 2022-06-27T12:01:16.346 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.323+0000 7faa88ed4700 1 -- start start 2022-06-27T12:01:16.347 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.324+0000 7faa88ed4700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faa8406aca0 0x7faa840bce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:16.347 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.324+0000 7faa88ed4700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faa840c0610 0x7faa84166c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:16.347 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.324+0000 7faa88ed4700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7faa840c3fc0 0x7faa8416a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:16.348 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.324+0000 7faa88ed4700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa841667f0 con 0x7faa8406aca0 2022-06-27T12:01:16.348 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.324+0000 7faa88ed4700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa84166200 con 0x7faa840c3fc0 2022-06-27T12:01:16.349 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.324+0000 7faa88ed4700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa84069320 con 0x7faa840c0610 2022-06-27T12:01:16.349 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.324+0000 7faa8259c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faa8406aca0 0x7faa840bce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:16.353 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.324+0000 7faa8259c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faa8406aca0 0x7faa840bce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:46464/0 (socket says 172.21.15.195:46464) 2022-06-27T12:01:16.354 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.324+0000 7faa8259c700 1 -- 172.21.15.195:0/428598842 learned_addr learned my addr 172.21.15.195:0/428598842 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:01:16.354 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.324+0000 7faa81d9b700 1 --2- 172.21.15.195:0/428598842 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faa840c0610 0x7faa84166c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:16.354 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.324+0000 7faa82d9d700 1 --2- 172.21.15.195:0/428598842 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7faa840c3fc0 0x7faa8416a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:16.355 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.324+0000 7faa8259c700 1 -- 172.21.15.195:0/428598842 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faa840c0610 msgr2=0x7faa84166c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:16.355 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.324+0000 7faa8259c700 1 --2- 172.21.15.195:0/428598842 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faa840c0610 0x7faa84166c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:16.355 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.324+0000 7faa8259c700 1 -- 172.21.15.195:0/428598842 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7faa840c3fc0 msgr2=0x7faa8416a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:16.356 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.324+0000 7faa8259c700 1 --2- 172.21.15.195:0/428598842 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7faa840c3fc0 0x7faa8416a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:16.356 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.324+0000 7faa8259c700 1 -- 172.21.15.195:0/428598842 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa7c009a20 con 0x7faa8406aca0 2022-06-27T12:01:16.358 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.324+0000 7faa8259c700 1 --2- 172.21.15.195:0/428598842 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faa8406aca0 0x7faa840bce60 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7faa7800c440 tx=0x7faa78007c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:01:16.358 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.325+0000 7faa727fc700 1 -- 172.21.15.195:0/428598842 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa780070d0 con 0x7faa8406aca0 2022-06-27T12:01:16.358 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.325+0000 7faa727fc700 1 -- 172.21.15.195:0/428598842 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa78007230 con 0x7faa8406aca0 2022-06-27T12:01:16.359 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.325+0000 7faa88ed4700 1 -- 172.21.15.195:0/428598842 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa84068100 con 0x7faa8406aca0 2022-06-27T12:01:16.359 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.325+0000 7faa727fc700 1 -- 172.21.15.195:0/428598842 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa780075f0 con 0x7faa8406aca0 2022-06-27T12:01:16.359 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.325+0000 7faa88ed4700 1 -- 172.21.15.195:0/428598842 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa840686c0 con 0x7faa8406aca0 2022-06-27T12:01:16.359 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.325+0000 7faa88ed4700 1 -- 172.21.15.195:0/428598842 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa840722c0 con 0x7faa8406aca0 2022-06-27T12:01:16.360 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.326+0000 7faa727fc700 1 -- 172.21.15.195:0/428598842 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7faa78019030 con 0x7faa8406aca0 2022-06-27T12:01:16.360 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.327+0000 7faa727fc700 1 --2- 172.21.15.195:0/428598842 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7faa60063cd0 0x7faa60066180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:16.360 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.327+0000 7faa727fc700 1 -- 172.21.15.195:0/428598842 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6718+0+0 (secure 0 0 0) 0x7faa7807fbe0 con 0x7faa8406aca0 2022-06-27T12:01:16.360 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.327+0000 7faa81d9b700 1 --2- 172.21.15.195:0/428598842 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7faa60063cd0 0x7faa60066180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:16.360 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.327+0000 7faa81d9b700 1 --2- 172.21.15.195:0/428598842 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7faa60063cd0 0x7faa60066180 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7faa840c4e60 tx=0x7faa74009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:01:16.361 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.331+0000 7faa727fc700 1 -- 172.21.15.195:0/428598842 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7faa7804e540 con 0x7faa8406aca0 2022-06-27T12:01:16.584 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.568+0000 7faa88ed4700 1 -- 172.21.15.195:0/428598842 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faa84166ad0 con 0x7faa8406aca0 2022-06-27T12:01:16.585 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.568+0000 7faa727fc700 1 -- 172.21.15.195:0/428598842 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+6125 (secure 0 0 0) 0x7faa78017030 con 0x7faa8406aca0 2022-06-27T12:01:16.588 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:01:16.588 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/417105338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":417105338},{"type":"v1","addr":"172.21.15.202:6835","nonce":417105338}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:59:23.081414+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5192},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5192":{"gid":5192,"name":"b","rank":0,"incarnation":43,"state":"up:active","state_seq":14,"addr":"172.21.15.202:6833/2651813808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":2651813808},{"type":"v1","addr":"172.21.15.202:6833","nonce":2651813808}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:01:16.590 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.570+0000 7faa88ed4700 1 -- 172.21.15.195:0/428598842 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7faa60063cd0 msgr2=0x7faa60066180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:16.591 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.570+0000 7faa88ed4700 1 --2- 172.21.15.195:0/428598842 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7faa60063cd0 0x7faa60066180 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7faa840c4e60 tx=0x7faa74009040 comp rx=0 tx=0).stop 2022-06-27T12:01:16.592 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.570+0000 7faa88ed4700 1 -- 172.21.15.195:0/428598842 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faa8406aca0 msgr2=0x7faa840bce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:16.592 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.570+0000 7faa88ed4700 1 --2- 172.21.15.195:0/428598842 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faa8406aca0 0x7faa840bce60 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7faa7800c440 tx=0x7faa78007c70 comp rx=0 tx=0).stop 2022-06-27T12:01:16.594 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.570+0000 7faa88ed4700 1 -- 172.21.15.195:0/428598842 shutdown_connections 2022-06-27T12:01:16.595 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.570+0000 7faa88ed4700 1 --2- 172.21.15.195:0/428598842 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7faa60063cd0 0x7faa60066180 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:16.596 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.570+0000 7faa88ed4700 1 --2- 172.21.15.195:0/428598842 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faa8406aca0 0x7faa840bce60 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:16.596 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.570+0000 7faa88ed4700 1 --2- 172.21.15.195:0/428598842 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7faa840c3fc0 0x7faa8416a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:16.597 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.570+0000 7faa88ed4700 1 --2- 172.21.15.195:0/428598842 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faa840c0610 0x7faa84166c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:16.597 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.570+0000 7faa88ed4700 1 -- 172.21.15.195:0/428598842 >> 172.21.15.195:0/428598842 conn(0x7faa840c8740 msgr2=0x7faa84153350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:01:16.602 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.570+0000 7faa88ed4700 1 -- 172.21.15.195:0/428598842 shutdown_connections 2022-06-27T12:01:16.602 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.570+0000 7faa88ed4700 1 -- 172.21.15.195:0/428598842 wait complete. 2022-06-27T12:01:16.603 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 48 2022-06-27T12:01:16.603 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2022-06-27T12:01:16.604 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-06-27T12:01:16.604 INFO:teuthology.orchestra.run:waiting for 300 2022-06-27T12:01:16.651 INFO:tasks.ceph.mds.b:Stopped 2022-06-27T12:01:16.652 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:01:16.779 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.778+0000 7fdaec3e6700 1 Processor -- start 2022-06-27T12:01:16.779 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.778+0000 7fdaec3e6700 1 -- start start 2022-06-27T12:01:16.821 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.779+0000 7fdaec3e6700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fdae40c3f50 0x7fdae415ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:16.822 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.779+0000 7fdaec3e6700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdae40c33b0 con 0x7fdae414e490 2022-06-27T12:01:16.822 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.779+0000 7fdaec3e6700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdae406f220 con 0x7fdae414e860 2022-06-27T12:01:16.823 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.779+0000 7fdaec3e6700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdae41542c0 con 0x7fdae40c3f50 2022-06-27T12:01:16.823 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.779+0000 7fdaea182700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fdae414e490 0x7fdae406cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:52660/0 (socket says 172.21.15.195:52660) 2022-06-27T12:01:16.824 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.779+0000 7fdaea182700 1 -- 172.21.15.195:0/2198834738 learned_addr learned my addr 172.21.15.195:0/2198834738 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:01:16.824 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.779+0000 7fdaea983700 1 --2- 172.21.15.195:0/2198834738 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fdae40c3f50 0x7fdae415ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:16.825 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.779+0000 7fdae9180700 1 -- 172.21.15.195:0/2198834738 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1613975739 0 0) 0x7fdae40c33b0 con 0x7fdae414e490 2022-06-27T12:01:16.825 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.779+0000 7fdae9180700 1 -- 172.21.15.195:0/2198834738 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdacc003430 con 0x7fdae414e490 2022-06-27T12:01:16.826 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.780+0000 7fdae9180700 1 -- 172.21.15.195:0/2198834738 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3468746235 0 0) 0x7fdacc003430 con 0x7fdae414e490 2022-06-27T12:01:16.826 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.780+0000 7fdae9180700 1 -- 172.21.15.195:0/2198834738 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fdae40c3f50 msgr2=0x7fdae415ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:01:16.828 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.780+0000 7fdae9180700 1 --2- 172.21.15.195:0/2198834738 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fdae40c3f50 0x7fdae415ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:16.829 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.780+0000 7fdae9180700 1 -- 172.21.15.195:0/2198834738 >> v1:172.21.15.202:6789/0 conn(0x7fdae414e860 legacy=0x7fdae40c3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:16.829 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.780+0000 7fdae9180700 1 -- 172.21.15.195:0/2198834738 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdae4154560 con 0x7fdae414e490 2022-06-27T12:01:16.830 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.780+0000 7fdae9180700 1 -- 172.21.15.195:0/2198834738 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7fdad4001470 con 0x7fdae414e490 2022-06-27T12:01:16.830 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.780+0000 7fdae9180700 1 -- 172.21.15.195:0/2198834738 >> v1:172.21.15.195:6789/0 conn(0x7fdae414e490 legacy=0x7fdae406cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:16.831 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.780+0000 7fdae9180700 1 --2- 172.21.15.195:0/2198834738 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fdae40c3f50 0x7fdacc003950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:16.831 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.780+0000 7fdae9180700 1 --2- 172.21.15.195:0/2198834738 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fdacc003e90 0x7fdacc0046d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:16.831 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.780+0000 7fdae9180700 1 --2- 172.21.15.195:0/2198834738 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdacc004c10 0x7fdacc005080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:16.832 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.780+0000 7fdae9180700 1 -- 172.21.15.195:0/2198834738 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdae41542c0 con 0x7fdacc004c10 2022-06-27T12:01:16.832 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.780+0000 7fdae9180700 1 -- 172.21.15.195:0/2198834738 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdacc00ba00 con 0x7fdacc003e90 2022-06-27T12:01:16.833 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.780+0000 7fdae9180700 1 -- 172.21.15.195:0/2198834738 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdacc00bbc0 con 0x7fdae40c3f50 2022-06-27T12:01:16.833 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.780+0000 7fdaea983700 1 --2- 172.21.15.195:0/2198834738 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdacc004c10 0x7fdacc005080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:16.834 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.780+0000 7fdaea182700 1 --2- 172.21.15.195:0/2198834738 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fdae40c3f50 0x7fdacc003950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:16.834 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.780+0000 7fdae9981700 1 --2- 172.21.15.195:0/2198834738 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fdacc003e90 0x7fdacc0046d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:16.835 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.780+0000 7fdaea983700 1 -- 172.21.15.195:0/2198834738 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fdae40c3f50 msgr2=0x7fdacc003950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:01:16.835 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.780+0000 7fdaea983700 1 --2- 172.21.15.195:0/2198834738 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fdae40c3f50 0x7fdacc003950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:16.835 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.780+0000 7fdaea983700 1 -- 172.21.15.195:0/2198834738 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fdacc003e90 msgr2=0x7fdacc0046d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:01:16.836 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.780+0000 7fdaea983700 1 --2- 172.21.15.195:0/2198834738 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fdacc003e90 0x7fdacc0046d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:16.836 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.780+0000 7fdaea983700 1 -- 172.21.15.195:0/2198834738 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdacc003430 con 0x7fdacc004c10 2022-06-27T12:01:16.837 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.780+0000 7fdaea983700 1 --2- 172.21.15.195:0/2198834738 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdacc004c10 0x7fdacc005080 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fdadc00e210 tx=0x7fdadc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f89e0bfcbddeb90 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:01:16.838 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.780+0000 7fdae9180700 1 -- 172.21.15.195:0/2198834738 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdadc00eac0 con 0x7fdacc004c10 2022-06-27T12:01:16.839 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.780+0000 7fdae9180700 1 -- 172.21.15.195:0/2198834738 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdadc00f040 con 0x7fdacc004c10 2022-06-27T12:01:16.840 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.780+0000 7fdae9180700 1 -- 172.21.15.195:0/2198834738 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdadc00ee70 con 0x7fdacc004c10 2022-06-27T12:01:16.840 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.781+0000 7fdaec3e6700 1 -- 172.21.15.195:0/2198834738 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdacc004c10 msgr2=0x7fdacc005080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:16.841 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.781+0000 7fdaec3e6700 1 --2- 172.21.15.195:0/2198834738 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdacc004c10 0x7fdacc005080 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fdadc00e210 tx=0x7fdadc00b040 comp rx=0 tx=0).stop 2022-06-27T12:01:16.841 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.781+0000 7fdaea983700 1 -- 172.21.15.195:0/2198834738 reap_dead start 2022-06-27T12:01:16.841 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.781+0000 7fdaec3e6700 1 -- 172.21.15.195:0/2198834738 shutdown_connections 2022-06-27T12:01:16.842 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.781+0000 7fdaec3e6700 1 -- 172.21.15.195:0/2198834738 >> 172.21.15.195:0/2198834738 conn(0x7fdae4063af0 msgr2=0x7fdae4063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:01:16.842 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.781+0000 7fdaec3e6700 1 -- 172.21.15.195:0/2198834738 shutdown_connections 2022-06-27T12:01:16.843 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.781+0000 7fdaec3e6700 1 -- 172.21.15.195:0/2198834738 wait complete. 2022-06-27T12:01:16.843 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.782+0000 7fdaec3e6700 1 Processor -- start 2022-06-27T12:01:16.843 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.782+0000 7fdaec3e6700 1 -- start start 2022-06-27T12:01:16.844 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.783+0000 7fdaec3e6700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fdacc004c10 0x7fdae414ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:16.844 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.783+0000 7fdaec3e6700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdae4153950 0x7fdae41656b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:16.845 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.783+0000 7fdaec3e6700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fdae4165bf0 0x7fdae40c7530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:16.860 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.783+0000 7fdaec3e6700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdae4166330 con 0x7fdae4153950 2022-06-27T12:01:16.861 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.783+0000 7fdaec3e6700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdae414b320 con 0x7fdacc004c10 2022-06-27T12:01:16.861 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.783+0000 7fdaec3e6700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdae414b600 con 0x7fdae4165bf0 2022-06-27T12:01:16.862 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.783+0000 7fdae9981700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdae4153950 0x7fdae41656b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:16.862 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.783+0000 7fdae9981700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdae4153950 0x7fdae41656b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:46486/0 (socket says 172.21.15.195:46486) 2022-06-27T12:01:16.862 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.783+0000 7fdae9981700 1 -- 172.21.15.195:0/415529482 learned_addr learned my addr 172.21.15.195:0/415529482 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:01:16.863 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.783+0000 7fdaea182700 1 --2- 172.21.15.195:0/415529482 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fdacc004c10 0x7fdae414ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:16.863 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.783+0000 7fdaea983700 1 --2- 172.21.15.195:0/415529482 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fdae4165bf0 0x7fdae40c7530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:16.864 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.783+0000 7fdae9981700 1 -- 172.21.15.195:0/415529482 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fdae4165bf0 msgr2=0x7fdae40c7530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:16.864 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.783+0000 7fdae9981700 1 --2- 172.21.15.195:0/415529482 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fdae4165bf0 0x7fdae40c7530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:16.866 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.783+0000 7fdae9981700 1 -- 172.21.15.195:0/415529482 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fdacc004c10 msgr2=0x7fdae414ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:16.867 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.783+0000 7fdae9981700 1 --2- 172.21.15.195:0/415529482 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fdacc004c10 0x7fdae414ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:16.867 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.783+0000 7fdae9981700 1 -- 172.21.15.195:0/415529482 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdae414b8e0 con 0x7fdae4153950 2022-06-27T12:01:16.868 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.783+0000 7fdae9981700 1 --2- 172.21.15.195:0/415529482 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdae4153950 0x7fdae41656b0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fdae000f8d0 tx=0x7fdae0009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:01:16.868 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.784+0000 7fdad27fc700 1 -- 172.21.15.195:0/415529482 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdae00052e0 con 0x7fdae4153950 2022-06-27T12:01:16.869 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.784+0000 7fdad27fc700 1 -- 172.21.15.195:0/415529482 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdae0003500 con 0x7fdae4153950 2022-06-27T12:01:16.869 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.784+0000 7fdaec3e6700 1 -- 172.21.15.195:0/415529482 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdae414bbc0 con 0x7fdae4153950 2022-06-27T12:01:16.870 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.784+0000 7fdad27fc700 1 -- 172.21.15.195:0/415529482 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdae00038c0 con 0x7fdae4153950 2022-06-27T12:01:16.871 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.784+0000 7fdaec3e6700 1 -- 172.21.15.195:0/415529482 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdae4153da0 con 0x7fdae4153950 2022-06-27T12:01:16.871 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.785+0000 7fdaec3e6700 1 -- 172.21.15.195:0/415529482 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdae40722c0 con 0x7fdae4153950 2022-06-27T12:01:16.872 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.785+0000 7fdad27fc700 1 -- 172.21.15.195:0/415529482 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fdae0003cb0 con 0x7fdae4153950 2022-06-27T12:01:16.872 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.785+0000 7fdad27fc700 1 --2- 172.21.15.195:0/415529482 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fdac4063d30 0x7fdac40661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:16.872 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.785+0000 7fdad27fc700 1 -- 172.21.15.195:0/415529482 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6718+0+0 (secure 0 0 0) 0x7fdae0080c20 con 0x7fdae4153950 2022-06-27T12:01:16.873 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.785+0000 7fdaea182700 1 --2- 172.21.15.195:0/415529482 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fdac4063d30 0x7fdac40661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:16.873 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.786+0000 7fdaea182700 1 --2- 172.21.15.195:0/415529482 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fdac4063d30 0x7fdac40661e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fdad40078a0 tx=0x7fdad4001bc0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:01:16.874 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:16.790+0000 7fdad27fc700 1 -- 172.21.15.195:0/415529482 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fdae004f580 con 0x7fdae4153950 2022-06-27T12:01:17.016 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:17.011+0000 7fdaec3e6700 1 -- 172.21.15.195:0/415529482 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdae40c7e90 con 0x7fdae4153950 2022-06-27T12:01:17.016 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:17.011+0000 7fdad27fc700 1 -- 172.21.15.195:0/415529482 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+6125 (secure 0 0 0) 0x7fdae0080020 con 0x7fdae4153950 2022-06-27T12:01:17.016 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:01:17.017 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/417105338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":417105338},{"type":"v1","addr":"172.21.15.202:6835","nonce":417105338}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:59:23.081414+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5192},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5192":{"gid":5192,"name":"b","rank":0,"incarnation":43,"state":"up:active","state_seq":14,"addr":"172.21.15.202:6833/2651813808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":2651813808},{"type":"v1","addr":"172.21.15.202:6833","nonce":2651813808}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:01:17.022 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:17.014+0000 7fdaec3e6700 1 -- 172.21.15.195:0/415529482 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fdac4063d30 msgr2=0x7fdac40661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:17.022 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:17.014+0000 7fdaec3e6700 1 --2- 172.21.15.195:0/415529482 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fdac4063d30 0x7fdac40661e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fdad40078a0 tx=0x7fdad4001bc0 comp rx=0 tx=0).stop 2022-06-27T12:01:17.025 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:17.014+0000 7fdaec3e6700 1 -- 172.21.15.195:0/415529482 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdae4153950 msgr2=0x7fdae41656b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:17.026 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:17.014+0000 7fdaec3e6700 1 --2- 172.21.15.195:0/415529482 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdae4153950 0x7fdae41656b0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fdae000f8d0 tx=0x7fdae0009040 comp rx=0 tx=0).stop 2022-06-27T12:01:17.042 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:17.014+0000 7fdaec3e6700 1 -- 172.21.15.195:0/415529482 shutdown_connections 2022-06-27T12:01:17.043 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:17.014+0000 7fdaec3e6700 1 --2- 172.21.15.195:0/415529482 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fdac4063d30 0x7fdac40661e0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:17.043 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:17.014+0000 7fdaec3e6700 1 --2- 172.21.15.195:0/415529482 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fdae4165bf0 0x7fdae40c7530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:17.044 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:17.014+0000 7fdaec3e6700 1 --2- 172.21.15.195:0/415529482 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fdacc004c10 0x7fdae414ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:17.044 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:17.014+0000 7fdaec3e6700 1 --2- 172.21.15.195:0/415529482 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdae4153950 0x7fdae41656b0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:17.045 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:17.014+0000 7fdaec3e6700 1 -- 172.21.15.195:0/415529482 >> 172.21.15.195:0/415529482 conn(0x7fdae4063af0 msgr2=0x7fdae40c3740 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:01:17.051 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:17.014+0000 7fdaec3e6700 1 -- 172.21.15.195:0/415529482 shutdown_connections 2022-06-27T12:01:17.051 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:17.014+0000 7fdaec3e6700 1 -- 172.21.15.195:0/415529482 wait complete. 2022-06-27T12:01:17.051 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 48 2022-06-27T12:01:17.054 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2022-06-27T12:01:19.055 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:01:19.205 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.195+0000 7fa45e5ea700 1 Processor -- start 2022-06-27T12:01:19.205 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.195+0000 7fa45e5ea700 1 -- start start 2022-06-27T12:01:19.205 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.196+0000 7fa45e5ea700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa45806f640 0x7fa458156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:19.206 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.196+0000 7fa45e5ea700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fa4580c4160 0x7fa4580c1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:19.206 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.196+0000 7fa45e5ea700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4580c1a20 con 0x7fa4580c39e0 2022-06-27T12:01:19.206 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.196+0000 7fa45e5ea700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4580c1130 con 0x7fa4580c4160 2022-06-27T12:01:19.206 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.196+0000 7fa45e5ea700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa45806ea90 con 0x7fa45806f640 2022-06-27T12:01:19.207 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.196+0000 7fa457fff700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa45806f640 0x7fa458156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:19.207 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.196+0000 7fa4577fe700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fa4580c4160 0x7fa4580c1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:19.207 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.196+0000 7fa4577fe700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fa4580c4160 0x7fa4580c1eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3300/0 says I am v2:172.21.15.195:44630/0 (socket says 172.21.15.195:44630) 2022-06-27T12:01:19.207 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.196+0000 7fa457fff700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa45806f640 0x7fa458156ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3301/0 says I am v2:172.21.15.195:39256/0 (socket says 172.21.15.195:39256) 2022-06-27T12:01:19.208 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.196+0000 7fa4577fe700 1 -- 172.21.15.195:0/1982019435 learned_addr learned my addr 172.21.15.195:0/1982019435 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:01:19.208 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.196+0000 7fa45cb87700 1 --1- >> v1:172.21.15.202:6789/0 conn(0x7fa4580c39e0 0x7fa4581535a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.202:6789/0 says I am v1:172.21.15.195:54550/0 (socket says 172.21.15.195:54550) 2022-06-27T12:01:19.208 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.197+0000 7fa4577fe700 1 -- 172.21.15.195:0/1982019435 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa45806f640 msgr2=0x7fa458156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:01:19.209 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.197+0000 7fa4577fe700 1 --2- 172.21.15.195:0/1982019435 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa45806f640 0x7fa458156ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:19.209 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.197+0000 7fa4577fe700 1 -- 172.21.15.195:0/1982019435 >> v1:172.21.15.202:6789/0 conn(0x7fa4580c39e0 legacy=0x7fa4581535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:19.209 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.197+0000 7fa4577fe700 1 -- 172.21.15.195:0/1982019435 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa45806ed70 con 0x7fa4580c4160 2022-06-27T12:01:19.209 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.197+0000 7fa456ffd700 1 -- 172.21.15.195:0/1982019435 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3075055775 0 0) 0x7fa4580c1a20 con 0x7fa4580c39e0 2022-06-27T12:01:19.210 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.197+0000 7fa457fff700 1 --2- 172.21.15.195:0/1982019435 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa45806f640 0x7fa458156ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:01:19.210 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.197+0000 7fa4577fe700 1 --2- 172.21.15.195:0/1982019435 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa4580c4160 0x7fa4580c1eb0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fa448009860 tx=0x7fa448009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f418381b73ba16f7 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:01:19.210 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.197+0000 7fa456ffd700 1 -- 172.21.15.195:0/1982019435 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa44800b690 con 0x7fa4580c4160 2022-06-27T12:01:19.210 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.197+0000 7fa456ffd700 1 -- 172.21.15.195:0/1982019435 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa44800b7f0 con 0x7fa4580c4160 2022-06-27T12:01:19.211 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.197+0000 7fa456ffd700 1 -- 172.21.15.195:0/1982019435 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa44800bb90 con 0x7fa4580c4160 2022-06-27T12:01:19.211 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.197+0000 7fa45e5ea700 1 -- 172.21.15.195:0/1982019435 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa4580c4160 msgr2=0x7fa4580c1eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:19.211 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.197+0000 7fa45e5ea700 1 --2- 172.21.15.195:0/1982019435 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa4580c4160 0x7fa4580c1eb0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fa448009860 tx=0x7fa448009120 comp rx=0 tx=0).stop 2022-06-27T12:01:19.211 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.198+0000 7fa45e5ea700 1 -- 172.21.15.195:0/1982019435 shutdown_connections 2022-06-27T12:01:19.212 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.198+0000 7fa45e5ea700 1 --2- 172.21.15.195:0/1982019435 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa45806f640 0x7fa458156ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:19.212 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.198+0000 7fa45e5ea700 1 --2- 172.21.15.195:0/1982019435 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa4580c4160 0x7fa4580c1eb0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:19.212 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.198+0000 7fa45e5ea700 1 -- 172.21.15.195:0/1982019435 >> 172.21.15.195:0/1982019435 conn(0x7fa4580c6350 msgr2=0x7fa4580c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:01:19.212 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.198+0000 7fa45e5ea700 1 -- 172.21.15.195:0/1982019435 shutdown_connections 2022-06-27T12:01:19.213 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.198+0000 7fa45e5ea700 1 -- 172.21.15.195:0/1982019435 wait complete. 2022-06-27T12:01:19.213 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.199+0000 7fa45e5ea700 1 Processor -- start 2022-06-27T12:01:19.213 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.199+0000 7fa45e5ea700 1 -- start start 2022-06-27T12:01:19.213 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.199+0000 7fa45e5ea700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa45806f640 0x7fa4580baac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:19.214 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.199+0000 7fa45e5ea700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa4580c39e0 0x7fa45814cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:19.214 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.200+0000 7fa45e5ea700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa4580c4160 0x7fa45814f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:19.214 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.200+0000 7fa45e5ea700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4580bfba0 con 0x7fa4580c39e0 2022-06-27T12:01:19.214 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.200+0000 7fa45e5ea700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4580be3b0 con 0x7fa45806f640 2022-06-27T12:01:19.215 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.200+0000 7fa45e5ea700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4580ba630 con 0x7fa4580c4160 2022-06-27T12:01:19.215 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.200+0000 7fa4577fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa4580c39e0 0x7fa45814cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:19.216 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.200+0000 7fa4577fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa4580c39e0 0x7fa45814cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:46508/0 (socket says 172.21.15.195:46508) 2022-06-27T12:01:19.216 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.200+0000 7fa4577fe700 1 -- 172.21.15.195:0/432757058 learned_addr learned my addr 172.21.15.195:0/432757058 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:01:19.216 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.200+0000 7fa45cb87700 1 --2- 172.21.15.195:0/432757058 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa4580c4160 0x7fa45814f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:19.217 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.200+0000 7fa457fff700 1 --2- 172.21.15.195:0/432757058 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa45806f640 0x7fa4580baac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:19.217 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.200+0000 7fa4577fe700 1 -- 172.21.15.195:0/432757058 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa4580c4160 msgr2=0x7fa45814f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:19.217 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.200+0000 7fa4577fe700 1 --2- 172.21.15.195:0/432757058 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa4580c4160 0x7fa45814f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:19.217 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.200+0000 7fa4577fe700 1 -- 172.21.15.195:0/432757058 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa45806f640 msgr2=0x7fa4580baac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:19.218 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.200+0000 7fa4577fe700 1 --2- 172.21.15.195:0/432757058 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa45806f640 0x7fa4580baac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:19.218 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.200+0000 7fa4577fe700 1 -- 172.21.15.195:0/432757058 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa44c0092e0 con 0x7fa4580c39e0 2022-06-27T12:01:19.218 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.200+0000 7fa4577fe700 1 --2- 172.21.15.195:0/432757058 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa4580c39e0 0x7fa45814cd40 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fa4580650f0 tx=0x7fa448017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:01:19.218 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.200+0000 7fa43ffff700 1 -- 172.21.15.195:0/432757058 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa448003730 con 0x7fa4580c39e0 2022-06-27T12:01:19.219 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.200+0000 7fa43ffff700 1 -- 172.21.15.195:0/432757058 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa448003890 con 0x7fa4580c39e0 2022-06-27T12:01:19.219 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.200+0000 7fa45e5ea700 1 -- 172.21.15.195:0/432757058 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4480093a0 con 0x7fa4580c39e0 2022-06-27T12:01:19.219 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.200+0000 7fa43ffff700 1 -- 172.21.15.195:0/432757058 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa448003c30 con 0x7fa4580c39e0 2022-06-27T12:01:19.220 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.200+0000 7fa45e5ea700 1 -- 172.21.15.195:0/432757058 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4580ba330 con 0x7fa4580c39e0 2022-06-27T12:01:19.220 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.201+0000 7fa45e5ea700 1 -- 172.21.15.195:0/432757058 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4580ba7a0 con 0x7fa4580c39e0 2022-06-27T12:01:19.220 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.202+0000 7fa43ffff700 1 -- 172.21.15.195:0/432757058 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fa44800f050 con 0x7fa4580c39e0 2022-06-27T12:01:19.220 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.202+0000 7fa43ffff700 1 --2- 172.21.15.195:0/432757058 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa438063d00 0x7fa4380661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:19.221 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.202+0000 7fa43ffff700 1 -- 172.21.15.195:0/432757058 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6718+0+0 (secure 0 0 0) 0x7fa4480821e0 con 0x7fa4580c39e0 2022-06-27T12:01:19.221 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.202+0000 7fa457fff700 1 --2- 172.21.15.195:0/432757058 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa438063d00 0x7fa4380661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:19.221 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.202+0000 7fa457fff700 1 --2- 172.21.15.195:0/432757058 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa438063d00 0x7fa4380661b0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fa44c0092b0 tx=0x7fa44c008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:01:19.222 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.206+0000 7fa43ffff700 1 -- 172.21.15.195:0/432757058 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fa448050b40 con 0x7fa4580c39e0 2022-06-27T12:01:19.437 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.434+0000 7fa45e5ea700 1 -- 172.21.15.195:0/432757058 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa4580722c0 con 0x7fa4580c39e0 2022-06-27T12:01:19.438 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.434+0000 7fa43ffff700 1 -- 172.21.15.195:0/432757058 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+6125 (secure 0 0 0) 0x7fa44802a020 con 0x7fa4580c39e0 2022-06-27T12:01:19.439 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:01:19.439 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/417105338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":417105338},{"type":"v1","addr":"172.21.15.202:6835","nonce":417105338}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:59:23.081414+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5192},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5192":{"gid":5192,"name":"b","rank":0,"incarnation":43,"state":"up:active","state_seq":14,"addr":"172.21.15.202:6833/2651813808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":2651813808},{"type":"v1","addr":"172.21.15.202:6833","nonce":2651813808}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:01:19.441 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.436+0000 7fa45e5ea700 1 -- 172.21.15.195:0/432757058 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa438063d00 msgr2=0x7fa4380661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:19.445 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.436+0000 7fa45e5ea700 1 --2- 172.21.15.195:0/432757058 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa438063d00 0x7fa4380661b0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fa44c0092b0 tx=0x7fa44c008040 comp rx=0 tx=0).stop 2022-06-27T12:01:19.446 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.436+0000 7fa45e5ea700 1 -- 172.21.15.195:0/432757058 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa4580c39e0 msgr2=0x7fa45814cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:19.447 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.436+0000 7fa45e5ea700 1 --2- 172.21.15.195:0/432757058 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa4580c39e0 0x7fa45814cd40 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fa4580650f0 tx=0x7fa448017000 comp rx=0 tx=0).stop 2022-06-27T12:01:19.447 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.436+0000 7fa45e5ea700 1 -- 172.21.15.195:0/432757058 shutdown_connections 2022-06-27T12:01:19.448 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.436+0000 7fa45e5ea700 1 --2- 172.21.15.195:0/432757058 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa438063d00 0x7fa4380661b0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:19.448 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.436+0000 7fa45e5ea700 1 --2- 172.21.15.195:0/432757058 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa4580c4160 0x7fa45814f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:19.449 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.436+0000 7fa45e5ea700 1 --2- 172.21.15.195:0/432757058 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa45806f640 0x7fa4580baac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:19.449 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.436+0000 7fa45e5ea700 1 --2- 172.21.15.195:0/432757058 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa4580c39e0 0x7fa45814cd40 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:19.450 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.436+0000 7fa45e5ea700 1 -- 172.21.15.195:0/432757058 >> 172.21.15.195:0/432757058 conn(0x7fa4580c6350 msgr2=0x7fa458152870 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:01:19.450 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.436+0000 7fa45e5ea700 1 -- 172.21.15.195:0/432757058 shutdown_connections 2022-06-27T12:01:19.451 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:19.436+0000 7fa45e5ea700 1 -- 172.21.15.195:0/432757058 wait complete. 2022-06-27T12:01:19.451 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 48 2022-06-27T12:01:19.458 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2022-06-27T12:01:21.460 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:01:21.601 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.596+0000 7f0629b05700 1 Processor -- start 2022-06-27T12:01:21.602 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.596+0000 7f0629b05700 1 -- start start 2022-06-27T12:01:21.602 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.597+0000 7f0629b05700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f062406fef0 0x7f06241570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:21.602 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.597+0000 7f0629b05700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f06240c34a0 0x7f06240c2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:21.602 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.597+0000 7f0629b05700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f062406fba0 con 0x7f062406e850 2022-06-27T12:01:21.603 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.597+0000 7f0629b05700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06241575e0 con 0x7f06240c34a0 2022-06-27T12:01:21.603 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.597+0000 7f0629b05700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f062406f5e0 con 0x7f062406fef0 2022-06-27T12:01:21.603 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.597+0000 7f0623fff700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f06240c34a0 0x7f06240c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:21.603 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.597+0000 7f0623fff700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f06240c34a0 0x7f06240c2920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3300/0 says I am v2:172.21.15.195:44646/0 (socket says 172.21.15.195:44646) 2022-06-27T12:01:21.604 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.597+0000 7f0623fff700 1 -- 172.21.15.195:0/4056951672 learned_addr learned my addr 172.21.15.195:0/4056951672 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:01:21.604 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.597+0000 7f0622ffd700 1 --2- 172.21.15.195:0/4056951672 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f062406fef0 0x7f06241570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:21.613 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.597+0000 7f0623fff700 1 -- 172.21.15.195:0/4056951672 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f062406fef0 msgr2=0x7f06241570a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:01:21.613 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.597+0000 7f0623fff700 1 --2- 172.21.15.195:0/4056951672 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f062406fef0 0x7f06241570a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:21.614 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.597+0000 7f0623fff700 1 -- 172.21.15.195:0/4056951672 >> v1:172.21.15.202:6789/0 conn(0x7f062406e850 legacy=0x7f0624069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:21.614 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.597+0000 7f0623fff700 1 -- 172.21.15.195:0/4056951672 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f062406f8c0 con 0x7f06240c34a0 2022-06-27T12:01:21.614 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.598+0000 7f0623fff700 1 --2- 172.21.15.195:0/4056951672 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f06240c34a0 0x7f06240c2920 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f061c009f20 tx=0x7f061c009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c6778eef0cb6fa27 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:01:21.615 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.598+0000 7f06227fc700 1 -- 172.21.15.195:0/4056951672 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f061c00e070 con 0x7f06240c34a0 2022-06-27T12:01:21.615 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.598+0000 7f06227fc700 1 -- 172.21.15.195:0/4056951672 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f061c00ba60 con 0x7f06240c34a0 2022-06-27T12:01:21.616 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.598+0000 7f06227fc700 1 -- 172.21.15.195:0/4056951672 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f061c0092e0 con 0x7f06240c34a0 2022-06-27T12:01:21.616 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.598+0000 7f0629b05700 1 -- 172.21.15.195:0/4056951672 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f06240c34a0 msgr2=0x7f06240c2920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:21.616 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.598+0000 7f0629b05700 1 --2- 172.21.15.195:0/4056951672 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f06240c34a0 0x7f06240c2920 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f061c009f20 tx=0x7f061c009840 comp rx=0 tx=0).stop 2022-06-27T12:01:21.617 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.598+0000 7f0629b05700 1 -- 172.21.15.195:0/4056951672 shutdown_connections 2022-06-27T12:01:21.617 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.598+0000 7f0629b05700 1 --2- 172.21.15.195:0/4056951672 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f06240c34a0 0x7f06240c2920 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:21.617 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.598+0000 7f0629b05700 1 --2- 172.21.15.195:0/4056951672 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f062406fef0 0x7f06241570a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:21.617 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.598+0000 7f0629b05700 1 -- 172.21.15.195:0/4056951672 >> 172.21.15.195:0/4056951672 conn(0x7f06240c6350 msgr2=0x7f06240c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:01:21.618 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.598+0000 7f0629b05700 1 -- 172.21.15.195:0/4056951672 shutdown_connections 2022-06-27T12:01:21.619 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.598+0000 7f0629b05700 1 -- 172.21.15.195:0/4056951672 wait complete. 2022-06-27T12:01:21.619 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.600+0000 7f0629b05700 1 Processor -- start 2022-06-27T12:01:21.620 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.600+0000 7f0629b05700 1 -- start start 2022-06-27T12:01:21.620 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.600+0000 7f0629b05700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f062406e850 0x7f06241532b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:21.621 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.600+0000 7f0629b05700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f06240c34a0 0x7f0624153ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:21.621 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.600+0000 7f0629b05700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0624150ff0 0x7f06240c0200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:21.621 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.600+0000 7f0629b05700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06241537f0 con 0x7f06240c34a0 2022-06-27T12:01:21.626 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.600+0000 7f0629b05700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0624152e20 con 0x7f0624150ff0 2022-06-27T12:01:21.627 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.600+0000 7f0629b05700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f062414b0b0 con 0x7f062406e850 2022-06-27T12:01:21.627 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.601+0000 7f0622ffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f06240c34a0 0x7f0624153ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:21.628 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.601+0000 7f0622ffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f06240c34a0 0x7f0624153ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:46524/0 (socket says 172.21.15.195:46524) 2022-06-27T12:01:21.628 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.601+0000 7f0622ffd700 1 -- 172.21.15.195:0/2164275181 learned_addr learned my addr 172.21.15.195:0/2164275181 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:01:21.629 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.601+0000 7f06237fe700 1 --2- 172.21.15.195:0/2164275181 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f062406e850 0x7f06241532b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:21.630 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.601+0000 7f0623fff700 1 --2- 172.21.15.195:0/2164275181 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0624150ff0 0x7f06240c0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:21.630 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.601+0000 7f0622ffd700 1 -- 172.21.15.195:0/2164275181 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f062406e850 msgr2=0x7f06241532b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:21.630 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.601+0000 7f0622ffd700 1 --2- 172.21.15.195:0/2164275181 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f062406e850 0x7f06241532b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:21.631 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.601+0000 7f0622ffd700 1 -- 172.21.15.195:0/2164275181 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0624150ff0 msgr2=0x7f06240c0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:21.631 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.601+0000 7f0622ffd700 1 --2- 172.21.15.195:0/2164275181 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0624150ff0 0x7f06240c0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:21.632 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.601+0000 7f0622ffd700 1 -- 172.21.15.195:0/2164275181 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f061c009ae0 con 0x7f06240c34a0 2022-06-27T12:01:21.632 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.601+0000 7f0622ffd700 1 --2- 172.21.15.195:0/2164275181 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f06240c34a0 0x7f0624153ca0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f061800e900 tx=0x7f0618009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:01:21.633 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.601+0000 7f060b7fe700 1 -- 172.21.15.195:0/2164275181 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f061800ee40 con 0x7f06240c34a0 2022-06-27T12:01:21.633 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.601+0000 7f060b7fe700 1 -- 172.21.15.195:0/2164275181 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f061800c850 con 0x7f06240c34a0 2022-06-27T12:01:21.634 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.601+0000 7f0629b05700 1 -- 172.21.15.195:0/2164275181 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f062414b390 con 0x7f06240c34a0 2022-06-27T12:01:21.634 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.601+0000 7f060b7fe700 1 -- 172.21.15.195:0/2164275181 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f061800cbd0 con 0x7f06240c34a0 2022-06-27T12:01:21.635 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.601+0000 7f0629b05700 1 -- 172.21.15.195:0/2164275181 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0624152ab0 con 0x7f06240c34a0 2022-06-27T12:01:21.635 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.602+0000 7f0629b05700 1 -- 172.21.15.195:0/2164275181 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06240722c0 con 0x7f06240c34a0 2022-06-27T12:01:21.636 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.603+0000 7f060b7fe700 1 -- 172.21.15.195:0/2164275181 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f0618019030 con 0x7f06240c34a0 2022-06-27T12:01:21.637 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.603+0000 7f060b7fe700 1 --2- 172.21.15.195:0/2164275181 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0604063d30 0x7f06040661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:21.637 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.603+0000 7f060b7fe700 1 -- 172.21.15.195:0/2164275181 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6718+0+0 (secure 0 0 0) 0x7f0618080a00 con 0x7f06240c34a0 2022-06-27T12:01:21.638 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.603+0000 7f06237fe700 1 --2- 172.21.15.195:0/2164275181 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0604063d30 0x7f06040661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:21.638 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.603+0000 7f06237fe700 1 --2- 172.21.15.195:0/2164275181 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0604063d30 0x7f06040661e0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f0614001670 tx=0x7f0614009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:01:21.639 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.607+0000 7f060b7fe700 1 -- 172.21.15.195:0/2164275181 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f061804f360 con 0x7f06240c34a0 2022-06-27T12:01:21.835 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.834+0000 7f0629b05700 1 -- 172.21.15.195:0/2164275181 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0624152f60 con 0x7f06240c34a0 2022-06-27T12:01:21.835 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.834+0000 7f060b7fe700 1 -- 172.21.15.195:0/2164275181 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+6125 (secure 0 0 0) 0x7f0618017090 con 0x7f06240c34a0 2022-06-27T12:01:21.839 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:01:21.839 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/417105338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":417105338},{"type":"v1","addr":"172.21.15.202:6835","nonce":417105338}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:59:23.081414+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5192},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5192":{"gid":5192,"name":"b","rank":0,"incarnation":43,"state":"up:active","state_seq":14,"addr":"172.21.15.202:6833/2651813808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":2651813808},{"type":"v1","addr":"172.21.15.202:6833","nonce":2651813808}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:01:21.841 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.836+0000 7f0629b05700 1 -- 172.21.15.195:0/2164275181 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0604063d30 msgr2=0x7f06040661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:21.842 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.836+0000 7f0629b05700 1 --2- 172.21.15.195:0/2164275181 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0604063d30 0x7f06040661e0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f0614001670 tx=0x7f0614009040 comp rx=0 tx=0).stop 2022-06-27T12:01:21.842 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.836+0000 7f0629b05700 1 -- 172.21.15.195:0/2164275181 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f06240c34a0 msgr2=0x7f0624153ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:21.842 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.836+0000 7f0629b05700 1 --2- 172.21.15.195:0/2164275181 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f06240c34a0 0x7f0624153ca0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f061800e900 tx=0x7f0618009060 comp rx=0 tx=0).stop 2022-06-27T12:01:21.843 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.836+0000 7f0629b05700 1 -- 172.21.15.195:0/2164275181 shutdown_connections 2022-06-27T12:01:21.843 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.836+0000 7f0629b05700 1 --2- 172.21.15.195:0/2164275181 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0604063d30 0x7f06040661e0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:21.844 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.836+0000 7f0629b05700 1 --2- 172.21.15.195:0/2164275181 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0624150ff0 0x7f06240c0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:21.853 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.836+0000 7f0629b05700 1 --2- 172.21.15.195:0/2164275181 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f062406e850 0x7f06241532b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:21.853 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.836+0000 7f0629b05700 1 --2- 172.21.15.195:0/2164275181 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f06240c34a0 0x7f0624153ca0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:21.854 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.836+0000 7f0629b05700 1 -- 172.21.15.195:0/2164275181 >> 172.21.15.195:0/2164275181 conn(0x7f06240c6350 msgr2=0x7f0624067ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:01:21.857 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.837+0000 7f0629b05700 1 -- 172.21.15.195:0/2164275181 shutdown_connections 2022-06-27T12:01:21.858 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:21.837+0000 7f0629b05700 1 -- 172.21.15.195:0/2164275181 wait complete. 2022-06-27T12:01:21.859 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 48 2022-06-27T12:01:21.860 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2022-06-27T12:01:23.861 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:01:24.009 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.002+0000 7f6a38179700 1 Processor -- start 2022-06-27T12:01:24.010 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.002+0000 7f6a38179700 1 -- start start 2022-06-27T12:01:24.010 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.003+0000 7f6a38179700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f6a3006eff0 0x7f6a300c61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:24.010 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.003+0000 7f6a38179700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6a300c31d0 0x7f6a300c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:24.011 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.003+0000 7f6a38179700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a3014d8a0 con 0x7f6a30067ed0 2022-06-27T12:01:24.011 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.003+0000 7f6a38179700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a300c47e0 con 0x7f6a3006eff0 2022-06-27T12:01:24.011 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.003+0000 7f6a38179700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a300c3e10 con 0x7f6a300c31d0 2022-06-27T12:01:24.011 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.003+0000 7f6a35714700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f6a30067ed0 0x7f6a300c2ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:52718/0 (socket says 172.21.15.195:52718) 2022-06-27T12:01:24.012 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.003+0000 7f6a35714700 1 -- 172.21.15.195:0/3157968346 learned_addr learned my addr 172.21.15.195:0/3157968346 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:01:24.012 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.003+0000 7f6a36716700 1 --2- 172.21.15.195:0/3157968346 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6a300c31d0 0x7f6a300c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:24.013 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.003+0000 7f6a35f15700 1 --2- 172.21.15.195:0/3157968346 >> v2:172.21.15.202:3300/0 conn(0x7f6a3006eff0 0x7f6a300c61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:24.013 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.003+0000 7f6a34f13700 1 -- 172.21.15.195:0/3157968346 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3905017721 0 0) 0x7f6a3014d8a0 con 0x7f6a30067ed0 2022-06-27T12:01:24.014 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.004+0000 7f6a34f13700 1 -- 172.21.15.195:0/3157968346 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a18003430 con 0x7f6a30067ed0 2022-06-27T12:01:24.014 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.004+0000 7f6a34f13700 1 -- 172.21.15.195:0/3157968346 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 602802270 0 0) 0x7f6a18003430 con 0x7f6a30067ed0 2022-06-27T12:01:24.014 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.004+0000 7f6a34f13700 1 -- 172.21.15.195:0/3157968346 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6a300c31d0 msgr2=0x7f6a300c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:01:24.014 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.004+0000 7f6a34f13700 1 --2- 172.21.15.195:0/3157968346 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6a300c31d0 0x7f6a300c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:24.015 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.004+0000 7f6a34f13700 1 -- 172.21.15.195:0/3157968346 >> v2:172.21.15.202:3300/0 conn(0x7f6a3006eff0 msgr2=0x7f6a300c61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:01:24.015 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.004+0000 7f6a34f13700 1 --2- 172.21.15.195:0/3157968346 >> v2:172.21.15.202:3300/0 conn(0x7f6a3006eff0 0x7f6a300c61e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:24.015 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.004+0000 7f6a34f13700 1 -- 172.21.15.195:0/3157968346 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a3006d970 con 0x7f6a30067ed0 2022-06-27T12:01:24.015 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.004+0000 7f6a34f13700 1 -- 172.21.15.195:0/3157968346 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7f6a2c001470 con 0x7f6a30067ed0 2022-06-27T12:01:24.016 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.004+0000 7f6a35f15700 1 --2- 172.21.15.195:0/3157968346 >> v2:172.21.15.202:3300/0 conn(0x7f6a3006eff0 0x7f6a300c61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:01:24.016 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.004+0000 7f6a36716700 1 --2- 172.21.15.195:0/3157968346 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6a300c31d0 0x7f6a300c42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:01:24.016 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.004+0000 7f6a34f13700 1 -- 172.21.15.195:0/3157968346 >> v1:172.21.15.195:6789/0 conn(0x7f6a30067ed0 legacy=0x7f6a300c2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:24.016 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.004+0000 7f6a34f13700 1 --2- 172.21.15.195:0/3157968346 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6a18003920 0x7f6a18003d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:24.017 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.004+0000 7f6a34f13700 1 --2- 172.21.15.195:0/3157968346 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6a180046d0 0x7f6a18004b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:24.017 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.004+0000 7f6a34f13700 1 --2- 172.21.15.195:0/3157968346 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6a18005080 0x7f6a1800b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:24.017 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.004+0000 7f6a34f13700 1 -- 172.21.15.195:0/3157968346 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a300c47e0 con 0x7f6a180046d0 2022-06-27T12:01:24.017 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.004+0000 7f6a34f13700 1 -- 172.21.15.195:0/3157968346 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a300c3e10 con 0x7f6a18003920 2022-06-27T12:01:24.017 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.004+0000 7f6a34f13700 1 -- 172.21.15.195:0/3157968346 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a18005520 con 0x7f6a18005080 2022-06-27T12:01:24.019 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.004+0000 7f6a35714700 1 --2- 172.21.15.195:0/3157968346 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6a180046d0 0x7f6a18004b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:24.019 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.004+0000 7f6a35f15700 1 --2- 172.21.15.195:0/3157968346 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6a18003920 0x7f6a18003d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:24.019 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.004+0000 7f6a36716700 1 --2- 172.21.15.195:0/3157968346 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6a18005080 0x7f6a1800b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:24.019 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.004+0000 7f6a35714700 1 -- 172.21.15.195:0/3157968346 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6a18005080 msgr2=0x7f6a1800b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:01:24.020 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.004+0000 7f6a35714700 1 --2- 172.21.15.195:0/3157968346 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6a18005080 0x7f6a1800b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:24.020 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.004+0000 7f6a35714700 1 -- 172.21.15.195:0/3157968346 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6a18003920 msgr2=0x7f6a18003d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:01:24.020 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.004+0000 7f6a35714700 1 --2- 172.21.15.195:0/3157968346 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6a18003920 0x7f6a18003d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:24.020 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.004+0000 7f6a35714700 1 -- 172.21.15.195:0/3157968346 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a28009a20 con 0x7f6a180046d0 2022-06-27T12:01:24.021 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.004+0000 7f6a35714700 1 --2- 172.21.15.195:0/3157968346 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6a180046d0 0x7f6a18004b40 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c00c2d0 tx=0x7f6a301530d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9a1cfc6d61b5e11 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:01:24.021 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.005+0000 7f6a34f13700 1 -- 172.21.15.195:0/3157968346 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a2c00cba0 con 0x7f6a180046d0 2022-06-27T12:01:24.021 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.005+0000 7f6a34f13700 1 -- 172.21.15.195:0/3157968346 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a2c00cd40 con 0x7f6a180046d0 2022-06-27T12:01:24.021 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.005+0000 7f6a34f13700 1 -- 172.21.15.195:0/3157968346 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a2c007320 con 0x7f6a180046d0 2022-06-27T12:01:24.022 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.005+0000 7f6a38179700 1 -- 172.21.15.195:0/3157968346 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6a180046d0 msgr2=0x7f6a18004b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:24.022 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.005+0000 7f6a38179700 1 --2- 172.21.15.195:0/3157968346 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6a180046d0 0x7f6a18004b40 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c00c2d0 tx=0x7f6a301530d0 comp rx=0 tx=0).stop 2022-06-27T12:01:24.023 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.005+0000 7f6a36716700 1 -- 172.21.15.195:0/3157968346 reap_dead start 2022-06-27T12:01:24.023 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.005+0000 7f6a38179700 1 -- 172.21.15.195:0/3157968346 shutdown_connections 2022-06-27T12:01:24.024 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.005+0000 7f6a38179700 1 -- 172.21.15.195:0/3157968346 >> 172.21.15.195:0/3157968346 conn(0x7f6a300c9b00 msgr2=0x7f6a300c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:01:24.024 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.005+0000 7f6a38179700 1 -- 172.21.15.195:0/3157968346 shutdown_connections 2022-06-27T12:01:24.024 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.005+0000 7f6a38179700 1 -- 172.21.15.195:0/3157968346 wait complete. 2022-06-27T12:01:24.027 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.006+0000 7f6a38179700 1 Processor -- start 2022-06-27T12:01:24.028 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.006+0000 7f6a38179700 1 -- start start 2022-06-27T12:01:24.028 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.007+0000 7f6a38179700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6a180046d0 0x7f6a30156bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:24.028 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.007+0000 7f6a38179700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6a300be250 0x7f6a3006dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:24.028 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.007+0000 7f6a38179700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6a3006e940 0x7f6a301542a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:24.029 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.007+0000 7f6a38179700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a30156040 con 0x7f6a300be250 2022-06-27T12:01:24.029 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.007+0000 7f6a38179700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a30157110 con 0x7f6a180046d0 2022-06-27T12:01:24.029 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.007+0000 7f6a38179700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a30156820 con 0x7f6a3006e940 2022-06-27T12:01:24.029 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.007+0000 7f6a35714700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6a300be250 0x7f6a3006dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:24.030 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.007+0000 7f6a35714700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6a300be250 0x7f6a3006dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:46544/0 (socket says 172.21.15.195:46544) 2022-06-27T12:01:24.030 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.007+0000 7f6a35714700 1 -- 172.21.15.195:0/3256022428 learned_addr learned my addr 172.21.15.195:0/3256022428 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:01:24.030 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.007+0000 7f6a35f15700 1 --2- 172.21.15.195:0/3256022428 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6a180046d0 0x7f6a30156bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:24.030 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.007+0000 7f6a36716700 1 --2- 172.21.15.195:0/3256022428 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6a3006e940 0x7f6a301542a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:24.031 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.007+0000 7f6a35714700 1 -- 172.21.15.195:0/3256022428 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6a3006e940 msgr2=0x7f6a301542a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:24.031 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.007+0000 7f6a35714700 1 --2- 172.21.15.195:0/3256022428 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6a3006e940 0x7f6a301542a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:24.031 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.007+0000 7f6a35714700 1 -- 172.21.15.195:0/3256022428 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6a180046d0 msgr2=0x7f6a30156bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:24.032 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.007+0000 7f6a35714700 1 --2- 172.21.15.195:0/3256022428 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6a180046d0 0x7f6a30156bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:24.032 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.007+0000 7f6a35714700 1 -- 172.21.15.195:0/3256022428 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a301659c0 con 0x7f6a300be250 2022-06-27T12:01:24.032 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.007+0000 7f6a35714700 1 --2- 172.21.15.195:0/3256022428 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6a300be250 0x7f6a3006dde0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f6a18005470 tx=0x7f6a2c012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:01:24.033 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.008+0000 7f6a1dffb700 1 -- 172.21.15.195:0/3256022428 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a2c007690 con 0x7f6a300be250 2022-06-27T12:01:24.033 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.008+0000 7f6a1dffb700 1 -- 172.21.15.195:0/3256022428 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a2c00ceb0 con 0x7f6a300be250 2022-06-27T12:01:24.033 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.008+0000 7f6a1dffb700 1 -- 172.21.15.195:0/3256022428 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a2c007a50 con 0x7f6a300be250 2022-06-27T12:01:24.033 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.008+0000 7f6a38179700 1 -- 172.21.15.195:0/3256022428 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a30157330 con 0x7f6a300be250 2022-06-27T12:01:24.034 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.008+0000 7f6a38179700 1 -- 172.21.15.195:0/3256022428 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a30165370 con 0x7f6a300be250 2022-06-27T12:01:24.034 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.009+0000 7f6a38179700 1 -- 172.21.15.195:0/3256022428 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a30165b70 con 0x7f6a300be250 2022-06-27T12:01:24.035 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.009+0000 7f6a1dffb700 1 -- 172.21.15.195:0/3256022428 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f6a2c010070 con 0x7f6a300be250 2022-06-27T12:01:24.035 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.010+0000 7f6a1dffb700 1 --2- 172.21.15.195:0/3256022428 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6a10063d00 0x7f6a100661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:24.035 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.010+0000 7f6a1dffb700 1 -- 172.21.15.195:0/3256022428 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6718+0+0 (secure 0 0 0) 0x7f6a2c0813e0 con 0x7f6a300be250 2022-06-27T12:01:24.035 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.010+0000 7f6a35f15700 1 --2- 172.21.15.195:0/3256022428 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6a10063d00 0x7f6a100661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:24.036 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.010+0000 7f6a35f15700 1 --2- 172.21.15.195:0/3256022428 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6a10063d00 0x7f6a100661b0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f6a20006d60 tx=0x7f6a20006cb0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:01:24.036 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.014+0000 7f6a1dffb700 1 -- 172.21.15.195:0/3256022428 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f6a30165b70 con 0x7f6a300be250 2022-06-27T12:01:24.243 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.243+0000 7f6a38179700 1 -- 172.21.15.195:0/3256022428 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6a30165b70 con 0x7f6a300be250 2022-06-27T12:01:24.248 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.243+0000 7f6a1dffb700 1 -- 172.21.15.195:0/3256022428 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+6125 (secure 0 0 0) 0x7f6a30165b70 con 0x7f6a300be250 2022-06-27T12:01:24.250 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:01:24.250 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/417105338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":417105338},{"type":"v1","addr":"172.21.15.202:6835","nonce":417105338}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:59:23.081414+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5192},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5192":{"gid":5192,"name":"b","rank":0,"incarnation":43,"state":"up:active","state_seq":14,"addr":"172.21.15.202:6833/2651813808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":2651813808},{"type":"v1","addr":"172.21.15.202:6833","nonce":2651813808}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:01:24.283 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.245+0000 7f6a38179700 1 -- 172.21.15.195:0/3256022428 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6a10063d00 msgr2=0x7f6a100661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:24.284 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.245+0000 7f6a38179700 1 --2- 172.21.15.195:0/3256022428 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6a10063d00 0x7f6a100661b0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f6a20006d60 tx=0x7f6a20006cb0 comp rx=0 tx=0).stop 2022-06-27T12:01:24.285 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.245+0000 7f6a38179700 1 -- 172.21.15.195:0/3256022428 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6a300be250 msgr2=0x7f6a3006dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:24.286 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.245+0000 7f6a38179700 1 --2- 172.21.15.195:0/3256022428 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6a300be250 0x7f6a3006dde0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f6a18005470 tx=0x7f6a2c012000 comp rx=0 tx=0).stop 2022-06-27T12:01:24.286 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.245+0000 7f6a38179700 1 -- 172.21.15.195:0/3256022428 shutdown_connections 2022-06-27T12:01:24.287 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.245+0000 7f6a38179700 1 --2- 172.21.15.195:0/3256022428 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6a10063d00 0x7f6a100661b0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:24.287 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.245+0000 7f6a38179700 1 --2- 172.21.15.195:0/3256022428 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6a3006e940 0x7f6a301542a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:24.287 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.245+0000 7f6a38179700 1 --2- 172.21.15.195:0/3256022428 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6a180046d0 0x7f6a30156bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:24.287 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.245+0000 7f6a38179700 1 --2- 172.21.15.195:0/3256022428 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6a300be250 0x7f6a3006dde0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:24.288 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.245+0000 7f6a38179700 1 -- 172.21.15.195:0/3256022428 >> 172.21.15.195:0/3256022428 conn(0x7f6a300c9b00 msgr2=0x7f6a30063c40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:01:24.288 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.246+0000 7f6a38179700 1 -- 172.21.15.195:0/3256022428 shutdown_connections 2022-06-27T12:01:24.288 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:24.246+0000 7f6a38179700 1 -- 172.21.15.195:0/3256022428 wait complete. 2022-06-27T12:01:24.288 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 48 2022-06-27T12:01:24.292 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2022-06-27T12:01:26.294 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:01:26.454 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.427+0000 7fefcaf13700 1 Processor -- start 2022-06-27T12:01:26.455 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.427+0000 7fefcaf13700 1 -- start start 2022-06-27T12:01:26.455 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.428+0000 7fefcaf13700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fefc40c3f50 0x7fefc415ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:26.456 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.428+0000 7fefcaf13700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefc40c33b0 con 0x7fefc414e490 2022-06-27T12:01:26.456 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.428+0000 7fefcaf13700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefc406f220 con 0x7fefc414e860 2022-06-27T12:01:26.456 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.428+0000 7fefcaf13700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefc41542c0 con 0x7fefc40c3f50 2022-06-27T12:01:26.458 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.428+0000 7fefc8caf700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fefc414e490 0x7fefc406cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:52740/0 (socket says 172.21.15.195:52740) 2022-06-27T12:01:26.458 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.428+0000 7fefc8caf700 1 -- 172.21.15.195:0/3206471802 learned_addr learned my addr 172.21.15.195:0/3206471802 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:01:26.459 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.429+0000 7fefc94b0700 1 --2- 172.21.15.195:0/3206471802 >> v2:172.21.15.202:3300/0 conn(0x7fefc40c3f50 0x7fefc415ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:26.459 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.429+0000 7fefc37fe700 1 -- 172.21.15.195:0/3206471802 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3184172042 0 0) 0x7fefc40c33b0 con 0x7fefc414e490 2022-06-27T12:01:26.460 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.429+0000 7fefc37fe700 1 -- 172.21.15.195:0/3206471802 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefa8003430 con 0x7fefc414e490 2022-06-27T12:01:26.460 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.429+0000 7fefc37fe700 1 -- 172.21.15.195:0/3206471802 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1720151080 0 0) 0x7fefa8003430 con 0x7fefc414e490 2022-06-27T12:01:26.461 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.429+0000 7fefc37fe700 1 -- 172.21.15.195:0/3206471802 >> v2:172.21.15.202:3300/0 conn(0x7fefc40c3f50 msgr2=0x7fefc415ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:01:26.461 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.429+0000 7fefc37fe700 1 --2- 172.21.15.195:0/3206471802 >> v2:172.21.15.202:3300/0 conn(0x7fefc40c3f50 0x7fefc415ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:26.461 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.429+0000 7fefc37fe700 1 -- 172.21.15.195:0/3206471802 >> v1:172.21.15.202:6789/0 conn(0x7fefc414e860 legacy=0x7fefc40c3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:26.462 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.429+0000 7fefc37fe700 1 -- 172.21.15.195:0/3206471802 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefc4154560 con 0x7fefc414e490 2022-06-27T12:01:26.463 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.429+0000 7fefc37fe700 1 -- 172.21.15.195:0/3206471802 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7fefb8001470 con 0x7fefc414e490 2022-06-27T12:01:26.464 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.429+0000 7fefc37fe700 1 -- 172.21.15.195:0/3206471802 >> v1:172.21.15.195:6789/0 conn(0x7fefc414e490 legacy=0x7fefc406cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:26.465 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.429+0000 7fefc37fe700 1 --2- 172.21.15.195:0/3206471802 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fefa8003920 0x7fefa8003d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:26.465 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.429+0000 7fefc37fe700 1 --2- 172.21.15.195:0/3206471802 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fefa80046d0 0x7fefa8004b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:26.465 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.429+0000 7fefc37fe700 1 --2- 172.21.15.195:0/3206471802 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fefa8005080 0x7fefa800b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:26.465 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.429+0000 7fefc37fe700 1 -- 172.21.15.195:0/3206471802 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefc41542c0 con 0x7fefa8003920 2022-06-27T12:01:26.466 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.429+0000 7fefc37fe700 1 -- 172.21.15.195:0/3206471802 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefa800bf90 con 0x7fefa80046d0 2022-06-27T12:01:26.467 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.429+0000 7fefc37fe700 1 -- 172.21.15.195:0/3206471802 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefa800c150 con 0x7fefa8005080 2022-06-27T12:01:26.467 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.429+0000 7fefc8caf700 1 --2- 172.21.15.195:0/3206471802 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fefa8003920 0x7fefa8003d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:26.468 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.429+0000 7fefc3fff700 1 --2- 172.21.15.195:0/3206471802 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fefa80046d0 0x7fefa8004b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:26.468 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.429+0000 7fefc94b0700 1 --2- 172.21.15.195:0/3206471802 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fefa8005080 0x7fefa800b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:26.469 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.429+0000 7fefc8caf700 1 -- 172.21.15.195:0/3206471802 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fefa8005080 msgr2=0x7fefa800b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:01:26.469 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.429+0000 7fefc8caf700 1 --2- 172.21.15.195:0/3206471802 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fefa8005080 0x7fefa800b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:26.469 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.429+0000 7fefc8caf700 1 -- 172.21.15.195:0/3206471802 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fefa80046d0 msgr2=0x7fefa8004b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:01:26.470 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.429+0000 7fefc8caf700 1 --2- 172.21.15.195:0/3206471802 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fefa80046d0 0x7fefa8004b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:26.470 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.429+0000 7fefc8caf700 1 -- 172.21.15.195:0/3206471802 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefa8003430 con 0x7fefa8003920 2022-06-27T12:01:26.471 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.429+0000 7fefc94b0700 1 -- 172.21.15.195:0/3206471802 reap_dead start 2022-06-27T12:01:26.471 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.430+0000 7fefc8caf700 1 --2- 172.21.15.195:0/3206471802 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fefa8003920 0x7fefa8003d30 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fefb800c1e0 tx=0x7fefc40c3220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63835ba2fbde9470 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:01:26.471 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.430+0000 7fefc37fe700 1 -- 172.21.15.195:0/3206471802 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefb800cab0 con 0x7fefa8003920 2022-06-27T12:01:26.472 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.430+0000 7fefc37fe700 1 -- 172.21.15.195:0/3206471802 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefb800cc10 con 0x7fefa8003920 2022-06-27T12:01:26.472 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.430+0000 7fefc37fe700 1 -- 172.21.15.195:0/3206471802 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefb80070d0 con 0x7fefa8003920 2022-06-27T12:01:26.473 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.430+0000 7fefcaf13700 1 -- 172.21.15.195:0/3206471802 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fefa8003920 msgr2=0x7fefa8003d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:26.473 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.430+0000 7fefcaf13700 1 --2- 172.21.15.195:0/3206471802 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fefa8003920 0x7fefa8003d30 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fefb800c1e0 tx=0x7fefc40c3220 comp rx=0 tx=0).stop 2022-06-27T12:01:26.474 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.430+0000 7fefcaf13700 1 -- 172.21.15.195:0/3206471802 shutdown_connections 2022-06-27T12:01:26.474 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.430+0000 7fefcaf13700 1 --2- 172.21.15.195:0/3206471802 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fefa8003920 0x7fefa8003d30 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:26.475 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.430+0000 7fefcaf13700 1 -- 172.21.15.195:0/3206471802 >> 172.21.15.195:0/3206471802 conn(0x7fefc4063af0 msgr2=0x7fefc4063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:01:26.475 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.430+0000 7fefcaf13700 1 -- 172.21.15.195:0/3206471802 shutdown_connections 2022-06-27T12:01:26.476 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.430+0000 7fefcaf13700 1 -- 172.21.15.195:0/3206471802 wait complete. 2022-06-27T12:01:26.476 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.431+0000 7fefcaf13700 1 Processor -- start 2022-06-27T12:01:26.476 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.432+0000 7fefcaf13700 1 -- start start 2022-06-27T12:01:26.477 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.432+0000 7fefcaf13700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fefa8003920 0x7fefc4167340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:26.477 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.432+0000 7fefcaf13700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fefc4069ac0 0x7fefc40c6e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:26.477 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.432+0000 7fefcaf13700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fefc4069ef0 0x7fefc40c0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:26.479 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.432+0000 7fefcaf13700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefc4150ef0 con 0x7fefa8003920 2022-06-27T12:01:26.479 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.432+0000 7fefcaf13700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefc41519f0 con 0x7fefc4069ef0 2022-06-27T12:01:26.479 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.432+0000 7fefcaf13700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefc4150b50 con 0x7fefc4069ac0 2022-06-27T12:01:26.480 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.432+0000 7fefc8caf700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fefa8003920 0x7fefc4167340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:26.480 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.432+0000 7fefc8caf700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fefa8003920 0x7fefc4167340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:46564/0 (socket says 172.21.15.195:46564) 2022-06-27T12:01:26.481 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.432+0000 7fefc8caf700 1 -- 172.21.15.195:0/3865581160 learned_addr learned my addr 172.21.15.195:0/3865581160 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:01:26.481 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.433+0000 7fefc8caf700 1 -- 172.21.15.195:0/3865581160 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fefc4069ac0 msgr2=0x7fefc40c6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:26.482 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.433+0000 7fefc94b0700 1 --2- 172.21.15.195:0/3865581160 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fefc4069ef0 0x7fefc40c0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:26.482 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.433+0000 7fefc3fff700 1 --2- 172.21.15.195:0/3865581160 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fefc4069ac0 0x7fefc40c6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:26.482 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.433+0000 7fefc8caf700 1 --2- 172.21.15.195:0/3865581160 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fefc4069ac0 0x7fefc40c6e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:26.483 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.433+0000 7fefc8caf700 1 -- 172.21.15.195:0/3865581160 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fefc4069ef0 msgr2=0x7fefc40c0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:26.483 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.433+0000 7fefc8caf700 1 --2- 172.21.15.195:0/3865581160 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fefc4069ef0 0x7fefc40c0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:26.484 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.433+0000 7fefc8caf700 1 -- 172.21.15.195:0/3865581160 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefc4157ee0 con 0x7fefa8003920 2022-06-27T12:01:26.484 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.433+0000 7fefc3fff700 1 --2- 172.21.15.195:0/3865581160 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fefc4069ac0 0x7fefc40c6e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T12:01:26.485 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.433+0000 7fefc94b0700 1 --2- 172.21.15.195:0/3865581160 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fefc4069ef0 0x7fefc40c0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T12:01:26.485 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.433+0000 7fefc8caf700 1 --2- 172.21.15.195:0/3865581160 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fefa8003920 0x7fefc4167340 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fefb800c7d0 tx=0x7fefb8015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:01:26.487 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.433+0000 7fefc0ff9700 1 -- 172.21.15.195:0/3865581160 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefb8000bc0 con 0x7fefa8003920 2022-06-27T12:01:26.487 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.433+0000 7fefc0ff9700 1 -- 172.21.15.195:0/3865581160 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefb80046f0 con 0x7fefa8003920 2022-06-27T12:01:26.487 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.433+0000 7fefcaf13700 1 -- 172.21.15.195:0/3865581160 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefc4069630 con 0x7fefa8003920 2022-06-27T12:01:26.488 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.433+0000 7fefc0ff9700 1 -- 172.21.15.195:0/3865581160 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefb8000d20 con 0x7fefa8003920 2022-06-27T12:01:26.488 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.433+0000 7fefcaf13700 1 -- 172.21.15.195:0/3865581160 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefc4158130 con 0x7fefa8003920 2022-06-27T12:01:26.489 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.434+0000 7fefcaf13700 1 -- 172.21.15.195:0/3865581160 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefc40722c0 con 0x7fefa8003920 2022-06-27T12:01:26.489 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.434+0000 7fefc0ff9700 1 -- 172.21.15.195:0/3865581160 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fefb8024020 con 0x7fefa8003920 2022-06-27T12:01:26.490 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.435+0000 7fefc0ff9700 1 --2- 172.21.15.195:0/3865581160 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fefac063d30 0x7fefac0661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:26.491 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.435+0000 7fefc0ff9700 1 -- 172.21.15.195:0/3865581160 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6718+0+0 (secure 0 0 0) 0x7fefb8081d40 con 0x7fefa8003920 2022-06-27T12:01:26.491 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.435+0000 7fefc3fff700 1 --2- 172.21.15.195:0/3865581160 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fefac063d30 0x7fefac0661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:26.492 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.435+0000 7fefc3fff700 1 --2- 172.21.15.195:0/3865581160 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fefac063d30 0x7fefac0661e0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fefb0005530 tx=0x7fefb0009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:01:26.492 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.439+0000 7fefc0ff9700 1 -- 172.21.15.195:0/3865581160 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fefb80506a0 con 0x7fefa8003920 2022-06-27T12:01:26.666 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.665+0000 7fefcaf13700 1 -- 172.21.15.195:0/3865581160 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fefc4069400 con 0x7fefa8003920 2022-06-27T12:01:26.666 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.665+0000 7fefc0ff9700 1 -- 172.21.15.195:0/3865581160 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+6125 (secure 0 0 0) 0x7fefc4069400 con 0x7fefa8003920 2022-06-27T12:01:26.667 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:01:26.668 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/417105338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":417105338},{"type":"v1","addr":"172.21.15.202:6835","nonce":417105338}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:59:23.081414+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5192},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5192":{"gid":5192,"name":"b","rank":0,"incarnation":43,"state":"up:active","state_seq":14,"addr":"172.21.15.202:6833/2651813808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":2651813808},{"type":"v1","addr":"172.21.15.202:6833","nonce":2651813808}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:01:26.670 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.667+0000 7fefcaf13700 1 -- 172.21.15.195:0/3865581160 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fefac063d30 msgr2=0x7fefac0661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:26.671 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.667+0000 7fefcaf13700 1 --2- 172.21.15.195:0/3865581160 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fefac063d30 0x7fefac0661e0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fefb0005530 tx=0x7fefb0009040 comp rx=0 tx=0).stop 2022-06-27T12:01:26.671 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.667+0000 7fefcaf13700 1 -- 172.21.15.195:0/3865581160 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fefa8003920 msgr2=0x7fefc4167340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:26.672 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.667+0000 7fefcaf13700 1 --2- 172.21.15.195:0/3865581160 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fefa8003920 0x7fefc4167340 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fefb800c7d0 tx=0x7fefb8015000 comp rx=0 tx=0).stop 2022-06-27T12:01:26.672 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.667+0000 7fefcaf13700 1 -- 172.21.15.195:0/3865581160 shutdown_connections 2022-06-27T12:01:26.673 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.667+0000 7fefcaf13700 1 --2- 172.21.15.195:0/3865581160 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fefac063d30 0x7fefac0661e0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:26.674 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.667+0000 7fefcaf13700 1 --2- 172.21.15.195:0/3865581160 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fefa8003920 0x7fefc4167340 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:26.675 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.667+0000 7fefcaf13700 1 --2- 172.21.15.195:0/3865581160 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fefc4069ef0 0x7fefc40c0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:26.675 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.667+0000 7fefcaf13700 1 --2- 172.21.15.195:0/3865581160 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fefc4069ac0 0x7fefc40c6e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:26.675 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.667+0000 7fefcaf13700 1 -- 172.21.15.195:0/3865581160 >> 172.21.15.195:0/3865581160 conn(0x7fefc4063af0 msgr2=0x7fefc414c730 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:01:26.675 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.668+0000 7fefcaf13700 1 -- 172.21.15.195:0/3865581160 shutdown_connections 2022-06-27T12:01:26.676 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:26.668+0000 7fefcaf13700 1 -- 172.21.15.195:0/3865581160 wait complete. 2022-06-27T12:01:26.676 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 48 2022-06-27T12:01:26.681 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2022-06-27T12:01:28.688 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:01:28.874 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:28.828+0000 7fb08c8a5700 1 Processor -- start 2022-06-27T12:01:28.875 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:28.828+0000 7fb08c8a5700 1 -- start start 2022-06-27T12:01:28.875 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:28.829+0000 7fb08c8a5700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fb08806fef0 0x7fb0881570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:28.876 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:28.829+0000 7fb08c8a5700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb0880c34a0 0x7fb0880c2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:28.876 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:28.829+0000 7fb08c8a5700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb08806fba0 con 0x7fb08806e850 2022-06-27T12:01:28.877 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:28.829+0000 7fb08c8a5700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0881575e0 con 0x7fb08806fef0 2022-06-27T12:01:28.877 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:28.829+0000 7fb08c8a5700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb08806f5e0 con 0x7fb0880c34a0 2022-06-27T12:01:28.877 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:28.829+0000 7fb085d9b700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fb08806fef0 0x7fb0881570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:28.878 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:28.829+0000 7fb08659c700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fb08806e850 0x7fb088069de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:52760/0 (socket says 172.21.15.195:52760) 2022-06-27T12:01:28.878 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:28.829+0000 7fb08659c700 1 -- 172.21.15.195:0/428772065 learned_addr learned my addr 172.21.15.195:0/428772065 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:01:28.879 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:28.829+0000 7fb086d9d700 1 --2- 172.21.15.195:0/428772065 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb0880c34a0 0x7fb0880c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:28.879 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:28.829+0000 7fb085d9b700 1 -- 172.21.15.195:0/428772065 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb0880c34a0 msgr2=0x7fb0880c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:01:28.880 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:28.829+0000 7fb085d9b700 1 --2- 172.21.15.195:0/428772065 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb0880c34a0 0x7fb0880c2920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:28.880 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:28.829+0000 7fb085d9b700 1 -- 172.21.15.195:0/428772065 >> v1:172.21.15.195:6789/0 conn(0x7fb08806e850 legacy=0x7fb088069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:28.881 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:28.829+0000 7fb08559a700 1 -- 172.21.15.195:0/428772065 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2449634823 0 0) 0x7fb08806fba0 con 0x7fb08806e850 2022-06-27T12:01:28.881 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:28.829+0000 7fb085d9b700 1 -- 172.21.15.195:0/428772065 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb08806f8c0 con 0x7fb08806fef0 2022-06-27T12:01:28.883 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:28.829+0000 7fb086d9d700 1 --2- 172.21.15.195:0/428772065 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb0880c34a0 0x7fb0880c2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:01:28.884 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:28.829+0000 7fb085d9b700 1 --2- 172.21.15.195:0/428772065 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb08806fef0 0x7fb0881570a0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fb0700097a0 tx=0x7fb070009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a00a948bad3618e server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:01:28.885 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:28.829+0000 7fb08559a700 1 -- 172.21.15.195:0/428772065 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb070011630 con 0x7fb08806fef0 2022-06-27T12:01:28.885 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:28.829+0000 7fb08559a700 1 -- 172.21.15.195:0/428772065 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb070011790 con 0x7fb08806fef0 2022-06-27T12:01:28.885 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:28.829+0000 7fb08559a700 1 -- 172.21.15.195:0/428772065 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb070011b30 con 0x7fb08806fef0 2022-06-27T12:01:28.886 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:28.830+0000 7fb08c8a5700 1 -- 172.21.15.195:0/428772065 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb08806fef0 msgr2=0x7fb0881570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:28.887 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:28.830+0000 7fb08c8a5700 1 --2- 172.21.15.195:0/428772065 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb08806fef0 0x7fb0881570a0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fb0700097a0 tx=0x7fb070009060 comp rx=0 tx=0).stop 2022-06-27T12:01:28.887 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:28.830+0000 7fb08c8a5700 1 -- 172.21.15.195:0/428772065 shutdown_connections 2022-06-27T12:01:28.887 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:28.830+0000 7fb08c8a5700 1 --2- 172.21.15.195:0/428772065 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb0880c34a0 0x7fb0880c2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:28.888 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:28.830+0000 7fb08c8a5700 1 --2- 172.21.15.195:0/428772065 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb08806fef0 0x7fb0881570a0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:28.888 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:28.830+0000 7fb08c8a5700 1 -- 172.21.15.195:0/428772065 >> 172.21.15.195:0/428772065 conn(0x7fb0880c6350 msgr2=0x7fb0880c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:01:28.889 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:28.830+0000 7fb08c8a5700 1 -- 172.21.15.195:0/428772065 shutdown_connections 2022-06-27T12:01:28.889 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:28.830+0000 7fb08c8a5700 1 -- 172.21.15.195:0/428772065 wait complete. 2022-06-27T12:01:28.890 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:28.831+0000 7fb08c8a5700 1 Processor -- start 2022-06-27T12:01:28.895 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:28.831+0000 7fb08c8a5700 1 -- start start 2022-06-27T12:01:28.895 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:28.832+0000 7fb08c8a5700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb08806e850 0x7fb088155850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:28.898 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:28.832+0000 7fb08c8a5700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb0880c34a0 0x7fb088153ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:28.898 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:28.832+0000 7fb08c8a5700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb0880b70e0 0x7fb088164b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:28.899 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:28.832+0000 7fb08c8a5700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0880b76c0 con 0x7fb0880b70e0 2022-06-27T12:01:28.899 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:28.832+0000 7fb08c8a5700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb088166930 con 0x7fb0880c34a0 2022-06-27T12:01:28.900 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:28.832+0000 7fb08c8a5700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0880c8520 con 0x7fb08806e850 2022-06-27T12:01:28.900 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:28.832+0000 7fb086d9d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb0880b70e0 0x7fb088164b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:28.901 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:28.832+0000 7fb086d9d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb0880b70e0 0x7fb088164b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:46582/0 (socket says 172.21.15.195:46582) 2022-06-27T12:01:28.901 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:28.832+0000 7fb086d9d700 1 -- 172.21.15.195:0/3316848888 learned_addr learned my addr 172.21.15.195:0/3316848888 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:01:28.903 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:28.832+0000 7fb08659c700 1 --2- 172.21.15.195:0/3316848888 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb08806e850 0x7fb088155850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:28.903 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:28.832+0000 7fb085d9b700 1 --2- 172.21.15.195:0/3316848888 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb0880c34a0 0x7fb088153ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:28.904 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:28.832+0000 7fb086d9d700 1 -- 172.21.15.195:0/3316848888 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb08806e850 msgr2=0x7fb088155850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:28.904 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:28.832+0000 7fb086d9d700 1 --2- 172.21.15.195:0/3316848888 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb08806e850 0x7fb088155850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:28.905 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:28.832+0000 7fb086d9d700 1 -- 172.21.15.195:0/3316848888 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb0880c34a0 msgr2=0x7fb088153ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:28.905 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:28.832+0000 7fb086d9d700 1 --2- 172.21.15.195:0/3316848888 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb0880c34a0 0x7fb088153ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:28.907 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:28.832+0000 7fb086d9d700 1 -- 172.21.15.195:0/3316848888 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0700092e0 con 0x7fb0880b70e0 2022-06-27T12:01:28.907 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:28.833+0000 7fb086d9d700 1 --2- 172.21.15.195:0/3316848888 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb0880b70e0 0x7fb088164b00 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fb08000d180 tx=0x7fb080009a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:01:28.908 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:28.833+0000 7fb07e7fc700 1 -- 172.21.15.195:0/3316848888 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb08000dcc0 con 0x7fb0880b70e0 2022-06-27T12:01:28.908 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:28.833+0000 7fb07e7fc700 1 -- 172.21.15.195:0/3316848888 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb08000f040 con 0x7fb0880b70e0 2022-06-27T12:01:28.909 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:28.833+0000 7fb08c8a5700 1 -- 172.21.15.195:0/3316848888 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb088166bd0 con 0x7fb0880b70e0 2022-06-27T12:01:28.909 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:28.833+0000 7fb07e7fc700 1 -- 172.21.15.195:0/3316848888 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb080003af0 con 0x7fb0880b70e0 2022-06-27T12:01:28.910 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:28.833+0000 7fb08c8a5700 1 -- 172.21.15.195:0/3316848888 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb088167190 con 0x7fb0880b70e0 2022-06-27T12:01:28.910 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:28.834+0000 7fb08c8a5700 1 -- 172.21.15.195:0/3316848888 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb088164820 con 0x7fb0880b70e0 2022-06-27T12:01:28.916 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:28.834+0000 7fb07e7fc700 1 -- 172.21.15.195:0/3316848888 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fb08001c030 con 0x7fb0880b70e0 2022-06-27T12:01:28.917 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:28.835+0000 7fb07e7fc700 1 --2- 172.21.15.195:0/3316848888 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb06c063d00 0x7fb06c0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:28.917 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:28.835+0000 7fb07e7fc700 1 -- 172.21.15.195:0/3316848888 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6718+0+0 (secure 0 0 0) 0x7fb080080b70 con 0x7fb0880b70e0 2022-06-27T12:01:28.917 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:28.835+0000 7fb08659c700 1 --2- 172.21.15.195:0/3316848888 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb06c063d00 0x7fb06c0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:28.918 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:28.835+0000 7fb08659c700 1 --2- 172.21.15.195:0/3316848888 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb06c063d00 0x7fb06c0661b0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fb078007fd0 tx=0x7fb078007d10 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:01:28.918 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:28.839+0000 7fb07e7fc700 1 -- 172.21.15.195:0/3316848888 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fb088164820 con 0x7fb0880b70e0 2022-06-27T12:01:29.071 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:29.064+0000 7fb08c8a5700 1 -- 172.21.15.195:0/3316848888 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb088164820 con 0x7fb0880b70e0 2022-06-27T12:01:29.071 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:29.065+0000 7fb07e7fc700 1 -- 172.21.15.195:0/3316848888 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+6125 (secure 0 0 0) 0x7fb088164820 con 0x7fb0880b70e0 2022-06-27T12:01:29.073 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:01:29.073 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/417105338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":417105338},{"type":"v1","addr":"172.21.15.202:6835","nonce":417105338}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T11:59:23.081414+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5192},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5192":{"gid":5192,"name":"b","rank":0,"incarnation":43,"state":"up:active","state_seq":14,"addr":"172.21.15.202:6833/2651813808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":2651813808},{"type":"v1","addr":"172.21.15.202:6833","nonce":2651813808}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:01:29.096 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:29.067+0000 7fb08c8a5700 1 -- 172.21.15.195:0/3316848888 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb06c063d00 msgr2=0x7fb06c0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:29.096 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:29.067+0000 7fb08c8a5700 1 --2- 172.21.15.195:0/3316848888 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb06c063d00 0x7fb06c0661b0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fb078007fd0 tx=0x7fb078007d10 comp rx=0 tx=0).stop 2022-06-27T12:01:29.097 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:29.067+0000 7fb08c8a5700 1 -- 172.21.15.195:0/3316848888 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb0880b70e0 msgr2=0x7fb088164b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:29.097 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:29.067+0000 7fb08c8a5700 1 --2- 172.21.15.195:0/3316848888 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb0880b70e0 0x7fb088164b00 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fb08000d180 tx=0x7fb080009a50 comp rx=0 tx=0).stop 2022-06-27T12:01:29.098 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:29.067+0000 7fb08c8a5700 1 -- 172.21.15.195:0/3316848888 shutdown_connections 2022-06-27T12:01:29.098 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:29.067+0000 7fb08c8a5700 1 --2- 172.21.15.195:0/3316848888 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb06c063d00 0x7fb06c0661b0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:29.099 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:29.067+0000 7fb08c8a5700 1 --2- 172.21.15.195:0/3316848888 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb0880b70e0 0x7fb088164b00 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:29.099 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:29.067+0000 7fb08c8a5700 1 --2- 172.21.15.195:0/3316848888 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb08806e850 0x7fb088155850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:29.100 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:29.067+0000 7fb08c8a5700 1 --2- 172.21.15.195:0/3316848888 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb0880c34a0 0x7fb088153ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:29.100 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:29.067+0000 7fb08c8a5700 1 -- 172.21.15.195:0/3316848888 >> 172.21.15.195:0/3316848888 conn(0x7fb0880c6350 msgr2=0x7fb088150550 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:01:29.100 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:29.067+0000 7fb08c8a5700 1 -- 172.21.15.195:0/3316848888 shutdown_connections 2022-06-27T12:01:29.101 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:29.067+0000 7fb08c8a5700 1 -- 172.21.15.195:0/3316848888 wait complete. 2022-06-27T12:01:29.101 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 48 2022-06-27T12:01:29.103 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2022-06-27T12:01:29.718 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:01:29.731 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:01:31.104 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:01:31.266 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.247+0000 7fd0e87a9700 1 Processor -- start 2022-06-27T12:01:31.267 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.247+0000 7fd0e87a9700 1 -- start start 2022-06-27T12:01:31.267 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.248+0000 7fd0e87a9700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd0e00c3030 0x7fd0e01570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:31.267 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.248+0000 7fd0e87a9700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0e006fba0 con 0x7fd0e006e850 2022-06-27T12:01:31.268 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.248+0000 7fd0e87a9700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0e006f5e0 con 0x7fd0e006fef0 2022-06-27T12:01:31.268 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.248+0000 7fd0e87a9700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0e01575e0 con 0x7fd0e00c3030 2022-06-27T12:01:31.268 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.248+0000 7fd0e6d46700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd0e00c3030 0x7fd0e01570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:31.268 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.248+0000 7fd0e6d46700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd0e00c3030 0x7fd0e01570a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:46594/0 (socket says 172.21.15.195:46594) 2022-06-27T12:01:31.268 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.248+0000 7fd0e6545700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fd0e006e850 0x7fd0e0069de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:52778/0 (socket says 172.21.15.195:52778) 2022-06-27T12:01:31.269 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.248+0000 7fd0e6d46700 1 -- 172.21.15.195:0/1625611682 learned_addr learned my addr 172.21.15.195:0/1625611682 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:01:31.269 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.248+0000 7fd0e6d46700 1 -- 172.21.15.195:0/1625611682 >> v1:172.21.15.202:6789/0 conn(0x7fd0e006fef0 legacy=0x7fd0e00c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:31.269 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.248+0000 7fd0e6d46700 1 -- 172.21.15.195:0/1625611682 >> v1:172.21.15.195:6789/0 conn(0x7fd0e006e850 legacy=0x7fd0e0069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:31.269 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.248+0000 7fd0e6d46700 1 -- 172.21.15.195:0/1625611682 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0e006f8c0 con 0x7fd0e00c3030 2022-06-27T12:01:31.270 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.248+0000 7fd0e5543700 1 -- 172.21.15.195:0/1625611682 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3743692159 0 0) 0x7fd0e006fba0 con 0x7fd0e006e850 2022-06-27T12:01:31.270 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.248+0000 7fd0e6d46700 1 --2- 172.21.15.195:0/1625611682 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd0e00c3030 0x7fd0e01570a0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fd0d8009f20 tx=0x7fd0d8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49b192192b18586a server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:01:31.270 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.248+0000 7fd0e5543700 1 -- 172.21.15.195:0/1625611682 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d800e070 con 0x7fd0e00c3030 2022-06-27T12:01:31.270 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.249+0000 7fd0e5543700 1 -- 172.21.15.195:0/1625611682 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0d800bb10 con 0x7fd0e00c3030 2022-06-27T12:01:31.271 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.249+0000 7fd0e5543700 1 -- 172.21.15.195:0/1625611682 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d80092e0 con 0x7fd0e00c3030 2022-06-27T12:01:31.271 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.249+0000 7fd0e87a9700 1 -- 172.21.15.195:0/1625611682 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd0e00c3030 msgr2=0x7fd0e01570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:31.271 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.249+0000 7fd0e87a9700 1 --2- 172.21.15.195:0/1625611682 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd0e00c3030 0x7fd0e01570a0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fd0d8009f20 tx=0x7fd0d8009840 comp rx=0 tx=0).stop 2022-06-27T12:01:31.271 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.249+0000 7fd0e87a9700 1 -- 172.21.15.195:0/1625611682 shutdown_connections 2022-06-27T12:01:31.272 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.249+0000 7fd0e87a9700 1 --2- 172.21.15.195:0/1625611682 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd0e00c3030 0x7fd0e01570a0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:31.272 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.249+0000 7fd0e87a9700 1 -- 172.21.15.195:0/1625611682 >> 172.21.15.195:0/1625611682 conn(0x7fd0e00c6350 msgr2=0x7fd0e00c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:01:31.272 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.249+0000 7fd0e87a9700 1 -- 172.21.15.195:0/1625611682 shutdown_connections 2022-06-27T12:01:31.272 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.249+0000 7fd0e87a9700 1 -- 172.21.15.195:0/1625611682 wait complete. 2022-06-27T12:01:31.273 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.250+0000 7fd0e87a9700 1 Processor -- start 2022-06-27T12:01:31.273 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.250+0000 7fd0e87a9700 1 -- start start 2022-06-27T12:01:31.273 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.251+0000 7fd0e87a9700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd0e006e850 0x7fd0e0168170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:31.274 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.251+0000 7fd0e87a9700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd0e00c3030 0x7fd0e00c78a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:31.274 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.251+0000 7fd0e87a9700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd0e0158c10 0x7fd0e0156500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:31.274 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.251+0000 7fd0e87a9700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e01597b0 con 0x7fd0e006e850 2022-06-27T12:01:31.275 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.251+0000 7fd0e87a9700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e0167ce0 con 0x7fd0e00c3030 2022-06-27T12:01:31.275 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.251+0000 7fd0e87a9700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e01591f0 con 0x7fd0e0158c10 2022-06-27T12:01:31.276 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.251+0000 7fd0e6545700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd0e006e850 0x7fd0e0168170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:31.276 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.251+0000 7fd0e6545700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd0e006e850 0x7fd0e0168170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:46596/0 (socket says 172.21.15.195:46596) 2022-06-27T12:01:31.276 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.251+0000 7fd0e6545700 1 -- 172.21.15.195:0/3167117325 learned_addr learned my addr 172.21.15.195:0/3167117325 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:01:31.276 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.251+0000 7fd0e6d46700 1 --2- 172.21.15.195:0/3167117325 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd0e0158c10 0x7fd0e0156500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:31.277 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.251+0000 7fd0e5d44700 1 --2- 172.21.15.195:0/3167117325 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd0e00c3030 0x7fd0e00c78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:31.277 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.251+0000 7fd0e6545700 1 -- 172.21.15.195:0/3167117325 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd0e0158c10 msgr2=0x7fd0e0156500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:31.277 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.251+0000 7fd0e6545700 1 --2- 172.21.15.195:0/3167117325 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd0e0158c10 0x7fd0e0156500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:31.277 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.251+0000 7fd0e6545700 1 -- 172.21.15.195:0/3167117325 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd0e00c3030 msgr2=0x7fd0e00c78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:31.277 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.251+0000 7fd0e6545700 1 --2- 172.21.15.195:0/3167117325 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd0e00c3030 0x7fd0e00c78a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:31.278 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.251+0000 7fd0e6545700 1 -- 172.21.15.195:0/3167117325 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0d8009ae0 con 0x7fd0e006e850 2022-06-27T12:01:31.278 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.251+0000 7fd0e6d46700 1 --2- 172.21.15.195:0/3167117325 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd0e0158c10 0x7fd0e0156500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T12:01:31.278 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.251+0000 7fd0e5d44700 1 --2- 172.21.15.195:0/3167117325 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd0e00c3030 0x7fd0e00c78a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T12:01:31.278 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.252+0000 7fd0e6545700 1 --2- 172.21.15.195:0/3167117325 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd0e006e850 0x7fd0e0168170 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fd0d000c4c0 tx=0x7fd0d0007de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:01:31.279 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.252+0000 7fd0ce7fc700 1 -- 172.21.15.195:0/3167117325 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d0007360 con 0x7fd0e006e850 2022-06-27T12:01:31.279 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.252+0000 7fd0ce7fc700 1 -- 172.21.15.195:0/3167117325 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0d00074c0 con 0x7fd0e006e850 2022-06-27T12:01:31.279 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.252+0000 7fd0e87a9700 1 -- 172.21.15.195:0/3167117325 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0e01594d0 con 0x7fd0e006e850 2022-06-27T12:01:31.279 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.252+0000 7fd0ce7fc700 1 -- 172.21.15.195:0/3167117325 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d0004940 con 0x7fd0e006e850 2022-06-27T12:01:31.280 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.252+0000 7fd0e87a9700 1 -- 172.21.15.195:0/3167117325 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0e01686b0 con 0x7fd0e006e850 2022-06-27T12:01:31.280 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.253+0000 7fd0e87a9700 1 -- 172.21.15.195:0/3167117325 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0e0167fc0 con 0x7fd0e006e850 2022-06-27T12:01:31.280 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.253+0000 7fd0ce7fc700 1 -- 172.21.15.195:0/3167117325 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fd0d001a020 con 0x7fd0e006e850 2022-06-27T12:01:31.280 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.253+0000 7fd0ce7fc700 1 --2- 172.21.15.195:0/3167117325 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd0c0063d30 0x7fd0c00661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:31.281 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.254+0000 7fd0ce7fc700 1 -- 172.21.15.195:0/3167117325 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6804+0+0 (secure 0 0 0) 0x7fd0d0080740 con 0x7fd0e006e850 2022-06-27T12:01:31.281 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.254+0000 7fd0e5d44700 1 --2- 172.21.15.195:0/3167117325 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd0c0063d30 0x7fd0c00661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:31.281 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.254+0000 7fd0e5d44700 1 --2- 172.21.15.195:0/3167117325 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd0c0063d30 0x7fd0c00661e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fd0e00c3a80 tx=0x7fd0dc009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:01:31.281 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.258+0000 7fd0ce7fc700 1 -- 172.21.15.195:0/3167117325 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fd0d004f0c0 con 0x7fd0e006e850 2022-06-27T12:01:31.486 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.478+0000 7fd0e87a9700 1 -- 172.21.15.195:0/3167117325 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd0e00722c0 con 0x7fd0e006e850 2022-06-27T12:01:31.487 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.479+0000 7fd0ce7fc700 1 -- 172.21.15.195:0/3167117325 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+5364 (secure 0 0 0) 0x7fd0d00526e0 con 0x7fd0e006e850 2022-06-27T12:01:31.512 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:01:31.512 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:01:30.014988+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5273":{"gid":5273,"name":"d","rank":0,"incarnation":49,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6835/417105338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":417105338},{"type":"v1","addr":"172.21.15.202:6835","nonce":417105338}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:01:31.516 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.481+0000 7fd0e87a9700 1 -- 172.21.15.195:0/3167117325 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd0c0063d30 msgr2=0x7fd0c00661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:31.517 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.481+0000 7fd0e87a9700 1 --2- 172.21.15.195:0/3167117325 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd0c0063d30 0x7fd0c00661e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fd0e00c3a80 tx=0x7fd0dc009040 comp rx=0 tx=0).stop 2022-06-27T12:01:31.517 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.481+0000 7fd0e87a9700 1 -- 172.21.15.195:0/3167117325 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd0e006e850 msgr2=0x7fd0e0168170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:31.518 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.481+0000 7fd0e87a9700 1 --2- 172.21.15.195:0/3167117325 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd0e006e850 0x7fd0e0168170 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fd0d000c4c0 tx=0x7fd0d0007de0 comp rx=0 tx=0).stop 2022-06-27T12:01:31.518 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.481+0000 7fd0e87a9700 1 -- 172.21.15.195:0/3167117325 shutdown_connections 2022-06-27T12:01:31.519 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.481+0000 7fd0e87a9700 1 --2- 172.21.15.195:0/3167117325 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd0c0063d30 0x7fd0c00661e0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:31.519 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.481+0000 7fd0e87a9700 1 --2- 172.21.15.195:0/3167117325 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd0e006e850 0x7fd0e0168170 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:31.519 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.481+0000 7fd0e87a9700 1 --2- 172.21.15.195:0/3167117325 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd0e0158c10 0x7fd0e0156500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:31.520 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.481+0000 7fd0e87a9700 1 --2- 172.21.15.195:0/3167117325 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd0e00c3030 0x7fd0e00c78a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:31.520 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.481+0000 7fd0e87a9700 1 -- 172.21.15.195:0/3167117325 >> 172.21.15.195:0/3167117325 conn(0x7fd0e00c6350 msgr2=0x7fd0e01503e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:01:31.521 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.481+0000 7fd0e87a9700 1 -- 172.21.15.195:0/3167117325 shutdown_connections 2022-06-27T12:01:31.521 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.481+0000 7fd0e87a9700 1 -- 172.21.15.195:0/3167117325 wait complete. 2022-06-27T12:01:31.522 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 49 2022-06-27T12:01:31.523 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2022-06-27T12:01:31.523 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2022-06-27T12:01:31.524 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:01:31.689 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.654+0000 7f30cba41700 1 Processor -- start 2022-06-27T12:01:31.690 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.654+0000 7f30cba41700 1 -- start start 2022-06-27T12:01:31.690 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.655+0000 7f30cba41700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f30c414e490 0x7f30c406cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:31.697 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.655+0000 7f30cba41700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30c40c33b0 con 0x7f30c40c3f50 2022-06-27T12:01:31.697 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.655+0000 7f30cba41700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30c406d520 con 0x7f30c414e860 2022-06-27T12:01:31.698 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.655+0000 7f30cba41700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30c406f220 con 0x7f30c414e490 2022-06-27T12:01:31.698 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.655+0000 7f30c97dd700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f30c414e490 0x7f30c406cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:31.699 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.655+0000 7f30c9fde700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f30c40c3f50 0x7f30c4157890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:52792/0 (socket says 172.21.15.195:52792) 2022-06-27T12:01:31.699 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.655+0000 7f30c97dd700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f30c414e490 0x7f30c406cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:46604/0 (socket says 172.21.15.195:46604) 2022-06-27T12:01:31.700 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.655+0000 7f30c9fde700 1 -- 172.21.15.195:0/119339616 learned_addr learned my addr 172.21.15.195:0/119339616 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:01:31.700 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.655+0000 7f30c97dd700 1 -- 172.21.15.195:0/119339616 >> v1:172.21.15.202:6789/0 conn(0x7f30c414e860 legacy=0x7f30c40c3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:01:31.701 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.655+0000 7f30c97dd700 1 -- 172.21.15.195:0/119339616 >> v1:172.21.15.195:6789/0 conn(0x7f30c40c3f50 legacy=0x7f30c4157890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:31.701 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.655+0000 7f30c97dd700 1 -- 172.21.15.195:0/119339616 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30c41542c0 con 0x7f30c414e490 2022-06-27T12:01:31.701 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.655+0000 7f30c97dd700 1 --2- 172.21.15.195:0/119339616 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f30c414e490 0x7f30c406cfe0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f30c00097a0 tx=0x7f30c0009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7ba9019b3ae113b server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:01:31.701 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.656+0000 7f30b3fff700 1 -- 172.21.15.195:0/119339616 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30c0004030 con 0x7f30c414e490 2022-06-27T12:01:31.703 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.656+0000 7f30b3fff700 1 -- 172.21.15.195:0/119339616 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30c0004190 con 0x7f30c414e490 2022-06-27T12:01:31.708 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.656+0000 7f30b3fff700 1 -- 172.21.15.195:0/119339616 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30c0011630 con 0x7f30c414e490 2022-06-27T12:01:31.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.656+0000 7f30cba41700 1 -- 172.21.15.195:0/119339616 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f30c414e490 msgr2=0x7f30c406cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:31.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.656+0000 7f30cba41700 1 --2- 172.21.15.195:0/119339616 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f30c414e490 0x7f30c406cfe0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f30c00097a0 tx=0x7f30c0009060 comp rx=0 tx=0).stop 2022-06-27T12:01:31.710 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.656+0000 7f30cba41700 1 -- 172.21.15.195:0/119339616 shutdown_connections 2022-06-27T12:01:31.710 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.656+0000 7f30cba41700 1 --2- 172.21.15.195:0/119339616 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f30c414e490 0x7f30c406cfe0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:31.711 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.656+0000 7f30cba41700 1 -- 172.21.15.195:0/119339616 >> 172.21.15.195:0/119339616 conn(0x7f30c4063af0 msgr2=0x7f30c4063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:01:31.711 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.656+0000 7f30cba41700 1 -- 172.21.15.195:0/119339616 shutdown_connections 2022-06-27T12:01:31.712 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.656+0000 7f30cba41700 1 -- 172.21.15.195:0/119339616 wait complete. 2022-06-27T12:01:31.712 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.658+0000 7f30cba41700 1 Processor -- start 2022-06-27T12:01:31.712 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.658+0000 7f30cba41700 1 -- start start 2022-06-27T12:01:31.713 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.659+0000 7f30cba41700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f30c40c3f50 0x7f30c4157da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:31.713 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.659+0000 7f30cba41700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f30c414e490 0x7f30c41566d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:31.714 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.659+0000 7f30cba41700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f30c414e860 0x7f30c4156c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:31.714 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.659+0000 7f30cba41700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c4158600 con 0x7f30c414e860 2022-06-27T12:01:31.715 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.659+0000 7f30cba41700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c4156240 con 0x7f30c40c3f50 2022-06-27T12:01:31.722 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.659+0000 7f30cba41700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30c4155630 con 0x7f30c414e490 2022-06-27T12:01:31.723 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.659+0000 7f30c9fde700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f30c414e860 0x7f30c4156c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:31.723 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.659+0000 7f30c9fde700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f30c414e860 0x7f30c4156c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:46610/0 (socket says 172.21.15.195:46610) 2022-06-27T12:01:31.723 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.659+0000 7f30c9fde700 1 -- 172.21.15.195:0/424908951 learned_addr learned my addr 172.21.15.195:0/424908951 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:01:31.729 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.659+0000 7f30c97dd700 1 --2- 172.21.15.195:0/424908951 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f30c40c3f50 0x7f30c4157da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:31.730 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.659+0000 7f30c8fdc700 1 --2- 172.21.15.195:0/424908951 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f30c414e490 0x7f30c41566d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:31.731 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.659+0000 7f30c9fde700 1 -- 172.21.15.195:0/424908951 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f30c414e490 msgr2=0x7f30c41566d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:31.731 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.659+0000 7f30c9fde700 1 --2- 172.21.15.195:0/424908951 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f30c414e490 0x7f30c41566d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:31.731 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.659+0000 7f30c9fde700 1 -- 172.21.15.195:0/424908951 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f30c40c3f50 msgr2=0x7f30c4157da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:31.732 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.659+0000 7f30c9fde700 1 --2- 172.21.15.195:0/424908951 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f30c40c3f50 0x7f30c4157da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:31.733 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.659+0000 7f30c9fde700 1 -- 172.21.15.195:0/424908951 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30c00092e0 con 0x7f30c414e860 2022-06-27T12:01:31.733 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.659+0000 7f30c8fdc700 1 --2- 172.21.15.195:0/424908951 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f30c414e490 0x7f30c41566d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T12:01:31.734 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.659+0000 7f30c97dd700 1 --2- 172.21.15.195:0/424908951 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f30c40c3f50 0x7f30c4157da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T12:01:31.734 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.659+0000 7f30c9fde700 1 --2- 172.21.15.195:0/424908951 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f30c414e860 0x7f30c4156c10 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f30bc00c7f0 tx=0x7f30bc001540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:01:31.735 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.659+0000 7f30b17fa700 1 -- 172.21.15.195:0/424908951 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30bc005d70 con 0x7f30c414e860 2022-06-27T12:01:31.735 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.660+0000 7f30b17fa700 1 -- 172.21.15.195:0/424908951 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30bc0070f0 con 0x7f30c414e860 2022-06-27T12:01:31.736 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.660+0000 7f30b17fa700 1 -- 172.21.15.195:0/424908951 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30bc007470 con 0x7f30c414e860 2022-06-27T12:01:31.736 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.660+0000 7f30cba41700 1 -- 172.21.15.195:0/424908951 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30c4155910 con 0x7f30c414e860 2022-06-27T12:01:31.737 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.660+0000 7f30cba41700 1 -- 172.21.15.195:0/424908951 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30c4155ed0 con 0x7f30c414e860 2022-06-27T12:01:31.737 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.661+0000 7f30cba41700 1 -- 172.21.15.195:0/424908951 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30c41563d0 con 0x7f30c414e860 2022-06-27T12:01:31.737 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.661+0000 7f30b17fa700 1 -- 172.21.15.195:0/424908951 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f30bc010070 con 0x7f30c414e860 2022-06-27T12:01:31.738 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.661+0000 7f30b17fa700 1 --2- 172.21.15.195:0/424908951 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f30a4063d30 0x7f30a40661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:31.738 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.661+0000 7f30c97dd700 1 --2- 172.21.15.195:0/424908951 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f30a4063d30 0x7f30a40661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:31.739 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.661+0000 7f30b17fa700 1 -- 172.21.15.195:0/424908951 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6804+0+0 (secure 0 0 0) 0x7f30bc080610 con 0x7f30c414e860 2022-06-27T12:01:31.739 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.662+0000 7f30c97dd700 1 --2- 172.21.15.195:0/424908951 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f30a4063d30 0x7f30a40661e0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f30c00097a0 tx=0x7f30c0016000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:01:31.739 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.666+0000 7f30b17fa700 1 -- 172.21.15.195:0/424908951 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f30bc04ef90 con 0x7f30c414e860 2022-06-27T12:01:31.895 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.887+0000 7f30cba41700 1 -- 172.21.15.195:0/424908951 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f30c4156090 con 0x7f30c414e860 2022-06-27T12:01:31.899 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.887+0000 7f30b17fa700 1 -- 172.21.15.195:0/424908951 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+5364 (secure 0 0 0) 0x7f30bc0525b0 con 0x7f30c414e860 2022-06-27T12:01:31.900 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:01:31.901 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:01:30.014988+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5273":{"gid":5273,"name":"d","rank":0,"incarnation":49,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6835/417105338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":417105338},{"type":"v1","addr":"172.21.15.202:6835","nonce":417105338}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:01:31.904 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.889+0000 7f30cba41700 1 -- 172.21.15.195:0/424908951 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f30a4063d30 msgr2=0x7f30a40661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:31.905 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.889+0000 7f30cba41700 1 --2- 172.21.15.195:0/424908951 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f30a4063d30 0x7f30a40661e0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f30c00097a0 tx=0x7f30c0016000 comp rx=0 tx=0).stop 2022-06-27T12:01:31.907 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.889+0000 7f30cba41700 1 -- 172.21.15.195:0/424908951 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f30c414e860 msgr2=0x7f30c4156c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:31.907 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.889+0000 7f30cba41700 1 --2- 172.21.15.195:0/424908951 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f30c414e860 0x7f30c4156c10 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f30bc00c7f0 tx=0x7f30bc001540 comp rx=0 tx=0).stop 2022-06-27T12:01:31.907 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.889+0000 7f30cba41700 1 -- 172.21.15.195:0/424908951 shutdown_connections 2022-06-27T12:01:31.908 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.889+0000 7f30cba41700 1 --2- 172.21.15.195:0/424908951 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f30a4063d30 0x7f30a40661e0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:31.908 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.889+0000 7f30cba41700 1 --2- 172.21.15.195:0/424908951 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f30c414e860 0x7f30c4156c10 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:31.909 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.889+0000 7f30cba41700 1 --2- 172.21.15.195:0/424908951 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f30c40c3f50 0x7f30c4157da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:31.909 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.889+0000 7f30cba41700 1 --2- 172.21.15.195:0/424908951 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f30c414e490 0x7f30c41566d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:31.910 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.889+0000 7f30cba41700 1 -- 172.21.15.195:0/424908951 >> 172.21.15.195:0/424908951 conn(0x7f30c4063af0 msgr2=0x7f30c40c9150 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:01:31.910 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.889+0000 7f30cba41700 1 -- 172.21.15.195:0/424908951 shutdown_connections 2022-06-27T12:01:31.911 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:31.889+0000 7f30cba41700 1 -- 172.21.15.195:0/424908951 wait complete. 2022-06-27T12:01:31.912 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 49 2022-06-27T12:01:31.917 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 49, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-06-27T11:47:52.358095+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 49, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5273": { "addr": "172.21.15.202:6835/417105338", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6834", "nonce": 417105338, "type": "v2" }, { "addr": "172.21.15.202:6835", "nonce": 417105338, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5273, "incarnation": 49, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 55, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-06-27T12:01:30.014988+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5273 } } } ], "standbys": [ { "addr": "172.21.15.202:6837/3305006841", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6836", "nonce": 3305006841, "type": "v2" }, { "addr": "172.21.15.202:6837", "nonce": 3305006841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6835/2060709822", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6834", "nonce": 2060709822, "type": "v2" }, { "addr": "172.21.15.195:6835", "nonce": 2060709822, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6838/1598751652", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6836", "nonce": 1598751652, "type": "v2" }, { "addr": "172.21.15.195:6838", "nonce": 1598751652, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4275, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6839/1313800661", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6837", "nonce": 1313800661, "type": "v2" }, { "addr": "172.21.15.195:6839", "nonce": 1313800661, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4278, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-06-27T12:01:33.942 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:01:34.078 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.075+0000 7f8070590700 1 Processor -- start 2022-06-27T12:01:34.079 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.075+0000 7f8070590700 1 -- start start 2022-06-27T12:01:34.079 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.076+0000 7f8070590700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f806814e490 0x7f806806cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:34.079 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.076+0000 7f8070590700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80680c33b0 con 0x7f80680c3f50 2022-06-27T12:01:34.080 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.076+0000 7f8070590700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f806806d520 con 0x7f806814e860 2022-06-27T12:01:34.080 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.076+0000 7f8070590700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f806806f220 con 0x7f806814e490 2022-06-27T12:01:34.080 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.076+0000 7f806e32c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f806814e490 0x7f806806cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:34.080 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.076+0000 7f806eb2d700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f80680c3f50 0x7f8068157890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:52806/0 (socket says 172.21.15.195:52806) 2022-06-27T12:01:34.081 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.076+0000 7f806eb2d700 1 -- 172.21.15.195:0/3209227347 learned_addr learned my addr 172.21.15.195:0/3209227347 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:01:34.081 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.076+0000 7f806e32c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f806814e490 0x7f806806cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:46614/0 (socket says 172.21.15.195:46614) 2022-06-27T12:01:34.081 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.076+0000 7f806e32c700 1 -- 172.21.15.195:0/3209227347 >> v1:172.21.15.202:6789/0 conn(0x7f806814e860 legacy=0x7f80680c3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:34.084 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.076+0000 7f806e32c700 1 -- 172.21.15.195:0/3209227347 >> v1:172.21.15.195:6789/0 conn(0x7f80680c3f50 legacy=0x7f8068157890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:34.095 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.076+0000 7f806e32c700 1 -- 172.21.15.195:0/3209227347 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80681542c0 con 0x7f806814e490 2022-06-27T12:01:34.095 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.076+0000 7f806d32a700 1 -- 172.21.15.195:0/3209227347 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3732755905 0 0) 0x7f80680c33b0 con 0x7f80680c3f50 2022-06-27T12:01:34.096 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.077+0000 7f806e32c700 1 --2- 172.21.15.195:0/3209227347 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f806814e490 0x7f806806cfe0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f80580097a0 tx=0x7f8058009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5145f12cc8a317f9 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:01:34.096 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.077+0000 7f806d32a700 1 -- 172.21.15.195:0/3209227347 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8058011630 con 0x7f806814e490 2022-06-27T12:01:34.096 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.077+0000 7f806d32a700 1 -- 172.21.15.195:0/3209227347 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8058011790 con 0x7f806814e490 2022-06-27T12:01:34.097 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.077+0000 7f806d32a700 1 -- 172.21.15.195:0/3209227347 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8058011b30 con 0x7f806814e490 2022-06-27T12:01:34.097 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.077+0000 7f8070590700 1 -- 172.21.15.195:0/3209227347 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f806814e490 msgr2=0x7f806806cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:34.097 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.077+0000 7f8070590700 1 --2- 172.21.15.195:0/3209227347 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f806814e490 0x7f806806cfe0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f80580097a0 tx=0x7f8058009060 comp rx=0 tx=0).stop 2022-06-27T12:01:34.097 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.077+0000 7f8070590700 1 -- 172.21.15.195:0/3209227347 shutdown_connections 2022-06-27T12:01:34.097 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.077+0000 7f8070590700 1 --2- 172.21.15.195:0/3209227347 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f806814e490 0x7f806806cfe0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:34.098 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.077+0000 7f8070590700 1 -- 172.21.15.195:0/3209227347 >> 172.21.15.195:0/3209227347 conn(0x7f8068063af0 msgr2=0x7f8068063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:01:34.098 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.077+0000 7f8070590700 1 -- 172.21.15.195:0/3209227347 shutdown_connections 2022-06-27T12:01:34.098 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.077+0000 7f8070590700 1 -- 172.21.15.195:0/3209227347 wait complete. 2022-06-27T12:01:34.098 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.079+0000 7f8070590700 1 Processor -- start 2022-06-27T12:01:34.099 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.079+0000 7f8070590700 1 -- start start 2022-06-27T12:01:34.103 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.079+0000 7f8070590700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f80680c3f50 0x7f80681493a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:34.103 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.079+0000 7f8070590700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f806814e490 0x7f8068069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:34.104 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.079+0000 7f8070590700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f806814e860 0x7f8068069870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:34.104 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.079+0000 7f8070590700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8068155490 con 0x7f806814e490 2022-06-27T12:01:34.105 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.079+0000 7f8070590700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8068155d30 con 0x7f80680c3f50 2022-06-27T12:01:34.105 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.079+0000 7f8070590700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8068157d80 con 0x7f806814e860 2022-06-27T12:01:34.105 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.079+0000 7f806db2b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f806814e490 0x7f8068069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:34.106 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.079+0000 7f806db2b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f806814e490 0x7f8068069330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:46622/0 (socket says 172.21.15.195:46622) 2022-06-27T12:01:34.106 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.079+0000 7f806db2b700 1 -- 172.21.15.195:0/3134859772 learned_addr learned my addr 172.21.15.195:0/3134859772 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:01:34.107 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.080+0000 7f806eb2d700 1 --2- 172.21.15.195:0/3134859772 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f806814e860 0x7f8068069870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:34.107 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.080+0000 7f806e32c700 1 --2- 172.21.15.195:0/3134859772 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f80680c3f50 0x7f80681493a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:34.107 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.080+0000 7f806db2b700 1 -- 172.21.15.195:0/3134859772 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f806814e860 msgr2=0x7f8068069870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:34.107 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.080+0000 7f806db2b700 1 --2- 172.21.15.195:0/3134859772 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f806814e860 0x7f8068069870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:34.108 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.080+0000 7f806db2b700 1 -- 172.21.15.195:0/3134859772 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f80680c3f50 msgr2=0x7f80681493a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:34.108 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.080+0000 7f806db2b700 1 --2- 172.21.15.195:0/3134859772 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f80680c3f50 0x7f80681493a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:34.108 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.080+0000 7f806db2b700 1 -- 172.21.15.195:0/3134859772 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80580092e0 con 0x7f806814e490 2022-06-27T12:01:34.108 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.080+0000 7f806e32c700 1 --2- 172.21.15.195:0/3134859772 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f80680c3f50 0x7f80681493a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:01:34.109 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.080+0000 7f806eb2d700 1 --2- 172.21.15.195:0/3134859772 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f806814e860 0x7f8068069870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:01:34.109 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.080+0000 7f806db2b700 1 --2- 172.21.15.195:0/3134859772 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f806814e490 0x7f8068069330 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f806400c440 tx=0x7f8064007e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:01:34.109 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.080+0000 7f80567fc700 1 -- 172.21.15.195:0/3134859772 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80640050d0 con 0x7f806814e490 2022-06-27T12:01:34.109 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.080+0000 7f80567fc700 1 -- 172.21.15.195:0/3134859772 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8064005250 con 0x7f806814e490 2022-06-27T12:01:34.110 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.080+0000 7f80567fc700 1 -- 172.21.15.195:0/3134859772 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80640117e0 con 0x7f806814e490 2022-06-27T12:01:34.110 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.080+0000 7f8070590700 1 -- 172.21.15.195:0/3134859772 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8068158410 con 0x7f806814e490 2022-06-27T12:01:34.111 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.080+0000 7f8070590700 1 -- 172.21.15.195:0/3134859772 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80681559c0 con 0x7f806814e490 2022-06-27T12:01:34.111 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.081+0000 7f8070590700 1 -- 172.21.15.195:0/3134859772 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8068069030 con 0x7f806814e490 2022-06-27T12:01:34.111 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.081+0000 7f80567fc700 1 -- 172.21.15.195:0/3134859772 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f8064019030 con 0x7f806814e490 2022-06-27T12:01:34.111 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.082+0000 7f80567fc700 1 --2- 172.21.15.195:0/3134859772 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8048063d00 0x7f80480661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:34.112 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.082+0000 7f806e32c700 1 --2- 172.21.15.195:0/3134859772 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8048063d00 0x7f80480661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:34.112 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.082+0000 7f80567fc700 1 -- 172.21.15.195:0/3134859772 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6804+0+0 (secure 0 0 0) 0x7f806407fc40 con 0x7f806814e490 2022-06-27T12:01:34.112 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.082+0000 7f806e32c700 1 --2- 172.21.15.195:0/3134859772 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8048063d00 0x7f80480661b0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f8058011fd0 tx=0x7f8058017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:01:34.112 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.086+0000 7f80567fc700 1 -- 172.21.15.195:0/3134859772 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f806404e540 con 0x7f806814e490 2022-06-27T12:01:34.311 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.306+0000 7f8070590700 1 -- 172.21.15.195:0/3134859772 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8068155b80 con 0x7f806814e490 2022-06-27T12:01:34.312 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.306+0000 7f80567fc700 1 -- 172.21.15.195:0/3134859772 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+5364 (secure 0 0 0) 0x7f8064017020 con 0x7f806814e490 2022-06-27T12:01:34.312 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:01:34.335 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:01:30.014988+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5273":{"gid":5273,"name":"d","rank":0,"incarnation":49,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6835/417105338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":417105338},{"type":"v1","addr":"172.21.15.202:6835","nonce":417105338}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:01:34.339 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.308+0000 7f8070590700 1 -- 172.21.15.195:0/3134859772 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8048063d00 msgr2=0x7f80480661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:34.339 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.308+0000 7f8070590700 1 --2- 172.21.15.195:0/3134859772 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8048063d00 0x7f80480661b0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f8058011fd0 tx=0x7f8058017000 comp rx=0 tx=0).stop 2022-06-27T12:01:34.340 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.308+0000 7f8070590700 1 -- 172.21.15.195:0/3134859772 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f806814e490 msgr2=0x7f8068069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:34.340 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.308+0000 7f8070590700 1 --2- 172.21.15.195:0/3134859772 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f806814e490 0x7f8068069330 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f806400c440 tx=0x7f8064007e40 comp rx=0 tx=0).stop 2022-06-27T12:01:34.341 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.308+0000 7f8070590700 1 -- 172.21.15.195:0/3134859772 shutdown_connections 2022-06-27T12:01:34.341 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.308+0000 7f8070590700 1 --2- 172.21.15.195:0/3134859772 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8048063d00 0x7f80480661b0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:34.342 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.308+0000 7f8070590700 1 --2- 172.21.15.195:0/3134859772 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f806814e860 0x7f8068069870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:34.342 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.308+0000 7f8070590700 1 --2- 172.21.15.195:0/3134859772 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f80680c3f50 0x7f80681493a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:34.343 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.308+0000 7f8070590700 1 --2- 172.21.15.195:0/3134859772 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f806814e490 0x7f8068069330 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:34.343 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.308+0000 7f8070590700 1 -- 172.21.15.195:0/3134859772 >> 172.21.15.195:0/3134859772 conn(0x7f8068063af0 msgr2=0x7f8068063ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:01:34.343 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.308+0000 7f8070590700 1 -- 172.21.15.195:0/3134859772 shutdown_connections 2022-06-27T12:01:34.344 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:34.308+0000 7f8070590700 1 -- 172.21.15.195:0/3134859772 wait complete. 2022-06-27T12:01:34.345 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 49 2022-06-27T12:01:34.346 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:01:36.348 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:01:36.486 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.485+0000 7f8541522700 1 Processor -- start 2022-06-27T12:01:36.486 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.486+0000 7f8541522700 1 -- start start 2022-06-27T12:01:36.503 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.486+0000 7f8541522700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f853c0c64f0 0x7f853c150bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:36.504 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.486+0000 7f8541522700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f853c0c5d80 con 0x7f853c06d280 2022-06-27T12:01:36.504 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.486+0000 7f8541522700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f853c06b010 con 0x7f853c0c5480 2022-06-27T12:01:36.504 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.486+0000 7f8541522700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f853c0c75e0 con 0x7f853c0c64f0 2022-06-27T12:01:36.505 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.487+0000 7f853b7fe700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f853c06d280 0x7f853c161010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:52820/0 (socket says 172.21.15.195:52820) 2022-06-27T12:01:36.505 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.487+0000 7f853b7fe700 1 -- 172.21.15.195:0/2015988196 learned_addr learned my addr 172.21.15.195:0/2015988196 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:01:36.506 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.487+0000 7f853a7fc700 1 --2- 172.21.15.195:0/2015988196 >> v2:172.21.15.202:3300/0 conn(0x7f853c0c64f0 0x7f853c150bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:36.506 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.487+0000 7f8539ffb700 1 -- 172.21.15.195:0/2015988196 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3224176204 0 0) 0x7f853c0c5d80 con 0x7f853c06d280 2022-06-27T12:01:36.507 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.487+0000 7f8539ffb700 1 -- 172.21.15.195:0/2015988196 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f851c003430 con 0x7f853c06d280 2022-06-27T12:01:36.507 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.487+0000 7f853a7fc700 1 -- 172.21.15.195:0/2015988196 >> v1:172.21.15.202:6789/0 conn(0x7f853c0c5480 legacy=0x7f853c1504c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:36.507 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.487+0000 7f8539ffb700 1 -- 172.21.15.195:0/2015988196 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3770820241 0 0) 0x7f853c06b010 con 0x7f853c0c5480 2022-06-27T12:01:36.508 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.487+0000 7f853a7fc700 1 -- 172.21.15.195:0/2015988196 >> v1:172.21.15.195:6789/0 conn(0x7f853c06d280 legacy=0x7f853c161010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:36.508 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.487+0000 7f853a7fc700 1 -- 172.21.15.195:0/2015988196 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f853c0c78c0 con 0x7f853c0c64f0 2022-06-27T12:01:36.509 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.487+0000 7f853a7fc700 1 --2- 172.21.15.195:0/2015988196 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f853c0c64f0 0x7f853c150bd0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f85240097a0 tx=0x7f8524009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a265cfea8ad5cd0e server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:01:36.509 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.487+0000 7f8539ffb700 1 -- 172.21.15.195:0/2015988196 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8524011630 con 0x7f853c0c64f0 2022-06-27T12:01:36.510 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.487+0000 7f8539ffb700 1 -- 172.21.15.195:0/2015988196 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8524011790 con 0x7f853c0c64f0 2022-06-27T12:01:36.510 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.487+0000 7f8539ffb700 1 -- 172.21.15.195:0/2015988196 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8524011b30 con 0x7f853c0c64f0 2022-06-27T12:01:36.511 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.488+0000 7f8541522700 1 -- 172.21.15.195:0/2015988196 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f853c0c64f0 msgr2=0x7f853c150bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:36.511 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.488+0000 7f8541522700 1 --2- 172.21.15.195:0/2015988196 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f853c0c64f0 0x7f853c150bd0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f85240097a0 tx=0x7f8524009060 comp rx=0 tx=0).stop 2022-06-27T12:01:36.512 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.488+0000 7f8541522700 1 -- 172.21.15.195:0/2015988196 shutdown_connections 2022-06-27T12:01:36.512 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.488+0000 7f8541522700 1 --2- 172.21.15.195:0/2015988196 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f853c0c64f0 0x7f853c150bd0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:36.512 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.488+0000 7f8541522700 1 -- 172.21.15.195:0/2015988196 >> 172.21.15.195:0/2015988196 conn(0x7f853c063af0 msgr2=0x7f853c063ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:01:36.513 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.488+0000 7f8541522700 1 -- 172.21.15.195:0/2015988196 shutdown_connections 2022-06-27T12:01:36.513 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.488+0000 7f8541522700 1 -- 172.21.15.195:0/2015988196 wait complete. 2022-06-27T12:01:36.514 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.489+0000 7f8541522700 1 Processor -- start 2022-06-27T12:01:36.514 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.489+0000 7f8541522700 1 -- start start 2022-06-27T12:01:36.515 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.490+0000 7f8541522700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f853c06d280 0x7f853c14acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:36.515 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.490+0000 7f8541522700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f853c0c5480 0x7f853c156a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:36.515 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.490+0000 7f8541522700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f853c0c64f0 0x7f853c169a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:36.516 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.490+0000 7f8541522700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f853c14bb90 con 0x7f853c06d280 2022-06-27T12:01:36.516 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.490+0000 7f8541522700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f853c14a840 con 0x7f853c0c64f0 2022-06-27T12:01:36.516 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.490+0000 7f8541522700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f853c14b210 con 0x7f853c0c5480 2022-06-27T12:01:36.517 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.490+0000 7f853affd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f853c06d280 0x7f853c14acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:36.518 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.490+0000 7f853affd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f853c06d280 0x7f853c14acd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:46634/0 (socket says 172.21.15.195:46634) 2022-06-27T12:01:36.518 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.490+0000 7f853affd700 1 -- 172.21.15.195:0/98535061 learned_addr learned my addr 172.21.15.195:0/98535061 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:01:36.519 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.490+0000 7f853b7fe700 1 --2- 172.21.15.195:0/98535061 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f853c0c64f0 0x7f853c169a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:36.519 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.490+0000 7f853a7fc700 1 --2- 172.21.15.195:0/98535061 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f853c0c5480 0x7f853c156a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:36.519 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.490+0000 7f853affd700 1 -- 172.21.15.195:0/98535061 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f853c0c5480 msgr2=0x7f853c156a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:36.520 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.490+0000 7f853affd700 1 --2- 172.21.15.195:0/98535061 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f853c0c5480 0x7f853c156a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:36.520 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.490+0000 7f853affd700 1 -- 172.21.15.195:0/98535061 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f853c0c64f0 msgr2=0x7f853c169a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:36.521 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.490+0000 7f853affd700 1 --2- 172.21.15.195:0/98535061 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f853c0c64f0 0x7f853c169a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:36.521 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.490+0000 7f853affd700 1 -- 172.21.15.195:0/98535061 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85240092e0 con 0x7f853c06d280 2022-06-27T12:01:36.522 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.491+0000 7f853affd700 1 --2- 172.21.15.195:0/98535061 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f853c06d280 0x7f853c14acd0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f852c00c5c0 tx=0x7f852c007cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:01:36.522 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.491+0000 7f8532ffd700 1 -- 172.21.15.195:0/98535061 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f852c0052e0 con 0x7f853c06d280 2022-06-27T12:01:36.522 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.491+0000 7f8532ffd700 1 -- 172.21.15.195:0/98535061 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f852c003500 con 0x7f853c06d280 2022-06-27T12:01:36.523 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.491+0000 7f8541522700 1 -- 172.21.15.195:0/98535061 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f853c156000 con 0x7f853c06d280 2022-06-27T12:01:36.523 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.491+0000 7f8532ffd700 1 -- 172.21.15.195:0/98535061 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f852c0038c0 con 0x7f853c06d280 2022-06-27T12:01:36.524 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.491+0000 7f8541522700 1 -- 172.21.15.195:0/98535061 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f853c1565c0 con 0x7f853c06d280 2022-06-27T12:01:36.532 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.492+0000 7f8541522700 1 -- 172.21.15.195:0/98535061 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f853c0722c0 con 0x7f853c06d280 2022-06-27T12:01:36.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.492+0000 7f8532ffd700 1 -- 172.21.15.195:0/98535061 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f852c019030 con 0x7f853c06d280 2022-06-27T12:01:36.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.492+0000 7f8532ffd700 1 --2- 172.21.15.195:0/98535061 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8520063d30 0x7f85200661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:36.534 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.492+0000 7f8532ffd700 1 -- 172.21.15.195:0/98535061 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6804+0+0 (secure 0 0 0) 0x7f852c07fda0 con 0x7f853c06d280 2022-06-27T12:01:36.534 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.493+0000 7f853a7fc700 1 --2- 172.21.15.195:0/98535061 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8520063d30 0x7f85200661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:36.535 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.493+0000 7f853a7fc700 1 --2- 172.21.15.195:0/98535061 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8520063d30 0x7f85200661e0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f85240097a0 tx=0x7f8524017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:01:36.535 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.497+0000 7f8532ffd700 1 -- 172.21.15.195:0/98535061 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f852c04e6a0 con 0x7f853c06d280 2022-06-27T12:01:36.729 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.725+0000 7f8541522700 1 -- 172.21.15.195:0/98535061 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f853c0c7ba0 con 0x7f853c06d280 2022-06-27T12:01:36.730 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.726+0000 7f8532ffd700 1 -- 172.21.15.195:0/98535061 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+5364 (secure 0 0 0) 0x7f852c051cc0 con 0x7f853c06d280 2022-06-27T12:01:36.731 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:01:36.732 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:01:30.014988+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5273":{"gid":5273,"name":"d","rank":0,"incarnation":49,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6835/417105338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":417105338},{"type":"v1","addr":"172.21.15.202:6835","nonce":417105338}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:01:36.734 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.728+0000 7f8541522700 1 -- 172.21.15.195:0/98535061 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8520063d30 msgr2=0x7f85200661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:36.735 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.728+0000 7f8541522700 1 --2- 172.21.15.195:0/98535061 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8520063d30 0x7f85200661e0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f85240097a0 tx=0x7f8524017000 comp rx=0 tx=0).stop 2022-06-27T12:01:36.735 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.728+0000 7f8541522700 1 -- 172.21.15.195:0/98535061 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f853c06d280 msgr2=0x7f853c14acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:36.758 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.728+0000 7f8541522700 1 --2- 172.21.15.195:0/98535061 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f853c06d280 0x7f853c14acd0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f852c00c5c0 tx=0x7f852c007cb0 comp rx=0 tx=0).stop 2022-06-27T12:01:36.759 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.728+0000 7f8541522700 1 -- 172.21.15.195:0/98535061 shutdown_connections 2022-06-27T12:01:36.759 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.728+0000 7f8541522700 1 --2- 172.21.15.195:0/98535061 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8520063d30 0x7f85200661e0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:36.759 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.728+0000 7f8541522700 1 --2- 172.21.15.195:0/98535061 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f853c06d280 0x7f853c14acd0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:36.760 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.728+0000 7f8541522700 1 --2- 172.21.15.195:0/98535061 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f853c0c64f0 0x7f853c169a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:36.760 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.728+0000 7f8541522700 1 --2- 172.21.15.195:0/98535061 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f853c0c5480 0x7f853c156a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:36.761 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.728+0000 7f8541522700 1 -- 172.21.15.195:0/98535061 >> 172.21.15.195:0/98535061 conn(0x7f853c063af0 msgr2=0x7f853c0ca180 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:01:36.761 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.728+0000 7f8541522700 1 -- 172.21.15.195:0/98535061 shutdown_connections 2022-06-27T12:01:36.762 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:36.728+0000 7f8541522700 1 -- 172.21.15.195:0/98535061 wait complete. 2022-06-27T12:01:36.762 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 49 2022-06-27T12:01:36.764 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:01:38.766 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:01:38.911 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:38.911+0000 7f1cc7cf0700 1 Processor -- start 2022-06-27T12:01:38.911 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:38.911+0000 7f1cc7cf0700 1 -- start start 2022-06-27T12:01:38.917 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:38.911+0000 7f1cc7cf0700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1cc006e850 0x7f1cc01570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:38.917 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:38.912+0000 7f1cc7cf0700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1cc00c34a0 0x7f1cc00c2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:38.917 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:38.912+0000 7f1cc7cf0700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cc006fba0 con 0x7f1cc006fef0 2022-06-27T12:01:38.917 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:38.912+0000 7f1cc7cf0700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cc01575e0 con 0x7f1cc00c34a0 2022-06-27T12:01:38.926 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:38.912+0000 7f1cc7cf0700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc006f5e0 con 0x7f1cc006e850 2022-06-27T12:01:38.926 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:38.912+0000 7f1cc628d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1cc00c34a0 0x7f1cc00c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:38.927 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:38.912+0000 7f1cc528b700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f1cc006fef0 0x7f1cc0069de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:52830/0 (socket says 172.21.15.195:52830) 2022-06-27T12:01:38.927 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:38.912+0000 7f1cc528b700 1 -- 172.21.15.195:0/4125103669 learned_addr learned my addr 172.21.15.195:0/4125103669 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:01:38.927 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:38.912+0000 7f1cc5a8c700 1 --2- 172.21.15.195:0/4125103669 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1cc006e850 0x7f1cc01570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:38.927 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:38.912+0000 7f1cc628d700 1 -- 172.21.15.195:0/4125103669 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1cc006e850 msgr2=0x7f1cc01570a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:01:38.928 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:38.912+0000 7f1cc628d700 1 --2- 172.21.15.195:0/4125103669 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1cc006e850 0x7f1cc01570a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:38.928 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:38.912+0000 7f1cc628d700 1 -- 172.21.15.195:0/4125103669 >> v1:172.21.15.195:6789/0 conn(0x7f1cc006fef0 legacy=0x7f1cc0069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:38.928 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:38.912+0000 7f1cc628d700 1 -- 172.21.15.195:0/4125103669 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cc006f8c0 con 0x7f1cc00c34a0 2022-06-27T12:01:38.929 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:38.912+0000 7f1cc4a8a700 1 -- 172.21.15.195:0/4125103669 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2223444296 0 0) 0x7f1cc006fba0 con 0x7f1cc006fef0 2022-06-27T12:01:38.929 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:38.912+0000 7f1cc5a8c700 1 --2- 172.21.15.195:0/4125103669 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1cc006e850 0x7f1cc01570a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:01:38.930 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:38.912+0000 7f1cc628d700 1 --2- 172.21.15.195:0/4125103669 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1cc00c34a0 0x7f1cc00c2920 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f1cb8009e60 tx=0x7f1cb8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a3823afae7e71f5 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:01:38.930 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:38.912+0000 7f1cc4a8a700 1 -- 172.21.15.195:0/4125103669 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb800e070 con 0x7f1cc00c34a0 2022-06-27T12:01:38.930 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:38.912+0000 7f1cc4a8a700 1 -- 172.21.15.195:0/4125103669 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cb800bbf0 con 0x7f1cc00c34a0 2022-06-27T12:01:38.930 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:38.912+0000 7f1cc4a8a700 1 -- 172.21.15.195:0/4125103669 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb8012430 con 0x7f1cc00c34a0 2022-06-27T12:01:38.931 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:38.913+0000 7f1cc7cf0700 1 -- 172.21.15.195:0/4125103669 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1cc00c34a0 msgr2=0x7f1cc00c2920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:38.931 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:38.913+0000 7f1cc7cf0700 1 --2- 172.21.15.195:0/4125103669 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1cc00c34a0 0x7f1cc00c2920 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f1cb8009e60 tx=0x7f1cb8009780 comp rx=0 tx=0).stop 2022-06-27T12:01:38.931 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:38.913+0000 7f1cc7cf0700 1 -- 172.21.15.195:0/4125103669 shutdown_connections 2022-06-27T12:01:38.932 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:38.913+0000 7f1cc7cf0700 1 --2- 172.21.15.195:0/4125103669 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1cc00c34a0 0x7f1cc00c2920 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:38.932 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:38.913+0000 7f1cc7cf0700 1 --2- 172.21.15.195:0/4125103669 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1cc006e850 0x7f1cc01570a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:38.933 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:38.913+0000 7f1cc7cf0700 1 -- 172.21.15.195:0/4125103669 >> 172.21.15.195:0/4125103669 conn(0x7f1cc00c6350 msgr2=0x7f1cc00c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:01:38.933 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:38.913+0000 7f1cc7cf0700 1 -- 172.21.15.195:0/4125103669 shutdown_connections 2022-06-27T12:01:38.934 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:38.913+0000 7f1cc7cf0700 1 -- 172.21.15.195:0/4125103669 wait complete. 2022-06-27T12:01:38.934 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:38.914+0000 7f1cc7cf0700 1 Processor -- start 2022-06-27T12:01:38.934 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:38.914+0000 7f1cc7cf0700 1 -- start start 2022-06-27T12:01:38.935 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:38.915+0000 7f1cc7cf0700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1cc006e850 0x7f1cc0145240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:38.935 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:38.915+0000 7f1cc7cf0700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1cc00c34a0 0x7f1cc0145780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:38.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:38.915+0000 7f1cc7cf0700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1cc0147760 0x7f1cc0156260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:38.937 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:38.915+0000 7f1cc7cf0700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc0154c60 con 0x7f1cc006e850 2022-06-27T12:01:38.937 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:38.915+0000 7f1cc7cf0700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc0155db0 con 0x7f1cc00c34a0 2022-06-27T12:01:38.942 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:38.915+0000 7f1cc7cf0700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc0149050 con 0x7f1cc0147760 2022-06-27T12:01:38.943 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:38.915+0000 7f1cc5a8c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1cc006e850 0x7f1cc0145240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:38.943 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:38.915+0000 7f1cc5a8c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1cc006e850 0x7f1cc0145240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:46646/0 (socket says 172.21.15.195:46646) 2022-06-27T12:01:38.943 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:38.915+0000 7f1cc5a8c700 1 -- 172.21.15.195:0/3368497475 learned_addr learned my addr 172.21.15.195:0/3368497475 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:01:38.944 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:38.915+0000 7f1cc528b700 1 --2- 172.21.15.195:0/3368497475 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1cc00c34a0 0x7f1cc0145780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:38.945 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:38.915+0000 7f1cc628d700 1 --2- 172.21.15.195:0/3368497475 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1cc0147760 0x7f1cc0156260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:38.958 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:38.915+0000 7f1cc5a8c700 1 -- 172.21.15.195:0/3368497475 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1cc0147760 msgr2=0x7f1cc0156260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:38.958 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:38.915+0000 7f1cc5a8c700 1 --2- 172.21.15.195:0/3368497475 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1cc0147760 0x7f1cc0156260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:38.959 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:38.915+0000 7f1cc5a8c700 1 -- 172.21.15.195:0/3368497475 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1cc00c34a0 msgr2=0x7f1cc0145780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:38.960 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:38.915+0000 7f1cc5a8c700 1 --2- 172.21.15.195:0/3368497475 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1cc00c34a0 0x7f1cc0145780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:38.960 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:38.915+0000 7f1cc5a8c700 1 -- 172.21.15.195:0/3368497475 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cb00092e0 con 0x7f1cc006e850 2022-06-27T12:01:38.961 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:38.915+0000 7f1cc5a8c700 1 --2- 172.21.15.195:0/3368497475 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1cc006e850 0x7f1cc0145240 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f1cb000ea50 tx=0x7f1cb0009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:01:38.961 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:38.915+0000 7f1cadffb700 1 -- 172.21.15.195:0/3368497475 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb000ee90 con 0x7f1cc006e850 2022-06-27T12:01:38.961 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:38.916+0000 7f1cadffb700 1 -- 172.21.15.195:0/3368497475 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cb000c850 con 0x7f1cc006e850 2022-06-27T12:01:38.962 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:38.916+0000 7f1cc7cf0700 1 -- 172.21.15.195:0/3368497475 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cb8009a20 con 0x7f1cc006e850 2022-06-27T12:01:38.962 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:38.916+0000 7f1cadffb700 1 -- 172.21.15.195:0/3368497475 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb000cbd0 con 0x7f1cc006e850 2022-06-27T12:01:38.963 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:38.916+0000 7f1cc7cf0700 1 -- 172.21.15.195:0/3368497475 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cc01492f0 con 0x7f1cc006e850 2022-06-27T12:01:38.964 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:38.917+0000 7f1cc7cf0700 1 -- 172.21.15.195:0/3368497475 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cc00722c0 con 0x7f1cc006e850 2022-06-27T12:01:38.964 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:38.917+0000 7f1cadffb700 1 -- 172.21.15.195:0/3368497475 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f1cb0023070 con 0x7f1cc006e850 2022-06-27T12:01:38.966 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:38.917+0000 7f1cadffb700 1 --2- 172.21.15.195:0/3368497475 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1ca0063d00 0x7f1ca00661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:38.967 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:38.918+0000 7f1cadffb700 1 -- 172.21.15.195:0/3368497475 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6804+0+0 (secure 0 0 0) 0x7f1cb0080f20 con 0x7f1cc006e850 2022-06-27T12:01:38.967 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:38.918+0000 7f1cc528b700 1 --2- 172.21.15.195:0/3368497475 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1ca0063d00 0x7f1ca00661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:38.968 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:38.918+0000 7f1cc528b700 1 --2- 172.21.15.195:0/3368497475 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1ca0063d00 0x7f1ca00661b0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f1cc0069230 tx=0x7f1cbc007bf0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:01:38.968 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:38.922+0000 7f1cadffb700 1 -- 172.21.15.195:0/3368497475 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f1cb004f820 con 0x7f1cc006e850 2022-06-27T12:01:39.156 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:39.153+0000 7f1cc7cf0700 1 -- 172.21.15.195:0/3368497475 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1cc00c0900 con 0x7f1cc006e850 2022-06-27T12:01:39.164 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:39.154+0000 7f1cadffb700 1 -- 172.21.15.195:0/3368497475 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+5364 (secure 0 0 0) 0x7f1cb0021090 con 0x7f1cc006e850 2022-06-27T12:01:39.165 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:01:39.165 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:01:30.014988+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5273":{"gid":5273,"name":"d","rank":0,"incarnation":49,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6835/417105338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":417105338},{"type":"v1","addr":"172.21.15.202:6835","nonce":417105338}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:01:39.167 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:39.156+0000 7f1cc7cf0700 1 -- 172.21.15.195:0/3368497475 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1ca0063d00 msgr2=0x7f1ca00661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:39.167 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:39.156+0000 7f1cc7cf0700 1 --2- 172.21.15.195:0/3368497475 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1ca0063d00 0x7f1ca00661b0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f1cc0069230 tx=0x7f1cbc007bf0 comp rx=0 tx=0).stop 2022-06-27T12:01:39.167 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:39.156+0000 7f1cc7cf0700 1 -- 172.21.15.195:0/3368497475 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1cc006e850 msgr2=0x7f1cc0145240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:39.168 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:39.156+0000 7f1cc7cf0700 1 --2- 172.21.15.195:0/3368497475 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1cc006e850 0x7f1cc0145240 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f1cb000ea50 tx=0x7f1cb0009060 comp rx=0 tx=0).stop 2022-06-27T12:01:39.168 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:39.156+0000 7f1cc7cf0700 1 -- 172.21.15.195:0/3368497475 shutdown_connections 2022-06-27T12:01:39.169 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:39.156+0000 7f1cc7cf0700 1 --2- 172.21.15.195:0/3368497475 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1ca0063d00 0x7f1ca00661b0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:39.169 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:39.156+0000 7f1cc7cf0700 1 --2- 172.21.15.195:0/3368497475 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1cc006e850 0x7f1cc0145240 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:39.170 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:39.156+0000 7f1cc7cf0700 1 --2- 172.21.15.195:0/3368497475 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1cc0147760 0x7f1cc0156260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:39.170 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:39.156+0000 7f1cc7cf0700 1 --2- 172.21.15.195:0/3368497475 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1cc00c34a0 0x7f1cc0145780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:39.170 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:39.156+0000 7f1cc7cf0700 1 -- 172.21.15.195:0/3368497475 >> 172.21.15.195:0/3368497475 conn(0x7f1cc00c6350 msgr2=0x7f1cc00c2cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:01:39.170 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:39.156+0000 7f1cc7cf0700 1 -- 172.21.15.195:0/3368497475 shutdown_connections 2022-06-27T12:01:39.171 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:39.156+0000 7f1cc7cf0700 1 -- 172.21.15.195:0/3368497475 wait complete. 2022-06-27T12:01:39.171 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 49 2022-06-27T12:01:39.173 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:01:41.176 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:01:41.331 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.322+0000 7f4ba523c700 1 Processor -- start 2022-06-27T12:01:41.332 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.322+0000 7f4ba523c700 1 -- start start 2022-06-27T12:01:41.333 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.323+0000 7f4ba523c700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f4ba006e850 0x7f4ba01570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:41.333 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.323+0000 7f4ba523c700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4ba00c34a0 0x7f4ba00c2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:41.334 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.323+0000 7f4ba523c700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ba006fba0 con 0x7f4ba006fef0 2022-06-27T12:01:41.334 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.323+0000 7f4ba523c700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ba01575e0 con 0x7f4ba006e850 2022-06-27T12:01:41.335 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.323+0000 7f4ba523c700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba006f5e0 con 0x7f4ba00c34a0 2022-06-27T12:01:41.335 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.323+0000 7f4b9e59c700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f4ba006fef0 0x7f4ba0069de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:52846/0 (socket says 172.21.15.195:52846) 2022-06-27T12:01:41.335 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.323+0000 7f4b9e59c700 1 -- 172.21.15.195:0/3211825111 learned_addr learned my addr 172.21.15.195:0/3211825111 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:01:41.336 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.323+0000 7f4b9f59e700 1 --2- 172.21.15.195:0/3211825111 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4ba00c34a0 0x7f4ba00c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:41.336 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.323+0000 7f4b9ed9d700 1 --2- 172.21.15.195:0/3211825111 >> v2:172.21.15.202:3300/0 conn(0x7f4ba006e850 0x7f4ba01570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:41.343 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.324+0000 7f4b9dd9b700 1 -- 172.21.15.195:0/3211825111 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 559013014 0 0) 0x7f4ba006fba0 con 0x7f4ba006fef0 2022-06-27T12:01:41.343 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.324+0000 7f4b9dd9b700 1 -- 172.21.15.195:0/3211825111 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b8c003430 con 0x7f4ba006fef0 2022-06-27T12:01:41.344 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.324+0000 7f4b9ed9d700 1 -- 172.21.15.195:0/3211825111 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4ba00c34a0 msgr2=0x7f4ba00c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:01:41.344 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.324+0000 7f4b9ed9d700 1 --2- 172.21.15.195:0/3211825111 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4ba00c34a0 0x7f4ba00c2920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:41.344 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.324+0000 7f4b9ed9d700 1 -- 172.21.15.195:0/3211825111 >> v1:172.21.15.195:6789/0 conn(0x7f4ba006fef0 legacy=0x7f4ba0069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:41.345 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.324+0000 7f4b9dd9b700 1 -- 172.21.15.195:0/3211825111 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2774007369 0 0) 0x7f4b8c003430 con 0x7f4ba006fef0 2022-06-27T12:01:41.345 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.324+0000 7f4b9ed9d700 1 -- 172.21.15.195:0/3211825111 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ba006f8c0 con 0x7f4ba006e850 2022-06-27T12:01:41.345 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.324+0000 7f4b9ed9d700 1 --2- 172.21.15.195:0/3211825111 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4ba006e850 0x7f4ba01570a0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f4b940097a0 tx=0x7f4b94009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f494cb0b56ed60d7 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:01:41.348 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.324+0000 7f4b9dd9b700 1 -- 172.21.15.195:0/3211825111 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b90001470 con 0x7f4ba006e850 2022-06-27T12:01:41.349 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.324+0000 7f4b9dd9b700 1 -- 172.21.15.195:0/3211825111 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b94004030 con 0x7f4ba006e850 2022-06-27T12:01:41.349 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.324+0000 7f4b9dd9b700 1 -- 172.21.15.195:0/3211825111 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b940043b0 con 0x7f4ba006e850 2022-06-27T12:01:41.350 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.325+0000 7f4ba523c700 1 -- 172.21.15.195:0/3211825111 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4ba006e850 msgr2=0x7f4ba01570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:41.350 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.325+0000 7f4ba523c700 1 --2- 172.21.15.195:0/3211825111 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4ba006e850 0x7f4ba01570a0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f4b940097a0 tx=0x7f4b94009060 comp rx=0 tx=0).stop 2022-06-27T12:01:41.351 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.325+0000 7f4ba523c700 1 -- 172.21.15.195:0/3211825111 shutdown_connections 2022-06-27T12:01:41.351 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.325+0000 7f4ba523c700 1 --2- 172.21.15.195:0/3211825111 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4ba00c34a0 0x7f4ba00c2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:41.352 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.325+0000 7f4ba523c700 1 --2- 172.21.15.195:0/3211825111 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4ba006e850 0x7f4ba01570a0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:41.352 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.325+0000 7f4ba523c700 1 -- 172.21.15.195:0/3211825111 >> 172.21.15.195:0/3211825111 conn(0x7f4ba00c6350 msgr2=0x7f4ba00c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:01:41.353 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.325+0000 7f4ba523c700 1 -- 172.21.15.195:0/3211825111 shutdown_connections 2022-06-27T12:01:41.353 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.325+0000 7f4ba523c700 1 -- 172.21.15.195:0/3211825111 wait complete. 2022-06-27T12:01:41.354 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.326+0000 7f4ba523c700 1 Processor -- start 2022-06-27T12:01:41.355 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.326+0000 7f4ba523c700 1 -- start start 2022-06-27T12:01:41.355 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.327+0000 7f4ba523c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4ba006e850 0x7f4ba0163960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:41.356 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.327+0000 7f4ba523c700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4ba00c34a0 0x7f4ba014f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:41.356 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.327+0000 7f4ba523c700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4ba0166cb0 0x7f4ba014b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:41.357 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.327+0000 7f4ba523c700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba00be720 con 0x7f4ba006e850 2022-06-27T12:01:41.357 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.327+0000 7f4ba523c700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba0166820 con 0x7f4ba0166cb0 2022-06-27T12:01:41.357 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.327+0000 7f4ba523c700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba00bfd20 con 0x7f4ba00c34a0 2022-06-27T12:01:41.358 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.327+0000 7f4b9ed9d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4ba006e850 0x7f4ba0163960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:41.359 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.327+0000 7f4b9ed9d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4ba006e850 0x7f4ba0163960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:46664/0 (socket says 172.21.15.195:46664) 2022-06-27T12:01:41.360 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.327+0000 7f4b9ed9d700 1 -- 172.21.15.195:0/1115821374 learned_addr learned my addr 172.21.15.195:0/1115821374 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:01:41.360 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.327+0000 7f4b9f59e700 1 --2- 172.21.15.195:0/1115821374 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4ba0166cb0 0x7f4ba014b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:41.361 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.327+0000 7f4b9e59c700 1 --2- 172.21.15.195:0/1115821374 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4ba00c34a0 0x7f4ba014f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:41.361 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.327+0000 7f4b9ed9d700 1 -- 172.21.15.195:0/1115821374 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4ba00c34a0 msgr2=0x7f4ba014f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:41.362 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.327+0000 7f4b9ed9d700 1 --2- 172.21.15.195:0/1115821374 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4ba00c34a0 0x7f4ba014f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:41.362 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.327+0000 7f4b9ed9d700 1 -- 172.21.15.195:0/1115821374 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4ba0166cb0 msgr2=0x7f4ba014b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:41.363 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.327+0000 7f4b9ed9d700 1 --2- 172.21.15.195:0/1115821374 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4ba0166cb0 0x7f4ba014b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:41.363 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.327+0000 7f4b9ed9d700 1 -- 172.21.15.195:0/1115821374 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b940092e0 con 0x7f4ba006e850 2022-06-27T12:01:41.363 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.327+0000 7f4b9ed9d700 1 --2- 172.21.15.195:0/1115821374 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4ba006e850 0x7f4ba0163960 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f4b94003de0 tx=0x7f4b94015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:01:41.364 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.327+0000 7f4b86ffd700 1 -- 172.21.15.195:0/1115821374 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b94011650 con 0x7f4ba006e850 2022-06-27T12:01:41.365 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.327+0000 7f4b86ffd700 1 -- 172.21.15.195:0/1115821374 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b940117b0 con 0x7f4ba006e850 2022-06-27T12:01:41.365 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.327+0000 7f4b86ffd700 1 -- 172.21.15.195:0/1115821374 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b94011b50 con 0x7f4ba006e850 2022-06-27T12:01:41.365 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.327+0000 7f4ba523c700 1 -- 172.21.15.195:0/1115821374 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ba00bffc0 con 0x7f4ba006e850 2022-06-27T12:01:41.365 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.328+0000 7f4ba523c700 1 -- 172.21.15.195:0/1115821374 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ba00c0580 con 0x7f4ba006e850 2022-06-27T12:01:41.366 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.328+0000 7f4ba523c700 1 -- 172.21.15.195:0/1115821374 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ba00722c0 con 0x7f4ba006e850 2022-06-27T12:01:41.366 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.329+0000 7f4b86ffd700 1 -- 172.21.15.195:0/1115821374 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f4b9400f070 con 0x7f4ba006e850 2022-06-27T12:01:41.366 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.329+0000 7f4b86ffd700 1 --2- 172.21.15.195:0/1115821374 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4b80063d00 0x7f4b800661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:41.366 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.329+0000 7f4b86ffd700 1 -- 172.21.15.195:0/1115821374 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6804+0+0 (secure 0 0 0) 0x7f4b940823f0 con 0x7f4ba006e850 2022-06-27T12:01:41.367 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.329+0000 7f4b9e59c700 1 --2- 172.21.15.195:0/1115821374 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4b80063d00 0x7f4b800661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:41.367 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.330+0000 7f4b9e59c700 1 --2- 172.21.15.195:0/1115821374 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4b80063d00 0x7f4b800661b0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f4ba01528e0 tx=0x7f4b90001970 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:01:41.367 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.334+0000 7f4b86ffd700 1 -- 172.21.15.195:0/1115821374 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f4b94050c00 con 0x7f4ba006e850 2022-06-27T12:01:41.569 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.568+0000 7f4ba523c700 1 -- 172.21.15.195:0/1115821374 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4ba006f240 con 0x7f4ba006e850 2022-06-27T12:01:41.582 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.568+0000 7f4b86ffd700 1 -- 172.21.15.195:0/1115821374 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+5364 (secure 0 0 0) 0x7f4b94054220 con 0x7f4ba006e850 2022-06-27T12:01:41.583 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:01:41.586 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:01:30.014988+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5273":{"gid":5273,"name":"d","rank":0,"incarnation":49,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6835/417105338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":417105338},{"type":"v1","addr":"172.21.15.202:6835","nonce":417105338}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:01:41.588 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.570+0000 7f4ba523c700 1 -- 172.21.15.195:0/1115821374 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4b80063d00 msgr2=0x7f4b800661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:41.588 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.570+0000 7f4ba523c700 1 --2- 172.21.15.195:0/1115821374 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4b80063d00 0x7f4b800661b0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f4ba01528e0 tx=0x7f4b90001970 comp rx=0 tx=0).stop 2022-06-27T12:01:41.588 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.570+0000 7f4ba523c700 1 -- 172.21.15.195:0/1115821374 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4ba006e850 msgr2=0x7f4ba0163960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:41.589 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.570+0000 7f4ba523c700 1 --2- 172.21.15.195:0/1115821374 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4ba006e850 0x7f4ba0163960 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f4b94003de0 tx=0x7f4b94015000 comp rx=0 tx=0).stop 2022-06-27T12:01:41.611 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.570+0000 7f4ba523c700 1 -- 172.21.15.195:0/1115821374 shutdown_connections 2022-06-27T12:01:41.612 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.570+0000 7f4ba523c700 1 --2- 172.21.15.195:0/1115821374 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4b80063d00 0x7f4b800661b0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:41.612 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.570+0000 7f4ba523c700 1 --2- 172.21.15.195:0/1115821374 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4ba006e850 0x7f4ba0163960 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:41.614 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.570+0000 7f4ba523c700 1 --2- 172.21.15.195:0/1115821374 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4ba0166cb0 0x7f4ba014b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:41.614 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.570+0000 7f4ba523c700 1 --2- 172.21.15.195:0/1115821374 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4ba00c34a0 0x7f4ba014f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:41.618 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.570+0000 7f4ba523c700 1 -- 172.21.15.195:0/1115821374 >> 172.21.15.195:0/1115821374 conn(0x7f4ba00c6350 msgr2=0x7f4ba0150550 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:01:41.618 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.570+0000 7f4ba523c700 1 -- 172.21.15.195:0/1115821374 shutdown_connections 2022-06-27T12:01:41.618 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:41.570+0000 7f4ba523c700 1 -- 172.21.15.195:0/1115821374 wait complete. 2022-06-27T12:01:41.619 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 49 2022-06-27T12:01:41.619 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:01:43.620 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:01:43.776 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:43.772+0000 7fd6e0444700 1 Processor -- start 2022-06-27T12:01:43.776 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:43.772+0000 7fd6e0444700 1 -- start start 2022-06-27T12:01:43.777 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:43.772+0000 7fd6e0444700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd6d806a450 0x7fd6d8063af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:43.778 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:43.772+0000 7fd6e0444700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd6d81508d0 0x7fd6d80bbf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:43.778 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:43.772+0000 7fd6e0444700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6d8064030 con 0x7fd6d8150500 2022-06-27T12:01:43.779 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:43.772+0000 7fd6e0444700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6d80c4160 con 0x7fd6d81508d0 2022-06-27T12:01:43.779 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:43.772+0000 7fd6e0444700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d80c4440 con 0x7fd6d806a450 2022-06-27T12:01:43.780 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:43.772+0000 7fd6de9e1700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd6d81508d0 0x7fd6d80bbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:43.781 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:43.772+0000 7fd6de9e1700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd6d81508d0 0x7fd6d80bbf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:46678/0 (socket says 172.21.15.195:46678) 2022-06-27T12:01:43.781 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:43.772+0000 7fd6de9e1700 1 -- 172.21.15.195:0/4143948946 learned_addr learned my addr 172.21.15.195:0/4143948946 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:01:43.783 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:43.772+0000 7fd6dd9df700 1 --2- 172.21.15.195:0/4143948946 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd6d806a450 0x7fd6d8063af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:43.783 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:43.772+0000 7fd6de9e1700 1 -- 172.21.15.195:0/4143948946 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd6d806a450 msgr2=0x7fd6d8063af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:01:43.784 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:43.772+0000 7fd6de9e1700 1 --2- 172.21.15.195:0/4143948946 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd6d806a450 0x7fd6d8063af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:43.784 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:43.772+0000 7fd6de9e1700 1 -- 172.21.15.195:0/4143948946 >> v1:172.21.15.202:6789/0 conn(0x7fd6d8150500 legacy=0x7fd6d80c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:43.785 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:43.772+0000 7fd6de9e1700 1 -- 172.21.15.195:0/4143948946 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6d815a800 con 0x7fd6d81508d0 2022-06-27T12:01:43.785 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:43.772+0000 7fd6de9e1700 1 --2- 172.21.15.195:0/4143948946 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd6d81508d0 0x7fd6d80bbf50 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fd6d0009f20 tx=0x7fd6d0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1030a9f7950b1a4d server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:01:43.795 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:43.772+0000 7fd6dd1de700 1 -- 172.21.15.195:0/4143948946 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d000e070 con 0x7fd6d81508d0 2022-06-27T12:01:43.795 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:43.773+0000 7fd6dd1de700 1 -- 172.21.15.195:0/4143948946 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6d000b9d0 con 0x7fd6d81508d0 2022-06-27T12:01:43.796 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:43.773+0000 7fd6dd1de700 1 -- 172.21.15.195:0/4143948946 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d00092e0 con 0x7fd6d81508d0 2022-06-27T12:01:43.796 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:43.773+0000 7fd6e0444700 1 -- 172.21.15.195:0/4143948946 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd6d81508d0 msgr2=0x7fd6d80bbf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:43.797 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:43.773+0000 7fd6e0444700 1 --2- 172.21.15.195:0/4143948946 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd6d81508d0 0x7fd6d80bbf50 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fd6d0009f20 tx=0x7fd6d0009840 comp rx=0 tx=0).stop 2022-06-27T12:01:43.798 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:43.773+0000 7fd6e0444700 1 -- 172.21.15.195:0/4143948946 shutdown_connections 2022-06-27T12:01:43.799 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:43.773+0000 7fd6e0444700 1 --2- 172.21.15.195:0/4143948946 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd6d81508d0 0x7fd6d80bbf50 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:43.799 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:43.773+0000 7fd6e0444700 1 --2- 172.21.15.195:0/4143948946 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd6d806a450 0x7fd6d8063af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:43.800 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:43.773+0000 7fd6e0444700 1 -- 172.21.15.195:0/4143948946 >> 172.21.15.195:0/4143948946 conn(0x7fd6d80c9b00 msgr2=0x7fd6d80c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:01:43.800 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:43.773+0000 7fd6e0444700 1 -- 172.21.15.195:0/4143948946 shutdown_connections 2022-06-27T12:01:43.801 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:43.773+0000 7fd6e0444700 1 -- 172.21.15.195:0/4143948946 wait complete. 2022-06-27T12:01:43.801 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:43.774+0000 7fd6e0444700 1 Processor -- start 2022-06-27T12:01:43.801 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:43.774+0000 7fd6e0444700 1 -- start start 2022-06-27T12:01:43.805 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:43.775+0000 7fd6e0444700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd6d806a450 0x7fd6d80685c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:43.806 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:43.775+0000 7fd6e0444700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd6d8150500 0x7fd6d814da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:43.806 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:43.775+0000 7fd6e0444700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd6d81508d0 0x7fd6d814df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:43.807 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:43.775+0000 7fd6e0444700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d80c0d30 con 0x7fd6d81508d0 2022-06-27T12:01:43.807 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:43.775+0000 7fd6e0444700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d80be130 con 0x7fd6d806a450 2022-06-27T12:01:43.808 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:43.775+0000 7fd6e0444700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d8166420 con 0x7fd6d8150500 2022-06-27T12:01:43.808 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:43.775+0000 7fd6de9e1700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd6d81508d0 0x7fd6d814df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:43.809 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:43.775+0000 7fd6de9e1700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd6d81508d0 0x7fd6d814df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:46684/0 (socket says 172.21.15.195:46684) 2022-06-27T12:01:43.810 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:43.775+0000 7fd6de9e1700 1 -- 172.21.15.195:0/2556450027 learned_addr learned my addr 172.21.15.195:0/2556450027 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:01:43.810 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:43.775+0000 7fd6dd9df700 1 --2- 172.21.15.195:0/2556450027 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd6d8150500 0x7fd6d814da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:43.811 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:43.775+0000 7fd6de1e0700 1 --2- 172.21.15.195:0/2556450027 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd6d806a450 0x7fd6d80685c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:43.811 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:43.776+0000 7fd6de9e1700 1 -- 172.21.15.195:0/2556450027 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd6d8150500 msgr2=0x7fd6d814da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:43.811 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:43.776+0000 7fd6de9e1700 1 --2- 172.21.15.195:0/2556450027 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd6d8150500 0x7fd6d814da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:43.812 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:43.776+0000 7fd6de9e1700 1 -- 172.21.15.195:0/2556450027 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd6d806a450 msgr2=0x7fd6d80685c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:43.812 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:43.776+0000 7fd6de9e1700 1 --2- 172.21.15.195:0/2556450027 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd6d806a450 0x7fd6d80685c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:43.813 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:43.776+0000 7fd6de9e1700 1 -- 172.21.15.195:0/2556450027 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6d0009ae0 con 0x7fd6d81508d0 2022-06-27T12:01:43.813 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:43.776+0000 7fd6de9e1700 1 --2- 172.21.15.195:0/2556450027 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd6d81508d0 0x7fd6d814df40 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fd6d000b610 tx=0x7fd6d0018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:01:43.815 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:43.776+0000 7fd6ca7fc700 1 -- 172.21.15.195:0/2556450027 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d00094e0 con 0x7fd6d81508d0 2022-06-27T12:01:43.815 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:43.776+0000 7fd6ca7fc700 1 -- 172.21.15.195:0/2556450027 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6d0004040 con 0x7fd6d81508d0 2022-06-27T12:01:43.817 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:43.776+0000 7fd6e0444700 1 -- 172.21.15.195:0/2556450027 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6d80be390 con 0x7fd6d81508d0 2022-06-27T12:01:43.818 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:43.776+0000 7fd6ca7fc700 1 -- 172.21.15.195:0/2556450027 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d0004400 con 0x7fd6d81508d0 2022-06-27T12:01:43.818 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:43.776+0000 7fd6e0444700 1 -- 172.21.15.195:0/2556450027 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6d80be950 con 0x7fd6d81508d0 2022-06-27T12:01:43.818 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:43.777+0000 7fd6e0444700 1 -- 172.21.15.195:0/2556450027 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6d81665e0 con 0x7fd6d81508d0 2022-06-27T12:01:43.819 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:43.777+0000 7fd6ca7fc700 1 -- 172.21.15.195:0/2556450027 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fd6d001d420 con 0x7fd6d81508d0 2022-06-27T12:01:43.819 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:43.778+0000 7fd6ca7fc700 1 --2- 172.21.15.195:0/2556450027 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd6b8063d00 0x7fd6b80661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:43.820 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:43.778+0000 7fd6ca7fc700 1 -- 172.21.15.195:0/2556450027 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6804+0+0 (secure 0 0 0) 0x7fd6d00828e0 con 0x7fd6d81508d0 2022-06-27T12:01:43.820 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:43.778+0000 7fd6de1e0700 1 --2- 172.21.15.195:0/2556450027 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd6b8063d00 0x7fd6b80661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:43.820 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:43.778+0000 7fd6de1e0700 1 --2- 172.21.15.195:0/2556450027 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd6b8063d00 0x7fd6b80661b0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fd6d4007fd0 tx=0x7fd6d4009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:01:43.821 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:43.783+0000 7fd6ca7fc700 1 -- 172.21.15.195:0/2556450027 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fd6d0051180 con 0x7fd6d81508d0 2022-06-27T12:01:44.011 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:44.011+0000 7fd6e0444700 1 -- 172.21.15.195:0/2556450027 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd6d8069fb0 con 0x7fd6d81508d0 2022-06-27T12:01:44.011 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:44.011+0000 7fd6ca7fc700 1 -- 172.21.15.195:0/2556450027 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+5368 (secure 0 0 0) 0x7fd6d00547a0 con 0x7fd6d81508d0 2022-06-27T12:01:44.012 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:01:44.013 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:01:43.204516+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5273":{"gid":5273,"name":"d","rank":0,"incarnation":49,"state":"up:reconnect","state_seq":36,"addr":"172.21.15.202:6835/417105338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":417105338},{"type":"v1","addr":"172.21.15.202:6835","nonce":417105338}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:01:44.019 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:44.013+0000 7fd6e0444700 1 -- 172.21.15.195:0/2556450027 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd6b8063d00 msgr2=0x7fd6b80661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:44.020 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:44.013+0000 7fd6e0444700 1 --2- 172.21.15.195:0/2556450027 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd6b8063d00 0x7fd6b80661b0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fd6d4007fd0 tx=0x7fd6d4009040 comp rx=0 tx=0).stop 2022-06-27T12:01:44.020 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:44.013+0000 7fd6e0444700 1 -- 172.21.15.195:0/2556450027 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd6d81508d0 msgr2=0x7fd6d814df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:44.021 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:44.013+0000 7fd6e0444700 1 --2- 172.21.15.195:0/2556450027 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd6d81508d0 0x7fd6d814df40 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fd6d000b610 tx=0x7fd6d0018000 comp rx=0 tx=0).stop 2022-06-27T12:01:44.021 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:44.013+0000 7fd6e0444700 1 -- 172.21.15.195:0/2556450027 shutdown_connections 2022-06-27T12:01:44.022 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:44.013+0000 7fd6e0444700 1 --2- 172.21.15.195:0/2556450027 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd6b8063d00 0x7fd6b80661b0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:44.022 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:44.013+0000 7fd6e0444700 1 --2- 172.21.15.195:0/2556450027 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd6d81508d0 0x7fd6d814df40 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:44.029 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:44.013+0000 7fd6e0444700 1 --2- 172.21.15.195:0/2556450027 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd6d806a450 0x7fd6d80685c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:44.030 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:44.013+0000 7fd6e0444700 1 --2- 172.21.15.195:0/2556450027 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd6d8150500 0x7fd6d814da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:44.030 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:44.013+0000 7fd6e0444700 1 -- 172.21.15.195:0/2556450027 >> 172.21.15.195:0/2556450027 conn(0x7fd6d80c9b00 msgr2=0x7fd6d814fbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:01:44.031 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:44.013+0000 7fd6e0444700 1 -- 172.21.15.195:0/2556450027 shutdown_connections 2022-06-27T12:01:44.031 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:44.013+0000 7fd6e0444700 1 -- 172.21.15.195:0/2556450027 wait complete. 2022-06-27T12:01:44.031 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 50 2022-06-27T12:01:44.037 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 50, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-06-27T11:47:52.358095+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 50, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5273": { "addr": "172.21.15.202:6835/417105338", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6834", "nonce": 417105338, "type": "v2" }, { "addr": "172.21.15.202:6835", "nonce": 417105338, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5273, "incarnation": 49, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:reconnect", "state_seq": 36 } }, "last_failure": 0, "last_failure_osd_epoch": 55, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-06-27T12:01:43.204516+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5273 } } } ], "standbys": [ { "addr": "172.21.15.202:6837/3305006841", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6836", "nonce": 3305006841, "type": "v2" }, { "addr": "172.21.15.202:6837", "nonce": 3305006841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6835/2060709822", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6834", "nonce": 2060709822, "type": "v2" }, { "addr": "172.21.15.195:6835", "nonce": 2060709822, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6838/1598751652", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6836", "nonce": 1598751652, "type": "v2" }, { "addr": "172.21.15.195:6838", "nonce": 1598751652, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4275, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6839/1313800661", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6837", "nonce": 1313800661, "type": "v2" }, { "addr": "172.21.15.195:6839", "nonce": 1313800661, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4278, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-06-27T12:01:46.039 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:01:46.173 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.173+0000 7f306af5c700 1 Processor -- start 2022-06-27T12:01:46.174 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.173+0000 7f306af5c700 1 -- start start 2022-06-27T12:01:46.175 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.174+0000 7f306af5c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f306406fef0 0x7f30641570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:46.175 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.174+0000 7f306af5c700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f30640c34a0 0x7f30640c2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:46.176 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.174+0000 7f306af5c700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f306406fba0 con 0x7f306406e850 2022-06-27T12:01:46.176 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.174+0000 7f306af5c700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30641575e0 con 0x7f306406fef0 2022-06-27T12:01:46.176 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.174+0000 7f306af5c700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f306406f5e0 con 0x7f30640c34a0 2022-06-27T12:01:46.176 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.174+0000 7f3063fff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f306406fef0 0x7f30641570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:46.177 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.174+0000 7f3063fff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f306406fef0 0x7f30641570a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:46692/0 (socket says 172.21.15.195:46692) 2022-06-27T12:01:46.177 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.174+0000 7f3063fff700 1 -- 172.21.15.195:0/496149340 learned_addr learned my addr 172.21.15.195:0/496149340 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:01:46.177 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.174+0000 7f30694f9700 1 --2- 172.21.15.195:0/496149340 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f30640c34a0 0x7f30640c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:46.184 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.174+0000 7f3063fff700 1 -- 172.21.15.195:0/496149340 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f30640c34a0 msgr2=0x7f30640c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:01:46.185 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.174+0000 7f3063fff700 1 --2- 172.21.15.195:0/496149340 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f30640c34a0 0x7f30640c2920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:46.185 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.174+0000 7f3063fff700 1 -- 172.21.15.195:0/496149340 >> v1:172.21.15.202:6789/0 conn(0x7f306406e850 legacy=0x7f3064069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:46.185 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.174+0000 7f3063fff700 1 -- 172.21.15.195:0/496149340 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f306406f8c0 con 0x7f306406fef0 2022-06-27T12:01:46.186 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.174+0000 7f3063fff700 1 --2- 172.21.15.195:0/496149340 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f306406fef0 0x7f30641570a0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f3050009860 tx=0x7f3050009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3e2050f07eed8a1 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:01:46.186 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.175+0000 7f30637fe700 1 -- 172.21.15.195:0/496149340 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f305000b710 con 0x7f306406fef0 2022-06-27T12:01:46.186 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.175+0000 7f30637fe700 1 -- 172.21.15.195:0/496149340 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f305000b870 con 0x7f306406fef0 2022-06-27T12:01:46.187 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.175+0000 7f30637fe700 1 -- 172.21.15.195:0/496149340 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f305000bc30 con 0x7f306406fef0 2022-06-27T12:01:46.187 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.175+0000 7f306af5c700 1 -- 172.21.15.195:0/496149340 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f306406fef0 msgr2=0x7f30641570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:46.187 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.175+0000 7f306af5c700 1 --2- 172.21.15.195:0/496149340 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f306406fef0 0x7f30641570a0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f3050009860 tx=0x7f3050009120 comp rx=0 tx=0).stop 2022-06-27T12:01:46.188 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.175+0000 7f306af5c700 1 -- 172.21.15.195:0/496149340 shutdown_connections 2022-06-27T12:01:46.188 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.175+0000 7f306af5c700 1 --2- 172.21.15.195:0/496149340 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f30640c34a0 0x7f30640c2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:46.188 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.175+0000 7f306af5c700 1 --2- 172.21.15.195:0/496149340 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f306406fef0 0x7f30641570a0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:46.189 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.175+0000 7f306af5c700 1 -- 172.21.15.195:0/496149340 >> 172.21.15.195:0/496149340 conn(0x7f30640c6350 msgr2=0x7f30640c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:01:46.189 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.175+0000 7f306af5c700 1 -- 172.21.15.195:0/496149340 shutdown_connections 2022-06-27T12:01:46.189 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.175+0000 7f306af5c700 1 -- 172.21.15.195:0/496149340 wait complete. 2022-06-27T12:01:46.190 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.177+0000 7f306af5c700 1 Processor -- start 2022-06-27T12:01:46.190 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.177+0000 7f306af5c700 1 -- start start 2022-06-27T12:01:46.190 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.178+0000 7f306af5c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f306406e850 0x7f30640c2c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:46.191 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.178+0000 7f306af5c700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f30640c34a0 0x7f306414b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:46.192 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.178+0000 7f306af5c700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f306414bc70 0x7f30641533c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:46.192 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.178+0000 7f306af5c700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f306414f380 con 0x7f306406e850 2022-06-27T12:01:46.192 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.178+0000 7f306af5c700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30640c3140 con 0x7f306414bc70 2022-06-27T12:01:46.192 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.178+0000 7f306af5c700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f306414ad20 con 0x7f30640c34a0 2022-06-27T12:01:46.193 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.178+0000 7f3068cf8700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f306406e850 0x7f30640c2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:46.193 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.178+0000 7f3068cf8700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f306406e850 0x7f30640c2c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:46696/0 (socket says 172.21.15.195:46696) 2022-06-27T12:01:46.193 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.178+0000 7f3068cf8700 1 -- 172.21.15.195:0/2950473212 learned_addr learned my addr 172.21.15.195:0/2950473212 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:01:46.193 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.178+0000 7f30694f9700 1 --2- 172.21.15.195:0/2950473212 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f306414bc70 0x7f30641533c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:46.194 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.178+0000 7f3063fff700 1 --2- 172.21.15.195:0/2950473212 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f30640c34a0 0x7f306414b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:46.194 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.178+0000 7f3068cf8700 1 -- 172.21.15.195:0/2950473212 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f30640c34a0 msgr2=0x7f306414b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:46.194 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.178+0000 7f3068cf8700 1 --2- 172.21.15.195:0/2950473212 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f30640c34a0 0x7f306414b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:46.194 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.178+0000 7f3068cf8700 1 -- 172.21.15.195:0/2950473212 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f306414bc70 msgr2=0x7f30641533c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:46.194 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.178+0000 7f3068cf8700 1 --2- 172.21.15.195:0/2950473212 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f306414bc70 0x7f30641533c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:46.195 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.178+0000 7f3068cf8700 1 -- 172.21.15.195:0/2950473212 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30500093a0 con 0x7f306406e850 2022-06-27T12:01:46.195 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.178+0000 7f3068cf8700 1 --2- 172.21.15.195:0/2950473212 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f306406e850 0x7f30640c2c00 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f305800c410 tx=0x7f3058007e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:01:46.195 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.178+0000 7f3060ff9700 1 -- 172.21.15.195:0/2950473212 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30580050d0 con 0x7f306406e850 2022-06-27T12:01:46.196 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.178+0000 7f3060ff9700 1 -- 172.21.15.195:0/2950473212 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3058005250 con 0x7f306406e850 2022-06-27T12:01:46.196 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.178+0000 7f306af5c700 1 -- 172.21.15.195:0/2950473212 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f306414b2a0 con 0x7f306406e850 2022-06-27T12:01:46.196 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.178+0000 7f3060ff9700 1 -- 172.21.15.195:0/2950473212 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30580117e0 con 0x7f306406e850 2022-06-27T12:01:46.196 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.178+0000 7f306af5c700 1 -- 172.21.15.195:0/2950473212 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f306414a0c0 con 0x7f306406e850 2022-06-27T12:01:46.197 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.179+0000 7f306af5c700 1 -- 172.21.15.195:0/2950473212 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30640722c0 con 0x7f306406e850 2022-06-27T12:01:46.197 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.180+0000 7f3060ff9700 1 -- 172.21.15.195:0/2950473212 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f3058019030 con 0x7f306406e850 2022-06-27T12:01:46.197 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.180+0000 7f3060ff9700 1 --2- 172.21.15.195:0/2950473212 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f304c063d00 0x7f304c0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:46.198 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.180+0000 7f3060ff9700 1 -- 172.21.15.195:0/2950473212 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6804+0+0 (secure 0 0 0) 0x7f305807fc20 con 0x7f306406e850 2022-06-27T12:01:46.198 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.180+0000 7f3063fff700 1 --2- 172.21.15.195:0/2950473212 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f304c063d00 0x7f304c0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:46.198 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.181+0000 7f3063fff700 1 --2- 172.21.15.195:0/2950473212 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f304c063d00 0x7f304c0661b0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f3050009160 tx=0x7f3050017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:01:46.198 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.184+0000 7f3060ff9700 1 -- 172.21.15.195:0/2950473212 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f305804e520 con 0x7f306406e850 2022-06-27T12:01:46.423 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.415+0000 7f306af5c700 1 -- 172.21.15.195:0/2950473212 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f306414b580 con 0x7f306406e850 2022-06-27T12:01:46.423 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.416+0000 7f3060ff9700 1 -- 172.21.15.195:0/2950473212 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+5368 (secure 0 0 0) 0x7f3058017020 con 0x7f306406e850 2022-06-27T12:01:46.424 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:01:46.425 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:01:43.204516+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5273":{"gid":5273,"name":"d","rank":0,"incarnation":49,"state":"up:reconnect","state_seq":36,"addr":"172.21.15.202:6835/417105338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":417105338},{"type":"v1","addr":"172.21.15.202:6835","nonce":417105338}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:01:46.445 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.418+0000 7f306af5c700 1 -- 172.21.15.195:0/2950473212 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f304c063d00 msgr2=0x7f304c0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:46.446 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.418+0000 7f306af5c700 1 --2- 172.21.15.195:0/2950473212 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f304c063d00 0x7f304c0661b0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f3050009160 tx=0x7f3050017000 comp rx=0 tx=0).stop 2022-06-27T12:01:46.449 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.418+0000 7f306af5c700 1 -- 172.21.15.195:0/2950473212 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f306406e850 msgr2=0x7f30640c2c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:46.454 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.418+0000 7f306af5c700 1 --2- 172.21.15.195:0/2950473212 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f306406e850 0x7f30640c2c00 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f305800c410 tx=0x7f3058007e40 comp rx=0 tx=0).stop 2022-06-27T12:01:46.455 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.418+0000 7f306af5c700 1 -- 172.21.15.195:0/2950473212 shutdown_connections 2022-06-27T12:01:46.456 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.418+0000 7f306af5c700 1 --2- 172.21.15.195:0/2950473212 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f304c063d00 0x7f304c0661b0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:46.456 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.418+0000 7f306af5c700 1 --2- 172.21.15.195:0/2950473212 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f306406e850 0x7f30640c2c00 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:46.456 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.418+0000 7f306af5c700 1 --2- 172.21.15.195:0/2950473212 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f306414bc70 0x7f30641533c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:46.457 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.418+0000 7f306af5c700 1 --2- 172.21.15.195:0/2950473212 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f30640c34a0 0x7f306414b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:46.457 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.418+0000 7f306af5c700 1 -- 172.21.15.195:0/2950473212 >> 172.21.15.195:0/2950473212 conn(0x7f30640c6350 msgr2=0x7f30640c8840 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:01:46.458 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.418+0000 7f306af5c700 1 -- 172.21.15.195:0/2950473212 shutdown_connections 2022-06-27T12:01:46.458 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:46.418+0000 7f306af5c700 1 -- 172.21.15.195:0/2950473212 wait complete. 2022-06-27T12:01:46.459 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 50 2022-06-27T12:01:46.461 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:01:48.462 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:01:48.635 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.598+0000 7f19b5b9f700 1 Processor -- start 2022-06-27T12:01:48.635 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.598+0000 7f19b5b9f700 1 -- start start 2022-06-27T12:01:48.636 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.599+0000 7f19b5b9f700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f19b00c3030 0x7f19b01570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:48.636 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.599+0000 7f19b5b9f700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19b006fba0 con 0x7f19b006e850 2022-06-27T12:01:48.636 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.599+0000 7f19b5b9f700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19b006f5e0 con 0x7f19b006fef0 2022-06-27T12:01:48.636 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.599+0000 7f19b5b9f700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19b01575e0 con 0x7f19b00c3030 2022-06-27T12:01:48.637 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.599+0000 7f19af7fe700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f19b006e850 0x7f19b0069de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:52892/0 (socket says 172.21.15.195:52892) 2022-06-27T12:01:48.637 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.599+0000 7f19af7fe700 1 -- 172.21.15.195:0/3574456842 learned_addr learned my addr 172.21.15.195:0/3574456842 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:01:48.637 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.599+0000 7f19ae7fc700 1 -- 172.21.15.195:0/3574456842 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 73579972 0 0) 0x7f19b006fba0 con 0x7f19b006e850 2022-06-27T12:01:48.637 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.599+0000 7f19affff700 1 --2- 172.21.15.195:0/3574456842 >> v2:172.21.15.202:3300/0 conn(0x7f19b00c3030 0x7f19b01570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:48.638 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.599+0000 7f19ae7fc700 1 -- 172.21.15.195:0/3574456842 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1990003430 con 0x7f19b006e850 2022-06-27T12:01:48.638 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.599+0000 7f19ae7fc700 1 -- 172.21.15.195:0/3574456842 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3954513797 0 0) 0x7f1990003430 con 0x7f19b006e850 2022-06-27T12:01:48.638 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.599+0000 7f19ae7fc700 1 -- 172.21.15.195:0/3574456842 >> v2:172.21.15.202:3300/0 conn(0x7f19b00c3030 msgr2=0x7f19b01570a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:01:48.639 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.599+0000 7f19ae7fc700 1 --2- 172.21.15.195:0/3574456842 >> v2:172.21.15.202:3300/0 conn(0x7f19b00c3030 0x7f19b01570a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:48.639 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.599+0000 7f19ae7fc700 1 -- 172.21.15.195:0/3574456842 >> v1:172.21.15.202:6789/0 conn(0x7f19b006fef0 legacy=0x7f19b00c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:48.639 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.599+0000 7f19ae7fc700 1 -- 172.21.15.195:0/3574456842 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19b006f8c0 con 0x7f19b006e850 2022-06-27T12:01:48.639 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.600+0000 7f19ae7fc700 1 -- 172.21.15.195:0/3574456842 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7f1998001470 con 0x7f19b006e850 2022-06-27T12:01:48.640 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.600+0000 7f19ae7fc700 1 -- 172.21.15.195:0/3574456842 >> v1:172.21.15.195:6789/0 conn(0x7f19b006e850 legacy=0x7f19b0069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:48.640 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.600+0000 7f19ae7fc700 1 --2- 172.21.15.195:0/3574456842 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1990003920 0x7f1990003cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:48.640 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.600+0000 7f19ae7fc700 1 --2- 172.21.15.195:0/3574456842 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f19900046d0 0x7f1990004b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:48.640 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.600+0000 7f19ae7fc700 1 --2- 172.21.15.195:0/3574456842 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1990005080 0x7f199000b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:48.641 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.600+0000 7f19ae7fc700 1 -- 172.21.15.195:0/3574456842 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19b01575e0 con 0x7f1990005080 2022-06-27T12:01:48.641 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.600+0000 7f19ae7fc700 1 -- 172.21.15.195:0/3574456842 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f199000bf10 con 0x7f1990003920 2022-06-27T12:01:48.641 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.600+0000 7f19ae7fc700 1 -- 172.21.15.195:0/3574456842 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f199000c0d0 con 0x7f19900046d0 2022-06-27T12:01:48.641 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.600+0000 7f19affff700 1 --2- 172.21.15.195:0/3574456842 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1990005080 0x7f199000b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:48.642 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.600+0000 7f19af7fe700 1 --2- 172.21.15.195:0/3574456842 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1990003920 0x7f1990003cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:48.642 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.600+0000 7f19aeffd700 1 --2- 172.21.15.195:0/3574456842 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f19900046d0 0x7f1990004b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:48.642 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.600+0000 7f19affff700 1 -- 172.21.15.195:0/3574456842 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f19900046d0 msgr2=0x7f1990004b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:01:48.642 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.600+0000 7f19affff700 1 --2- 172.21.15.195:0/3574456842 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f19900046d0 0x7f1990004b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:48.643 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.600+0000 7f19affff700 1 -- 172.21.15.195:0/3574456842 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1990003920 msgr2=0x7f1990003cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:01:48.643 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.600+0000 7f19affff700 1 --2- 172.21.15.195:0/3574456842 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1990003920 0x7f1990003cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:48.643 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.600+0000 7f19affff700 1 -- 172.21.15.195:0/3574456842 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1990003430 con 0x7f1990005080 2022-06-27T12:01:48.643 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.600+0000 7f19affff700 1 -- 172.21.15.195:0/3574456842 reap_dead start 2022-06-27T12:01:48.643 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.600+0000 7f19affff700 1 --2- 172.21.15.195:0/3574456842 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1990005080 0x7f199000b920 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f19b00c3450 tx=0x7f19a800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90778e0970404afa server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:01:48.644 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.600+0000 7f19ae7fc700 1 -- 172.21.15.195:0/3574456842 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19a800ea90 con 0x7f1990005080 2022-06-27T12:01:48.644 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.600+0000 7f19ae7fc700 1 -- 172.21.15.195:0/3574456842 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19a800f040 con 0x7f1990005080 2022-06-27T12:01:48.644 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.600+0000 7f19ae7fc700 1 -- 172.21.15.195:0/3574456842 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19a800ee40 con 0x7f1990005080 2022-06-27T12:01:48.645 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.601+0000 7f19b5b9f700 1 -- 172.21.15.195:0/3574456842 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1990005080 msgr2=0x7f199000b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:48.645 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.601+0000 7f19b5b9f700 1 --2- 172.21.15.195:0/3574456842 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1990005080 0x7f199000b920 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f19b00c3450 tx=0x7f19a800b040 comp rx=0 tx=0).stop 2022-06-27T12:01:48.645 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.601+0000 7f19b5b9f700 1 -- 172.21.15.195:0/3574456842 shutdown_connections 2022-06-27T12:01:48.645 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.601+0000 7f19b5b9f700 1 --2- 172.21.15.195:0/3574456842 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1990005080 0x7f199000b920 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:48.646 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.601+0000 7f19b5b9f700 1 -- 172.21.15.195:0/3574456842 >> 172.21.15.195:0/3574456842 conn(0x7f19b00c6350 msgr2=0x7f19b00c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:01:48.646 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.601+0000 7f19b5b9f700 1 -- 172.21.15.195:0/3574456842 shutdown_connections 2022-06-27T12:01:48.646 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.601+0000 7f19b5b9f700 1 -- 172.21.15.195:0/3574456842 wait complete. 2022-06-27T12:01:48.646 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.602+0000 7f19b5b9f700 1 Processor -- start 2022-06-27T12:01:48.646 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.602+0000 7f19b5b9f700 1 -- start start 2022-06-27T12:01:48.647 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.603+0000 7f19b5b9f700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1990005080 0x7f19b014e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:48.647 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.603+0000 7f19b5b9f700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f19b01542e0 0x7f19b014f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:48.647 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.603+0000 7f19b5b9f700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f19b014f570 0x7f19b00bb640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:48.647 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.603+0000 7f19b5b9f700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19b00bb1b0 con 0x7f1990005080 2022-06-27T12:01:48.648 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.603+0000 7f19b5b9f700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19b0166b30 con 0x7f19b01542e0 2022-06-27T12:01:48.648 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.603+0000 7f19b5b9f700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19b0166df0 con 0x7f19b014f570 2022-06-27T12:01:48.648 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.603+0000 7f19af7fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1990005080 0x7f19b014e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:48.648 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.603+0000 7f19af7fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1990005080 0x7f19b014e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:46716/0 (socket says 172.21.15.195:46716) 2022-06-27T12:01:48.649 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.603+0000 7f19af7fe700 1 -- 172.21.15.195:0/462992664 learned_addr learned my addr 172.21.15.195:0/462992664 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:01:48.649 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.603+0000 7f19affff700 1 --2- 172.21.15.195:0/462992664 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f19b014f570 0x7f19b00bb640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:48.649 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.603+0000 7f19aeffd700 1 --2- 172.21.15.195:0/462992664 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f19b01542e0 0x7f19b014f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:48.649 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.603+0000 7f19af7fe700 1 -- 172.21.15.195:0/462992664 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f19b014f570 msgr2=0x7f19b00bb640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:48.650 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.603+0000 7f19af7fe700 1 --2- 172.21.15.195:0/462992664 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f19b014f570 0x7f19b00bb640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:48.650 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.603+0000 7f19af7fe700 1 -- 172.21.15.195:0/462992664 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f19b01542e0 msgr2=0x7f19b014f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:48.650 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.603+0000 7f19af7fe700 1 --2- 172.21.15.195:0/462992664 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f19b01542e0 0x7f19b014f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:48.650 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.603+0000 7f19af7fe700 1 -- 172.21.15.195:0/462992664 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19b00ba810 con 0x7f1990005080 2022-06-27T12:01:48.651 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.603+0000 7f19aeffd700 1 --2- 172.21.15.195:0/462992664 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f19b01542e0 0x7f19b014f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:01:48.651 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.604+0000 7f19af7fe700 1 --2- 172.21.15.195:0/462992664 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1990005080 0x7f19b014e5c0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f1998004a20 tx=0x7f1998001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:01:48.652 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.604+0000 7f19a77fe700 1 -- 172.21.15.195:0/462992664 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1998018030 con 0x7f1990005080 2022-06-27T12:01:48.652 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.604+0000 7f19a77fe700 1 -- 172.21.15.195:0/462992664 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1998004e90 con 0x7f1990005080 2022-06-27T12:01:48.652 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.604+0000 7f19b5b9f700 1 -- 172.21.15.195:0/462992664 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19b00baaf0 con 0x7f1990005080 2022-06-27T12:01:48.652 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.604+0000 7f19a77fe700 1 -- 172.21.15.195:0/462992664 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1998000ce0 con 0x7f1990005080 2022-06-27T12:01:48.653 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.604+0000 7f19b5b9f700 1 -- 172.21.15.195:0/462992664 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19b006acc0 con 0x7f1990005080 2022-06-27T12:01:48.653 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.605+0000 7f19b5b9f700 1 -- 172.21.15.195:0/462992664 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19b014f940 con 0x7f1990005080 2022-06-27T12:01:48.653 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.605+0000 7f19a77fe700 1 -- 172.21.15.195:0/462992664 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f199801e470 con 0x7f1990005080 2022-06-27T12:01:48.653 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.605+0000 7f19a77fe700 1 --2- 172.21.15.195:0/462992664 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1994063d30 0x7f19940661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:48.654 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.605+0000 7f19a77fe700 1 -- 172.21.15.195:0/462992664 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6804+0+0 (secure 0 0 0) 0x7f1998019070 con 0x7f1990005080 2022-06-27T12:01:48.654 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.606+0000 7f19aeffd700 1 --2- 172.21.15.195:0/462992664 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1994063d30 0x7f19940661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:48.654 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.606+0000 7f19aeffd700 1 --2- 172.21.15.195:0/462992664 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1994063d30 0x7f19940661e0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f19a00054f0 tx=0x7f19a0009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:01:48.654 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.610+0000 7f19a77fe700 1 -- 172.21.15.195:0/462992664 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f199804f640 con 0x7f1990005080 2022-06-27T12:01:48.830 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.830+0000 7f19b5b9f700 1 -- 172.21.15.195:0/462992664 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f19b00bad30 con 0x7f1990005080 2022-06-27T12:01:48.831 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.831+0000 7f19a77fe700 1 -- 172.21.15.195:0/462992664 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+5365 (secure 0 0 0) 0x7f19b00bad30 con 0x7f1990005080 2022-06-27T12:01:48.832 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:01:48.833 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:01:48.219526+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5273":{"gid":5273,"name":"d","rank":0,"incarnation":49,"state":"up:rejoin","state_seq":38,"addr":"172.21.15.202:6835/417105338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":417105338},{"type":"v1","addr":"172.21.15.202:6835","nonce":417105338}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:01:48.841 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.833+0000 7f19b5b9f700 1 -- 172.21.15.195:0/462992664 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1994063d30 msgr2=0x7f19940661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:48.842 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.833+0000 7f19b5b9f700 1 --2- 172.21.15.195:0/462992664 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1994063d30 0x7f19940661e0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f19a00054f0 tx=0x7f19a0009040 comp rx=0 tx=0).stop 2022-06-27T12:01:48.842 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.833+0000 7f19b5b9f700 1 -- 172.21.15.195:0/462992664 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1990005080 msgr2=0x7f19b014e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:48.843 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.833+0000 7f19b5b9f700 1 --2- 172.21.15.195:0/462992664 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1990005080 0x7f19b014e5c0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f1998004a20 tx=0x7f1998001e50 comp rx=0 tx=0).stop 2022-06-27T12:01:48.843 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.833+0000 7f19b5b9f700 1 -- 172.21.15.195:0/462992664 shutdown_connections 2022-06-27T12:01:48.844 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.833+0000 7f19b5b9f700 1 --2- 172.21.15.195:0/462992664 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1994063d30 0x7f19940661e0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:48.844 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.833+0000 7f19b5b9f700 1 --2- 172.21.15.195:0/462992664 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1990005080 0x7f19b014e5c0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:48.844 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.833+0000 7f19b5b9f700 1 --2- 172.21.15.195:0/462992664 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f19b014f570 0x7f19b00bb640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:48.845 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.833+0000 7f19b5b9f700 1 --2- 172.21.15.195:0/462992664 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f19b01542e0 0x7f19b014f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:48.845 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.833+0000 7f19b5b9f700 1 -- 172.21.15.195:0/462992664 >> 172.21.15.195:0/462992664 conn(0x7f19b00c6350 msgr2=0x7f19b0068830 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:01:48.846 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.833+0000 7f19b5b9f700 1 -- 172.21.15.195:0/462992664 shutdown_connections 2022-06-27T12:01:48.846 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:48.833+0000 7f19b5b9f700 1 -- 172.21.15.195:0/462992664 wait complete. 2022-06-27T12:01:48.847 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 51 2022-06-27T12:01:48.849 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:01:50.852 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:01:50.998 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:50.987+0000 7f6001688700 1 Processor -- start 2022-06-27T12:01:50.998 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:50.987+0000 7f6001688700 1 -- start start 2022-06-27T12:01:50.999 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:50.988+0000 7f6001688700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f5ffc068f80 0x7f5ffc0c4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:50.999 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:50.988+0000 7f6001688700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5ffc0c7480 0x7f5ffc0683c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:51.000 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:50.988+0000 7f6001688700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ffc1573b0 con 0x7f5ffc069350 2022-06-27T12:01:51.000 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:50.988+0000 7f6001688700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ffc0c46c0 con 0x7f5ffc068f80 2022-06-27T12:01:51.001 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:50.988+0000 7f6001688700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ffc0697d0 con 0x7f5ffc0c7480 2022-06-27T12:01:51.001 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:50.988+0000 7f5ffaffd700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f5ffc068f80 0x7f5ffc0c4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:51.001 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:50.988+0000 7f5ffa7fc700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5ffc0c7480 0x7f5ffc0683c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:51.003 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:50.988+0000 7f5ffb7fe700 1 --1- >> v1:172.21.15.202:6789/0 conn(0x7f5ffc069350 0x7f5ffc156460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.202:6789/0 says I am v1:172.21.15.195:54780/0 (socket says 172.21.15.195:54780) 2022-06-27T12:01:51.003 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:50.988+0000 7f5ffa7fc700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5ffc0c7480 0x7f5ffc0683c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3301/0 says I am v2:172.21.15.195:39486/0 (socket says 172.21.15.195:39486) 2022-06-27T12:01:51.004 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:50.988+0000 7f5ffb7fe700 1 -- 172.21.15.195:0/845845140 learned_addr learned my addr 172.21.15.195:0/845845140 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:01:51.020 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:50.988+0000 7f5ffa7fc700 1 -- 172.21.15.195:0/845845140 >> v2:172.21.15.202:3300/0 conn(0x7f5ffc068f80 msgr2=0x7f5ffc0c4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:01:51.020 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:50.988+0000 7f5ffa7fc700 1 --2- 172.21.15.195:0/845845140 >> v2:172.21.15.202:3300/0 conn(0x7f5ffc068f80 0x7f5ffc0c4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:51.020 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:50.988+0000 7f5ffa7fc700 1 -- 172.21.15.195:0/845845140 >> v1:172.21.15.202:6789/0 conn(0x7f5ffc069350 legacy=0x7f5ffc156460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:51.021 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:50.988+0000 7f5ffa7fc700 1 -- 172.21.15.195:0/845845140 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ffc156b70 con 0x7f5ffc0c7480 2022-06-27T12:01:51.021 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:50.988+0000 7f5ffaffd700 1 --2- 172.21.15.195:0/845845140 >> v2:172.21.15.202:3300/0 conn(0x7f5ffc068f80 0x7f5ffc0c4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:01:51.022 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:50.988+0000 7f5ffa7fc700 1 --2- 172.21.15.195:0/845845140 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5ffc0c7480 0x7f5ffc0683c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5ff00097a0 tx=0x7f5ff0009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53d337f9d0b478d9 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:01:51.022 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:50.989+0000 7f5ff9ffb700 1 -- 172.21.15.195:0/845845140 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ff0011630 con 0x7f5ffc0c7480 2022-06-27T12:01:51.022 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:50.989+0000 7f5ff9ffb700 1 -- 172.21.15.195:0/845845140 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ff0011790 con 0x7f5ffc0c7480 2022-06-27T12:01:51.023 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:50.989+0000 7f5ff9ffb700 1 -- 172.21.15.195:0/845845140 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ff0011b30 con 0x7f5ffc0c7480 2022-06-27T12:01:51.023 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:50.989+0000 7f6001688700 1 -- 172.21.15.195:0/845845140 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5ffc0c7480 msgr2=0x7f5ffc0683c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:51.023 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:50.989+0000 7f6001688700 1 --2- 172.21.15.195:0/845845140 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5ffc0c7480 0x7f5ffc0683c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5ff00097a0 tx=0x7f5ff0009060 comp rx=0 tx=0).stop 2022-06-27T12:01:51.023 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:50.989+0000 7f6001688700 1 -- 172.21.15.195:0/845845140 shutdown_connections 2022-06-27T12:01:51.024 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:50.989+0000 7f6001688700 1 --2- 172.21.15.195:0/845845140 >> v2:172.21.15.202:3300/0 conn(0x7f5ffc068f80 0x7f5ffc0c4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:51.024 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:50.989+0000 7f6001688700 1 --2- 172.21.15.195:0/845845140 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5ffc0c7480 0x7f5ffc0683c0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:51.024 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:50.989+0000 7f6001688700 1 -- 172.21.15.195:0/845845140 >> 172.21.15.195:0/845845140 conn(0x7f5ffc0ccbd0 msgr2=0x7f5ffc0ccfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:01:51.024 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:50.989+0000 7f6001688700 1 -- 172.21.15.195:0/845845140 shutdown_connections 2022-06-27T12:01:51.025 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:50.989+0000 7f6001688700 1 -- 172.21.15.195:0/845845140 wait complete. 2022-06-27T12:01:51.025 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:50.991+0000 7f6001688700 1 Processor -- start 2022-06-27T12:01:51.025 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:50.991+0000 7f6001688700 1 -- start start 2022-06-27T12:01:51.026 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:50.991+0000 7f6001688700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5ffc068f80 0x7f5ffc155e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:51.026 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:50.991+0000 7f6001688700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5ffc069350 0x7f5ffc167e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:51.027 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:50.991+0000 7f6001688700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5ffc0c7480 0x7f5ffc168350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:51.027 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:50.991+0000 7f6001688700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ffc1675c0 con 0x7f5ffc068f80 2022-06-27T12:01:51.028 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:50.991+0000 7f6001688700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ffc15b7a0 con 0x7f5ffc0c7480 2022-06-27T12:01:51.028 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:50.991+0000 7f6001688700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ffc166c90 con 0x7f5ffc069350 2022-06-27T12:01:51.028 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:50.991+0000 7f5ffaffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5ffc068f80 0x7f5ffc155e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:51.028 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:50.991+0000 7f5ffaffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5ffc068f80 0x7f5ffc155e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:46734/0 (socket says 172.21.15.195:46734) 2022-06-27T12:01:51.028 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:50.991+0000 7f5ffaffd700 1 -- 172.21.15.195:0/1549204113 learned_addr learned my addr 172.21.15.195:0/1549204113 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:01:51.029 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:50.992+0000 7f5ffa7fc700 1 --2- 172.21.15.195:0/1549204113 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5ffc069350 0x7f5ffc167e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:51.029 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:50.992+0000 7f5ffb7fe700 1 --2- 172.21.15.195:0/1549204113 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5ffc0c7480 0x7f5ffc168350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:51.029 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:50.992+0000 7f5ffaffd700 1 -- 172.21.15.195:0/1549204113 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5ffc069350 msgr2=0x7f5ffc167e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:51.030 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:50.992+0000 7f5ffaffd700 1 --2- 172.21.15.195:0/1549204113 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5ffc069350 0x7f5ffc167e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:51.030 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:50.992+0000 7f5ffaffd700 1 -- 172.21.15.195:0/1549204113 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5ffc0c7480 msgr2=0x7f5ffc168350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:51.031 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:50.992+0000 7f5ffaffd700 1 --2- 172.21.15.195:0/1549204113 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5ffc0c7480 0x7f5ffc168350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:51.031 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:50.992+0000 7f5ffaffd700 1 -- 172.21.15.195:0/1549204113 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fec0092e0 con 0x7f5ffc068f80 2022-06-27T12:01:51.031 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:50.992+0000 7f5ffaffd700 1 --2- 172.21.15.195:0/1549204113 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5ffc068f80 0x7f5ffc155e60 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f5fec00ea50 tx=0x7f5fec009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:01:51.031 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:50.992+0000 7f5fe2ffd700 1 -- 172.21.15.195:0/1549204113 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fec00ee90 con 0x7f5ffc068f80 2022-06-27T12:01:51.032 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:50.992+0000 7f5fe2ffd700 1 -- 172.21.15.195:0/1549204113 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fec00c850 con 0x7f5ffc068f80 2022-06-27T12:01:51.032 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:50.992+0000 7f6001688700 1 -- 172.21.15.195:0/1549204113 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ff00092e0 con 0x7f5ffc068f80 2022-06-27T12:01:51.032 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:50.992+0000 7f5fe2ffd700 1 -- 172.21.15.195:0/1549204113 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fec00cbd0 con 0x7f5ffc068f80 2022-06-27T12:01:51.033 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:50.992+0000 7f6001688700 1 -- 172.21.15.195:0/1549204113 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ffc167250 con 0x7f5ffc068f80 2022-06-27T12:01:51.033 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:50.993+0000 7f6001688700 1 -- 172.21.15.195:0/1549204113 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ffc0722c0 con 0x7f5ffc068f80 2022-06-27T12:01:51.033 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:50.994+0000 7f5fe2ffd700 1 -- 172.21.15.195:0/1549204113 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f5fec019030 con 0x7f5ffc068f80 2022-06-27T12:01:51.033 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:50.994+0000 7f5fe2ffd700 1 --2- 172.21.15.195:0/1549204113 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5fdc063d00 0x7f5fdc0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:51.034 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:50.994+0000 7f5fe2ffd700 1 -- 172.21.15.195:0/1549204113 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6804+0+0 (secure 0 0 0) 0x7f5fec080b70 con 0x7f5ffc068f80 2022-06-27T12:01:51.034 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:50.994+0000 7f5ffa7fc700 1 --2- 172.21.15.195:0/1549204113 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5fdc063d00 0x7f5fdc0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:51.034 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:50.995+0000 7f5ffa7fc700 1 --2- 172.21.15.195:0/1549204113 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5fdc063d00 0x7f5fdc0661b0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f5ff0009260 tx=0x7f5ff0017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:01:51.035 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:50.998+0000 7f5fe2ffd700 1 -- 172.21.15.195:0/1549204113 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f5fec04f470 con 0x7f5ffc068f80 2022-06-27T12:01:51.231 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:51.226+0000 7f6001688700 1 -- 172.21.15.195:0/1549204113 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ffc06ac60 con 0x7f5ffc068f80 2022-06-27T12:01:51.231 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:51.226+0000 7f5fe2ffd700 1 -- 172.21.15.195:0/1549204113 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+5365 (secure 0 0 0) 0x7f5fec021090 con 0x7f5ffc068f80 2022-06-27T12:01:51.232 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:01:51.232 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:01:48.219526+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5273":{"gid":5273,"name":"d","rank":0,"incarnation":49,"state":"up:rejoin","state_seq":38,"addr":"172.21.15.202:6835/417105338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":417105338},{"type":"v1","addr":"172.21.15.202:6835","nonce":417105338}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:01:51.235 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:51.228+0000 7f6001688700 1 -- 172.21.15.195:0/1549204113 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5fdc063d00 msgr2=0x7f5fdc0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:51.235 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:51.228+0000 7f6001688700 1 --2- 172.21.15.195:0/1549204113 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5fdc063d00 0x7f5fdc0661b0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f5ff0009260 tx=0x7f5ff0017000 comp rx=0 tx=0).stop 2022-06-27T12:01:51.236 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:51.228+0000 7f6001688700 1 -- 172.21.15.195:0/1549204113 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5ffc068f80 msgr2=0x7f5ffc155e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:51.236 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:51.228+0000 7f6001688700 1 --2- 172.21.15.195:0/1549204113 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5ffc068f80 0x7f5ffc155e60 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f5fec00ea50 tx=0x7f5fec009060 comp rx=0 tx=0).stop 2022-06-27T12:01:51.236 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:51.228+0000 7f6001688700 1 -- 172.21.15.195:0/1549204113 shutdown_connections 2022-06-27T12:01:51.238 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:51.228+0000 7f6001688700 1 --2- 172.21.15.195:0/1549204113 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5fdc063d00 0x7f5fdc0661b0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:51.238 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:51.228+0000 7f6001688700 1 --2- 172.21.15.195:0/1549204113 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5ffc068f80 0x7f5ffc155e60 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:51.238 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:51.228+0000 7f6001688700 1 --2- 172.21.15.195:0/1549204113 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5ffc0c7480 0x7f5ffc168350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:51.239 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:51.228+0000 7f6001688700 1 --2- 172.21.15.195:0/1549204113 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5ffc069350 0x7f5ffc167e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:51.239 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:51.228+0000 7f6001688700 1 -- 172.21.15.195:0/1549204113 >> 172.21.15.195:0/1549204113 conn(0x7f5ffc0ccbd0 msgr2=0x7f5ffc1542a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:01:51.239 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:51.228+0000 7f6001688700 1 -- 172.21.15.195:0/1549204113 shutdown_connections 2022-06-27T12:01:51.240 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:51.228+0000 7f6001688700 1 -- 172.21.15.195:0/1549204113 wait complete. 2022-06-27T12:01:51.240 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 51 2022-06-27T12:01:51.243 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:01:53.261 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:01:53.420 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.401+0000 7f0a3ac56700 1 Processor -- start 2022-06-27T12:01:53.421 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.401+0000 7f0a3ac56700 1 -- start start 2022-06-27T12:01:53.421 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.402+0000 7f0a3ac56700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0a34156670 0x7f0a340c7440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:53.422 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.402+0000 7f0a3ac56700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f0a34067ed0 0x7f0a340c4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:53.423 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.402+0000 7f0a3ac56700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f0a34069390 0x7f0a340c4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:53.423 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.402+0000 7f0a3ac56700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a340c43d0 con 0x7f0a34067ed0 2022-06-27T12:01:53.424 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.402+0000 7f0a3ac56700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a34068430 con 0x7f0a34069390 2022-06-27T12:01:53.424 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.402+0000 7f0a3ac56700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a34068710 con 0x7f0a34156670 2022-06-27T12:01:53.424 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.402+0000 7f0a33fff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f0a34067ed0 0x7f0a340c4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:53.425 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.402+0000 7f0a33fff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f0a34067ed0 0x7f0a340c4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:46742/0 (socket says 172.21.15.195:46742) 2022-06-27T12:01:53.425 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.402+0000 7f0a33fff700 1 -- 172.21.15.195:0/4269045593 learned_addr learned my addr 172.21.15.195:0/4269045593 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:01:53.426 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.402+0000 7f0a389f2700 1 --2- 172.21.15.195:0/4269045593 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0a34156670 0x7f0a340c7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:53.426 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.402+0000 7f0a391f3700 1 --2- 172.21.15.195:0/4269045593 >> v2:172.21.15.202:3300/0 conn(0x7f0a34069390 0x7f0a340c4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:53.437 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.402+0000 7f0a33fff700 1 -- 172.21.15.195:0/4269045593 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0a34156670 msgr2=0x7f0a340c7440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:01:53.438 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.402+0000 7f0a33fff700 1 --2- 172.21.15.195:0/4269045593 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0a34156670 0x7f0a340c7440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:53.439 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.402+0000 7f0a33fff700 1 -- 172.21.15.195:0/4269045593 >> v2:172.21.15.202:3300/0 conn(0x7f0a34069390 msgr2=0x7f0a340c4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:01:53.439 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.402+0000 7f0a33fff700 1 --2- 172.21.15.195:0/4269045593 >> v2:172.21.15.202:3300/0 conn(0x7f0a34069390 0x7f0a340c4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:53.440 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.402+0000 7f0a33fff700 1 -- 172.21.15.195:0/4269045593 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a340689f0 con 0x7f0a34067ed0 2022-06-27T12:01:53.440 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.402+0000 7f0a33fff700 1 --2- 172.21.15.195:0/4269045593 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0a34067ed0 0x7f0a340c4860 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f0a34156420 tx=0x7f0a20009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdc0e0bf8cc60755 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:01:53.441 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.402+0000 7f0a337fe700 1 -- 172.21.15.195:0/4269045593 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a20011630 con 0x7f0a34067ed0 2022-06-27T12:01:53.441 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.402+0000 7f0a337fe700 1 -- 172.21.15.195:0/4269045593 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a20011790 con 0x7f0a34067ed0 2022-06-27T12:01:53.442 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.403+0000 7f0a337fe700 1 -- 172.21.15.195:0/4269045593 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a20011b30 con 0x7f0a34067ed0 2022-06-27T12:01:53.442 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.403+0000 7f0a3ac56700 1 -- 172.21.15.195:0/4269045593 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0a34067ed0 msgr2=0x7f0a340c4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:53.444 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.403+0000 7f0a3ac56700 1 --2- 172.21.15.195:0/4269045593 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0a34067ed0 0x7f0a340c4860 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f0a34156420 tx=0x7f0a20009120 comp rx=0 tx=0).stop 2022-06-27T12:01:53.445 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.403+0000 7f0a3ac56700 1 -- 172.21.15.195:0/4269045593 shutdown_connections 2022-06-27T12:01:53.445 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.403+0000 7f0a3ac56700 1 --2- 172.21.15.195:0/4269045593 >> v2:172.21.15.202:3300/0 conn(0x7f0a34069390 0x7f0a340c4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:53.445 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.403+0000 7f0a3ac56700 1 --2- 172.21.15.195:0/4269045593 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0a34156670 0x7f0a340c7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:53.446 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.403+0000 7f0a3ac56700 1 --2- 172.21.15.195:0/4269045593 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0a34067ed0 0x7f0a340c4860 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:53.446 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.403+0000 7f0a3ac56700 1 -- 172.21.15.195:0/4269045593 >> 172.21.15.195:0/4269045593 conn(0x7f0a34069e00 msgr2=0x7f0a3406a200 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:01:53.447 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.403+0000 7f0a3ac56700 1 -- 172.21.15.195:0/4269045593 shutdown_connections 2022-06-27T12:01:53.447 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.403+0000 7f0a3ac56700 1 -- 172.21.15.195:0/4269045593 wait complete. 2022-06-27T12:01:53.447 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.404+0000 7f0a3ac56700 1 Processor -- start 2022-06-27T12:01:53.448 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.404+0000 7f0a3ac56700 1 -- start start 2022-06-27T12:01:53.449 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.405+0000 7f0a3ac56700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0a34067ed0 0x7f0a34166170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:53.450 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.405+0000 7f0a3ac56700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0a34069390 0x7f0a340b2230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:53.450 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.405+0000 7f0a3ac56700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0a34156670 0x7f0a34152380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:53.451 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.405+0000 7f0a3ac56700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a340c7550 con 0x7f0a34156670 2022-06-27T12:01:53.451 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.405+0000 7f0a3ac56700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a34150430 con 0x7f0a34069390 2022-06-27T12:01:53.453 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.405+0000 7f0a3ac56700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a34150cd0 con 0x7f0a34067ed0 2022-06-27T12:01:53.453 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.405+0000 7f0a391f3700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0a34156670 0x7f0a34152380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:53.454 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.405+0000 7f0a391f3700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0a34156670 0x7f0a34152380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:46752/0 (socket says 172.21.15.195:46752) 2022-06-27T12:01:53.454 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.405+0000 7f0a391f3700 1 -- 172.21.15.195:0/2062321026 learned_addr learned my addr 172.21.15.195:0/2062321026 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:01:53.455 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.405+0000 7f0a33fff700 1 --2- 172.21.15.195:0/2062321026 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0a34069390 0x7f0a340b2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:53.456 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.405+0000 7f0a389f2700 1 --2- 172.21.15.195:0/2062321026 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0a34067ed0 0x7f0a34166170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:53.456 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.406+0000 7f0a391f3700 1 -- 172.21.15.195:0/2062321026 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0a34067ed0 msgr2=0x7f0a34166170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:53.457 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.406+0000 7f0a391f3700 1 --2- 172.21.15.195:0/2062321026 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0a34067ed0 0x7f0a34166170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:53.457 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.406+0000 7f0a391f3700 1 -- 172.21.15.195:0/2062321026 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0a34069390 msgr2=0x7f0a340b2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:53.458 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.406+0000 7f0a391f3700 1 --2- 172.21.15.195:0/2062321026 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0a34069390 0x7f0a340b2230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:53.458 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.406+0000 7f0a391f3700 1 -- 172.21.15.195:0/2062321026 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a200093a0 con 0x7f0a34156670 2022-06-27T12:01:53.459 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.406+0000 7f0a391f3700 1 --2- 172.21.15.195:0/2062321026 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0a34156670 0x7f0a34152380 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c009ec0 tx=0x7f0a2c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:01:53.461 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.406+0000 7f0a30ff9700 1 -- 172.21.15.195:0/2062321026 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a2c00cc00 con 0x7f0a34156670 2022-06-27T12:01:53.461 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.406+0000 7f0a30ff9700 1 -- 172.21.15.195:0/2062321026 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a2c00f040 con 0x7f0a34156670 2022-06-27T12:01:53.462 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.406+0000 7f0a3ac56700 1 -- 172.21.15.195:0/2062321026 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a34152e30 con 0x7f0a34156670 2022-06-27T12:01:53.462 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.406+0000 7f0a30ff9700 1 -- 172.21.15.195:0/2062321026 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a2c014480 con 0x7f0a34156670 2022-06-27T12:01:53.463 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.406+0000 7f0a3ac56700 1 -- 172.21.15.195:0/2062321026 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a34150680 con 0x7f0a34156670 2022-06-27T12:01:53.463 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.407+0000 7f0a3ac56700 1 -- 172.21.15.195:0/2062321026 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a340722c0 con 0x7f0a34156670 2022-06-27T12:01:53.463 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.408+0000 7f0a30ff9700 1 -- 172.21.15.195:0/2062321026 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f0a2c014830 con 0x7f0a34156670 2022-06-27T12:01:53.464 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.408+0000 7f0a30ff9700 1 --2- 172.21.15.195:0/2062321026 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0a1c063d00 0x7f0a1c0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:53.464 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.408+0000 7f0a389f2700 1 --2- 172.21.15.195:0/2062321026 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0a1c063d00 0x7f0a1c0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:53.464 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.408+0000 7f0a30ff9700 1 -- 172.21.15.195:0/2062321026 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6804+0+0 (secure 0 0 0) 0x7f0a2c081760 con 0x7f0a34156670 2022-06-27T12:01:53.465 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.408+0000 7f0a389f2700 1 --2- 172.21.15.195:0/2062321026 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0a1c063d00 0x7f0a1c0661b0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f0a28009b40 tx=0x7f0a28009060 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:01:53.465 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.412+0000 7f0a30ff9700 1 -- 172.21.15.195:0/2062321026 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f0a2c050060 con 0x7f0a34156670 2022-06-27T12:01:53.662 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.640+0000 7f0a3ac56700 1 -- 172.21.15.195:0/2062321026 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0a34150e10 con 0x7f0a34156670 2022-06-27T12:01:53.663 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.640+0000 7f0a30ff9700 1 -- 172.21.15.195:0/2062321026 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+5365 (secure 0 0 0) 0x7f0a2c053680 con 0x7f0a34156670 2022-06-27T12:01:53.664 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:01:53.664 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:01:48.219526+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5273":{"gid":5273,"name":"d","rank":0,"incarnation":49,"state":"up:rejoin","state_seq":38,"addr":"172.21.15.202:6835/417105338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":417105338},{"type":"v1","addr":"172.21.15.202:6835","nonce":417105338}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:01:53.666 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.642+0000 7f0a3ac56700 1 -- 172.21.15.195:0/2062321026 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0a1c063d00 msgr2=0x7f0a1c0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:53.666 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.642+0000 7f0a3ac56700 1 --2- 172.21.15.195:0/2062321026 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0a1c063d00 0x7f0a1c0661b0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f0a28009b40 tx=0x7f0a28009060 comp rx=0 tx=0).stop 2022-06-27T12:01:53.667 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.642+0000 7f0a3ac56700 1 -- 172.21.15.195:0/2062321026 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0a34156670 msgr2=0x7f0a34152380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:53.668 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.642+0000 7f0a3ac56700 1 --2- 172.21.15.195:0/2062321026 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0a34156670 0x7f0a34152380 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c009ec0 tx=0x7f0a2c00b040 comp rx=0 tx=0).stop 2022-06-27T12:01:53.668 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.642+0000 7f0a3ac56700 1 -- 172.21.15.195:0/2062321026 shutdown_connections 2022-06-27T12:01:53.669 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.642+0000 7f0a3ac56700 1 --2- 172.21.15.195:0/2062321026 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0a1c063d00 0x7f0a1c0661b0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:53.669 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.642+0000 7f0a3ac56700 1 --2- 172.21.15.195:0/2062321026 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0a34156670 0x7f0a34152380 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:53.669 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.642+0000 7f0a3ac56700 1 --2- 172.21.15.195:0/2062321026 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0a34067ed0 0x7f0a34166170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:53.677 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.642+0000 7f0a3ac56700 1 --2- 172.21.15.195:0/2062321026 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0a34069390 0x7f0a340b2230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:53.678 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.642+0000 7f0a3ac56700 1 -- 172.21.15.195:0/2062321026 >> 172.21.15.195:0/2062321026 conn(0x7f0a34069e00 msgr2=0x7f0a3406a740 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:01:53.678 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.642+0000 7f0a3ac56700 1 -- 172.21.15.195:0/2062321026 shutdown_connections 2022-06-27T12:01:53.679 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:53.642+0000 7f0a3ac56700 1 -- 172.21.15.195:0/2062321026 wait complete. 2022-06-27T12:01:53.679 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 51 2022-06-27T12:01:53.697 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:01:55.699 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:01:55.852 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:55.848+0000 7f70b41c4700 1 Processor -- start 2022-06-27T12:01:55.853 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:55.849+0000 7f70b41c4700 1 -- start start 2022-06-27T12:01:55.853 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:55.849+0000 7f70b41c4700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f70ac150500 0x7f70ac063af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:55.854 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:55.849+0000 7f70b41c4700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f70ac06a450 0x7f70ac0c48d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:55.854 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:55.849+0000 7f70b41c4700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70ac064030 con 0x7f70ac1508d0 2022-06-27T12:01:55.855 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:55.849+0000 7f70b41c4700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70ac0c4160 con 0x7f70ac06a450 2022-06-27T12:01:55.855 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:55.849+0000 7f70b41c4700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70ac0c4440 con 0x7f70ac150500 2022-06-27T12:01:55.856 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:55.849+0000 7f70b175f700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f70ac06a450 0x7f70ac0c48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:55.856 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:55.850+0000 7f70b175f700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f70ac06a450 0x7f70ac0c48d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:46756/0 (socket says 172.21.15.195:46756) 2022-06-27T12:01:55.857 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:55.850+0000 7f70b2761700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f70ac1508d0 0x7f70ac0bbf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:52946/0 (socket says 172.21.15.195:52946) 2022-06-27T12:01:55.857 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:55.850+0000 7f70b175f700 1 -- 172.21.15.195:0/2015846368 learned_addr learned my addr 172.21.15.195:0/2015846368 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:01:55.858 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:55.850+0000 7f70b1f60700 1 --2- 172.21.15.195:0/2015846368 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f70ac150500 0x7f70ac063af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:55.868 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:55.850+0000 7f70b175f700 1 -- 172.21.15.195:0/2015846368 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f70ac150500 msgr2=0x7f70ac063af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:01:55.869 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:55.850+0000 7f70b175f700 1 --2- 172.21.15.195:0/2015846368 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f70ac150500 0x7f70ac063af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:55.869 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:55.850+0000 7f70b175f700 1 -- 172.21.15.195:0/2015846368 >> v1:172.21.15.195:6789/0 conn(0x7f70ac1508d0 legacy=0x7f70ac0bbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:55.870 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:55.850+0000 7f70b175f700 1 -- 172.21.15.195:0/2015846368 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70ac15a800 con 0x7f70ac06a450 2022-06-27T12:01:55.870 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:55.850+0000 7f70b1f60700 1 --2- 172.21.15.195:0/2015846368 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f70ac150500 0x7f70ac063af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:01:55.870 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:55.850+0000 7f70b175f700 1 --2- 172.21.15.195:0/2015846368 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f70ac06a450 0x7f70ac0c48d0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f70a8009860 tx=0x7f70a8009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10eb0c0fab1dbdfc server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:01:55.871 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:55.850+0000 7f70b0f5e700 1 -- 172.21.15.195:0/2015846368 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70a800b680 con 0x7f70ac06a450 2022-06-27T12:01:55.872 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:55.850+0000 7f70b0f5e700 1 -- 172.21.15.195:0/2015846368 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70a800b7e0 con 0x7f70ac06a450 2022-06-27T12:01:55.872 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:55.850+0000 7f70b0f5e700 1 -- 172.21.15.195:0/2015846368 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70a800bba0 con 0x7f70ac06a450 2022-06-27T12:01:55.873 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:55.851+0000 7f70b41c4700 1 -- 172.21.15.195:0/2015846368 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f70ac06a450 msgr2=0x7f70ac0c48d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:55.878 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:55.851+0000 7f70b41c4700 1 --2- 172.21.15.195:0/2015846368 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f70ac06a450 0x7f70ac0c48d0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f70a8009860 tx=0x7f70a8009120 comp rx=0 tx=0).stop 2022-06-27T12:01:55.878 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:55.851+0000 7f70b41c4700 1 -- 172.21.15.195:0/2015846368 shutdown_connections 2022-06-27T12:01:55.879 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:55.851+0000 7f70b41c4700 1 --2- 172.21.15.195:0/2015846368 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f70ac150500 0x7f70ac063af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:55.880 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:55.851+0000 7f70b41c4700 1 --2- 172.21.15.195:0/2015846368 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f70ac06a450 0x7f70ac0c48d0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:55.880 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:55.851+0000 7f70b41c4700 1 -- 172.21.15.195:0/2015846368 >> 172.21.15.195:0/2015846368 conn(0x7f70ac0c9b00 msgr2=0x7f70ac0c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:01:55.886 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:55.851+0000 7f70b41c4700 1 -- 172.21.15.195:0/2015846368 shutdown_connections 2022-06-27T12:01:55.889 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:55.851+0000 7f70b41c4700 1 -- 172.21.15.195:0/2015846368 wait complete. 2022-06-27T12:01:55.890 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:55.852+0000 7f70b41c4700 1 Processor -- start 2022-06-27T12:01:55.891 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:55.852+0000 7f70b41c4700 1 -- start start 2022-06-27T12:01:55.891 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:55.853+0000 7f70b41c4700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f70ac06a450 0x7f70ac06de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:55.891 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:55.853+0000 7f70b41c4700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f70ac150500 0x7f70ac1663f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:55.892 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:55.853+0000 7f70b41c4700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f70ac1508d0 0x7f70ac14da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:55.892 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:55.853+0000 7f70b41c4700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70ac1558f0 con 0x7f70ac06a450 2022-06-27T12:01:55.892 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:55.853+0000 7f70b41c4700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70ac155d10 con 0x7f70ac150500 2022-06-27T12:01:55.892 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:55.853+0000 7f70b41c4700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70ac0c08d0 con 0x7f70ac1508d0 2022-06-27T12:01:55.892 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:55.853+0000 7f70b1f60700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f70ac06a450 0x7f70ac06de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:55.893 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:55.853+0000 7f70b1f60700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f70ac06a450 0x7f70ac06de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:46760/0 (socket says 172.21.15.195:46760) 2022-06-27T12:01:55.893 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:55.853+0000 7f70b1f60700 1 -- 172.21.15.195:0/212794725 learned_addr learned my addr 172.21.15.195:0/212794725 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:01:55.893 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:55.853+0000 7f70b2761700 1 --2- 172.21.15.195:0/212794725 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f70ac1508d0 0x7f70ac14da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:55.893 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:55.853+0000 7f70b175f700 1 --2- 172.21.15.195:0/212794725 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f70ac150500 0x7f70ac1663f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:55.894 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:55.853+0000 7f70b1f60700 1 -- 172.21.15.195:0/212794725 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f70ac1508d0 msgr2=0x7f70ac14da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:55.894 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:55.853+0000 7f70b1f60700 1 --2- 172.21.15.195:0/212794725 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f70ac1508d0 0x7f70ac14da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:55.895 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:55.853+0000 7f70b1f60700 1 -- 172.21.15.195:0/212794725 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f70ac150500 msgr2=0x7f70ac1663f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:55.895 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:55.853+0000 7f70b1f60700 1 --2- 172.21.15.195:0/212794725 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f70ac150500 0x7f70ac1663f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:55.896 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:55.853+0000 7f70b1f60700 1 -- 172.21.15.195:0/212794725 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70a80093a0 con 0x7f70ac06a450 2022-06-27T12:01:55.896 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:55.853+0000 7f70b2761700 1 --2- 172.21.15.195:0/212794725 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f70ac1508d0 0x7f70ac14da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T12:01:55.897 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:55.853+0000 7f70b175f700 1 --2- 172.21.15.195:0/212794725 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f70ac150500 0x7f70ac1663f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T12:01:55.897 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:55.853+0000 7f70b1f60700 1 --2- 172.21.15.195:0/212794725 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f70ac06a450 0x7f70ac06de90 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f709c00d8f0 tx=0x7f709c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:01:55.898 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:55.853+0000 7f7099ffb700 1 -- 172.21.15.195:0/212794725 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f709c004030 con 0x7f70ac06a450 2022-06-27T12:01:55.898 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:55.853+0000 7f7099ffb700 1 -- 172.21.15.195:0/212794725 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f709c004190 con 0x7f70ac06a450 2022-06-27T12:01:55.900 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:55.853+0000 7f70b41c4700 1 -- 172.21.15.195:0/212794725 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70ac0c0350 con 0x7f70ac06a450 2022-06-27T12:01:55.901 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:55.853+0000 7f7099ffb700 1 -- 172.21.15.195:0/212794725 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f709c0058c0 con 0x7f70ac06a450 2022-06-27T12:01:55.902 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:55.854+0000 7f70b41c4700 1 -- 172.21.15.195:0/212794725 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70ac06eb00 con 0x7f70ac06a450 2022-06-27T12:01:55.902 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:55.855+0000 7f70b41c4700 1 -- 172.21.15.195:0/212794725 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70ac0722c0 con 0x7f70ac06a450 2022-06-27T12:01:55.903 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:55.855+0000 7f7099ffb700 1 -- 172.21.15.195:0/212794725 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f709c01a020 con 0x7f70ac06a450 2022-06-27T12:01:55.903 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:55.855+0000 7f7099ffb700 1 --2- 172.21.15.195:0/212794725 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f708c063d30 0x7f708c0661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:55.904 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:55.855+0000 7f7099ffb700 1 -- 172.21.15.195:0/212794725 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6804+0+0 (secure 0 0 0) 0x7f709c080640 con 0x7f70ac06a450 2022-06-27T12:01:55.904 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:55.856+0000 7f70b175f700 1 --2- 172.21.15.195:0/212794725 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f708c063d30 0x7f708c0661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:55.905 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:55.856+0000 7f70b175f700 1 --2- 172.21.15.195:0/212794725 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f708c063d30 0x7f708c0661e0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f70a8009860 tx=0x7f70a8017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:01:55.905 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:55.860+0000 7f7099ffb700 1 -- 172.21.15.195:0/212794725 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f709c04bc90 con 0x7f70ac06a450 2022-06-27T12:01:56.087 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:56.087+0000 7f70b41c4700 1 -- 172.21.15.195:0/212794725 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f70ac155b60 con 0x7f70ac06a450 2022-06-27T12:01:56.087 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:56.087+0000 7f7099ffb700 1 -- 172.21.15.195:0/212794725 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+5371 (secure 0 0 0) 0x7f709c04ee50 con 0x7f70ac06a450 2022-06-27T12:01:56.088 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:01:56.088 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:01:54.244021+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5273":{"gid":5273,"name":"d","rank":0,"incarnation":49,"state":"up:clientreplay","state_seq":40,"addr":"172.21.15.202:6835/417105338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":417105338},{"type":"v1","addr":"172.21.15.202:6835","nonce":417105338}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:01:56.090 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:56.089+0000 7f70b41c4700 1 -- 172.21.15.195:0/212794725 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f708c063d30 msgr2=0x7f708c0661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:56.090 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:56.089+0000 7f70b41c4700 1 --2- 172.21.15.195:0/212794725 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f708c063d30 0x7f708c0661e0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f70a8009860 tx=0x7f70a8017000 comp rx=0 tx=0).stop 2022-06-27T12:01:56.090 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:56.089+0000 7f70b41c4700 1 -- 172.21.15.195:0/212794725 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f70ac06a450 msgr2=0x7f70ac06de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:56.090 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:56.089+0000 7f70b41c4700 1 --2- 172.21.15.195:0/212794725 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f70ac06a450 0x7f70ac06de90 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f709c00d8f0 tx=0x7f709c008040 comp rx=0 tx=0).stop 2022-06-27T12:01:56.091 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:56.089+0000 7f70b41c4700 1 -- 172.21.15.195:0/212794725 shutdown_connections 2022-06-27T12:01:56.091 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:56.089+0000 7f70b41c4700 1 --2- 172.21.15.195:0/212794725 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f708c063d30 0x7f708c0661e0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:56.091 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:56.089+0000 7f70b41c4700 1 --2- 172.21.15.195:0/212794725 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f70ac06a450 0x7f70ac06de90 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:56.092 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:56.089+0000 7f70b41c4700 1 --2- 172.21.15.195:0/212794725 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f70ac1508d0 0x7f70ac14da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:56.092 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:56.089+0000 7f70b41c4700 1 --2- 172.21.15.195:0/212794725 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f70ac150500 0x7f70ac1663f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:56.092 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:56.089+0000 7f70b41c4700 1 -- 172.21.15.195:0/212794725 >> 172.21.15.195:0/212794725 conn(0x7f70ac0c9b00 msgr2=0x7f70ac0c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:01:56.093 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:56.090+0000 7f70b41c4700 1 -- 172.21.15.195:0/212794725 shutdown_connections 2022-06-27T12:01:56.095 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:56.090+0000 7f70b41c4700 1 -- 172.21.15.195:0/212794725 wait complete. 2022-06-27T12:01:56.095 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 52 2022-06-27T12:01:56.103 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 52, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-06-27T11:47:52.358095+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 52, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5273": { "addr": "172.21.15.202:6835/417105338", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6834", "nonce": 417105338, "type": "v2" }, { "addr": "172.21.15.202:6835", "nonce": 417105338, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5273, "incarnation": 49, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:clientreplay", "state_seq": 40 } }, "last_failure": 0, "last_failure_osd_epoch": 55, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-06-27T12:01:54.244021+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5273 } } } ], "standbys": [ { "addr": "172.21.15.202:6837/3305006841", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6836", "nonce": 3305006841, "type": "v2" }, { "addr": "172.21.15.202:6837", "nonce": 3305006841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6835/2060709822", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6834", "nonce": 2060709822, "type": "v2" }, { "addr": "172.21.15.195:6835", "nonce": 2060709822, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6838/1598751652", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6836", "nonce": 1598751652, "type": "v2" }, { "addr": "172.21.15.195:6838", "nonce": 1598751652, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4275, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6839/1313800661", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6837", "nonce": 1313800661, "type": "v2" }, { "addr": "172.21.15.195:6839", "nonce": 1313800661, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4278, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-06-27T12:01:58.104 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:01:58.244 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.243+0000 7f999e3ed700 1 Processor -- start 2022-06-27T12:01:58.245 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.243+0000 7f999e3ed700 1 -- start start 2022-06-27T12:01:58.250 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.244+0000 7f999e3ed700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f9998150500 0x7f9998063af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:58.250 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.244+0000 7f999e3ed700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f999806a450 0x7f99980c48d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:58.251 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.244+0000 7f999e3ed700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9998064030 con 0x7f99981508d0 2022-06-27T12:01:58.251 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.244+0000 7f999e3ed700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99980c4160 con 0x7f999806a450 2022-06-27T12:01:58.252 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.244+0000 7f999e3ed700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99980c4440 con 0x7f9998150500 2022-06-27T12:01:58.252 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.244+0000 7f99977fe700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f999806a450 0x7f99980c48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:58.252 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.244+0000 7f99977fe700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f999806a450 0x7f99980c48d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:46780/0 (socket says 172.21.15.195:46780) 2022-06-27T12:01:58.253 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.244+0000 7f99977fe700 1 -- 172.21.15.195:0/2749510277 learned_addr learned my addr 172.21.15.195:0/2749510277 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:01:58.254 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.244+0000 7f9997fff700 1 --2- 172.21.15.195:0/2749510277 >> v2:172.21.15.202:3300/0 conn(0x7f9998150500 0x7f9998063af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:58.280 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.244+0000 7f99977fe700 1 -- 172.21.15.195:0/2749510277 >> v2:172.21.15.202:3300/0 conn(0x7f9998150500 msgr2=0x7f9998063af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:01:58.280 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.244+0000 7f99977fe700 1 --2- 172.21.15.195:0/2749510277 >> v2:172.21.15.202:3300/0 conn(0x7f9998150500 0x7f9998063af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:58.281 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.244+0000 7f99977fe700 1 -- 172.21.15.195:0/2749510277 >> v1:172.21.15.202:6789/0 conn(0x7f99981508d0 legacy=0x7f99980bbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:58.281 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.244+0000 7f99977fe700 1 -- 172.21.15.195:0/2749510277 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f999815a800 con 0x7f999806a450 2022-06-27T12:01:58.281 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.245+0000 7f99977fe700 1 --2- 172.21.15.195:0/2749510277 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f999806a450 0x7f99980c48d0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f999806f840 tx=0x7f9988009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e471fccadd526e7f server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:01:58.282 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.245+0000 7f9996ffd700 1 -- 172.21.15.195:0/2749510277 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9988011630 con 0x7f999806a450 2022-06-27T12:01:58.282 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.245+0000 7f9996ffd700 1 -- 172.21.15.195:0/2749510277 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9988011790 con 0x7f999806a450 2022-06-27T12:01:58.283 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.245+0000 7f9996ffd700 1 -- 172.21.15.195:0/2749510277 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9988011b30 con 0x7f999806a450 2022-06-27T12:01:58.283 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.245+0000 7f999e3ed700 1 -- 172.21.15.195:0/2749510277 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f999806a450 msgr2=0x7f99980c48d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:58.284 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.245+0000 7f999e3ed700 1 --2- 172.21.15.195:0/2749510277 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f999806a450 0x7f99980c48d0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f999806f840 tx=0x7f9988009120 comp rx=0 tx=0).stop 2022-06-27T12:01:58.285 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.245+0000 7f999e3ed700 1 -- 172.21.15.195:0/2749510277 shutdown_connections 2022-06-27T12:01:58.287 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.245+0000 7f999e3ed700 1 --2- 172.21.15.195:0/2749510277 >> v2:172.21.15.202:3300/0 conn(0x7f9998150500 0x7f9998063af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:58.288 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.245+0000 7f999e3ed700 1 --2- 172.21.15.195:0/2749510277 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f999806a450 0x7f99980c48d0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:58.288 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.245+0000 7f999e3ed700 1 -- 172.21.15.195:0/2749510277 >> 172.21.15.195:0/2749510277 conn(0x7f99980c9b00 msgr2=0x7f99980c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:01:58.289 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.245+0000 7f999e3ed700 1 -- 172.21.15.195:0/2749510277 shutdown_connections 2022-06-27T12:01:58.289 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.245+0000 7f999e3ed700 1 -- 172.21.15.195:0/2749510277 wait complete. 2022-06-27T12:01:58.290 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.246+0000 7f999e3ed700 1 Processor -- start 2022-06-27T12:01:58.291 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.247+0000 7f999e3ed700 1 -- start start 2022-06-27T12:01:58.291 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.247+0000 7f999e3ed700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f999806a450 0x7f99980c0e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:58.292 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.247+0000 7f999e3ed700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9998150500 0x7f999814c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:58.292 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.247+0000 7f999e3ed700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f99981508d0 0x7f99980c05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:58.293 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.247+0000 7f999e3ed700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9998166950 con 0x7f99981508d0 2022-06-27T12:01:58.293 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.247+0000 7f999e3ed700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9998152eb0 con 0x7f999806a450 2022-06-27T12:01:58.294 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.247+0000 7f999e3ed700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9998153190 con 0x7f9998150500 2022-06-27T12:01:58.294 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.248+0000 7f999c98a700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f99981508d0 0x7f99980c05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:58.295 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.248+0000 7f999c98a700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f99981508d0 0x7f99980c05c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:46786/0 (socket says 172.21.15.195:46786) 2022-06-27T12:01:58.295 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.248+0000 7f999c98a700 1 -- 172.21.15.195:0/948979049 learned_addr learned my addr 172.21.15.195:0/948979049 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:01:58.296 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.248+0000 7f99977fe700 1 --2- 172.21.15.195:0/948979049 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9998150500 0x7f999814c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:58.296 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.248+0000 7f9997fff700 1 --2- 172.21.15.195:0/948979049 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f999806a450 0x7f99980c0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:58.296 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.248+0000 7f999c98a700 1 -- 172.21.15.195:0/948979049 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9998150500 msgr2=0x7f999814c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:58.323 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.248+0000 7f999c98a700 1 --2- 172.21.15.195:0/948979049 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9998150500 0x7f999814c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:58.336 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.248+0000 7f999c98a700 1 -- 172.21.15.195:0/948979049 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f999806a450 msgr2=0x7f99980c0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:58.336 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.248+0000 7f999c98a700 1 --2- 172.21.15.195:0/948979049 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f999806a450 0x7f99980c0e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:58.336 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.248+0000 7f999c98a700 1 -- 172.21.15.195:0/948979049 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99880093a0 con 0x7f99981508d0 2022-06-27T12:01:58.337 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.248+0000 7f999c98a700 1 --2- 172.21.15.195:0/948979049 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f99981508d0 0x7f99980c05c0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f999000c710 tx=0x7f9990001540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:01:58.337 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.248+0000 7f997ffff700 1 -- 172.21.15.195:0/948979049 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9990003880 con 0x7f99981508d0 2022-06-27T12:01:58.337 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.248+0000 7f997ffff700 1 -- 172.21.15.195:0/948979049 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99900039e0 con 0x7f99981508d0 2022-06-27T12:01:58.337 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.248+0000 7f997ffff700 1 -- 172.21.15.195:0/948979049 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9990003da0 con 0x7f99981508d0 2022-06-27T12:01:58.337 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.248+0000 7f999e3ed700 1 -- 172.21.15.195:0/948979049 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9998153470 con 0x7f99981508d0 2022-06-27T12:01:58.338 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.248+0000 7f999e3ed700 1 -- 172.21.15.195:0/948979049 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99980c0b00 con 0x7f99981508d0 2022-06-27T12:01:58.338 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.249+0000 7f999e3ed700 1 -- 172.21.15.195:0/948979049 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9998166ac0 con 0x7f99981508d0 2022-06-27T12:01:58.338 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.250+0000 7f997ffff700 1 -- 172.21.15.195:0/948979049 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f9990019030 con 0x7f99981508d0 2022-06-27T12:01:58.338 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.250+0000 7f997ffff700 1 --2- 172.21.15.195:0/948979049 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9978063d00 0x7f99780661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:01:58.339 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.250+0000 7f997ffff700 1 -- 172.21.15.195:0/948979049 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6804+0+0 (secure 0 0 0) 0x7f999007fbd0 con 0x7f99981508d0 2022-06-27T12:01:58.339 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.250+0000 7f9997fff700 1 --2- 172.21.15.195:0/948979049 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9978063d00 0x7f99780661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:01:58.339 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.250+0000 7f9997fff700 1 --2- 172.21.15.195:0/948979049 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9978063d00 0x7f99780661b0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f998c009ce0 tx=0x7f998c009490 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:01:58.339 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.254+0000 7f997ffff700 1 -- 172.21.15.195:0/948979049 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f999004e4d0 con 0x7f99981508d0 2022-06-27T12:01:58.504 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.477+0000 7f999e3ed700 1 -- 172.21.15.195:0/948979049 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9998064030 con 0x7f99981508d0 2022-06-27T12:01:58.505 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.477+0000 7f997ffff700 1 -- 172.21.15.195:0/948979049 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+5371 (secure 0 0 0) 0x7f9998064030 con 0x7f99981508d0 2022-06-27T12:01:58.506 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:01:58.506 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:01:54.244021+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5273":{"gid":5273,"name":"d","rank":0,"incarnation":49,"state":"up:clientreplay","state_seq":40,"addr":"172.21.15.202:6835/417105338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":417105338},{"type":"v1","addr":"172.21.15.202:6835","nonce":417105338}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:01:58.510 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.479+0000 7f999e3ed700 1 -- 172.21.15.195:0/948979049 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9978063d00 msgr2=0x7f99780661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:58.510 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.479+0000 7f999e3ed700 1 --2- 172.21.15.195:0/948979049 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9978063d00 0x7f99780661b0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f998c009ce0 tx=0x7f998c009490 comp rx=0 tx=0).stop 2022-06-27T12:01:58.511 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.479+0000 7f999e3ed700 1 -- 172.21.15.195:0/948979049 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f99981508d0 msgr2=0x7f99980c05c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:01:58.511 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.479+0000 7f999e3ed700 1 --2- 172.21.15.195:0/948979049 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f99981508d0 0x7f99980c05c0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f999000c710 tx=0x7f9990001540 comp rx=0 tx=0).stop 2022-06-27T12:01:58.512 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.479+0000 7f999e3ed700 1 -- 172.21.15.195:0/948979049 shutdown_connections 2022-06-27T12:01:58.512 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.479+0000 7f999e3ed700 1 --2- 172.21.15.195:0/948979049 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9978063d00 0x7f99780661b0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:58.513 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.479+0000 7f999e3ed700 1 --2- 172.21.15.195:0/948979049 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f99981508d0 0x7f99980c05c0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:58.513 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.479+0000 7f999e3ed700 1 --2- 172.21.15.195:0/948979049 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f999806a450 0x7f99980c0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:58.514 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.479+0000 7f999e3ed700 1 --2- 172.21.15.195:0/948979049 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9998150500 0x7f999814c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:01:58.514 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.479+0000 7f999e3ed700 1 -- 172.21.15.195:0/948979049 >> 172.21.15.195:0/948979049 conn(0x7f99980c9b00 msgr2=0x7f999806dcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:01:58.515 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.479+0000 7f999e3ed700 1 -- 172.21.15.195:0/948979049 shutdown_connections 2022-06-27T12:01:58.515 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:01:58.479+0000 7f999e3ed700 1 -- 172.21.15.195:0/948979049 wait complete. 2022-06-27T12:01:58.515 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 52 2022-06-27T12:01:58.516 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:02:00.116 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:02:00.122 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:02:00.518 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:02:00.684 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.660+0000 7f90e9ab9700 1 Processor -- start 2022-06-27T12:02:00.685 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.660+0000 7f90e9ab9700 1 -- start start 2022-06-27T12:02:00.685 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.660+0000 7f90e9ab9700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f90e4150580 0x7f90e4156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:02:00.686 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.660+0000 7f90e9ab9700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f90e406f620 0x7f90e406d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:02:00.686 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.660+0000 7f90e9ab9700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90e40c76b0 con 0x7f90e4150950 2022-06-27T12:02:00.687 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.661+0000 7f90e9ab9700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90e4157430 con 0x7f90e406f620 2022-06-27T12:02:00.687 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.661+0000 7f90e9ab9700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90e40c8250 con 0x7f90e4150580 2022-06-27T12:02:00.688 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.661+0000 7f90e3fff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f90e406f620 0x7f90e406d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:02:00.693 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.661+0000 7f90e3fff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f90e406f620 0x7f90e406d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:46794/0 (socket says 172.21.15.195:46794) 2022-06-27T12:02:00.694 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.661+0000 7f90e3fff700 1 -- 172.21.15.195:0/3234464702 learned_addr learned my addr 172.21.15.195:0/3234464702 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:02:00.698 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.661+0000 7f90e37fe700 1 --2- 172.21.15.195:0/3234464702 >> v2:172.21.15.202:3300/0 conn(0x7f90e4150580 0x7f90e4156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:02:00.698 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.661+0000 7f90e3fff700 1 -- 172.21.15.195:0/3234464702 >> v2:172.21.15.202:3300/0 conn(0x7f90e4150580 msgr2=0x7f90e4156ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:02:00.699 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.661+0000 7f90e3fff700 1 --2- 172.21.15.195:0/3234464702 >> v2:172.21.15.202:3300/0 conn(0x7f90e4150580 0x7f90e4156ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:02:00.699 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.661+0000 7f90e3fff700 1 -- 172.21.15.195:0/3234464702 >> v1:172.21.15.202:6789/0 conn(0x7f90e4150950 legacy=0x7f90e40c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:02:00.699 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.661+0000 7f90e3fff700 1 -- 172.21.15.195:0/3234464702 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90e40bad70 con 0x7f90e406f620 2022-06-27T12:02:00.700 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.661+0000 7f90e3fff700 1 --2- 172.21.15.195:0/3234464702 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f90e406f620 0x7f90e406d6a0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f90e40c8470 tx=0x7f90dc009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=354e7f9f94003c95 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:02:00.700 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.661+0000 7f90e27fc700 1 -- 172.21.15.195:0/3234464702 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90dc011630 con 0x7f90e406f620 2022-06-27T12:02:00.701 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.661+0000 7f90e27fc700 1 -- 172.21.15.195:0/3234464702 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90dc011790 con 0x7f90e406f620 2022-06-27T12:02:00.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.661+0000 7f90e27fc700 1 -- 172.21.15.195:0/3234464702 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90dc011b30 con 0x7f90e406f620 2022-06-27T12:02:00.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.662+0000 7f90e9ab9700 1 -- 172.21.15.195:0/3234464702 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f90e406f620 msgr2=0x7f90e406d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:02:00.710 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.662+0000 7f90e9ab9700 1 --2- 172.21.15.195:0/3234464702 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f90e406f620 0x7f90e406d6a0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f90e40c8470 tx=0x7f90dc009120 comp rx=0 tx=0).stop 2022-06-27T12:02:00.710 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.662+0000 7f90e9ab9700 1 -- 172.21.15.195:0/3234464702 shutdown_connections 2022-06-27T12:02:00.710 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.662+0000 7f90e9ab9700 1 --2- 172.21.15.195:0/3234464702 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f90e406f620 0x7f90e406d6a0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:02:00.711 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.662+0000 7f90e9ab9700 1 --2- 172.21.15.195:0/3234464702 >> v2:172.21.15.202:3300/0 conn(0x7f90e4150580 0x7f90e4156ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:02:00.711 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.662+0000 7f90e9ab9700 1 -- 172.21.15.195:0/3234464702 >> 172.21.15.195:0/3234464702 conn(0x7f90e4063af0 msgr2=0x7f90e4063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:02:00.712 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.662+0000 7f90e9ab9700 1 -- 172.21.15.195:0/3234464702 shutdown_connections 2022-06-27T12:02:00.719 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.662+0000 7f90e9ab9700 1 -- 172.21.15.195:0/3234464702 wait complete. 2022-06-27T12:02:00.719 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.663+0000 7f90e9ab9700 1 Processor -- start 2022-06-27T12:02:00.720 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.663+0000 7f90e9ab9700 1 -- start start 2022-06-27T12:02:00.720 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.664+0000 7f90e9ab9700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f90e406f620 0x7f90e416a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:02:00.721 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.664+0000 7f90e9ab9700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f90e4150580 0x7f90e4166d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:02:00.721 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.664+0000 7f90e9ab9700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f90e4150950 0x7f90e40c1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:02:00.722 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.664+0000 7f90e9ab9700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90e414df40 con 0x7f90e4150950 2022-06-27T12:02:00.722 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.664+0000 7f90e9ab9700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90e4169f60 con 0x7f90e4150580 2022-06-27T12:02:00.726 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.664+0000 7f90e9ab9700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90e415a830 con 0x7f90e406f620 2022-06-27T12:02:00.727 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.664+0000 7f90e3fff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f90e4150950 0x7f90e40c1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:02:00.727 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.664+0000 7f90e3fff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f90e4150950 0x7f90e40c1fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:46800/0 (socket says 172.21.15.195:46800) 2022-06-27T12:02:00.728 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.664+0000 7f90e37fe700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f90e406f620 0x7f90e416a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:02:00.728 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.664+0000 7f90e3fff700 1 -- 172.21.15.195:0/2182608253 learned_addr learned my addr 172.21.15.195:0/2182608253 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:02:00.728 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.664+0000 7f90e2ffd700 1 --2- 172.21.15.195:0/2182608253 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f90e4150580 0x7f90e4166d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:02:00.730 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.664+0000 7f90e3fff700 1 -- 172.21.15.195:0/2182608253 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f90e406f620 msgr2=0x7f90e416a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:02:00.730 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.664+0000 7f90e3fff700 1 --2- 172.21.15.195:0/2182608253 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f90e406f620 0x7f90e416a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:02:00.731 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.664+0000 7f90e3fff700 1 -- 172.21.15.195:0/2182608253 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f90e4150580 msgr2=0x7f90e4166d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:02:00.731 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.664+0000 7f90e3fff700 1 --2- 172.21.15.195:0/2182608253 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f90e4150580 0x7f90e4166d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:02:00.731 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.664+0000 7f90e3fff700 1 -- 172.21.15.195:0/2182608253 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90dc0093a0 con 0x7f90e4150950 2022-06-27T12:02:00.732 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.664+0000 7f90e37fe700 1 --2- 172.21.15.195:0/2182608253 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f90e406f620 0x7f90e416a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:02:00.732 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.665+0000 7f90e3fff700 1 --2- 172.21.15.195:0/2182608253 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f90e4150950 0x7f90e40c1fc0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f90dc009ed0 tx=0x7f90dc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:02:00.733 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.665+0000 7f90db7fe700 1 -- 172.21.15.195:0/2182608253 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90dc011630 con 0x7f90e4150950 2022-06-27T12:02:00.733 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.665+0000 7f90db7fe700 1 -- 172.21.15.195:0/2182608253 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90dc004570 con 0x7f90e4150950 2022-06-27T12:02:00.733 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.665+0000 7f90e9ab9700 1 -- 172.21.15.195:0/2182608253 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90e415a270 con 0x7f90e4150950 2022-06-27T12:02:00.734 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.665+0000 7f90db7fe700 1 -- 172.21.15.195:0/2182608253 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90dc003ea0 con 0x7f90e4150950 2022-06-27T12:02:00.735 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.665+0000 7f90e9ab9700 1 -- 172.21.15.195:0/2182608253 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90e4169910 con 0x7f90e4150950 2022-06-27T12:02:00.735 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.666+0000 7f90e9ab9700 1 -- 172.21.15.195:0/2182608253 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90e40722c0 con 0x7f90e4150950 2022-06-27T12:02:00.735 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.666+0000 7f90db7fe700 1 -- 172.21.15.195:0/2182608253 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f90dc02b030 con 0x7f90e4150950 2022-06-27T12:02:00.736 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.667+0000 7f90db7fe700 1 --2- 172.21.15.195:0/2182608253 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f90c8063d00 0x7f90c80661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:02:00.736 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.667+0000 7f90e37fe700 1 --2- 172.21.15.195:0/2182608253 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f90c8063d00 0x7f90c80661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:02:00.737 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.667+0000 7f90db7fe700 1 -- 172.21.15.195:0/2182608253 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6804+0+0 (secure 0 0 0) 0x7f90dc082260 con 0x7f90e4150950 2022-06-27T12:02:00.737 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.667+0000 7f90e37fe700 1 --2- 172.21.15.195:0/2182608253 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f90c8063d00 0x7f90c80661b0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f90d400e950 tx=0x7f90d400b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:02:00.751 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.671+0000 7f90db7fe700 1 -- 172.21.15.195:0/2182608253 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f90dc050b00 con 0x7f90e4150950 2022-06-27T12:02:00.899 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.898+0000 7f90e9ab9700 1 -- 172.21.15.195:0/2182608253 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90e416a0a0 con 0x7f90e4150950 2022-06-27T12:02:00.900 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.898+0000 7f90db7fe700 1 -- 172.21.15.195:0/2182608253 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+5365 (secure 0 0 0) 0x7f90dc054120 con 0x7f90e4150950 2022-06-27T12:02:00.901 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:02:00.903 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:02:00.272060+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5273":{"gid":5273,"name":"d","rank":0,"incarnation":49,"state":"up:active","state_seq":42,"addr":"172.21.15.202:6835/417105338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":417105338},{"type":"v1","addr":"172.21.15.202:6835","nonce":417105338}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:02:00.906 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.900+0000 7f90e9ab9700 1 -- 172.21.15.195:0/2182608253 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f90c8063d00 msgr2=0x7f90c80661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:02:00.907 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.900+0000 7f90e9ab9700 1 --2- 172.21.15.195:0/2182608253 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f90c8063d00 0x7f90c80661b0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f90d400e950 tx=0x7f90d400b040 comp rx=0 tx=0).stop 2022-06-27T12:02:00.907 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.900+0000 7f90e9ab9700 1 -- 172.21.15.195:0/2182608253 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f90e4150950 msgr2=0x7f90e40c1fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:02:00.908 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.900+0000 7f90e9ab9700 1 --2- 172.21.15.195:0/2182608253 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f90e4150950 0x7f90e40c1fc0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f90dc009ed0 tx=0x7f90dc017000 comp rx=0 tx=0).stop 2022-06-27T12:02:00.908 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.900+0000 7f90e9ab9700 1 -- 172.21.15.195:0/2182608253 shutdown_connections 2022-06-27T12:02:00.909 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.900+0000 7f90e9ab9700 1 --2- 172.21.15.195:0/2182608253 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f90c8063d00 0x7f90c80661b0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:02:00.909 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.900+0000 7f90e9ab9700 1 --2- 172.21.15.195:0/2182608253 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f90e4150950 0x7f90e40c1fc0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:02:00.910 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.900+0000 7f90e9ab9700 1 --2- 172.21.15.195:0/2182608253 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f90e406f620 0x7f90e416a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:02:00.910 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.900+0000 7f90e9ab9700 1 --2- 172.21.15.195:0/2182608253 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f90e4150580 0x7f90e4166d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:02:00.911 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.900+0000 7f90e9ab9700 1 -- 172.21.15.195:0/2182608253 >> 172.21.15.195:0/2182608253 conn(0x7f90e4063af0 msgr2=0x7f90e415ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:02:00.911 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.900+0000 7f90e9ab9700 1 -- 172.21.15.195:0/2182608253 shutdown_connections 2022-06-27T12:02:00.912 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:00.901+0000 7f90e9ab9700 1 -- 172.21.15.195:0/2182608253 wait complete. 2022-06-27T12:02:00.912 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 53 2022-06-27T12:02:00.924 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d has gained rank=0, replacing gid=5192 2022-06-27T12:02:00.926 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 5 secs before reviving mds.b 2022-06-27T12:02:05.928 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2022-06-27T12:02:05.929 INFO:tasks.ceph.mds.b:Restarting daemon 2022-06-27T12:02:05.930 DEBUG:teuthology.orchestra.run.smithi202:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-06-27T12:02:05.939 INFO:tasks.ceph.mds.b:Started 2022-06-27T12:02:05.940 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:02:06.019 INFO:tasks.ceph.mds.b.smithi202.stdout:starting mds.b at 2022-06-27T12:02:06.078 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.077+0000 7f5db8366700 1 Processor -- start 2022-06-27T12:02:06.079 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.077+0000 7f5db8366700 1 -- start start 2022-06-27T12:02:06.087 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.079+0000 7f5db8366700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f5db0067ed0 0x7f5db00c7440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:02:06.088 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.079+0000 7f5db8366700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5db00c43d0 con 0x7f5db0156670 2022-06-27T12:02:06.088 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.079+0000 7f5db8366700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5db0068430 con 0x7f5db0069390 2022-06-27T12:02:06.089 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.079+0000 7f5db8366700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5db0068710 con 0x7f5db0067ed0 2022-06-27T12:02:06.089 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.079+0000 7f5db6102700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f5db0156670 0x7f5db00c4860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:52996/0 (socket says 172.21.15.195:52996) 2022-06-27T12:02:06.090 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.079+0000 7f5db6102700 1 -- 172.21.15.195:0/3702497234 learned_addr learned my addr 172.21.15.195:0/3702497234 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:02:06.090 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.079+0000 7f5db5901700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f5db0067ed0 0x7f5db00c7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:02:06.091 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.079+0000 7f5db5100700 1 -- 172.21.15.195:0/3702497234 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3146601067 0 0) 0x7f5db00c43d0 con 0x7f5db0156670 2022-06-27T12:02:06.091 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.079+0000 7f5db5100700 1 -- 172.21.15.195:0/3702497234 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d9c003430 con 0x7f5db0156670 2022-06-27T12:02:06.092 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.079+0000 7f5db5100700 1 -- 172.21.15.195:0/3702497234 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 727290623 0 0) 0x7f5db0068430 con 0x7f5db0069390 2022-06-27T12:02:06.092 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.079+0000 7f5db5901700 1 -- 172.21.15.195:0/3702497234 >> v1:172.21.15.202:6789/0 conn(0x7f5db0069390 legacy=0x7f5db01502b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:02:06.093 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.079+0000 7f5db5901700 1 -- 172.21.15.195:0/3702497234 >> v1:172.21.15.195:6789/0 conn(0x7f5db0156670 legacy=0x7f5db00c4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:02:06.101 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.079+0000 7f5db5901700 1 -- 172.21.15.195:0/3702497234 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5db00689f0 con 0x7f5db0067ed0 2022-06-27T12:02:06.102 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.080+0000 7f5db5901700 1 --2- 172.21.15.195:0/3702497234 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5db0067ed0 0x7f5db00c7440 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f5da40097a0 tx=0x7f5da4009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=149923b8518c139d server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:02:06.102 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.080+0000 7f5db5100700 1 -- 172.21.15.195:0/3702497234 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5da4011630 con 0x7f5db0067ed0 2022-06-27T12:02:06.103 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.080+0000 7f5db5100700 1 -- 172.21.15.195:0/3702497234 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5da4011790 con 0x7f5db0067ed0 2022-06-27T12:02:06.103 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.080+0000 7f5db5100700 1 -- 172.21.15.195:0/3702497234 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5da4011b30 con 0x7f5db0067ed0 2022-06-27T12:02:06.104 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.081+0000 7f5db8366700 1 -- 172.21.15.195:0/3702497234 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5db0067ed0 msgr2=0x7f5db00c7440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:02:06.104 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.081+0000 7f5db8366700 1 --2- 172.21.15.195:0/3702497234 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5db0067ed0 0x7f5db00c7440 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f5da40097a0 tx=0x7f5da4009060 comp rx=0 tx=0).stop 2022-06-27T12:02:06.128 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.081+0000 7f5db8366700 1 -- 172.21.15.195:0/3702497234 shutdown_connections 2022-06-27T12:02:06.128 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.081+0000 7f5db8366700 1 --2- 172.21.15.195:0/3702497234 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5db0067ed0 0x7f5db00c7440 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:02:06.128 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.081+0000 7f5db8366700 1 -- 172.21.15.195:0/3702497234 >> 172.21.15.195:0/3702497234 conn(0x7f5db0069e00 msgr2=0x7f5db006a200 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:02:06.128 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.081+0000 7f5db8366700 1 -- 172.21.15.195:0/3702497234 shutdown_connections 2022-06-27T12:02:06.129 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.081+0000 7f5db8366700 1 -- 172.21.15.195:0/3702497234 wait complete. 2022-06-27T12:02:06.129 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.082+0000 7f5db8366700 1 Processor -- start 2022-06-27T12:02:06.129 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.082+0000 7f5db8366700 1 -- start start 2022-06-27T12:02:06.129 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.083+0000 7f5db8366700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5db0067ed0 0x7f5db01699d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:02:06.130 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.083+0000 7f5db8366700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5db0069390 0x7f5db0169f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:02:06.130 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.083+0000 7f5db8366700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5db0156670 0x7f5db00cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:02:06.130 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.083+0000 7f5db8366700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5db0169580 con 0x7f5db0156670 2022-06-27T12:02:06.130 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.083+0000 7f5db8366700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5db00c9170 con 0x7f5db0069390 2022-06-27T12:02:06.131 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.083+0000 7f5db8366700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5db00c9410 con 0x7f5db0067ed0 2022-06-27T12:02:06.131 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.084+0000 7f5db6903700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5db0156670 0x7f5db00cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:02:06.131 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.084+0000 7f5db6903700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5db0156670 0x7f5db00cbd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:46818/0 (socket says 172.21.15.195:46818) 2022-06-27T12:02:06.131 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.084+0000 7f5db6903700 1 -- 172.21.15.195:0/2862819431 learned_addr learned my addr 172.21.15.195:0/2862819431 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:02:06.132 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.084+0000 7f5db5901700 1 --2- 172.21.15.195:0/2862819431 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5db0069390 0x7f5db0169f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:02:06.132 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.084+0000 7f5db6102700 1 --2- 172.21.15.195:0/2862819431 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5db0067ed0 0x7f5db01699d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:02:06.132 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.084+0000 7f5db5901700 1 -- 172.21.15.195:0/2862819431 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5db0067ed0 msgr2=0x7f5db01699d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:02:06.132 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.084+0000 7f5db5901700 1 --2- 172.21.15.195:0/2862819431 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5db0067ed0 0x7f5db01699d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:02:06.132 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.084+0000 7f5db5901700 1 -- 172.21.15.195:0/2862819431 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5db0156670 msgr2=0x7f5db00cbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:02:06.133 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.084+0000 7f5db5901700 1 --2- 172.21.15.195:0/2862819431 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5db0156670 0x7f5db00cbd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:02:06.133 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.084+0000 7f5db5901700 1 -- 172.21.15.195:0/2862819431 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5da40092e0 con 0x7f5db0069390 2022-06-27T12:02:06.133 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.084+0000 7f5db6903700 1 --2- 172.21.15.195:0/2862819431 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5db0156670 0x7f5db00cbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:02:06.133 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.084+0000 7f5db5901700 1 --2- 172.21.15.195:0/2862819431 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5db0069390 0x7f5db0169f10 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5da4016000 tx=0x7f5da4017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:02:06.134 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.085+0000 7f5dae7fc700 1 -- 172.21.15.195:0/2862819431 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5da4011900 con 0x7f5db0069390 2022-06-27T12:02:06.134 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.085+0000 7f5dae7fc700 1 -- 172.21.15.195:0/2862819431 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5da4011a60 con 0x7f5db0069390 2022-06-27T12:02:06.134 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.085+0000 7f5dae7fc700 1 -- 172.21.15.195:0/2862819431 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5da4025650 con 0x7f5db0069390 2022-06-27T12:02:06.134 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.085+0000 7f5db8366700 1 -- 172.21.15.195:0/2862819431 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5db00c96d0 con 0x7f5db0069390 2022-06-27T12:02:06.135 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.085+0000 7f5db8366700 1 -- 172.21.15.195:0/2862819431 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5db00c9c90 con 0x7f5db0069390 2022-06-27T12:02:06.135 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.086+0000 7f5db8366700 1 -- 172.21.15.195:0/2862819431 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5db01696f0 con 0x7f5db0069390 2022-06-27T12:02:06.135 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.086+0000 7f5dae7fc700 1 -- 172.21.15.195:0/2862819431 <== mon.1 v2:172.21.15.202:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f5da402b020 con 0x7f5db0069390 2022-06-27T12:02:06.135 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.086+0000 7f5dae7fc700 1 --2- 172.21.15.195:0/2862819431 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5d90063d30 0x7f5d900661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:02:06.136 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.087+0000 7f5dae7fc700 1 -- 172.21.15.195:0/2862819431 <== mon.1 v2:172.21.15.202:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6804+0+0 (secure 0 0 0) 0x7f5da40820e0 con 0x7f5db0069390 2022-06-27T12:02:06.136 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.087+0000 7f5db6102700 1 --2- 172.21.15.195:0/2862819431 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5d90063d30 0x7f5d900661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:02:06.136 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.087+0000 7f5db6102700 1 --2- 172.21.15.195:0/2862819431 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5d90063d30 0x7f5d900661e0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f5da0005790 tx=0x7f5da0009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:02:06.137 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.091+0000 7f5dae7fc700 1 -- 172.21.15.195:0/2862819431 <== mon.1 v2:172.21.15.202:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f5db01696f0 con 0x7f5db0069390 2022-06-27T12:02:06.322 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.320+0000 7f5db8366700 1 -- 172.21.15.195:0/2862819431 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5db01696f0 con 0x7f5db0069390 2022-06-27T12:02:06.322 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.321+0000 7f5dae7fc700 1 -- 172.21.15.195:0/2862819431 <== mon.1 v2:172.21.15.202:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+5365 (secure 0 0 0) 0x7f5db01696f0 con 0x7f5db0069390 2022-06-27T12:02:06.323 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:02:06.323 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:02:00.272060+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5273":{"gid":5273,"name":"d","rank":0,"incarnation":49,"state":"up:active","state_seq":42,"addr":"172.21.15.202:6835/417105338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":417105338},{"type":"v1","addr":"172.21.15.202:6835","nonce":417105338}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:02:06.324 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.322+0000 7f5db8366700 1 -- 172.21.15.195:0/2862819431 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5d90063d30 msgr2=0x7f5d900661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:02:06.325 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.322+0000 7f5db8366700 1 --2- 172.21.15.195:0/2862819431 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5d90063d30 0x7f5d900661e0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f5da0005790 tx=0x7f5da0009040 comp rx=0 tx=0).stop 2022-06-27T12:02:06.325 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.322+0000 7f5db8366700 1 -- 172.21.15.195:0/2862819431 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5db0069390 msgr2=0x7f5db0169f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:02:06.325 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.322+0000 7f5db8366700 1 --2- 172.21.15.195:0/2862819431 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5db0069390 0x7f5db0169f10 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5da4016000 tx=0x7f5da4017000 comp rx=0 tx=0).stop 2022-06-27T12:02:06.326 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.323+0000 7f5db8366700 1 -- 172.21.15.195:0/2862819431 shutdown_connections 2022-06-27T12:02:06.326 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.323+0000 7f5db8366700 1 --2- 172.21.15.195:0/2862819431 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5d90063d30 0x7f5d900661e0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:02:06.327 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.323+0000 7f5db8366700 1 --2- 172.21.15.195:0/2862819431 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5db0156670 0x7f5db00cbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:02:06.327 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.323+0000 7f5db8366700 1 --2- 172.21.15.195:0/2862819431 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5db0067ed0 0x7f5db01699d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:02:06.328 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.323+0000 7f5db8366700 1 --2- 172.21.15.195:0/2862819431 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5db0069390 0x7f5db0169f10 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:02:06.339 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.323+0000 7f5db8366700 1 -- 172.21.15.195:0/2862819431 >> 172.21.15.195:0/2862819431 conn(0x7f5db0069e00 msgr2=0x7f5db014f470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:02:06.340 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.323+0000 7f5db8366700 1 -- 172.21.15.195:0/2862819431 shutdown_connections 2022-06-27T12:02:06.340 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:06.323+0000 7f5db8366700 1 -- 172.21.15.195:0/2862819431 wait complete. 2022-06-27T12:02:06.341 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 53 2022-06-27T12:02:06.341 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2022-06-27T12:02:07.107 INFO:tasks.ceph.mds.d.smithi202.stderr:2022-06-27T12:02:07.102+0000 7f253a1a1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2022-06-27T12:02:08.343 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:02:08.478 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.478+0000 7f80651fb700 1 Processor -- start 2022-06-27T12:02:08.479 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.478+0000 7f80651fb700 1 -- start start 2022-06-27T12:02:08.488 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.479+0000 7f80651fb700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f806006a440 0x7f80600688a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:02:08.488 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.479+0000 7f80651fb700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f80601564d0 0x7f8060156950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:02:08.488 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.479+0000 7f80651fb700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80600c4b10 con 0x7f8060156100 2022-06-27T12:02:08.489 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.479+0000 7f80651fb700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80600c3dc0 con 0x7f806006a440 2022-06-27T12:02:08.491 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.479+0000 7f80651fb700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8060068f20 con 0x7f80601564d0 2022-06-27T12:02:08.491 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.479+0000 7f805e59c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f806006a440 0x7f80600688a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:02:08.491 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.479+0000 7f805e59c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f806006a440 0x7f80600688a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:46830/0 (socket says 172.21.15.195:46830) 2022-06-27T12:02:08.492 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.479+0000 7f805e59c700 1 -- 172.21.15.195:0/3489457034 learned_addr learned my addr 172.21.15.195:0/3489457034 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:02:08.493 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.479+0000 7f805f59e700 1 --2- 172.21.15.195:0/3489457034 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f80601564d0 0x7f8060156950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:02:08.493 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.479+0000 7f805e59c700 1 -- 172.21.15.195:0/3489457034 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f80601564d0 msgr2=0x7f8060156950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:02:08.494 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.479+0000 7f805e59c700 1 --2- 172.21.15.195:0/3489457034 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f80601564d0 0x7f8060156950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:02:08.494 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.479+0000 7f805e59c700 1 -- 172.21.15.195:0/3489457034 >> v1:172.21.15.202:6789/0 conn(0x7f8060156100 legacy=0x7f80600c4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:02:08.495 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.479+0000 7f805e59c700 1 -- 172.21.15.195:0/3489457034 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80600c56b0 con 0x7f806006a440 2022-06-27T12:02:08.495 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.479+0000 7f805f59e700 1 --2- 172.21.15.195:0/3489457034 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f80601564d0 0x7f8060156950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T12:02:08.496 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.479+0000 7f805e59c700 1 --2- 172.21.15.195:0/3489457034 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f806006a440 0x7f80600688a0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f8050009860 tx=0x7f8050009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e54abdf286d1bcee server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:02:08.499 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.480+0000 7f805dd9b700 1 -- 172.21.15.195:0/3489457034 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f805000b680 con 0x7f806006a440 2022-06-27T12:02:08.500 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.480+0000 7f805dd9b700 1 -- 172.21.15.195:0/3489457034 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f805000b7e0 con 0x7f806006a440 2022-06-27T12:02:08.500 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.480+0000 7f805dd9b700 1 -- 172.21.15.195:0/3489457034 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f805000bba0 con 0x7f806006a440 2022-06-27T12:02:08.501 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.480+0000 7f80651fb700 1 -- 172.21.15.195:0/3489457034 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f806006a440 msgr2=0x7f80600688a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:02:08.501 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.480+0000 7f80651fb700 1 --2- 172.21.15.195:0/3489457034 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f806006a440 0x7f80600688a0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f8050009860 tx=0x7f8050009120 comp rx=0 tx=0).stop 2022-06-27T12:02:08.502 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.480+0000 7f80651fb700 1 -- 172.21.15.195:0/3489457034 shutdown_connections 2022-06-27T12:02:08.502 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.480+0000 7f80651fb700 1 --2- 172.21.15.195:0/3489457034 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f80601564d0 0x7f8060156950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:02:08.502 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.480+0000 7f80651fb700 1 --2- 172.21.15.195:0/3489457034 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f806006a440 0x7f80600688a0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:02:08.502 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.480+0000 7f80651fb700 1 -- 172.21.15.195:0/3489457034 >> 172.21.15.195:0/3489457034 conn(0x7f80600c9b00 msgr2=0x7f80600c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:02:08.503 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.480+0000 7f80651fb700 1 -- 172.21.15.195:0/3489457034 shutdown_connections 2022-06-27T12:02:08.503 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.480+0000 7f80651fb700 1 -- 172.21.15.195:0/3489457034 wait complete. 2022-06-27T12:02:08.503 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.481+0000 7f80651fb700 1 Processor -- start 2022-06-27T12:02:08.503 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.482+0000 7f80651fb700 1 -- start start 2022-06-27T12:02:08.504 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.482+0000 7f80651fb700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f806006a440 0x7f80601670a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:02:08.504 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.482+0000 7f80651fb700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8060156100 0x7f8060154cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:02:08.504 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.482+0000 7f80651fb700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f80601564d0 0x7f80600bad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:02:08.504 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.482+0000 7f80651fb700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f806006ba90 con 0x7f80601564d0 2022-06-27T12:02:08.505 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.482+0000 7f80651fb700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80601675e0 con 0x7f8060156100 2022-06-27T12:02:08.505 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.482+0000 7f80651fb700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80600ba330 con 0x7f806006a440 2022-06-27T12:02:08.505 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.482+0000 7f805f59e700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f80601564d0 0x7f80600bad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:02:08.505 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.482+0000 7f805f59e700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f80601564d0 0x7f80600bad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:46838/0 (socket says 172.21.15.195:46838) 2022-06-27T12:02:08.506 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.482+0000 7f805f59e700 1 -- 172.21.15.195:0/1992141307 learned_addr learned my addr 172.21.15.195:0/1992141307 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:02:08.506 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.483+0000 7f805e59c700 1 --2- 172.21.15.195:0/1992141307 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8060156100 0x7f8060154cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:02:08.506 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.483+0000 7f805ed9d700 1 --2- 172.21.15.195:0/1992141307 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f806006a440 0x7f80601670a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:02:08.506 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.483+0000 7f805f59e700 1 -- 172.21.15.195:0/1992141307 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f806006a440 msgr2=0x7f80601670a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:02:08.507 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.483+0000 7f805f59e700 1 --2- 172.21.15.195:0/1992141307 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f806006a440 0x7f80601670a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:02:08.519 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.483+0000 7f805f59e700 1 -- 172.21.15.195:0/1992141307 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8060156100 msgr2=0x7f8060154cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:02:08.520 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.483+0000 7f805f59e700 1 --2- 172.21.15.195:0/1992141307 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8060156100 0x7f8060154cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:02:08.520 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.483+0000 7f805f59e700 1 -- 172.21.15.195:0/1992141307 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80500093a0 con 0x7f80601564d0 2022-06-27T12:02:08.520 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.483+0000 7f805f59e700 1 --2- 172.21.15.195:0/1992141307 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f80601564d0 0x7f80600bad60 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f805800cf20 tx=0x7f8058007860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:02:08.520 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.483+0000 7f8056ffd700 1 -- 172.21.15.195:0/1992141307 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f805800adf0 con 0x7f80601564d0 2022-06-27T12:02:08.521 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.483+0000 7f8056ffd700 1 -- 172.21.15.195:0/1992141307 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8058016420 con 0x7f80601564d0 2022-06-27T12:02:08.521 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.483+0000 7f80651fb700 1 -- 172.21.15.195:0/1992141307 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8060153f20 con 0x7f80601564d0 2022-06-27T12:02:08.521 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.483+0000 7f8056ffd700 1 -- 172.21.15.195:0/1992141307 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80580167e0 con 0x7f80601564d0 2022-06-27T12:02:08.521 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.483+0000 7f80651fb700 1 -- 172.21.15.195:0/1992141307 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80601544a0 con 0x7f80601564d0 2022-06-27T12:02:08.522 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.484+0000 7f80651fb700 1 -- 172.21.15.195:0/1992141307 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80600722c0 con 0x7f80601564d0 2022-06-27T12:02:08.522 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.484+0000 7f8056ffd700 1 -- 172.21.15.195:0/1992141307 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f8058016c10 con 0x7f80601564d0 2022-06-27T12:02:08.522 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.485+0000 7f8056ffd700 1 --2- 172.21.15.195:0/1992141307 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8044063d30 0x7f80440661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:02:08.522 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.485+0000 7f8056ffd700 1 -- 172.21.15.195:0/1992141307 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6804+0+0 (secure 0 0 0) 0x7f8058012070 con 0x7f80601564d0 2022-06-27T12:02:08.523 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.485+0000 7f805ed9d700 1 --2- 172.21.15.195:0/1992141307 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8044063d30 0x7f80440661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:02:08.523 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.485+0000 7f805ed9d700 1 --2- 172.21.15.195:0/1992141307 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8044063d30 0x7f80440661e0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f80480016f0 tx=0x7f8048009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:02:08.538 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.489+0000 7f8056ffd700 1 -- 172.21.15.195:0/1992141307 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f805804cc90 con 0x7f80601564d0 2022-06-27T12:02:08.729 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.720+0000 7f80651fb700 1 -- 172.21.15.195:0/1992141307 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f80600c6360 con 0x7f80601564d0 2022-06-27T12:02:08.730 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.721+0000 7f8056ffd700 1 -- 172.21.15.195:0/1992141307 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+6125 (secure 0 0 0) 0x7f805804fe50 con 0x7f80601564d0 2022-06-27T12:02:08.731 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:02:08.731 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":5818,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/3399027527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3399027527},{"type":"v1","addr":"172.21.15.202:6833","nonce":3399027527}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:02:00.272060+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5273":{"gid":5273,"name":"d","rank":0,"incarnation":49,"state":"up:active","state_seq":42,"addr":"172.21.15.202:6835/417105338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":417105338},{"type":"v1","addr":"172.21.15.202:6835","nonce":417105338}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:02:08.733 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.723+0000 7f80651fb700 1 -- 172.21.15.195:0/1992141307 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8044063d30 msgr2=0x7f80440661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:02:08.734 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.723+0000 7f80651fb700 1 --2- 172.21.15.195:0/1992141307 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8044063d30 0x7f80440661e0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f80480016f0 tx=0x7f8048009040 comp rx=0 tx=0).stop 2022-06-27T12:02:08.734 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.723+0000 7f80651fb700 1 -- 172.21.15.195:0/1992141307 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f80601564d0 msgr2=0x7f80600bad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:02:08.735 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.723+0000 7f80651fb700 1 --2- 172.21.15.195:0/1992141307 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f80601564d0 0x7f80600bad60 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f805800cf20 tx=0x7f8058007860 comp rx=0 tx=0).stop 2022-06-27T12:02:08.750 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.723+0000 7f80651fb700 1 -- 172.21.15.195:0/1992141307 shutdown_connections 2022-06-27T12:02:08.751 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.723+0000 7f80651fb700 1 --2- 172.21.15.195:0/1992141307 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8044063d30 0x7f80440661e0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:02:08.751 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.723+0000 7f80651fb700 1 --2- 172.21.15.195:0/1992141307 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f80601564d0 0x7f80600bad60 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:02:08.751 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.723+0000 7f80651fb700 1 --2- 172.21.15.195:0/1992141307 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f806006a440 0x7f80601670a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:02:08.752 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.723+0000 7f80651fb700 1 --2- 172.21.15.195:0/1992141307 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8060156100 0x7f8060154cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:02:08.752 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.723+0000 7f80651fb700 1 -- 172.21.15.195:0/1992141307 >> 172.21.15.195:0/1992141307 conn(0x7f80600c9b00 msgr2=0x7f80600c03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:02:08.753 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.723+0000 7f80651fb700 1 -- 172.21.15.195:0/1992141307 shutdown_connections 2022-06-27T12:02:08.753 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:02:08.723+0000 7f80651fb700 1 -- 172.21.15.195:0/1992141307 wait complete. 2022-06-27T12:02:08.753 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 54 2022-06-27T12:02:08.768 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2022-06-27T12:02:08.769 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 69 secs before thrashing 2022-06-27T12:02:30.321 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:02:30.334 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:03:00.160 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 1% (133/12226) Updating files: 1% (236/12226) Updating files: 2% (245/12226) Updating files: 2% (280/12226) Updating files: 2% (349/12226) Updating files: 3% (367/12226) Updating files: 3% (450/12226) Updating files: 4% (490/12226) Updating files: 4% (533/12226) Updating files: 5% (612/12226) Updating files: 5% (671/12226) Updating files: 5% (709/12226) Updating files: 5% (714/12226) Updating files: 5% (732/12226) Updating files: 6% (734/12226) Updating files: 7% (856/12226) Updating files: 7% (921/12226) Updating files: 8% (979/12226) Updating files: 8% (1025/12226) Updating files: 8% (1033/12226) Updating files: 8% (1080/12226) Updating files: 9% (1101/12226) Updating files: 10% (1223/12226) Updating files: 10% (1236/12226) Updating files: 10% (1282/12226) Updating files: 10% (1300/12226) Updating files: 11% (1345/12226) Updating files: 11% (1377/12226) Updating files: 11% (1463/12226) Updating files: 12% (1468/12226) Updating files: 12% (1476/12226) Updating files: 12% (1540/12226) Updating files: 12% (1560/12226) Updating files: 12% (1582/12226) Updating files: 13% (1590/12226) Updating files: 13% (1595/12226) Updating files: 13% (1638/12226) Updating files: 13% (1667/12226) Updating files: 13% (1708/12226) Updating files: 14% (1712/12226) Updating files: 14% (1766/12226) Updating files: 14% (1804/12226) Updating files: 15% (1834/12226) Updating files: 15% (1835/12226) Updating files: 15% (1920/12226) Updating files: 15% (1934/12226) Updating files: 16% (1957/12226) Updating files: 16% (2019/12226) Updating files: 17% (2079/12226) Updating files: 17% (2141/12226) Updating files: 18% (2201/12226) Updating files: 18% (2271/12226) Updating files: 19% (2323/12226) Updating files: 19% (2363/12226) Updating files: 19% (2391/12226) Updating files: 20% (2446/12226) Updating files: 20% (2502/12226) Updating files: 20% (2565/12226) Updating files: 21% (2568/12226) Updating files: 21% (2648/12226) Updating files: 22% (2690/12226) Updating files: 22% (2736/12226) Updating files: 22% (2745/12226) Updating files: 22% (2795/12226) Updating files: 23% (2812/12226) Updating files: 23% (2905/12226) Updating files: 24% (2935/12226) Updating files: 25% (3057/12226) Updating files: 25% (3059/12226) Updating files: 25% (3075/12226) Updating files: 25% (3114/12226) Updating files: 26% (3179/12226) Updating files: 26% (3188/12226) Updating files: 26% (3288/12226) Updating files: 27% (3302/12226) Updating files: 27% (3373/12226) Updating files: 28% (3424/12226) Updating files: 28% (3459/12226) Updating files: 28% (3497/12226) Updating files: 28% (3506/12226) Updating files: 29% (3546/12226) Updating files: 29% (3555/12226) Updating files: 29% (3663/12226) Updating files: 30% (3668/12226) Updating files: 30% (3722/12226) Updating files: 30% (3772/12226) Updating files: 30% (3785/12226) Updating files: 31% (3791/12226) Updating files: 31% (3846/12226) Updating files: 32% (3913/12226) Updating files: 32% (3939/12226) Updating files: 32% (3979/12226) Updating files: 32% (3990/12226) Updating files: 33% (4035/12226) Updating files: 33% (4078/12226) Updating files: 33% (4122/12226) Updating files: 33% (4141/12226) Updating files: 34% (4157/12226) Updating files: 34% (4175/12226) Updating files: 34% (4226/12226) Updating files: 35% (4280/12226) Updating files: 35% (4350/12226) Updating files: 36% (4402/12226) Updating files: 36% (4414/12226) Updating files: 36% (4438/12226) Updating files: 37% (4524/12226) Updating files: 37% (4541/12226) Updating files: 38% (4646/12226) Updating files: 38% (4680/12226) Updating files: 38% (4735/12226) Updating files: 39% (4769/12226) Updating files: 39% (4790/12226) Updating files: 39% (4847/12226) Updating files: 39% (4885/12226) Updating files: 40% (4891/12226) Updating files: 40% (4906/12226) Updating files: 40% (4946/12226) Updating files: 40% (4964/12226) Updating files: 40% (4985/12226) Updating files: 40% (4996/12226) Updating files: 41% (5013/12226) Updating files: 41% (5046/12226) Updating files: 41% (5108/12226) Updating files: 42% (5135/12226) Updating files: 43% (5258/12226) Updating files: 43% (5284/12226) Updating files: 43% (5350/12226) Updating files: 44% (5380/12226) Updating files: 44% (5424/12226) Updating files: 45% (5502/12226) Updating files: 45% (5569/12226) Updating files: 46% (5624/12226) Updating files: 46% (5688/12226) Updating files: 47% (5747/12226) Updating files: 48% (5869/12226) Updating files: 48% (5880/12226) Updating files: 48% (5894/12226) Updating files: 49% (5991/12226) Updating files: 49% (6004/12226) Updating files: 49% (6024/12226) Updating files: 50% (6113/12226) Updating files: 50% (6142/12226) Updating files: 50% (6205/12226) Updating files: 51% (6236/12226) Updating files: 51% (6330/12226) Updating files: 52% (6358/12226) Updating files: 52% (6451/12226) Updating files: 52% (6453/12226) Updating files: 53% (6480/12226) Updating files: 53% (6566/12226) Updating files: 54% (6603/12226) Updating files: 54% (6630/12226) Updating files: 54% (6667/12226) Updating files: 55% (6725/12226) Updating files: 56% (6847/12226) Updating files: 56% (6860/12226) Updating files: 56% (6928/12226) Updating files: 56% (6949/12226) Updating files: 57% (6969/12226) Updating files: 57% (6977/12226) Updating files: 57% (7022/12226) Updating files: 57% (7076/12226) Updating files: 58% (7092/12226) Updating files: 58% (7094/12226) Updating files: 58% (7136/12226) Updating files: 58% (7176/12226) Updating files: 58% (7207/12226) Updating files: 59% (7214/12226) Updating files: 59% (7277/12226) Updating files: 60% (7336/12226) Updating files: 60% (7345/12226) Updating files: 61% (7458/12226) Updating files: 61% (7573/12226) Updating files: 62% (7581/12226) Updating files: 62% (7620/12226) Updating files: 62% (7701/12226) Updating files: 63% (7703/12226) Updating files: 63% (7745/12226) Updating files: 63% (7767/12226) Updating files: 63% (7803/12226) Updating files: 64% (7825/12226) Updating files: 64% (7866/12226) Updating files: 64% (7887/12226) Updating files: 64% (7893/12226) Updating files: 64% (7910/12226) Updating files: 65% (7947/12226) Updating files: 65% (7950/12226) Updating files: 65% (8010/12226) Updating files: 65% (8040/12226) Updating files: 66% (8070/12226) Updating files: 66% (8075/12226) Updating files: 66% (8158/12226) Updating files: 66% (8175/12226) Updating files: 67% (8192/12226) Updating files: 67% (8209/12226) Updating files: 67% (8305/12226) Updating files: 68% (8314/12226) Updating files: 68% (8376/12226) Updating files: 69% (8436/12226) Updating files: 69% (8532/12226) Updating files: 70% (8559/12226) Updating files: 70% (8561/12226) Updating files: 71% (8681/12226) Updating files: 71% (8729/12226) Updating files: 71% (8750/12226) Updating files: 72% (8803/12226) Updating files: 72% (8808/12226) Updating files: 72% (8895/12226) Updating files: 72% (8922/12226) Updating files: 73% (8925/12226) Updating files: 73% (8996/12226) Updating files: 74% (9048/12226) Updating files: 74% (9159/12226) Updating files: 74% (9167/12226) Updating files: 75% (9170/12226) Updating files: 75% (9177/12226) Updating files: 75% (9285/12226) Updating files: 76% (9292/12226) Updating files: 76% (9386/12226) Updating files: 77% (9415/12226) Updating files: 77% (9448/12226) Updating files: 77% (9523/12226) Updating files: 78% (9537/12226) Updating files: 79% (9659/12226) Updating files: 79% (9681/12226) Updating files: 79% (9735/12226) Updating files: 79% (9754/12226) Updating files: 80% (9781/12226) Updating files: 80% (9897/12226) Updating files: 81% (9904/12226) Updating files: 81% (9941/12226) Updating files: 81% (10014/12226) Updating files: 82% (10026/12226) Updating files: 82% (10106/12226) Updating files: 83% (10148/12226) Updating files: 83% (10195/12226) Updating files: 83% (10252/12226) Updating files: 84% (10270/12226) Updating files: 84% (10324/12226) Updating files: 85% (10393/12226) Updating files: 85% (10407/12226) Updating files: 85% (10460/12226) Updating files: 85% (10491/12226) Updating files: 86% (10515/12226) Updating files: 86% (10617/12226) Updating files: 87% (10637/12226) Updating files: 87% (10672/12226) Updating files: 87% (10742/12226) Updating files: 88% (10759/12226) Updating files: 88% (10780/12226) Updating files: 88% (10840/12226) Updating files: 89% (10882/12226) Updating files: 90% (11004/12226) Updating files: 90% (11016/12226) Updating files: 90% (11056/12226) Updating files: 90% (11080/12226) Updating files: 91% (11126/12226) Updating files: 91% (11238/12226) Updating files: 91% (11240/12226) Updating files: 92% (11248/12226) Updating files: 92% (11266/12226) Updating files: 92% (11293/12226) Updating files: 93% (11371/12226) Updating files: 94% (11493/12226) Updating files: 94% (11559/12226) Updating files: 95% (11615/12226) Updating files: 96% (11737/12226) Updating files: 96% (11853/12226) Updating files: 97% (11860/12226) Updating files: 97% (11897/12226) Updating files: 97% (11977/12226) Updating files: 98% (11982/12226) Updating files: 98% (12032/12226) Updating files: 98% (12095/12226) Updating files: 99% (12104/12226) Updating files: 99% (12184/12226) Updating files: 100% (12226/12226) Updating files: 100% (12226/12226), done. 2022-06-27T12:03:00.407 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:03:00.414 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:03:01.427 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at 204c175eea3 msg: blobhash on entity_name_t causes strange badness... should look into that, but avoid for now 2022-06-27T12:03:01.519 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.1 2022-06-27T12:03:01.520 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:03:01.520 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 2 -eq 48 ']' 2022-06-27T12:03:01.520 INFO:tasks.workunit.client.0.smithi195.stdout:v0.2 2022-06-27T12:03:01.521 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.2 2022-06-27T12:03:01.522 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.2 2022-06-27T12:03:01.522 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.2 2022-06-27T12:03:04.162 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 69% (154/222) Updating files: 70% (156/222) Updating files: 71% (158/222) Updating files: 72% (160/222) Updating files: 73% (163/222) Updating files: 74% (165/222) Updating files: 75% (167/222) Updating files: 76% (169/222) Updating files: 77% (171/222) Updating files: 78% (174/222) Updating files: 79% (176/222) Updating files: 80% (178/222) Updating files: 81% (180/222) Updating files: 82% (183/222) Updating files: 83% (185/222) Updating files: 84% (187/222) Updating files: 85% (189/222) Updating files: 86% (191/222) Updating files: 87% (194/222) Updating files: 88% (196/222) Updating files: 89% (198/222) Updating files: 90% (200/222) Updating files: 91% (203/222) Updating files: 92% (205/222) Updating files: 93% (207/222) Updating files: 94% (209/222) Updating files: 95% (211/222) Updating files: 96% (214/222) Updating files: 97% (216/222) Updating files: 98% (218/222) Updating files: 99% (220/222) Updating files: 100% (222/222) Updating files: 100% (222/222), done. 2022-06-27T12:03:04.182 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at 2b5f76f48e9 readme 2022-06-27T12:03:04.185 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.2 2022-06-27T12:03:04.407 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:03:04.407 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 3 -eq 48 ']' 2022-06-27T12:03:04.408 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.3 2022-06-27T12:03:04.408 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.3 2022-06-27T12:03:04.408 INFO:tasks.workunit.client.0.smithi195.stdout:v0.3 2022-06-27T12:03:04.410 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.3 2022-06-27T12:03:11.427 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 26% (78/295) Updating files: 27% (80/295) Updating files: 28% (83/295) Updating files: 29% (86/295) Updating files: 30% (89/295) Updating files: 31% (92/295) Updating files: 31% (94/295) Updating files: 32% (95/295) Updating files: 33% (98/295) Updating files: 34% (101/295) Updating files: 35% (104/295) Updating files: 36% (107/295) Updating files: 37% (110/295) Updating files: 38% (113/295) Updating files: 39% (116/295) Updating files: 40% (118/295) Updating files: 41% (121/295) Updating files: 42% (124/295) Updating files: 43% (127/295) Updating files: 44% (130/295) Updating files: 45% (133/295) Updating files: 45% (134/295) Updating files: 46% (136/295) Updating files: 47% (139/295) Updating files: 48% (142/295) Updating files: 49% (145/295) Updating files: 49% (147/295) Updating files: 50% (148/295) Updating files: 51% (151/295) Updating files: 52% (154/295) Updating files: 53% (157/295) Updating files: 54% (160/295) Updating files: 55% (163/295) Updating files: 55% (164/295) Updating files: 56% (166/295) Updating files: 57% (169/295) Updating files: 58% (172/295) Updating files: 59% (175/295) Updating files: 60% (177/295) Updating files: 61% (180/295) Updating files: 62% (183/295) Updating files: 63% (186/295) Updating files: 64% (189/295) Updating files: 65% (192/295) Updating files: 66% (195/295) Updating files: 67% (198/295) Updating files: 68% (201/295) Updating files: 69% (204/295) Updating files: 70% (207/295) Updating files: 71% (210/295) Updating files: 72% (213/295) Updating files: 73% (216/295) Updating files: 74% (219/295) Updating files: 75% (222/295) Updating files: 76% (225/295) Updating files: 77% (228/295) Updating files: 78% (231/295) Updating files: 79% (234/295) Updating files: 80% (236/295) Updating files: 81% (239/295) Updating files: 82% (242/295) Updating files: 83% (245/295) Updating files: 83% (247/295) Updating files: 84% (248/295) Updating files: 85% (251/295) Updating files: 86% (254/295) Updating files: 87% (257/295) Updating files: 88% (260/295) Updating files: 89% (263/295) Updating files: 90% (266/295) Updating files: 91% (269/295) Updating files: 92% (272/295) Updating files: 93% (275/295) Updating files: 94% (278/295) Updating files: 95% (281/295) Updating files: 96% (284/295) Updating files: 97% (287/295) Updating files: 98% (290/295) Updating files: 98% (292/295) Updating files: 99% (293/295) Updating files: 100% (295/295) Updating files: 100% (295/295), done. 2022-06-27T12:03:11.447 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at 2e986fe9ddf include ceph.spec in configure.ac 2022-06-27T12:03:11.451 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.3 2022-06-27T12:03:12.135 INFO:tasks.workunit.client.0.smithi195.stdout:v0.4 2022-06-27T12:03:12.135 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:03:12.136 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 4 -eq 48 ']' 2022-06-27T12:03:12.136 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.4 2022-06-27T12:03:12.137 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.4 2022-06-27T12:03:12.137 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.4 2022-06-27T12:03:13.121 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at 13b1bf7c492 kclient: initializing kaddr (merge fix) 2022-06-27T12:03:13.124 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.4 2022-06-27T12:03:13.389 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:03:13.391 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 5 -eq 48 ']' 2022-06-27T12:03:13.391 INFO:tasks.workunit.client.0.smithi195.stdout:v0.5 2022-06-27T12:03:13.392 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.5 2022-06-27T12:03:13.393 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.5 2022-06-27T12:03:13.393 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.5 2022-06-27T12:03:17.771 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:03:17.907 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:17.907+0000 7f5b581cf700 1 Processor -- start 2022-06-27T12:03:17.908 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:17.907+0000 7f5b581cf700 1 -- start start 2022-06-27T12:03:17.923 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:17.908+0000 7f5b581cf700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f5b50150950 0x7f5b50156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:17.924 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:17.908+0000 7f5b581cf700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f5b5006f620 0x7f5b5006d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:17.924 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:17.908+0000 7f5b581cf700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b500c76b0 con 0x7f5b50150580 2022-06-27T12:03:17.925 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:17.908+0000 7f5b581cf700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b50157430 con 0x7f5b5006f620 2022-06-27T12:03:17.925 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:17.908+0000 7f5b581cf700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b500c8250 con 0x7f5b50150950 2022-06-27T12:03:17.926 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:17.908+0000 7f5b5676c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f5b5006f620 0x7f5b5006d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:17.926 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:17.908+0000 7f5b5676c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f5b5006f620 0x7f5b5006d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47084/0 (socket says 172.21.15.195:47084) 2022-06-27T12:03:17.926 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:17.908+0000 7f5b5676c700 1 -- 172.21.15.195:0/2128550128 learned_addr learned my addr 172.21.15.195:0/2128550128 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:03:17.927 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:17.908+0000 7f5b5576a700 1 --2- 172.21.15.195:0/2128550128 >> v2:172.21.15.202:3300/0 conn(0x7f5b50150950 0x7f5b50156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:17.927 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:17.908+0000 7f5b5676c700 1 -- 172.21.15.195:0/2128550128 >> v2:172.21.15.202:3300/0 conn(0x7f5b50150950 msgr2=0x7f5b50156ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:03:17.928 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:17.908+0000 7f5b5676c700 1 --2- 172.21.15.195:0/2128550128 >> v2:172.21.15.202:3300/0 conn(0x7f5b50150950 0x7f5b50156ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:17.928 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:17.908+0000 7f5b5676c700 1 -- 172.21.15.195:0/2128550128 >> v1:172.21.15.202:6789/0 conn(0x7f5b50150580 legacy=0x7f5b500c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:17.928 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:17.908+0000 7f5b5676c700 1 -- 172.21.15.195:0/2128550128 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b500bad70 con 0x7f5b5006f620 2022-06-27T12:03:17.929 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:17.908+0000 7f5b5676c700 1 --2- 172.21.15.195:0/2128550128 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5b5006f620 0x7f5b5006d6a0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f5b5006ea70 tx=0x7f5b48009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5f5ba4f576257e9 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:03:17.930 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:17.909+0000 7f5b54f69700 1 -- 172.21.15.195:0/2128550128 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b4800e070 con 0x7f5b5006f620 2022-06-27T12:03:17.930 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:17.909+0000 7f5b54f69700 1 -- 172.21.15.195:0/2128550128 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b4800ba50 con 0x7f5b5006f620 2022-06-27T12:03:17.932 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:17.909+0000 7f5b54f69700 1 -- 172.21.15.195:0/2128550128 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b480092e0 con 0x7f5b5006f620 2022-06-27T12:03:17.932 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:17.909+0000 7f5b581cf700 1 -- 172.21.15.195:0/2128550128 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5b5006f620 msgr2=0x7f5b5006d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:17.933 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:17.909+0000 7f5b581cf700 1 --2- 172.21.15.195:0/2128550128 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5b5006f620 0x7f5b5006d6a0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f5b5006ea70 tx=0x7f5b48009840 comp rx=0 tx=0).stop 2022-06-27T12:03:17.934 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:17.909+0000 7f5b581cf700 1 -- 172.21.15.195:0/2128550128 shutdown_connections 2022-06-27T12:03:17.934 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:17.909+0000 7f5b581cf700 1 --2- 172.21.15.195:0/2128550128 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5b5006f620 0x7f5b5006d6a0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:17.935 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:17.909+0000 7f5b581cf700 1 --2- 172.21.15.195:0/2128550128 >> v2:172.21.15.202:3300/0 conn(0x7f5b50150950 0x7f5b50156ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:17.935 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:17.909+0000 7f5b581cf700 1 -- 172.21.15.195:0/2128550128 >> 172.21.15.195:0/2128550128 conn(0x7f5b50063af0 msgr2=0x7f5b50063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:03:17.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:17.909+0000 7f5b581cf700 1 -- 172.21.15.195:0/2128550128 shutdown_connections 2022-06-27T12:03:17.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:17.909+0000 7f5b581cf700 1 -- 172.21.15.195:0/2128550128 wait complete. 2022-06-27T12:03:17.937 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:17.910+0000 7f5b581cf700 1 Processor -- start 2022-06-27T12:03:17.937 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:17.910+0000 7f5b581cf700 1 -- start start 2022-06-27T12:03:17.938 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:17.911+0000 7f5b581cf700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5b5006f620 0x7f5b500cb5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:17.938 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:17.911+0000 7f5b581cf700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5b50150580 0x7f5b500c5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:17.939 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:17.911+0000 7f5b581cf700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5b50150950 0x7f5b500bcff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:17.939 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:17.911+0000 7f5b581cf700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b500bc3d0 con 0x7f5b50150580 2022-06-27T12:03:17.940 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:17.911+0000 7f5b581cf700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b500bbf40 con 0x7f5b50150950 2022-06-27T12:03:17.941 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:17.911+0000 7f5b581cf700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b5016a820 con 0x7f5b5006f620 2022-06-27T12:03:17.942 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:17.911+0000 7f5b5576a700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5b50150580 0x7f5b500c5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:17.943 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:17.911+0000 7f5b5576a700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5b50150580 0x7f5b500c5240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47088/0 (socket says 172.21.15.195:47088) 2022-06-27T12:03:17.943 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:17.911+0000 7f5b5576a700 1 -- 172.21.15.195:0/1641769660 learned_addr learned my addr 172.21.15.195:0/1641769660 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:03:17.943 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:17.911+0000 7f5b5676c700 1 --2- 172.21.15.195:0/1641769660 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5b50150950 0x7f5b500bcff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:17.944 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:17.911+0000 7f5b55f6b700 1 --2- 172.21.15.195:0/1641769660 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5b5006f620 0x7f5b500cb5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:17.944 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:17.912+0000 7f5b5576a700 1 -- 172.21.15.195:0/1641769660 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5b5006f620 msgr2=0x7f5b500cb5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:17.945 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:17.912+0000 7f5b5576a700 1 --2- 172.21.15.195:0/1641769660 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5b5006f620 0x7f5b500cb5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:17.945 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:17.912+0000 7f5b5576a700 1 -- 172.21.15.195:0/1641769660 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5b50150950 msgr2=0x7f5b500bcff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:17.946 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:17.912+0000 7f5b5576a700 1 --2- 172.21.15.195:0/1641769660 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5b50150950 0x7f5b500bcff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:17.946 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:17.912+0000 7f5b5576a700 1 -- 172.21.15.195:0/1641769660 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b48009ae0 con 0x7f5b50150580 2022-06-27T12:03:17.947 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:17.912+0000 7f5b5576a700 1 --2- 172.21.15.195:0/1641769660 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5b50150580 0x7f5b500c5240 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f5b4000d890 tx=0x7f5b40008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:03:17.947 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:17.912+0000 7f5b3dffb700 1 -- 172.21.15.195:0/1641769660 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b40014620 con 0x7f5b50150580 2022-06-27T12:03:17.947 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:17.912+0000 7f5b3dffb700 1 -- 172.21.15.195:0/1641769660 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b40014780 con 0x7f5b50150580 2022-06-27T12:03:17.948 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:17.912+0000 7f5b581cf700 1 -- 172.21.15.195:0/1641769660 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b500bb330 con 0x7f5b50150580 2022-06-27T12:03:17.948 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:17.912+0000 7f5b3dffb700 1 -- 172.21.15.195:0/1641769660 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b40014b40 con 0x7f5b50150580 2022-06-27T12:03:17.948 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:17.912+0000 7f5b581cf700 1 -- 172.21.15.195:0/1641769660 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b500bb8f0 con 0x7f5b50150580 2022-06-27T12:03:17.948 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:17.913+0000 7f5b581cf700 1 -- 172.21.15.195:0/1641769660 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b500722c0 con 0x7f5b50150580 2022-06-27T12:03:17.949 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:17.914+0000 7f5b3dffb700 1 -- 172.21.15.195:0/1641769660 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f5b40024070 con 0x7f5b50150580 2022-06-27T12:03:17.949 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:17.914+0000 7f5b3dffb700 1 --2- 172.21.15.195:0/1641769660 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5b30063d30 0x7f5b300661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:17.949 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:17.914+0000 7f5b3dffb700 1 -- 172.21.15.195:0/1641769660 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6804+0+0 (secure 0 0 0) 0x7f5b400811b0 con 0x7f5b50150580 2022-06-27T12:03:17.949 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:17.914+0000 7f5b55f6b700 1 --2- 172.21.15.195:0/1641769660 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5b30063d30 0x7f5b300661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:17.950 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:17.914+0000 7f5b55f6b700 1 --2- 172.21.15.195:0/1641769660 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5b30063d30 0x7f5b300661e0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f5b4c001880 tx=0x7f5b4c009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:03:17.950 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:17.918+0000 7f5b3dffb700 1 -- 172.21.15.195:0/1641769660 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f5b4004fab0 con 0x7f5b50150580 2022-06-27T12:03:18.156 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.154+0000 7f5b581cf700 1 -- 172.21.15.195:0/1641769660 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5b500bc080 con 0x7f5b50150580 2022-06-27T12:03:18.157 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.155+0000 7f5b3dffb700 1 -- 172.21.15.195:0/1641769660 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+6125 (secure 0 0 0) 0x7f5b40018090 con 0x7f5b50150580 2022-06-27T12:03:18.158 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:03:18.158 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":5818,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/3399027527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3399027527},{"type":"v1","addr":"172.21.15.202:6833","nonce":3399027527}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:02:00.272060+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5273":{"gid":5273,"name":"d","rank":0,"incarnation":49,"state":"up:active","state_seq":42,"addr":"172.21.15.202:6835/417105338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":417105338},{"type":"v1","addr":"172.21.15.202:6835","nonce":417105338}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:03:18.161 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.157+0000 7f5b581cf700 1 -- 172.21.15.195:0/1641769660 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5b30063d30 msgr2=0x7f5b300661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:18.161 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.157+0000 7f5b581cf700 1 --2- 172.21.15.195:0/1641769660 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5b30063d30 0x7f5b300661e0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f5b4c001880 tx=0x7f5b4c009040 comp rx=0 tx=0).stop 2022-06-27T12:03:18.162 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.157+0000 7f5b581cf700 1 -- 172.21.15.195:0/1641769660 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5b50150580 msgr2=0x7f5b500c5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:18.163 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.157+0000 7f5b581cf700 1 --2- 172.21.15.195:0/1641769660 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5b50150580 0x7f5b500c5240 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f5b4000d890 tx=0x7f5b40008040 comp rx=0 tx=0).stop 2022-06-27T12:03:18.163 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.157+0000 7f5b581cf700 1 -- 172.21.15.195:0/1641769660 shutdown_connections 2022-06-27T12:03:18.163 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.157+0000 7f5b581cf700 1 --2- 172.21.15.195:0/1641769660 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5b30063d30 0x7f5b300661e0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:18.164 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.157+0000 7f5b581cf700 1 --2- 172.21.15.195:0/1641769660 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5b50150950 0x7f5b500bcff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:18.164 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.157+0000 7f5b581cf700 1 --2- 172.21.15.195:0/1641769660 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5b5006f620 0x7f5b500cb5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:18.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.157+0000 7f5b581cf700 1 --2- 172.21.15.195:0/1641769660 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5b50150580 0x7f5b500c5240 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:18.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.157+0000 7f5b581cf700 1 -- 172.21.15.195:0/1641769660 >> 172.21.15.195:0/1641769660 conn(0x7f5b50063af0 msgr2=0x7f5b5014ed50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:03:18.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.157+0000 7f5b581cf700 1 -- 172.21.15.195:0/1641769660 shutdown_connections 2022-06-27T12:03:18.166 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.157+0000 7f5b581cf700 1 -- 172.21.15.195:0/1641769660 wait complete. 2022-06-27T12:03:18.166 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 54 2022-06-27T12:03:18.170 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.d (rank=0) 2022-06-27T12:03:18.171 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2022-06-27T12:03:18.172 INFO:teuthology.orchestra.run:waiting for 300 2022-06-27T12:03:18.186 INFO:tasks.ceph.mds.d:Stopped 2022-06-27T12:03:18.186 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:03:18.343 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.332+0000 7f239ff7c700 1 Processor -- start 2022-06-27T12:03:18.344 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.333+0000 7f239ff7c700 1 -- start start 2022-06-27T12:03:18.345 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.333+0000 7f239ff7c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f2398150950 0x7f2398156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:18.345 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.333+0000 7f239ff7c700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f239806f620 0x7f239806d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:18.346 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.333+0000 7f239ff7c700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23980c76b0 con 0x7f2398150580 2022-06-27T12:03:18.346 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.333+0000 7f239ff7c700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2398157430 con 0x7f2398150950 2022-06-27T12:03:18.346 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.333+0000 7f239ff7c700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23980c8250 con 0x7f239806f620 2022-06-27T12:03:18.347 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.334+0000 7f239d517700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f2398150950 0x7f2398156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:18.347 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.334+0000 7f239d517700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f2398150950 0x7f2398156ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47096/0 (socket says 172.21.15.195:47096) 2022-06-27T12:03:18.348 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.334+0000 7f239dd18700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f2398150580 0x7f23980c7b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:53282/0 (socket says 172.21.15.195:53282) 2022-06-27T12:03:18.348 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.334+0000 7f239d517700 1 -- 172.21.15.195:0/3481123573 learned_addr learned my addr 172.21.15.195:0/3481123573 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:03:18.349 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.334+0000 7f239e519700 1 --2- 172.21.15.195:0/3481123573 >> v2:172.21.15.202:3300/0 conn(0x7f239806f620 0x7f239806d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:18.349 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.334+0000 7f239d517700 1 -- 172.21.15.195:0/3481123573 >> v2:172.21.15.202:3300/0 conn(0x7f239806f620 msgr2=0x7f239806d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:03:18.350 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.334+0000 7f239d517700 1 --2- 172.21.15.195:0/3481123573 >> v2:172.21.15.202:3300/0 conn(0x7f239806f620 0x7f239806d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:18.351 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.334+0000 7f239d517700 1 -- 172.21.15.195:0/3481123573 >> v1:172.21.15.195:6789/0 conn(0x7f2398150580 legacy=0x7f23980c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:18.351 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.334+0000 7f239d517700 1 -- 172.21.15.195:0/3481123573 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23980bad70 con 0x7f2398150950 2022-06-27T12:03:18.352 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.334+0000 7f239d517700 1 --2- 172.21.15.195:0/3481123573 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2398150950 0x7f2398156ef0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f23980c4f70 tx=0x7f2394009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efa37085e7d92f9 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:03:18.352 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.334+0000 7f239cd16700 1 -- 172.21.15.195:0/3481123573 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f239400b640 con 0x7f2398150950 2022-06-27T12:03:18.353 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.334+0000 7f239cd16700 1 -- 172.21.15.195:0/3481123573 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f239400b7a0 con 0x7f2398150950 2022-06-27T12:03:18.353 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.334+0000 7f239cd16700 1 -- 172.21.15.195:0/3481123573 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f239400bb40 con 0x7f2398150950 2022-06-27T12:03:18.354 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.335+0000 7f239ff7c700 1 -- 172.21.15.195:0/3481123573 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2398150950 msgr2=0x7f2398156ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:18.354 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.335+0000 7f239ff7c700 1 --2- 172.21.15.195:0/3481123573 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2398150950 0x7f2398156ef0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f23980c4f70 tx=0x7f2394009120 comp rx=0 tx=0).stop 2022-06-27T12:03:18.355 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.335+0000 7f239ff7c700 1 -- 172.21.15.195:0/3481123573 shutdown_connections 2022-06-27T12:03:18.355 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.335+0000 7f239ff7c700 1 --2- 172.21.15.195:0/3481123573 >> v2:172.21.15.202:3300/0 conn(0x7f239806f620 0x7f239806d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:18.356 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.335+0000 7f239ff7c700 1 --2- 172.21.15.195:0/3481123573 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2398150950 0x7f2398156ef0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:18.357 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.335+0000 7f239ff7c700 1 -- 172.21.15.195:0/3481123573 >> 172.21.15.195:0/3481123573 conn(0x7f2398063af0 msgr2=0x7f2398063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:03:18.357 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.335+0000 7f239ff7c700 1 -- 172.21.15.195:0/3481123573 shutdown_connections 2022-06-27T12:03:18.358 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.335+0000 7f239ff7c700 1 -- 172.21.15.195:0/3481123573 wait complete. 2022-06-27T12:03:18.358 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.336+0000 7f239ff7c700 1 Processor -- start 2022-06-27T12:03:18.359 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.336+0000 7f239ff7c700 1 -- start start 2022-06-27T12:03:18.360 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.336+0000 7f239ff7c700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f239806f620 0x7f23980c9a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:18.360 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.337+0000 7f239ff7c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2398150580 0x7f239814b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:18.362 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.337+0000 7f239ff7c700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2398150950 0x7f239814b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:18.362 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.337+0000 7f239ff7c700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23980c8900 con 0x7f2398150580 2022-06-27T12:03:18.363 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.337+0000 7f239ff7c700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f239816a910 con 0x7f2398150950 2022-06-27T12:03:18.363 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.337+0000 7f239ff7c700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23980bd570 con 0x7f239806f620 2022-06-27T12:03:18.363 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.337+0000 7f239d517700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2398150580 0x7f239814b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:18.364 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.337+0000 7f239d517700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2398150580 0x7f239814b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47102/0 (socket says 172.21.15.195:47102) 2022-06-27T12:03:18.364 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.337+0000 7f239d517700 1 -- 172.21.15.195:0/4053815874 learned_addr learned my addr 172.21.15.195:0/4053815874 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:03:18.365 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.337+0000 7f239e519700 1 --2- 172.21.15.195:0/4053815874 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2398150950 0x7f239814b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:18.365 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.337+0000 7f239dd18700 1 --2- 172.21.15.195:0/4053815874 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f239806f620 0x7f23980c9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:18.365 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.337+0000 7f239d517700 1 -- 172.21.15.195:0/4053815874 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f239806f620 msgr2=0x7f23980c9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:18.366 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.337+0000 7f239d517700 1 --2- 172.21.15.195:0/4053815874 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f239806f620 0x7f23980c9a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:18.366 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.337+0000 7f239d517700 1 -- 172.21.15.195:0/4053815874 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2398150950 msgr2=0x7f239814b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:18.367 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.337+0000 7f239d517700 1 --2- 172.21.15.195:0/4053815874 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2398150950 0x7f239814b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:18.367 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.337+0000 7f239d517700 1 -- 172.21.15.195:0/4053815874 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23940093a0 con 0x7f2398150580 2022-06-27T12:03:18.367 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.337+0000 7f239e519700 1 --2- 172.21.15.195:0/4053815874 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2398150950 0x7f239814b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T12:03:18.368 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.337+0000 7f239d517700 1 --2- 172.21.15.195:0/4053815874 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2398150580 0x7f239814b210 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f2394004370 tx=0x7f2394017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:03:18.369 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.337+0000 7f2385ffb700 1 -- 172.21.15.195:0/4053815874 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2394003730 con 0x7f2398150580 2022-06-27T12:03:18.369 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.337+0000 7f2385ffb700 1 -- 172.21.15.195:0/4053815874 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2394003890 con 0x7f2398150580 2022-06-27T12:03:18.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.337+0000 7f239ff7c700 1 -- 172.21.15.195:0/4053815874 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f239814bc90 con 0x7f2398150580 2022-06-27T12:03:18.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.337+0000 7f2385ffb700 1 -- 172.21.15.195:0/4053815874 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2394003c50 con 0x7f2398150580 2022-06-27T12:03:18.372 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.338+0000 7f239ff7c700 1 -- 172.21.15.195:0/4053815874 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f239806dc40 con 0x7f2398150580 2022-06-27T12:03:18.372 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.339+0000 7f239ff7c700 1 -- 172.21.15.195:0/4053815874 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f239806daa0 con 0x7f2398150580 2022-06-27T12:03:18.373 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.339+0000 7f2385ffb700 1 -- 172.21.15.195:0/4053815874 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f239402b030 con 0x7f2398150580 2022-06-27T12:03:18.373 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.339+0000 7f2385ffb700 1 --2- 172.21.15.195:0/4053815874 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2378063d00 0x7f23780661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:18.373 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.339+0000 7f2385ffb700 1 -- 172.21.15.195:0/4053815874 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6804+0+0 (secure 0 0 0) 0x7f23940820f0 con 0x7f2398150580 2022-06-27T12:03:18.373 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.339+0000 7f239dd18700 1 --2- 172.21.15.195:0/4053815874 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2378063d00 0x7f23780661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:18.373 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.340+0000 7f239dd18700 1 --2- 172.21.15.195:0/4053815874 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2378063d00 0x7f23780661b0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f2388001640 tx=0x7f2388009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:03:18.374 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.344+0000 7f2385ffb700 1 -- 172.21.15.195:0/4053815874 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f239806daa0 con 0x7f2398150580 2022-06-27T12:03:18.583 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.573+0000 7f239ff7c700 1 -- 172.21.15.195:0/4053815874 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f239814bc90 con 0x7f2398150580 2022-06-27T12:03:18.583 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.574+0000 7f2385ffb700 1 -- 172.21.15.195:0/4053815874 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+6125 (secure 0 0 0) 0x7f239814bc90 con 0x7f2398150580 2022-06-27T12:03:18.584 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:03:18.585 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":5818,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/3399027527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3399027527},{"type":"v1","addr":"172.21.15.202:6833","nonce":3399027527}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:02:00.272060+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5273":{"gid":5273,"name":"d","rank":0,"incarnation":49,"state":"up:active","state_seq":42,"addr":"172.21.15.202:6835/417105338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":417105338},{"type":"v1","addr":"172.21.15.202:6835","nonce":417105338}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:03:18.592 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.575+0000 7f239ff7c700 1 -- 172.21.15.195:0/4053815874 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2378063d00 msgr2=0x7f23780661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:18.592 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.575+0000 7f239ff7c700 1 --2- 172.21.15.195:0/4053815874 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2378063d00 0x7f23780661b0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f2388001640 tx=0x7f2388009040 comp rx=0 tx=0).stop 2022-06-27T12:03:18.593 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.575+0000 7f239ff7c700 1 -- 172.21.15.195:0/4053815874 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2398150580 msgr2=0x7f239814b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:18.593 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.575+0000 7f239ff7c700 1 --2- 172.21.15.195:0/4053815874 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2398150580 0x7f239814b210 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f2394004370 tx=0x7f2394017000 comp rx=0 tx=0).stop 2022-06-27T12:03:18.594 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.576+0000 7f239ff7c700 1 -- 172.21.15.195:0/4053815874 shutdown_connections 2022-06-27T12:03:18.594 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.576+0000 7f239ff7c700 1 --2- 172.21.15.195:0/4053815874 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2378063d00 0x7f23780661b0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:18.595 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.576+0000 7f239ff7c700 1 --2- 172.21.15.195:0/4053815874 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2398150950 0x7f239814b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:18.595 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.576+0000 7f239ff7c700 1 --2- 172.21.15.195:0/4053815874 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f239806f620 0x7f23980c9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:18.595 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.576+0000 7f239ff7c700 1 --2- 172.21.15.195:0/4053815874 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2398150580 0x7f239814b210 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:18.596 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.576+0000 7f239ff7c700 1 -- 172.21.15.195:0/4053815874 >> 172.21.15.195:0/4053815874 conn(0x7f2398063af0 msgr2=0x7f239814f800 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:03:18.596 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.576+0000 7f239ff7c700 1 -- 172.21.15.195:0/4053815874 shutdown_connections 2022-06-27T12:03:18.597 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:18.576+0000 7f239ff7c700 1 -- 172.21.15.195:0/4053815874 wait complete. 2022-06-27T12:03:18.597 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 54 2022-06-27T12:03:18.598 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-06-27T12:03:20.600 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:03:20.736 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.733+0000 7f21fc5cc700 1 Processor -- start 2022-06-27T12:03:20.736 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.734+0000 7f21fc5cc700 1 -- start start 2022-06-27T12:03:20.737 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.734+0000 7f21fc5cc700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f21f4150580 0x7f21f4156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:20.739 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.734+0000 7f21fc5cc700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f21f406f620 0x7f21f406d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:20.741 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.735+0000 7f21fc5cc700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21f40c76b0 con 0x7f21f4150950 2022-06-27T12:03:20.742 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.735+0000 7f21fc5cc700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21f4157430 con 0x7f21f406f620 2022-06-27T12:03:20.743 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.735+0000 7f21fc5cc700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21f40c8250 con 0x7f21f4150580 2022-06-27T12:03:20.743 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.735+0000 7f21fab69700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f21f406f620 0x7f21f406d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:20.744 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.735+0000 7f21f9b67700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f21f4150950 0x7f21f40c7b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:53310/0 (socket says 172.21.15.195:53310) 2022-06-27T12:03:20.745 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.735+0000 7f21f9b67700 1 -- 172.21.15.195:0/3008777946 learned_addr learned my addr 172.21.15.195:0/3008777946 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:03:20.746 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.735+0000 7f21fa368700 1 --2- 172.21.15.195:0/3008777946 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f21f4150580 0x7f21f4156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:20.754 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.735+0000 7f21fab69700 1 -- 172.21.15.195:0/3008777946 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f21f4150580 msgr2=0x7f21f4156ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:03:20.755 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.735+0000 7f21fab69700 1 --2- 172.21.15.195:0/3008777946 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f21f4150580 0x7f21f4156ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:20.755 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.735+0000 7f21fab69700 1 -- 172.21.15.195:0/3008777946 >> v1:172.21.15.195:6789/0 conn(0x7f21f4150950 legacy=0x7f21f40c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:20.755 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.735+0000 7f21f9366700 1 -- 172.21.15.195:0/3008777946 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4160345207 0 0) 0x7f21f40c76b0 con 0x7f21f4150950 2022-06-27T12:03:20.755 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.735+0000 7f21fab69700 1 -- 172.21.15.195:0/3008777946 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21f40bad70 con 0x7f21f406f620 2022-06-27T12:03:20.756 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.735+0000 7f21fa368700 1 --2- 172.21.15.195:0/3008777946 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f21f4150580 0x7f21f4156ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:03:20.757 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.735+0000 7f21fab69700 1 --2- 172.21.15.195:0/3008777946 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f21f406f620 0x7f21f406d6a0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f21ec009e90 tx=0x7f21ec009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59bd02c5290f403d server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:03:20.758 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.735+0000 7f21f9366700 1 -- 172.21.15.195:0/3008777946 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21ec00e070 con 0x7f21f406f620 2022-06-27T12:03:20.758 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.735+0000 7f21f9366700 1 -- 172.21.15.195:0/3008777946 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21ec00bbf0 con 0x7f21f406f620 2022-06-27T12:03:20.759 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.735+0000 7f21f9366700 1 -- 172.21.15.195:0/3008777946 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21ec012430 con 0x7f21f406f620 2022-06-27T12:03:20.759 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.736+0000 7f21fc5cc700 1 -- 172.21.15.195:0/3008777946 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f21f406f620 msgr2=0x7f21f406d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:20.759 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.736+0000 7f21fc5cc700 1 --2- 172.21.15.195:0/3008777946 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f21f406f620 0x7f21f406d6a0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f21ec009e90 tx=0x7f21ec009780 comp rx=0 tx=0).stop 2022-06-27T12:03:20.759 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.736+0000 7f21fc5cc700 1 -- 172.21.15.195:0/3008777946 shutdown_connections 2022-06-27T12:03:20.760 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.736+0000 7f21fc5cc700 1 --2- 172.21.15.195:0/3008777946 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f21f406f620 0x7f21f406d6a0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:20.760 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.736+0000 7f21fc5cc700 1 --2- 172.21.15.195:0/3008777946 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f21f4150580 0x7f21f4156ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:20.761 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.736+0000 7f21fc5cc700 1 -- 172.21.15.195:0/3008777946 >> 172.21.15.195:0/3008777946 conn(0x7f21f4063af0 msgr2=0x7f21f4063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:03:20.763 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.736+0000 7f21fc5cc700 1 -- 172.21.15.195:0/3008777946 shutdown_connections 2022-06-27T12:03:20.763 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.736+0000 7f21fc5cc700 1 -- 172.21.15.195:0/3008777946 wait complete. 2022-06-27T12:03:20.764 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.737+0000 7f21fc5cc700 1 Processor -- start 2022-06-27T12:03:20.764 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.737+0000 7f21fc5cc700 1 -- start start 2022-06-27T12:03:20.764 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.738+0000 7f21fc5cc700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f21f406f620 0x7f21f416a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:20.765 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.738+0000 7f21fc5cc700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f21f4150580 0x7f21f40c21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:20.765 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.738+0000 7f21fc5cc700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f21f4150950 0x7f21f414f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:20.765 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.738+0000 7f21fc5cc700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21f416a290 con 0x7f21f4150580 2022-06-27T12:03:20.766 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.738+0000 7f21fc5cc700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21f41510e0 con 0x7f21f406f620 2022-06-27T12:03:20.766 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.738+0000 7f21fc5cc700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21f4151380 con 0x7f21f4150950 2022-06-27T12:03:20.767 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.738+0000 7f21f9b67700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f21f4150580 0x7f21f40c21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:20.767 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.738+0000 7f21f9b67700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f21f4150580 0x7f21f40c21d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47126/0 (socket says 172.21.15.195:47126) 2022-06-27T12:03:20.768 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.738+0000 7f21f9b67700 1 -- 172.21.15.195:0/2075672775 learned_addr learned my addr 172.21.15.195:0/2075672775 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:03:20.768 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.738+0000 7f21fab69700 1 --2- 172.21.15.195:0/2075672775 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f21f4150950 0x7f21f414f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:20.768 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.738+0000 7f21fa368700 1 --2- 172.21.15.195:0/2075672775 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f21f406f620 0x7f21f416a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:20.768 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.738+0000 7f21f9b67700 1 -- 172.21.15.195:0/2075672775 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f21f4150950 msgr2=0x7f21f414f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:20.769 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.738+0000 7f21f9b67700 1 --2- 172.21.15.195:0/2075672775 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f21f4150950 0x7f21f414f340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:20.769 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.738+0000 7f21f9b67700 1 -- 172.21.15.195:0/2075672775 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f21f406f620 msgr2=0x7f21f416a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:20.769 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.738+0000 7f21f9b67700 1 --2- 172.21.15.195:0/2075672775 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f21f406f620 0x7f21f416a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:20.769 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.738+0000 7f21f9b67700 1 -- 172.21.15.195:0/2075672775 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21ec009a20 con 0x7f21f4150580 2022-06-27T12:03:20.773 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.738+0000 7f21f9b67700 1 --2- 172.21.15.195:0/2075672775 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f21f4150580 0x7f21f40c21d0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f21f000c590 tx=0x7f21f0007c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:03:20.773 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.738+0000 7f21e27fc700 1 -- 172.21.15.195:0/2075672775 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21f0007360 con 0x7f21f4150580 2022-06-27T12:03:20.773 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.738+0000 7f21e27fc700 1 -- 172.21.15.195:0/2075672775 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21f0007500 con 0x7f21f4150580 2022-06-27T12:03:20.774 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.738+0000 7f21fc5cc700 1 -- 172.21.15.195:0/2075672775 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21f4151660 con 0x7f21f4150580 2022-06-27T12:03:20.774 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.738+0000 7f21e27fc700 1 -- 172.21.15.195:0/2075672775 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21f0004940 con 0x7f21f4150580 2022-06-27T12:03:20.774 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.739+0000 7f21fc5cc700 1 -- 172.21.15.195:0/2075672775 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21f4150de0 con 0x7f21f4150580 2022-06-27T12:03:20.774 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.740+0000 7f21fc5cc700 1 -- 172.21.15.195:0/2075672775 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21f406eba0 con 0x7f21f4150580 2022-06-27T12:03:20.774 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.740+0000 7f21e27fc700 1 -- 172.21.15.195:0/2075672775 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f21f0010030 con 0x7f21f4150580 2022-06-27T12:03:20.775 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.740+0000 7f21e27fc700 1 --2- 172.21.15.195:0/2075672775 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f21d4063d00 0x7f21d40661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:20.775 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.740+0000 7f21e27fc700 1 -- 172.21.15.195:0/2075672775 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6804+0+0 (secure 0 0 0) 0x7f21f0080760 con 0x7f21f4150580 2022-06-27T12:03:20.775 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.740+0000 7f21fa368700 1 --2- 172.21.15.195:0/2075672775 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f21d4063d00 0x7f21d40661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:20.775 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.741+0000 7f21fa368700 1 --2- 172.21.15.195:0/2075672775 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f21d4063d00 0x7f21d40661b0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f21e4006dc0 tx=0x7f21e4008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:03:20.776 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.745+0000 7f21e27fc700 1 -- 172.21.15.195:0/2075672775 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f21f004f0e0 con 0x7f21f4150580 2022-06-27T12:03:20.972 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.972+0000 7f21fc5cc700 1 -- 172.21.15.195:0/2075672775 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f21f40c6ca0 con 0x7f21f4150580 2022-06-27T12:03:20.973 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.972+0000 7f21e27fc700 1 -- 172.21.15.195:0/2075672775 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+6125 (secure 0 0 0) 0x7f21f0052700 con 0x7f21f4150580 2022-06-27T12:03:20.974 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:03:20.974 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":5818,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/3399027527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3399027527},{"type":"v1","addr":"172.21.15.202:6833","nonce":3399027527}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:02:00.272060+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5273":{"gid":5273,"name":"d","rank":0,"incarnation":49,"state":"up:active","state_seq":42,"addr":"172.21.15.202:6835/417105338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":417105338},{"type":"v1","addr":"172.21.15.202:6835","nonce":417105338}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:03:20.991 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.974+0000 7f21fc5cc700 1 -- 172.21.15.195:0/2075672775 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f21d4063d00 msgr2=0x7f21d40661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:20.991 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.974+0000 7f21fc5cc700 1 --2- 172.21.15.195:0/2075672775 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f21d4063d00 0x7f21d40661b0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f21e4006dc0 tx=0x7f21e4008040 comp rx=0 tx=0).stop 2022-06-27T12:03:20.992 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.974+0000 7f21fc5cc700 1 -- 172.21.15.195:0/2075672775 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f21f4150580 msgr2=0x7f21f40c21d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:20.992 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.974+0000 7f21fc5cc700 1 --2- 172.21.15.195:0/2075672775 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f21f4150580 0x7f21f40c21d0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f21f000c590 tx=0x7f21f0007c80 comp rx=0 tx=0).stop 2022-06-27T12:03:20.992 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.974+0000 7f21fc5cc700 1 -- 172.21.15.195:0/2075672775 shutdown_connections 2022-06-27T12:03:20.993 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.974+0000 7f21fc5cc700 1 --2- 172.21.15.195:0/2075672775 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f21d4063d00 0x7f21d40661b0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:20.993 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.974+0000 7f21fc5cc700 1 --2- 172.21.15.195:0/2075672775 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f21f4150950 0x7f21f414f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:20.996 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.974+0000 7f21fc5cc700 1 --2- 172.21.15.195:0/2075672775 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f21f406f620 0x7f21f416a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:20.997 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.974+0000 7f21fc5cc700 1 --2- 172.21.15.195:0/2075672775 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f21f4150580 0x7f21f40c21d0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:20.997 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.974+0000 7f21fc5cc700 1 -- 172.21.15.195:0/2075672775 >> 172.21.15.195:0/2075672775 conn(0x7f21f4063af0 msgr2=0x7f21f414d7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:03:20.998 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.974+0000 7f21fc5cc700 1 -- 172.21.15.195:0/2075672775 shutdown_connections 2022-06-27T12:03:20.998 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:20.974+0000 7f21fc5cc700 1 -- 172.21.15.195:0/2075672775 wait complete. 2022-06-27T12:03:20.999 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 54 2022-06-27T12:03:21.000 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-06-27T12:03:23.002 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:03:23.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.136+0000 7fb280131700 1 Processor -- start 2022-06-27T12:03:23.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.137+0000 7fb280131700 1 -- start start 2022-06-27T12:03:23.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.137+0000 7fb280131700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fb27806e850 0x7fb2781570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:23.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.137+0000 7fb280131700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fb27806fef0 0x7fb278069de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:23.155 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.137+0000 7fb280131700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb27806fba0 con 0x7fb2780c34a0 2022-06-27T12:03:23.155 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.137+0000 7fb280131700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2781575e0 con 0x7fb27806e850 2022-06-27T12:03:23.156 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.137+0000 7fb280131700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb27806f5e0 con 0x7fb27806fef0 2022-06-27T12:03:23.156 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.138+0000 7fb27decd700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fb27806e850 0x7fb2781570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:23.157 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.138+0000 7fb27decd700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fb27806e850 0x7fb2781570a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47134/0 (socket says 172.21.15.195:47134) 2022-06-27T12:03:23.157 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.138+0000 7fb27decd700 1 -- 172.21.15.195:0/2499098450 learned_addr learned my addr 172.21.15.195:0/2499098450 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:03:23.158 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.138+0000 7fb27e6ce700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fb2780c34a0 0x7fb2780c2920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:53322/0 (socket says 172.21.15.195:53322) 2022-06-27T12:03:23.158 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.138+0000 7fb27d6cc700 1 --2- 172.21.15.195:0/2499098450 >> v2:172.21.15.202:3300/0 conn(0x7fb27806fef0 0x7fb278069de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:23.159 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.138+0000 7fb27decd700 1 -- 172.21.15.195:0/2499098450 >> v2:172.21.15.202:3300/0 conn(0x7fb27806fef0 msgr2=0x7fb278069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:03:23.159 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.138+0000 7fb27decd700 1 --2- 172.21.15.195:0/2499098450 >> v2:172.21.15.202:3300/0 conn(0x7fb27806fef0 0x7fb278069de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:23.160 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.138+0000 7fb27decd700 1 -- 172.21.15.195:0/2499098450 >> v1:172.21.15.195:6789/0 conn(0x7fb2780c34a0 legacy=0x7fb2780c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:23.160 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.138+0000 7fb27decd700 1 -- 172.21.15.195:0/2499098450 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb27806f8c0 con 0x7fb27806e850 2022-06-27T12:03:23.161 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.138+0000 7fb27decd700 1 --2- 172.21.15.195:0/2499098450 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb27806e850 0x7fb2781570a0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fb27806e600 tx=0x7fb268009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da4cbe9cdfbce0cb server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:03:23.161 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.138+0000 7fb27cecb700 1 -- 172.21.15.195:0/2499098450 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb26800b6d0 con 0x7fb27806e850 2022-06-27T12:03:23.162 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.138+0000 7fb27cecb700 1 -- 172.21.15.195:0/2499098450 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb26800b830 con 0x7fb27806e850 2022-06-27T12:03:23.162 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.138+0000 7fb27cecb700 1 -- 172.21.15.195:0/2499098450 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb26800bbd0 con 0x7fb27806e850 2022-06-27T12:03:23.163 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.139+0000 7fb280131700 1 -- 172.21.15.195:0/2499098450 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb27806e850 msgr2=0x7fb2781570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:23.163 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.139+0000 7fb280131700 1 --2- 172.21.15.195:0/2499098450 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb27806e850 0x7fb2781570a0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fb27806e600 tx=0x7fb268009120 comp rx=0 tx=0).stop 2022-06-27T12:03:23.164 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.139+0000 7fb280131700 1 -- 172.21.15.195:0/2499098450 shutdown_connections 2022-06-27T12:03:23.164 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.139+0000 7fb280131700 1 --2- 172.21.15.195:0/2499098450 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb27806e850 0x7fb2781570a0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:23.164 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.139+0000 7fb280131700 1 --2- 172.21.15.195:0/2499098450 >> v2:172.21.15.202:3300/0 conn(0x7fb27806fef0 0x7fb278069de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:23.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.139+0000 7fb280131700 1 -- 172.21.15.195:0/2499098450 >> 172.21.15.195:0/2499098450 conn(0x7fb2780c6350 msgr2=0x7fb2780c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:03:23.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.139+0000 7fb280131700 1 -- 172.21.15.195:0/2499098450 shutdown_connections 2022-06-27T12:03:23.166 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.139+0000 7fb280131700 1 -- 172.21.15.195:0/2499098450 wait complete. 2022-06-27T12:03:23.166 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.140+0000 7fb280131700 1 Processor -- start 2022-06-27T12:03:23.167 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.140+0000 7fb280131700 1 -- start start 2022-06-27T12:03:23.167 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.141+0000 7fb280131700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb27806e850 0x7fb278156480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:23.167 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.141+0000 7fb280131700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb2780c34a0 0x7fb27814f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:23.167 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.141+0000 7fb280131700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb27816a580 0x7fb278149d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:23.168 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.141+0000 7fb280131700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb27816aa50 con 0x7fb2780c34a0 2022-06-27T12:03:23.168 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.141+0000 7fb280131700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb278155ff0 con 0x7fb27806e850 2022-06-27T12:03:23.168 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.141+0000 7fb280131700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb278148a90 con 0x7fb27816a580 2022-06-27T12:03:23.168 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.141+0000 7fb27d6cc700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb2780c34a0 0x7fb27814f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:23.169 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.141+0000 7fb27d6cc700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb2780c34a0 0x7fb27814f5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47140/0 (socket says 172.21.15.195:47140) 2022-06-27T12:03:23.169 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.141+0000 7fb27d6cc700 1 -- 172.21.15.195:0/2047206723 learned_addr learned my addr 172.21.15.195:0/2047206723 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:03:23.170 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.141+0000 7fb27decd700 1 --2- 172.21.15.195:0/2047206723 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb27806e850 0x7fb278156480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:23.170 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.141+0000 7fb27d6cc700 1 -- 172.21.15.195:0/2047206723 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb27816a580 msgr2=0x7fb278149d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:23.170 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.141+0000 7fb27d6cc700 1 --2- 172.21.15.195:0/2047206723 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb27816a580 0x7fb278149d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:23.171 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.141+0000 7fb27d6cc700 1 -- 172.21.15.195:0/2047206723 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb27806e850 msgr2=0x7fb278156480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:23.171 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.141+0000 7fb27d6cc700 1 --2- 172.21.15.195:0/2047206723 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb27806e850 0x7fb278156480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:23.172 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.141+0000 7fb27d6cc700 1 -- 172.21.15.195:0/2047206723 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2680093a0 con 0x7fb2780c34a0 2022-06-27T12:03:23.173 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.141+0000 7fb27decd700 1 --2- 172.21.15.195:0/2047206723 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb27806e850 0x7fb278156480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T12:03:23.173 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.141+0000 7fb27d6cc700 1 --2- 172.21.15.195:0/2047206723 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb2780c34a0 0x7fb27814f5d0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fb27400d8c0 tx=0x7fb274008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:03:23.173 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.141+0000 7fb265ffb700 1 -- 172.21.15.195:0/2047206723 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb274009640 con 0x7fb2780c34a0 2022-06-27T12:03:23.174 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.141+0000 7fb265ffb700 1 -- 172.21.15.195:0/2047206723 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2740097a0 con 0x7fb2780c34a0 2022-06-27T12:03:23.174 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.142+0000 7fb265ffb700 1 -- 172.21.15.195:0/2047206723 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb274009b40 con 0x7fb2780c34a0 2022-06-27T12:03:23.175 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.142+0000 7fb280131700 1 -- 172.21.15.195:0/2047206723 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2781559a0 con 0x7fb2780c34a0 2022-06-27T12:03:23.175 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.142+0000 7fb280131700 1 -- 172.21.15.195:0/2047206723 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2781569c0 con 0x7fb2780c34a0 2022-06-27T12:03:23.176 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.143+0000 7fb280131700 1 -- 172.21.15.195:0/2047206723 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2780722c0 con 0x7fb2780c34a0 2022-06-27T12:03:23.177 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.143+0000 7fb265ffb700 1 -- 172.21.15.195:0/2047206723 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fb274024070 con 0x7fb2780c34a0 2022-06-27T12:03:23.177 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.143+0000 7fb265ffb700 1 --2- 172.21.15.195:0/2047206723 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb258063d30 0x7fb2580661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:23.177 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.143+0000 7fb265ffb700 1 -- 172.21.15.195:0/2047206723 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6804+0+0 (secure 0 0 0) 0x7fb2740811e0 con 0x7fb2780c34a0 2022-06-27T12:03:23.178 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.143+0000 7fb27decd700 1 --2- 172.21.15.195:0/2047206723 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb258063d30 0x7fb2580661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:23.178 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.144+0000 7fb27decd700 1 --2- 172.21.15.195:0/2047206723 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb258063d30 0x7fb2580661e0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fb268009160 tx=0x7fb268017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:03:23.179 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.148+0000 7fb265ffb700 1 -- 172.21.15.195:0/2047206723 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fb27404fae0 con 0x7fb2780c34a0 2022-06-27T12:03:23.374 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.370+0000 7fb280131700 1 -- 172.21.15.195:0/2047206723 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb278156130 con 0x7fb2780c34a0 2022-06-27T12:03:23.374 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.371+0000 7fb265ffb700 1 -- 172.21.15.195:0/2047206723 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+6125 (secure 0 0 0) 0x7fb274053100 con 0x7fb2780c34a0 2022-06-27T12:03:23.375 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:03:23.376 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":5818,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/3399027527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3399027527},{"type":"v1","addr":"172.21.15.202:6833","nonce":3399027527}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:02:00.272060+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5273":{"gid":5273,"name":"d","rank":0,"incarnation":49,"state":"up:active","state_seq":42,"addr":"172.21.15.202:6835/417105338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":417105338},{"type":"v1","addr":"172.21.15.202:6835","nonce":417105338}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:03:23.378 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.373+0000 7fb280131700 1 -- 172.21.15.195:0/2047206723 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb258063d30 msgr2=0x7fb2580661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:23.378 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.373+0000 7fb280131700 1 --2- 172.21.15.195:0/2047206723 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb258063d30 0x7fb2580661e0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fb268009160 tx=0x7fb268017000 comp rx=0 tx=0).stop 2022-06-27T12:03:23.378 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.373+0000 7fb280131700 1 -- 172.21.15.195:0/2047206723 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb2780c34a0 msgr2=0x7fb27814f5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:23.379 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.373+0000 7fb280131700 1 --2- 172.21.15.195:0/2047206723 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb2780c34a0 0x7fb27814f5d0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fb27400d8c0 tx=0x7fb274008040 comp rx=0 tx=0).stop 2022-06-27T12:03:23.379 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.373+0000 7fb280131700 1 -- 172.21.15.195:0/2047206723 shutdown_connections 2022-06-27T12:03:23.379 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.373+0000 7fb280131700 1 --2- 172.21.15.195:0/2047206723 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb258063d30 0x7fb2580661e0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:23.379 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.373+0000 7fb280131700 1 --2- 172.21.15.195:0/2047206723 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb27816a580 0x7fb278149d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:23.380 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.373+0000 7fb280131700 1 --2- 172.21.15.195:0/2047206723 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb27806e850 0x7fb278156480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:23.380 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.373+0000 7fb280131700 1 --2- 172.21.15.195:0/2047206723 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb2780c34a0 0x7fb27814f5d0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:23.380 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.373+0000 7fb280131700 1 -- 172.21.15.195:0/2047206723 >> 172.21.15.195:0/2047206723 conn(0x7fb2780c6350 msgr2=0x7fb278067ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:03:23.380 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.373+0000 7fb280131700 1 -- 172.21.15.195:0/2047206723 shutdown_connections 2022-06-27T12:03:23.381 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:23.373+0000 7fb280131700 1 -- 172.21.15.195:0/2047206723 wait complete. 2022-06-27T12:03:23.381 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 54 2022-06-27T12:03:23.385 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-06-27T12:03:25.386 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:03:25.539 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.520+0000 7fc5bed37700 1 Processor -- start 2022-06-27T12:03:25.540 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.520+0000 7fc5bed37700 1 -- start start 2022-06-27T12:03:25.540 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.521+0000 7fc5bed37700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fc5b8150500 0x7fc5b8063af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:25.540 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.521+0000 7fc5bed37700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc5b806a450 0x7fc5b80c48d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:25.541 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.521+0000 7fc5bed37700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fc5b81508d0 0x7fc5b80bbf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:25.541 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.521+0000 7fc5bed37700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5b8064030 con 0x7fc5b81508d0 2022-06-27T12:03:25.541 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.521+0000 7fc5bed37700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5b80c4160 con 0x7fc5b8150500 2022-06-27T12:03:25.541 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.521+0000 7fc5bed37700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b80c4440 con 0x7fc5b806a450 2022-06-27T12:03:25.542 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.521+0000 7fc5bd2d4700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fc5b81508d0 0x7fc5b80bbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:25.542 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.521+0000 7fc5bd2d4700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fc5b81508d0 0x7fc5b80bbf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47148/0 (socket says 172.21.15.195:47148) 2022-06-27T12:03:25.542 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.521+0000 7fc5bd2d4700 1 -- 172.21.15.195:0/3733416045 learned_addr learned my addr 172.21.15.195:0/3733416045 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:03:25.542 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.521+0000 7fc5bcad3700 1 --2- 172.21.15.195:0/3733416045 >> v2:172.21.15.202:3300/0 conn(0x7fc5b8150500 0x7fc5b8063af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:25.543 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.521+0000 7fc5b7fff700 1 --2- 172.21.15.195:0/3733416045 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc5b806a450 0x7fc5b80c48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:25.543 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.522+0000 7fc5bd2d4700 1 -- 172.21.15.195:0/3733416045 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc5b806a450 msgr2=0x7fc5b80c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:03:25.543 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.522+0000 7fc5bd2d4700 1 --2- 172.21.15.195:0/3733416045 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc5b806a450 0x7fc5b80c48d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:25.543 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.522+0000 7fc5bd2d4700 1 -- 172.21.15.195:0/3733416045 >> v2:172.21.15.202:3300/0 conn(0x7fc5b8150500 msgr2=0x7fc5b8063af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:03:25.544 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.522+0000 7fc5bd2d4700 1 --2- 172.21.15.195:0/3733416045 >> v2:172.21.15.202:3300/0 conn(0x7fc5b8150500 0x7fc5b8063af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:25.544 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.522+0000 7fc5bd2d4700 1 -- 172.21.15.195:0/3733416045 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5b815a800 con 0x7fc5b81508d0 2022-06-27T12:03:25.544 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.522+0000 7fc5bd2d4700 1 --2- 172.21.15.195:0/3733416045 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc5b81508d0 0x7fc5b80bbf50 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fc5b806f840 tx=0x7fc5b0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39326bccd5bdce3a server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:03:25.545 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.522+0000 7fc5b77fe700 1 -- 172.21.15.195:0/3733416045 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b000e070 con 0x7fc5b81508d0 2022-06-27T12:03:25.545 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.522+0000 7fc5b77fe700 1 -- 172.21.15.195:0/3733416045 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5b000ba50 con 0x7fc5b81508d0 2022-06-27T12:03:25.545 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.522+0000 7fc5b77fe700 1 -- 172.21.15.195:0/3733416045 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b00092e0 con 0x7fc5b81508d0 2022-06-27T12:03:25.545 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.522+0000 7fc5bed37700 1 -- 172.21.15.195:0/3733416045 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc5b81508d0 msgr2=0x7fc5b80bbf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:25.546 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.522+0000 7fc5bed37700 1 --2- 172.21.15.195:0/3733416045 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc5b81508d0 0x7fc5b80bbf50 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fc5b806f840 tx=0x7fc5b0009840 comp rx=0 tx=0).stop 2022-06-27T12:03:25.546 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.522+0000 7fc5bed37700 1 -- 172.21.15.195:0/3733416045 shutdown_connections 2022-06-27T12:03:25.547 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.522+0000 7fc5bed37700 1 --2- 172.21.15.195:0/3733416045 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc5b81508d0 0x7fc5b80bbf50 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:25.548 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.522+0000 7fc5bed37700 1 --2- 172.21.15.195:0/3733416045 >> v2:172.21.15.202:3300/0 conn(0x7fc5b8150500 0x7fc5b8063af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:25.548 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.522+0000 7fc5bed37700 1 --2- 172.21.15.195:0/3733416045 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc5b806a450 0x7fc5b80c48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:25.548 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.523+0000 7fc5bed37700 1 -- 172.21.15.195:0/3733416045 >> 172.21.15.195:0/3733416045 conn(0x7fc5b80c9b00 msgr2=0x7fc5b80c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:03:25.549 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.523+0000 7fc5bed37700 1 -- 172.21.15.195:0/3733416045 shutdown_connections 2022-06-27T12:03:25.549 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.523+0000 7fc5bed37700 1 -- 172.21.15.195:0/3733416045 wait complete. 2022-06-27T12:03:25.549 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.524+0000 7fc5bed37700 1 Processor -- start 2022-06-27T12:03:25.549 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.524+0000 7fc5bed37700 1 -- start start 2022-06-27T12:03:25.550 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.525+0000 7fc5bed37700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc5b806a450 0x7fc5b8166bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:25.550 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.525+0000 7fc5bed37700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc5b8150500 0x7fc5b80bd830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:25.551 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.525+0000 7fc5bed37700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc5b81508d0 0x7fc5b81589c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:25.551 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.525+0000 7fc5bed37700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b80bce20 con 0x7fc5b806a450 2022-06-27T12:03:25.551 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.525+0000 7fc5bed37700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b80bd3a0 con 0x7fc5b8150500 2022-06-27T12:03:25.552 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.525+0000 7fc5bed37700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b80bf6f0 con 0x7fc5b81508d0 2022-06-27T12:03:25.552 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.525+0000 7fc5bcad3700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc5b806a450 0x7fc5b8166bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:25.552 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.525+0000 7fc5bcad3700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc5b806a450 0x7fc5b8166bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47150/0 (socket says 172.21.15.195:47150) 2022-06-27T12:03:25.552 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.525+0000 7fc5bcad3700 1 -- 172.21.15.195:0/1891162264 learned_addr learned my addr 172.21.15.195:0/1891162264 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:03:25.553 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.525+0000 7fc5b7fff700 1 --2- 172.21.15.195:0/1891162264 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc5b8150500 0x7fc5b80bd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:25.553 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.525+0000 7fc5bd2d4700 1 --2- 172.21.15.195:0/1891162264 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc5b81508d0 0x7fc5b81589c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:25.553 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.526+0000 7fc5bcad3700 1 -- 172.21.15.195:0/1891162264 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc5b81508d0 msgr2=0x7fc5b81589c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:25.553 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.526+0000 7fc5bcad3700 1 --2- 172.21.15.195:0/1891162264 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc5b81508d0 0x7fc5b81589c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:25.554 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.526+0000 7fc5bcad3700 1 -- 172.21.15.195:0/1891162264 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc5b8150500 msgr2=0x7fc5b80bd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:25.554 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.526+0000 7fc5bcad3700 1 --2- 172.21.15.195:0/1891162264 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc5b8150500 0x7fc5b80bd830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:25.554 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.526+0000 7fc5bcad3700 1 -- 172.21.15.195:0/1891162264 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5b0009ae0 con 0x7fc5b806a450 2022-06-27T12:03:25.554 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.526+0000 7fc5bcad3700 1 --2- 172.21.15.195:0/1891162264 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc5b806a450 0x7fc5b8166bd0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac00d860 tx=0x7fc5ac008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:03:25.555 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.526+0000 7fc5b4ff9700 1 -- 172.21.15.195:0/1891162264 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5ac012630 con 0x7fc5b806a450 2022-06-27T12:03:25.555 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.526+0000 7fc5b4ff9700 1 -- 172.21.15.195:0/1891162264 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5ac012790 con 0x7fc5b806a450 2022-06-27T12:03:25.555 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.526+0000 7fc5bed37700 1 -- 172.21.15.195:0/1891162264 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5b80bf9d0 con 0x7fc5b806a450 2022-06-27T12:03:25.555 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.526+0000 7fc5b4ff9700 1 -- 172.21.15.195:0/1891162264 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5ac012b50 con 0x7fc5b806a450 2022-06-27T12:03:25.556 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.526+0000 7fc5bed37700 1 -- 172.21.15.195:0/1891162264 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5b80bd030 con 0x7fc5b806a450 2022-06-27T12:03:25.556 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.527+0000 7fc5bed37700 1 -- 172.21.15.195:0/1891162264 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5b80bd1f0 con 0x7fc5b806a450 2022-06-27T12:03:25.557 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.527+0000 7fc5b4ff9700 1 -- 172.21.15.195:0/1891162264 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fc5ac01d400 con 0x7fc5b806a450 2022-06-27T12:03:25.558 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.528+0000 7fc5b4ff9700 1 --2- 172.21.15.195:0/1891162264 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc5a0063d00 0x7fc5a00661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:25.558 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.528+0000 7fc5b4ff9700 1 -- 172.21.15.195:0/1891162264 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6804+0+0 (secure 0 0 0) 0x7fc5ac080fe0 con 0x7fc5b806a450 2022-06-27T12:03:25.558 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.528+0000 7fc5b7fff700 1 --2- 172.21.15.195:0/1891162264 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc5a0063d00 0x7fc5a00661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:25.559 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.528+0000 7fc5b7fff700 1 --2- 172.21.15.195:0/1891162264 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc5a0063d00 0x7fc5a00661b0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fc5a4004560 tx=0x7fc5a4008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:03:25.559 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.532+0000 7fc5b4ff9700 1 -- 172.21.15.195:0/1891162264 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fc5ac04f930 con 0x7fc5b806a450 2022-06-27T12:03:25.757 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.757+0000 7fc5bed37700 1 -- 172.21.15.195:0/1891162264 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc5b80722c0 con 0x7fc5b806a450 2022-06-27T12:03:25.759 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.758+0000 7fc5b4ff9700 1 -- 172.21.15.195:0/1891162264 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+6125 (secure 0 0 0) 0x7fc5ac052f50 con 0x7fc5b806a450 2022-06-27T12:03:25.762 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:03:25.763 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":5818,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/3399027527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3399027527},{"type":"v1","addr":"172.21.15.202:6833","nonce":3399027527}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:02:00.272060+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5273":{"gid":5273,"name":"d","rank":0,"incarnation":49,"state":"up:active","state_seq":42,"addr":"172.21.15.202:6835/417105338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":417105338},{"type":"v1","addr":"172.21.15.202:6835","nonce":417105338}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:03:25.769 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.760+0000 7fc5bed37700 1 -- 172.21.15.195:0/1891162264 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc5a0063d00 msgr2=0x7fc5a00661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:25.771 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.760+0000 7fc5bed37700 1 --2- 172.21.15.195:0/1891162264 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc5a0063d00 0x7fc5a00661b0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fc5a4004560 tx=0x7fc5a4008040 comp rx=0 tx=0).stop 2022-06-27T12:03:25.776 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.760+0000 7fc5bed37700 1 -- 172.21.15.195:0/1891162264 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc5b806a450 msgr2=0x7fc5b8166bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:25.778 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.760+0000 7fc5bed37700 1 --2- 172.21.15.195:0/1891162264 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc5b806a450 0x7fc5b8166bd0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac00d860 tx=0x7fc5ac008040 comp rx=0 tx=0).stop 2022-06-27T12:03:25.778 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.760+0000 7fc5bed37700 1 -- 172.21.15.195:0/1891162264 shutdown_connections 2022-06-27T12:03:25.779 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.760+0000 7fc5bed37700 1 --2- 172.21.15.195:0/1891162264 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc5a0063d00 0x7fc5a00661b0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:25.779 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.760+0000 7fc5bed37700 1 --2- 172.21.15.195:0/1891162264 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc5b806a450 0x7fc5b8166bd0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:25.779 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.760+0000 7fc5bed37700 1 --2- 172.21.15.195:0/1891162264 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc5b81508d0 0x7fc5b81589c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:25.780 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.760+0000 7fc5bed37700 1 --2- 172.21.15.195:0/1891162264 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc5b8150500 0x7fc5b80bd830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:25.780 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.760+0000 7fc5bed37700 1 -- 172.21.15.195:0/1891162264 >> 172.21.15.195:0/1891162264 conn(0x7fc5b80c9b00 msgr2=0x7fc5b80c8860 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:03:25.781 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.760+0000 7fc5bed37700 1 -- 172.21.15.195:0/1891162264 shutdown_connections 2022-06-27T12:03:25.784 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:25.760+0000 7fc5bed37700 1 -- 172.21.15.195:0/1891162264 wait complete. 2022-06-27T12:03:25.786 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 54 2022-06-27T12:03:25.788 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-06-27T12:03:27.791 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:03:27.961 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:27.933+0000 7f90cdbb0700 1 Processor -- start 2022-06-27T12:03:27.961 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:27.933+0000 7f90cdbb0700 1 -- start start 2022-06-27T12:03:27.962 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:27.934+0000 7f90cdbb0700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f90c80688a0 0x7f90c80c28c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:27.962 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:27.934+0000 7f90cdbb0700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90c80c4b10 con 0x7f90c8156100 2022-06-27T12:03:27.962 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:27.934+0000 7f90cdbb0700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90c8068f20 con 0x7f90c806a440 2022-06-27T12:03:27.962 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:27.934+0000 7f90cdbb0700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90c80c3dc0 con 0x7f90c80688a0 2022-06-27T12:03:27.963 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:27.934+0000 7f90c77fe700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f90c8156100 0x7f90c80c4fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:53368/0 (socket says 172.21.15.195:53368) 2022-06-27T12:03:27.963 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:27.934+0000 7f90c77fe700 1 -- 172.21.15.195:0/826240221 learned_addr learned my addr 172.21.15.195:0/826240221 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:03:27.963 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:27.934+0000 7f90c7fff700 1 --2- 172.21.15.195:0/826240221 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f90c80688a0 0x7f90c80c28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:27.963 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:27.934+0000 7f90c67fc700 1 -- 172.21.15.195:0/826240221 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3331110697 0 0) 0x7f90c80c4b10 con 0x7f90c8156100 2022-06-27T12:03:27.964 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:27.934+0000 7f90c67fc700 1 -- 172.21.15.195:0/826240221 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90b4003430 con 0x7f90c8156100 2022-06-27T12:03:27.964 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:27.934+0000 7f90c67fc700 1 -- 172.21.15.195:0/826240221 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3929025227 0 0) 0x7f90c8068f20 con 0x7f90c806a440 2022-06-27T12:03:27.964 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:27.934+0000 7f90c67fc700 1 -- 172.21.15.195:0/826240221 --> v1:172.21.15.202:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90c80c4b10 con 0x7f90c806a440 2022-06-27T12:03:27.978 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:27.934+0000 7f90c7fff700 1 -- 172.21.15.195:0/826240221 >> v1:172.21.15.202:6789/0 conn(0x7f90c806a440 legacy=0x7f90c81564d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:27.978 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:27.934+0000 7f90c7fff700 1 -- 172.21.15.195:0/826240221 >> v1:172.21.15.195:6789/0 conn(0x7f90c8156100 legacy=0x7f90c80c4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:27.979 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:27.934+0000 7f90c67fc700 1 -- 172.21.15.195:0/826240221 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3969000764 0 0) 0x7f90b4003430 con 0x7f90c8156100 2022-06-27T12:03:27.994 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:27.935+0000 7f90c7fff700 1 -- 172.21.15.195:0/826240221 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90c80c56b0 con 0x7f90c80688a0 2022-06-27T12:03:27.994 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:27.935+0000 7f90c7fff700 1 --2- 172.21.15.195:0/826240221 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f90c80688a0 0x7f90c80c28c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f90c0009e90 tx=0x7f90c0009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5803ef33f48c21e9 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:03:27.995 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:27.935+0000 7f90c67fc700 1 -- 172.21.15.195:0/826240221 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c000e070 con 0x7f90c80688a0 2022-06-27T12:03:27.995 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:27.935+0000 7f90c67fc700 1 -- 172.21.15.195:0/826240221 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90c00038a0 con 0x7f90c80688a0 2022-06-27T12:03:27.996 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:27.935+0000 7f90c67fc700 1 -- 172.21.15.195:0/826240221 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90bc0015f0 con 0x7f90c80688a0 2022-06-27T12:03:27.996 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:27.935+0000 7f90cdbb0700 1 -- 172.21.15.195:0/826240221 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f90c80688a0 msgr2=0x7f90c80c28c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:27.997 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:27.935+0000 7f90cdbb0700 1 --2- 172.21.15.195:0/826240221 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f90c80688a0 0x7f90c80c28c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f90c0009e90 tx=0x7f90c0009780 comp rx=0 tx=0).stop 2022-06-27T12:03:27.997 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:27.935+0000 7f90cdbb0700 1 -- 172.21.15.195:0/826240221 shutdown_connections 2022-06-27T12:03:27.997 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:27.935+0000 7f90cdbb0700 1 --2- 172.21.15.195:0/826240221 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f90c80688a0 0x7f90c80c28c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:27.998 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:27.935+0000 7f90cdbb0700 1 -- 172.21.15.195:0/826240221 >> 172.21.15.195:0/826240221 conn(0x7f90c80c9b00 msgr2=0x7f90c80c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:03:27.998 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:27.935+0000 7f90cdbb0700 1 -- 172.21.15.195:0/826240221 shutdown_connections 2022-06-27T12:03:27.999 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:27.935+0000 7f90cdbb0700 1 -- 172.21.15.195:0/826240221 wait complete. 2022-06-27T12:03:27.999 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:27.937+0000 7f90cdbb0700 1 Processor -- start 2022-06-27T12:03:28.000 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:27.937+0000 7f90cdbb0700 1 -- start start 2022-06-27T12:03:28.000 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:27.937+0000 7f90cdbb0700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f90c80688a0 0x7f90c81656b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:28.001 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:27.937+0000 7f90cdbb0700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f90c806a440 0x7f90c8144590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:28.001 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:27.938+0000 7f90cdbb0700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f90c8156100 0x7f90c8141410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:28.006 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:27.938+0000 7f90cdbb0700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90c8143860 con 0x7f90c806a440 2022-06-27T12:03:28.007 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:27.938+0000 7f90cdbb0700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90c8144100 con 0x7f90c80688a0 2022-06-27T12:03:28.007 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:27.938+0000 7f90cdbb0700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90c8144ad0 con 0x7f90c8156100 2022-06-27T12:03:28.007 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:27.938+0000 7f90c6ffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f90c806a440 0x7f90c8144590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:28.008 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:27.938+0000 7f90c6ffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f90c806a440 0x7f90c8144590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47188/0 (socket says 172.21.15.195:47188) 2022-06-27T12:03:28.008 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:27.938+0000 7f90c6ffd700 1 -- 172.21.15.195:0/3433294988 learned_addr learned my addr 172.21.15.195:0/3433294988 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:03:28.009 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:27.938+0000 7f90c77fe700 1 --2- 172.21.15.195:0/3433294988 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f90c80688a0 0x7f90c81656b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:28.009 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:27.938+0000 7f90c7fff700 1 --2- 172.21.15.195:0/3433294988 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f90c8156100 0x7f90c8141410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:28.009 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:27.938+0000 7f90c6ffd700 1 -- 172.21.15.195:0/3433294988 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f90c8156100 msgr2=0x7f90c8141410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:28.010 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:27.938+0000 7f90c6ffd700 1 --2- 172.21.15.195:0/3433294988 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f90c8156100 0x7f90c8141410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:28.010 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:27.938+0000 7f90c6ffd700 1 -- 172.21.15.195:0/3433294988 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f90c80688a0 msgr2=0x7f90c81656b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:28.010 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:27.938+0000 7f90c6ffd700 1 --2- 172.21.15.195:0/3433294988 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f90c80688a0 0x7f90c81656b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:28.011 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:27.938+0000 7f90c6ffd700 1 -- 172.21.15.195:0/3433294988 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90c0009a20 con 0x7f90c806a440 2022-06-27T12:03:28.011 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:27.938+0000 7f90c6ffd700 1 --2- 172.21.15.195:0/3433294988 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f90c806a440 0x7f90c8144590 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f90b800c4f0 tx=0x7f90b8007c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:03:28.011 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:27.938+0000 7f90af7fe700 1 -- 172.21.15.195:0/3433294988 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90b80050d0 con 0x7f90c806a440 2022-06-27T12:03:28.012 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:27.938+0000 7f90af7fe700 1 -- 172.21.15.195:0/3433294988 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90b8005250 con 0x7f90c806a440 2022-06-27T12:03:28.012 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:27.938+0000 7f90cdbb0700 1 -- 172.21.15.195:0/3433294988 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90c8142a10 con 0x7f90c806a440 2022-06-27T12:03:28.013 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:27.938+0000 7f90af7fe700 1 -- 172.21.15.195:0/3433294988 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90b80117e0 con 0x7f90c806a440 2022-06-27T12:03:28.013 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:27.938+0000 7f90cdbb0700 1 -- 172.21.15.195:0/3433294988 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90c8143ab0 con 0x7f90c806a440 2022-06-27T12:03:28.013 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:27.939+0000 7f90cdbb0700 1 -- 172.21.15.195:0/3433294988 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90c81442b0 con 0x7f90c806a440 2022-06-27T12:03:28.014 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:27.940+0000 7f90af7fe700 1 -- 172.21.15.195:0/3433294988 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f90b8019030 con 0x7f90c806a440 2022-06-27T12:03:28.014 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:27.940+0000 7f90af7fe700 1 --2- 172.21.15.195:0/3433294988 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f90a8063d30 0x7f90a80661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:28.014 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:27.940+0000 7f90af7fe700 1 -- 172.21.15.195:0/3433294988 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6804+0+0 (secure 0 0 0) 0x7f90b807fde0 con 0x7f90c806a440 2022-06-27T12:03:28.015 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:27.940+0000 7f90c77fe700 1 --2- 172.21.15.195:0/3433294988 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f90a8063d30 0x7f90a80661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:28.015 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:27.941+0000 7f90c77fe700 1 --2- 172.21.15.195:0/3433294988 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f90a8063d30 0x7f90a80661e0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f90bc000ef0 tx=0x7f90bc009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:03:28.016 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:27.944+0000 7f90af7fe700 1 -- 172.21.15.195:0/3433294988 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f90c81442b0 con 0x7f90c806a440 2022-06-27T12:03:28.192 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:28.178+0000 7f90cdbb0700 1 -- 172.21.15.195:0/3433294988 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90c80bd180 con 0x7f90c806a440 2022-06-27T12:03:28.193 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:28.179+0000 7f90af7fe700 1 -- 172.21.15.195:0/3433294988 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+6125 (secure 0 0 0) 0x7f90b8017020 con 0x7f90c806a440 2022-06-27T12:03:28.194 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:03:28.194 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":5818,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/3399027527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3399027527},{"type":"v1","addr":"172.21.15.202:6833","nonce":3399027527}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:02:00.272060+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5273":{"gid":5273,"name":"d","rank":0,"incarnation":49,"state":"up:active","state_seq":42,"addr":"172.21.15.202:6835/417105338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":417105338},{"type":"v1","addr":"172.21.15.202:6835","nonce":417105338}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:03:28.198 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:28.181+0000 7f90cdbb0700 1 -- 172.21.15.195:0/3433294988 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f90a8063d30 msgr2=0x7f90a80661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:28.198 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:28.181+0000 7f90cdbb0700 1 --2- 172.21.15.195:0/3433294988 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f90a8063d30 0x7f90a80661e0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f90bc000ef0 tx=0x7f90bc009040 comp rx=0 tx=0).stop 2022-06-27T12:03:28.199 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:28.181+0000 7f90cdbb0700 1 -- 172.21.15.195:0/3433294988 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f90c806a440 msgr2=0x7f90c8144590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:28.199 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:28.181+0000 7f90cdbb0700 1 --2- 172.21.15.195:0/3433294988 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f90c806a440 0x7f90c8144590 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f90b800c4f0 tx=0x7f90b8007c50 comp rx=0 tx=0).stop 2022-06-27T12:03:28.200 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:28.181+0000 7f90cdbb0700 1 -- 172.21.15.195:0/3433294988 shutdown_connections 2022-06-27T12:03:28.200 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:28.181+0000 7f90cdbb0700 1 --2- 172.21.15.195:0/3433294988 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f90a8063d30 0x7f90a80661e0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:28.200 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:28.181+0000 7f90cdbb0700 1 --2- 172.21.15.195:0/3433294988 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f90c8156100 0x7f90c8141410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:28.201 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:28.181+0000 7f90cdbb0700 1 --2- 172.21.15.195:0/3433294988 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f90c80688a0 0x7f90c81656b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:28.201 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:28.181+0000 7f90cdbb0700 1 --2- 172.21.15.195:0/3433294988 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f90c806a440 0x7f90c8144590 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:28.202 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:28.181+0000 7f90cdbb0700 1 -- 172.21.15.195:0/3433294988 >> 172.21.15.195:0/3433294988 conn(0x7f90c80c9b00 msgr2=0x7f90c8067ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:03:28.202 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:28.181+0000 7f90cdbb0700 1 -- 172.21.15.195:0/3433294988 shutdown_connections 2022-06-27T12:03:28.203 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:28.181+0000 7f90cdbb0700 1 -- 172.21.15.195:0/3433294988 wait complete. 2022-06-27T12:03:28.203 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 54 2022-06-27T12:03:28.204 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-06-27T12:03:30.205 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:03:30.344 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.345+0000 7f5bf9687700 1 Processor -- start 2022-06-27T12:03:30.344 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.345+0000 7f5bf9687700 1 -- start start 2022-06-27T12:03:30.360 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.346+0000 7f5bf9687700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5bf4154dd0 0x7f5bf4158100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:30.361 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.346+0000 7f5bf9687700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bf40c3980 con 0x7f5bf4154310 2022-06-27T12:03:30.361 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.346+0000 7f5bf9687700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bf40c3c40 con 0x7f5bf40c34b0 2022-06-27T12:03:30.361 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.346+0000 7f5bf9687700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf4151dd0 con 0x7f5bf4154dd0 2022-06-27T12:03:30.362 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.346+0000 7f5bf27fc700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f5bf4154310 0x7f5bf40c1dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:53388/0 (socket says 172.21.15.195:53388) 2022-06-27T12:03:30.362 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.346+0000 7f5bf27fc700 1 -- 172.21.15.195:0/435409151 learned_addr learned my addr 172.21.15.195:0/435409151 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:03:30.363 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.346+0000 7f5bf2ffd700 1 --2- 172.21.15.195:0/435409151 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5bf4154dd0 0x7f5bf4158100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:30.363 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.346+0000 7f5bf1ffb700 1 -- 172.21.15.195:0/435409151 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1123816243 0 0) 0x7f5bf40c3980 con 0x7f5bf4154310 2022-06-27T12:03:30.364 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.346+0000 7f5bf1ffb700 1 -- 172.21.15.195:0/435409151 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5be0003430 con 0x7f5bf4154310 2022-06-27T12:03:30.364 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.346+0000 7f5bf1ffb700 1 -- 172.21.15.195:0/435409151 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 945132474 0 0) 0x7f5bf40c3c40 con 0x7f5bf40c34b0 2022-06-27T12:03:30.365 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.346+0000 7f5bf1ffb700 1 -- 172.21.15.195:0/435409151 --> v1:172.21.15.202:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bf40c3980 con 0x7f5bf40c34b0 2022-06-27T12:03:30.365 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.346+0000 7f5bf1ffb700 1 -- 172.21.15.195:0/435409151 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4165568101 0 0) 0x7f5be0003430 con 0x7f5bf4154310 2022-06-27T12:03:30.366 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.346+0000 7f5bf2ffd700 1 -- 172.21.15.195:0/435409151 >> v1:172.21.15.202:6789/0 conn(0x7f5bf40c34b0 legacy=0x7f5bf415ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:30.366 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.346+0000 7f5bf2ffd700 1 -- 172.21.15.195:0/435409151 >> v1:172.21.15.195:6789/0 conn(0x7f5bf4154310 legacy=0x7f5bf40c1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:30.366 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.347+0000 7f5bf2ffd700 1 -- 172.21.15.195:0/435409151 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bf40c3f20 con 0x7f5bf4154dd0 2022-06-27T12:03:30.367 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.347+0000 7f5bf2ffd700 1 --2- 172.21.15.195:0/435409151 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5bf4154dd0 0x7f5bf4158100 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5be80097d0 tx=0x7f5be8009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=361627d7f8cbc737 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:03:30.367 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.347+0000 7f5bf1ffb700 1 -- 172.21.15.195:0/435409151 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5be4001470 con 0x7f5bf4154dd0 2022-06-27T12:03:30.368 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.347+0000 7f5bf1ffb700 1 -- 172.21.15.195:0/435409151 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5be8004030 con 0x7f5bf4154dd0 2022-06-27T12:03:30.369 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.347+0000 7f5bf1ffb700 1 -- 172.21.15.195:0/435409151 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5be80043b0 con 0x7f5bf4154dd0 2022-06-27T12:03:30.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.347+0000 7f5bf9687700 1 -- 172.21.15.195:0/435409151 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5bf4154dd0 msgr2=0x7f5bf4158100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:30.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.347+0000 7f5bf9687700 1 --2- 172.21.15.195:0/435409151 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5bf4154dd0 0x7f5bf4158100 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5be80097d0 tx=0x7f5be8009060 comp rx=0 tx=0).stop 2022-06-27T12:03:30.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.347+0000 7f5bf9687700 1 -- 172.21.15.195:0/435409151 shutdown_connections 2022-06-27T12:03:30.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.347+0000 7f5bf9687700 1 --2- 172.21.15.195:0/435409151 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5bf4154dd0 0x7f5bf4158100 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:30.372 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.347+0000 7f5bf9687700 1 -- 172.21.15.195:0/435409151 >> 172.21.15.195:0/435409151 conn(0x7f5bf40c8740 msgr2=0x7f5bf40c8b10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:03:30.372 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.347+0000 7f5bf9687700 1 -- 172.21.15.195:0/435409151 shutdown_connections 2022-06-27T12:03:30.372 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.347+0000 7f5bf9687700 1 -- 172.21.15.195:0/435409151 wait complete. 2022-06-27T12:03:30.373 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.349+0000 7f5bf9687700 1 Processor -- start 2022-06-27T12:03:30.373 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.349+0000 7f5bf9687700 1 -- start start 2022-06-27T12:03:30.373 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.349+0000 7f5bf9687700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5bf40c34b0 0x7f5bf4155640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:30.373 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.349+0000 7f5bf9687700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5bf4154310 0x7f5bf40ba9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:30.374 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.349+0000 7f5bf9687700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5bf4154dd0 0x7f5bf40baf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:30.374 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.349+0000 7f5bf9687700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf4156710 con 0x7f5bf40c34b0 2022-06-27T12:03:30.374 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.349+0000 7f5bf9687700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf40ba560 con 0x7f5bf4154310 2022-06-27T12:03:30.374 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.349+0000 7f5bf9687700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf4157bb0 con 0x7f5bf4154dd0 2022-06-27T12:03:30.375 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.350+0000 7f5bf2ffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5bf40c34b0 0x7f5bf4155640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:30.375 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.350+0000 7f5bf2ffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5bf40c34b0 0x7f5bf4155640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47204/0 (socket says 172.21.15.195:47204) 2022-06-27T12:03:30.375 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.350+0000 7f5bf2ffd700 1 -- 172.21.15.195:0/535896744 learned_addr learned my addr 172.21.15.195:0/535896744 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:03:30.376 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.350+0000 7f5bf27fc700 1 --2- 172.21.15.195:0/535896744 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5bf4154310 0x7f5bf40ba9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:30.376 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.350+0000 7f5bf37fe700 1 --2- 172.21.15.195:0/535896744 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5bf4154dd0 0x7f5bf40baf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:30.376 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.350+0000 7f5bf2ffd700 1 -- 172.21.15.195:0/535896744 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5bf4154dd0 msgr2=0x7f5bf40baf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:30.376 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.350+0000 7f5bf2ffd700 1 --2- 172.21.15.195:0/535896744 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5bf4154dd0 0x7f5bf40baf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:30.377 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.350+0000 7f5bf2ffd700 1 -- 172.21.15.195:0/535896744 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5bf4154310 msgr2=0x7f5bf40ba9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:30.377 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.350+0000 7f5bf2ffd700 1 --2- 172.21.15.195:0/535896744 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5bf4154310 0x7f5bf40ba9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:30.377 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.350+0000 7f5bf2ffd700 1 -- 172.21.15.195:0/535896744 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5be80092e0 con 0x7f5bf40c34b0 2022-06-27T12:03:30.378 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.350+0000 7f5bf37fe700 1 --2- 172.21.15.195:0/535896744 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5bf4154dd0 0x7f5bf40baf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T12:03:30.378 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.350+0000 7f5bf2ffd700 1 --2- 172.21.15.195:0/535896744 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5bf40c34b0 0x7f5bf4155640 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f5be8003de0 tx=0x7f5be8015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:03:30.378 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.350+0000 7f5bdaffd700 1 -- 172.21.15.195:0/535896744 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5be80116d0 con 0x7f5bf40c34b0 2022-06-27T12:03:30.378 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.350+0000 7f5bdaffd700 1 -- 172.21.15.195:0/535896744 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5be8011830 con 0x7f5bf40c34b0 2022-06-27T12:03:30.379 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.350+0000 7f5bf9687700 1 -- 172.21.15.195:0/535896744 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bf4156150 con 0x7f5bf40c34b0 2022-06-27T12:03:30.379 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.350+0000 7f5bdaffd700 1 -- 172.21.15.195:0/535896744 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5be8011bf0 con 0x7f5bf40c34b0 2022-06-27T12:03:30.379 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.350+0000 7f5bf9687700 1 -- 172.21.15.195:0/535896744 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bf40ba1f0 con 0x7f5bf40c34b0 2022-06-27T12:03:30.380 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.351+0000 7f5bf9687700 1 -- 172.21.15.195:0/535896744 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bf40ba840 con 0x7f5bf40c34b0 2022-06-27T12:03:30.380 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.351+0000 7f5bdaffd700 1 -- 172.21.15.195:0/535896744 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f5be800f070 con 0x7f5bf40c34b0 2022-06-27T12:03:30.380 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.352+0000 7f5bdaffd700 1 --2- 172.21.15.195:0/535896744 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5bd4063d00 0x7f5bd40661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:30.380 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.352+0000 7f5bdaffd700 1 -- 172.21.15.195:0/535896744 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6804+0+0 (secure 0 0 0) 0x7f5be8082b20 con 0x7f5bf40c34b0 2022-06-27T12:03:30.380 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.352+0000 7f5bf27fc700 1 --2- 172.21.15.195:0/535896744 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5bd4063d00 0x7f5bd40661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:30.381 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.352+0000 7f5bf27fc700 1 --2- 172.21.15.195:0/535896744 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5bd4063d00 0x7f5bd40661b0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f5bf4157dc0 tx=0x7f5be4001500 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:03:30.381 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.356+0000 7f5bdaffd700 1 -- 172.21.15.195:0/535896744 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f5be8051420 con 0x7f5bf40c34b0 2022-06-27T12:03:30.585 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.582+0000 7f5bf9687700 1 -- 172.21.15.195:0/535896744 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5bf40ba380 con 0x7f5bf40c34b0 2022-06-27T12:03:30.585 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.582+0000 7f5bdaffd700 1 -- 172.21.15.195:0/535896744 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+6125 (secure 0 0 0) 0x7f5be8054a40 con 0x7f5bf40c34b0 2022-06-27T12:03:30.586 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:03:30.586 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":5818,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/3399027527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3399027527},{"type":"v1","addr":"172.21.15.202:6833","nonce":3399027527}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:02:00.272060+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5273":{"gid":5273,"name":"d","rank":0,"incarnation":49,"state":"up:active","state_seq":42,"addr":"172.21.15.202:6835/417105338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":417105338},{"type":"v1","addr":"172.21.15.202:6835","nonce":417105338}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:03:30.588 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.584+0000 7f5bf9687700 1 -- 172.21.15.195:0/535896744 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5bd4063d00 msgr2=0x7f5bd40661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:30.589 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.584+0000 7f5bf9687700 1 --2- 172.21.15.195:0/535896744 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5bd4063d00 0x7f5bd40661b0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f5bf4157dc0 tx=0x7f5be4001500 comp rx=0 tx=0).stop 2022-06-27T12:03:30.589 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.584+0000 7f5bf9687700 1 -- 172.21.15.195:0/535896744 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5bf40c34b0 msgr2=0x7f5bf4155640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:30.589 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.584+0000 7f5bf9687700 1 --2- 172.21.15.195:0/535896744 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5bf40c34b0 0x7f5bf4155640 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f5be8003de0 tx=0x7f5be8015000 comp rx=0 tx=0).stop 2022-06-27T12:03:30.589 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.584+0000 7f5bf9687700 1 -- 172.21.15.195:0/535896744 shutdown_connections 2022-06-27T12:03:30.590 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.584+0000 7f5bf9687700 1 --2- 172.21.15.195:0/535896744 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5bd4063d00 0x7f5bd40661b0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:30.590 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.584+0000 7f5bf9687700 1 --2- 172.21.15.195:0/535896744 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5bf40c34b0 0x7f5bf4155640 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:30.590 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.584+0000 7f5bf9687700 1 --2- 172.21.15.195:0/535896744 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5bf4154dd0 0x7f5bf40baf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:30.590 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.584+0000 7f5bf9687700 1 --2- 172.21.15.195:0/535896744 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5bf4154310 0x7f5bf40ba9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:30.591 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.584+0000 7f5bf9687700 1 -- 172.21.15.195:0/535896744 >> 172.21.15.195:0/535896744 conn(0x7f5bf40c8740 msgr2=0x7f5bf41510f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:03:30.591 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.584+0000 7f5bf9687700 1 -- 172.21.15.195:0/535896744 shutdown_connections 2022-06-27T12:03:30.591 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:30.584+0000 7f5bf9687700 1 -- 172.21.15.195:0/535896744 wait complete. 2022-06-27T12:03:30.591 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 54 2022-06-27T12:03:30.596 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-06-27T12:03:30.675 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:03:30.678 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:03:32.598 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:03:32.732 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.732+0000 7f3e4a185700 1 Processor -- start 2022-06-27T12:03:32.732 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.732+0000 7f3e4a185700 1 -- start start 2022-06-27T12:03:32.740 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.732+0000 7f3e4a185700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f3e440c3f50 0x7f3e4415ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:32.846 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.732+0000 7f3e4a185700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e440c33b0 con 0x7f3e4414e490 2022-06-27T12:03:32.846 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.733+0000 7f3e4a185700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e4406f220 con 0x7f3e4414e860 2022-06-27T12:03:32.846 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.733+0000 7f3e4a185700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e441542c0 con 0x7f3e440c3f50 2022-06-27T12:03:32.846 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.733+0000 7f3e437fe700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f3e4414e490 0x7f3e4406cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:53400/0 (socket says 172.21.15.195:53400) 2022-06-27T12:03:32.847 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.733+0000 7f3e437fe700 1 -- 172.21.15.195:0/3132049624 learned_addr learned my addr 172.21.15.195:0/3132049624 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:03:32.847 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.733+0000 7f3e43fff700 1 --2- 172.21.15.195:0/3132049624 >> v2:172.21.15.202:3300/0 conn(0x7f3e440c3f50 0x7f3e4415ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:32.847 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.733+0000 7f3e427fc700 1 -- 172.21.15.195:0/3132049624 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4212533795 0 0) 0x7f3e440c33b0 con 0x7f3e4414e490 2022-06-27T12:03:32.848 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.733+0000 7f3e427fc700 1 -- 172.21.15.195:0/3132049624 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e24003430 con 0x7f3e4414e490 2022-06-27T12:03:32.848 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.733+0000 7f3e427fc700 1 -- 172.21.15.195:0/3132049624 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3392373676 0 0) 0x7f3e24003430 con 0x7f3e4414e490 2022-06-27T12:03:32.848 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.733+0000 7f3e427fc700 1 -- 172.21.15.195:0/3132049624 >> v2:172.21.15.202:3300/0 conn(0x7f3e440c3f50 msgr2=0x7f3e4415ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:03:32.849 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.733+0000 7f3e427fc700 1 --2- 172.21.15.195:0/3132049624 >> v2:172.21.15.202:3300/0 conn(0x7f3e440c3f50 0x7f3e4415ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:32.849 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.733+0000 7f3e427fc700 1 -- 172.21.15.195:0/3132049624 >> v1:172.21.15.202:6789/0 conn(0x7f3e4414e860 legacy=0x7f3e440c3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:32.849 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.733+0000 7f3e427fc700 1 -- 172.21.15.195:0/3132049624 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e44154560 con 0x7f3e4414e490 2022-06-27T12:03:32.849 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.733+0000 7f3e427fc700 1 -- 172.21.15.195:0/3132049624 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7f3e2c001470 con 0x7f3e4414e490 2022-06-27T12:03:32.850 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.733+0000 7f3e427fc700 1 -- 172.21.15.195:0/3132049624 >> v1:172.21.15.195:6789/0 conn(0x7f3e4414e490 legacy=0x7f3e4406cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:32.850 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.733+0000 7f3e427fc700 1 --2- 172.21.15.195:0/3132049624 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3e24003920 0x7f3e24003d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:32.850 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.733+0000 7f3e427fc700 1 --2- 172.21.15.195:0/3132049624 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3e240046d0 0x7f3e24004b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:32.850 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.733+0000 7f3e427fc700 1 --2- 172.21.15.195:0/3132049624 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3e24005080 0x7f3e2400b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:32.851 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.733+0000 7f3e427fc700 1 -- 172.21.15.195:0/3132049624 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e441542c0 con 0x7f3e24003920 2022-06-27T12:03:32.851 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.733+0000 7f3e427fc700 1 -- 172.21.15.195:0/3132049624 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2400bf90 con 0x7f3e24005080 2022-06-27T12:03:32.852 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.733+0000 7f3e427fc700 1 -- 172.21.15.195:0/3132049624 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2400c150 con 0x7f3e240046d0 2022-06-27T12:03:32.852 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.733+0000 7f3e437fe700 1 --2- 172.21.15.195:0/3132049624 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3e24003920 0x7f3e24003d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:32.852 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.733+0000 7f3e43fff700 1 --2- 172.21.15.195:0/3132049624 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3e24005080 0x7f3e2400b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:32.853 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.733+0000 7f3e42ffd700 1 --2- 172.21.15.195:0/3132049624 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3e240046d0 0x7f3e24004b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:32.853 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.734+0000 7f3e437fe700 1 -- 172.21.15.195:0/3132049624 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3e240046d0 msgr2=0x7f3e24004b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:03:32.853 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.734+0000 7f3e437fe700 1 --2- 172.21.15.195:0/3132049624 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3e240046d0 0x7f3e24004b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:32.853 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.734+0000 7f3e437fe700 1 -- 172.21.15.195:0/3132049624 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3e24005080 msgr2=0x7f3e2400b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:03:32.854 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.734+0000 7f3e437fe700 1 --2- 172.21.15.195:0/3132049624 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3e24005080 0x7f3e2400b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:32.854 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.734+0000 7f3e437fe700 1 -- 172.21.15.195:0/3132049624 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e24003430 con 0x7f3e24003920 2022-06-27T12:03:32.854 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.734+0000 7f3e43fff700 1 --2- 172.21.15.195:0/3132049624 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3e24005080 0x7f3e2400b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:03:32.855 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.734+0000 7f3e43fff700 1 -- 172.21.15.195:0/3132049624 reap_dead start 2022-06-27T12:03:32.855 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.734+0000 7f3e437fe700 1 --2- 172.21.15.195:0/3132049624 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3e24003920 0x7f3e24003d30 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c00c1e0 tx=0x7f3e440c3220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3eed929ac2b275a7 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:03:32.855 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.734+0000 7f3e427fc700 1 -- 172.21.15.195:0/3132049624 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e2c00cab0 con 0x7f3e24003920 2022-06-27T12:03:32.855 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.734+0000 7f3e427fc700 1 -- 172.21.15.195:0/3132049624 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e2c00cc10 con 0x7f3e24003920 2022-06-27T12:03:32.859 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.734+0000 7f3e427fc700 1 -- 172.21.15.195:0/3132049624 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e2c0070d0 con 0x7f3e24003920 2022-06-27T12:03:32.860 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.734+0000 7f3e4a185700 1 -- 172.21.15.195:0/3132049624 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3e24003920 msgr2=0x7f3e24003d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:32.864 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.734+0000 7f3e4a185700 1 --2- 172.21.15.195:0/3132049624 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3e24003920 0x7f3e24003d30 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c00c1e0 tx=0x7f3e440c3220 comp rx=0 tx=0).stop 2022-06-27T12:03:32.865 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.734+0000 7f3e4a185700 1 -- 172.21.15.195:0/3132049624 shutdown_connections 2022-06-27T12:03:32.865 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.734+0000 7f3e4a185700 1 --2- 172.21.15.195:0/3132049624 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3e24003920 0x7f3e24003d30 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:32.865 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.734+0000 7f3e4a185700 1 -- 172.21.15.195:0/3132049624 >> 172.21.15.195:0/3132049624 conn(0x7f3e44063af0 msgr2=0x7f3e44063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:03:32.866 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.734+0000 7f3e4a185700 1 -- 172.21.15.195:0/3132049624 shutdown_connections 2022-06-27T12:03:32.867 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.734+0000 7f3e4a185700 1 -- 172.21.15.195:0/3132049624 wait complete. 2022-06-27T12:03:32.867 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.736+0000 7f3e4a185700 1 Processor -- start 2022-06-27T12:03:32.868 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.736+0000 7f3e4a185700 1 -- start start 2022-06-27T12:03:32.868 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.736+0000 7f3e4a185700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3e24003920 0x7f3e44164e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:32.869 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.736+0000 7f3e4a185700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3e441582e0 0x7f3e4414f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:32.869 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.736+0000 7f3e4a185700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3e44151a50 0x7f3e44156a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:32.870 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.736+0000 7f3e4a185700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e44150be0 con 0x7f3e24003920 2022-06-27T12:03:32.870 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.736+0000 7f3e4a185700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e441653b0 con 0x7f3e44151a50 2022-06-27T12:03:32.870 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.736+0000 7f3e4a185700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e44153750 con 0x7f3e441582e0 2022-06-27T12:03:32.871 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.736+0000 7f3e437fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3e24003920 0x7f3e44164e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:32.871 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.737+0000 7f3e437fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3e24003920 0x7f3e44164e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47226/0 (socket says 172.21.15.195:47226) 2022-06-27T12:03:32.871 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.737+0000 7f3e437fe700 1 -- 172.21.15.195:0/2590056006 learned_addr learned my addr 172.21.15.195:0/2590056006 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:03:32.872 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.737+0000 7f3e43fff700 1 --2- 172.21.15.195:0/2590056006 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3e44151a50 0x7f3e44156a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:32.872 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.737+0000 7f3e42ffd700 1 --2- 172.21.15.195:0/2590056006 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3e441582e0 0x7f3e4414f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:32.872 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.737+0000 7f3e437fe700 1 -- 172.21.15.195:0/2590056006 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3e441582e0 msgr2=0x7f3e4414f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:32.872 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.737+0000 7f3e437fe700 1 --2- 172.21.15.195:0/2590056006 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3e441582e0 0x7f3e4414f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:32.873 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.737+0000 7f3e437fe700 1 -- 172.21.15.195:0/2590056006 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3e44151a50 msgr2=0x7f3e44156a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:32.873 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.737+0000 7f3e437fe700 1 --2- 172.21.15.195:0/2590056006 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3e44151a50 0x7f3e44156a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:32.873 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.737+0000 7f3e437fe700 1 -- 172.21.15.195:0/2590056006 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e44157e50 con 0x7f3e24003920 2022-06-27T12:03:32.873 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.737+0000 7f3e437fe700 1 --2- 172.21.15.195:0/2590056006 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3e24003920 0x7f3e44164e70 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f3e2400bf60 tx=0x7f3e2c015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:03:32.873 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.737+0000 7f3e3b7fe700 1 -- 172.21.15.195:0/2590056006 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e2c000bc0 con 0x7f3e24003920 2022-06-27T12:03:32.874 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.737+0000 7f3e3b7fe700 1 -- 172.21.15.195:0/2590056006 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e2c0046f0 con 0x7f3e24003920 2022-06-27T12:03:32.874 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.737+0000 7f3e4a185700 1 -- 172.21.15.195:0/2590056006 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e44157840 con 0x7f3e24003920 2022-06-27T12:03:32.875 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.737+0000 7f3e3b7fe700 1 -- 172.21.15.195:0/2590056006 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e2c000d20 con 0x7f3e24003920 2022-06-27T12:03:32.875 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.737+0000 7f3e4a185700 1 -- 172.21.15.195:0/2590056006 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e44153470 con 0x7f3e24003920 2022-06-27T12:03:32.875 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.738+0000 7f3e4a185700 1 -- 172.21.15.195:0/2590056006 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e440722c0 con 0x7f3e24003920 2022-06-27T12:03:32.876 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.738+0000 7f3e3b7fe700 1 -- 172.21.15.195:0/2590056006 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f3e2c024030 con 0x7f3e24003920 2022-06-27T12:03:32.876 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.739+0000 7f3e3b7fe700 1 --2- 172.21.15.195:0/2590056006 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3e28063d00 0x7f3e280661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:32.876 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.739+0000 7f3e3b7fe700 1 -- 172.21.15.195:0/2590056006 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6804+0+0 (secure 0 0 0) 0x7f3e2c0815d0 con 0x7f3e24003920 2022-06-27T12:03:32.876 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.739+0000 7f3e42ffd700 1 --2- 172.21.15.195:0/2590056006 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3e28063d00 0x7f3e280661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:32.877 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.739+0000 7f3e42ffd700 1 --2- 172.21.15.195:0/2590056006 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3e28063d00 0x7f3e280661b0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f3e441586d0 tx=0x7f3e34009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:03:32.878 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.743+0000 7f3e3b7fe700 1 -- 172.21.15.195:0/2590056006 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f3e2c04fed0 con 0x7f3e24003920 2022-06-27T12:03:32.969 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.969+0000 7f3e4a185700 1 -- 172.21.15.195:0/2590056006 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3e440c7250 con 0x7f3e24003920 2022-06-27T12:03:32.970 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.970+0000 7f3e3b7fe700 1 -- 172.21.15.195:0/2590056006 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+6125 (secure 0 0 0) 0x7f3e2c0534f0 con 0x7f3e24003920 2022-06-27T12:03:32.971 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:03:32.971 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":5818,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/3399027527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3399027527},{"type":"v1","addr":"172.21.15.202:6833","nonce":3399027527}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:02:00.272060+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5273":{"gid":5273,"name":"d","rank":0,"incarnation":49,"state":"up:active","state_seq":42,"addr":"172.21.15.202:6835/417105338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":417105338},{"type":"v1","addr":"172.21.15.202:6835","nonce":417105338}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:03:32.973 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.972+0000 7f3e4a185700 1 -- 172.21.15.195:0/2590056006 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3e28063d00 msgr2=0x7f3e280661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:32.973 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.972+0000 7f3e4a185700 1 --2- 172.21.15.195:0/2590056006 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3e28063d00 0x7f3e280661b0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f3e441586d0 tx=0x7f3e34009040 comp rx=0 tx=0).stop 2022-06-27T12:03:32.973 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.972+0000 7f3e4a185700 1 -- 172.21.15.195:0/2590056006 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3e24003920 msgr2=0x7f3e44164e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:32.974 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.972+0000 7f3e4a185700 1 --2- 172.21.15.195:0/2590056006 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3e24003920 0x7f3e44164e70 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f3e2400bf60 tx=0x7f3e2c015000 comp rx=0 tx=0).stop 2022-06-27T12:03:32.974 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.972+0000 7f3e4a185700 1 -- 172.21.15.195:0/2590056006 shutdown_connections 2022-06-27T12:03:32.974 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.972+0000 7f3e4a185700 1 --2- 172.21.15.195:0/2590056006 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3e28063d00 0x7f3e280661b0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:32.974 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.972+0000 7f3e4a185700 1 --2- 172.21.15.195:0/2590056006 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3e24003920 0x7f3e44164e70 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:32.975 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.972+0000 7f3e4a185700 1 --2- 172.21.15.195:0/2590056006 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3e44151a50 0x7f3e44156a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:32.975 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.972+0000 7f3e4a185700 1 --2- 172.21.15.195:0/2590056006 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3e441582e0 0x7f3e4414f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:32.976 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.972+0000 7f3e4a185700 1 -- 172.21.15.195:0/2590056006 >> 172.21.15.195:0/2590056006 conn(0x7f3e44063af0 msgr2=0x7f3e44152160 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:03:32.976 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.972+0000 7f3e4a185700 1 -- 172.21.15.195:0/2590056006 shutdown_connections 2022-06-27T12:03:32.976 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:32.972+0000 7f3e4a185700 1 -- 172.21.15.195:0/2590056006 wait complete. 2022-06-27T12:03:32.976 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 54 2022-06-27T12:03:32.984 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-06-27T12:03:34.988 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:03:35.131 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.129+0000 7fe12b02f700 1 Processor -- start 2022-06-27T12:03:35.131 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.129+0000 7fe12b02f700 1 -- start start 2022-06-27T12:03:35.131 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.130+0000 7fe12b02f700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fe1240c40d0 0x7fe124157010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:35.132 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.130+0000 7fe12b02f700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fe1240c6190 0x7fe124067ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:35.132 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.130+0000 7fe12b02f700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe124156b80 con 0x7fe1240c44a0 2022-06-27T12:03:35.132 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.130+0000 7fe12b02f700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe12406a040 con 0x7fe1240c40d0 2022-06-27T12:03:35.132 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.130+0000 7fe12b02f700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe12406a300 con 0x7fe1240c6190 2022-06-27T12:03:35.133 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.130+0000 7fe128dcb700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fe1240c40d0 0x7fe124157010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:35.133 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.130+0000 7fe1295cc700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fe1240c44a0 0x7fe12414cb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:53422/0 (socket says 172.21.15.195:53422) 2022-06-27T12:03:35.133 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.130+0000 7fe128dcb700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fe1240c40d0 0x7fe124157010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47234/0 (socket says 172.21.15.195:47234) 2022-06-27T12:03:35.133 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.130+0000 7fe1295cc700 1 -- 172.21.15.195:0/3945082316 learned_addr learned my addr 172.21.15.195:0/3945082316 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:03:35.136 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.130+0000 7fe123fff700 1 --2- 172.21.15.195:0/3945082316 >> v2:172.21.15.202:3300/0 conn(0x7fe1240c6190 0x7fe124067ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:35.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.131+0000 7fe128dcb700 1 -- 172.21.15.195:0/3945082316 >> v2:172.21.15.202:3300/0 conn(0x7fe1240c6190 msgr2=0x7fe124067ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:03:35.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.131+0000 7fe128dcb700 1 --2- 172.21.15.195:0/3945082316 >> v2:172.21.15.202:3300/0 conn(0x7fe1240c6190 0x7fe124067ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:35.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.131+0000 7fe128dcb700 1 -- 172.21.15.195:0/3945082316 >> v1:172.21.15.195:6789/0 conn(0x7fe1240c44a0 legacy=0x7fe12414cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:35.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.131+0000 7fe1237fe700 1 -- 172.21.15.195:0/3945082316 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1222993181 0 0) 0x7fe124156b80 con 0x7fe1240c44a0 2022-06-27T12:03:35.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.131+0000 7fe128dcb700 1 -- 172.21.15.195:0/3945082316 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe12406a5e0 con 0x7fe1240c40d0 2022-06-27T12:03:35.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.131+0000 7fe128dcb700 1 --2- 172.21.15.195:0/3945082316 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe1240c40d0 0x7fe124157010 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fe1240c0680 tx=0x7fe118009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a362c11cc385cee server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:03:35.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.131+0000 7fe1237fe700 1 -- 172.21.15.195:0/3945082316 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe118011630 con 0x7fe1240c40d0 2022-06-27T12:03:35.155 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.131+0000 7fe1237fe700 1 -- 172.21.15.195:0/3945082316 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe118011790 con 0x7fe1240c40d0 2022-06-27T12:03:35.155 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.131+0000 7fe1237fe700 1 -- 172.21.15.195:0/3945082316 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe118011b30 con 0x7fe1240c40d0 2022-06-27T12:03:35.155 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.131+0000 7fe12b02f700 1 -- 172.21.15.195:0/3945082316 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe1240c40d0 msgr2=0x7fe124157010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:35.155 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.131+0000 7fe12b02f700 1 --2- 172.21.15.195:0/3945082316 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe1240c40d0 0x7fe124157010 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fe1240c0680 tx=0x7fe118009060 comp rx=0 tx=0).stop 2022-06-27T12:03:35.156 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.131+0000 7fe12b02f700 1 -- 172.21.15.195:0/3945082316 shutdown_connections 2022-06-27T12:03:35.156 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.131+0000 7fe12b02f700 1 --2- 172.21.15.195:0/3945082316 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe1240c40d0 0x7fe124157010 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:35.156 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.131+0000 7fe12b02f700 1 --2- 172.21.15.195:0/3945082316 >> v2:172.21.15.202:3300/0 conn(0x7fe1240c6190 0x7fe124067ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:35.156 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.131+0000 7fe12b02f700 1 -- 172.21.15.195:0/3945082316 >> 172.21.15.195:0/3945082316 conn(0x7fe1240ccb80 msgr2=0x7fe1240ccf80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:03:35.156 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.132+0000 7fe12b02f700 1 -- 172.21.15.195:0/3945082316 shutdown_connections 2022-06-27T12:03:35.157 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.132+0000 7fe12b02f700 1 -- 172.21.15.195:0/3945082316 wait complete. 2022-06-27T12:03:35.157 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.133+0000 7fe12b02f700 1 Processor -- start 2022-06-27T12:03:35.157 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.133+0000 7fe12b02f700 1 -- start start 2022-06-27T12:03:35.157 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.134+0000 7fe12b02f700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe1240c40d0 0x7fe124166400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:35.158 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.134+0000 7fe12b02f700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe1240c44a0 0x7fe1240c2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:35.159 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.134+0000 7fe12b02f700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe1240c6190 0x7fe1240c0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:35.159 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.134+0000 7fe12b02f700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1240c3750 con 0x7fe1240c40d0 2022-06-27T12:03:35.159 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.134+0000 7fe12b02f700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1240c2640 con 0x7fe1240c44a0 2022-06-27T12:03:35.160 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.134+0000 7fe12b02f700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe124167850 con 0x7fe1240c6190 2022-06-27T12:03:35.160 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.134+0000 7fe128dcb700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe1240c40d0 0x7fe124166400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:35.160 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.134+0000 7fe128dcb700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe1240c40d0 0x7fe124166400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47240/0 (socket says 172.21.15.195:47240) 2022-06-27T12:03:35.161 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.134+0000 7fe128dcb700 1 -- 172.21.15.195:0/2481308573 learned_addr learned my addr 172.21.15.195:0/2481308573 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:03:35.161 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.135+0000 7fe1295cc700 1 --2- 172.21.15.195:0/2481308573 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe1240c6190 0x7fe1240c0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:35.161 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.135+0000 7fe123fff700 1 --2- 172.21.15.195:0/2481308573 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe1240c44a0 0x7fe1240c2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:35.162 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.135+0000 7fe128dcb700 1 -- 172.21.15.195:0/2481308573 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe1240c6190 msgr2=0x7fe1240c0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:35.162 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.135+0000 7fe128dcb700 1 --2- 172.21.15.195:0/2481308573 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe1240c6190 0x7fe1240c0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:35.162 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.135+0000 7fe128dcb700 1 -- 172.21.15.195:0/2481308573 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe1240c44a0 msgr2=0x7fe1240c2ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:35.162 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.135+0000 7fe128dcb700 1 --2- 172.21.15.195:0/2481308573 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe1240c44a0 0x7fe1240c2ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:35.163 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.135+0000 7fe128dcb700 1 -- 172.21.15.195:0/2481308573 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1180092e0 con 0x7fe1240c40d0 2022-06-27T12:03:35.163 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.135+0000 7fe1295cc700 1 --2- 172.21.15.195:0/2481308573 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe1240c6190 0x7fe1240c0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:03:35.163 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.135+0000 7fe123fff700 1 --2- 172.21.15.195:0/2481308573 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe1240c44a0 0x7fe1240c2ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:03:35.163 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.135+0000 7fe128dcb700 1 --2- 172.21.15.195:0/2481308573 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe1240c40d0 0x7fe124166400 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fe118009e10 tx=0x7fe118017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:03:35.164 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.135+0000 7fe120ff9700 1 -- 172.21.15.195:0/2481308573 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe118000c90 con 0x7fe1240c40d0 2022-06-27T12:03:35.164 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.135+0000 7fe120ff9700 1 -- 172.21.15.195:0/2481308573 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe118011900 con 0x7fe1240c40d0 2022-06-27T12:03:35.164 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.135+0000 7fe120ff9700 1 -- 172.21.15.195:0/2481308573 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe118025420 con 0x7fe1240c40d0 2022-06-27T12:03:35.164 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.135+0000 7fe12b02f700 1 -- 172.21.15.195:0/2481308573 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1240c13f0 con 0x7fe1240c40d0 2022-06-27T12:03:35.164 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.135+0000 7fe12b02f700 1 -- 172.21.15.195:0/2481308573 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1240c1990 con 0x7fe1240c40d0 2022-06-27T12:03:35.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.136+0000 7fe12b02f700 1 -- 172.21.15.195:0/2481308573 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1240c2920 con 0x7fe1240c40d0 2022-06-27T12:03:35.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.137+0000 7fe120ff9700 1 -- 172.21.15.195:0/2481308573 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fe11800f050 con 0x7fe1240c40d0 2022-06-27T12:03:35.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.137+0000 7fe120ff9700 1 --2- 172.21.15.195:0/2481308573 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe10c063d00 0x7fe10c0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:35.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.137+0000 7fe120ff9700 1 -- 172.21.15.195:0/2481308573 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6890+0+0 (secure 0 0 0) 0x7fe118081f60 con 0x7fe1240c40d0 2022-06-27T12:03:35.166 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.137+0000 7fe123fff700 1 --2- 172.21.15.195:0/2481308573 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe10c063d00 0x7fe10c0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:35.166 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.137+0000 7fe123fff700 1 --2- 172.21.15.195:0/2481308573 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe10c063d00 0x7fe10c0661b0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fe124156c90 tx=0x7fe110008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:03:35.166 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.141+0000 7fe120ff9700 1 -- 172.21.15.195:0/2481308573 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fe1180b1c90 con 0x7fe1240c40d0 2022-06-27T12:03:35.369 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.367+0000 7fe12b02f700 1 -- 172.21.15.195:0/2481308573 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe1240722c0 con 0x7fe1240c40d0 2022-06-27T12:03:35.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.368+0000 7fe120ff9700 1 -- 172.21.15.195:0/2481308573 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+5367 (secure 0 0 0) 0x7fe118015070 con 0x7fe1240c40d0 2022-06-27T12:03:35.371 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:03:35.371 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":55,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:03:35.042639+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5818},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5818":{"gid":5818,"name":"b","rank":0,"incarnation":55,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6833/3399027527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3399027527},{"type":"v1","addr":"172.21.15.202:6833","nonce":3399027527}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:03:35.373 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.370+0000 7fe12b02f700 1 -- 172.21.15.195:0/2481308573 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe10c063d00 msgr2=0x7fe10c0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:35.374 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.370+0000 7fe12b02f700 1 --2- 172.21.15.195:0/2481308573 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe10c063d00 0x7fe10c0661b0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fe124156c90 tx=0x7fe110008040 comp rx=0 tx=0).stop 2022-06-27T12:03:35.374 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.370+0000 7fe12b02f700 1 -- 172.21.15.195:0/2481308573 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe1240c40d0 msgr2=0x7fe124166400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:35.375 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.370+0000 7fe12b02f700 1 --2- 172.21.15.195:0/2481308573 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe1240c40d0 0x7fe124166400 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fe118009e10 tx=0x7fe118017000 comp rx=0 tx=0).stop 2022-06-27T12:03:35.376 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.370+0000 7fe12b02f700 1 -- 172.21.15.195:0/2481308573 shutdown_connections 2022-06-27T12:03:35.376 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.370+0000 7fe12b02f700 1 --2- 172.21.15.195:0/2481308573 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe10c063d00 0x7fe10c0661b0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:35.376 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.370+0000 7fe12b02f700 1 --2- 172.21.15.195:0/2481308573 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe1240c40d0 0x7fe124166400 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:35.377 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.370+0000 7fe12b02f700 1 --2- 172.21.15.195:0/2481308573 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe1240c6190 0x7fe1240c0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:35.377 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.370+0000 7fe12b02f700 1 --2- 172.21.15.195:0/2481308573 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe1240c44a0 0x7fe1240c2ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:35.377 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.370+0000 7fe12b02f700 1 -- 172.21.15.195:0/2481308573 >> 172.21.15.195:0/2481308573 conn(0x7fe1240ccb80 msgr2=0x7fe124150280 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:03:35.377 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.370+0000 7fe12b02f700 1 -- 172.21.15.195:0/2481308573 shutdown_connections 2022-06-27T12:03:35.378 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.370+0000 7fe12b02f700 1 -- 172.21.15.195:0/2481308573 wait complete. 2022-06-27T12:03:35.378 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 55 2022-06-27T12:03:35.382 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d down, removed from mdsmap 2022-06-27T12:03:35.382 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2022-06-27T12:03:35.383 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:03:35.524 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.506+0000 7fb555297700 1 Processor -- start 2022-06-27T12:03:35.524 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.506+0000 7fb555297700 1 -- start start 2022-06-27T12:03:35.528 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.507+0000 7fb555297700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb5500c40d0 0x7fb550157010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:35.528 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.507+0000 7fb555297700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fb550067ed0 0x7fb55014cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:35.528 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.507+0000 7fb555297700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb550156b80 con 0x7fb5500c6190 2022-06-27T12:03:35.529 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.507+0000 7fb555297700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb550068370 con 0x7fb550067ed0 2022-06-27T12:03:35.529 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.507+0000 7fb555297700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb55006a040 con 0x7fb5500c40d0 2022-06-27T12:03:35.529 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.507+0000 7fb54f7fe700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fb550067ed0 0x7fb55014cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:35.529 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.507+0000 7fb54f7fe700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fb550067ed0 0x7fb55014cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47248/0 (socket says 172.21.15.195:47248) 2022-06-27T12:03:35.530 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.507+0000 7fb54f7fe700 1 -- 172.21.15.195:0/144418812 learned_addr learned my addr 172.21.15.195:0/144418812 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:03:35.530 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.507+0000 7fb54effd700 1 --2- 172.21.15.195:0/144418812 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb5500c40d0 0x7fb550157010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:35.530 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.507+0000 7fb54f7fe700 1 -- 172.21.15.195:0/144418812 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb5500c40d0 msgr2=0x7fb550157010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:03:35.531 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.507+0000 7fb54f7fe700 1 --2- 172.21.15.195:0/144418812 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb5500c40d0 0x7fb550157010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:35.531 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.507+0000 7fb54f7fe700 1 -- 172.21.15.195:0/144418812 >> v1:172.21.15.202:6789/0 conn(0x7fb5500c6190 legacy=0x7fb5500c44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:35.532 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.507+0000 7fb54f7fe700 1 -- 172.21.15.195:0/144418812 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb55006a300 con 0x7fb550067ed0 2022-06-27T12:03:35.532 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.507+0000 7fb54effd700 1 --2- 172.21.15.195:0/144418812 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb5500c40d0 0x7fb550157010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T12:03:35.532 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.507+0000 7fb54f7fe700 1 --2- 172.21.15.195:0/144418812 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb550067ed0 0x7fb55014cb80 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fb548009f20 tx=0x7fb548009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5668649fe5ac557b server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:03:35.532 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.508+0000 7fb54dffb700 1 -- 172.21.15.195:0/144418812 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb54800e070 con 0x7fb550067ed0 2022-06-27T12:03:35.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.508+0000 7fb54dffb700 1 -- 172.21.15.195:0/144418812 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb54800ba80 con 0x7fb550067ed0 2022-06-27T12:03:35.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.508+0000 7fb54dffb700 1 -- 172.21.15.195:0/144418812 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5480092e0 con 0x7fb550067ed0 2022-06-27T12:03:35.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.508+0000 7fb555297700 1 -- 172.21.15.195:0/144418812 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb550067ed0 msgr2=0x7fb55014cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:35.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.508+0000 7fb555297700 1 --2- 172.21.15.195:0/144418812 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb550067ed0 0x7fb55014cb80 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fb548009f20 tx=0x7fb548009840 comp rx=0 tx=0).stop 2022-06-27T12:03:35.534 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.508+0000 7fb555297700 1 -- 172.21.15.195:0/144418812 shutdown_connections 2022-06-27T12:03:35.534 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.508+0000 7fb555297700 1 --2- 172.21.15.195:0/144418812 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb550067ed0 0x7fb55014cb80 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:35.534 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.508+0000 7fb555297700 1 --2- 172.21.15.195:0/144418812 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb5500c40d0 0x7fb550157010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:35.534 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.508+0000 7fb555297700 1 -- 172.21.15.195:0/144418812 >> 172.21.15.195:0/144418812 conn(0x7fb5500ccb80 msgr2=0x7fb5500ccf80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:03:35.535 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.508+0000 7fb555297700 1 -- 172.21.15.195:0/144418812 shutdown_connections 2022-06-27T12:03:35.535 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.508+0000 7fb555297700 1 -- 172.21.15.195:0/144418812 wait complete. 2022-06-27T12:03:35.535 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.509+0000 7fb555297700 1 Processor -- start 2022-06-27T12:03:35.535 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.509+0000 7fb555297700 1 -- start start 2022-06-27T12:03:35.536 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.510+0000 7fb555297700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb550067ed0 0x7fb5500c13e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:35.536 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.510+0000 7fb555297700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb5500c40d0 0x7fb5500c1d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:35.536 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.510+0000 7fb555297700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb5500c6190 0x7fb5500c22c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:35.536 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.510+0000 7fb555297700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb550152140 con 0x7fb5500c6190 2022-06-27T12:03:35.537 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.510+0000 7fb555297700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5501529e0 con 0x7fb550067ed0 2022-06-27T12:03:35.537 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.510+0000 7fb555297700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb550150e70 con 0x7fb5500c40d0 2022-06-27T12:03:35.537 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.510+0000 7fb54f7fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb5500c6190 0x7fb5500c22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:35.537 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.510+0000 7fb54f7fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb5500c6190 0x7fb5500c22c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47258/0 (socket says 172.21.15.195:47258) 2022-06-27T12:03:35.540 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.510+0000 7fb54f7fe700 1 -- 172.21.15.195:0/893848644 learned_addr learned my addr 172.21.15.195:0/893848644 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:03:35.540 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.510+0000 7fb54effd700 1 --2- 172.21.15.195:0/893848644 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb550067ed0 0x7fb5500c13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:35.540 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.510+0000 7fb54e7fc700 1 --2- 172.21.15.195:0/893848644 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb5500c40d0 0x7fb5500c1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:35.541 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.511+0000 7fb54f7fe700 1 -- 172.21.15.195:0/893848644 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb5500c40d0 msgr2=0x7fb5500c1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:35.541 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.511+0000 7fb54f7fe700 1 --2- 172.21.15.195:0/893848644 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb5500c40d0 0x7fb5500c1d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:35.542 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.511+0000 7fb54f7fe700 1 -- 172.21.15.195:0/893848644 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb550067ed0 msgr2=0x7fb5500c13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:35.542 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.511+0000 7fb54f7fe700 1 --2- 172.21.15.195:0/893848644 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb550067ed0 0x7fb5500c13e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:35.542 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.511+0000 7fb54f7fe700 1 -- 172.21.15.195:0/893848644 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb548009ae0 con 0x7fb5500c6190 2022-06-27T12:03:35.542 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.511+0000 7fb54effd700 1 --2- 172.21.15.195:0/893848644 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb550067ed0 0x7fb5500c13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:03:35.543 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.511+0000 7fb54e7fc700 1 --2- 172.21.15.195:0/893848644 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb5500c40d0 0x7fb5500c1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:03:35.543 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.511+0000 7fb54f7fe700 1 --2- 172.21.15.195:0/893848644 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb5500c6190 0x7fb5500c22c0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fb548009430 tx=0x7fb548018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:03:35.543 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.511+0000 7fb536ffd700 1 -- 172.21.15.195:0/893848644 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5480094e0 con 0x7fb5500c6190 2022-06-27T12:03:35.544 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.511+0000 7fb536ffd700 1 -- 172.21.15.195:0/893848644 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb548007820 con 0x7fb5500c6190 2022-06-27T12:03:35.544 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.511+0000 7fb536ffd700 1 -- 172.21.15.195:0/893848644 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb548007be0 con 0x7fb5500c6190 2022-06-27T12:03:35.545 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.511+0000 7fb555297700 1 -- 172.21.15.195:0/893848644 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb550151150 con 0x7fb5500c6190 2022-06-27T12:03:35.545 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.511+0000 7fb555297700 1 -- 172.21.15.195:0/893848644 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb550152670 con 0x7fb5500c6190 2022-06-27T12:03:35.545 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.512+0000 7fb555297700 1 -- 172.21.15.195:0/893848644 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5500722c0 con 0x7fb5500c6190 2022-06-27T12:03:35.546 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.512+0000 7fb536ffd700 1 -- 172.21.15.195:0/893848644 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fb548004040 con 0x7fb5500c6190 2022-06-27T12:03:35.546 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.513+0000 7fb536ffd700 1 --2- 172.21.15.195:0/893848644 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb530063d00 0x7fb5300661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:35.546 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.513+0000 7fb536ffd700 1 -- 172.21.15.195:0/893848644 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6890+0+0 (secure 0 0 0) 0x7fb548082c30 con 0x7fb5500c6190 2022-06-27T12:03:35.546 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.513+0000 7fb54effd700 1 --2- 172.21.15.195:0/893848644 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb530063d00 0x7fb5300661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:35.547 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.513+0000 7fb54effd700 1 --2- 172.21.15.195:0/893848644 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb530063d00 0x7fb5300661b0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fb54400aa60 tx=0x7fb544008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:03:35.547 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.517+0000 7fb536ffd700 1 -- 172.21.15.195:0/893848644 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fb548051480 con 0x7fb5500c6190 2022-06-27T12:03:35.747 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.745+0000 7fb555297700 1 -- 172.21.15.195:0/893848644 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb5500652c0 con 0x7fb5500c6190 2022-06-27T12:03:35.747 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.745+0000 7fb536ffd700 1 -- 172.21.15.195:0/893848644 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+5367 (secure 0 0 0) 0x7fb548054aa0 con 0x7fb5500c6190 2022-06-27T12:03:35.748 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:03:35.748 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":55,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:03:35.042639+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5818},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5818":{"gid":5818,"name":"b","rank":0,"incarnation":55,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6833/3399027527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3399027527},{"type":"v1","addr":"172.21.15.202:6833","nonce":3399027527}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:03:35.751 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.748+0000 7fb555297700 1 -- 172.21.15.195:0/893848644 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb530063d00 msgr2=0x7fb5300661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:35.751 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.748+0000 7fb555297700 1 --2- 172.21.15.195:0/893848644 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb530063d00 0x7fb5300661b0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fb54400aa60 tx=0x7fb544008040 comp rx=0 tx=0).stop 2022-06-27T12:03:35.751 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.748+0000 7fb555297700 1 -- 172.21.15.195:0/893848644 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb5500c6190 msgr2=0x7fb5500c22c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:35.752 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.748+0000 7fb555297700 1 --2- 172.21.15.195:0/893848644 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb5500c6190 0x7fb5500c22c0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fb548009430 tx=0x7fb548018000 comp rx=0 tx=0).stop 2022-06-27T12:03:35.752 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.748+0000 7fb555297700 1 -- 172.21.15.195:0/893848644 shutdown_connections 2022-06-27T12:03:35.752 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.748+0000 7fb555297700 1 --2- 172.21.15.195:0/893848644 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb530063d00 0x7fb5300661b0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:35.752 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.748+0000 7fb555297700 1 --2- 172.21.15.195:0/893848644 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb5500c6190 0x7fb5500c22c0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:35.753 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.748+0000 7fb555297700 1 --2- 172.21.15.195:0/893848644 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb550067ed0 0x7fb5500c13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:35.753 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.748+0000 7fb555297700 1 --2- 172.21.15.195:0/893848644 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb5500c40d0 0x7fb5500c1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:35.753 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.748+0000 7fb555297700 1 -- 172.21.15.195:0/893848644 >> 172.21.15.195:0/893848644 conn(0x7fb5500ccb80 msgr2=0x7fb5500c9620 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:03:35.753 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.748+0000 7fb555297700 1 -- 172.21.15.195:0/893848644 shutdown_connections 2022-06-27T12:03:35.754 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:35.748+0000 7fb555297700 1 -- 172.21.15.195:0/893848644 wait complete. 2022-06-27T12:03:35.754 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 55 2022-06-27T12:03:35.768 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 55, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-06-27T11:47:52.358095+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 55, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5818": { "addr": "172.21.15.202:6833/3399027527", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6832", "nonce": 3399027527, "type": "v2" }, { "addr": "172.21.15.202:6833", "nonce": 3399027527, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5818, "incarnation": 55, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 56, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-06-27T12:03:35.042639+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5818 } } } ], "standbys": [ { "addr": "172.21.15.202:6837/3305006841", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6836", "nonce": 3305006841, "type": "v2" }, { "addr": "172.21.15.202:6837", "nonce": 3305006841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6835/2060709822", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6834", "nonce": 2060709822, "type": "v2" }, { "addr": "172.21.15.195:6835", "nonce": 2060709822, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6838/1598751652", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6836", "nonce": 1598751652, "type": "v2" }, { "addr": "172.21.15.195:6838", "nonce": 1598751652, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4275, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6839/1313800661", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6837", "nonce": 1313800661, "type": "v2" }, { "addr": "172.21.15.195:6839", "nonce": 1313800661, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4278, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-06-27T12:03:37.770 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:03:37.917 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:37.903+0000 7fc827d98700 1 Processor -- start 2022-06-27T12:03:37.918 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:37.904+0000 7fc827d98700 1 -- start start 2022-06-27T12:03:37.919 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:37.904+0000 7fc827d98700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc82014e860 0x7fc8200c3840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:37.919 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:37.904+0000 7fc827d98700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fc8200c3d80 0x7fc820157890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:37.920 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:37.904+0000 7fc827d98700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8200c33b0 con 0x7fc82014e490 2022-06-27T12:03:37.921 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:37.904+0000 7fc827d98700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8200c4200 con 0x7fc8200c3d80 2022-06-27T12:03:37.921 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:37.904+0000 7fc827d98700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc82006f220 con 0x7fc82014e860 2022-06-27T12:03:37.921 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:37.905+0000 7fc825333700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc82014e860 0x7fc8200c3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:37.922 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:37.905+0000 7fc826335700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fc8200c3d80 0x7fc820157890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:37.923 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:37.905+0000 7fc826335700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fc8200c3d80 0x7fc820157890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3300/0 says I am v2:172.21.15.195:45396/0 (socket says 172.21.15.195:45396) 2022-06-27T12:03:37.924 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:37.905+0000 7fc825333700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc82014e860 0x7fc8200c3840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3301/0 says I am v2:172.21.15.195:40022/0 (socket says 172.21.15.195:40022) 2022-06-27T12:03:37.925 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:37.905+0000 7fc826335700 1 -- 172.21.15.195:0/1088637237 learned_addr learned my addr 172.21.15.195:0/1088637237 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:03:37.925 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:37.905+0000 7fc825b34700 1 --1- >> v1:172.21.15.202:6789/0 conn(0x7fc82014e490 0x7fc82006cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.202:6789/0 says I am v1:172.21.15.195:55314/0 (socket says 172.21.15.195:55314) 2022-06-27T12:03:37.926 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:37.905+0000 7fc826335700 1 -- 172.21.15.195:0/1088637237 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc82014e860 msgr2=0x7fc8200c3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:03:37.926 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:37.905+0000 7fc826335700 1 --2- 172.21.15.195:0/1088637237 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc82014e860 0x7fc8200c3840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:37.927 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:37.905+0000 7fc826335700 1 -- 172.21.15.195:0/1088637237 >> v1:172.21.15.202:6789/0 conn(0x7fc82014e490 legacy=0x7fc82006cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:37.927 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:37.905+0000 7fc824b32700 1 -- 172.21.15.195:0/1088637237 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 800904279 0 0) 0x7fc8200c33b0 con 0x7fc82014e490 2022-06-27T12:03:37.928 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:37.905+0000 7fc826335700 1 -- 172.21.15.195:0/1088637237 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8201542c0 con 0x7fc8200c3d80 2022-06-27T12:03:37.928 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:37.905+0000 7fc825333700 1 --2- 172.21.15.195:0/1088637237 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc82014e860 0x7fc8200c3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:03:37.929 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:37.905+0000 7fc826335700 1 --2- 172.21.15.195:0/1088637237 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc8200c3d80 0x7fc820157890 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fc818009f20 tx=0x7fc818009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=adf108069eca64fd server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:03:37.929 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:37.905+0000 7fc824b32700 1 -- 172.21.15.195:0/1088637237 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc81800e070 con 0x7fc8200c3d80 2022-06-27T12:03:37.929 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:37.905+0000 7fc824b32700 1 -- 172.21.15.195:0/1088637237 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc81800b9d0 con 0x7fc8200c3d80 2022-06-27T12:03:37.930 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:37.905+0000 7fc824b32700 1 -- 172.21.15.195:0/1088637237 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8180092e0 con 0x7fc8200c3d80 2022-06-27T12:03:37.930 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:37.906+0000 7fc827d98700 1 -- 172.21.15.195:0/1088637237 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc8200c3d80 msgr2=0x7fc820157890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:37.931 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:37.906+0000 7fc827d98700 1 --2- 172.21.15.195:0/1088637237 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc8200c3d80 0x7fc820157890 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fc818009f20 tx=0x7fc818009840 comp rx=0 tx=0).stop 2022-06-27T12:03:37.931 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:37.906+0000 7fc827d98700 1 -- 172.21.15.195:0/1088637237 shutdown_connections 2022-06-27T12:03:37.931 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:37.906+0000 7fc827d98700 1 --2- 172.21.15.195:0/1088637237 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc8200c3d80 0x7fc820157890 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:37.932 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:37.906+0000 7fc827d98700 1 --2- 172.21.15.195:0/1088637237 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc82014e860 0x7fc8200c3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:37.932 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:37.906+0000 7fc827d98700 1 -- 172.21.15.195:0/1088637237 >> 172.21.15.195:0/1088637237 conn(0x7fc820063af0 msgr2=0x7fc820063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:03:37.933 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:37.906+0000 7fc827d98700 1 -- 172.21.15.195:0/1088637237 shutdown_connections 2022-06-27T12:03:37.934 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:37.906+0000 7fc827d98700 1 -- 172.21.15.195:0/1088637237 wait complete. 2022-06-27T12:03:37.934 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:37.907+0000 7fc827d98700 1 Processor -- start 2022-06-27T12:03:37.935 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:37.907+0000 7fc827d98700 1 -- start start 2022-06-27T12:03:37.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:37.908+0000 7fc827d98700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc8200c3d80 0x7fc820153620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:37.942 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:37.908+0000 7fc827d98700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc82014e490 0x7fc820153b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:37.943 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:37.908+0000 7fc827d98700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc82014e860 0x7fc82014a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:37.964 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:37.908+0000 7fc827d98700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc820164180 con 0x7fc8200c3d80 2022-06-27T12:03:37.965 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:37.908+0000 7fc827d98700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc820152860 con 0x7fc82014e490 2022-06-27T12:03:37.965 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:37.908+0000 7fc827d98700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc820152b40 con 0x7fc82014e860 2022-06-27T12:03:37.966 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:37.908+0000 7fc825b34700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc8200c3d80 0x7fc820153620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:37.966 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:37.908+0000 7fc825b34700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc8200c3d80 0x7fc820153620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47274/0 (socket says 172.21.15.195:47274) 2022-06-27T12:03:37.967 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:37.908+0000 7fc825333700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc82014e490 0x7fc820153b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:37.967 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:37.908+0000 7fc825b34700 1 -- 172.21.15.195:0/1992131449 learned_addr learned my addr 172.21.15.195:0/1992131449 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:03:37.968 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:37.908+0000 7fc826335700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc82014e860 0x7fc82014a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:37.968 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:37.908+0000 7fc825b34700 1 -- 172.21.15.195:0/1992131449 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc82014e860 msgr2=0x7fc82014a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:37.968 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:37.908+0000 7fc825b34700 1 --2- 172.21.15.195:0/1992131449 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc82014e860 0x7fc82014a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:37.969 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:37.908+0000 7fc825b34700 1 -- 172.21.15.195:0/1992131449 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc82014e490 msgr2=0x7fc820153b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:37.969 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:37.908+0000 7fc825b34700 1 --2- 172.21.15.195:0/1992131449 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc82014e490 0x7fc820153b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:37.970 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:37.908+0000 7fc825b34700 1 -- 172.21.15.195:0/1992131449 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc81c0092e0 con 0x7fc8200c3d80 2022-06-27T12:03:37.970 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:37.908+0000 7fc825b34700 1 --2- 172.21.15.195:0/1992131449 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc8200c3d80 0x7fc820153620 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fc81000c590 tx=0x7fc810007cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:03:37.971 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:37.909+0000 7fc80dffb700 1 -- 172.21.15.195:0/1992131449 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc810007360 con 0x7fc8200c3d80 2022-06-27T12:03:37.971 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:37.909+0000 7fc80dffb700 1 -- 172.21.15.195:0/1992131449 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc810007500 con 0x7fc8200c3d80 2022-06-27T12:03:37.971 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:37.909+0000 7fc827d98700 1 -- 172.21.15.195:0/1992131449 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc818009ae0 con 0x7fc8200c3d80 2022-06-27T12:03:37.972 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:37.909+0000 7fc80dffb700 1 -- 172.21.15.195:0/1992131449 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc810004940 con 0x7fc8200c3d80 2022-06-27T12:03:37.975 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:37.909+0000 7fc827d98700 1 -- 172.21.15.195:0/1992131449 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc820153100 con 0x7fc8200c3d80 2022-06-27T12:03:37.975 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:37.910+0000 7fc827d98700 1 -- 172.21.15.195:0/1992131449 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8200722c0 con 0x7fc8200c3d80 2022-06-27T12:03:37.976 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:37.910+0000 7fc80dffb700 1 -- 172.21.15.195:0/1992131449 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fc810010030 con 0x7fc8200c3d80 2022-06-27T12:03:37.976 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:37.910+0000 7fc80dffb700 1 --2- 172.21.15.195:0/1992131449 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc800063d00 0x7fc8000661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:37.976 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:37.911+0000 7fc80dffb700 1 -- 172.21.15.195:0/1992131449 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6890+0+0 (secure 0 0 0) 0x7fc8100805e0 con 0x7fc8200c3d80 2022-06-27T12:03:37.977 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:37.911+0000 7fc825333700 1 --2- 172.21.15.195:0/1992131449 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc800063d00 0x7fc8000661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:37.977 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:37.911+0000 7fc825333700 1 --2- 172.21.15.195:0/1992131449 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc800063d00 0x7fc8000661b0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fc8200c8490 tx=0x7fc81c009060 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:03:37.978 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:37.915+0000 7fc80dffb700 1 -- 172.21.15.195:0/1992131449 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fc81004ef10 con 0x7fc8200c3d80 2022-06-27T12:03:38.145 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:38.141+0000 7fc827d98700 1 -- 172.21.15.195:0/1992131449 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc8200c2230 con 0x7fc8200c3d80 2022-06-27T12:03:38.145 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:38.141+0000 7fc80dffb700 1 -- 172.21.15.195:0/1992131449 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+5367 (secure 0 0 0) 0x7fc810052530 con 0x7fc8200c3d80 2022-06-27T12:03:38.160 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:03:38.161 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":55,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:03:35.042639+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5818},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5818":{"gid":5818,"name":"b","rank":0,"incarnation":55,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6833/3399027527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3399027527},{"type":"v1","addr":"172.21.15.202:6833","nonce":3399027527}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:03:38.168 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:38.143+0000 7fc827d98700 1 -- 172.21.15.195:0/1992131449 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc800063d00 msgr2=0x7fc8000661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:38.168 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:38.143+0000 7fc827d98700 1 --2- 172.21.15.195:0/1992131449 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc800063d00 0x7fc8000661b0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fc8200c8490 tx=0x7fc81c009060 comp rx=0 tx=0).stop 2022-06-27T12:03:38.169 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:38.143+0000 7fc827d98700 1 -- 172.21.15.195:0/1992131449 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc8200c3d80 msgr2=0x7fc820153620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:38.169 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:38.143+0000 7fc827d98700 1 --2- 172.21.15.195:0/1992131449 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc8200c3d80 0x7fc820153620 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fc81000c590 tx=0x7fc810007cb0 comp rx=0 tx=0).stop 2022-06-27T12:03:38.170 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:38.143+0000 7fc827d98700 1 -- 172.21.15.195:0/1992131449 shutdown_connections 2022-06-27T12:03:38.170 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:38.143+0000 7fc827d98700 1 --2- 172.21.15.195:0/1992131449 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc800063d00 0x7fc8000661b0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:38.171 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:38.143+0000 7fc827d98700 1 --2- 172.21.15.195:0/1992131449 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc8200c3d80 0x7fc820153620 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:38.171 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:38.143+0000 7fc827d98700 1 --2- 172.21.15.195:0/1992131449 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc82014e860 0x7fc82014a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:38.171 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:38.143+0000 7fc827d98700 1 --2- 172.21.15.195:0/1992131449 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc82014e490 0x7fc820153b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:38.172 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:38.143+0000 7fc827d98700 1 -- 172.21.15.195:0/1992131449 >> 172.21.15.195:0/1992131449 conn(0x7fc820063af0 msgr2=0x7fc8201455a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:03:38.172 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:38.144+0000 7fc827d98700 1 -- 172.21.15.195:0/1992131449 shutdown_connections 2022-06-27T12:03:38.175 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:38.144+0000 7fc827d98700 1 -- 172.21.15.195:0/1992131449 wait complete. 2022-06-27T12:03:38.175 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 55 2022-06-27T12:03:38.176 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:03:40.177 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:03:40.313 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.301+0000 7f3d6e536700 1 Processor -- start 2022-06-27T12:03:40.313 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.301+0000 7f3d6e536700 1 -- start start 2022-06-27T12:03:40.313 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.302+0000 7f3d6e536700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3d6806e850 0x7f3d681570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:40.314 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.302+0000 7f3d6e536700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f3d6806fef0 0x7f3d68069de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:40.314 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.302+0000 7f3d6e536700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d6806fba0 con 0x7f3d680c34a0 2022-06-27T12:03:40.314 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.302+0000 7f3d6e536700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d681575e0 con 0x7f3d6806fef0 2022-06-27T12:03:40.314 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.302+0000 7f3d6e536700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d6806f5e0 con 0x7f3d6806e850 2022-06-27T12:03:40.315 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.302+0000 7f3d677fe700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f3d6806fef0 0x7f3d68069de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:40.315 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.302+0000 7f3d677fe700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f3d6806fef0 0x7f3d68069de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47280/0 (socket says 172.21.15.195:47280) 2022-06-27T12:03:40.316 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.302+0000 7f3d677fe700 1 -- 172.21.15.195:0/2694595277 learned_addr learned my addr 172.21.15.195:0/2694595277 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:03:40.317 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.302+0000 7f3d67fff700 1 --2- 172.21.15.195:0/2694595277 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3d6806e850 0x7f3d681570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:40.317 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.302+0000 7f3d677fe700 1 -- 172.21.15.195:0/2694595277 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3d6806e850 msgr2=0x7f3d681570a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:03:40.317 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.302+0000 7f3d677fe700 1 --2- 172.21.15.195:0/2694595277 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3d6806e850 0x7f3d681570a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:40.318 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.302+0000 7f3d677fe700 1 -- 172.21.15.195:0/2694595277 >> v1:172.21.15.202:6789/0 conn(0x7f3d680c34a0 legacy=0x7f3d680c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:40.318 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.302+0000 7f3d677fe700 1 -- 172.21.15.195:0/2694595277 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d6806f8c0 con 0x7f3d6806fef0 2022-06-27T12:03:40.319 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.302+0000 7f3d677fe700 1 --2- 172.21.15.195:0/2694595277 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3d6806fef0 0x7f3d68069de0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f3d58009860 tx=0x7f3d58009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44071dfb4e68f1e server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:03:40.319 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.302+0000 7f3d66ffd700 1 -- 172.21.15.195:0/2694595277 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d5800b690 con 0x7f3d6806fef0 2022-06-27T12:03:40.320 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.303+0000 7f3d66ffd700 1 -- 172.21.15.195:0/2694595277 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d5800b7f0 con 0x7f3d6806fef0 2022-06-27T12:03:40.320 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.303+0000 7f3d66ffd700 1 -- 172.21.15.195:0/2694595277 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d5800bb90 con 0x7f3d6806fef0 2022-06-27T12:03:40.321 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.303+0000 7f3d6e536700 1 -- 172.21.15.195:0/2694595277 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3d6806fef0 msgr2=0x7f3d68069de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:40.322 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.303+0000 7f3d6e536700 1 --2- 172.21.15.195:0/2694595277 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3d6806fef0 0x7f3d68069de0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f3d58009860 tx=0x7f3d58009120 comp rx=0 tx=0).stop 2022-06-27T12:03:40.322 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.303+0000 7f3d6e536700 1 -- 172.21.15.195:0/2694595277 shutdown_connections 2022-06-27T12:03:40.323 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.303+0000 7f3d6e536700 1 --2- 172.21.15.195:0/2694595277 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3d6806e850 0x7f3d681570a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:40.323 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.303+0000 7f3d6e536700 1 --2- 172.21.15.195:0/2694595277 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3d6806fef0 0x7f3d68069de0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:40.324 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.303+0000 7f3d6e536700 1 -- 172.21.15.195:0/2694595277 >> 172.21.15.195:0/2694595277 conn(0x7f3d680c6350 msgr2=0x7f3d680c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:03:40.324 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.303+0000 7f3d6e536700 1 -- 172.21.15.195:0/2694595277 shutdown_connections 2022-06-27T12:03:40.324 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.303+0000 7f3d6e536700 1 -- 172.21.15.195:0/2694595277 wait complete. 2022-06-27T12:03:40.325 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.304+0000 7f3d6e536700 1 Processor -- start 2022-06-27T12:03:40.325 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.304+0000 7f3d6e536700 1 -- start start 2022-06-27T12:03:40.326 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.305+0000 7f3d6e536700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3d6806e850 0x7f3d68149ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:40.326 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.305+0000 7f3d6e536700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3d680c34a0 0x7f3d6816a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:40.327 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.305+0000 7f3d6e536700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3d68152980 0x7f3d6814f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:40.327 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.305+0000 7f3d6e536700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d6816a230 con 0x7f3d680c34a0 2022-06-27T12:03:40.328 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.305+0000 7f3d6e536700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d68151a30 con 0x7f3d68152980 2022-06-27T12:03:40.328 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.305+0000 7f3d6e536700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d680c2c30 con 0x7f3d6806e850 2022-06-27T12:03:40.329 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.305+0000 7f3d677fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3d680c34a0 0x7f3d6816a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:40.329 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.305+0000 7f3d677fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3d680c34a0 0x7f3d6816a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47288/0 (socket says 172.21.15.195:47288) 2022-06-27T12:03:40.330 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.305+0000 7f3d677fe700 1 -- 172.21.15.195:0/1947223465 learned_addr learned my addr 172.21.15.195:0/1947223465 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:03:40.330 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.305+0000 7f3d6cad3700 1 --2- 172.21.15.195:0/1947223465 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3d68152980 0x7f3d6814f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:40.331 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.305+0000 7f3d67fff700 1 --2- 172.21.15.195:0/1947223465 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3d6806e850 0x7f3d68149ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:40.331 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.306+0000 7f3d677fe700 1 -- 172.21.15.195:0/1947223465 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3d6806e850 msgr2=0x7f3d68149ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:40.332 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.306+0000 7f3d677fe700 1 --2- 172.21.15.195:0/1947223465 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3d6806e850 0x7f3d68149ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:40.333 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.306+0000 7f3d677fe700 1 -- 172.21.15.195:0/1947223465 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3d68152980 msgr2=0x7f3d6814f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:40.333 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.306+0000 7f3d677fe700 1 --2- 172.21.15.195:0/1947223465 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3d68152980 0x7f3d6814f340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:40.334 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.306+0000 7f3d677fe700 1 -- 172.21.15.195:0/1947223465 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d580093a0 con 0x7f3d680c34a0 2022-06-27T12:03:40.334 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.306+0000 7f3d677fe700 1 --2- 172.21.15.195:0/1947223465 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3d680c34a0 0x7f3d6816a6c0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f3d58013000 tx=0x7f3d58017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:03:40.335 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.306+0000 7f3d4ffff700 1 -- 172.21.15.195:0/1947223465 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d58003730 con 0x7f3d680c34a0 2022-06-27T12:03:40.335 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.306+0000 7f3d4ffff700 1 -- 172.21.15.195:0/1947223465 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d58003890 con 0x7f3d680c34a0 2022-06-27T12:03:40.335 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.306+0000 7f3d4ffff700 1 -- 172.21.15.195:0/1947223465 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d58003c50 con 0x7f3d680c34a0 2022-06-27T12:03:40.336 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.306+0000 7f3d6e536700 1 -- 172.21.15.195:0/1947223465 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d680c2ed0 con 0x7f3d680c34a0 2022-06-27T12:03:40.337 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.306+0000 7f3d6e536700 1 -- 172.21.15.195:0/1947223465 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d68169ec0 con 0x7f3d680c34a0 2022-06-27T12:03:40.337 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.307+0000 7f3d6e536700 1 -- 172.21.15.195:0/1947223465 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d6816a050 con 0x7f3d680c34a0 2022-06-27T12:03:40.337 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.307+0000 7f3d4ffff700 1 -- 172.21.15.195:0/1947223465 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f3d5800f050 con 0x7f3d680c34a0 2022-06-27T12:03:40.338 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.308+0000 7f3d4ffff700 1 --2- 172.21.15.195:0/1947223465 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3d48063d30 0x7f3d480661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:40.338 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.308+0000 7f3d4ffff700 1 -- 172.21.15.195:0/1947223465 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6890+0+0 (secure 0 0 0) 0x7f3d58081d60 con 0x7f3d680c34a0 2022-06-27T12:03:40.339 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.308+0000 7f3d67fff700 1 --2- 172.21.15.195:0/1947223465 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3d48063d30 0x7f3d480661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:40.339 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.308+0000 7f3d67fff700 1 --2- 172.21.15.195:0/1947223465 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3d48063d30 0x7f3d480661e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f3d5c009bd0 tx=0x7f3d5c009060 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:03:40.340 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.312+0000 7f3d4ffff700 1 -- 172.21.15.195:0/1947223465 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f3d58050610 con 0x7f3d680c34a0 2022-06-27T12:03:40.540 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.539+0000 7f3d6e536700 1 -- 172.21.15.195:0/1947223465 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d680722c0 con 0x7f3d680c34a0 2022-06-27T12:03:40.540 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.540+0000 7f3d4ffff700 1 -- 172.21.15.195:0/1947223465 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+5367 (secure 0 0 0) 0x7f3d58015020 con 0x7f3d680c34a0 2022-06-27T12:03:40.541 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:03:40.541 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":55,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:03:35.042639+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5818},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5818":{"gid":5818,"name":"b","rank":0,"incarnation":55,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6833/3399027527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3399027527},{"type":"v1","addr":"172.21.15.202:6833","nonce":3399027527}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:03:40.543 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.542+0000 7f3d6e536700 1 -- 172.21.15.195:0/1947223465 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3d48063d30 msgr2=0x7f3d480661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:40.544 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.542+0000 7f3d6e536700 1 --2- 172.21.15.195:0/1947223465 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3d48063d30 0x7f3d480661e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f3d5c009bd0 tx=0x7f3d5c009060 comp rx=0 tx=0).stop 2022-06-27T12:03:40.544 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.542+0000 7f3d6e536700 1 -- 172.21.15.195:0/1947223465 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3d680c34a0 msgr2=0x7f3d6816a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:40.544 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.542+0000 7f3d6e536700 1 --2- 172.21.15.195:0/1947223465 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3d680c34a0 0x7f3d6816a6c0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f3d58013000 tx=0x7f3d58017000 comp rx=0 tx=0).stop 2022-06-27T12:03:40.544 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.542+0000 7f3d6e536700 1 -- 172.21.15.195:0/1947223465 shutdown_connections 2022-06-27T12:03:40.545 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.542+0000 7f3d6e536700 1 --2- 172.21.15.195:0/1947223465 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3d48063d30 0x7f3d480661e0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:40.545 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.542+0000 7f3d6e536700 1 --2- 172.21.15.195:0/1947223465 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3d68152980 0x7f3d6814f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:40.545 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.542+0000 7f3d6e536700 1 --2- 172.21.15.195:0/1947223465 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3d6806e850 0x7f3d68149ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:40.545 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.542+0000 7f3d6e536700 1 --2- 172.21.15.195:0/1947223465 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3d680c34a0 0x7f3d6816a6c0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:40.546 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.542+0000 7f3d6e536700 1 -- 172.21.15.195:0/1947223465 >> 172.21.15.195:0/1947223465 conn(0x7f3d680c6350 msgr2=0x7f3d68150740 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:03:40.546 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.542+0000 7f3d6e536700 1 -- 172.21.15.195:0/1947223465 shutdown_connections 2022-06-27T12:03:40.546 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:40.542+0000 7f3d6e536700 1 -- 172.21.15.195:0/1947223465 wait complete. 2022-06-27T12:03:40.546 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 55 2022-06-27T12:03:40.555 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:03:42.556 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:03:42.693 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.689+0000 7f9278c8c700 1 Processor -- start 2022-06-27T12:03:42.693 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.689+0000 7f9278c8c700 1 -- start start 2022-06-27T12:03:42.694 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.690+0000 7f9278c8c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f927406e850 0x7f92741570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:42.694 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.690+0000 7f9278c8c700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f927406fef0 0x7f9274069de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:42.694 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.690+0000 7f9278c8c700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f927406fba0 con 0x7f92740c34a0 2022-06-27T12:03:42.694 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.690+0000 7f9278c8c700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92741575e0 con 0x7f927406e850 2022-06-27T12:03:42.695 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.690+0000 7f9278c8c700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f927406f5e0 con 0x7f927406fef0 2022-06-27T12:03:42.695 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.691+0000 7f927259c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f927406e850 0x7f92741570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:42.695 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.691+0000 7f927259c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f927406e850 0x7f92741570a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47294/0 (socket says 172.21.15.195:47294) 2022-06-27T12:03:42.695 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.691+0000 7f927259c700 1 -- 172.21.15.195:0/2281772571 learned_addr learned my addr 172.21.15.195:0/2281772571 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:03:42.695 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.691+0000 7f9271d9b700 1 --2- 172.21.15.195:0/2281772571 >> v2:172.21.15.202:3300/0 conn(0x7f927406fef0 0x7f9274069de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:42.696 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.691+0000 7f927259c700 1 -- 172.21.15.195:0/2281772571 >> v2:172.21.15.202:3300/0 conn(0x7f927406fef0 msgr2=0x7f9274069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:03:42.696 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.691+0000 7f927259c700 1 --2- 172.21.15.195:0/2281772571 >> v2:172.21.15.202:3300/0 conn(0x7f927406fef0 0x7f9274069de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:42.696 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.691+0000 7f927259c700 1 -- 172.21.15.195:0/2281772571 >> v1:172.21.15.202:6789/0 conn(0x7f92740c34a0 legacy=0x7f92740c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:42.696 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.691+0000 7f927259c700 1 -- 172.21.15.195:0/2281772571 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f927406f8c0 con 0x7f927406e850 2022-06-27T12:03:42.697 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.691+0000 7f927259c700 1 --2- 172.21.15.195:0/2281772571 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f927406e850 0x7f92741570a0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f927406e600 tx=0x7f9268009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75b755ddd7f12d2a server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:03:42.697 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.691+0000 7f927159a700 1 -- 172.21.15.195:0/2281772571 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9268011630 con 0x7f927406e850 2022-06-27T12:03:42.697 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.691+0000 7f927159a700 1 -- 172.21.15.195:0/2281772571 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9268011790 con 0x7f927406e850 2022-06-27T12:03:42.697 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.691+0000 7f927159a700 1 -- 172.21.15.195:0/2281772571 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9268011b30 con 0x7f927406e850 2022-06-27T12:03:42.698 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.692+0000 7f9278c8c700 1 -- 172.21.15.195:0/2281772571 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f927406e850 msgr2=0x7f92741570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:42.698 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.692+0000 7f9278c8c700 1 --2- 172.21.15.195:0/2281772571 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f927406e850 0x7f92741570a0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f927406e600 tx=0x7f9268009120 comp rx=0 tx=0).stop 2022-06-27T12:03:42.698 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.692+0000 7f9278c8c700 1 -- 172.21.15.195:0/2281772571 shutdown_connections 2022-06-27T12:03:42.698 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.692+0000 7f9278c8c700 1 --2- 172.21.15.195:0/2281772571 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f927406e850 0x7f92741570a0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:42.698 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.692+0000 7f9278c8c700 1 --2- 172.21.15.195:0/2281772571 >> v2:172.21.15.202:3300/0 conn(0x7f927406fef0 0x7f9274069de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:42.699 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.692+0000 7f9278c8c700 1 -- 172.21.15.195:0/2281772571 >> 172.21.15.195:0/2281772571 conn(0x7f92740c6350 msgr2=0x7f92740c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:03:42.699 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.692+0000 7f9278c8c700 1 -- 172.21.15.195:0/2281772571 shutdown_connections 2022-06-27T12:03:42.699 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.692+0000 7f9278c8c700 1 -- 172.21.15.195:0/2281772571 wait complete. 2022-06-27T12:03:42.700 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.693+0000 7f9278c8c700 1 Processor -- start 2022-06-27T12:03:42.700 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.693+0000 7f9278c8c700 1 -- start start 2022-06-27T12:03:42.703 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.694+0000 7f9278c8c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f927406e850 0x7f9274164cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:42.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.694+0000 7f9278c8c700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f92740c34a0 0x7f9274165210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:42.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.694+0000 7f9278c8c700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f92741538f0 0x7f9274163090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:42.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.694+0000 7f9278c8c700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9274164560 con 0x7f927406e850 2022-06-27T12:03:42.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.694+0000 7f9278c8c700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9274164840 con 0x7f92740c34a0 2022-06-27T12:03:42.705 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.694+0000 7f9278c8c700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9274153e30 con 0x7f92741538f0 2022-06-27T12:03:42.705 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.694+0000 7f927259c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f927406e850 0x7f9274164cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:42.705 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.694+0000 7f927259c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f927406e850 0x7f9274164cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47300/0 (socket says 172.21.15.195:47300) 2022-06-27T12:03:42.705 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.694+0000 7f927259c700 1 -- 172.21.15.195:0/1560920771 learned_addr learned my addr 172.21.15.195:0/1560920771 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:03:42.706 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.694+0000 7f9271d9b700 1 --2- 172.21.15.195:0/1560920771 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f92740c34a0 0x7f9274165210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:42.706 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.694+0000 7f9272d9d700 1 --2- 172.21.15.195:0/1560920771 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f92741538f0 0x7f9274163090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:42.707 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.694+0000 7f927259c700 1 -- 172.21.15.195:0/1560920771 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f92741538f0 msgr2=0x7f9274163090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:42.707 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.694+0000 7f927259c700 1 --2- 172.21.15.195:0/1560920771 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f92741538f0 0x7f9274163090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:42.707 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.694+0000 7f927259c700 1 -- 172.21.15.195:0/1560920771 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f92740c34a0 msgr2=0x7f9274165210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:42.707 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.694+0000 7f927259c700 1 --2- 172.21.15.195:0/1560920771 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f92740c34a0 0x7f9274165210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:42.708 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.694+0000 7f927259c700 1 -- 172.21.15.195:0/1560920771 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92680093a0 con 0x7f927406e850 2022-06-27T12:03:42.708 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.694+0000 7f9272d9d700 1 --2- 172.21.15.195:0/1560920771 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f92741538f0 0x7f9274163090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:03:42.708 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.694+0000 7f927259c700 1 --2- 172.21.15.195:0/1560920771 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f927406e850 0x7f9274164cd0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f9268011f70 tx=0x7f9268017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:03:42.708 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.694+0000 7f92627fc700 1 -- 172.21.15.195:0/1560920771 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9268021070 con 0x7f927406e850 2022-06-27T12:03:42.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.694+0000 7f92627fc700 1 -- 172.21.15.195:0/1560920771 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9268011b10 con 0x7f927406e850 2022-06-27T12:03:42.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.694+0000 7f9278c8c700 1 -- 172.21.15.195:0/1560920771 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f927414b060 con 0x7f927406e850 2022-06-27T12:03:42.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.694+0000 7f92627fc700 1 -- 172.21.15.195:0/1560920771 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92680042c0 con 0x7f927406e850 2022-06-27T12:03:42.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.694+0000 7f9278c8c700 1 -- 172.21.15.195:0/1560920771 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f927414b600 con 0x7f927406e850 2022-06-27T12:03:42.710 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.695+0000 7f9278c8c700 1 -- 172.21.15.195:0/1560920771 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92740722c0 con 0x7f927406e850 2022-06-27T12:03:42.710 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.696+0000 7f92627fc700 1 -- 172.21.15.195:0/1560920771 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f926802d030 con 0x7f927406e850 2022-06-27T12:03:42.711 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.697+0000 7f92627fc700 1 --2- 172.21.15.195:0/1560920771 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9250063d00 0x7f92500661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:42.711 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.697+0000 7f92627fc700 1 -- 172.21.15.195:0/1560920771 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6890+0+0 (secure 0 0 0) 0x7f92680842b0 con 0x7f927406e850 2022-06-27T12:03:42.711 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.697+0000 7f9271d9b700 1 --2- 172.21.15.195:0/1560920771 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9250063d00 0x7f92500661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:42.711 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.697+0000 7f9271d9b700 1 --2- 172.21.15.195:0/1560920771 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9250063d00 0x7f92500661b0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f9264009bd0 tx=0x7f9264009060 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:03:42.712 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.700+0000 7f92627fc700 1 -- 172.21.15.195:0/1560920771 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f9268052be0 con 0x7f927406e850 2022-06-27T12:03:42.931 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.929+0000 7f9278c8c700 1 -- 172.21.15.195:0/1560920771 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f92740c8a50 con 0x7f927406e850 2022-06-27T12:03:42.931 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.929+0000 7f92627fc700 1 -- 172.21.15.195:0/1560920771 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+5367 (secure 0 0 0) 0x7f926802b090 con 0x7f927406e850 2022-06-27T12:03:42.931 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:03:42.932 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":55,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:03:35.042639+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5818},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5818":{"gid":5818,"name":"b","rank":0,"incarnation":55,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6833/3399027527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3399027527},{"type":"v1","addr":"172.21.15.202:6833","nonce":3399027527}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:03:42.933 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.931+0000 7f9278c8c700 1 -- 172.21.15.195:0/1560920771 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9250063d00 msgr2=0x7f92500661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:42.933 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.931+0000 7f9278c8c700 1 --2- 172.21.15.195:0/1560920771 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9250063d00 0x7f92500661b0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f9264009bd0 tx=0x7f9264009060 comp rx=0 tx=0).stop 2022-06-27T12:03:42.934 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.931+0000 7f9278c8c700 1 -- 172.21.15.195:0/1560920771 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f927406e850 msgr2=0x7f9274164cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:42.934 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.931+0000 7f9278c8c700 1 --2- 172.21.15.195:0/1560920771 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f927406e850 0x7f9274164cd0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f9268011f70 tx=0x7f9268017000 comp rx=0 tx=0).stop 2022-06-27T12:03:42.934 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.931+0000 7f9278c8c700 1 -- 172.21.15.195:0/1560920771 shutdown_connections 2022-06-27T12:03:42.935 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.931+0000 7f9278c8c700 1 --2- 172.21.15.195:0/1560920771 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9250063d00 0x7f92500661b0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:42.935 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.931+0000 7f9278c8c700 1 --2- 172.21.15.195:0/1560920771 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f927406e850 0x7f9274164cd0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:42.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.931+0000 7f9278c8c700 1 --2- 172.21.15.195:0/1560920771 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f92741538f0 0x7f9274163090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:42.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.931+0000 7f9278c8c700 1 --2- 172.21.15.195:0/1560920771 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f92740c34a0 0x7f9274165210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:42.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.931+0000 7f9278c8c700 1 -- 172.21.15.195:0/1560920771 >> 172.21.15.195:0/1560920771 conn(0x7f92740c6350 msgr2=0x7f92740c76c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:03:42.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.931+0000 7f9278c8c700 1 -- 172.21.15.195:0/1560920771 shutdown_connections 2022-06-27T12:03:42.937 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:42.931+0000 7f9278c8c700 1 -- 172.21.15.195:0/1560920771 wait complete. 2022-06-27T12:03:42.937 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 55 2022-06-27T12:03:42.943 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:03:44.944 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:03:45.129 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.109+0000 7fb43a007700 1 Processor -- start 2022-06-27T12:03:45.129 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.109+0000 7fb43a007700 1 -- start start 2022-06-27T12:03:45.129 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.110+0000 7fb43a007700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb434154900 0x7fb4340c2120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:45.130 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.110+0000 7fb43a007700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fb434151930 0x7fb4340bf4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:45.130 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.110+0000 7fb43a007700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb434154da0 con 0x7fb434151560 2022-06-27T12:03:45.130 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.110+0000 7fb43a007700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4340c12a0 con 0x7fb434151930 2022-06-27T12:03:45.130 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.110+0000 7fb43a007700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4340c1580 con 0x7fb434154900 2022-06-27T12:03:45.131 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.110+0000 7fb433fff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fb434151930 0x7fb4340bf4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:45.131 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.110+0000 7fb433fff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fb434151930 0x7fb4340bf4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47310/0 (socket says 172.21.15.195:47310) 2022-06-27T12:03:45.131 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.110+0000 7fb433fff700 1 -- 172.21.15.195:0/3265739440 learned_addr learned my addr 172.21.15.195:0/3265739440 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:03:45.131 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.110+0000 7fb432ffd700 1 --2- 172.21.15.195:0/3265739440 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb434154900 0x7fb4340c2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:45.132 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.111+0000 7fb433fff700 1 -- 172.21.15.195:0/3265739440 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb434154900 msgr2=0x7fb4340c2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:03:45.132 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.111+0000 7fb433fff700 1 --2- 172.21.15.195:0/3265739440 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb434154900 0x7fb4340c2120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:45.132 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.111+0000 7fb433fff700 1 -- 172.21.15.195:0/3265739440 >> v1:172.21.15.202:6789/0 conn(0x7fb434151560 legacy=0x7fb4341541f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:45.132 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.111+0000 7fb433fff700 1 -- 172.21.15.195:0/3265739440 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4340c29f0 con 0x7fb434151930 2022-06-27T12:03:45.132 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.111+0000 7fb433fff700 1 --2- 172.21.15.195:0/3265739440 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb434151930 0x7fb4340bf4a0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fb42c009f20 tx=0x7fb42c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=491a62c5a22cef2c server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:03:45.133 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.111+0000 7fb4327fc700 1 -- 172.21.15.195:0/3265739440 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb42c00e070 con 0x7fb434151930 2022-06-27T12:03:45.143 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.111+0000 7fb4327fc700 1 -- 172.21.15.195:0/3265739440 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb42c00ba60 con 0x7fb434151930 2022-06-27T12:03:45.143 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.111+0000 7fb4327fc700 1 -- 172.21.15.195:0/3265739440 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb42c0092e0 con 0x7fb434151930 2022-06-27T12:03:45.143 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.111+0000 7fb43a007700 1 -- 172.21.15.195:0/3265739440 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb434151930 msgr2=0x7fb4340bf4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:45.144 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.111+0000 7fb43a007700 1 --2- 172.21.15.195:0/3265739440 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb434151930 0x7fb4340bf4a0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fb42c009f20 tx=0x7fb42c009840 comp rx=0 tx=0).stop 2022-06-27T12:03:45.144 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.111+0000 7fb43a007700 1 -- 172.21.15.195:0/3265739440 shutdown_connections 2022-06-27T12:03:45.145 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.111+0000 7fb43a007700 1 --2- 172.21.15.195:0/3265739440 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb434151930 0x7fb4340bf4a0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:45.145 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.111+0000 7fb43a007700 1 --2- 172.21.15.195:0/3265739440 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb434154900 0x7fb4340c2120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:45.145 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.111+0000 7fb43a007700 1 -- 172.21.15.195:0/3265739440 >> 172.21.15.195:0/3265739440 conn(0x7fb4340c6350 msgr2=0x7fb4340c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:03:45.146 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.111+0000 7fb43a007700 1 -- 172.21.15.195:0/3265739440 shutdown_connections 2022-06-27T12:03:45.146 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.111+0000 7fb43a007700 1 -- 172.21.15.195:0/3265739440 wait complete. 2022-06-27T12:03:45.146 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.113+0000 7fb43a007700 1 Processor -- start 2022-06-27T12:03:45.147 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.113+0000 7fb43a007700 1 -- start start 2022-06-27T12:03:45.147 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.113+0000 7fb43a007700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb434151560 0x7fb4341532c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:45.147 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.113+0000 7fb43a007700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb434151930 0x7fb4340bbad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:45.148 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.113+0000 7fb43a007700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb434154900 0x7fb4340bc010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:45.148 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.113+0000 7fb43a007700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4341671a0 con 0x7fb434151930 2022-06-27T12:03:45.149 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.113+0000 7fb43a007700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb434070080 con 0x7fb434151560 2022-06-27T12:03:45.149 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.113+0000 7fb43a007700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4340ba990 con 0x7fb434154900 2022-06-27T12:03:45.150 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.114+0000 7fb432ffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb434151930 0x7fb4340bbad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:45.150 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.114+0000 7fb432ffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb434151930 0x7fb4340bbad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47316/0 (socket says 172.21.15.195:47316) 2022-06-27T12:03:45.151 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.114+0000 7fb432ffd700 1 -- 172.21.15.195:0/2695127936 learned_addr learned my addr 172.21.15.195:0/2695127936 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:03:45.151 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.114+0000 7fb4337fe700 1 --2- 172.21.15.195:0/2695127936 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb434151560 0x7fb4341532c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:45.151 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.114+0000 7fb433fff700 1 --2- 172.21.15.195:0/2695127936 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb434154900 0x7fb4340bc010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:45.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.114+0000 7fb432ffd700 1 -- 172.21.15.195:0/2695127936 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb434154900 msgr2=0x7fb4340bc010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:45.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.114+0000 7fb432ffd700 1 --2- 172.21.15.195:0/2695127936 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb434154900 0x7fb4340bc010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:45.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.114+0000 7fb432ffd700 1 -- 172.21.15.195:0/2695127936 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb434151560 msgr2=0x7fb4341532c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:45.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.114+0000 7fb432ffd700 1 --2- 172.21.15.195:0/2695127936 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb434151560 0x7fb4341532c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:45.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.114+0000 7fb432ffd700 1 -- 172.21.15.195:0/2695127936 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb42c009ae0 con 0x7fb434151930 2022-06-27T12:03:45.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.114+0000 7fb432ffd700 1 --2- 172.21.15.195:0/2695127936 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb434151930 0x7fb4340bbad0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fb4240069c0 tx=0x7fb424009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:03:45.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.114+0000 7fb41b7fe700 1 -- 172.21.15.195:0/2695127936 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb424013680 con 0x7fb434151930 2022-06-27T12:03:45.155 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.114+0000 7fb41b7fe700 1 -- 172.21.15.195:0/2695127936 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4240137e0 con 0x7fb434151930 2022-06-27T12:03:45.155 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.114+0000 7fb43a007700 1 -- 172.21.15.195:0/2695127936 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4340bac70 con 0x7fb434151930 2022-06-27T12:03:45.156 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.114+0000 7fb41b7fe700 1 -- 172.21.15.195:0/2695127936 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb424013b80 con 0x7fb434151930 2022-06-27T12:03:45.156 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.114+0000 7fb43a007700 1 -- 172.21.15.195:0/2695127936 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4340bb230 con 0x7fb434151930 2022-06-27T12:03:45.157 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.115+0000 7fb43a007700 1 -- 172.21.15.195:0/2695127936 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4340bd020 con 0x7fb434151930 2022-06-27T12:03:45.157 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.116+0000 7fb41b7fe700 1 -- 172.21.15.195:0/2695127936 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fb424019030 con 0x7fb434151930 2022-06-27T12:03:45.157 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.116+0000 7fb41b7fe700 1 --2- 172.21.15.195:0/2695127936 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb414063d30 0x7fb4140661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:45.158 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.116+0000 7fb41b7fe700 1 -- 172.21.15.195:0/2695127936 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6890+0+0 (secure 0 0 0) 0x7fb424080b00 con 0x7fb434151930 2022-06-27T12:03:45.158 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.116+0000 7fb4337fe700 1 --2- 172.21.15.195:0/2695127936 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb414063d30 0x7fb4140661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:45.159 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.116+0000 7fb4337fe700 1 --2- 172.21.15.195:0/2695127936 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb414063d30 0x7fb4140661e0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fb4280017e0 tx=0x7fb428009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:03:45.159 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.120+0000 7fb41b7fe700 1 -- 172.21.15.195:0/2695127936 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fb42404f3b0 con 0x7fb434151930 2022-06-27T12:03:45.344 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.343+0000 7fb43a007700 1 -- 172.21.15.195:0/2695127936 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb4340722c0 con 0x7fb434151930 2022-06-27T12:03:45.345 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.344+0000 7fb41b7fe700 1 -- 172.21.15.195:0/2695127936 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+5367 (secure 0 0 0) 0x7fb424017090 con 0x7fb434151930 2022-06-27T12:03:45.345 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:03:45.346 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":55,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:03:35.042639+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5818},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5818":{"gid":5818,"name":"b","rank":0,"incarnation":55,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6833/3399027527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3399027527},{"type":"v1","addr":"172.21.15.202:6833","nonce":3399027527}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:03:45.348 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.346+0000 7fb43a007700 1 -- 172.21.15.195:0/2695127936 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb414063d30 msgr2=0x7fb4140661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:45.349 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.346+0000 7fb43a007700 1 --2- 172.21.15.195:0/2695127936 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb414063d30 0x7fb4140661e0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fb4280017e0 tx=0x7fb428009040 comp rx=0 tx=0).stop 2022-06-27T12:03:45.349 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.346+0000 7fb43a007700 1 -- 172.21.15.195:0/2695127936 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb434151930 msgr2=0x7fb4340bbad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:45.349 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.346+0000 7fb43a007700 1 --2- 172.21.15.195:0/2695127936 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb434151930 0x7fb4340bbad0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fb4240069c0 tx=0x7fb424009060 comp rx=0 tx=0).stop 2022-06-27T12:03:45.355 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.346+0000 7fb43a007700 1 -- 172.21.15.195:0/2695127936 shutdown_connections 2022-06-27T12:03:45.355 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.346+0000 7fb43a007700 1 --2- 172.21.15.195:0/2695127936 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb414063d30 0x7fb4140661e0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:45.355 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.346+0000 7fb43a007700 1 --2- 172.21.15.195:0/2695127936 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb434154900 0x7fb4340bc010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:45.356 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.346+0000 7fb43a007700 1 --2- 172.21.15.195:0/2695127936 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb434151560 0x7fb4341532c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:45.356 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.346+0000 7fb43a007700 1 --2- 172.21.15.195:0/2695127936 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb434151930 0x7fb4340bbad0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:45.356 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.346+0000 7fb43a007700 1 -- 172.21.15.195:0/2695127936 >> 172.21.15.195:0/2695127936 conn(0x7fb4340c6350 msgr2=0x7fb4340c53f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:03:45.356 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.346+0000 7fb43a007700 1 -- 172.21.15.195:0/2695127936 shutdown_connections 2022-06-27T12:03:45.357 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:45.346+0000 7fb43a007700 1 -- 172.21.15.195:0/2695127936 wait complete. 2022-06-27T12:03:45.357 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 55 2022-06-27T12:03:45.361 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:03:47.363 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:03:47.508 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.498+0000 7f9f94c2a700 1 Processor -- start 2022-06-27T12:03:47.509 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.498+0000 7f9f94c2a700 1 -- start start 2022-06-27T12:03:47.536 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.499+0000 7f9f94c2a700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f9f9006f640 0x7f9f90156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:47.537 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.499+0000 7f9f94c2a700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f9f900c4160 0x7f9f900c1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:47.539 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.499+0000 7f9f94c2a700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9f900c39e0 0x7f9f901535a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:47.539 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.499+0000 7f9f94c2a700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f900c1a20 con 0x7f9f900c4160 2022-06-27T12:03:47.540 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.499+0000 7f9f94c2a700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f900c1130 con 0x7f9f9006f640 2022-06-27T12:03:47.540 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.499+0000 7f9f94c2a700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9006ea90 con 0x7f9f900c39e0 2022-06-27T12:03:47.541 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.500+0000 7f9f8dd9b700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f9f900c4160 0x7f9f900c1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:47.541 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.500+0000 7f9f8dd9b700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f9f900c4160 0x7f9f900c1eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47322/0 (socket says 172.21.15.195:47322) 2022-06-27T12:03:47.542 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.500+0000 7f9f8dd9b700 1 -- 172.21.15.195:0/1790475531 learned_addr learned my addr 172.21.15.195:0/1790475531 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:03:47.542 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.500+0000 7f9f8ed9d700 1 --2- 172.21.15.195:0/1790475531 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9f900c39e0 0x7f9f901535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:47.543 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.500+0000 7f9f8e59c700 1 --2- 172.21.15.195:0/1790475531 >> v2:172.21.15.202:3300/0 conn(0x7f9f9006f640 0x7f9f90156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:47.544 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.500+0000 7f9f8dd9b700 1 -- 172.21.15.195:0/1790475531 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9f900c39e0 msgr2=0x7f9f901535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:03:47.544 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.500+0000 7f9f8dd9b700 1 --2- 172.21.15.195:0/1790475531 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9f900c39e0 0x7f9f901535a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:47.544 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.500+0000 7f9f8dd9b700 1 -- 172.21.15.195:0/1790475531 >> v2:172.21.15.202:3300/0 conn(0x7f9f9006f640 msgr2=0x7f9f90156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:03:47.545 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.500+0000 7f9f8dd9b700 1 --2- 172.21.15.195:0/1790475531 >> v2:172.21.15.202:3300/0 conn(0x7f9f9006f640 0x7f9f90156ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:47.545 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.500+0000 7f9f8dd9b700 1 -- 172.21.15.195:0/1790475531 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f9006ed70 con 0x7f9f900c4160 2022-06-27T12:03:47.551 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.500+0000 7f9f8dd9b700 1 --2- 172.21.15.195:0/1790475531 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9f900c4160 0x7f9f900c1eb0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f9f9006db30 tx=0x7f9f80009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0fc595bcaaeab54 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:03:47.551 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.500+0000 7f9f8d59a700 1 -- 172.21.15.195:0/1790475531 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f80011630 con 0x7f9f900c4160 2022-06-27T12:03:47.552 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.500+0000 7f9f8d59a700 1 -- 172.21.15.195:0/1790475531 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f80011790 con 0x7f9f900c4160 2022-06-27T12:03:47.555 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.500+0000 7f9f8d59a700 1 -- 172.21.15.195:0/1790475531 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f80011b30 con 0x7f9f900c4160 2022-06-27T12:03:47.555 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.501+0000 7f9f94c2a700 1 -- 172.21.15.195:0/1790475531 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9f900c4160 msgr2=0x7f9f900c1eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:47.556 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.501+0000 7f9f94c2a700 1 --2- 172.21.15.195:0/1790475531 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9f900c4160 0x7f9f900c1eb0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f9f9006db30 tx=0x7f9f80009120 comp rx=0 tx=0).stop 2022-06-27T12:03:47.557 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.501+0000 7f9f94c2a700 1 -- 172.21.15.195:0/1790475531 shutdown_connections 2022-06-27T12:03:47.557 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.501+0000 7f9f94c2a700 1 --2- 172.21.15.195:0/1790475531 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9f900c39e0 0x7f9f901535a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:47.557 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.501+0000 7f9f94c2a700 1 --2- 172.21.15.195:0/1790475531 >> v2:172.21.15.202:3300/0 conn(0x7f9f9006f640 0x7f9f90156ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:47.558 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.501+0000 7f9f94c2a700 1 --2- 172.21.15.195:0/1790475531 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9f900c4160 0x7f9f900c1eb0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:47.558 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.501+0000 7f9f94c2a700 1 -- 172.21.15.195:0/1790475531 >> 172.21.15.195:0/1790475531 conn(0x7f9f900c6350 msgr2=0x7f9f900c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:03:47.559 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.501+0000 7f9f94c2a700 1 -- 172.21.15.195:0/1790475531 shutdown_connections 2022-06-27T12:03:47.559 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.501+0000 7f9f94c2a700 1 -- 172.21.15.195:0/1790475531 wait complete. 2022-06-27T12:03:47.560 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.502+0000 7f9f94c2a700 1 Processor -- start 2022-06-27T12:03:47.560 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.502+0000 7f9f94c2a700 1 -- start start 2022-06-27T12:03:47.560 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.503+0000 7f9f94c2a700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9f9006f640 0x7f9f90153550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:47.573 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.503+0000 7f9f94c2a700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9f900c39e0 0x7f9f90157a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:47.573 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.503+0000 7f9f94c2a700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9f900c4160 0x7f9f90157f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:47.573 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.503+0000 7f9f94c2a700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9015a540 con 0x7f9f9006f640 2022-06-27T12:03:47.580 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.503+0000 7f9f94c2a700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f90157600 con 0x7f9f900c4160 2022-06-27T12:03:47.580 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.503+0000 7f9f94c2a700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9015a730 con 0x7f9f900c39e0 2022-06-27T12:03:47.581 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.503+0000 7f9f8e59c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9f9006f640 0x7f9f90153550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:47.581 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.503+0000 7f9f8e59c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9f9006f640 0x7f9f90153550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47328/0 (socket says 172.21.15.195:47328) 2022-06-27T12:03:47.581 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.503+0000 7f9f8e59c700 1 -- 172.21.15.195:0/417344016 learned_addr learned my addr 172.21.15.195:0/417344016 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:03:47.586 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.503+0000 7f9f8ed9d700 1 --2- 172.21.15.195:0/417344016 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9f900c4160 0x7f9f90157f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:47.587 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.503+0000 7f9f8dd9b700 1 --2- 172.21.15.195:0/417344016 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9f900c39e0 0x7f9f90157a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:47.587 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.503+0000 7f9f8e59c700 1 -- 172.21.15.195:0/417344016 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9f900c39e0 msgr2=0x7f9f90157a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:47.588 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.503+0000 7f9f8e59c700 1 --2- 172.21.15.195:0/417344016 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9f900c39e0 0x7f9f90157a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:47.588 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.503+0000 7f9f8e59c700 1 -- 172.21.15.195:0/417344016 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9f900c4160 msgr2=0x7f9f90157f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:47.588 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.503+0000 7f9f8e59c700 1 --2- 172.21.15.195:0/417344016 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9f900c4160 0x7f9f90157f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:47.589 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.503+0000 7f9f8e59c700 1 -- 172.21.15.195:0/417344016 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f800093a0 con 0x7f9f9006f640 2022-06-27T12:03:47.589 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.503+0000 7f9f8ed9d700 1 --2- 172.21.15.195:0/417344016 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9f900c4160 0x7f9f90157f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:03:47.590 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.503+0000 7f9f8e59c700 1 --2- 172.21.15.195:0/417344016 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9f9006f640 0x7f9f90153550 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f9f7800e970 tx=0x7f9f78009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:03:47.590 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.503+0000 7f9f867fc700 1 -- 172.21.15.195:0/417344016 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f78013680 con 0x7f9f9006f640 2022-06-27T12:03:47.591 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.503+0000 7f9f867fc700 1 -- 172.21.15.195:0/417344016 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f780137e0 con 0x7f9f9006f640 2022-06-27T12:03:47.591 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.503+0000 7f9f94c2a700 1 -- 172.21.15.195:0/417344016 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f9015aa10 con 0x7f9f9006f640 2022-06-27T12:03:47.592 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.503+0000 7f9f867fc700 1 -- 172.21.15.195:0/417344016 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f78013b60 con 0x7f9f9006f640 2022-06-27T12:03:47.593 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.503+0000 7f9f94c2a700 1 -- 172.21.15.195:0/417344016 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f90152bb0 con 0x7f9f9006f640 2022-06-27T12:03:47.593 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.504+0000 7f9f94c2a700 1 -- 172.21.15.195:0/417344016 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f901578a0 con 0x7f9f9006f640 2022-06-27T12:03:47.595 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.505+0000 7f9f867fc700 1 -- 172.21.15.195:0/417344016 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f9f78019030 con 0x7f9f9006f640 2022-06-27T12:03:47.598 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.506+0000 7f9f867fc700 1 --2- 172.21.15.195:0/417344016 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9f74063d00 0x7f9f740661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:47.599 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.506+0000 7f9f867fc700 1 -- 172.21.15.195:0/417344016 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6890+0+0 (secure 0 0 0) 0x7f9f78081390 con 0x7f9f9006f640 2022-06-27T12:03:47.599 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.506+0000 7f9f8dd9b700 1 --2- 172.21.15.195:0/417344016 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9f74063d00 0x7f9f740661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:47.599 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.506+0000 7f9f8dd9b700 1 --2- 172.21.15.195:0/417344016 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9f74063d00 0x7f9f740661b0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f9f80009160 tx=0x7f9f80017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:03:47.601 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.509+0000 7f9f867fc700 1 -- 172.21.15.195:0/417344016 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f9f7804fb70 con 0x7f9f9006f640 2022-06-27T12:03:47.743 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.737+0000 7f9f94c2a700 1 -- 172.21.15.195:0/417344016 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9f900722c0 con 0x7f9f9006f640 2022-06-27T12:03:47.744 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.737+0000 7f9f867fc700 1 -- 172.21.15.195:0/417344016 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+5367 (secure 0 0 0) 0x7f9f78053190 con 0x7f9f9006f640 2022-06-27T12:03:47.745 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:03:47.745 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":55,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:03:35.042639+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5818},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5818":{"gid":5818,"name":"b","rank":0,"incarnation":55,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6833/3399027527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3399027527},{"type":"v1","addr":"172.21.15.202:6833","nonce":3399027527}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:03:47.748 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.739+0000 7f9f94c2a700 1 -- 172.21.15.195:0/417344016 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9f74063d00 msgr2=0x7f9f740661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:47.749 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.739+0000 7f9f94c2a700 1 --2- 172.21.15.195:0/417344016 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9f74063d00 0x7f9f740661b0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f9f80009160 tx=0x7f9f80017000 comp rx=0 tx=0).stop 2022-06-27T12:03:47.749 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.739+0000 7f9f94c2a700 1 -- 172.21.15.195:0/417344016 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9f9006f640 msgr2=0x7f9f90153550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:47.749 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.739+0000 7f9f94c2a700 1 --2- 172.21.15.195:0/417344016 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9f9006f640 0x7f9f90153550 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f9f7800e970 tx=0x7f9f78009060 comp rx=0 tx=0).stop 2022-06-27T12:03:47.750 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.739+0000 7f9f94c2a700 1 -- 172.21.15.195:0/417344016 shutdown_connections 2022-06-27T12:03:47.750 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.739+0000 7f9f94c2a700 1 --2- 172.21.15.195:0/417344016 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9f74063d00 0x7f9f740661b0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:47.751 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.739+0000 7f9f94c2a700 1 --2- 172.21.15.195:0/417344016 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9f9006f640 0x7f9f90153550 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:47.751 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.739+0000 7f9f94c2a700 1 --2- 172.21.15.195:0/417344016 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9f900c4160 0x7f9f90157f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:47.751 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.739+0000 7f9f94c2a700 1 --2- 172.21.15.195:0/417344016 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9f900c39e0 0x7f9f90157a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:47.751 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.739+0000 7f9f94c2a700 1 -- 172.21.15.195:0/417344016 >> 172.21.15.195:0/417344016 conn(0x7f9f900c6350 msgr2=0x7f9f901505f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:03:47.752 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.739+0000 7f9f94c2a700 1 -- 172.21.15.195:0/417344016 shutdown_connections 2022-06-27T12:03:47.752 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:47.740+0000 7f9f94c2a700 1 -- 172.21.15.195:0/417344016 wait complete. 2022-06-27T12:03:47.753 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 55 2022-06-27T12:03:47.759 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 55, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-06-27T11:47:52.358095+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 55, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5818": { "addr": "172.21.15.202:6833/3399027527", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6832", "nonce": 3399027527, "type": "v2" }, { "addr": "172.21.15.202:6833", "nonce": 3399027527, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5818, "incarnation": 55, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 56, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-06-27T12:03:35.042639+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5818 } } } ], "standbys": [ { "addr": "172.21.15.202:6837/3305006841", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6836", "nonce": 3305006841, "type": "v2" }, { "addr": "172.21.15.202:6837", "nonce": 3305006841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6835/2060709822", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6834", "nonce": 2060709822, "type": "v2" }, { "addr": "172.21.15.195:6835", "nonce": 2060709822, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6838/1598751652", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6836", "nonce": 1598751652, "type": "v2" }, { "addr": "172.21.15.195:6838", "nonce": 1598751652, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4275, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6839/1313800661", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6837", "nonce": 1313800661, "type": "v2" }, { "addr": "172.21.15.195:6839", "nonce": 1313800661, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4278, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-06-27T12:03:49.763 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:03:49.905 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:49.905+0000 7fe6687aa700 1 Processor -- start 2022-06-27T12:03:49.905 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:49.905+0000 7fe6687aa700 1 -- start start 2022-06-27T12:03:49.908 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:49.906+0000 7fe6687aa700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fe660153cb0 0x7fe660156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:49.908 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:49.906+0000 7fe6687aa700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6600c1a20 con 0x7fe66006f640 2022-06-27T12:03:49.908 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:49.906+0000 7fe6687aa700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6600c1130 con 0x7fe6600c4160 2022-06-27T12:03:49.909 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:49.906+0000 7fe6687aa700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe66006ea90 con 0x7fe660153cb0 2022-06-27T12:03:49.909 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:49.906+0000 7fe666546700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fe66006f640 0x7fe6600c1eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:53530/0 (socket says 172.21.15.195:53530) 2022-06-27T12:03:49.909 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:49.906+0000 7fe666d47700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fe660153cb0 0x7fe660156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:49.909 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:49.906+0000 7fe666546700 1 -- 172.21.15.195:0/1067018962 learned_addr learned my addr 172.21.15.195:0/1067018962 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:03:49.910 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:49.906+0000 7fe666d47700 1 -- 172.21.15.195:0/1067018962 >> v1:172.21.15.202:6789/0 conn(0x7fe6600c4160 legacy=0x7fe6601535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:03:49.910 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:49.906+0000 7fe666d47700 1 -- 172.21.15.195:0/1067018962 >> v1:172.21.15.195:6789/0 conn(0x7fe66006f640 legacy=0x7fe6600c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:49.910 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:49.906+0000 7fe666d47700 1 -- 172.21.15.195:0/1067018962 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe66006ed70 con 0x7fe660153cb0 2022-06-27T12:03:49.910 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:49.906+0000 7fe666d47700 1 --2- 172.21.15.195:0/1067018962 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe660153cb0 0x7fe660156ed0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fe658009e60 tx=0x7fe658009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42078be4f2c5a7e0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:03:49.911 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:49.906+0000 7fe665544700 1 -- 172.21.15.195:0/1067018962 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe65800e070 con 0x7fe660153cb0 2022-06-27T12:03:49.911 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:49.907+0000 7fe665544700 1 -- 172.21.15.195:0/1067018962 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe65800bc80 con 0x7fe660153cb0 2022-06-27T12:03:49.911 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:49.907+0000 7fe665544700 1 -- 172.21.15.195:0/1067018962 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6580038a0 con 0x7fe660153cb0 2022-06-27T12:03:49.911 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:49.907+0000 7fe6687aa700 1 -- 172.21.15.195:0/1067018962 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe660153cb0 msgr2=0x7fe660156ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:49.912 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:49.907+0000 7fe6687aa700 1 --2- 172.21.15.195:0/1067018962 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe660153cb0 0x7fe660156ed0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fe658009e60 tx=0x7fe658009780 comp rx=0 tx=0).stop 2022-06-27T12:03:49.912 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:49.907+0000 7fe6687aa700 1 -- 172.21.15.195:0/1067018962 shutdown_connections 2022-06-27T12:03:49.912 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:49.907+0000 7fe6687aa700 1 --2- 172.21.15.195:0/1067018962 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe660153cb0 0x7fe660156ed0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:49.912 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:49.907+0000 7fe6687aa700 1 -- 172.21.15.195:0/1067018962 >> 172.21.15.195:0/1067018962 conn(0x7fe6600c6350 msgr2=0x7fe6600c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:03:49.912 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:49.907+0000 7fe6687aa700 1 -- 172.21.15.195:0/1067018962 shutdown_connections 2022-06-27T12:03:49.913 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:49.907+0000 7fe6687aa700 1 -- 172.21.15.195:0/1067018962 wait complete. 2022-06-27T12:03:49.913 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:49.908+0000 7fe6687aa700 1 Processor -- start 2022-06-27T12:03:49.914 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:49.908+0000 7fe6687aa700 1 -- start start 2022-06-27T12:03:49.917 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:49.909+0000 7fe6687aa700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe66006f640 0x7fe6600bd540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:49.917 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:49.909+0000 7fe6687aa700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe6600c4160 0x7fe6600bda80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:49.918 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:49.909+0000 7fe6687aa700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe660153cb0 0x7fe66006a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:49.918 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:49.909+0000 7fe6687aa700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe66006a090 con 0x7fe66006f640 2022-06-27T12:03:49.918 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:49.909+0000 7fe6687aa700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6600bc500 con 0x7fe660153cb0 2022-06-27T12:03:49.918 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:49.909+0000 7fe6687aa700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6600bc730 con 0x7fe6600c4160 2022-06-27T12:03:49.919 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:49.909+0000 7fe666546700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe66006f640 0x7fe6600bd540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:49.919 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:49.909+0000 7fe666546700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe66006f640 0x7fe6600bd540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47348/0 (socket says 172.21.15.195:47348) 2022-06-27T12:03:49.919 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:49.909+0000 7fe666546700 1 -- 172.21.15.195:0/2226055747 learned_addr learned my addr 172.21.15.195:0/2226055747 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:03:49.919 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:49.909+0000 7fe666d47700 1 --2- 172.21.15.195:0/2226055747 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe660153cb0 0x7fe66006a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:49.920 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:49.909+0000 7fe666546700 1 -- 172.21.15.195:0/2226055747 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe6600c4160 msgr2=0x7fe6600bda80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:49.920 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:49.909+0000 7fe665d45700 1 --2- 172.21.15.195:0/2226055747 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe6600c4160 0x7fe6600bda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:49.921 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:49.909+0000 7fe666546700 1 --2- 172.21.15.195:0/2226055747 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe6600c4160 0x7fe6600bda80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:49.921 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:49.909+0000 7fe666546700 1 -- 172.21.15.195:0/2226055747 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe660153cb0 msgr2=0x7fe66006a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:49.924 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:49.909+0000 7fe666546700 1 --2- 172.21.15.195:0/2226055747 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe660153cb0 0x7fe66006a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:49.924 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:49.909+0000 7fe666546700 1 -- 172.21.15.195:0/2226055747 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe658009a20 con 0x7fe66006f640 2022-06-27T12:03:49.925 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:49.909+0000 7fe666d47700 1 --2- 172.21.15.195:0/2226055747 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe660153cb0 0x7fe66006a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:03:49.925 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:49.909+0000 7fe665d45700 1 --2- 172.21.15.195:0/2226055747 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe6600c4160 0x7fe6600bda80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T12:03:49.927 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:49.910+0000 7fe666546700 1 --2- 172.21.15.195:0/2226055747 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe66006f640 0x7fe6600bd540 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fe65c007fd0 tx=0x7fe65c007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:03:49.927 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:49.910+0000 7fe64e7fc700 1 -- 172.21.15.195:0/2226055747 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe65c0072e0 con 0x7fe66006f640 2022-06-27T12:03:49.927 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:49.910+0000 7fe64e7fc700 1 -- 172.21.15.195:0/2226055747 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe65c007440 con 0x7fe66006f640 2022-06-27T12:03:49.928 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:49.910+0000 7fe6687aa700 1 -- 172.21.15.195:0/2226055747 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6601667b0 con 0x7fe66006f640 2022-06-27T12:03:49.928 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:49.910+0000 7fe64e7fc700 1 -- 172.21.15.195:0/2226055747 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe65c0057c0 con 0x7fe66006f640 2022-06-27T12:03:49.929 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:49.910+0000 7fe6687aa700 1 -- 172.21.15.195:0/2226055747 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe660166d70 con 0x7fe66006f640 2022-06-27T12:03:49.929 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:49.911+0000 7fe6687aa700 1 -- 172.21.15.195:0/2226055747 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6600c17b0 con 0x7fe66006f640 2022-06-27T12:03:49.930 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:49.911+0000 7fe64e7fc700 1 -- 172.21.15.195:0/2226055747 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fe65c019030 con 0x7fe66006f640 2022-06-27T12:03:49.930 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:49.911+0000 7fe64e7fc700 1 --2- 172.21.15.195:0/2226055747 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe640063d00 0x7fe6400661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:49.930 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:49.912+0000 7fe64e7fc700 1 -- 172.21.15.195:0/2226055747 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6890+0+0 (secure 0 0 0) 0x7fe65c07fd40 con 0x7fe66006f640 2022-06-27T12:03:49.931 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:49.912+0000 7fe665d45700 1 --2- 172.21.15.195:0/2226055747 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe640063d00 0x7fe6400661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:49.931 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:49.912+0000 7fe665d45700 1 --2- 172.21.15.195:0/2226055747 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe640063d00 0x7fe6400661b0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fe6600727b0 tx=0x7fe650009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:03:49.932 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:49.916+0000 7fe64e7fc700 1 -- 172.21.15.195:0/2226055747 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fe65c049c90 con 0x7fe66006f640 2022-06-27T12:03:50.139 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:50.137+0000 7fe6687aa700 1 -- 172.21.15.195:0/2226055747 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe6601673c0 con 0x7fe66006f640 2022-06-27T12:03:50.140 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:50.137+0000 7fe64e7fc700 1 -- 172.21.15.195:0/2226055747 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+5371 (secure 0 0 0) 0x7fe65c017070 con 0x7fe66006f640 2022-06-27T12:03:50.150 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:03:50.151 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:03:49.233804+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5818},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5818":{"gid":5818,"name":"b","rank":0,"incarnation":55,"state":"up:reconnect","state_seq":27,"addr":"172.21.15.202:6833/3399027527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3399027527},{"type":"v1","addr":"172.21.15.202:6833","nonce":3399027527}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:03:50.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:50.139+0000 7fe6687aa700 1 -- 172.21.15.195:0/2226055747 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe640063d00 msgr2=0x7fe6400661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:50.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:50.139+0000 7fe6687aa700 1 --2- 172.21.15.195:0/2226055747 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe640063d00 0x7fe6400661b0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fe6600727b0 tx=0x7fe650009040 comp rx=0 tx=0).stop 2022-06-27T12:03:50.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:50.139+0000 7fe6687aa700 1 -- 172.21.15.195:0/2226055747 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe66006f640 msgr2=0x7fe6600bd540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:50.155 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:50.139+0000 7fe6687aa700 1 --2- 172.21.15.195:0/2226055747 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe66006f640 0x7fe6600bd540 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fe65c007fd0 tx=0x7fe65c007df0 comp rx=0 tx=0).stop 2022-06-27T12:03:50.155 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:50.139+0000 7fe6687aa700 1 -- 172.21.15.195:0/2226055747 shutdown_connections 2022-06-27T12:03:50.155 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:50.139+0000 7fe6687aa700 1 --2- 172.21.15.195:0/2226055747 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe640063d00 0x7fe6400661b0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:50.155 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:50.139+0000 7fe6687aa700 1 --2- 172.21.15.195:0/2226055747 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe66006f640 0x7fe6600bd540 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:50.156 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:50.139+0000 7fe6687aa700 1 --2- 172.21.15.195:0/2226055747 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe660153cb0 0x7fe66006a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:50.156 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:50.139+0000 7fe6687aa700 1 --2- 172.21.15.195:0/2226055747 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe6600c4160 0x7fe6600bda80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:50.156 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:50.139+0000 7fe6687aa700 1 -- 172.21.15.195:0/2226055747 >> 172.21.15.195:0/2226055747 conn(0x7fe6600c6350 msgr2=0x7fe660151050 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:03:50.156 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:50.139+0000 7fe6687aa700 1 -- 172.21.15.195:0/2226055747 shutdown_connections 2022-06-27T12:03:50.157 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:50.139+0000 7fe6687aa700 1 -- 172.21.15.195:0/2226055747 wait complete. 2022-06-27T12:03:50.157 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 56 2022-06-27T12:03:50.158 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:03:52.169 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:03:52.307 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.304+0000 7f6ad1668700 1 Processor -- start 2022-06-27T12:03:52.325 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.304+0000 7f6ad1668700 1 -- start start 2022-06-27T12:03:52.325 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.305+0000 7f6ad1668700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f6acc0c40d0 0x7f6acc157010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:52.326 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.305+0000 7f6ad1668700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6acc067ed0 0x7f6acc14cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:52.326 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.305+0000 7f6ad1668700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6acc068330 con 0x7f6acc0c6190 2022-06-27T12:03:52.326 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.305+0000 7f6ad1668700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6acc156b80 con 0x7f6acc0c40d0 2022-06-27T12:03:52.326 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.305+0000 7f6ad1668700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6acc06a040 con 0x7f6acc067ed0 2022-06-27T12:03:52.326 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.305+0000 7f6aca7fc700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f6acc0c6190 0x7f6acc0c44a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:53544/0 (socket says 172.21.15.195:53544) 2022-06-27T12:03:52.327 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.305+0000 7f6aca7fc700 1 -- 172.21.15.195:0/1751235123 learned_addr learned my addr 172.21.15.195:0/1751235123 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:03:52.327 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.305+0000 7f6acaffd700 1 --2- 172.21.15.195:0/1751235123 >> v2:172.21.15.202:3300/0 conn(0x7f6acc0c40d0 0x7f6acc157010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:52.328 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.305+0000 7f6acb7fe700 1 --2- 172.21.15.195:0/1751235123 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6acc067ed0 0x7f6acc14cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:52.344 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.305+0000 7f6ac9ffb700 1 -- 172.21.15.195:0/1751235123 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4135703603 0 0) 0x7f6acc068330 con 0x7f6acc0c6190 2022-06-27T12:03:52.344 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.305+0000 7f6ac9ffb700 1 -- 172.21.15.195:0/1751235123 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6aac003430 con 0x7f6acc0c6190 2022-06-27T12:03:52.345 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.306+0000 7f6ac9ffb700 1 -- 172.21.15.195:0/1751235123 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4259313814 0 0) 0x7f6aac003430 con 0x7f6acc0c6190 2022-06-27T12:03:52.345 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.306+0000 7f6ac9ffb700 1 -- 172.21.15.195:0/1751235123 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6acc067ed0 msgr2=0x7f6acc14cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:03:52.346 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.306+0000 7f6ac9ffb700 1 --2- 172.21.15.195:0/1751235123 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6acc067ed0 0x7f6acc14cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:52.346 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.306+0000 7f6ac9ffb700 1 -- 172.21.15.195:0/1751235123 >> v2:172.21.15.202:3300/0 conn(0x7f6acc0c40d0 msgr2=0x7f6acc157010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:03:52.347 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.306+0000 7f6ac9ffb700 1 --2- 172.21.15.195:0/1751235123 >> v2:172.21.15.202:3300/0 conn(0x7f6acc0c40d0 0x7f6acc157010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:52.347 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.306+0000 7f6ac9ffb700 1 -- 172.21.15.195:0/1751235123 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6acc06a300 con 0x7f6acc0c6190 2022-06-27T12:03:52.348 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.306+0000 7f6ac9ffb700 1 -- 172.21.15.195:0/1751235123 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7f6ab4001470 con 0x7f6acc0c6190 2022-06-27T12:03:52.348 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.306+0000 7f6acaffd700 1 --2- 172.21.15.195:0/1751235123 >> v2:172.21.15.202:3300/0 conn(0x7f6acc0c40d0 0x7f6acc157010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:03:52.348 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.306+0000 7f6ac9ffb700 1 -- 172.21.15.195:0/1751235123 >> v1:172.21.15.195:6789/0 conn(0x7f6acc0c6190 legacy=0x7f6acc0c44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:52.349 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.306+0000 7f6ac9ffb700 1 --2- 172.21.15.195:0/1751235123 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6aac003920 0x7f6aac003d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:52.349 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.306+0000 7f6ac9ffb700 1 --2- 172.21.15.195:0/1751235123 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6aac0046d0 0x7f6aac004b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:52.350 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.306+0000 7f6ac9ffb700 1 --2- 172.21.15.195:0/1751235123 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6acc067ed0 0x7f6aac005080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:52.350 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.306+0000 7f6ac9ffb700 1 -- 172.21.15.195:0/1751235123 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6acc156b80 con 0x7f6aac003920 2022-06-27T12:03:52.350 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.306+0000 7f6ac9ffb700 1 -- 172.21.15.195:0/1751235123 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6acc06a040 con 0x7f6aac0046d0 2022-06-27T12:03:52.351 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.306+0000 7f6ac9ffb700 1 -- 172.21.15.195:0/1751235123 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6aac00b920 con 0x7f6acc067ed0 2022-06-27T12:03:52.351 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.306+0000 7f6acaffd700 1 --2- 172.21.15.195:0/1751235123 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6aac003920 0x7f6aac003d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:52.352 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.306+0000 7f6aca7fc700 1 --2- 172.21.15.195:0/1751235123 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6aac0046d0 0x7f6aac004b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:52.352 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.306+0000 7f6acb7fe700 1 --2- 172.21.15.195:0/1751235123 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6acc067ed0 0x7f6aac005080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:52.353 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.306+0000 7f6acaffd700 1 -- 172.21.15.195:0/1751235123 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6acc067ed0 msgr2=0x7f6aac005080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:03:52.353 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.306+0000 7f6acaffd700 1 --2- 172.21.15.195:0/1751235123 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6acc067ed0 0x7f6aac005080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:52.353 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.306+0000 7f6acaffd700 1 -- 172.21.15.195:0/1751235123 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6aac0046d0 msgr2=0x7f6aac004b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:03:52.354 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.306+0000 7f6acaffd700 1 --2- 172.21.15.195:0/1751235123 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6aac0046d0 0x7f6aac004b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:52.354 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.306+0000 7f6acaffd700 1 -- 172.21.15.195:0/1751235123 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aac003430 con 0x7f6aac003920 2022-06-27T12:03:52.355 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.306+0000 7f6acaffd700 1 --2- 172.21.15.195:0/1751235123 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6aac003920 0x7f6aac003d30 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f6abc009ce0 tx=0x7f6abc009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b01befc733c8e65b server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:03:52.355 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.306+0000 7f6ac9ffb700 1 -- 172.21.15.195:0/1751235123 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6abc00e770 con 0x7f6aac003920 2022-06-27T12:03:52.356 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.306+0000 7f6ac9ffb700 1 -- 172.21.15.195:0/1751235123 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6abc00e8d0 con 0x7f6aac003920 2022-06-27T12:03:52.356 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.306+0000 7f6ac9ffb700 1 -- 172.21.15.195:0/1751235123 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6abc00ec70 con 0x7f6aac003920 2022-06-27T12:03:52.356 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.307+0000 7f6ad1668700 1 -- 172.21.15.195:0/1751235123 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6aac003920 msgr2=0x7f6aac003d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:52.357 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.307+0000 7f6ad1668700 1 --2- 172.21.15.195:0/1751235123 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6aac003920 0x7f6aac003d30 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f6abc009ce0 tx=0x7f6abc009060 comp rx=0 tx=0).stop 2022-06-27T12:03:52.357 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.307+0000 7f6acb7fe700 1 -- 172.21.15.195:0/1751235123 reap_dead start 2022-06-27T12:03:52.358 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.307+0000 7f6ad1668700 1 -- 172.21.15.195:0/1751235123 shutdown_connections 2022-06-27T12:03:52.358 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.307+0000 7f6ad1668700 1 -- 172.21.15.195:0/1751235123 >> 172.21.15.195:0/1751235123 conn(0x7f6acc0ccb80 msgr2=0x7f6acc0ccf80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:03:52.359 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.307+0000 7f6ad1668700 1 -- 172.21.15.195:0/1751235123 shutdown_connections 2022-06-27T12:03:52.359 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.307+0000 7f6ad1668700 1 -- 172.21.15.195:0/1751235123 wait complete. 2022-06-27T12:03:52.360 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.309+0000 7f6ad1668700 1 Processor -- start 2022-06-27T12:03:52.360 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.309+0000 7f6ad1668700 1 -- start start 2022-06-27T12:03:52.360 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.310+0000 7f6ad1668700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6aac003920 0x7f6acc16a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:52.361 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.310+0000 7f6ad1668700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6acc14cd70 0x7f6acc149b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:52.361 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.310+0000 7f6ad1668700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6acc152260 0x7f6acc0c7730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:52.362 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.310+0000 7f6ad1668700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6acc0b24f0 con 0x7f6acc14cd70 2022-06-27T12:03:52.362 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.310+0000 7f6ad1668700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6acc166e80 con 0x7f6acc152260 2022-06-27T12:03:52.363 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.310+0000 7f6ad1668700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6acc16aa50 con 0x7f6aac003920 2022-06-27T12:03:52.363 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.310+0000 7f6aca7fc700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6acc14cd70 0x7f6acc149b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:52.364 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.310+0000 7f6aca7fc700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6acc14cd70 0x7f6acc149b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47370/0 (socket says 172.21.15.195:47370) 2022-06-27T12:03:52.364 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.310+0000 7f6aca7fc700 1 -- 172.21.15.195:0/186525450 learned_addr learned my addr 172.21.15.195:0/186525450 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:03:52.365 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.310+0000 7f6acaffd700 1 --2- 172.21.15.195:0/186525450 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6aac003920 0x7f6acc16a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:52.365 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.310+0000 7f6acb7fe700 1 --2- 172.21.15.195:0/186525450 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6acc152260 0x7f6acc0c7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:52.366 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.310+0000 7f6aca7fc700 1 -- 172.21.15.195:0/186525450 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6aac003920 msgr2=0x7f6acc16a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:52.367 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.310+0000 7f6aca7fc700 1 --2- 172.21.15.195:0/186525450 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6aac003920 0x7f6acc16a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:52.368 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.310+0000 7f6aca7fc700 1 -- 172.21.15.195:0/186525450 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6acc152260 msgr2=0x7f6acc0c7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:52.368 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.310+0000 7f6aca7fc700 1 --2- 172.21.15.195:0/186525450 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6acc152260 0x7f6acc0c7730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:52.369 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.310+0000 7f6aca7fc700 1 -- 172.21.15.195:0/186525450 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6acc14a040 con 0x7f6acc14cd70 2022-06-27T12:03:52.369 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.310+0000 7f6acaffd700 1 --2- 172.21.15.195:0/186525450 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6aac003920 0x7f6acc16a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:03:52.369 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.310+0000 7f6aca7fc700 1 --2- 172.21.15.195:0/186525450 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6acc14cd70 0x7f6acc149b00 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f6ab4001120 tx=0x7f6ab4001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:03:52.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.310+0000 7f6ac2ffd700 1 -- 172.21.15.195:0/186525450 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab400f880 con 0x7f6acc14cd70 2022-06-27T12:03:52.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.310+0000 7f6ac2ffd700 1 -- 172.21.15.195:0/186525450 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ab400f9e0 con 0x7f6acc14cd70 2022-06-27T12:03:52.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.310+0000 7f6ad1668700 1 -- 172.21.15.195:0/186525450 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6acc150e00 con 0x7f6acc14cd70 2022-06-27T12:03:52.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.310+0000 7f6ac2ffd700 1 -- 172.21.15.195:0/186525450 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab400fda0 con 0x7f6acc14cd70 2022-06-27T12:03:52.372 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.310+0000 7f6ad1668700 1 -- 172.21.15.195:0/186525450 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6acc1510c0 con 0x7f6acc14cd70 2022-06-27T12:03:52.372 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.311+0000 7f6ad1668700 1 -- 172.21.15.195:0/186525450 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6acc166ff0 con 0x7f6acc14cd70 2022-06-27T12:03:52.373 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.312+0000 7f6ac2ffd700 1 -- 172.21.15.195:0/186525450 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f6ab401c030 con 0x7f6acc14cd70 2022-06-27T12:03:52.373 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.312+0000 7f6ac2ffd700 1 --2- 172.21.15.195:0/186525450 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6ab0063d30 0x7f6ab00661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:52.373 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.312+0000 7f6ac2ffd700 1 -- 172.21.15.195:0/186525450 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6890+0+0 (secure 0 0 0) 0x7f6ab4080960 con 0x7f6acc14cd70 2022-06-27T12:03:52.374 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.312+0000 7f6acaffd700 1 --2- 172.21.15.195:0/186525450 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6ab0063d30 0x7f6ab00661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:52.375 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.313+0000 7f6acaffd700 1 --2- 172.21.15.195:0/186525450 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6ab0063d30 0x7f6ab00661e0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f6abc009a70 tx=0x7f6abc01a000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:03:52.375 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.317+0000 7f6ac2ffd700 1 -- 172.21.15.195:0/186525450 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f6ab404f120 con 0x7f6acc14cd70 2022-06-27T12:03:52.544 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.544+0000 7f6ad1668700 1 -- 172.21.15.195:0/186525450 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6acc069c90 con 0x7f6acc14cd70 2022-06-27T12:03:52.545 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.544+0000 7f6ac2ffd700 1 -- 172.21.15.195:0/186525450 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+5371 (secure 0 0 0) 0x7f6ab401a090 con 0x7f6acc14cd70 2022-06-27T12:03:52.545 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:03:52.546 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:03:49.233804+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5818},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5818":{"gid":5818,"name":"b","rank":0,"incarnation":55,"state":"up:reconnect","state_seq":27,"addr":"172.21.15.202:6833/3399027527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3399027527},{"type":"v1","addr":"172.21.15.202:6833","nonce":3399027527}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:03:52.547 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.546+0000 7f6ad1668700 1 -- 172.21.15.195:0/186525450 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6ab0063d30 msgr2=0x7f6ab00661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:52.547 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.546+0000 7f6ad1668700 1 --2- 172.21.15.195:0/186525450 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6ab0063d30 0x7f6ab00661e0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f6abc009a70 tx=0x7f6abc01a000 comp rx=0 tx=0).stop 2022-06-27T12:03:52.548 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.546+0000 7f6ad1668700 1 -- 172.21.15.195:0/186525450 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6acc14cd70 msgr2=0x7f6acc149b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:52.549 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.546+0000 7f6ad1668700 1 --2- 172.21.15.195:0/186525450 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6acc14cd70 0x7f6acc149b00 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f6ab4001120 tx=0x7f6ab4001e70 comp rx=0 tx=0).stop 2022-06-27T12:03:52.549 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.546+0000 7f6ad1668700 1 -- 172.21.15.195:0/186525450 shutdown_connections 2022-06-27T12:03:52.549 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.546+0000 7f6ad1668700 1 --2- 172.21.15.195:0/186525450 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6ab0063d30 0x7f6ab00661e0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:52.550 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.546+0000 7f6ad1668700 1 --2- 172.21.15.195:0/186525450 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6acc152260 0x7f6acc0c7730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:52.550 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.546+0000 7f6ad1668700 1 --2- 172.21.15.195:0/186525450 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6aac003920 0x7f6acc16a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:52.550 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.546+0000 7f6ad1668700 1 --2- 172.21.15.195:0/186525450 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6acc14cd70 0x7f6acc149b00 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:52.550 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.546+0000 7f6ad1668700 1 -- 172.21.15.195:0/186525450 >> 172.21.15.195:0/186525450 conn(0x7f6acc0ccb80 msgr2=0x7f6acc150200 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:03:52.551 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.546+0000 7f6ad1668700 1 -- 172.21.15.195:0/186525450 shutdown_connections 2022-06-27T12:03:52.551 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:52.546+0000 7f6ad1668700 1 -- 172.21.15.195:0/186525450 wait complete. 2022-06-27T12:03:52.551 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 56 2022-06-27T12:03:52.560 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:03:54.562 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:03:54.714 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.714+0000 7f6817d31700 1 Processor -- start 2022-06-27T12:03:54.715 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.715+0000 7f6817d31700 1 -- start start 2022-06-27T12:03:54.729 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.715+0000 7f6817d31700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f681006e850 0x7f68101570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:54.730 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.715+0000 7f6817d31700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f681006fef0 0x7f6810069de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:54.730 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.715+0000 7f6817d31700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f681006fba0 con 0x7f68100c34a0 2022-06-27T12:03:54.731 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.715+0000 7f6817d31700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68101575e0 con 0x7f681006e850 2022-06-27T12:03:54.731 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.715+0000 7f6817d31700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f681006f5e0 con 0x7f681006fef0 2022-06-27T12:03:54.731 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.716+0000 7f68162ce700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f68100c34a0 0x7f68100c2920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:53566/0 (socket says 172.21.15.195:53566) 2022-06-27T12:03:54.731 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.716+0000 7f68162ce700 1 -- 172.21.15.195:0/1007066681 learned_addr learned my addr 172.21.15.195:0/1007066681 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:03:54.732 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.716+0000 7f6815acd700 1 --2- 172.21.15.195:0/1007066681 >> v2:172.21.15.202:3300/0 conn(0x7f681006e850 0x7f68101570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:54.732 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.716+0000 7f68152cc700 1 --2- 172.21.15.195:0/1007066681 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f681006fef0 0x7f6810069de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:54.733 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.716+0000 7f6814acb700 1 -- 172.21.15.195:0/1007066681 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2574690438 0 0) 0x7f681006fba0 con 0x7f68100c34a0 2022-06-27T12:03:54.733 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.716+0000 7f6814acb700 1 -- 172.21.15.195:0/1007066681 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67f8003430 con 0x7f68100c34a0 2022-06-27T12:03:54.733 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.716+0000 7f6814acb700 1 -- 172.21.15.195:0/1007066681 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 181575771 0 0) 0x7f67f8003430 con 0x7f68100c34a0 2022-06-27T12:03:54.733 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.716+0000 7f6814acb700 1 -- 172.21.15.195:0/1007066681 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f681006fef0 msgr2=0x7f6810069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:03:54.749 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.716+0000 7f6814acb700 1 --2- 172.21.15.195:0/1007066681 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f681006fef0 0x7f6810069de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:54.749 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.716+0000 7f6814acb700 1 -- 172.21.15.195:0/1007066681 >> v2:172.21.15.202:3300/0 conn(0x7f681006e850 msgr2=0x7f68101570a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:03:54.749 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.716+0000 7f6814acb700 1 --2- 172.21.15.195:0/1007066681 >> v2:172.21.15.202:3300/0 conn(0x7f681006e850 0x7f68101570a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:54.750 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.716+0000 7f6814acb700 1 -- 172.21.15.195:0/1007066681 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f681006f8c0 con 0x7f68100c34a0 2022-06-27T12:03:54.750 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.716+0000 7f6814acb700 1 -- 172.21.15.195:0/1007066681 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7f68080017f0 con 0x7f68100c34a0 2022-06-27T12:03:54.750 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.716+0000 7f6814acb700 1 -- 172.21.15.195:0/1007066681 >> v1:172.21.15.195:6789/0 conn(0x7f68100c34a0 legacy=0x7f68100c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:54.750 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.716+0000 7f6814acb700 1 --2- 172.21.15.195:0/1007066681 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f67f8003920 0x7f67f8003d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:54.751 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.716+0000 7f6814acb700 1 --2- 172.21.15.195:0/1007066681 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f67f80046d0 0x7f67f8004b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:54.751 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.716+0000 7f6814acb700 1 --2- 172.21.15.195:0/1007066681 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f67f8005040 0x7f67f800b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:54.751 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.716+0000 7f6814acb700 1 -- 172.21.15.195:0/1007066681 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68101575e0 con 0x7f67f8005040 2022-06-27T12:03:54.751 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.716+0000 7f6814acb700 1 -- 172.21.15.195:0/1007066681 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681006f5e0 con 0x7f67f8003920 2022-06-27T12:03:54.752 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.716+0000 7f6814acb700 1 -- 172.21.15.195:0/1007066681 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67f800be60 con 0x7f67f80046d0 2022-06-27T12:03:54.752 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.716+0000 7f68162ce700 1 --2- 172.21.15.195:0/1007066681 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f67f8005040 0x7f67f800b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:54.752 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.716+0000 7f6815acd700 1 --2- 172.21.15.195:0/1007066681 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f67f8003920 0x7f67f8003d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:54.752 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.716+0000 7f68152cc700 1 --2- 172.21.15.195:0/1007066681 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f67f80046d0 0x7f67f8004b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:54.753 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.717+0000 7f68162ce700 1 -- 172.21.15.195:0/1007066681 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f67f80046d0 msgr2=0x7f67f8004b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:03:54.753 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.717+0000 7f68162ce700 1 --2- 172.21.15.195:0/1007066681 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f67f80046d0 0x7f67f8004b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:54.756 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.717+0000 7f68162ce700 1 -- 172.21.15.195:0/1007066681 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f67f8003920 msgr2=0x7f67f8003d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:03:54.756 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.717+0000 7f68162ce700 1 --2- 172.21.15.195:0/1007066681 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f67f8003920 0x7f67f8003d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:54.756 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.717+0000 7f68162ce700 1 -- 172.21.15.195:0/1007066681 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67f8003430 con 0x7f67f8005040 2022-06-27T12:03:54.756 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.717+0000 7f68162ce700 1 --2- 172.21.15.195:0/1007066681 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f67f8005040 0x7f67f800b920 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f680800c5a0 tx=0x7f6808001d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43dc3b46cfff8215 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:03:54.757 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.717+0000 7f6814acb700 1 -- 172.21.15.195:0/1007066681 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f680800ce90 con 0x7f67f8005040 2022-06-27T12:03:54.757 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.717+0000 7f6814acb700 1 -- 172.21.15.195:0/1007066681 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6808005b60 con 0x7f67f8005040 2022-06-27T12:03:54.758 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.717+0000 7f6814acb700 1 -- 172.21.15.195:0/1007066681 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68080070d0 con 0x7f67f8005040 2022-06-27T12:03:54.758 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.717+0000 7f6817d31700 1 -- 172.21.15.195:0/1007066681 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f67f8005040 msgr2=0x7f67f800b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:54.758 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.717+0000 7f6817d31700 1 --2- 172.21.15.195:0/1007066681 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f67f8005040 0x7f67f800b920 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f680800c5a0 tx=0x7f6808001d90 comp rx=0 tx=0).stop 2022-06-27T12:03:54.759 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.717+0000 7f68162ce700 1 -- 172.21.15.195:0/1007066681 reap_dead start 2022-06-27T12:03:54.759 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.717+0000 7f6817d31700 1 -- 172.21.15.195:0/1007066681 shutdown_connections 2022-06-27T12:03:54.760 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.717+0000 7f6817d31700 1 -- 172.21.15.195:0/1007066681 >> 172.21.15.195:0/1007066681 conn(0x7f68100c6350 msgr2=0x7f68100c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:03:54.760 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.718+0000 7f6817d31700 1 -- 172.21.15.195:0/1007066681 shutdown_connections 2022-06-27T12:03:54.760 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.718+0000 7f6817d31700 1 -- 172.21.15.195:0/1007066681 wait complete. 2022-06-27T12:03:54.760 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.719+0000 7f6817d31700 1 Processor -- start 2022-06-27T12:03:54.761 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.719+0000 7f6817d31700 1 -- start start 2022-06-27T12:03:54.761 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.719+0000 7f6817d31700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6810153400 0x7f681014f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:54.761 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.719+0000 7f6817d31700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f681014f760 0x7f681014fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:54.761 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.719+0000 7f6817d31700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f68101537d0 0x7f6810150300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:54.761 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.719+0000 7f6817d31700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681014f0c0 con 0x7f6810153400 2022-06-27T12:03:54.762 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.719+0000 7f6817d31700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68101649e0 con 0x7f68101537d0 2022-06-27T12:03:54.762 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.719+0000 7f6817d31700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6810164080 con 0x7f681014f760 2022-06-27T12:03:54.762 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.719+0000 7f6815acd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6810153400 0x7f681014f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:54.762 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.719+0000 7f6815acd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6810153400 0x7f681014f220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47388/0 (socket says 172.21.15.195:47388) 2022-06-27T12:03:54.763 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.719+0000 7f6815acd700 1 -- 172.21.15.195:0/313572472 learned_addr learned my addr 172.21.15.195:0/313572472 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:03:54.763 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.720+0000 7f68162ce700 1 --2- 172.21.15.195:0/313572472 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f68101537d0 0x7f6810150300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:54.763 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.720+0000 7f6815acd700 1 -- 172.21.15.195:0/313572472 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f681014f760 msgr2=0x7f681014fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:54.763 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.720+0000 7f6815acd700 1 --2- 172.21.15.195:0/313572472 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f681014f760 0x7f681014fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:54.770 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.720+0000 7f6815acd700 1 -- 172.21.15.195:0/313572472 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f68101537d0 msgr2=0x7f6810150300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:54.771 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.720+0000 7f6815acd700 1 --2- 172.21.15.195:0/313572472 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f68101537d0 0x7f6810150300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:54.771 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.720+0000 7f6815acd700 1 -- 172.21.15.195:0/313572472 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6810164e00 con 0x7f6810153400 2022-06-27T12:03:54.772 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.720+0000 7f68162ce700 1 --2- 172.21.15.195:0/313572472 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f68101537d0 0x7f6810150300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T12:03:54.772 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.720+0000 7f6815acd700 1 --2- 172.21.15.195:0/313572472 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6810153400 0x7f681014f220 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f6804011910 tx=0x7f6804009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:03:54.773 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.720+0000 7f6801ffb700 1 -- 172.21.15.195:0/313572472 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6804011eb0 con 0x7f6810153400 2022-06-27T12:03:54.773 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.720+0000 7f6801ffb700 1 -- 172.21.15.195:0/313572472 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6804004240 con 0x7f6810153400 2022-06-27T12:03:54.774 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.720+0000 7f6817d31700 1 -- 172.21.15.195:0/313572472 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6810168000 con 0x7f6810153400 2022-06-27T12:03:54.774 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.720+0000 7f6801ffb700 1 -- 172.21.15.195:0/313572472 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f680400f9f0 con 0x7f6810153400 2022-06-27T12:03:54.775 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.720+0000 7f6817d31700 1 -- 172.21.15.195:0/313572472 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6810164600 con 0x7f6810153400 2022-06-27T12:03:54.775 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.721+0000 7f6817d31700 1 -- 172.21.15.195:0/313572472 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68100722c0 con 0x7f6810153400 2022-06-27T12:03:54.776 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.722+0000 7f6801ffb700 1 -- 172.21.15.195:0/313572472 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f6804003520 con 0x7f6810153400 2022-06-27T12:03:54.776 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.722+0000 7f6801ffb700 1 --2- 172.21.15.195:0/313572472 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f67f0063d30 0x7f67f00661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:54.777 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.722+0000 7f6801ffb700 1 -- 172.21.15.195:0/313572472 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6890+0+0 (secure 0 0 0) 0x7f6804018030 con 0x7f6810153400 2022-06-27T12:03:54.777 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.722+0000 7f68152cc700 1 --2- 172.21.15.195:0/313572472 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f67f0063d30 0x7f67f00661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:54.778 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.723+0000 7f68152cc700 1 --2- 172.21.15.195:0/313572472 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f67f0063d30 0x7f67f00661e0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f680c006d70 tx=0x7f680c009060 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:03:54.778 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.726+0000 7f6801ffb700 1 -- 172.21.15.195:0/313572472 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f68040500e0 con 0x7f6810153400 2022-06-27T12:03:54.947 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.947+0000 7f6817d31700 1 -- 172.21.15.195:0/313572472 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f681006f240 con 0x7f6810153400 2022-06-27T12:03:54.948 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.948+0000 7f6801ffb700 1 -- 172.21.15.195:0/313572472 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+5371 (secure 0 0 0) 0x7f6804053700 con 0x7f6810153400 2022-06-27T12:03:54.948 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:03:54.948 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:03:49.233804+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5818},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5818":{"gid":5818,"name":"b","rank":0,"incarnation":55,"state":"up:reconnect","state_seq":27,"addr":"172.21.15.202:6833/3399027527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3399027527},{"type":"v1","addr":"172.21.15.202:6833","nonce":3399027527}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:03:54.950 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.950+0000 7f6817d31700 1 -- 172.21.15.195:0/313572472 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f67f0063d30 msgr2=0x7f67f00661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:54.950 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.950+0000 7f6817d31700 1 --2- 172.21.15.195:0/313572472 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f67f0063d30 0x7f67f00661e0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f680c006d70 tx=0x7f680c009060 comp rx=0 tx=0).stop 2022-06-27T12:03:54.951 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.950+0000 7f6817d31700 1 -- 172.21.15.195:0/313572472 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6810153400 msgr2=0x7f681014f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:54.951 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.950+0000 7f6817d31700 1 --2- 172.21.15.195:0/313572472 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6810153400 0x7f681014f220 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f6804011910 tx=0x7f6804009060 comp rx=0 tx=0).stop 2022-06-27T12:03:54.952 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.950+0000 7f6817d31700 1 -- 172.21.15.195:0/313572472 shutdown_connections 2022-06-27T12:03:54.952 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.950+0000 7f6817d31700 1 --2- 172.21.15.195:0/313572472 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f67f0063d30 0x7f67f00661e0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:54.952 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.950+0000 7f6817d31700 1 --2- 172.21.15.195:0/313572472 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6810153400 0x7f681014f220 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:54.953 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.950+0000 7f6817d31700 1 --2- 172.21.15.195:0/313572472 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f68101537d0 0x7f6810150300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:54.953 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.950+0000 7f6817d31700 1 --2- 172.21.15.195:0/313572472 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f681014f760 0x7f681014fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:54.969 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.950+0000 7f6817d31700 1 -- 172.21.15.195:0/313572472 >> 172.21.15.195:0/313572472 conn(0x7f68100c6350 msgr2=0x7f68100c8d20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:03:54.969 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.950+0000 7f6817d31700 1 -- 172.21.15.195:0/313572472 shutdown_connections 2022-06-27T12:03:54.971 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:54.950+0000 7f6817d31700 1 -- 172.21.15.195:0/313572472 wait complete. 2022-06-27T12:03:54.971 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 56 2022-06-27T12:03:54.985 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:03:56.992 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:03:57.125 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.124+0000 7f050c382700 1 Processor -- start 2022-06-27T12:03:57.127 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.125+0000 7f050c382700 1 -- start start 2022-06-27T12:03:57.141 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.125+0000 7f050c382700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f050406e850 0x7f05041570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:57.142 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.125+0000 7f050c382700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f05040c34a0 0x7f05040c2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:57.142 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.125+0000 7f050c382700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f050406fba0 con 0x7f050406fef0 2022-06-27T12:03:57.142 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.126+0000 7f050c382700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05041575e0 con 0x7f05040c34a0 2022-06-27T12:03:57.142 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.126+0000 7f050c382700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f050406f5e0 con 0x7f050406e850 2022-06-27T12:03:57.142 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.126+0000 7f050a11e700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f050406e850 0x7f05041570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:57.143 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.126+0000 7f050a91f700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f05040c34a0 0x7f05040c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:57.143 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.126+0000 7f050991d700 1 --1- >> v1:172.21.15.202:6789/0 conn(0x7f050406fef0 0x7f0504069de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.202:6789/0 says I am v1:172.21.15.195:55446/0 (socket says 172.21.15.195:55446) 2022-06-27T12:03:57.143 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.126+0000 7f050991d700 1 -- 172.21.15.195:0/701019161 learned_addr learned my addr 172.21.15.195:0/701019161 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:03:57.143 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.126+0000 7f050911c700 1 -- 172.21.15.195:0/701019161 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1450211390 0 0) 0x7f050406fba0 con 0x7f050406fef0 2022-06-27T12:03:57.144 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.126+0000 7f050911c700 1 -- 172.21.15.195:0/701019161 --> v1:172.21.15.202:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04ec003430 con 0x7f050406fef0 2022-06-27T12:03:57.144 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.126+0000 7f050a91f700 1 -- 172.21.15.195:0/701019161 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f050406e850 msgr2=0x7f05041570a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:03:57.144 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.126+0000 7f050a91f700 1 --2- 172.21.15.195:0/701019161 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f050406e850 0x7f05041570a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:57.144 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.126+0000 7f050a91f700 1 -- 172.21.15.195:0/701019161 >> v1:172.21.15.202:6789/0 conn(0x7f050406fef0 legacy=0x7f0504069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:57.145 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.126+0000 7f050a91f700 1 -- 172.21.15.195:0/701019161 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f050406f8c0 con 0x7f05040c34a0 2022-06-27T12:03:57.145 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.126+0000 7f050a11e700 1 --2- 172.21.15.195:0/701019161 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f050406e850 0x7f05041570a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:03:57.145 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.126+0000 7f050a91f700 1 --2- 172.21.15.195:0/701019161 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f05040c34a0 0x7f05040c2920 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f04fc009e60 tx=0x7f04fc009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa218c76d8963b1a server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:03:57.145 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.127+0000 7f050911c700 1 -- 172.21.15.195:0/701019161 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04fc00e070 con 0x7f05040c34a0 2022-06-27T12:03:57.145 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.127+0000 7f050911c700 1 -- 172.21.15.195:0/701019161 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04fc00bb40 con 0x7f05040c34a0 2022-06-27T12:03:57.146 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.127+0000 7f050911c700 1 -- 172.21.15.195:0/701019161 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04fc012430 con 0x7f05040c34a0 2022-06-27T12:03:57.146 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.127+0000 7f050c382700 1 -- 172.21.15.195:0/701019161 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f05040c34a0 msgr2=0x7f05040c2920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:57.146 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.127+0000 7f050c382700 1 --2- 172.21.15.195:0/701019161 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f05040c34a0 0x7f05040c2920 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f04fc009e60 tx=0x7f04fc009780 comp rx=0 tx=0).stop 2022-06-27T12:03:57.146 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.127+0000 7f050c382700 1 -- 172.21.15.195:0/701019161 shutdown_connections 2022-06-27T12:03:57.147 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.127+0000 7f050c382700 1 --2- 172.21.15.195:0/701019161 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f05040c34a0 0x7f05040c2920 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:57.147 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.127+0000 7f050c382700 1 --2- 172.21.15.195:0/701019161 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f050406e850 0x7f05041570a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:57.147 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.127+0000 7f050c382700 1 -- 172.21.15.195:0/701019161 >> 172.21.15.195:0/701019161 conn(0x7f05040c6350 msgr2=0x7f05040c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:03:57.147 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.127+0000 7f050c382700 1 -- 172.21.15.195:0/701019161 shutdown_connections 2022-06-27T12:03:57.148 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.127+0000 7f050c382700 1 -- 172.21.15.195:0/701019161 wait complete. 2022-06-27T12:03:57.148 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.129+0000 7f050c382700 1 Processor -- start 2022-06-27T12:03:57.148 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.129+0000 7f050c382700 1 -- start start 2022-06-27T12:03:57.148 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.130+0000 7f050c382700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f050406e850 0x7f0504162820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:57.149 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.130+0000 7f050c382700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f05040c34a0 0x7f0504148d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:57.149 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.130+0000 7f050c382700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f05040bd640 0x7f050414a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:57.149 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.130+0000 7f050c382700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05040bee20 con 0x7f050406e850 2022-06-27T12:03:57.149 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.130+0000 7f050c382700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0504166450 con 0x7f05040c34a0 2022-06-27T12:03:57.150 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.130+0000 7f050c382700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0504162390 con 0x7f05040bd640 2022-06-27T12:03:57.150 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.130+0000 7f050a11e700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f050406e850 0x7f0504162820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:57.150 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.130+0000 7f050a11e700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f050406e850 0x7f0504162820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47402/0 (socket says 172.21.15.195:47402) 2022-06-27T12:03:57.150 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.130+0000 7f050a11e700 1 -- 172.21.15.195:0/227586804 learned_addr learned my addr 172.21.15.195:0/227586804 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:03:57.150 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.130+0000 7f050a91f700 1 --2- 172.21.15.195:0/227586804 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f05040bd640 0x7f050414a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:57.151 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.130+0000 7f050991d700 1 --2- 172.21.15.195:0/227586804 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f05040c34a0 0x7f0504148d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:57.151 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.130+0000 7f050a11e700 1 -- 172.21.15.195:0/227586804 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f05040bd640 msgr2=0x7f050414a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:57.151 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.130+0000 7f050a11e700 1 --2- 172.21.15.195:0/227586804 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f05040bd640 0x7f050414a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:57.151 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.130+0000 7f050a11e700 1 -- 172.21.15.195:0/227586804 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f05040c34a0 msgr2=0x7f0504148d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:57.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.130+0000 7f050a11e700 1 --2- 172.21.15.195:0/227586804 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f05040c34a0 0x7f0504148d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:57.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.130+0000 7f050a11e700 1 -- 172.21.15.195:0/227586804 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04f40092e0 con 0x7f050406e850 2022-06-27T12:03:57.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.130+0000 7f050a11e700 1 --2- 172.21.15.195:0/227586804 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f050406e850 0x7f0504162820 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f04f400ea50 tx=0x7f04f4009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:03:57.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.130+0000 7f04f27fc700 1 -- 172.21.15.195:0/227586804 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04f400ee90 con 0x7f050406e850 2022-06-27T12:03:57.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.131+0000 7f04f27fc700 1 -- 172.21.15.195:0/227586804 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04f400c850 con 0x7f050406e850 2022-06-27T12:03:57.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.131+0000 7f050c382700 1 -- 172.21.15.195:0/227586804 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04fc009a20 con 0x7f050406e850 2022-06-27T12:03:57.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.131+0000 7f04f27fc700 1 -- 172.21.15.195:0/227586804 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04f400cbd0 con 0x7f050406e850 2022-06-27T12:03:57.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.131+0000 7f050c382700 1 -- 172.21.15.195:0/227586804 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05040bde50 con 0x7f050406e850 2022-06-27T12:03:57.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.131+0000 7f050c382700 1 -- 172.21.15.195:0/227586804 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0504162670 con 0x7f050406e850 2022-06-27T12:03:57.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.132+0000 7f04f27fc700 1 -- 172.21.15.195:0/227586804 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f04f4019030 con 0x7f050406e850 2022-06-27T12:03:57.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.132+0000 7f04f27fc700 1 --2- 172.21.15.195:0/227586804 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f04e4063d00 0x7f04e40661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:57.155 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.132+0000 7f04f27fc700 1 -- 172.21.15.195:0/227586804 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6890+0+0 (secure 0 0 0) 0x7f04f4080bc0 con 0x7f050406e850 2022-06-27T12:03:57.155 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.132+0000 7f050991d700 1 --2- 172.21.15.195:0/227586804 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f04e4063d00 0x7f04e40661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:57.155 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.133+0000 7f050991d700 1 --2- 172.21.15.195:0/227586804 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f04e4063d00 0x7f04e40661b0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f0500007fd0 tx=0x7f0500009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:03:57.155 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.137+0000 7f04f27fc700 1 -- 172.21.15.195:0/227586804 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f04f404f470 con 0x7f050406e850 2022-06-27T12:03:57.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.367+0000 7f050c382700 1 -- 172.21.15.195:0/227586804 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f05040722c0 con 0x7f050406e850 2022-06-27T12:03:57.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.368+0000 7f04f27fc700 1 -- 172.21.15.195:0/227586804 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+5368 (secure 0 0 0) 0x7f04f4021090 con 0x7f050406e850 2022-06-27T12:03:57.372 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:03:57.372 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:03:55.249002+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5818},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5818":{"gid":5818,"name":"b","rank":0,"incarnation":55,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.202:6833/3399027527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3399027527},{"type":"v1","addr":"172.21.15.202:6833","nonce":3399027527}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:03:57.380 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.370+0000 7f050c382700 1 -- 172.21.15.195:0/227586804 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f04e4063d00 msgr2=0x7f04e40661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:57.381 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.370+0000 7f050c382700 1 --2- 172.21.15.195:0/227586804 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f04e4063d00 0x7f04e40661b0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f0500007fd0 tx=0x7f0500009040 comp rx=0 tx=0).stop 2022-06-27T12:03:57.381 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.370+0000 7f050c382700 1 -- 172.21.15.195:0/227586804 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f050406e850 msgr2=0x7f0504162820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:57.382 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.370+0000 7f050c382700 1 --2- 172.21.15.195:0/227586804 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f050406e850 0x7f0504162820 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f04f400ea50 tx=0x7f04f4009060 comp rx=0 tx=0).stop 2022-06-27T12:03:57.382 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.370+0000 7f050c382700 1 -- 172.21.15.195:0/227586804 shutdown_connections 2022-06-27T12:03:57.382 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.370+0000 7f050c382700 1 --2- 172.21.15.195:0/227586804 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f04e4063d00 0x7f04e40661b0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:57.383 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.370+0000 7f050c382700 1 --2- 172.21.15.195:0/227586804 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f050406e850 0x7f0504162820 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:57.383 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.370+0000 7f050c382700 1 --2- 172.21.15.195:0/227586804 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f05040bd640 0x7f050414a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:57.390 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.370+0000 7f050c382700 1 --2- 172.21.15.195:0/227586804 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f05040c34a0 0x7f0504148d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:57.390 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.370+0000 7f050c382700 1 -- 172.21.15.195:0/227586804 >> 172.21.15.195:0/227586804 conn(0x7f05040c6350 msgr2=0x7f050414d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:03:57.391 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.370+0000 7f050c382700 1 -- 172.21.15.195:0/227586804 shutdown_connections 2022-06-27T12:03:57.391 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:57.370+0000 7f050c382700 1 -- 172.21.15.195:0/227586804 wait complete. 2022-06-27T12:03:57.392 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 57 2022-06-27T12:03:57.410 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:03:59.414 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:03:59.586 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.569+0000 7fda2fce4700 1 Processor -- start 2022-06-27T12:03:59.586 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.569+0000 7fda2fce4700 1 -- start start 2022-06-27T12:03:59.587 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.570+0000 7fda2fce4700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fda2806a450 0x7fda28063af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:59.587 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.570+0000 7fda2fce4700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda28064030 con 0x7fda28150500 2022-06-27T12:03:59.588 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.570+0000 7fda2fce4700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda280c4160 con 0x7fda281508d0 2022-06-27T12:03:59.588 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.570+0000 7fda2fce4700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda280c4440 con 0x7fda2806a450 2022-06-27T12:03:59.588 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.570+0000 7fda2d27f700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fda2806a450 0x7fda28063af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:59.589 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.570+0000 7fda2da80700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fda28150500 0x7fda280c48d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:53598/0 (socket says 172.21.15.195:53598) 2022-06-27T12:03:59.589 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.570+0000 7fda2da80700 1 -- 172.21.15.195:0/1825752864 learned_addr learned my addr 172.21.15.195:0/1825752864 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:03:59.590 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.570+0000 7fda2d27f700 1 -- 172.21.15.195:0/1825752864 >> v1:172.21.15.202:6789/0 conn(0x7fda281508d0 legacy=0x7fda280bbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:03:59.590 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.570+0000 7fda2d27f700 1 -- 172.21.15.195:0/1825752864 >> v1:172.21.15.195:6789/0 conn(0x7fda28150500 legacy=0x7fda280c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:59.590 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.570+0000 7fda2d27f700 1 -- 172.21.15.195:0/1825752864 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda2815a800 con 0x7fda2806a450 2022-06-27T12:03:59.591 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.570+0000 7fda2ca7e700 1 -- 172.21.15.195:0/1825752864 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 272309313 0 0) 0x7fda24001290 con 0x7fda28150500 2022-06-27T12:03:59.596 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.570+0000 7fda2d27f700 1 --2- 172.21.15.195:0/1825752864 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fda2806a450 0x7fda28063af0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fda180097a0 tx=0x7fda18009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9b88ae49e6f277c server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:03:59.596 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.571+0000 7fda2ca7e700 1 -- 172.21.15.195:0/1825752864 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda18004030 con 0x7fda2806a450 2022-06-27T12:03:59.596 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.571+0000 7fda2ca7e700 1 -- 172.21.15.195:0/1825752864 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda18004190 con 0x7fda2806a450 2022-06-27T12:03:59.597 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.571+0000 7fda2ca7e700 1 -- 172.21.15.195:0/1825752864 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda18011630 con 0x7fda2806a450 2022-06-27T12:03:59.597 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.571+0000 7fda2fce4700 1 -- 172.21.15.195:0/1825752864 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fda2806a450 msgr2=0x7fda28063af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:59.597 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.571+0000 7fda2fce4700 1 --2- 172.21.15.195:0/1825752864 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fda2806a450 0x7fda28063af0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fda180097a0 tx=0x7fda18009060 comp rx=0 tx=0).stop 2022-06-27T12:03:59.598 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.571+0000 7fda2fce4700 1 -- 172.21.15.195:0/1825752864 shutdown_connections 2022-06-27T12:03:59.598 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.571+0000 7fda2fce4700 1 --2- 172.21.15.195:0/1825752864 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fda2806a450 0x7fda28063af0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:59.598 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.571+0000 7fda2fce4700 1 -- 172.21.15.195:0/1825752864 >> 172.21.15.195:0/1825752864 conn(0x7fda280c9b00 msgr2=0x7fda280c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:03:59.599 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.571+0000 7fda2fce4700 1 -- 172.21.15.195:0/1825752864 shutdown_connections 2022-06-27T12:03:59.599 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.571+0000 7fda2fce4700 1 -- 172.21.15.195:0/1825752864 wait complete. 2022-06-27T12:03:59.599 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.572+0000 7fda2fce4700 1 Processor -- start 2022-06-27T12:03:59.599 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.572+0000 7fda2fce4700 1 -- start start 2022-06-27T12:03:59.599 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.573+0000 7fda2fce4700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fda2806a450 0x7fda280c0350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:59.600 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.573+0000 7fda2fce4700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fda28150500 0x7fda2816a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:59.600 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.573+0000 7fda2fce4700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fda281508d0 0x7fda2806de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:59.600 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.573+0000 7fda2fce4700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda2816ac90 con 0x7fda2806a450 2022-06-27T12:03:59.600 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.573+0000 7fda2fce4700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda280bf970 con 0x7fda28150500 2022-06-27T12:03:59.601 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.573+0000 7fda2fce4700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda2806ec00 con 0x7fda281508d0 2022-06-27T12:03:59.601 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.573+0000 7fda2da80700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fda2806a450 0x7fda280c0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:59.601 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.573+0000 7fda2da80700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fda2806a450 0x7fda280c0350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47416/0 (socket says 172.21.15.195:47416) 2022-06-27T12:03:59.601 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.573+0000 7fda2da80700 1 -- 172.21.15.195:0/3831988549 learned_addr learned my addr 172.21.15.195:0/3831988549 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:03:59.601 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.573+0000 7fda2d27f700 1 --2- 172.21.15.195:0/3831988549 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fda28150500 0x7fda2816a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:59.602 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.573+0000 7fda2e281700 1 --2- 172.21.15.195:0/3831988549 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fda281508d0 0x7fda2806de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:59.602 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.573+0000 7fda2da80700 1 -- 172.21.15.195:0/3831988549 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fda281508d0 msgr2=0x7fda2806de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:59.602 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.573+0000 7fda2da80700 1 --2- 172.21.15.195:0/3831988549 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fda281508d0 0x7fda2806de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:59.602 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.573+0000 7fda2da80700 1 -- 172.21.15.195:0/3831988549 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fda28150500 msgr2=0x7fda2816a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:59.603 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.573+0000 7fda2da80700 1 --2- 172.21.15.195:0/3831988549 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fda28150500 0x7fda2816a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:59.603 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.573+0000 7fda2da80700 1 -- 172.21.15.195:0/3831988549 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda180092e0 con 0x7fda2806a450 2022-06-27T12:03:59.604 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.574+0000 7fda2d27f700 1 --2- 172.21.15.195:0/3831988549 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fda28150500 0x7fda2816a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T12:03:59.604 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.574+0000 7fda2e281700 1 --2- 172.21.15.195:0/3831988549 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fda281508d0 0x7fda2806de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T12:03:59.604 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.574+0000 7fda2da80700 1 --2- 172.21.15.195:0/3831988549 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fda2806a450 0x7fda280c0350 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fda2400c5b0 tx=0x7fda24001890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:03:59.605 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.574+0000 7fda15ffb700 1 -- 172.21.15.195:0/3831988549 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda240072e0 con 0x7fda2806a450 2022-06-27T12:03:59.606 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.574+0000 7fda15ffb700 1 -- 172.21.15.195:0/3831988549 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda24007440 con 0x7fda2806a450 2022-06-27T12:03:59.609 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.574+0000 7fda15ffb700 1 -- 172.21.15.195:0/3831988549 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda24005900 con 0x7fda2806a450 2022-06-27T12:03:59.610 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.574+0000 7fda2fce4700 1 -- 172.21.15.195:0/3831988549 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda280bf640 con 0x7fda2806a450 2022-06-27T12:03:59.610 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.574+0000 7fda2fce4700 1 -- 172.21.15.195:0/3831988549 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda2806f180 con 0x7fda2806a450 2022-06-27T12:03:59.611 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.575+0000 7fda2fce4700 1 -- 172.21.15.195:0/3831988549 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda280bfce0 con 0x7fda2806a450 2022-06-27T12:03:59.611 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.575+0000 7fda15ffb700 1 -- 172.21.15.195:0/3831988549 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fda24019030 con 0x7fda2806a450 2022-06-27T12:03:59.612 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.576+0000 7fda15ffb700 1 --2- 172.21.15.195:0/3831988549 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fda08063d00 0x7fda080661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:03:59.612 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.576+0000 7fda15ffb700 1 -- 172.21.15.195:0/3831988549 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6890+0+0 (secure 0 0 0) 0x7fda2407fc80 con 0x7fda2806a450 2022-06-27T12:03:59.613 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.576+0000 7fda2d27f700 1 --2- 172.21.15.195:0/3831988549 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fda08063d00 0x7fda080661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:03:59.613 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.576+0000 7fda2d27f700 1 --2- 172.21.15.195:0/3831988549 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fda08063d00 0x7fda080661b0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fda180097a0 tx=0x7fda18016000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:03:59.613 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.580+0000 7fda15ffb700 1 -- 172.21.15.195:0/3831988549 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fda2404e4d0 con 0x7fda2806a450 2022-06-27T12:03:59.809 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.807+0000 7fda2fce4700 1 -- 172.21.15.195:0/3831988549 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fda28064030 con 0x7fda2806a450 2022-06-27T12:03:59.810 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.807+0000 7fda15ffb700 1 -- 172.21.15.195:0/3831988549 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+5368 (secure 0 0 0) 0x7fda28064030 con 0x7fda2806a450 2022-06-27T12:03:59.811 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:03:59.811 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:03:55.249002+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5818},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5818":{"gid":5818,"name":"b","rank":0,"incarnation":55,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.202:6833/3399027527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3399027527},{"type":"v1","addr":"172.21.15.202:6833","nonce":3399027527}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:03:59.815 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.809+0000 7fda2fce4700 1 -- 172.21.15.195:0/3831988549 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fda08063d00 msgr2=0x7fda080661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:59.816 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.809+0000 7fda2fce4700 1 --2- 172.21.15.195:0/3831988549 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fda08063d00 0x7fda080661b0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fda180097a0 tx=0x7fda18016000 comp rx=0 tx=0).stop 2022-06-27T12:03:59.817 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.809+0000 7fda2fce4700 1 -- 172.21.15.195:0/3831988549 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fda2806a450 msgr2=0x7fda280c0350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:03:59.817 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.809+0000 7fda2fce4700 1 --2- 172.21.15.195:0/3831988549 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fda2806a450 0x7fda280c0350 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fda2400c5b0 tx=0x7fda24001890 comp rx=0 tx=0).stop 2022-06-27T12:03:59.817 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.810+0000 7fda2fce4700 1 -- 172.21.15.195:0/3831988549 shutdown_connections 2022-06-27T12:03:59.818 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.810+0000 7fda2fce4700 1 --2- 172.21.15.195:0/3831988549 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fda08063d00 0x7fda080661b0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:59.818 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.810+0000 7fda2fce4700 1 --2- 172.21.15.195:0/3831988549 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fda2806a450 0x7fda280c0350 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:59.819 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.810+0000 7fda2fce4700 1 --2- 172.21.15.195:0/3831988549 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fda281508d0 0x7fda2806de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:59.819 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.810+0000 7fda2fce4700 1 --2- 172.21.15.195:0/3831988549 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fda28150500 0x7fda2816a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:03:59.820 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.810+0000 7fda2fce4700 1 -- 172.21.15.195:0/3831988549 >> 172.21.15.195:0/3831988549 conn(0x7fda280c9b00 msgr2=0x7fda2814fb60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:03:59.820 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.810+0000 7fda2fce4700 1 -- 172.21.15.195:0/3831988549 shutdown_connections 2022-06-27T12:03:59.821 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:03:59.810+0000 7fda2fce4700 1 -- 172.21.15.195:0/3831988549 wait complete. 2022-06-27T12:03:59.821 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 57 2022-06-27T12:03:59.825 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 57, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-06-27T11:47:52.358095+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 57, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5818": { "addr": "172.21.15.202:6833/3399027527", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6832", "nonce": 3399027527, "type": "v2" }, { "addr": "172.21.15.202:6833", "nonce": 3399027527, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5818, "incarnation": 55, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:rejoin", "state_seq": 29 } }, "last_failure": 0, "last_failure_osd_epoch": 56, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-06-27T12:03:55.249002+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5818 } } } ], "standbys": [ { "addr": "172.21.15.202:6837/3305006841", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6836", "nonce": 3305006841, "type": "v2" }, { "addr": "172.21.15.202:6837", "nonce": 3305006841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6835/2060709822", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6834", "nonce": 2060709822, "type": "v2" }, { "addr": "172.21.15.195:6835", "nonce": 2060709822, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6838/1598751652", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6836", "nonce": 1598751652, "type": "v2" }, { "addr": "172.21.15.195:6838", "nonce": 1598751652, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4275, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6839/1313800661", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6837", "nonce": 1313800661, "type": "v2" }, { "addr": "172.21.15.195:6839", "nonce": 1313800661, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4278, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-06-27T12:04:00.754 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:04:00.759 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:04:01.830 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:04:01.984 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:01.969+0000 7fcabdd32700 1 Processor -- start 2022-06-27T12:04:01.984 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:01.969+0000 7fcabdd32700 1 -- start start 2022-06-27T12:04:01.985 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:01.970+0000 7fcabdd32700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fcab814e490 0x7fcab806cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:01.985 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:01.970+0000 7fcabdd32700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fcab814e860 0x7fcab80c3840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:01.986 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:01.970+0000 7fcabdd32700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcab80c3d80 0x7fcab806f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:01.986 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:01.970+0000 7fcabdd32700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcab80c33b0 con 0x7fcab814e490 2022-06-27T12:04:01.987 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:01.970+0000 7fcabdd32700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcab806d520 con 0x7fcab814e860 2022-06-27T12:04:01.987 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:01.970+0000 7fcabdd32700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcab806f220 con 0x7fcab80c3d80 2022-06-27T12:04:01.987 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:01.970+0000 7fcab77fe700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fcab814e490 0x7fcab806cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:01.988 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:01.970+0000 7fcab77fe700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fcab814e490 0x7fcab806cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47424/0 (socket says 172.21.15.195:47424) 2022-06-27T12:04:01.988 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:01.970+0000 7fcab77fe700 1 -- 172.21.15.195:0/1107727877 learned_addr learned my addr 172.21.15.195:0/1107727877 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:04:01.989 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:01.970+0000 7fcab6ffd700 1 --2- 172.21.15.195:0/1107727877 >> v2:172.21.15.202:3300/0 conn(0x7fcab814e860 0x7fcab80c3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:01.989 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:01.970+0000 7fcab7fff700 1 --2- 172.21.15.195:0/1107727877 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcab80c3d80 0x7fcab806f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:01.998 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:01.970+0000 7fcab77fe700 1 -- 172.21.15.195:0/1107727877 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcab80c3d80 msgr2=0x7fcab806f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:04:01.998 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:01.970+0000 7fcab77fe700 1 --2- 172.21.15.195:0/1107727877 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcab80c3d80 0x7fcab806f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:01.999 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:01.970+0000 7fcab77fe700 1 -- 172.21.15.195:0/1107727877 >> v2:172.21.15.202:3300/0 conn(0x7fcab814e860 msgr2=0x7fcab80c3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:04:01.999 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:01.970+0000 7fcab77fe700 1 --2- 172.21.15.195:0/1107727877 >> v2:172.21.15.202:3300/0 conn(0x7fcab814e860 0x7fcab80c3840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:02.000 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:01.970+0000 7fcab77fe700 1 -- 172.21.15.195:0/1107727877 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcab81542c0 con 0x7fcab814e490 2022-06-27T12:04:02.000 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:01.970+0000 7fcab6ffd700 1 --2- 172.21.15.195:0/1107727877 >> v2:172.21.15.202:3300/0 conn(0x7fcab814e860 0x7fcab80c3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:04:02.000 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:01.970+0000 7fcab77fe700 1 --2- 172.21.15.195:0/1107727877 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcab814e490 0x7fcab806cfe0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fcab8065b80 tx=0x7fcaac009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c787a603172ce306 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:04:02.001 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:01.970+0000 7fcab67fc700 1 -- 172.21.15.195:0/1107727877 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaac00b640 con 0x7fcab814e490 2022-06-27T12:04:02.002 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:01.971+0000 7fcab67fc700 1 -- 172.21.15.195:0/1107727877 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcaac00b7a0 con 0x7fcab814e490 2022-06-27T12:04:02.002 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:01.971+0000 7fcab67fc700 1 -- 172.21.15.195:0/1107727877 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaac00bb40 con 0x7fcab814e490 2022-06-27T12:04:02.003 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:01.971+0000 7fcabdd32700 1 -- 172.21.15.195:0/1107727877 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcab814e490 msgr2=0x7fcab806cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:02.003 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:01.971+0000 7fcabdd32700 1 --2- 172.21.15.195:0/1107727877 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcab814e490 0x7fcab806cfe0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fcab8065b80 tx=0x7fcaac009120 comp rx=0 tx=0).stop 2022-06-27T12:04:02.004 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:01.971+0000 7fcabdd32700 1 -- 172.21.15.195:0/1107727877 shutdown_connections 2022-06-27T12:04:02.004 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:01.971+0000 7fcabdd32700 1 --2- 172.21.15.195:0/1107727877 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcab80c3d80 0x7fcab806f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:02.005 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:01.971+0000 7fcabdd32700 1 --2- 172.21.15.195:0/1107727877 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcab814e490 0x7fcab806cfe0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:02.005 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:01.971+0000 7fcabdd32700 1 --2- 172.21.15.195:0/1107727877 >> v2:172.21.15.202:3300/0 conn(0x7fcab814e860 0x7fcab80c3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:02.013 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:01.971+0000 7fcabdd32700 1 -- 172.21.15.195:0/1107727877 >> 172.21.15.195:0/1107727877 conn(0x7fcab8063af0 msgr2=0x7fcab8063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:04:02.014 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:01.971+0000 7fcabdd32700 1 -- 172.21.15.195:0/1107727877 shutdown_connections 2022-06-27T12:04:02.015 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:01.971+0000 7fcabdd32700 1 -- 172.21.15.195:0/1107727877 wait complete. 2022-06-27T12:04:02.018 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:01.972+0000 7fcabdd32700 1 Processor -- start 2022-06-27T12:04:02.019 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:01.972+0000 7fcabdd32700 1 -- start start 2022-06-27T12:04:02.019 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:01.973+0000 7fcabdd32700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcab80c3d80 0x7fcab8165140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:02.020 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:01.973+0000 7fcabdd32700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcab814e490 0x7fcab8156b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:02.020 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:01.973+0000 7fcabdd32700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcab814e860 0x7fcab814b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:02.021 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:01.973+0000 7fcabdd32700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcab80ca030 con 0x7fcab80c3d80 2022-06-27T12:04:02.021 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:01.973+0000 7fcabdd32700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcab8164cb0 con 0x7fcab814e860 2022-06-27T12:04:02.021 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:01.973+0000 7fcabdd32700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcab81570d0 con 0x7fcab814e490 2022-06-27T12:04:02.022 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:01.973+0000 7fcab77fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcab80c3d80 0x7fcab8165140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:02.022 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:01.973+0000 7fcab77fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcab80c3d80 0x7fcab8165140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47430/0 (socket says 172.21.15.195:47430) 2022-06-27T12:04:02.023 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:01.973+0000 7fcab77fe700 1 -- 172.21.15.195:0/3797186419 learned_addr learned my addr 172.21.15.195:0/3797186419 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:04:02.023 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:01.973+0000 7fcab7fff700 1 --2- 172.21.15.195:0/3797186419 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcab814e860 0x7fcab814b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:02.023 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:01.973+0000 7fcab6ffd700 1 --2- 172.21.15.195:0/3797186419 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcab814e490 0x7fcab8156b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:02.024 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:01.974+0000 7fcab77fe700 1 -- 172.21.15.195:0/3797186419 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcab814e490 msgr2=0x7fcab8156b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:02.025 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:01.974+0000 7fcab77fe700 1 --2- 172.21.15.195:0/3797186419 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcab814e490 0x7fcab8156b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:02.026 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:01.974+0000 7fcab77fe700 1 -- 172.21.15.195:0/3797186419 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcab814e860 msgr2=0x7fcab814b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:02.026 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:01.974+0000 7fcab77fe700 1 --2- 172.21.15.195:0/3797186419 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcab814e860 0x7fcab814b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:02.027 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:01.974+0000 7fcab77fe700 1 -- 172.21.15.195:0/3797186419 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaac0093a0 con 0x7fcab80c3d80 2022-06-27T12:04:02.027 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:01.974+0000 7fcab6ffd700 1 --2- 172.21.15.195:0/3797186419 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcab814e490 0x7fcab8156b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:04:02.027 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:01.974+0000 7fcab77fe700 1 --2- 172.21.15.195:0/3797186419 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcab80c3d80 0x7fcab8165140 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fcaac009f80 tx=0x7fcaac017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:04:02.028 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:01.974+0000 7fca9f7fe700 1 -- 172.21.15.195:0/3797186419 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaac003730 con 0x7fcab80c3d80 2022-06-27T12:04:02.028 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:01.974+0000 7fca9f7fe700 1 -- 172.21.15.195:0/3797186419 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcaac003890 con 0x7fcab80c3d80 2022-06-27T12:04:02.029 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:01.974+0000 7fcabdd32700 1 -- 172.21.15.195:0/3797186419 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcab81646c0 con 0x7fcab80c3d80 2022-06-27T12:04:02.029 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:01.974+0000 7fca9f7fe700 1 -- 172.21.15.195:0/3797186419 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaac003c50 con 0x7fcab80c3d80 2022-06-27T12:04:02.030 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:01.974+0000 7fcabdd32700 1 -- 172.21.15.195:0/3797186419 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcab8158140 con 0x7fcab80c3d80 2022-06-27T12:04:02.030 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:01.975+0000 7fcabdd32700 1 -- 172.21.15.195:0/3797186419 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcab8164f90 con 0x7fcab80c3d80 2022-06-27T12:04:02.033 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:01.975+0000 7fca9f7fe700 1 -- 172.21.15.195:0/3797186419 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fcaac02b030 con 0x7fcab80c3d80 2022-06-27T12:04:02.034 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:01.976+0000 7fca9f7fe700 1 --2- 172.21.15.195:0/3797186419 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fca98063d00 0x7fca980661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:02.034 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:01.976+0000 7fca9f7fe700 1 -- 172.21.15.195:0/3797186419 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6890+0+0 (secure 0 0 0) 0x7fcaac0821c0 con 0x7fcab80c3d80 2022-06-27T12:04:02.035 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:01.976+0000 7fcab6ffd700 1 --2- 172.21.15.195:0/3797186419 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fca98063d00 0x7fca980661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:02.035 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:01.976+0000 7fcab6ffd700 1 --2- 172.21.15.195:0/3797186419 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fca98063d00 0x7fca980661b0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fcab80c2f90 tx=0x7fcaa8008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:04:02.036 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:01.980+0000 7fca9f7fe700 1 -- 172.21.15.195:0/3797186419 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fcaac050a70 con 0x7fcab80c3d80 2022-06-27T12:04:02.207 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:02.207+0000 7fcabdd32700 1 -- 172.21.15.195:0/3797186419 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcab80722c0 con 0x7fcab80c3d80 2022-06-27T12:04:02.208 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:02.208+0000 7fca9f7fe700 1 -- 172.21.15.195:0/3797186419 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+5374 (secure 0 0 0) 0x7fcaac029090 con 0x7fcab80c3d80 2022-06-27T12:04:02.209 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:04:02.209 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:04:01.265250+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5818},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5818":{"gid":5818,"name":"b","rank":0,"incarnation":55,"state":"up:clientreplay","state_seq":31,"addr":"172.21.15.202:6833/3399027527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3399027527},{"type":"v1","addr":"172.21.15.202:6833","nonce":3399027527}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:04:02.212 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:02.210+0000 7fcabdd32700 1 -- 172.21.15.195:0/3797186419 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fca98063d00 msgr2=0x7fca980661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:02.213 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:02.210+0000 7fcabdd32700 1 --2- 172.21.15.195:0/3797186419 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fca98063d00 0x7fca980661b0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fcab80c2f90 tx=0x7fcaa8008040 comp rx=0 tx=0).stop 2022-06-27T12:04:02.213 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:02.210+0000 7fcabdd32700 1 -- 172.21.15.195:0/3797186419 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcab80c3d80 msgr2=0x7fcab8165140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:02.213 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:02.210+0000 7fcabdd32700 1 --2- 172.21.15.195:0/3797186419 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcab80c3d80 0x7fcab8165140 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fcaac009f80 tx=0x7fcaac017000 comp rx=0 tx=0).stop 2022-06-27T12:04:02.214 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:02.210+0000 7fcabdd32700 1 -- 172.21.15.195:0/3797186419 shutdown_connections 2022-06-27T12:04:02.214 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:02.210+0000 7fcabdd32700 1 --2- 172.21.15.195:0/3797186419 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fca98063d00 0x7fca980661b0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:02.215 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:02.210+0000 7fcabdd32700 1 --2- 172.21.15.195:0/3797186419 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcab80c3d80 0x7fcab8165140 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:02.215 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:02.210+0000 7fcabdd32700 1 --2- 172.21.15.195:0/3797186419 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcab814e860 0x7fcab814b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:02.216 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:02.210+0000 7fcabdd32700 1 --2- 172.21.15.195:0/3797186419 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcab814e490 0x7fcab8156b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:02.216 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:02.210+0000 7fcabdd32700 1 -- 172.21.15.195:0/3797186419 >> 172.21.15.195:0/3797186419 conn(0x7fcab8063af0 msgr2=0x7fcab80ca910 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:04:02.216 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:02.210+0000 7fcabdd32700 1 -- 172.21.15.195:0/3797186419 shutdown_connections 2022-06-27T12:04:02.217 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:02.210+0000 7fcabdd32700 1 -- 172.21.15.195:0/3797186419 wait complete. 2022-06-27T12:04:02.217 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 58 2022-06-27T12:04:02.223 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:04:04.224 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:04:04.416 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.415+0000 7fa6e6b19700 1 Processor -- start 2022-06-27T12:04:04.416 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.415+0000 7fa6e6b19700 1 -- start start 2022-06-27T12:04:04.419 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.416+0000 7fa6e6b19700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fa6e006a450 0x7fa6e0063af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:04.420 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.416+0000 7fa6e6b19700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa6e01508d0 0x7fa6e00bbf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:04.420 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.416+0000 7fa6e6b19700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6e0064030 con 0x7fa6e0150500 2022-06-27T12:04:04.421 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.416+0000 7fa6e6b19700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6e00c4160 con 0x7fa6e006a450 2022-06-27T12:04:04.421 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.416+0000 7fa6e6b19700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e00c4440 con 0x7fa6e01508d0 2022-06-27T12:04:04.422 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.417+0000 7fa6e50b6700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa6e01508d0 0x7fa6e00bbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:04.423 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.417+0000 7fa6dffff700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fa6e006a450 0x7fa6e0063af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:04.423 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.417+0000 7fa6e48b5700 1 --1- >> v1:172.21.15.202:6789/0 conn(0x7fa6e0150500 0x7fa6e00c48d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.202:6789/0 says I am v1:172.21.15.195:55486/0 (socket says 172.21.15.195:55486) 2022-06-27T12:04:04.424 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.417+0000 7fa6e48b5700 1 -- 172.21.15.195:0/1555505116 learned_addr learned my addr 172.21.15.195:0/1555505116 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:04:04.424 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.417+0000 7fa6dffff700 1 -- 172.21.15.195:0/1555505116 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa6e01508d0 msgr2=0x7fa6e00bbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:04:04.424 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.417+0000 7fa6dffff700 1 --2- 172.21.15.195:0/1555505116 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa6e01508d0 0x7fa6e00bbf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:04.425 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.417+0000 7fa6dffff700 1 -- 172.21.15.195:0/1555505116 >> v1:172.21.15.202:6789/0 conn(0x7fa6e0150500 legacy=0x7fa6e00c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:04.436 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.417+0000 7fa6df7fe700 1 -- 172.21.15.195:0/1555505116 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 10929359 0 0) 0x7fa6d4001160 con 0x7fa6e0150500 2022-06-27T12:04:04.436 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.417+0000 7fa6dffff700 1 -- 172.21.15.195:0/1555505116 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6e015a800 con 0x7fa6e006a450 2022-06-27T12:04:04.436 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.417+0000 7fa6dffff700 1 --2- 172.21.15.195:0/1555505116 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa6e006a450 0x7fa6e0063af0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fa6cc0097a0 tx=0x7fa6cc009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d41f690d63e2aa3 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:04:04.437 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.417+0000 7fa6df7fe700 1 -- 172.21.15.195:0/1555505116 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6cc011630 con 0x7fa6e006a450 2022-06-27T12:04:04.437 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.417+0000 7fa6df7fe700 1 -- 172.21.15.195:0/1555505116 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6cc011790 con 0x7fa6e006a450 2022-06-27T12:04:04.437 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.417+0000 7fa6df7fe700 1 -- 172.21.15.195:0/1555505116 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6cc011b30 con 0x7fa6e006a450 2022-06-27T12:04:04.437 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.418+0000 7fa6e6b19700 1 -- 172.21.15.195:0/1555505116 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa6e006a450 msgr2=0x7fa6e0063af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:04.438 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.418+0000 7fa6e6b19700 1 --2- 172.21.15.195:0/1555505116 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa6e006a450 0x7fa6e0063af0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fa6cc0097a0 tx=0x7fa6cc009060 comp rx=0 tx=0).stop 2022-06-27T12:04:04.438 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.418+0000 7fa6e6b19700 1 -- 172.21.15.195:0/1555505116 shutdown_connections 2022-06-27T12:04:04.439 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.418+0000 7fa6e6b19700 1 --2- 172.21.15.195:0/1555505116 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa6e01508d0 0x7fa6e00bbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:04.439 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.418+0000 7fa6e6b19700 1 --2- 172.21.15.195:0/1555505116 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa6e006a450 0x7fa6e0063af0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:04.439 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.418+0000 7fa6e6b19700 1 -- 172.21.15.195:0/1555505116 >> 172.21.15.195:0/1555505116 conn(0x7fa6e00c9b00 msgr2=0x7fa6e00c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:04:04.439 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.418+0000 7fa6e6b19700 1 -- 172.21.15.195:0/1555505116 shutdown_connections 2022-06-27T12:04:04.440 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.418+0000 7fa6e6b19700 1 -- 172.21.15.195:0/1555505116 wait complete. 2022-06-27T12:04:04.440 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.419+0000 7fa6e6b19700 1 Processor -- start 2022-06-27T12:04:04.440 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.419+0000 7fa6e6b19700 1 -- start start 2022-06-27T12:04:04.440 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.420+0000 7fa6e6b19700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa6e006a450 0x7fa6e00bd000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:04.441 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.420+0000 7fa6e6b19700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa6e0150500 0x7fa6e016a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:04.441 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.420+0000 7fa6e6b19700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa6e01508d0 0x7fa6e0166d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:04.441 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.420+0000 7fa6e6b19700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e016a130 con 0x7fa6e0150500 2022-06-27T12:04:04.441 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.420+0000 7fa6e6b19700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e0169e00 con 0x7fa6e01508d0 2022-06-27T12:04:04.441 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.420+0000 7fa6e6b19700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e01587e0 con 0x7fa6e006a450 2022-06-27T12:04:04.442 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.420+0000 7fa6dffff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa6e0150500 0x7fa6e016a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:04.443 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.420+0000 7fa6e48b5700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa6e006a450 0x7fa6e00bd000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:04.444 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.420+0000 7fa6e50b6700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa6e01508d0 0x7fa6e0166d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:04.445 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.420+0000 7fa6dffff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa6e0150500 0x7fa6e016a5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47448/0 (socket says 172.21.15.195:47448) 2022-06-27T12:04:04.445 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.420+0000 7fa6dffff700 1 -- 172.21.15.195:0/3855053796 learned_addr learned my addr 172.21.15.195:0/3855053796 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:04:04.446 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.420+0000 7fa6e50b6700 1 -- 172.21.15.195:0/3855053796 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa6e006a450 msgr2=0x7fa6e00bd000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:04.446 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.420+0000 7fa6e50b6700 1 --2- 172.21.15.195:0/3855053796 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa6e006a450 0x7fa6e00bd000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:04.447 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.420+0000 7fa6e50b6700 1 -- 172.21.15.195:0/3855053796 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa6e0150500 msgr2=0x7fa6e016a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:04.447 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.420+0000 7fa6e50b6700 1 --2- 172.21.15.195:0/3855053796 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa6e0150500 0x7fa6e016a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:04.448 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.420+0000 7fa6e50b6700 1 -- 172.21.15.195:0/3855053796 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6cc0092e0 con 0x7fa6e01508d0 2022-06-27T12:04:04.448 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.420+0000 7fa6dffff700 1 --2- 172.21.15.195:0/3855053796 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa6e0150500 0x7fa6e016a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:04:04.449 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.421+0000 7fa6e50b6700 1 --2- 172.21.15.195:0/3855053796 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa6e01508d0 0x7fa6e0166d70 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fa6d800ecc0 tx=0x7fa6d800b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:04:04.449 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.421+0000 7fa6dcff9700 1 -- 172.21.15.195:0/3855053796 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d800c8c0 con 0x7fa6e01508d0 2022-06-27T12:04:04.450 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.421+0000 7fa6dcff9700 1 -- 172.21.15.195:0/3855053796 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6d800f040 con 0x7fa6e01508d0 2022-06-27T12:04:04.450 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.421+0000 7fa6e6b19700 1 -- 172.21.15.195:0/3855053796 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6e00c5f20 con 0x7fa6e01508d0 2022-06-27T12:04:04.450 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.421+0000 7fa6dcff9700 1 -- 172.21.15.195:0/3855053796 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d800cc70 con 0x7fa6e01508d0 2022-06-27T12:04:04.450 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.421+0000 7fa6e6b19700 1 -- 172.21.15.195:0/3855053796 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6e0158d80 con 0x7fa6e01508d0 2022-06-27T12:04:04.451 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.422+0000 7fa6e6b19700 1 -- 172.21.15.195:0/3855053796 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6e016a2c0 con 0x7fa6e01508d0 2022-06-27T12:04:04.451 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.422+0000 7fa6dcff9700 1 -- 172.21.15.195:0/3855053796 <== mon.1 v2:172.21.15.202:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fa6d8014690 con 0x7fa6e01508d0 2022-06-27T12:04:04.451 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.423+0000 7fa6dcff9700 1 --2- 172.21.15.195:0/3855053796 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa6c8063d80 0x7fa6c8066230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:04.452 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.423+0000 7fa6dcff9700 1 -- 172.21.15.195:0/3855053796 <== mon.1 v2:172.21.15.202:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6890+0+0 (secure 0 0 0) 0x7fa6d8081df0 con 0x7fa6e01508d0 2022-06-27T12:04:04.452 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.423+0000 7fa6e48b5700 1 --2- 172.21.15.195:0/3855053796 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa6c8063d80 0x7fa6c8066230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:04.452 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.423+0000 7fa6e48b5700 1 --2- 172.21.15.195:0/3855053796 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa6c8063d80 0x7fa6c8066230 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fa6d4007600 tx=0x7fa6d4007a40 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:04:04.453 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.427+0000 7fa6dcff9700 1 -- 172.21.15.195:0/3855053796 <== mon.1 v2:172.21.15.202:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fa6d8050630 con 0x7fa6e01508d0 2022-06-27T12:04:04.655 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.653+0000 7fa6e6b19700 1 -- 172.21.15.195:0/3855053796 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6e00722c0 con 0x7fa6e01508d0 2022-06-27T12:04:04.656 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.654+0000 7fa6dcff9700 1 -- 172.21.15.195:0/3855053796 <== mon.1 v2:172.21.15.202:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+5374 (secure 0 0 0) 0x7fa6d8018020 con 0x7fa6e01508d0 2022-06-27T12:04:04.657 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:04:04.660 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:04:01.265250+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5818},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5818":{"gid":5818,"name":"b","rank":0,"incarnation":55,"state":"up:clientreplay","state_seq":31,"addr":"172.21.15.202:6833/3399027527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3399027527},{"type":"v1","addr":"172.21.15.202:6833","nonce":3399027527}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:04:04.662 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.656+0000 7fa6e6b19700 1 -- 172.21.15.195:0/3855053796 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa6c8063d80 msgr2=0x7fa6c8066230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:04.662 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.656+0000 7fa6e6b19700 1 --2- 172.21.15.195:0/3855053796 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa6c8063d80 0x7fa6c8066230 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fa6d4007600 tx=0x7fa6d4007a40 comp rx=0 tx=0).stop 2022-06-27T12:04:04.662 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.656+0000 7fa6e6b19700 1 -- 172.21.15.195:0/3855053796 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa6e01508d0 msgr2=0x7fa6e0166d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:04.663 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.656+0000 7fa6e6b19700 1 --2- 172.21.15.195:0/3855053796 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa6e01508d0 0x7fa6e0166d70 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fa6d800ecc0 tx=0x7fa6d800b040 comp rx=0 tx=0).stop 2022-06-27T12:04:04.663 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.656+0000 7fa6e6b19700 1 -- 172.21.15.195:0/3855053796 shutdown_connections 2022-06-27T12:04:04.664 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.656+0000 7fa6e6b19700 1 --2- 172.21.15.195:0/3855053796 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa6c8063d80 0x7fa6c8066230 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:04.664 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.656+0000 7fa6e6b19700 1 --2- 172.21.15.195:0/3855053796 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa6e01508d0 0x7fa6e0166d70 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:04.665 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.657+0000 7fa6e6b19700 1 --2- 172.21.15.195:0/3855053796 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa6e006a450 0x7fa6e00bd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:04.665 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.657+0000 7fa6e6b19700 1 --2- 172.21.15.195:0/3855053796 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa6e0150500 0x7fa6e016a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:04.666 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.657+0000 7fa6e6b19700 1 -- 172.21.15.195:0/3855053796 >> 172.21.15.195:0/3855053796 conn(0x7fa6e00c9b00 msgr2=0x7fa6e0067ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:04:04.666 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.657+0000 7fa6e6b19700 1 -- 172.21.15.195:0/3855053796 shutdown_connections 2022-06-27T12:04:04.666 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:04.657+0000 7fa6e6b19700 1 -- 172.21.15.195:0/3855053796 wait complete. 2022-06-27T12:04:04.666 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 58 2022-06-27T12:04:04.669 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:04:06.672 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:04:06.822 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:06.806+0000 7fbea9028700 1 Processor -- start 2022-06-27T12:04:06.822 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:06.806+0000 7fbea9028700 1 -- start start 2022-06-27T12:04:06.823 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:06.807+0000 7fbea9028700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbea4154900 0x7fbea40c2120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:06.823 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:06.807+0000 7fbea9028700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fbea4151930 0x7fbea40bf4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:06.824 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:06.807+0000 7fbea9028700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbea4154da0 con 0x7fbea4151560 2022-06-27T12:04:06.824 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:06.807+0000 7fbea9028700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbea40c12a0 con 0x7fbea4151930 2022-06-27T12:04:06.824 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:06.807+0000 7fbea9028700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbea40c1580 con 0x7fbea4154900 2022-06-27T12:04:06.825 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:06.807+0000 7fbea259c700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbea4154900 0x7fbea40c2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:06.825 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:06.807+0000 7fbea359e700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fbea4151930 0x7fbea40bf4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:06.826 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:06.807+0000 7fbea2d9d700 1 --1- >> v1:172.21.15.202:6789/0 conn(0x7fbea4151560 0x7fbea41541f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.202:6789/0 says I am v1:172.21.15.195:55502/0 (socket says 172.21.15.195:55502) 2022-06-27T12:04:06.826 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:06.807+0000 7fbea2d9d700 1 -- 172.21.15.195:0/1674032397 learned_addr learned my addr 172.21.15.195:0/1674032397 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:04:06.826 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:06.808+0000 7fbea359e700 1 -- 172.21.15.195:0/1674032397 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbea4154900 msgr2=0x7fbea40c2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:04:06.827 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:06.808+0000 7fbea359e700 1 --2- 172.21.15.195:0/1674032397 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbea4154900 0x7fbea40c2120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:06.827 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:06.808+0000 7fbea359e700 1 -- 172.21.15.195:0/1674032397 >> v1:172.21.15.202:6789/0 conn(0x7fbea4151560 legacy=0x7fbea41541f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:06.828 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:06.808+0000 7fbea359e700 1 -- 172.21.15.195:0/1674032397 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbea40c29f0 con 0x7fbea4151930 2022-06-27T12:04:06.828 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:06.808+0000 7fbea1d9b700 1 -- 172.21.15.195:0/1674032397 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3673802968 0 0) 0x7fbea4154da0 con 0x7fbea4151560 2022-06-27T12:04:06.828 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:06.808+0000 7fbea259c700 1 --2- 172.21.15.195:0/1674032397 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbea4154900 0x7fbea40c2120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:04:06.829 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:06.808+0000 7fbea359e700 1 --2- 172.21.15.195:0/1674032397 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbea4151930 0x7fbea40bf4a0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fbe9c009e60 tx=0x7fbe9c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4897b9a8a009089 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:04:06.829 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:06.808+0000 7fbea1d9b700 1 -- 172.21.15.195:0/1674032397 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe9c00e070 con 0x7fbea4151930 2022-06-27T12:04:06.829 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:06.808+0000 7fbea1d9b700 1 -- 172.21.15.195:0/1674032397 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe9c00bbf0 con 0x7fbea4151930 2022-06-27T12:04:06.830 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:06.808+0000 7fbea1d9b700 1 -- 172.21.15.195:0/1674032397 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe9c012430 con 0x7fbea4151930 2022-06-27T12:04:06.830 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:06.808+0000 7fbea9028700 1 -- 172.21.15.195:0/1674032397 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbea4151930 msgr2=0x7fbea40bf4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:06.830 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:06.808+0000 7fbea9028700 1 --2- 172.21.15.195:0/1674032397 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbea4151930 0x7fbea40bf4a0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fbe9c009e60 tx=0x7fbe9c009780 comp rx=0 tx=0).stop 2022-06-27T12:04:06.831 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:06.809+0000 7fbea9028700 1 -- 172.21.15.195:0/1674032397 shutdown_connections 2022-06-27T12:04:06.831 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:06.809+0000 7fbea9028700 1 --2- 172.21.15.195:0/1674032397 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbea4151930 0x7fbea40bf4a0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:06.832 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:06.809+0000 7fbea9028700 1 --2- 172.21.15.195:0/1674032397 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbea4154900 0x7fbea40c2120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:06.832 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:06.809+0000 7fbea9028700 1 -- 172.21.15.195:0/1674032397 >> 172.21.15.195:0/1674032397 conn(0x7fbea40c6350 msgr2=0x7fbea40c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:04:06.832 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:06.809+0000 7fbea9028700 1 -- 172.21.15.195:0/1674032397 shutdown_connections 2022-06-27T12:04:06.833 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:06.809+0000 7fbea9028700 1 -- 172.21.15.195:0/1674032397 wait complete. 2022-06-27T12:04:06.833 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:06.810+0000 7fbea9028700 1 Processor -- start 2022-06-27T12:04:06.833 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:06.810+0000 7fbea9028700 1 -- start start 2022-06-27T12:04:06.834 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:06.811+0000 7fbea9028700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbea4151560 0x7fbea414d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:06.834 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:06.811+0000 7fbea9028700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbea4151930 0x7fbea41502e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:06.835 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:06.811+0000 7fbea9028700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbea4154900 0x7fbea4150820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:06.835 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:06.811+0000 7fbea9028700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbea40c9670 con 0x7fbea4151930 2022-06-27T12:04:06.835 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:06.811+0000 7fbea9028700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbea406a3b0 con 0x7fbea4154900 2022-06-27T12:04:06.836 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:06.811+0000 7fbea9028700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbea4158500 con 0x7fbea4151560 2022-06-27T12:04:06.836 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:06.811+0000 7fbea259c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbea4151930 0x7fbea41502e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:06.836 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:06.811+0000 7fbea2d9d700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbea4151560 0x7fbea414d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:06.837 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:06.811+0000 7fbea259c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbea4151930 0x7fbea41502e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47462/0 (socket says 172.21.15.195:47462) 2022-06-27T12:04:06.837 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:06.811+0000 7fbea259c700 1 -- 172.21.15.195:0/882339882 learned_addr learned my addr 172.21.15.195:0/882339882 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:04:06.838 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:06.811+0000 7fbea359e700 1 --2- 172.21.15.195:0/882339882 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbea4154900 0x7fbea4150820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:06.838 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:06.811+0000 7fbea259c700 1 -- 172.21.15.195:0/882339882 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbea4151560 msgr2=0x7fbea414d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:06.838 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:06.811+0000 7fbea259c700 1 --2- 172.21.15.195:0/882339882 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbea4151560 0x7fbea414d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:06.839 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:06.811+0000 7fbea259c700 1 -- 172.21.15.195:0/882339882 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbea4154900 msgr2=0x7fbea4150820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:06.839 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:06.811+0000 7fbea259c700 1 --2- 172.21.15.195:0/882339882 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbea4154900 0x7fbea4150820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:06.840 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:06.811+0000 7fbea259c700 1 -- 172.21.15.195:0/882339882 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe940092e0 con 0x7fbea4151930 2022-06-27T12:04:06.840 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:06.811+0000 7fbea259c700 1 --2- 172.21.15.195:0/882339882 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbea4151930 0x7fbea41502e0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fbe9400ea00 tx=0x7fbe94009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:04:06.840 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:06.811+0000 7fbe8affd700 1 -- 172.21.15.195:0/882339882 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe9400c6a0 con 0x7fbea4151930 2022-06-27T12:04:06.841 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:06.811+0000 7fbe8affd700 1 -- 172.21.15.195:0/882339882 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe9400c800 con 0x7fbea4151930 2022-06-27T12:04:06.841 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:06.811+0000 7fbea9028700 1 -- 172.21.15.195:0/882339882 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe9c009a20 con 0x7fbea4151930 2022-06-27T12:04:06.841 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:06.811+0000 7fbe8affd700 1 -- 172.21.15.195:0/882339882 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe9400cba0 con 0x7fbea4151930 2022-06-27T12:04:06.842 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:06.811+0000 7fbea9028700 1 -- 172.21.15.195:0/882339882 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbea406a950 con 0x7fbea4151930 2022-06-27T12:04:06.843 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:06.812+0000 7fbea9028700 1 -- 172.21.15.195:0/882339882 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbea40722c0 con 0x7fbea4151930 2022-06-27T12:04:06.843 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:06.813+0000 7fbe8affd700 1 -- 172.21.15.195:0/882339882 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fbe94019030 con 0x7fbea4151930 2022-06-27T12:04:06.843 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:06.813+0000 7fbe8affd700 1 --2- 172.21.15.195:0/882339882 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbe84063d00 0x7fbe840661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:06.843 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:06.813+0000 7fbe8affd700 1 -- 172.21.15.195:0/882339882 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6890+0+0 (secure 0 0 0) 0x7fbe940808a0 con 0x7fbea4151930 2022-06-27T12:04:06.844 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:06.813+0000 7fbea2d9d700 1 --2- 172.21.15.195:0/882339882 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbe84063d00 0x7fbe840661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:06.844 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:06.814+0000 7fbea2d9d700 1 --2- 172.21.15.195:0/882339882 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbe84063d00 0x7fbe840661b0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fbe98007fd0 tx=0x7fbe98007d10 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:04:06.845 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:06.817+0000 7fbe8affd700 1 -- 172.21.15.195:0/882339882 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fbe9404f150 con 0x7fbea4151930 2022-06-27T12:04:07.075 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:07.073+0000 7fbea9028700 1 -- 172.21.15.195:0/882339882 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbea406b090 con 0x7fbea4151930 2022-06-27T12:04:07.075 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:07.073+0000 7fbe8affd700 1 -- 172.21.15.195:0/882339882 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+5374 (secure 0 0 0) 0x7fbe94052770 con 0x7fbea4151930 2022-06-27T12:04:07.076 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:04:07.076 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:04:01.265250+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5818},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5818":{"gid":5818,"name":"b","rank":0,"incarnation":55,"state":"up:clientreplay","state_seq":31,"addr":"172.21.15.202:6833/3399027527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3399027527},{"type":"v1","addr":"172.21.15.202:6833","nonce":3399027527}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:04:07.079 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:07.075+0000 7fbea9028700 1 -- 172.21.15.195:0/882339882 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbe84063d00 msgr2=0x7fbe840661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:07.079 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:07.075+0000 7fbea9028700 1 --2- 172.21.15.195:0/882339882 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbe84063d00 0x7fbe840661b0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fbe98007fd0 tx=0x7fbe98007d10 comp rx=0 tx=0).stop 2022-06-27T12:04:07.080 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:07.075+0000 7fbea9028700 1 -- 172.21.15.195:0/882339882 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbea4151930 msgr2=0x7fbea41502e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:07.080 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:07.075+0000 7fbea9028700 1 --2- 172.21.15.195:0/882339882 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbea4151930 0x7fbea41502e0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fbe9400ea00 tx=0x7fbe94009060 comp rx=0 tx=0).stop 2022-06-27T12:04:07.080 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:07.075+0000 7fbea9028700 1 -- 172.21.15.195:0/882339882 shutdown_connections 2022-06-27T12:04:07.081 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:07.075+0000 7fbea9028700 1 --2- 172.21.15.195:0/882339882 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbe84063d00 0x7fbe840661b0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:07.081 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:07.075+0000 7fbea9028700 1 --2- 172.21.15.195:0/882339882 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbea4154900 0x7fbea4150820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:07.082 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:07.075+0000 7fbea9028700 1 --2- 172.21.15.195:0/882339882 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbea4151560 0x7fbea414d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:07.082 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:07.075+0000 7fbea9028700 1 --2- 172.21.15.195:0/882339882 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbea4151930 0x7fbea41502e0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:07.082 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:07.075+0000 7fbea9028700 1 -- 172.21.15.195:0/882339882 >> 172.21.15.195:0/882339882 conn(0x7fbea40c6350 msgr2=0x7fbea40c6720 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:04:07.082 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:07.075+0000 7fbea9028700 1 -- 172.21.15.195:0/882339882 shutdown_connections 2022-06-27T12:04:07.083 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:07.075+0000 7fbea9028700 1 -- 172.21.15.195:0/882339882 wait complete. 2022-06-27T12:04:07.083 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 58 2022-06-27T12:04:07.088 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:04:09.090 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:04:09.231 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.216+0000 7fa94b2a5700 1 Processor -- start 2022-06-27T12:04:09.232 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.216+0000 7fa94b2a5700 1 -- start start 2022-06-27T12:04:09.232 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.217+0000 7fa94b2a5700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fa94406f640 0x7fa944156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:09.232 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.217+0000 7fa94b2a5700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa9440c4160 0x7fa9440c1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:09.233 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.217+0000 7fa94b2a5700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9440c1a20 con 0x7fa9440c39e0 2022-06-27T12:04:09.233 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.217+0000 7fa94b2a5700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9440c1130 con 0x7fa94406f640 2022-06-27T12:04:09.234 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.217+0000 7fa94b2a5700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa94406ea90 con 0x7fa9440c4160 2022-06-27T12:04:09.234 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.217+0000 7fa949842700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fa9440c39e0 0x7fa9441535a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:53668/0 (socket says 172.21.15.195:53668) 2022-06-27T12:04:09.234 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.217+0000 7fa949842700 1 -- 172.21.15.195:0/2976726113 learned_addr learned my addr 172.21.15.195:0/2976726113 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:04:09.235 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.218+0000 7fa948840700 1 --2- 172.21.15.195:0/2976726113 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa9440c4160 0x7fa9440c1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:09.235 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.218+0000 7fa949041700 1 --2- 172.21.15.195:0/2976726113 >> v2:172.21.15.202:3300/0 conn(0x7fa94406f640 0x7fa944156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:09.236 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.218+0000 7fa933fff700 1 -- 172.21.15.195:0/2976726113 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2088210239 0 0) 0x7fa9440c1a20 con 0x7fa9440c39e0 2022-06-27T12:04:09.236 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.218+0000 7fa933fff700 1 -- 172.21.15.195:0/2976726113 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa92c003430 con 0x7fa9440c39e0 2022-06-27T12:04:09.238 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.218+0000 7fa933fff700 1 -- 172.21.15.195:0/2976726113 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3024206250 0 0) 0x7fa92c003430 con 0x7fa9440c39e0 2022-06-27T12:04:09.238 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.218+0000 7fa933fff700 1 -- 172.21.15.195:0/2976726113 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa9440c4160 msgr2=0x7fa9440c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:04:09.238 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.218+0000 7fa933fff700 1 --2- 172.21.15.195:0/2976726113 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa9440c4160 0x7fa9440c1eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:09.239 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.218+0000 7fa933fff700 1 -- 172.21.15.195:0/2976726113 >> v2:172.21.15.202:3300/0 conn(0x7fa94406f640 msgr2=0x7fa944156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:04:09.239 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.218+0000 7fa933fff700 1 --2- 172.21.15.195:0/2976726113 >> v2:172.21.15.202:3300/0 conn(0x7fa94406f640 0x7fa944156ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:09.239 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.218+0000 7fa933fff700 1 -- 172.21.15.195:0/2976726113 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa94406ed70 con 0x7fa9440c39e0 2022-06-27T12:04:09.242 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.218+0000 7fa933fff700 1 -- 172.21.15.195:0/2976726113 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7fa93c0017f0 con 0x7fa9440c39e0 2022-06-27T12:04:09.242 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.218+0000 7fa933fff700 1 -- 172.21.15.195:0/2976726113 >> v1:172.21.15.195:6789/0 conn(0x7fa9440c39e0 legacy=0x7fa9441535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:09.242 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.218+0000 7fa933fff700 1 --2- 172.21.15.195:0/2976726113 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa92c003920 0x7fa92c003d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:09.244 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.218+0000 7fa933fff700 1 --2- 172.21.15.195:0/2976726113 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa9440c4160 0x7fa92c0046d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:09.245 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.218+0000 7fa933fff700 1 --2- 172.21.15.195:0/2976726113 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa92c004c10 0x7fa92c005080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:09.245 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.218+0000 7fa933fff700 1 -- 172.21.15.195:0/2976726113 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9440c1130 con 0x7fa92c003920 2022-06-27T12:04:09.245 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.218+0000 7fa933fff700 1 -- 172.21.15.195:0/2976726113 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa94406ea90 con 0x7fa92c004c10 2022-06-27T12:04:09.245 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.218+0000 7fa933fff700 1 -- 172.21.15.195:0/2976726113 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa92c00b920 con 0x7fa9440c4160 2022-06-27T12:04:09.246 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.218+0000 7fa949041700 1 --2- 172.21.15.195:0/2976726113 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa92c003920 0x7fa92c003d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:09.246 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.218+0000 7fa948840700 1 --2- 172.21.15.195:0/2976726113 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa9440c4160 0x7fa92c0046d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:09.247 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.218+0000 7fa949842700 1 --2- 172.21.15.195:0/2976726113 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa92c004c10 0x7fa92c005080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:09.247 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.218+0000 7fa949041700 1 -- 172.21.15.195:0/2976726113 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa9440c4160 msgr2=0x7fa92c0046d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:04:09.247 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.218+0000 7fa949041700 1 --2- 172.21.15.195:0/2976726113 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa9440c4160 0x7fa92c0046d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:09.247 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.218+0000 7fa949041700 1 -- 172.21.15.195:0/2976726113 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa92c004c10 msgr2=0x7fa92c005080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:04:09.248 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.218+0000 7fa949041700 1 --2- 172.21.15.195:0/2976726113 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa92c004c10 0x7fa92c005080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:09.248 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.218+0000 7fa949041700 1 -- 172.21.15.195:0/2976726113 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa92c003430 con 0x7fa92c003920 2022-06-27T12:04:09.248 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.219+0000 7fa949041700 1 --2- 172.21.15.195:0/2976726113 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa92c003920 0x7fa92c003d30 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fa934009ac0 tx=0x7fa934009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e33901af5a5ac4cb server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:04:09.248 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.219+0000 7fa933fff700 1 -- 172.21.15.195:0/2976726113 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa934009470 con 0x7fa92c003920 2022-06-27T12:04:09.249 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.219+0000 7fa933fff700 1 -- 172.21.15.195:0/2976726113 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa93400e5e0 con 0x7fa92c003920 2022-06-27T12:04:09.249 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.219+0000 7fa933fff700 1 -- 172.21.15.195:0/2976726113 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa93400e980 con 0x7fa92c003920 2022-06-27T12:04:09.249 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.219+0000 7fa94b2a5700 1 -- 172.21.15.195:0/2976726113 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa92c003920 msgr2=0x7fa92c003d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:09.250 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.219+0000 7fa94b2a5700 1 --2- 172.21.15.195:0/2976726113 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa92c003920 0x7fa92c003d30 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fa934009ac0 tx=0x7fa934009060 comp rx=0 tx=0).stop 2022-06-27T12:04:09.250 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.219+0000 7fa949842700 1 -- 172.21.15.195:0/2976726113 reap_dead start 2022-06-27T12:04:09.251 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.219+0000 7fa94b2a5700 1 -- 172.21.15.195:0/2976726113 shutdown_connections 2022-06-27T12:04:09.251 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.219+0000 7fa94b2a5700 1 -- 172.21.15.195:0/2976726113 >> 172.21.15.195:0/2976726113 conn(0x7fa9440c6350 msgr2=0x7fa9440c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:04:09.251 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.219+0000 7fa94b2a5700 1 -- 172.21.15.195:0/2976726113 shutdown_connections 2022-06-27T12:04:09.252 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.219+0000 7fa94b2a5700 1 -- 172.21.15.195:0/2976726113 wait complete. 2022-06-27T12:04:09.252 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.220+0000 7fa94b2a5700 1 Processor -- start 2022-06-27T12:04:09.252 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.220+0000 7fa94b2a5700 1 -- start start 2022-06-27T12:04:09.252 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.221+0000 7fa94b2a5700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa92c003920 0x7fa944168d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:09.253 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.221+0000 7fa94b2a5700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa9440bcb30 0x7fa9440c7990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:09.253 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.221+0000 7fa94b2a5700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa944169a80 0x7fa944169ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:09.253 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.221+0000 7fa94b2a5700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa944169240 con 0x7fa9440bcb30 2022-06-27T12:04:09.255 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.221+0000 7fa94b2a5700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa94406aff0 con 0x7fa944169a80 2022-06-27T12:04:09.255 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.221+0000 7fa94b2a5700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9441695f0 con 0x7fa92c003920 2022-06-27T12:04:09.255 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.221+0000 7fa948840700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa9440bcb30 0x7fa9440c7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:09.256 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.221+0000 7fa948840700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa9440bcb30 0x7fa9440c7990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47490/0 (socket says 172.21.15.195:47490) 2022-06-27T12:04:09.256 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.221+0000 7fa948840700 1 -- 172.21.15.195:0/4069986296 learned_addr learned my addr 172.21.15.195:0/4069986296 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:04:09.257 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.221+0000 7fa949041700 1 --2- 172.21.15.195:0/4069986296 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa92c003920 0x7fa944168d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:09.257 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.221+0000 7fa949842700 1 --2- 172.21.15.195:0/4069986296 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa944169a80 0x7fa944169ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:09.258 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.222+0000 7fa948840700 1 -- 172.21.15.195:0/4069986296 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa92c003920 msgr2=0x7fa944168d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:09.258 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.222+0000 7fa948840700 1 --2- 172.21.15.195:0/4069986296 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa92c003920 0x7fa944168d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:09.258 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.222+0000 7fa948840700 1 -- 172.21.15.195:0/4069986296 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa944169a80 msgr2=0x7fa944169ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:09.260 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.222+0000 7fa948840700 1 --2- 172.21.15.195:0/4069986296 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa944169a80 0x7fa944169ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:09.260 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.222+0000 7fa948840700 1 -- 172.21.15.195:0/4069986296 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa94406a7c0 con 0x7fa9440bcb30 2022-06-27T12:04:09.260 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.222+0000 7fa948840700 1 --2- 172.21.15.195:0/4069986296 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa9440bcb30 0x7fa9440c7990 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fa940011aa0 tx=0x7fa940009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:04:09.261 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.222+0000 7fa9317fa700 1 -- 172.21.15.195:0/4069986296 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa940011e80 con 0x7fa9440bcb30 2022-06-27T12:04:09.261 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.222+0000 7fa9317fa700 1 -- 172.21.15.195:0/4069986296 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa940004240 con 0x7fa9440bcb30 2022-06-27T12:04:09.262 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.222+0000 7fa94b2a5700 1 -- 172.21.15.195:0/4069986296 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9440c8cf0 con 0x7fa9440bcb30 2022-06-27T12:04:09.263 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.222+0000 7fa9317fa700 1 -- 172.21.15.195:0/4069986296 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa94000f920 con 0x7fa9440bcb30 2022-06-27T12:04:09.263 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.222+0000 7fa94b2a5700 1 -- 172.21.15.195:0/4069986296 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa94406aa30 con 0x7fa9440bcb30 2022-06-27T12:04:09.264 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.223+0000 7fa94b2a5700 1 -- 172.21.15.195:0/4069986296 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa944169780 con 0x7fa9440bcb30 2022-06-27T12:04:09.264 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.223+0000 7fa9317fa700 1 -- 172.21.15.195:0/4069986296 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fa940024070 con 0x7fa9440bcb30 2022-06-27T12:04:09.265 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.224+0000 7fa9317fa700 1 --2- 172.21.15.195:0/4069986296 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa924063d00 0x7fa9240661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:09.265 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.224+0000 7fa9317fa700 1 -- 172.21.15.195:0/4069986296 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6890+0+0 (secure 0 0 0) 0x7fa940018030 con 0x7fa9440bcb30 2022-06-27T12:04:09.266 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.224+0000 7fa949041700 1 --2- 172.21.15.195:0/4069986296 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa924063d00 0x7fa9240661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:09.266 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.224+0000 7fa949041700 1 --2- 172.21.15.195:0/4069986296 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa924063d00 0x7fa9240661b0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fa93400e5b0 tx=0x7fa93401a000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:04:09.267 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.228+0000 7fa9317fa700 1 -- 172.21.15.195:0/4069986296 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fa94004fa80 con 0x7fa9440bcb30 2022-06-27T12:04:09.480 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.464+0000 7fa94b2a5700 1 -- 172.21.15.195:0/4069986296 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa9440722c0 con 0x7fa9440bcb30 2022-06-27T12:04:09.480 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.465+0000 7fa9317fa700 1 -- 172.21.15.195:0/4069986296 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+5374 (secure 0 0 0) 0x7fa940022020 con 0x7fa9440bcb30 2022-06-27T12:04:09.481 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:04:09.481 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:04:01.265250+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5818},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5818":{"gid":5818,"name":"b","rank":0,"incarnation":55,"state":"up:clientreplay","state_seq":31,"addr":"172.21.15.202:6833/3399027527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3399027527},{"type":"v1","addr":"172.21.15.202:6833","nonce":3399027527}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:04:09.483 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.467+0000 7fa94b2a5700 1 -- 172.21.15.195:0/4069986296 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa924063d00 msgr2=0x7fa9240661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:09.483 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.467+0000 7fa94b2a5700 1 --2- 172.21.15.195:0/4069986296 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa924063d00 0x7fa9240661b0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fa93400e5b0 tx=0x7fa93401a000 comp rx=0 tx=0).stop 2022-06-27T12:04:09.484 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.467+0000 7fa94b2a5700 1 -- 172.21.15.195:0/4069986296 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa9440bcb30 msgr2=0x7fa9440c7990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:09.484 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.467+0000 7fa94b2a5700 1 --2- 172.21.15.195:0/4069986296 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa9440bcb30 0x7fa9440c7990 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fa940011aa0 tx=0x7fa940009060 comp rx=0 tx=0).stop 2022-06-27T12:04:09.484 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.467+0000 7fa94b2a5700 1 -- 172.21.15.195:0/4069986296 shutdown_connections 2022-06-27T12:04:09.485 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.467+0000 7fa94b2a5700 1 --2- 172.21.15.195:0/4069986296 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa924063d00 0x7fa9240661b0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:09.485 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.467+0000 7fa94b2a5700 1 --2- 172.21.15.195:0/4069986296 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa944169a80 0x7fa944169ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:09.485 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.467+0000 7fa94b2a5700 1 --2- 172.21.15.195:0/4069986296 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa92c003920 0x7fa944168d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:09.486 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.467+0000 7fa94b2a5700 1 --2- 172.21.15.195:0/4069986296 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa9440bcb30 0x7fa9440c7990 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:09.486 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.467+0000 7fa94b2a5700 1 -- 172.21.15.195:0/4069986296 >> 172.21.15.195:0/4069986296 conn(0x7fa9440c6350 msgr2=0x7fa9441504b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:04:09.487 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.467+0000 7fa94b2a5700 1 -- 172.21.15.195:0/4069986296 shutdown_connections 2022-06-27T12:04:09.487 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:09.467+0000 7fa94b2a5700 1 -- 172.21.15.195:0/4069986296 wait complete. 2022-06-27T12:04:09.488 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 58 2022-06-27T12:04:09.488 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:04:11.490 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:04:11.623 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.622+0000 7f79c4838700 1 Processor -- start 2022-06-27T12:04:11.623 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.622+0000 7f79c4838700 1 -- start start 2022-06-27T12:04:11.671 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.622+0000 7f79c4838700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f79c0150580 0x7f79c0156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:11.671 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.623+0000 7f79c4838700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f79c006f620 0x7f79c006d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:11.672 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.623+0000 7f79c4838700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79c00c76b0 con 0x7f79c0150950 2022-06-27T12:04:11.672 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.623+0000 7f79c4838700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79c0157430 con 0x7f79c0150580 2022-06-27T12:04:11.673 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.623+0000 7f79c4838700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79c00c8250 con 0x7f79c006f620 2022-06-27T12:04:11.673 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.623+0000 7f79be59c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f79c0150580 0x7f79c0156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:11.673 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.623+0000 7f79be59c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f79c0150580 0x7f79c0156ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47500/0 (socket says 172.21.15.195:47500) 2022-06-27T12:04:11.694 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.623+0000 7f79be59c700 1 -- 172.21.15.195:0/3018125975 learned_addr learned my addr 172.21.15.195:0/3018125975 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:04:11.695 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.623+0000 7f79bed9d700 1 --2- 172.21.15.195:0/3018125975 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f79c006f620 0x7f79c006d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:11.695 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.623+0000 7f79be59c700 1 -- 172.21.15.195:0/3018125975 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f79c006f620 msgr2=0x7f79c006d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:04:11.695 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.623+0000 7f79be59c700 1 --2- 172.21.15.195:0/3018125975 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f79c006f620 0x7f79c006d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:11.696 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.623+0000 7f79be59c700 1 -- 172.21.15.195:0/3018125975 >> v1:172.21.15.202:6789/0 conn(0x7f79c0150950 legacy=0x7f79c00c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:11.696 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.623+0000 7f79be59c700 1 -- 172.21.15.195:0/3018125975 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79c00bad70 con 0x7f79c0150580 2022-06-27T12:04:11.697 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.623+0000 7f79be59c700 1 --2- 172.21.15.195:0/3018125975 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f79c0150580 0x7f79c0156ef0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f79b4009860 tx=0x7f79b4009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b54669776e68229 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:04:11.697 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.623+0000 7f79bd59a700 1 -- 172.21.15.195:0/3018125975 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79b400b680 con 0x7f79c0150580 2022-06-27T12:04:11.699 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.623+0000 7f79bd59a700 1 -- 172.21.15.195:0/3018125975 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79b400b7e0 con 0x7f79c0150580 2022-06-27T12:04:11.699 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.623+0000 7f79bd59a700 1 -- 172.21.15.195:0/3018125975 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79b400bba0 con 0x7f79c0150580 2022-06-27T12:04:11.700 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.624+0000 7f79c4838700 1 -- 172.21.15.195:0/3018125975 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f79c0150580 msgr2=0x7f79c0156ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:11.700 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.624+0000 7f79c4838700 1 --2- 172.21.15.195:0/3018125975 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f79c0150580 0x7f79c0156ef0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f79b4009860 tx=0x7f79b4009120 comp rx=0 tx=0).stop 2022-06-27T12:04:11.700 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.624+0000 7f79c4838700 1 -- 172.21.15.195:0/3018125975 shutdown_connections 2022-06-27T12:04:11.701 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.624+0000 7f79c4838700 1 --2- 172.21.15.195:0/3018125975 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f79c006f620 0x7f79c006d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:11.701 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.624+0000 7f79c4838700 1 --2- 172.21.15.195:0/3018125975 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f79c0150580 0x7f79c0156ef0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:11.702 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.624+0000 7f79c4838700 1 -- 172.21.15.195:0/3018125975 >> 172.21.15.195:0/3018125975 conn(0x7f79c0063af0 msgr2=0x7f79c0063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:04:11.702 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.624+0000 7f79c4838700 1 -- 172.21.15.195:0/3018125975 shutdown_connections 2022-06-27T12:04:11.703 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.624+0000 7f79c4838700 1 -- 172.21.15.195:0/3018125975 wait complete. 2022-06-27T12:04:11.703 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.625+0000 7f79c4838700 1 Processor -- start 2022-06-27T12:04:11.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.625+0000 7f79c4838700 1 -- start start 2022-06-27T12:04:11.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.626+0000 7f79c4838700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f79c006f620 0x7f79c00c9d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:11.707 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.626+0000 7f79c4838700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f79c0150580 0x7f79c014a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:11.707 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.626+0000 7f79c4838700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f79c0150950 0x7f79c014a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:11.708 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.626+0000 7f79c4838700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79c00c4d80 con 0x7f79c006f620 2022-06-27T12:04:11.710 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.626+0000 7f79c4838700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79c00c4140 con 0x7f79c0150580 2022-06-27T12:04:11.711 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.626+0000 7f79c4838700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79c014adc0 con 0x7f79c0150950 2022-06-27T12:04:11.711 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.626+0000 7f79be59c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f79c006f620 0x7f79c00c9d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:11.711 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.626+0000 7f79be59c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f79c006f620 0x7f79c00c9d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47506/0 (socket says 172.21.15.195:47506) 2022-06-27T12:04:11.711 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.626+0000 7f79be59c700 1 -- 172.21.15.195:0/2596351932 learned_addr learned my addr 172.21.15.195:0/2596351932 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:04:11.711 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.626+0000 7f79bed9d700 1 --2- 172.21.15.195:0/2596351932 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f79c0150950 0x7f79c014a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:11.712 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.626+0000 7f79bdd9b700 1 --2- 172.21.15.195:0/2596351932 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f79c0150580 0x7f79c014a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:11.713 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.626+0000 7f79be59c700 1 -- 172.21.15.195:0/2596351932 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f79c0150950 msgr2=0x7f79c014a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:11.713 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.626+0000 7f79be59c700 1 --2- 172.21.15.195:0/2596351932 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f79c0150950 0x7f79c014a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:11.713 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.626+0000 7f79be59c700 1 -- 172.21.15.195:0/2596351932 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f79c0150580 msgr2=0x7f79c014a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:11.714 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.626+0000 7f79be59c700 1 --2- 172.21.15.195:0/2596351932 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f79c0150580 0x7f79c014a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:11.714 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.626+0000 7f79be59c700 1 -- 172.21.15.195:0/2596351932 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79b40093a0 con 0x7f79c006f620 2022-06-27T12:04:11.714 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.626+0000 7f79be59c700 1 --2- 172.21.15.195:0/2596351932 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f79c006f620 0x7f79c00c9d80 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f79c0065b00 tx=0x7f79b4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:04:11.714 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.627+0000 7f79ae7fc700 1 -- 172.21.15.195:0/2596351932 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79b400b950 con 0x7f79c006f620 2022-06-27T12:04:11.714 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.627+0000 7f79ae7fc700 1 -- 172.21.15.195:0/2596351932 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79b400bab0 con 0x7f79c006f620 2022-06-27T12:04:11.715 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.627+0000 7f79c4838700 1 -- 172.21.15.195:0/2596351932 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79c006a510 con 0x7f79c006f620 2022-06-27T12:04:11.715 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.627+0000 7f79ae7fc700 1 -- 172.21.15.195:0/2596351932 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79b401a680 con 0x7f79c006f620 2022-06-27T12:04:11.715 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.627+0000 7f79c4838700 1 -- 172.21.15.195:0/2596351932 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79c00c3810 con 0x7f79c006f620 2022-06-27T12:04:11.716 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.628+0000 7f79c4838700 1 -- 172.21.15.195:0/2596351932 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79c00722c0 con 0x7f79c006f620 2022-06-27T12:04:11.718 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.628+0000 7f79ae7fc700 1 -- 172.21.15.195:0/2596351932 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f79b400f050 con 0x7f79c006f620 2022-06-27T12:04:11.719 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.628+0000 7f79ae7fc700 1 --2- 172.21.15.195:0/2596351932 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f799c063d30 0x7f799c0661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:11.720 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.628+0000 7f79ae7fc700 1 -- 172.21.15.195:0/2596351932 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6890+0+0 (secure 0 0 0) 0x7f79b4082670 con 0x7f79c006f620 2022-06-27T12:04:11.720 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.628+0000 7f79bdd9b700 1 --2- 172.21.15.195:0/2596351932 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f799c063d30 0x7f799c0661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:11.720 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.629+0000 7f79bdd9b700 1 --2- 172.21.15.195:0/2596351932 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f799c063d30 0x7f799c0661e0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f79c00c7180 tx=0x7f79b0009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:04:11.720 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.633+0000 7f79ae7fc700 1 -- 172.21.15.195:0/2596351932 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f79b4050f20 con 0x7f79c006f620 2022-06-27T12:04:11.859 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.858+0000 7f79c4838700 1 -- 172.21.15.195:0/2596351932 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f79c006fe20 con 0x7f79c006f620 2022-06-27T12:04:11.860 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.859+0000 7f79ae7fc700 1 -- 172.21.15.195:0/2596351932 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+5374 (secure 0 0 0) 0x7f79b4015020 con 0x7f79c006f620 2022-06-27T12:04:11.860 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:04:11.860 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:04:01.265250+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5818},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5818":{"gid":5818,"name":"b","rank":0,"incarnation":55,"state":"up:clientreplay","state_seq":31,"addr":"172.21.15.202:6833/3399027527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3399027527},{"type":"v1","addr":"172.21.15.202:6833","nonce":3399027527}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:04:11.862 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.861+0000 7f79c4838700 1 -- 172.21.15.195:0/2596351932 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f799c063d30 msgr2=0x7f799c0661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:11.862 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.861+0000 7f79c4838700 1 --2- 172.21.15.195:0/2596351932 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f799c063d30 0x7f799c0661e0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f79c00c7180 tx=0x7f79b0009040 comp rx=0 tx=0).stop 2022-06-27T12:04:11.862 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.861+0000 7f79c4838700 1 -- 172.21.15.195:0/2596351932 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f79c006f620 msgr2=0x7f79c00c9d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:11.863 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.861+0000 7f79c4838700 1 --2- 172.21.15.195:0/2596351932 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f79c006f620 0x7f79c00c9d80 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f79c0065b00 tx=0x7f79b4017000 comp rx=0 tx=0).stop 2022-06-27T12:04:11.863 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.861+0000 7f79c4838700 1 -- 172.21.15.195:0/2596351932 shutdown_connections 2022-06-27T12:04:11.863 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.861+0000 7f79c4838700 1 --2- 172.21.15.195:0/2596351932 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f799c063d30 0x7f799c0661e0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:11.863 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.861+0000 7f79c4838700 1 --2- 172.21.15.195:0/2596351932 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f79c006f620 0x7f79c00c9d80 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:11.864 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.861+0000 7f79c4838700 1 --2- 172.21.15.195:0/2596351932 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f79c0150950 0x7f79c014a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:11.864 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.861+0000 7f79c4838700 1 --2- 172.21.15.195:0/2596351932 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f79c0150580 0x7f79c014a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:11.864 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.861+0000 7f79c4838700 1 -- 172.21.15.195:0/2596351932 >> 172.21.15.195:0/2596351932 conn(0x7f79c0063af0 msgr2=0x7f79c014f600 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:04:11.864 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.861+0000 7f79c4838700 1 -- 172.21.15.195:0/2596351932 shutdown_connections 2022-06-27T12:04:11.865 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:11.861+0000 7f79c4838700 1 -- 172.21.15.195:0/2596351932 wait complete. 2022-06-27T12:04:11.865 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 58 2022-06-27T12:04:11.875 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 58, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-06-27T11:47:52.358095+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 58, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5818": { "addr": "172.21.15.202:6833/3399027527", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6832", "nonce": 3399027527, "type": "v2" }, { "addr": "172.21.15.202:6833", "nonce": 3399027527, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5818, "incarnation": 55, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:clientreplay", "state_seq": 31 } }, "last_failure": 0, "last_failure_osd_epoch": 56, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-06-27T12:04:01.265250+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5818 } } } ], "standbys": [ { "addr": "172.21.15.202:6837/3305006841", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6836", "nonce": 3305006841, "type": "v2" }, { "addr": "172.21.15.202:6837", "nonce": 3305006841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6835/2060709822", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6834", "nonce": 2060709822, "type": "v2" }, { "addr": "172.21.15.195:6835", "nonce": 2060709822, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6838/1598751652", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6836", "nonce": 1598751652, "type": "v2" }, { "addr": "172.21.15.195:6838", "nonce": 1598751652, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4275, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6839/1313800661", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6837", "nonce": 1313800661, "type": "v2" }, { "addr": "172.21.15.195:6839", "nonce": 1313800661, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4278, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-06-27T12:04:13.878 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:04:14.014 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.012+0000 7f49e505d700 1 Processor -- start 2022-06-27T12:04:14.015 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.012+0000 7f49e505d700 1 -- start start 2022-06-27T12:04:14.046 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.013+0000 7f49e505d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f49e0150500 0x7f49e0063af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:14.047 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.013+0000 7f49e505d700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f49e006a450 0x7f49e00c48d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:14.047 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.013+0000 7f49e505d700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49e0064030 con 0x7f49e01508d0 2022-06-27T12:04:14.048 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.013+0000 7f49e505d700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49e00c4160 con 0x7f49e0150500 2022-06-27T12:04:14.048 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.013+0000 7f49e505d700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49e00c4440 con 0x7f49e006a450 2022-06-27T12:04:14.049 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.013+0000 7f49ded9d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f49e0150500 0x7f49e0063af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:14.049 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.013+0000 7f49df59e700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f49e01508d0 0x7f49e00bbf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:53704/0 (socket says 172.21.15.195:53704) 2022-06-27T12:04:14.051 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.013+0000 7f49ded9d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f49e0150500 0x7f49e0063af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47512/0 (socket says 172.21.15.195:47512) 2022-06-27T12:04:14.051 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.013+0000 7f49df59e700 1 -- 172.21.15.195:0/2446250606 learned_addr learned my addr 172.21.15.195:0/2446250606 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:04:14.052 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.014+0000 7f49de59c700 1 --2- 172.21.15.195:0/2446250606 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f49e006a450 0x7f49e00c48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:14.052 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.014+0000 7f49ded9d700 1 -- 172.21.15.195:0/2446250606 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f49e006a450 msgr2=0x7f49e00c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:04:14.053 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.014+0000 7f49ded9d700 1 --2- 172.21.15.195:0/2446250606 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f49e006a450 0x7f49e00c48d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:14.053 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.014+0000 7f49ded9d700 1 -- 172.21.15.195:0/2446250606 >> v1:172.21.15.195:6789/0 conn(0x7f49e01508d0 legacy=0x7f49e00bbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:14.054 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.014+0000 7f49ddd9b700 1 -- 172.21.15.195:0/2446250606 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 512739657 0 0) 0x7f49d8001490 con 0x7f49e01508d0 2022-06-27T12:04:14.054 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.014+0000 7f49ded9d700 1 -- 172.21.15.195:0/2446250606 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49e015a800 con 0x7f49e0150500 2022-06-27T12:04:14.054 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.014+0000 7f49de59c700 1 --2- 172.21.15.195:0/2446250606 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f49e006a450 0x7f49e00c48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T12:04:14.055 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.014+0000 7f49ded9d700 1 --2- 172.21.15.195:0/2446250606 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f49e0150500 0x7f49e0063af0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f49d4009720 tx=0x7f49d4009ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45dd4eb1713ee212 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:04:14.055 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.014+0000 7f49ddd9b700 1 -- 172.21.15.195:0/2446250606 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d4005ad0 con 0x7f49e0150500 2022-06-27T12:04:14.056 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.014+0000 7f49ddd9b700 1 -- 172.21.15.195:0/2446250606 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49d4005c30 con 0x7f49e0150500 2022-06-27T12:04:14.056 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.014+0000 7f49ddd9b700 1 -- 172.21.15.195:0/2446250606 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d4006030 con 0x7f49e0150500 2022-06-27T12:04:14.057 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.014+0000 7f49e505d700 1 -- 172.21.15.195:0/2446250606 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f49e0150500 msgr2=0x7f49e0063af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:14.057 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.014+0000 7f49e505d700 1 --2- 172.21.15.195:0/2446250606 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f49e0150500 0x7f49e0063af0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f49d4009720 tx=0x7f49d4009ae0 comp rx=0 tx=0).stop 2022-06-27T12:04:14.058 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.014+0000 7f49e505d700 1 -- 172.21.15.195:0/2446250606 shutdown_connections 2022-06-27T12:04:14.058 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.014+0000 7f49e505d700 1 --2- 172.21.15.195:0/2446250606 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f49e0150500 0x7f49e0063af0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:14.059 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.014+0000 7f49e505d700 1 --2- 172.21.15.195:0/2446250606 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f49e006a450 0x7f49e00c48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:14.059 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.014+0000 7f49e505d700 1 -- 172.21.15.195:0/2446250606 >> 172.21.15.195:0/2446250606 conn(0x7f49e00c9b00 msgr2=0x7f49e00c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:04:14.059 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.015+0000 7f49e505d700 1 -- 172.21.15.195:0/2446250606 shutdown_connections 2022-06-27T12:04:14.060 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.015+0000 7f49e505d700 1 -- 172.21.15.195:0/2446250606 wait complete. 2022-06-27T12:04:14.061 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.016+0000 7f49e505d700 1 Processor -- start 2022-06-27T12:04:14.061 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.016+0000 7f49e505d700 1 -- start start 2022-06-27T12:04:14.062 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.017+0000 7f49e505d700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f49e006a450 0x7f49e0167210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:14.062 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.017+0000 7f49e505d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f49e0150500 0x7f49e00c0550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:14.063 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.017+0000 7f49e505d700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f49e01508d0 0x7f49e0069780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:14.063 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.017+0000 7f49e505d700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49e0166840 con 0x7f49e0150500 2022-06-27T12:04:14.065 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.017+0000 7f49e505d700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49e0167750 con 0x7f49e01508d0 2022-06-27T12:04:14.065 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.017+0000 7f49e505d700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49e0166a30 con 0x7f49e006a450 2022-06-27T12:04:14.065 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.017+0000 7f49de59c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f49e0150500 0x7f49e00c0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:14.066 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.017+0000 7f49de59c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f49e0150500 0x7f49e00c0550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47520/0 (socket says 172.21.15.195:47520) 2022-06-27T12:04:14.066 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.017+0000 7f49de59c700 1 -- 172.21.15.195:0/872379453 learned_addr learned my addr 172.21.15.195:0/872379453 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:04:14.067 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.017+0000 7f49df59e700 1 --2- 172.21.15.195:0/872379453 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f49e01508d0 0x7f49e0069780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:14.067 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.017+0000 7f49ded9d700 1 --2- 172.21.15.195:0/872379453 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f49e006a450 0x7f49e0167210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:14.067 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.017+0000 7f49de59c700 1 -- 172.21.15.195:0/872379453 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f49e006a450 msgr2=0x7f49e0167210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:14.068 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.017+0000 7f49de59c700 1 --2- 172.21.15.195:0/872379453 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f49e006a450 0x7f49e0167210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:14.068 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.017+0000 7f49de59c700 1 -- 172.21.15.195:0/872379453 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f49e01508d0 msgr2=0x7f49e0069780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:14.069 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.017+0000 7f49de59c700 1 --2- 172.21.15.195:0/872379453 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f49e01508d0 0x7f49e0069780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:14.069 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.017+0000 7f49de59c700 1 -- 172.21.15.195:0/872379453 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49d4009290 con 0x7f49e0150500 2022-06-27T12:04:14.069 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.017+0000 7f49de59c700 1 --2- 172.21.15.195:0/872379453 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f49e0150500 0x7f49e00c0550 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f49d000c8b0 tx=0x7f49d0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:04:14.069 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.017+0000 7f49c6ffd700 1 -- 172.21.15.195:0/872379453 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d0011600 con 0x7f49e0150500 2022-06-27T12:04:14.070 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.017+0000 7f49c6ffd700 1 -- 172.21.15.195:0/872379453 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49d0011760 con 0x7f49e0150500 2022-06-27T12:04:14.070 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.017+0000 7f49e505d700 1 -- 172.21.15.195:0/872379453 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49e00c0a90 con 0x7f49e0150500 2022-06-27T12:04:14.070 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.017+0000 7f49c6ffd700 1 -- 172.21.15.195:0/872379453 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d0011b00 con 0x7f49e0150500 2022-06-27T12:04:14.071 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.017+0000 7f49e505d700 1 -- 172.21.15.195:0/872379453 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49e01568a0 con 0x7f49e0150500 2022-06-27T12:04:14.071 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.018+0000 7f49e505d700 1 -- 172.21.15.195:0/872379453 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49e00722c0 con 0x7f49e0150500 2022-06-27T12:04:14.071 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.019+0000 7f49c6ffd700 1 -- 172.21.15.195:0/872379453 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f49d0021070 con 0x7f49e0150500 2022-06-27T12:04:14.071 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.019+0000 7f49c6ffd700 1 --2- 172.21.15.195:0/872379453 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f49c0063d30 0x7f49c00661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:14.072 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.019+0000 7f49c6ffd700 1 -- 172.21.15.195:0/872379453 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6890+0+0 (secure 0 0 0) 0x7f49d00809d0 con 0x7f49e0150500 2022-06-27T12:04:14.072 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.019+0000 7f49ded9d700 1 --2- 172.21.15.195:0/872379453 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f49c0063d30 0x7f49c00661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:14.072 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.019+0000 7f49ded9d700 1 --2- 172.21.15.195:0/872379453 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f49c0063d30 0x7f49c00661e0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f49d4009b20 tx=0x7f49d4013000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:04:14.072 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.023+0000 7f49c6ffd700 1 -- 172.21.15.195:0/872379453 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f49d004f280 con 0x7f49e0150500 2022-06-27T12:04:14.250 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.250+0000 7f49e505d700 1 -- 172.21.15.195:0/872379453 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f49e00ba8e0 con 0x7f49e0150500 2022-06-27T12:04:14.251 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.250+0000 7f49c6ffd700 1 -- 172.21.15.195:0/872379453 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+5368 (secure 0 0 0) 0x7f49d001f090 con 0x7f49e0150500 2022-06-27T12:04:14.252 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:04:14.252 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:04:12.288201+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5818},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5818":{"gid":5818,"name":"b","rank":0,"incarnation":55,"state":"up:active","state_seq":34,"addr":"172.21.15.202:6833/3399027527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3399027527},{"type":"v1","addr":"172.21.15.202:6833","nonce":3399027527}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:04:14.254 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.252+0000 7f49e505d700 1 -- 172.21.15.195:0/872379453 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f49c0063d30 msgr2=0x7f49c00661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:14.255 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.252+0000 7f49e505d700 1 --2- 172.21.15.195:0/872379453 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f49c0063d30 0x7f49c00661e0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f49d4009b20 tx=0x7f49d4013000 comp rx=0 tx=0).stop 2022-06-27T12:04:14.255 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.252+0000 7f49e505d700 1 -- 172.21.15.195:0/872379453 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f49e0150500 msgr2=0x7f49e00c0550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:14.256 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.252+0000 7f49e505d700 1 --2- 172.21.15.195:0/872379453 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f49e0150500 0x7f49e00c0550 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f49d000c8b0 tx=0x7f49d0008040 comp rx=0 tx=0).stop 2022-06-27T12:04:14.256 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.252+0000 7f49e505d700 1 -- 172.21.15.195:0/872379453 shutdown_connections 2022-06-27T12:04:14.256 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.252+0000 7f49e505d700 1 --2- 172.21.15.195:0/872379453 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f49c0063d30 0x7f49c00661e0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:14.256 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.252+0000 7f49e505d700 1 --2- 172.21.15.195:0/872379453 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f49e01508d0 0x7f49e0069780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:14.258 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.252+0000 7f49e505d700 1 --2- 172.21.15.195:0/872379453 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f49e006a450 0x7f49e0167210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:14.258 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.252+0000 7f49e505d700 1 --2- 172.21.15.195:0/872379453 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f49e0150500 0x7f49e00c0550 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:14.258 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.252+0000 7f49e505d700 1 -- 172.21.15.195:0/872379453 >> 172.21.15.195:0/872379453 conn(0x7f49e00c9b00 msgr2=0x7f49e014fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:04:14.258 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.252+0000 7f49e505d700 1 -- 172.21.15.195:0/872379453 shutdown_connections 2022-06-27T12:04:14.259 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:14.253+0000 7f49e505d700 1 -- 172.21.15.195:0/872379453 wait complete. 2022-06-27T12:04:14.259 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 59 2022-06-27T12:04:14.265 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=5273 2022-06-27T12:04:14.266 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 3 secs before reviving mds.d 2022-06-27T12:04:17.267 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.d 2022-06-27T12:04:17.268 INFO:tasks.ceph.mds.d:Restarting daemon 2022-06-27T12:04:17.268 DEBUG:teuthology.orchestra.run.smithi202:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2022-06-27T12:04:17.272 INFO:tasks.ceph.mds.d:Started 2022-06-27T12:04:17.273 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:04:17.368 INFO:tasks.ceph.mds.d.smithi202.stdout:starting mds.d at 2022-06-27T12:04:17.430 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.427+0000 7ff9d1d4d700 1 Processor -- start 2022-06-27T12:04:17.431 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.427+0000 7ff9d1d4d700 1 -- start start 2022-06-27T12:04:17.431 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.428+0000 7ff9d1d4d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7ff9cc150580 0x7ff9cc156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:17.432 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.428+0000 7ff9d1d4d700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7ff9cc150950 0x7ff9cc0c7b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:17.432 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.428+0000 7ff9d1d4d700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9cc0c76b0 con 0x7ff9cc06f620 2022-06-27T12:04:17.432 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.428+0000 7ff9d1d4d700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9cc157430 con 0x7ff9cc150580 2022-06-27T12:04:17.433 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.428+0000 7ff9d1d4d700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9cc0bad70 con 0x7ff9cc150950 2022-06-27T12:04:17.433 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.428+0000 7ff9cb7fe700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7ff9cc150580 0x7ff9cc156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:17.434 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.428+0000 7ff9cb7fe700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7ff9cc150580 0x7ff9cc156ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47530/0 (socket says 172.21.15.195:47530) 2022-06-27T12:04:17.434 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.428+0000 7ff9cb7fe700 1 -- 172.21.15.195:0/331437387 learned_addr learned my addr 172.21.15.195:0/331437387 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:04:17.435 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.428+0000 7ff9cbfff700 1 --1- 172.21.15.195:0/331437387 >> v1:172.21.15.195:6789/0 conn(0x7ff9cc06f620 0x7ff9cc06d6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:53722/0 (socket says 172.21.15.195:53722) 2022-06-27T12:04:17.435 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.428+0000 7ff9caffd700 1 --2- 172.21.15.195:0/331437387 >> v2:172.21.15.202:3300/0 conn(0x7ff9cc150950 0x7ff9cc0c7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:17.436 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.429+0000 7ff9cb7fe700 1 -- 172.21.15.195:0/331437387 >> v2:172.21.15.202:3300/0 conn(0x7ff9cc150950 msgr2=0x7ff9cc0c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:04:17.436 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.429+0000 7ff9cb7fe700 1 --2- 172.21.15.195:0/331437387 >> v2:172.21.15.202:3300/0 conn(0x7ff9cc150950 0x7ff9cc0c7b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:17.437 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.429+0000 7ff9cb7fe700 1 -- 172.21.15.195:0/331437387 >> v1:172.21.15.195:6789/0 conn(0x7ff9cc06f620 legacy=0x7ff9cc06d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:17.437 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.429+0000 7ff9cb7fe700 1 -- 172.21.15.195:0/331437387 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9cc0ba4d0 con 0x7ff9cc150580 2022-06-27T12:04:17.438 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.429+0000 7ff9ca7fc700 1 -- 172.21.15.195:0/331437387 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4229689759 0 0) 0x7ff9cc0c76b0 con 0x7ff9cc06f620 2022-06-27T12:04:17.451 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.429+0000 7ff9cb7fe700 1 --2- 172.21.15.195:0/331437387 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff9cc150580 0x7ff9cc156ef0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc06ea70 tx=0x7ff9b4009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a34326668e1b3b8 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:04:17.451 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.429+0000 7ff9ca7fc700 1 -- 172.21.15.195:0/331437387 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9b4011630 con 0x7ff9cc150580 2022-06-27T12:04:17.451 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.429+0000 7ff9ca7fc700 1 -- 172.21.15.195:0/331437387 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9b4011790 con 0x7ff9cc150580 2022-06-27T12:04:17.452 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.429+0000 7ff9ca7fc700 1 -- 172.21.15.195:0/331437387 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9b4011b30 con 0x7ff9cc150580 2022-06-27T12:04:17.452 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.429+0000 7ff9d1d4d700 1 -- 172.21.15.195:0/331437387 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff9cc150580 msgr2=0x7ff9cc156ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:17.453 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.429+0000 7ff9d1d4d700 1 --2- 172.21.15.195:0/331437387 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff9cc150580 0x7ff9cc156ef0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc06ea70 tx=0x7ff9b4009120 comp rx=0 tx=0).stop 2022-06-27T12:04:17.453 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.429+0000 7ff9d1d4d700 1 -- 172.21.15.195:0/331437387 shutdown_connections 2022-06-27T12:04:17.454 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.429+0000 7ff9d1d4d700 1 --2- 172.21.15.195:0/331437387 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff9cc150580 0x7ff9cc156ef0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:17.454 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.429+0000 7ff9d1d4d700 1 --2- 172.21.15.195:0/331437387 >> v2:172.21.15.202:3300/0 conn(0x7ff9cc150950 0x7ff9cc0c7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:17.455 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.429+0000 7ff9d1d4d700 1 -- 172.21.15.195:0/331437387 >> 172.21.15.195:0/331437387 conn(0x7ff9cc063af0 msgr2=0x7ff9cc063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:04:17.455 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.429+0000 7ff9d1d4d700 1 -- 172.21.15.195:0/331437387 shutdown_connections 2022-06-27T12:04:17.455 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.429+0000 7ff9d1d4d700 1 -- 172.21.15.195:0/331437387 wait complete. 2022-06-27T12:04:17.456 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.431+0000 7ff9d1d4d700 1 Processor -- start 2022-06-27T12:04:17.456 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.431+0000 7ff9d1d4d700 1 -- start start 2022-06-27T12:04:17.457 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.431+0000 7ff9d1d4d700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff9cc06f620 0x7ff9cc0c38c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:17.457 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.431+0000 7ff9d1d4d700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff9cc150580 0x7ff9cc06d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:17.458 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.431+0000 7ff9d1d4d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff9cc150950 0x7ff9cc166c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:17.458 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.431+0000 7ff9d1d4d700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9cc06a3b0 con 0x7ff9cc150950 2022-06-27T12:04:17.458 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.431+0000 7ff9d1d4d700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9cc0c3f20 con 0x7ff9cc150580 2022-06-27T12:04:17.459 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.431+0000 7ff9d1d4d700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9cc0c4200 con 0x7ff9cc06f620 2022-06-27T12:04:17.460 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.432+0000 7ff9cbfff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff9cc150950 0x7ff9cc166c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:17.460 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.432+0000 7ff9cbfff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff9cc150950 0x7ff9cc166c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47540/0 (socket says 172.21.15.195:47540) 2022-06-27T12:04:17.461 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.432+0000 7ff9cbfff700 1 -- 172.21.15.195:0/172666875 learned_addr learned my addr 172.21.15.195:0/172666875 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:04:17.461 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.432+0000 7ff9caffd700 1 --2- 172.21.15.195:0/172666875 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff9cc150580 0x7ff9cc06d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:17.461 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.432+0000 7ff9cb7fe700 1 --2- 172.21.15.195:0/172666875 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff9cc06f620 0x7ff9cc0c38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:17.462 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.432+0000 7ff9cbfff700 1 -- 172.21.15.195:0/172666875 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff9cc06f620 msgr2=0x7ff9cc0c38c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:17.463 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.432+0000 7ff9cbfff700 1 --2- 172.21.15.195:0/172666875 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff9cc06f620 0x7ff9cc0c38c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:17.463 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.432+0000 7ff9cbfff700 1 -- 172.21.15.195:0/172666875 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff9cc150580 msgr2=0x7ff9cc06d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:17.464 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.432+0000 7ff9cbfff700 1 --2- 172.21.15.195:0/172666875 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff9cc150580 0x7ff9cc06d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:17.464 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.432+0000 7ff9cbfff700 1 -- 172.21.15.195:0/172666875 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9b40093a0 con 0x7ff9cc150950 2022-06-27T12:04:17.464 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.432+0000 7ff9cbfff700 1 --2- 172.21.15.195:0/172666875 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff9cc150950 0x7ff9cc166c50 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7ff9c400c810 tx=0x7ff9c4001540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:04:17.465 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.432+0000 7ff9c37fe700 1 -- 172.21.15.195:0/172666875 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c4004c80 con 0x7ff9cc150950 2022-06-27T12:04:17.465 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.432+0000 7ff9c37fe700 1 -- 172.21.15.195:0/172666875 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9c4004e00 con 0x7ff9cc150950 2022-06-27T12:04:17.466 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.432+0000 7ff9d1d4d700 1 -- 172.21.15.195:0/172666875 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9cc0c44e0 con 0x7ff9cc150950 2022-06-27T12:04:17.466 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.432+0000 7ff9c37fe700 1 -- 172.21.15.195:0/172666875 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c40073a0 con 0x7ff9cc150950 2022-06-27T12:04:17.467 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.432+0000 7ff9d1d4d700 1 -- 172.21.15.195:0/172666875 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9cc0c4aa0 con 0x7ff9cc150950 2022-06-27T12:04:17.467 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.433+0000 7ff9d1d4d700 1 -- 172.21.15.195:0/172666875 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9cc0722c0 con 0x7ff9cc150950 2022-06-27T12:04:17.468 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.434+0000 7ff9c37fe700 1 -- 172.21.15.195:0/172666875 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7ff9c401a030 con 0x7ff9cc150950 2022-06-27T12:04:17.468 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.434+0000 7ff9c37fe700 1 --2- 172.21.15.195:0/172666875 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff9b0063d30 0x7ff9b00661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:17.469 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.434+0000 7ff9c37fe700 1 -- 172.21.15.195:0/172666875 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6890+0+0 (secure 0 0 0) 0x7ff9c4080000 con 0x7ff9cc150950 2022-06-27T12:04:17.469 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.434+0000 7ff9cb7fe700 1 --2- 172.21.15.195:0/172666875 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff9b0063d30 0x7ff9b00661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:17.470 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.435+0000 7ff9cb7fe700 1 --2- 172.21.15.195:0/172666875 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff9b0063d30 0x7ff9b00661e0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7ff9b4009160 tx=0x7ff9b4017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:04:17.470 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.439+0000 7ff9c37fe700 1 -- 172.21.15.195:0/172666875 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7ff9c404e930 con 0x7ff9cc150950 2022-06-27T12:04:17.674 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.674+0000 7ff9d1d4d700 1 -- 172.21.15.195:0/172666875 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff9cc06fe20 con 0x7ff9cc150950 2022-06-27T12:04:17.675 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.674+0000 7ff9c37fe700 1 -- 172.21.15.195:0/172666875 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+5368 (secure 0 0 0) 0x7ff9c4018030 con 0x7ff9cc150950 2022-06-27T12:04:17.676 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:04:17.676 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:04:12.288201+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5818},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5818":{"gid":5818,"name":"b","rank":0,"incarnation":55,"state":"up:active","state_seq":34,"addr":"172.21.15.202:6833/3399027527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3399027527},{"type":"v1","addr":"172.21.15.202:6833","nonce":3399027527}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:04:17.679 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.676+0000 7ff9d1d4d700 1 -- 172.21.15.195:0/172666875 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff9b0063d30 msgr2=0x7ff9b00661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:17.679 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.676+0000 7ff9d1d4d700 1 --2- 172.21.15.195:0/172666875 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff9b0063d30 0x7ff9b00661e0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7ff9b4009160 tx=0x7ff9b4017000 comp rx=0 tx=0).stop 2022-06-27T12:04:17.680 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.676+0000 7ff9d1d4d700 1 -- 172.21.15.195:0/172666875 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff9cc150950 msgr2=0x7ff9cc166c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:17.680 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.676+0000 7ff9d1d4d700 1 --2- 172.21.15.195:0/172666875 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff9cc150950 0x7ff9cc166c50 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7ff9c400c810 tx=0x7ff9c4001540 comp rx=0 tx=0).stop 2022-06-27T12:04:17.680 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.676+0000 7ff9d1d4d700 1 -- 172.21.15.195:0/172666875 shutdown_connections 2022-06-27T12:04:17.681 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.676+0000 7ff9d1d4d700 1 --2- 172.21.15.195:0/172666875 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff9b0063d30 0x7ff9b00661e0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:17.682 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.676+0000 7ff9d1d4d700 1 --2- 172.21.15.195:0/172666875 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff9cc150950 0x7ff9cc166c50 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:17.682 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.676+0000 7ff9d1d4d700 1 --2- 172.21.15.195:0/172666875 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff9cc06f620 0x7ff9cc0c38c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:17.683 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.676+0000 7ff9d1d4d700 1 --2- 172.21.15.195:0/172666875 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff9cc150580 0x7ff9cc06d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:17.683 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.676+0000 7ff9d1d4d700 1 -- 172.21.15.195:0/172666875 >> 172.21.15.195:0/172666875 conn(0x7ff9cc063af0 msgr2=0x7ff9cc14d950 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:04:17.683 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.676+0000 7ff9d1d4d700 1 -- 172.21.15.195:0/172666875 shutdown_connections 2022-06-27T12:04:17.684 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:17.676+0000 7ff9d1d4d700 1 -- 172.21.15.195:0/172666875 wait complete. 2022-06-27T12:04:17.684 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 59 2022-06-27T12:04:17.689 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2022-06-27T12:04:18.239 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 41% (101/245) Updating files: 42% (103/245) Updating files: 43% (106/245) Updating files: 44% (108/245) Updating files: 45% (111/245) Updating files: 46% (113/245) Updating files: 47% (116/245) Updating files: 48% (118/245) Updating files: 49% (121/245) Updating files: 50% (123/245) Updating files: 51% (125/245) Updating files: 52% (128/245) Updating files: 53% (130/245) Updating files: 54% (133/245) Updating files: 55% (135/245) Updating files: 55% (136/245) Updating files: 56% (138/245) Updating files: 57% (140/245) Updating files: 58% (143/245) Updating files: 59% (145/245) Updating files: 60% (147/245) Updating files: 61% (150/245) Updating files: 62% (152/245) Updating files: 63% (155/245) Updating files: 64% (157/245) Updating files: 65% (160/245) Updating files: 66% (162/245) Updating files: 67% (165/245) Updating files: 68% (167/245) Updating files: 69% (170/245) Updating files: 70% (172/245) Updating files: 71% (174/245) Updating files: 72% (177/245) Updating files: 73% (179/245) Updating files: 74% (182/245) Updating files: 75% (184/245) Updating files: 76% (187/245) Updating files: 77% (189/245) Updating files: 78% (192/245) Updating files: 79% (194/245) Updating files: 80% (196/245) Updating files: 81% (199/245) Updating files: 82% (201/245) Updating files: 83% (204/245) Updating files: 84% (206/245) Updating files: 85% (209/245) Updating files: 86% (211/245) Updating files: 87% (214/245) Updating files: 87% (215/245) Updating files: 88% (216/245) Updating files: 89% (219/245) Updating files: 90% (221/245) Updating files: 91% (223/245) Updating files: 92% (226/245) Updating files: 93% (228/245) Updating files: 94% (231/245) Updating files: 95% (233/245) Updating files: 96% (236/245) Updating files: 97% (238/245) Updating files: 98% (241/245) Updating files: 99% (243/245) Updating files: 100% (245/245) Updating files: 100% (245/245), done. 2022-06-27T12:04:19.043 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at 1dd420957f9 journal: detect size of raw block devices properly 2022-06-27T12:04:19.048 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.5 2022-06-27T12:04:19.073 INFO:tasks.workunit.client.0.smithi195.stdout:v0.6 2022-06-27T12:04:19.074 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:04:19.075 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 6 -eq 48 ']' 2022-06-27T12:04:19.075 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.6 2022-06-27T12:04:19.075 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.6 2022-06-27T12:04:19.076 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.6 2022-06-27T12:04:19.691 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:04:19.820 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:19.820+0000 7f5160263700 1 Processor -- start 2022-06-27T12:04:19.842 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:19.820+0000 7f5160263700 1 -- start start 2022-06-27T12:04:19.842 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:19.821+0000 7f5160263700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f51580688a0 0x7f51580c28c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:19.843 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:19.821+0000 7f5160263700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51580c4b10 con 0x7f515806a440 2022-06-27T12:04:19.843 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:19.821+0000 7f5160263700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5158068f20 con 0x7f5158156100 2022-06-27T12:04:19.844 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:19.821+0000 7f5160263700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51580c3dc0 con 0x7f51580688a0 2022-06-27T12:04:19.844 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:19.821+0000 7f515d7fe700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f515806a440 0x7f51581564d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:53742/0 (socket says 172.21.15.195:53742) 2022-06-27T12:04:19.845 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:19.821+0000 7f515d7fe700 1 -- 172.21.15.195:0/1400309830 learned_addr learned my addr 172.21.15.195:0/1400309830 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:04:19.845 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:19.821+0000 7f515e800700 1 --2- 172.21.15.195:0/1400309830 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f51580688a0 0x7f51580c28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:19.846 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:19.821+0000 7f515cffd700 1 -- 172.21.15.195:0/1400309830 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2259204436 0 0) 0x7f51580c4b10 con 0x7f515806a440 2022-06-27T12:04:19.846 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:19.821+0000 7f515cffd700 1 -- 172.21.15.195:0/1400309830 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5140003430 con 0x7f515806a440 2022-06-27T12:04:19.846 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:19.821+0000 7f515cffd700 1 -- 172.21.15.195:0/1400309830 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3916834480 0 0) 0x7f5158068f20 con 0x7f5158156100 2022-06-27T12:04:19.847 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:19.821+0000 7f515cffd700 1 -- 172.21.15.195:0/1400309830 --> v1:172.21.15.202:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51580c4b10 con 0x7f5158156100 2022-06-27T12:04:19.848 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:19.821+0000 7f515e800700 1 -- 172.21.15.195:0/1400309830 >> v1:172.21.15.202:6789/0 conn(0x7f5158156100 legacy=0x7f51580c4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:19.848 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:19.821+0000 7f515e800700 1 -- 172.21.15.195:0/1400309830 >> v1:172.21.15.195:6789/0 conn(0x7f515806a440 legacy=0x7f51581564d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:19.849 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:19.821+0000 7f515e800700 1 -- 172.21.15.195:0/1400309830 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51580c56b0 con 0x7f51580688a0 2022-06-27T12:04:19.849 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:19.821+0000 7f515e800700 1 --2- 172.21.15.195:0/1400309830 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f51580688a0 0x7f51580c28c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f5150009e90 tx=0x7f5150009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2e4981e2fee175c server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:04:19.849 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:19.822+0000 7f515cffd700 1 -- 172.21.15.195:0/1400309830 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f515000e070 con 0x7f51580688a0 2022-06-27T12:04:19.849 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:19.822+0000 7f515cffd700 1 -- 172.21.15.195:0/1400309830 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f515000bb80 con 0x7f51580688a0 2022-06-27T12:04:19.850 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:19.822+0000 7f515cffd700 1 -- 172.21.15.195:0/1400309830 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5150012430 con 0x7f51580688a0 2022-06-27T12:04:19.850 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:19.822+0000 7f5160263700 1 -- 172.21.15.195:0/1400309830 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f51580688a0 msgr2=0x7f51580c28c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:19.850 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:19.822+0000 7f5160263700 1 --2- 172.21.15.195:0/1400309830 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f51580688a0 0x7f51580c28c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f5150009e90 tx=0x7f5150009780 comp rx=0 tx=0).stop 2022-06-27T12:04:19.851 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:19.822+0000 7f5160263700 1 -- 172.21.15.195:0/1400309830 shutdown_connections 2022-06-27T12:04:19.851 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:19.822+0000 7f5160263700 1 --2- 172.21.15.195:0/1400309830 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f51580688a0 0x7f51580c28c0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:19.855 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:19.822+0000 7f5160263700 1 -- 172.21.15.195:0/1400309830 >> 172.21.15.195:0/1400309830 conn(0x7f51580c9b00 msgr2=0x7f51580c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:04:19.855 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:19.822+0000 7f5160263700 1 -- 172.21.15.195:0/1400309830 shutdown_connections 2022-06-27T12:04:19.855 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:19.822+0000 7f5160263700 1 -- 172.21.15.195:0/1400309830 wait complete. 2022-06-27T12:04:19.856 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:19.823+0000 7f5160263700 1 Processor -- start 2022-06-27T12:04:19.856 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:19.824+0000 7f5160263700 1 -- start start 2022-06-27T12:04:19.856 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:19.824+0000 7f5160263700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f51580688a0 0x7f515806b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:19.857 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:19.824+0000 7f5160263700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f515806a440 0x7f51581564d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:19.857 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:19.824+0000 7f5160263700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5158156100 0x7f5158156a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:19.858 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:19.824+0000 7f5160263700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f515814c0c0 con 0x7f515806a440 2022-06-27T12:04:19.858 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:19.824+0000 7f5160263700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51580bbf10 con 0x7f51580688a0 2022-06-27T12:04:19.859 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:19.824+0000 7f5160263700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5158166520 con 0x7f5158156100 2022-06-27T12:04:19.859 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:19.824+0000 7f515d7fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f515806a440 0x7f51581564d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:19.860 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:19.824+0000 7f515d7fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f515806a440 0x7f51581564d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47560/0 (socket says 172.21.15.195:47560) 2022-06-27T12:04:19.860 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:19.824+0000 7f515dfff700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f51580688a0 0x7f515806b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:19.860 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:19.824+0000 7f515d7fe700 1 -- 172.21.15.195:0/3204036155 learned_addr learned my addr 172.21.15.195:0/3204036155 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:04:19.861 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:19.824+0000 7f515e800700 1 --2- 172.21.15.195:0/3204036155 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5158156100 0x7f5158156a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:19.861 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:19.825+0000 7f515d7fe700 1 -- 172.21.15.195:0/3204036155 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5158156100 msgr2=0x7f5158156a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:19.862 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:19.825+0000 7f515d7fe700 1 --2- 172.21.15.195:0/3204036155 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5158156100 0x7f5158156a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:19.862 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:19.825+0000 7f515d7fe700 1 -- 172.21.15.195:0/3204036155 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f51580688a0 msgr2=0x7f515806b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:19.862 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:19.825+0000 7f515d7fe700 1 --2- 172.21.15.195:0/3204036155 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f51580688a0 0x7f515806b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:19.863 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:19.825+0000 7f515d7fe700 1 -- 172.21.15.195:0/3204036155 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5150009a20 con 0x7f515806a440 2022-06-27T12:04:19.863 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:19.825+0000 7f515dfff700 1 --2- 172.21.15.195:0/3204036155 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f51580688a0 0x7f515806b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:04:19.864 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:19.825+0000 7f515d7fe700 1 --2- 172.21.15.195:0/3204036155 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f515806a440 0x7f51581564d0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f514800c590 tx=0x7f5148007cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:04:19.864 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:19.825+0000 7f5145ffb700 1 -- 172.21.15.195:0/3204036155 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51480072e0 con 0x7f515806a440 2022-06-27T12:04:19.865 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:19.825+0000 7f5145ffb700 1 -- 172.21.15.195:0/3204036155 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5148007440 con 0x7f515806a440 2022-06-27T12:04:19.865 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:19.825+0000 7f5145ffb700 1 -- 172.21.15.195:0/3204036155 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51480057c0 con 0x7f515806a440 2022-06-27T12:04:19.866 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:19.825+0000 7f5160263700 1 -- 172.21.15.195:0/3204036155 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51580bbc30 con 0x7f515806a440 2022-06-27T12:04:19.866 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:19.825+0000 7f5160263700 1 -- 172.21.15.195:0/3204036155 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5158166a10 con 0x7f515806a440 2022-06-27T12:04:19.867 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:19.826+0000 7f5160263700 1 -- 172.21.15.195:0/3204036155 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51580722c0 con 0x7f515806a440 2022-06-27T12:04:19.867 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:19.826+0000 7f5145ffb700 1 -- 172.21.15.195:0/3204036155 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f5148019030 con 0x7f515806a440 2022-06-27T12:04:19.868 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:19.827+0000 7f5145ffb700 1 --2- 172.21.15.195:0/3204036155 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5138063d00 0x7f51380661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:19.868 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:19.827+0000 7f5145ffb700 1 -- 172.21.15.195:0/3204036155 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6890+0+0 (secure 0 0 0) 0x7f514807fde0 con 0x7f515806a440 2022-06-27T12:04:19.868 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:19.827+0000 7f515dfff700 1 --2- 172.21.15.195:0/3204036155 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5138063d00 0x7f51380661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:19.869 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:19.827+0000 7f515dfff700 1 --2- 172.21.15.195:0/3204036155 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5138063d00 0x7f51380661b0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f5154001170 tx=0x7f5154009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:04:19.869 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:19.831+0000 7f5145ffb700 1 -- 172.21.15.195:0/3204036155 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f514804e690 con 0x7f515806a440 2022-06-27T12:04:20.104 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:20.103+0000 7f5160263700 1 -- 172.21.15.195:0/3204036155 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f51580c07d0 con 0x7f515806a440 2022-06-27T12:04:20.105 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:20.104+0000 7f5145ffb700 1 -- 172.21.15.195:0/3204036155 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+6128 (secure 0 0 0) 0x7f5148017070 con 0x7f515806a440 2022-06-27T12:04:20.106 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:04:20.106 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":5489,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/2342195382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2342195382},{"type":"v1","addr":"172.21.15.202:6835","nonce":2342195382}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:04:12.288201+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5818},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5818":{"gid":5818,"name":"b","rank":0,"incarnation":55,"state":"up:active","state_seq":34,"addr":"172.21.15.202:6833/3399027527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3399027527},{"type":"v1","addr":"172.21.15.202:6833","nonce":3399027527}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:04:20.111 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:20.106+0000 7f5160263700 1 -- 172.21.15.195:0/3204036155 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5138063d00 msgr2=0x7f51380661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:20.112 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:20.106+0000 7f5160263700 1 --2- 172.21.15.195:0/3204036155 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5138063d00 0x7f51380661b0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f5154001170 tx=0x7f5154009040 comp rx=0 tx=0).stop 2022-06-27T12:04:20.112 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:20.106+0000 7f5160263700 1 -- 172.21.15.195:0/3204036155 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f515806a440 msgr2=0x7f51581564d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:20.113 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:20.106+0000 7f5160263700 1 --2- 172.21.15.195:0/3204036155 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f515806a440 0x7f51581564d0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f514800c590 tx=0x7f5148007cb0 comp rx=0 tx=0).stop 2022-06-27T12:04:20.113 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:20.106+0000 7f5160263700 1 -- 172.21.15.195:0/3204036155 shutdown_connections 2022-06-27T12:04:20.113 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:20.106+0000 7f5160263700 1 --2- 172.21.15.195:0/3204036155 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5138063d00 0x7f51380661b0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:20.113 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:20.106+0000 7f5160263700 1 --2- 172.21.15.195:0/3204036155 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5158156100 0x7f5158156a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:20.114 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:20.106+0000 7f5160263700 1 --2- 172.21.15.195:0/3204036155 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f51580688a0 0x7f515806b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:20.114 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:20.106+0000 7f5160263700 1 --2- 172.21.15.195:0/3204036155 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f515806a440 0x7f51581564d0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:20.114 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:20.106+0000 7f5160263700 1 -- 172.21.15.195:0/3204036155 >> 172.21.15.195:0/3204036155 conn(0x7f51580c9b00 msgr2=0x7f51580c1870 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:04:20.114 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:20.106+0000 7f5160263700 1 -- 172.21.15.195:0/3204036155 shutdown_connections 2022-06-27T12:04:20.115 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:20.106+0000 7f5160263700 1 -- 172.21.15.195:0/3204036155 wait complete. 2022-06-27T12:04:20.115 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 60 2022-06-27T12:04:20.120 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d reported in up:standby state 2022-06-27T12:04:20.121 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 30 secs before thrashing 2022-06-27T12:04:22.036 INFO:tasks.ceph.mds.b.smithi202.stderr:2022-06-27T12:04:22.036+0000 7fb73a575700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2022-06-27T12:04:24.475 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 41% (94/227) Updating files: 42% (96/227) Updating files: 43% (98/227) Updating files: 44% (100/227) Updating files: 45% (103/227) Updating files: 46% (105/227) Updating files: 47% (107/227) Updating files: 48% (109/227) Updating files: 49% (112/227) Updating files: 50% (114/227) Updating files: 51% (116/227) Updating files: 51% (118/227) Updating files: 52% (119/227) Updating files: 53% (121/227) Updating files: 54% (123/227) Updating files: 55% (125/227) Updating files: 56% (128/227) Updating files: 57% (130/227) Updating files: 58% (132/227) Updating files: 59% (134/227) Updating files: 60% (137/227) Updating files: 61% (139/227) Updating files: 62% (141/227) Updating files: 63% (144/227) Updating files: 64% (146/227) Updating files: 65% (148/227) Updating files: 66% (150/227) Updating files: 67% (153/227) Updating files: 68% (155/227) Updating files: 69% (157/227) Updating files: 70% (159/227) Updating files: 71% (162/227) Updating files: 72% (164/227) Updating files: 73% (166/227) Updating files: 74% (168/227) Updating files: 75% (171/227) Updating files: 76% (173/227) Updating files: 77% (175/227) Updating files: 78% (178/227) Updating files: 79% (180/227) Updating files: 80% (182/227) Updating files: 81% (184/227) Updating files: 82% (187/227) Updating files: 83% (189/227) Updating files: 84% (191/227) Updating files: 85% (193/227) Updating files: 86% (196/227) Updating files: 87% (198/227) Updating files: 88% (200/227) Updating files: 89% (203/227) Updating files: 90% (205/227) Updating files: 91% (207/227) Updating files: 92% (209/227) Updating files: 93% (212/227) Updating files: 94% (214/227) Updating files: 95% (216/227) Updating files: 96% (218/227) Updating files: 97% (221/227) Updating files: 98% (223/227) Updating files: 99% (225/227) Updating files: 100% (227/227) Updating files: 100% (227/227), done. 2022-06-27T12:04:24.498 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at a4c752df713 v0.6 2022-06-27T12:04:24.499 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.6 2022-06-27T12:04:24.713 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:04:24.716 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 7 -eq 48 ']' 2022-06-27T12:04:24.716 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.7 2022-06-27T12:04:24.717 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.7 2022-06-27T12:04:24.717 INFO:tasks.workunit.client.0.smithi195.stdout:v0.7 2022-06-27T12:04:24.718 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.7 2022-06-27T12:04:28.427 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 21% (43/201) Updating files: 22% (45/201) Updating files: 23% (47/201) Updating files: 24% (49/201) Updating files: 25% (51/201) Updating files: 26% (53/201) Updating files: 27% (55/201) Updating files: 28% (57/201) Updating files: 29% (59/201) Updating files: 30% (61/201) Updating files: 31% (63/201) Updating files: 32% (65/201) Updating files: 33% (67/201) Updating files: 34% (69/201) Updating files: 35% (71/201) Updating files: 36% (73/201) Updating files: 37% (75/201) Updating files: 38% (77/201) Updating files: 39% (79/201) Updating files: 40% (81/201) Updating files: 41% (83/201) Updating files: 42% (85/201) Updating files: 43% (87/201) Updating files: 44% (89/201) Updating files: 45% (91/201) Updating files: 45% (92/201) Updating files: 46% (93/201) Updating files: 47% (95/201) Updating files: 48% (97/201) Updating files: 49% (99/201) Updating files: 50% (101/201) Updating files: 51% (103/201) Updating files: 52% (105/201) Updating files: 53% (107/201) Updating files: 54% (109/201) Updating files: 55% (111/201) Updating files: 56% (113/201) Updating files: 57% (115/201) Updating files: 58% (117/201) Updating files: 59% (119/201) Updating files: 60% (121/201) Updating files: 61% (123/201) Updating files: 62% (125/201) Updating files: 63% (127/201) Updating files: 64% (129/201) Updating files: 65% (131/201) Updating files: 66% (133/201) Updating files: 67% (135/201) Updating files: 68% (137/201) Updating files: 69% (139/201) Updating files: 70% (141/201) Updating files: 71% (143/201) Updating files: 72% (145/201) Updating files: 73% (147/201) Updating files: 74% (149/201) Updating files: 75% (151/201) Updating files: 76% (153/201) Updating files: 77% (155/201) Updating files: 78% (157/201) Updating files: 79% (159/201) Updating files: 80% (161/201) Updating files: 81% (163/201) Updating files: 82% (165/201) Updating files: 83% (167/201) Updating files: 84% (169/201) Updating files: 85% (171/201) Updating files: 86% (173/201) Updating files: 87% (175/201) Updating files: 88% (177/201) Updating files: 89% (179/201) Updating files: 90% (181/201) Updating files: 91% (183/201) Updating files: 92% (185/201) Updating files: 93% (187/201) Updating files: 94% (189/201) Updating files: 95% (191/201) Updating files: 96% (193/201) Updating files: 97% (195/201) Updating files: 98% (197/201) Updating files: 99% (199/201) Updating files: 100% (201/201) Updating files: 100% (201/201), done. 2022-06-27T12:04:28.443 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at c46e1b320c6 mkcephfs: btrfsctl -a between mkfs and mount 2022-06-27T12:04:28.454 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.7 2022-06-27T12:04:28.528 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:04:28.529 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 8 -eq 48 ']' 2022-06-27T12:04:28.529 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.8 2022-06-27T12:04:28.529 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.8 2022-06-27T12:04:28.530 INFO:tasks.workunit.client.0.smithi195.stdout:v0.8 2022-06-27T12:04:28.530 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.8 2022-06-27T12:04:30.849 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:04:30.855 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:04:33.147 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 52% (118/225) Updating files: 53% (120/225) Updating files: 54% (122/225) Updating files: 55% (124/225) Updating files: 56% (126/225) Updating files: 57% (129/225) Updating files: 58% (131/225) Updating files: 58% (132/225) Updating files: 59% (133/225) Updating files: 60% (135/225) Updating files: 61% (138/225) Updating files: 62% (140/225) Updating files: 63% (142/225) Updating files: 64% (144/225) Updating files: 65% (147/225) Updating files: 66% (149/225) Updating files: 67% (151/225) Updating files: 68% (153/225) Updating files: 69% (156/225) Updating files: 70% (158/225) Updating files: 71% (160/225) Updating files: 72% (162/225) Updating files: 73% (165/225) Updating files: 74% (167/225) Updating files: 75% (169/225) Updating files: 76% (171/225) Updating files: 77% (174/225) Updating files: 78% (176/225) Updating files: 79% (178/225) Updating files: 80% (180/225) Updating files: 81% (183/225) Updating files: 82% (185/225) Updating files: 83% (187/225) Updating files: 84% (189/225) Updating files: 85% (192/225) Updating files: 86% (194/225) Updating files: 87% (196/225) Updating files: 88% (198/225) Updating files: 89% (201/225) Updating files: 90% (203/225) Updating files: 91% (205/225) Updating files: 92% (207/225) Updating files: 93% (210/225) Updating files: 94% (212/225) Updating files: 95% (214/225) Updating files: 96% (216/225) Updating files: 97% (219/225) Updating files: 98% (221/225) Updating files: 98% (222/225) Updating files: 99% (223/225) Updating files: 100% (225/225) Updating files: 100% (225/225), done. 2022-06-27T12:04:33.174 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at 284161c00ab kclient: kill dput/d_drop debug cruft 2022-06-27T12:04:33.175 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.8 2022-06-27T12:04:33.222 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:04:33.222 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 9 -eq 48 ']' 2022-06-27T12:04:33.224 INFO:tasks.workunit.client.0.smithi195.stdout:v0.9 2022-06-27T12:04:33.225 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.9 2022-06-27T12:04:33.225 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.9 2022-06-27T12:04:33.226 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.9 2022-06-27T12:04:36.707 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 52% (102/193) Updating files: 53% (103/193) Updating files: 54% (105/193) Updating files: 55% (107/193) Updating files: 56% (109/193) Updating files: 57% (111/193) Updating files: 58% (112/193) Updating files: 59% (114/193) Updating files: 60% (116/193) Updating files: 61% (118/193) Updating files: 62% (120/193) Updating files: 63% (122/193) Updating files: 64% (124/193) Updating files: 65% (126/193) Updating files: 66% (128/193) Updating files: 67% (130/193) Updating files: 68% (132/193) Updating files: 69% (134/193) Updating files: 70% (136/193) Updating files: 71% (138/193) Updating files: 72% (139/193) Updating files: 73% (141/193) Updating files: 74% (143/193) Updating files: 74% (144/193) Updating files: 75% (145/193) Updating files: 76% (147/193) Updating files: 77% (149/193) Updating files: 78% (151/193) Updating files: 79% (153/193) Updating files: 80% (155/193) Updating files: 81% (157/193) Updating files: 82% (159/193) Updating files: 83% (161/193) Updating files: 84% (163/193) Updating files: 85% (165/193) Updating files: 86% (166/193) Updating files: 87% (168/193) Updating files: 88% (170/193) Updating files: 89% (172/193) Updating files: 90% (174/193) Updating files: 91% (176/193) Updating files: 92% (178/193) Updating files: 93% (180/193) Updating files: 94% (182/193) Updating files: 95% (184/193) Updating files: 96% (186/193) Updating files: 97% (188/193) Updating files: 98% (190/193) Updating files: 99% (192/193) Updating files: 100% (193/193) Updating files: 100% (193/193), done. 2022-06-27T12:04:36.738 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at 41481853924 kclient: strip out kernel version compatibility cruft 2022-06-27T12:04:36.741 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.9 2022-06-27T12:04:37.118 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:04:37.118 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 10 -eq 48 ']' 2022-06-27T12:04:37.119 INFO:tasks.workunit.client.0.smithi195.stdout:v0.10 2022-06-27T12:04:37.119 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.10 2022-06-27T12:04:37.120 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.10 2022-06-27T12:04:37.120 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.10 2022-06-27T12:04:38.586 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 41% (68/164) Updating files: 42% (69/164) Updating files: 43% (71/164) Updating files: 44% (73/164) Updating files: 45% (74/164) Updating files: 46% (76/164) Updating files: 47% (78/164) Updating files: 48% (79/164) Updating files: 49% (81/164) Updating files: 50% (82/164) Updating files: 51% (84/164) Updating files: 52% (86/164) Updating files: 53% (87/164) Updating files: 54% (89/164) Updating files: 55% (91/164) Updating files: 56% (92/164) Updating files: 57% (94/164) Updating files: 58% (96/164) Updating files: 59% (97/164) Updating files: 60% (99/164) Updating files: 61% (101/164) Updating files: 62% (102/164) Updating files: 63% (104/164) Updating files: 64% (105/164) Updating files: 65% (107/164) Updating files: 66% (109/164) Updating files: 67% (110/164) Updating files: 68% (112/164) Updating files: 69% (114/164) Updating files: 70% (115/164) Updating files: 71% (117/164) Updating files: 72% (119/164) Updating files: 73% (120/164) Updating files: 74% (122/164) Updating files: 75% (123/164) Updating files: 76% (125/164) Updating files: 77% (127/164) Updating files: 78% (128/164) Updating files: 79% (130/164) Updating files: 80% (132/164) Updating files: 81% (133/164) Updating files: 82% (135/164) Updating files: 83% (137/164) Updating files: 84% (138/164) Updating files: 85% (140/164) Updating files: 86% (142/164) Updating files: 87% (143/164) Updating files: 88% (145/164) Updating files: 89% (146/164) Updating files: 90% (148/164) Updating files: 91% (150/164) Updating files: 92% (151/164) Updating files: 93% (153/164) Updating files: 94% (155/164) Updating files: 95% (156/164) Updating files: 96% (158/164) Updating files: 97% (160/164) Updating files: 98% (161/164) Updating files: 99% (163/164) Updating files: 100% (164/164) Updating files: 100% (164/164), done. 2022-06-27T12:04:38.607 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at 007313fe844 v0.10 2022-06-27T12:04:38.611 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.10 2022-06-27T12:04:39.101 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:04:39.102 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 11 -eq 48 ']' 2022-06-27T12:04:39.103 INFO:tasks.workunit.client.0.smithi195.stdout:v0.11 2022-06-27T12:04:39.104 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.11 2022-06-27T12:04:39.104 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.11 2022-06-27T12:04:39.105 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.11 2022-06-27T12:04:40.686 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at 9fa6f5e7d09 v0.11 2022-06-27T12:04:40.692 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.11 2022-06-27T12:04:40.829 INFO:tasks.workunit.client.0.smithi195.stdout:v0.12 2022-06-27T12:04:40.830 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:04:40.830 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 12 -eq 48 ']' 2022-06-27T12:04:40.831 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.12 2022-06-27T12:04:40.831 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.12 2022-06-27T12:04:40.832 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.12 2022-06-27T12:04:44.138 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 49% (37/75) Updating files: 50% (38/75) Updating files: 52% (39/75) Updating files: 53% (40/75) Updating files: 54% (41/75) Updating files: 56% (42/75) Updating files: 57% (43/75) Updating files: 58% (44/75) Updating files: 60% (45/75) Updating files: 61% (46/75) Updating files: 62% (47/75) Updating files: 64% (48/75) Updating files: 65% (49/75) Updating files: 66% (50/75) Updating files: 68% (51/75) Updating files: 69% (52/75) Updating files: 70% (53/75) Updating files: 72% (54/75) Updating files: 73% (55/75) Updating files: 74% (56/75) Updating files: 76% (57/75) Updating files: 77% (58/75) Updating files: 78% (59/75) Updating files: 80% (60/75) Updating files: 81% (61/75) Updating files: 82% (62/75) Updating files: 84% (63/75) Updating files: 85% (64/75) Updating files: 86% (65/75) Updating files: 88% (66/75) Updating files: 89% (67/75) Updating files: 90% (68/75) Updating files: 92% (69/75) Updating files: 93% (70/75) Updating files: 94% (71/75) Updating files: 96% (72/75) Updating files: 97% (73/75) Updating files: 98% (74/75) Updating files: 100% (75/75) Updating files: 100% (75/75), done. 2022-06-27T12:04:44.161 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at f524150822a v0.12 2022-06-27T12:04:44.164 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.12 2022-06-27T12:04:44.589 INFO:tasks.workunit.client.0.smithi195.stdout:v0.13 2022-06-27T12:04:44.590 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:04:44.590 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 13 -eq 48 ']' 2022-06-27T12:04:44.591 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.13 2022-06-27T12:04:44.591 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.13 2022-06-27T12:04:44.591 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.13 2022-06-27T12:04:46.201 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 58% (78/133) Updating files: 59% (79/133) Updating files: 60% (80/133) Updating files: 61% (82/133) Updating files: 62% (83/133) Updating files: 63% (84/133) Updating files: 64% (86/133) Updating files: 65% (87/133) Updating files: 66% (88/133) Updating files: 67% (90/133) Updating files: 68% (91/133) Updating files: 69% (92/133) Updating files: 70% (94/133) Updating files: 71% (95/133) Updating files: 72% (96/133) Updating files: 73% (98/133) Updating files: 74% (99/133) Updating files: 75% (100/133) Updating files: 76% (102/133) Updating files: 77% (103/133) Updating files: 78% (104/133) Updating files: 79% (106/133) Updating files: 80% (107/133) Updating files: 81% (108/133) Updating files: 82% (110/133) Updating files: 83% (111/133) Updating files: 84% (112/133) Updating files: 85% (114/133) Updating files: 86% (115/133) Updating files: 87% (116/133) Updating files: 88% (118/133) Updating files: 89% (119/133) Updating files: 90% (120/133) Updating files: 91% (122/133) Updating files: 92% (123/133) Updating files: 93% (124/133) Updating files: 94% (126/133) Updating files: 95% (127/133) Updating files: 96% (128/133) Updating files: 97% (130/133) Updating files: 98% (131/133) Updating files: 99% (132/133) Updating files: 100% (133/133) Updating files: 100% (133/133), done. 2022-06-27T12:04:46.223 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at 2b32297267d v0.13 2022-06-27T12:04:46.226 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.13 2022-06-27T12:04:46.332 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:04:46.332 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 14 -eq 48 ']' 2022-06-27T12:04:46.333 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.14 2022-06-27T12:04:46.333 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.14 2022-06-27T12:04:46.334 INFO:tasks.workunit.client.0.smithi195.stdout:v0.14 2022-06-27T12:04:46.336 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.14 2022-06-27T12:04:48.324 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 85% (129/151) Updating files: 86% (130/151) Updating files: 87% (132/151) Updating files: 88% (133/151) Updating files: 89% (135/151) Updating files: 90% (136/151) Updating files: 91% (138/151) Updating files: 92% (139/151) Updating files: 93% (141/151) Updating files: 94% (142/151) Updating files: 95% (144/151) Updating files: 96% (145/151) Updating files: 97% (147/151) Updating files: 98% (148/151) Updating files: 99% (150/151) Updating files: 100% (151/151) Updating files: 100% (151/151), done. 2022-06-27T12:04:48.352 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at 1ec1e8d6fb6 v0.14 2022-06-27T12:04:48.353 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.14 2022-06-27T12:04:48.559 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:04:48.559 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 15 -eq 48 ']' 2022-06-27T12:04:48.560 INFO:tasks.workunit.client.0.smithi195.stdout:v0.15 2022-06-27T12:04:48.561 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.15 2022-06-27T12:04:48.567 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.15 2022-06-27T12:04:48.568 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.15 2022-06-27T12:04:49.422 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at ab3c421eb95 v0.15 2022-06-27T12:04:49.425 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.15 2022-06-27T12:04:49.578 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:04:49.578 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 16 -eq 48 ']' 2022-06-27T12:04:49.579 INFO:tasks.workunit.client.0.smithi195.stdout:v0.16 2022-06-27T12:04:49.580 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.16 2022-06-27T12:04:49.580 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.16 2022-06-27T12:04:49.580 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.16 2022-06-27T12:04:50.124 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:04:50.259 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.258+0000 7f239c9f7700 1 Processor -- start 2022-06-27T12:04:50.259 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.258+0000 7f239c9f7700 1 -- start start 2022-06-27T12:04:50.264 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.259+0000 7f239c9f7700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f23980c3740 0x7f2398153560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:50.264 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.259+0000 7f239c9f7700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f239806d790 0x7f23980c3b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:50.264 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.259+0000 7f239c9f7700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f239806dba0 0x7f2398156030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:50.265 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.259+0000 7f239c9f7700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f239806a3b0 con 0x7f239806dba0 2022-06-27T12:04:50.265 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.259+0000 7f239c9f7700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23980c4eb0 con 0x7f239806d790 2022-06-27T12:04:50.265 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.259+0000 7f239c9f7700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23980c5150 con 0x7f23980c3740 2022-06-27T12:04:50.265 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.259+0000 7f2396d9d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f239806dba0 0x7f2398156030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:50.266 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.259+0000 7f2396d9d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f239806dba0 0x7f2398156030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47664/0 (socket says 172.21.15.195:47664) 2022-06-27T12:04:50.266 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.259+0000 7f2396d9d700 1 -- 172.21.15.195:0/218902109 learned_addr learned my addr 172.21.15.195:0/218902109 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:04:50.266 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.260+0000 7f2395d9b700 1 --2- 172.21.15.195:0/218902109 >> v2:172.21.15.202:3300/0 conn(0x7f239806d790 0x7f23980c3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:50.266 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.260+0000 7f239659c700 1 --2- 172.21.15.195:0/218902109 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f23980c3740 0x7f2398153560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:50.267 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.260+0000 7f2396d9d700 1 -- 172.21.15.195:0/218902109 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f23980c3740 msgr2=0x7f2398153560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:04:50.267 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.260+0000 7f2396d9d700 1 --2- 172.21.15.195:0/218902109 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f23980c3740 0x7f2398153560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:50.267 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.260+0000 7f2396d9d700 1 -- 172.21.15.195:0/218902109 >> v2:172.21.15.202:3300/0 conn(0x7f239806d790 msgr2=0x7f23980c3b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:04:50.267 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.260+0000 7f2396d9d700 1 --2- 172.21.15.195:0/218902109 >> v2:172.21.15.202:3300/0 conn(0x7f239806d790 0x7f23980c3b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:50.267 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.260+0000 7f2396d9d700 1 -- 172.21.15.195:0/218902109 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23980c53f0 con 0x7f239806dba0 2022-06-27T12:04:50.268 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.260+0000 7f2395d9b700 1 --2- 172.21.15.195:0/218902109 >> v2:172.21.15.202:3300/0 conn(0x7f239806d790 0x7f23980c3b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:04:50.268 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.260+0000 7f2396d9d700 1 --2- 172.21.15.195:0/218902109 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f239806dba0 0x7f2398156030 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f2398069600 tx=0x7f239000a950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a0f8ab7fc7caac2 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:04:50.268 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.260+0000 7f239559a700 1 -- 172.21.15.195:0/218902109 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f239000cdf0 con 0x7f239806dba0 2022-06-27T12:04:50.268 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.260+0000 7f239559a700 1 -- 172.21.15.195:0/218902109 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f239000a0d0 con 0x7f239806dba0 2022-06-27T12:04:50.269 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.260+0000 7f239559a700 1 -- 172.21.15.195:0/218902109 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2390008510 con 0x7f239806dba0 2022-06-27T12:04:50.269 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.260+0000 7f239c9f7700 1 -- 172.21.15.195:0/218902109 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f239806dba0 msgr2=0x7f2398156030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:50.269 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.260+0000 7f239c9f7700 1 --2- 172.21.15.195:0/218902109 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f239806dba0 0x7f2398156030 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f2398069600 tx=0x7f239000a950 comp rx=0 tx=0).stop 2022-06-27T12:04:50.269 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.261+0000 7f239c9f7700 1 -- 172.21.15.195:0/218902109 shutdown_connections 2022-06-27T12:04:50.270 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.261+0000 7f239c9f7700 1 --2- 172.21.15.195:0/218902109 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f239806dba0 0x7f2398156030 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:50.270 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.261+0000 7f239c9f7700 1 --2- 172.21.15.195:0/218902109 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f23980c3740 0x7f2398153560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:50.270 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.261+0000 7f239c9f7700 1 --2- 172.21.15.195:0/218902109 >> v2:172.21.15.202:3300/0 conn(0x7f239806d790 0x7f23980c3b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:50.270 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.261+0000 7f239c9f7700 1 -- 172.21.15.195:0/218902109 >> 172.21.15.195:0/218902109 conn(0x7f2398067ed0 msgr2=0x7f23980682d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:04:50.270 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.261+0000 7f239c9f7700 1 -- 172.21.15.195:0/218902109 shutdown_connections 2022-06-27T12:04:50.271 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.261+0000 7f239c9f7700 1 -- 172.21.15.195:0/218902109 wait complete. 2022-06-27T12:04:50.271 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.262+0000 7f239c9f7700 1 Processor -- start 2022-06-27T12:04:50.271 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.262+0000 7f239c9f7700 1 -- start start 2022-06-27T12:04:50.271 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.262+0000 7f239c9f7700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f239806d790 0x7f239814bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:50.273 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.262+0000 7f239c9f7700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f239806dba0 0x7f23980c2740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:50.274 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.262+0000 7f239c9f7700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f23980c3740 0x7f23980c2c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:50.274 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.262+0000 7f239c9f7700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23980b77f0 con 0x7f239806dba0 2022-06-27T12:04:50.274 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.262+0000 7f239c9f7700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23980c22b0 con 0x7f23980c3740 2022-06-27T12:04:50.275 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.262+0000 7f239c9f7700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2398162850 con 0x7f239806d790 2022-06-27T12:04:50.275 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.263+0000 7f2395d9b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f239806dba0 0x7f23980c2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:50.276 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.263+0000 7f2395d9b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f239806dba0 0x7f23980c2740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47668/0 (socket says 172.21.15.195:47668) 2022-06-27T12:04:50.276 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.263+0000 7f2395d9b700 1 -- 172.21.15.195:0/2718794730 learned_addr learned my addr 172.21.15.195:0/2718794730 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:04:50.276 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.263+0000 7f239659c700 1 --2- 172.21.15.195:0/2718794730 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f239806d790 0x7f239814bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:50.277 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.263+0000 7f2396d9d700 1 --2- 172.21.15.195:0/2718794730 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f23980c3740 0x7f23980c2c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:50.277 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.263+0000 7f2395d9b700 1 -- 172.21.15.195:0/2718794730 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f239806d790 msgr2=0x7f239814bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:50.281 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.263+0000 7f2395d9b700 1 --2- 172.21.15.195:0/2718794730 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f239806d790 0x7f239814bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:50.282 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.263+0000 7f2395d9b700 1 -- 172.21.15.195:0/2718794730 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f23980c3740 msgr2=0x7f23980c2c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:50.282 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.263+0000 7f2395d9b700 1 --2- 172.21.15.195:0/2718794730 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f23980c3740 0x7f23980c2c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:50.283 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.263+0000 7f2395d9b700 1 -- 172.21.15.195:0/2718794730 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f239000abf0 con 0x7f239806dba0 2022-06-27T12:04:50.283 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.263+0000 7f2395d9b700 1 --2- 172.21.15.195:0/2718794730 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f239806dba0 0x7f23980c2740 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f238800d890 tx=0x7f2388008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:04:50.284 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.263+0000 7f23867fc700 1 -- 172.21.15.195:0/2718794730 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23880125f0 con 0x7f239806dba0 2022-06-27T12:04:50.284 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.263+0000 7f23867fc700 1 -- 172.21.15.195:0/2718794730 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2388012750 con 0x7f239806dba0 2022-06-27T12:04:50.284 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.263+0000 7f239c9f7700 1 -- 172.21.15.195:0/2718794730 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2398162b30 con 0x7f239806dba0 2022-06-27T12:04:50.285 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.263+0000 7f23867fc700 1 -- 172.21.15.195:0/2718794730 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2388012b10 con 0x7f239806dba0 2022-06-27T12:04:50.285 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.263+0000 7f239c9f7700 1 -- 172.21.15.195:0/2718794730 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23981630f0 con 0x7f239806dba0 2022-06-27T12:04:50.286 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.264+0000 7f239c9f7700 1 -- 172.21.15.195:0/2718794730 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23980c2440 con 0x7f239806dba0 2022-06-27T12:04:50.289 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.265+0000 7f23867fc700 1 -- 172.21.15.195:0/2718794730 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f238801e460 con 0x7f239806dba0 2022-06-27T12:04:50.289 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.265+0000 7f23867fc700 1 --2- 172.21.15.195:0/2718794730 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2374063d00 0x7f23740661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:50.290 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.265+0000 7f23867fc700 1 -- 172.21.15.195:0/2718794730 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6890+0+0 (secure 0 0 0) 0x7f2388081300 con 0x7f239806dba0 2022-06-27T12:04:50.291 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.265+0000 7f239659c700 1 --2- 172.21.15.195:0/2718794730 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2374063d00 0x7f23740661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:50.291 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.265+0000 7f239659c700 1 --2- 172.21.15.195:0/2718794730 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2374063d00 0x7f23740661b0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f238c006fd0 tx=0x7f238c008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:04:50.292 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.269+0000 7f23867fc700 1 -- 172.21.15.195:0/2718794730 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f238804fb50 con 0x7f239806dba0 2022-06-27T12:04:50.495 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.495+0000 7f239c9f7700 1 -- 172.21.15.195:0/2718794730 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f23980722c0 con 0x7f239806dba0 2022-06-27T12:04:50.496 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.495+0000 7f23867fc700 1 -- 172.21.15.195:0/2718794730 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+6128 (secure 0 0 0) 0x7f2388053170 con 0x7f239806dba0 2022-06-27T12:04:50.496 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:04:50.497 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":5489,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/2342195382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2342195382},{"type":"v1","addr":"172.21.15.202:6835","nonce":2342195382}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:04:12.288201+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5818},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5818":{"gid":5818,"name":"b","rank":0,"incarnation":55,"state":"up:active","state_seq":34,"addr":"172.21.15.202:6833/3399027527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3399027527},{"type":"v1","addr":"172.21.15.202:6833","nonce":3399027527}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:04:50.499 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.498+0000 7f239c9f7700 1 -- 172.21.15.195:0/2718794730 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2374063d00 msgr2=0x7f23740661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:50.499 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.498+0000 7f239c9f7700 1 --2- 172.21.15.195:0/2718794730 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2374063d00 0x7f23740661b0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f238c006fd0 tx=0x7f238c008040 comp rx=0 tx=0).stop 2022-06-27T12:04:50.500 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.498+0000 7f239c9f7700 1 -- 172.21.15.195:0/2718794730 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f239806dba0 msgr2=0x7f23980c2740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:50.501 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.498+0000 7f239c9f7700 1 --2- 172.21.15.195:0/2718794730 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f239806dba0 0x7f23980c2740 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f238800d890 tx=0x7f2388008040 comp rx=0 tx=0).stop 2022-06-27T12:04:50.501 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.498+0000 7f239c9f7700 1 -- 172.21.15.195:0/2718794730 shutdown_connections 2022-06-27T12:04:50.501 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.498+0000 7f239c9f7700 1 --2- 172.21.15.195:0/2718794730 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2374063d00 0x7f23740661b0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:50.501 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.498+0000 7f239c9f7700 1 --2- 172.21.15.195:0/2718794730 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f23980c3740 0x7f23980c2c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:50.502 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.498+0000 7f239c9f7700 1 --2- 172.21.15.195:0/2718794730 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f239806d790 0x7f239814bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:50.502 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.498+0000 7f239c9f7700 1 --2- 172.21.15.195:0/2718794730 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f239806dba0 0x7f23980c2740 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:50.503 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.498+0000 7f239c9f7700 1 -- 172.21.15.195:0/2718794730 >> 172.21.15.195:0/2718794730 conn(0x7f2398067ed0 msgr2=0x7f23980c73d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:04:50.503 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.498+0000 7f239c9f7700 1 -- 172.21.15.195:0/2718794730 shutdown_connections 2022-06-27T12:04:50.504 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.498+0000 7f239c9f7700 1 -- 172.21.15.195:0/2718794730 wait complete. 2022-06-27T12:04:50.504 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 60 2022-06-27T12:04:50.512 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2022-06-27T12:04:50.513 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-06-27T12:04:50.513 INFO:teuthology.orchestra.run:waiting for 300 2022-06-27T12:04:50.528 INFO:tasks.ceph.mds.b:Stopped 2022-06-27T12:04:50.529 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:04:50.657 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.657+0000 7f872cd9d700 1 Processor -- start 2022-06-27T12:04:50.658 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.657+0000 7f872cd9d700 1 -- start start 2022-06-27T12:04:50.665 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.658+0000 7f872cd9d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f872814e490 0x7f872806cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:50.665 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.658+0000 7f872cd9d700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f87280c3f50 0x7f8728157890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:50.665 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.658+0000 7f872cd9d700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87280c33b0 con 0x7f872814e860 2022-06-27T12:04:50.666 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.658+0000 7f872cd9d700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f872806d520 con 0x7f872814e490 2022-06-27T12:04:50.666 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.658+0000 7f872cd9d700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f872806f220 con 0x7f87280c3f50 2022-06-27T12:04:50.667 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.658+0000 7f872659c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f872814e490 0x7f872806cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:50.667 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.658+0000 7f8725d9b700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f872814e860 0x7f87280c3840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:53862/0 (socket says 172.21.15.195:53862) 2022-06-27T12:04:50.667 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.658+0000 7f8725d9b700 1 -- 172.21.15.195:0/1355755509 learned_addr learned my addr 172.21.15.195:0/1355755509 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:04:50.667 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.658+0000 7f8726d9d700 1 --2- 172.21.15.195:0/1355755509 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f87280c3f50 0x7f8728157890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:50.668 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.658+0000 7f872559a700 1 -- 172.21.15.195:0/1355755509 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2277265551 0 0) 0x7f87280c33b0 con 0x7f872814e860 2022-06-27T12:04:50.668 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.658+0000 7f872559a700 1 -- 172.21.15.195:0/1355755509 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f870c003430 con 0x7f872814e860 2022-06-27T12:04:50.668 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.658+0000 7f872659c700 1 -- 172.21.15.195:0/1355755509 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f87280c3f50 msgr2=0x7f8728157890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:04:50.668 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.658+0000 7f872659c700 1 --2- 172.21.15.195:0/1355755509 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f87280c3f50 0x7f8728157890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:50.668 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.658+0000 7f872659c700 1 -- 172.21.15.195:0/1355755509 >> v1:172.21.15.195:6789/0 conn(0x7f872814e860 legacy=0x7f87280c3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:50.669 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.658+0000 7f872659c700 1 -- 172.21.15.195:0/1355755509 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87281542c0 con 0x7f872814e490 2022-06-27T12:04:50.669 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.658+0000 7f8726d9d700 1 --2- 172.21.15.195:0/1355755509 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f87280c3f50 0x7f8728157890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:04:50.669 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.658+0000 7f872659c700 1 --2- 172.21.15.195:0/1355755509 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f872814e490 0x7f872806cfe0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f87180097a0 tx=0x7f8718009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e715984bd930677e server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:04:50.669 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.659+0000 7f872559a700 1 -- 172.21.15.195:0/1355755509 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8718011630 con 0x7f872814e490 2022-06-27T12:04:50.670 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.659+0000 7f872559a700 1 -- 172.21.15.195:0/1355755509 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8718011790 con 0x7f872814e490 2022-06-27T12:04:50.670 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.659+0000 7f872559a700 1 -- 172.21.15.195:0/1355755509 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8718011b30 con 0x7f872814e490 2022-06-27T12:04:50.670 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.659+0000 7f872cd9d700 1 -- 172.21.15.195:0/1355755509 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f872814e490 msgr2=0x7f872806cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:50.670 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.659+0000 7f872cd9d700 1 --2- 172.21.15.195:0/1355755509 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f872814e490 0x7f872806cfe0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f87180097a0 tx=0x7f8718009060 comp rx=0 tx=0).stop 2022-06-27T12:04:50.671 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.659+0000 7f872cd9d700 1 -- 172.21.15.195:0/1355755509 shutdown_connections 2022-06-27T12:04:50.671 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.659+0000 7f872cd9d700 1 --2- 172.21.15.195:0/1355755509 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f87280c3f50 0x7f8728157890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:50.671 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.659+0000 7f872cd9d700 1 --2- 172.21.15.195:0/1355755509 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f872814e490 0x7f872806cfe0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:50.671 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.659+0000 7f872cd9d700 1 -- 172.21.15.195:0/1355755509 >> 172.21.15.195:0/1355755509 conn(0x7f8728063af0 msgr2=0x7f8728063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:04:50.672 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.659+0000 7f872cd9d700 1 -- 172.21.15.195:0/1355755509 shutdown_connections 2022-06-27T12:04:50.672 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.659+0000 7f872cd9d700 1 -- 172.21.15.195:0/1355755509 wait complete. 2022-06-27T12:04:50.672 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.660+0000 7f872cd9d700 1 Processor -- start 2022-06-27T12:04:50.672 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.660+0000 7f872cd9d700 1 -- start start 2022-06-27T12:04:50.673 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.661+0000 7f872cd9d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f87280c3f50 0x7f8728148cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:50.673 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.661+0000 7f872cd9d700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f872814e490 0x7f8728149230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:50.673 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.661+0000 7f872cd9d700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f872814e860 0x7f872816a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:50.673 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.661+0000 7f872cd9d700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87280c72f0 con 0x7f87280c3f50 2022-06-27T12:04:50.673 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.661+0000 7f872cd9d700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8728150f50 con 0x7f872814e860 2022-06-27T12:04:50.674 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.661+0000 7f872cd9d700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87280c6d00 con 0x7f872814e490 2022-06-27T12:04:50.674 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.661+0000 7f872659c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f87280c3f50 0x7f8728148cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:50.674 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.661+0000 7f872659c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f87280c3f50 0x7f8728148cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47680/0 (socket says 172.21.15.195:47680) 2022-06-27T12:04:50.675 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.661+0000 7f872659c700 1 -- 172.21.15.195:0/3092442926 learned_addr learned my addr 172.21.15.195:0/3092442926 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:04:50.675 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.661+0000 7f8725d9b700 1 --2- 172.21.15.195:0/3092442926 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f872814e490 0x7f8728149230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:50.675 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.661+0000 7f8726d9d700 1 --2- 172.21.15.195:0/3092442926 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f872814e860 0x7f872816a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:50.675 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.661+0000 7f872659c700 1 -- 172.21.15.195:0/3092442926 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f872814e490 msgr2=0x7f8728149230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:50.675 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.661+0000 7f872659c700 1 --2- 172.21.15.195:0/3092442926 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f872814e490 0x7f8728149230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:50.676 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.661+0000 7f872659c700 1 -- 172.21.15.195:0/3092442926 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f872814e860 msgr2=0x7f872816a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:50.676 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.661+0000 7f872659c700 1 --2- 172.21.15.195:0/3092442926 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f872814e860 0x7f872816a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:50.676 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.661+0000 7f872659c700 1 -- 172.21.15.195:0/3092442926 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87180092e0 con 0x7f87280c3f50 2022-06-27T12:04:50.676 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.661+0000 7f8726d9d700 1 --2- 172.21.15.195:0/3092442926 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f872814e860 0x7f872816a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:04:50.677 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.662+0000 7f872659c700 1 --2- 172.21.15.195:0/3092442926 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f87280c3f50 0x7f8728148cf0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f8718009e40 tx=0x7f8718017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:04:50.677 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.662+0000 7f87167fc700 1 -- 172.21.15.195:0/3092442926 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87180045b0 con 0x7f87280c3f50 2022-06-27T12:04:50.677 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.662+0000 7f87167fc700 1 -- 172.21.15.195:0/3092442926 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f871801be00 con 0x7f87280c3f50 2022-06-27T12:04:50.677 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.662+0000 7f872cd9d700 1 -- 172.21.15.195:0/3092442926 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f872816aeb0 con 0x7f87280c3f50 2022-06-27T12:04:50.678 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.662+0000 7f87167fc700 1 -- 172.21.15.195:0/3092442926 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8718011b30 con 0x7f87280c3f50 2022-06-27T12:04:50.678 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.662+0000 7f872cd9d700 1 -- 172.21.15.195:0/3092442926 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f872806ade0 con 0x7f87280c3f50 2022-06-27T12:04:50.678 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.663+0000 7f872cd9d700 1 -- 172.21.15.195:0/3092442926 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f872806af70 con 0x7f87280c3f50 2022-06-27T12:04:50.678 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.663+0000 7f87167fc700 1 -- 172.21.15.195:0/3092442926 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f871802c030 con 0x7f87280c3f50 2022-06-27T12:04:50.679 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.663+0000 7f87167fc700 1 --2- 172.21.15.195:0/3092442926 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8704063d30 0x7f87040661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:50.679 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.663+0000 7f87167fc700 1 -- 172.21.15.195:0/3092442926 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6890+0+0 (secure 0 0 0) 0x7f8718081fa0 con 0x7f87280c3f50 2022-06-27T12:04:50.679 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.664+0000 7f8725d9b700 1 --2- 172.21.15.195:0/3092442926 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8704063d30 0x7f87040661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:50.680 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.664+0000 7f8725d9b700 1 --2- 172.21.15.195:0/3092442926 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8704063d30 0x7f87040661e0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f8728064b20 tx=0x7f871c009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:04:50.681 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.668+0000 7f87167fc700 1 -- 172.21.15.195:0/3092442926 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f872806af70 con 0x7f87280c3f50 2022-06-27T12:04:50.893 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.893+0000 7f872cd9d700 1 -- 172.21.15.195:0/3092442926 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87280c7430 con 0x7f87280c3f50 2022-06-27T12:04:50.894 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.893+0000 7f87167fc700 1 -- 172.21.15.195:0/3092442926 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+6128 (secure 0 0 0) 0x7f87280c7430 con 0x7f87280c3f50 2022-06-27T12:04:50.895 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:04:50.897 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":5489,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/2342195382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2342195382},{"type":"v1","addr":"172.21.15.202:6835","nonce":2342195382}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:04:12.288201+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5818},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5818":{"gid":5818,"name":"b","rank":0,"incarnation":55,"state":"up:active","state_seq":34,"addr":"172.21.15.202:6833/3399027527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3399027527},{"type":"v1","addr":"172.21.15.202:6833","nonce":3399027527}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:04:50.899 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.895+0000 7f872cd9d700 1 -- 172.21.15.195:0/3092442926 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8704063d30 msgr2=0x7f87040661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:50.899 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.895+0000 7f872cd9d700 1 --2- 172.21.15.195:0/3092442926 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8704063d30 0x7f87040661e0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f8728064b20 tx=0x7f871c009040 comp rx=0 tx=0).stop 2022-06-27T12:04:50.900 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.895+0000 7f872cd9d700 1 -- 172.21.15.195:0/3092442926 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f87280c3f50 msgr2=0x7f8728148cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:50.900 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.895+0000 7f872cd9d700 1 --2- 172.21.15.195:0/3092442926 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f87280c3f50 0x7f8728148cf0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f8718009e40 tx=0x7f8718017000 comp rx=0 tx=0).stop 2022-06-27T12:04:50.901 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.895+0000 7f872cd9d700 1 -- 172.21.15.195:0/3092442926 shutdown_connections 2022-06-27T12:04:50.901 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.895+0000 7f872cd9d700 1 --2- 172.21.15.195:0/3092442926 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8704063d30 0x7f87040661e0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:50.901 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.895+0000 7f872cd9d700 1 --2- 172.21.15.195:0/3092442926 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f87280c3f50 0x7f8728148cf0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:50.902 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.895+0000 7f872cd9d700 1 --2- 172.21.15.195:0/3092442926 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f872814e860 0x7f872816a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:50.902 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.895+0000 7f872cd9d700 1 --2- 172.21.15.195:0/3092442926 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f872814e490 0x7f8728149230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:50.903 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.895+0000 7f872cd9d700 1 -- 172.21.15.195:0/3092442926 >> 172.21.15.195:0/3092442926 conn(0x7f8728063af0 msgr2=0x7f872814d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:04:50.904 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.896+0000 7f872cd9d700 1 -- 172.21.15.195:0/3092442926 shutdown_connections 2022-06-27T12:04:50.905 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:50.896+0000 7f872cd9d700 1 -- 172.21.15.195:0/3092442926 wait complete. 2022-06-27T12:04:50.905 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 60 2022-06-27T12:04:50.910 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2022-06-27T12:04:52.912 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:04:53.045 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.045+0000 7f13b0aaa700 1 Processor -- start 2022-06-27T12:04:53.045 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.045+0000 7f13b0aaa700 1 -- start start 2022-06-27T12:04:53.051 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.046+0000 7f13b0aaa700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f13ac150500 0x7f13ac063af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:53.051 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.046+0000 7f13b0aaa700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f13ac06a450 0x7f13ac0c48d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:53.051 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.046+0000 7f13b0aaa700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13ac064030 con 0x7f13ac1508d0 2022-06-27T12:04:53.052 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.046+0000 7f13b0aaa700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13ac0c4160 con 0x7f13ac150500 2022-06-27T12:04:53.052 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.046+0000 7f13b0aaa700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13ac0c4440 con 0x7f13ac06a450 2022-06-27T12:04:53.052 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.046+0000 7f13aa59c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f13ac150500 0x7f13ac063af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:53.052 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.046+0000 7f13aa59c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f13ac150500 0x7f13ac063af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47694/0 (socket says 172.21.15.195:47694) 2022-06-27T12:04:53.053 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.046+0000 7f13aa59c700 1 -- 172.21.15.195:0/3886103861 learned_addr learned my addr 172.21.15.195:0/3886103861 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:04:53.053 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.046+0000 7f13a9d9b700 1 --2- 172.21.15.195:0/3886103861 >> v2:172.21.15.202:3300/0 conn(0x7f13ac06a450 0x7f13ac0c48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:53.053 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.047+0000 7f13aa59c700 1 -- 172.21.15.195:0/3886103861 >> v2:172.21.15.202:3300/0 conn(0x7f13ac06a450 msgr2=0x7f13ac0c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:04:53.053 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.047+0000 7f13aa59c700 1 --2- 172.21.15.195:0/3886103861 >> v2:172.21.15.202:3300/0 conn(0x7f13ac06a450 0x7f13ac0c48d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:53.054 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.047+0000 7f13aa59c700 1 -- 172.21.15.195:0/3886103861 >> v1:172.21.15.202:6789/0 conn(0x7f13ac1508d0 legacy=0x7f13ac0bbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:04:53.054 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.047+0000 7f13aa59c700 1 -- 172.21.15.195:0/3886103861 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13ac15a800 con 0x7f13ac150500 2022-06-27T12:04:53.054 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.047+0000 7f13aa59c700 1 --2- 172.21.15.195:0/3886103861 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f13ac150500 0x7f13ac063af0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f13ac06ce70 tx=0x7f13a00090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43c99cf1c3983e70 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:04:53.054 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.047+0000 7f13a959a700 1 -- 172.21.15.195:0/3886103861 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a0005ad0 con 0x7f13ac150500 2022-06-27T12:04:53.055 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.047+0000 7f13a959a700 1 -- 172.21.15.195:0/3886103861 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13a0005c30 con 0x7f13ac150500 2022-06-27T12:04:53.055 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.047+0000 7f13a959a700 1 -- 172.21.15.195:0/3886103861 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a0006030 con 0x7f13ac150500 2022-06-27T12:04:53.055 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.047+0000 7f13b0aaa700 1 -- 172.21.15.195:0/3886103861 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f13ac150500 msgr2=0x7f13ac063af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:53.055 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.047+0000 7f13b0aaa700 1 --2- 172.21.15.195:0/3886103861 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f13ac150500 0x7f13ac063af0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f13ac06ce70 tx=0x7f13a00090f0 comp rx=0 tx=0).stop 2022-06-27T12:04:53.056 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.047+0000 7f13b0aaa700 1 -- 172.21.15.195:0/3886103861 shutdown_connections 2022-06-27T12:04:53.056 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.047+0000 7f13b0aaa700 1 --2- 172.21.15.195:0/3886103861 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f13ac150500 0x7f13ac063af0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:53.056 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.047+0000 7f13b0aaa700 1 --2- 172.21.15.195:0/3886103861 >> v2:172.21.15.202:3300/0 conn(0x7f13ac06a450 0x7f13ac0c48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:53.056 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.047+0000 7f13b0aaa700 1 -- 172.21.15.195:0/3886103861 >> 172.21.15.195:0/3886103861 conn(0x7f13ac0c9b00 msgr2=0x7f13ac0c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:04:53.057 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.047+0000 7f13b0aaa700 1 -- 172.21.15.195:0/3886103861 shutdown_connections 2022-06-27T12:04:53.057 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.048+0000 7f13b0aaa700 1 -- 172.21.15.195:0/3886103861 wait complete. 2022-06-27T12:04:53.058 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.049+0000 7f13b0aaa700 1 Processor -- start 2022-06-27T12:04:53.058 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.049+0000 7f13b0aaa700 1 -- start start 2022-06-27T12:04:53.058 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.049+0000 7f13b0aaa700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f13ac06a450 0x7f13ac1669f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:53.058 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.049+0000 7f13b0aaa700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f13ac150500 0x7f13ac15ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:53.059 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.049+0000 7f13b0aaa700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f13ac1508d0 0x7f13ac14c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:53.059 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.049+0000 7f13b0aaa700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13ac0c5e80 con 0x7f13ac06a450 2022-06-27T12:04:53.059 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.049+0000 7f13b0aaa700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13ac1665d0 con 0x7f13ac150500 2022-06-27T12:04:53.059 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.049+0000 7f13b0aaa700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13ac0c5990 con 0x7f13ac1508d0 2022-06-27T12:04:53.060 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.049+0000 7f13aa59c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f13ac06a450 0x7f13ac1669f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:53.060 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.049+0000 7f13aa59c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f13ac06a450 0x7f13ac1669f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47698/0 (socket says 172.21.15.195:47698) 2022-06-27T12:04:53.060 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.049+0000 7f13aa59c700 1 -- 172.21.15.195:0/4095259386 learned_addr learned my addr 172.21.15.195:0/4095259386 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:04:53.061 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.050+0000 7f13a9d9b700 1 --2- 172.21.15.195:0/4095259386 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f13ac150500 0x7f13ac15ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:53.061 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.050+0000 7f13aa59c700 1 -- 172.21.15.195:0/4095259386 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f13ac1508d0 msgr2=0x7f13ac14c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:53.061 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.050+0000 7f13aa59c700 1 --2- 172.21.15.195:0/4095259386 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f13ac1508d0 0x7f13ac14c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:53.061 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.050+0000 7f13aa59c700 1 -- 172.21.15.195:0/4095259386 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f13ac150500 msgr2=0x7f13ac15ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:53.062 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.050+0000 7f13aa59c700 1 --2- 172.21.15.195:0/4095259386 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f13ac150500 0x7f13ac15ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:53.062 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.050+0000 7f13aa59c700 1 -- 172.21.15.195:0/4095259386 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13a0009350 con 0x7f13ac06a450 2022-06-27T12:04:53.062 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.050+0000 7f13a9d9b700 1 --2- 172.21.15.195:0/4095259386 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f13ac150500 0x7f13ac15ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T12:04:53.063 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.050+0000 7f13aa59c700 1 --2- 172.21.15.195:0/4095259386 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f13ac06a450 0x7f13ac1669f0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f13a0006900 tx=0x7f13a0016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:04:53.063 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.050+0000 7f139a7fc700 1 -- 172.21.15.195:0/4095259386 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a001d880 con 0x7f13ac06a450 2022-06-27T12:04:53.063 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.050+0000 7f139a7fc700 1 -- 172.21.15.195:0/4095259386 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13a001da20 con 0x7f13ac06a450 2022-06-27T12:04:53.068 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.050+0000 7f13b0aaa700 1 -- 172.21.15.195:0/4095259386 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13ac166f30 con 0x7f13ac06a450 2022-06-27T12:04:53.069 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.050+0000 7f139a7fc700 1 -- 172.21.15.195:0/4095259386 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a001dde0 con 0x7f13ac06a450 2022-06-27T12:04:53.069 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.050+0000 7f13b0aaa700 1 -- 172.21.15.195:0/4095259386 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13ac16a270 con 0x7f13ac06a450 2022-06-27T12:04:53.072 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.051+0000 7f13b0aaa700 1 -- 172.21.15.195:0/4095259386 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13ac0722c0 con 0x7f13ac06a450 2022-06-27T12:04:53.072 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.051+0000 7f139a7fc700 1 -- 172.21.15.195:0/4095259386 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f13a002c030 con 0x7f13ac06a450 2022-06-27T12:04:53.073 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.052+0000 7f139a7fc700 1 --2- 172.21.15.195:0/4095259386 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1388063d00 0x7f13880661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:53.073 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.052+0000 7f139a7fc700 1 -- 172.21.15.195:0/4095259386 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6890+0+0 (secure 0 0 0) 0x7f13a0082810 con 0x7f13ac06a450 2022-06-27T12:04:53.074 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.052+0000 7f13a9d9b700 1 --2- 172.21.15.195:0/4095259386 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1388063d00 0x7f13880661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:53.074 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.052+0000 7f13a9d9b700 1 --2- 172.21.15.195:0/4095259386 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1388063d00 0x7f13880661b0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f13ac157760 tx=0x7f139c008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:04:53.075 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.056+0000 7f139a7fc700 1 -- 172.21.15.195:0/4095259386 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f13a00510c0 con 0x7f13ac06a450 2022-06-27T12:04:53.295 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.294+0000 7f13b0aaa700 1 -- 172.21.15.195:0/4095259386 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f13ac064030 con 0x7f13ac06a450 2022-06-27T12:04:53.295 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.295+0000 7f139a7fc700 1 -- 172.21.15.195:0/4095259386 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+6128 (secure 0 0 0) 0x7f13ac064030 con 0x7f13ac06a450 2022-06-27T12:04:53.297 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:04:53.297 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":5489,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/2342195382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2342195382},{"type":"v1","addr":"172.21.15.202:6835","nonce":2342195382}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:04:12.288201+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5818},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5818":{"gid":5818,"name":"b","rank":0,"incarnation":55,"state":"up:active","state_seq":34,"addr":"172.21.15.202:6833/3399027527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3399027527},{"type":"v1","addr":"172.21.15.202:6833","nonce":3399027527}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:04:53.300 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.296+0000 7f13b0aaa700 1 -- 172.21.15.195:0/4095259386 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1388063d00 msgr2=0x7f13880661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:53.300 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.296+0000 7f13b0aaa700 1 --2- 172.21.15.195:0/4095259386 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1388063d00 0x7f13880661b0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f13ac157760 tx=0x7f139c008040 comp rx=0 tx=0).stop 2022-06-27T12:04:53.300 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.296+0000 7f13b0aaa700 1 -- 172.21.15.195:0/4095259386 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f13ac06a450 msgr2=0x7f13ac1669f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:53.301 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.296+0000 7f13b0aaa700 1 --2- 172.21.15.195:0/4095259386 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f13ac06a450 0x7f13ac1669f0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f13a0006900 tx=0x7f13a0016000 comp rx=0 tx=0).stop 2022-06-27T12:04:53.301 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.297+0000 7f13b0aaa700 1 -- 172.21.15.195:0/4095259386 shutdown_connections 2022-06-27T12:04:53.302 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.297+0000 7f13b0aaa700 1 --2- 172.21.15.195:0/4095259386 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1388063d00 0x7f13880661b0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:53.302 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.297+0000 7f13b0aaa700 1 --2- 172.21.15.195:0/4095259386 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f13ac06a450 0x7f13ac1669f0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:53.303 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.297+0000 7f13b0aaa700 1 --2- 172.21.15.195:0/4095259386 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f13ac1508d0 0x7f13ac14c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:53.303 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.297+0000 7f13b0aaa700 1 --2- 172.21.15.195:0/4095259386 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f13ac150500 0x7f13ac15ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:53.303 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.297+0000 7f13b0aaa700 1 -- 172.21.15.195:0/4095259386 >> 172.21.15.195:0/4095259386 conn(0x7f13ac0c9b00 msgr2=0x7f13ac14fb60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:04:53.303 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.297+0000 7f13b0aaa700 1 -- 172.21.15.195:0/4095259386 shutdown_connections 2022-06-27T12:04:53.304 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:53.297+0000 7f13b0aaa700 1 -- 172.21.15.195:0/4095259386 wait complete. 2022-06-27T12:04:53.304 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 60 2022-06-27T12:04:53.309 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2022-06-27T12:04:55.310 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:04:55.479 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.443+0000 7fdaefbc7700 1 Processor -- start 2022-06-27T12:04:55.479 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.444+0000 7fdaefbc7700 1 -- start start 2022-06-27T12:04:55.479 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.444+0000 7fdaefbc7700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fdae8156100 0x7fdae80688a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:55.480 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.444+0000 7fdaefbc7700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fdae806a440 0x7fdae80c4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:55.480 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.444+0000 7fdaefbc7700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdae80c4b10 con 0x7fdae81564d0 2022-06-27T12:04:55.480 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.444+0000 7fdaefbc7700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdae80c3dc0 con 0x7fdae8156100 2022-06-27T12:04:55.480 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.444+0000 7fdaefbc7700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdae8068f20 con 0x7fdae806a440 2022-06-27T12:04:55.481 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.445+0000 7fdaed963700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fdae8156100 0x7fdae80688a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:55.481 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.445+0000 7fdaed963700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fdae8156100 0x7fdae80688a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47714/0 (socket says 172.21.15.195:47714) 2022-06-27T12:04:55.481 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.445+0000 7fdaed963700 1 -- 172.21.15.195:0/3888654805 learned_addr learned my addr 172.21.15.195:0/3888654805 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:04:55.481 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.445+0000 7fdaed162700 1 --2- 172.21.15.195:0/3888654805 >> v2:172.21.15.202:3300/0 conn(0x7fdae806a440 0x7fdae80c4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:55.481 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.445+0000 7fdaed963700 1 -- 172.21.15.195:0/3888654805 >> v2:172.21.15.202:3300/0 conn(0x7fdae806a440 msgr2=0x7fdae80c4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:04:55.482 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.445+0000 7fdaed963700 1 --2- 172.21.15.195:0/3888654805 >> v2:172.21.15.202:3300/0 conn(0x7fdae806a440 0x7fdae80c4fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:55.482 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.445+0000 7fdaed963700 1 -- 172.21.15.195:0/3888654805 >> v1:172.21.15.202:6789/0 conn(0x7fdae81564d0 legacy=0x7fdae80c28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:04:55.482 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.445+0000 7fdaed963700 1 -- 172.21.15.195:0/3888654805 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdae80c6360 con 0x7fdae8156100 2022-06-27T12:04:55.482 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.445+0000 7fdaed963700 1 --2- 172.21.15.195:0/3888654805 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdae8156100 0x7fdae80688a0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fdae8072140 tx=0x7fdad8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d01e9b37d2519da server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:04:55.483 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.445+0000 7fdaec961700 1 -- 172.21.15.195:0/3888654805 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdad800e070 con 0x7fdae8156100 2022-06-27T12:04:55.483 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.445+0000 7fdaec961700 1 -- 172.21.15.195:0/3888654805 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdad800ba90 con 0x7fdae8156100 2022-06-27T12:04:55.483 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.445+0000 7fdaec961700 1 -- 172.21.15.195:0/3888654805 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdad80092e0 con 0x7fdae8156100 2022-06-27T12:04:55.483 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.446+0000 7fdaefbc7700 1 -- 172.21.15.195:0/3888654805 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdae8156100 msgr2=0x7fdae80688a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:55.484 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.446+0000 7fdaefbc7700 1 --2- 172.21.15.195:0/3888654805 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdae8156100 0x7fdae80688a0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fdae8072140 tx=0x7fdad8009840 comp rx=0 tx=0).stop 2022-06-27T12:04:55.484 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.446+0000 7fdaefbc7700 1 -- 172.21.15.195:0/3888654805 shutdown_connections 2022-06-27T12:04:55.484 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.446+0000 7fdaefbc7700 1 --2- 172.21.15.195:0/3888654805 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdae8156100 0x7fdae80688a0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:55.484 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.446+0000 7fdaefbc7700 1 --2- 172.21.15.195:0/3888654805 >> v2:172.21.15.202:3300/0 conn(0x7fdae806a440 0x7fdae80c4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:55.484 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.446+0000 7fdaefbc7700 1 -- 172.21.15.195:0/3888654805 >> 172.21.15.195:0/3888654805 conn(0x7fdae80c9b00 msgr2=0x7fdae80c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:04:55.485 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.446+0000 7fdaefbc7700 1 -- 172.21.15.195:0/3888654805 shutdown_connections 2022-06-27T12:04:55.485 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.446+0000 7fdaefbc7700 1 -- 172.21.15.195:0/3888654805 wait complete. 2022-06-27T12:04:55.485 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.447+0000 7fdaefbc7700 1 Processor -- start 2022-06-27T12:04:55.485 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.447+0000 7fdaefbc7700 1 -- start start 2022-06-27T12:04:55.486 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.448+0000 7fdaefbc7700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fdae806a440 0x7fdae80c8a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:55.486 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.448+0000 7fdaefbc7700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdae8156100 0x7fdae8164340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:55.486 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.448+0000 7fdaefbc7700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fdae81564d0 0x7fdae8144440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:55.486 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.448+0000 7fdaefbc7700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdae80c87b0 con 0x7fdae8156100 2022-06-27T12:04:55.487 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.448+0000 7fdaefbc7700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdae8168060 con 0x7fdae806a440 2022-06-27T12:04:55.487 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.448+0000 7fdaefbc7700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdae80c8250 con 0x7fdae81564d0 2022-06-27T12:04:55.487 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.448+0000 7fdaed162700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdae8156100 0x7fdae8164340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:55.487 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.448+0000 7fdaed162700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdae8156100 0x7fdae8164340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47722/0 (socket says 172.21.15.195:47722) 2022-06-27T12:04:55.488 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.448+0000 7fdaed162700 1 -- 172.21.15.195:0/1073727975 learned_addr learned my addr 172.21.15.195:0/1073727975 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:04:55.488 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.448+0000 7fdaed963700 1 --2- 172.21.15.195:0/1073727975 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fdae806a440 0x7fdae80c8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:55.488 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.448+0000 7fdaee164700 1 --2- 172.21.15.195:0/1073727975 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fdae81564d0 0x7fdae8144440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:55.488 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.448+0000 7fdaed162700 1 -- 172.21.15.195:0/1073727975 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fdae81564d0 msgr2=0x7fdae8144440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:55.489 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.448+0000 7fdaed162700 1 --2- 172.21.15.195:0/1073727975 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fdae81564d0 0x7fdae8144440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:55.489 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.448+0000 7fdaed162700 1 -- 172.21.15.195:0/1073727975 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fdae806a440 msgr2=0x7fdae80c8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:55.489 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.448+0000 7fdaed162700 1 --2- 172.21.15.195:0/1073727975 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fdae806a440 0x7fdae80c8a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:55.489 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.448+0000 7fdaed162700 1 -- 172.21.15.195:0/1073727975 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdad8009ae0 con 0x7fdae8156100 2022-06-27T12:04:55.489 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.448+0000 7fdaee164700 1 --2- 172.21.15.195:0/1073727975 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fdae81564d0 0x7fdae8144440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:04:55.490 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.448+0000 7fdaed162700 1 --2- 172.21.15.195:0/1073727975 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdae8156100 0x7fdae8164340 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fdadc006dc0 tx=0x7fdadc008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:04:55.490 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.449+0000 7fdae5ffb700 1 -- 172.21.15.195:0/1073727975 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdadc0125f0 con 0x7fdae8156100 2022-06-27T12:04:55.490 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.449+0000 7fdae5ffb700 1 -- 172.21.15.195:0/1073727975 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdadc012750 con 0x7fdae8156100 2022-06-27T12:04:55.490 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.449+0000 7fdaefbc7700 1 -- 172.21.15.195:0/1073727975 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdae80c84f0 con 0x7fdae8156100 2022-06-27T12:04:55.494 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.449+0000 7fdae5ffb700 1 -- 172.21.15.195:0/1073727975 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdadc012af0 con 0x7fdae8156100 2022-06-27T12:04:55.494 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.449+0000 7fdaefbc7700 1 -- 172.21.15.195:0/1073727975 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdae8167cf0 con 0x7fdae8156100 2022-06-27T12:04:55.495 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.450+0000 7fdaefbc7700 1 -- 172.21.15.195:0/1073727975 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdae81681d0 con 0x7fdae8156100 2022-06-27T12:04:55.495 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.450+0000 7fdae5ffb700 1 -- 172.21.15.195:0/1073727975 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fdadc01e460 con 0x7fdae8156100 2022-06-27T12:04:55.496 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.450+0000 7fdae5ffb700 1 --2- 172.21.15.195:0/1073727975 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fdac8063d00 0x7fdac80661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:55.496 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.451+0000 7fdae5ffb700 1 -- 172.21.15.195:0/1073727975 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6890+0+0 (secure 0 0 0) 0x7fdadc081110 con 0x7fdae8156100 2022-06-27T12:04:55.497 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.451+0000 7fdaed963700 1 --2- 172.21.15.195:0/1073727975 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fdac8063d00 0x7fdac80661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:55.497 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.451+0000 7fdaed963700 1 --2- 172.21.15.195:0/1073727975 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fdac8063d00 0x7fdac80661b0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fdad8009fd0 tx=0x7fdad8017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:04:55.498 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.455+0000 7fdae5ffb700 1 -- 172.21.15.195:0/1073727975 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fdadc04f960 con 0x7fdae8156100 2022-06-27T12:04:55.688 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.682+0000 7fdaefbc7700 1 -- 172.21.15.195:0/1073727975 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdae8167e80 con 0x7fdae8156100 2022-06-27T12:04:55.688 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.682+0000 7fdae5ffb700 1 -- 172.21.15.195:0/1073727975 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+6128 (secure 0 0 0) 0x7fdae8167e80 con 0x7fdae8156100 2022-06-27T12:04:55.689 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:04:55.690 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":5489,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/2342195382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2342195382},{"type":"v1","addr":"172.21.15.202:6835","nonce":2342195382}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:04:12.288201+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5818},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5818":{"gid":5818,"name":"b","rank":0,"incarnation":55,"state":"up:active","state_seq":34,"addr":"172.21.15.202:6833/3399027527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3399027527},{"type":"v1","addr":"172.21.15.202:6833","nonce":3399027527}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:04:55.691 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.684+0000 7fdaefbc7700 1 -- 172.21.15.195:0/1073727975 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fdac8063d00 msgr2=0x7fdac80661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:55.692 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.684+0000 7fdaefbc7700 1 --2- 172.21.15.195:0/1073727975 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fdac8063d00 0x7fdac80661b0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fdad8009fd0 tx=0x7fdad8017000 comp rx=0 tx=0).stop 2022-06-27T12:04:55.692 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.684+0000 7fdaefbc7700 1 -- 172.21.15.195:0/1073727975 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdae8156100 msgr2=0x7fdae8164340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:55.692 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.684+0000 7fdaefbc7700 1 --2- 172.21.15.195:0/1073727975 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdae8156100 0x7fdae8164340 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fdadc006dc0 tx=0x7fdadc008040 comp rx=0 tx=0).stop 2022-06-27T12:04:55.693 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.684+0000 7fdaefbc7700 1 -- 172.21.15.195:0/1073727975 shutdown_connections 2022-06-27T12:04:55.693 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.684+0000 7fdaefbc7700 1 --2- 172.21.15.195:0/1073727975 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fdac8063d00 0x7fdac80661b0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:55.693 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.684+0000 7fdaefbc7700 1 --2- 172.21.15.195:0/1073727975 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fdae81564d0 0x7fdae8144440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:55.699 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.684+0000 7fdaefbc7700 1 --2- 172.21.15.195:0/1073727975 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fdae806a440 0x7fdae80c8a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:55.699 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.684+0000 7fdaefbc7700 1 --2- 172.21.15.195:0/1073727975 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdae8156100 0x7fdae8164340 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:55.699 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.684+0000 7fdaefbc7700 1 -- 172.21.15.195:0/1073727975 >> 172.21.15.195:0/1073727975 conn(0x7fdae80c9b00 msgr2=0x7fdae814aba0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:04:55.700 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.684+0000 7fdaefbc7700 1 -- 172.21.15.195:0/1073727975 shutdown_connections 2022-06-27T12:04:55.700 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:55.684+0000 7fdaefbc7700 1 -- 172.21.15.195:0/1073727975 wait complete. 2022-06-27T12:04:55.702 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 60 2022-06-27T12:04:55.704 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2022-06-27T12:04:57.706 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:04:57.865 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:57.854+0000 7f3964877700 1 Processor -- start 2022-06-27T12:04:57.866 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:57.854+0000 7f3964877700 1 -- start start 2022-06-27T12:04:57.866 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:57.855+0000 7f3964877700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f396006e850 0x7f39601570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:57.866 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:57.855+0000 7f3964877700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f39600c34a0 0x7f39600c2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:57.867 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:57.855+0000 7f3964877700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f396006fba0 con 0x7f396006fef0 2022-06-27T12:04:57.867 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:57.855+0000 7f3964877700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39601575e0 con 0x7f39600c34a0 2022-06-27T12:04:57.867 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:57.855+0000 7f3964877700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f396006f5e0 con 0x7f396006e850 2022-06-27T12:04:57.867 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:57.855+0000 7f395ed9d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f39600c34a0 0x7f39600c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:57.868 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:57.855+0000 7f395ed9d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f39600c34a0 0x7f39600c2920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47732/0 (socket says 172.21.15.195:47732) 2022-06-27T12:04:57.868 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:57.855+0000 7f395ed9d700 1 -- 172.21.15.195:0/2327937511 learned_addr learned my addr 172.21.15.195:0/2327937511 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:04:57.868 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:57.855+0000 7f395e59c700 1 --2- 172.21.15.195:0/2327937511 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f396006e850 0x7f39601570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:57.868 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:57.855+0000 7f395ed9d700 1 -- 172.21.15.195:0/2327937511 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f396006e850 msgr2=0x7f39601570a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:04:57.869 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:57.855+0000 7f395ed9d700 1 --2- 172.21.15.195:0/2327937511 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f396006e850 0x7f39601570a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:57.869 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:57.855+0000 7f395ed9d700 1 -- 172.21.15.195:0/2327937511 >> v1:172.21.15.202:6789/0 conn(0x7f396006fef0 legacy=0x7f3960069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:57.869 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:57.855+0000 7f395ed9d700 1 -- 172.21.15.195:0/2327937511 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f396006f8c0 con 0x7f39600c34a0 2022-06-27T12:04:57.869 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:57.855+0000 7f395ed9d700 1 --2- 172.21.15.195:0/2327937511 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f39600c34a0 0x7f39600c2920 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f3958009f20 tx=0x7f3958009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be3c6a6867ed0c08 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:04:57.870 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:57.856+0000 7f395d59a700 1 -- 172.21.15.195:0/2327937511 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f395800e070 con 0x7f39600c34a0 2022-06-27T12:04:57.870 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:57.856+0000 7f395d59a700 1 -- 172.21.15.195:0/2327937511 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f395800bb10 con 0x7f39600c34a0 2022-06-27T12:04:57.870 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:57.856+0000 7f395d59a700 1 -- 172.21.15.195:0/2327937511 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39580092e0 con 0x7f39600c34a0 2022-06-27T12:04:57.870 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:57.856+0000 7f3964877700 1 -- 172.21.15.195:0/2327937511 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f39600c34a0 msgr2=0x7f39600c2920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:57.871 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:57.856+0000 7f3964877700 1 --2- 172.21.15.195:0/2327937511 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f39600c34a0 0x7f39600c2920 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f3958009f20 tx=0x7f3958009840 comp rx=0 tx=0).stop 2022-06-27T12:04:57.871 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:57.856+0000 7f3964877700 1 -- 172.21.15.195:0/2327937511 shutdown_connections 2022-06-27T12:04:57.871 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:57.856+0000 7f3964877700 1 --2- 172.21.15.195:0/2327937511 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f39600c34a0 0x7f39600c2920 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:57.871 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:57.856+0000 7f3964877700 1 --2- 172.21.15.195:0/2327937511 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f396006e850 0x7f39601570a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:57.871 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:57.856+0000 7f3964877700 1 -- 172.21.15.195:0/2327937511 >> 172.21.15.195:0/2327937511 conn(0x7f39600c6350 msgr2=0x7f39600c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:04:57.872 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:57.856+0000 7f3964877700 1 -- 172.21.15.195:0/2327937511 shutdown_connections 2022-06-27T12:04:57.872 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:57.856+0000 7f3964877700 1 -- 172.21.15.195:0/2327937511 wait complete. 2022-06-27T12:04:57.872 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:57.857+0000 7f3964877700 1 Processor -- start 2022-06-27T12:04:57.872 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:57.857+0000 7f3964877700 1 -- start start 2022-06-27T12:04:57.873 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:57.858+0000 7f3964877700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f396006e850 0x7f3960150ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:57.873 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:57.858+0000 7f3964877700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f39600c34a0 0x7f3960164750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:57.873 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:57.858+0000 7f3964877700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3960156710 0x7f3960148150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:57.873 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:57.858+0000 7f3964877700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3960156b80 con 0x7f3960156710 2022-06-27T12:04:57.873 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:57.858+0000 7f3964877700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39601642c0 con 0x7f39600c34a0 2022-06-27T12:04:57.874 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:57.858+0000 7f3964877700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39600bd6e0 con 0x7f396006e850 2022-06-27T12:04:57.874 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:57.858+0000 7f395ed9d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3960156710 0x7f3960148150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:57.875 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:57.858+0000 7f395ed9d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3960156710 0x7f3960148150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47738/0 (socket says 172.21.15.195:47738) 2022-06-27T12:04:57.875 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:57.858+0000 7f395ed9d700 1 -- 172.21.15.195:0/1155019929 learned_addr learned my addr 172.21.15.195:0/1155019929 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:04:57.875 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:57.858+0000 7f395e59c700 1 --2- 172.21.15.195:0/1155019929 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f396006e850 0x7f3960150ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:57.876 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:57.858+0000 7f395dd9b700 1 --2- 172.21.15.195:0/1155019929 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f39600c34a0 0x7f3960164750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:57.876 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:57.859+0000 7f395ed9d700 1 -- 172.21.15.195:0/1155019929 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f396006e850 msgr2=0x7f3960150ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:57.876 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:57.859+0000 7f395ed9d700 1 --2- 172.21.15.195:0/1155019929 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f396006e850 0x7f3960150ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:57.876 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:57.859+0000 7f395ed9d700 1 -- 172.21.15.195:0/1155019929 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f39600c34a0 msgr2=0x7f3960164750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:57.876 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:57.859+0000 7f395ed9d700 1 --2- 172.21.15.195:0/1155019929 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f39600c34a0 0x7f3960164750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:57.877 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:57.859+0000 7f395ed9d700 1 -- 172.21.15.195:0/1155019929 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3958009ae0 con 0x7f3960156710 2022-06-27T12:04:57.877 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:57.859+0000 7f395e59c700 1 --2- 172.21.15.195:0/1155019929 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f396006e850 0x7f3960150ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:04:57.877 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:57.859+0000 7f395ed9d700 1 --2- 172.21.15.195:0/1155019929 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3960156710 0x7f3960148150 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f3958003bd0 tx=0x7f3958018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:04:57.877 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:57.859+0000 7f394e7fc700 1 -- 172.21.15.195:0/1155019929 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39580092e0 con 0x7f3960156710 2022-06-27T12:04:57.878 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:57.859+0000 7f394e7fc700 1 -- 172.21.15.195:0/1155019929 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3958007820 con 0x7f3960156710 2022-06-27T12:04:57.878 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:57.859+0000 7f3964877700 1 -- 172.21.15.195:0/1155019929 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39600bd980 con 0x7f3960156710 2022-06-27T12:04:57.878 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:57.859+0000 7f394e7fc700 1 -- 172.21.15.195:0/1155019929 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3958007bc0 con 0x7f3960156710 2022-06-27T12:04:57.878 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:57.859+0000 7f3964877700 1 -- 172.21.15.195:0/1155019929 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39600bdc40 con 0x7f3960156710 2022-06-27T12:04:57.879 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:57.860+0000 7f3964877700 1 -- 172.21.15.195:0/1155019929 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3960164450 con 0x7f3960156710 2022-06-27T12:04:57.879 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:57.861+0000 7f394e7fc700 1 -- 172.21.15.195:0/1155019929 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f3958004040 con 0x7f3960156710 2022-06-27T12:04:57.879 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:57.861+0000 7f394e7fc700 1 --2- 172.21.15.195:0/1155019929 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f393c063d30 0x7f393c0661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:04:57.879 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:57.861+0000 7f394e7fc700 1 -- 172.21.15.195:0/1155019929 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6890+0+0 (secure 0 0 0) 0x7f3958083000 con 0x7f3960156710 2022-06-27T12:04:57.879 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:57.861+0000 7f395e59c700 1 --2- 172.21.15.195:0/1155019929 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f393c063d30 0x7f393c0661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:04:57.880 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:57.861+0000 7f395e59c700 1 --2- 172.21.15.195:0/1155019929 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f393c063d30 0x7f393c0661e0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f3950006fd0 tx=0x7f3950008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:04:57.880 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:57.865+0000 7f394e7fc700 1 -- 172.21.15.195:0/1155019929 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f3958051930 con 0x7f3960156710 2022-06-27T12:04:58.086 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:58.086+0000 7f3964877700 1 -- 172.21.15.195:0/1155019929 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f39600722c0 con 0x7f3960156710 2022-06-27T12:04:58.087 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:58.086+0000 7f394e7fc700 1 -- 172.21.15.195:0/1155019929 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+6128 (secure 0 0 0) 0x7f3958054f50 con 0x7f3960156710 2022-06-27T12:04:58.088 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:04:58.089 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":5489,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/2342195382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2342195382},{"type":"v1","addr":"172.21.15.202:6835","nonce":2342195382}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:04:12.288201+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5818},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5818":{"gid":5818,"name":"b","rank":0,"incarnation":55,"state":"up:active","state_seq":34,"addr":"172.21.15.202:6833/3399027527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3399027527},{"type":"v1","addr":"172.21.15.202:6833","nonce":3399027527}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:04:58.091 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:58.088+0000 7f3964877700 1 -- 172.21.15.195:0/1155019929 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f393c063d30 msgr2=0x7f393c0661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:58.091 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:58.088+0000 7f3964877700 1 --2- 172.21.15.195:0/1155019929 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f393c063d30 0x7f393c0661e0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f3950006fd0 tx=0x7f3950008040 comp rx=0 tx=0).stop 2022-06-27T12:04:58.092 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:58.088+0000 7f3964877700 1 -- 172.21.15.195:0/1155019929 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3960156710 msgr2=0x7f3960148150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:04:58.092 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:58.088+0000 7f3964877700 1 --2- 172.21.15.195:0/1155019929 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3960156710 0x7f3960148150 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f3958003bd0 tx=0x7f3958018000 comp rx=0 tx=0).stop 2022-06-27T12:04:58.093 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:58.088+0000 7f3964877700 1 -- 172.21.15.195:0/1155019929 shutdown_connections 2022-06-27T12:04:58.093 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:58.088+0000 7f3964877700 1 --2- 172.21.15.195:0/1155019929 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f393c063d30 0x7f393c0661e0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:58.094 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:58.088+0000 7f3964877700 1 --2- 172.21.15.195:0/1155019929 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3960156710 0x7f3960148150 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:58.094 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:58.088+0000 7f3964877700 1 --2- 172.21.15.195:0/1155019929 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f396006e850 0x7f3960150ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:58.095 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:58.088+0000 7f3964877700 1 --2- 172.21.15.195:0/1155019929 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f39600c34a0 0x7f3960164750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:04:58.095 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:58.088+0000 7f3964877700 1 -- 172.21.15.195:0/1155019929 >> 172.21.15.195:0/1155019929 conn(0x7f39600c6350 msgr2=0x7f3960068310 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:04:58.095 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:58.088+0000 7f3964877700 1 -- 172.21.15.195:0/1155019929 shutdown_connections 2022-06-27T12:04:58.095 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:04:58.088+0000 7f3964877700 1 -- 172.21.15.195:0/1155019929 wait complete. 2022-06-27T12:04:58.096 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 60 2022-06-27T12:04:58.102 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2022-06-27T12:05:00.103 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:05:00.246 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.235+0000 7f7ca58ac700 1 Processor -- start 2022-06-27T12:05:00.246 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.235+0000 7f7ca58ac700 1 -- start start 2022-06-27T12:05:00.247 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.236+0000 7f7ca58ac700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7ca00c40d0 0x7f7ca0157010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:00.247 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.236+0000 7f7ca58ac700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f7ca00c6190 0x7f7ca0067ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:00.247 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.236+0000 7f7ca58ac700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ca0156b80 con 0x7f7ca00c44a0 2022-06-27T12:05:00.248 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.236+0000 7f7ca58ac700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ca006a040 con 0x7f7ca00c6190 2022-06-27T12:05:00.248 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.236+0000 7f7ca58ac700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca006a300 con 0x7f7ca00c40d0 2022-06-27T12:05:00.249 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.236+0000 7f7c9effd700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7ca00c40d0 0x7f7ca0157010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:00.249 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.236+0000 7f7c9e7fc700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f7ca00c6190 0x7f7ca0067ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:00.249 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.236+0000 7f7c9f7fe700 1 --1- >> v1:172.21.15.202:6789/0 conn(0x7f7ca00c44a0 0x7f7ca014cb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.202:6789/0 says I am v1:172.21.15.195:55796/0 (socket says 172.21.15.195:55796) 2022-06-27T12:05:00.250 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.236+0000 7f7c9f7fe700 1 -- 172.21.15.195:0/2544219285 learned_addr learned my addr 172.21.15.195:0/2544219285 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:05:00.250 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.236+0000 7f7c9dffb700 1 -- 172.21.15.195:0/2544219285 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2451417874 0 0) 0x7f7ca0156b80 con 0x7f7ca00c44a0 2022-06-27T12:05:00.251 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.236+0000 7f7c9dffb700 1 -- 172.21.15.195:0/2544219285 --> v1:172.21.15.202:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c8c003430 con 0x7f7ca00c44a0 2022-06-27T12:05:00.251 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.236+0000 7f7c9effd700 1 -- 172.21.15.195:0/2544219285 >> v2:172.21.15.202:3300/0 conn(0x7f7ca00c6190 msgr2=0x7f7ca0067ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:05:00.252 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.236+0000 7f7c9effd700 1 --2- 172.21.15.195:0/2544219285 >> v2:172.21.15.202:3300/0 conn(0x7f7ca00c6190 0x7f7ca0067ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:00.252 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.236+0000 7f7c9effd700 1 -- 172.21.15.195:0/2544219285 >> v1:172.21.15.202:6789/0 conn(0x7f7ca00c44a0 legacy=0x7f7ca014cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:00.253 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.236+0000 7f7c9effd700 1 -- 172.21.15.195:0/2544219285 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ca006a5e0 con 0x7f7ca00c40d0 2022-06-27T12:05:00.253 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.236+0000 7f7c9e7fc700 1 --2- 172.21.15.195:0/2544219285 >> v2:172.21.15.202:3300/0 conn(0x7f7ca00c6190 0x7f7ca0067ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:05:00.254 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.236+0000 7f7c9effd700 1 --2- 172.21.15.195:0/2544219285 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7ca00c40d0 0x7f7ca0157010 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7c940097a0 tx=0x7f7c94009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=961e35ebda55bfbf server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:05:00.254 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.237+0000 7f7c9dffb700 1 -- 172.21.15.195:0/2544219285 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c94011630 con 0x7f7ca00c40d0 2022-06-27T12:05:00.255 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.237+0000 7f7c9dffb700 1 -- 172.21.15.195:0/2544219285 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c94011790 con 0x7f7ca00c40d0 2022-06-27T12:05:00.255 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.237+0000 7f7c9dffb700 1 -- 172.21.15.195:0/2544219285 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c94011b30 con 0x7f7ca00c40d0 2022-06-27T12:05:00.255 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.237+0000 7f7ca58ac700 1 -- 172.21.15.195:0/2544219285 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7ca00c40d0 msgr2=0x7f7ca0157010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:00.256 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.237+0000 7f7ca58ac700 1 --2- 172.21.15.195:0/2544219285 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7ca00c40d0 0x7f7ca0157010 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7c940097a0 tx=0x7f7c94009060 comp rx=0 tx=0).stop 2022-06-27T12:05:00.256 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.237+0000 7f7ca58ac700 1 -- 172.21.15.195:0/2544219285 shutdown_connections 2022-06-27T12:05:00.256 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.237+0000 7f7ca58ac700 1 --2- 172.21.15.195:0/2544219285 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7ca00c40d0 0x7f7ca0157010 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:00.256 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.237+0000 7f7ca58ac700 1 --2- 172.21.15.195:0/2544219285 >> v2:172.21.15.202:3300/0 conn(0x7f7ca00c6190 0x7f7ca0067ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:00.257 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.237+0000 7f7ca58ac700 1 -- 172.21.15.195:0/2544219285 >> 172.21.15.195:0/2544219285 conn(0x7f7ca00ccb80 msgr2=0x7f7ca00ccf80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:05:00.257 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.237+0000 7f7ca58ac700 1 -- 172.21.15.195:0/2544219285 shutdown_connections 2022-06-27T12:05:00.257 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.237+0000 7f7ca58ac700 1 -- 172.21.15.195:0/2544219285 wait complete. 2022-06-27T12:05:00.258 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.238+0000 7f7ca58ac700 1 Processor -- start 2022-06-27T12:05:00.258 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.239+0000 7f7ca58ac700 1 -- start start 2022-06-27T12:05:00.258 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.239+0000 7f7ca58ac700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7ca00c40d0 0x7f7ca014f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:00.258 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.239+0000 7f7ca58ac700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7ca00c44a0 0x7f7ca00c2520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:00.259 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.239+0000 7f7ca58ac700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7ca00c6190 0x7f7ca00c2a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:00.259 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.239+0000 7f7ca58ac700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca0159320 con 0x7f7ca00c40d0 2022-06-27T12:05:00.259 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.239+0000 7f7ca58ac700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca00c2090 con 0x7f7ca00c6190 2022-06-27T12:05:00.260 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.239+0000 7f7ca58ac700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca00c1ab0 con 0x7f7ca00c44a0 2022-06-27T12:05:00.260 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.239+0000 7f7c9effd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7ca00c40d0 0x7f7ca014f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:00.260 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.239+0000 7f7c9effd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7ca00c40d0 0x7f7ca014f450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47750/0 (socket says 172.21.15.195:47750) 2022-06-27T12:05:00.260 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.239+0000 7f7c9effd700 1 -- 172.21.15.195:0/3194504152 learned_addr learned my addr 172.21.15.195:0/3194504152 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:05:00.261 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.240+0000 7f7c9e7fc700 1 --2- 172.21.15.195:0/3194504152 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7ca00c44a0 0x7f7ca00c2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:00.261 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.240+0000 7f7c9f7fe700 1 --2- 172.21.15.195:0/3194504152 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7ca00c6190 0x7f7ca00c2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:00.261 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.240+0000 7f7c9effd700 1 -- 172.21.15.195:0/3194504152 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7ca00c44a0 msgr2=0x7f7ca00c2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:00.261 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.240+0000 7f7c9effd700 1 --2- 172.21.15.195:0/3194504152 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7ca00c44a0 0x7f7ca00c2520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:00.262 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.240+0000 7f7c9effd700 1 -- 172.21.15.195:0/3194504152 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7ca00c6190 msgr2=0x7f7ca00c2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:00.262 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.240+0000 7f7c9effd700 1 --2- 172.21.15.195:0/3194504152 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7ca00c6190 0x7f7ca00c2a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:00.262 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.240+0000 7f7c9effd700 1 -- 172.21.15.195:0/3194504152 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c900092e0 con 0x7f7ca00c40d0 2022-06-27T12:05:00.263 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.240+0000 7f7c9effd700 1 --2- 172.21.15.195:0/3194504152 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7ca00c40d0 0x7f7ca014f450 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f7c94011c30 tx=0x7f7c94017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:05:00.263 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.240+0000 7f7c86ffd700 1 -- 172.21.15.195:0/3194504152 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c94003800 con 0x7f7ca00c40d0 2022-06-27T12:05:00.263 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.240+0000 7f7c86ffd700 1 -- 172.21.15.195:0/3194504152 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c94003980 con 0x7f7ca00c40d0 2022-06-27T12:05:00.263 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.240+0000 7f7c86ffd700 1 -- 172.21.15.195:0/3194504152 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c94003d40 con 0x7f7ca00c40d0 2022-06-27T12:05:00.264 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.240+0000 7f7ca58ac700 1 -- 172.21.15.195:0/3194504152 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c940092e0 con 0x7f7ca00c40d0 2022-06-27T12:05:00.264 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.240+0000 7f7ca58ac700 1 -- 172.21.15.195:0/3194504152 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ca0159570 con 0x7f7ca00c40d0 2022-06-27T12:05:00.264 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.241+0000 7f7ca58ac700 1 -- 172.21.15.195:0/3194504152 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ca00c2370 con 0x7f7ca00c40d0 2022-06-27T12:05:00.265 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.241+0000 7f7c86ffd700 1 -- 172.21.15.195:0/3194504152 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f7c9402a020 con 0x7f7ca00c40d0 2022-06-27T12:05:00.265 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.242+0000 7f7c86ffd700 1 --2- 172.21.15.195:0/3194504152 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7c80063d00 0x7f7c800661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:00.265 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.242+0000 7f7c86ffd700 1 -- 172.21.15.195:0/3194504152 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6890+0+0 (secure 0 0 0) 0x7f7c940820d0 con 0x7f7ca00c40d0 2022-06-27T12:05:00.266 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.242+0000 7f7c9e7fc700 1 --2- 172.21.15.195:0/3194504152 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7c80063d00 0x7f7c800661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:00.266 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.242+0000 7f7c9e7fc700 1 --2- 172.21.15.195:0/3194504152 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7c80063d00 0x7f7c800661b0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f7c90006dc0 tx=0x7f7c90008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:05:00.267 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.246+0000 7f7c86ffd700 1 -- 172.21.15.195:0/3194504152 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f7c94050980 con 0x7f7ca00c40d0 2022-06-27T12:05:00.471 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.470+0000 7f7ca58ac700 1 -- 172.21.15.195:0/3194504152 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7ca016a830 con 0x7f7ca00c40d0 2022-06-27T12:05:00.472 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.471+0000 7f7c86ffd700 1 -- 172.21.15.195:0/3194504152 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+6128 (secure 0 0 0) 0x7f7c94053fa0 con 0x7f7ca00c40d0 2022-06-27T12:05:00.473 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:05:00.473 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":5489,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/2342195382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2342195382},{"type":"v1","addr":"172.21.15.202:6835","nonce":2342195382}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:04:12.288201+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5818},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5818":{"gid":5818,"name":"b","rank":0,"incarnation":55,"state":"up:active","state_seq":34,"addr":"172.21.15.202:6833/3399027527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3399027527},{"type":"v1","addr":"172.21.15.202:6833","nonce":3399027527}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:05:00.476 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.473+0000 7f7ca58ac700 1 -- 172.21.15.195:0/3194504152 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7c80063d00 msgr2=0x7f7c800661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:00.477 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.473+0000 7f7ca58ac700 1 --2- 172.21.15.195:0/3194504152 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7c80063d00 0x7f7c800661b0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f7c90006dc0 tx=0x7f7c90008040 comp rx=0 tx=0).stop 2022-06-27T12:05:00.477 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.473+0000 7f7ca58ac700 1 -- 172.21.15.195:0/3194504152 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7ca00c40d0 msgr2=0x7f7ca014f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:00.478 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.473+0000 7f7ca58ac700 1 --2- 172.21.15.195:0/3194504152 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7ca00c40d0 0x7f7ca014f450 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f7c94011c30 tx=0x7f7c94017000 comp rx=0 tx=0).stop 2022-06-27T12:05:00.478 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.473+0000 7f7ca58ac700 1 -- 172.21.15.195:0/3194504152 shutdown_connections 2022-06-27T12:05:00.479 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.473+0000 7f7ca58ac700 1 --2- 172.21.15.195:0/3194504152 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7c80063d00 0x7f7c800661b0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:00.480 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.473+0000 7f7ca58ac700 1 --2- 172.21.15.195:0/3194504152 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7ca00c40d0 0x7f7ca014f450 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:00.480 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.473+0000 7f7ca58ac700 1 --2- 172.21.15.195:0/3194504152 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7ca00c6190 0x7f7ca00c2a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:00.481 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.473+0000 7f7ca58ac700 1 --2- 172.21.15.195:0/3194504152 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7ca00c44a0 0x7f7ca00c2520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:00.481 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.473+0000 7f7ca58ac700 1 -- 172.21.15.195:0/3194504152 >> 172.21.15.195:0/3194504152 conn(0x7f7ca00ccb80 msgr2=0x7f7ca0154a80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:05:00.482 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.473+0000 7f7ca58ac700 1 -- 172.21.15.195:0/3194504152 shutdown_connections 2022-06-27T12:05:00.482 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:00.473+0000 7f7ca58ac700 1 -- 172.21.15.195:0/3194504152 wait complete. 2022-06-27T12:05:00.483 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 60 2022-06-27T12:05:00.496 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2022-06-27T12:05:00.939 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:05:00.942 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:05:02.497 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:05:02.636 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.633+0000 7f0a8bcca700 1 Processor -- start 2022-06-27T12:05:02.637 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.633+0000 7f0a8bcca700 1 -- start start 2022-06-27T12:05:02.638 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.634+0000 7f0a8bcca700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f0a8414e490 0x7f0a8406cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:02.638 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.634+0000 7f0a8bcca700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0a840c3f50 0x7f0a84157890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:02.638 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.634+0000 7f0a8bcca700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a840c33b0 con 0x7f0a8414e860 2022-06-27T12:05:02.639 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.634+0000 7f0a8bcca700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a8406d520 con 0x7f0a8414e490 2022-06-27T12:05:02.639 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.634+0000 7f0a8bcca700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a8406f220 con 0x7f0a840c3f50 2022-06-27T12:05:02.640 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.634+0000 7f0a89265700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f0a8414e860 0x7f0a840c3840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:53950/0 (socket says 172.21.15.195:53950) 2022-06-27T12:05:02.640 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.634+0000 7f0a89265700 1 -- 172.21.15.195:0/3438373057 learned_addr learned my addr 172.21.15.195:0/3438373057 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:05:02.641 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.634+0000 7f0a89a66700 1 --2- 172.21.15.195:0/3438373057 >> v2:172.21.15.202:3300/0 conn(0x7f0a8414e490 0x7f0a8406cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:02.641 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.634+0000 7f0a8a267700 1 --2- 172.21.15.195:0/3438373057 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0a840c3f50 0x7f0a84157890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:02.642 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.634+0000 7f0a88a64700 1 -- 172.21.15.195:0/3438373057 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3018084836 0 0) 0x7f0a840c33b0 con 0x7f0a8414e860 2022-06-27T12:05:02.649 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.634+0000 7f0a88a64700 1 -- 172.21.15.195:0/3438373057 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a6c003430 con 0x7f0a8414e860 2022-06-27T12:05:02.650 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.634+0000 7f0a89a66700 1 -- 172.21.15.195:0/3438373057 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0a840c3f50 msgr2=0x7f0a84157890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:05:02.650 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.634+0000 7f0a89a66700 1 --2- 172.21.15.195:0/3438373057 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0a840c3f50 0x7f0a84157890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:02.651 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.634+0000 7f0a89a66700 1 -- 172.21.15.195:0/3438373057 >> v1:172.21.15.195:6789/0 conn(0x7f0a8414e860 legacy=0x7f0a840c3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:02.651 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.634+0000 7f0a89a66700 1 -- 172.21.15.195:0/3438373057 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a841542c0 con 0x7f0a8414e490 2022-06-27T12:05:02.652 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.634+0000 7f0a8a267700 1 --2- 172.21.15.195:0/3438373057 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0a840c3f50 0x7f0a84157890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:05:02.652 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.635+0000 7f0a89a66700 1 --2- 172.21.15.195:0/3438373057 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0a8414e490 0x7f0a8406cfe0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f0a800097a0 tx=0x7f0a80009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa4e4fd54d7fce73 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:05:02.653 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.635+0000 7f0a88a64700 1 -- 172.21.15.195:0/3438373057 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a80011630 con 0x7f0a8414e490 2022-06-27T12:05:02.653 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.635+0000 7f0a88a64700 1 -- 172.21.15.195:0/3438373057 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a80011790 con 0x7f0a8414e490 2022-06-27T12:05:02.653 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.635+0000 7f0a88a64700 1 -- 172.21.15.195:0/3438373057 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a80011b10 con 0x7f0a8414e490 2022-06-27T12:05:02.654 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.635+0000 7f0a8bcca700 1 -- 172.21.15.195:0/3438373057 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0a8414e490 msgr2=0x7f0a8406cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:02.654 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.635+0000 7f0a8bcca700 1 --2- 172.21.15.195:0/3438373057 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0a8414e490 0x7f0a8406cfe0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f0a800097a0 tx=0x7f0a80009060 comp rx=0 tx=0).stop 2022-06-27T12:05:02.655 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.635+0000 7f0a8bcca700 1 -- 172.21.15.195:0/3438373057 shutdown_connections 2022-06-27T12:05:02.655 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.635+0000 7f0a8bcca700 1 --2- 172.21.15.195:0/3438373057 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0a840c3f50 0x7f0a84157890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:02.656 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.635+0000 7f0a8bcca700 1 --2- 172.21.15.195:0/3438373057 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0a8414e490 0x7f0a8406cfe0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:02.656 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.635+0000 7f0a8bcca700 1 -- 172.21.15.195:0/3438373057 >> 172.21.15.195:0/3438373057 conn(0x7f0a84063af0 msgr2=0x7f0a84063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:05:02.657 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.635+0000 7f0a8bcca700 1 -- 172.21.15.195:0/3438373057 shutdown_connections 2022-06-27T12:05:02.657 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.635+0000 7f0a8bcca700 1 -- 172.21.15.195:0/3438373057 wait complete. 2022-06-27T12:05:02.657 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.637+0000 7f0a8bcca700 1 Processor -- start 2022-06-27T12:05:02.658 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.637+0000 7f0a8bcca700 1 -- start start 2022-06-27T12:05:02.658 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.637+0000 7f0a8bcca700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0a840c3f50 0x7f0a841518e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:02.659 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.637+0000 7f0a8bcca700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0a8414e490 0x7f0a84167910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:02.659 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.637+0000 7f0a8bcca700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0a8414e860 0x7f0a8414fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:02.660 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.637+0000 7f0a8bcca700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a84157910 con 0x7f0a840c3f50 2022-06-27T12:05:02.660 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.637+0000 7f0a8bcca700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a841682c0 con 0x7f0a8414e490 2022-06-27T12:05:02.660 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.637+0000 7f0a8bcca700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a84157bb0 con 0x7f0a8414e860 2022-06-27T12:05:02.661 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.637+0000 7f0a89a66700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0a840c3f50 0x7f0a841518e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:02.661 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.637+0000 7f0a89a66700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0a840c3f50 0x7f0a841518e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47768/0 (socket says 172.21.15.195:47768) 2022-06-27T12:05:02.662 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.637+0000 7f0a89a66700 1 -- 172.21.15.195:0/535899834 learned_addr learned my addr 172.21.15.195:0/535899834 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:05:02.662 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.638+0000 7f0a8a267700 1 --2- 172.21.15.195:0/535899834 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0a8414e860 0x7f0a8414fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:02.662 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.638+0000 7f0a89a66700 1 -- 172.21.15.195:0/535899834 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0a8414e860 msgr2=0x7f0a8414fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:02.663 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.638+0000 7f0a89a66700 1 --2- 172.21.15.195:0/535899834 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0a8414e860 0x7f0a8414fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:02.663 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.638+0000 7f0a89265700 1 --2- 172.21.15.195:0/535899834 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0a8414e490 0x7f0a84167910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:02.664 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.638+0000 7f0a89a66700 1 -- 172.21.15.195:0/535899834 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0a8414e490 msgr2=0x7f0a84167910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:02.664 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.638+0000 7f0a89a66700 1 --2- 172.21.15.195:0/535899834 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0a8414e490 0x7f0a84167910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:02.664 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.638+0000 7f0a89a66700 1 -- 172.21.15.195:0/535899834 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a7c009a20 con 0x7f0a840c3f50 2022-06-27T12:05:02.665 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.638+0000 7f0a89a66700 1 --2- 172.21.15.195:0/535899834 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0a840c3f50 0x7f0a841518e0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f0a84070770 tx=0x7f0a80017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:05:02.665 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.638+0000 7f0a71ffb700 1 -- 172.21.15.195:0/535899834 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a80003800 con 0x7f0a840c3f50 2022-06-27T12:05:02.665 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.638+0000 7f0a71ffb700 1 -- 172.21.15.195:0/535899834 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a80003980 con 0x7f0a840c3f50 2022-06-27T12:05:02.671 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.638+0000 7f0a8bcca700 1 -- 172.21.15.195:0/535899834 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a800092e0 con 0x7f0a840c3f50 2022-06-27T12:05:02.672 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.638+0000 7f0a71ffb700 1 -- 172.21.15.195:0/535899834 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a80003d40 con 0x7f0a840c3f50 2022-06-27T12:05:02.672 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.638+0000 7f0a8bcca700 1 -- 172.21.15.195:0/535899834 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a8414f710 con 0x7f0a840c3f50 2022-06-27T12:05:02.673 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.639+0000 7f0a8bcca700 1 -- 172.21.15.195:0/535899834 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a84154130 con 0x7f0a840c3f50 2022-06-27T12:05:02.674 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.640+0000 7f0a71ffb700 1 -- 172.21.15.195:0/535899834 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f0a8000f050 con 0x7f0a840c3f50 2022-06-27T12:05:02.674 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.641+0000 7f0a71ffb700 1 --2- 172.21.15.195:0/535899834 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0a64063d00 0x7f0a640661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:02.675 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.641+0000 7f0a71ffb700 1 -- 172.21.15.195:0/535899834 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6890+0+0 (secure 0 0 0) 0x7f0a800826e0 con 0x7f0a840c3f50 2022-06-27T12:05:02.676 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.641+0000 7f0a89265700 1 --2- 172.21.15.195:0/535899834 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0a64063d00 0x7f0a640661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:02.677 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.641+0000 7f0a89265700 1 --2- 172.21.15.195:0/535899834 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0a64063d00 0x7f0a640661b0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f0a74005790 tx=0x7f0a74009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:05:02.677 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.644+0000 7f0a71ffb700 1 -- 172.21.15.195:0/535899834 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f0a80051010 con 0x7f0a840c3f50 2022-06-27T12:05:02.874 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.872+0000 7f0a8bcca700 1 -- 172.21.15.195:0/535899834 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0a840722c0 con 0x7f0a840c3f50 2022-06-27T12:05:02.875 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.872+0000 7f0a71ffb700 1 -- 172.21.15.195:0/535899834 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+6128 (secure 0 0 0) 0x7f0a80015020 con 0x7f0a840c3f50 2022-06-27T12:05:02.875 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:05:02.876 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":5489,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/2342195382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2342195382},{"type":"v1","addr":"172.21.15.202:6835","nonce":2342195382}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:04:12.288201+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5818},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5818":{"gid":5818,"name":"b","rank":0,"incarnation":55,"state":"up:active","state_seq":34,"addr":"172.21.15.202:6833/3399027527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3399027527},{"type":"v1","addr":"172.21.15.202:6833","nonce":3399027527}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:05:02.879 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.875+0000 7f0a8bcca700 1 -- 172.21.15.195:0/535899834 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0a64063d00 msgr2=0x7f0a640661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:02.879 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.875+0000 7f0a8bcca700 1 --2- 172.21.15.195:0/535899834 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0a64063d00 0x7f0a640661b0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f0a74005790 tx=0x7f0a74009040 comp rx=0 tx=0).stop 2022-06-27T12:05:02.880 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.875+0000 7f0a8bcca700 1 -- 172.21.15.195:0/535899834 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0a840c3f50 msgr2=0x7f0a841518e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:02.880 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.875+0000 7f0a8bcca700 1 --2- 172.21.15.195:0/535899834 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0a840c3f50 0x7f0a841518e0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f0a84070770 tx=0x7f0a80017000 comp rx=0 tx=0).stop 2022-06-27T12:05:02.880 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.875+0000 7f0a8bcca700 1 -- 172.21.15.195:0/535899834 shutdown_connections 2022-06-27T12:05:02.881 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.875+0000 7f0a8bcca700 1 --2- 172.21.15.195:0/535899834 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0a64063d00 0x7f0a640661b0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:02.883 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.875+0000 7f0a8bcca700 1 --2- 172.21.15.195:0/535899834 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0a840c3f50 0x7f0a841518e0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:02.883 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.875+0000 7f0a8bcca700 1 --2- 172.21.15.195:0/535899834 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0a8414e860 0x7f0a8414fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:02.884 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.875+0000 7f0a8bcca700 1 --2- 172.21.15.195:0/535899834 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0a8414e490 0x7f0a84167910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:02.884 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.875+0000 7f0a8bcca700 1 -- 172.21.15.195:0/535899834 >> 172.21.15.195:0/535899834 conn(0x7f0a84063af0 msgr2=0x7f0a8414d580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:05:02.884 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.875+0000 7f0a8bcca700 1 -- 172.21.15.195:0/535899834 shutdown_connections 2022-06-27T12:05:02.885 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:02.875+0000 7f0a8bcca700 1 -- 172.21.15.195:0/535899834 wait complete. 2022-06-27T12:05:02.885 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 60 2022-06-27T12:05:02.891 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2022-06-27T12:05:04.893 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:05:05.027 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.026+0000 7fa72c168700 1 Processor -- start 2022-06-27T12:05:05.027 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.027+0000 7fa72c168700 1 -- start start 2022-06-27T12:05:05.028 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.027+0000 7fa72c168700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa72414e490 0x7fa72406cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:05.029 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.027+0000 7fa72c168700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fa72414e860 0x7fa7240c3840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:05.029 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.027+0000 7fa72c168700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fa7240c3d80 0x7fa72406f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:05.029 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.028+0000 7fa72c168700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7240c33b0 con 0x7fa72414e860 2022-06-27T12:05:05.030 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.028+0000 7fa72c168700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa72406d520 con 0x7fa7240c3d80 2022-06-27T12:05:05.030 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.028+0000 7fa72c168700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa72406f220 con 0x7fa72414e490 2022-06-27T12:05:05.030 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.028+0000 7fa729703700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fa72414e860 0x7fa7240c3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:05.030 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.028+0000 7fa729703700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fa72414e860 0x7fa7240c3840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47778/0 (socket says 172.21.15.195:47778) 2022-06-27T12:05:05.030 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.028+0000 7fa729703700 1 -- 172.21.15.195:0/625630630 learned_addr learned my addr 172.21.15.195:0/625630630 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:05:05.031 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.028+0000 7fa729f04700 1 --2- 172.21.15.195:0/625630630 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa72414e490 0x7fa72406cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:05.040 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.028+0000 7fa72a705700 1 --2- 172.21.15.195:0/625630630 >> v2:172.21.15.202:3300/0 conn(0x7fa7240c3d80 0x7fa72406f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:05.040 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.028+0000 7fa729703700 1 -- 172.21.15.195:0/625630630 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa72414e490 msgr2=0x7fa72406cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:05:05.041 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.028+0000 7fa729703700 1 --2- 172.21.15.195:0/625630630 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa72414e490 0x7fa72406cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:05.041 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.028+0000 7fa729703700 1 -- 172.21.15.195:0/625630630 >> v2:172.21.15.202:3300/0 conn(0x7fa7240c3d80 msgr2=0x7fa72406f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:05:05.041 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.028+0000 7fa729703700 1 --2- 172.21.15.195:0/625630630 >> v2:172.21.15.202:3300/0 conn(0x7fa7240c3d80 0x7fa72406f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:05.042 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.028+0000 7fa729703700 1 -- 172.21.15.195:0/625630630 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7241542c0 con 0x7fa72414e860 2022-06-27T12:05:05.042 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.028+0000 7fa729703700 1 --2- 172.21.15.195:0/625630630 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa72414e860 0x7fa7240c3840 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fa72406ccd0 tx=0x7fa720009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=591e8f96e65128a9 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:05:05.042 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.028+0000 7fa728f02700 1 -- 172.21.15.195:0/625630630 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa72000b640 con 0x7fa72414e860 2022-06-27T12:05:05.043 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.028+0000 7fa728f02700 1 -- 172.21.15.195:0/625630630 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa72000b7a0 con 0x7fa72414e860 2022-06-27T12:05:05.043 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.028+0000 7fa728f02700 1 -- 172.21.15.195:0/625630630 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa72000bb40 con 0x7fa72414e860 2022-06-27T12:05:05.043 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.029+0000 7fa72c168700 1 -- 172.21.15.195:0/625630630 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa72414e860 msgr2=0x7fa7240c3840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:05.043 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.029+0000 7fa72c168700 1 --2- 172.21.15.195:0/625630630 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa72414e860 0x7fa7240c3840 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fa72406ccd0 tx=0x7fa720009120 comp rx=0 tx=0).stop 2022-06-27T12:05:05.043 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.029+0000 7fa72c168700 1 -- 172.21.15.195:0/625630630 shutdown_connections 2022-06-27T12:05:05.044 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.029+0000 7fa72c168700 1 --2- 172.21.15.195:0/625630630 >> v2:172.21.15.202:3300/0 conn(0x7fa7240c3d80 0x7fa72406f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:05.044 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.029+0000 7fa72c168700 1 --2- 172.21.15.195:0/625630630 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa72414e490 0x7fa72406cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:05.044 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.029+0000 7fa72c168700 1 --2- 172.21.15.195:0/625630630 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa72414e860 0x7fa7240c3840 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:05.044 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.029+0000 7fa72c168700 1 -- 172.21.15.195:0/625630630 >> 172.21.15.195:0/625630630 conn(0x7fa724063af0 msgr2=0x7fa724063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:05:05.045 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.029+0000 7fa72c168700 1 -- 172.21.15.195:0/625630630 shutdown_connections 2022-06-27T12:05:05.045 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.029+0000 7fa72c168700 1 -- 172.21.15.195:0/625630630 wait complete. 2022-06-27T12:05:05.045 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.030+0000 7fa72c168700 1 Processor -- start 2022-06-27T12:05:05.045 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.030+0000 7fa72c168700 1 -- start start 2022-06-27T12:05:05.045 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.031+0000 7fa72c168700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa7240c3d80 0x7fa7240bff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:05.046 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.031+0000 7fa72c168700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa72414e490 0x7fa7241510c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:05.046 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.031+0000 7fa72c168700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa72414e860 0x7fa724151600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:05.046 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.031+0000 7fa72c168700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa724157a70 con 0x7fa7240c3d80 2022-06-27T12:05:05.046 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.031+0000 7fa72c168700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa724151b40 con 0x7fa72414e860 2022-06-27T12:05:05.047 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.031+0000 7fa72c168700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7240bfae0 con 0x7fa72414e490 2022-06-27T12:05:05.047 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.031+0000 7fa729f04700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa7240c3d80 0x7fa7240bff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:05.047 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.031+0000 7fa729f04700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa7240c3d80 0x7fa7240bff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47784/0 (socket says 172.21.15.195:47784) 2022-06-27T12:05:05.047 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.031+0000 7fa729f04700 1 -- 172.21.15.195:0/4248162924 learned_addr learned my addr 172.21.15.195:0/4248162924 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:05:05.047 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.031+0000 7fa72a705700 1 --2- 172.21.15.195:0/4248162924 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa72414e860 0x7fa724151600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:05.048 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.031+0000 7fa729703700 1 --2- 172.21.15.195:0/4248162924 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa72414e490 0x7fa7241510c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:05.048 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.031+0000 7fa729f04700 1 -- 172.21.15.195:0/4248162924 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa72414e490 msgr2=0x7fa7241510c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:05.048 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.031+0000 7fa729f04700 1 --2- 172.21.15.195:0/4248162924 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa72414e490 0x7fa7241510c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:05.048 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.031+0000 7fa729f04700 1 -- 172.21.15.195:0/4248162924 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa72414e860 msgr2=0x7fa724151600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:05.049 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.031+0000 7fa729f04700 1 --2- 172.21.15.195:0/4248162924 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa72414e860 0x7fa724151600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:05.049 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.031+0000 7fa729f04700 1 -- 172.21.15.195:0/4248162924 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7200093a0 con 0x7fa7240c3d80 2022-06-27T12:05:05.049 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.031+0000 7fa729703700 1 --2- 172.21.15.195:0/4248162924 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa72414e490 0x7fa7241510c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:05:05.049 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.031+0000 7fa729f04700 1 --2- 172.21.15.195:0/4248162924 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa7240c3d80 0x7fa7240bff70 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fa71400d890 tx=0x7fa714008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:05:05.049 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.031+0000 7fa711ffb700 1 -- 172.21.15.195:0/4248162924 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa714014620 con 0x7fa7240c3d80 2022-06-27T12:05:05.050 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.031+0000 7fa711ffb700 1 -- 172.21.15.195:0/4248162924 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa714014780 con 0x7fa7240c3d80 2022-06-27T12:05:05.050 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.031+0000 7fa711ffb700 1 -- 172.21.15.195:0/4248162924 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa714014b40 con 0x7fa7240c3d80 2022-06-27T12:05:05.050 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.031+0000 7fa72c168700 1 -- 172.21.15.195:0/4248162924 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7240bf4d0 con 0x7fa7240c3d80 2022-06-27T12:05:05.050 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.032+0000 7fa72c168700 1 -- 172.21.15.195:0/4248162924 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa724158260 con 0x7fa7240c3d80 2022-06-27T12:05:05.051 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.033+0000 7fa72c168700 1 -- 172.21.15.195:0/4248162924 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7240722c0 con 0x7fa7240c3d80 2022-06-27T12:05:05.051 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.033+0000 7fa711ffb700 1 -- 172.21.15.195:0/4248162924 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fa714024070 con 0x7fa7240c3d80 2022-06-27T12:05:05.051 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.033+0000 7fa711ffb700 1 --2- 172.21.15.195:0/4248162924 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa704063cd0 0x7fa704066180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:05.051 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.033+0000 7fa711ffb700 1 -- 172.21.15.195:0/4248162924 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6890+0+0 (secure 0 0 0) 0x7fa714081170 con 0x7fa7240c3d80 2022-06-27T12:05:05.051 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.034+0000 7fa729703700 1 --2- 172.21.15.195:0/4248162924 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa704063cd0 0x7fa704066180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:05.052 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.034+0000 7fa729703700 1 --2- 172.21.15.195:0/4248162924 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa704063cd0 0x7fa704066180 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fa724065b00 tx=0x7fa720017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:05:05.052 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.038+0000 7fa711ffb700 1 -- 172.21.15.195:0/4248162924 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fa71404fa20 con 0x7fa7240c3d80 2022-06-27T12:05:05.271 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.270+0000 7fa72c168700 1 -- 172.21.15.195:0/4248162924 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa724064240 con 0x7fa7240c3d80 2022-06-27T12:05:05.271 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.271+0000 7fa711ffb700 1 -- 172.21.15.195:0/4248162924 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+5367 (secure 0 0 0) 0x7fa714053040 con 0x7fa7240c3d80 2022-06-27T12:05:05.272 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:05:05.272 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:05:05.063331+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5489},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5489":{"gid":5489,"name":"d","rank":0,"incarnation":61,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6835/2342195382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2342195382},{"type":"v1","addr":"172.21.15.202:6835","nonce":2342195382}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:05:05.274 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.272+0000 7fa72c168700 1 -- 172.21.15.195:0/4248162924 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa704063cd0 msgr2=0x7fa704066180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:05.274 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.272+0000 7fa72c168700 1 --2- 172.21.15.195:0/4248162924 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa704063cd0 0x7fa704066180 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fa724065b00 tx=0x7fa720017000 comp rx=0 tx=0).stop 2022-06-27T12:05:05.275 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.272+0000 7fa72c168700 1 -- 172.21.15.195:0/4248162924 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa7240c3d80 msgr2=0x7fa7240bff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:05.275 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.272+0000 7fa72c168700 1 --2- 172.21.15.195:0/4248162924 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa7240c3d80 0x7fa7240bff70 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fa71400d890 tx=0x7fa714008040 comp rx=0 tx=0).stop 2022-06-27T12:05:05.276 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.272+0000 7fa72c168700 1 -- 172.21.15.195:0/4248162924 shutdown_connections 2022-06-27T12:05:05.276 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.272+0000 7fa72c168700 1 --2- 172.21.15.195:0/4248162924 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa704063cd0 0x7fa704066180 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:05.277 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.272+0000 7fa72c168700 1 --2- 172.21.15.195:0/4248162924 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa7240c3d80 0x7fa7240bff70 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:05.277 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.272+0000 7fa72c168700 1 --2- 172.21.15.195:0/4248162924 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa72414e860 0x7fa724151600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:05.278 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.272+0000 7fa72c168700 1 --2- 172.21.15.195:0/4248162924 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa72414e490 0x7fa7241510c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:05.278 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.272+0000 7fa72c168700 1 -- 172.21.15.195:0/4248162924 >> 172.21.15.195:0/4248162924 conn(0x7fa724063af0 msgr2=0x7fa7240c8880 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:05:05.279 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.273+0000 7fa72c168700 1 -- 172.21.15.195:0/4248162924 shutdown_connections 2022-06-27T12:05:05.279 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.273+0000 7fa72c168700 1 -- 172.21.15.195:0/4248162924 wait complete. 2022-06-27T12:05:05.281 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 61 2022-06-27T12:05:05.289 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2022-06-27T12:05:05.289 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2022-06-27T12:05:05.290 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:05:05.458 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.448+0000 7f874e8cf700 1 Processor -- start 2022-06-27T12:05:05.459 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.448+0000 7f874e8cf700 1 -- start start 2022-06-27T12:05:05.459 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.449+0000 7f874e8cf700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f87480c3f50 0x7f874815ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:05.459 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.449+0000 7f874e8cf700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87480c33b0 con 0x7f874814e490 2022-06-27T12:05:05.460 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.449+0000 7f874e8cf700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f874806f220 con 0x7f874814e860 2022-06-27T12:05:05.460 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.449+0000 7f874e8cf700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87481542c0 con 0x7f87480c3f50 2022-06-27T12:05:05.460 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.449+0000 7f8747fff700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f874814e490 0x7f874806cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:53980/0 (socket says 172.21.15.195:53980) 2022-06-27T12:05:05.460 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.449+0000 7f8747fff700 1 -- 172.21.15.195:0/2181663312 learned_addr learned my addr 172.21.15.195:0/2181663312 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:05:05.461 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.449+0000 7f874ce6c700 1 --2- 172.21.15.195:0/2181663312 >> v2:172.21.15.202:3300/0 conn(0x7f87480c3f50 0x7f874815ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:05.461 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.449+0000 7f8746ffd700 1 -- 172.21.15.195:0/2181663312 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2489060418 0 0) 0x7f87480c33b0 con 0x7f874814e490 2022-06-27T12:05:05.461 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.449+0000 7f8746ffd700 1 -- 172.21.15.195:0/2181663312 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8734003430 con 0x7f874814e490 2022-06-27T12:05:05.461 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.449+0000 7f8746ffd700 1 -- 172.21.15.195:0/2181663312 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1510962365 0 0) 0x7f8734003430 con 0x7f874814e490 2022-06-27T12:05:05.461 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.449+0000 7f8746ffd700 1 -- 172.21.15.195:0/2181663312 >> v2:172.21.15.202:3300/0 conn(0x7f87480c3f50 msgr2=0x7f874815ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:05:05.462 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.449+0000 7f8746ffd700 1 --2- 172.21.15.195:0/2181663312 >> v2:172.21.15.202:3300/0 conn(0x7f87480c3f50 0x7f874815ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:05.462 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.449+0000 7f8746ffd700 1 -- 172.21.15.195:0/2181663312 >> v1:172.21.15.202:6789/0 conn(0x7f874814e860 legacy=0x7f87480c3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:05.462 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.449+0000 7f8746ffd700 1 -- 172.21.15.195:0/2181663312 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8748154560 con 0x7f874814e490 2022-06-27T12:05:05.463 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.449+0000 7f8746ffd700 1 -- 172.21.15.195:0/2181663312 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7f873c001470 con 0x7f874814e490 2022-06-27T12:05:05.463 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.449+0000 7f8746ffd700 1 -- 172.21.15.195:0/2181663312 >> v1:172.21.15.195:6789/0 conn(0x7f874814e490 legacy=0x7f874806cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:05.464 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.450+0000 7f8746ffd700 1 --2- 172.21.15.195:0/2181663312 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8734003920 0x7f8734003cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:05.464 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.450+0000 7f8746ffd700 1 --2- 172.21.15.195:0/2181663312 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f87340046d0 0x7f8734004b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:05.464 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.450+0000 7f8746ffd700 1 --2- 172.21.15.195:0/2181663312 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8734005080 0x7f873400b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:05.465 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.450+0000 7f8746ffd700 1 -- 172.21.15.195:0/2181663312 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87481542c0 con 0x7f87340046d0 2022-06-27T12:05:05.465 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.450+0000 7f8746ffd700 1 -- 172.21.15.195:0/2181663312 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f873400bf10 con 0x7f8734005080 2022-06-27T12:05:05.466 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.450+0000 7f8746ffd700 1 -- 172.21.15.195:0/2181663312 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f873400c0d0 con 0x7f8734003920 2022-06-27T12:05:05.466 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.450+0000 7f874ce6c700 1 --2- 172.21.15.195:0/2181663312 >> v2:172.21.15.202:3300/0 conn(0x7f87480c3f50 0x7f874815ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:05:05.466 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.450+0000 7f87477fe700 1 --2- 172.21.15.195:0/2181663312 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f87340046d0 0x7f8734004b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:05.466 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.450+0000 7f8747fff700 1 --2- 172.21.15.195:0/2181663312 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8734003920 0x7f8734003cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:05.467 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.450+0000 7f874ce6c700 1 --2- 172.21.15.195:0/2181663312 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8734005080 0x7f873400b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:05.467 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.450+0000 7f87477fe700 1 -- 172.21.15.195:0/2181663312 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8734003920 msgr2=0x7f8734003cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:05:05.467 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.450+0000 7f87477fe700 1 --2- 172.21.15.195:0/2181663312 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8734003920 0x7f8734003cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:05.467 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.450+0000 7f87477fe700 1 -- 172.21.15.195:0/2181663312 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8734005080 msgr2=0x7f873400b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:05:05.468 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.450+0000 7f87477fe700 1 --2- 172.21.15.195:0/2181663312 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8734005080 0x7f873400b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:05.468 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.450+0000 7f87477fe700 1 -- 172.21.15.195:0/2181663312 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8734003430 con 0x7f87340046d0 2022-06-27T12:05:05.468 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.450+0000 7f874ce6c700 1 --2- 172.21.15.195:0/2181663312 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8734005080 0x7f873400b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:05:05.469 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.450+0000 7f874ce6c700 1 -- 172.21.15.195:0/2181663312 reap_dead start 2022-06-27T12:05:05.469 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.450+0000 7f87477fe700 1 --2- 172.21.15.195:0/2181663312 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f87340046d0 0x7f8734004b40 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f8738001860 tx=0x7f8738007bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ef564e545926550 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:05:05.469 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.450+0000 7f8746ffd700 1 -- 172.21.15.195:0/2181663312 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87380050d0 con 0x7f87340046d0 2022-06-27T12:05:05.470 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.450+0000 7f8746ffd700 1 -- 172.21.15.195:0/2181663312 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8738005230 con 0x7f87340046d0 2022-06-27T12:05:05.470 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.450+0000 7f8746ffd700 1 -- 172.21.15.195:0/2181663312 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8738003750 con 0x7f87340046d0 2022-06-27T12:05:05.471 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.451+0000 7f874e8cf700 1 -- 172.21.15.195:0/2181663312 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f87340046d0 msgr2=0x7f8734004b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:05.471 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.451+0000 7f874e8cf700 1 --2- 172.21.15.195:0/2181663312 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f87340046d0 0x7f8734004b40 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f8738001860 tx=0x7f8738007bd0 comp rx=0 tx=0).stop 2022-06-27T12:05:05.471 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.451+0000 7f874e8cf700 1 -- 172.21.15.195:0/2181663312 shutdown_connections 2022-06-27T12:05:05.471 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.451+0000 7f874e8cf700 1 --2- 172.21.15.195:0/2181663312 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f87340046d0 0x7f8734004b40 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:05.472 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.451+0000 7f874e8cf700 1 -- 172.21.15.195:0/2181663312 >> 172.21.15.195:0/2181663312 conn(0x7f8748063af0 msgr2=0x7f8748063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:05:05.472 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.451+0000 7f874e8cf700 1 -- 172.21.15.195:0/2181663312 shutdown_connections 2022-06-27T12:05:05.472 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.451+0000 7f874e8cf700 1 -- 172.21.15.195:0/2181663312 wait complete. 2022-06-27T12:05:05.472 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.452+0000 7f874e8cf700 1 Processor -- start 2022-06-27T12:05:05.473 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.452+0000 7f874e8cf700 1 -- start start 2022-06-27T12:05:05.473 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.453+0000 7f874e8cf700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f87340046d0 0x7f8748156f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:05.473 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.453+0000 7f874e8cf700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f87481574c0 0x7f874814fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:05.474 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.453+0000 7f874e8cf700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8748159ca0 0x7f8748150240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:05.474 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.453+0000 7f874e8cf700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8748158220 con 0x7f87481574c0 2022-06-27T12:05:05.474 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.453+0000 7f874e8cf700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f874806aa00 con 0x7f8748159ca0 2022-06-27T12:05:05.474 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.453+0000 7f874e8cf700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f874814f870 con 0x7f87340046d0 2022-06-27T12:05:05.475 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.453+0000 7f87477fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f87481574c0 0x7f874814fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:05.475 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.453+0000 7f87477fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f87481574c0 0x7f874814fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47806/0 (socket says 172.21.15.195:47806) 2022-06-27T12:05:05.476 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.453+0000 7f87477fe700 1 -- 172.21.15.195:0/1112455229 learned_addr learned my addr 172.21.15.195:0/1112455229 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:05:05.476 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.453+0000 7f8747fff700 1 --2- 172.21.15.195:0/1112455229 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f87340046d0 0x7f8748156f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:05.477 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.453+0000 7f874ce6c700 1 --2- 172.21.15.195:0/1112455229 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8748159ca0 0x7f8748150240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:05.477 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.453+0000 7f87477fe700 1 -- 172.21.15.195:0/1112455229 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f87340046d0 msgr2=0x7f8748156f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:05.478 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.453+0000 7f87477fe700 1 --2- 172.21.15.195:0/1112455229 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f87340046d0 0x7f8748156f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:05.478 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.453+0000 7f87477fe700 1 -- 172.21.15.195:0/1112455229 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8748159ca0 msgr2=0x7f8748150240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:05.479 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.453+0000 7f87477fe700 1 --2- 172.21.15.195:0/1112455229 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8748159ca0 0x7f8748150240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:05.479 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.453+0000 7f87477fe700 1 -- 172.21.15.195:0/1112455229 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8748157950 con 0x7f87481574c0 2022-06-27T12:05:05.479 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.453+0000 7f87477fe700 1 --2- 172.21.15.195:0/1112455229 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f87481574c0 0x7f874814fd00 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f8738000e40 tx=0x7f8738015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:05:05.480 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.453+0000 7f872ffff700 1 -- 172.21.15.195:0/1112455229 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8738022070 con 0x7f87481574c0 2022-06-27T12:05:05.480 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.453+0000 7f872ffff700 1 -- 172.21.15.195:0/1112455229 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f873801e640 con 0x7f87481574c0 2022-06-27T12:05:05.481 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.453+0000 7f874e8cf700 1 -- 172.21.15.195:0/1112455229 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8748157bf0 con 0x7f87481574c0 2022-06-27T12:05:05.481 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.453+0000 7f872ffff700 1 -- 172.21.15.195:0/1112455229 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f873801ea00 con 0x7f87481574c0 2022-06-27T12:05:05.482 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.453+0000 7f874e8cf700 1 -- 172.21.15.195:0/1112455229 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f874806af30 con 0x7f87481574c0 2022-06-27T12:05:05.482 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.454+0000 7f874e8cf700 1 -- 172.21.15.195:0/1112455229 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87480722c0 con 0x7f87481574c0 2022-06-27T12:05:05.483 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.455+0000 7f872ffff700 1 -- 172.21.15.195:0/1112455229 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f8738010070 con 0x7f87481574c0 2022-06-27T12:05:05.483 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.455+0000 7f872ffff700 1 --2- 172.21.15.195:0/1112455229 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8728063d30 0x7f87280661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:05.484 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.455+0000 7f872ffff700 1 -- 172.21.15.195:0/1112455229 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6976+0+0 (secure 0 0 0) 0x7f8738081450 con 0x7f87481574c0 2022-06-27T12:05:05.484 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.455+0000 7f8747fff700 1 --2- 172.21.15.195:0/1112455229 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8728063d30 0x7f87280661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:05.484 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.455+0000 7f8747fff700 1 --2- 172.21.15.195:0/1112455229 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8728063d30 0x7f87280661e0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f873c001200 tx=0x7f87480c3220 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:05:05.485 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.459+0000 7f872ffff700 1 -- 172.21.15.195:0/1112455229 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f873804fd70 con 0x7f87481574c0 2022-06-27T12:05:05.692 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.682+0000 7f874e8cf700 1 -- 172.21.15.195:0/1112455229 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87480c60d0 con 0x7f87481574c0 2022-06-27T12:05:05.693 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.683+0000 7f872ffff700 1 -- 172.21.15.195:0/1112455229 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+5367 (secure 0 0 0) 0x7f8738017090 con 0x7f87481574c0 2022-06-27T12:05:05.694 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:05:05.695 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:05:05.063331+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5489},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5489":{"gid":5489,"name":"d","rank":0,"incarnation":61,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6835/2342195382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2342195382},{"type":"v1","addr":"172.21.15.202:6835","nonce":2342195382}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:05:05.696 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.685+0000 7f874e8cf700 1 -- 172.21.15.195:0/1112455229 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8728063d30 msgr2=0x7f87280661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:05.697 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.685+0000 7f874e8cf700 1 --2- 172.21.15.195:0/1112455229 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8728063d30 0x7f87280661e0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f873c001200 tx=0x7f87480c3220 comp rx=0 tx=0).stop 2022-06-27T12:05:05.697 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.685+0000 7f874e8cf700 1 -- 172.21.15.195:0/1112455229 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f87481574c0 msgr2=0x7f874814fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:05.697 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.685+0000 7f874e8cf700 1 --2- 172.21.15.195:0/1112455229 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f87481574c0 0x7f874814fd00 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f8738000e40 tx=0x7f8738015000 comp rx=0 tx=0).stop 2022-06-27T12:05:05.698 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.685+0000 7f874e8cf700 1 -- 172.21.15.195:0/1112455229 shutdown_connections 2022-06-27T12:05:05.699 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.685+0000 7f874e8cf700 1 --2- 172.21.15.195:0/1112455229 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8728063d30 0x7f87280661e0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:05.699 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.685+0000 7f874e8cf700 1 --2- 172.21.15.195:0/1112455229 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8748159ca0 0x7f8748150240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:05.699 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.685+0000 7f874e8cf700 1 --2- 172.21.15.195:0/1112455229 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f87340046d0 0x7f8748156f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:05.699 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.685+0000 7f874e8cf700 1 --2- 172.21.15.195:0/1112455229 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f87481574c0 0x7f874814fd00 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:05.699 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.685+0000 7f874e8cf700 1 -- 172.21.15.195:0/1112455229 >> 172.21.15.195:0/1112455229 conn(0x7f8748063af0 msgr2=0x7f874814d6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:05:05.700 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.685+0000 7f874e8cf700 1 -- 172.21.15.195:0/1112455229 shutdown_connections 2022-06-27T12:05:05.700 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:05.685+0000 7f874e8cf700 1 -- 172.21.15.195:0/1112455229 wait complete. 2022-06-27T12:05:05.700 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 61 2022-06-27T12:05:05.702 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 61, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-06-27T11:47:52.358095+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 61, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5489": { "addr": "172.21.15.202:6835/2342195382", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6834", "nonce": 2342195382, "type": "v2" }, { "addr": "172.21.15.202:6835", "nonce": 2342195382, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5489, "incarnation": 61, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 57, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-06-27T12:05:05.063331+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5489 } } } ], "standbys": [ { "addr": "172.21.15.202:6837/3305006841", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6836", "nonce": 3305006841, "type": "v2" }, { "addr": "172.21.15.202:6837", "nonce": 3305006841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6835/2060709822", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6834", "nonce": 2060709822, "type": "v2" }, { "addr": "172.21.15.195:6835", "nonce": 2060709822, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6838/1598751652", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6836", "nonce": 1598751652, "type": "v2" }, { "addr": "172.21.15.195:6838", "nonce": 1598751652, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4275, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6839/1313800661", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6837", "nonce": 1313800661, "type": "v2" }, { "addr": "172.21.15.195:6839", "nonce": 1313800661, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4278, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-06-27T12:05:07.703 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:05:07.848 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.836+0000 7fbdd06c8700 1 Processor -- start 2022-06-27T12:05:07.848 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.836+0000 7fbdd06c8700 1 -- start start 2022-06-27T12:05:07.848 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.837+0000 7fbdd06c8700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fbdc806a450 0x7fbdc8063af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:07.849 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.837+0000 7fbdd06c8700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdc8064030 con 0x7fbdc81508d0 2022-06-27T12:05:07.849 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.837+0000 7fbdd06c8700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdc80c4160 con 0x7fbdc8150500 2022-06-27T12:05:07.849 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.837+0000 7fbdd06c8700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdc80c4440 con 0x7fbdc806a450 2022-06-27T12:05:07.849 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.838+0000 7fbdcec65700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fbdc81508d0 0x7fbdc80bbf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:54012/0 (socket says 172.21.15.195:54012) 2022-06-27T12:05:07.850 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.838+0000 7fbdcec65700 1 -- 172.21.15.195:0/89550210 learned_addr learned my addr 172.21.15.195:0/89550210 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:05:07.850 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.838+0000 7fbdcdc63700 1 --2- 172.21.15.195:0/89550210 >> v2:172.21.15.202:3300/0 conn(0x7fbdc806a450 0x7fbdc8063af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:07.850 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.838+0000 7fbdcd462700 1 -- 172.21.15.195:0/89550210 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 797826705 0 0) 0x7fbdc0001490 con 0x7fbdc81508d0 2022-06-27T12:05:07.850 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.838+0000 7fbdcd462700 1 -- 172.21.15.195:0/89550210 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdb0003430 con 0x7fbdc81508d0 2022-06-27T12:05:07.850 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.838+0000 7fbdcd462700 1 -- 172.21.15.195:0/89550210 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1425706169 0 0) 0x7fbdb0003430 con 0x7fbdc81508d0 2022-06-27T12:05:07.851 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.838+0000 7fbdcd462700 1 -- 172.21.15.195:0/89550210 >> v2:172.21.15.202:3300/0 conn(0x7fbdc806a450 msgr2=0x7fbdc8063af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:05:07.851 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.838+0000 7fbdcd462700 1 --2- 172.21.15.195:0/89550210 >> v2:172.21.15.202:3300/0 conn(0x7fbdc806a450 0x7fbdc8063af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:07.851 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.838+0000 7fbdcd462700 1 -- 172.21.15.195:0/89550210 >> v1:172.21.15.202:6789/0 conn(0x7fbdc8150500 legacy=0x7fbdc80c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:07.851 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.838+0000 7fbdcd462700 1 -- 172.21.15.195:0/89550210 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdc815a800 con 0x7fbdc81508d0 2022-06-27T12:05:07.852 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.838+0000 7fbdcd462700 1 -- 172.21.15.195:0/89550210 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7fbdc0001980 con 0x7fbdc81508d0 2022-06-27T12:05:07.852 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.838+0000 7fbdcd462700 1 -- 172.21.15.195:0/89550210 >> v1:172.21.15.195:6789/0 conn(0x7fbdc81508d0 legacy=0x7fbdc80bbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:07.852 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.838+0000 7fbdcd462700 1 --2- 172.21.15.195:0/89550210 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbdb0003920 0x7fbdb0003cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:07.852 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.838+0000 7fbdcd462700 1 --2- 172.21.15.195:0/89550210 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbdb00046d0 0x7fbdb0004b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:07.853 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.838+0000 7fbdcd462700 1 --2- 172.21.15.195:0/89550210 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbdb0005080 0x7fbdb000b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:07.853 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.838+0000 7fbdcd462700 1 -- 172.21.15.195:0/89550210 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdc80c4440 con 0x7fbdb00046d0 2022-06-27T12:05:07.853 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.838+0000 7fbdcd462700 1 -- 172.21.15.195:0/89550210 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb000bf10 con 0x7fbdb0003920 2022-06-27T12:05:07.853 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.838+0000 7fbdcd462700 1 -- 172.21.15.195:0/89550210 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb000c0d0 con 0x7fbdb0005080 2022-06-27T12:05:07.854 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.838+0000 7fbdcdc63700 1 --2- 172.21.15.195:0/89550210 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbdb00046d0 0x7fbdb0004b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:07.854 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.838+0000 7fbdce464700 1 --2- 172.21.15.195:0/89550210 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbdb0003920 0x7fbdb0003cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:07.854 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.838+0000 7fbdcec65700 1 --2- 172.21.15.195:0/89550210 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbdb0005080 0x7fbdb000b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:07.854 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.839+0000 7fbdcdc63700 1 -- 172.21.15.195:0/89550210 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbdb0005080 msgr2=0x7fbdb000b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:05:07.854 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.839+0000 7fbdcdc63700 1 --2- 172.21.15.195:0/89550210 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbdb0005080 0x7fbdb000b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:07.855 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.839+0000 7fbdcdc63700 1 -- 172.21.15.195:0/89550210 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbdb0003920 msgr2=0x7fbdb0003cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:05:07.855 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.839+0000 7fbdcdc63700 1 --2- 172.21.15.195:0/89550210 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbdb0003920 0x7fbdb0003cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:07.855 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.839+0000 7fbdcdc63700 1 -- 172.21.15.195:0/89550210 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdb0003430 con 0x7fbdb00046d0 2022-06-27T12:05:07.855 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.839+0000 7fbdcec65700 1 --2- 172.21.15.195:0/89550210 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbdb0005080 0x7fbdb000b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:05:07.856 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.839+0000 7fbdcec65700 1 -- 172.21.15.195:0/89550210 reap_dead start 2022-06-27T12:05:07.856 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.839+0000 7fbdcdc63700 1 --2- 172.21.15.195:0/89550210 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbdb00046d0 0x7fbdb0004b40 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fbdc4009ac0 tx=0x7fbdc4009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bb5c01d23da57d server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:05:07.856 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.839+0000 7fbdcd462700 1 -- 172.21.15.195:0/89550210 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc4009470 con 0x7fbdb00046d0 2022-06-27T12:05:07.856 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.839+0000 7fbdcd462700 1 -- 172.21.15.195:0/89550210 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdc400e640 con 0x7fbdb00046d0 2022-06-27T12:05:07.856 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.839+0000 7fbdcd462700 1 -- 172.21.15.195:0/89550210 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc400e9e0 con 0x7fbdb00046d0 2022-06-27T12:05:07.857 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.839+0000 7fbdd06c8700 1 -- 172.21.15.195:0/89550210 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbdb00046d0 msgr2=0x7fbdb0004b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:07.857 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.839+0000 7fbdd06c8700 1 --2- 172.21.15.195:0/89550210 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbdb00046d0 0x7fbdb0004b40 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fbdc4009ac0 tx=0x7fbdc4009060 comp rx=0 tx=0).stop 2022-06-27T12:05:07.857 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.839+0000 7fbdd06c8700 1 -- 172.21.15.195:0/89550210 shutdown_connections 2022-06-27T12:05:07.857 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.840+0000 7fbdd06c8700 1 --2- 172.21.15.195:0/89550210 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbdb00046d0 0x7fbdb0004b40 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:07.858 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.840+0000 7fbdd06c8700 1 -- 172.21.15.195:0/89550210 >> 172.21.15.195:0/89550210 conn(0x7fbdc80c9b00 msgr2=0x7fbdc80c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:05:07.858 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.840+0000 7fbdd06c8700 1 -- 172.21.15.195:0/89550210 shutdown_connections 2022-06-27T12:05:07.858 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.840+0000 7fbdd06c8700 1 -- 172.21.15.195:0/89550210 wait complete. 2022-06-27T12:05:07.858 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.841+0000 7fbdd06c8700 1 Processor -- start 2022-06-27T12:05:07.859 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.841+0000 7fbdd06c8700 1 -- start start 2022-06-27T12:05:07.859 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.841+0000 7fbdd06c8700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbdb00046d0 0x7fbdc816a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:07.859 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.841+0000 7fbdd06c8700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbdc8157580 0x7fbdc80c7020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:07.859 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.841+0000 7fbdd06c8700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbdc80c7a40 0x7fbdc81590a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:07.859 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.841+0000 7fbdd06c8700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdc80c14c0 con 0x7fbdc8157580 2022-06-27T12:05:07.860 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.841+0000 7fbdd06c8700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdc8158930 con 0x7fbdb00046d0 2022-06-27T12:05:07.860 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.841+0000 7fbdd06c8700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdc8157a70 con 0x7fbdc80c7a40 2022-06-27T12:05:07.860 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.841+0000 7fbdcdc63700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbdc8157580 0x7fbdc80c7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:07.860 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.842+0000 7fbdcdc63700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbdc8157580 0x7fbdc80c7020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47834/0 (socket says 172.21.15.195:47834) 2022-06-27T12:05:07.861 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.842+0000 7fbdcdc63700 1 -- 172.21.15.195:0/1602132225 learned_addr learned my addr 172.21.15.195:0/1602132225 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:05:07.861 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.842+0000 7fbdce464700 1 --2- 172.21.15.195:0/1602132225 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbdb00046d0 0x7fbdc816a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:07.861 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.842+0000 7fbdcec65700 1 --2- 172.21.15.195:0/1602132225 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbdc80c7a40 0x7fbdc81590a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:07.861 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.842+0000 7fbdcdc63700 1 -- 172.21.15.195:0/1602132225 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbdc80c7a40 msgr2=0x7fbdc81590a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:07.861 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.842+0000 7fbdcdc63700 1 --2- 172.21.15.195:0/1602132225 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbdc80c7a40 0x7fbdc81590a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:07.862 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.842+0000 7fbdcdc63700 1 -- 172.21.15.195:0/1602132225 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbdb00046d0 msgr2=0x7fbdc816a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:07.862 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.842+0000 7fbdcdc63700 1 --2- 172.21.15.195:0/1602132225 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbdb00046d0 0x7fbdc816a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:07.862 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.842+0000 7fbdcdc63700 1 -- 172.21.15.195:0/1602132225 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdc8157d50 con 0x7fbdc8157580 2022-06-27T12:05:07.863 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.842+0000 7fbdcdc63700 1 --2- 172.21.15.195:0/1602132225 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbdc8157580 0x7fbdc80c7020 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fbdc40045a0 tx=0x7fbdc401a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:05:07.863 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.842+0000 7fbdb67fc700 1 -- 172.21.15.195:0/1602132225 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc4021070 con 0x7fbdc8157580 2022-06-27T12:05:07.863 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.842+0000 7fbdb67fc700 1 -- 172.21.15.195:0/1602132225 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdc4003520 con 0x7fbdc8157580 2022-06-27T12:05:07.863 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.842+0000 7fbdd06c8700 1 -- 172.21.15.195:0/1602132225 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdc8158030 con 0x7fbdc8157580 2022-06-27T12:05:07.864 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.842+0000 7fbdb67fc700 1 -- 172.21.15.195:0/1602132225 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc40038c0 con 0x7fbdc8157580 2022-06-27T12:05:07.864 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.842+0000 7fbdd06c8700 1 -- 172.21.15.195:0/1602132225 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdc8158b80 con 0x7fbdc8157580 2022-06-27T12:05:07.864 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.843+0000 7fbdd06c8700 1 -- 172.21.15.195:0/1602132225 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdc8158ef0 con 0x7fbdc8157580 2022-06-27T12:05:07.864 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.843+0000 7fbdb67fc700 1 -- 172.21.15.195:0/1602132225 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fbdc4019030 con 0x7fbdc8157580 2022-06-27T12:05:07.865 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.844+0000 7fbdb67fc700 1 --2- 172.21.15.195:0/1602132225 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbda8063d00 0x7fbda80661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:07.866 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.844+0000 7fbdb67fc700 1 -- 172.21.15.195:0/1602132225 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6976+0+0 (secure 0 0 0) 0x7fbdc4082140 con 0x7fbdc8157580 2022-06-27T12:05:07.866 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.844+0000 7fbdce464700 1 --2- 172.21.15.195:0/1602132225 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbda8063d00 0x7fbda80661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:07.866 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.844+0000 7fbdce464700 1 --2- 172.21.15.195:0/1602132225 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbda8063d00 0x7fbda80661b0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fbdb8007fd0 tx=0x7fbdb8009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:05:07.868 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:07.848+0000 7fbdb67fc700 1 -- 172.21.15.195:0/1602132225 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fbdc4050a60 con 0x7fbdc8157580 2022-06-27T12:05:08.077 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:08.076+0000 7fbdd06c8700 1 -- 172.21.15.195:0/1602132225 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbdc80722c0 con 0x7fbdc8157580 2022-06-27T12:05:08.078 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:08.077+0000 7fbdb67fc700 1 -- 172.21.15.195:0/1602132225 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+5367 (secure 0 0 0) 0x7fbdc4017090 con 0x7fbdc8157580 2022-06-27T12:05:08.079 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:05:08.079 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:05:05.063331+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5489},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5489":{"gid":5489,"name":"d","rank":0,"incarnation":61,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6835/2342195382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2342195382},{"type":"v1","addr":"172.21.15.202:6835","nonce":2342195382}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:05:08.081 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:08.079+0000 7fbdd06c8700 1 -- 172.21.15.195:0/1602132225 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbda8063d00 msgr2=0x7fbda80661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:08.082 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:08.079+0000 7fbdd06c8700 1 --2- 172.21.15.195:0/1602132225 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbda8063d00 0x7fbda80661b0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fbdb8007fd0 tx=0x7fbdb8009040 comp rx=0 tx=0).stop 2022-06-27T12:05:08.082 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:08.079+0000 7fbdd06c8700 1 -- 172.21.15.195:0/1602132225 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbdc8157580 msgr2=0x7fbdc80c7020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:08.083 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:08.079+0000 7fbdd06c8700 1 --2- 172.21.15.195:0/1602132225 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbdc8157580 0x7fbdc80c7020 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fbdc40045a0 tx=0x7fbdc401a000 comp rx=0 tx=0).stop 2022-06-27T12:05:08.083 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:08.079+0000 7fbdd06c8700 1 -- 172.21.15.195:0/1602132225 shutdown_connections 2022-06-27T12:05:08.083 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:08.079+0000 7fbdd06c8700 1 --2- 172.21.15.195:0/1602132225 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbda8063d00 0x7fbda80661b0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:08.084 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:08.079+0000 7fbdd06c8700 1 --2- 172.21.15.195:0/1602132225 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbdc80c7a40 0x7fbdc81590a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:08.084 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:08.079+0000 7fbdd06c8700 1 --2- 172.21.15.195:0/1602132225 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbdb00046d0 0x7fbdc816a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:08.085 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:08.079+0000 7fbdd06c8700 1 --2- 172.21.15.195:0/1602132225 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbdc8157580 0x7fbdc80c7020 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:08.085 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:08.079+0000 7fbdd06c8700 1 -- 172.21.15.195:0/1602132225 >> 172.21.15.195:0/1602132225 conn(0x7fbdc80c9b00 msgr2=0x7fbdc814fd70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:05:08.086 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:08.079+0000 7fbdd06c8700 1 -- 172.21.15.195:0/1602132225 shutdown_connections 2022-06-27T12:05:08.086 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:08.079+0000 7fbdd06c8700 1 -- 172.21.15.195:0/1602132225 wait complete. 2022-06-27T12:05:08.087 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 61 2022-06-27T12:05:08.092 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:05:10.094 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:05:10.236 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.223+0000 7f5a06952700 1 Processor -- start 2022-06-27T12:05:10.236 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.223+0000 7f5a06952700 1 -- start start 2022-06-27T12:05:10.237 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.224+0000 7f5a06952700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f5a000c4160 0x7f5a00156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:10.237 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.224+0000 7f5a06952700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a000c1a20 con 0x7f5a000c39e0 2022-06-27T12:05:10.238 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.224+0000 7f5a06952700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a000c1130 con 0x7f5a0006f640 2022-06-27T12:05:10.238 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.224+0000 7f5a06952700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a0006ea90 con 0x7f5a000c4160 2022-06-27T12:05:10.238 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.224+0000 7f59ff7fe700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f5a000c4160 0x7f5a00156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:10.239 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.224+0000 7f59ff7fe700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f5a000c4160 0x7f5a00156ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47852/0 (socket says 172.21.15.195:47852) 2022-06-27T12:05:10.240 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.224+0000 7f59ff7fe700 1 -- 172.21.15.195:0/3445766144 learned_addr learned my addr 172.21.15.195:0/3445766144 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:05:10.240 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.224+0000 7f5a04eef700 1 --1- 172.21.15.195:0/3445766144 >> v1:172.21.15.195:6789/0 conn(0x7f5a000c39e0 0x7f5a001535a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:54042/0 (socket says 172.21.15.195:54042) 2022-06-27T12:05:10.240 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.225+0000 7f59ff7fe700 1 -- 172.21.15.195:0/3445766144 >> v1:172.21.15.202:6789/0 conn(0x7f5a0006f640 legacy=0x7f5a000c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:10.241 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.225+0000 7f59ff7fe700 1 -- 172.21.15.195:0/3445766144 >> v1:172.21.15.195:6789/0 conn(0x7f5a000c39e0 legacy=0x7f5a001535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:10.241 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.225+0000 7f59feffd700 1 -- 172.21.15.195:0/3445766144 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2799356571 0 0) 0x7f5a000c1a20 con 0x7f5a000c39e0 2022-06-27T12:05:10.241 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.225+0000 7f59ff7fe700 1 -- 172.21.15.195:0/3445766144 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a0006ed70 con 0x7f5a000c4160 2022-06-27T12:05:10.241 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.225+0000 7f59ff7fe700 1 --2- 172.21.15.195:0/3445766144 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5a000c4160 0x7f5a00156ed0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f59f0009860 tx=0x7f59f0009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3da3a6098f45df6b server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:05:10.242 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.225+0000 7f59feffd700 1 -- 172.21.15.195:0/3445766144 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f000b710 con 0x7f5a000c4160 2022-06-27T12:05:10.242 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.225+0000 7f59feffd700 1 -- 172.21.15.195:0/3445766144 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59f000b870 con 0x7f5a000c4160 2022-06-27T12:05:10.243 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.225+0000 7f59feffd700 1 -- 172.21.15.195:0/3445766144 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f000bc30 con 0x7f5a000c4160 2022-06-27T12:05:10.243 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.225+0000 7f5a06952700 1 -- 172.21.15.195:0/3445766144 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5a000c4160 msgr2=0x7f5a00156ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:10.243 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.225+0000 7f5a06952700 1 --2- 172.21.15.195:0/3445766144 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5a000c4160 0x7f5a00156ed0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f59f0009860 tx=0x7f59f0009120 comp rx=0 tx=0).stop 2022-06-27T12:05:10.244 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.225+0000 7f5a06952700 1 -- 172.21.15.195:0/3445766144 shutdown_connections 2022-06-27T12:05:10.244 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.225+0000 7f5a06952700 1 --2- 172.21.15.195:0/3445766144 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5a000c4160 0x7f5a00156ed0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:10.245 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.225+0000 7f5a06952700 1 -- 172.21.15.195:0/3445766144 >> 172.21.15.195:0/3445766144 conn(0x7f5a000c6350 msgr2=0x7f5a000c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:05:10.245 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.225+0000 7f5a06952700 1 -- 172.21.15.195:0/3445766144 shutdown_connections 2022-06-27T12:05:10.245 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.225+0000 7f5a06952700 1 -- 172.21.15.195:0/3445766144 wait complete. 2022-06-27T12:05:10.246 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.227+0000 7f5a06952700 1 Processor -- start 2022-06-27T12:05:10.246 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.227+0000 7f5a06952700 1 -- start start 2022-06-27T12:05:10.246 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.227+0000 7f5a06952700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5a0006f640 0x7f5a00155d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:10.246 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.228+0000 7f5a06952700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5a000c39e0 0x7f5a001545b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:10.247 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.228+0000 7f5a06952700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5a000c4160 0x7f5a00154af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:10.247 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.228+0000 7f5a06952700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a00154120 con 0x7f5a000c39e0 2022-06-27T12:05:10.247 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.228+0000 7f5a06952700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a000bb480 con 0x7f5a000c4160 2022-06-27T12:05:10.247 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.228+0000 7f5a06952700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a000bce60 con 0x7f5a0006f640 2022-06-27T12:05:10.248 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.228+0000 7f59ff7fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5a000c39e0 0x7f5a001545b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:10.248 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.228+0000 7f59ff7fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5a000c39e0 0x7f5a001545b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47858/0 (socket says 172.21.15.195:47858) 2022-06-27T12:05:10.248 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.228+0000 7f59ff7fe700 1 -- 172.21.15.195:0/384488261 learned_addr learned my addr 172.21.15.195:0/384488261 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:05:10.248 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.228+0000 7f5a04eef700 1 --2- 172.21.15.195:0/384488261 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5a000c4160 0x7f5a00154af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:10.249 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.228+0000 7f59fffff700 1 --2- 172.21.15.195:0/384488261 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5a0006f640 0x7f5a00155d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:10.249 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.228+0000 7f59ff7fe700 1 -- 172.21.15.195:0/384488261 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5a0006f640 msgr2=0x7f5a00155d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:10.250 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.228+0000 7f59ff7fe700 1 --2- 172.21.15.195:0/384488261 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5a0006f640 0x7f5a00155d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:10.250 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.228+0000 7f59ff7fe700 1 -- 172.21.15.195:0/384488261 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5a000c4160 msgr2=0x7f5a00154af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:10.251 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.228+0000 7f59ff7fe700 1 --2- 172.21.15.195:0/384488261 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5a000c4160 0x7f5a00154af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:10.251 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.228+0000 7f59ff7fe700 1 -- 172.21.15.195:0/384488261 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59f00093a0 con 0x7f5a000c39e0 2022-06-27T12:05:10.251 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.228+0000 7f59ff7fe700 1 --2- 172.21.15.195:0/384488261 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5a000c39e0 0x7f5a001545b0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f5a000650f0 tx=0x7f59f0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:05:10.252 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.228+0000 7f59f7fff700 1 -- 172.21.15.195:0/384488261 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f0027070 con 0x7f5a000c39e0 2022-06-27T12:05:10.253 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.228+0000 7f59f7fff700 1 -- 172.21.15.195:0/384488261 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59f00118d0 con 0x7f5a000c39e0 2022-06-27T12:05:10.253 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.228+0000 7f59f7fff700 1 -- 172.21.15.195:0/384488261 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f0004590 con 0x7f5a000c39e0 2022-06-27T12:05:10.253 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.228+0000 7f5a06952700 1 -- 172.21.15.195:0/384488261 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a000bd140 con 0x7f5a000c39e0 2022-06-27T12:05:10.254 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.228+0000 7f5a06952700 1 -- 172.21.15.195:0/384488261 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a000bb760 con 0x7f5a000c39e0 2022-06-27T12:05:10.254 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.229+0000 7f5a06952700 1 -- 172.21.15.195:0/384488261 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a00154290 con 0x7f5a000c39e0 2022-06-27T12:05:10.255 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.230+0000 7f59f7fff700 1 -- 172.21.15.195:0/384488261 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f59f0034030 con 0x7f5a000c39e0 2022-06-27T12:05:10.255 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.230+0000 7f59f7fff700 1 --2- 172.21.15.195:0/384488261 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f59e4063d00 0x7f59e40661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:10.255 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.230+0000 7f59f7fff700 1 -- 172.21.15.195:0/384488261 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6976+0+0 (secure 0 0 0) 0x7f59f008a8e0 con 0x7f5a000c39e0 2022-06-27T12:05:10.256 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.230+0000 7f59fffff700 1 --2- 172.21.15.195:0/384488261 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f59e4063d00 0x7f59e40661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:10.256 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.230+0000 7f59fffff700 1 --2- 172.21.15.195:0/384488261 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f59e4063d00 0x7f59e40661b0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f59e8001880 tx=0x7f59e8009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:05:10.257 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.234+0000 7f59f7fff700 1 -- 172.21.15.195:0/384488261 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f59f0059140 con 0x7f5a000c39e0 2022-06-27T12:05:10.466 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.464+0000 7f5a06952700 1 -- 172.21.15.195:0/384488261 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a000722c0 con 0x7f5a000c39e0 2022-06-27T12:05:10.467 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.464+0000 7f59f7fff700 1 -- 172.21.15.195:0/384488261 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+5367 (secure 0 0 0) 0x7f59f0032090 con 0x7f5a000c39e0 2022-06-27T12:05:10.467 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:05:10.468 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:05:05.063331+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5489},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5489":{"gid":5489,"name":"d","rank":0,"incarnation":61,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6835/2342195382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2342195382},{"type":"v1","addr":"172.21.15.202:6835","nonce":2342195382}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:05:10.469 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.466+0000 7f5a06952700 1 -- 172.21.15.195:0/384488261 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f59e4063d00 msgr2=0x7f59e40661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:10.469 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.466+0000 7f5a06952700 1 --2- 172.21.15.195:0/384488261 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f59e4063d00 0x7f59e40661b0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f59e8001880 tx=0x7f59e8009040 comp rx=0 tx=0).stop 2022-06-27T12:05:10.470 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.466+0000 7f5a06952700 1 -- 172.21.15.195:0/384488261 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5a000c39e0 msgr2=0x7f5a001545b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:10.470 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.466+0000 7f5a06952700 1 --2- 172.21.15.195:0/384488261 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5a000c39e0 0x7f5a001545b0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f5a000650f0 tx=0x7f59f0017000 comp rx=0 tx=0).stop 2022-06-27T12:05:10.470 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.466+0000 7f5a06952700 1 -- 172.21.15.195:0/384488261 shutdown_connections 2022-06-27T12:05:10.470 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.466+0000 7f5a06952700 1 --2- 172.21.15.195:0/384488261 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f59e4063d00 0x7f59e40661b0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:10.471 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.466+0000 7f5a06952700 1 --2- 172.21.15.195:0/384488261 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5a000c4160 0x7f5a00154af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:10.471 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.466+0000 7f5a06952700 1 --2- 172.21.15.195:0/384488261 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5a0006f640 0x7f5a00155d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:10.471 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.466+0000 7f5a06952700 1 --2- 172.21.15.195:0/384488261 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5a000c39e0 0x7f5a001545b0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:10.471 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.466+0000 7f5a06952700 1 -- 172.21.15.195:0/384488261 >> 172.21.15.195:0/384488261 conn(0x7f5a000c6350 msgr2=0x7f5a0006cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:05:10.472 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.466+0000 7f5a06952700 1 -- 172.21.15.195:0/384488261 shutdown_connections 2022-06-27T12:05:10.472 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:10.466+0000 7f5a06952700 1 -- 172.21.15.195:0/384488261 wait complete. 2022-06-27T12:05:10.472 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 61 2022-06-27T12:05:10.478 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:05:12.482 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:05:12.622 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.622+0000 7ff5234a9700 1 Processor -- start 2022-06-27T12:05:12.623 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.623+0000 7ff5234a9700 1 -- start start 2022-06-27T12:05:12.631 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.623+0000 7ff5234a9700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7ff51c06f640 0x7ff51c156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:12.632 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.623+0000 7ff5234a9700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7ff51c0c39e0 0x7ff51c1535a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:12.632 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.623+0000 7ff5234a9700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff51c0c1a20 con 0x7ff51c0c4160 2022-06-27T12:05:12.632 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.623+0000 7ff5234a9700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff51c0c1130 con 0x7ff51c0c39e0 2022-06-27T12:05:12.632 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.623+0000 7ff5234a9700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff51c06ea90 con 0x7ff51c06f640 2022-06-27T12:05:12.633 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.623+0000 7ff521a46700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7ff51c0c39e0 0x7ff51c1535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:12.633 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.624+0000 7ff521a46700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7ff51c0c39e0 0x7ff51c1535a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47868/0 (socket says 172.21.15.195:47868) 2022-06-27T12:05:12.633 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.624+0000 7ff521a46700 1 -- 172.21.15.195:0/3641933058 learned_addr learned my addr 172.21.15.195:0/3641933058 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:05:12.633 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.624+0000 7ff521245700 1 --2- 172.21.15.195:0/3641933058 >> v2:172.21.15.202:3300/0 conn(0x7ff51c06f640 0x7ff51c156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:12.634 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.624+0000 7ff521a46700 1 -- 172.21.15.195:0/3641933058 >> v2:172.21.15.202:3300/0 conn(0x7ff51c06f640 msgr2=0x7ff51c156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:05:12.634 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.624+0000 7ff521a46700 1 --2- 172.21.15.195:0/3641933058 >> v2:172.21.15.202:3300/0 conn(0x7ff51c06f640 0x7ff51c156ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:12.635 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.624+0000 7ff521a46700 1 -- 172.21.15.195:0/3641933058 >> v1:172.21.15.202:6789/0 conn(0x7ff51c0c4160 legacy=0x7ff51c0c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:12.635 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.624+0000 7ff521a46700 1 -- 172.21.15.195:0/3641933058 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff51c06ed70 con 0x7ff51c0c39e0 2022-06-27T12:05:12.635 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.624+0000 7ff521a46700 1 --2- 172.21.15.195:0/3641933058 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff51c0c39e0 0x7ff51c1535a0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7ff51c06db30 tx=0x7ff514009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6859d17420589c5 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:05:12.635 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.624+0000 7ff50bfff700 1 -- 172.21.15.195:0/3641933058 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff51400e070 con 0x7ff51c0c39e0 2022-06-27T12:05:12.636 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.624+0000 7ff50bfff700 1 -- 172.21.15.195:0/3641933058 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff51400bab0 con 0x7ff51c0c39e0 2022-06-27T12:05:12.636 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.624+0000 7ff50bfff700 1 -- 172.21.15.195:0/3641933058 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5140092e0 con 0x7ff51c0c39e0 2022-06-27T12:05:12.636 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.625+0000 7ff5234a9700 1 -- 172.21.15.195:0/3641933058 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff51c0c39e0 msgr2=0x7ff51c1535a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:12.637 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.625+0000 7ff5234a9700 1 --2- 172.21.15.195:0/3641933058 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff51c0c39e0 0x7ff51c1535a0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7ff51c06db30 tx=0x7ff514009840 comp rx=0 tx=0).stop 2022-06-27T12:05:12.637 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.625+0000 7ff5234a9700 1 -- 172.21.15.195:0/3641933058 shutdown_connections 2022-06-27T12:05:12.637 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.625+0000 7ff5234a9700 1 --2- 172.21.15.195:0/3641933058 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff51c0c39e0 0x7ff51c1535a0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:12.638 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.625+0000 7ff5234a9700 1 --2- 172.21.15.195:0/3641933058 >> v2:172.21.15.202:3300/0 conn(0x7ff51c06f640 0x7ff51c156ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:12.638 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.625+0000 7ff5234a9700 1 -- 172.21.15.195:0/3641933058 >> 172.21.15.195:0/3641933058 conn(0x7ff51c0c6350 msgr2=0x7ff51c0c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:05:12.638 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.625+0000 7ff5234a9700 1 -- 172.21.15.195:0/3641933058 shutdown_connections 2022-06-27T12:05:12.638 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.625+0000 7ff5234a9700 1 -- 172.21.15.195:0/3641933058 wait complete. 2022-06-27T12:05:12.639 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.626+0000 7ff5234a9700 1 Processor -- start 2022-06-27T12:05:12.639 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.626+0000 7ff5234a9700 1 -- start start 2022-06-27T12:05:12.640 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.627+0000 7ff5234a9700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff51c06f640 0x7ff51c16aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:12.640 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.627+0000 7ff5234a9700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff51c0c39e0 0x7ff51c1514b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:12.641 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.627+0000 7ff5234a9700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff51c0c4160 0x7ff51c159b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:12.642 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.627+0000 7ff5234a9700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff51c152290 con 0x7ff51c0c4160 2022-06-27T12:05:12.642 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.627+0000 7ff5234a9700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff51c14c300 con 0x7ff51c06f640 2022-06-27T12:05:12.642 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.627+0000 7ff5234a9700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff51c14c120 con 0x7ff51c0c39e0 2022-06-27T12:05:12.643 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.627+0000 7ff521245700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff51c06f640 0x7ff51c16aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:12.643 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.627+0000 7ff521245700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff51c06f640 0x7ff51c16aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3300/0 says I am v2:172.21.15.195:45996/0 (socket says 172.21.15.195:45996) 2022-06-27T12:05:12.643 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.627+0000 7ff521245700 1 -- 172.21.15.195:0/840039965 learned_addr learned my addr 172.21.15.195:0/840039965 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:05:12.644 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.627+0000 7ff521a46700 1 --2- 172.21.15.195:0/840039965 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff51c0c4160 0x7ff51c159b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:12.644 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.627+0000 7ff520a44700 1 --2- 172.21.15.195:0/840039965 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff51c0c39e0 0x7ff51c1514b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:12.645 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.627+0000 7ff521a46700 1 -- 172.21.15.195:0/840039965 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff51c0c39e0 msgr2=0x7ff51c1514b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:12.645 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.627+0000 7ff521a46700 1 --2- 172.21.15.195:0/840039965 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff51c0c39e0 0x7ff51c1514b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:12.645 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.627+0000 7ff521a46700 1 -- 172.21.15.195:0/840039965 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff51c06f640 msgr2=0x7ff51c16aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:12.646 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.627+0000 7ff521a46700 1 --2- 172.21.15.195:0/840039965 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff51c06f640 0x7ff51c16aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:12.646 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.627+0000 7ff521a46700 1 -- 172.21.15.195:0/840039965 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff514009ae0 con 0x7ff51c0c4160 2022-06-27T12:05:12.647 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.627+0000 7ff521245700 1 --2- 172.21.15.195:0/840039965 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff51c06f640 0x7ff51c16aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:05:12.647 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.628+0000 7ff521a46700 1 --2- 172.21.15.195:0/840039965 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff51c0c4160 0x7ff51c159b40 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7ff51c06dd70 tx=0x7ff514018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:05:12.648 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.628+0000 7ff5097fa700 1 -- 172.21.15.195:0/840039965 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff514007610 con 0x7ff51c0c4160 2022-06-27T12:05:12.648 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.628+0000 7ff5097fa700 1 -- 172.21.15.195:0/840039965 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff514007770 con 0x7ff51c0c4160 2022-06-27T12:05:12.648 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.628+0000 7ff5097fa700 1 -- 172.21.15.195:0/840039965 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff514007b10 con 0x7ff51c0c4160 2022-06-27T12:05:12.649 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.628+0000 7ff5234a9700 1 -- 172.21.15.195:0/840039965 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff51c14c550 con 0x7ff51c0c4160 2022-06-27T12:05:12.649 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.628+0000 7ff5234a9700 1 -- 172.21.15.195:0/840039965 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff51c14cb10 con 0x7ff51c0c4160 2022-06-27T12:05:12.649 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.629+0000 7ff5234a9700 1 -- 172.21.15.195:0/840039965 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff51c0722c0 con 0x7ff51c0c4160 2022-06-27T12:05:12.650 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.629+0000 7ff5097fa700 1 -- 172.21.15.195:0/840039965 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7ff514004040 con 0x7ff51c0c4160 2022-06-27T12:05:12.650 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.629+0000 7ff5097fa700 1 --2- 172.21.15.195:0/840039965 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff4fc063d00 0x7ff4fc0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:12.651 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.630+0000 7ff521245700 1 --2- 172.21.15.195:0/840039965 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff4fc063d00 0x7ff4fc0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:12.651 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.630+0000 7ff5097fa700 1 -- 172.21.15.195:0/840039965 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6976+0+0 (secure 0 0 0) 0x7ff514082350 con 0x7ff51c0c4160 2022-06-27T12:05:12.652 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.630+0000 7ff521245700 1 --2- 172.21.15.195:0/840039965 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff4fc063d00 0x7ff4fc0661b0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7ff50c009fd0 tx=0x7ff50c009060 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:05:12.652 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.634+0000 7ff5097fa700 1 -- 172.21.15.195:0/840039965 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7ff514050bb0 con 0x7ff51c0c4160 2022-06-27T12:05:12.855 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.854+0000 7ff5234a9700 1 -- 172.21.15.195:0/840039965 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff51c155de0 con 0x7ff51c0c4160 2022-06-27T12:05:12.855 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.855+0000 7ff5097fa700 1 -- 172.21.15.195:0/840039965 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+5367 (secure 0 0 0) 0x7ff5140541d0 con 0x7ff51c0c4160 2022-06-27T12:05:12.856 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:05:12.856 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:05:05.063331+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5489},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5489":{"gid":5489,"name":"d","rank":0,"incarnation":61,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6835/2342195382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2342195382},{"type":"v1","addr":"172.21.15.202:6835","nonce":2342195382}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:05:12.857 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.857+0000 7ff5234a9700 1 -- 172.21.15.195:0/840039965 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff4fc063d00 msgr2=0x7ff4fc0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:12.858 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.857+0000 7ff5234a9700 1 --2- 172.21.15.195:0/840039965 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff4fc063d00 0x7ff4fc0661b0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7ff50c009fd0 tx=0x7ff50c009060 comp rx=0 tx=0).stop 2022-06-27T12:05:12.858 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.857+0000 7ff5234a9700 1 -- 172.21.15.195:0/840039965 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff51c0c4160 msgr2=0x7ff51c159b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:12.858 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.857+0000 7ff5234a9700 1 --2- 172.21.15.195:0/840039965 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff51c0c4160 0x7ff51c159b40 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7ff51c06dd70 tx=0x7ff514018000 comp rx=0 tx=0).stop 2022-06-27T12:05:12.859 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.857+0000 7ff5234a9700 1 -- 172.21.15.195:0/840039965 shutdown_connections 2022-06-27T12:05:12.859 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.857+0000 7ff5234a9700 1 --2- 172.21.15.195:0/840039965 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff4fc063d00 0x7ff4fc0661b0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:12.859 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.857+0000 7ff5234a9700 1 --2- 172.21.15.195:0/840039965 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff51c0c4160 0x7ff51c159b40 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:12.859 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.857+0000 7ff5234a9700 1 --2- 172.21.15.195:0/840039965 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff51c06f640 0x7ff51c16aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:12.859 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.857+0000 7ff5234a9700 1 --2- 172.21.15.195:0/840039965 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff51c0c39e0 0x7ff51c1514b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:12.860 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.857+0000 7ff5234a9700 1 -- 172.21.15.195:0/840039965 >> 172.21.15.195:0/840039965 conn(0x7ff51c0c6350 msgr2=0x7ff51c153370 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:05:12.860 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.857+0000 7ff5234a9700 1 -- 172.21.15.195:0/840039965 shutdown_connections 2022-06-27T12:05:12.860 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:12.857+0000 7ff5234a9700 1 -- 172.21.15.195:0/840039965 wait complete. 2022-06-27T12:05:12.861 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 61 2022-06-27T12:05:12.870 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:05:14.872 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:05:15.020 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.006+0000 7f64a8805700 1 Processor -- start 2022-06-27T12:05:15.021 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.006+0000 7f64a8805700 1 -- start start 2022-06-27T12:05:15.022 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.007+0000 7f64a8805700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f64a406e850 0x7f64a41570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:15.023 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.007+0000 7f64a8805700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f64a40c34a0 0x7f64a40c2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:15.023 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.007+0000 7f64a8805700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64a406fba0 con 0x7f64a406fef0 2022-06-27T12:05:15.024 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.007+0000 7f64a8805700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64a41575e0 con 0x7f64a40c34a0 2022-06-27T12:05:15.024 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.007+0000 7f64a8805700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64a406f5e0 con 0x7f64a406e850 2022-06-27T12:05:15.024 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.007+0000 7f64a2d9d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f64a40c34a0 0x7f64a40c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:15.025 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.007+0000 7f64a2d9d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f64a40c34a0 0x7f64a40c2920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47886/0 (socket says 172.21.15.195:47886) 2022-06-27T12:05:15.025 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.007+0000 7f64a2d9d700 1 -- 172.21.15.195:0/1773867842 learned_addr learned my addr 172.21.15.195:0/1773867842 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:05:15.026 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.007+0000 7f64a259c700 1 --2- 172.21.15.195:0/1773867842 >> v2:172.21.15.202:3300/0 conn(0x7f64a406e850 0x7f64a41570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:15.026 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.007+0000 7f64a2d9d700 1 -- 172.21.15.195:0/1773867842 >> v2:172.21.15.202:3300/0 conn(0x7f64a406e850 msgr2=0x7f64a41570a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:05:15.027 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.007+0000 7f64a2d9d700 1 --2- 172.21.15.195:0/1773867842 >> v2:172.21.15.202:3300/0 conn(0x7f64a406e850 0x7f64a41570a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:15.027 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.007+0000 7f64a2d9d700 1 -- 172.21.15.195:0/1773867842 >> v1:172.21.15.202:6789/0 conn(0x7f64a406fef0 legacy=0x7f64a4069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:15.028 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.007+0000 7f64a2d9d700 1 -- 172.21.15.195:0/1773867842 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64a406f8c0 con 0x7f64a40c34a0 2022-06-27T12:05:15.028 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.008+0000 7f64a2d9d700 1 --2- 172.21.15.195:0/1773867842 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f64a40c34a0 0x7f64a40c2920 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f64a406e120 tx=0x7f649c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f3dd2cacabd8370 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:05:15.028 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.008+0000 7f64a159a700 1 -- 172.21.15.195:0/1773867842 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f649c00e070 con 0x7f64a40c34a0 2022-06-27T12:05:15.030 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.008+0000 7f64a159a700 1 -- 172.21.15.195:0/1773867842 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f649c00bab0 con 0x7f64a40c34a0 2022-06-27T12:05:15.030 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.008+0000 7f64a159a700 1 -- 172.21.15.195:0/1773867842 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f649c0092e0 con 0x7f64a40c34a0 2022-06-27T12:05:15.031 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.008+0000 7f64a8805700 1 -- 172.21.15.195:0/1773867842 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f64a40c34a0 msgr2=0x7f64a40c2920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:15.031 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.008+0000 7f64a8805700 1 --2- 172.21.15.195:0/1773867842 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f64a40c34a0 0x7f64a40c2920 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f64a406e120 tx=0x7f649c009840 comp rx=0 tx=0).stop 2022-06-27T12:05:15.032 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.008+0000 7f64a8805700 1 -- 172.21.15.195:0/1773867842 shutdown_connections 2022-06-27T12:05:15.032 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.008+0000 7f64a8805700 1 --2- 172.21.15.195:0/1773867842 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f64a40c34a0 0x7f64a40c2920 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:15.033 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.008+0000 7f64a8805700 1 --2- 172.21.15.195:0/1773867842 >> v2:172.21.15.202:3300/0 conn(0x7f64a406e850 0x7f64a41570a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:15.033 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.008+0000 7f64a8805700 1 -- 172.21.15.195:0/1773867842 >> 172.21.15.195:0/1773867842 conn(0x7f64a40c6350 msgr2=0x7f64a40c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:05:15.036 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.009+0000 7f64a8805700 1 -- 172.21.15.195:0/1773867842 shutdown_connections 2022-06-27T12:05:15.036 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.009+0000 7f64a8805700 1 -- 172.21.15.195:0/1773867842 wait complete. 2022-06-27T12:05:15.036 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.010+0000 7f64a8805700 1 Processor -- start 2022-06-27T12:05:15.037 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.010+0000 7f64a8805700 1 -- start start 2022-06-27T12:05:15.037 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.010+0000 7f64a8805700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f64a406e850 0x7f64a40c28d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:15.038 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.010+0000 7f64a8805700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f64a40c34a0 0x7f64a4147790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:15.038 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.010+0000 7f64a8805700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f64a414e720 0x7f64a4147cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:15.041 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.010+0000 7f64a8805700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a4148520 con 0x7f64a40c34a0 2022-06-27T12:05:15.042 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.010+0000 7f64a8805700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a41487c0 con 0x7f64a406e850 2022-06-27T12:05:15.042 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.010+0000 7f64a8805700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64a4148aa0 con 0x7f64a414e720 2022-06-27T12:05:15.042 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.011+0000 7f64a1d9b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f64a40c34a0 0x7f64a4147790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:15.042 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.011+0000 7f64a1d9b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f64a40c34a0 0x7f64a4147790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47888/0 (socket says 172.21.15.195:47888) 2022-06-27T12:05:15.043 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.011+0000 7f64a1d9b700 1 -- 172.21.15.195:0/3986130530 learned_addr learned my addr 172.21.15.195:0/3986130530 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:05:15.043 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.011+0000 7f64a259c700 1 --2- 172.21.15.195:0/3986130530 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f64a406e850 0x7f64a40c28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:15.043 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.011+0000 7f64a2d9d700 1 --2- 172.21.15.195:0/3986130530 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f64a414e720 0x7f64a4147cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:15.044 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.011+0000 7f64a1d9b700 1 -- 172.21.15.195:0/3986130530 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f64a414e720 msgr2=0x7f64a4147cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:15.044 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.011+0000 7f64a1d9b700 1 --2- 172.21.15.195:0/3986130530 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f64a414e720 0x7f64a4147cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:15.044 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.011+0000 7f64a1d9b700 1 -- 172.21.15.195:0/3986130530 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f64a406e850 msgr2=0x7f64a40c28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:15.044 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.011+0000 7f64a1d9b700 1 --2- 172.21.15.195:0/3986130530 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f64a406e850 0x7f64a40c28d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:15.045 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.011+0000 7f64a1d9b700 1 -- 172.21.15.195:0/3986130530 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f649c009ae0 con 0x7f64a40c34a0 2022-06-27T12:05:15.045 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.011+0000 7f64a1d9b700 1 --2- 172.21.15.195:0/3986130530 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f64a40c34a0 0x7f64a4147790 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f649400c440 tx=0x7f6494007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:05:15.045 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.011+0000 7f64927fc700 1 -- 172.21.15.195:0/3986130530 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64940050d0 con 0x7f64a40c34a0 2022-06-27T12:05:15.045 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.011+0000 7f64927fc700 1 -- 172.21.15.195:0/3986130530 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6494005230 con 0x7f64a40c34a0 2022-06-27T12:05:15.046 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.011+0000 7f64927fc700 1 -- 172.21.15.195:0/3986130530 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6494003750 con 0x7f64a40c34a0 2022-06-27T12:05:15.046 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.011+0000 7f64a8805700 1 -- 172.21.15.195:0/3986130530 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64a4148d80 con 0x7f64a40c34a0 2022-06-27T12:05:15.046 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.011+0000 7f64a8805700 1 -- 172.21.15.195:0/3986130530 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64a4155840 con 0x7f64a40c34a0 2022-06-27T12:05:15.046 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.012+0000 7f64a8805700 1 -- 172.21.15.195:0/3986130530 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64a40722c0 con 0x7f64a40c34a0 2022-06-27T12:05:15.047 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.012+0000 7f64927fc700 1 -- 172.21.15.195:0/3986130530 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f6494019030 con 0x7f64a40c34a0 2022-06-27T12:05:15.047 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.013+0000 7f64927fc700 1 --2- 172.21.15.195:0/3986130530 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6480063d30 0x7f64800661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:15.047 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.013+0000 7f64a259c700 1 --2- 172.21.15.195:0/3986130530 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6480063d30 0x7f64800661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:15.047 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.013+0000 7f64927fc700 1 -- 172.21.15.195:0/3986130530 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6976+0+0 (secure 0 0 0) 0x7f649407fbe0 con 0x7f64a40c34a0 2022-06-27T12:05:15.048 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.013+0000 7f64a259c700 1 --2- 172.21.15.195:0/3986130530 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6480063d30 0x7f64800661e0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f6498009bd0 tx=0x7f6498009060 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:05:15.048 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.017+0000 7f64927fc700 1 -- 172.21.15.195:0/3986130530 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f649404e440 con 0x7f64a40c34a0 2022-06-27T12:05:15.243 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.240+0000 7f64a8805700 1 -- 172.21.15.195:0/3986130530 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f64a406f240 con 0x7f64a40c34a0 2022-06-27T12:05:15.244 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.240+0000 7f64927fc700 1 -- 172.21.15.195:0/3986130530 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+5367 (secure 0 0 0) 0x7f6494017020 con 0x7f64a40c34a0 2022-06-27T12:05:15.244 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:05:15.245 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:05:05.063331+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5489},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5489":{"gid":5489,"name":"d","rank":0,"incarnation":61,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6835/2342195382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2342195382},{"type":"v1","addr":"172.21.15.202:6835","nonce":2342195382}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:05:15.246 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.242+0000 7f64a8805700 1 -- 172.21.15.195:0/3986130530 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6480063d30 msgr2=0x7f64800661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:15.246 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.242+0000 7f64a8805700 1 --2- 172.21.15.195:0/3986130530 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6480063d30 0x7f64800661e0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f6498009bd0 tx=0x7f6498009060 comp rx=0 tx=0).stop 2022-06-27T12:05:15.247 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.242+0000 7f64a8805700 1 -- 172.21.15.195:0/3986130530 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f64a40c34a0 msgr2=0x7f64a4147790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:15.247 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.242+0000 7f64a8805700 1 --2- 172.21.15.195:0/3986130530 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f64a40c34a0 0x7f64a4147790 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f649400c440 tx=0x7f6494007e00 comp rx=0 tx=0).stop 2022-06-27T12:05:15.247 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.242+0000 7f64a8805700 1 -- 172.21.15.195:0/3986130530 shutdown_connections 2022-06-27T12:05:15.247 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.242+0000 7f64a8805700 1 --2- 172.21.15.195:0/3986130530 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6480063d30 0x7f64800661e0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:15.248 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.242+0000 7f64a8805700 1 --2- 172.21.15.195:0/3986130530 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f64a414e720 0x7f64a4147cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:15.248 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.242+0000 7f64a8805700 1 --2- 172.21.15.195:0/3986130530 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f64a406e850 0x7f64a40c28d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:15.248 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.242+0000 7f64a8805700 1 --2- 172.21.15.195:0/3986130530 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f64a40c34a0 0x7f64a4147790 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:15.248 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.242+0000 7f64a8805700 1 -- 172.21.15.195:0/3986130530 >> 172.21.15.195:0/3986130530 conn(0x7f64a40c6350 msgr2=0x7f64a4150430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:05:15.249 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.242+0000 7f64a8805700 1 -- 172.21.15.195:0/3986130530 shutdown_connections 2022-06-27T12:05:15.249 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:15.242+0000 7f64a8805700 1 -- 172.21.15.195:0/3986130530 wait complete. 2022-06-27T12:05:15.249 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 61 2022-06-27T12:05:15.255 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:05:17.256 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:05:17.405 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.391+0000 7f0228ad7700 1 Processor -- start 2022-06-27T12:05:17.406 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.391+0000 7f0228ad7700 1 -- start start 2022-06-27T12:05:17.406 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.392+0000 7f0228ad7700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f022406eff0 0x7f02240c61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:17.406 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.392+0000 7f0228ad7700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f02240c31d0 0x7f02240c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:17.407 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.392+0000 7f0228ad7700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f022414d8a0 con 0x7f0224067ed0 2022-06-27T12:05:17.407 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.392+0000 7f0228ad7700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02240c47e0 con 0x7f022406eff0 2022-06-27T12:05:17.407 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.392+0000 7f0228ad7700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02240c3e10 con 0x7f02240c31d0 2022-06-27T12:05:17.408 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.393+0000 7f022259c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f022406eff0 0x7f02240c61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:17.408 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.393+0000 7f022259c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f022406eff0 0x7f02240c61e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47896/0 (socket says 172.21.15.195:47896) 2022-06-27T12:05:17.408 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.393+0000 7f0221d9b700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f0224067ed0 0x7f02240c2ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:54086/0 (socket says 172.21.15.195:54086) 2022-06-27T12:05:17.409 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.393+0000 7f022259c700 1 -- 172.21.15.195:0/3978646598 learned_addr learned my addr 172.21.15.195:0/3978646598 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:05:17.409 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.393+0000 7f0222d9d700 1 --2- 172.21.15.195:0/3978646598 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f02240c31d0 0x7f02240c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:17.409 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.393+0000 7f022259c700 1 -- 172.21.15.195:0/3978646598 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f02240c31d0 msgr2=0x7f02240c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:05:17.409 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.393+0000 7f022259c700 1 --2- 172.21.15.195:0/3978646598 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f02240c31d0 0x7f02240c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:17.410 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.393+0000 7f022259c700 1 -- 172.21.15.195:0/3978646598 >> v1:172.21.15.195:6789/0 conn(0x7f0224067ed0 legacy=0x7f02240c2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:17.410 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.393+0000 7f022159a700 1 -- 172.21.15.195:0/3978646598 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3969592597 0 0) 0x7f022414d8a0 con 0x7f0224067ed0 2022-06-27T12:05:17.410 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.393+0000 7f022259c700 1 -- 172.21.15.195:0/3978646598 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f022406d970 con 0x7f022406eff0 2022-06-27T12:05:17.410 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.393+0000 7f022259c700 1 --2- 172.21.15.195:0/3978646598 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f022406eff0 0x7f02240c61e0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f0218006fd0 tx=0x7f02180090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ffa4e2e745d1974 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:05:17.411 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.393+0000 7f022159a700 1 -- 172.21.15.195:0/3978646598 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0218005ad0 con 0x7f022406eff0 2022-06-27T12:05:17.411 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.393+0000 7f022159a700 1 -- 172.21.15.195:0/3978646598 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0218005c30 con 0x7f022406eff0 2022-06-27T12:05:17.411 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.393+0000 7f022159a700 1 -- 172.21.15.195:0/3978646598 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0218004030 con 0x7f022406eff0 2022-06-27T12:05:17.411 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.394+0000 7f0228ad7700 1 -- 172.21.15.195:0/3978646598 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f022406eff0 msgr2=0x7f02240c61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:17.412 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.394+0000 7f0228ad7700 1 --2- 172.21.15.195:0/3978646598 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f022406eff0 0x7f02240c61e0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f0218006fd0 tx=0x7f02180090f0 comp rx=0 tx=0).stop 2022-06-27T12:05:17.412 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.394+0000 7f0228ad7700 1 -- 172.21.15.195:0/3978646598 shutdown_connections 2022-06-27T12:05:17.412 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.394+0000 7f0228ad7700 1 --2- 172.21.15.195:0/3978646598 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f02240c31d0 0x7f02240c42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:17.412 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.394+0000 7f0228ad7700 1 --2- 172.21.15.195:0/3978646598 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f022406eff0 0x7f02240c61e0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:17.413 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.394+0000 7f0228ad7700 1 -- 172.21.15.195:0/3978646598 >> 172.21.15.195:0/3978646598 conn(0x7f02240c9b00 msgr2=0x7f02240c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:05:17.413 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.394+0000 7f0228ad7700 1 -- 172.21.15.195:0/3978646598 shutdown_connections 2022-06-27T12:05:17.413 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.394+0000 7f0228ad7700 1 -- 172.21.15.195:0/3978646598 wait complete. 2022-06-27T12:05:17.413 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.395+0000 7f0228ad7700 1 Processor -- start 2022-06-27T12:05:17.414 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.395+0000 7f0228ad7700 1 -- start start 2022-06-27T12:05:17.414 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.396+0000 7f0228ad7700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0224067ed0 0x7f02241627b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:17.414 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.396+0000 7f0228ad7700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f022406eff0 0x7f02240bdf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:17.414 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.396+0000 7f0228ad7700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0224151af0 0x7f022414a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:17.415 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.396+0000 7f0228ad7700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02240bd3b0 con 0x7f0224151af0 2022-06-27T12:05:17.415 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.396+0000 7f0228ad7700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0224151660 con 0x7f0224067ed0 2022-06-27T12:05:17.415 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.396+0000 7f0228ad7700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0224152850 con 0x7f022406eff0 2022-06-27T12:05:17.415 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.396+0000 7f0222d9d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0224151af0 0x7f022414a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:17.416 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.396+0000 7f0222d9d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0224151af0 0x7f022414a3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47906/0 (socket says 172.21.15.195:47906) 2022-06-27T12:05:17.416 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.396+0000 7f0222d9d700 1 -- 172.21.15.195:0/2776107064 learned_addr learned my addr 172.21.15.195:0/2776107064 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:05:17.416 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.396+0000 7f0221d9b700 1 --2- 172.21.15.195:0/2776107064 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f022406eff0 0x7f02240bdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:17.416 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.396+0000 7f022259c700 1 --2- 172.21.15.195:0/2776107064 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0224067ed0 0x7f02241627b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:17.417 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.396+0000 7f0222d9d700 1 -- 172.21.15.195:0/2776107064 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f022406eff0 msgr2=0x7f02240bdf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:17.417 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.396+0000 7f0222d9d700 1 --2- 172.21.15.195:0/2776107064 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f022406eff0 0x7f02240bdf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:17.417 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.396+0000 7f0222d9d700 1 -- 172.21.15.195:0/2776107064 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0224067ed0 msgr2=0x7f02241627b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:17.417 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.396+0000 7f0222d9d700 1 --2- 172.21.15.195:0/2776107064 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0224067ed0 0x7f02241627b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:17.417 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.396+0000 7f0222d9d700 1 -- 172.21.15.195:0/2776107064 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0218009350 con 0x7f0224151af0 2022-06-27T12:05:17.418 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.396+0000 7f0222d9d700 1 --2- 172.21.15.195:0/2776107064 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0224151af0 0x7f022414a3f0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f021c009d70 tx=0x7f021c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:05:17.418 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.396+0000 7f02127fc700 1 -- 172.21.15.195:0/2776107064 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f021c00cb30 con 0x7f0224151af0 2022-06-27T12:05:17.418 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.396+0000 7f02127fc700 1 -- 172.21.15.195:0/2776107064 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f021c00f040 con 0x7f0224151af0 2022-06-27T12:05:17.418 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.396+0000 7f0228ad7700 1 -- 172.21.15.195:0/2776107064 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0224150da0 con 0x7f0224151af0 2022-06-27T12:05:17.419 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.396+0000 7f02127fc700 1 -- 172.21.15.195:0/2776107064 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f021c00cee0 con 0x7f0224151af0 2022-06-27T12:05:17.419 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.397+0000 7f0228ad7700 1 -- 172.21.15.195:0/2776107064 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0224151010 con 0x7f0224151af0 2022-06-27T12:05:17.419 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.397+0000 7f0228ad7700 1 -- 172.21.15.195:0/2776107064 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0224151810 con 0x7f0224151af0 2022-06-27T12:05:17.419 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.398+0000 7f02127fc700 1 -- 172.21.15.195:0/2776107064 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f021c014690 con 0x7f0224151af0 2022-06-27T12:05:17.420 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.398+0000 7f02127fc700 1 --2- 172.21.15.195:0/2776107064 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0200063d00 0x7f02000661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:17.420 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.398+0000 7f02127fc700 1 -- 172.21.15.195:0/2776107064 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6976+0+0 (secure 0 0 0) 0x7f021c081a30 con 0x7f0224151af0 2022-06-27T12:05:17.420 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.398+0000 7f022259c700 1 --2- 172.21.15.195:0/2776107064 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0200063d00 0x7f02000661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:17.421 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.399+0000 7f022259c700 1 --2- 172.21.15.195:0/2776107064 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0200063d00 0x7f02000661b0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f0218009130 tx=0x7f0218015000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:05:17.421 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.402+0000 7f02127fc700 1 -- 172.21.15.195:0/2776107064 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f021c083e60 con 0x7f0224151af0 2022-06-27T12:05:17.627 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.622+0000 7f0228ad7700 1 -- 172.21.15.195:0/2776107064 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f02240722c0 con 0x7f0224151af0 2022-06-27T12:05:17.628 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.622+0000 7f02127fc700 1 -- 172.21.15.195:0/2776107064 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+5367 (secure 0 0 0) 0x7f021c018090 con 0x7f0224151af0 2022-06-27T12:05:17.628 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:05:17.629 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:05:05.063331+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5489},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5489":{"gid":5489,"name":"d","rank":0,"incarnation":61,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6835/2342195382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2342195382},{"type":"v1","addr":"172.21.15.202:6835","nonce":2342195382}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:05:17.630 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.624+0000 7f0228ad7700 1 -- 172.21.15.195:0/2776107064 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0200063d00 msgr2=0x7f02000661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:17.630 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.624+0000 7f0228ad7700 1 --2- 172.21.15.195:0/2776107064 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0200063d00 0x7f02000661b0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f0218009130 tx=0x7f0218015000 comp rx=0 tx=0).stop 2022-06-27T12:05:17.631 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.624+0000 7f0228ad7700 1 -- 172.21.15.195:0/2776107064 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0224151af0 msgr2=0x7f022414a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:17.631 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.624+0000 7f0228ad7700 1 --2- 172.21.15.195:0/2776107064 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0224151af0 0x7f022414a3f0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f021c009d70 tx=0x7f021c00b040 comp rx=0 tx=0).stop 2022-06-27T12:05:17.631 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.625+0000 7f0228ad7700 1 -- 172.21.15.195:0/2776107064 shutdown_connections 2022-06-27T12:05:17.632 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.625+0000 7f0228ad7700 1 --2- 172.21.15.195:0/2776107064 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0200063d00 0x7f02000661b0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:17.632 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.625+0000 7f0228ad7700 1 --2- 172.21.15.195:0/2776107064 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0224151af0 0x7f022414a3f0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:17.632 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.625+0000 7f0228ad7700 1 --2- 172.21.15.195:0/2776107064 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0224067ed0 0x7f02241627b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:17.632 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.625+0000 7f0228ad7700 1 --2- 172.21.15.195:0/2776107064 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f022406eff0 0x7f02240bdf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:17.633 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.625+0000 7f0228ad7700 1 -- 172.21.15.195:0/2776107064 >> 172.21.15.195:0/2776107064 conn(0x7f02240c9b00 msgr2=0x7f0224063c40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:05:17.633 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.625+0000 7f0228ad7700 1 -- 172.21.15.195:0/2776107064 shutdown_connections 2022-06-27T12:05:17.633 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:17.625+0000 7f0228ad7700 1 -- 172.21.15.195:0/2776107064 wait complete. 2022-06-27T12:05:17.633 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 61 2022-06-27T12:05:17.641 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 61, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-06-27T11:47:52.358095+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 61, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5489": { "addr": "172.21.15.202:6835/2342195382", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6834", "nonce": 2342195382, "type": "v2" }, { "addr": "172.21.15.202:6835", "nonce": 2342195382, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5489, "incarnation": 61, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 57, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-06-27T12:05:05.063331+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5489 } } } ], "standbys": [ { "addr": "172.21.15.202:6837/3305006841", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6836", "nonce": 3305006841, "type": "v2" }, { "addr": "172.21.15.202:6837", "nonce": 3305006841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6835/2060709822", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6834", "nonce": 2060709822, "type": "v2" }, { "addr": "172.21.15.195:6835", "nonce": 2060709822, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6838/1598751652", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6836", "nonce": 1598751652, "type": "v2" }, { "addr": "172.21.15.195:6838", "nonce": 1598751652, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4275, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6839/1313800661", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6837", "nonce": 1313800661, "type": "v2" }, { "addr": "172.21.15.195:6839", "nonce": 1313800661, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4278, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-06-27T12:05:19.642 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:05:19.805 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:19.802+0000 7f1911d0b700 1 Processor -- start 2022-06-27T12:05:19.810 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:19.802+0000 7f1911d0b700 1 -- start start 2022-06-27T12:05:19.810 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:19.803+0000 7f1911d0b700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f190c14e490 0x7f190c06cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:19.815 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:19.803+0000 7f1911d0b700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f190c0c33b0 con 0x7f190c14e860 2022-06-27T12:05:19.815 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:19.803+0000 7f1911d0b700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f190c06d520 con 0x7f190c0c3f50 2022-06-27T12:05:19.815 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:19.803+0000 7f1911d0b700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f190c06f220 con 0x7f190c14e490 2022-06-27T12:05:19.816 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:19.803+0000 7f190affd700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f190c14e860 0x7f190c0c3840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:54106/0 (socket says 172.21.15.195:54106) 2022-06-27T12:05:19.816 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:19.803+0000 7f190affd700 1 -- 172.21.15.195:0/1751826852 learned_addr learned my addr 172.21.15.195:0/1751826852 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:05:19.816 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:19.803+0000 7f190b7fe700 1 --2- 172.21.15.195:0/1751826852 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f190c14e490 0x7f190c06cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:19.817 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:19.803+0000 7f190a7fc700 1 -- 172.21.15.195:0/1751826852 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3890289263 0 0) 0x7f190c0c33b0 con 0x7f190c14e860 2022-06-27T12:05:19.825 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:19.803+0000 7f190a7fc700 1 -- 172.21.15.195:0/1751826852 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18ec003430 con 0x7f190c14e860 2022-06-27T12:05:19.825 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:19.804+0000 7f190a7fc700 1 -- 172.21.15.195:0/1751826852 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1164121432 0 0) 0x7f190c06d520 con 0x7f190c0c3f50 2022-06-27T12:05:19.825 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:19.804+0000 7f190a7fc700 1 -- 172.21.15.195:0/1751826852 --> v1:172.21.15.202:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f190c0c33b0 con 0x7f190c0c3f50 2022-06-27T12:05:19.826 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:19.804+0000 7f190b7fe700 1 -- 172.21.15.195:0/1751826852 >> v1:172.21.15.202:6789/0 conn(0x7f190c0c3f50 legacy=0x7f190c157890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:19.826 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:19.804+0000 7f190b7fe700 1 -- 172.21.15.195:0/1751826852 >> v1:172.21.15.195:6789/0 conn(0x7f190c14e860 legacy=0x7f190c0c3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:19.826 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:19.804+0000 7f190b7fe700 1 -- 172.21.15.195:0/1751826852 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f190c1542c0 con 0x7f190c14e490 2022-06-27T12:05:19.827 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:19.804+0000 7f190b7fe700 1 --2- 172.21.15.195:0/1751826852 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f190c14e490 0x7f190c06cfe0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f18fc0097d0 tx=0x7f18fc009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dae958acafdaade server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:05:19.827 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:19.804+0000 7f190a7fc700 1 -- 172.21.15.195:0/1751826852 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18fc011630 con 0x7f190c14e490 2022-06-27T12:05:19.827 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:19.804+0000 7f190a7fc700 1 -- 172.21.15.195:0/1751826852 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18fc011790 con 0x7f190c14e490 2022-06-27T12:05:19.827 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:19.804+0000 7f190a7fc700 1 -- 172.21.15.195:0/1751826852 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18fc011b10 con 0x7f190c14e490 2022-06-27T12:05:19.828 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:19.804+0000 7f1911d0b700 1 -- 172.21.15.195:0/1751826852 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f190c14e490 msgr2=0x7f190c06cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:19.828 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:19.804+0000 7f1911d0b700 1 --2- 172.21.15.195:0/1751826852 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f190c14e490 0x7f190c06cfe0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f18fc0097d0 tx=0x7f18fc009060 comp rx=0 tx=0).stop 2022-06-27T12:05:19.828 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:19.805+0000 7f1911d0b700 1 -- 172.21.15.195:0/1751826852 shutdown_connections 2022-06-27T12:05:19.830 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:19.805+0000 7f1911d0b700 1 --2- 172.21.15.195:0/1751826852 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f190c14e490 0x7f190c06cfe0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:19.830 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:19.805+0000 7f1911d0b700 1 -- 172.21.15.195:0/1751826852 >> 172.21.15.195:0/1751826852 conn(0x7f190c063af0 msgr2=0x7f190c063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:05:19.831 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:19.805+0000 7f1911d0b700 1 -- 172.21.15.195:0/1751826852 shutdown_connections 2022-06-27T12:05:19.831 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:19.805+0000 7f1911d0b700 1 -- 172.21.15.195:0/1751826852 wait complete. 2022-06-27T12:05:19.831 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:19.806+0000 7f1911d0b700 1 Processor -- start 2022-06-27T12:05:19.831 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:19.806+0000 7f1911d0b700 1 -- start start 2022-06-27T12:05:19.832 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:19.806+0000 7f1911d0b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f190c0c3f50 0x7f190c16a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:19.832 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:19.806+0000 7f1911d0b700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f190c14e490 0x7f190c153390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:19.832 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:19.807+0000 7f1911d0b700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f190c14e860 0x7f190c1677a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:19.832 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:19.807+0000 7f1911d0b700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f190c06b080 con 0x7f190c0c3f50 2022-06-27T12:05:19.832 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:19.807+0000 7f1911d0b700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f190c069ab0 con 0x7f190c14e860 2022-06-27T12:05:19.833 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:19.807+0000 7f1911d0b700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f190c069f40 con 0x7f190c14e490 2022-06-27T12:05:19.833 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:19.807+0000 7f190b7fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f190c0c3f50 0x7f190c16a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:19.834 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:19.807+0000 7f190b7fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f190c0c3f50 0x7f190c16a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47922/0 (socket says 172.21.15.195:47922) 2022-06-27T12:05:19.834 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:19.807+0000 7f190b7fe700 1 -- 172.21.15.195:0/2913912570 learned_addr learned my addr 172.21.15.195:0/2913912570 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:05:19.834 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:19.807+0000 7f190affd700 1 --2- 172.21.15.195:0/2913912570 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f190c14e490 0x7f190c153390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:19.834 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:19.807+0000 7f190b7fe700 1 -- 172.21.15.195:0/2913912570 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f190c14e490 msgr2=0x7f190c153390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:19.835 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:19.807+0000 7f190bfff700 1 --2- 172.21.15.195:0/2913912570 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f190c14e860 0x7f190c1677a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:19.835 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:19.807+0000 7f190b7fe700 1 --2- 172.21.15.195:0/2913912570 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f190c14e490 0x7f190c153390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:19.840 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:19.807+0000 7f190b7fe700 1 -- 172.21.15.195:0/2913912570 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f190c14e860 msgr2=0x7f190c1677a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:19.841 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:19.807+0000 7f190b7fe700 1 --2- 172.21.15.195:0/2913912570 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f190c14e860 0x7f190c1677a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:19.841 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:19.807+0000 7f190b7fe700 1 -- 172.21.15.195:0/2913912570 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18fc0092e0 con 0x7f190c0c3f50 2022-06-27T12:05:19.841 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:19.807+0000 7f190b7fe700 1 --2- 172.21.15.195:0/2913912570 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f190c0c3f50 0x7f190c16a290 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f190c070770 tx=0x7f18fc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:05:19.841 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:19.807+0000 7f19037fe700 1 -- 172.21.15.195:0/2913912570 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18fc003900 con 0x7f190c0c3f50 2022-06-27T12:05:19.843 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:19.807+0000 7f19037fe700 1 -- 172.21.15.195:0/2913912570 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18fc003a60 con 0x7f190c0c3f50 2022-06-27T12:05:19.843 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:19.807+0000 7f1911d0b700 1 -- 172.21.15.195:0/2913912570 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f190c06ada0 con 0x7f190c0c3f50 2022-06-27T12:05:19.843 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:19.807+0000 7f19037fe700 1 -- 172.21.15.195:0/2913912570 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18fc003e20 con 0x7f190c0c3f50 2022-06-27T12:05:19.844 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:19.807+0000 7f1911d0b700 1 -- 172.21.15.195:0/2913912570 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f190c069460 con 0x7f190c0c3f50 2022-06-27T12:05:19.844 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:19.808+0000 7f1911d0b700 1 -- 172.21.15.195:0/2913912570 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f190c16a7d0 con 0x7f190c0c3f50 2022-06-27T12:05:19.844 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:19.809+0000 7f19037fe700 1 -- 172.21.15.195:0/2913912570 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f18fc029020 con 0x7f190c0c3f50 2022-06-27T12:05:19.845 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:19.809+0000 7f19037fe700 1 --2- 172.21.15.195:0/2913912570 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f18f0063d00 0x7f18f00661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:19.845 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:19.809+0000 7f19037fe700 1 -- 172.21.15.195:0/2913912570 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6976+0+0 (secure 0 0 0) 0x7f18fc081e20 con 0x7f190c0c3f50 2022-06-27T12:05:19.845 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:19.809+0000 7f190affd700 1 --2- 172.21.15.195:0/2913912570 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f18f0063d00 0x7f18f00661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:19.846 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:19.809+0000 7f190affd700 1 --2- 172.21.15.195:0/2913912570 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f18f0063d00 0x7f18f00661b0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f18f4005790 tx=0x7f18f4009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:05:19.846 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:19.813+0000 7f19037fe700 1 -- 172.21.15.195:0/2913912570 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f18fc050680 con 0x7f190c0c3f50 2022-06-27T12:05:20.045 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:20.044+0000 7f1911d0b700 1 -- 172.21.15.195:0/2913912570 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f190c069d90 con 0x7f190c0c3f50 2022-06-27T12:05:20.045 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:20.044+0000 7f19037fe700 1 -- 172.21.15.195:0/2913912570 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+5371 (secure 0 0 0) 0x7f18fc015070 con 0x7f190c0c3f50 2022-06-27T12:05:20.047 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:05:20.047 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:05:19.193001+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5489},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5489":{"gid":5489,"name":"d","rank":0,"incarnation":61,"state":"up:reconnect","state_seq":17,"addr":"172.21.15.202:6835/2342195382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2342195382},{"type":"v1","addr":"172.21.15.202:6835","nonce":2342195382}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:05:20.054 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:20.048+0000 7f1911d0b700 1 -- 172.21.15.195:0/2913912570 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f18f0063d00 msgr2=0x7f18f00661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:20.054 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:20.048+0000 7f1911d0b700 1 --2- 172.21.15.195:0/2913912570 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f18f0063d00 0x7f18f00661b0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f18f4005790 tx=0x7f18f4009040 comp rx=0 tx=0).stop 2022-06-27T12:05:20.054 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:20.048+0000 7f1911d0b700 1 -- 172.21.15.195:0/2913912570 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f190c0c3f50 msgr2=0x7f190c16a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:20.054 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:20.048+0000 7f1911d0b700 1 --2- 172.21.15.195:0/2913912570 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f190c0c3f50 0x7f190c16a290 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f190c070770 tx=0x7f18fc017000 comp rx=0 tx=0).stop 2022-06-27T12:05:20.055 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:20.049+0000 7f1911d0b700 1 -- 172.21.15.195:0/2913912570 shutdown_connections 2022-06-27T12:05:20.055 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:20.049+0000 7f1911d0b700 1 --2- 172.21.15.195:0/2913912570 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f18f0063d00 0x7f18f00661b0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:20.055 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:20.049+0000 7f1911d0b700 1 --2- 172.21.15.195:0/2913912570 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f190c0c3f50 0x7f190c16a290 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:20.056 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:20.049+0000 7f1911d0b700 1 --2- 172.21.15.195:0/2913912570 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f190c14e860 0x7f190c1677a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:20.056 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:20.049+0000 7f1911d0b700 1 --2- 172.21.15.195:0/2913912570 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f190c14e490 0x7f190c153390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:20.062 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:20.049+0000 7f1911d0b700 1 -- 172.21.15.195:0/2913912570 >> 172.21.15.195:0/2913912570 conn(0x7f190c063af0 msgr2=0x7f190c063f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:05:20.063 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:20.049+0000 7f1911d0b700 1 -- 172.21.15.195:0/2913912570 shutdown_connections 2022-06-27T12:05:20.063 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:20.049+0000 7f1911d0b700 1 -- 172.21.15.195:0/2913912570 wait complete. 2022-06-27T12:05:20.063 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 62 2022-06-27T12:05:20.065 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:05:22.068 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:05:22.202 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.202+0000 7f3f20753700 1 Processor -- start 2022-06-27T12:05:22.203 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.202+0000 7f3f20753700 1 -- start start 2022-06-27T12:05:22.214 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.203+0000 7f3f20753700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3f18156100 0x7f3f180688a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:22.214 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.203+0000 7f3f20753700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f3f181564d0 0x7f3f18156950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:22.215 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.203+0000 7f3f20753700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f180c4b10 con 0x7f3f1806a440 2022-06-27T12:05:22.215 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.203+0000 7f3f20753700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f180c3dc0 con 0x7f3f181564d0 2022-06-27T12:05:22.216 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.203+0000 7f3f20753700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f18068f20 con 0x7f3f18156100 2022-06-27T12:05:22.216 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.203+0000 7f3f1e4ef700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3f18156100 0x7f3f180688a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:22.217 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.203+0000 7f3f1ecf0700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f3f181564d0 0x7f3f18156950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:22.217 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.203+0000 7f3f1e4ef700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3f18156100 0x7f3f180688a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3301/0 says I am v2:172.21.15.195:40694/0 (socket says 172.21.15.195:40694) 2022-06-27T12:05:22.218 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.204+0000 7f3f1e4ef700 1 -- 172.21.15.195:0/819901817 learned_addr learned my addr 172.21.15.195:0/819901817 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:05:22.219 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.204+0000 7f3f1e4ef700 1 -- 172.21.15.195:0/819901817 >> v2:172.21.15.202:3300/0 conn(0x7f3f181564d0 msgr2=0x7f3f18156950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:05:22.219 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.204+0000 7f3f1e4ef700 1 --2- 172.21.15.195:0/819901817 >> v2:172.21.15.202:3300/0 conn(0x7f3f181564d0 0x7f3f18156950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:22.220 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.204+0000 7f3f1e4ef700 1 -- 172.21.15.195:0/819901817 >> v1:172.21.15.202:6789/0 conn(0x7f3f1806a440 legacy=0x7f3f180c4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:22.220 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.204+0000 7f3f1e4ef700 1 -- 172.21.15.195:0/819901817 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f180c56b0 con 0x7f3f18156100 2022-06-27T12:05:22.221 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.204+0000 7f3f1e4ef700 1 --2- 172.21.15.195:0/819901817 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3f18156100 0x7f3f180688a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3f14006fd0 tx=0x7f3f140090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec971676b42ee692 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:05:22.221 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.205+0000 7f3f1d4ed700 1 -- 172.21.15.195:0/819901817 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f14005ad0 con 0x7f3f18156100 2022-06-27T12:05:22.222 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.205+0000 7f3f1d4ed700 1 -- 172.21.15.195:0/819901817 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f14005c30 con 0x7f3f18156100 2022-06-27T12:05:22.223 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.205+0000 7f3f1d4ed700 1 -- 172.21.15.195:0/819901817 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f14004030 con 0x7f3f18156100 2022-06-27T12:05:22.223 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.205+0000 7f3f20753700 1 -- 172.21.15.195:0/819901817 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3f18156100 msgr2=0x7f3f180688a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:22.224 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.205+0000 7f3f20753700 1 --2- 172.21.15.195:0/819901817 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3f18156100 0x7f3f180688a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3f14006fd0 tx=0x7f3f140090f0 comp rx=0 tx=0).stop 2022-06-27T12:05:22.224 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.205+0000 7f3f20753700 1 -- 172.21.15.195:0/819901817 shutdown_connections 2022-06-27T12:05:22.225 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.205+0000 7f3f20753700 1 --2- 172.21.15.195:0/819901817 >> v2:172.21.15.202:3300/0 conn(0x7f3f181564d0 0x7f3f18156950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:22.225 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.205+0000 7f3f20753700 1 --2- 172.21.15.195:0/819901817 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3f18156100 0x7f3f180688a0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:22.226 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.205+0000 7f3f20753700 1 -- 172.21.15.195:0/819901817 >> 172.21.15.195:0/819901817 conn(0x7f3f180c9b00 msgr2=0x7f3f180c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:05:22.226 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.205+0000 7f3f20753700 1 -- 172.21.15.195:0/819901817 shutdown_connections 2022-06-27T12:05:22.227 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.205+0000 7f3f20753700 1 -- 172.21.15.195:0/819901817 wait complete. 2022-06-27T12:05:22.227 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.207+0000 7f3f20753700 1 Processor -- start 2022-06-27T12:05:22.227 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.207+0000 7f3f20753700 1 -- start start 2022-06-27T12:05:22.228 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.207+0000 7f3f20753700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3f1806a440 0x7f3f18164680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:22.228 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.207+0000 7f3f20753700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3f18156100 0x7f3f180bf210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:22.229 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.207+0000 7f3f20753700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3f181564d0 0x7f3f1814ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:22.229 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.207+0000 7f3f20753700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f1814eab0 con 0x7f3f1806a440 2022-06-27T12:05:22.230 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.207+0000 7f3f20753700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f1814ab10 con 0x7f3f181564d0 2022-06-27T12:05:22.230 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.207+0000 7f3f20753700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f1814ad70 con 0x7f3f18156100 2022-06-27T12:05:22.231 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.207+0000 7f3f1e4ef700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3f1806a440 0x7f3f18164680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:22.231 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.207+0000 7f3f1e4ef700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3f1806a440 0x7f3f18164680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47944/0 (socket says 172.21.15.195:47944) 2022-06-27T12:05:22.232 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.207+0000 7f3f1e4ef700 1 -- 172.21.15.195:0/2820378151 learned_addr learned my addr 172.21.15.195:0/2820378151 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:05:22.232 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.208+0000 7f3f1dcee700 1 --2- 172.21.15.195:0/2820378151 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3f18156100 0x7f3f180bf210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:22.233 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.208+0000 7f3f1ecf0700 1 --2- 172.21.15.195:0/2820378151 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3f181564d0 0x7f3f1814ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:22.233 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.208+0000 7f3f1e4ef700 1 -- 172.21.15.195:0/2820378151 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3f18156100 msgr2=0x7f3f180bf210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:22.234 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.208+0000 7f3f1e4ef700 1 --2- 172.21.15.195:0/2820378151 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3f18156100 0x7f3f180bf210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:22.234 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.208+0000 7f3f1e4ef700 1 -- 172.21.15.195:0/2820378151 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3f181564d0 msgr2=0x7f3f1814ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:22.235 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.208+0000 7f3f1e4ef700 1 --2- 172.21.15.195:0/2820378151 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3f181564d0 0x7f3f1814ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:22.235 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.208+0000 7f3f1e4ef700 1 -- 172.21.15.195:0/2820378151 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f14009350 con 0x7f3f1806a440 2022-06-27T12:05:22.236 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.208+0000 7f3f1e4ef700 1 --2- 172.21.15.195:0/2820378151 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3f1806a440 0x7f3f18164680 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f3f14009ed0 tx=0x7f3f14015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:05:22.236 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.208+0000 7f3f067fc700 1 -- 172.21.15.195:0/2820378151 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f1401d440 con 0x7f3f1806a440 2022-06-27T12:05:22.237 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.208+0000 7f3f067fc700 1 -- 172.21.15.195:0/2820378151 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f1401d5c0 con 0x7f3f1806a440 2022-06-27T12:05:22.237 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.208+0000 7f3f20753700 1 -- 172.21.15.195:0/2820378151 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f1814b050 con 0x7f3f1806a440 2022-06-27T12:05:22.238 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.208+0000 7f3f067fc700 1 -- 172.21.15.195:0/2820378151 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f1401d770 con 0x7f3f1806a440 2022-06-27T12:05:22.238 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.208+0000 7f3f20753700 1 -- 172.21.15.195:0/2820378151 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f1814b610 con 0x7f3f1806a440 2022-06-27T12:05:22.239 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.209+0000 7f3f067fc700 1 -- 172.21.15.195:0/2820378151 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f3f1402e070 con 0x7f3f1806a440 2022-06-27T12:05:22.239 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.209+0000 7f3f20753700 1 -- 172.21.15.195:0/2820378151 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f180722c0 con 0x7f3f1806a440 2022-06-27T12:05:22.239 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.210+0000 7f3f067fc700 1 --2- 172.21.15.195:0/2820378151 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3ef8063d00 0x7f3ef80661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:22.240 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.210+0000 7f3f067fc700 1 -- 172.21.15.195:0/2820378151 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6976+0+0 (secure 0 0 0) 0x7f3f14082ad0 con 0x7f3f1806a440 2022-06-27T12:05:22.240 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.210+0000 7f3f1dcee700 1 --2- 172.21.15.195:0/2820378151 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3ef8063d00 0x7f3ef80661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:22.241 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.210+0000 7f3f1dcee700 1 --2- 172.21.15.195:0/2820378151 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3ef8063d00 0x7f3ef80661b0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f3f18155790 tx=0x7f3f08009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:05:22.242 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.214+0000 7f3f067fc700 1 -- 172.21.15.195:0/2820378151 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f3f14051330 con 0x7f3f1806a440 2022-06-27T12:05:22.441 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.441+0000 7f3f20753700 1 -- 172.21.15.195:0/2820378151 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3f180c6360 con 0x7f3f1806a440 2022-06-27T12:05:22.442 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.441+0000 7f3f067fc700 1 -- 172.21.15.195:0/2820378151 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+5371 (secure 0 0 0) 0x7f3f1402c020 con 0x7f3f1806a440 2022-06-27T12:05:22.443 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:05:22.444 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:05:19.193001+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5489},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5489":{"gid":5489,"name":"d","rank":0,"incarnation":61,"state":"up:reconnect","state_seq":17,"addr":"172.21.15.202:6835/2342195382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2342195382},{"type":"v1","addr":"172.21.15.202:6835","nonce":2342195382}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:05:22.445 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.443+0000 7f3f20753700 1 -- 172.21.15.195:0/2820378151 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3ef8063d00 msgr2=0x7f3ef80661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:22.445 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.443+0000 7f3f20753700 1 --2- 172.21.15.195:0/2820378151 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3ef8063d00 0x7f3ef80661b0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f3f18155790 tx=0x7f3f08009040 comp rx=0 tx=0).stop 2022-06-27T12:05:22.446 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.443+0000 7f3f20753700 1 -- 172.21.15.195:0/2820378151 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3f1806a440 msgr2=0x7f3f18164680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:22.446 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.443+0000 7f3f20753700 1 --2- 172.21.15.195:0/2820378151 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3f1806a440 0x7f3f18164680 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f3f14009ed0 tx=0x7f3f14015000 comp rx=0 tx=0).stop 2022-06-27T12:05:22.446 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.443+0000 7f3f20753700 1 -- 172.21.15.195:0/2820378151 shutdown_connections 2022-06-27T12:05:22.447 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.443+0000 7f3f20753700 1 --2- 172.21.15.195:0/2820378151 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3ef8063d00 0x7f3ef80661b0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:22.447 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.443+0000 7f3f20753700 1 --2- 172.21.15.195:0/2820378151 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3f1806a440 0x7f3f18164680 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:22.447 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.443+0000 7f3f20753700 1 --2- 172.21.15.195:0/2820378151 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3f181564d0 0x7f3f1814ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:22.447 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.443+0000 7f3f20753700 1 --2- 172.21.15.195:0/2820378151 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3f18156100 0x7f3f180bf210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:22.448 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.443+0000 7f3f20753700 1 -- 172.21.15.195:0/2820378151 >> 172.21.15.195:0/2820378151 conn(0x7f3f180c9b00 msgr2=0x7f3f180c4fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:05:22.448 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.443+0000 7f3f20753700 1 -- 172.21.15.195:0/2820378151 shutdown_connections 2022-06-27T12:05:22.448 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:22.443+0000 7f3f20753700 1 -- 172.21.15.195:0/2820378151 wait complete. 2022-06-27T12:05:22.449 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 62 2022-06-27T12:05:22.456 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:05:24.457 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:05:24.616 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.591+0000 7fa8d59a8700 1 Processor -- start 2022-06-27T12:05:24.616 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.591+0000 7fa8d59a8700 1 -- start start 2022-06-27T12:05:24.616 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.591+0000 7fa8d59a8700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fa8d006eff0 0x7fa8d00c61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:24.617 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.591+0000 7fa8d59a8700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa8d0067ed0 0x7fa8d00c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:24.617 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.591+0000 7fa8d59a8700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8d014d8a0 con 0x7fa8d006f3c0 2022-06-27T12:05:24.617 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.591+0000 7fa8d59a8700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8d00c47e0 con 0x7fa8d006eff0 2022-06-27T12:05:24.617 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.591+0000 7fa8d59a8700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d00c3e10 con 0x7fa8d0067ed0 2022-06-27T12:05:24.617 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.592+0000 7fa8cf7fe700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fa8d006f3c0 0x7fa8d00c2ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:54144/0 (socket says 172.21.15.195:54144) 2022-06-27T12:05:24.618 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.592+0000 7fa8cf7fe700 1 -- 172.21.15.195:0/4006808280 learned_addr learned my addr 172.21.15.195:0/4006808280 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:05:24.618 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.592+0000 7fa8ce7fc700 1 --2- 172.21.15.195:0/4006808280 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa8d0067ed0 0x7fa8d00c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:24.618 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.592+0000 7fa8ceffd700 1 --2- 172.21.15.195:0/4006808280 >> v2:172.21.15.202:3300/0 conn(0x7fa8d006eff0 0x7fa8d00c61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:24.618 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.592+0000 7fa8cdffb700 1 -- 172.21.15.195:0/4006808280 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2228164275 0 0) 0x7fa8d014d8a0 con 0x7fa8d006f3c0 2022-06-27T12:05:24.619 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.592+0000 7fa8cdffb700 1 -- 172.21.15.195:0/4006808280 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8bc005640 con 0x7fa8d006f3c0 2022-06-27T12:05:24.619 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.592+0000 7fa8cdffb700 1 -- 172.21.15.195:0/4006808280 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 473177813 0 0) 0x7fa8bc005640 con 0x7fa8d006f3c0 2022-06-27T12:05:24.619 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.592+0000 7fa8cdffb700 1 -- 172.21.15.195:0/4006808280 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa8d0067ed0 msgr2=0x7fa8d00c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:05:24.619 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.592+0000 7fa8cdffb700 1 --2- 172.21.15.195:0/4006808280 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa8d0067ed0 0x7fa8d00c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:24.620 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.592+0000 7fa8cdffb700 1 -- 172.21.15.195:0/4006808280 >> v2:172.21.15.202:3300/0 conn(0x7fa8d006eff0 msgr2=0x7fa8d00c61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:05:24.620 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.592+0000 7fa8cdffb700 1 --2- 172.21.15.195:0/4006808280 >> v2:172.21.15.202:3300/0 conn(0x7fa8d006eff0 0x7fa8d00c61e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:24.620 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.592+0000 7fa8cdffb700 1 -- 172.21.15.195:0/4006808280 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8d006d970 con 0x7fa8d006f3c0 2022-06-27T12:05:24.621 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.592+0000 7fa8cdffb700 1 -- 172.21.15.195:0/4006808280 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7fa8c80017f0 con 0x7fa8d006f3c0 2022-06-27T12:05:24.621 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.592+0000 7fa8cdffb700 1 -- 172.21.15.195:0/4006808280 >> v1:172.21.15.195:6789/0 conn(0x7fa8d006f3c0 legacy=0x7fa8d00c2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:24.621 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.592+0000 7fa8cdffb700 1 --2- 172.21.15.195:0/4006808280 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa8d0067ed0 0x7fa8bc005b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:24.621 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.592+0000 7fa8cdffb700 1 --2- 172.21.15.195:0/4006808280 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa8bc0060b0 0x7fa8bc0068e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:24.621 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.592+0000 7fa8cdffb700 1 --2- 172.21.15.195:0/4006808280 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa8bc006e20 0x7fa8bc007290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:24.622 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.592+0000 7fa8cdffb700 1 -- 172.21.15.195:0/4006808280 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d00c47e0 con 0x7fa8bc006e20 2022-06-27T12:05:24.622 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.592+0000 7fa8cdffb700 1 -- 172.21.15.195:0/4006808280 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d00c3e10 con 0x7fa8bc0060b0 2022-06-27T12:05:24.622 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.592+0000 7fa8cdffb700 1 -- 172.21.15.195:0/4006808280 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8bc00db60 con 0x7fa8d0067ed0 2022-06-27T12:05:24.622 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.592+0000 7fa8ceffd700 1 --2- 172.21.15.195:0/4006808280 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa8d0067ed0 0x7fa8bc005b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:24.623 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.592+0000 7fa8cf7fe700 1 --2- 172.21.15.195:0/4006808280 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa8bc006e20 0x7fa8bc007290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:24.623 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.592+0000 7fa8ce7fc700 1 --2- 172.21.15.195:0/4006808280 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa8bc0060b0 0x7fa8bc0068e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:24.623 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.593+0000 7fa8cf7fe700 1 -- 172.21.15.195:0/4006808280 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa8d0067ed0 msgr2=0x7fa8bc005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:05:24.623 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.593+0000 7fa8cf7fe700 1 --2- 172.21.15.195:0/4006808280 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa8d0067ed0 0x7fa8bc005b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:24.624 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.593+0000 7fa8cf7fe700 1 -- 172.21.15.195:0/4006808280 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa8bc0060b0 msgr2=0x7fa8bc0068e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:05:24.624 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.593+0000 7fa8cf7fe700 1 --2- 172.21.15.195:0/4006808280 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa8bc0060b0 0x7fa8bc0068e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:24.625 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.593+0000 7fa8cf7fe700 1 -- 172.21.15.195:0/4006808280 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8bc005640 con 0x7fa8bc006e20 2022-06-27T12:05:24.625 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.593+0000 7fa8cf7fe700 1 --2- 172.21.15.195:0/4006808280 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa8bc006e20 0x7fa8bc007290 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fa8c800c570 tx=0x7fa8c80021f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23f4c3dca4fb1b6b server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:05:24.625 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.593+0000 7fa8cdffb700 1 -- 172.21.15.195:0/4006808280 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c800ce40 con 0x7fa8bc006e20 2022-06-27T12:05:24.625 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.593+0000 7fa8cdffb700 1 -- 172.21.15.195:0/4006808280 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8c8005b60 con 0x7fa8bc006e20 2022-06-27T12:05:24.626 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.593+0000 7fa8cdffb700 1 -- 172.21.15.195:0/4006808280 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c80070d0 con 0x7fa8bc006e20 2022-06-27T12:05:24.626 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.594+0000 7fa8d59a8700 1 -- 172.21.15.195:0/4006808280 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa8bc006e20 msgr2=0x7fa8bc007290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:24.626 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.594+0000 7fa8d59a8700 1 --2- 172.21.15.195:0/4006808280 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa8bc006e20 0x7fa8bc007290 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fa8c800c570 tx=0x7fa8c80021f0 comp rx=0 tx=0).stop 2022-06-27T12:05:24.626 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.594+0000 7fa8cf7fe700 1 -- 172.21.15.195:0/4006808280 reap_dead start 2022-06-27T12:05:24.627 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.594+0000 7fa8d59a8700 1 -- 172.21.15.195:0/4006808280 shutdown_connections 2022-06-27T12:05:24.627 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.594+0000 7fa8d59a8700 1 -- 172.21.15.195:0/4006808280 >> 172.21.15.195:0/4006808280 conn(0x7fa8d00c9b00 msgr2=0x7fa8d00c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:05:24.627 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.594+0000 7fa8d59a8700 1 -- 172.21.15.195:0/4006808280 shutdown_connections 2022-06-27T12:05:24.627 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.594+0000 7fa8d59a8700 1 -- 172.21.15.195:0/4006808280 wait complete. 2022-06-27T12:05:24.627 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.595+0000 7fa8d59a8700 1 Processor -- start 2022-06-27T12:05:24.628 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.595+0000 7fa8d59a8700 1 -- start start 2022-06-27T12:05:24.628 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.596+0000 7fa8d59a8700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa8d00bd9f0 0x7fa8d0152010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:24.628 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.596+0000 7fa8d59a8700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa8d0166c80 0x7fa8d0152550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:24.628 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.596+0000 7fa8d59a8700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa8d00bbd60 0x7fa8d014a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:24.629 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.596+0000 7fa8d59a8700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d0167180 con 0x7fa8d0166c80 2022-06-27T12:05:24.629 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.596+0000 7fa8d59a8700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d00bfea0 con 0x7fa8d00bd9f0 2022-06-27T12:05:24.629 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.596+0000 7fa8d59a8700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d00bd2c0 con 0x7fa8d00bbd60 2022-06-27T12:05:24.629 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.596+0000 7fa8ce7fc700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa8d0166c80 0x7fa8d0152550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:24.630 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.596+0000 7fa8ce7fc700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa8d0166c80 0x7fa8d0152550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47966/0 (socket says 172.21.15.195:47966) 2022-06-27T12:05:24.630 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.596+0000 7fa8ce7fc700 1 -- 172.21.15.195:0/249775862 learned_addr learned my addr 172.21.15.195:0/249775862 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:05:24.630 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.596+0000 7fa8ceffd700 1 --2- 172.21.15.195:0/249775862 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa8d00bd9f0 0x7fa8d0152010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:24.630 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.596+0000 7fa8cf7fe700 1 --2- 172.21.15.195:0/249775862 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa8d00bbd60 0x7fa8d014a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:24.631 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.596+0000 7fa8ce7fc700 1 -- 172.21.15.195:0/249775862 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa8d00bbd60 msgr2=0x7fa8d014a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:24.631 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.596+0000 7fa8ce7fc700 1 --2- 172.21.15.195:0/249775862 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa8d00bbd60 0x7fa8d014a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:24.631 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.596+0000 7fa8ce7fc700 1 -- 172.21.15.195:0/249775862 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa8d00bd9f0 msgr2=0x7fa8d0152010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:24.631 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.596+0000 7fa8ce7fc700 1 --2- 172.21.15.195:0/249775862 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa8d00bd9f0 0x7fa8d0152010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:24.631 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.596+0000 7fa8ce7fc700 1 -- 172.21.15.195:0/249775862 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8d00bf8e0 con 0x7fa8d0166c80 2022-06-27T12:05:24.632 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.596+0000 7fa8ce7fc700 1 --2- 172.21.15.195:0/249775862 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa8d0166c80 0x7fa8d0152550 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fa8c4011aa0 tx=0x7fa8c4009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:05:24.632 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.596+0000 7fa8b6ffd700 1 -- 172.21.15.195:0/249775862 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c4011e80 con 0x7fa8d0166c80 2022-06-27T12:05:24.632 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.596+0000 7fa8b6ffd700 1 -- 172.21.15.195:0/249775862 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8c4004240 con 0x7fa8d0166c80 2022-06-27T12:05:24.632 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.596+0000 7fa8d59a8700 1 -- 172.21.15.195:0/249775862 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8d00bfbc0 con 0x7fa8d0166c80 2022-06-27T12:05:24.633 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.596+0000 7fa8b6ffd700 1 -- 172.21.15.195:0/249775862 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c400f940 con 0x7fa8d0166c80 2022-06-27T12:05:24.633 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.597+0000 7fa8d59a8700 1 -- 172.21.15.195:0/249775862 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8d00c7810 con 0x7fa8d0166c80 2022-06-27T12:05:24.633 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.597+0000 7fa8d59a8700 1 -- 172.21.15.195:0/249775862 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8d00722c0 con 0x7fa8d0166c80 2022-06-27T12:05:24.633 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.598+0000 7fa8b6ffd700 1 -- 172.21.15.195:0/249775862 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fa8c4024070 con 0x7fa8d0166c80 2022-06-27T12:05:24.633 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.598+0000 7fa8b6ffd700 1 --2- 172.21.15.195:0/249775862 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa8b0063d00 0x7fa8b00661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:24.634 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.598+0000 7fa8b6ffd700 1 -- 172.21.15.195:0/249775862 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6976+0+0 (secure 0 0 0) 0x7fa8c4018030 con 0x7fa8d0166c80 2022-06-27T12:05:24.634 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.598+0000 7fa8ceffd700 1 --2- 172.21.15.195:0/249775862 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa8b0063d00 0x7fa8b00661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:24.634 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.599+0000 7fa8ceffd700 1 --2- 172.21.15.195:0/249775862 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa8b0063d00 0x7fa8b00661b0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fa8c0006d60 tx=0x7fa8c0006cb0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:05:24.634 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.603+0000 7fa8b6ffd700 1 -- 172.21.15.195:0/249775862 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fa8c40b0c90 con 0x7fa8d0166c80 2022-06-27T12:05:24.824 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.823+0000 7fa8d59a8700 1 -- 172.21.15.195:0/249775862 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa8d0153f20 con 0x7fa8d0166c80 2022-06-27T12:05:24.825 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.824+0000 7fa8b6ffd700 1 -- 172.21.15.195:0/249775862 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+5371 (secure 0 0 0) 0x7fa8c4022020 con 0x7fa8d0166c80 2022-06-27T12:05:24.826 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:05:24.826 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:05:19.193001+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5489},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5489":{"gid":5489,"name":"d","rank":0,"incarnation":61,"state":"up:reconnect","state_seq":17,"addr":"172.21.15.202:6835/2342195382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2342195382},{"type":"v1","addr":"172.21.15.202:6835","nonce":2342195382}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:05:24.829 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.826+0000 7fa8d59a8700 1 -- 172.21.15.195:0/249775862 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa8b0063d00 msgr2=0x7fa8b00661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:24.829 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.826+0000 7fa8d59a8700 1 --2- 172.21.15.195:0/249775862 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa8b0063d00 0x7fa8b00661b0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fa8c0006d60 tx=0x7fa8c0006cb0 comp rx=0 tx=0).stop 2022-06-27T12:05:24.831 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.826+0000 7fa8d59a8700 1 -- 172.21.15.195:0/249775862 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa8d0166c80 msgr2=0x7fa8d0152550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:24.831 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.826+0000 7fa8d59a8700 1 --2- 172.21.15.195:0/249775862 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa8d0166c80 0x7fa8d0152550 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fa8c4011aa0 tx=0x7fa8c4009060 comp rx=0 tx=0).stop 2022-06-27T12:05:24.831 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.826+0000 7fa8d59a8700 1 -- 172.21.15.195:0/249775862 shutdown_connections 2022-06-27T12:05:24.832 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.826+0000 7fa8d59a8700 1 --2- 172.21.15.195:0/249775862 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa8b0063d00 0x7fa8b00661b0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:24.832 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.826+0000 7fa8d59a8700 1 --2- 172.21.15.195:0/249775862 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa8d00bbd60 0x7fa8d014a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:24.832 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.826+0000 7fa8d59a8700 1 --2- 172.21.15.195:0/249775862 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa8d00bd9f0 0x7fa8d0152010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:24.832 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.826+0000 7fa8d59a8700 1 --2- 172.21.15.195:0/249775862 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa8d0166c80 0x7fa8d0152550 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:24.833 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.826+0000 7fa8d59a8700 1 -- 172.21.15.195:0/249775862 >> 172.21.15.195:0/249775862 conn(0x7fa8d00c9b00 msgr2=0x7fa8d0063c40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:05:24.833 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.826+0000 7fa8d59a8700 1 -- 172.21.15.195:0/249775862 shutdown_connections 2022-06-27T12:05:24.833 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:24.826+0000 7fa8d59a8700 1 -- 172.21.15.195:0/249775862 wait complete. 2022-06-27T12:05:24.834 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 62 2022-06-27T12:05:24.843 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:05:26.854 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:05:26.990 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:26.990+0000 7f5a44f3a700 1 Processor -- start 2022-06-27T12:05:27.002 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:26.990+0000 7f5a44f3a700 1 -- start start 2022-06-27T12:05:27.003 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:26.991+0000 7f5a44f3a700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f5a4006f640 0x7f5a40156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:27.003 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:26.991+0000 7f5a44f3a700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f5a400c39e0 0x7f5a401535a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:27.004 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:26.991+0000 7f5a44f3a700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a400c1a20 con 0x7f5a400c4160 2022-06-27T12:05:27.004 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:26.991+0000 7f5a44f3a700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a400c1130 con 0x7f5a400c39e0 2022-06-27T12:05:27.005 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:26.991+0000 7f5a44f3a700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a4006ea90 con 0x7f5a4006f640 2022-06-27T12:05:27.005 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:26.991+0000 7f5a3ed9d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f5a400c39e0 0x7f5a401535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:27.006 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:26.991+0000 7f5a3ed9d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f5a400c39e0 0x7f5a401535a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47972/0 (socket says 172.21.15.195:47972) 2022-06-27T12:05:27.006 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:26.991+0000 7f5a3ed9d700 1 -- 172.21.15.195:0/1143684417 learned_addr learned my addr 172.21.15.195:0/1143684417 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:05:27.007 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:26.991+0000 7f5a3e59c700 1 --2- 172.21.15.195:0/1143684417 >> v2:172.21.15.202:3300/0 conn(0x7f5a4006f640 0x7f5a40156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:27.007 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:26.992+0000 7f5a3ed9d700 1 -- 172.21.15.195:0/1143684417 >> v2:172.21.15.202:3300/0 conn(0x7f5a4006f640 msgr2=0x7f5a40156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:05:27.007 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:26.992+0000 7f5a3ed9d700 1 --2- 172.21.15.195:0/1143684417 >> v2:172.21.15.202:3300/0 conn(0x7f5a4006f640 0x7f5a40156ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:27.008 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:26.992+0000 7f5a3ed9d700 1 -- 172.21.15.195:0/1143684417 >> v1:172.21.15.202:6789/0 conn(0x7f5a400c4160 legacy=0x7f5a400c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:05:27.008 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:26.992+0000 7f5a3ed9d700 1 -- 172.21.15.195:0/1143684417 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a4006ed70 con 0x7f5a400c39e0 2022-06-27T12:05:27.009 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:26.992+0000 7f5a3e59c700 1 --2- 172.21.15.195:0/1143684417 >> v2:172.21.15.202:3300/0 conn(0x7f5a4006f640 0x7f5a40156ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T12:05:27.009 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:26.992+0000 7f5a3ed9d700 1 --2- 172.21.15.195:0/1143684417 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5a400c39e0 0x7f5a401535a0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f5a4006db30 tx=0x7f5a38009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=254a94b7f1ad65e server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:05:27.010 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:26.992+0000 7f5a3d59a700 1 -- 172.21.15.195:0/1143684417 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a3800e070 con 0x7f5a400c39e0 2022-06-27T12:05:27.010 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:26.992+0000 7f5a3d59a700 1 -- 172.21.15.195:0/1143684417 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a3800baa0 con 0x7f5a400c39e0 2022-06-27T12:05:27.011 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:26.992+0000 7f5a3d59a700 1 -- 172.21.15.195:0/1143684417 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a380092e0 con 0x7f5a400c39e0 2022-06-27T12:05:27.011 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:26.992+0000 7f5a44f3a700 1 -- 172.21.15.195:0/1143684417 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5a400c39e0 msgr2=0x7f5a401535a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:27.012 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:26.992+0000 7f5a44f3a700 1 --2- 172.21.15.195:0/1143684417 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5a400c39e0 0x7f5a401535a0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f5a4006db30 tx=0x7f5a38009840 comp rx=0 tx=0).stop 2022-06-27T12:05:27.012 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:26.992+0000 7f5a44f3a700 1 -- 172.21.15.195:0/1143684417 shutdown_connections 2022-06-27T12:05:27.013 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:26.992+0000 7f5a44f3a700 1 --2- 172.21.15.195:0/1143684417 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5a400c39e0 0x7f5a401535a0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:27.013 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:26.992+0000 7f5a44f3a700 1 --2- 172.21.15.195:0/1143684417 >> v2:172.21.15.202:3300/0 conn(0x7f5a4006f640 0x7f5a40156ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:27.013 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:26.992+0000 7f5a44f3a700 1 -- 172.21.15.195:0/1143684417 >> 172.21.15.195:0/1143684417 conn(0x7f5a400c6350 msgr2=0x7f5a400c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:05:27.014 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:26.993+0000 7f5a44f3a700 1 -- 172.21.15.195:0/1143684417 shutdown_connections 2022-06-27T12:05:27.014 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:26.993+0000 7f5a44f3a700 1 -- 172.21.15.195:0/1143684417 wait complete. 2022-06-27T12:05:27.015 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:26.994+0000 7f5a44f3a700 1 Processor -- start 2022-06-27T12:05:27.015 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:26.994+0000 7f5a44f3a700 1 -- start start 2022-06-27T12:05:27.016 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:26.994+0000 7f5a44f3a700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5a4006f640 0x7f5a4014cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:27.016 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:26.994+0000 7f5a44f3a700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5a400c39e0 0x7f5a400c9030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:27.017 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:26.994+0000 7f5a44f3a700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5a400c4160 0x7f5a401519c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:27.019 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:26.994+0000 7f5a44f3a700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a40166c30 con 0x7f5a400c4160 2022-06-27T12:05:27.020 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:26.994+0000 7f5a44f3a700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a4014e820 con 0x7f5a400c39e0 2022-06-27T12:05:27.021 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:26.994+0000 7f5a44f3a700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a4014ea90 con 0x7f5a4006f640 2022-06-27T12:05:27.021 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:26.994+0000 7f5a3ed9d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5a400c4160 0x7f5a401519c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:27.022 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:26.995+0000 7f5a3ed9d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5a400c4160 0x7f5a401519c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47982/0 (socket says 172.21.15.195:47982) 2022-06-27T12:05:27.022 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:26.995+0000 7f5a3ed9d700 1 -- 172.21.15.195:0/3952809838 learned_addr learned my addr 172.21.15.195:0/3952809838 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:05:27.023 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:26.995+0000 7f5a3dd9b700 1 --2- 172.21.15.195:0/3952809838 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5a400c39e0 0x7f5a400c9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:27.023 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:26.995+0000 7f5a3e59c700 1 --2- 172.21.15.195:0/3952809838 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5a4006f640 0x7f5a4014cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:27.024 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:26.995+0000 7f5a3ed9d700 1 -- 172.21.15.195:0/3952809838 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5a4006f640 msgr2=0x7f5a4014cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:27.024 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:26.995+0000 7f5a3ed9d700 1 --2- 172.21.15.195:0/3952809838 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5a4006f640 0x7f5a4014cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:27.024 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:26.995+0000 7f5a3ed9d700 1 -- 172.21.15.195:0/3952809838 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5a400c39e0 msgr2=0x7f5a400c9030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:27.025 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:26.995+0000 7f5a3ed9d700 1 --2- 172.21.15.195:0/3952809838 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5a400c39e0 0x7f5a400c9030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:27.025 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:26.995+0000 7f5a3ed9d700 1 -- 172.21.15.195:0/3952809838 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a38009ae0 con 0x7f5a400c4160 2022-06-27T12:05:27.026 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:26.995+0000 7f5a3ed9d700 1 --2- 172.21.15.195:0/3952809838 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5a400c4160 0x7f5a401519c0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f5a38003de0 tx=0x7f5a38017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:05:27.026 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:26.995+0000 7f5a2e7fc700 1 -- 172.21.15.195:0/3952809838 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a38004240 con 0x7f5a400c4160 2022-06-27T12:05:27.027 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:26.995+0000 7f5a2e7fc700 1 -- 172.21.15.195:0/3952809838 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a380043a0 con 0x7f5a400c4160 2022-06-27T12:05:27.027 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:26.995+0000 7f5a44f3a700 1 -- 172.21.15.195:0/3952809838 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a4014ed70 con 0x7f5a400c4160 2022-06-27T12:05:27.028 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:26.995+0000 7f5a2e7fc700 1 -- 172.21.15.195:0/3952809838 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a38004740 con 0x7f5a400c4160 2022-06-27T12:05:27.028 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:26.995+0000 7f5a44f3a700 1 -- 172.21.15.195:0/3952809838 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a4014f330 con 0x7f5a400c4160 2022-06-27T12:05:27.029 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:26.996+0000 7f5a44f3a700 1 -- 172.21.15.195:0/3952809838 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a40166d80 con 0x7f5a400c4160 2022-06-27T12:05:27.029 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:26.997+0000 7f5a2e7fc700 1 -- 172.21.15.195:0/3952809838 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f5a3802c070 con 0x7f5a400c4160 2022-06-27T12:05:27.030 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:26.997+0000 7f5a2e7fc700 1 --2- 172.21.15.195:0/3952809838 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5a1c063d00 0x7f5a1c0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:27.030 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:26.997+0000 7f5a2e7fc700 1 -- 172.21.15.195:0/3952809838 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6976+0+0 (secure 0 0 0) 0x7f5a38082170 con 0x7f5a400c4160 2022-06-27T12:05:27.030 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:26.997+0000 7f5a3e59c700 1 --2- 172.21.15.195:0/3952809838 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5a1c063d00 0x7f5a1c0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:27.031 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:26.997+0000 7f5a3e59c700 1 --2- 172.21.15.195:0/3952809838 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5a1c063d00 0x7f5a1c0661b0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f5a34008900 tx=0x7f5a34008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:05:27.031 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:27.001+0000 7f5a2e7fc700 1 -- 172.21.15.195:0/3952809838 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f5a40166d80 con 0x7f5a400c4160 2022-06-27T12:05:27.232 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:27.231+0000 7f5a44f3a700 1 -- 172.21.15.195:0/3952809838 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a40166d80 con 0x7f5a400c4160 2022-06-27T12:05:27.233 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:27.232+0000 7f5a2e7fc700 1 -- 172.21.15.195:0/3952809838 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+5368 (secure 0 0 0) 0x7f5a40166d80 con 0x7f5a400c4160 2022-06-27T12:05:27.234 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:05:27.234 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:05:25.209912+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5489},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5489":{"gid":5489,"name":"d","rank":0,"incarnation":61,"state":"up:rejoin","state_seq":19,"addr":"172.21.15.202:6835/2342195382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2342195382},{"type":"v1","addr":"172.21.15.202:6835","nonce":2342195382}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:05:27.237 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:27.234+0000 7f5a44f3a700 1 -- 172.21.15.195:0/3952809838 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5a1c063d00 msgr2=0x7f5a1c0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:27.238 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:27.234+0000 7f5a44f3a700 1 --2- 172.21.15.195:0/3952809838 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5a1c063d00 0x7f5a1c0661b0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f5a34008900 tx=0x7f5a34008040 comp rx=0 tx=0).stop 2022-06-27T12:05:27.238 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:27.234+0000 7f5a44f3a700 1 -- 172.21.15.195:0/3952809838 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5a400c4160 msgr2=0x7f5a401519c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:27.239 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:27.234+0000 7f5a44f3a700 1 --2- 172.21.15.195:0/3952809838 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5a400c4160 0x7f5a401519c0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f5a38003de0 tx=0x7f5a38017000 comp rx=0 tx=0).stop 2022-06-27T12:05:27.239 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:27.234+0000 7f5a44f3a700 1 -- 172.21.15.195:0/3952809838 shutdown_connections 2022-06-27T12:05:27.240 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:27.234+0000 7f5a44f3a700 1 --2- 172.21.15.195:0/3952809838 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5a1c063d00 0x7f5a1c0661b0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:27.240 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:27.234+0000 7f5a44f3a700 1 --2- 172.21.15.195:0/3952809838 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5a400c4160 0x7f5a401519c0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:27.241 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:27.234+0000 7f5a44f3a700 1 --2- 172.21.15.195:0/3952809838 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5a4006f640 0x7f5a4014cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:27.242 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:27.234+0000 7f5a44f3a700 1 --2- 172.21.15.195:0/3952809838 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5a400c39e0 0x7f5a400c9030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:27.243 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:27.234+0000 7f5a44f3a700 1 -- 172.21.15.195:0/3952809838 >> 172.21.15.195:0/3952809838 conn(0x7f5a400c6350 msgr2=0x7f5a400c0a10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:05:27.249 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:27.234+0000 7f5a44f3a700 1 -- 172.21.15.195:0/3952809838 shutdown_connections 2022-06-27T12:05:27.250 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:27.234+0000 7f5a44f3a700 1 -- 172.21.15.195:0/3952809838 wait complete. 2022-06-27T12:05:27.250 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 63 2022-06-27T12:05:27.254 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:05:29.256 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:05:29.391 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.391+0000 7f1f30ceb700 1 Processor -- start 2022-06-27T12:05:29.391 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.391+0000 7f1f30ceb700 1 -- start start 2022-06-27T12:05:29.407 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.391+0000 7f1f30ceb700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1f2c151560 0x7f1f2c0c2120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:29.407 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.392+0000 7f1f30ceb700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f1f2c151930 0x7f1f2c0bf4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:29.408 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.392+0000 7f1f30ceb700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1f2c0c2c70 0x7f1f2c1541f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:29.408 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.392+0000 7f1f30ceb700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f2c154da0 con 0x7f1f2c151560 2022-06-27T12:05:29.408 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.392+0000 7f1f30ceb700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f2c0c12a0 con 0x7f1f2c151930 2022-06-27T12:05:29.409 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.392+0000 7f1f30ceb700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f2c0c1580 con 0x7f1f2c0c2c70 2022-06-27T12:05:29.409 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.392+0000 7f1f2a59c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1f2c151560 0x7f1f2c0c2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:29.410 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.392+0000 7f1f2a59c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1f2c151560 0x7f1f2c0c2120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47988/0 (socket says 172.21.15.195:47988) 2022-06-27T12:05:29.410 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.392+0000 7f1f2a59c700 1 -- 172.21.15.195:0/2684702778 learned_addr learned my addr 172.21.15.195:0/2684702778 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:05:29.410 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.392+0000 7f1f29d9b700 1 --2- 172.21.15.195:0/2684702778 >> v2:172.21.15.202:3300/0 conn(0x7f1f2c151930 0x7f1f2c0bf4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:29.411 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.392+0000 7f1f2ad9d700 1 --2- 172.21.15.195:0/2684702778 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1f2c0c2c70 0x7f1f2c1541f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:29.411 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.392+0000 7f1f2a59c700 1 -- 172.21.15.195:0/2684702778 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1f2c0c2c70 msgr2=0x7f1f2c1541f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:05:29.411 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.392+0000 7f1f2a59c700 1 --2- 172.21.15.195:0/2684702778 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1f2c0c2c70 0x7f1f2c1541f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:29.412 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.392+0000 7f1f2a59c700 1 -- 172.21.15.195:0/2684702778 >> v2:172.21.15.202:3300/0 conn(0x7f1f2c151930 msgr2=0x7f1f2c0bf4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:05:29.412 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.392+0000 7f1f2a59c700 1 --2- 172.21.15.195:0/2684702778 >> v2:172.21.15.202:3300/0 conn(0x7f1f2c151930 0x7f1f2c0bf4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:29.413 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.392+0000 7f1f2a59c700 1 -- 172.21.15.195:0/2684702778 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f2c0c29f0 con 0x7f1f2c151560 2022-06-27T12:05:29.414 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.392+0000 7f1f2a59c700 1 --2- 172.21.15.195:0/2684702778 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1f2c151560 0x7f1f2c0c2120 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c0c2660 tx=0x7f1f20009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abe93a380a3839b7 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:05:29.414 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.392+0000 7f1f2959a700 1 -- 172.21.15.195:0/2684702778 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f20011630 con 0x7f1f2c151560 2022-06-27T12:05:29.416 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.392+0000 7f1f2959a700 1 -- 172.21.15.195:0/2684702778 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f20011790 con 0x7f1f2c151560 2022-06-27T12:05:29.416 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.392+0000 7f1f2959a700 1 -- 172.21.15.195:0/2684702778 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f20011b30 con 0x7f1f2c151560 2022-06-27T12:05:29.417 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.393+0000 7f1f30ceb700 1 -- 172.21.15.195:0/2684702778 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1f2c151560 msgr2=0x7f1f2c0c2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:29.417 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.393+0000 7f1f30ceb700 1 --2- 172.21.15.195:0/2684702778 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1f2c151560 0x7f1f2c0c2120 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c0c2660 tx=0x7f1f20009120 comp rx=0 tx=0).stop 2022-06-27T12:05:29.418 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.393+0000 7f1f30ceb700 1 -- 172.21.15.195:0/2684702778 shutdown_connections 2022-06-27T12:05:29.418 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.393+0000 7f1f30ceb700 1 --2- 172.21.15.195:0/2684702778 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1f2c0c2c70 0x7f1f2c1541f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:29.419 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.393+0000 7f1f30ceb700 1 --2- 172.21.15.195:0/2684702778 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1f2c151560 0x7f1f2c0c2120 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:29.419 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.393+0000 7f1f30ceb700 1 --2- 172.21.15.195:0/2684702778 >> v2:172.21.15.202:3300/0 conn(0x7f1f2c151930 0x7f1f2c0bf4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:29.420 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.393+0000 7f1f30ceb700 1 -- 172.21.15.195:0/2684702778 >> 172.21.15.195:0/2684702778 conn(0x7f1f2c0c6350 msgr2=0x7f1f2c0c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:05:29.420 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.393+0000 7f1f30ceb700 1 -- 172.21.15.195:0/2684702778 shutdown_connections 2022-06-27T12:05:29.421 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.393+0000 7f1f30ceb700 1 -- 172.21.15.195:0/2684702778 wait complete. 2022-06-27T12:05:29.422 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.394+0000 7f1f30ceb700 1 Processor -- start 2022-06-27T12:05:29.422 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.394+0000 7f1f30ceb700 1 -- start start 2022-06-27T12:05:29.423 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.395+0000 7f1f30ceb700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1f2c0c2c70 0x7f1f2c06a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:29.423 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.395+0000 7f1f30ceb700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1f2c151560 0x7f1f2c06ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:29.424 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.395+0000 7f1f30ceb700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1f2c151930 0x7f1f2c169e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:29.424 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.395+0000 7f1f30ceb700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f2c0701e0 con 0x7f1f2c151560 2022-06-27T12:05:29.425 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.395+0000 7f1f30ceb700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f2c0ba330 con 0x7f1f2c0c2c70 2022-06-27T12:05:29.425 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.395+0000 7f1f30ceb700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f2c06fcb0 con 0x7f1f2c151930 2022-06-27T12:05:29.426 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.395+0000 7f1f29d9b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1f2c151560 0x7f1f2c06ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:29.426 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.395+0000 7f1f29d9b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1f2c151560 0x7f1f2c06ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:47994/0 (socket says 172.21.15.195:47994) 2022-06-27T12:05:29.427 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.395+0000 7f1f29d9b700 1 -- 172.21.15.195:0/551757739 learned_addr learned my addr 172.21.15.195:0/551757739 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:05:29.427 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.395+0000 7f1f2a59c700 1 --2- 172.21.15.195:0/551757739 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1f2c0c2c70 0x7f1f2c06a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:29.428 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.395+0000 7f1f2ad9d700 1 --2- 172.21.15.195:0/551757739 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1f2c151930 0x7f1f2c169e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:29.428 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.395+0000 7f1f29d9b700 1 -- 172.21.15.195:0/551757739 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1f2c151930 msgr2=0x7f1f2c169e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:29.429 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.395+0000 7f1f29d9b700 1 --2- 172.21.15.195:0/551757739 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1f2c151930 0x7f1f2c169e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:29.429 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.395+0000 7f1f29d9b700 1 -- 172.21.15.195:0/551757739 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1f2c0c2c70 msgr2=0x7f1f2c06a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:29.430 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.395+0000 7f1f29d9b700 1 --2- 172.21.15.195:0/551757739 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1f2c0c2c70 0x7f1f2c06a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:29.430 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.395+0000 7f1f29d9b700 1 -- 172.21.15.195:0/551757739 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f200093a0 con 0x7f1f2c151560 2022-06-27T12:05:29.431 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.395+0000 7f1f29d9b700 1 --2- 172.21.15.195:0/551757739 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1f2c151560 0x7f1f2c06ec10 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c00e970 tx=0x7f1f1c009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:05:29.431 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.395+0000 7f1f1a7fc700 1 -- 172.21.15.195:0/551757739 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f1c013680 con 0x7f1f2c151560 2022-06-27T12:05:29.432 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.396+0000 7f1f1a7fc700 1 -- 172.21.15.195:0/551757739 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f1c0137e0 con 0x7f1f2c151560 2022-06-27T12:05:29.432 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.396+0000 7f1f30ceb700 1 -- 172.21.15.195:0/551757739 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f2c06e780 con 0x7f1f2c151560 2022-06-27T12:05:29.433 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.396+0000 7f1f1a7fc700 1 -- 172.21.15.195:0/551757739 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f1c013b60 con 0x7f1f2c151560 2022-06-27T12:05:29.433 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.396+0000 7f1f30ceb700 1 -- 172.21.15.195:0/551757739 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f2c158210 con 0x7f1f2c151560 2022-06-27T12:05:29.433 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.397+0000 7f1f30ceb700 1 -- 172.21.15.195:0/551757739 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f2c06e910 con 0x7f1f2c151560 2022-06-27T12:05:29.434 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.397+0000 7f1f1a7fc700 1 -- 172.21.15.195:0/551757739 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f1f1c023070 con 0x7f1f2c151560 2022-06-27T12:05:29.434 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.397+0000 7f1f1a7fc700 1 --2- 172.21.15.195:0/551757739 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1f08063d00 0x7f1f080661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:29.435 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.397+0000 7f1f1a7fc700 1 -- 172.21.15.195:0/551757739 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6976+0+0 (secure 0 0 0) 0x7f1f1c080e80 con 0x7f1f2c151560 2022-06-27T12:05:29.435 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.398+0000 7f1f2a59c700 1 --2- 172.21.15.195:0/551757739 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1f08063d00 0x7f1f080661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:29.435 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.398+0000 7f1f2a59c700 1 --2- 172.21.15.195:0/551757739 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1f08063d00 0x7f1f080661b0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f1f20009160 tx=0x7f1f20017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:05:29.436 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.402+0000 7f1f1a7fc700 1 -- 172.21.15.195:0/551757739 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f1f1c04f6e0 con 0x7f1f2c151560 2022-06-27T12:05:29.629 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.628+0000 7f1f30ceb700 1 -- 172.21.15.195:0/551757739 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f2c0722c0 con 0x7f1f2c151560 2022-06-27T12:05:29.630 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.628+0000 7f1f1a7fc700 1 -- 172.21.15.195:0/551757739 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+5368 (secure 0 0 0) 0x7f1f1c021090 con 0x7f1f2c151560 2022-06-27T12:05:29.630 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:05:29.631 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:05:25.209912+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5489},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5489":{"gid":5489,"name":"d","rank":0,"incarnation":61,"state":"up:rejoin","state_seq":19,"addr":"172.21.15.202:6835/2342195382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2342195382},{"type":"v1","addr":"172.21.15.202:6835","nonce":2342195382}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:05:29.632 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.630+0000 7f1f30ceb700 1 -- 172.21.15.195:0/551757739 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1f08063d00 msgr2=0x7f1f080661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:29.632 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.630+0000 7f1f30ceb700 1 --2- 172.21.15.195:0/551757739 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1f08063d00 0x7f1f080661b0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f1f20009160 tx=0x7f1f20017000 comp rx=0 tx=0).stop 2022-06-27T12:05:29.632 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.630+0000 7f1f30ceb700 1 -- 172.21.15.195:0/551757739 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1f2c151560 msgr2=0x7f1f2c06ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:29.633 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.630+0000 7f1f30ceb700 1 --2- 172.21.15.195:0/551757739 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1f2c151560 0x7f1f2c06ec10 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c00e970 tx=0x7f1f1c009060 comp rx=0 tx=0).stop 2022-06-27T12:05:29.633 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.630+0000 7f1f30ceb700 1 -- 172.21.15.195:0/551757739 shutdown_connections 2022-06-27T12:05:29.633 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.630+0000 7f1f30ceb700 1 --2- 172.21.15.195:0/551757739 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1f08063d00 0x7f1f080661b0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:29.633 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.630+0000 7f1f30ceb700 1 --2- 172.21.15.195:0/551757739 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1f2c151930 0x7f1f2c169e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:29.634 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.630+0000 7f1f30ceb700 1 --2- 172.21.15.195:0/551757739 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1f2c0c2c70 0x7f1f2c06a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:29.634 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.630+0000 7f1f30ceb700 1 --2- 172.21.15.195:0/551757739 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1f2c151560 0x7f1f2c06ec10 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:29.634 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.630+0000 7f1f30ceb700 1 -- 172.21.15.195:0/551757739 >> 172.21.15.195:0/551757739 conn(0x7f1f2c0c6350 msgr2=0x7f1f2c14f270 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:05:29.634 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.630+0000 7f1f30ceb700 1 -- 172.21.15.195:0/551757739 shutdown_connections 2022-06-27T12:05:29.635 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:29.630+0000 7f1f30ceb700 1 -- 172.21.15.195:0/551757739 wait complete. 2022-06-27T12:05:29.635 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 63 2022-06-27T12:05:29.643 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 63, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-06-27T11:47:52.358095+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 63, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5489": { "addr": "172.21.15.202:6835/2342195382", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6834", "nonce": 2342195382, "type": "v2" }, { "addr": "172.21.15.202:6835", "nonce": 2342195382, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5489, "incarnation": 61, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:rejoin", "state_seq": 19 } }, "last_failure": 0, "last_failure_osd_epoch": 57, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-06-27T12:05:25.209912+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5489 } } } ], "standbys": [ { "addr": "172.21.15.202:6837/3305006841", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6836", "nonce": 3305006841, "type": "v2" }, { "addr": "172.21.15.202:6837", "nonce": 3305006841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6835/2060709822", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6834", "nonce": 2060709822, "type": "v2" }, { "addr": "172.21.15.195:6835", "nonce": 2060709822, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6838/1598751652", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6836", "nonce": 1598751652, "type": "v2" }, { "addr": "172.21.15.195:6838", "nonce": 1598751652, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4275, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6839/1313800661", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6837", "nonce": 1313800661, "type": "v2" }, { "addr": "172.21.15.195:6839", "nonce": 1313800661, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4278, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-06-27T12:05:31.021 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:05:31.026 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:05:31.644 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:05:31.786 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:31.778+0000 7fb5949be700 1 Processor -- start 2022-06-27T12:05:31.786 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:31.778+0000 7fb5949be700 1 -- start start 2022-06-27T12:05:31.786 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:31.779+0000 7fb5949be700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb5900c3030 0x7fb5901570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:31.787 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:31.779+0000 7fb5949be700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb59006fba0 con 0x7fb59006fef0 2022-06-27T12:05:31.787 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:31.779+0000 7fb5949be700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb59006f5e0 con 0x7fb59006e850 2022-06-27T12:05:31.787 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:31.779+0000 7fb5949be700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5901575e0 con 0x7fb5900c3030 2022-06-27T12:05:31.787 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:31.779+0000 7fb58dd9b700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fb59006fef0 0x7fb5900c2920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:54188/0 (socket says 172.21.15.195:54188) 2022-06-27T12:05:31.787 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:31.779+0000 7fb58dd9b700 1 -- 172.21.15.195:0/1252189109 learned_addr learned my addr 172.21.15.195:0/1252189109 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:05:31.788 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:31.779+0000 7fb58ed9d700 1 --2- 172.21.15.195:0/1252189109 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb5900c3030 0x7fb5901570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:31.788 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:31.779+0000 7fb58d59a700 1 -- 172.21.15.195:0/1252189109 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3440389074 0 0) 0x7fb59006fba0 con 0x7fb59006fef0 2022-06-27T12:05:31.788 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:31.779+0000 7fb58d59a700 1 -- 172.21.15.195:0/1252189109 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb574003430 con 0x7fb59006fef0 2022-06-27T12:05:31.788 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:31.780+0000 7fb58d59a700 1 -- 172.21.15.195:0/1252189109 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1568673458 0 0) 0x7fb59006f5e0 con 0x7fb59006e850 2022-06-27T12:05:31.789 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:31.780+0000 7fb58d59a700 1 -- 172.21.15.195:0/1252189109 --> v1:172.21.15.202:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb59006fba0 con 0x7fb59006e850 2022-06-27T12:05:31.789 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:31.780+0000 7fb58ed9d700 1 -- 172.21.15.195:0/1252189109 >> v1:172.21.15.202:6789/0 conn(0x7fb59006e850 legacy=0x7fb590069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:31.789 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:31.780+0000 7fb58d59a700 1 -- 172.21.15.195:0/1252189109 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3159253680 0 0) 0x7fb574003430 con 0x7fb59006fef0 2022-06-27T12:05:31.790 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:31.780+0000 7fb58ed9d700 1 -- 172.21.15.195:0/1252189109 >> v1:172.21.15.195:6789/0 conn(0x7fb59006fef0 legacy=0x7fb5900c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:31.790 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:31.780+0000 7fb58ed9d700 1 -- 172.21.15.195:0/1252189109 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb59006f8c0 con 0x7fb5900c3030 2022-06-27T12:05:31.791 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:31.780+0000 7fb58ed9d700 1 --2- 172.21.15.195:0/1252189109 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb5900c3030 0x7fb5901570a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb588009e90 tx=0x7fb588009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f41c8cd9ac57cfe7 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:05:31.791 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:31.780+0000 7fb58d59a700 1 -- 172.21.15.195:0/1252189109 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb58800e070 con 0x7fb5900c3030 2022-06-27T12:05:31.791 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:31.780+0000 7fb58d59a700 1 -- 172.21.15.195:0/1252189109 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb58800b900 con 0x7fb5900c3030 2022-06-27T12:05:31.791 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:31.780+0000 7fb58d59a700 1 -- 172.21.15.195:0/1252189109 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb580001470 con 0x7fb5900c3030 2022-06-27T12:05:31.792 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:31.780+0000 7fb5949be700 1 -- 172.21.15.195:0/1252189109 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb5900c3030 msgr2=0x7fb5901570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:31.792 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:31.780+0000 7fb5949be700 1 --2- 172.21.15.195:0/1252189109 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb5900c3030 0x7fb5901570a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb588009e90 tx=0x7fb588009780 comp rx=0 tx=0).stop 2022-06-27T12:05:31.792 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:31.781+0000 7fb5949be700 1 -- 172.21.15.195:0/1252189109 shutdown_connections 2022-06-27T12:05:31.792 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:31.781+0000 7fb5949be700 1 --2- 172.21.15.195:0/1252189109 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb5900c3030 0x7fb5901570a0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:31.792 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:31.781+0000 7fb5949be700 1 -- 172.21.15.195:0/1252189109 >> 172.21.15.195:0/1252189109 conn(0x7fb5900c6350 msgr2=0x7fb5900c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:05:31.793 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:31.781+0000 7fb5949be700 1 -- 172.21.15.195:0/1252189109 shutdown_connections 2022-06-27T12:05:31.793 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:31.781+0000 7fb5949be700 1 -- 172.21.15.195:0/1252189109 wait complete. 2022-06-27T12:05:31.793 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:31.782+0000 7fb5949be700 1 Processor -- start 2022-06-27T12:05:31.793 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:31.782+0000 7fb5949be700 1 -- start start 2022-06-27T12:05:31.794 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:31.782+0000 7fb5949be700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb59006e850 0x7fb590152650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:31.794 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:31.782+0000 7fb5949be700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb5900c3030 0x7fb590069ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:31.794 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:31.782+0000 7fb5949be700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb5901536d0 0x7fb590153b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:31.794 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:31.782+0000 7fb5949be700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb590167790 con 0x7fb59006e850 2022-06-27T12:05:31.795 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:31.782+0000 7fb5949be700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb590168310 con 0x7fb5901536d0 2022-06-27T12:05:31.795 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:31.782+0000 7fb5949be700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb590154180 con 0x7fb5900c3030 2022-06-27T12:05:31.796 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:31.783+0000 7fb58e59c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb59006e850 0x7fb590152650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:31.796 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:31.783+0000 7fb58e59c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb59006e850 0x7fb590152650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48004/0 (socket says 172.21.15.195:48004) 2022-06-27T12:05:31.797 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:31.783+0000 7fb58e59c700 1 -- 172.21.15.195:0/3414853298 learned_addr learned my addr 172.21.15.195:0/3414853298 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:05:31.798 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:31.783+0000 7fb58dd9b700 1 --2- 172.21.15.195:0/3414853298 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb5900c3030 0x7fb590069ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:31.799 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:31.783+0000 7fb58ed9d700 1 --2- 172.21.15.195:0/3414853298 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb5901536d0 0x7fb590153b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:31.799 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:31.783+0000 7fb58e59c700 1 -- 172.21.15.195:0/3414853298 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb5900c3030 msgr2=0x7fb590069ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:31.800 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:31.783+0000 7fb58e59c700 1 --2- 172.21.15.195:0/3414853298 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb5900c3030 0x7fb590069ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:31.800 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:31.783+0000 7fb58e59c700 1 -- 172.21.15.195:0/3414853298 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb5901536d0 msgr2=0x7fb590153b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:31.801 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:31.783+0000 7fb58e59c700 1 --2- 172.21.15.195:0/3414853298 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb5901536d0 0x7fb590153b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:31.801 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:31.783+0000 7fb58e59c700 1 -- 172.21.15.195:0/3414853298 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb588009a20 con 0x7fb59006e850 2022-06-27T12:05:31.802 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:31.783+0000 7fb58e59c700 1 --2- 172.21.15.195:0/3414853298 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb59006e850 0x7fb590152650 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fb58400c470 tx=0x7fb584007cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:05:31.806 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:31.783+0000 7fb57e7fc700 1 -- 172.21.15.195:0/3414853298 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5840050d0 con 0x7fb59006e850 2022-06-27T12:05:31.812 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:31.783+0000 7fb57e7fc700 1 -- 172.21.15.195:0/3414853298 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb584005250 con 0x7fb59006e850 2022-06-27T12:05:31.813 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:31.783+0000 7fb57e7fc700 1 -- 172.21.15.195:0/3414853298 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb584003750 con 0x7fb59006e850 2022-06-27T12:05:31.813 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:31.783+0000 7fb5949be700 1 -- 172.21.15.195:0/3414853298 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5901679e0 con 0x7fb59006e850 2022-06-27T12:05:31.813 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:31.783+0000 7fb5949be700 1 -- 172.21.15.195:0/3414853298 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb590167fa0 con 0x7fb59006e850 2022-06-27T12:05:31.814 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:31.784+0000 7fb5949be700 1 -- 172.21.15.195:0/3414853298 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb590168680 con 0x7fb59006e850 2022-06-27T12:05:31.814 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:31.785+0000 7fb57e7fc700 1 -- 172.21.15.195:0/3414853298 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fb584019030 con 0x7fb59006e850 2022-06-27T12:05:31.814 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:31.785+0000 7fb57e7fc700 1 --2- 172.21.15.195:0/3414853298 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb56c063d00 0x7fb56c0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:31.814 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:31.785+0000 7fb57e7fc700 1 -- 172.21.15.195:0/3414853298 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6976+0+0 (secure 0 0 0) 0x7fb58407fe30 con 0x7fb59006e850 2022-06-27T12:05:31.815 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:31.785+0000 7fb58dd9b700 1 --2- 172.21.15.195:0/3414853298 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb56c063d00 0x7fb56c0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:31.815 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:31.786+0000 7fb58dd9b700 1 --2- 172.21.15.195:0/3414853298 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb56c063d00 0x7fb56c0661b0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fb580001eb0 tx=0x7fb580000bc0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:05:31.816 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:31.789+0000 7fb57e7fc700 1 -- 172.21.15.195:0/3414853298 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fb58404e690 con 0x7fb59006e850 2022-06-27T12:05:32.014 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:32.014+0000 7fb5949be700 1 -- 172.21.15.195:0/3414853298 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb5900722c0 con 0x7fb59006e850 2022-06-27T12:05:32.015 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:32.015+0000 7fb57e7fc700 1 -- 172.21.15.195:0/3414853298 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+5374 (secure 0 0 0) 0x7fb584017020 con 0x7fb59006e850 2022-06-27T12:05:32.015 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:05:32.016 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:05:31.228758+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5489},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5489":{"gid":5489,"name":"d","rank":0,"incarnation":61,"state":"up:clientreplay","state_seq":21,"addr":"172.21.15.202:6835/2342195382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2342195382},{"type":"v1","addr":"172.21.15.202:6835","nonce":2342195382}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:05:32.017 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:32.017+0000 7fb5949be700 1 -- 172.21.15.195:0/3414853298 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb56c063d00 msgr2=0x7fb56c0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:32.018 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:32.017+0000 7fb5949be700 1 --2- 172.21.15.195:0/3414853298 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb56c063d00 0x7fb56c0661b0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fb580001eb0 tx=0x7fb580000bc0 comp rx=0 tx=0).stop 2022-06-27T12:05:32.018 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:32.017+0000 7fb5949be700 1 -- 172.21.15.195:0/3414853298 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb59006e850 msgr2=0x7fb590152650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:32.019 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:32.017+0000 7fb5949be700 1 --2- 172.21.15.195:0/3414853298 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb59006e850 0x7fb590152650 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fb58400c470 tx=0x7fb584007cb0 comp rx=0 tx=0).stop 2022-06-27T12:05:32.019 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:32.017+0000 7fb5949be700 1 -- 172.21.15.195:0/3414853298 shutdown_connections 2022-06-27T12:05:32.020 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:32.017+0000 7fb5949be700 1 --2- 172.21.15.195:0/3414853298 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb56c063d00 0x7fb56c0661b0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:32.020 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:32.017+0000 7fb5949be700 1 --2- 172.21.15.195:0/3414853298 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb59006e850 0x7fb590152650 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:32.020 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:32.017+0000 7fb5949be700 1 --2- 172.21.15.195:0/3414853298 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb5901536d0 0x7fb590153b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:32.022 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:32.017+0000 7fb5949be700 1 --2- 172.21.15.195:0/3414853298 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb5900c3030 0x7fb590069ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:32.022 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:32.017+0000 7fb5949be700 1 -- 172.21.15.195:0/3414853298 >> 172.21.15.195:0/3414853298 conn(0x7fb5900c6350 msgr2=0x7fb59006be00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:05:32.022 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:32.017+0000 7fb5949be700 1 -- 172.21.15.195:0/3414853298 shutdown_connections 2022-06-27T12:05:32.022 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:32.017+0000 7fb5949be700 1 -- 172.21.15.195:0/3414853298 wait complete. 2022-06-27T12:05:32.023 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 64 2022-06-27T12:05:32.029 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:05:34.032 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:05:34.203 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.178+0000 7f60db575700 1 Processor -- start 2022-06-27T12:05:34.204 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.178+0000 7f60db575700 1 -- start start 2022-06-27T12:05:34.204 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.179+0000 7f60db575700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f60d40c7480 0x7f60d40c4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:34.205 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.179+0000 7f60db575700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f60d40683c0 0x7f60d4069350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:34.205 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.179+0000 7f60db575700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60d41573b0 con 0x7f60d4068f80 2022-06-27T12:05:34.205 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.179+0000 7f60db575700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60d4068840 con 0x7f60d40683c0 2022-06-27T12:05:34.205 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.179+0000 7f60db575700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60d40c46c0 con 0x7f60d40c7480 2022-06-27T12:05:34.206 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.179+0000 7f60d9b12700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f60d40683c0 0x7f60d4069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:34.206 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.179+0000 7f60d9b12700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f60d40683c0 0x7f60d4069350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48016/0 (socket says 172.21.15.195:48016) 2022-06-27T12:05:34.206 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.179+0000 7f60d9b12700 1 -- 172.21.15.195:0/175959817 learned_addr learned my addr 172.21.15.195:0/175959817 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:05:34.207 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.180+0000 7f60d8b10700 1 --2- 172.21.15.195:0/175959817 >> v2:172.21.15.202:3300/0 conn(0x7f60d40c7480 0x7f60d40c4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:34.207 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.180+0000 7f60d9b12700 1 -- 172.21.15.195:0/175959817 >> v2:172.21.15.202:3300/0 conn(0x7f60d40c7480 msgr2=0x7f60d40c4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:05:34.207 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.180+0000 7f60d9b12700 1 --2- 172.21.15.195:0/175959817 >> v2:172.21.15.202:3300/0 conn(0x7f60d40c7480 0x7f60d40c4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:34.207 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.180+0000 7f60d9b12700 1 -- 172.21.15.195:0/175959817 >> v1:172.21.15.202:6789/0 conn(0x7f60d4068f80 legacy=0x7f60d4156460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:34.208 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.180+0000 7f60d9b12700 1 -- 172.21.15.195:0/175959817 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60d4156b70 con 0x7f60d40683c0 2022-06-27T12:05:34.208 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.180+0000 7f60d9b12700 1 --2- 172.21.15.195:0/175959817 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f60d40683c0 0x7f60d4069350 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f60d40c6430 tx=0x7f60cc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c54f8828bbcf8b6 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:05:34.208 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.180+0000 7f60c3fff700 1 -- 172.21.15.195:0/175959817 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60cc00e070 con 0x7f60d40683c0 2022-06-27T12:05:34.208 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.180+0000 7f60c3fff700 1 -- 172.21.15.195:0/175959817 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60cc00ba50 con 0x7f60d40683c0 2022-06-27T12:05:34.209 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.180+0000 7f60c3fff700 1 -- 172.21.15.195:0/175959817 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60cc0092e0 con 0x7f60d40683c0 2022-06-27T12:05:34.209 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.180+0000 7f60db575700 1 -- 172.21.15.195:0/175959817 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f60d40683c0 msgr2=0x7f60d4069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:34.209 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.180+0000 7f60db575700 1 --2- 172.21.15.195:0/175959817 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f60d40683c0 0x7f60d4069350 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f60d40c6430 tx=0x7f60cc009840 comp rx=0 tx=0).stop 2022-06-27T12:05:34.209 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.180+0000 7f60db575700 1 -- 172.21.15.195:0/175959817 shutdown_connections 2022-06-27T12:05:34.210 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.180+0000 7f60db575700 1 --2- 172.21.15.195:0/175959817 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f60d40683c0 0x7f60d4069350 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:34.210 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.180+0000 7f60db575700 1 --2- 172.21.15.195:0/175959817 >> v2:172.21.15.202:3300/0 conn(0x7f60d40c7480 0x7f60d40c4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:34.210 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.180+0000 7f60db575700 1 -- 172.21.15.195:0/175959817 >> 172.21.15.195:0/175959817 conn(0x7f60d40ccbd0 msgr2=0x7f60d40ccfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:05:34.211 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.181+0000 7f60db575700 1 -- 172.21.15.195:0/175959817 shutdown_connections 2022-06-27T12:05:34.211 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.181+0000 7f60db575700 1 -- 172.21.15.195:0/175959817 wait complete. 2022-06-27T12:05:34.211 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.182+0000 7f60db575700 1 Processor -- start 2022-06-27T12:05:34.211 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.182+0000 7f60db575700 1 -- start start 2022-06-27T12:05:34.212 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.183+0000 7f60db575700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f60d40683c0 0x7f60d41539a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:34.212 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.183+0000 7f60db575700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f60d4068f80 0x7f60d40c9720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:34.212 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.183+0000 7f60db575700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f60d40c7480 0x7f60d4167900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:34.212 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.183+0000 7f60db575700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60d40c5420 con 0x7f60d40c7480 2022-06-27T12:05:34.213 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.183+0000 7f60db575700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60d4154050 con 0x7f60d40683c0 2022-06-27T12:05:34.213 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.183+0000 7f60db575700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60d416a600 con 0x7f60d4068f80 2022-06-27T12:05:34.213 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.183+0000 7f60d9b12700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f60d40c7480 0x7f60d4167900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:34.213 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.183+0000 7f60d9b12700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f60d40c7480 0x7f60d4167900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48022/0 (socket says 172.21.15.195:48022) 2022-06-27T12:05:34.213 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.183+0000 7f60d9b12700 1 -- 172.21.15.195:0/2995299241 learned_addr learned my addr 172.21.15.195:0/2995299241 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:05:34.214 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.183+0000 7f60d8b10700 1 --2- 172.21.15.195:0/2995299241 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f60d4068f80 0x7f60d40c9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:34.214 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.183+0000 7f60d9311700 1 --2- 172.21.15.195:0/2995299241 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f60d40683c0 0x7f60d41539a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:34.214 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.183+0000 7f60d9b12700 1 -- 172.21.15.195:0/2995299241 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f60d4068f80 msgr2=0x7f60d40c9720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:34.214 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.183+0000 7f60d9b12700 1 --2- 172.21.15.195:0/2995299241 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f60d4068f80 0x7f60d40c9720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:34.215 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.183+0000 7f60d9b12700 1 -- 172.21.15.195:0/2995299241 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f60d40683c0 msgr2=0x7f60d41539a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:34.215 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.183+0000 7f60d9b12700 1 --2- 172.21.15.195:0/2995299241 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f60d40683c0 0x7f60d41539a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:34.215 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.183+0000 7f60d9b12700 1 -- 172.21.15.195:0/2995299241 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60cc009ae0 con 0x7f60d40c7480 2022-06-27T12:05:34.215 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.183+0000 7f60d9311700 1 --2- 172.21.15.195:0/2995299241 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f60d40683c0 0x7f60d41539a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T12:05:34.216 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.183+0000 7f60d9b12700 1 --2- 172.21.15.195:0/2995299241 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f60d40c7480 0x7f60d4167900 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f60d40cdc40 tx=0x7f60cc018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:05:34.216 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.183+0000 7f60c17fa700 1 -- 172.21.15.195:0/2995299241 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60cc00bee0 con 0x7f60d40c7480 2022-06-27T12:05:34.216 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.183+0000 7f60c17fa700 1 -- 172.21.15.195:0/2995299241 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60cc007820 con 0x7f60d40c7480 2022-06-27T12:05:34.216 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.183+0000 7f60db575700 1 -- 172.21.15.195:0/2995299241 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60d41699f0 con 0x7f60d40c7480 2022-06-27T12:05:34.216 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.183+0000 7f60c17fa700 1 -- 172.21.15.195:0/2995299241 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60cc007be0 con 0x7f60d40c7480 2022-06-27T12:05:34.217 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.183+0000 7f60db575700 1 -- 172.21.15.195:0/2995299241 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60d4169fb0 con 0x7f60d40c7480 2022-06-27T12:05:34.217 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.184+0000 7f60db575700 1 -- 172.21.15.195:0/2995299241 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60d416a740 con 0x7f60d40c7480 2022-06-27T12:05:34.217 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.185+0000 7f60c17fa700 1 -- 172.21.15.195:0/2995299241 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f60cc004040 con 0x7f60d40c7480 2022-06-27T12:05:34.217 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.185+0000 7f60c17fa700 1 --2- 172.21.15.195:0/2995299241 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f60b4063d30 0x7f60b40661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:34.218 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.185+0000 7f60c17fa700 1 -- 172.21.15.195:0/2995299241 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6976+0+0 (secure 0 0 0) 0x7f60cc082550 con 0x7f60d40c7480 2022-06-27T12:05:34.218 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.185+0000 7f60d9311700 1 --2- 172.21.15.195:0/2995299241 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f60b4063d30 0x7f60b40661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:34.219 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.186+0000 7f60d9311700 1 --2- 172.21.15.195:0/2995299241 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f60b4063d30 0x7f60b40661e0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f60d00016f0 tx=0x7f60d0009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:05:34.219 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.189+0000 7f60c17fa700 1 -- 172.21.15.195:0/2995299241 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f60d416a740 con 0x7f60d40c7480 2022-06-27T12:05:34.419 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.419+0000 7f60db575700 1 -- 172.21.15.195:0/2995299241 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f60d416a740 con 0x7f60d40c7480 2022-06-27T12:05:34.420 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.420+0000 7f60c17fa700 1 -- 172.21.15.195:0/2995299241 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+5374 (secure 0 0 0) 0x7f60d416a740 con 0x7f60d40c7480 2022-06-27T12:05:34.421 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:05:34.421 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:05:31.228758+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5489},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5489":{"gid":5489,"name":"d","rank":0,"incarnation":61,"state":"up:clientreplay","state_seq":21,"addr":"172.21.15.202:6835/2342195382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2342195382},{"type":"v1","addr":"172.21.15.202:6835","nonce":2342195382}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:05:34.422 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.422+0000 7f60db575700 1 -- 172.21.15.195:0/2995299241 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f60b4063d30 msgr2=0x7f60b40661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:34.422 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.422+0000 7f60db575700 1 --2- 172.21.15.195:0/2995299241 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f60b4063d30 0x7f60b40661e0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f60d00016f0 tx=0x7f60d0009040 comp rx=0 tx=0).stop 2022-06-27T12:05:34.423 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.422+0000 7f60db575700 1 -- 172.21.15.195:0/2995299241 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f60d40c7480 msgr2=0x7f60d4167900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:34.423 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.422+0000 7f60db575700 1 --2- 172.21.15.195:0/2995299241 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f60d40c7480 0x7f60d4167900 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f60d40cdc40 tx=0x7f60cc018000 comp rx=0 tx=0).stop 2022-06-27T12:05:34.423 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.422+0000 7f60db575700 1 -- 172.21.15.195:0/2995299241 shutdown_connections 2022-06-27T12:05:34.424 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.422+0000 7f60db575700 1 --2- 172.21.15.195:0/2995299241 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f60b4063d30 0x7f60b40661e0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:34.424 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.422+0000 7f60db575700 1 --2- 172.21.15.195:0/2995299241 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f60d40c7480 0x7f60d4167900 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:34.424 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.422+0000 7f60db575700 1 --2- 172.21.15.195:0/2995299241 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f60d40683c0 0x7f60d41539a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:34.424 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.422+0000 7f60db575700 1 --2- 172.21.15.195:0/2995299241 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f60d4068f80 0x7f60d40c9720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:34.425 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.422+0000 7f60db575700 1 -- 172.21.15.195:0/2995299241 >> 172.21.15.195:0/2995299241 conn(0x7f60d40ccbd0 msgr2=0x7f60d40cb870 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:05:34.425 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.422+0000 7f60db575700 1 -- 172.21.15.195:0/2995299241 shutdown_connections 2022-06-27T12:05:34.425 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:34.422+0000 7f60db575700 1 -- 172.21.15.195:0/2995299241 wait complete. 2022-06-27T12:05:34.426 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 64 2022-06-27T12:05:34.434 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:05:36.435 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:05:36.593 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.575+0000 7fa3edb8f700 1 Processor -- start 2022-06-27T12:05:36.594 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.576+0000 7fa3edb8f700 1 -- start start 2022-06-27T12:05:36.594 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.576+0000 7fa3edb8f700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fa3e8153cb0 0x7fa3e8156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:36.594 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.576+0000 7fa3edb8f700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3e80c1a20 con 0x7fa3e80c4160 2022-06-27T12:05:36.595 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.576+0000 7fa3edb8f700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3e80c1130 con 0x7fa3e806f640 2022-06-27T12:05:36.595 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.576+0000 7fa3edb8f700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3e806ea90 con 0x7fa3e8153cb0 2022-06-27T12:05:36.595 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.576+0000 7fa3e7fff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fa3e8153cb0 0x7fa3e8156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:36.596 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.576+0000 7fa3e7fff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fa3e8153cb0 0x7fa3e8156ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48036/0 (socket says 172.21.15.195:48036) 2022-06-27T12:05:36.596 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.576+0000 7fa3e7fff700 1 -- 172.21.15.195:0/3890419033 learned_addr learned my addr 172.21.15.195:0/3890419033 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:05:36.596 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.576+0000 7fa3e6ffd700 1 --1- 172.21.15.195:0/3890419033 >> v1:172.21.15.195:6789/0 conn(0x7fa3e80c4160 0x7fa3e81535a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:54222/0 (socket says 172.21.15.195:54222) 2022-06-27T12:05:36.596 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.577+0000 7fa3e7fff700 1 -- 172.21.15.195:0/3890419033 >> v1:172.21.15.202:6789/0 conn(0x7fa3e806f640 legacy=0x7fa3e80c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:36.597 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.577+0000 7fa3e7fff700 1 -- 172.21.15.195:0/3890419033 >> v1:172.21.15.195:6789/0 conn(0x7fa3e80c4160 legacy=0x7fa3e81535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:36.597 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.577+0000 7fa3e7fff700 1 -- 172.21.15.195:0/3890419033 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3e806ed70 con 0x7fa3e8153cb0 2022-06-27T12:05:36.597 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.577+0000 7fa3e7fff700 1 --2- 172.21.15.195:0/3890419033 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa3e8153cb0 0x7fa3e8156ed0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fa3e0009f20 tx=0x7fa3e0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eacc34c45e0b9148 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:05:36.597 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.577+0000 7fa3e67fc700 1 -- 172.21.15.195:0/3890419033 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e000e070 con 0x7fa3e8153cb0 2022-06-27T12:05:36.598 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.577+0000 7fa3e67fc700 1 -- 172.21.15.195:0/3890419033 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3e000bb10 con 0x7fa3e8153cb0 2022-06-27T12:05:36.598 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.577+0000 7fa3e67fc700 1 -- 172.21.15.195:0/3890419033 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e00092e0 con 0x7fa3e8153cb0 2022-06-27T12:05:36.599 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.578+0000 7fa3edb8f700 1 -- 172.21.15.195:0/3890419033 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa3e8153cb0 msgr2=0x7fa3e8156ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:36.599 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.578+0000 7fa3edb8f700 1 --2- 172.21.15.195:0/3890419033 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa3e8153cb0 0x7fa3e8156ed0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fa3e0009f20 tx=0x7fa3e0009840 comp rx=0 tx=0).stop 2022-06-27T12:05:36.599 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.578+0000 7fa3edb8f700 1 -- 172.21.15.195:0/3890419033 shutdown_connections 2022-06-27T12:05:36.600 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.578+0000 7fa3edb8f700 1 --2- 172.21.15.195:0/3890419033 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa3e8153cb0 0x7fa3e8156ed0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:36.600 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.578+0000 7fa3edb8f700 1 -- 172.21.15.195:0/3890419033 >> 172.21.15.195:0/3890419033 conn(0x7fa3e80c6350 msgr2=0x7fa3e80c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:05:36.600 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.578+0000 7fa3edb8f700 1 -- 172.21.15.195:0/3890419033 shutdown_connections 2022-06-27T12:05:36.601 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.578+0000 7fa3edb8f700 1 -- 172.21.15.195:0/3890419033 wait complete. 2022-06-27T12:05:36.602 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.579+0000 7fa3edb8f700 1 Processor -- start 2022-06-27T12:05:36.602 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.579+0000 7fa3edb8f700 1 -- start start 2022-06-27T12:05:36.603 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.580+0000 7fa3edb8f700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa3e806f640 0x7fa3e8167100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:36.603 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.580+0000 7fa3edb8f700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa3e80c4160 0x7fa3e80c8b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:36.603 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.580+0000 7fa3edb8f700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa3e8153cb0 0x7fa3e80bdbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:36.603 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.580+0000 7fa3edb8f700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e8069de0 con 0x7fa3e806f640 2022-06-27T12:05:36.604 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.580+0000 7fa3edb8f700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e80680e0 con 0x7fa3e80c4160 2022-06-27T12:05:36.604 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.580+0000 7fa3edb8f700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e80bd6c0 con 0x7fa3e8153cb0 2022-06-27T12:05:36.604 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.580+0000 7fa3e77fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa3e806f640 0x7fa3e8167100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:36.605 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.580+0000 7fa3e77fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa3e806f640 0x7fa3e8167100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48038/0 (socket says 172.21.15.195:48038) 2022-06-27T12:05:36.605 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.580+0000 7fa3e77fe700 1 -- 172.21.15.195:0/3895469746 learned_addr learned my addr 172.21.15.195:0/3895469746 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:05:36.605 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.580+0000 7fa3e6ffd700 1 --2- 172.21.15.195:0/3895469746 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa3e80c4160 0x7fa3e80c8b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:36.606 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.580+0000 7fa3e7fff700 1 --2- 172.21.15.195:0/3895469746 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa3e8153cb0 0x7fa3e80bdbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:36.606 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.580+0000 7fa3e77fe700 1 -- 172.21.15.195:0/3895469746 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa3e8153cb0 msgr2=0x7fa3e80bdbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:36.607 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.580+0000 7fa3e77fe700 1 --2- 172.21.15.195:0/3895469746 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa3e8153cb0 0x7fa3e80bdbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:36.607 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.580+0000 7fa3e77fe700 1 -- 172.21.15.195:0/3895469746 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa3e80c4160 msgr2=0x7fa3e80c8b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:36.608 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.580+0000 7fa3e77fe700 1 --2- 172.21.15.195:0/3895469746 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa3e80c4160 0x7fa3e80c8b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:36.609 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.580+0000 7fa3e77fe700 1 -- 172.21.15.195:0/3895469746 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3e0009ae0 con 0x7fa3e806f640 2022-06-27T12:05:36.609 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.580+0000 7fa3e77fe700 1 --2- 172.21.15.195:0/3895469746 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa3e806f640 0x7fa3e8167100 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fa3d800c4c0 tx=0x7fa3d8007e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:05:36.610 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.580+0000 7fa3df7fe700 1 -- 172.21.15.195:0/3895469746 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3d80050d0 con 0x7fa3e806f640 2022-06-27T12:05:36.610 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.580+0000 7fa3df7fe700 1 -- 172.21.15.195:0/3895469746 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3d8005230 con 0x7fa3e806f640 2022-06-27T12:05:36.611 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.580+0000 7fa3edb8f700 1 -- 172.21.15.195:0/3895469746 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3e80bf0d0 con 0x7fa3e806f640 2022-06-27T12:05:36.611 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.581+0000 7fa3df7fe700 1 -- 172.21.15.195:0/3895469746 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3d80117a0 con 0x7fa3e806f640 2022-06-27T12:05:36.612 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.581+0000 7fa3edb8f700 1 -- 172.21.15.195:0/3895469746 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3e80bf690 con 0x7fa3e806f640 2022-06-27T12:05:36.612 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.581+0000 7fa3edb8f700 1 -- 172.21.15.195:0/3895469746 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3e80bda10 con 0x7fa3e806f640 2022-06-27T12:05:36.613 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.582+0000 7fa3df7fe700 1 -- 172.21.15.195:0/3895469746 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fa3d8019030 con 0x7fa3e806f640 2022-06-27T12:05:36.613 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.582+0000 7fa3df7fe700 1 --2- 172.21.15.195:0/3895469746 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa3cc063d00 0x7fa3cc0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:36.613 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.582+0000 7fa3df7fe700 1 -- 172.21.15.195:0/3895469746 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6976+0+0 (secure 0 0 0) 0x7fa3d807fbe0 con 0x7fa3e806f640 2022-06-27T12:05:36.614 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.582+0000 7fa3e6ffd700 1 --2- 172.21.15.195:0/3895469746 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa3cc063d00 0x7fa3cc0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:36.614 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.583+0000 7fa3e6ffd700 1 --2- 172.21.15.195:0/3895469746 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa3cc063d00 0x7fa3cc0661b0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fa3e80c0a30 tx=0x7fa3d0007d70 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:05:36.615 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.586+0000 7fa3df7fe700 1 -- 172.21.15.195:0/3895469746 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fa3d8049c90 con 0x7fa3e806f640 2022-06-27T12:05:36.820 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.819+0000 7fa3edb8f700 1 -- 172.21.15.195:0/3895469746 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa3e8167640 con 0x7fa3e806f640 2022-06-27T12:05:36.820 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.820+0000 7fa3df7fe700 1 -- 172.21.15.195:0/3895469746 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+5374 (secure 0 0 0) 0x7fa3d8017020 con 0x7fa3e806f640 2022-06-27T12:05:36.821 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:05:36.821 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:05:31.228758+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5489},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5489":{"gid":5489,"name":"d","rank":0,"incarnation":61,"state":"up:clientreplay","state_seq":21,"addr":"172.21.15.202:6835/2342195382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2342195382},{"type":"v1","addr":"172.21.15.202:6835","nonce":2342195382}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:05:36.823 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.822+0000 7fa3edb8f700 1 -- 172.21.15.195:0/3895469746 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa3cc063d00 msgr2=0x7fa3cc0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:36.824 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.822+0000 7fa3edb8f700 1 --2- 172.21.15.195:0/3895469746 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa3cc063d00 0x7fa3cc0661b0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fa3e80c0a30 tx=0x7fa3d0007d70 comp rx=0 tx=0).stop 2022-06-27T12:05:36.825 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.822+0000 7fa3edb8f700 1 -- 172.21.15.195:0/3895469746 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa3e806f640 msgr2=0x7fa3e8167100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:36.825 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.822+0000 7fa3edb8f700 1 --2- 172.21.15.195:0/3895469746 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa3e806f640 0x7fa3e8167100 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fa3d800c4c0 tx=0x7fa3d8007e40 comp rx=0 tx=0).stop 2022-06-27T12:05:36.825 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.822+0000 7fa3edb8f700 1 -- 172.21.15.195:0/3895469746 shutdown_connections 2022-06-27T12:05:36.825 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.822+0000 7fa3edb8f700 1 --2- 172.21.15.195:0/3895469746 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa3cc063d00 0x7fa3cc0661b0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:36.826 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.822+0000 7fa3edb8f700 1 --2- 172.21.15.195:0/3895469746 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa3e806f640 0x7fa3e8167100 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:36.826 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.822+0000 7fa3edb8f700 1 --2- 172.21.15.195:0/3895469746 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa3e8153cb0 0x7fa3e80bdbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:36.826 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.822+0000 7fa3edb8f700 1 --2- 172.21.15.195:0/3895469746 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa3e80c4160 0x7fa3e80c8b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:36.827 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.822+0000 7fa3edb8f700 1 -- 172.21.15.195:0/3895469746 >> 172.21.15.195:0/3895469746 conn(0x7fa3e80c6350 msgr2=0x7fa3e8152650 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:05:36.827 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.822+0000 7fa3edb8f700 1 -- 172.21.15.195:0/3895469746 shutdown_connections 2022-06-27T12:05:36.827 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:36.822+0000 7fa3edb8f700 1 -- 172.21.15.195:0/3895469746 wait complete. 2022-06-27T12:05:36.827 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 64 2022-06-27T12:05:36.835 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:05:38.837 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:05:38.987 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:38.958+0000 7fb254454700 1 Processor -- start 2022-06-27T12:05:38.987 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:38.958+0000 7fb254454700 1 -- start start 2022-06-27T12:05:38.988 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:38.959+0000 7fb254454700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fb24c151560 0x7fb24c0c2120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:38.988 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:38.959+0000 7fb254454700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb24c154900 0x7fb24c0bf4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:38.989 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:38.959+0000 7fb254454700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb24c154da0 con 0x7fb24c151930 2022-06-27T12:05:38.990 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:38.959+0000 7fb254454700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb24c0c12a0 con 0x7fb24c151560 2022-06-27T12:05:38.992 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:38.959+0000 7fb254454700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb24c0c1580 con 0x7fb24c154900 2022-06-27T12:05:38.992 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:38.959+0000 7fb2521f0700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fb24c151560 0x7fb24c0c2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:38.993 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:38.959+0000 7fb2521f0700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fb24c151560 0x7fb24c0c2120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48048/0 (socket says 172.21.15.195:48048) 2022-06-27T12:05:38.993 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:38.959+0000 7fb2521f0700 1 -- 172.21.15.195:0/2525894440 learned_addr learned my addr 172.21.15.195:0/2525894440 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:05:38.994 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:38.959+0000 7fb2519ef700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fb24c151930 0x7fb24c1541f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:54236/0 (socket says 172.21.15.195:54236) 2022-06-27T12:05:38.994 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:38.960+0000 7fb2529f1700 1 --2- 172.21.15.195:0/2525894440 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb24c154900 0x7fb24c0bf4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:38.994 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:38.960+0000 7fb2521f0700 1 -- 172.21.15.195:0/2525894440 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb24c154900 msgr2=0x7fb24c0bf4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:05:38.995 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:38.960+0000 7fb2521f0700 1 --2- 172.21.15.195:0/2525894440 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb24c154900 0x7fb24c0bf4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:38.995 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:38.960+0000 7fb2521f0700 1 -- 172.21.15.195:0/2525894440 >> v1:172.21.15.195:6789/0 conn(0x7fb24c151930 legacy=0x7fb24c1541f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:38.996 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:38.960+0000 7fb2511ee700 1 -- 172.21.15.195:0/2525894440 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2256833365 0 0) 0x7fb24c154da0 con 0x7fb24c151930 2022-06-27T12:05:38.996 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:38.960+0000 7fb2521f0700 1 -- 172.21.15.195:0/2525894440 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb24c0c29f0 con 0x7fb24c151560 2022-06-27T12:05:38.997 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:38.960+0000 7fb2521f0700 1 --2- 172.21.15.195:0/2525894440 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb24c151560 0x7fb24c0c2120 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fb23c009860 tx=0x7fb23c009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0866dd25111389c server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:05:38.997 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:38.960+0000 7fb2511ee700 1 -- 172.21.15.195:0/2525894440 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb23c00b690 con 0x7fb24c151560 2022-06-27T12:05:38.998 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:38.960+0000 7fb2511ee700 1 -- 172.21.15.195:0/2525894440 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb23c00b7f0 con 0x7fb24c151560 2022-06-27T12:05:38.999 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:38.960+0000 7fb2511ee700 1 -- 172.21.15.195:0/2525894440 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb23c00bb90 con 0x7fb24c151560 2022-06-27T12:05:38.999 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:38.961+0000 7fb254454700 1 -- 172.21.15.195:0/2525894440 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb24c151560 msgr2=0x7fb24c0c2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:39.000 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:38.961+0000 7fb254454700 1 --2- 172.21.15.195:0/2525894440 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb24c151560 0x7fb24c0c2120 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fb23c009860 tx=0x7fb23c009120 comp rx=0 tx=0).stop 2022-06-27T12:05:39.000 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:38.961+0000 7fb254454700 1 -- 172.21.15.195:0/2525894440 shutdown_connections 2022-06-27T12:05:39.001 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:38.961+0000 7fb254454700 1 --2- 172.21.15.195:0/2525894440 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb24c154900 0x7fb24c0bf4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:39.001 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:38.961+0000 7fb254454700 1 --2- 172.21.15.195:0/2525894440 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb24c151560 0x7fb24c0c2120 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:39.001 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:38.961+0000 7fb254454700 1 -- 172.21.15.195:0/2525894440 >> 172.21.15.195:0/2525894440 conn(0x7fb24c0c6350 msgr2=0x7fb24c0c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:05:39.003 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:38.961+0000 7fb254454700 1 -- 172.21.15.195:0/2525894440 shutdown_connections 2022-06-27T12:05:39.003 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:38.961+0000 7fb254454700 1 -- 172.21.15.195:0/2525894440 wait complete. 2022-06-27T12:05:39.004 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:38.962+0000 7fb254454700 1 Processor -- start 2022-06-27T12:05:39.004 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:38.962+0000 7fb254454700 1 -- start start 2022-06-27T12:05:39.005 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:38.963+0000 7fb254454700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb24c151560 0x7fb24c166500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:39.005 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:38.963+0000 7fb254454700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb24c151930 0x7fb24c0c06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:39.006 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:38.963+0000 7fb254454700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb24c154900 0x7fb24c1575d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:39.006 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:38.963+0000 7fb254454700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb24c16a9a0 con 0x7fb24c154900 2022-06-27T12:05:39.006 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:38.963+0000 7fb254454700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb24c0ba560 con 0x7fb24c151930 2022-06-27T12:05:39.007 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:38.963+0000 7fb254454700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb24c169e20 con 0x7fb24c151560 2022-06-27T12:05:39.007 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:38.963+0000 7fb2529f1700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb24c154900 0x7fb24c1575d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:39.007 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:38.963+0000 7fb2521f0700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb24c151560 0x7fb24c166500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:39.007 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:38.963+0000 7fb2529f1700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb24c154900 0x7fb24c1575d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48058/0 (socket says 172.21.15.195:48058) 2022-06-27T12:05:39.008 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:38.963+0000 7fb2529f1700 1 -- 172.21.15.195:0/4032127699 learned_addr learned my addr 172.21.15.195:0/4032127699 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:05:39.008 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:38.963+0000 7fb2519ef700 1 --2- 172.21.15.195:0/4032127699 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb24c151930 0x7fb24c0c06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:39.008 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:38.963+0000 7fb2529f1700 1 -- 172.21.15.195:0/4032127699 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb24c151560 msgr2=0x7fb24c166500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:39.008 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:38.963+0000 7fb2529f1700 1 --2- 172.21.15.195:0/4032127699 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb24c151560 0x7fb24c166500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:39.009 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:38.963+0000 7fb2529f1700 1 -- 172.21.15.195:0/4032127699 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb24c151930 msgr2=0x7fb24c0c06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:39.009 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:38.963+0000 7fb2529f1700 1 --2- 172.21.15.195:0/4032127699 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb24c151930 0x7fb24c0c06d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:39.009 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:38.963+0000 7fb2529f1700 1 -- 172.21.15.195:0/4032127699 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb23c0093a0 con 0x7fb24c154900 2022-06-27T12:05:39.009 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:38.963+0000 7fb2529f1700 1 --2- 172.21.15.195:0/4032127699 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb24c154900 0x7fb24c1575d0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fb244009df0 tx=0x7fb24400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:05:39.010 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:38.963+0000 7fb23a7fc700 1 -- 172.21.15.195:0/4032127699 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb24400cad0 con 0x7fb24c154900 2022-06-27T12:05:39.010 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:38.963+0000 7fb23a7fc700 1 -- 172.21.15.195:0/4032127699 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb24400f040 con 0x7fb24c154900 2022-06-27T12:05:39.011 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:38.963+0000 7fb254454700 1 -- 172.21.15.195:0/4032127699 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb24c16a100 con 0x7fb24c154900 2022-06-27T12:05:39.011 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:38.963+0000 7fb23a7fc700 1 -- 172.21.15.195:0/4032127699 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb24400cea0 con 0x7fb24c154900 2022-06-27T12:05:39.012 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:38.963+0000 7fb254454700 1 -- 172.21.15.195:0/4032127699 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb24c16a6c0 con 0x7fb24c154900 2022-06-27T12:05:39.012 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:38.964+0000 7fb254454700 1 -- 172.21.15.195:0/4032127699 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb24c0722c0 con 0x7fb24c154900 2022-06-27T12:05:39.013 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:38.965+0000 7fb23a7fc700 1 -- 172.21.15.195:0/4032127699 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fb244014690 con 0x7fb24c154900 2022-06-27T12:05:39.013 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:38.965+0000 7fb23a7fc700 1 --2- 172.21.15.195:0/4032127699 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb22c063d00 0x7fb22c0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:39.013 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:38.965+0000 7fb23a7fc700 1 -- 172.21.15.195:0/4032127699 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6976+0+0 (secure 0 0 0) 0x7fb244081770 con 0x7fb24c154900 2022-06-27T12:05:39.013 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:38.965+0000 7fb2521f0700 1 --2- 172.21.15.195:0/4032127699 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb22c063d00 0x7fb22c0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:39.014 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:38.965+0000 7fb2521f0700 1 --2- 172.21.15.195:0/4032127699 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb22c063d00 0x7fb22c0661b0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fb23c009160 tx=0x7fb23c017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:05:39.014 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:38.969+0000 7fb23a7fc700 1 -- 172.21.15.195:0/4032127699 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fb24404ffd0 con 0x7fb24c154900 2022-06-27T12:05:39.206 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:39.193+0000 7fb254454700 1 -- 172.21.15.195:0/4032127699 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb24c0ba6a0 con 0x7fb24c154900 2022-06-27T12:05:39.207 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:39.194+0000 7fb23a7fc700 1 -- 172.21.15.195:0/4032127699 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+5374 (secure 0 0 0) 0x7fb2440535f0 con 0x7fb24c154900 2022-06-27T12:05:39.208 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:05:39.208 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:05:31.228758+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5489},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5489":{"gid":5489,"name":"d","rank":0,"incarnation":61,"state":"up:clientreplay","state_seq":21,"addr":"172.21.15.202:6835/2342195382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2342195382},{"type":"v1","addr":"172.21.15.202:6835","nonce":2342195382}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:05:39.212 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:39.196+0000 7fb254454700 1 -- 172.21.15.195:0/4032127699 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb22c063d00 msgr2=0x7fb22c0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:39.212 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:39.196+0000 7fb254454700 1 --2- 172.21.15.195:0/4032127699 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb22c063d00 0x7fb22c0661b0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fb23c009160 tx=0x7fb23c017000 comp rx=0 tx=0).stop 2022-06-27T12:05:39.213 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:39.196+0000 7fb254454700 1 -- 172.21.15.195:0/4032127699 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb24c154900 msgr2=0x7fb24c1575d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:39.213 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:39.196+0000 7fb254454700 1 --2- 172.21.15.195:0/4032127699 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb24c154900 0x7fb24c1575d0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fb244009df0 tx=0x7fb24400b040 comp rx=0 tx=0).stop 2022-06-27T12:05:39.215 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:39.196+0000 7fb254454700 1 -- 172.21.15.195:0/4032127699 shutdown_connections 2022-06-27T12:05:39.215 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:39.196+0000 7fb254454700 1 --2- 172.21.15.195:0/4032127699 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb22c063d00 0x7fb22c0661b0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:39.216 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:39.196+0000 7fb254454700 1 --2- 172.21.15.195:0/4032127699 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb24c154900 0x7fb24c1575d0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:39.216 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:39.196+0000 7fb254454700 1 --2- 172.21.15.195:0/4032127699 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb24c151560 0x7fb24c166500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:39.217 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:39.196+0000 7fb254454700 1 --2- 172.21.15.195:0/4032127699 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb24c151930 0x7fb24c0c06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:39.217 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:39.196+0000 7fb254454700 1 -- 172.21.15.195:0/4032127699 >> 172.21.15.195:0/4032127699 conn(0x7fb24c0c6350 msgr2=0x7fb24c0c4ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:05:39.218 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:39.196+0000 7fb254454700 1 -- 172.21.15.195:0/4032127699 shutdown_connections 2022-06-27T12:05:39.219 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:39.196+0000 7fb254454700 1 -- 172.21.15.195:0/4032127699 wait complete. 2022-06-27T12:05:39.219 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 64 2022-06-27T12:05:39.220 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:05:41.222 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:05:41.366 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.355+0000 7f56564f4700 1 Processor -- start 2022-06-27T12:05:41.367 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.356+0000 7f56564f4700 1 -- start start 2022-06-27T12:05:41.367 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.356+0000 7f56564f4700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f565006e850 0x7f56501570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:41.368 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.356+0000 7f56564f4700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f56500c34a0 0x7f56500c2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:41.368 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.356+0000 7f56564f4700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f565006fba0 con 0x7f565006fef0 2022-06-27T12:05:41.368 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.356+0000 7f56564f4700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56501575e0 con 0x7f565006e850 2022-06-27T12:05:41.369 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.356+0000 7f56564f4700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f565006f5e0 con 0x7f56500c34a0 2022-06-27T12:05:41.369 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.357+0000 7f564ffff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f565006e850 0x7f56501570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:41.369 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.357+0000 7f564ffff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f565006e850 0x7f56501570a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48074/0 (socket says 172.21.15.195:48074) 2022-06-27T12:05:41.369 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.357+0000 7f564ffff700 1 -- 172.21.15.195:0/1058507218 learned_addr learned my addr 172.21.15.195:0/1058507218 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:05:41.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.357+0000 7f564f7fe700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f565006fef0 0x7f5650069de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:54264/0 (socket says 172.21.15.195:54264) 2022-06-27T12:05:41.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.357+0000 7f5654a91700 1 --2- 172.21.15.195:0/1058507218 >> v2:172.21.15.202:3300/0 conn(0x7f56500c34a0 0x7f56500c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:41.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.357+0000 7f564ffff700 1 -- 172.21.15.195:0/1058507218 >> v2:172.21.15.202:3300/0 conn(0x7f56500c34a0 msgr2=0x7f56500c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:05:41.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.357+0000 7f564ffff700 1 --2- 172.21.15.195:0/1058507218 >> v2:172.21.15.202:3300/0 conn(0x7f56500c34a0 0x7f56500c2920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:41.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.357+0000 7f564ffff700 1 -- 172.21.15.195:0/1058507218 >> v1:172.21.15.195:6789/0 conn(0x7f565006fef0 legacy=0x7f5650069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:41.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.357+0000 7f564ffff700 1 -- 172.21.15.195:0/1058507218 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f565006f8c0 con 0x7f565006e850 2022-06-27T12:05:41.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.357+0000 7f564effd700 1 -- 172.21.15.195:0/1058507218 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4231750408 0 0) 0x7f565006fba0 con 0x7f565006fef0 2022-06-27T12:05:41.372 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.357+0000 7f564ffff700 1 --2- 172.21.15.195:0/1058507218 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f565006e850 0x7f56501570a0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f56500c1420 tx=0x7f564000b3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba7857d37bbd6a19 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:05:41.372 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.357+0000 7f564effd700 1 -- 172.21.15.195:0/1058507218 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f564000dbd0 con 0x7f565006e850 2022-06-27T12:05:41.372 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.357+0000 7f564effd700 1 -- 172.21.15.195:0/1058507218 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f564000dd30 con 0x7f565006e850 2022-06-27T12:05:41.372 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.357+0000 7f564effd700 1 -- 172.21.15.195:0/1058507218 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f564000dea0 con 0x7f565006e850 2022-06-27T12:05:41.373 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.358+0000 7f56564f4700 1 -- 172.21.15.195:0/1058507218 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f565006e850 msgr2=0x7f56501570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:41.373 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.358+0000 7f56564f4700 1 --2- 172.21.15.195:0/1058507218 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f565006e850 0x7f56501570a0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f56500c1420 tx=0x7f564000b3b0 comp rx=0 tx=0).stop 2022-06-27T12:05:41.373 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.358+0000 7f56564f4700 1 -- 172.21.15.195:0/1058507218 shutdown_connections 2022-06-27T12:05:41.373 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.358+0000 7f56564f4700 1 --2- 172.21.15.195:0/1058507218 >> v2:172.21.15.202:3300/0 conn(0x7f56500c34a0 0x7f56500c2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:41.374 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.358+0000 7f56564f4700 1 --2- 172.21.15.195:0/1058507218 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f565006e850 0x7f56501570a0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:41.374 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.358+0000 7f56564f4700 1 -- 172.21.15.195:0/1058507218 >> 172.21.15.195:0/1058507218 conn(0x7f56500c6350 msgr2=0x7f56500c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:05:41.374 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.358+0000 7f56564f4700 1 -- 172.21.15.195:0/1058507218 shutdown_connections 2022-06-27T12:05:41.374 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.358+0000 7f56564f4700 1 -- 172.21.15.195:0/1058507218 wait complete. 2022-06-27T12:05:41.374 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.359+0000 7f56564f4700 1 Processor -- start 2022-06-27T12:05:41.375 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.359+0000 7f56564f4700 1 -- start start 2022-06-27T12:05:41.375 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.360+0000 7f56564f4700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f565006e850 0x7f5650152e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:41.375 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.360+0000 7f56564f4700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f56500c34a0 0x7f565014e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:41.375 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.360+0000 7f56564f4700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5650152670 0x7f5650069ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:41.376 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.360+0000 7f56564f4700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56500b87f0 con 0x7f5650152670 2022-06-27T12:05:41.376 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.360+0000 7f56564f4700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5650152b40 con 0x7f565006e850 2022-06-27T12:05:41.376 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.360+0000 7f56564f4700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5650162640 con 0x7f56500c34a0 2022-06-27T12:05:41.376 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.360+0000 7f5654a91700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5650152670 0x7f5650069ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:41.376 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.360+0000 7f5654a91700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5650152670 0x7f5650069ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48084/0 (socket says 172.21.15.195:48084) 2022-06-27T12:05:41.377 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.360+0000 7f5654a91700 1 -- 172.21.15.195:0/2682564180 learned_addr learned my addr 172.21.15.195:0/2682564180 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:05:41.377 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.360+0000 7f564ffff700 1 --2- 172.21.15.195:0/2682564180 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f565006e850 0x7f5650152e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:41.377 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.360+0000 7f564f7fe700 1 --2- 172.21.15.195:0/2682564180 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f56500c34a0 0x7f565014e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:41.377 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.360+0000 7f5654a91700 1 -- 172.21.15.195:0/2682564180 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f56500c34a0 msgr2=0x7f565014e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:41.379 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.360+0000 7f5654a91700 1 --2- 172.21.15.195:0/2682564180 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f56500c34a0 0x7f565014e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:41.379 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.360+0000 7f5654a91700 1 -- 172.21.15.195:0/2682564180 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f565006e850 msgr2=0x7f5650152e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:41.379 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.360+0000 7f5654a91700 1 --2- 172.21.15.195:0/2682564180 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f565006e850 0x7f5650152e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:41.380 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.360+0000 7f5654a91700 1 -- 172.21.15.195:0/2682564180 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f564000b650 con 0x7f5650152670 2022-06-27T12:05:41.380 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.360+0000 7f564ffff700 1 --2- 172.21.15.195:0/2682564180 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f565006e850 0x7f5650152e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T12:05:41.380 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.360+0000 7f5654a91700 1 --2- 172.21.15.195:0/2682564180 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5650152670 0x7f5650069ce0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f564800ceb0 tx=0x7f5648007870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:05:41.381 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.360+0000 7f5647fff700 1 -- 172.21.15.195:0/2682564180 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f564800ae40 con 0x7f5650152670 2022-06-27T12:05:41.381 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.360+0000 7f5647fff700 1 -- 172.21.15.195:0/2682564180 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56480038a0 con 0x7f5650152670 2022-06-27T12:05:41.381 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.360+0000 7f56564f4700 1 -- 172.21.15.195:0/2682564180 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5650162920 con 0x7f5650152670 2022-06-27T12:05:41.381 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.361+0000 7f5647fff700 1 -- 172.21.15.195:0/2682564180 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5648003c40 con 0x7f5650152670 2022-06-27T12:05:41.382 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.361+0000 7f56564f4700 1 -- 172.21.15.195:0/2682564180 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5650162330 con 0x7f5650152670 2022-06-27T12:05:41.383 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.361+0000 7f56564f4700 1 -- 172.21.15.195:0/2682564180 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5650163120 con 0x7f5650152670 2022-06-27T12:05:41.383 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.362+0000 7f5647fff700 1 -- 172.21.15.195:0/2682564180 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f5648004000 con 0x7f5650152670 2022-06-27T12:05:41.383 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.362+0000 7f5647fff700 1 --2- 172.21.15.195:0/2682564180 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5634063d00 0x7f56340661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:41.383 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.362+0000 7f5647fff700 1 -- 172.21.15.195:0/2682564180 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6976+0+0 (secure 0 0 0) 0x7f5648012070 con 0x7f5650152670 2022-06-27T12:05:41.384 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.362+0000 7f564ffff700 1 --2- 172.21.15.195:0/2682564180 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5634063d00 0x7f56340661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:41.384 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.363+0000 7f564ffff700 1 --2- 172.21.15.195:0/2682564180 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5634063d00 0x7f56340661b0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f564000b3f0 tx=0x7f564001b000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:05:41.384 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.367+0000 7f5647fff700 1 -- 172.21.15.195:0/2682564180 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f5650163120 con 0x7f5650152670 2022-06-27T12:05:41.595 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.595+0000 7f56564f4700 1 -- 172.21.15.195:0/2682564180 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f565006a640 con 0x7f5650152670 2022-06-27T12:05:41.596 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.595+0000 7f5647fff700 1 -- 172.21.15.195:0/2682564180 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+5368 (secure 0 0 0) 0x7f56480504b0 con 0x7f5650152670 2022-06-27T12:05:41.596 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:05:41.596 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:05:40.259006+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5489},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5489":{"gid":5489,"name":"d","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.202:6835/2342195382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2342195382},{"type":"v1","addr":"172.21.15.202:6835","nonce":2342195382}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:05:41.598 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.597+0000 7f56564f4700 1 -- 172.21.15.195:0/2682564180 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5634063d00 msgr2=0x7f56340661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:41.598 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.597+0000 7f56564f4700 1 --2- 172.21.15.195:0/2682564180 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5634063d00 0x7f56340661b0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f564000b3f0 tx=0x7f564001b000 comp rx=0 tx=0).stop 2022-06-27T12:05:41.598 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.597+0000 7f56564f4700 1 -- 172.21.15.195:0/2682564180 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5650152670 msgr2=0x7f5650069ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:41.599 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.597+0000 7f56564f4700 1 --2- 172.21.15.195:0/2682564180 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5650152670 0x7f5650069ce0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f564800ceb0 tx=0x7f5648007870 comp rx=0 tx=0).stop 2022-06-27T12:05:41.599 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.597+0000 7f56564f4700 1 -- 172.21.15.195:0/2682564180 shutdown_connections 2022-06-27T12:05:41.599 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.597+0000 7f56564f4700 1 --2- 172.21.15.195:0/2682564180 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5634063d00 0x7f56340661b0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:41.599 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.597+0000 7f56564f4700 1 --2- 172.21.15.195:0/2682564180 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5650152670 0x7f5650069ce0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:41.599 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.597+0000 7f56564f4700 1 --2- 172.21.15.195:0/2682564180 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f565006e850 0x7f5650152e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:41.600 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.597+0000 7f56564f4700 1 --2- 172.21.15.195:0/2682564180 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f56500c34a0 0x7f565014e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:41.600 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.597+0000 7f56564f4700 1 -- 172.21.15.195:0/2682564180 >> 172.21.15.195:0/2682564180 conn(0x7f56500c6350 msgr2=0x7f56500c8990 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:05:41.600 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.597+0000 7f56564f4700 1 -- 172.21.15.195:0/2682564180 shutdown_connections 2022-06-27T12:05:41.600 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:41.597+0000 7f56564f4700 1 -- 172.21.15.195:0/2682564180 wait complete. 2022-06-27T12:05:41.601 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 65 2022-06-27T12:05:41.609 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d has gained rank=0, replacing gid=5818 2022-06-27T12:05:41.610 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 6 secs before reviving mds.b 2022-06-27T12:05:46.414 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 93% (100/107) Updating files: 94% (101/107) Updating files: 95% (102/107) Updating files: 96% (103/107) Updating files: 97% (104/107) Updating files: 98% (105/107) Updating files: 99% (106/107) Updating files: 100% (107/107) Updating files: 100% (107/107), done. 2022-06-27T12:05:46.524 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at 38dbaa5175c v0.16 2022-06-27T12:05:46.526 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.16 2022-06-27T12:05:46.554 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:05:46.555 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 17 -eq 48 ']' 2022-06-27T12:05:46.555 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.17 2022-06-27T12:05:46.555 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.17 2022-06-27T12:05:46.556 INFO:tasks.workunit.client.0.smithi195.stdout:v0.17 2022-06-27T12:05:46.558 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.17 2022-06-27T12:05:47.610 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2022-06-27T12:05:47.611 INFO:tasks.ceph.mds.b:Restarting daemon 2022-06-27T12:05:47.612 DEBUG:teuthology.orchestra.run.smithi202:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-06-27T12:05:47.615 INFO:tasks.ceph.mds.b:Started 2022-06-27T12:05:47.616 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:05:47.694 INFO:tasks.ceph.mds.b.smithi202.stdout:starting mds.b at 2022-06-27T12:05:47.752 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.739+0000 7f00870b6700 1 Processor -- start 2022-06-27T12:05:47.753 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.739+0000 7f00870b6700 1 -- start start 2022-06-27T12:05:47.753 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.739+0000 7f00870b6700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f00800c40d0 0x7f0080157010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:47.754 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.739+0000 7f00870b6700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f00800c6190 0x7f0080067ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:47.754 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.739+0000 7f00870b6700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f00800c44a0 0x7f008014cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:47.755 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.739+0000 7f00870b6700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0080156b80 con 0x7f00800c6190 2022-06-27T12:05:47.755 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.739+0000 7f00870b6700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f008006a040 con 0x7f00800c44a0 2022-06-27T12:05:47.756 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.739+0000 7f00870b6700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f008006a300 con 0x7f00800c40d0 2022-06-27T12:05:47.757 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.740+0000 7f007ffff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f00800c6190 0x7f0080067ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:47.757 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.740+0000 7f007ffff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f00800c6190 0x7f0080067ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48102/0 (socket says 172.21.15.195:48102) 2022-06-27T12:05:47.758 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.740+0000 7f007ffff700 1 -- 172.21.15.195:0/2285437438 learned_addr learned my addr 172.21.15.195:0/2285437438 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:05:47.758 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.740+0000 7f0085653700 1 --2- 172.21.15.195:0/2285437438 >> v2:172.21.15.202:3300/0 conn(0x7f00800c44a0 0x7f008014cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:47.759 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.740+0000 7f0084e52700 1 --2- 172.21.15.195:0/2285437438 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f00800c40d0 0x7f0080157010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:47.759 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.740+0000 7f007ffff700 1 -- 172.21.15.195:0/2285437438 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f00800c40d0 msgr2=0x7f0080157010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:05:47.760 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.740+0000 7f007ffff700 1 --2- 172.21.15.195:0/2285437438 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f00800c40d0 0x7f0080157010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:47.761 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.740+0000 7f007ffff700 1 -- 172.21.15.195:0/2285437438 >> v2:172.21.15.202:3300/0 conn(0x7f00800c44a0 msgr2=0x7f008014cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:05:47.761 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.740+0000 7f007ffff700 1 --2- 172.21.15.195:0/2285437438 >> v2:172.21.15.202:3300/0 conn(0x7f00800c44a0 0x7f008014cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:47.761 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.740+0000 7f007ffff700 1 -- 172.21.15.195:0/2285437438 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f008006a5e0 con 0x7f00800c6190 2022-06-27T12:05:47.762 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.740+0000 7f007ffff700 1 --2- 172.21.15.195:0/2285437438 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f00800c6190 0x7f0080067ed0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f00800b2b60 tx=0x7f006c009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e8227658bca0d70 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:05:47.762 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.740+0000 7f007f7fe700 1 -- 172.21.15.195:0/2285437438 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f006c011630 con 0x7f00800c6190 2022-06-27T12:05:47.763 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.740+0000 7f007f7fe700 1 -- 172.21.15.195:0/2285437438 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f006c011790 con 0x7f00800c6190 2022-06-27T12:05:47.763 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.740+0000 7f007f7fe700 1 -- 172.21.15.195:0/2285437438 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f006c011b30 con 0x7f00800c6190 2022-06-27T12:05:47.764 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.741+0000 7f00870b6700 1 -- 172.21.15.195:0/2285437438 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f00800c6190 msgr2=0x7f0080067ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:47.764 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.741+0000 7f00870b6700 1 --2- 172.21.15.195:0/2285437438 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f00800c6190 0x7f0080067ed0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f00800b2b60 tx=0x7f006c009120 comp rx=0 tx=0).stop 2022-06-27T12:05:47.765 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.741+0000 7f00870b6700 1 -- 172.21.15.195:0/2285437438 shutdown_connections 2022-06-27T12:05:47.765 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.741+0000 7f00870b6700 1 --2- 172.21.15.195:0/2285437438 >> v2:172.21.15.202:3300/0 conn(0x7f00800c44a0 0x7f008014cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:47.765 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.741+0000 7f00870b6700 1 --2- 172.21.15.195:0/2285437438 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f00800c40d0 0x7f0080157010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:47.766 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.741+0000 7f00870b6700 1 --2- 172.21.15.195:0/2285437438 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f00800c6190 0x7f0080067ed0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:47.766 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.741+0000 7f00870b6700 1 -- 172.21.15.195:0/2285437438 >> 172.21.15.195:0/2285437438 conn(0x7f00800ccb80 msgr2=0x7f00800ccf80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:05:47.767 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.741+0000 7f00870b6700 1 -- 172.21.15.195:0/2285437438 shutdown_connections 2022-06-27T12:05:47.768 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.741+0000 7f00870b6700 1 -- 172.21.15.195:0/2285437438 wait complete. 2022-06-27T12:05:47.768 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.742+0000 7f00870b6700 1 Processor -- start 2022-06-27T12:05:47.768 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.742+0000 7f00870b6700 1 -- start start 2022-06-27T12:05:47.769 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.743+0000 7f00870b6700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f00800c40d0 0x7f00801529e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:47.769 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.743+0000 7f00870b6700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f00800c44a0 0x7f008014fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:47.771 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.743+0000 7f00870b6700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f00800c6190 0x7f00801549f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:47.771 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.743+0000 7f00870b6700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0080153f30 con 0x7f00800c44a0 2022-06-27T12:05:47.772 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.743+0000 7f00870b6700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0080147e10 con 0x7f00800c40d0 2022-06-27T12:05:47.772 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.743+0000 7f00870b6700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0080150e80 con 0x7f00800c6190 2022-06-27T12:05:47.773 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.743+0000 7f007ffff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f00800c44a0 0x7f008014fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:47.773 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.743+0000 7f007ffff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f00800c44a0 0x7f008014fe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48108/0 (socket says 172.21.15.195:48108) 2022-06-27T12:05:47.774 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.743+0000 7f007ffff700 1 -- 172.21.15.195:0/2925949570 learned_addr learned my addr 172.21.15.195:0/2925949570 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:05:47.774 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.743+0000 7f0085653700 1 --2- 172.21.15.195:0/2925949570 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f00800c6190 0x7f00801549f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:47.774 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.743+0000 7f0084e52700 1 --2- 172.21.15.195:0/2925949570 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f00800c40d0 0x7f00801529e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:47.775 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.743+0000 7f0085653700 1 -- 172.21.15.195:0/2925949570 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f00800c40d0 msgr2=0x7f00801529e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:47.776 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.743+0000 7f0085653700 1 --2- 172.21.15.195:0/2925949570 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f00800c40d0 0x7f00801529e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:47.776 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.743+0000 7f0085653700 1 -- 172.21.15.195:0/2925949570 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f00800c44a0 msgr2=0x7f008014fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:47.776 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.743+0000 7f0085653700 1 --2- 172.21.15.195:0/2925949570 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f00800c44a0 0x7f008014fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:47.777 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.743+0000 7f0085653700 1 -- 172.21.15.195:0/2925949570 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f006c0093a0 con 0x7f00800c6190 2022-06-27T12:05:47.778 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.743+0000 7f0085653700 1 --2- 172.21.15.195:0/2925949570 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f00800c6190 0x7f00801549f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f0078009e80 tx=0x7f007800b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:05:47.779 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.743+0000 7f007cff9700 1 -- 172.21.15.195:0/2925949570 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f007800cb30 con 0x7f00800c6190 2022-06-27T12:05:47.779 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.744+0000 7f007cff9700 1 -- 172.21.15.195:0/2925949570 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f007800f040 con 0x7f00800c6190 2022-06-27T12:05:47.779 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.744+0000 7f00870b6700 1 -- 172.21.15.195:0/2925949570 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0080151120 con 0x7f00800c6190 2022-06-27T12:05:47.780 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.744+0000 7f007cff9700 1 -- 172.21.15.195:0/2925949570 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0078014480 con 0x7f00800c6190 2022-06-27T12:05:47.782 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.744+0000 7f00870b6700 1 -- 172.21.15.195:0/2925949570 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0080147aa0 con 0x7f00800c6190 2022-06-27T12:05:47.783 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.744+0000 7f00870b6700 1 -- 172.21.15.195:0/2925949570 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0080152f80 con 0x7f00800c6190 2022-06-27T12:05:47.783 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.745+0000 7f007cff9700 1 -- 172.21.15.195:0/2925949570 <== mon.2 v2:172.21.15.202:3301/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f00780147f0 con 0x7f00800c6190 2022-06-27T12:05:47.783 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.745+0000 7f007cff9700 1 --2- 172.21.15.195:0/2925949570 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0068063d00 0x7f00680661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:47.784 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.745+0000 7f007cff9700 1 -- 172.21.15.195:0/2925949570 <== mon.2 v2:172.21.15.202:3301/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6976+0+0 (secure 0 0 0) 0x7f0078081b00 con 0x7f00800c6190 2022-06-27T12:05:47.784 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.745+0000 7f0084e52700 1 --2- 172.21.15.195:0/2925949570 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0068063d00 0x7f00680661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:47.784 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.745+0000 7f0084e52700 1 --2- 172.21.15.195:0/2925949570 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0068063d00 0x7f00680661b0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f0074009be0 tx=0x7f0074009060 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:05:47.785 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.750+0000 7f007cff9700 1 -- 172.21.15.195:0/2925949570 <== mon.2 v2:172.21.15.202:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f0078050360 con 0x7f00800c6190 2022-06-27T12:05:47.980 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.980+0000 7f00870b6700 1 -- 172.21.15.195:0/2925949570 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f00800722c0 con 0x7f00800c6190 2022-06-27T12:05:47.981 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.981+0000 7f007cff9700 1 -- 172.21.15.195:0/2925949570 <== mon.2 v2:172.21.15.202:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+5368 (secure 0 0 0) 0x7f0078018020 con 0x7f00800c6190 2022-06-27T12:05:47.985 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:05:47.985 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:05:40.259006+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5489},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5489":{"gid":5489,"name":"d","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.202:6835/2342195382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2342195382},{"type":"v1","addr":"172.21.15.202:6835","nonce":2342195382}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:05:47.987 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.983+0000 7f00870b6700 1 -- 172.21.15.195:0/2925949570 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0068063d00 msgr2=0x7f00680661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:47.987 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.983+0000 7f00870b6700 1 --2- 172.21.15.195:0/2925949570 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0068063d00 0x7f00680661b0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f0074009be0 tx=0x7f0074009060 comp rx=0 tx=0).stop 2022-06-27T12:05:47.988 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.983+0000 7f00870b6700 1 -- 172.21.15.195:0/2925949570 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f00800c6190 msgr2=0x7f00801549f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:47.988 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.983+0000 7f00870b6700 1 --2- 172.21.15.195:0/2925949570 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f00800c6190 0x7f00801549f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f0078009e80 tx=0x7f007800b040 comp rx=0 tx=0).stop 2022-06-27T12:05:47.989 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.983+0000 7f00870b6700 1 -- 172.21.15.195:0/2925949570 shutdown_connections 2022-06-27T12:05:47.989 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.983+0000 7f00870b6700 1 --2- 172.21.15.195:0/2925949570 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0068063d00 0x7f00680661b0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:47.989 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.983+0000 7f00870b6700 1 --2- 172.21.15.195:0/2925949570 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f00800c6190 0x7f00801549f0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:47.990 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.983+0000 7f00870b6700 1 --2- 172.21.15.195:0/2925949570 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f00800c40d0 0x7f00801529e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:47.992 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.983+0000 7f00870b6700 1 --2- 172.21.15.195:0/2925949570 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f00800c44a0 0x7f008014fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:47.993 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.983+0000 7f00870b6700 1 -- 172.21.15.195:0/2925949570 >> 172.21.15.195:0/2925949570 conn(0x7f00800ccb80 msgr2=0x7f00800cb410 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:05:47.993 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.983+0000 7f00870b6700 1 -- 172.21.15.195:0/2925949570 shutdown_connections 2022-06-27T12:05:47.994 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:47.983+0000 7f00870b6700 1 -- 172.21.15.195:0/2925949570 wait complete. 2022-06-27T12:05:47.994 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 65 2022-06-27T12:05:47.997 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2022-06-27T12:05:48.317 INFO:tasks.ceph.mds.d.smithi202.stderr:2022-06-27T12:05:48.316+0000 7fbcfd0f4700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2022-06-27T12:05:49.998 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:05:50.147 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.129+0000 7f6e0e2b6700 1 Processor -- start 2022-06-27T12:05:50.148 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.129+0000 7f6e0e2b6700 1 -- start start 2022-06-27T12:05:50.149 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.130+0000 7f6e0e2b6700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f6e080c3740 0x7f6e08153560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:50.149 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.130+0000 7f6e0e2b6700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f6e0806d790 0x7f6e080c3b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:50.150 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.130+0000 7f6e0e2b6700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e0806a3b0 con 0x7f6e0806dbd0 2022-06-27T12:05:50.150 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.130+0000 7f6e0e2b6700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e080c4eb0 con 0x7f6e080c3740 2022-06-27T12:05:50.150 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.130+0000 7f6e0e2b6700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e080c5150 con 0x7f6e0806d790 2022-06-27T12:05:50.151 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.130+0000 7f6e07fff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f6e080c3740 0x7f6e08153560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:50.151 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.130+0000 7f6e07fff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f6e080c3740 0x7f6e08153560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48118/0 (socket says 172.21.15.195:48118) 2022-06-27T12:05:50.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.130+0000 7f6e07fff700 1 -- 172.21.15.195:0/3738060421 learned_addr learned my addr 172.21.15.195:0/3738060421 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:05:50.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.130+0000 7f6e077fe700 1 --2- 172.21.15.195:0/3738060421 >> v2:172.21.15.202:3300/0 conn(0x7f6e0806d790 0x7f6e080c3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:50.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.130+0000 7f6e07fff700 1 -- 172.21.15.195:0/3738060421 >> v2:172.21.15.202:3300/0 conn(0x7f6e0806d790 msgr2=0x7f6e080c3b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:05:50.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.130+0000 7f6e07fff700 1 --2- 172.21.15.195:0/3738060421 >> v2:172.21.15.202:3300/0 conn(0x7f6e0806d790 0x7f6e080c3b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:50.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.130+0000 7f6e07fff700 1 -- 172.21.15.195:0/3738060421 >> v1:172.21.15.202:6789/0 conn(0x7f6e0806dbd0 legacy=0x7f6e08156030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:50.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.130+0000 7f6e07fff700 1 -- 172.21.15.195:0/3738060421 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e080c53f0 con 0x7f6e080c3740 2022-06-27T12:05:50.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.130+0000 7f6e07fff700 1 --2- 172.21.15.195:0/3738060421 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6e080c3740 0x7f6e08153560 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f6e080bfd10 tx=0x7f6df8009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30332c759e8baf34 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:05:50.155 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.130+0000 7f6e06ffd700 1 -- 172.21.15.195:0/3738060421 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df8011630 con 0x7f6e080c3740 2022-06-27T12:05:50.160 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.130+0000 7f6e06ffd700 1 -- 172.21.15.195:0/3738060421 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6df8011790 con 0x7f6e080c3740 2022-06-27T12:05:50.161 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.130+0000 7f6e06ffd700 1 -- 172.21.15.195:0/3738060421 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df8011b30 con 0x7f6e080c3740 2022-06-27T12:05:50.161 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.131+0000 7f6e0e2b6700 1 -- 172.21.15.195:0/3738060421 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6e080c3740 msgr2=0x7f6e08153560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:50.162 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.131+0000 7f6e0e2b6700 1 --2- 172.21.15.195:0/3738060421 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6e080c3740 0x7f6e08153560 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f6e080bfd10 tx=0x7f6df8009120 comp rx=0 tx=0).stop 2022-06-27T12:05:50.162 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.131+0000 7f6e0e2b6700 1 -- 172.21.15.195:0/3738060421 shutdown_connections 2022-06-27T12:05:50.163 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.131+0000 7f6e0e2b6700 1 --2- 172.21.15.195:0/3738060421 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6e080c3740 0x7f6e08153560 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:50.163 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.131+0000 7f6e0e2b6700 1 --2- 172.21.15.195:0/3738060421 >> v2:172.21.15.202:3300/0 conn(0x7f6e0806d790 0x7f6e080c3b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:50.163 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.131+0000 7f6e0e2b6700 1 -- 172.21.15.195:0/3738060421 >> 172.21.15.195:0/3738060421 conn(0x7f6e08067ed0 msgr2=0x7f6e080682d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:05:50.164 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.131+0000 7f6e0e2b6700 1 -- 172.21.15.195:0/3738060421 shutdown_connections 2022-06-27T12:05:50.164 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.131+0000 7f6e0e2b6700 1 -- 172.21.15.195:0/3738060421 wait complete. 2022-06-27T12:05:50.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.132+0000 7f6e0e2b6700 1 Processor -- start 2022-06-27T12:05:50.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.132+0000 7f6e0e2b6700 1 -- start start 2022-06-27T12:05:50.166 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.133+0000 7f6e0e2b6700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6e0806d790 0x7f6e080c0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:50.166 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.133+0000 7f6e0e2b6700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6e0806dbd0 0x7f6e080c09d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:50.167 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.133+0000 7f6e0e2b6700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6e080c3740 0x7f6e08141850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:50.167 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.133+0000 7f6e0e2b6700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e080c1960 con 0x7f6e080c3740 2022-06-27T12:05:50.167 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.133+0000 7f6e0e2b6700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e080bfd60 con 0x7f6e0806d790 2022-06-27T12:05:50.168 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.133+0000 7f6e0e2b6700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e080c0020 con 0x7f6e0806dbd0 2022-06-27T12:05:50.168 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.133+0000 7f6e0c853700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6e080c3740 0x7f6e08141850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:50.169 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.133+0000 7f6e07fff700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6e0806d790 0x7f6e080c0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:50.169 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.133+0000 7f6e0c853700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6e080c3740 0x7f6e08141850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48128/0 (socket says 172.21.15.195:48128) 2022-06-27T12:05:50.170 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.133+0000 7f6e07fff700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6e0806d790 0x7f6e080c0f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3300/0 says I am v2:172.21.15.195:46250/0 (socket says 172.21.15.195:46250) 2022-06-27T12:05:50.170 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.133+0000 7f6e077fe700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6e0806dbd0 0x7f6e080c09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:50.172 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.133+0000 7f6e0c853700 1 -- 172.21.15.195:0/988114609 learned_addr learned my addr 172.21.15.195:0/988114609 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:05:50.173 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.133+0000 7f6e0c853700 1 -- 172.21.15.195:0/988114609 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6e0806dbd0 msgr2=0x7f6e080c09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:50.174 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.133+0000 7f6e0c853700 1 --2- 172.21.15.195:0/988114609 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6e0806dbd0 0x7f6e080c09d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:50.174 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.133+0000 7f6e0c853700 1 -- 172.21.15.195:0/988114609 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6e0806d790 msgr2=0x7f6e080c0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:50.174 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.133+0000 7f6e0c853700 1 --2- 172.21.15.195:0/988114609 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6e0806d790 0x7f6e080c0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:50.175 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.133+0000 7f6e0c853700 1 -- 172.21.15.195:0/988114609 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6df80093a0 con 0x7f6e080c3740 2022-06-27T12:05:50.176 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.134+0000 7f6e0c853700 1 --2- 172.21.15.195:0/988114609 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6e080c3740 0x7f6e08141850 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f6e0000c7c0 tx=0x7f6e000015e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:05:50.176 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.134+0000 7f6deffff700 1 -- 172.21.15.195:0/988114609 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e00003880 con 0x7f6e080c3740 2022-06-27T12:05:50.177 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.134+0000 7f6deffff700 1 -- 172.21.15.195:0/988114609 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e000039e0 con 0x7f6e080c3740 2022-06-27T12:05:50.177 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.134+0000 7f6e0e2b6700 1 -- 172.21.15.195:0/988114609 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e08141d90 con 0x7f6e080c3740 2022-06-27T12:05:50.177 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.134+0000 7f6deffff700 1 -- 172.21.15.195:0/988114609 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e00003d60 con 0x7f6e080c3740 2022-06-27T12:05:50.178 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.134+0000 7f6e0e2b6700 1 -- 172.21.15.195:0/988114609 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e08156a30 con 0x7f6e080c3740 2022-06-27T12:05:50.178 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.135+0000 7f6e0e2b6700 1 -- 172.21.15.195:0/988114609 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e080c1ad0 con 0x7f6e080c3740 2022-06-27T12:05:50.179 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.136+0000 7f6deffff700 1 -- 172.21.15.195:0/988114609 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f6e00010050 con 0x7f6e080c3740 2022-06-27T12:05:50.179 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.136+0000 7f6deffff700 1 --2- 172.21.15.195:0/988114609 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6de8063d30 0x7f6de80661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:05:50.179 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.136+0000 7f6e07fff700 1 --2- 172.21.15.195:0/988114609 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6de8063d30 0x7f6de80661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:05:50.180 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.136+0000 7f6deffff700 1 -- 172.21.15.195:0/988114609 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6976+0+0 (secure 0 0 0) 0x7f6e000805a0 con 0x7f6e080c3740 2022-06-27T12:05:50.180 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.137+0000 7f6e07fff700 1 --2- 172.21.15.195:0/988114609 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6de8063d30 0x7f6de80661e0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f6df8009160 tx=0x7f6df8017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:05:50.181 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.141+0000 7f6deffff700 1 -- 172.21.15.195:0/988114609 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f6e080c1ad0 con 0x7f6e080c3740 2022-06-27T12:05:50.376 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.371+0000 7f6e0e2b6700 1 -- 172.21.15.195:0/988114609 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6e080c1aa0 con 0x7f6e080c3740 2022-06-27T12:05:50.376 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.372+0000 7f6deffff700 1 -- 172.21.15.195:0/988114609 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+6128 (secure 0 0 0) 0x7f6e0004ee00 con 0x7f6e080c3740 2022-06-27T12:05:50.376 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:05:50.377 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":5591,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/3022591968","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3022591968},{"type":"v1","addr":"172.21.15.202:6833","nonce":3022591968}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:05:40.259006+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5489},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5489":{"gid":5489,"name":"d","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.202:6835/2342195382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2342195382},{"type":"v1","addr":"172.21.15.202:6835","nonce":2342195382}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:05:50.382 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.374+0000 7f6e0e2b6700 1 -- 172.21.15.195:0/988114609 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6de8063d30 msgr2=0x7f6de80661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:50.382 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.374+0000 7f6e0e2b6700 1 --2- 172.21.15.195:0/988114609 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6de8063d30 0x7f6de80661e0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f6df8009160 tx=0x7f6df8017000 comp rx=0 tx=0).stop 2022-06-27T12:05:50.382 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.374+0000 7f6e0e2b6700 1 -- 172.21.15.195:0/988114609 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6e080c3740 msgr2=0x7f6e08141850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:05:50.383 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.374+0000 7f6e0e2b6700 1 --2- 172.21.15.195:0/988114609 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6e080c3740 0x7f6e08141850 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f6e0000c7c0 tx=0x7f6e000015e0 comp rx=0 tx=0).stop 2022-06-27T12:05:50.383 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.374+0000 7f6e0e2b6700 1 -- 172.21.15.195:0/988114609 shutdown_connections 2022-06-27T12:05:50.383 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.374+0000 7f6e0e2b6700 1 --2- 172.21.15.195:0/988114609 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6de8063d30 0x7f6de80661e0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:50.384 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.374+0000 7f6e0e2b6700 1 --2- 172.21.15.195:0/988114609 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6e080c3740 0x7f6e08141850 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:50.384 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.374+0000 7f6e0e2b6700 1 --2- 172.21.15.195:0/988114609 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6e0806d790 0x7f6e080c0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:50.384 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.374+0000 7f6e0e2b6700 1 --2- 172.21.15.195:0/988114609 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6e0806dbd0 0x7f6e080c09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:05:50.385 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.374+0000 7f6e0e2b6700 1 -- 172.21.15.195:0/988114609 >> 172.21.15.195:0/988114609 conn(0x7f6e08067ed0 msgr2=0x7f6e0814caf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:05:50.385 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.374+0000 7f6e0e2b6700 1 -- 172.21.15.195:0/988114609 shutdown_connections 2022-06-27T12:05:50.385 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:05:50.374+0000 7f6e0e2b6700 1 -- 172.21.15.195:0/988114609 wait complete. 2022-06-27T12:05:50.385 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 66 2022-06-27T12:05:50.387 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2022-06-27T12:05:50.388 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 27 secs before thrashing 2022-06-27T12:05:51.023 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 76% (166/217) Updating files: 77% (168/217) Updating files: 78% (170/217) Updating files: 79% (172/217) Updating files: 80% (174/217) Updating files: 81% (176/217) Updating files: 82% (178/217) Updating files: 83% (181/217) Updating files: 84% (183/217) Updating files: 85% (185/217) Updating files: 86% (187/217) Updating files: 87% (189/217) Updating files: 88% (191/217) Updating files: 89% (194/217) Updating files: 90% (196/217) Updating files: 91% (198/217) Updating files: 92% (200/217) Updating files: 93% (202/217) Updating files: 94% (204/217) Updating files: 95% (207/217) Updating files: 96% (209/217) Updating files: 97% (211/217) Updating files: 98% (213/217) Updating files: 99% (215/217) Updating files: 100% (217/217) Updating files: 100% (217/217), done. 2022-06-27T12:05:51.046 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at dac3a8c35fb v0.17 2022-06-27T12:05:51.047 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.17 2022-06-27T12:05:51.138 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:05:51.138 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 18 -eq 48 ']' 2022-06-27T12:05:51.138 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.18 2022-06-27T12:05:51.139 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.18 2022-06-27T12:05:51.139 INFO:tasks.workunit.client.0.smithi195.stdout:v0.18 2022-06-27T12:05:51.140 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.18 2022-06-27T12:06:01.117 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:06:01.121 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:06:03.180 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 4% (32/779) Updating files: 5% (39/779) Updating files: 6% (47/779) Updating files: 7% (55/779) Updating files: 8% (63/779) Updating files: 9% (71/779) Updating files: 10% (78/779) Updating files: 11% (86/779) Updating files: 12% (94/779) Updating files: 13% (102/779) Updating files: 14% (110/779) Updating files: 15% (117/779) Updating files: 16% (125/779) Updating files: 16% (127/779) Updating files: 17% (133/779) Updating files: 17% (136/779) Updating files: 18% (141/779) Updating files: 19% (149/779) Updating files: 20% (156/779) Updating files: 21% (164/779) Updating files: 21% (166/779) Updating files: 22% (172/779) Updating files: 22% (179/779) Updating files: 23% (180/779) Updating files: 23% (182/779) Updating files: 24% (187/779) Updating files: 25% (195/779) Updating files: 26% (203/779) Updating files: 27% (211/779) Updating files: 28% (219/779) Updating files: 29% (226/779) Updating files: 30% (234/779) Updating files: 31% (242/779) Updating files: 32% (250/779) Updating files: 33% (258/779) Updating files: 34% (265/779) Updating files: 35% (273/779) Updating files: 36% (281/779) Updating files: 36% (287/779) Updating files: 37% (289/779) Updating files: 38% (297/779) Updating files: 39% (304/779) Updating files: 40% (312/779) Updating files: 41% (320/779) Updating files: 42% (328/779) Updating files: 43% (335/779) Updating files: 44% (343/779) Updating files: 45% (351/779) Updating files: 46% (359/779) Updating files: 47% (367/779) Updating files: 48% (374/779) Updating files: 49% (382/779) Updating files: 50% (390/779) Updating files: 51% (398/779) Updating files: 52% (406/779) Updating files: 52% (407/779) Updating files: 53% (413/779) Updating files: 54% (421/779) Updating files: 55% (429/779) Updating files: 56% (437/779) Updating files: 57% (445/779) Updating files: 58% (452/779) Updating files: 59% (460/779) Updating files: 60% (468/779) Updating files: 61% (476/779) Updating files: 62% (483/779) Updating files: 63% (491/779) Updating files: 64% (499/779) Updating files: 65% (507/779) Updating files: 66% (515/779) Updating files: 67% (522/779) Updating files: 68% (530/779) Updating files: 69% (538/779) Updating files: 70% (546/779) Updating files: 71% (554/779) Updating files: 72% (561/779) Updating files: 73% (569/779) Updating files: 74% (577/779) Updating files: 75% (585/779) Updating files: 76% (593/779) Updating files: 77% (600/779) Updating files: 78% (608/779) Updating files: 78% (610/779) Updating files: 79% (616/779) Updating files: 80% (624/779) Updating files: 81% (631/779) Updating files: 82% (639/779) Updating files: 83% (647/779) Updating files: 84% (655/779) Updating files: 85% (663/779) Updating files: 86% (670/779) Updating files: 87% (678/779) Updating files: 88% (686/779) Updating files: 89% (694/779) Updating files: 90% (702/779) Updating files: 91% (709/779) Updating files: 92% (717/779) Updating files: 93% (725/779) Updating files: 94% (733/779) Updating files: 95% (741/779) Updating files: 96% (748/779) Updating files: 97% (756/779) Updating files: 98% (764/779) Updating files: 99% (772/779) Updating files: 100% (779/779) Updating files: 100% (779/779), done. 2022-06-27T12:06:03.391 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at 2f10137986a v0.18 2022-06-27T12:06:03.393 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.18 2022-06-27T12:06:03.474 INFO:tasks.workunit.client.0.smithi195.stdout:v0.19 2022-06-27T12:06:03.475 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:06:03.476 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 19 -eq 48 ']' 2022-06-27T12:06:03.476 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.19 2022-06-27T12:06:03.477 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.19 2022-06-27T12:06:03.477 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.19 2022-06-27T12:06:04.145 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at 98f5be53f85 v0.19 2022-06-27T12:06:04.148 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.19 2022-06-27T12:06:05.116 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:06:05.117 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 20 -eq 48 ']' 2022-06-27T12:06:05.117 INFO:tasks.workunit.client.0.smithi195.stdout:v0.20 2022-06-27T12:06:05.118 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.20 2022-06-27T12:06:05.120 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.20 2022-06-27T12:06:05.120 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.20 2022-06-27T12:06:08.386 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 11% (29/245) Updating files: 12% (30/245) Updating files: 13% (32/245) Updating files: 14% (35/245) Updating files: 15% (37/245) Updating files: 16% (40/245) Updating files: 17% (42/245) Updating files: 18% (45/245) Updating files: 19% (47/245) Updating files: 20% (49/245) Updating files: 21% (52/245) Updating files: 22% (54/245) Updating files: 23% (57/245) Updating files: 24% (59/245) Updating files: 25% (62/245) Updating files: 26% (64/245) Updating files: 27% (67/245) Updating files: 28% (69/245) Updating files: 29% (72/245) Updating files: 30% (74/245) Updating files: 31% (76/245) Updating files: 32% (79/245) Updating files: 33% (81/245) Updating files: 34% (84/245) Updating files: 35% (86/245) Updating files: 36% (89/245) Updating files: 37% (91/245) Updating files: 38% (94/245) Updating files: 39% (96/245) Updating files: 40% (98/245) Updating files: 41% (101/245) Updating files: 42% (103/245) Updating files: 43% (106/245) Updating files: 44% (108/245) Updating files: 45% (111/245) Updating files: 46% (113/245) Updating files: 47% (116/245) Updating files: 48% (118/245) Updating files: 49% (121/245) Updating files: 50% (123/245) Updating files: 51% (125/245) Updating files: 52% (128/245) Updating files: 53% (130/245) Updating files: 54% (133/245) Updating files: 55% (135/245) Updating files: 56% (138/245) Updating files: 57% (140/245) Updating files: 58% (143/245) Updating files: 59% (145/245) Updating files: 60% (147/245) Updating files: 61% (150/245) Updating files: 62% (152/245) Updating files: 63% (155/245) Updating files: 64% (157/245) Updating files: 65% (160/245) Updating files: 66% (162/245) Updating files: 67% (165/245) Updating files: 68% (167/245) Updating files: 69% (170/245) Updating files: 70% (172/245) Updating files: 71% (174/245) Updating files: 72% (177/245) Updating files: 73% (179/245) Updating files: 74% (182/245) Updating files: 75% (184/245) Updating files: 76% (187/245) Updating files: 77% (189/245) Updating files: 78% (192/245) Updating files: 79% (194/245) Updating files: 80% (196/245) Updating files: 81% (199/245) Updating files: 82% (201/245) Updating files: 83% (204/245) Updating files: 84% (206/245) Updating files: 85% (209/245) Updating files: 86% (211/245) Updating files: 87% (214/245) Updating files: 88% (216/245) Updating files: 89% (219/245) Updating files: 90% (221/245) Updating files: 91% (223/245) Updating files: 92% (226/245) Updating files: 93% (228/245) Updating files: 94% (231/245) Updating files: 95% (233/245) Updating files: 96% (236/245) Updating files: 97% (238/245) Updating files: 98% (241/245) Updating files: 99% (243/245) Updating files: 100% (245/245) Updating files: 100% (245/245), done. 2022-06-27T12:06:08.415 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at 266d2076b6c debian: fix up debian scripts 2022-06-27T12:06:08.416 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.20 2022-06-27T12:06:08.557 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:06:08.558 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 21 -eq 48 ']' 2022-06-27T12:06:08.558 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.21 2022-06-27T12:06:08.559 INFO:tasks.workunit.client.0.smithi195.stdout:v0.21 2022-06-27T12:06:08.560 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.21 2022-06-27T12:06:08.560 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.21 2022-06-27T12:06:17.389 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:06:17.563 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.560+0000 7f02aba0a700 1 Processor -- start 2022-06-27T12:06:17.564 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.560+0000 7f02aba0a700 1 -- start start 2022-06-27T12:06:17.564 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.561+0000 7f02aba0a700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f02a406e850 0x7f02a41570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:17.565 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.561+0000 7f02aba0a700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f02a406fef0 0x7f02a4069de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:17.565 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.561+0000 7f02aba0a700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f02a40c34a0 0x7f02a406a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:17.565 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.561+0000 7f02aba0a700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02a41575e0 con 0x7f02a406fef0 2022-06-27T12:06:17.568 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.561+0000 7f02aba0a700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02a406fba0 con 0x7f02a406e850 2022-06-27T12:06:17.568 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.561+0000 7f02aba0a700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02a406f5e0 con 0x7f02a40c34a0 2022-06-27T12:06:17.569 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.561+0000 7f02a8fa5700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f02a406fef0 0x7f02a4069de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:17.569 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.561+0000 7f02a8fa5700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f02a406fef0 0x7f02a4069de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48236/0 (socket says 172.21.15.195:48236) 2022-06-27T12:06:17.571 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.562+0000 7f02a8fa5700 1 -- 172.21.15.195:0/888032931 learned_addr learned my addr 172.21.15.195:0/888032931 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:06:17.573 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.562+0000 7f02a9fa7700 1 --2- 172.21.15.195:0/888032931 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f02a40c34a0 0x7f02a406a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:17.616 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.562+0000 7f02a97a6700 1 --2- 172.21.15.195:0/888032931 >> v2:172.21.15.202:3300/0 conn(0x7f02a406e850 0x7f02a41570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:17.616 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.562+0000 7f02a8fa5700 1 -- 172.21.15.195:0/888032931 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f02a40c34a0 msgr2=0x7f02a406a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:06:17.616 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.562+0000 7f02a8fa5700 1 --2- 172.21.15.195:0/888032931 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f02a40c34a0 0x7f02a406a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:17.617 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.562+0000 7f02a8fa5700 1 -- 172.21.15.195:0/888032931 >> v2:172.21.15.202:3300/0 conn(0x7f02a406e850 msgr2=0x7f02a41570a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:06:17.617 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.562+0000 7f02a8fa5700 1 --2- 172.21.15.195:0/888032931 >> v2:172.21.15.202:3300/0 conn(0x7f02a406e850 0x7f02a41570a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:17.617 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.562+0000 7f02a8fa5700 1 -- 172.21.15.195:0/888032931 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02a406f8c0 con 0x7f02a406fef0 2022-06-27T12:06:17.617 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.562+0000 7f02a8fa5700 1 --2- 172.21.15.195:0/888032931 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f02a406fef0 0x7f02a4069de0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f02a406e120 tx=0x7f02a0009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25bf7d1c6660281c server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:06:17.618 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.562+0000 7f0293fff700 1 -- 172.21.15.195:0/888032931 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a0011630 con 0x7f02a406fef0 2022-06-27T12:06:17.618 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.562+0000 7f0293fff700 1 -- 172.21.15.195:0/888032931 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02a0011790 con 0x7f02a406fef0 2022-06-27T12:06:17.618 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.562+0000 7f0293fff700 1 -- 172.21.15.195:0/888032931 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a0011b30 con 0x7f02a406fef0 2022-06-27T12:06:17.618 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.563+0000 7f02aba0a700 1 -- 172.21.15.195:0/888032931 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f02a406fef0 msgr2=0x7f02a4069de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:17.619 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.563+0000 7f02aba0a700 1 --2- 172.21.15.195:0/888032931 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f02a406fef0 0x7f02a4069de0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f02a406e120 tx=0x7f02a0009120 comp rx=0 tx=0).stop 2022-06-27T12:06:17.619 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.563+0000 7f02aba0a700 1 -- 172.21.15.195:0/888032931 shutdown_connections 2022-06-27T12:06:17.628 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.563+0000 7f02aba0a700 1 --2- 172.21.15.195:0/888032931 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f02a40c34a0 0x7f02a406a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:17.629 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.563+0000 7f02aba0a700 1 --2- 172.21.15.195:0/888032931 >> v2:172.21.15.202:3300/0 conn(0x7f02a406e850 0x7f02a41570a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:17.629 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.563+0000 7f02aba0a700 1 --2- 172.21.15.195:0/888032931 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f02a406fef0 0x7f02a4069de0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:17.631 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.563+0000 7f02aba0a700 1 -- 172.21.15.195:0/888032931 >> 172.21.15.195:0/888032931 conn(0x7f02a40c6350 msgr2=0x7f02a40c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:06:17.632 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.563+0000 7f02aba0a700 1 -- 172.21.15.195:0/888032931 shutdown_connections 2022-06-27T12:06:17.632 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.563+0000 7f02aba0a700 1 -- 172.21.15.195:0/888032931 wait complete. 2022-06-27T12:06:17.633 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.564+0000 7f02aba0a700 1 Processor -- start 2022-06-27T12:06:17.635 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.564+0000 7f02aba0a700 1 -- start start 2022-06-27T12:06:17.636 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.565+0000 7f02aba0a700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f02a406e850 0x7f02a40c8280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:17.637 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.565+0000 7f02aba0a700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f02a40c34a0 0x7f02a41504a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:17.637 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.565+0000 7f02aba0a700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f02a414bf40 0x7f02a414ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:17.638 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.565+0000 7f02aba0a700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02a414f4d0 con 0x7f02a406e850 2022-06-27T12:06:17.639 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.565+0000 7f02aba0a700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02a40c87c0 con 0x7f02a414bf40 2022-06-27T12:06:17.639 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.565+0000 7f02aba0a700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02a4168060 con 0x7f02a40c34a0 2022-06-27T12:06:17.643 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.565+0000 7f02a97a6700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f02a406e850 0x7f02a40c8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:17.644 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.565+0000 7f02a97a6700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f02a406e850 0x7f02a40c8280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48242/0 (socket says 172.21.15.195:48242) 2022-06-27T12:06:17.644 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.565+0000 7f02a97a6700 1 -- 172.21.15.195:0/889966150 learned_addr learned my addr 172.21.15.195:0/889966150 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:06:17.645 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.565+0000 7f02a8fa5700 1 --2- 172.21.15.195:0/889966150 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f02a40c34a0 0x7f02a41504a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:17.645 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.565+0000 7f02a9fa7700 1 --2- 172.21.15.195:0/889966150 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f02a414bf40 0x7f02a414ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:17.652 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.565+0000 7f02a97a6700 1 -- 172.21.15.195:0/889966150 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f02a40c34a0 msgr2=0x7f02a41504a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:17.654 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.565+0000 7f02a97a6700 1 --2- 172.21.15.195:0/889966150 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f02a40c34a0 0x7f02a41504a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:17.654 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.565+0000 7f02a97a6700 1 -- 172.21.15.195:0/889966150 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f02a414bf40 msgr2=0x7f02a414ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:17.655 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.565+0000 7f02a97a6700 1 --2- 172.21.15.195:0/889966150 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f02a414bf40 0x7f02a414ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:17.655 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.565+0000 7f02a97a6700 1 -- 172.21.15.195:0/889966150 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02a00093a0 con 0x7f02a406e850 2022-06-27T12:06:17.656 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.565+0000 7f02a97a6700 1 --2- 172.21.15.195:0/889966150 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f02a406e850 0x7f02a40c8280 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f029400e9a0 tx=0x7f0294009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:06:17.656 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.565+0000 7f02917fa700 1 -- 172.21.15.195:0/889966150 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0294013680 con 0x7f02a406e850 2022-06-27T12:06:17.656 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.565+0000 7f02917fa700 1 -- 172.21.15.195:0/889966150 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02940137e0 con 0x7f02a406e850 2022-06-27T12:06:17.657 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.565+0000 7f02917fa700 1 -- 172.21.15.195:0/889966150 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0294013b60 con 0x7f02a406e850 2022-06-27T12:06:17.657 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.565+0000 7f02aba0a700 1 -- 172.21.15.195:0/889966150 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02a41684f0 con 0x7f02a406e850 2022-06-27T12:06:17.658 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.566+0000 7f02aba0a700 1 -- 172.21.15.195:0/889966150 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02a4164f70 con 0x7f02a406e850 2022-06-27T12:06:17.658 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.566+0000 7f02aba0a700 1 -- 172.21.15.195:0/889966150 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02a40722c0 con 0x7f02a406e850 2022-06-27T12:06:17.658 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.567+0000 7f02917fa700 1 -- 172.21.15.195:0/889966150 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f0294023070 con 0x7f02a406e850 2022-06-27T12:06:17.659 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.567+0000 7f02917fa700 1 --2- 172.21.15.195:0/889966150 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0284063db0 0x7f0284066260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:17.659 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.567+0000 7f02917fa700 1 -- 172.21.15.195:0/889966150 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6976+0+0 (secure 0 0 0) 0x7f0294080eb0 con 0x7f02a406e850 2022-06-27T12:06:17.660 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.567+0000 7f02a8fa5700 1 --2- 172.21.15.195:0/889966150 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0284063db0 0x7f0284066260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:17.660 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.568+0000 7f02a8fa5700 1 --2- 172.21.15.195:0/889966150 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0284063db0 0x7f0284066260 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f02a0009160 tx=0x7f02a0017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:06:17.660 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.572+0000 7f02917fa700 1 -- 172.21.15.195:0/889966150 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f029404f5e0 con 0x7f02a406e850 2022-06-27T12:06:17.791 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.791+0000 7f02aba0a700 1 -- 172.21.15.195:0/889966150 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f02a406f240 con 0x7f02a406e850 2022-06-27T12:06:17.792 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.791+0000 7f02917fa700 1 -- 172.21.15.195:0/889966150 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+6128 (secure 0 0 0) 0x7f0294017090 con 0x7f02a406e850 2022-06-27T12:06:17.792 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:06:17.792 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":5591,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/3022591968","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3022591968},{"type":"v1","addr":"172.21.15.202:6833","nonce":3022591968}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:05:40.259006+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5489},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5489":{"gid":5489,"name":"d","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.202:6835/2342195382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2342195382},{"type":"v1","addr":"172.21.15.202:6835","nonce":2342195382}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:06:17.794 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.793+0000 7f02aba0a700 1 -- 172.21.15.195:0/889966150 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0284063db0 msgr2=0x7f0284066260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:17.794 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.793+0000 7f02aba0a700 1 --2- 172.21.15.195:0/889966150 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0284063db0 0x7f0284066260 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f02a0009160 tx=0x7f02a0017000 comp rx=0 tx=0).stop 2022-06-27T12:06:17.795 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.793+0000 7f02aba0a700 1 -- 172.21.15.195:0/889966150 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f02a406e850 msgr2=0x7f02a40c8280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:17.797 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.793+0000 7f02aba0a700 1 --2- 172.21.15.195:0/889966150 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f02a406e850 0x7f02a40c8280 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f029400e9a0 tx=0x7f0294009060 comp rx=0 tx=0).stop 2022-06-27T12:06:17.797 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.793+0000 7f02aba0a700 1 -- 172.21.15.195:0/889966150 shutdown_connections 2022-06-27T12:06:17.797 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.793+0000 7f02aba0a700 1 --2- 172.21.15.195:0/889966150 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0284063db0 0x7f0284066260 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:17.798 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.793+0000 7f02aba0a700 1 --2- 172.21.15.195:0/889966150 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f02a406e850 0x7f02a40c8280 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:17.798 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.793+0000 7f02aba0a700 1 --2- 172.21.15.195:0/889966150 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f02a414bf40 0x7f02a414ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:17.799 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.793+0000 7f02aba0a700 1 --2- 172.21.15.195:0/889966150 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f02a40c34a0 0x7f02a41504a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:17.799 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.793+0000 7f02aba0a700 1 -- 172.21.15.195:0/889966150 >> 172.21.15.195:0/889966150 conn(0x7f02a40c6350 msgr2=0x7f02a40686e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:06:17.800 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.793+0000 7f02aba0a700 1 -- 172.21.15.195:0/889966150 shutdown_connections 2022-06-27T12:06:17.800 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.793+0000 7f02aba0a700 1 -- 172.21.15.195:0/889966150 wait complete. 2022-06-27T12:06:17.800 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 66 2022-06-27T12:06:17.809 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.d (rank=0) 2022-06-27T12:06:17.810 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2022-06-27T12:06:17.811 INFO:teuthology.orchestra.run:waiting for 300 2022-06-27T12:06:17.826 INFO:tasks.ceph.mds.d:Stopped 2022-06-27T12:06:17.827 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:06:17.956 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.956+0000 7f84524d2700 1 Processor -- start 2022-06-27T12:06:17.956 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.956+0000 7f84524d2700 1 -- start start 2022-06-27T12:06:17.964 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.957+0000 7f84524d2700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f844c0c5000 0x7f844c0c2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:17.964 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.957+0000 7f84524d2700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f844c0c61e0 0x7f844c0c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:17.965 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.957+0000 7f84524d2700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f844c0bc490 con 0x7f844c0c2bf0 2022-06-27T12:06:17.965 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.957+0000 7f84524d2700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f844c06a0e0 con 0x7f844c0c61e0 2022-06-27T12:06:17.966 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.957+0000 7f84524d2700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f844c0bc6e0 con 0x7f844c0c5000 2022-06-27T12:06:17.966 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.957+0000 7f8450a6f700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f844c0c61e0 0x7f844c0c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:17.967 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.957+0000 7f8450a6f700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f844c0c61e0 0x7f844c0c3900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48254/0 (socket says 172.21.15.195:48254) 2022-06-27T12:06:17.967 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.957+0000 7f8450a6f700 1 -- 172.21.15.195:0/1726004496 learned_addr learned my addr 172.21.15.195:0/1726004496 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:06:17.968 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.957+0000 7f844b7fe700 1 --2- 172.21.15.195:0/1726004496 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f844c0c5000 0x7f844c0c2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:17.968 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.958+0000 7f8450a6f700 1 -- 172.21.15.195:0/1726004496 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f844c0c5000 msgr2=0x7f844c0c2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:06:17.968 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.958+0000 7f8450a6f700 1 --2- 172.21.15.195:0/1726004496 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f844c0c5000 0x7f844c0c2fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:17.969 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.958+0000 7f8450a6f700 1 -- 172.21.15.195:0/1726004496 >> v1:172.21.15.202:6789/0 conn(0x7f844c0c2bf0 legacy=0x7f844c0c48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:06:17.969 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.958+0000 7f8450a6f700 1 -- 172.21.15.195:0/1726004496 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f844c158220 con 0x7f844c0c61e0 2022-06-27T12:06:17.969 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.958+0000 7f844b7fe700 1 --2- 172.21.15.195:0/1726004496 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f844c0c5000 0x7f844c0c2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T12:06:17.970 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.958+0000 7f8450a6f700 1 --2- 172.21.15.195:0/1726004496 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f844c0c61e0 0x7f844c0c3900 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f8444009f20 tx=0x7f8444009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46b0f80b7e16e4ef server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:06:17.970 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.958+0000 7f844affd700 1 -- 172.21.15.195:0/1726004496 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f844400e070 con 0x7f844c0c61e0 2022-06-27T12:06:17.971 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.958+0000 7f844affd700 1 -- 172.21.15.195:0/1726004496 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f844400ba40 con 0x7f844c0c61e0 2022-06-27T12:06:17.971 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.958+0000 7f844affd700 1 -- 172.21.15.195:0/1726004496 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84440092e0 con 0x7f844c0c61e0 2022-06-27T12:06:17.971 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.958+0000 7f84524d2700 1 -- 172.21.15.195:0/1726004496 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f844c0c61e0 msgr2=0x7f844c0c3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:17.977 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.958+0000 7f84524d2700 1 --2- 172.21.15.195:0/1726004496 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f844c0c61e0 0x7f844c0c3900 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f8444009f20 tx=0x7f8444009840 comp rx=0 tx=0).stop 2022-06-27T12:06:17.978 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.958+0000 7f84524d2700 1 -- 172.21.15.195:0/1726004496 shutdown_connections 2022-06-27T12:06:17.978 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.958+0000 7f84524d2700 1 --2- 172.21.15.195:0/1726004496 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f844c0c61e0 0x7f844c0c3900 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:17.979 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.958+0000 7f84524d2700 1 --2- 172.21.15.195:0/1726004496 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f844c0c5000 0x7f844c0c2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:17.980 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.958+0000 7f84524d2700 1 -- 172.21.15.195:0/1726004496 >> 172.21.15.195:0/1726004496 conn(0x7f844c0c9b00 msgr2=0x7f844c0c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:06:17.982 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.958+0000 7f84524d2700 1 -- 172.21.15.195:0/1726004496 shutdown_connections 2022-06-27T12:06:17.984 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.958+0000 7f84524d2700 1 -- 172.21.15.195:0/1726004496 wait complete. 2022-06-27T12:06:17.984 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.960+0000 7f84524d2700 1 Processor -- start 2022-06-27T12:06:17.985 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.960+0000 7f84524d2700 1 -- start start 2022-06-27T12:06:17.986 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.960+0000 7f84524d2700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f844c0c2bf0 0x7f844c14a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:17.987 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.960+0000 7f84524d2700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f844c0c5000 0x7f844c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:17.987 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.960+0000 7f84524d2700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f844c0c61e0 0x7f844c163f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:17.988 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.960+0000 7f84524d2700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f844c168080 con 0x7f844c0c61e0 2022-06-27T12:06:17.988 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.960+0000 7f84524d2700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f844c149480 con 0x7f844c0c2bf0 2022-06-27T12:06:17.989 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.960+0000 7f84524d2700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f844c167470 con 0x7f844c0c5000 2022-06-27T12:06:17.990 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.961+0000 7f8450a6f700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f844c0c61e0 0x7f844c163f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:17.990 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.961+0000 7f8450a6f700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f844c0c61e0 0x7f844c163f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48260/0 (socket says 172.21.15.195:48260) 2022-06-27T12:06:17.991 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.961+0000 7f8450a6f700 1 -- 172.21.15.195:0/2096107487 learned_addr learned my addr 172.21.15.195:0/2096107487 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:06:17.992 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.961+0000 7f844b7fe700 1 --2- 172.21.15.195:0/2096107487 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f844c0c5000 0x7f844c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:17.992 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.961+0000 7f844bfff700 1 --2- 172.21.15.195:0/2096107487 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f844c0c2bf0 0x7f844c14a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:17.993 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.961+0000 7f8450a6f700 1 -- 172.21.15.195:0/2096107487 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f844c0c5000 msgr2=0x7f844c06e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:17.994 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.961+0000 7f8450a6f700 1 --2- 172.21.15.195:0/2096107487 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f844c0c5000 0x7f844c06e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:17.995 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.961+0000 7f8450a6f700 1 -- 172.21.15.195:0/2096107487 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f844c0c2bf0 msgr2=0x7f844c14a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:17.995 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.961+0000 7f8450a6f700 1 --2- 172.21.15.195:0/2096107487 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f844c0c2bf0 0x7f844c14a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:17.996 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.961+0000 7f8450a6f700 1 -- 172.21.15.195:0/2096107487 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8444009ae0 con 0x7f844c0c61e0 2022-06-27T12:06:17.996 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.961+0000 7f844bfff700 1 --2- 172.21.15.195:0/2096107487 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f844c0c2bf0 0x7f844c14a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T12:06:17.996 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.961+0000 7f8450a6f700 1 --2- 172.21.15.195:0/2096107487 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f844c0c61e0 0x7f844c163f60 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f844400b6c0 tx=0x7f8444017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:06:17.997 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.961+0000 7f8433fff700 1 -- 172.21.15.195:0/2096107487 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84440070e0 con 0x7f844c0c61e0 2022-06-27T12:06:17.997 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.961+0000 7f8433fff700 1 -- 172.21.15.195:0/2096107487 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8444004240 con 0x7f844c0c61e0 2022-06-27T12:06:17.998 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.961+0000 7f84524d2700 1 -- 172.21.15.195:0/2096107487 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f844c167750 con 0x7f844c0c61e0 2022-06-27T12:06:17.998 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.961+0000 7f8433fff700 1 -- 172.21.15.195:0/2096107487 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8444004600 con 0x7f844c0c61e0 2022-06-27T12:06:17.999 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.961+0000 7f84524d2700 1 -- 172.21.15.195:0/2096107487 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f844c167d10 con 0x7f844c0c61e0 2022-06-27T12:06:17.999 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.962+0000 7f84524d2700 1 -- 172.21.15.195:0/2096107487 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f844c0722c0 con 0x7f844c0c61e0 2022-06-27T12:06:17.999 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.963+0000 7f8433fff700 1 -- 172.21.15.195:0/2096107487 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f8444028460 con 0x7f844c0c61e0 2022-06-27T12:06:18.000 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.963+0000 7f8433fff700 1 --2- 172.21.15.195:0/2096107487 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f842c063d00 0x7f842c0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:18.000 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.963+0000 7f8433fff700 1 -- 172.21.15.195:0/2096107487 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6976+0+0 (secure 0 0 0) 0x7f8444082530 con 0x7f844c0c61e0 2022-06-27T12:06:18.001 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.963+0000 7f844bfff700 1 --2- 172.21.15.195:0/2096107487 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f842c063d00 0x7f842c0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:18.002 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.963+0000 7f844bfff700 1 --2- 172.21.15.195:0/2096107487 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f842c063d00 0x7f842c0661b0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f843c003d90 tx=0x7f843c009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:06:18.003 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:17.967+0000 7f8433fff700 1 -- 172.21.15.195:0/2096107487 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f8444050d30 con 0x7f844c0c61e0 2022-06-27T12:06:18.206 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:18.206+0000 7f84524d2700 1 -- 172.21.15.195:0/2096107487 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f844c151c50 con 0x7f844c0c61e0 2022-06-27T12:06:18.207 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:18.206+0000 7f8433fff700 1 -- 172.21.15.195:0/2096107487 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+6128 (secure 0 0 0) 0x7f8444054350 con 0x7f844c0c61e0 2022-06-27T12:06:18.207 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:06:18.208 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":5591,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/3022591968","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3022591968},{"type":"v1","addr":"172.21.15.202:6833","nonce":3022591968}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:05:40.259006+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5489},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5489":{"gid":5489,"name":"d","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.202:6835/2342195382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2342195382},{"type":"v1","addr":"172.21.15.202:6835","nonce":2342195382}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:06:18.209 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:18.208+0000 7f84524d2700 1 -- 172.21.15.195:0/2096107487 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f842c063d00 msgr2=0x7f842c0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:18.209 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:18.208+0000 7f84524d2700 1 --2- 172.21.15.195:0/2096107487 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f842c063d00 0x7f842c0661b0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f843c003d90 tx=0x7f843c009040 comp rx=0 tx=0).stop 2022-06-27T12:06:18.210 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:18.208+0000 7f84524d2700 1 -- 172.21.15.195:0/2096107487 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f844c0c61e0 msgr2=0x7f844c163f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:18.210 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:18.208+0000 7f84524d2700 1 --2- 172.21.15.195:0/2096107487 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f844c0c61e0 0x7f844c163f60 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f844400b6c0 tx=0x7f8444017000 comp rx=0 tx=0).stop 2022-06-27T12:06:18.211 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:18.208+0000 7f84524d2700 1 -- 172.21.15.195:0/2096107487 shutdown_connections 2022-06-27T12:06:18.211 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:18.208+0000 7f84524d2700 1 --2- 172.21.15.195:0/2096107487 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f842c063d00 0x7f842c0661b0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:18.212 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:18.208+0000 7f84524d2700 1 --2- 172.21.15.195:0/2096107487 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f844c0c61e0 0x7f844c163f60 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:18.212 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:18.208+0000 7f84524d2700 1 --2- 172.21.15.195:0/2096107487 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f844c0c2bf0 0x7f844c14a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:18.212 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:18.208+0000 7f84524d2700 1 --2- 172.21.15.195:0/2096107487 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f844c0c5000 0x7f844c06e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:18.212 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:18.208+0000 7f84524d2700 1 -- 172.21.15.195:0/2096107487 >> 172.21.15.195:0/2096107487 conn(0x7f844c0c9b00 msgr2=0x7f844c15aa80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:06:18.213 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:18.208+0000 7f84524d2700 1 -- 172.21.15.195:0/2096107487 shutdown_connections 2022-06-27T12:06:18.213 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:18.208+0000 7f84524d2700 1 -- 172.21.15.195:0/2096107487 wait complete. 2022-06-27T12:06:18.213 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 66 2022-06-27T12:06:18.221 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-06-27T12:06:20.225 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:06:20.373 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.359+0000 7f81d0745700 1 Processor -- start 2022-06-27T12:06:20.374 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.359+0000 7f81d0745700 1 -- start start 2022-06-27T12:06:20.374 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.360+0000 7f81d0745700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f81c8153560 0x7f81c80c3b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:20.374 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.360+0000 7f81d0745700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81c806a3b0 con 0x7f81c80c3740 2022-06-27T12:06:20.375 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.360+0000 7f81d0745700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81c80c4eb0 con 0x7f81c80c4050 2022-06-27T12:06:20.375 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.360+0000 7f81d0745700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81c806dea0 con 0x7f81c8153560 2022-06-27T12:06:20.376 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.360+0000 7f81cdce0700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f81c8153560 0x7f81c80c3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:20.376 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.360+0000 7f81ce4e1700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f81c80c3740 0x7f81c806d790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:54460/0 (socket says 172.21.15.195:54460) 2022-06-27T12:06:20.376 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.360+0000 7f81ce4e1700 1 -- 172.21.15.195:0/249182376 learned_addr learned my addr 172.21.15.195:0/249182376 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:06:20.377 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.360+0000 7f81cdce0700 1 -- 172.21.15.195:0/249182376 >> v1:172.21.15.202:6789/0 conn(0x7f81c80c4050 legacy=0x7f81c814c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:06:20.377 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.360+0000 7f81cdce0700 1 -- 172.21.15.195:0/249182376 >> v1:172.21.15.195:6789/0 conn(0x7f81c80c3740 legacy=0x7f81c806d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:20.377 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.360+0000 7f81cd4df700 1 -- 172.21.15.195:0/249182376 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4269632465 0 0) 0x7f81c806a3b0 con 0x7f81c80c3740 2022-06-27T12:06:20.377 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.360+0000 7f81cdce0700 1 -- 172.21.15.195:0/249182376 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81c80c5150 con 0x7f81c8153560 2022-06-27T12:06:20.378 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.361+0000 7f81cdce0700 1 --2- 172.21.15.195:0/249182376 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f81c8153560 0x7f81c80c3b10 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f81c40097a0 tx=0x7f81c4009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b392d17209a0a657 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:06:20.378 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.361+0000 7f81cd4df700 1 -- 172.21.15.195:0/249182376 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81c4004030 con 0x7f81c8153560 2022-06-27T12:06:20.378 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.361+0000 7f81cd4df700 1 -- 172.21.15.195:0/249182376 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81c4004190 con 0x7f81c8153560 2022-06-27T12:06:20.378 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.361+0000 7f81cd4df700 1 -- 172.21.15.195:0/249182376 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81c4011630 con 0x7f81c8153560 2022-06-27T12:06:20.379 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.361+0000 7f81d0745700 1 -- 172.21.15.195:0/249182376 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f81c8153560 msgr2=0x7f81c80c3b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:20.379 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.361+0000 7f81d0745700 1 --2- 172.21.15.195:0/249182376 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f81c8153560 0x7f81c80c3b10 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f81c40097a0 tx=0x7f81c4009060 comp rx=0 tx=0).stop 2022-06-27T12:06:20.379 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.361+0000 7f81d0745700 1 -- 172.21.15.195:0/249182376 shutdown_connections 2022-06-27T12:06:20.380 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.361+0000 7f81d0745700 1 --2- 172.21.15.195:0/249182376 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f81c8153560 0x7f81c80c3b10 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:20.380 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.361+0000 7f81d0745700 1 -- 172.21.15.195:0/249182376 >> 172.21.15.195:0/249182376 conn(0x7f81c8067ed0 msgr2=0x7f81c80682d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:06:20.380 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.361+0000 7f81d0745700 1 -- 172.21.15.195:0/249182376 shutdown_connections 2022-06-27T12:06:20.380 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.361+0000 7f81d0745700 1 -- 172.21.15.195:0/249182376 wait complete. 2022-06-27T12:06:20.380 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.363+0000 7f81d0745700 1 Processor -- start 2022-06-27T12:06:20.381 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.363+0000 7f81d0745700 1 -- start start 2022-06-27T12:06:20.381 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.364+0000 7f81d0745700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f81c80c3740 0x7f81c814d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:20.381 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.364+0000 7f81d0745700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f81c80c4050 0x7f81c814f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:20.381 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.364+0000 7f81d0745700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f81c8153560 0x7f81c80bfd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:20.382 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.364+0000 7f81d0745700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81c814e960 con 0x7f81c80c3740 2022-06-27T12:06:20.382 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.364+0000 7f81d0745700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81c815e390 con 0x7f81c8153560 2022-06-27T12:06:20.382 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.364+0000 7f81d0745700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81c814dab0 con 0x7f81c80c4050 2022-06-27T12:06:20.382 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.364+0000 7f81ce4e1700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f81c80c3740 0x7f81c814d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:20.383 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.364+0000 7f81ce4e1700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f81c80c3740 0x7f81c814d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48278/0 (socket says 172.21.15.195:48278) 2022-06-27T12:06:20.383 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.364+0000 7f81ce4e1700 1 -- 172.21.15.195:0/2081280988 learned_addr learned my addr 172.21.15.195:0/2081280988 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:06:20.383 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.364+0000 7f81cdce0700 1 --2- 172.21.15.195:0/2081280988 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f81c80c4050 0x7f81c814f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:20.383 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.364+0000 7f81cece2700 1 --2- 172.21.15.195:0/2081280988 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f81c8153560 0x7f81c80bfd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:20.384 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.364+0000 7f81ce4e1700 1 -- 172.21.15.195:0/2081280988 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f81c80c4050 msgr2=0x7f81c814f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:20.384 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.364+0000 7f81ce4e1700 1 --2- 172.21.15.195:0/2081280988 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f81c80c4050 0x7f81c814f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:20.384 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.364+0000 7f81ce4e1700 1 -- 172.21.15.195:0/2081280988 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f81c8153560 msgr2=0x7f81c80bfd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:20.385 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.364+0000 7f81ce4e1700 1 --2- 172.21.15.195:0/2081280988 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f81c8153560 0x7f81c80bfd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:20.386 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.364+0000 7f81ce4e1700 1 -- 172.21.15.195:0/2081280988 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81c40092e0 con 0x7f81c80c3740 2022-06-27T12:06:20.387 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.364+0000 7f81ce4e1700 1 --2- 172.21.15.195:0/2081280988 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f81c80c3740 0x7f81c814d420 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f81b800e7c0 tx=0x7f81b8007d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:06:20.387 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.365+0000 7f81b67fc700 1 -- 172.21.15.195:0/2081280988 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81b8007360 con 0x7f81c80c3740 2022-06-27T12:06:20.388 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.365+0000 7f81b67fc700 1 -- 172.21.15.195:0/2081280988 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81b80074c0 con 0x7f81c80c3740 2022-06-27T12:06:20.388 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.365+0000 7f81b67fc700 1 -- 172.21.15.195:0/2081280988 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81b8004940 con 0x7f81c80c3740 2022-06-27T12:06:20.389 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.365+0000 7f81d0745700 1 -- 172.21.15.195:0/2081280988 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81c814dd50 con 0x7f81c80c3740 2022-06-27T12:06:20.389 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.365+0000 7f81d0745700 1 -- 172.21.15.195:0/2081280988 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81c814e310 con 0x7f81c80c3740 2022-06-27T12:06:20.390 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.366+0000 7f81d0745700 1 -- 172.21.15.195:0/2081280988 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81c80722c0 con 0x7f81c80c3740 2022-06-27T12:06:20.390 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.366+0000 7f81b67fc700 1 -- 172.21.15.195:0/2081280988 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f81b8004cb0 con 0x7f81c80c3740 2022-06-27T12:06:20.391 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.366+0000 7f81b67fc700 1 --2- 172.21.15.195:0/2081280988 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f81a8063d00 0x7f81a80661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:20.391 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.366+0000 7f81b67fc700 1 -- 172.21.15.195:0/2081280988 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6976+0+0 (secure 0 0 0) 0x7f81b8083160 con 0x7f81c80c3740 2022-06-27T12:06:20.392 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.366+0000 7f81cdce0700 1 --2- 172.21.15.195:0/2081280988 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f81a8063d00 0x7f81a80661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:20.392 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.367+0000 7f81cdce0700 1 --2- 172.21.15.195:0/2081280988 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f81a8063d00 0x7f81a80661b0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f81c8145fd0 tx=0x7f81c4016000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:06:20.393 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.371+0000 7f81b67fc700 1 -- 172.21.15.195:0/2081280988 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f81b8051a40 con 0x7f81c80c3740 2022-06-27T12:06:20.598 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.598+0000 7f81d0745700 1 -- 172.21.15.195:0/2081280988 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f81c806b260 con 0x7f81c80c3740 2022-06-27T12:06:20.599 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.598+0000 7f81b67fc700 1 -- 172.21.15.195:0/2081280988 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+6128 (secure 0 0 0) 0x7f81b8015020 con 0x7f81c80c3740 2022-06-27T12:06:20.600 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:06:20.600 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":5591,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/3022591968","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3022591968},{"type":"v1","addr":"172.21.15.202:6833","nonce":3022591968}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:05:40.259006+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5489},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5489":{"gid":5489,"name":"d","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.202:6835/2342195382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2342195382},{"type":"v1","addr":"172.21.15.202:6835","nonce":2342195382}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:06:20.602 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.600+0000 7f81d0745700 1 -- 172.21.15.195:0/2081280988 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f81a8063d00 msgr2=0x7f81a80661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:20.603 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.600+0000 7f81d0745700 1 --2- 172.21.15.195:0/2081280988 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f81a8063d00 0x7f81a80661b0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f81c8145fd0 tx=0x7f81c4016000 comp rx=0 tx=0).stop 2022-06-27T12:06:20.604 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.600+0000 7f81d0745700 1 -- 172.21.15.195:0/2081280988 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f81c80c3740 msgr2=0x7f81c814d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:20.604 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.600+0000 7f81d0745700 1 --2- 172.21.15.195:0/2081280988 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f81c80c3740 0x7f81c814d420 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f81b800e7c0 tx=0x7f81b8007d40 comp rx=0 tx=0).stop 2022-06-27T12:06:20.604 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.600+0000 7f81d0745700 1 -- 172.21.15.195:0/2081280988 shutdown_connections 2022-06-27T12:06:20.605 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.600+0000 7f81d0745700 1 --2- 172.21.15.195:0/2081280988 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f81a8063d00 0x7f81a80661b0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:20.605 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.600+0000 7f81d0745700 1 --2- 172.21.15.195:0/2081280988 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f81c80c3740 0x7f81c814d420 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:20.606 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.600+0000 7f81d0745700 1 --2- 172.21.15.195:0/2081280988 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f81c8153560 0x7f81c80bfd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:20.607 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.600+0000 7f81d0745700 1 --2- 172.21.15.195:0/2081280988 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f81c80c4050 0x7f81c814f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:20.608 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.600+0000 7f81d0745700 1 -- 172.21.15.195:0/2081280988 >> 172.21.15.195:0/2081280988 conn(0x7f81c8067ed0 msgr2=0x7f81c814bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:06:20.608 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.600+0000 7f81d0745700 1 -- 172.21.15.195:0/2081280988 shutdown_connections 2022-06-27T12:06:20.608 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:20.600+0000 7f81d0745700 1 -- 172.21.15.195:0/2081280988 wait complete. 2022-06-27T12:06:20.609 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 66 2022-06-27T12:06:20.614 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-06-27T12:06:22.615 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:06:22.749 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.749+0000 7fcb251c2700 1 Processor -- start 2022-06-27T12:06:22.750 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.749+0000 7fcb251c2700 1 -- start start 2022-06-27T12:06:22.771 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.750+0000 7fcb251c2700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fcb2006e850 0x7fcb201570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:22.771 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.750+0000 7fcb251c2700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb2006fba0 con 0x7fcb2006fef0 2022-06-27T12:06:22.772 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.750+0000 7fcb251c2700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb2006f5e0 con 0x7fcb200c34a0 2022-06-27T12:06:22.772 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.750+0000 7fcb251c2700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb201575e0 con 0x7fcb2006e850 2022-06-27T12:06:22.773 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.750+0000 7fcb1ed9d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fcb2006e850 0x7fcb201570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:22.773 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.750+0000 7fcb1e59c700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fcb2006fef0 0x7fcb20069de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:54476/0 (socket says 172.21.15.195:54476) 2022-06-27T12:06:22.774 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.750+0000 7fcb1ed9d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fcb2006e850 0x7fcb201570a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48288/0 (socket says 172.21.15.195:48288) 2022-06-27T12:06:22.776 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.750+0000 7fcb1e59c700 1 -- 172.21.15.195:0/907657460 learned_addr learned my addr 172.21.15.195:0/907657460 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:06:22.777 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.751+0000 7fcb1ed9d700 1 -- 172.21.15.195:0/907657460 >> v1:172.21.15.202:6789/0 conn(0x7fcb200c34a0 legacy=0x7fcb200c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:22.777 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.751+0000 7fcb1ed9d700 1 -- 172.21.15.195:0/907657460 >> v1:172.21.15.195:6789/0 conn(0x7fcb2006fef0 legacy=0x7fcb20069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:22.778 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.751+0000 7fcb1ed9d700 1 -- 172.21.15.195:0/907657460 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb2006f8c0 con 0x7fcb2006e850 2022-06-27T12:06:22.778 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.751+0000 7fcb1dd9b700 1 -- 172.21.15.195:0/907657460 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1909691283 0 0) 0x7fcb2006fba0 con 0x7fcb2006fef0 2022-06-27T12:06:22.779 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.751+0000 7fcb1ed9d700 1 --2- 172.21.15.195:0/907657460 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcb2006e850 0x7fcb201570a0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fcb140097a0 tx=0x7fcb14009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da3df7234b640b1d server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:06:22.779 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.751+0000 7fcb1dd9b700 1 -- 172.21.15.195:0/907657460 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb14011630 con 0x7fcb2006e850 2022-06-27T12:06:22.780 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.751+0000 7fcb1dd9b700 1 -- 172.21.15.195:0/907657460 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb14011790 con 0x7fcb2006e850 2022-06-27T12:06:22.781 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.751+0000 7fcb1dd9b700 1 -- 172.21.15.195:0/907657460 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb14011b30 con 0x7fcb2006e850 2022-06-27T12:06:22.781 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.751+0000 7fcb251c2700 1 -- 172.21.15.195:0/907657460 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcb2006e850 msgr2=0x7fcb201570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:22.782 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.751+0000 7fcb251c2700 1 --2- 172.21.15.195:0/907657460 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcb2006e850 0x7fcb201570a0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fcb140097a0 tx=0x7fcb14009060 comp rx=0 tx=0).stop 2022-06-27T12:06:22.782 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.751+0000 7fcb251c2700 1 -- 172.21.15.195:0/907657460 shutdown_connections 2022-06-27T12:06:22.783 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.751+0000 7fcb251c2700 1 --2- 172.21.15.195:0/907657460 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcb2006e850 0x7fcb201570a0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:22.783 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.751+0000 7fcb251c2700 1 -- 172.21.15.195:0/907657460 >> 172.21.15.195:0/907657460 conn(0x7fcb200c6350 msgr2=0x7fcb200c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:06:22.783 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.751+0000 7fcb251c2700 1 -- 172.21.15.195:0/907657460 shutdown_connections 2022-06-27T12:06:22.784 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.751+0000 7fcb251c2700 1 -- 172.21.15.195:0/907657460 wait complete. 2022-06-27T12:06:22.784 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.753+0000 7fcb251c2700 1 Processor -- start 2022-06-27T12:06:22.785 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.753+0000 7fcb251c2700 1 -- start start 2022-06-27T12:06:22.785 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.753+0000 7fcb251c2700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcb2006e850 0x7fcb200680b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:22.785 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.753+0000 7fcb251c2700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcb200c34a0 0x7fcb201663b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:22.786 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.753+0000 7fcb251c2700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcb201542e0 0x7fcb200bf840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:22.786 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.753+0000 7fcb251c2700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb2016a350 con 0x7fcb2006e850 2022-06-27T12:06:22.787 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.753+0000 7fcb251c2700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb20154d80 con 0x7fcb200c34a0 2022-06-27T12:06:22.787 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.753+0000 7fcb251c2700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb2016a7e0 con 0x7fcb201542e0 2022-06-27T12:06:22.788 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.753+0000 7fcb1ed9d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcb2006e850 0x7fcb200680b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:22.788 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.754+0000 7fcb1ed9d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcb2006e850 0x7fcb200680b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48292/0 (socket says 172.21.15.195:48292) 2022-06-27T12:06:22.789 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.754+0000 7fcb1ed9d700 1 -- 172.21.15.195:0/1467066981 learned_addr learned my addr 172.21.15.195:0/1467066981 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:06:22.789 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.754+0000 7fcb1e59c700 1 --2- 172.21.15.195:0/1467066981 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcb200c34a0 0x7fcb201663b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:22.789 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.754+0000 7fcb1f59e700 1 --2- 172.21.15.195:0/1467066981 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcb201542e0 0x7fcb200bf840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:22.790 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.754+0000 7fcb1ed9d700 1 -- 172.21.15.195:0/1467066981 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcb201542e0 msgr2=0x7fcb200bf840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:22.790 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.754+0000 7fcb1ed9d700 1 --2- 172.21.15.195:0/1467066981 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcb201542e0 0x7fcb200bf840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:22.790 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.754+0000 7fcb1ed9d700 1 -- 172.21.15.195:0/1467066981 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcb200c34a0 msgr2=0x7fcb201663b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:22.791 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.754+0000 7fcb1ed9d700 1 --2- 172.21.15.195:0/1467066981 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcb200c34a0 0x7fcb201663b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:22.791 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.754+0000 7fcb1ed9d700 1 -- 172.21.15.195:0/1467066981 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb140092e0 con 0x7fcb2006e850 2022-06-27T12:06:22.791 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.754+0000 7fcb1ed9d700 1 --2- 172.21.15.195:0/1467066981 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcb2006e850 0x7fcb200680b0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fcb14009ef0 tx=0x7fcb14017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:06:22.792 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.754+0000 7fcb06ffd700 1 -- 172.21.15.195:0/1467066981 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb14003800 con 0x7fcb2006e850 2022-06-27T12:06:22.792 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.754+0000 7fcb06ffd700 1 -- 172.21.15.195:0/1467066981 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb14003960 con 0x7fcb2006e850 2022-06-27T12:06:22.792 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.754+0000 7fcb251c2700 1 -- 172.21.15.195:0/1467066981 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb20155040 con 0x7fcb2006e850 2022-06-27T12:06:22.792 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.754+0000 7fcb06ffd700 1 -- 172.21.15.195:0/1467066981 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb14003d20 con 0x7fcb2006e850 2022-06-27T12:06:22.793 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.754+0000 7fcb251c2700 1 -- 172.21.15.195:0/1467066981 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb20169d00 con 0x7fcb2006e850 2022-06-27T12:06:22.793 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.755+0000 7fcb251c2700 1 -- 172.21.15.195:0/1467066981 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb200722c0 con 0x7fcb2006e850 2022-06-27T12:06:22.793 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.756+0000 7fcb06ffd700 1 -- 172.21.15.195:0/1467066981 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fcb1402f070 con 0x7fcb2006e850 2022-06-27T12:06:22.793 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.756+0000 7fcb06ffd700 1 --2- 172.21.15.195:0/1467066981 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcb00063d00 0x7fcb000661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:22.794 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.756+0000 7fcb06ffd700 1 -- 172.21.15.195:0/1467066981 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6976+0+0 (secure 0 0 0) 0x7fcb140824e0 con 0x7fcb2006e850 2022-06-27T12:06:22.794 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.756+0000 7fcb1e59c700 1 --2- 172.21.15.195:0/1467066981 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcb00063d00 0x7fcb000661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:22.794 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.756+0000 7fcb1e59c700 1 --2- 172.21.15.195:0/1467066981 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcb00063d00 0x7fcb000661b0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fcb20068fb0 tx=0x7fcb10007d50 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:06:22.795 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.760+0000 7fcb06ffd700 1 -- 172.21.15.195:0/1467066981 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fcb14050d40 con 0x7fcb2006e850 2022-06-27T12:06:22.992 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.986+0000 7fcb251c2700 1 -- 172.21.15.195:0/1467066981 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcb2016a630 con 0x7fcb2006e850 2022-06-27T12:06:22.992 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.986+0000 7fcb06ffd700 1 -- 172.21.15.195:0/1467066981 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+6128 (secure 0 0 0) 0x7fcb14015070 con 0x7fcb2006e850 2022-06-27T12:06:22.993 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:06:22.993 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":5591,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/3022591968","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3022591968},{"type":"v1","addr":"172.21.15.202:6833","nonce":3022591968}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:05:40.259006+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5489},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5489":{"gid":5489,"name":"d","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.202:6835/2342195382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2342195382},{"type":"v1","addr":"172.21.15.202:6835","nonce":2342195382}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:06:22.995 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.988+0000 7fcb251c2700 1 -- 172.21.15.195:0/1467066981 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcb00063d00 msgr2=0x7fcb000661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:22.996 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.988+0000 7fcb251c2700 1 --2- 172.21.15.195:0/1467066981 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcb00063d00 0x7fcb000661b0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fcb20068fb0 tx=0x7fcb10007d50 comp rx=0 tx=0).stop 2022-06-27T12:06:22.998 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.988+0000 7fcb251c2700 1 -- 172.21.15.195:0/1467066981 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcb2006e850 msgr2=0x7fcb200680b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:22.999 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.988+0000 7fcb251c2700 1 --2- 172.21.15.195:0/1467066981 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcb2006e850 0x7fcb200680b0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fcb14009ef0 tx=0x7fcb14017000 comp rx=0 tx=0).stop 2022-06-27T12:06:22.999 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.988+0000 7fcb251c2700 1 -- 172.21.15.195:0/1467066981 shutdown_connections 2022-06-27T12:06:22.999 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.988+0000 7fcb251c2700 1 --2- 172.21.15.195:0/1467066981 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcb00063d00 0x7fcb000661b0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:23.000 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.988+0000 7fcb251c2700 1 --2- 172.21.15.195:0/1467066981 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcb2006e850 0x7fcb200680b0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:23.000 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.988+0000 7fcb251c2700 1 --2- 172.21.15.195:0/1467066981 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcb201542e0 0x7fcb200bf840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:23.001 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.988+0000 7fcb251c2700 1 --2- 172.21.15.195:0/1467066981 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcb200c34a0 0x7fcb201663b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:23.001 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.988+0000 7fcb251c2700 1 -- 172.21.15.195:0/1467066981 >> 172.21.15.195:0/1467066981 conn(0x7fcb200c6350 msgr2=0x7fcb2014f730 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:06:23.002 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.988+0000 7fcb251c2700 1 -- 172.21.15.195:0/1467066981 shutdown_connections 2022-06-27T12:06:23.003 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:22.988+0000 7fcb251c2700 1 -- 172.21.15.195:0/1467066981 wait complete. 2022-06-27T12:06:23.004 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 66 2022-06-27T12:06:23.010 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-06-27T12:06:25.012 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:06:25.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.149+0000 7f2ec2052700 1 Processor -- start 2022-06-27T12:06:25.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.149+0000 7f2ec2052700 1 -- start start 2022-06-27T12:06:25.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.150+0000 7f2ec2052700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f2ebc156100 0x7f2ebc0688a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:25.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.150+0000 7f2ec2052700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f2ebc06a440 0x7f2ebc0c4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:25.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.150+0000 7f2ec2052700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2ebc1564d0 0x7f2ebc156950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:25.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.150+0000 7f2ec2052700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ebc0c3dc0 con 0x7f2ebc156100 2022-06-27T12:06:25.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.150+0000 7f2ec2052700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ebc0c4b10 con 0x7f2ebc06a440 2022-06-27T12:06:25.155 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.150+0000 7f2ec2052700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ebc068f20 con 0x7f2ebc1564d0 2022-06-27T12:06:25.155 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.150+0000 7f2ebb7fe700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f2ebc156100 0x7f2ebc0688a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:25.155 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.150+0000 7f2ebb7fe700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f2ebc156100 0x7f2ebc0688a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48300/0 (socket says 172.21.15.195:48300) 2022-06-27T12:06:25.155 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.150+0000 7f2ebb7fe700 1 -- 172.21.15.195:0/1221322090 learned_addr learned my addr 172.21.15.195:0/1221322090 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:06:25.155 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.150+0000 7f2ebaffd700 1 --2- 172.21.15.195:0/1221322090 >> v2:172.21.15.202:3300/0 conn(0x7f2ebc06a440 0x7f2ebc0c4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:25.156 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.150+0000 7f2ebbfff700 1 --2- 172.21.15.195:0/1221322090 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2ebc1564d0 0x7f2ebc156950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:25.156 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.150+0000 7f2ebb7fe700 1 -- 172.21.15.195:0/1221322090 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2ebc1564d0 msgr2=0x7f2ebc156950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:06:25.156 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.150+0000 7f2ebb7fe700 1 --2- 172.21.15.195:0/1221322090 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2ebc1564d0 0x7f2ebc156950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:25.156 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.150+0000 7f2ebb7fe700 1 -- 172.21.15.195:0/1221322090 >> v2:172.21.15.202:3300/0 conn(0x7f2ebc06a440 msgr2=0x7f2ebc0c4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:06:25.157 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.150+0000 7f2ebb7fe700 1 --2- 172.21.15.195:0/1221322090 >> v2:172.21.15.202:3300/0 conn(0x7f2ebc06a440 0x7f2ebc0c4fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:25.157 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.150+0000 7f2ebb7fe700 1 -- 172.21.15.195:0/1221322090 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ebc0c6360 con 0x7f2ebc156100 2022-06-27T12:06:25.157 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.151+0000 7f2ebb7fe700 1 --2- 172.21.15.195:0/1221322090 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2ebc156100 0x7f2ebc0688a0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f2ebc072140 tx=0x7f2eb00090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b12bfce8392495fc server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:06:25.157 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.151+0000 7f2eba7fc700 1 -- 172.21.15.195:0/1221322090 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb0005ad0 con 0x7f2ebc156100 2022-06-27T12:06:25.157 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.151+0000 7f2eba7fc700 1 -- 172.21.15.195:0/1221322090 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2eb0005c30 con 0x7f2ebc156100 2022-06-27T12:06:25.158 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.151+0000 7f2eba7fc700 1 -- 172.21.15.195:0/1221322090 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb0004030 con 0x7f2ebc156100 2022-06-27T12:06:25.158 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.151+0000 7f2ec2052700 1 -- 172.21.15.195:0/1221322090 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2ebc156100 msgr2=0x7f2ebc0688a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:25.158 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.151+0000 7f2ec2052700 1 --2- 172.21.15.195:0/1221322090 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2ebc156100 0x7f2ebc0688a0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f2ebc072140 tx=0x7f2eb00090f0 comp rx=0 tx=0).stop 2022-06-27T12:06:25.159 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.151+0000 7f2ec2052700 1 -- 172.21.15.195:0/1221322090 shutdown_connections 2022-06-27T12:06:25.159 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.151+0000 7f2ec2052700 1 --2- 172.21.15.195:0/1221322090 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2ebc1564d0 0x7f2ebc156950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:25.159 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.151+0000 7f2ec2052700 1 --2- 172.21.15.195:0/1221322090 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2ebc156100 0x7f2ebc0688a0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:25.160 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.151+0000 7f2ec2052700 1 --2- 172.21.15.195:0/1221322090 >> v2:172.21.15.202:3300/0 conn(0x7f2ebc06a440 0x7f2ebc0c4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:25.160 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.151+0000 7f2ec2052700 1 -- 172.21.15.195:0/1221322090 >> 172.21.15.195:0/1221322090 conn(0x7f2ebc0c9b00 msgr2=0x7f2ebc0c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:06:25.160 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.151+0000 7f2ec2052700 1 -- 172.21.15.195:0/1221322090 shutdown_connections 2022-06-27T12:06:25.161 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.151+0000 7f2ec2052700 1 -- 172.21.15.195:0/1221322090 wait complete. 2022-06-27T12:06:25.161 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.153+0000 7f2ec2052700 1 Processor -- start 2022-06-27T12:06:25.161 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.153+0000 7f2ec2052700 1 -- start start 2022-06-27T12:06:25.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.153+0000 7f2ec2052700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2ebc06a440 0x7f2ebc155850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:25.171 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.153+0000 7f2ec2052700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2ebc156100 0x7f2ebc0c19e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:25.171 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.153+0000 7f2ec2052700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2ebc1564d0 0x7f2ebc165dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:25.172 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.153+0000 7f2ec2052700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ebc169d70 con 0x7f2ebc156100 2022-06-27T12:06:25.172 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.153+0000 7f2ec2052700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ebc16a200 con 0x7f2ebc1564d0 2022-06-27T12:06:25.173 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.153+0000 7f2ec2052700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ebc0c8990 con 0x7f2ebc06a440 2022-06-27T12:06:25.173 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.154+0000 7f2ebaffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2ebc156100 0x7f2ebc0c19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:25.174 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.154+0000 7f2ebaffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2ebc156100 0x7f2ebc0c19e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48308/0 (socket says 172.21.15.195:48308) 2022-06-27T12:06:25.174 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.154+0000 7f2ebaffd700 1 -- 172.21.15.195:0/2258176569 learned_addr learned my addr 172.21.15.195:0/2258176569 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:06:25.175 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.154+0000 7f2ebbfff700 1 --2- 172.21.15.195:0/2258176569 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2ebc1564d0 0x7f2ebc165dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:25.175 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.154+0000 7f2ebb7fe700 1 --2- 172.21.15.195:0/2258176569 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2ebc06a440 0x7f2ebc155850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:25.176 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.154+0000 7f2ebaffd700 1 -- 172.21.15.195:0/2258176569 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2ebc06a440 msgr2=0x7f2ebc155850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:25.176 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.154+0000 7f2ebaffd700 1 --2- 172.21.15.195:0/2258176569 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2ebc06a440 0x7f2ebc155850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:25.177 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.154+0000 7f2ebaffd700 1 -- 172.21.15.195:0/2258176569 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2ebc1564d0 msgr2=0x7f2ebc165dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:25.177 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.154+0000 7f2ebaffd700 1 --2- 172.21.15.195:0/2258176569 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2ebc1564d0 0x7f2ebc165dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:25.177 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.154+0000 7f2ebaffd700 1 -- 172.21.15.195:0/2258176569 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eb0009350 con 0x7f2ebc156100 2022-06-27T12:06:25.178 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.154+0000 7f2ebaffd700 1 --2- 172.21.15.195:0/2258176569 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2ebc156100 0x7f2ebc0c19e0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f2eac0069c0 tx=0x7f2eac009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:06:25.178 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.154+0000 7f2ea37fe700 1 -- 172.21.15.195:0/2258176569 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eac00c6a0 con 0x7f2ebc156100 2022-06-27T12:06:25.179 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.154+0000 7f2ea37fe700 1 -- 172.21.15.195:0/2258176569 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2eac00c800 con 0x7f2ebc156100 2022-06-27T12:06:25.179 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.154+0000 7f2ec2052700 1 -- 172.21.15.195:0/2258176569 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ebc0c8c70 con 0x7f2ebc156100 2022-06-27T12:06:25.180 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.154+0000 7f2ea37fe700 1 -- 172.21.15.195:0/2258176569 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eac00cba0 con 0x7f2ebc156100 2022-06-27T12:06:25.180 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.154+0000 7f2ec2052700 1 -- 172.21.15.195:0/2258176569 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ebc169720 con 0x7f2ebc156100 2022-06-27T12:06:25.180 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.155+0000 7f2ec2052700 1 -- 172.21.15.195:0/2258176569 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ebc169f20 con 0x7f2ebc156100 2022-06-27T12:06:25.181 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.156+0000 7f2ea37fe700 1 -- 172.21.15.195:0/2258176569 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f2eac019030 con 0x7f2ebc156100 2022-06-27T12:06:25.181 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.156+0000 7f2ea37fe700 1 --2- 172.21.15.195:0/2258176569 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2e9c063d30 0x7f2e9c0661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:25.182 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.156+0000 7f2ea37fe700 1 -- 172.21.15.195:0/2258176569 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6976+0+0 (secure 0 0 0) 0x7f2eac080b90 con 0x7f2ebc156100 2022-06-27T12:06:25.182 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.156+0000 7f2ebb7fe700 1 --2- 172.21.15.195:0/2258176569 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2e9c063d30 0x7f2e9c0661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:25.183 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.157+0000 7f2ebb7fe700 1 --2- 172.21.15.195:0/2258176569 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2e9c063d30 0x7f2e9c0661e0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f2eb0009130 tx=0x7f2eb0015000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:06:25.183 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.160+0000 7f2ea37fe700 1 -- 172.21.15.195:0/2258176569 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f2ebc169f20 con 0x7f2ebc156100 2022-06-27T12:06:25.388 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.389+0000 7f2ec2052700 1 -- 172.21.15.195:0/2258176569 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2ebc0722c0 con 0x7f2ebc156100 2022-06-27T12:06:25.389 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.389+0000 7f2ea37fe700 1 -- 172.21.15.195:0/2258176569 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+6128 (secure 0 0 0) 0x7f2eac017090 con 0x7f2ebc156100 2022-06-27T12:06:25.390 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:06:25.390 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":5591,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/3022591968","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3022591968},{"type":"v1","addr":"172.21.15.202:6833","nonce":3022591968}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:05:40.259006+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5489},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5489":{"gid":5489,"name":"d","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.202:6835/2342195382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2342195382},{"type":"v1","addr":"172.21.15.202:6835","nonce":2342195382}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:06:25.391 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.391+0000 7f2ec2052700 1 -- 172.21.15.195:0/2258176569 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2e9c063d30 msgr2=0x7f2e9c0661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:25.392 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.391+0000 7f2ec2052700 1 --2- 172.21.15.195:0/2258176569 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2e9c063d30 0x7f2e9c0661e0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f2eb0009130 tx=0x7f2eb0015000 comp rx=0 tx=0).stop 2022-06-27T12:06:25.392 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.391+0000 7f2ec2052700 1 -- 172.21.15.195:0/2258176569 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2ebc156100 msgr2=0x7f2ebc0c19e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:25.392 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.391+0000 7f2ec2052700 1 --2- 172.21.15.195:0/2258176569 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2ebc156100 0x7f2ebc0c19e0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f2eac0069c0 tx=0x7f2eac009060 comp rx=0 tx=0).stop 2022-06-27T12:06:25.393 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.391+0000 7f2ec2052700 1 -- 172.21.15.195:0/2258176569 shutdown_connections 2022-06-27T12:06:25.393 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.391+0000 7f2ec2052700 1 --2- 172.21.15.195:0/2258176569 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2e9c063d30 0x7f2e9c0661e0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:25.393 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.391+0000 7f2ec2052700 1 --2- 172.21.15.195:0/2258176569 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2ebc1564d0 0x7f2ebc165dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:25.393 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.391+0000 7f2ec2052700 1 --2- 172.21.15.195:0/2258176569 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2ebc06a440 0x7f2ebc155850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:25.393 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.391+0000 7f2ec2052700 1 --2- 172.21.15.195:0/2258176569 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2ebc156100 0x7f2ebc0c19e0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:25.394 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.391+0000 7f2ec2052700 1 -- 172.21.15.195:0/2258176569 >> 172.21.15.195:0/2258176569 conn(0x7f2ebc0c9b00 msgr2=0x7f2ebc071200 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:06:25.394 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.391+0000 7f2ec2052700 1 -- 172.21.15.195:0/2258176569 shutdown_connections 2022-06-27T12:06:25.394 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:25.391+0000 7f2ec2052700 1 -- 172.21.15.195:0/2258176569 wait complete. 2022-06-27T12:06:25.395 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 66 2022-06-27T12:06:25.404 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-06-27T12:06:27.405 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:06:27.538 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.537+0000 7f006ba2c700 1 Processor -- start 2022-06-27T12:06:27.539 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.538+0000 7f006ba2c700 1 -- start start 2022-06-27T12:06:27.547 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.539+0000 7f006ba2c700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f00640c4160 0x7f0064156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:27.547 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.539+0000 7f006ba2c700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00640c1a20 con 0x7f00640c39e0 2022-06-27T12:06:27.548 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.539+0000 7f006ba2c700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00640c1130 con 0x7f006406f640 2022-06-27T12:06:27.548 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.539+0000 7f006ba2c700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f006406ea90 con 0x7f00640c4160 2022-06-27T12:06:27.548 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.539+0000 7f0069fc9700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f00640c39e0 0x7f00641535a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:54508/0 (socket says 172.21.15.195:54508) 2022-06-27T12:06:27.548 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.539+0000 7f0069fc9700 1 -- 172.21.15.195:0/1546889878 learned_addr learned my addr 172.21.15.195:0/1546889878 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:06:27.549 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.539+0000 7f0068fc7700 1 --2- 172.21.15.195:0/1546889878 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f00640c4160 0x7f0064156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:27.549 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.539+0000 7f0057fff700 1 -- 172.21.15.195:0/1546889878 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3096354305 0 0) 0x7f00640c1a20 con 0x7f00640c39e0 2022-06-27T12:06:27.549 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.539+0000 7f0057fff700 1 -- 172.21.15.195:0/1546889878 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f004c003430 con 0x7f00640c39e0 2022-06-27T12:06:27.549 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.539+0000 7f0057fff700 1 -- 172.21.15.195:0/1546889878 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 378572498 0 0) 0x7f00640c1130 con 0x7f006406f640 2022-06-27T12:06:27.550 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.539+0000 7f0057fff700 1 -- 172.21.15.195:0/1546889878 --> v1:172.21.15.202:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00640c1a20 con 0x7f006406f640 2022-06-27T12:06:27.550 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.539+0000 7f0057fff700 1 -- 172.21.15.195:0/1546889878 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3940140055 0 0) 0x7f004c003430 con 0x7f00640c39e0 2022-06-27T12:06:27.550 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.539+0000 7f0057fff700 1 -- 172.21.15.195:0/1546889878 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f00640c4160 msgr2=0x7f0064156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:06:27.550 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.539+0000 7f0057fff700 1 --2- 172.21.15.195:0/1546889878 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f00640c4160 0x7f0064156ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:27.551 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.539+0000 7f0057fff700 1 -- 172.21.15.195:0/1546889878 >> v1:172.21.15.202:6789/0 conn(0x7f006406f640 legacy=0x7f00640c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:27.551 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.539+0000 7f0057fff700 1 -- 172.21.15.195:0/1546889878 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f006406ed70 con 0x7f00640c39e0 2022-06-27T12:06:27.551 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.539+0000 7f0057fff700 1 -- 172.21.15.195:0/1546889878 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7f005c0017f0 con 0x7f00640c39e0 2022-06-27T12:06:27.551 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.539+0000 7f0057fff700 1 -- 172.21.15.195:0/1546889878 >> v1:172.21.15.195:6789/0 conn(0x7f00640c39e0 legacy=0x7f00641535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:27.560 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.539+0000 7f0057fff700 1 --2- 172.21.15.195:0/1546889878 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f004c003ae0 0x7f004c003ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:27.560 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.539+0000 7f0057fff700 1 --2- 172.21.15.195:0/1546889878 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f004c0046d0 0x7f004c004b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:27.561 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.539+0000 7f0057fff700 1 --2- 172.21.15.195:0/1546889878 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f00640c4160 0x7f004c005080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:27.561 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.539+0000 7f0057fff700 1 -- 172.21.15.195:0/1546889878 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f006406ea90 con 0x7f004c0046d0 2022-06-27T12:06:27.562 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.539+0000 7f0057fff700 1 -- 172.21.15.195:0/1546889878 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f004c00ba00 con 0x7f004c003ae0 2022-06-27T12:06:27.562 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.539+0000 7f0057fff700 1 -- 172.21.15.195:0/1546889878 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f004c00bbc0 con 0x7f00640c4160 2022-06-27T12:06:27.563 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.540+0000 7f0068fc7700 1 --2- 172.21.15.195:0/1546889878 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f004c0046d0 0x7f004c004b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:27.563 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.540+0000 7f00697c8700 1 --2- 172.21.15.195:0/1546889878 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f004c003ae0 0x7f004c003ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:27.564 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.540+0000 7f0069fc9700 1 --2- 172.21.15.195:0/1546889878 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f00640c4160 0x7f004c005080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:27.564 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.540+0000 7f0068fc7700 1 -- 172.21.15.195:0/1546889878 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f00640c4160 msgr2=0x7f004c005080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:06:27.564 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.540+0000 7f0068fc7700 1 --2- 172.21.15.195:0/1546889878 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f00640c4160 0x7f004c005080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:27.565 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.540+0000 7f0068fc7700 1 -- 172.21.15.195:0/1546889878 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f004c003ae0 msgr2=0x7f004c003ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:06:27.565 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.540+0000 7f0068fc7700 1 --2- 172.21.15.195:0/1546889878 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f004c003ae0 0x7f004c003ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:27.566 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.540+0000 7f0068fc7700 1 -- 172.21.15.195:0/1546889878 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f004c003430 con 0x7f004c0046d0 2022-06-27T12:06:27.566 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.540+0000 7f0068fc7700 1 --2- 172.21.15.195:0/1546889878 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f004c0046d0 0x7f004c004b40 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f0060009ac0 tx=0x7f0060009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9296024f22c28f0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:06:27.567 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.540+0000 7f0057fff700 1 -- 172.21.15.195:0/1546889878 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0060009470 con 0x7f004c0046d0 2022-06-27T12:06:27.567 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.540+0000 7f0057fff700 1 -- 172.21.15.195:0/1546889878 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f006000e640 con 0x7f004c0046d0 2022-06-27T12:06:27.568 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.540+0000 7f0057fff700 1 -- 172.21.15.195:0/1546889878 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f006000e9e0 con 0x7f004c0046d0 2022-06-27T12:06:27.568 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.541+0000 7f006ba2c700 1 -- 172.21.15.195:0/1546889878 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f004c0046d0 msgr2=0x7f004c004b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:27.569 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.541+0000 7f006ba2c700 1 --2- 172.21.15.195:0/1546889878 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f004c0046d0 0x7f004c004b40 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f0060009ac0 tx=0x7f0060009060 comp rx=0 tx=0).stop 2022-06-27T12:06:27.569 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.541+0000 7f0069fc9700 1 -- 172.21.15.195:0/1546889878 reap_dead start 2022-06-27T12:06:27.570 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.541+0000 7f006ba2c700 1 -- 172.21.15.195:0/1546889878 shutdown_connections 2022-06-27T12:06:27.570 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.541+0000 7f006ba2c700 1 -- 172.21.15.195:0/1546889878 >> 172.21.15.195:0/1546889878 conn(0x7f00640c6350 msgr2=0x7f00640c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:06:27.571 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.541+0000 7f006ba2c700 1 -- 172.21.15.195:0/1546889878 shutdown_connections 2022-06-27T12:06:27.571 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.541+0000 7f006ba2c700 1 -- 172.21.15.195:0/1546889878 wait complete. 2022-06-27T12:06:27.572 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.542+0000 7f006ba2c700 1 Processor -- start 2022-06-27T12:06:27.572 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.542+0000 7f006ba2c700 1 -- start start 2022-06-27T12:06:27.572 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.543+0000 7f006ba2c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f004c0046d0 0x7f00641545a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:27.573 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.543+0000 7f006ba2c700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f00640be3b0 0x7f0064166980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:27.573 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.543+0000 7f006ba2c700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0064166ec0 0x7f006414e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:27.574 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.543+0000 7f006ba2c700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00640bd050 con 0x7f004c0046d0 2022-06-27T12:06:27.574 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.543+0000 7f006ba2c700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00640bc4b0 con 0x7f0064166ec0 2022-06-27T12:06:27.574 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.543+0000 7f006ba2c700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0064167330 con 0x7f00640be3b0 2022-06-27T12:06:27.575 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.543+0000 7f00697c8700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f004c0046d0 0x7f00641545a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:27.575 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.543+0000 7f00697c8700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f004c0046d0 0x7f00641545a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48328/0 (socket says 172.21.15.195:48328) 2022-06-27T12:06:27.576 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.543+0000 7f00697c8700 1 -- 172.21.15.195:0/285172322 learned_addr learned my addr 172.21.15.195:0/285172322 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:06:27.576 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.543+0000 7f0068fc7700 1 --2- 172.21.15.195:0/285172322 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f00640be3b0 0x7f0064166980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:27.576 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.543+0000 7f0069fc9700 1 --2- 172.21.15.195:0/285172322 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0064166ec0 0x7f006414e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:27.577 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.543+0000 7f00697c8700 1 -- 172.21.15.195:0/285172322 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f00640be3b0 msgr2=0x7f0064166980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:27.577 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.543+0000 7f00697c8700 1 --2- 172.21.15.195:0/285172322 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f00640be3b0 0x7f0064166980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:27.577 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.543+0000 7f00697c8700 1 -- 172.21.15.195:0/285172322 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0064166ec0 msgr2=0x7f006414e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:27.578 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.543+0000 7f00697c8700 1 --2- 172.21.15.195:0/285172322 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0064166ec0 0x7f006414e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:27.578 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.543+0000 7f00697c8700 1 -- 172.21.15.195:0/285172322 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00640bddf0 con 0x7f004c0046d0 2022-06-27T12:06:27.579 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.543+0000 7f00697c8700 1 --2- 172.21.15.195:0/285172322 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f004c0046d0 0x7f00641545a0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f005800f9e0 tx=0x7f0058009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:06:27.579 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.544+0000 7f00557fa700 1 -- 172.21.15.195:0/285172322 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00580052e0 con 0x7f004c0046d0 2022-06-27T12:06:27.579 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.544+0000 7f00557fa700 1 -- 172.21.15.195:0/285172322 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0058003500 con 0x7f004c0046d0 2022-06-27T12:06:27.580 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.544+0000 7f006ba2c700 1 -- 172.21.15.195:0/285172322 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00640bc720 con 0x7f004c0046d0 2022-06-27T12:06:27.580 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.544+0000 7f00557fa700 1 -- 172.21.15.195:0/285172322 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00580038c0 con 0x7f004c0046d0 2022-06-27T12:06:27.581 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.544+0000 7f006ba2c700 1 -- 172.21.15.195:0/285172322 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00640bcce0 con 0x7f004c0046d0 2022-06-27T12:06:27.581 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.545+0000 7f006ba2c700 1 -- 172.21.15.195:0/285172322 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00641667d0 con 0x7f004c0046d0 2022-06-27T12:06:27.581 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.545+0000 7f00557fa700 1 -- 172.21.15.195:0/285172322 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f0058003c70 con 0x7f004c0046d0 2022-06-27T12:06:27.582 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.545+0000 7f00557fa700 1 --2- 172.21.15.195:0/285172322 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0044063d30 0x7f00440661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:27.582 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.545+0000 7f00557fa700 1 -- 172.21.15.195:0/285172322 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6976+0+0 (secure 0 0 0) 0x7f0058080e10 con 0x7f004c0046d0 2022-06-27T12:06:27.583 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.545+0000 7f0068fc7700 1 --2- 172.21.15.195:0/285172322 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0044063d30 0x7f00440661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:27.583 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.546+0000 7f0068fc7700 1 --2- 172.21.15.195:0/285172322 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0044063d30 0x7f00440661e0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f00641676f0 tx=0x7f006001a000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:06:27.585 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.550+0000 7f00557fa700 1 -- 172.21.15.195:0/285172322 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f005804f670 con 0x7f004c0046d0 2022-06-27T12:06:27.774 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.774+0000 7f006ba2c700 1 -- 172.21.15.195:0/285172322 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f00640bce70 con 0x7f004c0046d0 2022-06-27T12:06:27.775 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.774+0000 7f00557fa700 1 -- 172.21.15.195:0/285172322 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+6128 (secure 0 0 0) 0x7f0058052c90 con 0x7f004c0046d0 2022-06-27T12:06:27.775 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:06:27.775 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":5591,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/3022591968","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3022591968},{"type":"v1","addr":"172.21.15.202:6833","nonce":3022591968}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:05:40.259006+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5489},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5489":{"gid":5489,"name":"d","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.202:6835/2342195382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2342195382},{"type":"v1","addr":"172.21.15.202:6835","nonce":2342195382}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:06:27.776 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.776+0000 7f006ba2c700 1 -- 172.21.15.195:0/285172322 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0044063d30 msgr2=0x7f00440661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:27.777 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.776+0000 7f006ba2c700 1 --2- 172.21.15.195:0/285172322 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0044063d30 0x7f00440661e0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f00641676f0 tx=0x7f006001a000 comp rx=0 tx=0).stop 2022-06-27T12:06:27.777 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.776+0000 7f006ba2c700 1 -- 172.21.15.195:0/285172322 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f004c0046d0 msgr2=0x7f00641545a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:27.777 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.776+0000 7f006ba2c700 1 --2- 172.21.15.195:0/285172322 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f004c0046d0 0x7f00641545a0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f005800f9e0 tx=0x7f0058009040 comp rx=0 tx=0).stop 2022-06-27T12:06:27.778 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.776+0000 7f006ba2c700 1 -- 172.21.15.195:0/285172322 shutdown_connections 2022-06-27T12:06:27.781 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.776+0000 7f006ba2c700 1 --2- 172.21.15.195:0/285172322 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0044063d30 0x7f00440661e0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:27.781 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.776+0000 7f006ba2c700 1 --2- 172.21.15.195:0/285172322 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f004c0046d0 0x7f00641545a0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:27.782 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.776+0000 7f006ba2c700 1 --2- 172.21.15.195:0/285172322 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0064166ec0 0x7f006414e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:27.782 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.776+0000 7f006ba2c700 1 --2- 172.21.15.195:0/285172322 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f00640be3b0 0x7f0064166980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:27.783 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.776+0000 7f006ba2c700 1 -- 172.21.15.195:0/285172322 >> 172.21.15.195:0/285172322 conn(0x7f00640c6350 msgr2=0x7f0064150650 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:06:27.784 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.776+0000 7f006ba2c700 1 -- 172.21.15.195:0/285172322 shutdown_connections 2022-06-27T12:06:27.784 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:27.776+0000 7f006ba2c700 1 -- 172.21.15.195:0/285172322 wait complete. 2022-06-27T12:06:27.785 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 66 2022-06-27T12:06:27.789 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-06-27T12:06:29.793 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:06:29.928 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:29.927+0000 7f31834b2700 1 Processor -- start 2022-06-27T12:06:29.928 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:29.928+0000 7f31834b2700 1 -- start start 2022-06-27T12:06:29.940 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:29.928+0000 7f31834b2700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f317c150500 0x7f317c063af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:29.940 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:29.928+0000 7f31834b2700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f317c1508d0 0x7f317c0bbf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:29.940 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:29.928+0000 7f31834b2700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f317c064030 con 0x7f317c06a450 2022-06-27T12:06:29.941 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:29.928+0000 7f31834b2700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f317c0c4160 con 0x7f317c150500 2022-06-27T12:06:29.941 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:29.928+0000 7f31834b2700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f317c0c4440 con 0x7f317c1508d0 2022-06-27T12:06:29.941 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:29.929+0000 7f318124e700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f317c150500 0x7f317c063af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:29.942 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:29.929+0000 7f318124e700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f317c150500 0x7f317c063af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48336/0 (socket says 172.21.15.195:48336) 2022-06-27T12:06:29.942 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:29.929+0000 7f318124e700 1 -- 172.21.15.195:0/2379495608 learned_addr learned my addr 172.21.15.195:0/2379495608 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:06:29.942 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:29.929+0000 7f318124e700 1 -- 172.21.15.195:0/2379495608 >> v2:172.21.15.202:3300/0 conn(0x7f317c1508d0 msgr2=0x7f317c0bbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:06:29.942 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:29.929+0000 7f318124e700 1 --2- 172.21.15.195:0/2379495608 >> v2:172.21.15.202:3300/0 conn(0x7f317c1508d0 0x7f317c0bbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:29.943 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:29.929+0000 7f318124e700 1 -- 172.21.15.195:0/2379495608 >> v1:172.21.15.202:6789/0 conn(0x7f317c06a450 legacy=0x7f317c0c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:06:29.943 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:29.929+0000 7f318124e700 1 -- 172.21.15.195:0/2379495608 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f317c15a800 con 0x7f317c150500 2022-06-27T12:06:29.943 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:29.929+0000 7f318124e700 1 --2- 172.21.15.195:0/2379495608 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f317c150500 0x7f317c063af0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f317c064800 tx=0x7f316c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2cdfb9cb74f290c server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:06:29.943 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:29.929+0000 7f316bfff700 1 -- 172.21.15.195:0/2379495608 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f316c005ad0 con 0x7f317c150500 2022-06-27T12:06:29.944 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:29.929+0000 7f316bfff700 1 -- 172.21.15.195:0/2379495608 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f316c005c30 con 0x7f317c150500 2022-06-27T12:06:29.944 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:29.929+0000 7f316bfff700 1 -- 172.21.15.195:0/2379495608 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f316c006030 con 0x7f317c150500 2022-06-27T12:06:29.944 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:29.930+0000 7f31834b2700 1 -- 172.21.15.195:0/2379495608 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f317c150500 msgr2=0x7f317c063af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:29.944 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:29.930+0000 7f31834b2700 1 --2- 172.21.15.195:0/2379495608 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f317c150500 0x7f317c063af0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f317c064800 tx=0x7f316c0090f0 comp rx=0 tx=0).stop 2022-06-27T12:06:29.945 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:29.930+0000 7f31834b2700 1 -- 172.21.15.195:0/2379495608 shutdown_connections 2022-06-27T12:06:29.945 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:29.930+0000 7f31834b2700 1 --2- 172.21.15.195:0/2379495608 >> v2:172.21.15.202:3300/0 conn(0x7f317c1508d0 0x7f317c0bbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:29.945 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:29.930+0000 7f31834b2700 1 --2- 172.21.15.195:0/2379495608 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f317c150500 0x7f317c063af0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:29.945 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:29.930+0000 7f31834b2700 1 -- 172.21.15.195:0/2379495608 >> 172.21.15.195:0/2379495608 conn(0x7f317c0c9b00 msgr2=0x7f317c0c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:06:29.945 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:29.930+0000 7f31834b2700 1 -- 172.21.15.195:0/2379495608 shutdown_connections 2022-06-27T12:06:29.946 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:29.930+0000 7f31834b2700 1 -- 172.21.15.195:0/2379495608 wait complete. 2022-06-27T12:06:29.946 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:29.931+0000 7f31834b2700 1 Processor -- start 2022-06-27T12:06:29.946 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:29.931+0000 7f31834b2700 1 -- start start 2022-06-27T12:06:29.946 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:29.932+0000 7f31834b2700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f317c06a450 0x7f317c159560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:29.947 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:29.932+0000 7f31834b2700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f317c150500 0x7f317c1589c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:29.947 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:29.932+0000 7f31834b2700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f317c1508d0 0x7f317c169ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:29.947 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:29.932+0000 7f31834b2700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f317c169ba0 con 0x7f317c1508d0 2022-06-27T12:06:29.947 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:29.932+0000 7f31834b2700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f317c167350 con 0x7f317c150500 2022-06-27T12:06:29.948 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:29.932+0000 7f31834b2700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f317c155f10 con 0x7f317c06a450 2022-06-27T12:06:29.948 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:29.932+0000 7f3181a4f700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f317c1508d0 0x7f317c169ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:29.948 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:29.932+0000 7f3181a4f700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f317c1508d0 0x7f317c169ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48344/0 (socket says 172.21.15.195:48344) 2022-06-27T12:06:29.948 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:29.932+0000 7f3181a4f700 1 -- 172.21.15.195:0/23352925 learned_addr learned my addr 172.21.15.195:0/23352925 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:06:29.949 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:29.932+0000 7f318124e700 1 --2- 172.21.15.195:0/23352925 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f317c06a450 0x7f317c159560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:29.949 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:29.932+0000 7f3180a4d700 1 --2- 172.21.15.195:0/23352925 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f317c150500 0x7f317c1589c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:29.949 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:29.932+0000 7f3181a4f700 1 -- 172.21.15.195:0/23352925 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f317c06a450 msgr2=0x7f317c159560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:29.949 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:29.932+0000 7f3181a4f700 1 --2- 172.21.15.195:0/23352925 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f317c06a450 0x7f317c159560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:29.950 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:29.932+0000 7f3181a4f700 1 -- 172.21.15.195:0/23352925 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f317c150500 msgr2=0x7f317c1589c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:29.950 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:29.932+0000 7f3181a4f700 1 --2- 172.21.15.195:0/23352925 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f317c150500 0x7f317c1589c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:29.950 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:29.932+0000 7f3181a4f700 1 -- 172.21.15.195:0/23352925 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f316c009350 con 0x7f317c1508d0 2022-06-27T12:06:29.950 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:29.932+0000 7f3180a4d700 1 --2- 172.21.15.195:0/23352925 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f317c150500 0x7f317c1589c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T12:06:29.951 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:29.932+0000 7f318124e700 1 --2- 172.21.15.195:0/23352925 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f317c06a450 0x7f317c159560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T12:06:29.951 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:29.932+0000 7f3181a4f700 1 --2- 172.21.15.195:0/23352925 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f317c1508d0 0x7f317c169ff0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f317400b8f0 tx=0x7f3174009400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:06:29.951 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:29.933+0000 7f31697fa700 1 -- 172.21.15.195:0/23352925 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f317400c7d0 con 0x7f317c1508d0 2022-06-27T12:06:29.951 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:29.933+0000 7f31697fa700 1 -- 172.21.15.195:0/23352925 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f317400c040 con 0x7f317c1508d0 2022-06-27T12:06:29.952 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:29.933+0000 7f31834b2700 1 -- 172.21.15.195:0/23352925 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f317c166e00 con 0x7f317c1508d0 2022-06-27T12:06:29.952 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:29.933+0000 7f31697fa700 1 -- 172.21.15.195:0/23352925 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f317400cb80 con 0x7f317c1508d0 2022-06-27T12:06:29.952 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:29.933+0000 7f31834b2700 1 -- 172.21.15.195:0/23352925 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f317c154410 con 0x7f317c1508d0 2022-06-27T12:06:29.952 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:29.934+0000 7f31834b2700 1 -- 172.21.15.195:0/23352925 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f317c169d10 con 0x7f317c1508d0 2022-06-27T12:06:29.953 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:29.934+0000 7f31697fa700 1 -- 172.21.15.195:0/23352925 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f317401b030 con 0x7f317c1508d0 2022-06-27T12:06:29.953 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:29.934+0000 7f31697fa700 1 --2- 172.21.15.195:0/23352925 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f315c063d00 0x7f315c0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:29.954 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:29.935+0000 7f31697fa700 1 -- 172.21.15.195:0/23352925 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6976+0+0 (secure 0 0 0) 0x7f31740802d0 con 0x7f317c1508d0 2022-06-27T12:06:29.954 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:29.935+0000 7f318124e700 1 --2- 172.21.15.195:0/23352925 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f315c063d00 0x7f315c0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:29.954 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:29.935+0000 7f318124e700 1 --2- 172.21.15.195:0/23352925 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f315c063d00 0x7f315c0661b0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f317c064800 tx=0x7f316c016000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:06:29.954 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:29.939+0000 7f31697fa700 1 -- 172.21.15.195:0/23352925 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f317c169d10 con 0x7f317c1508d0 2022-06-27T12:06:30.160 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:30.160+0000 7f31834b2700 1 -- 172.21.15.195:0/23352925 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f317c169ce0 con 0x7f317c1508d0 2022-06-27T12:06:30.161 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:30.160+0000 7f31697fa700 1 -- 172.21.15.195:0/23352925 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+6128 (secure 0 0 0) 0x7f3174019020 con 0x7f317c1508d0 2022-06-27T12:06:30.161 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:06:30.161 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":5591,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/3022591968","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3022591968},{"type":"v1","addr":"172.21.15.202:6833","nonce":3022591968}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:05:40.259006+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5489},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5489":{"gid":5489,"name":"d","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.202:6835/2342195382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2342195382},{"type":"v1","addr":"172.21.15.202:6835","nonce":2342195382}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:06:30.163 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:30.162+0000 7f31834b2700 1 -- 172.21.15.195:0/23352925 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f315c063d00 msgr2=0x7f315c0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:30.164 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:30.162+0000 7f31834b2700 1 --2- 172.21.15.195:0/23352925 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f315c063d00 0x7f315c0661b0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f317c064800 tx=0x7f316c016000 comp rx=0 tx=0).stop 2022-06-27T12:06:30.164 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:30.162+0000 7f31834b2700 1 -- 172.21.15.195:0/23352925 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f317c1508d0 msgr2=0x7f317c169ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:30.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:30.162+0000 7f31834b2700 1 --2- 172.21.15.195:0/23352925 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f317c1508d0 0x7f317c169ff0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f317400b8f0 tx=0x7f3174009400 comp rx=0 tx=0).stop 2022-06-27T12:06:30.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:30.162+0000 7f31834b2700 1 -- 172.21.15.195:0/23352925 shutdown_connections 2022-06-27T12:06:30.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:30.162+0000 7f31834b2700 1 --2- 172.21.15.195:0/23352925 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f315c063d00 0x7f315c0661b0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:30.166 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:30.162+0000 7f31834b2700 1 --2- 172.21.15.195:0/23352925 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f317c1508d0 0x7f317c169ff0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:30.166 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:30.162+0000 7f31834b2700 1 --2- 172.21.15.195:0/23352925 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f317c06a450 0x7f317c159560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:30.166 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:30.162+0000 7f31834b2700 1 --2- 172.21.15.195:0/23352925 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f317c150500 0x7f317c1589c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:30.166 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:30.162+0000 7f31834b2700 1 -- 172.21.15.195:0/23352925 >> 172.21.15.195:0/23352925 conn(0x7f317c0c9b00 msgr2=0x7f317c14f990 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:06:30.166 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:30.162+0000 7f31834b2700 1 -- 172.21.15.195:0/23352925 shutdown_connections 2022-06-27T12:06:30.167 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:30.162+0000 7f31834b2700 1 -- 172.21.15.195:0/23352925 wait complete. 2022-06-27T12:06:30.167 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 66 2022-06-27T12:06:30.174 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-06-27T12:06:31.540 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:06:31.550 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:06:32.175 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:06:32.314 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.309+0000 7ff1359ba700 1 Processor -- start 2022-06-27T12:06:32.315 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.309+0000 7ff1359ba700 1 -- start start 2022-06-27T12:06:32.315 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.310+0000 7ff1359ba700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7ff130154dd0 0x7ff130158100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:32.316 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.310+0000 7ff1359ba700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff1300c34b0 0x7ff13015ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:32.316 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.310+0000 7ff1359ba700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1300c3980 con 0x7ff130154310 2022-06-27T12:06:32.317 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.310+0000 7ff1359ba700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff130151dd0 con 0x7ff130154dd0 2022-06-27T12:06:32.318 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.310+0000 7ff1359ba700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1300c3c40 con 0x7ff1300c34b0 2022-06-27T12:06:32.318 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.310+0000 7ff12effd700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7ff130154dd0 0x7ff130158100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:32.319 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.310+0000 7ff12e7fc700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7ff130154310 0x7ff1300c1dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:54536/0 (socket says 172.21.15.195:54536) 2022-06-27T12:06:32.325 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.310+0000 7ff12e7fc700 1 -- 172.21.15.195:0/1280142554 learned_addr learned my addr 172.21.15.195:0/1280142554 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:06:32.326 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.310+0000 7ff12effd700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7ff130154dd0 0x7ff130158100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48348/0 (socket says 172.21.15.195:48348) 2022-06-27T12:06:32.326 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.310+0000 7ff12f7fe700 1 --2- 172.21.15.195:0/1280142554 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff1300c34b0 0x7ff13015ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:32.326 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.310+0000 7ff12effd700 1 -- 172.21.15.195:0/1280142554 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff1300c34b0 msgr2=0x7ff13015ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:06:32.327 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.310+0000 7ff12effd700 1 --2- 172.21.15.195:0/1280142554 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff1300c34b0 0x7ff13015ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:32.327 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.310+0000 7ff12effd700 1 -- 172.21.15.195:0/1280142554 >> v1:172.21.15.195:6789/0 conn(0x7ff130154310 legacy=0x7ff1300c1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:32.328 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.310+0000 7ff12effd700 1 -- 172.21.15.195:0/1280142554 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1300c3f20 con 0x7ff130154dd0 2022-06-27T12:06:32.328 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.310+0000 7ff12effd700 1 --2- 172.21.15.195:0/1280142554 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff130154dd0 0x7ff130158100 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7ff1240097a0 tx=0x7ff124009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5744699d3c0b25e server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:06:32.329 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.311+0000 7ff12dffb700 1 -- 172.21.15.195:0/1280142554 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff124011630 con 0x7ff130154dd0 2022-06-27T12:06:32.329 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.311+0000 7ff12dffb700 1 -- 172.21.15.195:0/1280142554 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff124011790 con 0x7ff130154dd0 2022-06-27T12:06:32.330 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.311+0000 7ff12dffb700 1 -- 172.21.15.195:0/1280142554 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff124011b30 con 0x7ff130154dd0 2022-06-27T12:06:32.331 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.311+0000 7ff1359ba700 1 -- 172.21.15.195:0/1280142554 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff130154dd0 msgr2=0x7ff130158100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:32.331 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.311+0000 7ff1359ba700 1 --2- 172.21.15.195:0/1280142554 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff130154dd0 0x7ff130158100 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7ff1240097a0 tx=0x7ff124009060 comp rx=0 tx=0).stop 2022-06-27T12:06:32.332 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.311+0000 7ff1359ba700 1 -- 172.21.15.195:0/1280142554 shutdown_connections 2022-06-27T12:06:32.332 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.311+0000 7ff1359ba700 1 --2- 172.21.15.195:0/1280142554 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff1300c34b0 0x7ff13015ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:32.333 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.311+0000 7ff1359ba700 1 --2- 172.21.15.195:0/1280142554 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff130154dd0 0x7ff130158100 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:32.333 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.311+0000 7ff1359ba700 1 -- 172.21.15.195:0/1280142554 >> 172.21.15.195:0/1280142554 conn(0x7ff1300c8740 msgr2=0x7ff1300c8b10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:06:32.334 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.311+0000 7ff1359ba700 1 -- 172.21.15.195:0/1280142554 shutdown_connections 2022-06-27T12:06:32.334 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.311+0000 7ff1359ba700 1 -- 172.21.15.195:0/1280142554 wait complete. 2022-06-27T12:06:32.349 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.312+0000 7ff1359ba700 1 Processor -- start 2022-06-27T12:06:32.350 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.313+0000 7ff1359ba700 1 -- start start 2022-06-27T12:06:32.351 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.313+0000 7ff1359ba700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff1300c34b0 0x7ff130168180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:32.351 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.313+0000 7ff1359ba700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff130154310 0x7ff13014d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:32.352 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.313+0000 7ff1359ba700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff130154dd0 0x7ff13014a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:32.352 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.313+0000 7ff1359ba700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff13014b6e0 con 0x7ff130154dd0 2022-06-27T12:06:32.352 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.313+0000 7ff1359ba700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff13006d670 con 0x7ff130154310 2022-06-27T12:06:32.352 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.313+0000 7ff1359ba700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff13014c220 con 0x7ff1300c34b0 2022-06-27T12:06:32.353 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.313+0000 7ff12f7fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff130154dd0 0x7ff13014a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:32.353 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.313+0000 7ff12f7fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff130154dd0 0x7ff13014a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48358/0 (socket says 172.21.15.195:48358) 2022-06-27T12:06:32.353 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.313+0000 7ff12f7fe700 1 -- 172.21.15.195:0/365705355 learned_addr learned my addr 172.21.15.195:0/365705355 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:06:32.353 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.313+0000 7ff12effd700 1 --2- 172.21.15.195:0/365705355 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff1300c34b0 0x7ff130168180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:32.354 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.314+0000 7ff12e7fc700 1 --2- 172.21.15.195:0/365705355 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff130154310 0x7ff13014d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:32.354 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.314+0000 7ff12f7fe700 1 -- 172.21.15.195:0/365705355 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff1300c34b0 msgr2=0x7ff130168180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:32.354 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.314+0000 7ff12f7fe700 1 --2- 172.21.15.195:0/365705355 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff1300c34b0 0x7ff130168180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:32.354 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.314+0000 7ff12f7fe700 1 -- 172.21.15.195:0/365705355 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff130154310 msgr2=0x7ff13014d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:32.355 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.314+0000 7ff12f7fe700 1 --2- 172.21.15.195:0/365705355 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff130154310 0x7ff13014d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:32.355 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.314+0000 7ff12f7fe700 1 -- 172.21.15.195:0/365705355 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1240092e0 con 0x7ff130154dd0 2022-06-27T12:06:32.355 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.314+0000 7ff12f7fe700 1 --2- 172.21.15.195:0/365705355 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff130154dd0 0x7ff13014a010 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7ff128007f50 tx=0x7ff1280098a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:06:32.355 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.314+0000 7ff116ffd700 1 -- 172.21.15.195:0/365705355 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff12800eb70 con 0x7ff130154dd0 2022-06-27T12:06:32.355 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.314+0000 7ff116ffd700 1 -- 172.21.15.195:0/365705355 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff12800ecf0 con 0x7ff130154dd0 2022-06-27T12:06:32.356 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.314+0000 7ff116ffd700 1 -- 172.21.15.195:0/365705355 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff128014670 con 0x7ff130154dd0 2022-06-27T12:06:32.356 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.314+0000 7ff1359ba700 1 -- 172.21.15.195:0/365705355 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff13014b1a0 con 0x7ff130154dd0 2022-06-27T12:06:32.356 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.314+0000 7ff1359ba700 1 -- 172.21.15.195:0/365705355 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff13014beb0 con 0x7ff130154dd0 2022-06-27T12:06:32.356 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.315+0000 7ff1359ba700 1 -- 172.21.15.195:0/365705355 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff13014c070 con 0x7ff130154dd0 2022-06-27T12:06:32.357 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.315+0000 7ff116ffd700 1 -- 172.21.15.195:0/365705355 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7ff128012050 con 0x7ff130154dd0 2022-06-27T12:06:32.357 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.316+0000 7ff116ffd700 1 --2- 172.21.15.195:0/365705355 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff110063d00 0x7ff1100661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:32.357 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.316+0000 7ff116ffd700 1 -- 172.21.15.195:0/365705355 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6976+0+0 (secure 0 0 0) 0x7ff1280810b0 con 0x7ff130154dd0 2022-06-27T12:06:32.357 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.316+0000 7ff12effd700 1 --2- 172.21.15.195:0/365705355 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff110063d00 0x7ff1100661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:32.358 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.316+0000 7ff12effd700 1 --2- 172.21.15.195:0/365705355 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff110063d00 0x7ff1100661b0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7ff124009260 tx=0x7ff124017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:06:32.358 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.320+0000 7ff116ffd700 1 -- 172.21.15.195:0/365705355 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7ff12804f910 con 0x7ff130154dd0 2022-06-27T12:06:32.550 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.548+0000 7ff1359ba700 1 -- 172.21.15.195:0/365705355 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff13014bd10 con 0x7ff130154dd0 2022-06-27T12:06:32.551 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.548+0000 7ff116ffd700 1 -- 172.21.15.195:0/365705355 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+6128 (secure 0 0 0) 0x7ff13014bd10 con 0x7ff130154dd0 2022-06-27T12:06:32.552 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:06:32.552 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":5591,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/3022591968","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3022591968},{"type":"v1","addr":"172.21.15.202:6833","nonce":3022591968}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:05:40.259006+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5489},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5489":{"gid":5489,"name":"d","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.202:6835/2342195382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2342195382},{"type":"v1","addr":"172.21.15.202:6835","nonce":2342195382}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:06:32.555 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.550+0000 7ff1359ba700 1 -- 172.21.15.195:0/365705355 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff110063d00 msgr2=0x7ff1100661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:32.555 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.550+0000 7ff1359ba700 1 --2- 172.21.15.195:0/365705355 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff110063d00 0x7ff1100661b0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7ff124009260 tx=0x7ff124017000 comp rx=0 tx=0).stop 2022-06-27T12:06:32.555 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.550+0000 7ff1359ba700 1 -- 172.21.15.195:0/365705355 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff130154dd0 msgr2=0x7ff13014a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:32.556 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.550+0000 7ff1359ba700 1 --2- 172.21.15.195:0/365705355 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff130154dd0 0x7ff13014a010 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7ff128007f50 tx=0x7ff1280098a0 comp rx=0 tx=0).stop 2022-06-27T12:06:32.557 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.550+0000 7ff1359ba700 1 -- 172.21.15.195:0/365705355 shutdown_connections 2022-06-27T12:06:32.557 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.550+0000 7ff1359ba700 1 --2- 172.21.15.195:0/365705355 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff110063d00 0x7ff1100661b0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:32.558 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.550+0000 7ff1359ba700 1 --2- 172.21.15.195:0/365705355 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff130154dd0 0x7ff13014a010 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:32.558 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.550+0000 7ff1359ba700 1 --2- 172.21.15.195:0/365705355 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff1300c34b0 0x7ff130168180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:32.559 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.550+0000 7ff1359ba700 1 --2- 172.21.15.195:0/365705355 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff130154310 0x7ff13014d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:32.559 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.550+0000 7ff1359ba700 1 -- 172.21.15.195:0/365705355 >> 172.21.15.195:0/365705355 conn(0x7ff1300c8740 msgr2=0x7ff1300b79d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:06:32.559 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.550+0000 7ff1359ba700 1 -- 172.21.15.195:0/365705355 shutdown_connections 2022-06-27T12:06:32.559 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:32.550+0000 7ff1359ba700 1 -- 172.21.15.195:0/365705355 wait complete. 2022-06-27T12:06:32.560 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 66 2022-06-27T12:06:32.563 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-06-27T12:06:34.564 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:06:34.694 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.693+0000 7f3d8334c700 1 Processor -- start 2022-06-27T12:06:34.695 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.694+0000 7f3d8334c700 1 -- start start 2022-06-27T12:06:34.707 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.694+0000 7f3d8334c700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f3d7c06fef0 0x7f3d7c1570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:34.707 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.694+0000 7f3d8334c700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d7c06fba0 con 0x7f3d7c06e850 2022-06-27T12:06:34.708 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.694+0000 7f3d8334c700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d7c06f5e0 con 0x7f3d7c0c34a0 2022-06-27T12:06:34.708 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.694+0000 7f3d8334c700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d7c1575e0 con 0x7f3d7c06fef0 2022-06-27T12:06:34.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.695+0000 7f3d810e8700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f3d7c06e850 0x7f3d7c069de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:54560/0 (socket says 172.21.15.195:54560) 2022-06-27T12:06:34.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.695+0000 7f3d810e8700 1 -- 172.21.15.195:0/4270458184 learned_addr learned my addr 172.21.15.195:0/4270458184 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:06:34.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.695+0000 7f3d808e7700 1 --2- 172.21.15.195:0/4270458184 >> v2:172.21.15.202:3300/0 conn(0x7f3d7c06fef0 0x7f3d7c1570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:34.710 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.695+0000 7f3d6bfff700 1 -- 172.21.15.195:0/4270458184 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 556539589 0 0) 0x7f3d7c06fba0 con 0x7f3d7c06e850 2022-06-27T12:06:34.710 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.695+0000 7f3d6bfff700 1 -- 172.21.15.195:0/4270458184 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d64003430 con 0x7f3d7c06e850 2022-06-27T12:06:34.710 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.695+0000 7f3d6bfff700 1 -- 172.21.15.195:0/4270458184 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3966879227 0 0) 0x7f3d7c06f5e0 con 0x7f3d7c0c34a0 2022-06-27T12:06:34.710 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.695+0000 7f3d6bfff700 1 -- 172.21.15.195:0/4270458184 --> v1:172.21.15.202:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d7c06fba0 con 0x7f3d7c0c34a0 2022-06-27T12:06:34.711 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.695+0000 7f3d808e7700 1 -- 172.21.15.195:0/4270458184 >> v1:172.21.15.202:6789/0 conn(0x7f3d7c0c34a0 legacy=0x7f3d7c0c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:34.712 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.695+0000 7f3d808e7700 1 -- 172.21.15.195:0/4270458184 >> v1:172.21.15.195:6789/0 conn(0x7f3d7c06e850 legacy=0x7f3d7c069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:34.712 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.695+0000 7f3d808e7700 1 -- 172.21.15.195:0/4270458184 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d7c06f8c0 con 0x7f3d7c06fef0 2022-06-27T12:06:34.713 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.695+0000 7f3d808e7700 1 --2- 172.21.15.195:0/4270458184 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3d7c06fef0 0x7f3d7c1570a0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f3d780097a0 tx=0x7f3d78009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=103d1c58772bcd64 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:06:34.713 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.695+0000 7f3d6bfff700 1 -- 172.21.15.195:0/4270458184 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d78011630 con 0x7f3d7c06fef0 2022-06-27T12:06:34.714 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.695+0000 7f3d6bfff700 1 -- 172.21.15.195:0/4270458184 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d78011790 con 0x7f3d7c06fef0 2022-06-27T12:06:34.714 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.695+0000 7f3d6bfff700 1 -- 172.21.15.195:0/4270458184 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d78011b30 con 0x7f3d7c06fef0 2022-06-27T12:06:34.714 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.696+0000 7f3d8334c700 1 -- 172.21.15.195:0/4270458184 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3d7c06fef0 msgr2=0x7f3d7c1570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:34.715 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.696+0000 7f3d8334c700 1 --2- 172.21.15.195:0/4270458184 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3d7c06fef0 0x7f3d7c1570a0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f3d780097a0 tx=0x7f3d78009060 comp rx=0 tx=0).stop 2022-06-27T12:06:34.715 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.696+0000 7f3d8334c700 1 -- 172.21.15.195:0/4270458184 shutdown_connections 2022-06-27T12:06:34.715 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.696+0000 7f3d8334c700 1 --2- 172.21.15.195:0/4270458184 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3d7c06fef0 0x7f3d7c1570a0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:34.716 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.696+0000 7f3d8334c700 1 -- 172.21.15.195:0/4270458184 >> 172.21.15.195:0/4270458184 conn(0x7f3d7c0c6350 msgr2=0x7f3d7c0c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:06:34.716 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.696+0000 7f3d8334c700 1 -- 172.21.15.195:0/4270458184 shutdown_connections 2022-06-27T12:06:34.716 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.696+0000 7f3d8334c700 1 -- 172.21.15.195:0/4270458184 wait complete. 2022-06-27T12:06:34.717 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.697+0000 7f3d8334c700 1 Processor -- start 2022-06-27T12:06:34.717 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.697+0000 7f3d8334c700 1 -- start start 2022-06-27T12:06:34.717 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.698+0000 7f3d8334c700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3d7c06e850 0x7f3d7c167d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:34.718 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.698+0000 7f3d8334c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3d7c0c34a0 0x7f3d7c0c6900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:34.718 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.698+0000 7f3d8334c700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3d7c156850 0x7f3d7c14cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:34.718 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.698+0000 7f3d8334c700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d7c0b98c0 con 0x7f3d7c0c34a0 2022-06-27T12:06:34.719 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.698+0000 7f3d8334c700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d7c0c7e70 con 0x7f3d7c156850 2022-06-27T12:06:34.719 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.698+0000 7f3d8334c700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d7c14e930 con 0x7f3d7c06e850 2022-06-27T12:06:34.719 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.698+0000 7f3d808e7700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3d7c0c34a0 0x7f3d7c0c6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:34.720 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.698+0000 7f3d808e7700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3d7c0c34a0 0x7f3d7c0c6900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48380/0 (socket says 172.21.15.195:48380) 2022-06-27T12:06:34.720 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.698+0000 7f3d808e7700 1 -- 172.21.15.195:0/3715314098 learned_addr learned my addr 172.21.15.195:0/3715314098 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:06:34.721 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.698+0000 7f3d818e9700 1 --2- 172.21.15.195:0/3715314098 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3d7c156850 0x7f3d7c14cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:34.721 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.698+0000 7f3d810e8700 1 --2- 172.21.15.195:0/3715314098 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3d7c06e850 0x7f3d7c167d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:34.721 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.698+0000 7f3d808e7700 1 -- 172.21.15.195:0/3715314098 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3d7c06e850 msgr2=0x7f3d7c167d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:34.722 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.698+0000 7f3d808e7700 1 --2- 172.21.15.195:0/3715314098 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3d7c06e850 0x7f3d7c167d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:34.723 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.698+0000 7f3d808e7700 1 -- 172.21.15.195:0/3715314098 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3d7c156850 msgr2=0x7f3d7c14cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:34.723 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.698+0000 7f3d808e7700 1 --2- 172.21.15.195:0/3715314098 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3d7c156850 0x7f3d7c14cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:34.723 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.698+0000 7f3d808e7700 1 -- 172.21.15.195:0/3715314098 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d780092e0 con 0x7f3d7c0c34a0 2022-06-27T12:06:34.724 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.699+0000 7f3d808e7700 1 --2- 172.21.15.195:0/3715314098 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3d7c0c34a0 0x7f3d7c0c6900 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f3d78009d90 tx=0x7f3d78017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:06:34.724 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.699+0000 7f3d697fa700 1 -- 172.21.15.195:0/3715314098 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d78011900 con 0x7f3d7c0c34a0 2022-06-27T12:06:34.725 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.699+0000 7f3d697fa700 1 -- 172.21.15.195:0/3715314098 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d78011a60 con 0x7f3d7c0c34a0 2022-06-27T12:06:34.725 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.699+0000 7f3d8334c700 1 -- 172.21.15.195:0/3715314098 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d7c1678d0 con 0x7f3d7c0c34a0 2022-06-27T12:06:34.725 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.699+0000 7f3d697fa700 1 -- 172.21.15.195:0/3715314098 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d78023600 con 0x7f3d7c0c34a0 2022-06-27T12:06:34.726 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.699+0000 7f3d8334c700 1 -- 172.21.15.195:0/3715314098 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d7c0c7b90 con 0x7f3d7c0c34a0 2022-06-27T12:06:34.726 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.700+0000 7f3d8334c700 1 -- 172.21.15.195:0/3715314098 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d7c167a80 con 0x7f3d7c0c34a0 2022-06-27T12:06:34.727 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.700+0000 7f3d697fa700 1 -- 172.21.15.195:0/3715314098 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f3d78029020 con 0x7f3d7c0c34a0 2022-06-27T12:06:34.727 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.700+0000 7f3d697fa700 1 --2- 172.21.15.195:0/3715314098 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3d5c063d00 0x7f3d5c0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:34.727 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.700+0000 7f3d697fa700 1 -- 172.21.15.195:0/3715314098 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6976+0+0 (secure 0 0 0) 0x7f3d78081bb0 con 0x7f3d7c0c34a0 2022-06-27T12:06:34.728 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.701+0000 7f3d810e8700 1 --2- 172.21.15.195:0/3715314098 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3d5c063d00 0x7f3d5c0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:34.728 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.701+0000 7f3d810e8700 1 --2- 172.21.15.195:0/3715314098 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3d5c063d00 0x7f3d5c0661b0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f3d6c005790 tx=0x7f3d6c009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:06:34.728 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.705+0000 7f3d697fa700 1 -- 172.21.15.195:0/3715314098 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f3d7c167a80 con 0x7f3d7c0c34a0 2022-06-27T12:06:34.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.935+0000 7f3d8334c700 1 -- 172.21.15.195:0/3715314098 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d7c167a80 con 0x7f3d7c0c34a0 2022-06-27T12:06:34.937 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.936+0000 7f3d697fa700 1 -- 172.21.15.195:0/3715314098 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+6128 (secure 0 0 0) 0x7f3d7c167a80 con 0x7f3d7c0c34a0 2022-06-27T12:06:34.937 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:06:34.942 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":5591,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/3022591968","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3022591968},{"type":"v1","addr":"172.21.15.202:6833","nonce":3022591968}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:05:40.259006+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5489},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5489":{"gid":5489,"name":"d","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.202:6835/2342195382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2342195382},{"type":"v1","addr":"172.21.15.202:6835","nonce":2342195382}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:06:34.947 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.938+0000 7f3d8334c700 1 -- 172.21.15.195:0/3715314098 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3d5c063d00 msgr2=0x7f3d5c0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:34.948 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.938+0000 7f3d8334c700 1 --2- 172.21.15.195:0/3715314098 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3d5c063d00 0x7f3d5c0661b0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f3d6c005790 tx=0x7f3d6c009040 comp rx=0 tx=0).stop 2022-06-27T12:06:34.948 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.938+0000 7f3d8334c700 1 -- 172.21.15.195:0/3715314098 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3d7c0c34a0 msgr2=0x7f3d7c0c6900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:34.950 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.938+0000 7f3d8334c700 1 --2- 172.21.15.195:0/3715314098 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3d7c0c34a0 0x7f3d7c0c6900 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f3d78009d90 tx=0x7f3d78017000 comp rx=0 tx=0).stop 2022-06-27T12:06:34.950 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.938+0000 7f3d8334c700 1 -- 172.21.15.195:0/3715314098 shutdown_connections 2022-06-27T12:06:34.951 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.938+0000 7f3d8334c700 1 --2- 172.21.15.195:0/3715314098 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3d5c063d00 0x7f3d5c0661b0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:34.951 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.938+0000 7f3d8334c700 1 --2- 172.21.15.195:0/3715314098 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3d7c156850 0x7f3d7c14cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:34.952 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.938+0000 7f3d8334c700 1 --2- 172.21.15.195:0/3715314098 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3d7c06e850 0x7f3d7c167d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:34.952 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.938+0000 7f3d8334c700 1 --2- 172.21.15.195:0/3715314098 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3d7c0c34a0 0x7f3d7c0c6900 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:34.952 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.938+0000 7f3d8334c700 1 -- 172.21.15.195:0/3715314098 >> 172.21.15.195:0/3715314098 conn(0x7f3d7c0c6350 msgr2=0x7f3d7c150280 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:06:34.953 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.938+0000 7f3d8334c700 1 -- 172.21.15.195:0/3715314098 shutdown_connections 2022-06-27T12:06:34.953 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:34.938+0000 7f3d8334c700 1 -- 172.21.15.195:0/3715314098 wait complete. 2022-06-27T12:06:34.955 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 66 2022-06-27T12:06:34.958 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-06-27T12:06:36.960 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:06:37.083 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.083+0000 7ff61604d700 1 Processor -- start 2022-06-27T12:06:37.084 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.083+0000 7ff61604d700 1 -- start start 2022-06-27T12:06:37.100 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.084+0000 7ff61604d700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff6100c3030 0x7ff6101570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:37.100 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.084+0000 7ff61604d700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff61006fba0 con 0x7ff61006fef0 2022-06-27T12:06:37.100 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.084+0000 7ff61604d700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff61006f5e0 con 0x7ff61006e850 2022-06-27T12:06:37.101 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.084+0000 7ff61604d700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6101575e0 con 0x7ff6100c3030 2022-06-27T12:06:37.101 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.084+0000 7ff60effd700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7ff61006fef0 0x7ff6100c2920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:54580/0 (socket says 172.21.15.195:54580) 2022-06-27T12:06:37.101 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.084+0000 7ff60effd700 1 -- 172.21.15.195:0/1344096922 learned_addr learned my addr 172.21.15.195:0/1344096922 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:06:37.102 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.084+0000 7ff60ffff700 1 --2- 172.21.15.195:0/1344096922 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff6100c3030 0x7ff6101570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:37.102 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.084+0000 7ff60e7fc700 1 -- 172.21.15.195:0/1344096922 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2926842813 0 0) 0x7ff61006fba0 con 0x7ff61006fef0 2022-06-27T12:06:37.102 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.084+0000 7ff60e7fc700 1 -- 172.21.15.195:0/1344096922 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5fc003430 con 0x7ff61006fef0 2022-06-27T12:06:37.103 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.085+0000 7ff60e7fc700 1 -- 172.21.15.195:0/1344096922 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 916463790 0 0) 0x7ff61006f5e0 con 0x7ff61006e850 2022-06-27T12:06:37.103 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.085+0000 7ff60e7fc700 1 -- 172.21.15.195:0/1344096922 --> v1:172.21.15.202:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff61006fba0 con 0x7ff61006e850 2022-06-27T12:06:37.103 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.085+0000 7ff60e7fc700 1 -- 172.21.15.195:0/1344096922 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3431584305 0 0) 0x7ff5fc003430 con 0x7ff61006fef0 2022-06-27T12:06:37.103 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.085+0000 7ff60e7fc700 1 -- 172.21.15.195:0/1344096922 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff6100c3030 msgr2=0x7ff6101570a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:06:37.104 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.085+0000 7ff60e7fc700 1 --2- 172.21.15.195:0/1344096922 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff6100c3030 0x7ff6101570a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:37.110 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.085+0000 7ff60e7fc700 1 -- 172.21.15.195:0/1344096922 >> v1:172.21.15.202:6789/0 conn(0x7ff61006e850 legacy=0x7ff610069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:37.110 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.085+0000 7ff60e7fc700 1 -- 172.21.15.195:0/1344096922 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff61006f8c0 con 0x7ff61006fef0 2022-06-27T12:06:37.111 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.085+0000 7ff60e7fc700 1 -- 172.21.15.195:0/1344096922 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7ff600001470 con 0x7ff61006fef0 2022-06-27T12:06:37.111 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.085+0000 7ff60e7fc700 1 -- 172.21.15.195:0/1344096922 >> v1:172.21.15.195:6789/0 conn(0x7ff61006fef0 legacy=0x7ff6100c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:37.111 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.085+0000 7ff60e7fc700 1 --2- 172.21.15.195:0/1344096922 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff5fc003ae0 0x7ff5fc003ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:37.112 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.085+0000 7ff60e7fc700 1 --2- 172.21.15.195:0/1344096922 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff5fc0046d0 0x7ff5fc004b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:37.112 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.085+0000 7ff60e7fc700 1 --2- 172.21.15.195:0/1344096922 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff6100c3030 0x7ff5fc0050e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:37.113 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.085+0000 7ff60e7fc700 1 -- 172.21.15.195:0/1344096922 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6101575e0 con 0x7ff5fc0046d0 2022-06-27T12:06:37.113 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.085+0000 7ff60e7fc700 1 -- 172.21.15.195:0/1344096922 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5fc00bab0 con 0x7ff5fc003ae0 2022-06-27T12:06:37.114 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.085+0000 7ff60e7fc700 1 -- 172.21.15.195:0/1344096922 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5fc00bc70 con 0x7ff6100c3030 2022-06-27T12:06:37.114 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.085+0000 7ff60effd700 1 --2- 172.21.15.195:0/1344096922 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff5fc0046d0 0x7ff5fc004b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:37.114 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.085+0000 7ff60f7fe700 1 --2- 172.21.15.195:0/1344096922 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff5fc003ae0 0x7ff5fc003ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:37.115 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.085+0000 7ff60ffff700 1 --2- 172.21.15.195:0/1344096922 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff6100c3030 0x7ff5fc0050e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:37.115 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.085+0000 7ff60effd700 1 -- 172.21.15.195:0/1344096922 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff6100c3030 msgr2=0x7ff5fc0050e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:06:37.115 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.085+0000 7ff60effd700 1 --2- 172.21.15.195:0/1344096922 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff6100c3030 0x7ff5fc0050e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:37.116 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.085+0000 7ff60effd700 1 -- 172.21.15.195:0/1344096922 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff5fc003ae0 msgr2=0x7ff5fc003ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:06:37.116 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.085+0000 7ff60effd700 1 --2- 172.21.15.195:0/1344096922 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff5fc003ae0 0x7ff5fc003ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:37.116 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.085+0000 7ff60effd700 1 -- 172.21.15.195:0/1344096922 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5fc003430 con 0x7ff5fc0046d0 2022-06-27T12:06:37.116 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.085+0000 7ff60effd700 1 --2- 172.21.15.195:0/1344096922 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff5fc0046d0 0x7ff5fc004b40 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7ff600007fa0 tx=0x7ff600001d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a07019713d6edf6c server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:06:37.117 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.085+0000 7ff60e7fc700 1 -- 172.21.15.195:0/1344096922 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff60000ca80 con 0x7ff5fc0046d0 2022-06-27T12:06:37.117 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.085+0000 7ff60e7fc700 1 -- 172.21.15.195:0/1344096922 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff60000cbe0 con 0x7ff5fc0046d0 2022-06-27T12:06:37.117 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.085+0000 7ff60e7fc700 1 -- 172.21.15.195:0/1344096922 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6000070d0 con 0x7ff5fc0046d0 2022-06-27T12:06:37.117 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.086+0000 7ff61604d700 1 -- 172.21.15.195:0/1344096922 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff5fc0046d0 msgr2=0x7ff5fc004b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:37.118 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.086+0000 7ff61604d700 1 --2- 172.21.15.195:0/1344096922 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff5fc0046d0 0x7ff5fc004b40 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7ff600007fa0 tx=0x7ff600001d20 comp rx=0 tx=0).stop 2022-06-27T12:06:37.118 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.086+0000 7ff60ffff700 1 -- 172.21.15.195:0/1344096922 reap_dead start 2022-06-27T12:06:37.118 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.086+0000 7ff61604d700 1 -- 172.21.15.195:0/1344096922 shutdown_connections 2022-06-27T12:06:37.118 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.086+0000 7ff61604d700 1 -- 172.21.15.195:0/1344096922 >> 172.21.15.195:0/1344096922 conn(0x7ff6100c6350 msgr2=0x7ff6100c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:06:37.119 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.086+0000 7ff61604d700 1 -- 172.21.15.195:0/1344096922 shutdown_connections 2022-06-27T12:06:37.119 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.086+0000 7ff61604d700 1 -- 172.21.15.195:0/1344096922 wait complete. 2022-06-27T12:06:37.119 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.087+0000 7ff61604d700 1 Processor -- start 2022-06-27T12:06:37.119 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.087+0000 7ff61604d700 1 -- start start 2022-06-27T12:06:37.120 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.088+0000 7ff61604d700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff5fc0046d0 0x7ff610069ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:37.120 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.088+0000 7ff61604d700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff610150770 0x7ff6100b6570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:37.120 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.088+0000 7ff61604d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff6100b6ab0 0x7ff61014e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:37.120 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.088+0000 7ff61604d700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6101675a0 con 0x7ff6100b6ab0 2022-06-27T12:06:37.121 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.088+0000 7ff61604d700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6100b60e0 con 0x7ff610150770 2022-06-27T12:06:37.121 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.088+0000 7ff61604d700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6100b6f20 con 0x7ff5fc0046d0 2022-06-27T12:06:37.121 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.088+0000 7ff60ffff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff6100b6ab0 0x7ff61014e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:37.121 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.088+0000 7ff60f7fe700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff5fc0046d0 0x7ff610069ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:37.121 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.088+0000 7ff60ffff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff6100b6ab0 0x7ff61014e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48404/0 (socket says 172.21.15.195:48404) 2022-06-27T12:06:37.122 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.088+0000 7ff60ffff700 1 -- 172.21.15.195:0/3215865149 learned_addr learned my addr 172.21.15.195:0/3215865149 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:06:37.122 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.088+0000 7ff60effd700 1 --2- 172.21.15.195:0/3215865149 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff610150770 0x7ff6100b6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:37.122 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.088+0000 7ff60ffff700 1 -- 172.21.15.195:0/3215865149 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff5fc0046d0 msgr2=0x7ff610069ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:37.122 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.088+0000 7ff60ffff700 1 --2- 172.21.15.195:0/3215865149 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff5fc0046d0 0x7ff610069ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:37.123 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.088+0000 7ff60ffff700 1 -- 172.21.15.195:0/3215865149 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff610150770 msgr2=0x7ff6100b6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:37.123 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.088+0000 7ff60ffff700 1 --2- 172.21.15.195:0/3215865149 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff610150770 0x7ff6100b6570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:37.123 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.088+0000 7ff60ffff700 1 -- 172.21.15.195:0/3215865149 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff610166a20 con 0x7ff6100b6ab0 2022-06-27T12:06:37.123 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.089+0000 7ff60ffff700 1 --2- 172.21.15.195:0/3215865149 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff6100b6ab0 0x7ff61014e120 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7ff6080094d0 tx=0x7ff60800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:06:37.124 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.089+0000 7ff5f77fe700 1 -- 172.21.15.195:0/3215865149 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff60800e040 con 0x7ff6100b6ab0 2022-06-27T12:06:37.124 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.089+0000 7ff5f77fe700 1 -- 172.21.15.195:0/3215865149 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff60800f040 con 0x7ff6100b6ab0 2022-06-27T12:06:37.124 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.089+0000 7ff61604d700 1 -- 172.21.15.195:0/3215865149 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff610166d00 con 0x7ff6100b6ab0 2022-06-27T12:06:37.124 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.089+0000 7ff5f77fe700 1 -- 172.21.15.195:0/3215865149 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff608011ee0 con 0x7ff6100b6ab0 2022-06-27T12:06:37.125 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.089+0000 7ff61604d700 1 -- 172.21.15.195:0/3215865149 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6101672c0 con 0x7ff6100b6ab0 2022-06-27T12:06:37.125 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.090+0000 7ff61604d700 1 -- 172.21.15.195:0/3215865149 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6100722c0 con 0x7ff6100b6ab0 2022-06-27T12:06:37.125 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.090+0000 7ff5f77fe700 1 -- 172.21.15.195:0/3215865149 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7ff608022030 con 0x7ff6100b6ab0 2022-06-27T12:06:37.125 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.091+0000 7ff5f77fe700 1 --2- 172.21.15.195:0/3215865149 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff5f0063d00 0x7ff5f00661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:37.126 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.091+0000 7ff60f7fe700 1 --2- 172.21.15.195:0/3215865149 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff5f0063d00 0x7ff5f00661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:37.126 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.091+0000 7ff5f77fe700 1 -- 172.21.15.195:0/3215865149 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7062+0+0 (secure 0 0 0) 0x7ff608082b90 con 0x7ff6100b6ab0 2022-06-27T12:06:37.126 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.091+0000 7ff60f7fe700 1 --2- 172.21.15.195:0/3215865149 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff5f0063d00 0x7ff5f00661b0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7ff604000c00 tx=0x7ff604009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:06:37.127 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.095+0000 7ff5f77fe700 1 -- 172.21.15.195:0/3215865149 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7ff608051390 con 0x7ff6100b6ab0 2022-06-27T12:06:37.319 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.319+0000 7ff61604d700 1 -- 172.21.15.195:0/3215865149 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff61006f550 con 0x7ff6100b6ab0 2022-06-27T12:06:37.320 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.319+0000 7ff5f77fe700 1 -- 172.21.15.195:0/3215865149 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+5367 (secure 0 0 0) 0x7ff608020020 con 0x7ff6100b6ab0 2022-06-27T12:06:37.320 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:06:37.321 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:06:35.081018+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5591},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5591":{"gid":5591,"name":"b","rank":0,"incarnation":67,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6833/3022591968","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3022591968},{"type":"v1","addr":"172.21.15.202:6833","nonce":3022591968}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:06:37.322 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.321+0000 7ff61604d700 1 -- 172.21.15.195:0/3215865149 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff5f0063d00 msgr2=0x7ff5f00661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:37.323 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.321+0000 7ff61604d700 1 --2- 172.21.15.195:0/3215865149 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff5f0063d00 0x7ff5f00661b0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7ff604000c00 tx=0x7ff604009040 comp rx=0 tx=0).stop 2022-06-27T12:06:37.323 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.321+0000 7ff61604d700 1 -- 172.21.15.195:0/3215865149 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff6100b6ab0 msgr2=0x7ff61014e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:37.323 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.321+0000 7ff61604d700 1 --2- 172.21.15.195:0/3215865149 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff6100b6ab0 0x7ff61014e120 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7ff6080094d0 tx=0x7ff60800b040 comp rx=0 tx=0).stop 2022-06-27T12:06:37.323 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.321+0000 7ff61604d700 1 -- 172.21.15.195:0/3215865149 shutdown_connections 2022-06-27T12:06:37.324 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.321+0000 7ff61604d700 1 --2- 172.21.15.195:0/3215865149 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff5f0063d00 0x7ff5f00661b0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:37.324 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.321+0000 7ff61604d700 1 --2- 172.21.15.195:0/3215865149 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff6100b6ab0 0x7ff61014e120 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:37.324 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.321+0000 7ff61604d700 1 --2- 172.21.15.195:0/3215865149 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff5fc0046d0 0x7ff610069ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:37.324 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.321+0000 7ff61604d700 1 --2- 172.21.15.195:0/3215865149 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff610150770 0x7ff6100b6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:37.324 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.321+0000 7ff61604d700 1 -- 172.21.15.195:0/3215865149 >> 172.21.15.195:0/3215865149 conn(0x7ff6100c6350 msgr2=0x7ff610153020 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:06:37.325 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.321+0000 7ff61604d700 1 -- 172.21.15.195:0/3215865149 shutdown_connections 2022-06-27T12:06:37.325 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.321+0000 7ff61604d700 1 -- 172.21.15.195:0/3215865149 wait complete. 2022-06-27T12:06:37.325 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 67 2022-06-27T12:06:37.335 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d down, removed from mdsmap 2022-06-27T12:06:37.335 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2022-06-27T12:06:37.336 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:06:37.456 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.456+0000 7f77bf5e5700 1 Processor -- start 2022-06-27T12:06:37.456 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.456+0000 7f77bf5e5700 1 -- start start 2022-06-27T12:06:37.459 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.457+0000 7f77bf5e5700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f77b806a450 0x7f77b8063af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:37.460 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.457+0000 7f77bf5e5700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f77b81508d0 0x7f77b80bbf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:37.460 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.457+0000 7f77bf5e5700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77b8064030 con 0x7f77b8150500 2022-06-27T12:06:37.460 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.457+0000 7f77bf5e5700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77b80c4160 con 0x7f77b81508d0 2022-06-27T12:06:37.460 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.457+0000 7f77bf5e5700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77b80c4440 con 0x7f77b806a450 2022-06-27T12:06:37.461 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.457+0000 7f77bcb80700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f77b806a450 0x7f77b8063af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:37.461 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.457+0000 7f77bdb82700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f77b81508d0 0x7f77b80bbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:37.461 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.457+0000 7f77bdb82700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f77b81508d0 0x7f77b80bbf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3300/0 says I am v2:172.21.15.195:46538/0 (socket says 172.21.15.195:46538) 2022-06-27T12:06:37.461 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.457+0000 7f77bcb80700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f77b806a450 0x7f77b8063af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3301/0 says I am v2:172.21.15.195:41166/0 (socket says 172.21.15.195:41166) 2022-06-27T12:06:37.462 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.457+0000 7f77bdb82700 1 -- 172.21.15.195:0/140015591 learned_addr learned my addr 172.21.15.195:0/140015591 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:06:37.462 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.457+0000 7f77bdb82700 1 -- 172.21.15.195:0/140015591 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f77b806a450 msgr2=0x7f77b8063af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:06:37.462 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.457+0000 7f77bdb82700 1 --2- 172.21.15.195:0/140015591 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f77b806a450 0x7f77b8063af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:37.462 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.457+0000 7f77bdb82700 1 -- 172.21.15.195:0/140015591 >> v1:172.21.15.202:6789/0 conn(0x7f77b8150500 legacy=0x7f77b80c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:37.463 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.457+0000 7f77bdb82700 1 -- 172.21.15.195:0/140015591 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77b815a800 con 0x7f77b81508d0 2022-06-27T12:06:37.463 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.457+0000 7f77bcb80700 1 --2- 172.21.15.195:0/140015591 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f77b806a450 0x7f77b8063af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:06:37.463 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.458+0000 7f77bdb82700 1 --2- 172.21.15.195:0/140015591 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f77b81508d0 0x7f77b80bbf50 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f77b0009f20 tx=0x7f77b0009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f4f3b35f833f8580 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:06:37.464 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.458+0000 7f77a7fff700 1 -- 172.21.15.195:0/140015591 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77b000e070 con 0x7f77b81508d0 2022-06-27T12:06:37.464 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.458+0000 7f77a7fff700 1 -- 172.21.15.195:0/140015591 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77b000b9d0 con 0x7f77b81508d0 2022-06-27T12:06:37.465 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.458+0000 7f77a7fff700 1 -- 172.21.15.195:0/140015591 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77b00092e0 con 0x7f77b81508d0 2022-06-27T12:06:37.465 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.458+0000 7f77bf5e5700 1 -- 172.21.15.195:0/140015591 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f77b81508d0 msgr2=0x7f77b80bbf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:37.465 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.458+0000 7f77bf5e5700 1 --2- 172.21.15.195:0/140015591 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f77b81508d0 0x7f77b80bbf50 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f77b0009f20 tx=0x7f77b0009840 comp rx=0 tx=0).stop 2022-06-27T12:06:37.465 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.458+0000 7f77bf5e5700 1 -- 172.21.15.195:0/140015591 shutdown_connections 2022-06-27T12:06:37.466 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.458+0000 7f77bf5e5700 1 --2- 172.21.15.195:0/140015591 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f77b81508d0 0x7f77b80bbf50 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:37.466 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.458+0000 7f77bf5e5700 1 --2- 172.21.15.195:0/140015591 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f77b806a450 0x7f77b8063af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:37.466 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.458+0000 7f77bf5e5700 1 -- 172.21.15.195:0/140015591 >> 172.21.15.195:0/140015591 conn(0x7f77b80c9b00 msgr2=0x7f77b80c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:06:37.467 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.458+0000 7f77bf5e5700 1 -- 172.21.15.195:0/140015591 shutdown_connections 2022-06-27T12:06:37.467 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.458+0000 7f77bf5e5700 1 -- 172.21.15.195:0/140015591 wait complete. 2022-06-27T12:06:37.467 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.459+0000 7f77bf5e5700 1 Processor -- start 2022-06-27T12:06:37.467 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.459+0000 7f77bf5e5700 1 -- start start 2022-06-27T12:06:37.471 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.460+0000 7f77bf5e5700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f77b806a450 0x7f77b80c1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:37.471 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.460+0000 7f77bf5e5700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f77b8150500 0x7f77b8158af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:37.471 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.460+0000 7f77bf5e5700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f77b81508d0 0x7f77b8159030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:37.472 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.460+0000 7f77bf5e5700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77b80bd830 con 0x7f77b8150500 2022-06-27T12:06:37.472 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.460+0000 7f77bf5e5700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77b80bce20 con 0x7f77b81508d0 2022-06-27T12:06:37.472 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.460+0000 7f77bf5e5700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77b80c7940 con 0x7f77b806a450 2022-06-27T12:06:37.472 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.460+0000 7f77bcb80700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f77b8150500 0x7f77b8158af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:37.473 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.461+0000 7f77bcb80700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f77b8150500 0x7f77b8158af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48416/0 (socket says 172.21.15.195:48416) 2022-06-27T12:06:37.473 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.461+0000 7f77bcb80700 1 -- 172.21.15.195:0/4234529342 learned_addr learned my addr 172.21.15.195:0/4234529342 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:06:37.473 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.461+0000 7f77bd381700 1 --2- 172.21.15.195:0/4234529342 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f77b806a450 0x7f77b80c1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:37.473 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.461+0000 7f77bdb82700 1 --2- 172.21.15.195:0/4234529342 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f77b81508d0 0x7f77b8159030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:37.474 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.461+0000 7f77bcb80700 1 -- 172.21.15.195:0/4234529342 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f77b806a450 msgr2=0x7f77b80c1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:37.474 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.461+0000 7f77bcb80700 1 --2- 172.21.15.195:0/4234529342 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f77b806a450 0x7f77b80c1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:37.475 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.461+0000 7f77bcb80700 1 -- 172.21.15.195:0/4234529342 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f77b81508d0 msgr2=0x7f77b8159030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:37.475 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.461+0000 7f77bcb80700 1 --2- 172.21.15.195:0/4234529342 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f77b81508d0 0x7f77b8159030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:37.475 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.461+0000 7f77bcb80700 1 -- 172.21.15.195:0/4234529342 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77a80092e0 con 0x7f77b8150500 2022-06-27T12:06:37.475 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.461+0000 7f77bdb82700 1 --2- 172.21.15.195:0/4234529342 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f77b81508d0 0x7f77b8159030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:06:37.476 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.461+0000 7f77bcb80700 1 --2- 172.21.15.195:0/4234529342 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f77b8150500 0x7f77b8158af0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f77a800ea80 tx=0x7f77a8009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:06:37.476 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.461+0000 7f77a57fa700 1 -- 172.21.15.195:0/4234529342 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a80095a0 con 0x7f77b8150500 2022-06-27T12:06:37.476 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.461+0000 7f77a57fa700 1 -- 172.21.15.195:0/4234529342 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77a800c850 con 0x7f77b8150500 2022-06-27T12:06:37.477 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.461+0000 7f77bf5e5700 1 -- 172.21.15.195:0/4234529342 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77b0009ae0 con 0x7f77b8150500 2022-06-27T12:06:37.477 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.461+0000 7f77a57fa700 1 -- 172.21.15.195:0/4234529342 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a800cbf0 con 0x7f77b8150500 2022-06-27T12:06:37.477 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.461+0000 7f77bf5e5700 1 -- 172.21.15.195:0/4234529342 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77b8068440 con 0x7f77b8150500 2022-06-27T12:06:37.477 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.462+0000 7f77bf5e5700 1 -- 172.21.15.195:0/4234529342 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77b80bcf90 con 0x7f77b8150500 2022-06-27T12:06:37.478 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.463+0000 7f77a57fa700 1 -- 172.21.15.195:0/4234529342 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f77a8019030 con 0x7f77b8150500 2022-06-27T12:06:37.478 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.463+0000 7f77a57fa700 1 --2- 172.21.15.195:0/4234529342 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7798063d00 0x7f77980661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:37.478 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.463+0000 7f77a57fa700 1 -- 172.21.15.195:0/4234529342 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7062+0+0 (secure 0 0 0) 0x7f77a8080be0 con 0x7f77b8150500 2022-06-27T12:06:37.478 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.463+0000 7f77bd381700 1 --2- 172.21.15.195:0/4234529342 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7798063d00 0x7f77980661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:37.479 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.463+0000 7f77bd381700 1 --2- 172.21.15.195:0/4234529342 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7798063d00 0x7f77980661b0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f77b4000c00 tx=0x7f77b4009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:06:37.479 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.467+0000 7f77a57fa700 1 -- 172.21.15.195:0/4234529342 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f77b80bcf90 con 0x7f77b8150500 2022-06-27T12:06:37.693 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.693+0000 7f77bf5e5700 1 -- 172.21.15.195:0/4234529342 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f77b80bcf90 con 0x7f77b8150500 2022-06-27T12:06:37.693 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.693+0000 7f77a57fa700 1 -- 172.21.15.195:0/4234529342 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+5367 (secure 0 0 0) 0x7f77b80bcf90 con 0x7f77b8150500 2022-06-27T12:06:37.694 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:06:37.694 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:06:35.081018+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5591},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5591":{"gid":5591,"name":"b","rank":0,"incarnation":67,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6833/3022591968","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3022591968},{"type":"v1","addr":"172.21.15.202:6833","nonce":3022591968}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:06:37.696 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.695+0000 7f77bf5e5700 1 -- 172.21.15.195:0/4234529342 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7798063d00 msgr2=0x7f77980661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:37.696 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.695+0000 7f77bf5e5700 1 --2- 172.21.15.195:0/4234529342 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7798063d00 0x7f77980661b0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f77b4000c00 tx=0x7f77b4009040 comp rx=0 tx=0).stop 2022-06-27T12:06:37.696 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.695+0000 7f77bf5e5700 1 -- 172.21.15.195:0/4234529342 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f77b8150500 msgr2=0x7f77b8158af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:37.696 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.695+0000 7f77bf5e5700 1 --2- 172.21.15.195:0/4234529342 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f77b8150500 0x7f77b8158af0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f77a800ea80 tx=0x7f77a8009060 comp rx=0 tx=0).stop 2022-06-27T12:06:37.697 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.695+0000 7f77bf5e5700 1 -- 172.21.15.195:0/4234529342 shutdown_connections 2022-06-27T12:06:37.697 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.695+0000 7f77bf5e5700 1 --2- 172.21.15.195:0/4234529342 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7798063d00 0x7f77980661b0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:37.697 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.695+0000 7f77bf5e5700 1 --2- 172.21.15.195:0/4234529342 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f77b81508d0 0x7f77b8159030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:37.697 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.695+0000 7f77bf5e5700 1 --2- 172.21.15.195:0/4234529342 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f77b806a450 0x7f77b80c1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:37.697 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.695+0000 7f77bf5e5700 1 --2- 172.21.15.195:0/4234529342 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f77b8150500 0x7f77b8158af0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:37.698 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.695+0000 7f77bf5e5700 1 -- 172.21.15.195:0/4234529342 >> 172.21.15.195:0/4234529342 conn(0x7f77b80c9b00 msgr2=0x7f77b814fd60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:06:37.698 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.696+0000 7f77bf5e5700 1 -- 172.21.15.195:0/4234529342 shutdown_connections 2022-06-27T12:06:37.698 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:37.696+0000 7f77bf5e5700 1 -- 172.21.15.195:0/4234529342 wait complete. 2022-06-27T12:06:37.699 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 67 2022-06-27T12:06:37.709 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 67, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-06-27T11:47:52.358095+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 67, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5591": { "addr": "172.21.15.202:6833/3022591968", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6832", "nonce": 3022591968, "type": "v2" }, { "addr": "172.21.15.202:6833", "nonce": 3022591968, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5591, "incarnation": 67, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 58, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-06-27T12:06:35.081018+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5591 } } } ], "standbys": [ { "addr": "172.21.15.202:6837/3305006841", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6836", "nonce": 3305006841, "type": "v2" }, { "addr": "172.21.15.202:6837", "nonce": 3305006841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6835/2060709822", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6834", "nonce": 2060709822, "type": "v2" }, { "addr": "172.21.15.195:6835", "nonce": 2060709822, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6838/1598751652", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6836", "nonce": 1598751652, "type": "v2" }, { "addr": "172.21.15.195:6838", "nonce": 1598751652, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4275, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6839/1313800661", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6837", "nonce": 1313800661, "type": "v2" }, { "addr": "172.21.15.195:6839", "nonce": 1313800661, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4278, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-06-27T12:06:39.711 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:06:39.860 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.844+0000 7f75bc3bf700 1 Processor -- start 2022-06-27T12:06:39.861 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.844+0000 7f75bc3bf700 1 -- start start 2022-06-27T12:06:39.861 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.845+0000 7f75bc3bf700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f75b414e490 0x7f75b406cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:39.863 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.845+0000 7f75bc3bf700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f75b414e860 0x7f75b40c3840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:39.863 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.845+0000 7f75bc3bf700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75b40c33b0 con 0x7f75b40c3d80 2022-06-27T12:06:39.864 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.845+0000 7f75bc3bf700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75b406d520 con 0x7f75b414e490 2022-06-27T12:06:39.864 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.845+0000 7f75bc3bf700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75b406f220 con 0x7f75b414e860 2022-06-27T12:06:39.865 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.845+0000 7f75ba95c700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f75b40c3d80 0x7f75b406f620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:54612/0 (socket says 172.21.15.195:54612) 2022-06-27T12:06:39.865 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.845+0000 7f75ba95c700 1 -- 172.21.15.195:0/1782257770 learned_addr learned my addr 172.21.15.195:0/1782257770 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:06:39.865 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.845+0000 7f75ba15b700 1 --2- 172.21.15.195:0/1782257770 >> v2:172.21.15.202:3300/0 conn(0x7f75b414e490 0x7f75b406cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:39.866 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.845+0000 7f75b995a700 1 --2- 172.21.15.195:0/1782257770 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f75b414e860 0x7f75b40c3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:39.866 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.845+0000 7f75b9159700 1 -- 172.21.15.195:0/1782257770 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 972372141 0 0) 0x7f75b40c33b0 con 0x7f75b40c3d80 2022-06-27T12:06:39.867 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.845+0000 7f75b9159700 1 -- 172.21.15.195:0/1782257770 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f759c003430 con 0x7f75b40c3d80 2022-06-27T12:06:39.867 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.845+0000 7f75b9159700 1 -- 172.21.15.195:0/1782257770 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3626233164 0 0) 0x7f759c003430 con 0x7f75b40c3d80 2022-06-27T12:06:39.868 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.845+0000 7f75b9159700 1 -- 172.21.15.195:0/1782257770 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f75b414e860 msgr2=0x7f75b40c3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:06:39.868 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.845+0000 7f75b9159700 1 --2- 172.21.15.195:0/1782257770 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f75b414e860 0x7f75b40c3840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:39.868 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.845+0000 7f75b9159700 1 -- 172.21.15.195:0/1782257770 >> v2:172.21.15.202:3300/0 conn(0x7f75b414e490 msgr2=0x7f75b406cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:06:39.868 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.845+0000 7f75b9159700 1 --2- 172.21.15.195:0/1782257770 >> v2:172.21.15.202:3300/0 conn(0x7f75b414e490 0x7f75b406cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:39.869 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.845+0000 7f75b9159700 1 -- 172.21.15.195:0/1782257770 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75b41542c0 con 0x7f75b40c3d80 2022-06-27T12:06:39.869 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.845+0000 7f75b9159700 1 -- 172.21.15.195:0/1782257770 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7f75ac0017f0 con 0x7f75b40c3d80 2022-06-27T12:06:39.869 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.846+0000 7f75b9159700 1 -- 172.21.15.195:0/1782257770 >> v1:172.21.15.195:6789/0 conn(0x7f75b40c3d80 legacy=0x7f75b406f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:39.869 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.846+0000 7f75b9159700 1 --2- 172.21.15.195:0/1782257770 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f759c003920 0x7f759c003d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:39.870 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.846+0000 7f75b9159700 1 --2- 172.21.15.195:0/1782257770 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f75b414e860 0x7f759c0046d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:39.870 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.846+0000 7f75b9159700 1 --2- 172.21.15.195:0/1782257770 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f759c004c10 0x7f759c005080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:39.870 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.846+0000 7f75b9159700 1 -- 172.21.15.195:0/1782257770 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75b406d520 con 0x7f759c003920 2022-06-27T12:06:39.870 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.846+0000 7f75b9159700 1 -- 172.21.15.195:0/1782257770 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75b406f220 con 0x7f759c004c10 2022-06-27T12:06:39.871 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.846+0000 7f75b9159700 1 -- 172.21.15.195:0/1782257770 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f759c00b920 con 0x7f75b414e860 2022-06-27T12:06:39.871 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.846+0000 7f75ba15b700 1 --2- 172.21.15.195:0/1782257770 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f759c003920 0x7f759c003d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:39.871 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.846+0000 7f75b995a700 1 --2- 172.21.15.195:0/1782257770 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f75b414e860 0x7f759c0046d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:39.871 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.846+0000 7f75ba95c700 1 --2- 172.21.15.195:0/1782257770 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f759c004c10 0x7f759c005080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:39.871 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.846+0000 7f75ba15b700 1 -- 172.21.15.195:0/1782257770 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f75b414e860 msgr2=0x7f759c0046d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:06:39.872 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.846+0000 7f75ba15b700 1 --2- 172.21.15.195:0/1782257770 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f75b414e860 0x7f759c0046d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:39.872 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.846+0000 7f75ba15b700 1 -- 172.21.15.195:0/1782257770 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f759c004c10 msgr2=0x7f759c005080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:06:39.872 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.846+0000 7f75ba15b700 1 --2- 172.21.15.195:0/1782257770 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f759c004c10 0x7f759c005080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:39.873 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.846+0000 7f75ba15b700 1 -- 172.21.15.195:0/1782257770 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f759c003430 con 0x7f759c003920 2022-06-27T12:06:39.873 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.846+0000 7f75ba95c700 1 --2- 172.21.15.195:0/1782257770 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f759c004c10 0x7f759c005080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:06:39.873 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.846+0000 7f75ba15b700 1 --2- 172.21.15.195:0/1782257770 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f759c003920 0x7f759c003d30 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f75b0009ac0 tx=0x7f75b0009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87ff6ee605981787 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:06:39.873 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.846+0000 7f75b9159700 1 -- 172.21.15.195:0/1782257770 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b0009470 con 0x7f759c003920 2022-06-27T12:06:39.874 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.846+0000 7f75b9159700 1 -- 172.21.15.195:0/1782257770 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75b000e5e0 con 0x7f759c003920 2022-06-27T12:06:39.874 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.846+0000 7f75b9159700 1 -- 172.21.15.195:0/1782257770 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b000e980 con 0x7f759c003920 2022-06-27T12:06:39.874 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.847+0000 7f75bc3bf700 1 -- 172.21.15.195:0/1782257770 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f759c003920 msgr2=0x7f759c003d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:39.874 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.847+0000 7f75bc3bf700 1 --2- 172.21.15.195:0/1782257770 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f759c003920 0x7f759c003d30 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f75b0009ac0 tx=0x7f75b0009060 comp rx=0 tx=0).stop 2022-06-27T12:06:39.874 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.847+0000 7f75ba95c700 1 -- 172.21.15.195:0/1782257770 reap_dead start 2022-06-27T12:06:39.875 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.847+0000 7f75bc3bf700 1 -- 172.21.15.195:0/1782257770 shutdown_connections 2022-06-27T12:06:39.875 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.847+0000 7f75bc3bf700 1 -- 172.21.15.195:0/1782257770 >> 172.21.15.195:0/1782257770 conn(0x7f75b4063af0 msgr2=0x7f75b4063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:06:39.875 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.847+0000 7f75bc3bf700 1 -- 172.21.15.195:0/1782257770 shutdown_connections 2022-06-27T12:06:39.875 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.847+0000 7f75bc3bf700 1 -- 172.21.15.195:0/1782257770 wait complete. 2022-06-27T12:06:39.876 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.848+0000 7f75bc3bf700 1 Processor -- start 2022-06-27T12:06:39.876 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.848+0000 7f75bc3bf700 1 -- start start 2022-06-27T12:06:39.876 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.849+0000 7f75bc3bf700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f759c003920 0x7f75b40c90e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:39.876 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.849+0000 7f75bc3bf700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f75b4145b90 0x7f75b4146000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:39.877 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.849+0000 7f75bc3bf700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f75b416a1a0 0x7f75b416a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:39.877 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.849+0000 7f75bc3bf700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75b414ab40 con 0x7f75b416a1a0 2022-06-27T12:06:39.877 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.849+0000 7f75bc3bf700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75b4146540 con 0x7f759c003920 2022-06-27T12:06:39.877 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.849+0000 7f75bc3bf700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75b4069110 con 0x7f75b4145b90 2022-06-27T12:06:39.878 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.849+0000 7f75ba95c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f75b416a1a0 0x7f75b416a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:39.878 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.849+0000 7f75ba95c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f75b416a1a0 0x7f75b416a610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48436/0 (socket says 172.21.15.195:48436) 2022-06-27T12:06:39.879 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.849+0000 7f75ba95c700 1 -- 172.21.15.195:0/2151390977 learned_addr learned my addr 172.21.15.195:0/2151390977 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:06:39.879 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.849+0000 7f75ba15b700 1 --2- 172.21.15.195:0/2151390977 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f759c003920 0x7f75b40c90e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:39.879 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.849+0000 7f75b995a700 1 --2- 172.21.15.195:0/2151390977 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f75b4145b90 0x7f75b4146000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:39.880 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.849+0000 7f75ba95c700 1 -- 172.21.15.195:0/2151390977 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f75b4145b90 msgr2=0x7f75b4146000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:39.880 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.849+0000 7f75ba95c700 1 --2- 172.21.15.195:0/2151390977 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f75b4145b90 0x7f75b4146000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:39.881 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.849+0000 7f75ba95c700 1 -- 172.21.15.195:0/2151390977 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f759c003920 msgr2=0x7f75b40c90e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:39.881 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.849+0000 7f75ba95c700 1 --2- 172.21.15.195:0/2151390977 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f759c003920 0x7f75b40c90e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:39.881 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.849+0000 7f75ba95c700 1 -- 172.21.15.195:0/2151390977 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75b40693f0 con 0x7f75b416a1a0 2022-06-27T12:06:39.882 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.849+0000 7f75ba95c700 1 --2- 172.21.15.195:0/2151390977 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f75b416a1a0 0x7f75b416a610 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f75ac000f80 tx=0x7f75b40c3220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:06:39.883 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.849+0000 7f75a27fc700 1 -- 172.21.15.195:0/2151390977 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ac00c750 con 0x7f75b416a1a0 2022-06-27T12:06:39.883 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.850+0000 7f75a27fc700 1 -- 172.21.15.195:0/2151390977 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75ac00c8b0 con 0x7f75b416a1a0 2022-06-27T12:06:39.883 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.850+0000 7f75bc3bf700 1 -- 172.21.15.195:0/2151390977 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75b40696d0 con 0x7f75b416a1a0 2022-06-27T12:06:39.884 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.850+0000 7f75a27fc700 1 -- 172.21.15.195:0/2151390977 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ac00cc50 con 0x7f75b416a1a0 2022-06-27T12:06:39.885 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.850+0000 7f75bc3bf700 1 -- 172.21.15.195:0/2151390977 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75b4069c90 con 0x7f75b416a1a0 2022-06-27T12:06:39.886 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.851+0000 7f75bc3bf700 1 -- 172.21.15.195:0/2151390977 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75b40722c0 con 0x7f75b416a1a0 2022-06-27T12:06:39.886 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.851+0000 7f75a27fc700 1 -- 172.21.15.195:0/2151390977 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f75ac00cdb0 con 0x7f75b416a1a0 2022-06-27T12:06:39.887 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.851+0000 7f75a27fc700 1 --2- 172.21.15.195:0/2151390977 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7594063d30 0x7f75940661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:39.887 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.852+0000 7f75a27fc700 1 -- 172.21.15.195:0/2151390977 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7062+0+0 (secure 0 0 0) 0x7f75ac080ca0 con 0x7f75b416a1a0 2022-06-27T12:06:39.887 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.852+0000 7f75ba15b700 1 --2- 172.21.15.195:0/2151390977 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7594063d30 0x7f75940661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:39.888 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.852+0000 7f75ba15b700 1 --2- 172.21.15.195:0/2151390977 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7594063d30 0x7f75940661e0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f75b0006d90 tx=0x7f75b001a000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:06:39.888 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:39.856+0000 7f75a27fc700 1 -- 172.21.15.195:0/2151390977 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f75ac085070 con 0x7f75b416a1a0 2022-06-27T12:06:40.085 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:40.085+0000 7f75bc3bf700 1 -- 172.21.15.195:0/2151390977 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f75b41586f0 con 0x7f75b416a1a0 2022-06-27T12:06:40.086 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:40.086+0000 7f75a27fc700 1 -- 172.21.15.195:0/2151390977 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+5367 (secure 0 0 0) 0x7f75ac04f4a0 con 0x7f75b416a1a0 2022-06-27T12:06:40.088 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:06:40.088 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:06:35.081018+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5591},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5591":{"gid":5591,"name":"b","rank":0,"incarnation":67,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6833/3022591968","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3022591968},{"type":"v1","addr":"172.21.15.202:6833","nonce":3022591968}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:06:40.091 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:40.088+0000 7f75bc3bf700 1 -- 172.21.15.195:0/2151390977 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7594063d30 msgr2=0x7f75940661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:40.091 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:40.088+0000 7f75bc3bf700 1 --2- 172.21.15.195:0/2151390977 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7594063d30 0x7f75940661e0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f75b0006d90 tx=0x7f75b001a000 comp rx=0 tx=0).stop 2022-06-27T12:06:40.092 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:40.088+0000 7f75bc3bf700 1 -- 172.21.15.195:0/2151390977 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f75b416a1a0 msgr2=0x7f75b416a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:40.092 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:40.088+0000 7f75bc3bf700 1 --2- 172.21.15.195:0/2151390977 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f75b416a1a0 0x7f75b416a610 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f75ac000f80 tx=0x7f75b40c3220 comp rx=0 tx=0).stop 2022-06-27T12:06:40.093 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:40.088+0000 7f75bc3bf700 1 -- 172.21.15.195:0/2151390977 shutdown_connections 2022-06-27T12:06:40.093 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:40.088+0000 7f75bc3bf700 1 --2- 172.21.15.195:0/2151390977 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7594063d30 0x7f75940661e0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:40.094 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:40.088+0000 7f75bc3bf700 1 --2- 172.21.15.195:0/2151390977 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f75b416a1a0 0x7f75b416a610 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:40.094 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:40.088+0000 7f75bc3bf700 1 --2- 172.21.15.195:0/2151390977 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f759c003920 0x7f75b40c90e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:40.095 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:40.088+0000 7f75bc3bf700 1 --2- 172.21.15.195:0/2151390977 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f75b4145b90 0x7f75b4146000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:40.095 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:40.088+0000 7f75bc3bf700 1 -- 172.21.15.195:0/2151390977 >> 172.21.15.195:0/2151390977 conn(0x7f75b4063af0 msgr2=0x7f75b40c59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:06:40.096 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:40.088+0000 7f75bc3bf700 1 -- 172.21.15.195:0/2151390977 shutdown_connections 2022-06-27T12:06:40.096 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:40.088+0000 7f75bc3bf700 1 -- 172.21.15.195:0/2151390977 wait complete. 2022-06-27T12:06:40.097 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 67 2022-06-27T12:06:40.103 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:06:42.105 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:06:42.238 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.237+0000 7f28a6c66700 1 Processor -- start 2022-06-27T12:06:42.238 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.237+0000 7f28a6c66700 1 -- start start 2022-06-27T12:06:42.248 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.238+0000 7f28a6c66700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f28a006e850 0x7f28a01570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:42.249 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.238+0000 7f28a6c66700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f28a006fef0 0x7f28a0069de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:42.249 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.238+0000 7f28a6c66700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28a006fba0 con 0x7f28a00c34a0 2022-06-27T12:06:42.250 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.238+0000 7f28a6c66700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28a01575e0 con 0x7f28a006fef0 2022-06-27T12:06:42.250 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.238+0000 7f28a6c66700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28a006f5e0 con 0x7f28a006e850 2022-06-27T12:06:42.250 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.238+0000 7f289ffff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f28a006fef0 0x7f28a0069de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:42.250 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.238+0000 7f289ffff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f28a006fef0 0x7f28a0069de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48440/0 (socket says 172.21.15.195:48440) 2022-06-27T12:06:42.251 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.238+0000 7f289ffff700 1 -- 172.21.15.195:0/4234488254 learned_addr learned my addr 172.21.15.195:0/4234488254 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:06:42.251 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.238+0000 7f28a4a02700 1 --2- 172.21.15.195:0/4234488254 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f28a006e850 0x7f28a01570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:42.251 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.239+0000 7f289ffff700 1 -- 172.21.15.195:0/4234488254 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f28a006e850 msgr2=0x7f28a01570a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:06:42.251 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.239+0000 7f289ffff700 1 --2- 172.21.15.195:0/4234488254 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f28a006e850 0x7f28a01570a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:42.252 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.239+0000 7f289ffff700 1 -- 172.21.15.195:0/4234488254 >> v1:172.21.15.202:6789/0 conn(0x7f28a00c34a0 legacy=0x7f28a00c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:42.252 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.239+0000 7f289ffff700 1 -- 172.21.15.195:0/4234488254 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28a006f8c0 con 0x7f28a006fef0 2022-06-27T12:06:42.252 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.239+0000 7f289ffff700 1 --2- 172.21.15.195:0/4234488254 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f28a006fef0 0x7f28a0069de0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f2894009860 tx=0x7f2894009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad6b0221e308c2e7 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:06:42.252 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.239+0000 7f289f7fe700 1 -- 172.21.15.195:0/4234488254 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f289400b710 con 0x7f28a006fef0 2022-06-27T12:06:42.253 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.239+0000 7f289f7fe700 1 -- 172.21.15.195:0/4234488254 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f289400b870 con 0x7f28a006fef0 2022-06-27T12:06:42.253 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.239+0000 7f289f7fe700 1 -- 172.21.15.195:0/4234488254 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f289400bc30 con 0x7f28a006fef0 2022-06-27T12:06:42.253 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.240+0000 7f28a6c66700 1 -- 172.21.15.195:0/4234488254 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f28a006fef0 msgr2=0x7f28a0069de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:42.254 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.240+0000 7f28a6c66700 1 --2- 172.21.15.195:0/4234488254 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f28a006fef0 0x7f28a0069de0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f2894009860 tx=0x7f2894009120 comp rx=0 tx=0).stop 2022-06-27T12:06:42.254 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.240+0000 7f28a6c66700 1 -- 172.21.15.195:0/4234488254 shutdown_connections 2022-06-27T12:06:42.254 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.240+0000 7f28a6c66700 1 --2- 172.21.15.195:0/4234488254 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f28a006e850 0x7f28a01570a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:42.254 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.240+0000 7f28a6c66700 1 --2- 172.21.15.195:0/4234488254 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f28a006fef0 0x7f28a0069de0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:42.254 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.240+0000 7f28a6c66700 1 -- 172.21.15.195:0/4234488254 >> 172.21.15.195:0/4234488254 conn(0x7f28a00c6350 msgr2=0x7f28a00c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:06:42.255 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.240+0000 7f28a6c66700 1 -- 172.21.15.195:0/4234488254 shutdown_connections 2022-06-27T12:06:42.255 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.240+0000 7f28a6c66700 1 -- 172.21.15.195:0/4234488254 wait complete. 2022-06-27T12:06:42.255 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.241+0000 7f28a6c66700 1 Processor -- start 2022-06-27T12:06:42.255 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.241+0000 7f28a6c66700 1 -- start start 2022-06-27T12:06:42.256 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.242+0000 7f28a6c66700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f28a006e850 0x7f28a00be120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:42.256 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.242+0000 7f28a6c66700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f28a00c34a0 0x7f28a00be660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:42.257 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.242+0000 7f28a6c66700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f28a00c03d0 0x7f28a01619d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:42.257 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.242+0000 7f28a6c66700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28a0160fc0 con 0x7f28a006e850 2022-06-27T12:06:42.257 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.242+0000 7f28a6c66700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28a0161540 con 0x7f28a00c34a0 2022-06-27T12:06:42.257 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.242+0000 7f28a6c66700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28a00c5870 con 0x7f28a00c03d0 2022-06-27T12:06:42.259 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.242+0000 7f28a4a02700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f28a006e850 0x7f28a00be120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:42.259 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.242+0000 7f28a4a02700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f28a006e850 0x7f28a00be120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48444/0 (socket says 172.21.15.195:48444) 2022-06-27T12:06:42.259 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.242+0000 7f28a4a02700 1 -- 172.21.15.195:0/869878076 learned_addr learned my addr 172.21.15.195:0/869878076 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:06:42.259 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.242+0000 7f289ffff700 1 --2- 172.21.15.195:0/869878076 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f28a00c34a0 0x7f28a00be660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:42.260 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.242+0000 7f28a5203700 1 --2- 172.21.15.195:0/869878076 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f28a00c03d0 0x7f28a01619d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:42.260 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.242+0000 7f28a4a02700 1 -- 172.21.15.195:0/869878076 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f28a00c03d0 msgr2=0x7f28a01619d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:42.260 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.242+0000 7f28a4a02700 1 --2- 172.21.15.195:0/869878076 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f28a00c03d0 0x7f28a01619d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:42.260 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.242+0000 7f28a4a02700 1 -- 172.21.15.195:0/869878076 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f28a00c34a0 msgr2=0x7f28a00be660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:42.261 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.242+0000 7f28a4a02700 1 --2- 172.21.15.195:0/869878076 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f28a00c34a0 0x7f28a00be660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:42.261 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.242+0000 7f28a4a02700 1 -- 172.21.15.195:0/869878076 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28940093a0 con 0x7f28a006e850 2022-06-27T12:06:42.261 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.242+0000 7f28a4a02700 1 --2- 172.21.15.195:0/869878076 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f28a006e850 0x7f28a00be120 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f289000fb00 tx=0x7f2890008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:06:42.261 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.242+0000 7f289cff9700 1 -- 172.21.15.195:0/869878076 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2890009850 con 0x7f28a006e850 2022-06-27T12:06:42.262 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.242+0000 7f289cff9700 1 -- 172.21.15.195:0/869878076 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28900099b0 con 0x7f28a006e850 2022-06-27T12:06:42.262 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.243+0000 7f28a6c66700 1 -- 172.21.15.195:0/869878076 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28a00beca0 con 0x7f28a006e850 2022-06-27T12:06:42.263 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.243+0000 7f289cff9700 1 -- 172.21.15.195:0/869878076 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2890009d50 con 0x7f28a006e850 2022-06-27T12:06:42.263 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.243+0000 7f28a6c66700 1 -- 172.21.15.195:0/869878076 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28a01611d0 con 0x7f28a006e850 2022-06-27T12:06:42.264 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.243+0000 7f28a6c66700 1 -- 172.21.15.195:0/869878076 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28a00722c0 con 0x7f28a006e850 2022-06-27T12:06:42.264 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.244+0000 7f289cff9700 1 -- 172.21.15.195:0/869878076 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f2890011660 con 0x7f28a006e850 2022-06-27T12:06:42.264 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.244+0000 7f289cff9700 1 --2- 172.21.15.195:0/869878076 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2880063d00 0x7f28800661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:42.264 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.244+0000 7f289cff9700 1 -- 172.21.15.195:0/869878076 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7062+0+0 (secure 0 0 0) 0x7f28900835e0 con 0x7f28a006e850 2022-06-27T12:06:42.265 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.244+0000 7f289ffff700 1 --2- 172.21.15.195:0/869878076 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2880063d00 0x7f28800661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:42.265 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.245+0000 7f289ffff700 1 --2- 172.21.15.195:0/869878076 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2880063d00 0x7f28800661b0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f28a00c73e0 tx=0x7f2894017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:06:42.265 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.248+0000 7f289cff9700 1 -- 172.21.15.195:0/869878076 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f2890051cf0 con 0x7f28a006e850 2022-06-27T12:06:42.473 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.473+0000 7f28a6c66700 1 -- 172.21.15.195:0/869878076 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f28a0161820 con 0x7f28a006e850 2022-06-27T12:06:42.476 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.473+0000 7f289cff9700 1 -- 172.21.15.195:0/869878076 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+5367 (secure 0 0 0) 0x7f2890055310 con 0x7f28a006e850 2022-06-27T12:06:42.480 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:06:42.480 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:06:35.081018+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5591},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5591":{"gid":5591,"name":"b","rank":0,"incarnation":67,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6833/3022591968","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3022591968},{"type":"v1","addr":"172.21.15.202:6833","nonce":3022591968}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:06:42.483 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.475+0000 7f28a6c66700 1 -- 172.21.15.195:0/869878076 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2880063d00 msgr2=0x7f28800661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:42.483 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.475+0000 7f28a6c66700 1 --2- 172.21.15.195:0/869878076 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2880063d00 0x7f28800661b0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f28a00c73e0 tx=0x7f2894017000 comp rx=0 tx=0).stop 2022-06-27T12:06:42.484 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.475+0000 7f28a6c66700 1 -- 172.21.15.195:0/869878076 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f28a006e850 msgr2=0x7f28a00be120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:42.484 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.475+0000 7f28a6c66700 1 --2- 172.21.15.195:0/869878076 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f28a006e850 0x7f28a00be120 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f289000fb00 tx=0x7f2890008040 comp rx=0 tx=0).stop 2022-06-27T12:06:42.485 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.475+0000 7f28a6c66700 1 -- 172.21.15.195:0/869878076 shutdown_connections 2022-06-27T12:06:42.485 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.475+0000 7f28a6c66700 1 --2- 172.21.15.195:0/869878076 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2880063d00 0x7f28800661b0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:42.486 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.475+0000 7f28a6c66700 1 --2- 172.21.15.195:0/869878076 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f28a006e850 0x7f28a00be120 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:42.486 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.475+0000 7f28a6c66700 1 --2- 172.21.15.195:0/869878076 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f28a00c03d0 0x7f28a01619d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:42.487 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.475+0000 7f28a6c66700 1 --2- 172.21.15.195:0/869878076 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f28a00c34a0 0x7f28a00be660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:42.487 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.475+0000 7f28a6c66700 1 -- 172.21.15.195:0/869878076 >> 172.21.15.195:0/869878076 conn(0x7f28a00c6350 msgr2=0x7f28a00c26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:06:42.487 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.475+0000 7f28a6c66700 1 -- 172.21.15.195:0/869878076 shutdown_connections 2022-06-27T12:06:42.488 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:42.475+0000 7f28a6c66700 1 -- 172.21.15.195:0/869878076 wait complete. 2022-06-27T12:06:42.488 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 67 2022-06-27T12:06:42.492 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:06:44.493 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:06:44.643 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.640+0000 7fa93007e700 1 Processor -- start 2022-06-27T12:06:44.643 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.640+0000 7fa93007e700 1 -- start start 2022-06-27T12:06:44.644 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.641+0000 7fa93007e700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fa92806e850 0x7fa9281570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:44.645 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.641+0000 7fa93007e700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa92806fba0 con 0x7fa92806fef0 2022-06-27T12:06:44.645 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.641+0000 7fa93007e700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa92806f5e0 con 0x7fa9280c34a0 2022-06-27T12:06:44.647 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.641+0000 7fa93007e700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9281575e0 con 0x7fa92806e850 2022-06-27T12:06:44.647 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.641+0000 7fa92de1a700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fa92806e850 0x7fa9281570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:44.648 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.641+0000 7fa92d619700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fa92806fef0 0x7fa928069de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:54640/0 (socket says 172.21.15.195:54640) 2022-06-27T12:06:44.648 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.641+0000 7fa92d619700 1 -- 172.21.15.195:0/2680810025 learned_addr learned my addr 172.21.15.195:0/2680810025 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:06:44.651 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.641+0000 7fa92de1a700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fa92806e850 0x7fa9281570a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48452/0 (socket says 172.21.15.195:48452) 2022-06-27T12:06:44.653 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.641+0000 7fa92de1a700 1 -- 172.21.15.195:0/2680810025 >> v1:172.21.15.202:6789/0 conn(0x7fa9280c34a0 legacy=0x7fa9280c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:44.664 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.641+0000 7fa92de1a700 1 -- 172.21.15.195:0/2680810025 >> v1:172.21.15.195:6789/0 conn(0x7fa92806fef0 legacy=0x7fa928069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:44.664 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.641+0000 7fa92de1a700 1 -- 172.21.15.195:0/2680810025 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa92806f8c0 con 0x7fa92806e850 2022-06-27T12:06:44.665 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.641+0000 7fa92ce18700 1 -- 172.21.15.195:0/2680810025 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3775462483 0 0) 0x7fa92806fba0 con 0x7fa92806fef0 2022-06-27T12:06:44.665 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.642+0000 7fa92de1a700 1 --2- 172.21.15.195:0/2680810025 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa92806e850 0x7fa9281570a0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fa9180097a0 tx=0x7fa918009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd543dbad5f35873 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:06:44.666 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.642+0000 7fa92ce18700 1 -- 172.21.15.195:0/2680810025 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa918011630 con 0x7fa92806e850 2022-06-27T12:06:44.666 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.642+0000 7fa92ce18700 1 -- 172.21.15.195:0/2680810025 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa918011790 con 0x7fa92806e850 2022-06-27T12:06:44.666 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.642+0000 7fa92ce18700 1 -- 172.21.15.195:0/2680810025 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa918011b30 con 0x7fa92806e850 2022-06-27T12:06:44.667 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.642+0000 7fa93007e700 1 -- 172.21.15.195:0/2680810025 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa92806e850 msgr2=0x7fa9281570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:44.667 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.642+0000 7fa93007e700 1 --2- 172.21.15.195:0/2680810025 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa92806e850 0x7fa9281570a0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fa9180097a0 tx=0x7fa918009060 comp rx=0 tx=0).stop 2022-06-27T12:06:44.668 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.642+0000 7fa93007e700 1 -- 172.21.15.195:0/2680810025 shutdown_connections 2022-06-27T12:06:44.668 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.642+0000 7fa93007e700 1 --2- 172.21.15.195:0/2680810025 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa92806e850 0x7fa9281570a0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:44.668 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.642+0000 7fa93007e700 1 -- 172.21.15.195:0/2680810025 >> 172.21.15.195:0/2680810025 conn(0x7fa9280c6350 msgr2=0x7fa9280c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:06:44.669 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.642+0000 7fa93007e700 1 -- 172.21.15.195:0/2680810025 shutdown_connections 2022-06-27T12:06:44.669 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.642+0000 7fa93007e700 1 -- 172.21.15.195:0/2680810025 wait complete. 2022-06-27T12:06:44.669 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.644+0000 7fa93007e700 1 Processor -- start 2022-06-27T12:06:44.670 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.644+0000 7fa93007e700 1 -- start start 2022-06-27T12:06:44.670 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.644+0000 7fa93007e700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa92806e850 0x7fa928152600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:44.670 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.644+0000 7fa93007e700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa9280c34a0 0x7fa9280c0080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:44.670 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.644+0000 7fa93007e700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa9280c05c0 0x7fa92814c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:44.671 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.644+0000 7fa93007e700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa928155b00 con 0x7fa9280c34a0 2022-06-27T12:06:44.671 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.644+0000 7fa93007e700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa928155d50 con 0x7fa9280c05c0 2022-06-27T12:06:44.672 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.644+0000 7fa93007e700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa928156030 con 0x7fa92806e850 2022-06-27T12:06:44.672 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.644+0000 7fa92d619700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa9280c34a0 0x7fa9280c0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:44.672 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.644+0000 7fa92d619700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa9280c34a0 0x7fa9280c0080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48460/0 (socket says 172.21.15.195:48460) 2022-06-27T12:06:44.673 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.644+0000 7fa92d619700 1 -- 172.21.15.195:0/799167874 learned_addr learned my addr 172.21.15.195:0/799167874 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:06:44.673 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.645+0000 7fa92de1a700 1 --2- 172.21.15.195:0/799167874 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa92806e850 0x7fa928152600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:44.673 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.645+0000 7fa92e61b700 1 --2- 172.21.15.195:0/799167874 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa9280c05c0 0x7fa92814c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:44.674 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.645+0000 7fa92d619700 1 -- 172.21.15.195:0/799167874 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa92806e850 msgr2=0x7fa928152600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:44.674 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.645+0000 7fa92d619700 1 --2- 172.21.15.195:0/799167874 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa92806e850 0x7fa928152600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:44.675 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.645+0000 7fa92d619700 1 -- 172.21.15.195:0/799167874 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa9280c05c0 msgr2=0x7fa92814c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:44.675 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.645+0000 7fa92d619700 1 --2- 172.21.15.195:0/799167874 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa9280c05c0 0x7fa92814c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:44.675 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.645+0000 7fa92d619700 1 -- 172.21.15.195:0/799167874 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9180092e0 con 0x7fa9280c34a0 2022-06-27T12:06:44.676 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.645+0000 7fa92d619700 1 --2- 172.21.15.195:0/799167874 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa9280c34a0 0x7fa9280c0080 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fa92400c490 tx=0x7fa924007de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:06:44.676 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.645+0000 7fa915ffb700 1 -- 172.21.15.195:0/799167874 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa924007360 con 0x7fa9280c34a0 2022-06-27T12:06:44.677 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.645+0000 7fa915ffb700 1 -- 172.21.15.195:0/799167874 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9240074c0 con 0x7fa9280c34a0 2022-06-27T12:06:44.677 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.645+0000 7fa93007e700 1 -- 172.21.15.195:0/799167874 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa928156310 con 0x7fa9280c34a0 2022-06-27T12:06:44.677 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.645+0000 7fa915ffb700 1 -- 172.21.15.195:0/799167874 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa924004940 con 0x7fa9280c34a0 2022-06-27T12:06:44.677 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.645+0000 7fa93007e700 1 -- 172.21.15.195:0/799167874 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa928152fe0 con 0x7fa9280c34a0 2022-06-27T12:06:44.678 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.646+0000 7fa93007e700 1 -- 172.21.15.195:0/799167874 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9280722c0 con 0x7fa9280c34a0 2022-06-27T12:06:44.678 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.646+0000 7fa915ffb700 1 -- 172.21.15.195:0/799167874 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fa92401a020 con 0x7fa9280c34a0 2022-06-27T12:06:44.679 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.647+0000 7fa915ffb700 1 --2- 172.21.15.195:0/799167874 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa908063d00 0x7fa9080661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:44.679 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.647+0000 7fa915ffb700 1 -- 172.21.15.195:0/799167874 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7062+0+0 (secure 0 0 0) 0x7fa924080660 con 0x7fa9280c34a0 2022-06-27T12:06:44.679 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.647+0000 7fa92de1a700 1 --2- 172.21.15.195:0/799167874 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa908063d00 0x7fa9080661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:44.679 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.647+0000 7fa92de1a700 1 --2- 172.21.15.195:0/799167874 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa908063d00 0x7fa9080661b0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fa918011fd0 tx=0x7fa918017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:06:44.680 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.651+0000 7fa915ffb700 1 -- 172.21.15.195:0/799167874 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fa92404eee0 con 0x7fa9280c34a0 2022-06-27T12:06:44.882 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.882+0000 7fa93007e700 1 -- 172.21.15.195:0/799167874 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa9280c8a90 con 0x7fa9280c34a0 2022-06-27T12:06:44.883 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.882+0000 7fa915ffb700 1 -- 172.21.15.195:0/799167874 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+5367 (secure 0 0 0) 0x7fa924052500 con 0x7fa9280c34a0 2022-06-27T12:06:44.884 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:06:44.884 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:06:35.081018+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5591},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5591":{"gid":5591,"name":"b","rank":0,"incarnation":67,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6833/3022591968","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3022591968},{"type":"v1","addr":"172.21.15.202:6833","nonce":3022591968}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:06:44.890 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.884+0000 7fa93007e700 1 -- 172.21.15.195:0/799167874 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa908063d00 msgr2=0x7fa9080661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:44.891 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.884+0000 7fa93007e700 1 --2- 172.21.15.195:0/799167874 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa908063d00 0x7fa9080661b0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fa918011fd0 tx=0x7fa918017000 comp rx=0 tx=0).stop 2022-06-27T12:06:44.891 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.884+0000 7fa93007e700 1 -- 172.21.15.195:0/799167874 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa9280c34a0 msgr2=0x7fa9280c0080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:44.891 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.884+0000 7fa93007e700 1 --2- 172.21.15.195:0/799167874 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa9280c34a0 0x7fa9280c0080 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fa92400c490 tx=0x7fa924007de0 comp rx=0 tx=0).stop 2022-06-27T12:06:44.891 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.884+0000 7fa93007e700 1 -- 172.21.15.195:0/799167874 shutdown_connections 2022-06-27T12:06:44.892 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.884+0000 7fa93007e700 1 --2- 172.21.15.195:0/799167874 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa908063d00 0x7fa9080661b0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:44.892 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.884+0000 7fa93007e700 1 --2- 172.21.15.195:0/799167874 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa9280c05c0 0x7fa92814c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:44.892 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.884+0000 7fa93007e700 1 --2- 172.21.15.195:0/799167874 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa92806e850 0x7fa928152600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:44.892 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.884+0000 7fa93007e700 1 --2- 172.21.15.195:0/799167874 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa9280c34a0 0x7fa9280c0080 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:44.893 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.884+0000 7fa93007e700 1 -- 172.21.15.195:0/799167874 >> 172.21.15.195:0/799167874 conn(0x7fa9280c6350 msgr2=0x7fa92814f730 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:06:44.893 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.884+0000 7fa93007e700 1 -- 172.21.15.195:0/799167874 shutdown_connections 2022-06-27T12:06:44.893 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:44.884+0000 7fa93007e700 1 -- 172.21.15.195:0/799167874 wait complete. 2022-06-27T12:06:44.893 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 67 2022-06-27T12:06:44.897 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:06:46.899 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:06:47.046 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.034+0000 7fbab6a0f700 1 Processor -- start 2022-06-27T12:06:47.047 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.034+0000 7fbab6a0f700 1 -- start start 2022-06-27T12:06:47.047 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.035+0000 7fbab6a0f700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fbab0150500 0x7fbab0063af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:47.048 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.035+0000 7fbab6a0f700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fbab006a450 0x7fbab00c48d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:47.048 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.035+0000 7fbab6a0f700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbab0064030 con 0x7fbab01508d0 2022-06-27T12:06:47.049 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.035+0000 7fbab6a0f700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbab00c4160 con 0x7fbab006a450 2022-06-27T12:06:47.049 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.035+0000 7fbab6a0f700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbab00c4440 con 0x7fbab0150500 2022-06-27T12:06:47.050 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.035+0000 7fbaaf7fe700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fbab006a450 0x7fbab00c48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:47.050 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.035+0000 7fbaaf7fe700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fbab006a450 0x7fbab00c48d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48470/0 (socket says 172.21.15.195:48470) 2022-06-27T12:06:47.050 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.035+0000 7fbaaf7fe700 1 -- 172.21.15.195:0/1355220837 learned_addr learned my addr 172.21.15.195:0/1355220837 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:06:47.051 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.035+0000 7fbaaffff700 1 --2- 172.21.15.195:0/1355220837 >> v2:172.21.15.202:3300/0 conn(0x7fbab0150500 0x7fbab0063af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:47.051 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.035+0000 7fbaaf7fe700 1 -- 172.21.15.195:0/1355220837 >> v2:172.21.15.202:3300/0 conn(0x7fbab0150500 msgr2=0x7fbab0063af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:06:47.052 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.035+0000 7fbaaf7fe700 1 --2- 172.21.15.195:0/1355220837 >> v2:172.21.15.202:3300/0 conn(0x7fbab0150500 0x7fbab0063af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:47.052 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.035+0000 7fbaaf7fe700 1 -- 172.21.15.195:0/1355220837 >> v1:172.21.15.202:6789/0 conn(0x7fbab01508d0 legacy=0x7fbab00bbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:47.053 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.035+0000 7fbaaf7fe700 1 -- 172.21.15.195:0/1355220837 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbab015a800 con 0x7fbab006a450 2022-06-27T12:06:47.053 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.035+0000 7fbaaf7fe700 1 --2- 172.21.15.195:0/1355220837 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbab006a450 0x7fbab00c48d0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fbab006f840 tx=0x7fbaa0009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bac35bf1f4af315 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:06:47.053 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.035+0000 7fbaaeffd700 1 -- 172.21.15.195:0/1355220837 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaa0011630 con 0x7fbab006a450 2022-06-27T12:06:47.054 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.036+0000 7fbaaeffd700 1 -- 172.21.15.195:0/1355220837 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaa0011790 con 0x7fbab006a450 2022-06-27T12:06:47.054 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.036+0000 7fbaaeffd700 1 -- 172.21.15.195:0/1355220837 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaa0011b30 con 0x7fbab006a450 2022-06-27T12:06:47.054 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.036+0000 7fbab6a0f700 1 -- 172.21.15.195:0/1355220837 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbab006a450 msgr2=0x7fbab00c48d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:47.055 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.036+0000 7fbab6a0f700 1 --2- 172.21.15.195:0/1355220837 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbab006a450 0x7fbab00c48d0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fbab006f840 tx=0x7fbaa0009120 comp rx=0 tx=0).stop 2022-06-27T12:06:47.055 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.036+0000 7fbab6a0f700 1 -- 172.21.15.195:0/1355220837 shutdown_connections 2022-06-27T12:06:47.056 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.036+0000 7fbab6a0f700 1 --2- 172.21.15.195:0/1355220837 >> v2:172.21.15.202:3300/0 conn(0x7fbab0150500 0x7fbab0063af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:47.056 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.036+0000 7fbab6a0f700 1 --2- 172.21.15.195:0/1355220837 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbab006a450 0x7fbab00c48d0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:47.057 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.036+0000 7fbab6a0f700 1 -- 172.21.15.195:0/1355220837 >> 172.21.15.195:0/1355220837 conn(0x7fbab00c9b00 msgr2=0x7fbab00c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:06:47.058 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.036+0000 7fbab6a0f700 1 -- 172.21.15.195:0/1355220837 shutdown_connections 2022-06-27T12:06:47.058 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.036+0000 7fbab6a0f700 1 -- 172.21.15.195:0/1355220837 wait complete. 2022-06-27T12:06:47.058 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.037+0000 7fbab6a0f700 1 Processor -- start 2022-06-27T12:06:47.058 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.037+0000 7fbab6a0f700 1 -- start start 2022-06-27T12:06:47.059 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.038+0000 7fbab6a0f700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbab006a450 0x7fbab00be850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:47.059 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.038+0000 7fbab6a0f700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbab0150500 0x7fbab00c25c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:47.059 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.038+0000 7fbab6a0f700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbab01508d0 0x7fbab014dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:47.059 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.038+0000 7fbab6a0f700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbab00c21a0 con 0x7fbab01508d0 2022-06-27T12:06:47.059 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.038+0000 7fbab6a0f700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbab006a1d0 con 0x7fbab006a450 2022-06-27T12:06:47.060 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.038+0000 7fbab6a0f700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbab00bd790 con 0x7fbab0150500 2022-06-27T12:06:47.060 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.038+0000 7fbab4fac700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbab01508d0 0x7fbab014dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:47.062 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.038+0000 7fbaaffff700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbab006a450 0x7fbab00be850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:47.062 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.038+0000 7fbaaf7fe700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbab0150500 0x7fbab00c25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:47.063 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.038+0000 7fbab4fac700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbab01508d0 0x7fbab014dc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48480/0 (socket says 172.21.15.195:48480) 2022-06-27T12:06:47.063 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.038+0000 7fbab4fac700 1 -- 172.21.15.195:0/1879368305 learned_addr learned my addr 172.21.15.195:0/1879368305 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:06:47.064 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.038+0000 7fbaaffff700 1 -- 172.21.15.195:0/1879368305 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbab0150500 msgr2=0x7fbab00c25c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:47.064 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.038+0000 7fbaaffff700 1 --2- 172.21.15.195:0/1879368305 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbab0150500 0x7fbab00c25c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:47.065 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.038+0000 7fbaaffff700 1 -- 172.21.15.195:0/1879368305 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbab01508d0 msgr2=0x7fbab014dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:47.065 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.039+0000 7fbaaffff700 1 --2- 172.21.15.195:0/1879368305 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbab01508d0 0x7fbab014dc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:47.065 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.039+0000 7fbaaffff700 1 -- 172.21.15.195:0/1879368305 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaa00093a0 con 0x7fbab006a450 2022-06-27T12:06:47.066 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.039+0000 7fbab4fac700 1 --2- 172.21.15.195:0/1879368305 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbab01508d0 0x7fbab014dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:06:47.066 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.039+0000 7fbaaffff700 1 --2- 172.21.15.195:0/1879368305 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbab006a450 0x7fbab00be850 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fba9800e900 tx=0x7fba98009490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:06:47.067 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.039+0000 7fbaa7fff700 1 -- 172.21.15.195:0/1879368305 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba98013680 con 0x7fbab006a450 2022-06-27T12:06:47.067 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.039+0000 7fbab6a0f700 1 -- 172.21.15.195:0/1879368305 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbab00c0720 con 0x7fbab006a450 2022-06-27T12:06:47.068 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.039+0000 7fbaa7fff700 1 -- 172.21.15.195:0/1879368305 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba980137e0 con 0x7fbab006a450 2022-06-27T12:06:47.068 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.039+0000 7fbaa7fff700 1 -- 172.21.15.195:0/1879368305 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba98013b80 con 0x7fbab006a450 2022-06-27T12:06:47.069 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.039+0000 7fbab6a0f700 1 -- 172.21.15.195:0/1879368305 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbab00bfa90 con 0x7fbab006a450 2022-06-27T12:06:47.069 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.040+0000 7fbab6a0f700 1 -- 172.21.15.195:0/1879368305 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbab00722c0 con 0x7fbab006a450 2022-06-27T12:06:47.069 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.040+0000 7fbaa7fff700 1 -- 172.21.15.195:0/1879368305 <== mon.1 v2:172.21.15.202:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fba98023070 con 0x7fbab006a450 2022-06-27T12:06:47.070 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.041+0000 7fbaa7fff700 1 --2- 172.21.15.195:0/1879368305 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fba94063d30 0x7fba940661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:47.070 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.041+0000 7fbaa7fff700 1 -- 172.21.15.195:0/1879368305 <== mon.1 v2:172.21.15.202:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7062+0+0 (secure 0 0 0) 0x7fba98080f80 con 0x7fbab006a450 2022-06-27T12:06:47.072 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.041+0000 7fbaaf7fe700 1 --2- 172.21.15.195:0/1879368305 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fba94063d30 0x7fba940661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:47.073 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.041+0000 7fbaaf7fe700 1 --2- 172.21.15.195:0/1879368305 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fba94063d30 0x7fba940661e0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fbaa0009160 tx=0x7fbaa0017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:06:47.074 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.049+0000 7fbaa7fff700 1 -- 172.21.15.195:0/1879368305 <== mon.1 v2:172.21.15.202:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fba9804f780 con 0x7fbab006a450 2022-06-27T12:06:47.277 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.277+0000 7fbab6a0f700 1 -- 172.21.15.195:0/1879368305 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbab00ba8e0 con 0x7fbab006a450 2022-06-27T12:06:47.279 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.279+0000 7fbaa7fff700 1 -- 172.21.15.195:0/1879368305 <== mon.1 v2:172.21.15.202:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+5367 (secure 0 0 0) 0x7fba98021090 con 0x7fbab006a450 2022-06-27T12:06:47.279 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:06:47.280 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:06:35.081018+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5591},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5591":{"gid":5591,"name":"b","rank":0,"incarnation":67,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6833/3022591968","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3022591968},{"type":"v1","addr":"172.21.15.202:6833","nonce":3022591968}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:06:47.283 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.280+0000 7fbab6a0f700 1 -- 172.21.15.195:0/1879368305 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fba94063d30 msgr2=0x7fba940661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:47.284 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.280+0000 7fbab6a0f700 1 --2- 172.21.15.195:0/1879368305 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fba94063d30 0x7fba940661e0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fbaa0009160 tx=0x7fbaa0017000 comp rx=0 tx=0).stop 2022-06-27T12:06:47.284 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.280+0000 7fbab6a0f700 1 -- 172.21.15.195:0/1879368305 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbab006a450 msgr2=0x7fbab00be850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:47.284 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.280+0000 7fbab6a0f700 1 --2- 172.21.15.195:0/1879368305 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbab006a450 0x7fbab00be850 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fba9800e900 tx=0x7fba98009490 comp rx=0 tx=0).stop 2022-06-27T12:06:47.284 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.281+0000 7fbab6a0f700 1 -- 172.21.15.195:0/1879368305 shutdown_connections 2022-06-27T12:06:47.285 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.281+0000 7fbab6a0f700 1 --2- 172.21.15.195:0/1879368305 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fba94063d30 0x7fba940661e0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:47.285 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.281+0000 7fbab6a0f700 1 --2- 172.21.15.195:0/1879368305 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbab01508d0 0x7fbab014dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:47.286 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.281+0000 7fbab6a0f700 1 --2- 172.21.15.195:0/1879368305 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbab006a450 0x7fbab00be850 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:47.286 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.281+0000 7fbab6a0f700 1 --2- 172.21.15.195:0/1879368305 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbab0150500 0x7fbab00c25c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:47.286 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.281+0000 7fbab6a0f700 1 -- 172.21.15.195:0/1879368305 >> 172.21.15.195:0/1879368305 conn(0x7fbab00c9b00 msgr2=0x7fbab00c6e40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:06:47.287 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.281+0000 7fbab6a0f700 1 -- 172.21.15.195:0/1879368305 shutdown_connections 2022-06-27T12:06:47.287 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:47.281+0000 7fbab6a0f700 1 -- 172.21.15.195:0/1879368305 wait complete. 2022-06-27T12:06:47.287 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 67 2022-06-27T12:06:47.301 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:06:49.303 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:06:49.471 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.437+0000 7f7b6b116700 1 Processor -- start 2022-06-27T12:06:49.471 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.437+0000 7f7b6b116700 1 -- start start 2022-06-27T12:06:49.471 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.437+0000 7f7b6b116700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f7b6406e850 0x7f7b641570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:49.472 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.437+0000 7f7b6b116700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7b640c34a0 0x7f7b640c2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:49.472 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.437+0000 7f7b6b116700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b6406fba0 con 0x7f7b6406fef0 2022-06-27T12:06:49.472 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.437+0000 7f7b6b116700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b641575e0 con 0x7f7b6406e850 2022-06-27T12:06:49.472 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.437+0000 7f7b6b116700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b6406f5e0 con 0x7f7b640c34a0 2022-06-27T12:06:49.473 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.438+0000 7f7b68eb2700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f7b6406e850 0x7f7b641570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:49.473 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.438+0000 7f7b696b3700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7b640c34a0 0x7f7b640c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:49.473 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.438+0000 7f7b63fff700 1 --1- >> v1:172.21.15.202:6789/0 conn(0x7f7b6406fef0 0x7f7b64069de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.202:6789/0 says I am v1:172.21.15.195:56534/0 (socket says 172.21.15.195:56534) 2022-06-27T12:06:49.473 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.438+0000 7f7b63fff700 1 -- 172.21.15.195:0/123348009 learned_addr learned my addr 172.21.15.195:0/123348009 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:06:49.474 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.438+0000 7f7b637fe700 1 -- 172.21.15.195:0/123348009 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3070584929 0 0) 0x7f7b6406fba0 con 0x7f7b6406fef0 2022-06-27T12:06:49.474 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.438+0000 7f7b637fe700 1 -- 172.21.15.195:0/123348009 --> v1:172.21.15.202:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b48003430 con 0x7f7b6406fef0 2022-06-27T12:06:49.474 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.438+0000 7f7b696b3700 1 -- 172.21.15.195:0/123348009 >> v2:172.21.15.202:3300/0 conn(0x7f7b6406e850 msgr2=0x7f7b641570a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:06:49.474 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.438+0000 7f7b696b3700 1 --2- 172.21.15.195:0/123348009 >> v2:172.21.15.202:3300/0 conn(0x7f7b6406e850 0x7f7b641570a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:49.475 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.438+0000 7f7b696b3700 1 -- 172.21.15.195:0/123348009 >> v1:172.21.15.202:6789/0 conn(0x7f7b6406fef0 legacy=0x7f7b64069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:49.475 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.438+0000 7f7b696b3700 1 -- 172.21.15.195:0/123348009 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b6406f8c0 con 0x7f7b640c34a0 2022-06-27T12:06:49.475 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.438+0000 7f7b68eb2700 1 --2- 172.21.15.195:0/123348009 >> v2:172.21.15.202:3300/0 conn(0x7f7b6406e850 0x7f7b641570a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:06:49.476 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.438+0000 7f7b696b3700 1 --2- 172.21.15.195:0/123348009 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7b640c34a0 0x7f7b640c2920 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c009e60 tx=0x7f7b5c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ded8f5bd707a765a server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:06:49.477 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.438+0000 7f7b637fe700 1 -- 172.21.15.195:0/123348009 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b5c00e070 con 0x7f7b640c34a0 2022-06-27T12:06:49.477 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.439+0000 7f7b637fe700 1 -- 172.21.15.195:0/123348009 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b5c00bb40 con 0x7f7b640c34a0 2022-06-27T12:06:49.477 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.439+0000 7f7b637fe700 1 -- 172.21.15.195:0/123348009 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b5c012430 con 0x7f7b640c34a0 2022-06-27T12:06:49.478 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.439+0000 7f7b6b116700 1 -- 172.21.15.195:0/123348009 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7b640c34a0 msgr2=0x7f7b640c2920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:49.478 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.439+0000 7f7b6b116700 1 --2- 172.21.15.195:0/123348009 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7b640c34a0 0x7f7b640c2920 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c009e60 tx=0x7f7b5c009780 comp rx=0 tx=0).stop 2022-06-27T12:06:49.478 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.439+0000 7f7b6b116700 1 -- 172.21.15.195:0/123348009 shutdown_connections 2022-06-27T12:06:49.478 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.439+0000 7f7b6b116700 1 --2- 172.21.15.195:0/123348009 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7b640c34a0 0x7f7b640c2920 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:49.478 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.439+0000 7f7b6b116700 1 --2- 172.21.15.195:0/123348009 >> v2:172.21.15.202:3300/0 conn(0x7f7b6406e850 0x7f7b641570a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:49.479 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.439+0000 7f7b6b116700 1 -- 172.21.15.195:0/123348009 >> 172.21.15.195:0/123348009 conn(0x7f7b640c6350 msgr2=0x7f7b640c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:06:49.479 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.439+0000 7f7b6b116700 1 -- 172.21.15.195:0/123348009 shutdown_connections 2022-06-27T12:06:49.479 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.439+0000 7f7b6b116700 1 -- 172.21.15.195:0/123348009 wait complete. 2022-06-27T12:06:49.479 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.440+0000 7f7b6b116700 1 Processor -- start 2022-06-27T12:06:49.480 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.440+0000 7f7b6b116700 1 -- start start 2022-06-27T12:06:49.480 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.441+0000 7f7b6b116700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7b6406e850 0x7f7b641660f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:49.480 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.441+0000 7f7b6b116700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7b640c34a0 0x7f7b640ba330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:49.480 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.441+0000 7f7b6b116700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7b640bad20 0x7f7b64155720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:49.481 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.441+0000 7f7b6b116700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b640bfda0 con 0x7f7b640c34a0 2022-06-27T12:06:49.481 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.441+0000 7f7b6b116700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b640bf230 con 0x7f7b6406e850 2022-06-27T12:06:49.481 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.441+0000 7f7b6b116700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b640bef30 con 0x7f7b640bad20 2022-06-27T12:06:49.481 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.441+0000 7f7b63fff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7b640c34a0 0x7f7b640ba330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:49.481 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.441+0000 7f7b63fff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7b640c34a0 0x7f7b640ba330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48494/0 (socket says 172.21.15.195:48494) 2022-06-27T12:06:49.482 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.441+0000 7f7b63fff700 1 -- 172.21.15.195:0/1130666548 learned_addr learned my addr 172.21.15.195:0/1130666548 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:06:49.483 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.441+0000 7f7b696b3700 1 --2- 172.21.15.195:0/1130666548 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7b640bad20 0x7f7b64155720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:49.483 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.441+0000 7f7b68eb2700 1 --2- 172.21.15.195:0/1130666548 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7b6406e850 0x7f7b641660f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:49.483 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.442+0000 7f7b63fff700 1 -- 172.21.15.195:0/1130666548 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7b640bad20 msgr2=0x7f7b64155720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:49.484 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.442+0000 7f7b63fff700 1 --2- 172.21.15.195:0/1130666548 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7b640bad20 0x7f7b64155720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:49.484 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.442+0000 7f7b63fff700 1 -- 172.21.15.195:0/1130666548 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7b6406e850 msgr2=0x7f7b641660f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:49.484 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.442+0000 7f7b63fff700 1 --2- 172.21.15.195:0/1130666548 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7b6406e850 0x7f7b641660f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:49.484 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.442+0000 7f7b63fff700 1 -- 172.21.15.195:0/1130666548 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b580092e0 con 0x7f7b640c34a0 2022-06-27T12:06:49.485 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.442+0000 7f7b696b3700 1 --2- 172.21.15.195:0/1130666548 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7b640bad20 0x7f7b64155720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:06:49.485 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.442+0000 7f7b63fff700 1 --2- 172.21.15.195:0/1130666548 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7b640c34a0 0x7f7b640ba330 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f7b5000c4f0 tx=0x7f7b50007cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:06:49.485 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.442+0000 7f7b60ff9700 1 -- 172.21.15.195:0/1130666548 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b500072e0 con 0x7f7b640c34a0 2022-06-27T12:06:49.485 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.442+0000 7f7b60ff9700 1 -- 172.21.15.195:0/1130666548 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b50007460 con 0x7f7b640c34a0 2022-06-27T12:06:49.486 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.442+0000 7f7b60ff9700 1 -- 172.21.15.195:0/1130666548 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b500057c0 con 0x7f7b640c34a0 2022-06-27T12:06:49.486 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.442+0000 7f7b6b116700 1 -- 172.21.15.195:0/1130666548 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b5c009a20 con 0x7f7b640c34a0 2022-06-27T12:06:49.486 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.442+0000 7f7b6b116700 1 -- 172.21.15.195:0/1130666548 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b640c02b0 con 0x7f7b640c34a0 2022-06-27T12:06:49.487 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.443+0000 7f7b6b116700 1 -- 172.21.15.195:0/1130666548 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b640bf3e0 con 0x7f7b640c34a0 2022-06-27T12:06:49.487 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.444+0000 7f7b60ff9700 1 -- 172.21.15.195:0/1130666548 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f7b50019030 con 0x7f7b640c34a0 2022-06-27T12:06:49.488 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.444+0000 7f7b60ff9700 1 --2- 172.21.15.195:0/1130666548 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7b4c063d00 0x7f7b4c0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:49.488 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.444+0000 7f7b60ff9700 1 -- 172.21.15.195:0/1130666548 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7062+0+0 (secure 0 0 0) 0x7f7b5007fdf0 con 0x7f7b640c34a0 2022-06-27T12:06:49.488 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.444+0000 7f7b68eb2700 1 --2- 172.21.15.195:0/1130666548 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7b4c063d00 0x7f7b4c0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:49.489 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.444+0000 7f7b68eb2700 1 --2- 172.21.15.195:0/1130666548 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7b4c063d00 0x7f7b4c0661b0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f7b58009d60 tx=0x7f7b58009060 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:06:49.489 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.448+0000 7f7b60ff9700 1 -- 172.21.15.195:0/1130666548 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f7b640bf3e0 con 0x7f7b640c34a0 2022-06-27T12:06:49.676 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.675+0000 7f7b6b116700 1 -- 172.21.15.195:0/1130666548 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7b640bf3e0 con 0x7f7b640c34a0 2022-06-27T12:06:49.677 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.676+0000 7f7b60ff9700 1 -- 172.21.15.195:0/1130666548 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+5371 (secure 0 0 0) 0x7f7b640bf3e0 con 0x7f7b640c34a0 2022-06-27T12:06:49.678 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:06:49.678 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":68,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:06:49.176649+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5591},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5591":{"gid":5591,"name":"b","rank":0,"incarnation":67,"state":"up:reconnect","state_seq":17,"addr":"172.21.15.202:6833/3022591968","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3022591968},{"type":"v1","addr":"172.21.15.202:6833","nonce":3022591968}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:06:49.680 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.678+0000 7f7b6b116700 1 -- 172.21.15.195:0/1130666548 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7b4c063d00 msgr2=0x7f7b4c0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:49.680 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.678+0000 7f7b6b116700 1 --2- 172.21.15.195:0/1130666548 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7b4c063d00 0x7f7b4c0661b0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f7b58009d60 tx=0x7f7b58009060 comp rx=0 tx=0).stop 2022-06-27T12:06:49.681 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.678+0000 7f7b6b116700 1 -- 172.21.15.195:0/1130666548 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7b640c34a0 msgr2=0x7f7b640ba330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:49.681 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.678+0000 7f7b6b116700 1 --2- 172.21.15.195:0/1130666548 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7b640c34a0 0x7f7b640ba330 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f7b5000c4f0 tx=0x7f7b50007cb0 comp rx=0 tx=0).stop 2022-06-27T12:06:49.681 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.678+0000 7f7b6b116700 1 -- 172.21.15.195:0/1130666548 shutdown_connections 2022-06-27T12:06:49.682 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.678+0000 7f7b6b116700 1 --2- 172.21.15.195:0/1130666548 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7b4c063d00 0x7f7b4c0661b0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:49.682 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.678+0000 7f7b6b116700 1 --2- 172.21.15.195:0/1130666548 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7b640bad20 0x7f7b64155720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:49.683 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.678+0000 7f7b6b116700 1 --2- 172.21.15.195:0/1130666548 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7b6406e850 0x7f7b641660f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:49.683 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.678+0000 7f7b6b116700 1 --2- 172.21.15.195:0/1130666548 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7b640c34a0 0x7f7b640ba330 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:49.684 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.678+0000 7f7b6b116700 1 -- 172.21.15.195:0/1130666548 >> 172.21.15.195:0/1130666548 conn(0x7f7b640c6350 msgr2=0x7f7b641506e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:06:49.685 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.678+0000 7f7b6b116700 1 -- 172.21.15.195:0/1130666548 shutdown_connections 2022-06-27T12:06:49.685 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:49.678+0000 7f7b6b116700 1 -- 172.21.15.195:0/1130666548 wait complete. 2022-06-27T12:06:49.685 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 68 2022-06-27T12:06:49.692 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 68, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-06-27T11:47:52.358095+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 68, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5591": { "addr": "172.21.15.202:6833/3022591968", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6832", "nonce": 3022591968, "type": "v2" }, { "addr": "172.21.15.202:6833", "nonce": 3022591968, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5591, "incarnation": 67, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:reconnect", "state_seq": 17 } }, "last_failure": 0, "last_failure_osd_epoch": 58, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-06-27T12:06:49.176649+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5591 } } } ], "standbys": [ { "addr": "172.21.15.202:6837/3305006841", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6836", "nonce": 3305006841, "type": "v2" }, { "addr": "172.21.15.202:6837", "nonce": 3305006841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6835/2060709822", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6834", "nonce": 2060709822, "type": "v2" }, { "addr": "172.21.15.195:6835", "nonce": 2060709822, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6838/1598751652", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6836", "nonce": 1598751652, "type": "v2" }, { "addr": "172.21.15.195:6838", "nonce": 1598751652, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4275, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6839/1313800661", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6837", "nonce": 1313800661, "type": "v2" }, { "addr": "172.21.15.195:6839", "nonce": 1313800661, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4278, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-06-27T12:06:51.693 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:06:51.842 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:51.827+0000 7feb33ee9700 1 Processor -- start 2022-06-27T12:06:51.842 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:51.827+0000 7feb33ee9700 1 -- start start 2022-06-27T12:06:51.843 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:51.828+0000 7feb33ee9700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7feb2c14e490 0x7feb2c06cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:51.843 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:51.828+0000 7feb33ee9700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7feb2c14e860 0x7feb2c0c3840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:51.843 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:51.828+0000 7feb33ee9700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb2c0c33b0 con 0x7feb2c0c3d80 2022-06-27T12:06:51.844 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:51.828+0000 7feb33ee9700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb2c06d520 con 0x7feb2c14e490 2022-06-27T12:06:51.844 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:51.828+0000 7feb33ee9700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb2c06f220 con 0x7feb2c14e860 2022-06-27T12:06:51.845 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:51.828+0000 7feb31c85700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7feb2c14e490 0x7feb2c06cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:51.845 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:51.828+0000 7feb31c85700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7feb2c14e490 0x7feb2c06cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48500/0 (socket says 172.21.15.195:48500) 2022-06-27T12:06:51.845 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:51.828+0000 7feb31c85700 1 -- 172.21.15.195:0/2390723441 learned_addr learned my addr 172.21.15.195:0/2390723441 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:06:51.846 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:51.828+0000 7feb31484700 1 --2- 172.21.15.195:0/2390723441 >> v2:172.21.15.202:3300/0 conn(0x7feb2c14e860 0x7feb2c0c3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:51.846 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:51.828+0000 7feb31c85700 1 -- 172.21.15.195:0/2390723441 >> v2:172.21.15.202:3300/0 conn(0x7feb2c14e860 msgr2=0x7feb2c0c3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:06:51.848 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:51.828+0000 7feb31c85700 1 --2- 172.21.15.195:0/2390723441 >> v2:172.21.15.202:3300/0 conn(0x7feb2c14e860 0x7feb2c0c3840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:51.848 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:51.828+0000 7feb31c85700 1 -- 172.21.15.195:0/2390723441 >> v1:172.21.15.202:6789/0 conn(0x7feb2c0c3d80 legacy=0x7feb2c06f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:06:51.849 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:51.828+0000 7feb31c85700 1 -- 172.21.15.195:0/2390723441 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb2c1542c0 con 0x7feb2c14e490 2022-06-27T12:06:51.849 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:51.828+0000 7feb31484700 1 --2- 172.21.15.195:0/2390723441 >> v2:172.21.15.202:3300/0 conn(0x7feb2c14e860 0x7feb2c0c3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T12:06:51.849 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:51.828+0000 7feb31c85700 1 --2- 172.21.15.195:0/2390723441 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7feb2c14e490 0x7feb2c06cfe0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7feb2c065b80 tx=0x7feb1c009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cde81c08cb74194d server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:06:51.850 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:51.829+0000 7feb30c83700 1 -- 172.21.15.195:0/2390723441 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb1c004030 con 0x7feb2c14e490 2022-06-27T12:06:51.850 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:51.829+0000 7feb30c83700 1 -- 172.21.15.195:0/2390723441 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb1c004190 con 0x7feb2c14e490 2022-06-27T12:06:51.850 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:51.829+0000 7feb30c83700 1 -- 172.21.15.195:0/2390723441 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb1c011630 con 0x7feb2c14e490 2022-06-27T12:06:51.851 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:51.829+0000 7feb33ee9700 1 -- 172.21.15.195:0/2390723441 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7feb2c14e490 msgr2=0x7feb2c06cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:51.851 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:51.829+0000 7feb33ee9700 1 --2- 172.21.15.195:0/2390723441 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7feb2c14e490 0x7feb2c06cfe0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7feb2c065b80 tx=0x7feb1c009120 comp rx=0 tx=0).stop 2022-06-27T12:06:51.851 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:51.829+0000 7feb33ee9700 1 -- 172.21.15.195:0/2390723441 shutdown_connections 2022-06-27T12:06:51.851 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:51.829+0000 7feb33ee9700 1 --2- 172.21.15.195:0/2390723441 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7feb2c14e490 0x7feb2c06cfe0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:51.852 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:51.829+0000 7feb33ee9700 1 --2- 172.21.15.195:0/2390723441 >> v2:172.21.15.202:3300/0 conn(0x7feb2c14e860 0x7feb2c0c3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:51.852 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:51.829+0000 7feb33ee9700 1 -- 172.21.15.195:0/2390723441 >> 172.21.15.195:0/2390723441 conn(0x7feb2c063af0 msgr2=0x7feb2c063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:06:51.852 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:51.829+0000 7feb33ee9700 1 -- 172.21.15.195:0/2390723441 shutdown_connections 2022-06-27T12:06:51.852 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:51.829+0000 7feb33ee9700 1 -- 172.21.15.195:0/2390723441 wait complete. 2022-06-27T12:06:51.853 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:51.830+0000 7feb33ee9700 1 Processor -- start 2022-06-27T12:06:51.853 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:51.830+0000 7feb33ee9700 1 -- start start 2022-06-27T12:06:51.853 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:51.831+0000 7feb33ee9700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7feb2c0c3d80 0x7feb2c156b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:51.853 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:51.831+0000 7feb33ee9700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7feb2c14e490 0x7feb2c14b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:51.854 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:51.831+0000 7feb33ee9700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7feb2c14e860 0x7feb2c068fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:51.854 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:51.831+0000 7feb33ee9700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb2c155cf0 con 0x7feb2c14e860 2022-06-27T12:06:51.854 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:51.831+0000 7feb33ee9700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb2c167430 con 0x7feb2c14e490 2022-06-27T12:06:51.855 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:51.831+0000 7feb33ee9700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb2c16aba0 con 0x7feb2c0c3d80 2022-06-27T12:06:51.855 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:51.831+0000 7feb32486700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7feb2c14e860 0x7feb2c068fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:51.855 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:51.831+0000 7feb32486700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7feb2c14e860 0x7feb2c068fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48508/0 (socket says 172.21.15.195:48508) 2022-06-27T12:06:51.856 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:51.831+0000 7feb32486700 1 -- 172.21.15.195:0/869329018 learned_addr learned my addr 172.21.15.195:0/869329018 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:06:51.856 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:51.831+0000 7feb31c85700 1 --2- 172.21.15.195:0/869329018 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7feb2c0c3d80 0x7feb2c156b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:51.856 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:51.831+0000 7feb31484700 1 --2- 172.21.15.195:0/869329018 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7feb2c14e490 0x7feb2c14b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:51.856 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:51.832+0000 7feb32486700 1 -- 172.21.15.195:0/869329018 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7feb2c0c3d80 msgr2=0x7feb2c156b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:51.856 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:51.832+0000 7feb32486700 1 --2- 172.21.15.195:0/869329018 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7feb2c0c3d80 0x7feb2c156b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:51.857 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:51.832+0000 7feb32486700 1 -- 172.21.15.195:0/869329018 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7feb2c14e490 msgr2=0x7feb2c14b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:51.857 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:51.832+0000 7feb32486700 1 --2- 172.21.15.195:0/869329018 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7feb2c14e490 0x7feb2c14b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:51.857 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:51.832+0000 7feb32486700 1 -- 172.21.15.195:0/869329018 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb1c0093a0 con 0x7feb2c14e860 2022-06-27T12:06:51.857 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:51.832+0000 7feb32486700 1 --2- 172.21.15.195:0/869329018 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7feb2c14e860 0x7feb2c068fe0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7feb2400b810 tx=0x7feb240092d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:06:51.858 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:51.832+0000 7feb19ffb700 1 -- 172.21.15.195:0/869329018 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb24011690 con 0x7feb2c14e860 2022-06-27T12:06:51.858 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:51.832+0000 7feb19ffb700 1 -- 172.21.15.195:0/869329018 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb24011810 con 0x7feb2c14e860 2022-06-27T12:06:51.859 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:51.832+0000 7feb33ee9700 1 -- 172.21.15.195:0/869329018 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb2c166b20 con 0x7feb2c14e860 2022-06-27T12:06:51.859 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:51.832+0000 7feb19ffb700 1 -- 172.21.15.195:0/869329018 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb24011bd0 con 0x7feb2c14e860 2022-06-27T12:06:51.859 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:51.832+0000 7feb33ee9700 1 -- 172.21.15.195:0/869329018 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb2c166de0 con 0x7feb2c14e860 2022-06-27T12:06:51.860 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:51.833+0000 7feb33ee9700 1 -- 172.21.15.195:0/869329018 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb2c1675e0 con 0x7feb2c14e860 2022-06-27T12:06:51.860 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:51.833+0000 7feb19ffb700 1 -- 172.21.15.195:0/869329018 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7feb240103e0 con 0x7feb2c14e860 2022-06-27T12:06:51.861 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:51.834+0000 7feb19ffb700 1 --2- 172.21.15.195:0/869329018 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7feb0c063d30 0x7feb0c0661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:51.861 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:51.834+0000 7feb19ffb700 1 -- 172.21.15.195:0/869329018 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7062+0+0 (secure 0 0 0) 0x7feb240808e0 con 0x7feb2c14e860 2022-06-27T12:06:51.861 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:51.834+0000 7feb31c85700 1 --2- 172.21.15.195:0/869329018 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7feb0c063d30 0x7feb0c0661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:51.862 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:51.834+0000 7feb31c85700 1 --2- 172.21.15.195:0/869329018 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7feb0c063d30 0x7feb0c0661e0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7feb1c009160 tx=0x7feb1c016000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:06:51.862 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:51.838+0000 7feb19ffb700 1 -- 172.21.15.195:0/869329018 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7feb2c1675e0 con 0x7feb2c14e860 2022-06-27T12:06:52.068 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:52.067+0000 7feb33ee9700 1 -- 172.21.15.195:0/869329018 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feb2c1675e0 con 0x7feb2c14e860 2022-06-27T12:06:52.069 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:52.068+0000 7feb19ffb700 1 -- 172.21.15.195:0/869329018 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+5371 (secure 0 0 0) 0x7feb2c1675e0 con 0x7feb2c14e860 2022-06-27T12:06:52.069 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:06:52.070 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":68,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:06:49.176649+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5591},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5591":{"gid":5591,"name":"b","rank":0,"incarnation":67,"state":"up:reconnect","state_seq":17,"addr":"172.21.15.202:6833/3022591968","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3022591968},{"type":"v1","addr":"172.21.15.202:6833","nonce":3022591968}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:06:52.071 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:52.070+0000 7feb33ee9700 1 -- 172.21.15.195:0/869329018 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7feb0c063d30 msgr2=0x7feb0c0661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:52.071 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:52.070+0000 7feb33ee9700 1 --2- 172.21.15.195:0/869329018 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7feb0c063d30 0x7feb0c0661e0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7feb1c009160 tx=0x7feb1c016000 comp rx=0 tx=0).stop 2022-06-27T12:06:52.072 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:52.070+0000 7feb33ee9700 1 -- 172.21.15.195:0/869329018 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7feb2c14e860 msgr2=0x7feb2c068fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:52.072 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:52.070+0000 7feb33ee9700 1 --2- 172.21.15.195:0/869329018 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7feb2c14e860 0x7feb2c068fe0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7feb2400b810 tx=0x7feb240092d0 comp rx=0 tx=0).stop 2022-06-27T12:06:52.072 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:52.070+0000 7feb33ee9700 1 -- 172.21.15.195:0/869329018 shutdown_connections 2022-06-27T12:06:52.072 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:52.070+0000 7feb33ee9700 1 --2- 172.21.15.195:0/869329018 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7feb0c063d30 0x7feb0c0661e0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:52.073 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:52.070+0000 7feb33ee9700 1 --2- 172.21.15.195:0/869329018 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7feb2c14e860 0x7feb2c068fe0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:52.073 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:52.070+0000 7feb33ee9700 1 --2- 172.21.15.195:0/869329018 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7feb2c0c3d80 0x7feb2c156b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:52.073 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:52.070+0000 7feb33ee9700 1 --2- 172.21.15.195:0/869329018 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7feb2c14e490 0x7feb2c14b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:52.073 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:52.070+0000 7feb33ee9700 1 -- 172.21.15.195:0/869329018 >> 172.21.15.195:0/869329018 conn(0x7feb2c063af0 msgr2=0x7feb2c0c9100 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:06:52.074 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:52.070+0000 7feb33ee9700 1 -- 172.21.15.195:0/869329018 shutdown_connections 2022-06-27T12:06:52.074 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:52.070+0000 7feb33ee9700 1 -- 172.21.15.195:0/869329018 wait complete. 2022-06-27T12:06:52.074 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 68 2022-06-27T12:06:52.083 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:06:54.084 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:06:54.240 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.226+0000 7fa6d0f00700 1 Processor -- start 2022-06-27T12:06:54.240 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.226+0000 7fa6d0f00700 1 -- start start 2022-06-27T12:06:54.241 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.227+0000 7fa6d0f00700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa6cc06fef0 0x7fa6cc1570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:54.241 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.227+0000 7fa6d0f00700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fa6cc0c34a0 0x7fa6cc0c2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:54.241 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.227+0000 7fa6d0f00700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6cc06fba0 con 0x7fa6cc06e850 2022-06-27T12:06:54.241 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.227+0000 7fa6d0f00700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6cc1575e0 con 0x7fa6cc0c34a0 2022-06-27T12:06:54.242 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.227+0000 7fa6d0f00700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6cc06f5e0 con 0x7fa6cc06fef0 2022-06-27T12:06:54.242 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.228+0000 7fa6cad9d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fa6cc0c34a0 0x7fa6cc0c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:54.242 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.228+0000 7fa6cad9d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fa6cc0c34a0 0x7fa6cc0c2920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48516/0 (socket says 172.21.15.195:48516) 2022-06-27T12:06:54.242 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.228+0000 7fa6cad9d700 1 -- 172.21.15.195:0/3473213436 learned_addr learned my addr 172.21.15.195:0/3473213436 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:06:54.243 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.228+0000 7fa6c9d9b700 1 --2- 172.21.15.195:0/3473213436 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa6cc06fef0 0x7fa6cc1570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:54.243 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.228+0000 7fa6cad9d700 1 -- 172.21.15.195:0/3473213436 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa6cc06fef0 msgr2=0x7fa6cc1570a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:06:54.244 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.228+0000 7fa6cad9d700 1 --2- 172.21.15.195:0/3473213436 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa6cc06fef0 0x7fa6cc1570a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:54.244 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.228+0000 7fa6cad9d700 1 -- 172.21.15.195:0/3473213436 >> v1:172.21.15.195:6789/0 conn(0x7fa6cc06e850 legacy=0x7fa6cc069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:54.244 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.228+0000 7fa6cad9d700 1 -- 172.21.15.195:0/3473213436 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6cc06f8c0 con 0x7fa6cc0c34a0 2022-06-27T12:06:54.245 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.228+0000 7fa6c959a700 1 -- 172.21.15.195:0/3473213436 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1894705502 0 0) 0x7fa6cc06fba0 con 0x7fa6cc06e850 2022-06-27T12:06:54.245 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.228+0000 7fa6cad9d700 1 --2- 172.21.15.195:0/3473213436 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa6cc0c34a0 0x7fa6cc0c2920 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fa6c4009f20 tx=0x7fa6c4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e55bd37d49c7d688 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:06:54.245 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.228+0000 7fa6c959a700 1 -- 172.21.15.195:0/3473213436 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c400e070 con 0x7fa6cc0c34a0 2022-06-27T12:06:54.245 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.228+0000 7fa6c959a700 1 -- 172.21.15.195:0/3473213436 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6c400ba60 con 0x7fa6cc0c34a0 2022-06-27T12:06:54.246 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.228+0000 7fa6c959a700 1 -- 172.21.15.195:0/3473213436 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c40092e0 con 0x7fa6cc0c34a0 2022-06-27T12:06:54.246 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.229+0000 7fa6d0f00700 1 -- 172.21.15.195:0/3473213436 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa6cc0c34a0 msgr2=0x7fa6cc0c2920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:54.246 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.229+0000 7fa6d0f00700 1 --2- 172.21.15.195:0/3473213436 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa6cc0c34a0 0x7fa6cc0c2920 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fa6c4009f20 tx=0x7fa6c4009840 comp rx=0 tx=0).stop 2022-06-27T12:06:54.246 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.229+0000 7fa6d0f00700 1 -- 172.21.15.195:0/3473213436 shutdown_connections 2022-06-27T12:06:54.247 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.229+0000 7fa6d0f00700 1 --2- 172.21.15.195:0/3473213436 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa6cc0c34a0 0x7fa6cc0c2920 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:54.247 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.229+0000 7fa6d0f00700 1 --2- 172.21.15.195:0/3473213436 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa6cc06fef0 0x7fa6cc1570a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:54.247 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.229+0000 7fa6d0f00700 1 -- 172.21.15.195:0/3473213436 >> 172.21.15.195:0/3473213436 conn(0x7fa6cc0c6350 msgr2=0x7fa6cc0c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:06:54.247 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.229+0000 7fa6d0f00700 1 -- 172.21.15.195:0/3473213436 shutdown_connections 2022-06-27T12:06:54.247 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.229+0000 7fa6d0f00700 1 -- 172.21.15.195:0/3473213436 wait complete. 2022-06-27T12:06:54.248 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.230+0000 7fa6d0f00700 1 Processor -- start 2022-06-27T12:06:54.248 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.230+0000 7fa6d0f00700 1 -- start start 2022-06-27T12:06:54.248 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.231+0000 7fa6d0f00700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa6cc06e850 0x7fa6cc142b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:54.248 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.231+0000 7fa6d0f00700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa6cc0c34a0 0x7fa6cc146a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:54.249 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.231+0000 7fa6d0f00700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa6cc150900 0x7fa6cc1443d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:54.249 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.231+0000 7fa6d0f00700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6cc142950 con 0x7fa6cc0c34a0 2022-06-27T12:06:54.250 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.231+0000 7fa6d0f00700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6cc143f40 con 0x7fa6cc150900 2022-06-27T12:06:54.250 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.231+0000 7fa6d0f00700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6cc143680 con 0x7fa6cc06e850 2022-06-27T12:06:54.250 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.231+0000 7fa6c9d9b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa6cc0c34a0 0x7fa6cc146a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:54.250 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.231+0000 7fa6c9d9b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa6cc0c34a0 0x7fa6cc146a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48520/0 (socket says 172.21.15.195:48520) 2022-06-27T12:06:54.251 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.231+0000 7fa6c9d9b700 1 -- 172.21.15.195:0/105741827 learned_addr learned my addr 172.21.15.195:0/105741827 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:06:54.251 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.231+0000 7fa6cad9d700 1 --2- 172.21.15.195:0/105741827 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa6cc150900 0x7fa6cc1443d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:54.251 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.231+0000 7fa6ca59c700 1 --2- 172.21.15.195:0/105741827 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa6cc06e850 0x7fa6cc142b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:54.251 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.231+0000 7fa6c9d9b700 1 -- 172.21.15.195:0/105741827 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa6cc06e850 msgr2=0x7fa6cc142b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:54.252 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.231+0000 7fa6c9d9b700 1 --2- 172.21.15.195:0/105741827 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa6cc06e850 0x7fa6cc142b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:54.252 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.231+0000 7fa6c9d9b700 1 -- 172.21.15.195:0/105741827 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa6cc150900 msgr2=0x7fa6cc1443d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:54.252 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.231+0000 7fa6c9d9b700 1 --2- 172.21.15.195:0/105741827 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa6cc150900 0x7fa6cc1443d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:54.253 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.231+0000 7fa6c9d9b700 1 -- 172.21.15.195:0/105741827 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6c4009ae0 con 0x7fa6cc0c34a0 2022-06-27T12:06:54.254 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.231+0000 7fa6c9d9b700 1 --2- 172.21.15.195:0/105741827 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa6cc0c34a0 0x7fa6cc146a80 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc00e930 tx=0x7fa6bc009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:06:54.254 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.231+0000 7fa6ba7fc700 1 -- 172.21.15.195:0/105741827 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6bc00ee70 con 0x7fa6cc0c34a0 2022-06-27T12:06:54.254 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.231+0000 7fa6ba7fc700 1 -- 172.21.15.195:0/105741827 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6bc00c850 con 0x7fa6cc0c34a0 2022-06-27T12:06:54.254 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.231+0000 7fa6d0f00700 1 -- 172.21.15.195:0/105741827 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6cc1481f0 con 0x7fa6cc0c34a0 2022-06-27T12:06:54.255 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.231+0000 7fa6ba7fc700 1 -- 172.21.15.195:0/105741827 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6bc00cbf0 con 0x7fa6cc0c34a0 2022-06-27T12:06:54.255 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.232+0000 7fa6d0f00700 1 -- 172.21.15.195:0/105741827 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6cc1438f0 con 0x7fa6cc0c34a0 2022-06-27T12:06:54.255 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.232+0000 7fa6d0f00700 1 -- 172.21.15.195:0/105741827 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6cc0722c0 con 0x7fa6cc0c34a0 2022-06-27T12:06:54.256 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.233+0000 7fa6ba7fc700 1 -- 172.21.15.195:0/105741827 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fa6bc019030 con 0x7fa6cc0c34a0 2022-06-27T12:06:54.256 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.233+0000 7fa6ba7fc700 1 --2- 172.21.15.195:0/105741827 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa6a8063d00 0x7fa6a80661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:54.256 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.233+0000 7fa6ba7fc700 1 -- 172.21.15.195:0/105741827 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7062+0+0 (secure 0 0 0) 0x7fa6bc080be0 con 0x7fa6cc0c34a0 2022-06-27T12:06:54.257 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.233+0000 7fa6ca59c700 1 --2- 172.21.15.195:0/105741827 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa6a8063d00 0x7fa6a80661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:54.257 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.233+0000 7fa6ca59c700 1 --2- 172.21.15.195:0/105741827 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa6a8063d00 0x7fa6a80661b0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fa6c0007fd0 tx=0x7fa6c0009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:06:54.258 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.237+0000 7fa6ba7fc700 1 -- 172.21.15.195:0/105741827 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fa6bc04f3e0 con 0x7fa6cc0c34a0 2022-06-27T12:06:54.463 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.463+0000 7fa6d0f00700 1 -- 172.21.15.195:0/105741827 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6cc144080 con 0x7fa6cc0c34a0 2022-06-27T12:06:54.464 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.463+0000 7fa6ba7fc700 1 -- 172.21.15.195:0/105741827 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+5371 (secure 0 0 0) 0x7fa6bc017090 con 0x7fa6cc0c34a0 2022-06-27T12:06:54.464 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:06:54.464 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":68,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:06:49.176649+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5591},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5591":{"gid":5591,"name":"b","rank":0,"incarnation":67,"state":"up:reconnect","state_seq":17,"addr":"172.21.15.202:6833/3022591968","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3022591968},{"type":"v1","addr":"172.21.15.202:6833","nonce":3022591968}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:06:54.467 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.465+0000 7fa6d0f00700 1 -- 172.21.15.195:0/105741827 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa6a8063d00 msgr2=0x7fa6a80661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:54.467 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.465+0000 7fa6d0f00700 1 --2- 172.21.15.195:0/105741827 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa6a8063d00 0x7fa6a80661b0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fa6c0007fd0 tx=0x7fa6c0009040 comp rx=0 tx=0).stop 2022-06-27T12:06:54.467 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.465+0000 7fa6d0f00700 1 -- 172.21.15.195:0/105741827 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa6cc0c34a0 msgr2=0x7fa6cc146a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:54.467 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.465+0000 7fa6d0f00700 1 --2- 172.21.15.195:0/105741827 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa6cc0c34a0 0x7fa6cc146a80 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc00e930 tx=0x7fa6bc009060 comp rx=0 tx=0).stop 2022-06-27T12:06:54.468 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.465+0000 7fa6d0f00700 1 -- 172.21.15.195:0/105741827 shutdown_connections 2022-06-27T12:06:54.468 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.465+0000 7fa6d0f00700 1 --2- 172.21.15.195:0/105741827 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa6a8063d00 0x7fa6a80661b0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:54.468 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.465+0000 7fa6d0f00700 1 --2- 172.21.15.195:0/105741827 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa6cc150900 0x7fa6cc1443d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:54.468 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.465+0000 7fa6d0f00700 1 --2- 172.21.15.195:0/105741827 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa6cc06e850 0x7fa6cc142b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:54.469 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.465+0000 7fa6d0f00700 1 --2- 172.21.15.195:0/105741827 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa6cc0c34a0 0x7fa6cc146a80 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:54.469 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.465+0000 7fa6d0f00700 1 -- 172.21.15.195:0/105741827 >> 172.21.15.195:0/105741827 conn(0x7fa6cc0c6350 msgr2=0x7fa6cc0c6720 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:06:54.469 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.465+0000 7fa6d0f00700 1 -- 172.21.15.195:0/105741827 shutdown_connections 2022-06-27T12:06:54.469 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:54.465+0000 7fa6d0f00700 1 -- 172.21.15.195:0/105741827 wait complete. 2022-06-27T12:06:54.470 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 68 2022-06-27T12:06:54.479 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:06:56.480 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:06:56.628 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.614+0000 7f86c034f700 1 Processor -- start 2022-06-27T12:06:56.628 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.614+0000 7f86c034f700 1 -- start start 2022-06-27T12:06:56.628 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.615+0000 7f86c034f700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f86b80c2bf0 0x7f86b80c2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:56.629 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.615+0000 7f86c034f700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f86b80c61e0 0x7f86b80c48f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:56.629 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.615+0000 7f86c034f700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86b80bc490 con 0x7f86b80c4e30 2022-06-27T12:06:56.629 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.615+0000 7f86c034f700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86b80c52d0 con 0x7f86b80c61e0 2022-06-27T12:06:56.630 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.615+0000 7f86c034f700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86b806a0e0 con 0x7f86b80c2bf0 2022-06-27T12:06:56.630 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.615+0000 7f86be8ec700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f86b80c4e30 0x7f86b80c3900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:54718/0 (socket says 172.21.15.195:54718) 2022-06-27T12:06:56.630 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.615+0000 7f86be8ec700 1 -- 172.21.15.195:0/3775304544 learned_addr learned my addr 172.21.15.195:0/3775304544 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:06:56.631 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.615+0000 7f86be0eb700 1 --2- 172.21.15.195:0/3775304544 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f86b80c2bf0 0x7f86b80c2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:56.631 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.615+0000 7f86bd8ea700 1 --2- 172.21.15.195:0/3775304544 >> v2:172.21.15.202:3300/0 conn(0x7f86b80c61e0 0x7f86b80c48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:56.631 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.615+0000 7f86bd0e9700 1 -- 172.21.15.195:0/3775304544 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2885893694 0 0) 0x7f86b80bc490 con 0x7f86b80c4e30 2022-06-27T12:06:56.631 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.615+0000 7f86bd0e9700 1 -- 172.21.15.195:0/3775304544 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86a0003430 con 0x7f86b80c4e30 2022-06-27T12:06:56.632 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.615+0000 7f86bd0e9700 1 -- 172.21.15.195:0/3775304544 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2526073962 0 0) 0x7f86a0003430 con 0x7f86b80c4e30 2022-06-27T12:06:56.632 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.615+0000 7f86bd0e9700 1 -- 172.21.15.195:0/3775304544 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f86b80c2bf0 msgr2=0x7f86b80c2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:06:56.632 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.615+0000 7f86bd0e9700 1 --2- 172.21.15.195:0/3775304544 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f86b80c2bf0 0x7f86b80c2fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:56.632 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.615+0000 7f86bd0e9700 1 -- 172.21.15.195:0/3775304544 >> v2:172.21.15.202:3300/0 conn(0x7f86b80c61e0 msgr2=0x7f86b80c48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:06:56.633 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.615+0000 7f86bd0e9700 1 --2- 172.21.15.195:0/3775304544 >> v2:172.21.15.202:3300/0 conn(0x7f86b80c61e0 0x7f86b80c48f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:56.633 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.615+0000 7f86bd0e9700 1 -- 172.21.15.195:0/3775304544 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86b80bc6e0 con 0x7f86b80c4e30 2022-06-27T12:06:56.633 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.615+0000 7f86bd0e9700 1 -- 172.21.15.195:0/3775304544 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7f86b00017f0 con 0x7f86b80c4e30 2022-06-27T12:06:56.633 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.616+0000 7f86bd8ea700 1 --2- 172.21.15.195:0/3775304544 >> v2:172.21.15.202:3300/0 conn(0x7f86b80c61e0 0x7f86b80c48f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:06:56.633 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.616+0000 7f86bd0e9700 1 -- 172.21.15.195:0/3775304544 >> v1:172.21.15.195:6789/0 conn(0x7f86b80c4e30 legacy=0x7f86b80c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:56.634 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.616+0000 7f86bd0e9700 1 --2- 172.21.15.195:0/3775304544 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f86a0003920 0x7f86a0003d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:56.634 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.616+0000 7f86bd0e9700 1 --2- 172.21.15.195:0/3775304544 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f86a00046d0 0x7f86a0004b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:56.634 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.616+0000 7f86bd0e9700 1 --2- 172.21.15.195:0/3775304544 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f86b80c2bf0 0x7f86a0005080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:56.634 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.616+0000 7f86bd0e9700 1 -- 172.21.15.195:0/3775304544 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86b806a0e0 con 0x7f86a0003920 2022-06-27T12:06:56.635 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.616+0000 7f86bd0e9700 1 -- 172.21.15.195:0/3775304544 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a000b920 con 0x7f86a00046d0 2022-06-27T12:06:56.635 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.616+0000 7f86bd0e9700 1 -- 172.21.15.195:0/3775304544 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86a000bae0 con 0x7f86b80c2bf0 2022-06-27T12:06:56.635 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.616+0000 7f86be0eb700 1 --2- 172.21.15.195:0/3775304544 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f86a0003920 0x7f86a0003d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:56.636 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.616+0000 7f86bd8ea700 1 --2- 172.21.15.195:0/3775304544 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f86a00046d0 0x7f86a0004b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:56.636 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.616+0000 7f86be8ec700 1 --2- 172.21.15.195:0/3775304544 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f86b80c2bf0 0x7f86a0005080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:56.637 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.616+0000 7f86be0eb700 1 -- 172.21.15.195:0/3775304544 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f86b80c2bf0 msgr2=0x7f86a0005080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:06:56.637 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.616+0000 7f86be0eb700 1 --2- 172.21.15.195:0/3775304544 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f86b80c2bf0 0x7f86a0005080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:56.638 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.616+0000 7f86be0eb700 1 -- 172.21.15.195:0/3775304544 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f86a00046d0 msgr2=0x7f86a0004b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:06:56.638 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.616+0000 7f86be0eb700 1 --2- 172.21.15.195:0/3775304544 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f86a00046d0 0x7f86a0004b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:56.639 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.616+0000 7f86be0eb700 1 -- 172.21.15.195:0/3775304544 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86a0003430 con 0x7f86a0003920 2022-06-27T12:06:56.639 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.616+0000 7f86be8ec700 1 --2- 172.21.15.195:0/3775304544 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f86b80c2bf0 0x7f86a0005080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:06:56.640 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.616+0000 7f86be0eb700 1 --2- 172.21.15.195:0/3775304544 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f86a0003920 0x7f86a0003d30 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f86b4009ae0 tx=0x7f86b4006cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c87722dce94a8900 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:06:56.640 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.616+0000 7f86bd0e9700 1 -- 172.21.15.195:0/3775304544 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b4009520 con 0x7f86a0003920 2022-06-27T12:06:56.641 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.616+0000 7f86bd0e9700 1 -- 172.21.15.195:0/3775304544 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86b40058c0 con 0x7f86a0003920 2022-06-27T12:06:56.641 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.616+0000 7f86bd0e9700 1 -- 172.21.15.195:0/3775304544 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b4005c60 con 0x7f86a0003920 2022-06-27T12:06:56.641 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.617+0000 7f86c034f700 1 -- 172.21.15.195:0/3775304544 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f86a0003920 msgr2=0x7f86a0003d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:56.642 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.617+0000 7f86c034f700 1 --2- 172.21.15.195:0/3775304544 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f86a0003920 0x7f86a0003d30 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f86b4009ae0 tx=0x7f86b4006cb0 comp rx=0 tx=0).stop 2022-06-27T12:06:56.642 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.617+0000 7f86be8ec700 1 -- 172.21.15.195:0/3775304544 reap_dead start 2022-06-27T12:06:56.643 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.617+0000 7f86c034f700 1 -- 172.21.15.195:0/3775304544 shutdown_connections 2022-06-27T12:06:56.643 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.617+0000 7f86c034f700 1 -- 172.21.15.195:0/3775304544 >> 172.21.15.195:0/3775304544 conn(0x7f86b80c9b00 msgr2=0x7f86b80c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:06:56.643 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.617+0000 7f86c034f700 1 -- 172.21.15.195:0/3775304544 shutdown_connections 2022-06-27T12:06:56.644 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.617+0000 7f86c034f700 1 -- 172.21.15.195:0/3775304544 wait complete. 2022-06-27T12:06:56.644 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.618+0000 7f86c034f700 1 Processor -- start 2022-06-27T12:06:56.644 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.618+0000 7f86c034f700 1 -- start start 2022-06-27T12:06:56.650 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.619+0000 7f86c034f700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f86a0003920 0x7f86b8147370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:56.651 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.619+0000 7f86c034f700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f86b8155d50 0x7f86b80c7b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:56.651 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.619+0000 7f86c034f700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f86b80c80d0 0x7f86b80695d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:56.651 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.619+0000 7f86c034f700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86b8068180 con 0x7f86a0003920 2022-06-27T12:06:56.652 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.619+0000 7f86c034f700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86b80bad30 con 0x7f86b8155d50 2022-06-27T12:06:56.652 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.619+0000 7f86c034f700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86b8069140 con 0x7f86b80c80d0 2022-06-27T12:06:56.653 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.619+0000 7f86be0eb700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f86a0003920 0x7f86b8147370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:56.653 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.619+0000 7f86be0eb700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f86a0003920 0x7f86b8147370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48538/0 (socket says 172.21.15.195:48538) 2022-06-27T12:06:56.654 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.619+0000 7f86be0eb700 1 -- 172.21.15.195:0/614672387 learned_addr learned my addr 172.21.15.195:0/614672387 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:06:56.654 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.619+0000 7f86be8ec700 1 --2- 172.21.15.195:0/614672387 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f86b80c80d0 0x7f86b80695d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:56.654 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.619+0000 7f86bd8ea700 1 --2- 172.21.15.195:0/614672387 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f86b8155d50 0x7f86b80c7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:56.655 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.619+0000 7f86be0eb700 1 -- 172.21.15.195:0/614672387 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f86b80c80d0 msgr2=0x7f86b80695d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:56.655 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.619+0000 7f86be0eb700 1 --2- 172.21.15.195:0/614672387 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f86b80c80d0 0x7f86b80695d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:56.656 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.619+0000 7f86be0eb700 1 -- 172.21.15.195:0/614672387 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f86b8155d50 msgr2=0x7f86b80c7b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:56.656 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.619+0000 7f86be0eb700 1 --2- 172.21.15.195:0/614672387 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f86b8155d50 0x7f86b80c7b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:56.656 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.619+0000 7f86be0eb700 1 -- 172.21.15.195:0/614672387 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86b81478b0 con 0x7f86a0003920 2022-06-27T12:06:56.657 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.619+0000 7f86bd8ea700 1 --2- 172.21.15.195:0/614672387 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f86b8155d50 0x7f86b80c7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:06:56.657 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.619+0000 7f86be0eb700 1 --2- 172.21.15.195:0/614672387 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f86a0003920 0x7f86b8147370 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f86b4006d60 tx=0x7f86b4019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:06:56.657 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.619+0000 7f86a67fc700 1 -- 172.21.15.195:0/614672387 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b4005bd0 con 0x7f86a0003920 2022-06-27T12:06:56.658 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.619+0000 7f86a67fc700 1 -- 172.21.15.195:0/614672387 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86b4004240 con 0x7f86a0003920 2022-06-27T12:06:56.658 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.619+0000 7f86a67fc700 1 -- 172.21.15.195:0/614672387 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b400e640 con 0x7f86a0003920 2022-06-27T12:06:56.658 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.619+0000 7f86c034f700 1 -- 172.21.15.195:0/614672387 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86b80ba770 con 0x7f86a0003920 2022-06-27T12:06:56.658 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.619+0000 7f86c034f700 1 -- 172.21.15.195:0/614672387 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86b8068af0 con 0x7f86a0003920 2022-06-27T12:06:56.659 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.620+0000 7f86c034f700 1 -- 172.21.15.195:0/614672387 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86b80c84a0 con 0x7f86a0003920 2022-06-27T12:06:56.659 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.621+0000 7f86a67fc700 1 -- 172.21.15.195:0/614672387 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f86b4024070 con 0x7f86a0003920 2022-06-27T12:06:56.659 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.621+0000 7f86a67fc700 1 --2- 172.21.15.195:0/614672387 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8698063d00 0x7f86980661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:56.660 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.621+0000 7f86a67fc700 1 -- 172.21.15.195:0/614672387 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7062+0+0 (secure 0 0 0) 0x7f86b40822c0 con 0x7f86a0003920 2022-06-27T12:06:56.660 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.621+0000 7f86bd8ea700 1 --2- 172.21.15.195:0/614672387 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8698063d00 0x7f86980661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:56.660 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.622+0000 7f86bd8ea700 1 --2- 172.21.15.195:0/614672387 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8698063d00 0x7f86980661b0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f86a8006c80 tx=0x7f86a8008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:06:56.660 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.625+0000 7f86a67fc700 1 -- 172.21.15.195:0/614672387 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f86b4050a30 con 0x7f86a0003920 2022-06-27T12:06:56.847 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.847+0000 7f86c034f700 1 -- 172.21.15.195:0/614672387 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f86b80722c0 con 0x7f86a0003920 2022-06-27T12:06:56.848 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.848+0000 7f86a67fc700 1 -- 172.21.15.195:0/614672387 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+5368 (secure 0 0 0) 0x7f86b402b090 con 0x7f86a0003920 2022-06-27T12:06:56.848 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:06:56.849 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":69,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:06:55.189811+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5591},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5591":{"gid":5591,"name":"b","rank":0,"incarnation":67,"state":"up:rejoin","state_seq":19,"addr":"172.21.15.202:6833/3022591968","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3022591968},{"type":"v1","addr":"172.21.15.202:6833","nonce":3022591968}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:06:56.851 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.849+0000 7f86c034f700 1 -- 172.21.15.195:0/614672387 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8698063d00 msgr2=0x7f86980661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:56.851 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.849+0000 7f86c034f700 1 --2- 172.21.15.195:0/614672387 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8698063d00 0x7f86980661b0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f86a8006c80 tx=0x7f86a8008040 comp rx=0 tx=0).stop 2022-06-27T12:06:56.852 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.849+0000 7f86c034f700 1 -- 172.21.15.195:0/614672387 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f86a0003920 msgr2=0x7f86b8147370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:56.852 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.849+0000 7f86c034f700 1 --2- 172.21.15.195:0/614672387 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f86a0003920 0x7f86b8147370 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f86b4006d60 tx=0x7f86b4019000 comp rx=0 tx=0).stop 2022-06-27T12:06:56.852 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.849+0000 7f86c034f700 1 -- 172.21.15.195:0/614672387 shutdown_connections 2022-06-27T12:06:56.853 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.849+0000 7f86c034f700 1 --2- 172.21.15.195:0/614672387 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8698063d00 0x7f86980661b0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:56.853 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.849+0000 7f86c034f700 1 --2- 172.21.15.195:0/614672387 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f86a0003920 0x7f86b8147370 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:56.854 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.849+0000 7f86c034f700 1 --2- 172.21.15.195:0/614672387 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f86b80c80d0 0x7f86b80695d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:56.854 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.849+0000 7f86c034f700 1 --2- 172.21.15.195:0/614672387 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f86b8155d50 0x7f86b80c7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:56.854 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.849+0000 7f86c034f700 1 -- 172.21.15.195:0/614672387 >> 172.21.15.195:0/614672387 conn(0x7f86b80c9b00 msgr2=0x7f86b806ead0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:06:56.855 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.850+0000 7f86c034f700 1 -- 172.21.15.195:0/614672387 shutdown_connections 2022-06-27T12:06:56.855 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:56.850+0000 7f86c034f700 1 -- 172.21.15.195:0/614672387 wait complete. 2022-06-27T12:06:56.856 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 69 2022-06-27T12:06:56.862 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:06:58.863 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:06:58.996 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:58.996+0000 7fa326824700 1 Processor -- start 2022-06-27T12:06:58.997 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:58.996+0000 7fa326824700 1 -- start start 2022-06-27T12:06:59.005 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:58.997+0000 7fa326824700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fa320150500 0x7fa320063af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:59.005 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:58.997+0000 7fa326824700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fa32006a450 0x7fa3200c48d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:59.005 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:58.997+0000 7fa326824700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa320064030 con 0x7fa3201508d0 2022-06-27T12:06:59.005 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:58.997+0000 7fa326824700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3200c4160 con 0x7fa32006a450 2022-06-27T12:06:59.006 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:58.997+0000 7fa326824700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3200c4440 con 0x7fa320150500 2022-06-27T12:06:59.006 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:58.997+0000 7fa31f7fe700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fa32006a450 0x7fa3200c48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:59.006 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:58.997+0000 7fa324dc1700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fa3201508d0 0x7fa3200bbf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:54738/0 (socket says 172.21.15.195:54738) 2022-06-27T12:06:59.006 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:58.997+0000 7fa31f7fe700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fa32006a450 0x7fa3200c48d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48546/0 (socket says 172.21.15.195:48546) 2022-06-27T12:06:59.007 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:58.997+0000 7fa324dc1700 1 -- 172.21.15.195:0/2003549323 learned_addr learned my addr 172.21.15.195:0/2003549323 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:06:59.007 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:58.998+0000 7fa31ffff700 1 --2- 172.21.15.195:0/2003549323 >> v2:172.21.15.202:3300/0 conn(0x7fa320150500 0x7fa320063af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:59.007 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:58.998+0000 7fa31f7fe700 1 -- 172.21.15.195:0/2003549323 >> v2:172.21.15.202:3300/0 conn(0x7fa320150500 msgr2=0x7fa320063af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:06:59.007 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:58.998+0000 7fa31f7fe700 1 --2- 172.21.15.195:0/2003549323 >> v2:172.21.15.202:3300/0 conn(0x7fa320150500 0x7fa320063af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:59.008 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:58.998+0000 7fa31f7fe700 1 -- 172.21.15.195:0/2003549323 >> v1:172.21.15.195:6789/0 conn(0x7fa3201508d0 legacy=0x7fa3200bbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:59.008 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:58.998+0000 7fa31f7fe700 1 -- 172.21.15.195:0/2003549323 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa32015a800 con 0x7fa32006a450 2022-06-27T12:06:59.008 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:58.998+0000 7fa31effd700 1 -- 172.21.15.195:0/2003549323 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1861687990 0 0) 0x7fa318001490 con 0x7fa3201508d0 2022-06-27T12:06:59.008 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:58.998+0000 7fa31ffff700 1 --2- 172.21.15.195:0/2003549323 >> v2:172.21.15.202:3300/0 conn(0x7fa320150500 0x7fa320063af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T12:06:59.009 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:58.998+0000 7fa31f7fe700 1 --2- 172.21.15.195:0/2003549323 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa32006a450 0x7fa3200c48d0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fa32006f840 tx=0x7fa310009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf0316364b452060 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:06:59.009 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:58.998+0000 7fa31effd700 1 -- 172.21.15.195:0/2003549323 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa310011630 con 0x7fa32006a450 2022-06-27T12:06:59.009 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:58.998+0000 7fa31effd700 1 -- 172.21.15.195:0/2003549323 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa310011790 con 0x7fa32006a450 2022-06-27T12:06:59.009 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:58.998+0000 7fa31effd700 1 -- 172.21.15.195:0/2003549323 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa310011b30 con 0x7fa32006a450 2022-06-27T12:06:59.010 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:58.998+0000 7fa326824700 1 -- 172.21.15.195:0/2003549323 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa32006a450 msgr2=0x7fa3200c48d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:59.010 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:58.998+0000 7fa326824700 1 --2- 172.21.15.195:0/2003549323 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa32006a450 0x7fa3200c48d0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fa32006f840 tx=0x7fa310009060 comp rx=0 tx=0).stop 2022-06-27T12:06:59.010 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:58.998+0000 7fa326824700 1 -- 172.21.15.195:0/2003549323 shutdown_connections 2022-06-27T12:06:59.010 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:58.998+0000 7fa326824700 1 --2- 172.21.15.195:0/2003549323 >> v2:172.21.15.202:3300/0 conn(0x7fa320150500 0x7fa320063af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:59.011 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:58.998+0000 7fa326824700 1 --2- 172.21.15.195:0/2003549323 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa32006a450 0x7fa3200c48d0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:59.011 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:58.998+0000 7fa326824700 1 -- 172.21.15.195:0/2003549323 >> 172.21.15.195:0/2003549323 conn(0x7fa3200c9b00 msgr2=0x7fa3200c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:06:59.011 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:58.999+0000 7fa326824700 1 -- 172.21.15.195:0/2003549323 shutdown_connections 2022-06-27T12:06:59.011 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:58.999+0000 7fa326824700 1 -- 172.21.15.195:0/2003549323 wait complete. 2022-06-27T12:06:59.012 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:59.000+0000 7fa326824700 1 Processor -- start 2022-06-27T12:06:59.012 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:59.000+0000 7fa326824700 1 -- start start 2022-06-27T12:06:59.012 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:59.000+0000 7fa326824700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa32006a450 0x7fa3200bf0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:59.012 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:59.000+0000 7fa326824700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa320150500 0x7fa320166ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:59.012 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:59.000+0000 7fa326824700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa3201508d0 0x7fa3200c0800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:59.013 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:59.001+0000 7fa326824700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa320166570 con 0x7fa3201508d0 2022-06-27T12:06:59.013 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:59.001+0000 7fa326824700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa320158d90 con 0x7fa320150500 2022-06-27T12:06:59.013 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:59.001+0000 7fa326824700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa320166810 con 0x7fa32006a450 2022-06-27T12:06:59.014 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:59.001+0000 7fa324dc1700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa3201508d0 0x7fa3200c0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:59.014 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:59.001+0000 7fa324dc1700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa3201508d0 0x7fa3200c0800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48556/0 (socket says 172.21.15.195:48556) 2022-06-27T12:06:59.014 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:59.001+0000 7fa324dc1700 1 -- 172.21.15.195:0/2540478470 learned_addr learned my addr 172.21.15.195:0/2540478470 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:06:59.014 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:59.001+0000 7fa31ffff700 1 --2- 172.21.15.195:0/2540478470 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa32006a450 0x7fa3200bf0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:59.015 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:59.001+0000 7fa31f7fe700 1 --2- 172.21.15.195:0/2540478470 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa320150500 0x7fa320166ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:59.015 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:59.001+0000 7fa324dc1700 1 -- 172.21.15.195:0/2540478470 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa32006a450 msgr2=0x7fa3200bf0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:59.015 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:59.001+0000 7fa324dc1700 1 --2- 172.21.15.195:0/2540478470 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa32006a450 0x7fa3200bf0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:59.015 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:59.001+0000 7fa324dc1700 1 -- 172.21.15.195:0/2540478470 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa320150500 msgr2=0x7fa320166ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:59.016 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:59.001+0000 7fa324dc1700 1 --2- 172.21.15.195:0/2540478470 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa320150500 0x7fa320166ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:59.016 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:59.001+0000 7fa324dc1700 1 -- 172.21.15.195:0/2540478470 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3100092e0 con 0x7fa3201508d0 2022-06-27T12:06:59.016 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:59.001+0000 7fa324dc1700 1 --2- 172.21.15.195:0/2540478470 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa3201508d0 0x7fa3200c0800 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fa31800c810 tx=0x7fa318001540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:06:59.016 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:59.001+0000 7fa317fff700 1 -- 172.21.15.195:0/2540478470 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa318004c80 con 0x7fa3201508d0 2022-06-27T12:06:59.017 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:59.001+0000 7fa317fff700 1 -- 172.21.15.195:0/2540478470 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa318004e20 con 0x7fa3201508d0 2022-06-27T12:06:59.017 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:59.001+0000 7fa317fff700 1 -- 172.21.15.195:0/2540478470 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3180073a0 con 0x7fa3201508d0 2022-06-27T12:06:59.017 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:59.001+0000 7fa326824700 1 -- 172.21.15.195:0/2540478470 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3200c80f0 con 0x7fa3201508d0 2022-06-27T12:06:59.017 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:59.001+0000 7fa326824700 1 -- 172.21.15.195:0/2540478470 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa320150230 con 0x7fa3201508d0 2022-06-27T12:06:59.017 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:59.002+0000 7fa326824700 1 -- 172.21.15.195:0/2540478470 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3200722c0 con 0x7fa3201508d0 2022-06-27T12:06:59.018 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:59.003+0000 7fa317fff700 1 -- 172.21.15.195:0/2540478470 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fa31801a030 con 0x7fa3201508d0 2022-06-27T12:06:59.018 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:59.003+0000 7fa317fff700 1 --2- 172.21.15.195:0/2540478470 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa304063d00 0x7fa3040661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:06:59.018 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:59.003+0000 7fa317fff700 1 -- 172.21.15.195:0/2540478470 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7062+0+0 (secure 0 0 0) 0x7fa31807ff80 con 0x7fa3201508d0 2022-06-27T12:06:59.018 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:59.003+0000 7fa31ffff700 1 --2- 172.21.15.195:0/2540478470 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa304063d00 0x7fa3040661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:06:59.019 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:59.003+0000 7fa31ffff700 1 --2- 172.21.15.195:0/2540478470 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa304063d00 0x7fa3040661b0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fa308006fd0 tx=0x7fa308008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:06:59.019 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:59.008+0000 7fa317fff700 1 -- 172.21.15.195:0/2540478470 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fa31804ac90 con 0x7fa3201508d0 2022-06-27T12:06:59.234 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:59.233+0000 7fa326824700 1 -- 172.21.15.195:0/2540478470 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa320064030 con 0x7fa3201508d0 2022-06-27T12:06:59.234 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:59.234+0000 7fa317fff700 1 -- 172.21.15.195:0/2540478470 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+5368 (secure 0 0 0) 0x7fa320064030 con 0x7fa3201508d0 2022-06-27T12:06:59.235 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:06:59.235 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":69,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:06:55.189811+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5591},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5591":{"gid":5591,"name":"b","rank":0,"incarnation":67,"state":"up:rejoin","state_seq":19,"addr":"172.21.15.202:6833/3022591968","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3022591968},{"type":"v1","addr":"172.21.15.202:6833","nonce":3022591968}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:06:59.238 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:59.236+0000 7fa326824700 1 -- 172.21.15.195:0/2540478470 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa304063d00 msgr2=0x7fa3040661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:59.238 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:59.236+0000 7fa326824700 1 --2- 172.21.15.195:0/2540478470 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa304063d00 0x7fa3040661b0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fa308006fd0 tx=0x7fa308008040 comp rx=0 tx=0).stop 2022-06-27T12:06:59.239 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:59.236+0000 7fa326824700 1 -- 172.21.15.195:0/2540478470 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa3201508d0 msgr2=0x7fa3200c0800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:06:59.239 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:59.236+0000 7fa326824700 1 --2- 172.21.15.195:0/2540478470 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa3201508d0 0x7fa3200c0800 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fa31800c810 tx=0x7fa318001540 comp rx=0 tx=0).stop 2022-06-27T12:06:59.239 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:59.236+0000 7fa326824700 1 -- 172.21.15.195:0/2540478470 shutdown_connections 2022-06-27T12:06:59.240 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:59.236+0000 7fa326824700 1 --2- 172.21.15.195:0/2540478470 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa304063d00 0x7fa3040661b0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:59.240 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:59.236+0000 7fa326824700 1 --2- 172.21.15.195:0/2540478470 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa3201508d0 0x7fa3200c0800 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:59.241 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:59.236+0000 7fa326824700 1 --2- 172.21.15.195:0/2540478470 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa32006a450 0x7fa3200bf0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:59.241 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:59.236+0000 7fa326824700 1 --2- 172.21.15.195:0/2540478470 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa320150500 0x7fa320166ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:06:59.241 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:59.236+0000 7fa326824700 1 -- 172.21.15.195:0/2540478470 >> 172.21.15.195:0/2540478470 conn(0x7fa3200c9b00 msgr2=0x7fa32014fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:06:59.242 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:59.236+0000 7fa326824700 1 -- 172.21.15.195:0/2540478470 shutdown_connections 2022-06-27T12:06:59.242 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:06:59.236+0000 7fa326824700 1 -- 172.21.15.195:0/2540478470 wait complete. 2022-06-27T12:06:59.243 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 69 2022-06-27T12:06:59.248 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:07:01.249 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:07:01.393 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.394+0000 7fc45ccdc700 1 Processor -- start 2022-06-27T12:07:01.393 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.394+0000 7fc45ccdc700 1 -- start start 2022-06-27T12:07:01.396 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.395+0000 7fc45ccdc700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc45806f640 0x7fc458156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:01.396 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.395+0000 7fc45ccdc700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fc4580c4160 0x7fc4580c1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:01.397 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.395+0000 7fc45ccdc700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fc4580c39e0 0x7fc4581535a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:01.397 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.395+0000 7fc45ccdc700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4580c1a20 con 0x7fc4580c39e0 2022-06-27T12:07:01.397 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.395+0000 7fc45ccdc700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4580c1130 con 0x7fc4580c4160 2022-06-27T12:07:01.397 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.395+0000 7fc45ccdc700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc45806ea90 con 0x7fc45806f640 2022-06-27T12:07:01.398 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.395+0000 7fc456d9d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fc4580c39e0 0x7fc4581535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:01.398 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.395+0000 7fc456d9d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fc4580c39e0 0x7fc4581535a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48572/0 (socket says 172.21.15.195:48572) 2022-06-27T12:07:01.398 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.395+0000 7fc456d9d700 1 -- 172.21.15.195:0/419530510 learned_addr learned my addr 172.21.15.195:0/419530510 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:07:01.398 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.395+0000 7fc45659c700 1 --2- 172.21.15.195:0/419530510 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc45806f640 0x7fc458156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:01.399 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.395+0000 7fc455d9b700 1 --2- 172.21.15.195:0/419530510 >> v2:172.21.15.202:3300/0 conn(0x7fc4580c4160 0x7fc4580c1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:01.399 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.395+0000 7fc456d9d700 1 -- 172.21.15.195:0/419530510 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc45806f640 msgr2=0x7fc458156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:07:01.399 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.395+0000 7fc456d9d700 1 --2- 172.21.15.195:0/419530510 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc45806f640 0x7fc458156ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:01.399 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.395+0000 7fc456d9d700 1 -- 172.21.15.195:0/419530510 >> v2:172.21.15.202:3300/0 conn(0x7fc4580c4160 msgr2=0x7fc4580c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:07:01.399 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.395+0000 7fc456d9d700 1 --2- 172.21.15.195:0/419530510 >> v2:172.21.15.202:3300/0 conn(0x7fc4580c4160 0x7fc4580c1eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:01.400 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.395+0000 7fc456d9d700 1 -- 172.21.15.195:0/419530510 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc45806ed70 con 0x7fc4580c39e0 2022-06-27T12:07:01.400 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.395+0000 7fc456d9d700 1 --2- 172.21.15.195:0/419530510 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc4580c39e0 0x7fc4581535a0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fc45806d100 tx=0x7fc450009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6633be4a05569078 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:07:01.400 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.396+0000 7fc45559a700 1 -- 172.21.15.195:0/419530510 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc45000e070 con 0x7fc4580c39e0 2022-06-27T12:07:01.400 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.396+0000 7fc45559a700 1 -- 172.21.15.195:0/419530510 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc45000ba00 con 0x7fc4580c39e0 2022-06-27T12:07:01.401 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.396+0000 7fc45559a700 1 -- 172.21.15.195:0/419530510 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4500092e0 con 0x7fc4580c39e0 2022-06-27T12:07:01.401 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.396+0000 7fc45ccdc700 1 -- 172.21.15.195:0/419530510 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc4580c39e0 msgr2=0x7fc4581535a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:01.401 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.396+0000 7fc45ccdc700 1 --2- 172.21.15.195:0/419530510 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc4580c39e0 0x7fc4581535a0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fc45806d100 tx=0x7fc450009840 comp rx=0 tx=0).stop 2022-06-27T12:07:01.401 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.396+0000 7fc45ccdc700 1 -- 172.21.15.195:0/419530510 shutdown_connections 2022-06-27T12:07:01.401 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.396+0000 7fc45ccdc700 1 --2- 172.21.15.195:0/419530510 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc4580c39e0 0x7fc4581535a0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:01.402 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.396+0000 7fc45ccdc700 1 --2- 172.21.15.195:0/419530510 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc45806f640 0x7fc458156ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:01.402 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.396+0000 7fc45ccdc700 1 --2- 172.21.15.195:0/419530510 >> v2:172.21.15.202:3300/0 conn(0x7fc4580c4160 0x7fc4580c1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:01.402 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.396+0000 7fc45ccdc700 1 -- 172.21.15.195:0/419530510 >> 172.21.15.195:0/419530510 conn(0x7fc4580c6350 msgr2=0x7fc4580c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:07:01.402 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.396+0000 7fc45ccdc700 1 -- 172.21.15.195:0/419530510 shutdown_connections 2022-06-27T12:07:01.403 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.396+0000 7fc45ccdc700 1 -- 172.21.15.195:0/419530510 wait complete. 2022-06-27T12:07:01.403 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.397+0000 7fc45ccdc700 1 Processor -- start 2022-06-27T12:07:01.403 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.397+0000 7fc45ccdc700 1 -- start start 2022-06-27T12:07:01.406 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.398+0000 7fc45ccdc700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc45806f640 0x7fc458167110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:01.406 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.398+0000 7fc45ccdc700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc4580c39e0 0x7fc45814f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:01.407 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.398+0000 7fc45ccdc700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc4580c4160 0x7fc45814f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:01.407 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.398+0000 7fc45ccdc700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc458069fb0 con 0x7fc4580c39e0 2022-06-27T12:07:01.407 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.398+0000 7fc45ccdc700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc458169de0 con 0x7fc45806f640 2022-06-27T12:07:01.407 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.398+0000 7fc45ccdc700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4580c7a00 con 0x7fc4580c4160 2022-06-27T12:07:01.408 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.398+0000 7fc455d9b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc4580c39e0 0x7fc45814f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:01.408 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.398+0000 7fc455d9b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc4580c39e0 0x7fc45814f450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48576/0 (socket says 172.21.15.195:48576) 2022-06-27T12:07:01.408 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.398+0000 7fc455d9b700 1 -- 172.21.15.195:0/765207324 learned_addr learned my addr 172.21.15.195:0/765207324 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:07:01.408 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.398+0000 7fc45659c700 1 --2- 172.21.15.195:0/765207324 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc45806f640 0x7fc458167110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:01.409 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.398+0000 7fc456d9d700 1 --2- 172.21.15.195:0/765207324 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc4580c4160 0x7fc45814f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:01.409 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.398+0000 7fc455d9b700 1 -- 172.21.15.195:0/765207324 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc4580c4160 msgr2=0x7fc45814f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:01.409 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.398+0000 7fc455d9b700 1 --2- 172.21.15.195:0/765207324 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc4580c4160 0x7fc45814f990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:01.409 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.398+0000 7fc455d9b700 1 -- 172.21.15.195:0/765207324 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc45806f640 msgr2=0x7fc458167110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:01.409 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.398+0000 7fc455d9b700 1 --2- 172.21.15.195:0/765207324 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc45806f640 0x7fc458167110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:01.410 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.398+0000 7fc455d9b700 1 -- 172.21.15.195:0/765207324 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc450009ae0 con 0x7fc4580c39e0 2022-06-27T12:07:01.410 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.399+0000 7fc455d9b700 1 --2- 172.21.15.195:0/765207324 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc4580c39e0 0x7fc45814f450 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fc44c00e900 tx=0x7fc44c009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:07:01.410 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.399+0000 7fc4467fc700 1 -- 172.21.15.195:0/765207324 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc44c013680 con 0x7fc4580c39e0 2022-06-27T12:07:01.410 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.399+0000 7fc4467fc700 1 -- 172.21.15.195:0/765207324 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc44c0137e0 con 0x7fc4580c39e0 2022-06-27T12:07:01.411 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.399+0000 7fc45ccdc700 1 -- 172.21.15.195:0/765207324 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc45816a0a0 con 0x7fc4580c39e0 2022-06-27T12:07:01.411 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.399+0000 7fc4467fc700 1 -- 172.21.15.195:0/765207324 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc44c013b80 con 0x7fc4580c39e0 2022-06-27T12:07:01.411 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.399+0000 7fc45ccdc700 1 -- 172.21.15.195:0/765207324 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc45816a660 con 0x7fc4580c39e0 2022-06-27T12:07:01.411 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.400+0000 7fc45ccdc700 1 -- 172.21.15.195:0/765207324 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4580722c0 con 0x7fc4580c39e0 2022-06-27T12:07:01.412 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.400+0000 7fc4467fc700 1 -- 172.21.15.195:0/765207324 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fc44c019030 con 0x7fc4580c39e0 2022-06-27T12:07:01.412 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.400+0000 7fc4467fc700 1 --2- 172.21.15.195:0/765207324 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc434063d00 0x7fc4340661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:01.412 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.401+0000 7fc4467fc700 1 -- 172.21.15.195:0/765207324 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7062+0+0 (secure 0 0 0) 0x7fc44c080920 con 0x7fc4580c39e0 2022-06-27T12:07:01.412 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.401+0000 7fc45659c700 1 --2- 172.21.15.195:0/765207324 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc434063d00 0x7fc4340661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:01.413 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.401+0000 7fc45659c700 1 --2- 172.21.15.195:0/765207324 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc434063d00 0x7fc4340661b0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fc448009c70 tx=0x7fc448009060 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:07:01.413 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.405+0000 7fc4467fc700 1 -- 172.21.15.195:0/765207324 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fc44c04f120 con 0x7fc4580c39e0 2022-06-27T12:07:01.627 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.628+0000 7fc45ccdc700 1 -- 172.21.15.195:0/765207324 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc45806f240 con 0x7fc4580c39e0 2022-06-27T12:07:01.629 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.629+0000 7fc4467fc700 1 -- 172.21.15.195:0/765207324 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+5374 (secure 0 0 0) 0x7fc44c052740 con 0x7fc4580c39e0 2022-06-27T12:07:01.629 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:07:01.630 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":70,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:07:01.205858+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5591},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5591":{"gid":5591,"name":"b","rank":0,"incarnation":67,"state":"up:clientreplay","state_seq":21,"addr":"172.21.15.202:6833/3022591968","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3022591968},{"type":"v1","addr":"172.21.15.202:6833","nonce":3022591968}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:07:01.631 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:07:01.634 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.631+0000 7fc45ccdc700 1 -- 172.21.15.195:0/765207324 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc434063d00 msgr2=0x7fc4340661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:01.634 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.631+0000 7fc45ccdc700 1 --2- 172.21.15.195:0/765207324 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc434063d00 0x7fc4340661b0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fc448009c70 tx=0x7fc448009060 comp rx=0 tx=0).stop 2022-06-27T12:07:01.634 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.631+0000 7fc45ccdc700 1 -- 172.21.15.195:0/765207324 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc4580c39e0 msgr2=0x7fc45814f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:01.635 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.631+0000 7fc45ccdc700 1 --2- 172.21.15.195:0/765207324 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc4580c39e0 0x7fc45814f450 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fc44c00e900 tx=0x7fc44c009060 comp rx=0 tx=0).stop 2022-06-27T12:07:01.635 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.631+0000 7fc45ccdc700 1 -- 172.21.15.195:0/765207324 shutdown_connections 2022-06-27T12:07:01.635 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.631+0000 7fc45ccdc700 1 --2- 172.21.15.195:0/765207324 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc434063d00 0x7fc4340661b0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:01.635 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.631+0000 7fc45ccdc700 1 --2- 172.21.15.195:0/765207324 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc4580c4160 0x7fc45814f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:01.636 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.631+0000 7fc45ccdc700 1 --2- 172.21.15.195:0/765207324 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc45806f640 0x7fc458167110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:01.636 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.631+0000 7fc45ccdc700 1 --2- 172.21.15.195:0/765207324 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc4580c39e0 0x7fc45814f450 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:01.636 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.631+0000 7fc45ccdc700 1 -- 172.21.15.195:0/765207324 >> 172.21.15.195:0/765207324 conn(0x7fc4580c6350 msgr2=0x7fc4580c1db0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:07:01.636 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.631+0000 7fc45ccdc700 1 -- 172.21.15.195:0/765207324 shutdown_connections 2022-06-27T12:07:01.637 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:01.631+0000 7fc45ccdc700 1 -- 172.21.15.195:0/765207324 wait complete. 2022-06-27T12:07:01.637 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 70 2022-06-27T12:07:01.639 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:07:01.644 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 70, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-06-27T11:47:52.358095+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 70, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5591": { "addr": "172.21.15.202:6833/3022591968", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6832", "nonce": 3022591968, "type": "v2" }, { "addr": "172.21.15.202:6833", "nonce": 3022591968, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5591, "incarnation": 67, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:clientreplay", "state_seq": 21 } }, "last_failure": 0, "last_failure_osd_epoch": 58, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-06-27T12:07:01.205858+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5591 } } } ], "standbys": [ { "addr": "172.21.15.202:6837/3305006841", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6836", "nonce": 3305006841, "type": "v2" }, { "addr": "172.21.15.202:6837", "nonce": 3305006841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6835/2060709822", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6834", "nonce": 2060709822, "type": "v2" }, { "addr": "172.21.15.195:6835", "nonce": 2060709822, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6838/1598751652", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6836", "nonce": 1598751652, "type": "v2" }, { "addr": "172.21.15.195:6838", "nonce": 1598751652, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4275, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6839/1313800661", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6837", "nonce": 1313800661, "type": "v2" }, { "addr": "172.21.15.195:6839", "nonce": 1313800661, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4278, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-06-27T12:07:03.646 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:07:03.780 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.779+0000 7efd085ed700 1 Processor -- start 2022-06-27T12:07:03.780 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.779+0000 7efd085ed700 1 -- start start 2022-06-27T12:07:03.787 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.780+0000 7efd085ed700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7efd0014e490 0x7efd0006cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:03.788 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.780+0000 7efd085ed700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd000c33b0 con 0x7efd000c3f50 2022-06-27T12:07:03.788 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.780+0000 7efd085ed700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd0006d520 con 0x7efd0014e860 2022-06-27T12:07:03.788 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.780+0000 7efd085ed700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd0006f220 con 0x7efd0014e490 2022-06-27T12:07:03.789 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.781+0000 7efd06b8a700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7efd000c3f50 0x7efd00157890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:54774/0 (socket says 172.21.15.195:54774) 2022-06-27T12:07:03.789 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.781+0000 7efd06b8a700 1 -- 172.21.15.195:0/1561764444 learned_addr learned my addr 172.21.15.195:0/1561764444 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:07:03.789 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.781+0000 7efd06389700 1 --2- 172.21.15.195:0/1561764444 >> v2:172.21.15.202:3300/0 conn(0x7efd0014e490 0x7efd0006cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:03.789 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.781+0000 7efd05387700 1 -- 172.21.15.195:0/1561764444 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3815536615 0 0) 0x7efd000c33b0 con 0x7efd000c3f50 2022-06-27T12:07:03.790 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.781+0000 7efd05387700 1 -- 172.21.15.195:0/1561764444 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efce8003430 con 0x7efd000c3f50 2022-06-27T12:07:03.790 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.781+0000 7efd05387700 1 -- 172.21.15.195:0/1561764444 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1430746899 0 0) 0x7efd0006d520 con 0x7efd0014e860 2022-06-27T12:07:03.790 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.781+0000 7efd05387700 1 -- 172.21.15.195:0/1561764444 --> v1:172.21.15.202:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd000c33b0 con 0x7efd0014e860 2022-06-27T12:07:03.790 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.781+0000 7efd05387700 1 -- 172.21.15.195:0/1561764444 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3294628931 0 0) 0x7efce8003430 con 0x7efd000c3f50 2022-06-27T12:07:03.791 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.781+0000 7efd05387700 1 -- 172.21.15.195:0/1561764444 >> v2:172.21.15.202:3300/0 conn(0x7efd0014e490 msgr2=0x7efd0006cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:07:03.791 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.781+0000 7efd05387700 1 --2- 172.21.15.195:0/1561764444 >> v2:172.21.15.202:3300/0 conn(0x7efd0014e490 0x7efd0006cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:03.791 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.781+0000 7efd05387700 1 -- 172.21.15.195:0/1561764444 >> v1:172.21.15.202:6789/0 conn(0x7efd0014e860 legacy=0x7efd000c3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:03.791 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.781+0000 7efd05387700 1 -- 172.21.15.195:0/1561764444 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd001542c0 con 0x7efd000c3f50 2022-06-27T12:07:03.792 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.781+0000 7efd05387700 1 -- 172.21.15.195:0/1561764444 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7efcf80017f0 con 0x7efd000c3f50 2022-06-27T12:07:03.792 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.781+0000 7efd05387700 1 -- 172.21.15.195:0/1561764444 >> v1:172.21.15.195:6789/0 conn(0x7efd000c3f50 legacy=0x7efd00157890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:03.792 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.781+0000 7efd05387700 1 --2- 172.21.15.195:0/1561764444 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7efce8003ab0 0x7efce8003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:03.792 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.781+0000 7efd05387700 1 --2- 172.21.15.195:0/1561764444 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efce80046d0 0x7efce8004b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:03.793 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.781+0000 7efd05387700 1 --2- 172.21.15.195:0/1561764444 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7efce8005080 0x7efce800b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:03.793 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.781+0000 7efd05387700 1 -- 172.21.15.195:0/1561764444 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd0006f220 con 0x7efce80046d0 2022-06-27T12:07:03.793 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.781+0000 7efd05387700 1 -- 172.21.15.195:0/1561764444 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efce8010380 con 0x7efce8005080 2022-06-27T12:07:03.793 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.781+0000 7efd05387700 1 -- 172.21.15.195:0/1561764444 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efce8010540 con 0x7efce8003ab0 2022-06-27T12:07:03.794 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.781+0000 7efd05b88700 1 --2- 172.21.15.195:0/1561764444 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efce80046d0 0x7efce8004b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:03.794 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.781+0000 7efd06b8a700 1 --2- 172.21.15.195:0/1561764444 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7efce8005080 0x7efce800b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:03.794 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.781+0000 7efd06389700 1 --2- 172.21.15.195:0/1561764444 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7efce8003ab0 0x7efce8003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:03.794 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.782+0000 7efd05b88700 1 -- 172.21.15.195:0/1561764444 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7efce8003ab0 msgr2=0x7efce8003ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:07:03.795 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.782+0000 7efd05b88700 1 --2- 172.21.15.195:0/1561764444 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7efce8003ab0 0x7efce8003ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:03.795 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.782+0000 7efd05b88700 1 -- 172.21.15.195:0/1561764444 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7efce8005080 msgr2=0x7efce800b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:07:03.795 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.782+0000 7efd05b88700 1 --2- 172.21.15.195:0/1561764444 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7efce8005080 0x7efce800b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:03.795 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.782+0000 7efd05b88700 1 -- 172.21.15.195:0/1561764444 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efce8003430 con 0x7efce80046d0 2022-06-27T12:07:03.796 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.782+0000 7efd06b8a700 1 -- 172.21.15.195:0/1561764444 reap_dead start 2022-06-27T12:07:03.796 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.782+0000 7efd05b88700 1 --2- 172.21.15.195:0/1561764444 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efce80046d0 0x7efce8004b40 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7efcfc0018d0 tx=0x7efcfc007bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1d095aa62e4ea7e server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:07:03.796 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.782+0000 7efd05387700 1 -- 172.21.15.195:0/1561764444 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcfc0050d0 con 0x7efce80046d0 2022-06-27T12:07:03.796 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.782+0000 7efd05387700 1 -- 172.21.15.195:0/1561764444 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcfc005250 con 0x7efce80046d0 2022-06-27T12:07:03.797 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.782+0000 7efd05387700 1 -- 172.21.15.195:0/1561764444 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcfc003750 con 0x7efce80046d0 2022-06-27T12:07:03.797 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.782+0000 7efd085ed700 1 -- 172.21.15.195:0/1561764444 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efce80046d0 msgr2=0x7efce8004b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:03.797 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.782+0000 7efd085ed700 1 --2- 172.21.15.195:0/1561764444 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efce80046d0 0x7efce8004b40 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7efcfc0018d0 tx=0x7efcfc007bf0 comp rx=0 tx=0).stop 2022-06-27T12:07:03.797 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.783+0000 7efd085ed700 1 -- 172.21.15.195:0/1561764444 shutdown_connections 2022-06-27T12:07:03.798 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.783+0000 7efd085ed700 1 --2- 172.21.15.195:0/1561764444 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efce80046d0 0x7efce8004b40 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:03.798 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.783+0000 7efd085ed700 1 -- 172.21.15.195:0/1561764444 >> 172.21.15.195:0/1561764444 conn(0x7efd00063af0 msgr2=0x7efd00063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:07:03.798 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.783+0000 7efd085ed700 1 -- 172.21.15.195:0/1561764444 shutdown_connections 2022-06-27T12:07:03.799 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.783+0000 7efd085ed700 1 -- 172.21.15.195:0/1561764444 wait complete. 2022-06-27T12:07:03.799 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.784+0000 7efd085ed700 1 Processor -- start 2022-06-27T12:07:03.799 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.784+0000 7efd085ed700 1 -- start start 2022-06-27T12:07:03.799 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.784+0000 7efd085ed700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efce80046d0 0x7efd000beb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:03.800 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.784+0000 7efd085ed700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7efd000c7bb0 0x7efd000c0380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:03.800 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.784+0000 7efd085ed700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7efd0014c000 0x7efd001483c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:03.800 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.784+0000 7efd085ed700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd0014aa10 con 0x7efce80046d0 2022-06-27T12:07:03.801 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.784+0000 7efd085ed700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd001525f0 con 0x7efd0014c000 2022-06-27T12:07:03.801 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.784+0000 7efd085ed700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd000c08c0 con 0x7efd000c7bb0 2022-06-27T12:07:03.801 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.785+0000 7efd06389700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efce80046d0 0x7efd000beb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:03.801 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.785+0000 7efd06389700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efce80046d0 0x7efd000beb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48594/0 (socket says 172.21.15.195:48594) 2022-06-27T12:07:03.802 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.785+0000 7efd06389700 1 -- 172.21.15.195:0/2985337900 learned_addr learned my addr 172.21.15.195:0/2985337900 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:07:03.802 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.785+0000 7efd05b88700 1 --2- 172.21.15.195:0/2985337900 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7efd000c7bb0 0x7efd000c0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:03.802 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.785+0000 7efd06b8a700 1 --2- 172.21.15.195:0/2985337900 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7efd0014c000 0x7efd001483c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:03.802 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.785+0000 7efd06389700 1 -- 172.21.15.195:0/2985337900 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7efd000c7bb0 msgr2=0x7efd000c0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:03.803 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.785+0000 7efd06389700 1 --2- 172.21.15.195:0/2985337900 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7efd000c7bb0 0x7efd000c0380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:03.803 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.785+0000 7efd06389700 1 -- 172.21.15.195:0/2985337900 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7efd0014c000 msgr2=0x7efd001483c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:03.803 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.785+0000 7efd06389700 1 --2- 172.21.15.195:0/2985337900 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7efd0014c000 0x7efd001483c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:03.803 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.785+0000 7efd06389700 1 -- 172.21.15.195:0/2985337900 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd0014ac60 con 0x7efce80046d0 2022-06-27T12:07:03.804 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.785+0000 7efd06389700 1 --2- 172.21.15.195:0/2985337900 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efce80046d0 0x7efd000beb20 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7efcf0011a70 tx=0x7efcf0009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:07:03.804 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.785+0000 7efcee7fc700 1 -- 172.21.15.195:0/2985337900 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcf0011eb0 con 0x7efce80046d0 2022-06-27T12:07:03.804 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.785+0000 7efcee7fc700 1 -- 172.21.15.195:0/2985337900 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcf0004240 con 0x7efce80046d0 2022-06-27T12:07:03.804 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.785+0000 7efcee7fc700 1 -- 172.21.15.195:0/2985337900 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcf000f980 con 0x7efce80046d0 2022-06-27T12:07:03.805 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.785+0000 7efd085ed700 1 -- 172.21.15.195:0/2985337900 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd000c7400 con 0x7efce80046d0 2022-06-27T12:07:03.805 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.785+0000 7efd085ed700 1 -- 172.21.15.195:0/2985337900 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd000c8050 con 0x7efce80046d0 2022-06-27T12:07:03.805 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.786+0000 7efd085ed700 1 -- 172.21.15.195:0/2985337900 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd000be970 con 0x7efce80046d0 2022-06-27T12:07:03.806 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.786+0000 7efcee7fc700 1 -- 172.21.15.195:0/2985337900 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7efcf0024070 con 0x7efce80046d0 2022-06-27T12:07:03.806 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.787+0000 7efcee7fc700 1 --2- 172.21.15.195:0/2985337900 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7efce0063d30 0x7efce00661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:03.806 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.787+0000 7efcee7fc700 1 -- 172.21.15.195:0/2985337900 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7062+0+0 (secure 0 0 0) 0x7efcf0018030 con 0x7efce80046d0 2022-06-27T12:07:03.806 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.787+0000 7efd05b88700 1 --2- 172.21.15.195:0/2985337900 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7efce0063d30 0x7efce00661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:03.807 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.787+0000 7efd05b88700 1 --2- 172.21.15.195:0/2985337900 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7efce0063d30 0x7efce00661e0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7efcfc000c00 tx=0x7efcfc015000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:07:03.807 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:03.791+0000 7efcee7fc700 1 -- 172.21.15.195:0/2985337900 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7efcf004f9f0 con 0x7efce80046d0 2022-06-27T12:07:04.021 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:04.021+0000 7efd085ed700 1 -- 172.21.15.195:0/2985337900 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efd000c8240 con 0x7efce80046d0 2022-06-27T12:07:04.022 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:04.022+0000 7efcee7fc700 1 -- 172.21.15.195:0/2985337900 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+5374 (secure 0 0 0) 0x7efcf0022020 con 0x7efce80046d0 2022-06-27T12:07:04.022 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:07:04.023 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":70,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:07:01.205858+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5591},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5591":{"gid":5591,"name":"b","rank":0,"incarnation":67,"state":"up:clientreplay","state_seq":21,"addr":"172.21.15.202:6833/3022591968","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3022591968},{"type":"v1","addr":"172.21.15.202:6833","nonce":3022591968}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:07:04.024 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:04.024+0000 7efd085ed700 1 -- 172.21.15.195:0/2985337900 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7efce0063d30 msgr2=0x7efce00661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:04.025 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:04.024+0000 7efd085ed700 1 --2- 172.21.15.195:0/2985337900 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7efce0063d30 0x7efce00661e0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7efcfc000c00 tx=0x7efcfc015000 comp rx=0 tx=0).stop 2022-06-27T12:07:04.025 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:04.024+0000 7efd085ed700 1 -- 172.21.15.195:0/2985337900 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efce80046d0 msgr2=0x7efd000beb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:04.025 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:04.024+0000 7efd085ed700 1 --2- 172.21.15.195:0/2985337900 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efce80046d0 0x7efd000beb20 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7efcf0011a70 tx=0x7efcf0009060 comp rx=0 tx=0).stop 2022-06-27T12:07:04.025 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:04.024+0000 7efd085ed700 1 -- 172.21.15.195:0/2985337900 shutdown_connections 2022-06-27T12:07:04.026 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:04.024+0000 7efd085ed700 1 --2- 172.21.15.195:0/2985337900 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7efce0063d30 0x7efce00661e0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:04.026 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:04.024+0000 7efd085ed700 1 --2- 172.21.15.195:0/2985337900 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efce80046d0 0x7efd000beb20 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:04.026 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:04.024+0000 7efd085ed700 1 --2- 172.21.15.195:0/2985337900 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7efd0014c000 0x7efd001483c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:04.026 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:04.024+0000 7efd085ed700 1 --2- 172.21.15.195:0/2985337900 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7efd000c7bb0 0x7efd000c0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:04.027 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:04.024+0000 7efd085ed700 1 -- 172.21.15.195:0/2985337900 >> 172.21.15.195:0/2985337900 conn(0x7efd00063af0 msgr2=0x7efd0014c9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:07:04.027 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:04.024+0000 7efd085ed700 1 -- 172.21.15.195:0/2985337900 shutdown_connections 2022-06-27T12:07:04.027 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:04.024+0000 7efd085ed700 1 -- 172.21.15.195:0/2985337900 wait complete. 2022-06-27T12:07:04.028 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 70 2022-06-27T12:07:04.036 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:07:06.038 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:07:06.171 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.171+0000 7fdf7c22a700 1 Processor -- start 2022-06-27T12:07:06.172 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.171+0000 7fdf7c22a700 1 -- start start 2022-06-27T12:07:06.184 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.172+0000 7fdf7c22a700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fdf740c4160 0x7fdf74156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:06.185 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.172+0000 7fdf7c22a700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fdf740c39e0 0x7fdf741535a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:06.186 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.172+0000 7fdf7c22a700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf740c1a20 con 0x7fdf7406f640 2022-06-27T12:07:06.186 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.172+0000 7fdf7c22a700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf740c1130 con 0x7fdf740c39e0 2022-06-27T12:07:06.187 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.172+0000 7fdf7c22a700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf7406ea90 con 0x7fdf740c4160 2022-06-27T12:07:06.187 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.172+0000 7fdf7a7c7700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fdf740c39e0 0x7fdf741535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:06.187 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.172+0000 7fdf79fc6700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fdf7406f640 0x7fdf740c1eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:54792/0 (socket says 172.21.15.195:54792) 2022-06-27T12:07:06.188 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.172+0000 7fdf79fc6700 1 -- 172.21.15.195:0/328234590 learned_addr learned my addr 172.21.15.195:0/328234590 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:07:06.188 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.172+0000 7fdf797c5700 1 --2- 172.21.15.195:0/328234590 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fdf740c4160 0x7fdf74156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:06.189 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.172+0000 7fdf7a7c7700 1 -- 172.21.15.195:0/328234590 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fdf740c4160 msgr2=0x7fdf74156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:07:06.189 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.172+0000 7fdf7a7c7700 1 --2- 172.21.15.195:0/328234590 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fdf740c4160 0x7fdf74156ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:06.190 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.172+0000 7fdf7a7c7700 1 -- 172.21.15.195:0/328234590 >> v1:172.21.15.195:6789/0 conn(0x7fdf7406f640 legacy=0x7fdf740c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:06.190 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.172+0000 7fdf78fc4700 1 -- 172.21.15.195:0/328234590 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3029289296 0 0) 0x7fdf740c1a20 con 0x7fdf7406f640 2022-06-27T12:07:06.191 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.172+0000 7fdf7a7c7700 1 -- 172.21.15.195:0/328234590 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf7406ed70 con 0x7fdf740c39e0 2022-06-27T12:07:06.192 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.172+0000 7fdf7a7c7700 1 --2- 172.21.15.195:0/328234590 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdf740c39e0 0x7fdf741535a0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c009e60 tx=0x7fdf6c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=271a0d05a233048 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:07:06.193 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.172+0000 7fdf78fc4700 1 -- 172.21.15.195:0/328234590 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf6c00e070 con 0x7fdf740c39e0 2022-06-27T12:07:06.193 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.172+0000 7fdf78fc4700 1 -- 172.21.15.195:0/328234590 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf6c00bcc0 con 0x7fdf740c39e0 2022-06-27T12:07:06.194 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.172+0000 7fdf78fc4700 1 -- 172.21.15.195:0/328234590 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf6c012680 con 0x7fdf740c39e0 2022-06-27T12:07:06.194 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.173+0000 7fdf7c22a700 1 -- 172.21.15.195:0/328234590 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdf740c39e0 msgr2=0x7fdf741535a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:06.195 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.173+0000 7fdf7c22a700 1 --2- 172.21.15.195:0/328234590 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdf740c39e0 0x7fdf741535a0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c009e60 tx=0x7fdf6c009780 comp rx=0 tx=0).stop 2022-06-27T12:07:06.195 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.173+0000 7fdf7c22a700 1 -- 172.21.15.195:0/328234590 shutdown_connections 2022-06-27T12:07:06.196 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.173+0000 7fdf7c22a700 1 --2- 172.21.15.195:0/328234590 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdf740c39e0 0x7fdf741535a0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:06.196 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.173+0000 7fdf7c22a700 1 --2- 172.21.15.195:0/328234590 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fdf740c4160 0x7fdf74156ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:06.197 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.173+0000 7fdf7c22a700 1 -- 172.21.15.195:0/328234590 >> 172.21.15.195:0/328234590 conn(0x7fdf740c6350 msgr2=0x7fdf740c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:07:06.197 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.173+0000 7fdf7c22a700 1 -- 172.21.15.195:0/328234590 shutdown_connections 2022-06-27T12:07:06.198 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.173+0000 7fdf7c22a700 1 -- 172.21.15.195:0/328234590 wait complete. 2022-06-27T12:07:06.198 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.175+0000 7fdf7c22a700 1 Processor -- start 2022-06-27T12:07:06.199 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.175+0000 7fdf7c22a700 1 -- start start 2022-06-27T12:07:06.200 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.175+0000 7fdf7c22a700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fdf7406f640 0x7fdf74155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:06.201 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.175+0000 7fdf7c22a700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdf740c39e0 0x7fdf74150730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:06.201 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.176+0000 7fdf7c22a700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fdf740c4160 0x7fdf741697a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:06.214 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.176+0000 7fdf7c22a700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf74166cd0 con 0x7fdf740c39e0 2022-06-27T12:07:06.214 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.176+0000 7fdf7c22a700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf741671c0 con 0x7fdf740c4160 2022-06-27T12:07:06.215 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.176+0000 7fdf7c22a700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf7414f7e0 con 0x7fdf7406f640 2022-06-27T12:07:06.215 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.176+0000 7fdf797c5700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdf740c39e0 0x7fdf74150730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:06.216 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.176+0000 7fdf797c5700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdf740c39e0 0x7fdf74150730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48612/0 (socket says 172.21.15.195:48612) 2022-06-27T12:07:06.216 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.176+0000 7fdf797c5700 1 -- 172.21.15.195:0/1762753552 learned_addr learned my addr 172.21.15.195:0/1762753552 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:07:06.217 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.176+0000 7fdf79fc6700 1 --2- 172.21.15.195:0/1762753552 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fdf7406f640 0x7fdf74155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:06.217 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.176+0000 7fdf7a7c7700 1 --2- 172.21.15.195:0/1762753552 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fdf740c4160 0x7fdf741697a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:06.218 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.176+0000 7fdf797c5700 1 -- 172.21.15.195:0/1762753552 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fdf7406f640 msgr2=0x7fdf74155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:06.219 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.176+0000 7fdf797c5700 1 --2- 172.21.15.195:0/1762753552 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fdf7406f640 0x7fdf74155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:06.220 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.176+0000 7fdf797c5700 1 -- 172.21.15.195:0/1762753552 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fdf740c4160 msgr2=0x7fdf741697a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:06.221 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.176+0000 7fdf797c5700 1 --2- 172.21.15.195:0/1762753552 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fdf740c4160 0x7fdf741697a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:06.221 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.176+0000 7fdf797c5700 1 -- 172.21.15.195:0/1762753552 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf6c009a20 con 0x7fdf740c39e0 2022-06-27T12:07:06.222 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.176+0000 7fdf7a7c7700 1 --2- 172.21.15.195:0/1762753552 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fdf740c4160 0x7fdf741697a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T12:07:06.222 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.176+0000 7fdf797c5700 1 --2- 172.21.15.195:0/1762753552 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdf740c39e0 0x7fdf74150730 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fdf6400d8f0 tx=0x7fdf64008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:07:06.223 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.176+0000 7fdf61ffb700 1 -- 172.21.15.195:0/1762753552 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf640095b0 con 0x7fdf740c39e0 2022-06-27T12:07:06.223 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.176+0000 7fdf61ffb700 1 -- 172.21.15.195:0/1762753552 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf64009710 con 0x7fdf740c39e0 2022-06-27T12:07:06.223 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.176+0000 7fdf7c22a700 1 -- 172.21.15.195:0/1762753552 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf741564f0 con 0x7fdf740c39e0 2022-06-27T12:07:06.224 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.176+0000 7fdf61ffb700 1 -- 172.21.15.195:0/1762753552 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf64009ab0 con 0x7fdf740c39e0 2022-06-27T12:07:06.224 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.177+0000 7fdf7c22a700 1 -- 172.21.15.195:0/1762753552 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf74166ee0 con 0x7fdf740c39e0 2022-06-27T12:07:06.225 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.178+0000 7fdf7c22a700 1 -- 172.21.15.195:0/1762753552 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf7414f030 con 0x7fdf740c39e0 2022-06-27T12:07:06.225 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.178+0000 7fdf61ffb700 1 -- 172.21.15.195:0/1762753552 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fdf64024070 con 0x7fdf740c39e0 2022-06-27T12:07:06.226 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.178+0000 7fdf61ffb700 1 --2- 172.21.15.195:0/1762753552 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fdf54063d00 0x7fdf540661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:06.226 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.179+0000 7fdf61ffb700 1 -- 172.21.15.195:0/1762753552 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7062+0+0 (secure 0 0 0) 0x7fdf64081260 con 0x7fdf740c39e0 2022-06-27T12:07:06.227 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.179+0000 7fdf79fc6700 1 --2- 172.21.15.195:0/1762753552 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fdf54063d00 0x7fdf540661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:06.227 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.179+0000 7fdf79fc6700 1 --2- 172.21.15.195:0/1762753552 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fdf54063d00 0x7fdf540661b0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fdf70007600 tx=0x7fdf70007b00 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:07:06.227 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.183+0000 7fdf61ffb700 1 -- 172.21.15.195:0/1762753552 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fdf7414f030 con 0x7fdf740c39e0 2022-06-27T12:07:06.530 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.528+0000 7fdf7c22a700 1 -- 172.21.15.195:0/1762753552 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf7414f030 con 0x7fdf740c39e0 2022-06-27T12:07:06.530 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.529+0000 7fdf61ffb700 1 -- 172.21.15.195:0/1762753552 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+5374 (secure 0 0 0) 0x7fdf7414f030 con 0x7fdf740c39e0 2022-06-27T12:07:06.531 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:07:06.531 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":70,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:07:01.205858+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5591},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5591":{"gid":5591,"name":"b","rank":0,"incarnation":67,"state":"up:clientreplay","state_seq":21,"addr":"172.21.15.202:6833/3022591968","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3022591968},{"type":"v1","addr":"172.21.15.202:6833","nonce":3022591968}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:07:06.535 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.534+0000 7fdf7c22a700 1 -- 172.21.15.195:0/1762753552 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fdf54063d00 msgr2=0x7fdf540661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:06.536 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.534+0000 7fdf7c22a700 1 --2- 172.21.15.195:0/1762753552 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fdf54063d00 0x7fdf540661b0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fdf70007600 tx=0x7fdf70007b00 comp rx=0 tx=0).stop 2022-06-27T12:07:06.536 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.534+0000 7fdf7c22a700 1 -- 172.21.15.195:0/1762753552 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdf740c39e0 msgr2=0x7fdf74150730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:06.536 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.534+0000 7fdf7c22a700 1 --2- 172.21.15.195:0/1762753552 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdf740c39e0 0x7fdf74150730 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fdf6400d8f0 tx=0x7fdf64008040 comp rx=0 tx=0).stop 2022-06-27T12:07:06.537 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.535+0000 7fdf7c22a700 1 -- 172.21.15.195:0/1762753552 shutdown_connections 2022-06-27T12:07:06.537 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.535+0000 7fdf7c22a700 1 --2- 172.21.15.195:0/1762753552 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fdf54063d00 0x7fdf540661b0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:06.538 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.535+0000 7fdf7c22a700 1 --2- 172.21.15.195:0/1762753552 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fdf740c4160 0x7fdf741697a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:06.538 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.535+0000 7fdf7c22a700 1 --2- 172.21.15.195:0/1762753552 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fdf7406f640 0x7fdf74155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:06.539 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.535+0000 7fdf7c22a700 1 --2- 172.21.15.195:0/1762753552 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdf740c39e0 0x7fdf74150730 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:06.539 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.535+0000 7fdf7c22a700 1 -- 172.21.15.195:0/1762753552 >> 172.21.15.195:0/1762753552 conn(0x7fdf740c6350 msgr2=0x7fdf7406cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:07:06.540 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.536+0000 7fdf7c22a700 1 -- 172.21.15.195:0/1762753552 shutdown_connections 2022-06-27T12:07:06.541 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:06.536+0000 7fdf7c22a700 1 -- 172.21.15.195:0/1762753552 wait complete. 2022-06-27T12:07:06.545 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 70 2022-06-27T12:07:06.560 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:07:08.561 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:07:08.683 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.682+0000 7f11ee089700 1 Processor -- start 2022-06-27T12:07:08.683 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.682+0000 7f11ee089700 1 -- start start 2022-06-27T12:07:08.691 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.683+0000 7f11ee089700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f11e806a450 0x7f11e8063af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:08.692 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.683+0000 7f11ee089700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11e8064030 con 0x7f11e81508d0 2022-06-27T12:07:08.692 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.683+0000 7f11ee089700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11e80c4160 con 0x7f11e8150500 2022-06-27T12:07:08.692 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.683+0000 7f11ee089700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11e80c4440 con 0x7f11e806a450 2022-06-27T12:07:08.693 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.683+0000 7f11e7fff700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f11e81508d0 0x7f11e80bbf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:54824/0 (socket says 172.21.15.195:54824) 2022-06-27T12:07:08.693 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.683+0000 7f11e7fff700 1 -- 172.21.15.195:0/3069002798 learned_addr learned my addr 172.21.15.195:0/3069002798 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:07:08.693 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.684+0000 7f11e6ffd700 1 --2- 172.21.15.195:0/3069002798 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f11e806a450 0x7f11e8063af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:08.693 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.684+0000 7f11e67fc700 1 -- 172.21.15.195:0/3069002798 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3709571473 0 0) 0x7f11e0001490 con 0x7f11e81508d0 2022-06-27T12:07:08.693 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.684+0000 7f11e67fc700 1 -- 172.21.15.195:0/3069002798 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11c8003430 con 0x7f11e81508d0 2022-06-27T12:07:08.694 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.684+0000 7f11e67fc700 1 -- 172.21.15.195:0/3069002798 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1507832174 0 0) 0x7f11e80c4160 con 0x7f11e8150500 2022-06-27T12:07:08.694 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.684+0000 7f11e6ffd700 1 -- 172.21.15.195:0/3069002798 >> v1:172.21.15.202:6789/0 conn(0x7f11e8150500 legacy=0x7f11e80c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:08.694 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.684+0000 7f11e6ffd700 1 -- 172.21.15.195:0/3069002798 >> v1:172.21.15.195:6789/0 conn(0x7f11e81508d0 legacy=0x7f11e80bbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:08.695 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.684+0000 7f11e6ffd700 1 -- 172.21.15.195:0/3069002798 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11e815a800 con 0x7f11e806a450 2022-06-27T12:07:08.695 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.684+0000 7f11e6ffd700 1 --2- 172.21.15.195:0/3069002798 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f11e806a450 0x7f11e8063af0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f11d00097d0 tx=0x7f11d0009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42ff3ef67b01a213 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:07:08.695 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.684+0000 7f11e67fc700 1 -- 172.21.15.195:0/3069002798 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11d0011630 con 0x7f11e806a450 2022-06-27T12:07:08.695 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.684+0000 7f11e67fc700 1 -- 172.21.15.195:0/3069002798 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11d0011790 con 0x7f11e806a450 2022-06-27T12:07:08.697 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.684+0000 7f11e67fc700 1 -- 172.21.15.195:0/3069002798 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11d0011b30 con 0x7f11e806a450 2022-06-27T12:07:08.697 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.685+0000 7f11ee089700 1 -- 172.21.15.195:0/3069002798 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f11e806a450 msgr2=0x7f11e8063af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:08.697 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.685+0000 7f11ee089700 1 --2- 172.21.15.195:0/3069002798 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f11e806a450 0x7f11e8063af0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f11d00097d0 tx=0x7f11d0009060 comp rx=0 tx=0).stop 2022-06-27T12:07:08.697 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.685+0000 7f11ee089700 1 -- 172.21.15.195:0/3069002798 shutdown_connections 2022-06-27T12:07:08.698 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.685+0000 7f11ee089700 1 --2- 172.21.15.195:0/3069002798 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f11e806a450 0x7f11e8063af0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:08.698 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.685+0000 7f11ee089700 1 -- 172.21.15.195:0/3069002798 >> 172.21.15.195:0/3069002798 conn(0x7f11e80c9b00 msgr2=0x7f11e80c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:07:08.698 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.685+0000 7f11ee089700 1 -- 172.21.15.195:0/3069002798 shutdown_connections 2022-06-27T12:07:08.698 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.685+0000 7f11ee089700 1 -- 172.21.15.195:0/3069002798 wait complete. 2022-06-27T12:07:08.699 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.686+0000 7f11ee089700 1 Processor -- start 2022-06-27T12:07:08.699 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.686+0000 7f11ee089700 1 -- start start 2022-06-27T12:07:08.699 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.687+0000 7f11ee089700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f11e806a450 0x7f11e814dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:08.699 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.687+0000 7f11ee089700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f11e8150500 0x7f11e8068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:08.699 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.687+0000 7f11ee089700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f11e81508d0 0x7f11e8068e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:08.700 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.687+0000 7f11ee089700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11e8155570 con 0x7f11e8150500 2022-06-27T12:07:08.700 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.687+0000 7f11ee089700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11e806f160 con 0x7f11e81508d0 2022-06-27T12:07:08.700 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.687+0000 7f11ee089700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11e80c31c0 con 0x7f11e806a450 2022-06-27T12:07:08.700 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.687+0000 7f11e6ffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f11e8150500 0x7f11e8068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:08.703 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.687+0000 7f11e6ffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f11e8150500 0x7f11e8068930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48640/0 (socket says 172.21.15.195:48640) 2022-06-27T12:07:08.703 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.687+0000 7f11e6ffd700 1 -- 172.21.15.195:0/3626219268 learned_addr learned my addr 172.21.15.195:0/3626219268 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:07:08.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.687+0000 7f11e7fff700 1 --2- 172.21.15.195:0/3626219268 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f11e81508d0 0x7f11e8068e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:08.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.687+0000 7f11e77fe700 1 --2- 172.21.15.195:0/3626219268 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f11e806a450 0x7f11e814dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:08.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.687+0000 7f11e6ffd700 1 -- 172.21.15.195:0/3626219268 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f11e806a450 msgr2=0x7f11e814dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:08.705 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.687+0000 7f11e6ffd700 1 --2- 172.21.15.195:0/3626219268 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f11e806a450 0x7f11e814dc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:08.705 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.687+0000 7f11e6ffd700 1 -- 172.21.15.195:0/3626219268 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f11e81508d0 msgr2=0x7f11e8068e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:08.708 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.687+0000 7f11e6ffd700 1 --2- 172.21.15.195:0/3626219268 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f11e81508d0 0x7f11e8068e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:08.708 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.687+0000 7f11e6ffd700 1 -- 172.21.15.195:0/3626219268 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11d00092e0 con 0x7f11e8150500 2022-06-27T12:07:08.708 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.687+0000 7f11e6ffd700 1 --2- 172.21.15.195:0/3626219268 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f11e8150500 0x7f11e8068930 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f11d0009e70 tx=0x7f11d0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:07:08.708 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.687+0000 7f11df7fe700 1 -- 172.21.15.195:0/3626219268 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11d0004590 con 0x7f11e8150500 2022-06-27T12:07:08.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.688+0000 7f11df7fe700 1 -- 172.21.15.195:0/3626219268 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11d0000bc0 con 0x7f11e8150500 2022-06-27T12:07:08.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.688+0000 7f11ee089700 1 -- 172.21.15.195:0/3626219268 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11e80c1270 con 0x7f11e8150500 2022-06-27T12:07:08.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.688+0000 7f11df7fe700 1 -- 172.21.15.195:0/3626219268 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11d0011b30 con 0x7f11e8150500 2022-06-27T12:07:08.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.688+0000 7f11ee089700 1 -- 172.21.15.195:0/3626219268 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11e80c1830 con 0x7f11e8150500 2022-06-27T12:07:08.710 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.688+0000 7f11ee089700 1 -- 172.21.15.195:0/3626219268 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11e80722c0 con 0x7f11e8150500 2022-06-27T12:07:08.710 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.689+0000 7f11df7fe700 1 -- 172.21.15.195:0/3626219268 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f11d000f050 con 0x7f11e8150500 2022-06-27T12:07:08.710 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.689+0000 7f11df7fe700 1 --2- 172.21.15.195:0/3626219268 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f11cc063d00 0x7f11cc0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:08.711 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.689+0000 7f11df7fe700 1 -- 172.21.15.195:0/3626219268 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7062+0+0 (secure 0 0 0) 0x7f11d0081ea0 con 0x7f11e8150500 2022-06-27T12:07:08.711 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.689+0000 7f11e77fe700 1 --2- 172.21.15.195:0/3626219268 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f11cc063d00 0x7f11cc0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:08.713 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.690+0000 7f11e77fe700 1 --2- 172.21.15.195:0/3626219268 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f11cc063d00 0x7f11cc0661b0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f11d8001a80 tx=0x7f11d8009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:07:08.714 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.693+0000 7f11df7fe700 1 -- 172.21.15.195:0/3626219268 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f11d00505b0 con 0x7f11e8150500 2022-06-27T12:07:08.914 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.914+0000 7f11ee089700 1 -- 172.21.15.195:0/3626219268 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f11e80ba8e0 con 0x7f11e8150500 2022-06-27T12:07:08.915 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.914+0000 7f11df7fe700 1 -- 172.21.15.195:0/3626219268 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+5368 (secure 0 0 0) 0x7f11d0015030 con 0x7f11e8150500 2022-06-27T12:07:08.915 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:07:08.916 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:07:08.223429+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5591},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5591":{"gid":5591,"name":"b","rank":0,"incarnation":67,"state":"up:active","state_seq":23,"addr":"172.21.15.202:6833/3022591968","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3022591968},{"type":"v1","addr":"172.21.15.202:6833","nonce":3022591968}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:07:08.917 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.916+0000 7f11ee089700 1 -- 172.21.15.195:0/3626219268 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f11cc063d00 msgr2=0x7f11cc0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:08.918 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.916+0000 7f11ee089700 1 --2- 172.21.15.195:0/3626219268 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f11cc063d00 0x7f11cc0661b0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f11d8001a80 tx=0x7f11d8009040 comp rx=0 tx=0).stop 2022-06-27T12:07:08.918 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.916+0000 7f11ee089700 1 -- 172.21.15.195:0/3626219268 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f11e8150500 msgr2=0x7f11e8068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:08.918 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.916+0000 7f11ee089700 1 --2- 172.21.15.195:0/3626219268 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f11e8150500 0x7f11e8068930 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f11d0009e70 tx=0x7f11d0017000 comp rx=0 tx=0).stop 2022-06-27T12:07:08.919 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.916+0000 7f11ee089700 1 -- 172.21.15.195:0/3626219268 shutdown_connections 2022-06-27T12:07:08.920 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.916+0000 7f11ee089700 1 --2- 172.21.15.195:0/3626219268 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f11cc063d00 0x7f11cc0661b0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:08.920 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.916+0000 7f11ee089700 1 --2- 172.21.15.195:0/3626219268 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f11e81508d0 0x7f11e8068e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:08.920 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.916+0000 7f11ee089700 1 --2- 172.21.15.195:0/3626219268 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f11e806a450 0x7f11e814dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:08.920 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.917+0000 7f11ee089700 1 --2- 172.21.15.195:0/3626219268 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f11e8150500 0x7f11e8068930 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:08.921 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.917+0000 7f11ee089700 1 -- 172.21.15.195:0/3626219268 >> 172.21.15.195:0/3626219268 conn(0x7f11e80c9b00 msgr2=0x7f11e80c7b00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:07:08.921 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.917+0000 7f11ee089700 1 -- 172.21.15.195:0/3626219268 shutdown_connections 2022-06-27T12:07:08.921 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:08.917+0000 7f11ee089700 1 -- 172.21.15.195:0/3626219268 wait complete. 2022-06-27T12:07:08.921 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 71 2022-06-27T12:07:08.929 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=5489 2022-06-27T12:07:08.929 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 5 secs before reviving mds.d 2022-06-27T12:07:13.289 INFO:tasks.ceph.mds.b.smithi202.stderr:2022-06-27T12:07:13.289+0000 7f74bbe2e700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2022-06-27T12:07:13.931 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.d 2022-06-27T12:07:13.931 INFO:tasks.ceph.mds.d:Restarting daemon 2022-06-27T12:07:13.932 DEBUG:teuthology.orchestra.run.smithi202:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2022-06-27T12:07:13.935 INFO:tasks.ceph.mds.d:Started 2022-06-27T12:07:13.936 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:07:14.011 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 9% (51/537) Updating files: 10% (54/537) Updating files: 11% (60/537) Updating files: 12% (65/537) Updating files: 13% (70/537) Updating files: 14% (76/537) Updating files: 15% (81/537) Updating files: 16% (86/537) Updating files: 17% (92/537) Updating files: 18% (97/537) Updating files: 19% (103/537) Updating files: 20% (108/537) Updating files: 21% (113/537) Updating files: 22% (119/537) Updating files: 23% (124/537) Updating files: 24% (129/537) Updating files: 24% (133/537) Updating files: 24% (134/537) Updating files: 25% (135/537) Updating files: 26% (140/537) Updating files: 27% (145/537) Updating files: 28% (151/537) Updating files: 29% (156/537) Updating files: 30% (162/537) Updating files: 31% (167/537) Updating files: 32% (172/537) Updating files: 33% (178/537) Updating files: 34% (183/537) Updating files: 35% (188/537) Updating files: 36% (194/537) Updating files: 37% (199/537) Updating files: 37% (200/537) Updating files: 38% (205/537) Updating files: 39% (210/537) Updating files: 40% (215/537) Updating files: 41% (221/537) Updating files: 42% (226/537) Updating files: 43% (231/537) Updating files: 44% (237/537) Updating files: 45% (242/537) Updating files: 46% (248/537) Updating files: 47% (253/537) Updating files: 48% (258/537) Updating files: 49% (264/537) Updating files: 50% (269/537) Updating files: 51% (274/537) Updating files: 52% (280/537) Updating files: 53% (285/537) Updating files: 54% (290/537) Updating files: 55% (296/537) Updating files: 56% (301/537) Updating files: 57% (307/537) Updating files: 58% (312/537) Updating files: 59% (317/537) Updating files: 60% (323/537) Updating files: 61% (328/537) Updating files: 62% (333/537) Updating files: 62% (336/537) Updating files: 63% (339/537) Updating files: 64% (344/537) Updating files: 65% (350/537) Updating files: 66% (355/537) Updating files: 67% (360/537) Updating files: 68% (366/537) Updating files: 69% (371/537) Updating files: 70% (376/537) Updating files: 71% (382/537) Updating files: 72% (387/537) Updating files: 73% (393/537) Updating files: 74% (398/537) Updating files: 75% (403/537) Updating files: 75% (408/537) Updating files: 76% (409/537) Updating files: 77% (414/537) Updating files: 78% (419/537) Updating files: 79% (425/537) Updating files: 80% (430/537) Updating files: 81% (435/537) Updating files: 82% (441/537) Updating files: 83% (446/537) Updating files: 84% (452/537) Updating files: 85% (457/537) Updating files: 86% (462/537) Updating files: 87% (468/537) Updating files: 88% (473/537) Updating files: 89% (478/537) Updating files: 90% (484/537) Updating files: 91% (489/537) Updating files: 92% (495/537) Updating files: 93% (500/537) Updating files: 94% (505/537) Updating files: 94% (506/537) Updating files: 95% (511/537) Updating files: 96% (516/537) Updating files: 97% (521/537) Updating files: 98% (527/537) Updating files: 99% (532/537) Updating files: 100% (537/537) Updating files: 100% (537/537), done. 2022-06-27T12:07:14.020 INFO:tasks.ceph.mds.d.smithi202.stdout:starting mds.d at 2022-06-27T12:07:14.079 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.079+0000 7f6f314b1700 1 Processor -- start 2022-06-27T12:07:14.080 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.079+0000 7f6f314b1700 1 -- start start 2022-06-27T12:07:14.088 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.080+0000 7f6f314b1700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f6f2c0c6190 0x7f6f2c157010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:14.089 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.080+0000 7f6f314b1700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f2c156b80 con 0x7f6f2c0c40d0 2022-06-27T12:07:14.089 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.080+0000 7f6f314b1700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f2c0682f0 con 0x7f6f2c067ed0 2022-06-27T12:07:14.089 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.080+0000 7f6f314b1700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f2c06a040 con 0x7f6f2c0c6190 2022-06-27T12:07:14.089 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.080+0000 7f6f2a7fc700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f6f2c0c6190 0x7f6f2c157010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:14.090 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.080+0000 7f6f2affd700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f6f2c0c40d0 0x7f6f2c0c44a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:54836/0 (socket says 172.21.15.195:54836) 2022-06-27T12:07:14.090 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.080+0000 7f6f2affd700 1 -- 172.21.15.195:0/1559683538 learned_addr learned my addr 172.21.15.195:0/1559683538 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:07:14.090 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.080+0000 7f6f2a7fc700 1 -- 172.21.15.195:0/1559683538 >> v1:172.21.15.202:6789/0 conn(0x7f6f2c067ed0 legacy=0x7f6f2c14cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:14.091 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.080+0000 7f6f2a7fc700 1 -- 172.21.15.195:0/1559683538 >> v1:172.21.15.195:6789/0 conn(0x7f6f2c0c40d0 legacy=0x7f6f2c0c44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:14.091 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.080+0000 7f6f2a7fc700 1 -- 172.21.15.195:0/1559683538 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f2c06a300 con 0x7f6f2c0c6190 2022-06-27T12:07:14.092 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.080+0000 7f6f29ffb700 1 -- 172.21.15.195:0/1559683538 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1123559370 0 0) 0x7f6f2c0682f0 con 0x7f6f2c067ed0 2022-06-27T12:07:14.092 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.080+0000 7f6f2a7fc700 1 --2- 172.21.15.195:0/1559683538 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6f2c0c6190 0x7f6f2c157010 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f6f200097a0 tx=0x7f6f20009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67c4dc7dbbbda776 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:07:14.092 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.081+0000 7f6f29ffb700 1 -- 172.21.15.195:0/1559683538 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f20011630 con 0x7f6f2c0c6190 2022-06-27T12:07:14.093 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.081+0000 7f6f29ffb700 1 -- 172.21.15.195:0/1559683538 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f20011790 con 0x7f6f2c0c6190 2022-06-27T12:07:14.093 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.081+0000 7f6f29ffb700 1 -- 172.21.15.195:0/1559683538 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f20011b30 con 0x7f6f2c0c6190 2022-06-27T12:07:14.093 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.081+0000 7f6f314b1700 1 -- 172.21.15.195:0/1559683538 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6f2c0c6190 msgr2=0x7f6f2c157010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:14.093 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.081+0000 7f6f314b1700 1 --2- 172.21.15.195:0/1559683538 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6f2c0c6190 0x7f6f2c157010 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f6f200097a0 tx=0x7f6f20009060 comp rx=0 tx=0).stop 2022-06-27T12:07:14.094 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.081+0000 7f6f314b1700 1 -- 172.21.15.195:0/1559683538 shutdown_connections 2022-06-27T12:07:14.094 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.081+0000 7f6f314b1700 1 --2- 172.21.15.195:0/1559683538 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6f2c0c6190 0x7f6f2c157010 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:14.094 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.081+0000 7f6f314b1700 1 -- 172.21.15.195:0/1559683538 >> 172.21.15.195:0/1559683538 conn(0x7f6f2c0ccb80 msgr2=0x7f6f2c0ccf80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:07:14.094 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.081+0000 7f6f314b1700 1 -- 172.21.15.195:0/1559683538 shutdown_connections 2022-06-27T12:07:14.095 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.081+0000 7f6f314b1700 1 -- 172.21.15.195:0/1559683538 wait complete. 2022-06-27T12:07:14.095 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.083+0000 7f6f314b1700 1 Processor -- start 2022-06-27T12:07:14.095 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.083+0000 7f6f314b1700 1 -- start start 2022-06-27T12:07:14.096 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.084+0000 7f6f314b1700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6f2c067ed0 0x7f6f2c152fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:14.096 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.084+0000 7f6f314b1700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6f2c0c40d0 0x7f6f2c14ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:14.096 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.084+0000 7f6f314b1700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6f2c0c6190 0x7f6f2c14d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:14.096 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.084+0000 7f6f314b1700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f2c15b3a0 con 0x7f6f2c0c40d0 2022-06-27T12:07:14.097 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.084+0000 7f6f314b1700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f2c1580a0 con 0x7f6f2c067ed0 2022-06-27T12:07:14.097 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.084+0000 7f6f314b1700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f2c166990 con 0x7f6f2c0c6190 2022-06-27T12:07:14.097 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.084+0000 7f6f2a7fc700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6f2c0c40d0 0x7f6f2c14ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:14.097 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.084+0000 7f6f2a7fc700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6f2c0c40d0 0x7f6f2c14ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48654/0 (socket says 172.21.15.195:48654) 2022-06-27T12:07:14.098 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.084+0000 7f6f2a7fc700 1 -- 172.21.15.195:0/1687899586 learned_addr learned my addr 172.21.15.195:0/1687899586 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:07:14.098 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.084+0000 7f6f2affd700 1 --2- 172.21.15.195:0/1687899586 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6f2c067ed0 0x7f6f2c152fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:14.098 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.084+0000 7f6f2b7fe700 1 --2- 172.21.15.195:0/1687899586 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6f2c0c6190 0x7f6f2c14d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:14.098 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.084+0000 7f6f2a7fc700 1 -- 172.21.15.195:0/1687899586 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6f2c0c6190 msgr2=0x7f6f2c14d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:14.099 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.084+0000 7f6f2a7fc700 1 --2- 172.21.15.195:0/1687899586 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6f2c0c6190 0x7f6f2c14d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:14.099 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.084+0000 7f6f2a7fc700 1 -- 172.21.15.195:0/1687899586 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6f2c067ed0 msgr2=0x7f6f2c152fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:14.099 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.084+0000 7f6f2a7fc700 1 --2- 172.21.15.195:0/1687899586 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6f2c067ed0 0x7f6f2c152fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:14.099 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.084+0000 7f6f2a7fc700 1 -- 172.21.15.195:0/1687899586 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f200092e0 con 0x7f6f2c0c40d0 2022-06-27T12:07:14.100 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.084+0000 7f6f2a7fc700 1 --2- 172.21.15.195:0/1687899586 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6f2c0c40d0 0x7f6f2c14ccb0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f6f2c0697c0 tx=0x7f6f20017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:07:14.100 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.085+0000 7f6f12ffd700 1 -- 172.21.15.195:0/1687899586 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f20000bc0 con 0x7f6f2c0c40d0 2022-06-27T12:07:14.100 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.085+0000 7f6f12ffd700 1 -- 172.21.15.195:0/1687899586 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f20011900 con 0x7f6f2c0c40d0 2022-06-27T12:07:14.100 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.085+0000 7f6f314b1700 1 -- 172.21.15.195:0/1687899586 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f2c166c70 con 0x7f6f2c0c40d0 2022-06-27T12:07:14.101 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.085+0000 7f6f12ffd700 1 -- 172.21.15.195:0/1687899586 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f20025420 con 0x7f6f2c0c40d0 2022-06-27T12:07:14.101 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.085+0000 7f6f314b1700 1 -- 172.21.15.195:0/1687899586 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f2c167230 con 0x7f6f2c0c40d0 2022-06-27T12:07:14.101 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.086+0000 7f6f314b1700 1 -- 172.21.15.195:0/1687899586 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f2c0722c0 con 0x7f6f2c0c40d0 2022-06-27T12:07:14.101 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.086+0000 7f6f12ffd700 1 -- 172.21.15.195:0/1687899586 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f6f2002c030 con 0x7f6f2c0c40d0 2022-06-27T12:07:14.102 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.086+0000 7f6f12ffd700 1 --2- 172.21.15.195:0/1687899586 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6f0c063d00 0x7f6f0c0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:14.102 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.086+0000 7f6f12ffd700 1 -- 172.21.15.195:0/1687899586 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7062+0+0 (secure 0 0 0) 0x7f6f20082120 con 0x7f6f2c0c40d0 2022-06-27T12:07:14.102 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.086+0000 7f6f2affd700 1 --2- 172.21.15.195:0/1687899586 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6f0c063d00 0x7f6f0c0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:14.102 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.087+0000 7f6f2affd700 1 --2- 172.21.15.195:0/1687899586 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6f0c063d00 0x7f6f0c0661b0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f6f1c000c00 tx=0x7f6f1c009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:07:14.103 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.090+0000 7f6f12ffd700 1 -- 172.21.15.195:0/1687899586 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f6f20050920 con 0x7f6f2c0c40d0 2022-06-27T12:07:14.329 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.329+0000 7f6f314b1700 1 -- 172.21.15.195:0/1687899586 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6f2c156550 con 0x7f6f2c0c40d0 2022-06-27T12:07:14.330 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.329+0000 7f6f12ffd700 1 -- 172.21.15.195:0/1687899586 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+6128 (secure 0 0 0) 0x7f6f2002a020 con 0x7f6f2c0c40d0 2022-06-27T12:07:14.331 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:07:14.331 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":6271,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/1603307620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":1603307620},{"type":"v1","addr":"172.21.15.202:6835","nonce":1603307620}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:07:08.223429+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5591},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5591":{"gid":5591,"name":"b","rank":0,"incarnation":67,"state":"up:active","state_seq":23,"addr":"172.21.15.202:6833/3022591968","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3022591968},{"type":"v1","addr":"172.21.15.202:6833","nonce":3022591968}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:07:14.333 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.331+0000 7f6f314b1700 1 -- 172.21.15.195:0/1687899586 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6f0c063d00 msgr2=0x7f6f0c0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:14.334 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.331+0000 7f6f314b1700 1 --2- 172.21.15.195:0/1687899586 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6f0c063d00 0x7f6f0c0661b0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f6f1c000c00 tx=0x7f6f1c009040 comp rx=0 tx=0).stop 2022-06-27T12:07:14.334 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.332+0000 7f6f314b1700 1 -- 172.21.15.195:0/1687899586 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6f2c0c40d0 msgr2=0x7f6f2c14ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:14.334 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.332+0000 7f6f314b1700 1 --2- 172.21.15.195:0/1687899586 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6f2c0c40d0 0x7f6f2c14ccb0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f6f2c0697c0 tx=0x7f6f20017000 comp rx=0 tx=0).stop 2022-06-27T12:07:14.334 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.332+0000 7f6f314b1700 1 -- 172.21.15.195:0/1687899586 shutdown_connections 2022-06-27T12:07:14.335 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.332+0000 7f6f314b1700 1 --2- 172.21.15.195:0/1687899586 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6f0c063d00 0x7f6f0c0661b0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:14.335 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.332+0000 7f6f314b1700 1 --2- 172.21.15.195:0/1687899586 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6f2c0c6190 0x7f6f2c14d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:14.335 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.332+0000 7f6f314b1700 1 --2- 172.21.15.195:0/1687899586 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6f2c067ed0 0x7f6f2c152fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:14.335 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.332+0000 7f6f314b1700 1 --2- 172.21.15.195:0/1687899586 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6f2c0c40d0 0x7f6f2c14ccb0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:14.337 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.332+0000 7f6f314b1700 1 -- 172.21.15.195:0/1687899586 >> 172.21.15.195:0/1687899586 conn(0x7f6f2c0ccb80 msgr2=0x7f6f2c150260 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:07:14.338 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.332+0000 7f6f314b1700 1 -- 172.21.15.195:0/1687899586 shutdown_connections 2022-06-27T12:07:14.338 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:14.332+0000 7f6f314b1700 1 -- 172.21.15.195:0/1687899586 wait complete. 2022-06-27T12:07:14.339 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 72 2022-06-27T12:07:14.346 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d reported in up:standby state 2022-06-27T12:07:14.347 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 19 secs before thrashing 2022-06-27T12:07:15.454 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at 090436f56ac v0.21 2022-06-27T12:07:15.458 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.21 2022-06-27T12:07:15.489 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:07:15.489 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 22 -eq 48 ']' 2022-06-27T12:07:15.490 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.22 2022-06-27T12:07:15.490 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.22 2022-06-27T12:07:15.491 INFO:tasks.workunit.client.0.smithi195.stdout:v0.22 2022-06-27T12:07:15.492 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.22 2022-06-27T12:07:21.398 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 79% (163/204) Updating files: 80% (164/204) Updating files: 81% (166/204) Updating files: 82% (168/204) Updating files: 83% (170/204) Updating files: 84% (172/204) Updating files: 85% (174/204) Updating files: 86% (176/204) Updating files: 87% (178/204) Updating files: 88% (180/204) Updating files: 89% (182/204) Updating files: 90% (184/204) Updating files: 91% (186/204) Updating files: 92% (188/204) Updating files: 93% (190/204) Updating files: 94% (192/204) Updating files: 95% (194/204) Updating files: 96% (196/204) Updating files: 97% (198/204) Updating files: 98% (200/204) Updating files: 99% (202/204) Updating files: 100% (204/204) Updating files: 100% (204/204), done. 2022-06-27T12:07:21.423 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at 8a7c95f60ad v0.22 2022-06-27T12:07:21.427 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.22 2022-06-27T12:07:21.553 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:07:21.554 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 23 -eq 48 ']' 2022-06-27T12:07:21.555 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.23 2022-06-27T12:07:21.555 INFO:tasks.workunit.client.0.smithi195.stdout:v0.23 2022-06-27T12:07:21.556 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.23 2022-06-27T12:07:21.557 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.23 2022-06-27T12:07:24.185 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 49% (81/163) Updating files: 50% (82/163) Updating files: 51% (84/163) Updating files: 52% (85/163) Updating files: 53% (87/163) Updating files: 54% (89/163) Updating files: 55% (90/163) Updating files: 56% (92/163) Updating files: 57% (93/163) Updating files: 58% (95/163) Updating files: 59% (97/163) Updating files: 60% (98/163) Updating files: 61% (100/163) Updating files: 62% (102/163) Updating files: 63% (103/163) Updating files: 64% (105/163) Updating files: 65% (106/163) Updating files: 66% (108/163) Updating files: 67% (110/163) Updating files: 68% (111/163) Updating files: 69% (113/163) Updating files: 70% (115/163) Updating files: 71% (116/163) Updating files: 72% (118/163) Updating files: 73% (119/163) Updating files: 74% (121/163) Updating files: 75% (123/163) Updating files: 76% (124/163) Updating files: 77% (126/163) Updating files: 78% (128/163) Updating files: 79% (129/163) Updating files: 80% (131/163) Updating files: 81% (133/163) Updating files: 82% (134/163) Updating files: 83% (136/163) Updating files: 84% (137/163) Updating files: 85% (139/163) Updating files: 86% (141/163) Updating files: 87% (142/163) Updating files: 88% (144/163) Updating files: 89% (146/163) Updating files: 90% (147/163) Updating files: 91% (149/163) Updating files: 92% (150/163) Updating files: 93% (152/163) Updating files: 94% (154/163) Updating files: 95% (155/163) Updating files: 96% (157/163) Updating files: 97% (159/163) Updating files: 98% (160/163) Updating files: 99% (162/163) Updating files: 100% (163/163) Updating files: 100% (163/163), done. 2022-06-27T12:07:24.213 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at 5d1d8d0c460 v0.23 2022-06-27T12:07:24.214 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.23 2022-06-27T12:07:24.877 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:07:24.877 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 24 -eq 48 ']' 2022-06-27T12:07:24.878 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.24 2022-06-27T12:07:24.878 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.24 2022-06-27T12:07:24.878 INFO:tasks.workunit.client.0.smithi195.stdout:v0.24 2022-06-27T12:07:24.879 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.24 2022-06-27T12:07:26.422 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at 180a4176035 v0.24 2022-06-27T12:07:26.426 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.24 2022-06-27T12:07:27.105 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:07:27.105 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 25 -eq 48 ']' 2022-06-27T12:07:27.106 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.25 2022-06-27T12:07:27.106 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.25 2022-06-27T12:07:27.106 INFO:tasks.workunit.client.0.smithi195.stdout:v0.25 2022-06-27T12:07:27.107 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.25 2022-06-27T12:07:31.716 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:07:31.733 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:07:33.351 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:07:33.487 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.487+0000 7fbbcad89700 1 Processor -- start 2022-06-27T12:07:33.487 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.487+0000 7fbbcad89700 1 -- start start 2022-06-27T12:07:33.494 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.487+0000 7fbbcad89700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbbc4150500 0x7fbbc4063af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:33.494 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.487+0000 7fbbcad89700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fbbc41508d0 0x7fbbc40bbf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:33.494 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.487+0000 7fbbcad89700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbc4064030 con 0x7fbbc406a450 2022-06-27T12:07:33.495 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.487+0000 7fbbcad89700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbc40c4160 con 0x7fbbc41508d0 2022-06-27T12:07:33.495 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.487+0000 7fbbcad89700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc40c4440 con 0x7fbbc4150500 2022-06-27T12:07:33.495 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.488+0000 7fbbc9326700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fbbc41508d0 0x7fbbc40bbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:33.495 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.488+0000 7fbbc8b25700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbbc4150500 0x7fbbc4063af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:33.496 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.488+0000 7fbbc3fff700 1 --1- >> v1:172.21.15.202:6789/0 conn(0x7fbbc406a450 0x7fbbc40c48d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.202:6789/0 says I am v1:172.21.15.195:56778/0 (socket says 172.21.15.195:56778) 2022-06-27T12:07:33.496 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.488+0000 7fbbc3fff700 1 -- 172.21.15.195:0/716900067 learned_addr learned my addr 172.21.15.195:0/716900067 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:07:33.496 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.488+0000 7fbbc9326700 1 -- 172.21.15.195:0/716900067 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbbc4150500 msgr2=0x7fbbc4063af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:07:33.497 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.488+0000 7fbbc9326700 1 --2- 172.21.15.195:0/716900067 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbbc4150500 0x7fbbc4063af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:33.497 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.488+0000 7fbbc9326700 1 -- 172.21.15.195:0/716900067 >> v1:172.21.15.202:6789/0 conn(0x7fbbc406a450 legacy=0x7fbbc40c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:33.497 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.488+0000 7fbbc37fe700 1 -- 172.21.15.195:0/716900067 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 802830723 0 0) 0x7fbbb0001110 con 0x7fbbc406a450 2022-06-27T12:07:33.497 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.488+0000 7fbbc9326700 1 -- 172.21.15.195:0/716900067 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbc415a800 con 0x7fbbc41508d0 2022-06-27T12:07:33.498 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.488+0000 7fbbc8b25700 1 --2- 172.21.15.195:0/716900067 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbbc4150500 0x7fbbc4063af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:07:33.498 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.488+0000 7fbbc9326700 1 --2- 172.21.15.195:0/716900067 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbbc41508d0 0x7fbbc40bbf50 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc009e60 tx=0x7fbbbc009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d6c3e086d08768cf server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:07:33.498 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.488+0000 7fbbc37fe700 1 -- 172.21.15.195:0/716900067 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbbc00e070 con 0x7fbbc41508d0 2022-06-27T12:07:33.499 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.488+0000 7fbbc37fe700 1 -- 172.21.15.195:0/716900067 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbbc00bb60 con 0x7fbbc41508d0 2022-06-27T12:07:33.499 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.488+0000 7fbbc37fe700 1 -- 172.21.15.195:0/716900067 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbbc012430 con 0x7fbbc41508d0 2022-06-27T12:07:33.499 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.489+0000 7fbbcad89700 1 -- 172.21.15.195:0/716900067 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbbc41508d0 msgr2=0x7fbbc40bbf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:33.499 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.489+0000 7fbbcad89700 1 --2- 172.21.15.195:0/716900067 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbbc41508d0 0x7fbbc40bbf50 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc009e60 tx=0x7fbbbc009780 comp rx=0 tx=0).stop 2022-06-27T12:07:33.500 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.489+0000 7fbbcad89700 1 -- 172.21.15.195:0/716900067 shutdown_connections 2022-06-27T12:07:33.500 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.489+0000 7fbbcad89700 1 --2- 172.21.15.195:0/716900067 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbbc41508d0 0x7fbbc40bbf50 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:33.500 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.489+0000 7fbbcad89700 1 --2- 172.21.15.195:0/716900067 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbbc4150500 0x7fbbc4063af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:33.500 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.489+0000 7fbbcad89700 1 -- 172.21.15.195:0/716900067 >> 172.21.15.195:0/716900067 conn(0x7fbbc40c9b00 msgr2=0x7fbbc40c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:07:33.501 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.489+0000 7fbbcad89700 1 -- 172.21.15.195:0/716900067 shutdown_connections 2022-06-27T12:07:33.501 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.489+0000 7fbbcad89700 1 -- 172.21.15.195:0/716900067 wait complete. 2022-06-27T12:07:33.501 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.490+0000 7fbbcad89700 1 Processor -- start 2022-06-27T12:07:33.502 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.490+0000 7fbbcad89700 1 -- start start 2022-06-27T12:07:33.502 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.491+0000 7fbbcad89700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbbc406a450 0x7fbbc406de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:33.502 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.491+0000 7fbbcad89700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbbc4150500 0x7fbbc414da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:33.502 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.491+0000 7fbbcad89700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbbc41508d0 0x7fbbc414df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:33.503 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.491+0000 7fbbcad89700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc40bd9f0 con 0x7fbbc406a450 2022-06-27T12:07:33.503 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.491+0000 7fbbcad89700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc40c2460 con 0x7fbbc4150500 2022-06-27T12:07:33.503 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.491+0000 7fbbcad89700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc41502b0 con 0x7fbbc41508d0 2022-06-27T12:07:33.503 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.491+0000 7fbbc8b25700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbbc406a450 0x7fbbc406de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:33.504 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.491+0000 7fbbc3fff700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbbc4150500 0x7fbbc414da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:33.504 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.491+0000 7fbbc8b25700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbbc406a450 0x7fbbc406de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48732/0 (socket says 172.21.15.195:48732) 2022-06-27T12:07:33.504 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.491+0000 7fbbc9326700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbbc41508d0 0x7fbbc414df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:33.504 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.491+0000 7fbbc8b25700 1 -- 172.21.15.195:0/3331010609 learned_addr learned my addr 172.21.15.195:0/3331010609 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:07:33.505 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.491+0000 7fbbc9326700 1 -- 172.21.15.195:0/3331010609 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbbc4150500 msgr2=0x7fbbc414da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:33.505 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.491+0000 7fbbc9326700 1 --2- 172.21.15.195:0/3331010609 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbbc4150500 0x7fbbc414da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:33.505 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.491+0000 7fbbc9326700 1 -- 172.21.15.195:0/3331010609 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbbc406a450 msgr2=0x7fbbc406de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:33.506 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.491+0000 7fbbc9326700 1 --2- 172.21.15.195:0/3331010609 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbbc406a450 0x7fbbc406de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:33.506 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.491+0000 7fbbc9326700 1 -- 172.21.15.195:0/3331010609 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbb8009290 con 0x7fbbc41508d0 2022-06-27T12:07:33.506 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.491+0000 7fbbc8b25700 1 --2- 172.21.15.195:0/3331010609 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbbc406a450 0x7fbbc406de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:07:33.507 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.491+0000 7fbbc9326700 1 --2- 172.21.15.195:0/3331010609 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbbc41508d0 0x7fbbc414df80 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc016000 tx=0x7fbbbc019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:07:33.507 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.492+0000 7fbbc0ff9700 1 -- 172.21.15.195:0/3331010609 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbbc003e20 con 0x7fbbc41508d0 2022-06-27T12:07:33.507 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.492+0000 7fbbc0ff9700 1 -- 172.21.15.195:0/3331010609 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbbc0129b0 con 0x7fbbc41508d0 2022-06-27T12:07:33.507 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.492+0000 7fbbc0ff9700 1 -- 172.21.15.195:0/3331010609 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbbc012d70 con 0x7fbbc41508d0 2022-06-27T12:07:33.508 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.492+0000 7fbbcad89700 1 -- 172.21.15.195:0/3331010609 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbbc009a20 con 0x7fbbc41508d0 2022-06-27T12:07:33.508 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.492+0000 7fbbcad89700 1 -- 172.21.15.195:0/3331010609 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbc40c1b30 con 0x7fbbc41508d0 2022-06-27T12:07:33.508 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.493+0000 7fbbcad89700 1 -- 172.21.15.195:0/3331010609 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbc40722c0 con 0x7fbbc41508d0 2022-06-27T12:07:33.509 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.493+0000 7fbbc0ff9700 1 -- 172.21.15.195:0/3331010609 <== mon.2 v2:172.21.15.202:3301/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fbbbc004250 con 0x7fbbc41508d0 2022-06-27T12:07:33.509 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.493+0000 7fbbc0ff9700 1 --2- 172.21.15.195:0/3331010609 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbbac063d30 0x7fbbac0661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:33.509 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.493+0000 7fbbc0ff9700 1 -- 172.21.15.195:0/3331010609 <== mon.2 v2:172.21.15.202:3301/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7062+0+0 (secure 0 0 0) 0x7fbbbc083380 con 0x7fbbc41508d0 2022-06-27T12:07:33.509 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.493+0000 7fbbc8b25700 1 --2- 172.21.15.195:0/3331010609 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbbac063d30 0x7fbbac0661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:33.510 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.494+0000 7fbbc8b25700 1 --2- 172.21.15.195:0/3331010609 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbbac063d30 0x7fbbac0661e0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fbbb8009fd0 tx=0x7fbbb8009640 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:07:33.511 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.498+0000 7fbbc0ff9700 1 -- 172.21.15.195:0/3331010609 <== mon.2 v2:172.21.15.202:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fbbbc051b80 con 0x7fbbc41508d0 2022-06-27T12:07:33.748 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.748+0000 7fbbcad89700 1 -- 172.21.15.195:0/3331010609 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbbc40ba8e0 con 0x7fbbc41508d0 2022-06-27T12:07:33.754 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.749+0000 7fbbc0ff9700 1 -- 172.21.15.195:0/3331010609 <== mon.2 v2:172.21.15.202:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+6128 (secure 0 0 0) 0x7fbbbc02c020 con 0x7fbbc41508d0 2022-06-27T12:07:33.755 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:07:33.755 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":6271,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/1603307620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":1603307620},{"type":"v1","addr":"172.21.15.202:6835","nonce":1603307620}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:07:08.223429+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5591},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5591":{"gid":5591,"name":"b","rank":0,"incarnation":67,"state":"up:active","state_seq":23,"addr":"172.21.15.202:6833/3022591968","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3022591968},{"type":"v1","addr":"172.21.15.202:6833","nonce":3022591968}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:07:33.758 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.751+0000 7fbbcad89700 1 -- 172.21.15.195:0/3331010609 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbbac063d30 msgr2=0x7fbbac0661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:33.759 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.751+0000 7fbbcad89700 1 --2- 172.21.15.195:0/3331010609 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbbac063d30 0x7fbbac0661e0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fbbb8009fd0 tx=0x7fbbb8009640 comp rx=0 tx=0).stop 2022-06-27T12:07:33.760 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.751+0000 7fbbcad89700 1 -- 172.21.15.195:0/3331010609 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbbc41508d0 msgr2=0x7fbbc414df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:33.760 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.751+0000 7fbbcad89700 1 --2- 172.21.15.195:0/3331010609 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbbc41508d0 0x7fbbc414df80 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc016000 tx=0x7fbbbc019000 comp rx=0 tx=0).stop 2022-06-27T12:07:33.761 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.751+0000 7fbbcad89700 1 -- 172.21.15.195:0/3331010609 shutdown_connections 2022-06-27T12:07:33.761 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.751+0000 7fbbcad89700 1 --2- 172.21.15.195:0/3331010609 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbbac063d30 0x7fbbac0661e0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:33.761 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.751+0000 7fbbcad89700 1 --2- 172.21.15.195:0/3331010609 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbbc406a450 0x7fbbc406de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:33.762 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.751+0000 7fbbcad89700 1 --2- 172.21.15.195:0/3331010609 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbbc41508d0 0x7fbbc414df80 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:33.762 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.751+0000 7fbbcad89700 1 --2- 172.21.15.195:0/3331010609 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbbc4150500 0x7fbbc414da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:33.763 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.751+0000 7fbbcad89700 1 -- 172.21.15.195:0/3331010609 >> 172.21.15.195:0/3331010609 conn(0x7fbbc40c9b00 msgr2=0x7fbbc40c36d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:07:33.763 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.751+0000 7fbbcad89700 1 -- 172.21.15.195:0/3331010609 shutdown_connections 2022-06-27T12:07:33.763 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.751+0000 7fbbcad89700 1 -- 172.21.15.195:0/3331010609 wait complete. 2022-06-27T12:07:33.764 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 72 2022-06-27T12:07:33.766 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2022-06-27T12:07:33.767 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-06-27T12:07:33.767 INFO:teuthology.orchestra.run:waiting for 300 2022-06-27T12:07:33.780 INFO:tasks.ceph.mds.b:Stopped 2022-06-27T12:07:33.780 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:07:33.902 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.902+0000 7fdff56c6700 1 Processor -- start 2022-06-27T12:07:33.902 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.902+0000 7fdff56c6700 1 -- start start 2022-06-27T12:07:33.906 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.903+0000 7fdff56c6700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fdff00c40d0 0x7fdff0157010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:33.906 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.903+0000 7fdff56c6700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fdff00c6190 0x7fdff0067ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:33.906 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.903+0000 7fdff56c6700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdff0156b80 con 0x7fdff00c44a0 2022-06-27T12:07:33.907 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.903+0000 7fdff56c6700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdff006a040 con 0x7fdff00c40d0 2022-06-27T12:07:33.907 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.903+0000 7fdff56c6700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdff006a300 con 0x7fdff00c6190 2022-06-27T12:07:33.907 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.903+0000 7fdfee7fc700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fdff00c6190 0x7fdff0067ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:33.907 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.903+0000 7fdfef7fe700 1 --1- >> v1:172.21.15.202:6789/0 conn(0x7fdff00c44a0 0x7fdff014cb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.202:6789/0 says I am v1:172.21.15.195:56794/0 (socket says 172.21.15.195:56794) 2022-06-27T12:07:33.907 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.903+0000 7fdfef7fe700 1 -- 172.21.15.195:0/327319177 learned_addr learned my addr 172.21.15.195:0/327319177 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:07:33.908 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.903+0000 7fdfeeffd700 1 --2- 172.21.15.195:0/327319177 >> v2:172.21.15.202:3300/0 conn(0x7fdff00c40d0 0x7fdff0157010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:33.908 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.904+0000 7fdfeeffd700 1 -- 172.21.15.195:0/327319177 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fdff00c6190 msgr2=0x7fdff0067ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:07:33.908 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.904+0000 7fdfeeffd700 1 --2- 172.21.15.195:0/327319177 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fdff00c6190 0x7fdff0067ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:33.908 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.904+0000 7fdfeeffd700 1 -- 172.21.15.195:0/327319177 >> v1:172.21.15.202:6789/0 conn(0x7fdff00c44a0 legacy=0x7fdff014cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:33.909 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.904+0000 7fdfedffb700 1 -- 172.21.15.195:0/327319177 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 424979672 0 0) 0x7fdff0156b80 con 0x7fdff00c44a0 2022-06-27T12:07:33.909 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.904+0000 7fdfeeffd700 1 -- 172.21.15.195:0/327319177 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdff006a5e0 con 0x7fdff00c40d0 2022-06-27T12:07:33.909 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.904+0000 7fdfeeffd700 1 --2- 172.21.15.195:0/327319177 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fdff00c40d0 0x7fdff0157010 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fdfe00097a0 tx=0x7fdfe0009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe087369c184715a server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:07:33.910 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.904+0000 7fdfedffb700 1 -- 172.21.15.195:0/327319177 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfe0011630 con 0x7fdff00c40d0 2022-06-27T12:07:33.910 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.904+0000 7fdfedffb700 1 -- 172.21.15.195:0/327319177 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfe0011790 con 0x7fdff00c40d0 2022-06-27T12:07:33.910 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.905+0000 7fdfedffb700 1 -- 172.21.15.195:0/327319177 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfe0011b30 con 0x7fdff00c40d0 2022-06-27T12:07:33.910 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.905+0000 7fdff56c6700 1 -- 172.21.15.195:0/327319177 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fdff00c40d0 msgr2=0x7fdff0157010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:33.911 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.905+0000 7fdff56c6700 1 --2- 172.21.15.195:0/327319177 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fdff00c40d0 0x7fdff0157010 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fdfe00097a0 tx=0x7fdfe0009060 comp rx=0 tx=0).stop 2022-06-27T12:07:33.911 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.905+0000 7fdff56c6700 1 -- 172.21.15.195:0/327319177 shutdown_connections 2022-06-27T12:07:33.911 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.905+0000 7fdff56c6700 1 --2- 172.21.15.195:0/327319177 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fdff00c40d0 0x7fdff0157010 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:33.911 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.905+0000 7fdff56c6700 1 --2- 172.21.15.195:0/327319177 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fdff00c6190 0x7fdff0067ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:33.912 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.905+0000 7fdff56c6700 1 -- 172.21.15.195:0/327319177 >> 172.21.15.195:0/327319177 conn(0x7fdff00ccb80 msgr2=0x7fdff00ccf80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:07:33.912 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.905+0000 7fdff56c6700 1 -- 172.21.15.195:0/327319177 shutdown_connections 2022-06-27T12:07:33.912 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.906+0000 7fdff56c6700 1 -- 172.21.15.195:0/327319177 wait complete. 2022-06-27T12:07:33.916 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.907+0000 7fdff56c6700 1 Processor -- start 2022-06-27T12:07:33.917 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.907+0000 7fdff56c6700 1 -- start start 2022-06-27T12:07:33.917 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.908+0000 7fdff56c6700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fdff00c40d0 0x7fdff00c06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:33.917 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.908+0000 7fdff56c6700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdff00c44a0 0x7fdff00c2f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:33.917 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.908+0000 7fdff56c6700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fdff00c6190 0x7fdff014edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:33.918 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.908+0000 7fdff56c6700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdff00c2570 con 0x7fdff00c44a0 2022-06-27T12:07:33.918 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.908+0000 7fdff56c6700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdff0154d10 con 0x7fdff00c40d0 2022-06-27T12:07:33.918 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.908+0000 7fdff56c6700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdff016a620 con 0x7fdff00c6190 2022-06-27T12:07:33.918 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.909+0000 7fdfee7fc700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdff00c44a0 0x7fdff00c2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:33.918 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.909+0000 7fdfee7fc700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdff00c44a0 0x7fdff00c2f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48752/0 (socket says 172.21.15.195:48752) 2022-06-27T12:07:33.919 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.909+0000 7fdfee7fc700 1 -- 172.21.15.195:0/1329772558 learned_addr learned my addr 172.21.15.195:0/1329772558 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:07:33.919 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.909+0000 7fdfef7fe700 1 --2- 172.21.15.195:0/1329772558 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fdff00c6190 0x7fdff014edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:33.919 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.909+0000 7fdfeeffd700 1 --2- 172.21.15.195:0/1329772558 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fdff00c40d0 0x7fdff00c06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:33.919 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.909+0000 7fdfee7fc700 1 -- 172.21.15.195:0/1329772558 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fdff00c6190 msgr2=0x7fdff014edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:33.920 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.909+0000 7fdfee7fc700 1 --2- 172.21.15.195:0/1329772558 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fdff00c6190 0x7fdff014edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:33.920 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.909+0000 7fdfee7fc700 1 -- 172.21.15.195:0/1329772558 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fdff00c40d0 msgr2=0x7fdff00c06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:33.920 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.909+0000 7fdfee7fc700 1 --2- 172.21.15.195:0/1329772558 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fdff00c40d0 0x7fdff00c06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:33.920 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.909+0000 7fdfee7fc700 1 -- 172.21.15.195:0/1329772558 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfe00092e0 con 0x7fdff00c44a0 2022-06-27T12:07:33.921 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.909+0000 7fdfef7fe700 1 --2- 172.21.15.195:0/1329772558 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fdff00c6190 0x7fdff014edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:07:33.921 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.909+0000 7fdfee7fc700 1 --2- 172.21.15.195:0/1329772558 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdff00c44a0 0x7fdff00c2f80 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fdfe400e900 tx=0x7fdfe4009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:07:33.921 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.910+0000 7fdfd6ffd700 1 -- 172.21.15.195:0/1329772558 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfe4013680 con 0x7fdff00c44a0 2022-06-27T12:07:33.921 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.910+0000 7fdfd6ffd700 1 -- 172.21.15.195:0/1329772558 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfe40137e0 con 0x7fdff00c44a0 2022-06-27T12:07:33.921 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.910+0000 7fdff56c6700 1 -- 172.21.15.195:0/1329772558 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdff0068bb0 con 0x7fdff00c44a0 2022-06-27T12:07:33.922 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.910+0000 7fdfd6ffd700 1 -- 172.21.15.195:0/1329772558 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfe4013b80 con 0x7fdff00c44a0 2022-06-27T12:07:33.922 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.910+0000 7fdff56c6700 1 -- 172.21.15.195:0/1329772558 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdff00c2780 con 0x7fdff00c44a0 2022-06-27T12:07:33.922 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.911+0000 7fdff56c6700 1 -- 172.21.15.195:0/1329772558 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdff00722c0 con 0x7fdff00c44a0 2022-06-27T12:07:33.922 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.911+0000 7fdfd6ffd700 1 -- 172.21.15.195:0/1329772558 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fdfe4019030 con 0x7fdff00c44a0 2022-06-27T12:07:33.923 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.912+0000 7fdfd6ffd700 1 --2- 172.21.15.195:0/1329772558 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fdfd0063d00 0x7fdfd00661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:33.923 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.912+0000 7fdfd6ffd700 1 -- 172.21.15.195:0/1329772558 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7062+0+0 (secure 0 0 0) 0x7fdfe40809f0 con 0x7fdff00c44a0 2022-06-27T12:07:33.923 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.912+0000 7fdfeeffd700 1 --2- 172.21.15.195:0/1329772558 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fdfd0063d00 0x7fdfd00661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:33.923 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.912+0000 7fdfeeffd700 1 --2- 172.21.15.195:0/1329772558 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fdfd0063d00 0x7fdfd00661b0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fdfe0009130 tx=0x7fdfe0017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:07:33.923 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:33.916+0000 7fdfd6ffd700 1 -- 172.21.15.195:0/1329772558 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fdfe4082e60 con 0x7fdff00c44a0 2022-06-27T12:07:34.143 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:34.142+0000 7fdff56c6700 1 -- 172.21.15.195:0/1329772558 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdff015a8d0 con 0x7fdff00c44a0 2022-06-27T12:07:34.144 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:34.143+0000 7fdfd6ffd700 1 -- 172.21.15.195:0/1329772558 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+6128 (secure 0 0 0) 0x7fdfe4017090 con 0x7fdff00c44a0 2022-06-27T12:07:34.145 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:07:34.146 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":6271,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/1603307620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":1603307620},{"type":"v1","addr":"172.21.15.202:6835","nonce":1603307620}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:07:08.223429+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5591},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5591":{"gid":5591,"name":"b","rank":0,"incarnation":67,"state":"up:active","state_seq":23,"addr":"172.21.15.202:6833/3022591968","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3022591968},{"type":"v1","addr":"172.21.15.202:6833","nonce":3022591968}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:07:34.148 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:34.145+0000 7fdff56c6700 1 -- 172.21.15.195:0/1329772558 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fdfd0063d00 msgr2=0x7fdfd00661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:34.149 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:34.145+0000 7fdff56c6700 1 --2- 172.21.15.195:0/1329772558 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fdfd0063d00 0x7fdfd00661b0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fdfe0009130 tx=0x7fdfe0017000 comp rx=0 tx=0).stop 2022-06-27T12:07:34.149 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:34.145+0000 7fdff56c6700 1 -- 172.21.15.195:0/1329772558 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdff00c44a0 msgr2=0x7fdff00c2f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:34.150 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:34.145+0000 7fdff56c6700 1 --2- 172.21.15.195:0/1329772558 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdff00c44a0 0x7fdff00c2f80 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fdfe400e900 tx=0x7fdfe4009060 comp rx=0 tx=0).stop 2022-06-27T12:07:34.150 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:34.145+0000 7fdff56c6700 1 -- 172.21.15.195:0/1329772558 shutdown_connections 2022-06-27T12:07:34.151 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:34.145+0000 7fdff56c6700 1 --2- 172.21.15.195:0/1329772558 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fdfd0063d00 0x7fdfd00661b0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:34.151 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:34.145+0000 7fdff56c6700 1 --2- 172.21.15.195:0/1329772558 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fdff00c6190 0x7fdff014edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:34.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:34.145+0000 7fdff56c6700 1 --2- 172.21.15.195:0/1329772558 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fdff00c40d0 0x7fdff00c06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:34.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:34.145+0000 7fdff56c6700 1 --2- 172.21.15.195:0/1329772558 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdff00c44a0 0x7fdff00c2f80 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:34.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:34.145+0000 7fdff56c6700 1 -- 172.21.15.195:0/1329772558 >> 172.21.15.195:0/1329772558 conn(0x7fdff00ccb80 msgr2=0x7fdff01502f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:07:34.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:34.145+0000 7fdff56c6700 1 -- 172.21.15.195:0/1329772558 shutdown_connections 2022-06-27T12:07:34.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:34.145+0000 7fdff56c6700 1 -- 172.21.15.195:0/1329772558 wait complete. 2022-06-27T12:07:34.154 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 72 2022-06-27T12:07:34.157 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2022-06-27T12:07:36.159 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:07:36.301 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.291+0000 7fa61c818700 1 Processor -- start 2022-06-27T12:07:36.301 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.291+0000 7fa61c818700 1 -- start start 2022-06-27T12:07:36.301 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.291+0000 7fa61c818700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fa618068f80 0x7fa6180c4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:36.302 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.292+0000 7fa61c818700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fa6180c7480 0x7fa6180683c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:36.302 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.292+0000 7fa61c818700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6181573b0 con 0x7fa618069350 2022-06-27T12:07:36.302 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.292+0000 7fa61c818700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6180c46c0 con 0x7fa6180c7480 2022-06-27T12:07:36.303 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.292+0000 7fa61c818700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6180697d0 con 0x7fa618068f80 2022-06-27T12:07:36.303 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.292+0000 7fa615d9b700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fa6180c7480 0x7fa6180683c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:36.303 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.292+0000 7fa615d9b700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fa6180c7480 0x7fa6180683c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48764/0 (socket says 172.21.15.195:48764) 2022-06-27T12:07:36.304 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.292+0000 7fa615d9b700 1 -- 172.21.15.195:0/4254247186 learned_addr learned my addr 172.21.15.195:0/4254247186 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:07:36.304 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.292+0000 7fa616d9d700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fa618069350 0x7fa618156460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:54956/0 (socket says 172.21.15.195:54956) 2022-06-27T12:07:36.305 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.292+0000 7fa61659c700 1 --2- 172.21.15.195:0/4254247186 >> v2:172.21.15.202:3300/0 conn(0x7fa618068f80 0x7fa6180c4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:36.305 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.292+0000 7fa615d9b700 1 -- 172.21.15.195:0/4254247186 >> v2:172.21.15.202:3300/0 conn(0x7fa618068f80 msgr2=0x7fa6180c4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:07:36.306 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.292+0000 7fa615d9b700 1 --2- 172.21.15.195:0/4254247186 >> v2:172.21.15.202:3300/0 conn(0x7fa618068f80 0x7fa6180c4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:36.306 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.292+0000 7fa615d9b700 1 -- 172.21.15.195:0/4254247186 >> v1:172.21.15.195:6789/0 conn(0x7fa618069350 legacy=0x7fa618156460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:36.306 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.292+0000 7fa61559a700 1 -- 172.21.15.195:0/4254247186 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2039092900 0 0) 0x7fa6181573b0 con 0x7fa618069350 2022-06-27T12:07:36.306 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.292+0000 7fa615d9b700 1 -- 172.21.15.195:0/4254247186 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa618156b70 con 0x7fa6180c7480 2022-06-27T12:07:36.307 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.292+0000 7fa615d9b700 1 --2- 172.21.15.195:0/4254247186 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa6180c7480 0x7fa6180683c0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fa618067b00 tx=0x7fa60c009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f22ac6efe42e1bea server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:07:36.307 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.292+0000 7fa61559a700 1 -- 172.21.15.195:0/4254247186 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa60c011630 con 0x7fa6180c7480 2022-06-27T12:07:36.307 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.292+0000 7fa61559a700 1 -- 172.21.15.195:0/4254247186 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa60c011790 con 0x7fa6180c7480 2022-06-27T12:07:36.307 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.292+0000 7fa61559a700 1 -- 172.21.15.195:0/4254247186 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa60c011b30 con 0x7fa6180c7480 2022-06-27T12:07:36.308 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.293+0000 7fa61c818700 1 -- 172.21.15.195:0/4254247186 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa6180c7480 msgr2=0x7fa6180683c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:36.308 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.293+0000 7fa61c818700 1 --2- 172.21.15.195:0/4254247186 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa6180c7480 0x7fa6180683c0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fa618067b00 tx=0x7fa60c009120 comp rx=0 tx=0).stop 2022-06-27T12:07:36.309 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.293+0000 7fa61c818700 1 -- 172.21.15.195:0/4254247186 shutdown_connections 2022-06-27T12:07:36.309 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.293+0000 7fa61c818700 1 --2- 172.21.15.195:0/4254247186 >> v2:172.21.15.202:3300/0 conn(0x7fa618068f80 0x7fa6180c4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:36.310 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.293+0000 7fa61c818700 1 --2- 172.21.15.195:0/4254247186 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa6180c7480 0x7fa6180683c0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:36.310 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.293+0000 7fa61c818700 1 -- 172.21.15.195:0/4254247186 >> 172.21.15.195:0/4254247186 conn(0x7fa6180ccbd0 msgr2=0x7fa6180ccfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:07:36.310 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.293+0000 7fa61c818700 1 -- 172.21.15.195:0/4254247186 shutdown_connections 2022-06-27T12:07:36.311 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.293+0000 7fa61c818700 1 -- 172.21.15.195:0/4254247186 wait complete. 2022-06-27T12:07:36.311 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.294+0000 7fa61c818700 1 Processor -- start 2022-06-27T12:07:36.311 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.295+0000 7fa61c818700 1 -- start start 2022-06-27T12:07:36.312 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.295+0000 7fa61c818700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa618068f80 0x7fa61816a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:36.312 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.295+0000 7fa61c818700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa618069350 0x7fa618153330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:36.312 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.295+0000 7fa61c818700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa6180c7480 0x7fa6180cb4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:36.312 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.295+0000 7fa61c818700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa61816a6c0 con 0x7fa6180c7480 2022-06-27T12:07:36.313 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.295+0000 7fa61c818700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa618166c00 con 0x7fa618069350 2022-06-27T12:07:36.313 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.295+0000 7fa61c818700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa618150f40 con 0x7fa618068f80 2022-06-27T12:07:36.313 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.296+0000 7fa616d9d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa6180c7480 0x7fa6180cb4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:36.314 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.296+0000 7fa616d9d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa6180c7480 0x7fa6180cb4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48774/0 (socket says 172.21.15.195:48774) 2022-06-27T12:07:36.314 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.296+0000 7fa616d9d700 1 -- 172.21.15.195:0/2927417297 learned_addr learned my addr 172.21.15.195:0/2927417297 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:07:36.315 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.296+0000 7fa615d9b700 1 --2- 172.21.15.195:0/2927417297 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa618069350 0x7fa618153330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:36.315 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.296+0000 7fa61659c700 1 --2- 172.21.15.195:0/2927417297 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa618068f80 0x7fa61816a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:36.315 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.296+0000 7fa616d9d700 1 -- 172.21.15.195:0/2927417297 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa618068f80 msgr2=0x7fa61816a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:36.316 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.296+0000 7fa616d9d700 1 --2- 172.21.15.195:0/2927417297 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa618068f80 0x7fa61816a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:36.317 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.296+0000 7fa616d9d700 1 -- 172.21.15.195:0/2927417297 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa618069350 msgr2=0x7fa618153330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:36.318 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.296+0000 7fa616d9d700 1 --2- 172.21.15.195:0/2927417297 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa618069350 0x7fa618153330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:36.318 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.296+0000 7fa616d9d700 1 -- 172.21.15.195:0/2927417297 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa60c0093a0 con 0x7fa6180c7480 2022-06-27T12:07:36.319 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.296+0000 7fa616d9d700 1 --2- 172.21.15.195:0/2927417297 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa6180c7480 0x7fa6180cb4c0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fa61000c860 tx=0x7fa610001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:07:36.319 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.296+0000 7fa6067fc700 1 -- 172.21.15.195:0/2927417297 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa610004c80 con 0x7fa6180c7480 2022-06-27T12:07:36.320 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.296+0000 7fa6067fc700 1 -- 172.21.15.195:0/2927417297 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa610004de0 con 0x7fa6180c7480 2022-06-27T12:07:36.320 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.296+0000 7fa61c818700 1 -- 172.21.15.195:0/2927417297 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa618151220 con 0x7fa6180c7480 2022-06-27T12:07:36.321 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.296+0000 7fa6067fc700 1 -- 172.21.15.195:0/2927417297 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa610007380 con 0x7fa6180c7480 2022-06-27T12:07:36.321 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.296+0000 7fa61c818700 1 -- 172.21.15.195:0/2927417297 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa618166890 con 0x7fa6180c7480 2022-06-27T12:07:36.321 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.297+0000 7fa61c818700 1 -- 172.21.15.195:0/2927417297 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6180722c0 con 0x7fa6180c7480 2022-06-27T12:07:36.321 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.298+0000 7fa6067fc700 1 -- 172.21.15.195:0/2927417297 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fa61001a030 con 0x7fa6180c7480 2022-06-27T12:07:36.322 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.298+0000 7fa6067fc700 1 --2- 172.21.15.195:0/2927417297 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa5f4063d00 0x7fa5f40661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:36.322 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.298+0000 7fa6067fc700 1 -- 172.21.15.195:0/2927417297 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7062+0+0 (secure 0 0 0) 0x7fa6100801e0 con 0x7fa6180c7480 2022-06-27T12:07:36.323 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.298+0000 7fa61659c700 1 --2- 172.21.15.195:0/2927417297 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa5f4063d00 0x7fa5f40661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:36.324 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.298+0000 7fa61659c700 1 --2- 172.21.15.195:0/2927417297 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa5f4063d00 0x7fa5f40661b0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fa608009bd0 tx=0x7fa608009060 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:07:36.325 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.302+0000 7fa6067fc700 1 -- 172.21.15.195:0/2927417297 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fa61004ea60 con 0x7fa6180c7480 2022-06-27T12:07:36.529 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.529+0000 7fa61c818700 1 -- 172.21.15.195:0/2927417297 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa61806ac60 con 0x7fa6180c7480 2022-06-27T12:07:36.530 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.530+0000 7fa6067fc700 1 -- 172.21.15.195:0/2927417297 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+6128 (secure 0 0 0) 0x7fa610018090 con 0x7fa6180c7480 2022-06-27T12:07:36.531 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:07:36.531 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":6271,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/1603307620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":1603307620},{"type":"v1","addr":"172.21.15.202:6835","nonce":1603307620}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:07:08.223429+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5591},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5591":{"gid":5591,"name":"b","rank":0,"incarnation":67,"state":"up:active","state_seq":23,"addr":"172.21.15.202:6833/3022591968","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3022591968},{"type":"v1","addr":"172.21.15.202:6833","nonce":3022591968}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:07:36.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.532+0000 7fa61c818700 1 -- 172.21.15.195:0/2927417297 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa5f4063d00 msgr2=0x7fa5f40661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:36.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.532+0000 7fa61c818700 1 --2- 172.21.15.195:0/2927417297 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa5f4063d00 0x7fa5f40661b0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fa608009bd0 tx=0x7fa608009060 comp rx=0 tx=0).stop 2022-06-27T12:07:36.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.532+0000 7fa61c818700 1 -- 172.21.15.195:0/2927417297 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa6180c7480 msgr2=0x7fa6180cb4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:36.534 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.532+0000 7fa61c818700 1 --2- 172.21.15.195:0/2927417297 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa6180c7480 0x7fa6180cb4c0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fa61000c860 tx=0x7fa610001630 comp rx=0 tx=0).stop 2022-06-27T12:07:36.534 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.532+0000 7fa61c818700 1 -- 172.21.15.195:0/2927417297 shutdown_connections 2022-06-27T12:07:36.534 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.532+0000 7fa61c818700 1 --2- 172.21.15.195:0/2927417297 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa5f4063d00 0x7fa5f40661b0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:36.535 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.532+0000 7fa61c818700 1 --2- 172.21.15.195:0/2927417297 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa6180c7480 0x7fa6180cb4c0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:36.535 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.532+0000 7fa61c818700 1 --2- 172.21.15.195:0/2927417297 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa618068f80 0x7fa61816a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:36.535 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.532+0000 7fa61c818700 1 --2- 172.21.15.195:0/2927417297 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa618069350 0x7fa618153330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:36.535 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.532+0000 7fa61c818700 1 -- 172.21.15.195:0/2927417297 >> 172.21.15.195:0/2927417297 conn(0x7fa6180ccbd0 msgr2=0x7fa618071330 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:07:36.535 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.532+0000 7fa61c818700 1 -- 172.21.15.195:0/2927417297 shutdown_connections 2022-06-27T12:07:36.536 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:36.532+0000 7fa61c818700 1 -- 172.21.15.195:0/2927417297 wait complete. 2022-06-27T12:07:36.536 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 72 2022-06-27T12:07:36.544 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2022-06-27T12:07:38.546 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:07:38.681 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.681+0000 7f34552d3700 1 Processor -- start 2022-06-27T12:07:38.681 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.681+0000 7f34552d3700 1 -- start start 2022-06-27T12:07:38.688 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.681+0000 7f34552d3700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f34500c5480 0x7f3450069a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:38.689 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.681+0000 7f34552d3700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f34500c64f0 0x7f34501504c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:38.689 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.681+0000 7f34552d3700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34500c5d80 con 0x7f345006d280 2022-06-27T12:07:38.690 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.681+0000 7f34552d3700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3450069f50 con 0x7f34500c64f0 2022-06-27T12:07:38.690 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.681+0000 7f34552d3700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f345006b010 con 0x7f34500c5480 2022-06-27T12:07:38.690 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.682+0000 7f344e7fc700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f34500c64f0 0x7f34501504c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:38.690 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.682+0000 7f344e7fc700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f34500c64f0 0x7f34501504c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48788/0 (socket says 172.21.15.195:48788) 2022-06-27T12:07:38.691 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.682+0000 7f344e7fc700 1 -- 172.21.15.195:0/3147200165 learned_addr learned my addr 172.21.15.195:0/3147200165 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:07:38.691 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.682+0000 7f344effd700 1 --2- 172.21.15.195:0/3147200165 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f34500c5480 0x7f3450069a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:38.691 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.682+0000 7f344e7fc700 1 -- 172.21.15.195:0/3147200165 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f34500c5480 msgr2=0x7f3450069a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:07:38.692 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.682+0000 7f344e7fc700 1 --2- 172.21.15.195:0/3147200165 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f34500c5480 0x7f3450069a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:38.692 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.682+0000 7f344e7fc700 1 -- 172.21.15.195:0/3147200165 >> v1:172.21.15.202:6789/0 conn(0x7f345006d280 legacy=0x7f3450150a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:38.692 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.682+0000 7f344e7fc700 1 -- 172.21.15.195:0/3147200165 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34500c75e0 con 0x7f34500c64f0 2022-06-27T12:07:38.692 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.682+0000 7f344e7fc700 1 --2- 172.21.15.195:0/3147200165 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f34500c64f0 0x7f34501504c0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f3440009860 tx=0x7f3440009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9a2117217c92c8e server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:07:38.693 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.682+0000 7f344dffb700 1 -- 172.21.15.195:0/3147200165 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f344000b680 con 0x7f34500c64f0 2022-06-27T12:07:38.693 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.682+0000 7f344dffb700 1 -- 172.21.15.195:0/3147200165 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f344000b7e0 con 0x7f34500c64f0 2022-06-27T12:07:38.693 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.682+0000 7f344dffb700 1 -- 172.21.15.195:0/3147200165 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f344000bba0 con 0x7f34500c64f0 2022-06-27T12:07:38.694 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.683+0000 7f34552d3700 1 -- 172.21.15.195:0/3147200165 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f34500c64f0 msgr2=0x7f34501504c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:38.694 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.683+0000 7f34552d3700 1 --2- 172.21.15.195:0/3147200165 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f34500c64f0 0x7f34501504c0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f3440009860 tx=0x7f3440009120 comp rx=0 tx=0).stop 2022-06-27T12:07:38.694 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.683+0000 7f34552d3700 1 -- 172.21.15.195:0/3147200165 shutdown_connections 2022-06-27T12:07:38.694 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.683+0000 7f34552d3700 1 --2- 172.21.15.195:0/3147200165 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f34500c5480 0x7f3450069a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:38.695 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.683+0000 7f34552d3700 1 --2- 172.21.15.195:0/3147200165 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f34500c64f0 0x7f34501504c0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:38.695 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.683+0000 7f34552d3700 1 -- 172.21.15.195:0/3147200165 >> 172.21.15.195:0/3147200165 conn(0x7f3450063af0 msgr2=0x7f3450063ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:07:38.695 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.683+0000 7f34552d3700 1 -- 172.21.15.195:0/3147200165 shutdown_connections 2022-06-27T12:07:38.696 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.683+0000 7f34552d3700 1 -- 172.21.15.195:0/3147200165 wait complete. 2022-06-27T12:07:38.696 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.684+0000 7f34552d3700 1 Processor -- start 2022-06-27T12:07:38.696 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.684+0000 7f34552d3700 1 -- start start 2022-06-27T12:07:38.696 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.685+0000 7f34552d3700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f345006d280 0x7f34500bdab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:38.697 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.685+0000 7f34552d3700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f34500c5480 0x7f34500bc9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:38.697 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.685+0000 7f34552d3700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f34500c64f0 0x7f34500bcee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:38.697 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.685+0000 7f34552d3700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3450167280 con 0x7f34500c5480 2022-06-27T12:07:38.697 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.685+0000 7f34552d3700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34500bc510 con 0x7f345006d280 2022-06-27T12:07:38.698 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.685+0000 7f34552d3700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3450166d50 con 0x7f34500c64f0 2022-06-27T12:07:38.698 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.685+0000 7f344e7fc700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f34500c5480 0x7f34500bc9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:38.698 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.685+0000 7f344e7fc700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f34500c5480 0x7f34500bc9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48794/0 (socket says 172.21.15.195:48794) 2022-06-27T12:07:38.699 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.685+0000 7f344e7fc700 1 -- 172.21.15.195:0/329541679 learned_addr learned my addr 172.21.15.195:0/329541679 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:07:38.699 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.685+0000 7f344effd700 1 --2- 172.21.15.195:0/329541679 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f345006d280 0x7f34500bdab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:38.699 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.685+0000 7f344f7fe700 1 --2- 172.21.15.195:0/329541679 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f34500c64f0 0x7f34500bcee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:38.699 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.685+0000 7f344e7fc700 1 -- 172.21.15.195:0/329541679 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f34500c64f0 msgr2=0x7f34500bcee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:38.700 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.685+0000 7f344e7fc700 1 --2- 172.21.15.195:0/329541679 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f34500c64f0 0x7f34500bcee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:38.700 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.685+0000 7f344e7fc700 1 -- 172.21.15.195:0/329541679 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f345006d280 msgr2=0x7f34500bdab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:38.700 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.685+0000 7f344e7fc700 1 --2- 172.21.15.195:0/329541679 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f345006d280 0x7f34500bdab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:38.701 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.685+0000 7f344e7fc700 1 -- 172.21.15.195:0/329541679 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34400093a0 con 0x7f34500c5480 2022-06-27T12:07:38.701 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.685+0000 7f344e7fc700 1 --2- 172.21.15.195:0/329541679 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f34500c5480 0x7f34500bc9a0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f3440009160 tx=0x7f3440017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:07:38.701 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.686+0000 7f3436ffd700 1 -- 172.21.15.195:0/329541679 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f344000b950 con 0x7f34500c5480 2022-06-27T12:07:38.702 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.686+0000 7f3436ffd700 1 -- 172.21.15.195:0/329541679 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f344000bab0 con 0x7f34500c5480 2022-06-27T12:07:38.702 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.686+0000 7f34552d3700 1 -- 172.21.15.195:0/329541679 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3450154780 con 0x7f34500c5480 2022-06-27T12:07:38.702 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.686+0000 7f3436ffd700 1 -- 172.21.15.195:0/329541679 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f344001a680 con 0x7f34500c5480 2022-06-27T12:07:38.702 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.686+0000 7f34552d3700 1 -- 172.21.15.195:0/329541679 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3450154a20 con 0x7f34500c5480 2022-06-27T12:07:38.703 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.686+0000 7f34552d3700 1 -- 172.21.15.195:0/329541679 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34501545e0 con 0x7f34500c5480 2022-06-27T12:07:38.703 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.687+0000 7f3436ffd700 1 -- 172.21.15.195:0/329541679 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f344000f050 con 0x7f34500c5480 2022-06-27T12:07:38.703 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.687+0000 7f3436ffd700 1 --2- 172.21.15.195:0/329541679 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3430063d30 0x7f34300661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:38.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.687+0000 7f3436ffd700 1 -- 172.21.15.195:0/329541679 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7062+0+0 (secure 0 0 0) 0x7f3440082550 con 0x7f34500c5480 2022-06-27T12:07:38.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.687+0000 7f344effd700 1 --2- 172.21.15.195:0/329541679 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3430063d30 0x7f34300661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:38.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.688+0000 7f344effd700 1 --2- 172.21.15.195:0/329541679 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3430063d30 0x7f34300661e0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f3444004560 tx=0x7f3444008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:07:38.705 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.692+0000 7f3436ffd700 1 -- 172.21.15.195:0/329541679 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f34501545e0 con 0x7f34500c5480 2022-06-27T12:07:38.923 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.923+0000 7f34552d3700 1 -- 172.21.15.195:0/329541679 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f34501545e0 con 0x7f34500c5480 2022-06-27T12:07:38.923 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.923+0000 7f3436ffd700 1 -- 172.21.15.195:0/329541679 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+6128 (secure 0 0 0) 0x7f34501545e0 con 0x7f34500c5480 2022-06-27T12:07:38.924 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:07:38.924 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":6271,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/1603307620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":1603307620},{"type":"v1","addr":"172.21.15.202:6835","nonce":1603307620}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:07:08.223429+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5591},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5591":{"gid":5591,"name":"b","rank":0,"incarnation":67,"state":"up:active","state_seq":23,"addr":"172.21.15.202:6833/3022591968","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3022591968},{"type":"v1","addr":"172.21.15.202:6833","nonce":3022591968}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:07:38.926 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.925+0000 7f34552d3700 1 -- 172.21.15.195:0/329541679 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3430063d30 msgr2=0x7f34300661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:38.926 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.925+0000 7f34552d3700 1 --2- 172.21.15.195:0/329541679 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3430063d30 0x7f34300661e0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f3444004560 tx=0x7f3444008040 comp rx=0 tx=0).stop 2022-06-27T12:07:38.927 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.925+0000 7f34552d3700 1 -- 172.21.15.195:0/329541679 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f34500c5480 msgr2=0x7f34500bc9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:38.927 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.925+0000 7f34552d3700 1 --2- 172.21.15.195:0/329541679 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f34500c5480 0x7f34500bc9a0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f3440009160 tx=0x7f3440017000 comp rx=0 tx=0).stop 2022-06-27T12:07:38.928 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.925+0000 7f34552d3700 1 -- 172.21.15.195:0/329541679 shutdown_connections 2022-06-27T12:07:38.928 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.925+0000 7f34552d3700 1 --2- 172.21.15.195:0/329541679 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3430063d30 0x7f34300661e0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:38.929 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.925+0000 7f34552d3700 1 --2- 172.21.15.195:0/329541679 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f34500c64f0 0x7f34500bcee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:38.929 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.925+0000 7f34552d3700 1 --2- 172.21.15.195:0/329541679 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f345006d280 0x7f34500bdab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:38.930 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.925+0000 7f34552d3700 1 --2- 172.21.15.195:0/329541679 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f34500c5480 0x7f34500bc9a0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:38.930 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.926+0000 7f34552d3700 1 -- 172.21.15.195:0/329541679 >> 172.21.15.195:0/329541679 conn(0x7f3450063af0 msgr2=0x7f34501538b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:07:38.931 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.926+0000 7f34552d3700 1 -- 172.21.15.195:0/329541679 shutdown_connections 2022-06-27T12:07:38.931 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:38.926+0000 7f34552d3700 1 -- 172.21.15.195:0/329541679 wait complete. 2022-06-27T12:07:38.932 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 72 2022-06-27T12:07:38.938 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2022-06-27T12:07:40.940 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:07:41.084 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.073+0000 7f08f1159700 1 Processor -- start 2022-06-27T12:07:41.084 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.073+0000 7f08f1159700 1 -- start start 2022-06-27T12:07:41.085 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.074+0000 7f08f1159700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f08ec06fef0 0x7f08ec1570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:41.085 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.074+0000 7f08f1159700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f08ec0c34a0 0x7f08ec0c2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:41.086 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.074+0000 7f08f1159700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08ec06fba0 con 0x7f08ec06e850 2022-06-27T12:07:41.086 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.074+0000 7f08f1159700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08ec1575e0 con 0x7f08ec06fef0 2022-06-27T12:07:41.087 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.074+0000 7f08f1159700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08ec06f5e0 con 0x7f08ec0c34a0 2022-06-27T12:07:41.087 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.074+0000 7f08eb59e700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f08ec0c34a0 0x7f08ec0c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:41.088 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.074+0000 7f08ea59c700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f08ec06fef0 0x7f08ec1570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:41.089 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.074+0000 7f08ead9d700 1 --1- >> v1:172.21.15.202:6789/0 conn(0x7f08ec06e850 0x7f08ec069de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.202:6789/0 says I am v1:172.21.15.195:56852/0 (socket says 172.21.15.195:56852) 2022-06-27T12:07:41.089 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.074+0000 7f08ea59c700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f08ec06fef0 0x7f08ec1570a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3300/0 says I am v2:172.21.15.195:46932/0 (socket says 172.21.15.195:46932) 2022-06-27T12:07:41.090 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.074+0000 7f08ead9d700 1 -- 172.21.15.195:0/2025722132 learned_addr learned my addr 172.21.15.195:0/2025722132 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:07:41.090 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.075+0000 7f08eb59e700 1 -- 172.21.15.195:0/2025722132 >> v2:172.21.15.202:3300/0 conn(0x7f08ec06fef0 msgr2=0x7f08ec1570a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:07:41.090 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.075+0000 7f08eb59e700 1 --2- 172.21.15.195:0/2025722132 >> v2:172.21.15.202:3300/0 conn(0x7f08ec06fef0 0x7f08ec1570a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:41.091 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.075+0000 7f08eb59e700 1 -- 172.21.15.195:0/2025722132 >> v1:172.21.15.202:6789/0 conn(0x7f08ec06e850 legacy=0x7f08ec069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:41.091 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.075+0000 7f08e9d9b700 1 -- 172.21.15.195:0/2025722132 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 142848955 0 0) 0x7f08ec06fba0 con 0x7f08ec06e850 2022-06-27T12:07:41.092 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.075+0000 7f08eb59e700 1 -- 172.21.15.195:0/2025722132 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08ec06f8c0 con 0x7f08ec0c34a0 2022-06-27T12:07:41.092 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.075+0000 7f08ea59c700 1 --2- 172.21.15.195:0/2025722132 >> v2:172.21.15.202:3300/0 conn(0x7f08ec06fef0 0x7f08ec1570a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:07:41.093 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.075+0000 7f08eb59e700 1 --2- 172.21.15.195:0/2025722132 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f08ec0c34a0 0x7f08ec0c2920 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f08e4009e60 tx=0x7f08e4009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=45dd4fec71e06e20 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:07:41.093 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.075+0000 7f08e9d9b700 1 -- 172.21.15.195:0/2025722132 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08e400e070 con 0x7f08ec0c34a0 2022-06-27T12:07:41.094 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.075+0000 7f08e9d9b700 1 -- 172.21.15.195:0/2025722132 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08e400bbf0 con 0x7f08ec0c34a0 2022-06-27T12:07:41.095 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.075+0000 7f08e9d9b700 1 -- 172.21.15.195:0/2025722132 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08e4012430 con 0x7f08ec0c34a0 2022-06-27T12:07:41.096 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.076+0000 7f08f1159700 1 -- 172.21.15.195:0/2025722132 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f08ec0c34a0 msgr2=0x7f08ec0c2920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:41.096 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.076+0000 7f08f1159700 1 --2- 172.21.15.195:0/2025722132 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f08ec0c34a0 0x7f08ec0c2920 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f08e4009e60 tx=0x7f08e4009780 comp rx=0 tx=0).stop 2022-06-27T12:07:41.097 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.076+0000 7f08f1159700 1 -- 172.21.15.195:0/2025722132 shutdown_connections 2022-06-27T12:07:41.097 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.076+0000 7f08f1159700 1 --2- 172.21.15.195:0/2025722132 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f08ec0c34a0 0x7f08ec0c2920 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:41.097 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.076+0000 7f08f1159700 1 --2- 172.21.15.195:0/2025722132 >> v2:172.21.15.202:3300/0 conn(0x7f08ec06fef0 0x7f08ec1570a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:41.098 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.076+0000 7f08f1159700 1 -- 172.21.15.195:0/2025722132 >> 172.21.15.195:0/2025722132 conn(0x7f08ec0c6350 msgr2=0x7f08ec0c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:07:41.099 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.076+0000 7f08f1159700 1 -- 172.21.15.195:0/2025722132 shutdown_connections 2022-06-27T12:07:41.099 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.076+0000 7f08f1159700 1 -- 172.21.15.195:0/2025722132 wait complete. 2022-06-27T12:07:41.100 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.077+0000 7f08f1159700 1 Processor -- start 2022-06-27T12:07:41.100 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.077+0000 7f08f1159700 1 -- start start 2022-06-27T12:07:41.100 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.078+0000 7f08f1159700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f08ec06e850 0x7f08ec06a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:41.101 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.078+0000 7f08f1159700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f08ec0c34a0 0x7f08ec14e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:41.102 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.078+0000 7f08f1159700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f08ec0c7800 0x7f08ec0b6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:41.102 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.078+0000 7f08f1159700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08ec1640f0 con 0x7f08ec0c7800 2022-06-27T12:07:41.104 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.078+0000 7f08f1159700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08ec1589c0 con 0x7f08ec06e850 2022-06-27T12:07:41.106 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.078+0000 7f08f1159700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08ec168170 con 0x7f08ec0c34a0 2022-06-27T12:07:41.106 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.078+0000 7f08eb59e700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f08ec0c7800 0x7f08ec0b6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:41.107 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.078+0000 7f08ead9d700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f08ec06e850 0x7f08ec06a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:41.107 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.078+0000 7f08eb59e700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f08ec0c7800 0x7f08ec0b6bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48814/0 (socket says 172.21.15.195:48814) 2022-06-27T12:07:41.108 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.078+0000 7f08ea59c700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f08ec0c34a0 0x7f08ec14e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:41.108 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.078+0000 7f08eb59e700 1 -- 172.21.15.195:0/4210471693 learned_addr learned my addr 172.21.15.195:0/4210471693 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:07:41.108 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.078+0000 7f08eb59e700 1 -- 172.21.15.195:0/4210471693 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f08ec0c34a0 msgr2=0x7f08ec14e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:41.109 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.078+0000 7f08eb59e700 1 --2- 172.21.15.195:0/4210471693 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f08ec0c34a0 0x7f08ec14e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:41.110 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.078+0000 7f08eb59e700 1 -- 172.21.15.195:0/4210471693 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f08ec06e850 msgr2=0x7f08ec06a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:41.110 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.078+0000 7f08eb59e700 1 --2- 172.21.15.195:0/4210471693 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f08ec06e850 0x7f08ec06a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:41.111 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.078+0000 7f08eb59e700 1 -- 172.21.15.195:0/4210471693 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08e00092e0 con 0x7f08ec0c7800 2022-06-27T12:07:41.111 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.078+0000 7f08eb59e700 1 --2- 172.21.15.195:0/4210471693 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f08ec0c7800 0x7f08ec0b6bc0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f08e400b6c0 tx=0x7f08e4019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:07:41.111 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.079+0000 7f08d2ffd700 1 -- 172.21.15.195:0/4210471693 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08e4009420 con 0x7f08ec0c7800 2022-06-27T12:07:41.112 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.079+0000 7f08d2ffd700 1 -- 172.21.15.195:0/4210471693 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08e40127a0 con 0x7f08ec0c7800 2022-06-27T12:07:41.113 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.079+0000 7f08f1159700 1 -- 172.21.15.195:0/4210471693 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08e4009a20 con 0x7f08ec0c7800 2022-06-27T12:07:41.114 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.079+0000 7f08d2ffd700 1 -- 172.21.15.195:0/4210471693 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08e4012b60 con 0x7f08ec0c7800 2022-06-27T12:07:41.114 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.079+0000 7f08f1159700 1 -- 172.21.15.195:0/4210471693 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08ec1643b0 con 0x7f08ec0c7800 2022-06-27T12:07:41.115 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.080+0000 7f08f1159700 1 -- 172.21.15.195:0/4210471693 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08ec0722c0 con 0x7f08ec0c7800 2022-06-27T12:07:41.115 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.080+0000 7f08d2ffd700 1 -- 172.21.15.195:0/4210471693 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f08e4004250 con 0x7f08ec0c7800 2022-06-27T12:07:41.115 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.080+0000 7f08d2ffd700 1 --2- 172.21.15.195:0/4210471693 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f08cc063d00 0x7f08cc0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:41.116 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.080+0000 7f08d2ffd700 1 -- 172.21.15.195:0/4210471693 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7062+0+0 (secure 0 0 0) 0x7f08e4083680 con 0x7f08ec0c7800 2022-06-27T12:07:41.117 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.080+0000 7f08ead9d700 1 --2- 172.21.15.195:0/4210471693 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f08cc063d00 0x7f08cc0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:41.117 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.081+0000 7f08ead9d700 1 --2- 172.21.15.195:0/4210471693 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f08cc063d00 0x7f08cc0661b0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f08dc007fd0 tx=0x7f08dc007d10 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:07:41.118 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.085+0000 7f08d2ffd700 1 -- 172.21.15.195:0/4210471693 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f08e4051e80 con 0x7f08ec0c7800 2022-06-27T12:07:41.313 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.313+0000 7f08f1159700 1 -- 172.21.15.195:0/4210471693 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f08ec0c4ec0 con 0x7f08ec0c7800 2022-06-27T12:07:41.314 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.313+0000 7f08d2ffd700 1 -- 172.21.15.195:0/4210471693 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+6128 (secure 0 0 0) 0x7f08e40554a0 con 0x7f08ec0c7800 2022-06-27T12:07:41.315 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:07:41.315 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":6271,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/1603307620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":1603307620},{"type":"v1","addr":"172.21.15.202:6835","nonce":1603307620}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:07:08.223429+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5591},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5591":{"gid":5591,"name":"b","rank":0,"incarnation":67,"state":"up:active","state_seq":23,"addr":"172.21.15.202:6833/3022591968","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3022591968},{"type":"v1","addr":"172.21.15.202:6833","nonce":3022591968}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:07:41.323 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.316+0000 7f08f1159700 1 -- 172.21.15.195:0/4210471693 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f08cc063d00 msgr2=0x7f08cc0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:41.324 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.316+0000 7f08f1159700 1 --2- 172.21.15.195:0/4210471693 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f08cc063d00 0x7f08cc0661b0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f08dc007fd0 tx=0x7f08dc007d10 comp rx=0 tx=0).stop 2022-06-27T12:07:41.324 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.316+0000 7f08f1159700 1 -- 172.21.15.195:0/4210471693 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f08ec0c7800 msgr2=0x7f08ec0b6bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:41.325 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.316+0000 7f08f1159700 1 --2- 172.21.15.195:0/4210471693 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f08ec0c7800 0x7f08ec0b6bc0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f08e400b6c0 tx=0x7f08e4019000 comp rx=0 tx=0).stop 2022-06-27T12:07:41.326 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.316+0000 7f08f1159700 1 -- 172.21.15.195:0/4210471693 shutdown_connections 2022-06-27T12:07:41.326 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.316+0000 7f08f1159700 1 --2- 172.21.15.195:0/4210471693 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f08cc063d00 0x7f08cc0661b0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:41.327 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.316+0000 7f08f1159700 1 --2- 172.21.15.195:0/4210471693 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f08ec0c7800 0x7f08ec0b6bc0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:41.327 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.316+0000 7f08f1159700 1 --2- 172.21.15.195:0/4210471693 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f08ec06e850 0x7f08ec06a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:41.328 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.316+0000 7f08f1159700 1 --2- 172.21.15.195:0/4210471693 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f08ec0c34a0 0x7f08ec14e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:41.328 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.316+0000 7f08f1159700 1 -- 172.21.15.195:0/4210471693 >> 172.21.15.195:0/4210471693 conn(0x7f08ec0c6350 msgr2=0x7f08ec0c6720 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:07:41.329 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.316+0000 7f08f1159700 1 -- 172.21.15.195:0/4210471693 shutdown_connections 2022-06-27T12:07:41.329 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:41.316+0000 7f08f1159700 1 -- 172.21.15.195:0/4210471693 wait complete. 2022-06-27T12:07:41.330 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 72 2022-06-27T12:07:41.332 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2022-06-27T12:07:43.333 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:07:43.467 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.467+0000 7f9b690e8700 1 Processor -- start 2022-06-27T12:07:43.467 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.467+0000 7f9b690e8700 1 -- start start 2022-06-27T12:07:43.479 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.468+0000 7f9b690e8700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9b640c4160 0x7f9b64156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:43.480 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.468+0000 7f9b690e8700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b640c1a20 con 0x7f9b640c39e0 2022-06-27T12:07:43.480 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.468+0000 7f9b690e8700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b640c1130 con 0x7f9b6406f640 2022-06-27T12:07:43.480 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.468+0000 7f9b690e8700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b6406ea90 con 0x7f9b640c4160 2022-06-27T12:07:43.481 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.468+0000 7f9b6359e700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f9b640c39e0 0x7f9b641535a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:55016/0 (socket says 172.21.15.195:55016) 2022-06-27T12:07:43.481 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.468+0000 7f9b6359e700 1 -- 172.21.15.195:0/2328841752 learned_addr learned my addr 172.21.15.195:0/2328841752 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:07:43.482 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.468+0000 7f9b6259c700 1 --2- 172.21.15.195:0/2328841752 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9b640c4160 0x7f9b64156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:43.482 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.468+0000 7f9b61d9b700 1 -- 172.21.15.195:0/2328841752 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2914625773 0 0) 0x7f9b640c1a20 con 0x7f9b640c39e0 2022-06-27T12:07:43.483 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.468+0000 7f9b61d9b700 1 -- 172.21.15.195:0/2328841752 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b44003430 con 0x7f9b640c39e0 2022-06-27T12:07:43.483 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.468+0000 7f9b61d9b700 1 -- 172.21.15.195:0/2328841752 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 825363579 0 0) 0x7f9b44003430 con 0x7f9b640c39e0 2022-06-27T12:07:43.484 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.468+0000 7f9b61d9b700 1 -- 172.21.15.195:0/2328841752 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9b640c4160 msgr2=0x7f9b64156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:07:43.484 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.468+0000 7f9b61d9b700 1 --2- 172.21.15.195:0/2328841752 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9b640c4160 0x7f9b64156ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:43.485 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.468+0000 7f9b61d9b700 1 -- 172.21.15.195:0/2328841752 >> v1:172.21.15.202:6789/0 conn(0x7f9b6406f640 legacy=0x7f9b640c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:43.485 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.468+0000 7f9b61d9b700 1 -- 172.21.15.195:0/2328841752 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b6406ed70 con 0x7f9b640c39e0 2022-06-27T12:07:43.486 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.469+0000 7f9b61d9b700 1 -- 172.21.15.195:0/2328841752 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7f9b5c0017f0 con 0x7f9b640c39e0 2022-06-27T12:07:43.488 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.469+0000 7f9b61d9b700 1 -- 172.21.15.195:0/2328841752 >> v1:172.21.15.195:6789/0 conn(0x7f9b640c39e0 legacy=0x7f9b641535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:43.488 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.469+0000 7f9b61d9b700 1 --2- 172.21.15.195:0/2328841752 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9b640c4160 0x7f9b44003950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:43.489 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.469+0000 7f9b61d9b700 1 --2- 172.21.15.195:0/2328841752 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9b44003ed0 0x7f9b440046d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:43.489 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.469+0000 7f9b61d9b700 1 --2- 172.21.15.195:0/2328841752 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9b44004c10 0x7f9b44005080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:43.490 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.469+0000 7f9b61d9b700 1 -- 172.21.15.195:0/2328841752 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b6406ea90 con 0x7f9b44003ed0 2022-06-27T12:07:43.490 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.469+0000 7f9b61d9b700 1 -- 172.21.15.195:0/2328841752 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b4400bab0 con 0x7f9b44004c10 2022-06-27T12:07:43.490 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.469+0000 7f9b61d9b700 1 -- 172.21.15.195:0/2328841752 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b4400bc70 con 0x7f9b640c4160 2022-06-27T12:07:43.491 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.469+0000 7f9b6259c700 1 --2- 172.21.15.195:0/2328841752 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9b44003ed0 0x7f9b440046d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:43.491 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.469+0000 7f9b62d9d700 1 --2- 172.21.15.195:0/2328841752 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9b640c4160 0x7f9b44003950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:43.491 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.469+0000 7f9b6359e700 1 --2- 172.21.15.195:0/2328841752 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9b44004c10 0x7f9b44005080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:43.492 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.469+0000 7f9b6259c700 1 -- 172.21.15.195:0/2328841752 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9b640c4160 msgr2=0x7f9b44003950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:07:43.492 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.469+0000 7f9b6259c700 1 --2- 172.21.15.195:0/2328841752 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9b640c4160 0x7f9b44003950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:43.492 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.469+0000 7f9b6259c700 1 -- 172.21.15.195:0/2328841752 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9b44004c10 msgr2=0x7f9b44005080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:07:43.493 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.469+0000 7f9b6259c700 1 --2- 172.21.15.195:0/2328841752 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9b44004c10 0x7f9b44005080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:43.493 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.469+0000 7f9b6259c700 1 -- 172.21.15.195:0/2328841752 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b44003430 con 0x7f9b44003ed0 2022-06-27T12:07:43.494 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.469+0000 7f9b6259c700 1 --2- 172.21.15.195:0/2328841752 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9b44003ed0 0x7f9b440046d0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f9b54006dc0 tx=0x7f9b54009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15d6d63bbfbaf966 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:07:43.494 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.469+0000 7f9b61d9b700 1 -- 172.21.15.195:0/2328841752 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b54009470 con 0x7f9b44003ed0 2022-06-27T12:07:43.495 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.469+0000 7f9b61d9b700 1 -- 172.21.15.195:0/2328841752 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b5400e640 con 0x7f9b44003ed0 2022-06-27T12:07:43.495 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.469+0000 7f9b61d9b700 1 -- 172.21.15.195:0/2328841752 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b5400e9e0 con 0x7f9b44003ed0 2022-06-27T12:07:43.495 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.470+0000 7f9b690e8700 1 -- 172.21.15.195:0/2328841752 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9b44003ed0 msgr2=0x7f9b440046d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:43.496 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.470+0000 7f9b690e8700 1 --2- 172.21.15.195:0/2328841752 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9b44003ed0 0x7f9b440046d0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f9b54006dc0 tx=0x7f9b54009060 comp rx=0 tx=0).stop 2022-06-27T12:07:43.496 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.470+0000 7f9b6359e700 1 -- 172.21.15.195:0/2328841752 reap_dead start 2022-06-27T12:07:43.497 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.470+0000 7f9b690e8700 1 -- 172.21.15.195:0/2328841752 shutdown_connections 2022-06-27T12:07:43.497 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.470+0000 7f9b690e8700 1 -- 172.21.15.195:0/2328841752 >> 172.21.15.195:0/2328841752 conn(0x7f9b640c6350 msgr2=0x7f9b640c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:07:43.497 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.470+0000 7f9b690e8700 1 -- 172.21.15.195:0/2328841752 shutdown_connections 2022-06-27T12:07:43.497 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.470+0000 7f9b690e8700 1 -- 172.21.15.195:0/2328841752 wait complete. 2022-06-27T12:07:43.497 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.471+0000 7f9b690e8700 1 Processor -- start 2022-06-27T12:07:43.498 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.472+0000 7f9b690e8700 1 -- start start 2022-06-27T12:07:43.499 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.472+0000 7f9b690e8700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9b44003ed0 0x7f9b6414f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:43.499 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.472+0000 7f9b690e8700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9b6416a1b0 0x7f9b64068330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:43.499 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.472+0000 7f9b690e8700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9b6406aeb0 0x7f9b64068870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:43.499 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.472+0000 7f9b690e8700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b64167040 con 0x7f9b6416a1b0 2022-06-27T12:07:43.500 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.472+0000 7f9b690e8700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b6416a6c0 con 0x7f9b44003ed0 2022-06-27T12:07:43.500 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.472+0000 7f9b690e8700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b64166430 con 0x7f9b6406aeb0 2022-06-27T12:07:43.500 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.473+0000 7f9b6259c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9b6416a1b0 0x7f9b64068330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:43.500 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.473+0000 7f9b6259c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9b6416a1b0 0x7f9b64068330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48838/0 (socket says 172.21.15.195:48838) 2022-06-27T12:07:43.500 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.473+0000 7f9b6259c700 1 -- 172.21.15.195:0/3924684714 learned_addr learned my addr 172.21.15.195:0/3924684714 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:07:43.501 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.473+0000 7f9b62d9d700 1 --2- 172.21.15.195:0/3924684714 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9b44003ed0 0x7f9b6414f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:43.501 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.473+0000 7f9b6359e700 1 --2- 172.21.15.195:0/3924684714 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9b6406aeb0 0x7f9b64068870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:43.501 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.473+0000 7f9b6259c700 1 -- 172.21.15.195:0/3924684714 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9b6406aeb0 msgr2=0x7f9b64068870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:43.501 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.473+0000 7f9b6259c700 1 --2- 172.21.15.195:0/3924684714 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9b6406aeb0 0x7f9b64068870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:43.502 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.473+0000 7f9b6259c700 1 -- 172.21.15.195:0/3924684714 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9b44003ed0 msgr2=0x7f9b6414f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:43.502 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.473+0000 7f9b6259c700 1 --2- 172.21.15.195:0/3924684714 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9b44003ed0 0x7f9b6414f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:43.502 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.473+0000 7f9b6259c700 1 -- 172.21.15.195:0/3924684714 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b64166710 con 0x7f9b6416a1b0 2022-06-27T12:07:43.502 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.473+0000 7f9b6259c700 1 --2- 172.21.15.195:0/3924684714 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9b6416a1b0 0x7f9b64068330 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f9b5400e4b0 tx=0x7f9b5401a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:07:43.503 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.473+0000 7f9b5affd700 1 -- 172.21.15.195:0/3924684714 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b54021070 con 0x7f9b6416a1b0 2022-06-27T12:07:43.503 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.473+0000 7f9b5affd700 1 -- 172.21.15.195:0/3924684714 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b540043f0 con 0x7f9b6416a1b0 2022-06-27T12:07:43.503 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.473+0000 7f9b690e8700 1 -- 172.21.15.195:0/3924684714 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b64167040 con 0x7f9b6416a1b0 2022-06-27T12:07:43.503 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.473+0000 7f9b5affd700 1 -- 172.21.15.195:0/3924684714 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b54003770 con 0x7f9b6416a1b0 2022-06-27T12:07:43.504 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.473+0000 7f9b690e8700 1 -- 172.21.15.195:0/3924684714 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b64166cd0 con 0x7f9b6416a1b0 2022-06-27T12:07:43.504 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.474+0000 7f9b690e8700 1 -- 172.21.15.195:0/3924684714 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b641671d0 con 0x7f9b6416a1b0 2022-06-27T12:07:43.504 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.474+0000 7f9b5affd700 1 -- 172.21.15.195:0/3924684714 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f9b5402e030 con 0x7f9b6416a1b0 2022-06-27T12:07:43.504 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.475+0000 7f9b5affd700 1 --2- 172.21.15.195:0/3924684714 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9b48063d00 0x7f9b480661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:43.504 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.475+0000 7f9b5affd700 1 -- 172.21.15.195:0/3924684714 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7062+0+0 (secure 0 0 0) 0x7f9b54082260 con 0x7f9b6416a1b0 2022-06-27T12:07:43.505 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.475+0000 7f9b62d9d700 1 --2- 172.21.15.195:0/3924684714 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9b48063d00 0x7f9b480661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:43.505 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.475+0000 7f9b62d9d700 1 --2- 172.21.15.195:0/3924684714 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9b48063d00 0x7f9b480661b0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f9b4c000c00 tx=0x7f9b4c009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:07:43.505 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.479+0000 7f9b5affd700 1 -- 172.21.15.195:0/3924684714 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f9b54050b40 con 0x7f9b6416a1b0 2022-06-27T12:07:43.705 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.705+0000 7f9b690e8700 1 -- 172.21.15.195:0/3924684714 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9b640722c0 con 0x7f9b6416a1b0 2022-06-27T12:07:43.706 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.705+0000 7f9b5affd700 1 -- 172.21.15.195:0/3924684714 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+6128 (secure 0 0 0) 0x7f9b540540e0 con 0x7f9b6416a1b0 2022-06-27T12:07:43.707 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:07:43.708 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":6271,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/1603307620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":1603307620},{"type":"v1","addr":"172.21.15.202:6835","nonce":1603307620}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:07:08.223429+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5591},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5591":{"gid":5591,"name":"b","rank":0,"incarnation":67,"state":"up:active","state_seq":23,"addr":"172.21.15.202:6833/3022591968","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3022591968},{"type":"v1","addr":"172.21.15.202:6833","nonce":3022591968}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:07:43.710 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.707+0000 7f9b690e8700 1 -- 172.21.15.195:0/3924684714 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9b48063d00 msgr2=0x7f9b480661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:43.711 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.707+0000 7f9b690e8700 1 --2- 172.21.15.195:0/3924684714 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9b48063d00 0x7f9b480661b0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f9b4c000c00 tx=0x7f9b4c009040 comp rx=0 tx=0).stop 2022-06-27T12:07:43.711 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.707+0000 7f9b690e8700 1 -- 172.21.15.195:0/3924684714 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9b6416a1b0 msgr2=0x7f9b64068330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:43.711 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.707+0000 7f9b690e8700 1 --2- 172.21.15.195:0/3924684714 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9b6416a1b0 0x7f9b64068330 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f9b5400e4b0 tx=0x7f9b5401a000 comp rx=0 tx=0).stop 2022-06-27T12:07:43.712 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.707+0000 7f9b690e8700 1 -- 172.21.15.195:0/3924684714 shutdown_connections 2022-06-27T12:07:43.712 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.707+0000 7f9b690e8700 1 --2- 172.21.15.195:0/3924684714 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9b48063d00 0x7f9b480661b0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:43.713 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.707+0000 7f9b690e8700 1 --2- 172.21.15.195:0/3924684714 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9b6406aeb0 0x7f9b64068870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:43.713 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.707+0000 7f9b690e8700 1 --2- 172.21.15.195:0/3924684714 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9b44003ed0 0x7f9b6414f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:43.714 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.707+0000 7f9b690e8700 1 --2- 172.21.15.195:0/3924684714 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9b6416a1b0 0x7f9b64068330 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:43.714 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.707+0000 7f9b690e8700 1 -- 172.21.15.195:0/3924684714 >> 172.21.15.195:0/3924684714 conn(0x7f9b640c6350 msgr2=0x7f9b640c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:07:43.715 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.707+0000 7f9b690e8700 1 -- 172.21.15.195:0/3924684714 shutdown_connections 2022-06-27T12:07:43.718 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:43.707+0000 7f9b690e8700 1 -- 172.21.15.195:0/3924684714 wait complete. 2022-06-27T12:07:43.718 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 72 2022-06-27T12:07:43.721 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2022-06-27T12:07:45.722 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:07:45.944 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:45.944+0000 7f5ec9ea4700 1 Processor -- start 2022-06-27T12:07:45.945 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:45.944+0000 7f5ec9ea4700 1 -- start start 2022-06-27T12:07:45.947 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:45.945+0000 7f5ec9ea4700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f5ec4154900 0x7f5ec40c2120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:45.947 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:45.945+0000 7f5ec9ea4700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ec4154da0 con 0x7f5ec4151560 2022-06-27T12:07:45.947 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:45.945+0000 7f5ec9ea4700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ec40c12a0 con 0x7f5ec4151930 2022-06-27T12:07:45.948 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:45.945+0000 7f5ec9ea4700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ec40c1580 con 0x7f5ec4154900 2022-06-27T12:07:45.948 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:45.945+0000 7f5ec37fe700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f5ec4151560 0x7f5ec41541f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:55052/0 (socket says 172.21.15.195:55052) 2022-06-27T12:07:45.948 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:45.945+0000 7f5ec37fe700 1 -- 172.21.15.195:0/2783590126 learned_addr learned my addr 172.21.15.195:0/2783590126 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:07:45.949 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:45.945+0000 7f5ec2ffd700 1 --2- 172.21.15.195:0/2783590126 >> v2:172.21.15.202:3300/0 conn(0x7f5ec4154900 0x7f5ec40c2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:45.949 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:45.945+0000 7f5ec27fc700 1 -- 172.21.15.195:0/2783590126 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3871851518 0 0) 0x7f5ec4154da0 con 0x7f5ec4151560 2022-06-27T12:07:45.950 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:45.945+0000 7f5ec27fc700 1 -- 172.21.15.195:0/2783590126 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5eb0003430 con 0x7f5ec4151560 2022-06-27T12:07:45.963 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:45.945+0000 7f5ec2ffd700 1 -- 172.21.15.195:0/2783590126 >> v1:172.21.15.202:6789/0 conn(0x7f5ec4151930 legacy=0x7f5ec415eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:45.964 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:45.945+0000 7f5ec2ffd700 1 -- 172.21.15.195:0/2783590126 >> v1:172.21.15.195:6789/0 conn(0x7f5ec4151560 legacy=0x7f5ec41541f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:45.964 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:45.945+0000 7f5ec2ffd700 1 -- 172.21.15.195:0/2783590126 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ec40c29f0 con 0x7f5ec4154900 2022-06-27T12:07:45.965 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:45.946+0000 7f5ec2ffd700 1 --2- 172.21.15.195:0/2783590126 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5ec4154900 0x7f5ec40c2120 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f5eb40097a0 tx=0x7f5eb4009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=159843102854ea6d server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:07:45.965 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:45.946+0000 7f5ec27fc700 1 -- 172.21.15.195:0/2783590126 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb4011630 con 0x7f5ec4154900 2022-06-27T12:07:45.966 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:45.946+0000 7f5ec27fc700 1 -- 172.21.15.195:0/2783590126 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5eb4011790 con 0x7f5ec4154900 2022-06-27T12:07:45.966 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:45.946+0000 7f5ec27fc700 1 -- 172.21.15.195:0/2783590126 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb4011b30 con 0x7f5ec4154900 2022-06-27T12:07:45.967 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:45.946+0000 7f5ec9ea4700 1 -- 172.21.15.195:0/2783590126 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5ec4154900 msgr2=0x7f5ec40c2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:45.967 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:45.946+0000 7f5ec9ea4700 1 --2- 172.21.15.195:0/2783590126 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5ec4154900 0x7f5ec40c2120 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f5eb40097a0 tx=0x7f5eb4009060 comp rx=0 tx=0).stop 2022-06-27T12:07:45.967 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:45.946+0000 7f5ec9ea4700 1 -- 172.21.15.195:0/2783590126 shutdown_connections 2022-06-27T12:07:45.968 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:45.946+0000 7f5ec9ea4700 1 --2- 172.21.15.195:0/2783590126 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5ec4154900 0x7f5ec40c2120 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:45.969 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:45.946+0000 7f5ec9ea4700 1 -- 172.21.15.195:0/2783590126 >> 172.21.15.195:0/2783590126 conn(0x7f5ec40c6350 msgr2=0x7f5ec40c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:07:45.969 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:45.946+0000 7f5ec9ea4700 1 -- 172.21.15.195:0/2783590126 shutdown_connections 2022-06-27T12:07:45.970 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:45.946+0000 7f5ec9ea4700 1 -- 172.21.15.195:0/2783590126 wait complete. 2022-06-27T12:07:45.970 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:45.948+0000 7f5ec9ea4700 1 Processor -- start 2022-06-27T12:07:45.971 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:45.948+0000 7f5ec9ea4700 1 -- start start 2022-06-27T12:07:45.971 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:45.949+0000 7f5ec9ea4700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5ec4151560 0x7f5ec40c49c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:45.971 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:45.949+0000 7f5ec9ea4700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5ec4151930 0x7f5ec41642b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:45.972 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:45.949+0000 7f5ec9ea4700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5ec4154900 0x7f5ec41647f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:45.972 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:45.949+0000 7f5ec9ea4700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ec414cda0 con 0x7f5ec4151930 2022-06-27T12:07:45.973 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:45.949+0000 7f5ec9ea4700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ec414c870 con 0x7f5ec4151560 2022-06-27T12:07:45.973 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:45.949+0000 7f5ec9ea4700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ec4158200 con 0x7f5ec4154900 2022-06-27T12:07:45.973 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:45.949+0000 7f5ec2ffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5ec4151930 0x7f5ec41642b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:45.974 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:45.949+0000 7f5ec2ffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5ec4151930 0x7f5ec41642b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48872/0 (socket says 172.21.15.195:48872) 2022-06-27T12:07:45.974 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:45.949+0000 7f5ec2ffd700 1 -- 172.21.15.195:0/937738737 learned_addr learned my addr 172.21.15.195:0/937738737 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:07:45.975 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:45.949+0000 7f5ec37fe700 1 --2- 172.21.15.195:0/937738737 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5ec4151560 0x7f5ec40c49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:45.975 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:45.949+0000 7f5ec3fff700 1 --2- 172.21.15.195:0/937738737 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5ec4154900 0x7f5ec41647f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:45.976 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:45.949+0000 7f5ec2ffd700 1 -- 172.21.15.195:0/937738737 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5ec4154900 msgr2=0x7f5ec41647f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:45.976 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:45.949+0000 7f5ec2ffd700 1 --2- 172.21.15.195:0/937738737 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5ec4154900 0x7f5ec41647f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:45.977 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:45.949+0000 7f5ec2ffd700 1 -- 172.21.15.195:0/937738737 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5ec4151560 msgr2=0x7f5ec40c49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:45.977 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:45.949+0000 7f5ec2ffd700 1 --2- 172.21.15.195:0/937738737 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5ec4151560 0x7f5ec40c49c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:45.979 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:45.949+0000 7f5ec2ffd700 1 -- 172.21.15.195:0/937738737 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eb40092e0 con 0x7f5ec4151930 2022-06-27T12:07:45.979 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:45.949+0000 7f5ec3fff700 1 --2- 172.21.15.195:0/937738737 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5ec4154900 0x7f5ec41647f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T12:07:45.980 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:45.950+0000 7f5ec2ffd700 1 --2- 172.21.15.195:0/937738737 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5ec4151930 0x7f5ec41642b0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f5ec40650f0 tx=0x7f5eb4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:07:45.980 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:45.950+0000 7f5eab7fe700 1 -- 172.21.15.195:0/937738737 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb401be00 con 0x7f5ec4151930 2022-06-27T12:07:45.981 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:45.950+0000 7f5eab7fe700 1 -- 172.21.15.195:0/937738737 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5eb4011900 con 0x7f5ec4151930 2022-06-27T12:07:45.981 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:45.950+0000 7f5eab7fe700 1 -- 172.21.15.195:0/937738737 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb4025420 con 0x7f5ec4151930 2022-06-27T12:07:45.981 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:45.950+0000 7f5ec9ea4700 1 -- 172.21.15.195:0/937738737 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ec414cac0 con 0x7f5ec4151930 2022-06-27T12:07:45.982 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:45.950+0000 7f5ec9ea4700 1 -- 172.21.15.195:0/937738737 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ec4157a00 con 0x7f5ec4151930 2022-06-27T12:07:45.982 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:45.951+0000 7f5eab7fe700 1 -- 172.21.15.195:0/937738737 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f5eb402c030 con 0x7f5ec4151930 2022-06-27T12:07:45.985 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:45.951+0000 7f5ec9ea4700 1 -- 172.21.15.195:0/937738737 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ec4157580 con 0x7f5ec4151930 2022-06-27T12:07:45.985 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:45.951+0000 7f5eab7fe700 1 --2- 172.21.15.195:0/937738737 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5ea4063d00 0x7f5ea40661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:45.985 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:45.951+0000 7f5eab7fe700 1 -- 172.21.15.195:0/937738737 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7062+0+0 (secure 0 0 0) 0x7f5eb40821d0 con 0x7f5ec4151930 2022-06-27T12:07:45.986 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:45.951+0000 7f5ec37fe700 1 --2- 172.21.15.195:0/937738737 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5ea4063d00 0x7f5ea40661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:45.986 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:45.952+0000 7f5ec37fe700 1 --2- 172.21.15.195:0/937738737 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5ea4063d00 0x7f5ea40661b0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f5eb8007fd0 tx=0x7f5eb8009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:07:45.986 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:45.956+0000 7f5eab7fe700 1 -- 172.21.15.195:0/937738737 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f5ec4157580 con 0x7f5ec4151930 2022-06-27T12:07:46.182 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:46.182+0000 7f5ec9ea4700 1 -- 172.21.15.195:0/937738737 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ec4157580 con 0x7f5ec4151930 2022-06-27T12:07:46.183 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:46.183+0000 7f5eab7fe700 1 -- 172.21.15.195:0/937738737 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+6128 (secure 0 0 0) 0x7f5ec4157580 con 0x7f5ec4151930 2022-06-27T12:07:46.184 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:07:46.185 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":6271,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/1603307620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":1603307620},{"type":"v1","addr":"172.21.15.202:6835","nonce":1603307620}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:07:08.223429+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5591},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5591":{"gid":5591,"name":"b","rank":0,"incarnation":67,"state":"up:active","state_seq":23,"addr":"172.21.15.202:6833/3022591968","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3022591968},{"type":"v1","addr":"172.21.15.202:6833","nonce":3022591968}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:07:46.187 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:46.184+0000 7f5ec9ea4700 1 -- 172.21.15.195:0/937738737 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5ea4063d00 msgr2=0x7f5ea40661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:46.188 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:46.184+0000 7f5ec9ea4700 1 --2- 172.21.15.195:0/937738737 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5ea4063d00 0x7f5ea40661b0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f5eb8007fd0 tx=0x7f5eb8009040 comp rx=0 tx=0).stop 2022-06-27T12:07:46.188 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:46.184+0000 7f5ec9ea4700 1 -- 172.21.15.195:0/937738737 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5ec4151930 msgr2=0x7f5ec41642b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:46.189 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:46.184+0000 7f5ec9ea4700 1 --2- 172.21.15.195:0/937738737 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5ec4151930 0x7f5ec41642b0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f5ec40650f0 tx=0x7f5eb4017000 comp rx=0 tx=0).stop 2022-06-27T12:07:46.189 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:46.185+0000 7f5ec9ea4700 1 -- 172.21.15.195:0/937738737 shutdown_connections 2022-06-27T12:07:46.190 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:46.185+0000 7f5ec9ea4700 1 --2- 172.21.15.195:0/937738737 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5ea4063d00 0x7f5ea40661b0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:46.190 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:46.185+0000 7f5ec9ea4700 1 --2- 172.21.15.195:0/937738737 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5ec4154900 0x7f5ec41647f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:46.191 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:46.185+0000 7f5ec9ea4700 1 --2- 172.21.15.195:0/937738737 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5ec4151560 0x7f5ec40c49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:46.191 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:46.185+0000 7f5ec9ea4700 1 --2- 172.21.15.195:0/937738737 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5ec4151930 0x7f5ec41642b0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:46.191 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:46.185+0000 7f5ec9ea4700 1 -- 172.21.15.195:0/937738737 >> 172.21.15.195:0/937738737 conn(0x7f5ec40c6350 msgr2=0x7f5ec406cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:07:46.192 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:46.185+0000 7f5ec9ea4700 1 -- 172.21.15.195:0/937738737 shutdown_connections 2022-06-27T12:07:46.192 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:46.185+0000 7f5ec9ea4700 1 -- 172.21.15.195:0/937738737 wait complete. 2022-06-27T12:07:46.193 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 72 2022-06-27T12:07:46.199 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2022-06-27T12:07:48.201 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:07:48.357 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.336+0000 7f8047b7d700 1 Processor -- start 2022-06-27T12:07:48.358 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.337+0000 7f8047b7d700 1 -- start start 2022-06-27T12:07:48.360 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.337+0000 7f8047b7d700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f804006e850 0x7f80401570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:48.360 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.338+0000 7f8047b7d700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f804006fba0 con 0x7f804006fef0 2022-06-27T12:07:48.361 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.338+0000 7f8047b7d700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f804006f5e0 con 0x7f80400c34a0 2022-06-27T12:07:48.361 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.338+0000 7f8047b7d700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80401575e0 con 0x7f804006e850 2022-06-27T12:07:48.362 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.338+0000 7f8045118700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f804006fef0 0x7f8040069de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:55066/0 (socket says 172.21.15.195:55066) 2022-06-27T12:07:48.362 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.338+0000 7f8045118700 1 -- 172.21.15.195:0/3752353498 learned_addr learned my addr 172.21.15.195:0/3752353498 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:07:48.363 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.338+0000 7f8044917700 1 -- 172.21.15.195:0/3752353498 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2645644393 0 0) 0x7f804006fba0 con 0x7f804006fef0 2022-06-27T12:07:48.363 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.338+0000 7f8045919700 1 --2- 172.21.15.195:0/3752353498 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f804006e850 0x7f80401570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:48.363 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.338+0000 7f8044917700 1 -- 172.21.15.195:0/3752353498 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8028003430 con 0x7f804006fef0 2022-06-27T12:07:48.364 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.338+0000 7f8044917700 1 -- 172.21.15.195:0/3752353498 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 306943273 0 0) 0x7f8028003430 con 0x7f804006fef0 2022-06-27T12:07:48.364 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.338+0000 7f8044917700 1 -- 172.21.15.195:0/3752353498 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f804006e850 msgr2=0x7f80401570a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:07:48.365 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.338+0000 7f8044917700 1 --2- 172.21.15.195:0/3752353498 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f804006e850 0x7f80401570a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:48.366 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.338+0000 7f8044917700 1 -- 172.21.15.195:0/3752353498 >> v1:172.21.15.202:6789/0 conn(0x7f80400c34a0 legacy=0x7f80400c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:48.367 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.338+0000 7f8044917700 1 -- 172.21.15.195:0/3752353498 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f804006f8c0 con 0x7f804006fef0 2022-06-27T12:07:48.368 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.338+0000 7f8044917700 1 -- 172.21.15.195:0/3752353498 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7f8034001470 con 0x7f804006fef0 2022-06-27T12:07:48.368 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.338+0000 7f8044917700 1 -- 172.21.15.195:0/3752353498 >> v1:172.21.15.195:6789/0 conn(0x7f804006fef0 legacy=0x7f8040069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:48.369 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.338+0000 7f8044917700 1 --2- 172.21.15.195:0/3752353498 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f804006e850 0x7f8028003950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:48.369 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.338+0000 7f8044917700 1 --2- 172.21.15.195:0/3752353498 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8028003e90 0x7f80280046d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:48.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.338+0000 7f8044917700 1 --2- 172.21.15.195:0/3752353498 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8028004c10 0x7f8028005080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:48.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.338+0000 7f8044917700 1 -- 172.21.15.195:0/3752353498 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80401575e0 con 0x7f8028003e90 2022-06-27T12:07:48.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.338+0000 7f8044917700 1 -- 172.21.15.195:0/3752353498 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f802800ba00 con 0x7f8028004c10 2022-06-27T12:07:48.372 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.338+0000 7f8044917700 1 -- 172.21.15.195:0/3752353498 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f802800bbc0 con 0x7f804006e850 2022-06-27T12:07:48.372 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.338+0000 7f8045118700 1 --2- 172.21.15.195:0/3752353498 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8028003e90 0x7f80280046d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:48.373 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.339+0000 7f8045919700 1 --2- 172.21.15.195:0/3752353498 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f804006e850 0x7f8028003950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:48.373 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.339+0000 7f804611a700 1 --2- 172.21.15.195:0/3752353498 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8028004c10 0x7f8028005080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:48.374 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.339+0000 7f8045118700 1 -- 172.21.15.195:0/3752353498 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f804006e850 msgr2=0x7f8028003950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:07:48.374 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.339+0000 7f8045118700 1 --2- 172.21.15.195:0/3752353498 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f804006e850 0x7f8028003950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:48.375 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.339+0000 7f8045118700 1 -- 172.21.15.195:0/3752353498 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8028004c10 msgr2=0x7f8028005080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:07:48.376 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.339+0000 7f8045118700 1 --2- 172.21.15.195:0/3752353498 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8028004c10 0x7f8028005080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:48.376 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.339+0000 7f8045118700 1 -- 172.21.15.195:0/3752353498 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8028003430 con 0x7f8028003e90 2022-06-27T12:07:48.376 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.339+0000 7f8045118700 1 --2- 172.21.15.195:0/3752353498 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8028003e90 0x7f80280046d0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f80400c73e0 tx=0x7f8034001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1031874c068d523b server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:07:48.377 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.339+0000 7f8044917700 1 -- 172.21.15.195:0/3752353498 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f803400ca60 con 0x7f8028003e90 2022-06-27T12:07:48.377 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.339+0000 7f8044917700 1 -- 172.21.15.195:0/3752353498 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f803400cbc0 con 0x7f8028003e90 2022-06-27T12:07:48.378 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.339+0000 7f8044917700 1 -- 172.21.15.195:0/3752353498 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80340070d0 con 0x7f8028003e90 2022-06-27T12:07:48.378 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.339+0000 7f8047b7d700 1 -- 172.21.15.195:0/3752353498 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8028003e90 msgr2=0x7f80280046d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:48.379 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.340+0000 7f8047b7d700 1 --2- 172.21.15.195:0/3752353498 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8028003e90 0x7f80280046d0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f80400c73e0 tx=0x7f8034001e50 comp rx=0 tx=0).stop 2022-06-27T12:07:48.379 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.340+0000 7f804611a700 1 -- 172.21.15.195:0/3752353498 reap_dead start 2022-06-27T12:07:48.380 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.340+0000 7f8047b7d700 1 -- 172.21.15.195:0/3752353498 shutdown_connections 2022-06-27T12:07:48.381 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.340+0000 7f8047b7d700 1 -- 172.21.15.195:0/3752353498 >> 172.21.15.195:0/3752353498 conn(0x7f80400c6350 msgr2=0x7f80400c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:07:48.382 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.340+0000 7f8047b7d700 1 -- 172.21.15.195:0/3752353498 shutdown_connections 2022-06-27T12:07:48.382 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.340+0000 7f8047b7d700 1 -- 172.21.15.195:0/3752353498 wait complete. 2022-06-27T12:07:48.382 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.341+0000 7f8047b7d700 1 Processor -- start 2022-06-27T12:07:48.383 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.341+0000 7f8047b7d700 1 -- start start 2022-06-27T12:07:48.383 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.341+0000 7f8047b7d700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8028003e90 0x7f8040154960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:48.384 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.342+0000 7f8047b7d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f80401542e0 0x7f804014c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:48.384 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.342+0000 7f8047b7d700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f80401552f0 0x7f80401667c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:48.385 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.342+0000 7f8047b7d700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8040166370 con 0x7f80401542e0 2022-06-27T12:07:48.385 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.342+0000 7f8047b7d700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8040146f90 con 0x7f80401552f0 2022-06-27T12:07:48.386 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.342+0000 7f8047b7d700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80400c3250 con 0x7f8028003e90 2022-06-27T12:07:48.386 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.342+0000 7f8045118700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f80401542e0 0x7f804014c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:48.387 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.342+0000 7f8045118700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f80401542e0 0x7f804014c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48892/0 (socket says 172.21.15.195:48892) 2022-06-27T12:07:48.387 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.342+0000 7f8045118700 1 -- 172.21.15.195:0/3867848679 learned_addr learned my addr 172.21.15.195:0/3867848679 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:07:48.388 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.342+0000 7f8045919700 1 --2- 172.21.15.195:0/3867848679 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8028003e90 0x7f8040154960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:48.388 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.342+0000 7f804611a700 1 --2- 172.21.15.195:0/3867848679 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f80401552f0 0x7f80401667c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:48.389 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.342+0000 7f8045118700 1 -- 172.21.15.195:0/3867848679 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8028003e90 msgr2=0x7f8040154960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:48.389 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.342+0000 7f8045118700 1 --2- 172.21.15.195:0/3867848679 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8028003e90 0x7f8040154960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:48.390 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.342+0000 7f8045118700 1 -- 172.21.15.195:0/3867848679 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f80401552f0 msgr2=0x7f80401667c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:48.394 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.342+0000 7f8045118700 1 --2- 172.21.15.195:0/3867848679 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f80401552f0 0x7f80401667c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:48.394 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.342+0000 7f8045118700 1 -- 172.21.15.195:0/3867848679 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8040147230 con 0x7f80401542e0 2022-06-27T12:07:48.395 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.342+0000 7f8045118700 1 --2- 172.21.15.195:0/3867848679 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f80401542e0 0x7f804014c600 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f802800b9d0 tx=0x7f8034017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:07:48.395 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.342+0000 7f8031ffb700 1 -- 172.21.15.195:0/3867848679 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8034001640 con 0x7f80401542e0 2022-06-27T12:07:48.396 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.342+0000 7f8031ffb700 1 -- 172.21.15.195:0/3867848679 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80340017c0 con 0x7f80401542e0 2022-06-27T12:07:48.396 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.342+0000 7f8047b7d700 1 -- 172.21.15.195:0/3867848679 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8040147510 con 0x7f80401542e0 2022-06-27T12:07:48.397 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.342+0000 7f8031ffb700 1 -- 172.21.15.195:0/3867848679 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8034004940 con 0x7f80401542e0 2022-06-27T12:07:48.397 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.343+0000 7f8047b7d700 1 -- 172.21.15.195:0/3867848679 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8040154ff0 con 0x7f80401542e0 2022-06-27T12:07:48.397 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.343+0000 7f8047b7d700 1 -- 172.21.15.195:0/3867848679 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80401664e0 con 0x7f80401542e0 2022-06-27T12:07:48.398 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.344+0000 7f8031ffb700 1 -- 172.21.15.195:0/3867848679 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f8034004cf0 con 0x7f80401542e0 2022-06-27T12:07:48.399 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.344+0000 7f8031ffb700 1 --2- 172.21.15.195:0/3867848679 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8020063d30 0x7f80200661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:48.399 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.344+0000 7f8031ffb700 1 -- 172.21.15.195:0/3867848679 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7062+0+0 (secure 0 0 0) 0x7f8034081f30 con 0x7f80401542e0 2022-06-27T12:07:48.400 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.344+0000 7f8045919700 1 --2- 172.21.15.195:0/3867848679 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8020063d30 0x7f80200661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:48.400 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.345+0000 7f8045919700 1 --2- 172.21.15.195:0/3867848679 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8020063d30 0x7f80200661e0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f803c006dc0 tx=0x7f803c008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:07:48.401 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.349+0000 7f8031ffb700 1 -- 172.21.15.195:0/3867848679 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f80340b1c90 con 0x7f80401542e0 2022-06-27T12:07:48.585 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.575+0000 7f8047b7d700 1 -- 172.21.15.195:0/3867848679 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f80400722c0 con 0x7f80401542e0 2022-06-27T12:07:48.585 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.576+0000 7f8031ffb700 1 -- 172.21.15.195:0/3867848679 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+6128 (secure 0 0 0) 0x7f8034050730 con 0x7f80401542e0 2022-06-27T12:07:48.586 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:07:48.586 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":6271,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/1603307620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":1603307620},{"type":"v1","addr":"172.21.15.202:6835","nonce":1603307620}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:07:08.223429+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5591},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5591":{"gid":5591,"name":"b","rank":0,"incarnation":67,"state":"up:active","state_seq":23,"addr":"172.21.15.202:6833/3022591968","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3022591968},{"type":"v1","addr":"172.21.15.202:6833","nonce":3022591968}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:07:48.590 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.577+0000 7f8047b7d700 1 -- 172.21.15.195:0/3867848679 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8020063d30 msgr2=0x7f80200661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:48.591 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.577+0000 7f8047b7d700 1 --2- 172.21.15.195:0/3867848679 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8020063d30 0x7f80200661e0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f803c006dc0 tx=0x7f803c008040 comp rx=0 tx=0).stop 2022-06-27T12:07:48.591 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.577+0000 7f8047b7d700 1 -- 172.21.15.195:0/3867848679 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f80401542e0 msgr2=0x7f804014c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:48.591 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.577+0000 7f8047b7d700 1 --2- 172.21.15.195:0/3867848679 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f80401542e0 0x7f804014c600 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f802800b9d0 tx=0x7f8034017000 comp rx=0 tx=0).stop 2022-06-27T12:07:48.592 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.578+0000 7f8047b7d700 1 -- 172.21.15.195:0/3867848679 shutdown_connections 2022-06-27T12:07:48.595 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.578+0000 7f8047b7d700 1 --2- 172.21.15.195:0/3867848679 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8020063d30 0x7f80200661e0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:48.595 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.578+0000 7f8047b7d700 1 --2- 172.21.15.195:0/3867848679 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f80401552f0 0x7f80401667c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:48.595 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.578+0000 7f8047b7d700 1 --2- 172.21.15.195:0/3867848679 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8028003e90 0x7f8040154960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:48.596 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.578+0000 7f8047b7d700 1 --2- 172.21.15.195:0/3867848679 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f80401542e0 0x7f804014c600 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:48.596 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.578+0000 7f8047b7d700 1 -- 172.21.15.195:0/3867848679 >> 172.21.15.195:0/3867848679 conn(0x7f80400c6350 msgr2=0x7f80400c4fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:07:48.596 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.578+0000 7f8047b7d700 1 -- 172.21.15.195:0/3867848679 shutdown_connections 2022-06-27T12:07:48.597 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:48.578+0000 7f8047b7d700 1 -- 172.21.15.195:0/3867848679 wait complete. 2022-06-27T12:07:48.597 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 72 2022-06-27T12:07:48.597 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2022-06-27T12:07:50.599 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:07:50.737 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.732+0000 7f6771b48700 1 Processor -- start 2022-06-27T12:07:50.740 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.732+0000 7f6771b48700 1 -- start start 2022-06-27T12:07:50.740 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.733+0000 7f6771b48700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f676c0c40d0 0x7f676c157010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:50.741 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.733+0000 7f6771b48700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f676c067ed0 0x7f676c14cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:50.741 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.733+0000 7f6771b48700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f676c156b80 con 0x7f676c0c6190 2022-06-27T12:07:50.742 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.733+0000 7f6771b48700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f676c0682f0 con 0x7f676c067ed0 2022-06-27T12:07:50.742 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.733+0000 7f6771b48700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f676c06a040 con 0x7f676c0c40d0 2022-06-27T12:07:50.753 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.733+0000 7f676bfff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f676c067ed0 0x7f676c14cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:50.754 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.733+0000 7f676bfff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f676c067ed0 0x7f676c14cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48902/0 (socket says 172.21.15.195:48902) 2022-06-27T12:07:50.755 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.733+0000 7f676bfff700 1 -- 172.21.15.195:0/665612654 learned_addr learned my addr 172.21.15.195:0/665612654 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:07:50.756 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.734+0000 7f676b7fe700 1 --2- 172.21.15.195:0/665612654 >> v2:172.21.15.202:3300/0 conn(0x7f676c0c40d0 0x7f676c157010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:50.756 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.734+0000 7f676bfff700 1 -- 172.21.15.195:0/665612654 >> v2:172.21.15.202:3300/0 conn(0x7f676c0c40d0 msgr2=0x7f676c157010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:07:50.757 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.734+0000 7f676bfff700 1 --2- 172.21.15.195:0/665612654 >> v2:172.21.15.202:3300/0 conn(0x7f676c0c40d0 0x7f676c157010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:50.757 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.734+0000 7f676bfff700 1 -- 172.21.15.195:0/665612654 >> v1:172.21.15.195:6789/0 conn(0x7f676c0c6190 legacy=0x7f676c0c44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:50.758 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.734+0000 7f676bfff700 1 -- 172.21.15.195:0/665612654 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f676c06a300 con 0x7f676c067ed0 2022-06-27T12:07:50.758 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.734+0000 7f676a7fc700 1 -- 172.21.15.195:0/665612654 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 694147274 0 0) 0x7f676c156b80 con 0x7f676c0c6190 2022-06-27T12:07:50.759 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.734+0000 7f676bfff700 1 --2- 172.21.15.195:0/665612654 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f676c067ed0 0x7f676c14cb80 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f676c0cdb00 tx=0x7f6764009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90e39a36d9b71910 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:07:50.759 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.734+0000 7f676a7fc700 1 -- 172.21.15.195:0/665612654 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f676400e070 con 0x7f676c067ed0 2022-06-27T12:07:50.760 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.734+0000 7f676a7fc700 1 -- 172.21.15.195:0/665612654 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f676400ba50 con 0x7f676c067ed0 2022-06-27T12:07:50.760 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.734+0000 7f676a7fc700 1 -- 172.21.15.195:0/665612654 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67640092e0 con 0x7f676c067ed0 2022-06-27T12:07:50.761 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.734+0000 7f6771b48700 1 -- 172.21.15.195:0/665612654 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f676c067ed0 msgr2=0x7f676c14cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:50.762 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.735+0000 7f6771b48700 1 --2- 172.21.15.195:0/665612654 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f676c067ed0 0x7f676c14cb80 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f676c0cdb00 tx=0x7f6764009840 comp rx=0 tx=0).stop 2022-06-27T12:07:50.762 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.735+0000 7f6771b48700 1 -- 172.21.15.195:0/665612654 shutdown_connections 2022-06-27T12:07:50.763 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.735+0000 7f6771b48700 1 --2- 172.21.15.195:0/665612654 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f676c067ed0 0x7f676c14cb80 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:50.764 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.735+0000 7f6771b48700 1 --2- 172.21.15.195:0/665612654 >> v2:172.21.15.202:3300/0 conn(0x7f676c0c40d0 0x7f676c157010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:50.765 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.735+0000 7f6771b48700 1 -- 172.21.15.195:0/665612654 >> 172.21.15.195:0/665612654 conn(0x7f676c0ccb80 msgr2=0x7f676c0ccf80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:07:50.765 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.735+0000 7f6771b48700 1 -- 172.21.15.195:0/665612654 shutdown_connections 2022-06-27T12:07:50.766 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.735+0000 7f6771b48700 1 -- 172.21.15.195:0/665612654 wait complete. 2022-06-27T12:07:50.767 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.736+0000 7f6771b48700 1 Processor -- start 2022-06-27T12:07:50.767 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.736+0000 7f6771b48700 1 -- start start 2022-06-27T12:07:50.768 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.737+0000 7f6771b48700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f676c067ed0 0x7f676c151cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:50.769 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.737+0000 7f6771b48700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f676c0c40d0 0x7f676c153120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:50.769 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.737+0000 7f6771b48700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f676c0c6190 0x7f676c0c0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:50.770 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.737+0000 7f6771b48700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f676c166c40 con 0x7f676c0c6190 2022-06-27T12:07:50.770 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.737+0000 7f6771b48700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f676c167100 con 0x7f676c0c40d0 2022-06-27T12:07:50.771 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.737+0000 7f6771b48700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f676c152680 con 0x7f676c067ed0 2022-06-27T12:07:50.771 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.737+0000 7f676bfff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f676c0c6190 0x7f676c0c0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:50.772 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.737+0000 7f676bfff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f676c0c6190 0x7f676c0c0890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48908/0 (socket says 172.21.15.195:48908) 2022-06-27T12:07:50.772 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.737+0000 7f676bfff700 1 -- 172.21.15.195:0/749749345 learned_addr learned my addr 172.21.15.195:0/749749345 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:07:50.773 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.737+0000 7f676affd700 1 --2- 172.21.15.195:0/749749345 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f676c0c40d0 0x7f676c153120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:50.773 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.737+0000 7f676b7fe700 1 --2- 172.21.15.195:0/749749345 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f676c067ed0 0x7f676c151cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:50.774 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.737+0000 7f676bfff700 1 -- 172.21.15.195:0/749749345 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f676c067ed0 msgr2=0x7f676c151cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:50.774 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.737+0000 7f676bfff700 1 --2- 172.21.15.195:0/749749345 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f676c067ed0 0x7f676c151cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:50.775 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.737+0000 7f676bfff700 1 -- 172.21.15.195:0/749749345 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f676c0c40d0 msgr2=0x7f676c153120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:50.775 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.737+0000 7f676bfff700 1 --2- 172.21.15.195:0/749749345 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f676c0c40d0 0x7f676c153120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:50.776 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.737+0000 7f676bfff700 1 -- 172.21.15.195:0/749749345 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6764009ae0 con 0x7f676c0c6190 2022-06-27T12:07:50.777 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.737+0000 7f676b7fe700 1 --2- 172.21.15.195:0/749749345 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f676c067ed0 0x7f676c151cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:07:50.777 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.737+0000 7f676bfff700 1 --2- 172.21.15.195:0/749749345 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f676c0c6190 0x7f676c0c0890 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f676c0697c0 tx=0x7f6764018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:07:50.778 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.737+0000 7f67637fe700 1 -- 172.21.15.195:0/749749345 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f676400be60 con 0x7f676c0c6190 2022-06-27T12:07:50.778 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.737+0000 7f67637fe700 1 -- 172.21.15.195:0/749749345 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6764003e00 con 0x7f676c0c6190 2022-06-27T12:07:50.779 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.737+0000 7f6771b48700 1 -- 172.21.15.195:0/749749345 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f676c166e20 con 0x7f676c0c6190 2022-06-27T12:07:50.780 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.737+0000 7f67637fe700 1 -- 172.21.15.195:0/749749345 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6764007a70 con 0x7f676c0c6190 2022-06-27T12:07:50.780 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.738+0000 7f6771b48700 1 -- 172.21.15.195:0/749749345 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f676c152c00 con 0x7f676c0c6190 2022-06-27T12:07:50.780 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.739+0000 7f6771b48700 1 -- 172.21.15.195:0/749749345 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f676c152e20 con 0x7f676c0c6190 2022-06-27T12:07:50.780 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.739+0000 7f67637fe700 1 -- 172.21.15.195:0/749749345 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f6764004040 con 0x7f676c0c6190 2022-06-27T12:07:50.781 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.739+0000 7f67637fe700 1 --2- 172.21.15.195:0/749749345 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6750063d00 0x7f67500661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:50.781 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.739+0000 7f67637fe700 1 -- 172.21.15.195:0/749749345 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7148+0+0 (secure 0 0 0) 0x7f6764082b30 con 0x7f676c0c6190 2022-06-27T12:07:50.781 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.739+0000 7f676b7fe700 1 --2- 172.21.15.195:0/749749345 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6750063d00 0x7f67500661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:50.781 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.740+0000 7f676b7fe700 1 --2- 172.21.15.195:0/749749345 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6750063d00 0x7f67500661b0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f6754006fd0 tx=0x7f6754008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:07:50.782 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.743+0000 7f67637fe700 1 -- 172.21.15.195:0/749749345 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f67640512e0 con 0x7f676c0c6190 2022-06-27T12:07:50.972 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.971+0000 7f6771b48700 1 -- 172.21.15.195:0/749749345 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f676c0722c0 con 0x7f676c0c6190 2022-06-27T12:07:50.973 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.972+0000 7f67637fe700 1 -- 172.21.15.195:0/749749345 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+5367 (secure 0 0 0) 0x7f676402a090 con 0x7f676c0c6190 2022-06-27T12:07:50.974 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:07:50.974 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:07:50.098891+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6271},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6271":{"gid":6271,"name":"d","rank":0,"incarnation":73,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6835/1603307620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":1603307620},{"type":"v1","addr":"172.21.15.202:6835","nonce":1603307620}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:07:50.979 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.973+0000 7f6771b48700 1 -- 172.21.15.195:0/749749345 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6750063d00 msgr2=0x7f67500661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:50.979 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.973+0000 7f6771b48700 1 --2- 172.21.15.195:0/749749345 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6750063d00 0x7f67500661b0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f6754006fd0 tx=0x7f6754008040 comp rx=0 tx=0).stop 2022-06-27T12:07:50.980 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.973+0000 7f6771b48700 1 -- 172.21.15.195:0/749749345 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f676c0c6190 msgr2=0x7f676c0c0890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:50.980 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.973+0000 7f6771b48700 1 --2- 172.21.15.195:0/749749345 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f676c0c6190 0x7f676c0c0890 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f676c0697c0 tx=0x7f6764018000 comp rx=0 tx=0).stop 2022-06-27T12:07:50.981 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.974+0000 7f6771b48700 1 -- 172.21.15.195:0/749749345 shutdown_connections 2022-06-27T12:07:50.981 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.974+0000 7f6771b48700 1 --2- 172.21.15.195:0/749749345 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6750063d00 0x7f67500661b0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:50.982 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.974+0000 7f6771b48700 1 --2- 172.21.15.195:0/749749345 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f676c0c6190 0x7f676c0c0890 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:50.982 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.974+0000 7f6771b48700 1 --2- 172.21.15.195:0/749749345 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f676c067ed0 0x7f676c151cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:50.983 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.974+0000 7f6771b48700 1 --2- 172.21.15.195:0/749749345 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f676c0c40d0 0x7f676c153120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:50.983 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.974+0000 7f6771b48700 1 -- 172.21.15.195:0/749749345 >> 172.21.15.195:0/749749345 conn(0x7f676c0ccb80 msgr2=0x7f676c0cbce0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:07:50.984 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.974+0000 7f6771b48700 1 -- 172.21.15.195:0/749749345 shutdown_connections 2022-06-27T12:07:50.984 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:50.974+0000 7f6771b48700 1 -- 172.21.15.195:0/749749345 wait complete. 2022-06-27T12:07:50.985 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 73 2022-06-27T12:07:50.987 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2022-06-27T12:07:50.988 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2022-06-27T12:07:50.988 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:07:51.123 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.111+0000 7f44ee1bd700 1 Processor -- start 2022-06-27T12:07:51.124 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.111+0000 7f44ee1bd700 1 -- start start 2022-06-27T12:07:51.124 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.112+0000 7f44ee1bd700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f44e806fef0 0x7f44e81570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:51.125 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.112+0000 7f44ee1bd700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f44e80c34a0 0x7f44e80c2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:51.125 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.112+0000 7f44ee1bd700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44e806fba0 con 0x7f44e806e850 2022-06-27T12:07:51.126 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.112+0000 7f44ee1bd700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44e81575e0 con 0x7f44e80c34a0 2022-06-27T12:07:51.127 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.112+0000 7f44ee1bd700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44e806f5e0 con 0x7f44e806fef0 2022-06-27T12:07:51.128 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.112+0000 7f44e6ffd700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f44e806fef0 0x7f44e81570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:51.128 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.112+0000 7f44e7fff700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f44e80c34a0 0x7f44e80c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:51.129 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.112+0000 7f44e6ffd700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f44e806fef0 0x7f44e81570a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3301/0 says I am v2:172.21.15.195:41670/0 (socket says 172.21.15.195:41670) 2022-06-27T12:07:51.129 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.112+0000 7f44e6ffd700 1 -- 172.21.15.195:0/4098964847 learned_addr learned my addr 172.21.15.195:0/4098964847 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:07:51.130 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.113+0000 7f44e7fff700 1 -- 172.21.15.195:0/4098964847 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f44e806fef0 msgr2=0x7f44e81570a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:07:51.130 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.113+0000 7f44e7fff700 1 --2- 172.21.15.195:0/4098964847 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f44e806fef0 0x7f44e81570a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:51.130 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.113+0000 7f44e7fff700 1 -- 172.21.15.195:0/4098964847 >> v1:172.21.15.202:6789/0 conn(0x7f44e806e850 legacy=0x7f44e8069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:51.131 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.113+0000 7f44e7fff700 1 -- 172.21.15.195:0/4098964847 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44e806f8c0 con 0x7f44e80c34a0 2022-06-27T12:07:51.131 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.113+0000 7f44e6ffd700 1 --2- 172.21.15.195:0/4098964847 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f44e806fef0 0x7f44e81570a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:07:51.132 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.113+0000 7f44e7fff700 1 --2- 172.21.15.195:0/4098964847 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f44e80c34a0 0x7f44e80c2920 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f44e0009e60 tx=0x7f44e0009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=397253a61bdb1fdb server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:07:51.132 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.113+0000 7f44e67fc700 1 -- 172.21.15.195:0/4098964847 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e000e070 con 0x7f44e80c34a0 2022-06-27T12:07:51.133 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.113+0000 7f44e67fc700 1 -- 172.21.15.195:0/4098964847 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44e000bbf0 con 0x7f44e80c34a0 2022-06-27T12:07:51.133 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.113+0000 7f44e67fc700 1 -- 172.21.15.195:0/4098964847 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e0012430 con 0x7f44e80c34a0 2022-06-27T12:07:51.134 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.114+0000 7f44ee1bd700 1 -- 172.21.15.195:0/4098964847 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f44e80c34a0 msgr2=0x7f44e80c2920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:51.134 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.114+0000 7f44ee1bd700 1 --2- 172.21.15.195:0/4098964847 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f44e80c34a0 0x7f44e80c2920 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f44e0009e60 tx=0x7f44e0009780 comp rx=0 tx=0).stop 2022-06-27T12:07:51.135 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.114+0000 7f44ee1bd700 1 -- 172.21.15.195:0/4098964847 shutdown_connections 2022-06-27T12:07:51.135 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.114+0000 7f44ee1bd700 1 --2- 172.21.15.195:0/4098964847 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f44e80c34a0 0x7f44e80c2920 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:51.136 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.114+0000 7f44ee1bd700 1 --2- 172.21.15.195:0/4098964847 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f44e806fef0 0x7f44e81570a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:51.136 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.114+0000 7f44ee1bd700 1 -- 172.21.15.195:0/4098964847 >> 172.21.15.195:0/4098964847 conn(0x7f44e80c6350 msgr2=0x7f44e80c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:07:51.137 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.114+0000 7f44ee1bd700 1 -- 172.21.15.195:0/4098964847 shutdown_connections 2022-06-27T12:07:51.138 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.114+0000 7f44ee1bd700 1 -- 172.21.15.195:0/4098964847 wait complete. 2022-06-27T12:07:51.138 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.115+0000 7f44ee1bd700 1 Processor -- start 2022-06-27T12:07:51.139 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.115+0000 7f44ee1bd700 1 -- start start 2022-06-27T12:07:51.139 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.116+0000 7f44ee1bd700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f44e806e850 0x7f44e8149850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:51.139 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.116+0000 7f44ee1bd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f44e80c34a0 0x7f44e80c43d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:51.140 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.116+0000 7f44ee1bd700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f44e814edf0 0x7f44e8156a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:51.140 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.116+0000 7f44ee1bd700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44e8149430 con 0x7f44e80c34a0 2022-06-27T12:07:51.141 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.116+0000 7f44ee1bd700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44e80bc020 con 0x7f44e806e850 2022-06-27T12:07:51.141 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.116+0000 7f44ee1bd700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44e81543c0 con 0x7f44e814edf0 2022-06-27T12:07:51.142 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.116+0000 7f44e6ffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f44e80c34a0 0x7f44e80c43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:51.142 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.116+0000 7f44e6ffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f44e80c34a0 0x7f44e80c43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48920/0 (socket says 172.21.15.195:48920) 2022-06-27T12:07:51.143 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.116+0000 7f44e6ffd700 1 -- 172.21.15.195:0/3900444937 learned_addr learned my addr 172.21.15.195:0/3900444937 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:07:51.144 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.116+0000 7f44e77fe700 1 --2- 172.21.15.195:0/3900444937 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f44e806e850 0x7f44e8149850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:51.144 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.116+0000 7f44e7fff700 1 --2- 172.21.15.195:0/3900444937 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f44e814edf0 0x7f44e8156a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:51.144 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.116+0000 7f44e6ffd700 1 -- 172.21.15.195:0/3900444937 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f44e814edf0 msgr2=0x7f44e8156a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:51.145 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.116+0000 7f44e6ffd700 1 --2- 172.21.15.195:0/3900444937 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f44e814edf0 0x7f44e8156a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:51.145 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.116+0000 7f44e6ffd700 1 -- 172.21.15.195:0/3900444937 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f44e806e850 msgr2=0x7f44e8149850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:51.146 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.116+0000 7f44e6ffd700 1 --2- 172.21.15.195:0/3900444937 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f44e806e850 0x7f44e8149850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:51.146 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.116+0000 7f44e6ffd700 1 -- 172.21.15.195:0/3900444937 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44d800b650 con 0x7f44e80c34a0 2022-06-27T12:07:51.147 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.116+0000 7f44e6ffd700 1 --2- 172.21.15.195:0/3900444937 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f44e80c34a0 0x7f44e80c43d0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f44d800ee10 tx=0x7f44d800b3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:07:51.147 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.116+0000 7f44df7fe700 1 -- 172.21.15.195:0/3900444937 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d800be30 con 0x7f44e80c34a0 2022-06-27T12:07:51.148 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.116+0000 7f44df7fe700 1 -- 172.21.15.195:0/3900444937 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44d800ddc0 con 0x7f44e80c34a0 2022-06-27T12:07:51.148 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.116+0000 7f44ee1bd700 1 -- 172.21.15.195:0/3900444937 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44e0009a20 con 0x7f44e80c34a0 2022-06-27T12:07:51.149 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.116+0000 7f44df7fe700 1 -- 172.21.15.195:0/3900444937 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d8010650 con 0x7f44e80c34a0 2022-06-27T12:07:51.149 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.116+0000 7f44ee1bd700 1 -- 172.21.15.195:0/3900444937 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44e8167240 con 0x7f44e80c34a0 2022-06-27T12:07:51.150 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.118+0000 7f44ee1bd700 1 -- 172.21.15.195:0/3900444937 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44e80722c0 con 0x7f44e80c34a0 2022-06-27T12:07:51.150 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.118+0000 7f44df7fe700 1 -- 172.21.15.195:0/3900444937 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f44d80109c0 con 0x7f44e80c34a0 2022-06-27T12:07:51.151 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.118+0000 7f44df7fe700 1 --2- 172.21.15.195:0/3900444937 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f44cc063d00 0x7f44cc0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:51.151 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.118+0000 7f44df7fe700 1 -- 172.21.15.195:0/3900444937 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7148+0+0 (secure 0 0 0) 0x7f44d8081140 con 0x7f44e80c34a0 2022-06-27T12:07:51.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.118+0000 7f44e77fe700 1 --2- 172.21.15.195:0/3900444937 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f44cc063d00 0x7f44cc0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:51.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.119+0000 7f44e77fe700 1 --2- 172.21.15.195:0/3900444937 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f44cc063d00 0x7f44cc0661b0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f44d00017e0 tx=0x7f44d0009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:07:51.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.123+0000 7f44df7fe700 1 -- 172.21.15.195:0/3900444937 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f44d804f8f0 con 0x7f44e80c34a0 2022-06-27T12:07:51.345 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.345+0000 7f44ee1bd700 1 -- 172.21.15.195:0/3900444937 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f44e80c9660 con 0x7f44e80c34a0 2022-06-27T12:07:51.346 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.345+0000 7f44df7fe700 1 -- 172.21.15.195:0/3900444937 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+5367 (secure 0 0 0) 0x7f44d801d020 con 0x7f44e80c34a0 2022-06-27T12:07:51.346 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:07:51.347 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:07:50.098891+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6271},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6271":{"gid":6271,"name":"d","rank":0,"incarnation":73,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6835/1603307620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":1603307620},{"type":"v1","addr":"172.21.15.202:6835","nonce":1603307620}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:07:51.353 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.348+0000 7f44ee1bd700 1 -- 172.21.15.195:0/3900444937 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f44cc063d00 msgr2=0x7f44cc0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:51.353 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.348+0000 7f44ee1bd700 1 --2- 172.21.15.195:0/3900444937 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f44cc063d00 0x7f44cc0661b0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f44d00017e0 tx=0x7f44d0009040 comp rx=0 tx=0).stop 2022-06-27T12:07:51.353 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.348+0000 7f44ee1bd700 1 -- 172.21.15.195:0/3900444937 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f44e80c34a0 msgr2=0x7f44e80c43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:51.354 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.348+0000 7f44ee1bd700 1 --2- 172.21.15.195:0/3900444937 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f44e80c34a0 0x7f44e80c43d0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f44d800ee10 tx=0x7f44d800b3b0 comp rx=0 tx=0).stop 2022-06-27T12:07:51.354 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.348+0000 7f44ee1bd700 1 -- 172.21.15.195:0/3900444937 shutdown_connections 2022-06-27T12:07:51.354 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.348+0000 7f44ee1bd700 1 --2- 172.21.15.195:0/3900444937 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f44cc063d00 0x7f44cc0661b0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:51.354 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.348+0000 7f44ee1bd700 1 --2- 172.21.15.195:0/3900444937 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f44e814edf0 0x7f44e8156a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:51.355 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.348+0000 7f44ee1bd700 1 --2- 172.21.15.195:0/3900444937 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f44e806e850 0x7f44e8149850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:51.355 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.348+0000 7f44ee1bd700 1 --2- 172.21.15.195:0/3900444937 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f44e80c34a0 0x7f44e80c43d0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:51.355 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.348+0000 7f44ee1bd700 1 -- 172.21.15.195:0/3900444937 >> 172.21.15.195:0/3900444937 conn(0x7f44e80c6350 msgr2=0x7f44e8069ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:07:51.355 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.348+0000 7f44ee1bd700 1 -- 172.21.15.195:0/3900444937 shutdown_connections 2022-06-27T12:07:51.356 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:51.348+0000 7f44ee1bd700 1 -- 172.21.15.195:0/3900444937 wait complete. 2022-06-27T12:07:51.356 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 73 2022-06-27T12:07:51.361 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 73, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-06-27T11:47:52.358095+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 73, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6271": { "addr": "172.21.15.202:6835/1603307620", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6834", "nonce": 1603307620, "type": "v2" }, { "addr": "172.21.15.202:6835", "nonce": 1603307620, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 6271, "incarnation": 73, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 59, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-06-27T12:07:50.098891+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6271 } } } ], "standbys": [ { "addr": "172.21.15.202:6837/3305006841", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6836", "nonce": 3305006841, "type": "v2" }, { "addr": "172.21.15.202:6837", "nonce": 3305006841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6835/2060709822", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6834", "nonce": 2060709822, "type": "v2" }, { "addr": "172.21.15.195:6835", "nonce": 2060709822, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6838/1598751652", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6836", "nonce": 1598751652, "type": "v2" }, { "addr": "172.21.15.195:6838", "nonce": 1598751652, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4275, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6839/1313800661", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6837", "nonce": 1313800661, "type": "v2" }, { "addr": "172.21.15.195:6839", "nonce": 1313800661, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4278, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-06-27T12:07:53.369 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:07:53.506 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.507+0000 7fe92c138700 1 Processor -- start 2022-06-27T12:07:53.507 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.507+0000 7fe92c138700 1 -- start start 2022-06-27T12:07:53.521 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.508+0000 7fe92c138700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe92406e850 0x7fe9241570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:53.521 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.508+0000 7fe92c138700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe92406fba0 con 0x7fe92406fef0 2022-06-27T12:07:53.522 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.508+0000 7fe92c138700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe92406f5e0 con 0x7fe9240c34a0 2022-06-27T12:07:53.522 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.508+0000 7fe92c138700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9241575e0 con 0x7fe92406e850 2022-06-27T12:07:53.523 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.508+0000 7fe9296d3700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fe92406fef0 0x7fe924069de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:55116/0 (socket says 172.21.15.195:55116) 2022-06-27T12:07:53.523 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.508+0000 7fe9296d3700 1 -- 172.21.15.195:0/785086228 learned_addr learned my addr 172.21.15.195:0/785086228 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:07:53.524 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.508+0000 7fe929ed4700 1 --2- 172.21.15.195:0/785086228 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe92406e850 0x7fe9241570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:53.525 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.508+0000 7fe928ed2700 1 -- 172.21.15.195:0/785086228 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2895151812 0 0) 0x7fe92406fba0 con 0x7fe92406fef0 2022-06-27T12:07:53.527 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.508+0000 7fe928ed2700 1 -- 172.21.15.195:0/785086228 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe90c003430 con 0x7fe92406fef0 2022-06-27T12:07:53.528 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.509+0000 7fe928ed2700 1 -- 172.21.15.195:0/785086228 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1816792661 0 0) 0x7fe92406f5e0 con 0x7fe9240c34a0 2022-06-27T12:07:53.528 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.509+0000 7fe928ed2700 1 -- 172.21.15.195:0/785086228 --> v1:172.21.15.202:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe92406fba0 con 0x7fe9240c34a0 2022-06-27T12:07:53.528 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.509+0000 7fe929ed4700 1 -- 172.21.15.195:0/785086228 >> v1:172.21.15.202:6789/0 conn(0x7fe9240c34a0 legacy=0x7fe9240c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:53.528 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.509+0000 7fe929ed4700 1 -- 172.21.15.195:0/785086228 >> v1:172.21.15.195:6789/0 conn(0x7fe92406fef0 legacy=0x7fe924069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:53.529 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.509+0000 7fe928ed2700 1 -- 172.21.15.195:0/785086228 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2604478260 0 0) 0x7fe90c003430 con 0x7fe92406fef0 2022-06-27T12:07:53.529 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.509+0000 7fe929ed4700 1 -- 172.21.15.195:0/785086228 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe92406f8c0 con 0x7fe92406e850 2022-06-27T12:07:53.529 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.509+0000 7fe929ed4700 1 --2- 172.21.15.195:0/785086228 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe92406e850 0x7fe9241570a0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe9140097d0 tx=0x7fe914009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14ca9eb2b1c7cfb9 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:07:53.529 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.509+0000 7fe928ed2700 1 -- 172.21.15.195:0/785086228 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe920001470 con 0x7fe92406e850 2022-06-27T12:07:53.529 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.509+0000 7fe928ed2700 1 -- 172.21.15.195:0/785086228 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe914004030 con 0x7fe92406e850 2022-06-27T12:07:53.530 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.509+0000 7fe928ed2700 1 -- 172.21.15.195:0/785086228 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9140043b0 con 0x7fe92406e850 2022-06-27T12:07:53.530 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.510+0000 7fe92c138700 1 -- 172.21.15.195:0/785086228 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe92406e850 msgr2=0x7fe9241570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:53.530 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.510+0000 7fe92c138700 1 --2- 172.21.15.195:0/785086228 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe92406e850 0x7fe9241570a0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe9140097d0 tx=0x7fe914009060 comp rx=0 tx=0).stop 2022-06-27T12:07:53.531 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.510+0000 7fe92c138700 1 -- 172.21.15.195:0/785086228 shutdown_connections 2022-06-27T12:07:53.531 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.510+0000 7fe92c138700 1 --2- 172.21.15.195:0/785086228 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe92406e850 0x7fe9241570a0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:53.531 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.510+0000 7fe92c138700 1 -- 172.21.15.195:0/785086228 >> 172.21.15.195:0/785086228 conn(0x7fe9240c6350 msgr2=0x7fe9240c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:07:53.531 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.510+0000 7fe92c138700 1 -- 172.21.15.195:0/785086228 shutdown_connections 2022-06-27T12:07:53.532 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.510+0000 7fe92c138700 1 -- 172.21.15.195:0/785086228 wait complete. 2022-06-27T12:07:53.532 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.511+0000 7fe92c138700 1 Processor -- start 2022-06-27T12:07:53.532 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.511+0000 7fe92c138700 1 -- start start 2022-06-27T12:07:53.532 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.511+0000 7fe92c138700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe92406e850 0x7fe924158a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:53.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.512+0000 7fe92c138700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe9240c34a0 0x7fe9241475e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:53.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.512+0000 7fe92c138700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe924159880 0x7fe924147b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:53.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.512+0000 7fe92c138700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9240c80f0 con 0x7fe92406e850 2022-06-27T12:07:53.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.512+0000 7fe92c138700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe924154380 con 0x7fe9240c34a0 2022-06-27T12:07:53.534 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.512+0000 7fe92c138700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9241545f0 con 0x7fe924159880 2022-06-27T12:07:53.534 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.512+0000 7fe929ed4700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe92406e850 0x7fe924158a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:53.535 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.512+0000 7fe929ed4700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe92406e850 0x7fe924158a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48930/0 (socket says 172.21.15.195:48930) 2022-06-27T12:07:53.535 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.512+0000 7fe929ed4700 1 -- 172.21.15.195:0/3228136068 learned_addr learned my addr 172.21.15.195:0/3228136068 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:07:53.536 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.512+0000 7fe9296d3700 1 --2- 172.21.15.195:0/3228136068 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe9240c34a0 0x7fe9241475e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:53.536 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.512+0000 7fe92a6d5700 1 --2- 172.21.15.195:0/3228136068 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe924159880 0x7fe924147b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:53.536 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.512+0000 7fe929ed4700 1 -- 172.21.15.195:0/3228136068 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe924159880 msgr2=0x7fe924147b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:53.536 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.512+0000 7fe929ed4700 1 --2- 172.21.15.195:0/3228136068 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe924159880 0x7fe924147b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:53.537 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.512+0000 7fe929ed4700 1 -- 172.21.15.195:0/3228136068 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe9240c34a0 msgr2=0x7fe9241475e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:53.537 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.512+0000 7fe929ed4700 1 --2- 172.21.15.195:0/3228136068 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe9240c34a0 0x7fe9241475e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:53.537 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.512+0000 7fe929ed4700 1 -- 172.21.15.195:0/3228136068 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9140092e0 con 0x7fe92406e850 2022-06-27T12:07:53.538 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.512+0000 7fe92a6d5700 1 --2- 172.21.15.195:0/3228136068 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe924159880 0x7fe924147b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:07:53.538 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.512+0000 7fe929ed4700 1 --2- 172.21.15.195:0/3228136068 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe92406e850 0x7fe924158a80 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fe914009dc0 tx=0x7fe914015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:07:53.538 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.512+0000 7fe911ffb700 1 -- 172.21.15.195:0/3228136068 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe914011650 con 0x7fe92406e850 2022-06-27T12:07:53.539 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.512+0000 7fe911ffb700 1 -- 172.21.15.195:0/3228136068 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9140117b0 con 0x7fe92406e850 2022-06-27T12:07:53.539 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.512+0000 7fe92c138700 1 -- 172.21.15.195:0/3228136068 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9240c7aa0 con 0x7fe92406e850 2022-06-27T12:07:53.539 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.512+0000 7fe911ffb700 1 -- 172.21.15.195:0/3228136068 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe914011b70 con 0x7fe92406e850 2022-06-27T12:07:53.539 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.513+0000 7fe92c138700 1 -- 172.21.15.195:0/3228136068 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9241548d0 con 0x7fe92406e850 2022-06-27T12:07:53.539 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.514+0000 7fe92c138700 1 -- 172.21.15.195:0/3228136068 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9240c8480 con 0x7fe92406e850 2022-06-27T12:07:53.540 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.514+0000 7fe911ffb700 1 -- 172.21.15.195:0/3228136068 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fe91400f070 con 0x7fe92406e850 2022-06-27T12:07:53.540 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.514+0000 7fe911ffb700 1 --2- 172.21.15.195:0/3228136068 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe904063d30 0x7fe9040661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:53.540 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.514+0000 7fe911ffb700 1 -- 172.21.15.195:0/3228136068 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7148+0+0 (secure 0 0 0) 0x7fe9140824d0 con 0x7fe92406e850 2022-06-27T12:07:53.540 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.514+0000 7fe9296d3700 1 --2- 172.21.15.195:0/3228136068 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe904063d30 0x7fe9040661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:53.541 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.515+0000 7fe9296d3700 1 --2- 172.21.15.195:0/3228136068 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe904063d30 0x7fe9040661e0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fe924069c70 tx=0x7fe920001a50 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:07:53.541 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.519+0000 7fe911ffb700 1 -- 172.21.15.195:0/3228136068 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fe9240c8480 con 0x7fe92406e850 2022-06-27T12:07:53.751 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.750+0000 7fe92c138700 1 -- 172.21.15.195:0/3228136068 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe9240c8480 con 0x7fe92406e850 2022-06-27T12:07:53.751 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.751+0000 7fe911ffb700 1 -- 172.21.15.195:0/3228136068 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+5367 (secure 0 0 0) 0x7fe9240c8480 con 0x7fe92406e850 2022-06-27T12:07:53.752 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:07:53.752 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:07:50.098891+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6271},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6271":{"gid":6271,"name":"d","rank":0,"incarnation":73,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6835/1603307620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":1603307620},{"type":"v1","addr":"172.21.15.202:6835","nonce":1603307620}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:07:53.754 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.752+0000 7fe92c138700 1 -- 172.21.15.195:0/3228136068 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe904063d30 msgr2=0x7fe9040661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:53.754 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.752+0000 7fe92c138700 1 --2- 172.21.15.195:0/3228136068 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe904063d30 0x7fe9040661e0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fe924069c70 tx=0x7fe920001a50 comp rx=0 tx=0).stop 2022-06-27T12:07:53.755 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.752+0000 7fe92c138700 1 -- 172.21.15.195:0/3228136068 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe92406e850 msgr2=0x7fe924158a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:53.755 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.752+0000 7fe92c138700 1 --2- 172.21.15.195:0/3228136068 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe92406e850 0x7fe924158a80 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fe914009dc0 tx=0x7fe914015000 comp rx=0 tx=0).stop 2022-06-27T12:07:53.755 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.753+0000 7fe92c138700 1 -- 172.21.15.195:0/3228136068 shutdown_connections 2022-06-27T12:07:53.756 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.753+0000 7fe92c138700 1 --2- 172.21.15.195:0/3228136068 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe904063d30 0x7fe9040661e0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:53.756 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.753+0000 7fe92c138700 1 --2- 172.21.15.195:0/3228136068 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe92406e850 0x7fe924158a80 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:53.756 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.753+0000 7fe92c138700 1 --2- 172.21.15.195:0/3228136068 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe924159880 0x7fe924147b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:53.756 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.753+0000 7fe92c138700 1 --2- 172.21.15.195:0/3228136068 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe9240c34a0 0x7fe9241475e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:53.757 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.753+0000 7fe92c138700 1 -- 172.21.15.195:0/3228136068 >> 172.21.15.195:0/3228136068 conn(0x7fe9240c6350 msgr2=0x7fe9240681f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:07:53.757 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.753+0000 7fe92c138700 1 -- 172.21.15.195:0/3228136068 shutdown_connections 2022-06-27T12:07:53.757 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:53.753+0000 7fe92c138700 1 -- 172.21.15.195:0/3228136068 wait complete. 2022-06-27T12:07:53.757 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 73 2022-06-27T12:07:53.764 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:07:55.766 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:07:55.904 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:55.903+0000 7f932d210700 1 Processor -- start 2022-06-27T12:07:55.905 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:55.903+0000 7f932d210700 1 -- start start 2022-06-27T12:07:55.905 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:55.904+0000 7f932d210700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f932814e860 0x7f93280c3840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:55.906 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:55.904+0000 7f932d210700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93280c33b0 con 0x7f932814e490 2022-06-27T12:07:55.906 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:55.904+0000 7f932d210700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f932806f220 con 0x7f93280c3d80 2022-06-27T12:07:55.906 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:55.904+0000 7f932d210700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93281542c0 con 0x7f932814e860 2022-06-27T12:07:55.906 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:55.904+0000 7f9326d9d700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f932814e490 0x7f932806cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:55130/0 (socket says 172.21.15.195:55130) 2022-06-27T12:07:55.907 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:55.904+0000 7f9326d9d700 1 -- 172.21.15.195:0/3274561476 learned_addr learned my addr 172.21.15.195:0/3274561476 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:07:55.915 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:55.904+0000 7f932659c700 1 --2- 172.21.15.195:0/3274561476 >> v2:172.21.15.202:3300/0 conn(0x7f932814e860 0x7f93280c3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:55.916 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:55.904+0000 7f9325d9b700 1 -- 172.21.15.195:0/3274561476 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1449515757 0 0) 0x7f93280c33b0 con 0x7f932814e490 2022-06-27T12:07:55.916 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:55.905+0000 7f9325d9b700 1 -- 172.21.15.195:0/3274561476 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9314003430 con 0x7f932814e490 2022-06-27T12:07:55.916 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:55.905+0000 7f9325d9b700 1 -- 172.21.15.195:0/3274561476 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3744333697 0 0) 0x7f932806f220 con 0x7f93280c3d80 2022-06-27T12:07:55.916 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:55.905+0000 7f9325d9b700 1 -- 172.21.15.195:0/3274561476 --> v1:172.21.15.202:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93280c33b0 con 0x7f93280c3d80 2022-06-27T12:07:55.917 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:55.905+0000 7f932659c700 1 -- 172.21.15.195:0/3274561476 >> v1:172.21.15.202:6789/0 conn(0x7f93280c3d80 legacy=0x7f9328157890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:55.918 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:55.905+0000 7f932659c700 1 -- 172.21.15.195:0/3274561476 >> v1:172.21.15.195:6789/0 conn(0x7f932814e490 legacy=0x7f932806cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:55.918 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:55.905+0000 7f932659c700 1 -- 172.21.15.195:0/3274561476 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9328154560 con 0x7f932814e860 2022-06-27T12:07:55.919 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:55.905+0000 7f932659c700 1 --2- 172.21.15.195:0/3274561476 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f932814e860 0x7f93280c3840 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f93180097a0 tx=0x7f9318009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a93e77f5e378645 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:07:55.919 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:55.905+0000 7f9325d9b700 1 -- 172.21.15.195:0/3274561476 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9318011630 con 0x7f932814e860 2022-06-27T12:07:55.919 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:55.905+0000 7f9325d9b700 1 -- 172.21.15.195:0/3274561476 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9318011790 con 0x7f932814e860 2022-06-27T12:07:55.920 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:55.905+0000 7f9325d9b700 1 -- 172.21.15.195:0/3274561476 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9318011b30 con 0x7f932814e860 2022-06-27T12:07:55.921 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:55.905+0000 7f932d210700 1 -- 172.21.15.195:0/3274561476 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f932814e860 msgr2=0x7f93280c3840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:55.921 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:55.905+0000 7f932d210700 1 --2- 172.21.15.195:0/3274561476 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f932814e860 0x7f93280c3840 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f93180097a0 tx=0x7f9318009060 comp rx=0 tx=0).stop 2022-06-27T12:07:55.922 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:55.905+0000 7f932d210700 1 -- 172.21.15.195:0/3274561476 shutdown_connections 2022-06-27T12:07:55.922 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:55.905+0000 7f932d210700 1 --2- 172.21.15.195:0/3274561476 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f932814e860 0x7f93280c3840 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:55.923 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:55.905+0000 7f932d210700 1 -- 172.21.15.195:0/3274561476 >> 172.21.15.195:0/3274561476 conn(0x7f9328063af0 msgr2=0x7f9328063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:07:55.923 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:55.906+0000 7f932d210700 1 -- 172.21.15.195:0/3274561476 shutdown_connections 2022-06-27T12:07:55.923 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:55.906+0000 7f932d210700 1 -- 172.21.15.195:0/3274561476 wait complete. 2022-06-27T12:07:55.924 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:55.907+0000 7f932d210700 1 Processor -- start 2022-06-27T12:07:55.924 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:55.907+0000 7f932d210700 1 -- start start 2022-06-27T12:07:55.924 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:55.908+0000 7f932d210700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f93280c3d80 0x7f9328165af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:55.925 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:55.908+0000 7f932d210700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f932814e490 0x7f9328157180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:55.925 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:55.908+0000 7f932d210700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f932814e860 0x7f9328165060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:55.925 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:55.908+0000 7f932d210700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9328156d60 con 0x7f932814e490 2022-06-27T12:07:55.926 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:55.908+0000 7f932d210700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f932806df40 con 0x7f93280c3d80 2022-06-27T12:07:55.927 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:55.908+0000 7f932d210700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93280c1050 con 0x7f932814e860 2022-06-27T12:07:55.927 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:55.908+0000 7f932659c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f932814e490 0x7f9328157180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:55.928 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:55.908+0000 7f932659c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f932814e490 0x7f9328157180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48950/0 (socket says 172.21.15.195:48950) 2022-06-27T12:07:55.929 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:55.908+0000 7f932659c700 1 -- 172.21.15.195:0/66010244 learned_addr learned my addr 172.21.15.195:0/66010244 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:07:55.929 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:55.908+0000 7f9326d9d700 1 --2- 172.21.15.195:0/66010244 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f93280c3d80 0x7f9328165af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:55.930 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:55.908+0000 7f932759e700 1 --2- 172.21.15.195:0/66010244 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f932814e860 0x7f9328165060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:55.930 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:55.908+0000 7f932659c700 1 -- 172.21.15.195:0/66010244 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f932814e860 msgr2=0x7f9328165060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:55.931 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:55.908+0000 7f932659c700 1 --2- 172.21.15.195:0/66010244 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f932814e860 0x7f9328165060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:55.931 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:55.908+0000 7f932659c700 1 -- 172.21.15.195:0/66010244 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f93280c3d80 msgr2=0x7f9328165af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:55.931 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:55.908+0000 7f932659c700 1 --2- 172.21.15.195:0/66010244 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f93280c3d80 0x7f9328165af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:55.932 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:55.908+0000 7f932659c700 1 -- 172.21.15.195:0/66010244 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93180092e0 con 0x7f932814e490 2022-06-27T12:07:55.932 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:55.908+0000 7f9326d9d700 1 --2- 172.21.15.195:0/66010244 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f93280c3d80 0x7f9328165af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:07:55.933 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:55.908+0000 7f932659c700 1 --2- 172.21.15.195:0/66010244 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f932814e490 0x7f9328157180 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f93180091c0 tx=0x7f9318017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:07:55.933 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:55.908+0000 7f930effd700 1 -- 172.21.15.195:0/66010244 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9318011900 con 0x7f932814e490 2022-06-27T12:07:55.933 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:55.908+0000 7f930effd700 1 -- 172.21.15.195:0/66010244 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9318011a60 con 0x7f932814e490 2022-06-27T12:07:55.934 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:55.908+0000 7f932d210700 1 -- 172.21.15.195:0/66010244 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93280c12f0 con 0x7f932814e490 2022-06-27T12:07:55.934 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:55.908+0000 7f930effd700 1 -- 172.21.15.195:0/66010244 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9318023600 con 0x7f932814e490 2022-06-27T12:07:55.934 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:55.908+0000 7f932d210700 1 -- 172.21.15.195:0/66010244 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93281566a0 con 0x7f932814e490 2022-06-27T12:07:55.935 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:55.909+0000 7f932d210700 1 -- 172.21.15.195:0/66010244 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93280722c0 con 0x7f932814e490 2022-06-27T12:07:55.935 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:55.910+0000 7f930effd700 1 -- 172.21.15.195:0/66010244 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f931800f050 con 0x7f932814e490 2022-06-27T12:07:55.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:55.910+0000 7f930effd700 1 --2- 172.21.15.195:0/66010244 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9308063d00 0x7f93080661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:55.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:55.910+0000 7f930effd700 1 -- 172.21.15.195:0/66010244 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7148+0+0 (secure 0 0 0) 0x7f9318081e20 con 0x7f932814e490 2022-06-27T12:07:55.942 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:55.910+0000 7f9326d9d700 1 --2- 172.21.15.195:0/66010244 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9308063d00 0x7f93080661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:55.944 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:55.910+0000 7f9326d9d700 1 --2- 172.21.15.195:0/66010244 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9308063d00 0x7f93080661b0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f931c005790 tx=0x7f931c009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:07:55.944 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:55.914+0000 7f930effd700 1 -- 172.21.15.195:0/66010244 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f9318050650 con 0x7f932814e490 2022-06-27T12:07:56.144 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:56.143+0000 7f932d210700 1 -- 172.21.15.195:0/66010244 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f93280c60d0 con 0x7f932814e490 2022-06-27T12:07:56.145 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:56.144+0000 7f930effd700 1 -- 172.21.15.195:0/66010244 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+5367 (secure 0 0 0) 0x7f9318015070 con 0x7f932814e490 2022-06-27T12:07:56.147 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:07:56.147 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:07:50.098891+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6271},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6271":{"gid":6271,"name":"d","rank":0,"incarnation":73,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6835/1603307620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":1603307620},{"type":"v1","addr":"172.21.15.202:6835","nonce":1603307620}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:07:56.150 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:56.146+0000 7f932d210700 1 -- 172.21.15.195:0/66010244 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9308063d00 msgr2=0x7f93080661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:56.150 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:56.146+0000 7f932d210700 1 --2- 172.21.15.195:0/66010244 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9308063d00 0x7f93080661b0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f931c005790 tx=0x7f931c009040 comp rx=0 tx=0).stop 2022-06-27T12:07:56.151 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:56.146+0000 7f932d210700 1 -- 172.21.15.195:0/66010244 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f932814e490 msgr2=0x7f9328157180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:56.151 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:56.146+0000 7f932d210700 1 --2- 172.21.15.195:0/66010244 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f932814e490 0x7f9328157180 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f93180091c0 tx=0x7f9318017000 comp rx=0 tx=0).stop 2022-06-27T12:07:56.151 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:56.146+0000 7f932d210700 1 -- 172.21.15.195:0/66010244 shutdown_connections 2022-06-27T12:07:56.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:56.146+0000 7f932d210700 1 --2- 172.21.15.195:0/66010244 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9308063d00 0x7f93080661b0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:56.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:56.146+0000 7f932d210700 1 --2- 172.21.15.195:0/66010244 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f932814e860 0x7f9328165060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:56.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:56.146+0000 7f932d210700 1 --2- 172.21.15.195:0/66010244 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f93280c3d80 0x7f9328165af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:56.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:56.146+0000 7f932d210700 1 --2- 172.21.15.195:0/66010244 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f932814e490 0x7f9328157180 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:56.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:56.146+0000 7f932d210700 1 -- 172.21.15.195:0/66010244 >> 172.21.15.195:0/66010244 conn(0x7f9328063af0 msgr2=0x7f93281528d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:07:56.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:56.146+0000 7f932d210700 1 -- 172.21.15.195:0/66010244 shutdown_connections 2022-06-27T12:07:56.155 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:56.146+0000 7f932d210700 1 -- 172.21.15.195:0/66010244 wait complete. 2022-06-27T12:07:56.155 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 73 2022-06-27T12:07:56.161 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:07:58.162 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:07:58.300 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.299+0000 7f87acb39700 1 Processor -- start 2022-06-27T12:07:58.301 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.299+0000 7f87acb39700 1 -- start start 2022-06-27T12:07:58.310 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.300+0000 7f87acb39700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f87a8151560 0x7f87a80c2120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:58.311 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.300+0000 7f87acb39700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f87a8154900 0x7f87a80bf4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:58.311 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.300+0000 7f87acb39700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87a8154da0 con 0x7f87a8151930 2022-06-27T12:07:58.312 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.300+0000 7f87acb39700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87a80c12a0 con 0x7f87a8154900 2022-06-27T12:07:58.312 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.300+0000 7f87acb39700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87a80c1580 con 0x7f87a8151560 2022-06-27T12:07:58.312 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.301+0000 7f87a6d9d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f87a8154900 0x7f87a80bf4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:58.313 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.301+0000 7f87a6d9d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f87a8154900 0x7f87a80bf4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48964/0 (socket says 172.21.15.195:48964) 2022-06-27T12:07:58.313 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.301+0000 7f87a5d9b700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f87a8151930 0x7f87a81541f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:55148/0 (socket says 172.21.15.195:55148) 2022-06-27T12:07:58.314 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.301+0000 7f87a6d9d700 1 -- 172.21.15.195:0/2022399127 learned_addr learned my addr 172.21.15.195:0/2022399127 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:07:58.314 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.301+0000 7f87a659c700 1 --2- 172.21.15.195:0/2022399127 >> v2:172.21.15.202:3300/0 conn(0x7f87a8151560 0x7f87a80c2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:58.314 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.301+0000 7f87a6d9d700 1 -- 172.21.15.195:0/2022399127 >> v2:172.21.15.202:3300/0 conn(0x7f87a8151560 msgr2=0x7f87a80c2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:07:58.315 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.301+0000 7f87a6d9d700 1 --2- 172.21.15.195:0/2022399127 >> v2:172.21.15.202:3300/0 conn(0x7f87a8151560 0x7f87a80c2120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:58.315 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.301+0000 7f87a6d9d700 1 -- 172.21.15.195:0/2022399127 >> v1:172.21.15.195:6789/0 conn(0x7f87a8151930 legacy=0x7f87a81541f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:58.316 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.301+0000 7f87a6d9d700 1 -- 172.21.15.195:0/2022399127 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87a80c29f0 con 0x7f87a8154900 2022-06-27T12:07:58.316 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.301+0000 7f87a559a700 1 -- 172.21.15.195:0/2022399127 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3924143857 0 0) 0x7f87a8154da0 con 0x7f87a8151930 2022-06-27T12:07:58.316 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.301+0000 7f87a6d9d700 1 --2- 172.21.15.195:0/2022399127 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f87a8154900 0x7f87a80bf4a0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f87a806e210 tx=0x7f87a0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb6dafd6aeb35b21 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:07:58.317 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.301+0000 7f87a559a700 1 -- 172.21.15.195:0/2022399127 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a000e070 con 0x7f87a8154900 2022-06-27T12:07:58.317 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.301+0000 7f87a559a700 1 -- 172.21.15.195:0/2022399127 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87a000bab0 con 0x7f87a8154900 2022-06-27T12:07:58.317 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.301+0000 7f87a559a700 1 -- 172.21.15.195:0/2022399127 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a00092e0 con 0x7f87a8154900 2022-06-27T12:07:58.317 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.302+0000 7f87acb39700 1 -- 172.21.15.195:0/2022399127 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f87a8154900 msgr2=0x7f87a80bf4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:58.318 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.302+0000 7f87acb39700 1 --2- 172.21.15.195:0/2022399127 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f87a8154900 0x7f87a80bf4a0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f87a806e210 tx=0x7f87a0009840 comp rx=0 tx=0).stop 2022-06-27T12:07:58.318 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.302+0000 7f87acb39700 1 -- 172.21.15.195:0/2022399127 shutdown_connections 2022-06-27T12:07:58.318 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.302+0000 7f87acb39700 1 --2- 172.21.15.195:0/2022399127 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f87a8154900 0x7f87a80bf4a0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:58.318 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.302+0000 7f87acb39700 1 --2- 172.21.15.195:0/2022399127 >> v2:172.21.15.202:3300/0 conn(0x7f87a8151560 0x7f87a80c2120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:58.319 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.302+0000 7f87acb39700 1 -- 172.21.15.195:0/2022399127 >> 172.21.15.195:0/2022399127 conn(0x7f87a80c6350 msgr2=0x7f87a80c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:07:58.319 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.302+0000 7f87acb39700 1 -- 172.21.15.195:0/2022399127 shutdown_connections 2022-06-27T12:07:58.319 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.302+0000 7f87acb39700 1 -- 172.21.15.195:0/2022399127 wait complete. 2022-06-27T12:07:58.319 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.303+0000 7f87acb39700 1 Processor -- start 2022-06-27T12:07:58.320 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.303+0000 7f87acb39700 1 -- start start 2022-06-27T12:07:58.320 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.304+0000 7f87acb39700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f87a8151560 0x7f87a8157d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:58.320 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.304+0000 7f87acb39700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f87a8151930 0x7f87a80c6900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:58.320 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.304+0000 7f87acb39700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f87a8154900 0x7f87a8155720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:58.320 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.304+0000 7f87acb39700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87a8069940 con 0x7f87a8151560 2022-06-27T12:07:58.321 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.304+0000 7f87acb39700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87a806a1e0 con 0x7f87a8151930 2022-06-27T12:07:58.321 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.304+0000 7f87acb39700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87a8164f50 con 0x7f87a8154900 2022-06-27T12:07:58.321 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.304+0000 7f87a659c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f87a8151560 0x7f87a8157d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:58.321 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.304+0000 7f87a659c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f87a8151560 0x7f87a8157d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48966/0 (socket says 172.21.15.195:48966) 2022-06-27T12:07:58.322 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.304+0000 7f87a659c700 1 -- 172.21.15.195:0/1496738028 learned_addr learned my addr 172.21.15.195:0/1496738028 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:07:58.322 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.304+0000 7f87a5d9b700 1 --2- 172.21.15.195:0/1496738028 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f87a8151930 0x7f87a80c6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:58.322 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.304+0000 7f87a6d9d700 1 --2- 172.21.15.195:0/1496738028 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f87a8154900 0x7f87a8155720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:58.322 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.304+0000 7f87a659c700 1 -- 172.21.15.195:0/1496738028 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f87a8154900 msgr2=0x7f87a8155720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:58.323 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.304+0000 7f87a659c700 1 --2- 172.21.15.195:0/1496738028 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f87a8154900 0x7f87a8155720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:58.323 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.304+0000 7f87a659c700 1 -- 172.21.15.195:0/1496738028 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f87a8151930 msgr2=0x7f87a80c6900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:58.323 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.304+0000 7f87a659c700 1 --2- 172.21.15.195:0/1496738028 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f87a8151930 0x7f87a80c6900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:58.323 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.304+0000 7f87a659c700 1 -- 172.21.15.195:0/1496738028 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87a0009ae0 con 0x7f87a8151560 2022-06-27T12:07:58.324 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.304+0000 7f87a659c700 1 --2- 172.21.15.195:0/1496738028 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f87a8151560 0x7f87a8157d90 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f879800e970 tx=0x7f8798009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:07:58.324 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.304+0000 7f879e7fc700 1 -- 172.21.15.195:0/1496738028 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8798013680 con 0x7f87a8151560 2022-06-27T12:07:58.324 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.304+0000 7f879e7fc700 1 -- 172.21.15.195:0/1496738028 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87980137e0 con 0x7f87a8151560 2022-06-27T12:07:58.324 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.304+0000 7f87acb39700 1 -- 172.21.15.195:0/1496738028 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87a81585d0 con 0x7f87a8151560 2022-06-27T12:07:58.325 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.304+0000 7f879e7fc700 1 -- 172.21.15.195:0/1496738028 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8798013b60 con 0x7f87a8151560 2022-06-27T12:07:58.325 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.304+0000 7f87acb39700 1 -- 172.21.15.195:0/1496738028 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87a8164620 con 0x7f87a8151560 2022-06-27T12:07:58.325 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.305+0000 7f87acb39700 1 -- 172.21.15.195:0/1496738028 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87a80722c0 con 0x7f87a8151560 2022-06-27T12:07:58.326 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.306+0000 7f879e7fc700 1 -- 172.21.15.195:0/1496738028 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f8798023070 con 0x7f87a8151560 2022-06-27T12:07:58.326 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.306+0000 7f879e7fc700 1 --2- 172.21.15.195:0/1496738028 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f878c063d00 0x7f878c0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:07:58.326 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.306+0000 7f879e7fc700 1 -- 172.21.15.195:0/1496738028 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7148+0+0 (secure 0 0 0) 0x7f8798080f90 con 0x7f87a8151560 2022-06-27T12:07:58.326 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.306+0000 7f87a5d9b700 1 --2- 172.21.15.195:0/1496738028 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f878c063d00 0x7f878c0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:07:58.327 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.306+0000 7f87a5d9b700 1 --2- 172.21.15.195:0/1496738028 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f878c063d00 0x7f878c0661b0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f87a8154f80 tx=0x7f8790007bf0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:07:58.327 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.310+0000 7f879e7fc700 1 -- 172.21.15.195:0/1496738028 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f879804f6e0 con 0x7f87a8151560 2022-06-27T12:07:58.538 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.538+0000 7f87acb39700 1 -- 172.21.15.195:0/1496738028 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87a80c1970 con 0x7f87a8151560 2022-06-27T12:07:58.539 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.538+0000 7f879e7fc700 1 -- 172.21.15.195:0/1496738028 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+5367 (secure 0 0 0) 0x7f8798017090 con 0x7f87a8151560 2022-06-27T12:07:58.539 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:07:58.539 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:07:50.098891+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6271},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6271":{"gid":6271,"name":"d","rank":0,"incarnation":73,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6835/1603307620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":1603307620},{"type":"v1","addr":"172.21.15.202:6835","nonce":1603307620}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:07:58.541 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.540+0000 7f87acb39700 1 -- 172.21.15.195:0/1496738028 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f878c063d00 msgr2=0x7f878c0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:58.542 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.540+0000 7f87acb39700 1 --2- 172.21.15.195:0/1496738028 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f878c063d00 0x7f878c0661b0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f87a8154f80 tx=0x7f8790007bf0 comp rx=0 tx=0).stop 2022-06-27T12:07:58.542 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.540+0000 7f87acb39700 1 -- 172.21.15.195:0/1496738028 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f87a8151560 msgr2=0x7f87a8157d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:07:58.542 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.540+0000 7f87acb39700 1 --2- 172.21.15.195:0/1496738028 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f87a8151560 0x7f87a8157d90 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f879800e970 tx=0x7f8798009060 comp rx=0 tx=0).stop 2022-06-27T12:07:58.542 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.540+0000 7f87acb39700 1 -- 172.21.15.195:0/1496738028 shutdown_connections 2022-06-27T12:07:58.543 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.540+0000 7f87acb39700 1 --2- 172.21.15.195:0/1496738028 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f878c063d00 0x7f878c0661b0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:58.543 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.540+0000 7f87acb39700 1 --2- 172.21.15.195:0/1496738028 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f87a8151560 0x7f87a8157d90 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:58.543 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.540+0000 7f87acb39700 1 --2- 172.21.15.195:0/1496738028 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f87a8154900 0x7f87a8155720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:58.544 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.540+0000 7f87acb39700 1 --2- 172.21.15.195:0/1496738028 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f87a8151930 0x7f87a80c6900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:07:58.544 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.540+0000 7f87acb39700 1 -- 172.21.15.195:0/1496738028 >> 172.21.15.195:0/1496738028 conn(0x7f87a80c6350 msgr2=0x7f87a80c3330 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:07:58.544 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.541+0000 7f87acb39700 1 -- 172.21.15.195:0/1496738028 shutdown_connections 2022-06-27T12:07:58.544 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:07:58.541+0000 7f87acb39700 1 -- 172.21.15.195:0/1496738028 wait complete. 2022-06-27T12:07:58.545 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 73 2022-06-27T12:07:58.553 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:08:00.554 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:08:00.691 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.691+0000 7faa3f816700 1 Processor -- start 2022-06-27T12:08:00.692 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.691+0000 7faa3f816700 1 -- start start 2022-06-27T12:08:00.698 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.692+0000 7faa3f816700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7faa380c2bf0 0x7faa380c2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:00.699 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.692+0000 7faa3f816700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faa380c5000 0x7faa380c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:00.699 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.692+0000 7faa3f816700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa380bc490 con 0x7faa380c61e0 2022-06-27T12:08:00.700 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.692+0000 7faa3f816700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa3806a0e0 con 0x7faa380c2bf0 2022-06-27T12:08:00.700 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.692+0000 7faa3f816700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa380bc6e0 con 0x7faa380c5000 2022-06-27T12:08:00.700 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.692+0000 7faa3d5b2700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7faa380c2bf0 0x7faa380c2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:00.700 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.692+0000 7faa3d5b2700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7faa380c2bf0 0x7faa380c2fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48976/0 (socket says 172.21.15.195:48976) 2022-06-27T12:08:00.701 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.692+0000 7faa3d5b2700 1 -- 172.21.15.195:0/3870660963 learned_addr learned my addr 172.21.15.195:0/3870660963 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:08:00.702 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.692+0000 7faa3ddb3700 1 --2- 172.21.15.195:0/3870660963 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faa380c5000 0x7faa380c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:00.702 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.692+0000 7faa3d5b2700 1 -- 172.21.15.195:0/3870660963 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faa380c5000 msgr2=0x7faa380c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:08:00.703 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.692+0000 7faa3d5b2700 1 --2- 172.21.15.195:0/3870660963 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faa380c5000 0x7faa380c3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:00.703 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.692+0000 7faa3d5b2700 1 -- 172.21.15.195:0/3870660963 >> v1:172.21.15.202:6789/0 conn(0x7faa380c61e0 legacy=0x7faa380c48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:00.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.692+0000 7faa3d5b2700 1 -- 172.21.15.195:0/3870660963 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa38158220 con 0x7faa380c2bf0 2022-06-27T12:08:00.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.692+0000 7faa3d5b2700 1 --2- 172.21.15.195:0/3870660963 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faa380c2bf0 0x7faa380c2fc0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7faa34006fd0 tx=0x7faa340090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d91a95355c16f17 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:08:00.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.693+0000 7faa27fff700 1 -- 172.21.15.195:0/3870660963 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa34005ad0 con 0x7faa380c2bf0 2022-06-27T12:08:00.705 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.693+0000 7faa27fff700 1 -- 172.21.15.195:0/3870660963 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa34005c30 con 0x7faa380c2bf0 2022-06-27T12:08:00.705 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.693+0000 7faa27fff700 1 -- 172.21.15.195:0/3870660963 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa34004030 con 0x7faa380c2bf0 2022-06-27T12:08:00.707 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.693+0000 7faa3f816700 1 -- 172.21.15.195:0/3870660963 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faa380c2bf0 msgr2=0x7faa380c2fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:00.707 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.693+0000 7faa3f816700 1 --2- 172.21.15.195:0/3870660963 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faa380c2bf0 0x7faa380c2fc0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7faa34006fd0 tx=0x7faa340090f0 comp rx=0 tx=0).stop 2022-06-27T12:08:00.708 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.693+0000 7faa3f816700 1 -- 172.21.15.195:0/3870660963 shutdown_connections 2022-06-27T12:08:00.708 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.693+0000 7faa3f816700 1 --2- 172.21.15.195:0/3870660963 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faa380c5000 0x7faa380c3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:00.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.693+0000 7faa3f816700 1 --2- 172.21.15.195:0/3870660963 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faa380c2bf0 0x7faa380c2fc0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:00.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.693+0000 7faa3f816700 1 -- 172.21.15.195:0/3870660963 >> 172.21.15.195:0/3870660963 conn(0x7faa380c9b00 msgr2=0x7faa380c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:08:00.710 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.693+0000 7faa3f816700 1 -- 172.21.15.195:0/3870660963 shutdown_connections 2022-06-27T12:08:00.710 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.693+0000 7faa3f816700 1 -- 172.21.15.195:0/3870660963 wait complete. 2022-06-27T12:08:00.710 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.694+0000 7faa3f816700 1 Processor -- start 2022-06-27T12:08:00.711 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.694+0000 7faa3f816700 1 -- start start 2022-06-27T12:08:00.712 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.695+0000 7faa3f816700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faa380c2bf0 0x7faa380ba8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:00.712 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.695+0000 7faa3f816700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faa380c5000 0x7faa380badf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:00.713 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.695+0000 7faa3f816700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7faa380c61e0 0x7faa38156000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:00.714 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.695+0000 7faa3f816700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa381596a0 con 0x7faa380c2bf0 2022-06-27T12:08:00.714 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.695+0000 7faa3f816700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa380ba460 con 0x7faa380c61e0 2022-06-27T12:08:00.715 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.695+0000 7faa3f816700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa38155770 con 0x7faa380c5000 2022-06-27T12:08:00.715 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.695+0000 7faa3d5b2700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faa380c2bf0 0x7faa380ba8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:00.716 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.695+0000 7faa3d5b2700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faa380c2bf0 0x7faa380ba8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48982/0 (socket says 172.21.15.195:48982) 2022-06-27T12:08:00.716 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.695+0000 7faa3d5b2700 1 -- 172.21.15.195:0/3696405099 learned_addr learned my addr 172.21.15.195:0/3696405099 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:08:00.717 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.695+0000 7faa3ddb3700 1 --2- 172.21.15.195:0/3696405099 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7faa380c61e0 0x7faa38156000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:00.717 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.695+0000 7faa3cdb1700 1 --2- 172.21.15.195:0/3696405099 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faa380c5000 0x7faa380badf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:00.717 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.696+0000 7faa3d5b2700 1 -- 172.21.15.195:0/3696405099 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faa380c5000 msgr2=0x7faa380badf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:00.718 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.696+0000 7faa3d5b2700 1 --2- 172.21.15.195:0/3696405099 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faa380c5000 0x7faa380badf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:00.718 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.696+0000 7faa3d5b2700 1 -- 172.21.15.195:0/3696405099 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7faa380c61e0 msgr2=0x7faa38156000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:00.719 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.696+0000 7faa3d5b2700 1 --2- 172.21.15.195:0/3696405099 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7faa380c61e0 0x7faa38156000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:00.719 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.696+0000 7faa3d5b2700 1 -- 172.21.15.195:0/3696405099 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa34009350 con 0x7faa380c2bf0 2022-06-27T12:08:00.720 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.696+0000 7faa3d5b2700 1 --2- 172.21.15.195:0/3696405099 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faa380c2bf0 0x7faa380ba8b0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7faa3806f810 tx=0x7faa34015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:08:00.720 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.696+0000 7faa257fa700 1 -- 172.21.15.195:0/3696405099 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa34004560 con 0x7faa380c2bf0 2022-06-27T12:08:00.721 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.696+0000 7faa257fa700 1 -- 172.21.15.195:0/3696405099 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa34003520 con 0x7faa380c2bf0 2022-06-27T12:08:00.721 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.696+0000 7faa3f816700 1 -- 172.21.15.195:0/3696405099 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa38159890 con 0x7faa380c2bf0 2022-06-27T12:08:00.721 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.696+0000 7faa257fa700 1 -- 172.21.15.195:0/3696405099 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa340036d0 con 0x7faa380c2bf0 2022-06-27T12:08:00.722 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.696+0000 7faa3f816700 1 -- 172.21.15.195:0/3696405099 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa380c5d50 con 0x7faa380c2bf0 2022-06-27T12:08:00.722 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.697+0000 7faa3f816700 1 -- 172.21.15.195:0/3696405099 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa3815a140 con 0x7faa380c2bf0 2022-06-27T12:08:00.723 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.697+0000 7faa257fa700 1 -- 172.21.15.195:0/3696405099 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7faa3402e070 con 0x7faa380c2bf0 2022-06-27T12:08:00.723 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.698+0000 7faa257fa700 1 --2- 172.21.15.195:0/3696405099 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7faa18063d30 0x7faa180661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:00.724 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.698+0000 7faa257fa700 1 -- 172.21.15.195:0/3696405099 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7148+0+0 (secure 0 0 0) 0x7faa34082910 con 0x7faa380c2bf0 2022-06-27T12:08:00.724 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.698+0000 7faa3cdb1700 1 --2- 172.21.15.195:0/3696405099 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7faa18063d30 0x7faa180661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:00.725 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.698+0000 7faa3cdb1700 1 --2- 172.21.15.195:0/3696405099 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7faa18063d30 0x7faa180661e0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7faa38159e60 tx=0x7faa28009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:08:00.726 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.702+0000 7faa257fa700 1 -- 172.21.15.195:0/3696405099 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7faa3815a140 con 0x7faa380c2bf0 2022-06-27T12:08:00.927 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.927+0000 7faa3f816700 1 -- 172.21.15.195:0/3696405099 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faa3815a140 con 0x7faa380c2bf0 2022-06-27T12:08:00.928 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.928+0000 7faa257fa700 1 -- 172.21.15.195:0/3696405099 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+5367 (secure 0 0 0) 0x7faa3815a140 con 0x7faa380c2bf0 2022-06-27T12:08:00.928 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:08:00.929 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:07:50.098891+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6271},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6271":{"gid":6271,"name":"d","rank":0,"incarnation":73,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6835/1603307620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":1603307620},{"type":"v1","addr":"172.21.15.202:6835","nonce":1603307620}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:08:00.934 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.929+0000 7faa3f816700 1 -- 172.21.15.195:0/3696405099 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7faa18063d30 msgr2=0x7faa180661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:00.934 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.929+0000 7faa3f816700 1 --2- 172.21.15.195:0/3696405099 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7faa18063d30 0x7faa180661e0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7faa38159e60 tx=0x7faa28009040 comp rx=0 tx=0).stop 2022-06-27T12:08:00.935 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.929+0000 7faa3f816700 1 -- 172.21.15.195:0/3696405099 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faa380c2bf0 msgr2=0x7faa380ba8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:00.935 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.929+0000 7faa3f816700 1 --2- 172.21.15.195:0/3696405099 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faa380c2bf0 0x7faa380ba8b0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7faa3806f810 tx=0x7faa34015000 comp rx=0 tx=0).stop 2022-06-27T12:08:00.935 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.930+0000 7faa3f816700 1 -- 172.21.15.195:0/3696405099 shutdown_connections 2022-06-27T12:08:00.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.930+0000 7faa3f816700 1 --2- 172.21.15.195:0/3696405099 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7faa18063d30 0x7faa180661e0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:00.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.930+0000 7faa3f816700 1 --2- 172.21.15.195:0/3696405099 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faa380c2bf0 0x7faa380ba8b0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:00.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.930+0000 7faa3f816700 1 --2- 172.21.15.195:0/3696405099 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7faa380c61e0 0x7faa38156000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:00.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.930+0000 7faa3f816700 1 --2- 172.21.15.195:0/3696405099 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faa380c5000 0x7faa380badf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:00.937 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.930+0000 7faa3f816700 1 -- 172.21.15.195:0/3696405099 >> 172.21.15.195:0/3696405099 conn(0x7faa380c9b00 msgr2=0x7faa3814da50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:08:00.937 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.930+0000 7faa3f816700 1 -- 172.21.15.195:0/3696405099 shutdown_connections 2022-06-27T12:08:00.937 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:00.930+0000 7faa3f816700 1 -- 172.21.15.195:0/3696405099 wait complete. 2022-06-27T12:08:00.937 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 73 2022-06-27T12:08:00.944 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:08:01.799 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:08:01.803 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:08:02.945 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:08:03.090 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.076+0000 7f6672f07700 1 Processor -- start 2022-06-27T12:08:03.092 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.076+0000 7f6672f07700 1 -- start start 2022-06-27T12:08:03.092 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.077+0000 7f6672f07700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f666c150500 0x7f666c063af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:03.093 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.077+0000 7f6672f07700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f666c064030 con 0x7f666c1508d0 2022-06-27T12:08:03.093 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.077+0000 7f6672f07700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f666c0c4160 con 0x7f666c06a450 2022-06-27T12:08:03.094 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.077+0000 7f6672f07700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f666c0c4440 con 0x7f666c150500 2022-06-27T12:08:03.094 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.077+0000 7f66714a4700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f666c1508d0 0x7f666c0bbf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:55180/0 (socket says 172.21.15.195:55180) 2022-06-27T12:08:03.094 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.077+0000 7f66714a4700 1 -- 172.21.15.195:0/2173021182 learned_addr learned my addr 172.21.15.195:0/2173021182 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:08:03.096 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.078+0000 7f6670ca3700 1 --2- 172.21.15.195:0/2173021182 >> v2:172.21.15.202:3300/0 conn(0x7f666c150500 0x7f666c063af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:03.097 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.078+0000 7f666b7fe700 1 -- 172.21.15.195:0/2173021182 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 770113561 0 0) 0x7f6664001490 con 0x7f666c1508d0 2022-06-27T12:08:03.097 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.078+0000 7f666b7fe700 1 -- 172.21.15.195:0/2173021182 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6650003430 con 0x7f666c1508d0 2022-06-27T12:08:03.098 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.078+0000 7f6670ca3700 1 -- 172.21.15.195:0/2173021182 >> v1:172.21.15.202:6789/0 conn(0x7f666c06a450 legacy=0x7f666c0c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:03.098 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.078+0000 7f6670ca3700 1 -- 172.21.15.195:0/2173021182 >> v1:172.21.15.195:6789/0 conn(0x7f666c1508d0 legacy=0x7f666c0bbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:03.099 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.078+0000 7f6670ca3700 1 -- 172.21.15.195:0/2173021182 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f666c15a800 con 0x7f666c150500 2022-06-27T12:08:03.099 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.078+0000 7f6670ca3700 1 --2- 172.21.15.195:0/2173021182 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f666c150500 0x7f666c063af0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f6660009720 tx=0x7f6660009a30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a5cbb2ef299820e server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:08:03.100 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.078+0000 7f666b7fe700 1 -- 172.21.15.195:0/2173021182 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6660005ad0 con 0x7f666c150500 2022-06-27T12:08:03.109 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.078+0000 7f666b7fe700 1 -- 172.21.15.195:0/2173021182 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6660005c30 con 0x7f666c150500 2022-06-27T12:08:03.110 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.078+0000 7f666b7fe700 1 -- 172.21.15.195:0/2173021182 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6660006030 con 0x7f666c150500 2022-06-27T12:08:03.110 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.079+0000 7f6672f07700 1 -- 172.21.15.195:0/2173021182 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f666c150500 msgr2=0x7f666c063af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:03.111 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.079+0000 7f6672f07700 1 --2- 172.21.15.195:0/2173021182 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f666c150500 0x7f666c063af0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f6660009720 tx=0x7f6660009a30 comp rx=0 tx=0).stop 2022-06-27T12:08:03.111 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.079+0000 7f6672f07700 1 -- 172.21.15.195:0/2173021182 shutdown_connections 2022-06-27T12:08:03.112 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.079+0000 7f6672f07700 1 --2- 172.21.15.195:0/2173021182 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f666c150500 0x7f666c063af0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:03.112 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.079+0000 7f6672f07700 1 -- 172.21.15.195:0/2173021182 >> 172.21.15.195:0/2173021182 conn(0x7f666c0c9b00 msgr2=0x7f666c0c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:08:03.113 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.079+0000 7f6672f07700 1 -- 172.21.15.195:0/2173021182 shutdown_connections 2022-06-27T12:08:03.113 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.079+0000 7f6672f07700 1 -- 172.21.15.195:0/2173021182 wait complete. 2022-06-27T12:08:03.114 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.080+0000 7f6672f07700 1 Processor -- start 2022-06-27T12:08:03.114 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.080+0000 7f6672f07700 1 -- start start 2022-06-27T12:08:03.115 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.081+0000 7f6672f07700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f666c06a450 0x7f666c14dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:03.115 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.081+0000 7f6672f07700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f666c150500 0x7f666c166450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:03.115 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.081+0000 7f6672f07700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f666c1508d0 0x7f666c166990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:03.116 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.081+0000 7f6672f07700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f666c0bdd40 con 0x7f666c06a450 2022-06-27T12:08:03.118 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.081+0000 7f6672f07700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f666c0c25a0 con 0x7f666c150500 2022-06-27T12:08:03.118 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.081+0000 7f6672f07700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f666c0bd2a0 con 0x7f666c1508d0 2022-06-27T12:08:03.118 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.081+0000 7f6670ca3700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f666c06a450 0x7f666c14dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:03.119 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.081+0000 7f6670ca3700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f666c06a450 0x7f666c14dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:48996/0 (socket says 172.21.15.195:48996) 2022-06-27T12:08:03.119 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.081+0000 7f6670ca3700 1 -- 172.21.15.195:0/2255036923 learned_addr learned my addr 172.21.15.195:0/2255036923 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:08:03.119 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.081+0000 7f666bfff700 1 --2- 172.21.15.195:0/2255036923 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f666c150500 0x7f666c166450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:03.119 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.081+0000 7f66714a4700 1 --2- 172.21.15.195:0/2255036923 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f666c1508d0 0x7f666c166990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:03.120 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.081+0000 7f6670ca3700 1 -- 172.21.15.195:0/2255036923 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f666c1508d0 msgr2=0x7f666c166990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:03.120 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.081+0000 7f6670ca3700 1 --2- 172.21.15.195:0/2255036923 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f666c1508d0 0x7f666c166990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:03.120 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.081+0000 7f6670ca3700 1 -- 172.21.15.195:0/2255036923 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f666c150500 msgr2=0x7f666c166450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:03.120 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.081+0000 7f6670ca3700 1 --2- 172.21.15.195:0/2255036923 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f666c150500 0x7f666c166450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:03.120 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.081+0000 7f6670ca3700 1 -- 172.21.15.195:0/2255036923 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6660009290 con 0x7f666c06a450 2022-06-27T12:08:03.121 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.081+0000 7f6670ca3700 1 --2- 172.21.15.195:0/2255036923 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f666c06a450 0x7f666c14dd40 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f6660009e30 tx=0x7f6660016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:08:03.123 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.081+0000 7f6668ff9700 1 -- 172.21.15.195:0/2255036923 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f666001a630 con 0x7f666c06a450 2022-06-27T12:08:03.124 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.081+0000 7f6668ff9700 1 -- 172.21.15.195:0/2255036923 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f666001a790 con 0x7f666c06a450 2022-06-27T12:08:03.124 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.081+0000 7f6672f07700 1 -- 172.21.15.195:0/2255036923 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f666c0c2110 con 0x7f666c06a450 2022-06-27T12:08:03.124 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.082+0000 7f6668ff9700 1 -- 172.21.15.195:0/2255036923 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f666001ab50 con 0x7f666c06a450 2022-06-27T12:08:03.125 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.082+0000 7f6672f07700 1 -- 172.21.15.195:0/2255036923 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f666c06f010 con 0x7f666c06a450 2022-06-27T12:08:03.125 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.083+0000 7f6672f07700 1 -- 172.21.15.195:0/2255036923 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f666c0c23f0 con 0x7f666c06a450 2022-06-27T12:08:03.126 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.083+0000 7f6668ff9700 1 -- 172.21.15.195:0/2255036923 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f666002d070 con 0x7f666c06a450 2022-06-27T12:08:03.126 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.083+0000 7f6668ff9700 1 --2- 172.21.15.195:0/2255036923 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6654063d00 0x7f66540661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:03.130 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.083+0000 7f6668ff9700 1 -- 172.21.15.195:0/2255036923 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7148+0+0 (secure 0 0 0) 0x7f6660082110 con 0x7f666c06a450 2022-06-27T12:08:03.131 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.083+0000 7f666bfff700 1 --2- 172.21.15.195:0/2255036923 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6654063d00 0x7f66540661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:03.131 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.084+0000 7f666bfff700 1 --2- 172.21.15.195:0/2255036923 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6654063d00 0x7f66540661b0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f666c157450 tx=0x7f6658009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:08:03.132 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.087+0000 7f6668ff9700 1 -- 172.21.15.195:0/2255036923 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f66600508c0 con 0x7f666c06a450 2022-06-27T12:08:03.315 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.308+0000 7f6672f07700 1 -- 172.21.15.195:0/2255036923 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f666c0722c0 con 0x7f666c06a450 2022-06-27T12:08:03.316 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.308+0000 7f6668ff9700 1 -- 172.21.15.195:0/2255036923 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+5367 (secure 0 0 0) 0x7f666002b090 con 0x7f666c06a450 2022-06-27T12:08:03.317 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:08:03.317 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:07:50.098891+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6271},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6271":{"gid":6271,"name":"d","rank":0,"incarnation":73,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6835/1603307620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":1603307620},{"type":"v1","addr":"172.21.15.202:6835","nonce":1603307620}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:08:03.319 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.310+0000 7f6672f07700 1 -- 172.21.15.195:0/2255036923 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6654063d00 msgr2=0x7f66540661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:03.319 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.310+0000 7f6672f07700 1 --2- 172.21.15.195:0/2255036923 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6654063d00 0x7f66540661b0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f666c157450 tx=0x7f6658009040 comp rx=0 tx=0).stop 2022-06-27T12:08:03.320 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.310+0000 7f6672f07700 1 -- 172.21.15.195:0/2255036923 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f666c06a450 msgr2=0x7f666c14dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:03.321 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.310+0000 7f6672f07700 1 --2- 172.21.15.195:0/2255036923 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f666c06a450 0x7f666c14dd40 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f6660009e30 tx=0x7f6660016000 comp rx=0 tx=0).stop 2022-06-27T12:08:03.322 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.310+0000 7f6672f07700 1 -- 172.21.15.195:0/2255036923 shutdown_connections 2022-06-27T12:08:03.322 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.310+0000 7f6672f07700 1 --2- 172.21.15.195:0/2255036923 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6654063d00 0x7f66540661b0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:03.323 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.310+0000 7f6672f07700 1 --2- 172.21.15.195:0/2255036923 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f666c06a450 0x7f666c14dd40 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:03.323 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.310+0000 7f6672f07700 1 --2- 172.21.15.195:0/2255036923 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f666c1508d0 0x7f666c166990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:03.324 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.310+0000 7f6672f07700 1 --2- 172.21.15.195:0/2255036923 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f666c150500 0x7f666c166450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:03.324 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.310+0000 7f6672f07700 1 -- 172.21.15.195:0/2255036923 >> 172.21.15.195:0/2255036923 conn(0x7f666c0c9b00 msgr2=0x7f666c14fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:08:03.327 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.310+0000 7f6672f07700 1 -- 172.21.15.195:0/2255036923 shutdown_connections 2022-06-27T12:08:03.327 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:03.310+0000 7f6672f07700 1 -- 172.21.15.195:0/2255036923 wait complete. 2022-06-27T12:08:03.329 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 73 2022-06-27T12:08:03.332 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 73, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-06-27T11:47:52.358095+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 73, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6271": { "addr": "172.21.15.202:6835/1603307620", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6834", "nonce": 1603307620, "type": "v2" }, { "addr": "172.21.15.202:6835", "nonce": 1603307620, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 6271, "incarnation": 73, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 59, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-06-27T12:07:50.098891+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6271 } } } ], "standbys": [ { "addr": "172.21.15.202:6837/3305006841", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6836", "nonce": 3305006841, "type": "v2" }, { "addr": "172.21.15.202:6837", "nonce": 3305006841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6835/2060709822", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6834", "nonce": 2060709822, "type": "v2" }, { "addr": "172.21.15.195:6835", "nonce": 2060709822, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6838/1598751652", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6836", "nonce": 1598751652, "type": "v2" }, { "addr": "172.21.15.195:6838", "nonce": 1598751652, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4275, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6839/1313800661", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6837", "nonce": 1313800661, "type": "v2" }, { "addr": "172.21.15.195:6839", "nonce": 1313800661, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4278, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-06-27T12:08:05.334 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:08:05.493 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.468+0000 7fad5b175700 1 Processor -- start 2022-06-27T12:08:05.494 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.468+0000 7fad5b175700 1 -- start start 2022-06-27T12:08:05.494 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.469+0000 7fad5b175700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fad5406fef0 0x7fad541570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:05.495 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.469+0000 7fad5b175700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad5406fba0 con 0x7fad540c34a0 2022-06-27T12:08:05.495 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.469+0000 7fad5b175700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad5406f5e0 con 0x7fad5406e850 2022-06-27T12:08:05.496 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.469+0000 7fad5b175700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad541575e0 con 0x7fad5406fef0 2022-06-27T12:08:05.496 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.469+0000 7fad53fff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fad5406fef0 0x7fad541570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:05.496 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.469+0000 7fad59712700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fad540c34a0 0x7fad540c2920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:55200/0 (socket says 172.21.15.195:55200) 2022-06-27T12:08:05.497 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.469+0000 7fad59712700 1 -- 172.21.15.195:0/1176669558 learned_addr learned my addr 172.21.15.195:0/1176669558 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:08:05.497 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.470+0000 7fad53fff700 1 -- 172.21.15.195:0/1176669558 >> v1:172.21.15.202:6789/0 conn(0x7fad5406e850 legacy=0x7fad54069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:05.498 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.470+0000 7fad53fff700 1 -- 172.21.15.195:0/1176669558 >> v1:172.21.15.195:6789/0 conn(0x7fad540c34a0 legacy=0x7fad540c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:05.498 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.470+0000 7fad53fff700 1 -- 172.21.15.195:0/1176669558 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad5406f8c0 con 0x7fad5406fef0 2022-06-27T12:08:05.499 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.470+0000 7fad537fe700 1 -- 172.21.15.195:0/1176669558 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3728740021 0 0) 0x7fad5406fba0 con 0x7fad540c34a0 2022-06-27T12:08:05.499 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.470+0000 7fad53fff700 1 --2- 172.21.15.195:0/1176669558 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fad5406fef0 0x7fad541570a0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fad400097a0 tx=0x7fad40009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acba9b766b81521f server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:08:05.500 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.470+0000 7fad537fe700 1 -- 172.21.15.195:0/1176669558 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad40011630 con 0x7fad5406fef0 2022-06-27T12:08:05.500 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.470+0000 7fad537fe700 1 -- 172.21.15.195:0/1176669558 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad40011790 con 0x7fad5406fef0 2022-06-27T12:08:05.501 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.470+0000 7fad537fe700 1 -- 172.21.15.195:0/1176669558 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad40011b30 con 0x7fad5406fef0 2022-06-27T12:08:05.501 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.470+0000 7fad5b175700 1 -- 172.21.15.195:0/1176669558 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fad5406fef0 msgr2=0x7fad541570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:05.502 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.470+0000 7fad5b175700 1 --2- 172.21.15.195:0/1176669558 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fad5406fef0 0x7fad541570a0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fad400097a0 tx=0x7fad40009060 comp rx=0 tx=0).stop 2022-06-27T12:08:05.502 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.470+0000 7fad5b175700 1 -- 172.21.15.195:0/1176669558 shutdown_connections 2022-06-27T12:08:05.502 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.470+0000 7fad5b175700 1 --2- 172.21.15.195:0/1176669558 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fad5406fef0 0x7fad541570a0 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:05.503 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.470+0000 7fad5b175700 1 -- 172.21.15.195:0/1176669558 >> 172.21.15.195:0/1176669558 conn(0x7fad540c6350 msgr2=0x7fad540c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:08:05.503 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.470+0000 7fad5b175700 1 -- 172.21.15.195:0/1176669558 shutdown_connections 2022-06-27T12:08:05.504 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.470+0000 7fad5b175700 1 -- 172.21.15.195:0/1176669558 wait complete. 2022-06-27T12:08:05.504 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.472+0000 7fad5b175700 1 Processor -- start 2022-06-27T12:08:05.505 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.472+0000 7fad5b175700 1 -- start start 2022-06-27T12:08:05.505 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.473+0000 7fad5b175700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fad5406e850 0x7fad541671c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:05.505 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.473+0000 7fad5b175700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fad540c34a0 0x7fad54153870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:05.505 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.473+0000 7fad5b175700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fad540baf90 0x7fad540bf750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:05.506 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.473+0000 7fad5b175700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad540bc000 con 0x7fad540c34a0 2022-06-27T12:08:05.506 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.473+0000 7fad5b175700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad541580b0 con 0x7fad540baf90 2022-06-27T12:08:05.506 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.473+0000 7fad5b175700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad540bad70 con 0x7fad5406e850 2022-06-27T12:08:05.506 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.473+0000 7fad53fff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fad540c34a0 0x7fad54153870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:05.507 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.473+0000 7fad53fff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fad540c34a0 0x7fad54153870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:49018/0 (socket says 172.21.15.195:49018) 2022-06-27T12:08:05.507 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.473+0000 7fad53fff700 1 -- 172.21.15.195:0/652656579 learned_addr learned my addr 172.21.15.195:0/652656579 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:08:05.507 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.473+0000 7fad59712700 1 --2- 172.21.15.195:0/652656579 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fad540baf90 0x7fad540bf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:05.507 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.473+0000 7fad58f11700 1 --2- 172.21.15.195:0/652656579 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fad5406e850 0x7fad541671c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:05.508 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.473+0000 7fad53fff700 1 -- 172.21.15.195:0/652656579 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fad5406e850 msgr2=0x7fad541671c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:05.508 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.473+0000 7fad53fff700 1 --2- 172.21.15.195:0/652656579 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fad5406e850 0x7fad541671c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:05.509 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.473+0000 7fad53fff700 1 -- 172.21.15.195:0/652656579 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fad540baf90 msgr2=0x7fad540bf750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:05.509 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.473+0000 7fad53fff700 1 --2- 172.21.15.195:0/652656579 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fad540baf90 0x7fad540bf750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:05.510 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.473+0000 7fad53fff700 1 -- 172.21.15.195:0/652656579 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad400092e0 con 0x7fad540c34a0 2022-06-27T12:08:05.510 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.473+0000 7fad59712700 1 --2- 172.21.15.195:0/652656579 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fad540baf90 0x7fad540bf750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:08:05.510 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.474+0000 7fad53fff700 1 --2- 172.21.15.195:0/652656579 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fad540c34a0 0x7fad54153870 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fad40009ef0 tx=0x7fad40017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:08:05.510 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.474+0000 7fad50ff9700 1 -- 172.21.15.195:0/652656579 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad40000d10 con 0x7fad540c34a0 2022-06-27T12:08:05.511 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.474+0000 7fad50ff9700 1 -- 172.21.15.195:0/652656579 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad40011900 con 0x7fad540c34a0 2022-06-27T12:08:05.511 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.474+0000 7fad5b175700 1 -- 172.21.15.195:0/652656579 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad54166d30 con 0x7fad540c34a0 2022-06-27T12:08:05.511 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.474+0000 7fad50ff9700 1 -- 172.21.15.195:0/652656579 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad40025420 con 0x7fad540c34a0 2022-06-27T12:08:05.511 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.474+0000 7fad5b175700 1 -- 172.21.15.195:0/652656579 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad540ba4d0 con 0x7fad540c34a0 2022-06-27T12:08:05.512 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.475+0000 7fad5b175700 1 -- 172.21.15.195:0/652656579 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad54166ec0 con 0x7fad540c34a0 2022-06-27T12:08:05.512 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.475+0000 7fad50ff9700 1 -- 172.21.15.195:0/652656579 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fad4002c030 con 0x7fad540c34a0 2022-06-27T12:08:05.512 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.475+0000 7fad50ff9700 1 --2- 172.21.15.195:0/652656579 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fad3c063d30 0x7fad3c0661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:05.512 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.476+0000 7fad50ff9700 1 -- 172.21.15.195:0/652656579 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7148+0+0 (secure 0 0 0) 0x7fad40082310 con 0x7fad540c34a0 2022-06-27T12:08:05.512 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.476+0000 7fad58f11700 1 --2- 172.21.15.195:0/652656579 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fad3c063d30 0x7fad3c0661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:05.513 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.476+0000 7fad58f11700 1 --2- 172.21.15.195:0/652656579 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fad3c063d30 0x7fad3c0661e0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fad480016f0 tx=0x7fad48009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:08:05.514 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.480+0000 7fad50ff9700 1 -- 172.21.15.195:0/652656579 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fad40050ac0 con 0x7fad540c34a0 2022-06-27T12:08:05.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.707+0000 7fad5b175700 1 -- 172.21.15.195:0/652656579 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fad540722c0 con 0x7fad540c34a0 2022-06-27T12:08:05.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.707+0000 7fad50ff9700 1 -- 172.21.15.195:0/652656579 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+5371 (secure 0 0 0) 0x7fad4002a020 con 0x7fad540c34a0 2022-06-27T12:08:05.710 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:08:05.711 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":74,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:08:04.355241+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6271},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6271":{"gid":6271,"name":"d","rank":0,"incarnation":73,"state":"up:reconnect","state_seq":14,"addr":"172.21.15.202:6835/1603307620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":1603307620},{"type":"v1","addr":"172.21.15.202:6835","nonce":1603307620}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:08:05.714 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.709+0000 7fad5b175700 1 -- 172.21.15.195:0/652656579 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fad3c063d30 msgr2=0x7fad3c0661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:05.714 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.709+0000 7fad5b175700 1 --2- 172.21.15.195:0/652656579 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fad3c063d30 0x7fad3c0661e0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fad480016f0 tx=0x7fad48009040 comp rx=0 tx=0).stop 2022-06-27T12:08:05.715 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.709+0000 7fad5b175700 1 -- 172.21.15.195:0/652656579 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fad540c34a0 msgr2=0x7fad54153870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:05.715 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.709+0000 7fad5b175700 1 --2- 172.21.15.195:0/652656579 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fad540c34a0 0x7fad54153870 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fad40009ef0 tx=0x7fad40017000 comp rx=0 tx=0).stop 2022-06-27T12:08:05.716 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.709+0000 7fad5b175700 1 -- 172.21.15.195:0/652656579 shutdown_connections 2022-06-27T12:08:05.716 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.709+0000 7fad5b175700 1 --2- 172.21.15.195:0/652656579 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fad3c063d30 0x7fad3c0661e0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:05.717 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.709+0000 7fad5b175700 1 --2- 172.21.15.195:0/652656579 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fad540baf90 0x7fad540bf750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:05.717 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.709+0000 7fad5b175700 1 --2- 172.21.15.195:0/652656579 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fad5406e850 0x7fad541671c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:05.717 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.709+0000 7fad5b175700 1 --2- 172.21.15.195:0/652656579 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fad540c34a0 0x7fad54153870 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:05.718 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.709+0000 7fad5b175700 1 -- 172.21.15.195:0/652656579 >> 172.21.15.195:0/652656579 conn(0x7fad540c6350 msgr2=0x7fad541505c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:08:05.718 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.710+0000 7fad5b175700 1 -- 172.21.15.195:0/652656579 shutdown_connections 2022-06-27T12:08:05.719 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:05.710+0000 7fad5b175700 1 -- 172.21.15.195:0/652656579 wait complete. 2022-06-27T12:08:05.719 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 74 2022-06-27T12:08:05.723 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:08:07.727 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:08:07.885 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:07.876+0000 7f60729b1700 1 Processor -- start 2022-06-27T12:08:07.888 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:07.876+0000 7f60729b1700 1 -- start start 2022-06-27T12:08:07.888 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:07.877+0000 7f60729b1700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f606c150580 0x7f606c156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:07.889 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:07.877+0000 7f60729b1700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f606c150950 0x7f606c0c7b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:07.889 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:07.877+0000 7f60729b1700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f606c0c76b0 con 0x7f606c06f620 2022-06-27T12:08:07.889 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:07.877+0000 7f60729b1700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f606c157430 con 0x7f606c150580 2022-06-27T12:08:07.890 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:07.877+0000 7f60729b1700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f606c0bad70 con 0x7f606c150950 2022-06-27T12:08:07.890 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:07.877+0000 7f606bfff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f606c150580 0x7f606c156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:07.890 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:07.877+0000 7f606bfff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f606c150580 0x7f606c156ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:49022/0 (socket says 172.21.15.195:49022) 2022-06-27T12:08:07.890 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:07.877+0000 7f606bfff700 1 -- 172.21.15.195:0/2786746952 learned_addr learned my addr 172.21.15.195:0/2786746952 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:08:07.891 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:07.877+0000 7f6070f4e700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f606c06f620 0x7f606c06d6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:55214/0 (socket says 172.21.15.195:55214) 2022-06-27T12:08:07.891 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:07.877+0000 7f606b7fe700 1 --2- 172.21.15.195:0/2786746952 >> v2:172.21.15.202:3300/0 conn(0x7f606c150950 0x7f606c0c7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:07.891 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:07.877+0000 7f606bfff700 1 -- 172.21.15.195:0/2786746952 >> v2:172.21.15.202:3300/0 conn(0x7f606c150950 msgr2=0x7f606c0c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:08:07.892 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:07.877+0000 7f606bfff700 1 --2- 172.21.15.195:0/2786746952 >> v2:172.21.15.202:3300/0 conn(0x7f606c150950 0x7f606c0c7b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:07.892 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:07.877+0000 7f606bfff700 1 -- 172.21.15.195:0/2786746952 >> v1:172.21.15.195:6789/0 conn(0x7f606c06f620 legacy=0x7f606c06d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:07.892 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:07.877+0000 7f606bfff700 1 -- 172.21.15.195:0/2786746952 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f606c0ba4d0 con 0x7f606c150580 2022-06-27T12:08:07.893 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:07.877+0000 7f606affd700 1 -- 172.21.15.195:0/2786746952 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1491574078 0 0) 0x7f606c0c76b0 con 0x7f606c06f620 2022-06-27T12:08:07.894 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:07.877+0000 7f606b7fe700 1 --2- 172.21.15.195:0/2786746952 >> v2:172.21.15.202:3300/0 conn(0x7f606c150950 0x7f606c0c7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:08:07.894 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:07.877+0000 7f606bfff700 1 --2- 172.21.15.195:0/2786746952 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f606c150580 0x7f606c156ef0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f606c06ea70 tx=0x7f6060009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e52195196b04a164 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:08:07.894 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:07.878+0000 7f606affd700 1 -- 172.21.15.195:0/2786746952 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f606000b640 con 0x7f606c150580 2022-06-27T12:08:07.894 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:07.878+0000 7f606affd700 1 -- 172.21.15.195:0/2786746952 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f606000b7a0 con 0x7f606c150580 2022-06-27T12:08:07.895 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:07.878+0000 7f606affd700 1 -- 172.21.15.195:0/2786746952 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f606000bb40 con 0x7f606c150580 2022-06-27T12:08:07.895 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:07.878+0000 7f60729b1700 1 -- 172.21.15.195:0/2786746952 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f606c150580 msgr2=0x7f606c156ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:07.895 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:07.878+0000 7f60729b1700 1 --2- 172.21.15.195:0/2786746952 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f606c150580 0x7f606c156ef0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f606c06ea70 tx=0x7f6060009120 comp rx=0 tx=0).stop 2022-06-27T12:08:07.896 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:07.878+0000 7f60729b1700 1 -- 172.21.15.195:0/2786746952 shutdown_connections 2022-06-27T12:08:07.896 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:07.878+0000 7f60729b1700 1 --2- 172.21.15.195:0/2786746952 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f606c150580 0x7f606c156ef0 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:07.896 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:07.878+0000 7f60729b1700 1 --2- 172.21.15.195:0/2786746952 >> v2:172.21.15.202:3300/0 conn(0x7f606c150950 0x7f606c0c7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:07.896 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:07.878+0000 7f60729b1700 1 -- 172.21.15.195:0/2786746952 >> 172.21.15.195:0/2786746952 conn(0x7f606c063af0 msgr2=0x7f606c063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:08:07.899 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:07.878+0000 7f60729b1700 1 -- 172.21.15.195:0/2786746952 shutdown_connections 2022-06-27T12:08:07.900 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:07.878+0000 7f60729b1700 1 -- 172.21.15.195:0/2786746952 wait complete. 2022-06-27T12:08:07.900 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:07.879+0000 7f60729b1700 1 Processor -- start 2022-06-27T12:08:07.900 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:07.880+0000 7f60729b1700 1 -- start start 2022-06-27T12:08:07.901 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:07.880+0000 7f60729b1700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f606c06f620 0x7f606c16a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:07.901 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:07.880+0000 7f60729b1700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f606c150580 0x7f606c166f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:07.901 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:07.880+0000 7f60729b1700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f606c150950 0x7f606c14e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:07.901 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:07.880+0000 7f60729b1700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f606c06a1c0 con 0x7f606c06f620 2022-06-27T12:08:07.902 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:07.880+0000 7f60729b1700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f606c0c8970 con 0x7f606c150580 2022-06-27T12:08:07.902 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:07.880+0000 7f60729b1700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f606c0c8c50 con 0x7f606c150950 2022-06-27T12:08:07.902 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:07.880+0000 7f606bfff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f606c06f620 0x7f606c16a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:07.902 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:07.880+0000 7f606bfff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f606c06f620 0x7f606c16a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:49028/0 (socket says 172.21.15.195:49028) 2022-06-27T12:08:07.903 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:07.880+0000 7f606bfff700 1 -- 172.21.15.195:0/494626163 learned_addr learned my addr 172.21.15.195:0/494626163 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:08:07.903 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:07.881+0000 7f606b7fe700 1 --2- 172.21.15.195:0/494626163 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f606c150580 0x7f606c166f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:07.903 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:07.881+0000 7f6070f4e700 1 --2- 172.21.15.195:0/494626163 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f606c150950 0x7f606c14e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:07.906 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:07.881+0000 7f606bfff700 1 -- 172.21.15.195:0/494626163 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f606c150950 msgr2=0x7f606c14e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:07.906 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:07.881+0000 7f606bfff700 1 --2- 172.21.15.195:0/494626163 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f606c150950 0x7f606c14e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:07.906 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:07.881+0000 7f606bfff700 1 -- 172.21.15.195:0/494626163 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f606c150580 msgr2=0x7f606c166f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:07.906 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:07.881+0000 7f606bfff700 1 --2- 172.21.15.195:0/494626163 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f606c150580 0x7f606c166f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:07.907 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:07.881+0000 7f606bfff700 1 -- 172.21.15.195:0/494626163 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60600093a0 con 0x7f606c06f620 2022-06-27T12:08:07.907 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:07.881+0000 7f606bfff700 1 --2- 172.21.15.195:0/494626163 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f606c06f620 0x7f606c16a640 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f606000a010 tx=0x7f6060017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:08:07.907 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:07.881+0000 7f6053fff700 1 -- 172.21.15.195:0/494626163 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f606000beb0 con 0x7f606c06f620 2022-06-27T12:08:07.912 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:07.881+0000 7f6053fff700 1 -- 172.21.15.195:0/494626163 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6060003730 con 0x7f606c06f620 2022-06-27T12:08:07.912 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:07.881+0000 7f60729b1700 1 -- 172.21.15.195:0/494626163 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f606c0c8f30 con 0x7f606c06f620 2022-06-27T12:08:07.913 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:07.881+0000 7f6053fff700 1 -- 172.21.15.195:0/494626163 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6060003b10 con 0x7f606c06f620 2022-06-27T12:08:07.913 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:07.881+0000 7f60729b1700 1 -- 172.21.15.195:0/494626163 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f606c1666a0 con 0x7f606c06f620 2022-06-27T12:08:07.913 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:07.882+0000 7f60729b1700 1 -- 172.21.15.195:0/494626163 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f606c166c30 con 0x7f606c06f620 2022-06-27T12:08:07.914 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:07.882+0000 7f6053fff700 1 -- 172.21.15.195:0/494626163 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f606000f050 con 0x7f606c06f620 2022-06-27T12:08:07.914 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:07.883+0000 7f6053fff700 1 --2- 172.21.15.195:0/494626163 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f604c063d00 0x7f604c0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:07.914 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:07.883+0000 7f6053fff700 1 -- 172.21.15.195:0/494626163 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7148+0+0 (secure 0 0 0) 0x7f60600821d0 con 0x7f606c06f620 2022-06-27T12:08:07.915 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:07.883+0000 7f606b7fe700 1 --2- 172.21.15.195:0/494626163 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f604c063d00 0x7f604c0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:07.915 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:07.883+0000 7f606b7fe700 1 --2- 172.21.15.195:0/494626163 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f604c063d00 0x7f604c0661b0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f606c150280 tx=0x7f605c008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:08:07.915 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:07.887+0000 7f6053fff700 1 -- 172.21.15.195:0/494626163 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f606c166c30 con 0x7f606c06f620 2022-06-27T12:08:08.110 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:08.108+0000 7f60729b1700 1 -- 172.21.15.195:0/494626163 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f606c166c30 con 0x7f606c06f620 2022-06-27T12:08:08.111 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:08.108+0000 7f6053fff700 1 -- 172.21.15.195:0/494626163 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+5371 (secure 0 0 0) 0x7f606c166c30 con 0x7f606c06f620 2022-06-27T12:08:08.111 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:08:08.112 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":74,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:08:04.355241+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6271},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6271":{"gid":6271,"name":"d","rank":0,"incarnation":73,"state":"up:reconnect","state_seq":14,"addr":"172.21.15.202:6835/1603307620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":1603307620},{"type":"v1","addr":"172.21.15.202:6835","nonce":1603307620}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:08:08.113 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:08.110+0000 7f60729b1700 1 -- 172.21.15.195:0/494626163 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f604c063d00 msgr2=0x7f604c0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:08.113 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:08.110+0000 7f60729b1700 1 --2- 172.21.15.195:0/494626163 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f604c063d00 0x7f604c0661b0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f606c150280 tx=0x7f605c008040 comp rx=0 tx=0).stop 2022-06-27T12:08:08.114 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:08.110+0000 7f60729b1700 1 -- 172.21.15.195:0/494626163 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f606c06f620 msgr2=0x7f606c16a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:08.114 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:08.110+0000 7f60729b1700 1 --2- 172.21.15.195:0/494626163 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f606c06f620 0x7f606c16a640 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f606000a010 tx=0x7f6060017000 comp rx=0 tx=0).stop 2022-06-27T12:08:08.114 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:08.110+0000 7f60729b1700 1 -- 172.21.15.195:0/494626163 shutdown_connections 2022-06-27T12:08:08.114 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:08.110+0000 7f60729b1700 1 --2- 172.21.15.195:0/494626163 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f604c063d00 0x7f604c0661b0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:08.115 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:08.110+0000 7f60729b1700 1 --2- 172.21.15.195:0/494626163 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f606c06f620 0x7f606c16a640 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:08.115 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:08.110+0000 7f60729b1700 1 --2- 172.21.15.195:0/494626163 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f606c150950 0x7f606c14e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:08.115 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:08.110+0000 7f60729b1700 1 --2- 172.21.15.195:0/494626163 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f606c150580 0x7f606c166f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:08.115 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:08.110+0000 7f60729b1700 1 -- 172.21.15.195:0/494626163 >> 172.21.15.195:0/494626163 conn(0x7f606c063af0 msgr2=0x7f606c0c5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:08:08.116 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:08.110+0000 7f60729b1700 1 -- 172.21.15.195:0/494626163 shutdown_connections 2022-06-27T12:08:08.116 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:08.111+0000 7f60729b1700 1 -- 172.21.15.195:0/494626163 wait complete. 2022-06-27T12:08:08.116 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 74 2022-06-27T12:08:08.135 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:08:10.137 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:08:10.278 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.272+0000 7fc5d7518700 1 Processor -- start 2022-06-27T12:08:10.278 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.272+0000 7fc5d7518700 1 -- start start 2022-06-27T12:08:10.278 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.273+0000 7fc5d7518700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fc5d0156100 0x7fc5d00688a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:10.279 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.273+0000 7fc5d7518700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fc5d006a440 0x7fc5d00c4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:10.279 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.273+0000 7fc5d7518700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5d00c4b10 con 0x7fc5d01564d0 2022-06-27T12:08:10.279 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.273+0000 7fc5d7518700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5d00c3dc0 con 0x7fc5d006a440 2022-06-27T12:08:10.279 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.273+0000 7fc5d7518700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5d0068f20 con 0x7fc5d0156100 2022-06-27T12:08:10.280 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.273+0000 7fc5d4ab3700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fc5d006a440 0x7fc5d00c4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:10.280 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.273+0000 7fc5d4ab3700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fc5d006a440 0x7fc5d00c4fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:49038/0 (socket says 172.21.15.195:49038) 2022-06-27T12:08:10.280 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.273+0000 7fc5d4ab3700 1 -- 172.21.15.195:0/1464477448 learned_addr learned my addr 172.21.15.195:0/1464477448 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:08:10.280 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.273+0000 7fc5d52b4700 1 --2- 172.21.15.195:0/1464477448 >> v2:172.21.15.202:3300/0 conn(0x7fc5d0156100 0x7fc5d00688a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:10.281 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.273+0000 7fc5d4ab3700 1 -- 172.21.15.195:0/1464477448 >> v2:172.21.15.202:3300/0 conn(0x7fc5d0156100 msgr2=0x7fc5d00688a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:08:10.281 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.273+0000 7fc5d4ab3700 1 --2- 172.21.15.195:0/1464477448 >> v2:172.21.15.202:3300/0 conn(0x7fc5d0156100 0x7fc5d00688a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:10.282 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.273+0000 7fc5d4ab3700 1 -- 172.21.15.195:0/1464477448 >> v1:172.21.15.202:6789/0 conn(0x7fc5d01564d0 legacy=0x7fc5d00c28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:10.283 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.273+0000 7fc5d4ab3700 1 -- 172.21.15.195:0/1464477448 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5d00c6360 con 0x7fc5d006a440 2022-06-27T12:08:10.283 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.273+0000 7fc5d4ab3700 1 --2- 172.21.15.195:0/1464477448 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc5d006a440 0x7fc5d00c4fa0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fc5d0070770 tx=0x7fc5c0009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b1043e305721c36 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:08:10.284 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.274+0000 7fc5bffff700 1 -- 172.21.15.195:0/1464477448 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c0011630 con 0x7fc5d006a440 2022-06-27T12:08:10.284 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.274+0000 7fc5bffff700 1 -- 172.21.15.195:0/1464477448 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5c0011790 con 0x7fc5d006a440 2022-06-27T12:08:10.285 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.274+0000 7fc5bffff700 1 -- 172.21.15.195:0/1464477448 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c0011b30 con 0x7fc5d006a440 2022-06-27T12:08:10.285 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.274+0000 7fc5d7518700 1 -- 172.21.15.195:0/1464477448 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc5d006a440 msgr2=0x7fc5d00c4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:10.285 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.274+0000 7fc5d7518700 1 --2- 172.21.15.195:0/1464477448 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc5d006a440 0x7fc5d00c4fa0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fc5d0070770 tx=0x7fc5c0009120 comp rx=0 tx=0).stop 2022-06-27T12:08:10.286 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.274+0000 7fc5d7518700 1 -- 172.21.15.195:0/1464477448 shutdown_connections 2022-06-27T12:08:10.286 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.274+0000 7fc5d7518700 1 --2- 172.21.15.195:0/1464477448 >> v2:172.21.15.202:3300/0 conn(0x7fc5d0156100 0x7fc5d00688a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:10.287 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.274+0000 7fc5d7518700 1 --2- 172.21.15.195:0/1464477448 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc5d006a440 0x7fc5d00c4fa0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:10.287 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.274+0000 7fc5d7518700 1 -- 172.21.15.195:0/1464477448 >> 172.21.15.195:0/1464477448 conn(0x7fc5d00c9b00 msgr2=0x7fc5d00c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:08:10.288 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.274+0000 7fc5d7518700 1 -- 172.21.15.195:0/1464477448 shutdown_connections 2022-06-27T12:08:10.288 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.274+0000 7fc5d7518700 1 -- 172.21.15.195:0/1464477448 wait complete. 2022-06-27T12:08:10.289 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.275+0000 7fc5d7518700 1 Processor -- start 2022-06-27T12:08:10.289 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.275+0000 7fc5d7518700 1 -- start start 2022-06-27T12:08:10.290 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.276+0000 7fc5d7518700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc5d006a440 0x7fc5d0147a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:10.290 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.276+0000 7fc5d7518700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc5d0156100 0x7fc5d01549f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:10.291 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.276+0000 7fc5d7518700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc5d01564d0 0x7fc5d0167aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:10.292 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.276+0000 7fc5d7518700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d00bfa60 con 0x7fc5d006a440 2022-06-27T12:08:10.292 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.276+0000 7fc5d7518700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d0152f70 con 0x7fc5d0156100 2022-06-27T12:08:10.293 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.276+0000 7fc5d7518700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d00bdff0 con 0x7fc5d01564d0 2022-06-27T12:08:10.293 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.276+0000 7fc5d52b4700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc5d006a440 0x7fc5d0147a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:10.294 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.276+0000 7fc5d52b4700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc5d006a440 0x7fc5d0147a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:49044/0 (socket says 172.21.15.195:49044) 2022-06-27T12:08:10.294 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.276+0000 7fc5d52b4700 1 -- 172.21.15.195:0/3075603263 learned_addr learned my addr 172.21.15.195:0/3075603263 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:08:10.295 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.277+0000 7fc5d4ab3700 1 --2- 172.21.15.195:0/3075603263 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc5d0156100 0x7fc5d01549f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:10.295 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.277+0000 7fc5d5ab5700 1 --2- 172.21.15.195:0/3075603263 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc5d01564d0 0x7fc5d0167aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:10.296 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.277+0000 7fc5d52b4700 1 -- 172.21.15.195:0/3075603263 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc5d01564d0 msgr2=0x7fc5d0167aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:10.296 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.277+0000 7fc5d52b4700 1 --2- 172.21.15.195:0/3075603263 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc5d01564d0 0x7fc5d0167aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:10.297 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.277+0000 7fc5d52b4700 1 -- 172.21.15.195:0/3075603263 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc5d0156100 msgr2=0x7fc5d01549f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:10.297 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.277+0000 7fc5d52b4700 1 --2- 172.21.15.195:0/3075603263 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc5d0156100 0x7fc5d01549f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:10.298 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.277+0000 7fc5d52b4700 1 -- 172.21.15.195:0/3075603263 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5c00093a0 con 0x7fc5d006a440 2022-06-27T12:08:10.298 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.277+0000 7fc5d52b4700 1 --2- 172.21.15.195:0/3075603263 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc5d006a440 0x7fc5d0147a90 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc00e960 tx=0x7fc5cc009550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:08:10.299 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.277+0000 7fc5bd7fa700 1 -- 172.21.15.195:0/3075603263 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5cc013680 con 0x7fc5d006a440 2022-06-27T12:08:10.299 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.277+0000 7fc5bd7fa700 1 -- 172.21.15.195:0/3075603263 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5cc0137e0 con 0x7fc5d006a440 2022-06-27T12:08:10.299 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.277+0000 7fc5d7518700 1 -- 172.21.15.195:0/3075603263 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5d0152380 con 0x7fc5d006a440 2022-06-27T12:08:10.300 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.277+0000 7fc5bd7fa700 1 -- 172.21.15.195:0/3075603263 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5cc013b60 con 0x7fc5d006a440 2022-06-27T12:08:10.300 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.277+0000 7fc5d7518700 1 -- 172.21.15.195:0/3075603263 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5d0152920 con 0x7fc5d006a440 2022-06-27T12:08:10.303 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.278+0000 7fc5d7518700 1 -- 172.21.15.195:0/3075603263 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5d00722c0 con 0x7fc5d006a440 2022-06-27T12:08:10.303 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.279+0000 7fc5bd7fa700 1 -- 172.21.15.195:0/3075603263 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fc5cc023070 con 0x7fc5d006a440 2022-06-27T12:08:10.304 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.279+0000 7fc5bd7fa700 1 --2- 172.21.15.195:0/3075603263 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc5b0063d00 0x7fc5b00661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:10.304 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.279+0000 7fc5bd7fa700 1 -- 172.21.15.195:0/3075603263 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7148+0+0 (secure 0 0 0) 0x7fc5cc080ff0 con 0x7fc5d006a440 2022-06-27T12:08:10.305 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.279+0000 7fc5d4ab3700 1 --2- 172.21.15.195:0/3075603263 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc5b0063d00 0x7fc5b00661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:10.305 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.279+0000 7fc5d4ab3700 1 --2- 172.21.15.195:0/3075603263 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc5b0063d00 0x7fc5b00661b0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fc5d006a860 tx=0x7fc5c0017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:08:10.306 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.283+0000 7fc5bd7fa700 1 -- 172.21.15.195:0/3075603263 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fc5cc04f7a0 con 0x7fc5d006a440 2022-06-27T12:08:10.512 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.512+0000 7fc5d7518700 1 -- 172.21.15.195:0/3075603263 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc5d00c6640 con 0x7fc5d006a440 2022-06-27T12:08:10.513 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.513+0000 7fc5bd7fa700 1 -- 172.21.15.195:0/3075603263 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+5368 (secure 0 0 0) 0x7fc5cc021090 con 0x7fc5d006a440 2022-06-27T12:08:10.514 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:08:10.514 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":75,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:08:10.380461+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6271},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6271":{"gid":6271,"name":"d","rank":0,"incarnation":73,"state":"up:rejoin","state_seq":16,"addr":"172.21.15.202:6835/1603307620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":1603307620},{"type":"v1","addr":"172.21.15.202:6835","nonce":1603307620}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:08:10.516 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.515+0000 7fc5d7518700 1 -- 172.21.15.195:0/3075603263 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc5b0063d00 msgr2=0x7fc5b00661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:10.516 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.515+0000 7fc5d7518700 1 --2- 172.21.15.195:0/3075603263 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc5b0063d00 0x7fc5b00661b0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fc5d006a860 tx=0x7fc5c0017000 comp rx=0 tx=0).stop 2022-06-27T12:08:10.517 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.515+0000 7fc5d7518700 1 -- 172.21.15.195:0/3075603263 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc5d006a440 msgr2=0x7fc5d0147a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:10.517 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.515+0000 7fc5d7518700 1 --2- 172.21.15.195:0/3075603263 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc5d006a440 0x7fc5d0147a90 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc00e960 tx=0x7fc5cc009550 comp rx=0 tx=0).stop 2022-06-27T12:08:10.517 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.515+0000 7fc5d7518700 1 -- 172.21.15.195:0/3075603263 shutdown_connections 2022-06-27T12:08:10.518 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.515+0000 7fc5d7518700 1 --2- 172.21.15.195:0/3075603263 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc5b0063d00 0x7fc5b00661b0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:10.518 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.515+0000 7fc5d7518700 1 --2- 172.21.15.195:0/3075603263 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc5d006a440 0x7fc5d0147a90 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:10.518 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.515+0000 7fc5d7518700 1 --2- 172.21.15.195:0/3075603263 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc5d01564d0 0x7fc5d0167aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:10.518 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.515+0000 7fc5d7518700 1 --2- 172.21.15.195:0/3075603263 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc5d0156100 0x7fc5d01549f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:10.519 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.515+0000 7fc5d7518700 1 -- 172.21.15.195:0/3075603263 >> 172.21.15.195:0/3075603263 conn(0x7fc5d00c9b00 msgr2=0x7fc5d00c1870 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:08:10.519 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.515+0000 7fc5d7518700 1 -- 172.21.15.195:0/3075603263 shutdown_connections 2022-06-27T12:08:10.520 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:10.515+0000 7fc5d7518700 1 -- 172.21.15.195:0/3075603263 wait complete. 2022-06-27T12:08:10.520 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 75 2022-06-27T12:08:10.528 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:08:12.529 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:08:12.663 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.663+0000 7f9e5df53700 1 Processor -- start 2022-06-27T12:08:12.664 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.663+0000 7f9e5df53700 1 -- start start 2022-06-27T12:08:12.667 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.664+0000 7f9e5df53700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9e58156100 0x7f9e580688a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:12.668 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.664+0000 7f9e5df53700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f9e581564d0 0x7f9e58156950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:12.668 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.664+0000 7f9e5df53700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e580c4b10 con 0x7f9e5806a440 2022-06-27T12:08:12.669 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.664+0000 7f9e5df53700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e580c3dc0 con 0x7f9e581564d0 2022-06-27T12:08:12.669 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.664+0000 7f9e5df53700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e58068f20 con 0x7f9e58156100 2022-06-27T12:08:12.669 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.664+0000 7f9e57fff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f9e581564d0 0x7f9e58156950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:12.670 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.664+0000 7f9e57fff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f9e581564d0 0x7f9e58156950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:49054/0 (socket says 172.21.15.195:49054) 2022-06-27T12:08:12.670 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.664+0000 7f9e57fff700 1 -- 172.21.15.195:0/2025687628 learned_addr learned my addr 172.21.15.195:0/2025687628 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:08:12.671 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.664+0000 7f9e57fff700 1 -- 172.21.15.195:0/2025687628 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9e58156100 msgr2=0x7f9e580688a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:08:12.671 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.664+0000 7f9e577fe700 1 --2- 172.21.15.195:0/2025687628 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9e58156100 0x7f9e580688a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:12.671 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.664+0000 7f9e57fff700 1 --2- 172.21.15.195:0/2025687628 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9e58156100 0x7f9e580688a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:12.673 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.664+0000 7f9e57fff700 1 -- 172.21.15.195:0/2025687628 >> v1:172.21.15.202:6789/0 conn(0x7f9e5806a440 legacy=0x7f9e580c4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:12.674 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.664+0000 7f9e57fff700 1 -- 172.21.15.195:0/2025687628 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e580c56b0 con 0x7f9e581564d0 2022-06-27T12:08:12.674 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.664+0000 7f9e577fe700 1 --2- 172.21.15.195:0/2025687628 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9e58156100 0x7f9e580688a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T12:08:12.675 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.665+0000 7f9e57fff700 1 --2- 172.21.15.195:0/2025687628 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9e581564d0 0x7f9e58156950 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f9e50009f20 tx=0x7f9e50009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60575e748e98963e server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:08:12.690 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.665+0000 7f9e567fc700 1 -- 172.21.15.195:0/2025687628 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e5000e070 con 0x7f9e581564d0 2022-06-27T12:08:12.691 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.665+0000 7f9e567fc700 1 -- 172.21.15.195:0/2025687628 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e5000ba80 con 0x7f9e581564d0 2022-06-27T12:08:12.693 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.665+0000 7f9e567fc700 1 -- 172.21.15.195:0/2025687628 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e500092e0 con 0x7f9e581564d0 2022-06-27T12:08:12.694 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.665+0000 7f9e5df53700 1 -- 172.21.15.195:0/2025687628 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9e581564d0 msgr2=0x7f9e58156950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:12.694 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.665+0000 7f9e5df53700 1 --2- 172.21.15.195:0/2025687628 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9e581564d0 0x7f9e58156950 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f9e50009f20 tx=0x7f9e50009840 comp rx=0 tx=0).stop 2022-06-27T12:08:12.695 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.665+0000 7f9e5df53700 1 -- 172.21.15.195:0/2025687628 shutdown_connections 2022-06-27T12:08:12.696 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.665+0000 7f9e5df53700 1 --2- 172.21.15.195:0/2025687628 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9e581564d0 0x7f9e58156950 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:12.696 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.665+0000 7f9e5df53700 1 --2- 172.21.15.195:0/2025687628 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9e58156100 0x7f9e580688a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:12.697 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.665+0000 7f9e5df53700 1 -- 172.21.15.195:0/2025687628 >> 172.21.15.195:0/2025687628 conn(0x7f9e580c9b00 msgr2=0x7f9e580c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:08:12.698 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.665+0000 7f9e5df53700 1 -- 172.21.15.195:0/2025687628 shutdown_connections 2022-06-27T12:08:12.698 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.665+0000 7f9e5df53700 1 -- 172.21.15.195:0/2025687628 wait complete. 2022-06-27T12:08:12.699 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.666+0000 7f9e5df53700 1 Processor -- start 2022-06-27T12:08:12.699 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.667+0000 7f9e5df53700 1 -- start start 2022-06-27T12:08:12.700 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.667+0000 7f9e5df53700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9e5806a440 0x7f9e580bb740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:12.701 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.667+0000 7f9e5df53700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9e58156100 0x7f9e580c1600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:12.702 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.667+0000 7f9e5df53700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9e581564d0 0x7f9e580c1b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:12.702 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.667+0000 7f9e5df53700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e58147a20 con 0x7f9e5806a440 2022-06-27T12:08:12.703 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.667+0000 7f9e5df53700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e580beff0 con 0x7f9e58156100 2022-06-27T12:08:12.703 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.667+0000 7f9e5df53700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e580bf290 con 0x7f9e581564d0 2022-06-27T12:08:12.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.667+0000 7f9e577fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9e5806a440 0x7f9e580bb740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:12.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.667+0000 7f9e577fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9e5806a440 0x7f9e580bb740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:49058/0 (socket says 172.21.15.195:49058) 2022-06-27T12:08:12.705 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.667+0000 7f9e577fe700 1 -- 172.21.15.195:0/2944346367 learned_addr learned my addr 172.21.15.195:0/2944346367 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:08:12.706 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.668+0000 7f9e57fff700 1 --2- 172.21.15.195:0/2944346367 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9e581564d0 0x7f9e580c1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:12.707 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.668+0000 7f9e56ffd700 1 --2- 172.21.15.195:0/2944346367 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9e58156100 0x7f9e580c1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:12.708 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.668+0000 7f9e577fe700 1 -- 172.21.15.195:0/2944346367 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9e581564d0 msgr2=0x7f9e580c1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:12.708 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.668+0000 7f9e577fe700 1 --2- 172.21.15.195:0/2944346367 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9e581564d0 0x7f9e580c1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:12.708 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.668+0000 7f9e577fe700 1 -- 172.21.15.195:0/2944346367 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9e58156100 msgr2=0x7f9e580c1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:12.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.668+0000 7f9e577fe700 1 --2- 172.21.15.195:0/2944346367 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9e58156100 0x7f9e580c1600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:12.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.668+0000 7f9e577fe700 1 -- 172.21.15.195:0/2944346367 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e50009ae0 con 0x7f9e5806a440 2022-06-27T12:08:12.710 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.668+0000 7f9e56ffd700 1 --2- 172.21.15.195:0/2944346367 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9e58156100 0x7f9e580c1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:08:12.710 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.668+0000 7f9e577fe700 1 --2- 172.21.15.195:0/2944346367 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9e5806a440 0x7f9e580bb740 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f9e4800c880 tx=0x7f9e48008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:08:12.723 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.668+0000 7f9e3f7fe700 1 -- 172.21.15.195:0/2944346367 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e480058c0 con 0x7f9e5806a440 2022-06-27T12:08:12.723 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.668+0000 7f9e3f7fe700 1 -- 172.21.15.195:0/2944346367 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e48005a20 con 0x7f9e5806a440 2022-06-27T12:08:12.724 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.668+0000 7f9e5df53700 1 -- 172.21.15.195:0/2944346367 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e581688f0 con 0x7f9e5806a440 2022-06-27T12:08:12.725 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.668+0000 7f9e3f7fe700 1 -- 172.21.15.195:0/2944346367 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e48005dc0 con 0x7f9e5806a440 2022-06-27T12:08:12.725 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.668+0000 7f9e5df53700 1 -- 172.21.15.195:0/2944346367 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e58168c80 con 0x7f9e5806a440 2022-06-27T12:08:12.728 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.669+0000 7f9e5df53700 1 -- 172.21.15.195:0/2944346367 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e580722c0 con 0x7f9e5806a440 2022-06-27T12:08:12.728 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.669+0000 7f9e3f7fe700 1 -- 172.21.15.195:0/2944346367 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f9e48020070 con 0x7f9e5806a440 2022-06-27T12:08:12.729 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.670+0000 7f9e3f7fe700 1 --2- 172.21.15.195:0/2944346367 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9e38063d00 0x7f9e380661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:12.729 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.670+0000 7f9e3f7fe700 1 -- 172.21.15.195:0/2944346367 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7148+0+0 (secure 0 0 0) 0x7f9e48080890 con 0x7f9e5806a440 2022-06-27T12:08:12.729 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.670+0000 7f9e56ffd700 1 --2- 172.21.15.195:0/2944346367 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9e38063d00 0x7f9e380661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:12.730 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.670+0000 7f9e56ffd700 1 --2- 172.21.15.195:0/2944346367 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9e38063d00 0x7f9e380661b0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f9e580c23a0 tx=0x7f9e4c009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:08:12.730 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.674+0000 7f9e3f7fe700 1 -- 172.21.15.195:0/2944346367 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f9e4804f040 con 0x7f9e5806a440 2022-06-27T12:08:12.896 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.892+0000 7f9e5df53700 1 -- 172.21.15.195:0/2944346367 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e580c6360 con 0x7f9e5806a440 2022-06-27T12:08:12.896 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.893+0000 7f9e3f7fe700 1 -- 172.21.15.195:0/2944346367 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+5368 (secure 0 0 0) 0x7f9e4801e070 con 0x7f9e5806a440 2022-06-27T12:08:12.897 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:08:12.897 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":75,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:08:10.380461+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6271},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6271":{"gid":6271,"name":"d","rank":0,"incarnation":73,"state":"up:rejoin","state_seq":16,"addr":"172.21.15.202:6835/1603307620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":1603307620},{"type":"v1","addr":"172.21.15.202:6835","nonce":1603307620}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:08:12.900 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.895+0000 7f9e5df53700 1 -- 172.21.15.195:0/2944346367 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9e38063d00 msgr2=0x7f9e380661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:12.901 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.895+0000 7f9e5df53700 1 --2- 172.21.15.195:0/2944346367 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9e38063d00 0x7f9e380661b0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f9e580c23a0 tx=0x7f9e4c009040 comp rx=0 tx=0).stop 2022-06-27T12:08:12.901 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.895+0000 7f9e5df53700 1 -- 172.21.15.195:0/2944346367 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9e5806a440 msgr2=0x7f9e580bb740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:12.901 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.895+0000 7f9e5df53700 1 --2- 172.21.15.195:0/2944346367 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9e5806a440 0x7f9e580bb740 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f9e4800c880 tx=0x7f9e48008040 comp rx=0 tx=0).stop 2022-06-27T12:08:12.901 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.895+0000 7f9e5df53700 1 -- 172.21.15.195:0/2944346367 shutdown_connections 2022-06-27T12:08:12.902 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.895+0000 7f9e5df53700 1 --2- 172.21.15.195:0/2944346367 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9e38063d00 0x7f9e380661b0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:12.902 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.895+0000 7f9e5df53700 1 --2- 172.21.15.195:0/2944346367 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9e5806a440 0x7f9e580bb740 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:12.902 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.895+0000 7f9e5df53700 1 --2- 172.21.15.195:0/2944346367 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9e581564d0 0x7f9e580c1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:12.903 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.895+0000 7f9e5df53700 1 --2- 172.21.15.195:0/2944346367 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9e58156100 0x7f9e580c1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:12.903 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.895+0000 7f9e5df53700 1 -- 172.21.15.195:0/2944346367 >> 172.21.15.195:0/2944346367 conn(0x7f9e580c9b00 msgr2=0x7f9e58071200 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:08:12.903 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.895+0000 7f9e5df53700 1 -- 172.21.15.195:0/2944346367 shutdown_connections 2022-06-27T12:08:12.904 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:12.895+0000 7f9e5df53700 1 -- 172.21.15.195:0/2944346367 wait complete. 2022-06-27T12:08:12.904 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 75 2022-06-27T12:08:12.908 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:08:14.910 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:08:15.048 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.044+0000 7f115a69d700 1 Processor -- start 2022-06-27T12:08:15.048 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.044+0000 7f115a69d700 1 -- start start 2022-06-27T12:08:15.048 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.044+0000 7f115a69d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f115406e850 0x7f11541570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:15.049 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.044+0000 7f115a69d700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f115406fef0 0x7f1154069de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:15.049 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.044+0000 7f115a69d700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f115406fba0 con 0x7f11540c34a0 2022-06-27T12:08:15.049 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.044+0000 7f115a69d700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11541575e0 con 0x7f115406e850 2022-06-27T12:08:15.050 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.045+0000 7f115a69d700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f115406f5e0 con 0x7f115406fef0 2022-06-27T12:08:15.050 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.045+0000 7f1153fff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f115406e850 0x7f11541570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:15.050 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.045+0000 7f1158c3a700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f11540c34a0 0x7f11540c2920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:55258/0 (socket says 172.21.15.195:55258) 2022-06-27T12:08:15.051 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.045+0000 7f1158c3a700 1 -- 172.21.15.195:0/126807108 learned_addr learned my addr 172.21.15.195:0/126807108 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:08:15.051 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.045+0000 7f11537fe700 1 --2- 172.21.15.195:0/126807108 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f115406fef0 0x7f1154069de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:15.051 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.045+0000 7f1152ffd700 1 -- 172.21.15.195:0/126807108 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 131752055 0 0) 0x7f115406fba0 con 0x7f11540c34a0 2022-06-27T12:08:15.052 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.045+0000 7f1152ffd700 1 -- 172.21.15.195:0/126807108 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1140003430 con 0x7f11540c34a0 2022-06-27T12:08:15.052 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.045+0000 7f1153fff700 1 -- 172.21.15.195:0/126807108 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f115406fef0 msgr2=0x7f1154069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:08:15.053 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.045+0000 7f1153fff700 1 --2- 172.21.15.195:0/126807108 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f115406fef0 0x7f1154069de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:15.053 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.045+0000 7f1153fff700 1 -- 172.21.15.195:0/126807108 >> v1:172.21.15.195:6789/0 conn(0x7f11540c34a0 legacy=0x7f11540c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:15.053 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.045+0000 7f1153fff700 1 -- 172.21.15.195:0/126807108 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f115406f8c0 con 0x7f115406e850 2022-06-27T12:08:15.054 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.045+0000 7f1153fff700 1 --2- 172.21.15.195:0/126807108 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f115406e850 0x7f11541570a0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f11440097a0 tx=0x7f1144009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c16766e1d1b3318 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:08:15.055 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.045+0000 7f1152ffd700 1 -- 172.21.15.195:0/126807108 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1144011630 con 0x7f115406e850 2022-06-27T12:08:15.055 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.045+0000 7f1152ffd700 1 -- 172.21.15.195:0/126807108 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1144011790 con 0x7f115406e850 2022-06-27T12:08:15.055 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.045+0000 7f1152ffd700 1 -- 172.21.15.195:0/126807108 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1144011b30 con 0x7f115406e850 2022-06-27T12:08:15.056 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.046+0000 7f115a69d700 1 -- 172.21.15.195:0/126807108 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f115406e850 msgr2=0x7f11541570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:15.057 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.046+0000 7f115a69d700 1 --2- 172.21.15.195:0/126807108 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f115406e850 0x7f11541570a0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f11440097a0 tx=0x7f1144009060 comp rx=0 tx=0).stop 2022-06-27T12:08:15.058 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.046+0000 7f115a69d700 1 -- 172.21.15.195:0/126807108 shutdown_connections 2022-06-27T12:08:15.058 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.046+0000 7f115a69d700 1 --2- 172.21.15.195:0/126807108 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f115406e850 0x7f11541570a0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:15.058 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.046+0000 7f115a69d700 1 --2- 172.21.15.195:0/126807108 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f115406fef0 0x7f1154069de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:15.059 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.046+0000 7f115a69d700 1 -- 172.21.15.195:0/126807108 >> 172.21.15.195:0/126807108 conn(0x7f11540c6350 msgr2=0x7f11540c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:08:15.059 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.046+0000 7f115a69d700 1 -- 172.21.15.195:0/126807108 shutdown_connections 2022-06-27T12:08:15.059 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.046+0000 7f115a69d700 1 -- 172.21.15.195:0/126807108 wait complete. 2022-06-27T12:08:15.060 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.047+0000 7f115a69d700 1 Processor -- start 2022-06-27T12:08:15.060 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.047+0000 7f115a69d700 1 -- start start 2022-06-27T12:08:15.065 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.048+0000 7f115a69d700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f115406e850 0x7f1154150480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:15.066 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.048+0000 7f115a69d700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f11540c34a0 0x7f11540becf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:15.066 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.048+0000 7f115a69d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f115414af30 0x7f11540bf230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:15.066 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.048+0000 7f115a69d700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11540be860 con 0x7f115414af30 2022-06-27T12:08:15.067 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.048+0000 7f115a69d700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1154153530 con 0x7f11540c34a0 2022-06-27T12:08:15.067 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.048+0000 7f115a69d700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11540c89a0 con 0x7f115406e850 2022-06-27T12:08:15.067 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.048+0000 7f1158c3a700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f115414af30 0x7f11540bf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:15.068 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.048+0000 7f1158c3a700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f115414af30 0x7f11540bf230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:49076/0 (socket says 172.21.15.195:49076) 2022-06-27T12:08:15.070 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.048+0000 7f1158c3a700 1 -- 172.21.15.195:0/3905159172 learned_addr learned my addr 172.21.15.195:0/3905159172 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:08:15.071 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.048+0000 7f1153fff700 1 --2- 172.21.15.195:0/3905159172 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f115406e850 0x7f1154150480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:15.071 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.048+0000 7f11537fe700 1 --2- 172.21.15.195:0/3905159172 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f11540c34a0 0x7f11540becf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:15.072 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.048+0000 7f1158c3a700 1 -- 172.21.15.195:0/3905159172 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f115406e850 msgr2=0x7f1154150480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:15.072 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.048+0000 7f1158c3a700 1 --2- 172.21.15.195:0/3905159172 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f115406e850 0x7f1154150480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:15.073 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.049+0000 7f1158c3a700 1 -- 172.21.15.195:0/3905159172 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f11540c34a0 msgr2=0x7f11540becf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:15.073 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.049+0000 7f1158c3a700 1 --2- 172.21.15.195:0/3905159172 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f11540c34a0 0x7f11540becf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:15.074 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.049+0000 7f1158c3a700 1 -- 172.21.15.195:0/3905159172 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11440092e0 con 0x7f115414af30 2022-06-27T12:08:15.075 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.049+0000 7f1158c3a700 1 --2- 172.21.15.195:0/3905159172 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f115414af30 0x7f11540bf230 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f114c001460 tx=0x7f114c001650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:08:15.075 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.049+0000 7f113bfff700 1 -- 172.21.15.195:0/3905159172 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f114c005d70 con 0x7f115414af30 2022-06-27T12:08:15.076 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.049+0000 7f113bfff700 1 -- 172.21.15.195:0/3905159172 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f114c0070f0 con 0x7f115414af30 2022-06-27T12:08:15.077 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.049+0000 7f115a69d700 1 -- 172.21.15.195:0/3905159172 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1154153780 con 0x7f115414af30 2022-06-27T12:08:15.080 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.049+0000 7f113bfff700 1 -- 172.21.15.195:0/3905159172 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f114c007470 con 0x7f115414af30 2022-06-27T12:08:15.080 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.049+0000 7f115a69d700 1 -- 172.21.15.195:0/3905159172 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1154153d40 con 0x7f115414af30 2022-06-27T12:08:15.081 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.050+0000 7f115a69d700 1 -- 172.21.15.195:0/3905159172 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11540bea10 con 0x7f115414af30 2022-06-27T12:08:15.082 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.050+0000 7f113bfff700 1 -- 172.21.15.195:0/3905159172 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f114c010070 con 0x7f115414af30 2022-06-27T12:08:15.082 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.051+0000 7f113bfff700 1 --2- 172.21.15.195:0/3905159172 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1134063d00 0x7f11340661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:15.083 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.051+0000 7f113bfff700 1 -- 172.21.15.195:0/3905159172 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7148+0+0 (secure 0 0 0) 0x7f114c080790 con 0x7f115414af30 2022-06-27T12:08:15.083 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.051+0000 7f1153fff700 1 --2- 172.21.15.195:0/3905159172 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1134063d00 0x7f11340661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:15.084 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.051+0000 7f1153fff700 1 --2- 172.21.15.195:0/3905159172 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1134063d00 0x7f11340661b0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f1144009260 tx=0x7f1144017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:08:15.084 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.055+0000 7f113bfff700 1 -- 172.21.15.195:0/3905159172 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f11540bea10 con 0x7f115414af30 2022-06-27T12:08:15.282 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.282+0000 7f115a69d700 1 -- 172.21.15.195:0/3905159172 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f11540bea10 con 0x7f115414af30 2022-06-27T12:08:15.283 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.282+0000 7f113bfff700 1 -- 172.21.15.195:0/3905159172 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+5368 (secure 0 0 0) 0x7f11540bea10 con 0x7f115414af30 2022-06-27T12:08:15.284 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:08:15.284 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":75,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:08:10.380461+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6271},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6271":{"gid":6271,"name":"d","rank":0,"incarnation":73,"state":"up:rejoin","state_seq":16,"addr":"172.21.15.202:6835/1603307620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":1603307620},{"type":"v1","addr":"172.21.15.202:6835","nonce":1603307620}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:08:15.305 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.284+0000 7f115a69d700 1 -- 172.21.15.195:0/3905159172 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1134063d00 msgr2=0x7f11340661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:15.305 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.284+0000 7f115a69d700 1 --2- 172.21.15.195:0/3905159172 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1134063d00 0x7f11340661b0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f1144009260 tx=0x7f1144017000 comp rx=0 tx=0).stop 2022-06-27T12:08:15.305 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.284+0000 7f115a69d700 1 -- 172.21.15.195:0/3905159172 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f115414af30 msgr2=0x7f11540bf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:15.306 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.284+0000 7f115a69d700 1 --2- 172.21.15.195:0/3905159172 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f115414af30 0x7f11540bf230 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f114c001460 tx=0x7f114c001650 comp rx=0 tx=0).stop 2022-06-27T12:08:15.306 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.284+0000 7f115a69d700 1 -- 172.21.15.195:0/3905159172 shutdown_connections 2022-06-27T12:08:15.306 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.284+0000 7f115a69d700 1 --2- 172.21.15.195:0/3905159172 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1134063d00 0x7f11340661b0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:15.307 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.284+0000 7f115a69d700 1 --2- 172.21.15.195:0/3905159172 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f115414af30 0x7f11540bf230 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:15.307 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.284+0000 7f115a69d700 1 --2- 172.21.15.195:0/3905159172 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f115406e850 0x7f1154150480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:15.307 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.284+0000 7f115a69d700 1 --2- 172.21.15.195:0/3905159172 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f11540c34a0 0x7f11540becf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:15.307 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.284+0000 7f115a69d700 1 -- 172.21.15.195:0/3905159172 >> 172.21.15.195:0/3905159172 conn(0x7f11540c6350 msgr2=0x7f1154156500 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:08:15.308 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.284+0000 7f115a69d700 1 -- 172.21.15.195:0/3905159172 shutdown_connections 2022-06-27T12:08:15.308 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:15.284+0000 7f115a69d700 1 -- 172.21.15.195:0/3905159172 wait complete. 2022-06-27T12:08:15.308 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 75 2022-06-27T12:08:15.309 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 75, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-06-27T11:47:52.358095+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 75, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6271": { "addr": "172.21.15.202:6835/1603307620", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6834", "nonce": 1603307620, "type": "v2" }, { "addr": "172.21.15.202:6835", "nonce": 1603307620, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 6271, "incarnation": 73, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:rejoin", "state_seq": 16 } }, "last_failure": 0, "last_failure_osd_epoch": 59, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-06-27T12:08:10.380461+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6271 } } } ], "standbys": [ { "addr": "172.21.15.202:6837/3305006841", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6836", "nonce": 3305006841, "type": "v2" }, { "addr": "172.21.15.202:6837", "nonce": 3305006841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6835/2060709822", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6834", "nonce": 2060709822, "type": "v2" }, { "addr": "172.21.15.195:6835", "nonce": 2060709822, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6838/1598751652", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6836", "nonce": 1598751652, "type": "v2" }, { "addr": "172.21.15.195:6838", "nonce": 1598751652, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4275, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6839/1313800661", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6837", "nonce": 1313800661, "type": "v2" }, { "addr": "172.21.15.195:6839", "nonce": 1313800661, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4278, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-06-27T12:08:17.321 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:08:17.515 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.465+0000 7f036a7ea700 1 Processor -- start 2022-06-27T12:08:17.516 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.465+0000 7f036a7ea700 1 -- start start 2022-06-27T12:08:17.516 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.465+0000 7f036a7ea700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f036406e850 0x7f03641570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:17.517 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.465+0000 7f036a7ea700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f03640c34a0 0x7f03640c2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:17.517 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.465+0000 7f036a7ea700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f036406fba0 con 0x7f036406fef0 2022-06-27T12:08:17.517 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.465+0000 7f036a7ea700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03641575e0 con 0x7f036406e850 2022-06-27T12:08:17.531 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.465+0000 7f036a7ea700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f036406f5e0 con 0x7f03640c34a0 2022-06-27T12:08:17.531 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.466+0000 7f0363fff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f036406e850 0x7f03641570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:17.532 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.466+0000 7f0363fff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f036406e850 0x7f03641570a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:49082/0 (socket says 172.21.15.195:49082) 2022-06-27T12:08:17.532 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.466+0000 7f0363fff700 1 -- 172.21.15.195:0/4001111997 learned_addr learned my addr 172.21.15.195:0/4001111997 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:08:17.532 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.466+0000 7f0368d87700 1 --2- 172.21.15.195:0/4001111997 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f03640c34a0 0x7f03640c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:17.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.466+0000 7f0363fff700 1 -- 172.21.15.195:0/4001111997 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f03640c34a0 msgr2=0x7f03640c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:08:17.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.466+0000 7f0363fff700 1 --2- 172.21.15.195:0/4001111997 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f03640c34a0 0x7f03640c2920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:17.539 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.466+0000 7f0363fff700 1 -- 172.21.15.195:0/4001111997 >> v1:172.21.15.202:6789/0 conn(0x7f036406fef0 legacy=0x7f0364069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:17.540 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.466+0000 7f0363fff700 1 -- 172.21.15.195:0/4001111997 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f036406f8c0 con 0x7f036406e850 2022-06-27T12:08:17.540 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.466+0000 7f0363fff700 1 --2- 172.21.15.195:0/4001111997 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f036406e850 0x7f03641570a0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f0354009860 tx=0x7f0354009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ffb521359a823c8 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:08:17.541 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.466+0000 7f0362ffd700 1 -- 172.21.15.195:0/4001111997 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f035400b710 con 0x7f036406e850 2022-06-27T12:08:17.541 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.466+0000 7f0362ffd700 1 -- 172.21.15.195:0/4001111997 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f035400b870 con 0x7f036406e850 2022-06-27T12:08:17.542 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.466+0000 7f0362ffd700 1 -- 172.21.15.195:0/4001111997 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f035400bc30 con 0x7f036406e850 2022-06-27T12:08:17.542 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.467+0000 7f036a7ea700 1 -- 172.21.15.195:0/4001111997 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f036406e850 msgr2=0x7f03641570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:17.543 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.467+0000 7f036a7ea700 1 --2- 172.21.15.195:0/4001111997 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f036406e850 0x7f03641570a0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f0354009860 tx=0x7f0354009120 comp rx=0 tx=0).stop 2022-06-27T12:08:17.543 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.467+0000 7f036a7ea700 1 -- 172.21.15.195:0/4001111997 shutdown_connections 2022-06-27T12:08:17.544 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.467+0000 7f036a7ea700 1 --2- 172.21.15.195:0/4001111997 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f03640c34a0 0x7f03640c2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:17.544 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.467+0000 7f036a7ea700 1 --2- 172.21.15.195:0/4001111997 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f036406e850 0x7f03641570a0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:17.544 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.467+0000 7f036a7ea700 1 -- 172.21.15.195:0/4001111997 >> 172.21.15.195:0/4001111997 conn(0x7f03640c6350 msgr2=0x7f03640c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:08:17.545 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.467+0000 7f036a7ea700 1 -- 172.21.15.195:0/4001111997 shutdown_connections 2022-06-27T12:08:17.545 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.467+0000 7f036a7ea700 1 -- 172.21.15.195:0/4001111997 wait complete. 2022-06-27T12:08:17.546 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.468+0000 7f036a7ea700 1 Processor -- start 2022-06-27T12:08:17.547 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.469+0000 7f036a7ea700 1 -- start start 2022-06-27T12:08:17.547 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.469+0000 7f036a7ea700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f036406e850 0x7f0364163050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:17.548 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.469+0000 7f036a7ea700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f03640c34a0 0x7f03640b7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:17.548 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.469+0000 7f036a7ea700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f03640b6aa0 0x7f03641665e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:17.548 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.469+0000 7f036a7ea700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03640b6610 con 0x7f036406e850 2022-06-27T12:08:17.549 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.469+0000 7f036a7ea700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03640b6ef0 con 0x7f03640b6aa0 2022-06-27T12:08:17.549 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.469+0000 7f036a7ea700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0364167450 con 0x7f03640c34a0 2022-06-27T12:08:17.550 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.470+0000 7f0363fff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f036406e850 0x7f0364163050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:17.550 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.470+0000 7f0363fff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f036406e850 0x7f0364163050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:49088/0 (socket says 172.21.15.195:49088) 2022-06-27T12:08:17.551 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.470+0000 7f0363fff700 1 -- 172.21.15.195:0/2834548015 learned_addr learned my addr 172.21.15.195:0/2834548015 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:08:17.551 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.470+0000 7f0363fff700 1 -- 172.21.15.195:0/2834548015 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f03640c34a0 msgr2=0x7f03640b7ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:17.551 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.470+0000 7f03637fe700 1 --2- 172.21.15.195:0/2834548015 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f03640c34a0 0x7f03640b7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:17.552 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.470+0000 7f0363fff700 1 --2- 172.21.15.195:0/2834548015 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f03640c34a0 0x7f03640b7ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:17.552 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.470+0000 7f0363fff700 1 -- 172.21.15.195:0/2834548015 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f03640b6aa0 msgr2=0x7f03641665e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:17.558 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.470+0000 7f0363fff700 1 --2- 172.21.15.195:0/2834548015 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f03640b6aa0 0x7f03641665e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:17.559 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.470+0000 7f0363fff700 1 -- 172.21.15.195:0/2834548015 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03540093a0 con 0x7f036406e850 2022-06-27T12:08:17.559 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.470+0000 7f0363fff700 1 --2- 172.21.15.195:0/2834548015 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f036406e850 0x7f0364163050 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f03640650f0 tx=0x7f0354017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:08:17.560 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.470+0000 7f034bfff700 1 -- 172.21.15.195:0/2834548015 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0354021070 con 0x7f036406e850 2022-06-27T12:08:17.560 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.470+0000 7f034bfff700 1 -- 172.21.15.195:0/2834548015 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0354004050 con 0x7f036406e850 2022-06-27T12:08:17.561 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.470+0000 7f036a7ea700 1 -- 172.21.15.195:0/2834548015 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0364166e90 con 0x7f036406e850 2022-06-27T12:08:17.561 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.470+0000 7f034bfff700 1 -- 172.21.15.195:0/2834548015 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0354004410 con 0x7f036406e850 2022-06-27T12:08:17.561 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.470+0000 7f036a7ea700 1 -- 172.21.15.195:0/2834548015 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03640b5fc0 con 0x7f036406e850 2022-06-27T12:08:17.562 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.471+0000 7f036a7ea700 1 -- 172.21.15.195:0/2834548015 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03640722c0 con 0x7f036406e850 2022-06-27T12:08:17.562 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.471+0000 7f034bfff700 1 -- 172.21.15.195:0/2834548015 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f035402d030 con 0x7f036406e850 2022-06-27T12:08:17.563 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.472+0000 7f034bfff700 1 --2- 172.21.15.195:0/2834548015 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0344063d00 0x7f03440661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:17.563 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.472+0000 7f034bfff700 1 -- 172.21.15.195:0/2834548015 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7148+0+0 (secure 0 0 0) 0x7f0354084300 con 0x7f036406e850 2022-06-27T12:08:17.564 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.472+0000 7f03637fe700 1 --2- 172.21.15.195:0/2834548015 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0344063d00 0x7f03440661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:17.564 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.472+0000 7f03637fe700 1 --2- 172.21.15.195:0/2834548015 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0344063d00 0x7f03440661b0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f036406c1e0 tx=0x7f0358009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:08:17.565 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.476+0000 7f034bfff700 1 -- 172.21.15.195:0/2834548015 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f0354052ab0 con 0x7f036406e850 2022-06-27T12:08:17.702 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.699+0000 7f036a7ea700 1 -- 172.21.15.195:0/2834548015 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f036406f240 con 0x7f036406e850 2022-06-27T12:08:17.707 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.700+0000 7f034bfff700 1 -- 172.21.15.195:0/2834548015 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) v1 ==== 94+0+5374 (secure 0 0 0) 0x7f035402b090 con 0x7f036406e850 2022-06-27T12:08:17.708 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:08:17.708 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":76,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:08:16.106013+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6271},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6271":{"gid":6271,"name":"d","rank":0,"incarnation":73,"state":"up:clientreplay","state_seq":18,"addr":"172.21.15.202:6835/1603307620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":1603307620},{"type":"v1","addr":"172.21.15.202:6835","nonce":1603307620}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:08:17.718 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.702+0000 7f036a7ea700 1 -- 172.21.15.195:0/2834548015 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0344063d00 msgr2=0x7f03440661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:17.719 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.702+0000 7f036a7ea700 1 --2- 172.21.15.195:0/2834548015 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0344063d00 0x7f03440661b0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f036406c1e0 tx=0x7f0358009040 comp rx=0 tx=0).stop 2022-06-27T12:08:17.719 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.702+0000 7f036a7ea700 1 -- 172.21.15.195:0/2834548015 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f036406e850 msgr2=0x7f0364163050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:17.719 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.702+0000 7f036a7ea700 1 --2- 172.21.15.195:0/2834548015 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f036406e850 0x7f0364163050 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f03640650f0 tx=0x7f0354017000 comp rx=0 tx=0).stop 2022-06-27T12:08:17.754 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.702+0000 7f036a7ea700 1 -- 172.21.15.195:0/2834548015 shutdown_connections 2022-06-27T12:08:17.755 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.702+0000 7f036a7ea700 1 --2- 172.21.15.195:0/2834548015 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0344063d00 0x7f03440661b0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:17.755 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.702+0000 7f036a7ea700 1 --2- 172.21.15.195:0/2834548015 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f036406e850 0x7f0364163050 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:17.756 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.702+0000 7f036a7ea700 1 --2- 172.21.15.195:0/2834548015 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f03640b6aa0 0x7f03641665e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:17.756 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.702+0000 7f036a7ea700 1 --2- 172.21.15.195:0/2834548015 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f03640c34a0 0x7f03640b7ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:17.757 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.702+0000 7f036a7ea700 1 -- 172.21.15.195:0/2834548015 >> 172.21.15.195:0/2834548015 conn(0x7f03640c6350 msgr2=0x7f036414f660 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:08:17.757 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.702+0000 7f036a7ea700 1 -- 172.21.15.195:0/2834548015 shutdown_connections 2022-06-27T12:08:17.757 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:17.702+0000 7f036a7ea700 1 -- 172.21.15.195:0/2834548015 wait complete. 2022-06-27T12:08:17.758 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 76 2022-06-27T12:08:17.759 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:08:19.769 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:08:19.916 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:19.907+0000 7f20359dc700 1 Processor -- start 2022-06-27T12:08:19.917 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:19.908+0000 7f20359dc700 1 -- start start 2022-06-27T12:08:19.917 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:19.908+0000 7f20359dc700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f20300c5480 0x7f2030069a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:19.918 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:19.908+0000 7f20359dc700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f203006d280 0x7f2030150bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:19.918 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:19.908+0000 7f20359dc700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20300c5d80 con 0x7f20300c64f0 2022-06-27T12:08:19.919 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:19.908+0000 7f20359dc700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2030069f50 con 0x7f20300c5480 2022-06-27T12:08:19.919 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:19.908+0000 7f20359dc700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f203006b010 con 0x7f203006d280 2022-06-27T12:08:19.919 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:19.909+0000 7f202f7fe700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f203006d280 0x7f2030150bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:19.919 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:19.909+0000 7f202effd700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f20300c5480 0x7f2030069a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:19.920 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:19.909+0000 7f202effd700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f20300c5480 0x7f2030069a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3300/0 says I am v2:172.21.15.195:47222/0 (socket says 172.21.15.195:47222) 2022-06-27T12:08:19.920 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:19.909+0000 7f202f7fe700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f203006d280 0x7f2030150bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3301/0 says I am v2:172.21.15.195:41852/0 (socket says 172.21.15.195:41852) 2022-06-27T12:08:19.920 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:19.909+0000 7f202effd700 1 -- 172.21.15.195:0/569560522 learned_addr learned my addr 172.21.15.195:0/569560522 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:08:19.920 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:19.909+0000 7f202dffb700 1 -- 172.21.15.195:0/569560522 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2473066758 0 0) 0x7f20300c5d80 con 0x7f20300c64f0 2022-06-27T12:08:19.921 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:19.909+0000 7f202dffb700 1 -- 172.21.15.195:0/569560522 --> v1:172.21.15.202:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f201c003430 con 0x7f20300c64f0 2022-06-27T12:08:19.921 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:19.909+0000 7f202f7fe700 1 -- 172.21.15.195:0/569560522 >> v2:172.21.15.202:3300/0 conn(0x7f20300c5480 msgr2=0x7f2030069a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:08:19.921 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:19.909+0000 7f202f7fe700 1 --2- 172.21.15.195:0/569560522 >> v2:172.21.15.202:3300/0 conn(0x7f20300c5480 0x7f2030069a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:19.921 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:19.909+0000 7f202f7fe700 1 -- 172.21.15.195:0/569560522 >> v1:172.21.15.202:6789/0 conn(0x7f20300c64f0 legacy=0x7f20301504c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:19.922 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:19.909+0000 7f202f7fe700 1 -- 172.21.15.195:0/569560522 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20300c75e0 con 0x7f203006d280 2022-06-27T12:08:19.922 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:19.909+0000 7f202effd700 1 --2- 172.21.15.195:0/569560522 >> v2:172.21.15.202:3300/0 conn(0x7f20300c5480 0x7f2030069a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:08:19.922 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:19.909+0000 7f202f7fe700 1 --2- 172.21.15.195:0/569560522 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f203006d280 0x7f2030150bd0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2028009e60 tx=0x7f2028009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f7e4773391123d8 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:08:19.922 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:19.909+0000 7f202dffb700 1 -- 172.21.15.195:0/569560522 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f202800e070 con 0x7f203006d280 2022-06-27T12:08:19.923 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:19.910+0000 7f202dffb700 1 -- 172.21.15.195:0/569560522 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f202800bab0 con 0x7f203006d280 2022-06-27T12:08:19.923 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:19.910+0000 7f202dffb700 1 -- 172.21.15.195:0/569560522 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f202800be30 con 0x7f203006d280 2022-06-27T12:08:19.923 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:19.910+0000 7f20359dc700 1 -- 172.21.15.195:0/569560522 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f203006d280 msgr2=0x7f2030150bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:19.923 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:19.910+0000 7f20359dc700 1 --2- 172.21.15.195:0/569560522 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f203006d280 0x7f2030150bd0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2028009e60 tx=0x7f2028009780 comp rx=0 tx=0).stop 2022-06-27T12:08:19.923 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:19.910+0000 7f20359dc700 1 -- 172.21.15.195:0/569560522 shutdown_connections 2022-06-27T12:08:19.924 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:19.910+0000 7f20359dc700 1 --2- 172.21.15.195:0/569560522 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f203006d280 0x7f2030150bd0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:19.924 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:19.910+0000 7f20359dc700 1 --2- 172.21.15.195:0/569560522 >> v2:172.21.15.202:3300/0 conn(0x7f20300c5480 0x7f2030069a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:19.924 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:19.910+0000 7f20359dc700 1 -- 172.21.15.195:0/569560522 >> 172.21.15.195:0/569560522 conn(0x7f2030063af0 msgr2=0x7f2030063ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:08:19.924 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:19.910+0000 7f20359dc700 1 -- 172.21.15.195:0/569560522 shutdown_connections 2022-06-27T12:08:19.925 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:19.910+0000 7f20359dc700 1 -- 172.21.15.195:0/569560522 wait complete. 2022-06-27T12:08:19.925 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:19.911+0000 7f20359dc700 1 Processor -- start 2022-06-27T12:08:19.933 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:19.911+0000 7f20359dc700 1 -- start start 2022-06-27T12:08:19.934 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:19.912+0000 7f20359dc700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f203006d280 0x7f2030159cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:19.934 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:19.912+0000 7f20359dc700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f20300c5480 0x7f203015a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:19.934 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:19.912+0000 7f20359dc700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f20300c64f0 0x7f2030168020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:19.935 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:19.912+0000 7f20359dc700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20301528a0 con 0x7f20300c5480 2022-06-27T12:08:19.935 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:19.912+0000 7f20359dc700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2030167b90 con 0x7f20300c64f0 2022-06-27T12:08:19.935 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:19.912+0000 7f20359dc700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2030149220 con 0x7f203006d280 2022-06-27T12:08:19.935 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:19.912+0000 7f202e7fc700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f20300c5480 0x7f203015a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:19.935 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:19.912+0000 7f202e7fc700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f20300c5480 0x7f203015a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:49104/0 (socket says 172.21.15.195:49104) 2022-06-27T12:08:19.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:19.912+0000 7f202e7fc700 1 -- 172.21.15.195:0/2628886497 learned_addr learned my addr 172.21.15.195:0/2628886497 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:08:19.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:19.912+0000 7f202effd700 1 --2- 172.21.15.195:0/2628886497 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f203006d280 0x7f2030159cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:19.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:19.912+0000 7f202f7fe700 1 --2- 172.21.15.195:0/2628886497 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f20300c64f0 0x7f2030168020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:19.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:19.913+0000 7f202e7fc700 1 -- 172.21.15.195:0/2628886497 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f203006d280 msgr2=0x7f2030159cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:19.937 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:19.913+0000 7f202e7fc700 1 --2- 172.21.15.195:0/2628886497 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f203006d280 0x7f2030159cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:19.937 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:19.913+0000 7f202e7fc700 1 -- 172.21.15.195:0/2628886497 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f20300c64f0 msgr2=0x7f2030168020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:19.937 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:19.913+0000 7f202e7fc700 1 --2- 172.21.15.195:0/2628886497 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f20300c64f0 0x7f2030168020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:19.937 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:19.913+0000 7f202e7fc700 1 -- 172.21.15.195:0/2628886497 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20240093a0 con 0x7f20300c5480 2022-06-27T12:08:19.939 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:19.913+0000 7f202f7fe700 1 --2- 172.21.15.195:0/2628886497 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f20300c64f0 0x7f2030168020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:08:19.939 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:19.913+0000 7f202effd700 1 --2- 172.21.15.195:0/2628886497 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f203006d280 0x7f2030159cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:08:19.940 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:19.913+0000 7f202e7fc700 1 --2- 172.21.15.195:0/2628886497 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f20300c5480 0x7f203015a210 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f202000c440 tx=0x7f2020007c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:08:19.940 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:19.913+0000 7f2016ffd700 1 -- 172.21.15.195:0/2628886497 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20200050d0 con 0x7f20300c5480 2022-06-27T12:08:19.941 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:19.913+0000 7f2016ffd700 1 -- 172.21.15.195:0/2628886497 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2020005230 con 0x7f20300c5480 2022-06-27T12:08:19.941 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:19.913+0000 7f20359dc700 1 -- 172.21.15.195:0/2628886497 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2028009a20 con 0x7f20300c5480 2022-06-27T12:08:19.942 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:19.913+0000 7f2016ffd700 1 -- 172.21.15.195:0/2628886497 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20200117e0 con 0x7f20300c5480 2022-06-27T12:08:19.942 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:19.913+0000 7f20359dc700 1 -- 172.21.15.195:0/2628886497 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20301522e0 con 0x7f20300c5480 2022-06-27T12:08:19.942 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:19.914+0000 7f20359dc700 1 -- 172.21.15.195:0/2628886497 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20300722c0 con 0x7f20300c5480 2022-06-27T12:08:19.943 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:19.915+0000 7f2016ffd700 1 -- 172.21.15.195:0/2628886497 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f2020019030 con 0x7f20300c5480 2022-06-27T12:08:19.943 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:19.915+0000 7f2016ffd700 1 --2- 172.21.15.195:0/2628886497 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2010063d00 0x7f20100661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:19.944 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:19.915+0000 7f2016ffd700 1 -- 172.21.15.195:0/2628886497 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7148+0+0 (secure 0 0 0) 0x7f20200805d0 con 0x7f20300c5480 2022-06-27T12:08:19.944 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:19.915+0000 7f202effd700 1 --2- 172.21.15.195:0/2628886497 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2010063d00 0x7f20100661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:19.944 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:19.916+0000 7f202effd700 1 --2- 172.21.15.195:0/2628886497 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2010063d00 0x7f20100661b0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f2024006c00 tx=0x7f2024008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:08:19.946 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:19.919+0000 7f2016ffd700 1 -- 172.21.15.195:0/2628886497 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f202004ee00 con 0x7f20300c5480 2022-06-27T12:08:20.147 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:20.147+0000 7f20359dc700 1 -- 172.21.15.195:0/2628886497 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f20300c78f0 con 0x7f20300c5480 2022-06-27T12:08:20.148 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:20.148+0000 7f2016ffd700 1 -- 172.21.15.195:0/2628886497 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) v1 ==== 94+0+5374 (secure 0 0 0) 0x7f2020052420 con 0x7f20300c5480 2022-06-27T12:08:20.148 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:08:20.149 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":76,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:08:16.106013+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6271},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6271":{"gid":6271,"name":"d","rank":0,"incarnation":73,"state":"up:clientreplay","state_seq":18,"addr":"172.21.15.202:6835/1603307620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":1603307620},{"type":"v1","addr":"172.21.15.202:6835","nonce":1603307620}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:08:20.157 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:20.150+0000 7f20359dc700 1 -- 172.21.15.195:0/2628886497 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2010063d00 msgr2=0x7f20100661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:20.157 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:20.150+0000 7f20359dc700 1 --2- 172.21.15.195:0/2628886497 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2010063d00 0x7f20100661b0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f2024006c00 tx=0x7f2024008040 comp rx=0 tx=0).stop 2022-06-27T12:08:20.158 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:20.150+0000 7f20359dc700 1 -- 172.21.15.195:0/2628886497 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f20300c5480 msgr2=0x7f203015a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:20.158 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:20.150+0000 7f20359dc700 1 --2- 172.21.15.195:0/2628886497 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f20300c5480 0x7f203015a210 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f202000c440 tx=0x7f2020007c70 comp rx=0 tx=0).stop 2022-06-27T12:08:20.158 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:20.150+0000 7f20359dc700 1 -- 172.21.15.195:0/2628886497 shutdown_connections 2022-06-27T12:08:20.159 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:20.150+0000 7f20359dc700 1 --2- 172.21.15.195:0/2628886497 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2010063d00 0x7f20100661b0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:20.159 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:20.150+0000 7f20359dc700 1 --2- 172.21.15.195:0/2628886497 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f20300c64f0 0x7f2030168020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:20.159 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:20.150+0000 7f20359dc700 1 --2- 172.21.15.195:0/2628886497 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f203006d280 0x7f2030159cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:20.159 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:20.150+0000 7f20359dc700 1 --2- 172.21.15.195:0/2628886497 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f20300c5480 0x7f203015a210 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:20.160 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:20.150+0000 7f20359dc700 1 -- 172.21.15.195:0/2628886497 >> 172.21.15.195:0/2628886497 conn(0x7f2030063af0 msgr2=0x7f20300cb2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:08:20.160 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:20.150+0000 7f20359dc700 1 -- 172.21.15.195:0/2628886497 shutdown_connections 2022-06-27T12:08:20.160 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:20.150+0000 7f20359dc700 1 -- 172.21.15.195:0/2628886497 wait complete. 2022-06-27T12:08:20.160 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 76 2022-06-27T12:08:20.163 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:08:22.164 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:08:22.302 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.292+0000 7f4a8ecc0700 1 Processor -- start 2022-06-27T12:08:22.302 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.292+0000 7f4a8ecc0700 1 -- start start 2022-06-27T12:08:22.303 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.292+0000 7f4a8ecc0700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f4a88150580 0x7f4a88156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:22.303 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.292+0000 7f4a8ecc0700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f4a8806f620 0x7f4a8806d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:22.303 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.293+0000 7f4a8ecc0700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a880c76b0 con 0x7f4a88150950 2022-06-27T12:08:22.303 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.293+0000 7f4a8ecc0700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a88157430 con 0x7f4a8806f620 2022-06-27T12:08:22.304 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.293+0000 7f4a8ecc0700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a880c8250 con 0x7f4a88150580 2022-06-27T12:08:22.304 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.293+0000 7f4a8d25d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f4a8806f620 0x7f4a8806d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:22.304 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.293+0000 7f4a8d25d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f4a8806f620 0x7f4a8806d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:49138/0 (socket says 172.21.15.195:49138) 2022-06-27T12:08:22.304 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.293+0000 7f4a8d25d700 1 -- 172.21.15.195:0/2783793137 learned_addr learned my addr 172.21.15.195:0/2783793137 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:08:22.305 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.293+0000 7f4a8ca5c700 1 --2- 172.21.15.195:0/2783793137 >> v2:172.21.15.202:3300/0 conn(0x7f4a88150580 0x7f4a88156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:22.305 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.293+0000 7f4a8d25d700 1 -- 172.21.15.195:0/2783793137 >> v2:172.21.15.202:3300/0 conn(0x7f4a88150580 msgr2=0x7f4a88156ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:08:22.305 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.293+0000 7f4a8d25d700 1 --2- 172.21.15.195:0/2783793137 >> v2:172.21.15.202:3300/0 conn(0x7f4a88150580 0x7f4a88156ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:22.305 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.293+0000 7f4a8d25d700 1 -- 172.21.15.195:0/2783793137 >> v1:172.21.15.202:6789/0 conn(0x7f4a88150950 legacy=0x7f4a880c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:22.306 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.293+0000 7f4a8d25d700 1 -- 172.21.15.195:0/2783793137 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a880bad70 con 0x7f4a8806f620 2022-06-27T12:08:22.306 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.293+0000 7f4a8d25d700 1 --2- 172.21.15.195:0/2783793137 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4a8806f620 0x7f4a8806d6a0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f4a880c8470 tx=0x7f4a80009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5d882e33b93bc84 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:08:22.307 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.293+0000 7f4a877fe700 1 -- 172.21.15.195:0/2783793137 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a8000e070 con 0x7f4a8806f620 2022-06-27T12:08:22.307 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.293+0000 7f4a877fe700 1 -- 172.21.15.195:0/2783793137 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a8000ba20 con 0x7f4a8806f620 2022-06-27T12:08:22.308 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.293+0000 7f4a877fe700 1 -- 172.21.15.195:0/2783793137 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a800092e0 con 0x7f4a8806f620 2022-06-27T12:08:22.308 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.294+0000 7f4a8ecc0700 1 -- 172.21.15.195:0/2783793137 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4a8806f620 msgr2=0x7f4a8806d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:22.309 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.294+0000 7f4a8ecc0700 1 --2- 172.21.15.195:0/2783793137 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4a8806f620 0x7f4a8806d6a0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f4a880c8470 tx=0x7f4a80009840 comp rx=0 tx=0).stop 2022-06-27T12:08:22.309 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.294+0000 7f4a8ecc0700 1 -- 172.21.15.195:0/2783793137 shutdown_connections 2022-06-27T12:08:22.309 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.294+0000 7f4a8ecc0700 1 --2- 172.21.15.195:0/2783793137 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4a8806f620 0x7f4a8806d6a0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:22.309 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.294+0000 7f4a8ecc0700 1 --2- 172.21.15.195:0/2783793137 >> v2:172.21.15.202:3300/0 conn(0x7f4a88150580 0x7f4a88156ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:22.310 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.294+0000 7f4a8ecc0700 1 -- 172.21.15.195:0/2783793137 >> 172.21.15.195:0/2783793137 conn(0x7f4a88063af0 msgr2=0x7f4a88063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:08:22.310 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.294+0000 7f4a8ecc0700 1 -- 172.21.15.195:0/2783793137 shutdown_connections 2022-06-27T12:08:22.310 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.294+0000 7f4a8ecc0700 1 -- 172.21.15.195:0/2783793137 wait complete. 2022-06-27T12:08:22.310 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.295+0000 7f4a8ecc0700 1 Processor -- start 2022-06-27T12:08:22.311 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.295+0000 7f4a8ecc0700 1 -- start start 2022-06-27T12:08:22.311 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.296+0000 7f4a8ecc0700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4a8806f620 0x7f4a880c1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:22.311 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.296+0000 7f4a8ecc0700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4a88150580 0x7f4a88159a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:22.311 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.296+0000 7f4a8ecc0700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4a88150950 0x7f4a880c2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:22.312 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.296+0000 7f4a8ecc0700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a88166bc0 con 0x7f4a8806f620 2022-06-27T12:08:22.312 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.296+0000 7f4a8ecc0700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a88156300 con 0x7f4a88150950 2022-06-27T12:08:22.312 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.296+0000 7f4a8ecc0700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a880c5ca0 con 0x7f4a88150580 2022-06-27T12:08:22.335 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.296+0000 7f4a8ca5c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4a8806f620 0x7f4a880c1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:22.336 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.296+0000 7f4a8ca5c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4a8806f620 0x7f4a880c1ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:49140/0 (socket says 172.21.15.195:49140) 2022-06-27T12:08:22.336 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.296+0000 7f4a8ca5c700 1 -- 172.21.15.195:0/3497676976 learned_addr learned my addr 172.21.15.195:0/3497676976 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:08:22.348 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.296+0000 7f4a8d25d700 1 --2- 172.21.15.195:0/3497676976 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4a88150950 0x7f4a880c2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:22.348 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.296+0000 7f4a87fff700 1 --2- 172.21.15.195:0/3497676976 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4a88150580 0x7f4a88159a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:22.349 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.296+0000 7f4a8ca5c700 1 -- 172.21.15.195:0/3497676976 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4a88150580 msgr2=0x7f4a88159a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:22.349 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.296+0000 7f4a8ca5c700 1 --2- 172.21.15.195:0/3497676976 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4a88150580 0x7f4a88159a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:22.349 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.296+0000 7f4a8ca5c700 1 -- 172.21.15.195:0/3497676976 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4a88150950 msgr2=0x7f4a880c2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:22.350 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.296+0000 7f4a8ca5c700 1 --2- 172.21.15.195:0/3497676976 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4a88150950 0x7f4a880c2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:22.350 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.296+0000 7f4a8ca5c700 1 -- 172.21.15.195:0/3497676976 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a80009ae0 con 0x7f4a8806f620 2022-06-27T12:08:22.350 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.296+0000 7f4a8ca5c700 1 --2- 172.21.15.195:0/3497676976 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4a8806f620 0x7f4a880c1ec0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f4a7c00e970 tx=0x7f4a7c009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:08:22.350 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.296+0000 7f4a84ff9700 1 -- 172.21.15.195:0/3497676976 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a7c013680 con 0x7f4a8806f620 2022-06-27T12:08:22.351 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.297+0000 7f4a84ff9700 1 -- 172.21.15.195:0/3497676976 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a7c0137e0 con 0x7f4a8806f620 2022-06-27T12:08:22.351 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.297+0000 7f4a8ecc0700 1 -- 172.21.15.195:0/3497676976 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a88167080 con 0x7f4a8806f620 2022-06-27T12:08:22.351 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.297+0000 7f4a84ff9700 1 -- 172.21.15.195:0/3497676976 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a7c013b60 con 0x7f4a8806f620 2022-06-27T12:08:22.351 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.297+0000 7f4a8ecc0700 1 -- 172.21.15.195:0/3497676976 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a88166da0 con 0x7f4a8806f620 2022-06-27T12:08:22.352 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.297+0000 7f4a8ecc0700 1 -- 172.21.15.195:0/3497676976 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a880722c0 con 0x7f4a8806f620 2022-06-27T12:08:22.352 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.298+0000 7f4a84ff9700 1 -- 172.21.15.195:0/3497676976 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f4a7c023070 con 0x7f4a8806f620 2022-06-27T12:08:22.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.298+0000 7f4a84ff9700 1 --2- 172.21.15.195:0/3497676976 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4a70063d00 0x7f4a700661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:22.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.298+0000 7f4a84ff9700 1 -- 172.21.15.195:0/3497676976 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7148+0+0 (secure 0 0 0) 0x7f4a7c080f20 con 0x7f4a8806f620 2022-06-27T12:08:22.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.298+0000 7f4a87fff700 1 --2- 172.21.15.195:0/3497676976 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4a70063d00 0x7f4a700661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:22.372 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.299+0000 7f4a87fff700 1 --2- 172.21.15.195:0/3497676976 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4a70063d00 0x7f4a700661b0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f4a880be3c0 tx=0x7f4a74009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:08:22.372 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.303+0000 7f4a84ff9700 1 -- 172.21.15.195:0/3497676976 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f4a7c04f670 con 0x7f4a8806f620 2022-06-27T12:08:22.534 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.533+0000 7f4a8ecc0700 1 -- 172.21.15.195:0/3497676976 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a8806fe20 con 0x7f4a8806f620 2022-06-27T12:08:22.535 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.534+0000 7f4a84ff9700 1 -- 172.21.15.195:0/3497676976 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) v1 ==== 94+0+5374 (secure 0 0 0) 0x7f4a7c017090 con 0x7f4a8806f620 2022-06-27T12:08:22.536 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:08:22.536 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":76,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:08:16.106013+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6271},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6271":{"gid":6271,"name":"d","rank":0,"incarnation":73,"state":"up:clientreplay","state_seq":18,"addr":"172.21.15.202:6835/1603307620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":1603307620},{"type":"v1","addr":"172.21.15.202:6835","nonce":1603307620}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:08:22.543 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.536+0000 7f4a8ecc0700 1 -- 172.21.15.195:0/3497676976 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4a70063d00 msgr2=0x7f4a700661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:22.543 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.536+0000 7f4a8ecc0700 1 --2- 172.21.15.195:0/3497676976 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4a70063d00 0x7f4a700661b0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f4a880be3c0 tx=0x7f4a74009040 comp rx=0 tx=0).stop 2022-06-27T12:08:22.544 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.536+0000 7f4a8ecc0700 1 -- 172.21.15.195:0/3497676976 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4a8806f620 msgr2=0x7f4a880c1ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:22.547 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.536+0000 7f4a8ecc0700 1 --2- 172.21.15.195:0/3497676976 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4a8806f620 0x7f4a880c1ec0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f4a7c00e970 tx=0x7f4a7c009060 comp rx=0 tx=0).stop 2022-06-27T12:08:22.547 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.536+0000 7f4a8ecc0700 1 -- 172.21.15.195:0/3497676976 shutdown_connections 2022-06-27T12:08:22.547 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.536+0000 7f4a8ecc0700 1 --2- 172.21.15.195:0/3497676976 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4a70063d00 0x7f4a700661b0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:22.548 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.536+0000 7f4a8ecc0700 1 --2- 172.21.15.195:0/3497676976 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4a8806f620 0x7f4a880c1ec0 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:22.548 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.536+0000 7f4a8ecc0700 1 --2- 172.21.15.195:0/3497676976 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4a88150950 0x7f4a880c2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:22.549 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.536+0000 7f4a8ecc0700 1 --2- 172.21.15.195:0/3497676976 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4a88150580 0x7f4a88159a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:22.549 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.536+0000 7f4a8ecc0700 1 -- 172.21.15.195:0/3497676976 >> 172.21.15.195:0/3497676976 conn(0x7f4a88063af0 msgr2=0x7f4a8814d580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:08:22.549 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.536+0000 7f4a8ecc0700 1 -- 172.21.15.195:0/3497676976 shutdown_connections 2022-06-27T12:08:22.549 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:22.536+0000 7f4a8ecc0700 1 -- 172.21.15.195:0/3497676976 wait complete. 2022-06-27T12:08:22.549 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 76 2022-06-27T12:08:22.560 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:08:24.562 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:08:24.731 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.698+0000 7f5765ba7700 1 Processor -- start 2022-06-27T12:08:24.732 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.698+0000 7f5765ba7700 1 -- start start 2022-06-27T12:08:24.733 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.699+0000 7f5765ba7700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f576006fef0 0x7f57601570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:24.733 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.699+0000 7f5765ba7700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f57600c34a0 0x7f57600c2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:24.734 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.699+0000 7f5765ba7700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f576006fba0 con 0x7f576006e850 2022-06-27T12:08:24.734 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.699+0000 7f5765ba7700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57601575e0 con 0x7f57600c34a0 2022-06-27T12:08:24.735 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.699+0000 7f5765ba7700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f576006f5e0 con 0x7f576006fef0 2022-06-27T12:08:24.735 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.699+0000 7f575ffff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f57600c34a0 0x7f57600c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:24.736 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.699+0000 7f575ffff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f57600c34a0 0x7f57600c2920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:49156/0 (socket says 172.21.15.195:49156) 2022-06-27T12:08:24.736 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.699+0000 7f575ffff700 1 -- 172.21.15.195:0/4281252642 learned_addr learned my addr 172.21.15.195:0/4281252642 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:08:24.737 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.699+0000 7f575effd700 1 --2- 172.21.15.195:0/4281252642 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f576006fef0 0x7f57601570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:24.737 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.699+0000 7f575ffff700 1 -- 172.21.15.195:0/4281252642 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f576006fef0 msgr2=0x7f57601570a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:08:24.744 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.699+0000 7f575ffff700 1 --2- 172.21.15.195:0/4281252642 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f576006fef0 0x7f57601570a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:24.745 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.699+0000 7f575ffff700 1 -- 172.21.15.195:0/4281252642 >> v1:172.21.15.202:6789/0 conn(0x7f576006e850 legacy=0x7f5760069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:24.745 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.699+0000 7f575ffff700 1 -- 172.21.15.195:0/4281252642 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f576006f8c0 con 0x7f57600c34a0 2022-06-27T12:08:24.746 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.700+0000 7f575ffff700 1 --2- 172.21.15.195:0/4281252642 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f57600c34a0 0x7f57600c2920 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f5758009f20 tx=0x7f5758009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=399fe1cd78c4f15 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:08:24.746 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.700+0000 7f575e7fc700 1 -- 172.21.15.195:0/4281252642 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f575800e070 con 0x7f57600c34a0 2022-06-27T12:08:24.747 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.700+0000 7f575e7fc700 1 -- 172.21.15.195:0/4281252642 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f575800ba60 con 0x7f57600c34a0 2022-06-27T12:08:24.747 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.700+0000 7f575e7fc700 1 -- 172.21.15.195:0/4281252642 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57580092e0 con 0x7f57600c34a0 2022-06-27T12:08:24.747 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.700+0000 7f5765ba7700 1 -- 172.21.15.195:0/4281252642 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f57600c34a0 msgr2=0x7f57600c2920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:24.748 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.700+0000 7f5765ba7700 1 --2- 172.21.15.195:0/4281252642 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f57600c34a0 0x7f57600c2920 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f5758009f20 tx=0x7f5758009840 comp rx=0 tx=0).stop 2022-06-27T12:08:24.749 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.700+0000 7f5765ba7700 1 -- 172.21.15.195:0/4281252642 shutdown_connections 2022-06-27T12:08:24.749 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.700+0000 7f5765ba7700 1 --2- 172.21.15.195:0/4281252642 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f57600c34a0 0x7f57600c2920 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:24.766 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.700+0000 7f5765ba7700 1 --2- 172.21.15.195:0/4281252642 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f576006fef0 0x7f57601570a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:24.767 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.700+0000 7f5765ba7700 1 -- 172.21.15.195:0/4281252642 >> 172.21.15.195:0/4281252642 conn(0x7f57600c6350 msgr2=0x7f57600c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:08:24.767 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.700+0000 7f5765ba7700 1 -- 172.21.15.195:0/4281252642 shutdown_connections 2022-06-27T12:08:24.768 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.700+0000 7f5765ba7700 1 -- 172.21.15.195:0/4281252642 wait complete. 2022-06-27T12:08:24.768 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.702+0000 7f5765ba7700 1 Processor -- start 2022-06-27T12:08:24.769 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.702+0000 7f5765ba7700 1 -- start start 2022-06-27T12:08:24.769 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.702+0000 7f5765ba7700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f576006e850 0x7f5760168170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:24.770 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.702+0000 7f5765ba7700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f57600c34a0 0x7f5760155670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:24.770 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.702+0000 7f5765ba7700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5760151300 0x7f576014b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:24.770 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.702+0000 7f5765ba7700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57600bddb0 con 0x7f5760151300 2022-06-27T12:08:24.771 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.702+0000 7f5765ba7700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57601569f0 con 0x7f57600c34a0 2022-06-27T12:08:24.771 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.702+0000 7f5765ba7700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5760161780 con 0x7f576006e850 2022-06-27T12:08:24.772 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.702+0000 7f575ffff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5760151300 0x7f576014b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:24.772 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.702+0000 7f575ffff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5760151300 0x7f576014b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:49162/0 (socket says 172.21.15.195:49162) 2022-06-27T12:08:24.773 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.702+0000 7f575ffff700 1 -- 172.21.15.195:0/1862001669 learned_addr learned my addr 172.21.15.195:0/1862001669 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:08:24.773 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.703+0000 7f575effd700 1 --2- 172.21.15.195:0/1862001669 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f57600c34a0 0x7f5760155670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:24.773 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.703+0000 7f575ffff700 1 -- 172.21.15.195:0/1862001669 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f576006e850 msgr2=0x7f5760168170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:24.774 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.703+0000 7f575f7fe700 1 --2- 172.21.15.195:0/1862001669 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f576006e850 0x7f5760168170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:24.774 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.703+0000 7f575ffff700 1 --2- 172.21.15.195:0/1862001669 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f576006e850 0x7f5760168170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:24.775 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.703+0000 7f575ffff700 1 -- 172.21.15.195:0/1862001669 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f57600c34a0 msgr2=0x7f5760155670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:24.775 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.703+0000 7f575ffff700 1 --2- 172.21.15.195:0/1862001669 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f57600c34a0 0x7f5760155670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:24.798 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.703+0000 7f575ffff700 1 -- 172.21.15.195:0/1862001669 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5758009ae0 con 0x7f5760151300 2022-06-27T12:08:24.799 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.703+0000 7f575effd700 1 --2- 172.21.15.195:0/1862001669 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f57600c34a0 0x7f5760155670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T12:08:24.808 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.703+0000 7f575f7fe700 1 --2- 172.21.15.195:0/1862001669 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f576006e850 0x7f5760168170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T12:08:24.808 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.703+0000 7f575ffff700 1 --2- 172.21.15.195:0/1862001669 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5760151300 0x7f576014b9b0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f575800b610 tx=0x7f5758018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:08:24.808 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.703+0000 7f57477fe700 1 -- 172.21.15.195:0/1862001669 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57580094e0 con 0x7f5760151300 2022-06-27T12:08:24.809 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.703+0000 7f57477fe700 1 -- 172.21.15.195:0/1862001669 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57580092e0 con 0x7f5760151300 2022-06-27T12:08:24.809 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.703+0000 7f5765ba7700 1 -- 172.21.15.195:0/1862001669 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5760161a60 con 0x7f5760151300 2022-06-27T12:08:24.810 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.703+0000 7f57477fe700 1 -- 172.21.15.195:0/1862001669 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5758007a70 con 0x7f5760151300 2022-06-27T12:08:24.810 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.703+0000 7f5765ba7700 1 -- 172.21.15.195:0/1862001669 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5760167970 con 0x7f5760151300 2022-06-27T12:08:24.811 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.704+0000 7f5765ba7700 1 -- 172.21.15.195:0/1862001669 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57600722c0 con 0x7f5760151300 2022-06-27T12:08:24.811 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.704+0000 7f57477fe700 1 -- 172.21.15.195:0/1862001669 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f5758004040 con 0x7f5760151300 2022-06-27T12:08:24.811 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.705+0000 7f57477fe700 1 --2- 172.21.15.195:0/1862001669 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5740063d00 0x7f57400661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:24.812 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.705+0000 7f57477fe700 1 -- 172.21.15.195:0/1862001669 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7148+0+0 (secure 0 0 0) 0x7f5758082c50 con 0x7f5760151300 2022-06-27T12:08:24.812 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.705+0000 7f575f7fe700 1 --2- 172.21.15.195:0/1862001669 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5740063d00 0x7f57400661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:24.813 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.705+0000 7f575f7fe700 1 --2- 172.21.15.195:0/1862001669 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5740063d00 0x7f57400661b0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f5754001650 tx=0x7f5754009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:08:24.813 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.709+0000 7f57477fe700 1 -- 172.21.15.195:0/1862001669 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f5758051400 con 0x7f5760151300 2022-06-27T12:08:24.945 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.939+0000 7f5765ba7700 1 -- 172.21.15.195:0/1862001669 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f576006f240 con 0x7f5760151300 2022-06-27T12:08:24.945 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.940+0000 7f57477fe700 1 -- 172.21.15.195:0/1862001669 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+5368 (secure 0 0 0) 0x7f575801ce60 con 0x7f5760151300 2022-06-27T12:08:24.947 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:08:24.951 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:08:23.133338+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6271},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6271":{"gid":6271,"name":"d","rank":0,"incarnation":73,"state":"up:active","state_seq":20,"addr":"172.21.15.202:6835/1603307620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":1603307620},{"type":"v1","addr":"172.21.15.202:6835","nonce":1603307620}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:08:24.987 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.942+0000 7f5765ba7700 1 -- 172.21.15.195:0/1862001669 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5740063d00 msgr2=0x7f57400661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:24.987 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.942+0000 7f5765ba7700 1 --2- 172.21.15.195:0/1862001669 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5740063d00 0x7f57400661b0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f5754001650 tx=0x7f5754009040 comp rx=0 tx=0).stop 2022-06-27T12:08:24.987 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.942+0000 7f5765ba7700 1 -- 172.21.15.195:0/1862001669 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5760151300 msgr2=0x7f576014b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:24.988 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.942+0000 7f5765ba7700 1 --2- 172.21.15.195:0/1862001669 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5760151300 0x7f576014b9b0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f575800b610 tx=0x7f5758018000 comp rx=0 tx=0).stop 2022-06-27T12:08:24.988 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.942+0000 7f5765ba7700 1 -- 172.21.15.195:0/1862001669 shutdown_connections 2022-06-27T12:08:24.988 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.942+0000 7f5765ba7700 1 --2- 172.21.15.195:0/1862001669 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5740063d00 0x7f57400661b0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:24.988 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.942+0000 7f5765ba7700 1 --2- 172.21.15.195:0/1862001669 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5760151300 0x7f576014b9b0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:24.989 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.942+0000 7f5765ba7700 1 --2- 172.21.15.195:0/1862001669 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f576006e850 0x7f5760168170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:24.989 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.942+0000 7f5765ba7700 1 --2- 172.21.15.195:0/1862001669 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f57600c34a0 0x7f5760155670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:24.989 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.942+0000 7f5765ba7700 1 -- 172.21.15.195:0/1862001669 >> 172.21.15.195:0/1862001669 conn(0x7f57600c6350 msgr2=0x7f57600c2cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:08:24.989 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.942+0000 7f5765ba7700 1 -- 172.21.15.195:0/1862001669 shutdown_connections 2022-06-27T12:08:24.998 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:24.942+0000 7f5765ba7700 1 -- 172.21.15.195:0/1862001669 wait complete. 2022-06-27T12:08:24.998 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 77 2022-06-27T12:08:24.999 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d has gained rank=0, replacing gid=5591 2022-06-27T12:08:24.999 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 0 secs before reviving mds.b 2022-06-27T12:08:25.000 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2022-06-27T12:08:25.000 INFO:tasks.ceph.mds.b:Restarting daemon 2022-06-27T12:08:25.001 DEBUG:teuthology.orchestra.run.smithi202:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-06-27T12:08:25.007 INFO:tasks.ceph.mds.b:Started 2022-06-27T12:08:25.007 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:08:25.099 INFO:tasks.ceph.mds.b.smithi202.stdout:starting mds.b at 2022-06-27T12:08:25.135 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.130+0000 7f9a69de1700 1 Processor -- start 2022-06-27T12:08:25.135 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.130+0000 7f9a69de1700 1 -- start start 2022-06-27T12:08:25.136 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.131+0000 7f9a69de1700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f9a64150580 0x7f9a64156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:25.136 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.131+0000 7f9a69de1700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f9a6406f620 0x7f9a6406d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:25.136 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.131+0000 7f9a69de1700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a640c76b0 con 0x7f9a64150950 2022-06-27T12:08:25.136 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.131+0000 7f9a69de1700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a64157430 con 0x7f9a6406f620 2022-06-27T12:08:25.137 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.131+0000 7f9a69de1700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a640c8250 con 0x7f9a64150580 2022-06-27T12:08:25.137 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.131+0000 7f9a63fff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f9a6406f620 0x7f9a6406d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:25.137 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.131+0000 7f9a62ffd700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f9a64150950 0x7f9a640c7b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:55356/0 (socket says 172.21.15.195:55356) 2022-06-27T12:08:25.137 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.131+0000 7f9a62ffd700 1 -- 172.21.15.195:0/1656969371 learned_addr learned my addr 172.21.15.195:0/1656969371 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:08:25.138 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.131+0000 7f9a637fe700 1 --2- 172.21.15.195:0/1656969371 >> v2:172.21.15.202:3300/0 conn(0x7f9a64150580 0x7f9a64156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:25.158 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.131+0000 7f9a63fff700 1 -- 172.21.15.195:0/1656969371 >> v2:172.21.15.202:3300/0 conn(0x7f9a64150580 msgr2=0x7f9a64156ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:08:25.159 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.131+0000 7f9a63fff700 1 --2- 172.21.15.195:0/1656969371 >> v2:172.21.15.202:3300/0 conn(0x7f9a64150580 0x7f9a64156ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:25.160 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.131+0000 7f9a63fff700 1 -- 172.21.15.195:0/1656969371 >> v1:172.21.15.195:6789/0 conn(0x7f9a64150950 legacy=0x7f9a640c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:25.167 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.131+0000 7f9a63fff700 1 -- 172.21.15.195:0/1656969371 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a640bad70 con 0x7f9a6406f620 2022-06-27T12:08:25.167 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.131+0000 7f9a627fc700 1 -- 172.21.15.195:0/1656969371 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2238382321 0 0) 0x7f9a640c76b0 con 0x7f9a64150950 2022-06-27T12:08:25.168 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.132+0000 7f9a63fff700 1 --2- 172.21.15.195:0/1656969371 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9a6406f620 0x7f9a6406d6a0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f9a640c8470 tx=0x7f9a5c009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30fe8bfdeb3826fa server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:08:25.168 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.132+0000 7f9a627fc700 1 -- 172.21.15.195:0/1656969371 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a5c011630 con 0x7f9a6406f620 2022-06-27T12:08:25.168 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.132+0000 7f9a627fc700 1 -- 172.21.15.195:0/1656969371 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a5c011790 con 0x7f9a6406f620 2022-06-27T12:08:25.169 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.132+0000 7f9a627fc700 1 -- 172.21.15.195:0/1656969371 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a5c011b30 con 0x7f9a6406f620 2022-06-27T12:08:25.169 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.132+0000 7f9a69de1700 1 -- 172.21.15.195:0/1656969371 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9a6406f620 msgr2=0x7f9a6406d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:25.171 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.132+0000 7f9a69de1700 1 --2- 172.21.15.195:0/1656969371 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9a6406f620 0x7f9a6406d6a0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f9a640c8470 tx=0x7f9a5c009060 comp rx=0 tx=0).stop 2022-06-27T12:08:25.176 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.132+0000 7f9a69de1700 1 -- 172.21.15.195:0/1656969371 shutdown_connections 2022-06-27T12:08:25.176 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.132+0000 7f9a69de1700 1 --2- 172.21.15.195:0/1656969371 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9a6406f620 0x7f9a6406d6a0 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:25.177 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.132+0000 7f9a69de1700 1 --2- 172.21.15.195:0/1656969371 >> v2:172.21.15.202:3300/0 conn(0x7f9a64150580 0x7f9a64156ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:25.177 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.132+0000 7f9a69de1700 1 -- 172.21.15.195:0/1656969371 >> 172.21.15.195:0/1656969371 conn(0x7f9a64063af0 msgr2=0x7f9a64063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:08:25.178 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.132+0000 7f9a69de1700 1 -- 172.21.15.195:0/1656969371 shutdown_connections 2022-06-27T12:08:25.178 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.132+0000 7f9a69de1700 1 -- 172.21.15.195:0/1656969371 wait complete. 2022-06-27T12:08:25.178 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.133+0000 7f9a69de1700 1 Processor -- start 2022-06-27T12:08:25.179 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.134+0000 7f9a69de1700 1 -- start start 2022-06-27T12:08:25.179 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.134+0000 7f9a69de1700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9a6406f620 0x7f9a6414d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:25.179 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.134+0000 7f9a69de1700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9a64150580 0x7f9a640bce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:25.180 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.134+0000 7f9a69de1700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9a64150950 0x7f9a640bd350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:25.180 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.134+0000 7f9a69de1700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a640c8930 con 0x7f9a64150580 2022-06-27T12:08:25.180 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.134+0000 7f9a69de1700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a640bbdc0 con 0x7f9a64150950 2022-06-27T12:08:25.181 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.134+0000 7f9a69de1700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a64166e20 con 0x7f9a6406f620 2022-06-27T12:08:25.181 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.134+0000 7f9a62ffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9a64150580 0x7f9a640bce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:25.181 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.134+0000 7f9a62ffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9a64150580 0x7f9a640bce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:49174/0 (socket says 172.21.15.195:49174) 2022-06-27T12:08:25.182 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.134+0000 7f9a62ffd700 1 -- 172.21.15.195:0/2733190791 learned_addr learned my addr 172.21.15.195:0/2733190791 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:08:25.182 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.134+0000 7f9a63fff700 1 --2- 172.21.15.195:0/2733190791 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9a64150950 0x7f9a640bd350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:25.182 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.134+0000 7f9a637fe700 1 --2- 172.21.15.195:0/2733190791 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9a6406f620 0x7f9a6414d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:25.183 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.135+0000 7f9a62ffd700 1 -- 172.21.15.195:0/2733190791 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9a6406f620 msgr2=0x7f9a6414d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:25.183 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.135+0000 7f9a62ffd700 1 --2- 172.21.15.195:0/2733190791 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9a6406f620 0x7f9a6414d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:25.183 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.135+0000 7f9a62ffd700 1 -- 172.21.15.195:0/2733190791 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9a64150950 msgr2=0x7f9a640bd350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:25.193 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.135+0000 7f9a62ffd700 1 --2- 172.21.15.195:0/2733190791 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9a64150950 0x7f9a640bd350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:25.193 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.135+0000 7f9a62ffd700 1 -- 172.21.15.195:0/2733190791 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a5c0092e0 con 0x7f9a64150580 2022-06-27T12:08:25.194 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.135+0000 7f9a62ffd700 1 --2- 172.21.15.195:0/2733190791 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9a64150580 0x7f9a640bce10 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f9a4c00c590 tx=0x7f9a4c007cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:08:25.194 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.135+0000 7f9a5b7fe700 1 -- 172.21.15.195:0/2733190791 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a4c007360 con 0x7f9a64150580 2022-06-27T12:08:25.195 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.135+0000 7f9a5b7fe700 1 -- 172.21.15.195:0/2733190791 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a4c007500 con 0x7f9a64150580 2022-06-27T12:08:25.195 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.135+0000 7f9a5b7fe700 1 -- 172.21.15.195:0/2733190791 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a4c004940 con 0x7f9a64150580 2022-06-27T12:08:25.196 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.135+0000 7f9a69de1700 1 -- 172.21.15.195:0/2733190791 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a64166b40 con 0x7f9a64150580 2022-06-27T12:08:25.196 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.135+0000 7f9a69de1700 1 -- 172.21.15.195:0/2733190791 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a640bbae0 con 0x7f9a64150580 2022-06-27T12:08:25.196 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.136+0000 7f9a69de1700 1 -- 172.21.15.195:0/2733190791 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a640722c0 con 0x7f9a64150580 2022-06-27T12:08:25.210 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.136+0000 7f9a5b7fe700 1 -- 172.21.15.195:0/2733190791 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f9a4c010030 con 0x7f9a64150580 2022-06-27T12:08:25.210 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.137+0000 7f9a5b7fe700 1 --2- 172.21.15.195:0/2733190791 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9a48063d00 0x7f9a480661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:25.211 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.137+0000 7f9a5b7fe700 1 -- 172.21.15.195:0/2733190791 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7148+0+0 (secure 0 0 0) 0x7f9a4c0806b0 con 0x7f9a64150580 2022-06-27T12:08:25.211 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.137+0000 7f9a637fe700 1 --2- 172.21.15.195:0/2733190791 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9a48063d00 0x7f9a480661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:25.213 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.137+0000 7f9a637fe700 1 --2- 172.21.15.195:0/2733190791 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9a48063d00 0x7f9a480661b0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f9a5400f570 tx=0x7f9a5400b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:08:25.213 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.141+0000 7f9a5b7fe700 1 -- 172.21.15.195:0/2733190791 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f9a4c04eee0 con 0x7f9a64150580 2022-06-27T12:08:25.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.369+0000 7f9a69de1700 1 -- 172.21.15.195:0/2733190791 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9a6406fe20 con 0x7f9a64150580 2022-06-27T12:08:25.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.369+0000 7f9a5b7fe700 1 -- 172.21.15.195:0/2733190791 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+5368 (secure 0 0 0) 0x7f9a4c052500 con 0x7f9a64150580 2022-06-27T12:08:25.372 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:08:25.372 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:08:23.133338+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6271},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6271":{"gid":6271,"name":"d","rank":0,"incarnation":73,"state":"up:active","state_seq":20,"addr":"172.21.15.202:6835/1603307620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":1603307620},{"type":"v1","addr":"172.21.15.202:6835","nonce":1603307620}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:08:25.375 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.371+0000 7f9a69de1700 1 -- 172.21.15.195:0/2733190791 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9a48063d00 msgr2=0x7f9a480661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:25.376 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.371+0000 7f9a69de1700 1 --2- 172.21.15.195:0/2733190791 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9a48063d00 0x7f9a480661b0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f9a5400f570 tx=0x7f9a5400b040 comp rx=0 tx=0).stop 2022-06-27T12:08:25.376 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.371+0000 7f9a69de1700 1 -- 172.21.15.195:0/2733190791 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9a64150580 msgr2=0x7f9a640bce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:25.377 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.371+0000 7f9a69de1700 1 --2- 172.21.15.195:0/2733190791 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9a64150580 0x7f9a640bce10 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f9a4c00c590 tx=0x7f9a4c007cb0 comp rx=0 tx=0).stop 2022-06-27T12:08:25.377 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.371+0000 7f9a69de1700 1 -- 172.21.15.195:0/2733190791 shutdown_connections 2022-06-27T12:08:25.378 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.371+0000 7f9a69de1700 1 --2- 172.21.15.195:0/2733190791 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9a48063d00 0x7f9a480661b0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:25.378 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.371+0000 7f9a69de1700 1 --2- 172.21.15.195:0/2733190791 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9a64150950 0x7f9a640bd350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:25.379 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.371+0000 7f9a69de1700 1 --2- 172.21.15.195:0/2733190791 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9a6406f620 0x7f9a6414d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:25.379 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.371+0000 7f9a69de1700 1 --2- 172.21.15.195:0/2733190791 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9a64150580 0x7f9a640bce10 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:25.380 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.371+0000 7f9a69de1700 1 -- 172.21.15.195:0/2733190791 >> 172.21.15.195:0/2733190791 conn(0x7f9a64063af0 msgr2=0x7f9a64069b00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:08:25.380 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.371+0000 7f9a69de1700 1 -- 172.21.15.195:0/2733190791 shutdown_connections 2022-06-27T12:08:25.381 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:25.371+0000 7f9a69de1700 1 -- 172.21.15.195:0/2733190791 wait complete. 2022-06-27T12:08:25.381 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 77 2022-06-27T12:08:25.393 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2022-06-27T12:08:27.397 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:08:27.540 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.531+0000 7fac09368700 1 Processor -- start 2022-06-27T12:08:27.541 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.531+0000 7fac09368700 1 -- start start 2022-06-27T12:08:27.541 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.531+0000 7fac09368700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fac04150580 0x7fac04156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:27.542 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.531+0000 7fac09368700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac040c76b0 con 0x7fac04150950 2022-06-27T12:08:27.542 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.531+0000 7fac09368700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac040c8250 con 0x7fac0406f620 2022-06-27T12:08:27.543 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.531+0000 7fac09368700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac04157430 con 0x7fac04150580 2022-06-27T12:08:27.543 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.532+0000 7fac027fc700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fac04150950 0x7fac040c7b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:55370/0 (socket says 172.21.15.195:55370) 2022-06-27T12:08:27.544 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.532+0000 7fac027fc700 1 -- 172.21.15.195:0/2242892724 learned_addr learned my addr 172.21.15.195:0/2242892724 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:08:27.544 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.532+0000 7fac02ffd700 1 --2- 172.21.15.195:0/2242892724 >> v2:172.21.15.202:3300/0 conn(0x7fac04150580 0x7fac04156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:27.544 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.532+0000 7fac01ffb700 1 -- 172.21.15.195:0/2242892724 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 716910640 0 0) 0x7fac040c76b0 con 0x7fac04150950 2022-06-27T12:08:27.545 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.532+0000 7fac01ffb700 1 -- 172.21.15.195:0/2242892724 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabf0003430 con 0x7fac04150950 2022-06-27T12:08:27.549 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.532+0000 7fac01ffb700 1 -- 172.21.15.195:0/2242892724 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1627993201 0 0) 0x7fabf0003430 con 0x7fac04150950 2022-06-27T12:08:27.550 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.532+0000 7fac01ffb700 1 -- 172.21.15.195:0/2242892724 >> v2:172.21.15.202:3300/0 conn(0x7fac04150580 msgr2=0x7fac04156ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:08:27.550 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.532+0000 7fac01ffb700 1 --2- 172.21.15.195:0/2242892724 >> v2:172.21.15.202:3300/0 conn(0x7fac04150580 0x7fac04156ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:27.551 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.532+0000 7fac01ffb700 1 -- 172.21.15.195:0/2242892724 >> v1:172.21.15.202:6789/0 conn(0x7fac0406f620 legacy=0x7fac0406d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:27.551 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.532+0000 7fac01ffb700 1 -- 172.21.15.195:0/2242892724 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac040bad70 con 0x7fac04150950 2022-06-27T12:08:27.587 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.532+0000 7fac01ffb700 1 -- 172.21.15.195:0/2242892724 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7fabf4001470 con 0x7fac04150950 2022-06-27T12:08:27.588 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.532+0000 7fac01ffb700 1 -- 172.21.15.195:0/2242892724 >> v1:172.21.15.195:6789/0 conn(0x7fac04150950 legacy=0x7fac040c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:27.588 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.532+0000 7fac01ffb700 1 --2- 172.21.15.195:0/2242892724 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fabf0003920 0x7fabf0003d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:27.589 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.532+0000 7fac01ffb700 1 --2- 172.21.15.195:0/2242892724 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fabf00046d0 0x7fabf0004b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:27.589 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.532+0000 7fac01ffb700 1 --2- 172.21.15.195:0/2242892724 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fabf0005080 0x7fabf000b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:27.590 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.532+0000 7fac01ffb700 1 -- 172.21.15.195:0/2242892724 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabf000bf60 con 0x7fabf00046d0 2022-06-27T12:08:27.603 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.532+0000 7fac01ffb700 1 -- 172.21.15.195:0/2242892724 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabf000c0d0 con 0x7fabf0003920 2022-06-27T12:08:27.606 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.532+0000 7fac01ffb700 1 -- 172.21.15.195:0/2242892724 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabf000c290 con 0x7fabf0005080 2022-06-27T12:08:27.607 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.532+0000 7fac027fc700 1 --2- 172.21.15.195:0/2242892724 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fabf00046d0 0x7fabf0004b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:27.607 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.532+0000 7fac02ffd700 1 --2- 172.21.15.195:0/2242892724 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fabf0003920 0x7fabf0003d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:27.608 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.533+0000 7fac037fe700 1 --2- 172.21.15.195:0/2242892724 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fabf0005080 0x7fabf000b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:27.608 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.533+0000 7fac027fc700 1 -- 172.21.15.195:0/2242892724 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fabf0005080 msgr2=0x7fabf000b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:08:27.609 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.533+0000 7fac027fc700 1 --2- 172.21.15.195:0/2242892724 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fabf0005080 0x7fabf000b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:27.609 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.533+0000 7fac027fc700 1 -- 172.21.15.195:0/2242892724 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fabf0003920 msgr2=0x7fabf0003d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:08:27.610 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.533+0000 7fac027fc700 1 --2- 172.21.15.195:0/2242892724 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fabf0003920 0x7fabf0003d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:27.611 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.533+0000 7fac027fc700 1 -- 172.21.15.195:0/2242892724 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabf0003430 con 0x7fabf00046d0 2022-06-27T12:08:27.612 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.533+0000 7fac037fe700 1 --2- 172.21.15.195:0/2242892724 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fabf0005080 0x7fabf000b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:08:27.612 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.533+0000 7fac02ffd700 1 --2- 172.21.15.195:0/2242892724 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fabf0003920 0x7fabf0003d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:08:27.612 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.533+0000 7fac037fe700 1 -- 172.21.15.195:0/2242892724 reap_dead start 2022-06-27T12:08:27.612 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.533+0000 7fac027fc700 1 --2- 172.21.15.195:0/2242892724 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fabf00046d0 0x7fabf0004b40 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fabf400c210 tx=0x7fabf4001e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86f00974d63baf21 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:08:27.613 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.533+0000 7fac01ffb700 1 -- 172.21.15.195:0/2242892724 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabf400cae0 con 0x7fabf00046d0 2022-06-27T12:08:27.613 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.533+0000 7fac01ffb700 1 -- 172.21.15.195:0/2242892724 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabf400cc60 con 0x7fabf00046d0 2022-06-27T12:08:27.613 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.533+0000 7fac01ffb700 1 -- 172.21.15.195:0/2242892724 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabf40070d0 con 0x7fabf00046d0 2022-06-27T12:08:27.613 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.533+0000 7fac09368700 1 -- 172.21.15.195:0/2242892724 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fabf00046d0 msgr2=0x7fabf0004b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:27.614 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.533+0000 7fac09368700 1 --2- 172.21.15.195:0/2242892724 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fabf00046d0 0x7fabf0004b40 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fabf400c210 tx=0x7fabf4001e30 comp rx=0 tx=0).stop 2022-06-27T12:08:27.614 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.533+0000 7fac09368700 1 -- 172.21.15.195:0/2242892724 shutdown_connections 2022-06-27T12:08:27.614 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.533+0000 7fac09368700 1 --2- 172.21.15.195:0/2242892724 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fabf00046d0 0x7fabf0004b40 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:27.621 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.534+0000 7fac09368700 1 -- 172.21.15.195:0/2242892724 >> 172.21.15.195:0/2242892724 conn(0x7fac04063af0 msgr2=0x7fac04063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:08:27.622 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.534+0000 7fac09368700 1 -- 172.21.15.195:0/2242892724 shutdown_connections 2022-06-27T12:08:27.622 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.534+0000 7fac09368700 1 -- 172.21.15.195:0/2242892724 wait complete. 2022-06-27T12:08:27.623 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.535+0000 7fac09368700 1 Processor -- start 2022-06-27T12:08:27.623 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.535+0000 7fac09368700 1 -- start start 2022-06-27T12:08:27.624 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.535+0000 7fac09368700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fabf00046d0 0x7fac0414f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:27.624 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.536+0000 7fac09368700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fac0414a190 0x7fac04157cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:27.624 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.536+0000 7fac09368700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fac0414a930 0x7fac04158200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:27.624 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.536+0000 7fac09368700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac041597e0 con 0x7fac0414a930 2022-06-27T12:08:27.625 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.536+0000 7fac09368700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac04157830 con 0x7fac0414a190 2022-06-27T12:08:27.625 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.536+0000 7fac09368700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac041589a0 con 0x7fabf00046d0 2022-06-27T12:08:27.626 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.536+0000 7fac037fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fac0414a930 0x7fac04158200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:27.626 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.536+0000 7fac037fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fac0414a930 0x7fac04158200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:49194/0 (socket says 172.21.15.195:49194) 2022-06-27T12:08:27.626 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.536+0000 7fac037fe700 1 -- 172.21.15.195:0/3985096349 learned_addr learned my addr 172.21.15.195:0/3985096349 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:08:27.627 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.536+0000 7fac027fc700 1 --2- 172.21.15.195:0/3985096349 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fac0414a190 0x7fac04157cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:27.627 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.536+0000 7fac02ffd700 1 --2- 172.21.15.195:0/3985096349 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fabf00046d0 0x7fac0414f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:27.628 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.536+0000 7fac037fe700 1 -- 172.21.15.195:0/3985096349 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fabf00046d0 msgr2=0x7fac0414f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:27.629 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.536+0000 7fac037fe700 1 --2- 172.21.15.195:0/3985096349 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fabf00046d0 0x7fac0414f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:27.641 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.536+0000 7fac037fe700 1 -- 172.21.15.195:0/3985096349 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fac0414a190 msgr2=0x7fac04157cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:27.641 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.536+0000 7fac037fe700 1 --2- 172.21.15.195:0/3985096349 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fac0414a190 0x7fac04157cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:27.651 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.536+0000 7fac037fe700 1 -- 172.21.15.195:0/3985096349 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac04158c60 con 0x7fac0414a930 2022-06-27T12:08:27.652 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.536+0000 7fac027fc700 1 --2- 172.21.15.195:0/3985096349 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fac0414a190 0x7fac04157cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T12:08:27.652 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.536+0000 7fac037fe700 1 --2- 172.21.15.195:0/3985096349 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fac0414a930 0x7fac04158200 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fac0406fdf0 tx=0x7fabfc009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:08:27.652 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.536+0000 7fabeaffd700 1 -- 172.21.15.195:0/3985096349 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabfc003a90 con 0x7fac0414a930 2022-06-27T12:08:27.653 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.536+0000 7fabeaffd700 1 -- 172.21.15.195:0/3985096349 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabfc003c10 con 0x7fac0414a930 2022-06-27T12:08:27.653 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.536+0000 7fac09368700 1 -- 172.21.15.195:0/3985096349 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac04158f40 con 0x7fac0414a930 2022-06-27T12:08:27.653 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.536+0000 7fabeaffd700 1 -- 172.21.15.195:0/3985096349 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabfc0050d0 con 0x7fac0414a930 2022-06-27T12:08:27.654 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.536+0000 7fac09368700 1 -- 172.21.15.195:0/3985096349 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac04159500 con 0x7fac0414a930 2022-06-27T12:08:27.654 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.537+0000 7fac09368700 1 -- 172.21.15.195:0/3985096349 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac040722c0 con 0x7fac0414a930 2022-06-27T12:08:27.655 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.538+0000 7fabeaffd700 1 -- 172.21.15.195:0/3985096349 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fabfc005440 con 0x7fac0414a930 2022-06-27T12:08:27.655 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.538+0000 7fabeaffd700 1 --2- 172.21.15.195:0/3985096349 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fabe4063d30 0x7fabe40661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:08:27.655 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.538+0000 7fabeaffd700 1 -- 172.21.15.195:0/3985096349 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7148+0+0 (secure 0 0 0) 0x7fabfc080270 con 0x7fac0414a930 2022-06-27T12:08:27.656 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.538+0000 7fac02ffd700 1 --2- 172.21.15.195:0/3985096349 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fabe4063d30 0x7fabe40661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:08:27.657 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.539+0000 7fac02ffd700 1 --2- 172.21.15.195:0/3985096349 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fabe4063d30 0x7fabe40661e0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fabf80095e0 tx=0x7fabf8008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:08:27.657 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.542+0000 7fabeaffd700 1 -- 172.21.15.195:0/3985096349 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fabfc04ac90 con 0x7fac0414a930 2022-06-27T12:08:27.774 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.769+0000 7fac09368700 1 -- 172.21.15.195:0/3985096349 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac04157970 con 0x7fac0414a930 2022-06-27T12:08:27.775 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.770+0000 7fabeaffd700 1 -- 172.21.15.195:0/3985096349 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+6125 (secure 0 0 0) 0x7fabfc019020 con 0x7fac0414a930 2022-06-27T12:08:27.775 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:08:27.775 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":78,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":6708,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/702244802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":702244802},{"type":"v1","addr":"172.21.15.202:6833","nonce":702244802}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:08:23.133338+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6271},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6271":{"gid":6271,"name":"d","rank":0,"incarnation":73,"state":"up:active","state_seq":20,"addr":"172.21.15.202:6835/1603307620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":1603307620},{"type":"v1","addr":"172.21.15.202:6835","nonce":1603307620}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:08:27.777 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.772+0000 7fac09368700 1 -- 172.21.15.195:0/3985096349 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fabe4063d30 msgr2=0x7fabe40661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:27.777 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.772+0000 7fac09368700 1 --2- 172.21.15.195:0/3985096349 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fabe4063d30 0x7fabe40661e0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fabf80095e0 tx=0x7fabf8008040 comp rx=0 tx=0).stop 2022-06-27T12:08:27.777 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.772+0000 7fac09368700 1 -- 172.21.15.195:0/3985096349 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fac0414a930 msgr2=0x7fac04158200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:08:27.778 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.772+0000 7fac09368700 1 --2- 172.21.15.195:0/3985096349 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fac0414a930 0x7fac04158200 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fac0406fdf0 tx=0x7fabfc009040 comp rx=0 tx=0).stop 2022-06-27T12:08:27.778 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.772+0000 7fac09368700 1 -- 172.21.15.195:0/3985096349 shutdown_connections 2022-06-27T12:08:27.778 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.772+0000 7fac09368700 1 --2- 172.21.15.195:0/3985096349 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fabe4063d30 0x7fabe40661e0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:27.778 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.772+0000 7fac09368700 1 --2- 172.21.15.195:0/3985096349 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fac0414a930 0x7fac04158200 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:27.779 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.772+0000 7fac09368700 1 --2- 172.21.15.195:0/3985096349 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fabf00046d0 0x7fac0414f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:27.779 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.772+0000 7fac09368700 1 --2- 172.21.15.195:0/3985096349 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fac0414a190 0x7fac04157cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:08:27.779 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.772+0000 7fac09368700 1 -- 172.21.15.195:0/3985096349 >> 172.21.15.195:0/3985096349 conn(0x7fac04063af0 msgr2=0x7fac04063ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:08:27.779 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.772+0000 7fac09368700 1 -- 172.21.15.195:0/3985096349 shutdown_connections 2022-06-27T12:08:27.780 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:08:27.772+0000 7fac09368700 1 -- 172.21.15.195:0/3985096349 wait complete. 2022-06-27T12:08:27.780 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 78 2022-06-27T12:08:27.788 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2022-06-27T12:08:27.789 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 41 secs before thrashing 2022-06-27T12:08:29.250 INFO:tasks.ceph.mds.d.smithi202.stderr:2022-06-27T12:08:29.247+0000 7fe5277fd700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2022-06-27T12:08:29.444 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 9% (60/622) Updating files: 10% (63/622) Updating files: 11% (69/622) Updating files: 12% (75/622) Updating files: 13% (81/622) Updating files: 14% (88/622) Updating files: 15% (94/622) Updating files: 16% (100/622) Updating files: 17% (106/622) Updating files: 18% (112/622) Updating files: 19% (119/622) Updating files: 20% (125/622) Updating files: 21% (131/622) Updating files: 22% (137/622) Updating files: 23% (144/622) Updating files: 24% (150/622) Updating files: 25% (156/622) Updating files: 26% (162/622) Updating files: 27% (168/622) Updating files: 28% (175/622) Updating files: 29% (181/622) Updating files: 30% (187/622) Updating files: 31% (193/622) Updating files: 32% (200/622) Updating files: 33% (206/622) Updating files: 34% (212/622) Updating files: 35% (218/622) Updating files: 36% (224/622) Updating files: 37% (231/622) Updating files: 38% (237/622) Updating files: 39% (243/622) Updating files: 40% (249/622) Updating files: 41% (256/622) Updating files: 42% (262/622) Updating files: 43% (268/622) Updating files: 44% (274/622) Updating files: 44% (277/622) Updating files: 45% (280/622) Updating files: 46% (287/622) Updating files: 47% (293/622) Updating files: 48% (299/622) Updating files: 49% (305/622) Updating files: 50% (311/622) Updating files: 51% (318/622) Updating files: 52% (324/622) Updating files: 53% (330/622) Updating files: 54% (336/622) Updating files: 55% (343/622) Updating files: 56% (349/622) Updating files: 56% (351/622) Updating files: 57% (355/622) Updating files: 58% (361/622) Updating files: 59% (367/622) Updating files: 60% (374/622) Updating files: 61% (380/622) Updating files: 62% (386/622) Updating files: 63% (392/622) Updating files: 64% (399/622) Updating files: 65% (405/622) Updating files: 66% (411/622) Updating files: 67% (417/622) Updating files: 68% (423/622) Updating files: 69% (430/622) Updating files: 70% (436/622) Updating files: 71% (442/622) Updating files: 71% (446/622) Updating files: 72% (448/622) Updating files: 73% (455/622) Updating files: 74% (461/622) Updating files: 75% (467/622) Updating files: 76% (473/622) Updating files: 77% (479/622) Updating files: 77% (485/622) Updating files: 78% (486/622) Updating files: 79% (492/622) Updating files: 80% (498/622) Updating files: 81% (504/622) Updating files: 82% (511/622) Updating files: 83% (517/622) Updating files: 84% (523/622) Updating files: 84% (524/622) Updating files: 85% (529/622) Updating files: 86% (535/622) Updating files: 86% (539/622) Updating files: 87% (542/622) Updating files: 88% (548/622) Updating files: 89% (554/622) Updating files: 90% (560/622) Updating files: 91% (567/622) Updating files: 92% (573/622) Updating files: 93% (579/622) Updating files: 94% (585/622) Updating files: 95% (591/622) Updating files: 96% (598/622) Updating files: 97% (604/622) Updating files: 97% (609/622) Updating files: 98% (610/622) Updating files: 99% (616/622) Updating files: 100% (622/622) Updating files: 100% (622/622), done. 2022-06-27T12:08:31.104 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at 3fa121bf0a8 v0.25 2022-06-27T12:08:31.117 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.25 2022-06-27T12:08:31.140 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:08:31.140 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 26 -eq 48 ']' 2022-06-27T12:08:31.141 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.26 2022-06-27T12:08:31.141 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.26 2022-06-27T12:08:31.142 INFO:tasks.workunit.client.0.smithi195.stdout:v0.26 2022-06-27T12:08:31.143 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.26 2022-06-27T12:08:31.870 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:08:31.877 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:08:34.641 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 89% (215/239) Updating files: 90% (216/239) Updating files: 91% (218/239) Updating files: 92% (220/239) Updating files: 93% (223/239) Updating files: 94% (225/239) Updating files: 95% (228/239) Updating files: 96% (230/239) Updating files: 97% (232/239) Updating files: 98% (235/239) Updating files: 99% (237/239) Updating files: 100% (239/239) Updating files: 100% (239/239), done. 2022-06-27T12:08:34.671 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at 9981ff90968 v0.26 2022-06-27T12:08:34.678 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.26 2022-06-27T12:08:34.863 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:08:34.863 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 27 -eq 48 ']' 2022-06-27T12:08:34.863 INFO:tasks.workunit.client.0.smithi195.stdout:v0.27 2022-06-27T12:08:34.864 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.27 2022-06-27T12:08:34.864 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.27 2022-06-27T12:08:34.865 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.27 2022-06-27T12:08:38.358 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 65% (166/253) Updating files: 66% (167/253) Updating files: 67% (170/253) Updating files: 68% (173/253) Updating files: 69% (175/253) Updating files: 70% (178/253) Updating files: 71% (180/253) Updating files: 72% (183/253) Updating files: 73% (185/253) Updating files: 74% (188/253) Updating files: 75% (190/253) Updating files: 75% (192/253) Updating files: 76% (193/253) Updating files: 77% (195/253) Updating files: 78% (198/253) Updating files: 79% (200/253) Updating files: 80% (203/253) Updating files: 81% (205/253) Updating files: 82% (208/253) Updating files: 83% (210/253) Updating files: 84% (213/253) Updating files: 85% (216/253) Updating files: 86% (218/253) Updating files: 87% (221/253) Updating files: 88% (223/253) Updating files: 89% (226/253) Updating files: 90% (228/253) Updating files: 91% (231/253) Updating files: 92% (233/253) Updating files: 93% (236/253) Updating files: 94% (238/253) Updating files: 95% (241/253) Updating files: 96% (243/253) Updating files: 97% (246/253) Updating files: 98% (248/253) Updating files: 99% (251/253) Updating files: 100% (253/253) Updating files: 100% (253/253), done. 2022-06-27T12:08:38.379 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at 793034c62c8 v0.27 2022-06-27T12:08:38.383 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.27 2022-06-27T12:08:38.447 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:08:38.448 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 28 -eq 48 ']' 2022-06-27T12:08:38.448 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.28 2022-06-27T12:08:38.448 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.28 2022-06-27T12:08:38.449 INFO:tasks.workunit.client.0.smithi195.stdout:v0.28 2022-06-27T12:08:38.449 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.28 2022-06-27T12:08:43.579 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 18% (43/227) Updating files: 19% (44/227) Updating files: 20% (46/227) Updating files: 21% (48/227) Updating files: 22% (50/227) Updating files: 23% (53/227) Updating files: 24% (55/227) Updating files: 25% (57/227) Updating files: 26% (60/227) Updating files: 27% (62/227) Updating files: 28% (64/227) Updating files: 29% (66/227) Updating files: 30% (69/227) Updating files: 31% (71/227) Updating files: 32% (73/227) Updating files: 33% (75/227) Updating files: 34% (78/227) Updating files: 35% (80/227) Updating files: 36% (82/227) Updating files: 37% (84/227) Updating files: 38% (87/227) Updating files: 39% (89/227) Updating files: 40% (91/227) Updating files: 41% (94/227) Updating files: 42% (96/227) Updating files: 43% (98/227) Updating files: 44% (100/227) Updating files: 45% (103/227) Updating files: 46% (105/227) Updating files: 47% (107/227) Updating files: 48% (109/227) Updating files: 49% (112/227) Updating files: 50% (114/227) Updating files: 50% (115/227) Updating files: 51% (116/227) Updating files: 52% (119/227) Updating files: 53% (121/227) Updating files: 54% (123/227) Updating files: 55% (125/227) Updating files: 56% (128/227) Updating files: 57% (130/227) Updating files: 58% (132/227) Updating files: 59% (134/227) Updating files: 60% (137/227) Updating files: 61% (139/227) Updating files: 62% (141/227) Updating files: 63% (144/227) Updating files: 64% (146/227) Updating files: 65% (148/227) Updating files: 66% (150/227) Updating files: 67% (153/227) Updating files: 68% (155/227) Updating files: 69% (157/227) Updating files: 70% (159/227) Updating files: 71% (162/227) Updating files: 72% (164/227) Updating files: 72% (165/227) Updating files: 73% (166/227) Updating files: 74% (168/227) Updating files: 75% (171/227) Updating files: 76% (173/227) Updating files: 77% (175/227) Updating files: 78% (178/227) Updating files: 79% (180/227) Updating files: 80% (182/227) Updating files: 81% (184/227) Updating files: 82% (187/227) Updating files: 83% (189/227) Updating files: 84% (191/227) Updating files: 85% (193/227) Updating files: 86% (196/227) Updating files: 87% (198/227) Updating files: 88% (200/227) Updating files: 89% (203/227) Updating files: 90% (205/227) Updating files: 91% (207/227) Updating files: 92% (209/227) Updating files: 93% (212/227) Updating files: 94% (214/227) Updating files: 95% (216/227) Updating files: 96% (218/227) Updating files: 97% (221/227) Updating files: 98% (223/227) Updating files: 99% (225/227) Updating files: 100% (227/227) Updating files: 100% (227/227), done. 2022-06-27T12:08:43.603 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at 071881d7e55 v0.28 2022-06-27T12:08:43.607 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.28 2022-06-27T12:08:43.706 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:08:43.707 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 29 -eq 48 ']' 2022-06-27T12:08:43.707 INFO:tasks.workunit.client.0.smithi195.stdout:v0.29 2022-06-27T12:08:43.708 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.29 2022-06-27T12:08:43.709 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.29 2022-06-27T12:08:43.709 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.29 2022-06-27T12:08:45.455 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 20% (32/159) Updating files: 21% (34/159) Updating files: 22% (35/159) Updating files: 23% (37/159) Updating files: 24% (39/159) Updating files: 25% (40/159) Updating files: 26% (42/159) Updating files: 27% (43/159) Updating files: 28% (45/159) Updating files: 29% (47/159) Updating files: 30% (48/159) Updating files: 31% (50/159) Updating files: 32% (51/159) Updating files: 33% (53/159) Updating files: 34% (55/159) Updating files: 35% (56/159) Updating files: 36% (58/159) Updating files: 37% (59/159) Updating files: 38% (61/159) Updating files: 39% (63/159) Updating files: 40% (64/159) Updating files: 41% (66/159) Updating files: 42% (67/159) Updating files: 43% (69/159) Updating files: 44% (70/159) Updating files: 45% (72/159) Updating files: 46% (74/159) Updating files: 47% (75/159) Updating files: 48% (77/159) Updating files: 49% (78/159) Updating files: 50% (80/159) Updating files: 51% (82/159) Updating files: 52% (83/159) Updating files: 53% (85/159) Updating files: 54% (86/159) Updating files: 55% (88/159) Updating files: 56% (90/159) Updating files: 57% (91/159) Updating files: 58% (93/159) Updating files: 59% (94/159) Updating files: 60% (96/159) Updating files: 61% (97/159) Updating files: 62% (99/159) Updating files: 63% (101/159) Updating files: 64% (102/159) Updating files: 65% (104/159) Updating files: 66% (105/159) Updating files: 67% (107/159) Updating files: 68% (109/159) Updating files: 69% (110/159) Updating files: 70% (112/159) Updating files: 71% (113/159) Updating files: 72% (115/159) Updating files: 73% (117/159) Updating files: 74% (118/159) Updating files: 75% (120/159) Updating files: 76% (121/159) Updating files: 77% (123/159) Updating files: 78% (125/159) Updating files: 79% (126/159) Updating files: 80% (128/159) Updating files: 81% (129/159) Updating files: 82% (131/159) Updating files: 83% (132/159) Updating files: 84% (134/159) Updating files: 85% (136/159) Updating files: 86% (137/159) Updating files: 87% (139/159) Updating files: 88% (140/159) Updating files: 89% (142/159) Updating files: 90% (144/159) Updating files: 91% (145/159) Updating files: 92% (147/159) Updating files: 93% (148/159) Updating files: 94% (150/159) Updating files: 95% (152/159) Updating files: 96% (153/159) Updating files: 97% (155/159) Updating files: 98% (156/159) Updating files: 99% (158/159) Updating files: 100% (159/159) Updating files: 100% (159/159), done. 2022-06-27T12:08:45.478 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at 8e69c39f699 v0.29 2022-06-27T12:08:45.482 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.29 2022-06-27T12:08:45.720 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:08:45.720 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 30 -eq 48 ']' 2022-06-27T12:08:45.723 INFO:tasks.workunit.client.0.smithi195.stdout:v0.30 2022-06-27T12:08:45.724 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.30 2022-06-27T12:08:45.725 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.30 2022-06-27T12:08:45.725 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.30 2022-06-27T12:08:50.948 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 49% (142/285) Updating files: 50% (143/285) Updating files: 51% (146/285) Updating files: 52% (149/285) Updating files: 53% (152/285) Updating files: 54% (154/285) Updating files: 55% (157/285) Updating files: 56% (160/285) Updating files: 57% (163/285) Updating files: 58% (166/285) Updating files: 59% (169/285) Updating files: 60% (171/285) Updating files: 61% (174/285) Updating files: 62% (177/285) Updating files: 63% (180/285) Updating files: 64% (183/285) Updating files: 65% (186/285) Updating files: 66% (189/285) Updating files: 67% (191/285) Updating files: 68% (194/285) Updating files: 69% (197/285) Updating files: 70% (200/285) Updating files: 71% (203/285) Updating files: 72% (206/285) Updating files: 73% (209/285) Updating files: 74% (211/285) Updating files: 75% (214/285) Updating files: 76% (217/285) Updating files: 77% (220/285) Updating files: 78% (223/285) Updating files: 79% (226/285) Updating files: 80% (228/285) Updating files: 81% (231/285) Updating files: 82% (234/285) Updating files: 83% (237/285) Updating files: 84% (240/285) Updating files: 84% (242/285) Updating files: 85% (243/285) Updating files: 86% (246/285) Updating files: 87% (248/285) Updating files: 88% (251/285) Updating files: 89% (254/285) Updating files: 90% (257/285) Updating files: 90% (258/285) Updating files: 91% (260/285) Updating files: 92% (263/285) Updating files: 93% (266/285) Updating files: 94% (268/285) Updating files: 95% (271/285) Updating files: 96% (274/285) Updating files: 97% (277/285) Updating files: 98% (280/285) Updating files: 99% (283/285) Updating files: 100% (285/285) Updating files: 100% (285/285), done. 2022-06-27T12:08:50.977 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at 64b1b2c70f0 v0.30 2022-06-27T12:08:50.983 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.30 2022-06-27T12:08:51.131 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:08:51.131 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 31 -eq 48 ']' 2022-06-27T12:08:51.132 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.31 2022-06-27T12:08:51.132 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.31 2022-06-27T12:08:51.132 INFO:tasks.workunit.client.0.smithi195.stdout:v0.31 2022-06-27T12:08:51.133 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.31 2022-06-27T12:08:54.695 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 12% (49/398) Updating files: 13% (52/398) Updating files: 14% (56/398) Updating files: 15% (60/398) Updating files: 16% (64/398) Updating files: 17% (68/398) Updating files: 18% (72/398) Updating files: 19% (76/398) Updating files: 20% (80/398) Updating files: 21% (84/398) Updating files: 22% (88/398) Updating files: 23% (92/398) Updating files: 24% (96/398) Updating files: 25% (100/398) Updating files: 26% (104/398) Updating files: 27% (108/398) Updating files: 28% (112/398) Updating files: 29% (116/398) Updating files: 30% (120/398) Updating files: 31% (124/398) Updating files: 32% (128/398) Updating files: 33% (132/398) Updating files: 34% (136/398) Updating files: 35% (140/398) Updating files: 36% (144/398) Updating files: 37% (148/398) Updating files: 38% (152/398) Updating files: 39% (156/398) Updating files: 40% (160/398) Updating files: 41% (164/398) Updating files: 42% (168/398) Updating files: 43% (172/398) Updating files: 44% (176/398) Updating files: 45% (180/398) Updating files: 46% (184/398) Updating files: 47% (188/398) Updating files: 48% (192/398) Updating files: 49% (196/398) Updating files: 50% (199/398) Updating files: 51% (203/398) Updating files: 52% (207/398) Updating files: 53% (211/398) Updating files: 54% (215/398) Updating files: 55% (219/398) Updating files: 56% (223/398) Updating files: 57% (227/398) Updating files: 58% (231/398) Updating files: 59% (235/398) Updating files: 60% (239/398) Updating files: 61% (243/398) Updating files: 62% (247/398) Updating files: 62% (250/398) Updating files: 63% (251/398) Updating files: 64% (255/398) Updating files: 65% (259/398) Updating files: 66% (263/398) Updating files: 67% (267/398) Updating files: 68% (271/398) Updating files: 69% (275/398) Updating files: 70% (279/398) Updating files: 71% (283/398) Updating files: 72% (287/398) Updating files: 73% (291/398) Updating files: 74% (295/398) Updating files: 75% (299/398) Updating files: 76% (303/398) Updating files: 77% (307/398) Updating files: 78% (311/398) Updating files: 79% (315/398) Updating files: 80% (319/398) Updating files: 81% (323/398) Updating files: 82% (327/398) Updating files: 83% (331/398) Updating files: 84% (335/398) Updating files: 85% (339/398) Updating files: 86% (343/398) Updating files: 87% (347/398) Updating files: 87% (348/398) Updating files: 88% (351/398) Updating files: 89% (355/398) Updating files: 90% (359/398) Updating files: 91% (363/398) Updating files: 92% (367/398) Updating files: 93% (371/398) Updating files: 94% (375/398) Updating files: 95% (379/398) Updating files: 96% (383/398) Updating files: 97% (387/398) Updating files: 98% (391/398) Updating files: 99% (395/398) Updating files: 100% (398/398) Updating files: 100% (398/398), done. 2022-06-27T12:08:54.718 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at 9019c6ce640 v0.31 2022-06-27T12:08:54.728 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.31 2022-06-27T12:08:54.973 INFO:tasks.workunit.client.0.smithi195.stdout:v0.32 2022-06-27T12:08:54.974 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:08:54.974 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 32 -eq 48 ']' 2022-06-27T12:08:54.975 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.32 2022-06-27T12:08:54.975 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.32 2022-06-27T12:08:54.976 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.32 2022-06-27T12:09:00.979 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 30% (59/195) Updating files: 31% (61/195) Updating files: 32% (63/195) Updating files: 32% (64/195) Updating files: 33% (65/195) Updating files: 34% (67/195) Updating files: 35% (69/195) Updating files: 36% (71/195) Updating files: 37% (73/195) Updating files: 38% (75/195) Updating files: 39% (77/195) Updating files: 40% (78/195) Updating files: 41% (80/195) Updating files: 42% (82/195) Updating files: 43% (84/195) Updating files: 44% (86/195) Updating files: 45% (88/195) Updating files: 46% (90/195) Updating files: 47% (92/195) Updating files: 48% (94/195) Updating files: 49% (96/195) Updating files: 50% (98/195) Updating files: 51% (100/195) Updating files: 52% (102/195) Updating files: 53% (104/195) Updating files: 54% (106/195) Updating files: 55% (108/195) Updating files: 56% (110/195) Updating files: 57% (112/195) Updating files: 58% (114/195) Updating files: 59% (116/195) Updating files: 60% (117/195) Updating files: 61% (119/195) Updating files: 62% (121/195) Updating files: 63% (123/195) Updating files: 64% (125/195) Updating files: 65% (127/195) Updating files: 66% (129/195) Updating files: 67% (131/195) Updating files: 67% (132/195) Updating files: 68% (133/195) Updating files: 69% (135/195) Updating files: 70% (137/195) Updating files: 71% (139/195) Updating files: 72% (141/195) Updating files: 73% (143/195) Updating files: 74% (145/195) Updating files: 75% (147/195) Updating files: 76% (149/195) Updating files: 77% (151/195) Updating files: 78% (153/195) Updating files: 79% (155/195) Updating files: 80% (156/195) Updating files: 81% (158/195) Updating files: 82% (160/195) Updating files: 83% (162/195) Updating files: 84% (164/195) Updating files: 85% (166/195) Updating files: 85% (167/195) Updating files: 86% (168/195) Updating files: 87% (170/195) Updating files: 88% (172/195) Updating files: 89% (174/195) Updating files: 90% (176/195) Updating files: 91% (178/195) Updating files: 92% (180/195) Updating files: 93% (182/195) Updating files: 94% (184/195) Updating files: 95% (186/195) Updating files: 96% (188/195) Updating files: 97% (190/195) Updating files: 98% (192/195) Updating files: 99% (194/195) Updating files: 100% (195/195) Updating files: 100% (195/195), done. 2022-06-27T12:09:01.001 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at c08d08baa6a v0.32 2022-06-27T12:09:01.003 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.32 2022-06-27T12:09:01.133 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:09:01.133 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 33 -eq 48 ']' 2022-06-27T12:09:01.134 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.33 2022-06-27T12:09:01.135 INFO:tasks.workunit.client.0.smithi195.stdout:v0.33 2022-06-27T12:09:01.135 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.33 2022-06-27T12:09:01.136 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.33 2022-06-27T12:09:01.958 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:09:01.966 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:09:02.773 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 42% (68/160) Updating files: 43% (69/160) Updating files: 44% (71/160) Updating files: 45% (72/160) Updating files: 46% (74/160) Updating files: 47% (76/160) Updating files: 48% (77/160) Updating files: 49% (79/160) Updating files: 50% (80/160) Updating files: 51% (82/160) Updating files: 52% (84/160) Updating files: 53% (85/160) Updating files: 54% (87/160) Updating files: 55% (88/160) Updating files: 56% (90/160) Updating files: 57% (92/160) Updating files: 58% (93/160) Updating files: 59% (95/160) Updating files: 60% (96/160) Updating files: 61% (98/160) Updating files: 62% (100/160) Updating files: 63% (101/160) Updating files: 64% (103/160) Updating files: 65% (104/160) Updating files: 66% (106/160) Updating files: 67% (108/160) Updating files: 68% (109/160) Updating files: 69% (111/160) Updating files: 70% (112/160) Updating files: 71% (114/160) Updating files: 72% (116/160) Updating files: 73% (117/160) Updating files: 74% (119/160) Updating files: 75% (120/160) Updating files: 76% (122/160) Updating files: 77% (124/160) Updating files: 78% (125/160) Updating files: 79% (127/160) Updating files: 80% (128/160) Updating files: 81% (130/160) Updating files: 82% (132/160) Updating files: 83% (133/160) Updating files: 84% (135/160) Updating files: 85% (136/160) Updating files: 86% (138/160) Updating files: 87% (140/160) Updating files: 88% (141/160) Updating files: 89% (143/160) Updating files: 90% (144/160) Updating files: 91% (146/160) Updating files: 92% (148/160) Updating files: 93% (149/160) Updating files: 94% (151/160) Updating files: 95% (152/160) Updating files: 96% (154/160) Updating files: 97% (156/160) Updating files: 98% (157/160) Updating files: 99% (159/160) Updating files: 100% (160/160) Updating files: 100% (160/160), done. 2022-06-27T12:09:02.807 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at 340b4348bf6 v0.33 2022-06-27T12:09:02.808 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.33 2022-06-27T12:09:03.104 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:09:03.105 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 34 -eq 48 ']' 2022-06-27T12:09:03.106 INFO:tasks.workunit.client.0.smithi195.stdout:v0.34 2022-06-27T12:09:03.108 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.34 2022-06-27T12:09:03.109 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.34 2022-06-27T12:09:03.109 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.34 2022-06-27T12:09:04.158 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 91% (118/129) Updating files: 92% (119/129) Updating files: 93% (120/129) Updating files: 94% (122/129) Updating files: 95% (123/129) Updating files: 96% (124/129) Updating files: 97% (126/129) Updating files: 98% (127/129) Updating files: 99% (128/129) Updating files: 100% (129/129) Updating files: 100% (129/129), done. 2022-06-27T12:09:04.195 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at 2f039eeeb74 v0.34 2022-06-27T12:09:04.204 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.34 2022-06-27T12:09:05.107 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:09:05.108 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 35 -eq 48 ']' 2022-06-27T12:09:05.108 INFO:tasks.workunit.client.0.smithi195.stdout:v0.35 2022-06-27T12:09:05.109 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.35 2022-06-27T12:09:05.109 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.35 2022-06-27T12:09:05.109 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.35 2022-06-27T12:09:08.791 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:09:08.960 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:08.927+0000 7fa7da0a9700 1 Processor -- start 2022-06-27T12:09:08.960 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:08.928+0000 7fa7da0a9700 1 -- start start 2022-06-27T12:09:08.961 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:08.928+0000 7fa7da0a9700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa7d414e860 0x7fa7d40c3840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:08.961 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:08.928+0000 7fa7da0a9700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fa7d40c3d80 0x7fa7d4157890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:08.962 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:08.928+0000 7fa7da0a9700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7d40c33b0 con 0x7fa7d414e490 2022-06-27T12:09:08.963 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:08.928+0000 7fa7da0a9700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7d40c4200 con 0x7fa7d40c3d80 2022-06-27T12:09:08.964 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:08.928+0000 7fa7da0a9700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7d406f220 con 0x7fa7d414e860 2022-06-27T12:09:08.964 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:08.928+0000 7fa7d3fff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fa7d40c3d80 0x7fa7d4157890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:08.965 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:08.929+0000 7fa7d3fff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fa7d40c3d80 0x7fa7d4157890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:49368/0 (socket says 172.21.15.195:49368) 2022-06-27T12:09:08.965 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:08.929+0000 7fa7d3fff700 1 -- 172.21.15.195:0/2701370953 learned_addr learned my addr 172.21.15.195:0/2701370953 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:09:08.966 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:08.929+0000 7fa7d2ffd700 1 --2- 172.21.15.195:0/2701370953 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa7d414e860 0x7fa7d40c3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:08.966 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:08.929+0000 7fa7d3fff700 1 -- 172.21.15.195:0/2701370953 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa7d414e860 msgr2=0x7fa7d40c3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:09:08.967 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:08.929+0000 7fa7d3fff700 1 --2- 172.21.15.195:0/2701370953 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa7d414e860 0x7fa7d40c3840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:08.967 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:08.929+0000 7fa7d3fff700 1 -- 172.21.15.195:0/2701370953 >> v1:172.21.15.195:6789/0 conn(0x7fa7d414e490 legacy=0x7fa7d406cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:08.967 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:08.929+0000 7fa7d27fc700 1 -- 172.21.15.195:0/2701370953 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3966171581 0 0) 0x7fa7d40c33b0 con 0x7fa7d414e490 2022-06-27T12:09:08.968 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:08.929+0000 7fa7d3fff700 1 -- 172.21.15.195:0/2701370953 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7d41542c0 con 0x7fa7d40c3d80 2022-06-27T12:09:08.968 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:08.929+0000 7fa7d3fff700 1 --2- 172.21.15.195:0/2701370953 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa7d40c3d80 0x7fa7d4157890 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fa7cc009f20 tx=0x7fa7cc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff08c2eab44a70ab server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:08.969 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:08.929+0000 7fa7d27fc700 1 -- 172.21.15.195:0/2701370953 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7cc00e070 con 0x7fa7d40c3d80 2022-06-27T12:09:08.969 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:08.929+0000 7fa7d27fc700 1 -- 172.21.15.195:0/2701370953 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7cc00b9d0 con 0x7fa7d40c3d80 2022-06-27T12:09:08.970 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:08.929+0000 7fa7d27fc700 1 -- 172.21.15.195:0/2701370953 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7cc0092e0 con 0x7fa7d40c3d80 2022-06-27T12:09:08.971 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:08.930+0000 7fa7da0a9700 1 -- 172.21.15.195:0/2701370953 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa7d40c3d80 msgr2=0x7fa7d4157890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:08.971 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:08.930+0000 7fa7da0a9700 1 --2- 172.21.15.195:0/2701370953 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa7d40c3d80 0x7fa7d4157890 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fa7cc009f20 tx=0x7fa7cc009840 comp rx=0 tx=0).stop 2022-06-27T12:09:08.972 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:08.930+0000 7fa7da0a9700 1 -- 172.21.15.195:0/2701370953 shutdown_connections 2022-06-27T12:09:08.972 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:08.930+0000 7fa7da0a9700 1 --2- 172.21.15.195:0/2701370953 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa7d40c3d80 0x7fa7d4157890 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:08.973 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:08.930+0000 7fa7da0a9700 1 --2- 172.21.15.195:0/2701370953 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa7d414e860 0x7fa7d40c3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:08.975 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:08.930+0000 7fa7da0a9700 1 -- 172.21.15.195:0/2701370953 >> 172.21.15.195:0/2701370953 conn(0x7fa7d4063af0 msgr2=0x7fa7d4063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:09:08.975 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:08.930+0000 7fa7da0a9700 1 -- 172.21.15.195:0/2701370953 shutdown_connections 2022-06-27T12:09:08.975 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:08.930+0000 7fa7da0a9700 1 -- 172.21.15.195:0/2701370953 wait complete. 2022-06-27T12:09:08.976 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:08.931+0000 7fa7da0a9700 1 Processor -- start 2022-06-27T12:09:08.976 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:08.931+0000 7fa7da0a9700 1 -- start start 2022-06-27T12:09:08.977 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:08.932+0000 7fa7da0a9700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa7d40c3d80 0x7fa7d40c8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:08.979 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:08.932+0000 7fa7da0a9700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa7d414e490 0x7fa7d406a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:08.979 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:08.932+0000 7fa7da0a9700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa7d414e860 0x7fa7d406a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:08.980 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:08.932+0000 7fa7da0a9700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7d406b020 con 0x7fa7d414e860 2022-06-27T12:09:08.981 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:08.932+0000 7fa7da0a9700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7d414ad80 con 0x7fa7d414e490 2022-06-27T12:09:08.984 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:08.932+0000 7fa7da0a9700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7d4069080 con 0x7fa7d40c3d80 2022-06-27T12:09:08.985 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:08.932+0000 7fa7d3fff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa7d414e860 0x7fa7d406a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:08.985 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:08.932+0000 7fa7d3fff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa7d414e860 0x7fa7d406a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:49374/0 (socket says 172.21.15.195:49374) 2022-06-27T12:09:08.986 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:08.932+0000 7fa7d3fff700 1 -- 172.21.15.195:0/2095491197 learned_addr learned my addr 172.21.15.195:0/2095491197 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:09:08.987 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:08.932+0000 7fa7d37fe700 1 --2- 172.21.15.195:0/2095491197 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa7d40c3d80 0x7fa7d40c8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:08.987 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:08.932+0000 7fa7d2ffd700 1 --2- 172.21.15.195:0/2095491197 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa7d414e490 0x7fa7d406a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:08.988 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:08.932+0000 7fa7d3fff700 1 -- 172.21.15.195:0/2095491197 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa7d40c3d80 msgr2=0x7fa7d40c8a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:08.988 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:08.932+0000 7fa7d3fff700 1 --2- 172.21.15.195:0/2095491197 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa7d40c3d80 0x7fa7d40c8a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:08.989 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:08.932+0000 7fa7d3fff700 1 -- 172.21.15.195:0/2095491197 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa7d414e490 msgr2=0x7fa7d406a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:08.989 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:08.932+0000 7fa7d3fff700 1 --2- 172.21.15.195:0/2095491197 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa7d414e490 0x7fa7d406a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:08.996 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:08.932+0000 7fa7d3fff700 1 -- 172.21.15.195:0/2095491197 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7cc009ae0 con 0x7fa7d414e860 2022-06-27T12:09:08.997 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:08.932+0000 7fa7d3fff700 1 --2- 172.21.15.195:0/2095491197 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa7d414e860 0x7fa7d406a620 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fa7cc00b610 tx=0x7fa7cc018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:08.998 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:08.933+0000 7fa7cb7fe700 1 -- 172.21.15.195:0/2095491197 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7cc003ea0 con 0x7fa7d414e860 2022-06-27T12:09:08.998 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:08.933+0000 7fa7cb7fe700 1 -- 172.21.15.195:0/2095491197 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7cc004250 con 0x7fa7d414e860 2022-06-27T12:09:08.998 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:08.933+0000 7fa7cb7fe700 1 -- 172.21.15.195:0/2095491197 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7cc004610 con 0x7fa7d414e860 2022-06-27T12:09:08.999 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:08.933+0000 7fa7da0a9700 1 -- 172.21.15.195:0/2095491197 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7d4069320 con 0x7fa7d414e860 2022-06-27T12:09:09.000 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:08.933+0000 7fa7da0a9700 1 -- 172.21.15.195:0/2095491197 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7d40698e0 con 0x7fa7d414e860 2022-06-27T12:09:09.000 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:08.934+0000 7fa7da0a9700 1 -- 172.21.15.195:0/2095491197 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7d40722c0 con 0x7fa7d414e860 2022-06-27T12:09:09.001 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:08.934+0000 7fa7cb7fe700 1 -- 172.21.15.195:0/2095491197 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fa7cc028460 con 0x7fa7d414e860 2022-06-27T12:09:09.001 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:08.934+0000 7fa7cb7fe700 1 --2- 172.21.15.195:0/2095491197 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa7b8063d00 0x7fa7b80661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:09.002 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:08.935+0000 7fa7cb7fe700 1 -- 172.21.15.195:0/2095491197 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7148+0+0 (secure 0 0 0) 0x7fa7cc082790 con 0x7fa7d414e860 2022-06-27T12:09:09.002 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:08.935+0000 7fa7d37fe700 1 --2- 172.21.15.195:0/2095491197 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa7b8063d00 0x7fa7b80661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:09.003 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:08.935+0000 7fa7d37fe700 1 --2- 172.21.15.195:0/2095491197 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa7b8063d00 0x7fa7b80661b0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc001900 tx=0x7fa7bc009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:09.004 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:08.940+0000 7fa7cb7fe700 1 -- 172.21.15.195:0/2095491197 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fa7cc050ee0 con 0x7fa7d414e860 2022-06-27T12:09:09.177 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.177+0000 7fa7da0a9700 1 -- 172.21.15.195:0/2095491197 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa7d40c60d0 con 0x7fa7d414e860 2022-06-27T12:09:09.179 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.178+0000 7fa7cb7fe700 1 -- 172.21.15.195:0/2095491197 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+6125 (secure 0 0 0) 0x7fa7cc054500 con 0x7fa7d414e860 2022-06-27T12:09:09.180 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:09:09.181 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":78,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":6708,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/702244802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":702244802},{"type":"v1","addr":"172.21.15.202:6833","nonce":702244802}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:08:23.133338+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6271},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6271":{"gid":6271,"name":"d","rank":0,"incarnation":73,"state":"up:active","state_seq":20,"addr":"172.21.15.202:6835/1603307620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":1603307620},{"type":"v1","addr":"172.21.15.202:6835","nonce":1603307620}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:09:09.183 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.180+0000 7fa7da0a9700 1 -- 172.21.15.195:0/2095491197 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa7b8063d00 msgr2=0x7fa7b80661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:09.184 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.180+0000 7fa7da0a9700 1 --2- 172.21.15.195:0/2095491197 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa7b8063d00 0x7fa7b80661b0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc001900 tx=0x7fa7bc009040 comp rx=0 tx=0).stop 2022-06-27T12:09:09.184 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.180+0000 7fa7da0a9700 1 -- 172.21.15.195:0/2095491197 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa7d414e860 msgr2=0x7fa7d406a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:09.185 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.180+0000 7fa7da0a9700 1 --2- 172.21.15.195:0/2095491197 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa7d414e860 0x7fa7d406a620 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fa7cc00b610 tx=0x7fa7cc018000 comp rx=0 tx=0).stop 2022-06-27T12:09:09.185 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.180+0000 7fa7da0a9700 1 -- 172.21.15.195:0/2095491197 shutdown_connections 2022-06-27T12:09:09.186 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.180+0000 7fa7da0a9700 1 --2- 172.21.15.195:0/2095491197 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa7b8063d00 0x7fa7b80661b0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:09.186 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.180+0000 7fa7da0a9700 1 --2- 172.21.15.195:0/2095491197 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa7d414e860 0x7fa7d406a620 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:09.186 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.180+0000 7fa7da0a9700 1 --2- 172.21.15.195:0/2095491197 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa7d40c3d80 0x7fa7d40c8a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:09.187 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.180+0000 7fa7da0a9700 1 --2- 172.21.15.195:0/2095491197 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa7d414e490 0x7fa7d406a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:09.187 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.180+0000 7fa7da0a9700 1 -- 172.21.15.195:0/2095491197 >> 172.21.15.195:0/2095491197 conn(0x7fa7d4063af0 msgr2=0x7fa7d414cae0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:09:09.187 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.180+0000 7fa7da0a9700 1 -- 172.21.15.195:0/2095491197 shutdown_connections 2022-06-27T12:09:09.188 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.180+0000 7fa7da0a9700 1 -- 172.21.15.195:0/2095491197 wait complete. 2022-06-27T12:09:09.188 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 78 2022-06-27T12:09:09.204 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.d (rank=0) 2022-06-27T12:09:09.205 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2022-06-27T12:09:09.207 INFO:teuthology.orchestra.run:waiting for 300 2022-06-27T12:09:09.229 INFO:tasks.ceph.mds.d:Stopped 2022-06-27T12:09:09.230 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:09:09.399 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.362+0000 7f21b724d700 1 Processor -- start 2022-06-27T12:09:09.400 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.362+0000 7f21b724d700 1 -- start start 2022-06-27T12:09:09.401 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.363+0000 7f21b724d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f21b006e850 0x7f21b01570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:09.402 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.363+0000 7f21b724d700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21b006fba0 con 0x7f21b006fef0 2022-06-27T12:09:09.402 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.363+0000 7f21b724d700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21b006f5e0 con 0x7f21b00c34a0 2022-06-27T12:09:09.403 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.363+0000 7f21b724d700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21b01575e0 con 0x7f21b006e850 2022-06-27T12:09:09.403 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.363+0000 7f21b4fe9700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f21b006e850 0x7f21b01570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:09.404 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.363+0000 7f21affff700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f21b006fef0 0x7f21b0069de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:55566/0 (socket says 172.21.15.195:55566) 2022-06-27T12:09:09.404 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.363+0000 7f21affff700 1 -- 172.21.15.195:0/362152382 learned_addr learned my addr 172.21.15.195:0/362152382 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:09:09.404 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.364+0000 7f21b4fe9700 1 -- 172.21.15.195:0/362152382 >> v1:172.21.15.202:6789/0 conn(0x7f21b00c34a0 legacy=0x7f21b00c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:09.405 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.364+0000 7f21b4fe9700 1 -- 172.21.15.195:0/362152382 >> v1:172.21.15.195:6789/0 conn(0x7f21b006fef0 legacy=0x7f21b0069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:09.405 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.364+0000 7f21b4fe9700 1 -- 172.21.15.195:0/362152382 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21b006f8c0 con 0x7f21b006e850 2022-06-27T12:09:09.406 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.364+0000 7f21af7fe700 1 -- 172.21.15.195:0/362152382 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1330253362 0 0) 0x7f21b006fba0 con 0x7f21b006fef0 2022-06-27T12:09:09.407 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.364+0000 7f21b4fe9700 1 --2- 172.21.15.195:0/362152382 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f21b006e850 0x7f21b01570a0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f21a400ba40 tx=0x7f21a400b300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c6a4783395165b9 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:09.407 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.364+0000 7f21af7fe700 1 -- 172.21.15.195:0/362152382 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a400db00 con 0x7f21b006e850 2022-06-27T12:09:09.410 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.364+0000 7f21af7fe700 1 -- 172.21.15.195:0/362152382 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21a400dc60 con 0x7f21b006e850 2022-06-27T12:09:09.422 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.364+0000 7f21af7fe700 1 -- 172.21.15.195:0/362152382 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a4019480 con 0x7f21b006e850 2022-06-27T12:09:09.423 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.364+0000 7f21b724d700 1 -- 172.21.15.195:0/362152382 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f21b006e850 msgr2=0x7f21b01570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:09.423 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.364+0000 7f21b724d700 1 --2- 172.21.15.195:0/362152382 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f21b006e850 0x7f21b01570a0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f21a400ba40 tx=0x7f21a400b300 comp rx=0 tx=0).stop 2022-06-27T12:09:09.424 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.364+0000 7f21b724d700 1 -- 172.21.15.195:0/362152382 shutdown_connections 2022-06-27T12:09:09.424 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.364+0000 7f21b724d700 1 --2- 172.21.15.195:0/362152382 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f21b006e850 0x7f21b01570a0 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:09.425 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.364+0000 7f21b724d700 1 -- 172.21.15.195:0/362152382 >> 172.21.15.195:0/362152382 conn(0x7f21b00c6350 msgr2=0x7f21b00c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:09:09.425 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.365+0000 7f21b724d700 1 -- 172.21.15.195:0/362152382 shutdown_connections 2022-06-27T12:09:09.426 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.365+0000 7f21b724d700 1 -- 172.21.15.195:0/362152382 wait complete. 2022-06-27T12:09:09.426 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.366+0000 7f21b724d700 1 Processor -- start 2022-06-27T12:09:09.427 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.366+0000 7f21b724d700 1 -- start start 2022-06-27T12:09:09.427 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.366+0000 7f21b724d700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f21b006e850 0x7f21b014e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:09.428 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.366+0000 7f21b724d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f21b00c34a0 0x7f21b00c2a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:09.428 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.366+0000 7f21b724d700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f21b0147600 0x7f21b0147a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:09.429 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.366+0000 7f21b724d700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21b0147fb0 con 0x7f21b00c34a0 2022-06-27T12:09:09.431 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.366+0000 7f21b724d700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21b014e9a0 con 0x7f21b0147600 2022-06-27T12:09:09.432 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.367+0000 7f21b724d700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21b014e710 con 0x7f21b006e850 2022-06-27T12:09:09.432 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.367+0000 7f21affff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f21b00c34a0 0x7f21b00c2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:09.433 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.367+0000 7f21affff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f21b00c34a0 0x7f21b00c2a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:49386/0 (socket says 172.21.15.195:49386) 2022-06-27T12:09:09.442 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.367+0000 7f21affff700 1 -- 172.21.15.195:0/992007660 learned_addr learned my addr 172.21.15.195:0/992007660 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:09:09.443 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.367+0000 7f21b4fe9700 1 --2- 172.21.15.195:0/992007660 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f21b006e850 0x7f21b014e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:09.443 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.367+0000 7f21b57ea700 1 --2- 172.21.15.195:0/992007660 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f21b0147600 0x7f21b0147a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:09.444 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.367+0000 7f21affff700 1 -- 172.21.15.195:0/992007660 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f21b006e850 msgr2=0x7f21b014e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:09.444 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.367+0000 7f21affff700 1 --2- 172.21.15.195:0/992007660 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f21b006e850 0x7f21b014e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:09.445 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.367+0000 7f21affff700 1 -- 172.21.15.195:0/992007660 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f21b0147600 msgr2=0x7f21b0147a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:09.461 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.367+0000 7f21affff700 1 --2- 172.21.15.195:0/992007660 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f21b0147600 0x7f21b0147a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:09.463 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.367+0000 7f21affff700 1 -- 172.21.15.195:0/992007660 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21a400b580 con 0x7f21b00c34a0 2022-06-27T12:09:09.464 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.367+0000 7f21affff700 1 --2- 172.21.15.195:0/992007660 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f21b00c34a0 0x7f21b00c2a80 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f219c00c490 tx=0x7f219c007de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:09.464 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.367+0000 7f21acff9700 1 -- 172.21.15.195:0/992007660 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f219c007360 con 0x7f21b00c34a0 2022-06-27T12:09:09.465 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.367+0000 7f21acff9700 1 -- 172.21.15.195:0/992007660 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f219c0074c0 con 0x7f21b00c34a0 2022-06-27T12:09:09.466 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.367+0000 7f21b724d700 1 -- 172.21.15.195:0/992007660 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21b0160c40 con 0x7f21b00c34a0 2022-06-27T12:09:09.466 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.367+0000 7f21acff9700 1 -- 172.21.15.195:0/992007660 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f219c004940 con 0x7f21b00c34a0 2022-06-27T12:09:09.467 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.367+0000 7f21b724d700 1 -- 172.21.15.195:0/992007660 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21b0166950 con 0x7f21b00c34a0 2022-06-27T12:09:09.467 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.368+0000 7f21b724d700 1 -- 172.21.15.195:0/992007660 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21b00c7850 con 0x7f21b00c34a0 2022-06-27T12:09:09.468 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.369+0000 7f21acff9700 1 -- 172.21.15.195:0/992007660 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f219c01a020 con 0x7f21b00c34a0 2022-06-27T12:09:09.468 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.369+0000 7f21acff9700 1 --2- 172.21.15.195:0/992007660 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2198063d00 0x7f21980661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:09.471 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.369+0000 7f21acff9700 1 -- 172.21.15.195:0/992007660 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7148+0+0 (secure 0 0 0) 0x7f219c0806e0 con 0x7f21b00c34a0 2022-06-27T12:09:09.474 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.369+0000 7f21b4fe9700 1 --2- 172.21.15.195:0/992007660 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2198063d00 0x7f21980661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:09.475 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.370+0000 7f21b4fe9700 1 --2- 172.21.15.195:0/992007660 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2198063d00 0x7f21980661b0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f21a400b340 tx=0x7f21a401b000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:09.475 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.373+0000 7f21acff9700 1 -- 172.21.15.195:0/992007660 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f219c04ef10 con 0x7f21b00c34a0 2022-06-27T12:09:09.597 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.597+0000 7f21b724d700 1 -- 172.21.15.195:0/992007660 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f21b00722c0 con 0x7f21b00c34a0 2022-06-27T12:09:09.599 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.598+0000 7f21acff9700 1 -- 172.21.15.195:0/992007660 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+6125 (secure 0 0 0) 0x7f219c052530 con 0x7f21b00c34a0 2022-06-27T12:09:09.600 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:09:09.600 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":78,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":6708,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/702244802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":702244802},{"type":"v1","addr":"172.21.15.202:6833","nonce":702244802}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:08:23.133338+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6271},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6271":{"gid":6271,"name":"d","rank":0,"incarnation":73,"state":"up:active","state_seq":20,"addr":"172.21.15.202:6835/1603307620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":1603307620},{"type":"v1","addr":"172.21.15.202:6835","nonce":1603307620}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:09:09.603 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.600+0000 7f21b724d700 1 -- 172.21.15.195:0/992007660 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2198063d00 msgr2=0x7f21980661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:09.603 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.600+0000 7f21b724d700 1 --2- 172.21.15.195:0/992007660 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2198063d00 0x7f21980661b0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f21a400b340 tx=0x7f21a401b000 comp rx=0 tx=0).stop 2022-06-27T12:09:09.603 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.600+0000 7f21b724d700 1 -- 172.21.15.195:0/992007660 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f21b00c34a0 msgr2=0x7f21b00c2a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:09.604 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.600+0000 7f21b724d700 1 --2- 172.21.15.195:0/992007660 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f21b00c34a0 0x7f21b00c2a80 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f219c00c490 tx=0x7f219c007de0 comp rx=0 tx=0).stop 2022-06-27T12:09:09.604 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.600+0000 7f21b724d700 1 -- 172.21.15.195:0/992007660 shutdown_connections 2022-06-27T12:09:09.604 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.600+0000 7f21b724d700 1 --2- 172.21.15.195:0/992007660 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2198063d00 0x7f21980661b0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:09.604 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.600+0000 7f21b724d700 1 --2- 172.21.15.195:0/992007660 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f21b0147600 0x7f21b0147a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:09.605 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.600+0000 7f21b724d700 1 --2- 172.21.15.195:0/992007660 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f21b006e850 0x7f21b014e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:09.605 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.600+0000 7f21b724d700 1 --2- 172.21.15.195:0/992007660 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f21b00c34a0 0x7f21b00c2a80 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:09.605 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.600+0000 7f21b724d700 1 -- 172.21.15.195:0/992007660 >> 172.21.15.195:0/992007660 conn(0x7f21b00c6350 msgr2=0x7f21b00c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:09:09.608 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.600+0000 7f21b724d700 1 -- 172.21.15.195:0/992007660 shutdown_connections 2022-06-27T12:09:09.609 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:09.600+0000 7f21b724d700 1 -- 172.21.15.195:0/992007660 wait complete. 2022-06-27T12:09:09.609 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 78 2022-06-27T12:09:09.612 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-06-27T12:09:11.614 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:09:11.755 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.755+0000 7fe67aad8700 1 Processor -- start 2022-06-27T12:09:11.756 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.755+0000 7fe67aad8700 1 -- start start 2022-06-27T12:09:11.779 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.756+0000 7fe67aad8700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fe67406e850 0x7fe6741570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:11.780 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.756+0000 7fe67aad8700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe67406fba0 con 0x7fe6740c34a0 2022-06-27T12:09:11.780 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.756+0000 7fe67aad8700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe67406f5e0 con 0x7fe67406fef0 2022-06-27T12:09:11.780 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.756+0000 7fe67aad8700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6741575e0 con 0x7fe67406e850 2022-06-27T12:09:11.781 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.756+0000 7fe678874700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fe67406e850 0x7fe6741570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:11.781 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.756+0000 7fe679075700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fe6740c34a0 0x7fe6740c2920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:55590/0 (socket says 172.21.15.195:55590) 2022-06-27T12:09:11.782 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.756+0000 7fe679075700 1 -- 172.21.15.195:0/2605103413 learned_addr learned my addr 172.21.15.195:0/2605103413 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:09:11.782 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.756+0000 7fe678874700 1 -- 172.21.15.195:0/2605103413 >> v1:172.21.15.202:6789/0 conn(0x7fe67406fef0 legacy=0x7fe674069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:09:11.782 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.756+0000 7fe678874700 1 -- 172.21.15.195:0/2605103413 >> v1:172.21.15.195:6789/0 conn(0x7fe6740c34a0 legacy=0x7fe6740c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:11.782 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.756+0000 7fe678874700 1 -- 172.21.15.195:0/2605103413 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe67406f8c0 con 0x7fe67406e850 2022-06-27T12:09:11.783 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.756+0000 7fe6737fe700 1 -- 172.21.15.195:0/2605103413 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2242317996 0 0) 0x7fe67406fba0 con 0x7fe6740c34a0 2022-06-27T12:09:11.783 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.756+0000 7fe678874700 1 --2- 172.21.15.195:0/2605103413 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe67406e850 0x7fe6741570a0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fe6680097a0 tx=0x7fe668009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4de9dc63319fc31 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:11.783 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.757+0000 7fe6737fe700 1 -- 172.21.15.195:0/2605103413 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe668004030 con 0x7fe67406e850 2022-06-27T12:09:11.783 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.757+0000 7fe6737fe700 1 -- 172.21.15.195:0/2605103413 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe668004190 con 0x7fe67406e850 2022-06-27T12:09:11.784 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.757+0000 7fe6737fe700 1 -- 172.21.15.195:0/2605103413 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe668011630 con 0x7fe67406e850 2022-06-27T12:09:11.784 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.757+0000 7fe67aad8700 1 -- 172.21.15.195:0/2605103413 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe67406e850 msgr2=0x7fe6741570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:11.784 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.757+0000 7fe67aad8700 1 --2- 172.21.15.195:0/2605103413 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe67406e850 0x7fe6741570a0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fe6680097a0 tx=0x7fe668009060 comp rx=0 tx=0).stop 2022-06-27T12:09:11.784 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.757+0000 7fe67aad8700 1 -- 172.21.15.195:0/2605103413 shutdown_connections 2022-06-27T12:09:11.785 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.757+0000 7fe67aad8700 1 --2- 172.21.15.195:0/2605103413 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe67406e850 0x7fe6741570a0 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:11.785 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.757+0000 7fe67aad8700 1 -- 172.21.15.195:0/2605103413 >> 172.21.15.195:0/2605103413 conn(0x7fe6740c6350 msgr2=0x7fe6740c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:09:11.785 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.757+0000 7fe67aad8700 1 -- 172.21.15.195:0/2605103413 shutdown_connections 2022-06-27T12:09:11.785 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.757+0000 7fe67aad8700 1 -- 172.21.15.195:0/2605103413 wait complete. 2022-06-27T12:09:11.786 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.758+0000 7fe67aad8700 1 Processor -- start 2022-06-27T12:09:11.786 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.758+0000 7fe67aad8700 1 -- start start 2022-06-27T12:09:11.786 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.759+0000 7fe67aad8700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe67406e850 0x7fe6740bf970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:11.786 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.759+0000 7fe67aad8700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe6740c34a0 0x7fe674069fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:11.787 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.759+0000 7fe67aad8700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe674158690 0x7fe674166a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:11.787 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.759+0000 7fe67aad8700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6741641c0 con 0x7fe67406e850 2022-06-27T12:09:11.787 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.759+0000 7fe67aad8700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6740c0290 con 0x7fe6740c34a0 2022-06-27T12:09:11.787 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.759+0000 7fe67aad8700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe674163540 con 0x7fe674158690 2022-06-27T12:09:11.788 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.759+0000 7fe678874700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe67406e850 0x7fe6740bf970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:11.788 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.759+0000 7fe678874700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe67406e850 0x7fe6740bf970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:49404/0 (socket says 172.21.15.195:49404) 2022-06-27T12:09:11.788 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.759+0000 7fe678874700 1 -- 172.21.15.195:0/2554111719 learned_addr learned my addr 172.21.15.195:0/2554111719 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:09:11.788 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.759+0000 7fe673fff700 1 --2- 172.21.15.195:0/2554111719 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe6740c34a0 0x7fe674069fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:11.789 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.759+0000 7fe679075700 1 --2- 172.21.15.195:0/2554111719 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe674158690 0x7fe674166a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:11.789 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.759+0000 7fe678874700 1 -- 172.21.15.195:0/2554111719 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe674158690 msgr2=0x7fe674166a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:11.789 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.759+0000 7fe678874700 1 --2- 172.21.15.195:0/2554111719 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe674158690 0x7fe674166a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:11.789 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.759+0000 7fe678874700 1 -- 172.21.15.195:0/2554111719 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe6740c34a0 msgr2=0x7fe674069fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:11.790 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.760+0000 7fe678874700 1 --2- 172.21.15.195:0/2554111719 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe6740c34a0 0x7fe674069fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:11.790 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.760+0000 7fe678874700 1 -- 172.21.15.195:0/2554111719 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6680092e0 con 0x7fe67406e850 2022-06-27T12:09:11.790 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.760+0000 7fe673fff700 1 --2- 172.21.15.195:0/2554111719 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe6740c34a0 0x7fe674069fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T12:09:11.791 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.760+0000 7fe678874700 1 --2- 172.21.15.195:0/2554111719 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe67406e850 0x7fe6740bf970 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fe668011d50 tx=0x7fe668016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:11.791 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.760+0000 7fe670ff9700 1 -- 172.21.15.195:0/2554111719 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe668020070 con 0x7fe67406e850 2022-06-27T12:09:11.791 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.760+0000 7fe670ff9700 1 -- 172.21.15.195:0/2554111719 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe66801b9b0 con 0x7fe67406e850 2022-06-27T12:09:11.791 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.760+0000 7fe67aad8700 1 -- 172.21.15.195:0/2554111719 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe674163820 con 0x7fe67406e850 2022-06-27T12:09:11.792 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.760+0000 7fe670ff9700 1 -- 172.21.15.195:0/2554111719 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe66801bd70 con 0x7fe67406e850 2022-06-27T12:09:11.792 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.760+0000 7fe67aad8700 1 -- 172.21.15.195:0/2554111719 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe674163de0 con 0x7fe67406e850 2022-06-27T12:09:11.792 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.761+0000 7fe67aad8700 1 -- 172.21.15.195:0/2554111719 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6740722c0 con 0x7fe67406e850 2022-06-27T12:09:11.792 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.761+0000 7fe670ff9700 1 -- 172.21.15.195:0/2554111719 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fe66802b030 con 0x7fe67406e850 2022-06-27T12:09:11.793 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.762+0000 7fe670ff9700 1 --2- 172.21.15.195:0/2554111719 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe65c063d00 0x7fe65c0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:11.793 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.762+0000 7fe670ff9700 1 -- 172.21.15.195:0/2554111719 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7148+0+0 (secure 0 0 0) 0x7fe668084050 con 0x7fe67406e850 2022-06-27T12:09:11.793 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.762+0000 7fe673fff700 1 --2- 172.21.15.195:0/2554111719 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe65c063d00 0x7fe65c0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:11.793 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.762+0000 7fe673fff700 1 --2- 172.21.15.195:0/2554111719 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe65c063d00 0x7fe65c0661b0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fe6740c2c80 tx=0x7fe660008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:11.794 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.766+0000 7fe670ff9700 1 -- 172.21.15.195:0/2554111719 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fe668052800 con 0x7fe67406e850 2022-06-27T12:09:11.993 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.993+0000 7fe67aad8700 1 -- 172.21.15.195:0/2554111719 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe67406f240 con 0x7fe67406e850 2022-06-27T12:09:11.995 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.993+0000 7fe670ff9700 1 -- 172.21.15.195:0/2554111719 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+6125 (secure 0 0 0) 0x7fe668029030 con 0x7fe67406e850 2022-06-27T12:09:11.995 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:09:11.996 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":78,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":6708,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/702244802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":702244802},{"type":"v1","addr":"172.21.15.202:6833","nonce":702244802}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:08:23.133338+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6271},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6271":{"gid":6271,"name":"d","rank":0,"incarnation":73,"state":"up:active","state_seq":20,"addr":"172.21.15.202:6835/1603307620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":1603307620},{"type":"v1","addr":"172.21.15.202:6835","nonce":1603307620}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:09:12.000 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.995+0000 7fe67aad8700 1 -- 172.21.15.195:0/2554111719 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe65c063d00 msgr2=0x7fe65c0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:12.000 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.995+0000 7fe67aad8700 1 --2- 172.21.15.195:0/2554111719 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe65c063d00 0x7fe65c0661b0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fe6740c2c80 tx=0x7fe660008040 comp rx=0 tx=0).stop 2022-06-27T12:09:12.001 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.995+0000 7fe67aad8700 1 -- 172.21.15.195:0/2554111719 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe67406e850 msgr2=0x7fe6740bf970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:12.001 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.995+0000 7fe67aad8700 1 --2- 172.21.15.195:0/2554111719 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe67406e850 0x7fe6740bf970 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fe668011d50 tx=0x7fe668016000 comp rx=0 tx=0).stop 2022-06-27T12:09:12.001 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.995+0000 7fe67aad8700 1 -- 172.21.15.195:0/2554111719 shutdown_connections 2022-06-27T12:09:12.001 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.995+0000 7fe67aad8700 1 --2- 172.21.15.195:0/2554111719 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe65c063d00 0x7fe65c0661b0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:12.002 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.995+0000 7fe67aad8700 1 --2- 172.21.15.195:0/2554111719 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe67406e850 0x7fe6740bf970 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:12.002 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.995+0000 7fe67aad8700 1 --2- 172.21.15.195:0/2554111719 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe674158690 0x7fe674166a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:12.003 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.995+0000 7fe67aad8700 1 --2- 172.21.15.195:0/2554111719 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe6740c34a0 0x7fe674069fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:12.003 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.995+0000 7fe67aad8700 1 -- 172.21.15.195:0/2554111719 >> 172.21.15.195:0/2554111719 conn(0x7fe6740c6350 msgr2=0x7fe674153020 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:09:12.011 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.995+0000 7fe67aad8700 1 -- 172.21.15.195:0/2554111719 shutdown_connections 2022-06-27T12:09:12.011 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:11.995+0000 7fe67aad8700 1 -- 172.21.15.195:0/2554111719 wait complete. 2022-06-27T12:09:12.012 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 78 2022-06-27T12:09:12.012 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-06-27T12:09:14.013 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:09:14.169 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.161+0000 7fdddb7e2700 1 Processor -- start 2022-06-27T12:09:14.169 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.162+0000 7fdddb7e2700 1 -- start start 2022-06-27T12:09:14.170 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.162+0000 7fdddb7e2700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fddd406f640 0x7fddd4156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:14.170 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.162+0000 7fdddb7e2700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fddd40c39e0 0x7fddd41535a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:14.177 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.162+0000 7fdddb7e2700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddd40c1a20 con 0x7fddd40c4160 2022-06-27T12:09:14.177 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.162+0000 7fdddb7e2700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddd40c1130 con 0x7fddd406f640 2022-06-27T12:09:14.180 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.163+0000 7fdddb7e2700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddd406ea90 con 0x7fddd40c39e0 2022-06-27T12:09:14.180 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.163+0000 7fddd957e700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fddd406f640 0x7fddd4156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:14.180 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.163+0000 7fddd957e700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fddd406f640 0x7fddd4156ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:49416/0 (socket says 172.21.15.195:49416) 2022-06-27T12:09:14.181 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.163+0000 7fddd957e700 1 -- 172.21.15.195:0/3451371124 learned_addr learned my addr 172.21.15.195:0/3451371124 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:09:14.181 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.163+0000 7fddd9d7f700 1 --2- 172.21.15.195:0/3451371124 >> v2:172.21.15.202:3300/0 conn(0x7fddd40c39e0 0x7fddd41535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:14.183 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.163+0000 7fddd957e700 1 -- 172.21.15.195:0/3451371124 >> v2:172.21.15.202:3300/0 conn(0x7fddd40c39e0 msgr2=0x7fddd41535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:09:14.224 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.163+0000 7fddd957e700 1 --2- 172.21.15.195:0/3451371124 >> v2:172.21.15.202:3300/0 conn(0x7fddd40c39e0 0x7fddd41535a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:14.224 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.163+0000 7fddd957e700 1 -- 172.21.15.195:0/3451371124 >> v1:172.21.15.202:6789/0 conn(0x7fddd40c4160 legacy=0x7fddd40c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:14.225 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.163+0000 7fddd957e700 1 -- 172.21.15.195:0/3451371124 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddd406ed70 con 0x7fddd406f640 2022-06-27T12:09:14.225 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.163+0000 7fddd9d7f700 1 --2- 172.21.15.195:0/3451371124 >> v2:172.21.15.202:3300/0 conn(0x7fddd40c39e0 0x7fddd41535a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:09:14.226 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.163+0000 7fddd957e700 1 --2- 172.21.15.195:0/3451371124 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fddd406f640 0x7fddd4156ed0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fddd4065ef0 tx=0x7fddc4009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccb95f7789732187 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:14.234 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.163+0000 7fddc3fff700 1 -- 172.21.15.195:0/3451371124 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc400b6d0 con 0x7fddd406f640 2022-06-27T12:09:14.234 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.163+0000 7fddc3fff700 1 -- 172.21.15.195:0/3451371124 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddc400b830 con 0x7fddd406f640 2022-06-27T12:09:14.235 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.163+0000 7fddc3fff700 1 -- 172.21.15.195:0/3451371124 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc400bbd0 con 0x7fddd406f640 2022-06-27T12:09:14.235 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.164+0000 7fdddb7e2700 1 -- 172.21.15.195:0/3451371124 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fddd406f640 msgr2=0x7fddd4156ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:14.236 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.164+0000 7fdddb7e2700 1 --2- 172.21.15.195:0/3451371124 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fddd406f640 0x7fddd4156ed0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fddd4065ef0 tx=0x7fddc4009120 comp rx=0 tx=0).stop 2022-06-27T12:09:14.236 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.164+0000 7fdddb7e2700 1 -- 172.21.15.195:0/3451371124 shutdown_connections 2022-06-27T12:09:14.236 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.164+0000 7fdddb7e2700 1 --2- 172.21.15.195:0/3451371124 >> v2:172.21.15.202:3300/0 conn(0x7fddd40c39e0 0x7fddd41535a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:14.249 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.164+0000 7fdddb7e2700 1 --2- 172.21.15.195:0/3451371124 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fddd406f640 0x7fddd4156ed0 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:14.250 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.164+0000 7fdddb7e2700 1 -- 172.21.15.195:0/3451371124 >> 172.21.15.195:0/3451371124 conn(0x7fddd40c6350 msgr2=0x7fddd40c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:09:14.250 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.164+0000 7fdddb7e2700 1 -- 172.21.15.195:0/3451371124 shutdown_connections 2022-06-27T12:09:14.251 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.164+0000 7fdddb7e2700 1 -- 172.21.15.195:0/3451371124 wait complete. 2022-06-27T12:09:14.251 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.165+0000 7fdddb7e2700 1 Processor -- start 2022-06-27T12:09:14.252 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.165+0000 7fdddb7e2700 1 -- start start 2022-06-27T12:09:14.252 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.166+0000 7fdddb7e2700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fddd406f640 0x7fddd4150530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:14.252 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.166+0000 7fdddb7e2700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fddd40c39e0 0x7fddd414f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:14.253 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.166+0000 7fdddb7e2700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fddd40c4160 0x7fddd41698d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:14.254 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.166+0000 7fdddb7e2700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddd414f130 con 0x7fddd40c4160 2022-06-27T12:09:14.258 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.166+0000 7fdddb7e2700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddd414e6b0 con 0x7fddd40c39e0 2022-06-27T12:09:14.259 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.166+0000 7fdddb7e2700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddd415b210 con 0x7fddd406f640 2022-06-27T12:09:14.260 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.166+0000 7fddd9d7f700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fddd40c4160 0x7fddd41698d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:14.260 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.166+0000 7fddd9d7f700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fddd40c4160 0x7fddd41698d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:49426/0 (socket says 172.21.15.195:49426) 2022-06-27T12:09:14.261 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.166+0000 7fddd9d7f700 1 -- 172.21.15.195:0/4075576102 learned_addr learned my addr 172.21.15.195:0/4075576102 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:09:14.261 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.166+0000 7fddd8d7d700 1 --2- 172.21.15.195:0/4075576102 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fddd40c39e0 0x7fddd414f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:14.262 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.166+0000 7fddd957e700 1 --2- 172.21.15.195:0/4075576102 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fddd406f640 0x7fddd4150530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:14.263 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.166+0000 7fddd9d7f700 1 -- 172.21.15.195:0/4075576102 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fddd406f640 msgr2=0x7fddd4150530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:14.263 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.166+0000 7fddd9d7f700 1 --2- 172.21.15.195:0/4075576102 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fddd406f640 0x7fddd4150530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:14.264 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.166+0000 7fddd9d7f700 1 -- 172.21.15.195:0/4075576102 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fddd40c39e0 msgr2=0x7fddd414f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:14.264 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.166+0000 7fddd9d7f700 1 --2- 172.21.15.195:0/4075576102 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fddd40c39e0 0x7fddd414f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:14.266 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.166+0000 7fddd9d7f700 1 -- 172.21.15.195:0/4075576102 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddc40093a0 con 0x7fddd40c4160 2022-06-27T12:09:14.267 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.167+0000 7fddd9d7f700 1 --2- 172.21.15.195:0/4075576102 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fddd40c4160 0x7fddd41698d0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fddcc00d1b0 tx=0x7fddcc009a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:14.267 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.167+0000 7fddc17fa700 1 -- 172.21.15.195:0/4075576102 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddcc00dcf0 con 0x7fddd40c4160 2022-06-27T12:09:14.268 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.167+0000 7fddc17fa700 1 -- 172.21.15.195:0/4075576102 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddcc00f040 con 0x7fddd40c4160 2022-06-27T12:09:14.268 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.167+0000 7fdddb7e2700 1 -- 172.21.15.195:0/4075576102 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddd415af90 con 0x7fddd40c4160 2022-06-27T12:09:14.269 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.167+0000 7fddc17fa700 1 -- 172.21.15.195:0/4075576102 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddcc003af0 con 0x7fddd40c4160 2022-06-27T12:09:14.271 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.167+0000 7fdddb7e2700 1 -- 172.21.15.195:0/4075576102 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddd4166fa0 con 0x7fddd40c4160 2022-06-27T12:09:14.272 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.168+0000 7fdddb7e2700 1 -- 172.21.15.195:0/4075576102 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddd4166e00 con 0x7fddd40c4160 2022-06-27T12:09:14.272 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.168+0000 7fddc17fa700 1 -- 172.21.15.195:0/4075576102 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fddcc01c030 con 0x7fddd40c4160 2022-06-27T12:09:14.273 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.168+0000 7fddc17fa700 1 --2- 172.21.15.195:0/4075576102 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fddb4063d00 0x7fddb40661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:14.273 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.169+0000 7fddc17fa700 1 -- 172.21.15.195:0/4075576102 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7148+0+0 (secure 0 0 0) 0x7fddcc080d30 con 0x7fddd40c4160 2022-06-27T12:09:14.274 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.169+0000 7fddd957e700 1 --2- 172.21.15.195:0/4075576102 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fddb4063d00 0x7fddb40661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:14.274 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.169+0000 7fddd957e700 1 --2- 172.21.15.195:0/4075576102 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fddb4063d00 0x7fddb40661b0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fddc4009160 tx=0x7fddc4017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:14.275 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.173+0000 7fddc17fa700 1 -- 172.21.15.195:0/4075576102 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fddd4166e00 con 0x7fddd40c4160 2022-06-27T12:09:14.405 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.394+0000 7fdddb7e2700 1 -- 172.21.15.195:0/4075576102 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fddd4166e00 con 0x7fddd40c4160 2022-06-27T12:09:14.405 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.394+0000 7fddc17fa700 1 -- 172.21.15.195:0/4075576102 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+6125 (secure 0 0 0) 0x7fddd4166e00 con 0x7fddd40c4160 2022-06-27T12:09:14.406 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:09:14.408 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":78,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":6708,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/702244802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":702244802},{"type":"v1","addr":"172.21.15.202:6833","nonce":702244802}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:08:23.133338+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6271},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6271":{"gid":6271,"name":"d","rank":0,"incarnation":73,"state":"up:active","state_seq":20,"addr":"172.21.15.202:6835/1603307620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":1603307620},{"type":"v1","addr":"172.21.15.202:6835","nonce":1603307620}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:09:14.410 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.396+0000 7fdddb7e2700 1 -- 172.21.15.195:0/4075576102 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fddb4063d00 msgr2=0x7fddb40661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:14.413 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.396+0000 7fdddb7e2700 1 --2- 172.21.15.195:0/4075576102 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fddb4063d00 0x7fddb40661b0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fddc4009160 tx=0x7fddc4017000 comp rx=0 tx=0).stop 2022-06-27T12:09:14.414 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.396+0000 7fdddb7e2700 1 -- 172.21.15.195:0/4075576102 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fddd40c4160 msgr2=0x7fddd41698d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:14.415 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.396+0000 7fdddb7e2700 1 --2- 172.21.15.195:0/4075576102 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fddd40c4160 0x7fddd41698d0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fddcc00d1b0 tx=0x7fddcc009a50 comp rx=0 tx=0).stop 2022-06-27T12:09:14.415 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.396+0000 7fdddb7e2700 1 -- 172.21.15.195:0/4075576102 shutdown_connections 2022-06-27T12:09:14.415 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.396+0000 7fdddb7e2700 1 --2- 172.21.15.195:0/4075576102 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fddb4063d00 0x7fddb40661b0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:14.416 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.396+0000 7fdddb7e2700 1 --2- 172.21.15.195:0/4075576102 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fddd40c4160 0x7fddd41698d0 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:14.426 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.396+0000 7fdddb7e2700 1 --2- 172.21.15.195:0/4075576102 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fddd406f640 0x7fddd4150530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:14.427 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.396+0000 7fdddb7e2700 1 --2- 172.21.15.195:0/4075576102 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fddd40c39e0 0x7fddd414f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:14.427 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.396+0000 7fdddb7e2700 1 -- 172.21.15.195:0/4075576102 >> 172.21.15.195:0/4075576102 conn(0x7fddd40c6350 msgr2=0x7fddd406c6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:09:14.428 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.396+0000 7fdddb7e2700 1 -- 172.21.15.195:0/4075576102 shutdown_connections 2022-06-27T12:09:14.428 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:14.396+0000 7fdddb7e2700 1 -- 172.21.15.195:0/4075576102 wait complete. 2022-06-27T12:09:14.429 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 78 2022-06-27T12:09:14.449 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-06-27T12:09:16.451 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:09:16.665 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.595+0000 7f31f2f4b700 1 Processor -- start 2022-06-27T12:09:16.665 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.595+0000 7f31f2f4b700 1 -- start start 2022-06-27T12:09:16.666 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.596+0000 7f31f2f4b700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f31ec06f640 0x7f31ec156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:16.666 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.596+0000 7f31f2f4b700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f31ec0c4160 0x7f31ec0c1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:16.666 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.596+0000 7f31f2f4b700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31ec0c1a20 con 0x7f31ec0c39e0 2022-06-27T12:09:16.667 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.596+0000 7f31f2f4b700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31ec0c1130 con 0x7f31ec06f640 2022-06-27T12:09:16.667 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.596+0000 7f31f2f4b700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31ec06ea90 con 0x7f31ec0c4160 2022-06-27T12:09:16.668 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.596+0000 7f31f0ce7700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f31ec06f640 0x7f31ec156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:16.668 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.596+0000 7f31f14e8700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f31ec0c39e0 0x7f31ec1535a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:55620/0 (socket says 172.21.15.195:55620) 2022-06-27T12:09:16.668 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.596+0000 7f31f14e8700 1 -- 172.21.15.195:0/4279509717 learned_addr learned my addr 172.21.15.195:0/4279509717 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:09:16.669 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.596+0000 7f31ebfff700 1 --2- 172.21.15.195:0/4279509717 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f31ec0c4160 0x7f31ec0c1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:16.677 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.596+0000 7f31f0ce7700 1 -- 172.21.15.195:0/4279509717 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f31ec0c4160 msgr2=0x7f31ec0c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:09:16.677 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.596+0000 7f31f0ce7700 1 --2- 172.21.15.195:0/4279509717 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f31ec0c4160 0x7f31ec0c1eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:16.678 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.596+0000 7f31f0ce7700 1 -- 172.21.15.195:0/4279509717 >> v1:172.21.15.195:6789/0 conn(0x7f31ec0c39e0 legacy=0x7f31ec1535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:16.678 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.596+0000 7f31f0ce7700 1 -- 172.21.15.195:0/4279509717 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31ec06ed70 con 0x7f31ec06f640 2022-06-27T12:09:16.682 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.596+0000 7f31eb7fe700 1 -- 172.21.15.195:0/4279509717 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1120143301 0 0) 0x7f31ec0c1a20 con 0x7f31ec0c39e0 2022-06-27T12:09:16.682 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.597+0000 7f31f0ce7700 1 --2- 172.21.15.195:0/4279509717 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f31ec06f640 0x7f31ec156ed0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f31e00097a0 tx=0x7f31e0009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0a0a031a06b5b5b server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:16.682 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.597+0000 7f31eb7fe700 1 -- 172.21.15.195:0/4279509717 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31e0011630 con 0x7f31ec06f640 2022-06-27T12:09:16.683 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.597+0000 7f31eb7fe700 1 -- 172.21.15.195:0/4279509717 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31e0011790 con 0x7f31ec06f640 2022-06-27T12:09:16.683 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.597+0000 7f31eb7fe700 1 -- 172.21.15.195:0/4279509717 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31e0011b30 con 0x7f31ec06f640 2022-06-27T12:09:16.683 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.597+0000 7f31f2f4b700 1 -- 172.21.15.195:0/4279509717 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f31ec06f640 msgr2=0x7f31ec156ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:16.683 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.597+0000 7f31f2f4b700 1 --2- 172.21.15.195:0/4279509717 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f31ec06f640 0x7f31ec156ed0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f31e00097a0 tx=0x7f31e0009060 comp rx=0 tx=0).stop 2022-06-27T12:09:16.684 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.597+0000 7f31f2f4b700 1 -- 172.21.15.195:0/4279509717 shutdown_connections 2022-06-27T12:09:16.684 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.597+0000 7f31f2f4b700 1 --2- 172.21.15.195:0/4279509717 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f31ec06f640 0x7f31ec156ed0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:16.684 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.597+0000 7f31f2f4b700 1 --2- 172.21.15.195:0/4279509717 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f31ec0c4160 0x7f31ec0c1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:16.684 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.597+0000 7f31f2f4b700 1 -- 172.21.15.195:0/4279509717 >> 172.21.15.195:0/4279509717 conn(0x7f31ec0c6350 msgr2=0x7f31ec0c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:09:16.685 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.597+0000 7f31f2f4b700 1 -- 172.21.15.195:0/4279509717 shutdown_connections 2022-06-27T12:09:16.685 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.597+0000 7f31f2f4b700 1 -- 172.21.15.195:0/4279509717 wait complete. 2022-06-27T12:09:16.685 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.598+0000 7f31f2f4b700 1 Processor -- start 2022-06-27T12:09:16.685 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.599+0000 7f31f2f4b700 1 -- start start 2022-06-27T12:09:16.685 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.599+0000 7f31f2f4b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f31ec06f640 0x7f31ec1664f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:16.686 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.599+0000 7f31f2f4b700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f31ec0c39e0 0x7f31ec0c86a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:16.686 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.599+0000 7f31f2f4b700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f31ec0c4160 0x7f31ec0be300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:16.729 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.599+0000 7f31f2f4b700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31ec0bdb90 con 0x7f31ec06f640 2022-06-27T12:09:16.729 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.599+0000 7f31f2f4b700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31ec14f520 con 0x7f31ec0c4160 2022-06-27T12:09:16.730 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.599+0000 7f31f2f4b700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31ec14f800 con 0x7f31ec0c39e0 2022-06-27T12:09:16.730 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.599+0000 7f31f0ce7700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f31ec06f640 0x7f31ec1664f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:16.730 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.599+0000 7f31f0ce7700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f31ec06f640 0x7f31ec1664f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:49434/0 (socket says 172.21.15.195:49434) 2022-06-27T12:09:16.730 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.600+0000 7f31f0ce7700 1 -- 172.21.15.195:0/2612372275 learned_addr learned my addr 172.21.15.195:0/2612372275 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:09:16.731 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.600+0000 7f31f14e8700 1 --2- 172.21.15.195:0/2612372275 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f31ec0c4160 0x7f31ec0be300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:16.731 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.600+0000 7f31ebfff700 1 --2- 172.21.15.195:0/2612372275 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f31ec0c39e0 0x7f31ec0c86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:16.731 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.600+0000 7f31f0ce7700 1 -- 172.21.15.195:0/2612372275 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f31ec0c39e0 msgr2=0x7f31ec0c86a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:16.731 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.600+0000 7f31f0ce7700 1 --2- 172.21.15.195:0/2612372275 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f31ec0c39e0 0x7f31ec0c86a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:16.732 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.600+0000 7f31f0ce7700 1 -- 172.21.15.195:0/2612372275 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f31ec0c4160 msgr2=0x7f31ec0be300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:16.732 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.600+0000 7f31f0ce7700 1 --2- 172.21.15.195:0/2612372275 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f31ec0c4160 0x7f31ec0be300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:16.732 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.600+0000 7f31f0ce7700 1 -- 172.21.15.195:0/2612372275 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31e00092e0 con 0x7f31ec06f640 2022-06-27T12:09:16.732 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.600+0000 7f31f0ce7700 1 --2- 172.21.15.195:0/2612372275 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f31ec06f640 0x7f31ec1664f0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f31e0009e40 tx=0x7f31e0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:16.733 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.600+0000 7f31e8ff9700 1 -- 172.21.15.195:0/2612372275 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31e0000bc0 con 0x7f31ec06f640 2022-06-27T12:09:16.733 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.600+0000 7f31e8ff9700 1 -- 172.21.15.195:0/2612372275 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31e0011900 con 0x7f31ec06f640 2022-06-27T12:09:16.733 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.600+0000 7f31f2f4b700 1 -- 172.21.15.195:0/2612372275 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31ec14fae0 con 0x7f31ec06f640 2022-06-27T12:09:16.733 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.600+0000 7f31e8ff9700 1 -- 172.21.15.195:0/2612372275 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31e0000d20 con 0x7f31ec06f640 2022-06-27T12:09:16.734 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.600+0000 7f31f2f4b700 1 -- 172.21.15.195:0/2612372275 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31ec0bdde0 con 0x7f31ec06f640 2022-06-27T12:09:16.734 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.601+0000 7f31f2f4b700 1 -- 172.21.15.195:0/2612372275 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31ec0be000 con 0x7f31ec06f640 2022-06-27T12:09:16.734 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.602+0000 7f31e8ff9700 1 -- 172.21.15.195:0/2612372275 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f31e002c030 con 0x7f31ec06f640 2022-06-27T12:09:16.734 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.602+0000 7f31e8ff9700 1 --2- 172.21.15.195:0/2612372275 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f31d4063d00 0x7f31d40661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:16.744 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.602+0000 7f31e8ff9700 1 -- 172.21.15.195:0/2612372275 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7148+0+0 (secure 0 0 0) 0x7f31e0082200 con 0x7f31ec06f640 2022-06-27T12:09:16.744 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.602+0000 7f31ebfff700 1 --2- 172.21.15.195:0/2612372275 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f31d4063d00 0x7f31d40661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:16.744 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.603+0000 7f31ebfff700 1 --2- 172.21.15.195:0/2612372275 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f31d4063d00 0x7f31d40661b0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f31ec06d0e0 tx=0x7f31d8009060 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:16.745 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.607+0000 7f31e8ff9700 1 -- 172.21.15.195:0/2612372275 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f31e0050950 con 0x7f31ec06f640 2022-06-27T12:09:16.862 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.830+0000 7f31f2f4b700 1 -- 172.21.15.195:0/2612372275 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f31ec0722c0 con 0x7f31ec06f640 2022-06-27T12:09:16.863 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.831+0000 7f31e8ff9700 1 -- 172.21.15.195:0/2612372275 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+6125 (secure 0 0 0) 0x7f31e0053f70 con 0x7f31ec06f640 2022-06-27T12:09:16.864 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:09:16.865 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":78,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":6708,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/702244802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":702244802},{"type":"v1","addr":"172.21.15.202:6833","nonce":702244802}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:08:23.133338+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6271},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6271":{"gid":6271,"name":"d","rank":0,"incarnation":73,"state":"up:active","state_seq":20,"addr":"172.21.15.202:6835/1603307620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":1603307620},{"type":"v1","addr":"172.21.15.202:6835","nonce":1603307620}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:09:16.869 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.833+0000 7f31f2f4b700 1 -- 172.21.15.195:0/2612372275 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f31d4063d00 msgr2=0x7f31d40661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:16.869 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.833+0000 7f31f2f4b700 1 --2- 172.21.15.195:0/2612372275 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f31d4063d00 0x7f31d40661b0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f31ec06d0e0 tx=0x7f31d8009060 comp rx=0 tx=0).stop 2022-06-27T12:09:16.869 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.833+0000 7f31f2f4b700 1 -- 172.21.15.195:0/2612372275 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f31ec06f640 msgr2=0x7f31ec1664f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:16.871 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.833+0000 7f31f2f4b700 1 --2- 172.21.15.195:0/2612372275 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f31ec06f640 0x7f31ec1664f0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f31e0009e40 tx=0x7f31e0017000 comp rx=0 tx=0).stop 2022-06-27T12:09:16.871 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.833+0000 7f31f2f4b700 1 -- 172.21.15.195:0/2612372275 shutdown_connections 2022-06-27T12:09:16.871 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.833+0000 7f31f2f4b700 1 --2- 172.21.15.195:0/2612372275 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f31d4063d00 0x7f31d40661b0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:16.872 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.833+0000 7f31f2f4b700 1 --2- 172.21.15.195:0/2612372275 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f31ec06f640 0x7f31ec1664f0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:16.872 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.833+0000 7f31f2f4b700 1 --2- 172.21.15.195:0/2612372275 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f31ec0c4160 0x7f31ec0be300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:16.873 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.833+0000 7f31f2f4b700 1 --2- 172.21.15.195:0/2612372275 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f31ec0c39e0 0x7f31ec0c86a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:16.873 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.833+0000 7f31f2f4b700 1 -- 172.21.15.195:0/2612372275 >> 172.21.15.195:0/2612372275 conn(0x7f31ec0c6350 msgr2=0x7f31ec150550 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:09:16.873 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.833+0000 7f31f2f4b700 1 -- 172.21.15.195:0/2612372275 shutdown_connections 2022-06-27T12:09:16.873 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:16.833+0000 7f31f2f4b700 1 -- 172.21.15.195:0/2612372275 wait complete. 2022-06-27T12:09:16.874 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 78 2022-06-27T12:09:16.874 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-06-27T12:09:18.894 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:09:19.058 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.028+0000 7f0c9d188700 1 Processor -- start 2022-06-27T12:09:19.059 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.028+0000 7f0c9d188700 1 -- start start 2022-06-27T12:09:19.061 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.029+0000 7f0c9d188700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f0c9806e850 0x7f0c981570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:19.061 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.029+0000 7f0c9d188700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f0c980c34a0 0x7f0c980c2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:19.062 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.029+0000 7f0c9d188700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c9806fba0 con 0x7f0c9806fef0 2022-06-27T12:09:19.063 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.029+0000 7f0c9d188700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c981575e0 con 0x7f0c9806e850 2022-06-27T12:09:19.063 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.029+0000 7f0c9d188700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c9806f5e0 con 0x7f0c980c34a0 2022-06-27T12:09:19.063 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.029+0000 7f0c96d9d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f0c9806e850 0x7f0c981570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:19.064 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.029+0000 7f0c96d9d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f0c9806e850 0x7f0c981570a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:49456/0 (socket says 172.21.15.195:49456) 2022-06-27T12:09:19.064 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.029+0000 7f0c96d9d700 1 -- 172.21.15.195:0/2905271186 learned_addr learned my addr 172.21.15.195:0/2905271186 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:09:19.065 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.029+0000 7f0c9759e700 1 --2- 172.21.15.195:0/2905271186 >> v2:172.21.15.202:3300/0 conn(0x7f0c980c34a0 0x7f0c980c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:19.065 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.029+0000 7f0c96d9d700 1 -- 172.21.15.195:0/2905271186 >> v2:172.21.15.202:3300/0 conn(0x7f0c980c34a0 msgr2=0x7f0c980c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:09:19.067 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.029+0000 7f0c96d9d700 1 --2- 172.21.15.195:0/2905271186 >> v2:172.21.15.202:3300/0 conn(0x7f0c980c34a0 0x7f0c980c2920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:19.067 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.029+0000 7f0c96d9d700 1 -- 172.21.15.195:0/2905271186 >> v1:172.21.15.202:6789/0 conn(0x7f0c9806fef0 legacy=0x7f0c98069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:19.068 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.029+0000 7f0c96d9d700 1 -- 172.21.15.195:0/2905271186 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c9806f8c0 con 0x7f0c9806e850 2022-06-27T12:09:19.068 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.029+0000 7f0c96d9d700 1 --2- 172.21.15.195:0/2905271186 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0c9806e850 0x7f0c981570a0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f0c980c1420 tx=0x7f0c88009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9b8b39a934eb571 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:19.070 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.029+0000 7f0c95d9b700 1 -- 172.21.15.195:0/2905271186 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c88011630 con 0x7f0c9806e850 2022-06-27T12:09:19.070 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.029+0000 7f0c95d9b700 1 -- 172.21.15.195:0/2905271186 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c88011790 con 0x7f0c9806e850 2022-06-27T12:09:19.071 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.029+0000 7f0c95d9b700 1 -- 172.21.15.195:0/2905271186 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c88011b30 con 0x7f0c9806e850 2022-06-27T12:09:19.071 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.030+0000 7f0c9d188700 1 -- 172.21.15.195:0/2905271186 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0c9806e850 msgr2=0x7f0c981570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:19.072 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.030+0000 7f0c9d188700 1 --2- 172.21.15.195:0/2905271186 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0c9806e850 0x7f0c981570a0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f0c980c1420 tx=0x7f0c88009120 comp rx=0 tx=0).stop 2022-06-27T12:09:19.072 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.030+0000 7f0c9d188700 1 -- 172.21.15.195:0/2905271186 shutdown_connections 2022-06-27T12:09:19.072 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.030+0000 7f0c9d188700 1 --2- 172.21.15.195:0/2905271186 >> v2:172.21.15.202:3300/0 conn(0x7f0c980c34a0 0x7f0c980c2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:19.073 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.030+0000 7f0c9d188700 1 --2- 172.21.15.195:0/2905271186 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0c9806e850 0x7f0c981570a0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:19.073 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.030+0000 7f0c9d188700 1 -- 172.21.15.195:0/2905271186 >> 172.21.15.195:0/2905271186 conn(0x7f0c980c6350 msgr2=0x7f0c980c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:09:19.074 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.030+0000 7f0c9d188700 1 -- 172.21.15.195:0/2905271186 shutdown_connections 2022-06-27T12:09:19.074 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.030+0000 7f0c9d188700 1 -- 172.21.15.195:0/2905271186 wait complete. 2022-06-27T12:09:19.075 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.031+0000 7f0c9d188700 1 Processor -- start 2022-06-27T12:09:19.075 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.031+0000 7f0c9d188700 1 -- start start 2022-06-27T12:09:19.075 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.032+0000 7f0c9d188700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0c9806e850 0x7f0c981537b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:19.076 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.032+0000 7f0c9d188700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0c980c34a0 0x7f0c98154ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:19.077 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.032+0000 7f0c9d188700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0c98141e90 0x7f0c981422e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:19.077 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.032+0000 7f0c9d188700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c98167880 con 0x7f0c980c34a0 2022-06-27T12:09:19.078 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.032+0000 7f0c9d188700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c98142820 con 0x7f0c98141e90 2022-06-27T12:09:19.078 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.032+0000 7f0c9d188700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c981542e0 con 0x7f0c9806e850 2022-06-27T12:09:19.079 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.032+0000 7f0c9659c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0c980c34a0 0x7f0c98154ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:19.080 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.032+0000 7f0c9659c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0c980c34a0 0x7f0c98154ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:49464/0 (socket says 172.21.15.195:49464) 2022-06-27T12:09:19.081 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.032+0000 7f0c9659c700 1 -- 172.21.15.195:0/488330215 learned_addr learned my addr 172.21.15.195:0/488330215 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:09:19.081 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.032+0000 7f0c9759e700 1 --2- 172.21.15.195:0/488330215 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0c98141e90 0x7f0c981422e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:19.082 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.032+0000 7f0c96d9d700 1 --2- 172.21.15.195:0/488330215 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0c9806e850 0x7f0c981537b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:19.083 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.033+0000 7f0c9659c700 1 -- 172.21.15.195:0/488330215 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0c9806e850 msgr2=0x7f0c981537b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:19.083 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.033+0000 7f0c9659c700 1 --2- 172.21.15.195:0/488330215 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0c9806e850 0x7f0c981537b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:19.084 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.033+0000 7f0c9659c700 1 -- 172.21.15.195:0/488330215 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0c98141e90 msgr2=0x7f0c981422e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:19.084 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.033+0000 7f0c9659c700 1 --2- 172.21.15.195:0/488330215 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0c98141e90 0x7f0c981422e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:19.085 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.033+0000 7f0c9659c700 1 -- 172.21.15.195:0/488330215 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c880093a0 con 0x7f0c980c34a0 2022-06-27T12:09:19.085 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.033+0000 7f0c9659c700 1 --2- 172.21.15.195:0/488330215 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0c980c34a0 0x7f0c98154ad0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c00c410 tx=0x7f0c8c007e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:19.085 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.033+0000 7f0c7effd700 1 -- 172.21.15.195:0/488330215 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c8c0050d0 con 0x7f0c980c34a0 2022-06-27T12:09:19.086 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.033+0000 7f0c7effd700 1 -- 172.21.15.195:0/488330215 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c8c005250 con 0x7f0c980c34a0 2022-06-27T12:09:19.087 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.033+0000 7f0c7effd700 1 -- 172.21.15.195:0/488330215 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c8c003750 con 0x7f0c980c34a0 2022-06-27T12:09:19.087 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.033+0000 7f0c9d188700 1 -- 172.21.15.195:0/488330215 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c98154010 con 0x7f0c980c34a0 2022-06-27T12:09:19.088 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.033+0000 7f0c9d188700 1 -- 172.21.15.195:0/488330215 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c98166f50 con 0x7f0c980c34a0 2022-06-27T12:09:19.088 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.034+0000 7f0c9d188700 1 -- 172.21.15.195:0/488330215 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c980722c0 con 0x7f0c980c34a0 2022-06-27T12:09:19.088 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.034+0000 7f0c7effd700 1 -- 172.21.15.195:0/488330215 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f0c8c019030 con 0x7f0c980c34a0 2022-06-27T12:09:19.089 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.035+0000 7f0c7effd700 1 --2- 172.21.15.195:0/488330215 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0c78063d00 0x7f0c780661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:19.089 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.035+0000 7f0c7effd700 1 -- 172.21.15.195:0/488330215 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7148+0+0 (secure 0 0 0) 0x7f0c8c07fcb0 con 0x7f0c980c34a0 2022-06-27T12:09:19.090 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.035+0000 7f0c96d9d700 1 --2- 172.21.15.195:0/488330215 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0c78063d00 0x7f0c780661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:19.091 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.035+0000 7f0c96d9d700 1 --2- 172.21.15.195:0/488330215 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0c78063d00 0x7f0c780661b0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f0c88009160 tx=0x7f0c88017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:19.091 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.039+0000 7f0c7effd700 1 -- 172.21.15.195:0/488330215 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f0c8c04e460 con 0x7f0c980c34a0 2022-06-27T12:09:19.257 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.257+0000 7f0c9d188700 1 -- 172.21.15.195:0/488330215 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0c9806f240 con 0x7f0c980c34a0 2022-06-27T12:09:19.258 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.257+0000 7f0c7effd700 1 -- 172.21.15.195:0/488330215 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+6125 (secure 0 0 0) 0x7f0c8c017020 con 0x7f0c980c34a0 2022-06-27T12:09:19.259 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:09:19.260 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":78,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":6708,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/702244802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":702244802},{"type":"v1","addr":"172.21.15.202:6833","nonce":702244802}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:08:23.133338+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6271},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6271":{"gid":6271,"name":"d","rank":0,"incarnation":73,"state":"up:active","state_seq":20,"addr":"172.21.15.202:6835/1603307620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":1603307620},{"type":"v1","addr":"172.21.15.202:6835","nonce":1603307620}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:09:19.263 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.259+0000 7f0c9d188700 1 -- 172.21.15.195:0/488330215 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0c78063d00 msgr2=0x7f0c780661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:19.264 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.259+0000 7f0c9d188700 1 --2- 172.21.15.195:0/488330215 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0c78063d00 0x7f0c780661b0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f0c88009160 tx=0x7f0c88017000 comp rx=0 tx=0).stop 2022-06-27T12:09:19.264 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.259+0000 7f0c9d188700 1 -- 172.21.15.195:0/488330215 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0c980c34a0 msgr2=0x7f0c98154ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:19.265 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.259+0000 7f0c9d188700 1 --2- 172.21.15.195:0/488330215 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0c980c34a0 0x7f0c98154ad0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c00c410 tx=0x7f0c8c007e40 comp rx=0 tx=0).stop 2022-06-27T12:09:19.265 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.259+0000 7f0c9d188700 1 -- 172.21.15.195:0/488330215 shutdown_connections 2022-06-27T12:09:19.267 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.259+0000 7f0c9d188700 1 --2- 172.21.15.195:0/488330215 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0c78063d00 0x7f0c780661b0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:19.276 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.259+0000 7f0c9d188700 1 --2- 172.21.15.195:0/488330215 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0c98141e90 0x7f0c981422e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:19.276 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.259+0000 7f0c9d188700 1 --2- 172.21.15.195:0/488330215 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0c9806e850 0x7f0c981537b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:19.281 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.259+0000 7f0c9d188700 1 --2- 172.21.15.195:0/488330215 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0c980c34a0 0x7f0c98154ad0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:19.282 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.259+0000 7f0c9d188700 1 -- 172.21.15.195:0/488330215 >> 172.21.15.195:0/488330215 conn(0x7f0c980c6350 msgr2=0x7f0c980c26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:09:19.282 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.260+0000 7f0c9d188700 1 -- 172.21.15.195:0/488330215 shutdown_connections 2022-06-27T12:09:19.283 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:19.260+0000 7f0c9d188700 1 -- 172.21.15.195:0/488330215 wait complete. 2022-06-27T12:09:19.283 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 78 2022-06-27T12:09:19.291 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-06-27T12:09:21.293 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:09:21.437 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.428+0000 7f8591b2f700 1 Processor -- start 2022-06-27T12:09:21.437 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.428+0000 7f8591b2f700 1 -- start start 2022-06-27T12:09:21.437 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.429+0000 7f8591b2f700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f858c14e490 0x7f858c06cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:21.438 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.429+0000 7f8591b2f700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f858c0c3f50 0x7f858c157890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:21.438 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.429+0000 7f8591b2f700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f858c0c33b0 con 0x7f858c14e860 2022-06-27T12:09:21.439 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.429+0000 7f8591b2f700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f858c06d520 con 0x7f858c0c3f50 2022-06-27T12:09:21.439 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.429+0000 7f8591b2f700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f858c06f220 con 0x7f858c14e490 2022-06-27T12:09:21.439 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.429+0000 7f858bfff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f858c0c3f50 0x7f858c157890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:21.439 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.429+0000 7f858bfff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f858c0c3f50 0x7f858c157890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:49474/0 (socket says 172.21.15.195:49474) 2022-06-27T12:09:21.440 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.429+0000 7f858bfff700 1 -- 172.21.15.195:0/3670374041 learned_addr learned my addr 172.21.15.195:0/3670374041 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:09:21.440 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.429+0000 7f858b7fe700 1 --2- 172.21.15.195:0/3670374041 >> v2:172.21.15.202:3300/0 conn(0x7f858c14e490 0x7f858c06cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:21.440 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.429+0000 7f858bfff700 1 -- 172.21.15.195:0/3670374041 >> v2:172.21.15.202:3300/0 conn(0x7f858c14e490 msgr2=0x7f858c06cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:09:21.440 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.429+0000 7f858bfff700 1 --2- 172.21.15.195:0/3670374041 >> v2:172.21.15.202:3300/0 conn(0x7f858c14e490 0x7f858c06cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:21.441 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.429+0000 7f858bfff700 1 -- 172.21.15.195:0/3670374041 >> v1:172.21.15.202:6789/0 conn(0x7f858c14e860 legacy=0x7f858c0c3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:09:21.441 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.429+0000 7f858bfff700 1 -- 172.21.15.195:0/3670374041 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f858c1542c0 con 0x7f858c0c3f50 2022-06-27T12:09:21.441 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.429+0000 7f858bfff700 1 --2- 172.21.15.195:0/3670374041 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f858c0c3f50 0x7f858c157890 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f858c064800 tx=0x7f8584009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7826dcc39e359e80 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:21.442 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.430+0000 7f858a7fc700 1 -- 172.21.15.195:0/3670374041 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f858400e070 con 0x7f858c0c3f50 2022-06-27T12:09:21.442 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.430+0000 7f858a7fc700 1 -- 172.21.15.195:0/3670374041 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f858400ba10 con 0x7f858c0c3f50 2022-06-27T12:09:21.442 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.430+0000 7f858a7fc700 1 -- 172.21.15.195:0/3670374041 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85840092e0 con 0x7f858c0c3f50 2022-06-27T12:09:21.442 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.430+0000 7f8591b2f700 1 -- 172.21.15.195:0/3670374041 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f858c0c3f50 msgr2=0x7f858c157890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:21.443 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.430+0000 7f8591b2f700 1 --2- 172.21.15.195:0/3670374041 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f858c0c3f50 0x7f858c157890 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f858c064800 tx=0x7f8584009840 comp rx=0 tx=0).stop 2022-06-27T12:09:21.443 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.430+0000 7f8591b2f700 1 -- 172.21.15.195:0/3670374041 shutdown_connections 2022-06-27T12:09:21.443 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.430+0000 7f8591b2f700 1 --2- 172.21.15.195:0/3670374041 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f858c0c3f50 0x7f858c157890 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:21.444 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.430+0000 7f8591b2f700 1 --2- 172.21.15.195:0/3670374041 >> v2:172.21.15.202:3300/0 conn(0x7f858c14e490 0x7f858c06cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:21.444 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.430+0000 7f8591b2f700 1 -- 172.21.15.195:0/3670374041 >> 172.21.15.195:0/3670374041 conn(0x7f858c063af0 msgr2=0x7f858c063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:09:21.444 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.430+0000 7f8591b2f700 1 -- 172.21.15.195:0/3670374041 shutdown_connections 2022-06-27T12:09:21.444 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.430+0000 7f8591b2f700 1 -- 172.21.15.195:0/3670374041 wait complete. 2022-06-27T12:09:21.445 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.431+0000 7f8591b2f700 1 Processor -- start 2022-06-27T12:09:21.445 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.432+0000 7f8591b2f700 1 -- start start 2022-06-27T12:09:21.445 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.432+0000 7f8591b2f700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f858c0c3f50 0x7f858c169de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:21.445 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.432+0000 7f8591b2f700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f858c14e490 0x7f858c14b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:21.446 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.432+0000 7f8591b2f700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f858c14e860 0x7f858c0c78f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:21.446 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.432+0000 7f8591b2f700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f858c166fb0 con 0x7f858c14e860 2022-06-27T12:09:21.446 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.432+0000 7f8591b2f700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f858c159250 con 0x7f858c14e490 2022-06-27T12:09:21.446 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.432+0000 7f8591b2f700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f858c158c90 con 0x7f858c0c3f50 2022-06-27T12:09:21.447 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.432+0000 7f858bfff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f858c14e860 0x7f858c0c78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:21.447 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.432+0000 7f858bfff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f858c14e860 0x7f858c0c78f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:49480/0 (socket says 172.21.15.195:49480) 2022-06-27T12:09:21.447 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.432+0000 7f858bfff700 1 -- 172.21.15.195:0/1784069341 learned_addr learned my addr 172.21.15.195:0/1784069341 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:09:21.447 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.432+0000 7f858b7fe700 1 --2- 172.21.15.195:0/1784069341 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f858c0c3f50 0x7f858c169de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:21.447 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.432+0000 7f858affd700 1 --2- 172.21.15.195:0/1784069341 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f858c14e490 0x7f858c14b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:21.448 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.433+0000 7f858bfff700 1 -- 172.21.15.195:0/1784069341 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f858c0c3f50 msgr2=0x7f858c169de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:21.448 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.433+0000 7f858bfff700 1 --2- 172.21.15.195:0/1784069341 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f858c0c3f50 0x7f858c169de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:21.448 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.433+0000 7f858bfff700 1 -- 172.21.15.195:0/1784069341 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f858c14e490 msgr2=0x7f858c14b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:21.449 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.433+0000 7f858bfff700 1 --2- 172.21.15.195:0/1784069341 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f858c14e490 0x7f858c14b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:21.449 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.433+0000 7f858bfff700 1 -- 172.21.15.195:0/1784069341 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8584009ae0 con 0x7f858c14e860 2022-06-27T12:09:21.449 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.433+0000 7f858bfff700 1 --2- 172.21.15.195:0/1784069341 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f858c14e860 0x7f858c0c78f0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f8584004000 tx=0x7f8584017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:21.449 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.433+0000 7f85837fe700 1 -- 172.21.15.195:0/1784069341 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85840092e0 con 0x7f858c14e860 2022-06-27T12:09:21.449 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.433+0000 7f85837fe700 1 -- 172.21.15.195:0/1784069341 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8584004240 con 0x7f858c14e860 2022-06-27T12:09:21.450 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.433+0000 7f8591b2f700 1 -- 172.21.15.195:0/1784069341 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f858c158f70 con 0x7f858c14e860 2022-06-27T12:09:21.450 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.433+0000 7f85837fe700 1 -- 172.21.15.195:0/1784069341 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85840045e0 con 0x7f858c14e860 2022-06-27T12:09:21.451 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.433+0000 7f8591b2f700 1 -- 172.21.15.195:0/1784069341 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f858c166c40 con 0x7f858c14e860 2022-06-27T12:09:21.451 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.434+0000 7f8591b2f700 1 -- 172.21.15.195:0/1784069341 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f858c167160 con 0x7f858c14e860 2022-06-27T12:09:21.451 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.434+0000 7f85837fe700 1 -- 172.21.15.195:0/1784069341 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f858402c070 con 0x7f858c14e860 2022-06-27T12:09:21.451 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.435+0000 7f85837fe700 1 --2- 172.21.15.195:0/1784069341 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8570063d00 0x7f85700661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:21.451 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.435+0000 7f85837fe700 1 -- 172.21.15.195:0/1784069341 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7148+0+0 (secure 0 0 0) 0x7f8584081fe0 con 0x7f858c14e860 2022-06-27T12:09:21.452 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.435+0000 7f858b7fe700 1 --2- 172.21.15.195:0/1784069341 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8570063d00 0x7f85700661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:21.452 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.435+0000 7f858b7fe700 1 --2- 172.21.15.195:0/1784069341 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8570063d00 0x7f85700661b0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f8574004560 tx=0x7f8574008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:21.453 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.439+0000 7f85837fe700 1 -- 172.21.15.195:0/1784069341 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f858c167160 con 0x7f858c14e860 2022-06-27T12:09:21.668 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.666+0000 7f8591b2f700 1 -- 172.21.15.195:0/1784069341 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f858c0722c0 con 0x7f858c14e860 2022-06-27T12:09:21.669 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.667+0000 7f85837fe700 1 -- 172.21.15.195:0/1784069341 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+6125 (secure 0 0 0) 0x7f858402a070 con 0x7f858c14e860 2022-06-27T12:09:21.674 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:09:21.675 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":78,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":6708,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/702244802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":702244802},{"type":"v1","addr":"172.21.15.202:6833","nonce":702244802}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:08:23.133338+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6271},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6271":{"gid":6271,"name":"d","rank":0,"incarnation":73,"state":"up:active","state_seq":20,"addr":"172.21.15.202:6835/1603307620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":1603307620},{"type":"v1","addr":"172.21.15.202:6835","nonce":1603307620}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:09:21.677 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.669+0000 7f8591b2f700 1 -- 172.21.15.195:0/1784069341 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8570063d00 msgr2=0x7f85700661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:21.677 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.669+0000 7f8591b2f700 1 --2- 172.21.15.195:0/1784069341 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8570063d00 0x7f85700661b0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f8574004560 tx=0x7f8574008040 comp rx=0 tx=0).stop 2022-06-27T12:09:21.678 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.669+0000 7f8591b2f700 1 -- 172.21.15.195:0/1784069341 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f858c14e860 msgr2=0x7f858c0c78f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:21.678 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.669+0000 7f8591b2f700 1 --2- 172.21.15.195:0/1784069341 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f858c14e860 0x7f858c0c78f0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f8584004000 tx=0x7f8584017000 comp rx=0 tx=0).stop 2022-06-27T12:09:21.679 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.669+0000 7f8591b2f700 1 -- 172.21.15.195:0/1784069341 shutdown_connections 2022-06-27T12:09:21.679 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.669+0000 7f8591b2f700 1 --2- 172.21.15.195:0/1784069341 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8570063d00 0x7f85700661b0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:21.679 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.669+0000 7f8591b2f700 1 --2- 172.21.15.195:0/1784069341 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f858c14e860 0x7f858c0c78f0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:21.680 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.669+0000 7f8591b2f700 1 --2- 172.21.15.195:0/1784069341 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f858c0c3f50 0x7f858c169de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:21.680 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.669+0000 7f8591b2f700 1 --2- 172.21.15.195:0/1784069341 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f858c14e490 0x7f858c14b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:21.681 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.669+0000 7f8591b2f700 1 -- 172.21.15.195:0/1784069341 >> 172.21.15.195:0/1784069341 conn(0x7f858c063af0 msgr2=0x7f858c063ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:09:21.681 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.669+0000 7f8591b2f700 1 -- 172.21.15.195:0/1784069341 shutdown_connections 2022-06-27T12:09:21.682 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:21.669+0000 7f8591b2f700 1 -- 172.21.15.195:0/1784069341 wait complete. 2022-06-27T12:09:21.682 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 78 2022-06-27T12:09:21.684 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-06-27T12:09:23.723 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:09:23.885 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:23.867+0000 7fbcf7fca700 1 Processor -- start 2022-06-27T12:09:23.886 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:23.868+0000 7fbcf7fca700 1 -- start start 2022-06-27T12:09:23.886 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:23.868+0000 7fbcf7fca700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fbcf0150500 0x7fbcf0063af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:23.887 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:23.868+0000 7fbcf7fca700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbcf01508d0 0x7fbcf00bbf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:23.887 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:23.868+0000 7fbcf7fca700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcf0064030 con 0x7fbcf006a450 2022-06-27T12:09:23.887 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:23.868+0000 7fbcf7fca700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcf00c4160 con 0x7fbcf0150500 2022-06-27T12:09:23.888 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:23.868+0000 7fbcf7fca700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf00c4440 con 0x7fbcf01508d0 2022-06-27T12:09:23.888 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:23.869+0000 7fbcf5d66700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fbcf0150500 0x7fbcf0063af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:23.889 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:23.869+0000 7fbcf5d66700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fbcf0150500 0x7fbcf0063af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:49484/0 (socket says 172.21.15.195:49484) 2022-06-27T12:09:23.889 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:23.869+0000 7fbcf5d66700 1 -- 172.21.15.195:0/32573154 learned_addr learned my addr 172.21.15.195:0/32573154 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:09:23.926 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:23.869+0000 7fbcf6567700 1 --2- 172.21.15.195:0/32573154 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbcf01508d0 0x7fbcf00bbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:23.926 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:23.869+0000 7fbcf5d66700 1 -- 172.21.15.195:0/32573154 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbcf01508d0 msgr2=0x7fbcf00bbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:09:23.927 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:23.869+0000 7fbcf5d66700 1 --2- 172.21.15.195:0/32573154 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbcf01508d0 0x7fbcf00bbf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:23.927 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:23.869+0000 7fbcf5d66700 1 -- 172.21.15.195:0/32573154 >> v1:172.21.15.202:6789/0 conn(0x7fbcf006a450 legacy=0x7fbcf00c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:23.928 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:23.869+0000 7fbcf5d66700 1 -- 172.21.15.195:0/32573154 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcf015a800 con 0x7fbcf0150500 2022-06-27T12:09:23.928 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:23.869+0000 7fbcf6567700 1 --2- 172.21.15.195:0/32573154 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbcf01508d0 0x7fbcf00bbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:09:23.929 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:23.869+0000 7fbcf5d66700 1 --2- 172.21.15.195:0/32573154 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbcf0150500 0x7fbcf0063af0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fbce0006fd0 tx=0x7fbce00090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e87f78696f7a8b9e server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:23.929 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:23.869+0000 7fbcf4d64700 1 -- 172.21.15.195:0/32573154 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce0005ad0 con 0x7fbcf0150500 2022-06-27T12:09:23.930 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:23.869+0000 7fbcf4d64700 1 -- 172.21.15.195:0/32573154 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbce0005c30 con 0x7fbcf0150500 2022-06-27T12:09:23.930 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:23.869+0000 7fbcf4d64700 1 -- 172.21.15.195:0/32573154 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce0004030 con 0x7fbcf0150500 2022-06-27T12:09:23.931 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:23.870+0000 7fbcf7fca700 1 -- 172.21.15.195:0/32573154 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbcf0150500 msgr2=0x7fbcf0063af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:23.931 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:23.870+0000 7fbcf7fca700 1 --2- 172.21.15.195:0/32573154 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbcf0150500 0x7fbcf0063af0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fbce0006fd0 tx=0x7fbce00090f0 comp rx=0 tx=0).stop 2022-06-27T12:09:23.932 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:23.870+0000 7fbcf7fca700 1 -- 172.21.15.195:0/32573154 shutdown_connections 2022-06-27T12:09:23.932 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:23.870+0000 7fbcf7fca700 1 --2- 172.21.15.195:0/32573154 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbcf01508d0 0x7fbcf00bbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:23.951 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:23.870+0000 7fbcf7fca700 1 --2- 172.21.15.195:0/32573154 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbcf0150500 0x7fbcf0063af0 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:23.952 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:23.870+0000 7fbcf7fca700 1 -- 172.21.15.195:0/32573154 >> 172.21.15.195:0/32573154 conn(0x7fbcf00c9b00 msgr2=0x7fbcf00c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:09:23.952 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:23.870+0000 7fbcf7fca700 1 -- 172.21.15.195:0/32573154 shutdown_connections 2022-06-27T12:09:23.953 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:23.870+0000 7fbcf7fca700 1 -- 172.21.15.195:0/32573154 wait complete. 2022-06-27T12:09:23.953 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:23.871+0000 7fbcf7fca700 1 Processor -- start 2022-06-27T12:09:23.953 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:23.871+0000 7fbcf7fca700 1 -- start start 2022-06-27T12:09:23.973 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:23.872+0000 7fbcf7fca700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbcf006a450 0x7fbcf0166ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:23.974 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:23.872+0000 7fbcf7fca700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbcf0150500 0x7fbcf0167400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:23.974 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:23.872+0000 7fbcf7fca700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbcf01508d0 0x7fbcf014b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:23.974 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:23.872+0000 7fbcf7fca700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf0166a30 con 0x7fbcf0150500 2022-06-27T12:09:23.975 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:23.872+0000 7fbcf7fca700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf00bf060 con 0x7fbcf01508d0 2022-06-27T12:09:23.975 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:23.872+0000 7fbcf7fca700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf00bf340 con 0x7fbcf006a450 2022-06-27T12:09:23.976 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:23.872+0000 7fbcf5565700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbcf0150500 0x7fbcf0167400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:23.976 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:23.872+0000 7fbcf5565700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbcf0150500 0x7fbcf0167400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:49492/0 (socket says 172.21.15.195:49492) 2022-06-27T12:09:23.980 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:23.872+0000 7fbcf5d66700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbcf006a450 0x7fbcf0166ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:23.981 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:23.872+0000 7fbcf5565700 1 -- 172.21.15.195:0/2028521492 learned_addr learned my addr 172.21.15.195:0/2028521492 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:09:23.981 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:23.872+0000 7fbcf6567700 1 --2- 172.21.15.195:0/2028521492 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbcf01508d0 0x7fbcf014b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:23.981 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:23.872+0000 7fbcf5565700 1 -- 172.21.15.195:0/2028521492 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbcf006a450 msgr2=0x7fbcf0166ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:23.982 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:23.872+0000 7fbcf5565700 1 --2- 172.21.15.195:0/2028521492 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbcf006a450 0x7fbcf0166ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:23.982 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:23.872+0000 7fbcf5565700 1 -- 172.21.15.195:0/2028521492 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbcf01508d0 msgr2=0x7fbcf014b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:23.983 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:23.872+0000 7fbcf5565700 1 --2- 172.21.15.195:0/2028521492 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbcf01508d0 0x7fbcf014b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:23.983 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:23.872+0000 7fbcf5565700 1 -- 172.21.15.195:0/2028521492 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbce0009350 con 0x7fbcf0150500 2022-06-27T12:09:23.984 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:23.872+0000 7fbcf5565700 1 --2- 172.21.15.195:0/2028521492 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbcf0150500 0x7fbcf0167400 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fbcec00c490 tx=0x7fbcec007e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:23.984 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:23.872+0000 7fbcddffb700 1 -- 172.21.15.195:0/2028521492 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcec00ce40 con 0x7fbcf0150500 2022-06-27T12:09:23.985 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:23.872+0000 7fbcddffb700 1 -- 172.21.15.195:0/2028521492 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcec0050d0 con 0x7fbcf0150500 2022-06-27T12:09:23.985 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:23.872+0000 7fbcf7fca700 1 -- 172.21.15.195:0/2028521492 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcf00bf620 con 0x7fbcf0150500 2022-06-27T12:09:23.985 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:23.872+0000 7fbcddffb700 1 -- 172.21.15.195:0/2028521492 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcec011590 con 0x7fbcf0150500 2022-06-27T12:09:23.993 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:23.873+0000 7fbcf7fca700 1 -- 172.21.15.195:0/2028521492 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcf00be720 con 0x7fbcf0150500 2022-06-27T12:09:23.994 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:23.874+0000 7fbcf7fca700 1 -- 172.21.15.195:0/2028521492 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcf0166be0 con 0x7fbcf0150500 2022-06-27T12:09:23.998 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:23.874+0000 7fbcddffb700 1 -- 172.21.15.195:0/2028521492 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fbcec019030 con 0x7fbcf0150500 2022-06-27T12:09:23.999 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:23.874+0000 7fbcddffb700 1 --2- 172.21.15.195:0/2028521492 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbcd0063d30 0x7fbcd00661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:23.999 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:23.874+0000 7fbcddffb700 1 -- 172.21.15.195:0/2028521492 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7148+0+0 (secure 0 0 0) 0x7fbcec07fad0 con 0x7fbcf0150500 2022-06-27T12:09:24.000 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:23.874+0000 7fbcf5d66700 1 --2- 172.21.15.195:0/2028521492 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbcd0063d30 0x7fbcd00661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:24.000 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:23.875+0000 7fbcf5d66700 1 --2- 172.21.15.195:0/2028521492 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbcd0063d30 0x7fbcd00661e0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fbce0009130 tx=0x7fbce0015000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:24.000 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:23.878+0000 7fbcddffb700 1 -- 172.21.15.195:0/2028521492 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fbcec049c90 con 0x7fbcf0150500 2022-06-27T12:09:24.107 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:24.106+0000 7fbcf7fca700 1 -- 172.21.15.195:0/2028521492 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbcf0064030 con 0x7fbcf0150500 2022-06-27T12:09:24.108 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:24.107+0000 7fbcddffb700 1 -- 172.21.15.195:0/2028521492 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+6125 (secure 0 0 0) 0x7fbcf0064030 con 0x7fbcf0150500 2022-06-27T12:09:24.109 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:09:24.109 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":78,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":6708,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/702244802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":702244802},{"type":"v1","addr":"172.21.15.202:6833","nonce":702244802}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:08:23.133338+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6271},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6271":{"gid":6271,"name":"d","rank":0,"incarnation":73,"state":"up:active","state_seq":20,"addr":"172.21.15.202:6835/1603307620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":1603307620},{"type":"v1","addr":"172.21.15.202:6835","nonce":1603307620}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:09:24.112 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:24.109+0000 7fbcf7fca700 1 -- 172.21.15.195:0/2028521492 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbcd0063d30 msgr2=0x7fbcd00661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:24.112 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:24.109+0000 7fbcf7fca700 1 --2- 172.21.15.195:0/2028521492 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbcd0063d30 0x7fbcd00661e0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fbce0009130 tx=0x7fbce0015000 comp rx=0 tx=0).stop 2022-06-27T12:09:24.113 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:24.109+0000 7fbcf7fca700 1 -- 172.21.15.195:0/2028521492 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbcf0150500 msgr2=0x7fbcf0167400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:24.113 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:24.109+0000 7fbcf7fca700 1 --2- 172.21.15.195:0/2028521492 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbcf0150500 0x7fbcf0167400 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fbcec00c490 tx=0x7fbcec007e40 comp rx=0 tx=0).stop 2022-06-27T12:09:24.114 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:24.109+0000 7fbcf7fca700 1 -- 172.21.15.195:0/2028521492 shutdown_connections 2022-06-27T12:09:24.114 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:24.109+0000 7fbcf7fca700 1 --2- 172.21.15.195:0/2028521492 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbcd0063d30 0x7fbcd00661e0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:24.115 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:24.109+0000 7fbcf7fca700 1 --2- 172.21.15.195:0/2028521492 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbcf01508d0 0x7fbcf014b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:24.115 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:24.109+0000 7fbcf7fca700 1 --2- 172.21.15.195:0/2028521492 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbcf006a450 0x7fbcf0166ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:24.116 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:24.109+0000 7fbcf7fca700 1 --2- 172.21.15.195:0/2028521492 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbcf0150500 0x7fbcf0167400 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:24.116 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:24.109+0000 7fbcf7fca700 1 -- 172.21.15.195:0/2028521492 >> 172.21.15.195:0/2028521492 conn(0x7fbcf00c9b00 msgr2=0x7fbcf014fa40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:09:24.117 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:24.109+0000 7fbcf7fca700 1 -- 172.21.15.195:0/2028521492 shutdown_connections 2022-06-27T12:09:24.117 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:24.109+0000 7fbcf7fca700 1 -- 172.21.15.195:0/2028521492 wait complete. 2022-06-27T12:09:24.117 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 78 2022-06-27T12:09:24.194 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-06-27T12:09:26.201 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:09:26.357 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.355+0000 7ffaf6442700 1 Processor -- start 2022-06-27T12:09:26.358 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.355+0000 7ffaf6442700 1 -- start start 2022-06-27T12:09:26.358 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.355+0000 7ffaf6442700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7ffaf006a450 0x7ffaf0063af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:26.359 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.355+0000 7ffaf6442700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ffaf01508d0 0x7ffaf00bbf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:26.359 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.355+0000 7ffaf6442700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaf0064030 con 0x7ffaf0150500 2022-06-27T12:09:26.359 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.355+0000 7ffaf6442700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffaf00c4160 con 0x7ffaf006a450 2022-06-27T12:09:26.361 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.355+0000 7ffaf6442700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf00c4440 con 0x7ffaf01508d0 2022-06-27T12:09:26.362 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.356+0000 7ffaef7fe700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7ffaf006a450 0x7ffaf0063af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:26.362 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.356+0000 7ffaf49df700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ffaf01508d0 0x7ffaf00bbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:26.363 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.356+0000 7ffaf49df700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ffaf01508d0 0x7ffaf00bbf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3301/0 says I am v2:172.21.15.195:42254/0 (socket says 172.21.15.195:42254) 2022-06-27T12:09:26.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.356+0000 7ffaf49df700 1 -- 172.21.15.195:0/4093270143 learned_addr learned my addr 172.21.15.195:0/4093270143 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:09:26.374 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.356+0000 7ffaef7fe700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7ffaf006a450 0x7ffaf0063af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3300/0 says I am v2:172.21.15.195:47624/0 (socket says 172.21.15.195:47624) 2022-06-27T12:09:26.375 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.356+0000 7ffaef7fe700 1 -- 172.21.15.195:0/4093270143 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ffaf01508d0 msgr2=0x7ffaf00bbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:09:26.375 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.356+0000 7ffaef7fe700 1 --2- 172.21.15.195:0/4093270143 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ffaf01508d0 0x7ffaf00bbf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:26.376 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.356+0000 7ffaef7fe700 1 -- 172.21.15.195:0/4093270143 >> v1:172.21.15.202:6789/0 conn(0x7ffaf0150500 legacy=0x7ffaf00c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:26.376 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.356+0000 7ffaef7fe700 1 -- 172.21.15.195:0/4093270143 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaf015a800 con 0x7ffaf006a450 2022-06-27T12:09:26.377 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.356+0000 7ffaef7fe700 1 --2- 172.21.15.195:0/4093270143 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ffaf006a450 0x7ffaf0063af0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7ffad80097a0 tx=0x7ffad8009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1b64eaa40f3e1d28 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:26.386 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.357+0000 7ffaeeffd700 1 -- 172.21.15.195:0/4093270143 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffad8011630 con 0x7ffaf006a450 2022-06-27T12:09:26.386 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.357+0000 7ffaeeffd700 1 -- 172.21.15.195:0/4093270143 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffad8011790 con 0x7ffaf006a450 2022-06-27T12:09:26.387 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.357+0000 7ffaeeffd700 1 -- 172.21.15.195:0/4093270143 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffad8011b30 con 0x7ffaf006a450 2022-06-27T12:09:26.387 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.357+0000 7ffaf6442700 1 -- 172.21.15.195:0/4093270143 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ffaf006a450 msgr2=0x7ffaf0063af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:26.388 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.357+0000 7ffaf6442700 1 --2- 172.21.15.195:0/4093270143 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ffaf006a450 0x7ffaf0063af0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7ffad80097a0 tx=0x7ffad8009060 comp rx=0 tx=0).stop 2022-06-27T12:09:26.389 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.357+0000 7ffaf6442700 1 -- 172.21.15.195:0/4093270143 shutdown_connections 2022-06-27T12:09:26.389 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.357+0000 7ffaf6442700 1 --2- 172.21.15.195:0/4093270143 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ffaf01508d0 0x7ffaf00bbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:26.389 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.357+0000 7ffaf6442700 1 --2- 172.21.15.195:0/4093270143 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ffaf006a450 0x7ffaf0063af0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:26.397 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.357+0000 7ffaf6442700 1 -- 172.21.15.195:0/4093270143 >> 172.21.15.195:0/4093270143 conn(0x7ffaf00c9b00 msgr2=0x7ffaf00c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:09:26.398 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.357+0000 7ffaf6442700 1 -- 172.21.15.195:0/4093270143 shutdown_connections 2022-06-27T12:09:26.399 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.357+0000 7ffaf6442700 1 -- 172.21.15.195:0/4093270143 wait complete. 2022-06-27T12:09:26.399 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.358+0000 7ffaf6442700 1 Processor -- start 2022-06-27T12:09:26.400 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.359+0000 7ffaf6442700 1 -- start start 2022-06-27T12:09:26.400 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.359+0000 7ffaf6442700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ffaf006a450 0x7ffaf0155010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:26.400 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.359+0000 7ffaf6442700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffaf0150500 0x7ffaf0166af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:26.414 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.359+0000 7ffaf6442700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ffaf01508d0 0x7ffaf015ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:26.415 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.359+0000 7ffaf6442700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf016a570 con 0x7ffaf0150500 2022-06-27T12:09:26.415 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.359+0000 7ffaf6442700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf01660e0 con 0x7ffaf006a450 2022-06-27T12:09:26.416 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.359+0000 7ffaf6442700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf0166660 con 0x7ffaf01508d0 2022-06-27T12:09:26.416 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.359+0000 7ffaef7fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffaf0150500 0x7ffaf0166af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:26.416 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.359+0000 7ffaef7fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffaf0150500 0x7ffaf0166af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:49504/0 (socket says 172.21.15.195:49504) 2022-06-27T12:09:26.417 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.359+0000 7ffaef7fe700 1 -- 172.21.15.195:0/3600839404 learned_addr learned my addr 172.21.15.195:0/3600839404 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:09:26.417 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.360+0000 7ffaf49df700 1 --2- 172.21.15.195:0/3600839404 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ffaf01508d0 0x7ffaf015ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:26.418 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.360+0000 7ffaeffff700 1 --2- 172.21.15.195:0/3600839404 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ffaf006a450 0x7ffaf0155010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:26.418 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.360+0000 7ffaef7fe700 1 -- 172.21.15.195:0/3600839404 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ffaf01508d0 msgr2=0x7ffaf015ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:26.419 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.360+0000 7ffaef7fe700 1 --2- 172.21.15.195:0/3600839404 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ffaf01508d0 0x7ffaf015ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:26.420 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.360+0000 7ffaef7fe700 1 -- 172.21.15.195:0/3600839404 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ffaf006a450 msgr2=0x7ffaf0155010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:26.420 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.360+0000 7ffaef7fe700 1 --2- 172.21.15.195:0/3600839404 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ffaf006a450 0x7ffaf0155010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:26.421 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.360+0000 7ffaef7fe700 1 -- 172.21.15.195:0/3600839404 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffad80092e0 con 0x7ffaf0150500 2022-06-27T12:09:26.422 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.360+0000 7ffaef7fe700 1 --2- 172.21.15.195:0/3600839404 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffaf0150500 0x7ffaf0166af0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7ffad8009e40 tx=0x7ffad8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:26.422 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.360+0000 7ffae7fff700 1 -- 172.21.15.195:0/3600839404 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffad8000d10 con 0x7ffaf0150500 2022-06-27T12:09:26.423 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.360+0000 7ffae7fff700 1 -- 172.21.15.195:0/3600839404 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffad8011900 con 0x7ffaf0150500 2022-06-27T12:09:26.423 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.360+0000 7ffaf6442700 1 -- 172.21.15.195:0/3600839404 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaf0167030 con 0x7ffaf0150500 2022-06-27T12:09:26.424 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.360+0000 7ffae7fff700 1 -- 172.21.15.195:0/3600839404 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffad8025420 con 0x7ffaf0150500 2022-06-27T12:09:26.424 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.360+0000 7ffaf6442700 1 -- 172.21.15.195:0/3600839404 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaf00bc5a0 con 0x7ffaf0150500 2022-06-27T12:09:26.425 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.361+0000 7ffaf6442700 1 -- 172.21.15.195:0/3600839404 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaf0166810 con 0x7ffaf0150500 2022-06-27T12:09:26.425 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.361+0000 7ffae7fff700 1 -- 172.21.15.195:0/3600839404 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7ffad800f050 con 0x7ffaf0150500 2022-06-27T12:09:26.426 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.362+0000 7ffae7fff700 1 --2- 172.21.15.195:0/3600839404 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ffad4063d00 0x7ffad40661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:26.426 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.362+0000 7ffae7fff700 1 -- 172.21.15.195:0/3600839404 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7234+0+0 (secure 0 0 0) 0x7ffad8082120 con 0x7ffaf0150500 2022-06-27T12:09:26.427 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.362+0000 7ffaeffff700 1 --2- 172.21.15.195:0/3600839404 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ffad4063d00 0x7ffad40661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:26.428 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.362+0000 7ffaeffff700 1 --2- 172.21.15.195:0/3600839404 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ffad4063d00 0x7ffad40661b0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7ffae00016f0 tx=0x7ffae0009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:26.428 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.366+0000 7ffae7fff700 1 -- 172.21.15.195:0/3600839404 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7ffaf0166810 con 0x7ffaf0150500 2022-06-27T12:09:26.595 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.594+0000 7ffaf6442700 1 -- 172.21.15.195:0/3600839404 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffaf0167030 con 0x7ffaf0150500 2022-06-27T12:09:26.596 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.595+0000 7ffae7fff700 1 -- 172.21.15.195:0/3600839404 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+5364 (secure 0 0 0) 0x7ffaf0167030 con 0x7ffaf0150500 2022-06-27T12:09:26.615 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:09:26.615 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:09:25.119670+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6708},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6708":{"gid":6708,"name":"b","rank":0,"incarnation":79,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6833/702244802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":702244802},{"type":"v1","addr":"172.21.15.202:6833","nonce":702244802}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:09:26.619 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.597+0000 7ffaf6442700 1 -- 172.21.15.195:0/3600839404 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ffad4063d00 msgr2=0x7ffad40661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:26.619 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.597+0000 7ffaf6442700 1 --2- 172.21.15.195:0/3600839404 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ffad4063d00 0x7ffad40661b0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7ffae00016f0 tx=0x7ffae0009040 comp rx=0 tx=0).stop 2022-06-27T12:09:26.619 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.597+0000 7ffaf6442700 1 -- 172.21.15.195:0/3600839404 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffaf0150500 msgr2=0x7ffaf0166af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:26.620 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.597+0000 7ffaf6442700 1 --2- 172.21.15.195:0/3600839404 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffaf0150500 0x7ffaf0166af0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7ffad8009e40 tx=0x7ffad8017000 comp rx=0 tx=0).stop 2022-06-27T12:09:26.620 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.597+0000 7ffaf6442700 1 -- 172.21.15.195:0/3600839404 shutdown_connections 2022-06-27T12:09:26.620 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.597+0000 7ffaf6442700 1 --2- 172.21.15.195:0/3600839404 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ffad4063d00 0x7ffad40661b0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:26.620 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.597+0000 7ffaf6442700 1 --2- 172.21.15.195:0/3600839404 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ffaf01508d0 0x7ffaf015ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:26.621 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.597+0000 7ffaf6442700 1 --2- 172.21.15.195:0/3600839404 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ffaf006a450 0x7ffaf0155010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:26.621 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.597+0000 7ffaf6442700 1 --2- 172.21.15.195:0/3600839404 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffaf0150500 0x7ffaf0166af0 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:26.621 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.597+0000 7ffaf6442700 1 -- 172.21.15.195:0/3600839404 >> 172.21.15.195:0/3600839404 conn(0x7ffaf00c9b00 msgr2=0x7ffaf0069b50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:09:26.622 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.597+0000 7ffaf6442700 1 -- 172.21.15.195:0/3600839404 shutdown_connections 2022-06-27T12:09:26.622 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.597+0000 7ffaf6442700 1 -- 172.21.15.195:0/3600839404 wait complete. 2022-06-27T12:09:26.622 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 79 2022-06-27T12:09:26.623 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d down, removed from mdsmap 2022-06-27T12:09:26.623 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2022-06-27T12:09:26.624 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:09:26.779 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.744+0000 7f8530dd3700 1 Processor -- start 2022-06-27T12:09:26.789 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.744+0000 7f8530dd3700 1 -- start start 2022-06-27T12:09:26.789 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.745+0000 7f8530dd3700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f852c150500 0x7f852c063af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:26.790 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.745+0000 7f8530dd3700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f852c06a450 0x7f852c0c48d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:26.790 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.745+0000 7f8530dd3700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f852c064030 con 0x7f852c1508d0 2022-06-27T12:09:26.791 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.745+0000 7f8530dd3700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f852c0c4160 con 0x7f852c06a450 2022-06-27T12:09:26.792 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.745+0000 7f8530dd3700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f852c0c4440 con 0x7f852c150500 2022-06-27T12:09:26.793 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.745+0000 7f8529d9b700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f852c06a450 0x7f852c0c48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:26.793 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.745+0000 7f852ad9d700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f852c1508d0 0x7f852c0bbf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:55700/0 (socket says 172.21.15.195:55700) 2022-06-27T12:09:26.794 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.745+0000 7f852ad9d700 1 -- 172.21.15.195:0/2376862919 learned_addr learned my addr 172.21.15.195:0/2376862919 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:09:26.795 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.745+0000 7f852a59c700 1 --2- 172.21.15.195:0/2376862919 >> v2:172.21.15.202:3300/0 conn(0x7f852c150500 0x7f852c063af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:26.795 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.745+0000 7f8529d9b700 1 -- 172.21.15.195:0/2376862919 >> v2:172.21.15.202:3300/0 conn(0x7f852c150500 msgr2=0x7f852c063af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:09:26.796 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.745+0000 7f8529d9b700 1 --2- 172.21.15.195:0/2376862919 >> v2:172.21.15.202:3300/0 conn(0x7f852c150500 0x7f852c063af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:26.796 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.745+0000 7f8529d9b700 1 -- 172.21.15.195:0/2376862919 >> v1:172.21.15.195:6789/0 conn(0x7f852c1508d0 legacy=0x7f852c0bbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:26.797 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.745+0000 7f852959a700 1 -- 172.21.15.195:0/2376862919 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1928326381 0 0) 0x7f8524001490 con 0x7f852c1508d0 2022-06-27T12:09:26.797 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.745+0000 7f8529d9b700 1 -- 172.21.15.195:0/2376862919 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f852c15a800 con 0x7f852c06a450 2022-06-27T12:09:26.798 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.746+0000 7f8529d9b700 1 --2- 172.21.15.195:0/2376862919 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f852c06a450 0x7f852c0c48d0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f852c06f840 tx=0x7f851c009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae12959e53f21b10 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:26.798 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.746+0000 7f852959a700 1 -- 172.21.15.195:0/2376862919 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f851c011630 con 0x7f852c06a450 2022-06-27T12:09:26.799 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.746+0000 7f852959a700 1 -- 172.21.15.195:0/2376862919 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f851c011790 con 0x7f852c06a450 2022-06-27T12:09:26.800 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.746+0000 7f852959a700 1 -- 172.21.15.195:0/2376862919 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f851c011b30 con 0x7f852c06a450 2022-06-27T12:09:26.800 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.746+0000 7f8530dd3700 1 -- 172.21.15.195:0/2376862919 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f852c06a450 msgr2=0x7f852c0c48d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:26.801 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.746+0000 7f8530dd3700 1 --2- 172.21.15.195:0/2376862919 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f852c06a450 0x7f852c0c48d0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f852c06f840 tx=0x7f851c009060 comp rx=0 tx=0).stop 2022-06-27T12:09:26.802 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.746+0000 7f8530dd3700 1 -- 172.21.15.195:0/2376862919 shutdown_connections 2022-06-27T12:09:26.802 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.746+0000 7f8530dd3700 1 --2- 172.21.15.195:0/2376862919 >> v2:172.21.15.202:3300/0 conn(0x7f852c150500 0x7f852c063af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:26.803 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.746+0000 7f8530dd3700 1 --2- 172.21.15.195:0/2376862919 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f852c06a450 0x7f852c0c48d0 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:26.804 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.746+0000 7f8530dd3700 1 -- 172.21.15.195:0/2376862919 >> 172.21.15.195:0/2376862919 conn(0x7f852c0c9b00 msgr2=0x7f852c0c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:09:26.804 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.746+0000 7f8530dd3700 1 -- 172.21.15.195:0/2376862919 shutdown_connections 2022-06-27T12:09:26.805 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.746+0000 7f8530dd3700 1 -- 172.21.15.195:0/2376862919 wait complete. 2022-06-27T12:09:26.805 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.748+0000 7f8530dd3700 1 Processor -- start 2022-06-27T12:09:26.806 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.748+0000 7f8530dd3700 1 -- start start 2022-06-27T12:09:26.807 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.749+0000 7f8530dd3700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f852c06a450 0x7f852c1591b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:26.807 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.749+0000 7f8530dd3700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f852c150500 0x7f852c16a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:26.808 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.749+0000 7f8530dd3700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f852c1508d0 0x7f852c150f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:26.808 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.749+0000 7f8530dd3700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f852c169f70 con 0x7f852c1508d0 2022-06-27T12:09:26.809 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.749+0000 7f8530dd3700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f852c069d40 con 0x7f852c06a450 2022-06-27T12:09:26.809 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.749+0000 7f8530dd3700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f852c152fe0 con 0x7f852c150500 2022-06-27T12:09:26.809 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.749+0000 7f852ad9d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f852c1508d0 0x7f852c150f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:26.810 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.749+0000 7f852ad9d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f852c1508d0 0x7f852c150f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:49518/0 (socket says 172.21.15.195:49518) 2022-06-27T12:09:26.810 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.749+0000 7f852ad9d700 1 -- 172.21.15.195:0/886403129 learned_addr learned my addr 172.21.15.195:0/886403129 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:09:26.811 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.749+0000 7f8529d9b700 1 --2- 172.21.15.195:0/886403129 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f852c150500 0x7f852c16a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:26.811 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.749+0000 7f852a59c700 1 --2- 172.21.15.195:0/886403129 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f852c06a450 0x7f852c1591b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:26.812 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.749+0000 7f852ad9d700 1 -- 172.21.15.195:0/886403129 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f852c150500 msgr2=0x7f852c16a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:26.813 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.749+0000 7f852ad9d700 1 --2- 172.21.15.195:0/886403129 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f852c150500 0x7f852c16a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:26.814 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.749+0000 7f852ad9d700 1 -- 172.21.15.195:0/886403129 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f852c06a450 msgr2=0x7f852c1591b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:26.815 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.749+0000 7f852ad9d700 1 --2- 172.21.15.195:0/886403129 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f852c06a450 0x7f852c1591b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:26.815 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.749+0000 7f852ad9d700 1 -- 172.21.15.195:0/886403129 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f851c0092e0 con 0x7f852c1508d0 2022-06-27T12:09:26.816 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.749+0000 7f852ad9d700 1 --2- 172.21.15.195:0/886403129 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f852c1508d0 0x7f852c150f70 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f852400c810 tx=0x7f8524001540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:26.816 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.750+0000 7f851a7fc700 1 -- 172.21.15.195:0/886403129 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8524004c80 con 0x7f852c1508d0 2022-06-27T12:09:26.817 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.750+0000 7f851a7fc700 1 -- 172.21.15.195:0/886403129 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8524004e00 con 0x7f852c1508d0 2022-06-27T12:09:26.817 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.750+0000 7f8530dd3700 1 -- 172.21.15.195:0/886403129 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f852c1532a0 con 0x7f852c1508d0 2022-06-27T12:09:26.818 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.750+0000 7f851a7fc700 1 -- 172.21.15.195:0/886403129 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85240073a0 con 0x7f852c1508d0 2022-06-27T12:09:26.818 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.750+0000 7f8530dd3700 1 -- 172.21.15.195:0/886403129 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f852c153860 con 0x7f852c1508d0 2022-06-27T12:09:26.819 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.751+0000 7f8530dd3700 1 -- 172.21.15.195:0/886403129 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f852c16a0e0 con 0x7f852c1508d0 2022-06-27T12:09:26.819 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.751+0000 7f851a7fc700 1 -- 172.21.15.195:0/886403129 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f852401a030 con 0x7f852c1508d0 2022-06-27T12:09:26.820 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.751+0000 7f851a7fc700 1 --2- 172.21.15.195:0/886403129 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8508063d00 0x7f85080661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:26.820 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.751+0000 7f851a7fc700 1 -- 172.21.15.195:0/886403129 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7234+0+0 (secure 0 0 0) 0x7f852407ff50 con 0x7f852c1508d0 2022-06-27T12:09:26.821 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.751+0000 7f852a59c700 1 --2- 172.21.15.195:0/886403129 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8508063d00 0x7f85080661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:26.821 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.752+0000 7f852a59c700 1 --2- 172.21.15.195:0/886403129 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8508063d00 0x7f85080661b0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f8520003740 tx=0x7f8520008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:26.821 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.755+0000 7f851a7fc700 1 -- 172.21.15.195:0/886403129 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f852404e6a0 con 0x7f852c1508d0 2022-06-27T12:09:26.978 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.978+0000 7f8530dd3700 1 -- 172.21.15.195:0/886403129 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f852c064030 con 0x7f852c1508d0 2022-06-27T12:09:26.979 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.978+0000 7f851a7fc700 1 -- 172.21.15.195:0/886403129 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+5364 (secure 0 0 0) 0x7f852c064030 con 0x7f852c1508d0 2022-06-27T12:09:26.980 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:09:26.980 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:09:25.119670+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6708},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6708":{"gid":6708,"name":"b","rank":0,"incarnation":79,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6833/702244802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":702244802},{"type":"v1","addr":"172.21.15.202:6833","nonce":702244802}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:09:26.984 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.980+0000 7f8530dd3700 1 -- 172.21.15.195:0/886403129 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8508063d00 msgr2=0x7f85080661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:26.984 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.980+0000 7f8530dd3700 1 --2- 172.21.15.195:0/886403129 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8508063d00 0x7f85080661b0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f8520003740 tx=0x7f8520008040 comp rx=0 tx=0).stop 2022-06-27T12:09:26.984 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.980+0000 7f8530dd3700 1 -- 172.21.15.195:0/886403129 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f852c1508d0 msgr2=0x7f852c150f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:26.984 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.980+0000 7f8530dd3700 1 --2- 172.21.15.195:0/886403129 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f852c1508d0 0x7f852c150f70 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f852400c810 tx=0x7f8524001540 comp rx=0 tx=0).stop 2022-06-27T12:09:26.985 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.980+0000 7f8530dd3700 1 -- 172.21.15.195:0/886403129 shutdown_connections 2022-06-27T12:09:26.985 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.980+0000 7f8530dd3700 1 --2- 172.21.15.195:0/886403129 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8508063d00 0x7f85080661b0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:26.985 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.980+0000 7f8530dd3700 1 --2- 172.21.15.195:0/886403129 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f852c1508d0 0x7f852c150f70 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:26.985 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.980+0000 7f8530dd3700 1 --2- 172.21.15.195:0/886403129 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f852c06a450 0x7f852c1591b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:26.986 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.980+0000 7f8530dd3700 1 --2- 172.21.15.195:0/886403129 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f852c150500 0x7f852c16a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:26.986 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.980+0000 7f8530dd3700 1 -- 172.21.15.195:0/886403129 >> 172.21.15.195:0/886403129 conn(0x7f852c0c9b00 msgr2=0x7f852c0c7920 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:09:26.986 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.980+0000 7f8530dd3700 1 -- 172.21.15.195:0/886403129 shutdown_connections 2022-06-27T12:09:26.986 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:26.980+0000 7f8530dd3700 1 -- 172.21.15.195:0/886403129 wait complete. 2022-06-27T12:09:26.987 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 79 2022-06-27T12:09:27.013 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 79, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-06-27T11:47:52.358095+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 79, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6708": { "addr": "172.21.15.202:6833/702244802", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6832", "nonce": 702244802, "type": "v2" }, { "addr": "172.21.15.202:6833", "nonce": 702244802, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 6708, "incarnation": 79, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-06-27T12:09:25.119670+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6708 } } } ], "standbys": [ { "addr": "172.21.15.202:6837/3305006841", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6836", "nonce": 3305006841, "type": "v2" }, { "addr": "172.21.15.202:6837", "nonce": 3305006841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6835/2060709822", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6834", "nonce": 2060709822, "type": "v2" }, { "addr": "172.21.15.195:6835", "nonce": 2060709822, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6838/1598751652", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6836", "nonce": 1598751652, "type": "v2" }, { "addr": "172.21.15.195:6838", "nonce": 1598751652, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4275, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6839/1313800661", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6837", "nonce": 1313800661, "type": "v2" }, { "addr": "172.21.15.195:6839", "nonce": 1313800661, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4278, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-06-27T12:09:29.014 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:09:29.148 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.148+0000 7f050072d700 1 Processor -- start 2022-06-27T12:09:29.148 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.148+0000 7f050072d700 1 -- start start 2022-06-27T12:09:29.151 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.149+0000 7f050072d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f04f80c3f50 0x7f04f815ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:29.151 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.149+0000 7f050072d700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04f80c33b0 con 0x7f04f814e860 2022-06-27T12:09:29.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.149+0000 7f050072d700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04f806f220 con 0x7f04f814e490 2022-06-27T12:09:29.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.149+0000 7f050072d700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04f81542c0 con 0x7f04f80c3f50 2022-06-27T12:09:29.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.149+0000 7f04fdcc8700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f04f814e860 0x7f04f80c3840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:55714/0 (socket says 172.21.15.195:55714) 2022-06-27T12:09:29.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.149+0000 7f04fdcc8700 1 -- 172.21.15.195:0/873366688 learned_addr learned my addr 172.21.15.195:0/873366688 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:09:29.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.149+0000 7f04fecca700 1 --2- 172.21.15.195:0/873366688 >> v2:172.21.15.195:3300/0 conn(0x7f04f80c3f50 0x7f04f815ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:29.162 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.149+0000 7f04fecca700 1 -- 172.21.15.195:0/873366688 >> v1:172.21.15.202:6789/0 conn(0x7f04f814e490 legacy=0x7f04f806cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:29.162 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.149+0000 7f04fecca700 1 -- 172.21.15.195:0/873366688 >> v1:172.21.15.195:6789/0 conn(0x7f04f814e860 legacy=0x7f04f80c3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:29.163 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.149+0000 7f04fecca700 1 -- 172.21.15.195:0/873366688 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04f8154560 con 0x7f04f80c3f50 2022-06-27T12:09:29.163 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.149+0000 7f04fd4c7700 1 -- 172.21.15.195:0/873366688 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3542482856 0 0) 0x7f04f80c33b0 con 0x7f04f814e860 2022-06-27T12:09:29.163 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.149+0000 7f04fecca700 1 --2- 172.21.15.195:0/873366688 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f04f80c3f50 0x7f04f815ec30 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f04f0009e60 tx=0x7f04f0009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ed6679060faab9e server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:29.163 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.150+0000 7f04fd4c7700 1 -- 172.21.15.195:0/873366688 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04f000e070 con 0x7f04f80c3f50 2022-06-27T12:09:29.164 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.150+0000 7f04fd4c7700 1 -- 172.21.15.195:0/873366688 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04f000bc30 con 0x7f04f80c3f50 2022-06-27T12:09:29.164 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.150+0000 7f04fd4c7700 1 -- 172.21.15.195:0/873366688 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04f0012430 con 0x7f04f80c3f50 2022-06-27T12:09:29.164 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.150+0000 7f050072d700 1 -- 172.21.15.195:0/873366688 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f04f80c3f50 msgr2=0x7f04f815ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:29.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.150+0000 7f050072d700 1 --2- 172.21.15.195:0/873366688 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f04f80c3f50 0x7f04f815ec30 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f04f0009e60 tx=0x7f04f0009780 comp rx=0 tx=0).stop 2022-06-27T12:09:29.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.150+0000 7f050072d700 1 -- 172.21.15.195:0/873366688 shutdown_connections 2022-06-27T12:09:29.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.150+0000 7f050072d700 1 --2- 172.21.15.195:0/873366688 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f04f80c3f50 0x7f04f815ec30 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:29.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.150+0000 7f050072d700 1 -- 172.21.15.195:0/873366688 >> 172.21.15.195:0/873366688 conn(0x7f04f8063af0 msgr2=0x7f04f8063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:09:29.167 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.150+0000 7f050072d700 1 -- 172.21.15.195:0/873366688 shutdown_connections 2022-06-27T12:09:29.167 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.150+0000 7f050072d700 1 -- 172.21.15.195:0/873366688 wait complete. 2022-06-27T12:09:29.171 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.152+0000 7f050072d700 1 Processor -- start 2022-06-27T12:09:29.171 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.152+0000 7f050072d700 1 -- start start 2022-06-27T12:09:29.171 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.153+0000 7f050072d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f04f80c3f50 0x7f04f80c7710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:29.172 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.153+0000 7f050072d700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f04f814e490 0x7f04f81654b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:29.172 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.153+0000 7f050072d700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f04f814e860 0x7f04f81676f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:29.172 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.153+0000 7f050072d700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04f8168490 con 0x7f04f80c3f50 2022-06-27T12:09:29.173 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.153+0000 7f050072d700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04f80c18a0 con 0x7f04f814e490 2022-06-27T12:09:29.173 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.153+0000 7f050072d700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04f80bf870 con 0x7f04f814e860 2022-06-27T12:09:29.173 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.153+0000 7f04fe4c9700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f04f80c3f50 0x7f04f80c7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:29.174 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.153+0000 7f04fe4c9700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f04f80c3f50 0x7f04f80c7710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:49530/0 (socket says 172.21.15.195:49530) 2022-06-27T12:09:29.174 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.153+0000 7f04fe4c9700 1 -- 172.21.15.195:0/4133999962 learned_addr learned my addr 172.21.15.195:0/4133999962 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:09:29.174 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.153+0000 7f04fdcc8700 1 --2- 172.21.15.195:0/4133999962 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f04f814e490 0x7f04f81654b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:29.175 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.153+0000 7f04fecca700 1 --2- 172.21.15.195:0/4133999962 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f04f814e860 0x7f04f81676f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:29.175 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.153+0000 7f04fe4c9700 1 -- 172.21.15.195:0/4133999962 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f04f814e860 msgr2=0x7f04f81676f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:29.175 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.153+0000 7f04fe4c9700 1 --2- 172.21.15.195:0/4133999962 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f04f814e860 0x7f04f81676f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:29.175 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.153+0000 7f04fe4c9700 1 -- 172.21.15.195:0/4133999962 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f04f814e490 msgr2=0x7f04f81654b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:29.176 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.153+0000 7f04fe4c9700 1 --2- 172.21.15.195:0/4133999962 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f04f814e490 0x7f04f81654b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:29.176 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.153+0000 7f04fe4c9700 1 -- 172.21.15.195:0/4133999962 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04f0009a20 con 0x7f04f80c3f50 2022-06-27T12:09:29.176 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.153+0000 7f04fecca700 1 --2- 172.21.15.195:0/4133999962 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f04f814e860 0x7f04f81676f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T12:09:29.176 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.153+0000 7f04fe4c9700 1 --2- 172.21.15.195:0/4133999962 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f04f80c3f50 0x7f04f80c7710 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f04f400c490 tx=0x7f04f4007e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:29.177 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.153+0000 7f04e67fc700 1 -- 172.21.15.195:0/4133999962 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04f40050d0 con 0x7f04f80c3f50 2022-06-27T12:09:29.177 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.153+0000 7f04e67fc700 1 -- 172.21.15.195:0/4133999962 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04f4005230 con 0x7f04f80c3f50 2022-06-27T12:09:29.177 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.153+0000 7f050072d700 1 -- 172.21.15.195:0/4133999962 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04f80c12e0 con 0x7f04f80c3f50 2022-06-27T12:09:29.177 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.153+0000 7f04e67fc700 1 -- 172.21.15.195:0/4133999962 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04f40117a0 con 0x7f04f80c3f50 2022-06-27T12:09:29.177 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.153+0000 7f050072d700 1 -- 172.21.15.195:0/4133999962 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04f80c5d20 con 0x7f04f80c3f50 2022-06-27T12:09:29.178 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.154+0000 7f050072d700 1 -- 172.21.15.195:0/4133999962 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04f80c5ee0 con 0x7f04f80c3f50 2022-06-27T12:09:29.178 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.155+0000 7f04e67fc700 1 -- 172.21.15.195:0/4133999962 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f04f4019030 con 0x7f04f80c3f50 2022-06-27T12:09:29.178 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.155+0000 7f04e67fc700 1 --2- 172.21.15.195:0/4133999962 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f04d8063d30 0x7f04d80661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:29.178 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.155+0000 7f04e67fc700 1 -- 172.21.15.195:0/4133999962 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7234+0+0 (secure 0 0 0) 0x7f04f407fdd0 con 0x7f04f80c3f50 2022-06-27T12:09:29.179 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.155+0000 7f04fdcc8700 1 --2- 172.21.15.195:0/4133999962 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f04d8063d30 0x7f04d80661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:29.179 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.156+0000 7f04fdcc8700 1 --2- 172.21.15.195:0/4133999962 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f04d8063d30 0x7f04d80661e0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f04f80c1ab0 tx=0x7f04e8007b00 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:29.179 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.159+0000 7f04e67fc700 1 -- 172.21.15.195:0/4133999962 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f04f4049c90 con 0x7f04f80c3f50 2022-06-27T12:09:29.395 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.389+0000 7f050072d700 1 -- 172.21.15.195:0/4133999962 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f04f80c36e0 con 0x7f04f80c3f50 2022-06-27T12:09:29.396 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.389+0000 7f04e67fc700 1 -- 172.21.15.195:0/4133999962 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+5364 (secure 0 0 0) 0x7f04f4017020 con 0x7f04f80c3f50 2022-06-27T12:09:29.398 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:09:29.398 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:09:25.119670+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6708},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6708":{"gid":6708,"name":"b","rank":0,"incarnation":79,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6833/702244802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":702244802},{"type":"v1","addr":"172.21.15.202:6833","nonce":702244802}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:09:29.413 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.391+0000 7f050072d700 1 -- 172.21.15.195:0/4133999962 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f04d8063d30 msgr2=0x7f04d80661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:29.440 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.391+0000 7f050072d700 1 --2- 172.21.15.195:0/4133999962 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f04d8063d30 0x7f04d80661e0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f04f80c1ab0 tx=0x7f04e8007b00 comp rx=0 tx=0).stop 2022-06-27T12:09:29.441 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.391+0000 7f050072d700 1 -- 172.21.15.195:0/4133999962 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f04f80c3f50 msgr2=0x7f04f80c7710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:29.441 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.391+0000 7f050072d700 1 --2- 172.21.15.195:0/4133999962 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f04f80c3f50 0x7f04f80c7710 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f04f400c490 tx=0x7f04f4007e40 comp rx=0 tx=0).stop 2022-06-27T12:09:29.441 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.392+0000 7f050072d700 1 -- 172.21.15.195:0/4133999962 shutdown_connections 2022-06-27T12:09:29.442 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.392+0000 7f050072d700 1 --2- 172.21.15.195:0/4133999962 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f04d8063d30 0x7f04d80661e0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:29.442 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.392+0000 7f050072d700 1 --2- 172.21.15.195:0/4133999962 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f04f80c3f50 0x7f04f80c7710 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:29.443 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.392+0000 7f050072d700 1 --2- 172.21.15.195:0/4133999962 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f04f814e860 0x7f04f81676f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:29.443 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.392+0000 7f050072d700 1 --2- 172.21.15.195:0/4133999962 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f04f814e490 0x7f04f81654b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:29.444 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.392+0000 7f050072d700 1 -- 172.21.15.195:0/4133999962 >> 172.21.15.195:0/4133999962 conn(0x7f04f8063af0 msgr2=0x7f04f806cfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:09:29.445 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.392+0000 7f050072d700 1 -- 172.21.15.195:0/4133999962 shutdown_connections 2022-06-27T12:09:29.445 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:29.392+0000 7f050072d700 1 -- 172.21.15.195:0/4133999962 wait complete. 2022-06-27T12:09:29.445 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 79 2022-06-27T12:09:29.446 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:09:31.448 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:09:31.610 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.582+0000 7f828a9d6700 1 Processor -- start 2022-06-27T12:09:31.611 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.582+0000 7f828a9d6700 1 -- start start 2022-06-27T12:09:31.611 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.583+0000 7f828a9d6700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8284067ed0 0x7f82840c7440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:31.612 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.583+0000 7f828a9d6700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f8284069390 0x7f82841502b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:31.612 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.583+0000 7f828a9d6700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82840c43d0 con 0x7f8284156670 2022-06-27T12:09:31.612 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.583+0000 7f828a9d6700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8284068430 con 0x7f8284069390 2022-06-27T12:09:31.613 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.583+0000 7f828a9d6700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8284068710 con 0x7f8284067ed0 2022-06-27T12:09:31.614 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.583+0000 7f8288f73700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f8284069390 0x7f82841502b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:31.614 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.583+0000 7f8283fff700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f8284156670 0x7f82840c4860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:55730/0 (socket says 172.21.15.195:55730) 2022-06-27T12:09:31.615 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.583+0000 7f8283fff700 1 -- 172.21.15.195:0/900669531 learned_addr learned my addr 172.21.15.195:0/900669531 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:09:31.616 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.583+0000 7f82837fe700 1 --2- 172.21.15.195:0/900669531 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8284067ed0 0x7f82840c7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:31.616 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.583+0000 7f8288f73700 1 -- 172.21.15.195:0/900669531 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8284067ed0 msgr2=0x7f82840c7440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:09:31.617 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.583+0000 7f8288f73700 1 --2- 172.21.15.195:0/900669531 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8284067ed0 0x7f82840c7440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:31.617 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.583+0000 7f8288f73700 1 -- 172.21.15.195:0/900669531 >> v1:172.21.15.195:6789/0 conn(0x7f8284156670 legacy=0x7f82840c4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:31.617 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.583+0000 7f8288f73700 1 -- 172.21.15.195:0/900669531 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82840689f0 con 0x7f8284069390 2022-06-27T12:09:31.619 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.583+0000 7f8282ffd700 1 -- 172.21.15.195:0/900669531 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 781218156 0 0) 0x7f82840c43d0 con 0x7f8284156670 2022-06-27T12:09:31.627 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.583+0000 7f8288f73700 1 --2- 172.21.15.195:0/900669531 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8284069390 0x7f82841502b0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f827c009e90 tx=0x7f827c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e0040cf4d3d4ae8 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:31.628 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.583+0000 7f8282ffd700 1 -- 172.21.15.195:0/900669531 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f827c00e070 con 0x7f8284069390 2022-06-27T12:09:31.628 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.583+0000 7f8282ffd700 1 -- 172.21.15.195:0/900669531 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f827c00bbf0 con 0x7f8284069390 2022-06-27T12:09:31.632 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.583+0000 7f8282ffd700 1 -- 172.21.15.195:0/900669531 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f827c012430 con 0x7f8284069390 2022-06-27T12:09:31.632 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.584+0000 7f828a9d6700 1 -- 172.21.15.195:0/900669531 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8284069390 msgr2=0x7f82841502b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:31.633 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.584+0000 7f828a9d6700 1 --2- 172.21.15.195:0/900669531 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8284069390 0x7f82841502b0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f827c009e90 tx=0x7f827c009780 comp rx=0 tx=0).stop 2022-06-27T12:09:31.634 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.584+0000 7f828a9d6700 1 -- 172.21.15.195:0/900669531 shutdown_connections 2022-06-27T12:09:31.634 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.584+0000 7f828a9d6700 1 --2- 172.21.15.195:0/900669531 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8284069390 0x7f82841502b0 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:31.634 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.584+0000 7f828a9d6700 1 --2- 172.21.15.195:0/900669531 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8284067ed0 0x7f82840c7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:31.635 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.584+0000 7f828a9d6700 1 -- 172.21.15.195:0/900669531 >> 172.21.15.195:0/900669531 conn(0x7f8284069e00 msgr2=0x7f828406a200 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:09:31.635 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.584+0000 7f828a9d6700 1 -- 172.21.15.195:0/900669531 shutdown_connections 2022-06-27T12:09:31.636 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.584+0000 7f828a9d6700 1 -- 172.21.15.195:0/900669531 wait complete. 2022-06-27T12:09:31.636 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.585+0000 7f828a9d6700 1 Processor -- start 2022-06-27T12:09:31.636 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.585+0000 7f828a9d6700 1 -- start start 2022-06-27T12:09:31.637 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.586+0000 7f828a9d6700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8284067ed0 0x7f828414c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:31.637 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.586+0000 7f828a9d6700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8284069390 0x7f828415cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:31.638 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.586+0000 7f828a9d6700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8284156670 0x7f828415d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:31.639 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.586+0000 7f828a9d6700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8284151250 con 0x7f8284156670 2022-06-27T12:09:31.639 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.586+0000 7f828a9d6700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f828416a6f0 con 0x7f8284067ed0 2022-06-27T12:09:31.640 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.586+0000 7f828a9d6700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8284159730 con 0x7f8284069390 2022-06-27T12:09:31.640 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.586+0000 7f8288f73700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8284156670 0x7f828415d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:31.641 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.586+0000 7f8288f73700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8284156670 0x7f828415d530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:49552/0 (socket says 172.21.15.195:49552) 2022-06-27T12:09:31.641 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.586+0000 7f8288f73700 1 -- 172.21.15.195:0/1350493489 learned_addr learned my addr 172.21.15.195:0/1350493489 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:09:31.641 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.586+0000 7f8283fff700 1 --2- 172.21.15.195:0/1350493489 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8284067ed0 0x7f828414c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:31.644 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.586+0000 7f82837fe700 1 --2- 172.21.15.195:0/1350493489 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8284069390 0x7f828415cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:31.645 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.586+0000 7f8288f73700 1 -- 172.21.15.195:0/1350493489 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8284069390 msgr2=0x7f828415cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:31.645 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.586+0000 7f8288f73700 1 --2- 172.21.15.195:0/1350493489 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8284069390 0x7f828415cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:31.646 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.586+0000 7f8288f73700 1 -- 172.21.15.195:0/1350493489 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8284067ed0 msgr2=0x7f828414c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:31.646 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.586+0000 7f8288f73700 1 --2- 172.21.15.195:0/1350493489 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8284067ed0 0x7f828414c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:31.647 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.586+0000 7f8288f73700 1 -- 172.21.15.195:0/1350493489 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f827c009a20 con 0x7f8284156670 2022-06-27T12:09:31.647 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.587+0000 7f8288f73700 1 --2- 172.21.15.195:0/1350493489 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8284156670 0x7f828415d530 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f827c00b6c0 tx=0x7f827c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:31.665 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.587+0000 7f827bfff700 1 -- 172.21.15.195:0/1350493489 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f827c012e90 con 0x7f8284156670 2022-06-27T12:09:31.666 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.587+0000 7f827bfff700 1 -- 172.21.15.195:0/1350493489 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f827c026480 con 0x7f8284156670 2022-06-27T12:09:31.666 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.587+0000 7f827bfff700 1 -- 172.21.15.195:0/1350493489 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f827c026820 con 0x7f8284156670 2022-06-27T12:09:31.667 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.587+0000 7f828a9d6700 1 -- 172.21.15.195:0/1350493489 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82841519d0 con 0x7f8284156670 2022-06-27T12:09:31.682 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.587+0000 7f828a9d6700 1 -- 172.21.15.195:0/1350493489 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8284169dc0 con 0x7f8284156670 2022-06-27T12:09:31.683 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.588+0000 7f828a9d6700 1 -- 172.21.15.195:0/1350493489 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f828416a860 con 0x7f8284156670 2022-06-27T12:09:31.683 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.588+0000 7f827bfff700 1 -- 172.21.15.195:0/1350493489 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f827c01c400 con 0x7f8284156670 2022-06-27T12:09:31.684 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.588+0000 7f827bfff700 1 --2- 172.21.15.195:0/1350493489 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8268063d30 0x7f82680661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:31.684 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.588+0000 7f827bfff700 1 -- 172.21.15.195:0/1350493489 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7234+0+0 (secure 0 0 0) 0x7f827c082e50 con 0x7f8284156670 2022-06-27T12:09:31.685 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.589+0000 7f8283fff700 1 --2- 172.21.15.195:0/1350493489 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8268063d30 0x7f82680661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:31.685 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.589+0000 7f8283fff700 1 --2- 172.21.15.195:0/1350493489 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8268063d30 0x7f82680661e0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f826c007fd0 tx=0x7f826c007d10 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:31.686 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.593+0000 7f827bfff700 1 -- 172.21.15.195:0/1350493489 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f827c051540 con 0x7f8284156670 2022-06-27T12:09:31.821 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.821+0000 7f828a9d6700 1 -- 172.21.15.195:0/1350493489 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f82840722c0 con 0x7f8284156670 2022-06-27T12:09:31.822 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.821+0000 7f827bfff700 1 -- 172.21.15.195:0/1350493489 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+5364 (secure 0 0 0) 0x7f827c02b090 con 0x7f8284156670 2022-06-27T12:09:31.822 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:09:31.823 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:09:25.119670+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6708},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6708":{"gid":6708,"name":"b","rank":0,"incarnation":79,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6833/702244802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":702244802},{"type":"v1","addr":"172.21.15.202:6833","nonce":702244802}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:09:31.825 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.823+0000 7f828a9d6700 1 -- 172.21.15.195:0/1350493489 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8268063d30 msgr2=0x7f82680661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:31.826 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.823+0000 7f828a9d6700 1 --2- 172.21.15.195:0/1350493489 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8268063d30 0x7f82680661e0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f826c007fd0 tx=0x7f826c007d10 comp rx=0 tx=0).stop 2022-06-27T12:09:31.826 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.823+0000 7f828a9d6700 1 -- 172.21.15.195:0/1350493489 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8284156670 msgr2=0x7f828415d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:31.826 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.823+0000 7f828a9d6700 1 --2- 172.21.15.195:0/1350493489 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8284156670 0x7f828415d530 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f827c00b6c0 tx=0x7f827c019000 comp rx=0 tx=0).stop 2022-06-27T12:09:31.827 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.823+0000 7f828a9d6700 1 -- 172.21.15.195:0/1350493489 shutdown_connections 2022-06-27T12:09:31.827 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.824+0000 7f828a9d6700 1 --2- 172.21.15.195:0/1350493489 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8268063d30 0x7f82680661e0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:31.828 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.824+0000 7f828a9d6700 1 --2- 172.21.15.195:0/1350493489 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8284156670 0x7f828415d530 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:31.828 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.824+0000 7f828a9d6700 1 --2- 172.21.15.195:0/1350493489 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8284067ed0 0x7f828414c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:31.828 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.824+0000 7f828a9d6700 1 --2- 172.21.15.195:0/1350493489 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8284069390 0x7f828415cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:31.828 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.824+0000 7f828a9d6700 1 -- 172.21.15.195:0/1350493489 >> 172.21.15.195:0/1350493489 conn(0x7f8284069e00 msgr2=0x7f828414e500 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:09:31.829 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.824+0000 7f828a9d6700 1 -- 172.21.15.195:0/1350493489 shutdown_connections 2022-06-27T12:09:31.829 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:31.824+0000 7f828a9d6700 1 -- 172.21.15.195:0/1350493489 wait complete. 2022-06-27T12:09:31.829 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 79 2022-06-27T12:09:31.836 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:09:32.432 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:09:32.457 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:09:33.838 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:09:33.985 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:33.979+0000 7f33a1370700 1 Processor -- start 2022-06-27T12:09:33.985 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:33.979+0000 7f33a1370700 1 -- start start 2022-06-27T12:09:33.986 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:33.980+0000 7f33a1370700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f339c0c64f0 0x7f339c150bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:33.986 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:33.980+0000 7f33a1370700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f339c0c5d80 con 0x7f339c06d280 2022-06-27T12:09:33.990 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:33.980+0000 7f33a1370700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f339c06b010 con 0x7f339c0c5480 2022-06-27T12:09:33.991 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:33.980+0000 7f33a1370700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f339c0c75e0 con 0x7f339c0c64f0 2022-06-27T12:09:33.993 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:33.980+0000 7f339b7fe700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f339c06d280 0x7f339c161010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:55750/0 (socket says 172.21.15.195:55750) 2022-06-27T12:09:33.998 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:33.980+0000 7f339b7fe700 1 -- 172.21.15.195:0/3027443142 learned_addr learned my addr 172.21.15.195:0/3027443142 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:09:33.999 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:33.980+0000 7f339a7fc700 1 --2- 172.21.15.195:0/3027443142 >> v2:172.21.15.202:3300/0 conn(0x7f339c0c64f0 0x7f339c150bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:33.999 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:33.980+0000 7f3399ffb700 1 -- 172.21.15.195:0/3027443142 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 104270167 0 0) 0x7f339c0c5d80 con 0x7f339c06d280 2022-06-27T12:09:34.021 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:33.980+0000 7f3399ffb700 1 -- 172.21.15.195:0/3027443142 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3388003430 con 0x7f339c06d280 2022-06-27T12:09:34.027 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:33.980+0000 7f339a7fc700 1 -- 172.21.15.195:0/3027443142 >> v1:172.21.15.202:6789/0 conn(0x7f339c0c5480 legacy=0x7f339c1504c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:34.027 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:33.980+0000 7f339a7fc700 1 -- 172.21.15.195:0/3027443142 >> v1:172.21.15.195:6789/0 conn(0x7f339c06d280 legacy=0x7f339c161010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:34.028 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:33.980+0000 7f3399ffb700 1 -- 172.21.15.195:0/3027443142 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 124513986 0 0) 0x7f3388003430 con 0x7f339c06d280 2022-06-27T12:09:34.028 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:33.980+0000 7f339a7fc700 1 -- 172.21.15.195:0/3027443142 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f339c0c78c0 con 0x7f339c0c64f0 2022-06-27T12:09:34.029 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:33.981+0000 7f339a7fc700 1 --2- 172.21.15.195:0/3027443142 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f339c0c64f0 0x7f339c150bd0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f338c0097a0 tx=0x7f338c009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f89ee5742d5cc256 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:34.029 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:33.981+0000 7f3399ffb700 1 -- 172.21.15.195:0/3027443142 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33940017f0 con 0x7f339c0c64f0 2022-06-27T12:09:34.033 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:33.981+0000 7f3399ffb700 1 -- 172.21.15.195:0/3027443142 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f338c004030 con 0x7f339c0c64f0 2022-06-27T12:09:34.034 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:33.981+0000 7f3399ffb700 1 -- 172.21.15.195:0/3027443142 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f338c0043b0 con 0x7f339c0c64f0 2022-06-27T12:09:34.035 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:33.981+0000 7f33a1370700 1 -- 172.21.15.195:0/3027443142 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f339c0c64f0 msgr2=0x7f339c150bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:34.035 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:33.981+0000 7f33a1370700 1 --2- 172.21.15.195:0/3027443142 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f339c0c64f0 0x7f339c150bd0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f338c0097a0 tx=0x7f338c009060 comp rx=0 tx=0).stop 2022-06-27T12:09:34.036 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:33.981+0000 7f33a1370700 1 -- 172.21.15.195:0/3027443142 shutdown_connections 2022-06-27T12:09:34.037 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:33.981+0000 7f33a1370700 1 --2- 172.21.15.195:0/3027443142 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f339c0c64f0 0x7f339c150bd0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:34.038 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:33.981+0000 7f33a1370700 1 -- 172.21.15.195:0/3027443142 >> 172.21.15.195:0/3027443142 conn(0x7f339c063af0 msgr2=0x7f339c063ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:09:34.038 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:33.981+0000 7f33a1370700 1 -- 172.21.15.195:0/3027443142 shutdown_connections 2022-06-27T12:09:34.038 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:33.981+0000 7f33a1370700 1 -- 172.21.15.195:0/3027443142 wait complete. 2022-06-27T12:09:34.039 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:33.983+0000 7f33a1370700 1 Processor -- start 2022-06-27T12:09:34.039 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:33.983+0000 7f33a1370700 1 -- start start 2022-06-27T12:09:34.040 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:33.983+0000 7f33a1370700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f339c06d280 0x7f339c166090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:34.040 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:33.983+0000 7f33a1370700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f339c0c5480 0x7f339c0c1e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:34.041 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:33.983+0000 7f33a1370700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f339c0c64f0 0x7f339c0c2370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:34.041 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:33.983+0000 7f33a1370700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f339c158f30 con 0x7f339c0c64f0 2022-06-27T12:09:34.042 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:33.983+0000 7f33a1370700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f339c0c96a0 con 0x7f339c06d280 2022-06-27T12:09:34.042 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:33.983+0000 7f33a1370700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f339c16a100 con 0x7f339c0c5480 2022-06-27T12:09:34.042 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:33.983+0000 7f339b7fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f339c0c64f0 0x7f339c0c2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:34.043 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:33.983+0000 7f339b7fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f339c0c64f0 0x7f339c0c2370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:49568/0 (socket says 172.21.15.195:49568) 2022-06-27T12:09:34.044 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:33.983+0000 7f339b7fe700 1 -- 172.21.15.195:0/3159459044 learned_addr learned my addr 172.21.15.195:0/3159459044 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:09:34.060 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:33.983+0000 7f339affd700 1 --2- 172.21.15.195:0/3159459044 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f339c06d280 0x7f339c166090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:34.061 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:33.984+0000 7f339a7fc700 1 --2- 172.21.15.195:0/3159459044 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f339c0c5480 0x7f339c0c1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:34.061 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:33.984+0000 7f339b7fe700 1 -- 172.21.15.195:0/3159459044 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f339c0c5480 msgr2=0x7f339c0c1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:34.062 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:33.984+0000 7f339b7fe700 1 --2- 172.21.15.195:0/3159459044 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f339c0c5480 0x7f339c0c1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:34.062 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:33.984+0000 7f339b7fe700 1 -- 172.21.15.195:0/3159459044 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f339c06d280 msgr2=0x7f339c166090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:34.062 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:33.984+0000 7f339b7fe700 1 --2- 172.21.15.195:0/3159459044 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f339c06d280 0x7f339c166090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:34.063 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:33.984+0000 7f339b7fe700 1 -- 172.21.15.195:0/3159459044 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f338c0092e0 con 0x7f339c0c64f0 2022-06-27T12:09:34.063 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:33.984+0000 7f339b7fe700 1 --2- 172.21.15.195:0/3159459044 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f339c0c64f0 0x7f339c0c2370 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f339400cb90 tx=0x7f3394001d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:34.064 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:33.984+0000 7f3382ffd700 1 -- 172.21.15.195:0/3159459044 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3394004c80 con 0x7f339c0c64f0 2022-06-27T12:09:34.064 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:33.984+0000 7f3382ffd700 1 -- 172.21.15.195:0/3159459044 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3394004de0 con 0x7f339c0c64f0 2022-06-27T12:09:34.064 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:33.984+0000 7f33a1370700 1 -- 172.21.15.195:0/3159459044 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f339c06ee70 con 0x7f339c0c64f0 2022-06-27T12:09:34.065 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:33.984+0000 7f3382ffd700 1 -- 172.21.15.195:0/3159459044 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3394007380 con 0x7f339c0c64f0 2022-06-27T12:09:34.065 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:33.984+0000 7f33a1370700 1 -- 172.21.15.195:0/3159459044 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f339c16a3a0 con 0x7f339c0c64f0 2022-06-27T12:09:34.066 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:33.985+0000 7f33a1370700 1 -- 172.21.15.195:0/3159459044 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f339c0c9850 con 0x7f339c0c64f0 2022-06-27T12:09:34.066 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:33.985+0000 7f3382ffd700 1 -- 172.21.15.195:0/3159459044 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f339401a030 con 0x7f339c0c64f0 2022-06-27T12:09:34.066 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:33.986+0000 7f3382ffd700 1 --2- 172.21.15.195:0/3159459044 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f337c063d30 0x7f337c0661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:34.067 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:33.986+0000 7f3382ffd700 1 -- 172.21.15.195:0/3159459044 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7234+0+0 (secure 0 0 0) 0x7f33940803f0 con 0x7f339c0c64f0 2022-06-27T12:09:34.067 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:33.986+0000 7f339affd700 1 --2- 172.21.15.195:0/3159459044 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f337c063d30 0x7f337c0661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:34.068 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:33.986+0000 7f339affd700 1 --2- 172.21.15.195:0/3159459044 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f337c063d30 0x7f337c0661e0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f3390001880 tx=0x7f3390009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:34.068 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:33.990+0000 7f3382ffd700 1 -- 172.21.15.195:0/3159459044 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f339404ebc0 con 0x7f339c0c64f0 2022-06-27T12:09:34.223 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:34.218+0000 7f33a1370700 1 -- 172.21.15.195:0/3159459044 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f339c0722c0 con 0x7f339c0c64f0 2022-06-27T12:09:34.223 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:34.219+0000 7f3382ffd700 1 -- 172.21.15.195:0/3159459044 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+5364 (secure 0 0 0) 0x7f3394018070 con 0x7f339c0c64f0 2022-06-27T12:09:34.224 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:09:34.224 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:09:25.119670+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6708},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6708":{"gid":6708,"name":"b","rank":0,"incarnation":79,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6833/702244802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":702244802},{"type":"v1","addr":"172.21.15.202:6833","nonce":702244802}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:09:34.235 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:34.221+0000 7f33a1370700 1 -- 172.21.15.195:0/3159459044 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f337c063d30 msgr2=0x7f337c0661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:34.236 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:34.221+0000 7f33a1370700 1 --2- 172.21.15.195:0/3159459044 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f337c063d30 0x7f337c0661e0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f3390001880 tx=0x7f3390009040 comp rx=0 tx=0).stop 2022-06-27T12:09:34.236 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:34.221+0000 7f33a1370700 1 -- 172.21.15.195:0/3159459044 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f339c0c64f0 msgr2=0x7f339c0c2370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:34.236 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:34.221+0000 7f33a1370700 1 --2- 172.21.15.195:0/3159459044 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f339c0c64f0 0x7f339c0c2370 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f339400cb90 tx=0x7f3394001d90 comp rx=0 tx=0).stop 2022-06-27T12:09:34.237 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:34.221+0000 7f33a1370700 1 -- 172.21.15.195:0/3159459044 shutdown_connections 2022-06-27T12:09:34.237 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:34.221+0000 7f33a1370700 1 --2- 172.21.15.195:0/3159459044 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f337c063d30 0x7f337c0661e0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:34.238 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:34.221+0000 7f33a1370700 1 --2- 172.21.15.195:0/3159459044 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f339c0c64f0 0x7f339c0c2370 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:34.238 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:34.221+0000 7f33a1370700 1 --2- 172.21.15.195:0/3159459044 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f339c06d280 0x7f339c166090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:34.239 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:34.221+0000 7f33a1370700 1 --2- 172.21.15.195:0/3159459044 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f339c0c5480 0x7f339c0c1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:34.239 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:34.221+0000 7f33a1370700 1 -- 172.21.15.195:0/3159459044 >> 172.21.15.195:0/3159459044 conn(0x7f339c063af0 msgr2=0x7f339c0c8800 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:09:34.239 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:34.221+0000 7f33a1370700 1 -- 172.21.15.195:0/3159459044 shutdown_connections 2022-06-27T12:09:34.240 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:34.221+0000 7f33a1370700 1 -- 172.21.15.195:0/3159459044 wait complete. 2022-06-27T12:09:34.240 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 79 2022-06-27T12:09:34.241 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:09:36.246 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:09:36.383 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.382+0000 7ff3b34bf700 1 Processor -- start 2022-06-27T12:09:36.383 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.382+0000 7ff3b34bf700 1 -- start start 2022-06-27T12:09:36.392 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.383+0000 7ff3b34bf700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7ff3ac150500 0x7ff3ac063af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:36.393 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.383+0000 7ff3b34bf700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff3ac1508d0 0x7ff3ac0bbf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:36.393 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.383+0000 7ff3b34bf700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3ac064030 con 0x7ff3ac06a450 2022-06-27T12:09:36.393 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.383+0000 7ff3b34bf700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3ac0c4160 con 0x7ff3ac150500 2022-06-27T12:09:36.394 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.383+0000 7ff3b34bf700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ac0c4440 con 0x7ff3ac1508d0 2022-06-27T12:09:36.394 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.383+0000 7ff3b125b700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7ff3ac150500 0x7ff3ac063af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:36.394 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.383+0000 7ff3b125b700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7ff3ac150500 0x7ff3ac063af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:49572/0 (socket says 172.21.15.195:49572) 2022-06-27T12:09:36.394 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.383+0000 7ff3b125b700 1 -- 172.21.15.195:0/246972823 learned_addr learned my addr 172.21.15.195:0/246972823 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:09:36.395 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.383+0000 7ff3b1a5c700 1 --2- 172.21.15.195:0/246972823 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff3ac1508d0 0x7ff3ac0bbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:36.395 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.384+0000 7ff3b125b700 1 -- 172.21.15.195:0/246972823 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff3ac1508d0 msgr2=0x7ff3ac0bbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:09:36.395 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.384+0000 7ff3b125b700 1 --2- 172.21.15.195:0/246972823 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff3ac1508d0 0x7ff3ac0bbf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:36.395 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.384+0000 7ff3b125b700 1 -- 172.21.15.195:0/246972823 >> v1:172.21.15.202:6789/0 conn(0x7ff3ac06a450 legacy=0x7ff3ac0c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:36.395 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.384+0000 7ff3b125b700 1 -- 172.21.15.195:0/246972823 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3ac15a800 con 0x7ff3ac150500 2022-06-27T12:09:36.396 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.384+0000 7ff3b125b700 1 --2- 172.21.15.195:0/246972823 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff3ac150500 0x7ff3ac063af0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7ff3a8006fd0 tx=0x7ff3a80090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc8e50d17481f35f server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:36.396 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.384+0000 7ff39bfff700 1 -- 172.21.15.195:0/246972823 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a8005ad0 con 0x7ff3ac150500 2022-06-27T12:09:36.396 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.384+0000 7ff39bfff700 1 -- 172.21.15.195:0/246972823 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3a8005c30 con 0x7ff3ac150500 2022-06-27T12:09:36.396 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.384+0000 7ff39bfff700 1 -- 172.21.15.195:0/246972823 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a8004030 con 0x7ff3ac150500 2022-06-27T12:09:36.397 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.384+0000 7ff3b34bf700 1 -- 172.21.15.195:0/246972823 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff3ac150500 msgr2=0x7ff3ac063af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:36.397 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.384+0000 7ff3b34bf700 1 --2- 172.21.15.195:0/246972823 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff3ac150500 0x7ff3ac063af0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7ff3a8006fd0 tx=0x7ff3a80090f0 comp rx=0 tx=0).stop 2022-06-27T12:09:36.397 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.384+0000 7ff3b34bf700 1 -- 172.21.15.195:0/246972823 shutdown_connections 2022-06-27T12:09:36.397 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.384+0000 7ff3b34bf700 1 --2- 172.21.15.195:0/246972823 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff3ac1508d0 0x7ff3ac0bbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:36.397 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.384+0000 7ff3b34bf700 1 --2- 172.21.15.195:0/246972823 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff3ac150500 0x7ff3ac063af0 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:36.398 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.386+0000 7ff3b34bf700 1 -- 172.21.15.195:0/246972823 >> 172.21.15.195:0/246972823 conn(0x7ff3ac0c9b00 msgr2=0x7ff3ac0c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:09:36.398 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.386+0000 7ff3b34bf700 1 -- 172.21.15.195:0/246972823 shutdown_connections 2022-06-27T12:09:36.398 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.386+0000 7ff3b34bf700 1 -- 172.21.15.195:0/246972823 wait complete. 2022-06-27T12:09:36.399 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.386+0000 7ff3b34bf700 1 Processor -- start 2022-06-27T12:09:36.399 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.386+0000 7ff3b34bf700 1 -- start start 2022-06-27T12:09:36.399 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.388+0000 7ff3b34bf700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff3ac06a450 0x7ff3ac150d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:36.399 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.388+0000 7ff3b34bf700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff3ac150500 0x7ff3ac16a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:36.399 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.388+0000 7ff3b34bf700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff3ac1508d0 0x7ff3ac166ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:36.400 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.388+0000 7ff3b34bf700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ac1523e0 con 0x7ff3ac1508d0 2022-06-27T12:09:36.400 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.388+0000 7ff3b34bf700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ac169de0 con 0x7ff3ac150500 2022-06-27T12:09:36.400 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.388+0000 7ff3b34bf700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ac153550 con 0x7ff3ac06a450 2022-06-27T12:09:36.400 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.388+0000 7ff3b1a5c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff3ac1508d0 0x7ff3ac166ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:36.401 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.388+0000 7ff3b125b700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff3ac06a450 0x7ff3ac150d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:36.401 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.388+0000 7ff3b0a5a700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff3ac150500 0x7ff3ac16a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:36.401 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.388+0000 7ff3b1a5c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff3ac1508d0 0x7ff3ac166ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:49582/0 (socket says 172.21.15.195:49582) 2022-06-27T12:09:36.401 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.388+0000 7ff3b125b700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff3ac06a450 0x7ff3ac150d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3301/0 says I am v2:172.21.15.195:42334/0 (socket says 172.21.15.195:42334) 2022-06-27T12:09:36.401 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.388+0000 7ff3b1a5c700 1 -- 172.21.15.195:0/4118567416 learned_addr learned my addr 172.21.15.195:0/4118567416 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:09:36.402 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.388+0000 7ff3b0a5a700 1 --2- 172.21.15.195:0/4118567416 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff3ac150500 0x7ff3ac16a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3300/0 says I am v2:172.21.15.195:47706/0 (socket says 172.21.15.195:47706) 2022-06-27T12:09:36.402 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.388+0000 7ff3b1a5c700 1 -- 172.21.15.195:0/4118567416 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff3ac06a450 msgr2=0x7ff3ac150d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:36.402 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.388+0000 7ff3b1a5c700 1 --2- 172.21.15.195:0/4118567416 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff3ac06a450 0x7ff3ac150d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:36.402 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.388+0000 7ff3b1a5c700 1 -- 172.21.15.195:0/4118567416 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff3ac150500 msgr2=0x7ff3ac16a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:36.403 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.388+0000 7ff3b1a5c700 1 --2- 172.21.15.195:0/4118567416 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff3ac150500 0x7ff3ac16a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:36.403 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.388+0000 7ff3b1a5c700 1 -- 172.21.15.195:0/4118567416 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3a8009350 con 0x7ff3ac1508d0 2022-06-27T12:09:36.403 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.388+0000 7ff3b1a5c700 1 --2- 172.21.15.195:0/4118567416 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff3ac1508d0 0x7ff3ac166ef0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7ff3a4009df0 tx=0x7ff3a400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:36.403 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.388+0000 7ff3997fa700 1 -- 172.21.15.195:0/4118567416 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a400cb30 con 0x7ff3ac1508d0 2022-06-27T12:09:36.403 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.388+0000 7ff3997fa700 1 -- 172.21.15.195:0/4118567416 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3a400f040 con 0x7ff3ac1508d0 2022-06-27T12:09:36.404 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.388+0000 7ff3b34bf700 1 -- 172.21.15.195:0/4118567416 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3ac1512d0 con 0x7ff3ac1508d0 2022-06-27T12:09:36.404 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.388+0000 7ff3997fa700 1 -- 172.21.15.195:0/4118567416 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a4014480 con 0x7ff3ac1508d0 2022-06-27T12:09:36.404 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.388+0000 7ff3b34bf700 1 -- 172.21.15.195:0/4118567416 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3ac152f90 con 0x7ff3ac1508d0 2022-06-27T12:09:36.404 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.389+0000 7ff3b34bf700 1 -- 172.21.15.195:0/4118567416 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3ac0c3f80 con 0x7ff3ac1508d0 2022-06-27T12:09:36.405 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.389+0000 7ff3997fa700 1 -- 172.21.15.195:0/4118567416 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7ff3a4014830 con 0x7ff3ac1508d0 2022-06-27T12:09:36.405 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.389+0000 7ff3997fa700 1 --2- 172.21.15.195:0/4118567416 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff38c063d00 0x7ff38c0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:36.405 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.389+0000 7ff3997fa700 1 -- 172.21.15.195:0/4118567416 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7234+0+0 (secure 0 0 0) 0x7ff3a4081860 con 0x7ff3ac1508d0 2022-06-27T12:09:36.405 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.389+0000 7ff3b125b700 1 --2- 172.21.15.195:0/4118567416 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff38c063d00 0x7ff38c0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:36.405 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.391+0000 7ff3b125b700 1 --2- 172.21.15.195:0/4118567416 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff38c063d00 0x7ff38c0661b0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7ff3a8009130 tx=0x7ff3a8015000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:36.414 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.393+0000 7ff3997fa700 1 -- 172.21.15.195:0/4118567416 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7ff3a404ffb0 con 0x7ff3ac1508d0 2022-06-27T12:09:36.623 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.622+0000 7ff3b34bf700 1 -- 172.21.15.195:0/4118567416 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff3ac0722c0 con 0x7ff3ac1508d0 2022-06-27T12:09:36.623 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.623+0000 7ff3997fa700 1 -- 172.21.15.195:0/4118567416 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+5364 (secure 0 0 0) 0x7ff3a40535d0 con 0x7ff3ac1508d0 2022-06-27T12:09:36.624 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:09:36.625 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:09:25.119670+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6708},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6708":{"gid":6708,"name":"b","rank":0,"incarnation":79,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6833/702244802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":702244802},{"type":"v1","addr":"172.21.15.202:6833","nonce":702244802}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:09:36.628 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.625+0000 7ff3b34bf700 1 -- 172.21.15.195:0/4118567416 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff38c063d00 msgr2=0x7ff38c0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:36.628 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.625+0000 7ff3b34bf700 1 --2- 172.21.15.195:0/4118567416 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff38c063d00 0x7ff38c0661b0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7ff3a8009130 tx=0x7ff3a8015000 comp rx=0 tx=0).stop 2022-06-27T12:09:36.628 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.625+0000 7ff3b34bf700 1 -- 172.21.15.195:0/4118567416 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff3ac1508d0 msgr2=0x7ff3ac166ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:36.629 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.625+0000 7ff3b34bf700 1 --2- 172.21.15.195:0/4118567416 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff3ac1508d0 0x7ff3ac166ef0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7ff3a4009df0 tx=0x7ff3a400b040 comp rx=0 tx=0).stop 2022-06-27T12:09:36.629 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.625+0000 7ff3b34bf700 1 -- 172.21.15.195:0/4118567416 shutdown_connections 2022-06-27T12:09:36.630 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.625+0000 7ff3b34bf700 1 --2- 172.21.15.195:0/4118567416 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff38c063d00 0x7ff38c0661b0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:36.630 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.625+0000 7ff3b34bf700 1 --2- 172.21.15.195:0/4118567416 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff3ac1508d0 0x7ff3ac166ef0 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:36.630 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.625+0000 7ff3b34bf700 1 --2- 172.21.15.195:0/4118567416 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff3ac06a450 0x7ff3ac150d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:36.631 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.625+0000 7ff3b34bf700 1 --2- 172.21.15.195:0/4118567416 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff3ac150500 0x7ff3ac16a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:36.631 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.625+0000 7ff3b34bf700 1 -- 172.21.15.195:0/4118567416 >> 172.21.15.195:0/4118567416 conn(0x7ff3ac0c9b00 msgr2=0x7ff3ac067ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:09:36.631 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.625+0000 7ff3b34bf700 1 -- 172.21.15.195:0/4118567416 shutdown_connections 2022-06-27T12:09:36.632 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:36.625+0000 7ff3b34bf700 1 -- 172.21.15.195:0/4118567416 wait complete. 2022-06-27T12:09:36.632 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 79 2022-06-27T12:09:36.637 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:09:38.638 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:09:38.782 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:38.771+0000 7f296d282700 1 Processor -- start 2022-06-27T12:09:38.783 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:38.771+0000 7f296d282700 1 -- start start 2022-06-27T12:09:38.783 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:38.772+0000 7f296d282700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f2968150570 0x7f29680c4160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:38.784 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:38.772+0000 7f296d282700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2968156ff0 0x7f296806e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:38.784 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:38.772+0000 7f296d282700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29680c1d40 con 0x7f2968150940 2022-06-27T12:09:38.790 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:38.772+0000 7f296d282700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29680c30e0 con 0x7f2968150570 2022-06-27T12:09:38.790 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:38.772+0000 7f296d282700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f296806fc80 con 0x7f2968156ff0 2022-06-27T12:09:38.791 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:38.772+0000 7f2966ffd700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f2968150570 0x7f29680c4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:38.791 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:38.772+0000 7f29677fe700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2968156ff0 0x7f296806e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:38.792 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:38.772+0000 7f29667fc700 1 --1- >> v1:172.21.15.202:6789/0 conn(0x7f2968150940 0x7f296806f3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.202:6789/0 says I am v1:172.21.15.195:57636/0 (socket says 172.21.15.195:57636) 2022-06-27T12:09:38.792 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:38.772+0000 7f29667fc700 1 -- 172.21.15.195:0/2506746509 learned_addr learned my addr 172.21.15.195:0/2506746509 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:09:38.793 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:38.773+0000 7f2966ffd700 1 -- 172.21.15.195:0/2506746509 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2968156ff0 msgr2=0x7f296806e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:09:38.793 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:38.773+0000 7f2966ffd700 1 --2- 172.21.15.195:0/2506746509 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2968156ff0 0x7f296806e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:38.807 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:38.773+0000 7f2966ffd700 1 -- 172.21.15.195:0/2506746509 >> v1:172.21.15.202:6789/0 conn(0x7f2968150940 legacy=0x7f296806f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:38.808 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:38.773+0000 7f2966ffd700 1 -- 172.21.15.195:0/2506746509 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f296806ff40 con 0x7f2968150570 2022-06-27T12:09:38.808 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:38.773+0000 7f2965ffb700 1 -- 172.21.15.195:0/2506746509 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 991928007 0 0) 0x7f29680c1d40 con 0x7f2968150940 2022-06-27T12:09:38.809 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:38.773+0000 7f29677fe700 1 --2- 172.21.15.195:0/2506746509 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2968156ff0 0x7f296806e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:09:38.809 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:38.773+0000 7f2966ffd700 1 --2- 172.21.15.195:0/2506746509 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2968150570 0x7f29680c4160 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f295c0097a0 tx=0x7f295c009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=13ffff87c7c2cb7b server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:38.810 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:38.773+0000 7f2965ffb700 1 -- 172.21.15.195:0/2506746509 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f295c011630 con 0x7f2968150570 2022-06-27T12:09:38.811 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:38.773+0000 7f2965ffb700 1 -- 172.21.15.195:0/2506746509 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f295c011790 con 0x7f2968150570 2022-06-27T12:09:38.811 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:38.773+0000 7f2965ffb700 1 -- 172.21.15.195:0/2506746509 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f295c011b30 con 0x7f2968150570 2022-06-27T12:09:38.812 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:38.774+0000 7f296d282700 1 -- 172.21.15.195:0/2506746509 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2968150570 msgr2=0x7f29680c4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:38.812 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:38.774+0000 7f296d282700 1 --2- 172.21.15.195:0/2506746509 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2968150570 0x7f29680c4160 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f295c0097a0 tx=0x7f295c009060 comp rx=0 tx=0).stop 2022-06-27T12:09:38.813 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:38.774+0000 7f296d282700 1 -- 172.21.15.195:0/2506746509 shutdown_connections 2022-06-27T12:09:38.814 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:38.774+0000 7f296d282700 1 --2- 172.21.15.195:0/2506746509 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2968156ff0 0x7f296806e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:38.814 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:38.774+0000 7f296d282700 1 --2- 172.21.15.195:0/2506746509 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2968150570 0x7f29680c4160 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:38.814 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:38.774+0000 7f296d282700 1 -- 172.21.15.195:0/2506746509 >> 172.21.15.195:0/2506746509 conn(0x7f29680c6350 msgr2=0x7f29680c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:09:38.815 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:38.774+0000 7f296d282700 1 -- 172.21.15.195:0/2506746509 shutdown_connections 2022-06-27T12:09:38.815 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:38.774+0000 7f296d282700 1 -- 172.21.15.195:0/2506746509 wait complete. 2022-06-27T12:09:38.815 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:38.775+0000 7f296d282700 1 Processor -- start 2022-06-27T12:09:38.815 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:38.775+0000 7f296d282700 1 -- start start 2022-06-27T12:09:38.828 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:38.776+0000 7f296d282700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2968150570 0x7f29680ba780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:38.829 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:38.776+0000 7f296d282700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2968150940 0x7f296816a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:38.830 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:38.776+0000 7f296d282700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2968156ff0 0x7f29681544e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:38.832 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:38.776+0000 7f296d282700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29681585c0 con 0x7f2968150570 2022-06-27T12:09:38.834 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:38.776+0000 7f296d282700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f296816a090 con 0x7f2968156ff0 2022-06-27T12:09:38.835 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:38.776+0000 7f296d282700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29680bcd70 con 0x7f2968150940 2022-06-27T12:09:38.836 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:38.776+0000 7f2966ffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2968150570 0x7f29680ba780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:38.836 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:38.776+0000 7f2966ffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2968150570 0x7f29680ba780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:49592/0 (socket says 172.21.15.195:49592) 2022-06-27T12:09:38.837 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:38.776+0000 7f2966ffd700 1 -- 172.21.15.195:0/2754279238 learned_addr learned my addr 172.21.15.195:0/2754279238 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:09:38.837 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:38.776+0000 7f29667fc700 1 --2- 172.21.15.195:0/2754279238 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2968150940 0x7f296816a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:38.838 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:38.776+0000 7f29677fe700 1 --2- 172.21.15.195:0/2754279238 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2968156ff0 0x7f29681544e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:38.839 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:38.776+0000 7f2966ffd700 1 -- 172.21.15.195:0/2754279238 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2968150940 msgr2=0x7f296816a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:38.840 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:38.776+0000 7f2966ffd700 1 --2- 172.21.15.195:0/2754279238 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2968150940 0x7f296816a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:38.840 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:38.776+0000 7f2966ffd700 1 -- 172.21.15.195:0/2754279238 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2968156ff0 msgr2=0x7f29681544e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:38.841 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:38.776+0000 7f2966ffd700 1 --2- 172.21.15.195:0/2754279238 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2968156ff0 0x7f29681544e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:38.857 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:38.776+0000 7f2966ffd700 1 -- 172.21.15.195:0/2754279238 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2960009a20 con 0x7f2968150570 2022-06-27T12:09:38.858 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:38.776+0000 7f29677fe700 1 --2- 172.21.15.195:0/2754279238 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2968156ff0 0x7f29681544e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:09:38.858 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:38.776+0000 7f2966ffd700 1 --2- 172.21.15.195:0/2754279238 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2968150570 0x7f29680ba780 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f295c009e40 tx=0x7f295c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:38.858 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:38.776+0000 7f294effd700 1 -- 172.21.15.195:0/2754279238 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f295c000d10 con 0x7f2968150570 2022-06-27T12:09:38.859 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:38.777+0000 7f294effd700 1 -- 172.21.15.195:0/2754279238 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f295c011900 con 0x7f2968150570 2022-06-27T12:09:38.873 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:38.777+0000 7f296d282700 1 -- 172.21.15.195:0/2754279238 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f295c0092e0 con 0x7f2968150570 2022-06-27T12:09:38.888 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:38.777+0000 7f294effd700 1 -- 172.21.15.195:0/2754279238 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f295c025420 con 0x7f2968150570 2022-06-27T12:09:38.888 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:38.777+0000 7f296d282700 1 -- 172.21.15.195:0/2754279238 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29680c0a60 con 0x7f2968150570 2022-06-27T12:09:38.889 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:38.777+0000 7f296d282700 1 -- 172.21.15.195:0/2754279238 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f296816a1d0 con 0x7f2968150570 2022-06-27T12:09:38.897 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:38.778+0000 7f294effd700 1 -- 172.21.15.195:0/2754279238 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f295c02c030 con 0x7f2968150570 2022-06-27T12:09:38.897 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:38.778+0000 7f294effd700 1 --2- 172.21.15.195:0/2754279238 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2948063d00 0x7f29480661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:38.898 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:38.778+0000 7f294effd700 1 -- 172.21.15.195:0/2754279238 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7234+0+0 (secure 0 0 0) 0x7f295c082340 con 0x7f2968150570 2022-06-27T12:09:38.898 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:38.778+0000 7f29667fc700 1 --2- 172.21.15.195:0/2754279238 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2948063d00 0x7f29480661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:38.899 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:38.779+0000 7f29667fc700 1 --2- 172.21.15.195:0/2754279238 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2948063d00 0x7f29480661b0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f29681561a0 tx=0x7f2958007b00 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:38.900 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:38.783+0000 7f294effd700 1 -- 172.21.15.195:0/2754279238 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f295c050a90 con 0x7f2968150570 2022-06-27T12:09:39.010 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:39.009+0000 7f296d282700 1 -- 172.21.15.195:0/2754279238 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f29680722c0 con 0x7f2968150570 2022-06-27T12:09:39.010 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:39.009+0000 7f294effd700 1 -- 172.21.15.195:0/2754279238 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+5364 (secure 0 0 0) 0x7f295c02a090 con 0x7f2968150570 2022-06-27T12:09:39.011 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:09:39.012 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:09:25.119670+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6708},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6708":{"gid":6708,"name":"b","rank":0,"incarnation":79,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6833/702244802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":702244802},{"type":"v1","addr":"172.21.15.202:6833","nonce":702244802}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:09:39.017 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:39.011+0000 7f296d282700 1 -- 172.21.15.195:0/2754279238 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2948063d00 msgr2=0x7f29480661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:39.018 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:39.011+0000 7f296d282700 1 --2- 172.21.15.195:0/2754279238 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2948063d00 0x7f29480661b0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f29681561a0 tx=0x7f2958007b00 comp rx=0 tx=0).stop 2022-06-27T12:09:39.018 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:39.011+0000 7f296d282700 1 -- 172.21.15.195:0/2754279238 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2968150570 msgr2=0x7f29680ba780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:39.019 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:39.011+0000 7f296d282700 1 --2- 172.21.15.195:0/2754279238 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2968150570 0x7f29680ba780 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f295c009e40 tx=0x7f295c017000 comp rx=0 tx=0).stop 2022-06-27T12:09:39.019 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:39.011+0000 7f296d282700 1 -- 172.21.15.195:0/2754279238 shutdown_connections 2022-06-27T12:09:39.020 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:39.011+0000 7f296d282700 1 --2- 172.21.15.195:0/2754279238 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2948063d00 0x7f29480661b0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:39.020 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:39.011+0000 7f296d282700 1 --2- 172.21.15.195:0/2754279238 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2968150570 0x7f29680ba780 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:39.021 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:39.011+0000 7f296d282700 1 --2- 172.21.15.195:0/2754279238 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2968156ff0 0x7f29681544e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:39.021 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:39.011+0000 7f296d282700 1 --2- 172.21.15.195:0/2754279238 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2968150940 0x7f296816a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:39.022 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:39.011+0000 7f296d282700 1 -- 172.21.15.195:0/2754279238 >> 172.21.15.195:0/2754279238 conn(0x7f29680c6350 msgr2=0x7f29680c9090 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:09:39.022 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:39.011+0000 7f296d282700 1 -- 172.21.15.195:0/2754279238 shutdown_connections 2022-06-27T12:09:39.023 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:39.011+0000 7f296d282700 1 -- 172.21.15.195:0/2754279238 wait complete. 2022-06-27T12:09:39.023 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 79 2022-06-27T12:09:39.046 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 79, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-06-27T11:47:52.358095+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 79, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6708": { "addr": "172.21.15.202:6833/702244802", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6832", "nonce": 702244802, "type": "v2" }, { "addr": "172.21.15.202:6833", "nonce": 702244802, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 6708, "incarnation": 79, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-06-27T12:09:25.119670+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6708 } } } ], "standbys": [ { "addr": "172.21.15.202:6837/3305006841", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6836", "nonce": 3305006841, "type": "v2" }, { "addr": "172.21.15.202:6837", "nonce": 3305006841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6835/2060709822", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6834", "nonce": 2060709822, "type": "v2" }, { "addr": "172.21.15.195:6835", "nonce": 2060709822, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6838/1598751652", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6836", "nonce": 1598751652, "type": "v2" }, { "addr": "172.21.15.195:6838", "nonce": 1598751652, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4275, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6839/1313800661", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6837", "nonce": 1313800661, "type": "v2" }, { "addr": "172.21.15.195:6839", "nonce": 1313800661, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4278, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-06-27T12:09:41.058 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:09:41.212 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.193+0000 7f3557618700 1 Processor -- start 2022-06-27T12:09:41.213 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.193+0000 7f3557618700 1 -- start start 2022-06-27T12:09:41.213 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.194+0000 7f3557618700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f3550150500 0x7f3550063af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:41.213 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.194+0000 7f3557618700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f355006a450 0x7f35500c48d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:41.213 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.194+0000 7f3557618700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f35501508d0 0x7f35500bbf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:41.214 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.194+0000 7f3557618700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3550064030 con 0x7f3550150500 2022-06-27T12:09:41.214 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.194+0000 7f3557618700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35500c4160 con 0x7f355006a450 2022-06-27T12:09:41.214 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.194+0000 7f3557618700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35500c4440 con 0x7f35501508d0 2022-06-27T12:09:41.214 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.194+0000 7f35553b4700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f3550150500 0x7f3550063af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:41.215 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.194+0000 7f35553b4700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f3550150500 0x7f3550063af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:49608/0 (socket says 172.21.15.195:49608) 2022-06-27T12:09:41.228 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.194+0000 7f35553b4700 1 -- 172.21.15.195:0/1919650652 learned_addr learned my addr 172.21.15.195:0/1919650652 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:09:41.228 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.194+0000 7f3554bb3700 1 --2- 172.21.15.195:0/1919650652 >> v2:172.21.15.202:3300/0 conn(0x7f355006a450 0x7f35500c48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:41.229 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.194+0000 7f3555bb5700 1 --2- 172.21.15.195:0/1919650652 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f35501508d0 0x7f35500bbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:41.230 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.194+0000 7f35553b4700 1 -- 172.21.15.195:0/1919650652 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f35501508d0 msgr2=0x7f35500bbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:09:41.230 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.194+0000 7f35553b4700 1 --2- 172.21.15.195:0/1919650652 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f35501508d0 0x7f35500bbf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:41.230 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.194+0000 7f35553b4700 1 -- 172.21.15.195:0/1919650652 >> v2:172.21.15.202:3300/0 conn(0x7f355006a450 msgr2=0x7f35500c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:09:41.231 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.194+0000 7f35553b4700 1 --2- 172.21.15.195:0/1919650652 >> v2:172.21.15.202:3300/0 conn(0x7f355006a450 0x7f35500c48d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:41.231 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.194+0000 7f35553b4700 1 -- 172.21.15.195:0/1919650652 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f355015a800 con 0x7f3550150500 2022-06-27T12:09:41.232 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.194+0000 7f35553b4700 1 --2- 172.21.15.195:0/1919650652 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3550150500 0x7f3550063af0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f355006ce70 tx=0x7f354c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b3314db607337a server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:41.233 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.195+0000 7f353ffff700 1 -- 172.21.15.195:0/1919650652 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f354c005ad0 con 0x7f3550150500 2022-06-27T12:09:41.233 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.195+0000 7f353ffff700 1 -- 172.21.15.195:0/1919650652 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f354c005c30 con 0x7f3550150500 2022-06-27T12:09:41.234 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.195+0000 7f353ffff700 1 -- 172.21.15.195:0/1919650652 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f354c004030 con 0x7f3550150500 2022-06-27T12:09:41.234 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.195+0000 7f3557618700 1 -- 172.21.15.195:0/1919650652 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3550150500 msgr2=0x7f3550063af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:41.236 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.195+0000 7f3557618700 1 --2- 172.21.15.195:0/1919650652 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3550150500 0x7f3550063af0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f355006ce70 tx=0x7f354c0090f0 comp rx=0 tx=0).stop 2022-06-27T12:09:41.236 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.195+0000 7f3557618700 1 -- 172.21.15.195:0/1919650652 shutdown_connections 2022-06-27T12:09:41.237 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.195+0000 7f3557618700 1 --2- 172.21.15.195:0/1919650652 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f35501508d0 0x7f35500bbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:41.237 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.195+0000 7f3557618700 1 --2- 172.21.15.195:0/1919650652 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3550150500 0x7f3550063af0 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:41.237 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.195+0000 7f3557618700 1 --2- 172.21.15.195:0/1919650652 >> v2:172.21.15.202:3300/0 conn(0x7f355006a450 0x7f35500c48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:41.246 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.195+0000 7f3557618700 1 -- 172.21.15.195:0/1919650652 >> 172.21.15.195:0/1919650652 conn(0x7f35500c9b00 msgr2=0x7f35500c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:09:41.246 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.195+0000 7f3557618700 1 -- 172.21.15.195:0/1919650652 shutdown_connections 2022-06-27T12:09:41.247 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.195+0000 7f3557618700 1 -- 172.21.15.195:0/1919650652 wait complete. 2022-06-27T12:09:41.247 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.196+0000 7f3557618700 1 Processor -- start 2022-06-27T12:09:41.248 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.196+0000 7f3557618700 1 -- start start 2022-06-27T12:09:41.248 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.197+0000 7f3557618700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f355006a450 0x7f35500c7020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:41.248 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.197+0000 7f3557618700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3550150500 0x7f355014d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:41.249 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.197+0000 7f3557618700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f35501508d0 0x7f355014deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:41.249 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.197+0000 7f3557618700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3550166eb0 con 0x7f35501508d0 2022-06-27T12:09:41.250 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.197+0000 7f3557618700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3550159b60 con 0x7f3550150500 2022-06-27T12:09:41.250 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.197+0000 7f3557618700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35501662e0 con 0x7f355006a450 2022-06-27T12:09:41.250 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.197+0000 7f3555bb5700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f35501508d0 0x7f355014deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:41.251 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.198+0000 7f3555bb5700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f35501508d0 0x7f355014deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:49616/0 (socket says 172.21.15.195:49616) 2022-06-27T12:09:41.251 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.198+0000 7f3555bb5700 1 -- 172.21.15.195:0/1777882599 learned_addr learned my addr 172.21.15.195:0/1777882599 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:09:41.252 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.198+0000 7f35553b4700 1 --2- 172.21.15.195:0/1777882599 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f355006a450 0x7f35500c7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:41.253 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.198+0000 7f3554bb3700 1 --2- 172.21.15.195:0/1777882599 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3550150500 0x7f355014d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:41.253 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.198+0000 7f3555bb5700 1 -- 172.21.15.195:0/1777882599 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f355006a450 msgr2=0x7f35500c7020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:41.254 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.198+0000 7f3555bb5700 1 --2- 172.21.15.195:0/1777882599 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f355006a450 0x7f35500c7020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:41.254 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.198+0000 7f3555bb5700 1 -- 172.21.15.195:0/1777882599 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3550150500 msgr2=0x7f355014d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:41.254 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.198+0000 7f3555bb5700 1 --2- 172.21.15.195:0/1777882599 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3550150500 0x7f355014d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:41.255 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.198+0000 7f3555bb5700 1 -- 172.21.15.195:0/1777882599 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f354c009350 con 0x7f35501508d0 2022-06-27T12:09:41.255 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.198+0000 7f3555bb5700 1 --2- 172.21.15.195:0/1777882599 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f35501508d0 0x7f355014deb0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f3548009df0 tx=0x7f354800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:41.256 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.198+0000 7f353d7fa700 1 -- 172.21.15.195:0/1777882599 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f354800cb30 con 0x7f35501508d0 2022-06-27T12:09:41.256 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.198+0000 7f353d7fa700 1 -- 172.21.15.195:0/1777882599 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f354800f040 con 0x7f35501508d0 2022-06-27T12:09:41.257 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.198+0000 7f3557618700 1 -- 172.21.15.195:0/1777882599 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3550166580 con 0x7f35501508d0 2022-06-27T12:09:41.257 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.198+0000 7f353d7fa700 1 -- 172.21.15.195:0/1777882599 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f354800cee0 con 0x7f35501508d0 2022-06-27T12:09:41.258 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.198+0000 7f3557618700 1 -- 172.21.15.195:0/1777882599 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3550166b40 con 0x7f35501508d0 2022-06-27T12:09:41.258 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.199+0000 7f3557618700 1 -- 172.21.15.195:0/1777882599 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3550167020 con 0x7f35501508d0 2022-06-27T12:09:41.258 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.200+0000 7f353d7fa700 1 -- 172.21.15.195:0/1777882599 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f3548014690 con 0x7f35501508d0 2022-06-27T12:09:41.259 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.200+0000 7f353d7fa700 1 --2- 172.21.15.195:0/1777882599 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3530063d00 0x7f35300661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:41.259 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.200+0000 7f353d7fa700 1 -- 172.21.15.195:0/1777882599 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7234+0+0 (secure 0 0 0) 0x7f3548082130 con 0x7f35501508d0 2022-06-27T12:09:41.259 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.200+0000 7f35553b4700 1 --2- 172.21.15.195:0/1777882599 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3530063d00 0x7f35300661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:41.260 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.201+0000 7f35553b4700 1 --2- 172.21.15.195:0/1777882599 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3530063d00 0x7f35300661b0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f354c009130 tx=0x7f354c015000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:41.260 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.204+0000 7f353d7fa700 1 -- 172.21.15.195:0/1777882599 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f3548050900 con 0x7f35501508d0 2022-06-27T12:09:41.436 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.432+0000 7f3557618700 1 -- 172.21.15.195:0/1777882599 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f35500722c0 con 0x7f35501508d0 2022-06-27T12:09:41.436 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.433+0000 7f353d7fa700 1 -- 172.21.15.195:0/1777882599 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+5368 (secure 0 0 0) 0x7f3548053f20 con 0x7f35501508d0 2022-06-27T12:09:41.437 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:09:41.439 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":80,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:09:39.287072+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6708},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6708":{"gid":6708,"name":"b","rank":0,"incarnation":79,"state":"up:reconnect","state_seq":20,"addr":"172.21.15.202:6833/702244802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":702244802},{"type":"v1","addr":"172.21.15.202:6833","nonce":702244802}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:09:41.442 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.435+0000 7f3557618700 1 -- 172.21.15.195:0/1777882599 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3530063d00 msgr2=0x7f35300661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:41.443 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.435+0000 7f3557618700 1 --2- 172.21.15.195:0/1777882599 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3530063d00 0x7f35300661b0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f354c009130 tx=0x7f354c015000 comp rx=0 tx=0).stop 2022-06-27T12:09:41.443 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.435+0000 7f3557618700 1 -- 172.21.15.195:0/1777882599 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f35501508d0 msgr2=0x7f355014deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:41.444 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.435+0000 7f3557618700 1 --2- 172.21.15.195:0/1777882599 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f35501508d0 0x7f355014deb0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f3548009df0 tx=0x7f354800b040 comp rx=0 tx=0).stop 2022-06-27T12:09:41.445 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.435+0000 7f3557618700 1 -- 172.21.15.195:0/1777882599 shutdown_connections 2022-06-27T12:09:41.446 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.435+0000 7f3557618700 1 --2- 172.21.15.195:0/1777882599 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3530063d00 0x7f35300661b0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:41.447 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.435+0000 7f3557618700 1 --2- 172.21.15.195:0/1777882599 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f35501508d0 0x7f355014deb0 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:41.447 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.435+0000 7f3557618700 1 --2- 172.21.15.195:0/1777882599 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f355006a450 0x7f35500c7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:41.448 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.435+0000 7f3557618700 1 --2- 172.21.15.195:0/1777882599 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3550150500 0x7f355014d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:41.449 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.435+0000 7f3557618700 1 -- 172.21.15.195:0/1777882599 >> 172.21.15.195:0/1777882599 conn(0x7f35500c9b00 msgr2=0x7f355014faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:09:41.449 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.435+0000 7f3557618700 1 -- 172.21.15.195:0/1777882599 shutdown_connections 2022-06-27T12:09:41.450 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:41.435+0000 7f3557618700 1 -- 172.21.15.195:0/1777882599 wait complete. 2022-06-27T12:09:41.450 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 80 2022-06-27T12:09:41.452 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:09:43.456 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:09:43.608 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.590+0000 7fed42f04700 1 Processor -- start 2022-06-27T12:09:43.609 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.590+0000 7fed42f04700 1 -- start start 2022-06-27T12:09:43.609 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.591+0000 7fed42f04700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fed3c06a450 0x7fed3c063af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:43.609 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.591+0000 7fed42f04700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fed3c1508d0 0x7fed3c0bbf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:43.609 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.591+0000 7fed42f04700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed3c064030 con 0x7fed3c150500 2022-06-27T12:09:43.610 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.591+0000 7fed42f04700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed3c0c4160 con 0x7fed3c06a450 2022-06-27T12:09:43.610 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.591+0000 7fed42f04700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed3c0c4440 con 0x7fed3c1508d0 2022-06-27T12:09:43.610 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.591+0000 7fed414a1700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fed3c1508d0 0x7fed3c0bbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:43.610 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.591+0000 7fed414a1700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fed3c1508d0 0x7fed3c0bbf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3301/0 says I am v2:172.21.15.195:42378/0 (socket says 172.21.15.195:42378) 2022-06-27T12:09:43.611 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.591+0000 7fed3bfff700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fed3c06a450 0x7fed3c063af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:43.611 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.591+0000 7fed414a1700 1 -- 172.21.15.195:0/932485071 learned_addr learned my addr 172.21.15.195:0/932485071 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:09:43.611 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.591+0000 7fed40ca0700 1 --1- 172.21.15.195:0/932485071 >> v1:172.21.15.202:6789/0 conn(0x7fed3c150500 0x7fed3c0c48d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.202:6789/0 says I am v1:172.21.15.195:57672/0 (socket says 172.21.15.195:57672) 2022-06-27T12:09:43.611 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.591+0000 7fed414a1700 1 -- 172.21.15.195:0/932485071 >> v2:172.21.15.202:3300/0 conn(0x7fed3c06a450 msgr2=0x7fed3c063af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:09:43.612 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.591+0000 7fed414a1700 1 --2- 172.21.15.195:0/932485071 >> v2:172.21.15.202:3300/0 conn(0x7fed3c06a450 0x7fed3c063af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:43.612 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.591+0000 7fed414a1700 1 -- 172.21.15.195:0/932485071 >> v1:172.21.15.202:6789/0 conn(0x7fed3c150500 legacy=0x7fed3c0c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:43.612 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.591+0000 7fed414a1700 1 -- 172.21.15.195:0/932485071 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed3c15a800 con 0x7fed3c1508d0 2022-06-27T12:09:43.612 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.591+0000 7fed3bfff700 1 --2- 172.21.15.195:0/932485071 >> v2:172.21.15.202:3300/0 conn(0x7fed3c06a450 0x7fed3c063af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:09:43.613 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.592+0000 7fed414a1700 1 --2- 172.21.15.195:0/932485071 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fed3c1508d0 0x7fed3c0bbf50 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fed34009f20 tx=0x7fed34009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95837110fadcc6d2 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:43.613 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.592+0000 7fed3b7fe700 1 -- 172.21.15.195:0/932485071 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed3400e070 con 0x7fed3c1508d0 2022-06-27T12:09:43.613 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.592+0000 7fed3b7fe700 1 -- 172.21.15.195:0/932485071 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed3400b9d0 con 0x7fed3c1508d0 2022-06-27T12:09:43.613 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.592+0000 7fed3b7fe700 1 -- 172.21.15.195:0/932485071 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed340092e0 con 0x7fed3c1508d0 2022-06-27T12:09:43.613 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.592+0000 7fed42f04700 1 -- 172.21.15.195:0/932485071 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fed3c1508d0 msgr2=0x7fed3c0bbf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:43.614 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.592+0000 7fed42f04700 1 --2- 172.21.15.195:0/932485071 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fed3c1508d0 0x7fed3c0bbf50 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fed34009f20 tx=0x7fed34009840 comp rx=0 tx=0).stop 2022-06-27T12:09:43.614 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.592+0000 7fed42f04700 1 -- 172.21.15.195:0/932485071 shutdown_connections 2022-06-27T12:09:43.614 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.592+0000 7fed42f04700 1 --2- 172.21.15.195:0/932485071 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fed3c1508d0 0x7fed3c0bbf50 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:43.614 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.592+0000 7fed42f04700 1 --2- 172.21.15.195:0/932485071 >> v2:172.21.15.202:3300/0 conn(0x7fed3c06a450 0x7fed3c063af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:43.615 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.592+0000 7fed42f04700 1 -- 172.21.15.195:0/932485071 >> 172.21.15.195:0/932485071 conn(0x7fed3c0c9b00 msgr2=0x7fed3c0c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:09:43.616 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.592+0000 7fed42f04700 1 -- 172.21.15.195:0/932485071 shutdown_connections 2022-06-27T12:09:43.616 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.592+0000 7fed42f04700 1 -- 172.21.15.195:0/932485071 wait complete. 2022-06-27T12:09:43.616 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.594+0000 7fed42f04700 1 Processor -- start 2022-06-27T12:09:43.616 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.594+0000 7fed42f04700 1 -- start start 2022-06-27T12:09:43.616 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.594+0000 7fed42f04700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fed3c06a450 0x7fed3c166500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:43.617 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.594+0000 7fed42f04700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fed3c150500 0x7fed3c06e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:43.617 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.594+0000 7fed42f04700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fed3c1508d0 0x7fed3c1591b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:43.617 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.594+0000 7fed42f04700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed3c155190 con 0x7fed3c06a450 2022-06-27T12:09:43.617 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.594+0000 7fed42f04700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed3c158d20 con 0x7fed3c150500 2022-06-27T12:09:43.618 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.594+0000 7fed42f04700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed3c1673c0 con 0x7fed3c1508d0 2022-06-27T12:09:43.618 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.595+0000 7fed40ca0700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fed3c06a450 0x7fed3c166500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:43.618 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.595+0000 7fed40ca0700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fed3c06a450 0x7fed3c166500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:49628/0 (socket says 172.21.15.195:49628) 2022-06-27T12:09:43.618 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.595+0000 7fed40ca0700 1 -- 172.21.15.195:0/2011202035 learned_addr learned my addr 172.21.15.195:0/2011202035 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:09:43.618 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.595+0000 7fed414a1700 1 --2- 172.21.15.195:0/2011202035 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fed3c1508d0 0x7fed3c1591b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:43.619 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.595+0000 7fed3bfff700 1 --2- 172.21.15.195:0/2011202035 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fed3c150500 0x7fed3c06e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:43.619 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.595+0000 7fed40ca0700 1 -- 172.21.15.195:0/2011202035 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fed3c1508d0 msgr2=0x7fed3c1591b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:43.619 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.595+0000 7fed40ca0700 1 --2- 172.21.15.195:0/2011202035 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fed3c1508d0 0x7fed3c1591b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:43.619 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.595+0000 7fed40ca0700 1 -- 172.21.15.195:0/2011202035 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fed3c150500 msgr2=0x7fed3c06e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:43.620 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.595+0000 7fed40ca0700 1 --2- 172.21.15.195:0/2011202035 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fed3c150500 0x7fed3c06e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:43.620 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.595+0000 7fed40ca0700 1 -- 172.21.15.195:0/2011202035 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed280092e0 con 0x7fed3c06a450 2022-06-27T12:09:43.620 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.595+0000 7fed40ca0700 1 --2- 172.21.15.195:0/2011202035 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fed3c06a450 0x7fed3c166500 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fed3000c4c0 tx=0x7fed30007e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:43.620 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.595+0000 7fed38ff9700 1 -- 172.21.15.195:0/2011202035 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed3000ce30 con 0x7fed3c06a450 2022-06-27T12:09:43.620 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.595+0000 7fed38ff9700 1 -- 172.21.15.195:0/2011202035 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed300072e0 con 0x7fed3c06a450 2022-06-27T12:09:43.621 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.595+0000 7fed42f04700 1 -- 172.21.15.195:0/2011202035 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed34009ae0 con 0x7fed3c06a450 2022-06-27T12:09:43.621 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.595+0000 7fed38ff9700 1 -- 172.21.15.195:0/2011202035 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed30004740 con 0x7fed3c06a450 2022-06-27T12:09:43.621 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.595+0000 7fed42f04700 1 -- 172.21.15.195:0/2011202035 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed3c166e00 con 0x7fed3c06a450 2022-06-27T12:09:43.621 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.596+0000 7fed42f04700 1 -- 172.21.15.195:0/2011202035 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed3c159000 con 0x7fed3c06a450 2022-06-27T12:09:43.622 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.597+0000 7fed38ff9700 1 -- 172.21.15.195:0/2011202035 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fed30010070 con 0x7fed3c06a450 2022-06-27T12:09:43.622 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.597+0000 7fed38ff9700 1 --2- 172.21.15.195:0/2011202035 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fed24063d30 0x7fed240661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:43.623 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.597+0000 7fed38ff9700 1 -- 172.21.15.195:0/2011202035 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7234+0+0 (secure 0 0 0) 0x7fed3007fb00 con 0x7fed3c06a450 2022-06-27T12:09:43.623 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.597+0000 7fed3bfff700 1 --2- 172.21.15.195:0/2011202035 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fed24063d30 0x7fed240661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:43.623 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.598+0000 7fed3bfff700 1 --2- 172.21.15.195:0/2011202035 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fed24063d30 0x7fed240661e0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fed3c06d770 tx=0x7fed28009060 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:43.623 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.601+0000 7fed38ff9700 1 -- 172.21.15.195:0/2011202035 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fed30049c90 con 0x7fed3c06a450 2022-06-27T12:09:43.822 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.821+0000 7fed42f04700 1 -- 172.21.15.195:0/2011202035 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fed3c064030 con 0x7fed3c06a450 2022-06-27T12:09:43.823 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.822+0000 7fed38ff9700 1 -- 172.21.15.195:0/2011202035 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+5368 (secure 0 0 0) 0x7fed3c064030 con 0x7fed3c06a450 2022-06-27T12:09:43.824 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:09:43.824 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":80,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:09:39.287072+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6708},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6708":{"gid":6708,"name":"b","rank":0,"incarnation":79,"state":"up:reconnect","state_seq":20,"addr":"172.21.15.202:6833/702244802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":702244802},{"type":"v1","addr":"172.21.15.202:6833","nonce":702244802}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:09:43.827 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.824+0000 7fed42f04700 1 -- 172.21.15.195:0/2011202035 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fed24063d30 msgr2=0x7fed240661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:43.828 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.824+0000 7fed42f04700 1 --2- 172.21.15.195:0/2011202035 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fed24063d30 0x7fed240661e0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fed3c06d770 tx=0x7fed28009060 comp rx=0 tx=0).stop 2022-06-27T12:09:43.828 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.824+0000 7fed42f04700 1 -- 172.21.15.195:0/2011202035 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fed3c06a450 msgr2=0x7fed3c166500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:43.829 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.824+0000 7fed42f04700 1 --2- 172.21.15.195:0/2011202035 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fed3c06a450 0x7fed3c166500 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fed3000c4c0 tx=0x7fed30007e20 comp rx=0 tx=0).stop 2022-06-27T12:09:43.829 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.824+0000 7fed42f04700 1 -- 172.21.15.195:0/2011202035 shutdown_connections 2022-06-27T12:09:43.830 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.824+0000 7fed42f04700 1 --2- 172.21.15.195:0/2011202035 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fed24063d30 0x7fed240661e0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:43.830 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.824+0000 7fed42f04700 1 --2- 172.21.15.195:0/2011202035 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fed3c06a450 0x7fed3c166500 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:43.830 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.824+0000 7fed42f04700 1 --2- 172.21.15.195:0/2011202035 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fed3c1508d0 0x7fed3c1591b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:43.831 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.824+0000 7fed42f04700 1 --2- 172.21.15.195:0/2011202035 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fed3c150500 0x7fed3c06e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:43.831 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.824+0000 7fed42f04700 1 -- 172.21.15.195:0/2011202035 >> 172.21.15.195:0/2011202035 conn(0x7fed3c0c9b00 msgr2=0x7fed3c14fc30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:09:43.832 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.824+0000 7fed42f04700 1 -- 172.21.15.195:0/2011202035 shutdown_connections 2022-06-27T12:09:43.832 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:43.824+0000 7fed42f04700 1 -- 172.21.15.195:0/2011202035 wait complete. 2022-06-27T12:09:43.833 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 80 2022-06-27T12:09:43.839 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:09:45.840 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:09:46.024 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.977+0000 7f75e948f700 1 Processor -- start 2022-06-27T12:09:46.025 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.977+0000 7f75e948f700 1 -- start start 2022-06-27T12:09:46.025 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.977+0000 7f75e948f700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f75e4068f80 0x7f75e40c4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:46.026 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.977+0000 7f75e948f700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75e41573b0 con 0x7f75e40683c0 2022-06-27T12:09:46.026 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.977+0000 7f75e948f700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75e40c46c0 con 0x7f75e40c7480 2022-06-27T12:09:46.027 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.977+0000 7f75e948f700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75e4068860 con 0x7f75e4068f80 2022-06-27T12:09:46.027 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.978+0000 7f75e37fe700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f75e40683c0 0x7f75e40c6510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:55824/0 (socket says 172.21.15.195:55824) 2022-06-27T12:09:46.028 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.978+0000 7f75e37fe700 1 -- 172.21.15.195:0/780294695 learned_addr learned my addr 172.21.15.195:0/780294695 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:09:46.028 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.978+0000 7f75e2ffd700 1 --2- 172.21.15.195:0/780294695 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f75e4068f80 0x7f75e40c4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:46.029 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.978+0000 7f75e1ffb700 1 -- 172.21.15.195:0/780294695 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1058378638 0 0) 0x7f75e41573b0 con 0x7f75e40683c0 2022-06-27T12:09:46.029 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.978+0000 7f75e1ffb700 1 -- 172.21.15.195:0/780294695 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75d0003430 con 0x7f75e40683c0 2022-06-27T12:09:46.029 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.978+0000 7f75e1ffb700 1 -- 172.21.15.195:0/780294695 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2112176456 0 0) 0x7f75d0003430 con 0x7f75e40683c0 2022-06-27T12:09:46.042 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.978+0000 7f75e1ffb700 1 -- 172.21.15.195:0/780294695 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f75e4068f80 msgr2=0x7f75e40c4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:09:46.043 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.978+0000 7f75e1ffb700 1 --2- 172.21.15.195:0/780294695 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f75e4068f80 0x7f75e40c4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:46.043 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.978+0000 7f75e1ffb700 1 -- 172.21.15.195:0/780294695 >> v1:172.21.15.202:6789/0 conn(0x7f75e40c7480 legacy=0x7f75e4156460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:46.043 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.978+0000 7f75e1ffb700 1 -- 172.21.15.195:0/780294695 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75e4156b70 con 0x7f75e40683c0 2022-06-27T12:09:46.044 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.978+0000 7f75e1ffb700 1 -- 172.21.15.195:0/780294695 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7f75dc0017f0 con 0x7f75e40683c0 2022-06-27T12:09:46.044 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.978+0000 7f75e1ffb700 1 -- 172.21.15.195:0/780294695 >> v1:172.21.15.195:6789/0 conn(0x7f75e40683c0 legacy=0x7f75e40c6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:46.044 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.978+0000 7f75e1ffb700 1 --2- 172.21.15.195:0/780294695 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f75e4068f80 0x7f75d0003950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:46.045 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.978+0000 7f75e1ffb700 1 --2- 172.21.15.195:0/780294695 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f75d0003ed0 0x7f75d00046d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:46.045 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.978+0000 7f75e1ffb700 1 --2- 172.21.15.195:0/780294695 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f75d0004c10 0x7f75d0005080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:46.062 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.978+0000 7f75e1ffb700 1 -- 172.21.15.195:0/780294695 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75d000bab0 con 0x7f75d0004c10 2022-06-27T12:09:46.062 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.978+0000 7f75e1ffb700 1 -- 172.21.15.195:0/780294695 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75d000bbf0 con 0x7f75d0003ed0 2022-06-27T12:09:46.063 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.978+0000 7f75e1ffb700 1 -- 172.21.15.195:0/780294695 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75d000bdb0 con 0x7f75e4068f80 2022-06-27T12:09:46.063 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.978+0000 7f75e37fe700 1 --2- 172.21.15.195:0/780294695 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f75d0004c10 0x7f75d0005080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:46.064 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.978+0000 7f75e27fc700 1 --2- 172.21.15.195:0/780294695 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f75d0003ed0 0x7f75d00046d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:46.064 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.978+0000 7f75e2ffd700 1 --2- 172.21.15.195:0/780294695 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f75e4068f80 0x7f75d0003950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:46.065 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.979+0000 7f75e37fe700 1 -- 172.21.15.195:0/780294695 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f75e4068f80 msgr2=0x7f75d0003950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:09:46.065 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.979+0000 7f75e37fe700 1 --2- 172.21.15.195:0/780294695 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f75e4068f80 0x7f75d0003950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:46.070 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.979+0000 7f75e37fe700 1 -- 172.21.15.195:0/780294695 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f75d0003ed0 msgr2=0x7f75d00046d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:09:46.071 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.979+0000 7f75e37fe700 1 --2- 172.21.15.195:0/780294695 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f75d0003ed0 0x7f75d00046d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:46.071 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.979+0000 7f75e37fe700 1 -- 172.21.15.195:0/780294695 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75d0003430 con 0x7f75d0004c10 2022-06-27T12:09:46.072 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.979+0000 7f75e37fe700 1 --2- 172.21.15.195:0/780294695 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f75d0004c10 0x7f75d0005080 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f75dc00c500 tx=0x7f75dc001d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8edd77f7206a707c server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:46.072 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.979+0000 7f75e1ffb700 1 -- 172.21.15.195:0/780294695 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75dc002040 con 0x7f75d0004c10 2022-06-27T12:09:46.073 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.979+0000 7f75e1ffb700 1 -- 172.21.15.195:0/780294695 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75dc00ce30 con 0x7f75d0004c10 2022-06-27T12:09:46.073 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.979+0000 7f75e1ffb700 1 -- 172.21.15.195:0/780294695 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75dc005db0 con 0x7f75d0004c10 2022-06-27T12:09:46.073 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.979+0000 7f75e948f700 1 -- 172.21.15.195:0/780294695 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f75d0004c10 msgr2=0x7f75d0005080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:46.074 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.979+0000 7f75e948f700 1 --2- 172.21.15.195:0/780294695 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f75d0004c10 0x7f75d0005080 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f75dc00c500 tx=0x7f75dc001d90 comp rx=0 tx=0).stop 2022-06-27T12:09:46.074 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.979+0000 7f75e37fe700 1 -- 172.21.15.195:0/780294695 reap_dead start 2022-06-27T12:09:46.075 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.979+0000 7f75e948f700 1 -- 172.21.15.195:0/780294695 shutdown_connections 2022-06-27T12:09:46.075 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.979+0000 7f75e948f700 1 -- 172.21.15.195:0/780294695 >> 172.21.15.195:0/780294695 conn(0x7f75e40ccbd0 msgr2=0x7f75e40ccfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:09:46.091 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.979+0000 7f75e948f700 1 -- 172.21.15.195:0/780294695 shutdown_connections 2022-06-27T12:09:46.091 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.979+0000 7f75e948f700 1 -- 172.21.15.195:0/780294695 wait complete. 2022-06-27T12:09:46.092 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.981+0000 7f75e948f700 1 Processor -- start 2022-06-27T12:09:46.092 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.981+0000 7f75e948f700 1 -- start start 2022-06-27T12:09:46.092 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.981+0000 7f75e948f700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f75d0004c10 0x7f75e415bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:46.093 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.981+0000 7f75e948f700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f75e415c260 0x7f75e406d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:46.093 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.981+0000 7f75e948f700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f75e406dba0 0x7f75e406c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:46.095 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.981+0000 7f75e948f700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75e406c1b0 con 0x7f75d0004c10 2022-06-27T12:09:46.096 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.981+0000 7f75e948f700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75e415c6d0 con 0x7f75e415c260 2022-06-27T12:09:46.096 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.981+0000 7f75e948f700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75e40b2800 con 0x7f75e406dba0 2022-06-27T12:09:46.097 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.982+0000 7f75e2ffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f75d0004c10 0x7f75e415bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:46.097 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.982+0000 7f75e2ffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f75d0004c10 0x7f75e415bd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:49646/0 (socket says 172.21.15.195:49646) 2022-06-27T12:09:46.097 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.982+0000 7f75e2ffd700 1 -- 172.21.15.195:0/4008271711 learned_addr learned my addr 172.21.15.195:0/4008271711 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:09:46.098 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.982+0000 7f75e37fe700 1 --2- 172.21.15.195:0/4008271711 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f75e406dba0 0x7f75e406c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:46.098 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.982+0000 7f75e27fc700 1 --2- 172.21.15.195:0/4008271711 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f75e415c260 0x7f75e406d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:46.098 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.982+0000 7f75e2ffd700 1 -- 172.21.15.195:0/4008271711 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f75e406dba0 msgr2=0x7f75e406c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:46.099 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.982+0000 7f75e2ffd700 1 --2- 172.21.15.195:0/4008271711 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f75e406dba0 0x7f75e406c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:46.099 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.982+0000 7f75e2ffd700 1 -- 172.21.15.195:0/4008271711 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f75e415c260 msgr2=0x7f75e406d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:46.100 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.982+0000 7f75e2ffd700 1 --2- 172.21.15.195:0/4008271711 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f75e415c260 0x7f75e406d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:46.100 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.982+0000 7f75e2ffd700 1 -- 172.21.15.195:0/4008271711 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75e406e010 con 0x7f75d0004c10 2022-06-27T12:09:46.100 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.982+0000 7f75e37fe700 1 --2- 172.21.15.195:0/4008271711 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f75e406dba0 0x7f75e406c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T12:09:46.101 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.982+0000 7f75e27fc700 1 --2- 172.21.15.195:0/4008271711 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f75e415c260 0x7f75e406d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T12:09:46.101 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.982+0000 7f75e2ffd700 1 --2- 172.21.15.195:0/4008271711 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f75d0004c10 0x7f75e415bd20 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f75d80119d0 tx=0x7f75d8009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:46.102 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.982+0000 7f75caffd700 1 -- 172.21.15.195:0/4008271711 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75d8004240 con 0x7f75d0004c10 2022-06-27T12:09:46.102 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.982+0000 7f75caffd700 1 -- 172.21.15.195:0/4008271711 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75d80043a0 con 0x7f75d0004c10 2022-06-27T12:09:46.103 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.982+0000 7f75caffd700 1 -- 172.21.15.195:0/4008271711 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75d800f940 con 0x7f75d0004c10 2022-06-27T12:09:46.103 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.982+0000 7f75e948f700 1 -- 172.21.15.195:0/4008271711 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75e416e570 con 0x7f75d0004c10 2022-06-27T12:09:46.106 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.982+0000 7f75e948f700 1 -- 172.21.15.195:0/4008271711 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75e406bbf0 con 0x7f75d0004c10 2022-06-27T12:09:46.107 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.983+0000 7f75e948f700 1 -- 172.21.15.195:0/4008271711 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75e406ba50 con 0x7f75d0004c10 2022-06-27T12:09:46.107 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.984+0000 7f75caffd700 1 -- 172.21.15.195:0/4008271711 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f75d8024070 con 0x7f75d0004c10 2022-06-27T12:09:46.107 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.984+0000 7f75caffd700 1 --2- 172.21.15.195:0/4008271711 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f75c4063d30 0x7f75c40661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:46.107 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.984+0000 7f75caffd700 1 -- 172.21.15.195:0/4008271711 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7234+0+0 (secure 0 0 0) 0x7f75d8018030 con 0x7f75d0004c10 2022-06-27T12:09:46.108 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.984+0000 7f75e27fc700 1 --2- 172.21.15.195:0/4008271711 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f75c4063d30 0x7f75c40661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:46.108 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.984+0000 7f75e27fc700 1 --2- 172.21.15.195:0/4008271711 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f75c4063d30 0x7f75c40661e0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f75d4007a40 tx=0x7f75d4009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:46.108 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:45.988+0000 7f75caffd700 1 -- 172.21.15.195:0/4008271711 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f75e406ba50 con 0x7f75d0004c10 2022-06-27T12:09:46.224 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:46.214+0000 7f75e948f700 1 -- 172.21.15.195:0/4008271711 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f75e406ba50 con 0x7f75d0004c10 2022-06-27T12:09:46.224 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:46.214+0000 7f75caffd700 1 -- 172.21.15.195:0/4008271711 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+5365 (secure 0 0 0) 0x7f75e406ba50 con 0x7f75d0004c10 2022-06-27T12:09:46.225 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:09:46.225 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":81,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:09:45.300745+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6708},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6708":{"gid":6708,"name":"b","rank":0,"incarnation":79,"state":"up:rejoin","state_seq":22,"addr":"172.21.15.202:6833/702244802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":702244802},{"type":"v1","addr":"172.21.15.202:6833","nonce":702244802}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:09:46.227 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:46.216+0000 7f75e948f700 1 -- 172.21.15.195:0/4008271711 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f75c4063d30 msgr2=0x7f75c40661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:46.227 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:46.216+0000 7f75e948f700 1 --2- 172.21.15.195:0/4008271711 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f75c4063d30 0x7f75c40661e0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f75d4007a40 tx=0x7f75d4009040 comp rx=0 tx=0).stop 2022-06-27T12:09:46.227 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:46.216+0000 7f75e948f700 1 -- 172.21.15.195:0/4008271711 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f75d0004c10 msgr2=0x7f75e415bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:46.228 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:46.216+0000 7f75e948f700 1 --2- 172.21.15.195:0/4008271711 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f75d0004c10 0x7f75e415bd20 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f75d80119d0 tx=0x7f75d8009060 comp rx=0 tx=0).stop 2022-06-27T12:09:46.228 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:46.216+0000 7f75e948f700 1 -- 172.21.15.195:0/4008271711 shutdown_connections 2022-06-27T12:09:46.228 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:46.216+0000 7f75e948f700 1 --2- 172.21.15.195:0/4008271711 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f75c4063d30 0x7f75c40661e0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:46.228 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:46.216+0000 7f75e948f700 1 --2- 172.21.15.195:0/4008271711 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f75d0004c10 0x7f75e415bd20 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:46.228 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:46.216+0000 7f75e948f700 1 --2- 172.21.15.195:0/4008271711 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f75e406dba0 0x7f75e406c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:46.229 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:46.216+0000 7f75e948f700 1 --2- 172.21.15.195:0/4008271711 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f75e415c260 0x7f75e406d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:46.229 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:46.216+0000 7f75e948f700 1 -- 172.21.15.195:0/4008271711 >> 172.21.15.195:0/4008271711 conn(0x7f75e40ccbd0 msgr2=0x7f75e4150240 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:09:46.230 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:46.216+0000 7f75e948f700 1 -- 172.21.15.195:0/4008271711 shutdown_connections 2022-06-27T12:09:46.230 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:46.216+0000 7f75e948f700 1 -- 172.21.15.195:0/4008271711 wait complete. 2022-06-27T12:09:46.230 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 81 2022-06-27T12:09:46.247 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:09:48.249 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:09:48.400 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.391+0000 7f9ecb126700 1 Processor -- start 2022-06-27T12:09:48.400 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.391+0000 7f9ecb126700 1 -- start start 2022-06-27T12:09:48.400 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.392+0000 7f9ecb126700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9ec4150580 0x7f9ec4156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:48.401 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.392+0000 7f9ecb126700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f9ec406f620 0x7f9ec406d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:48.423 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.392+0000 7f9ecb126700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ec40c76b0 con 0x7f9ec4150950 2022-06-27T12:09:48.423 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.392+0000 7f9ecb126700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ec4157430 con 0x7f9ec406f620 2022-06-27T12:09:48.423 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.392+0000 7f9ecb126700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec40c8250 con 0x7f9ec4150580 2022-06-27T12:09:48.423 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.392+0000 7f9ec96c3700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f9ec406f620 0x7f9ec406d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:48.423 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.392+0000 7f9ec96c3700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f9ec406f620 0x7f9ec406d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:49656/0 (socket says 172.21.15.195:49656) 2022-06-27T12:09:48.424 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.392+0000 7f9ec96c3700 1 -- 172.21.15.195:0/2562502198 learned_addr learned my addr 172.21.15.195:0/2562502198 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:09:48.424 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.392+0000 7f9ec8ec2700 1 --2- 172.21.15.195:0/2562502198 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9ec4150580 0x7f9ec4156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:48.424 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.393+0000 7f9ec96c3700 1 -- 172.21.15.195:0/2562502198 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9ec4150580 msgr2=0x7f9ec4156ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:09:48.424 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.393+0000 7f9ec96c3700 1 --2- 172.21.15.195:0/2562502198 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9ec4150580 0x7f9ec4156ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:48.504 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.393+0000 7f9ec96c3700 1 -- 172.21.15.195:0/2562502198 >> v1:172.21.15.202:6789/0 conn(0x7f9ec4150950 legacy=0x7f9ec40c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:48.504 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.393+0000 7f9ec96c3700 1 -- 172.21.15.195:0/2562502198 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ec40bad70 con 0x7f9ec406f620 2022-06-27T12:09:48.505 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.393+0000 7f9ec96c3700 1 --2- 172.21.15.195:0/2562502198 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9ec406f620 0x7f9ec406d6a0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f9ebc009f50 tx=0x7f9ebc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef53b2c802b1cd70 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:48.505 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.393+0000 7f9ec37fe700 1 -- 172.21.15.195:0/2562502198 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ebc00e070 con 0x7f9ec406f620 2022-06-27T12:09:48.506 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.393+0000 7f9ec37fe700 1 -- 172.21.15.195:0/2562502198 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ebc00ba70 con 0x7f9ec406f620 2022-06-27T12:09:48.506 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.393+0000 7f9ec37fe700 1 -- 172.21.15.195:0/2562502198 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ebc0092e0 con 0x7f9ec406f620 2022-06-27T12:09:48.507 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.393+0000 7f9ecb126700 1 -- 172.21.15.195:0/2562502198 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9ec406f620 msgr2=0x7f9ec406d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:48.508 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.393+0000 7f9ecb126700 1 --2- 172.21.15.195:0/2562502198 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9ec406f620 0x7f9ec406d6a0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f9ebc009f50 tx=0x7f9ebc009840 comp rx=0 tx=0).stop 2022-06-27T12:09:48.508 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.393+0000 7f9ecb126700 1 -- 172.21.15.195:0/2562502198 shutdown_connections 2022-06-27T12:09:48.508 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.393+0000 7f9ecb126700 1 --2- 172.21.15.195:0/2562502198 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9ec406f620 0x7f9ec406d6a0 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:48.509 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.393+0000 7f9ecb126700 1 --2- 172.21.15.195:0/2562502198 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9ec4150580 0x7f9ec4156ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:48.509 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.393+0000 7f9ecb126700 1 -- 172.21.15.195:0/2562502198 >> 172.21.15.195:0/2562502198 conn(0x7f9ec4063af0 msgr2=0x7f9ec4063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:09:48.510 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.393+0000 7f9ecb126700 1 -- 172.21.15.195:0/2562502198 shutdown_connections 2022-06-27T12:09:48.512 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.393+0000 7f9ecb126700 1 -- 172.21.15.195:0/2562502198 wait complete. 2022-06-27T12:09:48.513 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.395+0000 7f9ecb126700 1 Processor -- start 2022-06-27T12:09:48.514 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.395+0000 7f9ecb126700 1 -- start start 2022-06-27T12:09:48.514 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.395+0000 7f9ecb126700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9ec406f620 0x7f9ec41555f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:48.514 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.395+0000 7f9ecb126700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9ec4150580 0x7f9ec40c20a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:48.515 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.395+0000 7f9ecb126700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9ec4150950 0x7f9ec415ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:48.515 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.395+0000 7f9ecb126700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec4167290 con 0x7f9ec406f620 2022-06-27T12:09:48.515 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.395+0000 7f9ecb126700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec41666a0 con 0x7f9ec4150580 2022-06-27T12:09:48.516 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.395+0000 7f9ecb126700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec40c29c0 con 0x7f9ec4150950 2022-06-27T12:09:48.516 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.396+0000 7f9ec8ec2700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9ec406f620 0x7f9ec41555f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:48.517 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.396+0000 7f9ec8ec2700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9ec406f620 0x7f9ec41555f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:49660/0 (socket says 172.21.15.195:49660) 2022-06-27T12:09:48.517 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.396+0000 7f9ec8ec2700 1 -- 172.21.15.195:0/1837356251 learned_addr learned my addr 172.21.15.195:0/1837356251 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:09:48.517 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.396+0000 7f9ec96c3700 1 --2- 172.21.15.195:0/1837356251 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9ec4150950 0x7f9ec415ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:48.518 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.396+0000 7f9ec3fff700 1 --2- 172.21.15.195:0/1837356251 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9ec4150580 0x7f9ec40c20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:48.518 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.396+0000 7f9ec8ec2700 1 -- 172.21.15.195:0/1837356251 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9ec4150950 msgr2=0x7f9ec415ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:48.519 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.396+0000 7f9ec8ec2700 1 --2- 172.21.15.195:0/1837356251 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9ec4150950 0x7f9ec415ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:48.519 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.396+0000 7f9ec8ec2700 1 -- 172.21.15.195:0/1837356251 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9ec4150580 msgr2=0x7f9ec40c20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:48.520 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.396+0000 7f9ec8ec2700 1 --2- 172.21.15.195:0/1837356251 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9ec4150580 0x7f9ec40c20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:48.520 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.396+0000 7f9ec8ec2700 1 -- 172.21.15.195:0/1837356251 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ebc009ae0 con 0x7f9ec406f620 2022-06-27T12:09:48.520 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.396+0000 7f9ec96c3700 1 --2- 172.21.15.195:0/1837356251 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9ec4150950 0x7f9ec415ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:09:48.522 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.396+0000 7f9ec8ec2700 1 --2- 172.21.15.195:0/1837356251 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9ec406f620 0x7f9ec41555f0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f9eb800d860 tx=0x7f9eb8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:48.522 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.396+0000 7f9ec0ff9700 1 -- 172.21.15.195:0/1837356251 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eb8014620 con 0x7f9ec406f620 2022-06-27T12:09:48.524 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.396+0000 7f9ec0ff9700 1 -- 172.21.15.195:0/1837356251 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9eb8014780 con 0x7f9ec406f620 2022-06-27T12:09:48.524 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.396+0000 7f9ecb126700 1 -- 172.21.15.195:0/1837356251 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ec4166960 con 0x7f9ec406f620 2022-06-27T12:09:48.525 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.396+0000 7f9ec0ff9700 1 -- 172.21.15.195:0/1837356251 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eb8014b40 con 0x7f9ec406f620 2022-06-27T12:09:48.525 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.396+0000 7f9ecb126700 1 -- 172.21.15.195:0/1837356251 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ec4166f20 con 0x7f9ec406f620 2022-06-27T12:09:48.526 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.397+0000 7f9ecb126700 1 -- 172.21.15.195:0/1837356251 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ec4167570 con 0x7f9ec406f620 2022-06-27T12:09:48.526 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.398+0000 7f9ec0ff9700 1 -- 172.21.15.195:0/1837356251 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f9eb8024070 con 0x7f9ec406f620 2022-06-27T12:09:48.527 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.398+0000 7f9ec0ff9700 1 --2- 172.21.15.195:0/1837356251 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9eac063d00 0x7f9eac0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:48.527 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.398+0000 7f9ec0ff9700 1 -- 172.21.15.195:0/1837356251 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7234+0+0 (secure 0 0 0) 0x7f9eb80812c0 con 0x7f9ec406f620 2022-06-27T12:09:48.528 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.398+0000 7f9ec3fff700 1 --2- 172.21.15.195:0/1837356251 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9eac063d00 0x7f9eac0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:48.528 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.398+0000 7f9ec3fff700 1 --2- 172.21.15.195:0/1837356251 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9eac063d00 0x7f9eac0661b0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f9ec4068d20 tx=0x7f9eb0009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:48.528 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.402+0000 7f9ec0ff9700 1 -- 172.21.15.195:0/1837356251 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f9eb804f9b0 con 0x7f9ec406f620 2022-06-27T12:09:48.635 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.623+0000 7f9ecb126700 1 -- 172.21.15.195:0/1837356251 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9ec40722c0 con 0x7f9ec406f620 2022-06-27T12:09:48.635 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.624+0000 7f9ec0ff9700 1 -- 172.21.15.195:0/1837356251 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+5365 (secure 0 0 0) 0x7f9eb8052fd0 con 0x7f9ec406f620 2022-06-27T12:09:48.641 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:09:48.641 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":81,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:09:45.300745+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6708},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6708":{"gid":6708,"name":"b","rank":0,"incarnation":79,"state":"up:rejoin","state_seq":22,"addr":"172.21.15.202:6833/702244802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":702244802},{"type":"v1","addr":"172.21.15.202:6833","nonce":702244802}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:09:48.649 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.626+0000 7f9ecb126700 1 -- 172.21.15.195:0/1837356251 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9eac063d00 msgr2=0x7f9eac0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:48.649 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.626+0000 7f9ecb126700 1 --2- 172.21.15.195:0/1837356251 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9eac063d00 0x7f9eac0661b0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f9ec4068d20 tx=0x7f9eb0009040 comp rx=0 tx=0).stop 2022-06-27T12:09:48.651 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.626+0000 7f9ecb126700 1 -- 172.21.15.195:0/1837356251 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9ec406f620 msgr2=0x7f9ec41555f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:48.651 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.626+0000 7f9ecb126700 1 --2- 172.21.15.195:0/1837356251 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9ec406f620 0x7f9ec41555f0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f9eb800d860 tx=0x7f9eb8008040 comp rx=0 tx=0).stop 2022-06-27T12:09:48.680 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.626+0000 7f9ecb126700 1 -- 172.21.15.195:0/1837356251 shutdown_connections 2022-06-27T12:09:48.681 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.626+0000 7f9ecb126700 1 --2- 172.21.15.195:0/1837356251 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9eac063d00 0x7f9eac0661b0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:48.681 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.626+0000 7f9ecb126700 1 --2- 172.21.15.195:0/1837356251 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9ec406f620 0x7f9ec41555f0 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:48.683 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.626+0000 7f9ecb126700 1 --2- 172.21.15.195:0/1837356251 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9ec4150950 0x7f9ec415ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:48.683 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.626+0000 7f9ecb126700 1 --2- 172.21.15.195:0/1837356251 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9ec4150580 0x7f9ec40c20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:48.684 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.626+0000 7f9ecb126700 1 -- 172.21.15.195:0/1837356251 >> 172.21.15.195:0/1837356251 conn(0x7f9ec4063af0 msgr2=0x7f9ec40c5c70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:09:48.684 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.626+0000 7f9ecb126700 1 -- 172.21.15.195:0/1837356251 shutdown_connections 2022-06-27T12:09:48.685 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:48.626+0000 7f9ecb126700 1 -- 172.21.15.195:0/1837356251 wait complete. 2022-06-27T12:09:48.685 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 81 2022-06-27T12:09:48.686 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:09:50.689 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:09:50.861 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:50.830+0000 7efd12d82700 1 Processor -- start 2022-06-27T12:09:50.861 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:50.830+0000 7efd12d82700 1 -- start start 2022-06-27T12:09:50.862 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:50.831+0000 7efd12d82700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7efd0c150500 0x7efd0c063af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:50.863 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:50.831+0000 7efd12d82700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd0c064030 con 0x7efd0c1508d0 2022-06-27T12:09:50.864 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:50.831+0000 7efd12d82700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd0c0c4160 con 0x7efd0c06a450 2022-06-27T12:09:50.864 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:50.831+0000 7efd12d82700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd0c0c4440 con 0x7efd0c150500 2022-06-27T12:09:50.865 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:50.831+0000 7efd1131f700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7efd0c1508d0 0x7efd0c0bbf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:55858/0 (socket says 172.21.15.195:55858) 2022-06-27T12:09:50.865 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:50.831+0000 7efd1131f700 1 -- 172.21.15.195:0/4010809162 learned_addr learned my addr 172.21.15.195:0/4010809162 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:09:50.866 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:50.831+0000 7efd10b1e700 1 --2- 172.21.15.195:0/4010809162 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7efd0c150500 0x7efd0c063af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:50.868 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:50.831+0000 7efd0b7fe700 1 -- 172.21.15.195:0/4010809162 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2999366880 0 0) 0x7efd04001490 con 0x7efd0c1508d0 2022-06-27T12:09:50.869 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:50.831+0000 7efd0b7fe700 1 -- 172.21.15.195:0/4010809162 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcf0003430 con 0x7efd0c1508d0 2022-06-27T12:09:50.869 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:50.831+0000 7efd0b7fe700 1 -- 172.21.15.195:0/4010809162 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1313195925 0 0) 0x7efd0c0c4160 con 0x7efd0c06a450 2022-06-27T12:09:50.870 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:50.831+0000 7efd0b7fe700 1 -- 172.21.15.195:0/4010809162 --> v1:172.21.15.202:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd04001490 con 0x7efd0c06a450 2022-06-27T12:09:50.870 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:50.831+0000 7efd0b7fe700 1 -- 172.21.15.195:0/4010809162 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3926191949 0 0) 0x7efcf0003430 con 0x7efd0c1508d0 2022-06-27T12:09:50.870 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:50.831+0000 7efd10b1e700 1 -- 172.21.15.195:0/4010809162 >> v1:172.21.15.202:6789/0 conn(0x7efd0c06a450 legacy=0x7efd0c0c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:50.871 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:50.831+0000 7efd10b1e700 1 -- 172.21.15.195:0/4010809162 >> v1:172.21.15.195:6789/0 conn(0x7efd0c1508d0 legacy=0x7efd0c0bbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:50.871 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:50.831+0000 7efd10b1e700 1 -- 172.21.15.195:0/4010809162 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd0c15a800 con 0x7efd0c150500 2022-06-27T12:09:50.871 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:50.832+0000 7efd10b1e700 1 --2- 172.21.15.195:0/4010809162 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7efd0c150500 0x7efd0c063af0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7efd00009750 tx=0x7efd00009a60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f060dc1b0cab1f7 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:50.871 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:50.832+0000 7efd0b7fe700 1 -- 172.21.15.195:0/4010809162 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd04001980 con 0x7efd0c150500 2022-06-27T12:09:50.872 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:50.832+0000 7efd0b7fe700 1 -- 172.21.15.195:0/4010809162 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd00004030 con 0x7efd0c150500 2022-06-27T12:09:50.872 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:50.832+0000 7efd0b7fe700 1 -- 172.21.15.195:0/4010809162 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd000043d0 con 0x7efd0c150500 2022-06-27T12:09:50.872 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:50.832+0000 7efd12d82700 1 -- 172.21.15.195:0/4010809162 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7efd0c150500 msgr2=0x7efd0c063af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:50.873 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:50.832+0000 7efd12d82700 1 --2- 172.21.15.195:0/4010809162 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7efd0c150500 0x7efd0c063af0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7efd00009750 tx=0x7efd00009a60 comp rx=0 tx=0).stop 2022-06-27T12:09:50.873 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:50.832+0000 7efd12d82700 1 -- 172.21.15.195:0/4010809162 shutdown_connections 2022-06-27T12:09:50.873 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:50.832+0000 7efd12d82700 1 --2- 172.21.15.195:0/4010809162 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7efd0c150500 0x7efd0c063af0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:50.873 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:50.832+0000 7efd12d82700 1 -- 172.21.15.195:0/4010809162 >> 172.21.15.195:0/4010809162 conn(0x7efd0c0c9b00 msgr2=0x7efd0c0c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:09:50.874 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:50.832+0000 7efd12d82700 1 -- 172.21.15.195:0/4010809162 shutdown_connections 2022-06-27T12:09:50.874 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:50.832+0000 7efd12d82700 1 -- 172.21.15.195:0/4010809162 wait complete. 2022-06-27T12:09:50.875 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:50.833+0000 7efd12d82700 1 Processor -- start 2022-06-27T12:09:50.875 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:50.834+0000 7efd12d82700 1 -- start start 2022-06-27T12:09:50.876 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:50.834+0000 7efd12d82700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efd0c06a450 0x7efd0c166d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:50.876 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:50.834+0000 7efd12d82700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7efd0c150500 0x7efd0c0c15b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:50.876 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:50.834+0000 7efd12d82700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7efd0c1508d0 0x7efd0c0c3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:50.877 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:50.834+0000 7efd12d82700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd0c167310 con 0x7efd0c06a450 2022-06-27T12:09:50.877 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:50.834+0000 7efd12d82700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd0c14dc50 con 0x7efd0c1508d0 2022-06-27T12:09:50.878 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:50.834+0000 7efd12d82700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd0c14d040 con 0x7efd0c150500 2022-06-27T12:09:50.878 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:50.834+0000 7efd10b1e700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efd0c06a450 0x7efd0c166d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:50.879 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:50.834+0000 7efd10b1e700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efd0c06a450 0x7efd0c166d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:49672/0 (socket says 172.21.15.195:49672) 2022-06-27T12:09:50.879 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:50.834+0000 7efd10b1e700 1 -- 172.21.15.195:0/3826217543 learned_addr learned my addr 172.21.15.195:0/3826217543 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:09:50.880 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:50.835+0000 7efd0bfff700 1 --2- 172.21.15.195:0/3826217543 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7efd0c150500 0x7efd0c0c15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:50.880 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:50.835+0000 7efd1131f700 1 --2- 172.21.15.195:0/3826217543 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7efd0c1508d0 0x7efd0c0c3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:50.881 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:50.835+0000 7efd10b1e700 1 -- 172.21.15.195:0/3826217543 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7efd0c150500 msgr2=0x7efd0c0c15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:50.881 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:50.835+0000 7efd10b1e700 1 --2- 172.21.15.195:0/3826217543 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7efd0c150500 0x7efd0c0c15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:50.882 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:50.835+0000 7efd10b1e700 1 -- 172.21.15.195:0/3826217543 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7efd0c1508d0 msgr2=0x7efd0c0c3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:50.892 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:50.835+0000 7efd10b1e700 1 --2- 172.21.15.195:0/3826217543 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7efd0c1508d0 0x7efd0c0c3890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:50.892 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:50.835+0000 7efd10b1e700 1 -- 172.21.15.195:0/3826217543 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd00009290 con 0x7efd0c06a450 2022-06-27T12:09:50.893 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:50.835+0000 7efd1131f700 1 --2- 172.21.15.195:0/3826217543 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7efd0c1508d0 0x7efd0c0c3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:09:50.893 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:50.835+0000 7efd10b1e700 1 --2- 172.21.15.195:0/3826217543 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efd0c06a450 0x7efd0c166d20 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7efd0c06f810 tx=0x7efd00015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:50.894 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:50.835+0000 7efd08ff9700 1 -- 172.21.15.195:0/3826217543 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd0001b680 con 0x7efd0c06a450 2022-06-27T12:09:50.901 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:50.835+0000 7efd08ff9700 1 -- 172.21.15.195:0/3826217543 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd00022070 con 0x7efd0c06a450 2022-06-27T12:09:50.901 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:50.835+0000 7efd12d82700 1 -- 172.21.15.195:0/3826217543 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd0c14d320 con 0x7efd0c06a450 2022-06-27T12:09:50.902 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:50.835+0000 7efd08ff9700 1 -- 172.21.15.195:0/3826217543 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd0001b820 con 0x7efd0c06a450 2022-06-27T12:09:50.902 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:50.835+0000 7efd12d82700 1 -- 172.21.15.195:0/3826217543 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd0c14d8e0 con 0x7efd0c06a450 2022-06-27T12:09:50.903 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:50.836+0000 7efd12d82700 1 -- 172.21.15.195:0/3826217543 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd0c14daa0 con 0x7efd0c06a450 2022-06-27T12:09:50.903 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:50.837+0000 7efd08ff9700 1 -- 172.21.15.195:0/3826217543 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7efd0002e070 con 0x7efd0c06a450 2022-06-27T12:09:50.904 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:50.837+0000 7efd08ff9700 1 --2- 172.21.15.195:0/3826217543 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7efcf4063d00 0x7efcf40661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:50.905 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:50.837+0000 7efd08ff9700 1 -- 172.21.15.195:0/3826217543 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7234+0+0 (secure 0 0 0) 0x7efd00082660 con 0x7efd0c06a450 2022-06-27T12:09:50.906 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:50.837+0000 7efd0bfff700 1 --2- 172.21.15.195:0/3826217543 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7efcf4063d00 0x7efcf40661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:50.906 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:50.837+0000 7efd0bfff700 1 --2- 172.21.15.195:0/3826217543 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7efcf4063d00 0x7efcf40661b0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7efd0c14c0b0 tx=0x7efcf8009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:50.907 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:50.841+0000 7efd08ff9700 1 -- 172.21.15.195:0/3826217543 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7efd00050e10 con 0x7efd0c06a450 2022-06-27T12:09:51.064 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:51.060+0000 7efd12d82700 1 -- 172.21.15.195:0/3826217543 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efd0c0722c0 con 0x7efd0c06a450 2022-06-27T12:09:51.065 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:51.061+0000 7efd08ff9700 1 -- 172.21.15.195:0/3826217543 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+5365 (secure 0 0 0) 0x7efd00054430 con 0x7efd0c06a450 2022-06-27T12:09:51.066 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:09:51.067 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":81,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:09:45.300745+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6708},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6708":{"gid":6708,"name":"b","rank":0,"incarnation":79,"state":"up:rejoin","state_seq":22,"addr":"172.21.15.202:6833/702244802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":702244802},{"type":"v1","addr":"172.21.15.202:6833","nonce":702244802}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:09:51.069 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:51.063+0000 7efd12d82700 1 -- 172.21.15.195:0/3826217543 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7efcf4063d00 msgr2=0x7efcf40661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:51.071 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:51.063+0000 7efd12d82700 1 --2- 172.21.15.195:0/3826217543 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7efcf4063d00 0x7efcf40661b0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7efd0c14c0b0 tx=0x7efcf8009040 comp rx=0 tx=0).stop 2022-06-27T12:09:51.072 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:51.063+0000 7efd12d82700 1 -- 172.21.15.195:0/3826217543 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efd0c06a450 msgr2=0x7efd0c166d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:51.072 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:51.063+0000 7efd12d82700 1 --2- 172.21.15.195:0/3826217543 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efd0c06a450 0x7efd0c166d20 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7efd0c06f810 tx=0x7efd00015000 comp rx=0 tx=0).stop 2022-06-27T12:09:51.077 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:51.063+0000 7efd12d82700 1 -- 172.21.15.195:0/3826217543 shutdown_connections 2022-06-27T12:09:51.078 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:51.063+0000 7efd12d82700 1 --2- 172.21.15.195:0/3826217543 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7efcf4063d00 0x7efcf40661b0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:51.078 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:51.063+0000 7efd12d82700 1 --2- 172.21.15.195:0/3826217543 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efd0c06a450 0x7efd0c166d20 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:51.079 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:51.063+0000 7efd12d82700 1 --2- 172.21.15.195:0/3826217543 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7efd0c1508d0 0x7efd0c0c3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:51.079 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:51.063+0000 7efd12d82700 1 --2- 172.21.15.195:0/3826217543 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7efd0c150500 0x7efd0c0c15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:51.080 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:51.063+0000 7efd12d82700 1 -- 172.21.15.195:0/3826217543 >> 172.21.15.195:0/3826217543 conn(0x7efd0c0c9b00 msgr2=0x7efd0c14fd50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:09:51.080 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:51.063+0000 7efd12d82700 1 -- 172.21.15.195:0/3826217543 shutdown_connections 2022-06-27T12:09:51.081 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:51.063+0000 7efd12d82700 1 -- 172.21.15.195:0/3826217543 wait complete. 2022-06-27T12:09:51.082 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 81 2022-06-27T12:09:51.099 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 81, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-06-27T11:47:52.358095+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 81, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6708": { "addr": "172.21.15.202:6833/702244802", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6832", "nonce": 702244802, "type": "v2" }, { "addr": "172.21.15.202:6833", "nonce": 702244802, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 6708, "incarnation": 79, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:rejoin", "state_seq": 22 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-06-27T12:09:45.300745+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6708 } } } ], "standbys": [ { "addr": "172.21.15.202:6837/3305006841", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6836", "nonce": 3305006841, "type": "v2" }, { "addr": "172.21.15.202:6837", "nonce": 3305006841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6835/2060709822", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6834", "nonce": 2060709822, "type": "v2" }, { "addr": "172.21.15.195:6835", "nonce": 2060709822, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6838/1598751652", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6836", "nonce": 1598751652, "type": "v2" }, { "addr": "172.21.15.195:6838", "nonce": 1598751652, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4275, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6839/1313800661", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6837", "nonce": 1313800661, "type": "v2" }, { "addr": "172.21.15.195:6839", "nonce": 1313800661, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4278, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-06-27T12:09:53.115 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:09:53.296 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.285+0000 7fedc317c700 1 Processor -- start 2022-06-27T12:09:53.297 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.285+0000 7fedc317c700 1 -- start start 2022-06-27T12:09:53.297 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.286+0000 7fedc317c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fedbc153cb0 0x7fedbc156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:53.298 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.286+0000 7fedc317c700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedbc0c1a20 con 0x7fedbc06f640 2022-06-27T12:09:53.299 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.286+0000 7fedc317c700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedbc0c1130 con 0x7fedbc0c4160 2022-06-27T12:09:53.300 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.286+0000 7fedc317c700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedbc06ea90 con 0x7fedbc153cb0 2022-06-27T12:09:53.300 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.286+0000 7fedc1719700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fedbc153cb0 0x7fedbc156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:53.300 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.286+0000 7fedc0f18700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fedbc06f640 0x7fedbc0c1eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:55868/0 (socket says 172.21.15.195:55868) 2022-06-27T12:09:53.301 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.286+0000 7fedc0f18700 1 -- 172.21.15.195:0/295046518 learned_addr learned my addr 172.21.15.195:0/295046518 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:09:53.301 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.286+0000 7fedc1719700 1 -- 172.21.15.195:0/295046518 >> v1:172.21.15.202:6789/0 conn(0x7fedbc0c4160 legacy=0x7fedbc1535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:09:53.301 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.286+0000 7fedc1719700 1 -- 172.21.15.195:0/295046518 >> v1:172.21.15.195:6789/0 conn(0x7fedbc06f640 legacy=0x7fedbc0c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:53.302 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.286+0000 7fedc1719700 1 -- 172.21.15.195:0/295046518 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedbc06ed70 con 0x7fedbc153cb0 2022-06-27T12:09:53.302 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.286+0000 7fedbb7fe700 1 -- 172.21.15.195:0/295046518 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 109719494 0 0) 0x7fedbc0c1a20 con 0x7fedbc06f640 2022-06-27T12:09:53.302 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.286+0000 7fedc1719700 1 --2- 172.21.15.195:0/295046518 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fedbc153cb0 0x7fedbc156ed0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fedb4009e60 tx=0x7fedb4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23d107895147f0a3 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:53.303 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.287+0000 7fedbb7fe700 1 -- 172.21.15.195:0/295046518 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedb400e070 con 0x7fedbc153cb0 2022-06-27T12:09:53.303 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.287+0000 7fedbb7fe700 1 -- 172.21.15.195:0/295046518 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedb400bc80 con 0x7fedbc153cb0 2022-06-27T12:09:53.304 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.287+0000 7fedbb7fe700 1 -- 172.21.15.195:0/295046518 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedb40038a0 con 0x7fedbc153cb0 2022-06-27T12:09:53.304 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.287+0000 7fedc317c700 1 -- 172.21.15.195:0/295046518 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fedbc153cb0 msgr2=0x7fedbc156ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:53.304 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.287+0000 7fedc317c700 1 --2- 172.21.15.195:0/295046518 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fedbc153cb0 0x7fedbc156ed0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fedb4009e60 tx=0x7fedb4009780 comp rx=0 tx=0).stop 2022-06-27T12:09:53.314 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.287+0000 7fedc317c700 1 -- 172.21.15.195:0/295046518 shutdown_connections 2022-06-27T12:09:53.315 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.287+0000 7fedc317c700 1 --2- 172.21.15.195:0/295046518 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fedbc153cb0 0x7fedbc156ed0 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:53.315 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.287+0000 7fedc317c700 1 -- 172.21.15.195:0/295046518 >> 172.21.15.195:0/295046518 conn(0x7fedbc0c6350 msgr2=0x7fedbc0c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:09:53.316 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.287+0000 7fedc317c700 1 -- 172.21.15.195:0/295046518 shutdown_connections 2022-06-27T12:09:53.316 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.287+0000 7fedc317c700 1 -- 172.21.15.195:0/295046518 wait complete. 2022-06-27T12:09:53.317 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.288+0000 7fedc317c700 1 Processor -- start 2022-06-27T12:09:53.317 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.288+0000 7fedc317c700 1 -- start start 2022-06-27T12:09:53.317 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.289+0000 7fedc317c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fedbc06f640 0x7fedbc15ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:53.323 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.289+0000 7fedc317c700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fedbc0c4160 0x7fedbc15a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:53.323 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.289+0000 7fedc317c700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fedbc153cb0 0x7fedbc150500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:53.323 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.289+0000 7fedc317c700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedbc1542e0 con 0x7fedbc06f640 2022-06-27T12:09:53.324 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.289+0000 7fedc317c700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedbc16b180 con 0x7fedbc153cb0 2022-06-27T12:09:53.324 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.289+0000 7fedc317c700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedbc154c80 con 0x7fedbc0c4160 2022-06-27T12:09:53.324 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.289+0000 7fedc0f18700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fedbc06f640 0x7fedbc15ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:53.325 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.289+0000 7fedc0f18700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fedbc06f640 0x7fedbc15ae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:49686/0 (socket says 172.21.15.195:49686) 2022-06-27T12:09:53.325 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.289+0000 7fedc0f18700 1 -- 172.21.15.195:0/1015853152 learned_addr learned my addr 172.21.15.195:0/1015853152 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:09:53.325 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.290+0000 7fedc1719700 1 --2- 172.21.15.195:0/1015853152 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fedbc153cb0 0x7fedbc150500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:53.326 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.290+0000 7fedbbfff700 1 --2- 172.21.15.195:0/1015853152 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fedbc0c4160 0x7fedbc15a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:53.326 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.290+0000 7fedc0f18700 1 -- 172.21.15.195:0/1015853152 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fedbc0c4160 msgr2=0x7fedbc15a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:53.326 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.290+0000 7fedc0f18700 1 --2- 172.21.15.195:0/1015853152 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fedbc0c4160 0x7fedbc15a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:53.326 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.290+0000 7fedc0f18700 1 -- 172.21.15.195:0/1015853152 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fedbc153cb0 msgr2=0x7fedbc150500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:53.327 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.290+0000 7fedc0f18700 1 --2- 172.21.15.195:0/1015853152 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fedbc153cb0 0x7fedbc150500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:53.327 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.290+0000 7fedc0f18700 1 -- 172.21.15.195:0/1015853152 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedb4009a20 con 0x7fedbc06f640 2022-06-27T12:09:53.327 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.290+0000 7fedbbfff700 1 --2- 172.21.15.195:0/1015853152 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fedbc0c4160 0x7fedbc15a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T12:09:53.328 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.290+0000 7fedc0f18700 1 --2- 172.21.15.195:0/1015853152 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fedbc06f640 0x7fedbc15ae90 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fedb000c590 tx=0x7fedb0007cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:53.328 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.290+0000 7fedb8ff9700 1 -- 172.21.15.195:0/1015853152 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedb0007360 con 0x7fedbc06f640 2022-06-27T12:09:53.328 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.290+0000 7fedb8ff9700 1 -- 172.21.15.195:0/1015853152 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedb0007500 con 0x7fedbc06f640 2022-06-27T12:09:53.328 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.290+0000 7fedc317c700 1 -- 172.21.15.195:0/1015853152 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedbc0c0350 con 0x7fedbc06f640 2022-06-27T12:09:53.329 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.290+0000 7fedb8ff9700 1 -- 172.21.15.195:0/1015853152 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedb0004940 con 0x7fedbc06f640 2022-06-27T12:09:53.329 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.290+0000 7fedc317c700 1 -- 172.21.15.195:0/1015853152 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedbc0c08d0 con 0x7fedbc06f640 2022-06-27T12:09:53.329 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.291+0000 7fedc317c700 1 -- 172.21.15.195:0/1015853152 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedbc065f30 con 0x7fedbc06f640 2022-06-27T12:09:53.329 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.291+0000 7fedb8ff9700 1 -- 172.21.15.195:0/1015853152 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fedb0010030 con 0x7fedbc06f640 2022-06-27T12:09:53.331 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.292+0000 7fedb8ff9700 1 --2- 172.21.15.195:0/1015853152 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7feda4063d00 0x7feda40661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:53.331 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.292+0000 7fedb8ff9700 1 -- 172.21.15.195:0/1015853152 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7234+0+0 (secure 0 0 0) 0x7fedb00809d0 con 0x7fedbc06f640 2022-06-27T12:09:53.331 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.292+0000 7fedbbfff700 1 --2- 172.21.15.195:0/1015853152 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7feda4063d00 0x7feda40661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:53.331 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.292+0000 7fedbbfff700 1 --2- 172.21.15.195:0/1015853152 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7feda4063d00 0x7feda40661b0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fedbc15ac50 tx=0x7feda8009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:53.332 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.296+0000 7fedb8ff9700 1 -- 172.21.15.195:0/1015853152 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fedbc065f30 con 0x7fedbc06f640 2022-06-27T12:09:53.548 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.516+0000 7fedc317c700 1 -- 172.21.15.195:0/1015853152 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fedbc065f30 con 0x7fedbc06f640 2022-06-27T12:09:53.549 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.517+0000 7fedb8ff9700 1 -- 172.21.15.195:0/1015853152 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+5371 (secure 0 0 0) 0x7fedbc065f30 con 0x7fedbc06f640 2022-06-27T12:09:53.558 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:09:53.559 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:09:51.320525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6708},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6708":{"gid":6708,"name":"b","rank":0,"incarnation":79,"state":"up:clientreplay","state_seq":24,"addr":"172.21.15.202:6833/702244802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":702244802},{"type":"v1","addr":"172.21.15.202:6833","nonce":702244802}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:09:53.563 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.519+0000 7fedc317c700 1 -- 172.21.15.195:0/1015853152 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7feda4063d00 msgr2=0x7feda40661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:53.563 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.519+0000 7fedc317c700 1 --2- 172.21.15.195:0/1015853152 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7feda4063d00 0x7feda40661b0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fedbc15ac50 tx=0x7feda8009040 comp rx=0 tx=0).stop 2022-06-27T12:09:53.564 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.519+0000 7fedc317c700 1 -- 172.21.15.195:0/1015853152 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fedbc06f640 msgr2=0x7fedbc15ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:53.584 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.519+0000 7fedc317c700 1 --2- 172.21.15.195:0/1015853152 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fedbc06f640 0x7fedbc15ae90 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fedb000c590 tx=0x7fedb0007cb0 comp rx=0 tx=0).stop 2022-06-27T12:09:53.584 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.519+0000 7fedc317c700 1 -- 172.21.15.195:0/1015853152 shutdown_connections 2022-06-27T12:09:53.585 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.519+0000 7fedc317c700 1 --2- 172.21.15.195:0/1015853152 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7feda4063d00 0x7feda40661b0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:53.585 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.519+0000 7fedc317c700 1 --2- 172.21.15.195:0/1015853152 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fedbc06f640 0x7fedbc15ae90 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:53.586 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.519+0000 7fedc317c700 1 --2- 172.21.15.195:0/1015853152 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fedbc153cb0 0x7fedbc150500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:53.586 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.519+0000 7fedc317c700 1 --2- 172.21.15.195:0/1015853152 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fedbc0c4160 0x7fedbc15a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:53.586 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.519+0000 7fedc317c700 1 -- 172.21.15.195:0/1015853152 >> 172.21.15.195:0/1015853152 conn(0x7fedbc0c6350 msgr2=0x7fedbc06cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:09:53.587 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.519+0000 7fedc317c700 1 -- 172.21.15.195:0/1015853152 shutdown_connections 2022-06-27T12:09:53.587 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:53.519+0000 7fedc317c700 1 -- 172.21.15.195:0/1015853152 wait complete. 2022-06-27T12:09:53.587 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 82 2022-06-27T12:09:53.588 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:09:55.594 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:09:55.747 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.732+0000 7fd1e47b9700 1 Processor -- start 2022-06-27T12:09:55.748 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.732+0000 7fd1e47b9700 1 -- start start 2022-06-27T12:09:55.748 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.733+0000 7fd1e47b9700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fd1dc150500 0x7fd1dc063af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:55.749 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.733+0000 7fd1e47b9700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd1dc1508d0 0x7fd1dc0bbf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:55.749 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.733+0000 7fd1e47b9700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1dc064030 con 0x7fd1dc06a450 2022-06-27T12:09:55.749 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.733+0000 7fd1e47b9700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1dc0c4160 con 0x7fd1dc150500 2022-06-27T12:09:55.787 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.733+0000 7fd1e47b9700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1dc0c4440 con 0x7fd1dc1508d0 2022-06-27T12:09:55.787 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.733+0000 7fd1e1d54700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fd1dc06a450 0x7fd1dc0c48d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:55882/0 (socket says 172.21.15.195:55882) 2022-06-27T12:09:55.788 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.733+0000 7fd1e1d54700 1 -- 172.21.15.195:0/49072165 learned_addr learned my addr 172.21.15.195:0/49072165 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:09:55.788 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.733+0000 7fd1e2d56700 1 --2- 172.21.15.195:0/49072165 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd1dc1508d0 0x7fd1dc0bbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:55.789 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.733+0000 7fd1e2555700 1 --2- 172.21.15.195:0/49072165 >> v2:172.21.15.202:3300/0 conn(0x7fd1dc150500 0x7fd1dc063af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:55.789 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.734+0000 7fd1e1553700 1 -- 172.21.15.195:0/49072165 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3967542103 0 0) 0x7fd1d8001110 con 0x7fd1dc06a450 2022-06-27T12:09:55.802 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.734+0000 7fd1e1553700 1 -- 172.21.15.195:0/49072165 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1c4003430 con 0x7fd1dc06a450 2022-06-27T12:09:55.803 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.734+0000 7fd1e2555700 1 -- 172.21.15.195:0/49072165 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd1dc1508d0 msgr2=0x7fd1dc0bbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:09:55.803 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.734+0000 7fd1e1553700 1 -- 172.21.15.195:0/49072165 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2572824492 0 0) 0x7fd1c4003430 con 0x7fd1dc06a450 2022-06-27T12:09:55.804 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.734+0000 7fd1e2555700 1 --2- 172.21.15.195:0/49072165 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd1dc1508d0 0x7fd1dc0bbf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:55.804 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.734+0000 7fd1e2555700 1 -- 172.21.15.195:0/49072165 >> v1:172.21.15.195:6789/0 conn(0x7fd1dc06a450 legacy=0x7fd1dc0c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:55.805 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.734+0000 7fd1e2555700 1 -- 172.21.15.195:0/49072165 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1dc15a800 con 0x7fd1dc150500 2022-06-27T12:09:55.805 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.734+0000 7fd1e2d56700 1 --2- 172.21.15.195:0/49072165 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd1dc1508d0 0x7fd1dc0bbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:09:55.805 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.734+0000 7fd1e2555700 1 --2- 172.21.15.195:0/49072165 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd1dc150500 0x7fd1dc063af0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc009720 tx=0x7fd1cc009a30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7244de29459931f4 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:55.807 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.734+0000 7fd1e1553700 1 -- 172.21.15.195:0/49072165 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1d8001600 con 0x7fd1dc150500 2022-06-27T12:09:55.808 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.734+0000 7fd1e1553700 1 -- 172.21.15.195:0/49072165 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1cc004030 con 0x7fd1dc150500 2022-06-27T12:09:55.808 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.734+0000 7fd1e1553700 1 -- 172.21.15.195:0/49072165 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1cc0043b0 con 0x7fd1dc150500 2022-06-27T12:09:55.810 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.735+0000 7fd1e47b9700 1 -- 172.21.15.195:0/49072165 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd1dc150500 msgr2=0x7fd1dc063af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:55.810 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.735+0000 7fd1e47b9700 1 --2- 172.21.15.195:0/49072165 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd1dc150500 0x7fd1dc063af0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc009720 tx=0x7fd1cc009a30 comp rx=0 tx=0).stop 2022-06-27T12:09:55.810 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.735+0000 7fd1e47b9700 1 -- 172.21.15.195:0/49072165 shutdown_connections 2022-06-27T12:09:55.811 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.735+0000 7fd1e47b9700 1 --2- 172.21.15.195:0/49072165 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd1dc1508d0 0x7fd1dc0bbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:55.815 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.735+0000 7fd1e47b9700 1 --2- 172.21.15.195:0/49072165 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd1dc150500 0x7fd1dc063af0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:55.823 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.735+0000 7fd1e47b9700 1 -- 172.21.15.195:0/49072165 >> 172.21.15.195:0/49072165 conn(0x7fd1dc0c9b00 msgr2=0x7fd1dc0c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:09:55.834 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.735+0000 7fd1e47b9700 1 -- 172.21.15.195:0/49072165 shutdown_connections 2022-06-27T12:09:55.835 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.735+0000 7fd1e47b9700 1 -- 172.21.15.195:0/49072165 wait complete. 2022-06-27T12:09:55.837 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.736+0000 7fd1e47b9700 1 Processor -- start 2022-06-27T12:09:55.838 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.736+0000 7fd1e47b9700 1 -- start start 2022-06-27T12:09:55.839 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.737+0000 7fd1e47b9700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd1dc06a450 0x7fd1dc154cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:55.842 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.737+0000 7fd1e47b9700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd1dc150500 0x7fd1dc159230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:55.842 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.737+0000 7fd1e47b9700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd1dc1508d0 0x7fd1dc0c81a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:55.843 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.737+0000 7fd1e47b9700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1dc1587e0 con 0x7fd1dc150500 2022-06-27T12:09:55.843 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.737+0000 7fd1e47b9700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1dc159770 con 0x7fd1dc1508d0 2022-06-27T12:09:55.874 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.737+0000 7fd1e47b9700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1dc0c0350 con 0x7fd1dc06a450 2022-06-27T12:09:55.874 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.737+0000 7fd1e1d54700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd1dc150500 0x7fd1dc159230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:55.875 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.737+0000 7fd1e1d54700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd1dc150500 0x7fd1dc159230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:49700/0 (socket says 172.21.15.195:49700) 2022-06-27T12:09:55.875 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.737+0000 7fd1e1d54700 1 -- 172.21.15.195:0/3295146564 learned_addr learned my addr 172.21.15.195:0/3295146564 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:09:55.876 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.737+0000 7fd1e2555700 1 --2- 172.21.15.195:0/3295146564 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd1dc06a450 0x7fd1dc154cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:55.876 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.737+0000 7fd1e2d56700 1 --2- 172.21.15.195:0/3295146564 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd1dc1508d0 0x7fd1dc0c81a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:55.876 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.737+0000 7fd1e1d54700 1 -- 172.21.15.195:0/3295146564 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd1dc06a450 msgr2=0x7fd1dc154cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:55.877 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.737+0000 7fd1e1d54700 1 --2- 172.21.15.195:0/3295146564 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd1dc06a450 0x7fd1dc154cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:55.877 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.737+0000 7fd1e1d54700 1 -- 172.21.15.195:0/3295146564 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd1dc1508d0 msgr2=0x7fd1dc0c81a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:55.878 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.737+0000 7fd1e1d54700 1 --2- 172.21.15.195:0/3295146564 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd1dc1508d0 0x7fd1dc0c81a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:55.878 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.737+0000 7fd1e1d54700 1 -- 172.21.15.195:0/3295146564 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1d4009a20 con 0x7fd1dc150500 2022-06-27T12:09:55.879 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.737+0000 7fd1e1d54700 1 --2- 172.21.15.195:0/3295146564 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd1dc150500 0x7fd1dc159230 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fd1d800c810 tx=0x7fd1d8000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:55.879 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.737+0000 7fd1ca7fc700 1 -- 172.21.15.195:0/3295146564 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1d8007360 con 0x7fd1dc150500 2022-06-27T12:09:55.879 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.738+0000 7fd1ca7fc700 1 -- 172.21.15.195:0/3295146564 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1d80074e0 con 0x7fd1dc150500 2022-06-27T12:09:55.880 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.738+0000 7fd1e47b9700 1 -- 172.21.15.195:0/3295146564 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1cc009290 con 0x7fd1dc150500 2022-06-27T12:09:55.880 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.738+0000 7fd1ca7fc700 1 -- 172.21.15.195:0/3295146564 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1d8004940 con 0x7fd1dc150500 2022-06-27T12:09:55.881 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.738+0000 7fd1e47b9700 1 -- 172.21.15.195:0/3295146564 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1dc167080 con 0x7fd1dc150500 2022-06-27T12:09:55.881 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.739+0000 7fd1e47b9700 1 -- 172.21.15.195:0/3295146564 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1dc0722c0 con 0x7fd1dc150500 2022-06-27T12:09:55.881 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.739+0000 7fd1ca7fc700 1 -- 172.21.15.195:0/3295146564 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fd1d801a030 con 0x7fd1dc150500 2022-06-27T12:09:55.882 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.739+0000 7fd1ca7fc700 1 --2- 172.21.15.195:0/3295146564 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd1bc063d30 0x7fd1bc0661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:55.882 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.740+0000 7fd1ca7fc700 1 -- 172.21.15.195:0/3295146564 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7234+0+0 (secure 0 0 0) 0x7fd1d8080080 con 0x7fd1dc150500 2022-06-27T12:09:55.883 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.740+0000 7fd1e2555700 1 --2- 172.21.15.195:0/3295146564 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd1bc063d30 0x7fd1bc0661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:55.884 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.740+0000 7fd1e2555700 1 --2- 172.21.15.195:0/3295146564 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd1bc063d30 0x7fd1bc0661e0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc009a70 tx=0x7fd1cc015000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:55.884 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.744+0000 7fd1ca7fc700 1 -- 172.21.15.195:0/3295146564 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fd1d804e890 con 0x7fd1dc150500 2022-06-27T12:09:55.971 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.967+0000 7fd1e47b9700 1 -- 172.21.15.195:0/3295146564 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd1dc064030 con 0x7fd1dc150500 2022-06-27T12:09:55.972 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.967+0000 7fd1ca7fc700 1 -- 172.21.15.195:0/3295146564 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+5371 (secure 0 0 0) 0x7fd1dc064030 con 0x7fd1dc150500 2022-06-27T12:09:55.973 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:09:55.973 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:09:51.320525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6708},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6708":{"gid":6708,"name":"b","rank":0,"incarnation":79,"state":"up:clientreplay","state_seq":24,"addr":"172.21.15.202:6833/702244802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":702244802},{"type":"v1","addr":"172.21.15.202:6833","nonce":702244802}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:09:55.977 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.969+0000 7fd1e47b9700 1 -- 172.21.15.195:0/3295146564 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd1bc063d30 msgr2=0x7fd1bc0661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:55.977 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.969+0000 7fd1e47b9700 1 --2- 172.21.15.195:0/3295146564 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd1bc063d30 0x7fd1bc0661e0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc009a70 tx=0x7fd1cc015000 comp rx=0 tx=0).stop 2022-06-27T12:09:55.978 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.969+0000 7fd1e47b9700 1 -- 172.21.15.195:0/3295146564 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd1dc150500 msgr2=0x7fd1dc159230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:55.978 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.969+0000 7fd1e47b9700 1 --2- 172.21.15.195:0/3295146564 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd1dc150500 0x7fd1dc159230 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fd1d800c810 tx=0x7fd1d8000ed0 comp rx=0 tx=0).stop 2022-06-27T12:09:55.979 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.969+0000 7fd1e47b9700 1 -- 172.21.15.195:0/3295146564 shutdown_connections 2022-06-27T12:09:55.980 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.969+0000 7fd1e47b9700 1 --2- 172.21.15.195:0/3295146564 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd1bc063d30 0x7fd1bc0661e0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:55.980 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.969+0000 7fd1e47b9700 1 --2- 172.21.15.195:0/3295146564 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd1dc1508d0 0x7fd1dc0c81a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:55.981 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.969+0000 7fd1e47b9700 1 --2- 172.21.15.195:0/3295146564 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd1dc06a450 0x7fd1dc154cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:55.981 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.969+0000 7fd1e47b9700 1 --2- 172.21.15.195:0/3295146564 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd1dc150500 0x7fd1dc159230 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:55.981 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.969+0000 7fd1e47b9700 1 -- 172.21.15.195:0/3295146564 >> 172.21.15.195:0/3295146564 conn(0x7fd1dc0c9b00 msgr2=0x7fd1dc0bbe70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:09:55.982 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.969+0000 7fd1e47b9700 1 -- 172.21.15.195:0/3295146564 shutdown_connections 2022-06-27T12:09:55.982 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:55.969+0000 7fd1e47b9700 1 -- 172.21.15.195:0/3295146564 wait complete. 2022-06-27T12:09:55.983 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 82 2022-06-27T12:09:55.986 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:09:58.002 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:09:58.186 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.147+0000 7fb6380a5700 1 Processor -- start 2022-06-27T12:09:58.203 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.148+0000 7fb6380a5700 1 -- start start 2022-06-27T12:09:58.204 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.149+0000 7fb6380a5700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fb630150580 0x7fb630156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:58.204 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.149+0000 7fb6380a5700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fb630150950 0x7fb6300c7b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:58.205 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.149+0000 7fb6380a5700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6300c76b0 con 0x7fb63006f620 2022-06-27T12:09:58.205 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.149+0000 7fb6380a5700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb630157430 con 0x7fb630150950 2022-06-27T12:09:58.206 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.149+0000 7fb6380a5700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6300bad70 con 0x7fb630150580 2022-06-27T12:09:58.206 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.149+0000 7fb635640700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fb630150950 0x7fb6300c7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:58.207 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.150+0000 7fb635640700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fb630150950 0x7fb6300c7b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:49714/0 (socket says 172.21.15.195:49714) 2022-06-27T12:09:58.207 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.150+0000 7fb635640700 1 -- 172.21.15.195:0/3163264319 learned_addr learned my addr 172.21.15.195:0/3163264319 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:09:58.207 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.150+0000 7fb636642700 1 --1- >> v1:172.21.15.202:6789/0 conn(0x7fb63006f620 0x7fb63006d6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.202:6789/0 says I am v1:172.21.15.195:57766/0 (socket says 172.21.15.195:57766) 2022-06-27T12:09:58.208 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.150+0000 7fb635e41700 1 --2- 172.21.15.195:0/3163264319 >> v2:172.21.15.202:3300/0 conn(0x7fb630150580 0x7fb630156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:58.208 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.150+0000 7fb634e3f700 1 -- 172.21.15.195:0/3163264319 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3921214746 0 0) 0x7fb6300c76b0 con 0x7fb63006f620 2022-06-27T12:09:58.209 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.150+0000 7fb634e3f700 1 -- 172.21.15.195:0/3163264319 --> v1:172.21.15.202:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb618003430 con 0x7fb63006f620 2022-06-27T12:09:58.225 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.150+0000 7fb635640700 1 -- 172.21.15.195:0/3163264319 >> v2:172.21.15.202:3300/0 conn(0x7fb630150580 msgr2=0x7fb630156ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:09:58.225 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.150+0000 7fb635640700 1 --2- 172.21.15.195:0/3163264319 >> v2:172.21.15.202:3300/0 conn(0x7fb630150580 0x7fb630156ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:58.226 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.150+0000 7fb635640700 1 -- 172.21.15.195:0/3163264319 >> v1:172.21.15.202:6789/0 conn(0x7fb63006f620 legacy=0x7fb63006d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:58.226 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.150+0000 7fb635640700 1 -- 172.21.15.195:0/3163264319 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6300ba4d0 con 0x7fb630150950 2022-06-27T12:09:58.226 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.150+0000 7fb635640700 1 --2- 172.21.15.195:0/3163264319 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb630150950 0x7fb6300c7b40 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fb6300c8470 tx=0x7fb620009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b887414f8d8759f3 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:58.227 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.151+0000 7fb634e3f700 1 -- 172.21.15.195:0/3163264319 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb620011630 con 0x7fb630150950 2022-06-27T12:09:58.227 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.151+0000 7fb634e3f700 1 -- 172.21.15.195:0/3163264319 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb620011790 con 0x7fb630150950 2022-06-27T12:09:58.227 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.151+0000 7fb634e3f700 1 -- 172.21.15.195:0/3163264319 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb620011b10 con 0x7fb630150950 2022-06-27T12:09:58.228 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.151+0000 7fb6380a5700 1 -- 172.21.15.195:0/3163264319 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb630150950 msgr2=0x7fb6300c7b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:58.228 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.151+0000 7fb6380a5700 1 --2- 172.21.15.195:0/3163264319 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb630150950 0x7fb6300c7b40 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fb6300c8470 tx=0x7fb620009120 comp rx=0 tx=0).stop 2022-06-27T12:09:58.229 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.151+0000 7fb6380a5700 1 -- 172.21.15.195:0/3163264319 shutdown_connections 2022-06-27T12:09:58.229 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.151+0000 7fb6380a5700 1 --2- 172.21.15.195:0/3163264319 >> v2:172.21.15.202:3300/0 conn(0x7fb630150580 0x7fb630156ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:58.229 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.151+0000 7fb6380a5700 1 --2- 172.21.15.195:0/3163264319 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb630150950 0x7fb6300c7b40 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:58.230 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.151+0000 7fb6380a5700 1 -- 172.21.15.195:0/3163264319 >> 172.21.15.195:0/3163264319 conn(0x7fb630063af0 msgr2=0x7fb630063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:09:58.230 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.151+0000 7fb6380a5700 1 -- 172.21.15.195:0/3163264319 shutdown_connections 2022-06-27T12:09:58.230 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.151+0000 7fb6380a5700 1 -- 172.21.15.195:0/3163264319 wait complete. 2022-06-27T12:09:58.231 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.152+0000 7fb6380a5700 1 Processor -- start 2022-06-27T12:09:58.231 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.152+0000 7fb6380a5700 1 -- start start 2022-06-27T12:09:58.232 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.153+0000 7fb6380a5700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb63006f620 0x7fb630148430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:58.232 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.153+0000 7fb6380a5700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb630150580 0x7fb630142960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:58.232 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.153+0000 7fb6380a5700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb630150950 0x7fb63014e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:58.261 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.153+0000 7fb6380a5700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6301515d0 con 0x7fb630150950 2022-06-27T12:09:58.261 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.153+0000 7fb6380a5700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb630155f20 con 0x7fb630150580 2022-06-27T12:09:58.261 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.153+0000 7fb6380a5700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb630156200 con 0x7fb63006f620 2022-06-27T12:09:58.262 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.153+0000 7fb636642700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb630150950 0x7fb63014e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:58.262 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.153+0000 7fb636642700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb630150950 0x7fb63014e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:49724/0 (socket says 172.21.15.195:49724) 2022-06-27T12:09:58.262 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.153+0000 7fb636642700 1 -- 172.21.15.195:0/2909501121 learned_addr learned my addr 172.21.15.195:0/2909501121 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:09:58.263 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.153+0000 7fb635e41700 1 --2- 172.21.15.195:0/2909501121 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb63006f620 0x7fb630148430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:58.287 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.153+0000 7fb635640700 1 --2- 172.21.15.195:0/2909501121 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb630150580 0x7fb630142960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:58.288 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.154+0000 7fb636642700 1 -- 172.21.15.195:0/2909501121 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb63006f620 msgr2=0x7fb630148430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:58.288 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.154+0000 7fb636642700 1 --2- 172.21.15.195:0/2909501121 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb63006f620 0x7fb630148430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:58.288 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.154+0000 7fb636642700 1 -- 172.21.15.195:0/2909501121 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb630150580 msgr2=0x7fb630142960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:58.288 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.154+0000 7fb636642700 1 --2- 172.21.15.195:0/2909501121 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb630150580 0x7fb630142960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:58.289 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.154+0000 7fb636642700 1 -- 172.21.15.195:0/2909501121 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6200093a0 con 0x7fb630150950 2022-06-27T12:09:58.289 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.154+0000 7fb635640700 1 --2- 172.21.15.195:0/2909501121 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb630150580 0x7fb630142960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T12:09:58.309 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.154+0000 7fb636642700 1 --2- 172.21.15.195:0/2909501121 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb630150950 0x7fb63014e690 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fb62800c790 tx=0x7fb6280015b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:58.310 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.154+0000 7fb61dffb700 1 -- 172.21.15.195:0/2909501121 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb628005cb0 con 0x7fb630150950 2022-06-27T12:09:58.310 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.154+0000 7fb61dffb700 1 -- 172.21.15.195:0/2909501121 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb628005e10 con 0x7fb630150950 2022-06-27T12:09:58.311 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.154+0000 7fb61dffb700 1 -- 172.21.15.195:0/2909501121 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb628007300 con 0x7fb630150950 2022-06-27T12:09:58.311 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.154+0000 7fb6380a5700 1 -- 172.21.15.195:0/2909501121 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6300c3840 con 0x7fb630150950 2022-06-27T12:09:58.312 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.154+0000 7fb6380a5700 1 -- 172.21.15.195:0/2909501121 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6300c3e00 con 0x7fb630150950 2022-06-27T12:09:58.312 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.155+0000 7fb6380a5700 1 -- 172.21.15.195:0/2909501121 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb63004ccb0 con 0x7fb630150950 2022-06-27T12:09:58.312 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.155+0000 7fb61dffb700 1 -- 172.21.15.195:0/2909501121 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fb628010070 con 0x7fb630150950 2022-06-27T12:09:58.313 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.156+0000 7fb61dffb700 1 --2- 172.21.15.195:0/2909501121 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb610063d00 0x7fb6100661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:09:58.313 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.156+0000 7fb61dffb700 1 -- 172.21.15.195:0/2909501121 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7234+0+0 (secure 0 0 0) 0x7fb628080820 con 0x7fb630150950 2022-06-27T12:09:58.322 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.156+0000 7fb635e41700 1 --2- 172.21.15.195:0/2909501121 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb610063d00 0x7fb6100661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:09:58.322 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.156+0000 7fb635e41700 1 --2- 172.21.15.195:0/2909501121 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb610063d00 0x7fb6100661b0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fb62c00ffd0 tx=0x7fb62c00f880 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:09:58.323 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.160+0000 7fb61dffb700 1 -- 172.21.15.195:0/2909501121 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fb62804eff0 con 0x7fb630150950 2022-06-27T12:09:58.450 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.449+0000 7fb6380a5700 1 -- 172.21.15.195:0/2909501121 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb6300722c0 con 0x7fb630150950 2022-06-27T12:09:58.451 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.450+0000 7fb61dffb700 1 -- 172.21.15.195:0/2909501121 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+5371 (secure 0 0 0) 0x7fb628052610 con 0x7fb630150950 2022-06-27T12:09:58.451 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:09:58.451 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:09:51.320525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6708},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6708":{"gid":6708,"name":"b","rank":0,"incarnation":79,"state":"up:clientreplay","state_seq":24,"addr":"172.21.15.202:6833/702244802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":702244802},{"type":"v1","addr":"172.21.15.202:6833","nonce":702244802}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:09:58.454 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.452+0000 7fb6380a5700 1 -- 172.21.15.195:0/2909501121 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb610063d00 msgr2=0x7fb6100661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:58.454 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.452+0000 7fb6380a5700 1 --2- 172.21.15.195:0/2909501121 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb610063d00 0x7fb6100661b0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fb62c00ffd0 tx=0x7fb62c00f880 comp rx=0 tx=0).stop 2022-06-27T12:09:58.455 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.452+0000 7fb6380a5700 1 -- 172.21.15.195:0/2909501121 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb630150950 msgr2=0x7fb63014e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:09:58.455 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.452+0000 7fb6380a5700 1 --2- 172.21.15.195:0/2909501121 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb630150950 0x7fb63014e690 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fb62800c790 tx=0x7fb6280015b0 comp rx=0 tx=0).stop 2022-06-27T12:09:58.456 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.452+0000 7fb6380a5700 1 -- 172.21.15.195:0/2909501121 shutdown_connections 2022-06-27T12:09:58.456 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.452+0000 7fb6380a5700 1 --2- 172.21.15.195:0/2909501121 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb610063d00 0x7fb6100661b0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:58.456 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.452+0000 7fb6380a5700 1 --2- 172.21.15.195:0/2909501121 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb630150950 0x7fb63014e690 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:58.456 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.452+0000 7fb6380a5700 1 --2- 172.21.15.195:0/2909501121 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb63006f620 0x7fb630148430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:58.457 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.452+0000 7fb6380a5700 1 --2- 172.21.15.195:0/2909501121 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb630150580 0x7fb630142960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:09:58.457 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.452+0000 7fb6380a5700 1 -- 172.21.15.195:0/2909501121 >> 172.21.15.195:0/2909501121 conn(0x7fb630063af0 msgr2=0x7fb63006d500 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:09:58.457 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.452+0000 7fb6380a5700 1 -- 172.21.15.195:0/2909501121 shutdown_connections 2022-06-27T12:09:58.457 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:09:58.452+0000 7fb6380a5700 1 -- 172.21.15.195:0/2909501121 wait complete. 2022-06-27T12:09:58.458 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 82 2022-06-27T12:09:58.471 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:10:00.474 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:10:00.624 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.623+0000 7f7422d3c700 1 Processor -- start 2022-06-27T12:10:00.624 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.623+0000 7f7422d3c700 1 -- start start 2022-06-27T12:10:00.631 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.624+0000 7f7422d3c700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f741c06f640 0x7f741c156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:00.632 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.624+0000 7f7422d3c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f741c0c39e0 0x7f741c1535a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:00.632 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.624+0000 7f7422d3c700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f741c0c1a20 con 0x7f741c0c4160 2022-06-27T12:10:00.632 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.624+0000 7f7422d3c700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f741c0c1130 con 0x7f741c0c39e0 2022-06-27T12:10:00.632 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.624+0000 7f7422d3c700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f741c06ea90 con 0x7f741c06f640 2022-06-27T12:10:00.633 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.624+0000 7f74212d9700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f741c0c39e0 0x7f741c1535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:00.633 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.624+0000 7f741bfff700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f741c0c4160 0x7f741c0c1eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:55924/0 (socket says 172.21.15.195:55924) 2022-06-27T12:10:00.633 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.624+0000 7f74212d9700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f741c0c39e0 0x7f741c1535a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:49740/0 (socket says 172.21.15.195:49740) 2022-06-27T12:10:00.634 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.624+0000 7f741bfff700 1 -- 172.21.15.195:0/1790167397 learned_addr learned my addr 172.21.15.195:0/1790167397 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:10:00.640 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.625+0000 7f74212d9700 1 -- 172.21.15.195:0/1790167397 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f741c06f640 msgr2=0x7f741c156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:10:00.647 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.625+0000 7f7420ad8700 1 --2- 172.21.15.195:0/1790167397 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f741c06f640 0x7f741c156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:00.647 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.625+0000 7f74212d9700 1 --2- 172.21.15.195:0/1790167397 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f741c06f640 0x7f741c156ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:00.647 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.625+0000 7f74212d9700 1 -- 172.21.15.195:0/1790167397 >> v1:172.21.15.195:6789/0 conn(0x7f741c0c4160 legacy=0x7f741c0c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:00.648 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.625+0000 7f741b7fe700 1 -- 172.21.15.195:0/1790167397 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1910318209 0 0) 0x7f741c0c1a20 con 0x7f741c0c4160 2022-06-27T12:10:00.650 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.625+0000 7f74212d9700 1 -- 172.21.15.195:0/1790167397 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f741c06ed70 con 0x7f741c0c39e0 2022-06-27T12:10:00.650 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.625+0000 7f7420ad8700 1 --2- 172.21.15.195:0/1790167397 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f741c06f640 0x7f741c156ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T12:10:00.650 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.625+0000 7f74212d9700 1 --2- 172.21.15.195:0/1790167397 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f741c0c39e0 0x7f741c1535a0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f7414009e60 tx=0x7f7414009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee7d67f71c7904c server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:00.651 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.625+0000 7f741b7fe700 1 -- 172.21.15.195:0/1790167397 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f741400e070 con 0x7f741c0c39e0 2022-06-27T12:10:00.651 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.625+0000 7f741b7fe700 1 -- 172.21.15.195:0/1790167397 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f741400bcc0 con 0x7f741c0c39e0 2022-06-27T12:10:00.651 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.625+0000 7f741b7fe700 1 -- 172.21.15.195:0/1790167397 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7414012680 con 0x7f741c0c39e0 2022-06-27T12:10:00.666 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.625+0000 7f7422d3c700 1 -- 172.21.15.195:0/1790167397 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f741c0c39e0 msgr2=0x7f741c1535a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:00.667 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.625+0000 7f7422d3c700 1 --2- 172.21.15.195:0/1790167397 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f741c0c39e0 0x7f741c1535a0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f7414009e60 tx=0x7f7414009780 comp rx=0 tx=0).stop 2022-06-27T12:10:00.667 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.626+0000 7f7422d3c700 1 -- 172.21.15.195:0/1790167397 shutdown_connections 2022-06-27T12:10:00.668 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.626+0000 7f7422d3c700 1 --2- 172.21.15.195:0/1790167397 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f741c0c39e0 0x7f741c1535a0 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:00.668 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.626+0000 7f7422d3c700 1 --2- 172.21.15.195:0/1790167397 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f741c06f640 0x7f741c156ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:00.668 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.626+0000 7f7422d3c700 1 -- 172.21.15.195:0/1790167397 >> 172.21.15.195:0/1790167397 conn(0x7f741c0c6350 msgr2=0x7f741c0c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:10:00.668 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.626+0000 7f7422d3c700 1 -- 172.21.15.195:0/1790167397 shutdown_connections 2022-06-27T12:10:00.669 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.626+0000 7f7422d3c700 1 -- 172.21.15.195:0/1790167397 wait complete. 2022-06-27T12:10:00.676 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.627+0000 7f7422d3c700 1 Processor -- start 2022-06-27T12:10:00.676 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.627+0000 7f7422d3c700 1 -- start start 2022-06-27T12:10:00.698 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.628+0000 7f7422d3c700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f741c06f640 0x7f741c151a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:00.699 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.628+0000 7f7422d3c700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f741c0c39e0 0x7f741c16a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:00.700 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.628+0000 7f7422d3c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f741c0c4160 0x7f741c14fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:00.700 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.628+0000 7f7422d3c700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f741c0bf1b0 con 0x7f741c0c4160 2022-06-27T12:10:00.700 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.628+0000 7f7422d3c700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f741c1550c0 con 0x7f741c0c39e0 2022-06-27T12:10:00.701 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.628+0000 7f7422d3c700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f741c0bfc20 con 0x7f741c06f640 2022-06-27T12:10:00.701 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.628+0000 7f74212d9700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f741c0c4160 0x7f741c14fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:00.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.628+0000 7f74212d9700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f741c0c4160 0x7f741c14fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:49746/0 (socket says 172.21.15.195:49746) 2022-06-27T12:10:00.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.628+0000 7f74212d9700 1 -- 172.21.15.195:0/3277946955 learned_addr learned my addr 172.21.15.195:0/3277946955 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:10:00.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.628+0000 7f7420ad8700 1 --2- 172.21.15.195:0/3277946955 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f741c06f640 0x7f741c151a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:00.705 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.628+0000 7f741bfff700 1 --2- 172.21.15.195:0/3277946955 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f741c0c39e0 0x7f741c16a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:00.706 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.628+0000 7f74212d9700 1 -- 172.21.15.195:0/3277946955 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f741c06f640 msgr2=0x7f741c151a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:00.706 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.628+0000 7f74212d9700 1 --2- 172.21.15.195:0/3277946955 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f741c06f640 0x7f741c151a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:00.708 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.628+0000 7f74212d9700 1 -- 172.21.15.195:0/3277946955 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f741c0c39e0 msgr2=0x7f741c16a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:00.708 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.628+0000 7f74212d9700 1 --2- 172.21.15.195:0/3277946955 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f741c0c39e0 0x7f741c16a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:00.708 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.628+0000 7f74212d9700 1 -- 172.21.15.195:0/3277946955 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7414009a20 con 0x7f741c0c4160 2022-06-27T12:10:00.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.628+0000 7f74212d9700 1 --2- 172.21.15.195:0/3277946955 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f741c0c4160 0x7f741c14fb80 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f741400b6c0 tx=0x7f7414019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:00.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.628+0000 7f7418ff9700 1 -- 172.21.15.195:0/3277946955 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7414003b80 con 0x7f741c0c4160 2022-06-27T12:10:00.730 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.628+0000 7f7418ff9700 1 -- 172.21.15.195:0/3277946955 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7414003ce0 con 0x7f741c0c4160 2022-06-27T12:10:00.731 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.628+0000 7f7418ff9700 1 -- 172.21.15.195:0/3277946955 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7414004290 con 0x7f741c0c4160 2022-06-27T12:10:00.731 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.628+0000 7f7422d3c700 1 -- 172.21.15.195:0/3277946955 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f741c155550 con 0x7f741c0c4160 2022-06-27T12:10:00.731 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.628+0000 7f7422d3c700 1 -- 172.21.15.195:0/3277946955 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f741c1544a0 con 0x7f741c0c4160 2022-06-27T12:10:00.732 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.629+0000 7f7422d3c700 1 -- 172.21.15.195:0/3277946955 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f741c0722c0 con 0x7f741c0c4160 2022-06-27T12:10:00.732 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.630+0000 7f7418ff9700 1 -- 172.21.15.195:0/3277946955 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f7414004600 con 0x7f741c0c4160 2022-06-27T12:10:00.732 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.630+0000 7f7418ff9700 1 --2- 172.21.15.195:0/3277946955 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7404063d00 0x7f74040661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:00.732 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.630+0000 7f7418ff9700 1 -- 172.21.15.195:0/3277946955 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7234+0+0 (secure 0 0 0) 0x7f7414083830 con 0x7f741c0c4160 2022-06-27T12:10:00.733 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.630+0000 7f7420ad8700 1 --2- 172.21.15.195:0/3277946955 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7404063d00 0x7f74040661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:00.733 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.631+0000 7f7420ad8700 1 --2- 172.21.15.195:0/3277946955 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7404063d00 0x7f74040661b0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f741000aa40 tx=0x7f7410008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:00.734 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.634+0000 7f7418ff9700 1 -- 172.21.15.195:0/3277946955 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f7414051f80 con 0x7f741c0c4160 2022-06-27T12:10:00.862 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.861+0000 7f7422d3c700 1 -- 172.21.15.195:0/3277946955 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f741c06f240 con 0x7f741c0c4160 2022-06-27T12:10:00.863 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.862+0000 7f7418ff9700 1 -- 172.21.15.195:0/3277946955 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+5365 (secure 0 0 0) 0x7f741402c020 con 0x7f741c0c4160 2022-06-27T12:10:00.864 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:10:00.864 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":83,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:09:59.334885+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6708},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6708":{"gid":6708,"name":"b","rank":0,"incarnation":79,"state":"up:active","state_seq":27,"addr":"172.21.15.202:6833/702244802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":702244802},{"type":"v1","addr":"172.21.15.202:6833","nonce":702244802}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:10:00.913 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.864+0000 7f7422d3c700 1 -- 172.21.15.195:0/3277946955 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7404063d00 msgr2=0x7f74040661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:00.914 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.864+0000 7f7422d3c700 1 --2- 172.21.15.195:0/3277946955 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7404063d00 0x7f74040661b0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f741000aa40 tx=0x7f7410008040 comp rx=0 tx=0).stop 2022-06-27T12:10:00.914 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.864+0000 7f7422d3c700 1 -- 172.21.15.195:0/3277946955 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f741c0c4160 msgr2=0x7f741c14fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:00.915 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.864+0000 7f7422d3c700 1 --2- 172.21.15.195:0/3277946955 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f741c0c4160 0x7f741c14fb80 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f741400b6c0 tx=0x7f7414019000 comp rx=0 tx=0).stop 2022-06-27T12:10:00.915 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.864+0000 7f7422d3c700 1 -- 172.21.15.195:0/3277946955 shutdown_connections 2022-06-27T12:10:00.916 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.864+0000 7f7422d3c700 1 --2- 172.21.15.195:0/3277946955 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7404063d00 0x7f74040661b0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:00.916 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.864+0000 7f7422d3c700 1 --2- 172.21.15.195:0/3277946955 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f741c0c4160 0x7f741c14fb80 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:00.917 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.864+0000 7f7422d3c700 1 --2- 172.21.15.195:0/3277946955 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f741c06f640 0x7f741c151a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:00.918 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.864+0000 7f7422d3c700 1 --2- 172.21.15.195:0/3277946955 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f741c0c39e0 0x7f741c16a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:00.918 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.864+0000 7f7422d3c700 1 -- 172.21.15.195:0/3277946955 >> 172.21.15.195:0/3277946955 conn(0x7f741c0c6350 msgr2=0x7f741c153370 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:10:00.919 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.864+0000 7f7422d3c700 1 -- 172.21.15.195:0/3277946955 shutdown_connections 2022-06-27T12:10:00.919 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:00.864+0000 7f7422d3c700 1 -- 172.21.15.195:0/3277946955 wait complete. 2022-06-27T12:10:00.920 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 83 2022-06-27T12:10:00.922 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=6271 2022-06-27T12:10:00.924 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 3 secs before reviving mds.d 2022-06-27T12:10:02.555 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:10:02.583 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:10:03.946 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.d 2022-06-27T12:10:03.947 INFO:tasks.ceph.mds.d:Restarting daemon 2022-06-27T12:10:03.947 DEBUG:teuthology.orchestra.run.smithi202:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2022-06-27T12:10:03.961 INFO:tasks.ceph.mds.d:Started 2022-06-27T12:10:03.962 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:10:04.048 INFO:tasks.ceph.mds.d.smithi202.stdout:starting mds.d at 2022-06-27T12:10:04.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.097+0000 7fca90a9b700 1 Processor -- start 2022-06-27T12:10:04.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.097+0000 7fca90a9b700 1 -- start start 2022-06-27T12:10:04.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.098+0000 7fca90a9b700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fca8c150580 0x7fca8c156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:04.155 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.098+0000 7fca90a9b700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fca8c06f620 0x7fca8c06d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:04.155 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.098+0000 7fca90a9b700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca8c0c76b0 con 0x7fca8c150950 2022-06-27T12:10:04.156 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.098+0000 7fca90a9b700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca8c157430 con 0x7fca8c06f620 2022-06-27T12:10:04.156 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.098+0000 7fca90a9b700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca8c0c8250 con 0x7fca8c150580 2022-06-27T12:10:04.157 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.098+0000 7fca89d9b700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fca8c150950 0x7fca8c0c7b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:55950/0 (socket says 172.21.15.195:55950) 2022-06-27T12:10:04.157 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.098+0000 7fca89d9b700 1 -- 172.21.15.195:0/3175329354 learned_addr learned my addr 172.21.15.195:0/3175329354 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:10:04.158 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.098+0000 7fca8ad9d700 1 --2- 172.21.15.195:0/3175329354 >> v2:172.21.15.202:3300/0 conn(0x7fca8c06f620 0x7fca8c06d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:04.158 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.098+0000 7fca8a59c700 1 --2- 172.21.15.195:0/3175329354 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fca8c150580 0x7fca8c156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:04.159 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.099+0000 7fca8959a700 1 -- 172.21.15.195:0/3175329354 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1777111623 0 0) 0x7fca8c0c76b0 con 0x7fca8c150950 2022-06-27T12:10:04.159 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.099+0000 7fca8959a700 1 -- 172.21.15.195:0/3175329354 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca70003430 con 0x7fca8c150950 2022-06-27T12:10:04.160 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.099+0000 7fca8959a700 1 -- 172.21.15.195:0/3175329354 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3351095721 0 0) 0x7fca70003430 con 0x7fca8c150950 2022-06-27T12:10:04.160 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.099+0000 7fca8ad9d700 1 -- 172.21.15.195:0/3175329354 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fca8c150580 msgr2=0x7fca8c156ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:10:04.161 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.099+0000 7fca8ad9d700 1 --2- 172.21.15.195:0/3175329354 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fca8c150580 0x7fca8c156ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:04.163 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.099+0000 7fca8ad9d700 1 -- 172.21.15.195:0/3175329354 >> v1:172.21.15.195:6789/0 conn(0x7fca8c150950 legacy=0x7fca8c0c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:04.163 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.099+0000 7fca8ad9d700 1 -- 172.21.15.195:0/3175329354 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca8c0bad70 con 0x7fca8c06f620 2022-06-27T12:10:04.164 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.099+0000 7fca8a59c700 1 --2- 172.21.15.195:0/3175329354 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fca8c150580 0x7fca8c156ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:10:04.164 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.099+0000 7fca8ad9d700 1 --2- 172.21.15.195:0/3175329354 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fca8c06f620 0x7fca8c06d6a0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fca84009e90 tx=0x7fca84009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d8d8137ec0730ea1 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:04.164 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.099+0000 7fca8959a700 1 -- 172.21.15.195:0/3175329354 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca8400e070 con 0x7fca8c06f620 2022-06-27T12:10:04.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.099+0000 7fca8959a700 1 -- 172.21.15.195:0/3175329354 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca840038a0 con 0x7fca8c06f620 2022-06-27T12:10:04.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.099+0000 7fca8959a700 1 -- 172.21.15.195:0/3175329354 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca7c001470 con 0x7fca8c06f620 2022-06-27T12:10:04.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.100+0000 7fca90a9b700 1 -- 172.21.15.195:0/3175329354 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fca8c06f620 msgr2=0x7fca8c06d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:04.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.100+0000 7fca90a9b700 1 --2- 172.21.15.195:0/3175329354 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fca8c06f620 0x7fca8c06d6a0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fca84009e90 tx=0x7fca84009780 comp rx=0 tx=0).stop 2022-06-27T12:10:04.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.100+0000 7fca90a9b700 1 -- 172.21.15.195:0/3175329354 shutdown_connections 2022-06-27T12:10:04.166 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.100+0000 7fca90a9b700 1 --2- 172.21.15.195:0/3175329354 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fca8c06f620 0x7fca8c06d6a0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:04.166 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.100+0000 7fca90a9b700 1 --2- 172.21.15.195:0/3175329354 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fca8c150580 0x7fca8c156ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:04.166 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.100+0000 7fca90a9b700 1 -- 172.21.15.195:0/3175329354 >> 172.21.15.195:0/3175329354 conn(0x7fca8c063af0 msgr2=0x7fca8c063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:10:04.167 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.100+0000 7fca90a9b700 1 -- 172.21.15.195:0/3175329354 shutdown_connections 2022-06-27T12:10:04.174 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.100+0000 7fca90a9b700 1 -- 172.21.15.195:0/3175329354 wait complete. 2022-06-27T12:10:04.174 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.101+0000 7fca90a9b700 1 Processor -- start 2022-06-27T12:10:04.175 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.101+0000 7fca90a9b700 1 -- start start 2022-06-27T12:10:04.176 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.102+0000 7fca90a9b700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fca8c06f620 0x7fca8c0c4a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:04.179 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.102+0000 7fca90a9b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fca8c150580 0x7fca8c0be7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:04.179 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.102+0000 7fca90a9b700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fca8c150950 0x7fca8c0c2d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:04.180 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.102+0000 7fca90a9b700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca8c0bc880 con 0x7fca8c150580 2022-06-27T12:10:04.180 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.102+0000 7fca90a9b700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca8c0bbfe0 con 0x7fca8c150950 2022-06-27T12:10:04.180 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.102+0000 7fca90a9b700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca8c0be190 con 0x7fca8c06f620 2022-06-27T12:10:04.181 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.102+0000 7fca89d9b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fca8c150580 0x7fca8c0be7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:04.181 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.102+0000 7fca89d9b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fca8c150580 0x7fca8c0be7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:49770/0 (socket says 172.21.15.195:49770) 2022-06-27T12:10:04.182 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.102+0000 7fca89d9b700 1 -- 172.21.15.195:0/3253076014 learned_addr learned my addr 172.21.15.195:0/3253076014 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:10:04.183 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.102+0000 7fca8a59c700 1 --2- 172.21.15.195:0/3253076014 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fca8c06f620 0x7fca8c0c4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:04.184 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.102+0000 7fca8ad9d700 1 --2- 172.21.15.195:0/3253076014 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fca8c150950 0x7fca8c0c2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:04.184 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.102+0000 7fca89d9b700 1 -- 172.21.15.195:0/3253076014 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fca8c06f620 msgr2=0x7fca8c0c4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:04.185 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.102+0000 7fca89d9b700 1 --2- 172.21.15.195:0/3253076014 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fca8c06f620 0x7fca8c0c4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:04.185 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.102+0000 7fca89d9b700 1 -- 172.21.15.195:0/3253076014 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fca8c150950 msgr2=0x7fca8c0c2d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:04.185 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.102+0000 7fca89d9b700 1 --2- 172.21.15.195:0/3253076014 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fca8c150950 0x7fca8c0c2d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:04.185 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.102+0000 7fca89d9b700 1 -- 172.21.15.195:0/3253076014 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca800092e0 con 0x7fca8c150580 2022-06-27T12:10:04.195 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.102+0000 7fca89d9b700 1 --2- 172.21.15.195:0/3253076014 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fca8c150580 0x7fca8c0be7b0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fca7c00c8e0 tx=0x7fca7c001b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:04.195 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.102+0000 7fca7a7fc700 1 -- 172.21.15.195:0/3253076014 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca7c007360 con 0x7fca8c150580 2022-06-27T12:10:04.195 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.102+0000 7fca7a7fc700 1 -- 172.21.15.195:0/3253076014 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca7c0074c0 con 0x7fca8c150580 2022-06-27T12:10:04.196 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.102+0000 7fca90a9b700 1 -- 172.21.15.195:0/3253076014 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca84009a20 con 0x7fca8c150580 2022-06-27T12:10:04.196 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.102+0000 7fca7a7fc700 1 -- 172.21.15.195:0/3253076014 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca7c004920 con 0x7fca8c150580 2022-06-27T12:10:04.196 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.102+0000 7fca90a9b700 1 -- 172.21.15.195:0/3253076014 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca8c0bba20 con 0x7fca8c150580 2022-06-27T12:10:04.196 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.103+0000 7fca90a9b700 1 -- 172.21.15.195:0/3253076014 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca8c0722c0 con 0x7fca8c150580 2022-06-27T12:10:04.197 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.104+0000 7fca7a7fc700 1 -- 172.21.15.195:0/3253076014 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fca7c01a030 con 0x7fca8c150580 2022-06-27T12:10:04.206 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.104+0000 7fca7a7fc700 1 --2- 172.21.15.195:0/3253076014 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fca68063d30 0x7fca680661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:04.207 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.104+0000 7fca7a7fc700 1 -- 172.21.15.195:0/3253076014 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7234+0+0 (secure 0 0 0) 0x7fca7c080040 con 0x7fca8c150580 2022-06-27T12:10:04.212 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.104+0000 7fca8a59c700 1 --2- 172.21.15.195:0/3253076014 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fca68063d30 0x7fca680661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:04.212 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.105+0000 7fca8a59c700 1 --2- 172.21.15.195:0/3253076014 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fca68063d30 0x7fca680661e0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fca80009e00 tx=0x7fca80009060 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:04.213 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.109+0000 7fca7a7fc700 1 -- 172.21.15.195:0/3253076014 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fca7c04ac90 con 0x7fca8c150580 2022-06-27T12:10:04.340 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.335+0000 7fca90a9b700 1 -- 172.21.15.195:0/3253076014 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fca8c069c90 con 0x7fca8c150580 2022-06-27T12:10:04.341 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.335+0000 7fca7a7fc700 1 -- 172.21.15.195:0/3253076014 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+5365 (secure 0 0 0) 0x7fca7c018070 con 0x7fca8c150580 2022-06-27T12:10:04.341 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:10:04.342 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":83,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:09:59.334885+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6708},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6708":{"gid":6708,"name":"b","rank":0,"incarnation":79,"state":"up:active","state_seq":27,"addr":"172.21.15.202:6833/702244802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":702244802},{"type":"v1","addr":"172.21.15.202:6833","nonce":702244802}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:10:04.343 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.337+0000 7fca90a9b700 1 -- 172.21.15.195:0/3253076014 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fca68063d30 msgr2=0x7fca680661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:04.343 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.337+0000 7fca90a9b700 1 --2- 172.21.15.195:0/3253076014 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fca68063d30 0x7fca680661e0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fca80009e00 tx=0x7fca80009060 comp rx=0 tx=0).stop 2022-06-27T12:10:04.343 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.337+0000 7fca90a9b700 1 -- 172.21.15.195:0/3253076014 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fca8c150580 msgr2=0x7fca8c0be7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:04.344 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.337+0000 7fca90a9b700 1 --2- 172.21.15.195:0/3253076014 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fca8c150580 0x7fca8c0be7b0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fca7c00c8e0 tx=0x7fca7c001b80 comp rx=0 tx=0).stop 2022-06-27T12:10:04.344 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.337+0000 7fca90a9b700 1 -- 172.21.15.195:0/3253076014 shutdown_connections 2022-06-27T12:10:04.344 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.337+0000 7fca90a9b700 1 --2- 172.21.15.195:0/3253076014 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fca68063d30 0x7fca680661e0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:04.344 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.337+0000 7fca90a9b700 1 --2- 172.21.15.195:0/3253076014 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fca8c150950 0x7fca8c0c2d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:04.345 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.337+0000 7fca90a9b700 1 --2- 172.21.15.195:0/3253076014 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fca8c06f620 0x7fca8c0c4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:04.345 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.337+0000 7fca90a9b700 1 --2- 172.21.15.195:0/3253076014 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fca8c150580 0x7fca8c0be7b0 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:04.345 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.337+0000 7fca90a9b700 1 -- 172.21.15.195:0/3253076014 >> 172.21.15.195:0/3253076014 conn(0x7fca8c063af0 msgr2=0x7fca8c14d880 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:10:04.345 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.337+0000 7fca90a9b700 1 -- 172.21.15.195:0/3253076014 shutdown_connections 2022-06-27T12:10:04.350 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:04.337+0000 7fca90a9b700 1 -- 172.21.15.195:0/3253076014 wait complete. 2022-06-27T12:10:04.350 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 83 2022-06-27T12:10:04.353 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2022-06-27T12:10:06.118 INFO:tasks.ceph.mds.b.smithi202.stderr:2022-06-27T12:10:06.117+0000 7f7ee384c700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2022-06-27T12:10:06.362 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:10:06.514 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.500+0000 7fe6dc47f700 1 Processor -- start 2022-06-27T12:10:06.515 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.500+0000 7fe6dc47f700 1 -- start start 2022-06-27T12:10:06.515 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.501+0000 7fe6dc47f700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fe6d406e850 0x7fe6d41570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:06.516 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.501+0000 7fe6dc47f700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fe6d406fef0 0x7fe6d4069de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:06.516 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.501+0000 7fe6dc47f700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6d406fba0 con 0x7fe6d40c34a0 2022-06-27T12:10:06.517 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.501+0000 7fe6dc47f700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6d41575e0 con 0x7fe6d406e850 2022-06-27T12:10:06.517 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.501+0000 7fe6dc47f700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6d406f5e0 con 0x7fe6d406fef0 2022-06-27T12:10:06.517 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.501+0000 7fe6da21b700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fe6d406e850 0x7fe6d41570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:06.518 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.501+0000 7fe6daa1c700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fe6d40c34a0 0x7fe6d40c2920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:55968/0 (socket says 172.21.15.195:55968) 2022-06-27T12:10:06.519 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.501+0000 7fe6daa1c700 1 -- 172.21.15.195:0/3814729872 learned_addr learned my addr 172.21.15.195:0/3814729872 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:10:06.519 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.502+0000 7fe6d9a1a700 1 --2- 172.21.15.195:0/3814729872 >> v2:172.21.15.202:3300/0 conn(0x7fe6d406fef0 0x7fe6d4069de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:06.520 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.502+0000 7fe6da21b700 1 -- 172.21.15.195:0/3814729872 >> v2:172.21.15.202:3300/0 conn(0x7fe6d406fef0 msgr2=0x7fe6d4069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:10:06.520 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.502+0000 7fe6da21b700 1 --2- 172.21.15.195:0/3814729872 >> v2:172.21.15.202:3300/0 conn(0x7fe6d406fef0 0x7fe6d4069de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:06.520 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.502+0000 7fe6da21b700 1 -- 172.21.15.195:0/3814729872 >> v1:172.21.15.195:6789/0 conn(0x7fe6d40c34a0 legacy=0x7fe6d40c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:06.521 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.502+0000 7fe6da21b700 1 -- 172.21.15.195:0/3814729872 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6d406f8c0 con 0x7fe6d406e850 2022-06-27T12:10:06.522 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.502+0000 7fe6d9219700 1 -- 172.21.15.195:0/3814729872 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 291663541 0 0) 0x7fe6d406fba0 con 0x7fe6d40c34a0 2022-06-27T12:10:06.522 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.502+0000 7fe6da21b700 1 --2- 172.21.15.195:0/3814729872 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe6d406e850 0x7fe6d41570a0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fe6d406e600 tx=0x7fe6c4009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e6d38924f870b5d server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:06.524 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.502+0000 7fe6d9219700 1 -- 172.21.15.195:0/3814729872 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c4011630 con 0x7fe6d406e850 2022-06-27T12:10:06.524 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.502+0000 7fe6d9219700 1 -- 172.21.15.195:0/3814729872 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6c4011790 con 0x7fe6d406e850 2022-06-27T12:10:06.524 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.502+0000 7fe6d9219700 1 -- 172.21.15.195:0/3814729872 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c4011b30 con 0x7fe6d406e850 2022-06-27T12:10:06.525 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.502+0000 7fe6dc47f700 1 -- 172.21.15.195:0/3814729872 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe6d406e850 msgr2=0x7fe6d41570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:06.525 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.502+0000 7fe6dc47f700 1 --2- 172.21.15.195:0/3814729872 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe6d406e850 0x7fe6d41570a0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fe6d406e600 tx=0x7fe6c4009060 comp rx=0 tx=0).stop 2022-06-27T12:10:06.526 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.503+0000 7fe6dc47f700 1 -- 172.21.15.195:0/3814729872 shutdown_connections 2022-06-27T12:10:06.526 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.503+0000 7fe6dc47f700 1 --2- 172.21.15.195:0/3814729872 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe6d406e850 0x7fe6d41570a0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:06.526 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.503+0000 7fe6dc47f700 1 --2- 172.21.15.195:0/3814729872 >> v2:172.21.15.202:3300/0 conn(0x7fe6d406fef0 0x7fe6d4069de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:06.527 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.503+0000 7fe6dc47f700 1 -- 172.21.15.195:0/3814729872 >> 172.21.15.195:0/3814729872 conn(0x7fe6d40c6350 msgr2=0x7fe6d40c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:10:06.527 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.503+0000 7fe6dc47f700 1 -- 172.21.15.195:0/3814729872 shutdown_connections 2022-06-27T12:10:06.528 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.503+0000 7fe6dc47f700 1 -- 172.21.15.195:0/3814729872 wait complete. 2022-06-27T12:10:06.528 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.504+0000 7fe6dc47f700 1 Processor -- start 2022-06-27T12:10:06.528 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.504+0000 7fe6dc47f700 1 -- start start 2022-06-27T12:10:06.529 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.504+0000 7fe6dc47f700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe6d406e850 0x7fe6d414ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:06.529 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.504+0000 7fe6dc47f700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe6d40c34a0 0x7fe6d40c8380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:06.529 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.504+0000 7fe6dc47f700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe6d414db40 0x7fe6d414dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:06.530 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.504+0000 7fe6dc47f700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d414f510 con 0x7fe6d40c34a0 2022-06-27T12:10:06.530 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.504+0000 7fe6dc47f700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d414d550 con 0x7fe6d414db40 2022-06-27T12:10:06.531 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.504+0000 7fe6dc47f700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d414d270 con 0x7fe6d406e850 2022-06-27T12:10:06.531 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.504+0000 7fe6d9a1a700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe6d40c34a0 0x7fe6d40c8380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:06.531 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.505+0000 7fe6d9a1a700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe6d40c34a0 0x7fe6d40c8380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:49788/0 (socket says 172.21.15.195:49788) 2022-06-27T12:10:06.550 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.505+0000 7fe6d9a1a700 1 -- 172.21.15.195:0/3261336809 learned_addr learned my addr 172.21.15.195:0/3261336809 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:10:06.550 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.505+0000 7fe6daa1c700 1 --2- 172.21.15.195:0/3261336809 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe6d414db40 0x7fe6d414dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:06.551 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.505+0000 7fe6da21b700 1 --2- 172.21.15.195:0/3261336809 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe6d406e850 0x7fe6d414ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:06.552 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.505+0000 7fe6d9a1a700 1 -- 172.21.15.195:0/3261336809 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe6d406e850 msgr2=0x7fe6d414ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:06.552 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.505+0000 7fe6d9a1a700 1 --2- 172.21.15.195:0/3261336809 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe6d406e850 0x7fe6d414ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:06.553 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.505+0000 7fe6d9a1a700 1 -- 172.21.15.195:0/3261336809 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe6d414db40 msgr2=0x7fe6d414dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:06.553 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.505+0000 7fe6d9a1a700 1 --2- 172.21.15.195:0/3261336809 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe6d414db40 0x7fe6d414dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:06.553 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.505+0000 7fe6d9a1a700 1 -- 172.21.15.195:0/3261336809 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6c40092e0 con 0x7fe6d40c34a0 2022-06-27T12:10:06.554 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.505+0000 7fe6d9a1a700 1 --2- 172.21.15.195:0/3261336809 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe6d40c34a0 0x7fe6d40c8380 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7fe6d000d890 tx=0x7fe6d0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:06.554 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.505+0000 7fe6c27fc700 1 -- 172.21.15.195:0/3261336809 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d00125f0 con 0x7fe6d40c34a0 2022-06-27T12:10:06.554 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.505+0000 7fe6c27fc700 1 -- 172.21.15.195:0/3261336809 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6d0012750 con 0x7fe6d40c34a0 2022-06-27T12:10:06.554 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.505+0000 7fe6c27fc700 1 -- 172.21.15.195:0/3261336809 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d0012b10 con 0x7fe6d40c34a0 2022-06-27T12:10:06.555 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.505+0000 7fe6dc47f700 1 -- 172.21.15.195:0/3261336809 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6d414e4f0 con 0x7fe6d40c34a0 2022-06-27T12:10:06.555 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.505+0000 7fe6dc47f700 1 -- 172.21.15.195:0/3261336809 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6d414c6f0 con 0x7fe6d40c34a0 2022-06-27T12:10:06.555 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.506+0000 7fe6dc47f700 1 -- 172.21.15.195:0/3261336809 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6d414d6e0 con 0x7fe6d40c34a0 2022-06-27T12:10:06.555 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.507+0000 7fe6c27fc700 1 -- 172.21.15.195:0/3261336809 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fe6d001e460 con 0x7fe6d40c34a0 2022-06-27T12:10:06.556 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.507+0000 7fe6c27fc700 1 --2- 172.21.15.195:0/3261336809 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe6b4063d00 0x7fe6b40661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:06.556 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.507+0000 7fe6c27fc700 1 -- 172.21.15.195:0/3261336809 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7234+0+0 (secure 0 0 0) 0x7fe6d0081090 con 0x7fe6d40c34a0 2022-06-27T12:10:06.556 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.507+0000 7fe6da21b700 1 --2- 172.21.15.195:0/3261336809 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe6b4063d00 0x7fe6b40661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:06.556 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.507+0000 7fe6da21b700 1 --2- 172.21.15.195:0/3261336809 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe6b4063d00 0x7fe6b40661b0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fe6c4011fd0 tx=0x7fe6c4017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:06.557 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.511+0000 7fe6c27fc700 1 -- 172.21.15.195:0/3261336809 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fe6d004f780 con 0x7fe6d40c34a0 2022-06-27T12:10:06.741 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.733+0000 7fe6dc47f700 1 -- 172.21.15.195:0/3261336809 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe6d40722c0 con 0x7fe6d40c34a0 2022-06-27T12:10:06.742 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.733+0000 7fe6c27fc700 1 -- 172.21.15.195:0/3261336809 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+6125 (secure 0 0 0) 0x7fe6d0052da0 con 0x7fe6d40c34a0 2022-06-27T12:10:06.747 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:10:06.748 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":84,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":5933,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/1342896965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":1342896965},{"type":"v1","addr":"172.21.15.202:6835","nonce":1342896965}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:09:59.334885+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6708},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6708":{"gid":6708,"name":"b","rank":0,"incarnation":79,"state":"up:active","state_seq":27,"addr":"172.21.15.202:6833/702244802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":702244802},{"type":"v1","addr":"172.21.15.202:6833","nonce":702244802}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:10:06.751 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.735+0000 7fe6dc47f700 1 -- 172.21.15.195:0/3261336809 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe6b4063d00 msgr2=0x7fe6b40661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:06.760 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.735+0000 7fe6dc47f700 1 --2- 172.21.15.195:0/3261336809 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe6b4063d00 0x7fe6b40661b0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fe6c4011fd0 tx=0x7fe6c4017000 comp rx=0 tx=0).stop 2022-06-27T12:10:06.760 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.735+0000 7fe6dc47f700 1 -- 172.21.15.195:0/3261336809 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe6d40c34a0 msgr2=0x7fe6d40c8380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:06.761 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.735+0000 7fe6dc47f700 1 --2- 172.21.15.195:0/3261336809 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe6d40c34a0 0x7fe6d40c8380 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7fe6d000d890 tx=0x7fe6d0008040 comp rx=0 tx=0).stop 2022-06-27T12:10:06.761 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.735+0000 7fe6dc47f700 1 -- 172.21.15.195:0/3261336809 shutdown_connections 2022-06-27T12:10:06.762 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.735+0000 7fe6dc47f700 1 --2- 172.21.15.195:0/3261336809 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe6b4063d00 0x7fe6b40661b0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:06.762 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.735+0000 7fe6dc47f700 1 --2- 172.21.15.195:0/3261336809 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe6d414db40 0x7fe6d414dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:06.763 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.735+0000 7fe6dc47f700 1 --2- 172.21.15.195:0/3261336809 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe6d406e850 0x7fe6d414ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:06.763 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.735+0000 7fe6dc47f700 1 --2- 172.21.15.195:0/3261336809 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe6d40c34a0 0x7fe6d40c8380 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:06.763 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.735+0000 7fe6dc47f700 1 -- 172.21.15.195:0/3261336809 >> 172.21.15.195:0/3261336809 conn(0x7fe6d40c6350 msgr2=0x7fe6d40c2b80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:10:06.764 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.735+0000 7fe6dc47f700 1 -- 172.21.15.195:0/3261336809 shutdown_connections 2022-06-27T12:10:06.764 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:06.735+0000 7fe6dc47f700 1 -- 172.21.15.195:0/3261336809 wait complete. 2022-06-27T12:10:06.778 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 84 2022-06-27T12:10:06.779 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d reported in up:standby state 2022-06-27T12:10:06.779 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 9 secs before thrashing 2022-06-27T12:10:07.246 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 9% (33/334) Updating files: 10% (34/334) Updating files: 10% (36/334) Updating files: 11% (37/334) Updating files: 12% (41/334) Updating files: 13% (44/334) Updating files: 14% (47/334) Updating files: 15% (51/334) Updating files: 16% (54/334) Updating files: 17% (57/334) Updating files: 18% (61/334) Updating files: 19% (64/334) Updating files: 20% (67/334) Updating files: 21% (71/334) Updating files: 22% (74/334) Updating files: 23% (77/334) Updating files: 24% (81/334) Updating files: 25% (84/334) Updating files: 26% (87/334) Updating files: 27% (91/334) Updating files: 28% (94/334) Updating files: 29% (97/334) Updating files: 30% (101/334) Updating files: 31% (104/334) Updating files: 32% (107/334) Updating files: 33% (111/334) Updating files: 34% (114/334) Updating files: 35% (117/334) Updating files: 36% (121/334) Updating files: 37% (124/334) Updating files: 38% (127/334) Updating files: 39% (131/334) Updating files: 40% (134/334) Updating files: 41% (137/334) Updating files: 42% (141/334) Updating files: 43% (144/334) Updating files: 44% (147/334) Updating files: 45% (151/334) Updating files: 46% (154/334) Updating files: 47% (157/334) Updating files: 48% (161/334) Updating files: 48% (162/334) Updating files: 49% (164/334) Updating files: 50% (167/334) Updating files: 51% (171/334) Updating files: 52% (174/334) Updating files: 53% (178/334) Updating files: 54% (181/334) Updating files: 55% (184/334) Updating files: 56% (188/334) Updating files: 57% (191/334) Updating files: 58% (194/334) Updating files: 59% (198/334) Updating files: 60% (201/334) Updating files: 61% (204/334) Updating files: 62% (208/334) Updating files: 63% (211/334) Updating files: 64% (214/334) Updating files: 65% (218/334) Updating files: 66% (221/334) Updating files: 67% (224/334) Updating files: 68% (228/334) Updating files: 69% (231/334) Updating files: 70% (234/334) Updating files: 71% (238/334) Updating files: 72% (241/334) Updating files: 73% (244/334) Updating files: 73% (246/334) Updating files: 74% (248/334) Updating files: 75% (251/334) Updating files: 76% (254/334) Updating files: 76% (255/334) Updating files: 76% (256/334) Updating files: 77% (258/334) Updating files: 78% (261/334) Updating files: 79% (264/334) Updating files: 80% (268/334) Updating files: 81% (271/334) Updating files: 82% (274/334) Updating files: 83% (278/334) Updating files: 84% (281/334) Updating files: 85% (284/334) Updating files: 86% (288/334) Updating files: 87% (291/334) Updating files: 88% (294/334) Updating files: 89% (298/334) Updating files: 90% (301/334) Updating files: 91% (304/334) Updating files: 92% (308/334) Updating files: 93% (311/334) Updating files: 94% (314/334) Updating files: 95% (318/334) Updating files: 96% (321/334) Updating files: 96% (323/334) Updating files: 97% (324/334) Updating files: 98% (328/334) Updating files: 98% (329/334) Updating files: 99% (331/334) Updating files: 100% (334/334) Updating files: 100% (334/334), done. 2022-06-27T12:10:07.508 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at 0afda378124 v0.35 2022-06-27T12:10:07.509 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.35 2022-06-27T12:10:08.346 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:10:08.346 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 36 -eq 48 ']' 2022-06-27T12:10:08.347 INFO:tasks.workunit.client.0.smithi195.stdout:v0.36 2022-06-27T12:10:08.347 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.36 2022-06-27T12:10:08.348 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.36 2022-06-27T12:10:08.348 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.36 2022-06-27T12:10:15.787 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:10:15.942 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:15.942+0000 7fcbe0101700 1 Processor -- start 2022-06-27T12:10:15.943 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:15.942+0000 7fcbe0101700 1 -- start start 2022-06-27T12:10:15.946 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:15.943+0000 7fcbe0101700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fcbd806f640 0x7fcbd8156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:15.948 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:15.943+0000 7fcbe0101700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcbd80c39e0 0x7fcbd81535a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:15.948 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:15.943+0000 7fcbe0101700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbd80c1a20 con 0x7fcbd80c4160 2022-06-27T12:10:15.949 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:15.943+0000 7fcbe0101700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbd80c1130 con 0x7fcbd806f640 2022-06-27T12:10:15.949 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:15.943+0000 7fcbe0101700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd806ea90 con 0x7fcbd80c39e0 2022-06-27T12:10:15.950 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:15.943+0000 7fcbdde9d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fcbd806f640 0x7fcbd8156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:15.950 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:15.943+0000 7fcbdd69c700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fcbd80c4160 0x7fcbd80c1eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:56004/0 (socket says 172.21.15.195:56004) 2022-06-27T12:10:15.951 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:15.943+0000 7fcbdd69c700 1 -- 172.21.15.195:0/1577555818 learned_addr learned my addr 172.21.15.195:0/1577555818 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:10:15.951 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:15.943+0000 7fcbdde9d700 1 -- 172.21.15.195:0/1577555818 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcbd80c39e0 msgr2=0x7fcbd81535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:10:15.952 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:15.943+0000 7fcbdde9d700 1 --2- 172.21.15.195:0/1577555818 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcbd80c39e0 0x7fcbd81535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:15.953 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:15.943+0000 7fcbdde9d700 1 -- 172.21.15.195:0/1577555818 >> v1:172.21.15.195:6789/0 conn(0x7fcbd80c4160 legacy=0x7fcbd80c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:15.953 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:15.943+0000 7fcbdde9d700 1 -- 172.21.15.195:0/1577555818 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbd806ed70 con 0x7fcbd806f640 2022-06-27T12:10:15.954 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:15.943+0000 7fcbdce9b700 1 -- 172.21.15.195:0/1577555818 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2946083188 0 0) 0x7fcbd80c1a20 con 0x7fcbd80c4160 2022-06-27T12:10:15.986 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:15.943+0000 7fcbdde9d700 1 --2- 172.21.15.195:0/1577555818 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcbd806f640 0x7fcbd8156ed0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7fcbd40097a0 tx=0x7fcbd4009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7602275eb6c6bfc1 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:15.986 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:15.943+0000 7fcbdce9b700 1 -- 172.21.15.195:0/1577555818 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd4011630 con 0x7fcbd806f640 2022-06-27T12:10:15.987 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:15.943+0000 7fcbdce9b700 1 -- 172.21.15.195:0/1577555818 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbd4011790 con 0x7fcbd806f640 2022-06-27T12:10:15.987 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:15.943+0000 7fcbdce9b700 1 -- 172.21.15.195:0/1577555818 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd4011b30 con 0x7fcbd806f640 2022-06-27T12:10:15.988 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:15.944+0000 7fcbe0101700 1 -- 172.21.15.195:0/1577555818 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcbd806f640 msgr2=0x7fcbd8156ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:15.988 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:15.944+0000 7fcbe0101700 1 --2- 172.21.15.195:0/1577555818 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcbd806f640 0x7fcbd8156ed0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7fcbd40097a0 tx=0x7fcbd4009060 comp rx=0 tx=0).stop 2022-06-27T12:10:15.989 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:15.944+0000 7fcbe0101700 1 -- 172.21.15.195:0/1577555818 shutdown_connections 2022-06-27T12:10:15.989 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:15.944+0000 7fcbe0101700 1 --2- 172.21.15.195:0/1577555818 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcbd80c39e0 0x7fcbd81535a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:15.990 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:15.944+0000 7fcbe0101700 1 --2- 172.21.15.195:0/1577555818 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcbd806f640 0x7fcbd8156ed0 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:15.990 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:15.944+0000 7fcbe0101700 1 -- 172.21.15.195:0/1577555818 >> 172.21.15.195:0/1577555818 conn(0x7fcbd80c6350 msgr2=0x7fcbd80c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:10:15.991 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:15.944+0000 7fcbe0101700 1 -- 172.21.15.195:0/1577555818 shutdown_connections 2022-06-27T12:10:15.991 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:15.944+0000 7fcbe0101700 1 -- 172.21.15.195:0/1577555818 wait complete. 2022-06-27T12:10:15.991 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:15.946+0000 7fcbe0101700 1 Processor -- start 2022-06-27T12:10:15.992 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:15.946+0000 7fcbe0101700 1 -- start start 2022-06-27T12:10:15.992 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:15.946+0000 7fcbe0101700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcbd806f640 0x7fcbd8166c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:15.993 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:15.946+0000 7fcbe0101700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcbd80c39e0 0x7fcbd80c8e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:15.993 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:15.946+0000 7fcbe0101700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcbd80c4160 0x7fcbd815bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:15.994 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:15.947+0000 7fcbe0101700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd80ba590 con 0x7fcbd80c4160 2022-06-27T12:10:15.994 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:15.947+0000 7fcbe0101700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd81671d0 con 0x7fcbd80c39e0 2022-06-27T12:10:15.995 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:15.947+0000 7fcbe0101700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd80c8740 con 0x7fcbd806f640 2022-06-27T12:10:15.995 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:15.947+0000 7fcbde69e700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcbd80c4160 0x7fcbd815bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:15.996 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:15.947+0000 7fcbde69e700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcbd80c4160 0x7fcbd815bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:49824/0 (socket says 172.21.15.195:49824) 2022-06-27T12:10:15.996 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:15.947+0000 7fcbde69e700 1 -- 172.21.15.195:0/3927491304 learned_addr learned my addr 172.21.15.195:0/3927491304 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:10:15.997 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:15.947+0000 7fcbdde9d700 1 --2- 172.21.15.195:0/3927491304 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcbd806f640 0x7fcbd8166c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:15.997 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:15.947+0000 7fcbdd69c700 1 --2- 172.21.15.195:0/3927491304 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcbd80c39e0 0x7fcbd80c8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:16.002 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:15.947+0000 7fcbde69e700 1 -- 172.21.15.195:0/3927491304 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcbd806f640 msgr2=0x7fcbd8166c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:16.002 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:15.947+0000 7fcbde69e700 1 --2- 172.21.15.195:0/3927491304 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcbd806f640 0x7fcbd8166c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:16.003 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:15.947+0000 7fcbde69e700 1 -- 172.21.15.195:0/3927491304 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcbd80c39e0 msgr2=0x7fcbd80c8e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:16.004 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:15.947+0000 7fcbde69e700 1 --2- 172.21.15.195:0/3927491304 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcbd80c39e0 0x7fcbd80c8e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:16.014 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:15.947+0000 7fcbde69e700 1 -- 172.21.15.195:0/3927491304 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbd40092e0 con 0x7fcbd80c4160 2022-06-27T12:10:16.022 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:15.947+0000 7fcbde69e700 1 --2- 172.21.15.195:0/3927491304 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcbd80c4160 0x7fcbd815bcb0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fcbd000bc60 tx=0x7fcbd0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:16.023 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:15.947+0000 7fcbc5ffb700 1 -- 172.21.15.195:0/3927491304 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd00092e0 con 0x7fcbd80c4160 2022-06-27T12:10:16.023 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:15.947+0000 7fcbc5ffb700 1 -- 172.21.15.195:0/3927491304 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbd000c6c0 con 0x7fcbd80c4160 2022-06-27T12:10:16.023 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:15.947+0000 7fcbe0101700 1 -- 172.21.15.195:0/3927491304 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbd8155720 con 0x7fcbd80c4160 2022-06-27T12:10:16.024 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:15.947+0000 7fcbc5ffb700 1 -- 172.21.15.195:0/3927491304 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd000ca80 con 0x7fcbd80c4160 2022-06-27T12:10:16.025 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:15.948+0000 7fcbe0101700 1 -- 172.21.15.195:0/3927491304 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbd80c7850 con 0x7fcbd80c4160 2022-06-27T12:10:16.025 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:15.948+0000 7fcbe0101700 1 -- 172.21.15.195:0/3927491304 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbd80722c0 con 0x7fcbd80c4160 2022-06-27T12:10:16.026 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:15.949+0000 7fcbc5ffb700 1 -- 172.21.15.195:0/3927491304 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fcbd0007400 con 0x7fcbd80c4160 2022-06-27T12:10:16.026 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:15.949+0000 7fcbc5ffb700 1 --2- 172.21.15.195:0/3927491304 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcbb8063d00 0x7fcbb80661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:16.027 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:15.949+0000 7fcbc5ffb700 1 -- 172.21.15.195:0/3927491304 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7234+0+0 (secure 0 0 0) 0x7fcbd0080860 con 0x7fcbd80c4160 2022-06-27T12:10:16.027 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:15.949+0000 7fcbdde9d700 1 --2- 172.21.15.195:0/3927491304 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcbb8063d00 0x7fcbb80661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:16.028 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:15.949+0000 7fcbdde9d700 1 --2- 172.21.15.195:0/3927491304 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcbb8063d00 0x7fcbb80661b0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fcbd4009260 tx=0x7fcbd4017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:16.028 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:15.953+0000 7fcbc5ffb700 1 -- 172.21.15.195:0/3927491304 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fcbd004ef50 con 0x7fcbd80c4160 2022-06-27T12:10:16.196 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.188+0000 7fcbe0101700 1 -- 172.21.15.195:0/3927491304 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcbd80c88d0 con 0x7fcbd80c4160 2022-06-27T12:10:16.201 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.189+0000 7fcbc5ffb700 1 -- 172.21.15.195:0/3927491304 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+6125 (secure 0 0 0) 0x7fcbd0052570 con 0x7fcbd80c4160 2022-06-27T12:10:16.202 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:10:16.203 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":84,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":5933,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/1342896965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":1342896965},{"type":"v1","addr":"172.21.15.202:6835","nonce":1342896965}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:09:59.334885+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6708},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6708":{"gid":6708,"name":"b","rank":0,"incarnation":79,"state":"up:active","state_seq":27,"addr":"172.21.15.202:6833/702244802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":702244802},{"type":"v1","addr":"172.21.15.202:6833","nonce":702244802}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:10:16.205 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.191+0000 7fcbe0101700 1 -- 172.21.15.195:0/3927491304 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcbb8063d00 msgr2=0x7fcbb80661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:16.206 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.191+0000 7fcbe0101700 1 --2- 172.21.15.195:0/3927491304 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcbb8063d00 0x7fcbb80661b0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fcbd4009260 tx=0x7fcbd4017000 comp rx=0 tx=0).stop 2022-06-27T12:10:16.206 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.191+0000 7fcbe0101700 1 -- 172.21.15.195:0/3927491304 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcbd80c4160 msgr2=0x7fcbd815bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:16.206 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.191+0000 7fcbe0101700 1 --2- 172.21.15.195:0/3927491304 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcbd80c4160 0x7fcbd815bcb0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fcbd000bc60 tx=0x7fcbd0009840 comp rx=0 tx=0).stop 2022-06-27T12:10:16.207 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.191+0000 7fcbe0101700 1 -- 172.21.15.195:0/3927491304 shutdown_connections 2022-06-27T12:10:16.207 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.191+0000 7fcbe0101700 1 --2- 172.21.15.195:0/3927491304 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcbb8063d00 0x7fcbb80661b0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:16.208 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.191+0000 7fcbe0101700 1 --2- 172.21.15.195:0/3927491304 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcbd80c4160 0x7fcbd815bcb0 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:16.208 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.191+0000 7fcbe0101700 1 --2- 172.21.15.195:0/3927491304 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcbd806f640 0x7fcbd8166c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:16.209 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.191+0000 7fcbe0101700 1 --2- 172.21.15.195:0/3927491304 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcbd80c39e0 0x7fcbd80c8e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:16.209 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.191+0000 7fcbe0101700 1 -- 172.21.15.195:0/3927491304 >> 172.21.15.195:0/3927491304 conn(0x7fcbd80c6350 msgr2=0x7fcbd80c1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:10:16.209 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.191+0000 7fcbe0101700 1 -- 172.21.15.195:0/3927491304 shutdown_connections 2022-06-27T12:10:16.210 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.191+0000 7fcbe0101700 1 -- 172.21.15.195:0/3927491304 wait complete. 2022-06-27T12:10:16.211 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 84 2022-06-27T12:10:16.217 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2022-06-27T12:10:16.247 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-06-27T12:10:16.248 INFO:teuthology.orchestra.run:waiting for 300 2022-06-27T12:10:16.250 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 42% (104/244) Updating files: 43% (105/244) Updating files: 44% (108/244) Updating files: 45% (110/244) Updating files: 46% (113/244) Updating files: 47% (115/244) Updating files: 48% (118/244) Updating files: 49% (120/244) Updating files: 50% (122/244) Updating files: 51% (125/244) Updating files: 52% (127/244) Updating files: 53% (130/244) Updating files: 54% (132/244) Updating files: 55% (135/244) Updating files: 56% (137/244) Updating files: 57% (140/244) Updating files: 58% (142/244) Updating files: 59% (144/244) Updating files: 60% (147/244) Updating files: 61% (149/244) Updating files: 62% (152/244) Updating files: 63% (154/244) Updating files: 64% (157/244) Updating files: 65% (159/244) Updating files: 66% (162/244) Updating files: 67% (164/244) Updating files: 68% (166/244) Updating files: 69% (169/244) Updating files: 70% (171/244) Updating files: 71% (174/244) Updating files: 72% (176/244) Updating files: 73% (179/244) Updating files: 74% (181/244) Updating files: 75% (183/244) Updating files: 76% (186/244) Updating files: 77% (188/244) Updating files: 78% (191/244) Updating files: 79% (193/244) Updating files: 80% (196/244) Updating files: 81% (198/244) Updating files: 82% (201/244) Updating files: 83% (203/244) Updating files: 84% (205/244) Updating files: 85% (208/244) Updating files: 86% (210/244) Updating files: 87% (213/244) Updating files: 87% (214/244) Updating files: 88% (215/244) Updating files: 89% (218/244) Updating files: 90% (220/244) Updating files: 91% (223/244) Updating files: 92% (225/244) Updating files: 93% (227/244) Updating files: 94% (230/244) Updating files: 95% (232/244) Updating files: 96% (235/244) Updating files: 97% (237/244) Updating files: 98% (240/244) Updating files: 99% (242/244) Updating files: 100% (244/244) Updating files: 100% (244/244), done. 2022-06-27T12:10:16.251 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at 877cacbbdbb v0.36 2022-06-27T12:10:16.252 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.36 2022-06-27T12:10:16.269 INFO:tasks.ceph.mds.b:Stopped 2022-06-27T12:10:16.269 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:10:16.439 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.407+0000 7f2edbffa700 1 Processor -- start 2022-06-27T12:10:16.440 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.407+0000 7f2edbffa700 1 -- start start 2022-06-27T12:10:16.446 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.408+0000 7f2edbffa700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2ed4150950 0x7f2ed4156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:16.447 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.408+0000 7f2edbffa700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ed40c76b0 con 0x7f2ed4150580 2022-06-27T12:10:16.447 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.408+0000 7f2edbffa700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ed40c8250 con 0x7f2ed406f620 2022-06-27T12:10:16.447 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.408+0000 7f2edbffa700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed4157430 con 0x7f2ed4150950 2022-06-27T12:10:16.448 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.408+0000 7f2ed9d96700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f2ed4150580 0x7f2ed40c7b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:56018/0 (socket says 172.21.15.195:56018) 2022-06-27T12:10:16.448 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.408+0000 7f2ed9d96700 1 -- 172.21.15.195:0/3614386145 learned_addr learned my addr 172.21.15.195:0/3614386145 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:10:16.448 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.408+0000 7f2ed9595700 1 --2- 172.21.15.195:0/3614386145 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2ed4150950 0x7f2ed4156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:16.448 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.408+0000 7f2ed8d94700 1 -- 172.21.15.195:0/3614386145 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1490437011 0 0) 0x7f2ed40c76b0 con 0x7f2ed4150580 2022-06-27T12:10:16.449 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.408+0000 7f2ed8d94700 1 -- 172.21.15.195:0/3614386145 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ebc003430 con 0x7f2ed4150580 2022-06-27T12:10:16.449 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.408+0000 7f2ed8d94700 1 -- 172.21.15.195:0/3614386145 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3311957165 0 0) 0x7f2ed40c8250 con 0x7f2ed406f620 2022-06-27T12:10:16.449 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.408+0000 7f2ed8d94700 1 -- 172.21.15.195:0/3614386145 --> v1:172.21.15.202:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ed40c76b0 con 0x7f2ed406f620 2022-06-27T12:10:16.450 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.408+0000 7f2ed9595700 1 -- 172.21.15.195:0/3614386145 >> v1:172.21.15.202:6789/0 conn(0x7f2ed406f620 legacy=0x7f2ed406d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:16.450 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.408+0000 7f2ed9595700 1 -- 172.21.15.195:0/3614386145 >> v1:172.21.15.195:6789/0 conn(0x7f2ed4150580 legacy=0x7f2ed40c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:16.450 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.408+0000 7f2ed9595700 1 -- 172.21.15.195:0/3614386145 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ed40bad70 con 0x7f2ed4150950 2022-06-27T12:10:16.450 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.409+0000 7f2ed9595700 1 --2- 172.21.15.195:0/3614386145 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2ed4150950 0x7f2ed4156ef0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f2ec40097d0 tx=0x7f2ec4009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a74267edd4634be8 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:16.451 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.409+0000 7f2ed8d94700 1 -- 172.21.15.195:0/3614386145 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec4011630 con 0x7f2ed4150950 2022-06-27T12:10:16.451 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.409+0000 7f2ed8d94700 1 -- 172.21.15.195:0/3614386145 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ec4011790 con 0x7f2ed4150950 2022-06-27T12:10:16.451 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.409+0000 7f2ed8d94700 1 -- 172.21.15.195:0/3614386145 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec4011b10 con 0x7f2ed4150950 2022-06-27T12:10:16.451 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.409+0000 7f2edbffa700 1 -- 172.21.15.195:0/3614386145 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2ed4150950 msgr2=0x7f2ed4156ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:16.452 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.409+0000 7f2edbffa700 1 --2- 172.21.15.195:0/3614386145 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2ed4150950 0x7f2ed4156ef0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f2ec40097d0 tx=0x7f2ec4009060 comp rx=0 tx=0).stop 2022-06-27T12:10:16.452 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.409+0000 7f2edbffa700 1 -- 172.21.15.195:0/3614386145 shutdown_connections 2022-06-27T12:10:16.452 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.409+0000 7f2edbffa700 1 --2- 172.21.15.195:0/3614386145 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2ed4150950 0x7f2ed4156ef0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:16.453 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.409+0000 7f2edbffa700 1 -- 172.21.15.195:0/3614386145 >> 172.21.15.195:0/3614386145 conn(0x7f2ed4063af0 msgr2=0x7f2ed4063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:10:16.453 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.409+0000 7f2edbffa700 1 -- 172.21.15.195:0/3614386145 shutdown_connections 2022-06-27T12:10:16.453 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.409+0000 7f2edbffa700 1 -- 172.21.15.195:0/3614386145 wait complete. 2022-06-27T12:10:16.454 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.410+0000 7f2edbffa700 1 Processor -- start 2022-06-27T12:10:16.454 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.410+0000 7f2edbffa700 1 -- start start 2022-06-27T12:10:16.455 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.411+0000 7f2edbffa700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2ed406f620 0x7f2ed40c9df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:16.455 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.411+0000 7f2edbffa700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2ed4150580 0x7f2ed40bb290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:16.456 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.411+0000 7f2edbffa700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2ed4150950 0x7f2ed4169fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:16.456 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.411+0000 7f2edbffa700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed416ac40 con 0x7f2ed4150950 2022-06-27T12:10:16.457 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.411+0000 7f2edbffa700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed40c8c70 con 0x7f2ed406f620 2022-06-27T12:10:16.457 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.411+0000 7f2edbffa700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed40c2e10 con 0x7f2ed4150580 2022-06-27T12:10:16.458 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.411+0000 7f2eda597700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2ed4150950 0x7f2ed4169fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:16.458 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.411+0000 7f2eda597700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2ed4150950 0x7f2ed4169fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:49840/0 (socket says 172.21.15.195:49840) 2022-06-27T12:10:16.458 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.411+0000 7f2eda597700 1 -- 172.21.15.195:0/2927391992 learned_addr learned my addr 172.21.15.195:0/2927391992 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:10:16.459 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.411+0000 7f2ed9595700 1 --2- 172.21.15.195:0/2927391992 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2ed4150580 0x7f2ed40bb290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:16.459 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.411+0000 7f2ed9d96700 1 --2- 172.21.15.195:0/2927391992 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2ed406f620 0x7f2ed40c9df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:16.461 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.411+0000 7f2ed9595700 1 -- 172.21.15.195:0/2927391992 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2ed406f620 msgr2=0x7f2ed40c9df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:16.462 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.411+0000 7f2ed9595700 1 --2- 172.21.15.195:0/2927391992 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2ed406f620 0x7f2ed40c9df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:16.462 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.412+0000 7f2ed9595700 1 -- 172.21.15.195:0/2927391992 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2ed4150950 msgr2=0x7f2ed4169fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:16.463 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.412+0000 7f2ed9595700 1 --2- 172.21.15.195:0/2927391992 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2ed4150950 0x7f2ed4169fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:16.463 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.412+0000 7f2ed9595700 1 -- 172.21.15.195:0/2927391992 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ec40092e0 con 0x7f2ed4150580 2022-06-27T12:10:16.463 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.412+0000 7f2ed9595700 1 --2- 172.21.15.195:0/2927391992 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2ed4150580 0x7f2ed40bb290 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f2ed40c8470 tx=0x7f2ec4017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:16.464 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.412+0000 7f2ec1ffb700 1 -- 172.21.15.195:0/2927391992 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec4003730 con 0x7f2ed4150580 2022-06-27T12:10:16.464 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.412+0000 7f2ec1ffb700 1 -- 172.21.15.195:0/2927391992 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ec4003890 con 0x7f2ed4150580 2022-06-27T12:10:16.465 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.412+0000 7f2edbffa700 1 -- 172.21.15.195:0/2927391992 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ed40bb7d0 con 0x7f2ed4150580 2022-06-27T12:10:16.465 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.412+0000 7f2ec1ffb700 1 -- 172.21.15.195:0/2927391992 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec4003c30 con 0x7f2ed4150580 2022-06-27T12:10:16.466 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.412+0000 7f2edbffa700 1 -- 172.21.15.195:0/2927391992 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ed40c27c0 con 0x7f2ed4150580 2022-06-27T12:10:16.466 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.413+0000 7f2edbffa700 1 -- 172.21.15.195:0/2927391992 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ed40722c0 con 0x7f2ed4150580 2022-06-27T12:10:16.467 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.414+0000 7f2ec1ffb700 1 -- 172.21.15.195:0/2927391992 <== mon.2 v2:172.21.15.202:3301/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f2ec400f050 con 0x7f2ed4150580 2022-06-27T12:10:16.467 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.414+0000 7f2ec1ffb700 1 --2- 172.21.15.195:0/2927391992 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2eb4063d00 0x7f2eb40661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:16.468 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.414+0000 7f2ec1ffb700 1 -- 172.21.15.195:0/2927391992 <== mon.2 v2:172.21.15.202:3301/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7234+0+0 (secure 0 0 0) 0x7f2ec40820d0 con 0x7f2ed4150580 2022-06-27T12:10:16.468 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.414+0000 7f2ed9d96700 1 --2- 172.21.15.195:0/2927391992 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2eb4063d00 0x7f2eb40661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:16.468 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.415+0000 7f2ed9d96700 1 --2- 172.21.15.195:0/2927391992 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2eb4063d00 0x7f2eb40661b0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f2ed0007fd0 tx=0x7f2ed0009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:16.469 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.419+0000 7f2ec1ffb700 1 -- 172.21.15.195:0/2927391992 <== mon.2 v2:172.21.15.202:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f2ec4050940 con 0x7f2ed4150580 2022-06-27T12:10:16.646 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.646+0000 7f2edbffa700 1 -- 172.21.15.195:0/2927391992 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2ed406fe20 con 0x7f2ed4150580 2022-06-27T12:10:16.647 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.648+0000 7f2ec1ffb700 1 -- 172.21.15.195:0/2927391992 <== mon.2 v2:172.21.15.202:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+6125 (secure 0 0 0) 0x7f2ec4015070 con 0x7f2ed4150580 2022-06-27T12:10:16.648 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:10:16.648 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":84,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":5933,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/1342896965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":1342896965},{"type":"v1","addr":"172.21.15.202:6835","nonce":1342896965}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:09:59.334885+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6708},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6708":{"gid":6708,"name":"b","rank":0,"incarnation":79,"state":"up:active","state_seq":27,"addr":"172.21.15.202:6833/702244802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":702244802},{"type":"v1","addr":"172.21.15.202:6833","nonce":702244802}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:10:16.650 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.649+0000 7f2edbffa700 1 -- 172.21.15.195:0/2927391992 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2eb4063d00 msgr2=0x7f2eb40661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:16.650 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.649+0000 7f2edbffa700 1 --2- 172.21.15.195:0/2927391992 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2eb4063d00 0x7f2eb40661b0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f2ed0007fd0 tx=0x7f2ed0009040 comp rx=0 tx=0).stop 2022-06-27T12:10:16.650 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.649+0000 7f2edbffa700 1 -- 172.21.15.195:0/2927391992 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2ed4150580 msgr2=0x7f2ed40bb290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:16.650 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.649+0000 7f2edbffa700 1 --2- 172.21.15.195:0/2927391992 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2ed4150580 0x7f2ed40bb290 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f2ed40c8470 tx=0x7f2ec4017000 comp rx=0 tx=0).stop 2022-06-27T12:10:16.651 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.649+0000 7f2edbffa700 1 -- 172.21.15.195:0/2927391992 shutdown_connections 2022-06-27T12:10:16.651 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.649+0000 7f2edbffa700 1 --2- 172.21.15.195:0/2927391992 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2eb4063d00 0x7f2eb40661b0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:16.651 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.649+0000 7f2edbffa700 1 --2- 172.21.15.195:0/2927391992 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2ed4150950 0x7f2ed4169fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:16.651 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.649+0000 7f2edbffa700 1 --2- 172.21.15.195:0/2927391992 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2ed406f620 0x7f2ed40c9df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:16.652 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.649+0000 7f2edbffa700 1 --2- 172.21.15.195:0/2927391992 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2ed4150580 0x7f2ed40bb290 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:16.652 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.649+0000 7f2edbffa700 1 -- 172.21.15.195:0/2927391992 >> 172.21.15.195:0/2927391992 conn(0x7f2ed4063af0 msgr2=0x7f2ed414d950 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:10:16.652 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.649+0000 7f2edbffa700 1 -- 172.21.15.195:0/2927391992 shutdown_connections 2022-06-27T12:10:16.652 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:16.649+0000 7f2edbffa700 1 -- 172.21.15.195:0/2927391992 wait complete. 2022-06-27T12:10:16.663 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 84 2022-06-27T12:10:16.672 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2022-06-27T12:10:18.673 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:10:18.806 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.806+0000 7f870f4af700 1 Processor -- start 2022-06-27T12:10:18.807 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.806+0000 7f870f4af700 1 -- start start 2022-06-27T12:10:18.808 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.807+0000 7f870f4af700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f87080c3f40 0x7f8708158360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:18.808 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.807+0000 7f870f4af700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f870806d7f0 con 0x7f87080c48e0 2022-06-27T12:10:18.809 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.807+0000 7f870f4af700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8708154400 con 0x7f870806d070 2022-06-27T12:10:18.809 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.807+0000 7f870f4af700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87080c2060 con 0x7f87080c3f40 2022-06-27T12:10:18.809 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.807+0000 7f870d24b700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f87080c48e0 0x7f87080ca960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:56042/0 (socket says 172.21.15.195:56042) 2022-06-27T12:10:18.809 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.807+0000 7f870d24b700 1 -- 172.21.15.195:0/2561265195 learned_addr learned my addr 172.21.15.195:0/2561265195 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:10:18.810 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.807+0000 7f870da4c700 1 --2- 172.21.15.195:0/2561265195 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f87080c3f40 0x7f8708158360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:18.810 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.807+0000 7f86f7fff700 1 -- 172.21.15.195:0/2561265195 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 143252107 0 0) 0x7f870806d7f0 con 0x7f87080c48e0 2022-06-27T12:10:18.822 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.807+0000 7f86f7fff700 1 -- 172.21.15.195:0/2561265195 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86f0003430 con 0x7f87080c48e0 2022-06-27T12:10:18.824 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.808+0000 7f86f7fff700 1 -- 172.21.15.195:0/2561265195 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1681793453 0 0) 0x7f86f0003430 con 0x7f87080c48e0 2022-06-27T12:10:18.824 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.808+0000 7f86f7fff700 1 -- 172.21.15.195:0/2561265195 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f87080c3f40 msgr2=0x7f8708158360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:10:18.824 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.808+0000 7f86f7fff700 1 --2- 172.21.15.195:0/2561265195 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f87080c3f40 0x7f8708158360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:18.824 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.808+0000 7f86f7fff700 1 -- 172.21.15.195:0/2561265195 >> v1:172.21.15.202:6789/0 conn(0x7f870806d070 legacy=0x7f87080c3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:18.825 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.808+0000 7f86f7fff700 1 -- 172.21.15.195:0/2561265195 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87081546c0 con 0x7f87080c48e0 2022-06-27T12:10:18.825 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.808+0000 7f86f7fff700 1 -- 172.21.15.195:0/2561265195 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7f86f8001470 con 0x7f87080c48e0 2022-06-27T12:10:18.825 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.808+0000 7f86f7fff700 1 -- 172.21.15.195:0/2561265195 >> v1:172.21.15.195:6789/0 conn(0x7f87080c48e0 legacy=0x7f87080ca960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:18.825 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.808+0000 7f86f7fff700 1 --2- 172.21.15.195:0/2561265195 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f86f0003950 0x7f86f0003d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:18.826 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.808+0000 7f86f7fff700 1 --2- 172.21.15.195:0/2561265195 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f86f00046d0 0x7f86f0004b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:18.826 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.808+0000 7f86f7fff700 1 --2- 172.21.15.195:0/2561265195 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f87080c3f40 0x7f86f0005080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:18.826 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.808+0000 7f86f7fff700 1 -- 172.21.15.195:0/2561265195 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87080c2060 con 0x7f86f00046d0 2022-06-27T12:10:18.827 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.808+0000 7f86f7fff700 1 -- 172.21.15.195:0/2561265195 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86f000ba00 con 0x7f86f0003950 2022-06-27T12:10:18.827 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.808+0000 7f86f7fff700 1 -- 172.21.15.195:0/2561265195 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86f000bbc0 con 0x7f87080c3f40 2022-06-27T12:10:18.827 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.808+0000 7f870ca4a700 1 --2- 172.21.15.195:0/2561265195 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f86f00046d0 0x7f86f0004b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:18.827 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.808+0000 7f870d24b700 1 --2- 172.21.15.195:0/2561265195 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f86f0003950 0x7f86f0003d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:18.828 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.808+0000 7f870da4c700 1 --2- 172.21.15.195:0/2561265195 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f87080c3f40 0x7f86f0005080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:18.828 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.808+0000 7f870ca4a700 1 -- 172.21.15.195:0/2561265195 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f87080c3f40 msgr2=0x7f86f0005080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:10:18.829 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.808+0000 7f870ca4a700 1 --2- 172.21.15.195:0/2561265195 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f87080c3f40 0x7f86f0005080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:18.830 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.808+0000 7f870ca4a700 1 -- 172.21.15.195:0/2561265195 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f86f0003950 msgr2=0x7f86f0003d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:10:18.830 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.808+0000 7f870ca4a700 1 --2- 172.21.15.195:0/2561265195 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f86f0003950 0x7f86f0003d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:18.831 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.808+0000 7f870ca4a700 1 -- 172.21.15.195:0/2561265195 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86f0003430 con 0x7f86f00046d0 2022-06-27T12:10:18.831 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.808+0000 7f870ca4a700 1 --2- 172.21.15.195:0/2561265195 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f86f00046d0 0x7f86f0004b40 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f8704001860 tx=0x7f8704007bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1da327090e6f6e server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:18.832 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.808+0000 7f86f7fff700 1 -- 172.21.15.195:0/2561265195 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8704012420 con 0x7f86f00046d0 2022-06-27T12:10:18.832 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.809+0000 7f86f7fff700 1 -- 172.21.15.195:0/2561265195 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87040125a0 con 0x7f86f00046d0 2022-06-27T12:10:18.832 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.809+0000 7f86f7fff700 1 -- 172.21.15.195:0/2561265195 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8704012960 con 0x7f86f00046d0 2022-06-27T12:10:18.833 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.809+0000 7f870f4af700 1 -- 172.21.15.195:0/2561265195 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f86f00046d0 msgr2=0x7f86f0004b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:18.834 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.809+0000 7f870f4af700 1 --2- 172.21.15.195:0/2561265195 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f86f00046d0 0x7f86f0004b40 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f8704001860 tx=0x7f8704007bd0 comp rx=0 tx=0).stop 2022-06-27T12:10:18.834 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.809+0000 7f870da4c700 1 -- 172.21.15.195:0/2561265195 reap_dead start 2022-06-27T12:10:18.835 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.809+0000 7f870f4af700 1 -- 172.21.15.195:0/2561265195 shutdown_connections 2022-06-27T12:10:18.836 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.809+0000 7f870f4af700 1 -- 172.21.15.195:0/2561265195 >> 172.21.15.195:0/2561265195 conn(0x7f8708063af0 msgr2=0x7f8708063ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:10:18.836 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.809+0000 7f870f4af700 1 -- 172.21.15.195:0/2561265195 shutdown_connections 2022-06-27T12:10:18.837 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.809+0000 7f870f4af700 1 -- 172.21.15.195:0/2561265195 wait complete. 2022-06-27T12:10:18.838 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.810+0000 7f870f4af700 1 Processor -- start 2022-06-27T12:10:18.838 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.810+0000 7f870f4af700 1 -- start start 2022-06-27T12:10:18.838 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.811+0000 7f870f4af700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f86f00046d0 0x7f87080b9a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:18.839 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.811+0000 7f870f4af700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f87080ba080 0x7f87080c7280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:18.839 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.811+0000 7f870f4af700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f87081657e0 0x7f87080c08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:18.840 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.811+0000 7f870f4af700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87080b8c70 con 0x7f87081657e0 2022-06-27T12:10:18.840 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.811+0000 7f870f4af700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87080b80f0 con 0x7f86f00046d0 2022-06-27T12:10:18.840 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.811+0000 7f870f4af700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87080b9100 con 0x7f87080ba080 2022-06-27T12:10:18.841 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.811+0000 7f870da4c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f87081657e0 0x7f87080c08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:18.841 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.811+0000 7f870da4c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f87081657e0 0x7f87080c08c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:49870/0 (socket says 172.21.15.195:49870) 2022-06-27T12:10:18.842 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.811+0000 7f870da4c700 1 -- 172.21.15.195:0/2875150728 learned_addr learned my addr 172.21.15.195:0/2875150728 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:10:18.842 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.811+0000 7f870ca4a700 1 --2- 172.21.15.195:0/2875150728 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f87080ba080 0x7f87080c7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:18.843 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.812+0000 7f870d24b700 1 --2- 172.21.15.195:0/2875150728 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f86f00046d0 0x7f87080b9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:18.843 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.812+0000 7f870da4c700 1 -- 172.21.15.195:0/2875150728 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f87080ba080 msgr2=0x7f87080c7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:18.843 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.812+0000 7f870da4c700 1 --2- 172.21.15.195:0/2875150728 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f87080ba080 0x7f87080c7280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:18.844 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.812+0000 7f870da4c700 1 -- 172.21.15.195:0/2875150728 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f86f00046d0 msgr2=0x7f87080b9a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:18.845 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.812+0000 7f870da4c700 1 --2- 172.21.15.195:0/2875150728 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f86f00046d0 0x7f87080b9a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:18.846 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.812+0000 7f870da4c700 1 -- 172.21.15.195:0/2875150728 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8708165080 con 0x7f87081657e0 2022-06-27T12:10:18.846 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.812+0000 7f870d24b700 1 --2- 172.21.15.195:0/2875150728 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f86f00046d0 0x7f87080b9a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:10:18.847 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.812+0000 7f870da4c700 1 --2- 172.21.15.195:0/2875150728 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f87081657e0 0x7f87080c08c0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f87080c4330 tx=0x7f870000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:18.847 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.812+0000 7f86f57fa700 1 -- 172.21.15.195:0/2875150728 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f870000e040 con 0x7f87081657e0 2022-06-27T12:10:18.848 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.812+0000 7f86f57fa700 1 -- 172.21.15.195:0/2875150728 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f870000f040 con 0x7f87081657e0 2022-06-27T12:10:18.848 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.812+0000 7f870f4af700 1 -- 172.21.15.195:0/2875150728 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87080b8340 con 0x7f87081657e0 2022-06-27T12:10:18.849 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.812+0000 7f86f57fa700 1 -- 172.21.15.195:0/2875150728 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f870000fe20 con 0x7f87081657e0 2022-06-27T12:10:18.849 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.812+0000 7f870f4af700 1 -- 172.21.15.195:0/2875150728 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87080b8900 con 0x7f87081657e0 2022-06-27T12:10:18.850 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.813+0000 7f870f4af700 1 -- 172.21.15.195:0/2875150728 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87080b8e20 con 0x7f87081657e0 2022-06-27T12:10:18.850 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.813+0000 7f86f57fa700 1 -- 172.21.15.195:0/2875150728 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f8700003ab0 con 0x7f87081657e0 2022-06-27T12:10:18.851 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.814+0000 7f86f57fa700 1 --2- 172.21.15.195:0/2875150728 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f86e8063d00 0x7f86e80661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:18.851 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.814+0000 7f86f57fa700 1 -- 172.21.15.195:0/2875150728 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7234+0+0 (secure 0 0 0) 0x7f87000813f0 con 0x7f87081657e0 2022-06-27T12:10:18.851 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.814+0000 7f870d24b700 1 --2- 172.21.15.195:0/2875150728 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f86e8063d00 0x7f86e80661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:18.853 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.814+0000 7f870d24b700 1 --2- 172.21.15.195:0/2875150728 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f86e8063d00 0x7f86e80661b0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f86f80015a0 tx=0x7f86f8009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:18.853 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:18.818+0000 7f86f57fa700 1 -- 172.21.15.195:0/2875150728 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f87080b8e20 con 0x7f87081657e0 2022-06-27T12:10:19.055 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:19.039+0000 7f870f4af700 1 -- 172.21.15.195:0/2875150728 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87081551a0 con 0x7f87081657e0 2022-06-27T12:10:19.056 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:19.040+0000 7f86f57fa700 1 -- 172.21.15.195:0/2875150728 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+6125 (secure 0 0 0) 0x7f870004fb40 con 0x7f87081657e0 2022-06-27T12:10:19.057 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:10:19.057 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":84,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":5933,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/1342896965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":1342896965},{"type":"v1","addr":"172.21.15.202:6835","nonce":1342896965}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:09:59.334885+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6708},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6708":{"gid":6708,"name":"b","rank":0,"incarnation":79,"state":"up:active","state_seq":27,"addr":"172.21.15.202:6833/702244802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":702244802},{"type":"v1","addr":"172.21.15.202:6833","nonce":702244802}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:10:19.061 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:19.042+0000 7f870f4af700 1 -- 172.21.15.195:0/2875150728 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f86e8063d00 msgr2=0x7f86e80661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:19.061 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:19.042+0000 7f870f4af700 1 --2- 172.21.15.195:0/2875150728 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f86e8063d00 0x7f86e80661b0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f86f80015a0 tx=0x7f86f8009040 comp rx=0 tx=0).stop 2022-06-27T12:10:19.062 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:19.042+0000 7f870f4af700 1 -- 172.21.15.195:0/2875150728 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f87081657e0 msgr2=0x7f87080c08c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:19.062 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:19.042+0000 7f870f4af700 1 --2- 172.21.15.195:0/2875150728 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f87081657e0 0x7f87080c08c0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f87080c4330 tx=0x7f870000b040 comp rx=0 tx=0).stop 2022-06-27T12:10:19.063 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:19.042+0000 7f870f4af700 1 -- 172.21.15.195:0/2875150728 shutdown_connections 2022-06-27T12:10:19.063 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:19.042+0000 7f870f4af700 1 --2- 172.21.15.195:0/2875150728 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f86e8063d00 0x7f86e80661b0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:19.064 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:19.042+0000 7f870f4af700 1 --2- 172.21.15.195:0/2875150728 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f87081657e0 0x7f87080c08c0 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:19.064 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:19.042+0000 7f870f4af700 1 --2- 172.21.15.195:0/2875150728 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f86f00046d0 0x7f87080b9a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:19.065 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:19.042+0000 7f870f4af700 1 --2- 172.21.15.195:0/2875150728 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f87080ba080 0x7f87080c7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:19.065 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:19.042+0000 7f870f4af700 1 -- 172.21.15.195:0/2875150728 >> 172.21.15.195:0/2875150728 conn(0x7f8708063af0 msgr2=0x7f8708156220 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:10:19.066 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:19.042+0000 7f870f4af700 1 -- 172.21.15.195:0/2875150728 shutdown_connections 2022-06-27T12:10:19.067 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:19.042+0000 7f870f4af700 1 -- 172.21.15.195:0/2875150728 wait complete. 2022-06-27T12:10:19.067 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 84 2022-06-27T12:10:19.068 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2022-06-27T12:10:21.070 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:10:21.263 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.210+0000 7f60f7909700 1 Processor -- start 2022-06-27T12:10:21.263 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.211+0000 7f60f7909700 1 -- start start 2022-06-27T12:10:21.264 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.211+0000 7f60f7909700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f60f006f640 0x7f60f0156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:21.264 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.211+0000 7f60f7909700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f60f00c4160 0x7f60f00c1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:21.265 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.211+0000 7f60f7909700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60f00c1a20 con 0x7f60f00c39e0 2022-06-27T12:10:21.265 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.211+0000 7f60f7909700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60f00c1130 con 0x7f60f00c4160 2022-06-27T12:10:21.267 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.211+0000 7f60f7909700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60f006ea90 con 0x7f60f006f640 2022-06-27T12:10:21.268 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.212+0000 7f60f5ea6700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f60f00c39e0 0x7f60f01535a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:56074/0 (socket says 172.21.15.195:56074) 2022-06-27T12:10:21.268 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.212+0000 7f60f5ea6700 1 -- 172.21.15.195:0/616973864 learned_addr learned my addr 172.21.15.195:0/616973864 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:10:21.269 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.212+0000 7f60f4ea4700 1 --2- 172.21.15.195:0/616973864 >> v2:172.21.15.202:3300/0 conn(0x7f60f00c4160 0x7f60f00c1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:21.270 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.212+0000 7f60f56a5700 1 --2- 172.21.15.195:0/616973864 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f60f006f640 0x7f60f0156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:21.270 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.212+0000 7f60e3fff700 1 -- 172.21.15.195:0/616973864 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1804622492 0 0) 0x7f60f00c1a20 con 0x7f60f00c39e0 2022-06-27T12:10:21.271 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.212+0000 7f60e3fff700 1 -- 172.21.15.195:0/616973864 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60d8003430 con 0x7f60f00c39e0 2022-06-27T12:10:21.271 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.212+0000 7f60e3fff700 1 -- 172.21.15.195:0/616973864 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2111540155 0 0) 0x7f60d8003430 con 0x7f60f00c39e0 2022-06-27T12:10:21.271 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.212+0000 7f60e3fff700 1 -- 172.21.15.195:0/616973864 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f60f006f640 msgr2=0x7f60f0156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:10:21.272 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.212+0000 7f60e3fff700 1 --2- 172.21.15.195:0/616973864 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f60f006f640 0x7f60f0156ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:21.273 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.212+0000 7f60e3fff700 1 -- 172.21.15.195:0/616973864 >> v2:172.21.15.202:3300/0 conn(0x7f60f00c4160 msgr2=0x7f60f00c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:10:21.274 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.212+0000 7f60e3fff700 1 --2- 172.21.15.195:0/616973864 >> v2:172.21.15.202:3300/0 conn(0x7f60f00c4160 0x7f60f00c1eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:21.274 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.212+0000 7f60e3fff700 1 -- 172.21.15.195:0/616973864 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60f006ed70 con 0x7f60f00c39e0 2022-06-27T12:10:21.274 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.212+0000 7f60e3fff700 1 -- 172.21.15.195:0/616973864 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7f60e80017f0 con 0x7f60f00c39e0 2022-06-27T12:10:21.275 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.212+0000 7f60e3fff700 1 -- 172.21.15.195:0/616973864 >> v1:172.21.15.195:6789/0 conn(0x7f60f00c39e0 legacy=0x7f60f01535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:21.275 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.212+0000 7f60e3fff700 1 --2- 172.21.15.195:0/616973864 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f60d8003920 0x7f60d8003d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:21.276 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.212+0000 7f60e3fff700 1 --2- 172.21.15.195:0/616973864 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f60f006f640 0x7f60d80046d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:21.276 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.212+0000 7f60e3fff700 1 --2- 172.21.15.195:0/616973864 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f60d8004c10 0x7f60d8005080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:21.277 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.212+0000 7f60e3fff700 1 -- 172.21.15.195:0/616973864 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60f00c1130 con 0x7f60d8004c10 2022-06-27T12:10:21.278 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.212+0000 7f60e3fff700 1 -- 172.21.15.195:0/616973864 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60f006ea90 con 0x7f60d8003920 2022-06-27T12:10:21.278 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.212+0000 7f60e3fff700 1 -- 172.21.15.195:0/616973864 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60d800b920 con 0x7f60f006f640 2022-06-27T12:10:21.279 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.212+0000 7f60f56a5700 1 --2- 172.21.15.195:0/616973864 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f60d8003920 0x7f60d8003d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:21.279 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.212+0000 7f60f4ea4700 1 --2- 172.21.15.195:0/616973864 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f60f006f640 0x7f60d80046d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:21.280 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.212+0000 7f60f5ea6700 1 --2- 172.21.15.195:0/616973864 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f60d8004c10 0x7f60d8005080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:21.280 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.213+0000 7f60f56a5700 1 -- 172.21.15.195:0/616973864 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f60f006f640 msgr2=0x7f60d80046d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:10:21.281 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.213+0000 7f60f56a5700 1 --2- 172.21.15.195:0/616973864 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f60f006f640 0x7f60d80046d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:21.282 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.213+0000 7f60f56a5700 1 -- 172.21.15.195:0/616973864 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f60d8004c10 msgr2=0x7f60d8005080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:10:21.282 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.213+0000 7f60f56a5700 1 --2- 172.21.15.195:0/616973864 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f60d8004c10 0x7f60d8005080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:21.283 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.213+0000 7f60f56a5700 1 -- 172.21.15.195:0/616973864 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60d8003430 con 0x7f60d8003920 2022-06-27T12:10:21.284 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.213+0000 7f60f5ea6700 1 --2- 172.21.15.195:0/616973864 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f60d8004c10 0x7f60d8005080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:10:21.284 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.213+0000 7f60f4ea4700 1 --2- 172.21.15.195:0/616973864 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f60f006f640 0x7f60d80046d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:10:21.285 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.213+0000 7f60f56a5700 1 --2- 172.21.15.195:0/616973864 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f60d8003920 0x7f60d8003d30 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f60e4009ac0 tx=0x7f60e4009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f845680f4890dd server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:21.285 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.213+0000 7f60e3fff700 1 -- 172.21.15.195:0/616973864 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60e4009470 con 0x7f60d8003920 2022-06-27T12:10:21.286 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.213+0000 7f60e3fff700 1 -- 172.21.15.195:0/616973864 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60e400e640 con 0x7f60d8003920 2022-06-27T12:10:21.286 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.213+0000 7f60e3fff700 1 -- 172.21.15.195:0/616973864 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60e400e9e0 con 0x7f60d8003920 2022-06-27T12:10:21.287 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.213+0000 7f60f7909700 1 -- 172.21.15.195:0/616973864 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f60d8003920 msgr2=0x7f60d8003d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:21.288 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.213+0000 7f60f7909700 1 --2- 172.21.15.195:0/616973864 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f60d8003920 0x7f60d8003d30 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f60e4009ac0 tx=0x7f60e4009060 comp rx=0 tx=0).stop 2022-06-27T12:10:21.288 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.213+0000 7f60f5ea6700 1 -- 172.21.15.195:0/616973864 reap_dead start 2022-06-27T12:10:21.289 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.213+0000 7f60f7909700 1 -- 172.21.15.195:0/616973864 shutdown_connections 2022-06-27T12:10:21.289 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.213+0000 7f60f7909700 1 -- 172.21.15.195:0/616973864 >> 172.21.15.195:0/616973864 conn(0x7f60f00c6350 msgr2=0x7f60f00c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:10:21.290 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.214+0000 7f60f7909700 1 -- 172.21.15.195:0/616973864 shutdown_connections 2022-06-27T12:10:21.290 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.214+0000 7f60f7909700 1 -- 172.21.15.195:0/616973864 wait complete. 2022-06-27T12:10:21.290 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.215+0000 7f60f7909700 1 Processor -- start 2022-06-27T12:10:21.291 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.215+0000 7f60f7909700 1 -- start start 2022-06-27T12:10:21.291 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.216+0000 7f60f7909700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f60d8003920 0x7f60f016a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:21.292 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.216+0000 7f60f7909700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f60f0155a70 0x7f60f00c8500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:21.292 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.216+0000 7f60f7909700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f60f014b250 0x7f60f014b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:21.293 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.216+0000 7f60f7909700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60f00bc5f0 con 0x7f60f0155a70 2022-06-27T12:10:21.293 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.216+0000 7f60f7909700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60f00c0310 con 0x7f60d8003920 2022-06-27T12:10:21.294 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.216+0000 7f60f7909700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60f00c05f0 con 0x7f60f014b250 2022-06-27T12:10:21.295 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.216+0000 7f60f4ea4700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f60f0155a70 0x7f60f00c8500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:21.295 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.216+0000 7f60f4ea4700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f60f0155a70 0x7f60f00c8500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:49896/0 (socket says 172.21.15.195:49896) 2022-06-27T12:10:21.296 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.216+0000 7f60f4ea4700 1 -- 172.21.15.195:0/2156926901 learned_addr learned my addr 172.21.15.195:0/2156926901 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:10:21.296 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.216+0000 7f60f56a5700 1 --2- 172.21.15.195:0/2156926901 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f60d8003920 0x7f60f016a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:21.297 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.216+0000 7f60f5ea6700 1 --2- 172.21.15.195:0/2156926901 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f60f014b250 0x7f60f014b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:21.297 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.216+0000 7f60f4ea4700 1 -- 172.21.15.195:0/2156926901 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f60f014b250 msgr2=0x7f60f014b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:21.297 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.216+0000 7f60f4ea4700 1 --2- 172.21.15.195:0/2156926901 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f60f014b250 0x7f60f014b6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:21.298 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.216+0000 7f60f4ea4700 1 -- 172.21.15.195:0/2156926901 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f60d8003920 msgr2=0x7f60f016a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:21.299 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.216+0000 7f60f4ea4700 1 --2- 172.21.15.195:0/2156926901 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f60d8003920 0x7f60f016a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:21.299 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.216+0000 7f60f4ea4700 1 -- 172.21.15.195:0/2156926901 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60f00ba4d0 con 0x7f60f0155a70 2022-06-27T12:10:21.301 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.216+0000 7f60f5ea6700 1 --2- 172.21.15.195:0/2156926901 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f60f014b250 0x7f60f014b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:10:21.302 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.216+0000 7f60f4ea4700 1 --2- 172.21.15.195:0/2156926901 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f60f0155a70 0x7f60f00c8500 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f60ec011af0 tx=0x7f60ec009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:21.302 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.216+0000 7f60e17fa700 1 -- 172.21.15.195:0/2156926901 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60ec004020 con 0x7f60f0155a70 2022-06-27T12:10:21.303 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.216+0000 7f60e17fa700 1 -- 172.21.15.195:0/2156926901 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60ec004180 con 0x7f60f0155a70 2022-06-27T12:10:21.303 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.216+0000 7f60e17fa700 1 -- 172.21.15.195:0/2156926901 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60ec00fa10 con 0x7f60f0155a70 2022-06-27T12:10:21.304 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.216+0000 7f60f7909700 1 -- 172.21.15.195:0/2156926901 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60f00ba7b0 con 0x7f60f0155a70 2022-06-27T12:10:21.304 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.216+0000 7f60f7909700 1 -- 172.21.15.195:0/2156926901 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60f0166460 con 0x7f60f0155a70 2022-06-27T12:10:21.305 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.217+0000 7f60f7909700 1 -- 172.21.15.195:0/2156926901 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60f00ba330 con 0x7f60f0155a70 2022-06-27T12:10:21.322 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.218+0000 7f60e17fa700 1 -- 172.21.15.195:0/2156926901 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f60ec003520 con 0x7f60f0155a70 2022-06-27T12:10:21.322 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.218+0000 7f60e17fa700 1 --2- 172.21.15.195:0/2156926901 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f60d0063d30 0x7f60d00661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:21.323 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.218+0000 7f60e17fa700 1 -- 172.21.15.195:0/2156926901 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7234+0+0 (secure 0 0 0) 0x7f60ec018070 con 0x7f60f0155a70 2022-06-27T12:10:21.323 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.218+0000 7f60f56a5700 1 --2- 172.21.15.195:0/2156926901 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f60d0063d30 0x7f60d00661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:21.324 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.219+0000 7f60f56a5700 1 --2- 172.21.15.195:0/2156926901 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f60d0063d30 0x7f60d00661e0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f60e4009ac0 tx=0x7f60e401a000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:21.324 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.223+0000 7f60e17fa700 1 -- 172.21.15.195:0/2156926901 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f60f00ba330 con 0x7f60f0155a70 2022-06-27T12:10:21.452 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.448+0000 7f60f7909700 1 -- 172.21.15.195:0/2156926901 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f60f00ba330 con 0x7f60f0155a70 2022-06-27T12:10:21.453 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.449+0000 7f60e17fa700 1 -- 172.21.15.195:0/2156926901 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+6125 (secure 0 0 0) 0x7f60f00ba330 con 0x7f60f0155a70 2022-06-27T12:10:21.454 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:10:21.454 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":84,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":5933,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/1342896965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":1342896965},{"type":"v1","addr":"172.21.15.202:6835","nonce":1342896965}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:09:59.334885+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6708},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6708":{"gid":6708,"name":"b","rank":0,"incarnation":79,"state":"up:active","state_seq":27,"addr":"172.21.15.202:6833/702244802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":702244802},{"type":"v1","addr":"172.21.15.202:6833","nonce":702244802}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:10:21.456 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.451+0000 7f60f7909700 1 -- 172.21.15.195:0/2156926901 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f60d0063d30 msgr2=0x7f60d00661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:21.461 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.451+0000 7f60f7909700 1 --2- 172.21.15.195:0/2156926901 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f60d0063d30 0x7f60d00661e0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f60e4009ac0 tx=0x7f60e401a000 comp rx=0 tx=0).stop 2022-06-27T12:10:21.462 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.451+0000 7f60f7909700 1 -- 172.21.15.195:0/2156926901 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f60f0155a70 msgr2=0x7f60f00c8500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:21.462 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.451+0000 7f60f7909700 1 --2- 172.21.15.195:0/2156926901 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f60f0155a70 0x7f60f00c8500 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f60ec011af0 tx=0x7f60ec009060 comp rx=0 tx=0).stop 2022-06-27T12:10:21.463 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.451+0000 7f60f7909700 1 -- 172.21.15.195:0/2156926901 shutdown_connections 2022-06-27T12:10:21.463 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.451+0000 7f60f7909700 1 --2- 172.21.15.195:0/2156926901 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f60d0063d30 0x7f60d00661e0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:21.464 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.451+0000 7f60f7909700 1 --2- 172.21.15.195:0/2156926901 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f60f014b250 0x7f60f014b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:21.464 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.451+0000 7f60f7909700 1 --2- 172.21.15.195:0/2156926901 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f60d8003920 0x7f60f016a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:21.464 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.451+0000 7f60f7909700 1 --2- 172.21.15.195:0/2156926901 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f60f0155a70 0x7f60f00c8500 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:21.465 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.451+0000 7f60f7909700 1 -- 172.21.15.195:0/2156926901 >> 172.21.15.195:0/2156926901 conn(0x7f60f00c6350 msgr2=0x7f60f0152720 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:10:21.465 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.451+0000 7f60f7909700 1 -- 172.21.15.195:0/2156926901 shutdown_connections 2022-06-27T12:10:21.466 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:21.451+0000 7f60f7909700 1 -- 172.21.15.195:0/2156926901 wait complete. 2022-06-27T12:10:21.466 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 84 2022-06-27T12:10:21.483 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2022-06-27T12:10:23.486 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:10:23.622 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.621+0000 7f04ad40d700 1 Processor -- start 2022-06-27T12:10:23.622 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.621+0000 7f04ad40d700 1 -- start start 2022-06-27T12:10:23.635 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.622+0000 7f04ad40d700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f04a8150580 0x7f04a8156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:23.636 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.622+0000 7f04ad40d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f04a806f620 0x7f04a806d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:23.636 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.622+0000 7f04ad40d700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04a80c76b0 con 0x7f04a8150950 2022-06-27T12:10:23.637 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.622+0000 7f04ad40d700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04a8157430 con 0x7f04a806f620 2022-06-27T12:10:23.637 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.622+0000 7f04ad40d700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04a80c8250 con 0x7f04a8150580 2022-06-27T12:10:23.646 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.622+0000 7f04a77fe700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f04a806f620 0x7f04a806d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:23.647 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.622+0000 7f04a77fe700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f04a806f620 0x7f04a806d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:49906/0 (socket says 172.21.15.195:49906) 2022-06-27T12:10:23.647 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.622+0000 7f04a77fe700 1 -- 172.21.15.195:0/2430798634 learned_addr learned my addr 172.21.15.195:0/2430798634 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:10:23.648 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.622+0000 7f04a67fc700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f04a8150950 0x7f04a80c7b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:56090/0 (socket says 172.21.15.195:56090) 2022-06-27T12:10:23.648 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.623+0000 7f04a6ffd700 1 --2- 172.21.15.195:0/2430798634 >> v2:172.21.15.202:3300/0 conn(0x7f04a8150580 0x7f04a8156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:23.649 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.623+0000 7f04a77fe700 1 -- 172.21.15.195:0/2430798634 >> v2:172.21.15.202:3300/0 conn(0x7f04a8150580 msgr2=0x7f04a8156ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:10:23.650 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.623+0000 7f04a77fe700 1 --2- 172.21.15.195:0/2430798634 >> v2:172.21.15.202:3300/0 conn(0x7f04a8150580 0x7f04a8156ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:23.650 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.623+0000 7f04a77fe700 1 -- 172.21.15.195:0/2430798634 >> v1:172.21.15.195:6789/0 conn(0x7f04a8150950 legacy=0x7f04a80c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:23.651 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.623+0000 7f04a5ffb700 1 -- 172.21.15.195:0/2430798634 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3607887218 0 0) 0x7f04a80c76b0 con 0x7f04a8150950 2022-06-27T12:10:23.651 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.623+0000 7f04a77fe700 1 -- 172.21.15.195:0/2430798634 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04a80bad70 con 0x7f04a806f620 2022-06-27T12:10:23.652 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.623+0000 7f04a77fe700 1 --2- 172.21.15.195:0/2430798634 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f04a806f620 0x7f04a806d6a0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f04a80c8470 tx=0x7f04a0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a2df5bf360b7b5e server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:23.652 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.623+0000 7f04a5ffb700 1 -- 172.21.15.195:0/2430798634 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a000e070 con 0x7f04a806f620 2022-06-27T12:10:23.653 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.623+0000 7f04a5ffb700 1 -- 172.21.15.195:0/2430798634 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04a000ba50 con 0x7f04a806f620 2022-06-27T12:10:23.653 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.623+0000 7f04a5ffb700 1 -- 172.21.15.195:0/2430798634 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a00092e0 con 0x7f04a806f620 2022-06-27T12:10:23.654 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.623+0000 7f04ad40d700 1 -- 172.21.15.195:0/2430798634 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f04a806f620 msgr2=0x7f04a806d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:23.654 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.623+0000 7f04ad40d700 1 --2- 172.21.15.195:0/2430798634 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f04a806f620 0x7f04a806d6a0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f04a80c8470 tx=0x7f04a0009840 comp rx=0 tx=0).stop 2022-06-27T12:10:23.654 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.623+0000 7f04ad40d700 1 -- 172.21.15.195:0/2430798634 shutdown_connections 2022-06-27T12:10:23.655 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.623+0000 7f04ad40d700 1 --2- 172.21.15.195:0/2430798634 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f04a806f620 0x7f04a806d6a0 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:23.656 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.623+0000 7f04ad40d700 1 --2- 172.21.15.195:0/2430798634 >> v2:172.21.15.202:3300/0 conn(0x7f04a8150580 0x7f04a8156ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:23.656 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.623+0000 7f04ad40d700 1 -- 172.21.15.195:0/2430798634 >> 172.21.15.195:0/2430798634 conn(0x7f04a8063af0 msgr2=0x7f04a8063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:10:23.656 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.623+0000 7f04ad40d700 1 -- 172.21.15.195:0/2430798634 shutdown_connections 2022-06-27T12:10:23.657 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.624+0000 7f04ad40d700 1 -- 172.21.15.195:0/2430798634 wait complete. 2022-06-27T12:10:23.657 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.625+0000 7f04ad40d700 1 Processor -- start 2022-06-27T12:10:23.658 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.625+0000 7f04ad40d700 1 -- start start 2022-06-27T12:10:23.658 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.626+0000 7f04ad40d700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f04a806f620 0x7f04a814d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:23.659 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.626+0000 7f04ad40d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f04a8150580 0x7f04a80bce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:23.659 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.626+0000 7f04ad40d700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f04a8150950 0x7f04a80bd350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:23.660 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.626+0000 7f04ad40d700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04a80c8930 con 0x7f04a8150580 2022-06-27T12:10:23.660 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.626+0000 7f04ad40d700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04a80bbdc0 con 0x7f04a806f620 2022-06-27T12:10:23.661 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.626+0000 7f04ad40d700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04a8166e20 con 0x7f04a8150950 2022-06-27T12:10:23.667 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.626+0000 7f04a67fc700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f04a8150580 0x7f04a80bce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:23.667 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.626+0000 7f04a67fc700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f04a8150580 0x7f04a80bce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:49910/0 (socket says 172.21.15.195:49910) 2022-06-27T12:10:23.668 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.626+0000 7f04a67fc700 1 -- 172.21.15.195:0/3175620288 learned_addr learned my addr 172.21.15.195:0/3175620288 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:10:23.668 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.626+0000 7f04a6ffd700 1 --2- 172.21.15.195:0/3175620288 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f04a806f620 0x7f04a814d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:23.668 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.626+0000 7f04a77fe700 1 --2- 172.21.15.195:0/3175620288 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f04a8150950 0x7f04a80bd350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:23.669 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.627+0000 7f04a67fc700 1 -- 172.21.15.195:0/3175620288 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f04a8150950 msgr2=0x7f04a80bd350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:23.669 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.627+0000 7f04a67fc700 1 --2- 172.21.15.195:0/3175620288 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f04a8150950 0x7f04a80bd350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:23.670 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.627+0000 7f04a67fc700 1 -- 172.21.15.195:0/3175620288 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f04a806f620 msgr2=0x7f04a814d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:23.670 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.627+0000 7f04a67fc700 1 --2- 172.21.15.195:0/3175620288 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f04a806f620 0x7f04a814d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:23.671 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.627+0000 7f04a67fc700 1 -- 172.21.15.195:0/3175620288 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04a0009ae0 con 0x7f04a8150580 2022-06-27T12:10:23.671 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.627+0000 7f04a77fe700 1 --2- 172.21.15.195:0/3175620288 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f04a8150950 0x7f04a80bd350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T12:10:23.672 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.627+0000 7f04a67fc700 1 --2- 172.21.15.195:0/3175620288 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f04a8150580 0x7f04a80bce10 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f049800c590 tx=0x7f0498007cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:23.672 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.627+0000 7f048effd700 1 -- 172.21.15.195:0/3175620288 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0498007360 con 0x7f04a8150580 2022-06-27T12:10:23.673 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.627+0000 7f048effd700 1 -- 172.21.15.195:0/3175620288 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0498007500 con 0x7f04a8150580 2022-06-27T12:10:23.673 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.627+0000 7f04ad40d700 1 -- 172.21.15.195:0/3175620288 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04a8166b40 con 0x7f04a8150580 2022-06-27T12:10:23.674 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.627+0000 7f048effd700 1 -- 172.21.15.195:0/3175620288 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0498004940 con 0x7f04a8150580 2022-06-27T12:10:23.674 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.627+0000 7f04ad40d700 1 -- 172.21.15.195:0/3175620288 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04a80bbae0 con 0x7f04a8150580 2022-06-27T12:10:23.674 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.628+0000 7f04ad40d700 1 -- 172.21.15.195:0/3175620288 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04a80722c0 con 0x7f04a8150580 2022-06-27T12:10:23.675 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.628+0000 7f048effd700 1 -- 172.21.15.195:0/3175620288 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f0498010030 con 0x7f04a8150580 2022-06-27T12:10:23.675 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.629+0000 7f048effd700 1 --2- 172.21.15.195:0/3175620288 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0488063d00 0x7f04880661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:23.676 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.629+0000 7f048effd700 1 -- 172.21.15.195:0/3175620288 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7234+0+0 (secure 0 0 0) 0x7f04980808a0 con 0x7f04a8150580 2022-06-27T12:10:23.676 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.629+0000 7f04a6ffd700 1 --2- 172.21.15.195:0/3175620288 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0488063d00 0x7f04880661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:23.677 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.629+0000 7f04a6ffd700 1 --2- 172.21.15.195:0/3175620288 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0488063d00 0x7f04880661b0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f049c00aa30 tx=0x7f049c008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:23.677 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.633+0000 7f048effd700 1 -- 172.21.15.195:0/3175620288 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f049804f070 con 0x7f04a8150580 2022-06-27T12:10:23.858 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.858+0000 7f04ad40d700 1 -- 172.21.15.195:0/3175620288 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f04a806fe20 con 0x7f04a8150580 2022-06-27T12:10:23.859 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.858+0000 7f048effd700 1 -- 172.21.15.195:0/3175620288 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+6125 (secure 0 0 0) 0x7f0498052690 con 0x7f04a8150580 2022-06-27T12:10:23.861 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:10:23.861 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":84,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":5933,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/1342896965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":1342896965},{"type":"v1","addr":"172.21.15.202:6835","nonce":1342896965}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:09:59.334885+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6708},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6708":{"gid":6708,"name":"b","rank":0,"incarnation":79,"state":"up:active","state_seq":27,"addr":"172.21.15.202:6833/702244802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":702244802},{"type":"v1","addr":"172.21.15.202:6833","nonce":702244802}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:10:23.863 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.861+0000 7f04ad40d700 1 -- 172.21.15.195:0/3175620288 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0488063d00 msgr2=0x7f04880661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:23.863 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.861+0000 7f04ad40d700 1 --2- 172.21.15.195:0/3175620288 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0488063d00 0x7f04880661b0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f049c00aa30 tx=0x7f049c008040 comp rx=0 tx=0).stop 2022-06-27T12:10:23.863 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.861+0000 7f04ad40d700 1 -- 172.21.15.195:0/3175620288 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f04a8150580 msgr2=0x7f04a80bce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:23.863 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.861+0000 7f04ad40d700 1 --2- 172.21.15.195:0/3175620288 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f04a8150580 0x7f04a80bce10 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f049800c590 tx=0x7f0498007cb0 comp rx=0 tx=0).stop 2022-06-27T12:10:23.864 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.861+0000 7f04ad40d700 1 -- 172.21.15.195:0/3175620288 shutdown_connections 2022-06-27T12:10:23.864 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.861+0000 7f04ad40d700 1 --2- 172.21.15.195:0/3175620288 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0488063d00 0x7f04880661b0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:23.864 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.861+0000 7f04ad40d700 1 --2- 172.21.15.195:0/3175620288 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f04a8150950 0x7f04a80bd350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:23.864 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.861+0000 7f04ad40d700 1 --2- 172.21.15.195:0/3175620288 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f04a806f620 0x7f04a814d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:23.865 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.861+0000 7f04ad40d700 1 --2- 172.21.15.195:0/3175620288 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f04a8150580 0x7f04a80bce10 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:23.865 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.861+0000 7f04ad40d700 1 -- 172.21.15.195:0/3175620288 >> 172.21.15.195:0/3175620288 conn(0x7f04a8063af0 msgr2=0x7f04a8069b00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:10:23.865 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.861+0000 7f04ad40d700 1 -- 172.21.15.195:0/3175620288 shutdown_connections 2022-06-27T12:10:23.866 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:23.861+0000 7f04ad40d700 1 -- 172.21.15.195:0/3175620288 wait complete. 2022-06-27T12:10:23.866 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 84 2022-06-27T12:10:23.873 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2022-06-27T12:10:25.887 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:10:26.030 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.027+0000 7fd5f60d5700 1 Processor -- start 2022-06-27T12:10:26.031 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.028+0000 7fd5f60d5700 1 -- start start 2022-06-27T12:10:26.031 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.028+0000 7fd5f60d5700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fd5f006fef0 0x7fd5f01570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:26.031 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.028+0000 7fd5f60d5700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd5f00c34a0 0x7fd5f00c2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:26.032 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.028+0000 7fd5f60d5700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5f006fba0 con 0x7fd5f006e850 2022-06-27T12:10:26.032 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.028+0000 7fd5f60d5700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5f01575e0 con 0x7fd5f006fef0 2022-06-27T12:10:26.032 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.028+0000 7fd5f60d5700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5f006f5e0 con 0x7fd5f00c34a0 2022-06-27T12:10:26.032 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.028+0000 7fd5ef7fe700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fd5f006e850 0x7fd5f0069de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:56112/0 (socket says 172.21.15.195:56112) 2022-06-27T12:10:26.033 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.029+0000 7fd5ef7fe700 1 -- 172.21.15.195:0/3928855221 learned_addr learned my addr 172.21.15.195:0/3928855221 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:10:26.033 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.029+0000 7fd5eeffd700 1 --2- 172.21.15.195:0/3928855221 >> v2:172.21.15.202:3300/0 conn(0x7fd5f006fef0 0x7fd5f01570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:26.033 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.029+0000 7fd5effff700 1 --2- 172.21.15.195:0/3928855221 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd5f00c34a0 0x7fd5f00c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:26.055 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.029+0000 7fd5ee7fc700 1 -- 172.21.15.195:0/3928855221 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 259045476 0 0) 0x7fd5f006fba0 con 0x7fd5f006e850 2022-06-27T12:10:26.055 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.029+0000 7fd5ee7fc700 1 -- 172.21.15.195:0/3928855221 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5d0003430 con 0x7fd5f006e850 2022-06-27T12:10:26.056 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.029+0000 7fd5effff700 1 -- 172.21.15.195:0/3928855221 >> v2:172.21.15.202:3300/0 conn(0x7fd5f006fef0 msgr2=0x7fd5f01570a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:10:26.056 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.029+0000 7fd5effff700 1 --2- 172.21.15.195:0/3928855221 >> v2:172.21.15.202:3300/0 conn(0x7fd5f006fef0 0x7fd5f01570a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:26.056 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.029+0000 7fd5effff700 1 -- 172.21.15.195:0/3928855221 >> v1:172.21.15.195:6789/0 conn(0x7fd5f006e850 legacy=0x7fd5f0069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:26.057 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.029+0000 7fd5ee7fc700 1 -- 172.21.15.195:0/3928855221 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2932832449 0 0) 0x7fd5d0003430 con 0x7fd5f006e850 2022-06-27T12:10:26.088 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.029+0000 7fd5effff700 1 -- 172.21.15.195:0/3928855221 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5f006f8c0 con 0x7fd5f00c34a0 2022-06-27T12:10:26.088 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.029+0000 7fd5eeffd700 1 --2- 172.21.15.195:0/3928855221 >> v2:172.21.15.202:3300/0 conn(0x7fd5f006fef0 0x7fd5f01570a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:10:26.088 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.029+0000 7fd5effff700 1 --2- 172.21.15.195:0/3928855221 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd5f00c34a0 0x7fd5f00c2920 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd5e8009e60 tx=0x7fd5e8009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=738ac65ce9a0aca5 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:26.089 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.029+0000 7fd5ee7fc700 1 -- 172.21.15.195:0/3928855221 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5e800e070 con 0x7fd5f00c34a0 2022-06-27T12:10:26.089 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.029+0000 7fd5ee7fc700 1 -- 172.21.15.195:0/3928855221 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5e80038a0 con 0x7fd5f00c34a0 2022-06-27T12:10:26.089 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.029+0000 7fd5ee7fc700 1 -- 172.21.15.195:0/3928855221 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5d8001470 con 0x7fd5f00c34a0 2022-06-27T12:10:26.090 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.030+0000 7fd5f60d5700 1 -- 172.21.15.195:0/3928855221 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd5f00c34a0 msgr2=0x7fd5f00c2920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:26.090 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.030+0000 7fd5f60d5700 1 --2- 172.21.15.195:0/3928855221 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd5f00c34a0 0x7fd5f00c2920 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd5e8009e60 tx=0x7fd5e8009780 comp rx=0 tx=0).stop 2022-06-27T12:10:26.090 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.030+0000 7fd5f60d5700 1 -- 172.21.15.195:0/3928855221 shutdown_connections 2022-06-27T12:10:26.090 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.030+0000 7fd5f60d5700 1 --2- 172.21.15.195:0/3928855221 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd5f00c34a0 0x7fd5f00c2920 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:26.091 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.030+0000 7fd5f60d5700 1 --2- 172.21.15.195:0/3928855221 >> v2:172.21.15.202:3300/0 conn(0x7fd5f006fef0 0x7fd5f01570a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:26.091 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.030+0000 7fd5f60d5700 1 -- 172.21.15.195:0/3928855221 >> 172.21.15.195:0/3928855221 conn(0x7fd5f00c6350 msgr2=0x7fd5f00c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:10:26.092 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.030+0000 7fd5f60d5700 1 -- 172.21.15.195:0/3928855221 shutdown_connections 2022-06-27T12:10:26.092 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.030+0000 7fd5f60d5700 1 -- 172.21.15.195:0/3928855221 wait complete. 2022-06-27T12:10:26.093 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.031+0000 7fd5f60d5700 1 Processor -- start 2022-06-27T12:10:26.093 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.031+0000 7fd5f60d5700 1 -- start start 2022-06-27T12:10:26.098 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.032+0000 7fd5f60d5700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd5f006e850 0x7fd5f0164240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:26.099 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.032+0000 7fd5f60d5700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd5f00c34a0 0x7fd5f016a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:26.099 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.032+0000 7fd5f60d5700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd5f0164780 0x7fd5f0155670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:26.100 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.032+0000 7fd5f60d5700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5f0153fb0 con 0x7fd5f006e850 2022-06-27T12:10:26.100 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.032+0000 7fd5f60d5700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5f016d980 con 0x7fd5f00c34a0 2022-06-27T12:10:26.101 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.032+0000 7fd5f60d5700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5f016e700 con 0x7fd5f0164780 2022-06-27T12:10:26.101 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.032+0000 7fd5ef7fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd5f006e850 0x7fd5f0164240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:26.102 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.032+0000 7fd5ef7fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd5f006e850 0x7fd5f0164240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:49930/0 (socket says 172.21.15.195:49930) 2022-06-27T12:10:26.102 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.032+0000 7fd5ef7fe700 1 -- 172.21.15.195:0/3838119459 learned_addr learned my addr 172.21.15.195:0/3838119459 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:10:26.102 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.032+0000 7fd5eeffd700 1 --2- 172.21.15.195:0/3838119459 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd5f00c34a0 0x7fd5f016a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:26.103 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.032+0000 7fd5effff700 1 --2- 172.21.15.195:0/3838119459 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd5f0164780 0x7fd5f0155670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:26.103 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.032+0000 7fd5ef7fe700 1 -- 172.21.15.195:0/3838119459 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd5f0164780 msgr2=0x7fd5f0155670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:26.104 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.032+0000 7fd5ef7fe700 1 --2- 172.21.15.195:0/3838119459 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd5f0164780 0x7fd5f0155670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:26.104 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.032+0000 7fd5ef7fe700 1 -- 172.21.15.195:0/3838119459 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd5f00c34a0 msgr2=0x7fd5f016a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:26.105 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.032+0000 7fd5ef7fe700 1 --2- 172.21.15.195:0/3838119459 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd5f00c34a0 0x7fd5f016a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:26.105 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.032+0000 7fd5ef7fe700 1 -- 172.21.15.195:0/3838119459 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5e00092e0 con 0x7fd5f006e850 2022-06-27T12:10:26.106 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.032+0000 7fd5effff700 1 --2- 172.21.15.195:0/3838119459 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd5f0164780 0x7fd5f0155670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:10:26.106 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.033+0000 7fd5ef7fe700 1 --2- 172.21.15.195:0/3838119459 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd5f006e850 0x7fd5f0164240 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fd5d800c810 tx=0x7fd5d8001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:26.106 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.033+0000 7fd5e77fe700 1 -- 172.21.15.195:0/3838119459 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5d8007360 con 0x7fd5f006e850 2022-06-27T12:10:26.107 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.033+0000 7fd5e77fe700 1 -- 172.21.15.195:0/3838119459 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5d80074e0 con 0x7fd5f006e850 2022-06-27T12:10:26.107 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.033+0000 7fd5f60d5700 1 -- 172.21.15.195:0/3838119459 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5e8009a20 con 0x7fd5f006e850 2022-06-27T12:10:26.108 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.033+0000 7fd5e77fe700 1 -- 172.21.15.195:0/3838119459 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5d8004940 con 0x7fd5f006e850 2022-06-27T12:10:26.109 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.033+0000 7fd5f60d5700 1 -- 172.21.15.195:0/3838119459 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5f0153cd0 con 0x7fd5f006e850 2022-06-27T12:10:26.109 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.034+0000 7fd5f60d5700 1 -- 172.21.15.195:0/3838119459 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5f00722c0 con 0x7fd5f006e850 2022-06-27T12:10:26.110 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.034+0000 7fd5e77fe700 1 -- 172.21.15.195:0/3838119459 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fd5d801a030 con 0x7fd5f006e850 2022-06-27T12:10:26.110 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.034+0000 7fd5e77fe700 1 --2- 172.21.15.195:0/3838119459 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd5d4063d00 0x7fd5d40661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:26.110 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.035+0000 7fd5e77fe700 1 -- 172.21.15.195:0/3838119459 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7234+0+0 (secure 0 0 0) 0x7fd5d8080230 con 0x7fd5f006e850 2022-06-27T12:10:26.111 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.035+0000 7fd5eeffd700 1 --2- 172.21.15.195:0/3838119459 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd5d4063d00 0x7fd5d40661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:26.111 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.035+0000 7fd5eeffd700 1 --2- 172.21.15.195:0/3838119459 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd5d4063d00 0x7fd5d40661b0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fd5f0159ad0 tx=0x7fd5e0009060 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:26.112 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.039+0000 7fd5e77fe700 1 -- 172.21.15.195:0/3838119459 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fd5d804e980 con 0x7fd5f006e850 2022-06-27T12:10:26.278 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.266+0000 7fd5f60d5700 1 -- 172.21.15.195:0/3838119459 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd5f006f240 con 0x7fd5f006e850 2022-06-27T12:10:26.279 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.266+0000 7fd5e77fe700 1 -- 172.21.15.195:0/3838119459 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+6125 (secure 0 0 0) 0x7fd5d8018070 con 0x7fd5f006e850 2022-06-27T12:10:26.279 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:10:26.280 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":84,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":5933,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/1342896965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":1342896965},{"type":"v1","addr":"172.21.15.202:6835","nonce":1342896965}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:09:59.334885+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6708},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6708":{"gid":6708,"name":"b","rank":0,"incarnation":79,"state":"up:active","state_seq":27,"addr":"172.21.15.202:6833/702244802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":702244802},{"type":"v1","addr":"172.21.15.202:6833","nonce":702244802}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:10:26.301 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.268+0000 7fd5f60d5700 1 -- 172.21.15.195:0/3838119459 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd5d4063d00 msgr2=0x7fd5d40661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:26.301 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.268+0000 7fd5f60d5700 1 --2- 172.21.15.195:0/3838119459 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd5d4063d00 0x7fd5d40661b0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fd5f0159ad0 tx=0x7fd5e0009060 comp rx=0 tx=0).stop 2022-06-27T12:10:26.302 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.268+0000 7fd5f60d5700 1 -- 172.21.15.195:0/3838119459 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd5f006e850 msgr2=0x7fd5f0164240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:26.305 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.268+0000 7fd5f60d5700 1 --2- 172.21.15.195:0/3838119459 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd5f006e850 0x7fd5f0164240 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fd5d800c810 tx=0x7fd5d8001a00 comp rx=0 tx=0).stop 2022-06-27T12:10:26.305 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.268+0000 7fd5f60d5700 1 -- 172.21.15.195:0/3838119459 shutdown_connections 2022-06-27T12:10:26.305 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.268+0000 7fd5f60d5700 1 --2- 172.21.15.195:0/3838119459 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd5d4063d00 0x7fd5d40661b0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:26.306 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.268+0000 7fd5f60d5700 1 --2- 172.21.15.195:0/3838119459 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd5f006e850 0x7fd5f0164240 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:26.306 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.268+0000 7fd5f60d5700 1 --2- 172.21.15.195:0/3838119459 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd5f0164780 0x7fd5f0155670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:26.306 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.268+0000 7fd5f60d5700 1 --2- 172.21.15.195:0/3838119459 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd5f00c34a0 0x7fd5f016a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:26.306 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.268+0000 7fd5f60d5700 1 -- 172.21.15.195:0/3838119459 >> 172.21.15.195:0/3838119459 conn(0x7fd5f00c6350 msgr2=0x7fd5f00c6720 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:10:26.307 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.268+0000 7fd5f60d5700 1 -- 172.21.15.195:0/3838119459 shutdown_connections 2022-06-27T12:10:26.307 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:26.268+0000 7fd5f60d5700 1 -- 172.21.15.195:0/3838119459 wait complete. 2022-06-27T12:10:26.307 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 84 2022-06-27T12:10:26.308 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2022-06-27T12:10:28.310 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:10:28.504 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.444+0000 7fb7aba6e700 1 Processor -- start 2022-06-27T12:10:28.504 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.444+0000 7fb7aba6e700 1 -- start start 2022-06-27T12:10:28.504 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.445+0000 7fb7aba6e700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb7a4150580 0x7fb7a4156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:28.505 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.445+0000 7fb7aba6e700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fb7a406f620 0x7fb7a406d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:28.505 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.445+0000 7fb7aba6e700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7a40c76b0 con 0x7fb7a4150950 2022-06-27T12:10:28.505 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.445+0000 7fb7aba6e700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7a4157430 con 0x7fb7a406f620 2022-06-27T12:10:28.514 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.445+0000 7fb7aba6e700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a40c8250 con 0x7fb7a4150580 2022-06-27T12:10:28.515 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.445+0000 7fb7a9009700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fb7a4150950 0x7fb7a40c7b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:56128/0 (socket says 172.21.15.195:56128) 2022-06-27T12:10:28.516 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.445+0000 7fb7a9009700 1 -- 172.21.15.195:0/554590609 learned_addr learned my addr 172.21.15.195:0/554590609 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:10:28.518 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.446+0000 7fb7aa00b700 1 --2- 172.21.15.195:0/554590609 >> v2:172.21.15.202:3300/0 conn(0x7fb7a406f620 0x7fb7a406d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:28.518 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.446+0000 7fb7a980a700 1 --2- 172.21.15.195:0/554590609 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb7a4150580 0x7fb7a4156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:28.519 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.446+0000 7fb7a8808700 1 -- 172.21.15.195:0/554590609 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 509092410 0 0) 0x7fb7a40c76b0 con 0x7fb7a4150950 2022-06-27T12:10:28.519 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.446+0000 7fb7a8808700 1 -- 172.21.15.195:0/554590609 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb78c003430 con 0x7fb7a4150950 2022-06-27T12:10:28.520 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.446+0000 7fb7a8808700 1 -- 172.21.15.195:0/554590609 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3137158215 0 0) 0x7fb78c003430 con 0x7fb7a4150950 2022-06-27T12:10:28.520 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.446+0000 7fb7a8808700 1 -- 172.21.15.195:0/554590609 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb7a4150580 msgr2=0x7fb7a4156ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:10:28.520 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.446+0000 7fb7a8808700 1 --2- 172.21.15.195:0/554590609 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb7a4150580 0x7fb7a4156ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:28.521 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.446+0000 7fb7a8808700 1 -- 172.21.15.195:0/554590609 >> v2:172.21.15.202:3300/0 conn(0x7fb7a406f620 msgr2=0x7fb7a406d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:10:28.522 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.446+0000 7fb7a8808700 1 --2- 172.21.15.195:0/554590609 >> v2:172.21.15.202:3300/0 conn(0x7fb7a406f620 0x7fb7a406d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:28.522 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.446+0000 7fb7a8808700 1 -- 172.21.15.195:0/554590609 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7a40bad70 con 0x7fb7a4150950 2022-06-27T12:10:28.523 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.446+0000 7fb7a8808700 1 -- 172.21.15.195:0/554590609 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7fb7a0001470 con 0x7fb7a4150950 2022-06-27T12:10:28.523 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.446+0000 7fb7a8808700 1 -- 172.21.15.195:0/554590609 >> v1:172.21.15.195:6789/0 conn(0x7fb7a4150950 legacy=0x7fb7a40c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:28.525 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.446+0000 7fb7a8808700 1 --2- 172.21.15.195:0/554590609 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb78c003920 0x7fb78c003d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:28.526 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.446+0000 7fb7a8808700 1 --2- 172.21.15.195:0/554590609 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb7a4150580 0x7fb78c0046d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:28.526 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.446+0000 7fb7a8808700 1 --2- 172.21.15.195:0/554590609 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb78c004c10 0x7fb78c005080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:28.527 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.446+0000 7fb7a8808700 1 -- 172.21.15.195:0/554590609 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a40c8250 con 0x7fb78c003920 2022-06-27T12:10:28.528 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.446+0000 7fb7a8808700 1 -- 172.21.15.195:0/554590609 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb78c00b920 con 0x7fb78c004c10 2022-06-27T12:10:28.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.446+0000 7fb7a8808700 1 -- 172.21.15.195:0/554590609 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb78c00ba60 con 0x7fb7a4150580 2022-06-27T12:10:28.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.446+0000 7fb7a980a700 1 --2- 172.21.15.195:0/554590609 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb78c003920 0x7fb78c003d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:28.534 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.446+0000 7fb7a9009700 1 --2- 172.21.15.195:0/554590609 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb7a4150580 0x7fb78c0046d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:28.534 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.446+0000 7fb7aa00b700 1 --2- 172.21.15.195:0/554590609 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb78c004c10 0x7fb78c005080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:28.536 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.446+0000 7fb7a980a700 1 -- 172.21.15.195:0/554590609 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb7a4150580 msgr2=0x7fb78c0046d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:10:28.537 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.446+0000 7fb7a980a700 1 --2- 172.21.15.195:0/554590609 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb7a4150580 0x7fb78c0046d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:28.537 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.446+0000 7fb7a980a700 1 -- 172.21.15.195:0/554590609 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb78c004c10 msgr2=0x7fb78c005080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:10:28.538 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.446+0000 7fb7a980a700 1 --2- 172.21.15.195:0/554590609 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb78c004c10 0x7fb78c005080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:28.543 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.446+0000 7fb7a980a700 1 -- 172.21.15.195:0/554590609 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb78c003430 con 0x7fb78c003920 2022-06-27T12:10:28.544 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.446+0000 7fb7a980a700 1 --2- 172.21.15.195:0/554590609 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb78c003920 0x7fb78c003d30 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fb794009ac0 tx=0x7fb794009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25073344e3518894 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:28.544 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.447+0000 7fb7a8808700 1 -- 172.21.15.195:0/554590609 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb794009470 con 0x7fb78c003920 2022-06-27T12:10:28.545 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.447+0000 7fb7a8808700 1 -- 172.21.15.195:0/554590609 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb79400e5e0 con 0x7fb78c003920 2022-06-27T12:10:28.545 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.447+0000 7fb7a8808700 1 -- 172.21.15.195:0/554590609 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb79400e980 con 0x7fb78c003920 2022-06-27T12:10:28.545 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.447+0000 7fb7aba6e700 1 -- 172.21.15.195:0/554590609 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb78c003920 msgr2=0x7fb78c003d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:28.555 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.447+0000 7fb7aba6e700 1 --2- 172.21.15.195:0/554590609 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb78c003920 0x7fb78c003d30 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fb794009ac0 tx=0x7fb794009060 comp rx=0 tx=0).stop 2022-06-27T12:10:28.556 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.447+0000 7fb7aa00b700 1 -- 172.21.15.195:0/554590609 reap_dead start 2022-06-27T12:10:28.556 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.447+0000 7fb7aba6e700 1 -- 172.21.15.195:0/554590609 shutdown_connections 2022-06-27T12:10:28.557 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.447+0000 7fb7aba6e700 1 -- 172.21.15.195:0/554590609 >> 172.21.15.195:0/554590609 conn(0x7fb7a4063af0 msgr2=0x7fb7a4063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:10:28.557 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.447+0000 7fb7aba6e700 1 -- 172.21.15.195:0/554590609 shutdown_connections 2022-06-27T12:10:28.557 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.447+0000 7fb7aba6e700 1 -- 172.21.15.195:0/554590609 wait complete. 2022-06-27T12:10:28.558 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.448+0000 7fb7aba6e700 1 Processor -- start 2022-06-27T12:10:28.558 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.448+0000 7fb7aba6e700 1 -- start start 2022-06-27T12:10:28.559 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.449+0000 7fb7aba6e700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb78c003920 0x7fb7a40c9b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:28.559 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.449+0000 7fb7aba6e700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb7a4166fb0 0x7fb7a4148380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:28.560 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.449+0000 7fb7aba6e700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb7a4149810 0x7fb7a41488c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:28.562 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.449+0000 7fb7aba6e700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a4169b80 con 0x7fb7a4149810 2022-06-27T12:10:28.562 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.449+0000 7fb7aba6e700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a4149ec0 con 0x7fb78c003920 2022-06-27T12:10:28.563 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.449+0000 7fb7aba6e700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a414a1a0 con 0x7fb7a4166fb0 2022-06-27T12:10:28.570 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.449+0000 7fb7aa00b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb7a4149810 0x7fb7a41488c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:28.571 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.449+0000 7fb7aa00b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb7a4149810 0x7fb7a41488c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:49954/0 (socket says 172.21.15.195:49954) 2022-06-27T12:10:28.571 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.449+0000 7fb7a980a700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb78c003920 0x7fb7a40c9b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:28.572 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.449+0000 7fb7aa00b700 1 -- 172.21.15.195:0/94874492 learned_addr learned my addr 172.21.15.195:0/94874492 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:10:28.572 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.449+0000 7fb7a9009700 1 --2- 172.21.15.195:0/94874492 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb7a4166fb0 0x7fb7a4148380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:28.572 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.450+0000 7fb7aa00b700 1 -- 172.21.15.195:0/94874492 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb7a4166fb0 msgr2=0x7fb7a4148380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:28.573 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.450+0000 7fb7aa00b700 1 --2- 172.21.15.195:0/94874492 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb7a4166fb0 0x7fb7a4148380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:28.573 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.450+0000 7fb7aa00b700 1 -- 172.21.15.195:0/94874492 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb78c003920 msgr2=0x7fb7a40c9b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:28.574 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.450+0000 7fb7aa00b700 1 --2- 172.21.15.195:0/94874492 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb78c003920 0x7fb7a40c9b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:28.574 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.450+0000 7fb7aa00b700 1 -- 172.21.15.195:0/94874492 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7a414a480 con 0x7fb7a4149810 2022-06-27T12:10:28.585 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.450+0000 7fb7a980a700 1 --2- 172.21.15.195:0/94874492 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb78c003920 0x7fb7a40c9b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:10:28.586 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.450+0000 7fb7aa00b700 1 --2- 172.21.15.195:0/94874492 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb7a4149810 0x7fb7a41488c0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fb79c0094d0 tx=0x7fb79c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:28.586 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.450+0000 7fb791ffb700 1 -- 172.21.15.195:0/94874492 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb79c00e040 con 0x7fb7a4149810 2022-06-27T12:10:28.587 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.450+0000 7fb791ffb700 1 -- 172.21.15.195:0/94874492 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb79c00f040 con 0x7fb7a4149810 2022-06-27T12:10:28.587 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.450+0000 7fb7aba6e700 1 -- 172.21.15.195:0/94874492 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7a414a760 con 0x7fb7a4149810 2022-06-27T12:10:28.588 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.450+0000 7fb791ffb700 1 -- 172.21.15.195:0/94874492 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb79c00fd70 con 0x7fb7a4149810 2022-06-27T12:10:28.588 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.450+0000 7fb7aba6e700 1 -- 172.21.15.195:0/94874492 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7a4069ed0 con 0x7fb7a4149810 2022-06-27T12:10:28.588 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.451+0000 7fb7aba6e700 1 -- 172.21.15.195:0/94874492 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7a40722c0 con 0x7fb7a4149810 2022-06-27T12:10:28.589 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.452+0000 7fb791ffb700 1 -- 172.21.15.195:0/94874492 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fb79c003ab0 con 0x7fb7a4149810 2022-06-27T12:10:28.589 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.452+0000 7fb791ffb700 1 --2- 172.21.15.195:0/94874492 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb784063d00 0x7fb7840661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:28.590 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.452+0000 7fb791ffb700 1 -- 172.21.15.195:0/94874492 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7234+0+0 (secure 0 0 0) 0x7fb79c081280 con 0x7fb7a4149810 2022-06-27T12:10:28.590 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.452+0000 7fb7a980a700 1 --2- 172.21.15.195:0/94874492 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb784063d00 0x7fb7840661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:28.591 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.452+0000 7fb7a980a700 1 --2- 172.21.15.195:0/94874492 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb784063d00 0x7fb7840661b0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fb794006d90 tx=0x7fb79401a000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:28.591 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.456+0000 7fb791ffb700 1 -- 172.21.15.195:0/94874492 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fb79c04f9d0 con 0x7fb7a4149810 2022-06-27T12:10:28.695 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.675+0000 7fb7aba6e700 1 -- 172.21.15.195:0/94874492 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7a406fe20 con 0x7fb7a4149810 2022-06-27T12:10:28.695 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.676+0000 7fb791ffb700 1 -- 172.21.15.195:0/94874492 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+6125 (secure 0 0 0) 0x7fb79c052ff0 con 0x7fb7a4149810 2022-06-27T12:10:28.696 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:10:28.697 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":84,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":5933,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/1342896965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":1342896965},{"type":"v1","addr":"172.21.15.202:6835","nonce":1342896965}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:09:59.334885+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6708},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6708":{"gid":6708,"name":"b","rank":0,"incarnation":79,"state":"up:active","state_seq":27,"addr":"172.21.15.202:6833/702244802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":702244802},{"type":"v1","addr":"172.21.15.202:6833","nonce":702244802}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:10:28.724 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.678+0000 7fb7aba6e700 1 -- 172.21.15.195:0/94874492 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb784063d00 msgr2=0x7fb7840661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:28.724 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.678+0000 7fb7aba6e700 1 --2- 172.21.15.195:0/94874492 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb784063d00 0x7fb7840661b0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fb794006d90 tx=0x7fb79401a000 comp rx=0 tx=0).stop 2022-06-27T12:10:28.725 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.678+0000 7fb7aba6e700 1 -- 172.21.15.195:0/94874492 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb7a4149810 msgr2=0x7fb7a41488c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:28.725 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.678+0000 7fb7aba6e700 1 --2- 172.21.15.195:0/94874492 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb7a4149810 0x7fb7a41488c0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fb79c0094d0 tx=0x7fb79c00b040 comp rx=0 tx=0).stop 2022-06-27T12:10:28.733 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.678+0000 7fb7aba6e700 1 -- 172.21.15.195:0/94874492 shutdown_connections 2022-06-27T12:10:28.734 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.678+0000 7fb7aba6e700 1 --2- 172.21.15.195:0/94874492 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb784063d00 0x7fb7840661b0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:28.735 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.678+0000 7fb7aba6e700 1 --2- 172.21.15.195:0/94874492 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb7a4149810 0x7fb7a41488c0 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:28.735 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.678+0000 7fb7aba6e700 1 --2- 172.21.15.195:0/94874492 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb78c003920 0x7fb7a40c9b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:28.736 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.678+0000 7fb7aba6e700 1 --2- 172.21.15.195:0/94874492 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb7a4166fb0 0x7fb7a4148380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:28.736 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.678+0000 7fb7aba6e700 1 -- 172.21.15.195:0/94874492 >> 172.21.15.195:0/94874492 conn(0x7fb7a4063af0 msgr2=0x7fb7a414f6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:10:28.736 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.678+0000 7fb7aba6e700 1 -- 172.21.15.195:0/94874492 shutdown_connections 2022-06-27T12:10:28.737 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:28.678+0000 7fb7aba6e700 1 -- 172.21.15.195:0/94874492 wait complete. 2022-06-27T12:10:28.739 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 84 2022-06-27T12:10:28.740 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2022-06-27T12:10:30.742 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:10:30.901 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:30.877+0000 7fb07919f700 1 Processor -- start 2022-06-27T12:10:30.902 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:30.877+0000 7fb07919f700 1 -- start start 2022-06-27T12:10:30.902 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:30.878+0000 7fb07919f700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fb07406fef0 0x7fb0741570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:30.903 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:30.878+0000 7fb07919f700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb0740c34a0 0x7fb0740c2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:30.903 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:30.878+0000 7fb07919f700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb07406fba0 con 0x7fb07406e850 2022-06-27T12:10:30.904 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:30.878+0000 7fb07919f700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0741575e0 con 0x7fb07406fef0 2022-06-27T12:10:30.904 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:30.878+0000 7fb07919f700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb07406f5e0 con 0x7fb0740c34a0 2022-06-27T12:10:30.905 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:30.878+0000 7fb07259c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fb07406fef0 0x7fb0741570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:30.906 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:30.878+0000 7fb07259c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fb07406fef0 0x7fb0741570a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:49958/0 (socket says 172.21.15.195:49958) 2022-06-27T12:10:30.906 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:30.878+0000 7fb07259c700 1 -- 172.21.15.195:0/3260989495 learned_addr learned my addr 172.21.15.195:0/3260989495 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:10:30.907 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:30.879+0000 7fb07359e700 1 --2- 172.21.15.195:0/3260989495 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb0740c34a0 0x7fb0740c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:30.907 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:30.879+0000 7fb07259c700 1 -- 172.21.15.195:0/3260989495 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb0740c34a0 msgr2=0x7fb0740c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:10:30.908 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:30.879+0000 7fb07259c700 1 --2- 172.21.15.195:0/3260989495 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb0740c34a0 0x7fb0740c2920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:30.908 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:30.879+0000 7fb07259c700 1 -- 172.21.15.195:0/3260989495 >> v1:172.21.15.202:6789/0 conn(0x7fb07406e850 legacy=0x7fb074069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:30.909 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:30.879+0000 7fb07259c700 1 -- 172.21.15.195:0/3260989495 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb07406f8c0 con 0x7fb07406fef0 2022-06-27T12:10:30.910 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:30.879+0000 7fb07359e700 1 --2- 172.21.15.195:0/3260989495 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb0740c34a0 0x7fb0740c2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:10:30.911 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:30.879+0000 7fb07259c700 1 --2- 172.21.15.195:0/3260989495 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb07406fef0 0x7fb0741570a0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7fb064009860 tx=0x7fb064009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12150709fcae7b17 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:30.912 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:30.879+0000 7fb071d9b700 1 -- 172.21.15.195:0/3260989495 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb06400b710 con 0x7fb07406fef0 2022-06-27T12:10:30.912 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:30.879+0000 7fb071d9b700 1 -- 172.21.15.195:0/3260989495 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb06400b870 con 0x7fb07406fef0 2022-06-27T12:10:30.913 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:30.879+0000 7fb071d9b700 1 -- 172.21.15.195:0/3260989495 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb06400bc30 con 0x7fb07406fef0 2022-06-27T12:10:30.913 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:30.880+0000 7fb07919f700 1 -- 172.21.15.195:0/3260989495 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb07406fef0 msgr2=0x7fb0741570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:30.914 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:30.880+0000 7fb07919f700 1 --2- 172.21.15.195:0/3260989495 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb07406fef0 0x7fb0741570a0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7fb064009860 tx=0x7fb064009120 comp rx=0 tx=0).stop 2022-06-27T12:10:30.914 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:30.880+0000 7fb07919f700 1 -- 172.21.15.195:0/3260989495 shutdown_connections 2022-06-27T12:10:30.914 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:30.880+0000 7fb07919f700 1 --2- 172.21.15.195:0/3260989495 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb0740c34a0 0x7fb0740c2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:30.915 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:30.880+0000 7fb07919f700 1 --2- 172.21.15.195:0/3260989495 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb07406fef0 0x7fb0741570a0 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:30.918 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:30.880+0000 7fb07919f700 1 -- 172.21.15.195:0/3260989495 >> 172.21.15.195:0/3260989495 conn(0x7fb0740c6350 msgr2=0x7fb0740c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:10:30.919 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:30.880+0000 7fb07919f700 1 -- 172.21.15.195:0/3260989495 shutdown_connections 2022-06-27T12:10:30.919 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:30.880+0000 7fb07919f700 1 -- 172.21.15.195:0/3260989495 wait complete. 2022-06-27T12:10:30.920 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:30.881+0000 7fb07919f700 1 Processor -- start 2022-06-27T12:10:30.922 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:30.881+0000 7fb07919f700 1 -- start start 2022-06-27T12:10:30.923 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:30.882+0000 7fb07919f700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb07406e850 0x7fb07414d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:30.923 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:30.882+0000 7fb07919f700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb0740c34a0 0x7fb074155850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:30.924 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:30.882+0000 7fb07919f700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb074150ff0 0x7fb074151700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:30.924 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:30.882+0000 7fb07919f700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb07414d790 con 0x7fb0740c34a0 2022-06-27T12:10:30.925 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:30.882+0000 7fb07919f700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb07414cdc0 con 0x7fb07406e850 2022-06-27T12:10:30.925 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:30.882+0000 7fb07919f700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb074153b80 con 0x7fb074150ff0 2022-06-27T12:10:30.926 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:30.882+0000 7fb07259c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb0740c34a0 0x7fb074155850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:30.927 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:30.882+0000 7fb07259c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb0740c34a0 0x7fb074155850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:49966/0 (socket says 172.21.15.195:49966) 2022-06-27T12:10:30.939 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:30.882+0000 7fb07259c700 1 -- 172.21.15.195:0/3291841286 learned_addr learned my addr 172.21.15.195:0/3291841286 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:10:30.939 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:30.882+0000 7fb072d9d700 1 --2- 172.21.15.195:0/3291841286 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb07406e850 0x7fb07414d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:30.940 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:30.882+0000 7fb07359e700 1 --2- 172.21.15.195:0/3291841286 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb074150ff0 0x7fb074151700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:30.940 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:30.882+0000 7fb07259c700 1 -- 172.21.15.195:0/3291841286 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb074150ff0 msgr2=0x7fb074151700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:30.941 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:30.882+0000 7fb07259c700 1 --2- 172.21.15.195:0/3291841286 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb074150ff0 0x7fb074151700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:30.941 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:30.882+0000 7fb07259c700 1 -- 172.21.15.195:0/3291841286 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb07406e850 msgr2=0x7fb07414d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:30.942 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:30.882+0000 7fb07259c700 1 --2- 172.21.15.195:0/3291841286 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb07406e850 0x7fb07414d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:30.943 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:30.882+0000 7fb07259c700 1 -- 172.21.15.195:0/3291841286 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0640093a0 con 0x7fb0740c34a0 2022-06-27T12:10:30.943 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:30.882+0000 7fb07259c700 1 --2- 172.21.15.195:0/3291841286 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb0740c34a0 0x7fb074155850 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7fb064009fb0 tx=0x7fb064017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:30.944 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:30.882+0000 7fb05affd700 1 -- 172.21.15.195:0/3291841286 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb06400b9e0 con 0x7fb0740c34a0 2022-06-27T12:10:30.944 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:30.883+0000 7fb05affd700 1 -- 172.21.15.195:0/3291841286 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb06400bb40 con 0x7fb0740c34a0 2022-06-27T12:10:30.945 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:30.883+0000 7fb07919f700 1 -- 172.21.15.195:0/3291841286 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb074153dd0 con 0x7fb0740c34a0 2022-06-27T12:10:30.945 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:30.883+0000 7fb05affd700 1 -- 172.21.15.195:0/3291841286 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb064004280 con 0x7fb0740c34a0 2022-06-27T12:10:30.946 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:30.883+0000 7fb07919f700 1 -- 172.21.15.195:0/3291841286 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb07414c770 con 0x7fb0740c34a0 2022-06-27T12:10:30.946 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:30.884+0000 7fb07919f700 1 -- 172.21.15.195:0/3291841286 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb07414cf70 con 0x7fb0740c34a0 2022-06-27T12:10:30.947 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:30.884+0000 7fb05affd700 1 -- 172.21.15.195:0/3291841286 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fb06402b030 con 0x7fb0740c34a0 2022-06-27T12:10:30.947 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:30.885+0000 7fb05affd700 1 --2- 172.21.15.195:0/3291841286 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb054063d00 0x7fb0540661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:30.947 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:30.885+0000 7fb05affd700 1 -- 172.21.15.195:0/3291841286 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7320+0+0 (secure 0 0 0) 0x7fb0640827b0 con 0x7fb0740c34a0 2022-06-27T12:10:30.948 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:30.885+0000 7fb072d9d700 1 --2- 172.21.15.195:0/3291841286 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb054063d00 0x7fb0540661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:30.949 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:30.885+0000 7fb072d9d700 1 --2- 172.21.15.195:0/3291841286 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb054063d00 0x7fb0540661b0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fb0680017e0 tx=0x7fb068009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:30.949 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:30.888+0000 7fb05affd700 1 -- 172.21.15.195:0/3291841286 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fb064050eb0 con 0x7fb0740c34a0 2022-06-27T12:10:31.114 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.114+0000 7fb07919f700 1 -- 172.21.15.195:0/3291841286 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb07414cf70 con 0x7fb0740c34a0 2022-06-27T12:10:31.123 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.115+0000 7fb05affd700 1 -- 172.21.15.195:0/3291841286 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+5367 (secure 0 0 0) 0x7fb07414cf70 con 0x7fb0740c34a0 2022-06-27T12:10:31.124 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:10:31.125 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:10:30.134487+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5933},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5933":{"gid":5933,"name":"d","rank":0,"incarnation":85,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6835/1342896965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":1342896965},{"type":"v1","addr":"172.21.15.202:6835","nonce":1342896965}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:10:31.138 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.117+0000 7fb07919f700 1 -- 172.21.15.195:0/3291841286 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb054063d00 msgr2=0x7fb0540661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:31.138 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.117+0000 7fb07919f700 1 --2- 172.21.15.195:0/3291841286 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb054063d00 0x7fb0540661b0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fb0680017e0 tx=0x7fb068009040 comp rx=0 tx=0).stop 2022-06-27T12:10:31.139 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.117+0000 7fb07919f700 1 -- 172.21.15.195:0/3291841286 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb0740c34a0 msgr2=0x7fb074155850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:31.139 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.117+0000 7fb07919f700 1 --2- 172.21.15.195:0/3291841286 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb0740c34a0 0x7fb074155850 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7fb064009fb0 tx=0x7fb064017000 comp rx=0 tx=0).stop 2022-06-27T12:10:31.141 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.117+0000 7fb07919f700 1 -- 172.21.15.195:0/3291841286 shutdown_connections 2022-06-27T12:10:31.141 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.117+0000 7fb07919f700 1 --2- 172.21.15.195:0/3291841286 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb054063d00 0x7fb0540661b0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:31.141 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.117+0000 7fb07919f700 1 --2- 172.21.15.195:0/3291841286 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb074150ff0 0x7fb074151700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:31.142 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.117+0000 7fb07919f700 1 --2- 172.21.15.195:0/3291841286 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb07406e850 0x7fb07414d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:31.142 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.117+0000 7fb07919f700 1 --2- 172.21.15.195:0/3291841286 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb0740c34a0 0x7fb074155850 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:31.143 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.117+0000 7fb07919f700 1 -- 172.21.15.195:0/3291841286 >> 172.21.15.195:0/3291841286 conn(0x7fb0740c6350 msgr2=0x7fb0740c2d20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:10:31.143 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.117+0000 7fb07919f700 1 -- 172.21.15.195:0/3291841286 shutdown_connections 2022-06-27T12:10:31.143 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.117+0000 7fb07919f700 1 -- 172.21.15.195:0/3291841286 wait complete. 2022-06-27T12:10:31.144 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 85 2022-06-27T12:10:31.145 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2022-06-27T12:10:31.145 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2022-06-27T12:10:31.146 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:10:31.314 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.274+0000 7fcbf7b19700 1 Processor -- start 2022-06-27T12:10:31.315 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.274+0000 7fcbf7b19700 1 -- start start 2022-06-27T12:10:31.315 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.275+0000 7fcbf7b19700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcbf0156100 0x7fcbf00688a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:31.316 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.275+0000 7fcbf7b19700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fcbf01564d0 0x7fcbf0156950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:31.317 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.275+0000 7fcbf7b19700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbf00c4b10 con 0x7fcbf006a440 2022-06-27T12:10:31.327 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.275+0000 7fcbf7b19700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbf00c3dc0 con 0x7fcbf01564d0 2022-06-27T12:10:31.327 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.275+0000 7fcbf7b19700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf0068f20 con 0x7fcbf0156100 2022-06-27T12:10:31.328 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.275+0000 7fcbf50b4700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fcbf006a440 0x7fcbf00c4fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:56162/0 (socket says 172.21.15.195:56162) 2022-06-27T12:10:31.328 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.275+0000 7fcbf50b4700 1 -- 172.21.15.195:0/2892048486 learned_addr learned my addr 172.21.15.195:0/2892048486 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:10:31.329 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.275+0000 7fcbf60b6700 1 --2- 172.21.15.195:0/2892048486 >> v2:172.21.15.202:3300/0 conn(0x7fcbf01564d0 0x7fcbf0156950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:31.329 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.275+0000 7fcbf48b3700 1 -- 172.21.15.195:0/2892048486 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3890089156 0 0) 0x7fcbf00c4b10 con 0x7fcbf006a440 2022-06-27T12:10:31.331 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.275+0000 7fcbf48b3700 1 -- 172.21.15.195:0/2892048486 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbd8005640 con 0x7fcbf006a440 2022-06-27T12:10:31.331 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.275+0000 7fcbf58b5700 1 --2- 172.21.15.195:0/2892048486 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcbf0156100 0x7fcbf00688a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:31.332 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.275+0000 7fcbf48b3700 1 -- 172.21.15.195:0/2892048486 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 885844262 0 0) 0x7fcbd8005640 con 0x7fcbf006a440 2022-06-27T12:10:31.332 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.275+0000 7fcbf48b3700 1 -- 172.21.15.195:0/2892048486 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcbf0156100 msgr2=0x7fcbf00688a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:10:31.332 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.275+0000 7fcbf48b3700 1 --2- 172.21.15.195:0/2892048486 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcbf0156100 0x7fcbf00688a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:31.333 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.275+0000 7fcbf48b3700 1 -- 172.21.15.195:0/2892048486 >> v2:172.21.15.202:3300/0 conn(0x7fcbf01564d0 msgr2=0x7fcbf0156950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:10:31.333 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.275+0000 7fcbf48b3700 1 --2- 172.21.15.195:0/2892048486 >> v2:172.21.15.202:3300/0 conn(0x7fcbf01564d0 0x7fcbf0156950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:31.334 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.275+0000 7fcbf48b3700 1 -- 172.21.15.195:0/2892048486 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbf00c56b0 con 0x7fcbf006a440 2022-06-27T12:10:31.334 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.275+0000 7fcbf48b3700 1 -- 172.21.15.195:0/2892048486 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7fcbe0001470 con 0x7fcbf006a440 2022-06-27T12:10:31.335 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.276+0000 7fcbf48b3700 1 -- 172.21.15.195:0/2892048486 >> v1:172.21.15.195:6789/0 conn(0x7fcbf006a440 legacy=0x7fcbf00c4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:31.335 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.276+0000 7fcbf48b3700 1 --2- 172.21.15.195:0/2892048486 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcbd8005b30 0x7fcbd8005f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:31.335 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.276+0000 7fcbf48b3700 1 --2- 172.21.15.195:0/2892048486 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcbf0156100 0x7fcbd80068e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:31.336 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.276+0000 7fcbf48b3700 1 --2- 172.21.15.195:0/2892048486 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcbd8006e20 0x7fcbd8007290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:31.336 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.276+0000 7fcbf48b3700 1 -- 172.21.15.195:0/2892048486 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf00c3dc0 con 0x7fcbd8005b30 2022-06-27T12:10:31.337 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.276+0000 7fcbf48b3700 1 -- 172.21.15.195:0/2892048486 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf0068f20 con 0x7fcbd8006e20 2022-06-27T12:10:31.338 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.276+0000 7fcbf48b3700 1 -- 172.21.15.195:0/2892048486 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd800db30 con 0x7fcbf0156100 2022-06-27T12:10:31.338 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.276+0000 7fcbf58b5700 1 --2- 172.21.15.195:0/2892048486 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcbd8005b30 0x7fcbd8005f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:31.339 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.276+0000 7fcbf50b4700 1 --2- 172.21.15.195:0/2892048486 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcbf0156100 0x7fcbd80068e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:31.339 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.276+0000 7fcbf60b6700 1 --2- 172.21.15.195:0/2892048486 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcbd8006e20 0x7fcbd8007290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:31.340 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.276+0000 7fcbf58b5700 1 -- 172.21.15.195:0/2892048486 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcbf0156100 msgr2=0x7fcbd80068e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:10:31.340 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.276+0000 7fcbf58b5700 1 --2- 172.21.15.195:0/2892048486 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcbf0156100 0x7fcbd80068e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:31.340 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.276+0000 7fcbf58b5700 1 -- 172.21.15.195:0/2892048486 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcbd8006e20 msgr2=0x7fcbd8007290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:10:31.341 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.276+0000 7fcbf58b5700 1 --2- 172.21.15.195:0/2892048486 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcbd8006e20 0x7fcbd8007290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:31.341 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.276+0000 7fcbf58b5700 1 -- 172.21.15.195:0/2892048486 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbd8005640 con 0x7fcbd8005b30 2022-06-27T12:10:31.342 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.276+0000 7fcbf58b5700 1 --2- 172.21.15.195:0/2892048486 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcbd8005b30 0x7fcbd8005f40 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fcbec009ae0 tx=0x7fcbec006cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d6fdd3a008b6315 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:31.342 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.276+0000 7fcbf48b3700 1 -- 172.21.15.195:0/2892048486 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbec009520 con 0x7fcbd8005b30 2022-06-27T12:10:31.343 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.276+0000 7fcbf48b3700 1 -- 172.21.15.195:0/2892048486 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbec0058c0 con 0x7fcbd8005b30 2022-06-27T12:10:31.344 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.276+0000 7fcbf48b3700 1 -- 172.21.15.195:0/2892048486 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbec005c60 con 0x7fcbd8005b30 2022-06-27T12:10:31.344 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.277+0000 7fcbf7b19700 1 -- 172.21.15.195:0/2892048486 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcbd8005b30 msgr2=0x7fcbd8005f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:31.345 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.277+0000 7fcbf7b19700 1 --2- 172.21.15.195:0/2892048486 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcbd8005b30 0x7fcbd8005f40 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fcbec009ae0 tx=0x7fcbec006cb0 comp rx=0 tx=0).stop 2022-06-27T12:10:31.345 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.277+0000 7fcbf60b6700 1 -- 172.21.15.195:0/2892048486 reap_dead start 2022-06-27T12:10:31.345 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.277+0000 7fcbf7b19700 1 -- 172.21.15.195:0/2892048486 shutdown_connections 2022-06-27T12:10:31.347 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.277+0000 7fcbf7b19700 1 -- 172.21.15.195:0/2892048486 >> 172.21.15.195:0/2892048486 conn(0x7fcbf00c9b00 msgr2=0x7fcbf00c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:10:31.347 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.277+0000 7fcbf7b19700 1 -- 172.21.15.195:0/2892048486 shutdown_connections 2022-06-27T12:10:31.347 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.277+0000 7fcbf7b19700 1 -- 172.21.15.195:0/2892048486 wait complete. 2022-06-27T12:10:31.359 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.278+0000 7fcbf7b19700 1 Processor -- start 2022-06-27T12:10:31.359 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.278+0000 7fcbf7b19700 1 -- start start 2022-06-27T12:10:31.359 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.279+0000 7fcbf7b19700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcbd8005b30 0x7fcbf00c19f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:31.360 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.279+0000 7fcbf7b19700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcbf0149100 0x7fcbf0149930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:31.360 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.279+0000 7fcbf7b19700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcbf0149e70 0x7fcbf0145840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:31.361 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.279+0000 7fcbf7b19700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf0148db0 con 0x7fcbd8005b30 2022-06-27T12:10:31.361 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.279+0000 7fcbf7b19700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf0158240 con 0x7fcbf0149100 2022-06-27T12:10:31.365 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.279+0000 7fcbf7b19700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf01586d0 con 0x7fcbf0149e70 2022-06-27T12:10:31.367 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.279+0000 7fcbf58b5700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcbd8005b30 0x7fcbf00c19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:31.367 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.279+0000 7fcbf58b5700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcbd8005b30 0x7fcbf00c19f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:49984/0 (socket says 172.21.15.195:49984) 2022-06-27T12:10:31.368 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.279+0000 7fcbf58b5700 1 -- 172.21.15.195:0/184959880 learned_addr learned my addr 172.21.15.195:0/184959880 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:10:31.368 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.279+0000 7fcbf50b4700 1 --2- 172.21.15.195:0/184959880 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcbf0149100 0x7fcbf0149930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:31.368 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.279+0000 7fcbf60b6700 1 --2- 172.21.15.195:0/184959880 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcbf0149e70 0x7fcbf0145840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:31.369 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.279+0000 7fcbf58b5700 1 -- 172.21.15.195:0/184959880 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcbf0149e70 msgr2=0x7fcbf0145840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:31.369 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.279+0000 7fcbf58b5700 1 --2- 172.21.15.195:0/184959880 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcbf0149e70 0x7fcbf0145840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:31.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.279+0000 7fcbf58b5700 1 -- 172.21.15.195:0/184959880 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcbf0149100 msgr2=0x7fcbf0149930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:31.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.279+0000 7fcbf58b5700 1 --2- 172.21.15.195:0/184959880 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcbf0149100 0x7fcbf0149930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:31.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.279+0000 7fcbf58b5700 1 -- 172.21.15.195:0/184959880 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbf0155bc0 con 0x7fcbd8005b30 2022-06-27T12:10:31.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.279+0000 7fcbf50b4700 1 --2- 172.21.15.195:0/184959880 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcbf0149100 0x7fcbf0149930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:10:31.372 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.279+0000 7fcbf60b6700 1 --2- 172.21.15.195:0/184959880 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcbf0149e70 0x7fcbf0145840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:10:31.372 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.279+0000 7fcbf58b5700 1 --2- 172.21.15.195:0/184959880 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcbd8005b30 0x7fcbf00c19f0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fcbec006d60 tx=0x7fcbec019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:31.372 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.279+0000 7fcbddffb700 1 -- 172.21.15.195:0/184959880 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbec00e6f0 con 0x7fcbd8005b30 2022-06-27T12:10:31.373 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.279+0000 7fcbddffb700 1 -- 172.21.15.195:0/184959880 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbec00e870 con 0x7fcbd8005b30 2022-06-27T12:10:31.373 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.279+0000 7fcbf7b19700 1 -- 172.21.15.195:0/184959880 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbf0148550 con 0x7fcbd8005b30 2022-06-27T12:10:31.374 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.279+0000 7fcbddffb700 1 -- 172.21.15.195:0/184959880 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbec00ec30 con 0x7fcbd8005b30 2022-06-27T12:10:31.375 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.280+0000 7fcbf7b19700 1 -- 172.21.15.195:0/184959880 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbf0148ad0 con 0x7fcbd8005b30 2022-06-27T12:10:31.375 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.281+0000 7fcbf7b19700 1 -- 172.21.15.195:0/184959880 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbf00722c0 con 0x7fcbd8005b30 2022-06-27T12:10:31.375 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.281+0000 7fcbddffb700 1 -- 172.21.15.195:0/184959880 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fcbec024070 con 0x7fcbd8005b30 2022-06-27T12:10:31.376 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.281+0000 7fcbddffb700 1 --2- 172.21.15.195:0/184959880 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcbd0063d00 0x7fcbd00661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:31.376 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.281+0000 7fcbddffb700 1 -- 172.21.15.195:0/184959880 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7320+0+0 (secure 0 0 0) 0x7fcbec082370 con 0x7fcbd8005b30 2022-06-27T12:10:31.377 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.281+0000 7fcbf50b4700 1 --2- 172.21.15.195:0/184959880 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcbd0063d00 0x7fcbd00661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:31.377 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.282+0000 7fcbf50b4700 1 --2- 172.21.15.195:0/184959880 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcbd0063d00 0x7fcbd00661b0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fcbe0000e10 tx=0x7fcbe0009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:31.378 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.286+0000 7fcbddffb700 1 -- 172.21.15.195:0/184959880 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fcbec050a70 con 0x7fcbd8005b30 2022-06-27T12:10:31.513 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.512+0000 7fcbf7b19700 1 -- 172.21.15.195:0/184959880 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcbf00c6360 con 0x7fcbd8005b30 2022-06-27T12:10:31.514 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.512+0000 7fcbddffb700 1 -- 172.21.15.195:0/184959880 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+5367 (secure 0 0 0) 0x7fcbec054090 con 0x7fcbd8005b30 2022-06-27T12:10:31.515 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:10:31.515 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:10:30.134487+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5933},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5933":{"gid":5933,"name":"d","rank":0,"incarnation":85,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6835/1342896965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":1342896965},{"type":"v1","addr":"172.21.15.202:6835","nonce":1342896965}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:10:31.518 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.514+0000 7fcbf7b19700 1 -- 172.21.15.195:0/184959880 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcbd0063d00 msgr2=0x7fcbd00661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:31.518 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.514+0000 7fcbf7b19700 1 --2- 172.21.15.195:0/184959880 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcbd0063d00 0x7fcbd00661b0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fcbe0000e10 tx=0x7fcbe0009040 comp rx=0 tx=0).stop 2022-06-27T12:10:31.519 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.514+0000 7fcbf7b19700 1 -- 172.21.15.195:0/184959880 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcbd8005b30 msgr2=0x7fcbf00c19f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:31.519 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.514+0000 7fcbf7b19700 1 --2- 172.21.15.195:0/184959880 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcbd8005b30 0x7fcbf00c19f0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fcbec006d60 tx=0x7fcbec019000 comp rx=0 tx=0).stop 2022-06-27T12:10:31.520 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.514+0000 7fcbf7b19700 1 -- 172.21.15.195:0/184959880 shutdown_connections 2022-06-27T12:10:31.521 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.514+0000 7fcbf7b19700 1 --2- 172.21.15.195:0/184959880 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcbd0063d00 0x7fcbd00661b0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:31.521 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.514+0000 7fcbf7b19700 1 --2- 172.21.15.195:0/184959880 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcbd8005b30 0x7fcbf00c19f0 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:31.522 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.514+0000 7fcbf7b19700 1 --2- 172.21.15.195:0/184959880 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcbf0149e70 0x7fcbf0145840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:31.522 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.514+0000 7fcbf7b19700 1 --2- 172.21.15.195:0/184959880 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcbf0149100 0x7fcbf0149930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:31.522 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.514+0000 7fcbf7b19700 1 -- 172.21.15.195:0/184959880 >> 172.21.15.195:0/184959880 conn(0x7fcbf00c9b00 msgr2=0x7fcbf006f050 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:10:31.523 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.514+0000 7fcbf7b19700 1 -- 172.21.15.195:0/184959880 shutdown_connections 2022-06-27T12:10:31.523 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:31.515+0000 7fcbf7b19700 1 -- 172.21.15.195:0/184959880 wait complete. 2022-06-27T12:10:31.524 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 85 2022-06-27T12:10:31.529 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 85, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-06-27T11:47:52.358095+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 85, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5933": { "addr": "172.21.15.202:6835/1342896965", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6834", "nonce": 1342896965, "type": "v2" }, { "addr": "172.21.15.202:6835", "nonce": 1342896965, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5933, "incarnation": 85, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 61, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-06-27T12:10:30.134487+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5933 } } } ], "standbys": [ { "addr": "172.21.15.202:6837/3305006841", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6836", "nonce": 3305006841, "type": "v2" }, { "addr": "172.21.15.202:6837", "nonce": 3305006841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6835/2060709822", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6834", "nonce": 2060709822, "type": "v2" }, { "addr": "172.21.15.195:6835", "nonce": 2060709822, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6838/1598751652", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6836", "nonce": 1598751652, "type": "v2" }, { "addr": "172.21.15.195:6838", "nonce": 1598751652, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4275, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6839/1313800661", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6837", "nonce": 1313800661, "type": "v2" }, { "addr": "172.21.15.195:6839", "nonce": 1313800661, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4278, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-06-27T12:10:32.700 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:10:32.704 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:10:33.531 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:10:33.668 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.666+0000 7f8aa2f30700 1 Processor -- start 2022-06-27T12:10:33.668 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.667+0000 7f8aa2f30700 1 -- start start 2022-06-27T12:10:33.669 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.667+0000 7f8aa2f30700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8a9c06f640 0x7f8a9c156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:33.669 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.667+0000 7f8aa2f30700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f8a9c0c39e0 0x7f8a9c1535a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:33.672 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.667+0000 7f8aa2f30700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a9c0c1a20 con 0x7f8a9c0c4160 2022-06-27T12:10:33.672 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.667+0000 7f8aa2f30700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a9c0c1130 con 0x7f8a9c0c39e0 2022-06-27T12:10:33.672 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.667+0000 7f8aa2f30700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a9c06ea90 con 0x7f8a9c06f640 2022-06-27T12:10:33.682 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.668+0000 7f8aa14cd700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f8a9c0c39e0 0x7f8a9c1535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:33.682 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.668+0000 7f8aa0ccc700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8a9c06f640 0x7f8a9c156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:33.683 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.668+0000 7f8aa0ccc700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8a9c06f640 0x7f8a9c156ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3301/0 says I am v2:172.21.15.195:42748/0 (socket says 172.21.15.195:42748) 2022-06-27T12:10:33.683 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.668+0000 7f8a9bfff700 1 --1- >> v1:172.21.15.202:6789/0 conn(0x7f8a9c0c4160 0x7f8a9c0c1eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.202:6789/0 says I am v1:172.21.15.195:58044/0 (socket says 172.21.15.195:58044) 2022-06-27T12:10:33.683 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.668+0000 7f8aa14cd700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f8a9c0c39e0 0x7f8a9c1535a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3300/0 says I am v2:172.21.15.195:48120/0 (socket says 172.21.15.195:48120) 2022-06-27T12:10:33.683 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.668+0000 7f8aa0ccc700 1 -- 172.21.15.195:0/1703734600 learned_addr learned my addr 172.21.15.195:0/1703734600 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:10:33.684 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.668+0000 7f8aa14cd700 1 -- 172.21.15.195:0/1703734600 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8a9c06f640 msgr2=0x7f8a9c156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:10:33.684 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.668+0000 7f8aa14cd700 1 --2- 172.21.15.195:0/1703734600 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8a9c06f640 0x7f8a9c156ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:33.684 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.668+0000 7f8aa14cd700 1 -- 172.21.15.195:0/1703734600 >> v1:172.21.15.202:6789/0 conn(0x7f8a9c0c4160 legacy=0x7f8a9c0c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:33.684 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.668+0000 7f8a9b7fe700 1 -- 172.21.15.195:0/1703734600 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1563050919 0 0) 0x7f8a9c0c1a20 con 0x7f8a9c0c4160 2022-06-27T12:10:33.685 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.668+0000 7f8aa14cd700 1 -- 172.21.15.195:0/1703734600 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a9c06ed70 con 0x7f8a9c0c39e0 2022-06-27T12:10:33.685 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.668+0000 7f8aa14cd700 1 --2- 172.21.15.195:0/1703734600 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8a9c0c39e0 0x7f8a9c1535a0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f8a94009e60 tx=0x7f8a94009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce8da94b3297b50a server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:33.685 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.669+0000 7f8a9b7fe700 1 -- 172.21.15.195:0/1703734600 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a9400e070 con 0x7f8a9c0c39e0 2022-06-27T12:10:33.685 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.669+0000 7f8a9b7fe700 1 -- 172.21.15.195:0/1703734600 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a9400bbf0 con 0x7f8a9c0c39e0 2022-06-27T12:10:33.686 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.669+0000 7f8a9b7fe700 1 -- 172.21.15.195:0/1703734600 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a94012430 con 0x7f8a9c0c39e0 2022-06-27T12:10:33.686 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.669+0000 7f8aa2f30700 1 -- 172.21.15.195:0/1703734600 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8a9c0c39e0 msgr2=0x7f8a9c1535a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:33.686 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.669+0000 7f8aa2f30700 1 --2- 172.21.15.195:0/1703734600 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8a9c0c39e0 0x7f8a9c1535a0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f8a94009e60 tx=0x7f8a94009780 comp rx=0 tx=0).stop 2022-06-27T12:10:33.686 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.669+0000 7f8aa2f30700 1 -- 172.21.15.195:0/1703734600 shutdown_connections 2022-06-27T12:10:33.687 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.669+0000 7f8aa2f30700 1 --2- 172.21.15.195:0/1703734600 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8a9c0c39e0 0x7f8a9c1535a0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:33.687 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.669+0000 7f8aa2f30700 1 --2- 172.21.15.195:0/1703734600 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8a9c06f640 0x7f8a9c156ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:33.688 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.669+0000 7f8aa2f30700 1 -- 172.21.15.195:0/1703734600 >> 172.21.15.195:0/1703734600 conn(0x7f8a9c0c6350 msgr2=0x7f8a9c0c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:10:33.688 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.669+0000 7f8aa2f30700 1 -- 172.21.15.195:0/1703734600 shutdown_connections 2022-06-27T12:10:33.689 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.669+0000 7f8aa2f30700 1 -- 172.21.15.195:0/1703734600 wait complete. 2022-06-27T12:10:33.689 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.670+0000 7f8aa2f30700 1 Processor -- start 2022-06-27T12:10:33.689 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.671+0000 7f8aa2f30700 1 -- start start 2022-06-27T12:10:33.689 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.671+0000 7f8aa2f30700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8a9c06f640 0x7f8a9c151d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:33.691 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.671+0000 7f8aa2f30700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8a9c0c39e0 0x7f8a9c067ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:33.692 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.671+0000 7f8aa2f30700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8a9c0c4160 0x7f8a9c165ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:33.693 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.671+0000 7f8aa2f30700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a9c0beb30 con 0x7f8a9c0c39e0 2022-06-27T12:10:33.693 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.671+0000 7f8aa2f30700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a9c06a300 con 0x7f8a9c06f640 2022-06-27T12:10:33.694 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.671+0000 7f8aa2f30700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a9c0bcb30 con 0x7f8a9c0c4160 2022-06-27T12:10:33.694 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.671+0000 7f8a9bfff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8a9c0c39e0 0x7f8a9c067ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:33.694 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.671+0000 7f8a9bfff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8a9c0c39e0 0x7f8a9c067ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50000/0 (socket says 172.21.15.195:50000) 2022-06-27T12:10:33.695 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.671+0000 7f8a9bfff700 1 -- 172.21.15.195:0/72469475 learned_addr learned my addr 172.21.15.195:0/72469475 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:10:33.695 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.672+0000 7f8aa14cd700 1 --2- 172.21.15.195:0/72469475 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8a9c0c4160 0x7f8a9c165ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:33.696 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.672+0000 7f8aa0ccc700 1 --2- 172.21.15.195:0/72469475 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8a9c06f640 0x7f8a9c151d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:33.696 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.672+0000 7f8a9bfff700 1 -- 172.21.15.195:0/72469475 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8a9c0c4160 msgr2=0x7f8a9c165ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:33.696 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.672+0000 7f8a9bfff700 1 --2- 172.21.15.195:0/72469475 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8a9c0c4160 0x7f8a9c165ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:33.696 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.672+0000 7f8a9bfff700 1 -- 172.21.15.195:0/72469475 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8a9c06f640 msgr2=0x7f8a9c151d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:33.697 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.672+0000 7f8a9bfff700 1 --2- 172.21.15.195:0/72469475 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8a9c06f640 0x7f8a9c151d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:33.697 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.672+0000 7f8a9bfff700 1 -- 172.21.15.195:0/72469475 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a94009a20 con 0x7f8a9c0c39e0 2022-06-27T12:10:33.697 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.672+0000 7f8aa14cd700 1 --2- 172.21.15.195:0/72469475 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8a9c0c4160 0x7f8a9c165ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:10:33.698 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.672+0000 7f8a9bfff700 1 --2- 172.21.15.195:0/72469475 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8a9c0c39e0 0x7f8a9c067ed0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f8a8800c4c0 tx=0x7f8a88007c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:33.699 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.672+0000 7f8a98ff9700 1 -- 172.21.15.195:0/72469475 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a88007360 con 0x7f8a9c0c39e0 2022-06-27T12:10:33.699 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.672+0000 7f8a98ff9700 1 -- 172.21.15.195:0/72469475 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a880074c0 con 0x7f8a9c0c39e0 2022-06-27T12:10:33.700 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.672+0000 7f8aa2f30700 1 -- 172.21.15.195:0/72469475 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a9c0bcdd0 con 0x7f8a9c0c39e0 2022-06-27T12:10:33.700 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.672+0000 7f8a98ff9700 1 -- 172.21.15.195:0/72469475 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a88004900 con 0x7f8a9c0c39e0 2022-06-27T12:10:33.701 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.672+0000 7f8aa2f30700 1 -- 172.21.15.195:0/72469475 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a9c069f90 con 0x7f8a9c0c39e0 2022-06-27T12:10:33.701 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.673+0000 7f8aa2f30700 1 -- 172.21.15.195:0/72469475 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a9c0722c0 con 0x7f8a9c0c39e0 2022-06-27T12:10:33.702 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.673+0000 7f8a98ff9700 1 -- 172.21.15.195:0/72469475 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f8a88010030 con 0x7f8a9c0c39e0 2022-06-27T12:10:33.710 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.674+0000 7f8a98ff9700 1 --2- 172.21.15.195:0/72469475 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8a84063d00 0x7f8a840661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:33.719 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.674+0000 7f8a98ff9700 1 -- 172.21.15.195:0/72469475 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7320+0+0 (secure 0 0 0) 0x7f8a88080870 con 0x7f8a9c0c39e0 2022-06-27T12:10:33.719 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.674+0000 7f8aa0ccc700 1 --2- 172.21.15.195:0/72469475 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8a84063d00 0x7f8a840661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:33.719 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.674+0000 7f8aa0ccc700 1 --2- 172.21.15.195:0/72469475 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8a84063d00 0x7f8a840661b0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f8a90009bd0 tx=0x7f8a90009120 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:33.720 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.678+0000 7f8a98ff9700 1 -- 172.21.15.195:0/72469475 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f8a8804ef70 con 0x7f8a9c0c39e0 2022-06-27T12:10:33.909 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.906+0000 7f8aa2f30700 1 -- 172.21.15.195:0/72469475 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8a9c06f240 con 0x7f8a9c0c39e0 2022-06-27T12:10:33.910 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.907+0000 7f8a98ff9700 1 -- 172.21.15.195:0/72469475 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+5367 (secure 0 0 0) 0x7f8a88007d40 con 0x7f8a9c0c39e0 2022-06-27T12:10:33.911 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:10:33.911 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:10:30.134487+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5933},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5933":{"gid":5933,"name":"d","rank":0,"incarnation":85,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6835/1342896965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":1342896965},{"type":"v1","addr":"172.21.15.202:6835","nonce":1342896965}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:10:33.912 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.909+0000 7f8aa2f30700 1 -- 172.21.15.195:0/72469475 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8a84063d00 msgr2=0x7f8a840661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:33.913 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.909+0000 7f8aa2f30700 1 --2- 172.21.15.195:0/72469475 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8a84063d00 0x7f8a840661b0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f8a90009bd0 tx=0x7f8a90009120 comp rx=0 tx=0).stop 2022-06-27T12:10:33.913 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.909+0000 7f8aa2f30700 1 -- 172.21.15.195:0/72469475 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8a9c0c39e0 msgr2=0x7f8a9c067ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:33.913 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.909+0000 7f8aa2f30700 1 --2- 172.21.15.195:0/72469475 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8a9c0c39e0 0x7f8a9c067ed0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f8a8800c4c0 tx=0x7f8a88007c00 comp rx=0 tx=0).stop 2022-06-27T12:10:33.914 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.909+0000 7f8aa2f30700 1 -- 172.21.15.195:0/72469475 shutdown_connections 2022-06-27T12:10:33.914 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.909+0000 7f8aa2f30700 1 --2- 172.21.15.195:0/72469475 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8a84063d00 0x7f8a840661b0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:33.914 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.909+0000 7f8aa2f30700 1 --2- 172.21.15.195:0/72469475 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8a9c0c4160 0x7f8a9c165ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:33.914 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.909+0000 7f8aa2f30700 1 --2- 172.21.15.195:0/72469475 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8a9c06f640 0x7f8a9c151d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:33.915 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.909+0000 7f8aa2f30700 1 --2- 172.21.15.195:0/72469475 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8a9c0c39e0 0x7f8a9c067ed0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:33.915 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.909+0000 7f8aa2f30700 1 -- 172.21.15.195:0/72469475 >> 172.21.15.195:0/72469475 conn(0x7f8a9c0c6350 msgr2=0x7f8a9c0c7930 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:10:33.915 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.909+0000 7f8aa2f30700 1 -- 172.21.15.195:0/72469475 shutdown_connections 2022-06-27T12:10:33.915 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:33.909+0000 7f8aa2f30700 1 -- 172.21.15.195:0/72469475 wait complete. 2022-06-27T12:10:33.916 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 85 2022-06-27T12:10:33.921 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:10:35.942 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:10:36.085 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.084+0000 7f269bd66700 1 Processor -- start 2022-06-27T12:10:36.085 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.085+0000 7f269bd66700 1 -- start start 2022-06-27T12:10:36.087 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.085+0000 7f269bd66700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f26940c4160 0x7f2694156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:36.087 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.085+0000 7f269bd66700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f26940c39e0 0x7f26941535a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:36.087 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.085+0000 7f269bd66700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26940c1a20 con 0x7f269406f640 2022-06-27T12:10:36.087 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.085+0000 7f269bd66700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26940c1130 con 0x7f26940c4160 2022-06-27T12:10:36.088 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.085+0000 7f269bd66700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f269406ea90 con 0x7f26940c39e0 2022-06-27T12:10:36.088 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.086+0000 7f2699301700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f26940c4160 0x7f2694156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:36.088 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.086+0000 7f2699b02700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f269406f640 0x7f26940c1eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:56194/0 (socket says 172.21.15.195:56194) 2022-06-27T12:10:36.088 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.086+0000 7f2699b02700 1 -- 172.21.15.195:0/1373683722 learned_addr learned my addr 172.21.15.195:0/1373683722 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:10:36.089 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.086+0000 7f269a303700 1 --2- 172.21.15.195:0/1373683722 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f26940c39e0 0x7f26941535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:36.089 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.086+0000 7f2699301700 1 -- 172.21.15.195:0/1373683722 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f26940c39e0 msgr2=0x7f26941535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:10:36.089 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.086+0000 7f2699301700 1 --2- 172.21.15.195:0/1373683722 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f26940c39e0 0x7f26941535a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:36.089 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.086+0000 7f2699301700 1 -- 172.21.15.195:0/1373683722 >> v1:172.21.15.195:6789/0 conn(0x7f269406f640 legacy=0x7f26940c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:36.090 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.086+0000 7f2699301700 1 -- 172.21.15.195:0/1373683722 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f269406ed70 con 0x7f26940c4160 2022-06-27T12:10:36.092 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.086+0000 7f2698b00700 1 -- 172.21.15.195:0/1373683722 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1773554438 0 0) 0x7f26940c1a20 con 0x7f269406f640 2022-06-27T12:10:36.092 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.086+0000 7f2699301700 1 --2- 172.21.15.195:0/1373683722 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f26940c4160 0x7f2694156ed0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f26900097a0 tx=0x7f2690009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26a5188e274a6dfc server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:36.092 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.086+0000 7f2698b00700 1 -- 172.21.15.195:0/1373683722 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2690011630 con 0x7f26940c4160 2022-06-27T12:10:36.092 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.086+0000 7f2698b00700 1 -- 172.21.15.195:0/1373683722 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2690011790 con 0x7f26940c4160 2022-06-27T12:10:36.162 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.086+0000 7f2698b00700 1 -- 172.21.15.195:0/1373683722 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2690011b30 con 0x7f26940c4160 2022-06-27T12:10:36.162 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.087+0000 7f269bd66700 1 -- 172.21.15.195:0/1373683722 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f26940c4160 msgr2=0x7f2694156ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:36.163 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.087+0000 7f269bd66700 1 --2- 172.21.15.195:0/1373683722 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f26940c4160 0x7f2694156ed0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f26900097a0 tx=0x7f2690009060 comp rx=0 tx=0).stop 2022-06-27T12:10:36.163 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.087+0000 7f269bd66700 1 -- 172.21.15.195:0/1373683722 shutdown_connections 2022-06-27T12:10:36.164 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.087+0000 7f269bd66700 1 --2- 172.21.15.195:0/1373683722 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f26940c39e0 0x7f26941535a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:36.164 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.087+0000 7f269bd66700 1 --2- 172.21.15.195:0/1373683722 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f26940c4160 0x7f2694156ed0 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:36.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.087+0000 7f269bd66700 1 -- 172.21.15.195:0/1373683722 >> 172.21.15.195:0/1373683722 conn(0x7f26940c6350 msgr2=0x7f26940c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:10:36.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.087+0000 7f269bd66700 1 -- 172.21.15.195:0/1373683722 shutdown_connections 2022-06-27T12:10:36.171 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.087+0000 7f269bd66700 1 -- 172.21.15.195:0/1373683722 wait complete. 2022-06-27T12:10:36.171 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.088+0000 7f269bd66700 1 Processor -- start 2022-06-27T12:10:36.172 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.088+0000 7f269bd66700 1 -- start start 2022-06-27T12:10:36.185 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.089+0000 7f269bd66700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f269406f640 0x7f269414c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:36.186 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.089+0000 7f269bd66700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f26940c39e0 0x7f26940c78b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:36.186 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.089+0000 7f269bd66700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f26940c4160 0x7f26940c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:36.186 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.089+0000 7f269bd66700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f269414f3a0 con 0x7f26940c4160 2022-06-27T12:10:36.187 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.089+0000 7f269bd66700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26941592f0 con 0x7f26940c39e0 2022-06-27T12:10:36.187 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.089+0000 7f269bd66700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26940c0ac0 con 0x7f269406f640 2022-06-27T12:10:36.188 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.089+0000 7f269a303700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f26940c4160 0x7f26940c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:36.188 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.089+0000 7f269a303700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f26940c4160 0x7f26940c7df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50016/0 (socket says 172.21.15.195:50016) 2022-06-27T12:10:36.189 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.089+0000 7f269a303700 1 -- 172.21.15.195:0/875073009 learned_addr learned my addr 172.21.15.195:0/875073009 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:10:36.206 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.089+0000 7f2699b02700 1 --2- 172.21.15.195:0/875073009 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f269406f640 0x7f269414c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:36.207 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.089+0000 7f2699301700 1 --2- 172.21.15.195:0/875073009 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f26940c39e0 0x7f26940c78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:36.207 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.089+0000 7f269a303700 1 -- 172.21.15.195:0/875073009 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f269406f640 msgr2=0x7f269414c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:36.207 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.089+0000 7f269a303700 1 --2- 172.21.15.195:0/875073009 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f269406f640 0x7f269414c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:36.208 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.089+0000 7f269a303700 1 -- 172.21.15.195:0/875073009 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f26940c39e0 msgr2=0x7f26940c78b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:36.208 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.089+0000 7f269a303700 1 --2- 172.21.15.195:0/875073009 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f26940c39e0 0x7f26940c78b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:36.209 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.089+0000 7f269a303700 1 -- 172.21.15.195:0/875073009 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26900092e0 con 0x7f26940c4160 2022-06-27T12:10:36.210 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.089+0000 7f2699b02700 1 --2- 172.21.15.195:0/875073009 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f269406f640 0x7f269414c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:10:36.211 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.089+0000 7f269a303700 1 --2- 172.21.15.195:0/875073009 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f26940c4160 0x7f26940c7df0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f268c009ec0 tx=0x7f268c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:36.211 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.089+0000 7f2681ffb700 1 -- 172.21.15.195:0/875073009 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f268c00cc00 con 0x7f26940c4160 2022-06-27T12:10:36.211 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.090+0000 7f2681ffb700 1 -- 172.21.15.195:0/875073009 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f268c00f040 con 0x7f26940c4160 2022-06-27T12:10:36.212 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.090+0000 7f269bd66700 1 -- 172.21.15.195:0/875073009 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f269414ee20 con 0x7f26940c4160 2022-06-27T12:10:36.212 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.090+0000 7f2681ffb700 1 -- 172.21.15.195:0/875073009 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f268c014480 con 0x7f26940c4160 2022-06-27T12:10:36.212 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.090+0000 7f269bd66700 1 -- 172.21.15.195:0/875073009 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2694158f80 con 0x7f26940c4160 2022-06-27T12:10:36.213 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.091+0000 7f269bd66700 1 -- 172.21.15.195:0/875073009 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2694159480 con 0x7f26940c4160 2022-06-27T12:10:36.213 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.091+0000 7f2681ffb700 1 -- 172.21.15.195:0/875073009 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f268c014830 con 0x7f26940c4160 2022-06-27T12:10:36.214 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.091+0000 7f2681ffb700 1 --2- 172.21.15.195:0/875073009 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2674063d30 0x7f26740661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:36.214 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.091+0000 7f2681ffb700 1 -- 172.21.15.195:0/875073009 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7320+0+0 (secure 0 0 0) 0x7f268c0818e0 con 0x7f26940c4160 2022-06-27T12:10:36.215 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.092+0000 7f2699b02700 1 --2- 172.21.15.195:0/875073009 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2674063d30 0x7f26740661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:36.215 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.092+0000 7f2699b02700 1 --2- 172.21.15.195:0/875073009 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2674063d30 0x7f26740661e0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f2684001670 tx=0x7f2684007bf0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:36.216 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.096+0000 7f2681ffb700 1 -- 172.21.15.195:0/875073009 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f268c04ffe0 con 0x7f26940c4160 2022-06-27T12:10:36.335 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.327+0000 7f269bd66700 1 -- 172.21.15.195:0/875073009 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f26940722c0 con 0x7f26940c4160 2022-06-27T12:10:36.336 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.328+0000 7f2681ffb700 1 -- 172.21.15.195:0/875073009 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+5367 (secure 0 0 0) 0x7f268c053600 con 0x7f26940c4160 2022-06-27T12:10:36.336 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:10:36.337 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:10:30.134487+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5933},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5933":{"gid":5933,"name":"d","rank":0,"incarnation":85,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6835/1342896965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":1342896965},{"type":"v1","addr":"172.21.15.202:6835","nonce":1342896965}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:10:36.339 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.330+0000 7f269bd66700 1 -- 172.21.15.195:0/875073009 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2674063d30 msgr2=0x7f26740661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:36.339 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.330+0000 7f269bd66700 1 --2- 172.21.15.195:0/875073009 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2674063d30 0x7f26740661e0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f2684001670 tx=0x7f2684007bf0 comp rx=0 tx=0).stop 2022-06-27T12:10:36.339 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.330+0000 7f269bd66700 1 -- 172.21.15.195:0/875073009 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f26940c4160 msgr2=0x7f26940c7df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:36.339 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.330+0000 7f269bd66700 1 --2- 172.21.15.195:0/875073009 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f26940c4160 0x7f26940c7df0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f268c009ec0 tx=0x7f268c00b040 comp rx=0 tx=0).stop 2022-06-27T12:10:36.340 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.330+0000 7f269bd66700 1 -- 172.21.15.195:0/875073009 shutdown_connections 2022-06-27T12:10:36.340 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.330+0000 7f269bd66700 1 --2- 172.21.15.195:0/875073009 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2674063d30 0x7f26740661e0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:36.341 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.330+0000 7f269bd66700 1 --2- 172.21.15.195:0/875073009 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f26940c4160 0x7f26940c7df0 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:36.341 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.330+0000 7f269bd66700 1 --2- 172.21.15.195:0/875073009 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f269406f640 0x7f269414c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:36.341 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.330+0000 7f269bd66700 1 --2- 172.21.15.195:0/875073009 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f26940c39e0 0x7f26940c78b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:36.342 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.330+0000 7f269bd66700 1 -- 172.21.15.195:0/875073009 >> 172.21.15.195:0/875073009 conn(0x7f26940c6350 msgr2=0x7f2694152570 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:10:36.342 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.330+0000 7f269bd66700 1 -- 172.21.15.195:0/875073009 shutdown_connections 2022-06-27T12:10:36.342 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:36.330+0000 7f269bd66700 1 -- 172.21.15.195:0/875073009 wait complete. 2022-06-27T12:10:36.342 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 85 2022-06-27T12:10:36.344 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:10:38.346 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:10:38.481 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.478+0000 7f6ac1145700 1 Processor -- start 2022-06-27T12:10:38.485 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.478+0000 7f6ac1145700 1 -- start start 2022-06-27T12:10:38.485 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.479+0000 7f6ac1145700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6abc06e850 0x7f6abc1570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:38.486 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.479+0000 7f6ac1145700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f6abc06fef0 0x7f6abc069de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:38.487 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.479+0000 7f6ac1145700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6abc06fba0 con 0x7f6abc0c34a0 2022-06-27T12:10:38.487 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.479+0000 7f6ac1145700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6abc1575e0 con 0x7f6abc06fef0 2022-06-27T12:10:38.488 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.479+0000 7f6ac1145700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6abc06f5e0 con 0x7f6abc06e850 2022-06-27T12:10:38.488 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.479+0000 7f6aba59c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f6abc06fef0 0x7f6abc069de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:38.489 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.479+0000 7f6abb59e700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f6abc0c34a0 0x7f6abc0c2920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:56210/0 (socket says 172.21.15.195:56210) 2022-06-27T12:10:38.489 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.479+0000 7f6aba59c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f6abc06fef0 0x7f6abc069de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50020/0 (socket says 172.21.15.195:50020) 2022-06-27T12:10:38.490 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.479+0000 7f6abb59e700 1 -- 172.21.15.195:0/2734094650 learned_addr learned my addr 172.21.15.195:0/2734094650 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:10:38.502 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.479+0000 7f6abad9d700 1 --2- 172.21.15.195:0/2734094650 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6abc06e850 0x7f6abc1570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:38.503 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.480+0000 7f6aba59c700 1 -- 172.21.15.195:0/2734094650 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6abc06e850 msgr2=0x7f6abc1570a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:10:38.503 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.480+0000 7f6aba59c700 1 --2- 172.21.15.195:0/2734094650 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6abc06e850 0x7f6abc1570a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:38.504 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.480+0000 7f6aba59c700 1 -- 172.21.15.195:0/2734094650 >> v1:172.21.15.195:6789/0 conn(0x7f6abc0c34a0 legacy=0x7f6abc0c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:38.504 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.480+0000 7f6aba59c700 1 -- 172.21.15.195:0/2734094650 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6abc06f8c0 con 0x7f6abc06fef0 2022-06-27T12:10:38.505 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.480+0000 7f6aba59c700 1 --2- 172.21.15.195:0/2734094650 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6abc06fef0 0x7f6abc069de0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f6aac0097a0 tx=0x7f6aac009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbed63744c862b22 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:38.506 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.480+0000 7f6ab9d9b700 1 -- 172.21.15.195:0/2734094650 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aac004030 con 0x7f6abc06fef0 2022-06-27T12:10:38.506 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.480+0000 7f6ab9d9b700 1 -- 172.21.15.195:0/2734094650 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6aac004190 con 0x7f6abc06fef0 2022-06-27T12:10:38.507 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.480+0000 7f6ab9d9b700 1 -- 172.21.15.195:0/2734094650 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aac0105b0 con 0x7f6abc06fef0 2022-06-27T12:10:38.508 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.480+0000 7f6ac1145700 1 -- 172.21.15.195:0/2734094650 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6abc06fef0 msgr2=0x7f6abc069de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:38.508 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.480+0000 7f6ac1145700 1 --2- 172.21.15.195:0/2734094650 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6abc06fef0 0x7f6abc069de0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f6aac0097a0 tx=0x7f6aac009060 comp rx=0 tx=0).stop 2022-06-27T12:10:38.509 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.480+0000 7f6ac1145700 1 -- 172.21.15.195:0/2734094650 shutdown_connections 2022-06-27T12:10:38.509 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.480+0000 7f6ac1145700 1 --2- 172.21.15.195:0/2734094650 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6abc06e850 0x7f6abc1570a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:38.510 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.480+0000 7f6ac1145700 1 --2- 172.21.15.195:0/2734094650 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6abc06fef0 0x7f6abc069de0 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:38.512 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.480+0000 7f6ac1145700 1 -- 172.21.15.195:0/2734094650 >> 172.21.15.195:0/2734094650 conn(0x7f6abc0c6350 msgr2=0x7f6abc0c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:10:38.513 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.481+0000 7f6ac1145700 1 -- 172.21.15.195:0/2734094650 shutdown_connections 2022-06-27T12:10:38.513 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.481+0000 7f6ac1145700 1 -- 172.21.15.195:0/2734094650 wait complete. 2022-06-27T12:10:38.514 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.482+0000 7f6ac1145700 1 Processor -- start 2022-06-27T12:10:38.514 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.482+0000 7f6ac1145700 1 -- start start 2022-06-27T12:10:38.514 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.483+0000 7f6ac1145700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6abc06e850 0x7f6abc14dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:38.515 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.483+0000 7f6ac1145700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6abc0c34a0 0x7f6abc14e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:38.515 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.483+0000 7f6ac1145700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6abc0c6c30 0x7f6abc149280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:38.516 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.483+0000 7f6ac1145700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6abc068e20 con 0x7f6abc0c6c30 2022-06-27T12:10:38.516 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.483+0000 7f6ac1145700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6abc14d530 con 0x7f6abc06e850 2022-06-27T12:10:38.517 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.483+0000 7f6ac1145700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6abc149f60 con 0x7f6abc0c34a0 2022-06-27T12:10:38.517 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.483+0000 7f6abb59e700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6abc0c6c30 0x7f6abc149280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:38.518 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.483+0000 7f6abb59e700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6abc0c6c30 0x7f6abc149280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50028/0 (socket says 172.21.15.195:50028) 2022-06-27T12:10:38.519 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.483+0000 7f6abb59e700 1 -- 172.21.15.195:0/3806767037 learned_addr learned my addr 172.21.15.195:0/3806767037 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:10:38.519 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.483+0000 7f6aba59c700 1 --2- 172.21.15.195:0/3806767037 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6abc0c34a0 0x7f6abc14e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:38.521 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.483+0000 7f6abad9d700 1 --2- 172.21.15.195:0/3806767037 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6abc06e850 0x7f6abc14dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:38.521 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.483+0000 7f6abb59e700 1 -- 172.21.15.195:0/3806767037 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6abc0c34a0 msgr2=0x7f6abc14e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:38.522 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.483+0000 7f6abb59e700 1 --2- 172.21.15.195:0/3806767037 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6abc0c34a0 0x7f6abc14e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:38.523 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.483+0000 7f6abb59e700 1 -- 172.21.15.195:0/3806767037 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6abc06e850 msgr2=0x7f6abc14dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:38.523 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.483+0000 7f6abb59e700 1 --2- 172.21.15.195:0/3806767037 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6abc06e850 0x7f6abc14dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:38.524 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.483+0000 7f6abb59e700 1 -- 172.21.15.195:0/3806767037 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aac0092e0 con 0x7f6abc0c6c30 2022-06-27T12:10:38.524 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.483+0000 7f6abb59e700 1 --2- 172.21.15.195:0/3806767037 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6abc0c6c30 0x7f6abc149280 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f6ab400c960 tx=0x7f6ab40016f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:38.525 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.483+0000 7f6aa2ffd700 1 -- 172.21.15.195:0/3806767037 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab4004c80 con 0x7f6abc0c6c30 2022-06-27T12:10:38.525 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.483+0000 7f6aa2ffd700 1 -- 172.21.15.195:0/3806767037 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ab4004de0 con 0x7f6abc0c6c30 2022-06-27T12:10:38.526 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.483+0000 7f6ac1145700 1 -- 172.21.15.195:0/3806767037 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6abc14b0b0 con 0x7f6abc0c6c30 2022-06-27T12:10:38.526 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.483+0000 7f6aa2ffd700 1 -- 172.21.15.195:0/3806767037 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab40073a0 con 0x7f6abc0c6c30 2022-06-27T12:10:38.527 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.483+0000 7f6ac1145700 1 -- 172.21.15.195:0/3806767037 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6abc14d780 con 0x7f6abc0c6c30 2022-06-27T12:10:38.527 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.484+0000 7f6ac1145700 1 -- 172.21.15.195:0/3806767037 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6abc14b5d0 con 0x7f6abc0c6c30 2022-06-27T12:10:38.528 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.485+0000 7f6aa2ffd700 1 -- 172.21.15.195:0/3806767037 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f6ab401a030 con 0x7f6abc0c6c30 2022-06-27T12:10:38.528 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.485+0000 7f6aa2ffd700 1 --2- 172.21.15.195:0/3806767037 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6a9c063d00 0x7f6a9c0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:38.529 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.485+0000 7f6aa2ffd700 1 -- 172.21.15.195:0/3806767037 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7320+0+0 (secure 0 0 0) 0x7f6ab4080260 con 0x7f6abc0c6c30 2022-06-27T12:10:38.529 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.485+0000 7f6abad9d700 1 --2- 172.21.15.195:0/3806767037 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6a9c063d00 0x7f6a9c0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:38.539 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.486+0000 7f6abad9d700 1 --2- 172.21.15.195:0/3806767037 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6a9c063d00 0x7f6a9c0661b0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f6ab0009bd0 tx=0x7f6ab0009060 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:38.540 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.489+0000 7f6aa2ffd700 1 -- 172.21.15.195:0/3806767037 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f6ab404e9e0 con 0x7f6abc0c6c30 2022-06-27T12:10:38.730 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.726+0000 7f6ac1145700 1 -- 172.21.15.195:0/3806767037 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6abc14d970 con 0x7f6abc0c6c30 2022-06-27T12:10:38.730 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.726+0000 7f6aa2ffd700 1 -- 172.21.15.195:0/3806767037 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+5367 (secure 0 0 0) 0x7f6ab4018070 con 0x7f6abc0c6c30 2022-06-27T12:10:38.731 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:10:38.741 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:10:30.134487+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5933},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5933":{"gid":5933,"name":"d","rank":0,"incarnation":85,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6835/1342896965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":1342896965},{"type":"v1","addr":"172.21.15.202:6835","nonce":1342896965}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:10:38.744 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.728+0000 7f6ac1145700 1 -- 172.21.15.195:0/3806767037 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6a9c063d00 msgr2=0x7f6a9c0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:38.745 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.728+0000 7f6ac1145700 1 --2- 172.21.15.195:0/3806767037 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6a9c063d00 0x7f6a9c0661b0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f6ab0009bd0 tx=0x7f6ab0009060 comp rx=0 tx=0).stop 2022-06-27T12:10:38.745 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.728+0000 7f6ac1145700 1 -- 172.21.15.195:0/3806767037 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6abc0c6c30 msgr2=0x7f6abc149280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:38.746 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.728+0000 7f6ac1145700 1 --2- 172.21.15.195:0/3806767037 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6abc0c6c30 0x7f6abc149280 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f6ab400c960 tx=0x7f6ab40016f0 comp rx=0 tx=0).stop 2022-06-27T12:10:38.747 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.728+0000 7f6ac1145700 1 -- 172.21.15.195:0/3806767037 shutdown_connections 2022-06-27T12:10:38.747 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.728+0000 7f6ac1145700 1 --2- 172.21.15.195:0/3806767037 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6a9c063d00 0x7f6a9c0661b0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:38.747 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.728+0000 7f6ac1145700 1 --2- 172.21.15.195:0/3806767037 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6abc0c6c30 0x7f6abc149280 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:38.748 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.728+0000 7f6ac1145700 1 --2- 172.21.15.195:0/3806767037 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6abc06e850 0x7f6abc14dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:38.748 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.728+0000 7f6ac1145700 1 --2- 172.21.15.195:0/3806767037 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6abc0c34a0 0x7f6abc14e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:38.748 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.728+0000 7f6ac1145700 1 -- 172.21.15.195:0/3806767037 >> 172.21.15.195:0/3806767037 conn(0x7f6abc0c6350 msgr2=0x7f6abc156500 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:10:38.749 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.729+0000 7f6ac1145700 1 -- 172.21.15.195:0/3806767037 shutdown_connections 2022-06-27T12:10:38.749 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:38.729+0000 7f6ac1145700 1 -- 172.21.15.195:0/3806767037 wait complete. 2022-06-27T12:10:38.749 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 85 2022-06-27T12:10:38.760 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:10:40.766 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:10:40.927 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:40.916+0000 7f71a8b3a700 1 Processor -- start 2022-06-27T12:10:40.928 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:40.916+0000 7f71a8b3a700 1 -- start start 2022-06-27T12:10:40.928 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:40.917+0000 7f71a8b3a700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f71a406f620 0x7f71a4156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:40.928 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:40.917+0000 7f71a8b3a700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71a40c76b0 con 0x7f71a4150580 2022-06-27T12:10:40.928 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:40.917+0000 7f71a8b3a700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71a40c8250 con 0x7f71a4150950 2022-06-27T12:10:40.928 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:40.917+0000 7f71a8b3a700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71a4157430 con 0x7f71a406f620 2022-06-27T12:10:40.929 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:40.917+0000 7f71a259c700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f71a4150580 0x7f71a40c7b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:56222/0 (socket says 172.21.15.195:56222) 2022-06-27T12:10:40.929 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:40.917+0000 7f71a259c700 1 -- 172.21.15.195:0/2979704554 learned_addr learned my addr 172.21.15.195:0/2979704554 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:10:40.929 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:40.917+0000 7f71a2d9d700 1 --2- 172.21.15.195:0/2979704554 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f71a406f620 0x7f71a4156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:40.929 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:40.917+0000 7f71a159a700 1 -- 172.21.15.195:0/2979704554 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2026009025 0 0) 0x7f71a40c76b0 con 0x7f71a4150580 2022-06-27T12:10:40.939 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:40.917+0000 7f71a159a700 1 -- 172.21.15.195:0/2979704554 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7188003430 con 0x7f71a4150580 2022-06-27T12:10:40.939 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:40.917+0000 7f71a159a700 1 -- 172.21.15.195:0/2979704554 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3200928427 0 0) 0x7f71a40c8250 con 0x7f71a4150950 2022-06-27T12:10:40.940 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:40.917+0000 7f71a159a700 1 -- 172.21.15.195:0/2979704554 --> v1:172.21.15.202:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71a40c76b0 con 0x7f71a4150950 2022-06-27T12:10:40.964 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:40.917+0000 7f71a2d9d700 1 -- 172.21.15.195:0/2979704554 >> v1:172.21.15.202:6789/0 conn(0x7f71a4150950 legacy=0x7f71a406d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:40.964 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:40.917+0000 7f71a2d9d700 1 -- 172.21.15.195:0/2979704554 >> v1:172.21.15.195:6789/0 conn(0x7f71a4150580 legacy=0x7f71a40c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:40.965 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:40.917+0000 7f71a2d9d700 1 -- 172.21.15.195:0/2979704554 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71a40bad70 con 0x7f71a406f620 2022-06-27T12:10:40.965 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:40.918+0000 7f71a2d9d700 1 --2- 172.21.15.195:0/2979704554 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f71a406f620 0x7f71a4156ef0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f719c009e90 tx=0x7f719c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b74969b18c67fdc server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:40.965 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:40.918+0000 7f71a159a700 1 -- 172.21.15.195:0/2979704554 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f719c00e070 con 0x7f71a406f620 2022-06-27T12:10:40.966 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:40.918+0000 7f71a159a700 1 -- 172.21.15.195:0/2979704554 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f719c00bb80 con 0x7f71a406f620 2022-06-27T12:10:40.966 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:40.918+0000 7f71a159a700 1 -- 172.21.15.195:0/2979704554 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f719c012430 con 0x7f71a406f620 2022-06-27T12:10:40.967 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:40.918+0000 7f71a8b3a700 1 -- 172.21.15.195:0/2979704554 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f71a406f620 msgr2=0x7f71a4156ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:40.968 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:40.918+0000 7f71a8b3a700 1 --2- 172.21.15.195:0/2979704554 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f71a406f620 0x7f71a4156ef0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f719c009e90 tx=0x7f719c009780 comp rx=0 tx=0).stop 2022-06-27T12:10:40.968 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:40.918+0000 7f71a8b3a700 1 -- 172.21.15.195:0/2979704554 shutdown_connections 2022-06-27T12:10:40.968 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:40.918+0000 7f71a8b3a700 1 --2- 172.21.15.195:0/2979704554 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f71a406f620 0x7f71a4156ef0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:40.969 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:40.918+0000 7f71a8b3a700 1 -- 172.21.15.195:0/2979704554 >> 172.21.15.195:0/2979704554 conn(0x7f71a4063af0 msgr2=0x7f71a4063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:10:40.969 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:40.918+0000 7f71a8b3a700 1 -- 172.21.15.195:0/2979704554 shutdown_connections 2022-06-27T12:10:40.970 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:40.918+0000 7f71a8b3a700 1 -- 172.21.15.195:0/2979704554 wait complete. 2022-06-27T12:10:40.970 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:40.920+0000 7f71a8b3a700 1 Processor -- start 2022-06-27T12:10:40.971 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:40.920+0000 7f71a8b3a700 1 -- start start 2022-06-27T12:10:40.971 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:40.920+0000 7f71a8b3a700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f71a406f620 0x7f71a40c8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:40.971 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:40.920+0000 7f71a8b3a700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f71a4150580 0x7f71a4158ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:40.972 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:40.920+0000 7f71a8b3a700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f71a4150950 0x7f71a4169870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:40.972 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:40.920+0000 7f71a8b3a700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71a40c3930 con 0x7f71a4150950 2022-06-27T12:10:40.973 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:40.920+0000 7f71a8b3a700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71a40cb850 con 0x7f71a406f620 2022-06-27T12:10:40.973 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:40.920+0000 7f71a8b3a700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71a40c8ff0 con 0x7f71a4150580 2022-06-27T12:10:40.974 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:40.920+0000 7f71a2d9d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f71a4150950 0x7f71a4169870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:40.974 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:40.920+0000 7f71a259c700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f71a406f620 0x7f71a40c8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:40.975 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:40.920+0000 7f71a2d9d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f71a4150950 0x7f71a4169870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50044/0 (socket says 172.21.15.195:50044) 2022-06-27T12:10:40.975 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:40.920+0000 7f71a259c700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f71a406f620 0x7f71a40c8ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3300/0 says I am v2:172.21.15.195:48166/0 (socket says 172.21.15.195:48166) 2022-06-27T12:10:40.976 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:40.920+0000 7f71a1d9b700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f71a4150580 0x7f71a4158ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:40.976 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:40.920+0000 7f71a2d9d700 1 -- 172.21.15.195:0/32479749 learned_addr learned my addr 172.21.15.195:0/32479749 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:10:40.976 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:40.921+0000 7f71a2d9d700 1 -- 172.21.15.195:0/32479749 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f71a4150580 msgr2=0x7f71a4158ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:40.977 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:40.921+0000 7f71a2d9d700 1 --2- 172.21.15.195:0/32479749 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f71a4150580 0x7f71a4158ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:40.977 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:40.921+0000 7f71a2d9d700 1 -- 172.21.15.195:0/32479749 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f71a406f620 msgr2=0x7f71a40c8ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:40.978 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:40.921+0000 7f71a2d9d700 1 --2- 172.21.15.195:0/32479749 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f71a406f620 0x7f71a40c8ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:40.978 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:40.921+0000 7f71a2d9d700 1 -- 172.21.15.195:0/32479749 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f719c009a20 con 0x7f71a4150950 2022-06-27T12:10:40.979 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:40.921+0000 7f71a2d9d700 1 --2- 172.21.15.195:0/32479749 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f71a4150950 0x7f71a4169870 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f71a4065b00 tx=0x7f719c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:40.979 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:40.921+0000 7f71927fc700 1 -- 172.21.15.195:0/32479749 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f719c0263f0 con 0x7f71a4150950 2022-06-27T12:10:40.980 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:40.921+0000 7f71927fc700 1 -- 172.21.15.195:0/32479749 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f719c026570 con 0x7f71a4150950 2022-06-27T12:10:40.980 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:40.921+0000 7f71a8b3a700 1 -- 172.21.15.195:0/32479749 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71a40c47c0 con 0x7f71a4150950 2022-06-27T12:10:40.981 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:40.921+0000 7f71927fc700 1 -- 172.21.15.195:0/32479749 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f719c026930 con 0x7f71a4150950 2022-06-27T12:10:40.981 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:40.921+0000 7f71a8b3a700 1 -- 172.21.15.195:0/32479749 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71a40cb570 con 0x7f71a4150950 2022-06-27T12:10:40.981 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:40.922+0000 7f71a8b3a700 1 -- 172.21.15.195:0/32479749 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71a40722c0 con 0x7f71a4150950 2022-06-27T12:10:40.982 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:40.922+0000 7f71927fc700 1 -- 172.21.15.195:0/32479749 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f719c026ca0 con 0x7f71a4150950 2022-06-27T12:10:40.982 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:40.923+0000 7f71927fc700 1 --2- 172.21.15.195:0/32479749 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7180063d00 0x7f71800661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:40.983 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:40.923+0000 7f71927fc700 1 -- 172.21.15.195:0/32479749 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7320+0+0 (secure 0 0 0) 0x7f719c0830b0 con 0x7f71a4150950 2022-06-27T12:10:40.983 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:40.923+0000 7f71a259c700 1 --2- 172.21.15.195:0/32479749 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7180063d00 0x7f71800661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:40.984 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:40.923+0000 7f71a259c700 1 --2- 172.21.15.195:0/32479749 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7180063d00 0x7f71800661b0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f7194007fd0 tx=0x7f7194009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:40.985 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:40.927+0000 7f71927fc700 1 -- 172.21.15.195:0/32479749 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f719c0517b0 con 0x7f71a4150950 2022-06-27T12:10:41.180 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:41.159+0000 7f71a8b3a700 1 -- 172.21.15.195:0/32479749 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f71a414df20 con 0x7f71a4150950 2022-06-27T12:10:41.180 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:41.160+0000 7f71927fc700 1 -- 172.21.15.195:0/32479749 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+5367 (secure 0 0 0) 0x7f719c054dd0 con 0x7f71a4150950 2022-06-27T12:10:41.181 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:10:41.190 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:10:30.134487+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5933},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5933":{"gid":5933,"name":"d","rank":0,"incarnation":85,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6835/1342896965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":1342896965},{"type":"v1","addr":"172.21.15.202:6835","nonce":1342896965}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:10:41.195 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:41.162+0000 7f71a8b3a700 1 -- 172.21.15.195:0/32479749 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7180063d00 msgr2=0x7f71800661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:41.195 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:41.162+0000 7f71a8b3a700 1 --2- 172.21.15.195:0/32479749 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7180063d00 0x7f71800661b0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f7194007fd0 tx=0x7f7194009040 comp rx=0 tx=0).stop 2022-06-27T12:10:41.196 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:41.162+0000 7f71a8b3a700 1 -- 172.21.15.195:0/32479749 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f71a4150950 msgr2=0x7f71a4169870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:41.196 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:41.162+0000 7f71a8b3a700 1 --2- 172.21.15.195:0/32479749 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f71a4150950 0x7f71a4169870 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f71a4065b00 tx=0x7f719c019000 comp rx=0 tx=0).stop 2022-06-27T12:10:41.197 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:41.162+0000 7f71a8b3a700 1 -- 172.21.15.195:0/32479749 shutdown_connections 2022-06-27T12:10:41.198 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:41.162+0000 7f71a8b3a700 1 --2- 172.21.15.195:0/32479749 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7180063d00 0x7f71800661b0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:41.198 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:41.162+0000 7f71a8b3a700 1 --2- 172.21.15.195:0/32479749 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f71a4150950 0x7f71a4169870 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:41.199 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:41.162+0000 7f71a8b3a700 1 --2- 172.21.15.195:0/32479749 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f71a406f620 0x7f71a40c8ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:41.199 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:41.162+0000 7f71a8b3a700 1 --2- 172.21.15.195:0/32479749 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f71a4150580 0x7f71a4158ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:41.200 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:41.162+0000 7f71a8b3a700 1 -- 172.21.15.195:0/32479749 >> 172.21.15.195:0/32479749 conn(0x7f71a4063af0 msgr2=0x7f71a414e140 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:10:41.200 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:41.162+0000 7f71a8b3a700 1 -- 172.21.15.195:0/32479749 shutdown_connections 2022-06-27T12:10:41.200 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:41.162+0000 7f71a8b3a700 1 -- 172.21.15.195:0/32479749 wait complete. 2022-06-27T12:10:41.201 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 85 2022-06-27T12:10:41.202 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:10:43.203 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:10:43.373 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.345+0000 7f2b869e1700 1 Processor -- start 2022-06-27T12:10:43.373 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.345+0000 7f2b869e1700 1 -- start start 2022-06-27T12:10:43.374 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.346+0000 7f2b869e1700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2b8014e860 0x7f2b800c3840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:43.374 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.346+0000 7f2b869e1700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f2b800c3d80 0x7f2b80157890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:43.378 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.346+0000 7f2b869e1700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b800c33b0 con 0x7f2b8014e490 2022-06-27T12:10:43.379 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.346+0000 7f2b869e1700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b800c4200 con 0x7f2b800c3d80 2022-06-27T12:10:43.381 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.346+0000 7f2b869e1700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b8006f220 con 0x7f2b8014e860 2022-06-27T12:10:43.382 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.346+0000 7f2b7ffff700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f2b8014e490 0x7f2b8006cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:56238/0 (socket says 172.21.15.195:56238) 2022-06-27T12:10:43.382 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.346+0000 7f2b7ffff700 1 -- 172.21.15.195:0/912037900 learned_addr learned my addr 172.21.15.195:0/912037900 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:10:43.382 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.346+0000 7f2b7f7fe700 1 --2- 172.21.15.195:0/912037900 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2b8014e860 0x7f2b800c3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:43.383 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.346+0000 7f2b84f7e700 1 --2- 172.21.15.195:0/912037900 >> v2:172.21.15.202:3300/0 conn(0x7f2b800c3d80 0x7f2b80157890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:43.383 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.346+0000 7f2b7effd700 1 -- 172.21.15.195:0/912037900 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 107767247 0 0) 0x7f2b800c33b0 con 0x7f2b8014e490 2022-06-27T12:10:43.384 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.346+0000 7f2b7effd700 1 -- 172.21.15.195:0/912037900 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b60003430 con 0x7f2b8014e490 2022-06-27T12:10:43.384 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.346+0000 7f2b7f7fe700 1 -- 172.21.15.195:0/912037900 >> v2:172.21.15.202:3300/0 conn(0x7f2b800c3d80 msgr2=0x7f2b80157890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:10:43.385 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.346+0000 7f2b7f7fe700 1 --2- 172.21.15.195:0/912037900 >> v2:172.21.15.202:3300/0 conn(0x7f2b800c3d80 0x7f2b80157890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:43.385 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.346+0000 7f2b7f7fe700 1 -- 172.21.15.195:0/912037900 >> v1:172.21.15.195:6789/0 conn(0x7f2b8014e490 legacy=0x7f2b8006cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:43.385 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.346+0000 7f2b7f7fe700 1 -- 172.21.15.195:0/912037900 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b801542c0 con 0x7f2b8014e860 2022-06-27T12:10:43.386 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.347+0000 7f2b7f7fe700 1 --2- 172.21.15.195:0/912037900 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2b8014e860 0x7f2b800c3840 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2b700097a0 tx=0x7f2b70009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e74e0939cb4ad684 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:43.386 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.347+0000 7f2b7effd700 1 -- 172.21.15.195:0/912037900 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b70011630 con 0x7f2b8014e860 2022-06-27T12:10:43.399 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.347+0000 7f2b7effd700 1 -- 172.21.15.195:0/912037900 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b70011790 con 0x7f2b8014e860 2022-06-27T12:10:43.399 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.347+0000 7f2b7effd700 1 -- 172.21.15.195:0/912037900 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b70011b10 con 0x7f2b8014e860 2022-06-27T12:10:43.401 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.347+0000 7f2b869e1700 1 -- 172.21.15.195:0/912037900 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2b8014e860 msgr2=0x7f2b800c3840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:43.401 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.347+0000 7f2b869e1700 1 --2- 172.21.15.195:0/912037900 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2b8014e860 0x7f2b800c3840 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2b700097a0 tx=0x7f2b70009060 comp rx=0 tx=0).stop 2022-06-27T12:10:43.410 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.347+0000 7f2b869e1700 1 -- 172.21.15.195:0/912037900 shutdown_connections 2022-06-27T12:10:43.410 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.347+0000 7f2b869e1700 1 --2- 172.21.15.195:0/912037900 >> v2:172.21.15.202:3300/0 conn(0x7f2b800c3d80 0x7f2b80157890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:43.411 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.347+0000 7f2b869e1700 1 --2- 172.21.15.195:0/912037900 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2b8014e860 0x7f2b800c3840 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:43.411 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.347+0000 7f2b869e1700 1 -- 172.21.15.195:0/912037900 >> 172.21.15.195:0/912037900 conn(0x7f2b80063af0 msgr2=0x7f2b80063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:10:43.412 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.347+0000 7f2b869e1700 1 -- 172.21.15.195:0/912037900 shutdown_connections 2022-06-27T12:10:43.412 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.347+0000 7f2b869e1700 1 -- 172.21.15.195:0/912037900 wait complete. 2022-06-27T12:10:43.413 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.348+0000 7f2b869e1700 1 Processor -- start 2022-06-27T12:10:43.413 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.349+0000 7f2b869e1700 1 -- start start 2022-06-27T12:10:43.414 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.349+0000 7f2b869e1700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2b800c3d80 0x7f2b80165160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:43.414 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.349+0000 7f2b869e1700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2b8014e490 0x7f2b800caaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:43.415 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.349+0000 7f2b869e1700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2b8014e860 0x7f2b80165e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:43.415 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.349+0000 7f2b869e1700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b80156d00 con 0x7f2b8014e490 2022-06-27T12:10:43.416 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.349+0000 7f2b869e1700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b80157f90 con 0x7f2b8014e860 2022-06-27T12:10:43.416 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.349+0000 7f2b869e1700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b80155e10 con 0x7f2b800c3d80 2022-06-27T12:10:43.416 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.349+0000 7f2b7f7fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2b8014e490 0x7f2b800caaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:43.417 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.350+0000 7f2b7f7fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2b8014e490 0x7f2b800caaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50060/0 (socket says 172.21.15.195:50060) 2022-06-27T12:10:43.417 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.350+0000 7f2b7f7fe700 1 -- 172.21.15.195:0/3806137420 learned_addr learned my addr 172.21.15.195:0/3806137420 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:10:43.434 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.350+0000 7f2b7ffff700 1 --2- 172.21.15.195:0/3806137420 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2b800c3d80 0x7f2b80165160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:43.435 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.350+0000 7f2b7f7fe700 1 -- 172.21.15.195:0/3806137420 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2b800c3d80 msgr2=0x7f2b80165160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:43.436 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.350+0000 7f2b7f7fe700 1 --2- 172.21.15.195:0/3806137420 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2b800c3d80 0x7f2b80165160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:43.436 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.350+0000 7f2b7f7fe700 1 -- 172.21.15.195:0/3806137420 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2b8014e860 msgr2=0x7f2b80165e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:43.437 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.350+0000 7f2b84f7e700 1 --2- 172.21.15.195:0/3806137420 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2b8014e860 0x7f2b80165e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:43.437 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.350+0000 7f2b7f7fe700 1 --2- 172.21.15.195:0/3806137420 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2b8014e860 0x7f2b80165e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:43.438 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.350+0000 7f2b7f7fe700 1 -- 172.21.15.195:0/3806137420 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b700092e0 con 0x7f2b8014e490 2022-06-27T12:10:43.438 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.350+0000 7f2b84f7e700 1 --2- 172.21.15.195:0/3806137420 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2b8014e860 0x7f2b80165e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T12:10:43.439 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.350+0000 7f2b7f7fe700 1 --2- 172.21.15.195:0/3806137420 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2b8014e490 0x7f2b800caaf0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f2b800be550 tx=0x7f2b70017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:43.439 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.350+0000 7f2b77fff700 1 -- 172.21.15.195:0/3806137420 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b70003880 con 0x7f2b8014e490 2022-06-27T12:10:43.440 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.350+0000 7f2b77fff700 1 -- 172.21.15.195:0/3806137420 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b700039e0 con 0x7f2b8014e490 2022-06-27T12:10:43.440 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.350+0000 7f2b869e1700 1 -- 172.21.15.195:0/3806137420 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b801560f0 con 0x7f2b8014e490 2022-06-27T12:10:43.440 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.350+0000 7f2b77fff700 1 -- 172.21.15.195:0/3806137420 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b70003da0 con 0x7f2b8014e490 2022-06-27T12:10:43.441 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.350+0000 7f2b869e1700 1 -- 172.21.15.195:0/3806137420 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b801566b0 con 0x7f2b8014e490 2022-06-27T12:10:43.441 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.351+0000 7f2b869e1700 1 -- 172.21.15.195:0/3806137420 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b801656a0 con 0x7f2b8014e490 2022-06-27T12:10:43.442 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.352+0000 7f2b77fff700 1 -- 172.21.15.195:0/3806137420 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f2b7000f050 con 0x7f2b8014e490 2022-06-27T12:10:43.442 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.352+0000 7f2b77fff700 1 --2- 172.21.15.195:0/3806137420 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2b64063d00 0x7f2b640661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:43.443 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.352+0000 7f2b77fff700 1 -- 172.21.15.195:0/3806137420 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7320+0+0 (secure 0 0 0) 0x7f2b700824c0 con 0x7f2b8014e490 2022-06-27T12:10:43.443 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.352+0000 7f2b7ffff700 1 --2- 172.21.15.195:0/3806137420 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2b64063d00 0x7f2b640661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:43.444 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.352+0000 7f2b7ffff700 1 --2- 172.21.15.195:0/3806137420 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2b64063d00 0x7f2b640661b0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f2b68005790 tx=0x7f2b68009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:43.444 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.356+0000 7f2b77fff700 1 -- 172.21.15.195:0/3806137420 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f2b70050bc0 con 0x7f2b8014e490 2022-06-27T12:10:43.584 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.583+0000 7f2b869e1700 1 -- 172.21.15.195:0/3806137420 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2b800722c0 con 0x7f2b8014e490 2022-06-27T12:10:43.596 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.583+0000 7f2b77fff700 1 -- 172.21.15.195:0/3806137420 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+5371 (secure 0 0 0) 0x7f2b70015070 con 0x7f2b8014e490 2022-06-27T12:10:43.597 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:10:43.597 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:10:43.197876+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5933},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5933":{"gid":5933,"name":"d","rank":0,"incarnation":85,"state":"up:reconnect","state_seq":11,"addr":"172.21.15.202:6835/1342896965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":1342896965},{"type":"v1","addr":"172.21.15.202:6835","nonce":1342896965}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:10:43.601 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.586+0000 7f2b869e1700 1 -- 172.21.15.195:0/3806137420 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2b64063d00 msgr2=0x7f2b640661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:43.602 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.586+0000 7f2b869e1700 1 --2- 172.21.15.195:0/3806137420 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2b64063d00 0x7f2b640661b0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f2b68005790 tx=0x7f2b68009040 comp rx=0 tx=0).stop 2022-06-27T12:10:43.603 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.586+0000 7f2b869e1700 1 -- 172.21.15.195:0/3806137420 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2b8014e490 msgr2=0x7f2b800caaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:43.603 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.586+0000 7f2b869e1700 1 --2- 172.21.15.195:0/3806137420 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2b8014e490 0x7f2b800caaf0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f2b800be550 tx=0x7f2b70017000 comp rx=0 tx=0).stop 2022-06-27T12:10:43.604 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.586+0000 7f2b869e1700 1 -- 172.21.15.195:0/3806137420 shutdown_connections 2022-06-27T12:10:43.604 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.586+0000 7f2b869e1700 1 --2- 172.21.15.195:0/3806137420 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2b64063d00 0x7f2b640661b0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:43.605 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.586+0000 7f2b869e1700 1 --2- 172.21.15.195:0/3806137420 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2b8014e860 0x7f2b80165e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:43.605 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.586+0000 7f2b869e1700 1 --2- 172.21.15.195:0/3806137420 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2b800c3d80 0x7f2b80165160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:43.606 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.586+0000 7f2b869e1700 1 --2- 172.21.15.195:0/3806137420 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2b8014e490 0x7f2b800caaf0 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:43.606 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.586+0000 7f2b869e1700 1 -- 172.21.15.195:0/3806137420 >> 172.21.15.195:0/3806137420 conn(0x7f2b80063af0 msgr2=0x7f2b8006cfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:10:43.606 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.586+0000 7f2b869e1700 1 -- 172.21.15.195:0/3806137420 shutdown_connections 2022-06-27T12:10:43.607 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:43.586+0000 7f2b869e1700 1 -- 172.21.15.195:0/3806137420 wait complete. 2022-06-27T12:10:43.624 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 86 2022-06-27T12:10:43.657 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 86, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-06-27T11:47:52.358095+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 86, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5933": { "addr": "172.21.15.202:6835/1342896965", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6834", "nonce": 1342896965, "type": "v2" }, { "addr": "172.21.15.202:6835", "nonce": 1342896965, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5933, "incarnation": 85, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:reconnect", "state_seq": 11 } }, "last_failure": 0, "last_failure_osd_epoch": 61, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-06-27T12:10:43.197876+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5933 } } } ], "standbys": [ { "addr": "172.21.15.202:6837/3305006841", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6836", "nonce": 3305006841, "type": "v2" }, { "addr": "172.21.15.202:6837", "nonce": 3305006841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6835/2060709822", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6834", "nonce": 2060709822, "type": "v2" }, { "addr": "172.21.15.195:6835", "nonce": 2060709822, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6838/1598751652", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6836", "nonce": 1598751652, "type": "v2" }, { "addr": "172.21.15.195:6838", "nonce": 1598751652, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4275, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6839/1313800661", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6837", "nonce": 1313800661, "type": "v2" }, { "addr": "172.21.15.195:6839", "nonce": 1313800661, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4278, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-06-27T12:10:45.659 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:10:45.828 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:45.795+0000 7f2e23286700 1 Processor -- start 2022-06-27T12:10:45.828 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:45.795+0000 7f2e23286700 1 -- start start 2022-06-27T12:10:45.829 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:45.796+0000 7f2e23286700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f2e1c14e490 0x7f2e1c06cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:45.829 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:45.796+0000 7f2e23286700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f2e1c0c3f50 0x7f2e1c157890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:45.831 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:45.796+0000 7f2e23286700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e1c0c33b0 con 0x7f2e1c14e860 2022-06-27T12:10:45.831 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:45.796+0000 7f2e23286700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e1c06d520 con 0x7f2e1c0c3f50 2022-06-27T12:10:45.832 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:45.796+0000 7f2e23286700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e1c06f220 con 0x7f2e1c14e490 2022-06-27T12:10:45.832 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:45.796+0000 7f2e21823700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f2e1c0c3f50 0x7f2e1c157890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:45.833 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:45.796+0000 7f2e21823700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f2e1c0c3f50 0x7f2e1c157890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50068/0 (socket says 172.21.15.195:50068) 2022-06-27T12:10:45.833 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:45.796+0000 7f2e21823700 1 -- 172.21.15.195:0/3081716657 learned_addr learned my addr 172.21.15.195:0/3081716657 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:10:45.834 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:45.796+0000 7f2e21823700 1 -- 172.21.15.195:0/3081716657 >> v2:172.21.15.202:3300/0 conn(0x7f2e1c14e490 msgr2=0x7f2e1c06cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:10:45.834 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:45.796+0000 7f2e21823700 1 --2- 172.21.15.195:0/3081716657 >> v2:172.21.15.202:3300/0 conn(0x7f2e1c14e490 0x7f2e1c06cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:45.835 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:45.796+0000 7f2e21823700 1 -- 172.21.15.195:0/3081716657 >> v1:172.21.15.202:6789/0 conn(0x7f2e1c14e860 legacy=0x7f2e1c0c3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:10:45.835 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:45.797+0000 7f2e21823700 1 -- 172.21.15.195:0/3081716657 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e1c1542c0 con 0x7f2e1c0c3f50 2022-06-27T12:10:45.835 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:45.797+0000 7f2e21823700 1 --2- 172.21.15.195:0/3081716657 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2e1c0c3f50 0x7f2e1c157890 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f2e1c064800 tx=0x7f2e14009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=128a03a47bc28fb8 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:45.836 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:45.797+0000 7f2e0bfff700 1 -- 172.21.15.195:0/3081716657 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e1400e070 con 0x7f2e1c0c3f50 2022-06-27T12:10:45.836 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:45.797+0000 7f2e0bfff700 1 -- 172.21.15.195:0/3081716657 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e1400ba10 con 0x7f2e1c0c3f50 2022-06-27T12:10:45.837 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:45.797+0000 7f2e0bfff700 1 -- 172.21.15.195:0/3081716657 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e140092e0 con 0x7f2e1c0c3f50 2022-06-27T12:10:45.837 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:45.797+0000 7f2e23286700 1 -- 172.21.15.195:0/3081716657 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2e1c0c3f50 msgr2=0x7f2e1c157890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:45.838 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:45.797+0000 7f2e23286700 1 --2- 172.21.15.195:0/3081716657 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2e1c0c3f50 0x7f2e1c157890 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f2e1c064800 tx=0x7f2e14009840 comp rx=0 tx=0).stop 2022-06-27T12:10:45.840 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:45.797+0000 7f2e23286700 1 -- 172.21.15.195:0/3081716657 shutdown_connections 2022-06-27T12:10:45.840 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:45.797+0000 7f2e23286700 1 --2- 172.21.15.195:0/3081716657 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2e1c0c3f50 0x7f2e1c157890 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:45.840 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:45.797+0000 7f2e23286700 1 --2- 172.21.15.195:0/3081716657 >> v2:172.21.15.202:3300/0 conn(0x7f2e1c14e490 0x7f2e1c06cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:45.841 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:45.797+0000 7f2e23286700 1 -- 172.21.15.195:0/3081716657 >> 172.21.15.195:0/3081716657 conn(0x7f2e1c063af0 msgr2=0x7f2e1c063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:10:45.841 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:45.797+0000 7f2e23286700 1 -- 172.21.15.195:0/3081716657 shutdown_connections 2022-06-27T12:10:45.842 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:45.798+0000 7f2e23286700 1 -- 172.21.15.195:0/3081716657 wait complete. 2022-06-27T12:10:45.842 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:45.799+0000 7f2e23286700 1 Processor -- start 2022-06-27T12:10:45.843 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:45.799+0000 7f2e23286700 1 -- start start 2022-06-27T12:10:45.843 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:45.800+0000 7f2e23286700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2e1c0c3f50 0x7f2e1c157850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:45.843 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:45.800+0000 7f2e23286700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2e1c14e490 0x7f2e1c167460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:45.844 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:45.800+0000 7f2e23286700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2e1c14e860 0x7f2e1c0c0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:45.844 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:45.800+0000 7f2e23286700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e1c1664b0 con 0x7f2e1c14e490 2022-06-27T12:10:45.845 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:45.800+0000 7f2e23286700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e1c158760 con 0x7f2e1c0c3f50 2022-06-27T12:10:45.845 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:45.800+0000 7f2e23286700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e1c157d90 con 0x7f2e1c14e860 2022-06-27T12:10:45.845 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:45.800+0000 7f2e20821700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2e1c14e490 0x7f2e1c167460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:45.850 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:45.800+0000 7f2e20821700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2e1c14e490 0x7f2e1c167460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50074/0 (socket says 172.21.15.195:50074) 2022-06-27T12:10:45.851 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:45.800+0000 7f2e20821700 1 -- 172.21.15.195:0/557207013 learned_addr learned my addr 172.21.15.195:0/557207013 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:10:45.851 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:45.800+0000 7f2e21022700 1 --2- 172.21.15.195:0/557207013 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2e1c0c3f50 0x7f2e1c157850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:45.862 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:45.800+0000 7f2e21823700 1 --2- 172.21.15.195:0/557207013 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2e1c14e860 0x7f2e1c0c0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:45.862 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:45.801+0000 7f2e20821700 1 -- 172.21.15.195:0/557207013 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2e1c14e860 msgr2=0x7f2e1c0c0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:45.863 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:45.801+0000 7f2e20821700 1 --2- 172.21.15.195:0/557207013 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2e1c14e860 0x7f2e1c0c0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:45.863 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:45.801+0000 7f2e20821700 1 -- 172.21.15.195:0/557207013 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2e1c0c3f50 msgr2=0x7f2e1c157850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:45.864 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:45.801+0000 7f2e20821700 1 --2- 172.21.15.195:0/557207013 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2e1c0c3f50 0x7f2e1c157850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:45.864 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:45.801+0000 7f2e20821700 1 -- 172.21.15.195:0/557207013 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e14009ae0 con 0x7f2e1c14e490 2022-06-27T12:10:45.865 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:45.801+0000 7f2e20821700 1 --2- 172.21.15.195:0/557207013 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2e1c14e490 0x7f2e1c167460 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f2e1800b490 tx=0x7f2e18006f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:45.865 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:45.801+0000 7f2e097fa700 1 -- 172.21.15.195:0/557207013 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e18011420 con 0x7f2e1c14e490 2022-06-27T12:10:45.874 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:45.801+0000 7f2e097fa700 1 -- 172.21.15.195:0/557207013 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e180115a0 con 0x7f2e1c14e490 2022-06-27T12:10:45.875 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:45.801+0000 7f2e23286700 1 -- 172.21.15.195:0/557207013 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e1c166fd0 con 0x7f2e1c14e490 2022-06-27T12:10:45.875 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:45.801+0000 7f2e097fa700 1 -- 172.21.15.195:0/557207013 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e18011960 con 0x7f2e1c14e490 2022-06-27T12:10:45.876 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:45.801+0000 7f2e23286700 1 -- 172.21.15.195:0/557207013 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e1c1581c0 con 0x7f2e1c14e490 2022-06-27T12:10:45.876 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:45.802+0000 7f2e23286700 1 -- 172.21.15.195:0/557207013 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e1c0722c0 con 0x7f2e1c14e490 2022-06-27T12:10:45.877 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:45.802+0000 7f2e097fa700 1 -- 172.21.15.195:0/557207013 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f2e18017020 con 0x7f2e1c14e490 2022-06-27T12:10:45.877 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:45.802+0000 7f2e097fa700 1 --2- 172.21.15.195:0/557207013 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2dfc063d30 0x7f2dfc0661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:45.878 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:45.803+0000 7f2e097fa700 1 -- 172.21.15.195:0/557207013 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7320+0+0 (secure 0 0 0) 0x7f2e180806e0 con 0x7f2e1c14e490 2022-06-27T12:10:45.878 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:45.803+0000 7f2e21022700 1 --2- 172.21.15.195:0/557207013 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2dfc063d30 0x7f2dfc0661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:45.879 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:45.803+0000 7f2e21022700 1 --2- 172.21.15.195:0/557207013 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2dfc063d30 0x7f2dfc0661e0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c009b40 tx=0x7f2e0c009060 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:45.879 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:45.807+0000 7f2e097fa700 1 -- 172.21.15.195:0/557207013 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f2e1804ede0 con 0x7f2e1c14e490 2022-06-27T12:10:46.034 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:46.033+0000 7f2e23286700 1 -- 172.21.15.195:0/557207013 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2e1c0c60d0 con 0x7f2e1c14e490 2022-06-27T12:10:46.035 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:46.034+0000 7f2e097fa700 1 -- 172.21.15.195:0/557207013 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+5371 (secure 0 0 0) 0x7f2e18052400 con 0x7f2e1c14e490 2022-06-27T12:10:46.036 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:10:46.036 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:10:43.197876+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5933},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5933":{"gid":5933,"name":"d","rank":0,"incarnation":85,"state":"up:reconnect","state_seq":11,"addr":"172.21.15.202:6835/1342896965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":1342896965},{"type":"v1","addr":"172.21.15.202:6835","nonce":1342896965}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:10:46.039 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:46.036+0000 7f2e23286700 1 -- 172.21.15.195:0/557207013 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2dfc063d30 msgr2=0x7f2dfc0661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:46.040 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:46.036+0000 7f2e23286700 1 --2- 172.21.15.195:0/557207013 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2dfc063d30 0x7f2dfc0661e0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c009b40 tx=0x7f2e0c009060 comp rx=0 tx=0).stop 2022-06-27T12:10:46.040 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:46.036+0000 7f2e23286700 1 -- 172.21.15.195:0/557207013 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2e1c14e490 msgr2=0x7f2e1c167460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:46.041 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:46.036+0000 7f2e23286700 1 --2- 172.21.15.195:0/557207013 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2e1c14e490 0x7f2e1c167460 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f2e1800b490 tx=0x7f2e18006f40 comp rx=0 tx=0).stop 2022-06-27T12:10:46.041 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:46.036+0000 7f2e23286700 1 -- 172.21.15.195:0/557207013 shutdown_connections 2022-06-27T12:10:46.042 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:46.036+0000 7f2e23286700 1 --2- 172.21.15.195:0/557207013 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2dfc063d30 0x7f2dfc0661e0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:46.046 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:46.036+0000 7f2e23286700 1 --2- 172.21.15.195:0/557207013 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2e1c14e860 0x7f2e1c0c0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:46.047 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:46.036+0000 7f2e23286700 1 --2- 172.21.15.195:0/557207013 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2e1c0c3f50 0x7f2e1c157850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:46.047 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:46.036+0000 7f2e23286700 1 --2- 172.21.15.195:0/557207013 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2e1c14e490 0x7f2e1c167460 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:46.048 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:46.036+0000 7f2e23286700 1 -- 172.21.15.195:0/557207013 >> 172.21.15.195:0/557207013 conn(0x7f2e1c063af0 msgr2=0x7f2e1c0c8e50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:10:46.048 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:46.036+0000 7f2e23286700 1 -- 172.21.15.195:0/557207013 shutdown_connections 2022-06-27T12:10:46.049 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:46.036+0000 7f2e23286700 1 -- 172.21.15.195:0/557207013 wait complete. 2022-06-27T12:10:46.049 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 86 2022-06-27T12:10:46.072 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:10:48.074 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:10:48.280 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.230+0000 7f1f2d0e2700 1 Processor -- start 2022-06-27T12:10:48.281 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.231+0000 7f1f2d0e2700 1 -- start start 2022-06-27T12:10:48.281 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.232+0000 7f1f2d0e2700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f1f280c7480 0x7f1f280c4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:48.282 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.232+0000 7f1f2d0e2700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f281573b0 con 0x7f1f280683c0 2022-06-27T12:10:48.282 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.232+0000 7f1f2d0e2700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f280c46c0 con 0x7f1f28068f80 2022-06-27T12:10:48.283 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.232+0000 7f1f2d0e2700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f28068840 con 0x7f1f280c7480 2022-06-27T12:10:48.290 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.232+0000 7f1f2759e700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f1f280683c0 0x7f1f280c6510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:56272/0 (socket says 172.21.15.195:56272) 2022-06-27T12:10:48.291 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.232+0000 7f1f2759e700 1 -- 172.21.15.195:0/3365119337 learned_addr learned my addr 172.21.15.195:0/3365119337 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:10:48.291 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.232+0000 7f1f2659c700 1 --2- 172.21.15.195:0/3365119337 >> v2:172.21.15.202:3300/0 conn(0x7f1f280c7480 0x7f1f280c4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:48.292 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.232+0000 7f1f25d9b700 1 -- 172.21.15.195:0/3365119337 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 317553072 0 0) 0x7f1f281573b0 con 0x7f1f280683c0 2022-06-27T12:10:48.292 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.232+0000 7f1f25d9b700 1 -- 172.21.15.195:0/3365119337 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f14003430 con 0x7f1f280683c0 2022-06-27T12:10:48.292 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.232+0000 7f1f25d9b700 1 -- 172.21.15.195:0/3365119337 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 769727941 0 0) 0x7f1f14003430 con 0x7f1f280683c0 2022-06-27T12:10:48.293 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.232+0000 7f1f2659c700 1 -- 172.21.15.195:0/3365119337 >> v1:172.21.15.202:6789/0 conn(0x7f1f28068f80 legacy=0x7f1f28156460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:48.293 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.232+0000 7f1f2659c700 1 -- 172.21.15.195:0/3365119337 >> v1:172.21.15.195:6789/0 conn(0x7f1f280683c0 legacy=0x7f1f280c6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:48.294 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.232+0000 7f1f2659c700 1 -- 172.21.15.195:0/3365119337 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f28156b70 con 0x7f1f280c7480 2022-06-27T12:10:48.294 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.233+0000 7f1f2659c700 1 --2- 172.21.15.195:0/3365119337 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1f280c7480 0x7f1f280c4b50 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c0097a0 tx=0x7f1f1c009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=754425a3c4f78df0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:48.295 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.233+0000 7f1f25d9b700 1 -- 172.21.15.195:0/3365119337 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f200017f0 con 0x7f1f280c7480 2022-06-27T12:10:48.295 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.233+0000 7f1f25d9b700 1 -- 172.21.15.195:0/3365119337 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f1c004030 con 0x7f1f280c7480 2022-06-27T12:10:48.296 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.233+0000 7f1f25d9b700 1 -- 172.21.15.195:0/3365119337 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f1c0043b0 con 0x7f1f280c7480 2022-06-27T12:10:48.296 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.233+0000 7f1f2d0e2700 1 -- 172.21.15.195:0/3365119337 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1f280c7480 msgr2=0x7f1f280c4b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:48.296 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.233+0000 7f1f2d0e2700 1 --2- 172.21.15.195:0/3365119337 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1f280c7480 0x7f1f280c4b50 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c0097a0 tx=0x7f1f1c009060 comp rx=0 tx=0).stop 2022-06-27T12:10:48.297 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.233+0000 7f1f2d0e2700 1 -- 172.21.15.195:0/3365119337 shutdown_connections 2022-06-27T12:10:48.297 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.233+0000 7f1f2d0e2700 1 --2- 172.21.15.195:0/3365119337 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1f280c7480 0x7f1f280c4b50 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:48.307 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.233+0000 7f1f2d0e2700 1 -- 172.21.15.195:0/3365119337 >> 172.21.15.195:0/3365119337 conn(0x7f1f280ccbd0 msgr2=0x7f1f280ccfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:10:48.307 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.233+0000 7f1f2d0e2700 1 -- 172.21.15.195:0/3365119337 shutdown_connections 2022-06-27T12:10:48.308 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.233+0000 7f1f2d0e2700 1 -- 172.21.15.195:0/3365119337 wait complete. 2022-06-27T12:10:48.308 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.235+0000 7f1f2d0e2700 1 Processor -- start 2022-06-27T12:10:48.309 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.235+0000 7f1f2d0e2700 1 -- start start 2022-06-27T12:10:48.309 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.235+0000 7f1f2d0e2700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1f280683c0 0x7f1f2816a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:48.310 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.235+0000 7f1f2d0e2700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1f28068f80 0x7f1f280c9940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:48.310 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.235+0000 7f1f2d0e2700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1f280c7480 0x7f1f28151810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:48.311 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.235+0000 7f1f2d0e2700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f281521c0 con 0x7f1f280683c0 2022-06-27T12:10:48.311 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.235+0000 7f1f2d0e2700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f28167520 con 0x7f1f28068f80 2022-06-27T12:10:48.311 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.235+0000 7f1f2d0e2700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f28166930 con 0x7f1f280c7480 2022-06-27T12:10:48.312 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.235+0000 7f1f26d9d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1f280683c0 0x7f1f2816a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:48.312 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.235+0000 7f1f26d9d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1f280683c0 0x7f1f2816a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50086/0 (socket says 172.21.15.195:50086) 2022-06-27T12:10:48.312 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.235+0000 7f1f26d9d700 1 -- 172.21.15.195:0/3273928514 learned_addr learned my addr 172.21.15.195:0/3273928514 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:10:48.313 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.236+0000 7f1f2659c700 1 --2- 172.21.15.195:0/3273928514 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1f28068f80 0x7f1f280c9940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:48.313 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.236+0000 7f1f2759e700 1 --2- 172.21.15.195:0/3273928514 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1f280c7480 0x7f1f28151810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:48.314 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.236+0000 7f1f26d9d700 1 -- 172.21.15.195:0/3273928514 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1f280c7480 msgr2=0x7f1f28151810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:48.314 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.236+0000 7f1f26d9d700 1 --2- 172.21.15.195:0/3273928514 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1f280c7480 0x7f1f28151810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:48.314 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.236+0000 7f1f26d9d700 1 -- 172.21.15.195:0/3273928514 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1f28068f80 msgr2=0x7f1f280c9940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:48.315 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.236+0000 7f1f26d9d700 1 --2- 172.21.15.195:0/3273928514 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1f28068f80 0x7f1f280c9940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:48.315 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.236+0000 7f1f26d9d700 1 -- 172.21.15.195:0/3273928514 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f1c0092e0 con 0x7f1f280683c0 2022-06-27T12:10:48.315 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.236+0000 7f1f2659c700 1 --2- 172.21.15.195:0/3273928514 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1f28068f80 0x7f1f280c9940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:10:48.315 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.236+0000 7f1f26d9d700 1 --2- 172.21.15.195:0/3273928514 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1f280683c0 0x7f1f2816a750 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f1f1800c490 tx=0x7f1f18007e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:48.316 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.236+0000 7f1f0effd700 1 -- 172.21.15.195:0/3273928514 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f180050d0 con 0x7f1f280683c0 2022-06-27T12:10:48.316 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.236+0000 7f1f0effd700 1 -- 172.21.15.195:0/3273928514 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f18005230 con 0x7f1f280683c0 2022-06-27T12:10:48.316 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.236+0000 7f1f2d0e2700 1 -- 172.21.15.195:0/3273928514 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f28166bf0 con 0x7f1f280683c0 2022-06-27T12:10:48.316 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.236+0000 7f1f0effd700 1 -- 172.21.15.195:0/3273928514 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f180117a0 con 0x7f1f280683c0 2022-06-27T12:10:48.317 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.236+0000 7f1f2d0e2700 1 -- 172.21.15.195:0/3273928514 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f281671b0 con 0x7f1f280683c0 2022-06-27T12:10:48.317 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.237+0000 7f1f2d0e2700 1 -- 172.21.15.195:0/3273928514 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f28167370 con 0x7f1f280683c0 2022-06-27T12:10:48.317 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.238+0000 7f1f0effd700 1 -- 172.21.15.195:0/3273928514 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f1f18019030 con 0x7f1f280683c0 2022-06-27T12:10:48.318 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.238+0000 7f1f0effd700 1 --2- 172.21.15.195:0/3273928514 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1f08063d00 0x7f1f080661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:48.318 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.238+0000 7f1f0effd700 1 -- 172.21.15.195:0/3273928514 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7320+0+0 (secure 0 0 0) 0x7f1f1807fe30 con 0x7f1f280683c0 2022-06-27T12:10:48.318 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.238+0000 7f1f2659c700 1 --2- 172.21.15.195:0/3273928514 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1f08063d00 0x7f1f080661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:48.319 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.238+0000 7f1f2659c700 1 --2- 172.21.15.195:0/3273928514 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1f08063d00 0x7f1f080661b0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c00a010 tx=0x7f1f1c015000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:48.319 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.242+0000 7f1f0effd700 1 -- 172.21.15.195:0/3273928514 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f1f18049c90 con 0x7f1f280683c0 2022-06-27T12:10:48.479 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.465+0000 7f1f2d0e2700 1 -- 172.21.15.195:0/3273928514 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f280722c0 con 0x7f1f280683c0 2022-06-27T12:10:48.479 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.465+0000 7f1f0effd700 1 -- 172.21.15.195:0/3273928514 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+5371 (secure 0 0 0) 0x7f1f18017020 con 0x7f1f280683c0 2022-06-27T12:10:48.480 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:10:48.481 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:10:43.197876+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5933},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5933":{"gid":5933,"name":"d","rank":0,"incarnation":85,"state":"up:reconnect","state_seq":11,"addr":"172.21.15.202:6835/1342896965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":1342896965},{"type":"v1","addr":"172.21.15.202:6835","nonce":1342896965}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:10:48.502 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.467+0000 7f1f2d0e2700 1 -- 172.21.15.195:0/3273928514 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1f08063d00 msgr2=0x7f1f080661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:48.503 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.467+0000 7f1f2d0e2700 1 --2- 172.21.15.195:0/3273928514 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1f08063d00 0x7f1f080661b0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c00a010 tx=0x7f1f1c015000 comp rx=0 tx=0).stop 2022-06-27T12:10:48.503 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.467+0000 7f1f2d0e2700 1 -- 172.21.15.195:0/3273928514 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1f280683c0 msgr2=0x7f1f2816a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:48.503 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.467+0000 7f1f2d0e2700 1 --2- 172.21.15.195:0/3273928514 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1f280683c0 0x7f1f2816a750 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f1f1800c490 tx=0x7f1f18007e40 comp rx=0 tx=0).stop 2022-06-27T12:10:48.512 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.467+0000 7f1f2d0e2700 1 -- 172.21.15.195:0/3273928514 shutdown_connections 2022-06-27T12:10:48.512 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.467+0000 7f1f2d0e2700 1 --2- 172.21.15.195:0/3273928514 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1f08063d00 0x7f1f080661b0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:48.512 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.467+0000 7f1f2d0e2700 1 --2- 172.21.15.195:0/3273928514 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1f280683c0 0x7f1f2816a750 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:48.513 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.467+0000 7f1f2d0e2700 1 --2- 172.21.15.195:0/3273928514 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1f280c7480 0x7f1f28151810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:48.513 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.467+0000 7f1f2d0e2700 1 --2- 172.21.15.195:0/3273928514 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1f28068f80 0x7f1f280c9940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:48.514 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.467+0000 7f1f2d0e2700 1 -- 172.21.15.195:0/3273928514 >> 172.21.15.195:0/3273928514 conn(0x7f1f280ccbd0 msgr2=0x7f1f2815c360 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:10:48.514 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.467+0000 7f1f2d0e2700 1 -- 172.21.15.195:0/3273928514 shutdown_connections 2022-06-27T12:10:48.515 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:48.467+0000 7f1f2d0e2700 1 -- 172.21.15.195:0/3273928514 wait complete. 2022-06-27T12:10:48.517 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 86 2022-06-27T12:10:48.528 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:10:50.529 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:10:50.665 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.662+0000 7fe41c36c700 1 Processor -- start 2022-06-27T12:10:50.665 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.662+0000 7fe41c36c700 1 -- start start 2022-06-27T12:10:50.665 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.663+0000 7fe41c36c700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fe414151930 0x7fe4140c2120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:50.666 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.663+0000 7fe41c36c700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe414154da0 con 0x7fe414151560 2022-06-27T12:10:50.666 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.663+0000 7fe41c36c700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4140c12a0 con 0x7fe414154900 2022-06-27T12:10:50.666 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.663+0000 7fe41c36c700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4140c1580 con 0x7fe414151930 2022-06-27T12:10:50.666 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.663+0000 7fe41a108700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fe414151560 0x7fe4141541f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:56282/0 (socket says 172.21.15.195:56282) 2022-06-27T12:10:50.666 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.664+0000 7fe41a108700 1 -- 172.21.15.195:0/1312649318 learned_addr learned my addr 172.21.15.195:0/1312649318 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:10:50.674 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.664+0000 7fe419106700 1 -- 172.21.15.195:0/1312649318 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4242676315 0 0) 0x7fe414154da0 con 0x7fe414151560 2022-06-27T12:10:50.675 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.664+0000 7fe41a909700 1 --2- 172.21.15.195:0/1312649318 >> v2:172.21.15.202:3300/0 conn(0x7fe414151930 0x7fe4140c2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:50.675 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.664+0000 7fe419106700 1 -- 172.21.15.195:0/1312649318 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3fc003430 con 0x7fe414151560 2022-06-27T12:10:50.676 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.664+0000 7fe419106700 1 -- 172.21.15.195:0/1312649318 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 156412042 0 0) 0x7fe3fc003430 con 0x7fe414151560 2022-06-27T12:10:50.676 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.664+0000 7fe419106700 1 -- 172.21.15.195:0/1312649318 >> v2:172.21.15.202:3300/0 conn(0x7fe414151930 msgr2=0x7fe4140c2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:10:50.676 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.664+0000 7fe419106700 1 --2- 172.21.15.195:0/1312649318 >> v2:172.21.15.202:3300/0 conn(0x7fe414151930 0x7fe4140c2120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:50.676 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.664+0000 7fe419106700 1 -- 172.21.15.195:0/1312649318 >> v1:172.21.15.202:6789/0 conn(0x7fe414154900 legacy=0x7fe41415cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:50.677 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.664+0000 7fe419106700 1 -- 172.21.15.195:0/1312649318 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4140c29f0 con 0x7fe414151560 2022-06-27T12:10:50.677 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.664+0000 7fe419106700 1 -- 172.21.15.195:0/1312649318 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7fe404001470 con 0x7fe414151560 2022-06-27T12:10:50.677 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.664+0000 7fe419106700 1 -- 172.21.15.195:0/1312649318 >> v1:172.21.15.195:6789/0 conn(0x7fe414151560 legacy=0x7fe4141541f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:50.677 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.664+0000 7fe419106700 1 --2- 172.21.15.195:0/1312649318 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe3fc003920 0x7fe3fc003d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:50.677 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.664+0000 7fe419106700 1 --2- 172.21.15.195:0/1312649318 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe3fc0046d0 0x7fe3fc004b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:50.678 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.664+0000 7fe419106700 1 --2- 172.21.15.195:0/1312649318 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe3fc005080 0x7fe3fc00b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:50.678 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.664+0000 7fe419106700 1 -- 172.21.15.195:0/1312649318 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4140c1580 con 0x7fe3fc003920 2022-06-27T12:10:50.678 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.664+0000 7fe419106700 1 -- 172.21.15.195:0/1312649318 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3fc00bf90 con 0x7fe3fc005080 2022-06-27T12:10:50.678 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.664+0000 7fe419106700 1 -- 172.21.15.195:0/1312649318 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3fc00c150 con 0x7fe3fc0046d0 2022-06-27T12:10:50.679 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.664+0000 7fe41a108700 1 --2- 172.21.15.195:0/1312649318 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe3fc003920 0x7fe3fc003d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:50.679 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.664+0000 7fe41a909700 1 --2- 172.21.15.195:0/1312649318 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe3fc005080 0x7fe3fc00b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:50.679 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.664+0000 7fe419907700 1 --2- 172.21.15.195:0/1312649318 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe3fc0046d0 0x7fe3fc004b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:50.679 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.665+0000 7fe41a108700 1 -- 172.21.15.195:0/1312649318 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe3fc0046d0 msgr2=0x7fe3fc004b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:10:50.679 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.665+0000 7fe41a108700 1 --2- 172.21.15.195:0/1312649318 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe3fc0046d0 0x7fe3fc004b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:50.680 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.665+0000 7fe41a108700 1 -- 172.21.15.195:0/1312649318 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe3fc005080 msgr2=0x7fe3fc00b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:10:50.680 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.665+0000 7fe41a108700 1 --2- 172.21.15.195:0/1312649318 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe3fc005080 0x7fe3fc00b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:50.680 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.665+0000 7fe41a108700 1 -- 172.21.15.195:0/1312649318 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3fc003430 con 0x7fe3fc003920 2022-06-27T12:10:50.680 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.665+0000 7fe41a909700 1 -- 172.21.15.195:0/1312649318 reap_dead start 2022-06-27T12:10:50.681 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.665+0000 7fe419907700 1 --2- 172.21.15.195:0/1312649318 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe3fc0046d0 0x7fe3fc004b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:10:50.683 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.665+0000 7fe41a108700 1 --2- 172.21.15.195:0/1312649318 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe3fc003920 0x7fe3fc003d30 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fe40400c210 tx=0x7fe404001e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ad281d1594a1500 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:50.683 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.665+0000 7fe419106700 1 -- 172.21.15.195:0/1312649318 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe40400ca60 con 0x7fe3fc003920 2022-06-27T12:10:50.684 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.665+0000 7fe419106700 1 -- 172.21.15.195:0/1312649318 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe40400cbe0 con 0x7fe3fc003920 2022-06-27T12:10:50.684 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.665+0000 7fe419106700 1 -- 172.21.15.195:0/1312649318 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4040070d0 con 0x7fe3fc003920 2022-06-27T12:10:50.685 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.665+0000 7fe41c36c700 1 -- 172.21.15.195:0/1312649318 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe3fc003920 msgr2=0x7fe3fc003d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:50.685 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.665+0000 7fe41c36c700 1 --2- 172.21.15.195:0/1312649318 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe3fc003920 0x7fe3fc003d30 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fe40400c210 tx=0x7fe404001e30 comp rx=0 tx=0).stop 2022-06-27T12:10:50.685 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.665+0000 7fe41c36c700 1 -- 172.21.15.195:0/1312649318 shutdown_connections 2022-06-27T12:10:50.686 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.665+0000 7fe41c36c700 1 --2- 172.21.15.195:0/1312649318 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe3fc003920 0x7fe3fc003d30 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:50.686 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.665+0000 7fe41c36c700 1 -- 172.21.15.195:0/1312649318 >> 172.21.15.195:0/1312649318 conn(0x7fe4140c6350 msgr2=0x7fe4140c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:10:50.687 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.665+0000 7fe41c36c700 1 -- 172.21.15.195:0/1312649318 shutdown_connections 2022-06-27T12:10:50.687 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.666+0000 7fe41c36c700 1 -- 172.21.15.195:0/1312649318 wait complete. 2022-06-27T12:10:50.688 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.667+0000 7fe41c36c700 1 Processor -- start 2022-06-27T12:10:50.688 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.667+0000 7fe41c36c700 1 -- start start 2022-06-27T12:10:50.689 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.668+0000 7fe41c36c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe3fc003920 0x7fe414155380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:50.715 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.668+0000 7fe41c36c700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe41406aa00 0x7fe4140c8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:50.716 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.668+0000 7fe41c36c700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe41414e090 0x7fe41414e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:50.717 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.668+0000 7fe41c36c700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4140682c0 con 0x7fe3fc003920 2022-06-27T12:10:50.718 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.668+0000 7fe41c36c700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe414165070 con 0x7fe41406aa00 2022-06-27T12:10:50.718 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.668+0000 7fe41c36c700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe41414ea40 con 0x7fe41414e090 2022-06-27T12:10:50.718 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.668+0000 7fe41a108700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe3fc003920 0x7fe414155380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:50.719 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.668+0000 7fe41a108700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe3fc003920 0x7fe414155380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50106/0 (socket says 172.21.15.195:50106) 2022-06-27T12:10:50.719 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.668+0000 7fe41a108700 1 -- 172.21.15.195:0/2219663668 learned_addr learned my addr 172.21.15.195:0/2219663668 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:10:50.720 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.668+0000 7fe419907700 1 --2- 172.21.15.195:0/2219663668 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe41406aa00 0x7fe4140c8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:50.720 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.668+0000 7fe41a909700 1 --2- 172.21.15.195:0/2219663668 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe41414e090 0x7fe41414e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:50.720 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.668+0000 7fe41a108700 1 -- 172.21.15.195:0/2219663668 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe41414e090 msgr2=0x7fe41414e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:50.721 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.668+0000 7fe41a108700 1 --2- 172.21.15.195:0/2219663668 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe41414e090 0x7fe41414e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:50.721 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.668+0000 7fe41a108700 1 -- 172.21.15.195:0/2219663668 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe41406aa00 msgr2=0x7fe4140c8bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:50.722 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.668+0000 7fe41a108700 1 --2- 172.21.15.195:0/2219663668 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe41406aa00 0x7fe4140c8bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:50.722 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.668+0000 7fe41a108700 1 -- 172.21.15.195:0/2219663668 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe414067fe0 con 0x7fe3fc003920 2022-06-27T12:10:50.723 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.669+0000 7fe41a108700 1 --2- 172.21.15.195:0/2219663668 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe3fc003920 0x7fe414155380 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fe404001940 tx=0x7fe404015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:50.723 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.669+0000 7fe4027fc700 1 -- 172.21.15.195:0/2219663668 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe40400cd50 con 0x7fe3fc003920 2022-06-27T12:10:50.724 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.669+0000 7fe4027fc700 1 -- 172.21.15.195:0/2219663668 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe40400ced0 con 0x7fe3fc003920 2022-06-27T12:10:50.724 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.669+0000 7fe4027fc700 1 -- 172.21.15.195:0/2219663668 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe404004940 con 0x7fe3fc003920 2022-06-27T12:10:50.725 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.669+0000 7fe41c36c700 1 -- 172.21.15.195:0/2219663668 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe414164d90 con 0x7fe3fc003920 2022-06-27T12:10:50.725 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.669+0000 7fe41c36c700 1 -- 172.21.15.195:0/2219663668 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe414154640 con 0x7fe3fc003920 2022-06-27T12:10:50.725 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.670+0000 7fe41c36c700 1 -- 172.21.15.195:0/2219663668 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe414156150 con 0x7fe3fc003920 2022-06-27T12:10:50.726 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.670+0000 7fe4027fc700 1 -- 172.21.15.195:0/2219663668 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fe404010070 con 0x7fe3fc003920 2022-06-27T12:10:50.728 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.670+0000 7fe4027fc700 1 --2- 172.21.15.195:0/2219663668 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe3f4063d00 0x7fe3f40661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:50.728 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.670+0000 7fe4027fc700 1 -- 172.21.15.195:0/2219663668 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7320+0+0 (secure 0 0 0) 0x7fe404081360 con 0x7fe3fc003920 2022-06-27T12:10:50.729 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.670+0000 7fe419907700 1 --2- 172.21.15.195:0/2219663668 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe3f4063d00 0x7fe3f40661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:50.729 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.671+0000 7fe419907700 1 --2- 172.21.15.195:0/2219663668 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe3f4063d00 0x7fe3f40661b0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fe410000c00 tx=0x7fe410009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:50.730 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.675+0000 7fe4027fc700 1 -- 172.21.15.195:0/2219663668 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fe40404fa60 con 0x7fe3fc003920 2022-06-27T12:10:50.905 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.903+0000 7fe41c36c700 1 -- 172.21.15.195:0/2219663668 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe4140722c0 con 0x7fe3fc003920 2022-06-27T12:10:50.906 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.903+0000 7fe4027fc700 1 -- 172.21.15.195:0/2219663668 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+5368 (secure 0 0 0) 0x7fe404053080 con 0x7fe3fc003920 2022-06-27T12:10:50.906 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:10:50.907 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:10:49.214075+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5933},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5933":{"gid":5933,"name":"d","rank":0,"incarnation":85,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.202:6835/1342896965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":1342896965},{"type":"v1","addr":"172.21.15.202:6835","nonce":1342896965}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:10:50.909 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.905+0000 7fe41c36c700 1 -- 172.21.15.195:0/2219663668 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe3f4063d00 msgr2=0x7fe3f40661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:50.910 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.905+0000 7fe41c36c700 1 --2- 172.21.15.195:0/2219663668 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe3f4063d00 0x7fe3f40661b0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fe410000c00 tx=0x7fe410009040 comp rx=0 tx=0).stop 2022-06-27T12:10:50.911 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.905+0000 7fe41c36c700 1 -- 172.21.15.195:0/2219663668 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe3fc003920 msgr2=0x7fe414155380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:50.911 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.905+0000 7fe41c36c700 1 --2- 172.21.15.195:0/2219663668 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe3fc003920 0x7fe414155380 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fe404001940 tx=0x7fe404015000 comp rx=0 tx=0).stop 2022-06-27T12:10:50.912 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.905+0000 7fe41c36c700 1 -- 172.21.15.195:0/2219663668 shutdown_connections 2022-06-27T12:10:50.912 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.905+0000 7fe41c36c700 1 --2- 172.21.15.195:0/2219663668 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe3f4063d00 0x7fe3f40661b0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:50.913 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.905+0000 7fe41c36c700 1 --2- 172.21.15.195:0/2219663668 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe3fc003920 0x7fe414155380 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:50.913 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.905+0000 7fe41c36c700 1 --2- 172.21.15.195:0/2219663668 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe41414e090 0x7fe41414e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:50.913 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.905+0000 7fe41c36c700 1 --2- 172.21.15.195:0/2219663668 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe41406aa00 0x7fe4140c8bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:50.914 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.905+0000 7fe41c36c700 1 -- 172.21.15.195:0/2219663668 >> 172.21.15.195:0/2219663668 conn(0x7fe4140c6350 msgr2=0x7fe41414f5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:10:50.914 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.905+0000 7fe41c36c700 1 -- 172.21.15.195:0/2219663668 shutdown_connections 2022-06-27T12:10:50.915 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:50.905+0000 7fe41c36c700 1 -- 172.21.15.195:0/2219663668 wait complete. 2022-06-27T12:10:50.915 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 87 2022-06-27T12:10:50.931 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:10:52.933 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:10:53.095 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.081+0000 7f2f12f8a700 1 Processor -- start 2022-06-27T12:10:53.096 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.081+0000 7f2f12f8a700 1 -- start start 2022-06-27T12:10:53.096 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.082+0000 7f2f12f8a700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f2f0c06f640 0x7f2f0c156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:53.097 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.082+0000 7f2f12f8a700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2f0c0c4160 0x7f2f0c0c1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:53.097 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.082+0000 7f2f12f8a700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f0c0c1a20 con 0x7f2f0c0c39e0 2022-06-27T12:10:53.101 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.082+0000 7f2f12f8a700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f0c0c1130 con 0x7f2f0c06f640 2022-06-27T12:10:53.101 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.082+0000 7f2f12f8a700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0c06ea90 con 0x7f2f0c0c4160 2022-06-27T12:10:53.101 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.082+0000 7f2f10d26700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f2f0c06f640 0x7f2f0c156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:53.110 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.082+0000 7f2f10d26700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f2f0c06f640 0x7f2f0c156ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50116/0 (socket says 172.21.15.195:50116) 2022-06-27T12:10:53.111 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.082+0000 7f2f10d26700 1 -- 172.21.15.195:0/4234751878 learned_addr learned my addr 172.21.15.195:0/4234751878 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:10:53.111 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.082+0000 7f2f0bfff700 1 --2- 172.21.15.195:0/4234751878 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2f0c0c4160 0x7f2f0c0c1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:53.112 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.083+0000 7f2f10d26700 1 -- 172.21.15.195:0/4234751878 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2f0c0c4160 msgr2=0x7f2f0c0c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:10:53.112 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.083+0000 7f2f10d26700 1 --2- 172.21.15.195:0/4234751878 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2f0c0c4160 0x7f2f0c0c1eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:53.112 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.083+0000 7f2f10d26700 1 -- 172.21.15.195:0/4234751878 >> v1:172.21.15.202:6789/0 conn(0x7f2f0c0c39e0 legacy=0x7f2f0c1535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:53.113 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.083+0000 7f2f10d26700 1 -- 172.21.15.195:0/4234751878 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f0c06ed70 con 0x7f2f0c06f640 2022-06-27T12:10:53.113 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.083+0000 7f2f0bfff700 1 --2- 172.21.15.195:0/4234751878 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2f0c0c4160 0x7f2f0c0c1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:10:53.114 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.083+0000 7f2f10d26700 1 --2- 172.21.15.195:0/4234751878 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2f0c06f640 0x7f2f0c156ed0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f2f00009860 tx=0x7f2f00009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e8dea61f3f85482 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:53.114 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.083+0000 7f2f0b7fe700 1 -- 172.21.15.195:0/4234751878 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f0000b710 con 0x7f2f0c06f640 2022-06-27T12:10:53.115 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.083+0000 7f2f0b7fe700 1 -- 172.21.15.195:0/4234751878 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f0000b870 con 0x7f2f0c06f640 2022-06-27T12:10:53.115 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.083+0000 7f2f0b7fe700 1 -- 172.21.15.195:0/4234751878 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f0000bc30 con 0x7f2f0c06f640 2022-06-27T12:10:53.116 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.083+0000 7f2f12f8a700 1 -- 172.21.15.195:0/4234751878 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2f0c06f640 msgr2=0x7f2f0c156ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:53.118 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.083+0000 7f2f12f8a700 1 --2- 172.21.15.195:0/4234751878 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2f0c06f640 0x7f2f0c156ed0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f2f00009860 tx=0x7f2f00009120 comp rx=0 tx=0).stop 2022-06-27T12:10:53.119 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.083+0000 7f2f12f8a700 1 -- 172.21.15.195:0/4234751878 shutdown_connections 2022-06-27T12:10:53.119 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.083+0000 7f2f12f8a700 1 --2- 172.21.15.195:0/4234751878 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2f0c06f640 0x7f2f0c156ed0 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:53.120 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.083+0000 7f2f12f8a700 1 --2- 172.21.15.195:0/4234751878 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2f0c0c4160 0x7f2f0c0c1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:53.121 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.083+0000 7f2f12f8a700 1 -- 172.21.15.195:0/4234751878 >> 172.21.15.195:0/4234751878 conn(0x7f2f0c0c6350 msgr2=0x7f2f0c0c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:10:53.121 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.083+0000 7f2f12f8a700 1 -- 172.21.15.195:0/4234751878 shutdown_connections 2022-06-27T12:10:53.121 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.083+0000 7f2f12f8a700 1 -- 172.21.15.195:0/4234751878 wait complete. 2022-06-27T12:10:53.122 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.084+0000 7f2f12f8a700 1 Processor -- start 2022-06-27T12:10:53.122 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.085+0000 7f2f12f8a700 1 -- start start 2022-06-27T12:10:53.123 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.085+0000 7f2f12f8a700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2f0c06f640 0x7f2f0c14f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:53.123 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.085+0000 7f2f12f8a700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2f0c0c39e0 0x7f2f0c0bc500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:53.124 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.085+0000 7f2f12f8a700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2f0c0c4160 0x7f2f0c0bca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:53.124 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.085+0000 7f2f12f8a700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0c069270 con 0x7f2f0c0c4160 2022-06-27T12:10:53.125 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.085+0000 7f2f12f8a700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0c14fee0 con 0x7f2f0c06f640 2022-06-27T12:10:53.125 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.085+0000 7f2f12f8a700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0c16a7f0 con 0x7f2f0c0c39e0 2022-06-27T12:10:53.125 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.086+0000 7f2f11527700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2f0c0c4160 0x7f2f0c0bca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:53.126 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.086+0000 7f2f11527700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2f0c0c4160 0x7f2f0c0bca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50126/0 (socket says 172.21.15.195:50126) 2022-06-27T12:10:53.126 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.086+0000 7f2f11527700 1 -- 172.21.15.195:0/1890974249 learned_addr learned my addr 172.21.15.195:0/1890974249 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:10:53.126 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.086+0000 7f2f0bfff700 1 --2- 172.21.15.195:0/1890974249 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2f0c0c39e0 0x7f2f0c0bc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:53.127 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.086+0000 7f2f10d26700 1 --2- 172.21.15.195:0/1890974249 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2f0c06f640 0x7f2f0c14f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:53.128 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.086+0000 7f2f11527700 1 -- 172.21.15.195:0/1890974249 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2f0c0c39e0 msgr2=0x7f2f0c0bc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:53.129 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.086+0000 7f2f11527700 1 --2- 172.21.15.195:0/1890974249 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2f0c0c39e0 0x7f2f0c0bc500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:53.129 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.086+0000 7f2f11527700 1 -- 172.21.15.195:0/1890974249 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2f0c06f640 msgr2=0x7f2f0c14f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:53.129 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.086+0000 7f2f11527700 1 --2- 172.21.15.195:0/1890974249 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2f0c06f640 0x7f2f0c14f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:53.129 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.086+0000 7f2f11527700 1 -- 172.21.15.195:0/1890974249 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f000093a0 con 0x7f2f0c0c4160 2022-06-27T12:10:53.130 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.086+0000 7f2f10d26700 1 --2- 172.21.15.195:0/1890974249 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2f0c06f640 0x7f2f0c14f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T12:10:53.131 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.086+0000 7f2f0bfff700 1 --2- 172.21.15.195:0/1890974249 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2f0c0c39e0 0x7f2f0c0bc500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T12:10:53.131 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.086+0000 7f2f11527700 1 --2- 172.21.15.195:0/1890974249 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2f0c0c4160 0x7f2f0c0bca40 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f2f0400c7f0 tx=0x7f2f040015e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:53.131 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.086+0000 7f2f08ff9700 1 -- 172.21.15.195:0/1890974249 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f04003880 con 0x7f2f0c0c4160 2022-06-27T12:10:53.131 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.086+0000 7f2f08ff9700 1 -- 172.21.15.195:0/1890974249 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f040039e0 con 0x7f2f0c0c4160 2022-06-27T12:10:53.132 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.086+0000 7f2f08ff9700 1 -- 172.21.15.195:0/1890974249 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f04003d60 con 0x7f2f0c0c4160 2022-06-27T12:10:53.132 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.086+0000 7f2f12f8a700 1 -- 172.21.15.195:0/1890974249 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f0c169960 con 0x7f2f0c0c4160 2022-06-27T12:10:53.132 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.086+0000 7f2f12f8a700 1 -- 172.21.15.195:0/1890974249 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f0c169ec0 con 0x7f2f0c0c4160 2022-06-27T12:10:53.132 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.087+0000 7f2f12f8a700 1 -- 172.21.15.195:0/1890974249 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f0c166400 con 0x7f2f0c0c4160 2022-06-27T12:10:53.133 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.088+0000 7f2f08ff9700 1 -- 172.21.15.195:0/1890974249 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f2f04010050 con 0x7f2f0c0c4160 2022-06-27T12:10:53.133 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.088+0000 7f2f08ff9700 1 --2- 172.21.15.195:0/1890974249 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2ef4063d00 0x7f2ef40661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:53.133 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.088+0000 7f2f08ff9700 1 -- 172.21.15.195:0/1890974249 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7320+0+0 (secure 0 0 0) 0x7f2f04080830 con 0x7f2f0c0c4160 2022-06-27T12:10:53.133 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.088+0000 7f2f10d26700 1 --2- 172.21.15.195:0/1890974249 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2ef4063d00 0x7f2ef40661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:53.134 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.088+0000 7f2f10d26700 1 --2- 172.21.15.195:0/1890974249 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2ef4063d00 0x7f2ef40661b0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f2f00009320 tx=0x7f2f00017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:53.134 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.092+0000 7f2f08ff9700 1 -- 172.21.15.195:0/1890974249 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f2f0c166400 con 0x7f2f0c0c4160 2022-06-27T12:10:53.314 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.314+0000 7f2f12f8a700 1 -- 172.21.15.195:0/1890974249 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2f0c166400 con 0x7f2f0c0c4160 2022-06-27T12:10:53.324 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.314+0000 7f2f08ff9700 1 -- 172.21.15.195:0/1890974249 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+5368 (secure 0 0 0) 0x7f2f0c166400 con 0x7f2f0c0c4160 2022-06-27T12:10:53.326 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:10:53.326 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:10:49.214075+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5933},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5933":{"gid":5933,"name":"d","rank":0,"incarnation":85,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.202:6835/1342896965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":1342896965},{"type":"v1","addr":"172.21.15.202:6835","nonce":1342896965}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:10:53.330 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.316+0000 7f2f12f8a700 1 -- 172.21.15.195:0/1890974249 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2ef4063d00 msgr2=0x7f2ef40661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:53.330 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.316+0000 7f2f12f8a700 1 --2- 172.21.15.195:0/1890974249 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2ef4063d00 0x7f2ef40661b0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f2f00009320 tx=0x7f2f00017000 comp rx=0 tx=0).stop 2022-06-27T12:10:53.332 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.316+0000 7f2f12f8a700 1 -- 172.21.15.195:0/1890974249 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2f0c0c4160 msgr2=0x7f2f0c0bca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:53.334 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.316+0000 7f2f12f8a700 1 --2- 172.21.15.195:0/1890974249 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2f0c0c4160 0x7f2f0c0bca40 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f2f0400c7f0 tx=0x7f2f040015e0 comp rx=0 tx=0).stop 2022-06-27T12:10:53.335 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.316+0000 7f2f12f8a700 1 -- 172.21.15.195:0/1890974249 shutdown_connections 2022-06-27T12:10:53.335 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.316+0000 7f2f12f8a700 1 --2- 172.21.15.195:0/1890974249 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2ef4063d00 0x7f2ef40661b0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:53.336 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.316+0000 7f2f12f8a700 1 --2- 172.21.15.195:0/1890974249 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2f0c0c4160 0x7f2f0c0bca40 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:53.336 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.316+0000 7f2f12f8a700 1 --2- 172.21.15.195:0/1890974249 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2f0c06f640 0x7f2f0c14f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:53.337 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.316+0000 7f2f12f8a700 1 --2- 172.21.15.195:0/1890974249 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2f0c0c39e0 0x7f2f0c0bc500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:53.337 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.316+0000 7f2f12f8a700 1 -- 172.21.15.195:0/1890974249 >> 172.21.15.195:0/1890974249 conn(0x7f2f0c0c6350 msgr2=0x7f2f0c0c8570 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:10:53.337 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.316+0000 7f2f12f8a700 1 -- 172.21.15.195:0/1890974249 shutdown_connections 2022-06-27T12:10:53.338 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:53.316+0000 7f2f12f8a700 1 -- 172.21.15.195:0/1890974249 wait complete. 2022-06-27T12:10:53.338 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 87 2022-06-27T12:10:53.339 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:10:55.344 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:10:55.495 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.479+0000 7fec18285700 1 Processor -- start 2022-06-27T12:10:55.496 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.479+0000 7fec18285700 1 -- start start 2022-06-27T12:10:55.496 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.480+0000 7fec18285700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fec1006fef0 0x7fec101570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:55.497 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.480+0000 7fec18285700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fec100c34a0 0x7fec100c2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:55.497 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.480+0000 7fec18285700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec1006fba0 con 0x7fec1006e850 2022-06-27T12:10:55.498 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.480+0000 7fec18285700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec101575e0 con 0x7fec100c34a0 2022-06-27T12:10:55.498 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.480+0000 7fec18285700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec1006f5e0 con 0x7fec1006fef0 2022-06-27T12:10:55.518 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.480+0000 7fec16822700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fec100c34a0 0x7fec100c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:55.520 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.480+0000 7fec16822700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fec100c34a0 0x7fec100c2920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50134/0 (socket says 172.21.15.195:50134) 2022-06-27T12:10:55.520 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.480+0000 7fec16822700 1 -- 172.21.15.195:0/1942282250 learned_addr learned my addr 172.21.15.195:0/1942282250 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:10:55.521 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.480+0000 7fec15820700 1 --2- 172.21.15.195:0/1942282250 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fec1006fef0 0x7fec101570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:55.521 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.481+0000 7fec16822700 1 -- 172.21.15.195:0/1942282250 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fec1006fef0 msgr2=0x7fec101570a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:10:55.522 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.481+0000 7fec16822700 1 --2- 172.21.15.195:0/1942282250 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fec1006fef0 0x7fec101570a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:55.527 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.481+0000 7fec16822700 1 -- 172.21.15.195:0/1942282250 >> v1:172.21.15.202:6789/0 conn(0x7fec1006e850 legacy=0x7fec10069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:55.527 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.481+0000 7fec16822700 1 -- 172.21.15.195:0/1942282250 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec1006f8c0 con 0x7fec100c34a0 2022-06-27T12:10:55.528 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.481+0000 7fec16822700 1 --2- 172.21.15.195:0/1942282250 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fec100c34a0 0x7fec100c2920 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fec08009f20 tx=0x7fec08009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56facdb67bc53a12 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:55.528 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.481+0000 7fec1501f700 1 -- 172.21.15.195:0/1942282250 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec0800e070 con 0x7fec100c34a0 2022-06-27T12:10:55.529 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.481+0000 7fec1501f700 1 -- 172.21.15.195:0/1942282250 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec0800bb10 con 0x7fec100c34a0 2022-06-27T12:10:55.529 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.481+0000 7fec1501f700 1 -- 172.21.15.195:0/1942282250 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec080092e0 con 0x7fec100c34a0 2022-06-27T12:10:55.532 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.481+0000 7fec18285700 1 -- 172.21.15.195:0/1942282250 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fec100c34a0 msgr2=0x7fec100c2920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:55.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.481+0000 7fec18285700 1 --2- 172.21.15.195:0/1942282250 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fec100c34a0 0x7fec100c2920 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fec08009f20 tx=0x7fec08009840 comp rx=0 tx=0).stop 2022-06-27T12:10:55.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.481+0000 7fec18285700 1 -- 172.21.15.195:0/1942282250 shutdown_connections 2022-06-27T12:10:55.534 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.481+0000 7fec18285700 1 --2- 172.21.15.195:0/1942282250 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fec100c34a0 0x7fec100c2920 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:55.534 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.481+0000 7fec18285700 1 --2- 172.21.15.195:0/1942282250 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fec1006fef0 0x7fec101570a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:55.536 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.481+0000 7fec18285700 1 -- 172.21.15.195:0/1942282250 >> 172.21.15.195:0/1942282250 conn(0x7fec100c6350 msgr2=0x7fec100c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:10:55.536 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.482+0000 7fec18285700 1 -- 172.21.15.195:0/1942282250 shutdown_connections 2022-06-27T12:10:55.537 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.482+0000 7fec18285700 1 -- 172.21.15.195:0/1942282250 wait complete. 2022-06-27T12:10:55.537 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.483+0000 7fec18285700 1 Processor -- start 2022-06-27T12:10:55.538 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.483+0000 7fec18285700 1 -- start start 2022-06-27T12:10:55.538 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.483+0000 7fec18285700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fec1006e850 0x7fec10166c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:55.539 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.483+0000 7fec18285700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fec100c34a0 0x7fec1014b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:55.539 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.483+0000 7fec18285700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fec10068590 0x7fec10155700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:55.539 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.483+0000 7fec18285700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec10152a20 con 0x7fec1006e850 2022-06-27T12:10:55.540 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.483+0000 7fec18285700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec10153b60 con 0x7fec10068590 2022-06-27T12:10:55.541 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.483+0000 7fec18285700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec10152c70 con 0x7fec100c34a0 2022-06-27T12:10:55.541 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.483+0000 7fec16021700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fec1006e850 0x7fec10166c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:55.542 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.483+0000 7fec16021700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fec1006e850 0x7fec10166c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50136/0 (socket says 172.21.15.195:50136) 2022-06-27T12:10:55.542 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.483+0000 7fec16021700 1 -- 172.21.15.195:0/2181213793 learned_addr learned my addr 172.21.15.195:0/2181213793 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:10:55.543 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.484+0000 7fec15820700 1 --2- 172.21.15.195:0/2181213793 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fec100c34a0 0x7fec1014b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:55.543 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.484+0000 7fec16822700 1 --2- 172.21.15.195:0/2181213793 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fec10068590 0x7fec10155700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:55.544 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.484+0000 7fec16021700 1 -- 172.21.15.195:0/2181213793 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fec100c34a0 msgr2=0x7fec1014b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:55.544 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.484+0000 7fec16021700 1 --2- 172.21.15.195:0/2181213793 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fec100c34a0 0x7fec1014b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:55.545 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.484+0000 7fec16021700 1 -- 172.21.15.195:0/2181213793 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fec10068590 msgr2=0x7fec10155700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:55.545 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.484+0000 7fec16021700 1 --2- 172.21.15.195:0/2181213793 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fec10068590 0x7fec10155700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:55.546 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.484+0000 7fec16021700 1 -- 172.21.15.195:0/2181213793 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec08009ae0 con 0x7fec1006e850 2022-06-27T12:10:55.546 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.484+0000 7fec15820700 1 --2- 172.21.15.195:0/2181213793 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fec100c34a0 0x7fec1014b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T12:10:55.547 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.484+0000 7fec16822700 1 --2- 172.21.15.195:0/2181213793 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fec10068590 0x7fec10155700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T12:10:55.548 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.484+0000 7fec16021700 1 --2- 172.21.15.195:0/2181213793 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fec1006e850 0x7fec10166c40 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fec0000c440 tx=0x7fec00007e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:55.549 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.484+0000 7febfe7fc700 1 -- 172.21.15.195:0/2181213793 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec000050d0 con 0x7fec1006e850 2022-06-27T12:10:55.549 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.484+0000 7febfe7fc700 1 -- 172.21.15.195:0/2181213793 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec00005250 con 0x7fec1006e850 2022-06-27T12:10:55.550 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.484+0000 7febfe7fc700 1 -- 172.21.15.195:0/2181213793 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec000117e0 con 0x7fec1006e850 2022-06-27T12:10:55.550 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.484+0000 7fec18285700 1 -- 172.21.15.195:0/2181213793 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec10152f50 con 0x7fec1006e850 2022-06-27T12:10:55.551 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.484+0000 7fec18285700 1 -- 172.21.15.195:0/2181213793 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec10153510 con 0x7fec1006e850 2022-06-27T12:10:55.551 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.485+0000 7fec18285700 1 -- 172.21.15.195:0/2181213793 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec100722c0 con 0x7fec1006e850 2022-06-27T12:10:55.552 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.486+0000 7febfe7fc700 1 -- 172.21.15.195:0/2181213793 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fec00019030 con 0x7fec1006e850 2022-06-27T12:10:55.552 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.486+0000 7febfe7fc700 1 --2- 172.21.15.195:0/2181213793 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7febf0063d30 0x7febf00661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:55.553 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.486+0000 7febfe7fc700 1 -- 172.21.15.195:0/2181213793 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7320+0+0 (secure 0 0 0) 0x7fec0007fe40 con 0x7fec1006e850 2022-06-27T12:10:55.553 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.486+0000 7fec15820700 1 --2- 172.21.15.195:0/2181213793 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7febf0063d30 0x7febf00661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:55.554 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.486+0000 7fec15820700 1 --2- 172.21.15.195:0/2181213793 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7febf0063d30 0x7febf00661e0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fec10150dd0 tx=0x7fec0c008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:55.554 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.490+0000 7febfe7fc700 1 -- 172.21.15.195:0/2181213793 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fec0004e540 con 0x7fec1006e850 2022-06-27T12:10:55.719 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.717+0000 7fec18285700 1 -- 172.21.15.195:0/2181213793 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fec10156060 con 0x7fec1006e850 2022-06-27T12:10:55.719 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.718+0000 7febfe7fc700 1 -- 172.21.15.195:0/2181213793 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+5374 (secure 0 0 0) 0x7fec00017020 con 0x7fec1006e850 2022-06-27T12:10:55.720 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:10:55.722 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":88,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:10:55.231443+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5933},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5933":{"gid":5933,"name":"d","rank":0,"incarnation":85,"state":"up:clientreplay","state_seq":15,"addr":"172.21.15.202:6835/1342896965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":1342896965},{"type":"v1","addr":"172.21.15.202:6835","nonce":1342896965}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:10:55.731 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.720+0000 7fec18285700 1 -- 172.21.15.195:0/2181213793 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7febf0063d30 msgr2=0x7febf00661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:55.731 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.720+0000 7fec18285700 1 --2- 172.21.15.195:0/2181213793 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7febf0063d30 0x7febf00661e0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fec10150dd0 tx=0x7fec0c008040 comp rx=0 tx=0).stop 2022-06-27T12:10:55.732 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.720+0000 7fec18285700 1 -- 172.21.15.195:0/2181213793 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fec1006e850 msgr2=0x7fec10166c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:55.732 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.720+0000 7fec18285700 1 --2- 172.21.15.195:0/2181213793 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fec1006e850 0x7fec10166c40 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fec0000c440 tx=0x7fec00007e40 comp rx=0 tx=0).stop 2022-06-27T12:10:55.732 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.720+0000 7fec18285700 1 -- 172.21.15.195:0/2181213793 shutdown_connections 2022-06-27T12:10:55.732 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.720+0000 7fec18285700 1 --2- 172.21.15.195:0/2181213793 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7febf0063d30 0x7febf00661e0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:55.733 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.720+0000 7fec18285700 1 --2- 172.21.15.195:0/2181213793 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fec1006e850 0x7fec10166c40 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:55.733 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.720+0000 7fec18285700 1 --2- 172.21.15.195:0/2181213793 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fec10068590 0x7fec10155700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:55.733 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.720+0000 7fec18285700 1 --2- 172.21.15.195:0/2181213793 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fec100c34a0 0x7fec1014b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:55.733 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.720+0000 7fec18285700 1 -- 172.21.15.195:0/2181213793 >> 172.21.15.195:0/2181213793 conn(0x7fec100c6350 msgr2=0x7fec100c2cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:10:55.733 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.720+0000 7fec18285700 1 -- 172.21.15.195:0/2181213793 shutdown_connections 2022-06-27T12:10:55.734 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:55.720+0000 7fec18285700 1 -- 172.21.15.195:0/2181213793 wait complete. 2022-06-27T12:10:55.734 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 88 2022-06-27T12:10:55.736 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 88, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-06-27T11:47:52.358095+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 88, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5933": { "addr": "172.21.15.202:6835/1342896965", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6834", "nonce": 1342896965, "type": "v2" }, { "addr": "172.21.15.202:6835", "nonce": 1342896965, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5933, "incarnation": 85, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:clientreplay", "state_seq": 15 } }, "last_failure": 0, "last_failure_osd_epoch": 61, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-06-27T12:10:55.231443+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5933 } } } ], "standbys": [ { "addr": "172.21.15.202:6837/3305006841", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6836", "nonce": 3305006841, "type": "v2" }, { "addr": "172.21.15.202:6837", "nonce": 3305006841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6835/2060709822", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6834", "nonce": 2060709822, "type": "v2" }, { "addr": "172.21.15.195:6835", "nonce": 2060709822, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6838/1598751652", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6836", "nonce": 1598751652, "type": "v2" }, { "addr": "172.21.15.195:6838", "nonce": 1598751652, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4275, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6839/1313800661", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6837", "nonce": 1313800661, "type": "v2" }, { "addr": "172.21.15.195:6839", "nonce": 1313800661, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4278, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-06-27T12:10:57.740 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:10:57.901 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:57.899+0000 7f7ab1c04700 1 Processor -- start 2022-06-27T12:10:57.901 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:57.899+0000 7f7ab1c04700 1 -- start start 2022-06-27T12:10:57.902 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:57.900+0000 7f7ab1c04700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f7aac150580 0x7f7aac156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:57.902 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:57.900+0000 7f7ab1c04700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7aac150950 0x7f7aac0c7b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:57.902 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:57.900+0000 7f7ab1c04700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f7aac06f620 0x7f7aac06d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:57.902 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:57.900+0000 7f7ab1c04700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7aac157430 con 0x7f7aac06f620 2022-06-27T12:10:57.903 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:57.900+0000 7f7ab1c04700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7aac0c76b0 con 0x7f7aac150580 2022-06-27T12:10:57.903 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:57.900+0000 7f7ab1c04700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7aac0bad70 con 0x7f7aac150950 2022-06-27T12:10:57.904 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:57.900+0000 7f7aabfff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f7aac06f620 0x7f7aac06d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:57.904 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:57.900+0000 7f7aabfff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f7aac06f620 0x7f7aac06d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50148/0 (socket says 172.21.15.195:50148) 2022-06-27T12:10:57.904 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:57.900+0000 7f7aabfff700 1 -- 172.21.15.195:0/3270138850 learned_addr learned my addr 172.21.15.195:0/3270138850 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:10:57.904 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:57.900+0000 7f7aaaffd700 1 --2- 172.21.15.195:0/3270138850 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7aac150950 0x7f7aac0c7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:57.905 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:57.900+0000 7f7aab7fe700 1 --2- 172.21.15.195:0/3270138850 >> v2:172.21.15.202:3300/0 conn(0x7f7aac150580 0x7f7aac156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:57.907 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:57.900+0000 7f7aabfff700 1 -- 172.21.15.195:0/3270138850 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7aac150950 msgr2=0x7f7aac0c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:10:57.908 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:57.900+0000 7f7aabfff700 1 --2- 172.21.15.195:0/3270138850 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7aac150950 0x7f7aac0c7b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:57.908 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:57.900+0000 7f7aabfff700 1 -- 172.21.15.195:0/3270138850 >> v2:172.21.15.202:3300/0 conn(0x7f7aac150580 msgr2=0x7f7aac156ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:10:57.908 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:57.900+0000 7f7aabfff700 1 --2- 172.21.15.195:0/3270138850 >> v2:172.21.15.202:3300/0 conn(0x7f7aac150580 0x7f7aac156ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:57.909 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:57.900+0000 7f7aabfff700 1 -- 172.21.15.195:0/3270138850 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aac0ba4d0 con 0x7f7aac06f620 2022-06-27T12:10:57.909 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:57.900+0000 7f7aab7fe700 1 --2- 172.21.15.195:0/3270138850 >> v2:172.21.15.202:3300/0 conn(0x7f7aac150580 0x7f7aac156ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:10:57.909 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:57.901+0000 7f7aabfff700 1 --2- 172.21.15.195:0/3270138850 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7aac06f620 0x7f7aac06d6a0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f7aac0c8470 tx=0x7f7aa4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e35244924ee24383 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:57.910 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:57.901+0000 7f7aaa7fc700 1 -- 172.21.15.195:0/3270138850 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa400e070 con 0x7f7aac06f620 2022-06-27T12:10:57.910 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:57.901+0000 7f7aaa7fc700 1 -- 172.21.15.195:0/3270138850 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7aa400ba20 con 0x7f7aac06f620 2022-06-27T12:10:57.911 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:57.901+0000 7f7aaa7fc700 1 -- 172.21.15.195:0/3270138850 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa40092e0 con 0x7f7aac06f620 2022-06-27T12:10:57.911 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:57.901+0000 7f7ab1c04700 1 -- 172.21.15.195:0/3270138850 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7aac06f620 msgr2=0x7f7aac06d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:57.912 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:57.901+0000 7f7ab1c04700 1 --2- 172.21.15.195:0/3270138850 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7aac06f620 0x7f7aac06d6a0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f7aac0c8470 tx=0x7f7aa4009840 comp rx=0 tx=0).stop 2022-06-27T12:10:57.912 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:57.901+0000 7f7ab1c04700 1 -- 172.21.15.195:0/3270138850 shutdown_connections 2022-06-27T12:10:57.913 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:57.901+0000 7f7ab1c04700 1 --2- 172.21.15.195:0/3270138850 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7aac06f620 0x7f7aac06d6a0 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:57.913 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:57.901+0000 7f7ab1c04700 1 --2- 172.21.15.195:0/3270138850 >> v2:172.21.15.202:3300/0 conn(0x7f7aac150580 0x7f7aac156ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:57.914 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:57.901+0000 7f7ab1c04700 1 --2- 172.21.15.195:0/3270138850 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7aac150950 0x7f7aac0c7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:57.914 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:57.901+0000 7f7ab1c04700 1 -- 172.21.15.195:0/3270138850 >> 172.21.15.195:0/3270138850 conn(0x7f7aac063af0 msgr2=0x7f7aac063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:10:57.915 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:57.901+0000 7f7ab1c04700 1 -- 172.21.15.195:0/3270138850 shutdown_connections 2022-06-27T12:10:57.915 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:57.901+0000 7f7ab1c04700 1 -- 172.21.15.195:0/3270138850 wait complete. 2022-06-27T12:10:57.915 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:57.903+0000 7f7ab1c04700 1 Processor -- start 2022-06-27T12:10:57.916 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:57.903+0000 7f7ab1c04700 1 -- start start 2022-06-27T12:10:57.926 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:57.903+0000 7f7ab1c04700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7aac06f620 0x7f7aac14f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:57.927 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:57.903+0000 7f7ab1c04700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7aac150580 0x7f7aac166b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:57.927 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:57.903+0000 7f7ab1c04700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7aac150950 0x7f7aac167040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:57.927 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:57.903+0000 7f7ab1c04700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aac0bd7a0 con 0x7f7aac06f620 2022-06-27T12:10:57.928 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:57.903+0000 7f7ab1c04700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aac0ca320 con 0x7f7aac150580 2022-06-27T12:10:57.928 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:57.903+0000 7f7ab1c04700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7aac0bb330 con 0x7f7aac150950 2022-06-27T12:10:57.929 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:57.903+0000 7f7aab7fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7aac06f620 0x7f7aac14f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:57.929 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:57.903+0000 7f7aab7fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7aac06f620 0x7f7aac14f5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50150/0 (socket says 172.21.15.195:50150) 2022-06-27T12:10:57.930 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:57.903+0000 7f7aab7fe700 1 -- 172.21.15.195:0/3861757415 learned_addr learned my addr 172.21.15.195:0/3861757415 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:10:57.930 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:57.904+0000 7f7aabfff700 1 --2- 172.21.15.195:0/3861757415 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7aac150950 0x7f7aac167040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:57.931 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:57.904+0000 7f7aaaffd700 1 --2- 172.21.15.195:0/3861757415 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7aac150580 0x7f7aac166b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:57.931 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:57.904+0000 7f7aab7fe700 1 -- 172.21.15.195:0/3861757415 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7aac150950 msgr2=0x7f7aac167040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:57.932 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:57.904+0000 7f7aab7fe700 1 --2- 172.21.15.195:0/3861757415 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7aac150950 0x7f7aac167040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:57.932 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:57.904+0000 7f7aab7fe700 1 -- 172.21.15.195:0/3861757415 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7aac150580 msgr2=0x7f7aac166b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:57.933 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:57.904+0000 7f7aab7fe700 1 --2- 172.21.15.195:0/3861757415 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7aac150580 0x7f7aac166b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:57.933 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:57.904+0000 7f7aab7fe700 1 -- 172.21.15.195:0/3861757415 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aa4009ae0 con 0x7f7aac06f620 2022-06-27T12:10:57.933 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:57.904+0000 7f7aab7fe700 1 --2- 172.21.15.195:0/3861757415 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7aac06f620 0x7f7aac14f5b0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f7a9400d810 tx=0x7f7a94008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:57.934 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:57.904+0000 7f7aa37fe700 1 -- 172.21.15.195:0/3861757415 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a94004090 con 0x7f7aac06f620 2022-06-27T12:10:57.934 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:57.904+0000 7f7aa37fe700 1 -- 172.21.15.195:0/3861757415 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a940041f0 con 0x7f7aac06f620 2022-06-27T12:10:57.935 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:57.904+0000 7f7ab1c04700 1 -- 172.21.15.195:0/3861757415 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7aac0bb610 con 0x7f7aac06f620 2022-06-27T12:10:57.935 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:57.904+0000 7f7aa37fe700 1 -- 172.21.15.195:0/3861757415 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a940125f0 con 0x7f7aac06f620 2022-06-27T12:10:57.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:57.904+0000 7f7ab1c04700 1 -- 172.21.15.195:0/3861757415 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7aac0bbbd0 con 0x7f7aac06f620 2022-06-27T12:10:57.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:57.905+0000 7f7ab1c04700 1 -- 172.21.15.195:0/3861757415 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7aac0722c0 con 0x7f7aac06f620 2022-06-27T12:10:57.937 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:57.905+0000 7f7aa37fe700 1 -- 172.21.15.195:0/3861757415 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f7a940129c0 con 0x7f7aac06f620 2022-06-27T12:10:57.937 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:57.906+0000 7f7aa37fe700 1 --2- 172.21.15.195:0/3861757415 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7a90063cd0 0x7f7a90066180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:10:57.938 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:57.906+0000 7f7aa37fe700 1 -- 172.21.15.195:0/3861757415 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7320+0+0 (secure 0 0 0) 0x7f7a94081240 con 0x7f7aac06f620 2022-06-27T12:10:57.938 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:57.906+0000 7f7aaaffd700 1 --2- 172.21.15.195:0/3861757415 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7a90063cd0 0x7f7a90066180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:10:57.939 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:57.906+0000 7f7aaaffd700 1 --2- 172.21.15.195:0/3861757415 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7a90063cd0 0x7f7a90066180 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f7aac0c9cd0 tx=0x7f7a9c008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:10:57.939 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:57.910+0000 7f7aa37fe700 1 -- 172.21.15.195:0/3861757415 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f7a9404f940 con 0x7f7aac06f620 2022-06-27T12:10:58.147 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:58.135+0000 7f7ab1c04700 1 -- 172.21.15.195:0/3861757415 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7aac06fe20 con 0x7f7aac06f620 2022-06-27T12:10:58.148 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:58.136+0000 7f7aa37fe700 1 -- 172.21.15.195:0/3861757415 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+5374 (secure 0 0 0) 0x7f7a94052f60 con 0x7f7aac06f620 2022-06-27T12:10:58.151 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:10:58.152 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":88,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:10:55.231443+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5933},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5933":{"gid":5933,"name":"d","rank":0,"incarnation":85,"state":"up:clientreplay","state_seq":15,"addr":"172.21.15.202:6835/1342896965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":1342896965},{"type":"v1","addr":"172.21.15.202:6835","nonce":1342896965}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:10:58.155 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:58.138+0000 7f7ab1c04700 1 -- 172.21.15.195:0/3861757415 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7a90063cd0 msgr2=0x7f7a90066180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:58.155 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:58.138+0000 7f7ab1c04700 1 --2- 172.21.15.195:0/3861757415 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7a90063cd0 0x7f7a90066180 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f7aac0c9cd0 tx=0x7f7a9c008040 comp rx=0 tx=0).stop 2022-06-27T12:10:58.156 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:58.138+0000 7f7ab1c04700 1 -- 172.21.15.195:0/3861757415 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7aac06f620 msgr2=0x7f7aac14f5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:10:58.156 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:58.138+0000 7f7ab1c04700 1 --2- 172.21.15.195:0/3861757415 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7aac06f620 0x7f7aac14f5b0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f7a9400d810 tx=0x7f7a94008040 comp rx=0 tx=0).stop 2022-06-27T12:10:58.157 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:58.138+0000 7f7ab1c04700 1 -- 172.21.15.195:0/3861757415 shutdown_connections 2022-06-27T12:10:58.157 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:58.138+0000 7f7ab1c04700 1 --2- 172.21.15.195:0/3861757415 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7a90063cd0 0x7f7a90066180 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:58.157 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:58.138+0000 7f7ab1c04700 1 --2- 172.21.15.195:0/3861757415 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7aac06f620 0x7f7aac14f5b0 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:58.158 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:58.138+0000 7f7ab1c04700 1 --2- 172.21.15.195:0/3861757415 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7aac150950 0x7f7aac167040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:58.158 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:58.138+0000 7f7ab1c04700 1 --2- 172.21.15.195:0/3861757415 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7aac150580 0x7f7aac166b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:10:58.158 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:58.138+0000 7f7ab1c04700 1 -- 172.21.15.195:0/3861757415 >> 172.21.15.195:0/3861757415 conn(0x7f7aac063af0 msgr2=0x7f7aac06da50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:10:58.159 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:58.138+0000 7f7ab1c04700 1 -- 172.21.15.195:0/3861757415 shutdown_connections 2022-06-27T12:10:58.159 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:10:58.138+0000 7f7ab1c04700 1 -- 172.21.15.195:0/3861757415 wait complete. 2022-06-27T12:10:58.159 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 88 2022-06-27T12:10:58.160 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:11:00.162 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:11:00.327 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.319+0000 7f744b894700 1 Processor -- start 2022-06-27T12:11:00.328 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.321+0000 7f744b894700 1 -- start start 2022-06-27T12:11:00.328 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.322+0000 7f744b894700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f74440c40d0 0x7f7444157010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:00.328 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.322+0000 7f744b894700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f74440c6190 0x7f7444067ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:00.328 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.322+0000 7f744b894700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f74440c44a0 0x7f744414cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:00.329 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.322+0000 7f744b894700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7444156b80 con 0x7f74440c6190 2022-06-27T12:11:00.329 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.322+0000 7f744b894700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f744406a040 con 0x7f74440c44a0 2022-06-27T12:11:00.329 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.322+0000 7f744b894700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f744406a300 con 0x7f74440c40d0 2022-06-27T12:11:00.329 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.322+0000 7f7448e2f700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f74440c6190 0x7f7444067ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:00.329 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.322+0000 7f7448e2f700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f74440c6190 0x7f7444067ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50158/0 (socket says 172.21.15.195:50158) 2022-06-27T12:11:00.330 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.322+0000 7f7448e2f700 1 -- 172.21.15.195:0/465384521 learned_addr learned my addr 172.21.15.195:0/465384521 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:11:00.330 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.322+0000 7f7449e31700 1 --2- 172.21.15.195:0/465384521 >> v2:172.21.15.202:3300/0 conn(0x7f74440c44a0 0x7f744414cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:00.330 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.322+0000 7f7449630700 1 --2- 172.21.15.195:0/465384521 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f74440c40d0 0x7f7444157010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:00.331 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.323+0000 7f7449e31700 1 -- 172.21.15.195:0/465384521 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f74440c40d0 msgr2=0x7f7444157010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:11:00.331 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.323+0000 7f7449e31700 1 --2- 172.21.15.195:0/465384521 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f74440c40d0 0x7f7444157010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:00.331 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.323+0000 7f7449e31700 1 -- 172.21.15.195:0/465384521 >> v2:172.21.15.195:3300/0 conn(0x7f74440c6190 msgr2=0x7f7444067ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:11:00.331 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.323+0000 7f7449e31700 1 --2- 172.21.15.195:0/465384521 >> v2:172.21.15.195:3300/0 conn(0x7f74440c6190 0x7f7444067ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:00.331 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.323+0000 7f7449e31700 1 -- 172.21.15.195:0/465384521 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f744406a5e0 con 0x7f74440c44a0 2022-06-27T12:11:00.332 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.323+0000 7f7449e31700 1 --2- 172.21.15.195:0/465384521 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f74440c44a0 0x7f744414cb80 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f74440b2b60 tx=0x7f7440009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=768ef9726efb7474 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:00.332 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.323+0000 7f7433fff700 1 -- 172.21.15.195:0/465384521 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74400105b0 con 0x7f74440c44a0 2022-06-27T12:11:00.332 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.323+0000 7f7433fff700 1 -- 172.21.15.195:0/465384521 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7440010710 con 0x7f74440c44a0 2022-06-27T12:11:00.332 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.323+0000 7f7433fff700 1 -- 172.21.15.195:0/465384521 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7440010ab0 con 0x7f74440c44a0 2022-06-27T12:11:00.332 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.324+0000 7f744b894700 1 -- 172.21.15.195:0/465384521 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f74440c44a0 msgr2=0x7f744414cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:00.333 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.324+0000 7f744b894700 1 --2- 172.21.15.195:0/465384521 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f74440c44a0 0x7f744414cb80 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f74440b2b60 tx=0x7f7440009060 comp rx=0 tx=0).stop 2022-06-27T12:11:00.333 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.324+0000 7f744b894700 1 -- 172.21.15.195:0/465384521 shutdown_connections 2022-06-27T12:11:00.333 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.324+0000 7f744b894700 1 --2- 172.21.15.195:0/465384521 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f74440c44a0 0x7f744414cb80 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:00.334 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.324+0000 7f744b894700 1 --2- 172.21.15.195:0/465384521 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f74440c40d0 0x7f7444157010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:00.334 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.324+0000 7f744b894700 1 --2- 172.21.15.195:0/465384521 >> v2:172.21.15.195:3300/0 conn(0x7f74440c6190 0x7f7444067ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:00.334 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.324+0000 7f744b894700 1 -- 172.21.15.195:0/465384521 >> 172.21.15.195:0/465384521 conn(0x7f74440ccb80 msgr2=0x7f74440ccf80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:11:00.334 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.325+0000 7f744b894700 1 -- 172.21.15.195:0/465384521 shutdown_connections 2022-06-27T12:11:00.335 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.325+0000 7f744b894700 1 -- 172.21.15.195:0/465384521 wait complete. 2022-06-27T12:11:00.335 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.326+0000 7f744b894700 1 Processor -- start 2022-06-27T12:11:00.335 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.327+0000 7f744b894700 1 -- start start 2022-06-27T12:11:00.337 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.328+0000 7f744b894700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f74440c40d0 0x7f744416a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:00.338 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.328+0000 7f744b894700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f74440c44a0 0x7f7444157ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:00.338 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.328+0000 7f744b894700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f74440c6190 0x7f744415e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:00.338 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.328+0000 7f744b894700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7444158b70 con 0x7f74440c40d0 2022-06-27T12:11:00.338 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.328+0000 7f744b894700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7444158430 con 0x7f74440c6190 2022-06-27T12:11:00.339 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.328+0000 7f744b894700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f744414fd50 con 0x7f74440c44a0 2022-06-27T12:11:00.339 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.328+0000 7f7449630700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f74440c40d0 0x7f744416a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:00.339 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.328+0000 7f7449630700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f74440c40d0 0x7f744416a7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50162/0 (socket says 172.21.15.195:50162) 2022-06-27T12:11:00.339 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.328+0000 7f7448e2f700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f74440c44a0 0x7f7444157ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:00.340 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.328+0000 7f7449630700 1 -- 172.21.15.195:0/3533789476 learned_addr learned my addr 172.21.15.195:0/3533789476 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:11:00.340 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.328+0000 7f7449e31700 1 --2- 172.21.15.195:0/3533789476 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f74440c6190 0x7f744415e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:00.340 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.328+0000 7f7448e2f700 1 -- 172.21.15.195:0/3533789476 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f74440c6190 msgr2=0x7f744415e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:00.340 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.328+0000 7f7448e2f700 1 --2- 172.21.15.195:0/3533789476 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f74440c6190 0x7f744415e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:00.340 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.328+0000 7f7448e2f700 1 -- 172.21.15.195:0/3533789476 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f74440c40d0 msgr2=0x7f744416a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:00.341 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.328+0000 7f7448e2f700 1 --2- 172.21.15.195:0/3533789476 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f74440c40d0 0x7f744416a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:00.341 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.328+0000 7f7448e2f700 1 -- 172.21.15.195:0/3533789476 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74400092e0 con 0x7f74440c44a0 2022-06-27T12:11:00.341 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.328+0000 7f7448e2f700 1 --2- 172.21.15.195:0/3533789476 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f74440c44a0 0x7f7444157ef0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f743400e8e0 tx=0x7f7434009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:00.341 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.329+0000 7f74317fa700 1 -- 172.21.15.195:0/3533789476 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7434013680 con 0x7f74440c44a0 2022-06-27T12:11:00.342 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.329+0000 7f74317fa700 1 -- 172.21.15.195:0/3533789476 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74340137e0 con 0x7f74440c44a0 2022-06-27T12:11:00.342 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.329+0000 7f74317fa700 1 -- 172.21.15.195:0/3533789476 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7434013b60 con 0x7f74440c44a0 2022-06-27T12:11:00.342 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.329+0000 7f744b894700 1 -- 172.21.15.195:0/3533789476 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7444157a60 con 0x7f74440c44a0 2022-06-27T12:11:00.342 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.329+0000 7f744b894700 1 -- 172.21.15.195:0/3533789476 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f744414ff80 con 0x7f74440c44a0 2022-06-27T12:11:00.342 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.331+0000 7f744b894700 1 -- 172.21.15.195:0/3533789476 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7444157bf0 con 0x7f74440c44a0 2022-06-27T12:11:00.343 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.331+0000 7f74317fa700 1 -- 172.21.15.195:0/3533789476 <== mon.2 v2:172.21.15.202:3301/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f7434019030 con 0x7f74440c44a0 2022-06-27T12:11:00.343 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.332+0000 7f74317fa700 1 --2- 172.21.15.195:0/3533789476 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7424063d30 0x7f74240661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:00.343 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.332+0000 7f74317fa700 1 -- 172.21.15.195:0/3533789476 <== mon.2 v2:172.21.15.202:3301/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7320+0+0 (secure 0 0 0) 0x7f7434080c40 con 0x7f74440c44a0 2022-06-27T12:11:00.343 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.332+0000 7f7449630700 1 --2- 172.21.15.195:0/3533789476 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7424063d30 0x7f74240661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:00.344 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.332+0000 7f7449630700 1 --2- 172.21.15.195:0/3533789476 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7424063d30 0x7f74240661e0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f743c0094d0 tx=0x7f743c00b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:00.344 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.336+0000 7f74317fa700 1 -- 172.21.15.195:0/3533789476 <== mon.2 v2:172.21.15.202:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f743404f340 con 0x7f74440c44a0 2022-06-27T12:11:00.649 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.644+0000 7f744b894700 1 -- 172.21.15.195:0/3533789476 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f74440722c0 con 0x7f74440c44a0 2022-06-27T12:11:00.650 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.645+0000 7f74317fa700 1 -- 172.21.15.195:0/3533789476 <== mon.2 v2:172.21.15.202:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+5374 (secure 0 0 0) 0x7f7434017090 con 0x7f74440c44a0 2022-06-27T12:11:00.651 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:11:00.652 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":88,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:10:55.231443+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5933},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5933":{"gid":5933,"name":"d","rank":0,"incarnation":85,"state":"up:clientreplay","state_seq":15,"addr":"172.21.15.202:6835/1342896965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":1342896965},{"type":"v1","addr":"172.21.15.202:6835","nonce":1342896965}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:11:00.654 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.647+0000 7f744b894700 1 -- 172.21.15.195:0/3533789476 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7424063d30 msgr2=0x7f74240661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:00.654 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.647+0000 7f744b894700 1 --2- 172.21.15.195:0/3533789476 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7424063d30 0x7f74240661e0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f743c0094d0 tx=0x7f743c00b040 comp rx=0 tx=0).stop 2022-06-27T12:11:00.655 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.647+0000 7f744b894700 1 -- 172.21.15.195:0/3533789476 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f74440c44a0 msgr2=0x7f7444157ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:00.655 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.647+0000 7f744b894700 1 --2- 172.21.15.195:0/3533789476 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f74440c44a0 0x7f7444157ef0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f743400e8e0 tx=0x7f7434009120 comp rx=0 tx=0).stop 2022-06-27T12:11:00.655 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.647+0000 7f744b894700 1 -- 172.21.15.195:0/3533789476 shutdown_connections 2022-06-27T12:11:00.656 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.647+0000 7f744b894700 1 --2- 172.21.15.195:0/3533789476 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7424063d30 0x7f74240661e0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:00.656 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.647+0000 7f744b894700 1 --2- 172.21.15.195:0/3533789476 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f74440c40d0 0x7f744416a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:00.657 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.647+0000 7f744b894700 1 --2- 172.21.15.195:0/3533789476 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f74440c6190 0x7f744415e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:00.657 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.647+0000 7f744b894700 1 --2- 172.21.15.195:0/3533789476 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f74440c44a0 0x7f7444157ef0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:00.658 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.647+0000 7f744b894700 1 -- 172.21.15.195:0/3533789476 >> 172.21.15.195:0/3533789476 conn(0x7f74440ccb80 msgr2=0x7f74440c9600 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:11:00.658 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.647+0000 7f744b894700 1 -- 172.21.15.195:0/3533789476 shutdown_connections 2022-06-27T12:11:00.658 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:00.647+0000 7f744b894700 1 -- 172.21.15.195:0/3533789476 wait complete. 2022-06-27T12:11:00.659 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 88 2022-06-27T12:11:00.674 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:11:02.675 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:11:02.795 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:11:02.797 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:11:02.809 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:02.808+0000 7f83c49b0700 1 Processor -- start 2022-06-27T12:11:02.809 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:02.809+0000 7f83c49b0700 1 -- start start 2022-06-27T12:11:02.815 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:02.809+0000 7f83c49b0700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f83c006e850 0x7f83c01570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:02.815 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:02.810+0000 7f83c49b0700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f83c006fef0 0x7f83c0069de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:02.816 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:02.810+0000 7f83c49b0700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83c006fba0 con 0x7f83c00c34a0 2022-06-27T12:11:02.816 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:02.810+0000 7f83c49b0700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83c01575e0 con 0x7f83c006e850 2022-06-27T12:11:02.817 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:02.810+0000 7f83c49b0700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83c006f5e0 con 0x7f83c006fef0 2022-06-27T12:11:02.817 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:02.810+0000 7f83be59c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f83c006e850 0x7f83c01570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:02.817 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:02.810+0000 7f83be59c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f83c006e850 0x7f83c01570a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50170/0 (socket says 172.21.15.195:50170) 2022-06-27T12:11:02.817 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:02.810+0000 7f83bed9d700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f83c00c34a0 0x7f83c00c2920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:56362/0 (socket says 172.21.15.195:56362) 2022-06-27T12:11:02.818 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:02.810+0000 7f83be59c700 1 -- 172.21.15.195:0/91930617 learned_addr learned my addr 172.21.15.195:0/91930617 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:11:02.818 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:02.810+0000 7f83bdd9b700 1 --2- 172.21.15.195:0/91930617 >> v2:172.21.15.202:3300/0 conn(0x7f83c006fef0 0x7f83c0069de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:02.818 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:02.810+0000 7f83be59c700 1 -- 172.21.15.195:0/91930617 >> v2:172.21.15.202:3300/0 conn(0x7f83c006fef0 msgr2=0x7f83c0069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:11:02.819 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:02.810+0000 7f83be59c700 1 --2- 172.21.15.195:0/91930617 >> v2:172.21.15.202:3300/0 conn(0x7f83c006fef0 0x7f83c0069de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:02.819 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:02.810+0000 7f83be59c700 1 -- 172.21.15.195:0/91930617 >> v1:172.21.15.195:6789/0 conn(0x7f83c00c34a0 legacy=0x7f83c00c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:02.819 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:02.810+0000 7f83be59c700 1 -- 172.21.15.195:0/91930617 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83c006f8c0 con 0x7f83c006e850 2022-06-27T12:11:02.819 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:02.810+0000 7f83bd59a700 1 -- 172.21.15.195:0/91930617 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3034264663 0 0) 0x7f83c006fba0 con 0x7f83c00c34a0 2022-06-27T12:11:02.820 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:02.810+0000 7f83be59c700 1 --2- 172.21.15.195:0/91930617 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f83c006e850 0x7f83c01570a0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f83c006e600 tx=0x7f83b0009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f302e9b4bc98ac server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:02.820 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:02.810+0000 7f83bd59a700 1 -- 172.21.15.195:0/91930617 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83b0011630 con 0x7f83c006e850 2022-06-27T12:11:02.820 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:02.810+0000 7f83bd59a700 1 -- 172.21.15.195:0/91930617 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83b0011790 con 0x7f83c006e850 2022-06-27T12:11:02.821 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:02.810+0000 7f83bd59a700 1 -- 172.21.15.195:0/91930617 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83b0011b30 con 0x7f83c006e850 2022-06-27T12:11:02.821 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:02.811+0000 7f83c49b0700 1 -- 172.21.15.195:0/91930617 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f83c006e850 msgr2=0x7f83c01570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:02.821 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:02.811+0000 7f83c49b0700 1 --2- 172.21.15.195:0/91930617 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f83c006e850 0x7f83c01570a0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f83c006e600 tx=0x7f83b0009120 comp rx=0 tx=0).stop 2022-06-27T12:11:02.821 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:02.811+0000 7f83c49b0700 1 -- 172.21.15.195:0/91930617 shutdown_connections 2022-06-27T12:11:02.821 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:02.811+0000 7f83c49b0700 1 --2- 172.21.15.195:0/91930617 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f83c006e850 0x7f83c01570a0 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:02.822 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:02.811+0000 7f83c49b0700 1 --2- 172.21.15.195:0/91930617 >> v2:172.21.15.202:3300/0 conn(0x7f83c006fef0 0x7f83c0069de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:02.822 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:02.811+0000 7f83c49b0700 1 -- 172.21.15.195:0/91930617 >> 172.21.15.195:0/91930617 conn(0x7f83c00c6350 msgr2=0x7f83c00c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:11:02.822 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:02.811+0000 7f83c49b0700 1 -- 172.21.15.195:0/91930617 shutdown_connections 2022-06-27T12:11:02.822 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:02.811+0000 7f83c49b0700 1 -- 172.21.15.195:0/91930617 wait complete. 2022-06-27T12:11:02.823 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:02.812+0000 7f83c49b0700 1 Processor -- start 2022-06-27T12:11:02.823 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:02.812+0000 7f83c49b0700 1 -- start start 2022-06-27T12:11:02.823 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:02.813+0000 7f83c49b0700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f83c006e850 0x7f83c016a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:02.823 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:02.813+0000 7f83c49b0700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f83c00c34a0 0x7f83c00b72c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:02.823 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:02.813+0000 7f83c49b0700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f83c00b7a20 0x7f83c00c8380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:02.824 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:02.813+0000 7f83c49b0700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83c00b9c30 con 0x7f83c00c34a0 2022-06-27T12:11:02.824 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:02.813+0000 7f83c49b0700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83c00b7800 con 0x7f83c00b7a20 2022-06-27T12:11:02.824 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:02.813+0000 7f83c49b0700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83c00b6e50 con 0x7f83c006e850 2022-06-27T12:11:02.824 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:02.813+0000 7f83bdd9b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f83c00c34a0 0x7f83c00b72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:02.825 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:02.813+0000 7f83bdd9b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f83c00c34a0 0x7f83c00b72c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50178/0 (socket says 172.21.15.195:50178) 2022-06-27T12:11:02.825 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:02.813+0000 7f83bdd9b700 1 -- 172.21.15.195:0/1454759763 learned_addr learned my addr 172.21.15.195:0/1454759763 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:11:02.825 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:02.813+0000 7f83be59c700 1 --2- 172.21.15.195:0/1454759763 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f83c006e850 0x7f83c016a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:02.825 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:02.813+0000 7f83bed9d700 1 --2- 172.21.15.195:0/1454759763 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f83c00b7a20 0x7f83c00c8380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:02.825 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:02.814+0000 7f83bdd9b700 1 -- 172.21.15.195:0/1454759763 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f83c006e850 msgr2=0x7f83c016a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:02.826 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:02.814+0000 7f83bdd9b700 1 --2- 172.21.15.195:0/1454759763 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f83c006e850 0x7f83c016a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:02.826 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:02.814+0000 7f83bdd9b700 1 -- 172.21.15.195:0/1454759763 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f83c00b7a20 msgr2=0x7f83c00c8380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:02.826 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:02.814+0000 7f83bdd9b700 1 --2- 172.21.15.195:0/1454759763 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f83c00b7a20 0x7f83c00c8380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:02.826 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:02.814+0000 7f83bdd9b700 1 -- 172.21.15.195:0/1454759763 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b00093a0 con 0x7f83c00c34a0 2022-06-27T12:11:02.834 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:02.814+0000 7f83bdd9b700 1 --2- 172.21.15.195:0/1454759763 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f83c00c34a0 0x7f83c00b72c0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f83b400e960 tx=0x7f83b4009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:02.834 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:02.814+0000 7f83ae7fc700 1 -- 172.21.15.195:0/1454759763 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83b4013680 con 0x7f83c00c34a0 2022-06-27T12:11:02.835 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:02.814+0000 7f83ae7fc700 1 -- 172.21.15.195:0/1454759763 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83b40137e0 con 0x7f83c00c34a0 2022-06-27T12:11:02.835 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:02.814+0000 7f83ae7fc700 1 -- 172.21.15.195:0/1454759763 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83b4013b80 con 0x7f83c00c34a0 2022-06-27T12:11:02.836 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:02.814+0000 7f83c49b0700 1 -- 172.21.15.195:0/1454759763 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83c014fef0 con 0x7f83c00c34a0 2022-06-27T12:11:02.836 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:02.814+0000 7f83c49b0700 1 -- 172.21.15.195:0/1454759763 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83c00b6b70 con 0x7f83c00c34a0 2022-06-27T12:11:02.837 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:02.815+0000 7f83c49b0700 1 -- 172.21.15.195:0/1454759763 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83c00722c0 con 0x7f83c00c34a0 2022-06-27T12:11:02.839 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:02.815+0000 7f83ae7fc700 1 -- 172.21.15.195:0/1454759763 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f83b4019030 con 0x7f83c00c34a0 2022-06-27T12:11:02.840 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:02.816+0000 7f83ae7fc700 1 --2- 172.21.15.195:0/1454759763 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f839c063d00 0x7f839c0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:02.840 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:02.816+0000 7f83ae7fc700 1 -- 172.21.15.195:0/1454759763 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7320+0+0 (secure 0 0 0) 0x7f83b4080a50 con 0x7f83c00c34a0 2022-06-27T12:11:02.841 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:02.816+0000 7f83be59c700 1 --2- 172.21.15.195:0/1454759763 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f839c063d00 0x7f839c0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:02.841 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:02.816+0000 7f83be59c700 1 --2- 172.21.15.195:0/1454759763 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f839c063d00 0x7f839c0661b0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f83b0009160 tx=0x7f83b0017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:02.842 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:02.820+0000 7f83ae7fc700 1 -- 172.21.15.195:0/1454759763 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f83b404f150 con 0x7f83c00c34a0 2022-06-27T12:11:03.048 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:03.047+0000 7f83c49b0700 1 -- 172.21.15.195:0/1454759763 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f83c004c470 con 0x7f83c00c34a0 2022-06-27T12:11:03.048 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:03.047+0000 7f83ae7fc700 1 -- 172.21.15.195:0/1454759763 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+5368 (secure 0 0 0) 0x7f83b4017070 con 0x7f83c00c34a0 2022-06-27T12:11:03.049 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:11:03.049 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":89,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:11:01.244537+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5933},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5933":{"gid":5933,"name":"d","rank":0,"incarnation":85,"state":"up:active","state_seq":17,"addr":"172.21.15.202:6835/1342896965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":1342896965},{"type":"v1","addr":"172.21.15.202:6835","nonce":1342896965}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:11:03.051 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:03.049+0000 7f83c49b0700 1 -- 172.21.15.195:0/1454759763 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f839c063d00 msgr2=0x7f839c0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:03.052 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:03.049+0000 7f83c49b0700 1 --2- 172.21.15.195:0/1454759763 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f839c063d00 0x7f839c0661b0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f83b0009160 tx=0x7f83b0017000 comp rx=0 tx=0).stop 2022-06-27T12:11:03.052 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:03.049+0000 7f83c49b0700 1 -- 172.21.15.195:0/1454759763 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f83c00c34a0 msgr2=0x7f83c00b72c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:03.052 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:03.049+0000 7f83c49b0700 1 --2- 172.21.15.195:0/1454759763 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f83c00c34a0 0x7f83c00b72c0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f83b400e960 tx=0x7f83b4009060 comp rx=0 tx=0).stop 2022-06-27T12:11:03.053 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:03.049+0000 7f83c49b0700 1 -- 172.21.15.195:0/1454759763 shutdown_connections 2022-06-27T12:11:03.053 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:03.049+0000 7f83c49b0700 1 --2- 172.21.15.195:0/1454759763 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f839c063d00 0x7f839c0661b0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:03.054 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:03.049+0000 7f83c49b0700 1 --2- 172.21.15.195:0/1454759763 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f83c00b7a20 0x7f83c00c8380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:03.054 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:03.049+0000 7f83c49b0700 1 --2- 172.21.15.195:0/1454759763 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f83c006e850 0x7f83c016a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:03.054 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:03.049+0000 7f83c49b0700 1 --2- 172.21.15.195:0/1454759763 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f83c00c34a0 0x7f83c00b72c0 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:03.056 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:03.049+0000 7f83c49b0700 1 -- 172.21.15.195:0/1454759763 >> 172.21.15.195:0/1454759763 conn(0x7f83c00c6350 msgr2=0x7f83c00c2b80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:11:03.056 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:03.049+0000 7f83c49b0700 1 -- 172.21.15.195:0/1454759763 shutdown_connections 2022-06-27T12:11:03.069 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:03.049+0000 7f83c49b0700 1 -- 172.21.15.195:0/1454759763 wait complete. 2022-06-27T12:11:03.070 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 89 2022-06-27T12:11:03.080 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d has gained rank=0, replacing gid=6708 2022-06-27T12:11:03.080 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 4 secs before reviving mds.b 2022-06-27T12:11:06.448 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:11:06.449 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 37 -eq 48 ']' 2022-06-27T12:11:06.450 INFO:tasks.workunit.client.0.smithi195.stdout:v0.37 2022-06-27T12:11:06.450 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.37 2022-06-27T12:11:06.451 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.37 2022-06-27T12:11:06.451 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.37 2022-06-27T12:11:07.084 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2022-06-27T12:11:07.085 INFO:tasks.ceph.mds.b:Restarting daemon 2022-06-27T12:11:07.086 DEBUG:teuthology.orchestra.run.smithi202:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-06-27T12:11:07.110 INFO:tasks.ceph.mds.b:Started 2022-06-27T12:11:07.111 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:11:07.190 INFO:tasks.ceph.mds.b.smithi202.stdout:starting mds.b at 2022-06-27T12:11:07.239 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.238+0000 7f411bd5c700 1 Processor -- start 2022-06-27T12:11:07.240 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.238+0000 7f411bd5c700 1 -- start start 2022-06-27T12:11:07.276 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.239+0000 7f411bd5c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f41141508d0 0x7f4114063af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:07.277 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.239+0000 7f411bd5c700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4114064030 con 0x7f4114150500 2022-06-27T12:11:07.277 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.239+0000 7f411bd5c700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41140c4160 con 0x7f411406a450 2022-06-27T12:11:07.278 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.239+0000 7f411bd5c700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41140c4440 con 0x7f41141508d0 2022-06-27T12:11:07.278 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.239+0000 7f411a2f9700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f41141508d0 0x7f4114063af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:07.279 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.239+0000 7f411a2f9700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f41141508d0 0x7f4114063af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50192/0 (socket says 172.21.15.195:50192) 2022-06-27T12:11:07.279 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.239+0000 7f411a2f9700 1 -- 172.21.15.195:0/3285797579 learned_addr learned my addr 172.21.15.195:0/3285797579 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:11:07.280 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.239+0000 7f4119af8700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f4114150500 0x7f41140c48d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:56376/0 (socket says 172.21.15.195:56376) 2022-06-27T12:11:07.280 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.240+0000 7f411a2f9700 1 -- 172.21.15.195:0/3285797579 >> v1:172.21.15.202:6789/0 conn(0x7f411406a450 legacy=0x7f41140bbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:07.280 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.240+0000 7f411a2f9700 1 -- 172.21.15.195:0/3285797579 >> v1:172.21.15.195:6789/0 conn(0x7f4114150500 legacy=0x7f41140c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:07.280 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.240+0000 7f4118af6700 1 -- 172.21.15.195:0/3285797579 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3557768797 0 0) 0x7f41040011d0 con 0x7f4114150500 2022-06-27T12:11:07.281 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.240+0000 7f411a2f9700 1 -- 172.21.15.195:0/3285797579 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f411415a800 con 0x7f41141508d0 2022-06-27T12:11:07.281 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.240+0000 7f411a2f9700 1 --2- 172.21.15.195:0/3285797579 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f41141508d0 0x7f4114063af0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f410c009f20 tx=0x7f410c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cb89f6846f0ecd2 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:07.281 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.240+0000 7f4118af6700 1 -- 172.21.15.195:0/3285797579 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f410c00e070 con 0x7f41141508d0 2022-06-27T12:11:07.281 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.240+0000 7f4118af6700 1 -- 172.21.15.195:0/3285797579 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f410c00ba80 con 0x7f41141508d0 2022-06-27T12:11:07.282 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.240+0000 7f4118af6700 1 -- 172.21.15.195:0/3285797579 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f410c0092e0 con 0x7f41141508d0 2022-06-27T12:11:07.282 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.240+0000 7f411bd5c700 1 -- 172.21.15.195:0/3285797579 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f41141508d0 msgr2=0x7f4114063af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:07.282 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.240+0000 7f411bd5c700 1 --2- 172.21.15.195:0/3285797579 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f41141508d0 0x7f4114063af0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f410c009f20 tx=0x7f410c009840 comp rx=0 tx=0).stop 2022-06-27T12:11:07.282 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.240+0000 7f411bd5c700 1 -- 172.21.15.195:0/3285797579 shutdown_connections 2022-06-27T12:11:07.283 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.240+0000 7f411bd5c700 1 --2- 172.21.15.195:0/3285797579 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f41141508d0 0x7f4114063af0 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:07.283 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.240+0000 7f411bd5c700 1 -- 172.21.15.195:0/3285797579 >> 172.21.15.195:0/3285797579 conn(0x7f41140c9b00 msgr2=0x7f41140c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:11:07.283 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.241+0000 7f411bd5c700 1 -- 172.21.15.195:0/3285797579 shutdown_connections 2022-06-27T12:11:07.283 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.241+0000 7f411bd5c700 1 -- 172.21.15.195:0/3285797579 wait complete. 2022-06-27T12:11:07.284 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.242+0000 7f411bd5c700 1 Processor -- start 2022-06-27T12:11:07.284 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.242+0000 7f411bd5c700 1 -- start start 2022-06-27T12:11:07.284 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.243+0000 7f411bd5c700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f411406a450 0x7f41140bed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:07.284 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.243+0000 7f411bd5c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4114150500 0x7f41141589c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:07.285 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.243+0000 7f411bd5c700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f41141508d0 0x7f4114167c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:07.285 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.243+0000 7f411bd5c700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41140c7970 con 0x7f4114150500 2022-06-27T12:11:07.285 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.243+0000 7f411bd5c700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41140bf240 con 0x7f411406a450 2022-06-27T12:11:07.285 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.243+0000 7f411bd5c700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f411416a840 con 0x7f41141508d0 2022-06-27T12:11:07.286 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.243+0000 7f41192f7700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4114150500 0x7f41141589c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:07.286 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.243+0000 7f41192f7700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4114150500 0x7f41141589c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50196/0 (socket says 172.21.15.195:50196) 2022-06-27T12:11:07.286 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.243+0000 7f41192f7700 1 -- 172.21.15.195:0/1796895134 learned_addr learned my addr 172.21.15.195:0/1796895134 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:11:07.286 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.243+0000 7f4119af8700 1 --2- 172.21.15.195:0/1796895134 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f411406a450 0x7f41140bed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:07.286 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.243+0000 7f411a2f9700 1 --2- 172.21.15.195:0/1796895134 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f41141508d0 0x7f4114167c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:07.287 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.243+0000 7f41192f7700 1 -- 172.21.15.195:0/1796895134 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f41141508d0 msgr2=0x7f4114167c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:07.287 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.243+0000 7f41192f7700 1 --2- 172.21.15.195:0/1796895134 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f41141508d0 0x7f4114167c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:07.287 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.243+0000 7f41192f7700 1 -- 172.21.15.195:0/1796895134 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f411406a450 msgr2=0x7f41140bed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:07.287 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.243+0000 7f41192f7700 1 --2- 172.21.15.195:0/1796895134 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f411406a450 0x7f41140bed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:07.288 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.243+0000 7f41192f7700 1 -- 172.21.15.195:0/1796895134 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f410c009ae0 con 0x7f4114150500 2022-06-27T12:11:07.288 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.243+0000 7f411a2f9700 1 --2- 172.21.15.195:0/1796895134 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f41141508d0 0x7f4114167c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:11:07.288 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.244+0000 7f41192f7700 1 --2- 172.21.15.195:0/1796895134 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4114150500 0x7f41141589c0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f411000c490 tx=0x7f4110007e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:07.288 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.244+0000 7f4101ffb700 1 -- 172.21.15.195:0/1796895134 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41100050d0 con 0x7f4114150500 2022-06-27T12:11:07.289 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.244+0000 7f4101ffb700 1 -- 172.21.15.195:0/1796895134 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4110005230 con 0x7f4114150500 2022-06-27T12:11:07.289 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.244+0000 7f411bd5c700 1 -- 172.21.15.195:0/1796895134 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41140bf480 con 0x7f4114150500 2022-06-27T12:11:07.290 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.244+0000 7f4101ffb700 1 -- 172.21.15.195:0/1796895134 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41100117a0 con 0x7f4114150500 2022-06-27T12:11:07.290 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.244+0000 7f411bd5c700 1 -- 172.21.15.195:0/1796895134 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f411416a1f0 con 0x7f4114150500 2022-06-27T12:11:07.290 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.245+0000 7f411bd5c700 1 -- 172.21.15.195:0/1796895134 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41140722c0 con 0x7f4114150500 2022-06-27T12:11:07.290 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.245+0000 7f4101ffb700 1 -- 172.21.15.195:0/1796895134 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f4110019030 con 0x7f4114150500 2022-06-27T12:11:07.291 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.245+0000 7f4101ffb700 1 --2- 172.21.15.195:0/1796895134 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f40f4063d00 0x7f40f40661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:07.291 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.246+0000 7f4101ffb700 1 -- 172.21.15.195:0/1796895134 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7320+0+0 (secure 0 0 0) 0x7f411007fe50 con 0x7f4114150500 2022-06-27T12:11:07.291 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.246+0000 7f4119af8700 1 --2- 172.21.15.195:0/1796895134 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f40f4063d00 0x7f40f40661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:07.291 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.246+0000 7f4119af8700 1 --2- 172.21.15.195:0/1796895134 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f40f4063d00 0x7f40f40661b0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f4104000c00 tx=0x7f4104007940 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:07.292 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.250+0000 7f4101ffb700 1 -- 172.21.15.195:0/1796895134 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f4110049c90 con 0x7f4114150500 2022-06-27T12:11:07.480 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.472+0000 7f411bd5c700 1 -- 172.21.15.195:0/1796895134 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f41140ba8e0 con 0x7f4114150500 2022-06-27T12:11:07.482 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.472+0000 7f4101ffb700 1 -- 172.21.15.195:0/1796895134 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+6128 (secure 0 0 0) 0x7f4110017020 con 0x7f4114150500 2022-06-27T12:11:07.485 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:11:07.486 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":90,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":6050,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/3890799919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3890799919},{"type":"v1","addr":"172.21.15.202:6833","nonce":3890799919}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":90}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:11:01.244537+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5933},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5933":{"gid":5933,"name":"d","rank":0,"incarnation":85,"state":"up:active","state_seq":17,"addr":"172.21.15.202:6835/1342896965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":1342896965},{"type":"v1","addr":"172.21.15.202:6835","nonce":1342896965}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:11:07.489 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.474+0000 7f411bd5c700 1 -- 172.21.15.195:0/1796895134 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f40f4063d00 msgr2=0x7f40f40661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:07.490 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.474+0000 7f411bd5c700 1 --2- 172.21.15.195:0/1796895134 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f40f4063d00 0x7f40f40661b0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f4104000c00 tx=0x7f4104007940 comp rx=0 tx=0).stop 2022-06-27T12:11:07.491 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.474+0000 7f411bd5c700 1 -- 172.21.15.195:0/1796895134 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4114150500 msgr2=0x7f41141589c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:07.491 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.474+0000 7f411bd5c700 1 --2- 172.21.15.195:0/1796895134 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4114150500 0x7f41141589c0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f411000c490 tx=0x7f4110007e40 comp rx=0 tx=0).stop 2022-06-27T12:11:07.492 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.474+0000 7f411bd5c700 1 -- 172.21.15.195:0/1796895134 shutdown_connections 2022-06-27T12:11:07.492 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.474+0000 7f411bd5c700 1 --2- 172.21.15.195:0/1796895134 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f40f4063d00 0x7f40f40661b0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:07.493 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.474+0000 7f411bd5c700 1 --2- 172.21.15.195:0/1796895134 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f41141508d0 0x7f4114167c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:07.497 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.474+0000 7f411bd5c700 1 --2- 172.21.15.195:0/1796895134 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f411406a450 0x7f41140bed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:07.498 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.474+0000 7f411bd5c700 1 --2- 172.21.15.195:0/1796895134 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4114150500 0x7f41141589c0 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:07.498 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.474+0000 7f411bd5c700 1 -- 172.21.15.195:0/1796895134 >> 172.21.15.195:0/1796895134 conn(0x7f41140c9b00 msgr2=0x7f41140c6f20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:11:07.499 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.474+0000 7f411bd5c700 1 -- 172.21.15.195:0/1796895134 shutdown_connections 2022-06-27T12:11:07.499 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:07.474+0000 7f411bd5c700 1 -- 172.21.15.195:0/1796895134 wait complete. 2022-06-27T12:11:07.499 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 90 2022-06-27T12:11:07.500 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2022-06-27T12:11:07.501 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 3 secs before thrashing 2022-06-27T12:11:09.369 INFO:tasks.ceph.mds.d.smithi202.stderr:2022-06-27T12:11:09.369+0000 7f828aef6700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2022-06-27T12:11:10.502 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:11:10.695 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.635+0000 7fa4ab3e9700 1 Processor -- start 2022-06-27T12:11:10.695 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.636+0000 7fa4ab3e9700 1 -- start start 2022-06-27T12:11:10.696 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.636+0000 7fa4ab3e9700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fa4a406e850 0x7fa4a41570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:10.696 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.636+0000 7fa4ab3e9700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa4a406fef0 0x7fa4a4069de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:10.702 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.636+0000 7fa4ab3e9700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4a406fba0 con 0x7fa4a40c34a0 2022-06-27T12:11:10.703 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.636+0000 7fa4ab3e9700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4a41575e0 con 0x7fa4a406e850 2022-06-27T12:11:10.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.636+0000 7fa4ab3e9700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a406f5e0 con 0x7fa4a406fef0 2022-06-27T12:11:10.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.637+0000 7fa4a9185700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fa4a406e850 0x7fa4a41570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:10.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.637+0000 7fa4a8984700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa4a406fef0 0x7fa4a4069de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:10.705 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.637+0000 7fa4a9185700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fa4a406e850 0x7fa4a41570a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3300/0 says I am v2:172.21.15.195:48340/0 (socket says 172.21.15.195:48340) 2022-06-27T12:11:10.705 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.637+0000 7fa4a9986700 1 --1- >> v1:172.21.15.202:6789/0 conn(0x7fa4a40c34a0 0x7fa4a40c2920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.202:6789/0 says I am v1:172.21.15.195:58266/0 (socket says 172.21.15.195:58266) 2022-06-27T12:11:10.711 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.637+0000 7fa4a8984700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa4a406fef0 0x7fa4a4069de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3301/0 says I am v2:172.21.15.195:42970/0 (socket says 172.21.15.195:42970) 2022-06-27T12:11:10.712 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.637+0000 7fa4a9185700 1 -- 172.21.15.195:0/228490896 learned_addr learned my addr 172.21.15.195:0/228490896 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:11:10.712 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.637+0000 7fa4a9185700 1 -- 172.21.15.195:0/228490896 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa4a406fef0 msgr2=0x7fa4a4069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:11:10.719 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.637+0000 7fa4a9185700 1 --2- 172.21.15.195:0/228490896 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa4a406fef0 0x7fa4a4069de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:10.721 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.637+0000 7fa4a9185700 1 -- 172.21.15.195:0/228490896 >> v1:172.21.15.202:6789/0 conn(0x7fa4a40c34a0 legacy=0x7fa4a40c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:10.721 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.637+0000 7fa4a9185700 1 -- 172.21.15.195:0/228490896 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4a406f8c0 con 0x7fa4a406e850 2022-06-27T12:11:10.722 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.637+0000 7fa4a8984700 1 --2- 172.21.15.195:0/228490896 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa4a406fef0 0x7fa4a4069de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:11:10.722 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.637+0000 7fa4a9185700 1 --2- 172.21.15.195:0/228490896 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa4a406e850 0x7fa4a41570a0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fa494009860 tx=0x7fa494009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be6e9eeca7500809 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:10.723 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.638+0000 7fa493fff700 1 -- 172.21.15.195:0/228490896 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa49400b690 con 0x7fa4a406e850 2022-06-27T12:11:10.723 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.638+0000 7fa493fff700 1 -- 172.21.15.195:0/228490896 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa49400b7f0 con 0x7fa4a406e850 2022-06-27T12:11:10.724 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.638+0000 7fa493fff700 1 -- 172.21.15.195:0/228490896 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa49400bb90 con 0x7fa4a406e850 2022-06-27T12:11:10.724 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.638+0000 7fa4ab3e9700 1 -- 172.21.15.195:0/228490896 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa4a406e850 msgr2=0x7fa4a41570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:10.725 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.638+0000 7fa4ab3e9700 1 --2- 172.21.15.195:0/228490896 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa4a406e850 0x7fa4a41570a0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fa494009860 tx=0x7fa494009120 comp rx=0 tx=0).stop 2022-06-27T12:11:10.728 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.638+0000 7fa4ab3e9700 1 -- 172.21.15.195:0/228490896 shutdown_connections 2022-06-27T12:11:10.728 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.638+0000 7fa4ab3e9700 1 --2- 172.21.15.195:0/228490896 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa4a406e850 0x7fa4a41570a0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:10.729 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.638+0000 7fa4ab3e9700 1 --2- 172.21.15.195:0/228490896 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa4a406fef0 0x7fa4a4069de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:10.729 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.638+0000 7fa4ab3e9700 1 -- 172.21.15.195:0/228490896 >> 172.21.15.195:0/228490896 conn(0x7fa4a40c6350 msgr2=0x7fa4a40c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:11:10.730 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.638+0000 7fa4ab3e9700 1 -- 172.21.15.195:0/228490896 shutdown_connections 2022-06-27T12:11:10.731 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.638+0000 7fa4ab3e9700 1 -- 172.21.15.195:0/228490896 wait complete. 2022-06-27T12:11:10.731 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.639+0000 7fa4ab3e9700 1 Processor -- start 2022-06-27T12:11:10.732 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.639+0000 7fa4ab3e9700 1 -- start start 2022-06-27T12:11:10.733 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.640+0000 7fa4ab3e9700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa4a406e850 0x7fa4a4168740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:10.734 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.640+0000 7fa4ab3e9700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa4a40c34a0 0x7fa4a4149d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:10.734 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.640+0000 7fa4ab3e9700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa4a4149300 0x7fa4a41539b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:10.735 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.640+0000 7fa4ab3e9700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a4148ea0 con 0x7fa4a406e850 2022-06-27T12:11:10.735 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.640+0000 7fa4ab3e9700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a414b5f0 con 0x7fa4a40c34a0 2022-06-27T12:11:10.736 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.640+0000 7fa4ab3e9700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a41682b0 con 0x7fa4a4149300 2022-06-27T12:11:10.737 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.640+0000 7fa4a9185700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa4a406e850 0x7fa4a4168740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:10.737 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.641+0000 7fa4a9185700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa4a406e850 0x7fa4a4168740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50222/0 (socket says 172.21.15.195:50222) 2022-06-27T12:11:10.738 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.641+0000 7fa4a9185700 1 -- 172.21.15.195:0/575187692 learned_addr learned my addr 172.21.15.195:0/575187692 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:11:10.738 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.641+0000 7fa4a9986700 1 --2- 172.21.15.195:0/575187692 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa4a4149300 0x7fa4a41539b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:10.739 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.641+0000 7fa4a8984700 1 --2- 172.21.15.195:0/575187692 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa4a40c34a0 0x7fa4a4149d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:10.739 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.641+0000 7fa4a9185700 1 -- 172.21.15.195:0/575187692 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa4a4149300 msgr2=0x7fa4a41539b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:10.740 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.641+0000 7fa4a9185700 1 --2- 172.21.15.195:0/575187692 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa4a4149300 0x7fa4a41539b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:10.740 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.641+0000 7fa4a9185700 1 -- 172.21.15.195:0/575187692 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa4a40c34a0 msgr2=0x7fa4a4149d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:10.741 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.641+0000 7fa4a9185700 1 --2- 172.21.15.195:0/575187692 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa4a40c34a0 0x7fa4a4149d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:10.741 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.641+0000 7fa4a9185700 1 -- 172.21.15.195:0/575187692 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4a00092e0 con 0x7fa4a406e850 2022-06-27T12:11:10.741 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.641+0000 7fa4a8984700 1 --2- 172.21.15.195:0/575187692 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa4a40c34a0 0x7fa4a4149d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:11:10.742 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.641+0000 7fa4a9185700 1 --2- 172.21.15.195:0/575187692 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa4a406e850 0x7fa4a4168740 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fa4a40650f0 tx=0x7fa494017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:10.742 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.641+0000 7fa4917fa700 1 -- 172.21.15.195:0/575187692 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa494003730 con 0x7fa4a406e850 2022-06-27T12:11:10.743 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.641+0000 7fa4917fa700 1 -- 172.21.15.195:0/575187692 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa494003890 con 0x7fa4a406e850 2022-06-27T12:11:10.743 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.641+0000 7fa4917fa700 1 -- 172.21.15.195:0/575187692 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa494003c50 con 0x7fa4a406e850 2022-06-27T12:11:10.743 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.641+0000 7fa4ab3e9700 1 -- 172.21.15.195:0/575187692 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4940093a0 con 0x7fa4a406e850 2022-06-27T12:11:10.744 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.641+0000 7fa4ab3e9700 1 -- 172.21.15.195:0/575187692 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4a414b350 con 0x7fa4a406e850 2022-06-27T12:11:10.744 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.642+0000 7fa4ab3e9700 1 -- 172.21.15.195:0/575187692 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4a4168c80 con 0x7fa4a406e850 2022-06-27T12:11:10.744 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.642+0000 7fa4917fa700 1 -- 172.21.15.195:0/575187692 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fa49402b030 con 0x7fa4a406e850 2022-06-27T12:11:10.745 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.643+0000 7fa4917fa700 1 --2- 172.21.15.195:0/575187692 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa484063d00 0x7fa4840661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:10.766 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.643+0000 7fa4917fa700 1 -- 172.21.15.195:0/575187692 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7320+0+0 (secure 0 0 0) 0x7fa494082420 con 0x7fa4a406e850 2022-06-27T12:11:10.767 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.643+0000 7fa4a8984700 1 --2- 172.21.15.195:0/575187692 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa484063d00 0x7fa4840661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:10.767 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.643+0000 7fa4a8984700 1 --2- 172.21.15.195:0/575187692 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa484063d00 0x7fa4840661b0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fa4a00092b0 tx=0x7fa4a0008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:10.768 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.647+0000 7fa4917fa700 1 -- 172.21.15.195:0/575187692 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fa494050b20 con 0x7fa4a406e850 2022-06-27T12:11:10.891 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.884+0000 7fa4ab3e9700 1 -- 172.21.15.195:0/575187692 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa4a40722c0 con 0x7fa4a406e850 2022-06-27T12:11:10.892 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.885+0000 7fa4917fa700 1 -- 172.21.15.195:0/575187692 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+6128 (secure 0 0 0) 0x7fa494029090 con 0x7fa4a406e850 2022-06-27T12:11:10.893 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:11:10.893 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":90,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":6050,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/3890799919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3890799919},{"type":"v1","addr":"172.21.15.202:6833","nonce":3890799919}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":90}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:11:01.244537+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5933},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5933":{"gid":5933,"name":"d","rank":0,"incarnation":85,"state":"up:active","state_seq":17,"addr":"172.21.15.202:6835/1342896965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":1342896965},{"type":"v1","addr":"172.21.15.202:6835","nonce":1342896965}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:11:10.897 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.887+0000 7fa4ab3e9700 1 -- 172.21.15.195:0/575187692 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa484063d00 msgr2=0x7fa4840661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:10.897 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.887+0000 7fa4ab3e9700 1 --2- 172.21.15.195:0/575187692 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa484063d00 0x7fa4840661b0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fa4a00092b0 tx=0x7fa4a0008040 comp rx=0 tx=0).stop 2022-06-27T12:11:10.898 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.887+0000 7fa4ab3e9700 1 -- 172.21.15.195:0/575187692 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa4a406e850 msgr2=0x7fa4a4168740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:10.898 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.887+0000 7fa4ab3e9700 1 --2- 172.21.15.195:0/575187692 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa4a406e850 0x7fa4a4168740 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fa4a40650f0 tx=0x7fa494017000 comp rx=0 tx=0).stop 2022-06-27T12:11:10.899 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.887+0000 7fa4ab3e9700 1 -- 172.21.15.195:0/575187692 shutdown_connections 2022-06-27T12:11:10.899 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.887+0000 7fa4ab3e9700 1 --2- 172.21.15.195:0/575187692 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa484063d00 0x7fa4840661b0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:10.900 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.887+0000 7fa4ab3e9700 1 --2- 172.21.15.195:0/575187692 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa4a406e850 0x7fa4a4168740 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:10.900 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.887+0000 7fa4ab3e9700 1 --2- 172.21.15.195:0/575187692 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa4a4149300 0x7fa4a41539b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:10.901 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.887+0000 7fa4ab3e9700 1 --2- 172.21.15.195:0/575187692 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa4a40c34a0 0x7fa4a4149d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:10.902 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.887+0000 7fa4ab3e9700 1 -- 172.21.15.195:0/575187692 >> 172.21.15.195:0/575187692 conn(0x7fa4a40c6350 msgr2=0x7fa4a40c87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:11:10.902 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.887+0000 7fa4ab3e9700 1 -- 172.21.15.195:0/575187692 shutdown_connections 2022-06-27T12:11:10.903 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:10.887+0000 7fa4ab3e9700 1 -- 172.21.15.195:0/575187692 wait complete. 2022-06-27T12:11:10.903 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 90 2022-06-27T12:11:10.934 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.d (rank=0) 2022-06-27T12:11:10.935 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2022-06-27T12:11:10.935 INFO:teuthology.orchestra.run:waiting for 300 2022-06-27T12:11:10.950 INFO:tasks.ceph.mds.d:Stopped 2022-06-27T12:11:10.951 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:11:11.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.108+0000 7f743e412700 1 Processor -- start 2022-06-27T12:11:11.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.108+0000 7f743e412700 1 -- start start 2022-06-27T12:11:11.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.109+0000 7f743e412700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7438151560 0x7f74380c2120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:11.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.109+0000 7f743e412700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f7438154900 0x7f74380bf4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:11.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.109+0000 7f743e412700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7438154da0 con 0x7f7438151930 2022-06-27T12:11:11.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.109+0000 7f743e412700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74380c12a0 con 0x7f7438154900 2022-06-27T12:11:11.166 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.109+0000 7f743e412700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74380c1580 con 0x7f7438151560 2022-06-27T12:11:11.166 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.109+0000 7f74377fe700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f7438151930 0x7f74381541f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:56420/0 (socket says 172.21.15.195:56420) 2022-06-27T12:11:11.167 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.109+0000 7f74377fe700 1 -- 172.21.15.195:0/1926808509 learned_addr learned my addr 172.21.15.195:0/1926808509 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:11:11.167 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.109+0000 7f7436ffd700 1 -- 172.21.15.195:0/1926808509 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1453509075 0 0) 0x7f7438154da0 con 0x7f7438151930 2022-06-27T12:11:11.168 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.109+0000 7f7436ffd700 1 -- 172.21.15.195:0/1926808509 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7424003430 con 0x7f7438151930 2022-06-27T12:11:11.168 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.109+0000 7f743c9af700 1 --2- 172.21.15.195:0/1926808509 >> v2:172.21.15.202:3300/0 conn(0x7f7438154900 0x7f74380bf4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:11.168 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.109+0000 7f7437fff700 1 --2- 172.21.15.195:0/1926808509 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7438151560 0x7f74380c2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:11.175 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.109+0000 7f7436ffd700 1 -- 172.21.15.195:0/1926808509 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2431549960 0 0) 0x7f7424003430 con 0x7f7438151930 2022-06-27T12:11:11.176 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.109+0000 7f7436ffd700 1 -- 172.21.15.195:0/1926808509 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7438151560 msgr2=0x7f74380c2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:11:11.176 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.109+0000 7f7436ffd700 1 --2- 172.21.15.195:0/1926808509 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7438151560 0x7f74380c2120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:11.176 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.109+0000 7f7436ffd700 1 -- 172.21.15.195:0/1926808509 >> v2:172.21.15.202:3300/0 conn(0x7f7438154900 msgr2=0x7f74380bf4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:11:11.177 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.109+0000 7f7436ffd700 1 --2- 172.21.15.195:0/1926808509 >> v2:172.21.15.202:3300/0 conn(0x7f7438154900 0x7f74380bf4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:11.177 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.109+0000 7f7436ffd700 1 -- 172.21.15.195:0/1926808509 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74380c29f0 con 0x7f7438151930 2022-06-27T12:11:11.177 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.109+0000 7f7436ffd700 1 -- 172.21.15.195:0/1926808509 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7f7428001470 con 0x7f7438151930 2022-06-27T12:11:11.197 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.110+0000 7f7436ffd700 1 -- 172.21.15.195:0/1926808509 >> v1:172.21.15.195:6789/0 conn(0x7f7438151930 legacy=0x7f74381541f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:11.201 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.110+0000 7f7436ffd700 1 --2- 172.21.15.195:0/1926808509 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7438151560 0x7f7424003920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:11.202 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.110+0000 7f7436ffd700 1 --2- 172.21.15.195:0/1926808509 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7424003ea0 0x7f74240046d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:11.202 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.110+0000 7f7436ffd700 1 --2- 172.21.15.195:0/1926808509 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7424004c10 0x7f7424005080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:11.203 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.110+0000 7f7436ffd700 1 -- 172.21.15.195:0/1926808509 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74380c12a0 con 0x7f7424003ea0 2022-06-27T12:11:11.203 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.110+0000 7f7436ffd700 1 -- 172.21.15.195:0/1926808509 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74380c1580 con 0x7f7424004c10 2022-06-27T12:11:11.204 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.110+0000 7f7436ffd700 1 -- 172.21.15.195:0/1926808509 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f742400b950 con 0x7f7438151560 2022-06-27T12:11:11.204 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.110+0000 7f74377fe700 1 --2- 172.21.15.195:0/1926808509 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7424003ea0 0x7f74240046d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:11.204 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.110+0000 7f7437fff700 1 --2- 172.21.15.195:0/1926808509 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7438151560 0x7f7424003920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:11.205 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.110+0000 7f743c9af700 1 --2- 172.21.15.195:0/1926808509 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7424004c10 0x7f7424005080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:11.205 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.110+0000 7f74377fe700 1 -- 172.21.15.195:0/1926808509 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7438151560 msgr2=0x7f7424003920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:11:11.206 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.110+0000 7f74377fe700 1 --2- 172.21.15.195:0/1926808509 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7438151560 0x7f7424003920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:11.206 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.110+0000 7f74377fe700 1 -- 172.21.15.195:0/1926808509 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7424004c10 msgr2=0x7f7424005080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:11:11.207 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.110+0000 7f74377fe700 1 --2- 172.21.15.195:0/1926808509 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7424004c10 0x7f7424005080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:11.207 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.110+0000 7f74377fe700 1 -- 172.21.15.195:0/1926808509 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7424003430 con 0x7f7424003ea0 2022-06-27T12:11:11.208 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.110+0000 7f743c9af700 1 --2- 172.21.15.195:0/1926808509 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7424004c10 0x7f7424005080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:11:11.208 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.110+0000 7f74377fe700 1 --2- 172.21.15.195:0/1926808509 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7424003ea0 0x7f74240046d0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f742800c1e0 tx=0x7f7428001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62199218f1287173 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:11.209 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.110+0000 7f7436ffd700 1 -- 172.21.15.195:0/1926808509 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f742800cab0 con 0x7f7424003ea0 2022-06-27T12:11:11.209 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.110+0000 7f7436ffd700 1 -- 172.21.15.195:0/1926808509 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f742800cc10 con 0x7f7424003ea0 2022-06-27T12:11:11.210 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.110+0000 7f7436ffd700 1 -- 172.21.15.195:0/1926808509 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74280070d0 con 0x7f7424003ea0 2022-06-27T12:11:11.210 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.111+0000 7f743e412700 1 -- 172.21.15.195:0/1926808509 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7424003ea0 msgr2=0x7f74240046d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:11.210 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.111+0000 7f743e412700 1 --2- 172.21.15.195:0/1926808509 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7424003ea0 0x7f74240046d0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f742800c1e0 tx=0x7f7428001e70 comp rx=0 tx=0).stop 2022-06-27T12:11:11.211 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.111+0000 7f743c9af700 1 -- 172.21.15.195:0/1926808509 reap_dead start 2022-06-27T12:11:11.211 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.111+0000 7f743e412700 1 -- 172.21.15.195:0/1926808509 shutdown_connections 2022-06-27T12:11:11.212 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.111+0000 7f743e412700 1 -- 172.21.15.195:0/1926808509 >> 172.21.15.195:0/1926808509 conn(0x7f74380c6350 msgr2=0x7f74380c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:11:11.213 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.111+0000 7f743e412700 1 -- 172.21.15.195:0/1926808509 shutdown_connections 2022-06-27T12:11:11.213 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.111+0000 7f743e412700 1 -- 172.21.15.195:0/1926808509 wait complete. 2022-06-27T12:11:11.213 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.112+0000 7f743e412700 1 Processor -- start 2022-06-27T12:11:11.214 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.112+0000 7f743e412700 1 -- start start 2022-06-27T12:11:11.214 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.113+0000 7f743e412700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7424003ea0 0x7f7438153c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:11.215 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.113+0000 7f743e412700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7438153430 0x7f74380c0560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:11.215 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.113+0000 7f743e412700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f743814c260 0x7f743814c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:11.216 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.113+0000 7f743e412700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f743814cc10 con 0x7f743814c260 2022-06-27T12:11:11.216 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.113+0000 7f743e412700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74381677c0 con 0x7f7438153430 2022-06-27T12:11:11.216 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.113+0000 7f743e412700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7438167a80 con 0x7f7424003ea0 2022-06-27T12:11:11.217 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.113+0000 7f743c9af700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f743814c260 0x7f743814c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:11.217 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.113+0000 7f743c9af700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f743814c260 0x7f743814c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50244/0 (socket says 172.21.15.195:50244) 2022-06-27T12:11:11.218 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.113+0000 7f743c9af700 1 -- 172.21.15.195:0/4116719181 learned_addr learned my addr 172.21.15.195:0/4116719181 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:11:11.218 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.113+0000 7f7437fff700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7424003ea0 0x7f7438153c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:11.218 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.113+0000 7f74377fe700 1 --2- 172.21.15.195:0/4116719181 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7438153430 0x7f74380c0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:11.219 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.113+0000 7f743c9af700 1 -- 172.21.15.195:0/4116719181 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7424003ea0 msgr2=0x7f7438153c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:11.219 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.113+0000 7f743c9af700 1 --2- 172.21.15.195:0/4116719181 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7424003ea0 0x7f7438153c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:11.220 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.113+0000 7f743c9af700 1 -- 172.21.15.195:0/4116719181 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7438153430 msgr2=0x7f74380c0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:11.220 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.113+0000 7f743c9af700 1 --2- 172.21.15.195:0/4116719181 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7438153430 0x7f74380c0560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:11.221 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.113+0000 7f743c9af700 1 -- 172.21.15.195:0/4116719181 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7438167d60 con 0x7f743814c260 2022-06-27T12:11:11.221 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.113+0000 7f743c9af700 1 --2- 172.21.15.195:0/4116719181 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f743814c260 0x7f743814c6d0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f74380bd260 tx=0x7f743000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:11.221 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.113+0000 7f741ffff700 1 -- 172.21.15.195:0/4116719181 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f743000e050 con 0x7f743814c260 2022-06-27T12:11:11.222 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.113+0000 7f741ffff700 1 -- 172.21.15.195:0/4116719181 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f743000f040 con 0x7f743814c260 2022-06-27T12:11:11.222 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.113+0000 7f743e412700 1 -- 172.21.15.195:0/4116719181 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7438168040 con 0x7f743814c260 2022-06-27T12:11:11.223 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.113+0000 7f741ffff700 1 -- 172.21.15.195:0/4116719181 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7430010d40 con 0x7f743814c260 2022-06-27T12:11:11.223 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.114+0000 7f743e412700 1 -- 172.21.15.195:0/4116719181 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74380c3500 con 0x7f743814c260 2022-06-27T12:11:11.224 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.114+0000 7f743e412700 1 -- 172.21.15.195:0/4116719181 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7438152120 con 0x7f743814c260 2022-06-27T12:11:11.224 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.115+0000 7f741ffff700 1 -- 172.21.15.195:0/4116719181 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f7430015440 con 0x7f743814c260 2022-06-27T12:11:11.224 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.115+0000 7f741ffff700 1 --2- 172.21.15.195:0/4116719181 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7418063d30 0x7f74180661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:11.225 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.115+0000 7f741ffff700 1 -- 172.21.15.195:0/4116719181 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7320+0+0 (secure 0 0 0) 0x7f7430081140 con 0x7f743814c260 2022-06-27T12:11:11.225 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.115+0000 7f7437fff700 1 --2- 172.21.15.195:0/4116719181 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7418063d30 0x7f74180661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:11.226 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.115+0000 7f7437fff700 1 --2- 172.21.15.195:0/4116719181 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7418063d30 0x7f74180661e0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f742c006dc0 tx=0x7f742c009060 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:11.226 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.119+0000 7f741ffff700 1 -- 172.21.15.195:0/4116719181 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f7438152120 con 0x7f743814c260 2022-06-27T12:11:11.356 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.355+0000 7f743e412700 1 -- 172.21.15.195:0/4116719181 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7438152120 con 0x7f743814c260 2022-06-27T12:11:11.356 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.355+0000 7f741ffff700 1 -- 172.21.15.195:0/4116719181 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+6128 (secure 0 0 0) 0x7f7438152120 con 0x7f743814c260 2022-06-27T12:11:11.357 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:11:11.358 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":90,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":6050,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/3890799919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3890799919},{"type":"v1","addr":"172.21.15.202:6833","nonce":3890799919}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":90}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:11:01.244537+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5933},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5933":{"gid":5933,"name":"d","rank":0,"incarnation":85,"state":"up:active","state_seq":17,"addr":"172.21.15.202:6835/1342896965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":1342896965},{"type":"v1","addr":"172.21.15.202:6835","nonce":1342896965}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:11:11.360 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.357+0000 7f743e412700 1 -- 172.21.15.195:0/4116719181 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7418063d30 msgr2=0x7f74180661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:11.360 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.357+0000 7f743e412700 1 --2- 172.21.15.195:0/4116719181 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7418063d30 0x7f74180661e0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f742c006dc0 tx=0x7f742c009060 comp rx=0 tx=0).stop 2022-06-27T12:11:11.361 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.357+0000 7f743e412700 1 -- 172.21.15.195:0/4116719181 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f743814c260 msgr2=0x7f743814c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:11.361 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.357+0000 7f743e412700 1 --2- 172.21.15.195:0/4116719181 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f743814c260 0x7f743814c6d0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f74380bd260 tx=0x7f743000b040 comp rx=0 tx=0).stop 2022-06-27T12:11:11.361 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.357+0000 7f743e412700 1 -- 172.21.15.195:0/4116719181 shutdown_connections 2022-06-27T12:11:11.361 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.357+0000 7f743e412700 1 --2- 172.21.15.195:0/4116719181 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7418063d30 0x7f74180661e0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:11.361 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.357+0000 7f743e412700 1 --2- 172.21.15.195:0/4116719181 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f743814c260 0x7f743814c6d0 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:11.362 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.357+0000 7f743e412700 1 --2- 172.21.15.195:0/4116719181 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7424003ea0 0x7f7438153c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:11.362 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.357+0000 7f743e412700 1 --2- 172.21.15.195:0/4116719181 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7438153430 0x7f74380c0560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:11.363 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.357+0000 7f743e412700 1 -- 172.21.15.195:0/4116719181 >> 172.21.15.195:0/4116719181 conn(0x7f74380c6350 msgr2=0x7f74380c76c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:11:11.363 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.358+0000 7f743e412700 1 -- 172.21.15.195:0/4116719181 shutdown_connections 2022-06-27T12:11:11.364 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:11.358+0000 7f743e412700 1 -- 172.21.15.195:0/4116719181 wait complete. 2022-06-27T12:11:11.364 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 90 2022-06-27T12:11:11.371 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-06-27T12:11:13.373 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:11:13.517 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.507+0000 7f16c378c700 1 Processor -- start 2022-06-27T12:11:13.518 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.507+0000 7f16c378c700 1 -- start start 2022-06-27T12:11:13.519 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.508+0000 7f16c378c700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f16bc06d050 0x7f16bc15a890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:13.519 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.508+0000 7f16c378c700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16bc15ae90 con 0x7f16bc06a690 2022-06-27T12:11:13.520 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.508+0000 7f16c378c700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16bc15b170 con 0x7f16bc06aa60 2022-06-27T12:11:13.520 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.508+0000 7f16c378c700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16bc069be0 con 0x7f16bc06d050 2022-06-27T12:11:13.521 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.508+0000 7f16c1528700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f16bc06a690 0x7f16bc0c7c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:56442/0 (socket says 172.21.15.195:56442) 2022-06-27T12:11:13.521 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.508+0000 7f16c1528700 1 -- 172.21.15.195:0/3269533987 learned_addr learned my addr 172.21.15.195:0/3269533987 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:11:13.522 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.509+0000 7f16c0d27700 1 --2- 172.21.15.195:0/3269533987 >> v2:172.21.15.202:3300/0 conn(0x7f16bc06d050 0x7f16bc15a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:13.523 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.509+0000 7f16abfff700 1 -- 172.21.15.195:0/3269533987 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 530621470 0 0) 0x7f16bc15ae90 con 0x7f16bc06a690 2022-06-27T12:11:13.523 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.509+0000 7f16abfff700 1 -- 172.21.15.195:0/3269533987 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16a4003430 con 0x7f16bc06a690 2022-06-27T12:11:13.538 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.509+0000 7f16c0d27700 1 -- 172.21.15.195:0/3269533987 >> v1:172.21.15.202:6789/0 conn(0x7f16bc06aa60 legacy=0x7f16bc154700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:13.538 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.509+0000 7f16c0d27700 1 -- 172.21.15.195:0/3269533987 >> v1:172.21.15.195:6789/0 conn(0x7f16bc06a690 legacy=0x7f16bc0c7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:13.539 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.509+0000 7f16c0d27700 1 -- 172.21.15.195:0/3269533987 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16bc15b450 con 0x7f16bc06d050 2022-06-27T12:11:13.539 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.509+0000 7f16c0d27700 1 --2- 172.21.15.195:0/3269533987 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f16bc06d050 0x7f16bc15a890 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f16b80097d0 tx=0x7f16b8009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f52361dbf7570dd server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:13.540 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.509+0000 7f16abfff700 1 -- 172.21.15.195:0/3269533987 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b8011630 con 0x7f16bc06d050 2022-06-27T12:11:13.540 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.509+0000 7f16abfff700 1 -- 172.21.15.195:0/3269533987 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16b8011790 con 0x7f16bc06d050 2022-06-27T12:11:13.541 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.509+0000 7f16abfff700 1 -- 172.21.15.195:0/3269533987 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b8011b50 con 0x7f16bc06d050 2022-06-27T12:11:13.541 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.510+0000 7f16c378c700 1 -- 172.21.15.195:0/3269533987 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f16bc06d050 msgr2=0x7f16bc15a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:13.542 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.510+0000 7f16c378c700 1 --2- 172.21.15.195:0/3269533987 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f16bc06d050 0x7f16bc15a890 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f16b80097d0 tx=0x7f16b8009060 comp rx=0 tx=0).stop 2022-06-27T12:11:13.543 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.510+0000 7f16c378c700 1 -- 172.21.15.195:0/3269533987 shutdown_connections 2022-06-27T12:11:13.543 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.510+0000 7f16c378c700 1 --2- 172.21.15.195:0/3269533987 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f16bc06d050 0x7f16bc15a890 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:13.544 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.510+0000 7f16c378c700 1 -- 172.21.15.195:0/3269533987 >> 172.21.15.195:0/3269533987 conn(0x7f16bc063af0 msgr2=0x7f16bc063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:11:13.544 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.510+0000 7f16c378c700 1 -- 172.21.15.195:0/3269533987 shutdown_connections 2022-06-27T12:11:13.544 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.510+0000 7f16c378c700 1 -- 172.21.15.195:0/3269533987 wait complete. 2022-06-27T12:11:13.545 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.511+0000 7f16c378c700 1 Processor -- start 2022-06-27T12:11:13.545 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.511+0000 7f16c378c700 1 -- start start 2022-06-27T12:11:13.556 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.512+0000 7f16c378c700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f16bc06a690 0x7f16bc0c6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:13.557 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.512+0000 7f16c378c700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f16bc06aa60 0x7f16bc0c21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:13.557 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.512+0000 7f16c378c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f16bc06d050 0x7f16bc0c26e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:13.558 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.512+0000 7f16c378c700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16bc0c1d10 con 0x7f16bc06d050 2022-06-27T12:11:13.558 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.512+0000 7f16c378c700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16bc16c2a0 con 0x7f16bc06a690 2022-06-27T12:11:13.559 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.512+0000 7f16c378c700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16bc16c540 con 0x7f16bc06aa60 2022-06-27T12:11:13.559 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.512+0000 7f16c1d29700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f16bc06d050 0x7f16bc0c26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:13.559 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.512+0000 7f16c1d29700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f16bc06d050 0x7f16bc0c26e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50262/0 (socket says 172.21.15.195:50262) 2022-06-27T12:11:13.560 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.512+0000 7f16c1d29700 1 -- 172.21.15.195:0/2927209270 learned_addr learned my addr 172.21.15.195:0/2927209270 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:11:13.560 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.512+0000 7f16c1528700 1 --2- 172.21.15.195:0/2927209270 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f16bc06a690 0x7f16bc0c6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:13.561 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.512+0000 7f16c0d27700 1 --2- 172.21.15.195:0/2927209270 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f16bc06aa60 0x7f16bc0c21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:13.561 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.512+0000 7f16c1d29700 1 -- 172.21.15.195:0/2927209270 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f16bc06aa60 msgr2=0x7f16bc0c21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:13.561 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.512+0000 7f16c1d29700 1 --2- 172.21.15.195:0/2927209270 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f16bc06aa60 0x7f16bc0c21a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:13.562 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.512+0000 7f16c1d29700 1 -- 172.21.15.195:0/2927209270 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f16bc06a690 msgr2=0x7f16bc0c6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:13.562 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.512+0000 7f16c1d29700 1 --2- 172.21.15.195:0/2927209270 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f16bc06a690 0x7f16bc0c6180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:13.563 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.513+0000 7f16c1d29700 1 -- 172.21.15.195:0/2927209270 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16b80092e0 con 0x7f16bc06d050 2022-06-27T12:11:13.563 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.513+0000 7f16c1528700 1 --2- 172.21.15.195:0/2927209270 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f16bc06a690 0x7f16bc0c6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:11:13.564 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.513+0000 7f16c1d29700 1 --2- 172.21.15.195:0/2927209270 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f16bc06d050 0x7f16bc0c26e0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f16b400c7c0 tx=0x7f16b40015e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:13.564 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.513+0000 7f16a97fa700 1 -- 172.21.15.195:0/2927209270 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b4003880 con 0x7f16bc06d050 2022-06-27T12:11:13.566 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.513+0000 7f16a97fa700 1 -- 172.21.15.195:0/2927209270 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16b40039e0 con 0x7f16bc06d050 2022-06-27T12:11:13.567 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.513+0000 7f16c378c700 1 -- 172.21.15.195:0/2927209270 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16bc0c38e0 con 0x7f16bc06d050 2022-06-27T12:11:13.567 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.513+0000 7f16a97fa700 1 -- 172.21.15.195:0/2927209270 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b4003d60 con 0x7f16bc06d050 2022-06-27T12:11:13.567 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.513+0000 7f16c378c700 1 -- 172.21.15.195:0/2927209270 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16bc0c3ea0 con 0x7f16bc06d050 2022-06-27T12:11:13.568 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.514+0000 7f16c378c700 1 -- 172.21.15.195:0/2927209270 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16bc0c1ec0 con 0x7f16bc06d050 2022-06-27T12:11:13.569 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.514+0000 7f16a97fa700 1 -- 172.21.15.195:0/2927209270 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f16b4010050 con 0x7f16bc06d050 2022-06-27T12:11:13.569 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.514+0000 7f16a97fa700 1 --2- 172.21.15.195:0/2927209270 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f169c063d00 0x7f169c0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:13.570 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.515+0000 7f16a97fa700 1 -- 172.21.15.195:0/2927209270 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7320+0+0 (secure 0 0 0) 0x7f16b4080880 con 0x7f16bc06d050 2022-06-27T12:11:13.571 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.515+0000 7f16c1528700 1 --2- 172.21.15.195:0/2927209270 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f169c063d00 0x7f169c0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:13.571 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.515+0000 7f16c1528700 1 --2- 172.21.15.195:0/2927209270 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f169c063d00 0x7f169c0661b0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f16ac005790 tx=0x7f16ac009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:13.572 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.519+0000 7f16a97fa700 1 -- 172.21.15.195:0/2927209270 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f16bc0c1ec0 con 0x7f16bc06d050 2022-06-27T12:11:13.742 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.742+0000 7f16c378c700 1 -- 172.21.15.195:0/2927209270 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f16bc0c1ec0 con 0x7f16bc06d050 2022-06-27T12:11:13.743 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.742+0000 7f16a97fa700 1 -- 172.21.15.195:0/2927209270 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+6128 (secure 0 0 0) 0x7f16bc0c1ec0 con 0x7f16bc06d050 2022-06-27T12:11:13.744 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:11:13.745 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":90,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":6050,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/3890799919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3890799919},{"type":"v1","addr":"172.21.15.202:6833","nonce":3890799919}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":90}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:11:01.244537+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5933},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5933":{"gid":5933,"name":"d","rank":0,"incarnation":85,"state":"up:active","state_seq":17,"addr":"172.21.15.202:6835/1342896965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":1342896965},{"type":"v1","addr":"172.21.15.202:6835","nonce":1342896965}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:11:13.747 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.744+0000 7f16c378c700 1 -- 172.21.15.195:0/2927209270 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f169c063d00 msgr2=0x7f169c0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:13.747 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.744+0000 7f16c378c700 1 --2- 172.21.15.195:0/2927209270 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f169c063d00 0x7f169c0661b0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f16ac005790 tx=0x7f16ac009040 comp rx=0 tx=0).stop 2022-06-27T12:11:13.748 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.744+0000 7f16c378c700 1 -- 172.21.15.195:0/2927209270 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f16bc06d050 msgr2=0x7f16bc0c26e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:13.748 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.744+0000 7f16c378c700 1 --2- 172.21.15.195:0/2927209270 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f16bc06d050 0x7f16bc0c26e0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f16b400c7c0 tx=0x7f16b40015e0 comp rx=0 tx=0).stop 2022-06-27T12:11:13.749 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.744+0000 7f16c378c700 1 -- 172.21.15.195:0/2927209270 shutdown_connections 2022-06-27T12:11:13.749 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.744+0000 7f16c378c700 1 --2- 172.21.15.195:0/2927209270 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f169c063d00 0x7f169c0661b0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:13.763 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.744+0000 7f16c378c700 1 --2- 172.21.15.195:0/2927209270 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f16bc06d050 0x7f16bc0c26e0 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:13.764 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.744+0000 7f16c378c700 1 --2- 172.21.15.195:0/2927209270 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f16bc06a690 0x7f16bc0c6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:13.764 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.744+0000 7f16c378c700 1 --2- 172.21.15.195:0/2927209270 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f16bc06aa60 0x7f16bc0c21a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:13.764 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.744+0000 7f16c378c700 1 -- 172.21.15.195:0/2927209270 >> 172.21.15.195:0/2927209270 conn(0x7f16bc063af0 msgr2=0x7f16bc071200 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:11:13.764 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.744+0000 7f16c378c700 1 -- 172.21.15.195:0/2927209270 shutdown_connections 2022-06-27T12:11:13.765 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:13.744+0000 7f16c378c700 1 -- 172.21.15.195:0/2927209270 wait complete. 2022-06-27T12:11:13.765 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 90 2022-06-27T12:11:13.768 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-06-27T12:11:15.774 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:11:15.912 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:15.906+0000 7f2e513e2700 1 Processor -- start 2022-06-27T12:11:15.913 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:15.906+0000 7f2e513e2700 1 -- start start 2022-06-27T12:11:15.913 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:15.907+0000 7f2e513e2700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f2e4c0c40d0 0x7f2e4c157010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:15.914 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:15.907+0000 7f2e513e2700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f2e4c0c6190 0x7f2e4c067ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:15.915 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:15.907+0000 7f2e513e2700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e4c156b80 con 0x7f2e4c0c44a0 2022-06-27T12:11:15.915 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:15.907+0000 7f2e513e2700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e4c06a040 con 0x7f2e4c0c6190 2022-06-27T12:11:15.916 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:15.907+0000 7f2e513e2700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e4c06a300 con 0x7f2e4c0c40d0 2022-06-27T12:11:15.916 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:15.907+0000 7f2e4a7fc700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f2e4c0c6190 0x7f2e4c067ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:15.916 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:15.907+0000 7f2e4b7fe700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f2e4c0c44a0 0x7f2e4c14cb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:56462/0 (socket says 172.21.15.195:56462) 2022-06-27T12:11:15.917 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:15.907+0000 7f2e4b7fe700 1 -- 172.21.15.195:0/4235742522 learned_addr learned my addr 172.21.15.195:0/4235742522 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:11:15.917 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:15.907+0000 7f2e4affd700 1 --2- 172.21.15.195:0/4235742522 >> v2:172.21.15.202:3300/0 conn(0x7f2e4c0c40d0 0x7f2e4c157010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:15.918 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:15.907+0000 7f2e4a7fc700 1 -- 172.21.15.195:0/4235742522 >> v2:172.21.15.202:3300/0 conn(0x7f2e4c0c40d0 msgr2=0x7f2e4c157010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:11:15.918 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:15.907+0000 7f2e4a7fc700 1 --2- 172.21.15.195:0/4235742522 >> v2:172.21.15.202:3300/0 conn(0x7f2e4c0c40d0 0x7f2e4c157010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:15.918 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:15.908+0000 7f2e4a7fc700 1 -- 172.21.15.195:0/4235742522 >> v1:172.21.15.195:6789/0 conn(0x7f2e4c0c44a0 legacy=0x7f2e4c14cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:15.919 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:15.908+0000 7f2e49ffb700 1 -- 172.21.15.195:0/4235742522 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 69629858 0 0) 0x7f2e4c156b80 con 0x7f2e4c0c44a0 2022-06-27T12:11:15.919 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:15.908+0000 7f2e4a7fc700 1 -- 172.21.15.195:0/4235742522 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e4c06a5e0 con 0x7f2e4c0c6190 2022-06-27T12:11:15.919 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:15.908+0000 7f2e4a7fc700 1 --2- 172.21.15.195:0/4235742522 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2e4c0c6190 0x7f2e4c067ed0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c0cdb00 tx=0x7f2e3c009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8411169feb867b9d server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:15.920 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:15.908+0000 7f2e49ffb700 1 -- 172.21.15.195:0/4235742522 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e3c011630 con 0x7f2e4c0c6190 2022-06-27T12:11:15.920 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:15.908+0000 7f2e49ffb700 1 -- 172.21.15.195:0/4235742522 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e3c011790 con 0x7f2e4c0c6190 2022-06-27T12:11:15.921 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:15.908+0000 7f2e49ffb700 1 -- 172.21.15.195:0/4235742522 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e3c011b30 con 0x7f2e4c0c6190 2022-06-27T12:11:15.921 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:15.908+0000 7f2e513e2700 1 -- 172.21.15.195:0/4235742522 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2e4c0c6190 msgr2=0x7f2e4c067ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:15.921 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:15.908+0000 7f2e513e2700 1 --2- 172.21.15.195:0/4235742522 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2e4c0c6190 0x7f2e4c067ed0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c0cdb00 tx=0x7f2e3c009060 comp rx=0 tx=0).stop 2022-06-27T12:11:15.922 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:15.908+0000 7f2e513e2700 1 -- 172.21.15.195:0/4235742522 shutdown_connections 2022-06-27T12:11:15.922 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:15.908+0000 7f2e513e2700 1 --2- 172.21.15.195:0/4235742522 >> v2:172.21.15.202:3300/0 conn(0x7f2e4c0c40d0 0x7f2e4c157010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:15.923 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:15.908+0000 7f2e513e2700 1 --2- 172.21.15.195:0/4235742522 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2e4c0c6190 0x7f2e4c067ed0 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:15.923 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:15.908+0000 7f2e513e2700 1 -- 172.21.15.195:0/4235742522 >> 172.21.15.195:0/4235742522 conn(0x7f2e4c0ccb80 msgr2=0x7f2e4c0ccf80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:11:15.924 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:15.908+0000 7f2e513e2700 1 -- 172.21.15.195:0/4235742522 shutdown_connections 2022-06-27T12:11:15.924 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:15.908+0000 7f2e513e2700 1 -- 172.21.15.195:0/4235742522 wait complete. 2022-06-27T12:11:15.925 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:15.910+0000 7f2e513e2700 1 Processor -- start 2022-06-27T12:11:15.925 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:15.910+0000 7f2e513e2700 1 -- start start 2022-06-27T12:11:15.932 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:15.910+0000 7f2e513e2700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2e4c0c40d0 0x7f2e4c1642c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:15.932 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:15.910+0000 7f2e513e2700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2e4c0c44a0 0x7f2e4c14f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:15.933 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:15.910+0000 7f2e513e2700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2e4c0c6190 0x7f2e4c169830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:15.933 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:15.910+0000 7f2e513e2700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e4c16a680 con 0x7f2e4c0c40d0 2022-06-27T12:11:15.934 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:15.910+0000 7f2e513e2700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e4c158df0 con 0x7f2e4c0c6190 2022-06-27T12:11:15.934 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:15.910+0000 7f2e513e2700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e4c159280 con 0x7f2e4c0c44a0 2022-06-27T12:11:15.934 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:15.911+0000 7f2e4affd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2e4c0c40d0 0x7f2e4c1642c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:15.935 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:15.911+0000 7f2e4affd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2e4c0c40d0 0x7f2e4c1642c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50276/0 (socket says 172.21.15.195:50276) 2022-06-27T12:11:15.935 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:15.911+0000 7f2e4affd700 1 -- 172.21.15.195:0/104349097 learned_addr learned my addr 172.21.15.195:0/104349097 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:11:15.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:15.911+0000 7f2e4b7fe700 1 --2- 172.21.15.195:0/104349097 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2e4c0c6190 0x7f2e4c169830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:15.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:15.911+0000 7f2e4affd700 1 -- 172.21.15.195:0/104349097 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2e4c0c44a0 msgr2=0x7f2e4c14f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:15.937 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:15.911+0000 7f2e4a7fc700 1 --2- 172.21.15.195:0/104349097 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2e4c0c44a0 0x7f2e4c14f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:15.937 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:15.911+0000 7f2e4affd700 1 --2- 172.21.15.195:0/104349097 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2e4c0c44a0 0x7f2e4c14f980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:15.938 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:15.911+0000 7f2e4affd700 1 -- 172.21.15.195:0/104349097 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2e4c0c6190 msgr2=0x7f2e4c169830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:15.939 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:15.911+0000 7f2e4affd700 1 --2- 172.21.15.195:0/104349097 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2e4c0c6190 0x7f2e4c169830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:15.939 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:15.911+0000 7f2e4affd700 1 -- 172.21.15.195:0/104349097 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e3c0092e0 con 0x7f2e4c0c40d0 2022-06-27T12:11:15.939 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:15.911+0000 7f2e4affd700 1 --2- 172.21.15.195:0/104349097 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2e4c0c40d0 0x7f2e4c1642c0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f2e4000d860 tx=0x7f2e40008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:15.940 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:15.911+0000 7f2e32ffd700 1 -- 172.21.15.195:0/104349097 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e40014620 con 0x7f2e4c0c40d0 2022-06-27T12:11:15.940 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:15.911+0000 7f2e32ffd700 1 -- 172.21.15.195:0/104349097 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e40014780 con 0x7f2e4c0c40d0 2022-06-27T12:11:15.941 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:15.911+0000 7f2e513e2700 1 -- 172.21.15.195:0/104349097 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e4c158220 con 0x7f2e4c0c40d0 2022-06-27T12:11:15.941 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:15.911+0000 7f2e32ffd700 1 -- 172.21.15.195:0/104349097 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e40014b20 con 0x7f2e4c0c40d0 2022-06-27T12:11:15.942 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:15.911+0000 7f2e513e2700 1 -- 172.21.15.195:0/104349097 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e4c158820 con 0x7f2e4c0c40d0 2022-06-27T12:11:15.942 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:15.912+0000 7f2e513e2700 1 -- 172.21.15.195:0/104349097 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e4c1590d0 con 0x7f2e4c0c40d0 2022-06-27T12:11:15.942 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:15.913+0000 7f2e32ffd700 1 -- 172.21.15.195:0/104349097 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f2e40024070 con 0x7f2e4c0c40d0 2022-06-27T12:11:15.943 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:15.913+0000 7f2e32ffd700 1 --2- 172.21.15.195:0/104349097 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2e2c063d00 0x7f2e2c0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:15.943 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:15.913+0000 7f2e32ffd700 1 -- 172.21.15.195:0/104349097 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7320+0+0 (secure 0 0 0) 0x7f2e40081480 con 0x7f2e4c0c40d0 2022-06-27T12:11:15.943 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:15.913+0000 7f2e4a7fc700 1 --2- 172.21.15.195:0/104349097 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2e2c063d00 0x7f2e2c0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:15.944 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:15.913+0000 7f2e4a7fc700 1 --2- 172.21.15.195:0/104349097 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2e2c063d00 0x7f2e2c0661b0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c069490 tx=0x7f2e3c017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:15.944 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:15.917+0000 7f2e32ffd700 1 -- 172.21.15.195:0/104349097 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f2e4004fb30 con 0x7f2e4c0c40d0 2022-06-27T12:11:16.143 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:16.143+0000 7f2e513e2700 1 -- 172.21.15.195:0/104349097 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2e4c0722c0 con 0x7f2e4c0c40d0 2022-06-27T12:11:16.144 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:16.143+0000 7f2e32ffd700 1 -- 172.21.15.195:0/104349097 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+6128 (secure 0 0 0) 0x7f2e40053150 con 0x7f2e4c0c40d0 2022-06-27T12:11:16.145 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:11:16.145 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":90,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":6050,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/3890799919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3890799919},{"type":"v1","addr":"172.21.15.202:6833","nonce":3890799919}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":90}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:11:01.244537+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5933},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5933":{"gid":5933,"name":"d","rank":0,"incarnation":85,"state":"up:active","state_seq":17,"addr":"172.21.15.202:6835/1342896965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":1342896965},{"type":"v1","addr":"172.21.15.202:6835","nonce":1342896965}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:11:16.148 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:16.145+0000 7f2e513e2700 1 -- 172.21.15.195:0/104349097 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2e2c063d00 msgr2=0x7f2e2c0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:16.148 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:16.145+0000 7f2e513e2700 1 --2- 172.21.15.195:0/104349097 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2e2c063d00 0x7f2e2c0661b0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c069490 tx=0x7f2e3c017000 comp rx=0 tx=0).stop 2022-06-27T12:11:16.149 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:16.145+0000 7f2e513e2700 1 -- 172.21.15.195:0/104349097 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2e4c0c40d0 msgr2=0x7f2e4c1642c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:16.149 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:16.145+0000 7f2e513e2700 1 --2- 172.21.15.195:0/104349097 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2e4c0c40d0 0x7f2e4c1642c0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f2e4000d860 tx=0x7f2e40008040 comp rx=0 tx=0).stop 2022-06-27T12:11:16.150 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:16.145+0000 7f2e513e2700 1 -- 172.21.15.195:0/104349097 shutdown_connections 2022-06-27T12:11:16.150 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:16.145+0000 7f2e513e2700 1 --2- 172.21.15.195:0/104349097 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2e2c063d00 0x7f2e2c0661b0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:16.151 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:16.145+0000 7f2e513e2700 1 --2- 172.21.15.195:0/104349097 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2e4c0c40d0 0x7f2e4c1642c0 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:16.151 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:16.145+0000 7f2e513e2700 1 --2- 172.21.15.195:0/104349097 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2e4c0c6190 0x7f2e4c169830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:16.151 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:16.145+0000 7f2e513e2700 1 --2- 172.21.15.195:0/104349097 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2e4c0c44a0 0x7f2e4c14f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:16.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:16.145+0000 7f2e513e2700 1 -- 172.21.15.195:0/104349097 >> 172.21.15.195:0/104349097 conn(0x7f2e4c0ccb80 msgr2=0x7f2e4c0c9600 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:11:16.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:16.145+0000 7f2e513e2700 1 -- 172.21.15.195:0/104349097 shutdown_connections 2022-06-27T12:11:16.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:16.145+0000 7f2e513e2700 1 -- 172.21.15.195:0/104349097 wait complete. 2022-06-27T12:11:16.153 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 90 2022-06-27T12:11:16.158 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-06-27T12:11:18.163 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:11:18.339 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.314+0000 7f9119dee700 1 Processor -- start 2022-06-27T12:11:18.339 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.314+0000 7f9119dee700 1 -- start start 2022-06-27T12:11:18.340 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.315+0000 7f9119dee700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f911406f640 0x7f9114156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:18.340 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.315+0000 7f9119dee700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91140c1a20 con 0x7f91140c4160 2022-06-27T12:11:18.340 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.315+0000 7f9119dee700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91140c1130 con 0x7f91140c39e0 2022-06-27T12:11:18.341 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.315+0000 7f9119dee700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f911406ea90 con 0x7f911406f640 2022-06-27T12:11:18.341 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.315+0000 7f91137fe700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f911406f640 0x7f9114156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:18.342 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.315+0000 7f9112ffd700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f91140c4160 0x7f91140c1eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:56472/0 (socket says 172.21.15.195:56472) 2022-06-27T12:11:18.342 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.315+0000 7f9112ffd700 1 -- 172.21.15.195:0/3135159698 learned_addr learned my addr 172.21.15.195:0/3135159698 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:11:18.343 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.315+0000 7f91137fe700 1 -- 172.21.15.195:0/3135159698 >> v1:172.21.15.202:6789/0 conn(0x7f91140c39e0 legacy=0x7f91141535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:11:18.343 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.315+0000 7f91137fe700 1 -- 172.21.15.195:0/3135159698 >> v1:172.21.15.195:6789/0 conn(0x7f91140c4160 legacy=0x7f91140c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:18.344 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.315+0000 7f91137fe700 1 -- 172.21.15.195:0/3135159698 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f911406ed70 con 0x7f911406f640 2022-06-27T12:11:18.344 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.316+0000 7f91137fe700 1 --2- 172.21.15.195:0/3135159698 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f911406f640 0x7f9114156ed0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f90fc0097a0 tx=0x7f90fc009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6552d0ad2aba892f server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:18.345 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.316+0000 7f91127fc700 1 -- 172.21.15.195:0/3135159698 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90fc004030 con 0x7f911406f640 2022-06-27T12:11:18.345 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.316+0000 7f91127fc700 1 -- 172.21.15.195:0/3135159698 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90fc004190 con 0x7f911406f640 2022-06-27T12:11:18.346 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.316+0000 7f91127fc700 1 -- 172.21.15.195:0/3135159698 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90fc011630 con 0x7f911406f640 2022-06-27T12:11:18.346 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.316+0000 7f9119dee700 1 -- 172.21.15.195:0/3135159698 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f911406f640 msgr2=0x7f9114156ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:18.347 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.316+0000 7f9119dee700 1 --2- 172.21.15.195:0/3135159698 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f911406f640 0x7f9114156ed0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f90fc0097a0 tx=0x7f90fc009060 comp rx=0 tx=0).stop 2022-06-27T12:11:18.347 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.316+0000 7f9119dee700 1 -- 172.21.15.195:0/3135159698 shutdown_connections 2022-06-27T12:11:18.347 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.316+0000 7f9119dee700 1 --2- 172.21.15.195:0/3135159698 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f911406f640 0x7f9114156ed0 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:18.348 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.316+0000 7f9119dee700 1 -- 172.21.15.195:0/3135159698 >> 172.21.15.195:0/3135159698 conn(0x7f91140c6350 msgr2=0x7f91140c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:11:18.348 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.316+0000 7f9119dee700 1 -- 172.21.15.195:0/3135159698 shutdown_connections 2022-06-27T12:11:18.348 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.316+0000 7f9119dee700 1 -- 172.21.15.195:0/3135159698 wait complete. 2022-06-27T12:11:18.349 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.318+0000 7f9119dee700 1 Processor -- start 2022-06-27T12:11:18.349 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.318+0000 7f9119dee700 1 -- start start 2022-06-27T12:11:18.350 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.318+0000 7f9119dee700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f911406f640 0x7f91140c0180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:18.350 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.318+0000 7f9119dee700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f91140c39e0 0x7f91140bc6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:18.350 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.318+0000 7f9119dee700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f91140c4160 0x7f9114166c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:18.351 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.318+0000 7f9119dee700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f911406b200 con 0x7f91140c4160 2022-06-27T12:11:18.351 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.318+0000 7f9119dee700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91140bfd10 con 0x7f911406f640 2022-06-27T12:11:18.351 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.318+0000 7f9119dee700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91140bf680 con 0x7f91140c39e0 2022-06-27T12:11:18.352 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.319+0000 7f9113fff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f91140c4160 0x7f9114166c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:18.352 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.319+0000 7f9113fff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f91140c4160 0x7f9114166c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50294/0 (socket says 172.21.15.195:50294) 2022-06-27T12:11:18.353 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.319+0000 7f9113fff700 1 -- 172.21.15.195:0/3688843865 learned_addr learned my addr 172.21.15.195:0/3688843865 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:11:18.353 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.319+0000 7f9112ffd700 1 --2- 172.21.15.195:0/3688843865 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f91140c39e0 0x7f91140bc6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:18.354 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.319+0000 7f91137fe700 1 --2- 172.21.15.195:0/3688843865 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f911406f640 0x7f91140c0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:18.354 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.319+0000 7f9113fff700 1 -- 172.21.15.195:0/3688843865 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f91140c39e0 msgr2=0x7f91140bc6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:18.355 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.319+0000 7f9113fff700 1 --2- 172.21.15.195:0/3688843865 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f91140c39e0 0x7f91140bc6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:18.355 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.319+0000 7f9113fff700 1 -- 172.21.15.195:0/3688843865 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f911406f640 msgr2=0x7f91140c0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:18.355 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.319+0000 7f9113fff700 1 --2- 172.21.15.195:0/3688843865 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f911406f640 0x7f91140c0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:18.356 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.319+0000 7f9113fff700 1 -- 172.21.15.195:0/3688843865 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90fc0092e0 con 0x7f91140c4160 2022-06-27T12:11:18.356 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.319+0000 7f9113fff700 1 --2- 172.21.15.195:0/3688843865 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f91140c4160 0x7f9114166c50 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f910c00c810 tx=0x7f910c003bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:18.357 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.319+0000 7f910b7fe700 1 -- 172.21.15.195:0/3688843865 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f910c012630 con 0x7f91140c4160 2022-06-27T12:11:18.357 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.319+0000 7f910b7fe700 1 -- 172.21.15.195:0/3688843865 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f910c0127d0 con 0x7f91140c4160 2022-06-27T12:11:18.357 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.319+0000 7f9119dee700 1 -- 172.21.15.195:0/3688843865 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f911406acd0 con 0x7f91140c4160 2022-06-27T12:11:18.358 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.319+0000 7f910b7fe700 1 -- 172.21.15.195:0/3688843865 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f910c012b90 con 0x7f91140c4160 2022-06-27T12:11:18.358 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.319+0000 7f9119dee700 1 -- 172.21.15.195:0/3688843865 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9114069d80 con 0x7f91140c4160 2022-06-27T12:11:18.358 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.320+0000 7f9119dee700 1 -- 172.21.15.195:0/3688843865 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91140bfea0 con 0x7f91140c4160 2022-06-27T12:11:18.359 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.321+0000 7f910b7fe700 1 -- 172.21.15.195:0/3688843865 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f910c022070 con 0x7f91140c4160 2022-06-27T12:11:18.359 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.321+0000 7f910b7fe700 1 --2- 172.21.15.195:0/3688843865 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f90f8063d00 0x7f90f80661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:18.360 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.321+0000 7f910b7fe700 1 -- 172.21.15.195:0/3688843865 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7320+0+0 (secure 0 0 0) 0x7f910c080f30 con 0x7f91140c4160 2022-06-27T12:11:18.360 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.321+0000 7f91137fe700 1 --2- 172.21.15.195:0/3688843865 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f90f8063d00 0x7f90f80661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:18.360 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.321+0000 7f91137fe700 1 --2- 172.21.15.195:0/3688843865 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f90f8063d00 0x7f90f80661b0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f90fc0097a0 tx=0x7f90fc016000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:18.361 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.325+0000 7f910b7fe700 1 -- 172.21.15.195:0/3688843865 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f91140bfea0 con 0x7f91140c4160 2022-06-27T12:11:18.554 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.553+0000 7f9119dee700 1 -- 172.21.15.195:0/3688843865 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f91140bfea0 con 0x7f91140c4160 2022-06-27T12:11:18.555 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.554+0000 7f910b7fe700 1 -- 172.21.15.195:0/3688843865 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+6128 (secure 0 0 0) 0x7f91140bfea0 con 0x7f91140c4160 2022-06-27T12:11:18.556 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:11:18.556 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":90,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":6050,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/3890799919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3890799919},{"type":"v1","addr":"172.21.15.202:6833","nonce":3890799919}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":90}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:11:01.244537+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5933},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5933":{"gid":5933,"name":"d","rank":0,"incarnation":85,"state":"up:active","state_seq":17,"addr":"172.21.15.202:6835/1342896965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":1342896965},{"type":"v1","addr":"172.21.15.202:6835","nonce":1342896965}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:11:18.559 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.556+0000 7f9119dee700 1 -- 172.21.15.195:0/3688843865 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f90f8063d00 msgr2=0x7f90f80661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:18.560 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.556+0000 7f9119dee700 1 --2- 172.21.15.195:0/3688843865 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f90f8063d00 0x7f90f80661b0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f90fc0097a0 tx=0x7f90fc016000 comp rx=0 tx=0).stop 2022-06-27T12:11:18.560 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.556+0000 7f9119dee700 1 -- 172.21.15.195:0/3688843865 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f91140c4160 msgr2=0x7f9114166c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:18.560 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.556+0000 7f9119dee700 1 --2- 172.21.15.195:0/3688843865 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f91140c4160 0x7f9114166c50 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f910c00c810 tx=0x7f910c003bd0 comp rx=0 tx=0).stop 2022-06-27T12:11:18.561 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.556+0000 7f9119dee700 1 -- 172.21.15.195:0/3688843865 shutdown_connections 2022-06-27T12:11:18.561 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.556+0000 7f9119dee700 1 --2- 172.21.15.195:0/3688843865 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f90f8063d00 0x7f90f80661b0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:18.564 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.556+0000 7f9119dee700 1 --2- 172.21.15.195:0/3688843865 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f91140c4160 0x7f9114166c50 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:18.564 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.556+0000 7f9119dee700 1 --2- 172.21.15.195:0/3688843865 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f911406f640 0x7f91140c0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:18.565 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.556+0000 7f9119dee700 1 --2- 172.21.15.195:0/3688843865 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f91140c39e0 0x7f91140bc6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:18.565 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.556+0000 7f9119dee700 1 -- 172.21.15.195:0/3688843865 >> 172.21.15.195:0/3688843865 conn(0x7f91140c6350 msgr2=0x7f9114150f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:11:18.566 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.556+0000 7f9119dee700 1 -- 172.21.15.195:0/3688843865 shutdown_connections 2022-06-27T12:11:18.566 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:18.556+0000 7f9119dee700 1 -- 172.21.15.195:0/3688843865 wait complete. 2022-06-27T12:11:18.566 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 90 2022-06-27T12:11:18.579 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-06-27T12:11:20.581 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:11:20.748 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.716+0000 7f87f600c700 1 Processor -- start 2022-06-27T12:11:20.748 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.716+0000 7f87f600c700 1 -- start start 2022-06-27T12:11:20.749 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.717+0000 7f87f600c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f87f014e490 0x7f87f006cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:20.749 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.717+0000 7f87f600c700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87f00c33b0 con 0x7f87f00c3f50 2022-06-27T12:11:20.750 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.717+0000 7f87f600c700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87f006d520 con 0x7f87f014e860 2022-06-27T12:11:20.750 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.717+0000 7f87f600c700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87f006f220 con 0x7f87f014e490 2022-06-27T12:11:20.751 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.717+0000 7f87ef7fe700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f87f014e490 0x7f87f006cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:20.751 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.717+0000 7f87ef7fe700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f87f014e490 0x7f87f006cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50308/0 (socket says 172.21.15.195:50308) 2022-06-27T12:11:20.752 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.717+0000 7f87effff700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f87f00c3f50 0x7f87f0157890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:56500/0 (socket says 172.21.15.195:56500) 2022-06-27T12:11:20.752 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.717+0000 7f87ef7fe700 1 -- 172.21.15.195:0/372590510 learned_addr learned my addr 172.21.15.195:0/372590510 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:11:20.753 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.717+0000 7f87ef7fe700 1 -- 172.21.15.195:0/372590510 >> v1:172.21.15.202:6789/0 conn(0x7f87f014e860 legacy=0x7f87f00c3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:11:20.753 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.717+0000 7f87ee7fc700 1 -- 172.21.15.195:0/372590510 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2966182202 0 0) 0x7f87f00c33b0 con 0x7f87f00c3f50 2022-06-27T12:11:20.754 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.717+0000 7f87ef7fe700 1 -- 172.21.15.195:0/372590510 >> v1:172.21.15.195:6789/0 conn(0x7f87f00c3f50 legacy=0x7f87f0157890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:20.754 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.717+0000 7f87ef7fe700 1 -- 172.21.15.195:0/372590510 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f01542c0 con 0x7f87f014e490 2022-06-27T12:11:20.755 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.717+0000 7f87ef7fe700 1 --2- 172.21.15.195:0/372590510 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f87f014e490 0x7f87f006cfe0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f87e0009860 tx=0x7f87e0009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecc0b4e70c4a44c2 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:20.755 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.717+0000 7f87ee7fc700 1 -- 172.21.15.195:0/372590510 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e000b680 con 0x7f87f014e490 2022-06-27T12:11:20.756 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.718+0000 7f87ee7fc700 1 -- 172.21.15.195:0/372590510 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87e000b7e0 con 0x7f87f014e490 2022-06-27T12:11:20.756 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.718+0000 7f87ee7fc700 1 -- 172.21.15.195:0/372590510 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e000bba0 con 0x7f87f014e490 2022-06-27T12:11:20.756 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.718+0000 7f87f600c700 1 -- 172.21.15.195:0/372590510 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f87f014e490 msgr2=0x7f87f006cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:20.774 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.718+0000 7f87f600c700 1 --2- 172.21.15.195:0/372590510 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f87f014e490 0x7f87f006cfe0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f87e0009860 tx=0x7f87e0009120 comp rx=0 tx=0).stop 2022-06-27T12:11:20.775 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.718+0000 7f87f600c700 1 -- 172.21.15.195:0/372590510 shutdown_connections 2022-06-27T12:11:20.775 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.718+0000 7f87f600c700 1 --2- 172.21.15.195:0/372590510 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f87f014e490 0x7f87f006cfe0 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:20.776 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.718+0000 7f87f600c700 1 -- 172.21.15.195:0/372590510 >> 172.21.15.195:0/372590510 conn(0x7f87f0063af0 msgr2=0x7f87f0063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:11:20.776 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.718+0000 7f87f600c700 1 -- 172.21.15.195:0/372590510 shutdown_connections 2022-06-27T12:11:20.777 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.718+0000 7f87f600c700 1 -- 172.21.15.195:0/372590510 wait complete. 2022-06-27T12:11:20.777 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.719+0000 7f87f600c700 1 Processor -- start 2022-06-27T12:11:20.778 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.719+0000 7f87f600c700 1 -- start start 2022-06-27T12:11:20.779 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.720+0000 7f87f600c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f87f00c3f50 0x7f87f00c66d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:20.779 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.720+0000 7f87f600c700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f87f014e490 0x7f87f00c6c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:20.780 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.720+0000 7f87f600c700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f87f014e860 0x7f87f014c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:20.780 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.720+0000 7f87f600c700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f00c7d90 con 0x7f87f00c3f50 2022-06-27T12:11:20.781 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.720+0000 7f87f600c700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f00c7ff0 con 0x7f87f014e860 2022-06-27T12:11:20.781 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.720+0000 7f87f600c700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87f0157120 con 0x7f87f014e490 2022-06-27T12:11:20.781 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.720+0000 7f87ef7fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f87f00c3f50 0x7f87f00c66d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:20.782 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.720+0000 7f87ef7fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f87f00c3f50 0x7f87f00c66d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50314/0 (socket says 172.21.15.195:50314) 2022-06-27T12:11:20.782 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.720+0000 7f87ef7fe700 1 -- 172.21.15.195:0/1940865824 learned_addr learned my addr 172.21.15.195:0/1940865824 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:11:20.783 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.720+0000 7f87eeffd700 1 --2- 172.21.15.195:0/1940865824 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f87f014e490 0x7f87f00c6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:20.783 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.720+0000 7f87effff700 1 --2- 172.21.15.195:0/1940865824 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f87f014e860 0x7f87f014c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:20.784 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.721+0000 7f87ef7fe700 1 -- 172.21.15.195:0/1940865824 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f87f014e490 msgr2=0x7f87f00c6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:20.784 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.721+0000 7f87ef7fe700 1 --2- 172.21.15.195:0/1940865824 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f87f014e490 0x7f87f00c6c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:20.785 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.721+0000 7f87ef7fe700 1 -- 172.21.15.195:0/1940865824 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f87f014e860 msgr2=0x7f87f014c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:20.785 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.721+0000 7f87ef7fe700 1 --2- 172.21.15.195:0/1940865824 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f87f014e860 0x7f87f014c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:20.785 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.721+0000 7f87ef7fe700 1 -- 172.21.15.195:0/1940865824 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87e00093a0 con 0x7f87f00c3f50 2022-06-27T12:11:20.786 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.721+0000 7f87eeffd700 1 --2- 172.21.15.195:0/1940865824 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f87f014e490 0x7f87f00c6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:11:20.786 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.721+0000 7f87effff700 1 --2- 172.21.15.195:0/1940865824 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f87f014e860 0x7f87f014c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:11:20.788 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.721+0000 7f87ef7fe700 1 --2- 172.21.15.195:0/1940865824 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f87f00c3f50 0x7f87f00c66d0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f87f006e4b0 tx=0x7f87e0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:20.788 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.721+0000 7f87d77fe700 1 -- 172.21.15.195:0/1940865824 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e000b950 con 0x7f87f00c3f50 2022-06-27T12:11:20.789 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.721+0000 7f87d77fe700 1 -- 172.21.15.195:0/1940865824 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87e000bab0 con 0x7f87f00c3f50 2022-06-27T12:11:20.789 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.721+0000 7f87f600c700 1 -- 172.21.15.195:0/1940865824 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87f0157400 con 0x7f87f00c3f50 2022-06-27T12:11:20.790 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.721+0000 7f87d77fe700 1 -- 172.21.15.195:0/1940865824 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e001a680 con 0x7f87f00c3f50 2022-06-27T12:11:20.791 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.721+0000 7f87f600c700 1 -- 172.21.15.195:0/1940865824 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87f00c8290 con 0x7f87f00c3f50 2022-06-27T12:11:20.792 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.722+0000 7f87f600c700 1 -- 172.21.15.195:0/1940865824 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87f00722c0 con 0x7f87f00c3f50 2022-06-27T12:11:20.792 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.722+0000 7f87d77fe700 1 -- 172.21.15.195:0/1940865824 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f87e000f050 con 0x7f87f00c3f50 2022-06-27T12:11:20.792 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.723+0000 7f87d77fe700 1 --2- 172.21.15.195:0/1940865824 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f87d0063d00 0x7f87d00661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:20.793 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.723+0000 7f87d77fe700 1 -- 172.21.15.195:0/1940865824 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7320+0+0 (secure 0 0 0) 0x7f87e0082a50 con 0x7f87f00c3f50 2022-06-27T12:11:20.804 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.723+0000 7f87eeffd700 1 --2- 172.21.15.195:0/1940865824 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f87d0063d00 0x7f87d00661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:20.804 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.723+0000 7f87eeffd700 1 --2- 172.21.15.195:0/1940865824 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f87d0063d00 0x7f87d00661b0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f87f01597c0 tx=0x7f87e4009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:20.805 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.727+0000 7f87d77fe700 1 -- 172.21.15.195:0/1940865824 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f87e0051150 con 0x7f87f00c3f50 2022-06-27T12:11:20.954 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.950+0000 7f87f600c700 1 -- 172.21.15.195:0/1940865824 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87f0064340 con 0x7f87f00c3f50 2022-06-27T12:11:20.954 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.951+0000 7f87d77fe700 1 -- 172.21.15.195:0/1940865824 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+6128 (secure 0 0 0) 0x7f87e0015090 con 0x7f87f00c3f50 2022-06-27T12:11:20.955 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:11:20.956 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":90,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":6050,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/3890799919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3890799919},{"type":"v1","addr":"172.21.15.202:6833","nonce":3890799919}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":90}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:11:01.244537+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5933},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5933":{"gid":5933,"name":"d","rank":0,"incarnation":85,"state":"up:active","state_seq":17,"addr":"172.21.15.202:6835/1342896965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":1342896965},{"type":"v1","addr":"172.21.15.202:6835","nonce":1342896965}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:11:20.983 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.953+0000 7f87f600c700 1 -- 172.21.15.195:0/1940865824 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f87d0063d00 msgr2=0x7f87d00661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:20.984 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.953+0000 7f87f600c700 1 --2- 172.21.15.195:0/1940865824 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f87d0063d00 0x7f87d00661b0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f87f01597c0 tx=0x7f87e4009040 comp rx=0 tx=0).stop 2022-06-27T12:11:20.984 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.953+0000 7f87f600c700 1 -- 172.21.15.195:0/1940865824 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f87f00c3f50 msgr2=0x7f87f00c66d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:20.985 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.953+0000 7f87f600c700 1 --2- 172.21.15.195:0/1940865824 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f87f00c3f50 0x7f87f00c66d0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f87f006e4b0 tx=0x7f87e0017000 comp rx=0 tx=0).stop 2022-06-27T12:11:20.985 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.953+0000 7f87f600c700 1 -- 172.21.15.195:0/1940865824 shutdown_connections 2022-06-27T12:11:20.986 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.953+0000 7f87f600c700 1 --2- 172.21.15.195:0/1940865824 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f87d0063d00 0x7f87d00661b0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:20.986 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.953+0000 7f87f600c700 1 --2- 172.21.15.195:0/1940865824 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f87f00c3f50 0x7f87f00c66d0 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:20.986 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.953+0000 7f87f600c700 1 --2- 172.21.15.195:0/1940865824 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f87f014e860 0x7f87f014c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:20.987 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.953+0000 7f87f600c700 1 --2- 172.21.15.195:0/1940865824 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f87f014e490 0x7f87f00c6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:20.987 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.953+0000 7f87f600c700 1 -- 172.21.15.195:0/1940865824 >> 172.21.15.195:0/1940865824 conn(0x7f87f0063af0 msgr2=0x7f87f0158950 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:11:20.988 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.953+0000 7f87f600c700 1 -- 172.21.15.195:0/1940865824 shutdown_connections 2022-06-27T12:11:20.988 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:20.953+0000 7f87f600c700 1 -- 172.21.15.195:0/1940865824 wait complete. 2022-06-27T12:11:20.989 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 90 2022-06-27T12:11:20.990 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-06-27T12:11:22.997 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:11:23.145 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.132+0000 7fc1b503b700 1 Processor -- start 2022-06-27T12:11:23.146 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.132+0000 7fc1b503b700 1 -- start start 2022-06-27T12:11:23.146 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.133+0000 7fc1b503b700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fc1b0151560 0x7fc1b00c2120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:23.146 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.133+0000 7fc1b503b700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc1b0151930 0x7fc1b00bf4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:23.146 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.133+0000 7fc1b503b700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1b0154da0 con 0x7fc1b00c2c70 2022-06-27T12:11:23.147 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.133+0000 7fc1b503b700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1b00c12a0 con 0x7fc1b0151560 2022-06-27T12:11:23.147 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.133+0000 7fc1b503b700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b00c1580 con 0x7fc1b0151930 2022-06-27T12:11:23.147 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.133+0000 7fc1af59e700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fc1b00c2c70 0x7fc1b01541f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:56514/0 (socket says 172.21.15.195:56514) 2022-06-27T12:11:23.148 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.133+0000 7fc1af59e700 1 -- 172.21.15.195:0/1564371538 learned_addr learned my addr 172.21.15.195:0/1564371538 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:11:23.148 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.133+0000 7fc1ae59c700 1 --2- 172.21.15.195:0/1564371538 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc1b0151930 0x7fc1b00bf4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:23.149 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.134+0000 7fc1aed9d700 1 --2- 172.21.15.195:0/1564371538 >> v2:172.21.15.202:3300/0 conn(0x7fc1b0151560 0x7fc1b00c2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:23.149 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.134+0000 7fc1add9b700 1 -- 172.21.15.195:0/1564371538 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3173566559 0 0) 0x7fc1b0154da0 con 0x7fc1b00c2c70 2022-06-27T12:11:23.149 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.134+0000 7fc1add9b700 1 -- 172.21.15.195:0/1564371538 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc190003430 con 0x7fc1b00c2c70 2022-06-27T12:11:23.150 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.134+0000 7fc1ae59c700 1 -- 172.21.15.195:0/1564371538 >> v2:172.21.15.202:3300/0 conn(0x7fc1b0151560 msgr2=0x7fc1b00c2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:11:23.150 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.134+0000 7fc1ae59c700 1 --2- 172.21.15.195:0/1564371538 >> v2:172.21.15.202:3300/0 conn(0x7fc1b0151560 0x7fc1b00c2120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:23.150 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.134+0000 7fc1ae59c700 1 -- 172.21.15.195:0/1564371538 >> v1:172.21.15.195:6789/0 conn(0x7fc1b00c2c70 legacy=0x7fc1b01541f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:23.151 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.134+0000 7fc1ae59c700 1 -- 172.21.15.195:0/1564371538 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1b00c29f0 con 0x7fc1b0151930 2022-06-27T12:11:23.158 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.134+0000 7fc1ae59c700 1 --2- 172.21.15.195:0/1564371538 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc1b0151930 0x7fc1b00bf4a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc1980097a0 tx=0x7fc198009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e12375123942990 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:23.253 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.134+0000 7fc1add9b700 1 -- 172.21.15.195:0/1564371538 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc198011630 con 0x7fc1b0151930 2022-06-27T12:11:23.253 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.134+0000 7fc1add9b700 1 -- 172.21.15.195:0/1564371538 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc198011790 con 0x7fc1b0151930 2022-06-27T12:11:23.253 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.134+0000 7fc1add9b700 1 -- 172.21.15.195:0/1564371538 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc198011b10 con 0x7fc1b0151930 2022-06-27T12:11:23.253 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.135+0000 7fc1b503b700 1 -- 172.21.15.195:0/1564371538 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc1b0151930 msgr2=0x7fc1b00bf4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:23.254 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.135+0000 7fc1b503b700 1 --2- 172.21.15.195:0/1564371538 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc1b0151930 0x7fc1b00bf4a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc1980097a0 tx=0x7fc198009060 comp rx=0 tx=0).stop 2022-06-27T12:11:23.254 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.135+0000 7fc1b503b700 1 -- 172.21.15.195:0/1564371538 shutdown_connections 2022-06-27T12:11:23.254 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.135+0000 7fc1b503b700 1 --2- 172.21.15.195:0/1564371538 >> v2:172.21.15.202:3300/0 conn(0x7fc1b0151560 0x7fc1b00c2120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:23.254 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.135+0000 7fc1b503b700 1 --2- 172.21.15.195:0/1564371538 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc1b0151930 0x7fc1b00bf4a0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:23.255 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.135+0000 7fc1b503b700 1 -- 172.21.15.195:0/1564371538 >> 172.21.15.195:0/1564371538 conn(0x7fc1b00c6350 msgr2=0x7fc1b00c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:11:23.255 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.135+0000 7fc1b503b700 1 -- 172.21.15.195:0/1564371538 shutdown_connections 2022-06-27T12:11:23.255 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.135+0000 7fc1b503b700 1 -- 172.21.15.195:0/1564371538 wait complete. 2022-06-27T12:11:23.255 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.136+0000 7fc1b503b700 1 Processor -- start 2022-06-27T12:11:23.256 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.136+0000 7fc1b503b700 1 -- start start 2022-06-27T12:11:23.256 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.137+0000 7fc1b503b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc1b00c2c70 0x7fc1b016a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:23.256 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.137+0000 7fc1b503b700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc1b0151560 0x7fc1b014e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:23.256 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.137+0000 7fc1b503b700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc1b0151930 0x7fc1b006e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:23.257 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.137+0000 7fc1b503b700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b016acc0 con 0x7fc1b00c2c70 2022-06-27T12:11:23.257 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.137+0000 7fc1b503b700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b0167390 con 0x7fc1b0151930 2022-06-27T12:11:23.257 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.137+0000 7fc1b503b700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b01667a0 con 0x7fc1b0151560 2022-06-27T12:11:23.257 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.137+0000 7fc1aed9d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc1b00c2c70 0x7fc1b016a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:23.259 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.137+0000 7fc1aed9d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc1b00c2c70 0x7fc1b016a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50330/0 (socket says 172.21.15.195:50330) 2022-06-27T12:11:23.260 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.137+0000 7fc1aed9d700 1 -- 172.21.15.195:0/3680770684 learned_addr learned my addr 172.21.15.195:0/3680770684 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:11:23.260 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.137+0000 7fc1ae59c700 1 --2- 172.21.15.195:0/3680770684 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc1b0151560 0x7fc1b014e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:23.260 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.137+0000 7fc1af59e700 1 --2- 172.21.15.195:0/3680770684 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc1b0151930 0x7fc1b006e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:23.260 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.137+0000 7fc1aed9d700 1 -- 172.21.15.195:0/3680770684 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc1b0151560 msgr2=0x7fc1b014e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:23.261 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.137+0000 7fc1aed9d700 1 --2- 172.21.15.195:0/3680770684 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc1b0151560 0x7fc1b014e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:23.261 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.137+0000 7fc1aed9d700 1 -- 172.21.15.195:0/3680770684 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc1b0151930 msgr2=0x7fc1b006e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:23.261 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.137+0000 7fc1aed9d700 1 --2- 172.21.15.195:0/3680770684 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc1b0151930 0x7fc1b006e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:23.261 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.137+0000 7fc1aed9d700 1 -- 172.21.15.195:0/3680770684 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1980092e0 con 0x7fc1b00c2c70 2022-06-27T12:11:23.262 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.138+0000 7fc1aed9d700 1 --2- 172.21.15.195:0/3680770684 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc1b00c2c70 0x7fc1b016a780 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fc1a000e930 tx=0x7fc1a0009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:23.262 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.138+0000 7fc1a6ffd700 1 -- 172.21.15.195:0/3680770684 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a0013680 con 0x7fc1b00c2c70 2022-06-27T12:11:23.262 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.138+0000 7fc1a6ffd700 1 -- 172.21.15.195:0/3680770684 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1a00137e0 con 0x7fc1b00c2c70 2022-06-27T12:11:23.263 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.138+0000 7fc1b503b700 1 -- 172.21.15.195:0/3680770684 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1b006fd70 con 0x7fc1b00c2c70 2022-06-27T12:11:23.263 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.138+0000 7fc1a6ffd700 1 -- 172.21.15.195:0/3680770684 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a0013b80 con 0x7fc1b00c2c70 2022-06-27T12:11:23.263 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.138+0000 7fc1b503b700 1 -- 172.21.15.195:0/3680770684 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1b0166d40 con 0x7fc1b00c2c70 2022-06-27T12:11:23.263 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.139+0000 7fc1b503b700 1 -- 172.21.15.195:0/3680770684 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1b00722c0 con 0x7fc1b00c2c70 2022-06-27T12:11:23.263 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.139+0000 7fc1a6ffd700 1 -- 172.21.15.195:0/3680770684 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fc1a0019030 con 0x7fc1b00c2c70 2022-06-27T12:11:23.264 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.139+0000 7fc1a6ffd700 1 --2- 172.21.15.195:0/3680770684 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc194063d30 0x7fc1940661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:23.264 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.140+0000 7fc1a6ffd700 1 -- 172.21.15.195:0/3680770684 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7320+0+0 (secure 0 0 0) 0x7fc1a0080bf0 con 0x7fc1b00c2c70 2022-06-27T12:11:23.264 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.140+0000 7fc1ae59c700 1 --2- 172.21.15.195:0/3680770684 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc194063d30 0x7fc1940661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:23.265 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.140+0000 7fc1ae59c700 1 --2- 172.21.15.195:0/3680770684 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc194063d30 0x7fc1940661e0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fc198009260 tx=0x7fc198017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:23.265 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.144+0000 7fc1a6ffd700 1 -- 172.21.15.195:0/3680770684 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fc1a004f290 con 0x7fc1b00c2c70 2022-06-27T12:11:23.375 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.370+0000 7fc1b503b700 1 -- 172.21.15.195:0/3680770684 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc1b00c77b0 con 0x7fc1b00c2c70 2022-06-27T12:11:23.376 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.371+0000 7fc1a6ffd700 1 -- 172.21.15.195:0/3680770684 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+6128 (secure 0 0 0) 0x7fc1a0017090 con 0x7fc1b00c2c70 2022-06-27T12:11:23.377 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:11:23.380 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":90,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":6050,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/3890799919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3890799919},{"type":"v1","addr":"172.21.15.202:6833","nonce":3890799919}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":90}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:11:01.244537+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5933},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5933":{"gid":5933,"name":"d","rank":0,"incarnation":85,"state":"up:active","state_seq":17,"addr":"172.21.15.202:6835/1342896965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":1342896965},{"type":"v1","addr":"172.21.15.202:6835","nonce":1342896965}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:11:23.394 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.373+0000 7fc1b503b700 1 -- 172.21.15.195:0/3680770684 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc194063d30 msgr2=0x7fc1940661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:23.394 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.373+0000 7fc1b503b700 1 --2- 172.21.15.195:0/3680770684 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc194063d30 0x7fc1940661e0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fc198009260 tx=0x7fc198017000 comp rx=0 tx=0).stop 2022-06-27T12:11:23.394 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.373+0000 7fc1b503b700 1 -- 172.21.15.195:0/3680770684 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc1b00c2c70 msgr2=0x7fc1b016a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:23.395 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.373+0000 7fc1b503b700 1 --2- 172.21.15.195:0/3680770684 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc1b00c2c70 0x7fc1b016a780 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fc1a000e930 tx=0x7fc1a0009060 comp rx=0 tx=0).stop 2022-06-27T12:11:23.395 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.373+0000 7fc1b503b700 1 -- 172.21.15.195:0/3680770684 shutdown_connections 2022-06-27T12:11:23.395 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.373+0000 7fc1b503b700 1 --2- 172.21.15.195:0/3680770684 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc194063d30 0x7fc1940661e0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:23.395 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.373+0000 7fc1b503b700 1 --2- 172.21.15.195:0/3680770684 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc1b00c2c70 0x7fc1b016a780 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:23.396 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.373+0000 7fc1b503b700 1 --2- 172.21.15.195:0/3680770684 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc1b0151930 0x7fc1b006e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:23.396 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.373+0000 7fc1b503b700 1 --2- 172.21.15.195:0/3680770684 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc1b0151560 0x7fc1b014e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:23.396 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.373+0000 7fc1b503b700 1 -- 172.21.15.195:0/3680770684 >> 172.21.15.195:0/3680770684 conn(0x7fc1b00c6350 msgr2=0x7fc1b014f360 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:11:23.396 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.373+0000 7fc1b503b700 1 -- 172.21.15.195:0/3680770684 shutdown_connections 2022-06-27T12:11:23.397 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:23.373+0000 7fc1b503b700 1 -- 172.21.15.195:0/3680770684 wait complete. 2022-06-27T12:11:23.402 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 90 2022-06-27T12:11:23.404 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-06-27T12:11:25.411 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:11:25.571 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.554+0000 7f941e8aa700 1 Processor -- start 2022-06-27T12:11:25.572 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.554+0000 7f941e8aa700 1 -- start start 2022-06-27T12:11:25.573 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.555+0000 7f941e8aa700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f94180688a0 0x7f94180c28c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:25.573 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.555+0000 7f941e8aa700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94180c4b10 con 0x7f9418156100 2022-06-27T12:11:25.574 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.555+0000 7f941e8aa700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9418068f20 con 0x7f941806a440 2022-06-27T12:11:25.582 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.555+0000 7f941e8aa700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94180c3dc0 con 0x7f94180688a0 2022-06-27T12:11:25.583 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.555+0000 7f9417fff700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f9418156100 0x7f94180c4fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:56524/0 (socket says 172.21.15.195:56524) 2022-06-27T12:11:25.583 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.555+0000 7f9417fff700 1 -- 172.21.15.195:0/2951950120 learned_addr learned my addr 172.21.15.195:0/2951950120 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:11:25.583 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.555+0000 7f941ce47700 1 --2- 172.21.15.195:0/2951950120 >> v2:172.21.15.202:3300/0 conn(0x7f94180688a0 0x7f94180c28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:25.584 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.555+0000 7f9416ffd700 1 -- 172.21.15.195:0/2951950120 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2712712584 0 0) 0x7f94180c4b10 con 0x7f9418156100 2022-06-27T12:11:25.584 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.555+0000 7f9416ffd700 1 -- 172.21.15.195:0/2951950120 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9404004540 con 0x7f9418156100 2022-06-27T12:11:25.585 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.555+0000 7f9416ffd700 1 -- 172.21.15.195:0/2951950120 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 259152366 0 0) 0x7f9404004540 con 0x7f9418156100 2022-06-27T12:11:25.585 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.555+0000 7f9416ffd700 1 -- 172.21.15.195:0/2951950120 >> v2:172.21.15.202:3300/0 conn(0x7f94180688a0 msgr2=0x7f94180c28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:11:25.586 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.555+0000 7f9416ffd700 1 --2- 172.21.15.195:0/2951950120 >> v2:172.21.15.202:3300/0 conn(0x7f94180688a0 0x7f94180c28c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:25.587 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.556+0000 7f9416ffd700 1 -- 172.21.15.195:0/2951950120 >> v1:172.21.15.202:6789/0 conn(0x7f941806a440 legacy=0x7f94181564d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:25.587 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.556+0000 7f9416ffd700 1 -- 172.21.15.195:0/2951950120 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94180c56b0 con 0x7f9418156100 2022-06-27T12:11:25.588 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.556+0000 7f9416ffd700 1 -- 172.21.15.195:0/2951950120 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7f940c0015f0 con 0x7f9418156100 2022-06-27T12:11:25.588 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.556+0000 7f9416ffd700 1 -- 172.21.15.195:0/2951950120 >> v1:172.21.15.195:6789/0 conn(0x7f9418156100 legacy=0x7f94180c4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:25.589 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.556+0000 7f9416ffd700 1 --2- 172.21.15.195:0/2951950120 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9404004a30 0x7f9404004e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:25.589 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.556+0000 7f9416ffd700 1 --2- 172.21.15.195:0/2951950120 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f94040057e0 0x7f9404005c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:25.590 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.556+0000 7f9416ffd700 1 --2- 172.21.15.195:0/2951950120 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9404006190 0x7f940400ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:25.590 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.556+0000 7f9416ffd700 1 -- 172.21.15.195:0/2951950120 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94180c3dc0 con 0x7f94040057e0 2022-06-27T12:11:25.591 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.556+0000 7f9416ffd700 1 -- 172.21.15.195:0/2951950120 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f940400d0a0 con 0x7f9404004a30 2022-06-27T12:11:25.592 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.556+0000 7f9416ffd700 1 -- 172.21.15.195:0/2951950120 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f940400d260 con 0x7f9404006190 2022-06-27T12:11:25.592 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.556+0000 7f94177fe700 1 --2- 172.21.15.195:0/2951950120 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f94040057e0 0x7f9404005c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:25.592 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.556+0000 7f9417fff700 1 --2- 172.21.15.195:0/2951950120 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9404004a30 0x7f9404004e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:25.593 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.556+0000 7f941ce47700 1 --2- 172.21.15.195:0/2951950120 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9404006190 0x7f940400ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:25.593 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.556+0000 7f94177fe700 1 -- 172.21.15.195:0/2951950120 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9404006190 msgr2=0x7f940400ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:11:25.594 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.556+0000 7f94177fe700 1 --2- 172.21.15.195:0/2951950120 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9404006190 0x7f940400ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:25.594 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.556+0000 7f94177fe700 1 -- 172.21.15.195:0/2951950120 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9404004a30 msgr2=0x7f9404004e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:11:25.595 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.556+0000 7f94177fe700 1 --2- 172.21.15.195:0/2951950120 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9404004a30 0x7f9404004e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:25.595 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.556+0000 7f94177fe700 1 -- 172.21.15.195:0/2951950120 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9404004540 con 0x7f94040057e0 2022-06-27T12:11:25.595 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.556+0000 7f941ce47700 1 --2- 172.21.15.195:0/2951950120 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9404006190 0x7f940400ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:11:25.596 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.556+0000 7f941ce47700 1 -- 172.21.15.195:0/2951950120 reap_dead start 2022-06-27T12:11:25.597 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.556+0000 7f94177fe700 1 --2- 172.21.15.195:0/2951950120 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f94040057e0 0x7f9404005c50 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f9408001730 tx=0x7f9408007bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2598b314020097a server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:25.597 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.556+0000 7f9416ffd700 1 -- 172.21.15.195:0/2951950120 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94080050d0 con 0x7f94040057e0 2022-06-27T12:11:25.598 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.556+0000 7f9416ffd700 1 -- 172.21.15.195:0/2951950120 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9408005230 con 0x7f94040057e0 2022-06-27T12:11:25.598 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.556+0000 7f9416ffd700 1 -- 172.21.15.195:0/2951950120 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9408003750 con 0x7f94040057e0 2022-06-27T12:11:25.599 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.557+0000 7f941e8aa700 1 -- 172.21.15.195:0/2951950120 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f94040057e0 msgr2=0x7f9404005c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:25.599 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.557+0000 7f941e8aa700 1 --2- 172.21.15.195:0/2951950120 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f94040057e0 0x7f9404005c50 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f9408001730 tx=0x7f9408007bb0 comp rx=0 tx=0).stop 2022-06-27T12:11:25.600 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.557+0000 7f941e8aa700 1 -- 172.21.15.195:0/2951950120 shutdown_connections 2022-06-27T12:11:25.600 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.557+0000 7f941e8aa700 1 --2- 172.21.15.195:0/2951950120 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f94040057e0 0x7f9404005c50 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:25.600 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.557+0000 7f941e8aa700 1 -- 172.21.15.195:0/2951950120 >> 172.21.15.195:0/2951950120 conn(0x7f94180c9b00 msgr2=0x7f94180c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:11:25.600 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.557+0000 7f941e8aa700 1 -- 172.21.15.195:0/2951950120 shutdown_connections 2022-06-27T12:11:25.601 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.557+0000 7f941e8aa700 1 -- 172.21.15.195:0/2951950120 wait complete. 2022-06-27T12:11:25.602 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.558+0000 7f941e8aa700 1 Processor -- start 2022-06-27T12:11:25.602 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.558+0000 7f941e8aa700 1 -- start start 2022-06-27T12:11:25.602 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.559+0000 7f941e8aa700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f94040057e0 0x7f941814a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:25.603 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.559+0000 7f941e8aa700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f941814aea0 0x7f94180bb240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:25.603 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.559+0000 7f941e8aa700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f94180bb780 0x7f94181664c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:25.604 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.559+0000 7f941e8aa700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f941814b310 con 0x7f941814aea0 2022-06-27T12:11:25.604 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.559+0000 7f941e8aa700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94180c84b0 con 0x7f94040057e0 2022-06-27T12:11:25.605 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.559+0000 7f941e8aa700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94180bbbf0 con 0x7f94180bb780 2022-06-27T12:11:25.605 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.559+0000 7f94177fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f941814aea0 0x7f94180bb240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:25.606 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.559+0000 7f94177fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f941814aea0 0x7f94180bb240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50348/0 (socket says 172.21.15.195:50348) 2022-06-27T12:11:25.607 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.559+0000 7f94177fe700 1 -- 172.21.15.195:0/2608351803 learned_addr learned my addr 172.21.15.195:0/2608351803 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:11:25.608 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.559+0000 7f941ce47700 1 --2- 172.21.15.195:0/2608351803 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f94180bb780 0x7f94181664c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:25.608 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.559+0000 7f9417fff700 1 --2- 172.21.15.195:0/2608351803 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f94040057e0 0x7f941814a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:25.609 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.559+0000 7f94177fe700 1 -- 172.21.15.195:0/2608351803 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f94180bb780 msgr2=0x7f94181664c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:25.617 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.559+0000 7f94177fe700 1 --2- 172.21.15.195:0/2608351803 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f94180bb780 0x7f94181664c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:25.617 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.559+0000 7f94177fe700 1 -- 172.21.15.195:0/2608351803 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f94040057e0 msgr2=0x7f941814a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:25.617 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.559+0000 7f94177fe700 1 --2- 172.21.15.195:0/2608351803 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f94040057e0 0x7f941814a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:25.617 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.559+0000 7f94177fe700 1 -- 172.21.15.195:0/2608351803 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94180c7550 con 0x7f941814aea0 2022-06-27T12:11:25.618 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.560+0000 7f94177fe700 1 --2- 172.21.15.195:0/2608351803 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f941814aea0 0x7f94180bb240 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f9408001d20 tx=0x7f9408015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:25.618 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.560+0000 7f93fffff700 1 -- 172.21.15.195:0/2608351803 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9408003710 con 0x7f941814aea0 2022-06-27T12:11:25.618 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.560+0000 7f93fffff700 1 -- 172.21.15.195:0/2608351803 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9408012420 con 0x7f941814aea0 2022-06-27T12:11:25.618 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.560+0000 7f93fffff700 1 -- 172.21.15.195:0/2608351803 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9408012800 con 0x7f941814aea0 2022-06-27T12:11:25.619 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.560+0000 7f941e8aa700 1 -- 172.21.15.195:0/2608351803 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94180c7830 con 0x7f941814aea0 2022-06-27T12:11:25.619 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.560+0000 7f941e8aa700 1 -- 172.21.15.195:0/2608351803 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94180c89e0 con 0x7f941814aea0 2022-06-27T12:11:25.619 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.561+0000 7f941e8aa700 1 -- 172.21.15.195:0/2608351803 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94180722c0 con 0x7f941814aea0 2022-06-27T12:11:25.619 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.561+0000 7f93fffff700 1 -- 172.21.15.195:0/2608351803 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f9408010050 con 0x7f941814aea0 2022-06-27T12:11:25.620 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.561+0000 7f93fffff700 1 --2- 172.21.15.195:0/2608351803 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f93f8063d30 0x7f93f80661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:25.620 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.562+0000 7f93fffff700 1 -- 172.21.15.195:0/2608351803 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7406+0+0 (secure 0 0 0) 0x7f9408082240 con 0x7f941814aea0 2022-06-27T12:11:25.620 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.562+0000 7f9417fff700 1 --2- 172.21.15.195:0/2608351803 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f93f8063d30 0x7f93f80661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:25.620 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.562+0000 7f9417fff700 1 --2- 172.21.15.195:0/2608351803 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f93f8063d30 0x7f93f80661e0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f940c000e10 tx=0x7f940c009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:25.621 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.566+0000 7f93fffff700 1 -- 172.21.15.195:0/2608351803 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f9408050800 con 0x7f941814aea0 2022-06-27T12:11:25.785 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.785+0000 7f941e8aa700 1 -- 172.21.15.195:0/2608351803 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f94180bc120 con 0x7f941814aea0 2022-06-27T12:11:25.786 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.786+0000 7f93fffff700 1 -- 172.21.15.195:0/2608351803 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+5367 (secure 0 0 0) 0x7f9408053e20 con 0x7f941814aea0 2022-06-27T12:11:25.787 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:11:25.788 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:11:25.147223+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6050},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6050":{"gid":6050,"name":"b","rank":0,"incarnation":91,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6833/3890799919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3890799919},{"type":"v1","addr":"172.21.15.202:6833","nonce":3890799919}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:11:25.790 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.788+0000 7f941e8aa700 1 -- 172.21.15.195:0/2608351803 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f93f8063d30 msgr2=0x7f93f80661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:25.791 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.788+0000 7f941e8aa700 1 --2- 172.21.15.195:0/2608351803 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f93f8063d30 0x7f93f80661e0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f940c000e10 tx=0x7f940c009040 comp rx=0 tx=0).stop 2022-06-27T12:11:25.791 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.788+0000 7f941e8aa700 1 -- 172.21.15.195:0/2608351803 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f941814aea0 msgr2=0x7f94180bb240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:25.791 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.788+0000 7f941e8aa700 1 --2- 172.21.15.195:0/2608351803 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f941814aea0 0x7f94180bb240 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f9408001d20 tx=0x7f9408015000 comp rx=0 tx=0).stop 2022-06-27T12:11:25.792 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.788+0000 7f941e8aa700 1 -- 172.21.15.195:0/2608351803 shutdown_connections 2022-06-27T12:11:25.792 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.788+0000 7f941e8aa700 1 --2- 172.21.15.195:0/2608351803 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f93f8063d30 0x7f93f80661e0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:25.792 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.788+0000 7f941e8aa700 1 --2- 172.21.15.195:0/2608351803 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f94180bb780 0x7f94181664c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:25.793 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.788+0000 7f941e8aa700 1 --2- 172.21.15.195:0/2608351803 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f94040057e0 0x7f941814a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:25.793 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.788+0000 7f941e8aa700 1 --2- 172.21.15.195:0/2608351803 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f941814aea0 0x7f94180bb240 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:25.793 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.788+0000 7f941e8aa700 1 -- 172.21.15.195:0/2608351803 >> 172.21.15.195:0/2608351803 conn(0x7f94180c9b00 msgr2=0x7f94180c4fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:11:25.793 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.788+0000 7f941e8aa700 1 -- 172.21.15.195:0/2608351803 shutdown_connections 2022-06-27T12:11:25.794 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.788+0000 7f941e8aa700 1 -- 172.21.15.195:0/2608351803 wait complete. 2022-06-27T12:11:25.794 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 91 2022-06-27T12:11:25.800 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d down, removed from mdsmap 2022-06-27T12:11:25.801 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2022-06-27T12:11:25.801 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:11:25.938 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.921+0000 7f851196e700 1 Processor -- start 2022-06-27T12:11:25.939 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.921+0000 7f851196e700 1 -- start start 2022-06-27T12:11:25.939 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.922+0000 7f851196e700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f850c150580 0x7f850c156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:25.940 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.922+0000 7f851196e700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f850c150950 0x7f850c0c7b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:25.940 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.922+0000 7f851196e700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f850c0c76b0 con 0x7f850c06f620 2022-06-27T12:11:25.941 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.922+0000 7f851196e700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f850c157430 con 0x7f850c150950 2022-06-27T12:11:25.941 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.922+0000 7f851196e700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f850c0bad70 con 0x7f850c150580 2022-06-27T12:11:25.941 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.922+0000 7f850b7fe700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f850c06f620 0x7f850c06d6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:56544/0 (socket says 172.21.15.195:56544) 2022-06-27T12:11:25.941 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.922+0000 7f850b7fe700 1 -- 172.21.15.195:0/2141921807 learned_addr learned my addr 172.21.15.195:0/2141921807 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:11:25.942 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.923+0000 7f850affd700 1 --2- 172.21.15.195:0/2141921807 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f850c150580 0x7f850c156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:25.942 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.923+0000 7f8509ffb700 1 -- 172.21.15.195:0/2141921807 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4261298576 0 0) 0x7f850c0c76b0 con 0x7f850c06f620 2022-06-27T12:11:25.942 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.923+0000 7f8509ffb700 1 -- 172.21.15.195:0/2141921807 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84f8003430 con 0x7f850c06f620 2022-06-27T12:11:25.942 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.923+0000 7f850a7fc700 1 --2- 172.21.15.195:0/2141921807 >> v2:172.21.15.202:3300/0 conn(0x7f850c150950 0x7f850c0c7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:25.943 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.923+0000 7f8509ffb700 1 -- 172.21.15.195:0/2141921807 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1986375207 0 0) 0x7f84f8003430 con 0x7f850c06f620 2022-06-27T12:11:25.943 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.923+0000 7f8509ffb700 1 -- 172.21.15.195:0/2141921807 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f850c150580 msgr2=0x7f850c156ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:11:25.943 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.923+0000 7f8509ffb700 1 --2- 172.21.15.195:0/2141921807 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f850c150580 0x7f850c156ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:25.943 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.923+0000 7f8509ffb700 1 -- 172.21.15.195:0/2141921807 >> v2:172.21.15.202:3300/0 conn(0x7f850c150950 msgr2=0x7f850c0c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:11:25.944 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.923+0000 7f8509ffb700 1 --2- 172.21.15.195:0/2141921807 >> v2:172.21.15.202:3300/0 conn(0x7f850c150950 0x7f850c0c7b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:25.944 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.923+0000 7f8509ffb700 1 -- 172.21.15.195:0/2141921807 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f850c0ba4d0 con 0x7f850c06f620 2022-06-27T12:11:25.944 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.923+0000 7f8509ffb700 1 -- 172.21.15.195:0/2141921807 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7f85040017f0 con 0x7f850c06f620 2022-06-27T12:11:25.944 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.923+0000 7f850a7fc700 1 --2- 172.21.15.195:0/2141921807 >> v2:172.21.15.202:3300/0 conn(0x7f850c150950 0x7f850c0c7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:11:25.945 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.923+0000 7f8509ffb700 1 -- 172.21.15.195:0/2141921807 >> v1:172.21.15.195:6789/0 conn(0x7f850c06f620 legacy=0x7f850c06d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:25.945 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.923+0000 7f8509ffb700 1 --2- 172.21.15.195:0/2141921807 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f84f8003920 0x7f84f8003d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:25.945 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.923+0000 7f8509ffb700 1 --2- 172.21.15.195:0/2141921807 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f850c150580 0x7f84f80046d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:25.945 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.923+0000 7f8509ffb700 1 --2- 172.21.15.195:0/2141921807 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f84f8004c10 0x7f84f8005080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:25.945 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.923+0000 7f8509ffb700 1 -- 172.21.15.195:0/2141921807 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f850c0bad70 con 0x7f84f8004c10 2022-06-27T12:11:25.946 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.923+0000 7f8509ffb700 1 -- 172.21.15.195:0/2141921807 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84f800b920 con 0x7f84f8003920 2022-06-27T12:11:25.946 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.923+0000 7f8509ffb700 1 -- 172.21.15.195:0/2141921807 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84f800bae0 con 0x7f850c150580 2022-06-27T12:11:25.946 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.923+0000 7f850b7fe700 1 --2- 172.21.15.195:0/2141921807 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f84f8004c10 0x7f84f8005080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:25.947 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.923+0000 7f850affd700 1 --2- 172.21.15.195:0/2141921807 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f84f8003920 0x7f84f8003d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:25.947 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.923+0000 7f850a7fc700 1 --2- 172.21.15.195:0/2141921807 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f850c150580 0x7f84f80046d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:25.947 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.923+0000 7f850b7fe700 1 -- 172.21.15.195:0/2141921807 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f850c150580 msgr2=0x7f84f80046d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:11:25.947 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.923+0000 7f850b7fe700 1 --2- 172.21.15.195:0/2141921807 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f850c150580 0x7f84f80046d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:25.948 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.923+0000 7f850b7fe700 1 -- 172.21.15.195:0/2141921807 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f84f8003920 msgr2=0x7f84f8003d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:11:25.948 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.923+0000 7f850b7fe700 1 --2- 172.21.15.195:0/2141921807 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f84f8003920 0x7f84f8003d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:25.948 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.923+0000 7f850b7fe700 1 -- 172.21.15.195:0/2141921807 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84f8003430 con 0x7f84f8004c10 2022-06-27T12:11:25.948 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.924+0000 7f850b7fe700 1 --2- 172.21.15.195:0/2141921807 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f84f8004c10 0x7f84f8005080 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f850400c500 tx=0x7f8504001d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34c3dde07b28bdd1 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:25.949 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.924+0000 7f8509ffb700 1 -- 172.21.15.195:0/2141921807 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8504001fc0 con 0x7f84f8004c10 2022-06-27T12:11:25.949 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.924+0000 7f8509ffb700 1 -- 172.21.15.195:0/2141921807 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8504005b60 con 0x7f84f8004c10 2022-06-27T12:11:25.949 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.924+0000 7f8509ffb700 1 -- 172.21.15.195:0/2141921807 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85040070d0 con 0x7f84f8004c10 2022-06-27T12:11:25.949 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.924+0000 7f851196e700 1 -- 172.21.15.195:0/2141921807 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f84f8004c10 msgr2=0x7f84f8005080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:25.950 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.924+0000 7f851196e700 1 --2- 172.21.15.195:0/2141921807 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f84f8004c10 0x7f84f8005080 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f850400c500 tx=0x7f8504001d90 comp rx=0 tx=0).stop 2022-06-27T12:11:25.950 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.924+0000 7f850b7fe700 1 -- 172.21.15.195:0/2141921807 reap_dead start 2022-06-27T12:11:25.951 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.924+0000 7f851196e700 1 -- 172.21.15.195:0/2141921807 shutdown_connections 2022-06-27T12:11:25.951 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.924+0000 7f851196e700 1 -- 172.21.15.195:0/2141921807 >> 172.21.15.195:0/2141921807 conn(0x7f850c063af0 msgr2=0x7f850c063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:11:25.951 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.924+0000 7f851196e700 1 -- 172.21.15.195:0/2141921807 shutdown_connections 2022-06-27T12:11:25.951 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.924+0000 7f851196e700 1 -- 172.21.15.195:0/2141921807 wait complete. 2022-06-27T12:11:25.952 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.925+0000 7f851196e700 1 Processor -- start 2022-06-27T12:11:25.952 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.926+0000 7f851196e700 1 -- start start 2022-06-27T12:11:25.952 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.926+0000 7f851196e700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f850c0c5620 0x7f850c15c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:25.952 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.926+0000 7f851196e700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f850c0c9ee0 0x7f850c167840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:25.953 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.926+0000 7f851196e700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f850c15a900 0x7f850c166e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:25.953 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.926+0000 7f851196e700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f850c0c20a0 con 0x7f850c0c5620 2022-06-27T12:11:25.953 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.926+0000 7f851196e700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f850c15b370 con 0x7f850c0c9ee0 2022-06-27T12:11:25.954 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.926+0000 7f851196e700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f850c15adb0 con 0x7f850c15a900 2022-06-27T12:11:25.954 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.926+0000 7f850affd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f850c0c5620 0x7f850c15c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:25.954 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.926+0000 7f850affd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f850c0c5620 0x7f850c15c150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50366/0 (socket says 172.21.15.195:50366) 2022-06-27T12:11:25.954 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.926+0000 7f850affd700 1 -- 172.21.15.195:0/1633640095 learned_addr learned my addr 172.21.15.195:0/1633640095 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:11:25.954 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.926+0000 7f850b7fe700 1 --2- 172.21.15.195:0/1633640095 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f850c15a900 0x7f850c166e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:25.955 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.926+0000 7f850a7fc700 1 --2- 172.21.15.195:0/1633640095 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f850c0c9ee0 0x7f850c167840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:25.955 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.927+0000 7f850affd700 1 -- 172.21.15.195:0/1633640095 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f850c15a900 msgr2=0x7f850c166e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:25.955 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.927+0000 7f850affd700 1 --2- 172.21.15.195:0/1633640095 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f850c15a900 0x7f850c166e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:25.955 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.927+0000 7f850affd700 1 -- 172.21.15.195:0/1633640095 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f850c0c9ee0 msgr2=0x7f850c167840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:25.956 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.927+0000 7f850affd700 1 --2- 172.21.15.195:0/1633640095 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f850c0c9ee0 0x7f850c167840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:25.956 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.927+0000 7f850affd700 1 -- 172.21.15.195:0/1633640095 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f850c15b090 con 0x7f850c0c5620 2022-06-27T12:11:25.956 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.927+0000 7f850a7fc700 1 --2- 172.21.15.195:0/1633640095 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f850c0c9ee0 0x7f850c167840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:11:25.956 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.927+0000 7f850affd700 1 --2- 172.21.15.195:0/1633640095 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f850c0c5620 0x7f850c15c150 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f85000119a0 tx=0x7f8500009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:25.957 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.927+0000 7f84f2ffd700 1 -- 172.21.15.195:0/1633640095 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8500004240 con 0x7f850c0c5620 2022-06-27T12:11:25.958 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.927+0000 7f84f2ffd700 1 -- 172.21.15.195:0/1633640095 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85000043a0 con 0x7f850c0c5620 2022-06-27T12:11:25.958 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.927+0000 7f851196e700 1 -- 172.21.15.195:0/1633640095 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f850c15b670 con 0x7f850c0c5620 2022-06-27T12:11:25.958 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.927+0000 7f84f2ffd700 1 -- 172.21.15.195:0/1633640095 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f850000f940 con 0x7f850c0c5620 2022-06-27T12:11:25.959 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.927+0000 7f851196e700 1 -- 172.21.15.195:0/1633640095 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f850c15bc30 con 0x7f850c0c5620 2022-06-27T12:11:25.959 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.928+0000 7f851196e700 1 -- 172.21.15.195:0/1633640095 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f850c15be50 con 0x7f850c0c5620 2022-06-27T12:11:25.959 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.928+0000 7f84f2ffd700 1 -- 172.21.15.195:0/1633640095 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f8500024070 con 0x7f850c0c5620 2022-06-27T12:11:25.959 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.929+0000 7f84f2ffd700 1 --2- 172.21.15.195:0/1633640095 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f84ec063d00 0x7f84ec0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:25.960 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.929+0000 7f84f2ffd700 1 -- 172.21.15.195:0/1633640095 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7406+0+0 (secure 0 0 0) 0x7f8500018030 con 0x7f850c0c5620 2022-06-27T12:11:25.960 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.929+0000 7f850a7fc700 1 --2- 172.21.15.195:0/1633640095 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f84ec063d00 0x7f84ec0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:25.961 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.929+0000 7f850a7fc700 1 --2- 172.21.15.195:0/1633640095 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f84ec063d00 0x7f84ec0661b0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f850c0704f0 tx=0x7f84fc00b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:25.962 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:25.933+0000 7f84f2ffd700 1 -- 172.21.15.195:0/1633640095 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f850004fb30 con 0x7f850c0c5620 2022-06-27T12:11:26.161 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:26.159+0000 7f851196e700 1 -- 172.21.15.195:0/1633640095 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f850c0722c0 con 0x7f850c0c5620 2022-06-27T12:11:26.162 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:26.160+0000 7f84f2ffd700 1 -- 172.21.15.195:0/1633640095 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+5367 (secure 0 0 0) 0x7f8500022020 con 0x7f850c0c5620 2022-06-27T12:11:26.162 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:11:26.163 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:11:25.147223+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6050},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6050":{"gid":6050,"name":"b","rank":0,"incarnation":91,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6833/3890799919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3890799919},{"type":"v1","addr":"172.21.15.202:6833","nonce":3890799919}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:11:26.214 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:26.162+0000 7f851196e700 1 -- 172.21.15.195:0/1633640095 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f84ec063d00 msgr2=0x7f84ec0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:26.214 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:26.162+0000 7f851196e700 1 --2- 172.21.15.195:0/1633640095 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f84ec063d00 0x7f84ec0661b0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f850c0704f0 tx=0x7f84fc00b040 comp rx=0 tx=0).stop 2022-06-27T12:11:26.215 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:26.162+0000 7f851196e700 1 -- 172.21.15.195:0/1633640095 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f850c0c5620 msgr2=0x7f850c15c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:26.215 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:26.162+0000 7f851196e700 1 --2- 172.21.15.195:0/1633640095 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f850c0c5620 0x7f850c15c150 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f85000119a0 tx=0x7f8500009060 comp rx=0 tx=0).stop 2022-06-27T12:11:26.216 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:26.162+0000 7f851196e700 1 -- 172.21.15.195:0/1633640095 shutdown_connections 2022-06-27T12:11:26.216 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:26.162+0000 7f851196e700 1 --2- 172.21.15.195:0/1633640095 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f84ec063d00 0x7f84ec0661b0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:26.217 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:26.162+0000 7f851196e700 1 --2- 172.21.15.195:0/1633640095 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f850c0c5620 0x7f850c15c150 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:26.217 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:26.162+0000 7f851196e700 1 --2- 172.21.15.195:0/1633640095 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f850c15a900 0x7f850c166e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:26.226 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:26.162+0000 7f851196e700 1 --2- 172.21.15.195:0/1633640095 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f850c0c9ee0 0x7f850c167840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:26.227 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:26.162+0000 7f851196e700 1 -- 172.21.15.195:0/1633640095 >> 172.21.15.195:0/1633640095 conn(0x7f850c063af0 msgr2=0x7f850c0cb280 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:11:26.227 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:26.162+0000 7f851196e700 1 -- 172.21.15.195:0/1633640095 shutdown_connections 2022-06-27T12:11:26.227 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:26.162+0000 7f851196e700 1 -- 172.21.15.195:0/1633640095 wait complete. 2022-06-27T12:11:26.228 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 91 2022-06-27T12:11:26.242 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 91, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-06-27T11:47:52.358095+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 91, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6050": { "addr": "172.21.15.202:6833/3890799919", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6832", "nonce": 3890799919, "type": "v2" }, { "addr": "172.21.15.202:6833", "nonce": 3890799919, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 6050, "incarnation": 91, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 62, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-06-27T12:11:25.147223+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6050 } } } ], "standbys": [ { "addr": "172.21.15.202:6837/3305006841", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6836", "nonce": 3305006841, "type": "v2" }, { "addr": "172.21.15.202:6837", "nonce": 3305006841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6835/2060709822", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6834", "nonce": 2060709822, "type": "v2" }, { "addr": "172.21.15.195:6835", "nonce": 2060709822, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6838/1598751652", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6836", "nonce": 1598751652, "type": "v2" }, { "addr": "172.21.15.195:6838", "nonce": 1598751652, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4275, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6839/1313800661", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6837", "nonce": 1313800661, "type": "v2" }, { "addr": "172.21.15.195:6839", "nonce": 1313800661, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4278, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-06-27T12:11:28.255 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:11:28.422 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.422+0000 7f9e45b23700 1 Processor -- start 2022-06-27T12:11:28.423 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.422+0000 7f9e45b23700 1 -- start start 2022-06-27T12:11:28.435 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.423+0000 7f9e45b23700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f9e400c4160 0x7f9e40156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:28.435 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.423+0000 7f9e45b23700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f9e400c39e0 0x7f9e401535a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:28.435 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.423+0000 7f9e45b23700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e400c1a20 con 0x7f9e4006f640 2022-06-27T12:11:28.436 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.423+0000 7f9e45b23700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e400c1130 con 0x7f9e400c39e0 2022-06-27T12:11:28.436 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.423+0000 7f9e45b23700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e4006ea90 con 0x7f9e400c4160 2022-06-27T12:11:28.436 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.423+0000 7f9e3ffff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f9e400c39e0 0x7f9e401535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:28.436 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.423+0000 7f9e3ffff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f9e400c39e0 0x7f9e401535a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50380/0 (socket says 172.21.15.195:50380) 2022-06-27T12:11:28.437 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.423+0000 7f9e3ffff700 1 -- 172.21.15.195:0/467828059 learned_addr learned my addr 172.21.15.195:0/467828059 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:11:28.437 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.423+0000 7f9e3effd700 1 --2- 172.21.15.195:0/467828059 >> v2:172.21.15.202:3300/0 conn(0x7f9e400c4160 0x7f9e40156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:28.437 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.423+0000 7f9e3ffff700 1 -- 172.21.15.195:0/467828059 >> v2:172.21.15.202:3300/0 conn(0x7f9e400c4160 msgr2=0x7f9e40156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:11:28.437 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.423+0000 7f9e3ffff700 1 --2- 172.21.15.195:0/467828059 >> v2:172.21.15.202:3300/0 conn(0x7f9e400c4160 0x7f9e40156ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:28.438 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.423+0000 7f9e3ffff700 1 -- 172.21.15.195:0/467828059 >> v1:172.21.15.202:6789/0 conn(0x7f9e4006f640 legacy=0x7f9e400c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:11:28.438 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.423+0000 7f9e3ffff700 1 -- 172.21.15.195:0/467828059 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e4006ed70 con 0x7f9e400c39e0 2022-06-27T12:11:28.439 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.423+0000 7f9e3ffff700 1 --2- 172.21.15.195:0/467828059 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9e400c39e0 0x7f9e401535a0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f9e4006dd70 tx=0x7f9e38009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6361a895b3ad057 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:28.439 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.423+0000 7f9e3e7fc700 1 -- 172.21.15.195:0/467828059 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e3800e070 con 0x7f9e400c39e0 2022-06-27T12:11:28.439 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.423+0000 7f9e3e7fc700 1 -- 172.21.15.195:0/467828059 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e3800baa0 con 0x7f9e400c39e0 2022-06-27T12:11:28.439 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.423+0000 7f9e3e7fc700 1 -- 172.21.15.195:0/467828059 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e380092e0 con 0x7f9e400c39e0 2022-06-27T12:11:28.440 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.424+0000 7f9e45b23700 1 -- 172.21.15.195:0/467828059 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9e400c39e0 msgr2=0x7f9e401535a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:28.440 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.424+0000 7f9e45b23700 1 --2- 172.21.15.195:0/467828059 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9e400c39e0 0x7f9e401535a0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f9e4006dd70 tx=0x7f9e38009840 comp rx=0 tx=0).stop 2022-06-27T12:11:28.440 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.424+0000 7f9e45b23700 1 -- 172.21.15.195:0/467828059 shutdown_connections 2022-06-27T12:11:28.440 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.424+0000 7f9e45b23700 1 --2- 172.21.15.195:0/467828059 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9e400c39e0 0x7f9e401535a0 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:28.441 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.424+0000 7f9e45b23700 1 --2- 172.21.15.195:0/467828059 >> v2:172.21.15.202:3300/0 conn(0x7f9e400c4160 0x7f9e40156ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:28.441 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.424+0000 7f9e45b23700 1 -- 172.21.15.195:0/467828059 >> 172.21.15.195:0/467828059 conn(0x7f9e400c6350 msgr2=0x7f9e400c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:11:28.441 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.424+0000 7f9e45b23700 1 -- 172.21.15.195:0/467828059 shutdown_connections 2022-06-27T12:11:28.441 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.424+0000 7f9e45b23700 1 -- 172.21.15.195:0/467828059 wait complete. 2022-06-27T12:11:28.441 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.425+0000 7f9e45b23700 1 Processor -- start 2022-06-27T12:11:28.442 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.425+0000 7f9e45b23700 1 -- start start 2022-06-27T12:11:28.442 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.426+0000 7f9e45b23700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9e4006f640 0x7f9e4014afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:28.442 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.426+0000 7f9e45b23700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9e400c39e0 0x7f9e4014e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:28.443 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.426+0000 7f9e45b23700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9e400c4160 0x7f9e4014ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:28.443 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.426+0000 7f9e45b23700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e40169a30 con 0x7f9e400c4160 2022-06-27T12:11:28.443 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.426+0000 7f9e45b23700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e40169d10 con 0x7f9e400c39e0 2022-06-27T12:11:28.444 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.426+0000 7f9e45b23700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e40169ff0 con 0x7f9e4006f640 2022-06-27T12:11:28.444 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.426+0000 7f9e3ffff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9e400c4160 0x7f9e4014ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:28.445 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.426+0000 7f9e3ffff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9e400c4160 0x7f9e4014ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50386/0 (socket says 172.21.15.195:50386) 2022-06-27T12:11:28.445 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.426+0000 7f9e3ffff700 1 -- 172.21.15.195:0/1447613372 learned_addr learned my addr 172.21.15.195:0/1447613372 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:11:28.445 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.426+0000 7f9e3f7fe700 1 --2- 172.21.15.195:0/1447613372 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9e4006f640 0x7f9e4014afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:28.445 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.426+0000 7f9e3effd700 1 --2- 172.21.15.195:0/1447613372 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9e400c39e0 0x7f9e4014e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:28.446 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.426+0000 7f9e3ffff700 1 -- 172.21.15.195:0/1447613372 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9e4006f640 msgr2=0x7f9e4014afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:28.446 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.426+0000 7f9e3ffff700 1 --2- 172.21.15.195:0/1447613372 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9e4006f640 0x7f9e4014afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:28.446 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.426+0000 7f9e3ffff700 1 -- 172.21.15.195:0/1447613372 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9e400c39e0 msgr2=0x7f9e4014e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:28.447 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.426+0000 7f9e3ffff700 1 --2- 172.21.15.195:0/1447613372 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9e400c39e0 0x7f9e4014e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:28.447 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.426+0000 7f9e3ffff700 1 -- 172.21.15.195:0/1447613372 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e38009ae0 con 0x7f9e400c4160 2022-06-27T12:11:28.447 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.426+0000 7f9e3f7fe700 1 --2- 172.21.15.195:0/1447613372 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9e4006f640 0x7f9e4014afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:11:28.447 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.427+0000 7f9e3ffff700 1 --2- 172.21.15.195:0/1447613372 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9e400c4160 0x7f9e4014ef20 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f9e400650f0 tx=0x7f9e38017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:28.448 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.427+0000 7f9e277fe700 1 -- 172.21.15.195:0/1447613372 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e38004240 con 0x7f9e400c4160 2022-06-27T12:11:28.448 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.427+0000 7f9e277fe700 1 -- 172.21.15.195:0/1447613372 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e380043c0 con 0x7f9e400c4160 2022-06-27T12:11:28.448 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.427+0000 7f9e45b23700 1 -- 172.21.15.195:0/1447613372 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e4016a2d0 con 0x7f9e400c4160 2022-06-27T12:11:28.448 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.427+0000 7f9e277fe700 1 -- 172.21.15.195:0/1447613372 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e38004780 con 0x7f9e400c4160 2022-06-27T12:11:28.450 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.427+0000 7f9e45b23700 1 -- 172.21.15.195:0/1447613372 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e40154af0 con 0x7f9e400c4160 2022-06-27T12:11:28.450 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.428+0000 7f9e45b23700 1 -- 172.21.15.195:0/1447613372 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e400722c0 con 0x7f9e400c4160 2022-06-27T12:11:28.451 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.428+0000 7f9e277fe700 1 -- 172.21.15.195:0/1447613372 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f9e3802c070 con 0x7f9e400c4160 2022-06-27T12:11:28.451 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.429+0000 7f9e277fe700 1 --2- 172.21.15.195:0/1447613372 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9e20063d00 0x7f9e200661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:28.451 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.429+0000 7f9e277fe700 1 -- 172.21.15.195:0/1447613372 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7406+0+0 (secure 0 0 0) 0x7f9e380823a0 con 0x7f9e400c4160 2022-06-27T12:11:28.451 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.429+0000 7f9e3f7fe700 1 --2- 172.21.15.195:0/1447613372 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9e20063d00 0x7f9e200661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:28.452 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.429+0000 7f9e3f7fe700 1 --2- 172.21.15.195:0/1447613372 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9e20063d00 0x7f9e200661b0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f9e30001770 tx=0x7f9e30009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:28.452 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.433+0000 7f9e277fe700 1 -- 172.21.15.195:0/1447613372 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f9e38050a50 con 0x7f9e400c4160 2022-06-27T12:11:28.669 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.656+0000 7f9e45b23700 1 -- 172.21.15.195:0/1447613372 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e4014f5a0 con 0x7f9e400c4160 2022-06-27T12:11:28.670 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.657+0000 7f9e277fe700 1 -- 172.21.15.195:0/1447613372 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+5367 (secure 0 0 0) 0x7f9e3802a090 con 0x7f9e400c4160 2022-06-27T12:11:28.671 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:11:28.671 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:11:25.147223+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6050},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6050":{"gid":6050,"name":"b","rank":0,"incarnation":91,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6833/3890799919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3890799919},{"type":"v1","addr":"172.21.15.202:6833","nonce":3890799919}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:11:28.674 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.659+0000 7f9e45b23700 1 -- 172.21.15.195:0/1447613372 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9e20063d00 msgr2=0x7f9e200661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:28.675 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.659+0000 7f9e45b23700 1 --2- 172.21.15.195:0/1447613372 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9e20063d00 0x7f9e200661b0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f9e30001770 tx=0x7f9e30009040 comp rx=0 tx=0).stop 2022-06-27T12:11:28.675 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.659+0000 7f9e45b23700 1 -- 172.21.15.195:0/1447613372 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9e400c4160 msgr2=0x7f9e4014ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:28.676 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.659+0000 7f9e45b23700 1 --2- 172.21.15.195:0/1447613372 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9e400c4160 0x7f9e4014ef20 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f9e400650f0 tx=0x7f9e38017000 comp rx=0 tx=0).stop 2022-06-27T12:11:28.676 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.659+0000 7f9e45b23700 1 -- 172.21.15.195:0/1447613372 shutdown_connections 2022-06-27T12:11:28.678 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.659+0000 7f9e45b23700 1 --2- 172.21.15.195:0/1447613372 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9e20063d00 0x7f9e200661b0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:28.679 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.659+0000 7f9e45b23700 1 --2- 172.21.15.195:0/1447613372 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9e400c4160 0x7f9e4014ef20 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:28.686 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.659+0000 7f9e45b23700 1 --2- 172.21.15.195:0/1447613372 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9e4006f640 0x7f9e4014afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:28.686 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.659+0000 7f9e45b23700 1 --2- 172.21.15.195:0/1447613372 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9e400c39e0 0x7f9e4014e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:28.687 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.659+0000 7f9e45b23700 1 -- 172.21.15.195:0/1447613372 >> 172.21.15.195:0/1447613372 conn(0x7f9e400c6350 msgr2=0x7f9e400c1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:11:28.687 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.659+0000 7f9e45b23700 1 -- 172.21.15.195:0/1447613372 shutdown_connections 2022-06-27T12:11:28.688 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:28.659+0000 7f9e45b23700 1 -- 172.21.15.195:0/1447613372 wait complete. 2022-06-27T12:11:28.688 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 91 2022-06-27T12:11:28.689 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:11:30.694 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:11:30.846 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:30.824+0000 7fef59032700 1 Processor -- start 2022-06-27T12:11:30.847 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:30.824+0000 7fef59032700 1 -- start start 2022-06-27T12:11:30.847 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:30.824+0000 7fef59032700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fef54150950 0x7fef54156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:30.847 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:30.824+0000 7fef59032700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef540c76b0 con 0x7fef5406f620 2022-06-27T12:11:30.848 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:30.824+0000 7fef59032700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef540c8250 con 0x7fef54150580 2022-06-27T12:11:30.848 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:30.824+0000 7fef59032700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef54157430 con 0x7fef54150950 2022-06-27T12:11:30.848 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:30.825+0000 7fef5359e700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fef5406f620 0x7fef5406d6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:56580/0 (socket says 172.21.15.195:56580) 2022-06-27T12:11:30.848 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:30.825+0000 7fef5359e700 1 -- 172.21.15.195:0/2248142392 learned_addr learned my addr 172.21.15.195:0/2248142392 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:11:30.849 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:30.825+0000 7fef5259c700 1 --2- 172.21.15.195:0/2248142392 >> v2:172.21.15.202:3300/0 conn(0x7fef54150950 0x7fef54156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:30.849 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:30.825+0000 7fef51d9b700 1 -- 172.21.15.195:0/2248142392 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3597780192 0 0) 0x7fef540c76b0 con 0x7fef5406f620 2022-06-27T12:11:30.849 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:30.825+0000 7fef51d9b700 1 -- 172.21.15.195:0/2248142392 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef40003430 con 0x7fef5406f620 2022-06-27T12:11:30.849 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:30.825+0000 7fef51d9b700 1 -- 172.21.15.195:0/2248142392 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3553615929 0 0) 0x7fef540c8250 con 0x7fef54150580 2022-06-27T12:11:30.850 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:30.825+0000 7fef51d9b700 1 -- 172.21.15.195:0/2248142392 --> v1:172.21.15.202:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef540c76b0 con 0x7fef54150580 2022-06-27T12:11:30.872 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:30.825+0000 7fef5259c700 1 -- 172.21.15.195:0/2248142392 >> v1:172.21.15.202:6789/0 conn(0x7fef54150580 legacy=0x7fef540c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:30.873 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:30.825+0000 7fef5259c700 1 -- 172.21.15.195:0/2248142392 >> v1:172.21.15.195:6789/0 conn(0x7fef5406f620 legacy=0x7fef5406d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:30.873 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:30.825+0000 7fef5259c700 1 -- 172.21.15.195:0/2248142392 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef540bad70 con 0x7fef54150950 2022-06-27T12:11:30.873 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:30.825+0000 7fef5259c700 1 --2- 172.21.15.195:0/2248142392 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fef54150950 0x7fef54156ef0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fef440097a0 tx=0x7fef44009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1cba9a404adff744 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:30.873 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:30.825+0000 7fef51d9b700 1 -- 172.21.15.195:0/2248142392 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef44011630 con 0x7fef54150950 2022-06-27T12:11:30.873 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:30.825+0000 7fef51d9b700 1 -- 172.21.15.195:0/2248142392 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef44011790 con 0x7fef54150950 2022-06-27T12:11:30.874 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:30.825+0000 7fef51d9b700 1 -- 172.21.15.195:0/2248142392 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef44011b10 con 0x7fef54150950 2022-06-27T12:11:30.874 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:30.826+0000 7fef59032700 1 -- 172.21.15.195:0/2248142392 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fef54150950 msgr2=0x7fef54156ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:30.874 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:30.826+0000 7fef59032700 1 --2- 172.21.15.195:0/2248142392 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fef54150950 0x7fef54156ef0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fef440097a0 tx=0x7fef44009060 comp rx=0 tx=0).stop 2022-06-27T12:11:30.874 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:30.826+0000 7fef59032700 1 -- 172.21.15.195:0/2248142392 shutdown_connections 2022-06-27T12:11:30.875 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:30.826+0000 7fef59032700 1 --2- 172.21.15.195:0/2248142392 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fef54150950 0x7fef54156ef0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:30.875 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:30.826+0000 7fef59032700 1 -- 172.21.15.195:0/2248142392 >> 172.21.15.195:0/2248142392 conn(0x7fef54063af0 msgr2=0x7fef54063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:11:30.875 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:30.826+0000 7fef59032700 1 -- 172.21.15.195:0/2248142392 shutdown_connections 2022-06-27T12:11:30.875 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:30.826+0000 7fef59032700 1 -- 172.21.15.195:0/2248142392 wait complete. 2022-06-27T12:11:30.876 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:30.827+0000 7fef59032700 1 Processor -- start 2022-06-27T12:11:30.876 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:30.827+0000 7fef59032700 1 -- start start 2022-06-27T12:11:30.876 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:30.828+0000 7fef59032700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fef5406f620 0x7fef5415b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:30.876 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:30.828+0000 7fef59032700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fef54150580 0x7fef54166ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:30.876 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:30.828+0000 7fef59032700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fef54150950 0x7fef5414b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:30.877 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:30.828+0000 7fef59032700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef540bbd80 con 0x7fef5406f620 2022-06-27T12:11:30.877 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:30.828+0000 7fef59032700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef54159ff0 con 0x7fef54150950 2022-06-27T12:11:30.877 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:30.828+0000 7fef59032700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef54159470 con 0x7fef54150580 2022-06-27T12:11:30.877 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:30.828+0000 7fef52d9d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fef5406f620 0x7fef5415b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:30.878 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:30.828+0000 7fef52d9d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fef5406f620 0x7fef5415b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50394/0 (socket says 172.21.15.195:50394) 2022-06-27T12:11:30.879 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:30.828+0000 7fef52d9d700 1 -- 172.21.15.195:0/2390992967 learned_addr learned my addr 172.21.15.195:0/2390992967 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:11:30.879 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:30.828+0000 7fef5359e700 1 --2- 172.21.15.195:0/2390992967 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fef54150950 0x7fef5414b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:30.879 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:30.828+0000 7fef5259c700 1 --2- 172.21.15.195:0/2390992967 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fef54150580 0x7fef54166ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:30.879 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:30.828+0000 7fef52d9d700 1 -- 172.21.15.195:0/2390992967 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fef54150580 msgr2=0x7fef54166ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:30.880 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:30.828+0000 7fef52d9d700 1 --2- 172.21.15.195:0/2390992967 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fef54150580 0x7fef54166ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:30.880 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:30.828+0000 7fef52d9d700 1 -- 172.21.15.195:0/2390992967 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fef54150950 msgr2=0x7fef5414b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:30.880 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:30.828+0000 7fef52d9d700 1 --2- 172.21.15.195:0/2390992967 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fef54150950 0x7fef5414b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:30.881 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:30.828+0000 7fef52d9d700 1 -- 172.21.15.195:0/2390992967 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef440092e0 con 0x7fef5406f620 2022-06-27T12:11:30.882 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:30.828+0000 7fef5359e700 1 --2- 172.21.15.195:0/2390992967 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fef54150950 0x7fef5414b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:11:30.882 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:30.829+0000 7fef52d9d700 1 --2- 172.21.15.195:0/2390992967 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fef5406f620 0x7fef5415b560 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fef4800c4c0 tx=0x7fef48007c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:30.882 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:30.829+0000 7fef3affd700 1 -- 172.21.15.195:0/2390992967 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef4800ce30 con 0x7fef5406f620 2022-06-27T12:11:30.882 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:30.829+0000 7fef3affd700 1 -- 172.21.15.195:0/2390992967 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef480050d0 con 0x7fef5406f620 2022-06-27T12:11:30.883 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:30.829+0000 7fef59032700 1 -- 172.21.15.195:0/2390992967 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef54159750 con 0x7fef5406f620 2022-06-27T12:11:30.883 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:30.829+0000 7fef3affd700 1 -- 172.21.15.195:0/2390992967 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef48011590 con 0x7fef5406f620 2022-06-27T12:11:30.883 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:30.829+0000 7fef59032700 1 -- 172.21.15.195:0/2390992967 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef54159d10 con 0x7fef5406f620 2022-06-27T12:11:30.883 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:30.830+0000 7fef59032700 1 -- 172.21.15.195:0/2390992967 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef540722c0 con 0x7fef5406f620 2022-06-27T12:11:30.883 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:30.830+0000 7fef3affd700 1 -- 172.21.15.195:0/2390992967 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fef48019030 con 0x7fef5406f620 2022-06-27T12:11:30.884 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:30.831+0000 7fef3affd700 1 --2- 172.21.15.195:0/2390992967 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fef34063d00 0x7fef340661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:30.898 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:30.831+0000 7fef3affd700 1 -- 172.21.15.195:0/2390992967 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7406+0+0 (secure 0 0 0) 0x7fef4807fec0 con 0x7fef5406f620 2022-06-27T12:11:30.899 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:30.831+0000 7fef5259c700 1 --2- 172.21.15.195:0/2390992967 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fef34063d00 0x7fef340661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:30.899 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:30.831+0000 7fef5259c700 1 --2- 172.21.15.195:0/2390992967 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fef34063d00 0x7fef340661b0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fef540bb120 tx=0x7fef44017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:30.899 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:30.835+0000 7fef3affd700 1 -- 172.21.15.195:0/2390992967 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fef4804e510 con 0x7fef5406f620 2022-06-27T12:11:31.055 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:31.053+0000 7fef59032700 1 -- 172.21.15.195:0/2390992967 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef5406aa60 con 0x7fef5406f620 2022-06-27T12:11:31.055 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:31.053+0000 7fef3affd700 1 -- 172.21.15.195:0/2390992967 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+5367 (secure 0 0 0) 0x7fef48017090 con 0x7fef5406f620 2022-06-27T12:11:31.069 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:11:31.070 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:11:25.147223+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6050},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6050":{"gid":6050,"name":"b","rank":0,"incarnation":91,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6833/3890799919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3890799919},{"type":"v1","addr":"172.21.15.202:6833","nonce":3890799919}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:11:31.074 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:31.055+0000 7fef59032700 1 -- 172.21.15.195:0/2390992967 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fef34063d00 msgr2=0x7fef340661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:31.075 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:31.055+0000 7fef59032700 1 --2- 172.21.15.195:0/2390992967 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fef34063d00 0x7fef340661b0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fef540bb120 tx=0x7fef44017000 comp rx=0 tx=0).stop 2022-06-27T12:11:31.075 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:31.055+0000 7fef59032700 1 -- 172.21.15.195:0/2390992967 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fef5406f620 msgr2=0x7fef5415b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:31.076 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:31.055+0000 7fef59032700 1 --2- 172.21.15.195:0/2390992967 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fef5406f620 0x7fef5415b560 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fef4800c4c0 tx=0x7fef48007c70 comp rx=0 tx=0).stop 2022-06-27T12:11:31.076 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:31.055+0000 7fef59032700 1 -- 172.21.15.195:0/2390992967 shutdown_connections 2022-06-27T12:11:31.077 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:31.055+0000 7fef59032700 1 --2- 172.21.15.195:0/2390992967 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fef34063d00 0x7fef340661b0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:31.077 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:31.055+0000 7fef59032700 1 --2- 172.21.15.195:0/2390992967 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fef5406f620 0x7fef5415b560 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:31.078 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:31.055+0000 7fef59032700 1 --2- 172.21.15.195:0/2390992967 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fef54150950 0x7fef5414b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:31.078 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:31.055+0000 7fef59032700 1 --2- 172.21.15.195:0/2390992967 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fef54150580 0x7fef54166ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:31.079 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:31.055+0000 7fef59032700 1 -- 172.21.15.195:0/2390992967 >> 172.21.15.195:0/2390992967 conn(0x7fef54063af0 msgr2=0x7fef5414e720 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:11:31.079 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:31.055+0000 7fef59032700 1 -- 172.21.15.195:0/2390992967 shutdown_connections 2022-06-27T12:11:31.080 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:31.055+0000 7fef59032700 1 -- 172.21.15.195:0/2390992967 wait complete. 2022-06-27T12:11:31.080 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 91 2022-06-27T12:11:31.081 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:11:32.878 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:11:32.901 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:11:33.096 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:11:33.263 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.229+0000 7f7eda21b700 1 Processor -- start 2022-06-27T12:11:33.264 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.229+0000 7f7eda21b700 1 -- start start 2022-06-27T12:11:33.264 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.230+0000 7f7eda21b700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7ed40c4160 0x7f7ed4156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:33.265 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.230+0000 7f7eda21b700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ed40c1a20 con 0x7f7ed40c39e0 2022-06-27T12:11:33.265 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.230+0000 7f7eda21b700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ed40c1130 con 0x7f7ed406f640 2022-06-27T12:11:33.282 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.230+0000 7f7eda21b700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ed406ea90 con 0x7f7ed40c4160 2022-06-27T12:11:33.283 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.230+0000 7f7ed3fff700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f7ed40c39e0 0x7f7ed41535a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:56594/0 (socket says 172.21.15.195:56594) 2022-06-27T12:11:33.283 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.230+0000 7f7ed3fff700 1 -- 172.21.15.195:0/2955631508 learned_addr learned my addr 172.21.15.195:0/2955631508 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:11:33.284 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.230+0000 7f7ecbfff700 1 --2- 172.21.15.195:0/2955631508 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7ed40c4160 0x7f7ed4156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:33.284 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.231+0000 7f7ed2ffd700 1 -- 172.21.15.195:0/2955631508 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2153746691 0 0) 0x7f7ed40c1a20 con 0x7f7ed40c39e0 2022-06-27T12:11:33.285 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.231+0000 7f7ed2ffd700 1 -- 172.21.15.195:0/2955631508 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7eb4003430 con 0x7f7ed40c39e0 2022-06-27T12:11:33.285 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.231+0000 7f7ed2ffd700 1 -- 172.21.15.195:0/2955631508 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4026437044 0 0) 0x7f7ed40c1130 con 0x7f7ed406f640 2022-06-27T12:11:33.285 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.231+0000 7f7ed2ffd700 1 -- 172.21.15.195:0/2955631508 --> v1:172.21.15.202:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ed40c1a20 con 0x7f7ed406f640 2022-06-27T12:11:33.286 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.231+0000 7f7ecbfff700 1 -- 172.21.15.195:0/2955631508 >> v1:172.21.15.202:6789/0 conn(0x7f7ed406f640 legacy=0x7f7ed40c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:33.286 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.231+0000 7f7ecbfff700 1 -- 172.21.15.195:0/2955631508 >> v1:172.21.15.195:6789/0 conn(0x7f7ed40c39e0 legacy=0x7f7ed41535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:33.287 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.231+0000 7f7ecbfff700 1 -- 172.21.15.195:0/2955631508 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ed406ed70 con 0x7f7ed40c4160 2022-06-27T12:11:33.287 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.231+0000 7f7ecbfff700 1 --2- 172.21.15.195:0/2955631508 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7ed40c4160 0x7f7ed4156ed0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc0097d0 tx=0x7f7ebc009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7fa3ce2bb81931e server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:33.288 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.231+0000 7f7ed2ffd700 1 -- 172.21.15.195:0/2955631508 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ebc011630 con 0x7f7ed40c4160 2022-06-27T12:11:33.288 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.231+0000 7f7ed2ffd700 1 -- 172.21.15.195:0/2955631508 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ebc011790 con 0x7f7ed40c4160 2022-06-27T12:11:33.289 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.231+0000 7f7ed2ffd700 1 -- 172.21.15.195:0/2955631508 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ebc011b30 con 0x7f7ed40c4160 2022-06-27T12:11:33.289 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.232+0000 7f7eda21b700 1 -- 172.21.15.195:0/2955631508 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7ed40c4160 msgr2=0x7f7ed4156ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:33.289 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.232+0000 7f7eda21b700 1 --2- 172.21.15.195:0/2955631508 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7ed40c4160 0x7f7ed4156ed0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc0097d0 tx=0x7f7ebc009060 comp rx=0 tx=0).stop 2022-06-27T12:11:33.290 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.232+0000 7f7eda21b700 1 -- 172.21.15.195:0/2955631508 shutdown_connections 2022-06-27T12:11:33.291 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.232+0000 7f7eda21b700 1 --2- 172.21.15.195:0/2955631508 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7ed40c4160 0x7f7ed4156ed0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:33.291 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.232+0000 7f7eda21b700 1 -- 172.21.15.195:0/2955631508 >> 172.21.15.195:0/2955631508 conn(0x7f7ed40c6350 msgr2=0x7f7ed40c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:11:33.292 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.232+0000 7f7eda21b700 1 -- 172.21.15.195:0/2955631508 shutdown_connections 2022-06-27T12:11:33.292 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.232+0000 7f7eda21b700 1 -- 172.21.15.195:0/2955631508 wait complete. 2022-06-27T12:11:33.293 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.233+0000 7f7eda21b700 1 Processor -- start 2022-06-27T12:11:33.293 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.233+0000 7f7eda21b700 1 -- start start 2022-06-27T12:11:33.293 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.234+0000 7f7eda21b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7ed406f640 0x7f7ed4153550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:33.294 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.234+0000 7f7eda21b700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7ed40c39e0 0x7f7ed4169a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:33.294 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.234+0000 7f7eda21b700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7ed40c4160 0x7f7ed414e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:33.295 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.234+0000 7f7eda21b700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ed4068670 con 0x7f7ed406f640 2022-06-27T12:11:33.295 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.234+0000 7f7eda21b700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ed4067ed0 con 0x7f7ed40c39e0 2022-06-27T12:11:33.296 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.234+0000 7f7eda21b700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ed40c9080 con 0x7f7ed40c4160 2022-06-27T12:11:33.296 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.234+0000 7f7ed37fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7ed406f640 0x7f7ed4153550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:33.296 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.234+0000 7f7ed37fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7ed406f640 0x7f7ed4153550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50408/0 (socket says 172.21.15.195:50408) 2022-06-27T12:11:33.297 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.234+0000 7f7ed37fe700 1 -- 172.21.15.195:0/3146244200 learned_addr learned my addr 172.21.15.195:0/3146244200 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:11:33.297 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.234+0000 7f7ed3fff700 1 --2- 172.21.15.195:0/3146244200 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7ed40c4160 0x7f7ed414e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:33.297 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.234+0000 7f7ecbfff700 1 --2- 172.21.15.195:0/3146244200 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7ed40c39e0 0x7f7ed4169a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:33.298 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.234+0000 7f7ed37fe700 1 -- 172.21.15.195:0/3146244200 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7ed40c4160 msgr2=0x7f7ed414e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:33.298 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.234+0000 7f7ed37fe700 1 --2- 172.21.15.195:0/3146244200 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7ed40c4160 0x7f7ed414e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:33.299 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.234+0000 7f7ed37fe700 1 -- 172.21.15.195:0/3146244200 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7ed40c39e0 msgr2=0x7f7ed4169a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:33.299 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.234+0000 7f7ed37fe700 1 --2- 172.21.15.195:0/3146244200 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7ed40c39e0 0x7f7ed4169a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:33.299 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.234+0000 7f7ed37fe700 1 -- 172.21.15.195:0/3146244200 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ebc0092e0 con 0x7f7ed406f640 2022-06-27T12:11:33.300 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.234+0000 7f7ed37fe700 1 --2- 172.21.15.195:0/3146244200 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7ed406f640 0x7f7ed4153550 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f7ec400c590 tx=0x7f7ec4007cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:33.322 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.234+0000 7f7ecb7fe700 1 -- 172.21.15.195:0/3146244200 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ec40050d0 con 0x7f7ed406f640 2022-06-27T12:11:33.322 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.234+0000 7f7ecb7fe700 1 -- 172.21.15.195:0/3146244200 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ec4005250 con 0x7f7ed406f640 2022-06-27T12:11:33.330 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.234+0000 7f7eda21b700 1 -- 172.21.15.195:0/3146244200 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ed4152aa0 con 0x7f7ed406f640 2022-06-27T12:11:33.331 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.234+0000 7f7ecb7fe700 1 -- 172.21.15.195:0/3146244200 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ec40117e0 con 0x7f7ed406f640 2022-06-27T12:11:33.331 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.235+0000 7f7eda21b700 1 -- 172.21.15.195:0/3146244200 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ed4152f30 con 0x7f7ed406f640 2022-06-27T12:11:33.332 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.235+0000 7f7eda21b700 1 -- 172.21.15.195:0/3146244200 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ed40722c0 con 0x7f7ed406f640 2022-06-27T12:11:33.332 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.236+0000 7f7ecb7fe700 1 -- 172.21.15.195:0/3146244200 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f7ec4019030 con 0x7f7ed406f640 2022-06-27T12:11:33.333 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.236+0000 7f7ecb7fe700 1 --2- 172.21.15.195:0/3146244200 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7eb8063d30 0x7f7eb80661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:33.338 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.236+0000 7f7ecb7fe700 1 -- 172.21.15.195:0/3146244200 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7406+0+0 (secure 0 0 0) 0x7f7ec407ff90 con 0x7f7ed406f640 2022-06-27T12:11:33.339 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.236+0000 7f7ecbfff700 1 --2- 172.21.15.195:0/3146244200 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7eb8063d30 0x7f7eb80661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:33.339 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.237+0000 7f7ecbfff700 1 --2- 172.21.15.195:0/3146244200 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7eb8063d30 0x7f7eb80661e0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc009720 tx=0x7f7ebc017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:33.339 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.241+0000 7f7ecb7fe700 1 -- 172.21.15.195:0/3146244200 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f7ec404e640 con 0x7f7ed406f640 2022-06-27T12:11:33.469 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.469+0000 7f7eda21b700 1 -- 172.21.15.195:0/3146244200 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7ed40c1330 con 0x7f7ed406f640 2022-06-27T12:11:33.470 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.469+0000 7f7ecb7fe700 1 -- 172.21.15.195:0/3146244200 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+5367 (secure 0 0 0) 0x7f7ec4017020 con 0x7f7ed406f640 2022-06-27T12:11:33.470 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:11:33.471 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:11:25.147223+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6050},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6050":{"gid":6050,"name":"b","rank":0,"incarnation":91,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6833/3890799919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3890799919},{"type":"v1","addr":"172.21.15.202:6833","nonce":3890799919}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:11:33.483 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.471+0000 7f7eda21b700 1 -- 172.21.15.195:0/3146244200 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7eb8063d30 msgr2=0x7f7eb80661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:33.483 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.471+0000 7f7eda21b700 1 --2- 172.21.15.195:0/3146244200 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7eb8063d30 0x7f7eb80661e0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc009720 tx=0x7f7ebc017000 comp rx=0 tx=0).stop 2022-06-27T12:11:33.484 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.471+0000 7f7eda21b700 1 -- 172.21.15.195:0/3146244200 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7ed406f640 msgr2=0x7f7ed4153550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:33.484 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.471+0000 7f7eda21b700 1 --2- 172.21.15.195:0/3146244200 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7ed406f640 0x7f7ed4153550 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f7ec400c590 tx=0x7f7ec4007cb0 comp rx=0 tx=0).stop 2022-06-27T12:11:33.484 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.471+0000 7f7eda21b700 1 -- 172.21.15.195:0/3146244200 shutdown_connections 2022-06-27T12:11:33.485 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.471+0000 7f7eda21b700 1 --2- 172.21.15.195:0/3146244200 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7eb8063d30 0x7f7eb80661e0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:33.485 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.471+0000 7f7eda21b700 1 --2- 172.21.15.195:0/3146244200 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7ed406f640 0x7f7ed4153550 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:33.485 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.471+0000 7f7eda21b700 1 --2- 172.21.15.195:0/3146244200 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7ed40c4160 0x7f7ed414e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:33.485 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.471+0000 7f7eda21b700 1 --2- 172.21.15.195:0/3146244200 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7ed40c39e0 0x7f7ed4169a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:33.486 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.471+0000 7f7eda21b700 1 -- 172.21.15.195:0/3146244200 >> 172.21.15.195:0/3146244200 conn(0x7f7ed40c6350 msgr2=0x7f7ed41505e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:11:33.486 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.471+0000 7f7eda21b700 1 -- 172.21.15.195:0/3146244200 shutdown_connections 2022-06-27T12:11:33.486 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:33.471+0000 7f7eda21b700 1 -- 172.21.15.195:0/3146244200 wait complete. 2022-06-27T12:11:33.486 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 91 2022-06-27T12:11:33.496 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:11:35.498 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:11:35.627 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.625+0000 7f2bba33c700 1 Processor -- start 2022-06-27T12:11:35.628 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.626+0000 7f2bba33c700 1 -- start start 2022-06-27T12:11:35.628 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.626+0000 7f2bba33c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f2bb406f640 0x7f2bb4156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:35.641 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.626+0000 7f2bba33c700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2bb40c4160 0x7f2bb40c1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:35.641 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.626+0000 7f2bba33c700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f2bb40c39e0 0x7f2bb41535a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:35.642 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.626+0000 7f2bba33c700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bb40c1a20 con 0x7f2bb406f640 2022-06-27T12:11:35.642 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.626+0000 7f2bba33c700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bb40c1130 con 0x7f2bb40c39e0 2022-06-27T12:11:35.643 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.626+0000 7f2bba33c700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb406ea90 con 0x7f2bb40c4160 2022-06-27T12:11:35.643 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.626+0000 7f2bb3fff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f2bb406f640 0x7f2bb4156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:35.643 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.627+0000 7f2bb3fff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f2bb406f640 0x7f2bb4156ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50416/0 (socket says 172.21.15.195:50416) 2022-06-27T12:11:35.644 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.627+0000 7f2bb3fff700 1 -- 172.21.15.195:0/555027286 learned_addr learned my addr 172.21.15.195:0/555027286 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:11:35.644 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.627+0000 7f2bb37fe700 1 --2- 172.21.15.195:0/555027286 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2bb40c4160 0x7f2bb40c1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:35.645 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.627+0000 7f2bb88d9700 1 --2- 172.21.15.195:0/555027286 >> v2:172.21.15.202:3300/0 conn(0x7f2bb40c39e0 0x7f2bb41535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:35.645 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.627+0000 7f2bb3fff700 1 -- 172.21.15.195:0/555027286 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2bb40c4160 msgr2=0x7f2bb40c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:11:35.645 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.627+0000 7f2bb3fff700 1 --2- 172.21.15.195:0/555027286 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2bb40c4160 0x7f2bb40c1eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:35.647 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.627+0000 7f2bb3fff700 1 -- 172.21.15.195:0/555027286 >> v2:172.21.15.202:3300/0 conn(0x7f2bb40c39e0 msgr2=0x7f2bb41535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:11:35.647 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.627+0000 7f2bb3fff700 1 --2- 172.21.15.195:0/555027286 >> v2:172.21.15.202:3300/0 conn(0x7f2bb40c39e0 0x7f2bb41535a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:35.648 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.627+0000 7f2bb3fff700 1 -- 172.21.15.195:0/555027286 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bb406ed70 con 0x7f2bb406f640 2022-06-27T12:11:35.648 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.627+0000 7f2bb3fff700 1 --2- 172.21.15.195:0/555027286 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2bb406f640 0x7f2bb4156ed0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f2bb406b530 tx=0x7f2ba4009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68365c91f118f6f server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:35.649 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.627+0000 7f2bb2ffd700 1 -- 172.21.15.195:0/555027286 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba4011630 con 0x7f2bb406f640 2022-06-27T12:11:35.649 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.627+0000 7f2bb2ffd700 1 -- 172.21.15.195:0/555027286 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ba4011790 con 0x7f2bb406f640 2022-06-27T12:11:35.650 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.627+0000 7f2bb2ffd700 1 -- 172.21.15.195:0/555027286 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba4011b30 con 0x7f2bb406f640 2022-06-27T12:11:35.650 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.628+0000 7f2bba33c700 1 -- 172.21.15.195:0/555027286 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2bb406f640 msgr2=0x7f2bb4156ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:35.651 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.628+0000 7f2bba33c700 1 --2- 172.21.15.195:0/555027286 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2bb406f640 0x7f2bb4156ed0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f2bb406b530 tx=0x7f2ba4009120 comp rx=0 tx=0).stop 2022-06-27T12:11:35.658 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.628+0000 7f2bba33c700 1 -- 172.21.15.195:0/555027286 shutdown_connections 2022-06-27T12:11:35.659 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.628+0000 7f2bba33c700 1 --2- 172.21.15.195:0/555027286 >> v2:172.21.15.202:3300/0 conn(0x7f2bb40c39e0 0x7f2bb41535a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:35.659 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.628+0000 7f2bba33c700 1 --2- 172.21.15.195:0/555027286 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2bb406f640 0x7f2bb4156ed0 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:35.659 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.628+0000 7f2bba33c700 1 --2- 172.21.15.195:0/555027286 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2bb40c4160 0x7f2bb40c1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:35.660 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.628+0000 7f2bba33c700 1 -- 172.21.15.195:0/555027286 >> 172.21.15.195:0/555027286 conn(0x7f2bb40c6350 msgr2=0x7f2bb40c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:11:35.660 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.628+0000 7f2bba33c700 1 -- 172.21.15.195:0/555027286 shutdown_connections 2022-06-27T12:11:35.661 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.628+0000 7f2bba33c700 1 -- 172.21.15.195:0/555027286 wait complete. 2022-06-27T12:11:35.661 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.629+0000 7f2bba33c700 1 Processor -- start 2022-06-27T12:11:35.662 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.629+0000 7f2bba33c700 1 -- start start 2022-06-27T12:11:35.662 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.630+0000 7f2bba33c700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2bb406f640 0x7f2bb4152790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:35.663 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.630+0000 7f2bba33c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2bb40c39e0 0x7f2bb40c88a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:35.663 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.630+0000 7f2bba33c700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2bb40c4160 0x7f2bb4166c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:35.663 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.630+0000 7f2bba33c700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb4154ba0 con 0x7f2bb40c39e0 2022-06-27T12:11:35.664 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.630+0000 7f2bba33c700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb4152d80 con 0x7f2bb40c4160 2022-06-27T12:11:35.664 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.630+0000 7f2bba33c700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb416a370 con 0x7f2bb406f640 2022-06-27T12:11:35.665 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.630+0000 7f2bb37fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2bb40c39e0 0x7f2bb40c88a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:35.665 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.630+0000 7f2bb37fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2bb40c39e0 0x7f2bb40c88a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50422/0 (socket says 172.21.15.195:50422) 2022-06-27T12:11:35.666 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.630+0000 7f2bb3fff700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2bb406f640 0x7f2bb4152790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:35.666 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.630+0000 7f2bb37fe700 1 -- 172.21.15.195:0/1570740239 learned_addr learned my addr 172.21.15.195:0/1570740239 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:11:35.667 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.630+0000 7f2bb88d9700 1 --2- 172.21.15.195:0/1570740239 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2bb40c4160 0x7f2bb4166c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:35.668 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.630+0000 7f2bb37fe700 1 -- 172.21.15.195:0/1570740239 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2bb406f640 msgr2=0x7f2bb4152790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:35.668 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.630+0000 7f2bb37fe700 1 --2- 172.21.15.195:0/1570740239 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2bb406f640 0x7f2bb4152790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:35.669 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.630+0000 7f2bb37fe700 1 -- 172.21.15.195:0/1570740239 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2bb40c4160 msgr2=0x7f2bb4166c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:35.670 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.630+0000 7f2bb37fe700 1 --2- 172.21.15.195:0/1570740239 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2bb40c4160 0x7f2bb4166c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:35.670 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.630+0000 7f2bb37fe700 1 -- 172.21.15.195:0/1570740239 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ba40093a0 con 0x7f2bb40c39e0 2022-06-27T12:11:35.671 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.630+0000 7f2bb37fe700 1 --2- 172.21.15.195:0/1570740239 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2bb40c39e0 0x7f2bb40c88a0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f2ba800e900 tx=0x7f2ba8009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:35.671 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.630+0000 7f2b9bfff700 1 -- 172.21.15.195:0/1570740239 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba8013680 con 0x7f2bb40c39e0 2022-06-27T12:11:35.671 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.630+0000 7f2b9bfff700 1 -- 172.21.15.195:0/1570740239 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ba8013800 con 0x7f2bb40c39e0 2022-06-27T12:11:35.672 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.630+0000 7f2bba33c700 1 -- 172.21.15.195:0/1570740239 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bb416a650 con 0x7f2bb40c39e0 2022-06-27T12:11:35.672 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.630+0000 7f2b9bfff700 1 -- 172.21.15.195:0/1570740239 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba8013bc0 con 0x7f2bb40c39e0 2022-06-27T12:11:35.673 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.630+0000 7f2bba33c700 1 -- 172.21.15.195:0/1570740239 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bb416ac10 con 0x7f2bb40c39e0 2022-06-27T12:11:35.673 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.631+0000 7f2bba33c700 1 -- 172.21.15.195:0/1570740239 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bb40722c0 con 0x7f2bb40c39e0 2022-06-27T12:11:35.674 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.632+0000 7f2b9bfff700 1 -- 172.21.15.195:0/1570740239 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f2ba8019030 con 0x7f2bb40c39e0 2022-06-27T12:11:35.674 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.632+0000 7f2b9bfff700 1 --2- 172.21.15.195:0/1570740239 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2b94063d30 0x7f2b940661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:35.675 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.632+0000 7f2b9bfff700 1 -- 172.21.15.195:0/1570740239 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7406+0+0 (secure 0 0 0) 0x7f2ba8080c00 con 0x7f2bb40c39e0 2022-06-27T12:11:35.675 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.632+0000 7f2bb3fff700 1 --2- 172.21.15.195:0/1570740239 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2b94063d30 0x7f2b940661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:35.676 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.632+0000 7f2bb3fff700 1 --2- 172.21.15.195:0/1570740239 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2b94063d30 0x7f2b940661e0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f2ba4009160 tx=0x7f2ba4017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:35.676 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.636+0000 7f2b9bfff700 1 -- 172.21.15.195:0/1570740239 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f2ba8086020 con 0x7f2bb40c39e0 2022-06-27T12:11:35.861 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.861+0000 7f2bba33c700 1 -- 172.21.15.195:0/1570740239 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2bb40c1770 con 0x7f2bb40c39e0 2022-06-27T12:11:35.863 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.861+0000 7f2b9bfff700 1 -- 172.21.15.195:0/1570740239 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+5367 (secure 0 0 0) 0x7f2ba8017090 con 0x7f2bb40c39e0 2022-06-27T12:11:35.864 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:11:35.864 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:11:25.147223+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6050},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6050":{"gid":6050,"name":"b","rank":0,"incarnation":91,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6833/3890799919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3890799919},{"type":"v1","addr":"172.21.15.202:6833","nonce":3890799919}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:11:35.865 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.863+0000 7f2bba33c700 1 -- 172.21.15.195:0/1570740239 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2b94063d30 msgr2=0x7f2b940661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:35.875 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.863+0000 7f2bba33c700 1 --2- 172.21.15.195:0/1570740239 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2b94063d30 0x7f2b940661e0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f2ba4009160 tx=0x7f2ba4017000 comp rx=0 tx=0).stop 2022-06-27T12:11:35.875 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.863+0000 7f2bba33c700 1 -- 172.21.15.195:0/1570740239 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2bb40c39e0 msgr2=0x7f2bb40c88a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:35.875 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.863+0000 7f2bba33c700 1 --2- 172.21.15.195:0/1570740239 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2bb40c39e0 0x7f2bb40c88a0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f2ba800e900 tx=0x7f2ba8009060 comp rx=0 tx=0).stop 2022-06-27T12:11:35.876 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.864+0000 7f2bba33c700 1 -- 172.21.15.195:0/1570740239 shutdown_connections 2022-06-27T12:11:35.876 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.864+0000 7f2bba33c700 1 --2- 172.21.15.195:0/1570740239 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2b94063d30 0x7f2b940661e0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:35.876 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.864+0000 7f2bba33c700 1 --2- 172.21.15.195:0/1570740239 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2bb40c4160 0x7f2bb4166c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:35.876 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.864+0000 7f2bba33c700 1 --2- 172.21.15.195:0/1570740239 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2bb406f640 0x7f2bb4152790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:35.877 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.864+0000 7f2bba33c700 1 --2- 172.21.15.195:0/1570740239 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2bb40c39e0 0x7f2bb40c88a0 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:35.877 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.864+0000 7f2bba33c700 1 -- 172.21.15.195:0/1570740239 >> 172.21.15.195:0/1570740239 conn(0x7f2bb40c6350 msgr2=0x7f2bb41550a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:11:35.877 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.864+0000 7f2bba33c700 1 -- 172.21.15.195:0/1570740239 shutdown_connections 2022-06-27T12:11:35.877 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:35.864+0000 7f2bba33c700 1 -- 172.21.15.195:0/1570740239 wait complete. 2022-06-27T12:11:35.878 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 91 2022-06-27T12:11:35.879 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:11:37.894 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:11:38.053 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.029+0000 7f2fbb232700 1 Processor -- start 2022-06-27T12:11:38.053 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.029+0000 7f2fbb232700 1 -- start start 2022-06-27T12:11:38.058 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.029+0000 7f2fbb232700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f2fb4153cb0 0x7f2fb4156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:38.058 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.030+0000 7f2fbb232700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fb40c1a20 con 0x7f2fb406f640 2022-06-27T12:11:38.059 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.030+0000 7f2fbb232700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fb40c1130 con 0x7f2fb40c4160 2022-06-27T12:11:38.059 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.030+0000 7f2fbb232700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fb406ea90 con 0x7f2fb4153cb0 2022-06-27T12:11:38.060 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.030+0000 7f2fb97cf700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f2fb4153cb0 0x7f2fb4156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:38.060 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.030+0000 7f2fb97cf700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f2fb4153cb0 0x7f2fb4156ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50438/0 (socket says 172.21.15.195:50438) 2022-06-27T12:11:38.061 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.030+0000 7f2fb97cf700 1 -- 172.21.15.195:0/900216351 learned_addr learned my addr 172.21.15.195:0/900216351 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:11:38.061 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.030+0000 7f2fb97cf700 1 -- 172.21.15.195:0/900216351 >> v1:172.21.15.202:6789/0 conn(0x7f2fb40c4160 legacy=0x7f2fb41535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:38.075 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.030+0000 7f2fb97cf700 1 -- 172.21.15.195:0/900216351 >> v1:172.21.15.195:6789/0 conn(0x7f2fb406f640 legacy=0x7f2fb40c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:38.075 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.030+0000 7f2fb97cf700 1 -- 172.21.15.195:0/900216351 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fb406ed70 con 0x7f2fb4153cb0 2022-06-27T12:11:38.076 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.030+0000 7f2fb97cf700 1 --2- 172.21.15.195:0/900216351 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2fb4153cb0 0x7f2fb4156ed0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f2fac009f20 tx=0x7f2fac009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=978b5c221e706ed1 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:38.076 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.030+0000 7f2fb37fe700 1 -- 172.21.15.195:0/900216351 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fac00e070 con 0x7f2fb4153cb0 2022-06-27T12:11:38.077 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.030+0000 7f2fb37fe700 1 -- 172.21.15.195:0/900216351 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fac00bb10 con 0x7f2fb4153cb0 2022-06-27T12:11:38.077 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.030+0000 7f2fb37fe700 1 -- 172.21.15.195:0/900216351 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fac0092e0 con 0x7f2fb4153cb0 2022-06-27T12:11:38.078 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.031+0000 7f2fbb232700 1 -- 172.21.15.195:0/900216351 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2fb4153cb0 msgr2=0x7f2fb4156ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:38.078 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.031+0000 7f2fbb232700 1 --2- 172.21.15.195:0/900216351 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2fb4153cb0 0x7f2fb4156ed0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f2fac009f20 tx=0x7f2fac009840 comp rx=0 tx=0).stop 2022-06-27T12:11:38.079 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.031+0000 7f2fbb232700 1 -- 172.21.15.195:0/900216351 shutdown_connections 2022-06-27T12:11:38.079 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.031+0000 7f2fbb232700 1 --2- 172.21.15.195:0/900216351 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2fb4153cb0 0x7f2fb4156ed0 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:38.079 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.031+0000 7f2fbb232700 1 -- 172.21.15.195:0/900216351 >> 172.21.15.195:0/900216351 conn(0x7f2fb40c6350 msgr2=0x7f2fb40c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:11:38.080 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.031+0000 7f2fbb232700 1 -- 172.21.15.195:0/900216351 shutdown_connections 2022-06-27T12:11:38.080 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.031+0000 7f2fbb232700 1 -- 172.21.15.195:0/900216351 wait complete. 2022-06-27T12:11:38.081 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.032+0000 7f2fbb232700 1 Processor -- start 2022-06-27T12:11:38.081 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.032+0000 7f2fbb232700 1 -- start start 2022-06-27T12:11:38.082 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.033+0000 7f2fbb232700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2fb406f640 0x7f2fb406ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:38.082 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.033+0000 7f2fbb232700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2fb40c4160 0x7f2fb4167020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:38.083 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.033+0000 7f2fbb232700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2fb4153cb0 0x7f2fb4157c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:38.083 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.033+0000 7f2fbb232700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb4159270 con 0x7f2fb40c4160 2022-06-27T12:11:38.084 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.033+0000 7f2fbb232700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb4069db0 con 0x7f2fb4153cb0 2022-06-27T12:11:38.086 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.033+0000 7f2fbb232700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb415c830 con 0x7f2fb406f640 2022-06-27T12:11:38.086 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.033+0000 7f2fb3fff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2fb40c4160 0x7f2fb4167020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:38.087 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.033+0000 7f2fb3fff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2fb40c4160 0x7f2fb4167020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50442/0 (socket says 172.21.15.195:50442) 2022-06-27T12:11:38.087 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.033+0000 7f2fb3fff700 1 -- 172.21.15.195:0/2500885398 learned_addr learned my addr 172.21.15.195:0/2500885398 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:11:38.087 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.034+0000 7f2fb97cf700 1 --2- 172.21.15.195:0/2500885398 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2fb4153cb0 0x7f2fb4157c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:38.088 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.034+0000 7f2fb8fce700 1 --2- 172.21.15.195:0/2500885398 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2fb406f640 0x7f2fb406ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:38.088 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.034+0000 7f2fb3fff700 1 -- 172.21.15.195:0/2500885398 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2fb406f640 msgr2=0x7f2fb406ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:38.089 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.034+0000 7f2fb3fff700 1 --2- 172.21.15.195:0/2500885398 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2fb406f640 0x7f2fb406ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:38.089 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.034+0000 7f2fb3fff700 1 -- 172.21.15.195:0/2500885398 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2fb4153cb0 msgr2=0x7f2fb4157c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:38.090 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.034+0000 7f2fb3fff700 1 --2- 172.21.15.195:0/2500885398 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2fb4153cb0 0x7f2fb4157c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:38.091 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.034+0000 7f2fb3fff700 1 -- 172.21.15.195:0/2500885398 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fac009ae0 con 0x7f2fb40c4160 2022-06-27T12:11:38.091 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.034+0000 7f2fb3fff700 1 --2- 172.21.15.195:0/2500885398 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2fb40c4160 0x7f2fb4167020 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f2fa000c490 tx=0x7f2fa0007e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:38.092 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.034+0000 7f2fb0ff9700 1 -- 172.21.15.195:0/2500885398 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa000ce40 con 0x7f2fb40c4160 2022-06-27T12:11:38.092 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.034+0000 7f2fb0ff9700 1 -- 172.21.15.195:0/2500885398 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fa00050d0 con 0x7f2fb40c4160 2022-06-27T12:11:38.092 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.034+0000 7f2fbb232700 1 -- 172.21.15.195:0/2500885398 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fb406b320 con 0x7f2fb40c4160 2022-06-27T12:11:38.093 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.034+0000 7f2fb0ff9700 1 -- 172.21.15.195:0/2500885398 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa0011590 con 0x7f2fb40c4160 2022-06-27T12:11:38.093 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.034+0000 7f2fbb232700 1 -- 172.21.15.195:0/2500885398 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fb41588d0 con 0x7f2fb40c4160 2022-06-27T12:11:38.093 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.035+0000 7f2fbb232700 1 -- 172.21.15.195:0/2500885398 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fb41593b0 con 0x7f2fb40c4160 2022-06-27T12:11:38.094 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.035+0000 7f2fb0ff9700 1 -- 172.21.15.195:0/2500885398 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f2fa0019030 con 0x7f2fb40c4160 2022-06-27T12:11:38.094 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.036+0000 7f2fb0ff9700 1 --2- 172.21.15.195:0/2500885398 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2f9c063d00 0x7f2f9c0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:38.097 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.036+0000 7f2fb8fce700 1 --2- 172.21.15.195:0/2500885398 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2f9c063d00 0x7f2f9c0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:38.097 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.036+0000 7f2fb0ff9700 1 -- 172.21.15.195:0/2500885398 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7406+0+0 (secure 0 0 0) 0x7f2fa007fca0 con 0x7f2fb40c4160 2022-06-27T12:11:38.098 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.036+0000 7f2fb8fce700 1 --2- 172.21.15.195:0/2500885398 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2f9c063d00 0x7f2f9c0661b0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f2fa8001880 tx=0x7f2fa8009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:38.098 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.040+0000 7f2fb0ff9700 1 -- 172.21.15.195:0/2500885398 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f2fa0049c90 con 0x7f2fb40c4160 2022-06-27T12:11:38.286 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.269+0000 7f2fbb232700 1 -- 172.21.15.195:0/2500885398 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2fb40722c0 con 0x7f2fb40c4160 2022-06-27T12:11:38.286 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.269+0000 7f2fb0ff9700 1 -- 172.21.15.195:0/2500885398 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+5367 (secure 0 0 0) 0x7f2fa0017090 con 0x7f2fb40c4160 2022-06-27T12:11:38.287 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:11:38.288 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:11:25.147223+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6050},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6050":{"gid":6050,"name":"b","rank":0,"incarnation":91,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6833/3890799919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3890799919},{"type":"v1","addr":"172.21.15.202:6833","nonce":3890799919}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:11:38.291 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.271+0000 7f2fbb232700 1 -- 172.21.15.195:0/2500885398 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2f9c063d00 msgr2=0x7f2f9c0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:38.300 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.271+0000 7f2fbb232700 1 --2- 172.21.15.195:0/2500885398 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2f9c063d00 0x7f2f9c0661b0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f2fa8001880 tx=0x7f2fa8009040 comp rx=0 tx=0).stop 2022-06-27T12:11:38.300 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.271+0000 7f2fbb232700 1 -- 172.21.15.195:0/2500885398 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2fb40c4160 msgr2=0x7f2fb4167020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:38.300 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.271+0000 7f2fbb232700 1 --2- 172.21.15.195:0/2500885398 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2fb40c4160 0x7f2fb4167020 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f2fa000c490 tx=0x7f2fa0007e40 comp rx=0 tx=0).stop 2022-06-27T12:11:38.301 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.271+0000 7f2fbb232700 1 -- 172.21.15.195:0/2500885398 shutdown_connections 2022-06-27T12:11:38.301 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.271+0000 7f2fbb232700 1 --2- 172.21.15.195:0/2500885398 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2f9c063d00 0x7f2f9c0661b0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:38.301 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.271+0000 7f2fbb232700 1 --2- 172.21.15.195:0/2500885398 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2fb4153cb0 0x7f2fb4157c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:38.302 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.271+0000 7f2fbb232700 1 --2- 172.21.15.195:0/2500885398 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2fb406f640 0x7f2fb406ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:38.302 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.271+0000 7f2fbb232700 1 --2- 172.21.15.195:0/2500885398 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2fb40c4160 0x7f2fb4167020 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:38.303 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.271+0000 7f2fbb232700 1 -- 172.21.15.195:0/2500885398 >> 172.21.15.195:0/2500885398 conn(0x7f2fb40c6350 msgr2=0x7f2fb40c3970 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:11:38.303 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.271+0000 7f2fbb232700 1 -- 172.21.15.195:0/2500885398 shutdown_connections 2022-06-27T12:11:38.303 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:38.271+0000 7f2fbb232700 1 -- 172.21.15.195:0/2500885398 wait complete. 2022-06-27T12:11:38.304 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 91 2022-06-27T12:11:38.356 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 91, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-06-27T11:47:52.358095+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 91, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6050": { "addr": "172.21.15.202:6833/3890799919", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6832", "nonce": 3890799919, "type": "v2" }, { "addr": "172.21.15.202:6833", "nonce": 3890799919, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 6050, "incarnation": 91, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 62, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-06-27T12:11:25.147223+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6050 } } } ], "standbys": [ { "addr": "172.21.15.202:6837/3305006841", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6836", "nonce": 3305006841, "type": "v2" }, { "addr": "172.21.15.202:6837", "nonce": 3305006841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6835/2060709822", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6834", "nonce": 2060709822, "type": "v2" }, { "addr": "172.21.15.195:6835", "nonce": 2060709822, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6838/1598751652", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6836", "nonce": 1598751652, "type": "v2" }, { "addr": "172.21.15.195:6838", "nonce": 1598751652, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4275, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6839/1313800661", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6837", "nonce": 1313800661, "type": "v2" }, { "addr": "172.21.15.195:6839", "nonce": 1313800661, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4278, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-06-27T12:11:40.359 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:11:40.516 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.501+0000 7f9137f59700 1 Processor -- start 2022-06-27T12:11:40.516 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.502+0000 7f9137f59700 1 -- start start 2022-06-27T12:11:40.519 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.502+0000 7f9137f59700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f9130150500 0x7f9130063af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:40.520 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.502+0000 7f9137f59700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f913006a450 0x7f91300c48d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:40.520 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.502+0000 7f9137f59700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f91301508d0 0x7f91300bbf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:40.521 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.502+0000 7f9137f59700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9130064030 con 0x7f91301508d0 2022-06-27T12:11:40.521 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.502+0000 7f9137f59700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91300c4160 con 0x7f9130150500 2022-06-27T12:11:40.521 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.502+0000 7f9137f59700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91300c4440 con 0x7f913006a450 2022-06-27T12:11:40.523 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.503+0000 7f91364f6700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f91301508d0 0x7f91300bbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:40.523 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.503+0000 7f91364f6700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f91301508d0 0x7f91300bbf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50458/0 (socket says 172.21.15.195:50458) 2022-06-27T12:11:40.523 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.503+0000 7f91364f6700 1 -- 172.21.15.195:0/1743804355 learned_addr learned my addr 172.21.15.195:0/1743804355 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:11:40.524 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.503+0000 7f91354f4700 1 --2- 172.21.15.195:0/1743804355 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f913006a450 0x7f91300c48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:40.524 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.503+0000 7f9135cf5700 1 --2- 172.21.15.195:0/1743804355 >> v2:172.21.15.202:3300/0 conn(0x7f9130150500 0x7f9130063af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:40.525 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.503+0000 7f91364f6700 1 -- 172.21.15.195:0/1743804355 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f913006a450 msgr2=0x7f91300c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:11:40.525 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.503+0000 7f91364f6700 1 --2- 172.21.15.195:0/1743804355 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f913006a450 0x7f91300c48d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:40.525 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.503+0000 7f91364f6700 1 -- 172.21.15.195:0/1743804355 >> v2:172.21.15.202:3300/0 conn(0x7f9130150500 msgr2=0x7f9130063af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:11:40.542 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.503+0000 7f91364f6700 1 --2- 172.21.15.195:0/1743804355 >> v2:172.21.15.202:3300/0 conn(0x7f9130150500 0x7f9130063af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:40.542 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.503+0000 7f91364f6700 1 -- 172.21.15.195:0/1743804355 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f913015a800 con 0x7f91301508d0 2022-06-27T12:11:40.543 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.503+0000 7f91364f6700 1 --2- 172.21.15.195:0/1743804355 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f91301508d0 0x7f91300bbf50 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f913006f840 tx=0x7f9128009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de7522d7beb40f1d server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:40.543 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.503+0000 7f9134cf3700 1 -- 172.21.15.195:0/1743804355 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f912800e070 con 0x7f91301508d0 2022-06-27T12:11:40.544 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.503+0000 7f9134cf3700 1 -- 172.21.15.195:0/1743804355 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f912800ba50 con 0x7f91301508d0 2022-06-27T12:11:40.544 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.503+0000 7f9134cf3700 1 -- 172.21.15.195:0/1743804355 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91280092e0 con 0x7f91301508d0 2022-06-27T12:11:40.545 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.504+0000 7f9137f59700 1 -- 172.21.15.195:0/1743804355 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f91301508d0 msgr2=0x7f91300bbf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:40.546 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.504+0000 7f9137f59700 1 --2- 172.21.15.195:0/1743804355 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f91301508d0 0x7f91300bbf50 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f913006f840 tx=0x7f9128009840 comp rx=0 tx=0).stop 2022-06-27T12:11:40.546 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.504+0000 7f9137f59700 1 -- 172.21.15.195:0/1743804355 shutdown_connections 2022-06-27T12:11:40.547 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.504+0000 7f9137f59700 1 --2- 172.21.15.195:0/1743804355 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f91301508d0 0x7f91300bbf50 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:40.547 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.504+0000 7f9137f59700 1 --2- 172.21.15.195:0/1743804355 >> v2:172.21.15.202:3300/0 conn(0x7f9130150500 0x7f9130063af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:40.547 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.504+0000 7f9137f59700 1 --2- 172.21.15.195:0/1743804355 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f913006a450 0x7f91300c48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:40.548 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.504+0000 7f9137f59700 1 -- 172.21.15.195:0/1743804355 >> 172.21.15.195:0/1743804355 conn(0x7f91300c9b00 msgr2=0x7f91300c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:11:40.548 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.504+0000 7f9137f59700 1 -- 172.21.15.195:0/1743804355 shutdown_connections 2022-06-27T12:11:40.549 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.504+0000 7f9137f59700 1 -- 172.21.15.195:0/1743804355 wait complete. 2022-06-27T12:11:40.549 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.505+0000 7f9137f59700 1 Processor -- start 2022-06-27T12:11:40.550 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.505+0000 7f9137f59700 1 -- start start 2022-06-27T12:11:40.550 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.506+0000 7f9137f59700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f913006a450 0x7f9130166bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:40.551 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.506+0000 7f9137f59700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9130150500 0x7f91300bd830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:40.552 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.506+0000 7f9137f59700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f91301508d0 0x7f91301589c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:40.552 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.506+0000 7f9137f59700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91300bce20 con 0x7f913006a450 2022-06-27T12:11:40.552 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.506+0000 7f9137f59700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91300bd3a0 con 0x7f91301508d0 2022-06-27T12:11:40.553 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.506+0000 7f9137f59700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91300bf6f0 con 0x7f9130150500 2022-06-27T12:11:40.553 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.506+0000 7f9135cf5700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f913006a450 0x7f9130166bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:40.554 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.506+0000 7f9135cf5700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f913006a450 0x7f9130166bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50460/0 (socket says 172.21.15.195:50460) 2022-06-27T12:11:40.554 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.506+0000 7f9135cf5700 1 -- 172.21.15.195:0/1989434420 learned_addr learned my addr 172.21.15.195:0/1989434420 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:11:40.555 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.506+0000 7f91354f4700 1 --2- 172.21.15.195:0/1989434420 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9130150500 0x7f91300bd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:40.555 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.506+0000 7f91364f6700 1 --2- 172.21.15.195:0/1989434420 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f91301508d0 0x7f91301589c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:40.556 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.506+0000 7f9135cf5700 1 -- 172.21.15.195:0/1989434420 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9130150500 msgr2=0x7f91300bd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:40.557 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.506+0000 7f9135cf5700 1 --2- 172.21.15.195:0/1989434420 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9130150500 0x7f91300bd830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:40.558 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.506+0000 7f9135cf5700 1 -- 172.21.15.195:0/1989434420 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f91301508d0 msgr2=0x7f91301589c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:40.558 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.506+0000 7f9135cf5700 1 --2- 172.21.15.195:0/1989434420 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f91301508d0 0x7f91301589c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:40.559 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.506+0000 7f9135cf5700 1 -- 172.21.15.195:0/1989434420 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9128009ae0 con 0x7f913006a450 2022-06-27T12:11:40.559 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.506+0000 7f91354f4700 1 --2- 172.21.15.195:0/1989434420 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9130150500 0x7f91300bd830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:11:40.559 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.506+0000 7f9135cf5700 1 --2- 172.21.15.195:0/1989434420 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f913006a450 0x7f9130166bd0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f912c00d860 tx=0x7f912c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:40.560 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.506+0000 7f9121ffb700 1 -- 172.21.15.195:0/1989434420 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f912c012630 con 0x7f913006a450 2022-06-27T12:11:40.560 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.506+0000 7f9121ffb700 1 -- 172.21.15.195:0/1989434420 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f912c012790 con 0x7f913006a450 2022-06-27T12:11:40.561 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.506+0000 7f9137f59700 1 -- 172.21.15.195:0/1989434420 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91300bf9d0 con 0x7f913006a450 2022-06-27T12:11:40.561 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.506+0000 7f9121ffb700 1 -- 172.21.15.195:0/1989434420 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f912c012b30 con 0x7f913006a450 2022-06-27T12:11:40.563 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.507+0000 7f9137f59700 1 -- 172.21.15.195:0/1989434420 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91300bd030 con 0x7f913006a450 2022-06-27T12:11:40.563 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.508+0000 7f9137f59700 1 -- 172.21.15.195:0/1989434420 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91300bfe90 con 0x7f913006a450 2022-06-27T12:11:40.563 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.508+0000 7f9121ffb700 1 -- 172.21.15.195:0/1989434420 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f912c01e460 con 0x7f913006a450 2022-06-27T12:11:40.564 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.508+0000 7f9121ffb700 1 --2- 172.21.15.195:0/1989434420 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9110063d00 0x7f91100661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:40.564 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.508+0000 7f9121ffb700 1 -- 172.21.15.195:0/1989434420 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7406+0+0 (secure 0 0 0) 0x7f912c0814f0 con 0x7f913006a450 2022-06-27T12:11:40.565 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.508+0000 7f91354f4700 1 --2- 172.21.15.195:0/1989434420 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9110063d00 0x7f91100661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:40.565 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.509+0000 7f91354f4700 1 --2- 172.21.15.195:0/1989434420 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9110063d00 0x7f91100661b0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f913006cb90 tx=0x7f9124008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:40.565 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.513+0000 7f9121ffb700 1 -- 172.21.15.195:0/1989434420 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f912c04fb40 con 0x7f913006a450 2022-06-27T12:11:40.740 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.740+0000 7f9137f59700 1 -- 172.21.15.195:0/1989434420 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f91300722c0 con 0x7f913006a450 2022-06-27T12:11:40.741 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.741+0000 7f9121ffb700 1 -- 172.21.15.195:0/1989434420 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+5370 (secure 0 0 0) 0x7f912c053160 con 0x7f913006a450 2022-06-27T12:11:40.742 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:11:40.742 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":92,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:11:38.328748+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6050},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6050":{"gid":6050,"name":"b","rank":0,"incarnation":91,"state":"up:reconnect","state_seq":9,"addr":"172.21.15.202:6833/3890799919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3890799919},{"type":"v1","addr":"172.21.15.202:6833","nonce":3890799919}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:11:40.744 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.743+0000 7f9137f59700 1 -- 172.21.15.195:0/1989434420 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9110063d00 msgr2=0x7f91100661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:40.744 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.743+0000 7f9137f59700 1 --2- 172.21.15.195:0/1989434420 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9110063d00 0x7f91100661b0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f913006cb90 tx=0x7f9124008040 comp rx=0 tx=0).stop 2022-06-27T12:11:40.744 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.743+0000 7f9137f59700 1 -- 172.21.15.195:0/1989434420 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f913006a450 msgr2=0x7f9130166bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:40.744 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.743+0000 7f9137f59700 1 --2- 172.21.15.195:0/1989434420 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f913006a450 0x7f9130166bd0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f912c00d860 tx=0x7f912c008040 comp rx=0 tx=0).stop 2022-06-27T12:11:40.745 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.743+0000 7f9137f59700 1 -- 172.21.15.195:0/1989434420 shutdown_connections 2022-06-27T12:11:40.745 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.743+0000 7f9137f59700 1 --2- 172.21.15.195:0/1989434420 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9110063d00 0x7f91100661b0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:40.745 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.743+0000 7f9137f59700 1 --2- 172.21.15.195:0/1989434420 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f913006a450 0x7f9130166bd0 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:40.745 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.743+0000 7f9137f59700 1 --2- 172.21.15.195:0/1989434420 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f91301508d0 0x7f91301589c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:40.746 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.743+0000 7f9137f59700 1 --2- 172.21.15.195:0/1989434420 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9130150500 0x7f91300bd830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:40.746 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.743+0000 7f9137f59700 1 -- 172.21.15.195:0/1989434420 >> 172.21.15.195:0/1989434420 conn(0x7f91300c9b00 msgr2=0x7f91300c8860 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:11:40.746 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.743+0000 7f9137f59700 1 -- 172.21.15.195:0/1989434420 shutdown_connections 2022-06-27T12:11:40.746 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:40.743+0000 7f9137f59700 1 -- 172.21.15.195:0/1989434420 wait complete. 2022-06-27T12:11:40.747 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 92 2022-06-27T12:11:40.758 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:11:42.760 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:11:42.904 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:42.893+0000 7fe71a71c700 1 Processor -- start 2022-06-27T12:11:42.905 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:42.893+0000 7fe71a71c700 1 -- start start 2022-06-27T12:11:42.905 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:42.894+0000 7fe71a71c700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fe714150500 0x7fe714063af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:42.905 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:42.894+0000 7fe71a71c700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe71406a450 0x7fe7140c48d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:42.906 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:42.894+0000 7fe71a71c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fe7141508d0 0x7fe7140bbf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:42.906 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:42.894+0000 7fe71a71c700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe714064030 con 0x7fe7141508d0 2022-06-27T12:11:42.907 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:42.894+0000 7fe71a71c700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7140c4160 con 0x7fe714150500 2022-06-27T12:11:42.907 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:42.894+0000 7fe71a71c700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7140c4440 con 0x7fe71406a450 2022-06-27T12:11:42.907 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:42.894+0000 7fe718cb9700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fe7141508d0 0x7fe7140bbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:42.907 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:42.894+0000 7fe718cb9700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fe7141508d0 0x7fe7140bbf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50472/0 (socket says 172.21.15.195:50472) 2022-06-27T12:11:42.908 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:42.894+0000 7fe718cb9700 1 -- 172.21.15.195:0/2941427245 learned_addr learned my addr 172.21.15.195:0/2941427245 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:11:42.908 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:42.894+0000 7fe7137fe700 1 --2- 172.21.15.195:0/2941427245 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe71406a450 0x7fe7140c48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:42.908 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:42.894+0000 7fe713fff700 1 --2- 172.21.15.195:0/2941427245 >> v2:172.21.15.202:3300/0 conn(0x7fe714150500 0x7fe714063af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:42.908 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:42.894+0000 7fe718cb9700 1 -- 172.21.15.195:0/2941427245 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe71406a450 msgr2=0x7fe7140c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:11:42.909 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:42.894+0000 7fe718cb9700 1 --2- 172.21.15.195:0/2941427245 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe71406a450 0x7fe7140c48d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:42.909 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:42.894+0000 7fe718cb9700 1 -- 172.21.15.195:0/2941427245 >> v2:172.21.15.202:3300/0 conn(0x7fe714150500 msgr2=0x7fe714063af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:11:42.909 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:42.894+0000 7fe718cb9700 1 --2- 172.21.15.195:0/2941427245 >> v2:172.21.15.202:3300/0 conn(0x7fe714150500 0x7fe714063af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:42.909 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:42.894+0000 7fe718cb9700 1 -- 172.21.15.195:0/2941427245 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe71415a800 con 0x7fe7141508d0 2022-06-27T12:11:42.910 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:42.894+0000 7fe718cb9700 1 --2- 172.21.15.195:0/2941427245 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe7141508d0 0x7fe7140bbf50 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fe71406f840 tx=0x7fe70c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f645a692d46e352 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:42.911 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:42.895+0000 7fe712ffd700 1 -- 172.21.15.195:0/2941427245 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe70c00e070 con 0x7fe7141508d0 2022-06-27T12:11:42.911 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:42.895+0000 7fe712ffd700 1 -- 172.21.15.195:0/2941427245 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe70c00ba50 con 0x7fe7141508d0 2022-06-27T12:11:42.911 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:42.895+0000 7fe712ffd700 1 -- 172.21.15.195:0/2941427245 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe70c0092e0 con 0x7fe7141508d0 2022-06-27T12:11:42.912 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:42.895+0000 7fe71a71c700 1 -- 172.21.15.195:0/2941427245 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe7141508d0 msgr2=0x7fe7140bbf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:42.912 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:42.895+0000 7fe71a71c700 1 --2- 172.21.15.195:0/2941427245 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe7141508d0 0x7fe7140bbf50 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fe71406f840 tx=0x7fe70c009840 comp rx=0 tx=0).stop 2022-06-27T12:11:42.913 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:42.895+0000 7fe71a71c700 1 -- 172.21.15.195:0/2941427245 shutdown_connections 2022-06-27T12:11:42.913 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:42.895+0000 7fe71a71c700 1 --2- 172.21.15.195:0/2941427245 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe7141508d0 0x7fe7140bbf50 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:42.914 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:42.895+0000 7fe71a71c700 1 --2- 172.21.15.195:0/2941427245 >> v2:172.21.15.202:3300/0 conn(0x7fe714150500 0x7fe714063af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:42.914 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:42.895+0000 7fe71a71c700 1 --2- 172.21.15.195:0/2941427245 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe71406a450 0x7fe7140c48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:42.914 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:42.895+0000 7fe71a71c700 1 -- 172.21.15.195:0/2941427245 >> 172.21.15.195:0/2941427245 conn(0x7fe7140c9b00 msgr2=0x7fe7140c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:11:42.915 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:42.895+0000 7fe71a71c700 1 -- 172.21.15.195:0/2941427245 shutdown_connections 2022-06-27T12:11:42.915 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:42.895+0000 7fe71a71c700 1 -- 172.21.15.195:0/2941427245 wait complete. 2022-06-27T12:11:42.916 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:42.896+0000 7fe71a71c700 1 Processor -- start 2022-06-27T12:11:42.916 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:42.896+0000 7fe71a71c700 1 -- start start 2022-06-27T12:11:42.917 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:42.897+0000 7fe71a71c700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe71406a450 0x7fe71414fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:42.917 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:42.897+0000 7fe71a71c700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe714150500 0x7fe71414c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:42.918 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:42.897+0000 7fe71a71c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe7141508d0 0x7fe71414d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:42.918 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:42.897+0000 7fe71a71c700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe71414e920 con 0x7fe7141508d0 2022-06-27T12:11:42.919 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:42.897+0000 7fe71a71c700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe71414ec00 con 0x7fe71406a450 2022-06-27T12:11:42.919 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:42.897+0000 7fe71a71c700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe71414eee0 con 0x7fe714150500 2022-06-27T12:11:42.920 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:42.897+0000 7fe718cb9700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe7141508d0 0x7fe71414d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:42.920 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:42.897+0000 7fe718cb9700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe7141508d0 0x7fe71414d100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50478/0 (socket says 172.21.15.195:50478) 2022-06-27T12:11:42.921 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:42.897+0000 7fe718cb9700 1 -- 172.21.15.195:0/223789439 learned_addr learned my addr 172.21.15.195:0/223789439 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:11:42.922 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:42.897+0000 7fe7137fe700 1 --2- 172.21.15.195:0/223789439 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe714150500 0x7fe71414c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:42.922 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:42.898+0000 7fe713fff700 1 --2- 172.21.15.195:0/223789439 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe71406a450 0x7fe71414fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:42.922 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:42.898+0000 7fe718cb9700 1 -- 172.21.15.195:0/223789439 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe714150500 msgr2=0x7fe71414c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:42.923 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:42.898+0000 7fe718cb9700 1 --2- 172.21.15.195:0/223789439 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe714150500 0x7fe71414c1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:42.923 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:42.898+0000 7fe718cb9700 1 -- 172.21.15.195:0/223789439 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe71406a450 msgr2=0x7fe71414fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:42.924 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:42.898+0000 7fe718cb9700 1 --2- 172.21.15.195:0/223789439 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe71406a450 0x7fe71414fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:42.924 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:42.898+0000 7fe718cb9700 1 -- 172.21.15.195:0/223789439 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe70c009ae0 con 0x7fe7141508d0 2022-06-27T12:11:42.924 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:42.898+0000 7fe7137fe700 1 --2- 172.21.15.195:0/223789439 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe714150500 0x7fe71414c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:11:42.925 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:42.898+0000 7fe718cb9700 1 --2- 172.21.15.195:0/223789439 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe7141508d0 0x7fe71414d100 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fe70c009fd0 tx=0x7fe70c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:42.925 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:42.898+0000 7fe6fbfff700 1 -- 172.21.15.195:0/223789439 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe70c009940 con 0x7fe7141508d0 2022-06-27T12:11:42.925 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:42.898+0000 7fe6fbfff700 1 -- 172.21.15.195:0/223789439 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe70c007820 con 0x7fe7141508d0 2022-06-27T12:11:42.926 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:42.898+0000 7fe71a71c700 1 -- 172.21.15.195:0/223789439 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe71414f1c0 con 0x7fe7141508d0 2022-06-27T12:11:42.926 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:42.898+0000 7fe6fbfff700 1 -- 172.21.15.195:0/223789439 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe70c007be0 con 0x7fe7141508d0 2022-06-27T12:11:42.927 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:42.898+0000 7fe71a71c700 1 -- 172.21.15.195:0/223789439 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe71414f780 con 0x7fe7141508d0 2022-06-27T12:11:42.927 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:42.899+0000 7fe71a71c700 1 -- 172.21.15.195:0/223789439 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7140722c0 con 0x7fe7141508d0 2022-06-27T12:11:42.927 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:42.900+0000 7fe6fbfff700 1 -- 172.21.15.195:0/223789439 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fe70c004040 con 0x7fe7141508d0 2022-06-27T12:11:42.928 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:42.900+0000 7fe6fbfff700 1 --2- 172.21.15.195:0/223789439 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe6f4063d30 0x7fe6f40661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:42.928 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:42.900+0000 7fe6fbfff700 1 -- 172.21.15.195:0/223789439 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7406+0+0 (secure 0 0 0) 0x7fe70c082a00 con 0x7fe7141508d0 2022-06-27T12:11:42.929 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:42.900+0000 7fe713fff700 1 --2- 172.21.15.195:0/223789439 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe6f4063d30 0x7fe6f40661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:42.929 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:42.900+0000 7fe713fff700 1 --2- 172.21.15.195:0/223789439 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe6f4063d30 0x7fe6f40661e0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fe70800bb60 tx=0x7fe708008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:42.929 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:42.904+0000 7fe6fbfff700 1 -- 172.21.15.195:0/223789439 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fe70c0510b0 con 0x7fe7141508d0 2022-06-27T12:11:43.139 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:43.129+0000 7fe71a71c700 1 -- 172.21.15.195:0/223789439 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe7140ba8e0 con 0x7fe7141508d0 2022-06-27T12:11:43.140 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:43.129+0000 7fe6fbfff700 1 -- 172.21.15.195:0/223789439 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+5370 (secure 0 0 0) 0x7fe70c0546d0 con 0x7fe7141508d0 2022-06-27T12:11:43.141 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:11:43.141 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":92,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:11:38.328748+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6050},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6050":{"gid":6050,"name":"b","rank":0,"incarnation":91,"state":"up:reconnect","state_seq":9,"addr":"172.21.15.202:6833/3890799919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3890799919},{"type":"v1","addr":"172.21.15.202:6833","nonce":3890799919}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:11:43.149 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:43.131+0000 7fe71a71c700 1 -- 172.21.15.195:0/223789439 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe6f4063d30 msgr2=0x7fe6f40661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:43.151 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:43.131+0000 7fe71a71c700 1 --2- 172.21.15.195:0/223789439 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe6f4063d30 0x7fe6f40661e0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fe70800bb60 tx=0x7fe708008040 comp rx=0 tx=0).stop 2022-06-27T12:11:43.151 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:43.131+0000 7fe71a71c700 1 -- 172.21.15.195:0/223789439 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe7141508d0 msgr2=0x7fe71414d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:43.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:43.131+0000 7fe71a71c700 1 --2- 172.21.15.195:0/223789439 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe7141508d0 0x7fe71414d100 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fe70c009fd0 tx=0x7fe70c018000 comp rx=0 tx=0).stop 2022-06-27T12:11:43.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:43.131+0000 7fe71a71c700 1 -- 172.21.15.195:0/223789439 shutdown_connections 2022-06-27T12:11:43.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:43.131+0000 7fe71a71c700 1 --2- 172.21.15.195:0/223789439 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe6f4063d30 0x7fe6f40661e0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:43.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:43.131+0000 7fe71a71c700 1 --2- 172.21.15.195:0/223789439 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe7141508d0 0x7fe71414d100 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:43.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:43.131+0000 7fe71a71c700 1 --2- 172.21.15.195:0/223789439 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe71406a450 0x7fe71414fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:43.155 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:43.131+0000 7fe71a71c700 1 --2- 172.21.15.195:0/223789439 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe714150500 0x7fe71414c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:43.155 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:43.131+0000 7fe71a71c700 1 -- 172.21.15.195:0/223789439 >> 172.21.15.195:0/223789439 conn(0x7fe7140c9b00 msgr2=0x7fe7140c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:11:43.156 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:43.132+0000 7fe71a71c700 1 -- 172.21.15.195:0/223789439 shutdown_connections 2022-06-27T12:11:43.156 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:43.132+0000 7fe71a71c700 1 -- 172.21.15.195:0/223789439 wait complete. 2022-06-27T12:11:43.157 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 92 2022-06-27T12:11:43.158 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:11:45.162 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:11:45.305 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.301+0000 7f2867b4c700 1 Processor -- start 2022-06-27T12:11:45.306 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.302+0000 7f2867b4c700 1 -- start start 2022-06-27T12:11:45.306 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.302+0000 7f2867b4c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f286006f620 0x7f2860156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:45.307 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.302+0000 7f2867b4c700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28600c76b0 con 0x7f2860150950 2022-06-27T12:11:45.320 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.303+0000 7f2867b4c700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28600c8250 con 0x7f2860150580 2022-06-27T12:11:45.321 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.303+0000 7f2867b4c700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2860157430 con 0x7f286006f620 2022-06-27T12:11:45.321 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.303+0000 7f28660e9700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f286006f620 0x7f2860156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:45.321 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.303+0000 7f28650e7700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f2860150950 0x7f286006d6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:56672/0 (socket says 172.21.15.195:56672) 2022-06-27T12:11:45.322 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.303+0000 7f28650e7700 1 -- 172.21.15.195:0/3595387932 learned_addr learned my addr 172.21.15.195:0/3595387932 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:11:45.322 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.303+0000 7f28660e9700 1 -- 172.21.15.195:0/3595387932 >> v1:172.21.15.202:6789/0 conn(0x7f2860150580 legacy=0x7f28600c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:11:45.323 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.303+0000 7f28660e9700 1 -- 172.21.15.195:0/3595387932 >> v1:172.21.15.195:6789/0 conn(0x7f2860150950 legacy=0x7f286006d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:45.323 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.303+0000 7f28648e6700 1 -- 172.21.15.195:0/3595387932 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 359581199 0 0) 0x7f28600c76b0 con 0x7f2860150950 2022-06-27T12:11:45.324 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.303+0000 7f28660e9700 1 -- 172.21.15.195:0/3595387932 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28600bad70 con 0x7f286006f620 2022-06-27T12:11:45.324 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.303+0000 7f28660e9700 1 --2- 172.21.15.195:0/3595387932 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f286006f620 0x7f2860156ef0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f2858009e60 tx=0x7f2858009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54531e2d68795ebf server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:45.332 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.303+0000 7f28648e6700 1 -- 172.21.15.195:0/3595387932 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f285800e070 con 0x7f286006f620 2022-06-27T12:11:45.332 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.303+0000 7f28648e6700 1 -- 172.21.15.195:0/3595387932 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f285800bbf0 con 0x7f286006f620 2022-06-27T12:11:45.333 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.303+0000 7f28648e6700 1 -- 172.21.15.195:0/3595387932 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28580038a0 con 0x7f286006f620 2022-06-27T12:11:45.333 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.304+0000 7f2867b4c700 1 -- 172.21.15.195:0/3595387932 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f286006f620 msgr2=0x7f2860156ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:45.334 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.304+0000 7f2867b4c700 1 --2- 172.21.15.195:0/3595387932 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f286006f620 0x7f2860156ef0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f2858009e60 tx=0x7f2858009780 comp rx=0 tx=0).stop 2022-06-27T12:11:45.334 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.304+0000 7f2867b4c700 1 -- 172.21.15.195:0/3595387932 shutdown_connections 2022-06-27T12:11:45.335 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.304+0000 7f2867b4c700 1 --2- 172.21.15.195:0/3595387932 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f286006f620 0x7f2860156ef0 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:45.335 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.304+0000 7f2867b4c700 1 -- 172.21.15.195:0/3595387932 >> 172.21.15.195:0/3595387932 conn(0x7f2860063af0 msgr2=0x7f2860063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:11:45.335 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.304+0000 7f2867b4c700 1 -- 172.21.15.195:0/3595387932 shutdown_connections 2022-06-27T12:11:45.336 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.304+0000 7f2867b4c700 1 -- 172.21.15.195:0/3595387932 wait complete. 2022-06-27T12:11:45.336 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.305+0000 7f2867b4c700 1 Processor -- start 2022-06-27T12:11:45.337 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.305+0000 7f2867b4c700 1 -- start start 2022-06-27T12:11:45.337 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.306+0000 7f2867b4c700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f286006f620 0x7f28601584d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:45.337 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.306+0000 7f2867b4c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2860150580 0x7f2860158a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:45.338 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.306+0000 7f2867b4c700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2860150950 0x7f286016a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:45.338 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.306+0000 7f2867b4c700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2860159680 con 0x7f2860150580 2022-06-27T12:11:45.339 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.306+0000 7f2867b4c700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28600bb4a0 con 0x7f286006f620 2022-06-27T12:11:45.339 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.306+0000 7f2867b4c700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2860159920 con 0x7f2860150950 2022-06-27T12:11:45.341 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.306+0000 7f28650e7700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2860150580 0x7f2860158a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:45.342 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.306+0000 7f28650e7700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2860150580 0x7f2860158a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50492/0 (socket says 172.21.15.195:50492) 2022-06-27T12:11:45.342 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.306+0000 7f28650e7700 1 -- 172.21.15.195:0/4220199579 learned_addr learned my addr 172.21.15.195:0/4220199579 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:11:45.342 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.306+0000 7f28658e8700 1 --2- 172.21.15.195:0/4220199579 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f286006f620 0x7f28601584d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:45.343 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.306+0000 7f28660e9700 1 --2- 172.21.15.195:0/4220199579 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2860150950 0x7f286016a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:45.343 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.306+0000 7f28650e7700 1 -- 172.21.15.195:0/4220199579 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2860150950 msgr2=0x7f286016a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:45.344 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.306+0000 7f28650e7700 1 --2- 172.21.15.195:0/4220199579 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2860150950 0x7f286016a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:45.344 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.306+0000 7f28650e7700 1 -- 172.21.15.195:0/4220199579 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f286006f620 msgr2=0x7f28601584d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:45.345 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.306+0000 7f28650e7700 1 --2- 172.21.15.195:0/4220199579 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f286006f620 0x7f28601584d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:45.345 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.306+0000 7f28650e7700 1 -- 172.21.15.195:0/4220199579 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2858009a20 con 0x7f2860150580 2022-06-27T12:11:45.345 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.306+0000 7f28650e7700 1 --2- 172.21.15.195:0/4220199579 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2860150580 0x7f2860158a10 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f285c00c590 tx=0x7f285c007cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:45.346 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.307+0000 7f284dffb700 1 -- 172.21.15.195:0/4220199579 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f285c007360 con 0x7f2860150580 2022-06-27T12:11:45.346 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.307+0000 7f284dffb700 1 -- 172.21.15.195:0/4220199579 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f285c007500 con 0x7f2860150580 2022-06-27T12:11:45.347 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.307+0000 7f2867b4c700 1 -- 172.21.15.195:0/4220199579 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28600bb740 con 0x7f2860150580 2022-06-27T12:11:45.347 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.307+0000 7f284dffb700 1 -- 172.21.15.195:0/4220199579 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f285c004940 con 0x7f2860150580 2022-06-27T12:11:45.348 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.307+0000 7f2867b4c700 1 -- 172.21.15.195:0/4220199579 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2860157f90 con 0x7f2860150580 2022-06-27T12:11:45.349 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.308+0000 7f2867b4c700 1 -- 172.21.15.195:0/4220199579 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28600722c0 con 0x7f2860150580 2022-06-27T12:11:45.350 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.308+0000 7f284dffb700 1 -- 172.21.15.195:0/4220199579 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f285c010030 con 0x7f2860150580 2022-06-27T12:11:45.351 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.308+0000 7f284dffb700 1 --2- 172.21.15.195:0/4220199579 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2840063d00 0x7f28400661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:45.351 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.308+0000 7f284dffb700 1 -- 172.21.15.195:0/4220199579 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7406+0+0 (secure 0 0 0) 0x7f285c080870 con 0x7f2860150580 2022-06-27T12:11:45.351 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.309+0000 7f28658e8700 1 --2- 172.21.15.195:0/4220199579 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2840063d00 0x7f28400661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:45.352 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.309+0000 7f28658e8700 1 --2- 172.21.15.195:0/4220199579 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2840063d00 0x7f28400661b0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f2850001890 tx=0x7f2850009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:45.353 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.313+0000 7f284dffb700 1 -- 172.21.15.195:0/4220199579 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f285c04efa0 con 0x7f2860150580 2022-06-27T12:11:45.547 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.539+0000 7f2867b4c700 1 -- 172.21.15.195:0/4220199579 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f286006fe20 con 0x7f2860150580 2022-06-27T12:11:45.547 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.539+0000 7f284dffb700 1 -- 172.21.15.195:0/4220199579 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+5368 (secure 0 0 0) 0x7f285c0525c0 con 0x7f2860150580 2022-06-27T12:11:45.548 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:11:45.549 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:11:43.349160+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6050},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6050":{"gid":6050,"name":"b","rank":0,"incarnation":91,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.202:6833/3890799919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3890799919},{"type":"v1","addr":"172.21.15.202:6833","nonce":3890799919}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:11:45.551 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.541+0000 7f2867b4c700 1 -- 172.21.15.195:0/4220199579 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2840063d00 msgr2=0x7f28400661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:45.552 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.541+0000 7f2867b4c700 1 --2- 172.21.15.195:0/4220199579 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2840063d00 0x7f28400661b0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f2850001890 tx=0x7f2850009040 comp rx=0 tx=0).stop 2022-06-27T12:11:45.552 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.541+0000 7f2867b4c700 1 -- 172.21.15.195:0/4220199579 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2860150580 msgr2=0x7f2860158a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:45.553 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.541+0000 7f2867b4c700 1 --2- 172.21.15.195:0/4220199579 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2860150580 0x7f2860158a10 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f285c00c590 tx=0x7f285c007cb0 comp rx=0 tx=0).stop 2022-06-27T12:11:45.553 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.541+0000 7f2867b4c700 1 -- 172.21.15.195:0/4220199579 shutdown_connections 2022-06-27T12:11:45.553 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.541+0000 7f2867b4c700 1 --2- 172.21.15.195:0/4220199579 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2840063d00 0x7f28400661b0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:45.554 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.541+0000 7f2867b4c700 1 --2- 172.21.15.195:0/4220199579 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2860150950 0x7f286016a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:45.554 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.541+0000 7f2867b4c700 1 --2- 172.21.15.195:0/4220199579 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f286006f620 0x7f28601584d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:45.555 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.541+0000 7f2867b4c700 1 --2- 172.21.15.195:0/4220199579 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2860150580 0x7f2860158a10 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:45.555 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.541+0000 7f2867b4c700 1 -- 172.21.15.195:0/4220199579 >> 172.21.15.195:0/4220199579 conn(0x7f2860063af0 msgr2=0x7f286014d950 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:11:45.556 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.541+0000 7f2867b4c700 1 -- 172.21.15.195:0/4220199579 shutdown_connections 2022-06-27T12:11:45.556 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:45.541+0000 7f2867b4c700 1 -- 172.21.15.195:0/4220199579 wait complete. 2022-06-27T12:11:45.557 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 93 2022-06-27T12:11:45.564 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:11:47.570 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:11:47.711 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.704+0000 7fba165cc700 1 Processor -- start 2022-06-27T12:11:47.712 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.704+0000 7fba165cc700 1 -- start start 2022-06-27T12:11:47.712 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.705+0000 7fba165cc700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fba1006e850 0x7fba101570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:47.712 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.705+0000 7fba165cc700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fba1006fef0 0x7fba10069de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:47.712 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.705+0000 7fba165cc700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba1006fba0 con 0x7fba100c34a0 2022-06-27T12:11:47.713 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.705+0000 7fba165cc700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba101575e0 con 0x7fba1006fef0 2022-06-27T12:11:47.713 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.705+0000 7fba165cc700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba1006f5e0 con 0x7fba1006e850 2022-06-27T12:11:47.713 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.705+0000 7fba14b69700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fba100c34a0 0x7fba100c2920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:56686/0 (socket says 172.21.15.195:56686) 2022-06-27T12:11:47.713 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.705+0000 7fba14b69700 1 -- 172.21.15.195:0/3324010150 learned_addr learned my addr 172.21.15.195:0/3324010150 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:11:47.714 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.705+0000 7fba0f7fe700 1 --2- 172.21.15.195:0/3324010150 >> v2:172.21.15.202:3300/0 conn(0x7fba1006fef0 0x7fba10069de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:47.715 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.705+0000 7fba0ffff700 1 --2- 172.21.15.195:0/3324010150 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fba1006e850 0x7fba101570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:47.715 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.705+0000 7fba0effd700 1 -- 172.21.15.195:0/3324010150 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2115048593 0 0) 0x7fba1006fba0 con 0x7fba100c34a0 2022-06-27T12:11:47.715 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.705+0000 7fba0effd700 1 -- 172.21.15.195:0/3324010150 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9fc003430 con 0x7fba100c34a0 2022-06-27T12:11:47.716 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.705+0000 7fba0f7fe700 1 -- 172.21.15.195:0/3324010150 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fba1006e850 msgr2=0x7fba101570a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:11:47.716 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.705+0000 7fba0f7fe700 1 --2- 172.21.15.195:0/3324010150 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fba1006e850 0x7fba101570a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:47.716 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.705+0000 7fba0f7fe700 1 -- 172.21.15.195:0/3324010150 >> v1:172.21.15.195:6789/0 conn(0x7fba100c34a0 legacy=0x7fba100c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:47.716 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.705+0000 7fba0f7fe700 1 -- 172.21.15.195:0/3324010150 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba1006f8c0 con 0x7fba1006fef0 2022-06-27T12:11:47.717 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.705+0000 7fba0ffff700 1 --2- 172.21.15.195:0/3324010150 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fba1006e850 0x7fba101570a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:11:47.717 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.706+0000 7fba0f7fe700 1 --2- 172.21.15.195:0/3324010150 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fba1006fef0 0x7fba10069de0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fba000097a0 tx=0x7fba00009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6652e025fca9ee30 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:47.717 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.706+0000 7fba0effd700 1 -- 172.21.15.195:0/3324010150 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba00011630 con 0x7fba1006fef0 2022-06-27T12:11:47.717 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.706+0000 7fba0effd700 1 -- 172.21.15.195:0/3324010150 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba00011790 con 0x7fba1006fef0 2022-06-27T12:11:47.717 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.706+0000 7fba0effd700 1 -- 172.21.15.195:0/3324010150 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba00011b30 con 0x7fba1006fef0 2022-06-27T12:11:47.718 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.706+0000 7fba165cc700 1 -- 172.21.15.195:0/3324010150 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fba1006fef0 msgr2=0x7fba10069de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:47.718 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.706+0000 7fba165cc700 1 --2- 172.21.15.195:0/3324010150 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fba1006fef0 0x7fba10069de0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fba000097a0 tx=0x7fba00009060 comp rx=0 tx=0).stop 2022-06-27T12:11:47.718 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.706+0000 7fba165cc700 1 -- 172.21.15.195:0/3324010150 shutdown_connections 2022-06-27T12:11:47.718 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.706+0000 7fba165cc700 1 --2- 172.21.15.195:0/3324010150 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fba1006e850 0x7fba101570a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:47.719 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.706+0000 7fba165cc700 1 --2- 172.21.15.195:0/3324010150 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fba1006fef0 0x7fba10069de0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:47.720 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.706+0000 7fba165cc700 1 -- 172.21.15.195:0/3324010150 >> 172.21.15.195:0/3324010150 conn(0x7fba100c6350 msgr2=0x7fba100c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:11:47.721 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.706+0000 7fba165cc700 1 -- 172.21.15.195:0/3324010150 shutdown_connections 2022-06-27T12:11:47.721 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.706+0000 7fba165cc700 1 -- 172.21.15.195:0/3324010150 wait complete. 2022-06-27T12:11:47.721 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.708+0000 7fba165cc700 1 Processor -- start 2022-06-27T12:11:47.722 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.708+0000 7fba165cc700 1 -- start start 2022-06-27T12:11:47.728 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.708+0000 7fba165cc700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fba1006e850 0x7fba100bf780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:47.729 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.708+0000 7fba165cc700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fba100c34a0 0x7fba100bf1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:47.729 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.708+0000 7fba165cc700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fba100bed50 0x7fba100bd600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:47.729 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.709+0000 7fba165cc700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba1014fd30 con 0x7fba100c34a0 2022-06-27T12:11:47.729 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.709+0000 7fba165cc700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba1014f0a0 con 0x7fba100bed50 2022-06-27T12:11:47.730 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.709+0000 7fba165cc700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba100c5490 con 0x7fba1006e850 2022-06-27T12:11:47.730 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.709+0000 7fba0f7fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fba100c34a0 0x7fba100bf1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:47.730 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.709+0000 7fba0f7fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fba100c34a0 0x7fba100bf1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50506/0 (socket says 172.21.15.195:50506) 2022-06-27T12:11:47.731 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.709+0000 7fba0f7fe700 1 -- 172.21.15.195:0/897299957 learned_addr learned my addr 172.21.15.195:0/897299957 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:11:47.731 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.709+0000 7fba14b69700 1 --2- 172.21.15.195:0/897299957 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fba100bed50 0x7fba100bd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:47.732 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.709+0000 7fba0ffff700 1 --2- 172.21.15.195:0/897299957 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fba1006e850 0x7fba100bf780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:47.732 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.709+0000 7fba0f7fe700 1 -- 172.21.15.195:0/897299957 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fba1006e850 msgr2=0x7fba100bf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:47.732 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.709+0000 7fba0f7fe700 1 --2- 172.21.15.195:0/897299957 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fba1006e850 0x7fba100bf780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:47.733 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.709+0000 7fba0f7fe700 1 -- 172.21.15.195:0/897299957 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fba100bed50 msgr2=0x7fba100bd600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:47.733 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.709+0000 7fba0f7fe700 1 --2- 172.21.15.195:0/897299957 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fba100bed50 0x7fba100bd600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:47.733 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.709+0000 7fba0f7fe700 1 -- 172.21.15.195:0/897299957 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba040092e0 con 0x7fba100c34a0 2022-06-27T12:11:47.733 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.709+0000 7fba0f7fe700 1 --2- 172.21.15.195:0/897299957 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fba100c34a0 0x7fba100bf1c0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7fba100650f0 tx=0x7fba00017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:47.735 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.709+0000 7fb9f7fff700 1 -- 172.21.15.195:0/897299957 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba00003730 con 0x7fba100c34a0 2022-06-27T12:11:47.735 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.709+0000 7fb9f7fff700 1 -- 172.21.15.195:0/897299957 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba000038b0 con 0x7fba100c34a0 2022-06-27T12:11:47.735 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.709+0000 7fba165cc700 1 -- 172.21.15.195:0/897299957 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba000092e0 con 0x7fba100c34a0 2022-06-27T12:11:47.736 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.709+0000 7fb9f7fff700 1 -- 172.21.15.195:0/897299957 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba00003c70 con 0x7fba100c34a0 2022-06-27T12:11:47.737 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.709+0000 7fba165cc700 1 -- 172.21.15.195:0/897299957 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba1014f3d0 con 0x7fba100c34a0 2022-06-27T12:11:47.737 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.710+0000 7fba165cc700 1 -- 172.21.15.195:0/897299957 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba100722c0 con 0x7fba100c34a0 2022-06-27T12:11:47.737 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.711+0000 7fb9f7fff700 1 -- 172.21.15.195:0/897299957 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fba0000f050 con 0x7fba100c34a0 2022-06-27T12:11:47.737 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.711+0000 7fb9f7fff700 1 --2- 172.21.15.195:0/897299957 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb9f0063d00 0x7fb9f00661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:47.738 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.711+0000 7fb9f7fff700 1 -- 172.21.15.195:0/897299957 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7406+0+0 (secure 0 0 0) 0x7fba00082270 con 0x7fba100c34a0 2022-06-27T12:11:47.738 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.711+0000 7fba0ffff700 1 --2- 172.21.15.195:0/897299957 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb9f0063d00 0x7fb9f00661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:47.738 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.712+0000 7fba0ffff700 1 --2- 172.21.15.195:0/897299957 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb9f0063d00 0x7fb9f00661b0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fba04009d60 tx=0x7fba04009060 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:47.738 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.715+0000 7fb9f7fff700 1 -- 172.21.15.195:0/897299957 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fba00050920 con 0x7fba100c34a0 2022-06-27T12:11:47.957 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.946+0000 7fba165cc700 1 -- 172.21.15.195:0/897299957 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fba100c3130 con 0x7fba100c34a0 2022-06-27T12:11:47.957 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.947+0000 7fb9f7fff700 1 -- 172.21.15.195:0/897299957 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+5368 (secure 0 0 0) 0x7fba00015020 con 0x7fba100c34a0 2022-06-27T12:11:47.958 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:11:47.959 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:11:43.349160+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6050},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6050":{"gid":6050,"name":"b","rank":0,"incarnation":91,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.202:6833/3890799919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3890799919},{"type":"v1","addr":"172.21.15.202:6833","nonce":3890799919}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:11:47.961 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.949+0000 7fba165cc700 1 -- 172.21.15.195:0/897299957 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb9f0063d00 msgr2=0x7fb9f00661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:47.970 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.949+0000 7fba165cc700 1 --2- 172.21.15.195:0/897299957 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb9f0063d00 0x7fb9f00661b0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fba04009d60 tx=0x7fba04009060 comp rx=0 tx=0).stop 2022-06-27T12:11:47.971 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.949+0000 7fba165cc700 1 -- 172.21.15.195:0/897299957 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fba100c34a0 msgr2=0x7fba100bf1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:47.971 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.949+0000 7fba165cc700 1 --2- 172.21.15.195:0/897299957 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fba100c34a0 0x7fba100bf1c0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7fba100650f0 tx=0x7fba00017000 comp rx=0 tx=0).stop 2022-06-27T12:11:47.972 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.949+0000 7fba165cc700 1 -- 172.21.15.195:0/897299957 shutdown_connections 2022-06-27T12:11:47.972 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.949+0000 7fba165cc700 1 --2- 172.21.15.195:0/897299957 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb9f0063d00 0x7fb9f00661b0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:47.973 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.949+0000 7fba165cc700 1 --2- 172.21.15.195:0/897299957 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fba100bed50 0x7fba100bd600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:47.973 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.949+0000 7fba165cc700 1 --2- 172.21.15.195:0/897299957 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fba1006e850 0x7fba100bf780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:47.973 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.949+0000 7fba165cc700 1 --2- 172.21.15.195:0/897299957 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fba100c34a0 0x7fba100bf1c0 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:47.974 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.949+0000 7fba165cc700 1 -- 172.21.15.195:0/897299957 >> 172.21.15.195:0/897299957 conn(0x7fba100c6350 msgr2=0x7fba100c26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:11:47.975 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.949+0000 7fba165cc700 1 -- 172.21.15.195:0/897299957 shutdown_connections 2022-06-27T12:11:47.975 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:47.949+0000 7fba165cc700 1 -- 172.21.15.195:0/897299957 wait complete. 2022-06-27T12:11:47.976 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 93 2022-06-27T12:11:47.978 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:11:49.980 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:11:50.119 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.115+0000 7f4469450700 1 Processor -- start 2022-06-27T12:11:50.120 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.115+0000 7f4469450700 1 -- start start 2022-06-27T12:11:50.136 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.116+0000 7f4469450700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f44640c40d0 0x7f4464157010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:50.137 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.116+0000 7f4469450700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f44640c6190 0x7f4464067ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:50.137 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.116+0000 7f4469450700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4464156b80 con 0x7f44640c44a0 2022-06-27T12:11:50.139 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.116+0000 7f4469450700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f446406a040 con 0x7f44640c6190 2022-06-27T12:11:50.139 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.116+0000 7f4469450700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f446406a300 con 0x7f44640c40d0 2022-06-27T12:11:50.139 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.116+0000 7f44627fc700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f44640c6190 0x7f4464067ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:50.140 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.116+0000 7f44637fe700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f44640c44a0 0x7f446414cb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:56700/0 (socket says 172.21.15.195:56700) 2022-06-27T12:11:50.140 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.116+0000 7f44637fe700 1 -- 172.21.15.195:0/4154882988 learned_addr learned my addr 172.21.15.195:0/4154882988 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:11:50.141 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.116+0000 7f44627fc700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f44640c6190 0x7f4464067ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50512/0 (socket says 172.21.15.195:50512) 2022-06-27T12:11:50.141 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.116+0000 7f4462ffd700 1 --2- 172.21.15.195:0/4154882988 >> v2:172.21.15.202:3300/0 conn(0x7f44640c40d0 0x7f4464157010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:50.141 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.116+0000 7f44627fc700 1 -- 172.21.15.195:0/4154882988 >> v2:172.21.15.202:3300/0 conn(0x7f44640c40d0 msgr2=0x7f4464157010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:11:50.142 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.116+0000 7f44627fc700 1 --2- 172.21.15.195:0/4154882988 >> v2:172.21.15.202:3300/0 conn(0x7f44640c40d0 0x7f4464157010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:50.142 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.116+0000 7f44627fc700 1 -- 172.21.15.195:0/4154882988 >> v1:172.21.15.195:6789/0 conn(0x7f44640c44a0 legacy=0x7f446414cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:50.143 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.116+0000 7f4461ffb700 1 -- 172.21.15.195:0/4154882988 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1077903754 0 0) 0x7f4464156b80 con 0x7f44640c44a0 2022-06-27T12:11:50.144 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.116+0000 7f44627fc700 1 -- 172.21.15.195:0/4154882988 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f446406a5e0 con 0x7f44640c6190 2022-06-27T12:11:50.144 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.116+0000 7f4462ffd700 1 --2- 172.21.15.195:0/4154882988 >> v2:172.21.15.202:3300/0 conn(0x7f44640c40d0 0x7f4464157010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T12:11:50.145 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.116+0000 7f44627fc700 1 --2- 172.21.15.195:0/4154882988 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f44640c6190 0x7f4464067ed0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f44640cdb00 tx=0x7f4458009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=982b63abb08e3b5a server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:50.172 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.117+0000 7f4461ffb700 1 -- 172.21.15.195:0/4154882988 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4458011630 con 0x7f44640c6190 2022-06-27T12:11:50.173 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.117+0000 7f4461ffb700 1 -- 172.21.15.195:0/4154882988 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4458011790 con 0x7f44640c6190 2022-06-27T12:11:50.173 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.117+0000 7f4461ffb700 1 -- 172.21.15.195:0/4154882988 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4458011b30 con 0x7f44640c6190 2022-06-27T12:11:50.174 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.117+0000 7f4469450700 1 -- 172.21.15.195:0/4154882988 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f44640c6190 msgr2=0x7f4464067ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:50.174 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.117+0000 7f4469450700 1 --2- 172.21.15.195:0/4154882988 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f44640c6190 0x7f4464067ed0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f44640cdb00 tx=0x7f4458009060 comp rx=0 tx=0).stop 2022-06-27T12:11:50.175 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.117+0000 7f4469450700 1 -- 172.21.15.195:0/4154882988 shutdown_connections 2022-06-27T12:11:50.175 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.117+0000 7f4469450700 1 --2- 172.21.15.195:0/4154882988 >> v2:172.21.15.202:3300/0 conn(0x7f44640c40d0 0x7f4464157010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:50.176 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.117+0000 7f4469450700 1 --2- 172.21.15.195:0/4154882988 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f44640c6190 0x7f4464067ed0 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:50.176 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.117+0000 7f4469450700 1 -- 172.21.15.195:0/4154882988 >> 172.21.15.195:0/4154882988 conn(0x7f44640ccb80 msgr2=0x7f44640ccf80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:11:50.177 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.117+0000 7f4469450700 1 -- 172.21.15.195:0/4154882988 shutdown_connections 2022-06-27T12:11:50.177 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.117+0000 7f4469450700 1 -- 172.21.15.195:0/4154882988 wait complete. 2022-06-27T12:11:50.177 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.119+0000 7f4469450700 1 Processor -- start 2022-06-27T12:11:50.179 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.119+0000 7f4469450700 1 -- start start 2022-06-27T12:11:50.180 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.119+0000 7f4469450700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f44640c40d0 0x7f446415a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:50.180 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.119+0000 7f4469450700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f44640c44a0 0x7f446414f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:50.180 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.119+0000 7f4469450700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f44640c6190 0x7f446414fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:50.181 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.119+0000 7f4469450700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4464158ae0 con 0x7f44640c40d0 2022-06-27T12:11:50.182 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.119+0000 7f4469450700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4464159860 con 0x7f44640c6190 2022-06-27T12:11:50.182 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.119+0000 7f4469450700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4464153ca0 con 0x7f44640c44a0 2022-06-27T12:11:50.184 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.119+0000 7f4462ffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f44640c40d0 0x7f446415a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:50.186 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.120+0000 7f4462ffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f44640c40d0 0x7f446415a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50516/0 (socket says 172.21.15.195:50516) 2022-06-27T12:11:50.209 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.120+0000 7f4462ffd700 1 -- 172.21.15.195:0/3064758585 learned_addr learned my addr 172.21.15.195:0/3064758585 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:11:50.209 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.120+0000 7f44627fc700 1 --2- 172.21.15.195:0/3064758585 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f44640c44a0 0x7f446414f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:50.210 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.120+0000 7f44637fe700 1 --2- 172.21.15.195:0/3064758585 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f44640c6190 0x7f446414fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:50.210 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.120+0000 7f4462ffd700 1 -- 172.21.15.195:0/3064758585 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f44640c44a0 msgr2=0x7f446414f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:50.211 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.120+0000 7f4462ffd700 1 --2- 172.21.15.195:0/3064758585 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f44640c44a0 0x7f446414f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:50.211 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.120+0000 7f4462ffd700 1 -- 172.21.15.195:0/3064758585 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f44640c6190 msgr2=0x7f446414fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:50.212 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.120+0000 7f4462ffd700 1 --2- 172.21.15.195:0/3064758585 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f44640c6190 0x7f446414fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:50.212 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.120+0000 7f4462ffd700 1 -- 172.21.15.195:0/3064758585 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44580092e0 con 0x7f44640c40d0 2022-06-27T12:11:50.213 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.120+0000 7f4462ffd700 1 --2- 172.21.15.195:0/3064758585 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f44640c40d0 0x7f446415a2a0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f445400c8b0 tx=0x7f4454008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:50.213 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.120+0000 7f444affd700 1 -- 172.21.15.195:0/3064758585 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4454011600 con 0x7f44640c40d0 2022-06-27T12:11:50.213 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.120+0000 7f444affd700 1 -- 172.21.15.195:0/3064758585 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4454011760 con 0x7f44640c40d0 2022-06-27T12:11:50.214 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.120+0000 7f4469450700 1 -- 172.21.15.195:0/3064758585 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4464151450 con 0x7f44640c40d0 2022-06-27T12:11:50.214 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.120+0000 7f444affd700 1 -- 172.21.15.195:0/3064758585 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4454011b00 con 0x7f44640c40d0 2022-06-27T12:11:50.215 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.120+0000 7f4469450700 1 -- 172.21.15.195:0/3064758585 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44641593d0 con 0x7f44640c40d0 2022-06-27T12:11:50.215 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.121+0000 7f4469450700 1 -- 172.21.15.195:0/3064758585 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44640722c0 con 0x7f44640c40d0 2022-06-27T12:11:50.216 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.122+0000 7f444affd700 1 -- 172.21.15.195:0/3064758585 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f4454021070 con 0x7f44640c40d0 2022-06-27T12:11:50.216 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.122+0000 7f444affd700 1 --2- 172.21.15.195:0/3064758585 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4444063d00 0x7f44440661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:50.216 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.122+0000 7f444affd700 1 -- 172.21.15.195:0/3064758585 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7406+0+0 (secure 0 0 0) 0x7f4454080bc0 con 0x7f44640c40d0 2022-06-27T12:11:50.217 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.122+0000 7f44627fc700 1 --2- 172.21.15.195:0/3064758585 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4444063d00 0x7f44440661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:50.217 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.122+0000 7f44627fc700 1 --2- 172.21.15.195:0/3064758585 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4444063d00 0x7f44440661b0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f4458011fd0 tx=0x7f4458017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:50.218 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.126+0000 7f444affd700 1 -- 172.21.15.195:0/3064758585 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f445404f270 con 0x7f44640c40d0 2022-06-27T12:11:50.347 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.346+0000 7f4469450700 1 -- 172.21.15.195:0/3064758585 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f44641596b0 con 0x7f44640c40d0 2022-06-27T12:11:50.348 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.347+0000 7f444affd700 1 -- 172.21.15.195:0/3064758585 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+5368 (secure 0 0 0) 0x7f445401f090 con 0x7f44640c40d0 2022-06-27T12:11:50.348 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:11:50.349 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:11:43.349160+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6050},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6050":{"gid":6050,"name":"b","rank":0,"incarnation":91,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.202:6833/3890799919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3890799919},{"type":"v1","addr":"172.21.15.202:6833","nonce":3890799919}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:11:50.362 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.349+0000 7f4469450700 1 -- 172.21.15.195:0/3064758585 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4444063d00 msgr2=0x7f44440661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:50.363 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.349+0000 7f4469450700 1 --2- 172.21.15.195:0/3064758585 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4444063d00 0x7f44440661b0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f4458011fd0 tx=0x7f4458017000 comp rx=0 tx=0).stop 2022-06-27T12:11:50.363 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.349+0000 7f4469450700 1 -- 172.21.15.195:0/3064758585 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f44640c40d0 msgr2=0x7f446415a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:50.364 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.349+0000 7f4469450700 1 --2- 172.21.15.195:0/3064758585 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f44640c40d0 0x7f446415a2a0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f445400c8b0 tx=0x7f4454008040 comp rx=0 tx=0).stop 2022-06-27T12:11:50.364 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.349+0000 7f4469450700 1 -- 172.21.15.195:0/3064758585 shutdown_connections 2022-06-27T12:11:50.365 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.349+0000 7f4469450700 1 --2- 172.21.15.195:0/3064758585 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4444063d00 0x7f44440661b0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:50.365 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.349+0000 7f4469450700 1 --2- 172.21.15.195:0/3064758585 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f44640c40d0 0x7f446415a2a0 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:50.366 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.349+0000 7f4469450700 1 --2- 172.21.15.195:0/3064758585 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f44640c6190 0x7f446414fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:50.366 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.349+0000 7f4469450700 1 --2- 172.21.15.195:0/3064758585 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f44640c44a0 0x7f446414f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:50.367 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.349+0000 7f4469450700 1 -- 172.21.15.195:0/3064758585 >> 172.21.15.195:0/3064758585 conn(0x7f44640ccb80 msgr2=0x7f44641545c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:11:50.367 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.349+0000 7f4469450700 1 -- 172.21.15.195:0/3064758585 shutdown_connections 2022-06-27T12:11:50.368 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:50.349+0000 7f4469450700 1 -- 172.21.15.195:0/3064758585 wait complete. 2022-06-27T12:11:50.368 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 93 2022-06-27T12:11:50.370 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 93, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-06-27T11:47:52.358095+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 93, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6050": { "addr": "172.21.15.202:6833/3890799919", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6832", "nonce": 3890799919, "type": "v2" }, { "addr": "172.21.15.202:6833", "nonce": 3890799919, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 6050, "incarnation": 91, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:rejoin", "state_seq": 11 } }, "last_failure": 0, "last_failure_osd_epoch": 62, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-06-27T12:11:43.349160+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6050 } } } ], "standbys": [ { "addr": "172.21.15.202:6837/3305006841", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6836", "nonce": 3305006841, "type": "v2" }, { "addr": "172.21.15.202:6837", "nonce": 3305006841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6835/2060709822", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6834", "nonce": 2060709822, "type": "v2" }, { "addr": "172.21.15.195:6835", "nonce": 2060709822, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6838/1598751652", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6836", "nonce": 1598751652, "type": "v2" }, { "addr": "172.21.15.195:6838", "nonce": 1598751652, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4275, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6839/1313800661", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6837", "nonce": 1313800661, "type": "v2" }, { "addr": "172.21.15.195:6839", "nonce": 1313800661, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4278, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-06-27T12:11:52.372 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:11:52.556 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.530+0000 7f48a9ab9700 1 Processor -- start 2022-06-27T12:11:52.556 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.530+0000 7f48a9ab9700 1 -- start start 2022-06-27T12:11:52.557 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.531+0000 7f48a9ab9700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f48a406f640 0x7f48a4156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:52.557 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.531+0000 7f48a9ab9700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f48a40c4160 0x7f48a40c1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:52.558 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.531+0000 7f48a9ab9700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f48a40c39e0 0x7f48a41535a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:52.558 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.531+0000 7f48a9ab9700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48a40c1a20 con 0x7f48a406f640 2022-06-27T12:11:52.558 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.531+0000 7f48a9ab9700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48a40c1130 con 0x7f48a40c4160 2022-06-27T12:11:52.559 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.531+0000 7f48a9ab9700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48a406ea90 con 0x7f48a40c39e0 2022-06-27T12:11:52.559 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.531+0000 7f48a37fe700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f48a406f640 0x7f48a4156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:52.559 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.531+0000 7f48a37fe700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f48a406f640 0x7f48a4156ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50524/0 (socket says 172.21.15.195:50524) 2022-06-27T12:11:52.560 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.531+0000 7f48a37fe700 1 -- 172.21.15.195:0/171289969 learned_addr learned my addr 172.21.15.195:0/171289969 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:11:52.560 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.531+0000 7f48a3fff700 1 --2- 172.21.15.195:0/171289969 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f48a40c39e0 0x7f48a41535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:52.561 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.531+0000 7f48a2ffd700 1 --2- 172.21.15.195:0/171289969 >> v2:172.21.15.202:3300/0 conn(0x7f48a40c4160 0x7f48a40c1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:52.561 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.531+0000 7f48a37fe700 1 -- 172.21.15.195:0/171289969 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f48a40c39e0 msgr2=0x7f48a41535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:11:52.568 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.531+0000 7f48a37fe700 1 --2- 172.21.15.195:0/171289969 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f48a40c39e0 0x7f48a41535a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:52.568 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.531+0000 7f48a37fe700 1 -- 172.21.15.195:0/171289969 >> v2:172.21.15.202:3300/0 conn(0x7f48a40c4160 msgr2=0x7f48a40c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:11:52.569 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.531+0000 7f48a37fe700 1 --2- 172.21.15.195:0/171289969 >> v2:172.21.15.202:3300/0 conn(0x7f48a40c4160 0x7f48a40c1eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:52.569 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.531+0000 7f48a37fe700 1 -- 172.21.15.195:0/171289969 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48a406ed70 con 0x7f48a406f640 2022-06-27T12:11:52.569 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.531+0000 7f48a37fe700 1 --2- 172.21.15.195:0/171289969 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f48a406f640 0x7f48a4156ed0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f48a406dd70 tx=0x7f4898009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd585a2cf1b1e76b server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:52.570 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.532+0000 7f48a27fc700 1 -- 172.21.15.195:0/171289969 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f489800b6d0 con 0x7f48a406f640 2022-06-27T12:11:52.570 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.532+0000 7f48a27fc700 1 -- 172.21.15.195:0/171289969 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f489800b830 con 0x7f48a406f640 2022-06-27T12:11:52.571 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.532+0000 7f48a27fc700 1 -- 172.21.15.195:0/171289969 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f489800bbd0 con 0x7f48a406f640 2022-06-27T12:11:52.571 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.532+0000 7f48a9ab9700 1 -- 172.21.15.195:0/171289969 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f48a406f640 msgr2=0x7f48a4156ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:52.572 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.532+0000 7f48a9ab9700 1 --2- 172.21.15.195:0/171289969 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f48a406f640 0x7f48a4156ed0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f48a406dd70 tx=0x7f4898009120 comp rx=0 tx=0).stop 2022-06-27T12:11:52.572 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.532+0000 7f48a9ab9700 1 -- 172.21.15.195:0/171289969 shutdown_connections 2022-06-27T12:11:52.573 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.532+0000 7f48a9ab9700 1 --2- 172.21.15.195:0/171289969 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f48a40c39e0 0x7f48a41535a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:52.573 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.532+0000 7f48a9ab9700 1 --2- 172.21.15.195:0/171289969 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f48a406f640 0x7f48a4156ed0 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:52.574 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.532+0000 7f48a9ab9700 1 --2- 172.21.15.195:0/171289969 >> v2:172.21.15.202:3300/0 conn(0x7f48a40c4160 0x7f48a40c1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:52.575 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.532+0000 7f48a9ab9700 1 -- 172.21.15.195:0/171289969 >> 172.21.15.195:0/171289969 conn(0x7f48a40c6350 msgr2=0x7f48a40c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:11:52.575 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.532+0000 7f48a9ab9700 1 -- 172.21.15.195:0/171289969 shutdown_connections 2022-06-27T12:11:52.575 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.532+0000 7f48a9ab9700 1 -- 172.21.15.195:0/171289969 wait complete. 2022-06-27T12:11:52.576 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.534+0000 7f48a9ab9700 1 Processor -- start 2022-06-27T12:11:52.576 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.534+0000 7f48a9ab9700 1 -- start start 2022-06-27T12:11:52.577 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.535+0000 7f48a9ab9700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f48a406f640 0x7f48a4151190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:52.577 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.535+0000 7f48a9ab9700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f48a40c39e0 0x7f48a41516d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:52.606 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.535+0000 7f48a9ab9700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f48a40c4160 0x7f48a414c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:52.607 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.535+0000 7f48a9ab9700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48a40c0ce0 con 0x7f48a40c39e0 2022-06-27T12:11:52.607 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.535+0000 7f48a9ab9700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48a4158c90 con 0x7f48a406f640 2022-06-27T12:11:52.608 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.535+0000 7f48a9ab9700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48a40c0780 con 0x7f48a40c4160 2022-06-27T12:11:52.608 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.535+0000 7f48a2ffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f48a40c39e0 0x7f48a41516d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:52.609 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.535+0000 7f48a2ffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f48a40c39e0 0x7f48a41516d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50532/0 (socket says 172.21.15.195:50532) 2022-06-27T12:11:52.609 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.535+0000 7f48a2ffd700 1 -- 172.21.15.195:0/3134074000 learned_addr learned my addr 172.21.15.195:0/3134074000 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:11:52.610 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.535+0000 7f48a37fe700 1 --2- 172.21.15.195:0/3134074000 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f48a406f640 0x7f48a4151190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:52.610 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.535+0000 7f48a3fff700 1 --2- 172.21.15.195:0/3134074000 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f48a40c4160 0x7f48a414c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:52.611 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.535+0000 7f48a2ffd700 1 -- 172.21.15.195:0/3134074000 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f48a40c4160 msgr2=0x7f48a414c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:52.611 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.535+0000 7f48a2ffd700 1 --2- 172.21.15.195:0/3134074000 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f48a40c4160 0x7f48a414c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:52.612 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.535+0000 7f48a2ffd700 1 -- 172.21.15.195:0/3134074000 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f48a406f640 msgr2=0x7f48a4151190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:52.612 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.535+0000 7f48a2ffd700 1 --2- 172.21.15.195:0/3134074000 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f48a406f640 0x7f48a4151190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:52.629 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.535+0000 7f48a2ffd700 1 -- 172.21.15.195:0/3134074000 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48980093a0 con 0x7f48a40c39e0 2022-06-27T12:11:52.630 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.535+0000 7f48a2ffd700 1 --2- 172.21.15.195:0/3134074000 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f48a40c39e0 0x7f48a41516d0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f489400d890 tx=0x7f4894008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:52.630 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.536+0000 7f488b7fe700 1 -- 172.21.15.195:0/3134074000 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48940125f0 con 0x7f48a40c39e0 2022-06-27T12:11:52.631 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.536+0000 7f488b7fe700 1 -- 172.21.15.195:0/3134074000 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4894012750 con 0x7f48a40c39e0 2022-06-27T12:11:52.631 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.536+0000 7f48a9ab9700 1 -- 172.21.15.195:0/3134074000 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48a40c0a20 con 0x7f48a40c39e0 2022-06-27T12:11:52.632 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.536+0000 7f488b7fe700 1 -- 172.21.15.195:0/3134074000 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4894012af0 con 0x7f48a40c39e0 2022-06-27T12:11:52.632 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.536+0000 7f48a9ab9700 1 -- 172.21.15.195:0/3134074000 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48a4158640 con 0x7f48a40c39e0 2022-06-27T12:11:52.633 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.537+0000 7f48a9ab9700 1 -- 172.21.15.195:0/3134074000 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48a414c5b0 con 0x7f48a40c39e0 2022-06-27T12:11:52.633 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.537+0000 7f488b7fe700 1 -- 172.21.15.195:0/3134074000 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f489401e460 con 0x7f48a40c39e0 2022-06-27T12:11:52.633 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.537+0000 7f488b7fe700 1 --2- 172.21.15.195:0/3134074000 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4884063d00 0x7f48840661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:52.634 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.537+0000 7f488b7fe700 1 -- 172.21.15.195:0/3134074000 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7406+0+0 (secure 0 0 0) 0x7f4894081240 con 0x7f48a40c39e0 2022-06-27T12:11:52.634 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.537+0000 7f48a37fe700 1 --2- 172.21.15.195:0/3134074000 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4884063d00 0x7f48840661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:52.635 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.538+0000 7f48a37fe700 1 --2- 172.21.15.195:0/3134074000 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4884063d00 0x7f48840661b0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f4898009160 tx=0x7f4898017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:52.635 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.542+0000 7f488b7fe700 1 -- 172.21.15.195:0/3134074000 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f489404f890 con 0x7f48a40c39e0 2022-06-27T12:11:52.769 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.762+0000 7f48a9ab9700 1 -- 172.21.15.195:0/3134074000 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f48a4158dd0 con 0x7f48a40c39e0 2022-06-27T12:11:52.770 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.763+0000 7f488b7fe700 1 -- 172.21.15.195:0/3134074000 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+5374 (secure 0 0 0) 0x7f48a4158dd0 con 0x7f48a40c39e0 2022-06-27T12:11:52.770 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:11:52.771 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:11:50.368754+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6050},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6050":{"gid":6050,"name":"b","rank":0,"incarnation":91,"state":"up:clientreplay","state_seq":13,"addr":"172.21.15.202:6833/3890799919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3890799919},{"type":"v1","addr":"172.21.15.202:6833","nonce":3890799919}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:11:52.774 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.764+0000 7f48a9ab9700 1 -- 172.21.15.195:0/3134074000 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4884063d00 msgr2=0x7f48840661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:52.774 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.764+0000 7f48a9ab9700 1 --2- 172.21.15.195:0/3134074000 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4884063d00 0x7f48840661b0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f4898009160 tx=0x7f4898017000 comp rx=0 tx=0).stop 2022-06-27T12:11:52.775 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.764+0000 7f48a9ab9700 1 -- 172.21.15.195:0/3134074000 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f48a40c39e0 msgr2=0x7f48a41516d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:52.775 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.764+0000 7f48a9ab9700 1 --2- 172.21.15.195:0/3134074000 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f48a40c39e0 0x7f48a41516d0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f489400d890 tx=0x7f4894008040 comp rx=0 tx=0).stop 2022-06-27T12:11:52.776 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.764+0000 7f48a9ab9700 1 -- 172.21.15.195:0/3134074000 shutdown_connections 2022-06-27T12:11:52.776 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.764+0000 7f48a9ab9700 1 --2- 172.21.15.195:0/3134074000 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4884063d00 0x7f48840661b0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:52.777 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.764+0000 7f48a9ab9700 1 --2- 172.21.15.195:0/3134074000 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f48a40c4160 0x7f48a414c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:52.806 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.764+0000 7f48a9ab9700 1 --2- 172.21.15.195:0/3134074000 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f48a406f640 0x7f48a4151190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:52.807 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.764+0000 7f48a9ab9700 1 --2- 172.21.15.195:0/3134074000 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f48a40c39e0 0x7f48a41516d0 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:52.824 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.764+0000 7f48a9ab9700 1 -- 172.21.15.195:0/3134074000 >> 172.21.15.195:0/3134074000 conn(0x7f48a40c6350 msgr2=0x7f48a4153370 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:11:52.824 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.765+0000 7f48a9ab9700 1 -- 172.21.15.195:0/3134074000 shutdown_connections 2022-06-27T12:11:52.825 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:52.765+0000 7f48a9ab9700 1 -- 172.21.15.195:0/3134074000 wait complete. 2022-06-27T12:11:52.825 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 94 2022-06-27T12:11:52.943 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:11:54.950 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:11:55.144 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.116+0000 7f56f1d32700 1 Processor -- start 2022-06-27T12:11:55.144 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.116+0000 7f56f1d32700 1 -- start start 2022-06-27T12:11:55.145 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.117+0000 7f56f1d32700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f56ec06a450 0x7f56ec063af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:55.145 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.117+0000 7f56f1d32700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f56ec1508d0 0x7f56ec0bbf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:55.146 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.117+0000 7f56f1d32700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56ec064030 con 0x7f56ec150500 2022-06-27T12:11:55.146 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.117+0000 7f56f1d32700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56ec0c4160 con 0x7f56ec06a450 2022-06-27T12:11:55.146 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.117+0000 7f56f1d32700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56ec0c4440 con 0x7f56ec1508d0 2022-06-27T12:11:55.147 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.117+0000 7f56eb7fe700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f56ec150500 0x7f56ec0c48d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:56726/0 (socket says 172.21.15.195:56726) 2022-06-27T12:11:55.147 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.117+0000 7f56eb7fe700 1 -- 172.21.15.195:0/3193535363 learned_addr learned my addr 172.21.15.195:0/3193535363 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:11:55.148 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.117+0000 7f56ebfff700 1 --2- 172.21.15.195:0/3193535363 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f56ec1508d0 0x7f56ec0bbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:55.148 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.117+0000 7f56eaffd700 1 --2- 172.21.15.195:0/3193535363 >> v2:172.21.15.202:3300/0 conn(0x7f56ec06a450 0x7f56ec063af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:55.148 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.117+0000 7f56ea7fc700 1 -- 172.21.15.195:0/3193535363 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3809714421 0 0) 0x7f56dc001290 con 0x7f56ec150500 2022-06-27T12:11:55.149 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.117+0000 7f56ea7fc700 1 -- 172.21.15.195:0/3193535363 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56cc003430 con 0x7f56ec150500 2022-06-27T12:11:55.149 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.118+0000 7f56ebfff700 1 -- 172.21.15.195:0/3193535363 >> v2:172.21.15.202:3300/0 conn(0x7f56ec06a450 msgr2=0x7f56ec063af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:11:55.150 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.118+0000 7f56ebfff700 1 --2- 172.21.15.195:0/3193535363 >> v2:172.21.15.202:3300/0 conn(0x7f56ec06a450 0x7f56ec063af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:55.150 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.118+0000 7f56ebfff700 1 -- 172.21.15.195:0/3193535363 >> v1:172.21.15.195:6789/0 conn(0x7f56ec150500 legacy=0x7f56ec0c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:55.151 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.118+0000 7f56ebfff700 1 -- 172.21.15.195:0/3193535363 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56ec15a800 con 0x7f56ec1508d0 2022-06-27T12:11:55.151 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.118+0000 7f56eaffd700 1 --2- 172.21.15.195:0/3193535363 >> v2:172.21.15.202:3300/0 conn(0x7f56ec06a450 0x7f56ec063af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:11:55.151 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.118+0000 7f56ebfff700 1 --2- 172.21.15.195:0/3193535363 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f56ec1508d0 0x7f56ec0bbf50 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f56e4009e60 tx=0x7f56e4009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af1955f7945340d7 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:55.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.118+0000 7f56ea7fc700 1 -- 172.21.15.195:0/3193535363 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e400e070 con 0x7f56ec1508d0 2022-06-27T12:11:55.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.118+0000 7f56ea7fc700 1 -- 172.21.15.195:0/3193535363 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56e400bab0 con 0x7f56ec1508d0 2022-06-27T12:11:55.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.118+0000 7f56ea7fc700 1 -- 172.21.15.195:0/3193535363 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e400be70 con 0x7f56ec1508d0 2022-06-27T12:11:55.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.118+0000 7f56f1d32700 1 -- 172.21.15.195:0/3193535363 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f56ec1508d0 msgr2=0x7f56ec0bbf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:55.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.118+0000 7f56f1d32700 1 --2- 172.21.15.195:0/3193535363 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f56ec1508d0 0x7f56ec0bbf50 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f56e4009e60 tx=0x7f56e4009780 comp rx=0 tx=0).stop 2022-06-27T12:11:55.155 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.118+0000 7f56f1d32700 1 -- 172.21.15.195:0/3193535363 shutdown_connections 2022-06-27T12:11:55.155 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.118+0000 7f56f1d32700 1 --2- 172.21.15.195:0/3193535363 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f56ec1508d0 0x7f56ec0bbf50 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:55.156 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.118+0000 7f56f1d32700 1 --2- 172.21.15.195:0/3193535363 >> v2:172.21.15.202:3300/0 conn(0x7f56ec06a450 0x7f56ec063af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:55.156 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.118+0000 7f56f1d32700 1 -- 172.21.15.195:0/3193535363 >> 172.21.15.195:0/3193535363 conn(0x7f56ec0c9b00 msgr2=0x7f56ec0c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:11:55.157 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.119+0000 7f56f1d32700 1 -- 172.21.15.195:0/3193535363 shutdown_connections 2022-06-27T12:11:55.157 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.119+0000 7f56f1d32700 1 -- 172.21.15.195:0/3193535363 wait complete. 2022-06-27T12:11:55.158 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.120+0000 7f56f1d32700 1 Processor -- start 2022-06-27T12:11:55.158 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.120+0000 7f56f1d32700 1 -- start start 2022-06-27T12:11:55.158 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.121+0000 7f56f1d32700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f56ec06a450 0x7f56ec0bc050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:55.159 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.121+0000 7f56f1d32700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f56ec150500 0x7f56ec169690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:55.159 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.121+0000 7f56f1d32700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f56ec1508d0 0x7f56ec14da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:55.160 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.121+0000 7f56f1d32700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56ec154f30 con 0x7f56ec06a450 2022-06-27T12:11:55.160 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.121+0000 7f56f1d32700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56ec0c2e00 con 0x7f56ec1508d0 2022-06-27T12:11:55.160 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.121+0000 7f56f1d32700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56ec166c60 con 0x7f56ec150500 2022-06-27T12:11:55.161 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.121+0000 7f56eb7fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f56ec06a450 0x7f56ec0bc050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:55.161 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.121+0000 7f56eb7fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f56ec06a450 0x7f56ec0bc050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50544/0 (socket says 172.21.15.195:50544) 2022-06-27T12:11:55.161 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.121+0000 7f56eb7fe700 1 -- 172.21.15.195:0/3412757074 learned_addr learned my addr 172.21.15.195:0/3412757074 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:11:55.162 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.121+0000 7f56eaffd700 1 --2- 172.21.15.195:0/3412757074 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f56ec150500 0x7f56ec169690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:55.162 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.121+0000 7f56ebfff700 1 --2- 172.21.15.195:0/3412757074 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f56ec1508d0 0x7f56ec14da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:55.163 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.121+0000 7f56eb7fe700 1 -- 172.21.15.195:0/3412757074 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f56ec150500 msgr2=0x7f56ec169690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:55.164 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.121+0000 7f56eb7fe700 1 --2- 172.21.15.195:0/3412757074 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f56ec150500 0x7f56ec169690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:55.164 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.121+0000 7f56eb7fe700 1 -- 172.21.15.195:0/3412757074 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f56ec1508d0 msgr2=0x7f56ec14da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:55.166 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.121+0000 7f56eb7fe700 1 --2- 172.21.15.195:0/3412757074 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f56ec1508d0 0x7f56ec14da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:55.166 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.121+0000 7f56eb7fe700 1 -- 172.21.15.195:0/3412757074 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56d40092e0 con 0x7f56ec06a450 2022-06-27T12:11:55.167 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.121+0000 7f56ebfff700 1 --2- 172.21.15.195:0/3412757074 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f56ec1508d0 0x7f56ec14da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T12:11:55.167 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.121+0000 7f56eb7fe700 1 --2- 172.21.15.195:0/3412757074 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f56ec06a450 0x7f56ec0bc050 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f56dc00c510 tx=0x7f56dc001890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:55.168 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.121+0000 7f56e37fe700 1 -- 172.21.15.195:0/3412757074 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56dc0072e0 con 0x7f56ec06a450 2022-06-27T12:11:55.168 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.121+0000 7f56e37fe700 1 -- 172.21.15.195:0/3412757074 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56dc007440 con 0x7f56ec06a450 2022-06-27T12:11:55.169 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.121+0000 7f56e37fe700 1 -- 172.21.15.195:0/3412757074 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56dc005900 con 0x7f56ec06a450 2022-06-27T12:11:55.169 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.121+0000 7f56f1d32700 1 -- 172.21.15.195:0/3412757074 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56e4009a20 con 0x7f56ec06a450 2022-06-27T12:11:55.170 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.121+0000 7f56f1d32700 1 -- 172.21.15.195:0/3412757074 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56ec0c2460 con 0x7f56ec06a450 2022-06-27T12:11:55.170 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.122+0000 7f56f1d32700 1 -- 172.21.15.195:0/3412757074 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56ec0c3070 con 0x7f56ec06a450 2022-06-27T12:11:55.170 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.123+0000 7f56e37fe700 1 -- 172.21.15.195:0/3412757074 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f56dc019030 con 0x7f56ec06a450 2022-06-27T12:11:55.171 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.123+0000 7f56e37fe700 1 --2- 172.21.15.195:0/3412757074 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f56d0063d30 0x7f56d00661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:55.171 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.123+0000 7f56e37fe700 1 -- 172.21.15.195:0/3412757074 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7406+0+0 (secure 0 0 0) 0x7f56dc07ffb0 con 0x7f56ec06a450 2022-06-27T12:11:55.172 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.123+0000 7f56eaffd700 1 --2- 172.21.15.195:0/3412757074 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f56d0063d30 0x7f56d00661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:55.173 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.124+0000 7f56eaffd700 1 --2- 172.21.15.195:0/3412757074 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f56d0063d30 0x7f56d00661e0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f56d40092b0 tx=0x7f56d4008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:55.196 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.127+0000 7f56e37fe700 1 -- 172.21.15.195:0/3412757074 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f56dc04e660 con 0x7f56ec06a450 2022-06-27T12:11:55.361 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.360+0000 7f56f1d32700 1 -- 172.21.15.195:0/3412757074 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f56ec064030 con 0x7f56ec06a450 2022-06-27T12:11:55.361 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.361+0000 7f56e37fe700 1 -- 172.21.15.195:0/3412757074 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+5374 (secure 0 0 0) 0x7f56ec064030 con 0x7f56ec06a450 2022-06-27T12:11:55.362 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:11:55.363 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:11:50.368754+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6050},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6050":{"gid":6050,"name":"b","rank":0,"incarnation":91,"state":"up:clientreplay","state_seq":13,"addr":"172.21.15.202:6833/3890799919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3890799919},{"type":"v1","addr":"172.21.15.202:6833","nonce":3890799919}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:11:55.365 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.363+0000 7f56f1d32700 1 -- 172.21.15.195:0/3412757074 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f56d0063d30 msgr2=0x7f56d00661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:55.366 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.363+0000 7f56f1d32700 1 --2- 172.21.15.195:0/3412757074 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f56d0063d30 0x7f56d00661e0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f56d40092b0 tx=0x7f56d4008040 comp rx=0 tx=0).stop 2022-06-27T12:11:55.367 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.363+0000 7f56f1d32700 1 -- 172.21.15.195:0/3412757074 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f56ec06a450 msgr2=0x7f56ec0bc050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:55.367 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.363+0000 7f56f1d32700 1 --2- 172.21.15.195:0/3412757074 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f56ec06a450 0x7f56ec0bc050 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f56dc00c510 tx=0x7f56dc001890 comp rx=0 tx=0).stop 2022-06-27T12:11:55.367 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.363+0000 7f56f1d32700 1 -- 172.21.15.195:0/3412757074 shutdown_connections 2022-06-27T12:11:55.368 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.363+0000 7f56f1d32700 1 --2- 172.21.15.195:0/3412757074 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f56d0063d30 0x7f56d00661e0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:55.369 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.363+0000 7f56f1d32700 1 --2- 172.21.15.195:0/3412757074 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f56ec06a450 0x7f56ec0bc050 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:55.369 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.363+0000 7f56f1d32700 1 --2- 172.21.15.195:0/3412757074 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f56ec1508d0 0x7f56ec14da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:55.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.363+0000 7f56f1d32700 1 --2- 172.21.15.195:0/3412757074 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f56ec150500 0x7f56ec169690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:55.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.363+0000 7f56f1d32700 1 -- 172.21.15.195:0/3412757074 >> 172.21.15.195:0/3412757074 conn(0x7f56ec0c9b00 msgr2=0x7f56ec0c6e40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:11:55.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.363+0000 7f56f1d32700 1 -- 172.21.15.195:0/3412757074 shutdown_connections 2022-06-27T12:11:55.372 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:55.363+0000 7f56f1d32700 1 -- 172.21.15.195:0/3412757074 wait complete. 2022-06-27T12:11:55.372 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 94 2022-06-27T12:11:55.377 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:11:57.393 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:11:57.548 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.539+0000 7f890ab7e700 1 Processor -- start 2022-06-27T12:11:57.548 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.539+0000 7f890ab7e700 1 -- start start 2022-06-27T12:11:57.562 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.540+0000 7f890ab7e700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f890406f640 0x7f8904156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:57.563 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.540+0000 7f890ab7e700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f89040c4160 0x7f89040c1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:57.563 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.540+0000 7f890ab7e700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89040c1a20 con 0x7f89040c39e0 2022-06-27T12:11:57.563 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.540+0000 7f890ab7e700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89040c1130 con 0x7f89040c4160 2022-06-27T12:11:57.564 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.540+0000 7f890ab7e700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f890406ea90 con 0x7f890406f640 2022-06-27T12:11:57.565 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.540+0000 7f8903fff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f89040c4160 0x7f89040c1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:57.565 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.540+0000 7f890911b700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f89040c39e0 0x7f89041535a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:56746/0 (socket says 172.21.15.195:56746) 2022-06-27T12:11:57.566 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.540+0000 7f890911b700 1 -- 172.21.15.195:0/2261991637 learned_addr learned my addr 172.21.15.195:0/2261991637 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:11:57.567 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.540+0000 7f890891a700 1 --2- 172.21.15.195:0/2261991637 >> v2:172.21.15.202:3300/0 conn(0x7f890406f640 0x7f8904156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:57.568 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.540+0000 7f8903fff700 1 -- 172.21.15.195:0/2261991637 >> v2:172.21.15.202:3300/0 conn(0x7f890406f640 msgr2=0x7f8904156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:11:57.646 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.540+0000 7f8903fff700 1 --2- 172.21.15.195:0/2261991637 >> v2:172.21.15.202:3300/0 conn(0x7f890406f640 0x7f8904156ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:57.647 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.540+0000 7f8903fff700 1 -- 172.21.15.195:0/2261991637 >> v1:172.21.15.195:6789/0 conn(0x7f89040c39e0 legacy=0x7f89041535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:57.647 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.540+0000 7f89037fe700 1 -- 172.21.15.195:0/2261991637 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3802023793 0 0) 0x7f89040c1a20 con 0x7f89040c39e0 2022-06-27T12:11:57.647 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.540+0000 7f8903fff700 1 -- 172.21.15.195:0/2261991637 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f890406ed70 con 0x7f89040c4160 2022-06-27T12:11:57.648 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.540+0000 7f8903fff700 1 --2- 172.21.15.195:0/2261991637 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f89040c4160 0x7f89040c1eb0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f890406db30 tx=0x7f88f0009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22451a6044796633 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:57.648 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.541+0000 7f89037fe700 1 -- 172.21.15.195:0/2261991637 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f0011630 con 0x7f89040c4160 2022-06-27T12:11:57.649 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.541+0000 7f89037fe700 1 -- 172.21.15.195:0/2261991637 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88f0011790 con 0x7f89040c4160 2022-06-27T12:11:57.649 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.541+0000 7f89037fe700 1 -- 172.21.15.195:0/2261991637 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f0011b30 con 0x7f89040c4160 2022-06-27T12:11:57.650 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.541+0000 7f890ab7e700 1 -- 172.21.15.195:0/2261991637 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f89040c4160 msgr2=0x7f89040c1eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:57.650 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.541+0000 7f890ab7e700 1 --2- 172.21.15.195:0/2261991637 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f89040c4160 0x7f89040c1eb0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f890406db30 tx=0x7f88f0009060 comp rx=0 tx=0).stop 2022-06-27T12:11:57.651 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.541+0000 7f890ab7e700 1 -- 172.21.15.195:0/2261991637 shutdown_connections 2022-06-27T12:11:57.651 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.541+0000 7f890ab7e700 1 --2- 172.21.15.195:0/2261991637 >> v2:172.21.15.202:3300/0 conn(0x7f890406f640 0x7f8904156ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:57.652 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.541+0000 7f890ab7e700 1 --2- 172.21.15.195:0/2261991637 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f89040c4160 0x7f89040c1eb0 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:57.652 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.541+0000 7f890ab7e700 1 -- 172.21.15.195:0/2261991637 >> 172.21.15.195:0/2261991637 conn(0x7f89040c6350 msgr2=0x7f89040c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:11:57.653 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.541+0000 7f890ab7e700 1 -- 172.21.15.195:0/2261991637 shutdown_connections 2022-06-27T12:11:57.653 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.541+0000 7f890ab7e700 1 -- 172.21.15.195:0/2261991637 wait complete. 2022-06-27T12:11:57.654 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.542+0000 7f890ab7e700 1 Processor -- start 2022-06-27T12:11:57.656 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.542+0000 7f890ab7e700 1 -- start start 2022-06-27T12:11:57.657 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.543+0000 7f890ab7e700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f890406f640 0x7f89040ba330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:57.657 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.543+0000 7f890ab7e700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f89040c39e0 0x7f890406bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:57.657 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.543+0000 7f890ab7e700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f89040c4160 0x7f8904166a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:57.658 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.543+0000 7f890ab7e700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89040bba70 con 0x7f890406f640 2022-06-27T12:11:57.658 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.543+0000 7f890ab7e700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89040c8950 con 0x7f89040c39e0 2022-06-27T12:11:57.659 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.543+0000 7f890ab7e700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8904166600 con 0x7f89040c4160 2022-06-27T12:11:57.659 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.543+0000 7f890891a700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f890406f640 0x7f89040ba330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:57.660 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.543+0000 7f890891a700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f890406f640 0x7f89040ba330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50560/0 (socket says 172.21.15.195:50560) 2022-06-27T12:11:57.660 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.543+0000 7f890891a700 1 -- 172.21.15.195:0/2409052369 learned_addr learned my addr 172.21.15.195:0/2409052369 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:11:57.661 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.543+0000 7f8903fff700 1 --2- 172.21.15.195:0/2409052369 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f89040c39e0 0x7f890406bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:57.661 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.543+0000 7f890911b700 1 --2- 172.21.15.195:0/2409052369 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f89040c4160 0x7f8904166a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:57.662 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.544+0000 7f890891a700 1 -- 172.21.15.195:0/2409052369 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f89040c4160 msgr2=0x7f8904166a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:57.662 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.544+0000 7f890891a700 1 --2- 172.21.15.195:0/2409052369 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f89040c4160 0x7f8904166a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:57.665 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.544+0000 7f890891a700 1 -- 172.21.15.195:0/2409052369 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f89040c39e0 msgr2=0x7f890406bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:57.665 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.544+0000 7f890891a700 1 --2- 172.21.15.195:0/2409052369 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f89040c39e0 0x7f890406bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:57.666 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.544+0000 7f890891a700 1 -- 172.21.15.195:0/2409052369 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88f00092e0 con 0x7f890406f640 2022-06-27T12:11:57.667 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.544+0000 7f890891a700 1 --2- 172.21.15.195:0/2409052369 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f890406f640 0x7f89040ba330 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f88f800d890 tx=0x7f88f8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:57.667 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.544+0000 7f8900ff9700 1 -- 172.21.15.195:0/2409052369 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f8014620 con 0x7f890406f640 2022-06-27T12:11:57.668 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.544+0000 7f8900ff9700 1 -- 172.21.15.195:0/2409052369 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88f8014780 con 0x7f890406f640 2022-06-27T12:11:57.668 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.544+0000 7f8900ff9700 1 -- 172.21.15.195:0/2409052369 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f8014b40 con 0x7f890406f640 2022-06-27T12:11:57.668 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.544+0000 7f890ab7e700 1 -- 172.21.15.195:0/2409052369 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89040c9440 con 0x7f890406f640 2022-06-27T12:11:57.669 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.544+0000 7f890ab7e700 1 -- 172.21.15.195:0/2409052369 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89040c8e80 con 0x7f890406f640 2022-06-27T12:11:57.669 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.545+0000 7f890ab7e700 1 -- 172.21.15.195:0/2409052369 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89041668e0 con 0x7f890406f640 2022-06-27T12:11:57.670 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.545+0000 7f8900ff9700 1 -- 172.21.15.195:0/2409052369 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f88f8024070 con 0x7f890406f640 2022-06-27T12:11:57.670 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.546+0000 7f8900ff9700 1 --2- 172.21.15.195:0/2409052369 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f88ec063d30 0x7f88ec0661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:11:57.671 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.546+0000 7f8900ff9700 1 -- 172.21.15.195:0/2409052369 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7406+0+0 (secure 0 0 0) 0x7f88f80814c0 con 0x7f890406f640 2022-06-27T12:11:57.671 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.546+0000 7f8903fff700 1 --2- 172.21.15.195:0/2409052369 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f88ec063d30 0x7f88ec0661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:11:57.672 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.546+0000 7f8903fff700 1 --2- 172.21.15.195:0/2409052369 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f88ec063d30 0x7f88ec0661e0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f89041531a0 tx=0x7f88f0017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:11:57.672 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.550+0000 7f8900ff9700 1 -- 172.21.15.195:0/2409052369 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f88f804fb70 con 0x7f890406f640 2022-06-27T12:11:57.774 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.768+0000 7f890ab7e700 1 -- 172.21.15.195:0/2409052369 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8904154ba0 con 0x7f890406f640 2022-06-27T12:11:57.776 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.768+0000 7f8900ff9700 1 -- 172.21.15.195:0/2409052369 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+5368 (secure 0 0 0) 0x7f88f8053190 con 0x7f890406f640 2022-06-27T12:11:57.776 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:11:57.778 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":95,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:11:57.156947+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6050},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6050":{"gid":6050,"name":"b","rank":0,"incarnation":91,"state":"up:active","state_seq":15,"addr":"172.21.15.202:6833/3890799919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3890799919},{"type":"v1","addr":"172.21.15.202:6833","nonce":3890799919}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:11:57.796 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.770+0000 7f890ab7e700 1 -- 172.21.15.195:0/2409052369 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f88ec063d30 msgr2=0x7f88ec0661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:57.797 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.770+0000 7f890ab7e700 1 --2- 172.21.15.195:0/2409052369 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f88ec063d30 0x7f88ec0661e0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f89041531a0 tx=0x7f88f0017000 comp rx=0 tx=0).stop 2022-06-27T12:11:57.797 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.770+0000 7f890ab7e700 1 -- 172.21.15.195:0/2409052369 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f890406f640 msgr2=0x7f89040ba330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:11:57.798 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.770+0000 7f890ab7e700 1 --2- 172.21.15.195:0/2409052369 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f890406f640 0x7f89040ba330 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f88f800d890 tx=0x7f88f8008040 comp rx=0 tx=0).stop 2022-06-27T12:11:57.814 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.770+0000 7f890ab7e700 1 -- 172.21.15.195:0/2409052369 shutdown_connections 2022-06-27T12:11:57.819 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.770+0000 7f890ab7e700 1 --2- 172.21.15.195:0/2409052369 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f88ec063d30 0x7f88ec0661e0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:57.819 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.770+0000 7f890ab7e700 1 --2- 172.21.15.195:0/2409052369 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f890406f640 0x7f89040ba330 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:57.819 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.770+0000 7f890ab7e700 1 --2- 172.21.15.195:0/2409052369 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f89040c4160 0x7f8904166a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:57.820 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.770+0000 7f890ab7e700 1 --2- 172.21.15.195:0/2409052369 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f89040c39e0 0x7f890406bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:11:57.820 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.770+0000 7f890ab7e700 1 -- 172.21.15.195:0/2409052369 >> 172.21.15.195:0/2409052369 conn(0x7f89040c6350 msgr2=0x7f89041505e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:11:57.859 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.770+0000 7f890ab7e700 1 -- 172.21.15.195:0/2409052369 shutdown_connections 2022-06-27T12:11:57.860 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:11:57.770+0000 7f890ab7e700 1 -- 172.21.15.195:0/2409052369 wait complete. 2022-06-27T12:11:57.860 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 95 2022-06-27T12:11:57.861 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=5933 2022-06-27T12:11:57.867 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 6 secs before reviving mds.d 2022-06-27T12:12:02.272 INFO:tasks.ceph.mds.b.smithi202.stderr:2022-06-27T12:12:02.265+0000 7fe5d8be5700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2022-06-27T12:12:03.025 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:12:03.029 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:12:03.868 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.d 2022-06-27T12:12:03.869 INFO:tasks.ceph.mds.d:Restarting daemon 2022-06-27T12:12:03.870 DEBUG:teuthology.orchestra.run.smithi202:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2022-06-27T12:12:03.894 INFO:tasks.ceph.mds.d:Started 2022-06-27T12:12:03.895 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:12:03.991 INFO:tasks.ceph.mds.d.smithi202.stdout:starting mds.d at 2022-06-27T12:12:04.044 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.031+0000 7f0977d47700 1 Processor -- start 2022-06-27T12:12:04.045 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.031+0000 7f0977d47700 1 -- start start 2022-06-27T12:12:04.045 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.032+0000 7f0977d47700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f09700c6590 0x7f097015b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:04.045 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.032+0000 7f0977d47700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0970067ed0 con 0x7f0970069cf0 2022-06-27T12:12:04.055 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.032+0000 7f0977d47700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0970068120 con 0x7f097006a0c0 2022-06-27T12:12:04.056 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.032+0000 7f0977d47700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09700c48a0 con 0x7f09700c6590 2022-06-27T12:12:04.057 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.032+0000 7f0975ae3700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f0970069cf0 0x7f09700c4190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:56758/0 (socket says 172.21.15.195:56758) 2022-06-27T12:12:04.092 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.032+0000 7f0975ae3700 1 -- 172.21.15.195:0/4265139742 learned_addr learned my addr 172.21.15.195:0/4265139742 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:12:04.093 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.032+0000 7f09752e2700 1 --2- 172.21.15.195:0/4265139742 >> v2:172.21.15.202:3300/0 conn(0x7f09700c6590 0x7f097015b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:04.097 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.032+0000 7f0974ae1700 1 -- 172.21.15.195:0/4265139742 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2911000889 0 0) 0x7f0970067ed0 con 0x7f0970069cf0 2022-06-27T12:12:04.098 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.032+0000 7f0974ae1700 1 -- 172.21.15.195:0/4265139742 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0958003430 con 0x7f0970069cf0 2022-06-27T12:12:04.099 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.033+0000 7f0974ae1700 1 -- 172.21.15.195:0/4265139742 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1666046583 0 0) 0x7f0958003430 con 0x7f0970069cf0 2022-06-27T12:12:04.099 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.033+0000 7f0974ae1700 1 -- 172.21.15.195:0/4265139742 >> v2:172.21.15.202:3300/0 conn(0x7f09700c6590 msgr2=0x7f097015b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:12:04.100 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.033+0000 7f0974ae1700 1 --2- 172.21.15.195:0/4265139742 >> v2:172.21.15.202:3300/0 conn(0x7f09700c6590 0x7f097015b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:04.100 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.033+0000 7f0974ae1700 1 -- 172.21.15.195:0/4265139742 >> v1:172.21.15.202:6789/0 conn(0x7f097006a0c0 legacy=0x7f0970154650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:04.119 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.033+0000 7f0974ae1700 1 -- 172.21.15.195:0/4265139742 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0970068400 con 0x7f0970069cf0 2022-06-27T12:12:04.120 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.033+0000 7f0974ae1700 1 -- 172.21.15.195:0/4265139742 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7f09600015f0 con 0x7f0970069cf0 2022-06-27T12:12:04.120 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.033+0000 7f09752e2700 1 --2- 172.21.15.195:0/4265139742 >> v2:172.21.15.202:3300/0 conn(0x7f09700c6590 0x7f097015b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:12:04.121 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.033+0000 7f0974ae1700 1 -- 172.21.15.195:0/4265139742 >> v1:172.21.15.195:6789/0 conn(0x7f0970069cf0 legacy=0x7f09700c4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:04.121 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.033+0000 7f0974ae1700 1 --2- 172.21.15.195:0/4265139742 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0958003920 0x7f0958003d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:04.122 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.033+0000 7f0974ae1700 1 --2- 172.21.15.195:0/4265139742 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f09580046d0 0x7f0958004b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:04.122 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.033+0000 7f0974ae1700 1 --2- 172.21.15.195:0/4265139742 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0958005080 0x7f095800b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:04.123 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.033+0000 7f0974ae1700 1 -- 172.21.15.195:0/4265139742 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09700c48a0 con 0x7f09580046d0 2022-06-27T12:12:04.123 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.033+0000 7f0974ae1700 1 -- 172.21.15.195:0/4265139742 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f095800bf90 con 0x7f0958003920 2022-06-27T12:12:04.124 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.033+0000 7f0974ae1700 1 -- 172.21.15.195:0/4265139742 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f095800c150 con 0x7f0958005080 2022-06-27T12:12:04.124 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.033+0000 7f09752e2700 1 --2- 172.21.15.195:0/4265139742 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f09580046d0 0x7f0958004b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:04.125 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.033+0000 7f0975ae3700 1 --2- 172.21.15.195:0/4265139742 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0958003920 0x7f0958003d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:04.125 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.033+0000 7f09762e4700 1 --2- 172.21.15.195:0/4265139742 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0958005080 0x7f095800b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:04.125 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.033+0000 7f09752e2700 1 -- 172.21.15.195:0/4265139742 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0958005080 msgr2=0x7f095800b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:12:04.126 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.033+0000 7f09752e2700 1 --2- 172.21.15.195:0/4265139742 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0958005080 0x7f095800b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:04.126 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.033+0000 7f09752e2700 1 -- 172.21.15.195:0/4265139742 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0958003920 msgr2=0x7f0958003d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:12:04.127 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.033+0000 7f09752e2700 1 --2- 172.21.15.195:0/4265139742 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0958003920 0x7f0958003d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:04.127 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.033+0000 7f09762e4700 1 -- 172.21.15.195:0/4265139742 reap_dead start 2022-06-27T12:12:04.128 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.033+0000 7f09752e2700 1 -- 172.21.15.195:0/4265139742 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0958003430 con 0x7f09580046d0 2022-06-27T12:12:04.129 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.033+0000 7f09752e2700 1 --2- 172.21.15.195:0/4265139742 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f09580046d0 0x7f0958004b40 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f096c009cb0 tx=0x7f096c009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3008a6cfcfdd5cb2 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:12:04.129 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.033+0000 7f0974ae1700 1 -- 172.21.15.195:0/4265139742 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f096c009600 con 0x7f09580046d0 2022-06-27T12:12:04.130 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.033+0000 7f0974ae1700 1 -- 172.21.15.195:0/4265139742 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f096c00e770 con 0x7f09580046d0 2022-06-27T12:12:04.130 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.033+0000 7f0974ae1700 1 -- 172.21.15.195:0/4265139742 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f096c00eb10 con 0x7f09580046d0 2022-06-27T12:12:04.130 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.034+0000 7f0977d47700 1 -- 172.21.15.195:0/4265139742 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f09580046d0 msgr2=0x7f0958004b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:04.131 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.034+0000 7f0977d47700 1 --2- 172.21.15.195:0/4265139742 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f09580046d0 0x7f0958004b40 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f096c009cb0 tx=0x7f096c009060 comp rx=0 tx=0).stop 2022-06-27T12:12:04.132 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.034+0000 7f0977d47700 1 -- 172.21.15.195:0/4265139742 shutdown_connections 2022-06-27T12:12:04.132 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.034+0000 7f0977d47700 1 --2- 172.21.15.195:0/4265139742 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f09580046d0 0x7f0958004b40 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:04.132 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.034+0000 7f0977d47700 1 -- 172.21.15.195:0/4265139742 >> 172.21.15.195:0/4265139742 conn(0x7f09700c9b00 msgr2=0x7f09700c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:12:04.133 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.034+0000 7f0977d47700 1 -- 172.21.15.195:0/4265139742 shutdown_connections 2022-06-27T12:12:04.133 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.034+0000 7f0977d47700 1 -- 172.21.15.195:0/4265139742 wait complete. 2022-06-27T12:12:04.134 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.035+0000 7f0977d47700 1 Processor -- start 2022-06-27T12:12:04.136 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.035+0000 7f0977d47700 1 -- start start 2022-06-27T12:12:04.142 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.036+0000 7f0977d47700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f09580046d0 0x7f097015a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:04.143 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.036+0000 7f0977d47700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f097015b480 0x7f097015aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:04.143 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.036+0000 7f0977d47700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f097016e2a0 0x7f097016e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:04.146 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.036+0000 7f0977d47700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09700c2e80 con 0x7f097015b480 2022-06-27T12:12:04.146 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.036+0000 7f0977d47700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f097015a500 con 0x7f097016e2a0 2022-06-27T12:12:04.146 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.036+0000 7f0977d47700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f097016ec50 con 0x7f09580046d0 2022-06-27T12:12:04.147 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.036+0000 7f09752e2700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f097015b480 0x7f097015aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:04.147 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.036+0000 7f09752e2700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f097015b480 0x7f097015aed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50584/0 (socket says 172.21.15.195:50584) 2022-06-27T12:12:04.148 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.036+0000 7f09752e2700 1 -- 172.21.15.195:0/4114381782 learned_addr learned my addr 172.21.15.195:0/4114381782 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:12:04.148 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.036+0000 7f0975ae3700 1 --2- 172.21.15.195:0/4114381782 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f09580046d0 0x7f097015a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:04.149 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.036+0000 7f09762e4700 1 --2- 172.21.15.195:0/4114381782 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f097016e2a0 0x7f097016e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:04.149 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.036+0000 7f09752e2700 1 -- 172.21.15.195:0/4114381782 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f09580046d0 msgr2=0x7f097015a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:04.149 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.036+0000 7f09752e2700 1 --2- 172.21.15.195:0/4114381782 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f09580046d0 0x7f097015a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:04.162 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.036+0000 7f09752e2700 1 -- 172.21.15.195:0/4114381782 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f097016e2a0 msgr2=0x7f097016e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:04.163 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.036+0000 7f09752e2700 1 --2- 172.21.15.195:0/4114381782 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f097016e2a0 0x7f097016e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:04.163 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.036+0000 7f09752e2700 1 -- 172.21.15.195:0/4114381782 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0970153b10 con 0x7f097015b480 2022-06-27T12:12:04.163 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.036+0000 7f09752e2700 1 --2- 172.21.15.195:0/4114381782 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f097015b480 0x7f097015aed0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f096c00e4b0 tx=0x7f096c01a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:12:04.164 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.036+0000 7f095dffb700 1 -- 172.21.15.195:0/4114381782 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f096c021070 con 0x7f097015b480 2022-06-27T12:12:04.164 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.036+0000 7f095dffb700 1 -- 172.21.15.195:0/4114381782 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f096c009600 con 0x7f097015b480 2022-06-27T12:12:04.164 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.036+0000 7f0977d47700 1 -- 172.21.15.195:0/4114381782 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f097016eea0 con 0x7f097015b480 2022-06-27T12:12:04.164 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.037+0000 7f095dffb700 1 -- 172.21.15.195:0/4114381782 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f096c0039a0 con 0x7f097015b480 2022-06-27T12:12:04.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.037+0000 7f0977d47700 1 -- 172.21.15.195:0/4114381782 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0970159bd0 con 0x7f097015b480 2022-06-27T12:12:04.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.038+0000 7f0977d47700 1 -- 172.21.15.195:0/4114381782 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f097015a6b0 con 0x7f097015b480 2022-06-27T12:12:04.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.038+0000 7f095dffb700 1 -- 172.21.15.195:0/4114381782 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f096c02d030 con 0x7f097015b480 2022-06-27T12:12:04.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.038+0000 7f095dffb700 1 --2- 172.21.15.195:0/4114381782 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0950063d00 0x7f09500661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:04.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.038+0000 7f095dffb700 1 -- 172.21.15.195:0/4114381782 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7406+0+0 (secure 0 0 0) 0x7f096c082640 con 0x7f097015b480 2022-06-27T12:12:04.166 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.038+0000 7f0975ae3700 1 --2- 172.21.15.195:0/4114381782 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0950063d00 0x7f09500661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:04.166 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.039+0000 7f0975ae3700 1 --2- 172.21.15.195:0/4114381782 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0950063d00 0x7f09500661b0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f09600011d0 tx=0x7f0960009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:12:04.167 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.043+0000 7f095dffb700 1 -- 172.21.15.195:0/4114381782 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f097015a6b0 con 0x7f097015b480 2022-06-27T12:12:04.279 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.270+0000 7f0977d47700 1 -- 172.21.15.195:0/4114381782 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f097015a6b0 con 0x7f097015b480 2022-06-27T12:12:04.280 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.270+0000 7f095dffb700 1 -- 172.21.15.195:0/4114381782 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+6128 (secure 0 0 0) 0x7f097015a6b0 con 0x7f097015b480 2022-06-27T12:12:04.280 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:12:04.281 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":96,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":6865,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/2349973547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2349973547},{"type":"v1","addr":"172.21.15.202:6835","nonce":2349973547}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:11:57.156947+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6050},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6050":{"gid":6050,"name":"b","rank":0,"incarnation":91,"state":"up:active","state_seq":15,"addr":"172.21.15.202:6833/3890799919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3890799919},{"type":"v1","addr":"172.21.15.202:6833","nonce":3890799919}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:12:04.306 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.272+0000 7f0977d47700 1 -- 172.21.15.195:0/4114381782 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0950063d00 msgr2=0x7f09500661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:04.307 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.272+0000 7f0977d47700 1 --2- 172.21.15.195:0/4114381782 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0950063d00 0x7f09500661b0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f09600011d0 tx=0x7f0960009040 comp rx=0 tx=0).stop 2022-06-27T12:12:04.307 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.272+0000 7f0977d47700 1 -- 172.21.15.195:0/4114381782 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f097015b480 msgr2=0x7f097015aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:04.307 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.272+0000 7f0977d47700 1 --2- 172.21.15.195:0/4114381782 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f097015b480 0x7f097015aed0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f096c00e4b0 tx=0x7f096c01a000 comp rx=0 tx=0).stop 2022-06-27T12:12:04.307 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.272+0000 7f0977d47700 1 -- 172.21.15.195:0/4114381782 shutdown_connections 2022-06-27T12:12:04.308 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.272+0000 7f0977d47700 1 --2- 172.21.15.195:0/4114381782 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0950063d00 0x7f09500661b0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:04.311 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.272+0000 7f0977d47700 1 --2- 172.21.15.195:0/4114381782 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f097016e2a0 0x7f097016e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:04.311 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.272+0000 7f0977d47700 1 --2- 172.21.15.195:0/4114381782 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f09580046d0 0x7f097015a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:04.312 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.272+0000 7f0977d47700 1 --2- 172.21.15.195:0/4114381782 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f097015b480 0x7f097015aed0 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:04.312 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.272+0000 7f0977d47700 1 -- 172.21.15.195:0/4114381782 >> 172.21.15.195:0/4114381782 conn(0x7f09700c9b00 msgr2=0x7f09701502d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:12:04.313 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.272+0000 7f0977d47700 1 -- 172.21.15.195:0/4114381782 shutdown_connections 2022-06-27T12:12:04.313 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:04.272+0000 7f0977d47700 1 -- 172.21.15.195:0/4114381782 wait complete. 2022-06-27T12:12:04.314 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 96 2022-06-27T12:12:04.315 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d reported in up:standby state 2022-06-27T12:12:04.316 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 17 secs before thrashing 2022-06-27T12:12:10.701 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 31% (57/181) Updating files: 32% (58/181) Updating files: 33% (60/181) Updating files: 34% (62/181) Updating files: 35% (64/181) Updating files: 36% (66/181) Updating files: 37% (67/181) Updating files: 38% (69/181) Updating files: 39% (71/181) Updating files: 40% (73/181) Updating files: 41% (75/181) Updating files: 42% (77/181) Updating files: 43% (78/181) Updating files: 44% (80/181) Updating files: 45% (82/181) Updating files: 46% (84/181) Updating files: 47% (86/181) Updating files: 48% (87/181) Updating files: 49% (89/181) Updating files: 50% (91/181) Updating files: 51% (93/181) Updating files: 52% (95/181) Updating files: 53% (96/181) Updating files: 54% (98/181) Updating files: 55% (100/181) Updating files: 55% (101/181) Updating files: 56% (102/181) Updating files: 57% (104/181) Updating files: 58% (105/181) Updating files: 59% (107/181) Updating files: 60% (109/181) Updating files: 61% (111/181) Updating files: 62% (113/181) Updating files: 63% (115/181) Updating files: 64% (116/181) Updating files: 65% (118/181) Updating files: 66% (120/181) Updating files: 67% (122/181) Updating files: 68% (124/181) Updating files: 69% (125/181) Updating files: 70% (127/181) Updating files: 71% (129/181) Updating files: 72% (131/181) Updating files: 73% (133/181) Updating files: 74% (134/181) Updating files: 75% (136/181) Updating files: 76% (138/181) Updating files: 77% (140/181) Updating files: 78% (142/181) Updating files: 79% (143/181) Updating files: 80% (145/181) Updating files: 80% (146/181) Updating files: 81% (147/181) Updating files: 82% (149/181) Updating files: 83% (151/181) Updating files: 84% (153/181) Updating files: 85% (154/181) Updating files: 86% (156/181) Updating files: 87% (158/181) Updating files: 88% (160/181) Updating files: 89% (162/181) Updating files: 90% (163/181) Updating files: 91% (165/181) Updating files: 92% (167/181) Updating files: 93% (169/181) Updating files: 94% (171/181) Updating files: 95% (172/181) Updating files: 96% (174/181) Updating files: 97% (176/181) Updating files: 98% (178/181) Updating files: 99% (180/181) Updating files: 100% (181/181) Updating files: 100% (181/181), done. 2022-06-27T12:12:10.800 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at a6f3bbb744a v0.37 2022-06-27T12:12:10.804 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.37 2022-06-27T12:12:10.866 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:12:10.867 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 38 -eq 48 ']' 2022-06-27T12:12:10.867 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.38 2022-06-27T12:12:10.868 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.38 2022-06-27T12:12:10.868 INFO:tasks.workunit.client.0.smithi195.stdout:v0.38 2022-06-27T12:12:10.869 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.38 2022-06-27T12:12:16.446 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 46% (80/171) Updating files: 47% (81/171) Updating files: 48% (83/171) Updating files: 49% (84/171) Updating files: 50% (86/171) Updating files: 51% (88/171) Updating files: 52% (89/171) Updating files: 53% (91/171) Updating files: 54% (93/171) Updating files: 55% (95/171) Updating files: 56% (96/171) Updating files: 57% (98/171) Updating files: 58% (100/171) Updating files: 59% (101/171) Updating files: 60% (103/171) Updating files: 61% (105/171) Updating files: 61% (106/171) Updating files: 62% (107/171) Updating files: 63% (108/171) Updating files: 64% (110/171) Updating files: 65% (112/171) Updating files: 66% (113/171) Updating files: 67% (115/171) Updating files: 68% (117/171) Updating files: 69% (118/171) Updating files: 70% (120/171) Updating files: 71% (122/171) Updating files: 72% (124/171) Updating files: 73% (125/171) Updating files: 74% (127/171) Updating files: 75% (129/171) Updating files: 76% (130/171) Updating files: 77% (132/171) Updating files: 78% (134/171) Updating files: 79% (136/171) Updating files: 80% (137/171) Updating files: 81% (139/171) Updating files: 82% (141/171) Updating files: 83% (142/171) Updating files: 84% (144/171) Updating files: 85% (146/171) Updating files: 86% (148/171) Updating files: 87% (149/171) Updating files: 88% (151/171) Updating files: 89% (153/171) Updating files: 90% (154/171) Updating files: 91% (156/171) Updating files: 92% (158/171) Updating files: 93% (160/171) Updating files: 94% (161/171) Updating files: 95% (163/171) Updating files: 96% (165/171) Updating files: 97% (166/171) Updating files: 98% (168/171) Updating files: 99% (170/171) Updating files: 100% (171/171) Updating files: 100% (171/171), done. 2022-06-27T12:12:16.473 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at b600ec2ac7c v0.38 2022-06-27T12:12:16.474 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.38 2022-06-27T12:12:16.570 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:12:16.571 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 39 -eq 48 ']' 2022-06-27T12:12:16.571 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.39 2022-06-27T12:12:16.572 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.39 2022-06-27T12:12:16.573 INFO:tasks.workunit.client.0.smithi195.stdout:v0.39 2022-06-27T12:12:16.584 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.39 2022-06-27T12:12:21.318 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:12:21.457 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.442+0000 7f52ad321700 1 Processor -- start 2022-06-27T12:12:21.458 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.443+0000 7f52ad321700 1 -- start start 2022-06-27T12:12:21.458 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.443+0000 7f52ad321700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f52a80c5480 0x7f52a8069a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:21.458 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.443+0000 7f52ad321700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f52a80c64f0 0x7f52a81504c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:21.459 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.443+0000 7f52ad321700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52a80c5d80 con 0x7f52a806d280 2022-06-27T12:12:21.459 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.443+0000 7f52ad321700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52a8069f50 con 0x7f52a80c5480 2022-06-27T12:12:21.459 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.443+0000 7f52ad321700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52a806b010 con 0x7f52a80c64f0 2022-06-27T12:12:21.460 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.444+0000 7f52a6ffd700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f52a80c5480 0x7f52a8069a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:21.460 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.444+0000 7f52a6ffd700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f52a80c5480 0x7f52a8069a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50626/0 (socket says 172.21.15.195:50626) 2022-06-27T12:12:21.461 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.444+0000 7f52a6ffd700 1 -- 172.21.15.195:0/2134238483 learned_addr learned my addr 172.21.15.195:0/2134238483 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:12:21.461 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.444+0000 7f52a67fc700 1 --2- 172.21.15.195:0/2134238483 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f52a80c64f0 0x7f52a81504c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:21.462 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.444+0000 7f52a6ffd700 1 -- 172.21.15.195:0/2134238483 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f52a80c64f0 msgr2=0x7f52a81504c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:12:21.476 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.444+0000 7f52a6ffd700 1 --2- 172.21.15.195:0/2134238483 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f52a80c64f0 0x7f52a81504c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:21.476 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.444+0000 7f52a6ffd700 1 -- 172.21.15.195:0/2134238483 >> v1:172.21.15.202:6789/0 conn(0x7f52a806d280 legacy=0x7f52a8150a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:21.476 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.444+0000 7f52a6ffd700 1 -- 172.21.15.195:0/2134238483 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52a80c75e0 con 0x7f52a80c5480 2022-06-27T12:12:21.477 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.444+0000 7f52a6ffd700 1 --2- 172.21.15.195:0/2134238483 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f52a80c5480 0x7f52a8069a10 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f529c009860 tx=0x7f529c009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1dd5be67006ebe3 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:12:21.477 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.444+0000 7f52a5ffb700 1 -- 172.21.15.195:0/2134238483 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f529c011630 con 0x7f52a80c5480 2022-06-27T12:12:21.477 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.444+0000 7f52a5ffb700 1 -- 172.21.15.195:0/2134238483 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f529c011790 con 0x7f52a80c5480 2022-06-27T12:12:21.477 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.444+0000 7f52a5ffb700 1 -- 172.21.15.195:0/2134238483 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f529c011b30 con 0x7f52a80c5480 2022-06-27T12:12:21.478 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.445+0000 7f52ad321700 1 -- 172.21.15.195:0/2134238483 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f52a80c5480 msgr2=0x7f52a8069a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:21.478 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.445+0000 7f52ad321700 1 --2- 172.21.15.195:0/2134238483 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f52a80c5480 0x7f52a8069a10 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f529c009860 tx=0x7f529c009120 comp rx=0 tx=0).stop 2022-06-27T12:12:21.478 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.445+0000 7f52ad321700 1 -- 172.21.15.195:0/2134238483 shutdown_connections 2022-06-27T12:12:21.478 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.445+0000 7f52ad321700 1 --2- 172.21.15.195:0/2134238483 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f52a80c5480 0x7f52a8069a10 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:21.479 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.445+0000 7f52ad321700 1 --2- 172.21.15.195:0/2134238483 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f52a80c64f0 0x7f52a81504c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:21.479 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.445+0000 7f52ad321700 1 -- 172.21.15.195:0/2134238483 >> 172.21.15.195:0/2134238483 conn(0x7f52a8063af0 msgr2=0x7f52a8063ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:12:21.479 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.445+0000 7f52ad321700 1 -- 172.21.15.195:0/2134238483 shutdown_connections 2022-06-27T12:12:21.479 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.445+0000 7f52ad321700 1 -- 172.21.15.195:0/2134238483 wait complete. 2022-06-27T12:12:21.479 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.446+0000 7f52ad321700 1 Processor -- start 2022-06-27T12:12:21.480 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.446+0000 7f52ad321700 1 -- start start 2022-06-27T12:12:21.480 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.447+0000 7f52ad321700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f52a806d280 0x7f52a8148db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:21.480 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.447+0000 7f52ad321700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f52a80c5480 0x7f52a80bf3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:21.480 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.447+0000 7f52ad321700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f52a80c64f0 0x7f52a80bf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:21.481 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.447+0000 7f52ad321700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a80bef20 con 0x7f52a80c64f0 2022-06-27T12:12:21.481 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.447+0000 7f52ad321700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a80c0e60 con 0x7f52a80c5480 2022-06-27T12:12:21.481 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.447+0000 7f52ad321700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52a80c1140 con 0x7f52a806d280 2022-06-27T12:12:21.481 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.447+0000 7f52a6ffd700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f52a806d280 0x7f52a8148db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:21.482 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.447+0000 7f52a67fc700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f52a80c5480 0x7f52a80bf3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:21.482 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.447+0000 7f52a77fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f52a80c64f0 0x7f52a80bf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:21.482 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.447+0000 7f52a6ffd700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f52a806d280 0x7f52a8148db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3301/0 says I am v2:172.21.15.195:43386/0 (socket says 172.21.15.195:43386) 2022-06-27T12:12:21.482 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.447+0000 7f52a67fc700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f52a80c5480 0x7f52a80bf3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3300/0 says I am v2:172.21.15.195:48758/0 (socket says 172.21.15.195:48758) 2022-06-27T12:12:21.483 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.447+0000 7f52a6ffd700 1 -- 172.21.15.195:0/3945524939 learned_addr learned my addr 172.21.15.195:0/3945524939 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:12:21.483 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.447+0000 7f52a77fe700 1 -- 172.21.15.195:0/3945524939 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f52a806d280 msgr2=0x7f52a8148db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:21.483 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.447+0000 7f52a77fe700 1 --2- 172.21.15.195:0/3945524939 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f52a806d280 0x7f52a8148db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:21.483 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.447+0000 7f52a77fe700 1 -- 172.21.15.195:0/3945524939 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f52a80c5480 msgr2=0x7f52a80bf3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:21.483 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.447+0000 7f52a77fe700 1 --2- 172.21.15.195:0/3945524939 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f52a80c5480 0x7f52a80bf3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:21.485 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.447+0000 7f52a77fe700 1 -- 172.21.15.195:0/3945524939 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f529c0093a0 con 0x7f52a80c64f0 2022-06-27T12:12:21.485 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.447+0000 7f52a6ffd700 1 --2- 172.21.15.195:0/3945524939 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f52a806d280 0x7f52a8148db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:12:21.485 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.448+0000 7f52a77fe700 1 --2- 172.21.15.195:0/3945524939 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f52a80c64f0 0x7f52a80bf8f0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f52a000c6e0 tx=0x7f52a0001560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:12:21.486 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.448+0000 7f528effd700 1 -- 172.21.15.195:0/3945524939 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a0003880 con 0x7f52a80c64f0 2022-06-27T12:12:21.486 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.448+0000 7f528effd700 1 -- 172.21.15.195:0/3945524939 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52a00039e0 con 0x7f52a80c64f0 2022-06-27T12:12:21.487 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.448+0000 7f52ad321700 1 -- 172.21.15.195:0/3945524939 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52a80c1420 con 0x7f52a80c64f0 2022-06-27T12:12:21.487 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.448+0000 7f528effd700 1 -- 172.21.15.195:0/3945524939 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a0003d60 con 0x7f52a80c64f0 2022-06-27T12:12:21.487 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.448+0000 7f52ad321700 1 -- 172.21.15.195:0/3945524939 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52a80bebb0 con 0x7f52a80c64f0 2022-06-27T12:12:21.487 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.449+0000 7f52ad321700 1 -- 172.21.15.195:0/3945524939 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52a80bf0d0 con 0x7f52a80c64f0 2022-06-27T12:12:21.488 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.449+0000 7f528effd700 1 -- 172.21.15.195:0/3945524939 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f52a0019030 con 0x7f52a80c64f0 2022-06-27T12:12:21.488 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.449+0000 7f528effd700 1 --2- 172.21.15.195:0/3945524939 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5288063d00 0x7f52880661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:21.488 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.449+0000 7f528effd700 1 -- 172.21.15.195:0/3945524939 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7406+0+0 (secure 0 0 0) 0x7f52a007fff0 con 0x7f52a80c64f0 2022-06-27T12:12:21.488 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.450+0000 7f52a6ffd700 1 --2- 172.21.15.195:0/3945524939 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5288063d00 0x7f52880661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:21.488 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.450+0000 7f52a6ffd700 1 --2- 172.21.15.195:0/3945524939 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5288063d00 0x7f52880661b0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f529c011fd0 tx=0x7f529c017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:12:21.489 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.454+0000 7f528effd700 1 -- 172.21.15.195:0/3945524939 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f52a0049c90 con 0x7f52a80c64f0 2022-06-27T12:12:21.706 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.701+0000 7f52ad321700 1 -- 172.21.15.195:0/3945524939 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f52a80722c0 con 0x7f52a80c64f0 2022-06-27T12:12:21.707 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.701+0000 7f528effd700 1 -- 172.21.15.195:0/3945524939 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+6128 (secure 0 0 0) 0x7f52a0017020 con 0x7f52a80c64f0 2022-06-27T12:12:21.709 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:12:21.710 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":96,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":6865,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/2349973547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2349973547},{"type":"v1","addr":"172.21.15.202:6835","nonce":2349973547}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:11:57.156947+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6050},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6050":{"gid":6050,"name":"b","rank":0,"incarnation":91,"state":"up:active","state_seq":15,"addr":"172.21.15.202:6833/3890799919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3890799919},{"type":"v1","addr":"172.21.15.202:6833","nonce":3890799919}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:12:21.718 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.704+0000 7f52ad321700 1 -- 172.21.15.195:0/3945524939 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5288063d00 msgr2=0x7f52880661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:21.719 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.704+0000 7f52ad321700 1 --2- 172.21.15.195:0/3945524939 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5288063d00 0x7f52880661b0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f529c011fd0 tx=0x7f529c017000 comp rx=0 tx=0).stop 2022-06-27T12:12:21.719 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.704+0000 7f52ad321700 1 -- 172.21.15.195:0/3945524939 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f52a80c64f0 msgr2=0x7f52a80bf8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:21.720 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.704+0000 7f52ad321700 1 --2- 172.21.15.195:0/3945524939 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f52a80c64f0 0x7f52a80bf8f0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f52a000c6e0 tx=0x7f52a0001560 comp rx=0 tx=0).stop 2022-06-27T12:12:21.720 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.704+0000 7f52ad321700 1 -- 172.21.15.195:0/3945524939 shutdown_connections 2022-06-27T12:12:21.721 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.704+0000 7f52ad321700 1 --2- 172.21.15.195:0/3945524939 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5288063d00 0x7f52880661b0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:21.721 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.704+0000 7f52ad321700 1 --2- 172.21.15.195:0/3945524939 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f52a80c64f0 0x7f52a80bf8f0 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:21.722 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.704+0000 7f52ad321700 1 --2- 172.21.15.195:0/3945524939 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f52a806d280 0x7f52a8148db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:21.735 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.704+0000 7f52ad321700 1 --2- 172.21.15.195:0/3945524939 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f52a80c5480 0x7f52a80bf3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:21.742 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.704+0000 7f52ad321700 1 -- 172.21.15.195:0/3945524939 >> 172.21.15.195:0/3945524939 conn(0x7f52a8063af0 msgr2=0x7f52a80c8800 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:12:21.744 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.704+0000 7f52ad321700 1 -- 172.21.15.195:0/3945524939 shutdown_connections 2022-06-27T12:12:21.745 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.704+0000 7f52ad321700 1 -- 172.21.15.195:0/3945524939 wait complete. 2022-06-27T12:12:21.746 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 96 2022-06-27T12:12:21.751 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2022-06-27T12:12:21.752 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-06-27T12:12:21.753 INFO:teuthology.orchestra.run:waiting for 300 2022-06-27T12:12:21.766 INFO:tasks.ceph.mds.b:Stopped 2022-06-27T12:12:21.767 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:12:21.943 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.898+0000 7f1105b67700 1 Processor -- start 2022-06-27T12:12:21.943 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.898+0000 7f1105b67700 1 -- start start 2022-06-27T12:12:21.943 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.899+0000 7f1105b67700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f110006fef0 0x7f11001570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:21.943 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.899+0000 7f1105b67700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f11000c34a0 0x7f11000c2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:21.944 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.899+0000 7f1105b67700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f110006fba0 con 0x7f110006e850 2022-06-27T12:12:21.944 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.899+0000 7f1105b67700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11001575e0 con 0x7f110006fef0 2022-06-27T12:12:21.944 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.899+0000 7f1105b67700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f110006f5e0 con 0x7f11000c34a0 2022-06-27T12:12:21.944 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.899+0000 7f10feffd700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f110006fef0 0x7f11001570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:21.945 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.899+0000 7f10ff7fe700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f110006e850 0x7f1100069de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:56830/0 (socket says 172.21.15.195:56830) 2022-06-27T12:12:21.945 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.899+0000 7f10feffd700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f110006fef0 0x7f11001570a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50644/0 (socket says 172.21.15.195:50644) 2022-06-27T12:12:21.945 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.899+0000 7f10ff7fe700 1 -- 172.21.15.195:0/1042792809 learned_addr learned my addr 172.21.15.195:0/1042792809 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:12:21.945 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.899+0000 7f10fffff700 1 --2- 172.21.15.195:0/1042792809 >> v2:172.21.15.202:3300/0 conn(0x7f11000c34a0 0x7f11000c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:21.945 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.899+0000 7f10feffd700 1 -- 172.21.15.195:0/1042792809 >> v2:172.21.15.202:3300/0 conn(0x7f11000c34a0 msgr2=0x7f11000c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:12:21.964 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.899+0000 7f10feffd700 1 --2- 172.21.15.195:0/1042792809 >> v2:172.21.15.202:3300/0 conn(0x7f11000c34a0 0x7f11000c2920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:21.965 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.899+0000 7f10feffd700 1 -- 172.21.15.195:0/1042792809 >> v1:172.21.15.195:6789/0 conn(0x7f110006e850 legacy=0x7f1100069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:21.965 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.899+0000 7f10feffd700 1 -- 172.21.15.195:0/1042792809 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f110006f8c0 con 0x7f110006fef0 2022-06-27T12:12:21.965 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.899+0000 7f10fe7fc700 1 -- 172.21.15.195:0/1042792809 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3349902070 0 0) 0x7f110006fba0 con 0x7f110006e850 2022-06-27T12:12:21.965 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.900+0000 7f10feffd700 1 --2- 172.21.15.195:0/1042792809 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f110006fef0 0x7f11001570a0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f11000c1420 tx=0x7f10f0009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2b071f4115424b9 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:12:21.966 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.900+0000 7f10fe7fc700 1 -- 172.21.15.195:0/1042792809 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f0011630 con 0x7f110006fef0 2022-06-27T12:12:21.966 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.900+0000 7f10fe7fc700 1 -- 172.21.15.195:0/1042792809 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10f0011790 con 0x7f110006fef0 2022-06-27T12:12:21.966 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.900+0000 7f10fe7fc700 1 -- 172.21.15.195:0/1042792809 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f0011b30 con 0x7f110006fef0 2022-06-27T12:12:21.966 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.900+0000 7f1105b67700 1 -- 172.21.15.195:0/1042792809 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f110006fef0 msgr2=0x7f11001570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:21.967 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.900+0000 7f1105b67700 1 --2- 172.21.15.195:0/1042792809 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f110006fef0 0x7f11001570a0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f11000c1420 tx=0x7f10f0009060 comp rx=0 tx=0).stop 2022-06-27T12:12:21.967 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.900+0000 7f1105b67700 1 -- 172.21.15.195:0/1042792809 shutdown_connections 2022-06-27T12:12:21.967 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.900+0000 7f1105b67700 1 --2- 172.21.15.195:0/1042792809 >> v2:172.21.15.202:3300/0 conn(0x7f11000c34a0 0x7f11000c2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:21.967 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.900+0000 7f1105b67700 1 --2- 172.21.15.195:0/1042792809 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f110006fef0 0x7f11001570a0 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:21.968 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.900+0000 7f1105b67700 1 -- 172.21.15.195:0/1042792809 >> 172.21.15.195:0/1042792809 conn(0x7f11000c6350 msgr2=0x7f11000c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:12:21.968 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.900+0000 7f1105b67700 1 -- 172.21.15.195:0/1042792809 shutdown_connections 2022-06-27T12:12:21.968 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.900+0000 7f1105b67700 1 -- 172.21.15.195:0/1042792809 wait complete. 2022-06-27T12:12:21.968 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.901+0000 7f1105b67700 1 Processor -- start 2022-06-27T12:12:21.969 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.902+0000 7f1105b67700 1 -- start start 2022-06-27T12:12:21.969 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.902+0000 7f1105b67700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f110006e850 0x7f11000bc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:21.969 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.902+0000 7f1105b67700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f11000c34a0 0x7f11000c6900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:21.969 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.902+0000 7f1105b67700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f110014e210 0x7f110014e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:21.969 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.902+0000 7f1105b67700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1100167120 con 0x7f110006e850 2022-06-27T12:12:21.970 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.902+0000 7f1105b67700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f110014ebc0 con 0x7f110014e210 2022-06-27T12:12:21.970 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.902+0000 7f1105b67700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1100155ac0 con 0x7f11000c34a0 2022-06-27T12:12:21.970 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.902+0000 7f10ff7fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f110006e850 0x7f11000bc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:21.971 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.902+0000 7f10ff7fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f110006e850 0x7f11000bc5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50648/0 (socket says 172.21.15.195:50648) 2022-06-27T12:12:21.971 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.902+0000 7f10ff7fe700 1 -- 172.21.15.195:0/4018061141 learned_addr learned my addr 172.21.15.195:0/4018061141 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:12:21.971 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.903+0000 7f10feffd700 1 --2- 172.21.15.195:0/4018061141 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f11000c34a0 0x7f11000c6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:21.971 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.903+0000 7f10fffff700 1 --2- 172.21.15.195:0/4018061141 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f110014e210 0x7f110014e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:21.972 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.903+0000 7f10ff7fe700 1 -- 172.21.15.195:0/4018061141 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f11000c34a0 msgr2=0x7f11000c6900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:21.972 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.903+0000 7f10ff7fe700 1 --2- 172.21.15.195:0/4018061141 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f11000c34a0 0x7f11000c6900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:21.972 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.903+0000 7f10ff7fe700 1 -- 172.21.15.195:0/4018061141 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f110014e210 msgr2=0x7f110014e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:21.972 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.903+0000 7f10ff7fe700 1 --2- 172.21.15.195:0/4018061141 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f110014e210 0x7f110014e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:21.973 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.903+0000 7f10ff7fe700 1 -- 172.21.15.195:0/4018061141 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10f00092e0 con 0x7f110006e850 2022-06-27T12:12:21.973 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.903+0000 7f10ff7fe700 1 --2- 172.21.15.195:0/4018061141 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f110006e850 0x7f11000bc5c0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f10f400c490 tx=0x7f10f4007dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:12:21.973 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.903+0000 7f10e77fe700 1 -- 172.21.15.195:0/4018061141 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f4007430 con 0x7f110006e850 2022-06-27T12:12:21.973 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.903+0000 7f10e77fe700 1 -- 172.21.15.195:0/4018061141 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10f4007590 con 0x7f110006e850 2022-06-27T12:12:21.974 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.903+0000 7f1105b67700 1 -- 172.21.15.195:0/4018061141 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1100155d20 con 0x7f110006e850 2022-06-27T12:12:21.974 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.903+0000 7f10e77fe700 1 -- 172.21.15.195:0/4018061141 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f4004920 con 0x7f110006e850 2022-06-27T12:12:21.974 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.903+0000 7f1105b67700 1 -- 172.21.15.195:0/4018061141 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1100166ad0 con 0x7f110006e850 2022-06-27T12:12:21.974 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.904+0000 7f1105b67700 1 -- 172.21.15.195:0/4018061141 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1100167400 con 0x7f110006e850 2022-06-27T12:12:21.974 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.904+0000 7f10e77fe700 1 -- 172.21.15.195:0/4018061141 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f10f4010030 con 0x7f110006e850 2022-06-27T12:12:21.975 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.905+0000 7f10e77fe700 1 --2- 172.21.15.195:0/4018061141 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f10e0063d00 0x7f10e00661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:21.975 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.905+0000 7f10e77fe700 1 -- 172.21.15.195:0/4018061141 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7406+0+0 (secure 0 0 0) 0x7f10f4080900 con 0x7f110006e850 2022-06-27T12:12:21.975 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.905+0000 7f10feffd700 1 --2- 172.21.15.195:0/4018061141 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f10e0063d00 0x7f10e00661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:21.975 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.905+0000 7f10feffd700 1 --2- 172.21.15.195:0/4018061141 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f10e0063d00 0x7f10e00661b0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f10f0011fd0 tx=0x7f10f0017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:12:21.976 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:21.909+0000 7f10e77fe700 1 -- 172.21.15.195:0/4018061141 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f10f404f030 con 0x7f110006e850 2022-06-27T12:12:22.137 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:22.137+0000 7f1105b67700 1 -- 172.21.15.195:0/4018061141 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f11000722c0 con 0x7f110006e850 2022-06-27T12:12:22.137 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:22.137+0000 7f10e77fe700 1 -- 172.21.15.195:0/4018061141 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+6128 (secure 0 0 0) 0x7f10f4052650 con 0x7f110006e850 2022-06-27T12:12:22.138 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:12:22.138 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":96,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":6865,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/2349973547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2349973547},{"type":"v1","addr":"172.21.15.202:6835","nonce":2349973547}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:11:57.156947+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6050},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6050":{"gid":6050,"name":"b","rank":0,"incarnation":91,"state":"up:active","state_seq":15,"addr":"172.21.15.202:6833/3890799919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3890799919},{"type":"v1","addr":"172.21.15.202:6833","nonce":3890799919}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:12:22.144 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:22.139+0000 7f1105b67700 1 -- 172.21.15.195:0/4018061141 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f10e0063d00 msgr2=0x7f10e00661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:22.144 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:22.139+0000 7f1105b67700 1 --2- 172.21.15.195:0/4018061141 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f10e0063d00 0x7f10e00661b0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f10f0011fd0 tx=0x7f10f0017000 comp rx=0 tx=0).stop 2022-06-27T12:12:22.145 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:22.139+0000 7f1105b67700 1 -- 172.21.15.195:0/4018061141 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f110006e850 msgr2=0x7f11000bc5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:22.145 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:22.139+0000 7f1105b67700 1 --2- 172.21.15.195:0/4018061141 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f110006e850 0x7f11000bc5c0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f10f400c490 tx=0x7f10f4007dd0 comp rx=0 tx=0).stop 2022-06-27T12:12:22.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:22.139+0000 7f1105b67700 1 -- 172.21.15.195:0/4018061141 shutdown_connections 2022-06-27T12:12:22.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:22.139+0000 7f1105b67700 1 --2- 172.21.15.195:0/4018061141 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f10e0063d00 0x7f10e00661b0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:22.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:22.139+0000 7f1105b67700 1 --2- 172.21.15.195:0/4018061141 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f110006e850 0x7f11000bc5c0 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:22.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:22.139+0000 7f1105b67700 1 --2- 172.21.15.195:0/4018061141 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f110014e210 0x7f110014e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:22.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:22.139+0000 7f1105b67700 1 --2- 172.21.15.195:0/4018061141 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f11000c34a0 0x7f11000c6900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:22.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:22.139+0000 7f1105b67700 1 -- 172.21.15.195:0/4018061141 >> 172.21.15.195:0/4018061141 conn(0x7f11000c6350 msgr2=0x7f1100157720 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:12:22.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:22.139+0000 7f1105b67700 1 -- 172.21.15.195:0/4018061141 shutdown_connections 2022-06-27T12:12:22.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:22.139+0000 7f1105b67700 1 -- 172.21.15.195:0/4018061141 wait complete. 2022-06-27T12:12:22.155 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 96 2022-06-27T12:12:22.156 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2022-06-27T12:12:24.157 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:12:24.345 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.289+0000 7f011c81d700 1 Processor -- start 2022-06-27T12:12:24.346 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.289+0000 7f011c81d700 1 -- start start 2022-06-27T12:12:24.347 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.290+0000 7f011c81d700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f01180c5620 0x7f0118157330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:24.347 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.290+0000 7f011c81d700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f011806e200 con 0x7f01180c9700 2022-06-27T12:12:24.347 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.290+0000 7f011c81d700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f011806a010 con 0x7f01180c5250 2022-06-27T12:12:24.348 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.290+0000 7f011c81d700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0118157870 con 0x7f01180c5620 2022-06-27T12:12:24.348 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.290+0000 7f0116d9d700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f01180c9700 0x7f011806e690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:56852/0 (socket says 172.21.15.195:56852) 2022-06-27T12:12:24.349 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.290+0000 7f0116d9d700 1 -- 172.21.15.195:0/39431316 learned_addr learned my addr 172.21.15.195:0/39431316 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:12:24.356 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.290+0000 7f0115d9b700 1 --2- 172.21.15.195:0/39431316 >> v2:172.21.15.202:3300/0 conn(0x7f01180c5620 0x7f0118157330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:24.356 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.290+0000 7f011559a700 1 -- 172.21.15.195:0/39431316 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1135906028 0 0) 0x7f011806e200 con 0x7f01180c9700 2022-06-27T12:12:24.357 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.291+0000 7f011559a700 1 -- 172.21.15.195:0/39431316 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00fc003430 con 0x7f01180c9700 2022-06-27T12:12:24.357 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.291+0000 7f011559a700 1 -- 172.21.15.195:0/39431316 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2147813415 0 0) 0x7f00fc003430 con 0x7f01180c9700 2022-06-27T12:12:24.357 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.291+0000 7f011559a700 1 -- 172.21.15.195:0/39431316 >> v2:172.21.15.202:3300/0 conn(0x7f01180c5620 msgr2=0x7f0118157330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:12:24.358 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.291+0000 7f011559a700 1 --2- 172.21.15.195:0/39431316 >> v2:172.21.15.202:3300/0 conn(0x7f01180c5620 0x7f0118157330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:24.358 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.291+0000 7f011559a700 1 -- 172.21.15.195:0/39431316 >> v1:172.21.15.202:6789/0 conn(0x7f01180c5250 legacy=0x7f0118067ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:24.359 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.291+0000 7f011559a700 1 -- 172.21.15.195:0/39431316 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f011806dbf0 con 0x7f01180c9700 2022-06-27T12:12:24.359 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.291+0000 7f011559a700 1 -- 172.21.15.195:0/39431316 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7f01100017f0 con 0x7f01180c9700 2022-06-27T12:12:24.364 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.291+0000 7f0115d9b700 1 --2- 172.21.15.195:0/39431316 >> v2:172.21.15.202:3300/0 conn(0x7f01180c5620 0x7f0118157330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:12:24.364 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.291+0000 7f011559a700 1 -- 172.21.15.195:0/39431316 >> v1:172.21.15.195:6789/0 conn(0x7f01180c9700 legacy=0x7f011806e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:24.365 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.291+0000 7f011559a700 1 --2- 172.21.15.195:0/39431316 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f00fc003920 0x7f00fc003d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:24.366 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.291+0000 7f011559a700 1 --2- 172.21.15.195:0/39431316 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f00fc0046d0 0x7f00fc004b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:24.368 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.291+0000 7f011559a700 1 --2- 172.21.15.195:0/39431316 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f00fc005080 0x7f00fc00b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:24.368 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.291+0000 7f011559a700 1 -- 172.21.15.195:0/39431316 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0118157870 con 0x7f00fc003920 2022-06-27T12:12:24.369 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.291+0000 7f011559a700 1 -- 172.21.15.195:0/39431316 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00fc00bf90 con 0x7f00fc005080 2022-06-27T12:12:24.369 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.291+0000 7f011559a700 1 -- 172.21.15.195:0/39431316 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00fc00c150 con 0x7f00fc0046d0 2022-06-27T12:12:24.369 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.291+0000 7f011659c700 1 --2- 172.21.15.195:0/39431316 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f00fc003920 0x7f00fc003d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:24.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.291+0000 7f0115d9b700 1 --2- 172.21.15.195:0/39431316 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f00fc0046d0 0x7f00fc004b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:24.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.291+0000 7f0116d9d700 1 --2- 172.21.15.195:0/39431316 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f00fc005080 0x7f00fc00b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:24.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.291+0000 7f011659c700 1 -- 172.21.15.195:0/39431316 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f00fc0046d0 msgr2=0x7f00fc004b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:12:24.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.291+0000 7f011659c700 1 --2- 172.21.15.195:0/39431316 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f00fc0046d0 0x7f00fc004b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:24.372 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.291+0000 7f011659c700 1 -- 172.21.15.195:0/39431316 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f00fc005080 msgr2=0x7f00fc00b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:12:24.372 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.291+0000 7f011659c700 1 --2- 172.21.15.195:0/39431316 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f00fc005080 0x7f00fc00b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:24.380 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.291+0000 7f011659c700 1 -- 172.21.15.195:0/39431316 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00fc003430 con 0x7f00fc003920 2022-06-27T12:12:24.380 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.291+0000 7f0116d9d700 1 --2- 172.21.15.195:0/39431316 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f00fc005080 0x7f00fc00b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:12:24.381 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.291+0000 7f0116d9d700 1 -- 172.21.15.195:0/39431316 reap_dead start 2022-06-27T12:12:24.381 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.291+0000 7f011659c700 1 --2- 172.21.15.195:0/39431316 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f00fc003920 0x7f00fc003d30 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f010c001760 tx=0x7f010c007bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f1d9e5fc69ca1f9 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:12:24.381 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.291+0000 7f011559a700 1 -- 172.21.15.195:0/39431316 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f010c0050d0 con 0x7f00fc003920 2022-06-27T12:12:24.382 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.291+0000 7f011559a700 1 -- 172.21.15.195:0/39431316 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f010c005230 con 0x7f00fc003920 2022-06-27T12:12:24.382 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.292+0000 7f011559a700 1 -- 172.21.15.195:0/39431316 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f010c003750 con 0x7f00fc003920 2022-06-27T12:12:24.383 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.292+0000 7f011c81d700 1 -- 172.21.15.195:0/39431316 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f00fc003920 msgr2=0x7f00fc003d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:24.383 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.292+0000 7f011c81d700 1 --2- 172.21.15.195:0/39431316 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f00fc003920 0x7f00fc003d30 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f010c001760 tx=0x7f010c007bb0 comp rx=0 tx=0).stop 2022-06-27T12:12:24.383 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.292+0000 7f011c81d700 1 -- 172.21.15.195:0/39431316 shutdown_connections 2022-06-27T12:12:24.384 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.292+0000 7f011c81d700 1 --2- 172.21.15.195:0/39431316 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f00fc003920 0x7f00fc003d30 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:24.384 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.292+0000 7f011c81d700 1 -- 172.21.15.195:0/39431316 >> 172.21.15.195:0/39431316 conn(0x7f011806c2e0 msgr2=0x7f01180c8ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:12:24.385 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.292+0000 7f011c81d700 1 -- 172.21.15.195:0/39431316 shutdown_connections 2022-06-27T12:12:24.385 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.292+0000 7f011c81d700 1 -- 172.21.15.195:0/39431316 wait complete. 2022-06-27T12:12:24.385 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.293+0000 7f011c81d700 1 Processor -- start 2022-06-27T12:12:24.388 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.293+0000 7f011c81d700 1 -- start start 2022-06-27T12:12:24.389 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.294+0000 7f011c81d700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f00fc003920 0x7f0118167bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:24.389 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.294+0000 7f011c81d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f01180c8270 0x7f01181542a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:24.390 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.294+0000 7f011c81d700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f01180bfaa0 0x7f01181518a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:24.390 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.294+0000 7f011c81d700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0118153e10 con 0x7f01180c8270 2022-06-27T12:12:24.391 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.294+0000 7f011c81d700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0118153800 con 0x7f01180bfaa0 2022-06-27T12:12:24.391 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.294+0000 7f011c81d700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01180b8ea0 con 0x7f00fc003920 2022-06-27T12:12:24.392 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.294+0000 7f0115d9b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f01180c8270 0x7f01181542a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:24.392 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.294+0000 7f0115d9b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f01180c8270 0x7f01181542a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50674/0 (socket says 172.21.15.195:50674) 2022-06-27T12:12:24.393 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.294+0000 7f0115d9b700 1 -- 172.21.15.195:0/875926332 learned_addr learned my addr 172.21.15.195:0/875926332 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:12:24.393 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.294+0000 7f011659c700 1 --2- 172.21.15.195:0/875926332 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f00fc003920 0x7f0118167bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:24.393 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.294+0000 7f0116d9d700 1 --2- 172.21.15.195:0/875926332 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f01180bfaa0 0x7f01181518a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:24.394 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.294+0000 7f0115d9b700 1 -- 172.21.15.195:0/875926332 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f00fc003920 msgr2=0x7f0118167bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:24.394 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.294+0000 7f0115d9b700 1 --2- 172.21.15.195:0/875926332 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f00fc003920 0x7f0118167bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:24.395 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.294+0000 7f0115d9b700 1 -- 172.21.15.195:0/875926332 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f01180bfaa0 msgr2=0x7f01181518a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:24.395 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.294+0000 7f0115d9b700 1 --2- 172.21.15.195:0/875926332 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f01180bfaa0 0x7f01181518a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:24.396 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.294+0000 7f0115d9b700 1 -- 172.21.15.195:0/875926332 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f011816aa30 con 0x7f01180c8270 2022-06-27T12:12:24.396 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.294+0000 7f0116d9d700 1 --2- 172.21.15.195:0/875926332 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f01180bfaa0 0x7f01181518a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:12:24.396 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.294+0000 7f0115d9b700 1 --2- 172.21.15.195:0/875926332 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f01180c8270 0x7f01181542a0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f0108011d70 tx=0x7f0108009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:12:24.396 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.295+0000 7f01067fc700 1 -- 172.21.15.195:0/875926332 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0108004030 con 0x7f01180c8270 2022-06-27T12:12:24.397 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.295+0000 7f01067fc700 1 -- 172.21.15.195:0/875926332 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0108004190 con 0x7f01180c8270 2022-06-27T12:12:24.397 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.295+0000 7f011c81d700 1 -- 172.21.15.195:0/875926332 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0118154a30 con 0x7f01180c8270 2022-06-27T12:12:24.397 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.295+0000 7f01067fc700 1 -- 172.21.15.195:0/875926332 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f010800fac0 con 0x7f01180c8270 2022-06-27T12:12:24.397 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.295+0000 7f011c81d700 1 -- 172.21.15.195:0/875926332 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0118154d10 con 0x7f01180c8270 2022-06-27T12:12:24.397 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.295+0000 7f011c81d700 1 -- 172.21.15.195:0/875926332 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01180722c0 con 0x7f01180c8270 2022-06-27T12:12:24.398 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.296+0000 7f01067fc700 1 -- 172.21.15.195:0/875926332 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f0108003520 con 0x7f01180c8270 2022-06-27T12:12:24.398 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.296+0000 7f01067fc700 1 --2- 172.21.15.195:0/875926332 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f00f4063d00 0x7f00f40661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:24.398 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.296+0000 7f01067fc700 1 -- 172.21.15.195:0/875926332 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7406+0+0 (secure 0 0 0) 0x7f0108018030 con 0x7f01180c8270 2022-06-27T12:12:24.399 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.296+0000 7f011659c700 1 --2- 172.21.15.195:0/875926332 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f00f4063d00 0x7f00f40661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:24.399 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.297+0000 7f011659c700 1 --2- 172.21.15.195:0/875926332 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f00f4063d00 0x7f00f40661b0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f010c000ff0 tx=0x7f010c015000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:12:24.399 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.301+0000 7f01067fc700 1 -- 172.21.15.195:0/875926332 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f01080502a0 con 0x7f01180c8270 2022-06-27T12:12:24.527 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.526+0000 7f011c81d700 1 -- 172.21.15.195:0/875926332 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f01181553f0 con 0x7f01180c8270 2022-06-27T12:12:24.527 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.526+0000 7f01067fc700 1 -- 172.21.15.195:0/875926332 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+6128 (secure 0 0 0) 0x7f0108022070 con 0x7f01180c8270 2022-06-27T12:12:24.528 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:12:24.528 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":96,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":6865,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/2349973547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2349973547},{"type":"v1","addr":"172.21.15.202:6835","nonce":2349973547}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:11:57.156947+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6050},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6050":{"gid":6050,"name":"b","rank":0,"incarnation":91,"state":"up:active","state_seq":15,"addr":"172.21.15.202:6833/3890799919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3890799919},{"type":"v1","addr":"172.21.15.202:6833","nonce":3890799919}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:12:24.529 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.528+0000 7f011c81d700 1 -- 172.21.15.195:0/875926332 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f00f4063d00 msgr2=0x7f00f40661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:24.529 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.528+0000 7f011c81d700 1 --2- 172.21.15.195:0/875926332 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f00f4063d00 0x7f00f40661b0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f010c000ff0 tx=0x7f010c015000 comp rx=0 tx=0).stop 2022-06-27T12:12:24.529 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.528+0000 7f011c81d700 1 -- 172.21.15.195:0/875926332 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f01180c8270 msgr2=0x7f01181542a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:24.532 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.528+0000 7f011c81d700 1 --2- 172.21.15.195:0/875926332 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f01180c8270 0x7f01181542a0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f0108011d70 tx=0x7f0108009060 comp rx=0 tx=0).stop 2022-06-27T12:12:24.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.528+0000 7f011c81d700 1 -- 172.21.15.195:0/875926332 shutdown_connections 2022-06-27T12:12:24.542 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.528+0000 7f011c81d700 1 --2- 172.21.15.195:0/875926332 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f00f4063d00 0x7f00f40661b0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:24.543 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.528+0000 7f011c81d700 1 --2- 172.21.15.195:0/875926332 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f01180bfaa0 0x7f01181518a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:24.543 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.528+0000 7f011c81d700 1 --2- 172.21.15.195:0/875926332 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f00fc003920 0x7f0118167bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:24.543 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.528+0000 7f011c81d700 1 --2- 172.21.15.195:0/875926332 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f01180c8270 0x7f01181542a0 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:24.544 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.528+0000 7f011c81d700 1 -- 172.21.15.195:0/875926332 >> 172.21.15.195:0/875926332 conn(0x7f011806c2e0 msgr2=0x7f0118067ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:12:24.544 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.528+0000 7f011c81d700 1 -- 172.21.15.195:0/875926332 shutdown_connections 2022-06-27T12:12:24.544 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:24.528+0000 7f011c81d700 1 -- 172.21.15.195:0/875926332 wait complete. 2022-06-27T12:12:24.544 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 96 2022-06-27T12:12:24.545 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2022-06-27T12:12:26.552 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:12:26.784 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:26.783+0000 7f93abacd700 1 Processor -- start 2022-06-27T12:12:26.784 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:26.784+0000 7f93abacd700 1 -- start start 2022-06-27T12:12:26.799 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:26.784+0000 7f93abacd700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f93a406e850 0x7f93a41570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:26.809 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:26.784+0000 7f93abacd700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f93a406fef0 0x7f93a4069de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:26.809 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:26.784+0000 7f93abacd700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93a406fba0 con 0x7f93a40c34a0 2022-06-27T12:12:26.810 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:26.784+0000 7f93abacd700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93a41575e0 con 0x7f93a406e850 2022-06-27T12:12:26.810 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:26.784+0000 7f93abacd700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93a406f5e0 con 0x7f93a406fef0 2022-06-27T12:12:26.810 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:26.785+0000 7f93a9869700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f93a406e850 0x7f93a41570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:26.811 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:26.785+0000 7f93a9068700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f93a406fef0 0x7f93a4069de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:26.811 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:26.785+0000 7f93a9068700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f93a406fef0 0x7f93a4069de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3301/0 says I am v2:172.21.15.195:43440/0 (socket says 172.21.15.195:43440) 2022-06-27T12:12:26.812 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:26.785+0000 7f93a9869700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f93a406e850 0x7f93a41570a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3300/0 says I am v2:172.21.15.195:48810/0 (socket says 172.21.15.195:48810) 2022-06-27T12:12:26.812 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:26.785+0000 7f93a9068700 1 -- 172.21.15.195:0/622814367 learned_addr learned my addr 172.21.15.195:0/622814367 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:12:26.820 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:26.785+0000 7f93aa06a700 1 --1- >> v1:172.21.15.202:6789/0 conn(0x7f93a40c34a0 0x7f93a40c2920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.202:6789/0 says I am v1:172.21.15.195:58730/0 (socket says 172.21.15.195:58730) 2022-06-27T12:12:26.821 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:26.785+0000 7f93a9869700 1 -- 172.21.15.195:0/622814367 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f93a406fef0 msgr2=0x7f93a4069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:12:26.821 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:26.785+0000 7f93a9869700 1 --2- 172.21.15.195:0/622814367 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f93a406fef0 0x7f93a4069de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:26.822 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:26.785+0000 7f93a9869700 1 -- 172.21.15.195:0/622814367 >> v1:172.21.15.202:6789/0 conn(0x7f93a40c34a0 legacy=0x7f93a40c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:26.822 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:26.785+0000 7f93a8867700 1 -- 172.21.15.195:0/622814367 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2323278956 0 0) 0x7f93a406fba0 con 0x7f93a40c34a0 2022-06-27T12:12:26.823 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:26.785+0000 7f93a9869700 1 -- 172.21.15.195:0/622814367 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93a406f8c0 con 0x7f93a406e850 2022-06-27T12:12:26.824 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:26.785+0000 7f93a9068700 1 --2- 172.21.15.195:0/622814367 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f93a406fef0 0x7f93a4069de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:12:26.824 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:26.785+0000 7f93a9869700 1 --2- 172.21.15.195:0/622814367 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f93a406e850 0x7f93a41570a0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f93980097a0 tx=0x7f9398009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=35a2a1aab9bdf6bf server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:12:26.825 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:26.786+0000 7f93a8867700 1 -- 172.21.15.195:0/622814367 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9398011630 con 0x7f93a406e850 2022-06-27T12:12:26.826 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:26.786+0000 7f93a8867700 1 -- 172.21.15.195:0/622814367 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9398011790 con 0x7f93a406e850 2022-06-27T12:12:26.826 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:26.786+0000 7f93a8867700 1 -- 172.21.15.195:0/622814367 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9398011b30 con 0x7f93a406e850 2022-06-27T12:12:26.827 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:26.786+0000 7f93abacd700 1 -- 172.21.15.195:0/622814367 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f93a406e850 msgr2=0x7f93a41570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:26.827 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:26.786+0000 7f93abacd700 1 --2- 172.21.15.195:0/622814367 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f93a406e850 0x7f93a41570a0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f93980097a0 tx=0x7f9398009060 comp rx=0 tx=0).stop 2022-06-27T12:12:26.828 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:26.786+0000 7f93abacd700 1 -- 172.21.15.195:0/622814367 shutdown_connections 2022-06-27T12:12:26.828 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:26.786+0000 7f93abacd700 1 --2- 172.21.15.195:0/622814367 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f93a406e850 0x7f93a41570a0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:26.830 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:26.786+0000 7f93abacd700 1 --2- 172.21.15.195:0/622814367 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f93a406fef0 0x7f93a4069de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:26.830 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:26.786+0000 7f93abacd700 1 -- 172.21.15.195:0/622814367 >> 172.21.15.195:0/622814367 conn(0x7f93a40c6350 msgr2=0x7f93a40c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:12:26.831 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:26.786+0000 7f93abacd700 1 -- 172.21.15.195:0/622814367 shutdown_connections 2022-06-27T12:12:26.831 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:26.786+0000 7f93abacd700 1 -- 172.21.15.195:0/622814367 wait complete. 2022-06-27T12:12:26.832 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:26.787+0000 7f93abacd700 1 Processor -- start 2022-06-27T12:12:26.832 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:26.787+0000 7f93abacd700 1 -- start start 2022-06-27T12:12:26.833 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:26.788+0000 7f93abacd700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f93a406e850 0x7f93a40c0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:26.833 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:26.788+0000 7f93abacd700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f93a40c34a0 0x7f93a41535d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:26.833 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:26.788+0000 7f93abacd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f93a40bd600 0x7f93a4153b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:26.835 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:26.788+0000 7f93abacd700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93a40bde80 con 0x7f93a40bd600 2022-06-27T12:12:26.835 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:26.788+0000 7f93abacd700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93a40c0b60 con 0x7f93a406e850 2022-06-27T12:12:26.836 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:26.788+0000 7f93abacd700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93a4155670 con 0x7f93a40c34a0 2022-06-27T12:12:26.837 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:26.788+0000 7f93aa06a700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f93a40bd600 0x7f93a4153b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:26.837 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:26.788+0000 7f93a9068700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f93a40c34a0 0x7f93a41535d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:26.844 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:26.788+0000 7f93aa06a700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f93a40bd600 0x7f93a4153b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50692/0 (socket says 172.21.15.195:50692) 2022-06-27T12:12:26.844 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:26.788+0000 7f93a9869700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f93a406e850 0x7f93a40c0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:26.845 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:26.788+0000 7f93aa06a700 1 -- 172.21.15.195:0/4152202652 learned_addr learned my addr 172.21.15.195:0/4152202652 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:12:26.845 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:26.789+0000 7f93aa06a700 1 -- 172.21.15.195:0/4152202652 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f93a40c34a0 msgr2=0x7f93a41535d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:26.845 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:26.789+0000 7f93aa06a700 1 --2- 172.21.15.195:0/4152202652 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f93a40c34a0 0x7f93a41535d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:26.845 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:26.789+0000 7f93aa06a700 1 -- 172.21.15.195:0/4152202652 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f93a406e850 msgr2=0x7f93a40c0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:26.846 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:26.789+0000 7f93aa06a700 1 --2- 172.21.15.195:0/4152202652 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f93a406e850 0x7f93a40c0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:26.846 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:26.789+0000 7f93aa06a700 1 -- 172.21.15.195:0/4152202652 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93a00093a0 con 0x7f93a40bd600 2022-06-27T12:12:26.846 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:26.789+0000 7f93aa06a700 1 --2- 172.21.15.195:0/4152202652 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f93a40bd600 0x7f93a4153b10 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f939c00c890 tx=0x7f939c001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:12:26.846 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:26.789+0000 7f9395ffb700 1 -- 172.21.15.195:0/4152202652 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f939c004c80 con 0x7f93a40bd600 2022-06-27T12:12:26.847 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:26.789+0000 7f9395ffb700 1 -- 172.21.15.195:0/4152202652 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f939c004de0 con 0x7f93a40bd600 2022-06-27T12:12:26.847 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:26.789+0000 7f93abacd700 1 -- 172.21.15.195:0/4152202652 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93980092e0 con 0x7f93a40bd600 2022-06-27T12:12:26.847 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:26.789+0000 7f9395ffb700 1 -- 172.21.15.195:0/4152202652 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f939c0073a0 con 0x7f93a40bd600 2022-06-27T12:12:26.848 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:26.789+0000 7f93abacd700 1 -- 172.21.15.195:0/4152202652 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93a41515d0 con 0x7f93a40bd600 2022-06-27T12:12:26.848 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:26.790+0000 7f93abacd700 1 -- 172.21.15.195:0/4152202652 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93a40722c0 con 0x7f93a40bd600 2022-06-27T12:12:26.848 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:26.790+0000 7f9395ffb700 1 -- 172.21.15.195:0/4152202652 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f939c01a030 con 0x7f93a40bd600 2022-06-27T12:12:26.848 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:26.791+0000 7f9395ffb700 1 --2- 172.21.15.195:0/4152202652 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9384063d00 0x7f93840661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:26.849 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:26.791+0000 7f9395ffb700 1 -- 172.21.15.195:0/4152202652 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7406+0+0 (secure 0 0 0) 0x7f939c0802a0 con 0x7f93a40bd600 2022-06-27T12:12:26.849 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:26.791+0000 7f93a9869700 1 --2- 172.21.15.195:0/4152202652 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9384063d00 0x7f93840661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:26.849 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:26.791+0000 7f93a9869700 1 --2- 172.21.15.195:0/4152202652 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9384063d00 0x7f93840661b0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f9398009260 tx=0x7f9398017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:12:26.849 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:26.795+0000 7f9395ffb700 1 -- 172.21.15.195:0/4152202652 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f939c04e9d0 con 0x7f93a40bd600 2022-06-27T12:12:27.024 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:27.024+0000 7f93abacd700 1 -- 172.21.15.195:0/4152202652 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f93a4151950 con 0x7f93a40bd600 2022-06-27T12:12:27.025 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:27.024+0000 7f9395ffb700 1 -- 172.21.15.195:0/4152202652 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+6128 (secure 0 0 0) 0x7f939c051ff0 con 0x7f93a40bd600 2022-06-27T12:12:27.025 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:12:27.028 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":96,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":6865,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/2349973547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2349973547},{"type":"v1","addr":"172.21.15.202:6835","nonce":2349973547}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:11:57.156947+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6050},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6050":{"gid":6050,"name":"b","rank":0,"incarnation":91,"state":"up:active","state_seq":15,"addr":"172.21.15.202:6833/3890799919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3890799919},{"type":"v1","addr":"172.21.15.202:6833","nonce":3890799919}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:12:27.030 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:27.026+0000 7f93abacd700 1 -- 172.21.15.195:0/4152202652 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9384063d00 msgr2=0x7f93840661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:27.030 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:27.026+0000 7f93abacd700 1 --2- 172.21.15.195:0/4152202652 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9384063d00 0x7f93840661b0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f9398009260 tx=0x7f9398017000 comp rx=0 tx=0).stop 2022-06-27T12:12:27.031 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:27.026+0000 7f93abacd700 1 -- 172.21.15.195:0/4152202652 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f93a40bd600 msgr2=0x7f93a4153b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:27.031 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:27.026+0000 7f93abacd700 1 --2- 172.21.15.195:0/4152202652 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f93a40bd600 0x7f93a4153b10 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f939c00c890 tx=0x7f939c001630 comp rx=0 tx=0).stop 2022-06-27T12:12:27.031 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:27.026+0000 7f93abacd700 1 -- 172.21.15.195:0/4152202652 shutdown_connections 2022-06-27T12:12:27.032 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:27.026+0000 7f93abacd700 1 --2- 172.21.15.195:0/4152202652 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9384063d00 0x7f93840661b0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:27.032 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:27.026+0000 7f93abacd700 1 --2- 172.21.15.195:0/4152202652 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f93a40bd600 0x7f93a4153b10 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:27.032 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:27.026+0000 7f93abacd700 1 --2- 172.21.15.195:0/4152202652 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f93a406e850 0x7f93a40c0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:27.032 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:27.026+0000 7f93abacd700 1 --2- 172.21.15.195:0/4152202652 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f93a40c34a0 0x7f93a41535d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:27.033 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:27.026+0000 7f93abacd700 1 -- 172.21.15.195:0/4152202652 >> 172.21.15.195:0/4152202652 conn(0x7f93a40c6350 msgr2=0x7f93a406c6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:12:27.033 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:27.026+0000 7f93abacd700 1 -- 172.21.15.195:0/4152202652 shutdown_connections 2022-06-27T12:12:27.033 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:27.026+0000 7f93abacd700 1 -- 172.21.15.195:0/4152202652 wait complete. 2022-06-27T12:12:27.033 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 96 2022-06-27T12:12:27.039 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2022-06-27T12:12:29.040 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:12:29.175 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.175+0000 7feaac505700 1 Processor -- start 2022-06-27T12:12:29.175 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.175+0000 7feaac505700 1 -- start start 2022-06-27T12:12:29.178 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.176+0000 7feaac505700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7feaa406a450 0x7feaa4063af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:29.178 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.176+0000 7feaac505700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7feaa41508d0 0x7feaa40bbf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:29.179 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.176+0000 7feaac505700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaa4064030 con 0x7feaa4150500 2022-06-27T12:12:29.182 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.176+0000 7feaac505700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feaa40c4160 con 0x7feaa406a450 2022-06-27T12:12:29.182 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.176+0000 7feaac505700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feaa40c4440 con 0x7feaa41508d0 2022-06-27T12:12:29.183 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.176+0000 7feaa9aa0700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7feaa406a450 0x7feaa4063af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:29.183 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.176+0000 7feaa9aa0700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7feaa406a450 0x7feaa4063af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50696/0 (socket says 172.21.15.195:50696) 2022-06-27T12:12:29.183 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.176+0000 7feaa9aa0700 1 -- 172.21.15.195:0/461842066 learned_addr learned my addr 172.21.15.195:0/461842066 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:12:29.183 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.176+0000 7feaaa2a1700 1 --1- 172.21.15.195:0/461842066 >> v1:172.21.15.195:6789/0 conn(0x7feaa4150500 0x7feaa40c48d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:56884/0 (socket says 172.21.15.195:56884) 2022-06-27T12:12:29.184 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.176+0000 7feaaaaa2700 1 --2- 172.21.15.195:0/461842066 >> v2:172.21.15.202:3300/0 conn(0x7feaa41508d0 0x7feaa40bbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:29.184 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.176+0000 7feaa9aa0700 1 -- 172.21.15.195:0/461842066 >> v2:172.21.15.202:3300/0 conn(0x7feaa41508d0 msgr2=0x7feaa40bbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:12:29.184 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.176+0000 7feaa9aa0700 1 --2- 172.21.15.195:0/461842066 >> v2:172.21.15.202:3300/0 conn(0x7feaa41508d0 0x7feaa40bbf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:29.184 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.176+0000 7feaa9aa0700 1 -- 172.21.15.195:0/461842066 >> v1:172.21.15.195:6789/0 conn(0x7feaa4150500 legacy=0x7feaa40c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:29.185 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.176+0000 7feaa9aa0700 1 -- 172.21.15.195:0/461842066 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaa415a800 con 0x7feaa406a450 2022-06-27T12:12:29.185 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.176+0000 7feaa929f700 1 -- 172.21.15.195:0/461842066 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2731412444 0 0) 0x7feaa00011d0 con 0x7feaa4150500 2022-06-27T12:12:29.185 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.176+0000 7feaaaaa2700 1 --2- 172.21.15.195:0/461842066 >> v2:172.21.15.202:3300/0 conn(0x7feaa41508d0 0x7feaa40bbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T12:12:29.185 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.176+0000 7feaa9aa0700 1 --2- 172.21.15.195:0/461842066 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7feaa406a450 0x7feaa4063af0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7feaa4064800 tx=0x7fea94009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50257d0ed9d62c25 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:12:29.185 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.176+0000 7feaa929f700 1 -- 172.21.15.195:0/461842066 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea9400b640 con 0x7feaa406a450 2022-06-27T12:12:29.186 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.176+0000 7feaa929f700 1 -- 172.21.15.195:0/461842066 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea9400b7a0 con 0x7feaa406a450 2022-06-27T12:12:29.186 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.176+0000 7feaa929f700 1 -- 172.21.15.195:0/461842066 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea9400bb40 con 0x7feaa406a450 2022-06-27T12:12:29.186 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.177+0000 7feaac505700 1 -- 172.21.15.195:0/461842066 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7feaa406a450 msgr2=0x7feaa4063af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:29.186 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.177+0000 7feaac505700 1 --2- 172.21.15.195:0/461842066 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7feaa406a450 0x7feaa4063af0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7feaa4064800 tx=0x7fea94009120 comp rx=0 tx=0).stop 2022-06-27T12:12:29.187 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.177+0000 7feaac505700 1 -- 172.21.15.195:0/461842066 shutdown_connections 2022-06-27T12:12:29.187 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.177+0000 7feaac505700 1 --2- 172.21.15.195:0/461842066 >> v2:172.21.15.202:3300/0 conn(0x7feaa41508d0 0x7feaa40bbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:29.187 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.177+0000 7feaac505700 1 --2- 172.21.15.195:0/461842066 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7feaa406a450 0x7feaa4063af0 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:29.187 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.177+0000 7feaac505700 1 -- 172.21.15.195:0/461842066 >> 172.21.15.195:0/461842066 conn(0x7feaa40c9b00 msgr2=0x7feaa40c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:12:29.188 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.177+0000 7feaac505700 1 -- 172.21.15.195:0/461842066 shutdown_connections 2022-06-27T12:12:29.188 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.177+0000 7feaac505700 1 -- 172.21.15.195:0/461842066 wait complete. 2022-06-27T12:12:29.188 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.178+0000 7feaac505700 1 Processor -- start 2022-06-27T12:12:29.199 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.178+0000 7feaac505700 1 -- start start 2022-06-27T12:12:29.200 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.179+0000 7feaac505700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7feaa406a450 0x7feaa4150f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:29.201 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.179+0000 7feaac505700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7feaa4150500 0x7feaa4158a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:29.202 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.179+0000 7feaac505700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7feaa41508d0 0x7feaa4158fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:29.202 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.179+0000 7feaac505700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaa4152680 con 0x7feaa406a450 2022-06-27T12:12:29.203 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.179+0000 7feaac505700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaa4169980 con 0x7feaa41508d0 2022-06-27T12:12:29.203 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.179+0000 7feaac505700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feaa4169c40 con 0x7feaa4150500 2022-06-27T12:12:29.204 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.179+0000 7feaaa2a1700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7feaa406a450 0x7feaa4150f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:29.204 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.179+0000 7feaaa2a1700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7feaa406a450 0x7feaa4150f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50704/0 (socket says 172.21.15.195:50704) 2022-06-27T12:12:29.205 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.179+0000 7feaaa2a1700 1 -- 172.21.15.195:0/3786247998 learned_addr learned my addr 172.21.15.195:0/3786247998 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:12:29.205 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.179+0000 7feaa9aa0700 1 --2- 172.21.15.195:0/3786247998 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7feaa4150500 0x7feaa4158a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:29.206 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.179+0000 7feaaaaa2700 1 --2- 172.21.15.195:0/3786247998 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7feaa41508d0 0x7feaa4158fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:29.206 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.179+0000 7feaaa2a1700 1 -- 172.21.15.195:0/3786247998 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7feaa4150500 msgr2=0x7feaa4158a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:29.207 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.179+0000 7feaaa2a1700 1 --2- 172.21.15.195:0/3786247998 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7feaa4150500 0x7feaa4158a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:29.207 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.179+0000 7feaaa2a1700 1 -- 172.21.15.195:0/3786247998 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7feaa41508d0 msgr2=0x7feaa4158fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:29.208 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.179+0000 7feaaa2a1700 1 --2- 172.21.15.195:0/3786247998 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7feaa41508d0 0x7feaa4158fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:29.209 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.179+0000 7feaaa2a1700 1 -- 172.21.15.195:0/3786247998 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea940093a0 con 0x7feaa406a450 2022-06-27T12:12:29.209 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.180+0000 7feaaa2a1700 1 --2- 172.21.15.195:0/3786247998 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7feaa406a450 0x7feaa4150f70 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7feaa000c4e0 tx=0x7feaa0007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:12:29.209 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.180+0000 7fea927fc700 1 -- 172.21.15.195:0/3786247998 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaa00072e0 con 0x7feaa406a450 2022-06-27T12:12:29.210 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.180+0000 7fea927fc700 1 -- 172.21.15.195:0/3786247998 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaa0007440 con 0x7feaa406a450 2022-06-27T12:12:29.210 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.180+0000 7feaac505700 1 -- 172.21.15.195:0/3786247998 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feaa4169f20 con 0x7feaa406a450 2022-06-27T12:12:29.211 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.180+0000 7fea927fc700 1 -- 172.21.15.195:0/3786247998 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaa0005900 con 0x7feaa406a450 2022-06-27T12:12:29.211 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.180+0000 7feaac505700 1 -- 172.21.15.195:0/3786247998 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feaa416a4e0 con 0x7feaa406a450 2022-06-27T12:12:29.212 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.181+0000 7feaac505700 1 -- 172.21.15.195:0/3786247998 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feaa40722c0 con 0x7feaa406a450 2022-06-27T12:12:29.212 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.181+0000 7fea927fc700 1 -- 172.21.15.195:0/3786247998 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7feaa0019030 con 0x7feaa406a450 2022-06-27T12:12:29.215 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.182+0000 7fea927fc700 1 --2- 172.21.15.195:0/3786247998 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fea84063d00 0x7fea840661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:29.215 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.182+0000 7feaa9aa0700 1 --2- 172.21.15.195:0/3786247998 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fea84063d00 0x7fea840661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:29.216 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.182+0000 7fea927fc700 1 -- 172.21.15.195:0/3786247998 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7406+0+0 (secure 0 0 0) 0x7feaa007ffe0 con 0x7feaa406a450 2022-06-27T12:12:29.216 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.182+0000 7feaa9aa0700 1 --2- 172.21.15.195:0/3786247998 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fea84063d00 0x7fea840661b0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fea9400a010 tx=0x7fea94017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:12:29.217 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.186+0000 7fea927fc700 1 -- 172.21.15.195:0/3786247998 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7feaa004e5a0 con 0x7feaa406a450 2022-06-27T12:12:29.416 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.415+0000 7feaac505700 1 -- 172.21.15.195:0/3786247998 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feaa4064030 con 0x7feaa406a450 2022-06-27T12:12:29.417 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.415+0000 7fea927fc700 1 -- 172.21.15.195:0/3786247998 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+6128 (secure 0 0 0) 0x7feaa4064030 con 0x7feaa406a450 2022-06-27T12:12:29.421 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:12:29.421 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":96,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":6865,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/2349973547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2349973547},{"type":"v1","addr":"172.21.15.202:6835","nonce":2349973547}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:11:57.156947+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6050},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6050":{"gid":6050,"name":"b","rank":0,"incarnation":91,"state":"up:active","state_seq":15,"addr":"172.21.15.202:6833/3890799919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3890799919},{"type":"v1","addr":"172.21.15.202:6833","nonce":3890799919}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:12:29.424 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.417+0000 7feaac505700 1 -- 172.21.15.195:0/3786247998 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fea84063d00 msgr2=0x7fea840661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:29.425 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.417+0000 7feaac505700 1 --2- 172.21.15.195:0/3786247998 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fea84063d00 0x7fea840661b0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fea9400a010 tx=0x7fea94017000 comp rx=0 tx=0).stop 2022-06-27T12:12:29.428 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.417+0000 7feaac505700 1 -- 172.21.15.195:0/3786247998 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7feaa406a450 msgr2=0x7feaa4150f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:29.428 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.417+0000 7feaac505700 1 --2- 172.21.15.195:0/3786247998 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7feaa406a450 0x7feaa4150f70 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7feaa000c4e0 tx=0x7feaa0007a40 comp rx=0 tx=0).stop 2022-06-27T12:12:29.431 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.417+0000 7feaac505700 1 -- 172.21.15.195:0/3786247998 shutdown_connections 2022-06-27T12:12:29.432 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.417+0000 7feaac505700 1 --2- 172.21.15.195:0/3786247998 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fea84063d00 0x7fea840661b0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:29.433 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.417+0000 7feaac505700 1 --2- 172.21.15.195:0/3786247998 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7feaa406a450 0x7feaa4150f70 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:29.437 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.417+0000 7feaac505700 1 --2- 172.21.15.195:0/3786247998 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7feaa41508d0 0x7feaa4158fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:29.439 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.417+0000 7feaac505700 1 --2- 172.21.15.195:0/3786247998 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7feaa4150500 0x7feaa4158a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:29.443 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.417+0000 7feaac505700 1 -- 172.21.15.195:0/3786247998 >> 172.21.15.195:0/3786247998 conn(0x7feaa40c9b00 msgr2=0x7feaa406eec0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:12:29.444 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.417+0000 7feaac505700 1 -- 172.21.15.195:0/3786247998 shutdown_connections 2022-06-27T12:12:29.446 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:29.417+0000 7feaac505700 1 -- 172.21.15.195:0/3786247998 wait complete. 2022-06-27T12:12:29.447 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 96 2022-06-27T12:12:29.450 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2022-06-27T12:12:31.452 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:12:31.616 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.585+0000 7fbec3510700 1 Processor -- start 2022-06-27T12:12:31.617 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.586+0000 7fbec3510700 1 -- start start 2022-06-27T12:12:31.617 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.586+0000 7fbec3510700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fbebc06fef0 0x7fbebc1570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:31.617 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.586+0000 7fbec3510700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbebc0c34a0 0x7fbebc0c2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:31.617 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.586+0000 7fbec3510700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbebc06fba0 con 0x7fbebc06e850 2022-06-27T12:12:31.618 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.586+0000 7fbec3510700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbebc1575e0 con 0x7fbebc06fef0 2022-06-27T12:12:31.618 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.586+0000 7fbec3510700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbebc06f5e0 con 0x7fbebc0c34a0 2022-06-27T12:12:31.628 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.586+0000 7fbec0aab700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fbebc06fef0 0x7fbebc1570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:31.628 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.587+0000 7fbec0aab700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fbebc06fef0 0x7fbebc1570a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50714/0 (socket says 172.21.15.195:50714) 2022-06-27T12:12:31.631 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.587+0000 7fbec0aab700 1 -- 172.21.15.195:0/3281169789 learned_addr learned my addr 172.21.15.195:0/3281169789 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:12:31.632 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.587+0000 7fbec1aad700 1 --2- 172.21.15.195:0/3281169789 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbebc0c34a0 0x7fbebc0c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:31.633 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.587+0000 7fbec0aab700 1 -- 172.21.15.195:0/3281169789 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbebc0c34a0 msgr2=0x7fbebc0c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:12:31.633 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.587+0000 7fbec0aab700 1 --2- 172.21.15.195:0/3281169789 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbebc0c34a0 0x7fbebc0c2920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:31.634 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.587+0000 7fbec0aab700 1 -- 172.21.15.195:0/3281169789 >> v1:172.21.15.202:6789/0 conn(0x7fbebc06e850 legacy=0x7fbebc069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:31.634 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.587+0000 7fbec0aab700 1 -- 172.21.15.195:0/3281169789 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbebc06f8c0 con 0x7fbebc06fef0 2022-06-27T12:12:31.635 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.587+0000 7fbec0aab700 1 --2- 172.21.15.195:0/3281169789 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbebc06fef0 0x7fbebc1570a0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fbeb8009860 tx=0x7fbeb8009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ba766b1a7f22a11 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:12:31.635 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.587+0000 7fbeaffff700 1 -- 172.21.15.195:0/3281169789 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeb800b690 con 0x7fbebc06fef0 2022-06-27T12:12:31.636 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.587+0000 7fbeaffff700 1 -- 172.21.15.195:0/3281169789 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbeb800b7f0 con 0x7fbebc06fef0 2022-06-27T12:12:31.636 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.587+0000 7fbeaffff700 1 -- 172.21.15.195:0/3281169789 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeb800bb90 con 0x7fbebc06fef0 2022-06-27T12:12:31.637 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.588+0000 7fbec3510700 1 -- 172.21.15.195:0/3281169789 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbebc06fef0 msgr2=0x7fbebc1570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:31.637 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.588+0000 7fbec3510700 1 --2- 172.21.15.195:0/3281169789 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbebc06fef0 0x7fbebc1570a0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fbeb8009860 tx=0x7fbeb8009120 comp rx=0 tx=0).stop 2022-06-27T12:12:31.638 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.588+0000 7fbec3510700 1 -- 172.21.15.195:0/3281169789 shutdown_connections 2022-06-27T12:12:31.638 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.588+0000 7fbec3510700 1 --2- 172.21.15.195:0/3281169789 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbebc0c34a0 0x7fbebc0c2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:31.639 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.588+0000 7fbec3510700 1 --2- 172.21.15.195:0/3281169789 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbebc06fef0 0x7fbebc1570a0 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:31.639 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.588+0000 7fbec3510700 1 -- 172.21.15.195:0/3281169789 >> 172.21.15.195:0/3281169789 conn(0x7fbebc0c6350 msgr2=0x7fbebc0c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:12:31.640 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.588+0000 7fbec3510700 1 -- 172.21.15.195:0/3281169789 shutdown_connections 2022-06-27T12:12:31.640 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.588+0000 7fbec3510700 1 -- 172.21.15.195:0/3281169789 wait complete. 2022-06-27T12:12:31.645 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.589+0000 7fbec3510700 1 Processor -- start 2022-06-27T12:12:31.646 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.589+0000 7fbec3510700 1 -- start start 2022-06-27T12:12:31.646 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.590+0000 7fbec3510700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbebc06e850 0x7fbebc1531e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:31.647 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.590+0000 7fbec3510700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbebc0c34a0 0x7fbebc158a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:31.647 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.590+0000 7fbec3510700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbebc1502a0 0x7fbebc150710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:31.648 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.590+0000 7fbec3510700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbebc1653f0 con 0x7fbebc1502a0 2022-06-27T12:12:31.648 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.590+0000 7fbec3510700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbebc0c8580 con 0x7fbebc06e850 2022-06-27T12:12:31.648 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.590+0000 7fbec3510700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbebc150c50 con 0x7fbebc0c34a0 2022-06-27T12:12:31.649 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.591+0000 7fbec1aad700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbebc1502a0 0x7fbebc150710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:31.649 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.591+0000 7fbec1aad700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbebc1502a0 0x7fbebc150710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50722/0 (socket says 172.21.15.195:50722) 2022-06-27T12:12:31.650 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.591+0000 7fbec1aad700 1 -- 172.21.15.195:0/1131058865 learned_addr learned my addr 172.21.15.195:0/1131058865 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:12:31.650 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.591+0000 7fbec0aab700 1 --2- 172.21.15.195:0/1131058865 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbebc0c34a0 0x7fbebc158a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:31.651 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.591+0000 7fbec12ac700 1 --2- 172.21.15.195:0/1131058865 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbebc06e850 0x7fbebc1531e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:31.651 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.591+0000 7fbec1aad700 1 -- 172.21.15.195:0/1131058865 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbebc0c34a0 msgr2=0x7fbebc158a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:31.652 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.591+0000 7fbec1aad700 1 --2- 172.21.15.195:0/1131058865 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbebc0c34a0 0x7fbebc158a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:31.652 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.591+0000 7fbec1aad700 1 -- 172.21.15.195:0/1131058865 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbebc06e850 msgr2=0x7fbebc1531e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:31.652 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.591+0000 7fbec1aad700 1 --2- 172.21.15.195:0/1131058865 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbebc06e850 0x7fbebc1531e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:31.653 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.591+0000 7fbec1aad700 1 -- 172.21.15.195:0/1131058865 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeb80093a0 con 0x7fbebc1502a0 2022-06-27T12:12:31.653 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.591+0000 7fbec1aad700 1 --2- 172.21.15.195:0/1131058865 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbebc1502a0 0x7fbebc150710 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fbeb4009ec0 tx=0x7fbeb400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:12:31.659 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.591+0000 7fbead7fa700 1 -- 172.21.15.195:0/1131058865 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeb400ccb0 con 0x7fbebc1502a0 2022-06-27T12:12:31.659 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.591+0000 7fbead7fa700 1 -- 172.21.15.195:0/1131058865 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbeb400f040 con 0x7fbebc1502a0 2022-06-27T12:12:31.660 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.591+0000 7fbead7fa700 1 -- 172.21.15.195:0/1131058865 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeb4014480 con 0x7fbebc1502a0 2022-06-27T12:12:31.660 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.591+0000 7fbec3510700 1 -- 172.21.15.195:0/1131058865 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbebc164590 con 0x7fbebc1502a0 2022-06-27T12:12:31.660 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.591+0000 7fbec3510700 1 -- 172.21.15.195:0/1131058865 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbebc164b50 con 0x7fbebc1502a0 2022-06-27T12:12:31.660 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.592+0000 7fbec3510700 1 -- 172.21.15.195:0/1131058865 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbebc0722c0 con 0x7fbebc1502a0 2022-06-27T12:12:31.661 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.593+0000 7fbead7fa700 1 -- 172.21.15.195:0/1131058865 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fbeb4014830 con 0x7fbebc1502a0 2022-06-27T12:12:31.661 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.593+0000 7fbead7fa700 1 --2- 172.21.15.195:0/1131058865 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbe9c063d00 0x7fbe9c0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:31.661 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.593+0000 7fbead7fa700 1 -- 172.21.15.195:0/1131058865 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7406+0+0 (secure 0 0 0) 0x7fbeb4081b50 con 0x7fbebc1502a0 2022-06-27T12:12:31.661 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.593+0000 7fbec12ac700 1 --2- 172.21.15.195:0/1131058865 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbe9c063d00 0x7fbe9c0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:31.662 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.593+0000 7fbec12ac700 1 --2- 172.21.15.195:0/1131058865 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbe9c063d00 0x7fbe9c0661b0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fbeb0001670 tx=0x7fbeb0009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:12:31.662 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.597+0000 7fbead7fa700 1 -- 172.21.15.195:0/1131058865 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fbeb4050200 con 0x7fbebc1502a0 2022-06-27T12:12:31.839 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.819+0000 7fbec3510700 1 -- 172.21.15.195:0/1131058865 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbebc1539e0 con 0x7fbebc1502a0 2022-06-27T12:12:31.839 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.820+0000 7fbead7fa700 1 -- 172.21.15.195:0/1131058865 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+6128 (secure 0 0 0) 0x7fbeb4018090 con 0x7fbebc1502a0 2022-06-27T12:12:31.840 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:12:31.841 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":96,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":6865,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/2349973547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2349973547},{"type":"v1","addr":"172.21.15.202:6835","nonce":2349973547}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:11:57.156947+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6050},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6050":{"gid":6050,"name":"b","rank":0,"incarnation":91,"state":"up:active","state_seq":15,"addr":"172.21.15.202:6833/3890799919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3890799919},{"type":"v1","addr":"172.21.15.202:6833","nonce":3890799919}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:12:31.855 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.822+0000 7fbec3510700 1 -- 172.21.15.195:0/1131058865 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbe9c063d00 msgr2=0x7fbe9c0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:31.856 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.822+0000 7fbec3510700 1 --2- 172.21.15.195:0/1131058865 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbe9c063d00 0x7fbe9c0661b0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fbeb0001670 tx=0x7fbeb0009040 comp rx=0 tx=0).stop 2022-06-27T12:12:31.857 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.822+0000 7fbec3510700 1 -- 172.21.15.195:0/1131058865 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbebc1502a0 msgr2=0x7fbebc150710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:31.857 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.822+0000 7fbec3510700 1 --2- 172.21.15.195:0/1131058865 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbebc1502a0 0x7fbebc150710 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fbeb4009ec0 tx=0x7fbeb400b040 comp rx=0 tx=0).stop 2022-06-27T12:12:31.857 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.822+0000 7fbec3510700 1 -- 172.21.15.195:0/1131058865 shutdown_connections 2022-06-27T12:12:31.858 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.822+0000 7fbec3510700 1 --2- 172.21.15.195:0/1131058865 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbe9c063d00 0x7fbe9c0661b0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:31.858 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.822+0000 7fbec3510700 1 --2- 172.21.15.195:0/1131058865 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbebc1502a0 0x7fbebc150710 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:31.859 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.822+0000 7fbec3510700 1 --2- 172.21.15.195:0/1131058865 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbebc06e850 0x7fbebc1531e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:31.859 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.822+0000 7fbec3510700 1 --2- 172.21.15.195:0/1131058865 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbebc0c34a0 0x7fbebc158a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:31.860 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.822+0000 7fbec3510700 1 -- 172.21.15.195:0/1131058865 >> 172.21.15.195:0/1131058865 conn(0x7fbebc0c6350 msgr2=0x7fbebc156500 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:12:31.860 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.822+0000 7fbec3510700 1 -- 172.21.15.195:0/1131058865 shutdown_connections 2022-06-27T12:12:31.861 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:31.822+0000 7fbec3510700 1 -- 172.21.15.195:0/1131058865 wait complete. 2022-06-27T12:12:31.861 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 96 2022-06-27T12:12:31.862 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2022-06-27T12:12:33.124 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:12:33.161 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:12:33.865 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:12:34.013 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.000+0000 7f5dcf161700 1 Processor -- start 2022-06-27T12:12:34.014 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.000+0000 7f5dcf161700 1 -- start start 2022-06-27T12:12:34.015 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.001+0000 7f5dcf161700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5dc8150950 0x7f5dc8156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:34.016 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.001+0000 7f5dcf161700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f5dc806f620 0x7f5dc806d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:34.017 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.001+0000 7f5dcf161700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dc80c76b0 con 0x7f5dc8150580 2022-06-27T12:12:34.028 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.001+0000 7f5dcf161700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dc8157430 con 0x7f5dc806f620 2022-06-27T12:12:34.028 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.001+0000 7f5dcf161700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc80c8250 con 0x7f5dc8150950 2022-06-27T12:12:34.028 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.001+0000 7f5dc7fff700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5dc8150950 0x7f5dc8156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:34.029 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.001+0000 7f5dcd6fe700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f5dc806f620 0x7f5dc806d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:34.029 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.001+0000 7f5dc7fff700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5dc8150950 0x7f5dc8156ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3301/0 says I am v2:172.21.15.195:43482/0 (socket says 172.21.15.195:43482) 2022-06-27T12:12:34.030 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.001+0000 7f5dc7fff700 1 -- 172.21.15.195:0/3325047103 learned_addr learned my addr 172.21.15.195:0/3325047103 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:12:34.030 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.001+0000 7f5dc7fff700 1 -- 172.21.15.195:0/3325047103 >> v2:172.21.15.202:3300/0 conn(0x7f5dc806f620 msgr2=0x7f5dc806d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:12:34.030 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.001+0000 7f5dc7fff700 1 --2- 172.21.15.195:0/3325047103 >> v2:172.21.15.202:3300/0 conn(0x7f5dc806f620 0x7f5dc806d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:34.031 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.001+0000 7f5dc7fff700 1 -- 172.21.15.195:0/3325047103 >> v1:172.21.15.202:6789/0 conn(0x7f5dc8150580 legacy=0x7f5dc80c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:34.031 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.001+0000 7f5dc7fff700 1 -- 172.21.15.195:0/3325047103 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dc80bad70 con 0x7f5dc8150950 2022-06-27T12:12:34.032 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.001+0000 7f5dcd6fe700 1 --2- 172.21.15.195:0/3325047103 >> v2:172.21.15.202:3300/0 conn(0x7f5dc806f620 0x7f5dc806d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:12:34.032 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.001+0000 7f5dc7fff700 1 --2- 172.21.15.195:0/3325047103 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5dc8150950 0x7f5dc8156ef0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f5db4009860 tx=0x7f5db4009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=875581a2600fb530 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:12:34.033 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.002+0000 7f5dc77fe700 1 -- 172.21.15.195:0/3325047103 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5db4011630 con 0x7f5dc8150950 2022-06-27T12:12:34.033 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.002+0000 7f5dc77fe700 1 -- 172.21.15.195:0/3325047103 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5db4011790 con 0x7f5dc8150950 2022-06-27T12:12:34.046 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.002+0000 7f5dc77fe700 1 -- 172.21.15.195:0/3325047103 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5db4011b30 con 0x7f5dc8150950 2022-06-27T12:12:34.047 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.002+0000 7f5dcf161700 1 -- 172.21.15.195:0/3325047103 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5dc8150950 msgr2=0x7f5dc8156ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:34.048 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.002+0000 7f5dcf161700 1 --2- 172.21.15.195:0/3325047103 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5dc8150950 0x7f5dc8156ef0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f5db4009860 tx=0x7f5db4009120 comp rx=0 tx=0).stop 2022-06-27T12:12:34.048 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.002+0000 7f5dcf161700 1 -- 172.21.15.195:0/3325047103 shutdown_connections 2022-06-27T12:12:34.049 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.002+0000 7f5dcf161700 1 --2- 172.21.15.195:0/3325047103 >> v2:172.21.15.202:3300/0 conn(0x7f5dc806f620 0x7f5dc806d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:34.050 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.002+0000 7f5dcf161700 1 --2- 172.21.15.195:0/3325047103 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5dc8150950 0x7f5dc8156ef0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:34.050 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.002+0000 7f5dcf161700 1 -- 172.21.15.195:0/3325047103 >> 172.21.15.195:0/3325047103 conn(0x7f5dc8063af0 msgr2=0x7f5dc8063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:12:34.051 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.002+0000 7f5dcf161700 1 -- 172.21.15.195:0/3325047103 shutdown_connections 2022-06-27T12:12:34.051 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.002+0000 7f5dcf161700 1 -- 172.21.15.195:0/3325047103 wait complete. 2022-06-27T12:12:34.052 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.003+0000 7f5dcf161700 1 Processor -- start 2022-06-27T12:12:34.052 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.004+0000 7f5dcf161700 1 -- start start 2022-06-27T12:12:34.053 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.004+0000 7f5dcf161700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5dc806f620 0x7f5dc80c9820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:34.053 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.004+0000 7f5dcf161700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5dc8150580 0x7f5dc80c1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:34.054 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.004+0000 7f5dcf161700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5dc80c31c0 0x7f5dc81474b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:34.054 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.004+0000 7f5dcf161700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc80bb9f0 con 0x7f5dc806f620 2022-06-27T12:12:34.054 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.004+0000 7f5dcf161700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc80bb500 con 0x7f5dc80c31c0 2022-06-27T12:12:34.055 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.004+0000 7f5dcf161700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc8166e00 con 0x7f5dc8150580 2022-06-27T12:12:34.055 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.004+0000 7f5dccefd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5dc806f620 0x7f5dc80c9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:34.056 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.004+0000 7f5dccefd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5dc806f620 0x7f5dc80c9820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50730/0 (socket says 172.21.15.195:50730) 2022-06-27T12:12:34.057 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.004+0000 7f5dccefd700 1 -- 172.21.15.195:0/3720182071 learned_addr learned my addr 172.21.15.195:0/3720182071 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:12:34.057 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.005+0000 7f5dc7fff700 1 --2- 172.21.15.195:0/3720182071 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5dc8150580 0x7f5dc80c1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:34.063 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.005+0000 7f5dcd6fe700 1 --2- 172.21.15.195:0/3720182071 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5dc80c31c0 0x7f5dc81474b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:34.064 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.005+0000 7f5dccefd700 1 -- 172.21.15.195:0/3720182071 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5dc8150580 msgr2=0x7f5dc80c1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:34.066 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.005+0000 7f5dccefd700 1 --2- 172.21.15.195:0/3720182071 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5dc8150580 0x7f5dc80c1ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:34.067 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.005+0000 7f5dccefd700 1 -- 172.21.15.195:0/3720182071 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5dc80c31c0 msgr2=0x7f5dc81474b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:34.067 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.005+0000 7f5dccefd700 1 --2- 172.21.15.195:0/3720182071 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5dc80c31c0 0x7f5dc81474b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:34.067 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.005+0000 7f5dccefd700 1 -- 172.21.15.195:0/3720182071 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dc0009a20 con 0x7f5dc806f620 2022-06-27T12:12:34.068 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.005+0000 7f5dccefd700 1 --2- 172.21.15.195:0/3720182071 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5dc806f620 0x7f5dc80c9820 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc00c490 tx=0x7f5dbc007e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:12:34.068 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.005+0000 7f5dc4ff9700 1 -- 172.21.15.195:0/3720182071 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dbc0050d0 con 0x7f5dc806f620 2022-06-27T12:12:34.068 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.005+0000 7f5dc4ff9700 1 -- 172.21.15.195:0/3720182071 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dbc005230 con 0x7f5dc806f620 2022-06-27T12:12:34.068 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.005+0000 7f5dc4ff9700 1 -- 172.21.15.195:0/3720182071 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dbc0117a0 con 0x7f5dc806f620 2022-06-27T12:12:34.069 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.005+0000 7f5dcf161700 1 -- 172.21.15.195:0/3720182071 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5db40093a0 con 0x7f5dc806f620 2022-06-27T12:12:34.069 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.005+0000 7f5dcf161700 1 -- 172.21.15.195:0/3720182071 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dc8167010 con 0x7f5dc806f620 2022-06-27T12:12:34.078 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.006+0000 7f5dcf161700 1 -- 172.21.15.195:0/3720182071 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dc80722c0 con 0x7f5dc806f620 2022-06-27T12:12:34.079 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.007+0000 7f5dc4ff9700 1 -- 172.21.15.195:0/3720182071 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f5dbc019030 con 0x7f5dc806f620 2022-06-27T12:12:34.079 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.007+0000 7f5dc4ff9700 1 --2- 172.21.15.195:0/3720182071 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5db0063d30 0x7f5db00661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:34.079 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.007+0000 7f5dc4ff9700 1 -- 172.21.15.195:0/3720182071 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7406+0+0 (secure 0 0 0) 0x7f5dbc07fda0 con 0x7f5dc806f620 2022-06-27T12:12:34.079 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.007+0000 7f5dc7fff700 1 --2- 172.21.15.195:0/3720182071 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5db0063d30 0x7f5db00661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:34.080 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.007+0000 7f5dc7fff700 1 --2- 172.21.15.195:0/3720182071 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5db0063d30 0x7f5db00661e0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f5dc806b1c0 tx=0x7f5db4017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:12:34.080 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.011+0000 7f5dc4ff9700 1 -- 172.21.15.195:0/3720182071 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f5dbc049c90 con 0x7f5dc806f620 2022-06-27T12:12:34.251 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.241+0000 7f5dcf161700 1 -- 172.21.15.195:0/3720182071 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5dc80c7400 con 0x7f5dc806f620 2022-06-27T12:12:34.251 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.241+0000 7f5dc4ff9700 1 -- 172.21.15.195:0/3720182071 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+6128 (secure 0 0 0) 0x7f5dbc017020 con 0x7f5dc806f620 2022-06-27T12:12:34.254 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:12:34.255 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":96,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":6865,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/2349973547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2349973547},{"type":"v1","addr":"172.21.15.202:6835","nonce":2349973547}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:11:57.156947+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6050},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6050":{"gid":6050,"name":"b","rank":0,"incarnation":91,"state":"up:active","state_seq":15,"addr":"172.21.15.202:6833/3890799919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3890799919},{"type":"v1","addr":"172.21.15.202:6833","nonce":3890799919}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:12:34.259 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.243+0000 7f5dcf161700 1 -- 172.21.15.195:0/3720182071 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5db0063d30 msgr2=0x7f5db00661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:34.261 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.243+0000 7f5dcf161700 1 --2- 172.21.15.195:0/3720182071 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5db0063d30 0x7f5db00661e0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f5dc806b1c0 tx=0x7f5db4017000 comp rx=0 tx=0).stop 2022-06-27T12:12:34.262 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.243+0000 7f5dcf161700 1 -- 172.21.15.195:0/3720182071 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5dc806f620 msgr2=0x7f5dc80c9820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:34.262 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.243+0000 7f5dcf161700 1 --2- 172.21.15.195:0/3720182071 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5dc806f620 0x7f5dc80c9820 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc00c490 tx=0x7f5dbc007e40 comp rx=0 tx=0).stop 2022-06-27T12:12:34.262 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.244+0000 7f5dcf161700 1 -- 172.21.15.195:0/3720182071 shutdown_connections 2022-06-27T12:12:34.263 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.244+0000 7f5dcf161700 1 --2- 172.21.15.195:0/3720182071 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5db0063d30 0x7f5db00661e0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:34.263 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.244+0000 7f5dcf161700 1 --2- 172.21.15.195:0/3720182071 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5dc806f620 0x7f5dc80c9820 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:34.263 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.244+0000 7f5dcf161700 1 --2- 172.21.15.195:0/3720182071 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5dc80c31c0 0x7f5dc81474b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:34.264 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.244+0000 7f5dcf161700 1 --2- 172.21.15.195:0/3720182071 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5dc8150580 0x7f5dc80c1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:34.264 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.244+0000 7f5dcf161700 1 -- 172.21.15.195:0/3720182071 >> 172.21.15.195:0/3720182071 conn(0x7f5dc8063af0 msgr2=0x7f5dc8156b30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:12:34.264 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.244+0000 7f5dcf161700 1 -- 172.21.15.195:0/3720182071 shutdown_connections 2022-06-27T12:12:34.265 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:34.244+0000 7f5dcf161700 1 -- 172.21.15.195:0/3720182071 wait complete. 2022-06-27T12:12:34.265 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 96 2022-06-27T12:12:34.266 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2022-06-27T12:12:36.274 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:12:36.444 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.411+0000 7f03bbb0b700 1 Processor -- start 2022-06-27T12:12:36.445 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.412+0000 7f03bbb0b700 1 -- start start 2022-06-27T12:12:36.445 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.412+0000 7f03bbb0b700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f03b406fef0 0x7f03b41570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:36.446 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.412+0000 7f03bbb0b700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f03b40c34a0 0x7f03b40c2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:36.446 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.412+0000 7f03bbb0b700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03b406fba0 con 0x7f03b406e850 2022-06-27T12:12:36.447 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.412+0000 7f03bbb0b700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03b41575e0 con 0x7f03b406fef0 2022-06-27T12:12:36.447 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.412+0000 7f03bbb0b700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03b406f5e0 con 0x7f03b40c34a0 2022-06-27T12:12:36.448 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.413+0000 7f03ba0a8700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f03b40c34a0 0x7f03b40c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:36.448 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.413+0000 7f03b90a6700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f03b406fef0 0x7f03b41570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:36.449 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.413+0000 7f03ba0a8700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f03b40c34a0 0x7f03b40c2920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3301/0 says I am v2:172.21.15.195:43502/0 (socket says 172.21.15.195:43502) 2022-06-27T12:12:36.450 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.413+0000 7f03ba0a8700 1 -- 172.21.15.195:0/863121375 learned_addr learned my addr 172.21.15.195:0/863121375 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:12:36.450 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.413+0000 7f03ba0a8700 1 -- 172.21.15.195:0/863121375 >> v2:172.21.15.202:3300/0 conn(0x7f03b406fef0 msgr2=0x7f03b41570a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:12:36.451 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.413+0000 7f03ba0a8700 1 --2- 172.21.15.195:0/863121375 >> v2:172.21.15.202:3300/0 conn(0x7f03b406fef0 0x7f03b41570a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:36.451 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.413+0000 7f03ba0a8700 1 -- 172.21.15.195:0/863121375 >> v1:172.21.15.202:6789/0 conn(0x7f03b406e850 legacy=0x7f03b4069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:36.452 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.413+0000 7f03ba0a8700 1 -- 172.21.15.195:0/863121375 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03b406f8c0 con 0x7f03b40c34a0 2022-06-27T12:12:36.452 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.413+0000 7f03b90a6700 1 --2- 172.21.15.195:0/863121375 >> v2:172.21.15.202:3300/0 conn(0x7f03b406fef0 0x7f03b41570a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:12:36.453 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.413+0000 7f03ba0a8700 1 --2- 172.21.15.195:0/863121375 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f03b40c34a0 0x7f03b40c2920 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f03ac009f20 tx=0x7f03ac009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fce2f292bbbf2d07 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:12:36.453 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.413+0000 7f03b88a5700 1 -- 172.21.15.195:0/863121375 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03ac00e070 con 0x7f03b40c34a0 2022-06-27T12:12:36.453 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.413+0000 7f03b88a5700 1 -- 172.21.15.195:0/863121375 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03ac00ba60 con 0x7f03b40c34a0 2022-06-27T12:12:36.454 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.414+0000 7f03b88a5700 1 -- 172.21.15.195:0/863121375 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03ac0092e0 con 0x7f03b40c34a0 2022-06-27T12:12:36.454 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.414+0000 7f03bbb0b700 1 -- 172.21.15.195:0/863121375 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f03b40c34a0 msgr2=0x7f03b40c2920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:36.455 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.414+0000 7f03bbb0b700 1 --2- 172.21.15.195:0/863121375 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f03b40c34a0 0x7f03b40c2920 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f03ac009f20 tx=0x7f03ac009840 comp rx=0 tx=0).stop 2022-06-27T12:12:36.455 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.414+0000 7f03bbb0b700 1 -- 172.21.15.195:0/863121375 shutdown_connections 2022-06-27T12:12:36.456 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.414+0000 7f03bbb0b700 1 --2- 172.21.15.195:0/863121375 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f03b40c34a0 0x7f03b40c2920 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:36.456 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.414+0000 7f03bbb0b700 1 --2- 172.21.15.195:0/863121375 >> v2:172.21.15.202:3300/0 conn(0x7f03b406fef0 0x7f03b41570a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:36.456 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.414+0000 7f03bbb0b700 1 -- 172.21.15.195:0/863121375 >> 172.21.15.195:0/863121375 conn(0x7f03b40c6350 msgr2=0x7f03b40c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:12:36.457 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.414+0000 7f03bbb0b700 1 -- 172.21.15.195:0/863121375 shutdown_connections 2022-06-27T12:12:36.457 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.414+0000 7f03bbb0b700 1 -- 172.21.15.195:0/863121375 wait complete. 2022-06-27T12:12:36.458 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.415+0000 7f03bbb0b700 1 Processor -- start 2022-06-27T12:12:36.459 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.415+0000 7f03bbb0b700 1 -- start start 2022-06-27T12:12:36.459 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.416+0000 7f03bbb0b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f03b406e850 0x7f03b40688a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:36.460 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.416+0000 7f03bbb0b700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f03b40c34a0 0x7f03b4152520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:36.461 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.416+0000 7f03bbb0b700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f03b414e0b0 0x7f03b414e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:36.461 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.416+0000 7f03bbb0b700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03b4155fd0 con 0x7f03b406e850 2022-06-27T12:12:36.461 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.416+0000 7f03bbb0b700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03b4068de0 con 0x7f03b40c34a0 2022-06-27T12:12:36.462 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.416+0000 7f03bbb0b700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03b4155670 con 0x7f03b414e0b0 2022-06-27T12:12:36.462 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.416+0000 7f03b98a7700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f03b406e850 0x7f03b40688a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:36.463 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.416+0000 7f03b98a7700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f03b406e850 0x7f03b40688a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50748/0 (socket says 172.21.15.195:50748) 2022-06-27T12:12:36.463 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.416+0000 7f03b98a7700 1 -- 172.21.15.195:0/4020665972 learned_addr learned my addr 172.21.15.195:0/4020665972 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:12:36.464 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.416+0000 7f03ba0a8700 1 --2- 172.21.15.195:0/4020665972 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f03b414e0b0 0x7f03b414e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:36.464 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.416+0000 7f03b90a6700 1 --2- 172.21.15.195:0/4020665972 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f03b40c34a0 0x7f03b4152520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:36.465 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.416+0000 7f03b98a7700 1 -- 172.21.15.195:0/4020665972 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f03b414e0b0 msgr2=0x7f03b414e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:36.466 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.416+0000 7f03b98a7700 1 --2- 172.21.15.195:0/4020665972 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f03b414e0b0 0x7f03b414e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:36.466 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.416+0000 7f03b98a7700 1 -- 172.21.15.195:0/4020665972 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f03b40c34a0 msgr2=0x7f03b4152520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:36.467 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.416+0000 7f03b98a7700 1 --2- 172.21.15.195:0/4020665972 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f03b40c34a0 0x7f03b4152520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:36.467 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.416+0000 7f03b98a7700 1 -- 172.21.15.195:0/4020665972 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03b00092e0 con 0x7f03b406e850 2022-06-27T12:12:36.468 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.417+0000 7f03b98a7700 1 --2- 172.21.15.195:0/4020665972 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f03b406e850 0x7f03b40688a0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f03a400c440 tx=0x7f03a4007e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:12:36.468 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.417+0000 7f03a1ffb700 1 -- 172.21.15.195:0/4020665972 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03a40050d0 con 0x7f03b406e850 2022-06-27T12:12:36.469 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.417+0000 7f03a1ffb700 1 -- 172.21.15.195:0/4020665972 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03a4005250 con 0x7f03b406e850 2022-06-27T12:12:36.469 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.417+0000 7f03a1ffb700 1 -- 172.21.15.195:0/4020665972 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03a4003750 con 0x7f03b406e850 2022-06-27T12:12:36.472 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.417+0000 7f03bbb0b700 1 -- 172.21.15.195:0/4020665972 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03ac009ae0 con 0x7f03b406e850 2022-06-27T12:12:36.472 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.417+0000 7f03bbb0b700 1 -- 172.21.15.195:0/4020665972 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03b41517b0 con 0x7f03b406e850 2022-06-27T12:12:36.473 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.418+0000 7f03bbb0b700 1 -- 172.21.15.195:0/4020665972 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03b40722c0 con 0x7f03b406e850 2022-06-27T12:12:36.474 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.418+0000 7f03a1ffb700 1 -- 172.21.15.195:0/4020665972 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f03a4019030 con 0x7f03b406e850 2022-06-27T12:12:36.474 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.419+0000 7f03a1ffb700 1 --2- 172.21.15.195:0/4020665972 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0394063e00 0x7f03940662b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:36.474 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.419+0000 7f03a1ffb700 1 -- 172.21.15.195:0/4020665972 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7406+0+0 (secure 0 0 0) 0x7f03a407fdc0 con 0x7f03b406e850 2022-06-27T12:12:36.475 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.419+0000 7f03b90a6700 1 --2- 172.21.15.195:0/4020665972 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0394063e00 0x7f03940662b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:36.475 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.419+0000 7f03b90a6700 1 --2- 172.21.15.195:0/4020665972 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0394063e00 0x7f03940662b0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f03b4069a70 tx=0x7f03b0009060 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:12:36.476 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.423+0000 7f03a1ffb700 1 -- 172.21.15.195:0/4020665972 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f03a404e400 con 0x7f03b406e850 2022-06-27T12:12:36.651 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.650+0000 7f03bbb0b700 1 -- 172.21.15.195:0/4020665972 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f03b406f240 con 0x7f03b406e850 2022-06-27T12:12:36.652 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.651+0000 7f03a1ffb700 1 -- 172.21.15.195:0/4020665972 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+6128 (secure 0 0 0) 0x7f03a4017090 con 0x7f03b406e850 2022-06-27T12:12:36.653 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:12:36.653 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":96,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":6865,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/2349973547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2349973547},{"type":"v1","addr":"172.21.15.202:6835","nonce":2349973547}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:11:57.156947+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6050},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6050":{"gid":6050,"name":"b","rank":0,"incarnation":91,"state":"up:active","state_seq":15,"addr":"172.21.15.202:6833/3890799919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3890799919},{"type":"v1","addr":"172.21.15.202:6833","nonce":3890799919}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:12:36.659 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.653+0000 7f03bbb0b700 1 -- 172.21.15.195:0/4020665972 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0394063e00 msgr2=0x7f03940662b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:36.660 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.653+0000 7f03bbb0b700 1 --2- 172.21.15.195:0/4020665972 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0394063e00 0x7f03940662b0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f03b4069a70 tx=0x7f03b0009060 comp rx=0 tx=0).stop 2022-06-27T12:12:36.660 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.653+0000 7f03bbb0b700 1 -- 172.21.15.195:0/4020665972 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f03b406e850 msgr2=0x7f03b40688a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:36.691 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.653+0000 7f03bbb0b700 1 --2- 172.21.15.195:0/4020665972 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f03b406e850 0x7f03b40688a0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f03a400c440 tx=0x7f03a4007e40 comp rx=0 tx=0).stop 2022-06-27T12:12:36.692 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.653+0000 7f03bbb0b700 1 -- 172.21.15.195:0/4020665972 shutdown_connections 2022-06-27T12:12:36.692 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.653+0000 7f03bbb0b700 1 --2- 172.21.15.195:0/4020665972 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0394063e00 0x7f03940662b0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:36.693 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.653+0000 7f03bbb0b700 1 --2- 172.21.15.195:0/4020665972 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f03b406e850 0x7f03b40688a0 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:36.693 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.653+0000 7f03bbb0b700 1 --2- 172.21.15.195:0/4020665972 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f03b414e0b0 0x7f03b414e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:36.694 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.653+0000 7f03bbb0b700 1 --2- 172.21.15.195:0/4020665972 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f03b40c34a0 0x7f03b4152520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:36.694 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.653+0000 7f03bbb0b700 1 -- 172.21.15.195:0/4020665972 >> 172.21.15.195:0/4020665972 conn(0x7f03b40c6350 msgr2=0x7f03b41504d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:12:36.695 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.653+0000 7f03bbb0b700 1 -- 172.21.15.195:0/4020665972 shutdown_connections 2022-06-27T12:12:36.695 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:36.653+0000 7f03bbb0b700 1 -- 172.21.15.195:0/4020665972 wait complete. 2022-06-27T12:12:36.696 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 96 2022-06-27T12:12:36.699 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2022-06-27T12:12:38.704 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:12:38.840 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:38.837+0000 7f342940f700 1 Processor -- start 2022-06-27T12:12:38.841 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:38.837+0000 7f342940f700 1 -- start start 2022-06-27T12:12:38.841 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:38.838+0000 7f342940f700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f3424068f80 0x7f34240c4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:38.860 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:38.838+0000 7f342940f700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34241573b0 con 0x7f34240c7480 2022-06-27T12:12:38.861 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:38.838+0000 7f342940f700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34240c46c0 con 0x7f34240683c0 2022-06-27T12:12:38.861 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:38.838+0000 7f342940f700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3424068840 con 0x7f3424068f80 2022-06-27T12:12:38.869 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:38.838+0000 7f34227fc700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f34240c7480 0x7f3424156460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:56946/0 (socket says 172.21.15.195:56946) 2022-06-27T12:12:38.871 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:38.838+0000 7f34227fc700 1 -- 172.21.15.195:0/3420014193 learned_addr learned my addr 172.21.15.195:0/3420014193 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:12:38.871 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:38.838+0000 7f3422ffd700 1 --2- 172.21.15.195:0/3420014193 >> v2:172.21.15.202:3300/0 conn(0x7f3424068f80 0x7f34240c4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:38.893 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:38.838+0000 7f3421ffb700 1 -- 172.21.15.195:0/3420014193 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1922501444 0 0) 0x7f34241573b0 con 0x7f34240c7480 2022-06-27T12:12:38.896 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:38.838+0000 7f3421ffb700 1 -- 172.21.15.195:0/3420014193 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3410003430 con 0x7f34240c7480 2022-06-27T12:12:38.908 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:38.838+0000 7f3422ffd700 1 -- 172.21.15.195:0/3420014193 >> v1:172.21.15.202:6789/0 conn(0x7f34240683c0 legacy=0x7f34240c6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:38.909 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:38.839+0000 7f3422ffd700 1 -- 172.21.15.195:0/3420014193 >> v1:172.21.15.195:6789/0 conn(0x7f34240c7480 legacy=0x7f3424156460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:38.909 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:38.839+0000 7f3422ffd700 1 -- 172.21.15.195:0/3420014193 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3424156b70 con 0x7f3424068f80 2022-06-27T12:12:38.910 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:38.839+0000 7f3422ffd700 1 --2- 172.21.15.195:0/3420014193 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3424068f80 0x7f34240c4b50 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f34180097a0 tx=0x7f3418009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=481e774b6fdb6ee6 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:12:38.910 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:38.839+0000 7f3421ffb700 1 -- 172.21.15.195:0/3420014193 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3418011630 con 0x7f3424068f80 2022-06-27T12:12:38.911 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:38.839+0000 7f3421ffb700 1 -- 172.21.15.195:0/3420014193 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3418011790 con 0x7f3424068f80 2022-06-27T12:12:38.911 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:38.839+0000 7f3421ffb700 1 -- 172.21.15.195:0/3420014193 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3418011b10 con 0x7f3424068f80 2022-06-27T12:12:38.912 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:38.839+0000 7f342940f700 1 -- 172.21.15.195:0/3420014193 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3424068f80 msgr2=0x7f34240c4b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:38.913 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:38.839+0000 7f342940f700 1 --2- 172.21.15.195:0/3420014193 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3424068f80 0x7f34240c4b50 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f34180097a0 tx=0x7f3418009060 comp rx=0 tx=0).stop 2022-06-27T12:12:38.913 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:38.839+0000 7f342940f700 1 -- 172.21.15.195:0/3420014193 shutdown_connections 2022-06-27T12:12:38.914 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:38.839+0000 7f342940f700 1 --2- 172.21.15.195:0/3420014193 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3424068f80 0x7f34240c4b50 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:38.921 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:38.839+0000 7f342940f700 1 -- 172.21.15.195:0/3420014193 >> 172.21.15.195:0/3420014193 conn(0x7f34240ccbd0 msgr2=0x7f34240ccfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:12:38.921 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:38.839+0000 7f342940f700 1 -- 172.21.15.195:0/3420014193 shutdown_connections 2022-06-27T12:12:38.923 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:38.839+0000 7f342940f700 1 -- 172.21.15.195:0/3420014193 wait complete. 2022-06-27T12:12:38.924 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:38.841+0000 7f342940f700 1 Processor -- start 2022-06-27T12:12:38.924 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:38.841+0000 7f342940f700 1 -- start start 2022-06-27T12:12:38.925 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:38.841+0000 7f342940f700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f34240683c0 0x7f342415b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:38.925 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:38.841+0000 7f342940f700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3424068f80 0x7f3424150da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:38.926 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:38.841+0000 7f342940f700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f34240c7480 0x7f342414dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:38.926 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:38.841+0000 7f342940f700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3424157630 con 0x7f34240683c0 2022-06-27T12:12:38.927 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:38.841+0000 7f342940f700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3424158170 con 0x7f34240c7480 2022-06-27T12:12:38.927 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:38.841+0000 7f342940f700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f342416a560 con 0x7f3424068f80 2022-06-27T12:12:38.928 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:38.842+0000 7f3422ffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f34240683c0 0x7f342415b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:38.928 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:38.842+0000 7f3422ffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f34240683c0 0x7f342415b190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50764/0 (socket says 172.21.15.195:50764) 2022-06-27T12:12:38.929 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:38.842+0000 7f3422ffd700 1 -- 172.21.15.195:0/1888221647 learned_addr learned my addr 172.21.15.195:0/1888221647 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:12:38.929 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:38.842+0000 7f34227fc700 1 --2- 172.21.15.195:0/1888221647 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3424068f80 0x7f3424150da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:38.930 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:38.842+0000 7f34237fe700 1 --2- 172.21.15.195:0/1888221647 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f34240c7480 0x7f342414dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:38.930 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:38.842+0000 7f3422ffd700 1 -- 172.21.15.195:0/1888221647 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3424068f80 msgr2=0x7f3424150da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:38.931 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:38.842+0000 7f3422ffd700 1 --2- 172.21.15.195:0/1888221647 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3424068f80 0x7f3424150da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:38.931 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:38.842+0000 7f3422ffd700 1 -- 172.21.15.195:0/1888221647 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f34240c7480 msgr2=0x7f342414dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:38.931 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:38.842+0000 7f3422ffd700 1 --2- 172.21.15.195:0/1888221647 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f34240c7480 0x7f342414dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:38.932 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:38.842+0000 7f3422ffd700 1 -- 172.21.15.195:0/1888221647 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34180092e0 con 0x7f34240683c0 2022-06-27T12:12:38.933 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:38.842+0000 7f34237fe700 1 --2- 172.21.15.195:0/1888221647 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f34240c7480 0x7f342414dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:12:38.934 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:38.842+0000 7f34227fc700 1 --2- 172.21.15.195:0/1888221647 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3424068f80 0x7f3424150da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:12:38.934 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:38.842+0000 7f3422ffd700 1 --2- 172.21.15.195:0/1888221647 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f34240683c0 0x7f342415b190 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f3418009e40 tx=0x7f3418017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:12:38.935 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:38.842+0000 7f340affd700 1 -- 172.21.15.195:0/1888221647 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3418000c40 con 0x7f34240683c0 2022-06-27T12:12:38.940 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:38.842+0000 7f340affd700 1 -- 172.21.15.195:0/1888221647 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3418011900 con 0x7f34240683c0 2022-06-27T12:12:38.941 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:38.842+0000 7f342940f700 1 -- 172.21.15.195:0/1888221647 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34241525b0 con 0x7f34240683c0 2022-06-27T12:12:38.941 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:38.842+0000 7f340affd700 1 -- 172.21.15.195:0/1888221647 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3418025630 con 0x7f34240683c0 2022-06-27T12:12:38.942 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:38.842+0000 7f342940f700 1 -- 172.21.15.195:0/1888221647 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3424157b20 con 0x7f34240683c0 2022-06-27T12:12:38.942 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:38.844+0000 7f342940f700 1 -- 172.21.15.195:0/1888221647 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34240722c0 con 0x7f34240683c0 2022-06-27T12:12:38.943 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:38.844+0000 7f340affd700 1 -- 172.21.15.195:0/1888221647 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f341802c030 con 0x7f34240683c0 2022-06-27T12:12:38.943 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:38.844+0000 7f340affd700 1 --2- 172.21.15.195:0/1888221647 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3404063d00 0x7f34040661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:38.945 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:38.844+0000 7f340affd700 1 -- 172.21.15.195:0/1888221647 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7406+0+0 (secure 0 0 0) 0x7f3418082340 con 0x7f34240683c0 2022-06-27T12:12:38.945 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:38.844+0000 7f34227fc700 1 --2- 172.21.15.195:0/1888221647 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3404063d00 0x7f34040661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:38.946 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:38.845+0000 7f34227fc700 1 --2- 172.21.15.195:0/1888221647 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3404063d00 0x7f34040661b0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f34240b2880 tx=0x7f3414009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:12:38.946 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:38.849+0000 7f340affd700 1 -- 172.21.15.195:0/1888221647 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f3418050aa0 con 0x7f34240683c0 2022-06-27T12:12:39.075 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:39.072+0000 7f342940f700 1 -- 172.21.15.195:0/1888221647 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f34240c4c60 con 0x7f34240683c0 2022-06-27T12:12:39.075 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:39.073+0000 7f340affd700 1 -- 172.21.15.195:0/1888221647 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+6128 (secure 0 0 0) 0x7f341802a020 con 0x7f34240683c0 2022-06-27T12:12:39.076 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:12:39.077 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":96,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":6865,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/2349973547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2349973547},{"type":"v1","addr":"172.21.15.202:6835","nonce":2349973547}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:11:57.156947+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6050},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6050":{"gid":6050,"name":"b","rank":0,"incarnation":91,"state":"up:active","state_seq":15,"addr":"172.21.15.202:6833/3890799919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":3890799919},{"type":"v1","addr":"172.21.15.202:6833","nonce":3890799919}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:12:39.096 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:39.075+0000 7f342940f700 1 -- 172.21.15.195:0/1888221647 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3404063d00 msgr2=0x7f34040661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:39.097 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:39.075+0000 7f342940f700 1 --2- 172.21.15.195:0/1888221647 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3404063d00 0x7f34040661b0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f34240b2880 tx=0x7f3414009040 comp rx=0 tx=0).stop 2022-06-27T12:12:39.110 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:39.075+0000 7f342940f700 1 -- 172.21.15.195:0/1888221647 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f34240683c0 msgr2=0x7f342415b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:39.110 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:39.075+0000 7f342940f700 1 --2- 172.21.15.195:0/1888221647 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f34240683c0 0x7f342415b190 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f3418009e40 tx=0x7f3418017000 comp rx=0 tx=0).stop 2022-06-27T12:12:39.111 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:39.075+0000 7f342940f700 1 -- 172.21.15.195:0/1888221647 shutdown_connections 2022-06-27T12:12:39.111 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:39.075+0000 7f342940f700 1 --2- 172.21.15.195:0/1888221647 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3404063d00 0x7f34040661b0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:39.112 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:39.075+0000 7f342940f700 1 --2- 172.21.15.195:0/1888221647 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f34240683c0 0x7f342415b190 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:39.112 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:39.075+0000 7f342940f700 1 --2- 172.21.15.195:0/1888221647 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f34240c7480 0x7f342414dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:39.113 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:39.075+0000 7f342940f700 1 --2- 172.21.15.195:0/1888221647 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3424068f80 0x7f3424150da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:39.113 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:39.075+0000 7f342940f700 1 -- 172.21.15.195:0/1888221647 >> 172.21.15.195:0/1888221647 conn(0x7f34240ccbd0 msgr2=0x7f342415ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:12:39.113 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:39.075+0000 7f342940f700 1 -- 172.21.15.195:0/1888221647 shutdown_connections 2022-06-27T12:12:39.120 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:39.075+0000 7f342940f700 1 -- 172.21.15.195:0/1888221647 wait complete. 2022-06-27T12:12:39.120 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 96 2022-06-27T12:12:39.121 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2022-06-27T12:12:41.124 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:12:41.274 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.272+0000 7f6c1ab5e700 1 Processor -- start 2022-06-27T12:12:41.275 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.272+0000 7f6c1ab5e700 1 -- start start 2022-06-27T12:12:41.284 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.273+0000 7f6c1ab5e700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6c14150950 0x7f6c14156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:41.284 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.273+0000 7f6c1ab5e700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f6c1406f620 0x7f6c1406d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:41.284 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.273+0000 7f6c1ab5e700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c140c76b0 con 0x7f6c14150580 2022-06-27T12:12:41.285 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.273+0000 7f6c1ab5e700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c14157430 con 0x7f6c1406f620 2022-06-27T12:12:41.285 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.273+0000 7f6c1ab5e700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c140c8250 con 0x7f6c14150950 2022-06-27T12:12:41.294 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.273+0000 7f6c190fb700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f6c1406f620 0x7f6c1406d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:41.294 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.273+0000 7f6c188fa700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f6c14150580 0x7f6c140c7b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:56966/0 (socket says 172.21.15.195:56966) 2022-06-27T12:12:41.294 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.273+0000 7f6c188fa700 1 -- 172.21.15.195:0/516104236 learned_addr learned my addr 172.21.15.195:0/516104236 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:12:41.295 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.273+0000 7f6c13fff700 1 --2- 172.21.15.195:0/516104236 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6c14150950 0x7f6c14156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:41.295 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.273+0000 7f6c190fb700 1 -- 172.21.15.195:0/516104236 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6c14150950 msgr2=0x7f6c14156ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:12:41.295 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.273+0000 7f6c190fb700 1 --2- 172.21.15.195:0/516104236 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6c14150950 0x7f6c14156ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:41.295 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.273+0000 7f6c190fb700 1 -- 172.21.15.195:0/516104236 >> v1:172.21.15.195:6789/0 conn(0x7f6c14150580 legacy=0x7f6c140c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:41.295 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.273+0000 7f6c190fb700 1 -- 172.21.15.195:0/516104236 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c140bad70 con 0x7f6c1406f620 2022-06-27T12:12:41.296 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.273+0000 7f6c137fe700 1 -- 172.21.15.195:0/516104236 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3952799475 0 0) 0x7f6c140c76b0 con 0x7f6c14150580 2022-06-27T12:12:41.296 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.273+0000 7f6c190fb700 1 --2- 172.21.15.195:0/516104236 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6c1406f620 0x7f6c1406d6a0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c009e60 tx=0x7f6c0c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9160575f529180c5 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:12:41.296 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.273+0000 7f6c137fe700 1 -- 172.21.15.195:0/516104236 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c0c00e070 con 0x7f6c1406f620 2022-06-27T12:12:41.296 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.273+0000 7f6c137fe700 1 -- 172.21.15.195:0/516104236 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c0c00bc30 con 0x7f6c1406f620 2022-06-27T12:12:41.297 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.273+0000 7f6c137fe700 1 -- 172.21.15.195:0/516104236 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c0c012430 con 0x7f6c1406f620 2022-06-27T12:12:41.297 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.274+0000 7f6c1ab5e700 1 -- 172.21.15.195:0/516104236 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6c1406f620 msgr2=0x7f6c1406d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:41.315 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.274+0000 7f6c1ab5e700 1 --2- 172.21.15.195:0/516104236 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6c1406f620 0x7f6c1406d6a0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c009e60 tx=0x7f6c0c009780 comp rx=0 tx=0).stop 2022-06-27T12:12:41.316 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.274+0000 7f6c1ab5e700 1 -- 172.21.15.195:0/516104236 shutdown_connections 2022-06-27T12:12:41.317 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.274+0000 7f6c1ab5e700 1 --2- 172.21.15.195:0/516104236 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6c1406f620 0x7f6c1406d6a0 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:41.317 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.274+0000 7f6c1ab5e700 1 --2- 172.21.15.195:0/516104236 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6c14150950 0x7f6c14156ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:41.317 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.274+0000 7f6c1ab5e700 1 -- 172.21.15.195:0/516104236 >> 172.21.15.195:0/516104236 conn(0x7f6c14063af0 msgr2=0x7f6c14063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:12:41.318 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.274+0000 7f6c1ab5e700 1 -- 172.21.15.195:0/516104236 shutdown_connections 2022-06-27T12:12:41.318 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.274+0000 7f6c1ab5e700 1 -- 172.21.15.195:0/516104236 wait complete. 2022-06-27T12:12:41.319 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.276+0000 7f6c1ab5e700 1 Processor -- start 2022-06-27T12:12:41.319 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.276+0000 7f6c1ab5e700 1 -- start start 2022-06-27T12:12:41.320 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.277+0000 7f6c1ab5e700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6c1406f620 0x7f6c14155240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:41.320 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.277+0000 7f6c1ab5e700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6c14150580 0x7f6c141515c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:41.321 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.277+0000 7f6c1ab5e700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6c14150950 0x7f6c14151b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:41.321 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.277+0000 7f6c1ab5e700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c14167880 con 0x7f6c1406f620 2022-06-27T12:12:41.322 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.277+0000 7f6c1ab5e700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c14151130 con 0x7f6c14150580 2022-06-27T12:12:41.323 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.277+0000 7f6c1ab5e700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c14150de0 con 0x7f6c14150950 2022-06-27T12:12:41.323 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.277+0000 7f6c188fa700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6c1406f620 0x7f6c14155240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:41.323 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.277+0000 7f6c188fa700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6c1406f620 0x7f6c14155240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50784/0 (socket says 172.21.15.195:50784) 2022-06-27T12:12:41.324 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.277+0000 7f6c188fa700 1 -- 172.21.15.195:0/2137539803 learned_addr learned my addr 172.21.15.195:0/2137539803 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:12:41.324 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.277+0000 7f6c190fb700 1 --2- 172.21.15.195:0/2137539803 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6c14150950 0x7f6c14151b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:41.325 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.277+0000 7f6c13fff700 1 --2- 172.21.15.195:0/2137539803 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6c14150580 0x7f6c141515c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:41.325 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.277+0000 7f6c188fa700 1 -- 172.21.15.195:0/2137539803 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6c14150950 msgr2=0x7f6c14151b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:41.327 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.277+0000 7f6c188fa700 1 --2- 172.21.15.195:0/2137539803 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6c14150950 0x7f6c14151b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:41.327 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.277+0000 7f6c188fa700 1 -- 172.21.15.195:0/2137539803 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6c14150580 msgr2=0x7f6c141515c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:41.328 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.277+0000 7f6c188fa700 1 --2- 172.21.15.195:0/2137539803 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6c14150580 0x7f6c141515c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:41.328 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.277+0000 7f6c188fa700 1 -- 172.21.15.195:0/2137539803 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c0c009a20 con 0x7f6c1406f620 2022-06-27T12:12:41.329 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.277+0000 7f6c188fa700 1 --2- 172.21.15.195:0/2137539803 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6c1406f620 0x7f6c14155240 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f6c0800c5c0 tx=0x7f6c08007cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:12:41.329 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.277+0000 7f6c10ff9700 1 -- 172.21.15.195:0/2137539803 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c08007360 con 0x7f6c1406f620 2022-06-27T12:12:41.330 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.277+0000 7f6c10ff9700 1 -- 172.21.15.195:0/2137539803 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c08007500 con 0x7f6c1406f620 2022-06-27T12:12:41.330 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.277+0000 7f6c1ab5e700 1 -- 172.21.15.195:0/2137539803 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c140bdb50 con 0x7f6c1406f620 2022-06-27T12:12:41.331 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.277+0000 7f6c10ff9700 1 -- 172.21.15.195:0/2137539803 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c08004940 con 0x7f6c1406f620 2022-06-27T12:12:41.331 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.277+0000 7f6c1ab5e700 1 -- 172.21.15.195:0/2137539803 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c140be110 con 0x7f6c1406f620 2022-06-27T12:12:41.332 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.278+0000 7f6c1ab5e700 1 -- 172.21.15.195:0/2137539803 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c140722c0 con 0x7f6c1406f620 2022-06-27T12:12:41.332 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.279+0000 7f6c10ff9700 1 -- 172.21.15.195:0/2137539803 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f6c08010030 con 0x7f6c1406f620 2022-06-27T12:12:41.333 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.279+0000 7f6c10ff9700 1 --2- 172.21.15.195:0/2137539803 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6bfc063d30 0x7f6bfc0661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:41.333 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.279+0000 7f6c10ff9700 1 -- 172.21.15.195:0/2137539803 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7492+0+0 (secure 0 0 0) 0x7f6c08080840 con 0x7f6c1406f620 2022-06-27T12:12:41.334 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.279+0000 7f6c13fff700 1 --2- 172.21.15.195:0/2137539803 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6bfc063d30 0x7f6bfc0661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:41.335 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.280+0000 7f6c13fff700 1 --2- 172.21.15.195:0/2137539803 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6bfc063d30 0x7f6bfc0661e0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f6c00004560 tx=0x7f6c00008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:12:41.335 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.283+0000 7f6c10ff9700 1 -- 172.21.15.195:0/2137539803 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f6c0804ef10 con 0x7f6c1406f620 2022-06-27T12:12:41.516 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.511+0000 7f6c1ab5e700 1 -- 172.21.15.195:0/2137539803 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6c1414df20 con 0x7f6c1406f620 2022-06-27T12:12:41.517 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.511+0000 7f6c10ff9700 1 -- 172.21.15.195:0/2137539803 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+5367 (secure 0 0 0) 0x7f6c08052530 con 0x7f6c1406f620 2022-06-27T12:12:41.518 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:12:41.519 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:12:40.165257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6865},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6865":{"gid":6865,"name":"d","rank":0,"incarnation":97,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6835/2349973547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2349973547},{"type":"v1","addr":"172.21.15.202:6835","nonce":2349973547}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:12:41.521 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.513+0000 7f6c1ab5e700 1 -- 172.21.15.195:0/2137539803 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6bfc063d30 msgr2=0x7f6bfc0661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:41.522 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.513+0000 7f6c1ab5e700 1 --2- 172.21.15.195:0/2137539803 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6bfc063d30 0x7f6bfc0661e0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f6c00004560 tx=0x7f6c00008040 comp rx=0 tx=0).stop 2022-06-27T12:12:41.522 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.513+0000 7f6c1ab5e700 1 -- 172.21.15.195:0/2137539803 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6c1406f620 msgr2=0x7f6c14155240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:41.523 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.513+0000 7f6c1ab5e700 1 --2- 172.21.15.195:0/2137539803 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6c1406f620 0x7f6c14155240 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f6c0800c5c0 tx=0x7f6c08007cb0 comp rx=0 tx=0).stop 2022-06-27T12:12:41.523 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.513+0000 7f6c1ab5e700 1 -- 172.21.15.195:0/2137539803 shutdown_connections 2022-06-27T12:12:41.523 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.513+0000 7f6c1ab5e700 1 --2- 172.21.15.195:0/2137539803 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6bfc063d30 0x7f6bfc0661e0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:41.524 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.513+0000 7f6c1ab5e700 1 --2- 172.21.15.195:0/2137539803 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6c1406f620 0x7f6c14155240 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:41.524 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.513+0000 7f6c1ab5e700 1 --2- 172.21.15.195:0/2137539803 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6c14150950 0x7f6c14151b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:41.525 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.513+0000 7f6c1ab5e700 1 --2- 172.21.15.195:0/2137539803 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6c14150580 0x7f6c141515c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:41.525 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.513+0000 7f6c1ab5e700 1 -- 172.21.15.195:0/2137539803 >> 172.21.15.195:0/2137539803 conn(0x7f6c14063af0 msgr2=0x7f6c1414f280 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:12:41.525 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.513+0000 7f6c1ab5e700 1 -- 172.21.15.195:0/2137539803 shutdown_connections 2022-06-27T12:12:41.526 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.513+0000 7f6c1ab5e700 1 -- 172.21.15.195:0/2137539803 wait complete. 2022-06-27T12:12:41.526 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 97 2022-06-27T12:12:41.533 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2022-06-27T12:12:41.534 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2022-06-27T12:12:41.534 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:12:41.666 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.654+0000 7f96e2100700 1 Processor -- start 2022-06-27T12:12:41.667 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.654+0000 7f96e2100700 1 -- start start 2022-06-27T12:12:41.667 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.655+0000 7f96e2100700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f96dc06a450 0x7f96dc063af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:41.667 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.655+0000 7f96e2100700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f96dc1508d0 0x7f96dc0bbf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:41.668 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.655+0000 7f96e2100700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96dc064030 con 0x7f96dc150500 2022-06-27T12:12:41.668 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.655+0000 7f96e2100700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96dc0c4160 con 0x7f96dc1508d0 2022-06-27T12:12:41.668 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.655+0000 7f96e2100700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96dc0c4440 con 0x7f96dc06a450 2022-06-27T12:12:41.669 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.655+0000 7f96db7fe700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f96dc150500 0x7f96dc0c48d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:56980/0 (socket says 172.21.15.195:56980) 2022-06-27T12:12:41.669 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.655+0000 7f96db7fe700 1 -- 172.21.15.195:0/818315552 learned_addr learned my addr 172.21.15.195:0/818315552 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:12:41.669 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.655+0000 7f96dbfff700 1 --2- 172.21.15.195:0/818315552 >> v2:172.21.15.195:3300/0 conn(0x7f96dc1508d0 0x7f96dc0bbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:41.669 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.655+0000 7f96daffd700 1 --2- 172.21.15.195:0/818315552 >> v2:172.21.15.202:3300/0 conn(0x7f96dc06a450 0x7f96dc063af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:41.670 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.656+0000 7f96da7fc700 1 -- 172.21.15.195:0/818315552 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 970586775 0 0) 0x7f96d0001290 con 0x7f96dc150500 2022-06-27T12:12:41.670 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.656+0000 7f96da7fc700 1 -- 172.21.15.195:0/818315552 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96c8003430 con 0x7f96dc150500 2022-06-27T12:12:41.670 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.656+0000 7f96dbfff700 1 -- 172.21.15.195:0/818315552 >> v2:172.21.15.202:3300/0 conn(0x7f96dc06a450 msgr2=0x7f96dc063af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:12:41.671 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.656+0000 7f96dbfff700 1 --2- 172.21.15.195:0/818315552 >> v2:172.21.15.202:3300/0 conn(0x7f96dc06a450 0x7f96dc063af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:41.671 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.656+0000 7f96dbfff700 1 -- 172.21.15.195:0/818315552 >> v1:172.21.15.195:6789/0 conn(0x7f96dc150500 legacy=0x7f96dc0c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:41.671 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.656+0000 7f96dbfff700 1 -- 172.21.15.195:0/818315552 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96dc15a800 con 0x7f96dc1508d0 2022-06-27T12:12:41.672 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.656+0000 7f96daffd700 1 --2- 172.21.15.195:0/818315552 >> v2:172.21.15.202:3300/0 conn(0x7f96dc06a450 0x7f96dc063af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:12:41.673 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.656+0000 7f96dbfff700 1 --2- 172.21.15.195:0/818315552 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f96dc1508d0 0x7f96dc0bbf50 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f96dc06ce70 tx=0x7f96d4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9151bd52558451f4 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:12:41.673 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.656+0000 7f96da7fc700 1 -- 172.21.15.195:0/818315552 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96d400e070 con 0x7f96dc1508d0 2022-06-27T12:12:41.673 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.656+0000 7f96da7fc700 1 -- 172.21.15.195:0/818315552 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96d400bb60 con 0x7f96dc1508d0 2022-06-27T12:12:41.674 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.656+0000 7f96da7fc700 1 -- 172.21.15.195:0/818315552 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96d4012430 con 0x7f96dc1508d0 2022-06-27T12:12:41.674 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.656+0000 7f96e2100700 1 -- 172.21.15.195:0/818315552 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f96dc1508d0 msgr2=0x7f96dc0bbf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:41.674 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.656+0000 7f96e2100700 1 --2- 172.21.15.195:0/818315552 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f96dc1508d0 0x7f96dc0bbf50 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f96dc06ce70 tx=0x7f96d4009780 comp rx=0 tx=0).stop 2022-06-27T12:12:41.674 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.657+0000 7f96e2100700 1 -- 172.21.15.195:0/818315552 shutdown_connections 2022-06-27T12:12:41.675 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.657+0000 7f96e2100700 1 --2- 172.21.15.195:0/818315552 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f96dc1508d0 0x7f96dc0bbf50 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:41.675 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.657+0000 7f96e2100700 1 --2- 172.21.15.195:0/818315552 >> v2:172.21.15.202:3300/0 conn(0x7f96dc06a450 0x7f96dc063af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:41.675 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.657+0000 7f96e2100700 1 -- 172.21.15.195:0/818315552 >> 172.21.15.195:0/818315552 conn(0x7f96dc0c9b00 msgr2=0x7f96dc0c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:12:41.676 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.657+0000 7f96e2100700 1 -- 172.21.15.195:0/818315552 shutdown_connections 2022-06-27T12:12:41.676 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.657+0000 7f96e2100700 1 -- 172.21.15.195:0/818315552 wait complete. 2022-06-27T12:12:41.676 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.658+0000 7f96e2100700 1 Processor -- start 2022-06-27T12:12:41.676 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.658+0000 7f96e2100700 1 -- start start 2022-06-27T12:12:41.676 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.659+0000 7f96e2100700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f96dc06a450 0x7f96dc15b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:41.677 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.659+0000 7f96e2100700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f96dc150500 0x7f96dc166550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:41.677 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.659+0000 7f96e2100700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f96dc1508d0 0x7f96dc166a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:41.677 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.659+0000 7f96e2100700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96dc154d80 con 0x7f96dc1508d0 2022-06-27T12:12:41.677 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.659+0000 7f96e2100700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96dc16a120 con 0x7f96dc150500 2022-06-27T12:12:41.678 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.659+0000 7f96e2100700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96dc1570d0 con 0x7f96dc06a450 2022-06-27T12:12:41.678 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.659+0000 7f96dbfff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f96dc1508d0 0x7f96dc166a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:41.678 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.659+0000 7f96dbfff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f96dc1508d0 0x7f96dc166a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50802/0 (socket says 172.21.15.195:50802) 2022-06-27T12:12:41.679 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.659+0000 7f96dbfff700 1 -- 172.21.15.195:0/3801289251 learned_addr learned my addr 172.21.15.195:0/3801289251 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:12:41.679 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.659+0000 7f96daffd700 1 --2- 172.21.15.195:0/3801289251 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f96dc150500 0x7f96dc166550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:41.679 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.659+0000 7f96db7fe700 1 --2- 172.21.15.195:0/3801289251 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f96dc06a450 0x7f96dc15b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:41.679 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.659+0000 7f96dbfff700 1 -- 172.21.15.195:0/3801289251 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f96dc06a450 msgr2=0x7f96dc15b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:41.680 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.659+0000 7f96dbfff700 1 --2- 172.21.15.195:0/3801289251 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f96dc06a450 0x7f96dc15b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:41.680 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.659+0000 7f96dbfff700 1 -- 172.21.15.195:0/3801289251 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f96dc150500 msgr2=0x7f96dc166550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:41.680 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.659+0000 7f96dbfff700 1 --2- 172.21.15.195:0/3801289251 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f96dc150500 0x7f96dc166550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:41.681 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.659+0000 7f96dbfff700 1 -- 172.21.15.195:0/3801289251 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96d4009a20 con 0x7f96dc1508d0 2022-06-27T12:12:41.681 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.659+0000 7f96db7fe700 1 --2- 172.21.15.195:0/3801289251 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f96dc06a450 0x7f96dc15b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:12:41.681 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.659+0000 7f96dbfff700 1 --2- 172.21.15.195:0/3801289251 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f96dc1508d0 0x7f96dc166a90 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f96dc06f810 tx=0x7f96d4019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:12:41.681 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.659+0000 7f96c37fe700 1 -- 172.21.15.195:0/3801289251 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96d4003e00 con 0x7f96dc1508d0 2022-06-27T12:12:41.682 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.659+0000 7f96c37fe700 1 -- 172.21.15.195:0/3801289251 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96d4012a00 con 0x7f96dc1508d0 2022-06-27T12:12:41.682 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.659+0000 7f96e2100700 1 -- 172.21.15.195:0/3801289251 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96dc16a5e0 con 0x7f96dc1508d0 2022-06-27T12:12:41.682 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.659+0000 7f96c37fe700 1 -- 172.21.15.195:0/3801289251 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96d4012dc0 con 0x7f96dc1508d0 2022-06-27T12:12:41.683 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.659+0000 7f96e2100700 1 -- 172.21.15.195:0/3801289251 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96dc16a300 con 0x7f96dc1508d0 2022-06-27T12:12:41.683 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.660+0000 7f96e2100700 1 -- 172.21.15.195:0/3801289251 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96dc166270 con 0x7f96dc1508d0 2022-06-27T12:12:41.683 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.661+0000 7f96c37fe700 1 -- 172.21.15.195:0/3801289251 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f96d4004250 con 0x7f96dc1508d0 2022-06-27T12:12:41.683 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.661+0000 7f96c37fe700 1 --2- 172.21.15.195:0/3801289251 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f96bc063d30 0x7f96bc0661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:41.684 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.661+0000 7f96c37fe700 1 -- 172.21.15.195:0/3801289251 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7492+0+0 (secure 0 0 0) 0x7f96d40835d0 con 0x7f96dc1508d0 2022-06-27T12:12:41.684 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.661+0000 7f96db7fe700 1 --2- 172.21.15.195:0/3801289251 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f96bc063d30 0x7f96bc0661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:41.684 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.661+0000 7f96db7fe700 1 --2- 172.21.15.195:0/3801289251 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f96bc063d30 0x7f96bc0661e0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f96d00013f0 tx=0x7f96d0009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:12:41.684 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.665+0000 7f96c37fe700 1 -- 172.21.15.195:0/3801289251 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f96dc166270 con 0x7f96dc1508d0 2022-06-27T12:12:41.895 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.894+0000 7f96e2100700 1 -- 172.21.15.195:0/3801289251 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f96dc166270 con 0x7f96dc1508d0 2022-06-27T12:12:41.896 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.895+0000 7f96c37fe700 1 -- 172.21.15.195:0/3801289251 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+5367 (secure 0 0 0) 0x7f96dc166270 con 0x7f96dc1508d0 2022-06-27T12:12:41.897 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:12:41.897 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:12:40.165257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6865},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6865":{"gid":6865,"name":"d","rank":0,"incarnation":97,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6835/2349973547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2349973547},{"type":"v1","addr":"172.21.15.202:6835","nonce":2349973547}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:12:41.900 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.897+0000 7f96e2100700 1 -- 172.21.15.195:0/3801289251 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f96bc063d30 msgr2=0x7f96bc0661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:41.900 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.897+0000 7f96e2100700 1 --2- 172.21.15.195:0/3801289251 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f96bc063d30 0x7f96bc0661e0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f96d00013f0 tx=0x7f96d0009040 comp rx=0 tx=0).stop 2022-06-27T12:12:41.901 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.897+0000 7f96e2100700 1 -- 172.21.15.195:0/3801289251 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f96dc1508d0 msgr2=0x7f96dc166a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:41.901 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.897+0000 7f96e2100700 1 --2- 172.21.15.195:0/3801289251 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f96dc1508d0 0x7f96dc166a90 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f96dc06f810 tx=0x7f96d4019000 comp rx=0 tx=0).stop 2022-06-27T12:12:41.901 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.897+0000 7f96e2100700 1 -- 172.21.15.195:0/3801289251 shutdown_connections 2022-06-27T12:12:41.903 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.897+0000 7f96e2100700 1 --2- 172.21.15.195:0/3801289251 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f96bc063d30 0x7f96bc0661e0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:41.903 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.897+0000 7f96e2100700 1 --2- 172.21.15.195:0/3801289251 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f96dc1508d0 0x7f96dc166a90 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:41.904 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.897+0000 7f96e2100700 1 --2- 172.21.15.195:0/3801289251 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f96dc06a450 0x7f96dc15b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:41.904 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.897+0000 7f96e2100700 1 --2- 172.21.15.195:0/3801289251 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f96dc150500 0x7f96dc166550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:41.904 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.897+0000 7f96e2100700 1 -- 172.21.15.195:0/3801289251 >> 172.21.15.195:0/3801289251 conn(0x7f96dc0c9b00 msgr2=0x7f96dc14fc30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:12:41.905 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.897+0000 7f96e2100700 1 -- 172.21.15.195:0/3801289251 shutdown_connections 2022-06-27T12:12:41.905 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:41.897+0000 7f96e2100700 1 -- 172.21.15.195:0/3801289251 wait complete. 2022-06-27T12:12:41.906 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 97 2022-06-27T12:12:41.920 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 97, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-06-27T11:47:52.358095+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 97, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6865": { "addr": "172.21.15.202:6835/2349973547", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6834", "nonce": 2349973547, "type": "v2" }, { "addr": "172.21.15.202:6835", "nonce": 2349973547, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 6865, "incarnation": 97, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 63, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-06-27T12:12:40.165257+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6865 } } } ], "standbys": [ { "addr": "172.21.15.202:6837/3305006841", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6836", "nonce": 3305006841, "type": "v2" }, { "addr": "172.21.15.202:6837", "nonce": 3305006841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6835/2060709822", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6834", "nonce": 2060709822, "type": "v2" }, { "addr": "172.21.15.195:6835", "nonce": 2060709822, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6838/1598751652", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6836", "nonce": 1598751652, "type": "v2" }, { "addr": "172.21.15.195:6838", "nonce": 1598751652, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4275, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6839/1313800661", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6837", "nonce": 1313800661, "type": "v2" }, { "addr": "172.21.15.195:6839", "nonce": 1313800661, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4278, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-06-27T12:12:43.922 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:12:44.077 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.058+0000 7fc6d3b4b700 1 Processor -- start 2022-06-27T12:12:44.077 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.058+0000 7fc6d3b4b700 1 -- start start 2022-06-27T12:12:44.077 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.059+0000 7fc6d3b4b700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fc6cc0688a0 0x7fc6cc0c28c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:44.078 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.059+0000 7fc6d3b4b700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6cc0c4b10 con 0x7fc6cc156100 2022-06-27T12:12:44.079 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.059+0000 7fc6d3b4b700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6cc068f20 con 0x7fc6cc06a440 2022-06-27T12:12:44.079 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.059+0000 7fc6d3b4b700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6cc0c3dc0 con 0x7fc6cc0688a0 2022-06-27T12:12:44.079 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.059+0000 7fc6d18e7700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fc6cc156100 0x7fc6cc0c4fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:56994/0 (socket says 172.21.15.195:56994) 2022-06-27T12:12:44.079 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.059+0000 7fc6d18e7700 1 -- 172.21.15.195:0/1487503771 learned_addr learned my addr 172.21.15.195:0/1487503771 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:12:44.080 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.059+0000 7fc6d20e8700 1 --2- 172.21.15.195:0/1487503771 >> v2:172.21.15.202:3300/0 conn(0x7fc6cc0688a0 0x7fc6cc0c28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:44.080 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.059+0000 7fc6d08e5700 1 -- 172.21.15.195:0/1487503771 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1576669125 0 0) 0x7fc6cc0c4b10 con 0x7fc6cc156100 2022-06-27T12:12:44.080 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.059+0000 7fc6d08e5700 1 -- 172.21.15.195:0/1487503771 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6b4003430 con 0x7fc6cc156100 2022-06-27T12:12:44.080 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.060+0000 7fc6d08e5700 1 -- 172.21.15.195:0/1487503771 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3315925523 0 0) 0x7fc6b4003430 con 0x7fc6cc156100 2022-06-27T12:12:44.081 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.060+0000 7fc6d08e5700 1 -- 172.21.15.195:0/1487503771 >> v2:172.21.15.202:3300/0 conn(0x7fc6cc0688a0 msgr2=0x7fc6cc0c28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:12:44.081 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.060+0000 7fc6d08e5700 1 --2- 172.21.15.195:0/1487503771 >> v2:172.21.15.202:3300/0 conn(0x7fc6cc0688a0 0x7fc6cc0c28c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:44.081 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.060+0000 7fc6d08e5700 1 -- 172.21.15.195:0/1487503771 >> v1:172.21.15.202:6789/0 conn(0x7fc6cc06a440 legacy=0x7fc6cc1564d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:44.081 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.060+0000 7fc6d08e5700 1 -- 172.21.15.195:0/1487503771 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6cc0c56b0 con 0x7fc6cc156100 2022-06-27T12:12:44.082 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.060+0000 7fc6d08e5700 1 -- 172.21.15.195:0/1487503771 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7fc6c80015f0 con 0x7fc6cc156100 2022-06-27T12:12:44.082 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.060+0000 7fc6d08e5700 1 -- 172.21.15.195:0/1487503771 >> v1:172.21.15.195:6789/0 conn(0x7fc6cc156100 legacy=0x7fc6cc0c4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:44.082 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.060+0000 7fc6d08e5700 1 --2- 172.21.15.195:0/1487503771 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc6b4003920 0x7fc6b4003cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:44.082 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.060+0000 7fc6d08e5700 1 --2- 172.21.15.195:0/1487503771 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc6b40046d0 0x7fc6b4004b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:44.083 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.060+0000 7fc6d08e5700 1 --2- 172.21.15.195:0/1487503771 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc6b4005080 0x7fc6b400b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:44.083 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.060+0000 7fc6d08e5700 1 -- 172.21.15.195:0/1487503771 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6cc0c3dc0 con 0x7fc6b4005080 2022-06-27T12:12:44.083 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.060+0000 7fc6d08e5700 1 -- 172.21.15.195:0/1487503771 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b400bf10 con 0x7fc6b40046d0 2022-06-27T12:12:44.083 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.060+0000 7fc6d08e5700 1 -- 172.21.15.195:0/1487503771 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b400e2e0 con 0x7fc6b4003920 2022-06-27T12:12:44.083 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.060+0000 7fc6d20e8700 1 --2- 172.21.15.195:0/1487503771 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc6b4005080 0x7fc6b400b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:44.084 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.060+0000 7fc6d10e6700 1 --2- 172.21.15.195:0/1487503771 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc6b40046d0 0x7fc6b4004b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:44.085 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.060+0000 7fc6d18e7700 1 --2- 172.21.15.195:0/1487503771 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc6b4003920 0x7fc6b4003cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:44.085 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.060+0000 7fc6d20e8700 1 -- 172.21.15.195:0/1487503771 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc6b4003920 msgr2=0x7fc6b4003cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:12:44.086 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.060+0000 7fc6d20e8700 1 --2- 172.21.15.195:0/1487503771 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc6b4003920 0x7fc6b4003cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:44.087 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.060+0000 7fc6d20e8700 1 -- 172.21.15.195:0/1487503771 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc6b40046d0 msgr2=0x7fc6b4004b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:12:44.087 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.060+0000 7fc6d20e8700 1 --2- 172.21.15.195:0/1487503771 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc6b40046d0 0x7fc6b4004b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:44.087 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.060+0000 7fc6d20e8700 1 -- 172.21.15.195:0/1487503771 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6b4003430 con 0x7fc6b4005080 2022-06-27T12:12:44.088 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.060+0000 7fc6d20e8700 1 -- 172.21.15.195:0/1487503771 reap_dead start 2022-06-27T12:12:44.088 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.060+0000 7fc6d20e8700 1 --2- 172.21.15.195:0/1487503771 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc6b4005080 0x7fc6b400b920 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fc6c400a2c0 tx=0x7fc6c400a890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0752ca0f7b91253 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:12:44.089 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.060+0000 7fc6d08e5700 1 -- 172.21.15.195:0/1487503771 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c4018070 con 0x7fc6b4005080 2022-06-27T12:12:44.089 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.060+0000 7fc6d08e5700 1 -- 172.21.15.195:0/1487503771 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6c4010040 con 0x7fc6b4005080 2022-06-27T12:12:44.089 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.060+0000 7fc6d08e5700 1 -- 172.21.15.195:0/1487503771 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c4009e20 con 0x7fc6b4005080 2022-06-27T12:12:44.089 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.061+0000 7fc6d3b4b700 1 -- 172.21.15.195:0/1487503771 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc6b4005080 msgr2=0x7fc6b400b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:44.090 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.061+0000 7fc6d3b4b700 1 --2- 172.21.15.195:0/1487503771 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc6b4005080 0x7fc6b400b920 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fc6c400a2c0 tx=0x7fc6c400a890 comp rx=0 tx=0).stop 2022-06-27T12:12:44.090 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.061+0000 7fc6d3b4b700 1 -- 172.21.15.195:0/1487503771 shutdown_connections 2022-06-27T12:12:44.091 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.061+0000 7fc6d3b4b700 1 --2- 172.21.15.195:0/1487503771 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc6b4005080 0x7fc6b400b920 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:44.091 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.061+0000 7fc6d3b4b700 1 -- 172.21.15.195:0/1487503771 >> 172.21.15.195:0/1487503771 conn(0x7fc6cc0c9b00 msgr2=0x7fc6cc0c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:12:44.092 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.061+0000 7fc6d3b4b700 1 -- 172.21.15.195:0/1487503771 shutdown_connections 2022-06-27T12:12:44.094 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.061+0000 7fc6d3b4b700 1 -- 172.21.15.195:0/1487503771 wait complete. 2022-06-27T12:12:44.094 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.062+0000 7fc6d3b4b700 1 Processor -- start 2022-06-27T12:12:44.095 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.062+0000 7fc6d3b4b700 1 -- start start 2022-06-27T12:12:44.095 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.063+0000 7fc6d3b4b700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc6b4005080 0x7fc6cc165f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:44.096 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.063+0000 7fc6d3b4b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc6cc0c8aa0 0x7fc6cc14a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:44.096 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.063+0000 7fc6d3b4b700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc6cc06b960 0x7fc6cc0bed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:44.097 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.063+0000 7fc6d3b4b700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6cc166440 con 0x7fc6cc0c8aa0 2022-06-27T12:12:44.097 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.063+0000 7fc6d3b4b700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6cc0c77a0 con 0x7fc6cc06b960 2022-06-27T12:12:44.097 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.063+0000 7fc6d3b4b700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6cc0c6ee0 con 0x7fc6b4005080 2022-06-27T12:12:44.097 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.063+0000 7fc6d10e6700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc6cc0c8aa0 0x7fc6cc14a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:44.097 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.063+0000 7fc6d10e6700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc6cc0c8aa0 0x7fc6cc14a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50818/0 (socket says 172.21.15.195:50818) 2022-06-27T12:12:44.098 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.063+0000 7fc6d10e6700 1 -- 172.21.15.195:0/852930261 learned_addr learned my addr 172.21.15.195:0/852930261 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:12:44.098 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.063+0000 7fc6d18e7700 1 --2- 172.21.15.195:0/852930261 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc6b4005080 0x7fc6cc165f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:44.099 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.063+0000 7fc6d20e8700 1 --2- 172.21.15.195:0/852930261 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc6cc06b960 0x7fc6cc0bed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:44.099 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.063+0000 7fc6d10e6700 1 -- 172.21.15.195:0/852930261 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc6b4005080 msgr2=0x7fc6cc165f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:44.100 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.063+0000 7fc6d10e6700 1 --2- 172.21.15.195:0/852930261 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc6b4005080 0x7fc6cc165f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:44.100 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.063+0000 7fc6d10e6700 1 -- 172.21.15.195:0/852930261 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc6cc06b960 msgr2=0x7fc6cc0bed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:44.101 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.063+0000 7fc6d10e6700 1 --2- 172.21.15.195:0/852930261 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc6cc06b960 0x7fc6cc0bed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:44.101 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.063+0000 7fc6d10e6700 1 -- 172.21.15.195:0/852930261 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6cc0c8250 con 0x7fc6cc0c8aa0 2022-06-27T12:12:44.102 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.063+0000 7fc6d10e6700 1 --2- 172.21.15.195:0/852930261 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc6cc0c8aa0 0x7fc6cc14a320 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7fc6bc00f900 tx=0x7fc6bc009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:12:44.102 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.064+0000 7fc6b9ffb700 1 -- 172.21.15.195:0/852930261 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6bc0052e0 con 0x7fc6cc0c8aa0 2022-06-27T12:12:44.103 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.064+0000 7fc6b9ffb700 1 -- 172.21.15.195:0/852930261 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6bc003500 con 0x7fc6cc0c8aa0 2022-06-27T12:12:44.103 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.064+0000 7fc6d3b4b700 1 -- 172.21.15.195:0/852930261 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6cc0c84f0 con 0x7fc6cc0c8aa0 2022-06-27T12:12:44.104 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.064+0000 7fc6b9ffb700 1 -- 172.21.15.195:0/852930261 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6bc0038a0 con 0x7fc6cc0c8aa0 2022-06-27T12:12:44.104 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.064+0000 7fc6d3b4b700 1 -- 172.21.15.195:0/852930261 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6cc0c7430 con 0x7fc6cc0c8aa0 2022-06-27T12:12:44.104 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.065+0000 7fc6d3b4b700 1 -- 172.21.15.195:0/852930261 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6cc0722c0 con 0x7fc6cc0c8aa0 2022-06-27T12:12:44.104 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.065+0000 7fc6b9ffb700 1 -- 172.21.15.195:0/852930261 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fc6bc003c50 con 0x7fc6cc0c8aa0 2022-06-27T12:12:44.105 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.065+0000 7fc6b9ffb700 1 --2- 172.21.15.195:0/852930261 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc6ac063d00 0x7fc6ac0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:44.105 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.066+0000 7fc6b9ffb700 1 -- 172.21.15.195:0/852930261 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7492+0+0 (secure 0 0 0) 0x7fc6bc080dc0 con 0x7fc6cc0c8aa0 2022-06-27T12:12:44.106 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.066+0000 7fc6d18e7700 1 --2- 172.21.15.195:0/852930261 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc6ac063d00 0x7fc6ac0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:44.106 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.066+0000 7fc6d18e7700 1 --2- 172.21.15.195:0/852930261 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc6ac063d00 0x7fc6ac0661b0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fc6c8001720 tx=0x7fc6c8009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:12:44.107 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.070+0000 7fc6b9ffb700 1 -- 172.21.15.195:0/852930261 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fc6bc04f4d0 con 0x7fc6cc0c8aa0 2022-06-27T12:12:44.299 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.292+0000 7fc6d3b4b700 1 -- 172.21.15.195:0/852930261 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc6cc0c3ba0 con 0x7fc6cc0c8aa0 2022-06-27T12:12:44.299 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.293+0000 7fc6b9ffb700 1 -- 172.21.15.195:0/852930261 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+5367 (secure 0 0 0) 0x7fc6bc052af0 con 0x7fc6cc0c8aa0 2022-06-27T12:12:44.300 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:12:44.300 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:12:40.165257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6865},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6865":{"gid":6865,"name":"d","rank":0,"incarnation":97,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6835/2349973547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2349973547},{"type":"v1","addr":"172.21.15.202:6835","nonce":2349973547}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:12:44.301 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.295+0000 7fc6d3b4b700 1 -- 172.21.15.195:0/852930261 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc6ac063d00 msgr2=0x7fc6ac0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:44.301 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.295+0000 7fc6d3b4b700 1 --2- 172.21.15.195:0/852930261 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc6ac063d00 0x7fc6ac0661b0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fc6c8001720 tx=0x7fc6c8009040 comp rx=0 tx=0).stop 2022-06-27T12:12:44.302 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.295+0000 7fc6d3b4b700 1 -- 172.21.15.195:0/852930261 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc6cc0c8aa0 msgr2=0x7fc6cc14a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:44.302 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.295+0000 7fc6d3b4b700 1 --2- 172.21.15.195:0/852930261 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc6cc0c8aa0 0x7fc6cc14a320 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7fc6bc00f900 tx=0x7fc6bc009040 comp rx=0 tx=0).stop 2022-06-27T12:12:44.302 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.295+0000 7fc6d3b4b700 1 -- 172.21.15.195:0/852930261 shutdown_connections 2022-06-27T12:12:44.303 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.295+0000 7fc6d3b4b700 1 --2- 172.21.15.195:0/852930261 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc6ac063d00 0x7fc6ac0661b0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:44.303 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.295+0000 7fc6d3b4b700 1 --2- 172.21.15.195:0/852930261 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc6cc06b960 0x7fc6cc0bed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:44.303 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.295+0000 7fc6d3b4b700 1 --2- 172.21.15.195:0/852930261 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc6b4005080 0x7fc6cc165f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:44.304 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.295+0000 7fc6d3b4b700 1 --2- 172.21.15.195:0/852930261 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc6cc0c8aa0 0x7fc6cc14a320 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:44.304 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.295+0000 7fc6d3b4b700 1 -- 172.21.15.195:0/852930261 >> 172.21.15.195:0/852930261 conn(0x7fc6cc0c9b00 msgr2=0x7fc6cc071250 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:12:44.305 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.295+0000 7fc6d3b4b700 1 -- 172.21.15.195:0/852930261 shutdown_connections 2022-06-27T12:12:44.305 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:44.295+0000 7fc6d3b4b700 1 -- 172.21.15.195:0/852930261 wait complete. 2022-06-27T12:12:44.305 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 97 2022-06-27T12:12:44.308 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:12:46.309 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:12:46.513 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.471+0000 7f05cdb3b700 1 Processor -- start 2022-06-27T12:12:46.513 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.471+0000 7f05cdb3b700 1 -- start start 2022-06-27T12:12:46.514 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.472+0000 7f05cdb3b700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f05c814e490 0x7f05c806cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:46.514 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.472+0000 7f05cdb3b700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05c80c33b0 con 0x7f05c80c3f50 2022-06-27T12:12:46.515 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.472+0000 7f05cdb3b700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05c806d520 con 0x7f05c814e860 2022-06-27T12:12:46.515 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.472+0000 7f05cdb3b700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05c806f220 con 0x7f05c814e490 2022-06-27T12:12:46.515 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.472+0000 7f05c7fff700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f05c80c3f50 0x7f05c8157890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:57014/0 (socket says 172.21.15.195:57014) 2022-06-27T12:12:46.516 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.472+0000 7f05c7fff700 1 -- 172.21.15.195:0/2710900291 learned_addr learned my addr 172.21.15.195:0/2710900291 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:12:46.516 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.472+0000 7f05c77fe700 1 --2- 172.21.15.195:0/2710900291 >> v2:172.21.15.202:3300/0 conn(0x7f05c814e490 0x7f05c806cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:46.517 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.473+0000 7f05c67fc700 1 -- 172.21.15.195:0/2710900291 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2594814450 0 0) 0x7f05c80c33b0 con 0x7f05c80c3f50 2022-06-27T12:12:46.518 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.473+0000 7f05c67fc700 1 -- 172.21.15.195:0/2710900291 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05a8003430 con 0x7f05c80c3f50 2022-06-27T12:12:46.518 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.473+0000 7f05c77fe700 1 -- 172.21.15.195:0/2710900291 >> v1:172.21.15.202:6789/0 conn(0x7f05c814e860 legacy=0x7f05c80c3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:46.520 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.473+0000 7f05c67fc700 1 -- 172.21.15.195:0/2710900291 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2068259417 0 0) 0x7f05a8003430 con 0x7f05c80c3f50 2022-06-27T12:12:46.521 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.473+0000 7f05c77fe700 1 -- 172.21.15.195:0/2710900291 >> v1:172.21.15.195:6789/0 conn(0x7f05c80c3f50 legacy=0x7f05c8157890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:46.521 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.473+0000 7f05c77fe700 1 -- 172.21.15.195:0/2710900291 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05c81542c0 con 0x7f05c814e490 2022-06-27T12:12:46.521 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.473+0000 7f05c77fe700 1 --2- 172.21.15.195:0/2710900291 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f05c814e490 0x7f05c806cfe0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f05b00097a0 tx=0x7f05b0009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b33ca0b655c69d4c server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:12:46.522 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.473+0000 7f05c67fc700 1 -- 172.21.15.195:0/2710900291 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05c00017f0 con 0x7f05c814e490 2022-06-27T12:12:46.522 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.473+0000 7f05c67fc700 1 -- 172.21.15.195:0/2710900291 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05b0011400 con 0x7f05c814e490 2022-06-27T12:12:46.523 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.473+0000 7f05c67fc700 1 -- 172.21.15.195:0/2710900291 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b00117a0 con 0x7f05c814e490 2022-06-27T12:12:46.523 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.474+0000 7f05cdb3b700 1 -- 172.21.15.195:0/2710900291 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f05c814e490 msgr2=0x7f05c806cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:46.524 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.474+0000 7f05cdb3b700 1 --2- 172.21.15.195:0/2710900291 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f05c814e490 0x7f05c806cfe0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f05b00097a0 tx=0x7f05b0009060 comp rx=0 tx=0).stop 2022-06-27T12:12:46.525 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.474+0000 7f05cdb3b700 1 -- 172.21.15.195:0/2710900291 shutdown_connections 2022-06-27T12:12:46.525 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.474+0000 7f05cdb3b700 1 --2- 172.21.15.195:0/2710900291 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f05c814e490 0x7f05c806cfe0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:46.526 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.474+0000 7f05cdb3b700 1 -- 172.21.15.195:0/2710900291 >> 172.21.15.195:0/2710900291 conn(0x7f05c8063af0 msgr2=0x7f05c8063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:12:46.526 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.474+0000 7f05cdb3b700 1 -- 172.21.15.195:0/2710900291 shutdown_connections 2022-06-27T12:12:46.527 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.474+0000 7f05cdb3b700 1 -- 172.21.15.195:0/2710900291 wait complete. 2022-06-27T12:12:46.527 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.475+0000 7f05cdb3b700 1 Processor -- start 2022-06-27T12:12:46.527 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.475+0000 7f05cdb3b700 1 -- start start 2022-06-27T12:12:46.528 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.476+0000 7f05cdb3b700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f05c80c3f50 0x7f05c8158380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:46.528 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.476+0000 7f05cdb3b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f05c814e490 0x7f05c80c0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:46.528 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.476+0000 7f05cdb3b700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f05c814e860 0x7f05c8153250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:46.536 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.476+0000 7f05cdb3b700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05c8166450 con 0x7f05c814e490 2022-06-27T12:12:46.537 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.476+0000 7f05cdb3b700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05c8166f70 con 0x7f05c80c3f50 2022-06-27T12:12:46.537 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.476+0000 7f05cdb3b700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05c80c0bd0 con 0x7f05c814e860 2022-06-27T12:12:46.537 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.476+0000 7f05c6ffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f05c814e490 0x7f05c80c0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:46.538 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.476+0000 7f05c6ffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f05c814e490 0x7f05c80c0690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50830/0 (socket says 172.21.15.195:50830) 2022-06-27T12:12:46.538 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.476+0000 7f05c6ffd700 1 -- 172.21.15.195:0/3763251360 learned_addr learned my addr 172.21.15.195:0/3763251360 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:12:46.539 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.476+0000 7f05c77fe700 1 --2- 172.21.15.195:0/3763251360 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f05c80c3f50 0x7f05c8158380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:46.540 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.476+0000 7f05c7fff700 1 --2- 172.21.15.195:0/3763251360 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f05c814e860 0x7f05c8153250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:46.540 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.476+0000 7f05c6ffd700 1 -- 172.21.15.195:0/3763251360 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f05c814e860 msgr2=0x7f05c8153250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:46.541 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.476+0000 7f05c6ffd700 1 --2- 172.21.15.195:0/3763251360 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f05c814e860 0x7f05c8153250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:46.541 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.476+0000 7f05c6ffd700 1 -- 172.21.15.195:0/3763251360 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f05c80c3f50 msgr2=0x7f05c8158380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:46.543 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.476+0000 7f05c6ffd700 1 --2- 172.21.15.195:0/3763251360 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f05c80c3f50 0x7f05c8158380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:46.551 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.476+0000 7f05c6ffd700 1 -- 172.21.15.195:0/3763251360 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05b00092e0 con 0x7f05c814e490 2022-06-27T12:12:46.551 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.476+0000 7f05c77fe700 1 --2- 172.21.15.195:0/3763251360 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f05c80c3f50 0x7f05c8158380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:12:46.551 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.476+0000 7f05c6ffd700 1 --2- 172.21.15.195:0/3763251360 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f05c814e490 0x7f05c80c0690 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f05b800c5c0 tx=0x7f05b8007dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:12:46.552 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.476+0000 7f05bf7fe700 1 -- 172.21.15.195:0/3763251360 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b80052e0 con 0x7f05c814e490 2022-06-27T12:12:46.553 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.477+0000 7f05bf7fe700 1 -- 172.21.15.195:0/3763251360 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05b8003500 con 0x7f05c814e490 2022-06-27T12:12:46.554 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.477+0000 7f05bf7fe700 1 -- 172.21.15.195:0/3763251360 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b80038c0 con 0x7f05c814e490 2022-06-27T12:12:46.554 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.477+0000 7f05cdb3b700 1 -- 172.21.15.195:0/3763251360 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05c80c0e60 con 0x7f05c814e490 2022-06-27T12:12:46.554 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.477+0000 7f05cdb3b700 1 -- 172.21.15.195:0/3763251360 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05c8166920 con 0x7f05c814e490 2022-06-27T12:12:46.555 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.477+0000 7f05cdb3b700 1 -- 172.21.15.195:0/3763251360 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05c80722c0 con 0x7f05c814e490 2022-06-27T12:12:46.555 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.478+0000 7f05bf7fe700 1 -- 172.21.15.195:0/3763251360 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f05b8019030 con 0x7f05c814e490 2022-06-27T12:12:46.556 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.478+0000 7f05bf7fe700 1 --2- 172.21.15.195:0/3763251360 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f05ac063d00 0x7f05ac0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:46.556 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.478+0000 7f05bf7fe700 1 -- 172.21.15.195:0/3763251360 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7492+0+0 (secure 0 0 0) 0x7f05b8080100 con 0x7f05c814e490 2022-06-27T12:12:46.556 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.478+0000 7f05c77fe700 1 --2- 172.21.15.195:0/3763251360 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f05ac063d00 0x7f05ac0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:46.557 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.479+0000 7f05c77fe700 1 --2- 172.21.15.195:0/3763251360 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f05ac063d00 0x7f05ac0661b0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f05b000a010 tx=0x7f05b0017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:12:46.557 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.483+0000 7f05bf7fe700 1 -- 172.21.15.195:0/3763251360 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f05b804e750 con 0x7f05c814e490 2022-06-27T12:12:46.723 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.718+0000 7f05cdb3b700 1 -- 172.21.15.195:0/3763251360 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f05c80c7250 con 0x7f05c814e490 2022-06-27T12:12:46.724 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.719+0000 7f05bf7fe700 1 -- 172.21.15.195:0/3763251360 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+5367 (secure 0 0 0) 0x7f05b8017070 con 0x7f05c814e490 2022-06-27T12:12:46.725 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:12:46.725 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:12:40.165257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6865},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6865":{"gid":6865,"name":"d","rank":0,"incarnation":97,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6835/2349973547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2349973547},{"type":"v1","addr":"172.21.15.202:6835","nonce":2349973547}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:12:46.729 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.721+0000 7f05cdb3b700 1 -- 172.21.15.195:0/3763251360 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f05ac063d00 msgr2=0x7f05ac0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:46.729 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.721+0000 7f05cdb3b700 1 --2- 172.21.15.195:0/3763251360 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f05ac063d00 0x7f05ac0661b0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f05b000a010 tx=0x7f05b0017000 comp rx=0 tx=0).stop 2022-06-27T12:12:46.730 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.721+0000 7f05cdb3b700 1 -- 172.21.15.195:0/3763251360 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f05c814e490 msgr2=0x7f05c80c0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:46.730 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.721+0000 7f05cdb3b700 1 --2- 172.21.15.195:0/3763251360 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f05c814e490 0x7f05c80c0690 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f05b800c5c0 tx=0x7f05b8007dc0 comp rx=0 tx=0).stop 2022-06-27T12:12:46.731 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.721+0000 7f05cdb3b700 1 -- 172.21.15.195:0/3763251360 shutdown_connections 2022-06-27T12:12:46.731 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.721+0000 7f05cdb3b700 1 --2- 172.21.15.195:0/3763251360 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f05ac063d00 0x7f05ac0661b0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:46.731 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.721+0000 7f05cdb3b700 1 --2- 172.21.15.195:0/3763251360 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f05c814e860 0x7f05c8153250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:46.732 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.721+0000 7f05cdb3b700 1 --2- 172.21.15.195:0/3763251360 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f05c80c3f50 0x7f05c8158380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:46.732 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.721+0000 7f05cdb3b700 1 --2- 172.21.15.195:0/3763251360 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f05c814e490 0x7f05c80c0690 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:46.733 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.721+0000 7f05cdb3b700 1 -- 172.21.15.195:0/3763251360 >> 172.21.15.195:0/3763251360 conn(0x7f05c8063af0 msgr2=0x7f05c814c800 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:12:46.733 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.721+0000 7f05cdb3b700 1 -- 172.21.15.195:0/3763251360 shutdown_connections 2022-06-27T12:12:46.734 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:46.721+0000 7f05cdb3b700 1 -- 172.21.15.195:0/3763251360 wait complete. 2022-06-27T12:12:46.739 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 97 2022-06-27T12:12:46.747 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:12:48.757 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:12:48.911 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:48.887+0000 7fc26e42a700 1 Processor -- start 2022-06-27T12:12:48.912 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:48.887+0000 7fc26e42a700 1 -- start start 2022-06-27T12:12:48.913 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:48.888+0000 7fc26e42a700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc268068f80 0x7fc2680c4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:48.913 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:48.888+0000 7fc26e42a700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fc2680c7480 0x7fc2680683c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:48.914 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:48.888+0000 7fc26e42a700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2680697f0 con 0x7fc268069350 2022-06-27T12:12:48.914 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:48.888+0000 7fc26e42a700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2681573b0 con 0x7fc2680c7480 2022-06-27T12:12:48.915 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:48.888+0000 7fc26e42a700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2680c46c0 con 0x7fc268068f80 2022-06-27T12:12:48.915 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:48.888+0000 7fc26c9c7700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fc268069350 0x7fc268156460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:57030/0 (socket says 172.21.15.195:57030) 2022-06-27T12:12:48.916 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:48.888+0000 7fc26c9c7700 1 -- 172.21.15.195:0/3725964715 learned_addr learned my addr 172.21.15.195:0/3725964715 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:12:48.916 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:48.888+0000 7fc267fff700 1 --2- 172.21.15.195:0/3725964715 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc268068f80 0x7fc2680c4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:48.916 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:48.888+0000 7fc2677fe700 1 --2- 172.21.15.195:0/3725964715 >> v2:172.21.15.202:3300/0 conn(0x7fc2680c7480 0x7fc2680683c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:48.917 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:48.888+0000 7fc266ffd700 1 -- 172.21.15.195:0/3725964715 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3837514815 0 0) 0x7fc2680697f0 con 0x7fc268069350 2022-06-27T12:12:48.917 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:48.888+0000 7fc266ffd700 1 -- 172.21.15.195:0/3725964715 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc254003430 con 0x7fc268069350 2022-06-27T12:12:48.918 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:48.888+0000 7fc267fff700 1 -- 172.21.15.195:0/3725964715 >> v2:172.21.15.202:3300/0 conn(0x7fc2680c7480 msgr2=0x7fc2680683c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:12:48.918 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:48.888+0000 7fc267fff700 1 --2- 172.21.15.195:0/3725964715 >> v2:172.21.15.202:3300/0 conn(0x7fc2680c7480 0x7fc2680683c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:48.919 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:48.888+0000 7fc267fff700 1 -- 172.21.15.195:0/3725964715 >> v1:172.21.15.195:6789/0 conn(0x7fc268069350 legacy=0x7fc268156460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:48.919 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:48.888+0000 7fc267fff700 1 -- 172.21.15.195:0/3725964715 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc268156b70 con 0x7fc268068f80 2022-06-27T12:12:48.920 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:48.888+0000 7fc2677fe700 1 --2- 172.21.15.195:0/3725964715 >> v2:172.21.15.202:3300/0 conn(0x7fc2680c7480 0x7fc2680683c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:12:48.920 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:48.889+0000 7fc267fff700 1 --2- 172.21.15.195:0/3725964715 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc268068f80 0x7fc2680c4b50 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc25c0097a0 tx=0x7fc25c009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6545948fe12723f server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:12:48.921 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:48.889+0000 7fc266ffd700 1 -- 172.21.15.195:0/3725964715 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc25c011630 con 0x7fc268068f80 2022-06-27T12:12:48.922 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:48.889+0000 7fc266ffd700 1 -- 172.21.15.195:0/3725964715 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc25c011790 con 0x7fc268068f80 2022-06-27T12:12:48.922 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:48.889+0000 7fc266ffd700 1 -- 172.21.15.195:0/3725964715 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc25c011b30 con 0x7fc268068f80 2022-06-27T12:12:48.923 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:48.889+0000 7fc26e42a700 1 -- 172.21.15.195:0/3725964715 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc268068f80 msgr2=0x7fc2680c4b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:48.923 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:48.889+0000 7fc26e42a700 1 --2- 172.21.15.195:0/3725964715 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc268068f80 0x7fc2680c4b50 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc25c0097a0 tx=0x7fc25c009060 comp rx=0 tx=0).stop 2022-06-27T12:12:48.924 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:48.889+0000 7fc26e42a700 1 -- 172.21.15.195:0/3725964715 shutdown_connections 2022-06-27T12:12:48.924 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:48.889+0000 7fc26e42a700 1 --2- 172.21.15.195:0/3725964715 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc268068f80 0x7fc2680c4b50 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:48.924 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:48.889+0000 7fc26e42a700 1 --2- 172.21.15.195:0/3725964715 >> v2:172.21.15.202:3300/0 conn(0x7fc2680c7480 0x7fc2680683c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:48.925 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:48.889+0000 7fc26e42a700 1 -- 172.21.15.195:0/3725964715 >> 172.21.15.195:0/3725964715 conn(0x7fc2680ccbd0 msgr2=0x7fc2680ccfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:12:48.925 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:48.889+0000 7fc26e42a700 1 -- 172.21.15.195:0/3725964715 shutdown_connections 2022-06-27T12:12:48.926 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:48.889+0000 7fc26e42a700 1 -- 172.21.15.195:0/3725964715 wait complete. 2022-06-27T12:12:48.926 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:48.890+0000 7fc26e42a700 1 Processor -- start 2022-06-27T12:12:48.927 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:48.891+0000 7fc26e42a700 1 -- start start 2022-06-27T12:12:48.927 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:48.891+0000 7fc26e42a700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc268068f80 0x7fc2681673f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:48.928 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:48.891+0000 7fc26e42a700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc268069350 0x7fc2680b1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:48.928 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:48.891+0000 7fc26e42a700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc2680c7480 0x7fc2681521a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:48.928 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:48.891+0000 7fc26e42a700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc268151f80 con 0x7fc2680c7480 2022-06-27T12:12:48.929 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:48.891+0000 7fc26e42a700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc268156560 con 0x7fc268069350 2022-06-27T12:12:48.929 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:48.891+0000 7fc26e42a700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2680b21f0 con 0x7fc268068f80 2022-06-27T12:12:48.930 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:48.891+0000 7fc26c9c7700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc2680c7480 0x7fc2681521a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:48.930 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:48.891+0000 7fc267fff700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc268068f80 0x7fc2681673f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:48.931 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:48.892+0000 7fc267fff700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc268068f80 0x7fc2681673f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3301/0 says I am v2:172.21.15.195:43600/0 (socket says 172.21.15.195:43600) 2022-06-27T12:12:48.931 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:48.892+0000 7fc2677fe700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc268069350 0x7fc2680b1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:48.932 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:48.892+0000 7fc267fff700 1 -- 172.21.15.195:0/1879509897 learned_addr learned my addr 172.21.15.195:0/1879509897 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:12:48.932 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:48.892+0000 7fc26c9c7700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc2680c7480 0x7fc2681521a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50848/0 (socket says 172.21.15.195:50848) 2022-06-27T12:12:48.933 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:48.892+0000 7fc2677fe700 1 -- 172.21.15.195:0/1879509897 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc268068f80 msgr2=0x7fc2681673f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:48.933 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:48.892+0000 7fc2677fe700 1 --2- 172.21.15.195:0/1879509897 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc268068f80 0x7fc2681673f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:48.933 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:48.892+0000 7fc2677fe700 1 -- 172.21.15.195:0/1879509897 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc2680c7480 msgr2=0x7fc2681521a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:48.934 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:48.892+0000 7fc2677fe700 1 --2- 172.21.15.195:0/1879509897 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc2680c7480 0x7fc2681521a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:48.934 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:48.892+0000 7fc2677fe700 1 -- 172.21.15.195:0/1879509897 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2580092e0 con 0x7fc268069350 2022-06-27T12:12:48.935 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:48.892+0000 7fc2677fe700 1 --2- 172.21.15.195:0/1879509897 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc268069350 0x7fc2680b1cb0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fc25800e930 tx=0x7fc258009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:12:48.935 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:48.892+0000 7fc24ffff700 1 -- 172.21.15.195:0/1879509897 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc258013680 con 0x7fc268069350 2022-06-27T12:12:48.935 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:48.892+0000 7fc24ffff700 1 -- 172.21.15.195:0/1879509897 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2580137e0 con 0x7fc268069350 2022-06-27T12:12:48.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:48.892+0000 7fc26e42a700 1 -- 172.21.15.195:0/1879509897 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc25c0092e0 con 0x7fc268069350 2022-06-27T12:12:48.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:48.892+0000 7fc24ffff700 1 -- 172.21.15.195:0/1879509897 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc258013ba0 con 0x7fc268069350 2022-06-27T12:12:48.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:48.892+0000 7fc26e42a700 1 -- 172.21.15.195:0/1879509897 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc268151060 con 0x7fc268069350 2022-06-27T12:12:48.937 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:48.893+0000 7fc26e42a700 1 -- 172.21.15.195:0/1879509897 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2680c4cc0 con 0x7fc268069350 2022-06-27T12:12:48.937 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:48.894+0000 7fc24ffff700 1 -- 172.21.15.195:0/1879509897 <== mon.1 v2:172.21.15.202:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fc258023070 con 0x7fc268069350 2022-06-27T12:12:48.938 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:48.894+0000 7fc24ffff700 1 --2- 172.21.15.195:0/1879509897 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc248063d00 0x7fc2480661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:48.938 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:48.894+0000 7fc24ffff700 1 -- 172.21.15.195:0/1879509897 <== mon.1 v2:172.21.15.202:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7492+0+0 (secure 0 0 0) 0x7fc258081150 con 0x7fc268069350 2022-06-27T12:12:48.938 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:48.894+0000 7fc267fff700 1 --2- 172.21.15.195:0/1879509897 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc248063d00 0x7fc2480661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:48.939 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:48.895+0000 7fc267fff700 1 --2- 172.21.15.195:0/1879509897 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc248063d00 0x7fc2480661b0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fc25c009260 tx=0x7fc25c017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:12:48.939 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:48.899+0000 7fc24ffff700 1 -- 172.21.15.195:0/1879509897 <== mon.1 v2:172.21.15.202:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fc25804f740 con 0x7fc268069350 2022-06-27T12:12:49.131 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:49.131+0000 7fc26e42a700 1 -- 172.21.15.195:0/1879509897 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc26806ac60 con 0x7fc268069350 2022-06-27T12:12:49.132 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:49.131+0000 7fc24ffff700 1 -- 172.21.15.195:0/1879509897 <== mon.1 v2:172.21.15.202:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+5367 (secure 0 0 0) 0x7fc258017090 con 0x7fc268069350 2022-06-27T12:12:49.133 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:12:49.133 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:12:40.165257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6865},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6865":{"gid":6865,"name":"d","rank":0,"incarnation":97,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6835/2349973547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2349973547},{"type":"v1","addr":"172.21.15.202:6835","nonce":2349973547}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:12:49.143 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:49.133+0000 7fc26e42a700 1 -- 172.21.15.195:0/1879509897 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc248063d00 msgr2=0x7fc2480661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:49.144 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:49.133+0000 7fc26e42a700 1 --2- 172.21.15.195:0/1879509897 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc248063d00 0x7fc2480661b0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fc25c009260 tx=0x7fc25c017000 comp rx=0 tx=0).stop 2022-06-27T12:12:49.144 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:49.133+0000 7fc26e42a700 1 -- 172.21.15.195:0/1879509897 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc268069350 msgr2=0x7fc2680b1cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:49.145 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:49.133+0000 7fc26e42a700 1 --2- 172.21.15.195:0/1879509897 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc268069350 0x7fc2680b1cb0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fc25800e930 tx=0x7fc258009060 comp rx=0 tx=0).stop 2022-06-27T12:12:49.145 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:49.133+0000 7fc26e42a700 1 -- 172.21.15.195:0/1879509897 shutdown_connections 2022-06-27T12:12:49.146 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:49.133+0000 7fc26e42a700 1 --2- 172.21.15.195:0/1879509897 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc248063d00 0x7fc2480661b0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:49.146 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:49.133+0000 7fc26e42a700 1 --2- 172.21.15.195:0/1879509897 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc2680c7480 0x7fc2681521a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:49.147 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:49.133+0000 7fc26e42a700 1 --2- 172.21.15.195:0/1879509897 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc268068f80 0x7fc2681673f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:49.147 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:49.133+0000 7fc26e42a700 1 --2- 172.21.15.195:0/1879509897 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc268069350 0x7fc2680b1cb0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:49.148 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:49.133+0000 7fc26e42a700 1 -- 172.21.15.195:0/1879509897 >> 172.21.15.195:0/1879509897 conn(0x7fc2680ccbd0 msgr2=0x7fc268150040 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:12:49.148 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:49.133+0000 7fc26e42a700 1 -- 172.21.15.195:0/1879509897 shutdown_connections 2022-06-27T12:12:49.148 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:49.133+0000 7fc26e42a700 1 -- 172.21.15.195:0/1879509897 wait complete. 2022-06-27T12:12:49.149 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 97 2022-06-27T12:12:49.162 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:12:51.171 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:12:51.351 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.316+0000 7ff2127d5700 1 Processor -- start 2022-06-27T12:12:51.352 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.316+0000 7ff2127d5700 1 -- start start 2022-06-27T12:12:51.352 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.317+0000 7ff2127d5700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff20c06fef0 0x7ff20c1570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:51.353 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.317+0000 7ff2127d5700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7ff20c0c34a0 0x7ff20c0c2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:51.353 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.317+0000 7ff2127d5700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff20c06fba0 con 0x7ff20c06e850 2022-06-27T12:12:51.354 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.317+0000 7ff2127d5700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff20c1575e0 con 0x7ff20c0c34a0 2022-06-27T12:12:51.354 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.317+0000 7ff2127d5700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff20c06f5e0 con 0x7ff20c06fef0 2022-06-27T12:12:51.355 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.317+0000 7ff20bfff700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7ff20c06e850 0x7ff20c069de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:57048/0 (socket says 172.21.15.195:57048) 2022-06-27T12:12:51.355 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.317+0000 7ff20bfff700 1 -- 172.21.15.195:0/371436777 learned_addr learned my addr 172.21.15.195:0/371436777 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:12:51.356 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.318+0000 7ff20b7fe700 1 --2- 172.21.15.195:0/371436777 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff20c06fef0 0x7ff20c1570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:51.360 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.318+0000 7ff210d72700 1 --2- 172.21.15.195:0/371436777 >> v2:172.21.15.202:3300/0 conn(0x7ff20c0c34a0 0x7ff20c0c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:51.360 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.318+0000 7ff20affd700 1 -- 172.21.15.195:0/371436777 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2339710535 0 0) 0x7ff20c06fba0 con 0x7ff20c06e850 2022-06-27T12:12:51.361 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.318+0000 7ff20affd700 1 -- 172.21.15.195:0/371436777 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1f8003430 con 0x7ff20c06e850 2022-06-27T12:12:51.361 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.318+0000 7ff20b7fe700 1 -- 172.21.15.195:0/371436777 >> v2:172.21.15.202:3300/0 conn(0x7ff20c0c34a0 msgr2=0x7ff20c0c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:12:51.362 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.318+0000 7ff20b7fe700 1 --2- 172.21.15.195:0/371436777 >> v2:172.21.15.202:3300/0 conn(0x7ff20c0c34a0 0x7ff20c0c2920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:51.363 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.318+0000 7ff20b7fe700 1 -- 172.21.15.195:0/371436777 >> v1:172.21.15.195:6789/0 conn(0x7ff20c06e850 legacy=0x7ff20c069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:51.363 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.318+0000 7ff20b7fe700 1 -- 172.21.15.195:0/371436777 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff20c06f8c0 con 0x7ff20c06fef0 2022-06-27T12:12:51.364 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.318+0000 7ff20b7fe700 1 --2- 172.21.15.195:0/371436777 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff20c06fef0 0x7ff20c1570a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc0097a0 tx=0x7ff1fc009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3884fdd46455da33 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:12:51.364 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.318+0000 7ff20affd700 1 -- 172.21.15.195:0/371436777 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1fc011630 con 0x7ff20c06fef0 2022-06-27T12:12:51.364 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.318+0000 7ff20affd700 1 -- 172.21.15.195:0/371436777 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1fc011790 con 0x7ff20c06fef0 2022-06-27T12:12:51.365 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.318+0000 7ff20affd700 1 -- 172.21.15.195:0/371436777 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1fc011b10 con 0x7ff20c06fef0 2022-06-27T12:12:51.365 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.319+0000 7ff2127d5700 1 -- 172.21.15.195:0/371436777 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff20c06fef0 msgr2=0x7ff20c1570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:51.366 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.319+0000 7ff2127d5700 1 --2- 172.21.15.195:0/371436777 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff20c06fef0 0x7ff20c1570a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc0097a0 tx=0x7ff1fc009060 comp rx=0 tx=0).stop 2022-06-27T12:12:51.366 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.319+0000 7ff2127d5700 1 -- 172.21.15.195:0/371436777 shutdown_connections 2022-06-27T12:12:51.367 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.319+0000 7ff2127d5700 1 --2- 172.21.15.195:0/371436777 >> v2:172.21.15.202:3300/0 conn(0x7ff20c0c34a0 0x7ff20c0c2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:51.367 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.319+0000 7ff2127d5700 1 --2- 172.21.15.195:0/371436777 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff20c06fef0 0x7ff20c1570a0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:51.368 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.319+0000 7ff2127d5700 1 -- 172.21.15.195:0/371436777 >> 172.21.15.195:0/371436777 conn(0x7ff20c0c6350 msgr2=0x7ff20c0c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:12:51.368 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.319+0000 7ff2127d5700 1 -- 172.21.15.195:0/371436777 shutdown_connections 2022-06-27T12:12:51.369 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.319+0000 7ff2127d5700 1 -- 172.21.15.195:0/371436777 wait complete. 2022-06-27T12:12:51.369 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.320+0000 7ff2127d5700 1 Processor -- start 2022-06-27T12:12:51.369 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.320+0000 7ff2127d5700 1 -- start start 2022-06-27T12:12:51.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.321+0000 7ff2127d5700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff20c06e850 0x7ff20c14b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:51.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.321+0000 7ff2127d5700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff20c0c34a0 0x7ff20c067ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:51.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.321+0000 7ff2127d5700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff20c1542e0 0x7ff20c068410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:51.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.321+0000 7ff2127d5700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff20c152520 con 0x7ff20c06e850 2022-06-27T12:12:51.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.321+0000 7ff2127d5700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff20c151770 con 0x7ff20c1542e0 2022-06-27T12:12:51.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.321+0000 7ff2127d5700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff20c14bba0 con 0x7ff20c0c34a0 2022-06-27T12:12:51.372 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.321+0000 7ff20bfff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff20c06e850 0x7ff20c14b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:51.372 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.321+0000 7ff20bfff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff20c06e850 0x7ff20c14b660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50864/0 (socket says 172.21.15.195:50864) 2022-06-27T12:12:51.373 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.321+0000 7ff20bfff700 1 -- 172.21.15.195:0/3875272509 learned_addr learned my addr 172.21.15.195:0/3875272509 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:12:51.373 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.321+0000 7ff210d72700 1 --2- 172.21.15.195:0/3875272509 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff20c1542e0 0x7ff20c068410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:51.374 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.321+0000 7ff20b7fe700 1 --2- 172.21.15.195:0/3875272509 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff20c0c34a0 0x7ff20c067ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:51.374 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.321+0000 7ff20bfff700 1 -- 172.21.15.195:0/3875272509 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff20c0c34a0 msgr2=0x7ff20c067ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:51.375 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.321+0000 7ff20bfff700 1 --2- 172.21.15.195:0/3875272509 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff20c0c34a0 0x7ff20c067ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:51.375 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.321+0000 7ff20bfff700 1 -- 172.21.15.195:0/3875272509 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff20c1542e0 msgr2=0x7ff20c068410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:51.376 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.321+0000 7ff20bfff700 1 --2- 172.21.15.195:0/3875272509 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff20c1542e0 0x7ff20c068410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:51.376 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.321+0000 7ff20bfff700 1 -- 172.21.15.195:0/3875272509 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1fc0092e0 con 0x7ff20c06e850 2022-06-27T12:12:51.377 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.321+0000 7ff20b7fe700 1 --2- 172.21.15.195:0/3875272509 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff20c0c34a0 0x7ff20c067ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:12:51.377 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.321+0000 7ff210d72700 1 --2- 172.21.15.195:0/3875272509 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff20c1542e0 0x7ff20c068410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:12:51.378 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.321+0000 7ff20bfff700 1 --2- 172.21.15.195:0/3875272509 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff20c06e850 0x7ff20c14b660 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7ff20000c440 tx=0x7ff200007c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:12:51.378 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.322+0000 7ff1f3fff700 1 -- 172.21.15.195:0/3875272509 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2000070d0 con 0x7ff20c06e850 2022-06-27T12:12:51.379 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.322+0000 7ff1f3fff700 1 -- 172.21.15.195:0/3875272509 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff200007230 con 0x7ff20c06e850 2022-06-27T12:12:51.379 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.322+0000 7ff2127d5700 1 -- 172.21.15.195:0/3875272509 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff20c156970 con 0x7ff20c06e850 2022-06-27T12:12:51.380 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.322+0000 7ff1f3fff700 1 -- 172.21.15.195:0/3875272509 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2000075f0 con 0x7ff20c06e850 2022-06-27T12:12:51.380 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.322+0000 7ff2127d5700 1 -- 172.21.15.195:0/3875272509 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff20c156430 con 0x7ff20c06e850 2022-06-27T12:12:51.380 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.323+0000 7ff2127d5700 1 -- 172.21.15.195:0/3875272509 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff20c0722c0 con 0x7ff20c06e850 2022-06-27T12:12:51.381 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.323+0000 7ff1f3fff700 1 -- 172.21.15.195:0/3875272509 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7ff200019030 con 0x7ff20c06e850 2022-06-27T12:12:51.381 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.323+0000 7ff1f3fff700 1 --2- 172.21.15.195:0/3875272509 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff1ec063d00 0x7ff1ec0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:51.381 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.323+0000 7ff1f3fff700 1 -- 172.21.15.195:0/3875272509 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7492+0+0 (secure 0 0 0) 0x7ff20007fed0 con 0x7ff20c06e850 2022-06-27T12:12:51.382 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.324+0000 7ff20b7fe700 1 --2- 172.21.15.195:0/3875272509 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff1ec063d00 0x7ff1ec0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:51.382 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.324+0000 7ff20b7fe700 1 --2- 172.21.15.195:0/3875272509 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff1ec063d00 0x7ff1ec0661b0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7ff20c16ab10 tx=0x7ff1fc017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:12:51.383 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.328+0000 7ff1f3fff700 1 -- 172.21.15.195:0/3875272509 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7ff20004e520 con 0x7ff20c06e850 2022-06-27T12:12:51.563 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.561+0000 7ff2127d5700 1 -- 172.21.15.195:0/3875272509 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff20c06f240 con 0x7ff20c06e850 2022-06-27T12:12:51.564 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.561+0000 7ff1f3fff700 1 -- 172.21.15.195:0/3875272509 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+5367 (secure 0 0 0) 0x7ff200017090 con 0x7ff20c06e850 2022-06-27T12:12:51.566 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:12:51.567 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:12:40.165257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6865},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6865":{"gid":6865,"name":"d","rank":0,"incarnation":97,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6835/2349973547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2349973547},{"type":"v1","addr":"172.21.15.202:6835","nonce":2349973547}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:12:51.569 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.563+0000 7ff2127d5700 1 -- 172.21.15.195:0/3875272509 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff1ec063d00 msgr2=0x7ff1ec0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:51.570 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.563+0000 7ff2127d5700 1 --2- 172.21.15.195:0/3875272509 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff1ec063d00 0x7ff1ec0661b0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7ff20c16ab10 tx=0x7ff1fc017000 comp rx=0 tx=0).stop 2022-06-27T12:12:51.570 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.563+0000 7ff2127d5700 1 -- 172.21.15.195:0/3875272509 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff20c06e850 msgr2=0x7ff20c14b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:51.571 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.563+0000 7ff2127d5700 1 --2- 172.21.15.195:0/3875272509 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff20c06e850 0x7ff20c14b660 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7ff20000c440 tx=0x7ff200007c70 comp rx=0 tx=0).stop 2022-06-27T12:12:51.571 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.563+0000 7ff2127d5700 1 -- 172.21.15.195:0/3875272509 shutdown_connections 2022-06-27T12:12:51.572 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.563+0000 7ff2127d5700 1 --2- 172.21.15.195:0/3875272509 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff1ec063d00 0x7ff1ec0661b0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:51.572 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.563+0000 7ff2127d5700 1 --2- 172.21.15.195:0/3875272509 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff20c06e850 0x7ff20c14b660 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:51.574 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.563+0000 7ff2127d5700 1 --2- 172.21.15.195:0/3875272509 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff20c1542e0 0x7ff20c068410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:51.575 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.563+0000 7ff2127d5700 1 --2- 172.21.15.195:0/3875272509 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff20c0c34a0 0x7ff20c067ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:51.575 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.563+0000 7ff2127d5700 1 -- 172.21.15.195:0/3875272509 >> 172.21.15.195:0/3875272509 conn(0x7ff20c0c6350 msgr2=0x7ff20c0c26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:12:51.575 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.563+0000 7ff2127d5700 1 -- 172.21.15.195:0/3875272509 shutdown_connections 2022-06-27T12:12:51.577 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:51.563+0000 7ff2127d5700 1 -- 172.21.15.195:0/3875272509 wait complete. 2022-06-27T12:12:51.577 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 97 2022-06-27T12:12:51.579 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:12:53.581 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:12:53.723 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.715+0000 7f319ef1b700 1 Processor -- start 2022-06-27T12:12:53.724 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.715+0000 7f319ef1b700 1 -- start start 2022-06-27T12:12:53.724 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.715+0000 7f319ef1b700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f31980c2bf0 0x7f31980c2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:53.724 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.715+0000 7f319ef1b700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f31980c61e0 0x7f31980c48f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:53.725 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.715+0000 7f319ef1b700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31980bc490 con 0x7f31980c4e30 2022-06-27T12:12:53.725 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.715+0000 7f319ef1b700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31980c52b0 con 0x7f31980c61e0 2022-06-27T12:12:53.725 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.715+0000 7f319ef1b700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f319806a0e0 con 0x7f31980c2bf0 2022-06-27T12:12:53.726 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.716+0000 7f3197fff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f31980c61e0 0x7f31980c48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:53.726 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.716+0000 7f3197fff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f31980c61e0 0x7f31980c48f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50876/0 (socket says 172.21.15.195:50876) 2022-06-27T12:12:53.726 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.716+0000 7f3197fff700 1 -- 172.21.15.195:0/271009438 learned_addr learned my addr 172.21.15.195:0/271009438 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:12:53.727 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.716+0000 7f319ccb7700 1 --2- 172.21.15.195:0/271009438 >> v2:172.21.15.202:3300/0 conn(0x7f31980c2bf0 0x7f31980c2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:53.727 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.716+0000 7f3197fff700 1 -- 172.21.15.195:0/271009438 >> v2:172.21.15.202:3300/0 conn(0x7f31980c2bf0 msgr2=0x7f31980c2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:12:53.728 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.716+0000 7f3197fff700 1 --2- 172.21.15.195:0/271009438 >> v2:172.21.15.202:3300/0 conn(0x7f31980c2bf0 0x7f31980c2fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:53.728 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.716+0000 7f3197fff700 1 -- 172.21.15.195:0/271009438 >> v1:172.21.15.202:6789/0 conn(0x7f31980c4e30 legacy=0x7f31980c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:12:53.728 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.716+0000 7f3197fff700 1 -- 172.21.15.195:0/271009438 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31980bc6e0 con 0x7f31980c61e0 2022-06-27T12:12:53.734 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.716+0000 7f3197fff700 1 --2- 172.21.15.195:0/271009438 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f31980c61e0 0x7f31980c48f0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f319806f840 tx=0x7f3184009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e06a328e3694ff60 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:12:53.734 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.716+0000 7f31977fe700 1 -- 172.21.15.195:0/271009438 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3184004030 con 0x7f31980c61e0 2022-06-27T12:12:53.734 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.716+0000 7f31977fe700 1 -- 172.21.15.195:0/271009438 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3184004190 con 0x7f31980c61e0 2022-06-27T12:12:53.735 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.716+0000 7f31977fe700 1 -- 172.21.15.195:0/271009438 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3184011630 con 0x7f31980c61e0 2022-06-27T12:12:53.735 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.717+0000 7f319ef1b700 1 -- 172.21.15.195:0/271009438 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f31980c61e0 msgr2=0x7f31980c48f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:53.735 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.717+0000 7f319ef1b700 1 --2- 172.21.15.195:0/271009438 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f31980c61e0 0x7f31980c48f0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f319806f840 tx=0x7f3184009120 comp rx=0 tx=0).stop 2022-06-27T12:12:53.736 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.717+0000 7f319ef1b700 1 -- 172.21.15.195:0/271009438 shutdown_connections 2022-06-27T12:12:53.736 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.717+0000 7f319ef1b700 1 --2- 172.21.15.195:0/271009438 >> v2:172.21.15.202:3300/0 conn(0x7f31980c2bf0 0x7f31980c2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:53.736 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.717+0000 7f319ef1b700 1 --2- 172.21.15.195:0/271009438 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f31980c61e0 0x7f31980c48f0 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:53.736 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.717+0000 7f319ef1b700 1 -- 172.21.15.195:0/271009438 >> 172.21.15.195:0/271009438 conn(0x7f31980c9b00 msgr2=0x7f31980c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:12:53.737 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.717+0000 7f319ef1b700 1 -- 172.21.15.195:0/271009438 shutdown_connections 2022-06-27T12:12:53.737 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.717+0000 7f319ef1b700 1 -- 172.21.15.195:0/271009438 wait complete. 2022-06-27T12:12:53.737 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.718+0000 7f319ef1b700 1 Processor -- start 2022-06-27T12:12:53.737 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.719+0000 7f319ef1b700 1 -- start start 2022-06-27T12:12:53.738 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.719+0000 7f319ef1b700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f31980c2bf0 0x7f3198144f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:53.738 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.719+0000 7f319ef1b700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f31980c4e30 0x7f3198163c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:53.738 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.719+0000 7f319ef1b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f31980c61e0 0x7f319814d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:53.739 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.719+0000 7f319ef1b700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3198159ea0 con 0x7f31980c61e0 2022-06-27T12:12:53.739 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.719+0000 7f319ef1b700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3198158b10 con 0x7f31980c4e30 2022-06-27T12:12:53.739 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.719+0000 7f319ef1b700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3198158db0 con 0x7f31980c2bf0 2022-06-27T12:12:53.740 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.719+0000 7f319d4b8700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f31980c61e0 0x7f319814d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:53.740 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.720+0000 7f319d4b8700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f31980c61e0 0x7f319814d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50886/0 (socket says 172.21.15.195:50886) 2022-06-27T12:12:53.740 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.720+0000 7f319d4b8700 1 -- 172.21.15.195:0/395680194 learned_addr learned my addr 172.21.15.195:0/395680194 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:12:53.740 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.720+0000 7f3197fff700 1 --2- 172.21.15.195:0/395680194 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f31980c4e30 0x7f3198163c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:53.741 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.720+0000 7f319ccb7700 1 --2- 172.21.15.195:0/395680194 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f31980c2bf0 0x7f3198144f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:53.741 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.720+0000 7f319d4b8700 1 -- 172.21.15.195:0/395680194 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f31980c2bf0 msgr2=0x7f3198144f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:53.741 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.720+0000 7f319d4b8700 1 --2- 172.21.15.195:0/395680194 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f31980c2bf0 0x7f3198144f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:53.741 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.720+0000 7f319d4b8700 1 -- 172.21.15.195:0/395680194 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f31980c4e30 msgr2=0x7f3198163c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:53.742 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.720+0000 7f319d4b8700 1 --2- 172.21.15.195:0/395680194 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f31980c4e30 0x7f3198163c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:53.742 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.720+0000 7f319d4b8700 1 -- 172.21.15.195:0/395680194 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31840093a0 con 0x7f31980c61e0 2022-06-27T12:12:53.742 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.720+0000 7f319d4b8700 1 --2- 172.21.15.195:0/395680194 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f31980c61e0 0x7f319814d9b0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f319000c810 tx=0x7f3190003bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:12:53.742 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.720+0000 7f3194ff9700 1 -- 172.21.15.195:0/395680194 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3190012630 con 0x7f31980c61e0 2022-06-27T12:12:53.743 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.720+0000 7f3194ff9700 1 -- 172.21.15.195:0/395680194 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3190012790 con 0x7f31980c61e0 2022-06-27T12:12:53.743 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.720+0000 7f319ef1b700 1 -- 172.21.15.195:0/395680194 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31981590e0 con 0x7f31980c61e0 2022-06-27T12:12:53.743 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.720+0000 7f3194ff9700 1 -- 172.21.15.195:0/395680194 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3190012b50 con 0x7f31980c61e0 2022-06-27T12:12:53.743 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.720+0000 7f319ef1b700 1 -- 172.21.15.195:0/395680194 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31981596a0 con 0x7f31980c61e0 2022-06-27T12:12:53.744 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.721+0000 7f319ef1b700 1 -- 172.21.15.195:0/395680194 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31980722c0 con 0x7f31980c61e0 2022-06-27T12:12:53.744 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.721+0000 7f3194ff9700 1 -- 172.21.15.195:0/395680194 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f3190010050 con 0x7f31980c61e0 2022-06-27T12:12:53.744 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.722+0000 7f3194ff9700 1 --2- 172.21.15.195:0/395680194 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3180063d30 0x7f31800661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:53.744 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.722+0000 7f3194ff9700 1 -- 172.21.15.195:0/395680194 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7492+0+0 (secure 0 0 0) 0x7f3190080be0 con 0x7f31980c61e0 2022-06-27T12:12:53.745 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.722+0000 7f319ccb7700 1 --2- 172.21.15.195:0/395680194 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3180063d30 0x7f31800661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:53.745 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.722+0000 7f319ccb7700 1 --2- 172.21.15.195:0/395680194 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3180063d30 0x7f31800661e0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f318c003740 tx=0x7f318c008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:12:53.745 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.726+0000 7f3194ff9700 1 -- 172.21.15.195:0/395680194 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f319004f230 con 0x7f31980c61e0 2022-06-27T12:12:53.969 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.949+0000 7f319ef1b700 1 -- 172.21.15.195:0/395680194 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3198155130 con 0x7f31980c61e0 2022-06-27T12:12:53.970 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.950+0000 7f3194ff9700 1 -- 172.21.15.195:0/395680194 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+5371 (secure 0 0 0) 0x7f3190016020 con 0x7f31980c61e0 2022-06-27T12:12:53.971 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:12:53.971 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":98,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:12:53.284952+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6865},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6865":{"gid":6865,"name":"d","rank":0,"incarnation":97,"state":"up:reconnect","state_seq":14,"addr":"172.21.15.202:6835/2349973547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2349973547},{"type":"v1","addr":"172.21.15.202:6835","nonce":2349973547}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:12:53.980 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.952+0000 7f319ef1b700 1 -- 172.21.15.195:0/395680194 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3180063d30 msgr2=0x7f31800661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:53.981 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.952+0000 7f319ef1b700 1 --2- 172.21.15.195:0/395680194 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3180063d30 0x7f31800661e0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f318c003740 tx=0x7f318c008040 comp rx=0 tx=0).stop 2022-06-27T12:12:53.981 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.952+0000 7f319ef1b700 1 -- 172.21.15.195:0/395680194 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f31980c61e0 msgr2=0x7f319814d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:53.982 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.952+0000 7f319ef1b700 1 --2- 172.21.15.195:0/395680194 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f31980c61e0 0x7f319814d9b0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f319000c810 tx=0x7f3190003bd0 comp rx=0 tx=0).stop 2022-06-27T12:12:53.982 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.952+0000 7f319ef1b700 1 -- 172.21.15.195:0/395680194 shutdown_connections 2022-06-27T12:12:53.983 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.952+0000 7f319ef1b700 1 --2- 172.21.15.195:0/395680194 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3180063d30 0x7f31800661e0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:53.983 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.952+0000 7f319ef1b700 1 --2- 172.21.15.195:0/395680194 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f31980c61e0 0x7f319814d9b0 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:53.984 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.952+0000 7f319ef1b700 1 --2- 172.21.15.195:0/395680194 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f31980c2bf0 0x7f3198144f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:53.984 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.952+0000 7f319ef1b700 1 --2- 172.21.15.195:0/395680194 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f31980c4e30 0x7f3198163c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:53.985 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.952+0000 7f319ef1b700 1 -- 172.21.15.195:0/395680194 >> 172.21.15.195:0/395680194 conn(0x7f31980c9b00 msgr2=0x7f319806dcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:12:53.985 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.952+0000 7f319ef1b700 1 -- 172.21.15.195:0/395680194 shutdown_connections 2022-06-27T12:12:53.986 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:53.952+0000 7f319ef1b700 1 -- 172.21.15.195:0/395680194 wait complete. 2022-06-27T12:12:53.986 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 98 2022-06-27T12:12:53.988 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 98, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-06-27T11:47:52.358095+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 98, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6865": { "addr": "172.21.15.202:6835/2349973547", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6834", "nonce": 2349973547, "type": "v2" }, { "addr": "172.21.15.202:6835", "nonce": 2349973547, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 6865, "incarnation": 97, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:reconnect", "state_seq": 14 } }, "last_failure": 0, "last_failure_osd_epoch": 63, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-06-27T12:12:53.284952+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6865 } } } ], "standbys": [ { "addr": "172.21.15.202:6837/3305006841", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6836", "nonce": 3305006841, "type": "v2" }, { "addr": "172.21.15.202:6837", "nonce": 3305006841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6835/2060709822", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6834", "nonce": 2060709822, "type": "v2" }, { "addr": "172.21.15.195:6835", "nonce": 2060709822, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6838/1598751652", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6836", "nonce": 1598751652, "type": "v2" }, { "addr": "172.21.15.195:6838", "nonce": 1598751652, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4275, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6839/1313800661", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6837", "nonce": 1313800661, "type": "v2" }, { "addr": "172.21.15.195:6839", "nonce": 1313800661, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4278, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-06-27T12:12:55.990 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:12:56.159 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.126+0000 7f93b2662700 1 Processor -- start 2022-06-27T12:12:56.161 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.126+0000 7f93b2662700 1 -- start start 2022-06-27T12:12:56.161 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.127+0000 7f93b2662700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f93ac150940 0x7f93ac06f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:56.162 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.127+0000 7f93b2662700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f93ac156ff0 0x7f93ac06e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:56.162 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.127+0000 7f93b2662700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93ac0c1d40 con 0x7f93ac150570 2022-06-27T12:12:56.163 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.127+0000 7f93b2662700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93ac0c30e0 con 0x7f93ac156ff0 2022-06-27T12:12:56.163 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.127+0000 7f93b2662700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93ac06fc80 con 0x7f93ac150940 2022-06-27T12:12:56.169 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.127+0000 7f93b0bff700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f93ac156ff0 0x7f93ac06e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:56.169 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.127+0000 7f93b0bff700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f93ac156ff0 0x7f93ac06e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3300/0 says I am v2:172.21.15.195:49020/0 (socket says 172.21.15.195:49020) 2022-06-27T12:12:56.171 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.127+0000 7f93b0bff700 1 -- 172.21.15.195:0/1418940733 learned_addr learned my addr 172.21.15.195:0/1418940733 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:12:56.171 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.127+0000 7f93abfff700 1 --1- 172.21.15.195:0/1418940733 >> v1:172.21.15.202:6789/0 conn(0x7f93ac150570 0x7f93ac0c4160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.202:6789/0 says I am v1:172.21.15.195:58938/0 (socket says 172.21.15.195:58938) 2022-06-27T12:12:56.171 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.127+0000 7f93ab7fe700 1 --2- 172.21.15.195:0/1418940733 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f93ac150940 0x7f93ac06f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:56.172 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.128+0000 7f93ab7fe700 1 -- 172.21.15.195:0/1418940733 >> v2:172.21.15.202:3300/0 conn(0x7f93ac156ff0 msgr2=0x7f93ac06e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:12:56.172 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.128+0000 7f93ab7fe700 1 --2- 172.21.15.195:0/1418940733 >> v2:172.21.15.202:3300/0 conn(0x7f93ac156ff0 0x7f93ac06e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:56.173 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.128+0000 7f93ab7fe700 1 -- 172.21.15.195:0/1418940733 >> v1:172.21.15.202:6789/0 conn(0x7f93ac150570 legacy=0x7f93ac0c4160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:56.173 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.128+0000 7f93aaffd700 1 -- 172.21.15.195:0/1418940733 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 93158297 0 0) 0x7f93ac0c1d40 con 0x7f93ac150570 2022-06-27T12:12:56.174 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.128+0000 7f93ab7fe700 1 -- 172.21.15.195:0/1418940733 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93ac06ff40 con 0x7f93ac150940 2022-06-27T12:12:56.175 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.128+0000 7f93b0bff700 1 --2- 172.21.15.195:0/1418940733 >> v2:172.21.15.202:3300/0 conn(0x7f93ac156ff0 0x7f93ac06e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:12:56.176 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.128+0000 7f93ab7fe700 1 --2- 172.21.15.195:0/1418940733 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f93ac150940 0x7f93ac06f3b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f93a00097a0 tx=0x7f93a0009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7b53056721f99d8 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:12:56.176 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.128+0000 7f93aaffd700 1 -- 172.21.15.195:0/1418940733 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93a0011630 con 0x7f93ac150940 2022-06-27T12:12:56.177 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.128+0000 7f93aaffd700 1 -- 172.21.15.195:0/1418940733 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93a0011790 con 0x7f93ac150940 2022-06-27T12:12:56.177 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.128+0000 7f93aaffd700 1 -- 172.21.15.195:0/1418940733 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93a0011b30 con 0x7f93ac150940 2022-06-27T12:12:56.177 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.129+0000 7f93b2662700 1 -- 172.21.15.195:0/1418940733 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f93ac150940 msgr2=0x7f93ac06f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:56.178 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.129+0000 7f93b2662700 1 --2- 172.21.15.195:0/1418940733 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f93ac150940 0x7f93ac06f3b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f93a00097a0 tx=0x7f93a0009060 comp rx=0 tx=0).stop 2022-06-27T12:12:56.178 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.129+0000 7f93b2662700 1 -- 172.21.15.195:0/1418940733 shutdown_connections 2022-06-27T12:12:56.179 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.129+0000 7f93b2662700 1 --2- 172.21.15.195:0/1418940733 >> v2:172.21.15.202:3300/0 conn(0x7f93ac156ff0 0x7f93ac06e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:56.179 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.129+0000 7f93b2662700 1 --2- 172.21.15.195:0/1418940733 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f93ac150940 0x7f93ac06f3b0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:56.180 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.129+0000 7f93b2662700 1 -- 172.21.15.195:0/1418940733 >> 172.21.15.195:0/1418940733 conn(0x7f93ac0c6350 msgr2=0x7f93ac0c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:12:56.180 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.129+0000 7f93b2662700 1 -- 172.21.15.195:0/1418940733 shutdown_connections 2022-06-27T12:12:56.181 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.129+0000 7f93b2662700 1 -- 172.21.15.195:0/1418940733 wait complete. 2022-06-27T12:12:56.182 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.130+0000 7f93b2662700 1 Processor -- start 2022-06-27T12:12:56.182 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.130+0000 7f93b2662700 1 -- start start 2022-06-27T12:12:56.182 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.131+0000 7f93b2662700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f93ac150570 0x7f93ac15af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:56.183 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.131+0000 7f93b2662700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f93ac150940 0x7f93ac169c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:56.183 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.131+0000 7f93b2662700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f93ac156ff0 0x7f93ac16a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:56.184 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.131+0000 7f93b2662700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93ac157a00 con 0x7f93ac150570 2022-06-27T12:12:56.184 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.131+0000 7f93b2662700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93ac1536c0 con 0x7f93ac150940 2022-06-27T12:12:56.185 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.131+0000 7f93b2662700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93ac157c50 con 0x7f93ac156ff0 2022-06-27T12:12:56.186 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.131+0000 7f93abfff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f93ac150570 0x7f93ac15af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:56.186 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.131+0000 7f93abfff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f93ac150570 0x7f93ac15af70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50896/0 (socket says 172.21.15.195:50896) 2022-06-27T12:12:56.187 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.131+0000 7f93abfff700 1 -- 172.21.15.195:0/1802256715 learned_addr learned my addr 172.21.15.195:0/1802256715 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:12:56.187 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.131+0000 7f93b0bff700 1 --2- 172.21.15.195:0/1802256715 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f93ac156ff0 0x7f93ac16a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:56.188 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.131+0000 7f93ab7fe700 1 --2- 172.21.15.195:0/1802256715 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f93ac150940 0x7f93ac169c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:56.188 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.131+0000 7f93abfff700 1 -- 172.21.15.195:0/1802256715 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f93ac156ff0 msgr2=0x7f93ac16a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:56.188 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.131+0000 7f93abfff700 1 --2- 172.21.15.195:0/1802256715 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f93ac156ff0 0x7f93ac16a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:56.189 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.131+0000 7f93abfff700 1 -- 172.21.15.195:0/1802256715 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f93ac150940 msgr2=0x7f93ac169c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:56.189 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.131+0000 7f93abfff700 1 --2- 172.21.15.195:0/1802256715 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f93ac150940 0x7f93ac169c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:56.191 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.131+0000 7f93abfff700 1 -- 172.21.15.195:0/1802256715 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93a4009ae0 con 0x7f93ac150570 2022-06-27T12:12:56.192 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.131+0000 7f93abfff700 1 --2- 172.21.15.195:0/1802256715 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f93ac150570 0x7f93ac15af70 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f939c00c4c0 tx=0x7f939c007dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:12:56.192 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.131+0000 7f9393fff700 1 -- 172.21.15.195:0/1802256715 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f939c007360 con 0x7f93ac150570 2022-06-27T12:12:56.193 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.131+0000 7f9393fff700 1 -- 172.21.15.195:0/1802256715 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f939c0074c0 con 0x7f93ac150570 2022-06-27T12:12:56.193 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.131+0000 7f93b2662700 1 -- 172.21.15.195:0/1802256715 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93a00092e0 con 0x7f93ac150570 2022-06-27T12:12:56.194 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.131+0000 7f9393fff700 1 -- 172.21.15.195:0/1802256715 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f939c004940 con 0x7f93ac150570 2022-06-27T12:12:56.194 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.132+0000 7f93b2662700 1 -- 172.21.15.195:0/1802256715 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93ac152d90 con 0x7f93ac150570 2022-06-27T12:12:56.194 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.132+0000 7f93b2662700 1 -- 172.21.15.195:0/1802256715 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93ac0722c0 con 0x7f93ac150570 2022-06-27T12:12:56.195 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.133+0000 7f9393fff700 1 -- 172.21.15.195:0/1802256715 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f939c010030 con 0x7f93ac150570 2022-06-27T12:12:56.195 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.133+0000 7f9393fff700 1 --2- 172.21.15.195:0/1802256715 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f938c063d30 0x7f938c0661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:56.198 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.133+0000 7f9393fff700 1 -- 172.21.15.195:0/1802256715 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7492+0+0 (secure 0 0 0) 0x7f939c0807f0 con 0x7f93ac150570 2022-06-27T12:12:56.198 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.133+0000 7f93ab7fe700 1 --2- 172.21.15.195:0/1802256715 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f938c063d30 0x7f938c0661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:56.199 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.134+0000 7f93ab7fe700 1 --2- 172.21.15.195:0/1802256715 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f938c063d30 0x7f938c0661e0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f93ac150d50 tx=0x7f93a0017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:12:56.199 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.137+0000 7f9393fff700 1 -- 172.21.15.195:0/1802256715 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f939c04eec0 con 0x7f93ac150570 2022-06-27T12:12:56.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.369+0000 7f93b2662700 1 -- 172.21.15.195:0/1802256715 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f93ac0c1130 con 0x7f93ac150570 2022-06-27T12:12:56.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.369+0000 7f9393fff700 1 -- 172.21.15.195:0/1802256715 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+5371 (secure 0 0 0) 0x7f939c0524e0 con 0x7f93ac150570 2022-06-27T12:12:56.372 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:12:56.374 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":98,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:12:53.284952+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6865},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6865":{"gid":6865,"name":"d","rank":0,"incarnation":97,"state":"up:reconnect","state_seq":14,"addr":"172.21.15.202:6835/2349973547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2349973547},{"type":"v1","addr":"172.21.15.202:6835","nonce":2349973547}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:12:56.379 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.371+0000 7f93b2662700 1 -- 172.21.15.195:0/1802256715 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f938c063d30 msgr2=0x7f938c0661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:56.383 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.371+0000 7f93b2662700 1 --2- 172.21.15.195:0/1802256715 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f938c063d30 0x7f938c0661e0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f93ac150d50 tx=0x7f93a0017000 comp rx=0 tx=0).stop 2022-06-27T12:12:56.383 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.371+0000 7f93b2662700 1 -- 172.21.15.195:0/1802256715 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f93ac150570 msgr2=0x7f93ac15af70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:56.383 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.371+0000 7f93b2662700 1 --2- 172.21.15.195:0/1802256715 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f93ac150570 0x7f93ac15af70 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f939c00c4c0 tx=0x7f939c007dd0 comp rx=0 tx=0).stop 2022-06-27T12:12:56.384 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.371+0000 7f93b2662700 1 -- 172.21.15.195:0/1802256715 shutdown_connections 2022-06-27T12:12:56.384 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.371+0000 7f93b2662700 1 --2- 172.21.15.195:0/1802256715 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f938c063d30 0x7f938c0661e0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:56.384 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.371+0000 7f93b2662700 1 --2- 172.21.15.195:0/1802256715 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f93ac150570 0x7f93ac15af70 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:56.385 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.371+0000 7f93b2662700 1 --2- 172.21.15.195:0/1802256715 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f93ac156ff0 0x7f93ac16a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:56.385 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.371+0000 7f93b2662700 1 --2- 172.21.15.195:0/1802256715 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f93ac150940 0x7f93ac169c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:56.385 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.371+0000 7f93b2662700 1 -- 172.21.15.195:0/1802256715 >> 172.21.15.195:0/1802256715 conn(0x7f93ac0c6350 msgr2=0x7f93ac0c3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:12:56.386 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.371+0000 7f93b2662700 1 -- 172.21.15.195:0/1802256715 shutdown_connections 2022-06-27T12:12:56.386 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:56.371+0000 7f93b2662700 1 -- 172.21.15.195:0/1802256715 wait complete. 2022-06-27T12:12:56.387 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 98 2022-06-27T12:12:56.389 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:12:58.394 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:12:58.555 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.539+0000 7f1e4bbc4700 1 Processor -- start 2022-06-27T12:12:58.556 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.539+0000 7f1e4bbc4700 1 -- start start 2022-06-27T12:12:58.556 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.540+0000 7f1e4bbc4700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1e4406d050 0x7f1e4415a890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:58.557 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.540+0000 7f1e4bbc4700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1e4406aa60 0x7f1e4415e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:58.557 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.540+0000 7f1e4bbc4700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e4415ae90 con 0x7f1e4406a690 2022-06-27T12:12:58.558 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.540+0000 7f1e4bbc4700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e44069be0 con 0x7f1e4406aa60 2022-06-27T12:12:58.559 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.540+0000 7f1e4bbc4700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e4415edb0 con 0x7f1e4406d050 2022-06-27T12:12:58.559 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.540+0000 7f1e4a161700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1e4406aa60 0x7f1e4415e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:58.560 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.540+0000 7f1e4a161700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1e4406aa60 0x7f1e4415e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50908/0 (socket says 172.21.15.195:50908) 2022-06-27T12:12:58.560 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.540+0000 7f1e4a161700 1 -- 172.21.15.195:0/3330285 learned_addr learned my addr 172.21.15.195:0/3330285 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:12:58.561 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.540+0000 7f1e4915f700 1 --2- 172.21.15.195:0/3330285 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1e4406d050 0x7f1e4415a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:58.561 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.540+0000 7f1e4a161700 1 -- 172.21.15.195:0/3330285 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1e4406d050 msgr2=0x7f1e4415a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:12:58.562 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.540+0000 7f1e4a161700 1 --2- 172.21.15.195:0/3330285 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1e4406d050 0x7f1e4415a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:58.562 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.540+0000 7f1e4a161700 1 -- 172.21.15.195:0/3330285 >> v1:172.21.15.202:6789/0 conn(0x7f1e4406a690 legacy=0x7f1e440c7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:58.563 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.540+0000 7f1e4a161700 1 -- 172.21.15.195:0/3330285 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e4415b170 con 0x7f1e4406aa60 2022-06-27T12:12:58.563 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.540+0000 7f1e4915f700 1 --2- 172.21.15.195:0/3330285 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1e4406d050 0x7f1e4415a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:12:58.564 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.540+0000 7f1e4a161700 1 --2- 172.21.15.195:0/3330285 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1e4406aa60 0x7f1e4415e870 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f1e3c009f20 tx=0x7f1e3c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=342fe49517d10ae6 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:12:58.564 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.540+0000 7f1e4895e700 1 -- 172.21.15.195:0/3330285 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e3c00e070 con 0x7f1e4406aa60 2022-06-27T12:12:58.572 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.540+0000 7f1e4895e700 1 -- 172.21.15.195:0/3330285 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e3c00bb10 con 0x7f1e4406aa60 2022-06-27T12:12:58.573 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.541+0000 7f1e4895e700 1 -- 172.21.15.195:0/3330285 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e3c0092e0 con 0x7f1e4406aa60 2022-06-27T12:12:58.573 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.541+0000 7f1e4bbc4700 1 -- 172.21.15.195:0/3330285 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1e4406aa60 msgr2=0x7f1e4415e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:58.573 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.541+0000 7f1e4bbc4700 1 --2- 172.21.15.195:0/3330285 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1e4406aa60 0x7f1e4415e870 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f1e3c009f20 tx=0x7f1e3c009840 comp rx=0 tx=0).stop 2022-06-27T12:12:58.574 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.541+0000 7f1e4bbc4700 1 -- 172.21.15.195:0/3330285 shutdown_connections 2022-06-27T12:12:58.575 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.541+0000 7f1e4bbc4700 1 --2- 172.21.15.195:0/3330285 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1e4406aa60 0x7f1e4415e870 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:58.575 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.541+0000 7f1e4bbc4700 1 --2- 172.21.15.195:0/3330285 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1e4406d050 0x7f1e4415a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:58.576 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.541+0000 7f1e4bbc4700 1 -- 172.21.15.195:0/3330285 >> 172.21.15.195:0/3330285 conn(0x7f1e44063af0 msgr2=0x7f1e44063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:12:58.577 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.541+0000 7f1e4bbc4700 1 -- 172.21.15.195:0/3330285 shutdown_connections 2022-06-27T12:12:58.578 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.541+0000 7f1e4bbc4700 1 -- 172.21.15.195:0/3330285 wait complete. 2022-06-27T12:12:58.578 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.542+0000 7f1e4bbc4700 1 Processor -- start 2022-06-27T12:12:58.579 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.542+0000 7f1e4bbc4700 1 -- start start 2022-06-27T12:12:58.579 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.543+0000 7f1e4bbc4700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1e4406a690 0x7f1e44154f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:58.580 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.543+0000 7f1e4bbc4700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1e4406aa60 0x7f1e440c0500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:58.581 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.543+0000 7f1e4bbc4700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1e4406d050 0x7f1e4416e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:58.581 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.543+0000 7f1e4bbc4700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e440c0070 con 0x7f1e4406aa60 2022-06-27T12:12:58.582 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.543+0000 7f1e4bbc4700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e4416b5f0 con 0x7f1e4406a690 2022-06-27T12:12:58.583 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.543+0000 7f1e4bbc4700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e4416b8d0 con 0x7f1e4406d050 2022-06-27T12:12:58.583 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.543+0000 7f1e4915f700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1e4406aa60 0x7f1e440c0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:58.584 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.543+0000 7f1e4915f700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1e4406aa60 0x7f1e440c0500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50912/0 (socket says 172.21.15.195:50912) 2022-06-27T12:12:58.584 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.543+0000 7f1e4915f700 1 -- 172.21.15.195:0/3348390103 learned_addr learned my addr 172.21.15.195:0/3348390103 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:12:58.585 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.543+0000 7f1e49960700 1 --2- 172.21.15.195:0/3348390103 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1e4406a690 0x7f1e44154f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:58.586 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.543+0000 7f1e4a161700 1 --2- 172.21.15.195:0/3348390103 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1e4406d050 0x7f1e4416e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:58.595 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.543+0000 7f1e4915f700 1 -- 172.21.15.195:0/3348390103 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1e4406d050 msgr2=0x7f1e4416e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:58.595 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.543+0000 7f1e4915f700 1 --2- 172.21.15.195:0/3348390103 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1e4406d050 0x7f1e4416e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:58.596 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.543+0000 7f1e4915f700 1 -- 172.21.15.195:0/3348390103 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1e4406a690 msgr2=0x7f1e44154f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:58.596 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.543+0000 7f1e4915f700 1 --2- 172.21.15.195:0/3348390103 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1e4406a690 0x7f1e44154f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:58.597 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.543+0000 7f1e4915f700 1 -- 172.21.15.195:0/3348390103 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e3c009ae0 con 0x7f1e4406aa60 2022-06-27T12:12:58.597 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.543+0000 7f1e4915f700 1 --2- 172.21.15.195:0/3348390103 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1e4406aa60 0x7f1e440c0500 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f1e3800d890 tx=0x7f1e38008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:12:58.608 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.544+0000 7f1e35ffb700 1 -- 172.21.15.195:0/3348390103 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e38004030 con 0x7f1e4406aa60 2022-06-27T12:12:58.608 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.544+0000 7f1e35ffb700 1 -- 172.21.15.195:0/3348390103 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e38004190 con 0x7f1e4406aa60 2022-06-27T12:12:58.609 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.544+0000 7f1e35ffb700 1 -- 172.21.15.195:0/3348390103 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e38014620 con 0x7f1e4406aa60 2022-06-27T12:12:58.609 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.544+0000 7f1e4bbc4700 1 -- 172.21.15.195:0/3348390103 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e4416bbb0 con 0x7f1e4406aa60 2022-06-27T12:12:58.610 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.544+0000 7f1e4bbc4700 1 -- 172.21.15.195:0/3348390103 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e440bfd00 con 0x7f1e4406aa60 2022-06-27T12:12:58.610 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.545+0000 7f1e4bbc4700 1 -- 172.21.15.195:0/3348390103 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e44152cd0 con 0x7f1e4406aa60 2022-06-27T12:12:58.611 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.545+0000 7f1e35ffb700 1 -- 172.21.15.195:0/3348390103 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f1e3801a020 con 0x7f1e4406aa60 2022-06-27T12:12:58.622 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.545+0000 7f1e35ffb700 1 --2- 172.21.15.195:0/3348390103 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1e24063d00 0x7f1e240661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:12:58.623 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.546+0000 7f1e35ffb700 1 -- 172.21.15.195:0/3348390103 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7492+0+0 (secure 0 0 0) 0x7f1e38081040 con 0x7f1e4406aa60 2022-06-27T12:12:58.624 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.546+0000 7f1e49960700 1 --2- 172.21.15.195:0/3348390103 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1e24063d00 0x7f1e240661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:12:58.624 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.546+0000 7f1e49960700 1 --2- 172.21.15.195:0/3348390103 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1e24063d00 0x7f1e240661b0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f1e40001880 tx=0x7f1e40009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:12:58.625 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.550+0000 7f1e35ffb700 1 -- 172.21.15.195:0/3348390103 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f1e3804f690 con 0x7f1e4406aa60 2022-06-27T12:12:58.775 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.774+0000 7f1e4bbc4700 1 -- 172.21.15.195:0/3348390103 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1e440722c0 con 0x7f1e4406aa60 2022-06-27T12:12:58.775 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.774+0000 7f1e35ffb700 1 -- 172.21.15.195:0/3348390103 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+5371 (secure 0 0 0) 0x7f1e38052cb0 con 0x7f1e4406aa60 2022-06-27T12:12:58.776 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:12:58.776 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":98,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:12:53.284952+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6865},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6865":{"gid":6865,"name":"d","rank":0,"incarnation":97,"state":"up:reconnect","state_seq":14,"addr":"172.21.15.202:6835/2349973547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2349973547},{"type":"v1","addr":"172.21.15.202:6835","nonce":2349973547}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:12:58.779 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.776+0000 7f1e4bbc4700 1 -- 172.21.15.195:0/3348390103 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1e24063d00 msgr2=0x7f1e240661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:58.779 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.776+0000 7f1e4bbc4700 1 --2- 172.21.15.195:0/3348390103 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1e24063d00 0x7f1e240661b0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f1e40001880 tx=0x7f1e40009040 comp rx=0 tx=0).stop 2022-06-27T12:12:58.780 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.776+0000 7f1e4bbc4700 1 -- 172.21.15.195:0/3348390103 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1e4406aa60 msgr2=0x7f1e440c0500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:12:58.780 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.776+0000 7f1e4bbc4700 1 --2- 172.21.15.195:0/3348390103 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1e4406aa60 0x7f1e440c0500 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f1e3800d890 tx=0x7f1e38008040 comp rx=0 tx=0).stop 2022-06-27T12:12:58.780 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.776+0000 7f1e4bbc4700 1 -- 172.21.15.195:0/3348390103 shutdown_connections 2022-06-27T12:12:58.781 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.776+0000 7f1e4bbc4700 1 --2- 172.21.15.195:0/3348390103 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1e24063d00 0x7f1e240661b0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:58.781 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.776+0000 7f1e4bbc4700 1 --2- 172.21.15.195:0/3348390103 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1e4406d050 0x7f1e4416e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:58.782 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.776+0000 7f1e4bbc4700 1 --2- 172.21.15.195:0/3348390103 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1e4406a690 0x7f1e44154f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:58.782 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.776+0000 7f1e4bbc4700 1 --2- 172.21.15.195:0/3348390103 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1e4406aa60 0x7f1e440c0500 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:12:58.782 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.776+0000 7f1e4bbc4700 1 -- 172.21.15.195:0/3348390103 >> 172.21.15.195:0/3348390103 conn(0x7f1e44063af0 msgr2=0x7f1e441525c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:12:58.783 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.776+0000 7f1e4bbc4700 1 -- 172.21.15.195:0/3348390103 shutdown_connections 2022-06-27T12:12:58.783 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:12:58.776+0000 7f1e4bbc4700 1 -- 172.21.15.195:0/3348390103 wait complete. 2022-06-27T12:12:58.784 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 98 2022-06-27T12:12:58.805 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:13:00.807 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:13:00.952 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:00.940+0000 7f7b0eb75700 1 Processor -- start 2022-06-27T12:13:00.952 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:00.940+0000 7f7b0eb75700 1 -- start start 2022-06-27T12:13:00.952 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:00.941+0000 7f7b0eb75700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f7b08156100 0x7f7b080688a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:13:00.953 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:00.941+0000 7f7b0eb75700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7b0806a440 0x7f7b080c4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:13:00.953 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:00.941+0000 7f7b0eb75700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b080c4b10 con 0x7f7b081564d0 2022-06-27T12:13:00.953 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:00.941+0000 7f7b0eb75700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b080c3dc0 con 0x7f7b08156100 2022-06-27T12:13:00.953 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:00.941+0000 7f7b0eb75700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b08068f20 con 0x7f7b0806a440 2022-06-27T12:13:00.959 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:00.941+0000 7f7b07fff700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7b0806a440 0x7f7b080c4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:13:00.959 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:00.941+0000 7f7b0c911700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f7b08156100 0x7f7b080688a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:13:00.960 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:00.941+0000 7f7b07fff700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7b0806a440 0x7f7b080c4fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3301/0 says I am v2:172.21.15.195:43674/0 (socket says 172.21.15.195:43674) 2022-06-27T12:13:00.960 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:00.941+0000 7f7b0d112700 1 --1- >> v1:172.21.15.202:6789/0 conn(0x7f7b081564d0 0x7f7b080c28c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.202:6789/0 says I am v1:172.21.15.195:58970/0 (socket says 172.21.15.195:58970) 2022-06-27T12:13:00.961 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:00.941+0000 7f7b07fff700 1 -- 172.21.15.195:0/2744594913 learned_addr learned my addr 172.21.15.195:0/2744594913 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:13:00.961 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:00.941+0000 7f7b07fff700 1 -- 172.21.15.195:0/2744594913 >> v2:172.21.15.202:3300/0 conn(0x7f7b08156100 msgr2=0x7f7b080688a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:13:00.962 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:00.941+0000 7f7b07fff700 1 --2- 172.21.15.195:0/2744594913 >> v2:172.21.15.202:3300/0 conn(0x7f7b08156100 0x7f7b080688a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:13:00.963 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:00.941+0000 7f7b07fff700 1 -- 172.21.15.195:0/2744594913 >> v1:172.21.15.202:6789/0 conn(0x7f7b081564d0 legacy=0x7f7b080c28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:13:00.964 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:00.941+0000 7f7b07fff700 1 -- 172.21.15.195:0/2744594913 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b080c6360 con 0x7f7b0806a440 2022-06-27T12:13:00.964 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:00.941+0000 7f7b0c911700 1 --2- 172.21.15.195:0/2744594913 >> v2:172.21.15.202:3300/0 conn(0x7f7b08156100 0x7f7b080688a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:13:00.965 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:00.942+0000 7f7b07fff700 1 --2- 172.21.15.195:0/2744594913 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7b0806a440 0x7f7b080c4fa0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7af4009860 tx=0x7f7af4009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af0ca74113415c89 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:13:00.965 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:00.942+0000 7f7b077fe700 1 -- 172.21.15.195:0/2744594913 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af4011630 con 0x7f7b0806a440 2022-06-27T12:13:00.966 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:00.942+0000 7f7b077fe700 1 -- 172.21.15.195:0/2744594913 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7af4011790 con 0x7f7b0806a440 2022-06-27T12:13:00.967 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:00.942+0000 7f7b077fe700 1 -- 172.21.15.195:0/2744594913 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af4011b30 con 0x7f7b0806a440 2022-06-27T12:13:00.967 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:00.942+0000 7f7b0eb75700 1 -- 172.21.15.195:0/2744594913 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7b0806a440 msgr2=0x7f7b080c4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:13:00.968 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:00.942+0000 7f7b0eb75700 1 --2- 172.21.15.195:0/2744594913 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7b0806a440 0x7f7b080c4fa0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7af4009860 tx=0x7f7af4009120 comp rx=0 tx=0).stop 2022-06-27T12:13:00.969 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:00.942+0000 7f7b0eb75700 1 -- 172.21.15.195:0/2744594913 shutdown_connections 2022-06-27T12:13:00.969 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:00.942+0000 7f7b0eb75700 1 --2- 172.21.15.195:0/2744594913 >> v2:172.21.15.202:3300/0 conn(0x7f7b08156100 0x7f7b080688a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:13:00.969 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:00.942+0000 7f7b0eb75700 1 --2- 172.21.15.195:0/2744594913 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7b0806a440 0x7f7b080c4fa0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:13:00.970 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:00.942+0000 7f7b0eb75700 1 -- 172.21.15.195:0/2744594913 >> 172.21.15.195:0/2744594913 conn(0x7f7b080c9b00 msgr2=0x7f7b080c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:13:00.970 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:00.942+0000 7f7b0eb75700 1 -- 172.21.15.195:0/2744594913 shutdown_connections 2022-06-27T12:13:00.971 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:00.942+0000 7f7b0eb75700 1 -- 172.21.15.195:0/2744594913 wait complete. 2022-06-27T12:13:00.971 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:00.944+0000 7f7b0eb75700 1 Processor -- start 2022-06-27T12:13:00.972 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:00.944+0000 7f7b0eb75700 1 -- start start 2022-06-27T12:13:00.972 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:00.944+0000 7f7b0eb75700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7b0806a440 0x7f7b080bf870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:13:00.973 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:00.944+0000 7f7b0eb75700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7b08156100 0x7f7b08154f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:13:00.973 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:00.944+0000 7f7b0eb75700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7b081564d0 0x7f7b080bd1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:13:00.973 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:00.945+0000 7f7b0eb75700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b0814e170 con 0x7f7b08156100 2022-06-27T12:13:00.974 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:00.945+0000 7f7b0eb75700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b08144780 con 0x7f7b081564d0 2022-06-27T12:13:00.974 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:00.945+0000 7f7b0eb75700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b0814d880 con 0x7f7b0806a440 2022-06-27T12:13:00.974 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:00.945+0000 7f7b07fff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7b08156100 0x7f7b08154f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:13:00.975 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:00.945+0000 7f7b07fff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7b08156100 0x7f7b08154f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50926/0 (socket says 172.21.15.195:50926) 2022-06-27T12:13:00.975 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:00.945+0000 7f7b07fff700 1 -- 172.21.15.195:0/2669272642 learned_addr learned my addr 172.21.15.195:0/2669272642 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:13:00.976 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:00.945+0000 7f7b0c911700 1 --2- 172.21.15.195:0/2669272642 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7b0806a440 0x7f7b080bf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:13:00.977 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:00.945+0000 7f7b0d112700 1 --2- 172.21.15.195:0/2669272642 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7b081564d0 0x7f7b080bd1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:13:00.977 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:00.945+0000 7f7b07fff700 1 -- 172.21.15.195:0/2669272642 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7b0806a440 msgr2=0x7f7b080bf870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:13:00.978 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:00.945+0000 7f7b07fff700 1 --2- 172.21.15.195:0/2669272642 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7b0806a440 0x7f7b080bf870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:13:00.980 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:00.945+0000 7f7b07fff700 1 -- 172.21.15.195:0/2669272642 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7b081564d0 msgr2=0x7f7b080bd1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:13:00.980 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:00.945+0000 7f7b07fff700 1 --2- 172.21.15.195:0/2669272642 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7b081564d0 0x7f7b080bd1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:13:00.980 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:00.945+0000 7f7b07fff700 1 -- 172.21.15.195:0/2669272642 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7afc00a420 con 0x7f7b08156100 2022-06-27T12:13:00.981 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:00.945+0000 7f7b07fff700 1 --2- 172.21.15.195:0/2669272642 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7b08156100 0x7f7b08154f60 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f7af4009f00 tx=0x7f7af4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:13:00.981 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:00.945+0000 7f7b04ff9700 1 -- 172.21.15.195:0/2669272642 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af4003730 con 0x7f7b08156100 2022-06-27T12:13:00.983 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:00.945+0000 7f7b04ff9700 1 -- 172.21.15.195:0/2669272642 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7af4003890 con 0x7f7b08156100 2022-06-27T12:13:00.984 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:00.945+0000 7f7b04ff9700 1 -- 172.21.15.195:0/2669272642 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af4003c90 con 0x7f7b08156100 2022-06-27T12:13:00.984 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:00.945+0000 7f7b0eb75700 1 -- 172.21.15.195:0/2669272642 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7af40093a0 con 0x7f7b08156100 2022-06-27T12:13:00.985 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:00.945+0000 7f7b0eb75700 1 -- 172.21.15.195:0/2669272642 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b08165890 con 0x7f7b08156100 2022-06-27T12:13:00.985 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:00.946+0000 7f7b0eb75700 1 -- 172.21.15.195:0/2669272642 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b0814e320 con 0x7f7b08156100 2022-06-27T12:13:00.985 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:00.947+0000 7f7b04ff9700 1 -- 172.21.15.195:0/2669272642 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f7af400f050 con 0x7f7b08156100 2022-06-27T12:13:00.986 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:00.947+0000 7f7b04ff9700 1 --2- 172.21.15.195:0/2669272642 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7af0063d00 0x7f7af00661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:13:00.986 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:00.947+0000 7f7b04ff9700 1 -- 172.21.15.195:0/2669272642 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7492+0+0 (secure 0 0 0) 0x7f7af4082410 con 0x7f7b08156100 2022-06-27T12:13:00.987 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:00.947+0000 7f7b0c911700 1 --2- 172.21.15.195:0/2669272642 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7af0063d00 0x7f7af00661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:13:00.988 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:00.947+0000 7f7b0c911700 1 --2- 172.21.15.195:0/2669272642 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7af0063d00 0x7f7af00661b0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f7afc00a3f0 tx=0x7f7afc00a200 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:13:00.988 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:00.951+0000 7f7b04ff9700 1 -- 172.21.15.195:0/2669272642 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f7b0814e320 con 0x7f7b08156100 2022-06-27T12:13:01.175 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:01.172+0000 7f7b0eb75700 1 -- 172.21.15.195:0/2669272642 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7b0814e320 con 0x7f7b08156100 2022-06-27T12:13:01.175 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:01.173+0000 7f7b04ff9700 1 -- 172.21.15.195:0/2669272642 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+5368 (secure 0 0 0) 0x7f7b0814e320 con 0x7f7b08156100 2022-06-27T12:13:01.176 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:13:01.176 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":99,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:12:59.298631+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6865},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6865":{"gid":6865,"name":"d","rank":0,"incarnation":97,"state":"up:rejoin","state_seq":16,"addr":"172.21.15.202:6835/2349973547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2349973547},{"type":"v1","addr":"172.21.15.202:6835","nonce":2349973547}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:13:01.177 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:01.175+0000 7f7b0eb75700 1 -- 172.21.15.195:0/2669272642 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7af0063d00 msgr2=0x7f7af00661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:13:01.178 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:01.175+0000 7f7b0eb75700 1 --2- 172.21.15.195:0/2669272642 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7af0063d00 0x7f7af00661b0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f7afc00a3f0 tx=0x7f7afc00a200 comp rx=0 tx=0).stop 2022-06-27T12:13:01.179 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:01.175+0000 7f7b0eb75700 1 -- 172.21.15.195:0/2669272642 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7b08156100 msgr2=0x7f7b08154f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:13:01.179 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:01.175+0000 7f7b0eb75700 1 --2- 172.21.15.195:0/2669272642 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7b08156100 0x7f7b08154f60 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f7af4009f00 tx=0x7f7af4017000 comp rx=0 tx=0).stop 2022-06-27T12:13:01.179 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:01.175+0000 7f7b0eb75700 1 -- 172.21.15.195:0/2669272642 shutdown_connections 2022-06-27T12:13:01.179 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:01.175+0000 7f7b0eb75700 1 --2- 172.21.15.195:0/2669272642 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7af0063d00 0x7f7af00661b0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:13:01.179 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:01.175+0000 7f7b0eb75700 1 --2- 172.21.15.195:0/2669272642 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f7b081564d0 0x7f7b080bd1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:13:01.180 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:01.175+0000 7f7b0eb75700 1 --2- 172.21.15.195:0/2669272642 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f7b0806a440 0x7f7b080bf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:13:01.180 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:01.175+0000 7f7b0eb75700 1 --2- 172.21.15.195:0/2669272642 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7b08156100 0x7f7b08154f60 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:13:01.180 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:01.175+0000 7f7b0eb75700 1 -- 172.21.15.195:0/2669272642 >> 172.21.15.195:0/2669272642 conn(0x7f7b080c9b00 msgr2=0x7f7b080c2ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:13:01.180 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:01.175+0000 7f7b0eb75700 1 -- 172.21.15.195:0/2669272642 shutdown_connections 2022-06-27T12:13:01.181 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:01.175+0000 7f7b0eb75700 1 -- 172.21.15.195:0/2669272642 wait complete. 2022-06-27T12:13:01.181 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 99 2022-06-27T12:13:01.209 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:13:03.210 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:13:03.282 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:13:03.286 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:13:03.373 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.347+0000 7f1a3e399700 1 Processor -- start 2022-06-27T12:13:03.374 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.347+0000 7f1a3e399700 1 -- start start 2022-06-27T12:13:03.374 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.348+0000 7f1a3e399700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1a38153cb0 0x7f1a38156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:13:03.375 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.348+0000 7f1a3e399700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a380c1a20 con 0x7f1a380c4160 2022-06-27T12:13:03.375 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.348+0000 7f1a3e399700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a380c1130 con 0x7f1a3806f640 2022-06-27T12:13:03.376 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.348+0000 7f1a3e399700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a3806ea90 con 0x7f1a38153cb0 2022-06-27T12:13:03.376 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.348+0000 7f1a3c936700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1a38153cb0 0x7f1a38156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:13:03.376 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.348+0000 7f1a377fe700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f1a380c4160 0x7f1a381535a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:57120/0 (socket says 172.21.15.195:57120) 2022-06-27T12:13:03.377 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.348+0000 7f1a377fe700 1 -- 172.21.15.195:0/3581129384 learned_addr learned my addr 172.21.15.195:0/3581129384 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:13:03.377 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.348+0000 7f1a3c936700 1 -- 172.21.15.195:0/3581129384 >> v1:172.21.15.202:6789/0 conn(0x7f1a3806f640 legacy=0x7f1a380c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:13:03.395 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.348+0000 7f1a3c936700 1 -- 172.21.15.195:0/3581129384 >> v1:172.21.15.195:6789/0 conn(0x7f1a380c4160 legacy=0x7f1a381535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:13:03.396 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.348+0000 7f1a3c936700 1 -- 172.21.15.195:0/3581129384 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a3806ed70 con 0x7f1a38153cb0 2022-06-27T12:13:03.396 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.348+0000 7f1a36ffd700 1 -- 172.21.15.195:0/3581129384 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3523420208 0 0) 0x7f1a380c1a20 con 0x7f1a380c4160 2022-06-27T12:13:03.397 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.349+0000 7f1a3c936700 1 --2- 172.21.15.195:0/3581129384 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1a38153cb0 0x7f1a38156ed0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f1a30009e60 tx=0x7f1a30009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2108ad84fd4ddca1 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:13:03.400 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.349+0000 7f1a36ffd700 1 -- 172.21.15.195:0/3581129384 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a3000e070 con 0x7f1a38153cb0 2022-06-27T12:13:03.401 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.349+0000 7f1a36ffd700 1 -- 172.21.15.195:0/3581129384 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a3000bcc0 con 0x7f1a38153cb0 2022-06-27T12:13:03.401 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.349+0000 7f1a36ffd700 1 -- 172.21.15.195:0/3581129384 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a30012680 con 0x7f1a38153cb0 2022-06-27T12:13:03.402 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.349+0000 7f1a3e399700 1 -- 172.21.15.195:0/3581129384 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1a38153cb0 msgr2=0x7f1a38156ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:13:03.402 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.349+0000 7f1a3e399700 1 --2- 172.21.15.195:0/3581129384 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1a38153cb0 0x7f1a38156ed0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f1a30009e60 tx=0x7f1a30009780 comp rx=0 tx=0).stop 2022-06-27T12:13:03.403 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.349+0000 7f1a3e399700 1 -- 172.21.15.195:0/3581129384 shutdown_connections 2022-06-27T12:13:03.403 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.349+0000 7f1a3e399700 1 --2- 172.21.15.195:0/3581129384 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1a38153cb0 0x7f1a38156ed0 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:13:03.405 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.349+0000 7f1a3e399700 1 -- 172.21.15.195:0/3581129384 >> 172.21.15.195:0/3581129384 conn(0x7f1a380c6350 msgr2=0x7f1a380c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:13:03.406 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.349+0000 7f1a3e399700 1 -- 172.21.15.195:0/3581129384 shutdown_connections 2022-06-27T12:13:03.406 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.349+0000 7f1a3e399700 1 -- 172.21.15.195:0/3581129384 wait complete. 2022-06-27T12:13:03.407 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.351+0000 7f1a3e399700 1 Processor -- start 2022-06-27T12:13:03.407 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.351+0000 7f1a3e399700 1 -- start start 2022-06-27T12:13:03.408 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.351+0000 7f1a3e399700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1a3806f640 0x7f1a380bcbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:13:03.408 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.352+0000 7f1a3e399700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1a380c4160 0x7f1a380c4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:13:03.408 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.352+0000 7f1a3e399700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1a38153cb0 0x7f1a380bdc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:13:03.409 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.352+0000 7f1a3e399700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a380bed80 con 0x7f1a38153cb0 2022-06-27T12:13:03.410 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.352+0000 7f1a3e399700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a38167400 con 0x7f1a3806f640 2022-06-27T12:13:03.411 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.352+0000 7f1a3e399700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a381668a0 con 0x7f1a380c4160 2022-06-27T12:13:03.411 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.352+0000 7f1a3c936700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1a38153cb0 0x7f1a380bdc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:13:03.412 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.352+0000 7f1a3c936700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1a38153cb0 0x7f1a380bdc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50940/0 (socket says 172.21.15.195:50940) 2022-06-27T12:13:03.412 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.352+0000 7f1a3c936700 1 -- 172.21.15.195:0/1802739805 learned_addr learned my addr 172.21.15.195:0/1802739805 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:13:03.412 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.352+0000 7f1a37fff700 1 --2- 172.21.15.195:0/1802739805 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1a3806f640 0x7f1a380bcbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:13:03.413 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.352+0000 7f1a377fe700 1 --2- 172.21.15.195:0/1802739805 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1a380c4160 0x7f1a380c4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:13:03.413 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.352+0000 7f1a3c936700 1 -- 172.21.15.195:0/1802739805 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1a380c4160 msgr2=0x7f1a380c4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:13:03.414 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.352+0000 7f1a3c936700 1 --2- 172.21.15.195:0/1802739805 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1a380c4160 0x7f1a380c4810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:13:03.414 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.352+0000 7f1a3c936700 1 -- 172.21.15.195:0/1802739805 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1a3806f640 msgr2=0x7f1a380bcbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:13:03.416 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.352+0000 7f1a3c936700 1 --2- 172.21.15.195:0/1802739805 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1a3806f640 0x7f1a380bcbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:13:03.417 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.352+0000 7f1a3c936700 1 -- 172.21.15.195:0/1802739805 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a30009a20 con 0x7f1a38153cb0 2022-06-27T12:13:03.417 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.352+0000 7f1a3c936700 1 --2- 172.21.15.195:0/1802739805 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1a38153cb0 0x7f1a380bdc00 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f1a3806e350 tx=0x7f1a30019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:13:03.418 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.352+0000 7f1a1ffff700 1 -- 172.21.15.195:0/1802739805 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a30003b80 con 0x7f1a38153cb0 2022-06-27T12:13:03.418 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.352+0000 7f1a1ffff700 1 -- 172.21.15.195:0/1802739805 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a30003d00 con 0x7f1a38153cb0 2022-06-27T12:13:03.419 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.352+0000 7f1a3e399700 1 -- 172.21.15.195:0/1802739805 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a38166b60 con 0x7f1a38153cb0 2022-06-27T12:13:03.419 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.352+0000 7f1a1ffff700 1 -- 172.21.15.195:0/1802739805 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a30004290 con 0x7f1a38153cb0 2022-06-27T12:13:03.420 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.352+0000 7f1a3e399700 1 -- 172.21.15.195:0/1802739805 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a38167120 con 0x7f1a38153cb0 2022-06-27T12:13:03.420 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.353+0000 7f1a3e399700 1 -- 172.21.15.195:0/1802739805 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a380722c0 con 0x7f1a38153cb0 2022-06-27T12:13:03.421 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.354+0000 7f1a1ffff700 1 -- 172.21.15.195:0/1802739805 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f1a30004640 con 0x7f1a38153cb0 2022-06-27T12:13:03.422 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.354+0000 7f1a1ffff700 1 --2- 172.21.15.195:0/1802739805 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1a18063d00 0x7f1a180661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:13:03.422 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.354+0000 7f1a1ffff700 1 -- 172.21.15.195:0/1802739805 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7492+0+0 (secure 0 0 0) 0x7f1a30083380 con 0x7f1a38153cb0 2022-06-27T12:13:03.422 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.354+0000 7f1a37fff700 1 --2- 172.21.15.195:0/1802739805 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1a18063d00 0x7f1a180661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:13:03.423 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.355+0000 7f1a37fff700 1 --2- 172.21.15.195:0/1802739805 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1a18063d00 0x7f1a180661b0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f1a28001730 tx=0x7f1a28009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:13:03.423 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.359+0000 7f1a1ffff700 1 -- 172.21.15.195:0/1802739805 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f1a300519d0 con 0x7f1a38153cb0 2022-06-27T12:13:03.595 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.591+0000 7f1a3e399700 1 -- 172.21.15.195:0/1802739805 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a38152eb0 con 0x7f1a38153cb0 2022-06-27T12:13:03.596 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.592+0000 7f1a1ffff700 1 -- 172.21.15.195:0/1802739805 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+5368 (secure 0 0 0) 0x7f1a30054ff0 con 0x7f1a38153cb0 2022-06-27T12:13:03.596 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:13:03.596 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":99,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:12:59.298631+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6865},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6865":{"gid":6865,"name":"d","rank":0,"incarnation":97,"state":"up:rejoin","state_seq":16,"addr":"172.21.15.202:6835/2349973547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2349973547},{"type":"v1","addr":"172.21.15.202:6835","nonce":2349973547}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:13:03.607 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.594+0000 7f1a3e399700 1 -- 172.21.15.195:0/1802739805 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1a18063d00 msgr2=0x7f1a180661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:13:03.607 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.594+0000 7f1a3e399700 1 --2- 172.21.15.195:0/1802739805 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1a18063d00 0x7f1a180661b0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f1a28001730 tx=0x7f1a28009040 comp rx=0 tx=0).stop 2022-06-27T12:13:03.608 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.594+0000 7f1a3e399700 1 -- 172.21.15.195:0/1802739805 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1a38153cb0 msgr2=0x7f1a380bdc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:13:03.608 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.594+0000 7f1a3e399700 1 --2- 172.21.15.195:0/1802739805 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1a38153cb0 0x7f1a380bdc00 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f1a3806e350 tx=0x7f1a30019000 comp rx=0 tx=0).stop 2022-06-27T12:13:03.609 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.594+0000 7f1a3e399700 1 -- 172.21.15.195:0/1802739805 shutdown_connections 2022-06-27T12:13:03.609 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.594+0000 7f1a3e399700 1 --2- 172.21.15.195:0/1802739805 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1a18063d00 0x7f1a180661b0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:13:03.610 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.594+0000 7f1a3e399700 1 --2- 172.21.15.195:0/1802739805 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1a38153cb0 0x7f1a380bdc00 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:13:03.610 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.594+0000 7f1a3e399700 1 --2- 172.21.15.195:0/1802739805 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1a3806f640 0x7f1a380bcbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:13:03.611 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.594+0000 7f1a3e399700 1 --2- 172.21.15.195:0/1802739805 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1a380c4160 0x7f1a380c4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:13:03.611 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.594+0000 7f1a3e399700 1 -- 172.21.15.195:0/1802739805 >> 172.21.15.195:0/1802739805 conn(0x7f1a380c6350 msgr2=0x7f1a380c6720 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:13:03.612 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.594+0000 7f1a3e399700 1 -- 172.21.15.195:0/1802739805 shutdown_connections 2022-06-27T12:13:03.612 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:03.594+0000 7f1a3e399700 1 -- 172.21.15.195:0/1802739805 wait complete. 2022-06-27T12:13:03.612 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 99 2022-06-27T12:13:03.628 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:13:05.630 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:13:05.781 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:05.764+0000 7f1b024d8700 1 Processor -- start 2022-06-27T12:13:05.782 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:05.764+0000 7f1b024d8700 1 -- start start 2022-06-27T12:13:05.782 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:05.765+0000 7f1b024d8700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1afc06f640 0x7f1afc156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:13:05.782 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:05.765+0000 7f1b024d8700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1afc0c1a20 con 0x7f1afc0c4160 2022-06-27T12:13:05.782 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:05.765+0000 7f1b024d8700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1afc0c1130 con 0x7f1afc0c39e0 2022-06-27T12:13:05.783 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:05.765+0000 7f1b024d8700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1afc06ea90 con 0x7f1afc06f640 2022-06-27T12:13:05.783 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:05.765+0000 7f1afbfff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1afc06f640 0x7f1afc156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:13:05.784 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:05.765+0000 7f1afb7fe700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f1afc0c4160 0x7f1afc0c1eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:57134/0 (socket says 172.21.15.195:57134) 2022-06-27T12:13:05.784 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:05.765+0000 7f1afb7fe700 1 -- 172.21.15.195:0/2024670204 learned_addr learned my addr 172.21.15.195:0/2024670204 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:13:05.784 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:05.765+0000 7f1afbfff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1afc06f640 0x7f1afc156ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50946/0 (socket says 172.21.15.195:50946) 2022-06-27T12:13:05.784 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:05.766+0000 7f1afbfff700 1 -- 172.21.15.195:0/2024670204 >> v1:172.21.15.202:6789/0 conn(0x7f1afc0c39e0 legacy=0x7f1afc1535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:13:05.785 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:05.766+0000 7f1afbfff700 1 -- 172.21.15.195:0/2024670204 >> v1:172.21.15.195:6789/0 conn(0x7f1afc0c4160 legacy=0x7f1afc0c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:13:05.785 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:05.766+0000 7f1afbfff700 1 -- 172.21.15.195:0/2024670204 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1afc06ed70 con 0x7f1afc06f640 2022-06-27T12:13:05.785 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:05.766+0000 7f1afaffd700 1 -- 172.21.15.195:0/2024670204 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1147569742 0 0) 0x7f1afc0c1a20 con 0x7f1afc0c4160 2022-06-27T12:13:05.785 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:05.766+0000 7f1afbfff700 1 --2- 172.21.15.195:0/2024670204 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1afc06f640 0x7f1afc156ed0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f1af00097a0 tx=0x7f1af0009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f44cb8d360c5745d server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:13:05.786 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:05.766+0000 7f1afaffd700 1 -- 172.21.15.195:0/2024670204 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1af0004030 con 0x7f1afc06f640 2022-06-27T12:13:05.786 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:05.766+0000 7f1afaffd700 1 -- 172.21.15.195:0/2024670204 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1af0004190 con 0x7f1afc06f640 2022-06-27T12:13:05.786 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:05.766+0000 7f1afaffd700 1 -- 172.21.15.195:0/2024670204 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1af0011630 con 0x7f1afc06f640 2022-06-27T12:13:05.787 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:05.767+0000 7f1b024d8700 1 -- 172.21.15.195:0/2024670204 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1afc06f640 msgr2=0x7f1afc156ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:13:05.787 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:05.767+0000 7f1b024d8700 1 --2- 172.21.15.195:0/2024670204 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1afc06f640 0x7f1afc156ed0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f1af00097a0 tx=0x7f1af0009060 comp rx=0 tx=0).stop 2022-06-27T12:13:05.788 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:05.767+0000 7f1b024d8700 1 -- 172.21.15.195:0/2024670204 shutdown_connections 2022-06-27T12:13:05.789 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:05.767+0000 7f1b024d8700 1 --2- 172.21.15.195:0/2024670204 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1afc06f640 0x7f1afc156ed0 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:13:05.789 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:05.767+0000 7f1b024d8700 1 -- 172.21.15.195:0/2024670204 >> 172.21.15.195:0/2024670204 conn(0x7f1afc0c6350 msgr2=0x7f1afc0c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:13:05.789 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:05.767+0000 7f1b024d8700 1 -- 172.21.15.195:0/2024670204 shutdown_connections 2022-06-27T12:13:05.790 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:05.767+0000 7f1b024d8700 1 -- 172.21.15.195:0/2024670204 wait complete. 2022-06-27T12:13:05.790 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:05.768+0000 7f1b024d8700 1 Processor -- start 2022-06-27T12:13:05.791 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:05.768+0000 7f1b024d8700 1 -- start start 2022-06-27T12:13:05.792 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:05.769+0000 7f1b024d8700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1afc06f640 0x7f1afc069210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:13:05.792 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:05.769+0000 7f1b024d8700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1afc0c39e0 0x7f1afc06a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:13:05.792 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:05.769+0000 7f1b024d8700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1afc0c4160 0x7f1afc06ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:13:05.792 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:05.769+0000 7f1b024d8700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1afc06b3a0 con 0x7f1afc0c39e0 2022-06-27T12:13:05.793 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:05.769+0000 7f1b024d8700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1afc068d80 con 0x7f1afc0c4160 2022-06-27T12:13:05.793 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:05.769+0000 7f1b024d8700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1afc0c0b70 con 0x7f1afc06f640 2022-06-27T12:13:05.793 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:05.769+0000 7f1afb7fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1afc0c39e0 0x7f1afc06a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:13:05.794 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:05.769+0000 7f1afb7fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1afc0c39e0 0x7f1afc06a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50954/0 (socket says 172.21.15.195:50954) 2022-06-27T12:13:05.794 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:05.769+0000 7f1afb7fe700 1 -- 172.21.15.195:0/3313280371 learned_addr learned my addr 172.21.15.195:0/3313280371 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:13:05.794 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:05.769+0000 7f1afbfff700 1 --2- 172.21.15.195:0/3313280371 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1afc06f640 0x7f1afc069210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:13:05.795 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:05.769+0000 7f1b00a75700 1 --2- 172.21.15.195:0/3313280371 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1afc0c4160 0x7f1afc06ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:13:05.795 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:05.769+0000 7f1afb7fe700 1 -- 172.21.15.195:0/3313280371 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1afc06f640 msgr2=0x7f1afc069210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:13:05.795 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:05.769+0000 7f1afb7fe700 1 --2- 172.21.15.195:0/3313280371 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1afc06f640 0x7f1afc069210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:13:05.796 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:05.769+0000 7f1afb7fe700 1 -- 172.21.15.195:0/3313280371 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1afc0c4160 msgr2=0x7f1afc06ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:13:05.796 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:05.769+0000 7f1afb7fe700 1 --2- 172.21.15.195:0/3313280371 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1afc0c4160 0x7f1afc06ae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:13:05.796 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:05.769+0000 7f1afb7fe700 1 -- 172.21.15.195:0/3313280371 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1af00092e0 con 0x7f1afc0c39e0 2022-06-27T12:13:05.797 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:05.769+0000 7f1afbfff700 1 --2- 172.21.15.195:0/3313280371 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1afc06f640 0x7f1afc069210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T12:13:05.797 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:05.769+0000 7f1afb7fe700 1 --2- 172.21.15.195:0/3313280371 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1afc0c39e0 0x7f1afc06a920 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f1aec00c490 tx=0x7f1aec007dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:13:05.797 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:05.769+0000 7f1ae3fff700 1 -- 172.21.15.195:0/3313280371 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aec007360 con 0x7f1afc0c39e0 2022-06-27T12:13:05.797 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:05.769+0000 7f1ae3fff700 1 -- 172.21.15.195:0/3313280371 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1aec0074c0 con 0x7f1afc0c39e0 2022-06-27T12:13:05.799 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:05.769+0000 7f1b024d8700 1 -- 172.21.15.195:0/3313280371 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1afc067ed0 con 0x7f1afc0c39e0 2022-06-27T12:13:05.799 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:05.769+0000 7f1ae3fff700 1 -- 172.21.15.195:0/3313280371 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aec004940 con 0x7f1afc0c39e0 2022-06-27T12:13:05.799 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:05.769+0000 7f1b024d8700 1 -- 172.21.15.195:0/3313280371 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1afc068170 con 0x7f1afc0c39e0 2022-06-27T12:13:05.802 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:05.770+0000 7f1b024d8700 1 -- 172.21.15.195:0/3313280371 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1afc0722c0 con 0x7f1afc0c39e0 2022-06-27T12:13:05.803 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:05.771+0000 7f1ae3fff700 1 -- 172.21.15.195:0/3313280371 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f1aec010030 con 0x7f1afc0c39e0 2022-06-27T12:13:05.803 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:05.771+0000 7f1ae3fff700 1 --2- 172.21.15.195:0/3313280371 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1adc063d00 0x7f1adc0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:13:05.803 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:05.771+0000 7f1ae3fff700 1 -- 172.21.15.195:0/3313280371 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7492+0+0 (secure 0 0 0) 0x7f1aec080930 con 0x7f1afc0c39e0 2022-06-27T12:13:05.804 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:05.771+0000 7f1afbfff700 1 --2- 172.21.15.195:0/3313280371 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1adc063d00 0x7f1adc0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:13:05.804 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:05.772+0000 7f1afbfff700 1 --2- 172.21.15.195:0/3313280371 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1adc063d00 0x7f1adc0661b0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f1af00097a0 tx=0x7f1af0016000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:13:05.804 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:05.776+0000 7f1ae3fff700 1 -- 172.21.15.195:0/3313280371 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f1aec04f000 con 0x7f1afc0c39e0 2022-06-27T12:13:06.006 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:06.005+0000 7f1b024d8700 1 -- 172.21.15.195:0/3313280371 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1afc0704f0 con 0x7f1afc0c39e0 2022-06-27T12:13:06.007 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:06.005+0000 7f1ae3fff700 1 -- 172.21.15.195:0/3313280371 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+5370 (secure 0 0 0) 0x7f1aec052620 con 0x7f1afc0c39e0 2022-06-27T12:13:06.007 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:13:06.008 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":100,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:13:05.314357+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6865},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6865":{"gid":6865,"name":"d","rank":0,"incarnation":97,"state":"up:active","state_seq":18,"addr":"172.21.15.202:6835/2349973547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2349973547},{"type":"v1","addr":"172.21.15.202:6835","nonce":2349973547}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:13:06.010 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:06.007+0000 7f1b024d8700 1 -- 172.21.15.195:0/3313280371 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1adc063d00 msgr2=0x7f1adc0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:13:06.011 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:06.007+0000 7f1b024d8700 1 --2- 172.21.15.195:0/3313280371 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1adc063d00 0x7f1adc0661b0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f1af00097a0 tx=0x7f1af0016000 comp rx=0 tx=0).stop 2022-06-27T12:13:06.011 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:06.007+0000 7f1b024d8700 1 -- 172.21.15.195:0/3313280371 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1afc0c39e0 msgr2=0x7f1afc06a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:13:06.012 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:06.007+0000 7f1b024d8700 1 --2- 172.21.15.195:0/3313280371 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1afc0c39e0 0x7f1afc06a920 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f1aec00c490 tx=0x7f1aec007dd0 comp rx=0 tx=0).stop 2022-06-27T12:13:06.012 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:06.007+0000 7f1b024d8700 1 -- 172.21.15.195:0/3313280371 shutdown_connections 2022-06-27T12:13:06.012 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:06.007+0000 7f1b024d8700 1 --2- 172.21.15.195:0/3313280371 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1adc063d00 0x7f1adc0661b0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:13:06.013 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:06.007+0000 7f1b024d8700 1 --2- 172.21.15.195:0/3313280371 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1afc0c4160 0x7f1afc06ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:13:06.013 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:06.007+0000 7f1b024d8700 1 --2- 172.21.15.195:0/3313280371 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1afc06f640 0x7f1afc069210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:13:06.014 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:06.007+0000 7f1b024d8700 1 --2- 172.21.15.195:0/3313280371 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1afc0c39e0 0x7f1afc06a920 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:13:06.014 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:06.007+0000 7f1b024d8700 1 -- 172.21.15.195:0/3313280371 >> 172.21.15.195:0/3313280371 conn(0x7f1afc0c6350 msgr2=0x7f1afc0c5490 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:13:06.015 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:06.007+0000 7f1b024d8700 1 -- 172.21.15.195:0/3313280371 shutdown_connections 2022-06-27T12:13:06.015 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:06.007+0000 7f1b024d8700 1 -- 172.21.15.195:0/3313280371 wait complete. 2022-06-27T12:13:06.015 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 100 2022-06-27T12:13:06.026 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d has gained rank=0, replacing gid=6050 2022-06-27T12:13:06.027 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 5 secs before reviving mds.b 2022-06-27T12:13:11.033 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2022-06-27T12:13:11.035 INFO:tasks.ceph.mds.b:Restarting daemon 2022-06-27T12:13:11.040 DEBUG:teuthology.orchestra.run.smithi202:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-06-27T12:13:11.052 INFO:tasks.ceph.mds.b:Started 2022-06-27T12:13:11.053 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:13:11.128 INFO:tasks.ceph.mds.b.smithi202.stdout:starting mds.b at 2022-06-27T12:13:11.175 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.175+0000 7f7573c9f700 1 Processor -- start 2022-06-27T12:13:11.176 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.175+0000 7f7573c9f700 1 -- start start 2022-06-27T12:13:11.179 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.176+0000 7f7573c9f700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f756c06a450 0x7f756c063af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:13:11.179 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.176+0000 7f7573c9f700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f756c1508d0 0x7f756c0bbf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:13:11.180 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.176+0000 7f7573c9f700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f756c064030 con 0x7f756c150500 2022-06-27T12:13:11.180 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.176+0000 7f7573c9f700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f756c0c4160 con 0x7f756c1508d0 2022-06-27T12:13:11.180 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.176+0000 7f7573c9f700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f756c0c4440 con 0x7f756c06a450 2022-06-27T12:13:11.180 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.176+0000 7f7571a3b700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f756c150500 0x7f756c0c48d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:57158/0 (socket says 172.21.15.195:57158) 2022-06-27T12:13:11.180 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.176+0000 7f7571a3b700 1 -- 172.21.15.195:0/3367170202 learned_addr learned my addr 172.21.15.195:0/3367170202 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:13:11.181 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.176+0000 7f757123a700 1 --2- 172.21.15.195:0/3367170202 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f756c06a450 0x7f756c063af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:13:11.181 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.176+0000 7f757223c700 1 --2- 172.21.15.195:0/3367170202 >> v2:172.21.15.202:3300/0 conn(0x7f756c1508d0 0x7f756c0bbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:13:11.181 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.177+0000 7f7570a39700 1 -- 172.21.15.195:0/3367170202 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4083106362 0 0) 0x7f755c001290 con 0x7f756c150500 2022-06-27T12:13:11.181 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.177+0000 7f7570a39700 1 -- 172.21.15.195:0/3367170202 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7554003430 con 0x7f756c150500 2022-06-27T12:13:11.182 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.177+0000 7f757123a700 1 -- 172.21.15.195:0/3367170202 >> v2:172.21.15.202:3300/0 conn(0x7f756c1508d0 msgr2=0x7f756c0bbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:13:11.182 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.177+0000 7f757123a700 1 --2- 172.21.15.195:0/3367170202 >> v2:172.21.15.202:3300/0 conn(0x7f756c1508d0 0x7f756c0bbf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:13:11.182 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.177+0000 7f757123a700 1 -- 172.21.15.195:0/3367170202 >> v1:172.21.15.195:6789/0 conn(0x7f756c150500 legacy=0x7f756c0c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:13:11.182 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.177+0000 7f757123a700 1 -- 172.21.15.195:0/3367170202 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f756c15a800 con 0x7f756c06a450 2022-06-27T12:13:11.182 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.177+0000 7f757223c700 1 --2- 172.21.15.195:0/3367170202 >> v2:172.21.15.202:3300/0 conn(0x7f756c1508d0 0x7f756c0bbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:13:11.183 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.177+0000 7f757123a700 1 --2- 172.21.15.195:0/3367170202 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f756c06a450 0x7f756c063af0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f75680097a0 tx=0x7f7568009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6805d6d68ea19073 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:13:11.183 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.177+0000 7f7570a39700 1 -- 172.21.15.195:0/3367170202 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7568011630 con 0x7f756c06a450 2022-06-27T12:13:11.183 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.177+0000 7f7570a39700 1 -- 172.21.15.195:0/3367170202 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7568011790 con 0x7f756c06a450 2022-06-27T12:13:11.184 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.177+0000 7f7570a39700 1 -- 172.21.15.195:0/3367170202 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7568011b30 con 0x7f756c06a450 2022-06-27T12:13:11.184 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.178+0000 7f7573c9f700 1 -- 172.21.15.195:0/3367170202 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f756c06a450 msgr2=0x7f756c063af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:13:11.184 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.178+0000 7f7573c9f700 1 --2- 172.21.15.195:0/3367170202 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f756c06a450 0x7f756c063af0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f75680097a0 tx=0x7f7568009060 comp rx=0 tx=0).stop 2022-06-27T12:13:11.184 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.178+0000 7f7573c9f700 1 -- 172.21.15.195:0/3367170202 shutdown_connections 2022-06-27T12:13:11.185 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.178+0000 7f7573c9f700 1 --2- 172.21.15.195:0/3367170202 >> v2:172.21.15.202:3300/0 conn(0x7f756c1508d0 0x7f756c0bbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:13:11.185 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.178+0000 7f7573c9f700 1 --2- 172.21.15.195:0/3367170202 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f756c06a450 0x7f756c063af0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:13:11.185 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.178+0000 7f7573c9f700 1 -- 172.21.15.195:0/3367170202 >> 172.21.15.195:0/3367170202 conn(0x7f756c0c9b00 msgr2=0x7f756c0c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:13:11.185 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.178+0000 7f7573c9f700 1 -- 172.21.15.195:0/3367170202 shutdown_connections 2022-06-27T12:13:11.185 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.178+0000 7f7573c9f700 1 -- 172.21.15.195:0/3367170202 wait complete. 2022-06-27T12:13:11.186 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.179+0000 7f7573c9f700 1 Processor -- start 2022-06-27T12:13:11.186 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.179+0000 7f7573c9f700 1 -- start start 2022-06-27T12:13:11.189 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.179+0000 7f7573c9f700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f756c06a450 0x7f756c14f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:13:11.190 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.179+0000 7f7573c9f700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f756c150500 0x7f756c167780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:13:11.199 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.179+0000 7f7573c9f700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f756c1508d0 0x7f756c167cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:13:11.199 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.179+0000 7f7573c9f700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f756c0bfe20 con 0x7f756c1508d0 2022-06-27T12:13:11.199 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.179+0000 7f7573c9f700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f756c0c2050 con 0x7f756c150500 2022-06-27T12:13:11.200 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.179+0000 7f7573c9f700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f756c169850 con 0x7f756c06a450 2022-06-27T12:13:11.200 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.180+0000 7f757223c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f756c1508d0 0x7f756c167cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:13:11.200 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.180+0000 7f757223c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f756c1508d0 0x7f756c167cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:50980/0 (socket says 172.21.15.195:50980) 2022-06-27T12:13:11.200 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.180+0000 7f757223c700 1 -- 172.21.15.195:0/1474773938 learned_addr learned my addr 172.21.15.195:0/1474773938 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:13:11.200 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.180+0000 7f757123a700 1 --2- 172.21.15.195:0/1474773938 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f756c150500 0x7f756c167780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:13:11.201 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.180+0000 7f7571a3b700 1 --2- 172.21.15.195:0/1474773938 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f756c06a450 0x7f756c14f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:13:11.201 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.180+0000 7f757223c700 1 -- 172.21.15.195:0/1474773938 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f756c06a450 msgr2=0x7f756c14f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:13:11.201 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.180+0000 7f757223c700 1 --2- 172.21.15.195:0/1474773938 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f756c06a450 0x7f756c14f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:13:11.201 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.180+0000 7f757223c700 1 -- 172.21.15.195:0/1474773938 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f756c150500 msgr2=0x7f756c167780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:13:11.202 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.180+0000 7f757223c700 1 --2- 172.21.15.195:0/1474773938 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f756c150500 0x7f756c167780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:13:11.202 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.180+0000 7f757223c700 1 -- 172.21.15.195:0/1474773938 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7564009a20 con 0x7f756c1508d0 2022-06-27T12:13:11.202 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.180+0000 7f757223c700 1 --2- 172.21.15.195:0/1474773938 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f756c1508d0 0x7f756c167cc0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f7564009f50 tx=0x7f756400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:13:11.202 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.180+0000 7f7559ffb700 1 -- 172.21.15.195:0/1474773938 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f756400cb50 con 0x7f756c1508d0 2022-06-27T12:13:11.202 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.180+0000 7f7559ffb700 1 -- 172.21.15.195:0/1474773938 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f756400f040 con 0x7f756c1508d0 2022-06-27T12:13:11.203 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.180+0000 7f7573c9f700 1 -- 172.21.15.195:0/1474773938 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75680092e0 con 0x7f756c1508d0 2022-06-27T12:13:11.203 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.180+0000 7f7559ffb700 1 -- 172.21.15.195:0/1474773938 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75640038a0 con 0x7f756c1508d0 2022-06-27T12:13:11.203 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.180+0000 7f7573c9f700 1 -- 172.21.15.195:0/1474773938 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f756c0c1440 con 0x7f756c1508d0 2022-06-27T12:13:11.204 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.181+0000 7f7573c9f700 1 -- 172.21.15.195:0/1474773938 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f756c0c23e0 con 0x7f756c1508d0 2022-06-27T12:13:11.204 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.182+0000 7f7559ffb700 1 -- 172.21.15.195:0/1474773938 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f7564014480 con 0x7f756c1508d0 2022-06-27T12:13:11.204 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.182+0000 7f7559ffb700 1 --2- 172.21.15.195:0/1474773938 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f754c063d30 0x7f754c0661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:13:11.204 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.182+0000 7f7559ffb700 1 -- 172.21.15.195:0/1474773938 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7492+0+0 (secure 0 0 0) 0x7f75640817d0 con 0x7f756c1508d0 2022-06-27T12:13:11.204 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.182+0000 7f7571a3b700 1 --2- 172.21.15.195:0/1474773938 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f754c063d30 0x7f754c0661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:13:11.205 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.183+0000 7f7571a3b700 1 --2- 172.21.15.195:0/1474773938 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f754c063d30 0x7f754c0661e0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f755c001eb0 tx=0x7f755c0017f0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:13:11.205 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.186+0000 7f7559ffb700 1 -- 172.21.15.195:0/1474773938 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f756404fdc0 con 0x7f756c1508d0 2022-06-27T12:13:11.426 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.410+0000 7f7573c9f700 1 -- 172.21.15.195:0/1474773938 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f756c0722f0 con 0x7f756c1508d0 2022-06-27T12:13:11.426 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.411+0000 7f7559ffb700 1 -- 172.21.15.195:0/1474773938 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+6131 (secure 0 0 0) 0x7f75640533e0 con 0x7f756c1508d0 2022-06-27T12:13:11.427 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:13:11.427 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":101,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":6254,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/1317767101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1317767101},{"type":"v1","addr":"172.21.15.202:6833","nonce":1317767101}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":101}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:13:05.314357+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6865},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6865":{"gid":6865,"name":"d","rank":0,"incarnation":97,"state":"up:active","state_seq":18,"addr":"172.21.15.202:6835/2349973547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2349973547},{"type":"v1","addr":"172.21.15.202:6835","nonce":2349973547}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:13:11.439 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.413+0000 7f7573c9f700 1 -- 172.21.15.195:0/1474773938 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f754c063d30 msgr2=0x7f754c0661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:13:11.440 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.413+0000 7f7573c9f700 1 --2- 172.21.15.195:0/1474773938 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f754c063d30 0x7f754c0661e0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f755c001eb0 tx=0x7f755c0017f0 comp rx=0 tx=0).stop 2022-06-27T12:13:11.440 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.413+0000 7f7573c9f700 1 -- 172.21.15.195:0/1474773938 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f756c1508d0 msgr2=0x7f756c167cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:13:11.441 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.413+0000 7f7573c9f700 1 --2- 172.21.15.195:0/1474773938 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f756c1508d0 0x7f756c167cc0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f7564009f50 tx=0x7f756400b040 comp rx=0 tx=0).stop 2022-06-27T12:13:11.441 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.413+0000 7f7573c9f700 1 -- 172.21.15.195:0/1474773938 shutdown_connections 2022-06-27T12:13:11.442 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.413+0000 7f7573c9f700 1 --2- 172.21.15.195:0/1474773938 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f754c063d30 0x7f754c0661e0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:13:11.443 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.413+0000 7f7573c9f700 1 --2- 172.21.15.195:0/1474773938 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f756c1508d0 0x7f756c167cc0 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:13:11.443 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.413+0000 7f7573c9f700 1 --2- 172.21.15.195:0/1474773938 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f756c06a450 0x7f756c14f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:13:11.443 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.413+0000 7f7573c9f700 1 --2- 172.21.15.195:0/1474773938 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f756c150500 0x7f756c167780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:13:11.444 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.413+0000 7f7573c9f700 1 -- 172.21.15.195:0/1474773938 >> 172.21.15.195:0/1474773938 conn(0x7f756c0c9b00 msgr2=0x7f756c068200 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:13:11.444 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.413+0000 7f7573c9f700 1 -- 172.21.15.195:0/1474773938 shutdown_connections 2022-06-27T12:13:11.451 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:13:11.413+0000 7f7573c9f700 1 -- 172.21.15.195:0/1474773938 wait complete. 2022-06-27T12:13:11.452 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 101 2022-06-27T12:13:11.454 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2022-06-27T12:13:11.456 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 116 secs before thrashing 2022-06-27T12:13:14.188 INFO:tasks.ceph.mds.d.smithi202.stderr:2022-06-27T12:13:14.184+0000 7f81bbc8f700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2022-06-27T12:13:14.700 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 19% (51/258) Updating files: 20% (52/258) Updating files: 21% (55/258) Updating files: 22% (57/258) Updating files: 23% (60/258) Updating files: 24% (62/258) Updating files: 25% (65/258) Updating files: 26% (68/258) Updating files: 27% (70/258) Updating files: 27% (71/258) Updating files: 28% (73/258) Updating files: 29% (75/258) Updating files: 30% (78/258) Updating files: 31% (80/258) Updating files: 32% (83/258) Updating files: 33% (86/258) Updating files: 34% (88/258) Updating files: 35% (91/258) Updating files: 36% (93/258) Updating files: 37% (96/258) Updating files: 38% (99/258) Updating files: 39% (101/258) Updating files: 40% (104/258) Updating files: 41% (106/258) Updating files: 42% (109/258) Updating files: 43% (111/258) Updating files: 44% (114/258) Updating files: 45% (117/258) Updating files: 46% (119/258) Updating files: 47% (122/258) Updating files: 48% (124/258) Updating files: 49% (127/258) Updating files: 50% (129/258) Updating files: 50% (131/258) Updating files: 51% (132/258) Updating files: 52% (135/258) Updating files: 53% (137/258) Updating files: 54% (140/258) Updating files: 55% (142/258) Updating files: 56% (145/258) Updating files: 57% (148/258) Updating files: 58% (150/258) Updating files: 59% (153/258) Updating files: 60% (155/258) Updating files: 61% (158/258) Updating files: 62% (160/258) Updating files: 63% (163/258) Updating files: 64% (166/258) Updating files: 65% (168/258) Updating files: 66% (171/258) Updating files: 66% (172/258) Updating files: 67% (173/258) Updating files: 68% (176/258) Updating files: 69% (179/258) Updating files: 70% (181/258) Updating files: 71% (184/258) Updating files: 72% (186/258) Updating files: 73% (189/258) Updating files: 74% (191/258) Updating files: 75% (194/258) Updating files: 76% (197/258) Updating files: 77% (199/258) Updating files: 78% (202/258) Updating files: 79% (204/258) Updating files: 80% (207/258) Updating files: 81% (209/258) Updating files: 82% (212/258) Updating files: 83% (215/258) Updating files: 84% (217/258) Updating files: 85% (220/258) Updating files: 86% (222/258) Updating files: 87% (225/258) Updating files: 88% (228/258) Updating files: 89% (230/258) Updating files: 90% (233/258) Updating files: 91% (235/258) Updating files: 92% (238/258) Updating files: 93% (240/258) Updating files: 93% (242/258) Updating files: 94% (243/258) Updating files: 95% (246/258) Updating files: 96% (248/258) Updating files: 97% (251/258) Updating files: 98% (253/258) Updating files: 99% (256/258) Updating files: 100% (258/258) Updating files: 100% (258/258), done. 2022-06-27T12:13:15.110 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at 321ecdaba2c v0.39 2022-06-27T12:13:15.121 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.39 2022-06-27T12:13:15.158 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:13:15.159 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 40 -eq 48 ']' 2022-06-27T12:13:15.159 INFO:tasks.workunit.client.0.smithi195.stdout:v0.40 2022-06-27T12:13:15.160 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.40 2022-06-27T12:13:15.161 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.40 2022-06-27T12:13:15.174 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.40 2022-06-27T12:13:18.980 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 73% (166/226) Updating files: 74% (168/226) Updating files: 75% (170/226) Updating files: 76% (172/226) Updating files: 77% (175/226) Updating files: 78% (177/226) Updating files: 79% (179/226) Updating files: 80% (181/226) Updating files: 81% (184/226) Updating files: 82% (186/226) Updating files: 83% (188/226) Updating files: 84% (190/226) Updating files: 85% (193/226) Updating files: 86% (195/226) Updating files: 87% (197/226) Updating files: 88% (199/226) Updating files: 89% (202/226) Updating files: 90% (204/226) Updating files: 91% (206/226) Updating files: 92% (208/226) Updating files: 93% (211/226) Updating files: 94% (213/226) Updating files: 95% (215/226) Updating files: 96% (217/226) Updating files: 97% (220/226) Updating files: 98% (222/226) Updating files: 99% (224/226) Updating files: 100% (226/226) Updating files: 100% (226/226), done. 2022-06-27T12:13:19.007 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at 7eea40ea37f v0.40 2022-06-27T12:13:19.012 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.40 2022-06-27T12:13:19.080 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:13:19.081 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 41 -eq 48 ']' 2022-06-27T12:13:19.082 INFO:tasks.workunit.client.0.smithi195.stdout:v0.41 2022-06-27T12:13:19.087 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.41 2022-06-27T12:13:19.087 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.41 2022-06-27T12:13:19.088 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.41 2022-06-27T12:13:21.016 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 57% (50/87) Updating files: 58% (51/87) Updating files: 59% (52/87) Updating files: 60% (53/87) Updating files: 62% (54/87) Updating files: 63% (55/87) Updating files: 64% (56/87) Updating files: 65% (57/87) Updating files: 66% (58/87) Updating files: 67% (59/87) Updating files: 68% (60/87) Updating files: 70% (61/87) Updating files: 71% (62/87) Updating files: 72% (63/87) Updating files: 73% (64/87) Updating files: 74% (65/87) Updating files: 75% (66/87) Updating files: 77% (67/87) Updating files: 78% (68/87) Updating files: 79% (69/87) Updating files: 80% (70/87) Updating files: 81% (71/87) Updating files: 82% (72/87) Updating files: 83% (73/87) Updating files: 85% (74/87) Updating files: 86% (75/87) Updating files: 87% (76/87) Updating files: 88% (77/87) Updating files: 89% (78/87) Updating files: 90% (79/87) Updating files: 91% (80/87) Updating files: 93% (81/87) Updating files: 94% (82/87) Updating files: 95% (83/87) Updating files: 96% (84/87) Updating files: 97% (85/87) Updating files: 98% (86/87) Updating files: 100% (87/87) Updating files: 100% (87/87), done. 2022-06-27T12:13:21.045 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at c1345f7136a v0.41 2022-06-27T12:13:21.046 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.41 2022-06-27T12:13:21.108 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:13:21.109 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 42 -eq 48 ']' 2022-06-27T12:13:21.109 INFO:tasks.workunit.client.0.smithi195.stdout:v0.42 2022-06-27T12:13:21.110 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.42 2022-06-27T12:13:21.110 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.42 2022-06-27T12:13:21.111 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.42 2022-06-27T12:13:23.620 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 54% (135/250) Updating files: 55% (138/250) Updating files: 56% (140/250) Updating files: 57% (143/250) Updating files: 58% (145/250) Updating files: 59% (148/250) Updating files: 60% (150/250) Updating files: 61% (153/250) Updating files: 62% (155/250) Updating files: 63% (158/250) Updating files: 64% (160/250) Updating files: 65% (163/250) Updating files: 66% (165/250) Updating files: 67% (168/250) Updating files: 68% (170/250) Updating files: 69% (173/250) Updating files: 70% (175/250) Updating files: 71% (178/250) Updating files: 72% (180/250) Updating files: 73% (183/250) Updating files: 74% (185/250) Updating files: 75% (188/250) Updating files: 76% (190/250) Updating files: 77% (193/250) Updating files: 78% (195/250) Updating files: 79% (198/250) Updating files: 80% (200/250) Updating files: 81% (203/250) Updating files: 82% (205/250) Updating files: 83% (208/250) Updating files: 84% (210/250) Updating files: 85% (213/250) Updating files: 86% (215/250) Updating files: 87% (218/250) Updating files: 88% (220/250) Updating files: 89% (223/250) Updating files: 90% (225/250) Updating files: 91% (228/250) Updating files: 92% (230/250) Updating files: 93% (233/250) Updating files: 94% (235/250) Updating files: 95% (238/250) Updating files: 96% (240/250) Updating files: 97% (243/250) Updating files: 98% (245/250) Updating files: 99% (248/250) Updating files: 100% (250/250) Updating files: 100% (250/250), done. 2022-06-27T12:13:23.645 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at b205c64c687 v0.42 2022-06-27T12:13:23.646 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.42 2022-06-27T12:13:23.763 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:13:23.765 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 43 -eq 48 ']' 2022-06-27T12:13:23.765 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.43 2022-06-27T12:13:23.766 INFO:tasks.workunit.client.0.smithi195.stdout:v0.43 2022-06-27T12:13:23.767 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.43 2022-06-27T12:13:23.767 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.43 2022-06-27T12:13:26.172 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 64% (72/111) Updating files: 65% (73/111) Updating files: 66% (74/111) Updating files: 67% (75/111) Updating files: 68% (76/111) Updating files: 69% (77/111) Updating files: 70% (78/111) Updating files: 71% (79/111) Updating files: 72% (80/111) Updating files: 73% (82/111) Updating files: 74% (83/111) Updating files: 75% (84/111) Updating files: 76% (85/111) Updating files: 77% (86/111) Updating files: 78% (87/111) Updating files: 79% (88/111) Updating files: 80% (89/111) Updating files: 81% (90/111) Updating files: 82% (92/111) Updating files: 83% (93/111) Updating files: 84% (94/111) Updating files: 85% (95/111) Updating files: 86% (96/111) Updating files: 87% (97/111) Updating files: 88% (98/111) Updating files: 89% (99/111) Updating files: 90% (100/111) Updating files: 91% (102/111) Updating files: 92% (103/111) Updating files: 93% (104/111) Updating files: 94% (105/111) Updating files: 95% (106/111) Updating files: 96% (107/111) Updating files: 97% (108/111) Updating files: 98% (109/111) Updating files: 99% (110/111) Updating files: 100% (111/111) Updating files: 100% (111/111), done. 2022-06-27T12:13:26.201 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at 9fa8781c014 v0.43 2022-06-27T12:13:26.204 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.43 2022-06-27T12:13:26.261 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:13:26.262 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 44 -eq 48 ']' 2022-06-27T12:13:26.262 INFO:tasks.workunit.client.0.smithi195.stdout:v0.44 2022-06-27T12:13:26.263 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.44 2022-06-27T12:13:26.263 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.44 2022-06-27T12:13:26.263 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.44 2022-06-27T12:13:30.845 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 39% (73/186) Updating files: 40% (75/186) Updating files: 41% (77/186) Updating files: 42% (79/186) Updating files: 43% (80/186) Updating files: 44% (82/186) Updating files: 45% (84/186) Updating files: 46% (86/186) Updating files: 47% (88/186) Updating files: 48% (90/186) Updating files: 49% (92/186) Updating files: 50% (93/186) Updating files: 51% (95/186) Updating files: 52% (97/186) Updating files: 53% (99/186) Updating files: 54% (101/186) Updating files: 55% (103/186) Updating files: 56% (105/186) Updating files: 57% (107/186) Updating files: 58% (108/186) Updating files: 59% (110/186) Updating files: 60% (112/186) Updating files: 61% (114/186) Updating files: 62% (116/186) Updating files: 63% (118/186) Updating files: 64% (120/186) Updating files: 65% (121/186) Updating files: 66% (123/186) Updating files: 67% (125/186) Updating files: 68% (127/186) Updating files: 69% (129/186) Updating files: 70% (131/186) Updating files: 71% (133/186) Updating files: 72% (134/186) Updating files: 72% (135/186) Updating files: 73% (136/186) Updating files: 74% (138/186) Updating files: 75% (140/186) Updating files: 76% (142/186) Updating files: 77% (144/186) Updating files: 78% (146/186) Updating files: 79% (147/186) Updating files: 80% (149/186) Updating files: 81% (151/186) Updating files: 82% (153/186) Updating files: 83% (155/186) Updating files: 84% (157/186) Updating files: 85% (159/186) Updating files: 86% (160/186) Updating files: 87% (162/186) Updating files: 88% (164/186) Updating files: 89% (166/186) Updating files: 90% (168/186) Updating files: 91% (170/186) Updating files: 92% (172/186) Updating files: 93% (173/186) Updating files: 94% (175/186) Updating files: 95% (177/186) Updating files: 96% (179/186) Updating files: 97% (181/186) Updating files: 98% (183/186) Updating files: 99% (185/186) Updating files: 100% (186/186) Updating files: 100% (186/186), done. 2022-06-27T12:13:30.865 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at cdd5298d276 v0.44 2022-06-27T12:13:30.872 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.44 2022-06-27T12:13:30.979 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:13:30.980 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 45 -eq 48 ']' 2022-06-27T12:13:30.981 INFO:tasks.workunit.client.0.smithi195.stdout:v0.45 2022-06-27T12:13:30.982 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.45 2022-06-27T12:13:30.983 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.45 2022-06-27T12:13:30.983 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.45 2022-06-27T12:13:31.766 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at 0aea1cb1df5 v0.45 2022-06-27T12:13:31.767 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.45 2022-06-27T12:13:31.882 INFO:tasks.workunit.client.0.smithi195.stdout:v0.46 2022-06-27T12:13:31.883 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:13:31.883 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 46 -eq 48 ']' 2022-06-27T12:13:31.883 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.46 2022-06-27T12:13:31.884 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.46 2022-06-27T12:13:31.886 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.46 2022-06-27T12:13:33.478 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:13:33.485 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:13:35.710 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 16% (50/309) Updating files: 17% (53/309) Updating files: 18% (56/309) Updating files: 19% (59/309) Updating files: 20% (62/309) Updating files: 21% (65/309) Updating files: 22% (68/309) Updating files: 23% (72/309) Updating files: 24% (75/309) Updating files: 25% (78/309) Updating files: 26% (81/309) Updating files: 27% (84/309) Updating files: 28% (87/309) Updating files: 29% (90/309) Updating files: 30% (93/309) Updating files: 31% (96/309) Updating files: 32% (99/309) Updating files: 33% (102/309) Updating files: 34% (106/309) Updating files: 35% (109/309) Updating files: 36% (112/309) Updating files: 37% (115/309) Updating files: 38% (118/309) Updating files: 39% (121/309) Updating files: 40% (124/309) Updating files: 41% (127/309) Updating files: 42% (130/309) Updating files: 43% (133/309) Updating files: 44% (136/309) Updating files: 45% (140/309) Updating files: 46% (143/309) Updating files: 47% (146/309) Updating files: 48% (149/309) Updating files: 48% (150/309) Updating files: 49% (152/309) Updating files: 50% (155/309) Updating files: 51% (158/309) Updating files: 52% (161/309) Updating files: 53% (164/309) Updating files: 54% (167/309) Updating files: 55% (170/309) Updating files: 56% (174/309) Updating files: 57% (177/309) Updating files: 58% (180/309) Updating files: 59% (183/309) Updating files: 60% (186/309) Updating files: 61% (189/309) Updating files: 62% (192/309) Updating files: 63% (195/309) Updating files: 64% (198/309) Updating files: 65% (201/309) Updating files: 66% (204/309) Updating files: 67% (208/309) Updating files: 68% (211/309) Updating files: 69% (214/309) Updating files: 70% (217/309) Updating files: 71% (220/309) Updating files: 72% (223/309) Updating files: 73% (226/309) Updating files: 74% (229/309) Updating files: 75% (232/309) Updating files: 76% (235/309) Updating files: 77% (238/309) Updating files: 78% (242/309) Updating files: 79% (245/309) Updating files: 80% (248/309) Updating files: 81% (251/309) Updating files: 82% (254/309) Updating files: 83% (257/309) Updating files: 84% (260/309) Updating files: 85% (263/309) Updating files: 86% (266/309) Updating files: 87% (269/309) Updating files: 88% (272/309) Updating files: 89% (276/309) Updating files: 89% (277/309) Updating files: 90% (279/309) Updating files: 91% (282/309) Updating files: 92% (285/309) Updating files: 93% (288/309) Updating files: 94% (291/309) Updating files: 95% (294/309) Updating files: 96% (297/309) Updating files: 97% (300/309) Updating files: 98% (303/309) Updating files: 99% (306/309) Updating files: 100% (309/309) Updating files: 100% (309/309), done. 2022-06-27T12:13:35.740 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at cb7f1c9c752 v0.46 2022-06-27T12:13:35.747 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.46 2022-06-27T12:13:36.293 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:13:36.296 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 47 -eq 48 ']' 2022-06-27T12:13:36.297 INFO:tasks.workunit.client.0.smithi195.stdout:v0.47 2022-06-27T12:13:36.299 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.47 2022-06-27T12:13:36.300 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.47 2022-06-27T12:13:36.301 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.47 2022-06-27T12:13:40.236 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 25% (55/219) Updating files: 26% (57/219) Updating files: 27% (60/219) Updating files: 28% (62/219) Updating files: 29% (64/219) Updating files: 30% (66/219) Updating files: 31% (68/219) Updating files: 32% (71/219) Updating files: 33% (73/219) Updating files: 34% (75/219) Updating files: 35% (77/219) Updating files: 36% (79/219) Updating files: 37% (82/219) Updating files: 37% (83/219) Updating files: 38% (84/219) Updating files: 39% (86/219) Updating files: 40% (88/219) Updating files: 41% (90/219) Updating files: 42% (92/219) Updating files: 43% (95/219) Updating files: 44% (97/219) Updating files: 45% (99/219) Updating files: 46% (101/219) Updating files: 47% (103/219) Updating files: 48% (106/219) Updating files: 49% (108/219) Updating files: 50% (110/219) Updating files: 51% (112/219) Updating files: 52% (114/219) Updating files: 53% (117/219) Updating files: 54% (119/219) Updating files: 55% (121/219) Updating files: 56% (123/219) Updating files: 57% (125/219) Updating files: 58% (128/219) Updating files: 59% (130/219) Updating files: 60% (132/219) Updating files: 61% (134/219) Updating files: 62% (136/219) Updating files: 63% (138/219) Updating files: 64% (141/219) Updating files: 65% (143/219) Updating files: 66% (145/219) Updating files: 67% (147/219) Updating files: 68% (149/219) Updating files: 69% (152/219) Updating files: 70% (154/219) Updating files: 71% (156/219) Updating files: 72% (158/219) Updating files: 73% (160/219) Updating files: 74% (163/219) Updating files: 75% (165/219) Updating files: 76% (167/219) Updating files: 77% (169/219) Updating files: 78% (171/219) Updating files: 79% (174/219) Updating files: 80% (176/219) Updating files: 81% (178/219) Updating files: 82% (180/219) Updating files: 83% (182/219) Updating files: 84% (184/219) Updating files: 85% (187/219) Updating files: 86% (189/219) Updating files: 87% (191/219) Updating files: 88% (193/219) Updating files: 89% (195/219) Updating files: 90% (198/219) Updating files: 91% (200/219) Updating files: 92% (202/219) Updating files: 93% (204/219) Updating files: 94% (206/219) Updating files: 95% (209/219) Updating files: 96% (211/219) Updating files: 97% (213/219) Updating files: 98% (215/219) Updating files: 99% (217/219) Updating files: 100% (219/219) Updating files: 100% (219/219), done. 2022-06-27T12:13:40.247 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at ad663d58c89 v0.47 2022-06-27T12:13:40.254 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.47 2022-06-27T12:13:40.362 INFO:tasks.workunit.client.0.smithi195.stdout:v0.49 2022-06-27T12:13:40.363 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:13:40.364 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 48 -eq 48 ']' 2022-06-27T12:13:40.364 INFO:tasks.workunit.client.0.smithi195.stderr:+ continue 2022-06-27T12:13:40.365 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:13:40.365 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 49 -eq 48 ']' 2022-06-27T12:13:40.365 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.49 2022-06-27T12:13:40.368 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.49 2022-06-27T12:13:40.369 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.49 2022-06-27T12:13:46.740 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 4% (21/481) Updating files: 5% (25/481) Updating files: 6% (29/481) Updating files: 7% (34/481) Updating files: 8% (39/481) Updating files: 9% (44/481) Updating files: 10% (49/481) Updating files: 10% (51/481) Updating files: 11% (53/481) Updating files: 12% (58/481) Updating files: 13% (63/481) Updating files: 14% (68/481) Updating files: 15% (73/481) Updating files: 16% (77/481) Updating files: 17% (82/481) Updating files: 18% (87/481) Updating files: 19% (92/481) Updating files: 20% (97/481) Updating files: 21% (102/481) Updating files: 22% (106/481) Updating files: 23% (111/481) Updating files: 24% (116/481) Updating files: 25% (121/481) Updating files: 26% (126/481) Updating files: 27% (130/481) Updating files: 28% (135/481) Updating files: 29% (140/481) Updating files: 30% (145/481) Updating files: 31% (150/481) Updating files: 32% (154/481) Updating files: 33% (159/481) Updating files: 34% (164/481) Updating files: 34% (168/481) Updating files: 35% (169/481) Updating files: 36% (174/481) Updating files: 37% (178/481) Updating files: 38% (183/481) Updating files: 39% (188/481) Updating files: 40% (193/481) Updating files: 41% (198/481) Updating files: 42% (203/481) Updating files: 43% (207/481) Updating files: 44% (212/481) Updating files: 45% (217/481) Updating files: 46% (222/481) Updating files: 47% (227/481) Updating files: 48% (231/481) Updating files: 49% (236/481) Updating files: 50% (241/481) Updating files: 51% (246/481) Updating files: 52% (251/481) Updating files: 53% (255/481) Updating files: 54% (260/481) Updating files: 55% (265/481) Updating files: 56% (270/481) Updating files: 57% (275/481) Updating files: 58% (279/481) Updating files: 59% (284/481) Updating files: 60% (289/481) Updating files: 61% (294/481) Updating files: 62% (299/481) Updating files: 63% (304/481) Updating files: 64% (308/481) Updating files: 65% (313/481) Updating files: 65% (316/481) Updating files: 66% (318/481) Updating files: 67% (323/481) Updating files: 68% (328/481) Updating files: 69% (332/481) Updating files: 70% (337/481) Updating files: 71% (342/481) Updating files: 72% (347/481) Updating files: 73% (352/481) Updating files: 74% (356/481) Updating files: 75% (361/481) Updating files: 76% (366/481) Updating files: 77% (371/481) Updating files: 78% (376/481) Updating files: 79% (380/481) Updating files: 80% (385/481) Updating files: 81% (390/481) Updating files: 82% (395/481) Updating files: 82% (397/481) Updating files: 83% (400/481) Updating files: 84% (405/481) Updating files: 85% (409/481) Updating files: 86% (414/481) Updating files: 87% (419/481) Updating files: 88% (424/481) Updating files: 89% (429/481) Updating files: 90% (433/481) Updating files: 91% (438/481) Updating files: 92% (443/481) Updating files: 93% (448/481) Updating files: 94% (453/481) Updating files: 95% (457/481) Updating files: 96% (462/481) Updating files: 97% (467/481) Updating files: 98% (472/481) Updating files: 99% (477/481) Updating files: 100% (481/481) Updating files: 100% (481/481), done. 2022-06-27T12:13:46.785 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at ca6265d0f4d v0.49 2022-06-27T12:13:46.786 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.49 2022-06-27T12:13:47.138 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:13:47.139 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 50 -eq 48 ']' 2022-06-27T12:13:47.139 INFO:tasks.workunit.client.0.smithi195.stdout:v0.50 2022-06-27T12:13:47.140 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.50 2022-06-27T12:13:47.140 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.50 2022-06-27T12:13:47.140 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.50 2022-06-27T12:13:49.136 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 24% (33/137) Updating files: 25% (35/137) Updating files: 26% (36/137) Updating files: 27% (37/137) Updating files: 28% (39/137) Updating files: 29% (40/137) Updating files: 30% (42/137) Updating files: 31% (43/137) Updating files: 32% (44/137) Updating files: 33% (46/137) Updating files: 34% (47/137) Updating files: 35% (48/137) Updating files: 36% (50/137) Updating files: 37% (51/137) Updating files: 38% (53/137) Updating files: 39% (54/137) Updating files: 40% (55/137) Updating files: 41% (57/137) Updating files: 42% (58/137) Updating files: 43% (59/137) Updating files: 44% (61/137) Updating files: 45% (62/137) Updating files: 46% (64/137) Updating files: 47% (65/137) Updating files: 48% (66/137) Updating files: 49% (68/137) Updating files: 50% (69/137) Updating files: 51% (70/137) Updating files: 52% (72/137) Updating files: 53% (73/137) Updating files: 54% (74/137) Updating files: 55% (76/137) Updating files: 56% (77/137) Updating files: 57% (79/137) Updating files: 58% (80/137) Updating files: 59% (81/137) Updating files: 60% (83/137) Updating files: 61% (84/137) Updating files: 62% (85/137) Updating files: 63% (87/137) Updating files: 64% (88/137) Updating files: 65% (90/137) Updating files: 66% (91/137) Updating files: 67% (92/137) Updating files: 68% (94/137) Updating files: 69% (95/137) Updating files: 70% (96/137) Updating files: 71% (98/137) Updating files: 72% (99/137) Updating files: 73% (101/137) Updating files: 74% (102/137) Updating files: 75% (103/137) Updating files: 76% (105/137) Updating files: 77% (106/137) Updating files: 78% (107/137) Updating files: 79% (109/137) Updating files: 80% (110/137) Updating files: 81% (111/137) Updating files: 82% (113/137) Updating files: 83% (114/137) Updating files: 84% (116/137) Updating files: 85% (117/137) Updating files: 86% (118/137) Updating files: 87% (120/137) Updating files: 88% (121/137) Updating files: 89% (122/137) Updating files: 90% (124/137) Updating files: 91% (125/137) Updating files: 92% (127/137) Updating files: 93% (128/137) Updating files: 94% (129/137) Updating files: 95% (131/137) Updating files: 96% (132/137) Updating files: 97% (133/137) Updating files: 98% (135/137) Updating files: 99% (136/137) Updating files: 100% (137/137) Updating files: 100% (137/137), done. 2022-06-27T12:13:49.161 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at a4428bd0e19 v0.50 2022-06-27T12:13:49.167 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.50 2022-06-27T12:13:50.141 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:13:50.141 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 51 -eq 48 ']' 2022-06-27T12:13:50.142 INFO:tasks.workunit.client.0.smithi195.stdout:v0.51 2022-06-27T12:13:50.143 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.51 2022-06-27T12:13:50.143 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.51 2022-06-27T12:13:50.143 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.51 2022-06-27T12:13:53.240 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 21% (23/105) Updating files: 22% (24/105) Updating files: 23% (25/105) Updating files: 24% (26/105) Updating files: 25% (27/105) Updating files: 26% (28/105) Updating files: 27% (29/105) Updating files: 28% (30/105) Updating files: 29% (31/105) Updating files: 30% (32/105) Updating files: 31% (33/105) Updating files: 32% (34/105) Updating files: 33% (35/105) Updating files: 34% (36/105) Updating files: 35% (37/105) Updating files: 36% (38/105) Updating files: 37% (39/105) Updating files: 38% (40/105) Updating files: 39% (41/105) Updating files: 40% (42/105) Updating files: 41% (44/105) Updating files: 42% (45/105) Updating files: 43% (46/105) Updating files: 44% (47/105) Updating files: 45% (48/105) Updating files: 46% (49/105) Updating files: 47% (50/105) Updating files: 48% (51/105) Updating files: 49% (52/105) Updating files: 50% (53/105) Updating files: 51% (54/105) Updating files: 52% (55/105) Updating files: 53% (56/105) Updating files: 54% (57/105) Updating files: 55% (58/105) Updating files: 56% (59/105) Updating files: 57% (60/105) Updating files: 58% (61/105) Updating files: 59% (62/105) Updating files: 60% (63/105) Updating files: 61% (65/105) Updating files: 62% (66/105) Updating files: 63% (67/105) Updating files: 64% (68/105) Updating files: 65% (69/105) Updating files: 66% (70/105) Updating files: 67% (71/105) Updating files: 68% (72/105) Updating files: 69% (73/105) Updating files: 70% (74/105) Updating files: 71% (75/105) Updating files: 72% (76/105) Updating files: 73% (77/105) Updating files: 74% (78/105) Updating files: 75% (79/105) Updating files: 76% (80/105) Updating files: 77% (81/105) Updating files: 78% (82/105) Updating files: 79% (83/105) Updating files: 80% (84/105) Updating files: 81% (86/105) Updating files: 82% (87/105) Updating files: 83% (88/105) Updating files: 84% (89/105) Updating files: 85% (90/105) Updating files: 86% (91/105) Updating files: 87% (92/105) Updating files: 88% (93/105) Updating files: 89% (94/105) Updating files: 90% (95/105) Updating files: 91% (96/105) Updating files: 92% (97/105) Updating files: 93% (98/105) Updating files: 94% (99/105) Updating files: 95% (100/105) Updating files: 96% (101/105) Updating files: 97% (102/105) Updating files: 98% (103/105) Updating files: 99% (104/105) Updating files: 100% (105/105) Updating files: 100% (105/105), done. 2022-06-27T12:13:53.483 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at c03ca95d235 v0.51 2022-06-27T12:13:53.487 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.51 2022-06-27T12:13:53.592 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:13:53.593 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 52 -eq 48 ']' 2022-06-27T12:13:53.594 INFO:tasks.workunit.client.0.smithi195.stdout:v0.52 2022-06-27T12:13:53.595 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.52 2022-06-27T12:13:53.598 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.52 2022-06-27T12:13:53.599 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.52 2022-06-27T12:13:59.794 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 34% (64/183) Updating files: 35% (65/183) Updating files: 36% (66/183) Updating files: 37% (68/183) Updating files: 38% (70/183) Updating files: 39% (72/183) Updating files: 40% (74/183) Updating files: 41% (76/183) Updating files: 42% (77/183) Updating files: 43% (79/183) Updating files: 44% (81/183) Updating files: 45% (83/183) Updating files: 46% (85/183) Updating files: 47% (87/183) Updating files: 48% (88/183) Updating files: 49% (90/183) Updating files: 50% (92/183) Updating files: 51% (94/183) Updating files: 52% (96/183) Updating files: 53% (97/183) Updating files: 54% (99/183) Updating files: 55% (101/183) Updating files: 56% (103/183) Updating files: 57% (105/183) Updating files: 58% (107/183) Updating files: 59% (108/183) Updating files: 60% (110/183) Updating files: 61% (112/183) Updating files: 62% (114/183) Updating files: 63% (116/183) Updating files: 64% (118/183) Updating files: 65% (119/183) Updating files: 66% (121/183) Updating files: 67% (123/183) Updating files: 68% (125/183) Updating files: 69% (127/183) Updating files: 70% (129/183) Updating files: 71% (130/183) Updating files: 72% (132/183) Updating files: 73% (134/183) Updating files: 74% (136/183) Updating files: 74% (137/183) Updating files: 75% (138/183) Updating files: 76% (140/183) Updating files: 77% (141/183) Updating files: 78% (143/183) Updating files: 79% (145/183) Updating files: 80% (147/183) Updating files: 81% (149/183) Updating files: 82% (151/183) Updating files: 83% (152/183) Updating files: 84% (154/183) Updating files: 85% (156/183) Updating files: 86% (158/183) Updating files: 87% (160/183) Updating files: 88% (162/183) Updating files: 89% (163/183) Updating files: 89% (164/183) Updating files: 90% (165/183) Updating files: 91% (167/183) Updating files: 92% (169/183) Updating files: 93% (171/183) Updating files: 94% (173/183) Updating files: 95% (174/183) Updating files: 96% (176/183) Updating files: 97% (178/183) Updating files: 98% (180/183) Updating files: 99% (182/183) Updating files: 100% (183/183) Updating files: 100% (183/183), done. 2022-06-27T12:13:59.823 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at e48859474c4 v0.52 2022-06-27T12:13:59.824 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.52 2022-06-27T12:13:59.915 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:13:59.946 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 53 -eq 48 ']' 2022-06-27T12:13:59.947 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.53 2022-06-27T12:13:59.947 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.53 2022-06-27T12:13:59.948 INFO:tasks.workunit.client.0.smithi195.stdout:v0.53 2022-06-27T12:13:59.949 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.53 2022-06-27T12:14:03.558 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:14:03.635 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:14:07.148 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 7% (28/386) Updating files: 8% (31/386) Updating files: 9% (35/386) Updating files: 10% (39/386) Updating files: 11% (43/386) Updating files: 12% (47/386) Updating files: 13% (51/386) Updating files: 13% (53/386) Updating files: 14% (55/386) Updating files: 15% (58/386) Updating files: 16% (62/386) Updating files: 17% (66/386) Updating files: 18% (70/386) Updating files: 19% (74/386) Updating files: 20% (78/386) Updating files: 21% (82/386) Updating files: 22% (85/386) Updating files: 23% (89/386) Updating files: 24% (93/386) Updating files: 25% (97/386) Updating files: 26% (101/386) Updating files: 27% (105/386) Updating files: 28% (109/386) Updating files: 29% (112/386) Updating files: 30% (116/386) Updating files: 31% (120/386) Updating files: 31% (121/386) Updating files: 32% (124/386) Updating files: 33% (128/386) Updating files: 34% (132/386) Updating files: 35% (136/386) Updating files: 36% (139/386) Updating files: 37% (143/386) Updating files: 38% (147/386) Updating files: 39% (151/386) Updating files: 40% (155/386) Updating files: 41% (159/386) Updating files: 42% (163/386) Updating files: 43% (166/386) Updating files: 44% (170/386) Updating files: 45% (174/386) Updating files: 46% (178/386) Updating files: 47% (182/386) Updating files: 48% (186/386) Updating files: 49% (190/386) Updating files: 50% (193/386) Updating files: 51% (197/386) Updating files: 52% (201/386) Updating files: 53% (205/386) Updating files: 54% (209/386) Updating files: 55% (213/386) Updating files: 56% (217/386) Updating files: 56% (219/386) Updating files: 57% (221/386) Updating files: 58% (224/386) Updating files: 59% (228/386) Updating files: 60% (232/386) Updating files: 61% (236/386) Updating files: 62% (240/386) Updating files: 63% (244/386) Updating files: 64% (248/386) Updating files: 65% (251/386) Updating files: 66% (255/386) Updating files: 67% (259/386) Updating files: 67% (262/386) Updating files: 68% (263/386) Updating files: 69% (267/386) Updating files: 70% (271/386) Updating files: 70% (272/386) Updating files: 71% (275/386) Updating files: 72% (278/386) Updating files: 73% (282/386) Updating files: 74% (286/386) Updating files: 75% (290/386) Updating files: 76% (294/386) Updating files: 77% (298/386) Updating files: 78% (302/386) Updating files: 79% (305/386) Updating files: 80% (309/386) Updating files: 81% (313/386) Updating files: 82% (317/386) Updating files: 83% (321/386) Updating files: 84% (325/386) Updating files: 85% (329/386) Updating files: 86% (332/386) Updating files: 87% (336/386) Updating files: 88% (340/386) Updating files: 89% (344/386) Updating files: 90% (348/386) Updating files: 91% (352/386) Updating files: 92% (356/386) Updating files: 93% (359/386) Updating files: 94% (363/386) Updating files: 95% (367/386) Updating files: 96% (371/386) Updating files: 97% (375/386) Updating files: 98% (379/386) Updating files: 99% (383/386) Updating files: 100% (386/386) Updating files: 100% (386/386), done. 2022-06-27T12:14:07.186 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at 2528b5ee105 v0.43 2022-06-27T12:14:07.194 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.53 2022-06-27T12:14:07.546 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:14:07.547 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 54 -eq 48 ']' 2022-06-27T12:14:07.547 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.54 2022-06-27T12:14:07.547 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.54 2022-06-27T12:14:07.548 INFO:tasks.workunit.client.0.smithi195.stdout:v0.54 2022-06-27T12:14:07.549 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.54 2022-06-27T12:14:11.891 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 5% (12/212) Updating files: 6% (13/212) Updating files: 7% (15/212) Updating files: 8% (17/212) Updating files: 9% (20/212) Updating files: 10% (22/212) Updating files: 11% (24/212) Updating files: 12% (26/212) Updating files: 13% (28/212) Updating files: 14% (30/212) Updating files: 15% (32/212) Updating files: 16% (34/212) Updating files: 17% (37/212) Updating files: 18% (39/212) Updating files: 19% (41/212) Updating files: 20% (43/212) Updating files: 21% (45/212) Updating files: 22% (47/212) Updating files: 23% (49/212) Updating files: 24% (51/212) Updating files: 25% (53/212) Updating files: 26% (56/212) Updating files: 27% (58/212) Updating files: 28% (60/212) Updating files: 29% (62/212) Updating files: 30% (64/212) Updating files: 31% (66/212) Updating files: 32% (68/212) Updating files: 33% (70/212) Updating files: 34% (73/212) Updating files: 35% (75/212) Updating files: 36% (77/212) Updating files: 37% (79/212) Updating files: 38% (81/212) Updating files: 39% (83/212) Updating files: 40% (85/212) Updating files: 41% (87/212) Updating files: 42% (90/212) Updating files: 43% (92/212) Updating files: 44% (94/212) Updating files: 45% (96/212) Updating files: 46% (98/212) Updating files: 47% (100/212) Updating files: 48% (102/212) Updating files: 49% (104/212) Updating files: 50% (106/212) Updating files: 51% (109/212) Updating files: 52% (111/212) Updating files: 53% (113/212) Updating files: 54% (115/212) Updating files: 55% (117/212) Updating files: 56% (119/212) Updating files: 57% (121/212) Updating files: 58% (123/212) Updating files: 59% (126/212) Updating files: 60% (128/212) Updating files: 61% (130/212) Updating files: 62% (132/212) Updating files: 63% (134/212) Updating files: 64% (136/212) Updating files: 65% (138/212) Updating files: 66% (140/212) Updating files: 67% (143/212) Updating files: 68% (145/212) Updating files: 69% (147/212) Updating files: 70% (149/212) Updating files: 71% (151/212) Updating files: 72% (153/212) Updating files: 73% (155/212) Updating files: 74% (157/212) Updating files: 75% (159/212) Updating files: 76% (162/212) Updating files: 77% (164/212) Updating files: 78% (166/212) Updating files: 79% (168/212) Updating files: 80% (170/212) Updating files: 81% (172/212) Updating files: 82% (174/212) Updating files: 83% (176/212) Updating files: 84% (179/212) Updating files: 85% (181/212) Updating files: 86% (183/212) Updating files: 87% (185/212) Updating files: 88% (187/212) Updating files: 89% (189/212) Updating files: 90% (191/212) Updating files: 91% (193/212) Updating files: 92% (196/212) Updating files: 93% (198/212) Updating files: 94% (200/212) Updating files: 95% (202/212) Updating files: 96% (204/212) Updating files: 97% (206/212) Updating files: 98% (208/212) Updating files: 99% (210/212) Updating files: 100% (212/212) Updating files: 100% (212/212), done. 2022-06-27T12:14:11.906 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at 60b84b095b1 v0.54 2022-06-27T12:14:11.909 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.54 2022-06-27T12:14:12.094 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:14:12.094 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 55 -eq 48 ']' 2022-06-27T12:14:12.095 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.55 2022-06-27T12:14:12.095 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.55 2022-06-27T12:14:12.095 INFO:tasks.workunit.client.0.smithi195.stdout:v0.55 2022-06-27T12:14:12.096 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.55 2022-06-27T12:14:20.187 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 15% (85/540) Updating files: 16% (87/540) Updating files: 17% (92/540) Updating files: 18% (98/540) Updating files: 19% (103/540) Updating files: 20% (108/540) Updating files: 21% (114/540) Updating files: 22% (119/540) Updating files: 23% (125/540) Updating files: 24% (130/540) Updating files: 25% (135/540) Updating files: 26% (141/540) Updating files: 27% (146/540) Updating files: 28% (152/540) Updating files: 29% (157/540) Updating files: 30% (162/540) Updating files: 31% (168/540) Updating files: 32% (173/540) Updating files: 33% (179/540) Updating files: 34% (184/540) Updating files: 35% (189/540) Updating files: 36% (195/540) Updating files: 37% (200/540) Updating files: 38% (206/540) Updating files: 39% (211/540) Updating files: 40% (216/540) Updating files: 41% (222/540) Updating files: 42% (227/540) Updating files: 43% (233/540) Updating files: 44% (238/540) Updating files: 45% (243/540) Updating files: 46% (249/540) Updating files: 47% (254/540) Updating files: 48% (260/540) Updating files: 49% (265/540) Updating files: 49% (268/540) Updating files: 50% (270/540) Updating files: 51% (276/540) Updating files: 52% (281/540) Updating files: 53% (287/540) Updating files: 54% (292/540) Updating files: 55% (297/540) Updating files: 56% (303/540) Updating files: 57% (308/540) Updating files: 58% (314/540) Updating files: 59% (319/540) Updating files: 60% (324/540) Updating files: 61% (330/540) Updating files: 62% (335/540) Updating files: 63% (341/540) Updating files: 64% (346/540) Updating files: 65% (351/540) Updating files: 66% (357/540) Updating files: 67% (362/540) Updating files: 68% (368/540) Updating files: 69% (373/540) Updating files: 70% (378/540) Updating files: 71% (384/540) Updating files: 72% (389/540) Updating files: 73% (395/540) Updating files: 74% (400/540) Updating files: 75% (405/540) Updating files: 76% (411/540) Updating files: 77% (416/540) Updating files: 78% (422/540) Updating files: 79% (427/540) Updating files: 80% (432/540) Updating files: 81% (438/540) Updating files: 81% (439/540) Updating files: 82% (443/540) Updating files: 83% (449/540) Updating files: 84% (454/540) Updating files: 85% (459/540) Updating files: 86% (465/540) Updating files: 87% (470/540) Updating files: 88% (476/540) Updating files: 89% (481/540) Updating files: 90% (486/540) Updating files: 91% (492/540) Updating files: 92% (497/540) Updating files: 93% (503/540) Updating files: 94% (508/540) Updating files: 95% (513/540) Updating files: 96% (519/540) Updating files: 97% (524/540) Updating files: 97% (526/540) Updating files: 98% (530/540) Updating files: 99% (535/540) Updating files: 100% (540/540) Updating files: 100% (540/540), done. 2022-06-27T12:14:21.187 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at 690f8175606 v0.55 2022-06-27T12:14:21.194 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.55 2022-06-27T12:14:21.243 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:14:21.244 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 56 -eq 48 ']' 2022-06-27T12:14:21.245 INFO:tasks.workunit.client.0.smithi195.stdout:v0.56 2022-06-27T12:14:21.247 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.56 2022-06-27T12:14:21.247 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.56 2022-06-27T12:14:21.248 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.56 2022-06-27T12:14:24.341 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 57% (142/246) Updating files: 58% (143/246) Updating files: 59% (146/246) Updating files: 60% (148/246) Updating files: 61% (151/246) Updating files: 62% (153/246) Updating files: 63% (155/246) Updating files: 64% (158/246) Updating files: 65% (160/246) Updating files: 66% (163/246) Updating files: 67% (165/246) Updating files: 68% (168/246) Updating files: 69% (170/246) Updating files: 70% (173/246) Updating files: 71% (175/246) Updating files: 72% (178/246) Updating files: 73% (180/246) Updating files: 74% (183/246) Updating files: 75% (185/246) Updating files: 76% (187/246) Updating files: 77% (190/246) Updating files: 78% (192/246) Updating files: 79% (195/246) Updating files: 80% (197/246) Updating files: 81% (200/246) Updating files: 82% (202/246) Updating files: 83% (205/246) Updating files: 84% (207/246) Updating files: 85% (210/246) Updating files: 86% (212/246) Updating files: 87% (215/246) Updating files: 88% (217/246) Updating files: 89% (219/246) Updating files: 90% (222/246) Updating files: 91% (224/246) Updating files: 92% (227/246) Updating files: 93% (229/246) Updating files: 94% (232/246) Updating files: 95% (234/246) Updating files: 96% (237/246) Updating files: 97% (239/246) Updating files: 98% (242/246) Updating files: 99% (244/246) Updating files: 100% (246/246) Updating files: 100% (246/246), done. 2022-06-27T12:14:24.647 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at 1a32f0a0b42 v0.56 2022-06-27T12:14:24.654 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.56 2022-06-27T12:14:24.932 INFO:tasks.workunit.client.0.smithi195.stdout:v0.57 2022-06-27T12:14:24.933 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:14:24.933 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 57 -eq 48 ']' 2022-06-27T12:14:24.933 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.57 2022-06-27T12:14:24.933 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.57 2022-06-27T12:14:24.934 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.57 2022-06-27T12:14:29.429 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 21% (66/304) Updating files: 22% (67/304) Updating files: 23% (70/304) Updating files: 24% (73/304) Updating files: 25% (76/304) Updating files: 26% (80/304) Updating files: 27% (83/304) Updating files: 28% (86/304) Updating files: 29% (89/304) Updating files: 30% (92/304) Updating files: 31% (95/304) Updating files: 32% (98/304) Updating files: 33% (101/304) Updating files: 34% (104/304) Updating files: 35% (107/304) Updating files: 36% (110/304) Updating files: 37% (113/304) Updating files: 38% (116/304) Updating files: 39% (119/304) Updating files: 39% (120/304) Updating files: 40% (122/304) Updating files: 41% (125/304) Updating files: 42% (128/304) Updating files: 43% (131/304) Updating files: 44% (134/304) Updating files: 45% (137/304) Updating files: 46% (140/304) Updating files: 47% (143/304) Updating files: 48% (146/304) Updating files: 49% (149/304) Updating files: 50% (152/304) Updating files: 51% (156/304) Updating files: 52% (159/304) Updating files: 53% (162/304) Updating files: 54% (165/304) Updating files: 55% (168/304) Updating files: 56% (171/304) Updating files: 57% (174/304) Updating files: 58% (177/304) Updating files: 59% (180/304) Updating files: 60% (183/304) Updating files: 61% (186/304) Updating files: 62% (189/304) Updating files: 63% (192/304) Updating files: 64% (195/304) Updating files: 65% (198/304) Updating files: 66% (201/304) Updating files: 66% (202/304) Updating files: 67% (204/304) Updating files: 68% (207/304) Updating files: 69% (210/304) Updating files: 70% (213/304) Updating files: 71% (216/304) Updating files: 72% (219/304) Updating files: 73% (222/304) Updating files: 74% (225/304) Updating files: 75% (228/304) Updating files: 76% (232/304) Updating files: 77% (235/304) Updating files: 78% (238/304) Updating files: 79% (241/304) Updating files: 80% (244/304) Updating files: 81% (247/304) Updating files: 82% (250/304) Updating files: 83% (253/304) Updating files: 84% (256/304) Updating files: 85% (259/304) Updating files: 86% (262/304) Updating files: 87% (265/304) Updating files: 88% (268/304) Updating files: 89% (271/304) Updating files: 90% (274/304) Updating files: 91% (277/304) Updating files: 92% (280/304) Updating files: 93% (283/304) Updating files: 94% (286/304) Updating files: 95% (289/304) Updating files: 96% (292/304) Updating files: 97% (295/304) Updating files: 98% (298/304) Updating files: 99% (301/304) Updating files: 100% (304/304) Updating files: 100% (304/304), done. 2022-06-27T12:14:29.457 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at 9a7a9d06c06 v0.57 2022-06-27T12:14:29.469 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.57 2022-06-27T12:14:29.613 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:14:29.616 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 58 -eq 48 ']' 2022-06-27T12:14:29.616 INFO:tasks.workunit.client.0.smithi195.stdout:v0.58 2022-06-27T12:14:29.617 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.58 2022-06-27T12:14:29.617 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.58 2022-06-27T12:14:29.617 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.58 2022-06-27T12:14:33.889 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:14:33.920 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:14:36.903 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 25% (68/268) Updating files: 26% (70/268) Updating files: 27% (73/268) Updating files: 28% (76/268) Updating files: 29% (78/268) Updating files: 30% (81/268) Updating files: 31% (84/268) Updating files: 32% (86/268) Updating files: 33% (89/268) Updating files: 34% (92/268) Updating files: 35% (94/268) Updating files: 36% (97/268) Updating files: 37% (100/268) Updating files: 38% (102/268) Updating files: 39% (105/268) Updating files: 40% (108/268) Updating files: 41% (110/268) Updating files: 42% (113/268) Updating files: 43% (116/268) Updating files: 44% (118/268) Updating files: 45% (121/268) Updating files: 45% (123/268) Updating files: 46% (124/268) Updating files: 47% (126/268) Updating files: 48% (129/268) Updating files: 49% (132/268) Updating files: 50% (134/268) Updating files: 51% (137/268) Updating files: 52% (140/268) Updating files: 52% (142/268) Updating files: 53% (143/268) Updating files: 54% (145/268) Updating files: 55% (148/268) Updating files: 56% (151/268) Updating files: 57% (153/268) Updating files: 58% (156/268) Updating files: 59% (159/268) Updating files: 60% (161/268) Updating files: 61% (164/268) Updating files: 62% (167/268) Updating files: 63% (169/268) Updating files: 64% (172/268) Updating files: 65% (175/268) Updating files: 66% (177/268) Updating files: 67% (180/268) Updating files: 68% (183/268) Updating files: 69% (185/268) Updating files: 70% (188/268) Updating files: 71% (191/268) Updating files: 72% (193/268) Updating files: 73% (196/268) Updating files: 74% (199/268) Updating files: 75% (201/268) Updating files: 76% (204/268) Updating files: 77% (207/268) Updating files: 78% (210/268) Updating files: 79% (212/268) Updating files: 80% (215/268) Updating files: 81% (218/268) Updating files: 82% (220/268) Updating files: 83% (223/268) Updating files: 84% (226/268) Updating files: 85% (228/268) Updating files: 86% (231/268) Updating files: 87% (234/268) Updating files: 88% (236/268) Updating files: 89% (239/268) Updating files: 90% (242/268) Updating files: 90% (243/268) Updating files: 91% (244/268) Updating files: 91% (245/268) Updating files: 92% (247/268) Updating files: 93% (250/268) Updating files: 94% (252/268) Updating files: 95% (255/268) Updating files: 96% (258/268) Updating files: 97% (260/268) Updating files: 98% (263/268) Updating files: 99% (266/268) Updating files: 100% (268/268) Updating files: 100% (268/268), done. 2022-06-27T12:14:36.922 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at ba3f91e7504 v0.58 2022-06-27T12:14:36.923 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.58 2022-06-27T12:14:37.779 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:14:37.779 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 59 -eq 48 ']' 2022-06-27T12:14:37.780 INFO:tasks.workunit.client.0.smithi195.stdout:v0.59 2022-06-27T12:14:37.781 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.59 2022-06-27T12:14:37.781 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.59 2022-06-27T12:14:37.782 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.59 2022-06-27T12:14:42.723 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 20% (44/211) Updating files: 21% (45/211) Updating files: 22% (47/211) Updating files: 23% (49/211) Updating files: 24% (51/211) Updating files: 25% (53/211) Updating files: 26% (55/211) Updating files: 27% (57/211) Updating files: 28% (60/211) Updating files: 29% (62/211) Updating files: 30% (64/211) Updating files: 31% (66/211) Updating files: 32% (68/211) Updating files: 33% (70/211) Updating files: 34% (72/211) Updating files: 35% (74/211) Updating files: 36% (76/211) Updating files: 37% (79/211) Updating files: 38% (81/211) Updating files: 39% (83/211) Updating files: 40% (85/211) Updating files: 41% (87/211) Updating files: 42% (89/211) Updating files: 43% (91/211) Updating files: 43% (92/211) Updating files: 44% (93/211) Updating files: 45% (95/211) Updating files: 46% (98/211) Updating files: 47% (100/211) Updating files: 48% (102/211) Updating files: 49% (104/211) Updating files: 50% (106/211) Updating files: 50% (107/211) Updating files: 51% (108/211) Updating files: 52% (110/211) Updating files: 53% (112/211) Updating files: 54% (114/211) Updating files: 55% (117/211) Updating files: 56% (119/211) Updating files: 57% (121/211) Updating files: 58% (123/211) Updating files: 59% (125/211) Updating files: 60% (127/211) Updating files: 61% (129/211) Updating files: 62% (131/211) Updating files: 63% (133/211) Updating files: 64% (136/211) Updating files: 65% (138/211) Updating files: 66% (140/211) Updating files: 67% (142/211) Updating files: 68% (144/211) Updating files: 69% (146/211) Updating files: 70% (148/211) Updating files: 71% (150/211) Updating files: 72% (152/211) Updating files: 73% (155/211) Updating files: 74% (157/211) Updating files: 75% (159/211) Updating files: 76% (161/211) Updating files: 77% (163/211) Updating files: 78% (165/211) Updating files: 79% (167/211) Updating files: 80% (169/211) Updating files: 81% (171/211) Updating files: 82% (174/211) Updating files: 83% (176/211) Updating files: 84% (178/211) Updating files: 85% (180/211) Updating files: 86% (182/211) Updating files: 87% (184/211) Updating files: 88% (186/211) Updating files: 89% (188/211) Updating files: 90% (190/211) Updating files: 91% (193/211) Updating files: 92% (195/211) Updating files: 93% (197/211) Updating files: 94% (199/211) Updating files: 95% (201/211) Updating files: 96% (203/211) Updating files: 97% (205/211) Updating files: 98% (207/211) Updating files: 99% (209/211) Updating files: 100% (211/211) Updating files: 100% (211/211), done. 2022-06-27T12:14:42.745 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at cbae6a435c6 v0.59 2022-06-27T12:14:42.746 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.59 2022-06-27T12:14:42.861 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:14:42.862 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 60 -eq 48 ']' 2022-06-27T12:14:42.862 INFO:tasks.workunit.client.0.smithi195.stdout:v0.60 2022-06-27T12:14:42.863 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.60 2022-06-27T12:14:42.863 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.60 2022-06-27T12:14:42.864 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.60 2022-06-27T12:14:45.334 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 31% (81/256) Updating files: 32% (82/256) Updating files: 33% (85/256) Updating files: 34% (88/256) Updating files: 35% (90/256) Updating files: 36% (93/256) Updating files: 37% (95/256) Updating files: 38% (98/256) Updating files: 39% (100/256) Updating files: 40% (103/256) Updating files: 41% (105/256) Updating files: 42% (108/256) Updating files: 43% (111/256) Updating files: 44% (113/256) Updating files: 45% (116/256) Updating files: 46% (118/256) Updating files: 47% (121/256) Updating files: 48% (123/256) Updating files: 49% (126/256) Updating files: 50% (128/256) Updating files: 51% (131/256) Updating files: 52% (134/256) Updating files: 53% (136/256) Updating files: 54% (139/256) Updating files: 55% (141/256) Updating files: 56% (144/256) Updating files: 57% (146/256) Updating files: 58% (149/256) Updating files: 59% (152/256) Updating files: 60% (154/256) Updating files: 61% (157/256) Updating files: 62% (159/256) Updating files: 63% (162/256) Updating files: 64% (164/256) Updating files: 65% (167/256) Updating files: 66% (169/256) Updating files: 67% (172/256) Updating files: 68% (175/256) Updating files: 69% (177/256) Updating files: 70% (180/256) Updating files: 71% (182/256) Updating files: 72% (185/256) Updating files: 73% (187/256) Updating files: 74% (190/256) Updating files: 75% (192/256) Updating files: 76% (195/256) Updating files: 77% (198/256) Updating files: 78% (200/256) Updating files: 79% (203/256) Updating files: 80% (205/256) Updating files: 81% (208/256) Updating files: 82% (210/256) Updating files: 83% (213/256) Updating files: 84% (216/256) Updating files: 85% (218/256) Updating files: 86% (221/256) Updating files: 87% (223/256) Updating files: 88% (226/256) Updating files: 89% (228/256) Updating files: 90% (231/256) Updating files: 91% (233/256) Updating files: 92% (236/256) Updating files: 93% (239/256) Updating files: 94% (241/256) Updating files: 95% (244/256) Updating files: 96% (246/256) Updating files: 97% (249/256) Updating files: 98% (251/256) Updating files: 99% (254/256) Updating files: 100% (256/256) Updating files: 100% (256/256), done. 2022-06-27T12:14:45.367 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at f26f7a39021 v0.60 2022-06-27T12:14:45.368 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.60 2022-06-27T12:14:45.484 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:14:45.485 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 61 -eq 48 ']' 2022-06-27T12:14:45.485 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.61 2022-06-27T12:14:45.486 INFO:tasks.workunit.client.0.smithi195.stdout:v0.61 2022-06-27T12:14:45.487 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.61 2022-06-27T12:14:45.487 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.61 2022-06-27T12:14:54.731 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 39% (149/376) Updating files: 40% (151/376) Updating files: 41% (155/376) Updating files: 42% (158/376) Updating files: 43% (162/376) Updating files: 44% (166/376) Updating files: 45% (170/376) Updating files: 46% (173/376) Updating files: 46% (174/376) Updating files: 46% (175/376) Updating files: 47% (177/376) Updating files: 48% (181/376) Updating files: 49% (185/376) Updating files: 50% (188/376) Updating files: 51% (192/376) Updating files: 52% (196/376) Updating files: 53% (200/376) Updating files: 54% (204/376) Updating files: 55% (207/376) Updating files: 56% (211/376) Updating files: 57% (215/376) Updating files: 58% (219/376) Updating files: 59% (222/376) Updating files: 60% (226/376) Updating files: 61% (230/376) Updating files: 62% (234/376) Updating files: 63% (237/376) Updating files: 64% (241/376) Updating files: 65% (245/376) Updating files: 66% (249/376) Updating files: 67% (252/376) Updating files: 68% (256/376) Updating files: 69% (260/376) Updating files: 70% (264/376) Updating files: 71% (267/376) Updating files: 71% (268/376) Updating files: 72% (271/376) Updating files: 73% (275/376) Updating files: 74% (279/376) Updating files: 75% (282/376) Updating files: 76% (286/376) Updating files: 77% (290/376) Updating files: 78% (294/376) Updating files: 79% (298/376) Updating files: 80% (301/376) Updating files: 81% (305/376) Updating files: 82% (309/376) Updating files: 83% (313/376) Updating files: 84% (316/376) Updating files: 85% (320/376) Updating files: 86% (324/376) Updating files: 87% (328/376) Updating files: 88% (331/376) Updating files: 89% (335/376) Updating files: 90% (339/376) Updating files: 91% (343/376) Updating files: 92% (346/376) Updating files: 93% (350/376) Updating files: 94% (354/376) Updating files: 95% (358/376) Updating files: 95% (359/376) Updating files: 96% (361/376) Updating files: 96% (364/376) Updating files: 97% (365/376) Updating files: 98% (369/376) Updating files: 99% (373/376) Updating files: 100% (376/376) Updating files: 100% (376/376), done. 2022-06-27T12:14:54.774 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at 237f3f1e8d8 v0.61 2022-06-27T12:14:54.776 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.61 2022-06-27T12:14:54.881 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:14:54.881 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 62 -eq 48 ']' 2022-06-27T12:14:54.882 INFO:tasks.workunit.client.0.smithi195.stdout:v0.62 2022-06-27T12:14:54.882 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.62 2022-06-27T12:14:54.882 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.62 2022-06-27T12:14:54.890 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.62 2022-06-27T12:14:55.446 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at 9382379cc6b v0.62 2022-06-27T12:14:55.450 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.62 2022-06-27T12:14:56.228 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:14:56.229 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 63 -eq 48 ']' 2022-06-27T12:14:56.230 INFO:tasks.workunit.client.0.smithi195.stdout:v0.63 2022-06-27T12:14:56.231 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.63 2022-06-27T12:14:56.233 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.63 2022-06-27T12:14:56.234 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.63 2022-06-27T12:14:57.997 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 55% (81/146) Updating files: 56% (82/146) Updating files: 57% (84/146) Updating files: 58% (85/146) Updating files: 59% (87/146) Updating files: 60% (88/146) Updating files: 61% (90/146) Updating files: 62% (91/146) Updating files: 63% (92/146) Updating files: 64% (94/146) Updating files: 65% (95/146) Updating files: 66% (97/146) Updating files: 67% (98/146) Updating files: 68% (100/146) Updating files: 69% (101/146) Updating files: 70% (103/146) Updating files: 71% (104/146) Updating files: 72% (106/146) Updating files: 73% (107/146) Updating files: 74% (109/146) Updating files: 75% (110/146) Updating files: 76% (111/146) Updating files: 77% (113/146) Updating files: 78% (114/146) Updating files: 79% (116/146) Updating files: 80% (117/146) Updating files: 81% (119/146) Updating files: 82% (120/146) Updating files: 83% (122/146) Updating files: 84% (123/146) Updating files: 85% (125/146) Updating files: 86% (126/146) Updating files: 87% (128/146) Updating files: 88% (129/146) Updating files: 89% (130/146) Updating files: 90% (132/146) Updating files: 91% (133/146) Updating files: 92% (135/146) Updating files: 93% (136/146) Updating files: 94% (138/146) Updating files: 95% (139/146) Updating files: 96% (141/146) Updating files: 97% (142/146) Updating files: 98% (144/146) Updating files: 99% (145/146) Updating files: 100% (146/146) Updating files: 100% (146/146), done. 2022-06-27T12:14:58.030 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at 054e96cf79e v0.63 2022-06-27T12:14:58.038 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.63 2022-06-27T12:14:58.139 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:14:58.140 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 64 -eq 48 ']' 2022-06-27T12:14:58.140 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.64 2022-06-27T12:14:58.141 INFO:tasks.workunit.client.0.smithi195.stdout:v0.64 2022-06-27T12:14:58.142 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.64 2022-06-27T12:14:58.142 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.64 2022-06-27T12:15:01.498 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 3% (5/143) Updating files: 4% (6/143) Updating files: 5% (8/143) Updating files: 6% (9/143) Updating files: 7% (11/143) Updating files: 8% (12/143) Updating files: 9% (13/143) Updating files: 10% (15/143) Updating files: 11% (16/143) Updating files: 12% (18/143) Updating files: 13% (19/143) Updating files: 14% (21/143) Updating files: 15% (22/143) Updating files: 16% (23/143) Updating files: 17% (25/143) Updating files: 18% (26/143) Updating files: 19% (28/143) Updating files: 20% (29/143) Updating files: 21% (31/143) Updating files: 22% (32/143) Updating files: 23% (33/143) Updating files: 24% (35/143) Updating files: 25% (36/143) Updating files: 26% (38/143) Updating files: 27% (39/143) Updating files: 28% (41/143) Updating files: 29% (42/143) Updating files: 30% (43/143) Updating files: 31% (45/143) Updating files: 32% (46/143) Updating files: 33% (48/143) Updating files: 34% (49/143) Updating files: 35% (51/143) Updating files: 36% (52/143) Updating files: 37% (53/143) Updating files: 37% (54/143) Updating files: 38% (55/143) Updating files: 39% (56/143) Updating files: 40% (58/143) Updating files: 41% (59/143) Updating files: 42% (61/143) Updating files: 43% (62/143) Updating files: 44% (63/143) Updating files: 45% (65/143) Updating files: 46% (66/143) Updating files: 47% (68/143) Updating files: 48% (69/143) Updating files: 49% (71/143) Updating files: 50% (72/143) Updating files: 51% (73/143) Updating files: 52% (75/143) Updating files: 53% (76/143) Updating files: 54% (78/143) Updating files: 55% (79/143) Updating files: 56% (81/143) Updating files: 57% (82/143) Updating files: 58% (83/143) Updating files: 59% (85/143) Updating files: 60% (86/143) Updating files: 61% (88/143) Updating files: 62% (89/143) Updating files: 63% (91/143) Updating files: 64% (92/143) Updating files: 65% (93/143) Updating files: 66% (95/143) Updating files: 67% (96/143) Updating files: 68% (98/143) Updating files: 69% (99/143) Updating files: 70% (101/143) Updating files: 71% (102/143) Updating files: 72% (103/143) Updating files: 73% (105/143) Updating files: 74% (106/143) Updating files: 75% (108/143) Updating files: 76% (109/143) Updating files: 77% (111/143) Updating files: 78% (112/143) Updating files: 79% (113/143) Updating files: 80% (115/143) Updating files: 81% (116/143) Updating files: 82% (118/143) Updating files: 83% (119/143) Updating files: 84% (121/143) Updating files: 85% (122/143) Updating files: 86% (123/143) Updating files: 87% (125/143) Updating files: 88% (126/143) Updating files: 89% (128/143) Updating files: 90% (129/143) Updating files: 91% (131/143) Updating files: 92% (132/143) Updating files: 93% (133/143) Updating files: 93% (134/143) Updating files: 94% (135/143) Updating files: 95% (136/143) Updating files: 96% (138/143) Updating files: 97% (139/143) Updating files: 98% (141/143) Updating files: 99% (142/143) Updating files: 100% (143/143) Updating files: 100% (143/143), done. 2022-06-27T12:15:01.534 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at 42e06c12db6 v0.64 2022-06-27T12:15:01.537 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.64 2022-06-27T12:15:01.611 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:15:01.612 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 65 -eq 48 ']' 2022-06-27T12:15:01.612 INFO:tasks.workunit.client.0.smithi195.stdout:v0.65 2022-06-27T12:15:01.613 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.65 2022-06-27T12:15:01.613 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.65 2022-06-27T12:15:01.613 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.65 2022-06-27T12:15:02.516 INFO:tasks.ceph.mds.f.smithi202.stderr:2022-06-27T12:15:02.516+0000 7f38e894d700 -1 received signal: Hangup from killall ceph-mds -1 (PID: 101199) UID: 0 2022-06-27T12:15:02.517 INFO:tasks.ceph.mds.d.smithi202.stderr:2022-06-27T12:15:02.518+0000 7f81bfc97700 -1 received signal: Hangup from killall ceph-mds -1 (PID: 101199) UID: 0 2022-06-27T12:15:02.518 INFO:tasks.ceph.mds.b.smithi202.stderr:2022-06-27T12:15:02.518+0000 7f02e078c700 -1 received signal: Hangup from killall ceph-mds -1 (PID: 101199) UID: 0 2022-06-27T12:15:03.300 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 68% (139/203) Updating files: 69% (141/203) Updating files: 70% (143/203) Updating files: 71% (145/203) Updating files: 72% (147/203) Updating files: 73% (149/203) Updating files: 74% (151/203) Updating files: 75% (153/203) Updating files: 76% (155/203) Updating files: 77% (157/203) Updating files: 78% (159/203) Updating files: 79% (161/203) Updating files: 80% (163/203) Updating files: 81% (165/203) Updating files: 82% (167/203) Updating files: 83% (169/203) Updating files: 84% (171/203) Updating files: 85% (173/203) Updating files: 86% (175/203) Updating files: 87% (177/203) Updating files: 88% (179/203) Updating files: 89% (181/203) Updating files: 90% (183/203) Updating files: 91% (185/203) Updating files: 92% (187/203) Updating files: 93% (189/203) Updating files: 94% (191/203) Updating files: 95% (193/203) Updating files: 96% (195/203) Updating files: 97% (197/203) Updating files: 98% (199/203) Updating files: 99% (201/203) Updating files: 100% (203/203) Updating files: 100% (203/203), done. 2022-06-27T12:15:03.335 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at c2d517ef966 v0.65 2022-06-27T12:15:03.336 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.65 2022-06-27T12:15:03.434 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:15:03.435 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 66 -eq 48 ']' 2022-06-27T12:15:03.435 INFO:tasks.workunit.client.0.smithi195.stdout:v0.66 2022-06-27T12:15:03.436 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.66 2022-06-27T12:15:03.436 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.66 2022-06-27T12:15:03.436 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.66 2022-06-27T12:15:04.002 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:15:04.013 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:15:06.252 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 13% (20/152) Updating files: 14% (22/152) Updating files: 15% (23/152) Updating files: 16% (25/152) Updating files: 17% (26/152) Updating files: 18% (28/152) Updating files: 19% (29/152) Updating files: 20% (31/152) Updating files: 21% (32/152) Updating files: 22% (34/152) Updating files: 23% (35/152) Updating files: 24% (37/152) Updating files: 25% (38/152) Updating files: 26% (40/152) Updating files: 27% (42/152) Updating files: 28% (43/152) Updating files: 29% (45/152) Updating files: 30% (46/152) Updating files: 31% (48/152) Updating files: 32% (49/152) Updating files: 33% (51/152) Updating files: 34% (52/152) Updating files: 35% (54/152) Updating files: 36% (55/152) Updating files: 37% (57/152) Updating files: 38% (58/152) Updating files: 39% (60/152) Updating files: 40% (61/152) Updating files: 41% (63/152) Updating files: 42% (64/152) Updating files: 43% (66/152) Updating files: 44% (67/152) Updating files: 45% (69/152) Updating files: 46% (70/152) Updating files: 47% (72/152) Updating files: 48% (73/152) Updating files: 49% (75/152) Updating files: 50% (76/152) Updating files: 51% (78/152) Updating files: 52% (80/152) Updating files: 53% (81/152) Updating files: 54% (83/152) Updating files: 55% (84/152) Updating files: 56% (86/152) Updating files: 57% (87/152) Updating files: 58% (89/152) Updating files: 59% (90/152) Updating files: 60% (92/152) Updating files: 61% (93/152) Updating files: 62% (95/152) Updating files: 63% (96/152) Updating files: 64% (98/152) Updating files: 65% (99/152) Updating files: 66% (101/152) Updating files: 67% (102/152) Updating files: 68% (104/152) Updating files: 69% (105/152) Updating files: 70% (107/152) Updating files: 71% (108/152) Updating files: 72% (110/152) Updating files: 73% (111/152) Updating files: 74% (113/152) Updating files: 75% (114/152) Updating files: 76% (116/152) Updating files: 77% (118/152) Updating files: 78% (119/152) Updating files: 79% (121/152) Updating files: 80% (122/152) Updating files: 81% (124/152) Updating files: 82% (125/152) Updating files: 83% (127/152) Updating files: 84% (128/152) Updating files: 85% (130/152) Updating files: 86% (131/152) Updating files: 87% (133/152) Updating files: 88% (134/152) Updating files: 89% (136/152) Updating files: 90% (137/152) Updating files: 91% (139/152) Updating files: 92% (140/152) Updating files: 93% (142/152) Updating files: 94% (143/152) Updating files: 95% (145/152) Updating files: 96% (146/152) Updating files: 97% (148/152) Updating files: 98% (149/152) Updating files: 99% (151/152) Updating files: 100% (152/152) Updating files: 100% (152/152), done. 2022-06-27T12:15:06.292 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at b6b48dbefad v0.66 2022-06-27T12:15:06.303 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.66 2022-06-27T12:15:06.626 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:15:06.626 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 67 -eq 48 ']' 2022-06-27T12:15:06.627 INFO:tasks.workunit.client.0.smithi195.stdout:v0.67 2022-06-27T12:15:06.628 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.67 2022-06-27T12:15:06.628 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.67 2022-06-27T12:15:06.628 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.67 2022-06-27T12:15:07.458 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:15:07.669 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.587+0000 7fdde1cb2700 1 Processor -- start 2022-06-27T12:15:07.670 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.587+0000 7fdde1cb2700 1 -- start start 2022-06-27T12:15:07.671 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.588+0000 7fdde1cb2700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fdddc150580 0x7fdddc156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:07.671 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.588+0000 7fdde1cb2700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fdddc150950 0x7fdddc0c7b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:07.682 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.588+0000 7fdde1cb2700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdddc0c76b0 con 0x7fdddc06f620 2022-06-27T12:15:07.683 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.588+0000 7fdde1cb2700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdddc157430 con 0x7fdddc150580 2022-06-27T12:15:07.683 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.588+0000 7fdde1cb2700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdddc0bad70 con 0x7fdddc150950 2022-06-27T12:15:07.684 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.588+0000 7fdddbfff700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fdddc06f620 0x7fdddc06d6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:57686/0 (socket says 172.21.15.195:57686) 2022-06-27T12:15:07.684 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.588+0000 7fdddbfff700 1 -- 172.21.15.195:0/1430100469 learned_addr learned my addr 172.21.15.195:0/1430100469 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:15:07.685 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.588+0000 7fdddb7fe700 1 --2- 172.21.15.195:0/1430100469 >> v2:172.21.15.202:3300/0 conn(0x7fdddc150580 0x7fdddc156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:07.685 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.588+0000 7fdddaffd700 1 --2- 172.21.15.195:0/1430100469 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fdddc150950 0x7fdddc0c7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:07.685 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.588+0000 7fddda7fc700 1 -- 172.21.15.195:0/1430100469 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4099601832 0 0) 0x7fdddc0c76b0 con 0x7fdddc06f620 2022-06-27T12:15:07.691 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.588+0000 7fddda7fc700 1 -- 172.21.15.195:0/1430100469 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddc8003430 con 0x7fdddc06f620 2022-06-27T12:15:07.691 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.588+0000 7fdddaffd700 1 -- 172.21.15.195:0/1430100469 >> v2:172.21.15.202:3300/0 conn(0x7fdddc150580 msgr2=0x7fdddc156ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:15:07.691 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.588+0000 7fdddaffd700 1 --2- 172.21.15.195:0/1430100469 >> v2:172.21.15.202:3300/0 conn(0x7fdddc150580 0x7fdddc156ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:07.692 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.588+0000 7fdddaffd700 1 -- 172.21.15.195:0/1430100469 >> v1:172.21.15.195:6789/0 conn(0x7fdddc06f620 legacy=0x7fdddc06d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:07.692 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.588+0000 7fdddaffd700 1 -- 172.21.15.195:0/1430100469 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdddc0ba4d0 con 0x7fdddc150950 2022-06-27T12:15:07.692 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.588+0000 7fdddaffd700 1 --2- 172.21.15.195:0/1430100469 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fdddc150950 0x7fdddc0c7b40 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fddd00097a0 tx=0x7fddd0009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8098d183df6529f6 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:07.692 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.589+0000 7fddda7fc700 1 -- 172.21.15.195:0/1430100469 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd0011630 con 0x7fdddc150950 2022-06-27T12:15:07.693 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.589+0000 7fddda7fc700 1 -- 172.21.15.195:0/1430100469 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddd0011790 con 0x7fdddc150950 2022-06-27T12:15:07.693 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.589+0000 7fddda7fc700 1 -- 172.21.15.195:0/1430100469 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd0011b30 con 0x7fdddc150950 2022-06-27T12:15:07.693 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.589+0000 7fdde1cb2700 1 -- 172.21.15.195:0/1430100469 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fdddc150950 msgr2=0x7fdddc0c7b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:07.693 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.589+0000 7fdde1cb2700 1 --2- 172.21.15.195:0/1430100469 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fdddc150950 0x7fdddc0c7b40 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fddd00097a0 tx=0x7fddd0009060 comp rx=0 tx=0).stop 2022-06-27T12:15:07.694 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.589+0000 7fdde1cb2700 1 -- 172.21.15.195:0/1430100469 shutdown_connections 2022-06-27T12:15:07.694 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.589+0000 7fdde1cb2700 1 --2- 172.21.15.195:0/1430100469 >> v2:172.21.15.202:3300/0 conn(0x7fdddc150580 0x7fdddc156ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:07.694 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.589+0000 7fdde1cb2700 1 --2- 172.21.15.195:0/1430100469 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fdddc150950 0x7fdddc0c7b40 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:07.695 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.589+0000 7fdde1cb2700 1 -- 172.21.15.195:0/1430100469 >> 172.21.15.195:0/1430100469 conn(0x7fdddc063af0 msgr2=0x7fdddc063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:15:07.695 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.589+0000 7fdde1cb2700 1 -- 172.21.15.195:0/1430100469 shutdown_connections 2022-06-27T12:15:07.695 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.589+0000 7fdde1cb2700 1 -- 172.21.15.195:0/1430100469 wait complete. 2022-06-27T12:15:07.696 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.590+0000 7fdde1cb2700 1 Processor -- start 2022-06-27T12:15:07.696 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.590+0000 7fdde1cb2700 1 -- start start 2022-06-27T12:15:07.696 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.591+0000 7fdde1cb2700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fdddc06f620 0x7fdddc0ca900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:07.696 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.591+0000 7fdde1cb2700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fdddc150580 0x7fdddc069ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:07.697 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.591+0000 7fdde1cb2700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdddc150950 0x7fdddc149f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:07.697 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.591+0000 7fdde1cb2700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdddc0ca470 con 0x7fdddc150950 2022-06-27T12:15:07.697 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.591+0000 7fdde1cb2700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdddc0c9e90 con 0x7fdddc06f620 2022-06-27T12:15:07.697 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.591+0000 7fdde1cb2700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdddc0c47a0 con 0x7fdddc150580 2022-06-27T12:15:07.698 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.591+0000 7fdddbfff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdddc150950 0x7fdddc149f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:07.698 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.591+0000 7fdddbfff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdddc150950 0x7fdddc149f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:51504/0 (socket says 172.21.15.195:51504) 2022-06-27T12:15:07.698 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.591+0000 7fdddbfff700 1 -- 172.21.15.195:0/2922243939 learned_addr learned my addr 172.21.15.195:0/2922243939 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:15:07.698 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.591+0000 7fdddb7fe700 1 --2- 172.21.15.195:0/2922243939 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fdddc06f620 0x7fdddc0ca900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:07.699 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.591+0000 7fdddaffd700 1 --2- 172.21.15.195:0/2922243939 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fdddc150580 0x7fdddc069ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:07.699 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.592+0000 7fdddbfff700 1 -- 172.21.15.195:0/2922243939 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fdddc150580 msgr2=0x7fdddc069ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:07.700 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.592+0000 7fdddbfff700 1 --2- 172.21.15.195:0/2922243939 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fdddc150580 0x7fdddc069ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:07.702 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.592+0000 7fdddbfff700 1 -- 172.21.15.195:0/2922243939 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fdddc06f620 msgr2=0x7fdddc0ca900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:07.702 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.592+0000 7fdddbfff700 1 --2- 172.21.15.195:0/2922243939 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fdddc06f620 0x7fdddc0ca900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:07.703 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.592+0000 7fdddbfff700 1 -- 172.21.15.195:0/2922243939 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddd00092e0 con 0x7fdddc150950 2022-06-27T12:15:07.703 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.592+0000 7fdddaffd700 1 --2- 172.21.15.195:0/2922243939 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fdddc150580 0x7fdddc069ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:15:07.703 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.592+0000 7fdddb7fe700 1 --2- 172.21.15.195:0/2922243939 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fdddc06f620 0x7fdddc0ca900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:15:07.703 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.592+0000 7fdddbfff700 1 --2- 172.21.15.195:0/2922243939 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdddc150950 0x7fdddc149f10 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7fddd4005b10 tx=0x7fddd40015b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:07.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.592+0000 7fddc37fe700 1 -- 172.21.15.195:0/2922243939 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd4005d70 con 0x7fdddc150950 2022-06-27T12:15:07.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.592+0000 7fddc37fe700 1 -- 172.21.15.195:0/2922243939 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddd40070d0 con 0x7fdddc150950 2022-06-27T12:15:07.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.592+0000 7fdde1cb2700 1 -- 172.21.15.195:0/2922243939 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdddc0cae40 con 0x7fdddc150950 2022-06-27T12:15:07.705 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.592+0000 7fddc37fe700 1 -- 172.21.15.195:0/2922243939 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd4007450 con 0x7fdddc150950 2022-06-27T12:15:07.705 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.592+0000 7fdde1cb2700 1 -- 172.21.15.195:0/2922243939 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdddc0c3810 con 0x7fdddc150950 2022-06-27T12:15:07.705 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.593+0000 7fdde1cb2700 1 -- 172.21.15.195:0/2922243939 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdddc0ca620 con 0x7fdddc150950 2022-06-27T12:15:07.705 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.593+0000 7fddc37fe700 1 -- 172.21.15.195:0/2922243939 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fddd4019030 con 0x7fdddc150950 2022-06-27T12:15:07.706 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.594+0000 7fddc37fe700 1 --2- 172.21.15.195:0/2922243939 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fddbc063cd0 0x7fddbc066180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:07.707 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.594+0000 7fddc37fe700 1 -- 172.21.15.195:0/2922243939 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7492+0+0 (secure 0 0 0) 0x7fddd40800f0 con 0x7fdddc150950 2022-06-27T12:15:07.711 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.594+0000 7fdddb7fe700 1 --2- 172.21.15.195:0/2922243939 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fddbc063cd0 0x7fddbc066180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:07.712 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.594+0000 7fdddb7fe700 1 --2- 172.21.15.195:0/2922243939 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fddbc063cd0 0x7fddbc066180 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fddcc0113e0 tx=0x7fddcc009060 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:07.712 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.598+0000 7fddc37fe700 1 -- 172.21.15.195:0/2922243939 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fddd4049c90 con 0x7fdddc150950 2022-06-27T12:15:07.839 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.830+0000 7fdde1cb2700 1 -- 172.21.15.195:0/2922243939 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdddc0722c0 con 0x7fdddc150950 2022-06-27T12:15:07.840 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.830+0000 7fddc37fe700 1 -- 172.21.15.195:0/2922243939 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+6131 (secure 0 0 0) 0x7fddd4017070 con 0x7fdddc150950 2022-06-27T12:15:07.841 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:15:07.841 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":101,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":6254,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/1317767101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1317767101},{"type":"v1","addr":"172.21.15.202:6833","nonce":1317767101}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":101}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:13:05.314357+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6865},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6865":{"gid":6865,"name":"d","rank":0,"incarnation":97,"state":"up:active","state_seq":18,"addr":"172.21.15.202:6835/2349973547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2349973547},{"type":"v1","addr":"172.21.15.202:6835","nonce":2349973547}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:15:07.843 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.833+0000 7fdde1cb2700 1 -- 172.21.15.195:0/2922243939 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fddbc063cd0 msgr2=0x7fddbc066180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:07.843 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.833+0000 7fdde1cb2700 1 --2- 172.21.15.195:0/2922243939 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fddbc063cd0 0x7fddbc066180 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fddcc0113e0 tx=0x7fddcc009060 comp rx=0 tx=0).stop 2022-06-27T12:15:07.844 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.833+0000 7fdde1cb2700 1 -- 172.21.15.195:0/2922243939 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdddc150950 msgr2=0x7fdddc149f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:07.844 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.833+0000 7fdde1cb2700 1 --2- 172.21.15.195:0/2922243939 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdddc150950 0x7fdddc149f10 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7fddd4005b10 tx=0x7fddd40015b0 comp rx=0 tx=0).stop 2022-06-27T12:15:07.845 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.834+0000 7fdde1cb2700 1 -- 172.21.15.195:0/2922243939 shutdown_connections 2022-06-27T12:15:07.845 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.834+0000 7fdde1cb2700 1 --2- 172.21.15.195:0/2922243939 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fddbc063cd0 0x7fddbc066180 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:07.845 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.834+0000 7fdde1cb2700 1 --2- 172.21.15.195:0/2922243939 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdddc150950 0x7fdddc149f10 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:07.846 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.834+0000 7fdde1cb2700 1 --2- 172.21.15.195:0/2922243939 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fdddc06f620 0x7fdddc0ca900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:07.846 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.834+0000 7fdde1cb2700 1 --2- 172.21.15.195:0/2922243939 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fdddc150580 0x7fdddc069ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:07.847 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.834+0000 7fdde1cb2700 1 -- 172.21.15.195:0/2922243939 >> 172.21.15.195:0/2922243939 conn(0x7fdddc063af0 msgr2=0x7fdddc14dbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:15:07.847 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.834+0000 7fdde1cb2700 1 -- 172.21.15.195:0/2922243939 shutdown_connections 2022-06-27T12:15:07.848 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:07.834+0000 7fdde1cb2700 1 -- 172.21.15.195:0/2922243939 wait complete. 2022-06-27T12:15:07.849 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 101 2022-06-27T12:15:07.851 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.d (rank=0) 2022-06-27T12:15:07.852 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2022-06-27T12:15:07.853 INFO:teuthology.orchestra.run:waiting for 300 2022-06-27T12:15:07.879 INFO:tasks.ceph.mds.d:Stopped 2022-06-27T12:15:07.880 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:15:08.011 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.007+0000 7efd7e124700 1 Processor -- start 2022-06-27T12:15:08.011 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.007+0000 7efd7e124700 1 -- start start 2022-06-27T12:15:08.012 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.008+0000 7efd7e124700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7efd7814e490 0x7efd7806cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:08.013 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.008+0000 7efd7e124700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd780c33b0 con 0x7efd7814e860 2022-06-27T12:15:08.014 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.008+0000 7efd7e124700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd7806d520 con 0x7efd780c3f50 2022-06-27T12:15:08.015 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.008+0000 7efd7e124700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd7806f220 con 0x7efd7814e490 2022-06-27T12:15:08.015 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.008+0000 7efd76ffd700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7efd7814e860 0x7efd780c3840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:57700/0 (socket says 172.21.15.195:57700) 2022-06-27T12:15:08.015 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.008+0000 7efd76ffd700 1 -- 172.21.15.195:0/948754246 learned_addr learned my addr 172.21.15.195:0/948754246 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:15:08.016 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.008+0000 7efd777fe700 1 --2- 172.21.15.195:0/948754246 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7efd7814e490 0x7efd7806cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:08.016 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.009+0000 7efd767fc700 1 -- 172.21.15.195:0/948754246 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 692514644 0 0) 0x7efd780c33b0 con 0x7efd7814e860 2022-06-27T12:15:08.017 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.009+0000 7efd767fc700 1 -- 172.21.15.195:0/948754246 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd58003430 con 0x7efd7814e860 2022-06-27T12:15:08.017 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.009+0000 7efd767fc700 1 -- 172.21.15.195:0/948754246 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3888863439 0 0) 0x7efd7806d520 con 0x7efd780c3f50 2022-06-27T12:15:08.017 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.009+0000 7efd767fc700 1 -- 172.21.15.195:0/948754246 --> v1:172.21.15.202:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd780c33b0 con 0x7efd780c3f50 2022-06-27T12:15:08.026 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.009+0000 7efd777fe700 1 -- 172.21.15.195:0/948754246 >> v1:172.21.15.202:6789/0 conn(0x7efd780c3f50 legacy=0x7efd78157890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:08.027 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.009+0000 7efd777fe700 1 -- 172.21.15.195:0/948754246 >> v1:172.21.15.195:6789/0 conn(0x7efd7814e860 legacy=0x7efd780c3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:08.027 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.009+0000 7efd777fe700 1 -- 172.21.15.195:0/948754246 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd781542c0 con 0x7efd7814e490 2022-06-27T12:15:08.027 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.009+0000 7efd777fe700 1 --2- 172.21.15.195:0/948754246 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7efd7814e490 0x7efd7806cfe0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7efd680097d0 tx=0x7efd68009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d759796a890205ab server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:08.028 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.009+0000 7efd767fc700 1 -- 172.21.15.195:0/948754246 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd68011630 con 0x7efd7814e490 2022-06-27T12:15:08.030 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.009+0000 7efd767fc700 1 -- 172.21.15.195:0/948754246 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd68011790 con 0x7efd7814e490 2022-06-27T12:15:08.030 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.009+0000 7efd767fc700 1 -- 172.21.15.195:0/948754246 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd68011b10 con 0x7efd7814e490 2022-06-27T12:15:08.031 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.010+0000 7efd7e124700 1 -- 172.21.15.195:0/948754246 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7efd7814e490 msgr2=0x7efd7806cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:08.031 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.010+0000 7efd7e124700 1 --2- 172.21.15.195:0/948754246 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7efd7814e490 0x7efd7806cfe0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7efd680097d0 tx=0x7efd68009060 comp rx=0 tx=0).stop 2022-06-27T12:15:08.031 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.010+0000 7efd7e124700 1 -- 172.21.15.195:0/948754246 shutdown_connections 2022-06-27T12:15:08.032 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.010+0000 7efd7e124700 1 --2- 172.21.15.195:0/948754246 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7efd7814e490 0x7efd7806cfe0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:08.032 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.010+0000 7efd7e124700 1 -- 172.21.15.195:0/948754246 >> 172.21.15.195:0/948754246 conn(0x7efd78063af0 msgr2=0x7efd78063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:15:08.032 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.010+0000 7efd7e124700 1 -- 172.21.15.195:0/948754246 shutdown_connections 2022-06-27T12:15:08.033 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.010+0000 7efd7e124700 1 -- 172.21.15.195:0/948754246 wait complete. 2022-06-27T12:15:08.033 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.011+0000 7efd7e124700 1 Processor -- start 2022-06-27T12:15:08.034 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.011+0000 7efd7e124700 1 -- start start 2022-06-27T12:15:08.035 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.012+0000 7efd7e124700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efd780c3f50 0x7efd78156c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:08.035 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.012+0000 7efd7e124700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7efd7814e490 0x7efd7814a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:08.035 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.012+0000 7efd7e124700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7efd7814e860 0x7efd7814a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:08.036 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.012+0000 7efd7e124700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd780bf5c0 con 0x7efd780c3f50 2022-06-27T12:15:08.036 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.012+0000 7efd7e124700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd78149010 con 0x7efd7814e490 2022-06-27T12:15:08.036 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.012+0000 7efd7e124700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd78149b90 con 0x7efd7814e860 2022-06-27T12:15:08.037 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.012+0000 7efd777fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efd780c3f50 0x7efd78156c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:08.037 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.012+0000 7efd777fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efd780c3f50 0x7efd78156c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:51516/0 (socket says 172.21.15.195:51516) 2022-06-27T12:15:08.037 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.012+0000 7efd777fe700 1 -- 172.21.15.195:0/2344447334 learned_addr learned my addr 172.21.15.195:0/2344447334 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:15:08.038 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.012+0000 7efd77fff700 1 --2- 172.21.15.195:0/2344447334 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7efd7814e860 0x7efd7814a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:08.038 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.012+0000 7efd76ffd700 1 --2- 172.21.15.195:0/2344447334 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7efd7814e490 0x7efd7814a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:08.043 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.012+0000 7efd777fe700 1 -- 172.21.15.195:0/2344447334 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7efd7814e860 msgr2=0x7efd7814a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:08.044 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.012+0000 7efd777fe700 1 --2- 172.21.15.195:0/2344447334 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7efd7814e860 0x7efd7814a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:08.044 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.012+0000 7efd777fe700 1 -- 172.21.15.195:0/2344447334 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7efd7814e490 msgr2=0x7efd7814a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:08.045 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.012+0000 7efd777fe700 1 --2- 172.21.15.195:0/2344447334 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7efd7814e490 0x7efd7814a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:08.045 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.012+0000 7efd777fe700 1 -- 172.21.15.195:0/2344447334 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd680092e0 con 0x7efd780c3f50 2022-06-27T12:15:08.046 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.012+0000 7efd777fe700 1 --2- 172.21.15.195:0/2344447334 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efd780c3f50 0x7efd78156c10 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7efd68009dc0 tx=0x7efd68017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:08.046 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.012+0000 7efd6f7fe700 1 -- 172.21.15.195:0/2344447334 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd68003800 con 0x7efd780c3f50 2022-06-27T12:15:08.046 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.012+0000 7efd6f7fe700 1 -- 172.21.15.195:0/2344447334 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd68003960 con 0x7efd780c3f50 2022-06-27T12:15:08.047 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.012+0000 7efd6f7fe700 1 -- 172.21.15.195:0/2344447334 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd68003d20 con 0x7efd780c3f50 2022-06-27T12:15:08.047 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.012+0000 7efd7e124700 1 -- 172.21.15.195:0/2344447334 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd78158560 con 0x7efd780c3f50 2022-06-27T12:15:08.048 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.012+0000 7efd7e124700 1 -- 172.21.15.195:0/2344447334 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd78149540 con 0x7efd780c3f50 2022-06-27T12:15:08.048 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.013+0000 7efd7e124700 1 -- 172.21.15.195:0/2344447334 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd7806eb40 con 0x7efd780c3f50 2022-06-27T12:15:08.049 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.014+0000 7efd6f7fe700 1 -- 172.21.15.195:0/2344447334 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7efd68029020 con 0x7efd780c3f50 2022-06-27T12:15:08.049 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.014+0000 7efd6f7fe700 1 --2- 172.21.15.195:0/2344447334 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7efd5c063d30 0x7efd5c0661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:08.050 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.014+0000 7efd6f7fe700 1 -- 172.21.15.195:0/2344447334 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7492+0+0 (secure 0 0 0) 0x7efd68081ec0 con 0x7efd780c3f50 2022-06-27T12:15:08.051 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.014+0000 7efd76ffd700 1 --2- 172.21.15.195:0/2344447334 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7efd5c063d30 0x7efd5c0661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:08.051 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.015+0000 7efd76ffd700 1 --2- 172.21.15.195:0/2344447334 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7efd5c063d30 0x7efd5c0661e0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7efd780c80f0 tx=0x7efd60009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:08.051 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.018+0000 7efd6f7fe700 1 -- 172.21.15.195:0/2344447334 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7efd68050510 con 0x7efd780c3f50 2022-06-27T12:15:08.259 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.255+0000 7efd7e124700 1 -- 172.21.15.195:0/2344447334 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efd78154ab0 con 0x7efd780c3f50 2022-06-27T12:15:08.263 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.255+0000 7efd6f7fe700 1 -- 172.21.15.195:0/2344447334 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+6131 (secure 0 0 0) 0x7efd68015070 con 0x7efd780c3f50 2022-06-27T12:15:08.268 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:15:08.269 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":101,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":6254,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/1317767101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1317767101},{"type":"v1","addr":"172.21.15.202:6833","nonce":1317767101}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":101}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:13:05.314357+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6865},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6865":{"gid":6865,"name":"d","rank":0,"incarnation":97,"state":"up:active","state_seq":18,"addr":"172.21.15.202:6835/2349973547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2349973547},{"type":"v1","addr":"172.21.15.202:6835","nonce":2349973547}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:15:08.272 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.257+0000 7efd7e124700 1 -- 172.21.15.195:0/2344447334 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7efd5c063d30 msgr2=0x7efd5c0661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:08.272 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.257+0000 7efd7e124700 1 --2- 172.21.15.195:0/2344447334 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7efd5c063d30 0x7efd5c0661e0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7efd780c80f0 tx=0x7efd60009040 comp rx=0 tx=0).stop 2022-06-27T12:15:08.277 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.257+0000 7efd7e124700 1 -- 172.21.15.195:0/2344447334 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efd780c3f50 msgr2=0x7efd78156c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:08.278 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.257+0000 7efd7e124700 1 --2- 172.21.15.195:0/2344447334 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efd780c3f50 0x7efd78156c10 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7efd68009dc0 tx=0x7efd68017000 comp rx=0 tx=0).stop 2022-06-27T12:15:08.279 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.257+0000 7efd7e124700 1 -- 172.21.15.195:0/2344447334 shutdown_connections 2022-06-27T12:15:08.279 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.257+0000 7efd7e124700 1 --2- 172.21.15.195:0/2344447334 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7efd5c063d30 0x7efd5c0661e0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:08.280 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.257+0000 7efd7e124700 1 --2- 172.21.15.195:0/2344447334 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efd780c3f50 0x7efd78156c10 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:08.280 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.257+0000 7efd7e124700 1 --2- 172.21.15.195:0/2344447334 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7efd7814e860 0x7efd7814a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:08.281 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.257+0000 7efd7e124700 1 --2- 172.21.15.195:0/2344447334 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7efd7814e490 0x7efd7814a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:08.281 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.257+0000 7efd7e124700 1 -- 172.21.15.195:0/2344447334 >> 172.21.15.195:0/2344447334 conn(0x7efd78063af0 msgr2=0x7efd780c97b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:15:08.282 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.257+0000 7efd7e124700 1 -- 172.21.15.195:0/2344447334 shutdown_connections 2022-06-27T12:15:08.283 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:08.257+0000 7efd7e124700 1 -- 172.21.15.195:0/2344447334 wait complete. 2022-06-27T12:15:08.283 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 101 2022-06-27T12:15:08.289 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-06-27T12:15:10.294 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:15:10.448 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.433+0000 7f2fa08e1700 1 Processor -- start 2022-06-27T12:15:10.448 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.433+0000 7f2fa08e1700 1 -- start start 2022-06-27T12:15:10.449 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.434+0000 7f2fa08e1700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f2f9c0c4160 0x7f2f9c156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:10.451 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.434+0000 7f2fa08e1700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2f9c0c39e0 0x7f2f9c1535a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:10.451 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.434+0000 7f2fa08e1700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f9c0c1a20 con 0x7f2f9c06f640 2022-06-27T12:15:10.451 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.434+0000 7f2fa08e1700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f9c0c1130 con 0x7f2f9c0c4160 2022-06-27T12:15:10.452 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.434+0000 7f2fa08e1700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f9c06ea90 con 0x7f2f9c0c39e0 2022-06-27T12:15:10.452 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.435+0000 7f2f9ad9d700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2f9c0c39e0 0x7f2f9c1535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:10.452 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.435+0000 7f2f99d9b700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f2f9c0c4160 0x7f2f9c156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:10.452 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.435+0000 7f2f9a59c700 1 --1- >> v1:172.21.15.202:6789/0 conn(0x7f2f9c06f640 0x7f2f9c0c1eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.202:6789/0 says I am v1:172.21.15.195:59584/0 (socket says 172.21.15.195:59584) 2022-06-27T12:15:10.453 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.435+0000 7f2f9a59c700 1 -- 172.21.15.195:0/1881660491 learned_addr learned my addr 172.21.15.195:0/1881660491 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:15:10.453 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.435+0000 7f2f9ad9d700 1 -- 172.21.15.195:0/1881660491 >> v2:172.21.15.202:3300/0 conn(0x7f2f9c0c4160 msgr2=0x7f2f9c156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:15:10.453 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.435+0000 7f2f9ad9d700 1 --2- 172.21.15.195:0/1881660491 >> v2:172.21.15.202:3300/0 conn(0x7f2f9c0c4160 0x7f2f9c156ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:10.454 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.435+0000 7f2f9ad9d700 1 -- 172.21.15.195:0/1881660491 >> v1:172.21.15.202:6789/0 conn(0x7f2f9c06f640 legacy=0x7f2f9c0c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:10.454 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.435+0000 7f2f9ad9d700 1 -- 172.21.15.195:0/1881660491 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f9c06ed70 con 0x7f2f9c0c39e0 2022-06-27T12:15:10.455 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.435+0000 7f2f99d9b700 1 --2- 172.21.15.195:0/1881660491 >> v2:172.21.15.202:3300/0 conn(0x7f2f9c0c4160 0x7f2f9c156ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:15:10.455 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.435+0000 7f2f9ad9d700 1 --2- 172.21.15.195:0/1881660491 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2f9c0c39e0 0x7f2f9c1535a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f2f94009e60 tx=0x7f2f94009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c72ff9d8a0bb5ee3 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:10.456 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.435+0000 7f2f9959a700 1 -- 172.21.15.195:0/1881660491 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f9400e070 con 0x7f2f9c0c39e0 2022-06-27T12:15:10.456 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.435+0000 7f2f9959a700 1 -- 172.21.15.195:0/1881660491 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f9400bbf0 con 0x7f2f9c0c39e0 2022-06-27T12:15:10.456 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.435+0000 7f2f9959a700 1 -- 172.21.15.195:0/1881660491 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f94012430 con 0x7f2f9c0c39e0 2022-06-27T12:15:10.459 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.436+0000 7f2fa08e1700 1 -- 172.21.15.195:0/1881660491 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2f9c0c39e0 msgr2=0x7f2f9c1535a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:10.460 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.436+0000 7f2fa08e1700 1 --2- 172.21.15.195:0/1881660491 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2f9c0c39e0 0x7f2f9c1535a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f2f94009e60 tx=0x7f2f94009780 comp rx=0 tx=0).stop 2022-06-27T12:15:10.461 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.436+0000 7f2fa08e1700 1 -- 172.21.15.195:0/1881660491 shutdown_connections 2022-06-27T12:15:10.461 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.436+0000 7f2fa08e1700 1 --2- 172.21.15.195:0/1881660491 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2f9c0c39e0 0x7f2f9c1535a0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:10.461 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.436+0000 7f2fa08e1700 1 --2- 172.21.15.195:0/1881660491 >> v2:172.21.15.202:3300/0 conn(0x7f2f9c0c4160 0x7f2f9c156ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:10.461 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.436+0000 7f2fa08e1700 1 -- 172.21.15.195:0/1881660491 >> 172.21.15.195:0/1881660491 conn(0x7f2f9c0c6350 msgr2=0x7f2f9c0c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:15:10.463 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.436+0000 7f2fa08e1700 1 -- 172.21.15.195:0/1881660491 shutdown_connections 2022-06-27T12:15:10.463 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.436+0000 7f2fa08e1700 1 -- 172.21.15.195:0/1881660491 wait complete. 2022-06-27T12:15:10.463 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.437+0000 7f2fa08e1700 1 Processor -- start 2022-06-27T12:15:10.464 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.437+0000 7f2fa08e1700 1 -- start start 2022-06-27T12:15:10.464 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.438+0000 7f2fa08e1700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2f9c06f640 0x7f2f9c0ba330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:10.464 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.438+0000 7f2fa08e1700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2f9c0c39e0 0x7f2f9c154e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:10.465 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.438+0000 7f2fa08e1700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2f9c0c4160 0x7f2f9c1553b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:10.465 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.438+0000 7f2fa08e1700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f9c14eec0 con 0x7f2f9c06f640 2022-06-27T12:15:10.474 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.438+0000 7f2fa08e1700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f9c0c0890 con 0x7f2f9c0c4160 2022-06-27T12:15:10.475 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.438+0000 7f2fa08e1700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f9c159210 con 0x7f2f9c0c39e0 2022-06-27T12:15:10.475 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.438+0000 7f2f9a59c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2f9c06f640 0x7f2f9c0ba330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:10.475 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.438+0000 7f2f99d9b700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2f9c0c39e0 0x7f2f9c154e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:10.476 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.438+0000 7f2f99d9b700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2f9c0c39e0 0x7f2f9c154e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3301/0 says I am v2:172.21.15.195:44300/0 (socket says 172.21.15.195:44300) 2022-06-27T12:15:10.476 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.438+0000 7f2f99d9b700 1 -- 172.21.15.195:0/2212096043 learned_addr learned my addr 172.21.15.195:0/2212096043 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:15:10.476 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.438+0000 7f2f9ad9d700 1 --2- 172.21.15.195:0/2212096043 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2f9c0c4160 0x7f2f9c1553b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:10.476 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.438+0000 7f2f99d9b700 1 -- 172.21.15.195:0/2212096043 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2f9c0c4160 msgr2=0x7f2f9c1553b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:10.477 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.438+0000 7f2f99d9b700 1 --2- 172.21.15.195:0/2212096043 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2f9c0c4160 0x7f2f9c1553b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:10.477 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.438+0000 7f2f99d9b700 1 -- 172.21.15.195:0/2212096043 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2f9c06f640 msgr2=0x7f2f9c0ba330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:10.477 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.438+0000 7f2f99d9b700 1 --2- 172.21.15.195:0/2212096043 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2f9c06f640 0x7f2f9c0ba330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:10.477 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.438+0000 7f2f99d9b700 1 -- 172.21.15.195:0/2212096043 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f8c0092e0 con 0x7f2f9c0c39e0 2022-06-27T12:15:10.480 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.438+0000 7f2f99d9b700 1 --2- 172.21.15.195:0/2212096043 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2f9c0c39e0 0x7f2f9c154e70 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f2f8c00ea50 tx=0x7f2f8c009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:10.481 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.439+0000 7f2f8a7fc700 1 -- 172.21.15.195:0/2212096043 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f8c00c640 con 0x7f2f9c0c39e0 2022-06-27T12:15:10.481 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.439+0000 7f2f8a7fc700 1 -- 172.21.15.195:0/2212096043 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f8c00c7a0 con 0x7f2f9c0c39e0 2022-06-27T12:15:10.481 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.439+0000 7f2fa08e1700 1 -- 172.21.15.195:0/2212096043 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f94009a20 con 0x7f2f9c0c39e0 2022-06-27T12:15:10.481 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.439+0000 7f2f8a7fc700 1 -- 172.21.15.195:0/2212096043 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f8c00cb40 con 0x7f2f9c0c39e0 2022-06-27T12:15:10.482 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.439+0000 7f2fa08e1700 1 -- 172.21.15.195:0/2212096043 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f9c0c0ae0 con 0x7f2f9c0c39e0 2022-06-27T12:15:10.482 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.440+0000 7f2fa08e1700 1 -- 172.21.15.195:0/2212096043 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f9c158170 con 0x7f2f9c0c39e0 2022-06-27T12:15:10.483 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.440+0000 7f2f8a7fc700 1 -- 172.21.15.195:0/2212096043 <== mon.2 v2:172.21.15.202:3301/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f2f8c019030 con 0x7f2f9c0c39e0 2022-06-27T12:15:10.483 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.440+0000 7f2f8a7fc700 1 --2- 172.21.15.195:0/2212096043 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2f78063d00 0x7f2f780661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:10.483 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.440+0000 7f2f8a7fc700 1 -- 172.21.15.195:0/2212096043 <== mon.2 v2:172.21.15.202:3301/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7492+0+0 (secure 0 0 0) 0x7f2f8c080ce0 con 0x7f2f9c0c39e0 2022-06-27T12:15:10.484 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.440+0000 7f2f9a59c700 1 --2- 172.21.15.195:0/2212096043 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2f78063d00 0x7f2f780661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:10.484 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.441+0000 7f2f9a59c700 1 --2- 172.21.15.195:0/2212096043 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2f78063d00 0x7f2f780661b0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f2f90007fd0 tx=0x7f2f90007d70 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:10.485 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.445+0000 7f2f8a7fc700 1 -- 172.21.15.195:0/2212096043 <== mon.2 v2:172.21.15.202:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f2f8c04f330 con 0x7f2f9c0c39e0 2022-06-27T12:15:10.671 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.669+0000 7f2fa08e1700 1 -- 172.21.15.195:0/2212096043 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2f9c0722c0 con 0x7f2f9c0c39e0 2022-06-27T12:15:10.671 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.670+0000 7f2f8a7fc700 1 -- 172.21.15.195:0/2212096043 <== mon.2 v2:172.21.15.202:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+6131 (secure 0 0 0) 0x7f2f8c052950 con 0x7f2f9c0c39e0 2022-06-27T12:15:10.673 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:15:10.673 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":101,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":6254,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/1317767101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1317767101},{"type":"v1","addr":"172.21.15.202:6833","nonce":1317767101}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":101}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:13:05.314357+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6865},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6865":{"gid":6865,"name":"d","rank":0,"incarnation":97,"state":"up:active","state_seq":18,"addr":"172.21.15.202:6835/2349973547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2349973547},{"type":"v1","addr":"172.21.15.202:6835","nonce":2349973547}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:15:10.675 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.672+0000 7f2fa08e1700 1 -- 172.21.15.195:0/2212096043 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2f78063d00 msgr2=0x7f2f780661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:10.675 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.672+0000 7f2fa08e1700 1 --2- 172.21.15.195:0/2212096043 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2f78063d00 0x7f2f780661b0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f2f90007fd0 tx=0x7f2f90007d70 comp rx=0 tx=0).stop 2022-06-27T12:15:10.676 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.672+0000 7f2fa08e1700 1 -- 172.21.15.195:0/2212096043 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2f9c0c39e0 msgr2=0x7f2f9c154e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:10.676 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.672+0000 7f2fa08e1700 1 --2- 172.21.15.195:0/2212096043 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2f9c0c39e0 0x7f2f9c154e70 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f2f8c00ea50 tx=0x7f2f8c009060 comp rx=0 tx=0).stop 2022-06-27T12:15:10.676 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.672+0000 7f2fa08e1700 1 -- 172.21.15.195:0/2212096043 shutdown_connections 2022-06-27T12:15:10.676 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.672+0000 7f2fa08e1700 1 --2- 172.21.15.195:0/2212096043 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2f78063d00 0x7f2f780661b0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:10.677 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.672+0000 7f2fa08e1700 1 --2- 172.21.15.195:0/2212096043 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2f9c06f640 0x7f2f9c0ba330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:10.677 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.672+0000 7f2fa08e1700 1 --2- 172.21.15.195:0/2212096043 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2f9c0c4160 0x7f2f9c1553b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:10.677 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.672+0000 7f2fa08e1700 1 --2- 172.21.15.195:0/2212096043 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2f9c0c39e0 0x7f2f9c154e70 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:10.677 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.672+0000 7f2fa08e1700 1 -- 172.21.15.195:0/2212096043 >> 172.21.15.195:0/2212096043 conn(0x7f2f9c0c6350 msgr2=0x7f2f9c06c570 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:15:10.678 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.672+0000 7f2fa08e1700 1 -- 172.21.15.195:0/2212096043 shutdown_connections 2022-06-27T12:15:10.678 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:10.672+0000 7f2fa08e1700 1 -- 172.21.15.195:0/2212096043 wait complete. 2022-06-27T12:15:10.679 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 101 2022-06-27T12:15:10.692 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-06-27T12:15:12.693 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:15:12.832 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:12.832+0000 7fcfb43d9700 1 Processor -- start 2022-06-27T12:15:12.832 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:12.832+0000 7fcfb43d9700 1 -- start start 2022-06-27T12:15:12.841 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:12.833+0000 7fcfb43d9700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fcfac06e850 0x7fcfac1570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:12.842 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:12.833+0000 7fcfb43d9700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcfac06fef0 0x7fcfac069de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:12.842 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:12.833+0000 7fcfb43d9700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfac06fba0 con 0x7fcfac0c34a0 2022-06-27T12:15:12.842 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:12.833+0000 7fcfb43d9700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfac1575e0 con 0x7fcfac06e850 2022-06-27T12:15:12.843 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:12.833+0000 7fcfb43d9700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfac06f5e0 con 0x7fcfac06fef0 2022-06-27T12:15:12.843 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:12.833+0000 7fcfb2175700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fcfac06e850 0x7fcfac1570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:12.843 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:12.833+0000 7fcfb2175700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fcfac06e850 0x7fcfac1570a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:51552/0 (socket says 172.21.15.195:51552) 2022-06-27T12:15:12.843 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:12.833+0000 7fcfb2976700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fcfac0c34a0 0x7fcfac0c2920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:57744/0 (socket says 172.21.15.195:57744) 2022-06-27T12:15:12.844 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:12.833+0000 7fcfb2175700 1 -- 172.21.15.195:0/105915453 learned_addr learned my addr 172.21.15.195:0/105915453 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:15:12.844 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:12.833+0000 7fcfb1974700 1 --2- 172.21.15.195:0/105915453 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcfac06fef0 0x7fcfac069de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:12.844 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:12.833+0000 7fcfb2175700 1 -- 172.21.15.195:0/105915453 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcfac06fef0 msgr2=0x7fcfac069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:15:12.844 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:12.833+0000 7fcfb2175700 1 --2- 172.21.15.195:0/105915453 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcfac06fef0 0x7fcfac069de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:12.845 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:12.833+0000 7fcfb2175700 1 -- 172.21.15.195:0/105915453 >> v1:172.21.15.195:6789/0 conn(0x7fcfac0c34a0 legacy=0x7fcfac0c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:12.845 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:12.833+0000 7fcfb2175700 1 -- 172.21.15.195:0/105915453 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfac06f8c0 con 0x7fcfac06e850 2022-06-27T12:15:12.845 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:12.833+0000 7fcfb1173700 1 -- 172.21.15.195:0/105915453 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3369954331 0 0) 0x7fcfac06fba0 con 0x7fcfac0c34a0 2022-06-27T12:15:12.845 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:12.833+0000 7fcfb2175700 1 --2- 172.21.15.195:0/105915453 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcfac06e850 0x7fcfac1570a0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7fcfa8009860 tx=0x7fcfa8009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd64d210d6a741f8 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:12.846 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:12.834+0000 7fcfb1173700 1 -- 172.21.15.195:0/105915453 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa800b690 con 0x7fcfac06e850 2022-06-27T12:15:12.846 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:12.834+0000 7fcfb1173700 1 -- 172.21.15.195:0/105915453 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfa800b7f0 con 0x7fcfac06e850 2022-06-27T12:15:12.847 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:12.834+0000 7fcfb1173700 1 -- 172.21.15.195:0/105915453 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa800bb90 con 0x7fcfac06e850 2022-06-27T12:15:12.847 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:12.834+0000 7fcfb43d9700 1 -- 172.21.15.195:0/105915453 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcfac06e850 msgr2=0x7fcfac1570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:12.847 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:12.834+0000 7fcfb43d9700 1 --2- 172.21.15.195:0/105915453 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcfac06e850 0x7fcfac1570a0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7fcfa8009860 tx=0x7fcfa8009120 comp rx=0 tx=0).stop 2022-06-27T12:15:12.847 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:12.834+0000 7fcfb43d9700 1 -- 172.21.15.195:0/105915453 shutdown_connections 2022-06-27T12:15:12.847 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:12.834+0000 7fcfb43d9700 1 --2- 172.21.15.195:0/105915453 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcfac06e850 0x7fcfac1570a0 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:12.848 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:12.834+0000 7fcfb43d9700 1 --2- 172.21.15.195:0/105915453 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcfac06fef0 0x7fcfac069de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:12.848 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:12.834+0000 7fcfb43d9700 1 -- 172.21.15.195:0/105915453 >> 172.21.15.195:0/105915453 conn(0x7fcfac0c6350 msgr2=0x7fcfac0c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:15:12.848 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:12.834+0000 7fcfb43d9700 1 -- 172.21.15.195:0/105915453 shutdown_connections 2022-06-27T12:15:12.848 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:12.834+0000 7fcfb43d9700 1 -- 172.21.15.195:0/105915453 wait complete. 2022-06-27T12:15:12.849 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:12.835+0000 7fcfb43d9700 1 Processor -- start 2022-06-27T12:15:12.849 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:12.835+0000 7fcfb43d9700 1 -- start start 2022-06-27T12:15:12.849 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:12.836+0000 7fcfb43d9700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcfac06e850 0x7fcfac14d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:12.849 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:12.836+0000 7fcfb43d9700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcfac0c34a0 0x7fcfac0c8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:12.850 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:12.836+0000 7fcfb43d9700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcfac0c07c0 0x7fcfac162380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:12.851 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:12.836+0000 7fcfb43d9700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfac0c72e0 con 0x7fcfac0c34a0 2022-06-27T12:15:12.851 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:12.836+0000 7fcfb43d9700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfac14ee10 con 0x7fcfac0c07c0 2022-06-27T12:15:12.852 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:12.836+0000 7fcfb43d9700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfac14df00 con 0x7fcfac06e850 2022-06-27T12:15:12.852 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:12.836+0000 7fcfb1974700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcfac0c34a0 0x7fcfac0c8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:12.853 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:12.836+0000 7fcfb1974700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcfac0c34a0 0x7fcfac0c8260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:51560/0 (socket says 172.21.15.195:51560) 2022-06-27T12:15:12.853 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:12.836+0000 7fcfb1974700 1 -- 172.21.15.195:0/1027114071 learned_addr learned my addr 172.21.15.195:0/1027114071 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:15:12.853 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:12.836+0000 7fcfb2175700 1 --2- 172.21.15.195:0/1027114071 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcfac06e850 0x7fcfac14d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:12.854 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:12.836+0000 7fcfb2976700 1 --2- 172.21.15.195:0/1027114071 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcfac0c07c0 0x7fcfac162380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:12.855 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:12.837+0000 7fcfb1974700 1 -- 172.21.15.195:0/1027114071 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcfac06e850 msgr2=0x7fcfac14d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:12.855 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:12.837+0000 7fcfb1974700 1 --2- 172.21.15.195:0/1027114071 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcfac06e850 0x7fcfac14d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:12.855 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:12.837+0000 7fcfb1974700 1 -- 172.21.15.195:0/1027114071 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcfac0c07c0 msgr2=0x7fcfac162380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:12.856 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:12.837+0000 7fcfb1974700 1 --2- 172.21.15.195:0/1027114071 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcfac0c07c0 0x7fcfac162380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:12.856 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:12.837+0000 7fcfb1974700 1 -- 172.21.15.195:0/1027114071 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfa80093a0 con 0x7fcfac0c34a0 2022-06-27T12:15:12.856 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:12.837+0000 7fcfb1974700 1 --2- 172.21.15.195:0/1027114071 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcfac0c34a0 0x7fcfac0c8260 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c00e930 tx=0x7fcf9c009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:12.857 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:12.837+0000 7fcf9a7fc700 1 -- 172.21.15.195:0/1027114071 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf9c013680 con 0x7fcfac0c34a0 2022-06-27T12:15:12.857 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:12.837+0000 7fcf9a7fc700 1 -- 172.21.15.195:0/1027114071 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf9c0137e0 con 0x7fcfac0c34a0 2022-06-27T12:15:12.857 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:12.837+0000 7fcfb43d9700 1 -- 172.21.15.195:0/1027114071 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfac14e4e0 con 0x7fcfac0c34a0 2022-06-27T12:15:12.857 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:12.837+0000 7fcf9a7fc700 1 -- 172.21.15.195:0/1027114071 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf9c013b80 con 0x7fcfac0c34a0 2022-06-27T12:15:12.858 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:12.837+0000 7fcfb43d9700 1 -- 172.21.15.195:0/1027114071 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfac14eaa0 con 0x7fcfac0c34a0 2022-06-27T12:15:12.858 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:12.838+0000 7fcfb43d9700 1 -- 172.21.15.195:0/1027114071 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfac0722c0 con 0x7fcfac0c34a0 2022-06-27T12:15:12.858 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:12.838+0000 7fcf9a7fc700 1 -- 172.21.15.195:0/1027114071 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fcf9c019030 con 0x7fcfac0c34a0 2022-06-27T12:15:12.858 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:12.839+0000 7fcf9a7fc700 1 --2- 172.21.15.195:0/1027114071 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcf8c063d00 0x7fcf8c0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:12.859 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:12.839+0000 7fcf9a7fc700 1 -- 172.21.15.195:0/1027114071 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7492+0+0 (secure 0 0 0) 0x7fcf9c080bd0 con 0x7fcfac0c34a0 2022-06-27T12:15:12.859 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:12.839+0000 7fcfb2175700 1 --2- 172.21.15.195:0/1027114071 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcf8c063d00 0x7fcf8c0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:12.859 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:12.839+0000 7fcfb2175700 1 --2- 172.21.15.195:0/1027114071 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcf8c063d00 0x7fcf8c0661b0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fcfa8009160 tx=0x7fcfa8017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:12.860 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:12.843+0000 7fcf9a7fc700 1 -- 172.21.15.195:0/1027114071 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fcf9c04f220 con 0x7fcfac0c34a0 2022-06-27T12:15:13.067 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:13.065+0000 7fcfb43d9700 1 -- 172.21.15.195:0/1027114071 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcfac06f240 con 0x7fcfac0c34a0 2022-06-27T12:15:13.068 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:13.066+0000 7fcf9a7fc700 1 -- 172.21.15.195:0/1027114071 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+6131 (secure 0 0 0) 0x7fcf9c017070 con 0x7fcfac0c34a0 2022-06-27T12:15:13.068 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:15:13.069 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":101,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":6254,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/1317767101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1317767101},{"type":"v1","addr":"172.21.15.202:6833","nonce":1317767101}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":101}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:13:05.314357+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6865},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6865":{"gid":6865,"name":"d","rank":0,"incarnation":97,"state":"up:active","state_seq":18,"addr":"172.21.15.202:6835/2349973547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2349973547},{"type":"v1","addr":"172.21.15.202:6835","nonce":2349973547}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:15:13.070 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:13.068+0000 7fcfb43d9700 1 -- 172.21.15.195:0/1027114071 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcf8c063d00 msgr2=0x7fcf8c0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:13.070 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:13.068+0000 7fcfb43d9700 1 --2- 172.21.15.195:0/1027114071 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcf8c063d00 0x7fcf8c0661b0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fcfa8009160 tx=0x7fcfa8017000 comp rx=0 tx=0).stop 2022-06-27T12:15:13.070 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:13.068+0000 7fcfb43d9700 1 -- 172.21.15.195:0/1027114071 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcfac0c34a0 msgr2=0x7fcfac0c8260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:13.071 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:13.068+0000 7fcfb43d9700 1 --2- 172.21.15.195:0/1027114071 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcfac0c34a0 0x7fcfac0c8260 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c00e930 tx=0x7fcf9c009060 comp rx=0 tx=0).stop 2022-06-27T12:15:13.071 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:13.068+0000 7fcfb43d9700 1 -- 172.21.15.195:0/1027114071 shutdown_connections 2022-06-27T12:15:13.071 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:13.068+0000 7fcfb43d9700 1 --2- 172.21.15.195:0/1027114071 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcf8c063d00 0x7fcf8c0661b0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:13.072 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:13.068+0000 7fcfb43d9700 1 --2- 172.21.15.195:0/1027114071 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcfac0c07c0 0x7fcfac162380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:13.072 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:13.068+0000 7fcfb43d9700 1 --2- 172.21.15.195:0/1027114071 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcfac06e850 0x7fcfac14d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:13.072 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:13.068+0000 7fcfb43d9700 1 --2- 172.21.15.195:0/1027114071 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcfac0c34a0 0x7fcfac0c8260 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:13.072 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:13.068+0000 7fcfb43d9700 1 -- 172.21.15.195:0/1027114071 >> 172.21.15.195:0/1027114071 conn(0x7fcfac0c6350 msgr2=0x7fcfac06c860 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:15:13.073 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:13.068+0000 7fcfb43d9700 1 -- 172.21.15.195:0/1027114071 shutdown_connections 2022-06-27T12:15:13.073 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:13.068+0000 7fcfb43d9700 1 -- 172.21.15.195:0/1027114071 wait complete. 2022-06-27T12:15:13.075 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 101 2022-06-27T12:15:13.082 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-06-27T12:15:15.090 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:15:15.245 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.239+0000 7f8b248ca700 1 Processor -- start 2022-06-27T12:15:15.263 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.239+0000 7f8b248ca700 1 -- start start 2022-06-27T12:15:15.263 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.240+0000 7f8b248ca700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8b2014e860 0x7f8b200c3840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:15.264 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.240+0000 7f8b248ca700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f8b200c3d80 0x7f8b20157890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:15.264 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.240+0000 7f8b248ca700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b200c33b0 con 0x7f8b2014e490 2022-06-27T12:15:15.264 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.240+0000 7f8b248ca700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b200c4200 con 0x7f8b200c3d80 2022-06-27T12:15:15.265 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.240+0000 7f8b248ca700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b2006f220 con 0x7f8b2014e860 2022-06-27T12:15:15.266 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.240+0000 7f8b1e59c700 1 --1- >> v1:172.21.15.202:6789/0 conn(0x7f8b2014e490 0x7f8b2006cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.202:6789/0 says I am v1:172.21.15.195:59616/0 (socket says 172.21.15.195:59616) 2022-06-27T12:15:15.266 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.240+0000 7f8b1ed9d700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f8b200c3d80 0x7f8b20157890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:15.266 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.240+0000 7f8b1e59c700 1 -- 172.21.15.195:0/2018946582 learned_addr learned my addr 172.21.15.195:0/2018946582 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:15:15.266 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.240+0000 7f8b1dd9b700 1 --2- 172.21.15.195:0/2018946582 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8b2014e860 0x7f8b200c3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:15.267 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.240+0000 7f8b1ed9d700 1 -- 172.21.15.195:0/2018946582 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8b2014e860 msgr2=0x7f8b200c3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:15:15.267 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.240+0000 7f8b1ed9d700 1 --2- 172.21.15.195:0/2018946582 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8b2014e860 0x7f8b200c3840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:15.267 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.240+0000 7f8b1ed9d700 1 -- 172.21.15.195:0/2018946582 >> v1:172.21.15.202:6789/0 conn(0x7f8b2014e490 legacy=0x7f8b2006cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:15.267 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.240+0000 7f8b1d59a700 1 -- 172.21.15.195:0/2018946582 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 419410225 0 0) 0x7f8b200c33b0 con 0x7f8b2014e490 2022-06-27T12:15:15.268 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.240+0000 7f8b1ed9d700 1 -- 172.21.15.195:0/2018946582 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b201542c0 con 0x7f8b200c3d80 2022-06-27T12:15:15.281 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.240+0000 7f8b1dd9b700 1 --2- 172.21.15.195:0/2018946582 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8b2014e860 0x7f8b200c3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:15:15.282 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.240+0000 7f8b1ed9d700 1 --2- 172.21.15.195:0/2018946582 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8b200c3d80 0x7f8b20157890 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f8b18009e60 tx=0x7f8b18009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9fc34ab51decea2 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:15.282 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.241+0000 7f8b1d59a700 1 -- 172.21.15.195:0/2018946582 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b1800e070 con 0x7f8b200c3d80 2022-06-27T12:15:15.283 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.241+0000 7f8b1d59a700 1 -- 172.21.15.195:0/2018946582 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b1800bb60 con 0x7f8b200c3d80 2022-06-27T12:15:15.283 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.241+0000 7f8b1d59a700 1 -- 172.21.15.195:0/2018946582 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b18012430 con 0x7f8b200c3d80 2022-06-27T12:15:15.283 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.241+0000 7f8b248ca700 1 -- 172.21.15.195:0/2018946582 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8b200c3d80 msgr2=0x7f8b20157890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:15.284 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.241+0000 7f8b248ca700 1 --2- 172.21.15.195:0/2018946582 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8b200c3d80 0x7f8b20157890 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f8b18009e60 tx=0x7f8b18009780 comp rx=0 tx=0).stop 2022-06-27T12:15:15.284 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.241+0000 7f8b248ca700 1 -- 172.21.15.195:0/2018946582 shutdown_connections 2022-06-27T12:15:15.285 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.241+0000 7f8b248ca700 1 --2- 172.21.15.195:0/2018946582 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8b200c3d80 0x7f8b20157890 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:15.285 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.241+0000 7f8b248ca700 1 --2- 172.21.15.195:0/2018946582 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8b2014e860 0x7f8b200c3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:15.285 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.241+0000 7f8b248ca700 1 -- 172.21.15.195:0/2018946582 >> 172.21.15.195:0/2018946582 conn(0x7f8b20063af0 msgr2=0x7f8b20063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:15:15.286 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.241+0000 7f8b248ca700 1 -- 172.21.15.195:0/2018946582 shutdown_connections 2022-06-27T12:15:15.287 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.241+0000 7f8b248ca700 1 -- 172.21.15.195:0/2018946582 wait complete. 2022-06-27T12:15:15.287 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.242+0000 7f8b248ca700 1 Processor -- start 2022-06-27T12:15:15.288 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.242+0000 7f8b248ca700 1 -- start start 2022-06-27T12:15:15.288 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.243+0000 7f8b248ca700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8b200c3d80 0x7f8b20169b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:15.288 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.243+0000 7f8b248ca700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8b2014e490 0x7f8b20167050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:15.289 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.243+0000 7f8b248ca700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8b2014e860 0x7f8b20157ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:15.289 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.243+0000 7f8b248ca700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b20156540 con 0x7f8b200c3d80 2022-06-27T12:15:15.290 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.243+0000 7f8b248ca700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b20166590 con 0x7f8b2014e860 2022-06-27T12:15:15.290 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.243+0000 7f8b248ca700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b2006a8e0 con 0x7f8b2014e490 2022-06-27T12:15:15.291 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.243+0000 7f8b1e59c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8b200c3d80 0x7f8b20169b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:15.291 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.243+0000 7f8b1dd9b700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8b2014e490 0x7f8b20167050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:15.292 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.243+0000 7f8b1ed9d700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8b2014e860 0x7f8b20157ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:15.292 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.243+0000 7f8b1e59c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8b200c3d80 0x7f8b20169b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:51574/0 (socket says 172.21.15.195:51574) 2022-06-27T12:15:15.293 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.243+0000 7f8b1dd9b700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8b2014e490 0x7f8b20167050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3301/0 says I am v2:172.21.15.195:44332/0 (socket says 172.21.15.195:44332) 2022-06-27T12:15:15.293 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.243+0000 7f8b1e59c700 1 -- 172.21.15.195:0/2689919208 learned_addr learned my addr 172.21.15.195:0/2689919208 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:15:15.294 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.244+0000 7f8b1e59c700 1 -- 172.21.15.195:0/2689919208 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8b2014e490 msgr2=0x7f8b20167050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:15.294 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.244+0000 7f8b1e59c700 1 --2- 172.21.15.195:0/2689919208 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8b2014e490 0x7f8b20167050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:15.295 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.244+0000 7f8b1e59c700 1 -- 172.21.15.195:0/2689919208 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8b2014e860 msgr2=0x7f8b20157ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:15.295 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.244+0000 7f8b1e59c700 1 --2- 172.21.15.195:0/2689919208 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8b2014e860 0x7f8b20157ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:15.296 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.244+0000 7f8b1e59c700 1 -- 172.21.15.195:0/2689919208 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b140092e0 con 0x7f8b200c3d80 2022-06-27T12:15:15.296 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.244+0000 7f8b1dd9b700 1 --2- 172.21.15.195:0/2689919208 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8b2014e490 0x7f8b20167050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:15:15.296 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.244+0000 7f8b1ed9d700 1 --2- 172.21.15.195:0/2689919208 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8b2014e860 0x7f8b20157ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:15:15.297 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.244+0000 7f8b1e59c700 1 --2- 172.21.15.195:0/2689919208 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8b200c3d80 0x7f8b20169b60 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f8b1000c590 tx=0x7f8b10007cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:15.297 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.244+0000 7f8b0e7fc700 1 -- 172.21.15.195:0/2689919208 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b10007360 con 0x7f8b200c3d80 2022-06-27T12:15:15.298 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.244+0000 7f8b0e7fc700 1 -- 172.21.15.195:0/2689919208 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b10007500 con 0x7f8b200c3d80 2022-06-27T12:15:15.298 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.244+0000 7f8b248ca700 1 -- 172.21.15.195:0/2689919208 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b18009a20 con 0x7f8b200c3d80 2022-06-27T12:15:15.298 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.244+0000 7f8b0e7fc700 1 -- 172.21.15.195:0/2689919208 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b10004940 con 0x7f8b200c3d80 2022-06-27T12:15:15.299 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.244+0000 7f8b248ca700 1 -- 172.21.15.195:0/2689919208 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b2006aea0 con 0x7f8b200c3d80 2022-06-27T12:15:15.299 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.245+0000 7f8b248ca700 1 -- 172.21.15.195:0/2689919208 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b200722c0 con 0x7f8b200c3d80 2022-06-27T12:15:15.299 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.245+0000 7f8b0e7fc700 1 -- 172.21.15.195:0/2689919208 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f8b10010030 con 0x7f8b200c3d80 2022-06-27T12:15:15.300 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.246+0000 7f8b0e7fc700 1 --2- 172.21.15.195:0/2689919208 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8afc063d00 0x7f8afc0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:15.300 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.246+0000 7f8b0e7fc700 1 -- 172.21.15.195:0/2689919208 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7492+0+0 (secure 0 0 0) 0x7f8b100808c0 con 0x7f8b200c3d80 2022-06-27T12:15:15.301 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.246+0000 7f8b1dd9b700 1 --2- 172.21.15.195:0/2689919208 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8afc063d00 0x7f8afc0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:15.301 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.246+0000 7f8b1dd9b700 1 --2- 172.21.15.195:0/2689919208 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8afc063d00 0x7f8afc0661b0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f8b2014ae00 tx=0x7f8b14009060 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:15.302 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.250+0000 7f8b0e7fc700 1 -- 172.21.15.195:0/2689919208 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f8b1004ef90 con 0x7f8b200c3d80 2022-06-27T12:15:15.480 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.479+0000 7f8b248ca700 1 -- 172.21.15.195:0/2689919208 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b20064900 con 0x7f8b200c3d80 2022-06-27T12:15:15.481 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.480+0000 7f8b0e7fc700 1 -- 172.21.15.195:0/2689919208 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+6131 (secure 0 0 0) 0x7f8b100525b0 con 0x7f8b200c3d80 2022-06-27T12:15:15.482 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:15:15.483 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":101,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":6254,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/1317767101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1317767101},{"type":"v1","addr":"172.21.15.202:6833","nonce":1317767101}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":101}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:13:05.314357+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6865},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6865":{"gid":6865,"name":"d","rank":0,"incarnation":97,"state":"up:active","state_seq":18,"addr":"172.21.15.202:6835/2349973547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2349973547},{"type":"v1","addr":"172.21.15.202:6835","nonce":2349973547}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:15:15.486 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.482+0000 7f8b248ca700 1 -- 172.21.15.195:0/2689919208 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8afc063d00 msgr2=0x7f8afc0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:15.486 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.482+0000 7f8b248ca700 1 --2- 172.21.15.195:0/2689919208 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8afc063d00 0x7f8afc0661b0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f8b2014ae00 tx=0x7f8b14009060 comp rx=0 tx=0).stop 2022-06-27T12:15:15.487 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.482+0000 7f8b248ca700 1 -- 172.21.15.195:0/2689919208 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8b200c3d80 msgr2=0x7f8b20169b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:15.487 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.482+0000 7f8b248ca700 1 --2- 172.21.15.195:0/2689919208 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8b200c3d80 0x7f8b20169b60 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f8b1000c590 tx=0x7f8b10007cb0 comp rx=0 tx=0).stop 2022-06-27T12:15:15.487 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.482+0000 7f8b248ca700 1 -- 172.21.15.195:0/2689919208 shutdown_connections 2022-06-27T12:15:15.488 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.482+0000 7f8b248ca700 1 --2- 172.21.15.195:0/2689919208 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8afc063d00 0x7f8afc0661b0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:15.488 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.482+0000 7f8b248ca700 1 --2- 172.21.15.195:0/2689919208 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8b200c3d80 0x7f8b20169b60 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:15.489 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.482+0000 7f8b248ca700 1 --2- 172.21.15.195:0/2689919208 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8b2014e860 0x7f8b20157ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:15.489 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.482+0000 7f8b248ca700 1 --2- 172.21.15.195:0/2689919208 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8b2014e490 0x7f8b20167050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:15.490 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.482+0000 7f8b248ca700 1 -- 172.21.15.195:0/2689919208 >> 172.21.15.195:0/2689919208 conn(0x7f8b20063af0 msgr2=0x7f8b201526c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:15:15.490 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.482+0000 7f8b248ca700 1 -- 172.21.15.195:0/2689919208 shutdown_connections 2022-06-27T12:15:15.490 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:15.482+0000 7f8b248ca700 1 -- 172.21.15.195:0/2689919208 wait complete. 2022-06-27T12:15:15.491 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 101 2022-06-27T12:15:15.508 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-06-27T12:15:17.511 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:15:17.651 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.646+0000 7f824d83a700 1 Processor -- start 2022-06-27T12:15:17.651 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.646+0000 7f824d83a700 1 -- start start 2022-06-27T12:15:17.652 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.647+0000 7f824d83a700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f82480c64f0 0x7f8248150bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:17.652 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.647+0000 7f824d83a700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f824806d280 0x7f8248069a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:17.652 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.647+0000 7f824d83a700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82480c5d80 con 0x7f82480c5480 2022-06-27T12:15:17.652 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.647+0000 7f824d83a700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8248069f50 con 0x7f82480c64f0 2022-06-27T12:15:17.653 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.647+0000 7f824d83a700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f824806b010 con 0x7f824806d280 2022-06-27T12:15:17.653 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.647+0000 7f82467fc700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f82480c64f0 0x7f8248150bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:17.653 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.647+0000 7f8246ffd700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f82480c5480 0x7f82481504c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:57772/0 (socket says 172.21.15.195:57772) 2022-06-27T12:15:17.653 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.647+0000 7f8246ffd700 1 -- 172.21.15.195:0/3708616699 learned_addr learned my addr 172.21.15.195:0/3708616699 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:15:17.663 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.647+0000 7f82477fe700 1 --2- 172.21.15.195:0/3708616699 >> v2:172.21.15.202:3300/0 conn(0x7f824806d280 0x7f8248069a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:17.675 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.647+0000 7f82467fc700 1 -- 172.21.15.195:0/3708616699 >> v2:172.21.15.202:3300/0 conn(0x7f824806d280 msgr2=0x7f8248069a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:15:17.676 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.647+0000 7f82467fc700 1 --2- 172.21.15.195:0/3708616699 >> v2:172.21.15.202:3300/0 conn(0x7f824806d280 0x7f8248069a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:17.676 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.647+0000 7f82467fc700 1 -- 172.21.15.195:0/3708616699 >> v1:172.21.15.195:6789/0 conn(0x7f82480c5480 legacy=0x7f82481504c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:17.676 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.647+0000 7f8245ffb700 1 -- 172.21.15.195:0/3708616699 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1160756777 0 0) 0x7f82480c5d80 con 0x7f82480c5480 2022-06-27T12:15:17.677 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.647+0000 7f82467fc700 1 -- 172.21.15.195:0/3708616699 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82480c75e0 con 0x7f82480c64f0 2022-06-27T12:15:17.677 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.648+0000 7f82467fc700 1 --2- 172.21.15.195:0/3708616699 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f82480c64f0 0x7f8248150bd0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f82480c1c00 tx=0x7f8238009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=417edecfdab1e042 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:17.677 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.648+0000 7f8245ffb700 1 -- 172.21.15.195:0/3708616699 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8238011630 con 0x7f82480c64f0 2022-06-27T12:15:17.677 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.648+0000 7f8245ffb700 1 -- 172.21.15.195:0/3708616699 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8238011790 con 0x7f82480c64f0 2022-06-27T12:15:17.678 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.648+0000 7f8245ffb700 1 -- 172.21.15.195:0/3708616699 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8238011b30 con 0x7f82480c64f0 2022-06-27T12:15:17.678 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.648+0000 7f824d83a700 1 -- 172.21.15.195:0/3708616699 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f82480c64f0 msgr2=0x7f8248150bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:17.678 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.648+0000 7f824d83a700 1 --2- 172.21.15.195:0/3708616699 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f82480c64f0 0x7f8248150bd0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f82480c1c00 tx=0x7f8238009060 comp rx=0 tx=0).stop 2022-06-27T12:15:17.679 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.648+0000 7f824d83a700 1 -- 172.21.15.195:0/3708616699 shutdown_connections 2022-06-27T12:15:17.679 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.648+0000 7f824d83a700 1 --2- 172.21.15.195:0/3708616699 >> v2:172.21.15.202:3300/0 conn(0x7f824806d280 0x7f8248069a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:17.679 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.648+0000 7f824d83a700 1 --2- 172.21.15.195:0/3708616699 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f82480c64f0 0x7f8248150bd0 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:17.680 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.648+0000 7f824d83a700 1 -- 172.21.15.195:0/3708616699 >> 172.21.15.195:0/3708616699 conn(0x7f8248063af0 msgr2=0x7f8248063ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:15:17.681 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.648+0000 7f824d83a700 1 -- 172.21.15.195:0/3708616699 shutdown_connections 2022-06-27T12:15:17.684 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.648+0000 7f824d83a700 1 -- 172.21.15.195:0/3708616699 wait complete. 2022-06-27T12:15:17.684 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.650+0000 7f824d83a700 1 Processor -- start 2022-06-27T12:15:17.684 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.650+0000 7f824d83a700 1 -- start start 2022-06-27T12:15:17.685 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.650+0000 7f824d83a700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f824806d280 0x7f8248154670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:17.685 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.650+0000 7f824d83a700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f82480c5480 0x7f8248154bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:17.686 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.650+0000 7f824d83a700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f82480c64f0 0x7f8248169890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:17.686 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.650+0000 7f824d83a700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82480c3050 con 0x7f824806d280 2022-06-27T12:15:17.686 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.650+0000 7f824d83a700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f824815c6c0 con 0x7f82480c5480 2022-06-27T12:15:17.686 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.650+0000 7f824d83a700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f824815bab0 con 0x7f82480c64f0 2022-06-27T12:15:17.687 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.651+0000 7f8246ffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f824806d280 0x7f8248154670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:17.687 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.651+0000 7f8246ffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f824806d280 0x7f8248154670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:51590/0 (socket says 172.21.15.195:51590) 2022-06-27T12:15:17.687 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.651+0000 7f8246ffd700 1 -- 172.21.15.195:0/1457092783 learned_addr learned my addr 172.21.15.195:0/1457092783 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:15:17.688 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.651+0000 7f82467fc700 1 --2- 172.21.15.195:0/1457092783 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f82480c5480 0x7f8248154bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:17.688 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.651+0000 7f82477fe700 1 --2- 172.21.15.195:0/1457092783 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f82480c64f0 0x7f8248169890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:17.688 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.651+0000 7f8246ffd700 1 -- 172.21.15.195:0/1457092783 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f82480c64f0 msgr2=0x7f8248169890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:17.688 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.651+0000 7f8246ffd700 1 --2- 172.21.15.195:0/1457092783 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f82480c64f0 0x7f8248169890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:17.689 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.651+0000 7f8246ffd700 1 -- 172.21.15.195:0/1457092783 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f82480c5480 msgr2=0x7f8248154bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:17.689 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.651+0000 7f8246ffd700 1 --2- 172.21.15.195:0/1457092783 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f82480c5480 0x7f8248154bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:17.689 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.651+0000 7f8246ffd700 1 -- 172.21.15.195:0/1457092783 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82380092e0 con 0x7f824806d280 2022-06-27T12:15:17.690 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.651+0000 7f8246ffd700 1 --2- 172.21.15.195:0/1457092783 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f824806d280 0x7f8248154670 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f823c00c590 tx=0x7f823c007cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:17.690 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.651+0000 7f822effd700 1 -- 172.21.15.195:0/1457092783 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f823c007360 con 0x7f824806d280 2022-06-27T12:15:17.690 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.651+0000 7f822effd700 1 -- 172.21.15.195:0/1457092783 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f823c007500 con 0x7f824806d280 2022-06-27T12:15:17.690 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.651+0000 7f824d83a700 1 -- 172.21.15.195:0/1457092783 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f824815bd90 con 0x7f824806d280 2022-06-27T12:15:17.691 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.651+0000 7f822effd700 1 -- 172.21.15.195:0/1457092783 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f823c004940 con 0x7f824806d280 2022-06-27T12:15:17.691 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.651+0000 7f824d83a700 1 -- 172.21.15.195:0/1457092783 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f824815c350 con 0x7f824806d280 2022-06-27T12:15:17.691 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.652+0000 7f824d83a700 1 -- 172.21.15.195:0/1457092783 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82480722c0 con 0x7f824806d280 2022-06-27T12:15:17.692 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.653+0000 7f822effd700 1 -- 172.21.15.195:0/1457092783 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f823c010030 con 0x7f824806d280 2022-06-27T12:15:17.692 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.653+0000 7f822effd700 1 --2- 172.21.15.195:0/1457092783 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8228063d30 0x7f82280661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:17.693 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.653+0000 7f822effd700 1 -- 172.21.15.195:0/1457092783 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7492+0+0 (secure 0 0 0) 0x7f823c080810 con 0x7f824806d280 2022-06-27T12:15:17.693 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.653+0000 7f82467fc700 1 --2- 172.21.15.195:0/1457092783 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8228063d30 0x7f82280661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:17.693 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.653+0000 7f82467fc700 1 --2- 172.21.15.195:0/1457092783 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8228063d30 0x7f82280661e0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f82480c1c00 tx=0x7f8238017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:17.694 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.657+0000 7f822effd700 1 -- 172.21.15.195:0/1457092783 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f823c04eee0 con 0x7f824806d280 2022-06-27T12:15:17.885 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.884+0000 7f824d83a700 1 -- 172.21.15.195:0/1457092783 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f82480c78c0 con 0x7f824806d280 2022-06-27T12:15:17.886 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.884+0000 7f822effd700 1 -- 172.21.15.195:0/1457092783 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+6131 (secure 0 0 0) 0x7f823c052500 con 0x7f824806d280 2022-06-27T12:15:17.887 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:15:17.887 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":101,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":6254,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/1317767101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1317767101},{"type":"v1","addr":"172.21.15.202:6833","nonce":1317767101}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":101}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:13:05.314357+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6865},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6865":{"gid":6865,"name":"d","rank":0,"incarnation":97,"state":"up:active","state_seq":18,"addr":"172.21.15.202:6835/2349973547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2349973547},{"type":"v1","addr":"172.21.15.202:6835","nonce":2349973547}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:15:17.895 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.886+0000 7f824d83a700 1 -- 172.21.15.195:0/1457092783 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8228063d30 msgr2=0x7f82280661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:17.895 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.886+0000 7f824d83a700 1 --2- 172.21.15.195:0/1457092783 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8228063d30 0x7f82280661e0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f82480c1c00 tx=0x7f8238017000 comp rx=0 tx=0).stop 2022-06-27T12:15:17.896 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.886+0000 7f824d83a700 1 -- 172.21.15.195:0/1457092783 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f824806d280 msgr2=0x7f8248154670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:17.897 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.886+0000 7f824d83a700 1 --2- 172.21.15.195:0/1457092783 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f824806d280 0x7f8248154670 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f823c00c590 tx=0x7f823c007cb0 comp rx=0 tx=0).stop 2022-06-27T12:15:17.898 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.886+0000 7f824d83a700 1 -- 172.21.15.195:0/1457092783 shutdown_connections 2022-06-27T12:15:17.898 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.886+0000 7f824d83a700 1 --2- 172.21.15.195:0/1457092783 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8228063d30 0x7f82280661e0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:17.899 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.886+0000 7f824d83a700 1 --2- 172.21.15.195:0/1457092783 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f824806d280 0x7f8248154670 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:17.899 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.886+0000 7f824d83a700 1 --2- 172.21.15.195:0/1457092783 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f82480c64f0 0x7f8248169890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:17.899 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.886+0000 7f824d83a700 1 --2- 172.21.15.195:0/1457092783 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f82480c5480 0x7f8248154bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:17.900 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.886+0000 7f824d83a700 1 -- 172.21.15.195:0/1457092783 >> 172.21.15.195:0/1457092783 conn(0x7f8248063af0 msgr2=0x7f824806d7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:15:17.900 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.887+0000 7f824d83a700 1 -- 172.21.15.195:0/1457092783 shutdown_connections 2022-06-27T12:15:17.901 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:17.887+0000 7f824d83a700 1 -- 172.21.15.195:0/1457092783 wait complete. 2022-06-27T12:15:17.901 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 101 2022-06-27T12:15:17.908 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-06-27T12:15:19.910 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:15:20.059 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.045+0000 7f4826c4a700 1 Processor -- start 2022-06-27T12:15:20.061 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.045+0000 7f4826c4a700 1 -- start start 2022-06-27T12:15:20.062 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.046+0000 7f4826c4a700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f4820068f80 0x7f48200c4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:20.064 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.046+0000 7f4826c4a700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f48200683c0 0x7f4820069350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:20.065 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.046+0000 7f4826c4a700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48201573b0 con 0x7f48200c7480 2022-06-27T12:15:20.065 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.046+0000 7f4826c4a700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4820068840 con 0x7f4820068f80 2022-06-27T12:15:20.066 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.046+0000 7f4826c4a700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48200c46c0 con 0x7f48200683c0 2022-06-27T12:15:20.066 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.046+0000 7f48249e6700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f4820068f80 0x7f48200c4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:20.066 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.046+0000 7f48249e6700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f4820068f80 0x7f48200c4b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:51596/0 (socket says 172.21.15.195:51596) 2022-06-27T12:15:20.066 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.046+0000 7f48249e6700 1 -- 172.21.15.195:0/2148958141 learned_addr learned my addr 172.21.15.195:0/2148958141 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:15:20.067 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.046+0000 7f481ffff700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f48200c7480 0x7f4820156460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:57786/0 (socket says 172.21.15.195:57786) 2022-06-27T12:15:20.067 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.046+0000 7f48251e7700 1 --2- 172.21.15.195:0/2148958141 >> v2:172.21.15.202:3300/0 conn(0x7f48200683c0 0x7f4820069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:20.067 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.047+0000 7f48249e6700 1 -- 172.21.15.195:0/2148958141 >> v2:172.21.15.202:3300/0 conn(0x7f48200683c0 msgr2=0x7f4820069350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:15:20.067 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.047+0000 7f48249e6700 1 --2- 172.21.15.195:0/2148958141 >> v2:172.21.15.202:3300/0 conn(0x7f48200683c0 0x7f4820069350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:20.068 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.047+0000 7f48249e6700 1 -- 172.21.15.195:0/2148958141 >> v1:172.21.15.195:6789/0 conn(0x7f48200c7480 legacy=0x7f4820156460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:20.068 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.047+0000 7f481f7fe700 1 -- 172.21.15.195:0/2148958141 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1180878216 0 0) 0x7f48201573b0 con 0x7f48200c7480 2022-06-27T12:15:20.068 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.047+0000 7f48249e6700 1 -- 172.21.15.195:0/2148958141 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4820156b70 con 0x7f4820068f80 2022-06-27T12:15:20.068 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.047+0000 7f48249e6700 1 --2- 172.21.15.195:0/2148958141 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4820068f80 0x7f48200c4b50 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f48200705a0 tx=0x7f4814009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3b6e2db88f6ab7b server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:20.069 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.047+0000 7f481f7fe700 1 -- 172.21.15.195:0/2148958141 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4814011630 con 0x7f4820068f80 2022-06-27T12:15:20.069 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.047+0000 7f481f7fe700 1 -- 172.21.15.195:0/2148958141 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4814011790 con 0x7f4820068f80 2022-06-27T12:15:20.069 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.047+0000 7f481f7fe700 1 -- 172.21.15.195:0/2148958141 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4814011b30 con 0x7f4820068f80 2022-06-27T12:15:20.069 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.047+0000 7f4826c4a700 1 -- 172.21.15.195:0/2148958141 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4820068f80 msgr2=0x7f48200c4b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:20.070 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.047+0000 7f4826c4a700 1 --2- 172.21.15.195:0/2148958141 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4820068f80 0x7f48200c4b50 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f48200705a0 tx=0x7f4814009120 comp rx=0 tx=0).stop 2022-06-27T12:15:20.070 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.048+0000 7f4826c4a700 1 -- 172.21.15.195:0/2148958141 shutdown_connections 2022-06-27T12:15:20.070 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.048+0000 7f4826c4a700 1 --2- 172.21.15.195:0/2148958141 >> v2:172.21.15.202:3300/0 conn(0x7f48200683c0 0x7f4820069350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:20.070 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.048+0000 7f4826c4a700 1 --2- 172.21.15.195:0/2148958141 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4820068f80 0x7f48200c4b50 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:20.071 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.048+0000 7f4826c4a700 1 -- 172.21.15.195:0/2148958141 >> 172.21.15.195:0/2148958141 conn(0x7f48200ccbd0 msgr2=0x7f48200ccfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:15:20.071 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.048+0000 7f4826c4a700 1 -- 172.21.15.195:0/2148958141 shutdown_connections 2022-06-27T12:15:20.071 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.048+0000 7f4826c4a700 1 -- 172.21.15.195:0/2148958141 wait complete. 2022-06-27T12:15:20.071 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.049+0000 7f4826c4a700 1 Processor -- start 2022-06-27T12:15:20.072 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.049+0000 7f4826c4a700 1 -- start start 2022-06-27T12:15:20.072 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.050+0000 7f4826c4a700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f48200683c0 0x7f48200c9820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:20.072 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.050+0000 7f4826c4a700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4820068f80 0x7f482014dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:20.072 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.050+0000 7f4826c4a700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f48200c7480 0x7f482014e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:20.073 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.050+0000 7f4826c4a700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48201665a0 con 0x7f48200683c0 2022-06-27T12:15:20.073 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.050+0000 7f4826c4a700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48201667f0 con 0x7f48200c7480 2022-06-27T12:15:20.073 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.050+0000 7f4826c4a700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4820166ad0 con 0x7f4820068f80 2022-06-27T12:15:20.074 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.050+0000 7f48249e6700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f48200683c0 0x7f48200c9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:20.075 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.050+0000 7f48249e6700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f48200683c0 0x7f48200c9820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:51602/0 (socket says 172.21.15.195:51602) 2022-06-27T12:15:20.075 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.050+0000 7f48249e6700 1 -- 172.21.15.195:0/2008906818 learned_addr learned my addr 172.21.15.195:0/2008906818 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:15:20.076 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.050+0000 7f48251e7700 1 --2- 172.21.15.195:0/2008906818 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f48200c7480 0x7f482014e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:20.076 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.050+0000 7f481ffff700 1 --2- 172.21.15.195:0/2008906818 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4820068f80 0x7f482014dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:20.077 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.050+0000 7f48249e6700 1 -- 172.21.15.195:0/2008906818 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4820068f80 msgr2=0x7f482014dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:20.077 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.050+0000 7f48249e6700 1 --2- 172.21.15.195:0/2008906818 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4820068f80 0x7f482014dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:20.078 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.050+0000 7f48249e6700 1 -- 172.21.15.195:0/2008906818 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f48200c7480 msgr2=0x7f482014e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:20.078 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.050+0000 7f48249e6700 1 --2- 172.21.15.195:0/2008906818 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f48200c7480 0x7f482014e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:20.078 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.050+0000 7f48249e6700 1 -- 172.21.15.195:0/2008906818 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48140093a0 con 0x7f48200683c0 2022-06-27T12:15:20.079 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.050+0000 7f48249e6700 1 --2- 172.21.15.195:0/2008906818 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f48200683c0 0x7f48200c9820 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f48200cdc40 tx=0x7f4814017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:20.079 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.050+0000 7f481cff9700 1 -- 172.21.15.195:0/2008906818 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4814003730 con 0x7f48200683c0 2022-06-27T12:15:20.080 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.050+0000 7f481cff9700 1 -- 172.21.15.195:0/2008906818 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4814003890 con 0x7f48200683c0 2022-06-27T12:15:20.080 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.050+0000 7f4826c4a700 1 -- 172.21.15.195:0/2008906818 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4820166db0 con 0x7f48200683c0 2022-06-27T12:15:20.081 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.050+0000 7f481cff9700 1 -- 172.21.15.195:0/2008906818 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4814003c50 con 0x7f48200683c0 2022-06-27T12:15:20.081 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.050+0000 7f4826c4a700 1 -- 172.21.15.195:0/2008906818 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4820156750 con 0x7f48200683c0 2022-06-27T12:15:20.084 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.051+0000 7f4826c4a700 1 -- 172.21.15.195:0/2008906818 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48200722c0 con 0x7f48200683c0 2022-06-27T12:15:20.084 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.052+0000 7f481cff9700 1 -- 172.21.15.195:0/2008906818 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f481402b030 con 0x7f48200683c0 2022-06-27T12:15:20.085 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.052+0000 7f481cff9700 1 --2- 172.21.15.195:0/2008906818 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4808063d00 0x7f48080661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:20.085 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.052+0000 7f481cff9700 1 -- 172.21.15.195:0/2008906818 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7492+0+0 (secure 0 0 0) 0x7f48140823f0 con 0x7f48200683c0 2022-06-27T12:15:20.086 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.052+0000 7f481ffff700 1 --2- 172.21.15.195:0/2008906818 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4808063d00 0x7f48080661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:20.086 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.053+0000 7f481ffff700 1 --2- 172.21.15.195:0/2008906818 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4808063d00 0x7f48080661b0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f48201590b0 tx=0x7f480c007bf0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:20.086 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.056+0000 7f481cff9700 1 -- 172.21.15.195:0/2008906818 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f4814050a40 con 0x7f48200683c0 2022-06-27T12:15:20.285 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.285+0000 7f4826c4a700 1 -- 172.21.15.195:0/2008906818 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f48200c39c0 con 0x7f48200683c0 2022-06-27T12:15:20.286 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.285+0000 7f481cff9700 1 -- 172.21.15.195:0/2008906818 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+5370 (secure 0 0 0) 0x7f4814015090 con 0x7f48200683c0 2022-06-27T12:15:20.286 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:15:20.287 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":102,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:15:20.199910+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6254":{"gid":6254,"name":"b","rank":0,"incarnation":102,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6833/1317767101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1317767101},{"type":"v1","addr":"172.21.15.202:6833","nonce":1317767101}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:15:20.290 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.288+0000 7f4826c4a700 1 -- 172.21.15.195:0/2008906818 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4808063d00 msgr2=0x7f48080661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:20.290 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.288+0000 7f4826c4a700 1 --2- 172.21.15.195:0/2008906818 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4808063d00 0x7f48080661b0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f48201590b0 tx=0x7f480c007bf0 comp rx=0 tx=0).stop 2022-06-27T12:15:20.291 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.288+0000 7f4826c4a700 1 -- 172.21.15.195:0/2008906818 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f48200683c0 msgr2=0x7f48200c9820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:20.291 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.288+0000 7f4826c4a700 1 --2- 172.21.15.195:0/2008906818 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f48200683c0 0x7f48200c9820 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f48200cdc40 tx=0x7f4814017000 comp rx=0 tx=0).stop 2022-06-27T12:15:20.291 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.288+0000 7f4826c4a700 1 -- 172.21.15.195:0/2008906818 shutdown_connections 2022-06-27T12:15:20.292 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.288+0000 7f4826c4a700 1 --2- 172.21.15.195:0/2008906818 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4808063d00 0x7f48080661b0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:20.292 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.288+0000 7f4826c4a700 1 --2- 172.21.15.195:0/2008906818 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f48200683c0 0x7f48200c9820 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:20.292 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.288+0000 7f4826c4a700 1 --2- 172.21.15.195:0/2008906818 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f48200c7480 0x7f482014e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:20.292 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.288+0000 7f4826c4a700 1 --2- 172.21.15.195:0/2008906818 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4820068f80 0x7f482014dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:20.293 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.288+0000 7f4826c4a700 1 -- 172.21.15.195:0/2008906818 >> 172.21.15.195:0/2008906818 conn(0x7f48200ccbd0 msgr2=0x7f4820150240 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:15:20.293 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.288+0000 7f4826c4a700 1 -- 172.21.15.195:0/2008906818 shutdown_connections 2022-06-27T12:15:20.293 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.288+0000 7f4826c4a700 1 -- 172.21.15.195:0/2008906818 wait complete. 2022-06-27T12:15:20.294 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 102 2022-06-27T12:15:20.301 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d down, removed from mdsmap 2022-06-27T12:15:20.301 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2022-06-27T12:15:20.302 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:15:20.423 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.422+0000 7ffb6adbe700 1 Processor -- start 2022-06-27T12:15:20.423 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.422+0000 7ffb6adbe700 1 -- start start 2022-06-27T12:15:20.436 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.423+0000 7ffb6adbe700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ffb6414e490 0x7ffb6406cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:20.437 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.423+0000 7ffb6adbe700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb640c33b0 con 0x7ffb6414e860 2022-06-27T12:15:20.437 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.423+0000 7ffb6adbe700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb6406d520 con 0x7ffb640c3f50 2022-06-27T12:15:20.437 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.423+0000 7ffb6adbe700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb6406f220 con 0x7ffb6414e490 2022-06-27T12:15:20.437 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.423+0000 7ffb63fff700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7ffb6414e860 0x7ffb640c3840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:57800/0 (socket says 172.21.15.195:57800) 2022-06-27T12:15:20.440 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.423+0000 7ffb63fff700 1 -- 172.21.15.195:0/1347248632 learned_addr learned my addr 172.21.15.195:0/1347248632 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:15:20.446 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.423+0000 7ffb68b5a700 1 --2- 172.21.15.195:0/1347248632 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ffb6414e490 0x7ffb6406cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:20.449 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.424+0000 7ffb637fe700 1 -- 172.21.15.195:0/1347248632 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1527330892 0 0) 0x7ffb640c33b0 con 0x7ffb6414e860 2022-06-27T12:15:20.449 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.424+0000 7ffb637fe700 1 -- 172.21.15.195:0/1347248632 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb48003430 con 0x7ffb6414e860 2022-06-27T12:15:20.452 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.424+0000 7ffb637fe700 1 -- 172.21.15.195:0/1347248632 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2589960778 0 0) 0x7ffb6406d520 con 0x7ffb640c3f50 2022-06-27T12:15:20.452 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.424+0000 7ffb637fe700 1 -- 172.21.15.195:0/1347248632 --> v1:172.21.15.202:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb640c33b0 con 0x7ffb640c3f50 2022-06-27T12:15:20.453 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.424+0000 7ffb68b5a700 1 -- 172.21.15.195:0/1347248632 >> v1:172.21.15.202:6789/0 conn(0x7ffb640c3f50 legacy=0x7ffb64157890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:20.453 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.424+0000 7ffb68b5a700 1 -- 172.21.15.195:0/1347248632 >> v1:172.21.15.195:6789/0 conn(0x7ffb6414e860 legacy=0x7ffb640c3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:20.454 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.424+0000 7ffb68b5a700 1 -- 172.21.15.195:0/1347248632 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb641542c0 con 0x7ffb6414e490 2022-06-27T12:15:20.454 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.424+0000 7ffb68b5a700 1 --2- 172.21.15.195:0/1347248632 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ffb6414e490 0x7ffb6406cfe0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ffb580097d0 tx=0x7ffb58009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b94459ad87cdaef server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:20.454 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.424+0000 7ffb637fe700 1 -- 172.21.15.195:0/1347248632 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb58011630 con 0x7ffb6414e490 2022-06-27T12:15:20.455 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.424+0000 7ffb637fe700 1 -- 172.21.15.195:0/1347248632 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb58011790 con 0x7ffb6414e490 2022-06-27T12:15:20.455 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.424+0000 7ffb637fe700 1 -- 172.21.15.195:0/1347248632 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb58011b10 con 0x7ffb6414e490 2022-06-27T12:15:20.456 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.425+0000 7ffb6adbe700 1 -- 172.21.15.195:0/1347248632 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ffb6414e490 msgr2=0x7ffb6406cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:20.456 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.425+0000 7ffb6adbe700 1 --2- 172.21.15.195:0/1347248632 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ffb6414e490 0x7ffb6406cfe0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ffb580097d0 tx=0x7ffb58009060 comp rx=0 tx=0).stop 2022-06-27T12:15:20.457 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.425+0000 7ffb6adbe700 1 -- 172.21.15.195:0/1347248632 shutdown_connections 2022-06-27T12:15:20.457 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.425+0000 7ffb6adbe700 1 --2- 172.21.15.195:0/1347248632 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ffb6414e490 0x7ffb6406cfe0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:20.458 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.425+0000 7ffb6adbe700 1 -- 172.21.15.195:0/1347248632 >> 172.21.15.195:0/1347248632 conn(0x7ffb64063af0 msgr2=0x7ffb64063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:15:20.458 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.425+0000 7ffb6adbe700 1 -- 172.21.15.195:0/1347248632 shutdown_connections 2022-06-27T12:15:20.458 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.425+0000 7ffb6adbe700 1 -- 172.21.15.195:0/1347248632 wait complete. 2022-06-27T12:15:20.459 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.427+0000 7ffb6adbe700 1 Processor -- start 2022-06-27T12:15:20.459 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.427+0000 7ffb6adbe700 1 -- start start 2022-06-27T12:15:20.460 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.428+0000 7ffb6adbe700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ffb640c3f50 0x7ffb640bf170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:20.463 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.428+0000 7ffb6adbe700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ffb6414e490 0x7ffb6414b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:20.463 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.428+0000 7ffb6adbe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffb6414e860 0x7ffb640c0b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:20.463 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.428+0000 7ffb6adbe700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb640c7f90 con 0x7ffb6414e860 2022-06-27T12:15:20.464 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.428+0000 7ffb6adbe700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb640c9610 con 0x7ffb640c3f50 2022-06-27T12:15:20.464 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.428+0000 7ffb6adbe700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb64166dd0 con 0x7ffb6414e490 2022-06-27T12:15:20.465 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.428+0000 7ffb6935b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffb6414e860 0x7ffb640c0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:20.465 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.428+0000 7ffb68b5a700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ffb640c3f50 0x7ffb640bf170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:20.466 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.428+0000 7ffb63fff700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ffb6414e490 0x7ffb6414b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:20.467 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.428+0000 7ffb6935b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffb6414e860 0x7ffb640c0b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:51620/0 (socket says 172.21.15.195:51620) 2022-06-27T12:15:20.467 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.428+0000 7ffb6935b700 1 -- 172.21.15.195:0/746291832 learned_addr learned my addr 172.21.15.195:0/746291832 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:15:20.468 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.428+0000 7ffb68b5a700 1 -- 172.21.15.195:0/746291832 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ffb6414e490 msgr2=0x7ffb6414b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:20.468 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.428+0000 7ffb68b5a700 1 --2- 172.21.15.195:0/746291832 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ffb6414e490 0x7ffb6414b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:20.469 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.428+0000 7ffb68b5a700 1 -- 172.21.15.195:0/746291832 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffb6414e860 msgr2=0x7ffb640c0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:20.469 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.428+0000 7ffb68b5a700 1 --2- 172.21.15.195:0/746291832 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffb6414e860 0x7ffb640c0b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:20.470 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.428+0000 7ffb68b5a700 1 -- 172.21.15.195:0/746291832 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb580092e0 con 0x7ffb640c3f50 2022-06-27T12:15:20.470 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.428+0000 7ffb6935b700 1 --2- 172.21.15.195:0/746291832 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffb6414e860 0x7ffb640c0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:15:20.471 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.428+0000 7ffb68b5a700 1 --2- 172.21.15.195:0/746291832 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ffb640c3f50 0x7ffb640bf170 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7ffb58016000 tx=0x7ffb58017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:20.473 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.428+0000 7ffb60ff9700 1 -- 172.21.15.195:0/746291832 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb58003800 con 0x7ffb640c3f50 2022-06-27T12:15:20.473 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.429+0000 7ffb60ff9700 1 -- 172.21.15.195:0/746291832 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb58003980 con 0x7ffb640c3f50 2022-06-27T12:15:20.474 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.429+0000 7ffb6adbe700 1 -- 172.21.15.195:0/746291832 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb641661e0 con 0x7ffb640c3f50 2022-06-27T12:15:20.474 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.429+0000 7ffb60ff9700 1 -- 172.21.15.195:0/746291832 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb58003d40 con 0x7ffb640c3f50 2022-06-27T12:15:20.475 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.429+0000 7ffb6adbe700 1 -- 172.21.15.195:0/746291832 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb641664a0 con 0x7ffb640c3f50 2022-06-27T12:15:20.475 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.429+0000 7ffb6adbe700 1 -- 172.21.15.195:0/746291832 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb640722c0 con 0x7ffb640c3f50 2022-06-27T12:15:20.476 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.430+0000 7ffb60ff9700 1 -- 172.21.15.195:0/746291832 <== mon.1 v2:172.21.15.202:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7ffb5800f050 con 0x7ffb640c3f50 2022-06-27T12:15:20.476 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.430+0000 7ffb60ff9700 1 --2- 172.21.15.195:0/746291832 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ffb4c063d00 0x7ffb4c0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:20.476 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.430+0000 7ffb60ff9700 1 -- 172.21.15.195:0/746291832 <== mon.1 v2:172.21.15.202:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7578+0+0 (secure 0 0 0) 0x7ffb58081b10 con 0x7ffb640c3f50 2022-06-27T12:15:20.477 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.430+0000 7ffb63fff700 1 --2- 172.21.15.195:0/746291832 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ffb4c063d00 0x7ffb4c0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:20.477 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.431+0000 7ffb63fff700 1 --2- 172.21.15.195:0/746291832 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ffb4c063d00 0x7ffb4c0661b0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7ffb64152f30 tx=0x7ffb50009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:20.478 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.435+0000 7ffb60ff9700 1 -- 172.21.15.195:0/746291832 <== mon.1 v2:172.21.15.202:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7ffb58050110 con 0x7ffb640c3f50 2022-06-27T12:15:20.679 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.668+0000 7ffb6adbe700 1 -- 172.21.15.195:0/746291832 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffb640c9c00 con 0x7ffb640c3f50 2022-06-27T12:15:20.680 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.668+0000 7ffb60ff9700 1 -- 172.21.15.195:0/746291832 <== mon.1 v2:172.21.15.202:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+5370 (secure 0 0 0) 0x7ffb58015090 con 0x7ffb640c3f50 2022-06-27T12:15:20.689 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:15:20.690 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":102,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:15:20.199910+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6254":{"gid":6254,"name":"b","rank":0,"incarnation":102,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6833/1317767101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1317767101},{"type":"v1","addr":"172.21.15.202:6833","nonce":1317767101}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:15:20.694 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.671+0000 7ffb6adbe700 1 -- 172.21.15.195:0/746291832 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ffb4c063d00 msgr2=0x7ffb4c0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:20.695 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.671+0000 7ffb6adbe700 1 --2- 172.21.15.195:0/746291832 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ffb4c063d00 0x7ffb4c0661b0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7ffb64152f30 tx=0x7ffb50009040 comp rx=0 tx=0).stop 2022-06-27T12:15:20.695 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.671+0000 7ffb6adbe700 1 -- 172.21.15.195:0/746291832 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ffb640c3f50 msgr2=0x7ffb640bf170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:20.695 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.671+0000 7ffb6adbe700 1 --2- 172.21.15.195:0/746291832 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ffb640c3f50 0x7ffb640bf170 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7ffb58016000 tx=0x7ffb58017000 comp rx=0 tx=0).stop 2022-06-27T12:15:20.695 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.671+0000 7ffb6adbe700 1 -- 172.21.15.195:0/746291832 shutdown_connections 2022-06-27T12:15:20.696 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.671+0000 7ffb6adbe700 1 --2- 172.21.15.195:0/746291832 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ffb4c063d00 0x7ffb4c0661b0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:20.696 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.671+0000 7ffb6adbe700 1 --2- 172.21.15.195:0/746291832 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffb6414e860 0x7ffb640c0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:20.696 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.671+0000 7ffb6adbe700 1 --2- 172.21.15.195:0/746291832 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ffb640c3f50 0x7ffb640bf170 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:20.696 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.671+0000 7ffb6adbe700 1 --2- 172.21.15.195:0/746291832 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ffb6414e490 0x7ffb6414b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:20.697 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.671+0000 7ffb6adbe700 1 -- 172.21.15.195:0/746291832 >> 172.21.15.195:0/746291832 conn(0x7ffb64063af0 msgr2=0x7ffb64063ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:15:20.697 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.671+0000 7ffb6adbe700 1 -- 172.21.15.195:0/746291832 shutdown_connections 2022-06-27T12:15:20.697 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:20.671+0000 7ffb6adbe700 1 -- 172.21.15.195:0/746291832 wait complete. 2022-06-27T12:15:20.697 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 102 2022-06-27T12:15:20.699 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 102, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-06-27T11:47:52.358095+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 102, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6254": { "addr": "172.21.15.202:6833/1317767101", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6832", "nonce": 1317767101, "type": "v2" }, { "addr": "172.21.15.202:6833", "nonce": 1317767101, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 6254, "incarnation": 102, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 64, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-06-27T12:15:20.199910+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6254 } } } ], "standbys": [ { "addr": "172.21.15.202:6837/3305006841", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6836", "nonce": 3305006841, "type": "v2" }, { "addr": "172.21.15.202:6837", "nonce": 3305006841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6835/2060709822", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6834", "nonce": 2060709822, "type": "v2" }, { "addr": "172.21.15.195:6835", "nonce": 2060709822, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6838/1598751652", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6836", "nonce": 1598751652, "type": "v2" }, { "addr": "172.21.15.195:6838", "nonce": 1598751652, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4275, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6839/1313800661", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6837", "nonce": 1313800661, "type": "v2" }, { "addr": "172.21.15.195:6839", "nonce": 1313800661, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4278, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-06-27T12:15:22.701 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:15:22.836 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:22.834+0000 7fa635bb3700 1 Processor -- start 2022-06-27T12:15:22.837 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:22.834+0000 7fa635bb3700 1 -- start start 2022-06-27T12:15:22.837 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:22.834+0000 7fa635bb3700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa63014e490 0x7fa63006cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:22.837 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:22.834+0000 7fa635bb3700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fa6300c3f50 0x7fa630157890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:22.838 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:22.834+0000 7fa635bb3700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6300c33b0 con 0x7fa63014e860 2022-06-27T12:15:22.838 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:22.834+0000 7fa635bb3700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa63006d520 con 0x7fa6300c3f50 2022-06-27T12:15:22.839 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:22.835+0000 7fa635bb3700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa63006f220 con 0x7fa63014e490 2022-06-27T12:15:22.839 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:22.835+0000 7fa62ffff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fa6300c3f50 0x7fa630157890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:22.839 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:22.835+0000 7fa62effd700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fa63014e860 0x7fa6300c3840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:57812/0 (socket says 172.21.15.195:57812) 2022-06-27T12:15:22.839 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:22.835+0000 7fa62effd700 1 -- 172.21.15.195:0/2826192741 learned_addr learned my addr 172.21.15.195:0/2826192741 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:15:22.840 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:22.835+0000 7fa62f7fe700 1 --2- 172.21.15.195:0/2826192741 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa63014e490 0x7fa63006cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:22.840 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:22.835+0000 7fa62ffff700 1 -- 172.21.15.195:0/2826192741 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa63014e490 msgr2=0x7fa63006cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:15:22.840 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:22.835+0000 7fa62ffff700 1 --2- 172.21.15.195:0/2826192741 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa63014e490 0x7fa63006cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:22.840 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:22.835+0000 7fa62ffff700 1 -- 172.21.15.195:0/2826192741 >> v1:172.21.15.195:6789/0 conn(0x7fa63014e860 legacy=0x7fa6300c3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:22.841 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:22.835+0000 7fa62ffff700 1 -- 172.21.15.195:0/2826192741 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6301542c0 con 0x7fa6300c3f50 2022-06-27T12:15:22.841 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:22.835+0000 7fa62e7fc700 1 -- 172.21.15.195:0/2826192741 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3708121537 0 0) 0x7fa6300c33b0 con 0x7fa63014e860 2022-06-27T12:15:22.841 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:22.835+0000 7fa62ffff700 1 --2- 172.21.15.195:0/2826192741 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa6300c3f50 0x7fa630157890 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fa628009e60 tx=0x7fa628009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=518c4d00ddf71f89 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:22.841 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:22.835+0000 7fa62e7fc700 1 -- 172.21.15.195:0/2826192741 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa62800e070 con 0x7fa6300c3f50 2022-06-27T12:15:22.842 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:22.835+0000 7fa62e7fc700 1 -- 172.21.15.195:0/2826192741 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa62800bb60 con 0x7fa6300c3f50 2022-06-27T12:15:22.842 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:22.835+0000 7fa62e7fc700 1 -- 172.21.15.195:0/2826192741 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa628012430 con 0x7fa6300c3f50 2022-06-27T12:15:22.842 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:22.836+0000 7fa635bb3700 1 -- 172.21.15.195:0/2826192741 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa6300c3f50 msgr2=0x7fa630157890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:22.843 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:22.836+0000 7fa635bb3700 1 --2- 172.21.15.195:0/2826192741 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa6300c3f50 0x7fa630157890 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fa628009e60 tx=0x7fa628009780 comp rx=0 tx=0).stop 2022-06-27T12:15:22.843 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:22.836+0000 7fa635bb3700 1 -- 172.21.15.195:0/2826192741 shutdown_connections 2022-06-27T12:15:22.843 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:22.836+0000 7fa635bb3700 1 --2- 172.21.15.195:0/2826192741 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa6300c3f50 0x7fa630157890 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:22.843 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:22.836+0000 7fa635bb3700 1 --2- 172.21.15.195:0/2826192741 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa63014e490 0x7fa63006cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:22.844 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:22.836+0000 7fa635bb3700 1 -- 172.21.15.195:0/2826192741 >> 172.21.15.195:0/2826192741 conn(0x7fa630063af0 msgr2=0x7fa630063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:15:22.844 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:22.836+0000 7fa635bb3700 1 -- 172.21.15.195:0/2826192741 shutdown_connections 2022-06-27T12:15:22.844 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:22.836+0000 7fa635bb3700 1 -- 172.21.15.195:0/2826192741 wait complete. 2022-06-27T12:15:22.846 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:22.837+0000 7fa635bb3700 1 Processor -- start 2022-06-27T12:15:22.846 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:22.837+0000 7fa635bb3700 1 -- start start 2022-06-27T12:15:22.847 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:22.838+0000 7fa635bb3700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa6300c3f50 0x7fa630167e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:22.847 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:22.838+0000 7fa635bb3700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa63014e490 0x7fa63014bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:22.847 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:22.838+0000 7fa635bb3700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa63014e860 0x7fa6301460a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:22.847 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:22.838+0000 7fa635bb3700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa630164f50 con 0x7fa63014e860 2022-06-27T12:15:22.848 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:22.838+0000 7fa635bb3700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6301640f0 con 0x7fa6300c3f50 2022-06-27T12:15:22.848 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:22.838+0000 7fa635bb3700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6301643d0 con 0x7fa63014e490 2022-06-27T12:15:22.849 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:22.838+0000 7fa62ffff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa63014e860 0x7fa6301460a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:22.849 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:22.838+0000 7fa62ffff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa63014e860 0x7fa6301460a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:51634/0 (socket says 172.21.15.195:51634) 2022-06-27T12:15:22.849 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:22.838+0000 7fa62ffff700 1 -- 172.21.15.195:0/2919142899 learned_addr learned my addr 172.21.15.195:0/2919142899 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:15:22.850 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:22.838+0000 7fa62f7fe700 1 --2- 172.21.15.195:0/2919142899 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa6300c3f50 0x7fa630167e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:22.850 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:22.838+0000 7fa62effd700 1 --2- 172.21.15.195:0/2919142899 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa63014e490 0x7fa63014bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:22.850 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:22.838+0000 7fa62ffff700 1 -- 172.21.15.195:0/2919142899 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa63014e490 msgr2=0x7fa63014bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:22.850 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:22.838+0000 7fa62ffff700 1 --2- 172.21.15.195:0/2919142899 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa63014e490 0x7fa63014bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:22.851 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:22.838+0000 7fa62ffff700 1 -- 172.21.15.195:0/2919142899 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa6300c3f50 msgr2=0x7fa630167e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:22.851 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:22.838+0000 7fa62ffff700 1 --2- 172.21.15.195:0/2919142899 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa6300c3f50 0x7fa630167e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:22.851 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:22.838+0000 7fa62ffff700 1 -- 172.21.15.195:0/2919142899 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa628009a20 con 0x7fa63014e860 2022-06-27T12:15:22.852 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:22.838+0000 7fa62ffff700 1 --2- 172.21.15.195:0/2919142899 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa63014e860 0x7fa6301460a0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7fa630070770 tx=0x7fa628019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:22.852 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:22.839+0000 7fa6177fe700 1 -- 172.21.15.195:0/2919142899 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa628003ea0 con 0x7fa63014e860 2022-06-27T12:15:22.853 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:22.839+0000 7fa6177fe700 1 -- 172.21.15.195:0/2919142899 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6280129b0 con 0x7fa63014e860 2022-06-27T12:15:22.854 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:22.839+0000 7fa6177fe700 1 -- 172.21.15.195:0/2919142899 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa628012d70 con 0x7fa63014e860 2022-06-27T12:15:22.855 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:22.839+0000 7fa635bb3700 1 -- 172.21.15.195:0/2919142899 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6301646b0 con 0x7fa63014e860 2022-06-27T12:15:22.855 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:22.839+0000 7fa635bb3700 1 -- 172.21.15.195:0/2919142899 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa630164c70 con 0x7fa63014e860 2022-06-27T12:15:22.855 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:22.840+0000 7fa635bb3700 1 -- 172.21.15.195:0/2919142899 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6300722c0 con 0x7fa63014e860 2022-06-27T12:15:22.855 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:22.840+0000 7fa6177fe700 1 -- 172.21.15.195:0/2919142899 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fa628004250 con 0x7fa63014e860 2022-06-27T12:15:22.856 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:22.840+0000 7fa6177fe700 1 --2- 172.21.15.195:0/2919142899 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa610063d30 0x7fa6100661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:22.856 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:22.840+0000 7fa6177fe700 1 -- 172.21.15.195:0/2919142899 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7578+0+0 (secure 0 0 0) 0x7fa628083720 con 0x7fa63014e860 2022-06-27T12:15:22.856 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:22.840+0000 7fa62f7fe700 1 --2- 172.21.15.195:0/2919142899 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa610063d30 0x7fa6100661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:22.866 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:22.841+0000 7fa62f7fe700 1 --2- 172.21.15.195:0/2919142899 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa610063d30 0x7fa6100661e0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fa620009c70 tx=0x7fa620009060 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:22.867 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:22.845+0000 7fa6177fe700 1 -- 172.21.15.195:0/2919142899 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fa628051d20 con 0x7fa63014e860 2022-06-27T12:15:23.077 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:23.077+0000 7fa635bb3700 1 -- 172.21.15.195:0/2919142899 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa63014d770 con 0x7fa63014e860 2022-06-27T12:15:23.079 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:23.077+0000 7fa6177fe700 1 -- 172.21.15.195:0/2919142899 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+5370 (secure 0 0 0) 0x7fa62802c020 con 0x7fa63014e860 2022-06-27T12:15:23.080 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:15:23.080 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":102,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:15:20.199910+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6254":{"gid":6254,"name":"b","rank":0,"incarnation":102,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6833/1317767101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1317767101},{"type":"v1","addr":"172.21.15.202:6833","nonce":1317767101}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:15:23.081 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:23.079+0000 7fa635bb3700 1 -- 172.21.15.195:0/2919142899 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa610063d30 msgr2=0x7fa6100661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:23.082 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:23.079+0000 7fa635bb3700 1 --2- 172.21.15.195:0/2919142899 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa610063d30 0x7fa6100661e0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fa620009c70 tx=0x7fa620009060 comp rx=0 tx=0).stop 2022-06-27T12:15:23.082 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:23.079+0000 7fa635bb3700 1 -- 172.21.15.195:0/2919142899 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa63014e860 msgr2=0x7fa6301460a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:23.082 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:23.079+0000 7fa635bb3700 1 --2- 172.21.15.195:0/2919142899 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa63014e860 0x7fa6301460a0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7fa630070770 tx=0x7fa628019000 comp rx=0 tx=0).stop 2022-06-27T12:15:23.083 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:23.079+0000 7fa635bb3700 1 -- 172.21.15.195:0/2919142899 shutdown_connections 2022-06-27T12:15:23.083 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:23.079+0000 7fa635bb3700 1 --2- 172.21.15.195:0/2919142899 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa610063d30 0x7fa6100661e0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:23.083 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:23.079+0000 7fa635bb3700 1 --2- 172.21.15.195:0/2919142899 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa63014e860 0x7fa6301460a0 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:23.083 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:23.079+0000 7fa635bb3700 1 --2- 172.21.15.195:0/2919142899 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa6300c3f50 0x7fa630167e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:23.084 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:23.079+0000 7fa635bb3700 1 --2- 172.21.15.195:0/2919142899 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa63014e490 0x7fa63014bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:23.084 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:23.079+0000 7fa635bb3700 1 -- 172.21.15.195:0/2919142899 >> 172.21.15.195:0/2919142899 conn(0x7fa630063af0 msgr2=0x7fa6300c9da0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:15:23.084 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:23.079+0000 7fa635bb3700 1 -- 172.21.15.195:0/2919142899 shutdown_connections 2022-06-27T12:15:23.084 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:23.079+0000 7fa635bb3700 1 -- 172.21.15.195:0/2919142899 wait complete. 2022-06-27T12:15:23.085 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 102 2022-06-27T12:15:23.092 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:15:25.093 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:15:25.255 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.234+0000 7f1718a18700 1 Processor -- start 2022-06-27T12:15:25.256 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.234+0000 7f1718a18700 1 -- start start 2022-06-27T12:15:25.257 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.235+0000 7f1718a18700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f17140c48e0 0x7f1714158360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:25.257 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.235+0000 7f1718a18700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f17140c3830 0x7f17140c3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:25.257 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.235+0000 7f1718a18700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f171406d7f0 con 0x7f171406d070 2022-06-27T12:15:25.258 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.235+0000 7f1718a18700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17140c4210 con 0x7f17140c3830 2022-06-27T12:15:25.262 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.235+0000 7f1718a18700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1714154400 con 0x7f17140c48e0 2022-06-27T12:15:25.263 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.235+0000 7f1711d9b700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f171406d070 0x7f17140ca960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:57840/0 (socket says 172.21.15.195:57840) 2022-06-27T12:15:25.263 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.235+0000 7f1711d9b700 1 -- 172.21.15.195:0/3781624946 learned_addr learned my addr 172.21.15.195:0/3781624946 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:15:25.263 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.235+0000 7f171159a700 1 -- 172.21.15.195:0/3781624946 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2942801799 0 0) 0x7f171406d7f0 con 0x7f171406d070 2022-06-27T12:15:25.264 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.235+0000 7f171159a700 1 -- 172.21.15.195:0/3781624946 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16f8003430 con 0x7f171406d070 2022-06-27T12:15:25.264 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.235+0000 7f1712d9d700 1 --2- 172.21.15.195:0/3781624946 >> v2:172.21.15.202:3300/0 conn(0x7f17140c3830 0x7f17140c3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:25.264 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.235+0000 7f171259c700 1 --2- 172.21.15.195:0/3781624946 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f17140c48e0 0x7f1714158360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:25.265 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.236+0000 7f171159a700 1 -- 172.21.15.195:0/3781624946 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2893257947 0 0) 0x7f16f8003430 con 0x7f171406d070 2022-06-27T12:15:25.265 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.236+0000 7f171159a700 1 -- 172.21.15.195:0/3781624946 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f17140c48e0 msgr2=0x7f1714158360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:15:25.265 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.236+0000 7f171159a700 1 --2- 172.21.15.195:0/3781624946 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f17140c48e0 0x7f1714158360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:25.265 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.236+0000 7f171159a700 1 -- 172.21.15.195:0/3781624946 >> v2:172.21.15.202:3300/0 conn(0x7f17140c3830 msgr2=0x7f17140c3cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:15:25.266 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.236+0000 7f171159a700 1 --2- 172.21.15.195:0/3781624946 >> v2:172.21.15.202:3300/0 conn(0x7f17140c3830 0x7f17140c3cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:25.266 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.236+0000 7f171159a700 1 -- 172.21.15.195:0/3781624946 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17140c2060 con 0x7f171406d070 2022-06-27T12:15:25.266 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.236+0000 7f171159a700 1 -- 172.21.15.195:0/3781624946 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7f1708001470 con 0x7f171406d070 2022-06-27T12:15:25.266 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.236+0000 7f1712d9d700 1 --2- 172.21.15.195:0/3781624946 >> v2:172.21.15.202:3300/0 conn(0x7f17140c3830 0x7f17140c3cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:15:25.267 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.236+0000 7f171159a700 1 -- 172.21.15.195:0/3781624946 >> v1:172.21.15.195:6789/0 conn(0x7f171406d070 legacy=0x7f17140ca960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:25.267 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.236+0000 7f171159a700 1 --2- 172.21.15.195:0/3781624946 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f16f8003920 0x7f16f8003d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:25.267 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.236+0000 7f171159a700 1 --2- 172.21.15.195:0/3781624946 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f17140c48e0 0x7f16f80046d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:25.267 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.236+0000 7f171159a700 1 --2- 172.21.15.195:0/3781624946 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f16f8004c10 0x7f16f8005080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:25.268 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.236+0000 7f171159a700 1 -- 172.21.15.195:0/3781624946 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1714154400 con 0x7f16f8003920 2022-06-27T12:15:25.268 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.236+0000 7f171159a700 1 -- 172.21.15.195:0/3781624946 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f800b9d0 con 0x7f16f8004c10 2022-06-27T12:15:25.268 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.236+0000 7f171159a700 1 -- 172.21.15.195:0/3781624946 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16f800bb90 con 0x7f17140c48e0 2022-06-27T12:15:25.268 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.236+0000 7f171259c700 1 --2- 172.21.15.195:0/3781624946 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f16f8003920 0x7f16f8003d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:25.269 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.236+0000 7f1712d9d700 1 --2- 172.21.15.195:0/3781624946 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f16f8004c10 0x7f16f8005080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:25.269 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.236+0000 7f1711d9b700 1 --2- 172.21.15.195:0/3781624946 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f17140c48e0 0x7f16f80046d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:25.269 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.236+0000 7f171259c700 1 -- 172.21.15.195:0/3781624946 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f17140c48e0 msgr2=0x7f16f80046d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:15:25.269 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.236+0000 7f171259c700 1 --2- 172.21.15.195:0/3781624946 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f17140c48e0 0x7f16f80046d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:25.270 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.236+0000 7f171259c700 1 -- 172.21.15.195:0/3781624946 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f16f8004c10 msgr2=0x7f16f8005080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:15:25.270 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.236+0000 7f171259c700 1 --2- 172.21.15.195:0/3781624946 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f16f8004c10 0x7f16f8005080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:25.270 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.236+0000 7f171259c700 1 -- 172.21.15.195:0/3781624946 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f8003430 con 0x7f16f8003920 2022-06-27T12:15:25.270 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.236+0000 7f171259c700 1 --2- 172.21.15.195:0/3781624946 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f16f8003920 0x7f16f8003d30 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f170400ba20 tx=0x7f1704008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=545e5c3aac79f58c server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:25.271 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.236+0000 7f171159a700 1 -- 172.21.15.195:0/3781624946 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1704017070 con 0x7f16f8003920 2022-06-27T12:15:25.271 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.236+0000 7f171159a700 1 -- 172.21.15.195:0/3781624946 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f170400be60 con 0x7f16f8003920 2022-06-27T12:15:25.271 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.236+0000 7f171159a700 1 -- 172.21.15.195:0/3781624946 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1704004270 con 0x7f16f8003920 2022-06-27T12:15:25.271 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.237+0000 7f1718a18700 1 -- 172.21.15.195:0/3781624946 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f16f8003920 msgr2=0x7f16f8003d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:25.272 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.237+0000 7f1718a18700 1 --2- 172.21.15.195:0/3781624946 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f16f8003920 0x7f16f8003d30 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f170400ba20 tx=0x7f1704008040 comp rx=0 tx=0).stop 2022-06-27T12:15:25.272 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.237+0000 7f1712d9d700 1 -- 172.21.15.195:0/3781624946 reap_dead start 2022-06-27T12:15:25.272 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.237+0000 7f1718a18700 1 -- 172.21.15.195:0/3781624946 shutdown_connections 2022-06-27T12:15:25.272 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.237+0000 7f1718a18700 1 -- 172.21.15.195:0/3781624946 >> 172.21.15.195:0/3781624946 conn(0x7f1714063af0 msgr2=0x7f1714063ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:15:25.273 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.237+0000 7f1718a18700 1 -- 172.21.15.195:0/3781624946 shutdown_connections 2022-06-27T12:15:25.273 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.237+0000 7f1718a18700 1 -- 172.21.15.195:0/3781624946 wait complete. 2022-06-27T12:15:25.273 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.239+0000 7f1718a18700 1 Processor -- start 2022-06-27T12:15:25.273 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.239+0000 7f1718a18700 1 -- start start 2022-06-27T12:15:25.274 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.240+0000 7f1718a18700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f16f8003920 0x7f17140c9fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:25.287 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.240+0000 7f1718a18700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f171415a1a0 0x7f1714164db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:25.288 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.240+0000 7f1718a18700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f171414ea90 0x7f171414ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:25.288 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.240+0000 7f1718a18700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1714148ff0 con 0x7f16f8003920 2022-06-27T12:15:25.288 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.240+0000 7f1718a18700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1714156220 con 0x7f171415a1a0 2022-06-27T12:15:25.288 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.240+0000 7f1718a18700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17141652f0 con 0x7f171414ea90 2022-06-27T12:15:25.289 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.240+0000 7f171259c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f16f8003920 0x7f17140c9fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:25.289 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.240+0000 7f171259c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f16f8003920 0x7f17140c9fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:51662/0 (socket says 172.21.15.195:51662) 2022-06-27T12:15:25.289 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.240+0000 7f171259c700 1 -- 172.21.15.195:0/4048195261 learned_addr learned my addr 172.21.15.195:0/4048195261 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:15:25.290 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.240+0000 7f1712d9d700 1 --2- 172.21.15.195:0/4048195261 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f171414ea90 0x7f171414ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:25.290 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.240+0000 7f1711d9b700 1 --2- 172.21.15.195:0/4048195261 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f171415a1a0 0x7f1714164db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:25.290 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.240+0000 7f171259c700 1 -- 172.21.15.195:0/4048195261 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f171414ea90 msgr2=0x7f171414ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:25.290 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.240+0000 7f171259c700 1 --2- 172.21.15.195:0/4048195261 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f171414ea90 0x7f171414ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:25.291 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.240+0000 7f171259c700 1 -- 172.21.15.195:0/4048195261 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f171415a1a0 msgr2=0x7f1714164db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:25.291 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.240+0000 7f171259c700 1 --2- 172.21.15.195:0/4048195261 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f171415a1a0 0x7f1714164db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:25.291 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.240+0000 7f171259c700 1 -- 172.21.15.195:0/4048195261 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1714155610 con 0x7f16f8003920 2022-06-27T12:15:25.291 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.240+0000 7f171259c700 1 --2- 172.21.15.195:0/4048195261 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f16f8003920 0x7f17140c9fd0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f170400db90 tx=0x7f1704019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:25.291 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.240+0000 7f17027fc700 1 -- 172.21.15.195:0/4048195261 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f170400de80 con 0x7f16f8003920 2022-06-27T12:15:25.292 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.240+0000 7f17027fc700 1 -- 172.21.15.195:0/4048195261 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1704003730 con 0x7f16f8003920 2022-06-27T12:15:25.292 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.240+0000 7f1718a18700 1 -- 172.21.15.195:0/4048195261 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17141558f0 con 0x7f16f8003920 2022-06-27T12:15:25.292 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.240+0000 7f17027fc700 1 -- 172.21.15.195:0/4048195261 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1704003af0 con 0x7f16f8003920 2022-06-27T12:15:25.292 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.240+0000 7f1718a18700 1 -- 172.21.15.195:0/4048195261 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1714155eb0 con 0x7f16f8003920 2022-06-27T12:15:25.293 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.241+0000 7f1718a18700 1 -- 172.21.15.195:0/4048195261 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1714156040 con 0x7f16f8003920 2022-06-27T12:15:25.293 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.242+0000 7f17027fc700 1 -- 172.21.15.195:0/4048195261 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f1704013480 con 0x7f16f8003920 2022-06-27T12:15:25.293 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.242+0000 7f17027fc700 1 --2- 172.21.15.195:0/4048195261 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f16f0063d00 0x7f16f00661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:25.294 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.242+0000 7f1711d9b700 1 --2- 172.21.15.195:0/4048195261 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f16f0063d00 0x7f16f00661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:25.294 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.242+0000 7f17027fc700 1 -- 172.21.15.195:0/4048195261 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7578+0+0 (secure 0 0 0) 0x7f1704082d50 con 0x7f16f8003920 2022-06-27T12:15:25.294 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.243+0000 7f1711d9b700 1 --2- 172.21.15.195:0/4048195261 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f16f0063d00 0x7f16f00661b0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f1708000fb0 tx=0x7f1708001e70 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:25.294 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.247+0000 7f17027fc700 1 -- 172.21.15.195:0/4048195261 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f17040512f0 con 0x7f16f8003920 2022-06-27T12:15:25.485 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.474+0000 7f1718a18700 1 -- 172.21.15.195:0/4048195261 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1714148ff0 con 0x7f16f8003920 2022-06-27T12:15:25.486 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:15:25.487 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":102,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:15:20.199910+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6254":{"gid":6254,"name":"b","rank":0,"incarnation":102,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6833/1317767101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1317767101},{"type":"v1","addr":"172.21.15.202:6833","nonce":1317767101}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:15:25.577 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.475+0000 7f17027fc700 1 -- 172.21.15.195:0/4048195261 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+5370 (secure 0 0 0) 0x7f1704054910 con 0x7f16f8003920 2022-06-27T12:15:25.579 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.477+0000 7f1718a18700 1 -- 172.21.15.195:0/4048195261 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f16f0063d00 msgr2=0x7f16f00661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:25.580 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.477+0000 7f1718a18700 1 --2- 172.21.15.195:0/4048195261 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f16f0063d00 0x7f16f00661b0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f1708000fb0 tx=0x7f1708001e70 comp rx=0 tx=0).stop 2022-06-27T12:15:25.580 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.477+0000 7f1718a18700 1 -- 172.21.15.195:0/4048195261 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f16f8003920 msgr2=0x7f17140c9fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:25.581 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.477+0000 7f1718a18700 1 --2- 172.21.15.195:0/4048195261 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f16f8003920 0x7f17140c9fd0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f170400db90 tx=0x7f1704019000 comp rx=0 tx=0).stop 2022-06-27T12:15:25.581 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.477+0000 7f1718a18700 1 -- 172.21.15.195:0/4048195261 shutdown_connections 2022-06-27T12:15:25.582 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.477+0000 7f1718a18700 1 --2- 172.21.15.195:0/4048195261 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f16f0063d00 0x7f16f00661b0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:25.584 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.477+0000 7f1718a18700 1 --2- 172.21.15.195:0/4048195261 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f16f8003920 0x7f17140c9fd0 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:25.584 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.477+0000 7f1718a18700 1 --2- 172.21.15.195:0/4048195261 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f171414ea90 0x7f171414ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:25.584 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.477+0000 7f1718a18700 1 --2- 172.21.15.195:0/4048195261 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f171415a1a0 0x7f1714164db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:25.584 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.477+0000 7f1718a18700 1 -- 172.21.15.195:0/4048195261 >> 172.21.15.195:0/4048195261 conn(0x7f1714063af0 msgr2=0x7f17141537d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:15:25.585 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.477+0000 7f1718a18700 1 -- 172.21.15.195:0/4048195261 shutdown_connections 2022-06-27T12:15:25.585 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:25.477+0000 7f1718a18700 1 -- 172.21.15.195:0/4048195261 wait complete. 2022-06-27T12:15:25.585 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 102 2022-06-27T12:15:25.586 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:15:27.592 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:15:27.757 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.735+0000 7ff3ef873700 1 Processor -- start 2022-06-27T12:15:27.757 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.736+0000 7ff3ef873700 1 -- start start 2022-06-27T12:15:27.758 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.736+0000 7ff3ef873700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7ff3e814e490 0x7ff3e806cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:27.758 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.736+0000 7ff3ef873700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff3e80c3f50 0x7ff3e8157890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:27.758 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.736+0000 7ff3ef873700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3e80c33b0 con 0x7ff3e814e860 2022-06-27T12:15:27.759 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.736+0000 7ff3ef873700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3e806d520 con 0x7ff3e814e490 2022-06-27T12:15:27.759 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.736+0000 7ff3ef873700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e806f220 con 0x7ff3e80c3f50 2022-06-27T12:15:27.759 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.737+0000 7ff3ece0e700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7ff3e814e860 0x7ff3e80c3840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:57862/0 (socket says 172.21.15.195:57862) 2022-06-27T12:15:27.760 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.737+0000 7ff3ece0e700 1 -- 172.21.15.195:0/3365036130 learned_addr learned my addr 172.21.15.195:0/3365036130 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:15:27.760 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.737+0000 7ff3ede10700 1 --2- 172.21.15.195:0/3365036130 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff3e80c3f50 0x7ff3e8157890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:27.761 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.737+0000 7ff3ed60f700 1 --2- 172.21.15.195:0/3365036130 >> v2:172.21.15.202:3300/0 conn(0x7ff3e814e490 0x7ff3e806cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:27.761 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.737+0000 7ff3dbfff700 1 -- 172.21.15.195:0/3365036130 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4272389033 0 0) 0x7ff3e80c33b0 con 0x7ff3e814e860 2022-06-27T12:15:27.762 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.737+0000 7ff3dbfff700 1 -- 172.21.15.195:0/3365036130 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3d0003430 con 0x7ff3e814e860 2022-06-27T12:15:27.762 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.737+0000 7ff3ede10700 1 -- 172.21.15.195:0/3365036130 >> v2:172.21.15.202:3300/0 conn(0x7ff3e814e490 msgr2=0x7ff3e806cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:15:27.763 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.737+0000 7ff3ede10700 1 --2- 172.21.15.195:0/3365036130 >> v2:172.21.15.202:3300/0 conn(0x7ff3e814e490 0x7ff3e806cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:27.763 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.737+0000 7ff3dbfff700 1 -- 172.21.15.195:0/3365036130 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2499341602 0 0) 0x7ff3d0003430 con 0x7ff3e814e860 2022-06-27T12:15:27.764 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.737+0000 7ff3ede10700 1 -- 172.21.15.195:0/3365036130 >> v1:172.21.15.195:6789/0 conn(0x7ff3e814e860 legacy=0x7ff3e80c3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:27.764 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.737+0000 7ff3ede10700 1 -- 172.21.15.195:0/3365036130 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3e81542c0 con 0x7ff3e80c3f50 2022-06-27T12:15:27.765 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.737+0000 7ff3ed60f700 1 --2- 172.21.15.195:0/3365036130 >> v2:172.21.15.202:3300/0 conn(0x7ff3e814e490 0x7ff3e806cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:15:27.765 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.737+0000 7ff3ede10700 1 --2- 172.21.15.195:0/3365036130 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff3e80c3f50 0x7ff3e8157890 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff3e0009e60 tx=0x7ff3e0009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3295290cf3483d2 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:27.779 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.737+0000 7ff3dbfff700 1 -- 172.21.15.195:0/3365036130 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3e000e070 con 0x7ff3e80c3f50 2022-06-27T12:15:27.782 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.737+0000 7ff3dbfff700 1 -- 172.21.15.195:0/3365036130 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3e00038a0 con 0x7ff3e80c3f50 2022-06-27T12:15:27.782 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.737+0000 7ff3dbfff700 1 -- 172.21.15.195:0/3365036130 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3e4001470 con 0x7ff3e80c3f50 2022-06-27T12:15:27.783 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.738+0000 7ff3ef873700 1 -- 172.21.15.195:0/3365036130 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff3e80c3f50 msgr2=0x7ff3e8157890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:27.783 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.738+0000 7ff3ef873700 1 --2- 172.21.15.195:0/3365036130 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff3e80c3f50 0x7ff3e8157890 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff3e0009e60 tx=0x7ff3e0009780 comp rx=0 tx=0).stop 2022-06-27T12:15:27.784 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.738+0000 7ff3ef873700 1 -- 172.21.15.195:0/3365036130 shutdown_connections 2022-06-27T12:15:27.784 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.738+0000 7ff3ef873700 1 --2- 172.21.15.195:0/3365036130 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff3e80c3f50 0x7ff3e8157890 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:27.785 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.738+0000 7ff3ef873700 1 --2- 172.21.15.195:0/3365036130 >> v2:172.21.15.202:3300/0 conn(0x7ff3e814e490 0x7ff3e806cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:27.785 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.738+0000 7ff3ef873700 1 -- 172.21.15.195:0/3365036130 >> 172.21.15.195:0/3365036130 conn(0x7ff3e8063af0 msgr2=0x7ff3e8063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:15:27.785 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.738+0000 7ff3ef873700 1 -- 172.21.15.195:0/3365036130 shutdown_connections 2022-06-27T12:15:27.786 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.738+0000 7ff3ef873700 1 -- 172.21.15.195:0/3365036130 wait complete. 2022-06-27T12:15:27.786 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.739+0000 7ff3ef873700 1 Processor -- start 2022-06-27T12:15:27.787 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.739+0000 7ff3ef873700 1 -- start start 2022-06-27T12:15:27.787 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.740+0000 7ff3ef873700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff3e80c3f50 0x7ff3e8157600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:27.787 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.740+0000 7ff3ef873700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff3e814e490 0x7ff3e8157b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:27.788 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.740+0000 7ff3ef873700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff3e814e860 0x7ff3e80c0f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:27.788 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.740+0000 7ff3ef873700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e8157170 con 0x7ff3e814e860 2022-06-27T12:15:27.789 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.740+0000 7ff3ef873700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e81530f0 con 0x7ff3e80c3f50 2022-06-27T12:15:27.789 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.740+0000 7ff3ef873700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e80beb60 con 0x7ff3e814e490 2022-06-27T12:15:27.790 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.740+0000 7ff3ede10700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff3e814e860 0x7ff3e80c0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:27.790 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.740+0000 7ff3ece0e700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff3e814e490 0x7ff3e8157b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:27.791 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.740+0000 7ff3ede10700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff3e814e860 0x7ff3e80c0f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:51684/0 (socket says 172.21.15.195:51684) 2022-06-27T12:15:27.793 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.740+0000 7ff3ede10700 1 -- 172.21.15.195:0/1396990719 learned_addr learned my addr 172.21.15.195:0/1396990719 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:15:27.793 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.740+0000 7ff3ed60f700 1 --2- 172.21.15.195:0/1396990719 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff3e80c3f50 0x7ff3e8157600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:27.793 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.741+0000 7ff3ede10700 1 -- 172.21.15.195:0/1396990719 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff3e814e490 msgr2=0x7ff3e8157b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:27.794 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.741+0000 7ff3ede10700 1 --2- 172.21.15.195:0/1396990719 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff3e814e490 0x7ff3e8157b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:27.795 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.741+0000 7ff3ede10700 1 -- 172.21.15.195:0/1396990719 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff3e80c3f50 msgr2=0x7ff3e8157600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:27.795 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.741+0000 7ff3ede10700 1 --2- 172.21.15.195:0/1396990719 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff3e80c3f50 0x7ff3e8157600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:27.805 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.741+0000 7ff3ede10700 1 -- 172.21.15.195:0/1396990719 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3dc0092e0 con 0x7ff3e814e860 2022-06-27T12:15:27.806 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.741+0000 7ff3ede10700 1 --2- 172.21.15.195:0/1396990719 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff3e814e860 0x7ff3e80c0f20 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7ff3e806f360 tx=0x7ff3e0018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:27.806 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.741+0000 7ff3d97fa700 1 -- 172.21.15.195:0/1396990719 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3e0004030 con 0x7ff3e814e860 2022-06-27T12:15:27.807 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.741+0000 7ff3d97fa700 1 -- 172.21.15.195:0/1396990719 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3e0004190 con 0x7ff3e814e860 2022-06-27T12:15:27.807 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.741+0000 7ff3ef873700 1 -- 172.21.15.195:0/1396990719 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3e0009a20 con 0x7ff3e814e860 2022-06-27T12:15:27.823 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.741+0000 7ff3d97fa700 1 -- 172.21.15.195:0/1396990719 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3e0004550 con 0x7ff3e814e860 2022-06-27T12:15:27.823 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.741+0000 7ff3ef873700 1 -- 172.21.15.195:0/1396990719 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3e80bf120 con 0x7ff3e814e860 2022-06-27T12:15:27.824 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.742+0000 7ff3ef873700 1 -- 172.21.15.195:0/1396990719 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3e81572e0 con 0x7ff3e814e860 2022-06-27T12:15:27.824 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.742+0000 7ff3d97fa700 1 -- 172.21.15.195:0/1396990719 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7ff3e001e430 con 0x7ff3e814e860 2022-06-27T12:15:27.825 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.743+0000 7ff3d97fa700 1 --2- 172.21.15.195:0/1396990719 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff3c8063d00 0x7ff3c80661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:27.825 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.743+0000 7ff3ed60f700 1 --2- 172.21.15.195:0/1396990719 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff3c8063d00 0x7ff3c80661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:27.828 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.743+0000 7ff3d97fa700 1 -- 172.21.15.195:0/1396990719 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7578+0+0 (secure 0 0 0) 0x7ff3e0082720 con 0x7ff3e814e860 2022-06-27T12:15:27.828 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.743+0000 7ff3ed60f700 1 --2- 172.21.15.195:0/1396990719 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff3c8063d00 0x7ff3c80661b0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7ff3dc009e00 tx=0x7ff3dc009060 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:27.829 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.747+0000 7ff3d97fa700 1 -- 172.21.15.195:0/1396990719 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7ff3e0050cc0 con 0x7ff3e814e860 2022-06-27T12:15:27.978 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.976+0000 7ff3ef873700 1 -- 172.21.15.195:0/1396990719 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff3e80722c0 con 0x7ff3e814e860 2022-06-27T12:15:27.979 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.977+0000 7ff3d97fa700 1 -- 172.21.15.195:0/1396990719 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+5370 (secure 0 0 0) 0x7ff3e00542e0 con 0x7ff3e814e860 2022-06-27T12:15:27.979 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:15:27.980 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":102,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:15:20.199910+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6254":{"gid":6254,"name":"b","rank":0,"incarnation":102,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6833/1317767101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1317767101},{"type":"v1","addr":"172.21.15.202:6833","nonce":1317767101}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:15:27.981 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.979+0000 7ff3ef873700 1 -- 172.21.15.195:0/1396990719 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff3c8063d00 msgr2=0x7ff3c80661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:27.982 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.979+0000 7ff3ef873700 1 --2- 172.21.15.195:0/1396990719 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff3c8063d00 0x7ff3c80661b0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7ff3dc009e00 tx=0x7ff3dc009060 comp rx=0 tx=0).stop 2022-06-27T12:15:27.982 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.979+0000 7ff3ef873700 1 -- 172.21.15.195:0/1396990719 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff3e814e860 msgr2=0x7ff3e80c0f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:27.983 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.979+0000 7ff3ef873700 1 --2- 172.21.15.195:0/1396990719 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff3e814e860 0x7ff3e80c0f20 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7ff3e806f360 tx=0x7ff3e0018000 comp rx=0 tx=0).stop 2022-06-27T12:15:27.983 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.979+0000 7ff3ef873700 1 -- 172.21.15.195:0/1396990719 shutdown_connections 2022-06-27T12:15:27.984 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.979+0000 7ff3ef873700 1 --2- 172.21.15.195:0/1396990719 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff3c8063d00 0x7ff3c80661b0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:28.001 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.979+0000 7ff3ef873700 1 --2- 172.21.15.195:0/1396990719 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff3e814e860 0x7ff3e80c0f20 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:28.007 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.979+0000 7ff3ef873700 1 --2- 172.21.15.195:0/1396990719 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff3e80c3f50 0x7ff3e8157600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:28.008 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.979+0000 7ff3ef873700 1 --2- 172.21.15.195:0/1396990719 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff3e814e490 0x7ff3e8157b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:28.008 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.979+0000 7ff3ef873700 1 -- 172.21.15.195:0/1396990719 >> 172.21.15.195:0/1396990719 conn(0x7ff3e8063af0 msgr2=0x7ff3e80c8880 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:15:28.008 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.979+0000 7ff3ef873700 1 -- 172.21.15.195:0/1396990719 shutdown_connections 2022-06-27T12:15:28.010 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:27.979+0000 7ff3ef873700 1 -- 172.21.15.195:0/1396990719 wait complete. 2022-06-27T12:15:28.011 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 102 2022-06-27T12:15:28.011 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:15:30.018 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:15:30.159 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.157+0000 7f70540a1700 1 Processor -- start 2022-06-27T12:15:30.160 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.157+0000 7f70540a1700 1 -- start start 2022-06-27T12:15:30.160 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.158+0000 7f70540a1700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f704c06eff0 0x7f704c0c61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:30.161 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.158+0000 7f70540a1700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f704c0c31d0 0x7f704c0c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:30.161 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.158+0000 7f70540a1700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f704c14d8a0 con 0x7f704c067ed0 2022-06-27T12:15:30.161 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.158+0000 7f70540a1700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f704c0c47e0 con 0x7f704c0c31d0 2022-06-27T12:15:30.163 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.158+0000 7f70540a1700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f704c0c3e10 con 0x7f704c06eff0 2022-06-27T12:15:30.191 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.158+0000 7f705263e700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f704c0c31d0 0x7f704c0c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:30.192 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.158+0000 7f7051e3d700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f704c06eff0 0x7f704c0c61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:30.192 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.158+0000 7f705263e700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f704c0c31d0 0x7f704c0c42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3300/0 says I am v2:172.21.15.195:49822/0 (socket says 172.21.15.195:49822) 2022-06-27T12:15:30.193 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.158+0000 7f705163c700 1 --1- >> v1:172.21.15.202:6789/0 conn(0x7f704c067ed0 0x7f704c0c2ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.202:6789/0 says I am v1:172.21.15.195:59740/0 (socket says 172.21.15.195:59740) 2022-06-27T12:15:30.193 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.158+0000 7f705263e700 1 -- 172.21.15.195:0/247075068 learned_addr learned my addr 172.21.15.195:0/247075068 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:15:30.196 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.159+0000 7f705263e700 1 -- 172.21.15.195:0/247075068 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f704c06eff0 msgr2=0x7f704c0c61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:15:30.196 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.159+0000 7f705263e700 1 --2- 172.21.15.195:0/247075068 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f704c06eff0 0x7f704c0c61e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:30.197 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.159+0000 7f705263e700 1 -- 172.21.15.195:0/247075068 >> v1:172.21.15.202:6789/0 conn(0x7f704c067ed0 legacy=0x7f704c0c2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:30.197 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.159+0000 7f705263e700 1 -- 172.21.15.195:0/247075068 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f704c06d970 con 0x7f704c0c31d0 2022-06-27T12:15:30.198 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.159+0000 7f705263e700 1 --2- 172.21.15.195:0/247075068 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f704c0c31d0 0x7f704c0c42a0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f7044009f20 tx=0x7f7044009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a9964af56dd0069c server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:30.200 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.159+0000 7f7050e3b700 1 -- 172.21.15.195:0/247075068 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70440092e0 con 0x7f704c0c31d0 2022-06-27T12:15:30.200 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.159+0000 7f7050e3b700 1 -- 172.21.15.195:0/247075068 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f704400ba60 con 0x7f704c0c31d0 2022-06-27T12:15:30.201 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.159+0000 7f7050e3b700 1 -- 172.21.15.195:0/247075068 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f704400be20 con 0x7f704c0c31d0 2022-06-27T12:15:30.201 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.159+0000 7f70540a1700 1 -- 172.21.15.195:0/247075068 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f704c0c31d0 msgr2=0x7f704c0c42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:30.202 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.159+0000 7f70540a1700 1 --2- 172.21.15.195:0/247075068 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f704c0c31d0 0x7f704c0c42a0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f7044009f20 tx=0x7f7044009840 comp rx=0 tx=0).stop 2022-06-27T12:15:30.202 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.160+0000 7f70540a1700 1 -- 172.21.15.195:0/247075068 shutdown_connections 2022-06-27T12:15:30.203 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.160+0000 7f70540a1700 1 --2- 172.21.15.195:0/247075068 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f704c0c31d0 0x7f704c0c42a0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:30.203 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.160+0000 7f70540a1700 1 --2- 172.21.15.195:0/247075068 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f704c06eff0 0x7f704c0c61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:30.204 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.160+0000 7f70540a1700 1 -- 172.21.15.195:0/247075068 >> 172.21.15.195:0/247075068 conn(0x7f704c0c9b00 msgr2=0x7f704c0c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:15:30.204 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.160+0000 7f70540a1700 1 -- 172.21.15.195:0/247075068 shutdown_connections 2022-06-27T12:15:30.205 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.160+0000 7f70540a1700 1 -- 172.21.15.195:0/247075068 wait complete. 2022-06-27T12:15:30.208 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.161+0000 7f70540a1700 1 Processor -- start 2022-06-27T12:15:30.208 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.161+0000 7f70540a1700 1 -- start start 2022-06-27T12:15:30.208 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.161+0000 7f70540a1700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f704c067ed0 0x7f704c165400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:30.209 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.161+0000 7f70540a1700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f704c06eff0 0x7f704c163270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:30.209 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.161+0000 7f70540a1700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f704c157350 0x7f704c1637b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:30.210 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.161+0000 7f70540a1700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f704c166170 con 0x7f704c067ed0 2022-06-27T12:15:30.210 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.161+0000 7f70540a1700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f704c155310 con 0x7f704c157350 2022-06-27T12:15:30.211 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.161+0000 7f70540a1700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f704c165e90 con 0x7f704c06eff0 2022-06-27T12:15:30.211 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.162+0000 7f7051e3d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f704c067ed0 0x7f704c165400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:30.212 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.162+0000 7f7051e3d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f704c067ed0 0x7f704c165400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:51698/0 (socket says 172.21.15.195:51698) 2022-06-27T12:15:30.212 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.162+0000 7f7051e3d700 1 -- 172.21.15.195:0/3879314369 learned_addr learned my addr 172.21.15.195:0/3879314369 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:15:30.213 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.162+0000 7f705263e700 1 --2- 172.21.15.195:0/3879314369 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f704c157350 0x7f704c1637b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:30.213 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.162+0000 7f705163c700 1 --2- 172.21.15.195:0/3879314369 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f704c06eff0 0x7f704c163270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:30.214 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.162+0000 7f7051e3d700 1 -- 172.21.15.195:0/3879314369 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f704c06eff0 msgr2=0x7f704c163270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:30.214 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.162+0000 7f7051e3d700 1 --2- 172.21.15.195:0/3879314369 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f704c06eff0 0x7f704c163270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:30.215 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.162+0000 7f7051e3d700 1 -- 172.21.15.195:0/3879314369 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f704c157350 msgr2=0x7f704c1637b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:30.216 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.162+0000 7f7051e3d700 1 --2- 172.21.15.195:0/3879314369 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f704c157350 0x7f704c1637b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:30.219 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.162+0000 7f7051e3d700 1 -- 172.21.15.195:0/3879314369 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7044009ae0 con 0x7f704c067ed0 2022-06-27T12:15:30.219 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.162+0000 7f705263e700 1 --2- 172.21.15.195:0/3879314369 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f704c157350 0x7f704c1637b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:15:30.220 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.162+0000 7f7051e3d700 1 --2- 172.21.15.195:0/3879314369 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f704c067ed0 0x7f704c165400 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f704800e960 tx=0x7f7048009550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:30.220 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.162+0000 7f7039ffb700 1 -- 172.21.15.195:0/3879314369 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7048013680 con 0x7f704c067ed0 2022-06-27T12:15:30.221 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.162+0000 7f7039ffb700 1 -- 172.21.15.195:0/3879314369 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70480137e0 con 0x7f704c067ed0 2022-06-27T12:15:30.221 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.162+0000 7f70540a1700 1 -- 172.21.15.195:0/3879314369 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f704c1555b0 con 0x7f704c067ed0 2022-06-27T12:15:30.222 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.162+0000 7f7039ffb700 1 -- 172.21.15.195:0/3879314369 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7048013b60 con 0x7f704c067ed0 2022-06-27T12:15:30.222 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.162+0000 7f70540a1700 1 -- 172.21.15.195:0/3879314369 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f704c155b70 con 0x7f704c067ed0 2022-06-27T12:15:30.223 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.163+0000 7f70540a1700 1 -- 172.21.15.195:0/3879314369 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f704c0722c0 con 0x7f704c067ed0 2022-06-27T12:15:30.223 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.164+0000 7f7039ffb700 1 -- 172.21.15.195:0/3879314369 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f7048023070 con 0x7f704c067ed0 2022-06-27T12:15:30.224 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.164+0000 7f7039ffb700 1 --2- 172.21.15.195:0/3879314369 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f702c063d30 0x7f702c0661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:30.224 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.164+0000 7f7039ffb700 1 -- 172.21.15.195:0/3879314369 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7578+0+0 (secure 0 0 0) 0x7f70480811a0 con 0x7f704c067ed0 2022-06-27T12:15:30.225 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.164+0000 7f705163c700 1 --2- 172.21.15.195:0/3879314369 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f702c063d30 0x7f702c0661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:30.225 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.165+0000 7f705163c700 1 --2- 172.21.15.195:0/3879314369 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f702c063d30 0x7f702c0661e0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f704c068650 tx=0x7f703c007d30 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:30.225 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.168+0000 7f7039ffb700 1 -- 172.21.15.195:0/3879314369 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f704804f7a0 con 0x7f704c067ed0 2022-06-27T12:15:30.425 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.425+0000 7f70540a1700 1 -- 172.21.15.195:0/3879314369 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f704c153f20 con 0x7f704c067ed0 2022-06-27T12:15:30.426 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.425+0000 7f7039ffb700 1 -- 172.21.15.195:0/3879314369 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+5374 (secure 0 0 0) 0x7f7048021090 con 0x7f704c067ed0 2022-06-27T12:15:30.426 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:15:30.427 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":103,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:15:30.312602+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6254":{"gid":6254,"name":"b","rank":0,"incarnation":102,"state":"up:reconnect","state_seq":36,"addr":"172.21.15.202:6833/1317767101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1317767101},{"type":"v1","addr":"172.21.15.202:6833","nonce":1317767101}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:15:30.429 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.427+0000 7f70540a1700 1 -- 172.21.15.195:0/3879314369 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f702c063d30 msgr2=0x7f702c0661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:30.430 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.427+0000 7f70540a1700 1 --2- 172.21.15.195:0/3879314369 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f702c063d30 0x7f702c0661e0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f704c068650 tx=0x7f703c007d30 comp rx=0 tx=0).stop 2022-06-27T12:15:30.430 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.427+0000 7f70540a1700 1 -- 172.21.15.195:0/3879314369 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f704c067ed0 msgr2=0x7f704c165400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:30.431 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.427+0000 7f70540a1700 1 --2- 172.21.15.195:0/3879314369 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f704c067ed0 0x7f704c165400 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f704800e960 tx=0x7f7048009550 comp rx=0 tx=0).stop 2022-06-27T12:15:30.431 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.427+0000 7f70540a1700 1 -- 172.21.15.195:0/3879314369 shutdown_connections 2022-06-27T12:15:30.431 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.427+0000 7f70540a1700 1 --2- 172.21.15.195:0/3879314369 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f702c063d30 0x7f702c0661e0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:30.432 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.427+0000 7f70540a1700 1 --2- 172.21.15.195:0/3879314369 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f704c067ed0 0x7f704c165400 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:30.432 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.427+0000 7f70540a1700 1 --2- 172.21.15.195:0/3879314369 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f704c157350 0x7f704c1637b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:30.432 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.427+0000 7f70540a1700 1 --2- 172.21.15.195:0/3879314369 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f704c06eff0 0x7f704c163270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:30.433 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.427+0000 7f70540a1700 1 -- 172.21.15.195:0/3879314369 >> 172.21.15.195:0/3879314369 conn(0x7f704c0c9b00 msgr2=0x7f704c14cdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:15:30.433 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.427+0000 7f70540a1700 1 -- 172.21.15.195:0/3879314369 shutdown_connections 2022-06-27T12:15:30.433 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:30.427+0000 7f70540a1700 1 -- 172.21.15.195:0/3879314369 wait complete. 2022-06-27T12:15:30.434 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 103 2022-06-27T12:15:30.441 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:15:32.443 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:15:32.590 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.582+0000 7fedca326700 1 Processor -- start 2022-06-27T12:15:32.591 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.582+0000 7fedca326700 1 -- start start 2022-06-27T12:15:32.596 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.583+0000 7fedca326700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fedc4150580 0x7fedc4156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:32.596 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.583+0000 7fedca326700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fedc4150950 0x7fedc40c7b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:32.597 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.583+0000 7fedca326700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedc40c76b0 con 0x7fedc406f620 2022-06-27T12:15:32.597 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.583+0000 7fedca326700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedc4157430 con 0x7fedc4150580 2022-06-27T12:15:32.598 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.583+0000 7fedca326700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedc40bad70 con 0x7fedc4150950 2022-06-27T12:15:32.598 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.583+0000 7fedc37fe700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fedc4150950 0x7fedc40c7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:32.599 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.583+0000 7fedc3fff700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fedc4150580 0x7fedc4156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:32.600 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.583+0000 7fedc88c3700 1 --1- >> v1:172.21.15.202:6789/0 conn(0x7fedc406f620 0x7fedc406d6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.202:6789/0 says I am v1:172.21.15.195:59776/0 (socket says 172.21.15.195:59776) 2022-06-27T12:15:32.600 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.583+0000 7fedc37fe700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fedc4150950 0x7fedc40c7b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3301/0 says I am v2:172.21.15.195:44480/0 (socket says 172.21.15.195:44480) 2022-06-27T12:15:32.600 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.583+0000 7fedc88c3700 1 -- 172.21.15.195:0/3999041881 learned_addr learned my addr 172.21.15.195:0/3999041881 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:15:32.601 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.583+0000 7fedc3fff700 1 -- 172.21.15.195:0/3999041881 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fedc4150950 msgr2=0x7fedc40c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:15:32.602 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.583+0000 7fedc3fff700 1 --2- 172.21.15.195:0/3999041881 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fedc4150950 0x7fedc40c7b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:32.602 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.583+0000 7fedc3fff700 1 -- 172.21.15.195:0/3999041881 >> v1:172.21.15.202:6789/0 conn(0x7fedc406f620 legacy=0x7fedc406d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:32.603 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.583+0000 7fedc2ffd700 1 -- 172.21.15.195:0/3999041881 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1007805967 0 0) 0x7fedc40c76b0 con 0x7fedc406f620 2022-06-27T12:15:32.603 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.583+0000 7fedc3fff700 1 -- 172.21.15.195:0/3999041881 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedc40ba4d0 con 0x7fedc4150580 2022-06-27T12:15:32.605 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.584+0000 7fedc3fff700 1 --2- 172.21.15.195:0/3999041881 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fedc4150580 0x7fedc4156ef0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fedb80097a0 tx=0x7fedb8009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=95ce003437a951e2 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:32.605 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.584+0000 7fedc2ffd700 1 -- 172.21.15.195:0/3999041881 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedb8011630 con 0x7fedc4150580 2022-06-27T12:15:32.610 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.584+0000 7fedc2ffd700 1 -- 172.21.15.195:0/3999041881 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedb8011790 con 0x7fedc4150580 2022-06-27T12:15:32.615 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.584+0000 7fedc2ffd700 1 -- 172.21.15.195:0/3999041881 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedb8011b30 con 0x7fedc4150580 2022-06-27T12:15:32.618 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.584+0000 7fedca326700 1 -- 172.21.15.195:0/3999041881 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fedc4150580 msgr2=0x7fedc4156ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:32.618 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.584+0000 7fedca326700 1 --2- 172.21.15.195:0/3999041881 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fedc4150580 0x7fedc4156ef0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fedb80097a0 tx=0x7fedb8009060 comp rx=0 tx=0).stop 2022-06-27T12:15:32.618 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.584+0000 7fedca326700 1 -- 172.21.15.195:0/3999041881 shutdown_connections 2022-06-27T12:15:32.618 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.584+0000 7fedca326700 1 --2- 172.21.15.195:0/3999041881 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fedc4150580 0x7fedc4156ef0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:32.619 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.584+0000 7fedca326700 1 --2- 172.21.15.195:0/3999041881 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fedc4150950 0x7fedc40c7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:32.619 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.584+0000 7fedca326700 1 -- 172.21.15.195:0/3999041881 >> 172.21.15.195:0/3999041881 conn(0x7fedc4063af0 msgr2=0x7fedc4063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:15:32.619 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.584+0000 7fedca326700 1 -- 172.21.15.195:0/3999041881 shutdown_connections 2022-06-27T12:15:32.619 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.584+0000 7fedca326700 1 -- 172.21.15.195:0/3999041881 wait complete. 2022-06-27T12:15:32.620 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.585+0000 7fedca326700 1 Processor -- start 2022-06-27T12:15:32.620 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.586+0000 7fedca326700 1 -- start start 2022-06-27T12:15:32.620 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.586+0000 7fedca326700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fedc406f620 0x7fedc40bc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:32.620 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.586+0000 7fedca326700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fedc4150580 0x7fedc40bca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:32.621 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.586+0000 7fedca326700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fedc4150950 0x7fedc40bd7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:32.621 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.586+0000 7fedca326700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedc40c93e0 con 0x7fedc4150580 2022-06-27T12:15:32.621 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.586+0000 7fedca326700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedc40ca960 con 0x7fedc4150950 2022-06-27T12:15:32.621 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.586+0000 7fedca326700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedc40bc050 con 0x7fedc406f620 2022-06-27T12:15:32.621 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.586+0000 7fedc37fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fedc4150580 0x7fedc40bca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:32.624 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.586+0000 7fedc3fff700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fedc406f620 0x7fedc40bc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:32.624 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.586+0000 7fedc3fff700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fedc406f620 0x7fedc40bc4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3301/0 says I am v2:172.21.15.195:44486/0 (socket says 172.21.15.195:44486) 2022-06-27T12:15:32.625 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.586+0000 7fedc88c3700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fedc4150950 0x7fedc40bd7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:32.625 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.586+0000 7fedc3fff700 1 -- 172.21.15.195:0/4170446408 learned_addr learned my addr 172.21.15.195:0/4170446408 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:15:32.626 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.587+0000 7fedc88c3700 1 -- 172.21.15.195:0/4170446408 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fedc406f620 msgr2=0x7fedc40bc4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:32.626 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.587+0000 7fedc88c3700 1 --2- 172.21.15.195:0/4170446408 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fedc406f620 0x7fedc40bc4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:32.627 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.587+0000 7fedc88c3700 1 -- 172.21.15.195:0/4170446408 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fedc4150580 msgr2=0x7fedc40bca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:32.627 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.587+0000 7fedc88c3700 1 --2- 172.21.15.195:0/4170446408 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fedc4150580 0x7fedc40bca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:32.627 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.587+0000 7fedc88c3700 1 -- 172.21.15.195:0/4170446408 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedb80092e0 con 0x7fedc4150950 2022-06-27T12:15:32.628 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.587+0000 7fedc3fff700 1 --2- 172.21.15.195:0/4170446408 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fedc406f620 0x7fedc40bc4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:15:32.628 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.587+0000 7fedc88c3700 1 --2- 172.21.15.195:0/4170446408 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fedc4150950 0x7fedc40bd7e0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fedbc00c810 tx=0x7fedbc001490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:32.629 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.587+0000 7fedabfff700 1 -- 172.21.15.195:0/4170446408 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedbc004c80 con 0x7fedc4150950 2022-06-27T12:15:32.629 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.587+0000 7fedabfff700 1 -- 172.21.15.195:0/4170446408 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedbc004e00 con 0x7fedc4150950 2022-06-27T12:15:32.630 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.587+0000 7fedca326700 1 -- 172.21.15.195:0/4170446408 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedc40bb3a0 con 0x7fedc4150950 2022-06-27T12:15:32.630 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.587+0000 7fedabfff700 1 -- 172.21.15.195:0/4170446408 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedbc0073a0 con 0x7fedc4150950 2022-06-27T12:15:32.631 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.587+0000 7fedca326700 1 -- 172.21.15.195:0/4170446408 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedc40cabb0 con 0x7fedc4150950 2022-06-27T12:15:32.631 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.588+0000 7fedca326700 1 -- 172.21.15.195:0/4170446408 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedc40bc1e0 con 0x7fedc4150950 2022-06-27T12:15:32.632 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.589+0000 7fedabfff700 1 -- 172.21.15.195:0/4170446408 <== mon.1 v2:172.21.15.202:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fedbc010030 con 0x7fedc4150950 2022-06-27T12:15:32.632 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.589+0000 7fedabfff700 1 --2- 172.21.15.195:0/4170446408 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7feda4063d00 0x7feda40661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:32.632 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.589+0000 7fedabfff700 1 -- 172.21.15.195:0/4170446408 <== mon.1 v2:172.21.15.202:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7578+0+0 (secure 0 0 0) 0x7fedbc080ae0 con 0x7fedc4150950 2022-06-27T12:15:32.633 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.589+0000 7fedc3fff700 1 --2- 172.21.15.195:0/4170446408 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7feda4063d00 0x7feda40661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:32.633 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.589+0000 7fedc3fff700 1 --2- 172.21.15.195:0/4170446408 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7feda4063d00 0x7feda40661b0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fedb8011fd0 tx=0x7fedb8017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:32.634 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.597+0000 7fedabfff700 1 -- 172.21.15.195:0/4170446408 <== mon.1 v2:172.21.15.202:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fedbc04eff0 con 0x7fedc4150950 2022-06-27T12:15:32.827 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.827+0000 7fedca326700 1 -- 172.21.15.195:0/4170446408 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fedc40722c0 con 0x7fedc4150950 2022-06-27T12:15:32.828 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.828+0000 7fedabfff700 1 -- 172.21.15.195:0/4170446408 <== mon.1 v2:172.21.15.202:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+5374 (secure 0 0 0) 0x7fedbc052610 con 0x7fedc4150950 2022-06-27T12:15:32.828 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:15:32.828 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":103,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:15:30.312602+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6254":{"gid":6254,"name":"b","rank":0,"incarnation":102,"state":"up:reconnect","state_seq":36,"addr":"172.21.15.202:6833/1317767101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1317767101},{"type":"v1","addr":"172.21.15.202:6833","nonce":1317767101}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:15:32.832 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.830+0000 7fedca326700 1 -- 172.21.15.195:0/4170446408 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7feda4063d00 msgr2=0x7feda40661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:32.832 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.830+0000 7fedca326700 1 --2- 172.21.15.195:0/4170446408 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7feda4063d00 0x7feda40661b0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fedb8011fd0 tx=0x7fedb8017000 comp rx=0 tx=0).stop 2022-06-27T12:15:32.833 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.830+0000 7fedca326700 1 -- 172.21.15.195:0/4170446408 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fedc4150950 msgr2=0x7fedc40bd7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:32.833 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.830+0000 7fedca326700 1 --2- 172.21.15.195:0/4170446408 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fedc4150950 0x7fedc40bd7e0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fedbc00c810 tx=0x7fedbc001490 comp rx=0 tx=0).stop 2022-06-27T12:15:32.834 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.830+0000 7fedca326700 1 -- 172.21.15.195:0/4170446408 shutdown_connections 2022-06-27T12:15:32.834 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.830+0000 7fedca326700 1 --2- 172.21.15.195:0/4170446408 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7feda4063d00 0x7feda40661b0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:32.835 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.830+0000 7fedca326700 1 --2- 172.21.15.195:0/4170446408 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fedc4150950 0x7fedc40bd7e0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:32.835 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.830+0000 7fedca326700 1 --2- 172.21.15.195:0/4170446408 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fedc406f620 0x7fedc40bc4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:32.836 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.830+0000 7fedca326700 1 --2- 172.21.15.195:0/4170446408 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fedc4150580 0x7fedc40bca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:32.836 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.830+0000 7fedca326700 1 -- 172.21.15.195:0/4170446408 >> 172.21.15.195:0/4170446408 conn(0x7fedc4063af0 msgr2=0x7fedc414e6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:15:32.836 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.830+0000 7fedca326700 1 -- 172.21.15.195:0/4170446408 shutdown_connections 2022-06-27T12:15:32.837 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:32.830+0000 7fedca326700 1 -- 172.21.15.195:0/4170446408 wait complete. 2022-06-27T12:15:32.837 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 103 2022-06-27T12:15:32.852 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 103, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-06-27T11:47:52.358095+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 103, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6254": { "addr": "172.21.15.202:6833/1317767101", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6832", "nonce": 1317767101, "type": "v2" }, { "addr": "172.21.15.202:6833", "nonce": 1317767101, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 6254, "incarnation": 102, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:reconnect", "state_seq": 36 } }, "last_failure": 0, "last_failure_osd_epoch": 64, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-06-27T12:15:30.312602+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6254 } } } ], "standbys": [ { "addr": "172.21.15.202:6837/3305006841", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6836", "nonce": 3305006841, "type": "v2" }, { "addr": "172.21.15.202:6837", "nonce": 3305006841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6835/2060709822", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6834", "nonce": 2060709822, "type": "v2" }, { "addr": "172.21.15.195:6835", "nonce": 2060709822, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6838/1598751652", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6836", "nonce": 1598751652, "type": "v2" }, { "addr": "172.21.15.195:6838", "nonce": 1598751652, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4275, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6839/1313800661", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6837", "nonce": 1313800661, "type": "v2" }, { "addr": "172.21.15.195:6839", "nonce": 1313800661, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4278, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-06-27T12:15:34.198 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:15:34.204 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:15:34.854 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:15:34.977 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:34.977+0000 7f008f6a9700 1 Processor -- start 2022-06-27T12:15:34.978 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:34.977+0000 7f008f6a9700 1 -- start start 2022-06-27T12:15:34.981 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:34.978+0000 7f008f6a9700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0088156100 0x7f00880688a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:34.981 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:34.978+0000 7f008f6a9700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00880c4b10 con 0x7f00881564d0 2022-06-27T12:15:34.982 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:34.978+0000 7f008f6a9700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0088068f20 con 0x7f008806a440 2022-06-27T12:15:34.982 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:34.978+0000 7f008f6a9700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00880c3dc0 con 0x7f0088156100 2022-06-27T12:15:34.983 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:34.978+0000 7f008dc46700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f00881564d0 0x7f00880c28c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:57946/0 (socket says 172.21.15.195:57946) 2022-06-27T12:15:34.983 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:34.978+0000 7f008dc46700 1 -- 172.21.15.195:0/2573007727 learned_addr learned my addr 172.21.15.195:0/2573007727 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:15:34.983 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:34.978+0000 7f008d445700 1 --2- 172.21.15.195:0/2573007727 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0088156100 0x7f00880688a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:34.984 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:34.978+0000 7f0077fff700 1 -- 172.21.15.195:0/2573007727 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1418490297 0 0) 0x7f00880c4b10 con 0x7f00881564d0 2022-06-27T12:15:34.984 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:34.978+0000 7f0077fff700 1 -- 172.21.15.195:0/2573007727 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0070004540 con 0x7f00881564d0 2022-06-27T12:15:35.004 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:34.979+0000 7f0077fff700 1 -- 172.21.15.195:0/2573007727 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1822959109 0 0) 0x7f0088068f20 con 0x7f008806a440 2022-06-27T12:15:35.005 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:34.979+0000 7f0077fff700 1 -- 172.21.15.195:0/2573007727 --> v1:172.21.15.202:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00880c4b10 con 0x7f008806a440 2022-06-27T12:15:35.005 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:34.979+0000 7f008d445700 1 -- 172.21.15.195:0/2573007727 >> v1:172.21.15.202:6789/0 conn(0x7f008806a440 legacy=0x7f00880c4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:35.005 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:34.979+0000 7f0077fff700 1 -- 172.21.15.195:0/2573007727 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1243959049 0 0) 0x7f0070004540 con 0x7f00881564d0 2022-06-27T12:15:35.005 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:34.979+0000 7f008d445700 1 -- 172.21.15.195:0/2573007727 >> v1:172.21.15.195:6789/0 conn(0x7f00881564d0 legacy=0x7f00880c28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:35.006 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:34.979+0000 7f008d445700 1 -- 172.21.15.195:0/2573007727 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00880c56b0 con 0x7f0088156100 2022-06-27T12:15:35.006 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:34.979+0000 7f008d445700 1 --2- 172.21.15.195:0/2573007727 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0088156100 0x7f00880688a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0084009750 tx=0x7f0084009a60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab38532a50d944f5 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:35.006 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:34.979+0000 7f0077fff700 1 -- 172.21.15.195:0/2573007727 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00800017f0 con 0x7f0088156100 2022-06-27T12:15:35.006 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:34.979+0000 7f0077fff700 1 -- 172.21.15.195:0/2573007727 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0084004030 con 0x7f0088156100 2022-06-27T12:15:35.007 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:34.979+0000 7f0077fff700 1 -- 172.21.15.195:0/2573007727 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00840043b0 con 0x7f0088156100 2022-06-27T12:15:35.007 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:34.979+0000 7f008f6a9700 1 -- 172.21.15.195:0/2573007727 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0088156100 msgr2=0x7f00880688a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:35.007 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:34.979+0000 7f008f6a9700 1 --2- 172.21.15.195:0/2573007727 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0088156100 0x7f00880688a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0084009750 tx=0x7f0084009a60 comp rx=0 tx=0).stop 2022-06-27T12:15:35.007 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:34.980+0000 7f008f6a9700 1 -- 172.21.15.195:0/2573007727 shutdown_connections 2022-06-27T12:15:35.008 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:34.980+0000 7f008f6a9700 1 --2- 172.21.15.195:0/2573007727 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0088156100 0x7f00880688a0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:35.008 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:34.980+0000 7f008f6a9700 1 -- 172.21.15.195:0/2573007727 >> 172.21.15.195:0/2573007727 conn(0x7f00880c9b00 msgr2=0x7f00880c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:15:35.008 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:34.980+0000 7f008f6a9700 1 -- 172.21.15.195:0/2573007727 shutdown_connections 2022-06-27T12:15:35.008 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:34.980+0000 7f008f6a9700 1 -- 172.21.15.195:0/2573007727 wait complete. 2022-06-27T12:15:35.008 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:34.981+0000 7f008f6a9700 1 Processor -- start 2022-06-27T12:15:35.009 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:34.981+0000 7f008f6a9700 1 -- start start 2022-06-27T12:15:35.009 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:34.982+0000 7f008f6a9700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f008806a440 0x7f00880c7020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:35.009 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:34.982+0000 7f008f6a9700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0088156100 0x7f0088166650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:35.009 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:34.982+0000 7f008f6a9700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f00881564d0 0x7f0088166b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:35.009 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:34.982+0000 7f008f6a9700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00880c8b30 con 0x7f0088156100 2022-06-27T12:15:35.014 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:34.982+0000 7f008f6a9700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0088159d80 con 0x7f00881564d0 2022-06-27T12:15:35.014 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:34.982+0000 7f008f6a9700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00880c7de0 con 0x7f008806a440 2022-06-27T12:15:35.015 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:34.982+0000 7f008cc44700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0088156100 0x7f0088166650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:35.015 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:34.982+0000 7f008cc44700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0088156100 0x7f0088166650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:51762/0 (socket says 172.21.15.195:51762) 2022-06-27T12:15:35.015 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:34.982+0000 7f008cc44700 1 -- 172.21.15.195:0/126560774 learned_addr learned my addr 172.21.15.195:0/126560774 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:15:35.015 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:34.982+0000 7f008d445700 1 --2- 172.21.15.195:0/126560774 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f008806a440 0x7f00880c7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:35.016 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:34.982+0000 7f008dc46700 1 --2- 172.21.15.195:0/126560774 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f00881564d0 0x7f0088166b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:35.016 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:34.982+0000 7f008cc44700 1 -- 172.21.15.195:0/126560774 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f008806a440 msgr2=0x7f00880c7020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:35.016 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:34.982+0000 7f008cc44700 1 --2- 172.21.15.195:0/126560774 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f008806a440 0x7f00880c7020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:35.016 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:34.982+0000 7f008cc44700 1 -- 172.21.15.195:0/126560774 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f00881564d0 msgr2=0x7f0088166b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:35.016 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:34.982+0000 7f008cc44700 1 --2- 172.21.15.195:0/126560774 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f00881564d0 0x7f0088166b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:35.017 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:34.982+0000 7f008cc44700 1 -- 172.21.15.195:0/126560774 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0084009290 con 0x7f0088156100 2022-06-27T12:15:35.017 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:34.982+0000 7f008cc44700 1 --2- 172.21.15.195:0/126560774 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0088156100 0x7f0088166650 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f007800c440 tx=0x7f0078007c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:35.017 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:34.982+0000 7f00757fa700 1 -- 172.21.15.195:0/126560774 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00780050d0 con 0x7f0088156100 2022-06-27T12:15:35.017 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:34.982+0000 7f00757fa700 1 -- 172.21.15.195:0/126560774 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0078005250 con 0x7f0088156100 2022-06-27T12:15:35.018 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:34.982+0000 7f008f6a9700 1 -- 172.21.15.195:0/126560774 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00880c84e0 con 0x7f0088156100 2022-06-27T12:15:35.018 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:34.982+0000 7f00757fa700 1 -- 172.21.15.195:0/126560774 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0078003750 con 0x7f0088156100 2022-06-27T12:15:35.019 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:34.982+0000 7f008f6a9700 1 -- 172.21.15.195:0/126560774 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00881591d0 con 0x7f0088156100 2022-06-27T12:15:35.019 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:34.983+0000 7f008f6a9700 1 -- 172.21.15.195:0/126560774 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00880c8c70 con 0x7f0088156100 2022-06-27T12:15:35.019 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:34.984+0000 7f00757fa700 1 -- 172.21.15.195:0/126560774 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f0078019030 con 0x7f0088156100 2022-06-27T12:15:35.020 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:34.984+0000 7f00757fa700 1 --2- 172.21.15.195:0/126560774 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0068063d00 0x7f00680661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:35.020 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:34.984+0000 7f00757fa700 1 -- 172.21.15.195:0/126560774 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7578+0+0 (secure 0 0 0) 0x7f007807fdf0 con 0x7f0088156100 2022-06-27T12:15:35.020 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:34.984+0000 7f008d445700 1 --2- 172.21.15.195:0/126560774 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0068063d00 0x7f00680661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:35.020 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:34.985+0000 7f008d445700 1 --2- 172.21.15.195:0/126560774 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0068063d00 0x7f00680661b0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f0084009aa0 tx=0x7f0084015000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:35.020 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:34.988+0000 7f00757fa700 1 -- 172.21.15.195:0/126560774 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f00880c8c70 con 0x7f0088156100 2022-06-27T12:15:35.216 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:35.214+0000 7f008f6a9700 1 -- 172.21.15.195:0/126560774 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f00880c8c70 con 0x7f0088156100 2022-06-27T12:15:35.216 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:35.214+0000 7f00757fa700 1 -- 172.21.15.195:0/126560774 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+5374 (secure 0 0 0) 0x7f00880c8c70 con 0x7f0088156100 2022-06-27T12:15:35.217 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:15:35.217 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":103,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:15:30.312602+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6254":{"gid":6254,"name":"b","rank":0,"incarnation":102,"state":"up:reconnect","state_seq":36,"addr":"172.21.15.202:6833/1317767101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1317767101},{"type":"v1","addr":"172.21.15.202:6833","nonce":1317767101}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:15:35.245 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:35.216+0000 7f008f6a9700 1 -- 172.21.15.195:0/126560774 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0068063d00 msgr2=0x7f00680661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:35.258 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:35.216+0000 7f008f6a9700 1 --2- 172.21.15.195:0/126560774 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0068063d00 0x7f00680661b0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f0084009aa0 tx=0x7f0084015000 comp rx=0 tx=0).stop 2022-06-27T12:15:35.258 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:35.216+0000 7f008f6a9700 1 -- 172.21.15.195:0/126560774 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0088156100 msgr2=0x7f0088166650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:35.259 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:35.216+0000 7f008f6a9700 1 --2- 172.21.15.195:0/126560774 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0088156100 0x7f0088166650 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f007800c440 tx=0x7f0078007c50 comp rx=0 tx=0).stop 2022-06-27T12:15:35.259 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:35.216+0000 7f008f6a9700 1 -- 172.21.15.195:0/126560774 shutdown_connections 2022-06-27T12:15:35.259 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:35.216+0000 7f008f6a9700 1 --2- 172.21.15.195:0/126560774 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0068063d00 0x7f00680661b0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:35.259 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:35.216+0000 7f008f6a9700 1 --2- 172.21.15.195:0/126560774 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f00881564d0 0x7f0088166b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:35.260 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:35.216+0000 7f008f6a9700 1 --2- 172.21.15.195:0/126560774 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f008806a440 0x7f00880c7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:35.260 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:35.216+0000 7f008f6a9700 1 --2- 172.21.15.195:0/126560774 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0088156100 0x7f0088166650 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:35.260 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:35.216+0000 7f008f6a9700 1 -- 172.21.15.195:0/126560774 >> 172.21.15.195:0/126560774 conn(0x7f00880c9b00 msgr2=0x7f00880c0980 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:15:35.261 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:35.216+0000 7f008f6a9700 1 -- 172.21.15.195:0/126560774 shutdown_connections 2022-06-27T12:15:35.261 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:35.216+0000 7f008f6a9700 1 -- 172.21.15.195:0/126560774 wait complete. 2022-06-27T12:15:35.261 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 103 2022-06-27T12:15:35.271 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:15:37.277 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:15:37.435 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.434+0000 7fce26587700 1 Processor -- start 2022-06-27T12:15:37.435 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.434+0000 7fce26587700 1 -- start start 2022-06-27T12:15:37.450 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.435+0000 7fce26587700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fce2006a450 0x7fce20063af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:37.451 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.435+0000 7fce26587700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fce201508d0 0x7fce200bbf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:37.451 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.435+0000 7fce26587700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce20064030 con 0x7fce20150500 2022-06-27T12:15:37.452 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.435+0000 7fce26587700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce200c4160 con 0x7fce201508d0 2022-06-27T12:15:37.452 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.435+0000 7fce26587700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce200c4440 con 0x7fce2006a450 2022-06-27T12:15:37.452 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.435+0000 7fce24b24700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fce201508d0 0x7fce200bbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:37.452 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.436+0000 7fce24b24700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fce201508d0 0x7fce200bbf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:51774/0 (socket says 172.21.15.195:51774) 2022-06-27T12:15:37.453 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.436+0000 7fce24b24700 1 -- 172.21.15.195:0/2494007559 learned_addr learned my addr 172.21.15.195:0/2494007559 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:15:37.453 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.436+0000 7fce1f7fe700 1 --2- 172.21.15.195:0/2494007559 >> v2:172.21.15.202:3300/0 conn(0x7fce2006a450 0x7fce20063af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:37.453 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.436+0000 7fce24b24700 1 -- 172.21.15.195:0/2494007559 >> v2:172.21.15.202:3300/0 conn(0x7fce2006a450 msgr2=0x7fce20063af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:15:37.453 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.436+0000 7fce24b24700 1 --2- 172.21.15.195:0/2494007559 >> v2:172.21.15.202:3300/0 conn(0x7fce2006a450 0x7fce20063af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:37.454 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.436+0000 7fce24b24700 1 -- 172.21.15.195:0/2494007559 >> v1:172.21.15.202:6789/0 conn(0x7fce20150500 legacy=0x7fce200c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:37.454 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.436+0000 7fce24b24700 1 -- 172.21.15.195:0/2494007559 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce2015a800 con 0x7fce201508d0 2022-06-27T12:15:37.454 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.436+0000 7fce24b24700 1 --2- 172.21.15.195:0/2494007559 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fce201508d0 0x7fce200bbf50 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7fce2006ce70 tx=0x7fce18009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1bea1138d8ae15c server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:37.454 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.436+0000 7fce1effd700 1 -- 172.21.15.195:0/2494007559 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce1800e070 con 0x7fce201508d0 2022-06-27T12:15:37.454 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.436+0000 7fce1effd700 1 -- 172.21.15.195:0/2494007559 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce1800ba20 con 0x7fce201508d0 2022-06-27T12:15:37.455 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.436+0000 7fce1effd700 1 -- 172.21.15.195:0/2494007559 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce180092e0 con 0x7fce201508d0 2022-06-27T12:15:37.455 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.437+0000 7fce26587700 1 -- 172.21.15.195:0/2494007559 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fce201508d0 msgr2=0x7fce200bbf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:37.455 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.437+0000 7fce26587700 1 --2- 172.21.15.195:0/2494007559 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fce201508d0 0x7fce200bbf50 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7fce2006ce70 tx=0x7fce18009840 comp rx=0 tx=0).stop 2022-06-27T12:15:37.455 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.437+0000 7fce26587700 1 -- 172.21.15.195:0/2494007559 shutdown_connections 2022-06-27T12:15:37.456 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.437+0000 7fce26587700 1 --2- 172.21.15.195:0/2494007559 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fce201508d0 0x7fce200bbf50 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:37.456 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.437+0000 7fce26587700 1 --2- 172.21.15.195:0/2494007559 >> v2:172.21.15.202:3300/0 conn(0x7fce2006a450 0x7fce20063af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:37.456 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.437+0000 7fce26587700 1 -- 172.21.15.195:0/2494007559 >> 172.21.15.195:0/2494007559 conn(0x7fce200c9b00 msgr2=0x7fce200c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:15:37.457 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.437+0000 7fce26587700 1 -- 172.21.15.195:0/2494007559 shutdown_connections 2022-06-27T12:15:37.457 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.437+0000 7fce26587700 1 -- 172.21.15.195:0/2494007559 wait complete. 2022-06-27T12:15:37.457 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.438+0000 7fce26587700 1 Processor -- start 2022-06-27T12:15:37.458 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.438+0000 7fce26587700 1 -- start start 2022-06-27T12:15:37.458 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.439+0000 7fce26587700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fce2006a450 0x7fce200c0350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:37.458 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.439+0000 7fce26587700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fce20150500 0x7fce200680b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:37.458 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.439+0000 7fce26587700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fce201508d0 0x7fce201663a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:37.458 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.439+0000 7fce26587700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce200c0890 con 0x7fce2006a450 2022-06-27T12:15:37.459 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.439+0000 7fce26587700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce200c2380 con 0x7fce201508d0 2022-06-27T12:15:37.459 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.439+0000 7fce26587700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce201599e0 con 0x7fce20150500 2022-06-27T12:15:37.459 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.439+0000 7fce1ffff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fce2006a450 0x7fce200c0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:37.459 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.439+0000 7fce1ffff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fce2006a450 0x7fce200c0350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:51776/0 (socket says 172.21.15.195:51776) 2022-06-27T12:15:37.460 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.439+0000 7fce1ffff700 1 -- 172.21.15.195:0/1950403437 learned_addr learned my addr 172.21.15.195:0/1950403437 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:15:37.460 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.439+0000 7fce1f7fe700 1 --2- 172.21.15.195:0/1950403437 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fce20150500 0x7fce200680b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:37.460 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.439+0000 7fce24b24700 1 --2- 172.21.15.195:0/1950403437 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fce201508d0 0x7fce201663a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:37.460 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.439+0000 7fce1ffff700 1 -- 172.21.15.195:0/1950403437 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fce20150500 msgr2=0x7fce200680b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:37.460 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.439+0000 7fce1ffff700 1 --2- 172.21.15.195:0/1950403437 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fce20150500 0x7fce200680b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:37.461 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.439+0000 7fce1ffff700 1 -- 172.21.15.195:0/1950403437 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fce201508d0 msgr2=0x7fce201663a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:37.461 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.439+0000 7fce1ffff700 1 --2- 172.21.15.195:0/1950403437 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fce201508d0 0x7fce201663a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:37.461 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.439+0000 7fce1ffff700 1 -- 172.21.15.195:0/1950403437 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce18009ae0 con 0x7fce2006a450 2022-06-27T12:15:37.461 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.439+0000 7fce1ffff700 1 --2- 172.21.15.195:0/1950403437 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fce2006a450 0x7fce200c0350 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7fce1000c4b0 tx=0x7fce10007e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:37.462 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.440+0000 7fce17fff700 1 -- 172.21.15.195:0/1950403437 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce1000ce60 con 0x7fce2006a450 2022-06-27T12:15:37.462 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.440+0000 7fce17fff700 1 -- 172.21.15.195:0/1950403437 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce100050d0 con 0x7fce2006a450 2022-06-27T12:15:37.462 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.440+0000 7fce17fff700 1 -- 172.21.15.195:0/1950403437 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce10011590 con 0x7fce2006a450 2022-06-27T12:15:37.462 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.440+0000 7fce26587700 1 -- 172.21.15.195:0/1950403437 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce200bd160 con 0x7fce2006a450 2022-06-27T12:15:37.463 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.440+0000 7fce26587700 1 -- 172.21.15.195:0/1950403437 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce200c1d30 con 0x7fce2006a450 2022-06-27T12:15:37.463 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.440+0000 7fce26587700 1 -- 172.21.15.195:0/1950403437 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce200722c0 con 0x7fce2006a450 2022-06-27T12:15:37.463 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.441+0000 7fce17fff700 1 -- 172.21.15.195:0/1950403437 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fce10019030 con 0x7fce2006a450 2022-06-27T12:15:37.464 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.441+0000 7fce17fff700 1 --2- 172.21.15.195:0/1950403437 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fce04063d30 0x7fce040661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:37.464 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.442+0000 7fce1f7fe700 1 --2- 172.21.15.195:0/1950403437 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fce04063d30 0x7fce040661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:37.465 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.442+0000 7fce17fff700 1 -- 172.21.15.195:0/1950403437 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7578+0+0 (secure 0 0 0) 0x7fce1007fd40 con 0x7fce2006a450 2022-06-27T12:15:37.465 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.442+0000 7fce1f7fe700 1 --2- 172.21.15.195:0/1950403437 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fce04063d30 0x7fce040661e0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fce200c00b0 tx=0x7fce08009060 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:37.466 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.446+0000 7fce17fff700 1 -- 172.21.15.195:0/1950403437 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fce1004e340 con 0x7fce2006a450 2022-06-27T12:15:37.677 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.673+0000 7fce26587700 1 -- 172.21.15.195:0/1950403437 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fce20064030 con 0x7fce2006a450 2022-06-27T12:15:37.677 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.673+0000 7fce17fff700 1 -- 172.21.15.195:0/1950403437 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+5371 (secure 0 0 0) 0x7fce20064030 con 0x7fce2006a450 2022-06-27T12:15:37.679 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:15:37.679 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":104,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:15:36.333922+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6254":{"gid":6254,"name":"b","rank":0,"incarnation":102,"state":"up:rejoin","state_seq":38,"addr":"172.21.15.202:6833/1317767101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1317767101},{"type":"v1","addr":"172.21.15.202:6833","nonce":1317767101}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:15:37.682 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.675+0000 7fce26587700 1 -- 172.21.15.195:0/1950403437 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fce04063d30 msgr2=0x7fce040661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:37.682 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.675+0000 7fce26587700 1 --2- 172.21.15.195:0/1950403437 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fce04063d30 0x7fce040661e0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fce200c00b0 tx=0x7fce08009060 comp rx=0 tx=0).stop 2022-06-27T12:15:37.683 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.675+0000 7fce26587700 1 -- 172.21.15.195:0/1950403437 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fce2006a450 msgr2=0x7fce200c0350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:37.683 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.675+0000 7fce26587700 1 --2- 172.21.15.195:0/1950403437 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fce2006a450 0x7fce200c0350 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7fce1000c4b0 tx=0x7fce10007e30 comp rx=0 tx=0).stop 2022-06-27T12:15:37.684 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.675+0000 7fce26587700 1 -- 172.21.15.195:0/1950403437 shutdown_connections 2022-06-27T12:15:37.686 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.675+0000 7fce26587700 1 --2- 172.21.15.195:0/1950403437 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fce04063d30 0x7fce040661e0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:37.687 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.675+0000 7fce26587700 1 --2- 172.21.15.195:0/1950403437 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fce2006a450 0x7fce200c0350 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:37.688 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.675+0000 7fce26587700 1 --2- 172.21.15.195:0/1950403437 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fce201508d0 0x7fce201663a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:37.689 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.675+0000 7fce26587700 1 --2- 172.21.15.195:0/1950403437 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fce20150500 0x7fce200680b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:37.689 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.675+0000 7fce26587700 1 -- 172.21.15.195:0/1950403437 >> 172.21.15.195:0/1950403437 conn(0x7fce200c9b00 msgr2=0x7fce200c48d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:15:37.689 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.675+0000 7fce26587700 1 -- 172.21.15.195:0/1950403437 shutdown_connections 2022-06-27T12:15:37.690 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:37.675+0000 7fce26587700 1 -- 172.21.15.195:0/1950403437 wait complete. 2022-06-27T12:15:37.690 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 104 2022-06-27T12:15:37.693 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:15:39.705 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:15:39.892 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:39.836+0000 7fcb806e3700 1 Processor -- start 2022-06-27T12:15:39.893 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:39.836+0000 7fcb806e3700 1 -- start start 2022-06-27T12:15:39.894 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:39.836+0000 7fcb806e3700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcb7814e860 0x7fcb780c3840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:39.894 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:39.836+0000 7fcb806e3700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fcb780c3d80 0x7fcb78157890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:39.895 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:39.836+0000 7fcb806e3700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb780c33b0 con 0x7fcb7814e490 2022-06-27T12:15:39.895 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:39.836+0000 7fcb806e3700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb780c4200 con 0x7fcb780c3d80 2022-06-27T12:15:39.896 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:39.837+0000 7fcb806e3700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb7806f220 con 0x7fcb7814e860 2022-06-27T12:15:39.896 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:39.837+0000 7fcb7ec80700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fcb780c3d80 0x7fcb78157890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:39.896 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:39.837+0000 7fcb7e47f700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fcb7814e490 0x7fcb7806cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:57976/0 (socket says 172.21.15.195:57976) 2022-06-27T12:15:39.897 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:39.837+0000 7fcb7e47f700 1 -- 172.21.15.195:0/1394159033 learned_addr learned my addr 172.21.15.195:0/1394159033 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:15:39.897 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:39.837+0000 7fcb7ec80700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fcb780c3d80 0x7fcb78157890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:51792/0 (socket says 172.21.15.195:51792) 2022-06-27T12:15:39.910 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:39.837+0000 7fcb7dc7e700 1 --2- 172.21.15.195:0/1394159033 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcb7814e860 0x7fcb780c3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:39.911 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:39.837+0000 7fcb7ec80700 1 -- 172.21.15.195:0/1394159033 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcb7814e860 msgr2=0x7fcb780c3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:15:39.911 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:39.837+0000 7fcb7ec80700 1 --2- 172.21.15.195:0/1394159033 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcb7814e860 0x7fcb780c3840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:39.914 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:39.837+0000 7fcb7ec80700 1 -- 172.21.15.195:0/1394159033 >> v1:172.21.15.195:6789/0 conn(0x7fcb7814e490 legacy=0x7fcb7806cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:39.915 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:39.837+0000 7fcb7ec80700 1 -- 172.21.15.195:0/1394159033 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb781542c0 con 0x7fcb780c3d80 2022-06-27T12:15:39.915 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:39.837+0000 7fcb7d47d700 1 -- 172.21.15.195:0/1394159033 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 668702517 0 0) 0x7fcb780c33b0 con 0x7fcb7814e490 2022-06-27T12:15:39.916 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:39.837+0000 7fcb7ec80700 1 --2- 172.21.15.195:0/1394159033 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcb780c3d80 0x7fcb78157890 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7fcb70009e60 tx=0x7fcb70009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8e2f65195bf80 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:39.916 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:39.837+0000 7fcb7d47d700 1 -- 172.21.15.195:0/1394159033 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb7000e070 con 0x7fcb780c3d80 2022-06-27T12:15:39.917 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:39.837+0000 7fcb7d47d700 1 -- 172.21.15.195:0/1394159033 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb7000bb60 con 0x7fcb780c3d80 2022-06-27T12:15:39.917 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:39.837+0000 7fcb7d47d700 1 -- 172.21.15.195:0/1394159033 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb70012430 con 0x7fcb780c3d80 2022-06-27T12:15:39.917 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:39.838+0000 7fcb806e3700 1 -- 172.21.15.195:0/1394159033 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcb780c3d80 msgr2=0x7fcb78157890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:39.924 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:39.838+0000 7fcb806e3700 1 --2- 172.21.15.195:0/1394159033 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcb780c3d80 0x7fcb78157890 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7fcb70009e60 tx=0x7fcb70009780 comp rx=0 tx=0).stop 2022-06-27T12:15:39.925 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:39.838+0000 7fcb806e3700 1 -- 172.21.15.195:0/1394159033 shutdown_connections 2022-06-27T12:15:39.925 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:39.838+0000 7fcb806e3700 1 --2- 172.21.15.195:0/1394159033 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcb780c3d80 0x7fcb78157890 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:39.926 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:39.838+0000 7fcb806e3700 1 --2- 172.21.15.195:0/1394159033 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcb7814e860 0x7fcb780c3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:39.926 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:39.838+0000 7fcb806e3700 1 -- 172.21.15.195:0/1394159033 >> 172.21.15.195:0/1394159033 conn(0x7fcb78063af0 msgr2=0x7fcb78063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:15:39.927 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:39.838+0000 7fcb806e3700 1 -- 172.21.15.195:0/1394159033 shutdown_connections 2022-06-27T12:15:39.927 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:39.838+0000 7fcb806e3700 1 -- 172.21.15.195:0/1394159033 wait complete. 2022-06-27T12:15:39.928 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:39.839+0000 7fcb806e3700 1 Processor -- start 2022-06-27T12:15:39.938 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:39.839+0000 7fcb806e3700 1 -- start start 2022-06-27T12:15:39.939 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:39.840+0000 7fcb806e3700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcb780c3d80 0x7fcb78152830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:39.939 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:39.840+0000 7fcb806e3700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcb7814e490 0x7fcb78152d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:39.940 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:39.840+0000 7fcb806e3700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcb7814e860 0x7fcb78153600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:39.940 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:39.840+0000 7fcb806e3700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb7806aef0 con 0x7fcb7814e490 2022-06-27T12:15:39.942 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:39.840+0000 7fcb806e3700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb7806a000 con 0x7fcb7814e860 2022-06-27T12:15:39.942 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:39.840+0000 7fcb806e3700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb7806a2e0 con 0x7fcb780c3d80 2022-06-27T12:15:39.943 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:39.840+0000 7fcb7dc7e700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcb7814e490 0x7fcb78152d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:39.943 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:39.840+0000 7fcb7dc7e700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcb7814e490 0x7fcb78152d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:51796/0 (socket says 172.21.15.195:51796) 2022-06-27T12:15:39.943 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:39.840+0000 7fcb7dc7e700 1 -- 172.21.15.195:0/1642884570 learned_addr learned my addr 172.21.15.195:0/1642884570 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:15:39.944 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:39.840+0000 7fcb7e47f700 1 --2- 172.21.15.195:0/1642884570 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcb780c3d80 0x7fcb78152830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:39.944 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:39.840+0000 7fcb7ec80700 1 --2- 172.21.15.195:0/1642884570 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcb7814e860 0x7fcb78153600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:39.945 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:39.840+0000 7fcb7dc7e700 1 -- 172.21.15.195:0/1642884570 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcb780c3d80 msgr2=0x7fcb78152830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:39.945 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:39.840+0000 7fcb7dc7e700 1 --2- 172.21.15.195:0/1642884570 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcb780c3d80 0x7fcb78152830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:39.946 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:39.840+0000 7fcb7dc7e700 1 -- 172.21.15.195:0/1642884570 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcb7814e860 msgr2=0x7fcb78153600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:39.946 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:39.840+0000 7fcb7dc7e700 1 --2- 172.21.15.195:0/1642884570 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcb7814e860 0x7fcb78153600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:39.947 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:39.840+0000 7fcb7dc7e700 1 -- 172.21.15.195:0/1642884570 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb70009a20 con 0x7fcb7814e490 2022-06-27T12:15:39.947 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:39.840+0000 7fcb7dc7e700 1 --2- 172.21.15.195:0/1642884570 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcb7814e490 0x7fcb78152d70 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7fcb7400e8d0 tx=0x7fcb74009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:39.948 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:39.840+0000 7fcb6a7fc700 1 -- 172.21.15.195:0/1642884570 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb74013680 con 0x7fcb7814e490 2022-06-27T12:15:39.948 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:39.840+0000 7fcb6a7fc700 1 -- 172.21.15.195:0/1642884570 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb740137e0 con 0x7fcb7814e490 2022-06-27T12:15:39.948 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:39.840+0000 7fcb6a7fc700 1 -- 172.21.15.195:0/1642884570 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb74013b80 con 0x7fcb7814e490 2022-06-27T12:15:39.949 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:39.840+0000 7fcb806e3700 1 -- 172.21.15.195:0/1642884570 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb7806a5c0 con 0x7fcb7814e490 2022-06-27T12:15:39.949 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:39.841+0000 7fcb806e3700 1 -- 172.21.15.195:0/1642884570 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb7806ab80 con 0x7fcb7814e490 2022-06-27T12:15:39.950 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:39.841+0000 7fcb806e3700 1 -- 172.21.15.195:0/1642884570 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb780722c0 con 0x7fcb7814e490 2022-06-27T12:15:39.950 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:39.842+0000 7fcb6a7fc700 1 -- 172.21.15.195:0/1642884570 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fcb74019030 con 0x7fcb7814e490 2022-06-27T12:15:39.950 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:39.842+0000 7fcb6a7fc700 1 --2- 172.21.15.195:0/1642884570 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcb58063d30 0x7fcb580661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:39.950 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:39.842+0000 7fcb6a7fc700 1 -- 172.21.15.195:0/1642884570 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7578+0+0 (secure 0 0 0) 0x7fcb74080c80 con 0x7fcb7814e490 2022-06-27T12:15:39.951 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:39.842+0000 7fcb7e47f700 1 --2- 172.21.15.195:0/1642884570 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcb58063d30 0x7fcb580661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:39.951 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:39.843+0000 7fcb7e47f700 1 --2- 172.21.15.195:0/1642884570 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcb58063d30 0x7fcb580661e0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fcb6c004810 tx=0x7fcb6c007bf0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:39.951 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:39.846+0000 7fcb6a7fc700 1 -- 172.21.15.195:0/1642884570 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fcb7404f280 con 0x7fcb7814e490 2022-06-27T12:15:40.073 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:40.073+0000 7fcb806e3700 1 -- 172.21.15.195:0/1642884570 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcb780c60d0 con 0x7fcb7814e490 2022-06-27T12:15:40.073 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:40.073+0000 7fcb6a7fc700 1 -- 172.21.15.195:0/1642884570 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+5371 (secure 0 0 0) 0x7fcb74017090 con 0x7fcb7814e490 2022-06-27T12:15:40.074 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:15:40.074 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":104,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:15:36.333922+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6254":{"gid":6254,"name":"b","rank":0,"incarnation":102,"state":"up:rejoin","state_seq":38,"addr":"172.21.15.202:6833/1317767101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1317767101},{"type":"v1","addr":"172.21.15.202:6833","nonce":1317767101}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:15:40.075 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:40.075+0000 7fcb806e3700 1 -- 172.21.15.195:0/1642884570 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcb58063d30 msgr2=0x7fcb580661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:40.076 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:40.075+0000 7fcb806e3700 1 --2- 172.21.15.195:0/1642884570 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcb58063d30 0x7fcb580661e0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fcb6c004810 tx=0x7fcb6c007bf0 comp rx=0 tx=0).stop 2022-06-27T12:15:40.076 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:40.075+0000 7fcb806e3700 1 -- 172.21.15.195:0/1642884570 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcb7814e490 msgr2=0x7fcb78152d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:40.077 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:40.075+0000 7fcb806e3700 1 --2- 172.21.15.195:0/1642884570 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcb7814e490 0x7fcb78152d70 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7fcb7400e8d0 tx=0x7fcb74009060 comp rx=0 tx=0).stop 2022-06-27T12:15:40.077 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:40.075+0000 7fcb806e3700 1 -- 172.21.15.195:0/1642884570 shutdown_connections 2022-06-27T12:15:40.077 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:40.075+0000 7fcb806e3700 1 --2- 172.21.15.195:0/1642884570 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcb58063d30 0x7fcb580661e0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:40.077 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:40.075+0000 7fcb806e3700 1 --2- 172.21.15.195:0/1642884570 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcb7814e860 0x7fcb78153600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:40.078 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:40.075+0000 7fcb806e3700 1 --2- 172.21.15.195:0/1642884570 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcb780c3d80 0x7fcb78152830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:40.078 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:40.075+0000 7fcb806e3700 1 --2- 172.21.15.195:0/1642884570 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcb7814e490 0x7fcb78152d70 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:40.078 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:40.075+0000 7fcb806e3700 1 -- 172.21.15.195:0/1642884570 >> 172.21.15.195:0/1642884570 conn(0x7fcb78063af0 msgr2=0x7fcb781455a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:15:40.078 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:40.075+0000 7fcb806e3700 1 -- 172.21.15.195:0/1642884570 shutdown_connections 2022-06-27T12:15:40.078 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:40.075+0000 7fcb806e3700 1 -- 172.21.15.195:0/1642884570 wait complete. 2022-06-27T12:15:40.079 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 104 2022-06-27T12:15:40.087 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:15:42.088 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:15:42.248 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.229+0000 7f91c952d700 1 Processor -- start 2022-06-27T12:15:42.248 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.229+0000 7f91c952d700 1 -- start start 2022-06-27T12:15:42.248 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.229+0000 7f91c952d700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f91c40c5250 0x7f91c4157330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:42.257 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.229+0000 7f91c952d700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f91c40c5620 0x7f91c4067ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:42.257 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.229+0000 7f91c952d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f91c40c9700 0x7f91c4068410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:42.259 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.229+0000 7f91c952d700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91c4157870 con 0x7f91c40c9700 2022-06-27T12:15:42.260 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.229+0000 7f91c952d700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91c406e200 con 0x7f91c40c5620 2022-06-27T12:15:42.260 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.229+0000 7f91c952d700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91c406a010 con 0x7f91c40c5250 2022-06-27T12:15:42.260 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.229+0000 7f91c37fe700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f91c40c9700 0x7f91c4068410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:42.260 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.229+0000 7f91c37fe700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f91c40c9700 0x7f91c4068410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:51806/0 (socket says 172.21.15.195:51806) 2022-06-27T12:15:42.260 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.229+0000 7f91c37fe700 1 -- 172.21.15.195:0/1308155995 learned_addr learned my addr 172.21.15.195:0/1308155995 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:15:42.261 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.229+0000 7f91c27fc700 1 --2- 172.21.15.195:0/1308155995 >> v2:172.21.15.202:3300/0 conn(0x7f91c40c5620 0x7f91c4067ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:42.261 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.229+0000 7f91c2ffd700 1 --2- 172.21.15.195:0/1308155995 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f91c40c5250 0x7f91c4157330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:42.261 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.229+0000 7f91c37fe700 1 -- 172.21.15.195:0/1308155995 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f91c40c5250 msgr2=0x7f91c4157330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:15:42.262 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.229+0000 7f91c37fe700 1 --2- 172.21.15.195:0/1308155995 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f91c40c5250 0x7f91c4157330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:42.262 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.229+0000 7f91c37fe700 1 -- 172.21.15.195:0/1308155995 >> v2:172.21.15.202:3300/0 conn(0x7f91c40c5620 msgr2=0x7f91c4067ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:15:42.262 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.229+0000 7f91c37fe700 1 --2- 172.21.15.195:0/1308155995 >> v2:172.21.15.202:3300/0 conn(0x7f91c40c5620 0x7f91c4067ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:42.263 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.229+0000 7f91c37fe700 1 -- 172.21.15.195:0/1308155995 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91c406dbf0 con 0x7f91c40c9700 2022-06-27T12:15:42.263 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.229+0000 7f91c37fe700 1 --2- 172.21.15.195:0/1308155995 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f91c40c9700 0x7f91c4068410 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f91c406d380 tx=0x7f91bc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bce2c92926813170 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:42.263 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.229+0000 7f91c1ffb700 1 -- 172.21.15.195:0/1308155995 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91bc00e070 con 0x7f91c40c9700 2022-06-27T12:15:42.263 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.229+0000 7f91c1ffb700 1 -- 172.21.15.195:0/1308155995 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91bc00ba00 con 0x7f91c40c9700 2022-06-27T12:15:42.264 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.229+0000 7f91c1ffb700 1 -- 172.21.15.195:0/1308155995 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91bc0092e0 con 0x7f91c40c9700 2022-06-27T12:15:42.264 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.230+0000 7f91c952d700 1 -- 172.21.15.195:0/1308155995 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f91c40c9700 msgr2=0x7f91c4068410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:42.264 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.230+0000 7f91c952d700 1 --2- 172.21.15.195:0/1308155995 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f91c40c9700 0x7f91c4068410 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f91c406d380 tx=0x7f91bc009840 comp rx=0 tx=0).stop 2022-06-27T12:15:42.265 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.230+0000 7f91c952d700 1 -- 172.21.15.195:0/1308155995 shutdown_connections 2022-06-27T12:15:42.265 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.230+0000 7f91c952d700 1 --2- 172.21.15.195:0/1308155995 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f91c40c9700 0x7f91c4068410 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:42.265 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.230+0000 7f91c952d700 1 --2- 172.21.15.195:0/1308155995 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f91c40c5250 0x7f91c4157330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:42.266 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.230+0000 7f91c952d700 1 --2- 172.21.15.195:0/1308155995 >> v2:172.21.15.202:3300/0 conn(0x7f91c40c5620 0x7f91c4067ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:42.268 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.230+0000 7f91c952d700 1 -- 172.21.15.195:0/1308155995 >> 172.21.15.195:0/1308155995 conn(0x7f91c406c2e0 msgr2=0x7f91c40c8ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:15:42.268 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.230+0000 7f91c952d700 1 -- 172.21.15.195:0/1308155995 shutdown_connections 2022-06-27T12:15:42.268 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.230+0000 7f91c952d700 1 -- 172.21.15.195:0/1308155995 wait complete. 2022-06-27T12:15:42.269 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.231+0000 7f91c952d700 1 Processor -- start 2022-06-27T12:15:42.269 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.231+0000 7f91c952d700 1 -- start start 2022-06-27T12:15:42.269 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.232+0000 7f91c952d700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f91c40c5250 0x7f91c40bfb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:42.269 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.232+0000 7f91c952d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f91c40c5620 0x7f91c41518a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:42.270 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.232+0000 7f91c952d700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f91c40c9700 0x7f91c4167f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:42.270 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.232+0000 7f91c952d700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91c40b8430 con 0x7f91c40c5620 2022-06-27T12:15:42.270 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.232+0000 7f91c952d700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91c4167800 con 0x7f91c40c5250 2022-06-27T12:15:42.270 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.232+0000 7f91c952d700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91c4167ae0 con 0x7f91c40c9700 2022-06-27T12:15:42.271 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.232+0000 7f91c27fc700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f91c40c5620 0x7f91c41518a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:42.271 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.232+0000 7f91c2ffd700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f91c40c5250 0x7f91c40bfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:42.271 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.232+0000 7f91c27fc700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f91c40c5620 0x7f91c41518a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:51810/0 (socket says 172.21.15.195:51810) 2022-06-27T12:15:42.272 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.232+0000 7f91c27fc700 1 -- 172.21.15.195:0/781069927 learned_addr learned my addr 172.21.15.195:0/781069927 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:15:42.272 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.232+0000 7f91c37fe700 1 --2- 172.21.15.195:0/781069927 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f91c40c9700 0x7f91c4167f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:42.272 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.232+0000 7f91c2ffd700 1 -- 172.21.15.195:0/781069927 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f91c40c9700 msgr2=0x7f91c4167f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:42.272 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.232+0000 7f91c2ffd700 1 --2- 172.21.15.195:0/781069927 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f91c40c9700 0x7f91c4167f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:42.273 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.232+0000 7f91c2ffd700 1 -- 172.21.15.195:0/781069927 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f91c40c5620 msgr2=0x7f91c41518a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:42.273 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.232+0000 7f91c2ffd700 1 --2- 172.21.15.195:0/781069927 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f91c40c5620 0x7f91c41518a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:42.273 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.232+0000 7f91c2ffd700 1 -- 172.21.15.195:0/781069927 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91bc009ae0 con 0x7f91c40c5250 2022-06-27T12:15:42.273 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.232+0000 7f91c27fc700 1 --2- 172.21.15.195:0/781069927 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f91c40c5620 0x7f91c41518a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:15:42.273 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.232+0000 7f91c2ffd700 1 --2- 172.21.15.195:0/781069927 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f91c40c5250 0x7f91c40bfb20 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f91b400e930 tx=0x7f91b4009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:42.274 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.233+0000 7f91aaffd700 1 -- 172.21.15.195:0/781069927 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91b4013680 con 0x7f91c40c5250 2022-06-27T12:15:42.274 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.233+0000 7f91aaffd700 1 -- 172.21.15.195:0/781069927 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91b40137e0 con 0x7f91c40c5250 2022-06-27T12:15:42.274 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.233+0000 7f91c952d700 1 -- 172.21.15.195:0/781069927 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91c40bc340 con 0x7f91c40c5250 2022-06-27T12:15:42.274 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.233+0000 7f91aaffd700 1 -- 172.21.15.195:0/781069927 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91b4013ba0 con 0x7f91c40c5250 2022-06-27T12:15:42.275 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.233+0000 7f91c952d700 1 -- 172.21.15.195:0/781069927 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91c40bf5d0 con 0x7f91c40c5250 2022-06-27T12:15:42.275 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.234+0000 7f91c952d700 1 -- 172.21.15.195:0/781069927 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91c4167dc0 con 0x7f91c40c5250 2022-06-27T12:15:42.275 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.234+0000 7f91aaffd700 1 -- 172.21.15.195:0/781069927 <== mon.1 v2:172.21.15.202:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f91b4019030 con 0x7f91c40c5250 2022-06-27T12:15:42.275 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.234+0000 7f91aaffd700 1 --2- 172.21.15.195:0/781069927 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f91a4063d00 0x7f91a40661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:42.276 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.234+0000 7f91aaffd700 1 -- 172.21.15.195:0/781069927 <== mon.1 v2:172.21.15.202:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7578+0+0 (secure 0 0 0) 0x7f91b4080be0 con 0x7f91c40c5250 2022-06-27T12:15:42.276 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.234+0000 7f91c27fc700 1 --2- 172.21.15.195:0/781069927 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f91a4063d00 0x7f91a40661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:42.276 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.235+0000 7f91c27fc700 1 --2- 172.21.15.195:0/781069927 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f91a4063d00 0x7f91a40661b0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f91c406f180 tx=0x7f91b8009060 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:42.276 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.240+0000 7f91aaffd700 1 -- 172.21.15.195:0/781069927 <== mon.1 v2:172.21.15.202:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f91b404f180 con 0x7f91c40c5250 2022-06-27T12:15:42.474 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.467+0000 7f91c952d700 1 -- 172.21.15.195:0/781069927 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f91c40bee20 con 0x7f91c40c5250 2022-06-27T12:15:42.475 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.467+0000 7f91aaffd700 1 -- 172.21.15.195:0/781069927 <== mon.1 v2:172.21.15.202:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+5377 (secure 0 0 0) 0x7f91c40bee20 con 0x7f91c40c5250 2022-06-27T12:15:42.477 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:15:42.477 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":105,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:15:42.380083+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6254":{"gid":6254,"name":"b","rank":0,"incarnation":102,"state":"up:clientreplay","state_seq":40,"addr":"172.21.15.202:6833/1317767101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1317767101},{"type":"v1","addr":"172.21.15.202:6833","nonce":1317767101}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:15:42.480 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.469+0000 7f91c952d700 1 -- 172.21.15.195:0/781069927 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f91a4063d00 msgr2=0x7f91a40661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:42.480 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.469+0000 7f91c952d700 1 --2- 172.21.15.195:0/781069927 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f91a4063d00 0x7f91a40661b0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f91c406f180 tx=0x7f91b8009060 comp rx=0 tx=0).stop 2022-06-27T12:15:42.480 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.469+0000 7f91c952d700 1 -- 172.21.15.195:0/781069927 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f91c40c5250 msgr2=0x7f91c40bfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:42.481 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.469+0000 7f91c952d700 1 --2- 172.21.15.195:0/781069927 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f91c40c5250 0x7f91c40bfb20 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f91b400e930 tx=0x7f91b4009060 comp rx=0 tx=0).stop 2022-06-27T12:15:42.481 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.470+0000 7f91c952d700 1 -- 172.21.15.195:0/781069927 shutdown_connections 2022-06-27T12:15:42.481 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.470+0000 7f91c952d700 1 --2- 172.21.15.195:0/781069927 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f91a4063d00 0x7f91a40661b0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:42.482 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.470+0000 7f91c952d700 1 --2- 172.21.15.195:0/781069927 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f91c40c9700 0x7f91c4167f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:42.482 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.470+0000 7f91c952d700 1 --2- 172.21.15.195:0/781069927 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f91c40c5250 0x7f91c40bfb20 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:42.483 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.470+0000 7f91c952d700 1 --2- 172.21.15.195:0/781069927 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f91c40c5620 0x7f91c41518a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:42.483 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.470+0000 7f91c952d700 1 -- 172.21.15.195:0/781069927 >> 172.21.15.195:0/781069927 conn(0x7f91c406c2e0 msgr2=0x7f91c406e590 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:15:42.483 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.470+0000 7f91c952d700 1 -- 172.21.15.195:0/781069927 shutdown_connections 2022-06-27T12:15:42.484 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:42.470+0000 7f91c952d700 1 -- 172.21.15.195:0/781069927 wait complete. 2022-06-27T12:15:42.484 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 105 2022-06-27T12:15:42.492 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:15:44.494 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:15:44.634 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.633+0000 7f4def5b5700 1 Processor -- start 2022-06-27T12:15:44.634 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.633+0000 7f4def5b5700 1 -- start start 2022-06-27T12:15:44.646 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.634+0000 7f4def5b5700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f4de8150580 0x7f4de8156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:44.646 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.634+0000 7f4def5b5700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4de806f620 0x7f4de806d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:44.647 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.634+0000 7f4def5b5700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4de80c76b0 con 0x7f4de8150950 2022-06-27T12:15:44.647 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.634+0000 7f4def5b5700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4de8157430 con 0x7f4de8150580 2022-06-27T12:15:44.647 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.634+0000 7f4def5b5700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4de80c8250 con 0x7f4de806f620 2022-06-27T12:15:44.647 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.634+0000 7f4decb50700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f4de8150950 0x7f4de80c7b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:58008/0 (socket says 172.21.15.195:58008) 2022-06-27T12:15:44.648 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.634+0000 7f4decb50700 1 -- 172.21.15.195:0/767537890 learned_addr learned my addr 172.21.15.195:0/767537890 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:15:44.648 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.635+0000 7f4dedb52700 1 --2- 172.21.15.195:0/767537890 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4de806f620 0x7f4de806d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:44.648 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.635+0000 7f4ded351700 1 --2- 172.21.15.195:0/767537890 >> v2:172.21.15.202:3300/0 conn(0x7f4de8150580 0x7f4de8156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:44.649 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.635+0000 7f4dd7fff700 1 -- 172.21.15.195:0/767537890 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3441758687 0 0) 0x7f4de80c76b0 con 0x7f4de8150950 2022-06-27T12:15:44.649 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.635+0000 7f4dd7fff700 1 -- 172.21.15.195:0/767537890 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4dd0003430 con 0x7f4de8150950 2022-06-27T12:15:44.649 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.635+0000 7f4dedb52700 1 -- 172.21.15.195:0/767537890 >> v2:172.21.15.202:3300/0 conn(0x7f4de8150580 msgr2=0x7f4de8156ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:15:44.650 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.635+0000 7f4dedb52700 1 --2- 172.21.15.195:0/767537890 >> v2:172.21.15.202:3300/0 conn(0x7f4de8150580 0x7f4de8156ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:44.650 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.635+0000 7f4dedb52700 1 -- 172.21.15.195:0/767537890 >> v1:172.21.15.195:6789/0 conn(0x7f4de8150950 legacy=0x7f4de80c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:44.651 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.635+0000 7f4dedb52700 1 -- 172.21.15.195:0/767537890 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4de80bad70 con 0x7f4de806f620 2022-06-27T12:15:44.651 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.635+0000 7f4ded351700 1 --2- 172.21.15.195:0/767537890 >> v2:172.21.15.202:3300/0 conn(0x7f4de8150580 0x7f4de8156ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:15:44.652 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.635+0000 7f4dedb52700 1 --2- 172.21.15.195:0/767537890 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4de806f620 0x7f4de806d6a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f4de0009e60 tx=0x7f4de0009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93422b0d6d2abcff server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:44.652 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.635+0000 7f4dd7fff700 1 -- 172.21.15.195:0/767537890 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4de000e070 con 0x7f4de806f620 2022-06-27T12:15:44.653 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.635+0000 7f4dd7fff700 1 -- 172.21.15.195:0/767537890 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4de000bab0 con 0x7f4de806f620 2022-06-27T12:15:44.653 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.635+0000 7f4dd7fff700 1 -- 172.21.15.195:0/767537890 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4de000be30 con 0x7f4de806f620 2022-06-27T12:15:44.654 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.636+0000 7f4def5b5700 1 -- 172.21.15.195:0/767537890 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4de806f620 msgr2=0x7f4de806d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:44.654 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.636+0000 7f4def5b5700 1 --2- 172.21.15.195:0/767537890 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4de806f620 0x7f4de806d6a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f4de0009e60 tx=0x7f4de0009780 comp rx=0 tx=0).stop 2022-06-27T12:15:44.654 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.636+0000 7f4def5b5700 1 -- 172.21.15.195:0/767537890 shutdown_connections 2022-06-27T12:15:44.655 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.636+0000 7f4def5b5700 1 --2- 172.21.15.195:0/767537890 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4de806f620 0x7f4de806d6a0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:44.656 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.636+0000 7f4def5b5700 1 --2- 172.21.15.195:0/767537890 >> v2:172.21.15.202:3300/0 conn(0x7f4de8150580 0x7f4de8156ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:44.657 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.636+0000 7f4def5b5700 1 -- 172.21.15.195:0/767537890 >> 172.21.15.195:0/767537890 conn(0x7f4de8063af0 msgr2=0x7f4de8063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:15:44.658 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.636+0000 7f4def5b5700 1 -- 172.21.15.195:0/767537890 shutdown_connections 2022-06-27T12:15:44.658 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.636+0000 7f4def5b5700 1 -- 172.21.15.195:0/767537890 wait complete. 2022-06-27T12:15:44.659 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.637+0000 7f4def5b5700 1 Processor -- start 2022-06-27T12:15:44.659 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.637+0000 7f4def5b5700 1 -- start start 2022-06-27T12:15:44.660 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.638+0000 7f4def5b5700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4de806f620 0x7f4de815acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:44.660 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.638+0000 7f4def5b5700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4de8150580 0x7f4de80bc490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:44.661 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.638+0000 7f4def5b5700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4de8150950 0x7f4de814b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:44.661 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.638+0000 7f4def5b5700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de80bc000 con 0x7f4de8150580 2022-06-27T12:15:44.661 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.638+0000 7f4def5b5700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de80bb3a0 con 0x7f4de806f620 2022-06-27T12:15:44.663 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.638+0000 7f4def5b5700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4de80bb680 con 0x7f4de8150950 2022-06-27T12:15:44.664 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.638+0000 7f4decb50700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4de8150580 0x7f4de80bc490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:44.664 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.638+0000 7f4decb50700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4de8150580 0x7f4de80bc490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:51826/0 (socket says 172.21.15.195:51826) 2022-06-27T12:15:44.665 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.638+0000 7f4ded351700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4de806f620 0x7f4de815acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:44.665 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.638+0000 7f4decb50700 1 -- 172.21.15.195:0/1916241246 learned_addr learned my addr 172.21.15.195:0/1916241246 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:15:44.666 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.638+0000 7f4dedb52700 1 --2- 172.21.15.195:0/1916241246 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4de8150950 0x7f4de814b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:44.666 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.638+0000 7f4decb50700 1 -- 172.21.15.195:0/1916241246 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4de8150950 msgr2=0x7f4de814b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:44.667 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.638+0000 7f4decb50700 1 --2- 172.21.15.195:0/1916241246 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4de8150950 0x7f4de814b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:44.667 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.638+0000 7f4decb50700 1 -- 172.21.15.195:0/1916241246 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4de806f620 msgr2=0x7f4de815acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:44.667 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.638+0000 7f4decb50700 1 --2- 172.21.15.195:0/1916241246 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4de806f620 0x7f4de815acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:44.668 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.638+0000 7f4decb50700 1 -- 172.21.15.195:0/1916241246 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dd80092e0 con 0x7f4de8150580 2022-06-27T12:15:44.668 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.638+0000 7f4dedb52700 1 --2- 172.21.15.195:0/1916241246 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4de8150950 0x7f4de814b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:15:44.669 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.638+0000 7f4ded351700 1 --2- 172.21.15.195:0/1916241246 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4de806f620 0x7f4de815acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:15:44.669 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.639+0000 7f4decb50700 1 --2- 172.21.15.195:0/1916241246 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4de8150580 0x7f4de80bc490 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f4de400c410 tx=0x7f4de4007c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:44.669 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.639+0000 7f4dd57fa700 1 -- 172.21.15.195:0/1916241246 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4de40070d0 con 0x7f4de8150580 2022-06-27T12:15:44.677 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.639+0000 7f4dd57fa700 1 -- 172.21.15.195:0/1916241246 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4de4007230 con 0x7f4de8150580 2022-06-27T12:15:44.678 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.639+0000 7f4def5b5700 1 -- 172.21.15.195:0/1916241246 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4de0009a20 con 0x7f4de8150580 2022-06-27T12:15:44.678 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.639+0000 7f4dd57fa700 1 -- 172.21.15.195:0/1916241246 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4de40075f0 con 0x7f4de8150580 2022-06-27T12:15:44.679 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.639+0000 7f4def5b5700 1 -- 172.21.15.195:0/1916241246 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4de80bbc90 con 0x7f4de8150580 2022-06-27T12:15:44.679 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.640+0000 7f4def5b5700 1 -- 172.21.15.195:0/1916241246 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4de80bc1b0 con 0x7f4de8150580 2022-06-27T12:15:44.680 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.640+0000 7f4dd57fa700 1 -- 172.21.15.195:0/1916241246 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f4de4019030 con 0x7f4de8150580 2022-06-27T12:15:44.680 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.641+0000 7f4dd57fa700 1 --2- 172.21.15.195:0/1916241246 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4dc8063cd0 0x7f4dc8066180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:44.681 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.641+0000 7f4dd57fa700 1 -- 172.21.15.195:0/1916241246 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7578+0+0 (secure 0 0 0) 0x7f4de407ff40 con 0x7f4de8150580 2022-06-27T12:15:44.681 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.641+0000 7f4ded351700 1 --2- 172.21.15.195:0/1916241246 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4dc8063cd0 0x7f4dc8066180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:44.681 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.641+0000 7f4ded351700 1 --2- 172.21.15.195:0/1916241246 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4dc8063cd0 0x7f4dc8066180 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f4dd80114c0 tx=0x7f4dd8009060 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:44.683 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.645+0000 7f4dd57fa700 1 -- 172.21.15.195:0/1916241246 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f4de80bc1b0 con 0x7f4de8150580 2022-06-27T12:15:44.870 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.869+0000 7f4def5b5700 1 -- 172.21.15.195:0/1916241246 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4de806aba0 con 0x7f4de8150580 2022-06-27T12:15:44.871 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.870+0000 7f4dd57fa700 1 -- 172.21.15.195:0/1916241246 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+5377 (secure 0 0 0) 0x7f4de4017030 con 0x7f4de8150580 2022-06-27T12:15:44.872 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:15:44.875 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":105,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:15:42.380083+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6254":{"gid":6254,"name":"b","rank":0,"incarnation":102,"state":"up:clientreplay","state_seq":40,"addr":"172.21.15.202:6833/1317767101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1317767101},{"type":"v1","addr":"172.21.15.202:6833","nonce":1317767101}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:15:44.879 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.872+0000 7f4def5b5700 1 -- 172.21.15.195:0/1916241246 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4dc8063cd0 msgr2=0x7f4dc8066180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:44.880 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.872+0000 7f4def5b5700 1 --2- 172.21.15.195:0/1916241246 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4dc8063cd0 0x7f4dc8066180 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f4dd80114c0 tx=0x7f4dd8009060 comp rx=0 tx=0).stop 2022-06-27T12:15:44.903 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.872+0000 7f4def5b5700 1 -- 172.21.15.195:0/1916241246 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4de8150580 msgr2=0x7f4de80bc490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:44.903 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.872+0000 7f4def5b5700 1 --2- 172.21.15.195:0/1916241246 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4de8150580 0x7f4de80bc490 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f4de400c410 tx=0x7f4de4007c70 comp rx=0 tx=0).stop 2022-06-27T12:15:44.903 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.872+0000 7f4def5b5700 1 -- 172.21.15.195:0/1916241246 shutdown_connections 2022-06-27T12:15:44.904 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.872+0000 7f4def5b5700 1 --2- 172.21.15.195:0/1916241246 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4dc8063cd0 0x7f4dc8066180 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:44.904 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.872+0000 7f4def5b5700 1 --2- 172.21.15.195:0/1916241246 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4de8150950 0x7f4de814b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:44.905 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.872+0000 7f4def5b5700 1 --2- 172.21.15.195:0/1916241246 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4de806f620 0x7f4de815acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:44.905 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.872+0000 7f4def5b5700 1 --2- 172.21.15.195:0/1916241246 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4de8150580 0x7f4de80bc490 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:44.905 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.872+0000 7f4def5b5700 1 -- 172.21.15.195:0/1916241246 >> 172.21.15.195:0/1916241246 conn(0x7f4de8063af0 msgr2=0x7f4de806a490 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:15:44.906 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.872+0000 7f4def5b5700 1 -- 172.21.15.195:0/1916241246 shutdown_connections 2022-06-27T12:15:44.906 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:44.872+0000 7f4def5b5700 1 -- 172.21.15.195:0/1916241246 wait complete. 2022-06-27T12:15:44.912 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 105 2022-06-27T12:15:44.921 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 105, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-06-27T11:47:52.358095+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 105, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6254": { "addr": "172.21.15.202:6833/1317767101", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6832", "nonce": 1317767101, "type": "v2" }, { "addr": "172.21.15.202:6833", "nonce": 1317767101, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 6254, "incarnation": 102, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:clientreplay", "state_seq": 40 } }, "last_failure": 0, "last_failure_osd_epoch": 64, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-06-27T12:15:42.380083+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6254 } } } ], "standbys": [ { "addr": "172.21.15.202:6837/3305006841", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6836", "nonce": 3305006841, "type": "v2" }, { "addr": "172.21.15.202:6837", "nonce": 3305006841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6835/2060709822", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6834", "nonce": 2060709822, "type": "v2" }, { "addr": "172.21.15.195:6835", "nonce": 2060709822, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6838/1598751652", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6836", "nonce": 1598751652, "type": "v2" }, { "addr": "172.21.15.195:6838", "nonce": 1598751652, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4275, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6839/1313800661", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6837", "nonce": 1313800661, "type": "v2" }, { "addr": "172.21.15.195:6839", "nonce": 1313800661, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4278, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-06-27T12:15:46.931 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:15:47.070 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.065+0000 7f1817c3c700 1 Processor -- start 2022-06-27T12:15:47.070 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.066+0000 7f1817c3c700 1 -- start start 2022-06-27T12:15:47.071 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.066+0000 7f1817c3c700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f181006e850 0x7f18101570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:47.071 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.066+0000 7f1817c3c700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f18100c34a0 0x7f18100c2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:47.071 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.066+0000 7f1817c3c700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f181006fba0 con 0x7f181006fef0 2022-06-27T12:15:47.072 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.066+0000 7f1817c3c700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18101575e0 con 0x7f18100c34a0 2022-06-27T12:15:47.072 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.066+0000 7f1817c3c700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f181006f5e0 con 0x7f181006e850 2022-06-27T12:15:47.072 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.067+0000 7f18151d7700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f181006fef0 0x7f1810069de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:58022/0 (socket says 172.21.15.195:58022) 2022-06-27T12:15:47.073 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.067+0000 7f18151d7700 1 -- 172.21.15.195:0/1413115504 learned_addr learned my addr 172.21.15.195:0/1413115504 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:15:47.073 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.067+0000 7f18159d8700 1 --2- 172.21.15.195:0/1413115504 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f181006e850 0x7f18101570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:47.073 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.067+0000 7f18161d9700 1 --2- 172.21.15.195:0/1413115504 >> v2:172.21.15.202:3300/0 conn(0x7f18100c34a0 0x7f18100c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:47.073 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.067+0000 7f18149d6700 1 -- 172.21.15.195:0/1413115504 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 993178818 0 0) 0x7f181006fba0 con 0x7f181006fef0 2022-06-27T12:15:47.074 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.067+0000 7f18149d6700 1 -- 172.21.15.195:0/1413115504 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17f8003430 con 0x7f181006fef0 2022-06-27T12:15:47.074 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.067+0000 7f18159d8700 1 -- 172.21.15.195:0/1413115504 >> v2:172.21.15.202:3300/0 conn(0x7f18100c34a0 msgr2=0x7f18100c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:15:47.074 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.067+0000 7f18159d8700 1 --2- 172.21.15.195:0/1413115504 >> v2:172.21.15.202:3300/0 conn(0x7f18100c34a0 0x7f18100c2920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:47.074 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.067+0000 7f18159d8700 1 -- 172.21.15.195:0/1413115504 >> v1:172.21.15.195:6789/0 conn(0x7f181006fef0 legacy=0x7f1810069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:47.074 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.067+0000 7f18159d8700 1 -- 172.21.15.195:0/1413115504 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f181006f8c0 con 0x7f181006e850 2022-06-27T12:15:47.076 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.067+0000 7f18161d9700 1 --2- 172.21.15.195:0/1413115504 >> v2:172.21.15.202:3300/0 conn(0x7f18100c34a0 0x7f18100c2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:15:47.076 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.067+0000 7f18159d8700 1 --2- 172.21.15.195:0/1413115504 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f181006e850 0x7f18101570a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f180c0097a0 tx=0x7f180c009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d324a51349141b35 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:47.076 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.067+0000 7f18149d6700 1 -- 172.21.15.195:0/1413115504 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f180c011630 con 0x7f181006e850 2022-06-27T12:15:47.076 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.067+0000 7f18149d6700 1 -- 172.21.15.195:0/1413115504 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f180c011790 con 0x7f181006e850 2022-06-27T12:15:47.077 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.067+0000 7f18149d6700 1 -- 172.21.15.195:0/1413115504 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f180c011b30 con 0x7f181006e850 2022-06-27T12:15:47.077 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.068+0000 7f1817c3c700 1 -- 172.21.15.195:0/1413115504 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f181006e850 msgr2=0x7f18101570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:47.077 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.068+0000 7f1817c3c700 1 --2- 172.21.15.195:0/1413115504 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f181006e850 0x7f18101570a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f180c0097a0 tx=0x7f180c009060 comp rx=0 tx=0).stop 2022-06-27T12:15:47.077 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.068+0000 7f1817c3c700 1 -- 172.21.15.195:0/1413115504 shutdown_connections 2022-06-27T12:15:47.078 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.068+0000 7f1817c3c700 1 --2- 172.21.15.195:0/1413115504 >> v2:172.21.15.202:3300/0 conn(0x7f18100c34a0 0x7f18100c2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:47.078 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.068+0000 7f1817c3c700 1 --2- 172.21.15.195:0/1413115504 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f181006e850 0x7f18101570a0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:47.078 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.068+0000 7f1817c3c700 1 -- 172.21.15.195:0/1413115504 >> 172.21.15.195:0/1413115504 conn(0x7f18100c6350 msgr2=0x7f18100c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:15:47.078 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.068+0000 7f1817c3c700 1 -- 172.21.15.195:0/1413115504 shutdown_connections 2022-06-27T12:15:47.079 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.068+0000 7f1817c3c700 1 -- 172.21.15.195:0/1413115504 wait complete. 2022-06-27T12:15:47.079 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.069+0000 7f1817c3c700 1 Processor -- start 2022-06-27T12:15:47.099 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.069+0000 7f1817c3c700 1 -- start start 2022-06-27T12:15:47.099 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.070+0000 7f1817c3c700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f181006e850 0x7f18101682c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:47.100 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.070+0000 7f1817c3c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f18100c34a0 0x7f1810153400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:47.100 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.070+0000 7f1817c3c700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f181006a330 0x7f181014ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:47.101 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.070+0000 7f1817c3c700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1810165340 con 0x7f18100c34a0 2022-06-27T12:15:47.101 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.070+0000 7f1817c3c700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1810167e30 con 0x7f181006a330 2022-06-27T12:15:47.101 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.070+0000 7f1817c3c700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1810164e00 con 0x7f181006e850 2022-06-27T12:15:47.101 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.070+0000 7f18151d7700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f18100c34a0 0x7f1810153400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:47.101 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.070+0000 7f18159d8700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f181006e850 0x7f18101682c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:47.102 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.070+0000 7f18151d7700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f18100c34a0 0x7f1810153400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:51840/0 (socket says 172.21.15.195:51840) 2022-06-27T12:15:47.102 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.070+0000 7f18151d7700 1 -- 172.21.15.195:0/354516978 learned_addr learned my addr 172.21.15.195:0/354516978 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:15:47.102 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.070+0000 7f18161d9700 1 --2- 172.21.15.195:0/354516978 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f181006a330 0x7f181014ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:47.102 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.070+0000 7f18151d7700 1 -- 172.21.15.195:0/354516978 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f181006e850 msgr2=0x7f18101682c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:47.103 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.070+0000 7f18151d7700 1 --2- 172.21.15.195:0/354516978 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f181006e850 0x7f18101682c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:47.103 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.070+0000 7f18151d7700 1 -- 172.21.15.195:0/354516978 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f181006a330 msgr2=0x7f181014ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:47.103 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.070+0000 7f18151d7700 1 --2- 172.21.15.195:0/354516978 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f181006a330 0x7f181014ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:47.103 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.070+0000 7f18151d7700 1 -- 172.21.15.195:0/354516978 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1808009a20 con 0x7f18100c34a0 2022-06-27T12:15:47.104 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.070+0000 7f18159d8700 1 --2- 172.21.15.195:0/354516978 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f181006e850 0x7f18101682c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:15:47.104 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.070+0000 7f18161d9700 1 --2- 172.21.15.195:0/354516978 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f181006a330 0x7f181014ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:15:47.104 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.071+0000 7f18151d7700 1 --2- 172.21.15.195:0/354516978 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f18100c34a0 0x7f1810153400 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f180000c490 tx=0x7f1800007c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:47.104 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.071+0000 7f17fdffb700 1 -- 172.21.15.195:0/354516978 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18000072e0 con 0x7f18100c34a0 2022-06-27T12:15:47.104 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.071+0000 7f17fdffb700 1 -- 172.21.15.195:0/354516978 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1800007460 con 0x7f18100c34a0 2022-06-27T12:15:47.105 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.071+0000 7f1817c3c700 1 -- 172.21.15.195:0/354516978 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f180c0092e0 con 0x7f18100c34a0 2022-06-27T12:15:47.105 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.071+0000 7f17fdffb700 1 -- 172.21.15.195:0/354516978 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18000057c0 con 0x7f18100c34a0 2022-06-27T12:15:47.105 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.071+0000 7f1817c3c700 1 -- 172.21.15.195:0/354516978 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18100bd780 con 0x7f18100c34a0 2022-06-27T12:15:47.105 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.072+0000 7f1817c3c700 1 -- 172.21.15.195:0/354516978 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1810167960 con 0x7f18100c34a0 2022-06-27T12:15:47.106 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.072+0000 7f17fdffb700 1 -- 172.21.15.195:0/354516978 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f1800019030 con 0x7f18100c34a0 2022-06-27T12:15:47.106 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.073+0000 7f17fdffb700 1 --2- 172.21.15.195:0/354516978 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f17f0063d00 0x7f17f00661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:47.106 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.073+0000 7f17fdffb700 1 -- 172.21.15.195:0/354516978 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7578+0+0 (secure 0 0 0) 0x7f1800080000 con 0x7f18100c34a0 2022-06-27T12:15:47.107 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.073+0000 7f18159d8700 1 --2- 172.21.15.195:0/354516978 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f17f0063d00 0x7f17f00661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:47.107 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.073+0000 7f18159d8700 1 --2- 172.21.15.195:0/354516978 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f17f0063d00 0x7f17f00661b0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f180c009260 tx=0x7f180c017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:47.107 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.077+0000 7f17fdffb700 1 -- 172.21.15.195:0/354516978 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f1810167960 con 0x7f18100c34a0 2022-06-27T12:15:47.312 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.301+0000 7f1817c3c700 1 -- 172.21.15.195:0/354516978 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1810167960 con 0x7f18100c34a0 2022-06-27T12:15:47.312 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.302+0000 7f17fdffb700 1 -- 172.21.15.195:0/354516978 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+5377 (secure 0 0 0) 0x7f1810167960 con 0x7f18100c34a0 2022-06-27T12:15:47.313 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:15:47.316 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":105,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:15:42.380083+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6254":{"gid":6254,"name":"b","rank":0,"incarnation":102,"state":"up:clientreplay","state_seq":40,"addr":"172.21.15.202:6833/1317767101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1317767101},{"type":"v1","addr":"172.21.15.202:6833","nonce":1317767101}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:15:47.319 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.304+0000 7f1817c3c700 1 -- 172.21.15.195:0/354516978 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f17f0063d00 msgr2=0x7f17f00661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:47.330 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.304+0000 7f1817c3c700 1 --2- 172.21.15.195:0/354516978 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f17f0063d00 0x7f17f00661b0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f180c009260 tx=0x7f180c017000 comp rx=0 tx=0).stop 2022-06-27T12:15:47.331 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.304+0000 7f1817c3c700 1 -- 172.21.15.195:0/354516978 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f18100c34a0 msgr2=0x7f1810153400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:47.331 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.304+0000 7f1817c3c700 1 --2- 172.21.15.195:0/354516978 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f18100c34a0 0x7f1810153400 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f180000c490 tx=0x7f1800007c50 comp rx=0 tx=0).stop 2022-06-27T12:15:47.332 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.304+0000 7f1817c3c700 1 -- 172.21.15.195:0/354516978 shutdown_connections 2022-06-27T12:15:47.332 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.304+0000 7f1817c3c700 1 --2- 172.21.15.195:0/354516978 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f17f0063d00 0x7f17f00661b0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:47.333 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.304+0000 7f1817c3c700 1 --2- 172.21.15.195:0/354516978 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f181006a330 0x7f181014ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:47.333 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.304+0000 7f1817c3c700 1 --2- 172.21.15.195:0/354516978 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f181006e850 0x7f18101682c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:47.333 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.304+0000 7f1817c3c700 1 --2- 172.21.15.195:0/354516978 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f18100c34a0 0x7f1810153400 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:47.334 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.304+0000 7f1817c3c700 1 -- 172.21.15.195:0/354516978 >> 172.21.15.195:0/354516978 conn(0x7f18100c6350 msgr2=0x7f18100c7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:15:47.335 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.304+0000 7f1817c3c700 1 -- 172.21.15.195:0/354516978 shutdown_connections 2022-06-27T12:15:47.335 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:47.304+0000 7f1817c3c700 1 -- 172.21.15.195:0/354516978 wait complete. 2022-06-27T12:15:47.336 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 105 2022-06-27T12:15:47.339 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:15:49.342 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:15:49.493 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.483+0000 7f25e186f700 1 Processor -- start 2022-06-27T12:15:49.495 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.483+0000 7f25e186f700 1 -- start start 2022-06-27T12:15:49.496 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.483+0000 7f25e186f700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f25dc0c40d0 0x7f25dc157010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:49.496 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.483+0000 7f25e186f700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f25dc067ed0 0x7f25dc14cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:49.497 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.483+0000 7f25e186f700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25dc156b80 con 0x7f25dc0c6190 2022-06-27T12:15:49.497 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.483+0000 7f25e186f700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25dc0682f0 con 0x7f25dc067ed0 2022-06-27T12:15:49.497 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.483+0000 7f25e186f700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25dc06a040 con 0x7f25dc0c40d0 2022-06-27T12:15:49.498 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.484+0000 7f25db7fe700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f25dc067ed0 0x7f25dc14cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:49.498 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.484+0000 7f25db7fe700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f25dc067ed0 0x7f25dc14cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:51852/0 (socket says 172.21.15.195:51852) 2022-06-27T12:15:49.499 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.484+0000 7f25db7fe700 1 -- 172.21.15.195:0/3806521548 learned_addr learned my addr 172.21.15.195:0/3806521548 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:15:49.499 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.484+0000 7f25daffd700 1 --2- 172.21.15.195:0/3806521548 >> v2:172.21.15.202:3300/0 conn(0x7f25dc0c40d0 0x7f25dc157010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:49.500 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.484+0000 7f25db7fe700 1 -- 172.21.15.195:0/3806521548 >> v2:172.21.15.202:3300/0 conn(0x7f25dc0c40d0 msgr2=0x7f25dc157010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:15:49.500 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.484+0000 7f25db7fe700 1 --2- 172.21.15.195:0/3806521548 >> v2:172.21.15.202:3300/0 conn(0x7f25dc0c40d0 0x7f25dc157010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:49.501 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.484+0000 7f25db7fe700 1 -- 172.21.15.195:0/3806521548 >> v1:172.21.15.202:6789/0 conn(0x7f25dc0c6190 legacy=0x7f25dc0c44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:49.501 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.484+0000 7f25db7fe700 1 -- 172.21.15.195:0/3806521548 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25dc06a300 con 0x7f25dc067ed0 2022-06-27T12:15:49.503 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.484+0000 7f25db7fe700 1 --2- 172.21.15.195:0/3806521548 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f25dc067ed0 0x7f25dc14cb80 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f25dc0cdb00 tx=0x7f25d4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=876248f06dbe7d6f server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:49.503 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.484+0000 7f25d9ffb700 1 -- 172.21.15.195:0/3806521548 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25d400e070 con 0x7f25dc067ed0 2022-06-27T12:15:49.504 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.484+0000 7f25d9ffb700 1 -- 172.21.15.195:0/3806521548 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25d400ba20 con 0x7f25dc067ed0 2022-06-27T12:15:49.504 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.484+0000 7f25d9ffb700 1 -- 172.21.15.195:0/3806521548 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25d40092e0 con 0x7f25dc067ed0 2022-06-27T12:15:49.504 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.485+0000 7f25e186f700 1 -- 172.21.15.195:0/3806521548 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f25dc067ed0 msgr2=0x7f25dc14cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:49.505 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.485+0000 7f25e186f700 1 --2- 172.21.15.195:0/3806521548 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f25dc067ed0 0x7f25dc14cb80 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f25dc0cdb00 tx=0x7f25d4009840 comp rx=0 tx=0).stop 2022-06-27T12:15:49.524 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.485+0000 7f25e186f700 1 -- 172.21.15.195:0/3806521548 shutdown_connections 2022-06-27T12:15:49.524 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.485+0000 7f25e186f700 1 --2- 172.21.15.195:0/3806521548 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f25dc067ed0 0x7f25dc14cb80 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:49.525 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.485+0000 7f25e186f700 1 --2- 172.21.15.195:0/3806521548 >> v2:172.21.15.202:3300/0 conn(0x7f25dc0c40d0 0x7f25dc157010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:49.525 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.485+0000 7f25e186f700 1 -- 172.21.15.195:0/3806521548 >> 172.21.15.195:0/3806521548 conn(0x7f25dc0ccb80 msgr2=0x7f25dc0ccf80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:15:49.529 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.485+0000 7f25e186f700 1 -- 172.21.15.195:0/3806521548 shutdown_connections 2022-06-27T12:15:49.538 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.485+0000 7f25e186f700 1 -- 172.21.15.195:0/3806521548 wait complete. 2022-06-27T12:15:49.539 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.486+0000 7f25e186f700 1 Processor -- start 2022-06-27T12:15:49.539 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.486+0000 7f25e186f700 1 -- start start 2022-06-27T12:15:49.545 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.487+0000 7f25e186f700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f25dc067ed0 0x7f25dc169460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:49.546 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.487+0000 7f25e186f700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f25dc0c40d0 0x7f25dc14e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:49.546 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.487+0000 7f25e186f700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f25dc0c6190 0x7f25dc0c7c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:49.546 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.487+0000 7f25e186f700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25dc0c3a20 con 0x7f25dc0c40d0 2022-06-27T12:15:49.547 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.487+0000 7f25e186f700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25dc0c2bc0 con 0x7f25dc0c6190 2022-06-27T12:15:49.547 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.487+0000 7f25e186f700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25dc0c2ea0 con 0x7f25dc067ed0 2022-06-27T12:15:49.547 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.487+0000 7f25da7fc700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f25dc0c40d0 0x7f25dc14e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:49.547 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.487+0000 7f25da7fc700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f25dc0c40d0 0x7f25dc14e0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:51856/0 (socket says 172.21.15.195:51856) 2022-06-27T12:15:49.549 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.487+0000 7f25da7fc700 1 -- 172.21.15.195:0/1447477128 learned_addr learned my addr 172.21.15.195:0/1447477128 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:15:49.549 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.487+0000 7f25daffd700 1 --2- 172.21.15.195:0/1447477128 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f25dc067ed0 0x7f25dc169460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:49.549 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.487+0000 7f25db7fe700 1 --2- 172.21.15.195:0/1447477128 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f25dc0c6190 0x7f25dc0c7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:49.549 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.487+0000 7f25da7fc700 1 -- 172.21.15.195:0/1447477128 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f25dc067ed0 msgr2=0x7f25dc169460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:49.550 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.487+0000 7f25da7fc700 1 --2- 172.21.15.195:0/1447477128 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f25dc067ed0 0x7f25dc169460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:49.550 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.487+0000 7f25da7fc700 1 -- 172.21.15.195:0/1447477128 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f25dc0c6190 msgr2=0x7f25dc0c7c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:49.550 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.487+0000 7f25da7fc700 1 --2- 172.21.15.195:0/1447477128 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f25dc0c6190 0x7f25dc0c7c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:49.550 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.487+0000 7f25da7fc700 1 -- 172.21.15.195:0/1447477128 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25d4009ae0 con 0x7f25dc0c40d0 2022-06-27T12:15:49.550 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.487+0000 7f25da7fc700 1 --2- 172.21.15.195:0/1447477128 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f25dc0c40d0 0x7f25dc14e0d0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f25d000c410 tx=0x7f25d0007e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:49.551 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.488+0000 7f25c2ffd700 1 -- 172.21.15.195:0/1447477128 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25d00050d0 con 0x7f25dc0c40d0 2022-06-27T12:15:49.551 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.488+0000 7f25c2ffd700 1 -- 172.21.15.195:0/1447477128 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25d0005250 con 0x7f25dc0c40d0 2022-06-27T12:15:49.551 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.488+0000 7f25c2ffd700 1 -- 172.21.15.195:0/1447477128 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25d0003750 con 0x7f25dc0c40d0 2022-06-27T12:15:49.551 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.488+0000 7f25e186f700 1 -- 172.21.15.195:0/1447477128 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25dc0c3180 con 0x7f25dc0c40d0 2022-06-27T12:15:49.552 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.488+0000 7f25e186f700 1 -- 172.21.15.195:0/1447477128 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25dc0c3740 con 0x7f25dc0c40d0 2022-06-27T12:15:49.552 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.489+0000 7f25e186f700 1 -- 172.21.15.195:0/1447477128 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25dc0c06b0 con 0x7f25dc0c40d0 2022-06-27T12:15:49.552 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.489+0000 7f25c2ffd700 1 -- 172.21.15.195:0/1447477128 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f25d0019030 con 0x7f25dc0c40d0 2022-06-27T12:15:49.552 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.489+0000 7f25c2ffd700 1 --2- 172.21.15.195:0/1447477128 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f25bc063d00 0x7f25bc0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:49.552 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.490+0000 7f25c2ffd700 1 -- 172.21.15.195:0/1447477128 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7578+0+0 (secure 0 0 0) 0x7f25d007fe60 con 0x7f25dc0c40d0 2022-06-27T12:15:49.553 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.490+0000 7f25daffd700 1 --2- 172.21.15.195:0/1447477128 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f25bc063d00 0x7f25bc0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:49.553 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.490+0000 7f25daffd700 1 --2- 172.21.15.195:0/1447477128 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f25bc063d00 0x7f25bc0661b0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f25cc009c70 tx=0x7f25cc009060 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:49.554 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.494+0000 7f25c2ffd700 1 -- 172.21.15.195:0/1447477128 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f25dc0c06b0 con 0x7f25dc0c40d0 2022-06-27T12:15:49.733 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.723+0000 7f25e186f700 1 -- 172.21.15.195:0/1447477128 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25dc0c06b0 con 0x7f25dc0c40d0 2022-06-27T12:15:49.734 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.723+0000 7f25c2ffd700 1 -- 172.21.15.195:0/1447477128 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+5377 (secure 0 0 0) 0x7f25dc0c06b0 con 0x7f25dc0c40d0 2022-06-27T12:15:49.735 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:15:49.736 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":105,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:15:42.380083+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6254":{"gid":6254,"name":"b","rank":0,"incarnation":102,"state":"up:clientreplay","state_seq":40,"addr":"172.21.15.202:6833/1317767101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1317767101},{"type":"v1","addr":"172.21.15.202:6833","nonce":1317767101}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:15:49.739 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.725+0000 7f25e186f700 1 -- 172.21.15.195:0/1447477128 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f25bc063d00 msgr2=0x7f25bc0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:49.739 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.725+0000 7f25e186f700 1 --2- 172.21.15.195:0/1447477128 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f25bc063d00 0x7f25bc0661b0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f25cc009c70 tx=0x7f25cc009060 comp rx=0 tx=0).stop 2022-06-27T12:15:49.740 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.725+0000 7f25e186f700 1 -- 172.21.15.195:0/1447477128 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f25dc0c40d0 msgr2=0x7f25dc14e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:49.740 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.725+0000 7f25e186f700 1 --2- 172.21.15.195:0/1447477128 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f25dc0c40d0 0x7f25dc14e0d0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f25d000c410 tx=0x7f25d0007e40 comp rx=0 tx=0).stop 2022-06-27T12:15:49.740 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.725+0000 7f25e186f700 1 -- 172.21.15.195:0/1447477128 shutdown_connections 2022-06-27T12:15:49.742 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.725+0000 7f25e186f700 1 --2- 172.21.15.195:0/1447477128 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f25bc063d00 0x7f25bc0661b0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:49.742 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.725+0000 7f25e186f700 1 --2- 172.21.15.195:0/1447477128 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f25dc0c6190 0x7f25dc0c7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:49.743 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.725+0000 7f25e186f700 1 --2- 172.21.15.195:0/1447477128 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f25dc067ed0 0x7f25dc169460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:49.744 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.725+0000 7f25e186f700 1 --2- 172.21.15.195:0/1447477128 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f25dc0c40d0 0x7f25dc14e0d0 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:49.744 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.725+0000 7f25e186f700 1 -- 172.21.15.195:0/1447477128 >> 172.21.15.195:0/1447477128 conn(0x7f25dc0ccb80 msgr2=0x7f25dc0cb450 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:15:49.744 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.725+0000 7f25e186f700 1 -- 172.21.15.195:0/1447477128 shutdown_connections 2022-06-27T12:15:49.745 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:49.725+0000 7f25e186f700 1 -- 172.21.15.195:0/1447477128 wait complete. 2022-06-27T12:15:49.745 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 105 2022-06-27T12:15:49.746 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:15:51.757 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:15:51.890 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:51.883+0000 7f8aace3c700 1 Processor -- start 2022-06-27T12:15:51.890 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:51.883+0000 7f8aace3c700 1 -- start start 2022-06-27T12:15:51.891 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:51.884+0000 7f8aace3c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f8aa8153cb0 0x7f8aa8156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:51.891 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:51.884+0000 7f8aace3c700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8aa80c1a20 con 0x7f8aa80c4160 2022-06-27T12:15:51.892 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:51.884+0000 7f8aace3c700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8aa80c1130 con 0x7f8aa806f640 2022-06-27T12:15:51.892 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:51.884+0000 7f8aace3c700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8aa806ea90 con 0x7f8aa8153cb0 2022-06-27T12:15:51.892 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:51.884+0000 7f8aa6d9d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f8aa8153cb0 0x7f8aa8156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:51.893 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:51.884+0000 7f8aa5d9b700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f8aa80c4160 0x7f8aa81535a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:58066/0 (socket says 172.21.15.195:58066) 2022-06-27T12:15:51.893 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:51.884+0000 7f8aa5d9b700 1 -- 172.21.15.195:0/2402764313 learned_addr learned my addr 172.21.15.195:0/2402764313 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:15:51.894 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:51.884+0000 7f8aa6d9d700 1 -- 172.21.15.195:0/2402764313 >> v1:172.21.15.202:6789/0 conn(0x7f8aa806f640 legacy=0x7f8aa80c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:51.894 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:51.884+0000 7f8aa6d9d700 1 -- 172.21.15.195:0/2402764313 >> v1:172.21.15.195:6789/0 conn(0x7f8aa80c4160 legacy=0x7f8aa81535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:51.894 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:51.884+0000 7f8aa559a700 1 -- 172.21.15.195:0/2402764313 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2913145176 0 0) 0x7f8aa80c1a20 con 0x7f8aa80c4160 2022-06-27T12:15:51.895 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:51.884+0000 7f8aa6d9d700 1 -- 172.21.15.195:0/2402764313 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aa806ed70 con 0x7f8aa8153cb0 2022-06-27T12:15:51.895 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:51.884+0000 7f8aa6d9d700 1 --2- 172.21.15.195:0/2402764313 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8aa8153cb0 0x7f8aa8156ed0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f8aa0009e60 tx=0x7f8aa0009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c883bd616007ed6 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:51.895 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:51.884+0000 7f8aa559a700 1 -- 172.21.15.195:0/2402764313 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aa000e070 con 0x7f8aa8153cb0 2022-06-27T12:15:51.896 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:51.884+0000 7f8aa559a700 1 -- 172.21.15.195:0/2402764313 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8aa000bcc0 con 0x7f8aa8153cb0 2022-06-27T12:15:51.896 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:51.885+0000 7f8aa559a700 1 -- 172.21.15.195:0/2402764313 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aa0012680 con 0x7f8aa8153cb0 2022-06-27T12:15:51.897 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:51.885+0000 7f8aace3c700 1 -- 172.21.15.195:0/2402764313 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8aa8153cb0 msgr2=0x7f8aa8156ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:51.897 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:51.885+0000 7f8aace3c700 1 --2- 172.21.15.195:0/2402764313 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8aa8153cb0 0x7f8aa8156ed0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f8aa0009e60 tx=0x7f8aa0009780 comp rx=0 tx=0).stop 2022-06-27T12:15:51.899 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:51.885+0000 7f8aace3c700 1 -- 172.21.15.195:0/2402764313 shutdown_connections 2022-06-27T12:15:51.899 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:51.885+0000 7f8aace3c700 1 --2- 172.21.15.195:0/2402764313 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8aa8153cb0 0x7f8aa8156ed0 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:51.907 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:51.885+0000 7f8aace3c700 1 -- 172.21.15.195:0/2402764313 >> 172.21.15.195:0/2402764313 conn(0x7f8aa80c6350 msgr2=0x7f8aa80c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:15:51.908 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:51.885+0000 7f8aace3c700 1 -- 172.21.15.195:0/2402764313 shutdown_connections 2022-06-27T12:15:51.910 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:51.885+0000 7f8aace3c700 1 -- 172.21.15.195:0/2402764313 wait complete. 2022-06-27T12:15:51.911 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:51.886+0000 7f8aace3c700 1 Processor -- start 2022-06-27T12:15:51.911 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:51.886+0000 7f8aace3c700 1 -- start start 2022-06-27T12:15:51.912 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:51.887+0000 7f8aace3c700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8aa806f640 0x7f8aa814f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:51.913 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:51.887+0000 7f8aace3c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8aa80c4160 0x7f8aa80c4530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:51.913 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:51.887+0000 7f8aace3c700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8aa8153cb0 0x7f8aa80c4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:51.913 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:51.887+0000 7f8aace3c700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa81572b0 con 0x7f8aa80c4160 2022-06-27T12:15:51.914 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:51.887+0000 7f8aace3c700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa8069b70 con 0x7f8aa8153cb0 2022-06-27T12:15:51.914 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:51.887+0000 7f8aace3c700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa80be150 con 0x7f8aa806f640 2022-06-27T12:15:51.915 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:51.887+0000 7f8aa5d9b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8aa80c4160 0x7f8aa80c4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:51.915 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:51.887+0000 7f8aa5d9b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8aa80c4160 0x7f8aa80c4530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:51884/0 (socket says 172.21.15.195:51884) 2022-06-27T12:15:51.916 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:51.887+0000 7f8aa5d9b700 1 -- 172.21.15.195:0/3470927784 learned_addr learned my addr 172.21.15.195:0/3470927784 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:15:51.916 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:51.887+0000 7f8aa6d9d700 1 --2- 172.21.15.195:0/3470927784 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8aa8153cb0 0x7f8aa80c4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:51.916 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:51.887+0000 7f8aa659c700 1 --2- 172.21.15.195:0/3470927784 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8aa806f640 0x7f8aa814f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:51.917 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:51.887+0000 7f8aa5d9b700 1 -- 172.21.15.195:0/3470927784 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8aa806f640 msgr2=0x7f8aa814f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:51.917 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:51.887+0000 7f8aa5d9b700 1 --2- 172.21.15.195:0/3470927784 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8aa806f640 0x7f8aa814f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:51.918 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:51.887+0000 7f8aa5d9b700 1 -- 172.21.15.195:0/3470927784 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8aa8153cb0 msgr2=0x7f8aa80c4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:51.918 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:51.887+0000 7f8aa5d9b700 1 --2- 172.21.15.195:0/3470927784 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8aa8153cb0 0x7f8aa80c4a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:51.919 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:51.887+0000 7f8aa5d9b700 1 -- 172.21.15.195:0/3470927784 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aa0009a20 con 0x7f8aa80c4160 2022-06-27T12:15:51.920 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:51.887+0000 7f8aa5d9b700 1 --2- 172.21.15.195:0/3470927784 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8aa80c4160 0x7f8aa80c4530 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f8a9800c5c0 tx=0x7f8a98007cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:51.921 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:51.888+0000 7f8a967fc700 1 -- 172.21.15.195:0/3470927784 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a98007360 con 0x7f8aa80c4160 2022-06-27T12:15:51.921 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:51.888+0000 7f8a967fc700 1 -- 172.21.15.195:0/3470927784 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a98007500 con 0x7f8aa80c4160 2022-06-27T12:15:51.922 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:51.888+0000 7f8aace3c700 1 -- 172.21.15.195:0/3470927784 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8aa80c0b10 con 0x7f8aa80c4160 2022-06-27T12:15:51.922 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:51.888+0000 7f8a967fc700 1 -- 172.21.15.195:0/3470927784 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a98004940 con 0x7f8aa80c4160 2022-06-27T12:15:51.923 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:51.888+0000 7f8aace3c700 1 -- 172.21.15.195:0/3470927784 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8aa80bd540 con 0x7f8aa80c4160 2022-06-27T12:15:51.923 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:51.888+0000 7f8aace3c700 1 -- 172.21.15.195:0/3470927784 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8aa80722c0 con 0x7f8aa80c4160 2022-06-27T12:15:51.923 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:51.889+0000 7f8a967fc700 1 -- 172.21.15.195:0/3470927784 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f8a98010030 con 0x7f8aa80c4160 2022-06-27T12:15:51.924 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:51.889+0000 7f8a967fc700 1 --2- 172.21.15.195:0/3470927784 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8a84063d00 0x7f8a840661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:51.924 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:51.889+0000 7f8a967fc700 1 -- 172.21.15.195:0/3470927784 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7578+0+0 (secure 0 0 0) 0x7f8a98080990 con 0x7f8aa80c4160 2022-06-27T12:15:51.925 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:51.889+0000 7f8aa659c700 1 --2- 172.21.15.195:0/3470927784 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8a84063d00 0x7f8a840661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:51.925 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:51.890+0000 7f8aa659c700 1 --2- 172.21.15.195:0/3470927784 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8a84063d00 0x7f8a840661b0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c001880 tx=0x7f8a9c009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:51.926 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:51.894+0000 7f8a967fc700 1 -- 172.21.15.195:0/3470927784 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f8a9804ef90 con 0x7f8aa80c4160 2022-06-27T12:15:52.166 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:52.164+0000 7f8aace3c700 1 -- 172.21.15.195:0/3470927784 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8aa806f240 con 0x7f8aa80c4160 2022-06-27T12:15:52.166 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:52.165+0000 7f8a967fc700 1 -- 172.21.15.195:0/3470927784 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+5377 (secure 0 0 0) 0x7f8a980525b0 con 0x7f8aa80c4160 2022-06-27T12:15:52.167 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:15:52.168 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":105,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:15:42.380083+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6254":{"gid":6254,"name":"b","rank":0,"incarnation":102,"state":"up:clientreplay","state_seq":40,"addr":"172.21.15.202:6833/1317767101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1317767101},{"type":"v1","addr":"172.21.15.202:6833","nonce":1317767101}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:15:52.170 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:52.166+0000 7f8aace3c700 1 -- 172.21.15.195:0/3470927784 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8a84063d00 msgr2=0x7f8a840661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:52.171 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:52.166+0000 7f8aace3c700 1 --2- 172.21.15.195:0/3470927784 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8a84063d00 0x7f8a840661b0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c001880 tx=0x7f8a9c009040 comp rx=0 tx=0).stop 2022-06-27T12:15:52.171 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:52.166+0000 7f8aace3c700 1 -- 172.21.15.195:0/3470927784 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8aa80c4160 msgr2=0x7f8aa80c4530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:52.172 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:52.166+0000 7f8aace3c700 1 --2- 172.21.15.195:0/3470927784 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8aa80c4160 0x7f8aa80c4530 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f8a9800c5c0 tx=0x7f8a98007cb0 comp rx=0 tx=0).stop 2022-06-27T12:15:52.172 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:52.166+0000 7f8aace3c700 1 -- 172.21.15.195:0/3470927784 shutdown_connections 2022-06-27T12:15:52.173 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:52.166+0000 7f8aace3c700 1 --2- 172.21.15.195:0/3470927784 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8a84063d00 0x7f8a840661b0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:52.174 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:52.166+0000 7f8aace3c700 1 --2- 172.21.15.195:0/3470927784 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8aa8153cb0 0x7f8aa80c4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:52.174 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:52.166+0000 7f8aace3c700 1 --2- 172.21.15.195:0/3470927784 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8aa806f640 0x7f8aa814f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:52.175 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:52.166+0000 7f8aace3c700 1 --2- 172.21.15.195:0/3470927784 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8aa80c4160 0x7f8aa80c4530 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:52.175 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:52.166+0000 7f8aace3c700 1 -- 172.21.15.195:0/3470927784 >> 172.21.15.195:0/3470927784 conn(0x7f8aa80c6350 msgr2=0x7f8aa806cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:15:52.175 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:52.167+0000 7f8aace3c700 1 -- 172.21.15.195:0/3470927784 shutdown_connections 2022-06-27T12:15:52.176 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:52.167+0000 7f8aace3c700 1 -- 172.21.15.195:0/3470927784 wait complete. 2022-06-27T12:15:52.176 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 105 2022-06-27T12:15:52.180 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:15:54.183 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:15:54.316 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.316+0000 7ff24fa05700 1 Processor -- start 2022-06-27T12:15:54.319 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.316+0000 7ff24fa05700 1 -- start start 2022-06-27T12:15:54.334 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.316+0000 7ff24fa05700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7ff24814e860 0x7ff2480c3840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:54.335 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.316+0000 7ff24fa05700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff2480c3d80 0x7ff248157890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:54.335 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.317+0000 7ff24fa05700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2480c33b0 con 0x7ff24814e490 2022-06-27T12:15:54.336 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.317+0000 7ff24fa05700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff24806f220 con 0x7ff24814e860 2022-06-27T12:15:54.336 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.317+0000 7ff24fa05700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2481542c0 con 0x7ff2480c3d80 2022-06-27T12:15:54.337 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.317+0000 7ff24d7a1700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7ff24814e490 0x7ff24806cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:58084/0 (socket says 172.21.15.195:58084) 2022-06-27T12:15:54.337 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.317+0000 7ff24cfa0700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7ff24814e860 0x7ff2480c3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:54.337 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.317+0000 7ff24d7a1700 1 -- 172.21.15.195:0/3279708811 learned_addr learned my addr 172.21.15.195:0/3279708811 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:15:54.338 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.317+0000 7ff24dfa2700 1 --2- 172.21.15.195:0/3279708811 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff2480c3d80 0x7ff248157890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:54.338 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.317+0000 7ff24cfa0700 1 -- 172.21.15.195:0/3279708811 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff2480c3d80 msgr2=0x7ff248157890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:15:54.338 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.317+0000 7ff24cfa0700 1 --2- 172.21.15.195:0/3279708811 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff2480c3d80 0x7ff248157890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:54.339 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.317+0000 7ff24cfa0700 1 -- 172.21.15.195:0/3279708811 >> v1:172.21.15.195:6789/0 conn(0x7ff24814e490 legacy=0x7ff24806cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:54.339 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.317+0000 7ff24cfa0700 1 -- 172.21.15.195:0/3279708811 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff248154560 con 0x7ff24814e860 2022-06-27T12:15:54.339 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.317+0000 7ff23bfff700 1 -- 172.21.15.195:0/3279708811 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3979440584 0 0) 0x7ff2480c33b0 con 0x7ff24814e490 2022-06-27T12:15:54.339 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.317+0000 7ff24cfa0700 1 --2- 172.21.15.195:0/3279708811 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff24814e860 0x7ff2480c3840 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7ff2440097a0 tx=0x7ff244009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d658e181f11a21ef server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:54.340 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.317+0000 7ff23bfff700 1 -- 172.21.15.195:0/3279708811 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff244011630 con 0x7ff24814e860 2022-06-27T12:15:54.342 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.317+0000 7ff23bfff700 1 -- 172.21.15.195:0/3279708811 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff244011790 con 0x7ff24814e860 2022-06-27T12:15:54.342 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.317+0000 7ff23bfff700 1 -- 172.21.15.195:0/3279708811 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff244011b30 con 0x7ff24814e860 2022-06-27T12:15:54.342 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.318+0000 7ff24fa05700 1 -- 172.21.15.195:0/3279708811 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff24814e860 msgr2=0x7ff2480c3840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:54.343 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.318+0000 7ff24fa05700 1 --2- 172.21.15.195:0/3279708811 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff24814e860 0x7ff2480c3840 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7ff2440097a0 tx=0x7ff244009060 comp rx=0 tx=0).stop 2022-06-27T12:15:54.343 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.318+0000 7ff24fa05700 1 -- 172.21.15.195:0/3279708811 shutdown_connections 2022-06-27T12:15:54.343 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.318+0000 7ff24fa05700 1 --2- 172.21.15.195:0/3279708811 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff2480c3d80 0x7ff248157890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:54.343 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.318+0000 7ff24fa05700 1 --2- 172.21.15.195:0/3279708811 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff24814e860 0x7ff2480c3840 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:54.344 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.318+0000 7ff24fa05700 1 -- 172.21.15.195:0/3279708811 >> 172.21.15.195:0/3279708811 conn(0x7ff248063af0 msgr2=0x7ff248063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:15:54.344 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.318+0000 7ff24fa05700 1 -- 172.21.15.195:0/3279708811 shutdown_connections 2022-06-27T12:15:54.344 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.318+0000 7ff24fa05700 1 -- 172.21.15.195:0/3279708811 wait complete. 2022-06-27T12:15:54.345 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.319+0000 7ff24fa05700 1 Processor -- start 2022-06-27T12:15:54.345 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.319+0000 7ff24fa05700 1 -- start start 2022-06-27T12:15:54.345 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.320+0000 7ff24fa05700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff2480c3d80 0x7ff248166b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:54.346 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.320+0000 7ff24fa05700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff24814e490 0x7ff248152990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:54.346 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.320+0000 7ff24fa05700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff24814e860 0x7ff248157720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:54.347 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.320+0000 7ff24fa05700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2480becf0 con 0x7ff24814e490 2022-06-27T12:15:54.348 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.320+0000 7ff24fa05700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff248157290 con 0x7ff24814e860 2022-06-27T12:15:54.348 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.320+0000 7ff24fa05700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff248068fe0 con 0x7ff2480c3d80 2022-06-27T12:15:54.348 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.320+0000 7ff24cfa0700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff24814e490 0x7ff248152990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:54.349 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.320+0000 7ff24d7a1700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff2480c3d80 0x7ff248166b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:54.349 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.320+0000 7ff24cfa0700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff24814e490 0x7ff248152990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:51904/0 (socket says 172.21.15.195:51904) 2022-06-27T12:15:54.350 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.320+0000 7ff24cfa0700 1 -- 172.21.15.195:0/3810238304 learned_addr learned my addr 172.21.15.195:0/3810238304 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:15:54.351 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.320+0000 7ff24dfa2700 1 --2- 172.21.15.195:0/3810238304 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff24814e860 0x7ff248157720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:54.352 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.320+0000 7ff24cfa0700 1 -- 172.21.15.195:0/3810238304 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff2480c3d80 msgr2=0x7ff248166b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:54.352 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.320+0000 7ff24cfa0700 1 --2- 172.21.15.195:0/3810238304 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff2480c3d80 0x7ff248166b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:54.353 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.320+0000 7ff24cfa0700 1 -- 172.21.15.195:0/3810238304 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff24814e860 msgr2=0x7ff248157720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:54.353 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.320+0000 7ff24cfa0700 1 --2- 172.21.15.195:0/3810238304 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff24814e860 0x7ff248157720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:54.354 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.320+0000 7ff24cfa0700 1 -- 172.21.15.195:0/3810238304 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2440092e0 con 0x7ff24814e490 2022-06-27T12:15:54.354 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.320+0000 7ff24d7a1700 1 --2- 172.21.15.195:0/3810238304 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff2480c3d80 0x7ff248166b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:15:54.355 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.320+0000 7ff24dfa2700 1 --2- 172.21.15.195:0/3810238304 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff24814e860 0x7ff248157720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:15:54.355 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.321+0000 7ff24cfa0700 1 --2- 172.21.15.195:0/3810238304 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff24814e490 0x7ff248152990 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7ff244009e40 tx=0x7ff244017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:54.356 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.321+0000 7ff2397fa700 1 -- 172.21.15.195:0/3810238304 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff24401be00 con 0x7ff24814e490 2022-06-27T12:15:54.356 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.321+0000 7ff2397fa700 1 -- 172.21.15.195:0/3810238304 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff244011900 con 0x7ff24814e490 2022-06-27T12:15:54.357 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.321+0000 7ff24fa05700 1 -- 172.21.15.195:0/3810238304 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff24814bb60 con 0x7ff24814e490 2022-06-27T12:15:54.357 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.321+0000 7ff2397fa700 1 -- 172.21.15.195:0/3810238304 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff244025420 con 0x7ff24814e490 2022-06-27T12:15:54.358 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.321+0000 7ff24fa05700 1 -- 172.21.15.195:0/3810238304 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff248156c40 con 0x7ff24814e490 2022-06-27T12:15:54.358 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.322+0000 7ff24fa05700 1 -- 172.21.15.195:0/3810238304 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff248156aa0 con 0x7ff24814e490 2022-06-27T12:15:54.359 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.322+0000 7ff2397fa700 1 -- 172.21.15.195:0/3810238304 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7ff24402c030 con 0x7ff24814e490 2022-06-27T12:15:54.359 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.322+0000 7ff2397fa700 1 --2- 172.21.15.195:0/3810238304 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff228063d00 0x7ff2280661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:54.359 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.323+0000 7ff2397fa700 1 -- 172.21.15.195:0/3810238304 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7578+0+0 (secure 0 0 0) 0x7ff2440821a0 con 0x7ff24814e490 2022-06-27T12:15:54.361 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.323+0000 7ff24d7a1700 1 --2- 172.21.15.195:0/3810238304 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff228063d00 0x7ff2280661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:54.361 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.323+0000 7ff24d7a1700 1 --2- 172.21.15.195:0/3810238304 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff228063d00 0x7ff2280661b0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7ff23c001670 tx=0x7ff23c007bf0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:54.361 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.327+0000 7ff2397fa700 1 -- 172.21.15.195:0/3810238304 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7ff248156aa0 con 0x7ff24814e490 2022-06-27T12:15:54.575 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.553+0000 7ff24fa05700 1 -- 172.21.15.195:0/3810238304 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff248156aa0 con 0x7ff24814e490 2022-06-27T12:15:54.576 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.553+0000 7ff2397fa700 1 -- 172.21.15.195:0/3810238304 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+5371 (secure 0 0 0) 0x7ff248156aa0 con 0x7ff24814e490 2022-06-27T12:15:54.577 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:15:54.578 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":106,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:15:52.408825+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6254":{"gid":6254,"name":"b","rank":0,"incarnation":102,"state":"up:active","state_seq":43,"addr":"172.21.15.202:6833/1317767101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1317767101},{"type":"v1","addr":"172.21.15.202:6833","nonce":1317767101}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:15:54.581 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.555+0000 7ff24fa05700 1 -- 172.21.15.195:0/3810238304 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff228063d00 msgr2=0x7ff2280661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:54.582 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.555+0000 7ff24fa05700 1 --2- 172.21.15.195:0/3810238304 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff228063d00 0x7ff2280661b0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7ff23c001670 tx=0x7ff23c007bf0 comp rx=0 tx=0).stop 2022-06-27T12:15:54.582 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.555+0000 7ff24fa05700 1 -- 172.21.15.195:0/3810238304 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff24814e490 msgr2=0x7ff248152990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:54.583 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.555+0000 7ff24fa05700 1 --2- 172.21.15.195:0/3810238304 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff24814e490 0x7ff248152990 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7ff244009e40 tx=0x7ff244017000 comp rx=0 tx=0).stop 2022-06-27T12:15:54.583 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.555+0000 7ff24fa05700 1 -- 172.21.15.195:0/3810238304 shutdown_connections 2022-06-27T12:15:54.584 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.555+0000 7ff24fa05700 1 --2- 172.21.15.195:0/3810238304 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff228063d00 0x7ff2280661b0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:54.584 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.555+0000 7ff24fa05700 1 --2- 172.21.15.195:0/3810238304 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff24814e860 0x7ff248157720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:54.585 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.555+0000 7ff24fa05700 1 --2- 172.21.15.195:0/3810238304 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff2480c3d80 0x7ff248166b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:54.585 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.555+0000 7ff24fa05700 1 --2- 172.21.15.195:0/3810238304 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff24814e490 0x7ff248152990 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:54.585 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.555+0000 7ff24fa05700 1 -- 172.21.15.195:0/3810238304 >> 172.21.15.195:0/3810238304 conn(0x7ff248063af0 msgr2=0x7ff2480ca0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:15:54.585 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.555+0000 7ff24fa05700 1 -- 172.21.15.195:0/3810238304 shutdown_connections 2022-06-27T12:15:54.586 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:54.556+0000 7ff24fa05700 1 -- 172.21.15.195:0/3810238304 wait complete. 2022-06-27T12:15:54.586 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 106 2022-06-27T12:15:54.586 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=6865 2022-06-27T12:15:54.587 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 1 secs before reviving mds.d 2022-06-27T12:15:55.587 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.d 2022-06-27T12:15:55.588 INFO:tasks.ceph.mds.d:Restarting daemon 2022-06-27T12:15:55.589 DEBUG:teuthology.orchestra.run.smithi202:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2022-06-27T12:15:55.598 INFO:tasks.ceph.mds.d:Started 2022-06-27T12:15:55.599 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:15:55.674 INFO:tasks.ceph.mds.d.smithi202.stdout:starting mds.d at 2022-06-27T12:15:55.747 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.734+0000 7fcfde11d700 1 Processor -- start 2022-06-27T12:15:55.747 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.734+0000 7fcfde11d700 1 -- start start 2022-06-27T12:15:55.748 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.735+0000 7fcfde11d700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fcfd814e490 0x7fcfd806cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:55.748 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.735+0000 7fcfde11d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fcfd80c3f50 0x7fcfd8157890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:55.748 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.735+0000 7fcfde11d700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfd80c33b0 con 0x7fcfd814e860 2022-06-27T12:15:55.749 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.735+0000 7fcfde11d700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfd806d520 con 0x7fcfd80c3f50 2022-06-27T12:15:55.749 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.735+0000 7fcfde11d700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfd806f220 con 0x7fcfd814e490 2022-06-27T12:15:55.749 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.735+0000 7fcfd7fff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fcfd80c3f50 0x7fcfd8157890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:55.750 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.735+0000 7fcfd6ffd700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fcfd814e860 0x7fcfd80c3840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:58098/0 (socket says 172.21.15.195:58098) 2022-06-27T12:15:55.750 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.735+0000 7fcfd6ffd700 1 -- 172.21.15.195:0/3685845726 learned_addr learned my addr 172.21.15.195:0/3685845726 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:15:55.750 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.735+0000 7fcfd77fe700 1 --2- 172.21.15.195:0/3685845726 >> v2:172.21.15.202:3300/0 conn(0x7fcfd814e490 0x7fcfd806cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:55.750 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.735+0000 7fcfd7fff700 1 -- 172.21.15.195:0/3685845726 >> v2:172.21.15.202:3300/0 conn(0x7fcfd814e490 msgr2=0x7fcfd806cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:15:55.751 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.735+0000 7fcfd7fff700 1 --2- 172.21.15.195:0/3685845726 >> v2:172.21.15.202:3300/0 conn(0x7fcfd814e490 0x7fcfd806cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:55.751 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.735+0000 7fcfd7fff700 1 -- 172.21.15.195:0/3685845726 >> v1:172.21.15.195:6789/0 conn(0x7fcfd814e860 legacy=0x7fcfd80c3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:55.751 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.735+0000 7fcfd67fc700 1 -- 172.21.15.195:0/3685845726 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2416599069 0 0) 0x7fcfd80c33b0 con 0x7fcfd814e860 2022-06-27T12:15:55.752 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.735+0000 7fcfd7fff700 1 -- 172.21.15.195:0/3685845726 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfd81542c0 con 0x7fcfd80c3f50 2022-06-27T12:15:55.752 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.736+0000 7fcfd7fff700 1 --2- 172.21.15.195:0/3685845726 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcfd80c3f50 0x7fcfd8157890 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7fcfd8064800 tx=0x7fcfd0009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41e3dad2a4da0e2 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:55.752 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.736+0000 7fcfd67fc700 1 -- 172.21.15.195:0/3685845726 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd000e070 con 0x7fcfd80c3f50 2022-06-27T12:15:55.753 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.736+0000 7fcfd67fc700 1 -- 172.21.15.195:0/3685845726 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfd000bb60 con 0x7fcfd80c3f50 2022-06-27T12:15:55.753 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.736+0000 7fcfd67fc700 1 -- 172.21.15.195:0/3685845726 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd0012430 con 0x7fcfd80c3f50 2022-06-27T12:15:55.754 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.736+0000 7fcfde11d700 1 -- 172.21.15.195:0/3685845726 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcfd80c3f50 msgr2=0x7fcfd8157890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:55.754 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.736+0000 7fcfde11d700 1 --2- 172.21.15.195:0/3685845726 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcfd80c3f50 0x7fcfd8157890 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7fcfd8064800 tx=0x7fcfd0009780 comp rx=0 tx=0).stop 2022-06-27T12:15:55.754 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.736+0000 7fcfde11d700 1 -- 172.21.15.195:0/3685845726 shutdown_connections 2022-06-27T12:15:55.755 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.736+0000 7fcfde11d700 1 --2- 172.21.15.195:0/3685845726 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcfd80c3f50 0x7fcfd8157890 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:55.755 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.736+0000 7fcfde11d700 1 --2- 172.21.15.195:0/3685845726 >> v2:172.21.15.202:3300/0 conn(0x7fcfd814e490 0x7fcfd806cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:55.755 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.736+0000 7fcfde11d700 1 -- 172.21.15.195:0/3685845726 >> 172.21.15.195:0/3685845726 conn(0x7fcfd8063af0 msgr2=0x7fcfd8063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:15:55.756 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.736+0000 7fcfde11d700 1 -- 172.21.15.195:0/3685845726 shutdown_connections 2022-06-27T12:15:55.757 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.736+0000 7fcfde11d700 1 -- 172.21.15.195:0/3685845726 wait complete. 2022-06-27T12:15:55.757 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.738+0000 7fcfde11d700 1 Processor -- start 2022-06-27T12:15:55.757 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.738+0000 7fcfde11d700 1 -- start start 2022-06-27T12:15:55.758 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.738+0000 7fcfde11d700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcfd80c3f50 0x7fcfd80c69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:55.758 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.738+0000 7fcfde11d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcfd814e490 0x7fcfd80c6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:55.759 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.738+0000 7fcfde11d700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcfd814e860 0x7fcfd8159320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:55.759 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.738+0000 7fcfde11d700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd80c8360 con 0x7fcfd814e490 2022-06-27T12:15:55.759 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.738+0000 7fcfde11d700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd8148590 con 0x7fcfd80c3f50 2022-06-27T12:15:55.760 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.738+0000 7fcfde11d700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd80c75a0 con 0x7fcfd814e860 2022-06-27T12:15:55.761 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.738+0000 7fcfd6ffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcfd814e490 0x7fcfd80c6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:55.761 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.738+0000 7fcfd6ffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcfd814e490 0x7fcfd80c6ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:51916/0 (socket says 172.21.15.195:51916) 2022-06-27T12:15:55.761 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.738+0000 7fcfd6ffd700 1 -- 172.21.15.195:0/2289601768 learned_addr learned my addr 172.21.15.195:0/2289601768 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:15:55.762 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.738+0000 7fcfd7fff700 1 --2- 172.21.15.195:0/2289601768 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcfd814e860 0x7fcfd8159320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:55.762 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.738+0000 7fcfd77fe700 1 --2- 172.21.15.195:0/2289601768 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcfd80c3f50 0x7fcfd80c69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:55.763 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.739+0000 7fcfd6ffd700 1 -- 172.21.15.195:0/2289601768 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcfd814e860 msgr2=0x7fcfd8159320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:55.763 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.739+0000 7fcfd6ffd700 1 --2- 172.21.15.195:0/2289601768 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcfd814e860 0x7fcfd8159320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:55.764 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.739+0000 7fcfd6ffd700 1 -- 172.21.15.195:0/2289601768 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcfd80c3f50 msgr2=0x7fcfd80c69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:55.764 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.739+0000 7fcfd6ffd700 1 --2- 172.21.15.195:0/2289601768 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcfd80c3f50 0x7fcfd80c69a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:55.765 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.739+0000 7fcfd6ffd700 1 -- 172.21.15.195:0/2289601768 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfd0009a20 con 0x7fcfd814e490 2022-06-27T12:15:55.765 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.739+0000 7fcfd6ffd700 1 --2- 172.21.15.195:0/2289601768 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcfd814e490 0x7fcfd80c6ee0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7fcfc000c490 tx=0x7fcfc0007dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:55.765 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.739+0000 7fcfcf7fe700 1 -- 172.21.15.195:0/2289601768 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfc0007150 con 0x7fcfd814e490 2022-06-27T12:15:55.766 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.739+0000 7fcfcf7fe700 1 -- 172.21.15.195:0/2289601768 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfc00072b0 con 0x7fcfd814e490 2022-06-27T12:15:55.766 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.739+0000 7fcfde11d700 1 -- 172.21.15.195:0/2289601768 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfd80c7da0 con 0x7fcfd814e490 2022-06-27T12:15:55.767 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.739+0000 7fcfcf7fe700 1 -- 172.21.15.195:0/2289601768 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfc00046f0 con 0x7fcfd814e490 2022-06-27T12:15:55.767 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.739+0000 7fcfde11d700 1 -- 172.21.15.195:0/2289601768 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfd8148220 con 0x7fcfd814e490 2022-06-27T12:15:55.768 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.740+0000 7fcfde11d700 1 -- 172.21.15.195:0/2289601768 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfd8148740 con 0x7fcfd814e490 2022-06-27T12:15:55.768 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.741+0000 7fcfcf7fe700 1 -- 172.21.15.195:0/2289601768 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fcfc0010030 con 0x7fcfd814e490 2022-06-27T12:15:55.768 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.741+0000 7fcfcf7fe700 1 --2- 172.21.15.195:0/2289601768 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcfbc063d30 0x7fcfbc0661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:55.769 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.741+0000 7fcfcf7fe700 1 -- 172.21.15.195:0/2289601768 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7578+0+0 (secure 0 0 0) 0x7fcfc00809c0 con 0x7fcfd814e490 2022-06-27T12:15:55.769 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.741+0000 7fcfd77fe700 1 --2- 172.21.15.195:0/2289601768 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcfbc063d30 0x7fcfbc0661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:55.770 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.741+0000 7fcfd77fe700 1 --2- 172.21.15.195:0/2289601768 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcfbc063d30 0x7fcfbc0661e0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8009c70 tx=0x7fcfc8009060 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:55.770 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.745+0000 7fcfcf7fe700 1 -- 172.21.15.195:0/2289601768 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fcfd8148740 con 0x7fcfd814e490 2022-06-27T12:15:55.976 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.971+0000 7fcfde11d700 1 -- 172.21.15.195:0/2289601768 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcfd8148740 con 0x7fcfd814e490 2022-06-27T12:15:55.977 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.971+0000 7fcfcf7fe700 1 -- 172.21.15.195:0/2289601768 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+5371 (secure 0 0 0) 0x7fcfd8148740 con 0x7fcfd814e490 2022-06-27T12:15:55.977 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:15:55.978 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":106,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:15:52.408825+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6254":{"gid":6254,"name":"b","rank":0,"incarnation":102,"state":"up:active","state_seq":43,"addr":"172.21.15.202:6833/1317767101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1317767101},{"type":"v1","addr":"172.21.15.202:6833","nonce":1317767101}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:15:55.982 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.973+0000 7fcfde11d700 1 -- 172.21.15.195:0/2289601768 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcfbc063d30 msgr2=0x7fcfbc0661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:55.983 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.973+0000 7fcfde11d700 1 --2- 172.21.15.195:0/2289601768 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcfbc063d30 0x7fcfbc0661e0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8009c70 tx=0x7fcfc8009060 comp rx=0 tx=0).stop 2022-06-27T12:15:55.984 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.973+0000 7fcfde11d700 1 -- 172.21.15.195:0/2289601768 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcfd814e490 msgr2=0x7fcfd80c6ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:55.984 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.973+0000 7fcfde11d700 1 --2- 172.21.15.195:0/2289601768 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcfd814e490 0x7fcfd80c6ee0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7fcfc000c490 tx=0x7fcfc0007dd0 comp rx=0 tx=0).stop 2022-06-27T12:15:55.985 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.973+0000 7fcfde11d700 1 -- 172.21.15.195:0/2289601768 shutdown_connections 2022-06-27T12:15:55.985 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.973+0000 7fcfde11d700 1 --2- 172.21.15.195:0/2289601768 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcfbc063d30 0x7fcfbc0661e0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:55.985 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.973+0000 7fcfde11d700 1 --2- 172.21.15.195:0/2289601768 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcfd814e860 0x7fcfd8159320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:55.987 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.973+0000 7fcfde11d700 1 --2- 172.21.15.195:0/2289601768 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcfd80c3f50 0x7fcfd80c69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:55.987 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.973+0000 7fcfde11d700 1 --2- 172.21.15.195:0/2289601768 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcfd814e490 0x7fcfd80c6ee0 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:55.987 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.973+0000 7fcfde11d700 1 -- 172.21.15.195:0/2289601768 >> 172.21.15.195:0/2289601768 conn(0x7fcfd8063af0 msgr2=0x7fcfd814d730 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:15:55.988 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.973+0000 7fcfde11d700 1 -- 172.21.15.195:0/2289601768 shutdown_connections 2022-06-27T12:15:55.988 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:55.973+0000 7fcfde11d700 1 -- 172.21.15.195:0/2289601768 wait complete. 2022-06-27T12:15:55.988 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 106 2022-06-27T12:15:55.990 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2022-06-27T12:15:57.993 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:15:58.133 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.119+0000 7f997e55a700 1 Processor -- start 2022-06-27T12:15:58.134 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.119+0000 7f997e55a700 1 -- start start 2022-06-27T12:15:58.134 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.120+0000 7f997e55a700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f997806e850 0x7f99781570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:58.134 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.120+0000 7f997e55a700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f99780c34a0 0x7f99780c2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:58.135 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.120+0000 7f997e55a700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f997806fba0 con 0x7f997806fef0 2022-06-27T12:15:58.135 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.120+0000 7f997e55a700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99781575e0 con 0x7f997806e850 2022-06-27T12:15:58.136 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.120+0000 7f997e55a700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f997806f5e0 con 0x7f99780c34a0 2022-06-27T12:15:58.136 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.120+0000 7f99777fe700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f997806fef0 0x7f9978069de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:58130/0 (socket says 172.21.15.195:58130) 2022-06-27T12:15:58.136 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.120+0000 7f99777fe700 1 -- 172.21.15.195:0/134605797 learned_addr learned my addr 172.21.15.195:0/134605797 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:15:58.136 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.120+0000 7f9977fff700 1 --2- 172.21.15.195:0/134605797 >> v2:172.21.15.202:3300/0 conn(0x7f997806e850 0x7f99781570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:58.137 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.120+0000 7f997caf7700 1 --2- 172.21.15.195:0/134605797 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f99780c34a0 0x7f99780c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:58.137 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.120+0000 7f9976ffd700 1 -- 172.21.15.195:0/134605797 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 53722249 0 0) 0x7f997806fba0 con 0x7f997806fef0 2022-06-27T12:15:58.137 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.120+0000 7f9976ffd700 1 -- 172.21.15.195:0/134605797 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9964003430 con 0x7f997806fef0 2022-06-27T12:15:58.138 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.120+0000 7f997caf7700 1 -- 172.21.15.195:0/134605797 >> v2:172.21.15.202:3300/0 conn(0x7f997806e850 msgr2=0x7f99781570a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:15:58.138 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.120+0000 7f997caf7700 1 --2- 172.21.15.195:0/134605797 >> v2:172.21.15.202:3300/0 conn(0x7f997806e850 0x7f99781570a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:58.138 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.121+0000 7f997caf7700 1 -- 172.21.15.195:0/134605797 >> v1:172.21.15.195:6789/0 conn(0x7f997806fef0 legacy=0x7f9978069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:58.138 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.121+0000 7f997caf7700 1 -- 172.21.15.195:0/134605797 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f997806f8c0 con 0x7f99780c34a0 2022-06-27T12:15:58.139 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.121+0000 7f9977fff700 1 --2- 172.21.15.195:0/134605797 >> v2:172.21.15.202:3300/0 conn(0x7f997806e850 0x7f99781570a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:15:58.139 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.121+0000 7f997caf7700 1 --2- 172.21.15.195:0/134605797 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f99780c34a0 0x7f99780c2920 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f9970009e60 tx=0x7f9970009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87f62fa5478d8bb3 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:58.140 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.121+0000 7f9976ffd700 1 -- 172.21.15.195:0/134605797 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f997000e070 con 0x7f99780c34a0 2022-06-27T12:15:58.140 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.121+0000 7f9976ffd700 1 -- 172.21.15.195:0/134605797 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f997000bb40 con 0x7f99780c34a0 2022-06-27T12:15:58.140 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.121+0000 7f9976ffd700 1 -- 172.21.15.195:0/134605797 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9970012430 con 0x7f99780c34a0 2022-06-27T12:15:58.140 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.121+0000 7f997e55a700 1 -- 172.21.15.195:0/134605797 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f99780c34a0 msgr2=0x7f99780c2920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:58.140 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.121+0000 7f997e55a700 1 --2- 172.21.15.195:0/134605797 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f99780c34a0 0x7f99780c2920 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f9970009e60 tx=0x7f9970009780 comp rx=0 tx=0).stop 2022-06-27T12:15:58.141 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.121+0000 7f997e55a700 1 -- 172.21.15.195:0/134605797 shutdown_connections 2022-06-27T12:15:58.141 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.121+0000 7f997e55a700 1 --2- 172.21.15.195:0/134605797 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f99780c34a0 0x7f99780c2920 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:58.141 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.121+0000 7f997e55a700 1 --2- 172.21.15.195:0/134605797 >> v2:172.21.15.202:3300/0 conn(0x7f997806e850 0x7f99781570a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:58.141 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.121+0000 7f997e55a700 1 -- 172.21.15.195:0/134605797 >> 172.21.15.195:0/134605797 conn(0x7f99780c6350 msgr2=0x7f99780c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:15:58.143 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.121+0000 7f997e55a700 1 -- 172.21.15.195:0/134605797 shutdown_connections 2022-06-27T12:15:58.143 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.121+0000 7f997e55a700 1 -- 172.21.15.195:0/134605797 wait complete. 2022-06-27T12:15:58.144 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.122+0000 7f997e55a700 1 Processor -- start 2022-06-27T12:15:58.145 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.123+0000 7f997e55a700 1 -- start start 2022-06-27T12:15:58.146 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.123+0000 7f997e55a700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f997806e850 0x7f99780bf2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:58.147 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.123+0000 7f997e55a700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f99780c34a0 0x7f9978155720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:58.147 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.123+0000 7f997e55a700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f99780bc9a0 0x7f997814f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:58.148 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.123+0000 7f997e55a700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99780bf900 con 0x7f997806e850 2022-06-27T12:15:58.149 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.123+0000 7f997e55a700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99781674b0 con 0x7f99780c34a0 2022-06-27T12:15:58.149 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.124+0000 7f997e55a700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9978164970 con 0x7f99780bc9a0 2022-06-27T12:15:58.151 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.124+0000 7f9977fff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f997806e850 0x7f99780bf2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:58.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.124+0000 7f9977fff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f997806e850 0x7f99780bf2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:51946/0 (socket says 172.21.15.195:51946) 2022-06-27T12:15:58.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.124+0000 7f9977fff700 1 -- 172.21.15.195:0/643913732 learned_addr learned my addr 172.21.15.195:0/643913732 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:15:58.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.124+0000 7f99777fe700 1 --2- 172.21.15.195:0/643913732 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f99780c34a0 0x7f9978155720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:58.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.124+0000 7f997caf7700 1 --2- 172.21.15.195:0/643913732 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f99780bc9a0 0x7f997814f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:58.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.124+0000 7f9977fff700 1 -- 172.21.15.195:0/643913732 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f99780bc9a0 msgr2=0x7f997814f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:58.157 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.124+0000 7f9977fff700 1 --2- 172.21.15.195:0/643913732 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f99780bc9a0 0x7f997814f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:58.158 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.124+0000 7f9977fff700 1 -- 172.21.15.195:0/643913732 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f99780c34a0 msgr2=0x7f9978155720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:58.158 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.124+0000 7f9977fff700 1 --2- 172.21.15.195:0/643913732 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f99780c34a0 0x7f9978155720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:58.158 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.124+0000 7f9977fff700 1 -- 172.21.15.195:0/643913732 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f996c0092e0 con 0x7f997806e850 2022-06-27T12:15:58.159 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.124+0000 7f9977fff700 1 --2- 172.21.15.195:0/643913732 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f997806e850 0x7f99780bf2d0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f996c00ea80 tx=0x7f996c009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:58.159 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.124+0000 7f995ffff700 1 -- 172.21.15.195:0/643913732 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f996c00ee60 con 0x7f997806e850 2022-06-27T12:15:58.160 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.124+0000 7f995ffff700 1 -- 172.21.15.195:0/643913732 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f996c00c850 con 0x7f997806e850 2022-06-27T12:15:58.160 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.124+0000 7f997e55a700 1 -- 172.21.15.195:0/643913732 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9970009a20 con 0x7f997806e850 2022-06-27T12:15:58.161 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.124+0000 7f995ffff700 1 -- 172.21.15.195:0/643913732 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f996c00cbd0 con 0x7f997806e850 2022-06-27T12:15:58.162 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.124+0000 7f997e55a700 1 -- 172.21.15.195:0/643913732 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99780bfb50 con 0x7f997806e850 2022-06-27T12:15:58.163 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.125+0000 7f997e55a700 1 -- 172.21.15.195:0/643913732 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99780722c0 con 0x7f997806e850 2022-06-27T12:15:58.163 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.126+0000 7f995ffff700 1 -- 172.21.15.195:0/643913732 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f996c023070 con 0x7f997806e850 2022-06-27T12:15:58.163 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.126+0000 7f995ffff700 1 --2- 172.21.15.195:0/643913732 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9958063d00 0x7f99580661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:15:58.164 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.126+0000 7f99777fe700 1 --2- 172.21.15.195:0/643913732 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9958063d00 0x7f99580661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:15:58.164 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.126+0000 7f995ffff700 1 -- 172.21.15.195:0/643913732 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7578+0+0 (secure 0 0 0) 0x7f996c081160 con 0x7f997806e850 2022-06-27T12:15:58.164 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.127+0000 7f99777fe700 1 --2- 172.21.15.195:0/643913732 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9958063d00 0x7f99580661b0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f99780c0810 tx=0x7f9968009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:15:58.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.131+0000 7f995ffff700 1 -- 172.21.15.195:0/643913732 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f996c04f700 con 0x7f997806e850 2022-06-27T12:15:58.365 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.363+0000 7f997e55a700 1 -- 172.21.15.195:0/643913732 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9978167790 con 0x7f997806e850 2022-06-27T12:15:58.366 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.363+0000 7f995ffff700 1 -- 172.21.15.195:0/643913732 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+6132 (secure 0 0 0) 0x7f996c017090 con 0x7f997806e850 2022-06-27T12:15:58.366 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:15:58.367 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":107,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":7165,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/3551478822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":3551478822},{"type":"v1","addr":"172.21.15.202:6835","nonce":3551478822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":107}],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:15:52.408825+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6254":{"gid":6254,"name":"b","rank":0,"incarnation":102,"state":"up:active","state_seq":43,"addr":"172.21.15.202:6833/1317767101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1317767101},{"type":"v1","addr":"172.21.15.202:6833","nonce":1317767101}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:15:58.370 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.365+0000 7f997e55a700 1 -- 172.21.15.195:0/643913732 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9958063d00 msgr2=0x7f99580661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:58.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.365+0000 7f997e55a700 1 --2- 172.21.15.195:0/643913732 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9958063d00 0x7f99580661b0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f99780c0810 tx=0x7f9968009040 comp rx=0 tx=0).stop 2022-06-27T12:15:58.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.365+0000 7f997e55a700 1 -- 172.21.15.195:0/643913732 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f997806e850 msgr2=0x7f99780bf2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:15:58.372 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.365+0000 7f997e55a700 1 --2- 172.21.15.195:0/643913732 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f997806e850 0x7f99780bf2d0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f996c00ea80 tx=0x7f996c009060 comp rx=0 tx=0).stop 2022-06-27T12:15:58.372 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.366+0000 7f997e55a700 1 -- 172.21.15.195:0/643913732 shutdown_connections 2022-06-27T12:15:58.373 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.366+0000 7f997e55a700 1 --2- 172.21.15.195:0/643913732 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9958063d00 0x7f99580661b0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:58.374 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.366+0000 7f997e55a700 1 --2- 172.21.15.195:0/643913732 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f997806e850 0x7f99780bf2d0 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:58.375 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.366+0000 7f997e55a700 1 --2- 172.21.15.195:0/643913732 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f99780bc9a0 0x7f997814f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:58.375 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.366+0000 7f997e55a700 1 --2- 172.21.15.195:0/643913732 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f99780c34a0 0x7f9978155720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:15:58.388 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.366+0000 7f997e55a700 1 -- 172.21.15.195:0/643913732 >> 172.21.15.195:0/643913732 conn(0x7f99780c6350 msgr2=0x7f99781506e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:15:58.389 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.366+0000 7f997e55a700 1 -- 172.21.15.195:0/643913732 shutdown_connections 2022-06-27T12:15:58.389 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:15:58.366+0000 7f997e55a700 1 -- 172.21.15.195:0/643913732 wait complete. 2022-06-27T12:15:58.390 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 107 2022-06-27T12:15:58.392 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d reported in up:standby state 2022-06-27T12:15:58.420 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 46 secs before thrashing 2022-06-27T12:16:01.349 INFO:tasks.ceph.mds.b.smithi202.stderr:2022-06-27T12:16:01.339+0000 7f02dc784700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2022-06-27T12:16:04.399 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 11% (40/355) Updating files: 12% (43/355) Updating files: 13% (47/355) Updating files: 14% (50/355) Updating files: 15% (54/355) Updating files: 16% (57/355) Updating files: 17% (61/355) Updating files: 18% (64/355) Updating files: 19% (68/355) Updating files: 20% (71/355) Updating files: 21% (75/355) Updating files: 22% (79/355) Updating files: 23% (82/355) Updating files: 24% (86/355) Updating files: 24% (87/355) Updating files: 25% (89/355) Updating files: 26% (93/355) Updating files: 27% (96/355) Updating files: 28% (100/355) Updating files: 29% (103/355) Updating files: 29% (104/355) Updating files: 30% (107/355) Updating files: 31% (111/355) Updating files: 32% (114/355) Updating files: 33% (118/355) Updating files: 34% (121/355) Updating files: 35% (125/355) Updating files: 36% (128/355) Updating files: 37% (132/355) Updating files: 38% (135/355) Updating files: 39% (139/355) Updating files: 40% (142/355) Updating files: 41% (146/355) Updating files: 42% (150/355) Updating files: 43% (153/355) Updating files: 44% (157/355) Updating files: 45% (160/355) Updating files: 46% (164/355) Updating files: 47% (167/355) Updating files: 48% (171/355) Updating files: 49% (174/355) Updating files: 50% (178/355) Updating files: 51% (182/355) Updating files: 52% (185/355) Updating files: 53% (189/355) Updating files: 54% (192/355) Updating files: 55% (196/355) Updating files: 55% (197/355) Updating files: 56% (199/355) Updating files: 57% (203/355) Updating files: 57% (205/355) Updating files: 58% (206/355) Updating files: 59% (210/355) Updating files: 60% (213/355) Updating files: 61% (217/355) Updating files: 62% (221/355) Updating files: 62% (223/355) Updating files: 63% (224/355) Updating files: 64% (228/355) Updating files: 65% (231/355) Updating files: 66% (235/355) Updating files: 67% (238/355) Updating files: 68% (242/355) Updating files: 69% (245/355) Updating files: 70% (249/355) Updating files: 71% (253/355) Updating files: 72% (256/355) Updating files: 73% (260/355) Updating files: 74% (263/355) Updating files: 75% (267/355) Updating files: 76% (270/355) Updating files: 77% (274/355) Updating files: 78% (277/355) Updating files: 79% (281/355) Updating files: 80% (284/355) Updating files: 81% (288/355) Updating files: 82% (292/355) Updating files: 83% (295/355) Updating files: 84% (299/355) Updating files: 85% (302/355) Updating files: 85% (305/355) Updating files: 86% (306/355) Updating files: 87% (309/355) Updating files: 88% (313/355) Updating files: 89% (316/355) Updating files: 90% (320/355) Updating files: 91% (324/355) Updating files: 92% (327/355) Updating files: 93% (331/355) Updating files: 94% (334/355) Updating files: 95% (338/355) Updating files: 96% (341/355) Updating files: 97% (345/355) Updating files: 98% (348/355) Updating files: 99% (352/355) Updating files: 100% (355/355) Updating files: 100% (355/355), done. 2022-06-27T12:16:04.603 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:16:04.623 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:16:04.625 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at e3b7bc5bce8 v0.67 2022-06-27T12:16:04.626 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.67 2022-06-27T12:16:04.673 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:16:04.677 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 68 -eq 48 ']' 2022-06-27T12:16:04.678 INFO:tasks.workunit.client.0.smithi195.stdout:v0.68 2022-06-27T12:16:04.679 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.68 2022-06-27T12:16:04.679 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.68 2022-06-27T12:16:04.679 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.68 2022-06-27T12:16:18.166 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 16% (43/265) Updating files: 16% (44/265) Updating files: 17% (46/265) Updating files: 18% (48/265) Updating files: 19% (51/265) Updating files: 20% (53/265) Updating files: 21% (56/265) Updating files: 22% (59/265) Updating files: 23% (61/265) Updating files: 24% (64/265) Updating files: 25% (67/265) Updating files: 26% (69/265) Updating files: 27% (72/265) Updating files: 28% (75/265) Updating files: 29% (77/265) Updating files: 30% (80/265) Updating files: 31% (83/265) Updating files: 32% (85/265) Updating files: 33% (88/265) Updating files: 34% (91/265) Updating files: 35% (93/265) Updating files: 36% (96/265) Updating files: 36% (97/265) Updating files: 37% (99/265) Updating files: 38% (101/265) Updating files: 39% (104/265) Updating files: 40% (106/265) Updating files: 41% (109/265) Updating files: 42% (112/265) Updating files: 43% (114/265) Updating files: 44% (117/265) Updating files: 45% (120/265) Updating files: 46% (122/265) Updating files: 47% (125/265) Updating files: 48% (128/265) Updating files: 49% (130/265) Updating files: 50% (133/265) Updating files: 51% (136/265) Updating files: 52% (138/265) Updating files: 53% (141/265) Updating files: 54% (144/265) Updating files: 55% (146/265) Updating files: 56% (149/265) Updating files: 57% (152/265) Updating files: 58% (154/265) Updating files: 59% (157/265) Updating files: 60% (159/265) Updating files: 61% (162/265) Updating files: 62% (165/265) Updating files: 63% (167/265) Updating files: 64% (170/265) Updating files: 65% (173/265) Updating files: 66% (175/265) Updating files: 67% (178/265) Updating files: 68% (181/265) Updating files: 69% (183/265) Updating files: 70% (186/265) Updating files: 71% (189/265) Updating files: 72% (191/265) Updating files: 73% (194/265) Updating files: 74% (197/265) Updating files: 75% (199/265) Updating files: 76% (202/265) Updating files: 77% (205/265) Updating files: 78% (207/265) Updating files: 79% (210/265) Updating files: 80% (212/265) Updating files: 81% (215/265) Updating files: 82% (218/265) Updating files: 83% (220/265) Updating files: 84% (223/265) Updating files: 85% (226/265) Updating files: 86% (228/265) Updating files: 87% (231/265) Updating files: 88% (234/265) Updating files: 89% (236/265) Updating files: 89% (237/265) Updating files: 90% (239/265) Updating files: 91% (242/265) Updating files: 92% (244/265) Updating files: 93% (247/265) Updating files: 94% (250/265) Updating files: 95% (252/265) Updating files: 96% (255/265) Updating files: 97% (258/265) Updating files: 98% (260/265) Updating files: 99% (263/265) Updating files: 100% (265/265) Updating files: 100% (265/265), done. 2022-06-27T12:16:18.192 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at b4cf0f2574b v0.68 2022-06-27T12:16:18.193 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.68 2022-06-27T12:16:18.268 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:16:18.268 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 69 -eq 48 ']' 2022-06-27T12:16:18.268 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.69 2022-06-27T12:16:18.269 INFO:tasks.workunit.client.0.smithi195.stdout:v0.69 2022-06-27T12:16:18.270 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.69 2022-06-27T12:16:18.270 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.69 2022-06-27T12:16:20.150 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 49% (77/155) Updating files: 50% (78/155) Updating files: 51% (80/155) Updating files: 52% (81/155) Updating files: 53% (83/155) Updating files: 54% (84/155) Updating files: 55% (86/155) Updating files: 56% (87/155) Updating files: 57% (89/155) Updating files: 58% (90/155) Updating files: 59% (92/155) Updating files: 60% (93/155) Updating files: 61% (95/155) Updating files: 62% (97/155) Updating files: 63% (98/155) Updating files: 64% (100/155) Updating files: 65% (101/155) Updating files: 66% (103/155) Updating files: 67% (104/155) Updating files: 68% (106/155) Updating files: 69% (107/155) Updating files: 70% (109/155) Updating files: 71% (111/155) Updating files: 72% (112/155) Updating files: 73% (114/155) Updating files: 74% (115/155) Updating files: 75% (117/155) Updating files: 76% (118/155) Updating files: 77% (120/155) Updating files: 78% (121/155) Updating files: 79% (123/155) Updating files: 80% (124/155) Updating files: 81% (126/155) Updating files: 82% (128/155) Updating files: 83% (129/155) Updating files: 84% (131/155) Updating files: 85% (132/155) Updating files: 86% (134/155) Updating files: 87% (135/155) Updating files: 88% (137/155) Updating files: 89% (138/155) Updating files: 90% (140/155) Updating files: 91% (142/155) Updating files: 92% (143/155) Updating files: 93% (145/155) Updating files: 94% (146/155) Updating files: 95% (148/155) Updating files: 96% (149/155) Updating files: 97% (151/155) Updating files: 98% (152/155) Updating files: 99% (154/155) Updating files: 100% (155/155) Updating files: 100% (155/155), done. 2022-06-27T12:16:20.176 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at 6ca6f2f9f75 v0.69 2022-06-27T12:16:20.180 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.69 2022-06-27T12:16:20.299 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:16:20.299 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 70 -eq 48 ']' 2022-06-27T12:16:20.300 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.70 2022-06-27T12:16:20.300 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.70 2022-06-27T12:16:20.301 INFO:tasks.workunit.client.0.smithi195.stdout:v0.70 2022-06-27T12:16:20.302 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.70 2022-06-27T12:16:20.517 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at e3bb0656d92 v0.70 2022-06-27T12:16:20.520 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.70 2022-06-27T12:16:20.632 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:16:20.633 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 71 -eq 48 ']' 2022-06-27T12:16:20.634 INFO:tasks.workunit.client.0.smithi195.stdout:v0.71 2022-06-27T12:16:20.635 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.71 2022-06-27T12:16:20.636 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.71 2022-06-27T12:16:20.636 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.71 2022-06-27T12:16:25.781 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 13% (54/405) Updating files: 14% (57/405) Updating files: 15% (61/405) Updating files: 16% (65/405) Updating files: 17% (69/405) Updating files: 18% (73/405) Updating files: 19% (77/405) Updating files: 20% (81/405) Updating files: 21% (86/405) Updating files: 22% (90/405) Updating files: 23% (94/405) Updating files: 24% (98/405) Updating files: 25% (102/405) Updating files: 26% (106/405) Updating files: 27% (110/405) Updating files: 28% (114/405) Updating files: 29% (118/405) Updating files: 30% (122/405) Updating files: 31% (126/405) Updating files: 32% (130/405) Updating files: 33% (134/405) Updating files: 34% (138/405) Updating files: 35% (142/405) Updating files: 36% (146/405) Updating files: 37% (150/405) Updating files: 38% (154/405) Updating files: 39% (158/405) Updating files: 40% (162/405) Updating files: 41% (167/405) Updating files: 42% (171/405) Updating files: 43% (175/405) Updating files: 44% (179/405) Updating files: 45% (183/405) Updating files: 46% (187/405) Updating files: 47% (191/405) Updating files: 48% (195/405) Updating files: 49% (199/405) Updating files: 50% (203/405) Updating files: 51% (207/405) Updating files: 52% (211/405) Updating files: 53% (215/405) Updating files: 53% (216/405) Updating files: 54% (219/405) Updating files: 55% (223/405) Updating files: 56% (227/405) Updating files: 57% (231/405) Updating files: 58% (235/405) Updating files: 59% (239/405) Updating files: 60% (243/405) Updating files: 61% (248/405) Updating files: 62% (252/405) Updating files: 63% (256/405) Updating files: 64% (260/405) Updating files: 65% (264/405) Updating files: 65% (265/405) Updating files: 66% (268/405) Updating files: 67% (272/405) Updating files: 67% (274/405) Updating files: 68% (276/405) Updating files: 69% (280/405) Updating files: 70% (284/405) Updating files: 71% (288/405) Updating files: 72% (292/405) Updating files: 73% (296/405) Updating files: 74% (300/405) Updating files: 75% (304/405) Updating files: 76% (308/405) Updating files: 77% (312/405) Updating files: 78% (316/405) Updating files: 79% (320/405) Updating files: 80% (324/405) Updating files: 81% (329/405) Updating files: 82% (333/405) Updating files: 83% (337/405) Updating files: 84% (341/405) Updating files: 85% (345/405) Updating files: 86% (349/405) Updating files: 87% (353/405) Updating files: 88% (357/405) Updating files: 89% (361/405) Updating files: 90% (365/405) Updating files: 91% (369/405) Updating files: 92% (373/405) Updating files: 93% (377/405) Updating files: 94% (381/405) Updating files: 95% (385/405) Updating files: 96% (389/405) Updating files: 97% (393/405) Updating files: 98% (397/405) Updating files: 99% (401/405) Updating files: 100% (405/405) Updating files: 100% (405/405), done. 2022-06-27T12:16:26.277 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at e509cb1e69c v0.71 2022-06-27T12:16:26.277 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.71 2022-06-27T12:16:27.089 INFO:tasks.workunit.client.0.smithi195.stdout:v0.72 2022-06-27T12:16:27.090 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:16:27.090 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 72 -eq 48 ']' 2022-06-27T12:16:27.091 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.72 2022-06-27T12:16:27.091 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.72 2022-06-27T12:16:27.092 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.72 2022-06-27T12:16:31.810 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 45% (87/191) Updating files: 46% (88/191) Updating files: 47% (90/191) Updating files: 48% (92/191) Updating files: 49% (94/191) Updating files: 50% (96/191) Updating files: 51% (98/191) Updating files: 52% (100/191) Updating files: 53% (102/191) Updating files: 54% (104/191) Updating files: 55% (106/191) Updating files: 56% (107/191) Updating files: 57% (109/191) Updating files: 58% (111/191) Updating files: 59% (113/191) Updating files: 60% (115/191) Updating files: 61% (117/191) Updating files: 62% (119/191) Updating files: 63% (121/191) Updating files: 64% (123/191) Updating files: 65% (125/191) Updating files: 66% (127/191) Updating files: 67% (128/191) Updating files: 68% (130/191) Updating files: 69% (132/191) Updating files: 70% (134/191) Updating files: 71% (136/191) Updating files: 72% (138/191) Updating files: 73% (140/191) Updating files: 74% (142/191) Updating files: 75% (144/191) Updating files: 76% (146/191) Updating files: 77% (148/191) Updating files: 78% (149/191) Updating files: 79% (151/191) Updating files: 80% (153/191) Updating files: 81% (155/191) Updating files: 82% (157/191) Updating files: 83% (159/191) Updating files: 84% (161/191) Updating files: 85% (163/191) Updating files: 86% (165/191) Updating files: 87% (167/191) Updating files: 88% (169/191) Updating files: 89% (170/191) Updating files: 90% (172/191) Updating files: 91% (174/191) Updating files: 92% (176/191) Updating files: 93% (178/191) Updating files: 94% (180/191) Updating files: 95% (182/191) Updating files: 96% (184/191) Updating files: 97% (186/191) Updating files: 98% (188/191) Updating files: 99% (190/191) Updating files: 100% (191/191) Updating files: 100% (191/191), done. 2022-06-27T12:16:31.838 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at 5832e2603c7 v0.72 2022-06-27T12:16:31.843 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.72 2022-06-27T12:16:31.977 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:16:31.978 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 73 -eq 48 ']' 2022-06-27T12:16:31.979 INFO:tasks.workunit.client.0.smithi195.stdout:v0.73 2022-06-27T12:16:31.979 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.73 2022-06-27T12:16:31.980 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.73 2022-06-27T12:16:31.980 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.73 2022-06-27T12:16:33.298 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 57% (85/148) Updating files: 58% (86/148) Updating files: 59% (88/148) Updating files: 60% (89/148) Updating files: 61% (91/148) Updating files: 62% (92/148) Updating files: 63% (94/148) Updating files: 64% (95/148) Updating files: 65% (97/148) Updating files: 66% (98/148) Updating files: 67% (100/148) Updating files: 68% (101/148) Updating files: 69% (103/148) Updating files: 70% (104/148) Updating files: 71% (106/148) Updating files: 72% (107/148) Updating files: 73% (109/148) Updating files: 74% (110/148) Updating files: 75% (111/148) Updating files: 76% (113/148) Updating files: 77% (114/148) Updating files: 78% (116/148) Updating files: 79% (117/148) Updating files: 80% (119/148) Updating files: 81% (120/148) Updating files: 82% (122/148) Updating files: 83% (123/148) Updating files: 84% (125/148) Updating files: 85% (126/148) Updating files: 86% (128/148) Updating files: 87% (129/148) Updating files: 88% (131/148) Updating files: 89% (132/148) Updating files: 90% (134/148) Updating files: 91% (135/148) Updating files: 92% (137/148) Updating files: 93% (138/148) Updating files: 94% (140/148) Updating files: 95% (141/148) Updating files: 96% (143/148) Updating files: 97% (144/148) Updating files: 98% (146/148) Updating files: 99% (147/148) Updating files: 100% (148/148) Updating files: 100% (148/148), done. 2022-06-27T12:16:33.627 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at d8ad51ee8ab v0.73 2022-06-27T12:16:33.628 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.73 2022-06-27T12:16:33.716 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:16:33.717 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 74 -eq 48 ']' 2022-06-27T12:16:33.717 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.74 2022-06-27T12:16:33.726 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.74 2022-06-27T12:16:33.727 INFO:tasks.workunit.client.0.smithi195.stdout:v0.74 2022-06-27T12:16:33.728 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.74 2022-06-27T12:16:34.699 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:16:34.703 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:16:36.668 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 26% (50/192) Updating files: 27% (52/192) Updating files: 28% (54/192) Updating files: 29% (56/192) Updating files: 30% (58/192) Updating files: 31% (60/192) Updating files: 32% (62/192) Updating files: 33% (64/192) Updating files: 34% (66/192) Updating files: 35% (68/192) Updating files: 36% (70/192) Updating files: 37% (72/192) Updating files: 38% (73/192) Updating files: 39% (75/192) Updating files: 40% (77/192) Updating files: 41% (79/192) Updating files: 42% (81/192) Updating files: 43% (83/192) Updating files: 44% (85/192) Updating files: 45% (87/192) Updating files: 46% (89/192) Updating files: 47% (91/192) Updating files: 48% (93/192) Updating files: 49% (95/192) Updating files: 50% (96/192) Updating files: 51% (98/192) Updating files: 52% (100/192) Updating files: 53% (102/192) Updating files: 54% (104/192) Updating files: 55% (106/192) Updating files: 56% (108/192) Updating files: 57% (110/192) Updating files: 58% (112/192) Updating files: 59% (114/192) Updating files: 60% (116/192) Updating files: 61% (118/192) Updating files: 62% (120/192) Updating files: 63% (121/192) Updating files: 64% (123/192) Updating files: 65% (125/192) Updating files: 66% (127/192) Updating files: 67% (129/192) Updating files: 68% (131/192) Updating files: 69% (133/192) Updating files: 70% (135/192) Updating files: 71% (137/192) Updating files: 72% (139/192) Updating files: 73% (141/192) Updating files: 74% (143/192) Updating files: 75% (144/192) Updating files: 76% (146/192) Updating files: 77% (148/192) Updating files: 78% (150/192) Updating files: 79% (152/192) Updating files: 80% (154/192) Updating files: 81% (156/192) Updating files: 82% (158/192) Updating files: 83% (160/192) Updating files: 84% (162/192) Updating files: 85% (164/192) Updating files: 86% (166/192) Updating files: 87% (168/192) Updating files: 88% (169/192) Updating files: 89% (171/192) Updating files: 90% (173/192) Updating files: 91% (175/192) Updating files: 92% (177/192) Updating files: 93% (179/192) Updating files: 94% (181/192) Updating files: 95% (183/192) Updating files: 96% (185/192) Updating files: 97% (187/192) Updating files: 98% (189/192) Updating files: 99% (191/192) Updating files: 100% (192/192) Updating files: 100% (192/192), done. 2022-06-27T12:16:36.982 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at c165483bc72 v0.74 2022-06-27T12:16:36.983 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.74 2022-06-27T12:16:37.085 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:16:37.086 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 75 -eq 48 ']' 2022-06-27T12:16:37.087 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.75 2022-06-27T12:16:37.087 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.75 2022-06-27T12:16:37.088 INFO:tasks.workunit.client.0.smithi195.stdout:v0.75 2022-06-27T12:16:37.089 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.75 2022-06-27T12:16:40.290 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 29% (87/291) Updating files: 30% (88/291) Updating files: 31% (91/291) Updating files: 32% (94/291) Updating files: 33% (97/291) Updating files: 34% (99/291) Updating files: 35% (102/291) Updating files: 36% (105/291) Updating files: 37% (108/291) Updating files: 38% (111/291) Updating files: 39% (114/291) Updating files: 40% (117/291) Updating files: 41% (120/291) Updating files: 42% (123/291) Updating files: 43% (126/291) Updating files: 44% (129/291) Updating files: 45% (131/291) Updating files: 46% (134/291) Updating files: 47% (137/291) Updating files: 48% (140/291) Updating files: 49% (143/291) Updating files: 50% (146/291) Updating files: 51% (149/291) Updating files: 52% (152/291) Updating files: 53% (155/291) Updating files: 54% (158/291) Updating files: 55% (161/291) Updating files: 56% (163/291) Updating files: 57% (166/291) Updating files: 58% (169/291) Updating files: 59% (172/291) Updating files: 60% (175/291) Updating files: 61% (178/291) Updating files: 62% (181/291) Updating files: 63% (184/291) Updating files: 64% (187/291) Updating files: 65% (190/291) Updating files: 66% (193/291) Updating files: 67% (195/291) Updating files: 68% (198/291) Updating files: 69% (201/291) Updating files: 70% (204/291) Updating files: 71% (207/291) Updating files: 72% (210/291) Updating files: 73% (213/291) Updating files: 74% (216/291) Updating files: 75% (219/291) Updating files: 76% (222/291) Updating files: 77% (225/291) Updating files: 78% (227/291) Updating files: 79% (230/291) Updating files: 80% (233/291) Updating files: 81% (236/291) Updating files: 82% (239/291) Updating files: 82% (241/291) Updating files: 83% (242/291) Updating files: 84% (245/291) Updating files: 85% (248/291) Updating files: 86% (251/291) Updating files: 87% (254/291) Updating files: 88% (257/291) Updating files: 89% (259/291) Updating files: 90% (262/291) Updating files: 91% (265/291) Updating files: 92% (268/291) Updating files: 93% (271/291) Updating files: 94% (274/291) Updating files: 95% (277/291) Updating files: 96% (280/291) Updating files: 97% (283/291) Updating files: 98% (286/291) Updating files: 99% (289/291) Updating files: 100% (291/291) Updating files: 100% (291/291), done. 2022-06-27T12:16:40.325 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at 946d6036958 v0.75 2022-06-27T12:16:40.327 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.75 2022-06-27T12:16:40.476 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:16:40.477 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 76 -eq 48 ']' 2022-06-27T12:16:40.477 INFO:tasks.workunit.client.0.smithi195.stdout:v0.76 2022-06-27T12:16:40.478 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.76 2022-06-27T12:16:40.478 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.76 2022-06-27T12:16:40.499 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.76 2022-06-27T12:16:43.596 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 31% (53/168) Updating files: 32% (54/168) Updating files: 33% (56/168) Updating files: 34% (58/168) Updating files: 35% (59/168) Updating files: 36% (61/168) Updating files: 37% (63/168) Updating files: 38% (64/168) Updating files: 39% (66/168) Updating files: 40% (68/168) Updating files: 41% (69/168) Updating files: 42% (71/168) Updating files: 43% (73/168) Updating files: 44% (74/168) Updating files: 45% (76/168) Updating files: 46% (78/168) Updating files: 47% (79/168) Updating files: 48% (81/168) Updating files: 49% (83/168) Updating files: 50% (84/168) Updating files: 51% (86/168) Updating files: 52% (88/168) Updating files: 53% (90/168) Updating files: 54% (91/168) Updating files: 55% (93/168) Updating files: 56% (95/168) Updating files: 57% (96/168) Updating files: 58% (98/168) Updating files: 59% (100/168) Updating files: 60% (101/168) Updating files: 60% (102/168) Updating files: 61% (103/168) Updating files: 62% (105/168) Updating files: 63% (106/168) Updating files: 64% (108/168) Updating files: 65% (110/168) Updating files: 66% (111/168) Updating files: 67% (113/168) Updating files: 68% (115/168) Updating files: 69% (116/168) Updating files: 70% (118/168) Updating files: 71% (120/168) Updating files: 72% (121/168) Updating files: 73% (123/168) Updating files: 74% (125/168) Updating files: 75% (126/168) Updating files: 76% (128/168) Updating files: 77% (130/168) Updating files: 78% (132/168) Updating files: 79% (133/168) Updating files: 80% (135/168) Updating files: 81% (137/168) Updating files: 82% (138/168) Updating files: 83% (140/168) Updating files: 84% (142/168) Updating files: 85% (143/168) Updating files: 86% (145/168) Updating files: 87% (147/168) Updating files: 88% (148/168) Updating files: 89% (150/168) Updating files: 90% (152/168) Updating files: 91% (153/168) Updating files: 92% (155/168) Updating files: 93% (157/168) Updating files: 94% (158/168) Updating files: 95% (160/168) Updating files: 96% (162/168) Updating files: 97% (163/168) Updating files: 98% (165/168) Updating files: 99% (167/168) Updating files: 100% (168/168) Updating files: 100% (168/168), done. 2022-06-27T12:16:43.654 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at 3b990136bfa v0.76 2022-06-27T12:16:43.655 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.76 2022-06-27T12:16:43.742 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:16:43.742 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 77 -eq 48 ']' 2022-06-27T12:16:43.743 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.77 2022-06-27T12:16:43.743 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.77 2022-06-27T12:16:43.751 INFO:tasks.workunit.client.0.smithi195.stdout:v0.77 2022-06-27T12:16:43.753 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.77 2022-06-27T12:16:44.422 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:16:44.596 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.587+0000 7fe9cc811700 1 Processor -- start 2022-06-27T12:16:44.597 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.587+0000 7fe9cc811700 1 -- start start 2022-06-27T12:16:44.597 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.587+0000 7fe9cc811700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fe9c8068f80 0x7fe9c80c4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:16:44.597 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.588+0000 7fe9cc811700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe9c80c7480 0x7fe9c80683c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:16:44.597 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.588+0000 7fe9cc811700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9c81573b0 con 0x7fe9c8069350 2022-06-27T12:16:44.598 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.588+0000 7fe9cc811700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9c80c46c0 con 0x7fe9c8068f80 2022-06-27T12:16:44.598 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.588+0000 7fe9cc811700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c80697d0 con 0x7fe9c80c7480 2022-06-27T12:16:44.598 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.588+0000 7fe9c6d9d700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fe9c8069350 0x7fe9c8156460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:58336/0 (socket says 172.21.15.195:58336) 2022-06-27T12:16:44.598 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.588+0000 7fe9c6d9d700 1 -- 172.21.15.195:0/1552491154 learned_addr learned my addr 172.21.15.195:0/1552491154 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:16:44.599 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.588+0000 7fe9c5d9b700 1 --2- 172.21.15.195:0/1552491154 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe9c80c7480 0x7fe9c80683c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:16:44.599 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.588+0000 7fe9c659c700 1 --2- 172.21.15.195:0/1552491154 >> v2:172.21.15.202:3300/0 conn(0x7fe9c8068f80 0x7fe9c80c4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:16:44.599 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.588+0000 7fe9c559a700 1 -- 172.21.15.195:0/1552491154 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3365327977 0 0) 0x7fe9c81573b0 con 0x7fe9c8069350 2022-06-27T12:16:44.600 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.588+0000 7fe9c559a700 1 -- 172.21.15.195:0/1552491154 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9ac003430 con 0x7fe9c8069350 2022-06-27T12:16:44.600 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.588+0000 7fe9c5d9b700 1 -- 172.21.15.195:0/1552491154 >> v2:172.21.15.202:3300/0 conn(0x7fe9c8068f80 msgr2=0x7fe9c80c4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:16:44.600 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.588+0000 7fe9c5d9b700 1 --2- 172.21.15.195:0/1552491154 >> v2:172.21.15.202:3300/0 conn(0x7fe9c8068f80 0x7fe9c80c4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:44.601 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.588+0000 7fe9c5d9b700 1 -- 172.21.15.195:0/1552491154 >> v1:172.21.15.195:6789/0 conn(0x7fe9c8069350 legacy=0x7fe9c8156460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:16:44.601 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.588+0000 7fe9c5d9b700 1 -- 172.21.15.195:0/1552491154 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c8156b70 con 0x7fe9c80c7480 2022-06-27T12:16:44.601 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.588+0000 7fe9c659c700 1 --2- 172.21.15.195:0/1552491154 >> v2:172.21.15.202:3300/0 conn(0x7fe9c8068f80 0x7fe9c80c4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:16:44.602 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.588+0000 7fe9c5d9b700 1 --2- 172.21.15.195:0/1552491154 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe9c80c7480 0x7fe9c80683c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc0097a0 tx=0x7fe9bc009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=44d48e1caa2af6f0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:16:44.602 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.588+0000 7fe9c559a700 1 -- 172.21.15.195:0/1552491154 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9bc011630 con 0x7fe9c80c7480 2022-06-27T12:16:44.603 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.588+0000 7fe9c559a700 1 -- 172.21.15.195:0/1552491154 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9bc011790 con 0x7fe9c80c7480 2022-06-27T12:16:44.603 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.588+0000 7fe9c559a700 1 -- 172.21.15.195:0/1552491154 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9bc011b30 con 0x7fe9c80c7480 2022-06-27T12:16:44.603 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.589+0000 7fe9cc811700 1 -- 172.21.15.195:0/1552491154 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe9c80c7480 msgr2=0x7fe9c80683c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:16:44.604 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.589+0000 7fe9cc811700 1 --2- 172.21.15.195:0/1552491154 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe9c80c7480 0x7fe9c80683c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc0097a0 tx=0x7fe9bc009060 comp rx=0 tx=0).stop 2022-06-27T12:16:44.604 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.589+0000 7fe9cc811700 1 -- 172.21.15.195:0/1552491154 shutdown_connections 2022-06-27T12:16:44.604 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.589+0000 7fe9cc811700 1 --2- 172.21.15.195:0/1552491154 >> v2:172.21.15.202:3300/0 conn(0x7fe9c8068f80 0x7fe9c80c4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:44.604 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.589+0000 7fe9cc811700 1 --2- 172.21.15.195:0/1552491154 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe9c80c7480 0x7fe9c80683c0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:44.605 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.589+0000 7fe9cc811700 1 -- 172.21.15.195:0/1552491154 >> 172.21.15.195:0/1552491154 conn(0x7fe9c80ccbd0 msgr2=0x7fe9c80ccfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:16:44.605 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.589+0000 7fe9cc811700 1 -- 172.21.15.195:0/1552491154 shutdown_connections 2022-06-27T12:16:44.605 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.589+0000 7fe9cc811700 1 -- 172.21.15.195:0/1552491154 wait complete. 2022-06-27T12:16:44.605 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.590+0000 7fe9cc811700 1 Processor -- start 2022-06-27T12:16:44.606 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.590+0000 7fe9cc811700 1 -- start start 2022-06-27T12:16:44.606 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.591+0000 7fe9cc811700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe9c8068f80 0x7fe9c8166eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:16:44.606 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.591+0000 7fe9cc811700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe9c8069350 0x7fe9c814f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:16:44.606 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.591+0000 7fe9cc811700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe9c80c7480 0x7fe9c80b1e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:16:44.607 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.591+0000 7fe9cc811700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c815b2a0 con 0x7fe9c8069350 2022-06-27T12:16:44.607 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.591+0000 7fe9cc811700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c816a6e0 con 0x7fe9c8068f80 2022-06-27T12:16:44.607 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.591+0000 7fe9cc811700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c81568a0 con 0x7fe9c80c7480 2022-06-27T12:16:44.607 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.591+0000 7fe9c5d9b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe9c8069350 0x7fe9c814f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:16:44.608 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.591+0000 7fe9c659c700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe9c8068f80 0x7fe9c8166eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:16:44.608 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.591+0000 7fe9c659c700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe9c8068f80 0x7fe9c8166eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3300/0 says I am v2:172.21.15.195:50276/0 (socket says 172.21.15.195:50276) 2022-06-27T12:16:44.608 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.591+0000 7fe9c5d9b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe9c8069350 0x7fe9c814f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:52152/0 (socket says 172.21.15.195:52152) 2022-06-27T12:16:44.608 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.591+0000 7fe9c659c700 1 -- 172.21.15.195:0/57567913 learned_addr learned my addr 172.21.15.195:0/57567913 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:16:44.609 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.591+0000 7fe9c6d9d700 1 --2- 172.21.15.195:0/57567913 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe9c80c7480 0x7fe9c80b1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:16:44.609 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.591+0000 7fe9c5d9b700 1 -- 172.21.15.195:0/57567913 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe9c80c7480 msgr2=0x7fe9c80b1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:16:44.609 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.591+0000 7fe9c5d9b700 1 --2- 172.21.15.195:0/57567913 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe9c80c7480 0x7fe9c80b1e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:44.610 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.591+0000 7fe9c5d9b700 1 -- 172.21.15.195:0/57567913 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe9c8068f80 msgr2=0x7fe9c8166eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:16:44.610 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.591+0000 7fe9c5d9b700 1 --2- 172.21.15.195:0/57567913 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe9c8068f80 0x7fe9c8166eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:44.610 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.591+0000 7fe9c5d9b700 1 -- 172.21.15.195:0/57567913 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9b80092e0 con 0x7fe9c8069350 2022-06-27T12:16:44.611 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.591+0000 7fe9c6d9d700 1 --2- 172.21.15.195:0/57567913 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe9c80c7480 0x7fe9c80b1e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:16:44.611 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.591+0000 7fe9c659c700 1 --2- 172.21.15.195:0/57567913 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe9c8068f80 0x7fe9c8166eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:16:44.611 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.592+0000 7fe9c5d9b700 1 --2- 172.21.15.195:0/57567913 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe9c8069350 0x7fe9c814f620 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7fe9c80c3ca0 tx=0x7fe9bc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:16:44.611 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.592+0000 7fe9b67fc700 1 -- 172.21.15.195:0/57567913 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9bc003730 con 0x7fe9c8069350 2022-06-27T12:16:44.612 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.592+0000 7fe9b67fc700 1 -- 172.21.15.195:0/57567913 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9bc0038b0 con 0x7fe9c8069350 2022-06-27T12:16:44.612 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.592+0000 7fe9b67fc700 1 -- 172.21.15.195:0/57567913 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9bc003cb0 con 0x7fe9c8069350 2022-06-27T12:16:44.612 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.592+0000 7fe9cc811700 1 -- 172.21.15.195:0/57567913 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9bc0092e0 con 0x7fe9c8069350 2022-06-27T12:16:44.613 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.592+0000 7fe9cc811700 1 -- 172.21.15.195:0/57567913 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9c8169d40 con 0x7fe9c8069350 2022-06-27T12:16:44.613 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.593+0000 7fe9cc811700 1 -- 172.21.15.195:0/57567913 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9c816a820 con 0x7fe9c8069350 2022-06-27T12:16:44.613 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.593+0000 7fe9b67fc700 1 -- 172.21.15.195:0/57567913 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fe9bc00f050 con 0x7fe9c8069350 2022-06-27T12:16:44.613 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.593+0000 7fe9b67fc700 1 --2- 172.21.15.195:0/57567913 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe9a4063d00 0x7fe9a40661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:16:44.614 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.593+0000 7fe9b67fc700 1 -- 172.21.15.195:0/57567913 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7578+0+0 (secure 0 0 0) 0x7fe9bc082530 con 0x7fe9c8069350 2022-06-27T12:16:44.615 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.594+0000 7fe9c659c700 1 --2- 172.21.15.195:0/57567913 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe9a4063d00 0x7fe9a40661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:16:44.615 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.594+0000 7fe9c659c700 1 --2- 172.21.15.195:0/57567913 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe9a4063d00 0x7fe9a40661b0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fe9b8011520 tx=0x7fe9b8009060 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:16:44.616 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.598+0000 7fe9b67fc700 1 -- 172.21.15.195:0/57567913 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fe9bc050b30 con 0x7fe9c8069350 2022-06-27T12:16:44.827 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.826+0000 7fe9cc811700 1 -- 172.21.15.195:0/57567913 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe9c806ac60 con 0x7fe9c8069350 2022-06-27T12:16:44.827 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.827+0000 7fe9b67fc700 1 -- 172.21.15.195:0/57567913 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+6132 (secure 0 0 0) 0x7fe9bc015090 con 0x7fe9c8069350 2022-06-27T12:16:44.828 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:16:44.828 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":107,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":7165,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/3551478822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":3551478822},{"type":"v1","addr":"172.21.15.202:6835","nonce":3551478822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":107}],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:15:52.408825+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6254":{"gid":6254,"name":"b","rank":0,"incarnation":102,"state":"up:active","state_seq":43,"addr":"172.21.15.202:6833/1317767101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1317767101},{"type":"v1","addr":"172.21.15.202:6833","nonce":1317767101}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:16:44.836 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.829+0000 7fe9cc811700 1 -- 172.21.15.195:0/57567913 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe9a4063d00 msgr2=0x7fe9a40661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:16:44.836 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.829+0000 7fe9cc811700 1 --2- 172.21.15.195:0/57567913 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe9a4063d00 0x7fe9a40661b0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fe9b8011520 tx=0x7fe9b8009060 comp rx=0 tx=0).stop 2022-06-27T12:16:44.837 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.829+0000 7fe9cc811700 1 -- 172.21.15.195:0/57567913 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe9c8069350 msgr2=0x7fe9c814f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:16:44.838 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.829+0000 7fe9cc811700 1 --2- 172.21.15.195:0/57567913 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe9c8069350 0x7fe9c814f620 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7fe9c80c3ca0 tx=0x7fe9bc017000 comp rx=0 tx=0).stop 2022-06-27T12:16:44.839 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.829+0000 7fe9cc811700 1 -- 172.21.15.195:0/57567913 shutdown_connections 2022-06-27T12:16:44.839 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.829+0000 7fe9cc811700 1 --2- 172.21.15.195:0/57567913 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe9a4063d00 0x7fe9a40661b0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:44.840 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.829+0000 7fe9cc811700 1 --2- 172.21.15.195:0/57567913 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe9c80c7480 0x7fe9c80b1e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:44.840 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.829+0000 7fe9cc811700 1 --2- 172.21.15.195:0/57567913 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe9c8068f80 0x7fe9c8166eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:44.842 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.829+0000 7fe9cc811700 1 --2- 172.21.15.195:0/57567913 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe9c8069350 0x7fe9c814f620 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:44.842 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.829+0000 7fe9cc811700 1 -- 172.21.15.195:0/57567913 >> 172.21.15.195:0/57567913 conn(0x7fe9c80ccbd0 msgr2=0x7fe9c8150340 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:16:44.843 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.829+0000 7fe9cc811700 1 -- 172.21.15.195:0/57567913 shutdown_connections 2022-06-27T12:16:44.843 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:44.829+0000 7fe9cc811700 1 -- 172.21.15.195:0/57567913 wait complete. 2022-06-27T12:16:44.843 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 107 2022-06-27T12:16:44.846 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2022-06-27T12:16:44.847 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-06-27T12:16:44.847 INFO:teuthology.orchestra.run:waiting for 300 2022-06-27T12:16:44.868 INFO:tasks.ceph.mds.b:Stopped 2022-06-27T12:16:44.869 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:16:45.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.151+0000 7f9b19093700 1 Processor -- start 2022-06-27T12:16:45.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.152+0000 7f9b19093700 1 -- start start 2022-06-27T12:16:45.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.154+0000 7f9b19093700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9b14156670 0x7f9b140c7440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:16:45.154 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.154+0000 7f9b19093700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f9b14067ed0 0x7f9b140c4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:16:45.155 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.154+0000 7f9b19093700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b140c43d0 con 0x7f9b14069390 2022-06-27T12:16:45.155 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.154+0000 7f9b19093700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b14068430 con 0x7f9b14067ed0 2022-06-27T12:16:45.155 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.154+0000 7f9b19093700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b14068710 con 0x7f9b14156670 2022-06-27T12:16:45.156 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.155+0000 7f9b1359e700 1 --1- >> v1:172.21.15.202:6789/0 conn(0x7f9b14069390 0x7f9b141502b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.202:6789/0 says I am v1:172.21.15.195:60228/0 (socket says 172.21.15.195:60228) 2022-06-27T12:16:45.156 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.155+0000 7f9b1359e700 1 -- 172.21.15.195:0/2522318834 learned_addr learned my addr 172.21.15.195:0/2522318834 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:16:45.156 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.155+0000 7f9b12d9d700 1 --2- 172.21.15.195:0/2522318834 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9b14156670 0x7f9b140c7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:16:45.156 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.155+0000 7f9b1259c700 1 --2- 172.21.15.195:0/2522318834 >> v2:172.21.15.202:3300/0 conn(0x7f9b14067ed0 0x7f9b140c4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:16:45.157 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.157+0000 7f9b1259c700 1 -- 172.21.15.195:0/2522318834 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9b14156670 msgr2=0x7f9b140c7440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:16:45.158 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.157+0000 7f9b1259c700 1 --2- 172.21.15.195:0/2522318834 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9b14156670 0x7f9b140c7440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:45.158 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.157+0000 7f9b1259c700 1 -- 172.21.15.195:0/2522318834 >> v1:172.21.15.202:6789/0 conn(0x7f9b14069390 legacy=0x7f9b141502b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:16:45.158 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.157+0000 7f9b1259c700 1 -- 172.21.15.195:0/2522318834 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b140689f0 con 0x7f9b14067ed0 2022-06-27T12:16:45.161 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.158+0000 7f9b1259c700 1 --2- 172.21.15.195:0/2522318834 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9b14067ed0 0x7f9b140c4860 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f9b080097a0 tx=0x7f9b08009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=97dd645a305cbfa server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:16:45.161 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.161+0000 7f9b11d9b700 1 -- 172.21.15.195:0/2522318834 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b08004030 con 0x7f9b14067ed0 2022-06-27T12:16:45.161 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.161+0000 7f9b11d9b700 1 -- 172.21.15.195:0/2522318834 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b08004190 con 0x7f9b14067ed0 2022-06-27T12:16:45.162 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.161+0000 7f9b11d9b700 1 -- 172.21.15.195:0/2522318834 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b08011630 con 0x7f9b14067ed0 2022-06-27T12:16:45.162 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.161+0000 7f9b19093700 1 -- 172.21.15.195:0/2522318834 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9b14067ed0 msgr2=0x7f9b140c4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:16:45.163 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.161+0000 7f9b19093700 1 --2- 172.21.15.195:0/2522318834 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9b14067ed0 0x7f9b140c4860 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f9b080097a0 tx=0x7f9b08009060 comp rx=0 tx=0).stop 2022-06-27T12:16:45.164 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.164+0000 7f9b19093700 1 -- 172.21.15.195:0/2522318834 shutdown_connections 2022-06-27T12:16:45.164 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.164+0000 7f9b19093700 1 --2- 172.21.15.195:0/2522318834 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9b14156670 0x7f9b140c7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:45.164 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.164+0000 7f9b19093700 1 --2- 172.21.15.195:0/2522318834 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9b14067ed0 0x7f9b140c4860 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:45.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.164+0000 7f9b19093700 1 -- 172.21.15.195:0/2522318834 >> 172.21.15.195:0/2522318834 conn(0x7f9b14069e00 msgr2=0x7f9b1406a200 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:16:45.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.164+0000 7f9b19093700 1 -- 172.21.15.195:0/2522318834 shutdown_connections 2022-06-27T12:16:45.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.164+0000 7f9b19093700 1 -- 172.21.15.195:0/2522318834 wait complete. 2022-06-27T12:16:45.166 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.165+0000 7f9b19093700 1 Processor -- start 2022-06-27T12:16:45.166 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.166+0000 7f9b19093700 1 -- start start 2022-06-27T12:16:45.167 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.167+0000 7f9b19093700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9b14067ed0 0x7f9b1416a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:16:45.167 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.167+0000 7f9b19093700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9b14069390 0x7f9b14154f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:16:45.167 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.167+0000 7f9b19093700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9b14156670 0x7f9b14167d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:16:45.168 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.167+0000 7f9b19093700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b140a8640 con 0x7f9b14069390 2022-06-27T12:16:45.168 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.167+0000 7f9b19093700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b141672c0 con 0x7f9b14156670 2022-06-27T12:16:45.168 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.167+0000 7f9b19093700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b14153400 con 0x7f9b14067ed0 2022-06-27T12:16:45.169 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.168+0000 7f9b1359e700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9b14156670 0x7f9b14167d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:16:45.170 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.168+0000 7f9b1359e700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9b14156670 0x7f9b14167d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3300/0 says I am v2:172.21.15.195:50308/0 (socket says 172.21.15.195:50308) 2022-06-27T12:16:45.170 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.168+0000 7f9b1359e700 1 -- 172.21.15.195:0/4244189122 learned_addr learned my addr 172.21.15.195:0/4244189122 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:16:45.170 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.168+0000 7f9b1359e700 1 -- 172.21.15.195:0/4244189122 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9b14067ed0 msgr2=0x7f9b1416a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:16:45.180 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.168+0000 7f9b1359e700 1 --2- 172.21.15.195:0/4244189122 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9b14067ed0 0x7f9b1416a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:45.180 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.168+0000 7f9b1359e700 1 -- 172.21.15.195:0/4244189122 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9b14069390 msgr2=0x7f9b14154f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:16:45.180 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.168+0000 7f9b1359e700 1 --2- 172.21.15.195:0/4244189122 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9b14069390 0x7f9b14154f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:45.180 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.168+0000 7f9b1359e700 1 -- 172.21.15.195:0/4244189122 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b080092e0 con 0x7f9b14156670 2022-06-27T12:16:45.181 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.169+0000 7f9b1359e700 1 --2- 172.21.15.195:0/4244189122 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9b14156670 0x7f9b14167d50 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f9b0c00c810 tx=0x7f9b0c004e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:16:45.181 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.169+0000 7f9afaffd700 1 -- 172.21.15.195:0/4244189122 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b0c005d70 con 0x7f9b14156670 2022-06-27T12:16:45.183 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.169+0000 7f9afaffd700 1 -- 172.21.15.195:0/4244189122 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b0c0070f0 con 0x7f9b14156670 2022-06-27T12:16:45.183 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.169+0000 7f9afaffd700 1 -- 172.21.15.195:0/4244189122 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b0c0074b0 con 0x7f9b14156670 2022-06-27T12:16:45.183 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.169+0000 7f9b19093700 1 -- 172.21.15.195:0/4244189122 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b141536a0 con 0x7f9b14156670 2022-06-27T12:16:45.183 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.169+0000 7f9b19093700 1 -- 172.21.15.195:0/4244189122 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b14166990 con 0x7f9b14156670 2022-06-27T12:16:45.184 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.171+0000 7f9afaffd700 1 -- 172.21.15.195:0/4244189122 <== mon.1 v2:172.21.15.202:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f9b0c010070 con 0x7f9b14156670 2022-06-27T12:16:45.184 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.172+0000 7f9afaffd700 1 --2- 172.21.15.195:0/4244189122 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9af4063d00 0x7f9af40661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:16:45.184 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.172+0000 7f9afaffd700 1 -- 172.21.15.195:0/4244189122 <== mon.1 v2:172.21.15.202:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7578+0+0 (secure 0 0 0) 0x7f9b0c080a80 con 0x7f9b14156670 2022-06-27T12:16:45.184 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.172+0000 7f9b12d9d700 1 --2- 172.21.15.195:0/4244189122 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9af4063d00 0x7f9af40661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:16:45.185 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.173+0000 7f9b12d9d700 1 --2- 172.21.15.195:0/4244189122 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9af4063d00 0x7f9af40661b0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f9b040099d0 tx=0x7f9b04009060 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:16:45.185 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.181+0000 7f9b19093700 1 -- 172.21.15.195:0/4244189122 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9aec005170 con 0x7f9b14156670 2022-06-27T12:16:45.187 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.187+0000 7f9afaffd700 1 -- 172.21.15.195:0/4244189122 <== mon.1 v2:172.21.15.202:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f9b0c04f100 con 0x7f9b14156670 2022-06-27T12:16:45.446 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.445+0000 7f9b19093700 1 -- 172.21.15.195:0/4244189122 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9aec005dc0 con 0x7f9b14156670 2022-06-27T12:16:45.447 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.446+0000 7f9afaffd700 1 -- 172.21.15.195:0/4244189122 <== mon.1 v2:172.21.15.202:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+6132 (secure 0 0 0) 0x7f9b0c052720 con 0x7f9b14156670 2022-06-27T12:16:45.448 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:16:45.459 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":107,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":7165,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/3551478822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":3551478822},{"type":"v1","addr":"172.21.15.202:6835","nonce":3551478822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":107}],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:15:52.408825+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6254":{"gid":6254,"name":"b","rank":0,"incarnation":102,"state":"up:active","state_seq":43,"addr":"172.21.15.202:6833/1317767101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1317767101},{"type":"v1","addr":"172.21.15.202:6833","nonce":1317767101}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:16:45.463 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.448+0000 7f9b19093700 1 -- 172.21.15.195:0/4244189122 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9af4063d00 msgr2=0x7f9af40661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:16:45.464 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.448+0000 7f9b19093700 1 --2- 172.21.15.195:0/4244189122 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9af4063d00 0x7f9af40661b0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f9b040099d0 tx=0x7f9b04009060 comp rx=0 tx=0).stop 2022-06-27T12:16:45.465 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.448+0000 7f9b19093700 1 -- 172.21.15.195:0/4244189122 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9b14156670 msgr2=0x7f9b14167d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:16:45.465 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.448+0000 7f9b19093700 1 --2- 172.21.15.195:0/4244189122 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9b14156670 0x7f9b14167d50 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f9b0c00c810 tx=0x7f9b0c004e40 comp rx=0 tx=0).stop 2022-06-27T12:16:45.466 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.448+0000 7f9b19093700 1 -- 172.21.15.195:0/4244189122 shutdown_connections 2022-06-27T12:16:45.467 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.448+0000 7f9b19093700 1 --2- 172.21.15.195:0/4244189122 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9af4063d00 0x7f9af40661b0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:45.467 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.448+0000 7f9b19093700 1 --2- 172.21.15.195:0/4244189122 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9b14156670 0x7f9b14167d50 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:45.467 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.448+0000 7f9b19093700 1 --2- 172.21.15.195:0/4244189122 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9b14067ed0 0x7f9b1416a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:45.468 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.448+0000 7f9b19093700 1 --2- 172.21.15.195:0/4244189122 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9b14069390 0x7f9b14154f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:45.468 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.448+0000 7f9b19093700 1 -- 172.21.15.195:0/4244189122 >> 172.21.15.195:0/4244189122 conn(0x7f9b14069e00 msgr2=0x7f9b1406a800 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:16:45.469 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.448+0000 7f9b19093700 1 -- 172.21.15.195:0/4244189122 shutdown_connections 2022-06-27T12:16:45.469 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:45.448+0000 7f9b19093700 1 -- 172.21.15.195:0/4244189122 wait complete. 2022-06-27T12:16:45.469 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 107 2022-06-27T12:16:45.470 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2022-06-27T12:16:47.473 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:16:47.639 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.632+0000 7fce01efb700 1 Processor -- start 2022-06-27T12:16:47.640 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.632+0000 7fce01efb700 1 -- start start 2022-06-27T12:16:47.640 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.633+0000 7fce01efb700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fcdfc151560 0x7fcdfc0c2120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:16:47.640 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.633+0000 7fce01efb700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fcdfc151930 0x7fcdfc0bf4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:16:47.641 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.633+0000 7fce01efb700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcdfc0c2c70 0x7fcdfc1541f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:16:47.641 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.633+0000 7fce01efb700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdfc154da0 con 0x7fcdfc151560 2022-06-27T12:16:47.641 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.633+0000 7fce01efb700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdfc0c12a0 con 0x7fcdfc151930 2022-06-27T12:16:47.641 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.633+0000 7fce01efb700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdfc0c1580 con 0x7fcdfc0c2c70 2022-06-27T12:16:47.642 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.633+0000 7fcdfb7fe700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fcdfc151560 0x7fcdfc0c2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:16:47.642 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.633+0000 7fcdfb7fe700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fcdfc151560 0x7fcdfc0c2120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:52198/0 (socket says 172.21.15.195:52198) 2022-06-27T12:16:47.642 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.633+0000 7fcdfb7fe700 1 -- 172.21.15.195:0/2007033093 learned_addr learned my addr 172.21.15.195:0/2007033093 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:16:47.643 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.633+0000 7fcdfaffd700 1 --2- 172.21.15.195:0/2007033093 >> v2:172.21.15.202:3300/0 conn(0x7fcdfc151930 0x7fcdfc0bf4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:16:47.648 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.633+0000 7fcdfbfff700 1 --2- 172.21.15.195:0/2007033093 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcdfc0c2c70 0x7fcdfc1541f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:16:47.648 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.634+0000 7fcdfb7fe700 1 -- 172.21.15.195:0/2007033093 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcdfc0c2c70 msgr2=0x7fcdfc1541f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:16:47.649 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.634+0000 7fcdfb7fe700 1 --2- 172.21.15.195:0/2007033093 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcdfc0c2c70 0x7fcdfc1541f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:47.649 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.634+0000 7fcdfb7fe700 1 -- 172.21.15.195:0/2007033093 >> v2:172.21.15.202:3300/0 conn(0x7fcdfc151930 msgr2=0x7fcdfc0bf4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:16:47.649 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.634+0000 7fcdfb7fe700 1 --2- 172.21.15.195:0/2007033093 >> v2:172.21.15.202:3300/0 conn(0x7fcdfc151930 0x7fcdfc0bf4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:47.649 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.634+0000 7fcdfb7fe700 1 -- 172.21.15.195:0/2007033093 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdfc0c29f0 con 0x7fcdfc151560 2022-06-27T12:16:47.650 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.634+0000 7fcdfaffd700 1 --2- 172.21.15.195:0/2007033093 >> v2:172.21.15.202:3300/0 conn(0x7fcdfc151930 0x7fcdfc0bf4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:16:47.650 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.634+0000 7fcdfb7fe700 1 --2- 172.21.15.195:0/2007033093 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcdfc151560 0x7fcdfc0c2120 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc0c2660 tx=0x7fcdf4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b424c484e338412 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:16:47.650 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.634+0000 7fcdfa7fc700 1 -- 172.21.15.195:0/2007033093 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf400e070 con 0x7fcdfc151560 2022-06-27T12:16:47.651 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.634+0000 7fcdfa7fc700 1 -- 172.21.15.195:0/2007033093 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdf400ba00 con 0x7fcdfc151560 2022-06-27T12:16:47.651 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.634+0000 7fcdfa7fc700 1 -- 172.21.15.195:0/2007033093 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf40092e0 con 0x7fcdfc151560 2022-06-27T12:16:47.651 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.635+0000 7fce01efb700 1 -- 172.21.15.195:0/2007033093 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcdfc151560 msgr2=0x7fcdfc0c2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:16:47.651 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.635+0000 7fce01efb700 1 --2- 172.21.15.195:0/2007033093 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcdfc151560 0x7fcdfc0c2120 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc0c2660 tx=0x7fcdf4009840 comp rx=0 tx=0).stop 2022-06-27T12:16:47.652 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.635+0000 7fce01efb700 1 -- 172.21.15.195:0/2007033093 shutdown_connections 2022-06-27T12:16:47.652 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.635+0000 7fce01efb700 1 --2- 172.21.15.195:0/2007033093 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcdfc0c2c70 0x7fcdfc1541f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:47.652 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.635+0000 7fce01efb700 1 --2- 172.21.15.195:0/2007033093 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcdfc151560 0x7fcdfc0c2120 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:47.652 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.635+0000 7fce01efb700 1 --2- 172.21.15.195:0/2007033093 >> v2:172.21.15.202:3300/0 conn(0x7fcdfc151930 0x7fcdfc0bf4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:47.652 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.635+0000 7fce01efb700 1 -- 172.21.15.195:0/2007033093 >> 172.21.15.195:0/2007033093 conn(0x7fcdfc0c6350 msgr2=0x7fcdfc0c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:16:47.653 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.635+0000 7fce01efb700 1 -- 172.21.15.195:0/2007033093 shutdown_connections 2022-06-27T12:16:47.653 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.635+0000 7fce01efb700 1 -- 172.21.15.195:0/2007033093 wait complete. 2022-06-27T12:16:47.653 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.636+0000 7fce01efb700 1 Processor -- start 2022-06-27T12:16:47.654 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.636+0000 7fce01efb700 1 -- start start 2022-06-27T12:16:47.654 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.637+0000 7fce01efb700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcdfc0c2c70 0x7fcdfc14c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:16:47.654 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.637+0000 7fce01efb700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcdfc151560 0x7fcdfc14b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:16:47.654 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.637+0000 7fce01efb700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcdfc151930 0x7fcdfc14b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:16:47.655 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.637+0000 7fce01efb700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdfc14c540 con 0x7fcdfc151930 2022-06-27T12:16:47.655 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.637+0000 7fce01efb700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdfc14bb70 con 0x7fcdfc151560 2022-06-27T12:16:47.655 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.637+0000 7fce01efb700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdfc0c8980 con 0x7fcdfc0c2c70 2022-06-27T12:16:47.655 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.637+0000 7fcdfbfff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcdfc151930 0x7fcdfc14b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:16:47.655 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.637+0000 7fcdfbfff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcdfc151930 0x7fcdfc14b630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:52208/0 (socket says 172.21.15.195:52208) 2022-06-27T12:16:47.656 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.637+0000 7fcdfbfff700 1 -- 172.21.15.195:0/1343343629 learned_addr learned my addr 172.21.15.195:0/1343343629 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:16:47.656 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.637+0000 7fcdfb7fe700 1 --2- 172.21.15.195:0/1343343629 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcdfc0c2c70 0x7fcdfc14c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:16:47.656 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.637+0000 7fcdfaffd700 1 --2- 172.21.15.195:0/1343343629 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcdfc151560 0x7fcdfc14b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:16:47.656 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.637+0000 7fcdfbfff700 1 -- 172.21.15.195:0/1343343629 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcdfc0c2c70 msgr2=0x7fcdfc14c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:16:47.657 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.637+0000 7fcdfbfff700 1 --2- 172.21.15.195:0/1343343629 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcdfc0c2c70 0x7fcdfc14c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:47.657 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.637+0000 7fcdfbfff700 1 -- 172.21.15.195:0/1343343629 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcdfc151560 msgr2=0x7fcdfc14b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:16:47.657 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.637+0000 7fcdfbfff700 1 --2- 172.21.15.195:0/1343343629 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcdfc151560 0x7fcdfc14b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:47.657 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.637+0000 7fcdfbfff700 1 -- 172.21.15.195:0/1343343629 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcde40092e0 con 0x7fcdfc151930 2022-06-27T12:16:47.658 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.637+0000 7fcdfbfff700 1 --2- 172.21.15.195:0/1343343629 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcdfc151930 0x7fcdfc14b630 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7fcdec00e930 tx=0x7fcdec009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:16:47.658 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.637+0000 7fcdf37fe700 1 -- 172.21.15.195:0/1343343629 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdec013680 con 0x7fcdfc151930 2022-06-27T12:16:47.660 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.637+0000 7fcdf37fe700 1 -- 172.21.15.195:0/1343343629 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdec013800 con 0x7fcdfc151930 2022-06-27T12:16:47.660 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.637+0000 7fcdf37fe700 1 -- 172.21.15.195:0/1343343629 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdec013bc0 con 0x7fcdfc151930 2022-06-27T12:16:47.661 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.637+0000 7fce01efb700 1 -- 172.21.15.195:0/1343343629 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdf4009ae0 con 0x7fcdfc151930 2022-06-27T12:16:47.661 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.637+0000 7fce01efb700 1 -- 172.21.15.195:0/1343343629 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdfc14fd50 con 0x7fcdfc151930 2022-06-27T12:16:47.662 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.638+0000 7fce01efb700 1 -- 172.21.15.195:0/1343343629 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdfc14cf10 con 0x7fcdfc151930 2022-06-27T12:16:47.662 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.639+0000 7fcdf37fe700 1 -- 172.21.15.195:0/1343343629 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fcdec019030 con 0x7fcdfc151930 2022-06-27T12:16:47.663 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.639+0000 7fcdf37fe700 1 --2- 172.21.15.195:0/1343343629 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcde0063d30 0x7fcde00661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:16:47.663 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.639+0000 7fcdf37fe700 1 -- 172.21.15.195:0/1343343629 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7578+0+0 (secure 0 0 0) 0x7fcdec080bd0 con 0x7fcdfc151930 2022-06-27T12:16:47.664 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.639+0000 7fcdfb7fe700 1 --2- 172.21.15.195:0/1343343629 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcde0063d30 0x7fcde00661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:16:47.664 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.640+0000 7fcdfb7fe700 1 --2- 172.21.15.195:0/1343343629 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcde0063d30 0x7fcde00661e0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fcdf4009880 tx=0x7fcdf4018000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:16:47.664 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.643+0000 7fcdf37fe700 1 -- 172.21.15.195:0/1343343629 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fcdec04f1d0 con 0x7fcdfc151930 2022-06-27T12:16:47.870 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.868+0000 7fce01efb700 1 -- 172.21.15.195:0/1343343629 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcdfc14c6f0 con 0x7fcdfc151930 2022-06-27T12:16:47.878 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.868+0000 7fcdf37fe700 1 -- 172.21.15.195:0/1343343629 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+6132 (secure 0 0 0) 0x7fcdfc14c6f0 con 0x7fcdfc151930 2022-06-27T12:16:47.879 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:16:47.879 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":107,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":7165,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/3551478822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":3551478822},{"type":"v1","addr":"172.21.15.202:6835","nonce":3551478822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":107}],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:15:52.408825+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6254":{"gid":6254,"name":"b","rank":0,"incarnation":102,"state":"up:active","state_seq":43,"addr":"172.21.15.202:6833/1317767101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1317767101},{"type":"v1","addr":"172.21.15.202:6833","nonce":1317767101}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:16:47.881 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.870+0000 7fce01efb700 1 -- 172.21.15.195:0/1343343629 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcde0063d30 msgr2=0x7fcde00661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:16:47.881 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.870+0000 7fce01efb700 1 --2- 172.21.15.195:0/1343343629 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcde0063d30 0x7fcde00661e0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fcdf4009880 tx=0x7fcdf4018000 comp rx=0 tx=0).stop 2022-06-27T12:16:47.882 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.870+0000 7fce01efb700 1 -- 172.21.15.195:0/1343343629 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcdfc151930 msgr2=0x7fcdfc14b630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:16:47.882 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.870+0000 7fce01efb700 1 --2- 172.21.15.195:0/1343343629 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcdfc151930 0x7fcdfc14b630 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7fcdec00e930 tx=0x7fcdec009060 comp rx=0 tx=0).stop 2022-06-27T12:16:47.882 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.870+0000 7fce01efb700 1 -- 172.21.15.195:0/1343343629 shutdown_connections 2022-06-27T12:16:47.884 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.870+0000 7fce01efb700 1 --2- 172.21.15.195:0/1343343629 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fcde0063d30 0x7fcde00661e0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:47.885 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.870+0000 7fce01efb700 1 --2- 172.21.15.195:0/1343343629 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcdfc151930 0x7fcdfc14b630 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:47.885 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.870+0000 7fce01efb700 1 --2- 172.21.15.195:0/1343343629 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fcdfc0c2c70 0x7fcdfc14c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:47.886 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.870+0000 7fce01efb700 1 --2- 172.21.15.195:0/1343343629 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fcdfc151560 0x7fcdfc14b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:47.887 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.870+0000 7fce01efb700 1 -- 172.21.15.195:0/1343343629 >> 172.21.15.195:0/1343343629 conn(0x7fcdfc0c6350 msgr2=0x7fcdfc0c8e80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:16:47.887 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.871+0000 7fce01efb700 1 -- 172.21.15.195:0/1343343629 shutdown_connections 2022-06-27T12:16:47.887 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:47.871+0000 7fce01efb700 1 -- 172.21.15.195:0/1343343629 wait complete. 2022-06-27T12:16:47.888 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 107 2022-06-27T12:16:47.893 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2022-06-27T12:16:49.894 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:16:50.063 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.028+0000 7f9ba2409700 1 Processor -- start 2022-06-27T12:16:50.063 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.028+0000 7f9ba2409700 1 -- start start 2022-06-27T12:16:50.064 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.028+0000 7f9ba2409700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9b9c06e850 0x7f9b9c1570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:16:50.065 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.028+0000 7f9ba2409700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f9b9c0c34a0 0x7f9b9c0c2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:16:50.065 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.028+0000 7f9ba2409700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b9c06fba0 con 0x7f9b9c06fef0 2022-06-27T12:16:50.065 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.029+0000 7f9ba2409700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b9c1575e0 con 0x7f9b9c0c34a0 2022-06-27T12:16:50.066 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.029+0000 7f9ba2409700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b9c06f5e0 con 0x7f9b9c06e850 2022-06-27T12:16:50.066 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.029+0000 7f9ba09a6700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f9b9c0c34a0 0x7f9b9c0c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:16:50.066 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.029+0000 7f9ba09a6700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f9b9c0c34a0 0x7f9b9c0c2920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:52222/0 (socket says 172.21.15.195:52222) 2022-06-27T12:16:50.066 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.029+0000 7f9ba09a6700 1 -- 172.21.15.195:0/2172748991 learned_addr learned my addr 172.21.15.195:0/2172748991 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:16:50.069 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.029+0000 7f9b9bfff700 1 --2- 172.21.15.195:0/2172748991 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9b9c06e850 0x7f9b9c1570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:16:50.070 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.029+0000 7f9ba09a6700 1 -- 172.21.15.195:0/2172748991 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9b9c06e850 msgr2=0x7f9b9c1570a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:16:50.070 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.029+0000 7f9ba09a6700 1 --2- 172.21.15.195:0/2172748991 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9b9c06e850 0x7f9b9c1570a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:50.071 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.029+0000 7f9b9affd700 1 -- 172.21.15.195:0/2172748991 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2061822487 0 0) 0x7f9b9c06fba0 con 0x7f9b9c06fef0 2022-06-27T12:16:50.071 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.029+0000 7f9ba09a6700 1 -- 172.21.15.195:0/2172748991 >> v1:172.21.15.202:6789/0 conn(0x7f9b9c06fef0 legacy=0x7f9b9c069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:16:50.071 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.029+0000 7f9ba09a6700 1 -- 172.21.15.195:0/2172748991 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b9c06f8c0 con 0x7f9b9c0c34a0 2022-06-27T12:16:50.072 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.029+0000 7f9ba09a6700 1 --2- 172.21.15.195:0/2172748991 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9b9c0c34a0 0x7f9b9c0c2920 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f9b94009f20 tx=0x7f9b94009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7f8efc890acb9f3 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:16:50.072 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.029+0000 7f9b9affd700 1 -- 172.21.15.195:0/2172748991 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b9400e070 con 0x7f9b9c0c34a0 2022-06-27T12:16:50.072 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.029+0000 7f9b9affd700 1 -- 172.21.15.195:0/2172748991 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b9400ba60 con 0x7f9b9c0c34a0 2022-06-27T12:16:50.073 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.029+0000 7f9b9affd700 1 -- 172.21.15.195:0/2172748991 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b940092e0 con 0x7f9b9c0c34a0 2022-06-27T12:16:50.073 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.030+0000 7f9ba2409700 1 -- 172.21.15.195:0/2172748991 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9b9c0c34a0 msgr2=0x7f9b9c0c2920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:16:50.073 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.030+0000 7f9ba2409700 1 --2- 172.21.15.195:0/2172748991 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9b9c0c34a0 0x7f9b9c0c2920 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f9b94009f20 tx=0x7f9b94009840 comp rx=0 tx=0).stop 2022-06-27T12:16:50.073 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.030+0000 7f9ba2409700 1 -- 172.21.15.195:0/2172748991 shutdown_connections 2022-06-27T12:16:50.073 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.030+0000 7f9ba2409700 1 --2- 172.21.15.195:0/2172748991 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9b9c0c34a0 0x7f9b9c0c2920 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:50.074 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.030+0000 7f9ba2409700 1 --2- 172.21.15.195:0/2172748991 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9b9c06e850 0x7f9b9c1570a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:50.074 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.030+0000 7f9ba2409700 1 -- 172.21.15.195:0/2172748991 >> 172.21.15.195:0/2172748991 conn(0x7f9b9c0c6350 msgr2=0x7f9b9c0c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:16:50.076 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.030+0000 7f9ba2409700 1 -- 172.21.15.195:0/2172748991 shutdown_connections 2022-06-27T12:16:50.076 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.030+0000 7f9ba2409700 1 -- 172.21.15.195:0/2172748991 wait complete. 2022-06-27T12:16:50.076 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.031+0000 7f9ba2409700 1 Processor -- start 2022-06-27T12:16:50.076 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.031+0000 7f9ba2409700 1 -- start start 2022-06-27T12:16:50.077 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.032+0000 7f9ba2409700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9b9c06e850 0x7f9b9c145240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:16:50.077 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.032+0000 7f9ba2409700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9b9c0c34a0 0x7f9b9c145780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:16:50.077 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.032+0000 7f9ba2409700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9b9c147760 0x7f9b9c156260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:16:50.078 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.032+0000 7f9ba2409700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b9c154c60 con 0x7f9b9c0c34a0 2022-06-27T12:16:50.078 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.032+0000 7f9ba2409700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b9c155db0 con 0x7f9b9c06e850 2022-06-27T12:16:50.078 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.032+0000 7f9ba2409700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b9c149050 con 0x7f9b9c147760 2022-06-27T12:16:50.078 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.032+0000 7f9b9b7fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9b9c0c34a0 0x7f9b9c145780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:16:50.078 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.032+0000 7f9b9bfff700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9b9c06e850 0x7f9b9c145240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:16:50.079 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.032+0000 7f9ba09a6700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9b9c147760 0x7f9b9c156260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:16:50.079 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.032+0000 7f9b9b7fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9b9c0c34a0 0x7f9b9c145780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:52228/0 (socket says 172.21.15.195:52228) 2022-06-27T12:16:50.079 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.032+0000 7f9ba09a6700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9b9c147760 0x7f9b9c156260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3301/0 says I am v2:172.21.15.195:44982/0 (socket says 172.21.15.195:44982) 2022-06-27T12:16:50.080 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.032+0000 7f9b9bfff700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9b9c06e850 0x7f9b9c145240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.202:3300/0 says I am v2:172.21.15.195:50350/0 (socket says 172.21.15.195:50350) 2022-06-27T12:16:50.080 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.032+0000 7f9b9b7fe700 1 -- 172.21.15.195:0/2275062480 learned_addr learned my addr 172.21.15.195:0/2275062480 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:16:50.081 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.032+0000 7f9b9b7fe700 1 -- 172.21.15.195:0/2275062480 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9b9c147760 msgr2=0x7f9b9c156260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:16:50.081 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.032+0000 7f9b9b7fe700 1 --2- 172.21.15.195:0/2275062480 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9b9c147760 0x7f9b9c156260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:50.082 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.033+0000 7f9b9b7fe700 1 -- 172.21.15.195:0/2275062480 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9b9c06e850 msgr2=0x7f9b9c145240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:16:50.082 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.033+0000 7f9b9b7fe700 1 --2- 172.21.15.195:0/2275062480 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9b9c06e850 0x7f9b9c145240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:50.082 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.033+0000 7f9b9b7fe700 1 -- 172.21.15.195:0/2275062480 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b94009ae0 con 0x7f9b9c0c34a0 2022-06-27T12:16:50.082 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.033+0000 7f9b9bfff700 1 --2- 172.21.15.195:0/2275062480 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9b9c06e850 0x7f9b9c145240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:16:50.083 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.033+0000 7f9b9b7fe700 1 --2- 172.21.15.195:0/2275062480 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9b9c0c34a0 0x7f9b9c145780 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f9b9000c4a0 tx=0x7f9b90007c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:16:50.083 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.033+0000 7f9b83fff700 1 -- 172.21.15.195:0/2275062480 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b900052e0 con 0x7f9b9c0c34a0 2022-06-27T12:16:50.083 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.033+0000 7f9b83fff700 1 -- 172.21.15.195:0/2275062480 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b90003500 con 0x7f9b9c0c34a0 2022-06-27T12:16:50.083 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.033+0000 7f9ba2409700 1 -- 172.21.15.195:0/2275062480 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b9c13e840 con 0x7f9b9c0c34a0 2022-06-27T12:16:50.084 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.033+0000 7f9b83fff700 1 -- 172.21.15.195:0/2275062480 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b900038c0 con 0x7f9b9c0c34a0 2022-06-27T12:16:50.084 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.033+0000 7f9ba2409700 1 -- 172.21.15.195:0/2275062480 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b9c13eb00 con 0x7f9b9c0c34a0 2022-06-27T12:16:50.084 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.034+0000 7f9ba2409700 1 -- 172.21.15.195:0/2275062480 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b9c155f80 con 0x7f9b9c0c34a0 2022-06-27T12:16:50.084 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.034+0000 7f9b83fff700 1 -- 172.21.15.195:0/2275062480 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f9b90019030 con 0x7f9b9c0c34a0 2022-06-27T12:16:50.085 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.034+0000 7f9b83fff700 1 --2- 172.21.15.195:0/2275062480 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9b7c063d00 0x7f9b7c0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:16:50.085 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.035+0000 7f9b83fff700 1 -- 172.21.15.195:0/2275062480 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7578+0+0 (secure 0 0 0) 0x7f9b90080110 con 0x7f9b9c0c34a0 2022-06-27T12:16:50.085 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.035+0000 7f9b9bfff700 1 --2- 172.21.15.195:0/2275062480 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9b7c063d00 0x7f9b7c0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:16:50.085 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.035+0000 7f9b9bfff700 1 --2- 172.21.15.195:0/2275062480 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9b7c063d00 0x7f9b7c0661b0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c009fd0 tx=0x7f9b8c009060 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:16:50.085 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.039+0000 7f9b83fff700 1 -- 172.21.15.195:0/2275062480 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f9b9c155f80 con 0x7f9b9c0c34a0 2022-06-27T12:16:50.266 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.266+0000 7f9ba2409700 1 -- 172.21.15.195:0/2275062480 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9b9c155f80 con 0x7f9b9c0c34a0 2022-06-27T12:16:50.267 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.266+0000 7f9b83fff700 1 -- 172.21.15.195:0/2275062480 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+6132 (secure 0 0 0) 0x7f9b9c155f80 con 0x7f9b9c0c34a0 2022-06-27T12:16:50.269 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:16:50.270 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":107,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":7165,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/3551478822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":3551478822},{"type":"v1","addr":"172.21.15.202:6835","nonce":3551478822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":107}],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:15:52.408825+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6254":{"gid":6254,"name":"b","rank":0,"incarnation":102,"state":"up:active","state_seq":43,"addr":"172.21.15.202:6833/1317767101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1317767101},{"type":"v1","addr":"172.21.15.202:6833","nonce":1317767101}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:16:50.272 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.268+0000 7f9ba2409700 1 -- 172.21.15.195:0/2275062480 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9b7c063d00 msgr2=0x7f9b7c0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:16:50.273 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.268+0000 7f9ba2409700 1 --2- 172.21.15.195:0/2275062480 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9b7c063d00 0x7f9b7c0661b0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c009fd0 tx=0x7f9b8c009060 comp rx=0 tx=0).stop 2022-06-27T12:16:50.273 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.268+0000 7f9ba2409700 1 -- 172.21.15.195:0/2275062480 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9b9c0c34a0 msgr2=0x7f9b9c145780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:16:50.284 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.268+0000 7f9ba2409700 1 --2- 172.21.15.195:0/2275062480 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9b9c0c34a0 0x7f9b9c145780 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f9b9000c4a0 tx=0x7f9b90007c50 comp rx=0 tx=0).stop 2022-06-27T12:16:50.285 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.268+0000 7f9ba2409700 1 -- 172.21.15.195:0/2275062480 shutdown_connections 2022-06-27T12:16:50.285 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.268+0000 7f9ba2409700 1 --2- 172.21.15.195:0/2275062480 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9b7c063d00 0x7f9b7c0661b0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:50.285 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.268+0000 7f9ba2409700 1 --2- 172.21.15.195:0/2275062480 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9b9c147760 0x7f9b9c156260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:50.286 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.268+0000 7f9ba2409700 1 --2- 172.21.15.195:0/2275062480 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9b9c06e850 0x7f9b9c145240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:50.286 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.268+0000 7f9ba2409700 1 --2- 172.21.15.195:0/2275062480 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9b9c0c34a0 0x7f9b9c145780 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:50.287 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.268+0000 7f9ba2409700 1 -- 172.21.15.195:0/2275062480 >> 172.21.15.195:0/2275062480 conn(0x7f9b9c0c6350 msgr2=0x7f9b9c0c2cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:16:50.287 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.268+0000 7f9ba2409700 1 -- 172.21.15.195:0/2275062480 shutdown_connections 2022-06-27T12:16:50.288 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:50.268+0000 7f9ba2409700 1 -- 172.21.15.195:0/2275062480 wait complete. 2022-06-27T12:16:50.288 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 107 2022-06-27T12:16:50.292 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2022-06-27T12:16:52.294 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:16:52.428 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.428+0000 7fec67d6e700 1 Processor -- start 2022-06-27T12:16:52.429 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.428+0000 7fec67d6e700 1 -- start start 2022-06-27T12:16:52.441 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.429+0000 7fec67d6e700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fec6006fef0 0x7fec601570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:16:52.442 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.429+0000 7fec67d6e700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fec600c34a0 0x7fec600c2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:16:52.443 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.429+0000 7fec67d6e700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec6006fba0 con 0x7fec6006e850 2022-06-27T12:16:52.447 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.429+0000 7fec67d6e700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec601575e0 con 0x7fec600c34a0 2022-06-27T12:16:52.448 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.429+0000 7fec67d6e700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec6006f5e0 con 0x7fec6006fef0 2022-06-27T12:16:52.448 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.429+0000 7fec6630b700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fec600c34a0 0x7fec600c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:16:52.456 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.429+0000 7fec65b0a700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fec6006e850 0x7fec60069de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:58420/0 (socket says 172.21.15.195:58420) 2022-06-27T12:16:52.457 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.429+0000 7fec6630b700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fec600c34a0 0x7fec600c2920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:52236/0 (socket says 172.21.15.195:52236) 2022-06-27T12:16:52.463 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.429+0000 7fec65b0a700 1 -- 172.21.15.195:0/3695951961 learned_addr learned my addr 172.21.15.195:0/3695951961 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:16:52.464 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.429+0000 7fec65309700 1 --2- 172.21.15.195:0/3695951961 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fec6006fef0 0x7fec601570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:16:52.464 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.429+0000 7fec6630b700 1 -- 172.21.15.195:0/3695951961 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fec6006fef0 msgr2=0x7fec601570a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:16:52.464 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.429+0000 7fec6630b700 1 --2- 172.21.15.195:0/3695951961 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fec6006fef0 0x7fec601570a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:52.464 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.429+0000 7fec6630b700 1 -- 172.21.15.195:0/3695951961 >> v1:172.21.15.195:6789/0 conn(0x7fec6006e850 legacy=0x7fec60069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:16:52.465 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.429+0000 7fec64b08700 1 -- 172.21.15.195:0/3695951961 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3051633384 0 0) 0x7fec6006fba0 con 0x7fec6006e850 2022-06-27T12:16:52.470 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.429+0000 7fec6630b700 1 -- 172.21.15.195:0/3695951961 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec6006f8c0 con 0x7fec600c34a0 2022-06-27T12:16:52.470 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.429+0000 7fec6630b700 1 --2- 172.21.15.195:0/3695951961 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fec600c34a0 0x7fec600c2920 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7fec58009e60 tx=0x7fec58009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75bcd147a7add62e server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:16:52.471 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.430+0000 7fec64b08700 1 -- 172.21.15.195:0/3695951961 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec5800e070 con 0x7fec600c34a0 2022-06-27T12:16:52.471 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.430+0000 7fec64b08700 1 -- 172.21.15.195:0/3695951961 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec5800bcc0 con 0x7fec600c34a0 2022-06-27T12:16:52.471 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.430+0000 7fec64b08700 1 -- 172.21.15.195:0/3695951961 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec58012680 con 0x7fec600c34a0 2022-06-27T12:16:52.471 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.430+0000 7fec67d6e700 1 -- 172.21.15.195:0/3695951961 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fec600c34a0 msgr2=0x7fec600c2920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:16:52.472 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.430+0000 7fec67d6e700 1 --2- 172.21.15.195:0/3695951961 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fec600c34a0 0x7fec600c2920 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7fec58009e60 tx=0x7fec58009780 comp rx=0 tx=0).stop 2022-06-27T12:16:52.472 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.430+0000 7fec67d6e700 1 -- 172.21.15.195:0/3695951961 shutdown_connections 2022-06-27T12:16:52.472 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.430+0000 7fec67d6e700 1 --2- 172.21.15.195:0/3695951961 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fec600c34a0 0x7fec600c2920 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:52.472 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.430+0000 7fec67d6e700 1 --2- 172.21.15.195:0/3695951961 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fec6006fef0 0x7fec601570a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:52.473 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.430+0000 7fec67d6e700 1 -- 172.21.15.195:0/3695951961 >> 172.21.15.195:0/3695951961 conn(0x7fec600c6350 msgr2=0x7fec600c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:16:52.473 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.430+0000 7fec67d6e700 1 -- 172.21.15.195:0/3695951961 shutdown_connections 2022-06-27T12:16:52.473 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.430+0000 7fec67d6e700 1 -- 172.21.15.195:0/3695951961 wait complete. 2022-06-27T12:16:52.473 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.431+0000 7fec67d6e700 1 Processor -- start 2022-06-27T12:16:52.474 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.431+0000 7fec67d6e700 1 -- start start 2022-06-27T12:16:52.474 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.432+0000 7fec67d6e700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fec6006e850 0x7fec601447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:16:52.474 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.432+0000 7fec67d6e700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fec600c34a0 0x7fec60143040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:16:52.474 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.432+0000 7fec67d6e700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fec60154aa0 0x7fec60147900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:16:52.475 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.432+0000 7fec67d6e700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec60155c80 con 0x7fec6006e850 2022-06-27T12:16:52.475 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.432+0000 7fec67d6e700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec601459e0 con 0x7fec60154aa0 2022-06-27T12:16:52.475 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.432+0000 7fec67d6e700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec601450f0 con 0x7fec600c34a0 2022-06-27T12:16:52.475 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.432+0000 7fec65b0a700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fec6006e850 0x7fec601447b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:16:52.476 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.432+0000 7fec65b0a700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fec6006e850 0x7fec601447b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:52238/0 (socket says 172.21.15.195:52238) 2022-06-27T12:16:52.476 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.432+0000 7fec65b0a700 1 -- 172.21.15.195:0/2189429651 learned_addr learned my addr 172.21.15.195:0/2189429651 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:16:52.476 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.432+0000 7fec65309700 1 --2- 172.21.15.195:0/2189429651 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fec600c34a0 0x7fec60143040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:16:52.476 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.432+0000 7fec6630b700 1 --2- 172.21.15.195:0/2189429651 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fec60154aa0 0x7fec60147900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:16:52.477 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.432+0000 7fec65b0a700 1 -- 172.21.15.195:0/2189429651 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fec600c34a0 msgr2=0x7fec60143040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:16:52.477 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.432+0000 7fec65b0a700 1 --2- 172.21.15.195:0/2189429651 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fec600c34a0 0x7fec60143040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:52.477 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.432+0000 7fec65b0a700 1 -- 172.21.15.195:0/2189429651 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fec60154aa0 msgr2=0x7fec60147900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:16:52.477 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.432+0000 7fec65b0a700 1 --2- 172.21.15.195:0/2189429651 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fec60154aa0 0x7fec60147900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:52.478 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.432+0000 7fec65b0a700 1 -- 172.21.15.195:0/2189429651 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec58009a20 con 0x7fec6006e850 2022-06-27T12:16:52.478 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.433+0000 7fec65b0a700 1 --2- 172.21.15.195:0/2189429651 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fec6006e850 0x7fec601447b0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fec5c00c4c0 tx=0x7fec5c007dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:16:52.478 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.433+0000 7fec4dffb700 1 -- 172.21.15.195:0/2189429651 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec5c007430 con 0x7fec6006e850 2022-06-27T12:16:52.478 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.433+0000 7fec4dffb700 1 -- 172.21.15.195:0/2189429651 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec5c007590 con 0x7fec6006e850 2022-06-27T12:16:52.479 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.433+0000 7fec4dffb700 1 -- 172.21.15.195:0/2189429651 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec5c004920 con 0x7fec6006e850 2022-06-27T12:16:52.479 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.433+0000 7fec67d6e700 1 -- 172.21.15.195:0/2189429651 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec601436d0 con 0x7fec6006e850 2022-06-27T12:16:52.479 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.433+0000 7fec67d6e700 1 -- 172.21.15.195:0/2189429651 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec60145670 con 0x7fec6006e850 2022-06-27T12:16:52.479 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.434+0000 7fec67d6e700 1 -- 172.21.15.195:0/2189429651 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec600722c0 con 0x7fec6006e850 2022-06-27T12:16:52.480 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.434+0000 7fec4dffb700 1 -- 172.21.15.195:0/2189429651 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fec5c010030 con 0x7fec6006e850 2022-06-27T12:16:52.483 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.435+0000 7fec4dffb700 1 --2- 172.21.15.195:0/2189429651 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fec40063d30 0x7fec400661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:16:52.483 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.435+0000 7fec4dffb700 1 -- 172.21.15.195:0/2189429651 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7578+0+0 (secure 0 0 0) 0x7fec5c0809f0 con 0x7fec6006e850 2022-06-27T12:16:52.484 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.435+0000 7fec65309700 1 --2- 172.21.15.195:0/2189429651 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fec40063d30 0x7fec400661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:16:52.484 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.435+0000 7fec65309700 1 --2- 172.21.15.195:0/2189429651 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fec40063d30 0x7fec400661e0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fec50006fd0 tx=0x7fec50008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:16:52.484 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.439+0000 7fec4dffb700 1 -- 172.21.15.195:0/2189429651 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fec5c082e60 con 0x7fec6006e850 2022-06-27T12:16:52.663 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.663+0000 7fec67d6e700 1 -- 172.21.15.195:0/2189429651 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fec6006f240 con 0x7fec6006e850 2022-06-27T12:16:52.665 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.664+0000 7fec4dffb700 1 -- 172.21.15.195:0/2189429651 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+6132 (secure 0 0 0) 0x7fec5c04f010 con 0x7fec6006e850 2022-06-27T12:16:52.665 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:16:52.666 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":107,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":7165,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/3551478822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":3551478822},{"type":"v1","addr":"172.21.15.202:6835","nonce":3551478822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":107}],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:15:52.408825+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6254":{"gid":6254,"name":"b","rank":0,"incarnation":102,"state":"up:active","state_seq":43,"addr":"172.21.15.202:6833/1317767101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1317767101},{"type":"v1","addr":"172.21.15.202:6833","nonce":1317767101}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:16:52.667 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.665+0000 7fec67d6e700 1 -- 172.21.15.195:0/2189429651 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fec40063d30 msgr2=0x7fec400661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:16:52.667 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.665+0000 7fec67d6e700 1 --2- 172.21.15.195:0/2189429651 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fec40063d30 0x7fec400661e0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fec50006fd0 tx=0x7fec50008040 comp rx=0 tx=0).stop 2022-06-27T12:16:52.668 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.665+0000 7fec67d6e700 1 -- 172.21.15.195:0/2189429651 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fec6006e850 msgr2=0x7fec601447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:16:52.668 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.665+0000 7fec67d6e700 1 --2- 172.21.15.195:0/2189429651 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fec6006e850 0x7fec601447b0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fec5c00c4c0 tx=0x7fec5c007dd0 comp rx=0 tx=0).stop 2022-06-27T12:16:52.668 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.666+0000 7fec67d6e700 1 -- 172.21.15.195:0/2189429651 shutdown_connections 2022-06-27T12:16:52.668 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.666+0000 7fec67d6e700 1 --2- 172.21.15.195:0/2189429651 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fec40063d30 0x7fec400661e0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:52.669 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.666+0000 7fec67d6e700 1 --2- 172.21.15.195:0/2189429651 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fec6006e850 0x7fec601447b0 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:52.669 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.666+0000 7fec67d6e700 1 --2- 172.21.15.195:0/2189429651 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fec60154aa0 0x7fec60147900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:52.669 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.666+0000 7fec67d6e700 1 --2- 172.21.15.195:0/2189429651 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fec600c34a0 0x7fec60143040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:52.669 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.666+0000 7fec67d6e700 1 -- 172.21.15.195:0/2189429651 >> 172.21.15.195:0/2189429651 conn(0x7fec600c6350 msgr2=0x7fec600c6720 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:16:52.670 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.666+0000 7fec67d6e700 1 -- 172.21.15.195:0/2189429651 shutdown_connections 2022-06-27T12:16:52.670 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:52.666+0000 7fec67d6e700 1 -- 172.21.15.195:0/2189429651 wait complete. 2022-06-27T12:16:52.671 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 107 2022-06-27T12:16:52.680 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2022-06-27T12:16:54.682 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:16:54.820 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:54.817+0000 7fc401a82700 1 Processor -- start 2022-06-27T12:16:54.820 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:54.817+0000 7fc401a82700 1 -- start start 2022-06-27T12:16:54.821 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:54.817+0000 7fc401a82700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fc3fc06f640 0x7fc3fc156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:16:54.821 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:54.817+0000 7fc401a82700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fc3fc0c4160 0x7fc3fc0c1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:16:54.822 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:54.817+0000 7fc401a82700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3fc0c1a20 con 0x7fc3fc0c39e0 2022-06-27T12:16:54.822 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:54.817+0000 7fc401a82700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3fc0c1130 con 0x7fc3fc06f640 2022-06-27T12:16:54.822 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:54.817+0000 7fc401a82700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3fc06ea90 con 0x7fc3fc0c4160 2022-06-27T12:16:54.823 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:54.818+0000 7fc3fb7fe700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fc3fc06f640 0x7fc3fc156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:16:54.823 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:54.818+0000 7fc3fb7fe700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fc3fc06f640 0x7fc3fc156ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:52248/0 (socket says 172.21.15.195:52248) 2022-06-27T12:16:54.823 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:54.818+0000 7fc3fb7fe700 1 -- 172.21.15.195:0/1522405947 learned_addr learned my addr 172.21.15.195:0/1522405947 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:16:54.823 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:54.818+0000 7fc3faffd700 1 --2- 172.21.15.195:0/1522405947 >> v2:172.21.15.202:3300/0 conn(0x7fc3fc0c4160 0x7fc3fc0c1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:16:54.824 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:54.818+0000 7fc3fb7fe700 1 -- 172.21.15.195:0/1522405947 >> v2:172.21.15.202:3300/0 conn(0x7fc3fc0c4160 msgr2=0x7fc3fc0c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:16:54.824 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:54.818+0000 7fc3fb7fe700 1 --2- 172.21.15.195:0/1522405947 >> v2:172.21.15.202:3300/0 conn(0x7fc3fc0c4160 0x7fc3fc0c1eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:54.824 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:54.818+0000 7fc3fb7fe700 1 -- 172.21.15.195:0/1522405947 >> v1:172.21.15.202:6789/0 conn(0x7fc3fc0c39e0 legacy=0x7fc3fc1535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:16:54.824 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:54.818+0000 7fc3fb7fe700 1 -- 172.21.15.195:0/1522405947 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3fc06ed70 con 0x7fc3fc06f640 2022-06-27T12:16:54.825 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:54.818+0000 7fc3faffd700 1 --2- 172.21.15.195:0/1522405947 >> v2:172.21.15.202:3300/0 conn(0x7fc3fc0c4160 0x7fc3fc0c1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:16:54.825 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:54.818+0000 7fc3fb7fe700 1 --2- 172.21.15.195:0/1522405947 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc3fc06f640 0x7fc3fc156ed0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7fc3fc06dd70 tx=0x7fc3f0009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2db73de5fbdd301 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:16:54.825 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:54.818+0000 7fc3fa7fc700 1 -- 172.21.15.195:0/1522405947 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f000b6d0 con 0x7fc3fc06f640 2022-06-27T12:16:54.825 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:54.818+0000 7fc3fa7fc700 1 -- 172.21.15.195:0/1522405947 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3f000b830 con 0x7fc3fc06f640 2022-06-27T12:16:54.826 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:54.818+0000 7fc3fa7fc700 1 -- 172.21.15.195:0/1522405947 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f000bbd0 con 0x7fc3fc06f640 2022-06-27T12:16:54.826 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:54.819+0000 7fc401a82700 1 -- 172.21.15.195:0/1522405947 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc3fc06f640 msgr2=0x7fc3fc156ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:16:54.826 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:54.819+0000 7fc401a82700 1 --2- 172.21.15.195:0/1522405947 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc3fc06f640 0x7fc3fc156ed0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7fc3fc06dd70 tx=0x7fc3f0009120 comp rx=0 tx=0).stop 2022-06-27T12:16:54.826 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:54.819+0000 7fc401a82700 1 -- 172.21.15.195:0/1522405947 shutdown_connections 2022-06-27T12:16:54.827 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:54.819+0000 7fc401a82700 1 --2- 172.21.15.195:0/1522405947 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc3fc06f640 0x7fc3fc156ed0 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:54.827 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:54.819+0000 7fc401a82700 1 --2- 172.21.15.195:0/1522405947 >> v2:172.21.15.202:3300/0 conn(0x7fc3fc0c4160 0x7fc3fc0c1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:54.827 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:54.819+0000 7fc401a82700 1 -- 172.21.15.195:0/1522405947 >> 172.21.15.195:0/1522405947 conn(0x7fc3fc0c6350 msgr2=0x7fc3fc0c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:16:54.827 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:54.819+0000 7fc401a82700 1 -- 172.21.15.195:0/1522405947 shutdown_connections 2022-06-27T12:16:54.828 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:54.819+0000 7fc401a82700 1 -- 172.21.15.195:0/1522405947 wait complete. 2022-06-27T12:16:54.828 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:54.820+0000 7fc401a82700 1 Processor -- start 2022-06-27T12:16:54.828 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:54.820+0000 7fc401a82700 1 -- start start 2022-06-27T12:16:54.832 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:54.821+0000 7fc401a82700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc3fc06f640 0x7fc3fc158fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:16:54.833 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:54.821+0000 7fc401a82700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc3fc0c39e0 0x7fc3fc16a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:16:54.833 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:54.821+0000 7fc401a82700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc3fc0c4160 0x7fc3fc152680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:16:54.833 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:54.821+0000 7fc401a82700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3fc0bd830 con 0x7fc3fc06f640 2022-06-27T12:16:54.834 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:54.821+0000 7fc401a82700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3fc0bdb10 con 0x7fc3fc0c4160 2022-06-27T12:16:54.834 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:54.821+0000 7fc401a82700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3fc14ef70 con 0x7fc3fc0c39e0 2022-06-27T12:16:54.834 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:54.821+0000 7fc3fb7fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc3fc06f640 0x7fc3fc158fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:16:54.835 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:54.821+0000 7fc3fb7fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc3fc06f640 0x7fc3fc158fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:52254/0 (socket says 172.21.15.195:52254) 2022-06-27T12:16:54.835 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:54.821+0000 7fc3fb7fe700 1 -- 172.21.15.195:0/4023689981 learned_addr learned my addr 172.21.15.195:0/4023689981 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:16:54.835 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:54.821+0000 7fc3faffd700 1 --2- 172.21.15.195:0/4023689981 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc3fc0c39e0 0x7fc3fc16a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:16:54.836 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:54.821+0000 7fc3fbfff700 1 --2- 172.21.15.195:0/4023689981 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc3fc0c4160 0x7fc3fc152680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:16:54.836 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:54.821+0000 7fc3fb7fe700 1 -- 172.21.15.195:0/4023689981 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc3fc0c39e0 msgr2=0x7fc3fc16a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:16:54.836 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:54.821+0000 7fc3fb7fe700 1 --2- 172.21.15.195:0/4023689981 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc3fc0c39e0 0x7fc3fc16a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:54.836 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:54.821+0000 7fc3fb7fe700 1 -- 172.21.15.195:0/4023689981 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc3fc0c4160 msgr2=0x7fc3fc152680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:16:54.837 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:54.821+0000 7fc3fb7fe700 1 --2- 172.21.15.195:0/4023689981 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc3fc0c4160 0x7fc3fc152680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:54.837 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:54.821+0000 7fc3fb7fe700 1 -- 172.21.15.195:0/4023689981 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3f00093a0 con 0x7fc3fc06f640 2022-06-27T12:16:54.837 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:54.821+0000 7fc3fb7fe700 1 --2- 172.21.15.195:0/4023689981 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc3fc06f640 0x7fc3fc158fd0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7fc3f0009f80 tx=0x7fc3f0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:16:54.837 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:54.821+0000 7fc3e37fe700 1 -- 172.21.15.195:0/4023689981 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f0003730 con 0x7fc3fc06f640 2022-06-27T12:16:54.838 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:54.822+0000 7fc3e37fe700 1 -- 172.21.15.195:0/4023689981 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3f0003890 con 0x7fc3fc06f640 2022-06-27T12:16:54.838 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:54.822+0000 7fc401a82700 1 -- 172.21.15.195:0/4023689981 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3fc14f250 con 0x7fc3fc06f640 2022-06-27T12:16:54.838 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:54.822+0000 7fc3e37fe700 1 -- 172.21.15.195:0/4023689981 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f0003c50 con 0x7fc3fc06f640 2022-06-27T12:16:54.839 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:54.822+0000 7fc401a82700 1 -- 172.21.15.195:0/4023689981 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3fc14f810 con 0x7fc3fc06f640 2022-06-27T12:16:54.839 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:54.823+0000 7fc401a82700 1 -- 172.21.15.195:0/4023689981 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3fc0722c0 con 0x7fc3fc06f640 2022-06-27T12:16:54.839 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:54.823+0000 7fc3e37fe700 1 -- 172.21.15.195:0/4023689981 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fc3f002b030 con 0x7fc3fc06f640 2022-06-27T12:16:54.839 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:54.823+0000 7fc3e37fe700 1 --2- 172.21.15.195:0/4023689981 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc3dc063d30 0x7fc3dc0661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:16:54.840 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:54.823+0000 7fc3e37fe700 1 -- 172.21.15.195:0/4023689981 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7578+0+0 (secure 0 0 0) 0x7fc3f0082530 con 0x7fc3fc06f640 2022-06-27T12:16:54.840 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:54.823+0000 7fc3faffd700 1 --2- 172.21.15.195:0/4023689981 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc3dc063d30 0x7fc3dc0661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:16:54.840 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:54.824+0000 7fc3faffd700 1 --2- 172.21.15.195:0/4023689981 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc3dc063d30 0x7fc3dc0661e0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fc3fc161600 tx=0x7fc3ec008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:16:54.840 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:54.828+0000 7fc3e37fe700 1 -- 172.21.15.195:0/4023689981 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fc3f0050b30 con 0x7fc3fc06f640 2022-06-27T12:16:55.057 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:55.056+0000 7fc401a82700 1 -- 172.21.15.195:0/4023689981 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc3fc06d180 con 0x7fc3fc06f640 2022-06-27T12:16:55.057 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:55.056+0000 7fc3e37fe700 1 -- 172.21.15.195:0/4023689981 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+6132 (secure 0 0 0) 0x7fc3f0029090 con 0x7fc3fc06f640 2022-06-27T12:16:55.058 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:16:55.058 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":107,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":7165,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/3551478822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":3551478822},{"type":"v1","addr":"172.21.15.202:6835","nonce":3551478822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":107}],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:15:52.408825+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6254":{"gid":6254,"name":"b","rank":0,"incarnation":102,"state":"up:active","state_seq":43,"addr":"172.21.15.202:6833/1317767101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1317767101},{"type":"v1","addr":"172.21.15.202:6833","nonce":1317767101}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:16:55.059 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:55.058+0000 7fc401a82700 1 -- 172.21.15.195:0/4023689981 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc3dc063d30 msgr2=0x7fc3dc0661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:16:55.059 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:55.058+0000 7fc401a82700 1 --2- 172.21.15.195:0/4023689981 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc3dc063d30 0x7fc3dc0661e0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fc3fc161600 tx=0x7fc3ec008040 comp rx=0 tx=0).stop 2022-06-27T12:16:55.060 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:55.058+0000 7fc401a82700 1 -- 172.21.15.195:0/4023689981 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc3fc06f640 msgr2=0x7fc3fc158fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:16:55.060 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:55.058+0000 7fc401a82700 1 --2- 172.21.15.195:0/4023689981 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc3fc06f640 0x7fc3fc158fd0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7fc3f0009f80 tx=0x7fc3f0017000 comp rx=0 tx=0).stop 2022-06-27T12:16:55.060 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:55.058+0000 7fc401a82700 1 -- 172.21.15.195:0/4023689981 shutdown_connections 2022-06-27T12:16:55.066 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:55.058+0000 7fc401a82700 1 --2- 172.21.15.195:0/4023689981 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc3dc063d30 0x7fc3dc0661e0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:55.066 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:55.058+0000 7fc401a82700 1 --2- 172.21.15.195:0/4023689981 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc3fc06f640 0x7fc3fc158fd0 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:55.066 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:55.058+0000 7fc401a82700 1 --2- 172.21.15.195:0/4023689981 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc3fc0c4160 0x7fc3fc152680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:55.067 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:55.058+0000 7fc401a82700 1 --2- 172.21.15.195:0/4023689981 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc3fc0c39e0 0x7fc3fc16a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:55.067 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:55.058+0000 7fc401a82700 1 -- 172.21.15.195:0/4023689981 >> 172.21.15.195:0/4023689981 conn(0x7fc3fc0c6350 msgr2=0x7fc3fc1559f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:16:55.067 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:55.059+0000 7fc401a82700 1 -- 172.21.15.195:0/4023689981 shutdown_connections 2022-06-27T12:16:55.067 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:55.059+0000 7fc401a82700 1 -- 172.21.15.195:0/4023689981 wait complete. 2022-06-27T12:16:55.068 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 107 2022-06-27T12:16:55.073 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2022-06-27T12:16:57.078 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:16:57.216 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.211+0000 7f9e49636700 1 Processor -- start 2022-06-27T12:16:57.216 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.211+0000 7f9e49636700 1 -- start start 2022-06-27T12:16:57.217 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.212+0000 7f9e49636700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f9e440c7480 0x7f9e440c4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:16:57.217 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.212+0000 7f9e49636700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f9e440683c0 0x7f9e44069350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:16:57.217 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.212+0000 7f9e49636700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e441573b0 con 0x7f9e44068f80 2022-06-27T12:16:57.217 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.212+0000 7f9e49636700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e44068840 con 0x7f9e440c7480 2022-06-27T12:16:57.218 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.212+0000 7f9e49636700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e440c46c0 con 0x7f9e440683c0 2022-06-27T12:16:57.218 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.212+0000 7f9e427fc700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f9e440c7480 0x7f9e440c4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:16:57.218 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.212+0000 7f9e427fc700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f9e440c7480 0x7f9e440c4b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:52264/0 (socket says 172.21.15.195:52264) 2022-06-27T12:16:57.218 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.212+0000 7f9e427fc700 1 -- 172.21.15.195:0/232463828 learned_addr learned my addr 172.21.15.195:0/232463828 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:16:57.219 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.212+0000 7f9e437fe700 1 --2- 172.21.15.195:0/232463828 >> v2:172.21.15.202:3300/0 conn(0x7f9e440683c0 0x7f9e44069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:16:57.219 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.212+0000 7f9e427fc700 1 -- 172.21.15.195:0/232463828 >> v2:172.21.15.202:3300/0 conn(0x7f9e440683c0 msgr2=0x7f9e44069350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:16:57.219 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.212+0000 7f9e427fc700 1 --2- 172.21.15.195:0/232463828 >> v2:172.21.15.202:3300/0 conn(0x7f9e440683c0 0x7f9e44069350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:57.219 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.212+0000 7f9e427fc700 1 -- 172.21.15.195:0/232463828 >> v1:172.21.15.195:6789/0 conn(0x7f9e44068f80 legacy=0x7f9e44156460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:16:57.219 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.212+0000 7f9e427fc700 1 -- 172.21.15.195:0/232463828 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e44156b70 con 0x7f9e440c7480 2022-06-27T12:16:57.229 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.212+0000 7f9e427fc700 1 --2- 172.21.15.195:0/232463828 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9e440c7480 0x7f9e440c4b50 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f9e440705a0 tx=0x7f9e34009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9772d60d0abceda6 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:16:57.230 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.212+0000 7f9e41ffb700 1 -- 172.21.15.195:0/232463828 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e3400b640 con 0x7f9e440c7480 2022-06-27T12:16:57.230 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.213+0000 7f9e41ffb700 1 -- 172.21.15.195:0/232463828 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e3400b7a0 con 0x7f9e440c7480 2022-06-27T12:16:57.230 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.213+0000 7f9e41ffb700 1 -- 172.21.15.195:0/232463828 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e3400bb40 con 0x7f9e440c7480 2022-06-27T12:16:57.231 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.213+0000 7f9e49636700 1 -- 172.21.15.195:0/232463828 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9e440c7480 msgr2=0x7f9e440c4b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:16:57.239 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.213+0000 7f9e49636700 1 --2- 172.21.15.195:0/232463828 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9e440c7480 0x7f9e440c4b50 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f9e440705a0 tx=0x7f9e34009120 comp rx=0 tx=0).stop 2022-06-27T12:16:57.240 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.213+0000 7f9e49636700 1 -- 172.21.15.195:0/232463828 shutdown_connections 2022-06-27T12:16:57.243 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.213+0000 7f9e49636700 1 --2- 172.21.15.195:0/232463828 >> v2:172.21.15.202:3300/0 conn(0x7f9e440683c0 0x7f9e44069350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:57.247 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.213+0000 7f9e49636700 1 --2- 172.21.15.195:0/232463828 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9e440c7480 0x7f9e440c4b50 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:57.248 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.213+0000 7f9e49636700 1 -- 172.21.15.195:0/232463828 >> 172.21.15.195:0/232463828 conn(0x7f9e440ccbd0 msgr2=0x7f9e440ccfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:16:57.260 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.213+0000 7f9e49636700 1 -- 172.21.15.195:0/232463828 shutdown_connections 2022-06-27T12:16:57.260 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.213+0000 7f9e49636700 1 -- 172.21.15.195:0/232463828 wait complete. 2022-06-27T12:16:57.261 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.215+0000 7f9e49636700 1 Processor -- start 2022-06-27T12:16:57.261 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.215+0000 7f9e49636700 1 -- start start 2022-06-27T12:16:57.262 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.215+0000 7f9e49636700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9e440683c0 0x7f9e440c6680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:16:57.263 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.215+0000 7f9e49636700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9e44068f80 0x7f9e4414bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:16:57.264 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.215+0000 7f9e49636700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9e440c7480 0x7f9e4414c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:16:57.264 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.215+0000 7f9e49636700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e4414d790 con 0x7f9e440c7480 2022-06-27T12:16:57.264 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.215+0000 7f9e49636700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e44167180 con 0x7f9e44068f80 2022-06-27T12:16:57.265 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.215+0000 7f9e49636700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e4415bc10 con 0x7f9e440683c0 2022-06-27T12:16:57.268 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.215+0000 7f9e437fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9e440c7480 0x7f9e4414c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:16:57.269 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.215+0000 7f9e437fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9e440c7480 0x7f9e4414c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:52272/0 (socket says 172.21.15.195:52272) 2022-06-27T12:16:57.269 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.215+0000 7f9e437fe700 1 -- 172.21.15.195:0/2359177363 learned_addr learned my addr 172.21.15.195:0/2359177363 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:16:57.270 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.216+0000 7f9e427fc700 1 --2- 172.21.15.195:0/2359177363 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9e44068f80 0x7f9e4414bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:16:57.270 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.216+0000 7f9e42ffd700 1 --2- 172.21.15.195:0/2359177363 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9e440683c0 0x7f9e440c6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:16:57.271 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.216+0000 7f9e437fe700 1 -- 172.21.15.195:0/2359177363 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9e440683c0 msgr2=0x7f9e440c6680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:16:57.271 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.216+0000 7f9e437fe700 1 --2- 172.21.15.195:0/2359177363 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9e440683c0 0x7f9e440c6680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:57.272 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.216+0000 7f9e437fe700 1 -- 172.21.15.195:0/2359177363 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9e44068f80 msgr2=0x7f9e4414bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:16:57.272 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.216+0000 7f9e437fe700 1 --2- 172.21.15.195:0/2359177363 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9e44068f80 0x7f9e4414bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:57.272 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.216+0000 7f9e437fe700 1 -- 172.21.15.195:0/2359177363 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e340093a0 con 0x7f9e440c7480 2022-06-27T12:16:57.273 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.216+0000 7f9e42ffd700 1 --2- 172.21.15.195:0/2359177363 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9e440683c0 0x7f9e440c6680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T12:16:57.273 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.216+0000 7f9e437fe700 1 --2- 172.21.15.195:0/2359177363 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9e440c7480 0x7f9e4414c460 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f9e3c007f20 tx=0x7f9e3c009860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:16:57.274 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.216+0000 7f9e3affd700 1 -- 172.21.15.195:0/2359177363 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e3c00eca0 con 0x7f9e440c7480 2022-06-27T12:16:57.275 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.216+0000 7f9e3affd700 1 -- 172.21.15.195:0/2359177363 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e3c00ee20 con 0x7f9e440c7480 2022-06-27T12:16:57.277 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.216+0000 7f9e49636700 1 -- 172.21.15.195:0/2359177363 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e4415bef0 con 0x7f9e440c7480 2022-06-27T12:16:57.277 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.216+0000 7f9e3affd700 1 -- 172.21.15.195:0/2359177363 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e3c014670 con 0x7f9e440c7480 2022-06-27T12:16:57.277 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.216+0000 7f9e49636700 1 -- 172.21.15.195:0/2359177363 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e4415c4b0 con 0x7f9e440c7480 2022-06-27T12:16:57.278 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.217+0000 7f9e49636700 1 -- 172.21.15.195:0/2359177363 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e4415c9d0 con 0x7f9e440c7480 2022-06-27T12:16:57.279 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.217+0000 7f9e3affd700 1 -- 172.21.15.195:0/2359177363 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f9e3c012050 con 0x7f9e440c7480 2022-06-27T12:16:57.279 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.218+0000 7f9e3affd700 1 --2- 172.21.15.195:0/2359177363 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9e28063d00 0x7f9e280661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:16:57.280 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.218+0000 7f9e3affd700 1 -- 172.21.15.195:0/2359177363 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7578+0+0 (secure 0 0 0) 0x7f9e3c081470 con 0x7f9e440c7480 2022-06-27T12:16:57.280 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.218+0000 7f9e42ffd700 1 --2- 172.21.15.195:0/2359177363 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9e28063d00 0x7f9e280661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:16:57.281 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.218+0000 7f9e42ffd700 1 --2- 172.21.15.195:0/2359177363 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9e28063d00 0x7f9e280661b0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f9e2c0016f0 tx=0x7f9e2c009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:16:57.281 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.222+0000 7f9e3affd700 1 -- 172.21.15.195:0/2359177363 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f9e4415c9d0 con 0x7f9e440c7480 2022-06-27T12:16:57.452 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.451+0000 7f9e49636700 1 -- 172.21.15.195:0/2359177363 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e4415c9d0 con 0x7f9e440c7480 2022-06-27T12:16:57.453 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.451+0000 7f9e3affd700 1 -- 172.21.15.195:0/2359177363 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+6132 (secure 0 0 0) 0x7f9e4415c9d0 con 0x7f9e440c7480 2022-06-27T12:16:57.454 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:16:57.454 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":107,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":7165,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/3551478822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":3551478822},{"type":"v1","addr":"172.21.15.202:6835","nonce":3551478822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":107}],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:15:52.408825+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6254":{"gid":6254,"name":"b","rank":0,"incarnation":102,"state":"up:active","state_seq":43,"addr":"172.21.15.202:6833/1317767101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1317767101},{"type":"v1","addr":"172.21.15.202:6833","nonce":1317767101}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:16:57.462 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.453+0000 7f9e49636700 1 -- 172.21.15.195:0/2359177363 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9e28063d00 msgr2=0x7f9e280661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:16:57.463 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.453+0000 7f9e49636700 1 --2- 172.21.15.195:0/2359177363 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9e28063d00 0x7f9e280661b0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f9e2c0016f0 tx=0x7f9e2c009040 comp rx=0 tx=0).stop 2022-06-27T12:16:57.463 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.453+0000 7f9e49636700 1 -- 172.21.15.195:0/2359177363 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9e440c7480 msgr2=0x7f9e4414c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:16:57.463 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.453+0000 7f9e49636700 1 --2- 172.21.15.195:0/2359177363 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9e440c7480 0x7f9e4414c460 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f9e3c007f20 tx=0x7f9e3c009860 comp rx=0 tx=0).stop 2022-06-27T12:16:57.463 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.453+0000 7f9e49636700 1 -- 172.21.15.195:0/2359177363 shutdown_connections 2022-06-27T12:16:57.464 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.453+0000 7f9e49636700 1 --2- 172.21.15.195:0/2359177363 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f9e28063d00 0x7f9e280661b0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:57.464 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.453+0000 7f9e49636700 1 --2- 172.21.15.195:0/2359177363 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9e440c7480 0x7f9e4414c460 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:57.464 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.453+0000 7f9e49636700 1 --2- 172.21.15.195:0/2359177363 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f9e440683c0 0x7f9e440c6680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:57.465 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.453+0000 7f9e49636700 1 --2- 172.21.15.195:0/2359177363 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f9e44068f80 0x7f9e4414bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:57.465 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.453+0000 7f9e49636700 1 -- 172.21.15.195:0/2359177363 >> 172.21.15.195:0/2359177363 conn(0x7f9e440ccbd0 msgr2=0x7f9e440c8570 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:16:57.466 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.453+0000 7f9e49636700 1 -- 172.21.15.195:0/2359177363 shutdown_connections 2022-06-27T12:16:57.466 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:57.453+0000 7f9e49636700 1 -- 172.21.15.195:0/2359177363 wait complete. 2022-06-27T12:16:57.466 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 107 2022-06-27T12:16:57.468 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2022-06-27T12:16:59.473 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:16:59.629 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.622+0000 7f38db7c1700 1 Processor -- start 2022-06-27T12:16:59.629 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.622+0000 7f38db7c1700 1 -- start start 2022-06-27T12:16:59.640 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.623+0000 7f38db7c1700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f38d414e490 0x7f38d406cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:16:59.640 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.623+0000 7f38db7c1700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38d40c33b0 con 0x7f38d40c3f50 2022-06-27T12:16:59.641 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.623+0000 7f38db7c1700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38d406d520 con 0x7f38d414e860 2022-06-27T12:16:59.641 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.623+0000 7f38db7c1700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38d406f220 con 0x7f38d414e490 2022-06-27T12:16:59.642 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.623+0000 7f38d9d5e700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f38d40c3f50 0x7f38d4157890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:58470/0 (socket says 172.21.15.195:58470) 2022-06-27T12:16:59.643 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.623+0000 7f38d9d5e700 1 -- 172.21.15.195:0/106483150 learned_addr learned my addr 172.21.15.195:0/106483150 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:16:59.643 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.623+0000 7f38d955d700 1 --2- 172.21.15.195:0/106483150 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f38d414e490 0x7f38d406cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:16:59.644 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.623+0000 7f38c3fff700 1 -- 172.21.15.195:0/106483150 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2873182848 0 0) 0x7f38d40c33b0 con 0x7f38d40c3f50 2022-06-27T12:16:59.644 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.624+0000 7f38c3fff700 1 -- 172.21.15.195:0/106483150 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38bc003430 con 0x7f38d40c3f50 2022-06-27T12:16:59.669 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.624+0000 7f38c3fff700 1 -- 172.21.15.195:0/106483150 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3678243611 0 0) 0x7f38d406d520 con 0x7f38d414e860 2022-06-27T12:16:59.669 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.624+0000 7f38c3fff700 1 -- 172.21.15.195:0/106483150 --> v1:172.21.15.202:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38d40c33b0 con 0x7f38d414e860 2022-06-27T12:16:59.670 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.624+0000 7f38d955d700 1 -- 172.21.15.195:0/106483150 >> v1:172.21.15.202:6789/0 conn(0x7f38d414e860 legacy=0x7f38d40c3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:16:59.683 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.624+0000 7f38d955d700 1 -- 172.21.15.195:0/106483150 >> v1:172.21.15.195:6789/0 conn(0x7f38d40c3f50 legacy=0x7f38d4157890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:16:59.684 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.624+0000 7f38c3fff700 1 -- 172.21.15.195:0/106483150 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 782674529 0 0) 0x7f38bc003430 con 0x7f38d40c3f50 2022-06-27T12:16:59.684 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.624+0000 7f38d955d700 1 -- 172.21.15.195:0/106483150 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38d41542c0 con 0x7f38d414e490 2022-06-27T12:16:59.684 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.624+0000 7f38d955d700 1 --2- 172.21.15.195:0/106483150 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f38d414e490 0x7f38d406cfe0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f38c40097d0 tx=0x7f38c4009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab5e01eba42acd8b server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:16:59.684 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.624+0000 7f38c3fff700 1 -- 172.21.15.195:0/106483150 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38cc0017f0 con 0x7f38d414e490 2022-06-27T12:16:59.694 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.624+0000 7f38c3fff700 1 -- 172.21.15.195:0/106483150 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38c4004030 con 0x7f38d414e490 2022-06-27T12:16:59.694 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.624+0000 7f38c3fff700 1 -- 172.21.15.195:0/106483150 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38c40043b0 con 0x7f38d414e490 2022-06-27T12:16:59.694 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.625+0000 7f38db7c1700 1 -- 172.21.15.195:0/106483150 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f38d414e490 msgr2=0x7f38d406cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:16:59.695 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.625+0000 7f38db7c1700 1 --2- 172.21.15.195:0/106483150 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f38d414e490 0x7f38d406cfe0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f38c40097d0 tx=0x7f38c4009060 comp rx=0 tx=0).stop 2022-06-27T12:16:59.695 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.625+0000 7f38db7c1700 1 -- 172.21.15.195:0/106483150 shutdown_connections 2022-06-27T12:16:59.698 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.625+0000 7f38db7c1700 1 --2- 172.21.15.195:0/106483150 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f38d414e490 0x7f38d406cfe0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:59.698 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.625+0000 7f38db7c1700 1 -- 172.21.15.195:0/106483150 >> 172.21.15.195:0/106483150 conn(0x7f38d4063af0 msgr2=0x7f38d4063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:16:59.699 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.625+0000 7f38db7c1700 1 -- 172.21.15.195:0/106483150 shutdown_connections 2022-06-27T12:16:59.699 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.625+0000 7f38db7c1700 1 -- 172.21.15.195:0/106483150 wait complete. 2022-06-27T12:16:59.700 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.626+0000 7f38db7c1700 1 Processor -- start 2022-06-27T12:16:59.701 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.626+0000 7f38db7c1700 1 -- start start 2022-06-27T12:16:59.701 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.627+0000 7f38db7c1700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f38d40c3f50 0x7f38d40caaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:16:59.702 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.627+0000 7f38db7c1700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f38d414e490 0x7f38d4156400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:16:59.703 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.627+0000 7f38db7c1700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f38d414e860 0x7f38d4169cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:16:59.706 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.627+0000 7f38db7c1700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38d4167720 con 0x7f38d414e860 2022-06-27T12:16:59.708 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.627+0000 7f38db7c1700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38d406b010 con 0x7f38d40c3f50 2022-06-27T12:16:59.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.627+0000 7f38db7c1700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38d41671b0 con 0x7f38d414e490 2022-06-27T12:16:59.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.627+0000 7f38d9d5e700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f38d414e860 0x7f38d4169cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:16:59.710 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.627+0000 7f38d9d5e700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f38d414e860 0x7f38d4169cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:52288/0 (socket says 172.21.15.195:52288) 2022-06-27T12:16:59.710 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.627+0000 7f38d9d5e700 1 -- 172.21.15.195:0/757807063 learned_addr learned my addr 172.21.15.195:0/757807063 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:16:59.718 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.627+0000 7f38d8d5c700 1 --2- 172.21.15.195:0/757807063 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f38d414e490 0x7f38d4156400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:16:59.718 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.627+0000 7f38d955d700 1 --2- 172.21.15.195:0/757807063 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f38d40c3f50 0x7f38d40caaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:16:59.719 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.627+0000 7f38d9d5e700 1 -- 172.21.15.195:0/757807063 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f38d414e490 msgr2=0x7f38d4156400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:16:59.719 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.627+0000 7f38d9d5e700 1 --2- 172.21.15.195:0/757807063 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f38d414e490 0x7f38d4156400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:59.720 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.627+0000 7f38d9d5e700 1 -- 172.21.15.195:0/757807063 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f38d40c3f50 msgr2=0x7f38d40caaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:16:59.720 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.627+0000 7f38d9d5e700 1 --2- 172.21.15.195:0/757807063 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f38d40c3f50 0x7f38d40caaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:59.732 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.627+0000 7f38d9d5e700 1 -- 172.21.15.195:0/757807063 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38c40092e0 con 0x7f38d414e860 2022-06-27T12:16:59.735 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.627+0000 7f38d955d700 1 --2- 172.21.15.195:0/757807063 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f38d40c3f50 0x7f38d40caaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T12:16:59.735 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.627+0000 7f38d9d5e700 1 --2- 172.21.15.195:0/757807063 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f38d414e860 0x7f38d4169cc0 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f38cc00cb60 tx=0x7f38cc0018e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:16:59.736 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.627+0000 7f38c17fa700 1 -- 172.21.15.195:0/757807063 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38cc004c80 con 0x7f38d414e860 2022-06-27T12:16:59.736 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.628+0000 7f38c17fa700 1 -- 172.21.15.195:0/757807063 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38cc004de0 con 0x7f38d414e860 2022-06-27T12:16:59.737 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.628+0000 7f38c17fa700 1 -- 172.21.15.195:0/757807063 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38cc007380 con 0x7f38d414e860 2022-06-27T12:16:59.737 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.628+0000 7f38db7c1700 1 -- 172.21.15.195:0/757807063 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38d4167440 con 0x7f38d414e860 2022-06-27T12:16:59.738 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.628+0000 7f38db7c1700 1 -- 172.21.15.195:0/757807063 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38d4069460 con 0x7f38d414e860 2022-06-27T12:16:59.738 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.629+0000 7f38db7c1700 1 -- 172.21.15.195:0/757807063 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38d40722c0 con 0x7f38d414e860 2022-06-27T12:16:59.739 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.629+0000 7f38c17fa700 1 -- 172.21.15.195:0/757807063 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f38cc010070 con 0x7f38d414e860 2022-06-27T12:16:59.739 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.629+0000 7f38c17fa700 1 --2- 172.21.15.195:0/757807063 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f38b4063d30 0x7f38b40661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:16:59.740 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.629+0000 7f38c17fa700 1 -- 172.21.15.195:0/757807063 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7578+0+0 (secure 0 0 0) 0x7f38cc0803d0 con 0x7f38d414e860 2022-06-27T12:16:59.740 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.630+0000 7f38d955d700 1 --2- 172.21.15.195:0/757807063 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f38b4063d30 0x7f38b40661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:16:59.740 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.630+0000 7f38d955d700 1 --2- 172.21.15.195:0/757807063 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f38b4063d30 0x7f38b40661e0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f38c4000e50 tx=0x7f38c4015000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:16:59.741 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.634+0000 7f38c17fa700 1 -- 172.21.15.195:0/757807063 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f38cc04e960 con 0x7f38d414e860 2022-06-27T12:16:59.871 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.861+0000 7f38db7c1700 1 -- 172.21.15.195:0/757807063 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f38d406d0f0 con 0x7f38d414e860 2022-06-27T12:16:59.872 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.862+0000 7f38c17fa700 1 -- 172.21.15.195:0/757807063 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+6132 (secure 0 0 0) 0x7f38cc018020 con 0x7f38d414e860 2022-06-27T12:16:59.873 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:16:59.873 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":107,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":7165,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/3551478822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":3551478822},{"type":"v1","addr":"172.21.15.202:6835","nonce":3551478822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":107}],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:15:52.408825+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6254":{"gid":6254,"name":"b","rank":0,"incarnation":102,"state":"up:active","state_seq":43,"addr":"172.21.15.202:6833/1317767101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1317767101},{"type":"v1","addr":"172.21.15.202:6833","nonce":1317767101}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:16:59.889 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.864+0000 7f38db7c1700 1 -- 172.21.15.195:0/757807063 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f38b4063d30 msgr2=0x7f38b40661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:16:59.889 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.864+0000 7f38db7c1700 1 --2- 172.21.15.195:0/757807063 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f38b4063d30 0x7f38b40661e0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f38c4000e50 tx=0x7f38c4015000 comp rx=0 tx=0).stop 2022-06-27T12:16:59.894 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.864+0000 7f38db7c1700 1 -- 172.21.15.195:0/757807063 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f38d414e860 msgr2=0x7f38d4169cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:16:59.895 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.864+0000 7f38db7c1700 1 --2- 172.21.15.195:0/757807063 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f38d414e860 0x7f38d4169cc0 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f38cc00cb60 tx=0x7f38cc0018e0 comp rx=0 tx=0).stop 2022-06-27T12:16:59.895 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.864+0000 7f38db7c1700 1 -- 172.21.15.195:0/757807063 shutdown_connections 2022-06-27T12:16:59.910 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.864+0000 7f38db7c1700 1 --2- 172.21.15.195:0/757807063 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f38b4063d30 0x7f38b40661e0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:59.910 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.864+0000 7f38db7c1700 1 --2- 172.21.15.195:0/757807063 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f38d414e860 0x7f38d4169cc0 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:59.910 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.864+0000 7f38db7c1700 1 --2- 172.21.15.195:0/757807063 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f38d40c3f50 0x7f38d40caaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:59.910 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.864+0000 7f38db7c1700 1 --2- 172.21.15.195:0/757807063 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f38d414e490 0x7f38d4156400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:16:59.911 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.864+0000 7f38db7c1700 1 -- 172.21.15.195:0/757807063 >> 172.21.15.195:0/757807063 conn(0x7f38d4063af0 msgr2=0x7f38d4063ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:16:59.911 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.864+0000 7f38db7c1700 1 -- 172.21.15.195:0/757807063 shutdown_connections 2022-06-27T12:16:59.913 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:16:59.864+0000 7f38db7c1700 1 -- 172.21.15.195:0/757807063 wait complete. 2022-06-27T12:16:59.913 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 107 2022-06-27T12:16:59.919 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2022-06-27T12:17:01.920 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:17:02.054 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.054+0000 7feb37f05700 1 Processor -- start 2022-06-27T12:17:02.067 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.055+0000 7feb37f05700 1 -- start start 2022-06-27T12:17:02.068 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.055+0000 7feb37f05700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7feb30150500 0x7feb30063af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:02.068 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.055+0000 7feb37f05700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7feb3006a450 0x7feb300c48d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:02.068 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.055+0000 7feb37f05700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7feb301508d0 0x7feb300bbf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:02.069 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.055+0000 7feb37f05700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb30064030 con 0x7feb3006a450 2022-06-27T12:17:02.069 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.055+0000 7feb37f05700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb300c4160 con 0x7feb301508d0 2022-06-27T12:17:02.069 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.055+0000 7feb37f05700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb300c4440 con 0x7feb30150500 2022-06-27T12:17:02.070 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.055+0000 7feb354a0700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7feb3006a450 0x7feb300c48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:02.070 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.056+0000 7feb354a0700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7feb3006a450 0x7feb300c48d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:52292/0 (socket says 172.21.15.195:52292) 2022-06-27T12:17:02.070 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.056+0000 7feb354a0700 1 -- 172.21.15.195:0/1665617099 learned_addr learned my addr 172.21.15.195:0/1665617099 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:17:02.070 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.056+0000 7feb364a2700 1 --2- 172.21.15.195:0/1665617099 >> v2:172.21.15.202:3300/0 conn(0x7feb301508d0 0x7feb300bbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:02.071 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.056+0000 7feb35ca1700 1 --2- 172.21.15.195:0/1665617099 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7feb30150500 0x7feb30063af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:02.071 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.056+0000 7feb354a0700 1 -- 172.21.15.195:0/1665617099 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7feb30150500 msgr2=0x7feb30063af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:17:02.071 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.056+0000 7feb354a0700 1 --2- 172.21.15.195:0/1665617099 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7feb30150500 0x7feb30063af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:02.071 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.056+0000 7feb354a0700 1 -- 172.21.15.195:0/1665617099 >> v2:172.21.15.202:3300/0 conn(0x7feb301508d0 msgr2=0x7feb300bbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:17:02.072 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.056+0000 7feb354a0700 1 --2- 172.21.15.195:0/1665617099 >> v2:172.21.15.202:3300/0 conn(0x7feb301508d0 0x7feb300bbf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:02.072 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.056+0000 7feb354a0700 1 -- 172.21.15.195:0/1665617099 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb3015a800 con 0x7feb3006a450 2022-06-27T12:17:02.072 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.056+0000 7feb354a0700 1 --2- 172.21.15.195:0/1665617099 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7feb3006a450 0x7feb300c48d0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7feb3006b400 tx=0x7feb2c009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be2258318911e4ad server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:17:02.073 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.056+0000 7feb34c9f700 1 -- 172.21.15.195:0/1665617099 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb2c00b640 con 0x7feb3006a450 2022-06-27T12:17:02.073 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.056+0000 7feb34c9f700 1 -- 172.21.15.195:0/1665617099 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb2c00b7a0 con 0x7feb3006a450 2022-06-27T12:17:02.073 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.056+0000 7feb34c9f700 1 -- 172.21.15.195:0/1665617099 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb2c00bb40 con 0x7feb3006a450 2022-06-27T12:17:02.073 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.057+0000 7feb37f05700 1 -- 172.21.15.195:0/1665617099 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7feb3006a450 msgr2=0x7feb300c48d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:02.074 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.057+0000 7feb37f05700 1 --2- 172.21.15.195:0/1665617099 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7feb3006a450 0x7feb300c48d0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7feb3006b400 tx=0x7feb2c009120 comp rx=0 tx=0).stop 2022-06-27T12:17:02.074 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.057+0000 7feb37f05700 1 -- 172.21.15.195:0/1665617099 shutdown_connections 2022-06-27T12:17:02.074 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.057+0000 7feb37f05700 1 --2- 172.21.15.195:0/1665617099 >> v2:172.21.15.202:3300/0 conn(0x7feb301508d0 0x7feb300bbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:02.074 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.057+0000 7feb37f05700 1 --2- 172.21.15.195:0/1665617099 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7feb30150500 0x7feb30063af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:02.075 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.057+0000 7feb37f05700 1 --2- 172.21.15.195:0/1665617099 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7feb3006a450 0x7feb300c48d0 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:02.075 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.057+0000 7feb37f05700 1 -- 172.21.15.195:0/1665617099 >> 172.21.15.195:0/1665617099 conn(0x7feb300c9b00 msgr2=0x7feb300c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:17:02.075 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.057+0000 7feb37f05700 1 -- 172.21.15.195:0/1665617099 shutdown_connections 2022-06-27T12:17:02.075 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.057+0000 7feb37f05700 1 -- 172.21.15.195:0/1665617099 wait complete. 2022-06-27T12:17:02.076 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.058+0000 7feb37f05700 1 Processor -- start 2022-06-27T12:17:02.076 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.058+0000 7feb37f05700 1 -- start start 2022-06-27T12:17:02.076 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.059+0000 7feb37f05700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7feb3006a450 0x7feb300c7300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:02.077 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.059+0000 7feb37f05700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7feb30150500 0x7feb300be080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:02.077 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.059+0000 7feb37f05700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7feb301508d0 0x7feb300be5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:02.077 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.059+0000 7feb37f05700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb300c1260 con 0x7feb301508d0 2022-06-27T12:17:02.078 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.059+0000 7feb37f05700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb300c0ed0 con 0x7feb3006a450 2022-06-27T12:17:02.078 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.059+0000 7feb37f05700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb300bd0d0 con 0x7feb30150500 2022-06-27T12:17:02.079 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.059+0000 7feb364a2700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7feb301508d0 0x7feb300be5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:02.079 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.059+0000 7feb364a2700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7feb301508d0 0x7feb300be5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:52300/0 (socket says 172.21.15.195:52300) 2022-06-27T12:17:02.080 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.059+0000 7feb364a2700 1 -- 172.21.15.195:0/1976935590 learned_addr learned my addr 172.21.15.195:0/1976935590 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:17:02.080 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.059+0000 7feb35ca1700 1 --2- 172.21.15.195:0/1976935590 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7feb3006a450 0x7feb300c7300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:02.081 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.059+0000 7feb354a0700 1 --2- 172.21.15.195:0/1976935590 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7feb30150500 0x7feb300be080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:02.081 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.059+0000 7feb364a2700 1 -- 172.21.15.195:0/1976935590 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7feb30150500 msgr2=0x7feb300be080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:02.081 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.059+0000 7feb364a2700 1 --2- 172.21.15.195:0/1976935590 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7feb30150500 0x7feb300be080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:02.082 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.059+0000 7feb364a2700 1 -- 172.21.15.195:0/1976935590 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7feb3006a450 msgr2=0x7feb300c7300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:02.082 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.059+0000 7feb364a2700 1 --2- 172.21.15.195:0/1976935590 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7feb3006a450 0x7feb300c7300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:02.083 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.059+0000 7feb364a2700 1 -- 172.21.15.195:0/1976935590 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb2c0093a0 con 0x7feb301508d0 2022-06-27T12:17:02.083 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.059+0000 7feb364a2700 1 --2- 172.21.15.195:0/1976935590 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7feb301508d0 0x7feb300be5c0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7feb2800ce10 tx=0x7feb280077d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:17:02.084 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.059+0000 7feb1dffb700 1 -- 172.21.15.195:0/1976935590 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb2800a9d0 con 0x7feb301508d0 2022-06-27T12:17:02.084 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.059+0000 7feb1dffb700 1 -- 172.21.15.195:0/1976935590 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb2800ab30 con 0x7feb301508d0 2022-06-27T12:17:02.085 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.059+0000 7feb1dffb700 1 -- 172.21.15.195:0/1976935590 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb280038a0 con 0x7feb301508d0 2022-06-27T12:17:02.085 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.059+0000 7feb37f05700 1 -- 172.21.15.195:0/1976935590 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb300c8020 con 0x7feb301508d0 2022-06-27T12:17:02.085 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.060+0000 7feb37f05700 1 -- 172.21.15.195:0/1976935590 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb300c03f0 con 0x7feb301508d0 2022-06-27T12:17:02.090 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.061+0000 7feb37f05700 1 -- 172.21.15.195:0/1976935590 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb300722c0 con 0x7feb301508d0 2022-06-27T12:17:02.090 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.061+0000 7feb1dffb700 1 -- 172.21.15.195:0/1976935590 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7feb2800f400 con 0x7feb301508d0 2022-06-27T12:17:02.091 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.061+0000 7feb1dffb700 1 --2- 172.21.15.195:0/1976935590 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7feb10063d00 0x7feb100661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:02.091 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.061+0000 7feb35ca1700 1 --2- 172.21.15.195:0/1976935590 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7feb10063d00 0x7feb100661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:02.092 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.061+0000 7feb1dffb700 1 -- 172.21.15.195:0/1976935590 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7664+0+0 (secure 0 0 0) 0x7feb28013070 con 0x7feb301508d0 2022-06-27T12:17:02.092 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.062+0000 7feb35ca1700 1 --2- 172.21.15.195:0/1976935590 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7feb10063d00 0x7feb100661b0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7feb20006fd0 tx=0x7feb20008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:17:02.092 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.066+0000 7feb1dffb700 1 -- 172.21.15.195:0/1976935590 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7feb2804f8a0 con 0x7feb301508d0 2022-06-27T12:17:02.291 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.291+0000 7feb37f05700 1 -- 172.21.15.195:0/1976935590 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feb300ba8e0 con 0x7feb301508d0 2022-06-27T12:17:02.292 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.291+0000 7feb1dffb700 1 -- 172.21.15.195:0/1976935590 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5370 (secure 0 0 0) 0x7feb28052ec0 con 0x7feb301508d0 2022-06-27T12:17:02.293 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:17:02.293 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:17:00.220308+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7165},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7165":{"gid":7165,"name":"d","rank":0,"incarnation":108,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6835/3551478822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":3551478822},{"type":"v1","addr":"172.21.15.202:6835","nonce":3551478822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:17:02.296 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.293+0000 7feb37f05700 1 -- 172.21.15.195:0/1976935590 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7feb10063d00 msgr2=0x7feb100661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:02.296 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.293+0000 7feb37f05700 1 --2- 172.21.15.195:0/1976935590 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7feb10063d00 0x7feb100661b0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7feb20006fd0 tx=0x7feb20008040 comp rx=0 tx=0).stop 2022-06-27T12:17:02.296 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.293+0000 7feb37f05700 1 -- 172.21.15.195:0/1976935590 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7feb301508d0 msgr2=0x7feb300be5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:02.297 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.293+0000 7feb37f05700 1 --2- 172.21.15.195:0/1976935590 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7feb301508d0 0x7feb300be5c0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7feb2800ce10 tx=0x7feb280077d0 comp rx=0 tx=0).stop 2022-06-27T12:17:02.297 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.293+0000 7feb37f05700 1 -- 172.21.15.195:0/1976935590 shutdown_connections 2022-06-27T12:17:02.298 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.293+0000 7feb37f05700 1 --2- 172.21.15.195:0/1976935590 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7feb10063d00 0x7feb100661b0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:02.298 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.293+0000 7feb37f05700 1 --2- 172.21.15.195:0/1976935590 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7feb301508d0 0x7feb300be5c0 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:02.299 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.293+0000 7feb37f05700 1 --2- 172.21.15.195:0/1976935590 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7feb3006a450 0x7feb300c7300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:02.299 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.293+0000 7feb37f05700 1 --2- 172.21.15.195:0/1976935590 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7feb30150500 0x7feb300be080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:02.300 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.293+0000 7feb37f05700 1 -- 172.21.15.195:0/1976935590 >> 172.21.15.195:0/1976935590 conn(0x7feb300c9b00 msgr2=0x7feb3014fbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:17:02.300 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.293+0000 7feb37f05700 1 -- 172.21.15.195:0/1976935590 shutdown_connections 2022-06-27T12:17:02.301 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.293+0000 7feb37f05700 1 -- 172.21.15.195:0/1976935590 wait complete. 2022-06-27T12:17:02.301 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 108 2022-06-27T12:17:02.313 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2022-06-27T12:17:02.314 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2022-06-27T12:17:02.314 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:17:02.448 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.434+0000 7fb856353700 1 Processor -- start 2022-06-27T12:17:02.448 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.434+0000 7fb856353700 1 -- start start 2022-06-27T12:17:02.449 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.435+0000 7fb856353700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb8500c4160 0x7fb850156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:02.449 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.435+0000 7fb856353700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fb8500c39e0 0x7fb8501535a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:02.449 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.435+0000 7fb856353700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8500c1a20 con 0x7fb85006f640 2022-06-27T12:17:02.449 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.435+0000 7fb856353700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8500c1130 con 0x7fb8500c39e0 2022-06-27T12:17:02.450 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.435+0000 7fb856353700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb85006ea90 con 0x7fb8500c4160 2022-06-27T12:17:02.450 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.435+0000 7fb8548f0700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fb8500c39e0 0x7fb8501535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:02.450 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.435+0000 7fb84ffff700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fb85006f640 0x7fb8500c1eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:58492/0 (socket says 172.21.15.195:58492) 2022-06-27T12:17:02.450 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.435+0000 7fb84ffff700 1 -- 172.21.15.195:0/545343068 learned_addr learned my addr 172.21.15.195:0/545343068 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:17:02.451 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.435+0000 7fb84f7fe700 1 --2- 172.21.15.195:0/545343068 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb8500c4160 0x7fb850156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:02.451 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.435+0000 7fb84effd700 1 -- 172.21.15.195:0/545343068 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3098576877 0 0) 0x7fb8500c1a20 con 0x7fb85006f640 2022-06-27T12:17:02.451 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.435+0000 7fb84effd700 1 -- 172.21.15.195:0/545343068 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb83c003430 con 0x7fb85006f640 2022-06-27T12:17:02.451 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.435+0000 7fb8548f0700 1 -- 172.21.15.195:0/545343068 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb8500c4160 msgr2=0x7fb850156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:17:02.452 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.435+0000 7fb8548f0700 1 --2- 172.21.15.195:0/545343068 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb8500c4160 0x7fb850156ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:02.452 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.435+0000 7fb8548f0700 1 -- 172.21.15.195:0/545343068 >> v1:172.21.15.195:6789/0 conn(0x7fb85006f640 legacy=0x7fb8500c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:02.452 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.435+0000 7fb8548f0700 1 -- 172.21.15.195:0/545343068 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb85006ed70 con 0x7fb8500c39e0 2022-06-27T12:17:02.452 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.436+0000 7fb8548f0700 1 --2- 172.21.15.195:0/545343068 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb8500c39e0 0x7fb8501535a0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7fb848009e60 tx=0x7fb848009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=add646d5e632bb5d server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:17:02.453 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.436+0000 7fb84effd700 1 -- 172.21.15.195:0/545343068 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb84800e070 con 0x7fb8500c39e0 2022-06-27T12:17:02.453 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.436+0000 7fb84effd700 1 -- 172.21.15.195:0/545343068 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb84800bb40 con 0x7fb8500c39e0 2022-06-27T12:17:02.453 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.436+0000 7fb84effd700 1 -- 172.21.15.195:0/545343068 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb848012430 con 0x7fb8500c39e0 2022-06-27T12:17:02.453 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.436+0000 7fb856353700 1 -- 172.21.15.195:0/545343068 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb8500c39e0 msgr2=0x7fb8501535a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:02.454 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.436+0000 7fb856353700 1 --2- 172.21.15.195:0/545343068 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb8500c39e0 0x7fb8501535a0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7fb848009e60 tx=0x7fb848009780 comp rx=0 tx=0).stop 2022-06-27T12:17:02.454 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.436+0000 7fb856353700 1 -- 172.21.15.195:0/545343068 shutdown_connections 2022-06-27T12:17:02.455 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.436+0000 7fb856353700 1 --2- 172.21.15.195:0/545343068 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb8500c39e0 0x7fb8501535a0 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:02.455 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.436+0000 7fb856353700 1 --2- 172.21.15.195:0/545343068 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb8500c4160 0x7fb850156ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:02.455 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.436+0000 7fb856353700 1 -- 172.21.15.195:0/545343068 >> 172.21.15.195:0/545343068 conn(0x7fb8500c6350 msgr2=0x7fb8500c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:17:02.455 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.436+0000 7fb856353700 1 -- 172.21.15.195:0/545343068 shutdown_connections 2022-06-27T12:17:02.456 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.436+0000 7fb856353700 1 -- 172.21.15.195:0/545343068 wait complete. 2022-06-27T12:17:02.456 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.437+0000 7fb856353700 1 Processor -- start 2022-06-27T12:17:02.456 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.438+0000 7fb856353700 1 -- start start 2022-06-27T12:17:02.457 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.438+0000 7fb856353700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb85006f640 0x7fb850153550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:02.457 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.438+0000 7fb856353700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb8500c39e0 0x7fb8500bee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:02.457 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.438+0000 7fb856353700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb8500c4160 0x7fb8500bf340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:02.458 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.438+0000 7fb856353700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8500c04d0 con 0x7fb8500c39e0 2022-06-27T12:17:02.459 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.438+0000 7fb856353700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb850155890 con 0x7fb8500c4160 2022-06-27T12:17:02.459 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.438+0000 7fb856353700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb850155350 con 0x7fb85006f640 2022-06-27T12:17:02.459 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.438+0000 7fb84f7fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb8500c39e0 0x7fb8500bee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:02.459 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.439+0000 7fb84f7fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb8500c39e0 0x7fb8500bee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:52310/0 (socket says 172.21.15.195:52310) 2022-06-27T12:17:02.460 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.439+0000 7fb84f7fe700 1 -- 172.21.15.195:0/2773065767 learned_addr learned my addr 172.21.15.195:0/2773065767 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:17:02.460 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.439+0000 7fb84ffff700 1 --2- 172.21.15.195:0/2773065767 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb85006f640 0x7fb850153550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:02.460 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.439+0000 7fb8548f0700 1 --2- 172.21.15.195:0/2773065767 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb8500c4160 0x7fb8500bf340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:02.461 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.439+0000 7fb84f7fe700 1 -- 172.21.15.195:0/2773065767 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb85006f640 msgr2=0x7fb850153550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:02.461 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.439+0000 7fb84f7fe700 1 --2- 172.21.15.195:0/2773065767 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb85006f640 0x7fb850153550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:02.461 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.439+0000 7fb84f7fe700 1 -- 172.21.15.195:0/2773065767 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb8500c4160 msgr2=0x7fb8500bf340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:02.461 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.439+0000 7fb84f7fe700 1 --2- 172.21.15.195:0/2773065767 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb8500c4160 0x7fb8500bf340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:02.462 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.439+0000 7fb84f7fe700 1 -- 172.21.15.195:0/2773065767 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb848009a20 con 0x7fb8500c39e0 2022-06-27T12:17:02.462 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.439+0000 7fb84ffff700 1 --2- 172.21.15.195:0/2773065767 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb85006f640 0x7fb850153550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T12:17:02.463 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.439+0000 7fb8548f0700 1 --2- 172.21.15.195:0/2773065767 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb8500c4160 0x7fb8500bf340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T12:17:02.463 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.439+0000 7fb84f7fe700 1 --2- 172.21.15.195:0/2773065767 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb8500c39e0 0x7fb8500bee00 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7fb84000e900 tx=0x7fb840009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:17:02.463 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.439+0000 7fb837fff700 1 -- 172.21.15.195:0/2773065767 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb84000ee40 con 0x7fb8500c39e0 2022-06-27T12:17:02.464 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.439+0000 7fb837fff700 1 -- 172.21.15.195:0/2773065767 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb84000c850 con 0x7fb8500c39e0 2022-06-27T12:17:02.464 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.439+0000 7fb856353700 1 -- 172.21.15.195:0/2773065767 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8501555f0 con 0x7fb8500c39e0 2022-06-27T12:17:02.465 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.439+0000 7fb837fff700 1 -- 172.21.15.195:0/2773065767 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb84000cbf0 con 0x7fb8500c39e0 2022-06-27T12:17:02.465 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.439+0000 7fb856353700 1 -- 172.21.15.195:0/2773065767 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb850156840 con 0x7fb8500c39e0 2022-06-27T12:17:02.465 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.440+0000 7fb856353700 1 -- 172.21.15.195:0/2773065767 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8500722c0 con 0x7fb8500c39e0 2022-06-27T12:17:02.466 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.441+0000 7fb837fff700 1 -- 172.21.15.195:0/2773065767 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fb840023070 con 0x7fb8500c39e0 2022-06-27T12:17:02.466 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.441+0000 7fb837fff700 1 --2- 172.21.15.195:0/2773065767 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb830063d30 0x7fb8300661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:02.467 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.441+0000 7fb837fff700 1 -- 172.21.15.195:0/2773065767 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7664+0+0 (secure 0 0 0) 0x7fb840081160 con 0x7fb8500c39e0 2022-06-27T12:17:02.467 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.441+0000 7fb84ffff700 1 --2- 172.21.15.195:0/2773065767 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb830063d30 0x7fb8300661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:02.467 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.441+0000 7fb84ffff700 1 --2- 172.21.15.195:0/2773065767 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb830063d30 0x7fb8300661e0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fb844005790 tx=0x7fb844009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:17:02.468 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.445+0000 7fb837fff700 1 -- 172.21.15.195:0/2773065767 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fb84004f710 con 0x7fb8500c39e0 2022-06-27T12:17:02.672 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.672+0000 7fb856353700 1 -- 172.21.15.195:0/2773065767 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb85006f240 con 0x7fb8500c39e0 2022-06-27T12:17:02.673 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.672+0000 7fb837fff700 1 -- 172.21.15.195:0/2773065767 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5370 (secure 0 0 0) 0x7fb840021090 con 0x7fb8500c39e0 2022-06-27T12:17:02.674 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:17:02.675 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:17:00.220308+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7165},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7165":{"gid":7165,"name":"d","rank":0,"incarnation":108,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6835/3551478822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":3551478822},{"type":"v1","addr":"172.21.15.202:6835","nonce":3551478822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:17:02.678 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.674+0000 7fb856353700 1 -- 172.21.15.195:0/2773065767 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb830063d30 msgr2=0x7fb8300661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:02.678 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.674+0000 7fb856353700 1 --2- 172.21.15.195:0/2773065767 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb830063d30 0x7fb8300661e0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fb844005790 tx=0x7fb844009040 comp rx=0 tx=0).stop 2022-06-27T12:17:02.679 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.674+0000 7fb856353700 1 -- 172.21.15.195:0/2773065767 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb8500c39e0 msgr2=0x7fb8500bee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:02.679 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.674+0000 7fb856353700 1 --2- 172.21.15.195:0/2773065767 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb8500c39e0 0x7fb8500bee00 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7fb84000e900 tx=0x7fb840009060 comp rx=0 tx=0).stop 2022-06-27T12:17:02.680 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.674+0000 7fb856353700 1 -- 172.21.15.195:0/2773065767 shutdown_connections 2022-06-27T12:17:02.680 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.674+0000 7fb856353700 1 --2- 172.21.15.195:0/2773065767 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fb830063d30 0x7fb8300661e0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:02.681 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.674+0000 7fb856353700 1 --2- 172.21.15.195:0/2773065767 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fb8500c4160 0x7fb8500bf340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:02.681 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.674+0000 7fb856353700 1 --2- 172.21.15.195:0/2773065767 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fb85006f640 0x7fb850153550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:02.681 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.674+0000 7fb856353700 1 --2- 172.21.15.195:0/2773065767 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb8500c39e0 0x7fb8500bee00 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:02.682 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.674+0000 7fb856353700 1 -- 172.21.15.195:0/2773065767 >> 172.21.15.195:0/2773065767 conn(0x7fb8500c6350 msgr2=0x7fb8500c1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:17:02.682 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.674+0000 7fb856353700 1 -- 172.21.15.195:0/2773065767 shutdown_connections 2022-06-27T12:17:02.683 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:02.674+0000 7fb856353700 1 -- 172.21.15.195:0/2773065767 wait complete. 2022-06-27T12:17:02.683 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 108 2022-06-27T12:17:02.689 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 108, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-06-27T11:47:52.358095+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 108, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7165": { "addr": "172.21.15.202:6835/3551478822", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6834", "nonce": 3551478822, "type": "v2" }, { "addr": "172.21.15.202:6835", "nonce": 3551478822, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7165, "incarnation": 108, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 65, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-06-27T12:17:00.220308+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7165 } } } ], "standbys": [ { "addr": "172.21.15.202:6837/3305006841", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6836", "nonce": 3305006841, "type": "v2" }, { "addr": "172.21.15.202:6837", "nonce": 3305006841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6835/2060709822", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6834", "nonce": 2060709822, "type": "v2" }, { "addr": "172.21.15.195:6835", "nonce": 2060709822, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6838/1598751652", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6836", "nonce": 1598751652, "type": "v2" }, { "addr": "172.21.15.195:6838", "nonce": 1598751652, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4275, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6839/1313800661", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6837", "nonce": 1313800661, "type": "v2" }, { "addr": "172.21.15.195:6839", "nonce": 1313800661, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4278, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-06-27T12:17:04.698 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:17:04.850 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:04.843+0000 7f123cb7f700 1 Processor -- start 2022-06-27T12:17:04.851 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:04.843+0000 7f123cb7f700 1 -- start start 2022-06-27T12:17:04.862 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:04.844+0000 7f123cb7f700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1238067ed0 0x7f12380c7440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:04.862 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:04.844+0000 7f123cb7f700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f1238069390 0x7f12381502b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:04.863 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:04.844+0000 7f123cb7f700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12380c43d0 con 0x7f1238156670 2022-06-27T12:17:04.863 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:04.844+0000 7f123cb7f700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1238068430 con 0x7f1238067ed0 2022-06-27T12:17:04.865 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:04.844+0000 7f123cb7f700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1238068710 con 0x7f1238069390 2022-06-27T12:17:04.865 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:04.845+0000 7f1235d9b700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1238067ed0 0x7f12380c7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:04.866 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:04.845+0000 7f1235d9b700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1238067ed0 0x7f12380c7440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:52322/0 (socket says 172.21.15.195:52322) 2022-06-27T12:17:04.867 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:04.845+0000 7f1235d9b700 1 -- 172.21.15.195:0/2821948270 learned_addr learned my addr 172.21.15.195:0/2821948270 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:17:04.867 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:04.845+0000 7f1236d9d700 1 --2- 172.21.15.195:0/2821948270 >> v2:172.21.15.202:3300/0 conn(0x7f1238069390 0x7f12381502b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:04.868 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:04.845+0000 7f1235d9b700 1 -- 172.21.15.195:0/2821948270 >> v2:172.21.15.202:3300/0 conn(0x7f1238069390 msgr2=0x7f12381502b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:17:04.868 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:04.845+0000 7f1235d9b700 1 --2- 172.21.15.195:0/2821948270 >> v2:172.21.15.202:3300/0 conn(0x7f1238069390 0x7f12381502b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:04.869 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:04.845+0000 7f1235d9b700 1 -- 172.21.15.195:0/2821948270 >> v1:172.21.15.202:6789/0 conn(0x7f1238156670 legacy=0x7f12380c4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:04.869 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:04.845+0000 7f1235d9b700 1 -- 172.21.15.195:0/2821948270 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12380689f0 con 0x7f1238067ed0 2022-06-27T12:17:04.870 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:04.845+0000 7f1235d9b700 1 --2- 172.21.15.195:0/2821948270 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1238067ed0 0x7f12380c7440 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f12380c5d60 tx=0x7f1228009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ce13bf9facea051 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:17:04.870 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:04.845+0000 7f123559a700 1 -- 172.21.15.195:0/2821948270 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f122800b640 con 0x7f1238067ed0 2022-06-27T12:17:04.870 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:04.845+0000 7f123559a700 1 -- 172.21.15.195:0/2821948270 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f122800b7a0 con 0x7f1238067ed0 2022-06-27T12:17:04.875 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:04.845+0000 7f123559a700 1 -- 172.21.15.195:0/2821948270 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f122800bb40 con 0x7f1238067ed0 2022-06-27T12:17:04.876 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:04.846+0000 7f123cb7f700 1 -- 172.21.15.195:0/2821948270 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1238067ed0 msgr2=0x7f12380c7440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:04.876 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:04.846+0000 7f123cb7f700 1 --2- 172.21.15.195:0/2821948270 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1238067ed0 0x7f12380c7440 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f12380c5d60 tx=0x7f1228009120 comp rx=0 tx=0).stop 2022-06-27T12:17:04.877 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:04.846+0000 7f123cb7f700 1 -- 172.21.15.195:0/2821948270 shutdown_connections 2022-06-27T12:17:04.877 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:04.846+0000 7f123cb7f700 1 --2- 172.21.15.195:0/2821948270 >> v2:172.21.15.202:3300/0 conn(0x7f1238069390 0x7f12381502b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:04.877 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:04.846+0000 7f123cb7f700 1 --2- 172.21.15.195:0/2821948270 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1238067ed0 0x7f12380c7440 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:04.878 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:04.846+0000 7f123cb7f700 1 -- 172.21.15.195:0/2821948270 >> 172.21.15.195:0/2821948270 conn(0x7f1238069e00 msgr2=0x7f123806a200 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:17:04.878 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:04.846+0000 7f123cb7f700 1 -- 172.21.15.195:0/2821948270 shutdown_connections 2022-06-27T12:17:04.879 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:04.846+0000 7f123cb7f700 1 -- 172.21.15.195:0/2821948270 wait complete. 2022-06-27T12:17:04.879 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:04.847+0000 7f123cb7f700 1 Processor -- start 2022-06-27T12:17:04.880 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:04.847+0000 7f123cb7f700 1 -- start start 2022-06-27T12:17:04.880 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:04.848+0000 7f123cb7f700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1238067ed0 0x7f1238150cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:04.881 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:04.848+0000 7f123cb7f700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1238069390 0x7f1238167930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:04.881 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:04.848+0000 7f123cb7f700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1238156670 0x7f12380b1e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:04.881 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:04.848+0000 7f123cb7f700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12380c2350 con 0x7f1238156670 2022-06-27T12:17:04.890 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:04.848+0000 7f123cb7f700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12380c25a0 con 0x7f1238067ed0 2022-06-27T12:17:04.891 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:04.848+0000 7f123cb7f700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12380c2880 con 0x7f1238069390 2022-06-27T12:17:04.891 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:04.848+0000 7f1236d9d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1238156670 0x7f12380b1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:04.899 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:04.848+0000 7f1236d9d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1238156670 0x7f12380b1e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:52330/0 (socket says 172.21.15.195:52330) 2022-06-27T12:17:04.900 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:04.848+0000 7f1236d9d700 1 -- 172.21.15.195:0/3134054983 learned_addr learned my addr 172.21.15.195:0/3134054983 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:17:04.900 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:04.848+0000 7f1235d9b700 1 --2- 172.21.15.195:0/3134054983 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1238069390 0x7f1238167930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:04.901 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:04.848+0000 7f123659c700 1 --2- 172.21.15.195:0/3134054983 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1238067ed0 0x7f1238150cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:04.901 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:04.848+0000 7f1236d9d700 1 -- 172.21.15.195:0/3134054983 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1238069390 msgr2=0x7f1238167930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:04.902 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:04.848+0000 7f1236d9d700 1 --2- 172.21.15.195:0/3134054983 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1238069390 0x7f1238167930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:04.902 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:04.848+0000 7f1236d9d700 1 -- 172.21.15.195:0/3134054983 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1238067ed0 msgr2=0x7f1238150cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:04.902 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:04.848+0000 7f1236d9d700 1 --2- 172.21.15.195:0/3134054983 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1238067ed0 0x7f1238150cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:04.903 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:04.848+0000 7f1236d9d700 1 -- 172.21.15.195:0/3134054983 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12280093a0 con 0x7f1238156670 2022-06-27T12:17:04.903 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:04.848+0000 7f1236d9d700 1 --2- 172.21.15.195:0/3134054983 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1238156670 0x7f12380b1e50 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f1230009fa0 tx=0x7f1230009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:17:04.903 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:04.848+0000 7f12267fc700 1 -- 172.21.15.195:0/3134054983 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f123000ebc0 con 0x7f1238156670 2022-06-27T12:17:04.904 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:04.848+0000 7f12267fc700 1 -- 172.21.15.195:0/3134054983 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f123000ed20 con 0x7f1238156670 2022-06-27T12:17:04.907 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:04.848+0000 7f123cb7f700 1 -- 172.21.15.195:0/3134054983 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12380c2b60 con 0x7f1238156670 2022-06-27T12:17:04.908 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:04.848+0000 7f12267fc700 1 -- 172.21.15.195:0/3134054983 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1230014670 con 0x7f1238156670 2022-06-27T12:17:04.908 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:04.848+0000 7f123cb7f700 1 -- 172.21.15.195:0/3134054983 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12380c3120 con 0x7f1238156670 2022-06-27T12:17:04.908 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:04.849+0000 7f123cb7f700 1 -- 172.21.15.195:0/3134054983 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12380722c0 con 0x7f1238156670 2022-06-27T12:17:04.909 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:04.850+0000 7f12267fc700 1 -- 172.21.15.195:0/3134054983 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f1230012050 con 0x7f1238156670 2022-06-27T12:17:04.909 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:04.850+0000 7f12267fc700 1 --2- 172.21.15.195:0/3134054983 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1214063d80 0x7f1214066230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:04.909 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:04.850+0000 7f12267fc700 1 -- 172.21.15.195:0/3134054983 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7664+0+0 (secure 0 0 0) 0x7f12300814e0 con 0x7f1238156670 2022-06-27T12:17:04.910 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:04.850+0000 7f123659c700 1 --2- 172.21.15.195:0/3134054983 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1214063d80 0x7f1214066230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:04.910 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:04.851+0000 7f123659c700 1 --2- 172.21.15.195:0/3134054983 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1214063d80 0x7f1214066230 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f122c0017e0 tx=0x7f122c009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:17:04.911 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:04.854+0000 7f12267fc700 1 -- 172.21.15.195:0/3134054983 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f123004f960 con 0x7f1238156670 2022-06-27T12:17:05.099 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:05.083+0000 7f123cb7f700 1 -- 172.21.15.195:0/3134054983 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1238068cd0 con 0x7f1238156670 2022-06-27T12:17:05.100 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:05.084+0000 7f12267fc700 1 -- 172.21.15.195:0/3134054983 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5370 (secure 0 0 0) 0x7f1230052f80 con 0x7f1238156670 2022-06-27T12:17:05.101 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:17:05.101 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:17:00.220308+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7165},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7165":{"gid":7165,"name":"d","rank":0,"incarnation":108,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6835/3551478822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":3551478822},{"type":"v1","addr":"172.21.15.202:6835","nonce":3551478822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:17:05.105 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:05.086+0000 7f123cb7f700 1 -- 172.21.15.195:0/3134054983 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1214063d80 msgr2=0x7f1214066230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:05.106 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:05.086+0000 7f123cb7f700 1 --2- 172.21.15.195:0/3134054983 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1214063d80 0x7f1214066230 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f122c0017e0 tx=0x7f122c009040 comp rx=0 tx=0).stop 2022-06-27T12:17:05.106 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:05.086+0000 7f123cb7f700 1 -- 172.21.15.195:0/3134054983 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1238156670 msgr2=0x7f12380b1e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:05.107 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:05.086+0000 7f123cb7f700 1 --2- 172.21.15.195:0/3134054983 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1238156670 0x7f12380b1e50 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f1230009fa0 tx=0x7f1230009900 comp rx=0 tx=0).stop 2022-06-27T12:17:05.108 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:05.086+0000 7f123cb7f700 1 -- 172.21.15.195:0/3134054983 shutdown_connections 2022-06-27T12:17:05.109 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:05.086+0000 7f123cb7f700 1 --2- 172.21.15.195:0/3134054983 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1214063d80 0x7f1214066230 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:05.109 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:05.086+0000 7f123cb7f700 1 --2- 172.21.15.195:0/3134054983 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1238156670 0x7f12380b1e50 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:05.109 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:05.086+0000 7f123cb7f700 1 --2- 172.21.15.195:0/3134054983 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1238067ed0 0x7f1238150cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:05.110 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:05.086+0000 7f123cb7f700 1 --2- 172.21.15.195:0/3134054983 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1238069390 0x7f1238167930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:05.110 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:05.086+0000 7f123cb7f700 1 -- 172.21.15.195:0/3134054983 >> 172.21.15.195:0/3134054983 conn(0x7f1238069e00 msgr2=0x7f123814ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:17:05.111 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:05.086+0000 7f123cb7f700 1 -- 172.21.15.195:0/3134054983 shutdown_connections 2022-06-27T12:17:05.111 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:05.086+0000 7f123cb7f700 1 -- 172.21.15.195:0/3134054983 wait complete. 2022-06-27T12:17:05.112 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 108 2022-06-27T12:17:05.113 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:17:05.313 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:17:05.334 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:17:07.118 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:17:07.277 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.251+0000 7ff1f6525700 1 Processor -- start 2022-06-27T12:17:07.277 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.251+0000 7ff1f6525700 1 -- start start 2022-06-27T12:17:07.278 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.252+0000 7ff1f6525700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7ff1f0156100 0x7ff1f00688a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:07.278 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.252+0000 7ff1f6525700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1f00c4b10 con 0x7ff1f01564d0 2022-06-27T12:17:07.278 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.252+0000 7ff1f6525700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1f0068f20 con 0x7ff1f006a440 2022-06-27T12:17:07.279 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.252+0000 7ff1f6525700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1f00c3dc0 con 0x7ff1f0156100 2022-06-27T12:17:07.279 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.252+0000 7ff1effff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7ff1f0156100 0x7ff1f00688a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:07.279 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.252+0000 7ff1f4ac2700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7ff1f01564d0 0x7ff1f00c28c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:58522/0 (socket says 172.21.15.195:58522) 2022-06-27T12:17:07.279 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.252+0000 7ff1effff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7ff1f0156100 0x7ff1f00688a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:52334/0 (socket says 172.21.15.195:52334) 2022-06-27T12:17:07.279 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.252+0000 7ff1f4ac2700 1 -- 172.21.15.195:0/2657704539 learned_addr learned my addr 172.21.15.195:0/2657704539 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:17:07.280 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.252+0000 7ff1effff700 1 -- 172.21.15.195:0/2657704539 >> v1:172.21.15.202:6789/0 conn(0x7ff1f006a440 legacy=0x7ff1f00c4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:07.280 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.252+0000 7ff1effff700 1 -- 172.21.15.195:0/2657704539 >> v1:172.21.15.195:6789/0 conn(0x7ff1f01564d0 legacy=0x7ff1f00c28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:07.280 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.252+0000 7ff1effff700 1 -- 172.21.15.195:0/2657704539 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1f00c56b0 con 0x7ff1f0156100 2022-06-27T12:17:07.280 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.253+0000 7ff1effff700 1 --2- 172.21.15.195:0/2657704539 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff1f0156100 0x7ff1f00688a0 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7ff1e4009720 tx=0x7ff1e4009ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b04056a1defd01f server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:17:07.281 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.253+0000 7ff1eeffd700 1 -- 172.21.15.195:0/2657704539 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1e4005ad0 con 0x7ff1f0156100 2022-06-27T12:17:07.281 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.253+0000 7ff1eeffd700 1 -- 172.21.15.195:0/2657704539 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1e4005c30 con 0x7ff1f0156100 2022-06-27T12:17:07.281 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.253+0000 7ff1eeffd700 1 -- 172.21.15.195:0/2657704539 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1e4006030 con 0x7ff1f0156100 2022-06-27T12:17:07.281 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.253+0000 7ff1f6525700 1 -- 172.21.15.195:0/2657704539 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff1f0156100 msgr2=0x7ff1f00688a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:07.282 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.253+0000 7ff1f6525700 1 --2- 172.21.15.195:0/2657704539 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff1f0156100 0x7ff1f00688a0 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7ff1e4009720 tx=0x7ff1e4009ae0 comp rx=0 tx=0).stop 2022-06-27T12:17:07.282 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.253+0000 7ff1f6525700 1 -- 172.21.15.195:0/2657704539 shutdown_connections 2022-06-27T12:17:07.283 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.253+0000 7ff1f6525700 1 --2- 172.21.15.195:0/2657704539 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff1f0156100 0x7ff1f00688a0 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:07.283 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.253+0000 7ff1f6525700 1 -- 172.21.15.195:0/2657704539 >> 172.21.15.195:0/2657704539 conn(0x7ff1f00c9b00 msgr2=0x7ff1f00c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:17:07.283 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.253+0000 7ff1f6525700 1 -- 172.21.15.195:0/2657704539 shutdown_connections 2022-06-27T12:17:07.283 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.253+0000 7ff1f6525700 1 -- 172.21.15.195:0/2657704539 wait complete. 2022-06-27T12:17:07.284 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.255+0000 7ff1f6525700 1 Processor -- start 2022-06-27T12:17:07.284 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.255+0000 7ff1f6525700 1 -- start start 2022-06-27T12:17:07.284 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.256+0000 7ff1f6525700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff1f006a440 0x7ff1f014d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:07.284 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.256+0000 7ff1f6525700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff1f0156100 0x7ff1f01468f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:07.285 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.256+0000 7ff1f6525700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff1f01564d0 0x7ff1f0162670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:07.285 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.256+0000 7ff1f6525700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1f01476e0 con 0x7ff1f006a440 2022-06-27T12:17:07.285 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.256+0000 7ff1f6525700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1f0148750 con 0x7ff1f0156100 2022-06-27T12:17:07.285 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.256+0000 7ff1f6525700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1f0149320 con 0x7ff1f01564d0 2022-06-27T12:17:07.286 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.256+0000 7ff1effff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff1f006a440 0x7ff1f014d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:07.286 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.256+0000 7ff1effff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff1f006a440 0x7ff1f014d7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:52338/0 (socket says 172.21.15.195:52338) 2022-06-27T12:17:07.286 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.256+0000 7ff1effff700 1 -- 172.21.15.195:0/4153757437 learned_addr learned my addr 172.21.15.195:0/4153757437 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:17:07.286 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.256+0000 7ff1f4ac2700 1 --2- 172.21.15.195:0/4153757437 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff1f01564d0 0x7ff1f0162670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:07.286 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.256+0000 7ff1ef7fe700 1 --2- 172.21.15.195:0/4153757437 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff1f0156100 0x7ff1f01468f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:07.287 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.256+0000 7ff1effff700 1 -- 172.21.15.195:0/4153757437 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff1f01564d0 msgr2=0x7ff1f0162670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:07.287 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.256+0000 7ff1effff700 1 --2- 172.21.15.195:0/4153757437 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff1f01564d0 0x7ff1f0162670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:07.287 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.256+0000 7ff1effff700 1 -- 172.21.15.195:0/4153757437 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff1f0156100 msgr2=0x7ff1f01468f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:07.287 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.256+0000 7ff1effff700 1 --2- 172.21.15.195:0/4153757437 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff1f0156100 0x7ff1f01468f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:07.288 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.256+0000 7ff1effff700 1 -- 172.21.15.195:0/4153757437 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1e4009290 con 0x7ff1f006a440 2022-06-27T12:17:07.288 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.256+0000 7ff1effff700 1 --2- 172.21.15.195:0/4153757437 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff1f006a440 0x7ff1f014d7a0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7ff1e4009ee0 tx=0x7ff1e4013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:17:07.288 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.256+0000 7ff1d7fff700 1 -- 172.21.15.195:0/4153757437 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1e401d7b0 con 0x7ff1f006a440 2022-06-27T12:17:07.288 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.256+0000 7ff1d7fff700 1 -- 172.21.15.195:0/4153757437 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1e401d910 con 0x7ff1f006a440 2022-06-27T12:17:07.289 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.256+0000 7ff1f6525700 1 -- 172.21.15.195:0/4153757437 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1f00c84e0 con 0x7ff1f006a440 2022-06-27T12:17:07.289 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.256+0000 7ff1d7fff700 1 -- 172.21.15.195:0/4153757437 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1e401dcb0 con 0x7ff1f006a440 2022-06-27T12:17:07.289 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.257+0000 7ff1f6525700 1 -- 172.21.15.195:0/4153757437 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1f0148cd0 con 0x7ff1f006a440 2022-06-27T12:17:07.289 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.257+0000 7ff1f6525700 1 -- 172.21.15.195:0/4153757437 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1f00bc180 con 0x7ff1f006a440 2022-06-27T12:17:07.290 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.258+0000 7ff1d7fff700 1 -- 172.21.15.195:0/4153757437 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7ff1e402c030 con 0x7ff1f006a440 2022-06-27T12:17:07.290 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.258+0000 7ff1d7fff700 1 --2- 172.21.15.195:0/4153757437 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff1d0063d00 0x7ff1d00661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:07.290 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.258+0000 7ff1d7fff700 1 -- 172.21.15.195:0/4153757437 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7664+0+0 (secure 0 0 0) 0x7ff1e4082a80 con 0x7ff1f006a440 2022-06-27T12:17:07.290 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.259+0000 7ff1ef7fe700 1 --2- 172.21.15.195:0/4153757437 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff1d0063d00 0x7ff1d00661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:07.291 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.259+0000 7ff1ef7fe700 1 --2- 172.21.15.195:0/4153757437 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff1d0063d00 0x7ff1d00661b0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7ff1f006dd50 tx=0x7ff1e0009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:17:07.291 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.263+0000 7ff1d7fff700 1 -- 172.21.15.195:0/4153757437 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7ff1e4051030 con 0x7ff1f006a440 2022-06-27T12:17:07.491 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.489+0000 7ff1f6525700 1 -- 172.21.15.195:0/4153757437 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff1f00722c0 con 0x7ff1f006a440 2022-06-27T12:17:07.492 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.490+0000 7ff1d7fff700 1 -- 172.21.15.195:0/4153757437 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5370 (secure 0 0 0) 0x7ff1e402a030 con 0x7ff1f006a440 2022-06-27T12:17:07.492 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:17:07.493 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:17:00.220308+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7165},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7165":{"gid":7165,"name":"d","rank":0,"incarnation":108,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6835/3551478822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":3551478822},{"type":"v1","addr":"172.21.15.202:6835","nonce":3551478822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:17:07.495 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.492+0000 7ff1f6525700 1 -- 172.21.15.195:0/4153757437 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff1d0063d00 msgr2=0x7ff1d00661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:07.495 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.492+0000 7ff1f6525700 1 --2- 172.21.15.195:0/4153757437 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff1d0063d00 0x7ff1d00661b0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7ff1f006dd50 tx=0x7ff1e0009040 comp rx=0 tx=0).stop 2022-06-27T12:17:07.495 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.492+0000 7ff1f6525700 1 -- 172.21.15.195:0/4153757437 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff1f006a440 msgr2=0x7ff1f014d7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:07.495 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.492+0000 7ff1f6525700 1 --2- 172.21.15.195:0/4153757437 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff1f006a440 0x7ff1f014d7a0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7ff1e4009ee0 tx=0x7ff1e4013000 comp rx=0 tx=0).stop 2022-06-27T12:17:07.496 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.492+0000 7ff1f6525700 1 -- 172.21.15.195:0/4153757437 shutdown_connections 2022-06-27T12:17:07.496 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.492+0000 7ff1f6525700 1 --2- 172.21.15.195:0/4153757437 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7ff1d0063d00 0x7ff1d00661b0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:07.496 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.492+0000 7ff1f6525700 1 --2- 172.21.15.195:0/4153757437 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff1f006a440 0x7ff1f014d7a0 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:07.496 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.492+0000 7ff1f6525700 1 --2- 172.21.15.195:0/4153757437 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7ff1f01564d0 0x7ff1f0162670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:07.497 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.492+0000 7ff1f6525700 1 --2- 172.21.15.195:0/4153757437 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7ff1f0156100 0x7ff1f01468f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:07.497 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.492+0000 7ff1f6525700 1 -- 172.21.15.195:0/4153757437 >> 172.21.15.195:0/4153757437 conn(0x7ff1f00c9b00 msgr2=0x7ff1f00c0400 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:17:07.497 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.492+0000 7ff1f6525700 1 -- 172.21.15.195:0/4153757437 shutdown_connections 2022-06-27T12:17:07.497 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:07.492+0000 7ff1f6525700 1 -- 172.21.15.195:0/4153757437 wait complete. 2022-06-27T12:17:07.498 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 108 2022-06-27T12:17:07.514 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:17:09.528 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:17:09.701 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.689+0000 7fe783641700 1 Processor -- start 2022-06-27T12:17:09.701 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.689+0000 7fe783641700 1 -- start start 2022-06-27T12:17:09.701 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.690+0000 7fe783641700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fe77c06f640 0x7fe77c156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:09.702 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.690+0000 7fe783641700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe77c0c1a20 con 0x7fe77c0c39e0 2022-06-27T12:17:09.702 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.690+0000 7fe783641700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe77c0c1130 con 0x7fe77c0c4160 2022-06-27T12:17:09.702 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.690+0000 7fe783641700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe77c06ea90 con 0x7fe77c06f640 2022-06-27T12:17:09.702 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.690+0000 7fe7813dd700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fe77c06f640 0x7fe77c156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:09.703 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.690+0000 7fe781bde700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fe77c0c39e0 0x7fe77c1535a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:58548/0 (socket says 172.21.15.195:58548) 2022-06-27T12:17:09.703 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.690+0000 7fe781bde700 1 -- 172.21.15.195:0/3630762034 learned_addr learned my addr 172.21.15.195:0/3630762034 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:17:09.703 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.690+0000 7fe7813dd700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fe77c06f640 0x7fe77c156ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:52356/0 (socket says 172.21.15.195:52356) 2022-06-27T12:17:09.703 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.691+0000 7fe7813dd700 1 -- 172.21.15.195:0/3630762034 >> v1:172.21.15.202:6789/0 conn(0x7fe77c0c4160 legacy=0x7fe77c0c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:09.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.691+0000 7fe7813dd700 1 -- 172.21.15.195:0/3630762034 >> v1:172.21.15.195:6789/0 conn(0x7fe77c0c39e0 legacy=0x7fe77c1535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:09.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.691+0000 7fe76bfff700 1 -- 172.21.15.195:0/3630762034 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3895677463 0 0) 0x7fe77c0c1a20 con 0x7fe77c0c39e0 2022-06-27T12:17:09.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.691+0000 7fe7813dd700 1 -- 172.21.15.195:0/3630762034 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe77c06ed70 con 0x7fe77c06f640 2022-06-27T12:17:09.705 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.691+0000 7fe7813dd700 1 --2- 172.21.15.195:0/3630762034 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe77c06f640 0x7fe77c156ed0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7fe76c0097a0 tx=0x7fe76c009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7a3d7702c9e45b4 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:17:09.705 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.691+0000 7fe76bfff700 1 -- 172.21.15.195:0/3630762034 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe76c011630 con 0x7fe77c06f640 2022-06-27T12:17:09.705 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.691+0000 7fe76bfff700 1 -- 172.21.15.195:0/3630762034 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe76c011790 con 0x7fe77c06f640 2022-06-27T12:17:09.705 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.691+0000 7fe76bfff700 1 -- 172.21.15.195:0/3630762034 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe76c011b30 con 0x7fe77c06f640 2022-06-27T12:17:09.706 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.691+0000 7fe783641700 1 -- 172.21.15.195:0/3630762034 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe77c06f640 msgr2=0x7fe77c156ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:09.706 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.691+0000 7fe783641700 1 --2- 172.21.15.195:0/3630762034 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe77c06f640 0x7fe77c156ed0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7fe76c0097a0 tx=0x7fe76c009060 comp rx=0 tx=0).stop 2022-06-27T12:17:09.706 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.691+0000 7fe783641700 1 -- 172.21.15.195:0/3630762034 shutdown_connections 2022-06-27T12:17:09.707 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.691+0000 7fe783641700 1 --2- 172.21.15.195:0/3630762034 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe77c06f640 0x7fe77c156ed0 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:09.707 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.691+0000 7fe783641700 1 -- 172.21.15.195:0/3630762034 >> 172.21.15.195:0/3630762034 conn(0x7fe77c0c6350 msgr2=0x7fe77c0c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:17:09.707 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.692+0000 7fe783641700 1 -- 172.21.15.195:0/3630762034 shutdown_connections 2022-06-27T12:17:09.708 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.692+0000 7fe783641700 1 -- 172.21.15.195:0/3630762034 wait complete. 2022-06-27T12:17:09.708 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.693+0000 7fe783641700 1 Processor -- start 2022-06-27T12:17:09.708 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.693+0000 7fe783641700 1 -- start start 2022-06-27T12:17:09.708 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.693+0000 7fe783641700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe77c06f640 0x7fe77c14f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:09.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.694+0000 7fe783641700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe77c0c39e0 0x7fe77c0bf710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:09.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.694+0000 7fe783641700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe77c0c4160 0x7fe77c0bfc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:09.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.694+0000 7fe783641700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe77c169a10 con 0x7fe77c0c4160 2022-06-27T12:17:09.710 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.694+0000 7fe783641700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe77c1511e0 con 0x7fe77c06f640 2022-06-27T12:17:09.711 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.694+0000 7fe783641700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe77c14fda0 con 0x7fe77c0c39e0 2022-06-27T12:17:09.711 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.694+0000 7fe781bde700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe77c0c4160 0x7fe77c0bfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:09.711 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.694+0000 7fe781bde700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe77c0c4160 0x7fe77c0bfc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:52366/0 (socket says 172.21.15.195:52366) 2022-06-27T12:17:09.712 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.694+0000 7fe781bde700 1 -- 172.21.15.195:0/3155129383 learned_addr learned my addr 172.21.15.195:0/3155129383 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:17:09.712 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.694+0000 7fe7813dd700 1 --2- 172.21.15.195:0/3155129383 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe77c06f640 0x7fe77c14f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:09.712 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.694+0000 7fe780bdc700 1 --2- 172.21.15.195:0/3155129383 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe77c0c39e0 0x7fe77c0bf710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:09.715 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.694+0000 7fe781bde700 1 -- 172.21.15.195:0/3155129383 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe77c0c39e0 msgr2=0x7fe77c0bf710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:09.715 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.694+0000 7fe781bde700 1 --2- 172.21.15.195:0/3155129383 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe77c0c39e0 0x7fe77c0bf710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:09.715 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.694+0000 7fe781bde700 1 -- 172.21.15.195:0/3155129383 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe77c06f640 msgr2=0x7fe77c14f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:09.716 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.694+0000 7fe781bde700 1 --2- 172.21.15.195:0/3155129383 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe77c06f640 0x7fe77c14f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:09.716 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.694+0000 7fe781bde700 1 -- 172.21.15.195:0/3155129383 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe76c0092e0 con 0x7fe77c0c4160 2022-06-27T12:17:09.716 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.694+0000 7fe781bde700 1 --2- 172.21.15.195:0/3155129383 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe77c0c4160 0x7fe77c0bfc50 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7fe77400c860 tx=0x7fe774001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:17:09.716 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.694+0000 7fe7697fa700 1 -- 172.21.15.195:0/3155129383 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe774004c80 con 0x7fe77c0c4160 2022-06-27T12:17:09.717 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.694+0000 7fe7697fa700 1 -- 172.21.15.195:0/3155129383 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe774004de0 con 0x7fe77c0c4160 2022-06-27T12:17:09.717 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.694+0000 7fe783641700 1 -- 172.21.15.195:0/3155129383 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe77c1666a0 con 0x7fe77c0c4160 2022-06-27T12:17:09.717 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.694+0000 7fe7697fa700 1 -- 172.21.15.195:0/3155129383 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe774007380 con 0x7fe77c0c4160 2022-06-27T12:17:09.717 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.694+0000 7fe783641700 1 -- 172.21.15.195:0/3155129383 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe77c166c40 con 0x7fe77c0c4160 2022-06-27T12:17:09.718 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.695+0000 7fe783641700 1 -- 172.21.15.195:0/3155129383 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe77c169b80 con 0x7fe77c0c4160 2022-06-27T12:17:09.718 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.696+0000 7fe7697fa700 1 -- 172.21.15.195:0/3155129383 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fe77401a030 con 0x7fe77c0c4160 2022-06-27T12:17:09.718 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.696+0000 7fe7697fa700 1 --2- 172.21.15.195:0/3155129383 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe75c063d00 0x7fe75c0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:09.718 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.696+0000 7fe7697fa700 1 -- 172.21.15.195:0/3155129383 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7664+0+0 (secure 0 0 0) 0x7fe7740803d0 con 0x7fe77c0c4160 2022-06-27T12:17:09.719 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.696+0000 7fe7813dd700 1 --2- 172.21.15.195:0/3155129383 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe75c063d00 0x7fe75c0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:09.719 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.697+0000 7fe7813dd700 1 --2- 172.21.15.195:0/3155129383 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe75c063d00 0x7fe75c0661b0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fe76c011fd0 tx=0x7fe76c017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:17:09.720 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.700+0000 7fe7697fa700 1 -- 172.21.15.195:0/3155129383 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fe77c169b80 con 0x7fe77c0c4160 2022-06-27T12:17:09.933 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.931+0000 7fe783641700 1 -- 172.21.15.195:0/3155129383 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe77c169b50 con 0x7fe77c0c4160 2022-06-27T12:17:09.933 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.931+0000 7fe7697fa700 1 -- 172.21.15.195:0/3155129383 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 109 v109) v1 ==== 95+0+5374 (secure 0 0 0) 0x7fe774018070 con 0x7fe77c0c4160 2022-06-27T12:17:09.934 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:17:09.934 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":109,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:17:09.331749+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7165},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7165":{"gid":7165,"name":"d","rank":0,"incarnation":108,"state":"up:reconnect","state_seq":20,"addr":"172.21.15.202:6835/3551478822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":3551478822},{"type":"v1","addr":"172.21.15.202:6835","nonce":3551478822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:17:09.935 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.933+0000 7fe783641700 1 -- 172.21.15.195:0/3155129383 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe75c063d00 msgr2=0x7fe75c0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:09.935 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.933+0000 7fe783641700 1 --2- 172.21.15.195:0/3155129383 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe75c063d00 0x7fe75c0661b0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fe76c011fd0 tx=0x7fe76c017000 comp rx=0 tx=0).stop 2022-06-27T12:17:09.935 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.933+0000 7fe783641700 1 -- 172.21.15.195:0/3155129383 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe77c0c4160 msgr2=0x7fe77c0bfc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:09.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.933+0000 7fe783641700 1 --2- 172.21.15.195:0/3155129383 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe77c0c4160 0x7fe77c0bfc50 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7fe77400c860 tx=0x7fe774001630 comp rx=0 tx=0).stop 2022-06-27T12:17:09.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.933+0000 7fe783641700 1 -- 172.21.15.195:0/3155129383 shutdown_connections 2022-06-27T12:17:09.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.933+0000 7fe783641700 1 --2- 172.21.15.195:0/3155129383 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe75c063d00 0x7fe75c0661b0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:09.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.933+0000 7fe783641700 1 --2- 172.21.15.195:0/3155129383 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe77c0c4160 0x7fe77c0bfc50 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:09.937 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.933+0000 7fe783641700 1 --2- 172.21.15.195:0/3155129383 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe77c06f640 0x7fe77c14f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:09.937 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.933+0000 7fe783641700 1 --2- 172.21.15.195:0/3155129383 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe77c0c39e0 0x7fe77c0bf710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:09.942 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.934+0000 7fe783641700 1 -- 172.21.15.195:0/3155129383 >> 172.21.15.195:0/3155129383 conn(0x7fe77c0c6350 msgr2=0x7fe77c156c00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:17:09.943 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.934+0000 7fe783641700 1 -- 172.21.15.195:0/3155129383 shutdown_connections 2022-06-27T12:17:09.943 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:09.934+0000 7fe783641700 1 -- 172.21.15.195:0/3155129383 wait complete. 2022-06-27T12:17:09.944 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 109 2022-06-27T12:17:09.951 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:17:11.952 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:17:12.089 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.088+0000 7f8e56168700 1 Processor -- start 2022-06-27T12:17:12.089 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.088+0000 7f8e56168700 1 -- start start 2022-06-27T12:17:12.103 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.089+0000 7f8e56168700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f8e50154900 0x7f8e500c2120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:12.103 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.089+0000 7f8e56168700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e50154da0 con 0x7f8e50151560 2022-06-27T12:17:12.104 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.089+0000 7f8e56168700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e500c12a0 con 0x7f8e50151930 2022-06-27T12:17:12.104 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.089+0000 7f8e56168700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e500c1580 con 0x7f8e50154900 2022-06-27T12:17:12.105 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.089+0000 7f8e4effd700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f8e50154900 0x7f8e500c2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:12.105 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.089+0000 7f8e4f7fe700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f8e50151560 0x7f8e501541f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:58562/0 (socket says 172.21.15.195:58562) 2022-06-27T12:17:12.107 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.089+0000 7f8e4effd700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f8e50154900 0x7f8e500c2120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:52374/0 (socket says 172.21.15.195:52374) 2022-06-27T12:17:12.108 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.089+0000 7f8e4f7fe700 1 -- 172.21.15.195:0/1725590897 learned_addr learned my addr 172.21.15.195:0/1725590897 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:17:12.108 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.089+0000 7f8e4effd700 1 -- 172.21.15.195:0/1725590897 >> v1:172.21.15.202:6789/0 conn(0x7f8e50151930 legacy=0x7f8e5015eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:12.108 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.089+0000 7f8e4effd700 1 -- 172.21.15.195:0/1725590897 >> v1:172.21.15.195:6789/0 conn(0x7f8e50151560 legacy=0x7f8e501541f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:12.108 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.089+0000 7f8e4e7fc700 1 -- 172.21.15.195:0/1725590897 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1852110433 0 0) 0x7f8e50154da0 con 0x7f8e50151560 2022-06-27T12:17:12.109 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.089+0000 7f8e4effd700 1 -- 172.21.15.195:0/1725590897 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e500c29f0 con 0x7f8e50154900 2022-06-27T12:17:12.109 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.090+0000 7f8e4effd700 1 --2- 172.21.15.195:0/1725590897 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8e50154900 0x7f8e500c2120 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f8e400097a0 tx=0x7f8e40009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3db351569875cabd server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:17:12.109 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.090+0000 7f8e4e7fc700 1 -- 172.21.15.195:0/1725590897 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e40011630 con 0x7f8e50154900 2022-06-27T12:17:12.109 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.090+0000 7f8e4e7fc700 1 -- 172.21.15.195:0/1725590897 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e40011790 con 0x7f8e50154900 2022-06-27T12:17:12.109 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.090+0000 7f8e4e7fc700 1 -- 172.21.15.195:0/1725590897 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e40011b30 con 0x7f8e50154900 2022-06-27T12:17:12.110 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.090+0000 7f8e56168700 1 -- 172.21.15.195:0/1725590897 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8e50154900 msgr2=0x7f8e500c2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:12.110 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.090+0000 7f8e56168700 1 --2- 172.21.15.195:0/1725590897 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8e50154900 0x7f8e500c2120 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f8e400097a0 tx=0x7f8e40009060 comp rx=0 tx=0).stop 2022-06-27T12:17:12.110 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.090+0000 7f8e56168700 1 -- 172.21.15.195:0/1725590897 shutdown_connections 2022-06-27T12:17:12.111 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.090+0000 7f8e56168700 1 --2- 172.21.15.195:0/1725590897 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8e50154900 0x7f8e500c2120 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:12.111 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.090+0000 7f8e56168700 1 -- 172.21.15.195:0/1725590897 >> 172.21.15.195:0/1725590897 conn(0x7f8e500c6350 msgr2=0x7f8e500c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:17:12.111 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.090+0000 7f8e56168700 1 -- 172.21.15.195:0/1725590897 shutdown_connections 2022-06-27T12:17:12.111 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.090+0000 7f8e56168700 1 -- 172.21.15.195:0/1725590897 wait complete. 2022-06-27T12:17:12.111 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.092+0000 7f8e56168700 1 Processor -- start 2022-06-27T12:17:12.112 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.092+0000 7f8e56168700 1 -- start start 2022-06-27T12:17:12.112 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.092+0000 7f8e56168700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8e50151560 0x7f8e500c77b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:12.112 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.092+0000 7f8e56168700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8e50151930 0x7f8e50155f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:12.112 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.092+0000 7f8e56168700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8e50154900 0x7f8e501564b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:12.113 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.092+0000 7f8e56168700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e50155a70 con 0x7f8e50154900 2022-06-27T12:17:12.113 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.092+0000 7f8e56168700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e50168720 con 0x7f8e50151930 2022-06-27T12:17:12.113 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.092+0000 7f8e56168700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e50168160 con 0x7f8e50151560 2022-06-27T12:17:12.114 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.092+0000 7f8e4ffff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8e50154900 0x7f8e501564b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:12.114 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.092+0000 7f8e4ffff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8e50154900 0x7f8e501564b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:52384/0 (socket says 172.21.15.195:52384) 2022-06-27T12:17:12.114 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.092+0000 7f8e4ffff700 1 -- 172.21.15.195:0/4137073099 learned_addr learned my addr 172.21.15.195:0/4137073099 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:17:12.114 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.093+0000 7f8e4f7fe700 1 --2- 172.21.15.195:0/4137073099 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8e50151560 0x7f8e500c77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:12.115 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.093+0000 7f8e4effd700 1 --2- 172.21.15.195:0/4137073099 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8e50151930 0x7f8e50155f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:12.115 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.093+0000 7f8e4ffff700 1 -- 172.21.15.195:0/4137073099 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8e50151560 msgr2=0x7f8e500c77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:12.116 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.093+0000 7f8e4ffff700 1 --2- 172.21.15.195:0/4137073099 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8e50151560 0x7f8e500c77b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:12.116 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.093+0000 7f8e4ffff700 1 -- 172.21.15.195:0/4137073099 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8e50151930 msgr2=0x7f8e50155f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:12.116 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.093+0000 7f8e4ffff700 1 --2- 172.21.15.195:0/4137073099 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8e50151930 0x7f8e50155f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:12.116 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.093+0000 7f8e4ffff700 1 -- 172.21.15.195:0/4137073099 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e400092e0 con 0x7f8e50154900 2022-06-27T12:17:12.117 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.093+0000 7f8e4ffff700 1 --2- 172.21.15.195:0/4137073099 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8e50154900 0x7f8e501564b0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f8e4800c7c0 tx=0x7f8e480015e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:17:12.117 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.093+0000 7f8e477fe700 1 -- 172.21.15.195:0/4137073099 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e48003880 con 0x7f8e50154900 2022-06-27T12:17:12.117 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.093+0000 7f8e477fe700 1 -- 172.21.15.195:0/4137073099 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e480039e0 con 0x7f8e50154900 2022-06-27T12:17:12.118 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.093+0000 7f8e477fe700 1 -- 172.21.15.195:0/4137073099 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e48003d60 con 0x7f8e50154900 2022-06-27T12:17:12.118 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.093+0000 7f8e56168700 1 -- 172.21.15.195:0/4137073099 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e50168440 con 0x7f8e50154900 2022-06-27T12:17:12.118 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.093+0000 7f8e56168700 1 -- 172.21.15.195:0/4137073099 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e50155700 con 0x7f8e50154900 2022-06-27T12:17:12.119 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.094+0000 7f8e56168700 1 -- 172.21.15.195:0/4137073099 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e501558c0 con 0x7f8e50154900 2022-06-27T12:17:12.121 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.095+0000 7f8e477fe700 1 -- 172.21.15.195:0/4137073099 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f8e48010050 con 0x7f8e50154900 2022-06-27T12:17:12.121 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.095+0000 7f8e477fe700 1 --2- 172.21.15.195:0/4137073099 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8e34063d00 0x7f8e340661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:12.122 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.095+0000 7f8e477fe700 1 -- 172.21.15.195:0/4137073099 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7664+0+0 (secure 0 0 0) 0x7f8e48081370 con 0x7f8e50154900 2022-06-27T12:17:12.122 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.096+0000 7f8e4f7fe700 1 --2- 172.21.15.195:0/4137073099 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8e34063d00 0x7f8e340661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:12.123 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.096+0000 7f8e4f7fe700 1 --2- 172.21.15.195:0/4137073099 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8e34063d00 0x7f8e340661b0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f8e380011e0 tx=0x7f8e38007bf0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:17:12.123 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.099+0000 7f8e477fe700 1 -- 172.21.15.195:0/4137073099 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f8e4804f9a0 con 0x7f8e50154900 2022-06-27T12:17:12.334 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.327+0000 7f8e56168700 1 -- 172.21.15.195:0/4137073099 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8e500722c0 con 0x7f8e50154900 2022-06-27T12:17:12.335 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.328+0000 7f8e477fe700 1 -- 172.21.15.195:0/4137073099 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 109 v109) v1 ==== 95+0+5374 (secure 0 0 0) 0x7f8e48052fc0 con 0x7f8e50154900 2022-06-27T12:17:12.335 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:17:12.335 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":109,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:17:09.331749+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7165},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7165":{"gid":7165,"name":"d","rank":0,"incarnation":108,"state":"up:reconnect","state_seq":20,"addr":"172.21.15.202:6835/3551478822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":3551478822},{"type":"v1","addr":"172.21.15.202:6835","nonce":3551478822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:17:12.337 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.330+0000 7f8e56168700 1 -- 172.21.15.195:0/4137073099 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8e34063d00 msgr2=0x7f8e340661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:12.337 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.330+0000 7f8e56168700 1 --2- 172.21.15.195:0/4137073099 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8e34063d00 0x7f8e340661b0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f8e380011e0 tx=0x7f8e38007bf0 comp rx=0 tx=0).stop 2022-06-27T12:17:12.338 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.330+0000 7f8e56168700 1 -- 172.21.15.195:0/4137073099 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8e50154900 msgr2=0x7f8e501564b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:12.338 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.330+0000 7f8e56168700 1 --2- 172.21.15.195:0/4137073099 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8e50154900 0x7f8e501564b0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f8e4800c7c0 tx=0x7f8e480015e0 comp rx=0 tx=0).stop 2022-06-27T12:17:12.339 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.330+0000 7f8e56168700 1 -- 172.21.15.195:0/4137073099 shutdown_connections 2022-06-27T12:17:12.339 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.330+0000 7f8e56168700 1 --2- 172.21.15.195:0/4137073099 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8e34063d00 0x7f8e340661b0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:12.339 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.330+0000 7f8e56168700 1 --2- 172.21.15.195:0/4137073099 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8e50154900 0x7f8e501564b0 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:12.340 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.330+0000 7f8e56168700 1 --2- 172.21.15.195:0/4137073099 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f8e50151560 0x7f8e500c77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:12.340 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.330+0000 7f8e56168700 1 --2- 172.21.15.195:0/4137073099 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f8e50151930 0x7f8e50155f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:12.340 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.330+0000 7f8e56168700 1 -- 172.21.15.195:0/4137073099 >> 172.21.15.195:0/4137073099 conn(0x7f8e500c6350 msgr2=0x7f8e500c6720 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:17:12.340 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.330+0000 7f8e56168700 1 -- 172.21.15.195:0/4137073099 shutdown_connections 2022-06-27T12:17:12.341 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:12.330+0000 7f8e56168700 1 -- 172.21.15.195:0/4137073099 wait complete. 2022-06-27T12:17:12.341 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 109 2022-06-27T12:17:12.343 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:17:14.344 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:17:14.496 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.492+0000 7f13a3092700 1 Processor -- start 2022-06-27T12:17:14.496 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.492+0000 7f13a3092700 1 -- start start 2022-06-27T12:17:14.496 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.493+0000 7f13a3092700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f139c150580 0x7f139c156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:14.497 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.493+0000 7f13a3092700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f139c06f620 0x7f139c06d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:14.497 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.493+0000 7f13a3092700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f139c0c76b0 con 0x7f139c150950 2022-06-27T12:17:14.497 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.493+0000 7f13a3092700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f139c157430 con 0x7f139c06f620 2022-06-27T12:17:14.498 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.493+0000 7f13a3092700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f139c0c8250 con 0x7f139c150580 2022-06-27T12:17:14.498 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.493+0000 7f13a162f700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f139c06f620 0x7f139c06d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:14.498 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.493+0000 7f139bfff700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f139c150950 0x7f139c0c7b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:58576/0 (socket says 172.21.15.195:58576) 2022-06-27T12:17:14.498 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.493+0000 7f13a162f700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f139c06f620 0x7f139c06d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:52392/0 (socket says 172.21.15.195:52392) 2022-06-27T12:17:14.499 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.493+0000 7f139bfff700 1 -- 172.21.15.195:0/2669818272 learned_addr learned my addr 172.21.15.195:0/2669818272 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:17:14.499 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.493+0000 7f13a0e2e700 1 --2- 172.21.15.195:0/2669818272 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f139c150580 0x7f139c156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:14.499 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.493+0000 7f13a162f700 1 -- 172.21.15.195:0/2669818272 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f139c150580 msgr2=0x7f139c156ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:17:14.499 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.493+0000 7f13a162f700 1 --2- 172.21.15.195:0/2669818272 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f139c150580 0x7f139c156ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:14.500 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.493+0000 7f13a162f700 1 -- 172.21.15.195:0/2669818272 >> v1:172.21.15.195:6789/0 conn(0x7f139c150950 legacy=0x7f139c0c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:14.500 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.493+0000 7f139b7fe700 1 -- 172.21.15.195:0/2669818272 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3439732447 0 0) 0x7f139c0c76b0 con 0x7f139c150950 2022-06-27T12:17:14.501 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.493+0000 7f13a162f700 1 -- 172.21.15.195:0/2669818272 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f139c0bad70 con 0x7f139c06f620 2022-06-27T12:17:14.501 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.493+0000 7f13a0e2e700 1 --2- 172.21.15.195:0/2669818272 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f139c150580 0x7f139c156ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:17:14.501 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.493+0000 7f13a162f700 1 --2- 172.21.15.195:0/2669818272 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f139c06f620 0x7f139c06d6a0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f1394009e90 tx=0x7f1394009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=640563b7ab976862 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:17:14.502 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.494+0000 7f139b7fe700 1 -- 172.21.15.195:0/2669818272 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f139400e070 con 0x7f139c06f620 2022-06-27T12:17:14.502 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.494+0000 7f139b7fe700 1 -- 172.21.15.195:0/2669818272 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f139400bbf0 con 0x7f139c06f620 2022-06-27T12:17:14.503 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.494+0000 7f139b7fe700 1 -- 172.21.15.195:0/2669818272 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1394012430 con 0x7f139c06f620 2022-06-27T12:17:14.503 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.494+0000 7f13a3092700 1 -- 172.21.15.195:0/2669818272 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f139c06f620 msgr2=0x7f139c06d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:14.503 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.494+0000 7f13a3092700 1 --2- 172.21.15.195:0/2669818272 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f139c06f620 0x7f139c06d6a0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f1394009e90 tx=0x7f1394009780 comp rx=0 tx=0).stop 2022-06-27T12:17:14.504 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.494+0000 7f13a3092700 1 -- 172.21.15.195:0/2669818272 shutdown_connections 2022-06-27T12:17:14.504 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.494+0000 7f13a3092700 1 --2- 172.21.15.195:0/2669818272 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f139c06f620 0x7f139c06d6a0 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:14.504 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.494+0000 7f13a3092700 1 --2- 172.21.15.195:0/2669818272 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f139c150580 0x7f139c156ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:14.504 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.494+0000 7f13a3092700 1 -- 172.21.15.195:0/2669818272 >> 172.21.15.195:0/2669818272 conn(0x7f139c063af0 msgr2=0x7f139c063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:17:14.504 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.494+0000 7f13a3092700 1 -- 172.21.15.195:0/2669818272 shutdown_connections 2022-06-27T12:17:14.505 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.494+0000 7f13a3092700 1 -- 172.21.15.195:0/2669818272 wait complete. 2022-06-27T12:17:14.505 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.496+0000 7f13a3092700 1 Processor -- start 2022-06-27T12:17:14.505 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.496+0000 7f13a3092700 1 -- start start 2022-06-27T12:17:14.509 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.497+0000 7f13a3092700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f139c06f620 0x7f139c16a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:14.509 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.497+0000 7f13a3092700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f139c150580 0x7f139c0c21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:14.509 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.497+0000 7f13a3092700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f139c150950 0x7f139c14f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:14.509 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.497+0000 7f13a3092700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f139c16a290 con 0x7f139c150580 2022-06-27T12:17:14.510 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.497+0000 7f13a3092700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f139c1510e0 con 0x7f139c06f620 2022-06-27T12:17:14.510 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.497+0000 7f13a3092700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f139c151380 con 0x7f139c150950 2022-06-27T12:17:14.510 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.497+0000 7f139bfff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f139c150580 0x7f139c0c21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:14.511 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.497+0000 7f139bfff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f139c150580 0x7f139c0c21d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:52396/0 (socket says 172.21.15.195:52396) 2022-06-27T12:17:14.511 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.497+0000 7f139bfff700 1 -- 172.21.15.195:0/3947717062 learned_addr learned my addr 172.21.15.195:0/3947717062 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:17:14.512 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.497+0000 7f13a162f700 1 --2- 172.21.15.195:0/3947717062 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f139c150950 0x7f139c14f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:14.512 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.497+0000 7f13a0e2e700 1 --2- 172.21.15.195:0/3947717062 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f139c06f620 0x7f139c16a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:14.512 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.497+0000 7f139bfff700 1 -- 172.21.15.195:0/3947717062 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f139c150950 msgr2=0x7f139c14f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:14.513 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.497+0000 7f139bfff700 1 --2- 172.21.15.195:0/3947717062 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f139c150950 0x7f139c14f340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:14.513 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.497+0000 7f139bfff700 1 -- 172.21.15.195:0/3947717062 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f139c06f620 msgr2=0x7f139c16a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:14.514 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.497+0000 7f139bfff700 1 --2- 172.21.15.195:0/3947717062 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f139c06f620 0x7f139c16a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:14.514 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.497+0000 7f139bfff700 1 -- 172.21.15.195:0/3947717062 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1394009a20 con 0x7f139c150580 2022-06-27T12:17:14.515 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.497+0000 7f13a0e2e700 1 --2- 172.21.15.195:0/3947717062 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f139c06f620 0x7f139c16a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T12:17:14.515 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.497+0000 7f139bfff700 1 --2- 172.21.15.195:0/3947717062 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f139c150580 0x7f139c0c21d0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f138800c590 tx=0x7f1388007c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:17:14.516 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.498+0000 7f1398ff9700 1 -- 172.21.15.195:0/3947717062 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1388007360 con 0x7f139c150580 2022-06-27T12:17:14.516 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.498+0000 7f1398ff9700 1 -- 172.21.15.195:0/3947717062 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1388007500 con 0x7f139c150580 2022-06-27T12:17:14.517 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.498+0000 7f1398ff9700 1 -- 172.21.15.195:0/3947717062 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1388004940 con 0x7f139c150580 2022-06-27T12:17:14.517 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.498+0000 7f13a3092700 1 -- 172.21.15.195:0/3947717062 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f139c151660 con 0x7f139c150580 2022-06-27T12:17:14.518 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.498+0000 7f13a3092700 1 -- 172.21.15.195:0/3947717062 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f139c150de0 con 0x7f139c150580 2022-06-27T12:17:14.518 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.499+0000 7f13a3092700 1 -- 172.21.15.195:0/3947717062 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f139c06eba0 con 0x7f139c150580 2022-06-27T12:17:14.519 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.499+0000 7f1398ff9700 1 -- 172.21.15.195:0/3947717062 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f1388010030 con 0x7f139c150580 2022-06-27T12:17:14.519 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.499+0000 7f1398ff9700 1 --2- 172.21.15.195:0/3947717062 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1384063d00 0x7f13840661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:14.520 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.500+0000 7f1398ff9700 1 -- 172.21.15.195:0/3947717062 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7664+0+0 (secure 0 0 0) 0x7f1388080b80 con 0x7f139c150580 2022-06-27T12:17:14.520 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.500+0000 7f13a0e2e700 1 --2- 172.21.15.195:0/3947717062 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1384063d00 0x7f13840661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:14.521 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.500+0000 7f13a0e2e700 1 --2- 172.21.15.195:0/3947717062 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1384063d00 0x7f13840661b0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f1390006dc0 tx=0x7f1390008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:17:14.521 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.504+0000 7f1398ff9700 1 -- 172.21.15.195:0/3947717062 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f138804f0e0 con 0x7f139c150580 2022-06-27T12:17:14.725 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.724+0000 7f13a3092700 1 -- 172.21.15.195:0/3947717062 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f139c0c6ca0 con 0x7f139c150580 2022-06-27T12:17:14.726 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.725+0000 7f1398ff9700 1 -- 172.21.15.195:0/3947717062 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 109 v109) v1 ==== 95+0+5374 (secure 0 0 0) 0x7f1388052700 con 0x7f139c150580 2022-06-27T12:17:14.727 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:17:14.727 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":109,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:17:09.331749+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7165},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7165":{"gid":7165,"name":"d","rank":0,"incarnation":108,"state":"up:reconnect","state_seq":20,"addr":"172.21.15.202:6835/3551478822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":3551478822},{"type":"v1","addr":"172.21.15.202:6835","nonce":3551478822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:17:14.729 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.727+0000 7f13a3092700 1 -- 172.21.15.195:0/3947717062 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1384063d00 msgr2=0x7f13840661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:14.731 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.727+0000 7f13a3092700 1 --2- 172.21.15.195:0/3947717062 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1384063d00 0x7f13840661b0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f1390006dc0 tx=0x7f1390008040 comp rx=0 tx=0).stop 2022-06-27T12:17:14.731 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.727+0000 7f13a3092700 1 -- 172.21.15.195:0/3947717062 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f139c150580 msgr2=0x7f139c0c21d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:14.732 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.727+0000 7f13a3092700 1 --2- 172.21.15.195:0/3947717062 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f139c150580 0x7f139c0c21d0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f138800c590 tx=0x7f1388007c80 comp rx=0 tx=0).stop 2022-06-27T12:17:14.732 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.727+0000 7f13a3092700 1 -- 172.21.15.195:0/3947717062 shutdown_connections 2022-06-27T12:17:14.733 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.727+0000 7f13a3092700 1 --2- 172.21.15.195:0/3947717062 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1384063d00 0x7f13840661b0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:14.733 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.727+0000 7f13a3092700 1 --2- 172.21.15.195:0/3947717062 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f139c150950 0x7f139c14f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:14.733 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.727+0000 7f13a3092700 1 --2- 172.21.15.195:0/3947717062 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f139c06f620 0x7f139c16a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:14.735 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.727+0000 7f13a3092700 1 --2- 172.21.15.195:0/3947717062 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f139c150580 0x7f139c0c21d0 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:14.736 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.727+0000 7f13a3092700 1 -- 172.21.15.195:0/3947717062 >> 172.21.15.195:0/3947717062 conn(0x7f139c063af0 msgr2=0x7f139c14d7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:17:14.737 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.727+0000 7f13a3092700 1 -- 172.21.15.195:0/3947717062 shutdown_connections 2022-06-27T12:17:14.737 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:14.727+0000 7f13a3092700 1 -- 172.21.15.195:0/3947717062 wait complete. 2022-06-27T12:17:14.738 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 109 2022-06-27T12:17:14.742 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 109, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-06-27T11:47:52.358095+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 109, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7165": { "addr": "172.21.15.202:6835/3551478822", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6834", "nonce": 3551478822, "type": "v2" }, { "addr": "172.21.15.202:6835", "nonce": 3551478822, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7165, "incarnation": 108, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:reconnect", "state_seq": 20 } }, "last_failure": 0, "last_failure_osd_epoch": 65, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-06-27T12:17:09.331749+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7165 } } } ], "standbys": [ { "addr": "172.21.15.202:6837/3305006841", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6836", "nonce": 3305006841, "type": "v2" }, { "addr": "172.21.15.202:6837", "nonce": 3305006841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6835/2060709822", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6834", "nonce": 2060709822, "type": "v2" }, { "addr": "172.21.15.195:6835", "nonce": 2060709822, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6838/1598751652", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6836", "nonce": 1598751652, "type": "v2" }, { "addr": "172.21.15.195:6838", "nonce": 1598751652, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4275, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6839/1313800661", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6837", "nonce": 1313800661, "type": "v2" }, { "addr": "172.21.15.195:6839", "nonce": 1313800661, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4278, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-06-27T12:17:16.743 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:17:16.878 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.877+0000 7f52404e8700 1 Processor -- start 2022-06-27T12:17:16.878 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.877+0000 7f52404e8700 1 -- start start 2022-06-27T12:17:16.889 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.878+0000 7f52404e8700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f52380c4160 0x7f5238156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:16.889 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.878+0000 7f52404e8700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52380c1a20 con 0x7f52380c39e0 2022-06-27T12:17:16.889 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.878+0000 7f52404e8700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52380c1130 con 0x7f523806f640 2022-06-27T12:17:16.890 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.878+0000 7f52404e8700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f523806ea90 con 0x7f52380c4160 2022-06-27T12:17:16.890 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.878+0000 7f523ea85700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f52380c39e0 0x7f52381535a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:58594/0 (socket says 172.21.15.195:58594) 2022-06-27T12:17:16.890 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.878+0000 7f523ea85700 1 -- 172.21.15.195:0/754474853 learned_addr learned my addr 172.21.15.195:0/754474853 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:17:16.890 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.879+0000 7f523da83700 1 --2- 172.21.15.195:0/754474853 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f52380c4160 0x7f5238156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:16.891 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.879+0000 7f523d282700 1 -- 172.21.15.195:0/754474853 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2443175041 0 0) 0x7f52380c1a20 con 0x7f52380c39e0 2022-06-27T12:17:16.891 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.879+0000 7f523d282700 1 -- 172.21.15.195:0/754474853 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5220003430 con 0x7f52380c39e0 2022-06-27T12:17:16.891 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.879+0000 7f523d282700 1 -- 172.21.15.195:0/754474853 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 436688264 0 0) 0x7f5220003430 con 0x7f52380c39e0 2022-06-27T12:17:16.891 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.879+0000 7f523d282700 1 -- 172.21.15.195:0/754474853 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f52380c4160 msgr2=0x7f5238156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:17:16.892 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.879+0000 7f523d282700 1 --2- 172.21.15.195:0/754474853 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f52380c4160 0x7f5238156ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:16.892 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.879+0000 7f523d282700 1 -- 172.21.15.195:0/754474853 >> v1:172.21.15.202:6789/0 conn(0x7f523806f640 legacy=0x7f52380c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:16.892 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.879+0000 7f523d282700 1 -- 172.21.15.195:0/754474853 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f523806ed70 con 0x7f52380c39e0 2022-06-27T12:17:16.893 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.879+0000 7f523d282700 1 -- 172.21.15.195:0/754474853 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7f52300017f0 con 0x7f52380c39e0 2022-06-27T12:17:16.893 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.879+0000 7f523d282700 1 -- 172.21.15.195:0/754474853 >> v1:172.21.15.195:6789/0 conn(0x7f52380c39e0 legacy=0x7f52381535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:16.894 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.879+0000 7f523d282700 1 --2- 172.21.15.195:0/754474853 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5220003950 0x7f5220003d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:16.894 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.879+0000 7f523d282700 1 --2- 172.21.15.195:0/754474853 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f52380c4160 0x7f52200046d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:16.894 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.879+0000 7f523d282700 1 --2- 172.21.15.195:0/754474853 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5220004c10 0x7f5220005080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:16.895 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.879+0000 7f523d282700 1 -- 172.21.15.195:0/754474853 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f523806ea90 con 0x7f5220003950 2022-06-27T12:17:16.895 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.879+0000 7f523d282700 1 -- 172.21.15.195:0/754474853 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f522000ba30 con 0x7f5220004c10 2022-06-27T12:17:16.895 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.879+0000 7f523d282700 1 -- 172.21.15.195:0/754474853 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f522000bbf0 con 0x7f52380c4160 2022-06-27T12:17:16.896 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.879+0000 7f523e284700 1 --2- 172.21.15.195:0/754474853 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5220003950 0x7f5220003d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:16.896 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.879+0000 7f523da83700 1 --2- 172.21.15.195:0/754474853 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f52380c4160 0x7f52200046d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:16.896 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.879+0000 7f523ea85700 1 --2- 172.21.15.195:0/754474853 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5220004c10 0x7f5220005080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:16.896 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.879+0000 7f523e284700 1 -- 172.21.15.195:0/754474853 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f52380c4160 msgr2=0x7f52200046d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:17:16.896 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.879+0000 7f523e284700 1 --2- 172.21.15.195:0/754474853 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f52380c4160 0x7f52200046d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:16.897 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.879+0000 7f523e284700 1 -- 172.21.15.195:0/754474853 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5220004c10 msgr2=0x7f5220005080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:17:16.897 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.879+0000 7f523e284700 1 --2- 172.21.15.195:0/754474853 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5220004c10 0x7f5220005080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:16.897 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.879+0000 7f523e284700 1 -- 172.21.15.195:0/754474853 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5220003430 con 0x7f5220003950 2022-06-27T12:17:16.898 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.879+0000 7f523ea85700 1 --2- 172.21.15.195:0/754474853 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f5220004c10 0x7f5220005080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:17:16.898 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.880+0000 7f523e284700 1 --2- 172.21.15.195:0/754474853 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5220003950 0x7f5220003d60 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f5234001760 tx=0x7f5234007bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b7ff3c5ccbbdba1 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:17:16.898 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.880+0000 7f523d282700 1 -- 172.21.15.195:0/754474853 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52340050d0 con 0x7f5220003950 2022-06-27T12:17:16.899 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.880+0000 7f523d282700 1 -- 172.21.15.195:0/754474853 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5234005230 con 0x7f5220003950 2022-06-27T12:17:16.899 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.880+0000 7f523d282700 1 -- 172.21.15.195:0/754474853 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5234003750 con 0x7f5220003950 2022-06-27T12:17:16.899 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.880+0000 7f52404e8700 1 -- 172.21.15.195:0/754474853 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5220003950 msgr2=0x7f5220003d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:16.899 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.880+0000 7f52404e8700 1 --2- 172.21.15.195:0/754474853 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5220003950 0x7f5220003d60 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f5234001760 tx=0x7f5234007bb0 comp rx=0 tx=0).stop 2022-06-27T12:17:16.900 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.880+0000 7f523ea85700 1 -- 172.21.15.195:0/754474853 reap_dead start 2022-06-27T12:17:16.900 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.880+0000 7f52404e8700 1 -- 172.21.15.195:0/754474853 shutdown_connections 2022-06-27T12:17:16.900 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.880+0000 7f52404e8700 1 -- 172.21.15.195:0/754474853 >> 172.21.15.195:0/754474853 conn(0x7f52380c6350 msgr2=0x7f52380c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:17:16.900 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.880+0000 7f52404e8700 1 -- 172.21.15.195:0/754474853 shutdown_connections 2022-06-27T12:17:16.900 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.880+0000 7f52404e8700 1 -- 172.21.15.195:0/754474853 wait complete. 2022-06-27T12:17:16.901 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.881+0000 7f52404e8700 1 Processor -- start 2022-06-27T12:17:16.901 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.882+0000 7f52404e8700 1 -- start start 2022-06-27T12:17:16.901 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.882+0000 7f52404e8700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5220003950 0x7f52380c4750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:16.901 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.882+0000 7f52404e8700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f52381544c0 0x7f523814e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:16.902 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.882+0000 7f52404e8700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5238156570 0x7f523814e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:16.902 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.882+0000 7f52404e8700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5238154200 con 0x7f5238156570 2022-06-27T12:17:16.902 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.882+0000 7f52404e8700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52381549b0 con 0x7f52381544c0 2022-06-27T12:17:16.903 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.882+0000 7f52404e8700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5238156ca0 con 0x7f5220003950 2022-06-27T12:17:16.903 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.882+0000 7f523ea85700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5238156570 0x7f523814e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:16.903 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.882+0000 7f523ea85700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5238156570 0x7f523814e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:52420/0 (socket says 172.21.15.195:52420) 2022-06-27T12:17:16.904 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.882+0000 7f523ea85700 1 -- 172.21.15.195:0/1740837911 learned_addr learned my addr 172.21.15.195:0/1740837911 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:17:16.904 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.882+0000 7f523e284700 1 --2- 172.21.15.195:0/1740837911 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5220003950 0x7f52380c4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:16.905 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.882+0000 7f523da83700 1 --2- 172.21.15.195:0/1740837911 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f52381544c0 0x7f523814e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:16.906 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.883+0000 7f523ea85700 1 -- 172.21.15.195:0/1740837911 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5220003950 msgr2=0x7f52380c4750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:16.906 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.883+0000 7f523ea85700 1 --2- 172.21.15.195:0/1740837911 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5220003950 0x7f52380c4750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:16.907 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.883+0000 7f523ea85700 1 -- 172.21.15.195:0/1740837911 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f52381544c0 msgr2=0x7f523814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:16.907 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.883+0000 7f523ea85700 1 --2- 172.21.15.195:0/1740837911 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f52381544c0 0x7f523814e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:16.907 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.883+0000 7f523ea85700 1 -- 172.21.15.195:0/1740837911 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5238156f80 con 0x7f5238156570 2022-06-27T12:17:16.908 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.883+0000 7f523ea85700 1 --2- 172.21.15.195:0/1740837911 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5238156570 0x7f523814e7c0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f5238065ef0 tx=0x7f5230007f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:17:16.908 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.883+0000 7f52267fc700 1 -- 172.21.15.195:0/1740837911 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52300079e0 con 0x7f5238156570 2022-06-27T12:17:16.909 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.883+0000 7f52267fc700 1 -- 172.21.15.195:0/1740837911 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f523000e8c0 con 0x7f5238156570 2022-06-27T12:17:16.909 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.883+0000 7f52267fc700 1 -- 172.21.15.195:0/1740837911 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f523000ec80 con 0x7f5238156570 2022-06-27T12:17:16.910 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.883+0000 7f52404e8700 1 -- 172.21.15.195:0/1740837911 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5238157260 con 0x7f5238156570 2022-06-27T12:17:16.911 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.883+0000 7f52404e8700 1 -- 172.21.15.195:0/1740837911 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5238068250 con 0x7f5238156570 2022-06-27T12:17:16.911 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.884+0000 7f52404e8700 1 -- 172.21.15.195:0/1740837911 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52380722c0 con 0x7f5238156570 2022-06-27T12:17:16.912 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.884+0000 7f52267fc700 1 -- 172.21.15.195:0/1740837911 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f5230013630 con 0x7f5238156570 2022-06-27T12:17:16.912 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.885+0000 7f52267fc700 1 --2- 172.21.15.195:0/1740837911 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5218063d30 0x7f52180661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:16.913 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.885+0000 7f52267fc700 1 -- 172.21.15.195:0/1740837911 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7664+0+0 (secure 0 0 0) 0x7f5230081150 con 0x7f5238156570 2022-06-27T12:17:16.913 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.885+0000 7f523e284700 1 --2- 172.21.15.195:0/1740837911 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5218063d30 0x7f52180661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:16.914 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.885+0000 7f523e284700 1 --2- 172.21.15.195:0/1740837911 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5218063d30 0x7f52180661e0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f5234000ff0 tx=0x7f5234015000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:17:16.914 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:16.889+0000 7f52267fc700 1 -- 172.21.15.195:0/1740837911 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f523004f700 con 0x7f5238156570 2022-06-27T12:17:17.119 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:17.119+0000 7f52404e8700 1 -- 172.21.15.195:0/1740837911 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5238156a80 con 0x7f5238156570 2022-06-27T12:17:17.132 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:17.119+0000 7f52267fc700 1 -- 172.21.15.195:0/1740837911 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) v1 ==== 95+0+5371 (secure 0 0 0) 0x7f5230052d20 con 0x7f5238156570 2022-06-27T12:17:17.133 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:17:17.134 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":110,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:17:15.342067+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7165},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7165":{"gid":7165,"name":"d","rank":0,"incarnation":108,"state":"up:rejoin","state_seq":22,"addr":"172.21.15.202:6835/3551478822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":3551478822},{"type":"v1","addr":"172.21.15.202:6835","nonce":3551478822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:17:17.137 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:17.121+0000 7f52404e8700 1 -- 172.21.15.195:0/1740837911 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5218063d30 msgr2=0x7f52180661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:17.139 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:17.121+0000 7f52404e8700 1 --2- 172.21.15.195:0/1740837911 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5218063d30 0x7f52180661e0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f5234000ff0 tx=0x7f5234015000 comp rx=0 tx=0).stop 2022-06-27T12:17:17.139 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:17.121+0000 7f52404e8700 1 -- 172.21.15.195:0/1740837911 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5238156570 msgr2=0x7f523814e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:17.140 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:17.121+0000 7f52404e8700 1 --2- 172.21.15.195:0/1740837911 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5238156570 0x7f523814e7c0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f5238065ef0 tx=0x7f5230007f10 comp rx=0 tx=0).stop 2022-06-27T12:17:17.140 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:17.121+0000 7f52404e8700 1 -- 172.21.15.195:0/1740837911 shutdown_connections 2022-06-27T12:17:17.141 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:17.121+0000 7f52404e8700 1 --2- 172.21.15.195:0/1740837911 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5218063d30 0x7f52180661e0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:17.142 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:17.121+0000 7f52404e8700 1 --2- 172.21.15.195:0/1740837911 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5238156570 0x7f523814e7c0 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:17.142 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:17.121+0000 7f52404e8700 1 --2- 172.21.15.195:0/1740837911 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f5220003950 0x7f52380c4750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:17.143 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:17.121+0000 7f52404e8700 1 --2- 172.21.15.195:0/1740837911 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f52381544c0 0x7f523814e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:17.143 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:17.121+0000 7f52404e8700 1 -- 172.21.15.195:0/1740837911 >> 172.21.15.195:0/1740837911 conn(0x7f52380c6350 msgr2=0x7f52380c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:17:17.144 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:17.121+0000 7f52404e8700 1 -- 172.21.15.195:0/1740837911 shutdown_connections 2022-06-27T12:17:17.144 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:17.121+0000 7f52404e8700 1 -- 172.21.15.195:0/1740837911 wait complete. 2022-06-27T12:17:17.144 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 110 2022-06-27T12:17:17.145 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:17:19.146 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:17:19.314 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.281+0000 7fa8a7d5e700 1 Processor -- start 2022-06-27T12:17:19.315 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.281+0000 7fa8a7d5e700 1 -- start start 2022-06-27T12:17:19.316 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.282+0000 7fa8a7d5e700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa8a006f640 0x7fa8a0156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:19.316 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.282+0000 7fa8a7d5e700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8a00c1a20 con 0x7fa8a00c39e0 2022-06-27T12:17:19.316 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.282+0000 7fa8a7d5e700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8a00c1130 con 0x7fa8a00c4160 2022-06-27T12:17:19.317 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.282+0000 7fa8a7d5e700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a006ea90 con 0x7fa8a006f640 2022-06-27T12:17:19.317 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.282+0000 7fa8a62fb700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fa8a00c39e0 0x7fa8a01535a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:58614/0 (socket says 172.21.15.195:58614) 2022-06-27T12:17:19.318 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.282+0000 7fa8a62fb700 1 -- 172.21.15.195:0/2313151187 learned_addr learned my addr 172.21.15.195:0/2313151187 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:17:19.318 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.282+0000 7fa8a5afa700 1 --2- 172.21.15.195:0/2313151187 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa8a006f640 0x7fa8a0156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:19.319 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.282+0000 7fa8a4af8700 1 -- 172.21.15.195:0/2313151187 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 506391266 0 0) 0x7fa8a00c1a20 con 0x7fa8a00c39e0 2022-06-27T12:17:19.319 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.282+0000 7fa8a4af8700 1 -- 172.21.15.195:0/2313151187 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa888003430 con 0x7fa8a00c39e0 2022-06-27T12:17:19.320 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.282+0000 7fa8a4af8700 1 -- 172.21.15.195:0/2313151187 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3390653958 0 0) 0x7fa888003430 con 0x7fa8a00c39e0 2022-06-27T12:17:19.320 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.283+0000 7fa8a4af8700 1 -- 172.21.15.195:0/2313151187 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa8a006f640 msgr2=0x7fa8a0156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:17:19.320 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.283+0000 7fa8a4af8700 1 --2- 172.21.15.195:0/2313151187 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa8a006f640 0x7fa8a0156ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:19.321 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.283+0000 7fa8a4af8700 1 -- 172.21.15.195:0/2313151187 >> v1:172.21.15.202:6789/0 conn(0x7fa8a00c4160 legacy=0x7fa8a00c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:19.321 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.283+0000 7fa8a4af8700 1 -- 172.21.15.195:0/2313151187 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8a006ed70 con 0x7fa8a00c39e0 2022-06-27T12:17:19.322 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.283+0000 7fa8a4af8700 1 -- 172.21.15.195:0/2313151187 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7fa8980017f0 con 0x7fa8a00c39e0 2022-06-27T12:17:19.322 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.283+0000 7fa8a4af8700 1 -- 172.21.15.195:0/2313151187 >> v1:172.21.15.195:6789/0 conn(0x7fa8a00c39e0 legacy=0x7fa8a01535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:19.322 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.283+0000 7fa8a4af8700 1 --2- 172.21.15.195:0/2313151187 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa888003950 0x7fa888003d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:19.323 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.283+0000 7fa8a4af8700 1 --2- 172.21.15.195:0/2313151187 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa8a006f640 0x7fa8880046d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:19.323 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.283+0000 7fa8a4af8700 1 --2- 172.21.15.195:0/2313151187 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa888004c10 0x7fa888005080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:19.325 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.283+0000 7fa8a4af8700 1 -- 172.21.15.195:0/2313151187 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a006ea90 con 0x7fa888003950 2022-06-27T12:17:19.325 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.283+0000 7fa8a4af8700 1 -- 172.21.15.195:0/2313151187 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa88800b9d0 con 0x7fa888004c10 2022-06-27T12:17:19.326 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.283+0000 7fa8a4af8700 1 -- 172.21.15.195:0/2313151187 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa88800bb90 con 0x7fa8a006f640 2022-06-27T12:17:19.326 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.283+0000 7fa8a5afa700 1 --2- 172.21.15.195:0/2313151187 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa888003950 0x7fa888003d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:19.326 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.283+0000 7fa8a52f9700 1 --2- 172.21.15.195:0/2313151187 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa8a006f640 0x7fa8880046d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:19.327 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.283+0000 7fa8a62fb700 1 --2- 172.21.15.195:0/2313151187 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa888004c10 0x7fa888005080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:19.327 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.283+0000 7fa8a5afa700 1 -- 172.21.15.195:0/2313151187 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa8a006f640 msgr2=0x7fa8880046d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:17:19.328 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.283+0000 7fa8a5afa700 1 --2- 172.21.15.195:0/2313151187 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa8a006f640 0x7fa8880046d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:19.328 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.283+0000 7fa8a5afa700 1 -- 172.21.15.195:0/2313151187 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa888004c10 msgr2=0x7fa888005080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:17:19.329 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.283+0000 7fa8a5afa700 1 --2- 172.21.15.195:0/2313151187 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa888004c10 0x7fa888005080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:19.331 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.283+0000 7fa8a5afa700 1 -- 172.21.15.195:0/2313151187 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa888003430 con 0x7fa888003950 2022-06-27T12:17:19.331 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.283+0000 7fa8a62fb700 1 --2- 172.21.15.195:0/2313151187 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa888004c10 0x7fa888005080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:17:19.331 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.283+0000 7fa8a5afa700 1 --2- 172.21.15.195:0/2313151187 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa888003950 0x7fa888003d20 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fa89c009ac0 tx=0x7fa89c009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28fbc5f8af1610ab server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:17:19.332 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.283+0000 7fa8a4af8700 1 -- 172.21.15.195:0/2313151187 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa89c009470 con 0x7fa888003950 2022-06-27T12:17:19.332 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.283+0000 7fa8a4af8700 1 -- 172.21.15.195:0/2313151187 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa89c00e640 con 0x7fa888003950 2022-06-27T12:17:19.332 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.283+0000 7fa8a4af8700 1 -- 172.21.15.195:0/2313151187 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa89c00e9e0 con 0x7fa888003950 2022-06-27T12:17:19.332 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.284+0000 7fa8a7d5e700 1 -- 172.21.15.195:0/2313151187 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa888003950 msgr2=0x7fa888003d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:19.332 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.284+0000 7fa8a7d5e700 1 --2- 172.21.15.195:0/2313151187 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa888003950 0x7fa888003d20 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fa89c009ac0 tx=0x7fa89c009060 comp rx=0 tx=0).stop 2022-06-27T12:17:19.333 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.284+0000 7fa8a62fb700 1 -- 172.21.15.195:0/2313151187 reap_dead start 2022-06-27T12:17:19.333 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.284+0000 7fa8a7d5e700 1 -- 172.21.15.195:0/2313151187 shutdown_connections 2022-06-27T12:17:19.337 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.284+0000 7fa8a7d5e700 1 -- 172.21.15.195:0/2313151187 >> 172.21.15.195:0/2313151187 conn(0x7fa8a00c6350 msgr2=0x7fa8a00c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:17:19.338 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.284+0000 7fa8a7d5e700 1 -- 172.21.15.195:0/2313151187 shutdown_connections 2022-06-27T12:17:19.338 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.284+0000 7fa8a7d5e700 1 -- 172.21.15.195:0/2313151187 wait complete. 2022-06-27T12:17:19.338 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.285+0000 7fa8a7d5e700 1 Processor -- start 2022-06-27T12:17:19.339 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.286+0000 7fa8a7d5e700 1 -- start start 2022-06-27T12:17:19.339 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.286+0000 7fa8a7d5e700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa888003950 0x7fa8a00bc990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:19.340 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.286+0000 7fa8a7d5e700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa8a00bd540 0x7fa8a0156a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:19.340 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.286+0000 7fa8a7d5e700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa8a01664b0 0x7fa8a01668e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:19.341 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.286+0000 7fa8a7d5e700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a00c05c0 con 0x7fa8a01664b0 2022-06-27T12:17:19.341 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.286+0000 7fa8a7d5e700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a00bed50 con 0x7fa8a00bd540 2022-06-27T12:17:19.341 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.286+0000 7fa8a7d5e700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a0166e20 con 0x7fa888003950 2022-06-27T12:17:19.342 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.287+0000 7fa8a62fb700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa8a01664b0 0x7fa8a01668e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:19.343 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.287+0000 7fa8a62fb700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa8a01664b0 0x7fa8a01668e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:52438/0 (socket says 172.21.15.195:52438) 2022-06-27T12:17:19.343 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.287+0000 7fa8a62fb700 1 -- 172.21.15.195:0/2551152254 learned_addr learned my addr 172.21.15.195:0/2551152254 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:17:19.344 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.287+0000 7fa8a5afa700 1 --2- 172.21.15.195:0/2551152254 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa888003950 0x7fa8a00bc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:19.344 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.287+0000 7fa8a52f9700 1 --2- 172.21.15.195:0/2551152254 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa8a00bd540 0x7fa8a0156a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:19.344 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.287+0000 7fa8a62fb700 1 -- 172.21.15.195:0/2551152254 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa888003950 msgr2=0x7fa8a00bc990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:19.345 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.287+0000 7fa8a62fb700 1 --2- 172.21.15.195:0/2551152254 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa888003950 0x7fa8a00bc990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:19.345 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.287+0000 7fa8a62fb700 1 -- 172.21.15.195:0/2551152254 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa8a00bd540 msgr2=0x7fa8a0156a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:19.348 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.287+0000 7fa8a62fb700 1 --2- 172.21.15.195:0/2551152254 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa8a00bd540 0x7fa8a0156a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:19.348 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.287+0000 7fa8a62fb700 1 -- 172.21.15.195:0/2551152254 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8a00bd050 con 0x7fa8a01664b0 2022-06-27T12:17:19.349 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.287+0000 7fa8a62fb700 1 --2- 172.21.15.195:0/2551152254 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa8a01664b0 0x7fa8a01668e0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7fa898000f80 tx=0x7fa898001300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:17:19.349 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.287+0000 7fa88dffb700 1 -- 172.21.15.195:0/2551152254 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa89800e8c0 con 0x7fa8a01664b0 2022-06-27T12:17:19.350 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.287+0000 7fa88dffb700 1 -- 172.21.15.195:0/2551152254 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa89800ea20 con 0x7fa8a01664b0 2022-06-27T12:17:19.350 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.287+0000 7fa8a7d5e700 1 -- 172.21.15.195:0/2551152254 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8a00be140 con 0x7fa8a01664b0 2022-06-27T12:17:19.350 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.287+0000 7fa88dffb700 1 -- 172.21.15.195:0/2551152254 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa89800ede0 con 0x7fa8a01664b0 2022-06-27T12:17:19.351 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.288+0000 7fa8a7d5e700 1 -- 172.21.15.195:0/2551152254 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8a00be700 con 0x7fa8a01664b0 2022-06-27T12:17:19.351 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.289+0000 7fa8a7d5e700 1 -- 172.21.15.195:0/2551152254 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8a00722c0 con 0x7fa8a01664b0 2022-06-27T12:17:19.352 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.289+0000 7fa88dffb700 1 -- 172.21.15.195:0/2551152254 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fa898014640 con 0x7fa8a01664b0 2022-06-27T12:17:19.352 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.289+0000 7fa88dffb700 1 --2- 172.21.15.195:0/2551152254 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa880063d30 0x7fa8800661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:19.353 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.289+0000 7fa88dffb700 1 -- 172.21.15.195:0/2551152254 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7664+0+0 (secure 0 0 0) 0x7fa898080c60 con 0x7fa8a01664b0 2022-06-27T12:17:19.353 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.289+0000 7fa8a5afa700 1 --2- 172.21.15.195:0/2551152254 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa880063d30 0x7fa8800661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:19.353 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.290+0000 7fa8a5afa700 1 --2- 172.21.15.195:0/2551152254 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa880063d30 0x7fa8800661e0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fa89c006d90 tx=0x7fa89c01a000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:17:19.354 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.293+0000 7fa88dffb700 1 -- 172.21.15.195:0/2551152254 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fa89804ac90 con 0x7fa8a01664b0 2022-06-27T12:17:19.524 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.521+0000 7fa8a7d5e700 1 -- 172.21.15.195:0/2551152254 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa8a006f240 con 0x7fa8a01664b0 2022-06-27T12:17:19.526 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.521+0000 7fa88dffb700 1 -- 172.21.15.195:0/2551152254 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) v1 ==== 95+0+5371 (secure 0 0 0) 0x7fa89804f200 con 0x7fa8a01664b0 2022-06-27T12:17:19.527 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:17:19.527 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":110,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:17:15.342067+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7165},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7165":{"gid":7165,"name":"d","rank":0,"incarnation":108,"state":"up:rejoin","state_seq":22,"addr":"172.21.15.202:6835/3551478822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":3551478822},{"type":"v1","addr":"172.21.15.202:6835","nonce":3551478822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:17:19.528 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.523+0000 7fa8a7d5e700 1 -- 172.21.15.195:0/2551152254 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa880063d30 msgr2=0x7fa8800661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:19.530 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.523+0000 7fa8a7d5e700 1 --2- 172.21.15.195:0/2551152254 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa880063d30 0x7fa8800661e0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fa89c006d90 tx=0x7fa89c01a000 comp rx=0 tx=0).stop 2022-06-27T12:17:19.530 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.523+0000 7fa8a7d5e700 1 -- 172.21.15.195:0/2551152254 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa8a01664b0 msgr2=0x7fa8a01668e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:19.531 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.523+0000 7fa8a7d5e700 1 --2- 172.21.15.195:0/2551152254 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa8a01664b0 0x7fa8a01668e0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7fa898000f80 tx=0x7fa898001300 comp rx=0 tx=0).stop 2022-06-27T12:17:19.533 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.523+0000 7fa8a7d5e700 1 -- 172.21.15.195:0/2551152254 shutdown_connections 2022-06-27T12:17:19.534 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.523+0000 7fa8a7d5e700 1 --2- 172.21.15.195:0/2551152254 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fa880063d30 0x7fa8800661e0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:19.534 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.523+0000 7fa8a7d5e700 1 --2- 172.21.15.195:0/2551152254 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa8a01664b0 0x7fa8a01668e0 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:19.535 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.523+0000 7fa8a7d5e700 1 --2- 172.21.15.195:0/2551152254 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fa888003950 0x7fa8a00bc990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:19.536 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.523+0000 7fa8a7d5e700 1 --2- 172.21.15.195:0/2551152254 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fa8a00bd540 0x7fa8a0156a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:19.536 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.523+0000 7fa8a7d5e700 1 -- 172.21.15.195:0/2551152254 >> 172.21.15.195:0/2551152254 conn(0x7fa8a00c6350 msgr2=0x7fa8a01504e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:17:19.537 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.523+0000 7fa8a7d5e700 1 -- 172.21.15.195:0/2551152254 shutdown_connections 2022-06-27T12:17:19.537 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:19.523+0000 7fa8a7d5e700 1 -- 172.21.15.195:0/2551152254 wait complete. 2022-06-27T12:17:19.537 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 110 2022-06-27T12:17:19.539 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:17:21.542 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:17:21.691 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.677+0000 7f10015e0700 1 Processor -- start 2022-06-27T12:17:21.692 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.677+0000 7f10015e0700 1 -- start start 2022-06-27T12:17:21.692 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.678+0000 7f10015e0700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0ffc0c40d0 0x7f0ffc157010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:21.692 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.678+0000 7f10015e0700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ffc156b80 con 0x7f0ffc067ed0 2022-06-27T12:17:21.692 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.678+0000 7f10015e0700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ffc06a040 con 0x7f0ffc0c6190 2022-06-27T12:17:21.693 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.678+0000 7f10015e0700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ffc068370 con 0x7f0ffc0c40d0 2022-06-27T12:17:21.694 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.678+0000 7f0ffb7fe700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f0ffc067ed0 0x7f0ffc14cb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:58634/0 (socket says 172.21.15.195:58634) 2022-06-27T12:17:21.694 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.678+0000 7f0ffb7fe700 1 -- 172.21.15.195:0/2439350152 learned_addr learned my addr 172.21.15.195:0/2439350152 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:17:21.695 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.678+0000 7f0ffaffd700 1 --2- 172.21.15.195:0/2439350152 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0ffc0c40d0 0x7f0ffc157010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:21.695 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.678+0000 7f0ff9ffb700 1 -- 172.21.15.195:0/2439350152 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 343571806 0 0) 0x7f0ffc156b80 con 0x7f0ffc067ed0 2022-06-27T12:17:21.695 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.678+0000 7f0ff9ffb700 1 -- 172.21.15.195:0/2439350152 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fe8003430 con 0x7f0ffc067ed0 2022-06-27T12:17:21.695 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.678+0000 7f0ff9ffb700 1 -- 172.21.15.195:0/2439350152 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 381322831 0 0) 0x7f0ffc06a040 con 0x7f0ffc0c6190 2022-06-27T12:17:21.696 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.678+0000 7f0ff9ffb700 1 -- 172.21.15.195:0/2439350152 --> v1:172.21.15.202:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ffc156b80 con 0x7f0ffc0c6190 2022-06-27T12:17:21.696 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.678+0000 7f0ff9ffb700 1 -- 172.21.15.195:0/2439350152 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3455007713 0 0) 0x7f0fe8003430 con 0x7f0ffc067ed0 2022-06-27T12:17:21.698 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.678+0000 7f0ff9ffb700 1 -- 172.21.15.195:0/2439350152 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0ffc0c40d0 msgr2=0x7f0ffc157010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:17:21.698 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.678+0000 7f0ff9ffb700 1 --2- 172.21.15.195:0/2439350152 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0ffc0c40d0 0x7f0ffc157010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:21.699 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.678+0000 7f0ff9ffb700 1 -- 172.21.15.195:0/2439350152 >> v1:172.21.15.202:6789/0 conn(0x7f0ffc0c6190 legacy=0x7f0ffc0c44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:21.699 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.678+0000 7f0ff9ffb700 1 -- 172.21.15.195:0/2439350152 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ffc06a300 con 0x7f0ffc067ed0 2022-06-27T12:17:21.700 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.678+0000 7f0ff9ffb700 1 -- 172.21.15.195:0/2439350152 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7f0ff40017f0 con 0x7f0ffc067ed0 2022-06-27T12:17:21.700 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.678+0000 7f0ff9ffb700 1 -- 172.21.15.195:0/2439350152 >> v1:172.21.15.195:6789/0 conn(0x7f0ffc067ed0 legacy=0x7f0ffc14cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:21.701 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.678+0000 7f0ff9ffb700 1 --2- 172.21.15.195:0/2439350152 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0fe8003ae0 0x7f0fe8003ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:21.701 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.678+0000 7f0ff9ffb700 1 --2- 172.21.15.195:0/2439350152 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0fe80046d0 0x7f0fe8004b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:21.702 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.679+0000 7f0ff9ffb700 1 --2- 172.21.15.195:0/2439350152 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0ffc0c40d0 0x7f0fe80050e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:21.702 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.679+0000 7f0ff9ffb700 1 -- 172.21.15.195:0/2439350152 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe800bab0 con 0x7f0fe80046d0 2022-06-27T12:17:21.703 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.679+0000 7f0ff9ffb700 1 -- 172.21.15.195:0/2439350152 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe800bbf0 con 0x7f0fe8003ae0 2022-06-27T12:17:21.703 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.679+0000 7f0ff9ffb700 1 -- 172.21.15.195:0/2439350152 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe800bdb0 con 0x7f0ffc0c40d0 2022-06-27T12:17:21.703 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.679+0000 7f0ffaffd700 1 --2- 172.21.15.195:0/2439350152 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0fe8003ae0 0x7f0fe8003ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:21.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.679+0000 7f0ffa7fc700 1 --2- 172.21.15.195:0/2439350152 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0fe80046d0 0x7f0fe8004b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:21.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.679+0000 7f0ffb7fe700 1 --2- 172.21.15.195:0/2439350152 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0ffc0c40d0 0x7f0fe80050e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:21.705 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.679+0000 7f0ffa7fc700 1 -- 172.21.15.195:0/2439350152 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0ffc0c40d0 msgr2=0x7f0fe80050e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:17:21.705 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.679+0000 7f0ffa7fc700 1 --2- 172.21.15.195:0/2439350152 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0ffc0c40d0 0x7f0fe80050e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:21.707 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.679+0000 7f0ffa7fc700 1 -- 172.21.15.195:0/2439350152 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0fe8003ae0 msgr2=0x7f0fe8003ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:17:21.707 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.679+0000 7f0ffa7fc700 1 --2- 172.21.15.195:0/2439350152 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0fe8003ae0 0x7f0fe8003ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:21.707 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.679+0000 7f0ffa7fc700 1 -- 172.21.15.195:0/2439350152 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fe8003430 con 0x7f0fe80046d0 2022-06-27T12:17:21.707 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.679+0000 7f0ffaffd700 1 --2- 172.21.15.195:0/2439350152 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0fe8003ae0 0x7f0fe8003ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:17:21.708 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.679+0000 7f0ffa7fc700 1 --2- 172.21.15.195:0/2439350152 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0fe80046d0 0x7f0fe8004b40 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f0ff00018a0 tx=0x7f0ff0007bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3629e564187f35c3 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:17:21.708 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.679+0000 7f0ff9ffb700 1 -- 172.21.15.195:0/2439350152 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff00050d0 con 0x7f0fe80046d0 2022-06-27T12:17:21.708 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.679+0000 7f0ff9ffb700 1 -- 172.21.15.195:0/2439350152 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ff0005250 con 0x7f0fe80046d0 2022-06-27T12:17:21.708 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.679+0000 7f0ff9ffb700 1 -- 172.21.15.195:0/2439350152 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff0003750 con 0x7f0fe80046d0 2022-06-27T12:17:21.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.680+0000 7f10015e0700 1 -- 172.21.15.195:0/2439350152 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0fe80046d0 msgr2=0x7f0fe8004b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:21.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.680+0000 7f10015e0700 1 --2- 172.21.15.195:0/2439350152 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0fe80046d0 0x7f0fe8004b40 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f0ff00018a0 tx=0x7f0ff0007bf0 comp rx=0 tx=0).stop 2022-06-27T12:17:21.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.680+0000 7f0ffb7fe700 1 -- 172.21.15.195:0/2439350152 reap_dead start 2022-06-27T12:17:21.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.680+0000 7f10015e0700 1 -- 172.21.15.195:0/2439350152 shutdown_connections 2022-06-27T12:17:21.710 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.680+0000 7f10015e0700 1 -- 172.21.15.195:0/2439350152 >> 172.21.15.195:0/2439350152 conn(0x7f0ffc0ccb80 msgr2=0x7f0ffc0ccf80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:17:21.710 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.680+0000 7f10015e0700 1 -- 172.21.15.195:0/2439350152 shutdown_connections 2022-06-27T12:17:21.710 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.680+0000 7f10015e0700 1 -- 172.21.15.195:0/2439350152 wait complete. 2022-06-27T12:17:21.710 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.681+0000 7f10015e0700 1 Processor -- start 2022-06-27T12:17:21.711 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.681+0000 7f10015e0700 1 -- start start 2022-06-27T12:17:21.711 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.682+0000 7f10015e0700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0fe80046d0 0x7f0ffc1538c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:21.711 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.682+0000 7f10015e0700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0ffc159610 0x7f0ffc15bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:21.712 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.682+0000 7f10015e0700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0ffc14fce0 0x7f0ffc150130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:21.712 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.682+0000 7f10015e0700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ffc0c0f10 con 0x7f0fe80046d0 2022-06-27T12:17:21.712 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.682+0000 7f10015e0700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ffc166030 con 0x7f0ffc14fce0 2022-06-27T12:17:21.712 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.682+0000 7f10015e0700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ffc166890 con 0x7f0ffc159610 2022-06-27T12:17:21.713 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.682+0000 7f0ffaffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0fe80046d0 0x7f0ffc1538c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:21.713 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.682+0000 7f0ffaffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0fe80046d0 0x7f0ffc1538c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:52454/0 (socket says 172.21.15.195:52454) 2022-06-27T12:17:21.713 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.682+0000 7f0ffaffd700 1 -- 172.21.15.195:0/421116437 learned_addr learned my addr 172.21.15.195:0/421116437 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:17:21.713 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.682+0000 7f0ffa7fc700 1 --2- 172.21.15.195:0/421116437 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0ffc159610 0x7f0ffc15bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:21.714 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.682+0000 7f0ffb7fe700 1 --2- 172.21.15.195:0/421116437 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0ffc14fce0 0x7f0ffc150130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:21.714 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.682+0000 7f0ffaffd700 1 -- 172.21.15.195:0/421116437 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0ffc159610 msgr2=0x7f0ffc15bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:21.714 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.682+0000 7f0ffaffd700 1 --2- 172.21.15.195:0/421116437 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0ffc159610 0x7f0ffc15bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:21.714 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.682+0000 7f0ffaffd700 1 -- 172.21.15.195:0/421116437 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0ffc14fce0 msgr2=0x7f0ffc150130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:21.715 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.682+0000 7f0ffaffd700 1 --2- 172.21.15.195:0/421116437 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0ffc14fce0 0x7f0ffc150130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:21.715 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.682+0000 7f0ffaffd700 1 -- 172.21.15.195:0/421116437 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ffc0c06b0 con 0x7f0fe80046d0 2022-06-27T12:17:21.715 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.682+0000 7f0ffa7fc700 1 --2- 172.21.15.195:0/421116437 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0ffc159610 0x7f0ffc15bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:17:21.715 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.682+0000 7f0ffaffd700 1 --2- 172.21.15.195:0/421116437 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0fe80046d0 0x7f0ffc1538c0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f0fec011d40 tx=0x7f0fec009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:17:21.716 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.683+0000 7f0fe2ffd700 1 -- 172.21.15.195:0/421116437 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fec004020 con 0x7f0fe80046d0 2022-06-27T12:17:21.716 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.683+0000 7f0fe2ffd700 1 -- 172.21.15.195:0/421116437 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fec004180 con 0x7f0fe80046d0 2022-06-27T12:17:21.716 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.683+0000 7f10015e0700 1 -- 172.21.15.195:0/421116437 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ffc0c0950 con 0x7f0fe80046d0 2022-06-27T12:17:21.716 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.683+0000 7f0fe2ffd700 1 -- 172.21.15.195:0/421116437 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fec00faa0 con 0x7f0fe80046d0 2022-06-27T12:17:21.716 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.683+0000 7f10015e0700 1 -- 172.21.15.195:0/421116437 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ffc166240 con 0x7f0fe80046d0 2022-06-27T12:17:21.717 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.683+0000 7f10015e0700 1 -- 172.21.15.195:0/421116437 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ffc0722c0 con 0x7f0fe80046d0 2022-06-27T12:17:21.717 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.684+0000 7f0fe2ffd700 1 -- 172.21.15.195:0/421116437 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f0fec003520 con 0x7f0fe80046d0 2022-06-27T12:17:21.717 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.684+0000 7f0fe2ffd700 1 --2- 172.21.15.195:0/421116437 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0fdc063d30 0x7f0fdc0661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:21.717 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.684+0000 7f0fe2ffd700 1 -- 172.21.15.195:0/421116437 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7664+0+0 (secure 0 0 0) 0x7f0fec018070 con 0x7f0fe80046d0 2022-06-27T12:17:21.718 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.684+0000 7f0ffa7fc700 1 --2- 172.21.15.195:0/421116437 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0fdc063d30 0x7f0fdc0661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:21.718 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.685+0000 7f0ffa7fc700 1 --2- 172.21.15.195:0/421116437 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0fdc063d30 0x7f0fdc0661e0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc15a6d0 tx=0x7f0ff0015000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:17:21.718 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.689+0000 7f0fe2ffd700 1 -- 172.21.15.195:0/421116437 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f0fec04fea0 con 0x7f0fe80046d0 2022-06-27T12:17:21.921 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.921+0000 7f10015e0700 1 -- 172.21.15.195:0/421116437 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ffc069d80 con 0x7f0fe80046d0 2022-06-27T12:17:21.922 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.922+0000 7f0fe2ffd700 1 -- 172.21.15.195:0/421116437 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) v1 ==== 95+0+5371 (secure 0 0 0) 0x7f0fec0534c0 con 0x7f0fe80046d0 2022-06-27T12:17:21.923 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:17:21.923 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":110,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:17:15.342067+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7165},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7165":{"gid":7165,"name":"d","rank":0,"incarnation":108,"state":"up:rejoin","state_seq":22,"addr":"172.21.15.202:6835/3551478822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":3551478822},{"type":"v1","addr":"172.21.15.202:6835","nonce":3551478822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:17:21.927 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.924+0000 7f10015e0700 1 -- 172.21.15.195:0/421116437 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0fdc063d30 msgr2=0x7f0fdc0661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:21.927 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.924+0000 7f10015e0700 1 --2- 172.21.15.195:0/421116437 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0fdc063d30 0x7f0fdc0661e0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc15a6d0 tx=0x7f0ff0015000 comp rx=0 tx=0).stop 2022-06-27T12:17:21.928 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.924+0000 7f10015e0700 1 -- 172.21.15.195:0/421116437 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0fe80046d0 msgr2=0x7f0ffc1538c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:21.929 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.924+0000 7f10015e0700 1 --2- 172.21.15.195:0/421116437 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0fe80046d0 0x7f0ffc1538c0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f0fec011d40 tx=0x7f0fec009060 comp rx=0 tx=0).stop 2022-06-27T12:17:21.929 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.924+0000 7f10015e0700 1 -- 172.21.15.195:0/421116437 shutdown_connections 2022-06-27T12:17:21.930 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.924+0000 7f10015e0700 1 --2- 172.21.15.195:0/421116437 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0fdc063d30 0x7f0fdc0661e0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:21.931 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.924+0000 7f10015e0700 1 --2- 172.21.15.195:0/421116437 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0fe80046d0 0x7f0ffc1538c0 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:21.931 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.924+0000 7f10015e0700 1 --2- 172.21.15.195:0/421116437 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f0ffc14fce0 0x7f0ffc150130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:21.931 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.924+0000 7f10015e0700 1 --2- 172.21.15.195:0/421116437 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f0ffc159610 0x7f0ffc15bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:21.932 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.924+0000 7f10015e0700 1 -- 172.21.15.195:0/421116437 >> 172.21.15.195:0/421116437 conn(0x7f0ffc0ccb80 msgr2=0x7f0ffc0cc170 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:17:21.932 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.924+0000 7f10015e0700 1 -- 172.21.15.195:0/421116437 shutdown_connections 2022-06-27T12:17:21.933 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:21.924+0000 7f10015e0700 1 -- 172.21.15.195:0/421116437 wait complete. 2022-06-27T12:17:21.934 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 110 2022-06-27T12:17:21.937 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:17:23.948 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:17:24.089 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.089+0000 7fac71734700 1 Processor -- start 2022-06-27T12:17:24.091 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.089+0000 7fac71734700 1 -- start start 2022-06-27T12:17:24.098 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.090+0000 7fac71734700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fac6c0c6190 0x7fac6c157010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:24.098 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.090+0000 7fac71734700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fac6c067ed0 0x7fac6c14cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:24.099 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.090+0000 7fac71734700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac6c156b80 con 0x7fac6c0c40d0 2022-06-27T12:17:24.099 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.090+0000 7fac71734700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac6c068350 con 0x7fac6c067ed0 2022-06-27T12:17:24.099 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.090+0000 7fac71734700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac6c06a040 con 0x7fac6c0c6190 2022-06-27T12:17:24.099 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.090+0000 7fac6affd700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fac6c0c40d0 0x7fac6c0c44a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:58652/0 (socket says 172.21.15.195:58652) 2022-06-27T12:17:24.100 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.090+0000 7fac6affd700 1 -- 172.21.15.195:0/1337343183 learned_addr learned my addr 172.21.15.195:0/1337343183 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:17:24.100 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.090+0000 7fac6a7fc700 1 --2- 172.21.15.195:0/1337343183 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fac6c0c6190 0x7fac6c157010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:24.100 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.090+0000 7fac6b7fe700 1 --2- 172.21.15.195:0/1337343183 >> v2:172.21.15.202:3300/0 conn(0x7fac6c067ed0 0x7fac6c14cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:24.100 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.090+0000 7fac69ffb700 1 -- 172.21.15.195:0/1337343183 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 197648913 0 0) 0x7fac6c156b80 con 0x7fac6c0c40d0 2022-06-27T12:17:24.101 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.090+0000 7fac69ffb700 1 -- 172.21.15.195:0/1337343183 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac58003430 con 0x7fac6c0c40d0 2022-06-27T12:17:24.102 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.091+0000 7fac69ffb700 1 -- 172.21.15.195:0/1337343183 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2602749914 0 0) 0x7fac58003430 con 0x7fac6c0c40d0 2022-06-27T12:17:24.102 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.091+0000 7fac6a7fc700 1 -- 172.21.15.195:0/1337343183 >> v2:172.21.15.202:3300/0 conn(0x7fac6c067ed0 msgr2=0x7fac6c14cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:17:24.103 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.091+0000 7fac6a7fc700 1 --2- 172.21.15.195:0/1337343183 >> v2:172.21.15.202:3300/0 conn(0x7fac6c067ed0 0x7fac6c14cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:24.103 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.091+0000 7fac6a7fc700 1 -- 172.21.15.195:0/1337343183 >> v1:172.21.15.195:6789/0 conn(0x7fac6c0c40d0 legacy=0x7fac6c0c44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:24.103 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.091+0000 7fac6a7fc700 1 -- 172.21.15.195:0/1337343183 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac6c06a300 con 0x7fac6c0c6190 2022-06-27T12:17:24.103 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.091+0000 7fac6b7fe700 1 --2- 172.21.15.195:0/1337343183 >> v2:172.21.15.202:3300/0 conn(0x7fac6c067ed0 0x7fac6c14cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:17:24.104 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.091+0000 7fac6a7fc700 1 --2- 172.21.15.195:0/1337343183 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fac6c0c6190 0x7fac6c157010 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fac600097a0 tx=0x7fac60009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77e895372d984bc9 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:17:24.104 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.091+0000 7fac69ffb700 1 -- 172.21.15.195:0/1337343183 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac5c0017f0 con 0x7fac6c0c6190 2022-06-27T12:17:24.105 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.091+0000 7fac69ffb700 1 -- 172.21.15.195:0/1337343183 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac60004030 con 0x7fac6c0c6190 2022-06-27T12:17:24.105 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.091+0000 7fac69ffb700 1 -- 172.21.15.195:0/1337343183 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac600043b0 con 0x7fac6c0c6190 2022-06-27T12:17:24.105 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.091+0000 7fac71734700 1 -- 172.21.15.195:0/1337343183 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fac6c0c6190 msgr2=0x7fac6c157010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:24.105 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.091+0000 7fac71734700 1 --2- 172.21.15.195:0/1337343183 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fac6c0c6190 0x7fac6c157010 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fac600097a0 tx=0x7fac60009060 comp rx=0 tx=0).stop 2022-06-27T12:17:24.106 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.092+0000 7fac71734700 1 -- 172.21.15.195:0/1337343183 shutdown_connections 2022-06-27T12:17:24.106 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.092+0000 7fac71734700 1 --2- 172.21.15.195:0/1337343183 >> v2:172.21.15.202:3300/0 conn(0x7fac6c067ed0 0x7fac6c14cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:24.106 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.092+0000 7fac71734700 1 --2- 172.21.15.195:0/1337343183 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fac6c0c6190 0x7fac6c157010 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:24.106 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.092+0000 7fac71734700 1 -- 172.21.15.195:0/1337343183 >> 172.21.15.195:0/1337343183 conn(0x7fac6c0ccb80 msgr2=0x7fac6c0ccf80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:17:24.107 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.092+0000 7fac71734700 1 -- 172.21.15.195:0/1337343183 shutdown_connections 2022-06-27T12:17:24.107 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.092+0000 7fac71734700 1 -- 172.21.15.195:0/1337343183 wait complete. 2022-06-27T12:17:24.107 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.093+0000 7fac71734700 1 Processor -- start 2022-06-27T12:17:24.107 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.093+0000 7fac71734700 1 -- start start 2022-06-27T12:17:24.108 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.094+0000 7fac71734700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fac6c067ed0 0x7fac6c155770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:24.108 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.094+0000 7fac71734700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fac6c0c40d0 0x7fac6c1523c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:24.108 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.094+0000 7fac71734700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fac6c0c6190 0x7fac6c14ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:24.108 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.094+0000 7fac71734700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac6c167150 con 0x7fac6c067ed0 2022-06-27T12:17:24.110 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.094+0000 7fac71734700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac6c14d5c0 con 0x7fac6c0c40d0 2022-06-27T12:17:24.110 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.094+0000 7fac71734700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac6c167610 con 0x7fac6c0c6190 2022-06-27T12:17:24.110 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.094+0000 7fac6affd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fac6c067ed0 0x7fac6c155770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:24.111 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.094+0000 7fac6affd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fac6c067ed0 0x7fac6c155770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:52470/0 (socket says 172.21.15.195:52470) 2022-06-27T12:17:24.111 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.094+0000 7fac6affd700 1 -- 172.21.15.195:0/4124115570 learned_addr learned my addr 172.21.15.195:0/4124115570 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:17:24.112 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.094+0000 7fac6b7fe700 1 --2- 172.21.15.195:0/4124115570 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fac6c0c6190 0x7fac6c14ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:24.112 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.094+0000 7fac6a7fc700 1 --2- 172.21.15.195:0/4124115570 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fac6c0c40d0 0x7fac6c1523c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:24.113 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.094+0000 7fac6affd700 1 -- 172.21.15.195:0/4124115570 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fac6c0c6190 msgr2=0x7fac6c14ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:24.113 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.094+0000 7fac6affd700 1 --2- 172.21.15.195:0/4124115570 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fac6c0c6190 0x7fac6c14ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:24.114 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.094+0000 7fac6affd700 1 -- 172.21.15.195:0/4124115570 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fac6c0c40d0 msgr2=0x7fac6c1523c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:24.114 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.094+0000 7fac6affd700 1 --2- 172.21.15.195:0/4124115570 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fac6c0c40d0 0x7fac6c1523c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:24.115 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.094+0000 7fac6affd700 1 -- 172.21.15.195:0/4124115570 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac640092e0 con 0x7fac6c067ed0 2022-06-27T12:17:24.115 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.094+0000 7fac6b7fe700 1 --2- 172.21.15.195:0/4124115570 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fac6c0c6190 0x7fac6c14ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:17:24.116 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.094+0000 7fac6affd700 1 --2- 172.21.15.195:0/4124115570 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fac6c067ed0 0x7fac6c155770 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7fac5c00cae0 tx=0x7fac5c007be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:17:24.116 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.094+0000 7fac52ffd700 1 -- 172.21.15.195:0/4124115570 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac5c004c80 con 0x7fac6c067ed0 2022-06-27T12:17:24.116 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.094+0000 7fac52ffd700 1 -- 172.21.15.195:0/4124115570 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac5c004de0 con 0x7fac6c067ed0 2022-06-27T12:17:24.117 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.094+0000 7fac71734700 1 -- 172.21.15.195:0/4124115570 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac600092e0 con 0x7fac6c067ed0 2022-06-27T12:17:24.117 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.094+0000 7fac52ffd700 1 -- 172.21.15.195:0/4124115570 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac5c0073a0 con 0x7fac6c067ed0 2022-06-27T12:17:24.117 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.094+0000 7fac71734700 1 -- 172.21.15.195:0/4124115570 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac6c14da50 con 0x7fac6c067ed0 2022-06-27T12:17:24.118 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.095+0000 7fac71734700 1 -- 172.21.15.195:0/4124115570 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac6c14d8a0 con 0x7fac6c067ed0 2022-06-27T12:17:24.118 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.096+0000 7fac52ffd700 1 -- 172.21.15.195:0/4124115570 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fac5c01a030 con 0x7fac6c067ed0 2022-06-27T12:17:24.118 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.096+0000 7fac52ffd700 1 --2- 172.21.15.195:0/4124115570 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fac4c063d00 0x7fac4c0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:24.118 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.096+0000 7fac52ffd700 1 -- 172.21.15.195:0/4124115570 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7664+0+0 (secure 0 0 0) 0x7fac5c080790 con 0x7fac6c067ed0 2022-06-27T12:17:24.119 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.096+0000 7fac6a7fc700 1 --2- 172.21.15.195:0/4124115570 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fac4c063d00 0x7fac4c0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:24.119 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.097+0000 7fac6a7fc700 1 --2- 172.21.15.195:0/4124115570 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fac4c063d00 0x7fac4c0661b0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fac6c0ca2b0 tx=0x7fac60015000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:17:24.120 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.100+0000 7fac52ffd700 1 -- 172.21.15.195:0/4124115570 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fac5c04edc0 con 0x7fac6c067ed0 2022-06-27T12:17:24.329 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.329+0000 7fac71734700 1 -- 172.21.15.195:0/4124115570 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac6c0722c0 con 0x7fac6c067ed0 2022-06-27T12:17:24.329 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.329+0000 7fac52ffd700 1 -- 172.21.15.195:0/4124115570 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) v1 ==== 95+0+5377 (secure 0 0 0) 0x7fac5c018070 con 0x7fac6c067ed0 2022-06-27T12:17:24.330 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:17:24.331 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":111,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:17:22.357174+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7165},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7165":{"gid":7165,"name":"d","rank":0,"incarnation":108,"state":"up:clientreplay","state_seq":24,"addr":"172.21.15.202:6835/3551478822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":3551478822},{"type":"v1","addr":"172.21.15.202:6835","nonce":3551478822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:17:24.335 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.331+0000 7fac71734700 1 -- 172.21.15.195:0/4124115570 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fac4c063d00 msgr2=0x7fac4c0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:24.336 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.331+0000 7fac71734700 1 --2- 172.21.15.195:0/4124115570 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fac4c063d00 0x7fac4c0661b0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fac6c0ca2b0 tx=0x7fac60015000 comp rx=0 tx=0).stop 2022-06-27T12:17:24.336 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.331+0000 7fac71734700 1 -- 172.21.15.195:0/4124115570 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fac6c067ed0 msgr2=0x7fac6c155770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:24.337 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.331+0000 7fac71734700 1 --2- 172.21.15.195:0/4124115570 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fac6c067ed0 0x7fac6c155770 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7fac5c00cae0 tx=0x7fac5c007be0 comp rx=0 tx=0).stop 2022-06-27T12:17:24.337 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.331+0000 7fac71734700 1 -- 172.21.15.195:0/4124115570 shutdown_connections 2022-06-27T12:17:24.338 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.331+0000 7fac71734700 1 --2- 172.21.15.195:0/4124115570 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fac4c063d00 0x7fac4c0661b0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:24.338 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.331+0000 7fac71734700 1 --2- 172.21.15.195:0/4124115570 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fac6c067ed0 0x7fac6c155770 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:24.339 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.331+0000 7fac71734700 1 --2- 172.21.15.195:0/4124115570 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fac6c0c6190 0x7fac6c14ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:24.339 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.331+0000 7fac71734700 1 --2- 172.21.15.195:0/4124115570 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fac6c0c40d0 0x7fac6c1523c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:24.339 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.331+0000 7fac71734700 1 -- 172.21.15.195:0/4124115570 >> 172.21.15.195:0/4124115570 conn(0x7fac6c0ccb80 msgr2=0x7fac6c0cc190 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:17:24.340 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.331+0000 7fac71734700 1 -- 172.21.15.195:0/4124115570 shutdown_connections 2022-06-27T12:17:24.340 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:24.331+0000 7fac71734700 1 -- 172.21.15.195:0/4124115570 wait complete. 2022-06-27T12:17:24.340 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 111 2022-06-27T12:17:24.345 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:17:26.350 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:17:26.496 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.485+0000 7f91e2a93700 1 Processor -- start 2022-06-27T12:17:26.497 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.485+0000 7f91e2a93700 1 -- start start 2022-06-27T12:17:26.497 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.485+0000 7f91e2a93700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f91dc14e490 0x7f91dc06cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:26.497 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.485+0000 7f91e2a93700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91dc0c33b0 con 0x7f91dc0c3f50 2022-06-27T12:17:26.498 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.485+0000 7f91e2a93700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91dc06d520 con 0x7f91dc14e860 2022-06-27T12:17:26.498 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.485+0000 7f91e2a93700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91dc06f220 con 0x7f91dc14e490 2022-06-27T12:17:26.499 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.486+0000 7f91e1030700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f91dc0c3f50 0x7f91dc157890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:58666/0 (socket says 172.21.15.195:58666) 2022-06-27T12:17:26.499 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.486+0000 7f91e1030700 1 -- 172.21.15.195:0/3274853384 learned_addr learned my addr 172.21.15.195:0/3274853384 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:17:26.500 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.486+0000 7f91e082f700 1 --2- 172.21.15.195:0/3274853384 >> v2:172.21.15.202:3300/0 conn(0x7f91dc14e490 0x7f91dc06cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:26.500 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.486+0000 7f91db7fe700 1 -- 172.21.15.195:0/3274853384 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2216074476 0 0) 0x7f91dc0c33b0 con 0x7f91dc0c3f50 2022-06-27T12:17:26.500 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.486+0000 7f91db7fe700 1 -- 172.21.15.195:0/3274853384 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91c0003430 con 0x7f91dc0c3f50 2022-06-27T12:17:26.536 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.486+0000 7f91e082f700 1 -- 172.21.15.195:0/3274853384 >> v1:172.21.15.202:6789/0 conn(0x7f91dc14e860 legacy=0x7f91dc0c3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:26.537 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.486+0000 7f91e082f700 1 -- 172.21.15.195:0/3274853384 >> v1:172.21.15.195:6789/0 conn(0x7f91dc0c3f50 legacy=0x7f91dc157890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:26.537 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.486+0000 7f91db7fe700 1 -- 172.21.15.195:0/3274853384 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2566842419 0 0) 0x7f91c0003430 con 0x7f91dc0c3f50 2022-06-27T12:17:26.537 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.486+0000 7f91e082f700 1 -- 172.21.15.195:0/3274853384 --> v2:172.21.15.202:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91dc1542c0 con 0x7f91dc14e490 2022-06-27T12:17:26.538 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.486+0000 7f91e082f700 1 --2- 172.21.15.195:0/3274853384 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f91dc14e490 0x7f91dc06cfe0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f91d00097a0 tx=0x7f91d0009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=71de4c6dab5fdd4 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:17:26.538 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.486+0000 7f91db7fe700 1 -- 172.21.15.195:0/3274853384 <== mon.1 v2:172.21.15.202:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d40017f0 con 0x7f91dc14e490 2022-06-27T12:17:26.538 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.486+0000 7f91db7fe700 1 -- 172.21.15.195:0/3274853384 <== mon.1 v2:172.21.15.202:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91d0004030 con 0x7f91dc14e490 2022-06-27T12:17:26.539 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.486+0000 7f91db7fe700 1 -- 172.21.15.195:0/3274853384 <== mon.1 v2:172.21.15.202:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d00043b0 con 0x7f91dc14e490 2022-06-27T12:17:26.539 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.487+0000 7f91e2a93700 1 -- 172.21.15.195:0/3274853384 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f91dc14e490 msgr2=0x7f91dc06cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:26.540 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.487+0000 7f91e2a93700 1 --2- 172.21.15.195:0/3274853384 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f91dc14e490 0x7f91dc06cfe0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f91d00097a0 tx=0x7f91d0009060 comp rx=0 tx=0).stop 2022-06-27T12:17:26.540 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.487+0000 7f91e2a93700 1 -- 172.21.15.195:0/3274853384 shutdown_connections 2022-06-27T12:17:26.541 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.487+0000 7f91e2a93700 1 --2- 172.21.15.195:0/3274853384 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f91dc14e490 0x7f91dc06cfe0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:26.541 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.487+0000 7f91e2a93700 1 -- 172.21.15.195:0/3274853384 >> 172.21.15.195:0/3274853384 conn(0x7f91dc063af0 msgr2=0x7f91dc063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:17:26.542 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.487+0000 7f91e2a93700 1 -- 172.21.15.195:0/3274853384 shutdown_connections 2022-06-27T12:17:26.542 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.487+0000 7f91e2a93700 1 -- 172.21.15.195:0/3274853384 wait complete. 2022-06-27T12:17:26.543 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.488+0000 7f91e2a93700 1 Processor -- start 2022-06-27T12:17:26.543 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.488+0000 7f91e2a93700 1 -- start start 2022-06-27T12:17:26.545 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.489+0000 7f91e2a93700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f91dc0c3f50 0x7f91dc149570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:26.546 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.489+0000 7f91e2a93700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f91dc14e490 0x7f91dc1579a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:26.546 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.489+0000 7f91e2a93700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f91dc14e860 0x7f91dc157ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:26.547 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.489+0000 7f91e2a93700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91dc166930 con 0x7f91dc0c3f50 2022-06-27T12:17:26.547 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.489+0000 7f91e2a93700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91dc169fc0 con 0x7f91dc14e490 2022-06-27T12:17:26.548 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.489+0000 7f91e2a93700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91dc16a2a0 con 0x7f91dc14e860 2022-06-27T12:17:26.548 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.489+0000 7f91e082f700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f91dc0c3f50 0x7f91dc149570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:26.549 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.489+0000 7f91e082f700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f91dc0c3f50 0x7f91dc149570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:52484/0 (socket says 172.21.15.195:52484) 2022-06-27T12:17:26.549 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.489+0000 7f91e082f700 1 -- 172.21.15.195:0/2725435324 learned_addr learned my addr 172.21.15.195:0/2725435324 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:17:26.550 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.489+0000 7f91e1030700 1 --2- 172.21.15.195:0/2725435324 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f91dc14e860 0x7f91dc157ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:26.550 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.489+0000 7f91dbfff700 1 --2- 172.21.15.195:0/2725435324 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f91dc14e490 0x7f91dc1579a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:26.551 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.490+0000 7f91e082f700 1 -- 172.21.15.195:0/2725435324 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f91dc14e860 msgr2=0x7f91dc157ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:26.551 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.490+0000 7f91e082f700 1 --2- 172.21.15.195:0/2725435324 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f91dc14e860 0x7f91dc157ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:26.551 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.490+0000 7f91e082f700 1 -- 172.21.15.195:0/2725435324 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f91dc14e490 msgr2=0x7f91dc1579a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:26.552 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.490+0000 7f91e082f700 1 --2- 172.21.15.195:0/2725435324 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f91dc14e490 0x7f91dc1579a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:26.552 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.490+0000 7f91e082f700 1 -- 172.21.15.195:0/2725435324 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91d00092e0 con 0x7f91dc0c3f50 2022-06-27T12:17:26.553 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.490+0000 7f91e082f700 1 --2- 172.21.15.195:0/2725435324 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f91dc0c3f50 0x7f91dc149570 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f91dc06e4b0 tx=0x7f91d0015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:17:26.553 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.490+0000 7f91d8ff9700 1 -- 172.21.15.195:0/2725435324 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d0011650 con 0x7f91dc0c3f50 2022-06-27T12:17:26.554 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.490+0000 7f91d8ff9700 1 -- 172.21.15.195:0/2725435324 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91d00117b0 con 0x7f91dc0c3f50 2022-06-27T12:17:26.554 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.490+0000 7f91e2a93700 1 -- 172.21.15.195:0/2725435324 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91dc16a580 con 0x7f91dc0c3f50 2022-06-27T12:17:26.555 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.490+0000 7f91d8ff9700 1 -- 172.21.15.195:0/2725435324 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d0011b30 con 0x7f91dc0c3f50 2022-06-27T12:17:26.556 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.490+0000 7f91e2a93700 1 -- 172.21.15.195:0/2725435324 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91dc16ab40 con 0x7f91dc0c3f50 2022-06-27T12:17:26.556 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.491+0000 7f91e2a93700 1 -- 172.21.15.195:0/2725435324 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91dc0722c0 con 0x7f91dc0c3f50 2022-06-27T12:17:26.557 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.491+0000 7f91d8ff9700 1 -- 172.21.15.195:0/2725435324 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f91d000f070 con 0x7f91dc0c3f50 2022-06-27T12:17:26.557 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.492+0000 7f91d8ff9700 1 --2- 172.21.15.195:0/2725435324 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f91c4063d00 0x7f91c40661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:26.557 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.492+0000 7f91d8ff9700 1 -- 172.21.15.195:0/2725435324 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7664+0+0 (secure 0 0 0) 0x7f91d0082d40 con 0x7f91dc0c3f50 2022-06-27T12:17:26.558 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.492+0000 7f91dbfff700 1 --2- 172.21.15.195:0/2725435324 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f91c4063d00 0x7f91c40661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:26.558 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.492+0000 7f91dbfff700 1 --2- 172.21.15.195:0/2725435324 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f91c4063d00 0x7f91c40661b0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f91dc154290 tx=0x7f91c8009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:17:26.559 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.496+0000 7f91d8ff9700 1 -- 172.21.15.195:0/2725435324 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f91d0086030 con 0x7f91dc0c3f50 2022-06-27T12:17:26.726 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.724+0000 7f91e2a93700 1 -- 172.21.15.195:0/2725435324 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f91dc0c98f0 con 0x7f91dc0c3f50 2022-06-27T12:17:26.727 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.725+0000 7f91d8ff9700 1 -- 172.21.15.195:0/2725435324 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) v1 ==== 95+0+5377 (secure 0 0 0) 0x7f91d002b090 con 0x7f91dc0c3f50 2022-06-27T12:17:26.727 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:17:26.728 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":111,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:17:22.357174+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7165},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7165":{"gid":7165,"name":"d","rank":0,"incarnation":108,"state":"up:clientreplay","state_seq":24,"addr":"172.21.15.202:6835/3551478822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":3551478822},{"type":"v1","addr":"172.21.15.202:6835","nonce":3551478822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:17:26.745 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.727+0000 7f91e2a93700 1 -- 172.21.15.195:0/2725435324 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f91c4063d00 msgr2=0x7f91c40661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:26.745 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.727+0000 7f91e2a93700 1 --2- 172.21.15.195:0/2725435324 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f91c4063d00 0x7f91c40661b0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f91dc154290 tx=0x7f91c8009040 comp rx=0 tx=0).stop 2022-06-27T12:17:26.745 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.727+0000 7f91e2a93700 1 -- 172.21.15.195:0/2725435324 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f91dc0c3f50 msgr2=0x7f91dc149570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:26.746 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.727+0000 7f91e2a93700 1 --2- 172.21.15.195:0/2725435324 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f91dc0c3f50 0x7f91dc149570 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f91dc06e4b0 tx=0x7f91d0015000 comp rx=0 tx=0).stop 2022-06-27T12:17:26.746 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.727+0000 7f91e2a93700 1 -- 172.21.15.195:0/2725435324 shutdown_connections 2022-06-27T12:17:26.746 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.727+0000 7f91e2a93700 1 --2- 172.21.15.195:0/2725435324 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f91c4063d00 0x7f91c40661b0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:26.746 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.727+0000 7f91e2a93700 1 --2- 172.21.15.195:0/2725435324 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f91dc0c3f50 0x7f91dc149570 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:26.747 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.727+0000 7f91e2a93700 1 --2- 172.21.15.195:0/2725435324 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f91dc14e860 0x7f91dc157ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:26.747 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.727+0000 7f91e2a93700 1 --2- 172.21.15.195:0/2725435324 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f91dc14e490 0x7f91dc1579a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:26.747 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.727+0000 7f91e2a93700 1 -- 172.21.15.195:0/2725435324 >> 172.21.15.195:0/2725435324 conn(0x7f91dc063af0 msgr2=0x7f91dc0c9070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:17:26.747 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.727+0000 7f91e2a93700 1 -- 172.21.15.195:0/2725435324 shutdown_connections 2022-06-27T12:17:26.748 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:26.727+0000 7f91e2a93700 1 -- 172.21.15.195:0/2725435324 wait complete. 2022-06-27T12:17:26.748 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 111 2022-06-27T12:17:26.749 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 111, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-06-27T11:47:52.358095+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 111, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7165": { "addr": "172.21.15.202:6835/3551478822", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6834", "nonce": 3551478822, "type": "v2" }, { "addr": "172.21.15.202:6835", "nonce": 3551478822, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7165, "incarnation": 108, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:clientreplay", "state_seq": 24 } }, "last_failure": 0, "last_failure_osd_epoch": 65, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-06-27T12:17:22.357174+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7165 } } } ], "standbys": [ { "addr": "172.21.15.202:6837/3305006841", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6836", "nonce": 3305006841, "type": "v2" }, { "addr": "172.21.15.202:6837", "nonce": 3305006841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6835/2060709822", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6834", "nonce": 2060709822, "type": "v2" }, { "addr": "172.21.15.195:6835", "nonce": 2060709822, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6838/1598751652", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6836", "nonce": 1598751652, "type": "v2" }, { "addr": "172.21.15.195:6838", "nonce": 1598751652, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4275, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6839/1313800661", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6837", "nonce": 1313800661, "type": "v2" }, { "addr": "172.21.15.195:6839", "nonce": 1313800661, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4278, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-06-27T12:17:28.751 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:17:28.901 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:28.873+0000 7f2b489ec700 1 Processor -- start 2022-06-27T12:17:28.901 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:28.873+0000 7f2b489ec700 1 -- start start 2022-06-27T12:17:28.904 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:28.874+0000 7f2b489ec700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f2b4406e850 0x7f2b441570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:28.905 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:28.874+0000 7f2b489ec700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f2b440c34a0 0x7f2b440c2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:28.905 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:28.874+0000 7f2b489ec700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b4406fba0 con 0x7f2b4406fef0 2022-06-27T12:17:28.906 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:28.874+0000 7f2b489ec700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b441575e0 con 0x7f2b4406e850 2022-06-27T12:17:28.906 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:28.874+0000 7f2b489ec700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b4406f5e0 con 0x7f2b440c34a0 2022-06-27T12:17:28.907 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:28.874+0000 7f2b4259c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f2b4406e850 0x7f2b441570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:28.907 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:28.874+0000 7f2b41d9b700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f2b4406fef0 0x7f2b44069de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:58694/0 (socket says 172.21.15.195:58694) 2022-06-27T12:17:28.908 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:28.874+0000 7f2b41d9b700 1 -- 172.21.15.195:0/1027064017 learned_addr learned my addr 172.21.15.195:0/1027064017 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:17:28.908 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:28.874+0000 7f2b42d9d700 1 --2- 172.21.15.195:0/1027064017 >> v2:172.21.15.202:3300/0 conn(0x7f2b440c34a0 0x7f2b440c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:28.909 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:28.874+0000 7f2b4259c700 1 -- 172.21.15.195:0/1027064017 >> v2:172.21.15.202:3300/0 conn(0x7f2b440c34a0 msgr2=0x7f2b440c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:17:28.909 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:28.874+0000 7f2b4259c700 1 --2- 172.21.15.195:0/1027064017 >> v2:172.21.15.202:3300/0 conn(0x7f2b440c34a0 0x7f2b440c2920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:28.909 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:28.874+0000 7f2b4259c700 1 -- 172.21.15.195:0/1027064017 >> v1:172.21.15.195:6789/0 conn(0x7f2b4406fef0 legacy=0x7f2b44069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:28.911 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:28.874+0000 7f2b4159a700 1 -- 172.21.15.195:0/1027064017 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4248693323 0 0) 0x7f2b4406fba0 con 0x7f2b4406fef0 2022-06-27T12:17:28.911 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:28.874+0000 7f2b4259c700 1 -- 172.21.15.195:0/1027064017 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b4406f8c0 con 0x7f2b4406e850 2022-06-27T12:17:28.912 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:28.874+0000 7f2b4259c700 1 --2- 172.21.15.195:0/1027064017 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2b4406e850 0x7f2b441570a0 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f2b440c1420 tx=0x7f2b38009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d321188cf724a8d0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:17:28.912 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:28.875+0000 7f2b4159a700 1 -- 172.21.15.195:0/1027064017 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b38011630 con 0x7f2b4406e850 2022-06-27T12:17:28.913 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:28.875+0000 7f2b4159a700 1 -- 172.21.15.195:0/1027064017 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b38011790 con 0x7f2b4406e850 2022-06-27T12:17:28.913 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:28.875+0000 7f2b4159a700 1 -- 172.21.15.195:0/1027064017 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b38011b30 con 0x7f2b4406e850 2022-06-27T12:17:28.913 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:28.875+0000 7f2b489ec700 1 -- 172.21.15.195:0/1027064017 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2b4406e850 msgr2=0x7f2b441570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:28.914 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:28.875+0000 7f2b489ec700 1 --2- 172.21.15.195:0/1027064017 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2b4406e850 0x7f2b441570a0 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f2b440c1420 tx=0x7f2b38009060 comp rx=0 tx=0).stop 2022-06-27T12:17:28.914 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:28.875+0000 7f2b489ec700 1 -- 172.21.15.195:0/1027064017 shutdown_connections 2022-06-27T12:17:28.915 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:28.875+0000 7f2b489ec700 1 --2- 172.21.15.195:0/1027064017 >> v2:172.21.15.202:3300/0 conn(0x7f2b440c34a0 0x7f2b440c2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:28.915 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:28.875+0000 7f2b489ec700 1 --2- 172.21.15.195:0/1027064017 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2b4406e850 0x7f2b441570a0 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:28.916 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:28.875+0000 7f2b489ec700 1 -- 172.21.15.195:0/1027064017 >> 172.21.15.195:0/1027064017 conn(0x7f2b440c6350 msgr2=0x7f2b440c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:17:28.916 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:28.875+0000 7f2b489ec700 1 -- 172.21.15.195:0/1027064017 shutdown_connections 2022-06-27T12:17:28.917 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:28.875+0000 7f2b489ec700 1 -- 172.21.15.195:0/1027064017 wait complete. 2022-06-27T12:17:28.918 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:28.876+0000 7f2b489ec700 1 Processor -- start 2022-06-27T12:17:28.918 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:28.876+0000 7f2b489ec700 1 -- start start 2022-06-27T12:17:28.919 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:28.877+0000 7f2b489ec700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2b4406e850 0x7f2b44166870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:28.920 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:28.877+0000 7f2b489ec700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2b440c34a0 0x7f2b44169ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:28.920 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:28.877+0000 7f2b489ec700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2b441567b0 0x7f2b440c6720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:28.921 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:28.877+0000 7f2b489ec700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b441480c0 con 0x7f2b4406e850 2022-06-27T12:17:28.922 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:28.877+0000 7f2b489ec700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b44166db0 con 0x7f2b441567b0 2022-06-27T12:17:28.922 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:28.877+0000 7f2b489ec700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b44169810 con 0x7f2b440c34a0 2022-06-27T12:17:28.923 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:28.877+0000 7f2b4259c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2b4406e850 0x7f2b44166870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:28.923 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:28.877+0000 7f2b4259c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2b4406e850 0x7f2b44166870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:52510/0 (socket says 172.21.15.195:52510) 2022-06-27T12:17:28.924 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:28.877+0000 7f2b4259c700 1 -- 172.21.15.195:0/4180244762 learned_addr learned my addr 172.21.15.195:0/4180244762 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:17:28.938 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:28.877+0000 7f2b41d9b700 1 --2- 172.21.15.195:0/4180244762 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2b440c34a0 0x7f2b44169ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:28.939 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:28.877+0000 7f2b42d9d700 1 --2- 172.21.15.195:0/4180244762 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2b441567b0 0x7f2b440c6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:28.939 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:28.878+0000 7f2b4259c700 1 -- 172.21.15.195:0/4180244762 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2b440c34a0 msgr2=0x7f2b44169ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:28.940 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:28.878+0000 7f2b4259c700 1 --2- 172.21.15.195:0/4180244762 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2b440c34a0 0x7f2b44169ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:28.940 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:28.878+0000 7f2b4259c700 1 -- 172.21.15.195:0/4180244762 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2b441567b0 msgr2=0x7f2b440c6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:28.940 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:28.878+0000 7f2b4259c700 1 --2- 172.21.15.195:0/4180244762 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2b441567b0 0x7f2b440c6720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:28.941 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:28.878+0000 7f2b4259c700 1 -- 172.21.15.195:0/4180244762 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b380092e0 con 0x7f2b4406e850 2022-06-27T12:17:28.941 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:28.878+0000 7f2b4259c700 1 --2- 172.21.15.195:0/4180244762 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2b4406e850 0x7f2b44166870 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f2b38011c30 tx=0x7f2b38017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:17:28.942 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:28.878+0000 7f2b327fc700 1 -- 172.21.15.195:0/4180244762 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b38003730 con 0x7f2b4406e850 2022-06-27T12:17:28.943 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:28.878+0000 7f2b327fc700 1 -- 172.21.15.195:0/4180244762 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b380038b0 con 0x7f2b4406e850 2022-06-27T12:17:28.943 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:28.878+0000 7f2b489ec700 1 -- 172.21.15.195:0/4180244762 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b4414d0f0 con 0x7f2b4406e850 2022-06-27T12:17:28.944 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:28.878+0000 7f2b327fc700 1 -- 172.21.15.195:0/4180244762 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b38003c70 con 0x7f2b4406e850 2022-06-27T12:17:28.944 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:28.878+0000 7f2b489ec700 1 -- 172.21.15.195:0/4180244762 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b441694e0 con 0x7f2b4406e850 2022-06-27T12:17:28.945 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:28.879+0000 7f2b327fc700 1 -- 172.21.15.195:0/4180244762 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f2b3800f050 con 0x7f2b4406e850 2022-06-27T12:17:28.945 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:28.879+0000 7f2b489ec700 1 -- 172.21.15.195:0/4180244762 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b440722c0 con 0x7f2b4406e850 2022-06-27T12:17:28.953 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:28.880+0000 7f2b327fc700 1 --2- 172.21.15.195:0/4180244762 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2b20063d00 0x7f2b200661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:28.954 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:28.880+0000 7f2b327fc700 1 -- 172.21.15.195:0/4180244762 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7664+0+0 (secure 0 0 0) 0x7f2b38082460 con 0x7f2b4406e850 2022-06-27T12:17:28.954 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:28.880+0000 7f2b41d9b700 1 --2- 172.21.15.195:0/4180244762 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2b20063d00 0x7f2b200661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:28.955 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:28.880+0000 7f2b41d9b700 1 --2- 172.21.15.195:0/4180244762 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2b20063d00 0x7f2b200661b0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f2b440c0000 tx=0x7f2b34007d10 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:17:28.955 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:28.884+0000 7f2b327fc700 1 -- 172.21.15.195:0/4180244762 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f2b380509b0 con 0x7f2b4406e850 2022-06-27T12:17:29.121 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:29.121+0000 7f2b489ec700 1 -- 172.21.15.195:0/4180244762 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2b44169af0 con 0x7f2b4406e850 2022-06-27T12:17:29.122 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:29.121+0000 7f2b327fc700 1 -- 172.21.15.195:0/4180244762 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) v1 ==== 95+0+5377 (secure 0 0 0) 0x7f2b38053fd0 con 0x7f2b4406e850 2022-06-27T12:17:29.123 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:17:29.123 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":111,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:17:22.357174+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7165},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7165":{"gid":7165,"name":"d","rank":0,"incarnation":108,"state":"up:clientreplay","state_seq":24,"addr":"172.21.15.202:6835/3551478822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":3551478822},{"type":"v1","addr":"172.21.15.202:6835","nonce":3551478822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:17:29.127 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:29.123+0000 7f2b489ec700 1 -- 172.21.15.195:0/4180244762 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2b20063d00 msgr2=0x7f2b200661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:29.128 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:29.123+0000 7f2b489ec700 1 --2- 172.21.15.195:0/4180244762 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2b20063d00 0x7f2b200661b0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f2b440c0000 tx=0x7f2b34007d10 comp rx=0 tx=0).stop 2022-06-27T12:17:29.128 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:29.123+0000 7f2b489ec700 1 -- 172.21.15.195:0/4180244762 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2b4406e850 msgr2=0x7f2b44166870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:29.128 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:29.123+0000 7f2b489ec700 1 --2- 172.21.15.195:0/4180244762 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2b4406e850 0x7f2b44166870 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f2b38011c30 tx=0x7f2b38017000 comp rx=0 tx=0).stop 2022-06-27T12:17:29.129 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:29.123+0000 7f2b489ec700 1 -- 172.21.15.195:0/4180244762 shutdown_connections 2022-06-27T12:17:29.129 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:29.123+0000 7f2b489ec700 1 --2- 172.21.15.195:0/4180244762 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f2b20063d00 0x7f2b200661b0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:29.130 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:29.123+0000 7f2b489ec700 1 --2- 172.21.15.195:0/4180244762 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2b4406e850 0x7f2b44166870 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:29.131 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:29.123+0000 7f2b489ec700 1 --2- 172.21.15.195:0/4180244762 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f2b441567b0 0x7f2b440c6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:29.131 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:29.124+0000 7f2b489ec700 1 --2- 172.21.15.195:0/4180244762 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f2b440c34a0 0x7f2b44169ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:29.134 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:29.124+0000 7f2b489ec700 1 -- 172.21.15.195:0/4180244762 >> 172.21.15.195:0/4180244762 conn(0x7f2b440c6350 msgr2=0x7f2b440c26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:17:29.134 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:29.124+0000 7f2b489ec700 1 -- 172.21.15.195:0/4180244762 shutdown_connections 2022-06-27T12:17:29.135 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:29.124+0000 7f2b489ec700 1 -- 172.21.15.195:0/4180244762 wait complete. 2022-06-27T12:17:29.135 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 111 2022-06-27T12:17:29.145 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:17:31.153 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:17:31.308 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.296+0000 7f3c8f2bf700 1 Processor -- start 2022-06-27T12:17:31.309 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.297+0000 7f3c8f2bf700 1 -- start start 2022-06-27T12:17:31.309 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.297+0000 7f3c8f2bf700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f3c881509c0 0x7f3c880c7440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:31.310 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.297+0000 7f3c8f2bf700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c880c43d0 con 0x7f3c88156670 2022-06-27T12:17:31.310 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.297+0000 7f3c8f2bf700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c88068430 con 0x7f3c88067ed0 2022-06-27T12:17:31.311 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.297+0000 7f3c8f2bf700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c88068710 con 0x7f3c881509c0 2022-06-27T12:17:31.311 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.298+0000 7f3c8d85c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f3c881509c0 0x7f3c880c7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:31.312 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.298+0000 7f3c8d05b700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f3c88156670 0x7f3c880c4860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:58712/0 (socket says 172.21.15.195:58712) 2022-06-27T12:17:31.312 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.298+0000 7f3c8d05b700 1 -- 172.21.15.195:0/3417522952 learned_addr learned my addr 172.21.15.195:0/3417522952 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:17:31.313 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.298+0000 7f3c8d85c700 1 -- 172.21.15.195:0/3417522952 >> v1:172.21.15.202:6789/0 conn(0x7f3c88067ed0 legacy=0x7f3c881502b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:31.313 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.298+0000 7f3c8d85c700 1 -- 172.21.15.195:0/3417522952 >> v1:172.21.15.195:6789/0 conn(0x7f3c88156670 legacy=0x7f3c880c4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:31.314 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.298+0000 7f3c8d85c700 1 -- 172.21.15.195:0/3417522952 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c880689f0 con 0x7f3c881509c0 2022-06-27T12:17:31.314 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.298+0000 7f3c7bfff700 1 -- 172.21.15.195:0/3417522952 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4073559548 0 0) 0x7f3c880c43d0 con 0x7f3c88156670 2022-06-27T12:17:31.315 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.298+0000 7f3c8d85c700 1 --2- 172.21.15.195:0/3417522952 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3c881509c0 0x7f3c880c7440 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f3c80009e60 tx=0x7f3c80009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47b9794724adb3b1 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:17:31.315 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.298+0000 7f3c7bfff700 1 -- 172.21.15.195:0/3417522952 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c8000e070 con 0x7f3c881509c0 2022-06-27T12:17:31.316 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.298+0000 7f3c7bfff700 1 -- 172.21.15.195:0/3417522952 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c8000bc30 con 0x7f3c881509c0 2022-06-27T12:17:31.317 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.298+0000 7f3c7bfff700 1 -- 172.21.15.195:0/3417522952 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c80012430 con 0x7f3c881509c0 2022-06-27T12:17:31.317 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.299+0000 7f3c8f2bf700 1 -- 172.21.15.195:0/3417522952 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3c881509c0 msgr2=0x7f3c880c7440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:31.318 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.299+0000 7f3c8f2bf700 1 --2- 172.21.15.195:0/3417522952 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3c881509c0 0x7f3c880c7440 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f3c80009e60 tx=0x7f3c80009780 comp rx=0 tx=0).stop 2022-06-27T12:17:31.319 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.299+0000 7f3c8f2bf700 1 -- 172.21.15.195:0/3417522952 shutdown_connections 2022-06-27T12:17:31.319 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.299+0000 7f3c8f2bf700 1 --2- 172.21.15.195:0/3417522952 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3c881509c0 0x7f3c880c7440 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:31.320 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.299+0000 7f3c8f2bf700 1 -- 172.21.15.195:0/3417522952 >> 172.21.15.195:0/3417522952 conn(0x7f3c88069e00 msgr2=0x7f3c8806a200 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:17:31.320 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.299+0000 7f3c8f2bf700 1 -- 172.21.15.195:0/3417522952 shutdown_connections 2022-06-27T12:17:31.320 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.299+0000 7f3c8f2bf700 1 -- 172.21.15.195:0/3417522952 wait complete. 2022-06-27T12:17:31.321 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.300+0000 7f3c8f2bf700 1 Processor -- start 2022-06-27T12:17:31.321 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.300+0000 7f3c8f2bf700 1 -- start start 2022-06-27T12:17:31.322 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.301+0000 7f3c8f2bf700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3c88067ed0 0x7f3c880c3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:31.322 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.301+0000 7f3c8f2bf700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3c881509c0 0x7f3c88167730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:31.323 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.301+0000 7f3c8f2bf700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3c88156670 0x7f3c88167c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:31.323 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.301+0000 7f3c8f2bf700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c880a8650 con 0x7f3c88067ed0 2022-06-27T12:17:31.323 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.301+0000 7f3c8f2bf700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c88152910 con 0x7f3c88156670 2022-06-27T12:17:31.324 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.301+0000 7f3c8f2bf700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c88153720 con 0x7f3c881509c0 2022-06-27T12:17:31.325 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.301+0000 7f3c8d05b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3c88067ed0 0x7f3c880c3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:31.325 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.301+0000 7f3c8d05b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3c88067ed0 0x7f3c880c3600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:52530/0 (socket says 172.21.15.195:52530) 2022-06-27T12:17:31.326 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.301+0000 7f3c8d05b700 1 -- 172.21.15.195:0/2577733574 learned_addr learned my addr 172.21.15.195:0/2577733574 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:17:31.327 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.301+0000 7f3c8d85c700 1 --2- 172.21.15.195:0/2577733574 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3c88156670 0x7f3c88167c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:31.327 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.301+0000 7f3c8c85a700 1 --2- 172.21.15.195:0/2577733574 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3c881509c0 0x7f3c88167730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:31.328 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.301+0000 7f3c8d05b700 1 -- 172.21.15.195:0/2577733574 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3c881509c0 msgr2=0x7f3c88167730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:31.328 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.301+0000 7f3c8d05b700 1 --2- 172.21.15.195:0/2577733574 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3c881509c0 0x7f3c88167730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:31.328 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.301+0000 7f3c8d05b700 1 -- 172.21.15.195:0/2577733574 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3c88156670 msgr2=0x7f3c88167c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:31.329 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.301+0000 7f3c8d05b700 1 --2- 172.21.15.195:0/2577733574 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3c88156670 0x7f3c88167c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:31.329 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.301+0000 7f3c8d05b700 1 -- 172.21.15.195:0/2577733574 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c80009a20 con 0x7f3c88067ed0 2022-06-27T12:17:31.330 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.302+0000 7f3c8d05b700 1 --2- 172.21.15.195:0/2577733574 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3c88067ed0 0x7f3c880c3600 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c00c4c0 tx=0x7f3c7c007dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:17:31.330 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.302+0000 7f3c797fa700 1 -- 172.21.15.195:0/2577733574 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c7c007360 con 0x7f3c88067ed0 2022-06-27T12:17:31.331 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.302+0000 7f3c797fa700 1 -- 172.21.15.195:0/2577733574 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c7c0074c0 con 0x7f3c88067ed0 2022-06-27T12:17:31.331 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.302+0000 7f3c8f2bf700 1 -- 172.21.15.195:0/2577733574 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c88152b10 con 0x7f3c88067ed0 2022-06-27T12:17:31.332 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.302+0000 7f3c797fa700 1 -- 172.21.15.195:0/2577733574 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c7c004940 con 0x7f3c88067ed0 2022-06-27T12:17:31.332 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.302+0000 7f3c8f2bf700 1 -- 172.21.15.195:0/2577733574 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c881530d0 con 0x7f3c88067ed0 2022-06-27T12:17:31.333 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.303+0000 7f3c8f2bf700 1 -- 172.21.15.195:0/2577733574 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c880722c0 con 0x7f3c88067ed0 2022-06-27T12:17:31.333 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.303+0000 7f3c797fa700 1 -- 172.21.15.195:0/2577733574 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f3c7c010030 con 0x7f3c88067ed0 2022-06-27T12:17:31.333 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.304+0000 7f3c797fa700 1 --2- 172.21.15.195:0/2577733574 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3c68063d00 0x7f3c680661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:31.334 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.304+0000 7f3c797fa700 1 -- 172.21.15.195:0/2577733574 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7664+0+0 (secure 0 0 0) 0x7f3c7c0808f0 con 0x7f3c88067ed0 2022-06-27T12:17:31.334 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.304+0000 7f3c8c85a700 1 --2- 172.21.15.195:0/2577733574 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3c68063d00 0x7f3c680661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:31.335 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.304+0000 7f3c8c85a700 1 --2- 172.21.15.195:0/2577733574 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3c68063d00 0x7f3c680661b0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f3c8814f130 tx=0x7f3c84009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:17:31.336 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.308+0000 7f3c797fa700 1 -- 172.21.15.195:0/2577733574 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f3c7c04eec0 con 0x7f3c88067ed0 2022-06-27T12:17:31.535 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.530+0000 7f3c8f2bf700 1 -- 172.21.15.195:0/2577733574 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c88068cd0 con 0x7f3c88067ed0 2022-06-27T12:17:31.535 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.530+0000 7f3c797fa700 1 -- 172.21.15.195:0/2577733574 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+5371 (secure 0 0 0) 0x7f3c7c0524e0 con 0x7f3c88067ed0 2022-06-27T12:17:31.537 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:17:31.538 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":112,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:17:30.381992+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7165},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7165":{"gid":7165,"name":"d","rank":0,"incarnation":108,"state":"up:active","state_seq":27,"addr":"172.21.15.202:6835/3551478822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":3551478822},{"type":"v1","addr":"172.21.15.202:6835","nonce":3551478822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:17:31.548 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.532+0000 7f3c8f2bf700 1 -- 172.21.15.195:0/2577733574 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3c68063d00 msgr2=0x7f3c680661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:31.549 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.532+0000 7f3c8f2bf700 1 --2- 172.21.15.195:0/2577733574 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3c68063d00 0x7f3c680661b0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f3c8814f130 tx=0x7f3c84009040 comp rx=0 tx=0).stop 2022-06-27T12:17:31.549 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.532+0000 7f3c8f2bf700 1 -- 172.21.15.195:0/2577733574 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3c88067ed0 msgr2=0x7f3c880c3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:31.550 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.532+0000 7f3c8f2bf700 1 --2- 172.21.15.195:0/2577733574 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3c88067ed0 0x7f3c880c3600 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c00c4c0 tx=0x7f3c7c007dd0 comp rx=0 tx=0).stop 2022-06-27T12:17:31.550 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.532+0000 7f3c8f2bf700 1 -- 172.21.15.195:0/2577733574 shutdown_connections 2022-06-27T12:17:31.550 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.532+0000 7f3c8f2bf700 1 --2- 172.21.15.195:0/2577733574 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f3c68063d00 0x7f3c680661b0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:31.551 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.532+0000 7f3c8f2bf700 1 --2- 172.21.15.195:0/2577733574 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3c88067ed0 0x7f3c880c3600 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:31.551 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.532+0000 7f3c8f2bf700 1 --2- 172.21.15.195:0/2577733574 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f3c88156670 0x7f3c88167c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:31.551 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.532+0000 7f3c8f2bf700 1 --2- 172.21.15.195:0/2577733574 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f3c881509c0 0x7f3c88167730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:31.551 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.532+0000 7f3c8f2bf700 1 -- 172.21.15.195:0/2577733574 >> 172.21.15.195:0/2577733574 conn(0x7f3c88069e00 msgr2=0x7f3c8814f720 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:17:31.552 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.532+0000 7f3c8f2bf700 1 -- 172.21.15.195:0/2577733574 shutdown_connections 2022-06-27T12:17:31.552 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:31.532+0000 7f3c8f2bf700 1 -- 172.21.15.195:0/2577733574 wait complete. 2022-06-27T12:17:31.552 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 112 2022-06-27T12:17:31.553 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d has gained rank=0, replacing gid=6254 2022-06-27T12:17:31.553 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 8 secs before reviving mds.b 2022-06-27T12:17:35.458 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:17:35.462 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:17:36.439 INFO:tasks.ceph.mds.d.smithi202.stderr:2022-06-27T12:17:36.431+0000 7f6a5f912700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2022-06-27T12:17:39.554 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2022-06-27T12:17:39.555 INFO:tasks.ceph.mds.b:Restarting daemon 2022-06-27T12:17:39.556 DEBUG:teuthology.orchestra.run.smithi202:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-06-27T12:17:39.560 INFO:tasks.ceph.mds.b:Started 2022-06-27T12:17:39.560 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:17:39.636 INFO:tasks.ceph.mds.b.smithi202.stdout:starting mds.b at 2022-06-27T12:17:39.698 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.686+0000 7f1ced087700 1 Processor -- start 2022-06-27T12:17:39.698 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.687+0000 7f1ced087700 1 -- start start 2022-06-27T12:17:39.699 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.687+0000 7f1ced087700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1ce80688a0 0x7f1ce80c28c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:39.699 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.687+0000 7f1ced087700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ce80c4b10 con 0x7f1ce806a440 2022-06-27T12:17:39.700 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.687+0000 7f1ced087700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ce8068f20 con 0x7f1ce8156100 2022-06-27T12:17:39.700 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.687+0000 7f1ced087700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce80c3dc0 con 0x7f1ce80688a0 2022-06-27T12:17:39.700 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.687+0000 7f1ce659c700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f1ce806a440 0x7f1ce81564d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:58756/0 (socket says 172.21.15.195:58756) 2022-06-27T12:17:39.702 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.687+0000 7f1ce659c700 1 -- 172.21.15.195:0/1315444701 learned_addr learned my addr 172.21.15.195:0/1315444701 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:17:39.702 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.688+0000 7f1ce759e700 1 --2- 172.21.15.195:0/1315444701 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1ce80688a0 0x7f1ce80c28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:39.703 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.688+0000 7f1ce5d9b700 1 -- 172.21.15.195:0/1315444701 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2148725390 0 0) 0x7f1ce80c4b10 con 0x7f1ce806a440 2022-06-27T12:17:39.703 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.688+0000 7f1ce5d9b700 1 -- 172.21.15.195:0/1315444701 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cd4004540 con 0x7f1ce806a440 2022-06-27T12:17:39.703 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.688+0000 7f1ce5d9b700 1 -- 172.21.15.195:0/1315444701 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 663429161 0 0) 0x7f1ce8068f20 con 0x7f1ce8156100 2022-06-27T12:17:39.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.688+0000 7f1ce5d9b700 1 -- 172.21.15.195:0/1315444701 --> v1:172.21.15.202:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ce80c4b10 con 0x7f1ce8156100 2022-06-27T12:17:39.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.688+0000 7f1ce759e700 1 -- 172.21.15.195:0/1315444701 >> v1:172.21.15.202:6789/0 conn(0x7f1ce8156100 legacy=0x7f1ce80c4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:39.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.688+0000 7f1ce759e700 1 -- 172.21.15.195:0/1315444701 >> v1:172.21.15.195:6789/0 conn(0x7f1ce806a440 legacy=0x7f1ce81564d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:39.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.688+0000 7f1ce759e700 1 -- 172.21.15.195:0/1315444701 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ce80c56b0 con 0x7f1ce80688a0 2022-06-27T12:17:39.705 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.688+0000 7f1ce759e700 1 --2- 172.21.15.195:0/1315444701 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1ce80688a0 0x7f1ce80c28c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1ce0009e90 tx=0x7f1ce0009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93f46c545818f13c server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:17:39.705 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.688+0000 7f1ce5d9b700 1 -- 172.21.15.195:0/1315444701 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ce000e070 con 0x7f1ce80688a0 2022-06-27T12:17:39.705 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.688+0000 7f1ce5d9b700 1 -- 172.21.15.195:0/1315444701 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ce000bb80 con 0x7f1ce80688a0 2022-06-27T12:17:39.706 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.688+0000 7f1ce5d9b700 1 -- 172.21.15.195:0/1315444701 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ce0012430 con 0x7f1ce80688a0 2022-06-27T12:17:39.706 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.689+0000 7f1ced087700 1 -- 172.21.15.195:0/1315444701 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1ce80688a0 msgr2=0x7f1ce80c28c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:39.706 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.689+0000 7f1ced087700 1 --2- 172.21.15.195:0/1315444701 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1ce80688a0 0x7f1ce80c28c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1ce0009e90 tx=0x7f1ce0009780 comp rx=0 tx=0).stop 2022-06-27T12:17:39.706 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.689+0000 7f1ced087700 1 -- 172.21.15.195:0/1315444701 shutdown_connections 2022-06-27T12:17:39.707 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.689+0000 7f1ced087700 1 --2- 172.21.15.195:0/1315444701 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1ce80688a0 0x7f1ce80c28c0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:39.708 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.689+0000 7f1ced087700 1 -- 172.21.15.195:0/1315444701 >> 172.21.15.195:0/1315444701 conn(0x7f1ce80c9b00 msgr2=0x7f1ce80c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:17:39.708 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.689+0000 7f1ced087700 1 -- 172.21.15.195:0/1315444701 shutdown_connections 2022-06-27T12:17:39.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.689+0000 7f1ced087700 1 -- 172.21.15.195:0/1315444701 wait complete. 2022-06-27T12:17:39.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.690+0000 7f1ced087700 1 Processor -- start 2022-06-27T12:17:39.710 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.690+0000 7f1ced087700 1 -- start start 2022-06-27T12:17:39.710 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.691+0000 7f1ced087700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1ce80688a0 0x7f1ce8142b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:39.710 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.691+0000 7f1ced087700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1ce806a440 0x7f1ce806b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:39.710 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.691+0000 7f1ced087700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1ce8156100 0x7f1ce806bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:39.711 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.691+0000 7f1ced087700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce814c6e0 con 0x7f1ce80688a0 2022-06-27T12:17:39.711 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.691+0000 7f1ced087700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce80bbd50 con 0x7f1ce8156100 2022-06-27T12:17:39.711 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.691+0000 7f1ced087700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce8166870 con 0x7f1ce806a440 2022-06-27T12:17:39.711 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.691+0000 7f1ce6d9d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1ce80688a0 0x7f1ce8142b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:39.711 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.691+0000 7f1ce6d9d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1ce80688a0 0x7f1ce8142b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:52572/0 (socket says 172.21.15.195:52572) 2022-06-27T12:17:39.712 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.691+0000 7f1ce6d9d700 1 -- 172.21.15.195:0/3838583471 learned_addr learned my addr 172.21.15.195:0/3838583471 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:17:39.712 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.691+0000 7f1ce659c700 1 --2- 172.21.15.195:0/3838583471 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1ce806a440 0x7f1ce806b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:39.712 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.691+0000 7f1ce759e700 1 --2- 172.21.15.195:0/3838583471 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1ce8156100 0x7f1ce806bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:39.713 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.691+0000 7f1ce6d9d700 1 -- 172.21.15.195:0/3838583471 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1ce806a440 msgr2=0x7f1ce806b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:39.713 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.691+0000 7f1ce6d9d700 1 --2- 172.21.15.195:0/3838583471 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1ce806a440 0x7f1ce806b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:39.713 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.691+0000 7f1ce6d9d700 1 -- 172.21.15.195:0/3838583471 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1ce8156100 msgr2=0x7f1ce806bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:39.713 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.691+0000 7f1ce6d9d700 1 --2- 172.21.15.195:0/3838583471 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1ce8156100 0x7f1ce806bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:39.714 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.691+0000 7f1ce6d9d700 1 -- 172.21.15.195:0/3838583471 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ce0009a20 con 0x7f1ce80688a0 2022-06-27T12:17:39.714 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.691+0000 7f1ce6d9d700 1 --2- 172.21.15.195:0/3838583471 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1ce80688a0 0x7f1ce8142b90 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc00c4e0 tx=0x7f1cdc007c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:17:39.714 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.692+0000 7f1cceffd700 1 -- 172.21.15.195:0/3838583471 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cdc00ce90 con 0x7f1ce80688a0 2022-06-27T12:17:39.714 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.692+0000 7f1cceffd700 1 -- 172.21.15.195:0/3838583471 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cdc0050d0 con 0x7f1ce80688a0 2022-06-27T12:17:39.715 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.692+0000 7f1ced087700 1 -- 172.21.15.195:0/3838583471 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ce8166b50 con 0x7f1ce80688a0 2022-06-27T12:17:39.715 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.692+0000 7f1cceffd700 1 -- 172.21.15.195:0/3838583471 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cdc011590 con 0x7f1ce80688a0 2022-06-27T12:17:39.715 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.692+0000 7f1ced087700 1 -- 172.21.15.195:0/3838583471 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ce8167110 con 0x7f1ce80688a0 2022-06-27T12:17:39.716 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.692+0000 7f1ced087700 1 -- 172.21.15.195:0/3838583471 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ce80722c0 con 0x7f1ce80688a0 2022-06-27T12:17:39.716 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.693+0000 7f1cceffd700 1 -- 172.21.15.195:0/3838583471 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f1cdc019030 con 0x7f1ce80688a0 2022-06-27T12:17:39.716 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.693+0000 7f1cceffd700 1 --2- 172.21.15.195:0/3838583471 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1cc8063d00 0x7f1cc80661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:39.716 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.693+0000 7f1cceffd700 1 -- 172.21.15.195:0/3838583471 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7664+0+0 (secure 0 0 0) 0x7f1cdc080050 con 0x7f1ce80688a0 2022-06-27T12:17:39.717 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.693+0000 7f1ce659c700 1 --2- 172.21.15.195:0/3838583471 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1cc8063d00 0x7f1cc80661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:39.717 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.694+0000 7f1ce659c700 1 --2- 172.21.15.195:0/3838583471 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1cc8063d00 0x7f1cc80661b0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f1cd8007fd0 tx=0x7f1cd8009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:17:39.717 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.698+0000 7f1cceffd700 1 -- 172.21.15.195:0/3838583471 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f1cdc04e5a0 con 0x7f1ce80688a0 2022-06-27T12:17:39.930 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.922+0000 7f1ced087700 1 -- 172.21.15.195:0/3838583471 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1ce80bb190 con 0x7f1ce80688a0 2022-06-27T12:17:39.931 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.923+0000 7f1cceffd700 1 -- 172.21.15.195:0/3838583471 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+5371 (secure 0 0 0) 0x7f1cdc017090 con 0x7f1ce80688a0 2022-06-27T12:17:39.931 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:17:39.932 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":112,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:17:30.381992+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7165},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7165":{"gid":7165,"name":"d","rank":0,"incarnation":108,"state":"up:active","state_seq":27,"addr":"172.21.15.202:6835/3551478822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":3551478822},{"type":"v1","addr":"172.21.15.202:6835","nonce":3551478822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:17:39.933 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.925+0000 7f1ced087700 1 -- 172.21.15.195:0/3838583471 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1cc8063d00 msgr2=0x7f1cc80661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:39.933 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.925+0000 7f1ced087700 1 --2- 172.21.15.195:0/3838583471 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1cc8063d00 0x7f1cc80661b0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f1cd8007fd0 tx=0x7f1cd8009040 comp rx=0 tx=0).stop 2022-06-27T12:17:39.934 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.925+0000 7f1ced087700 1 -- 172.21.15.195:0/3838583471 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1ce80688a0 msgr2=0x7f1ce8142b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:39.934 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.925+0000 7f1ced087700 1 --2- 172.21.15.195:0/3838583471 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1ce80688a0 0x7f1ce8142b90 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc00c4e0 tx=0x7f1cdc007c60 comp rx=0 tx=0).stop 2022-06-27T12:17:39.934 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.925+0000 7f1ced087700 1 -- 172.21.15.195:0/3838583471 shutdown_connections 2022-06-27T12:17:39.935 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.925+0000 7f1ced087700 1 --2- 172.21.15.195:0/3838583471 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f1cc8063d00 0x7f1cc80661b0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:39.935 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.925+0000 7f1ced087700 1 --2- 172.21.15.195:0/3838583471 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1ce80688a0 0x7f1ce8142b90 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:39.935 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.925+0000 7f1ced087700 1 --2- 172.21.15.195:0/3838583471 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f1ce8156100 0x7f1ce806bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:39.935 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.925+0000 7f1ced087700 1 --2- 172.21.15.195:0/3838583471 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f1ce806a440 0x7f1ce806b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:39.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.925+0000 7f1ced087700 1 -- 172.21.15.195:0/3838583471 >> 172.21.15.195:0/3838583471 conn(0x7f1ce80c9b00 msgr2=0x7f1ce80c1870 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:17:39.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.925+0000 7f1ced087700 1 -- 172.21.15.195:0/3838583471 shutdown_connections 2022-06-27T12:17:39.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:39.925+0000 7f1ced087700 1 -- 172.21.15.195:0/3838583471 wait complete. 2022-06-27T12:17:39.937 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 112 2022-06-27T12:17:39.939 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2022-06-27T12:17:40.662 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 22% (68/298) Updating files: 23% (69/298) Updating files: 24% (72/298) Updating files: 25% (75/298) Updating files: 26% (78/298) Updating files: 27% (81/298) Updating files: 28% (84/298) Updating files: 29% (87/298) Updating files: 30% (90/298) Updating files: 31% (93/298) Updating files: 32% (96/298) Updating files: 33% (99/298) Updating files: 34% (102/298) Updating files: 35% (105/298) Updating files: 36% (108/298) Updating files: 37% (111/298) Updating files: 38% (114/298) Updating files: 39% (117/298) Updating files: 40% (120/298) Updating files: 41% (123/298) Updating files: 42% (126/298) Updating files: 43% (129/298) Updating files: 44% (132/298) Updating files: 45% (135/298) Updating files: 46% (138/298) Updating files: 47% (141/298) Updating files: 47% (143/298) Updating files: 48% (144/298) Updating files: 49% (147/298) Updating files: 50% (149/298) Updating files: 51% (152/298) Updating files: 52% (155/298) Updating files: 53% (158/298) Updating files: 54% (161/298) Updating files: 55% (164/298) Updating files: 56% (167/298) Updating files: 57% (170/298) Updating files: 58% (173/298) Updating files: 59% (176/298) Updating files: 60% (179/298) Updating files: 61% (182/298) Updating files: 62% (185/298) Updating files: 63% (188/298) Updating files: 64% (191/298) Updating files: 65% (194/298) Updating files: 66% (197/298) Updating files: 67% (200/298) Updating files: 68% (203/298) Updating files: 69% (206/298) Updating files: 70% (209/298) Updating files: 71% (212/298) Updating files: 72% (215/298) Updating files: 73% (218/298) Updating files: 74% (221/298) Updating files: 75% (224/298) Updating files: 75% (225/298) Updating files: 76% (227/298) Updating files: 77% (230/298) Updating files: 78% (233/298) Updating files: 79% (236/298) Updating files: 80% (239/298) Updating files: 81% (242/298) Updating files: 82% (245/298) Updating files: 82% (246/298) Updating files: 83% (248/298) Updating files: 84% (251/298) Updating files: 85% (254/298) Updating files: 86% (257/298) Updating files: 87% (260/298) Updating files: 88% (263/298) Updating files: 89% (266/298) Updating files: 90% (269/298) Updating files: 91% (272/298) Updating files: 92% (275/298) Updating files: 93% (278/298) Updating files: 94% (281/298) Updating files: 95% (284/298) Updating files: 96% (287/298) Updating files: 97% (290/298) Updating files: 98% (293/298) Updating files: 99% (296/298) Updating files: 100% (298/298) Updating files: 100% (298/298), done. 2022-06-27T12:17:40.989 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at 1bca9c5c412 v0.77 2022-06-27T12:17:40.993 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.77 2022-06-27T12:17:41.035 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:17:41.036 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 78 -eq 48 ']' 2022-06-27T12:17:41.036 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.78 2022-06-27T12:17:41.037 INFO:tasks.workunit.client.0.smithi195.stdout:v0.78 2022-06-27T12:17:41.037 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.78 2022-06-27T12:17:41.038 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.78 2022-06-27T12:17:41.941 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:17:42.064 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.064+0000 7f406966c700 1 Processor -- start 2022-06-27T12:17:42.064 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.064+0000 7f406966c700 1 -- start start 2022-06-27T12:17:42.069 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.065+0000 7f406966c700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f40640c40d0 0x7f4064157010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:42.070 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.065+0000 7f406966c700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4064156b80 con 0x7f4064067ed0 2022-06-27T12:17:42.070 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.065+0000 7f406966c700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f406406a040 con 0x7f40640c6190 2022-06-27T12:17:42.070 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.065+0000 7f406966c700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4064068370 con 0x7f40640c40d0 2022-06-27T12:17:42.070 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.065+0000 7f40637fe700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f4064067ed0 0x7f406414cb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:58794/0 (socket says 172.21.15.195:58794) 2022-06-27T12:17:42.071 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.065+0000 7f40637fe700 1 -- 172.21.15.195:0/3263061072 learned_addr learned my addr 172.21.15.195:0/3263061072 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:17:42.071 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.065+0000 7f4062ffd700 1 --2- 172.21.15.195:0/3263061072 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f40640c40d0 0x7f4064157010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:42.071 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.065+0000 7f4061ffb700 1 -- 172.21.15.195:0/3263061072 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 510834703 0 0) 0x7f4064156b80 con 0x7f4064067ed0 2022-06-27T12:17:42.071 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.065+0000 7f4061ffb700 1 -- 172.21.15.195:0/3263061072 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4050003430 con 0x7f4064067ed0 2022-06-27T12:17:42.072 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.065+0000 7f4061ffb700 1 -- 172.21.15.195:0/3263061072 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2771907368 0 0) 0x7f406406a040 con 0x7f40640c6190 2022-06-27T12:17:42.072 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.065+0000 7f4061ffb700 1 -- 172.21.15.195:0/3263061072 --> v1:172.21.15.202:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4064156b80 con 0x7f40640c6190 2022-06-27T12:17:42.072 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.065+0000 7f4062ffd700 1 -- 172.21.15.195:0/3263061072 >> v1:172.21.15.202:6789/0 conn(0x7f40640c6190 legacy=0x7f40640c44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:42.072 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.065+0000 7f4062ffd700 1 -- 172.21.15.195:0/3263061072 >> v1:172.21.15.195:6789/0 conn(0x7f4064067ed0 legacy=0x7f406414cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:42.073 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.065+0000 7f4062ffd700 1 -- 172.21.15.195:0/3263061072 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f406406a300 con 0x7f40640c40d0 2022-06-27T12:17:42.073 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.066+0000 7f4062ffd700 1 --2- 172.21.15.195:0/3263061072 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f40640c40d0 0x7f4064157010 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f40580097d0 tx=0x7f4058009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a3522ee043da865 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:17:42.073 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.066+0000 7f4061ffb700 1 -- 172.21.15.195:0/3263061072 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4058011630 con 0x7f40640c40d0 2022-06-27T12:17:42.073 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.066+0000 7f4061ffb700 1 -- 172.21.15.195:0/3263061072 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4058011790 con 0x7f40640c40d0 2022-06-27T12:17:42.073 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.066+0000 7f4061ffb700 1 -- 172.21.15.195:0/3263061072 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4058011b30 con 0x7f40640c40d0 2022-06-27T12:17:42.074 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.066+0000 7f406966c700 1 -- 172.21.15.195:0/3263061072 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f40640c40d0 msgr2=0x7f4064157010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:42.074 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.066+0000 7f406966c700 1 --2- 172.21.15.195:0/3263061072 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f40640c40d0 0x7f4064157010 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f40580097d0 tx=0x7f4058009060 comp rx=0 tx=0).stop 2022-06-27T12:17:42.074 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.066+0000 7f406966c700 1 -- 172.21.15.195:0/3263061072 shutdown_connections 2022-06-27T12:17:42.074 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.066+0000 7f406966c700 1 --2- 172.21.15.195:0/3263061072 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f40640c40d0 0x7f4064157010 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:42.075 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.066+0000 7f406966c700 1 -- 172.21.15.195:0/3263061072 >> 172.21.15.195:0/3263061072 conn(0x7f40640ccb80 msgr2=0x7f40640ccf80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:17:42.075 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.066+0000 7f406966c700 1 -- 172.21.15.195:0/3263061072 shutdown_connections 2022-06-27T12:17:42.075 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.066+0000 7f406966c700 1 -- 172.21.15.195:0/3263061072 wait complete. 2022-06-27T12:17:42.075 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.068+0000 7f406966c700 1 Processor -- start 2022-06-27T12:17:42.075 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.068+0000 7f406966c700 1 -- start start 2022-06-27T12:17:42.076 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.068+0000 7f406966c700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4064067ed0 0x7f4064157690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:42.076 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.068+0000 7f406966c700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f40640c40d0 0x7f406414fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:42.076 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.068+0000 7f406966c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f40640c6190 0x7f406415c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:42.076 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.068+0000 7f406966c700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4064158f30 con 0x7f40640c6190 2022-06-27T12:17:42.077 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.068+0000 7f406966c700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4064157bd0 con 0x7f40640c40d0 2022-06-27T12:17:42.077 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.068+0000 7f406966c700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40641670d0 con 0x7f4064067ed0 2022-06-27T12:17:42.077 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.068+0000 7f40637fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f40640c6190 0x7f406415c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:42.077 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.068+0000 7f40637fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f40640c6190 0x7f406415c0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:52612/0 (socket says 172.21.15.195:52612) 2022-06-27T12:17:42.077 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.068+0000 7f40637fe700 1 -- 172.21.15.195:0/776086125 learned_addr learned my addr 172.21.15.195:0/776086125 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:17:42.078 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.069+0000 7f4062ffd700 1 --2- 172.21.15.195:0/776086125 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4064067ed0 0x7f4064157690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:42.078 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.069+0000 7f40627fc700 1 --2- 172.21.15.195:0/776086125 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f40640c40d0 0x7f406414fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:42.078 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.069+0000 7f40637fe700 1 -- 172.21.15.195:0/776086125 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4064067ed0 msgr2=0x7f4064157690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:42.078 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.069+0000 7f40637fe700 1 --2- 172.21.15.195:0/776086125 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4064067ed0 0x7f4064157690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:42.079 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.069+0000 7f40637fe700 1 -- 172.21.15.195:0/776086125 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f40640c40d0 msgr2=0x7f406414fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:42.079 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.069+0000 7f40637fe700 1 --2- 172.21.15.195:0/776086125 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f40640c40d0 0x7f406414fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:42.079 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.069+0000 7f40637fe700 1 -- 172.21.15.195:0/776086125 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40580092e0 con 0x7f40640c6190 2022-06-27T12:17:42.079 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.069+0000 7f40637fe700 1 --2- 172.21.15.195:0/776086125 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f40640c6190 0x7f406415c0b0 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f405c00c910 tx=0x7f405c001650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:17:42.080 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.069+0000 7f404affd700 1 -- 172.21.15.195:0/776086125 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f405c005d70 con 0x7f40640c6190 2022-06-27T12:17:42.080 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.069+0000 7f404affd700 1 -- 172.21.15.195:0/776086125 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f405c0070f0 con 0x7f40640c6190 2022-06-27T12:17:42.080 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.069+0000 7f404affd700 1 -- 172.21.15.195:0/776086125 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f405c007470 con 0x7f40640c6190 2022-06-27T12:17:42.080 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.069+0000 7f406966c700 1 -- 172.21.15.195:0/776086125 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4064156750 con 0x7f40640c6190 2022-06-27T12:17:42.080 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.069+0000 7f406966c700 1 -- 172.21.15.195:0/776086125 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40640c2a00 con 0x7f40640c6190 2022-06-27T12:17:42.082 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.070+0000 7f406966c700 1 -- 172.21.15.195:0/776086125 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40640722c0 con 0x7f40640c6190 2022-06-27T12:17:42.082 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.070+0000 7f404affd700 1 -- 172.21.15.195:0/776086125 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f405c010070 con 0x7f40640c6190 2022-06-27T12:17:42.082 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.071+0000 7f404affd700 1 --2- 172.21.15.195:0/776086125 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4044063d00 0x7f40440661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:17:42.083 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.071+0000 7f404affd700 1 -- 172.21.15.195:0/776086125 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7664+0+0 (secure 0 0 0) 0x7f405c080a40 con 0x7f40640c6190 2022-06-27T12:17:42.083 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.071+0000 7f4062ffd700 1 --2- 172.21.15.195:0/776086125 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4044063d00 0x7f40440661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:17:42.083 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.071+0000 7f4062ffd700 1 --2- 172.21.15.195:0/776086125 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4044063d00 0x7f40440661b0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f4058009130 tx=0x7f4058017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:17:42.083 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.075+0000 7f404affd700 1 -- 172.21.15.195:0/776086125 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f405c04f070 con 0x7f40640c6190 2022-06-27T12:17:42.302 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.302+0000 7f406966c700 1 -- 172.21.15.195:0/776086125 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4064167210 con 0x7f40640c6190 2022-06-27T12:17:42.302 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.302+0000 7f404affd700 1 -- 172.21.15.195:0/776086125 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+6132 (secure 0 0 0) 0x7f405c052690 con 0x7f40640c6190 2022-06-27T12:17:42.303 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:17:42.303 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":113,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":7300,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/1965783412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1965783412},{"type":"v1","addr":"172.21.15.202:6833","nonce":1965783412}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":113}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:17:30.381992+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7165},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7165":{"gid":7165,"name":"d","rank":0,"incarnation":108,"state":"up:active","state_seq":27,"addr":"172.21.15.202:6835/3551478822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":3551478822},{"type":"v1","addr":"172.21.15.202:6835","nonce":3551478822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:17:42.304 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.304+0000 7f406966c700 1 -- 172.21.15.195:0/776086125 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4044063d00 msgr2=0x7f40440661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:42.305 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.304+0000 7f406966c700 1 --2- 172.21.15.195:0/776086125 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4044063d00 0x7f40440661b0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f4058009130 tx=0x7f4058017000 comp rx=0 tx=0).stop 2022-06-27T12:17:42.305 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.304+0000 7f406966c700 1 -- 172.21.15.195:0/776086125 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f40640c6190 msgr2=0x7f406415c0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:17:42.305 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.304+0000 7f406966c700 1 --2- 172.21.15.195:0/776086125 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f40640c6190 0x7f406415c0b0 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f405c00c910 tx=0x7f405c001650 comp rx=0 tx=0).stop 2022-06-27T12:17:42.305 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.304+0000 7f406966c700 1 -- 172.21.15.195:0/776086125 shutdown_connections 2022-06-27T12:17:42.306 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.304+0000 7f406966c700 1 --2- 172.21.15.195:0/776086125 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4044063d00 0x7f40440661b0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:42.306 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.304+0000 7f406966c700 1 --2- 172.21.15.195:0/776086125 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f40640c6190 0x7f406415c0b0 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:42.306 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.304+0000 7f406966c700 1 --2- 172.21.15.195:0/776086125 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4064067ed0 0x7f4064157690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:42.306 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.304+0000 7f406966c700 1 --2- 172.21.15.195:0/776086125 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f40640c40d0 0x7f406414fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:17:42.307 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.304+0000 7f406966c700 1 -- 172.21.15.195:0/776086125 >> 172.21.15.195:0/776086125 conn(0x7f40640ccb80 msgr2=0x7f40640cc090 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:17:42.307 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.304+0000 7f406966c700 1 -- 172.21.15.195:0/776086125 shutdown_connections 2022-06-27T12:17:42.307 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:17:42.304+0000 7f406966c700 1 -- 172.21.15.195:0/776086125 wait complete. 2022-06-27T12:17:42.307 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 113 2022-06-27T12:17:42.317 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2022-06-27T12:17:42.318 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 32 secs before thrashing 2022-06-27T12:17:54.312 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 16% (83/496) Updating files: 17% (85/496) Updating files: 18% (90/496) Updating files: 19% (95/496) Updating files: 20% (100/496) Updating files: 21% (105/496) Updating files: 22% (110/496) Updating files: 23% (115/496) Updating files: 24% (120/496) Updating files: 25% (124/496) Updating files: 26% (129/496) Updating files: 26% (131/496) Updating files: 27% (134/496) Updating files: 28% (139/496) Updating files: 29% (144/496) Updating files: 30% (149/496) Updating files: 31% (154/496) Updating files: 32% (159/496) Updating files: 33% (164/496) Updating files: 34% (169/496) Updating files: 35% (174/496) Updating files: 36% (179/496) Updating files: 37% (184/496) Updating files: 38% (189/496) Updating files: 39% (194/496) Updating files: 40% (199/496) Updating files: 41% (204/496) Updating files: 42% (209/496) Updating files: 43% (214/496) Updating files: 44% (219/496) Updating files: 45% (224/496) Updating files: 46% (229/496) Updating files: 47% (234/496) Updating files: 48% (239/496) Updating files: 49% (244/496) Updating files: 50% (248/496) Updating files: 51% (253/496) Updating files: 52% (258/496) Updating files: 53% (263/496) Updating files: 54% (268/496) Updating files: 55% (273/496) Updating files: 56% (278/496) Updating files: 57% (283/496) Updating files: 58% (288/496) Updating files: 59% (293/496) Updating files: 60% (298/496) Updating files: 61% (303/496) Updating files: 62% (308/496) Updating files: 63% (313/496) Updating files: 64% (318/496) Updating files: 65% (323/496) Updating files: 66% (328/496) Updating files: 67% (333/496) Updating files: 68% (338/496) Updating files: 69% (343/496) Updating files: 70% (348/496) Updating files: 71% (353/496) Updating files: 72% (358/496) Updating files: 73% (363/496) Updating files: 74% (368/496) Updating files: 75% (372/496) Updating files: 76% (377/496) Updating files: 77% (382/496) Updating files: 78% (387/496) Updating files: 79% (392/496) Updating files: 80% (397/496) Updating files: 81% (402/496) Updating files: 82% (407/496) Updating files: 83% (412/496) Updating files: 83% (415/496) Updating files: 84% (417/496) Updating files: 85% (422/496) Updating files: 86% (427/496) Updating files: 87% (432/496) Updating files: 88% (437/496) Updating files: 89% (442/496) Updating files: 90% (447/496) Updating files: 91% (452/496) Updating files: 92% (457/496) Updating files: 93% (462/496) Updating files: 94% (467/496) Updating files: 95% (472/496) Updating files: 96% (477/496) Updating files: 97% (482/496) Updating files: 98% (487/496) Updating files: 99% (492/496) Updating files: 100% (496/496) Updating files: 100% (496/496), done. 2022-06-27T12:17:54.468 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at f6c746c314d 0.78 2022-06-27T12:17:54.472 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.78 2022-06-27T12:17:54.520 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:17:54.521 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 79 -eq 48 ']' 2022-06-27T12:17:54.521 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.79 2022-06-27T12:17:54.521 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.79 2022-06-27T12:17:54.522 INFO:tasks.workunit.client.0.smithi195.stdout:v0.79 2022-06-27T12:17:54.523 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.79 2022-06-27T12:18:00.162 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 6% (20/329) Updating files: 7% (24/329) Updating files: 8% (27/329) Updating files: 9% (30/329) Updating files: 10% (33/329) Updating files: 11% (37/329) Updating files: 12% (40/329) Updating files: 13% (43/329) Updating files: 14% (47/329) Updating files: 15% (50/329) Updating files: 16% (53/329) Updating files: 17% (56/329) Updating files: 18% (60/329) Updating files: 19% (63/329) Updating files: 20% (66/329) Updating files: 21% (70/329) Updating files: 22% (73/329) Updating files: 23% (76/329) Updating files: 24% (79/329) Updating files: 25% (83/329) Updating files: 26% (86/329) Updating files: 27% (89/329) Updating files: 28% (93/329) Updating files: 29% (96/329) Updating files: 30% (99/329) Updating files: 31% (102/329) Updating files: 32% (106/329) Updating files: 33% (109/329) Updating files: 34% (112/329) Updating files: 35% (116/329) Updating files: 36% (119/329) Updating files: 37% (122/329) Updating files: 38% (126/329) Updating files: 39% (129/329) Updating files: 40% (132/329) Updating files: 41% (135/329) Updating files: 42% (139/329) Updating files: 43% (142/329) Updating files: 44% (145/329) Updating files: 45% (149/329) Updating files: 46% (152/329) Updating files: 47% (155/329) Updating files: 48% (158/329) Updating files: 49% (162/329) Updating files: 50% (165/329) Updating files: 50% (166/329) Updating files: 51% (168/329) Updating files: 52% (172/329) Updating files: 53% (175/329) Updating files: 54% (178/329) Updating files: 55% (181/329) Updating files: 56% (185/329) Updating files: 57% (188/329) Updating files: 58% (191/329) Updating files: 59% (195/329) Updating files: 60% (198/329) Updating files: 61% (201/329) Updating files: 62% (204/329) Updating files: 63% (208/329) Updating files: 64% (211/329) Updating files: 65% (214/329) Updating files: 66% (218/329) Updating files: 67% (221/329) Updating files: 68% (224/329) Updating files: 69% (228/329) Updating files: 70% (231/329) Updating files: 71% (234/329) Updating files: 72% (237/329) Updating files: 73% (241/329) Updating files: 74% (244/329) Updating files: 75% (247/329) Updating files: 76% (251/329) Updating files: 77% (254/329) Updating files: 78% (257/329) Updating files: 79% (260/329) Updating files: 80% (264/329) Updating files: 81% (267/329) Updating files: 82% (270/329) Updating files: 83% (274/329) Updating files: 84% (277/329) Updating files: 85% (280/329) Updating files: 86% (283/329) Updating files: 87% (287/329) Updating files: 88% (290/329) Updating files: 89% (293/329) Updating files: 90% (297/329) Updating files: 91% (300/329) Updating files: 92% (303/329) Updating files: 93% (306/329) Updating files: 94% (310/329) Updating files: 95% (313/329) Updating files: 96% (316/329) Updating files: 97% (320/329) Updating files: 98% (323/329) Updating files: 99% (326/329) Updating files: 100% (329/329) Updating files: 100% (329/329), done. 2022-06-27T12:18:00.205 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at 4c2d73a5095 0.79 2022-06-27T12:18:00.213 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.79 2022-06-27T12:18:01.086 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:18:01.087 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 80 -eq 48 ']' 2022-06-27T12:18:01.087 INFO:tasks.workunit.client.0.smithi195.stdout:v0.80 2022-06-27T12:18:01.088 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.80 2022-06-27T12:18:01.089 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.80 2022-06-27T12:18:01.089 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.80 2022-06-27T12:18:03.962 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 14% (25/176) Updating files: 15% (27/176) Updating files: 16% (29/176) Updating files: 17% (30/176) Updating files: 18% (32/176) Updating files: 19% (34/176) Updating files: 20% (36/176) Updating files: 21% (37/176) Updating files: 22% (39/176) Updating files: 23% (41/176) Updating files: 24% (43/176) Updating files: 25% (44/176) Updating files: 26% (46/176) Updating files: 27% (48/176) Updating files: 28% (50/176) Updating files: 29% (52/176) Updating files: 30% (53/176) Updating files: 31% (55/176) Updating files: 32% (57/176) Updating files: 33% (59/176) Updating files: 34% (60/176) Updating files: 35% (62/176) Updating files: 36% (64/176) Updating files: 37% (66/176) Updating files: 38% (67/176) Updating files: 39% (69/176) Updating files: 40% (71/176) Updating files: 41% (73/176) Updating files: 42% (74/176) Updating files: 43% (76/176) Updating files: 44% (78/176) Updating files: 45% (80/176) Updating files: 46% (81/176) Updating files: 47% (83/176) Updating files: 48% (85/176) Updating files: 49% (87/176) Updating files: 50% (88/176) Updating files: 51% (90/176) Updating files: 52% (92/176) Updating files: 53% (94/176) Updating files: 53% (95/176) Updating files: 54% (96/176) Updating files: 55% (97/176) Updating files: 56% (99/176) Updating files: 57% (101/176) Updating files: 58% (103/176) Updating files: 59% (104/176) Updating files: 60% (106/176) Updating files: 61% (108/176) Updating files: 62% (110/176) Updating files: 63% (111/176) Updating files: 64% (113/176) Updating files: 65% (115/176) Updating files: 66% (117/176) Updating files: 67% (118/176) Updating files: 68% (120/176) Updating files: 69% (122/176) Updating files: 70% (124/176) Updating files: 71% (125/176) Updating files: 72% (127/176) Updating files: 73% (129/176) Updating files: 74% (131/176) Updating files: 75% (132/176) Updating files: 76% (134/176) Updating files: 77% (136/176) Updating files: 78% (138/176) Updating files: 79% (140/176) Updating files: 80% (141/176) Updating files: 81% (143/176) Updating files: 82% (145/176) Updating files: 83% (147/176) Updating files: 84% (148/176) Updating files: 85% (150/176) Updating files: 86% (152/176) Updating files: 87% (154/176) Updating files: 88% (155/176) Updating files: 89% (157/176) Updating files: 90% (159/176) Updating files: 91% (161/176) Updating files: 92% (162/176) Updating files: 93% (164/176) Updating files: 94% (166/176) Updating files: 95% (168/176) Updating files: 96% (169/176) Updating files: 97% (171/176) Updating files: 98% (173/176) Updating files: 99% (175/176) Updating files: 100% (176/176) Updating files: 100% (176/176), done. 2022-06-27T12:18:05.045 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at b78644e7dee 0.80 2022-06-27T12:18:05.049 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.80 2022-06-27T12:18:05.188 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:18:05.188 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 81 -eq 48 ']' 2022-06-27T12:18:05.189 INFO:tasks.workunit.client.0.smithi195.stdout:v0.81 2022-06-27T12:18:05.191 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.81 2022-06-27T12:18:05.192 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.81 2022-06-27T12:18:05.193 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.81 2022-06-27T12:18:05.539 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:18:05.542 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:18:08.545 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 60% (188/309) Updating files: 61% (189/309) Updating files: 62% (192/309) Updating files: 63% (195/309) Updating files: 64% (198/309) Updating files: 65% (201/309) Updating files: 66% (204/309) Updating files: 67% (208/309) Updating files: 67% (210/309) Updating files: 68% (211/309) Updating files: 69% (214/309) Updating files: 70% (217/309) Updating files: 71% (220/309) Updating files: 72% (223/309) Updating files: 73% (226/309) Updating files: 74% (229/309) Updating files: 75% (232/309) Updating files: 76% (235/309) Updating files: 77% (238/309) Updating files: 78% (242/309) Updating files: 79% (245/309) Updating files: 80% (248/309) Updating files: 81% (251/309) Updating files: 82% (254/309) Updating files: 83% (257/309) Updating files: 84% (260/309) Updating files: 85% (263/309) Updating files: 86% (266/309) Updating files: 87% (269/309) Updating files: 88% (272/309) Updating files: 89% (276/309) Updating files: 90% (279/309) Updating files: 91% (282/309) Updating files: 92% (285/309) Updating files: 93% (288/309) Updating files: 94% (291/309) Updating files: 94% (293/309) Updating files: 95% (294/309) Updating files: 96% (297/309) Updating files: 97% (300/309) Updating files: 98% (303/309) Updating files: 99% (306/309) Updating files: 100% (309/309) Updating files: 100% (309/309), done. 2022-06-27T12:18:08.648 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at 8de9501df27 0.81 2022-06-27T12:18:08.655 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.81 2022-06-27T12:18:08.708 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:18:08.708 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 82 -eq 48 ']' 2022-06-27T12:18:08.709 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.82 2022-06-27T12:18:08.709 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.82 2022-06-27T12:18:08.710 INFO:tasks.workunit.client.0.smithi195.stdout:v0.82 2022-06-27T12:18:08.711 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.82 2022-06-27T12:18:13.887 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 60% (124/204) Updating files: 61% (125/204) Updating files: 62% (127/204) Updating files: 63% (129/204) Updating files: 64% (131/204) Updating files: 65% (133/204) Updating files: 66% (135/204) Updating files: 66% (136/204) Updating files: 67% (137/204) Updating files: 68% (139/204) Updating files: 69% (141/204) Updating files: 70% (143/204) Updating files: 71% (145/204) Updating files: 72% (147/204) Updating files: 73% (149/204) Updating files: 74% (151/204) Updating files: 75% (153/204) Updating files: 76% (156/204) Updating files: 77% (158/204) Updating files: 78% (160/204) Updating files: 79% (162/204) Updating files: 80% (164/204) Updating files: 81% (166/204) Updating files: 82% (168/204) Updating files: 83% (170/204) Updating files: 84% (172/204) Updating files: 85% (174/204) Updating files: 86% (176/204) Updating files: 87% (178/204) Updating files: 88% (180/204) Updating files: 89% (182/204) Updating files: 90% (184/204) Updating files: 91% (186/204) Updating files: 92% (188/204) Updating files: 93% (190/204) Updating files: 94% (192/204) Updating files: 95% (194/204) Updating files: 96% (196/204) Updating files: 97% (198/204) Updating files: 98% (200/204) Updating files: 99% (202/204) Updating files: 100% (204/204) Updating files: 100% (204/204), done. 2022-06-27T12:18:13.957 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at 14085f42ddd 0.82 2022-06-27T12:18:13.962 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.82 2022-06-27T12:18:14.319 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:18:14.462 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.441+0000 7f6680910700 1 Processor -- start 2022-06-27T12:18:14.463 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.441+0000 7f6680910700 1 -- start start 2022-06-27T12:18:14.463 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.442+0000 7f6680910700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f667c14e860 0x7f667c0c3840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:14.464 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.442+0000 7f6680910700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f667c0c3d80 0x7f667c157890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:14.464 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.442+0000 7f6680910700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f667c0c33b0 con 0x7f667c14e490 2022-06-27T12:18:14.465 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.442+0000 7f6680910700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f667c06f220 con 0x7f667c14e860 2022-06-27T12:18:14.465 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.442+0000 7f6680910700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f667c1542c0 con 0x7f667c0c3d80 2022-06-27T12:18:14.466 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.442+0000 7f6679d9b700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f667c14e860 0x7f667c0c3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:14.466 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.442+0000 7f667a59c700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f667c14e490 0x7f667c06cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:58954/0 (socket says 172.21.15.195:58954) 2022-06-27T12:18:14.466 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.442+0000 7f667a59c700 1 -- 172.21.15.195:0/2998553022 learned_addr learned my addr 172.21.15.195:0/2998553022 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:18:14.467 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.442+0000 7f667ad9d700 1 --2- 172.21.15.195:0/2998553022 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f667c0c3d80 0x7f667c157890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:14.467 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.442+0000 7f6679d9b700 1 -- 172.21.15.195:0/2998553022 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f667c0c3d80 msgr2=0x7f667c157890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:18:14.468 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.442+0000 7f6679d9b700 1 --2- 172.21.15.195:0/2998553022 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f667c0c3d80 0x7f667c157890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:14.468 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.442+0000 7f6679d9b700 1 -- 172.21.15.195:0/2998553022 >> v1:172.21.15.195:6789/0 conn(0x7f667c14e490 legacy=0x7f667c06cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:14.468 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.442+0000 7f6679d9b700 1 -- 172.21.15.195:0/2998553022 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f667c154560 con 0x7f667c14e860 2022-06-27T12:18:14.469 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.442+0000 7f667ad9d700 1 --2- 172.21.15.195:0/2998553022 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f667c0c3d80 0x7f667c157890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:18:14.469 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.443+0000 7f6679d9b700 1 --2- 172.21.15.195:0/2998553022 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f667c14e860 0x7f667c0c3840 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f66700097a0 tx=0x7f6670009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b69d1c752efd22f9 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:18:14.469 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.443+0000 7f667959a700 1 -- 172.21.15.195:0/2998553022 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6670011630 con 0x7f667c14e860 2022-06-27T12:18:14.470 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.443+0000 7f667959a700 1 -- 172.21.15.195:0/2998553022 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6670011790 con 0x7f667c14e860 2022-06-27T12:18:14.470 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.443+0000 7f667959a700 1 -- 172.21.15.195:0/2998553022 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6670011b30 con 0x7f667c14e860 2022-06-27T12:18:14.471 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.443+0000 7f6680910700 1 -- 172.21.15.195:0/2998553022 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f667c14e860 msgr2=0x7f667c0c3840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:14.471 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.443+0000 7f6680910700 1 --2- 172.21.15.195:0/2998553022 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f667c14e860 0x7f667c0c3840 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f66700097a0 tx=0x7f6670009060 comp rx=0 tx=0).stop 2022-06-27T12:18:14.471 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.443+0000 7f6680910700 1 -- 172.21.15.195:0/2998553022 shutdown_connections 2022-06-27T12:18:14.472 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.443+0000 7f6680910700 1 --2- 172.21.15.195:0/2998553022 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f667c0c3d80 0x7f667c157890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:14.472 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.443+0000 7f6680910700 1 --2- 172.21.15.195:0/2998553022 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f667c14e860 0x7f667c0c3840 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:14.473 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.443+0000 7f6680910700 1 -- 172.21.15.195:0/2998553022 >> 172.21.15.195:0/2998553022 conn(0x7f667c063af0 msgr2=0x7f667c063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:18:14.473 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.443+0000 7f6680910700 1 -- 172.21.15.195:0/2998553022 shutdown_connections 2022-06-27T12:18:14.473 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.443+0000 7f6680910700 1 -- 172.21.15.195:0/2998553022 wait complete. 2022-06-27T12:18:14.474 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.444+0000 7f6680910700 1 Processor -- start 2022-06-27T12:18:14.474 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.444+0000 7f6680910700 1 -- start start 2022-06-27T12:18:14.474 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.445+0000 7f6680910700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f667c0c3d80 0x7f667c164360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:14.475 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.445+0000 7f6680910700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f667c14e490 0x7f667c14b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:14.475 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.445+0000 7f6680910700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f667c14e860 0x7f667c157410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:14.476 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.445+0000 7f6680910700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f667c151390 con 0x7f667c0c3d80 2022-06-27T12:18:14.476 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.445+0000 7f6680910700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f667c156470 con 0x7f667c14e490 2022-06-27T12:18:14.476 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.445+0000 7f6680910700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f667c151cd0 con 0x7f667c14e860 2022-06-27T12:18:14.477 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.445+0000 7f667a59c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f667c0c3d80 0x7f667c164360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:14.477 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.445+0000 7f667a59c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f667c0c3d80 0x7f667c164360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:52772/0 (socket says 172.21.15.195:52772) 2022-06-27T12:18:14.478 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.445+0000 7f667a59c700 1 -- 172.21.15.195:0/3760560275 learned_addr learned my addr 172.21.15.195:0/3760560275 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:18:14.478 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.445+0000 7f667ad9d700 1 --2- 172.21.15.195:0/3760560275 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f667c14e860 0x7f667c157410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:14.479 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.445+0000 7f6679d9b700 1 --2- 172.21.15.195:0/3760560275 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f667c14e490 0x7f667c14b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:14.479 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.446+0000 7f667a59c700 1 -- 172.21.15.195:0/3760560275 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f667c14e860 msgr2=0x7f667c157410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:14.480 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.446+0000 7f667a59c700 1 --2- 172.21.15.195:0/3760560275 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f667c14e860 0x7f667c157410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:14.480 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.446+0000 7f667a59c700 1 -- 172.21.15.195:0/3760560275 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f667c14e490 msgr2=0x7f667c14b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:14.482 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.446+0000 7f667a59c700 1 --2- 172.21.15.195:0/3760560275 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f667c14e490 0x7f667c14b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:14.483 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.446+0000 7f667a59c700 1 -- 172.21.15.195:0/3760560275 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66700092e0 con 0x7f667c0c3d80 2022-06-27T12:18:14.484 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.446+0000 7f667a59c700 1 --2- 172.21.15.195:0/3760560275 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f667c0c3d80 0x7f667c164360 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f666c00c510 tx=0x7f666c007e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:18:14.484 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.446+0000 7f666a7fc700 1 -- 172.21.15.195:0/3760560275 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f666c00ce80 con 0x7f667c0c3d80 2022-06-27T12:18:14.485 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.446+0000 7f666a7fc700 1 -- 172.21.15.195:0/3760560275 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f666c0072e0 con 0x7f667c0c3d80 2022-06-27T12:18:14.485 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.446+0000 7f6680910700 1 -- 172.21.15.195:0/3760560275 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f667c1555c0 con 0x7f667c0c3d80 2022-06-27T12:18:14.485 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.446+0000 7f666a7fc700 1 -- 172.21.15.195:0/3760560275 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f666c007680 con 0x7f667c0c3d80 2022-06-27T12:18:14.486 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.446+0000 7f6680910700 1 -- 172.21.15.195:0/3760560275 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f667c155b40 con 0x7f667c0c3d80 2022-06-27T12:18:14.486 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.447+0000 7f6680910700 1 -- 172.21.15.195:0/3760560275 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f667c0722c0 con 0x7f667c0c3d80 2022-06-27T12:18:14.487 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.447+0000 7f666a7fc700 1 -- 172.21.15.195:0/3760560275 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f666c010070 con 0x7f667c0c3d80 2022-06-27T12:18:14.487 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.448+0000 7f666a7fc700 1 --2- 172.21.15.195:0/3760560275 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6658063d30 0x7f66580661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:14.488 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.448+0000 7f666a7fc700 1 -- 172.21.15.195:0/3760560275 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7664+0+0 (secure 0 0 0) 0x7f666c0804c0 con 0x7f667c0c3d80 2022-06-27T12:18:14.488 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.448+0000 7f6679d9b700 1 --2- 172.21.15.195:0/3760560275 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6658063d30 0x7f66580661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:14.489 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.448+0000 7f6679d9b700 1 --2- 172.21.15.195:0/3760560275 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6658063d30 0x7f66580661e0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f6670009260 tx=0x7f6670017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:18:14.489 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.452+0000 7f666a7fc700 1 -- 172.21.15.195:0/3760560275 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f666c04e9a0 con 0x7f667c0c3d80 2022-06-27T12:18:14.715 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:18:14.716 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":113,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":7300,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/1965783412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1965783412},{"type":"v1","addr":"172.21.15.202:6833","nonce":1965783412}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":113}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:17:30.381992+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7165},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7165":{"gid":7165,"name":"d","rank":0,"incarnation":108,"state":"up:active","state_seq":27,"addr":"172.21.15.202:6835/3551478822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":3551478822},{"type":"v1","addr":"172.21.15.202:6835","nonce":3551478822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:18:14.718 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.709+0000 7f6680910700 1 -- 172.21.15.195:0/3760560275 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f667c0c60d0 con 0x7f667c0c3d80 2022-06-27T12:18:14.718 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.710+0000 7f666a7fc700 1 -- 172.21.15.195:0/3760560275 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+6132 (secure 0 0 0) 0x7f666c051fc0 con 0x7f667c0c3d80 2022-06-27T12:18:14.719 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.713+0000 7f6680910700 1 -- 172.21.15.195:0/3760560275 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6658063d30 msgr2=0x7f66580661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:14.719 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.713+0000 7f6680910700 1 --2- 172.21.15.195:0/3760560275 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6658063d30 0x7f66580661e0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f6670009260 tx=0x7f6670017000 comp rx=0 tx=0).stop 2022-06-27T12:18:14.720 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.713+0000 7f6680910700 1 -- 172.21.15.195:0/3760560275 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f667c0c3d80 msgr2=0x7f667c164360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:14.720 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.713+0000 7f6680910700 1 --2- 172.21.15.195:0/3760560275 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f667c0c3d80 0x7f667c164360 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f666c00c510 tx=0x7f666c007e30 comp rx=0 tx=0).stop 2022-06-27T12:18:14.721 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.714+0000 7f6680910700 1 -- 172.21.15.195:0/3760560275 shutdown_connections 2022-06-27T12:18:14.721 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.714+0000 7f6680910700 1 --2- 172.21.15.195:0/3760560275 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6658063d30 0x7f66580661e0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:14.721 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.714+0000 7f6680910700 1 --2- 172.21.15.195:0/3760560275 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f667c0c3d80 0x7f667c164360 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:14.722 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.714+0000 7f6680910700 1 --2- 172.21.15.195:0/3760560275 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f667c14e860 0x7f667c157410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:14.722 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.714+0000 7f6680910700 1 --2- 172.21.15.195:0/3760560275 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f667c14e490 0x7f667c14b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:14.723 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.714+0000 7f6680910700 1 -- 172.21.15.195:0/3760560275 >> 172.21.15.195:0/3760560275 conn(0x7f667c063af0 msgr2=0x7f667c0c8880 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:18:14.723 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.714+0000 7f6680910700 1 -- 172.21.15.195:0/3760560275 shutdown_connections 2022-06-27T12:18:14.723 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.716+0000 7f6680910700 1 -- 172.21.15.195:0/3760560275 wait complete. 2022-06-27T12:18:14.724 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 113 2022-06-27T12:18:14.731 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.d (rank=0) 2022-06-27T12:18:14.732 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2022-06-27T12:18:14.732 INFO:teuthology.orchestra.run:waiting for 300 2022-06-27T12:18:14.753 INFO:tasks.ceph.mds.d:Stopped 2022-06-27T12:18:14.754 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:18:14.880 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.879+0000 7f5568165700 1 Processor -- start 2022-06-27T12:18:14.880 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.880+0000 7f5568165700 1 -- start start 2022-06-27T12:18:14.909 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.880+0000 7f5568165700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f556006f640 0x7f5560156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:14.909 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.880+0000 7f5568165700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f55600c39e0 0x7f55601535a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:14.910 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.880+0000 7f5568165700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55600c1a20 con 0x7f55600c4160 2022-06-27T12:18:14.910 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.880+0000 7f5568165700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55600c1130 con 0x7f556006f640 2022-06-27T12:18:14.911 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.880+0000 7f5568165700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f556006ea90 con 0x7f55600c39e0 2022-06-27T12:18:14.912 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.880+0000 7f5565f01700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f556006f640 0x7f5560156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:14.913 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.881+0000 7f5565f01700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f556006f640 0x7f5560156ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:52782/0 (socket says 172.21.15.195:52782) 2022-06-27T12:18:14.913 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.881+0000 7f5565f01700 1 -- 172.21.15.195:0/1483242144 learned_addr learned my addr 172.21.15.195:0/1483242144 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:18:14.914 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.881+0000 7f5566702700 1 --2- 172.21.15.195:0/1483242144 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f55600c39e0 0x7f55601535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:14.914 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.881+0000 7f5565f01700 1 -- 172.21.15.195:0/1483242144 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f55600c39e0 msgr2=0x7f55601535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:18:14.918 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.881+0000 7f5565f01700 1 --2- 172.21.15.195:0/1483242144 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f55600c39e0 0x7f55601535a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:14.918 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.881+0000 7f5565f01700 1 -- 172.21.15.195:0/1483242144 >> v1:172.21.15.202:6789/0 conn(0x7f55600c4160 legacy=0x7f55600c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:18:14.919 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.881+0000 7f5565f01700 1 -- 172.21.15.195:0/1483242144 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f556006ed70 con 0x7f556006f640 2022-06-27T12:18:14.919 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.881+0000 7f5565f01700 1 --2- 172.21.15.195:0/1483242144 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f556006f640 0x7f5560156ed0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f5550009860 tx=0x7f5550009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18675ec0735e3fd4 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:18:14.920 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.881+0000 7f5564eff700 1 -- 172.21.15.195:0/1483242144 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5550004030 con 0x7f556006f640 2022-06-27T12:18:14.921 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.881+0000 7f5564eff700 1 -- 172.21.15.195:0/1483242144 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5550004190 con 0x7f556006f640 2022-06-27T12:18:14.921 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.881+0000 7f5564eff700 1 -- 172.21.15.195:0/1483242144 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f555000b6d0 con 0x7f556006f640 2022-06-27T12:18:14.922 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.882+0000 7f5568165700 1 -- 172.21.15.195:0/1483242144 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f556006f640 msgr2=0x7f5560156ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:14.922 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.882+0000 7f5568165700 1 --2- 172.21.15.195:0/1483242144 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f556006f640 0x7f5560156ed0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f5550009860 tx=0x7f5550009120 comp rx=0 tx=0).stop 2022-06-27T12:18:14.923 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.882+0000 7f5568165700 1 -- 172.21.15.195:0/1483242144 shutdown_connections 2022-06-27T12:18:14.923 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.882+0000 7f5568165700 1 --2- 172.21.15.195:0/1483242144 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f55600c39e0 0x7f55601535a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:14.924 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.882+0000 7f5568165700 1 --2- 172.21.15.195:0/1483242144 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f556006f640 0x7f5560156ed0 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:14.924 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.882+0000 7f5568165700 1 -- 172.21.15.195:0/1483242144 >> 172.21.15.195:0/1483242144 conn(0x7f55600c6350 msgr2=0x7f55600c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:18:14.925 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.882+0000 7f5568165700 1 -- 172.21.15.195:0/1483242144 shutdown_connections 2022-06-27T12:18:14.925 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.882+0000 7f5568165700 1 -- 172.21.15.195:0/1483242144 wait complete. 2022-06-27T12:18:14.925 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.883+0000 7f5568165700 1 Processor -- start 2022-06-27T12:18:14.926 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.883+0000 7f5568165700 1 -- start start 2022-06-27T12:18:14.926 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.884+0000 7f5568165700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f556006f640 0x7f556006c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:14.927 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.884+0000 7f5568165700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f55600c39e0 0x7f556016a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:14.928 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.884+0000 7f5568165700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f55600c4160 0x7f556016a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:14.928 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.884+0000 7f5568165700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5560166600 con 0x7f55600c39e0 2022-06-27T12:18:14.929 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.884+0000 7f5568165700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5560169bb0 con 0x7f556006f640 2022-06-27T12:18:14.930 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.884+0000 7f5568165700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55601667f0 con 0x7f55600c4160 2022-06-27T12:18:14.930 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.884+0000 7f5565700700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f55600c39e0 0x7f556016a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:14.931 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.884+0000 7f5565700700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f55600c39e0 0x7f556016a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:52790/0 (socket says 172.21.15.195:52790) 2022-06-27T12:18:14.931 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.884+0000 7f5565700700 1 -- 172.21.15.195:0/1667074461 learned_addr learned my addr 172.21.15.195:0/1667074461 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:18:14.932 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.884+0000 7f5565f01700 1 --2- 172.21.15.195:0/1667074461 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f556006f640 0x7f556006c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:14.932 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.884+0000 7f5566702700 1 --2- 172.21.15.195:0/1667074461 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f55600c4160 0x7f556016a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:14.933 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.884+0000 7f5565700700 1 -- 172.21.15.195:0/1667074461 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f55600c4160 msgr2=0x7f556016a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:14.933 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.884+0000 7f5565700700 1 --2- 172.21.15.195:0/1667074461 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f55600c4160 0x7f556016a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:14.934 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.884+0000 7f5565700700 1 -- 172.21.15.195:0/1667074461 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f556006f640 msgr2=0x7f556006c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:14.935 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.884+0000 7f5565700700 1 --2- 172.21.15.195:0/1667074461 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f556006f640 0x7f556006c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:14.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.884+0000 7f5565700700 1 -- 172.21.15.195:0/1667074461 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55500093a0 con 0x7f55600c39e0 2022-06-27T12:18:14.936 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.884+0000 7f5565f01700 1 --2- 172.21.15.195:0/1667074461 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f556006f640 0x7f556006c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T12:18:14.947 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.884+0000 7f5566702700 1 --2- 172.21.15.195:0/1667074461 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f55600c4160 0x7f556016a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T12:18:14.948 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.884+0000 7f5565700700 1 --2- 172.21.15.195:0/1667074461 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f55600c39e0 0x7f556016a040 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f555c00b470 tx=0x7f555c006f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:18:14.950 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.884+0000 7f554dffb700 1 -- 172.21.15.195:0/1667074461 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f555c011630 con 0x7f55600c39e0 2022-06-27T12:18:14.950 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.884+0000 7f554dffb700 1 -- 172.21.15.195:0/1667074461 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f555c0117d0 con 0x7f55600c39e0 2022-06-27T12:18:14.951 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.884+0000 7f5568165700 1 -- 172.21.15.195:0/1667074461 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5560166ad0 con 0x7f55600c39e0 2022-06-27T12:18:14.952 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.884+0000 7f554dffb700 1 -- 172.21.15.195:0/1667074461 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f555c011b90 con 0x7f55600c39e0 2022-06-27T12:18:14.952 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.885+0000 7f5568165700 1 -- 172.21.15.195:0/1667074461 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5560167090 con 0x7f55600c39e0 2022-06-27T12:18:14.953 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.885+0000 7f5568165700 1 -- 172.21.15.195:0/1667074461 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5560169d60 con 0x7f55600c39e0 2022-06-27T12:18:14.954 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.886+0000 7f554dffb700 1 -- 172.21.15.195:0/1667074461 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f555c00f050 con 0x7f55600c39e0 2022-06-27T12:18:14.954 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.886+0000 7f554dffb700 1 --2- 172.21.15.195:0/1667074461 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5540063d30 0x7f55400661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:14.954 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.886+0000 7f554dffb700 1 -- 172.21.15.195:0/1667074461 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7664+0+0 (secure 0 0 0) 0x7f555c080740 con 0x7f55600c39e0 2022-06-27T12:18:14.955 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.886+0000 7f5565f01700 1 --2- 172.21.15.195:0/1667074461 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5540063d30 0x7f55400661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:14.955 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.887+0000 7f5565f01700 1 --2- 172.21.15.195:0/1667074461 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5540063d30 0x7f55400661e0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f5550009860 tx=0x7f5550016000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:18:14.956 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:14.891+0000 7f554dffb700 1 -- 172.21.15.195:0/1667074461 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f5560169d60 con 0x7f55600c39e0 2022-06-27T12:18:15.119 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:15.117+0000 7f5568165700 1 -- 172.21.15.195:0/1667074461 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5560169d60 con 0x7f55600c39e0 2022-06-27T12:18:15.119 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:15.118+0000 7f554dffb700 1 -- 172.21.15.195:0/1667074461 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+6132 (secure 0 0 0) 0x7f5560169d60 con 0x7f55600c39e0 2022-06-27T12:18:15.120 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:18:15.120 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":113,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":7300,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/1965783412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1965783412},{"type":"v1","addr":"172.21.15.202:6833","nonce":1965783412}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":113}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:17:30.381992+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7165},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7165":{"gid":7165,"name":"d","rank":0,"incarnation":108,"state":"up:active","state_seq":27,"addr":"172.21.15.202:6835/3551478822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":3551478822},{"type":"v1","addr":"172.21.15.202:6835","nonce":3551478822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:18:15.122 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:15.120+0000 7f5568165700 1 -- 172.21.15.195:0/1667074461 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5540063d30 msgr2=0x7f55400661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:15.122 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:15.120+0000 7f5568165700 1 --2- 172.21.15.195:0/1667074461 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5540063d30 0x7f55400661e0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f5550009860 tx=0x7f5550016000 comp rx=0 tx=0).stop 2022-06-27T12:18:15.123 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:15.120+0000 7f5568165700 1 -- 172.21.15.195:0/1667074461 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f55600c39e0 msgr2=0x7f556016a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:15.123 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:15.120+0000 7f5568165700 1 --2- 172.21.15.195:0/1667074461 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f55600c39e0 0x7f556016a040 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f555c00b470 tx=0x7f555c006f60 comp rx=0 tx=0).stop 2022-06-27T12:18:15.124 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:15.120+0000 7f5568165700 1 -- 172.21.15.195:0/1667074461 shutdown_connections 2022-06-27T12:18:15.124 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:15.120+0000 7f5568165700 1 --2- 172.21.15.195:0/1667074461 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f5540063d30 0x7f55400661e0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:15.124 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:15.120+0000 7f5568165700 1 --2- 172.21.15.195:0/1667074461 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f55600c4160 0x7f556016a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:15.125 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:15.120+0000 7f5568165700 1 --2- 172.21.15.195:0/1667074461 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f556006f640 0x7f556006c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:15.125 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:15.120+0000 7f5568165700 1 --2- 172.21.15.195:0/1667074461 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f55600c39e0 0x7f556016a040 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:15.127 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:15.120+0000 7f5568165700 1 -- 172.21.15.195:0/1667074461 >> 172.21.15.195:0/1667074461 conn(0x7f55600c6350 msgr2=0x7f556014bf40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:18:15.127 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:15.120+0000 7f5568165700 1 -- 172.21.15.195:0/1667074461 shutdown_connections 2022-06-27T12:18:15.128 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:15.120+0000 7f5568165700 1 -- 172.21.15.195:0/1667074461 wait complete. 2022-06-27T12:18:15.128 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 113 2022-06-27T12:18:15.134 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-06-27T12:18:17.135 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:18:17.284 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.265+0000 7fe58f8dd700 1 Processor -- start 2022-06-27T12:18:17.285 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.266+0000 7fe58f8dd700 1 -- start start 2022-06-27T12:18:17.285 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.266+0000 7fe58f8dd700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fe58806e850 0x7fe5881570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:17.287 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.266+0000 7fe58f8dd700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fe58806fef0 0x7fe588069de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:17.287 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.266+0000 7fe58f8dd700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe58806fba0 con 0x7fe5880c34a0 2022-06-27T12:18:17.288 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.266+0000 7fe58f8dd700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5881575e0 con 0x7fe58806e850 2022-06-27T12:18:17.288 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.266+0000 7fe58f8dd700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe58806f5e0 con 0x7fe58806fef0 2022-06-27T12:18:17.289 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.267+0000 7fe58d679700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fe58806e850 0x7fe5881570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:17.289 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.267+0000 7fe58de7a700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fe5880c34a0 0x7fe5880c2920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:58992/0 (socket says 172.21.15.195:58992) 2022-06-27T12:18:17.289 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.267+0000 7fe58de7a700 1 -- 172.21.15.195:0/3620714057 learned_addr learned my addr 172.21.15.195:0/3620714057 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:18:17.290 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.267+0000 7fe58ce78700 1 --2- 172.21.15.195:0/3620714057 >> v2:172.21.15.202:3300/0 conn(0x7fe58806fef0 0x7fe588069de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:17.290 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.267+0000 7fe58d679700 1 -- 172.21.15.195:0/3620714057 >> v2:172.21.15.202:3300/0 conn(0x7fe58806fef0 msgr2=0x7fe588069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:18:17.291 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.267+0000 7fe58d679700 1 --2- 172.21.15.195:0/3620714057 >> v2:172.21.15.202:3300/0 conn(0x7fe58806fef0 0x7fe588069de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:17.291 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.267+0000 7fe58d679700 1 -- 172.21.15.195:0/3620714057 >> v1:172.21.15.195:6789/0 conn(0x7fe5880c34a0 legacy=0x7fe5880c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:17.292 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.267+0000 7fe58d679700 1 -- 172.21.15.195:0/3620714057 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe58806f8c0 con 0x7fe58806e850 2022-06-27T12:18:17.292 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.267+0000 7fe58d679700 1 --2- 172.21.15.195:0/3620714057 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe58806e850 0x7fe5881570a0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7fe58806e600 tx=0x7fe578009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57f9724a1398483a server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:18:17.293 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.267+0000 7fe577fff700 1 -- 172.21.15.195:0/3620714057 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe578011630 con 0x7fe58806e850 2022-06-27T12:18:17.294 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.267+0000 7fe577fff700 1 -- 172.21.15.195:0/3620714057 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe578011790 con 0x7fe58806e850 2022-06-27T12:18:17.295 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.267+0000 7fe577fff700 1 -- 172.21.15.195:0/3620714057 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe578011b30 con 0x7fe58806e850 2022-06-27T12:18:17.295 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.268+0000 7fe58f8dd700 1 -- 172.21.15.195:0/3620714057 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe58806e850 msgr2=0x7fe5881570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:17.295 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.268+0000 7fe58f8dd700 1 --2- 172.21.15.195:0/3620714057 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe58806e850 0x7fe5881570a0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7fe58806e600 tx=0x7fe578009060 comp rx=0 tx=0).stop 2022-06-27T12:18:17.296 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.268+0000 7fe58f8dd700 1 -- 172.21.15.195:0/3620714057 shutdown_connections 2022-06-27T12:18:17.296 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.268+0000 7fe58f8dd700 1 --2- 172.21.15.195:0/3620714057 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe58806e850 0x7fe5881570a0 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:17.296 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.268+0000 7fe58f8dd700 1 --2- 172.21.15.195:0/3620714057 >> v2:172.21.15.202:3300/0 conn(0x7fe58806fef0 0x7fe588069de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:17.296 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.268+0000 7fe58f8dd700 1 -- 172.21.15.195:0/3620714057 >> 172.21.15.195:0/3620714057 conn(0x7fe5880c6350 msgr2=0x7fe5880c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:18:17.297 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.268+0000 7fe58f8dd700 1 -- 172.21.15.195:0/3620714057 shutdown_connections 2022-06-27T12:18:17.297 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.268+0000 7fe58f8dd700 1 -- 172.21.15.195:0/3620714057 wait complete. 2022-06-27T12:18:17.297 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.269+0000 7fe58f8dd700 1 Processor -- start 2022-06-27T12:18:17.297 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.269+0000 7fe58f8dd700 1 -- start start 2022-06-27T12:18:17.298 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.270+0000 7fe58f8dd700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe58806e850 0x7fe588149f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:17.298 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.270+0000 7fe58f8dd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe5880c34a0 0x7fe588154b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:17.298 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.270+0000 7fe58f8dd700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe588143120 0x7fe588155080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:17.298 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.270+0000 7fe58f8dd700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5881415e0 con 0x7fe5880c34a0 2022-06-27T12:18:17.298 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.270+0000 7fe58f8dd700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe588147bb0 con 0x7fe58806e850 2022-06-27T12:18:17.299 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.270+0000 7fe58f8dd700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe588147e90 con 0x7fe588143120 2022-06-27T12:18:17.299 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.270+0000 7fe58ce78700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe5880c34a0 0x7fe588154b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:17.299 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.270+0000 7fe58ce78700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe5880c34a0 0x7fe588154b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:52812/0 (socket says 172.21.15.195:52812) 2022-06-27T12:18:17.299 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.270+0000 7fe58ce78700 1 -- 172.21.15.195:0/1235523409 learned_addr learned my addr 172.21.15.195:0/1235523409 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:18:17.300 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.270+0000 7fe58d679700 1 --2- 172.21.15.195:0/1235523409 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe58806e850 0x7fe588149f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:17.300 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.270+0000 7fe58de7a700 1 --2- 172.21.15.195:0/1235523409 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe588143120 0x7fe588155080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:17.300 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.270+0000 7fe58ce78700 1 -- 172.21.15.195:0/1235523409 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe588143120 msgr2=0x7fe588155080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:17.300 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.270+0000 7fe58ce78700 1 --2- 172.21.15.195:0/1235523409 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe588143120 0x7fe588155080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:17.301 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.270+0000 7fe58ce78700 1 -- 172.21.15.195:0/1235523409 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe58806e850 msgr2=0x7fe588149f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:17.301 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.270+0000 7fe58ce78700 1 --2- 172.21.15.195:0/1235523409 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe58806e850 0x7fe588149f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:17.301 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.270+0000 7fe58ce78700 1 -- 172.21.15.195:0/1235523409 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5780092e0 con 0x7fe5880c34a0 2022-06-27T12:18:17.302 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.270+0000 7fe58de7a700 1 --2- 172.21.15.195:0/1235523409 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe588143120 0x7fe588155080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T12:18:17.302 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.270+0000 7fe58ce78700 1 --2- 172.21.15.195:0/1235523409 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe5880c34a0 0x7fe588154b40 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7fe58400d890 tx=0x7fe584008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:18:17.302 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.270+0000 7fe5757fa700 1 -- 172.21.15.195:0/1235523409 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe584014620 con 0x7fe5880c34a0 2022-06-27T12:18:17.303 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.270+0000 7fe5757fa700 1 -- 172.21.15.195:0/1235523409 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe584014780 con 0x7fe5880c34a0 2022-06-27T12:18:17.303 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.270+0000 7fe58f8dd700 1 -- 172.21.15.195:0/1235523409 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe588148170 con 0x7fe5880c34a0 2022-06-27T12:18:17.303 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.270+0000 7fe5757fa700 1 -- 172.21.15.195:0/1235523409 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe584014b20 con 0x7fe5880c34a0 2022-06-27T12:18:17.303 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.271+0000 7fe58f8dd700 1 -- 172.21.15.195:0/1235523409 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe588148730 con 0x7fe5880c34a0 2022-06-27T12:18:17.304 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.271+0000 7fe58f8dd700 1 -- 172.21.15.195:0/1235523409 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5880722c0 con 0x7fe5880c34a0 2022-06-27T12:18:17.304 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.272+0000 7fe5757fa700 1 -- 172.21.15.195:0/1235523409 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fe584024070 con 0x7fe5880c34a0 2022-06-27T12:18:17.304 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.272+0000 7fe5757fa700 1 --2- 172.21.15.195:0/1235523409 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe568063d00 0x7fe5680661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:17.304 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.272+0000 7fe5757fa700 1 -- 172.21.15.195:0/1235523409 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7664+0+0 (secure 0 0 0) 0x7fe584081520 con 0x7fe5880c34a0 2022-06-27T12:18:17.305 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.272+0000 7fe58d679700 1 --2- 172.21.15.195:0/1235523409 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe568063d00 0x7fe5680661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:17.305 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.273+0000 7fe58d679700 1 --2- 172.21.15.195:0/1235523409 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe568063d00 0x7fe5680661b0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fe578011fd0 tx=0x7fe578017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:18:17.305 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.277+0000 7fe5757fa700 1 -- 172.21.15.195:0/1235523409 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fe58404fa70 con 0x7fe5880c34a0 2022-06-27T12:18:17.516 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.511+0000 7fe58f8dd700 1 -- 172.21.15.195:0/1235523409 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe58806f240 con 0x7fe5880c34a0 2022-06-27T12:18:17.517 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.511+0000 7fe5757fa700 1 -- 172.21.15.195:0/1235523409 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+6132 (secure 0 0 0) 0x7fe584053090 con 0x7fe5880c34a0 2022-06-27T12:18:17.518 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:18:17.518 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":113,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":7300,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/1965783412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1965783412},{"type":"v1","addr":"172.21.15.202:6833","nonce":1965783412}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":113}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:17:30.381992+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7165},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7165":{"gid":7165,"name":"d","rank":0,"incarnation":108,"state":"up:active","state_seq":27,"addr":"172.21.15.202:6835/3551478822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":3551478822},{"type":"v1","addr":"172.21.15.202:6835","nonce":3551478822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:18:17.520 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.513+0000 7fe58f8dd700 1 -- 172.21.15.195:0/1235523409 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe568063d00 msgr2=0x7fe5680661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:17.521 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.513+0000 7fe58f8dd700 1 --2- 172.21.15.195:0/1235523409 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe568063d00 0x7fe5680661b0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fe578011fd0 tx=0x7fe578017000 comp rx=0 tx=0).stop 2022-06-27T12:18:17.521 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.513+0000 7fe58f8dd700 1 -- 172.21.15.195:0/1235523409 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe5880c34a0 msgr2=0x7fe588154b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:17.521 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.513+0000 7fe58f8dd700 1 --2- 172.21.15.195:0/1235523409 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe5880c34a0 0x7fe588154b40 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7fe58400d890 tx=0x7fe584008040 comp rx=0 tx=0).stop 2022-06-27T12:18:17.522 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.513+0000 7fe58f8dd700 1 -- 172.21.15.195:0/1235523409 shutdown_connections 2022-06-27T12:18:17.522 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.513+0000 7fe58f8dd700 1 --2- 172.21.15.195:0/1235523409 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe568063d00 0x7fe5680661b0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:17.527 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.513+0000 7fe58f8dd700 1 --2- 172.21.15.195:0/1235523409 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe588143120 0x7fe588155080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:17.527 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.513+0000 7fe58f8dd700 1 --2- 172.21.15.195:0/1235523409 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe58806e850 0x7fe588149f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:17.528 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.513+0000 7fe58f8dd700 1 --2- 172.21.15.195:0/1235523409 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe5880c34a0 0x7fe588154b40 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:17.528 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.513+0000 7fe58f8dd700 1 -- 172.21.15.195:0/1235523409 >> 172.21.15.195:0/1235523409 conn(0x7fe5880c6350 msgr2=0x7fe588067ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:18:17.529 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.513+0000 7fe58f8dd700 1 -- 172.21.15.195:0/1235523409 shutdown_connections 2022-06-27T12:18:17.529 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:17.513+0000 7fe58f8dd700 1 -- 172.21.15.195:0/1235523409 wait complete. 2022-06-27T12:18:17.529 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 113 2022-06-27T12:18:17.531 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-06-27T12:18:19.533 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:18:19.666 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.666+0000 7f0662f90700 1 Processor -- start 2022-06-27T12:18:19.666 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.666+0000 7f0662f90700 1 -- start start 2022-06-27T12:18:19.671 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.667+0000 7f0662f90700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f065c06e850 0x7f065c1570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:19.672 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.667+0000 7f0662f90700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f065c06fef0 0x7f065c069de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:19.672 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.667+0000 7f0662f90700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f065c0c34a0 0x7f065c06a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:19.674 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.667+0000 7f0662f90700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f065c1575e0 con 0x7f065c0c34a0 2022-06-27T12:18:19.674 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.667+0000 7f0662f90700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f065c06fba0 con 0x7f065c06fef0 2022-06-27T12:18:19.675 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.667+0000 7f0662f90700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f065c06f5e0 con 0x7f065c06e850 2022-06-27T12:18:19.675 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.667+0000 7f066152d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f065c0c34a0 0x7f065c06a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:19.675 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.667+0000 7f066152d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f065c0c34a0 0x7f065c06a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:52822/0 (socket says 172.21.15.195:52822) 2022-06-27T12:18:19.676 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.667+0000 7f066152d700 1 -- 172.21.15.195:0/3456969034 learned_addr learned my addr 172.21.15.195:0/3456969034 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:18:19.677 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.667+0000 7f0660d2c700 1 --2- 172.21.15.195:0/3456969034 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f065c06e850 0x7f065c1570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:19.677 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.667+0000 7f065bfff700 1 --2- 172.21.15.195:0/3456969034 >> v2:172.21.15.202:3300/0 conn(0x7f065c06fef0 0x7f065c069de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:19.678 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.667+0000 7f066152d700 1 -- 172.21.15.195:0/3456969034 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f065c06e850 msgr2=0x7f065c1570a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:18:19.678 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.667+0000 7f066152d700 1 --2- 172.21.15.195:0/3456969034 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f065c06e850 0x7f065c1570a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:19.678 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.667+0000 7f066152d700 1 -- 172.21.15.195:0/3456969034 >> v2:172.21.15.202:3300/0 conn(0x7f065c06fef0 msgr2=0x7f065c069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:18:19.679 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.667+0000 7f066152d700 1 --2- 172.21.15.195:0/3456969034 >> v2:172.21.15.202:3300/0 conn(0x7f065c06fef0 0x7f065c069de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:19.680 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.667+0000 7f066152d700 1 -- 172.21.15.195:0/3456969034 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f065c06f8c0 con 0x7f065c0c34a0 2022-06-27T12:18:19.680 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.667+0000 7f066152d700 1 --2- 172.21.15.195:0/3456969034 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f065c0c34a0 0x7f065c06a320 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f065c06e240 tx=0x7f0654009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9197c176be1a688 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:18:19.681 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.667+0000 7f065b7fe700 1 -- 172.21.15.195:0/3456969034 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f065400e070 con 0x7f065c0c34a0 2022-06-27T12:18:19.681 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.667+0000 7f065b7fe700 1 -- 172.21.15.195:0/3456969034 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f065400bae0 con 0x7f065c0c34a0 2022-06-27T12:18:19.682 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.667+0000 7f065b7fe700 1 -- 172.21.15.195:0/3456969034 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06540092e0 con 0x7f065c0c34a0 2022-06-27T12:18:19.682 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.668+0000 7f0662f90700 1 -- 172.21.15.195:0/3456969034 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f065c0c34a0 msgr2=0x7f065c06a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:19.686 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.668+0000 7f0662f90700 1 --2- 172.21.15.195:0/3456969034 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f065c0c34a0 0x7f065c06a320 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f065c06e240 tx=0x7f0654009840 comp rx=0 tx=0).stop 2022-06-27T12:18:19.686 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.668+0000 7f0662f90700 1 -- 172.21.15.195:0/3456969034 shutdown_connections 2022-06-27T12:18:19.687 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.668+0000 7f0662f90700 1 --2- 172.21.15.195:0/3456969034 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f065c0c34a0 0x7f065c06a320 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:19.687 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.668+0000 7f0662f90700 1 --2- 172.21.15.195:0/3456969034 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f065c06e850 0x7f065c1570a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:19.688 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.668+0000 7f0662f90700 1 --2- 172.21.15.195:0/3456969034 >> v2:172.21.15.202:3300/0 conn(0x7f065c06fef0 0x7f065c069de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:19.688 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.668+0000 7f0662f90700 1 -- 172.21.15.195:0/3456969034 >> 172.21.15.195:0/3456969034 conn(0x7f065c0c6350 msgr2=0x7f065c0c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:18:19.689 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.668+0000 7f0662f90700 1 -- 172.21.15.195:0/3456969034 shutdown_connections 2022-06-27T12:18:19.689 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.668+0000 7f0662f90700 1 -- 172.21.15.195:0/3456969034 wait complete. 2022-06-27T12:18:19.690 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.669+0000 7f0662f90700 1 Processor -- start 2022-06-27T12:18:19.690 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.669+0000 7f0662f90700 1 -- start start 2022-06-27T12:18:19.692 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.670+0000 7f0662f90700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f065c06e850 0x7f065c1638a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:19.692 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.670+0000 7f0662f90700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f065c0c34a0 0x7f065c14e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:19.693 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.670+0000 7f0662f90700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f065c14aea0 0x7f065c148d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:19.693 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.670+0000 7f0662f90700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f065c149e30 con 0x7f065c14aea0 2022-06-27T12:18:19.694 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.670+0000 7f0662f90700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f065c14e090 con 0x7f065c06e850 2022-06-27T12:18:19.702 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.670+0000 7f0662f90700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f065c070540 con 0x7f065c0c34a0 2022-06-27T12:18:19.702 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.670+0000 7f066152d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f065c14aea0 0x7f065c148d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:19.703 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.670+0000 7f066152d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f065c14aea0 0x7f065c148d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:52830/0 (socket says 172.21.15.195:52830) 2022-06-27T12:18:19.703 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.670+0000 7f066152d700 1 -- 172.21.15.195:0/3718156380 learned_addr learned my addr 172.21.15.195:0/3718156380 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:18:19.703 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.670+0000 7f0660d2c700 1 --2- 172.21.15.195:0/3718156380 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f065c06e850 0x7f065c1638a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:19.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.670+0000 7f065bfff700 1 --2- 172.21.15.195:0/3718156380 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f065c0c34a0 0x7f065c14e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:19.704 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.670+0000 7f066152d700 1 -- 172.21.15.195:0/3718156380 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f065c0c34a0 msgr2=0x7f065c14e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:19.705 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.670+0000 7f066152d700 1 --2- 172.21.15.195:0/3718156380 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f065c0c34a0 0x7f065c14e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:19.705 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.670+0000 7f066152d700 1 -- 172.21.15.195:0/3718156380 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f065c06e850 msgr2=0x7f065c1638a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:19.706 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.670+0000 7f066152d700 1 --2- 172.21.15.195:0/3718156380 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f065c06e850 0x7f065c1638a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:19.706 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.670+0000 7f066152d700 1 -- 172.21.15.195:0/3718156380 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0654009ae0 con 0x7f065c14aea0 2022-06-27T12:18:19.706 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.671+0000 7f066152d700 1 --2- 172.21.15.195:0/3718156380 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f065c14aea0 0x7f065c148d40 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f0654009fd0 tx=0x7f0654018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:18:19.707 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.671+0000 7f0658ff9700 1 -- 172.21.15.195:0/3718156380 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0654021070 con 0x7f065c14aea0 2022-06-27T12:18:19.707 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.671+0000 7f0658ff9700 1 -- 172.21.15.195:0/3718156380 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0654009960 con 0x7f065c14aea0 2022-06-27T12:18:19.708 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.671+0000 7f0658ff9700 1 -- 172.21.15.195:0/3718156380 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0654007840 con 0x7f065c14aea0 2022-06-27T12:18:19.708 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.671+0000 7f0662f90700 1 -- 172.21.15.195:0/3718156380 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f065c14cfd0 con 0x7f065c14aea0 2022-06-27T12:18:19.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.671+0000 7f0662f90700 1 -- 172.21.15.195:0/3718156380 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f065c14d590 con 0x7f065c14aea0 2022-06-27T12:18:19.709 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.672+0000 7f0662f90700 1 -- 172.21.15.195:0/3718156380 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f065c14e200 con 0x7f065c14aea0 2022-06-27T12:18:19.728 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.672+0000 7f0658ff9700 1 -- 172.21.15.195:0/3718156380 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f0654007c30 con 0x7f065c14aea0 2022-06-27T12:18:19.728 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.673+0000 7f0658ff9700 1 --2- 172.21.15.195:0/3718156380 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0644063d00 0x7f06440661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:19.729 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.673+0000 7f0658ff9700 1 -- 172.21.15.195:0/3718156380 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7664+0+0 (secure 0 0 0) 0x7f06540845f0 con 0x7f065c14aea0 2022-06-27T12:18:19.729 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.673+0000 7f0660d2c700 1 --2- 172.21.15.195:0/3718156380 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0644063d00 0x7f06440661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:19.730 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.673+0000 7f0660d2c700 1 --2- 172.21.15.195:0/3718156380 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0644063d00 0x7f06440661b0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f0650006fd0 tx=0x7f0650008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:18:19.730 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.676+0000 7f0658ff9700 1 -- 172.21.15.195:0/3718156380 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f0654052c60 con 0x7f065c14aea0 2022-06-27T12:18:19.906 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.903+0000 7f0662f90700 1 -- 172.21.15.195:0/3718156380 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f065c0722c0 con 0x7f065c14aea0 2022-06-27T12:18:19.907 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.904+0000 7f0658ff9700 1 -- 172.21.15.195:0/3718156380 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+6132 (secure 0 0 0) 0x7f0654033020 con 0x7f065c14aea0 2022-06-27T12:18:19.908 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:18:19.908 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":113,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":7300,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/1965783412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1965783412},{"type":"v1","addr":"172.21.15.202:6833","nonce":1965783412}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":113}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:17:30.381992+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7165},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7165":{"gid":7165,"name":"d","rank":0,"incarnation":108,"state":"up:active","state_seq":27,"addr":"172.21.15.202:6835/3551478822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":3551478822},{"type":"v1","addr":"172.21.15.202:6835","nonce":3551478822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:18:19.911 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.906+0000 7f0662f90700 1 -- 172.21.15.195:0/3718156380 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0644063d00 msgr2=0x7f06440661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:19.913 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.906+0000 7f0662f90700 1 --2- 172.21.15.195:0/3718156380 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0644063d00 0x7f06440661b0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f0650006fd0 tx=0x7f0650008040 comp rx=0 tx=0).stop 2022-06-27T12:18:19.913 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.906+0000 7f0662f90700 1 -- 172.21.15.195:0/3718156380 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f065c14aea0 msgr2=0x7f065c148d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:19.914 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.906+0000 7f0662f90700 1 --2- 172.21.15.195:0/3718156380 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f065c14aea0 0x7f065c148d40 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f0654009fd0 tx=0x7f0654018000 comp rx=0 tx=0).stop 2022-06-27T12:18:19.915 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.906+0000 7f0662f90700 1 -- 172.21.15.195:0/3718156380 shutdown_connections 2022-06-27T12:18:19.916 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.906+0000 7f0662f90700 1 --2- 172.21.15.195:0/3718156380 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f0644063d00 0x7f06440661b0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:19.916 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.906+0000 7f0662f90700 1 --2- 172.21.15.195:0/3718156380 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f065c14aea0 0x7f065c148d40 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:19.916 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.906+0000 7f0662f90700 1 --2- 172.21.15.195:0/3718156380 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f065c06e850 0x7f065c1638a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:19.917 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.906+0000 7f0662f90700 1 --2- 172.21.15.195:0/3718156380 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f065c0c34a0 0x7f065c14e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:19.917 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.906+0000 7f0662f90700 1 -- 172.21.15.195:0/3718156380 >> 172.21.15.195:0/3718156380 conn(0x7f065c0c6350 msgr2=0x7f065c150660 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:18:19.917 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.906+0000 7f0662f90700 1 -- 172.21.15.195:0/3718156380 shutdown_connections 2022-06-27T12:18:19.918 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:19.906+0000 7f0662f90700 1 -- 172.21.15.195:0/3718156380 wait complete. 2022-06-27T12:18:19.919 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 113 2022-06-27T12:18:19.922 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-06-27T12:18:21.927 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:18:22.060 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.061+0000 7fd173885700 1 Processor -- start 2022-06-27T12:18:22.060 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.061+0000 7fd173885700 1 -- start start 2022-06-27T12:18:22.062 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.062+0000 7fd173885700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fd16c06f640 0x7fd16c156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:22.063 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.062+0000 7fd173885700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd16c0c4160 0x7fd16c0c1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:22.063 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.062+0000 7fd173885700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd16c0c39e0 0x7fd16c1535a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:22.063 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.062+0000 7fd173885700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd16c0c1a20 con 0x7fd16c0c4160 2022-06-27T12:18:22.064 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.062+0000 7fd173885700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd16c0c1130 con 0x7fd16c06f640 2022-06-27T12:18:22.064 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.062+0000 7fd173885700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd16c06ea90 con 0x7fd16c0c39e0 2022-06-27T12:18:22.064 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.062+0000 7fd170e20700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd16c0c4160 0x7fd16c0c1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:22.065 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.062+0000 7fd170e20700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd16c0c4160 0x7fd16c0c1eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:52838/0 (socket says 172.21.15.195:52838) 2022-06-27T12:18:22.065 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.062+0000 7fd170e20700 1 -- 172.21.15.195:0/2522171206 learned_addr learned my addr 172.21.15.195:0/2522171206 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:18:22.065 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.062+0000 7fd171e22700 1 --2- 172.21.15.195:0/2522171206 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd16c0c39e0 0x7fd16c1535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:22.065 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.062+0000 7fd171621700 1 --2- 172.21.15.195:0/2522171206 >> v2:172.21.15.202:3300/0 conn(0x7fd16c06f640 0x7fd16c156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:22.073 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.062+0000 7fd170e20700 1 -- 172.21.15.195:0/2522171206 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd16c0c39e0 msgr2=0x7fd16c1535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:18:22.073 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.062+0000 7fd170e20700 1 --2- 172.21.15.195:0/2522171206 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd16c0c39e0 0x7fd16c1535a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:22.074 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.062+0000 7fd170e20700 1 -- 172.21.15.195:0/2522171206 >> v2:172.21.15.202:3300/0 conn(0x7fd16c06f640 msgr2=0x7fd16c156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:18:22.074 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.062+0000 7fd170e20700 1 --2- 172.21.15.195:0/2522171206 >> v2:172.21.15.202:3300/0 conn(0x7fd16c06f640 0x7fd16c156ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:22.075 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.062+0000 7fd170e20700 1 -- 172.21.15.195:0/2522171206 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd16c06ed70 con 0x7fd16c0c4160 2022-06-27T12:18:22.075 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.062+0000 7fd170e20700 1 --2- 172.21.15.195:0/2522171206 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd16c0c4160 0x7fd16c0c1eb0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7fd16c06db30 tx=0x7fd15c009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efad1398d912bd36 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:18:22.076 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.063+0000 7fd15bfff700 1 -- 172.21.15.195:0/2522171206 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd15c00b6d0 con 0x7fd16c0c4160 2022-06-27T12:18:22.076 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.063+0000 7fd15bfff700 1 -- 172.21.15.195:0/2522171206 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd15c00b830 con 0x7fd16c0c4160 2022-06-27T12:18:22.077 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.063+0000 7fd15bfff700 1 -- 172.21.15.195:0/2522171206 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd15c00bbd0 con 0x7fd16c0c4160 2022-06-27T12:18:22.077 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.063+0000 7fd173885700 1 -- 172.21.15.195:0/2522171206 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd16c0c4160 msgr2=0x7fd16c0c1eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:22.078 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.063+0000 7fd173885700 1 --2- 172.21.15.195:0/2522171206 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd16c0c4160 0x7fd16c0c1eb0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7fd16c06db30 tx=0x7fd15c009120 comp rx=0 tx=0).stop 2022-06-27T12:18:22.078 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.063+0000 7fd173885700 1 -- 172.21.15.195:0/2522171206 shutdown_connections 2022-06-27T12:18:22.079 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.063+0000 7fd173885700 1 --2- 172.21.15.195:0/2522171206 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd16c0c39e0 0x7fd16c1535a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:22.079 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.063+0000 7fd173885700 1 --2- 172.21.15.195:0/2522171206 >> v2:172.21.15.202:3300/0 conn(0x7fd16c06f640 0x7fd16c156ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:22.079 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.063+0000 7fd173885700 1 --2- 172.21.15.195:0/2522171206 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd16c0c4160 0x7fd16c0c1eb0 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:22.079 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.063+0000 7fd173885700 1 -- 172.21.15.195:0/2522171206 >> 172.21.15.195:0/2522171206 conn(0x7fd16c0c6350 msgr2=0x7fd16c0c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:18:22.080 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.063+0000 7fd173885700 1 -- 172.21.15.195:0/2522171206 shutdown_connections 2022-06-27T12:18:22.080 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.063+0000 7fd173885700 1 -- 172.21.15.195:0/2522171206 wait complete. 2022-06-27T12:18:22.081 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.064+0000 7fd173885700 1 Processor -- start 2022-06-27T12:18:22.081 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.064+0000 7fd173885700 1 -- start start 2022-06-27T12:18:22.081 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.065+0000 7fd173885700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd16c06f640 0x7fd16c1671e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:22.081 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.065+0000 7fd173885700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd16c0c39e0 0x7fd16c153550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:22.081 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.065+0000 7fd173885700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd16c0c4160 0x7fd16c0bb4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:22.082 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.065+0000 7fd173885700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd16c152f20 con 0x7fd16c0c4160 2022-06-27T12:18:22.082 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.065+0000 7fd173885700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd16c152030 con 0x7fd16c06f640 2022-06-27T12:18:22.084 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.065+0000 7fd173885700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd16c152310 con 0x7fd16c0c39e0 2022-06-27T12:18:22.084 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.065+0000 7fd171e22700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd16c0c4160 0x7fd16c0bb4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:22.084 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.065+0000 7fd171e22700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd16c0c4160 0x7fd16c0bb4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:52848/0 (socket says 172.21.15.195:52848) 2022-06-27T12:18:22.085 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.065+0000 7fd171e22700 1 -- 172.21.15.195:0/3165067833 learned_addr learned my addr 172.21.15.195:0/3165067833 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:18:22.085 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.066+0000 7fd171621700 1 --2- 172.21.15.195:0/3165067833 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd16c06f640 0x7fd16c1671e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:22.086 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.066+0000 7fd170e20700 1 --2- 172.21.15.195:0/3165067833 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd16c0c39e0 0x7fd16c153550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:22.086 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.066+0000 7fd171e22700 1 -- 172.21.15.195:0/3165067833 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd16c0c39e0 msgr2=0x7fd16c153550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:22.087 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.066+0000 7fd171e22700 1 --2- 172.21.15.195:0/3165067833 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd16c0c39e0 0x7fd16c153550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:22.087 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.066+0000 7fd171e22700 1 -- 172.21.15.195:0/3165067833 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd16c06f640 msgr2=0x7fd16c1671e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:22.088 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.066+0000 7fd171e22700 1 --2- 172.21.15.195:0/3165067833 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd16c06f640 0x7fd16c1671e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:22.088 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.066+0000 7fd171e22700 1 -- 172.21.15.195:0/3165067833 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd15c0093a0 con 0x7fd16c0c4160 2022-06-27T12:18:22.089 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.066+0000 7fd171e22700 1 --2- 172.21.15.195:0/3165067833 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd16c0c4160 0x7fd16c0bb4b0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7fd164009fa0 tx=0x7fd164009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:18:22.089 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.066+0000 7fd1597fa700 1 -- 172.21.15.195:0/3165067833 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd16400ebc0 con 0x7fd16c0c4160 2022-06-27T12:18:22.089 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.066+0000 7fd1597fa700 1 -- 172.21.15.195:0/3165067833 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd16400ed20 con 0x7fd16c0c4160 2022-06-27T12:18:22.090 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.066+0000 7fd173885700 1 -- 172.21.15.195:0/3165067833 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd16c1525f0 con 0x7fd16c0c4160 2022-06-27T12:18:22.090 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.066+0000 7fd1597fa700 1 -- 172.21.15.195:0/3165067833 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd164014670 con 0x7fd16c0c4160 2022-06-27T12:18:22.091 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.066+0000 7fd173885700 1 -- 172.21.15.195:0/3165067833 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd16c152bb0 con 0x7fd16c0c4160 2022-06-27T12:18:22.091 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.067+0000 7fd173885700 1 -- 172.21.15.195:0/3165067833 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd16c0bb320 con 0x7fd16c0c4160 2022-06-27T12:18:22.092 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.067+0000 7fd1597fa700 1 -- 172.21.15.195:0/3165067833 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fd164012050 con 0x7fd16c0c4160 2022-06-27T12:18:22.092 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.068+0000 7fd1597fa700 1 --2- 172.21.15.195:0/3165067833 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd14c063d00 0x7fd14c0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:22.093 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.068+0000 7fd1597fa700 1 -- 172.21.15.195:0/3165067833 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7664+0+0 (secure 0 0 0) 0x7fd1640813e0 con 0x7fd16c0c4160 2022-06-27T12:18:22.093 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.068+0000 7fd171621700 1 --2- 172.21.15.195:0/3165067833 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd14c063d00 0x7fd14c0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:22.094 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.068+0000 7fd171621700 1 --2- 172.21.15.195:0/3165067833 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd14c063d00 0x7fd14c0661b0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fd168004560 tx=0x7fd168008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:18:22.095 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.072+0000 7fd1597fa700 1 -- 172.21.15.195:0/3165067833 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fd16404f930 con 0x7fd16c0c4160 2022-06-27T12:18:22.300 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.301+0000 7fd173885700 1 -- 172.21.15.195:0/3165067833 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd16c0722c0 con 0x7fd16c0c4160 2022-06-27T12:18:22.301 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.301+0000 7fd1597fa700 1 -- 172.21.15.195:0/3165067833 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+6132 (secure 0 0 0) 0x7fd164052f50 con 0x7fd16c0c4160 2022-06-27T12:18:22.301 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:18:22.302 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":113,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":7300,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/1965783412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1965783412},{"type":"v1","addr":"172.21.15.202:6833","nonce":1965783412}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":113}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:17:30.381992+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7165},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7165":{"gid":7165,"name":"d","rank":0,"incarnation":108,"state":"up:active","state_seq":27,"addr":"172.21.15.202:6835/3551478822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":3551478822},{"type":"v1","addr":"172.21.15.202:6835","nonce":3551478822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:18:22.313 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.303+0000 7fd173885700 1 -- 172.21.15.195:0/3165067833 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd14c063d00 msgr2=0x7fd14c0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:22.313 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.303+0000 7fd173885700 1 --2- 172.21.15.195:0/3165067833 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd14c063d00 0x7fd14c0661b0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fd168004560 tx=0x7fd168008040 comp rx=0 tx=0).stop 2022-06-27T12:18:22.314 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.303+0000 7fd173885700 1 -- 172.21.15.195:0/3165067833 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd16c0c4160 msgr2=0x7fd16c0bb4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:22.314 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.303+0000 7fd173885700 1 --2- 172.21.15.195:0/3165067833 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd16c0c4160 0x7fd16c0bb4b0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7fd164009fa0 tx=0x7fd164009900 comp rx=0 tx=0).stop 2022-06-27T12:18:22.315 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.303+0000 7fd173885700 1 -- 172.21.15.195:0/3165067833 shutdown_connections 2022-06-27T12:18:22.315 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.303+0000 7fd173885700 1 --2- 172.21.15.195:0/3165067833 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fd14c063d00 0x7fd14c0661b0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:22.316 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.303+0000 7fd173885700 1 --2- 172.21.15.195:0/3165067833 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd16c0c4160 0x7fd16c0bb4b0 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:22.316 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.303+0000 7fd173885700 1 --2- 172.21.15.195:0/3165067833 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fd16c06f640 0x7fd16c1671e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:22.317 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.303+0000 7fd173885700 1 --2- 172.21.15.195:0/3165067833 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fd16c0c39e0 0x7fd16c153550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:22.317 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.303+0000 7fd173885700 1 -- 172.21.15.195:0/3165067833 >> 172.21.15.195:0/3165067833 conn(0x7fd16c0c6350 msgr2=0x7fd16c1506e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:18:22.318 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.303+0000 7fd173885700 1 -- 172.21.15.195:0/3165067833 shutdown_connections 2022-06-27T12:18:22.319 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:22.303+0000 7fd173885700 1 -- 172.21.15.195:0/3165067833 wait complete. 2022-06-27T12:18:22.319 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 113 2022-06-27T12:18:22.323 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-06-27T12:18:24.324 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:18:24.461 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.461+0000 7f39f7090700 1 Processor -- start 2022-06-27T12:18:24.465 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.461+0000 7f39f7090700 1 -- start start 2022-06-27T12:18:24.467 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.461+0000 7f39f7090700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f39f006b800 0x7f39f0067ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:24.468 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.461+0000 7f39f7090700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f39f006c400 0x7f39f00694d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:24.468 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.461+0000 7f39f7090700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39f00c5220 con 0x7f39f00c3840 2022-06-27T12:18:24.469 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.462+0000 7f39f7090700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39f006f680 con 0x7f39f006c400 2022-06-27T12:18:24.469 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.462+0000 7f39f7090700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39f00c5490 con 0x7f39f006b800 2022-06-27T12:18:24.470 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.462+0000 7f39effff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f39f006c400 0x7f39f00694d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:24.471 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.462+0000 7f39effff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f39f006c400 0x7f39f00694d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:52856/0 (socket says 172.21.15.195:52856) 2022-06-27T12:18:24.471 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.462+0000 7f39effff700 1 -- 172.21.15.195:0/978449072 learned_addr learned my addr 172.21.15.195:0/978449072 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:18:24.472 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.462+0000 7f39f4e2c700 1 --2- 172.21.15.195:0/978449072 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f39f006b800 0x7f39f0067ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:24.488 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.462+0000 7f39effff700 1 -- 172.21.15.195:0/978449072 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f39f006b800 msgr2=0x7f39f0067ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:18:24.488 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.462+0000 7f39effff700 1 --2- 172.21.15.195:0/978449072 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f39f006b800 0x7f39f0067ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:24.489 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.462+0000 7f39effff700 1 -- 172.21.15.195:0/978449072 >> v1:172.21.15.202:6789/0 conn(0x7f39f00c3840 legacy=0x7f39f0156cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:24.491 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.462+0000 7f39effff700 1 -- 172.21.15.195:0/978449072 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39f00c5770 con 0x7f39f006c400 2022-06-27T12:18:24.492 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.462+0000 7f39f4e2c700 1 --2- 172.21.15.195:0/978449072 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f39f006b800 0x7f39f0067ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:18:24.492 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.462+0000 7f39effff700 1 --2- 172.21.15.195:0/978449072 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f39f006c400 0x7f39f00694d0 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f39dc009860 tx=0x7f39dc009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acbed50c9631b97f server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:18:24.492 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.462+0000 7f39ef7fe700 1 -- 172.21.15.195:0/978449072 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39dc00b680 con 0x7f39f006c400 2022-06-27T12:18:24.493 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.462+0000 7f39ef7fe700 1 -- 172.21.15.195:0/978449072 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39dc00b7e0 con 0x7f39f006c400 2022-06-27T12:18:24.493 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.462+0000 7f39ef7fe700 1 -- 172.21.15.195:0/978449072 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39dc00bba0 con 0x7f39f006c400 2022-06-27T12:18:24.494 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.463+0000 7f39f7090700 1 -- 172.21.15.195:0/978449072 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f39f006c400 msgr2=0x7f39f00694d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:24.494 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.463+0000 7f39f7090700 1 --2- 172.21.15.195:0/978449072 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f39f006c400 0x7f39f00694d0 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f39dc009860 tx=0x7f39dc009120 comp rx=0 tx=0).stop 2022-06-27T12:18:24.495 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.463+0000 7f39f7090700 1 -- 172.21.15.195:0/978449072 shutdown_connections 2022-06-27T12:18:24.495 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.463+0000 7f39f7090700 1 --2- 172.21.15.195:0/978449072 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f39f006b800 0x7f39f0067ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:24.496 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.463+0000 7f39f7090700 1 --2- 172.21.15.195:0/978449072 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f39f006c400 0x7f39f00694d0 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:24.496 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.463+0000 7f39f7090700 1 -- 172.21.15.195:0/978449072 >> 172.21.15.195:0/978449072 conn(0x7f39f00c8f90 msgr2=0x7f39f00c9390 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:18:24.497 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.463+0000 7f39f7090700 1 -- 172.21.15.195:0/978449072 shutdown_connections 2022-06-27T12:18:24.497 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.463+0000 7f39f7090700 1 -- 172.21.15.195:0/978449072 wait complete. 2022-06-27T12:18:24.497 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.464+0000 7f39f7090700 1 Processor -- start 2022-06-27T12:18:24.498 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.464+0000 7f39f7090700 1 -- start start 2022-06-27T12:18:24.498 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.465+0000 7f39f7090700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f39f006b800 0x7f39f0154520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:24.499 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.465+0000 7f39f7090700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f39f006c400 0x7f39f0150fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:24.499 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.465+0000 7f39f7090700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f39f00c3840 0x7f39f01665c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:24.500 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.465+0000 7f39f7090700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39f016a5e0 con 0x7f39f00c3840 2022-06-27T12:18:24.501 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.465+0000 7f39f7090700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39f00c0700 con 0x7f39f006b800 2022-06-27T12:18:24.502 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.465+0000 7f39f7090700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39f00c0180 con 0x7f39f006c400 2022-06-27T12:18:24.503 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.465+0000 7f39f562d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f39f00c3840 0x7f39f01665c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:24.503 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.465+0000 7f39f562d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f39f00c3840 0x7f39f01665c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:52864/0 (socket says 172.21.15.195:52864) 2022-06-27T12:18:24.504 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.465+0000 7f39f562d700 1 -- 172.21.15.195:0/3793227786 learned_addr learned my addr 172.21.15.195:0/3793227786 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:18:24.504 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.465+0000 7f39effff700 1 --2- 172.21.15.195:0/3793227786 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f39f006c400 0x7f39f0150fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:24.505 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.465+0000 7f39f4e2c700 1 --2- 172.21.15.195:0/3793227786 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f39f006b800 0x7f39f0154520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:24.505 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.465+0000 7f39f562d700 1 -- 172.21.15.195:0/3793227786 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f39f006c400 msgr2=0x7f39f0150fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:24.505 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.465+0000 7f39f562d700 1 --2- 172.21.15.195:0/3793227786 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f39f006c400 0x7f39f0150fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:24.506 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.465+0000 7f39f562d700 1 -- 172.21.15.195:0/3793227786 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f39f006b800 msgr2=0x7f39f0154520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:24.507 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.465+0000 7f39f562d700 1 --2- 172.21.15.195:0/3793227786 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f39f006b800 0x7f39f0154520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:24.507 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.465+0000 7f39f562d700 1 -- 172.21.15.195:0/3793227786 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39dc0093a0 con 0x7f39f00c3840 2022-06-27T12:18:24.508 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.466+0000 7f39f562d700 1 --2- 172.21.15.195:0/3793227786 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f39f00c3840 0x7f39f01665c0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f39e800c7c0 tx=0x7f39e80015e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:18:24.508 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.466+0000 7f39ecff9700 1 -- 172.21.15.195:0/3793227786 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39e8003880 con 0x7f39f00c3840 2022-06-27T12:18:24.508 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.466+0000 7f39ecff9700 1 -- 172.21.15.195:0/3793227786 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39e80039e0 con 0x7f39f00c3840 2022-06-27T12:18:24.509 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.466+0000 7f39ecff9700 1 -- 172.21.15.195:0/3793227786 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39e8003d60 con 0x7f39f00c3840 2022-06-27T12:18:24.509 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.466+0000 7f39f7090700 1 -- 172.21.15.195:0/3793227786 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39f00c0420 con 0x7f39f00c3840 2022-06-27T12:18:24.510 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.466+0000 7f39f7090700 1 -- 172.21.15.195:0/3793227786 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39f0169f20 con 0x7f39f00c3840 2022-06-27T12:18:24.510 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.467+0000 7f39f7090700 1 -- 172.21.15.195:0/3793227786 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39f016a720 con 0x7f39f00c3840 2022-06-27T12:18:24.511 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.467+0000 7f39ecff9700 1 -- 172.21.15.195:0/3793227786 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f39e8010050 con 0x7f39f00c3840 2022-06-27T12:18:24.511 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.468+0000 7f39ecff9700 1 --2- 172.21.15.195:0/3793227786 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f39d8063d00 0x7f39d80661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:24.512 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.468+0000 7f39ecff9700 1 -- 172.21.15.195:0/3793227786 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7664+0+0 (secure 0 0 0) 0x7f39e8080920 con 0x7f39f00c3840 2022-06-27T12:18:24.512 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.468+0000 7f39f4e2c700 1 --2- 172.21.15.195:0/3793227786 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f39d8063d00 0x7f39d80661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:24.513 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.468+0000 7f39f4e2c700 1 --2- 172.21.15.195:0/3793227786 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f39d8063d00 0x7f39d80661b0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f39e4006fd0 tx=0x7f39e4008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:18:24.513 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.472+0000 7f39ecff9700 1 -- 172.21.15.195:0/3793227786 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f39f016a720 con 0x7f39f00c3840 2022-06-27T12:18:24.705 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.702+0000 7f39f7090700 1 -- 172.21.15.195:0/3793227786 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f39f016a720 con 0x7f39f00c3840 2022-06-27T12:18:24.707 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.703+0000 7f39ecff9700 1 -- 172.21.15.195:0/3793227786 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+6132 (secure 0 0 0) 0x7f39f016a720 con 0x7f39f00c3840 2022-06-27T12:18:24.708 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:18:24.709 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":113,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":7300,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/1965783412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1965783412},{"type":"v1","addr":"172.21.15.202:6833","nonce":1965783412}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":113}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:17:30.381992+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7165},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7165":{"gid":7165,"name":"d","rank":0,"incarnation":108,"state":"up:active","state_seq":27,"addr":"172.21.15.202:6835/3551478822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":3551478822},{"type":"v1","addr":"172.21.15.202:6835","nonce":3551478822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:18:24.711 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.705+0000 7f39f7090700 1 -- 172.21.15.195:0/3793227786 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f39d8063d00 msgr2=0x7f39d80661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:24.711 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.705+0000 7f39f7090700 1 --2- 172.21.15.195:0/3793227786 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f39d8063d00 0x7f39d80661b0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f39e4006fd0 tx=0x7f39e4008040 comp rx=0 tx=0).stop 2022-06-27T12:18:24.712 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.705+0000 7f39f7090700 1 -- 172.21.15.195:0/3793227786 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f39f00c3840 msgr2=0x7f39f01665c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:24.712 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.705+0000 7f39f7090700 1 --2- 172.21.15.195:0/3793227786 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f39f00c3840 0x7f39f01665c0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f39e800c7c0 tx=0x7f39e80015e0 comp rx=0 tx=0).stop 2022-06-27T12:18:24.713 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.705+0000 7f39f7090700 1 -- 172.21.15.195:0/3793227786 shutdown_connections 2022-06-27T12:18:24.713 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.705+0000 7f39f7090700 1 --2- 172.21.15.195:0/3793227786 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f39d8063d00 0x7f39d80661b0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:24.713 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.705+0000 7f39f7090700 1 --2- 172.21.15.195:0/3793227786 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f39f00c3840 0x7f39f01665c0 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:24.714 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.705+0000 7f39f7090700 1 --2- 172.21.15.195:0/3793227786 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f39f006b800 0x7f39f0154520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:24.714 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.705+0000 7f39f7090700 1 --2- 172.21.15.195:0/3793227786 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f39f006c400 0x7f39f0150fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:24.715 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.705+0000 7f39f7090700 1 -- 172.21.15.195:0/3793227786 >> 172.21.15.195:0/3793227786 conn(0x7f39f00c8f90 msgr2=0x7f39f0068450 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:18:24.715 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.705+0000 7f39f7090700 1 -- 172.21.15.195:0/3793227786 shutdown_connections 2022-06-27T12:18:24.715 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:24.705+0000 7f39f7090700 1 -- 172.21.15.195:0/3793227786 wait complete. 2022-06-27T12:18:24.716 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 113 2022-06-27T12:18:24.718 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-06-27T12:18:26.719 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:18:26.857 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:26.857+0000 7fec6422e700 1 Processor -- start 2022-06-27T12:18:26.857 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:26.857+0000 7fec6422e700 1 -- start start 2022-06-27T12:18:26.860 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:26.858+0000 7fec6422e700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fec5c150500 0x7fec5c063af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:26.861 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:26.858+0000 7fec6422e700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fec5c1508d0 0x7fec5c0bbf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:26.861 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:26.858+0000 7fec6422e700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec5c064030 con 0x7fec5c06a450 2022-06-27T12:18:26.861 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:26.858+0000 7fec6422e700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec5c0c4160 con 0x7fec5c150500 2022-06-27T12:18:26.862 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:26.858+0000 7fec6422e700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec5c0c4440 con 0x7fec5c1508d0 2022-06-27T12:18:26.862 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:26.858+0000 7fec61fca700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fec5c150500 0x7fec5c063af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:26.862 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:26.858+0000 7fec61fca700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fec5c150500 0x7fec5c063af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:52868/0 (socket says 172.21.15.195:52868) 2022-06-27T12:18:26.862 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:26.858+0000 7fec61fca700 1 -- 172.21.15.195:0/3320497018 learned_addr learned my addr 172.21.15.195:0/3320497018 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:18:26.863 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:26.858+0000 7fec627cb700 1 --2- 172.21.15.195:0/3320497018 >> v2:172.21.15.202:3300/0 conn(0x7fec5c1508d0 0x7fec5c0bbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:26.863 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:26.858+0000 7fec61fca700 1 -- 172.21.15.195:0/3320497018 >> v2:172.21.15.202:3300/0 conn(0x7fec5c1508d0 msgr2=0x7fec5c0bbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:18:26.863 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:26.858+0000 7fec61fca700 1 --2- 172.21.15.195:0/3320497018 >> v2:172.21.15.202:3300/0 conn(0x7fec5c1508d0 0x7fec5c0bbf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:26.863 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:26.858+0000 7fec61fca700 1 -- 172.21.15.195:0/3320497018 >> v1:172.21.15.202:6789/0 conn(0x7fec5c06a450 legacy=0x7fec5c0c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:18:26.864 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:26.858+0000 7fec61fca700 1 -- 172.21.15.195:0/3320497018 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec5c15a800 con 0x7fec5c150500 2022-06-27T12:18:26.864 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:26.858+0000 7fec627cb700 1 --2- 172.21.15.195:0/3320497018 >> v2:172.21.15.202:3300/0 conn(0x7fec5c1508d0 0x7fec5c0bbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T12:18:26.864 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:26.858+0000 7fec61fca700 1 --2- 172.21.15.195:0/3320497018 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fec5c150500 0x7fec5c063af0 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7fec5c064800 tx=0x7fec580090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6a8a4f5192cdcf7 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:18:26.864 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:26.859+0000 7fec60fc8700 1 -- 172.21.15.195:0/3320497018 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec58005ad0 con 0x7fec5c150500 2022-06-27T12:18:26.865 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:26.859+0000 7fec60fc8700 1 -- 172.21.15.195:0/3320497018 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec58005c30 con 0x7fec5c150500 2022-06-27T12:18:26.865 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:26.859+0000 7fec60fc8700 1 -- 172.21.15.195:0/3320497018 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec58006030 con 0x7fec5c150500 2022-06-27T12:18:26.865 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:26.859+0000 7fec6422e700 1 -- 172.21.15.195:0/3320497018 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fec5c150500 msgr2=0x7fec5c063af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:26.865 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:26.859+0000 7fec6422e700 1 --2- 172.21.15.195:0/3320497018 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fec5c150500 0x7fec5c063af0 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7fec5c064800 tx=0x7fec580090f0 comp rx=0 tx=0).stop 2022-06-27T12:18:26.866 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:26.859+0000 7fec6422e700 1 -- 172.21.15.195:0/3320497018 shutdown_connections 2022-06-27T12:18:26.866 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:26.859+0000 7fec6422e700 1 --2- 172.21.15.195:0/3320497018 >> v2:172.21.15.202:3300/0 conn(0x7fec5c1508d0 0x7fec5c0bbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:26.866 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:26.859+0000 7fec6422e700 1 --2- 172.21.15.195:0/3320497018 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fec5c150500 0x7fec5c063af0 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:26.869 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:26.859+0000 7fec6422e700 1 -- 172.21.15.195:0/3320497018 >> 172.21.15.195:0/3320497018 conn(0x7fec5c0c9b00 msgr2=0x7fec5c0c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:18:26.870 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:26.859+0000 7fec6422e700 1 -- 172.21.15.195:0/3320497018 shutdown_connections 2022-06-27T12:18:26.871 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:26.859+0000 7fec6422e700 1 -- 172.21.15.195:0/3320497018 wait complete. 2022-06-27T12:18:26.871 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:26.860+0000 7fec6422e700 1 Processor -- start 2022-06-27T12:18:26.872 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:26.860+0000 7fec6422e700 1 -- start start 2022-06-27T12:18:26.875 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:26.861+0000 7fec6422e700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fec5c06a450 0x7fec5c06dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:26.875 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:26.861+0000 7fec6422e700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fec5c150500 0x7fec5c153260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:26.876 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:26.861+0000 7fec6422e700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fec5c1508d0 0x7fec5c1537a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:26.876 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:26.861+0000 7fec6422e700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec5c152550 con 0x7fec5c1508d0 2022-06-27T12:18:26.876 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:26.861+0000 7fec6422e700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec5c0c7fa0 con 0x7fec5c150500 2022-06-27T12:18:26.876 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:26.861+0000 7fec6422e700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec5c1663a0 con 0x7fec5c06a450 2022-06-27T12:18:26.877 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:26.861+0000 7fec627cb700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fec5c1508d0 0x7fec5c1537a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:26.877 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:26.861+0000 7fec627cb700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fec5c1508d0 0x7fec5c1537a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:52878/0 (socket says 172.21.15.195:52878) 2022-06-27T12:18:26.877 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:26.861+0000 7fec627cb700 1 -- 172.21.15.195:0/1124944961 learned_addr learned my addr 172.21.15.195:0/1124944961 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:18:26.878 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:26.861+0000 7fec61fca700 1 --2- 172.21.15.195:0/1124944961 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fec5c06a450 0x7fec5c06dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:26.878 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:26.861+0000 7fec617c9700 1 --2- 172.21.15.195:0/1124944961 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fec5c150500 0x7fec5c153260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:26.879 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:26.862+0000 7fec627cb700 1 -- 172.21.15.195:0/1124944961 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fec5c06a450 msgr2=0x7fec5c06dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:26.879 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:26.862+0000 7fec627cb700 1 --2- 172.21.15.195:0/1124944961 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fec5c06a450 0x7fec5c06dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:26.879 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:26.862+0000 7fec627cb700 1 -- 172.21.15.195:0/1124944961 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fec5c150500 msgr2=0x7fec5c153260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:26.880 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:26.862+0000 7fec627cb700 1 --2- 172.21.15.195:0/1124944961 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fec5c150500 0x7fec5c153260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:26.880 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:26.862+0000 7fec627cb700 1 -- 172.21.15.195:0/1124944961 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec58009350 con 0x7fec5c1508d0 2022-06-27T12:18:26.880 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:26.862+0000 7fec627cb700 1 --2- 172.21.15.195:0/1124944961 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fec5c1508d0 0x7fec5c1537a0 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7fec5400cef0 tx=0x7fec54007860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:18:26.880 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:26.862+0000 7fec49ffb700 1 -- 172.21.15.195:0/1124944961 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec5400adb0 con 0x7fec5c1508d0 2022-06-27T12:18:26.882 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:26.862+0000 7fec49ffb700 1 -- 172.21.15.195:0/1124944961 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec54016420 con 0x7fec5c1508d0 2022-06-27T12:18:26.882 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:26.862+0000 7fec6422e700 1 -- 172.21.15.195:0/1124944961 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec5c152780 con 0x7fec5c1508d0 2022-06-27T12:18:26.883 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:26.862+0000 7fec49ffb700 1 -- 172.21.15.195:0/1124944961 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec540167e0 con 0x7fec5c1508d0 2022-06-27T12:18:26.883 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:26.862+0000 7fec6422e700 1 -- 172.21.15.195:0/1124944961 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec5c152d40 con 0x7fec5c1508d0 2022-06-27T12:18:26.884 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:26.863+0000 7fec6422e700 1 -- 172.21.15.195:0/1124944961 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec5c0722c0 con 0x7fec5c1508d0 2022-06-27T12:18:26.884 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:26.863+0000 7fec49ffb700 1 -- 172.21.15.195:0/1124944961 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fec54016bd0 con 0x7fec5c1508d0 2022-06-27T12:18:26.885 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:26.864+0000 7fec49ffb700 1 --2- 172.21.15.195:0/1124944961 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fec3c063d00 0x7fec3c0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:26.885 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:26.864+0000 7fec49ffb700 1 -- 172.21.15.195:0/1124944961 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7664+0+0 (secure 0 0 0) 0x7fec54012070 con 0x7fec5c1508d0 2022-06-27T12:18:26.885 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:26.864+0000 7fec61fca700 1 --2- 172.21.15.195:0/1124944961 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fec3c063d00 0x7fec3c0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:26.886 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:26.864+0000 7fec61fca700 1 --2- 172.21.15.195:0/1124944961 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fec3c063d00 0x7fec3c0661b0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fec5c064800 tx=0x7fec58016000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:18:26.886 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:26.868+0000 7fec49ffb700 1 -- 172.21.15.195:0/1124944961 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fec5404cc90 con 0x7fec5c1508d0 2022-06-27T12:18:27.095 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:27.092+0000 7fec6422e700 1 -- 172.21.15.195:0/1124944961 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fec5c0ba8e0 con 0x7fec5c1508d0 2022-06-27T12:18:27.095 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:27.093+0000 7fec49ffb700 1 -- 172.21.15.195:0/1124944961 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+6132 (secure 0 0 0) 0x7fec54016e60 con 0x7fec5c1508d0 2022-06-27T12:18:27.096 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:18:27.097 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":113,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":7300,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/1965783412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1965783412},{"type":"v1","addr":"172.21.15.202:6833","nonce":1965783412}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":113}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:17:30.381992+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7165},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7165":{"gid":7165,"name":"d","rank":0,"incarnation":108,"state":"up:active","state_seq":27,"addr":"172.21.15.202:6835/3551478822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":3551478822},{"type":"v1","addr":"172.21.15.202:6835","nonce":3551478822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:18:27.099 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:27.094+0000 7fec6422e700 1 -- 172.21.15.195:0/1124944961 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fec3c063d00 msgr2=0x7fec3c0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:27.099 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:27.094+0000 7fec6422e700 1 --2- 172.21.15.195:0/1124944961 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fec3c063d00 0x7fec3c0661b0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fec5c064800 tx=0x7fec58016000 comp rx=0 tx=0).stop 2022-06-27T12:18:27.100 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:27.094+0000 7fec6422e700 1 -- 172.21.15.195:0/1124944961 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fec5c1508d0 msgr2=0x7fec5c1537a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:27.100 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:27.094+0000 7fec6422e700 1 --2- 172.21.15.195:0/1124944961 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fec5c1508d0 0x7fec5c1537a0 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7fec5400cef0 tx=0x7fec54007860 comp rx=0 tx=0).stop 2022-06-27T12:18:27.101 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:27.094+0000 7fec6422e700 1 -- 172.21.15.195:0/1124944961 shutdown_connections 2022-06-27T12:18:27.102 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:27.094+0000 7fec6422e700 1 --2- 172.21.15.195:0/1124944961 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fec3c063d00 0x7fec3c0661b0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:27.102 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:27.094+0000 7fec6422e700 1 --2- 172.21.15.195:0/1124944961 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fec5c1508d0 0x7fec5c1537a0 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:27.102 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:27.094+0000 7fec6422e700 1 --2- 172.21.15.195:0/1124944961 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fec5c06a450 0x7fec5c06dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:27.103 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:27.094+0000 7fec6422e700 1 --2- 172.21.15.195:0/1124944961 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fec5c150500 0x7fec5c153260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:27.103 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:27.094+0000 7fec6422e700 1 -- 172.21.15.195:0/1124944961 >> 172.21.15.195:0/1124944961 conn(0x7fec5c0c9b00 msgr2=0x7fec5c0c6e40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:18:27.103 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:27.095+0000 7fec6422e700 1 -- 172.21.15.195:0/1124944961 shutdown_connections 2022-06-27T12:18:27.103 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:27.095+0000 7fec6422e700 1 -- 172.21.15.195:0/1124944961 wait complete. 2022-06-27T12:18:27.104 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 113 2022-06-27T12:18:27.107 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-06-27T12:18:29.111 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:18:29.263 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.253+0000 7fbc67eb9700 1 Processor -- start 2022-06-27T12:18:29.264 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.253+0000 7fbc67eb9700 1 -- start start 2022-06-27T12:18:29.264 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.254+0000 7fbc67eb9700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fbc60067ed0 0x7fbc600c61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:29.264 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.254+0000 7fbc67eb9700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc6014d8a0 con 0x7fbc600c31d0 2022-06-27T12:18:29.264 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.254+0000 7fbc67eb9700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc600c3e10 con 0x7fbc6006eff0 2022-06-27T12:18:29.265 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.254+0000 7fbc67eb9700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc6006d970 con 0x7fbc60067ed0 2022-06-27T12:18:29.265 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.254+0000 7fbc65454700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fbc600c31d0 0x7fbc6015ba30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:59072/0 (socket says 172.21.15.195:59072) 2022-06-27T12:18:29.265 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.254+0000 7fbc66456700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fbc60067ed0 0x7fbc600c61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:29.265 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.254+0000 7fbc65454700 1 -- 172.21.15.195:0/2771178709 learned_addr learned my addr 172.21.15.195:0/2771178709 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:18:29.267 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.254+0000 7fbc66456700 1 -- 172.21.15.195:0/2771178709 >> v1:172.21.15.202:6789/0 conn(0x7fbc6006eff0 legacy=0x7fbc600c2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:29.267 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.254+0000 7fbc66456700 1 -- 172.21.15.195:0/2771178709 >> v1:172.21.15.195:6789/0 conn(0x7fbc600c31d0 legacy=0x7fbc6015ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:29.267 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.254+0000 7fbc66456700 1 -- 172.21.15.195:0/2771178709 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc60153f20 con 0x7fbc60067ed0 2022-06-27T12:18:29.268 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.254+0000 7fbc64c53700 1 -- 172.21.15.195:0/2771178709 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1830128063 0 0) 0x7fbc6014d8a0 con 0x7fbc600c31d0 2022-06-27T12:18:29.268 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.254+0000 7fbc66456700 1 --2- 172.21.15.195:0/2771178709 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbc60067ed0 0x7fbc600c61e0 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7fbc58009e60 tx=0x7fbc58009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51676dd7751c60e4 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:18:29.268 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.255+0000 7fbc64c53700 1 -- 172.21.15.195:0/2771178709 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc5800e070 con 0x7fbc60067ed0 2022-06-27T12:18:29.268 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.255+0000 7fbc64c53700 1 -- 172.21.15.195:0/2771178709 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc5800bcc0 con 0x7fbc60067ed0 2022-06-27T12:18:29.269 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.255+0000 7fbc64c53700 1 -- 172.21.15.195:0/2771178709 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc58012680 con 0x7fbc60067ed0 2022-06-27T12:18:29.269 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.255+0000 7fbc67eb9700 1 -- 172.21.15.195:0/2771178709 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbc60067ed0 msgr2=0x7fbc600c61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:29.269 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.255+0000 7fbc67eb9700 1 --2- 172.21.15.195:0/2771178709 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbc60067ed0 0x7fbc600c61e0 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7fbc58009e60 tx=0x7fbc58009780 comp rx=0 tx=0).stop 2022-06-27T12:18:29.269 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.255+0000 7fbc67eb9700 1 -- 172.21.15.195:0/2771178709 shutdown_connections 2022-06-27T12:18:29.270 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.255+0000 7fbc67eb9700 1 --2- 172.21.15.195:0/2771178709 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbc60067ed0 0x7fbc600c61e0 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:29.270 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.255+0000 7fbc67eb9700 1 -- 172.21.15.195:0/2771178709 >> 172.21.15.195:0/2771178709 conn(0x7fbc600c9b00 msgr2=0x7fbc600c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:18:29.270 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.255+0000 7fbc67eb9700 1 -- 172.21.15.195:0/2771178709 shutdown_connections 2022-06-27T12:18:29.271 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.255+0000 7fbc67eb9700 1 -- 172.21.15.195:0/2771178709 wait complete. 2022-06-27T12:18:29.271 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.256+0000 7fbc67eb9700 1 Processor -- start 2022-06-27T12:18:29.271 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.256+0000 7fbc67eb9700 1 -- start start 2022-06-27T12:18:29.271 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.257+0000 7fbc67eb9700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbc60067ed0 0x7fbc601665f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:29.272 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.257+0000 7fbc67eb9700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbc6006eff0 0x7fbc6014f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:29.272 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.257+0000 7fbc67eb9700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbc600c8350 0x7fbc6014aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:29.272 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.257+0000 7fbc67eb9700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc601661d0 con 0x7fbc60067ed0 2022-06-27T12:18:29.272 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.257+0000 7fbc67eb9700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc60068cf0 con 0x7fbc600c8350 2022-06-27T12:18:29.273 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.257+0000 7fbc67eb9700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc6016a0e0 con 0x7fbc6006eff0 2022-06-27T12:18:29.273 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.257+0000 7fbc65c55700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbc60067ed0 0x7fbc601665f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:29.273 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.257+0000 7fbc65c55700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbc60067ed0 0x7fbc601665f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:52888/0 (socket says 172.21.15.195:52888) 2022-06-27T12:18:29.273 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.257+0000 7fbc65c55700 1 -- 172.21.15.195:0/3641030002 learned_addr learned my addr 172.21.15.195:0/3641030002 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:18:29.276 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.257+0000 7fbc65454700 1 --2- 172.21.15.195:0/3641030002 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbc6006eff0 0x7fbc6014f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:29.276 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.257+0000 7fbc66456700 1 --2- 172.21.15.195:0/3641030002 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbc600c8350 0x7fbc6014aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:29.276 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.257+0000 7fbc65c55700 1 -- 172.21.15.195:0/3641030002 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbc6006eff0 msgr2=0x7fbc6014f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:29.277 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.257+0000 7fbc65c55700 1 --2- 172.21.15.195:0/3641030002 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbc6006eff0 0x7fbc6014f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:29.277 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.257+0000 7fbc65c55700 1 -- 172.21.15.195:0/3641030002 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbc600c8350 msgr2=0x7fbc6014aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:29.277 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.257+0000 7fbc65c55700 1 --2- 172.21.15.195:0/3641030002 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbc600c8350 0x7fbc6014aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:29.277 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.257+0000 7fbc65c55700 1 -- 172.21.15.195:0/3641030002 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc58009a20 con 0x7fbc60067ed0 2022-06-27T12:18:29.278 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.257+0000 7fbc66456700 1 --2- 172.21.15.195:0/3641030002 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbc600c8350 0x7fbc6014aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T12:18:29.279 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.258+0000 7fbc65c55700 1 --2- 172.21.15.195:0/3641030002 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbc60067ed0 0x7fbc601665f0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c00c4c0 tx=0x7fbc5c007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:18:29.279 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.258+0000 7fbc4dffb700 1 -- 172.21.15.195:0/3641030002 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc5c0050d0 con 0x7fbc60067ed0 2022-06-27T12:18:29.279 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.258+0000 7fbc4dffb700 1 -- 172.21.15.195:0/3641030002 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc5c005250 con 0x7fbc60067ed0 2022-06-27T12:18:29.279 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.258+0000 7fbc67eb9700 1 -- 172.21.15.195:0/3641030002 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc6014fce0 con 0x7fbc60067ed0 2022-06-27T12:18:29.280 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.258+0000 7fbc4dffb700 1 -- 172.21.15.195:0/3641030002 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc5c0117e0 con 0x7fbc60067ed0 2022-06-27T12:18:29.280 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.258+0000 7fbc67eb9700 1 -- 172.21.15.195:0/3641030002 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc60165df0 con 0x7fbc60067ed0 2022-06-27T12:18:29.280 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.259+0000 7fbc67eb9700 1 -- 172.21.15.195:0/3641030002 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc60165f80 con 0x7fbc60067ed0 2022-06-27T12:18:29.280 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.259+0000 7fbc4dffb700 1 -- 172.21.15.195:0/3641030002 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fbc5c019030 con 0x7fbc60067ed0 2022-06-27T12:18:29.281 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.260+0000 7fbc4dffb700 1 --2- 172.21.15.195:0/3641030002 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbc40063d00 0x7fbc400661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:29.281 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.260+0000 7fbc4dffb700 1 -- 172.21.15.195:0/3641030002 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7664+0+0 (secure 0 0 0) 0x7fbc5c080130 con 0x7fbc60067ed0 2022-06-27T12:18:29.281 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.260+0000 7fbc65454700 1 --2- 172.21.15.195:0/3641030002 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbc40063d00 0x7fbc400661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:29.281 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.260+0000 7fbc65454700 1 --2- 172.21.15.195:0/3641030002 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbc40063d00 0x7fbc400661b0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fbc60068c80 tx=0x7fbc50007b90 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:18:29.282 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.264+0000 7fbc4dffb700 1 -- 172.21.15.195:0/3641030002 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fbc5c04e6e0 con 0x7fbc60067ed0 2022-06-27T12:18:29.492 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.491+0000 7fbc67eb9700 1 -- 172.21.15.195:0/3641030002 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbc600722c0 con 0x7fbc60067ed0 2022-06-27T12:18:29.492 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.492+0000 7fbc4dffb700 1 -- 172.21.15.195:0/3641030002 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+6132 (secure 0 0 0) 0x7fbc5c017020 con 0x7fbc60067ed0 2022-06-27T12:18:29.493 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:18:29.493 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":113,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":7300,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6833/1965783412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1965783412},{"type":"v1","addr":"172.21.15.202:6833","nonce":1965783412}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":113}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:17:30.381992+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7165},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7165":{"gid":7165,"name":"d","rank":0,"incarnation":108,"state":"up:active","state_seq":27,"addr":"172.21.15.202:6835/3551478822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":3551478822},{"type":"v1","addr":"172.21.15.202:6835","nonce":3551478822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:18:29.495 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.493+0000 7fbc67eb9700 1 -- 172.21.15.195:0/3641030002 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbc40063d00 msgr2=0x7fbc400661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:29.495 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.493+0000 7fbc67eb9700 1 --2- 172.21.15.195:0/3641030002 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbc40063d00 0x7fbc400661b0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fbc60068c80 tx=0x7fbc50007b90 comp rx=0 tx=0).stop 2022-06-27T12:18:29.495 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.493+0000 7fbc67eb9700 1 -- 172.21.15.195:0/3641030002 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbc60067ed0 msgr2=0x7fbc601665f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:29.495 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.493+0000 7fbc67eb9700 1 --2- 172.21.15.195:0/3641030002 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbc60067ed0 0x7fbc601665f0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c00c4c0 tx=0x7fbc5c007df0 comp rx=0 tx=0).stop 2022-06-27T12:18:29.496 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.494+0000 7fbc67eb9700 1 -- 172.21.15.195:0/3641030002 shutdown_connections 2022-06-27T12:18:29.496 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.494+0000 7fbc67eb9700 1 --2- 172.21.15.195:0/3641030002 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbc40063d00 0x7fbc400661b0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:29.496 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.494+0000 7fbc67eb9700 1 --2- 172.21.15.195:0/3641030002 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbc60067ed0 0x7fbc601665f0 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:29.497 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.494+0000 7fbc67eb9700 1 --2- 172.21.15.195:0/3641030002 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbc600c8350 0x7fbc6014aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:29.497 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.494+0000 7fbc67eb9700 1 --2- 172.21.15.195:0/3641030002 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbc6006eff0 0x7fbc6014f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:29.497 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.494+0000 7fbc67eb9700 1 -- 172.21.15.195:0/3641030002 >> 172.21.15.195:0/3641030002 conn(0x7fbc600c9b00 msgr2=0x7fbc6014c9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:18:29.497 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.494+0000 7fbc67eb9700 1 -- 172.21.15.195:0/3641030002 shutdown_connections 2022-06-27T12:18:29.498 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:29.494+0000 7fbc67eb9700 1 -- 172.21.15.195:0/3641030002 wait complete. 2022-06-27T12:18:29.498 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 113 2022-06-27T12:18:29.508 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-06-27T12:18:31.510 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:18:31.646 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.645+0000 7fe80471d700 1 Processor -- start 2022-06-27T12:18:31.647 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.645+0000 7fe80471d700 1 -- start start 2022-06-27T12:18:31.661 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.646+0000 7fe80471d700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fe7fc14e490 0x7fe7fc06cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:31.661 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.646+0000 7fe80471d700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7fc0c33b0 con 0x7fe7fc14e860 2022-06-27T12:18:31.662 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.646+0000 7fe80471d700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7fc06d520 con 0x7fe7fc0c3f50 2022-06-27T12:18:31.662 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.646+0000 7fe80471d700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7fc06f220 con 0x7fe7fc14e490 2022-06-27T12:18:31.662 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.646+0000 7fe801cb8700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fe7fc14e860 0x7fe7fc0c3840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:59084/0 (socket says 172.21.15.195:59084) 2022-06-27T12:18:31.662 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.646+0000 7fe801cb8700 1 -- 172.21.15.195:0/2046649881 learned_addr learned my addr 172.21.15.195:0/2046649881 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:18:31.663 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.646+0000 7fe8024b9700 1 --2- 172.21.15.195:0/2046649881 >> v2:172.21.15.202:3300/0 conn(0x7fe7fc14e490 0x7fe7fc06cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:31.663 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.646+0000 7fe8014b7700 1 -- 172.21.15.195:0/2046649881 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3842946391 0 0) 0x7fe7fc0c33b0 con 0x7fe7fc14e860 2022-06-27T12:18:31.663 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.646+0000 7fe8014b7700 1 -- 172.21.15.195:0/2046649881 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7e4003430 con 0x7fe7fc14e860 2022-06-27T12:18:31.663 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.647+0000 7fe8014b7700 1 -- 172.21.15.195:0/2046649881 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 178634190 0 0) 0x7fe7e4003430 con 0x7fe7fc14e860 2022-06-27T12:18:31.664 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.647+0000 7fe8014b7700 1 -- 172.21.15.195:0/2046649881 >> v2:172.21.15.202:3300/0 conn(0x7fe7fc14e490 msgr2=0x7fe7fc06cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:18:31.664 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.647+0000 7fe8014b7700 1 --2- 172.21.15.195:0/2046649881 >> v2:172.21.15.202:3300/0 conn(0x7fe7fc14e490 0x7fe7fc06cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:31.664 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.647+0000 7fe8014b7700 1 -- 172.21.15.195:0/2046649881 >> v1:172.21.15.202:6789/0 conn(0x7fe7fc0c3f50 legacy=0x7fe7fc157890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:31.664 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.647+0000 7fe8014b7700 1 -- 172.21.15.195:0/2046649881 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7fc1542c0 con 0x7fe7fc14e860 2022-06-27T12:18:31.665 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.647+0000 7fe8014b7700 1 -- 172.21.15.195:0/2046649881 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7fe7f8001470 con 0x7fe7fc14e860 2022-06-27T12:18:31.665 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.647+0000 7fe8014b7700 1 -- 172.21.15.195:0/2046649881 >> v1:172.21.15.195:6789/0 conn(0x7fe7fc14e860 legacy=0x7fe7fc0c3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:31.666 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.647+0000 7fe8014b7700 1 --2- 172.21.15.195:0/2046649881 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe7e4003920 0x7fe7e4003d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:31.666 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.647+0000 7fe8014b7700 1 --2- 172.21.15.195:0/2046649881 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe7e40046d0 0x7fe7e4004b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:31.666 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.647+0000 7fe8014b7700 1 --2- 172.21.15.195:0/2046649881 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe7e4005080 0x7fe7e400b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:31.666 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.647+0000 7fe8014b7700 1 -- 172.21.15.195:0/2046649881 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7fc06f220 con 0x7fe7e4003920 2022-06-27T12:18:31.667 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.647+0000 7fe8014b7700 1 -- 172.21.15.195:0/2046649881 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e400bf90 con 0x7fe7e4005080 2022-06-27T12:18:31.667 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.647+0000 7fe8014b7700 1 -- 172.21.15.195:0/2046649881 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e400c150 con 0x7fe7e40046d0 2022-06-27T12:18:31.667 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.647+0000 7fe8024b9700 1 --2- 172.21.15.195:0/2046649881 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe7e4003920 0x7fe7e4003d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:31.667 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.647+0000 7fe802cba700 1 --2- 172.21.15.195:0/2046649881 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe7e4005080 0x7fe7e400b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:31.668 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.647+0000 7fe801cb8700 1 --2- 172.21.15.195:0/2046649881 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe7e40046d0 0x7fe7e4004b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:31.668 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.647+0000 7fe8024b9700 1 -- 172.21.15.195:0/2046649881 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe7e40046d0 msgr2=0x7fe7e4004b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:18:31.668 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.647+0000 7fe8024b9700 1 --2- 172.21.15.195:0/2046649881 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe7e40046d0 0x7fe7e4004b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:31.668 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.647+0000 7fe8024b9700 1 -- 172.21.15.195:0/2046649881 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe7e4005080 msgr2=0x7fe7e400b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:18:31.668 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.647+0000 7fe8024b9700 1 --2- 172.21.15.195:0/2046649881 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe7e4005080 0x7fe7e400b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:31.669 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.647+0000 7fe802cba700 1 -- 172.21.15.195:0/2046649881 reap_dead start 2022-06-27T12:18:31.669 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.647+0000 7fe8024b9700 1 -- 172.21.15.195:0/2046649881 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7e4003430 con 0x7fe7e4003920 2022-06-27T12:18:31.669 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.647+0000 7fe8024b9700 1 --2- 172.21.15.195:0/2046649881 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe7e4003920 0x7fe7e4003d30 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7fe7ec009ac0 tx=0x7fe7ec009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f529e8bd503da40 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:18:31.669 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.647+0000 7fe8014b7700 1 -- 172.21.15.195:0/2046649881 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7ec009470 con 0x7fe7e4003920 2022-06-27T12:18:31.670 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.647+0000 7fe8014b7700 1 -- 172.21.15.195:0/2046649881 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7ec00e640 con 0x7fe7e4003920 2022-06-27T12:18:31.670 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.647+0000 7fe8014b7700 1 -- 172.21.15.195:0/2046649881 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7ec00e9e0 con 0x7fe7e4003920 2022-06-27T12:18:31.671 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.648+0000 7fe80471d700 1 -- 172.21.15.195:0/2046649881 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe7e4003920 msgr2=0x7fe7e4003d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:31.671 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.648+0000 7fe80471d700 1 --2- 172.21.15.195:0/2046649881 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe7e4003920 0x7fe7e4003d30 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7fe7ec009ac0 tx=0x7fe7ec009060 comp rx=0 tx=0).stop 2022-06-27T12:18:31.671 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.648+0000 7fe80471d700 1 -- 172.21.15.195:0/2046649881 shutdown_connections 2022-06-27T12:18:31.671 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.648+0000 7fe80471d700 1 --2- 172.21.15.195:0/2046649881 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe7e4003920 0x7fe7e4003d30 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:31.672 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.648+0000 7fe80471d700 1 -- 172.21.15.195:0/2046649881 >> 172.21.15.195:0/2046649881 conn(0x7fe7fc063af0 msgr2=0x7fe7fc063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:18:31.672 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.648+0000 7fe80471d700 1 -- 172.21.15.195:0/2046649881 shutdown_connections 2022-06-27T12:18:31.672 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.648+0000 7fe80471d700 1 -- 172.21.15.195:0/2046649881 wait complete. 2022-06-27T12:18:31.672 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.649+0000 7fe80471d700 1 Processor -- start 2022-06-27T12:18:31.673 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.649+0000 7fe80471d700 1 -- start start 2022-06-27T12:18:31.673 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.650+0000 7fe80471d700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe7e4003920 0x7fe7fc0bfe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:31.673 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.650+0000 7fe80471d700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe7fc16a030 0x7fe7fc16a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:31.673 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.650+0000 7fe80471d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe7fc16a9e0 0x7fe7fc148e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:31.673 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.650+0000 7fe80471d700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7fc0be990 con 0x7fe7fc16a9e0 2022-06-27T12:18:31.674 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.650+0000 7fe80471d700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7fc06a310 con 0x7fe7e4003920 2022-06-27T12:18:31.674 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.650+0000 7fe80471d700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7fc069420 con 0x7fe7fc16a030 2022-06-27T12:18:31.674 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.650+0000 7fe802cba700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe7fc16a9e0 0x7fe7fc148e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:31.674 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.650+0000 7fe802cba700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe7fc16a9e0 0x7fe7fc148e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:52910/0 (socket says 172.21.15.195:52910) 2022-06-27T12:18:31.675 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.650+0000 7fe802cba700 1 -- 172.21.15.195:0/1833331404 learned_addr learned my addr 172.21.15.195:0/1833331404 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:18:31.675 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.650+0000 7fe801cb8700 1 --2- 172.21.15.195:0/1833331404 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe7fc16a030 0x7fe7fc16a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:31.675 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.650+0000 7fe8024b9700 1 --2- 172.21.15.195:0/1833331404 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe7e4003920 0x7fe7fc0bfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:31.676 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.650+0000 7fe802cba700 1 -- 172.21.15.195:0/1833331404 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe7fc16a030 msgr2=0x7fe7fc16a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:31.676 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.650+0000 7fe802cba700 1 --2- 172.21.15.195:0/1833331404 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe7fc16a030 0x7fe7fc16a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:31.676 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.650+0000 7fe802cba700 1 -- 172.21.15.195:0/1833331404 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe7e4003920 msgr2=0x7fe7fc0bfe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:31.676 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.650+0000 7fe802cba700 1 --2- 172.21.15.195:0/1833331404 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe7e4003920 0x7fe7fc0bfe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:31.677 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.650+0000 7fe802cba700 1 -- 172.21.15.195:0/1833331404 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7fc069700 con 0x7fe7fc16a9e0 2022-06-27T12:18:31.677 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.651+0000 7fe802cba700 1 --2- 172.21.15.195:0/1833331404 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe7fc16a9e0 0x7fe7fc148e00 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc0c43b0 tx=0x7fe7f4009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:18:31.677 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.651+0000 7fe7ea7fc700 1 -- 172.21.15.195:0/1833331404 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f400ee40 con 0x7fe7fc16a9e0 2022-06-27T12:18:31.677 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.651+0000 7fe7ea7fc700 1 -- 172.21.15.195:0/1833331404 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7f4003a90 con 0x7fe7fc16a9e0 2022-06-27T12:18:31.677 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.651+0000 7fe80471d700 1 -- 172.21.15.195:0/1833331404 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7fc0699e0 con 0x7fe7fc16a9e0 2022-06-27T12:18:31.678 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.651+0000 7fe7ea7fc700 1 -- 172.21.15.195:0/1833331404 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f4003e30 con 0x7fe7fc16a9e0 2022-06-27T12:18:31.678 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.651+0000 7fe80471d700 1 -- 172.21.15.195:0/1833331404 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7fc069fa0 con 0x7fe7fc16a9e0 2022-06-27T12:18:31.678 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.652+0000 7fe80471d700 1 -- 172.21.15.195:0/1833331404 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7fc0722c0 con 0x7fe7fc16a9e0 2022-06-27T12:18:31.678 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.652+0000 7fe7ea7fc700 1 -- 172.21.15.195:0/1833331404 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fe7f401b030 con 0x7fe7fc16a9e0 2022-06-27T12:18:31.679 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.652+0000 7fe7ea7fc700 1 --2- 172.21.15.195:0/1833331404 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe7dc063d00 0x7fe7dc0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:31.679 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.653+0000 7fe7ea7fc700 1 -- 172.21.15.195:0/1833331404 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7750+0+0 (secure 0 0 0) 0x7fe7f4080510 con 0x7fe7fc16a9e0 2022-06-27T12:18:31.679 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.653+0000 7fe8024b9700 1 --2- 172.21.15.195:0/1833331404 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe7dc063d00 0x7fe7dc0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:31.679 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.653+0000 7fe8024b9700 1 --2- 172.21.15.195:0/1833331404 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe7dc063d00 0x7fe7dc0661b0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fe7ec009880 tx=0x7fe7ec01a000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:18:31.679 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.657+0000 7fe7ea7fc700 1 -- 172.21.15.195:0/1833331404 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fe7f404ea60 con 0x7fe7fc16a9e0 2022-06-27T12:18:31.879 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.879+0000 7fe80471d700 1 -- 172.21.15.195:0/1833331404 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe7fc06a450 con 0x7fe7fc16a9e0 2022-06-27T12:18:31.880 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.879+0000 7fe7ea7fc700 1 -- 172.21.15.195:0/1833331404 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) v1 ==== 95+0+5370 (secure 0 0 0) 0x7fe7f4019020 con 0x7fe7fc16a9e0 2022-06-27T12:18:31.880 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:18:31.881 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":114,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:18:30.240443+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7300},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7300":{"gid":7300,"name":"b","rank":0,"incarnation":114,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6833/1965783412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1965783412},{"type":"v1","addr":"172.21.15.202:6833","nonce":1965783412}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:18:31.884 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.881+0000 7fe80471d700 1 -- 172.21.15.195:0/1833331404 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe7dc063d00 msgr2=0x7fe7dc0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:31.885 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.881+0000 7fe80471d700 1 --2- 172.21.15.195:0/1833331404 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe7dc063d00 0x7fe7dc0661b0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fe7ec009880 tx=0x7fe7ec01a000 comp rx=0 tx=0).stop 2022-06-27T12:18:31.885 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.881+0000 7fe80471d700 1 -- 172.21.15.195:0/1833331404 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe7fc16a9e0 msgr2=0x7fe7fc148e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:31.885 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.881+0000 7fe80471d700 1 --2- 172.21.15.195:0/1833331404 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe7fc16a9e0 0x7fe7fc148e00 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc0c43b0 tx=0x7fe7f4009040 comp rx=0 tx=0).stop 2022-06-27T12:18:31.886 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.881+0000 7fe80471d700 1 -- 172.21.15.195:0/1833331404 shutdown_connections 2022-06-27T12:18:31.886 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.881+0000 7fe80471d700 1 --2- 172.21.15.195:0/1833331404 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fe7dc063d00 0x7fe7dc0661b0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:31.886 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.881+0000 7fe80471d700 1 --2- 172.21.15.195:0/1833331404 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe7fc16a9e0 0x7fe7fc148e00 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:31.887 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.881+0000 7fe80471d700 1 --2- 172.21.15.195:0/1833331404 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fe7e4003920 0x7fe7fc0bfe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:31.887 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.881+0000 7fe80471d700 1 --2- 172.21.15.195:0/1833331404 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fe7fc16a030 0x7fe7fc16a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:31.887 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.881+0000 7fe80471d700 1 -- 172.21.15.195:0/1833331404 >> 172.21.15.195:0/1833331404 conn(0x7fe7fc063af0 msgr2=0x7fe7fc0c9100 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:18:31.888 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.881+0000 7fe80471d700 1 -- 172.21.15.195:0/1833331404 shutdown_connections 2022-06-27T12:18:31.888 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:31.881+0000 7fe80471d700 1 -- 172.21.15.195:0/1833331404 wait complete. 2022-06-27T12:18:31.889 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 114 2022-06-27T12:18:31.894 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d down, removed from mdsmap 2022-06-27T12:18:31.895 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2022-06-27T12:18:31.895 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:18:32.027 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.016+0000 7faaa5d0c700 1 Processor -- start 2022-06-27T12:18:32.027 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.016+0000 7faaa5d0c700 1 -- start start 2022-06-27T12:18:32.027 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.017+0000 7faaa5d0c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7faaa006f640 0x7faaa0156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:32.028 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.017+0000 7faaa5d0c700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faaa00c4160 0x7faaa00c1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:32.028 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.017+0000 7faaa5d0c700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7faaa00c39e0 0x7faaa01535a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:32.028 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.017+0000 7faaa5d0c700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faaa00c1a20 con 0x7faaa006f640 2022-06-27T12:18:32.028 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.017+0000 7faaa5d0c700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faaa00c1130 con 0x7faaa00c39e0 2022-06-27T12:18:32.029 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.017+0000 7faaa5d0c700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaa006ea90 con 0x7faaa00c4160 2022-06-27T12:18:32.030 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.017+0000 7faa9f7fe700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7faaa006f640 0x7faaa0156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:32.030 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.017+0000 7faa9f7fe700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7faaa006f640 0x7faaa0156ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:52914/0 (socket says 172.21.15.195:52914) 2022-06-27T12:18:32.031 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.017+0000 7faa9f7fe700 1 -- 172.21.15.195:0/3822620778 learned_addr learned my addr 172.21.15.195:0/3822620778 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:18:32.031 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.017+0000 7faa9effd700 1 --2- 172.21.15.195:0/3822620778 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faaa00c4160 0x7faaa00c1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:32.031 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.017+0000 7faa9ffff700 1 --2- 172.21.15.195:0/3822620778 >> v2:172.21.15.202:3300/0 conn(0x7faaa00c39e0 0x7faaa01535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:32.032 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.017+0000 7faa9f7fe700 1 -- 172.21.15.195:0/3822620778 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faaa00c4160 msgr2=0x7faaa00c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:18:32.033 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.017+0000 7faa9f7fe700 1 --2- 172.21.15.195:0/3822620778 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faaa00c4160 0x7faaa00c1eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:32.033 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.017+0000 7faa9f7fe700 1 -- 172.21.15.195:0/3822620778 >> v2:172.21.15.202:3300/0 conn(0x7faaa00c39e0 msgr2=0x7faaa01535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:18:32.033 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.017+0000 7faa9f7fe700 1 --2- 172.21.15.195:0/3822620778 >> v2:172.21.15.202:3300/0 conn(0x7faaa00c39e0 0x7faaa01535a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:32.033 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.017+0000 7faa9f7fe700 1 -- 172.21.15.195:0/3822620778 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaa006ed70 con 0x7faaa006f640 2022-06-27T12:18:32.034 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.017+0000 7faa9ffff700 1 --2- 172.21.15.195:0/3822620778 >> v2:172.21.15.202:3300/0 conn(0x7faaa00c39e0 0x7faaa01535a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:18:32.034 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.017+0000 7faa9f7fe700 1 --2- 172.21.15.195:0/3822620778 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faaa006f640 0x7faaa0156ed0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7faaa006b530 tx=0x7faa94009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f38096022ff22e7c server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:18:32.034 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.017+0000 7faa9e7fc700 1 -- 172.21.15.195:0/3822620778 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa9400b6d0 con 0x7faaa006f640 2022-06-27T12:18:32.034 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.018+0000 7faa9e7fc700 1 -- 172.21.15.195:0/3822620778 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa9400b830 con 0x7faaa006f640 2022-06-27T12:18:32.035 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.018+0000 7faa9e7fc700 1 -- 172.21.15.195:0/3822620778 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa9400bbd0 con 0x7faaa006f640 2022-06-27T12:18:32.035 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.018+0000 7faaa5d0c700 1 -- 172.21.15.195:0/3822620778 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faaa006f640 msgr2=0x7faaa0156ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:32.035 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.018+0000 7faaa5d0c700 1 --2- 172.21.15.195:0/3822620778 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faaa006f640 0x7faaa0156ed0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7faaa006b530 tx=0x7faa94009120 comp rx=0 tx=0).stop 2022-06-27T12:18:32.035 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.018+0000 7faaa5d0c700 1 -- 172.21.15.195:0/3822620778 shutdown_connections 2022-06-27T12:18:32.036 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.018+0000 7faaa5d0c700 1 --2- 172.21.15.195:0/3822620778 >> v2:172.21.15.202:3300/0 conn(0x7faaa00c39e0 0x7faaa01535a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:32.036 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.018+0000 7faaa5d0c700 1 --2- 172.21.15.195:0/3822620778 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faaa006f640 0x7faaa0156ed0 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:32.036 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.018+0000 7faaa5d0c700 1 --2- 172.21.15.195:0/3822620778 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faaa00c4160 0x7faaa00c1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:32.037 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.018+0000 7faaa5d0c700 1 -- 172.21.15.195:0/3822620778 >> 172.21.15.195:0/3822620778 conn(0x7faaa00c6350 msgr2=0x7faaa00c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:18:32.037 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.018+0000 7faaa5d0c700 1 -- 172.21.15.195:0/3822620778 shutdown_connections 2022-06-27T12:18:32.037 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.018+0000 7faaa5d0c700 1 -- 172.21.15.195:0/3822620778 wait complete. 2022-06-27T12:18:32.038 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.019+0000 7faaa5d0c700 1 Processor -- start 2022-06-27T12:18:32.038 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.019+0000 7faaa5d0c700 1 -- start start 2022-06-27T12:18:32.039 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.020+0000 7faaa5d0c700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faaa006f640 0x7faaa015c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:32.039 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.020+0000 7faaa5d0c700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7faaa00c39e0 0x7faaa00bb040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:32.040 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.020+0000 7faaa5d0c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faaa00c4160 0x7faaa00bb580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:32.040 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.020+0000 7faaa5d0c700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaa0166250 con 0x7faaa00c4160 2022-06-27T12:18:32.041 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.020+0000 7faaa5d0c700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaa00bd240 con 0x7faaa00c39e0 2022-06-27T12:18:32.041 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.020+0000 7faaa5d0c700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaa0166d50 con 0x7faaa006f640 2022-06-27T12:18:32.042 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.020+0000 7faa9ffff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faaa00c4160 0x7faaa00bb580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:32.042 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.020+0000 7faa9ffff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faaa00c4160 0x7faaa00bb580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:52924/0 (socket says 172.21.15.195:52924) 2022-06-27T12:18:32.043 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.020+0000 7faa9ffff700 1 -- 172.21.15.195:0/2697322682 learned_addr learned my addr 172.21.15.195:0/2697322682 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:18:32.043 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.020+0000 7faa9f7fe700 1 --2- 172.21.15.195:0/2697322682 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faaa006f640 0x7faaa015c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:32.043 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.020+0000 7faa9effd700 1 --2- 172.21.15.195:0/2697322682 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7faaa00c39e0 0x7faaa00bb040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:32.044 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.020+0000 7faa9ffff700 1 -- 172.21.15.195:0/2697322682 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faaa006f640 msgr2=0x7faaa015c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:32.044 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.020+0000 7faa9ffff700 1 --2- 172.21.15.195:0/2697322682 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faaa006f640 0x7faaa015c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:32.045 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.020+0000 7faa9ffff700 1 -- 172.21.15.195:0/2697322682 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7faaa00c39e0 msgr2=0x7faaa00bb040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:32.045 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.020+0000 7faa9ffff700 1 --2- 172.21.15.195:0/2697322682 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7faaa00c39e0 0x7faaa00bb040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:32.046 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.020+0000 7faa9ffff700 1 -- 172.21.15.195:0/2697322682 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa940093a0 con 0x7faaa00c4160 2022-06-27T12:18:32.046 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.021+0000 7faa9ffff700 1 --2- 172.21.15.195:0/2697322682 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faaa00c4160 0x7faaa00bb580 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7faa9800d180 tx=0x7faa98009a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:18:32.047 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.021+0000 7faa877fe700 1 -- 172.21.15.195:0/2697322682 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa9800dd50 con 0x7faaa00c4160 2022-06-27T12:18:32.047 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.021+0000 7faa877fe700 1 -- 172.21.15.195:0/2697322682 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa9800f040 con 0x7faaa00c4160 2022-06-27T12:18:32.048 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.021+0000 7faaa5d0c700 1 -- 172.21.15.195:0/2697322682 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaa0166460 con 0x7faaa00c4160 2022-06-27T12:18:32.048 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.021+0000 7faa877fe700 1 -- 172.21.15.195:0/2697322682 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa98003af0 con 0x7faaa00c4160 2022-06-27T12:18:32.048 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.021+0000 7faaa5d0c700 1 -- 172.21.15.195:0/2697322682 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faaa01544a0 con 0x7faaa00c4160 2022-06-27T12:18:32.049 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.022+0000 7faaa5d0c700 1 -- 172.21.15.195:0/2697322682 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faaa0166ec0 con 0x7faaa00c4160 2022-06-27T12:18:32.049 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.022+0000 7faa877fe700 1 -- 172.21.15.195:0/2697322682 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7faa98013070 con 0x7faaa00c4160 2022-06-27T12:18:32.050 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.023+0000 7faa877fe700 1 --2- 172.21.15.195:0/2697322682 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7faa80063d30 0x7faa800661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:32.050 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.023+0000 7faa877fe700 1 -- 172.21.15.195:0/2697322682 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7750+0+0 (secure 0 0 0) 0x7faa98080c70 con 0x7faaa00c4160 2022-06-27T12:18:32.051 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.023+0000 7faa9f7fe700 1 --2- 172.21.15.195:0/2697322682 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7faa80063d30 0x7faa800661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:32.052 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.023+0000 7faa9f7fe700 1 --2- 172.21.15.195:0/2697322682 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7faa80063d30 0x7faa800661e0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7faa94009160 tx=0x7faa94017000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:18:32.052 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.027+0000 7faa877fe700 1 -- 172.21.15.195:0/2697322682 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7faa9804f1c0 con 0x7faaa00c4160 2022-06-27T12:18:32.255 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.253+0000 7faaa5d0c700 1 -- 172.21.15.195:0/2697322682 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faaa00722c0 con 0x7faaa00c4160 2022-06-27T12:18:32.256 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.254+0000 7faa877fe700 1 -- 172.21.15.195:0/2697322682 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) v1 ==== 95+0+5370 (secure 0 0 0) 0x7faa9801a090 con 0x7faaa00c4160 2022-06-27T12:18:32.256 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:18:32.265 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":114,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:18:30.240443+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7300},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7300":{"gid":7300,"name":"b","rank":0,"incarnation":114,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6833/1965783412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1965783412},{"type":"v1","addr":"172.21.15.202:6833","nonce":1965783412}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:18:32.267 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.256+0000 7faaa5d0c700 1 -- 172.21.15.195:0/2697322682 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7faa80063d30 msgr2=0x7faa800661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:32.267 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.256+0000 7faaa5d0c700 1 --2- 172.21.15.195:0/2697322682 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7faa80063d30 0x7faa800661e0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7faa94009160 tx=0x7faa94017000 comp rx=0 tx=0).stop 2022-06-27T12:18:32.267 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.256+0000 7faaa5d0c700 1 -- 172.21.15.195:0/2697322682 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faaa00c4160 msgr2=0x7faaa00bb580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:32.267 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.256+0000 7faaa5d0c700 1 --2- 172.21.15.195:0/2697322682 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faaa00c4160 0x7faaa00bb580 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7faa9800d180 tx=0x7faa98009a50 comp rx=0 tx=0).stop 2022-06-27T12:18:32.268 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.256+0000 7faaa5d0c700 1 -- 172.21.15.195:0/2697322682 shutdown_connections 2022-06-27T12:18:32.268 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.256+0000 7faaa5d0c700 1 --2- 172.21.15.195:0/2697322682 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7faa80063d30 0x7faa800661e0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:32.268 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.256+0000 7faaa5d0c700 1 --2- 172.21.15.195:0/2697322682 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faaa00c4160 0x7faaa00bb580 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:32.268 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.256+0000 7faaa5d0c700 1 --2- 172.21.15.195:0/2697322682 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faaa006f640 0x7faaa015c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:32.269 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.256+0000 7faaa5d0c700 1 --2- 172.21.15.195:0/2697322682 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7faaa00c39e0 0x7faaa00bb040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:32.269 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.256+0000 7faaa5d0c700 1 -- 172.21.15.195:0/2697322682 >> 172.21.15.195:0/2697322682 conn(0x7faaa00c6350 msgr2=0x7faaa0151b10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:18:32.269 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.256+0000 7faaa5d0c700 1 -- 172.21.15.195:0/2697322682 shutdown_connections 2022-06-27T12:18:32.269 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:32.256+0000 7faaa5d0c700 1 -- 172.21.15.195:0/2697322682 wait complete. 2022-06-27T12:18:32.270 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 114 2022-06-27T12:18:32.272 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 114, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-06-27T11:47:52.358095+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 114, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7300": { "addr": "172.21.15.202:6833/1965783412", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6832", "nonce": 1965783412, "type": "v2" }, { "addr": "172.21.15.202:6833", "nonce": 1965783412, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7300, "incarnation": 114, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 66, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-06-27T12:18:30.240443+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7300 } } } ], "standbys": [ { "addr": "172.21.15.202:6837/3305006841", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6836", "nonce": 3305006841, "type": "v2" }, { "addr": "172.21.15.202:6837", "nonce": 3305006841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6835/2060709822", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6834", "nonce": 2060709822, "type": "v2" }, { "addr": "172.21.15.195:6835", "nonce": 2060709822, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6838/1598751652", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6836", "nonce": 1598751652, "type": "v2" }, { "addr": "172.21.15.195:6838", "nonce": 1598751652, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4275, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6839/1313800661", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6837", "nonce": 1313800661, "type": "v2" }, { "addr": "172.21.15.195:6839", "nonce": 1313800661, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4278, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-06-27T12:18:34.273 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:18:34.415 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.407+0000 7faaf2bea700 1 Processor -- start 2022-06-27T12:18:34.416 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.407+0000 7faaf2bea700 1 -- start start 2022-06-27T12:18:34.416 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.408+0000 7faaf2bea700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faaec153cb0 0x7faaec156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:34.416 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.408+0000 7faaf2bea700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaec0c1a20 con 0x7faaec0c4160 2022-06-27T12:18:34.417 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.408+0000 7faaf2bea700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaec0c1130 con 0x7faaec06f640 2022-06-27T12:18:34.417 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.408+0000 7faaf2bea700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaec06ea90 con 0x7faaec153cb0 2022-06-27T12:18:34.417 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.408+0000 7faaebfff700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7faaec0c4160 0x7faaec1535a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:59118/0 (socket says 172.21.15.195:59118) 2022-06-27T12:18:34.417 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.408+0000 7faaebfff700 1 -- 172.21.15.195:0/1741308946 learned_addr learned my addr 172.21.15.195:0/1741308946 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:18:34.417 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.409+0000 7faaf1187700 1 --2- 172.21.15.195:0/1741308946 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faaec153cb0 0x7faaec156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:34.418 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.409+0000 7faaeb7fe700 1 -- 172.21.15.195:0/1741308946 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3810970796 0 0) 0x7faaec0c1a20 con 0x7faaec0c4160 2022-06-27T12:18:34.418 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.409+0000 7faaeb7fe700 1 -- 172.21.15.195:0/1741308946 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faad0003430 con 0x7faaec0c4160 2022-06-27T12:18:34.418 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.409+0000 7faaeb7fe700 1 -- 172.21.15.195:0/1741308946 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3372255001 0 0) 0x7faaec0c1130 con 0x7faaec06f640 2022-06-27T12:18:34.419 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.409+0000 7faaeb7fe700 1 -- 172.21.15.195:0/1741308946 --> v1:172.21.15.202:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faaec0c1a20 con 0x7faaec06f640 2022-06-27T12:18:34.419 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.409+0000 7faaeb7fe700 1 -- 172.21.15.195:0/1741308946 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1046483452 0 0) 0x7faad0003430 con 0x7faaec0c4160 2022-06-27T12:18:34.419 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.409+0000 7faaeb7fe700 1 -- 172.21.15.195:0/1741308946 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faaec153cb0 msgr2=0x7faaec156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:18:34.420 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.409+0000 7faaeb7fe700 1 --2- 172.21.15.195:0/1741308946 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faaec153cb0 0x7faaec156ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:34.420 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.409+0000 7faaeb7fe700 1 -- 172.21.15.195:0/1741308946 >> v1:172.21.15.202:6789/0 conn(0x7faaec06f640 legacy=0x7faaec0c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:34.421 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.409+0000 7faaeb7fe700 1 -- 172.21.15.195:0/1741308946 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaec06ed70 con 0x7faaec0c4160 2022-06-27T12:18:34.421 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.409+0000 7faaeb7fe700 1 -- 172.21.15.195:0/1741308946 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7faad8001470 con 0x7faaec0c4160 2022-06-27T12:18:34.421 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.409+0000 7faaeb7fe700 1 -- 172.21.15.195:0/1741308946 >> v1:172.21.15.195:6789/0 conn(0x7faaec0c4160 legacy=0x7faaec1535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:34.422 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.409+0000 7faaeb7fe700 1 --2- 172.21.15.195:0/1741308946 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faaec153cb0 0x7faad0003ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:34.422 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.409+0000 7faaeb7fe700 1 --2- 172.21.15.195:0/1741308946 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7faad0004090 0x7faad00046d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:34.422 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.409+0000 7faaeb7fe700 1 --2- 172.21.15.195:0/1741308946 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faad0004c10 0x7faad0005080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:34.422 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.409+0000 7faaeb7fe700 1 -- 172.21.15.195:0/1741308946 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaec06ea90 con 0x7faad0004c10 2022-06-27T12:18:34.423 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.409+0000 7faaeb7fe700 1 -- 172.21.15.195:0/1741308946 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faad000bab0 con 0x7faad0004090 2022-06-27T12:18:34.423 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.409+0000 7faaeb7fe700 1 -- 172.21.15.195:0/1741308946 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faad000bc70 con 0x7faaec153cb0 2022-06-27T12:18:34.423 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.409+0000 7faaf1187700 1 --2- 172.21.15.195:0/1741308946 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faad0004c10 0x7faad0005080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:34.423 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.409+0000 7faaf0986700 1 --2- 172.21.15.195:0/1741308946 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faaec153cb0 0x7faad0003ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:34.423 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.409+0000 7faaebfff700 1 --2- 172.21.15.195:0/1741308946 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7faad0004090 0x7faad00046d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:34.424 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.409+0000 7faaf1187700 1 -- 172.21.15.195:0/1741308946 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faaec153cb0 msgr2=0x7faad0003ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:18:34.424 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.409+0000 7faaf1187700 1 --2- 172.21.15.195:0/1741308946 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faaec153cb0 0x7faad0003ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:34.424 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.409+0000 7faaf1187700 1 -- 172.21.15.195:0/1741308946 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7faad0004090 msgr2=0x7faad00046d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:18:34.424 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.409+0000 7faaf1187700 1 --2- 172.21.15.195:0/1741308946 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7faad0004090 0x7faad00046d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:34.425 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.410+0000 7faaf1187700 1 -- 172.21.15.195:0/1741308946 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faad0003430 con 0x7faad0004c10 2022-06-27T12:18:34.425 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.410+0000 7faaf1187700 1 --2- 172.21.15.195:0/1741308946 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faad0004c10 0x7faad0005080 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7faaec065ef0 tx=0x7faae400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e9a72968148d61b server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:18:34.425 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.410+0000 7faaeb7fe700 1 -- 172.21.15.195:0/1741308946 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae400ea90 con 0x7faad0004c10 2022-06-27T12:18:34.426 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.410+0000 7faaeb7fe700 1 -- 172.21.15.195:0/1741308946 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faae400f040 con 0x7faad0004c10 2022-06-27T12:18:34.426 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.410+0000 7faaeb7fe700 1 -- 172.21.15.195:0/1741308946 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae400ee40 con 0x7faad0004c10 2022-06-27T12:18:34.426 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.410+0000 7faaf2bea700 1 -- 172.21.15.195:0/1741308946 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faad0004c10 msgr2=0x7faad0005080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:34.426 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.410+0000 7faaf2bea700 1 --2- 172.21.15.195:0/1741308946 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faad0004c10 0x7faad0005080 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7faaec065ef0 tx=0x7faae400b040 comp rx=0 tx=0).stop 2022-06-27T12:18:34.427 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.410+0000 7faaf1187700 1 -- 172.21.15.195:0/1741308946 reap_dead start 2022-06-27T12:18:34.427 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.410+0000 7faaf2bea700 1 -- 172.21.15.195:0/1741308946 shutdown_connections 2022-06-27T12:18:34.427 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.410+0000 7faaf2bea700 1 -- 172.21.15.195:0/1741308946 >> 172.21.15.195:0/1741308946 conn(0x7faaec0c6350 msgr2=0x7faaec0c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:18:34.428 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.410+0000 7faaf2bea700 1 -- 172.21.15.195:0/1741308946 shutdown_connections 2022-06-27T12:18:34.428 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.410+0000 7faaf2bea700 1 -- 172.21.15.195:0/1741308946 wait complete. 2022-06-27T12:18:34.428 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.411+0000 7faaf2bea700 1 Processor -- start 2022-06-27T12:18:34.428 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.412+0000 7faaf2bea700 1 -- start start 2022-06-27T12:18:34.429 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.412+0000 7faaf2bea700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faaec0c83a0 0x7faaec167150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:34.429 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.412+0000 7faaf2bea700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7faaec0c8770 0x7faaec0baa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:34.429 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.412+0000 7faaf2bea700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faaec0bcec0 0x7faaec0bafb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:34.429 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.412+0000 7faaf2bea700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaec167690 con 0x7faaec0c83a0 2022-06-27T12:18:34.430 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.412+0000 7faaf2bea700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaec0ba5e0 con 0x7faaec0c8770 2022-06-27T12:18:34.430 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.412+0000 7faaf2bea700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaec1560a0 con 0x7faaec0bcec0 2022-06-27T12:18:34.430 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.412+0000 7faaf0986700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faaec0c83a0 0x7faaec167150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:34.431 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.413+0000 7faaf0986700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faaec0c83a0 0x7faaec167150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:52940/0 (socket says 172.21.15.195:52940) 2022-06-27T12:18:34.431 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.413+0000 7faaf0986700 1 -- 172.21.15.195:0/1968400154 learned_addr learned my addr 172.21.15.195:0/1968400154 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:18:34.431 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.413+0000 7faaebfff700 1 --2- 172.21.15.195:0/1968400154 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7faaec0c8770 0x7faaec0baa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:34.432 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.413+0000 7faaf1187700 1 --2- 172.21.15.195:0/1968400154 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faaec0bcec0 0x7faaec0bafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:34.432 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.413+0000 7faaf0986700 1 -- 172.21.15.195:0/1968400154 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faaec0bcec0 msgr2=0x7faaec0bafb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:34.432 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.413+0000 7faaf0986700 1 --2- 172.21.15.195:0/1968400154 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faaec0bcec0 0x7faaec0bafb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:34.432 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.413+0000 7faaf0986700 1 -- 172.21.15.195:0/1968400154 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7faaec0c8770 msgr2=0x7faaec0baa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:34.432 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.413+0000 7faaf0986700 1 --2- 172.21.15.195:0/1968400154 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7faaec0c8770 0x7faaec0baa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:34.433 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.413+0000 7faaf0986700 1 -- 172.21.15.195:0/1968400154 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaec156380 con 0x7faaec0c83a0 2022-06-27T12:18:34.433 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.413+0000 7faaf0986700 1 --2- 172.21.15.195:0/1968400154 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faaec0c83a0 0x7faaec167150 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7faae000f8b0 tx=0x7faae0009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:18:34.433 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.413+0000 7faae8ff9700 1 -- 172.21.15.195:0/1968400154 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae00052e0 con 0x7faaec0c83a0 2022-06-27T12:18:34.433 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.413+0000 7faae8ff9700 1 -- 172.21.15.195:0/1968400154 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faae0003500 con 0x7faaec0c83a0 2022-06-27T12:18:34.434 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.413+0000 7faaf2bea700 1 -- 172.21.15.195:0/1968400154 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaec156660 con 0x7faaec0c83a0 2022-06-27T12:18:34.434 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.413+0000 7faae8ff9700 1 -- 172.21.15.195:0/1968400154 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae00038c0 con 0x7faaec0c83a0 2022-06-27T12:18:34.434 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.413+0000 7faaf2bea700 1 -- 172.21.15.195:0/1968400154 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faaec156c20 con 0x7faaec0c83a0 2022-06-27T12:18:34.434 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.414+0000 7faaf2bea700 1 -- 172.21.15.195:0/1968400154 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faaec156de0 con 0x7faaec0c83a0 2022-06-27T12:18:34.435 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.415+0000 7faae8ff9700 1 -- 172.21.15.195:0/1968400154 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7faae0003c30 con 0x7faaec0c83a0 2022-06-27T12:18:34.435 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.415+0000 7faae8ff9700 1 --2- 172.21.15.195:0/1968400154 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7faad4063d30 0x7faad40661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:34.435 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.415+0000 7faae8ff9700 1 -- 172.21.15.195:0/1968400154 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7750+0+0 (secure 0 0 0) 0x7faae00810d0 con 0x7faaec0c83a0 2022-06-27T12:18:34.435 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.415+0000 7faaebfff700 1 --2- 172.21.15.195:0/1968400154 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7faad4063d30 0x7faad40661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:34.435 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.415+0000 7faaebfff700 1 --2- 172.21.15.195:0/1968400154 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7faad4063d30 0x7faad40661e0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7faaec06e570 tx=0x7faad8009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:18:34.436 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.419+0000 7faae8ff9700 1 -- 172.21.15.195:0/1968400154 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7faaec156de0 con 0x7faaec0c83a0 2022-06-27T12:18:34.646 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.644+0000 7faaf2bea700 1 -- 172.21.15.195:0/1968400154 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faaec156de0 con 0x7faaec0c83a0 2022-06-27T12:18:34.647 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.645+0000 7faae8ff9700 1 -- 172.21.15.195:0/1968400154 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) v1 ==== 95+0+5370 (secure 0 0 0) 0x7faaec156de0 con 0x7faaec0c83a0 2022-06-27T12:18:34.648 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:18:34.649 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":114,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:18:30.240443+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7300},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7300":{"gid":7300,"name":"b","rank":0,"incarnation":114,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6833/1965783412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1965783412},{"type":"v1","addr":"172.21.15.202:6833","nonce":1965783412}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:18:34.653 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.646+0000 7faaf2bea700 1 -- 172.21.15.195:0/1968400154 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7faad4063d30 msgr2=0x7faad40661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:34.653 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.646+0000 7faaf2bea700 1 --2- 172.21.15.195:0/1968400154 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7faad4063d30 0x7faad40661e0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7faaec06e570 tx=0x7faad8009040 comp rx=0 tx=0).stop 2022-06-27T12:18:34.654 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.647+0000 7faaf2bea700 1 -- 172.21.15.195:0/1968400154 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faaec0c83a0 msgr2=0x7faaec167150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:34.654 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.647+0000 7faaf2bea700 1 --2- 172.21.15.195:0/1968400154 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faaec0c83a0 0x7faaec167150 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7faae000f8b0 tx=0x7faae0009040 comp rx=0 tx=0).stop 2022-06-27T12:18:34.655 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.647+0000 7faaf2bea700 1 -- 172.21.15.195:0/1968400154 shutdown_connections 2022-06-27T12:18:34.655 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.647+0000 7faaf2bea700 1 --2- 172.21.15.195:0/1968400154 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7faad4063d30 0x7faad40661e0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:34.655 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.647+0000 7faaf2bea700 1 --2- 172.21.15.195:0/1968400154 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faaec0c83a0 0x7faaec167150 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:34.656 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.647+0000 7faaf2bea700 1 --2- 172.21.15.195:0/1968400154 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faaec0bcec0 0x7faaec0bafb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:34.656 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.647+0000 7faaf2bea700 1 --2- 172.21.15.195:0/1968400154 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7faaec0c8770 0x7faaec0baa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:34.656 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.647+0000 7faaf2bea700 1 -- 172.21.15.195:0/1968400154 >> 172.21.15.195:0/1968400154 conn(0x7faaec0c6350 msgr2=0x7faaec0c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:18:34.657 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.647+0000 7faaf2bea700 1 -- 172.21.15.195:0/1968400154 shutdown_connections 2022-06-27T12:18:34.657 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:34.647+0000 7faaf2bea700 1 -- 172.21.15.195:0/1968400154 wait complete. 2022-06-27T12:18:34.658 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 114 2022-06-27T12:18:34.660 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:18:35.610 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:18:35.613 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:18:36.662 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:18:36.801 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:36.800+0000 7f12d747f700 1 Processor -- start 2022-06-27T12:18:36.801 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:36.801+0000 7f12d747f700 1 -- start start 2022-06-27T12:18:36.804 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:36.801+0000 7f12d747f700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f12d006a450 0x7f12d0063af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:36.804 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:36.801+0000 7f12d747f700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f12d01508d0 0x7f12d00bbf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:36.804 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:36.801+0000 7f12d747f700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12d0064030 con 0x7f12d0150500 2022-06-27T12:18:36.804 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:36.801+0000 7f12d747f700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12d00c4160 con 0x7f12d01508d0 2022-06-27T12:18:36.805 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:36.801+0000 7f12d747f700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12d00c4440 con 0x7f12d006a450 2022-06-27T12:18:36.805 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:36.801+0000 7f12d5a1c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f12d01508d0 0x7f12d00bbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:36.805 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:36.802+0000 7f12d521b700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f12d0150500 0x7f12d00c48d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:59136/0 (socket says 172.21.15.195:59136) 2022-06-27T12:18:36.805 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:36.802+0000 7f12d521b700 1 -- 172.21.15.195:0/2986434029 learned_addr learned my addr 172.21.15.195:0/2986434029 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:18:36.806 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:36.802+0000 7f12d4a1a700 1 --2- 172.21.15.195:0/2986434029 >> v2:172.21.15.202:3300/0 conn(0x7f12d006a450 0x7f12d0063af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:36.806 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:36.802+0000 7f12d5a1c700 1 -- 172.21.15.195:0/2986434029 >> v2:172.21.15.202:3300/0 conn(0x7f12d006a450 msgr2=0x7f12d0063af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:18:36.806 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:36.802+0000 7f12d5a1c700 1 --2- 172.21.15.195:0/2986434029 >> v2:172.21.15.202:3300/0 conn(0x7f12d006a450 0x7f12d0063af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:36.806 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:36.802+0000 7f12d5a1c700 1 -- 172.21.15.195:0/2986434029 >> v1:172.21.15.195:6789/0 conn(0x7f12d0150500 legacy=0x7f12d00c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:36.807 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:36.802+0000 7f12c3fff700 1 -- 172.21.15.195:0/2986434029 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4227810848 0 0) 0x7f12cc001290 con 0x7f12d0150500 2022-06-27T12:18:36.807 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:36.802+0000 7f12d5a1c700 1 -- 172.21.15.195:0/2986434029 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12d015a800 con 0x7f12d01508d0 2022-06-27T12:18:36.807 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:36.802+0000 7f12d5a1c700 1 --2- 172.21.15.195:0/2986434029 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f12d01508d0 0x7f12d00bbf50 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f12d006ce70 tx=0x7f12c8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=334c0a86bd5ed7f1 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:18:36.807 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:36.802+0000 7f12c3fff700 1 -- 172.21.15.195:0/2986434029 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12c800e070 con 0x7f12d01508d0 2022-06-27T12:18:36.807 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:36.802+0000 7f12c3fff700 1 -- 172.21.15.195:0/2986434029 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12c800bb60 con 0x7f12d01508d0 2022-06-27T12:18:36.808 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:36.802+0000 7f12c3fff700 1 -- 172.21.15.195:0/2986434029 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12c8012430 con 0x7f12d01508d0 2022-06-27T12:18:36.808 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:36.803+0000 7f12d747f700 1 -- 172.21.15.195:0/2986434029 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f12d01508d0 msgr2=0x7f12d00bbf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:36.808 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:36.803+0000 7f12d747f700 1 --2- 172.21.15.195:0/2986434029 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f12d01508d0 0x7f12d00bbf50 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f12d006ce70 tx=0x7f12c8009780 comp rx=0 tx=0).stop 2022-06-27T12:18:36.808 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:36.803+0000 7f12d747f700 1 -- 172.21.15.195:0/2986434029 shutdown_connections 2022-06-27T12:18:36.809 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:36.803+0000 7f12d747f700 1 --2- 172.21.15.195:0/2986434029 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f12d01508d0 0x7f12d00bbf50 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:36.809 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:36.803+0000 7f12d747f700 1 --2- 172.21.15.195:0/2986434029 >> v2:172.21.15.202:3300/0 conn(0x7f12d006a450 0x7f12d0063af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:36.809 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:36.803+0000 7f12d747f700 1 -- 172.21.15.195:0/2986434029 >> 172.21.15.195:0/2986434029 conn(0x7f12d00c9b00 msgr2=0x7f12d00c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:18:36.809 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:36.803+0000 7f12d747f700 1 -- 172.21.15.195:0/2986434029 shutdown_connections 2022-06-27T12:18:36.809 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:36.803+0000 7f12d747f700 1 -- 172.21.15.195:0/2986434029 wait complete. 2022-06-27T12:18:36.810 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:36.804+0000 7f12d747f700 1 Processor -- start 2022-06-27T12:18:36.810 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:36.804+0000 7f12d747f700 1 -- start start 2022-06-27T12:18:36.813 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:36.805+0000 7f12d747f700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f12d006a450 0x7f12d00680b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:36.813 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:36.805+0000 7f12d747f700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f12d0150500 0x7f12d0169c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:36.813 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:36.805+0000 7f12d747f700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f12d01508d0 0x7f12d016a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:36.813 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:36.805+0000 7f12d747f700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12d00c2bc0 con 0x7f12d0150500 2022-06-27T12:18:36.814 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:36.805+0000 7f12d747f700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12d01539f0 con 0x7f12d01508d0 2022-06-27T12:18:36.814 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:36.805+0000 7f12d747f700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12d00c2600 con 0x7f12d006a450 2022-06-27T12:18:36.814 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:36.805+0000 7f12d4a1a700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f12d0150500 0x7f12d0169c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:36.814 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:36.805+0000 7f12d4a1a700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f12d0150500 0x7f12d0169c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:52954/0 (socket says 172.21.15.195:52954) 2022-06-27T12:18:36.815 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:36.805+0000 7f12d4a1a700 1 -- 172.21.15.195:0/3595606854 learned_addr learned my addr 172.21.15.195:0/3595606854 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:18:36.815 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:36.805+0000 7f12d521b700 1 --2- 172.21.15.195:0/3595606854 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f12d006a450 0x7f12d00680b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:36.815 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:36.805+0000 7f12d5a1c700 1 --2- 172.21.15.195:0/3595606854 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f12d01508d0 0x7f12d016a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:36.815 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:36.805+0000 7f12d4a1a700 1 -- 172.21.15.195:0/3595606854 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f12d006a450 msgr2=0x7f12d00680b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:36.815 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:36.805+0000 7f12d4a1a700 1 --2- 172.21.15.195:0/3595606854 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f12d006a450 0x7f12d00680b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:36.816 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:36.805+0000 7f12d4a1a700 1 -- 172.21.15.195:0/3595606854 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f12d01508d0 msgr2=0x7f12d016a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:36.816 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:36.805+0000 7f12d4a1a700 1 --2- 172.21.15.195:0/3595606854 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f12d01508d0 0x7f12d016a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:36.816 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:36.805+0000 7f12d4a1a700 1 -- 172.21.15.195:0/3595606854 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12c8009a20 con 0x7f12d0150500 2022-06-27T12:18:36.816 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:36.805+0000 7f12d4a1a700 1 --2- 172.21.15.195:0/3595606854 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f12d0150500 0x7f12d0169c30 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f12c400e9a0 tx=0x7f12c4009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:18:36.817 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:36.806+0000 7f12c17fa700 1 -- 172.21.15.195:0/3595606854 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12c4013680 con 0x7f12d0150500 2022-06-27T12:18:36.817 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:36.806+0000 7f12c17fa700 1 -- 172.21.15.195:0/3595606854 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12c40137e0 con 0x7f12d0150500 2022-06-27T12:18:36.817 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:36.806+0000 7f12d747f700 1 -- 172.21.15.195:0/3595606854 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12d00c28e0 con 0x7f12d0150500 2022-06-27T12:18:36.817 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:36.806+0000 7f12c17fa700 1 -- 172.21.15.195:0/3595606854 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12c4013b60 con 0x7f12d0150500 2022-06-27T12:18:36.817 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:36.806+0000 7f12d747f700 1 -- 172.21.15.195:0/3595606854 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12d0153680 con 0x7f12d0150500 2022-06-27T12:18:36.818 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:36.807+0000 7f12d747f700 1 -- 172.21.15.195:0/3595606854 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12d00722c0 con 0x7f12d0150500 2022-06-27T12:18:36.818 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:36.807+0000 7f12c17fa700 1 -- 172.21.15.195:0/3595606854 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f12c4019030 con 0x7f12d0150500 2022-06-27T12:18:36.818 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:36.807+0000 7f12c17fa700 1 --2- 172.21.15.195:0/3595606854 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f12b0063d00 0x7f12b00661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:36.818 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:36.807+0000 7f12c17fa700 1 -- 172.21.15.195:0/3595606854 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7750+0+0 (secure 0 0 0) 0x7f12c4080de0 con 0x7f12d0150500 2022-06-27T12:18:36.819 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:36.808+0000 7f12d521b700 1 --2- 172.21.15.195:0/3595606854 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f12b0063d00 0x7f12b00661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:36.819 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:36.808+0000 7f12d521b700 1 --2- 172.21.15.195:0/3595606854 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f12b0063d00 0x7f12b00661b0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f12cc0014f0 tx=0x7f12cc0017d0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:18:36.819 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:36.812+0000 7f12c17fa700 1 -- 172.21.15.195:0/3595606854 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f12c404f330 con 0x7f12d0150500 2022-06-27T12:18:37.035 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:37.034+0000 7f12d747f700 1 -- 172.21.15.195:0/3595606854 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f12d00ba8e0 con 0x7f12d0150500 2022-06-27T12:18:37.035 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:37.035+0000 7f12c17fa700 1 -- 172.21.15.195:0/3595606854 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) v1 ==== 95+0+5370 (secure 0 0 0) 0x7f12c4017070 con 0x7f12d0150500 2022-06-27T12:18:37.036 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:18:37.036 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":114,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:18:30.240443+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7300},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7300":{"gid":7300,"name":"b","rank":0,"incarnation":114,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6833/1965783412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1965783412},{"type":"v1","addr":"172.21.15.202:6833","nonce":1965783412}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:18:37.039 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:37.037+0000 7f12d747f700 1 -- 172.21.15.195:0/3595606854 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f12b0063d00 msgr2=0x7f12b00661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:37.039 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:37.037+0000 7f12d747f700 1 --2- 172.21.15.195:0/3595606854 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f12b0063d00 0x7f12b00661b0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f12cc0014f0 tx=0x7f12cc0017d0 comp rx=0 tx=0).stop 2022-06-27T12:18:37.039 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:37.037+0000 7f12d747f700 1 -- 172.21.15.195:0/3595606854 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f12d0150500 msgr2=0x7f12d0169c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:37.039 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:37.037+0000 7f12d747f700 1 --2- 172.21.15.195:0/3595606854 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f12d0150500 0x7f12d0169c30 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f12c400e9a0 tx=0x7f12c4009060 comp rx=0 tx=0).stop 2022-06-27T12:18:37.040 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:37.037+0000 7f12d747f700 1 -- 172.21.15.195:0/3595606854 shutdown_connections 2022-06-27T12:18:37.040 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:37.037+0000 7f12d747f700 1 --2- 172.21.15.195:0/3595606854 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f12b0063d00 0x7f12b00661b0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:37.041 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:37.037+0000 7f12d747f700 1 --2- 172.21.15.195:0/3595606854 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f12d01508d0 0x7f12d016a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:37.041 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:37.037+0000 7f12d747f700 1 --2- 172.21.15.195:0/3595606854 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f12d006a450 0x7f12d00680b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:37.041 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:37.037+0000 7f12d747f700 1 --2- 172.21.15.195:0/3595606854 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f12d0150500 0x7f12d0169c30 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:37.042 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:37.037+0000 7f12d747f700 1 -- 172.21.15.195:0/3595606854 >> 172.21.15.195:0/3595606854 conn(0x7f12d00c9b00 msgr2=0x7f12d00c48d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:18:37.043 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:37.037+0000 7f12d747f700 1 -- 172.21.15.195:0/3595606854 shutdown_connections 2022-06-27T12:18:37.043 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:37.037+0000 7f12d747f700 1 -- 172.21.15.195:0/3595606854 wait complete. 2022-06-27T12:18:37.043 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 114 2022-06-27T12:18:37.049 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:18:39.054 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:18:39.211 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.207+0000 7f82acb20700 1 Processor -- start 2022-06-27T12:18:39.212 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.207+0000 7f82acb20700 1 -- start start 2022-06-27T12:18:39.212 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.208+0000 7f82acb20700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f82a806a450 0x7f82a8063af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:39.213 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.208+0000 7f82acb20700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f82a81508d0 0x7f82a80bbf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:39.213 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.208+0000 7f82acb20700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82a8064030 con 0x7f82a8150500 2022-06-27T12:18:39.213 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.208+0000 7f82acb20700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82a80c4160 con 0x7f82a806a450 2022-06-27T12:18:39.214 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.208+0000 7f82acb20700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82a80c4440 con 0x7f82a81508d0 2022-06-27T12:18:39.215 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.209+0000 7f82a5d9b700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f82a806a450 0x7f82a8063af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:39.215 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.209+0000 7f82a659c700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f82a8150500 0x7f82a80c48d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:59150/0 (socket says 172.21.15.195:59150) 2022-06-27T12:18:39.215 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.209+0000 7f82a659c700 1 -- 172.21.15.195:0/10085906 learned_addr learned my addr 172.21.15.195:0/10085906 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:18:39.216 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.209+0000 7f82a6d9d700 1 --2- 172.21.15.195:0/10085906 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f82a81508d0 0x7f82a80bbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:39.216 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.209+0000 7f82a5d9b700 1 -- 172.21.15.195:0/10085906 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f82a81508d0 msgr2=0x7f82a80bbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:18:39.216 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.209+0000 7f82a5d9b700 1 --2- 172.21.15.195:0/10085906 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f82a81508d0 0x7f82a80bbf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:39.217 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.209+0000 7f82a5d9b700 1 -- 172.21.15.195:0/10085906 >> v1:172.21.15.195:6789/0 conn(0x7f82a8150500 legacy=0x7f82a80c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:39.224 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.209+0000 7f82a559a700 1 -- 172.21.15.195:0/10085906 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 526336048 0 0) 0x7f8298001290 con 0x7f82a8150500 2022-06-27T12:18:39.225 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.209+0000 7f82a5d9b700 1 -- 172.21.15.195:0/10085906 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82a815a800 con 0x7f82a806a450 2022-06-27T12:18:39.225 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.209+0000 7f82a5d9b700 1 --2- 172.21.15.195:0/10085906 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f82a806a450 0x7f82a8063af0 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f829c0097a0 tx=0x7f829c009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f06a125bc6122b56 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:18:39.225 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.209+0000 7f82a559a700 1 -- 172.21.15.195:0/10085906 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f829c011630 con 0x7f82a806a450 2022-06-27T12:18:39.225 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.209+0000 7f82a559a700 1 -- 172.21.15.195:0/10085906 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f829c011790 con 0x7f82a806a450 2022-06-27T12:18:39.226 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.209+0000 7f82a559a700 1 -- 172.21.15.195:0/10085906 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f829c011b30 con 0x7f82a806a450 2022-06-27T12:18:39.226 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.210+0000 7f82acb20700 1 -- 172.21.15.195:0/10085906 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f82a806a450 msgr2=0x7f82a8063af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:39.226 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.210+0000 7f82acb20700 1 --2- 172.21.15.195:0/10085906 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f82a806a450 0x7f82a8063af0 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f829c0097a0 tx=0x7f829c009060 comp rx=0 tx=0).stop 2022-06-27T12:18:39.227 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.210+0000 7f82acb20700 1 -- 172.21.15.195:0/10085906 shutdown_connections 2022-06-27T12:18:39.227 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.210+0000 7f82acb20700 1 --2- 172.21.15.195:0/10085906 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f82a81508d0 0x7f82a80bbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:39.227 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.210+0000 7f82acb20700 1 --2- 172.21.15.195:0/10085906 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f82a806a450 0x7f82a8063af0 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:39.227 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.210+0000 7f82acb20700 1 -- 172.21.15.195:0/10085906 >> 172.21.15.195:0/10085906 conn(0x7f82a80c9b00 msgr2=0x7f82a80c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:18:39.228 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.210+0000 7f82acb20700 1 -- 172.21.15.195:0/10085906 shutdown_connections 2022-06-27T12:18:39.228 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.210+0000 7f82acb20700 1 -- 172.21.15.195:0/10085906 wait complete. 2022-06-27T12:18:39.228 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.211+0000 7f82acb20700 1 Processor -- start 2022-06-27T12:18:39.229 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.211+0000 7f82acb20700 1 -- start start 2022-06-27T12:18:39.229 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.212+0000 7f82acb20700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f82a806a450 0x7f82a80bd000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:39.230 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.212+0000 7f82acb20700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f82a8150500 0x7f82a816a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:39.230 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.212+0000 7f82acb20700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f82a81508d0 0x7f82a8166d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:39.231 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.212+0000 7f82acb20700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82a816a130 con 0x7f82a81508d0 2022-06-27T12:18:39.231 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.212+0000 7f82acb20700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82a8169e00 con 0x7f82a8150500 2022-06-27T12:18:39.231 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.212+0000 7f82acb20700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82a81587e0 con 0x7f82a806a450 2022-06-27T12:18:39.231 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.212+0000 7f82a6d9d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f82a81508d0 0x7f82a8166d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:39.232 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.212+0000 7f82a6d9d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f82a81508d0 0x7f82a8166d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:52972/0 (socket says 172.21.15.195:52972) 2022-06-27T12:18:39.232 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.212+0000 7f82a6d9d700 1 -- 172.21.15.195:0/3643647527 learned_addr learned my addr 172.21.15.195:0/3643647527 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:18:39.232 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.212+0000 7f82a659c700 1 --2- 172.21.15.195:0/3643647527 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f82a806a450 0x7f82a80bd000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:39.232 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.212+0000 7f82a5d9b700 1 --2- 172.21.15.195:0/3643647527 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f82a8150500 0x7f82a816a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:39.233 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.212+0000 7f82a6d9d700 1 -- 172.21.15.195:0/3643647527 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f82a806a450 msgr2=0x7f82a80bd000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:39.233 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.212+0000 7f82a6d9d700 1 --2- 172.21.15.195:0/3643647527 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f82a806a450 0x7f82a80bd000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:39.233 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.212+0000 7f82a6d9d700 1 -- 172.21.15.195:0/3643647527 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f82a8150500 msgr2=0x7f82a816a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:39.234 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.212+0000 7f82a6d9d700 1 --2- 172.21.15.195:0/3643647527 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f82a8150500 0x7f82a816a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:39.234 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.212+0000 7f82a6d9d700 1 -- 172.21.15.195:0/3643647527 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f829c0092e0 con 0x7f82a81508d0 2022-06-27T12:18:39.234 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.212+0000 7f82a6d9d700 1 --2- 172.21.15.195:0/3643647527 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f82a81508d0 0x7f82a8166d70 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f82a0009df0 tx=0x7f82a000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:18:39.235 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.212+0000 7f82967fc700 1 -- 172.21.15.195:0/3643647527 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82a000caa0 con 0x7f82a81508d0 2022-06-27T12:18:39.235 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.212+0000 7f82967fc700 1 -- 172.21.15.195:0/3643647527 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82a000f040 con 0x7f82a81508d0 2022-06-27T12:18:39.235 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.213+0000 7f82acb20700 1 -- 172.21.15.195:0/3643647527 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82a80c5f20 con 0x7f82a81508d0 2022-06-27T12:18:39.236 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.213+0000 7f82967fc700 1 -- 172.21.15.195:0/3643647527 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82a000ce50 con 0x7f82a81508d0 2022-06-27T12:18:39.237 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.213+0000 7f82acb20700 1 -- 172.21.15.195:0/3643647527 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82a8158d80 con 0x7f82a81508d0 2022-06-27T12:18:39.237 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.213+0000 7f82acb20700 1 -- 172.21.15.195:0/3643647527 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82a816a2c0 con 0x7f82a81508d0 2022-06-27T12:18:39.239 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.214+0000 7f82967fc700 1 -- 172.21.15.195:0/3643647527 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f82a0014690 con 0x7f82a81508d0 2022-06-27T12:18:39.240 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.214+0000 7f82967fc700 1 --2- 172.21.15.195:0/3643647527 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8284063d00 0x7f82840661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:39.241 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.214+0000 7f82967fc700 1 -- 172.21.15.195:0/3643647527 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7750+0+0 (secure 0 0 0) 0x7f82a00819c0 con 0x7f82a81508d0 2022-06-27T12:18:39.242 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.214+0000 7f82a659c700 1 --2- 172.21.15.195:0/3643647527 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8284063d00 0x7f82840661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:39.242 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.215+0000 7f82a659c700 1 --2- 172.21.15.195:0/3643647527 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8284063d00 0x7f82840661b0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f8298001eb0 tx=0x7f82980017d0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:18:39.242 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.219+0000 7f82967fc700 1 -- 172.21.15.195:0/3643647527 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f82a004fe20 con 0x7f82a81508d0 2022-06-27T12:18:39.446 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.445+0000 7f82acb20700 1 -- 172.21.15.195:0/3643647527 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f82a80722c0 con 0x7f82a81508d0 2022-06-27T12:18:39.446 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.445+0000 7f82967fc700 1 -- 172.21.15.195:0/3643647527 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) v1 ==== 95+0+5370 (secure 0 0 0) 0x7f82a0018070 con 0x7f82a81508d0 2022-06-27T12:18:39.447 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:18:39.447 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":114,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:18:30.240443+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7300},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7300":{"gid":7300,"name":"b","rank":0,"incarnation":114,"state":"up:replay","state_seq":1,"addr":"172.21.15.202:6833/1965783412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1965783412},{"type":"v1","addr":"172.21.15.202:6833","nonce":1965783412}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:18:39.450 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.448+0000 7f82acb20700 1 -- 172.21.15.195:0/3643647527 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8284063d00 msgr2=0x7f82840661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:39.450 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.448+0000 7f82acb20700 1 --2- 172.21.15.195:0/3643647527 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8284063d00 0x7f82840661b0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f8298001eb0 tx=0x7f82980017d0 comp rx=0 tx=0).stop 2022-06-27T12:18:39.451 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.448+0000 7f82acb20700 1 -- 172.21.15.195:0/3643647527 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f82a81508d0 msgr2=0x7f82a8166d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:39.451 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.448+0000 7f82acb20700 1 --2- 172.21.15.195:0/3643647527 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f82a81508d0 0x7f82a8166d70 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f82a0009df0 tx=0x7f82a000b040 comp rx=0 tx=0).stop 2022-06-27T12:18:39.452 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.448+0000 7f82acb20700 1 -- 172.21.15.195:0/3643647527 shutdown_connections 2022-06-27T12:18:39.452 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.448+0000 7f82acb20700 1 --2- 172.21.15.195:0/3643647527 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f8284063d00 0x7f82840661b0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:39.453 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.448+0000 7f82acb20700 1 --2- 172.21.15.195:0/3643647527 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f82a81508d0 0x7f82a8166d70 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:39.453 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.448+0000 7f82acb20700 1 --2- 172.21.15.195:0/3643647527 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f82a806a450 0x7f82a80bd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:39.454 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.448+0000 7f82acb20700 1 --2- 172.21.15.195:0/3643647527 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f82a8150500 0x7f82a816a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:39.454 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.448+0000 7f82acb20700 1 -- 172.21.15.195:0/3643647527 >> 172.21.15.195:0/3643647527 conn(0x7f82a80c9b00 msgr2=0x7f82a8067ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:18:39.455 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.448+0000 7f82acb20700 1 -- 172.21.15.195:0/3643647527 shutdown_connections 2022-06-27T12:18:39.455 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:39.448+0000 7f82acb20700 1 -- 172.21.15.195:0/3643647527 wait complete. 2022-06-27T12:18:39.455 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 114 2022-06-27T12:18:39.460 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:18:41.461 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:18:41.595 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.595+0000 7f6eedffe700 1 Processor -- start 2022-06-27T12:18:41.596 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.595+0000 7f6eedffe700 1 -- start start 2022-06-27T12:18:41.600 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.596+0000 7f6eedffe700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f6ee806f640 0x7f6ee8156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:41.601 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.596+0000 7f6eedffe700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f6ee80c4160 0x7f6ee80c1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:41.601 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.596+0000 7f6eedffe700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6ee80c39e0 0x7f6ee81535a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:41.602 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.596+0000 7f6eedffe700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ee80c1a20 con 0x7f6ee80c4160 2022-06-27T12:18:41.602 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.596+0000 7f6eedffe700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ee80c1130 con 0x7f6ee806f640 2022-06-27T12:18:41.603 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.596+0000 7f6eedffe700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee806ea90 con 0x7f6ee80c39e0 2022-06-27T12:18:41.604 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.596+0000 7f6ee6ffd700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f6ee80c4160 0x7f6ee80c1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:41.604 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.596+0000 7f6ee6ffd700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f6ee80c4160 0x7f6ee80c1eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:52984/0 (socket says 172.21.15.195:52984) 2022-06-27T12:18:41.605 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.596+0000 7f6ee6ffd700 1 -- 172.21.15.195:0/17678301 learned_addr learned my addr 172.21.15.195:0/17678301 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:18:41.605 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.596+0000 7f6ee77fe700 1 --2- 172.21.15.195:0/17678301 >> v2:172.21.15.202:3300/0 conn(0x7f6ee806f640 0x7f6ee8156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:41.605 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.596+0000 7f6ee7fff700 1 --2- 172.21.15.195:0/17678301 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6ee80c39e0 0x7f6ee81535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:41.606 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.596+0000 7f6ee6ffd700 1 -- 172.21.15.195:0/17678301 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6ee80c39e0 msgr2=0x7f6ee81535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:18:41.606 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.596+0000 7f6ee6ffd700 1 --2- 172.21.15.195:0/17678301 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6ee80c39e0 0x7f6ee81535a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:41.607 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.596+0000 7f6ee6ffd700 1 -- 172.21.15.195:0/17678301 >> v2:172.21.15.202:3300/0 conn(0x7f6ee806f640 msgr2=0x7f6ee8156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:18:41.607 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.596+0000 7f6ee6ffd700 1 --2- 172.21.15.195:0/17678301 >> v2:172.21.15.202:3300/0 conn(0x7f6ee806f640 0x7f6ee8156ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:41.608 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.596+0000 7f6ee6ffd700 1 -- 172.21.15.195:0/17678301 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ee806ed70 con 0x7f6ee80c4160 2022-06-27T12:18:41.608 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.596+0000 7f6ee7fff700 1 --2- 172.21.15.195:0/17678301 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6ee80c39e0 0x7f6ee81535a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T12:18:41.609 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.596+0000 7f6ee6ffd700 1 --2- 172.21.15.195:0/17678301 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6ee80c4160 0x7f6ee80c1eb0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f6ee806db30 tx=0x7f6ed8009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=212e4017c22b32a0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:18:41.609 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.597+0000 7f6ee67fc700 1 -- 172.21.15.195:0/17678301 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed800b6d0 con 0x7f6ee80c4160 2022-06-27T12:18:41.610 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.597+0000 7f6ee67fc700 1 -- 172.21.15.195:0/17678301 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ed800b830 con 0x7f6ee80c4160 2022-06-27T12:18:41.610 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.597+0000 7f6ee67fc700 1 -- 172.21.15.195:0/17678301 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed800bbd0 con 0x7f6ee80c4160 2022-06-27T12:18:41.610 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.597+0000 7f6eedffe700 1 -- 172.21.15.195:0/17678301 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6ee80c4160 msgr2=0x7f6ee80c1eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:41.611 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.597+0000 7f6eedffe700 1 --2- 172.21.15.195:0/17678301 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6ee80c4160 0x7f6ee80c1eb0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f6ee806db30 tx=0x7f6ed8009120 comp rx=0 tx=0).stop 2022-06-27T12:18:41.611 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.597+0000 7f6eedffe700 1 -- 172.21.15.195:0/17678301 shutdown_connections 2022-06-27T12:18:41.612 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.597+0000 7f6eedffe700 1 --2- 172.21.15.195:0/17678301 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6ee80c39e0 0x7f6ee81535a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:41.613 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.597+0000 7f6eedffe700 1 --2- 172.21.15.195:0/17678301 >> v2:172.21.15.202:3300/0 conn(0x7f6ee806f640 0x7f6ee8156ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:41.613 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.597+0000 7f6eedffe700 1 --2- 172.21.15.195:0/17678301 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6ee80c4160 0x7f6ee80c1eb0 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:41.613 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.597+0000 7f6eedffe700 1 -- 172.21.15.195:0/17678301 >> 172.21.15.195:0/17678301 conn(0x7f6ee80c6350 msgr2=0x7f6ee80c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:18:41.614 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.597+0000 7f6eedffe700 1 -- 172.21.15.195:0/17678301 shutdown_connections 2022-06-27T12:18:41.615 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.597+0000 7f6eedffe700 1 -- 172.21.15.195:0/17678301 wait complete. 2022-06-27T12:18:41.615 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.598+0000 7f6eedffe700 1 Processor -- start 2022-06-27T12:18:41.616 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.599+0000 7f6eedffe700 1 -- start start 2022-06-27T12:18:41.616 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.599+0000 7f6eedffe700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6ee806f640 0x7f6ee8158920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:41.624 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.599+0000 7f6eedffe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6ee80c39e0 0x7f6ee80c8930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:41.624 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.599+0000 7f6eedffe700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6ee80c4160 0x7f6ee80c8e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:41.625 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.599+0000 7f6eedffe700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee8152a80 con 0x7f6ee80c39e0 2022-06-27T12:18:41.625 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.599+0000 7f6eedffe700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee816a3f0 con 0x7f6ee806f640 2022-06-27T12:18:41.626 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.599+0000 7f6eedffe700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee81480c0 con 0x7f6ee80c4160 2022-06-27T12:18:41.626 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.599+0000 7f6ee6ffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6ee80c39e0 0x7f6ee80c8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:41.627 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.599+0000 7f6ee6ffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6ee80c39e0 0x7f6ee80c8930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:52992/0 (socket says 172.21.15.195:52992) 2022-06-27T12:18:41.627 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.599+0000 7f6ee6ffd700 1 -- 172.21.15.195:0/1598782835 learned_addr learned my addr 172.21.15.195:0/1598782835 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:18:41.627 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.599+0000 7f6ee77fe700 1 --2- 172.21.15.195:0/1598782835 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6ee806f640 0x7f6ee8158920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:41.628 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.599+0000 7f6ee7fff700 1 --2- 172.21.15.195:0/1598782835 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6ee80c4160 0x7f6ee80c8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:41.628 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.600+0000 7f6ee6ffd700 1 -- 172.21.15.195:0/1598782835 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6ee80c4160 msgr2=0x7f6ee80c8e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:41.629 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.600+0000 7f6ee6ffd700 1 --2- 172.21.15.195:0/1598782835 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6ee80c4160 0x7f6ee80c8e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:41.629 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.600+0000 7f6ee6ffd700 1 -- 172.21.15.195:0/1598782835 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6ee806f640 msgr2=0x7f6ee8158920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:41.630 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.600+0000 7f6ee6ffd700 1 --2- 172.21.15.195:0/1598782835 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6ee806f640 0x7f6ee8158920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:41.630 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.600+0000 7f6ee6ffd700 1 -- 172.21.15.195:0/1598782835 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ed80093a0 con 0x7f6ee80c39e0 2022-06-27T12:18:41.631 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.600+0000 7f6ee6ffd700 1 --2- 172.21.15.195:0/1598782835 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6ee80c39e0 0x7f6ee80c8930 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f6ee80650f0 tx=0x7f6ed8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:18:41.632 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.600+0000 7f6edf7fe700 1 -- 172.21.15.195:0/1598782835 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed8003730 con 0x7f6ee80c39e0 2022-06-27T12:18:41.632 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.600+0000 7f6edf7fe700 1 -- 172.21.15.195:0/1598782835 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ed8003890 con 0x7f6ee80c39e0 2022-06-27T12:18:41.633 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.600+0000 7f6eedffe700 1 -- 172.21.15.195:0/1598782835 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ee815b680 con 0x7f6ee80c39e0 2022-06-27T12:18:41.633 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.600+0000 7f6edf7fe700 1 -- 172.21.15.195:0/1598782835 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed8003c30 con 0x7f6ee80c39e0 2022-06-27T12:18:41.634 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.600+0000 7f6eedffe700 1 -- 172.21.15.195:0/1598782835 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ee815b8e0 con 0x7f6ee80c39e0 2022-06-27T12:18:41.634 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.601+0000 7f6eedffe700 1 -- 172.21.15.195:0/1598782835 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ee816a5a0 con 0x7f6ee80c39e0 2022-06-27T12:18:41.635 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.601+0000 7f6edf7fe700 1 -- 172.21.15.195:0/1598782835 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f6ed802b030 con 0x7f6ee80c39e0 2022-06-27T12:18:41.635 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.602+0000 7f6edf7fe700 1 --2- 172.21.15.195:0/1598782835 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6ecc063d00 0x7f6ecc0661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:41.636 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.602+0000 7f6edf7fe700 1 -- 172.21.15.195:0/1598782835 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7750+0+0 (secure 0 0 0) 0x7f6ed8082530 con 0x7f6ee80c39e0 2022-06-27T12:18:41.636 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.602+0000 7f6ee77fe700 1 --2- 172.21.15.195:0/1598782835 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6ecc063d00 0x7f6ecc0661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:41.637 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.602+0000 7f6ee77fe700 1 --2- 172.21.15.195:0/1598782835 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6ecc063d00 0x7f6ecc0661b0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f6ed0004560 tx=0x7f6ed0008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:18:41.637 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.606+0000 7f6edf7fe700 1 -- 172.21.15.195:0/1598782835 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f6ee816a5a0 con 0x7f6ee80c39e0 2022-06-27T12:18:41.836 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.835+0000 7f6eedffe700 1 -- 172.21.15.195:0/1598782835 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ee816a5a0 con 0x7f6ee80c39e0 2022-06-27T12:18:41.837 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.836+0000 7f6edf7fe700 1 -- 172.21.15.195:0/1598782835 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+5374 (secure 0 0 0) 0x7f6ee816a5a0 con 0x7f6ee80c39e0 2022-06-27T12:18:41.838 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:18:41.839 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":115,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:18:40.339756+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7300},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7300":{"gid":7300,"name":"b","rank":0,"incarnation":114,"state":"up:reconnect","state_seq":17,"addr":"172.21.15.202:6833/1965783412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1965783412},{"type":"v1","addr":"172.21.15.202:6833","nonce":1965783412}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:18:41.841 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.838+0000 7f6eedffe700 1 -- 172.21.15.195:0/1598782835 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6ecc063d00 msgr2=0x7f6ecc0661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:41.841 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.838+0000 7f6eedffe700 1 --2- 172.21.15.195:0/1598782835 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6ecc063d00 0x7f6ecc0661b0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f6ed0004560 tx=0x7f6ed0008040 comp rx=0 tx=0).stop 2022-06-27T12:18:41.842 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.838+0000 7f6eedffe700 1 -- 172.21.15.195:0/1598782835 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6ee80c39e0 msgr2=0x7f6ee80c8930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:41.842 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.838+0000 7f6eedffe700 1 --2- 172.21.15.195:0/1598782835 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6ee80c39e0 0x7f6ee80c8930 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f6ee80650f0 tx=0x7f6ed8017000 comp rx=0 tx=0).stop 2022-06-27T12:18:41.842 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.838+0000 7f6eedffe700 1 -- 172.21.15.195:0/1598782835 shutdown_connections 2022-06-27T12:18:41.843 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.838+0000 7f6eedffe700 1 --2- 172.21.15.195:0/1598782835 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6ecc063d00 0x7f6ecc0661b0 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:41.843 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.838+0000 7f6eedffe700 1 --2- 172.21.15.195:0/1598782835 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6ee80c4160 0x7f6ee80c8e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:41.843 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.838+0000 7f6eedffe700 1 --2- 172.21.15.195:0/1598782835 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6ee806f640 0x7f6ee8158920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:41.843 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.838+0000 7f6eedffe700 1 --2- 172.21.15.195:0/1598782835 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6ee80c39e0 0x7f6ee80c8930 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:41.844 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.838+0000 7f6eedffe700 1 -- 172.21.15.195:0/1598782835 >> 172.21.15.195:0/1598782835 conn(0x7f6ee80c6350 msgr2=0x7f6ee81504b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:18:41.844 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.838+0000 7f6eedffe700 1 -- 172.21.15.195:0/1598782835 shutdown_connections 2022-06-27T12:18:41.844 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:41.838+0000 7f6eedffe700 1 -- 172.21.15.195:0/1598782835 wait complete. 2022-06-27T12:18:41.844 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 115 2022-06-27T12:18:41.850 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:18:43.851 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:18:43.994 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:43.985+0000 7f442d755700 1 Processor -- start 2022-06-27T12:18:43.995 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:43.985+0000 7f442d755700 1 -- start start 2022-06-27T12:18:43.995 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:43.986+0000 7f442d755700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f442806e850 0x7f44281570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:43.995 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:43.986+0000 7f442d755700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f44280c34a0 0x7f44280c2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:43.995 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:43.986+0000 7f442d755700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f442806fba0 con 0x7f442806fef0 2022-06-27T12:18:43.996 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:43.986+0000 7f442d755700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44281575e0 con 0x7f44280c34a0 2022-06-27T12:18:43.996 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:43.986+0000 7f442d755700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f442806f5e0 con 0x7f442806e850 2022-06-27T12:18:43.996 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:43.986+0000 7f44277fe700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f44280c34a0 0x7f44280c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:43.997 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:43.986+0000 7f44267fc700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f442806fef0 0x7f4428069de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:59188/0 (socket says 172.21.15.195:59188) 2022-06-27T12:18:43.997 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:43.986+0000 7f44267fc700 1 -- 172.21.15.195:0/1880475089 learned_addr learned my addr 172.21.15.195:0/1880475089 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:18:43.997 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:43.987+0000 7f4426ffd700 1 --2- 172.21.15.195:0/1880475089 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f442806e850 0x7f44281570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:43.997 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:43.987+0000 7f44277fe700 1 -- 172.21.15.195:0/1880475089 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f442806e850 msgr2=0x7f44281570a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:18:43.998 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:43.987+0000 7f44277fe700 1 --2- 172.21.15.195:0/1880475089 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f442806e850 0x7f44281570a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:43.998 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:43.987+0000 7f44277fe700 1 -- 172.21.15.195:0/1880475089 >> v1:172.21.15.195:6789/0 conn(0x7f442806fef0 legacy=0x7f4428069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:43.998 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:43.987+0000 7f44277fe700 1 -- 172.21.15.195:0/1880475089 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f442806f8c0 con 0x7f44280c34a0 2022-06-27T12:18:43.998 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:43.987+0000 7f4425ffb700 1 -- 172.21.15.195:0/1880475089 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1042439417 0 0) 0x7f442806fba0 con 0x7f442806fef0 2022-06-27T12:18:43.999 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:43.987+0000 7f4426ffd700 1 --2- 172.21.15.195:0/1880475089 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f442806e850 0x7f44281570a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:18:43.999 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:43.987+0000 7f44277fe700 1 --2- 172.21.15.195:0/1880475089 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f44280c34a0 0x7f44280c2920 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f4420009e60 tx=0x7f4420009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6e8149181a0258c server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:18:43.999 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:43.987+0000 7f4425ffb700 1 -- 172.21.15.195:0/1880475089 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f442000e070 con 0x7f44280c34a0 2022-06-27T12:18:43.999 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:43.987+0000 7f4425ffb700 1 -- 172.21.15.195:0/1880475089 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f442000bcc0 con 0x7f44280c34a0 2022-06-27T12:18:43.999 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:43.987+0000 7f4425ffb700 1 -- 172.21.15.195:0/1880475089 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4420012680 con 0x7f44280c34a0 2022-06-27T12:18:44.000 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:43.988+0000 7f442d755700 1 -- 172.21.15.195:0/1880475089 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f44280c34a0 msgr2=0x7f44280c2920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:44.000 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:43.988+0000 7f442d755700 1 --2- 172.21.15.195:0/1880475089 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f44280c34a0 0x7f44280c2920 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f4420009e60 tx=0x7f4420009780 comp rx=0 tx=0).stop 2022-06-27T12:18:44.000 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:43.988+0000 7f442d755700 1 -- 172.21.15.195:0/1880475089 shutdown_connections 2022-06-27T12:18:44.001 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:43.988+0000 7f442d755700 1 --2- 172.21.15.195:0/1880475089 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f44280c34a0 0x7f44280c2920 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:44.001 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:43.988+0000 7f442d755700 1 --2- 172.21.15.195:0/1880475089 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f442806e850 0x7f44281570a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:44.001 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:43.988+0000 7f442d755700 1 -- 172.21.15.195:0/1880475089 >> 172.21.15.195:0/1880475089 conn(0x7f44280c6350 msgr2=0x7f44280c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:18:44.001 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:43.988+0000 7f442d755700 1 -- 172.21.15.195:0/1880475089 shutdown_connections 2022-06-27T12:18:44.002 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:43.988+0000 7f442d755700 1 -- 172.21.15.195:0/1880475089 wait complete. 2022-06-27T12:18:44.002 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:43.989+0000 7f442d755700 1 Processor -- start 2022-06-27T12:18:44.002 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:43.989+0000 7f442d755700 1 -- start start 2022-06-27T12:18:44.002 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:43.990+0000 7f442d755700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f442806e850 0x7f44281525a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:44.002 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:43.990+0000 7f442d755700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f44280c34a0 0x7f44280c8740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:44.003 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:43.990+0000 7f442d755700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f44280ba630 0x7f44281667f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:44.003 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:43.990+0000 7f442d755700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4428151150 con 0x7f442806e850 2022-06-27T12:18:44.003 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:43.990+0000 7f442d755700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44280c32a0 con 0x7f44280c34a0 2022-06-27T12:18:44.004 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:43.990+0000 7f442d755700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44280bd210 con 0x7f44280ba630 2022-06-27T12:18:44.004 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:43.990+0000 7f4426ffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f442806e850 0x7f44281525a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:44.004 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:43.990+0000 7f4426ffd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f442806e850 0x7f44281525a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:53004/0 (socket says 172.21.15.195:53004) 2022-06-27T12:18:44.005 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:43.990+0000 7f4426ffd700 1 -- 172.21.15.195:0/508444291 learned_addr learned my addr 172.21.15.195:0/508444291 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:18:44.005 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:43.990+0000 7f44267fc700 1 --2- 172.21.15.195:0/508444291 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f44280c34a0 0x7f44280c8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:44.010 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:43.990+0000 7f44277fe700 1 --2- 172.21.15.195:0/508444291 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f44280ba630 0x7f44281667f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:44.010 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:43.990+0000 7f4426ffd700 1 -- 172.21.15.195:0/508444291 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f44280ba630 msgr2=0x7f44281667f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:44.011 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:43.990+0000 7f4426ffd700 1 --2- 172.21.15.195:0/508444291 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f44280ba630 0x7f44281667f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:44.012 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:43.990+0000 7f4426ffd700 1 -- 172.21.15.195:0/508444291 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f44280c34a0 msgr2=0x7f44280c8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:44.013 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:43.990+0000 7f4426ffd700 1 --2- 172.21.15.195:0/508444291 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f44280c34a0 0x7f44280c8740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:44.013 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:43.990+0000 7f4426ffd700 1 -- 172.21.15.195:0/508444291 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4420009a20 con 0x7f442806e850 2022-06-27T12:18:44.014 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:43.990+0000 7f44267fc700 1 --2- 172.21.15.195:0/508444291 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f44280c34a0 0x7f44280c8740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:18:44.014 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:43.990+0000 7f4426ffd700 1 --2- 172.21.15.195:0/508444291 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f442806e850 0x7f44281525a0 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f441c00d8c0 tx=0x7f441c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:18:44.015 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:43.990+0000 7f440effd700 1 -- 172.21.15.195:0/508444291 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f441c004030 con 0x7f442806e850 2022-06-27T12:18:44.015 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:43.990+0000 7f440effd700 1 -- 172.21.15.195:0/508444291 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f441c004190 con 0x7f442806e850 2022-06-27T12:18:44.016 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:43.990+0000 7f442d755700 1 -- 172.21.15.195:0/508444291 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44280c2d20 con 0x7f442806e850 2022-06-27T12:18:44.016 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:43.990+0000 7f440effd700 1 -- 172.21.15.195:0/508444291 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f441c009550 con 0x7f442806e850 2022-06-27T12:18:44.016 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:43.990+0000 7f442d755700 1 -- 172.21.15.195:0/508444291 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44280c2fc0 con 0x7f442806e850 2022-06-27T12:18:44.017 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:43.991+0000 7f442d755700 1 -- 172.21.15.195:0/508444291 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44280722c0 con 0x7f442806e850 2022-06-27T12:18:44.017 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:43.992+0000 7f440effd700 1 -- 172.21.15.195:0/508444291 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f441c01a020 con 0x7f442806e850 2022-06-27T12:18:44.018 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:43.992+0000 7f440effd700 1 --2- 172.21.15.195:0/508444291 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4408063d00 0x7f44080661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:44.018 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:43.992+0000 7f440effd700 1 -- 172.21.15.195:0/508444291 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7750+0+0 (secure 0 0 0) 0x7f441c0811f0 con 0x7f442806e850 2022-06-27T12:18:44.019 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:43.992+0000 7f44267fc700 1 --2- 172.21.15.195:0/508444291 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4408063d00 0x7f44080661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:44.019 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:43.993+0000 7f44267fc700 1 --2- 172.21.15.195:0/508444291 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4408063d00 0x7f44080661b0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f44280c4360 tx=0x7f4418007bd0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:18:44.020 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:43.996+0000 7f440effd700 1 -- 172.21.15.195:0/508444291 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f441c04f740 con 0x7f442806e850 2022-06-27T12:18:44.226 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:44.226+0000 7f442d755700 1 -- 172.21.15.195:0/508444291 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f44280bb2f0 con 0x7f442806e850 2022-06-27T12:18:44.228 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:44.227+0000 7f440effd700 1 -- 172.21.15.195:0/508444291 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+5374 (secure 0 0 0) 0x7f441c052d60 con 0x7f442806e850 2022-06-27T12:18:44.228 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:18:44.229 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":115,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:18:40.339756+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7300},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7300":{"gid":7300,"name":"b","rank":0,"incarnation":114,"state":"up:reconnect","state_seq":17,"addr":"172.21.15.202:6833/1965783412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1965783412},{"type":"v1","addr":"172.21.15.202:6833","nonce":1965783412}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:18:44.232 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:44.228+0000 7f442d755700 1 -- 172.21.15.195:0/508444291 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4408063d00 msgr2=0x7f44080661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:44.232 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:44.228+0000 7f442d755700 1 --2- 172.21.15.195:0/508444291 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4408063d00 0x7f44080661b0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f44280c4360 tx=0x7f4418007bd0 comp rx=0 tx=0).stop 2022-06-27T12:18:44.232 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:44.228+0000 7f442d755700 1 -- 172.21.15.195:0/508444291 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f442806e850 msgr2=0x7f44281525a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:44.233 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:44.228+0000 7f442d755700 1 --2- 172.21.15.195:0/508444291 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f442806e850 0x7f44281525a0 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f441c00d8c0 tx=0x7f441c008040 comp rx=0 tx=0).stop 2022-06-27T12:18:44.233 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:44.229+0000 7f442d755700 1 -- 172.21.15.195:0/508444291 shutdown_connections 2022-06-27T12:18:44.233 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:44.229+0000 7f442d755700 1 --2- 172.21.15.195:0/508444291 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4408063d00 0x7f44080661b0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:44.233 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:44.229+0000 7f442d755700 1 --2- 172.21.15.195:0/508444291 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f442806e850 0x7f44281525a0 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:44.234 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:44.229+0000 7f442d755700 1 --2- 172.21.15.195:0/508444291 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f44280ba630 0x7f44281667f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:44.234 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:44.229+0000 7f442d755700 1 --2- 172.21.15.195:0/508444291 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f44280c34a0 0x7f44280c8740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:44.234 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:44.229+0000 7f442d755700 1 -- 172.21.15.195:0/508444291 >> 172.21.15.195:0/508444291 conn(0x7f44280c6350 msgr2=0x7f44280c4960 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:18:44.234 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:44.229+0000 7f442d755700 1 -- 172.21.15.195:0/508444291 shutdown_connections 2022-06-27T12:18:44.234 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:44.229+0000 7f442d755700 1 -- 172.21.15.195:0/508444291 wait complete. 2022-06-27T12:18:44.235 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 115 2022-06-27T12:18:44.242 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 115, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-06-27T11:47:52.358095+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 115, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7300": { "addr": "172.21.15.202:6833/1965783412", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6832", "nonce": 1965783412, "type": "v2" }, { "addr": "172.21.15.202:6833", "nonce": 1965783412, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7300, "incarnation": 114, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:reconnect", "state_seq": 17 } }, "last_failure": 0, "last_failure_osd_epoch": 66, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-06-27T12:18:40.339756+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7300 } } } ], "standbys": [ { "addr": "172.21.15.202:6837/3305006841", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6836", "nonce": 3305006841, "type": "v2" }, { "addr": "172.21.15.202:6837", "nonce": 3305006841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6835/2060709822", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6834", "nonce": 2060709822, "type": "v2" }, { "addr": "172.21.15.195:6835", "nonce": 2060709822, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6838/1598751652", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6836", "nonce": 1598751652, "type": "v2" }, { "addr": "172.21.15.195:6838", "nonce": 1598751652, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4275, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6839/1313800661", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6837", "nonce": 1313800661, "type": "v2" }, { "addr": "172.21.15.195:6839", "nonce": 1313800661, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4278, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-06-27T12:18:46.244 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:18:46.395 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.378+0000 7faea7249700 1 Processor -- start 2022-06-27T12:18:46.396 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.378+0000 7faea7249700 1 -- start start 2022-06-27T12:18:46.396 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.379+0000 7faea7249700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faea00c1d60 0x7faea006d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:46.397 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.379+0000 7faea7249700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faea006b500 con 0x7faea006d090 2022-06-27T12:18:46.397 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.379+0000 7faea7249700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faea0155010 con 0x7faea006d9a0 2022-06-27T12:18:46.398 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.379+0000 7faea7249700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faea006ba30 con 0x7faea00c1d60 2022-06-27T12:18:46.398 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.379+0000 7faea4fe5700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7faea006d090 0x7faea00c5b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:59200/0 (socket says 172.21.15.195:59200) 2022-06-27T12:18:46.398 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.379+0000 7faea4fe5700 1 -- 172.21.15.195:0/2052115929 learned_addr learned my addr 172.21.15.195:0/2052115929 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:18:46.398 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.379+0000 7fae9ffff700 1 --2- 172.21.15.195:0/2052115929 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faea00c1d60 0x7faea006d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:46.399 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.379+0000 7fae9f7fe700 1 -- 172.21.15.195:0/2052115929 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1750127675 0 0) 0x7faea006b500 con 0x7faea006d090 2022-06-27T12:18:46.399 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.379+0000 7fae9f7fe700 1 -- 172.21.15.195:0/2052115929 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae84003430 con 0x7faea006d090 2022-06-27T12:18:46.399 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.379+0000 7fae9f7fe700 1 -- 172.21.15.195:0/2052115929 <== mon.1 v1:172.21.15.202:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2908654371 0 0) 0x7faea0155010 con 0x7faea006d9a0 2022-06-27T12:18:46.399 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.379+0000 7fae9f7fe700 1 -- 172.21.15.195:0/2052115929 --> v1:172.21.15.202:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faea006b500 con 0x7faea006d9a0 2022-06-27T12:18:46.400 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.379+0000 7fae9f7fe700 1 -- 172.21.15.195:0/2052115929 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 293313095 0 0) 0x7fae84003430 con 0x7faea006d090 2022-06-27T12:18:46.400 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.379+0000 7fae9f7fe700 1 -- 172.21.15.195:0/2052115929 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faea00c1d60 msgr2=0x7faea006d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:18:46.400 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.379+0000 7fae9f7fe700 1 --2- 172.21.15.195:0/2052115929 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faea00c1d60 0x7faea006d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:46.400 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.379+0000 7fae9f7fe700 1 -- 172.21.15.195:0/2052115929 >> v1:172.21.15.202:6789/0 conn(0x7faea006d9a0 legacy=0x7faea0151370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:46.401 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.379+0000 7fae9f7fe700 1 -- 172.21.15.195:0/2052115929 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faea006b750 con 0x7faea006d090 2022-06-27T12:18:46.401 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.379+0000 7fae9f7fe700 1 -- 172.21.15.195:0/2052115929 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7fae94001470 con 0x7faea006d090 2022-06-27T12:18:46.401 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.379+0000 7fae9f7fe700 1 -- 172.21.15.195:0/2052115929 >> v1:172.21.15.195:6789/0 conn(0x7faea006d090 legacy=0x7faea00c5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:46.402 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.379+0000 7fae9f7fe700 1 --2- 172.21.15.195:0/2052115929 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fae84003ae0 0x7fae84003ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:46.402 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.379+0000 7fae9f7fe700 1 --2- 172.21.15.195:0/2052115929 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fae840046d0 0x7fae84004b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:46.403 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.380+0000 7fae9f7fe700 1 --2- 172.21.15.195:0/2052115929 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fae84005080 0x7fae8400b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:46.403 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.380+0000 7fae9f7fe700 1 -- 172.21.15.195:0/2052115929 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faea006ba30 con 0x7fae840046d0 2022-06-27T12:18:46.404 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.380+0000 7fae9f7fe700 1 -- 172.21.15.195:0/2052115929 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae8400bee0 con 0x7fae84005080 2022-06-27T12:18:46.404 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.380+0000 7fae9f7fe700 1 -- 172.21.15.195:0/2052115929 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae8400c0a0 con 0x7fae84003ae0 2022-06-27T12:18:46.405 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.380+0000 7fae9ffff700 1 --2- 172.21.15.195:0/2052115929 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faea00c1d60 0x7faea006d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:18:46.405 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.380+0000 7faea4fe5700 1 --2- 172.21.15.195:0/2052115929 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fae84003ae0 0x7fae84003ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:46.406 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.380+0000 7faea57e6700 1 --2- 172.21.15.195:0/2052115929 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fae84005080 0x7fae8400b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:46.406 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.380+0000 7fae9ffff700 1 --2- 172.21.15.195:0/2052115929 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fae840046d0 0x7fae84004b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:46.407 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.380+0000 7fae9ffff700 1 -- 172.21.15.195:0/2052115929 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fae84003ae0 msgr2=0x7fae84003ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:18:46.407 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.380+0000 7fae9ffff700 1 --2- 172.21.15.195:0/2052115929 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fae84003ae0 0x7fae84003ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:46.408 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.380+0000 7fae9ffff700 1 -- 172.21.15.195:0/2052115929 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fae84005080 msgr2=0x7fae8400b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:18:46.408 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.380+0000 7fae9ffff700 1 --2- 172.21.15.195:0/2052115929 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fae84005080 0x7fae8400b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:46.409 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.380+0000 7fae9ffff700 1 -- 172.21.15.195:0/2052115929 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae84003430 con 0x7fae840046d0 2022-06-27T12:18:46.409 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.380+0000 7fae9ffff700 1 --2- 172.21.15.195:0/2052115929 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fae840046d0 0x7fae84004b40 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7faea006f5d0 tx=0x7fae8c009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5ef6433777c08c9 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:18:46.410 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.380+0000 7fae9f7fe700 1 -- 172.21.15.195:0/2052115929 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae8c009510 con 0x7fae840046d0 2022-06-27T12:18:46.411 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.380+0000 7fae9f7fe700 1 -- 172.21.15.195:0/2052115929 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae8c004030 con 0x7fae840046d0 2022-06-27T12:18:46.411 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.380+0000 7fae9f7fe700 1 -- 172.21.15.195:0/2052115929 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae8c0043d0 con 0x7fae840046d0 2022-06-27T12:18:46.412 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.381+0000 7faea7249700 1 -- 172.21.15.195:0/2052115929 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fae840046d0 msgr2=0x7fae84004b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:46.412 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.381+0000 7faea7249700 1 --2- 172.21.15.195:0/2052115929 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fae840046d0 0x7fae84004b40 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7faea006f5d0 tx=0x7fae8c009060 comp rx=0 tx=0).stop 2022-06-27T12:18:46.413 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.381+0000 7faea57e6700 1 -- 172.21.15.195:0/2052115929 reap_dead start 2022-06-27T12:18:46.413 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.381+0000 7faea7249700 1 -- 172.21.15.195:0/2052115929 shutdown_connections 2022-06-27T12:18:46.413 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.381+0000 7faea7249700 1 -- 172.21.15.195:0/2052115929 >> 172.21.15.195:0/2052115929 conn(0x7faea0063af0 msgr2=0x7faea0063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:18:46.414 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.381+0000 7faea7249700 1 -- 172.21.15.195:0/2052115929 shutdown_connections 2022-06-27T12:18:46.414 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.381+0000 7faea7249700 1 -- 172.21.15.195:0/2052115929 wait complete. 2022-06-27T12:18:46.415 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.382+0000 7faea7249700 1 Processor -- start 2022-06-27T12:18:46.415 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.383+0000 7faea7249700 1 -- start start 2022-06-27T12:18:46.415 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.383+0000 7faea7249700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faea00c9dc0 0x7faea00ca190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:46.416 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.383+0000 7faea7249700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faea01503a0 0x7faea00b7360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:46.416 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.384+0000 7faea7249700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7faea00b78a0 0x7faea015abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:46.417 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.384+0000 7faea7249700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faea00ca6d0 con 0x7faea01503a0 2022-06-27T12:18:46.418 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.384+0000 7faea7249700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faea0167840 con 0x7faea00b78a0 2022-06-27T12:18:46.419 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.384+0000 7faea7249700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faea01686a0 con 0x7faea00c9dc0 2022-06-27T12:18:46.419 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.384+0000 7fae9ffff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faea01503a0 0x7faea00b7360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:46.420 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.384+0000 7fae9ffff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faea01503a0 0x7faea00b7360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:53026/0 (socket says 172.21.15.195:53026) 2022-06-27T12:18:46.420 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.384+0000 7fae9ffff700 1 -- 172.21.15.195:0/2582122073 learned_addr learned my addr 172.21.15.195:0/2582122073 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:18:46.421 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.384+0000 7faea4fe5700 1 --2- 172.21.15.195:0/2582122073 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faea00c9dc0 0x7faea00ca190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:46.421 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.384+0000 7faea57e6700 1 --2- 172.21.15.195:0/2582122073 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7faea00b78a0 0x7faea015abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:46.422 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.384+0000 7fae9ffff700 1 -- 172.21.15.195:0/2582122073 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faea00c9dc0 msgr2=0x7faea00ca190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:46.422 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.384+0000 7fae9ffff700 1 --2- 172.21.15.195:0/2582122073 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faea00c9dc0 0x7faea00ca190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:46.423 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.384+0000 7fae9ffff700 1 -- 172.21.15.195:0/2582122073 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7faea00b78a0 msgr2=0x7faea015abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:46.423 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.384+0000 7fae9ffff700 1 --2- 172.21.15.195:0/2582122073 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7faea00b78a0 0x7faea015abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:46.423 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.384+0000 7fae9ffff700 1 -- 172.21.15.195:0/2582122073 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faea0154230 con 0x7faea01503a0 2022-06-27T12:18:46.424 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.384+0000 7faea4fe5700 1 --2- 172.21.15.195:0/2582122073 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faea00c9dc0 0x7faea00ca190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T12:18:46.424 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.384+0000 7faea57e6700 1 --2- 172.21.15.195:0/2582122073 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7faea00b78a0 0x7faea015abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T12:18:46.425 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.384+0000 7fae9ffff700 1 --2- 172.21.15.195:0/2582122073 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faea01503a0 0x7faea00b7360 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7fae8c00e4b0 tx=0x7fae8c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:18:46.425 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.384+0000 7fae9cff9700 1 -- 172.21.15.195:0/2582122073 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae8c022070 con 0x7faea01503a0 2022-06-27T12:18:46.426 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.384+0000 7fae9cff9700 1 -- 172.21.15.195:0/2582122073 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae8c0038b0 con 0x7faea01503a0 2022-06-27T12:18:46.426 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.384+0000 7faea7249700 1 -- 172.21.15.195:0/2582122073 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faea01544d0 con 0x7faea01503a0 2022-06-27T12:18:46.426 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.384+0000 7fae9cff9700 1 -- 172.21.15.195:0/2582122073 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae8c003c70 con 0x7faea01503a0 2022-06-27T12:18:46.426 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.384+0000 7faea7249700 1 -- 172.21.15.195:0/2582122073 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faea0167d70 con 0x7faea01503a0 2022-06-27T12:18:46.427 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.385+0000 7faea7249700 1 -- 172.21.15.195:0/2582122073 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faea00b7c70 con 0x7faea01503a0 2022-06-27T12:18:46.427 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.386+0000 7fae9cff9700 1 -- 172.21.15.195:0/2582122073 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fae8c013470 con 0x7faea01503a0 2022-06-27T12:18:46.427 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.386+0000 7fae9cff9700 1 --2- 172.21.15.195:0/2582122073 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fae88063d30 0x7fae880661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:46.427 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.386+0000 7fae9cff9700 1 -- 172.21.15.195:0/2582122073 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7750+0+0 (secure 0 0 0) 0x7fae8c0827f0 con 0x7faea01503a0 2022-06-27T12:18:46.428 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.386+0000 7faea4fe5700 1 --2- 172.21.15.195:0/2582122073 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fae88063d30 0x7fae880661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:46.428 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.386+0000 7faea4fe5700 1 --2- 172.21.15.195:0/2582122073 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fae88063d30 0x7fae880661e0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fae94001c80 tx=0x7fae94009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:18:46.428 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.390+0000 7fae9cff9700 1 -- 172.21.15.195:0/2582122073 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fae8c050da0 con 0x7faea01503a0 2022-06-27T12:18:46.611 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.611+0000 7faea7249700 1 -- 172.21.15.195:0/2582122073 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faea00722c0 con 0x7faea01503a0 2022-06-27T12:18:46.613 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.612+0000 7fae9cff9700 1 -- 172.21.15.195:0/2582122073 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+5371 (secure 0 0 0) 0x7fae8c006b70 con 0x7faea01503a0 2022-06-27T12:18:46.613 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:18:46.614 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":116,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:18:46.244827+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7300},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7300":{"gid":7300,"name":"b","rank":0,"incarnation":114,"state":"up:rejoin","state_seq":19,"addr":"172.21.15.202:6833/1965783412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1965783412},{"type":"v1","addr":"172.21.15.202:6833","nonce":1965783412}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:18:46.616 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.614+0000 7faea7249700 1 -- 172.21.15.195:0/2582122073 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fae88063d30 msgr2=0x7fae880661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:46.616 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.614+0000 7faea7249700 1 --2- 172.21.15.195:0/2582122073 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fae88063d30 0x7fae880661e0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fae94001c80 tx=0x7fae94009040 comp rx=0 tx=0).stop 2022-06-27T12:18:46.617 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.614+0000 7faea7249700 1 -- 172.21.15.195:0/2582122073 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faea01503a0 msgr2=0x7faea00b7360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:46.617 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.614+0000 7faea7249700 1 --2- 172.21.15.195:0/2582122073 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faea01503a0 0x7faea00b7360 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7fae8c00e4b0 tx=0x7fae8c019000 comp rx=0 tx=0).stop 2022-06-27T12:18:46.618 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.614+0000 7faea7249700 1 -- 172.21.15.195:0/2582122073 shutdown_connections 2022-06-27T12:18:46.618 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.614+0000 7faea7249700 1 --2- 172.21.15.195:0/2582122073 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fae88063d30 0x7fae880661e0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:46.619 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.614+0000 7faea7249700 1 --2- 172.21.15.195:0/2582122073 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7faea00b78a0 0x7faea015abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:46.619 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.614+0000 7faea7249700 1 --2- 172.21.15.195:0/2582122073 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7faea00c9dc0 0x7faea00ca190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:46.619 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.614+0000 7faea7249700 1 --2- 172.21.15.195:0/2582122073 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faea01503a0 0x7faea00b7360 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:46.619 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.614+0000 7faea7249700 1 -- 172.21.15.195:0/2582122073 >> 172.21.15.195:0/2582122073 conn(0x7faea0063af0 msgr2=0x7faea0155a30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:18:46.620 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.614+0000 7faea7249700 1 -- 172.21.15.195:0/2582122073 shutdown_connections 2022-06-27T12:18:46.620 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:46.614+0000 7faea7249700 1 -- 172.21.15.195:0/2582122073 wait complete. 2022-06-27T12:18:46.620 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 116 2022-06-27T12:18:46.628 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:18:48.630 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:18:48.764 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:48.763+0000 7f6a8f552700 1 Processor -- start 2022-06-27T12:18:48.764 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:48.764+0000 7f6a8f552700 1 -- start start 2022-06-27T12:18:48.772 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:48.764+0000 7f6a8f552700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7f6a88150580 0x7f6a88156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:48.772 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:48.764+0000 7f6a8f552700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6a88150950 0x7f6a880c7b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:48.772 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:48.764+0000 7f6a8f552700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a880c76b0 con 0x7f6a8806f620 2022-06-27T12:18:48.773 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:48.764+0000 7f6a8f552700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a88157430 con 0x7f6a88150580 2022-06-27T12:18:48.773 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:48.764+0000 7f6a8f552700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a880bad70 con 0x7f6a88150950 2022-06-27T12:18:48.773 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:48.765+0000 7f6a8daef700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f6a8806f620 0x7f6a8806d6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:59220/0 (socket says 172.21.15.195:59220) 2022-06-27T12:18:48.773 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:48.765+0000 7f6a8daef700 1 -- 172.21.15.195:0/1645280360 learned_addr learned my addr 172.21.15.195:0/1645280360 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:18:48.774 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:48.765+0000 7f6a8caed700 1 --2- 172.21.15.195:0/1645280360 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6a88150950 0x7f6a880c7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:48.774 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:48.765+0000 7f6a8d2ee700 1 --2- 172.21.15.195:0/1645280360 >> v2:172.21.15.202:3300/0 conn(0x7f6a88150580 0x7f6a88156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:48.774 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:48.765+0000 7f6a77fff700 1 -- 172.21.15.195:0/1645280360 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1907677919 0 0) 0x7f6a880c76b0 con 0x7f6a8806f620 2022-06-27T12:18:48.774 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:48.765+0000 7f6a77fff700 1 -- 172.21.15.195:0/1645280360 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a70003430 con 0x7f6a8806f620 2022-06-27T12:18:48.775 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:48.765+0000 7f6a8caed700 1 -- 172.21.15.195:0/1645280360 >> v2:172.21.15.202:3300/0 conn(0x7f6a88150580 msgr2=0x7f6a88156ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:18:48.775 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:48.765+0000 7f6a8caed700 1 --2- 172.21.15.195:0/1645280360 >> v2:172.21.15.202:3300/0 conn(0x7f6a88150580 0x7f6a88156ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:48.775 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:48.765+0000 7f6a8caed700 1 -- 172.21.15.195:0/1645280360 >> v1:172.21.15.195:6789/0 conn(0x7f6a8806f620 legacy=0x7f6a8806d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:48.775 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:48.765+0000 7f6a8caed700 1 -- 172.21.15.195:0/1645280360 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a880ba4d0 con 0x7f6a88150950 2022-06-27T12:18:48.776 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:48.765+0000 7f6a8d2ee700 1 --2- 172.21.15.195:0/1645280360 >> v2:172.21.15.202:3300/0 conn(0x7f6a88150580 0x7f6a88156ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:18:48.776 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:48.765+0000 7f6a8caed700 1 --2- 172.21.15.195:0/1645280360 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6a88150950 0x7f6a880c7b40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6a840097a0 tx=0x7f6a84009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac04ef09bfc1b2e2 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:18:48.776 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:48.765+0000 7f6a77fff700 1 -- 172.21.15.195:0/1645280360 <== mon.2 v2:172.21.15.202:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a84011630 con 0x7f6a88150950 2022-06-27T12:18:48.776 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:48.765+0000 7f6a77fff700 1 -- 172.21.15.195:0/1645280360 <== mon.2 v2:172.21.15.202:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a84011790 con 0x7f6a88150950 2022-06-27T12:18:48.777 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:48.765+0000 7f6a77fff700 1 -- 172.21.15.195:0/1645280360 <== mon.2 v2:172.21.15.202:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a84011b10 con 0x7f6a88150950 2022-06-27T12:18:48.777 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:48.766+0000 7f6a8f552700 1 -- 172.21.15.195:0/1645280360 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6a88150950 msgr2=0x7f6a880c7b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:48.777 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:48.766+0000 7f6a8f552700 1 --2- 172.21.15.195:0/1645280360 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6a88150950 0x7f6a880c7b40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6a840097a0 tx=0x7f6a84009060 comp rx=0 tx=0).stop 2022-06-27T12:18:48.777 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:48.766+0000 7f6a8f552700 1 -- 172.21.15.195:0/1645280360 shutdown_connections 2022-06-27T12:18:48.778 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:48.766+0000 7f6a8f552700 1 --2- 172.21.15.195:0/1645280360 >> v2:172.21.15.202:3300/0 conn(0x7f6a88150580 0x7f6a88156ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:48.778 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:48.766+0000 7f6a8f552700 1 --2- 172.21.15.195:0/1645280360 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6a88150950 0x7f6a880c7b40 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:48.778 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:48.766+0000 7f6a8f552700 1 -- 172.21.15.195:0/1645280360 >> 172.21.15.195:0/1645280360 conn(0x7f6a88063af0 msgr2=0x7f6a88063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:18:48.778 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:48.766+0000 7f6a8f552700 1 -- 172.21.15.195:0/1645280360 shutdown_connections 2022-06-27T12:18:48.779 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:48.766+0000 7f6a8f552700 1 -- 172.21.15.195:0/1645280360 wait complete. 2022-06-27T12:18:48.779 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:48.767+0000 7f6a8f552700 1 Processor -- start 2022-06-27T12:18:48.779 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:48.767+0000 7f6a8f552700 1 -- start start 2022-06-27T12:18:48.779 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:48.768+0000 7f6a8f552700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6a8806f620 0x7f6a880bb220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:48.780 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:48.768+0000 7f6a8f552700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6a88150580 0x7f6a88169de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:48.780 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:48.768+0000 7f6a8f552700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6a88150950 0x7f6a880bc340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:48.780 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:48.768+0000 7f6a8f552700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a880bbbd0 con 0x7f6a88150580 2022-06-27T12:18:48.780 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:48.768+0000 7f6a8f552700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a880bbeb0 con 0x7f6a8806f620 2022-06-27T12:18:48.780 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:48.768+0000 7f6a8f552700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a880bb760 con 0x7f6a88150950 2022-06-27T12:18:48.781 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:48.768+0000 7f6a8caed700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6a88150580 0x7f6a88169de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:48.781 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:48.768+0000 7f6a8d2ee700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6a8806f620 0x7f6a880bb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:48.781 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:48.768+0000 7f6a8caed700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6a88150580 0x7f6a88169de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:53040/0 (socket says 172.21.15.195:53040) 2022-06-27T12:18:48.781 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:48.768+0000 7f6a8caed700 1 -- 172.21.15.195:0/3738626550 learned_addr learned my addr 172.21.15.195:0/3738626550 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:18:48.782 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:48.768+0000 7f6a8daef700 1 --2- 172.21.15.195:0/3738626550 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6a88150950 0x7f6a880bc340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:48.782 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:48.768+0000 7f6a8caed700 1 -- 172.21.15.195:0/3738626550 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6a88150950 msgr2=0x7f6a880bc340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:48.782 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:48.768+0000 7f6a8caed700 1 --2- 172.21.15.195:0/3738626550 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6a88150950 0x7f6a880bc340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:48.782 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:48.769+0000 7f6a8caed700 1 -- 172.21.15.195:0/3738626550 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6a8806f620 msgr2=0x7f6a880bb220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:48.783 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:48.769+0000 7f6a8caed700 1 --2- 172.21.15.195:0/3738626550 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6a8806f620 0x7f6a880bb220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:48.783 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:48.769+0000 7f6a8caed700 1 -- 172.21.15.195:0/3738626550 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a780092e0 con 0x7f6a88150580 2022-06-27T12:18:48.783 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:48.769+0000 7f6a8d2ee700 1 --2- 172.21.15.195:0/3738626550 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6a8806f620 0x7f6a880bb220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-06-27T12:18:48.783 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:48.769+0000 7f6a8caed700 1 --2- 172.21.15.195:0/3738626550 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6a88150580 0x7f6a88169de0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f6a84011c30 tx=0x7f6a84017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:18:48.783 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:48.769+0000 7f6a757fa700 1 -- 172.21.15.195:0/3738626550 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a84003730 con 0x7f6a88150580 2022-06-27T12:18:48.784 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:48.769+0000 7f6a757fa700 1 -- 172.21.15.195:0/3738626550 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a84003890 con 0x7f6a88150580 2022-06-27T12:18:48.784 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:48.769+0000 7f6a8f552700 1 -- 172.21.15.195:0/3738626550 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a840092e0 con 0x7f6a88150580 2022-06-27T12:18:48.784 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:48.769+0000 7f6a757fa700 1 -- 172.21.15.195:0/3738626550 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a84003c50 con 0x7f6a88150580 2022-06-27T12:18:48.784 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:48.769+0000 7f6a8f552700 1 -- 172.21.15.195:0/3738626550 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a880bdc20 con 0x7f6a88150580 2022-06-27T12:18:48.785 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:48.770+0000 7f6a8f552700 1 -- 172.21.15.195:0/3738626550 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a880722c0 con 0x7f6a88150580 2022-06-27T12:18:48.785 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:48.770+0000 7f6a757fa700 1 -- 172.21.15.195:0/3738626550 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f6a8402a020 con 0x7f6a88150580 2022-06-27T12:18:48.785 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:48.771+0000 7f6a757fa700 1 --2- 172.21.15.195:0/3738626550 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6a68063d00 0x7f6a680661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:48.785 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:48.771+0000 7f6a757fa700 1 -- 172.21.15.195:0/3738626550 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7750+0+0 (secure 0 0 0) 0x7f6a84082320 con 0x7f6a88150580 2022-06-27T12:18:48.785 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:48.771+0000 7f6a8d2ee700 1 --2- 172.21.15.195:0/3738626550 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6a68063d00 0x7f6a680661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:48.786 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:48.771+0000 7f6a8d2ee700 1 --2- 172.21.15.195:0/3738626550 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6a68063d00 0x7f6a680661b0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f6a780114c0 tx=0x7f6a78009060 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:18:48.786 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:48.775+0000 7f6a757fa700 1 -- 172.21.15.195:0/3738626550 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f6a84050920 con 0x7f6a88150580 2022-06-27T12:18:49.001 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:49.001+0000 7f6a8f552700 1 -- 172.21.15.195:0/3738626550 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6a880bbff0 con 0x7f6a88150580 2022-06-27T12:18:49.002 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:49.002+0000 7f6a757fa700 1 -- 172.21.15.195:0/3738626550 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+5371 (secure 0 0 0) 0x7f6a84015030 con 0x7f6a88150580 2022-06-27T12:18:49.003 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:18:49.003 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":116,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:18:46.244827+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7300},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7300":{"gid":7300,"name":"b","rank":0,"incarnation":114,"state":"up:rejoin","state_seq":19,"addr":"172.21.15.202:6833/1965783412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1965783412},{"type":"v1","addr":"172.21.15.202:6833","nonce":1965783412}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:18:49.005 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:49.003+0000 7f6a8f552700 1 -- 172.21.15.195:0/3738626550 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6a68063d00 msgr2=0x7f6a680661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:49.005 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:49.003+0000 7f6a8f552700 1 --2- 172.21.15.195:0/3738626550 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6a68063d00 0x7f6a680661b0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f6a780114c0 tx=0x7f6a78009060 comp rx=0 tx=0).stop 2022-06-27T12:18:49.006 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:49.003+0000 7f6a8f552700 1 -- 172.21.15.195:0/3738626550 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6a88150580 msgr2=0x7f6a88169de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:49.006 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:49.003+0000 7f6a8f552700 1 --2- 172.21.15.195:0/3738626550 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6a88150580 0x7f6a88169de0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f6a84011c30 tx=0x7f6a84017000 comp rx=0 tx=0).stop 2022-06-27T12:18:49.006 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:49.004+0000 7f6a8f552700 1 -- 172.21.15.195:0/3738626550 shutdown_connections 2022-06-27T12:18:49.007 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:49.004+0000 7f6a8f552700 1 --2- 172.21.15.195:0/3738626550 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f6a68063d00 0x7f6a680661b0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:49.007 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:49.004+0000 7f6a8f552700 1 --2- 172.21.15.195:0/3738626550 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f6a88150950 0x7f6a880bc340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:49.007 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:49.004+0000 7f6a8f552700 1 --2- 172.21.15.195:0/3738626550 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f6a8806f620 0x7f6a880bb220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:49.007 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:49.004+0000 7f6a8f552700 1 --2- 172.21.15.195:0/3738626550 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6a88150580 0x7f6a88169de0 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:49.008 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:49.004+0000 7f6a8f552700 1 -- 172.21.15.195:0/3738626550 >> 172.21.15.195:0/3738626550 conn(0x7f6a88063af0 msgr2=0x7f6a8814dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:18:49.008 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:49.004+0000 7f6a8f552700 1 -- 172.21.15.195:0/3738626550 shutdown_connections 2022-06-27T12:18:49.008 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:49.004+0000 7f6a8f552700 1 -- 172.21.15.195:0/3738626550 wait complete. 2022-06-27T12:18:49.008 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 116 2022-06-27T12:18:49.016 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:18:51.018 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:18:51.152 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.152+0000 7fc08c663700 1 Processor -- start 2022-06-27T12:18:51.153 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.152+0000 7fc08c663700 1 -- start start 2022-06-27T12:18:51.163 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.153+0000 7fc08c663700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc08406f640 0x7fc084156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:51.164 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.153+0000 7fc08c663700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fc0840c39e0 0x7fc0841535a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:51.164 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.153+0000 7fc08c663700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0840c1a20 con 0x7fc0840c4160 2022-06-27T12:18:51.164 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.153+0000 7fc08c663700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0840c1130 con 0x7fc0840c39e0 2022-06-27T12:18:51.164 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.153+0000 7fc08c663700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc08406ea90 con 0x7fc08406f640 2022-06-27T12:18:51.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.153+0000 7fc089bfe700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fc0840c4160 0x7fc0840c1eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:59232/0 (socket says 172.21.15.195:59232) 2022-06-27T12:18:51.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.153+0000 7fc08ac00700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fc0840c39e0 0x7fc0841535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:51.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.153+0000 7fc089bfe700 1 -- 172.21.15.195:0/4049393231 learned_addr learned my addr 172.21.15.195:0/4049393231 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:18:51.165 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.153+0000 7fc08a3ff700 1 --2- 172.21.15.195:0/4049393231 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc08406f640 0x7fc084156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:51.166 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.153+0000 7fc0893fd700 1 -- 172.21.15.195:0/4049393231 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1619224119 0 0) 0x7fc0840c1a20 con 0x7fc0840c4160 2022-06-27T12:18:51.167 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.153+0000 7fc0893fd700 1 -- 172.21.15.195:0/4049393231 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc06c003430 con 0x7fc0840c4160 2022-06-27T12:18:51.168 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.154+0000 7fc08ac00700 1 -- 172.21.15.195:0/4049393231 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc08406f640 msgr2=0x7fc084156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:18:51.169 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.154+0000 7fc08ac00700 1 --2- 172.21.15.195:0/4049393231 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc08406f640 0x7fc084156ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:51.169 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.154+0000 7fc08ac00700 1 -- 172.21.15.195:0/4049393231 >> v1:172.21.15.195:6789/0 conn(0x7fc0840c4160 legacy=0x7fc0840c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:51.170 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.154+0000 7fc08ac00700 1 -- 172.21.15.195:0/4049393231 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc08406ed70 con 0x7fc0840c39e0 2022-06-27T12:18:51.170 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.154+0000 7fc08ac00700 1 --2- 172.21.15.195:0/4049393231 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc0840c39e0 0x7fc0841535a0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7fc07c009e60 tx=0x7fc07c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c7838dd3fdb2dfa server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:18:51.171 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.154+0000 7fc0893fd700 1 -- 172.21.15.195:0/4049393231 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc07c00e070 con 0x7fc0840c39e0 2022-06-27T12:18:51.171 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.154+0000 7fc0893fd700 1 -- 172.21.15.195:0/4049393231 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc07c00bb40 con 0x7fc0840c39e0 2022-06-27T12:18:51.171 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.154+0000 7fc0893fd700 1 -- 172.21.15.195:0/4049393231 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc07c012430 con 0x7fc0840c39e0 2022-06-27T12:18:51.172 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.154+0000 7fc08c663700 1 -- 172.21.15.195:0/4049393231 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc0840c39e0 msgr2=0x7fc0841535a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:51.172 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.154+0000 7fc08c663700 1 --2- 172.21.15.195:0/4049393231 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc0840c39e0 0x7fc0841535a0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7fc07c009e60 tx=0x7fc07c009780 comp rx=0 tx=0).stop 2022-06-27T12:18:51.173 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.155+0000 7fc08c663700 1 -- 172.21.15.195:0/4049393231 shutdown_connections 2022-06-27T12:18:51.173 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.155+0000 7fc08c663700 1 --2- 172.21.15.195:0/4049393231 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc0840c39e0 0x7fc0841535a0 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:51.174 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.155+0000 7fc08c663700 1 --2- 172.21.15.195:0/4049393231 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc08406f640 0x7fc084156ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:51.174 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.155+0000 7fc08c663700 1 -- 172.21.15.195:0/4049393231 >> 172.21.15.195:0/4049393231 conn(0x7fc0840c6350 msgr2=0x7fc0840c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:18:51.175 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.155+0000 7fc08c663700 1 -- 172.21.15.195:0/4049393231 shutdown_connections 2022-06-27T12:18:51.175 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.155+0000 7fc08c663700 1 -- 172.21.15.195:0/4049393231 wait complete. 2022-06-27T12:18:51.176 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.156+0000 7fc08c663700 1 Processor -- start 2022-06-27T12:18:51.177 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.156+0000 7fc08c663700 1 -- start start 2022-06-27T12:18:51.178 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.156+0000 7fc08c663700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc08406f640 0x7fc0840680b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:51.178 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.156+0000 7fc08c663700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc0840c39e0 0x7fc08414f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:51.179 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.156+0000 7fc08c663700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc0840c4160 0x7fc08414f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:51.180 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.157+0000 7fc08c663700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0840bd2b0 con 0x7fc0840c39e0 2022-06-27T12:18:51.180 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.157+0000 7fc08c663700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc08406a040 con 0x7fc08406f640 2022-06-27T12:18:51.180 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.157+0000 7fc08c663700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc08406a2e0 con 0x7fc0840c4160 2022-06-27T12:18:51.181 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.157+0000 7fc089bfe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc0840c39e0 0x7fc08414f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:51.181 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.157+0000 7fc089bfe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc0840c39e0 0x7fc08414f290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:53050/0 (socket says 172.21.15.195:53050) 2022-06-27T12:18:51.181 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.157+0000 7fc089bfe700 1 -- 172.21.15.195:0/1438946727 learned_addr learned my addr 172.21.15.195:0/1438946727 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:18:51.181 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.157+0000 7fc08a3ff700 1 --2- 172.21.15.195:0/1438946727 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc08406f640 0x7fc0840680b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:51.182 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.157+0000 7fc08ac00700 1 --2- 172.21.15.195:0/1438946727 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc0840c4160 0x7fc08414f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:51.182 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.157+0000 7fc089bfe700 1 -- 172.21.15.195:0/1438946727 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc0840c4160 msgr2=0x7fc08414f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:51.182 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.157+0000 7fc089bfe700 1 --2- 172.21.15.195:0/1438946727 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc0840c4160 0x7fc08414f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:51.182 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.157+0000 7fc089bfe700 1 -- 172.21.15.195:0/1438946727 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc08406f640 msgr2=0x7fc0840680b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:51.183 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.157+0000 7fc089bfe700 1 --2- 172.21.15.195:0/1438946727 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc08406f640 0x7fc0840680b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:51.183 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.157+0000 7fc089bfe700 1 -- 172.21.15.195:0/1438946727 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc07c009a20 con 0x7fc0840c39e0 2022-06-27T12:18:51.183 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.157+0000 7fc089bfe700 1 --2- 172.21.15.195:0/1438946727 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc0840c39e0 0x7fc08414f290 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7fc08000c3f0 tx=0x7fc080007bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:18:51.183 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.157+0000 7fc0727fc700 1 -- 172.21.15.195:0/1438946727 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0800072e0 con 0x7fc0840c39e0 2022-06-27T12:18:51.184 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.157+0000 7fc0727fc700 1 -- 172.21.15.195:0/1438946727 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc080007440 con 0x7fc0840c39e0 2022-06-27T12:18:51.184 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.157+0000 7fc08c663700 1 -- 172.21.15.195:0/1438946727 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc08406a5c0 con 0x7fc0840c39e0 2022-06-27T12:18:51.184 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.157+0000 7fc0727fc700 1 -- 172.21.15.195:0/1438946727 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0800057e0 con 0x7fc0840c39e0 2022-06-27T12:18:51.184 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.157+0000 7fc08c663700 1 -- 172.21.15.195:0/1438946727 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc084069690 con 0x7fc0840c39e0 2022-06-27T12:18:51.184 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.158+0000 7fc08c663700 1 -- 172.21.15.195:0/1438946727 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc08404ccb0 con 0x7fc0840c39e0 2022-06-27T12:18:51.185 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.158+0000 7fc0727fc700 1 -- 172.21.15.195:0/1438946727 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fc080019030 con 0x7fc0840c39e0 2022-06-27T12:18:51.185 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.159+0000 7fc0727fc700 1 --2- 172.21.15.195:0/1438946727 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc064063d00 0x7fc0640661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:51.185 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.159+0000 7fc0727fc700 1 -- 172.21.15.195:0/1438946727 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7750+0+0 (secure 0 0 0) 0x7fc080080090 con 0x7fc0840c39e0 2022-06-27T12:18:51.185 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.159+0000 7fc08a3ff700 1 --2- 172.21.15.195:0/1438946727 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc064063d00 0x7fc0640661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:51.186 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.159+0000 7fc08a3ff700 1 --2- 172.21.15.195:0/1438946727 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc064063d00 0x7fc0640661b0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fc074006dc0 tx=0x7fc074008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:18:51.186 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.163+0000 7fc0727fc700 1 -- 172.21.15.195:0/1438946727 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fc08004e5e0 con 0x7fc0840c39e0 2022-06-27T12:18:51.408 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.408+0000 7fc08c663700 1 -- 172.21.15.195:0/1438946727 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc0840722c0 con 0x7fc0840c39e0 2022-06-27T12:18:51.409 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.408+0000 7fc0727fc700 1 -- 172.21.15.195:0/1438946727 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+5371 (secure 0 0 0) 0x7fc080017020 con 0x7fc0840c39e0 2022-06-27T12:18:51.409 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:18:51.409 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":116,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:18:46.244827+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7300},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7300":{"gid":7300,"name":"b","rank":0,"incarnation":114,"state":"up:rejoin","state_seq":19,"addr":"172.21.15.202:6833/1965783412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1965783412},{"type":"v1","addr":"172.21.15.202:6833","nonce":1965783412}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:18:51.412 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.410+0000 7fc08c663700 1 -- 172.21.15.195:0/1438946727 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc064063d00 msgr2=0x7fc0640661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:51.412 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.410+0000 7fc08c663700 1 --2- 172.21.15.195:0/1438946727 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc064063d00 0x7fc0640661b0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fc074006dc0 tx=0x7fc074008040 comp rx=0 tx=0).stop 2022-06-27T12:18:51.412 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.410+0000 7fc08c663700 1 -- 172.21.15.195:0/1438946727 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc0840c39e0 msgr2=0x7fc08414f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:51.413 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.410+0000 7fc08c663700 1 --2- 172.21.15.195:0/1438946727 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc0840c39e0 0x7fc08414f290 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7fc08000c3f0 tx=0x7fc080007bb0 comp rx=0 tx=0).stop 2022-06-27T12:18:51.413 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.410+0000 7fc08c663700 1 -- 172.21.15.195:0/1438946727 shutdown_connections 2022-06-27T12:18:51.414 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.410+0000 7fc08c663700 1 --2- 172.21.15.195:0/1438946727 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fc064063d00 0x7fc0640661b0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:51.414 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.410+0000 7fc08c663700 1 --2- 172.21.15.195:0/1438946727 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fc0840c4160 0x7fc08414f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:51.415 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.410+0000 7fc08c663700 1 --2- 172.21.15.195:0/1438946727 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fc08406f640 0x7fc0840680b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:51.415 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.410+0000 7fc08c663700 1 --2- 172.21.15.195:0/1438946727 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc0840c39e0 0x7fc08414f290 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:51.416 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.410+0000 7fc08c663700 1 -- 172.21.15.195:0/1438946727 >> 172.21.15.195:0/1438946727 conn(0x7fc0840c6350 msgr2=0x7fc084151b30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:18:51.416 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.410+0000 7fc08c663700 1 -- 172.21.15.195:0/1438946727 shutdown_connections 2022-06-27T12:18:51.417 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:51.410+0000 7fc08c663700 1 -- 172.21.15.195:0/1438946727 wait complete. 2022-06-27T12:18:51.417 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 116 2022-06-27T12:18:51.423 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:18:53.425 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:18:53.571 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.560+0000 7fce66eb2700 1 Processor -- start 2022-06-27T12:18:53.572 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.560+0000 7fce66eb2700 1 -- start start 2022-06-27T12:18:53.572 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.561+0000 7fce66eb2700 1 --2- >> v2:172.21.15.202:3300/0 conn(0x7fce6006f640 0x7fce60156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:53.573 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.561+0000 7fce66eb2700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fce600c4160 0x7fce600c1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:53.573 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.561+0000 7fce66eb2700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce600c1a20 con 0x7fce600c39e0 2022-06-27T12:18:53.573 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.561+0000 7fce66eb2700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce600c1130 con 0x7fce600c4160 2022-06-27T12:18:53.574 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.561+0000 7fce66eb2700 1 -- --> v2:172.21.15.202:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce6006ea90 con 0x7fce6006f640 2022-06-27T12:18:53.575 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.561+0000 7fce5ffff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fce600c4160 0x7fce600c1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:53.575 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.561+0000 7fce6544f700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fce600c39e0 0x7fce601535a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:59252/0 (socket says 172.21.15.195:59252) 2022-06-27T12:18:53.576 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.561+0000 7fce6544f700 1 -- 172.21.15.195:0/1279346250 learned_addr learned my addr 172.21.15.195:0/1279346250 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:18:53.576 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.561+0000 7fce64c4e700 1 --2- 172.21.15.195:0/1279346250 >> v2:172.21.15.202:3300/0 conn(0x7fce6006f640 0x7fce60156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:53.577 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.561+0000 7fce5ffff700 1 -- 172.21.15.195:0/1279346250 >> v2:172.21.15.202:3300/0 conn(0x7fce6006f640 msgr2=0x7fce60156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:18:53.577 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.561+0000 7fce5ffff700 1 --2- 172.21.15.195:0/1279346250 >> v2:172.21.15.202:3300/0 conn(0x7fce6006f640 0x7fce60156ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:53.578 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.561+0000 7fce5ffff700 1 -- 172.21.15.195:0/1279346250 >> v1:172.21.15.195:6789/0 conn(0x7fce600c39e0 legacy=0x7fce601535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:53.579 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.561+0000 7fce5ffff700 1 -- 172.21.15.195:0/1279346250 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce6006ed70 con 0x7fce600c4160 2022-06-27T12:18:53.579 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.561+0000 7fce5f7fe700 1 -- 172.21.15.195:0/1279346250 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1951664512 0 0) 0x7fce600c1a20 con 0x7fce600c39e0 2022-06-27T12:18:53.580 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.561+0000 7fce5ffff700 1 --2- 172.21.15.195:0/1279346250 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fce600c4160 0x7fce600c1eb0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7fce6006db30 tx=0x7fce4c009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd4033264ebef011 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:18:53.580 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.561+0000 7fce5f7fe700 1 -- 172.21.15.195:0/1279346250 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce4c011630 con 0x7fce600c4160 2022-06-27T12:18:53.581 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.561+0000 7fce5f7fe700 1 -- 172.21.15.195:0/1279346250 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce4c011790 con 0x7fce600c4160 2022-06-27T12:18:53.581 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.561+0000 7fce5f7fe700 1 -- 172.21.15.195:0/1279346250 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce4c011b30 con 0x7fce600c4160 2022-06-27T12:18:53.582 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.562+0000 7fce66eb2700 1 -- 172.21.15.195:0/1279346250 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fce600c4160 msgr2=0x7fce600c1eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:53.583 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.562+0000 7fce66eb2700 1 --2- 172.21.15.195:0/1279346250 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fce600c4160 0x7fce600c1eb0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7fce6006db30 tx=0x7fce4c009060 comp rx=0 tx=0).stop 2022-06-27T12:18:53.583 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.562+0000 7fce66eb2700 1 -- 172.21.15.195:0/1279346250 shutdown_connections 2022-06-27T12:18:53.584 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.562+0000 7fce66eb2700 1 --2- 172.21.15.195:0/1279346250 >> v2:172.21.15.202:3300/0 conn(0x7fce6006f640 0x7fce60156ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:53.584 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.562+0000 7fce66eb2700 1 --2- 172.21.15.195:0/1279346250 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fce600c4160 0x7fce600c1eb0 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:53.584 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.562+0000 7fce66eb2700 1 -- 172.21.15.195:0/1279346250 >> 172.21.15.195:0/1279346250 conn(0x7fce600c6350 msgr2=0x7fce600c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:18:53.585 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.562+0000 7fce66eb2700 1 -- 172.21.15.195:0/1279346250 shutdown_connections 2022-06-27T12:18:53.585 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.562+0000 7fce66eb2700 1 -- 172.21.15.195:0/1279346250 wait complete. 2022-06-27T12:18:53.586 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.563+0000 7fce66eb2700 1 Processor -- start 2022-06-27T12:18:53.586 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.563+0000 7fce66eb2700 1 -- start start 2022-06-27T12:18:53.587 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.564+0000 7fce66eb2700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fce6006f640 0x7fce6016aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:53.587 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.564+0000 7fce66eb2700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fce600c39e0 0x7fce60153550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:53.588 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.564+0000 7fce66eb2700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fce600c4160 0x7fce6014e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:53.588 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.564+0000 7fce66eb2700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce6016a570 con 0x7fce600c4160 2022-06-27T12:18:53.589 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.564+0000 7fce66eb2700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce60152f70 con 0x7fce600c39e0 2022-06-27T12:18:53.589 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.564+0000 7fce66eb2700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce601526d0 con 0x7fce6006f640 2022-06-27T12:18:53.590 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.564+0000 7fce6544f700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fce600c4160 0x7fce6014e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:53.590 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.564+0000 7fce6544f700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fce600c4160 0x7fce6014e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:53070/0 (socket says 172.21.15.195:53070) 2022-06-27T12:18:53.591 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.564+0000 7fce5ffff700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fce600c39e0 0x7fce60153550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:53.591 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.564+0000 7fce6544f700 1 -- 172.21.15.195:0/3976673758 learned_addr learned my addr 172.21.15.195:0/3976673758 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:18:53.591 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.564+0000 7fce64c4e700 1 --2- 172.21.15.195:0/3976673758 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fce6006f640 0x7fce6016aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:53.592 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.564+0000 7fce6544f700 1 -- 172.21.15.195:0/3976673758 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fce6006f640 msgr2=0x7fce6016aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:53.592 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.564+0000 7fce6544f700 1 --2- 172.21.15.195:0/3976673758 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fce6006f640 0x7fce6016aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:53.593 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.564+0000 7fce6544f700 1 -- 172.21.15.195:0/3976673758 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fce600c39e0 msgr2=0x7fce60153550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:53.593 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.564+0000 7fce6544f700 1 --2- 172.21.15.195:0/3976673758 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fce600c39e0 0x7fce60153550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:53.594 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.564+0000 7fce6544f700 1 -- 172.21.15.195:0/3976673758 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce4c0092e0 con 0x7fce600c4160 2022-06-27T12:18:53.594 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.565+0000 7fce6544f700 1 --2- 172.21.15.195:0/3976673758 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fce600c4160 0x7fce6014e3e0 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7fce5800c810 tx=0x7fce58007f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:18:53.595 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.565+0000 7fce5cff9700 1 -- 172.21.15.195:0/3976673758 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce58004c80 con 0x7fce600c4160 2022-06-27T12:18:53.595 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.565+0000 7fce5cff9700 1 -- 172.21.15.195:0/3976673758 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce58004e00 con 0x7fce600c4160 2022-06-27T12:18:53.596 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.565+0000 7fce66eb2700 1 -- 172.21.15.195:0/3976673758 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce601529b0 con 0x7fce600c4160 2022-06-27T12:18:53.596 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.565+0000 7fce5cff9700 1 -- 172.21.15.195:0/3976673758 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce580073a0 con 0x7fce600c4160 2022-06-27T12:18:53.597 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.565+0000 7fce66eb2700 1 -- 172.21.15.195:0/3976673758 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce6016a200 con 0x7fce600c4160 2022-06-27T12:18:53.597 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.566+0000 7fce66eb2700 1 -- 172.21.15.195:0/3976673758 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce600722c0 con 0x7fce600c4160 2022-06-27T12:18:53.597 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.566+0000 7fce5cff9700 1 -- 172.21.15.195:0/3976673758 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fce5801a030 con 0x7fce600c4160 2022-06-27T12:18:53.598 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.566+0000 7fce5cff9700 1 --2- 172.21.15.195:0/3976673758 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fce48063d30 0x7fce480661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:53.598 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.567+0000 7fce5cff9700 1 -- 172.21.15.195:0/3976673758 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7750+0+0 (secure 0 0 0) 0x7fce58080280 con 0x7fce600c4160 2022-06-27T12:18:53.599 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.567+0000 7fce64c4e700 1 --2- 172.21.15.195:0/3976673758 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fce48063d30 0x7fce480661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:53.599 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.567+0000 7fce64c4e700 1 --2- 172.21.15.195:0/3976673758 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fce48063d30 0x7fce480661e0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fce54004560 tx=0x7fce54008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:18:53.600 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.571+0000 7fce5cff9700 1 -- 172.21.15.195:0/3976673758 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fce5804e850 con 0x7fce600c4160 2022-06-27T12:18:53.799 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.799+0000 7fce66eb2700 1 -- 172.21.15.195:0/3976673758 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fce600c60d0 con 0x7fce600c4160 2022-06-27T12:18:53.800 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.799+0000 7fce5cff9700 1 -- 172.21.15.195:0/3976673758 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+5377 (secure 0 0 0) 0x7fce58018020 con 0x7fce600c4160 2022-06-27T12:18:53.801 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:18:53.801 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":117,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:18:52.379960+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7300},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7300":{"gid":7300,"name":"b","rank":0,"incarnation":114,"state":"up:clientreplay","state_seq":21,"addr":"172.21.15.202:6833/1965783412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1965783412},{"type":"v1","addr":"172.21.15.202:6833","nonce":1965783412}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:18:53.809 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.801+0000 7fce66eb2700 1 -- 172.21.15.195:0/3976673758 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fce48063d30 msgr2=0x7fce480661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:53.810 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.801+0000 7fce66eb2700 1 --2- 172.21.15.195:0/3976673758 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fce48063d30 0x7fce480661e0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fce54004560 tx=0x7fce54008040 comp rx=0 tx=0).stop 2022-06-27T12:18:53.811 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.801+0000 7fce66eb2700 1 -- 172.21.15.195:0/3976673758 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fce600c4160 msgr2=0x7fce6014e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:53.811 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.801+0000 7fce66eb2700 1 --2- 172.21.15.195:0/3976673758 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fce600c4160 0x7fce6014e3e0 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7fce5800c810 tx=0x7fce58007f50 comp rx=0 tx=0).stop 2022-06-27T12:18:53.812 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.801+0000 7fce66eb2700 1 -- 172.21.15.195:0/3976673758 shutdown_connections 2022-06-27T12:18:53.812 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.801+0000 7fce66eb2700 1 --2- 172.21.15.195:0/3976673758 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fce48063d30 0x7fce480661e0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:53.813 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.801+0000 7fce66eb2700 1 --2- 172.21.15.195:0/3976673758 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fce600c4160 0x7fce6014e3e0 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:53.813 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.801+0000 7fce66eb2700 1 --2- 172.21.15.195:0/3976673758 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fce6006f640 0x7fce6016aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:53.813 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.801+0000 7fce66eb2700 1 --2- 172.21.15.195:0/3976673758 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fce600c39e0 0x7fce60153550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:53.815 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.801+0000 7fce66eb2700 1 -- 172.21.15.195:0/3976673758 >> 172.21.15.195:0/3976673758 conn(0x7fce600c6350 msgr2=0x7fce600c1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:18:53.815 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.801+0000 7fce66eb2700 1 -- 172.21.15.195:0/3976673758 shutdown_connections 2022-06-27T12:18:53.816 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:53.801+0000 7fce66eb2700 1 -- 172.21.15.195:0/3976673758 wait complete. 2022-06-27T12:18:53.816 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 117 2022-06-27T12:18:53.834 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-06-27T12:18:55.836 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:18:55.971 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.970+0000 7fbfd4f27700 1 Processor -- start 2022-06-27T12:18:55.971 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.971+0000 7fbfd4f27700 1 -- start start 2022-06-27T12:18:55.973 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.972+0000 7fbfd4f27700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbfd006f640 0x7fbfd0156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:55.973 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.972+0000 7fbfd4f27700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfd00c1a20 con 0x7fbfd00c4160 2022-06-27T12:18:55.974 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.972+0000 7fbfd4f27700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfd00c1130 con 0x7fbfd00c39e0 2022-06-27T12:18:55.974 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.972+0000 7fbfd4f27700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd006ea90 con 0x7fbfd006f640 2022-06-27T12:18:55.974 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.972+0000 7fbfcdd9b700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fbfd00c4160 0x7fbfd00c1eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:59262/0 (socket says 172.21.15.195:59262) 2022-06-27T12:18:55.975 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.972+0000 7fbfcdd9b700 1 -- 172.21.15.195:0/2739964755 learned_addr learned my addr 172.21.15.195:0/2739964755 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:18:55.975 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.972+0000 7fbfce59c700 1 --2- 172.21.15.195:0/2739964755 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbfd006f640 0x7fbfd0156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:55.976 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.972+0000 7fbfcd59a700 1 -- 172.21.15.195:0/2739964755 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2111991834 0 0) 0x7fbfd00c1a20 con 0x7fbfd00c4160 2022-06-27T12:18:55.994 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.972+0000 7fbfcd59a700 1 -- 172.21.15.195:0/2739964755 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfb4003430 con 0x7fbfd00c4160 2022-06-27T12:18:55.995 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.972+0000 7fbfcd59a700 1 -- 172.21.15.195:0/2739964755 <== mon.0 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2376670656 0 0) 0x7fbfb4003430 con 0x7fbfd00c4160 2022-06-27T12:18:55.995 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.972+0000 7fbfcd59a700 1 -- 172.21.15.195:0/2739964755 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbfd006f640 msgr2=0x7fbfd0156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:18:55.996 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.972+0000 7fbfcd59a700 1 --2- 172.21.15.195:0/2739964755 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbfd006f640 0x7fbfd0156ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:55.996 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.972+0000 7fbfcd59a700 1 -- 172.21.15.195:0/2739964755 >> v1:172.21.15.202:6789/0 conn(0x7fbfd00c39e0 legacy=0x7fbfd01535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:55.997 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.972+0000 7fbfcd59a700 1 -- 172.21.15.195:0/2739964755 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfd006ed70 con 0x7fbfd00c4160 2022-06-27T12:18:56.000 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.972+0000 7fbfcd59a700 1 -- 172.21.15.195:0/2739964755 <== mon.0 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1200511073 0 0) 0x7fbfc0001470 con 0x7fbfd00c4160 2022-06-27T12:18:56.010 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.972+0000 7fbfcd59a700 1 -- 172.21.15.195:0/2739964755 >> v1:172.21.15.195:6789/0 conn(0x7fbfd00c4160 legacy=0x7fbfd00c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:56.011 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.972+0000 7fbfcd59a700 1 --2- 172.21.15.195:0/2739964755 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbfb4003950 0x7fbfb4003d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:56.012 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.972+0000 7fbfcd59a700 1 --2- 172.21.15.195:0/2739964755 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbfb40046d0 0x7fbfb4004b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:56.012 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.972+0000 7fbfcd59a700 1 --2- 172.21.15.195:0/2739964755 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbfd006f640 0x7fbfb4005080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:56.013 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.972+0000 7fbfcd59a700 1 -- 172.21.15.195:0/2739964755 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd006ea90 con 0x7fbfb40046d0 2022-06-27T12:18:56.013 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.972+0000 7fbfcd59a700 1 -- 172.21.15.195:0/2739964755 --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb400b9d0 con 0x7fbfb4003950 2022-06-27T12:18:56.014 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.972+0000 7fbfcd59a700 1 -- 172.21.15.195:0/2739964755 --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb400bb90 con 0x7fbfd006f640 2022-06-27T12:18:56.014 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.973+0000 7fbfcdd9b700 1 --2- 172.21.15.195:0/2739964755 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbfb40046d0 0x7fbfb4004b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:56.015 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.973+0000 7fbfce59c700 1 --2- 172.21.15.195:0/2739964755 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbfb4003950 0x7fbfb4003d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:56.015 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.973+0000 7fbfced9d700 1 --2- 172.21.15.195:0/2739964755 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbfd006f640 0x7fbfb4005080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:56.016 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.973+0000 7fbfcdd9b700 1 -- 172.21.15.195:0/2739964755 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbfd006f640 msgr2=0x7fbfb4005080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:18:56.016 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.973+0000 7fbfcdd9b700 1 --2- 172.21.15.195:0/2739964755 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbfd006f640 0x7fbfb4005080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:56.017 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.973+0000 7fbfcdd9b700 1 -- 172.21.15.195:0/2739964755 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbfb4003950 msgr2=0x7fbfb4003d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:18:56.017 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.973+0000 7fbfcdd9b700 1 --2- 172.21.15.195:0/2739964755 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbfb4003950 0x7fbfb4003d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:56.018 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.973+0000 7fbfcdd9b700 1 -- 172.21.15.195:0/2739964755 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfb4003430 con 0x7fbfb40046d0 2022-06-27T12:18:56.018 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.973+0000 7fbfced9d700 1 --2- 172.21.15.195:0/2739964755 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbfd006f640 0x7fbfb4005080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:18:56.020 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.973+0000 7fbfcdd9b700 1 --2- 172.21.15.195:0/2739964755 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbfb40046d0 0x7fbfb4004b40 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7fbfd006dd70 tx=0x7fbfc0007800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3097bc06311420e0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:18:56.021 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.973+0000 7fbfcd59a700 1 -- 172.21.15.195:0/2739964755 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfc000c9e0 con 0x7fbfb40046d0 2022-06-27T12:18:56.022 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.973+0000 7fbfcd59a700 1 -- 172.21.15.195:0/2739964755 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfc000cb40 con 0x7fbfb40046d0 2022-06-27T12:18:56.023 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.973+0000 7fbfcd59a700 1 -- 172.21.15.195:0/2739964755 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfc00070d0 con 0x7fbfb40046d0 2022-06-27T12:18:56.023 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.974+0000 7fbfd4f27700 1 -- 172.21.15.195:0/2739964755 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbfb40046d0 msgr2=0x7fbfb4004b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:56.024 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.974+0000 7fbfd4f27700 1 --2- 172.21.15.195:0/2739964755 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbfb40046d0 0x7fbfb4004b40 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7fbfd006dd70 tx=0x7fbfc0007800 comp rx=0 tx=0).stop 2022-06-27T12:18:56.024 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.974+0000 7fbfced9d700 1 -- 172.21.15.195:0/2739964755 reap_dead start 2022-06-27T12:18:56.025 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.974+0000 7fbfd4f27700 1 -- 172.21.15.195:0/2739964755 shutdown_connections 2022-06-27T12:18:56.025 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.974+0000 7fbfd4f27700 1 -- 172.21.15.195:0/2739964755 >> 172.21.15.195:0/2739964755 conn(0x7fbfd00c6350 msgr2=0x7fbfd00c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:18:56.038 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.974+0000 7fbfd4f27700 1 -- 172.21.15.195:0/2739964755 shutdown_connections 2022-06-27T12:18:56.039 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.974+0000 7fbfd4f27700 1 -- 172.21.15.195:0/2739964755 wait complete. 2022-06-27T12:18:56.039 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.975+0000 7fbfd4f27700 1 Processor -- start 2022-06-27T12:18:56.040 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.975+0000 7fbfd4f27700 1 -- start start 2022-06-27T12:18:56.040 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.976+0000 7fbfd4f27700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbfb40046d0 0x7fbfd0155c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:56.046 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.976+0000 7fbfd4f27700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbfd014e880 0x7fbfd016a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:56.046 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.976+0000 7fbfd4f27700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbfd00c4710 0x7fbfd00bc990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:56.047 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.976+0000 7fbfd4f27700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd00c04c0 con 0x7fbfb40046d0 2022-06-27T12:18:56.047 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.976+0000 7fbfd4f27700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd016a050 con 0x7fbfd00c4710 2022-06-27T12:18:56.048 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.976+0000 7fbfd4f27700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd0169d50 con 0x7fbfd014e880 2022-06-27T12:18:56.048 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.976+0000 7fbfce59c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbfb40046d0 0x7fbfd0155c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:56.049 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.976+0000 7fbfce59c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbfb40046d0 0x7fbfd0155c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:53084/0 (socket says 172.21.15.195:53084) 2022-06-27T12:18:56.049 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.976+0000 7fbfce59c700 1 -- 172.21.15.195:0/2562273840 learned_addr learned my addr 172.21.15.195:0/2562273840 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:18:56.050 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.976+0000 7fbfcdd9b700 1 --2- 172.21.15.195:0/2562273840 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbfd014e880 0x7fbfd016a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:56.050 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.976+0000 7fbfced9d700 1 --2- 172.21.15.195:0/2562273840 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbfd00c4710 0x7fbfd00bc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:56.051 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.976+0000 7fbfce59c700 1 -- 172.21.15.195:0/2562273840 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbfd014e880 msgr2=0x7fbfd016a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:56.052 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.976+0000 7fbfce59c700 1 --2- 172.21.15.195:0/2562273840 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbfd014e880 0x7fbfd016a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:56.052 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.976+0000 7fbfce59c700 1 -- 172.21.15.195:0/2562273840 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbfd00c4710 msgr2=0x7fbfd00bc990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:56.054 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.976+0000 7fbfce59c700 1 --2- 172.21.15.195:0/2562273840 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbfd00c4710 0x7fbfd00bc990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:56.055 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.976+0000 7fbfce59c700 1 -- 172.21.15.195:0/2562273840 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfd014ecd0 con 0x7fbfb40046d0 2022-06-27T12:18:56.056 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.976+0000 7fbfce59c700 1 --2- 172.21.15.195:0/2562273840 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbfb40046d0 0x7fbfd0155c50 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7fbfc4011ad0 tx=0x7fbfc4009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:18:56.056 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.976+0000 7fbfbe7fc700 1 -- 172.21.15.195:0/2562273840 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfc4004240 con 0x7fbfb40046d0 2022-06-27T12:18:56.057 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.976+0000 7fbfbe7fc700 1 -- 172.21.15.195:0/2562273840 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfc40043a0 con 0x7fbfb40046d0 2022-06-27T12:18:56.057 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.976+0000 7fbfd4f27700 1 -- 172.21.15.195:0/2562273840 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfd00bd650 con 0x7fbfb40046d0 2022-06-27T12:18:56.058 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.976+0000 7fbfbe7fc700 1 -- 172.21.15.195:0/2562273840 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfc400fa10 con 0x7fbfb40046d0 2022-06-27T12:18:56.058 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.976+0000 7fbfd4f27700 1 -- 172.21.15.195:0/2562273840 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfd00bdbd0 con 0x7fbfb40046d0 2022-06-27T12:18:56.059 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.977+0000 7fbfd4f27700 1 -- 172.21.15.195:0/2562273840 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfd0169e90 con 0x7fbfb40046d0 2022-06-27T12:18:56.059 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.978+0000 7fbfbe7fc700 1 -- 172.21.15.195:0/2562273840 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7fbfc4024070 con 0x7fbfb40046d0 2022-06-27T12:18:56.060 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.978+0000 7fbfbe7fc700 1 --2- 172.21.15.195:0/2562273840 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbfac063d30 0x7fbfac0661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:56.060 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.978+0000 7fbfbe7fc700 1 -- 172.21.15.195:0/2562273840 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7750+0+0 (secure 0 0 0) 0x7fbfc4018030 con 0x7fbfb40046d0 2022-06-27T12:18:56.060 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.978+0000 7fbfcdd9b700 1 --2- 172.21.15.195:0/2562273840 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbfac063d30 0x7fbfac0661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:56.061 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.978+0000 7fbfcdd9b700 1 --2- 172.21.15.195:0/2562273840 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbfac063d30 0x7fbfac0661e0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fbfc0014000 tx=0x7fbfc0015000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:18:56.061 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:55.982+0000 7fbfbe7fc700 1 -- 172.21.15.195:0/2562273840 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7fbfd0169e90 con 0x7fbfb40046d0 2022-06-27T12:18:56.210 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:56.207+0000 7fbfd4f27700 1 -- 172.21.15.195:0/2562273840 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbfd0169e90 con 0x7fbfb40046d0 2022-06-27T12:18:56.211 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:56.208+0000 7fbfbe7fc700 1 -- 172.21.15.195:0/2562273840 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+5377 (secure 0 0 0) 0x7fbfd0169e90 con 0x7fbfb40046d0 2022-06-27T12:18:56.212 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:18:56.212 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":117,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:18:52.379960+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7300},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7300":{"gid":7300,"name":"b","rank":0,"incarnation":114,"state":"up:clientreplay","state_seq":21,"addr":"172.21.15.202:6833/1965783412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1965783412},{"type":"v1","addr":"172.21.15.202:6833","nonce":1965783412}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:18:56.216 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:56.209+0000 7fbfd4f27700 1 -- 172.21.15.195:0/2562273840 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbfac063d30 msgr2=0x7fbfac0661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:56.216 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:56.210+0000 7fbfd4f27700 1 --2- 172.21.15.195:0/2562273840 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbfac063d30 0x7fbfac0661e0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fbfc0014000 tx=0x7fbfc0015000 comp rx=0 tx=0).stop 2022-06-27T12:18:56.217 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:56.210+0000 7fbfd4f27700 1 -- 172.21.15.195:0/2562273840 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbfb40046d0 msgr2=0x7fbfd0155c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:56.217 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:56.210+0000 7fbfd4f27700 1 --2- 172.21.15.195:0/2562273840 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbfb40046d0 0x7fbfd0155c50 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7fbfc4011ad0 tx=0x7fbfc4009060 comp rx=0 tx=0).stop 2022-06-27T12:18:56.218 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:56.210+0000 7fbfd4f27700 1 -- 172.21.15.195:0/2562273840 shutdown_connections 2022-06-27T12:18:56.218 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:56.210+0000 7fbfd4f27700 1 --2- 172.21.15.195:0/2562273840 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7fbfac063d30 0x7fbfac0661e0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:56.219 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:56.210+0000 7fbfd4f27700 1 --2- 172.21.15.195:0/2562273840 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbfb40046d0 0x7fbfd0155c50 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:56.219 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:56.210+0000 7fbfd4f27700 1 --2- 172.21.15.195:0/2562273840 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7fbfd00c4710 0x7fbfd00bc990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:56.220 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:56.210+0000 7fbfd4f27700 1 --2- 172.21.15.195:0/2562273840 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7fbfd014e880 0x7fbfd016a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:56.220 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:56.210+0000 7fbfd4f27700 1 -- 172.21.15.195:0/2562273840 >> 172.21.15.195:0/2562273840 conn(0x7fbfd00c6350 msgr2=0x7fbfd01507b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:18:56.220 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:56.210+0000 7fbfd4f27700 1 -- 172.21.15.195:0/2562273840 shutdown_connections 2022-06-27T12:18:56.221 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:56.210+0000 7fbfd4f27700 1 -- 172.21.15.195:0/2562273840 wait complete. 2022-06-27T12:18:56.221 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 117 2022-06-27T12:18:56.229 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 117, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-06-27T11:47:52.358095+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 117, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7300": { "addr": "172.21.15.202:6833/1965783412", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6832", "nonce": 1965783412, "type": "v2" }, { "addr": "172.21.15.202:6833", "nonce": 1965783412, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7300, "incarnation": 114, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:clientreplay", "state_seq": 21 } }, "last_failure": 0, "last_failure_osd_epoch": 66, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-06-27T12:18:52.379960+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7300 } } } ], "standbys": [ { "addr": "172.21.15.202:6837/3305006841", "addrs": { "addrvec": [ { "addr": "172.21.15.202:6836", "nonce": 3305006841, "type": "v2" }, { "addr": "172.21.15.202:6837", "nonce": 3305006841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6835/2060709822", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6834", "nonce": 2060709822, "type": "v2" }, { "addr": "172.21.15.195:6835", "nonce": 2060709822, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6838/1598751652", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6836", "nonce": 1598751652, "type": "v2" }, { "addr": "172.21.15.195:6838", "nonce": 1598751652, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4275, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.195:6839/1313800661", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6837", "nonce": 1313800661, "type": "v2" }, { "addr": "172.21.15.195:6839", "nonce": 1313800661, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4278, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-06-27T12:18:58.232 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:18:58.357 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.357+0000 7f733715d700 1 Processor -- start 2022-06-27T12:18:58.358 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.357+0000 7f733715d700 1 -- start start 2022-06-27T12:18:58.360 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.358+0000 7f733715d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f733006fef0 0x7f73301570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:58.361 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.358+0000 7f733715d700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f73300c34a0 0x7f73300c2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:58.361 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.358+0000 7f733715d700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f733006fba0 con 0x7f733006e850 2022-06-27T12:18:58.361 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.358+0000 7f733715d700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73301575e0 con 0x7f733006fef0 2022-06-27T12:18:58.361 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.358+0000 7f733715d700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f733006f5e0 con 0x7f73300c34a0 2022-06-27T12:18:58.362 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.358+0000 7f732ffff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f733006fef0 0x7f73301570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:58.362 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.358+0000 7f732ffff700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f733006fef0 0x7f73301570a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:53094/0 (socket says 172.21.15.195:53094) 2022-06-27T12:18:58.363 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.358+0000 7f732ffff700 1 -- 172.21.15.195:0/78458600 learned_addr learned my addr 172.21.15.195:0/78458600 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:18:58.363 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.358+0000 7f73356fa700 1 --2- 172.21.15.195:0/78458600 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f73300c34a0 0x7f73300c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:58.363 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.358+0000 7f732ffff700 1 -- 172.21.15.195:0/78458600 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f73300c34a0 msgr2=0x7f73300c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:18:58.364 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.358+0000 7f732ffff700 1 --2- 172.21.15.195:0/78458600 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f73300c34a0 0x7f73300c2920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:58.364 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.358+0000 7f732ffff700 1 -- 172.21.15.195:0/78458600 >> v1:172.21.15.202:6789/0 conn(0x7f733006e850 legacy=0x7f7330069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:18:58.364 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.358+0000 7f732ffff700 1 -- 172.21.15.195:0/78458600 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f733006f8c0 con 0x7f733006fef0 2022-06-27T12:18:58.365 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.358+0000 7f732ffff700 1 --2- 172.21.15.195:0/78458600 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f733006fef0 0x7f73301570a0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f731c009860 tx=0x7f731c009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e44969aa8e4e8716 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:18:58.365 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.358+0000 7f732f7fe700 1 -- 172.21.15.195:0/78458600 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f731c004030 con 0x7f733006fef0 2022-06-27T12:18:58.365 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.359+0000 7f732f7fe700 1 -- 172.21.15.195:0/78458600 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f731c004190 con 0x7f733006fef0 2022-06-27T12:18:58.366 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.359+0000 7f732f7fe700 1 -- 172.21.15.195:0/78458600 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f731c00b6d0 con 0x7f733006fef0 2022-06-27T12:18:58.366 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.359+0000 7f733715d700 1 -- 172.21.15.195:0/78458600 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f733006fef0 msgr2=0x7f73301570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:58.367 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.359+0000 7f733715d700 1 --2- 172.21.15.195:0/78458600 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f733006fef0 0x7f73301570a0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f731c009860 tx=0x7f731c009120 comp rx=0 tx=0).stop 2022-06-27T12:18:58.371 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.359+0000 7f733715d700 1 -- 172.21.15.195:0/78458600 shutdown_connections 2022-06-27T12:18:58.372 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.359+0000 7f733715d700 1 --2- 172.21.15.195:0/78458600 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f73300c34a0 0x7f73300c2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:58.372 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.359+0000 7f733715d700 1 --2- 172.21.15.195:0/78458600 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f733006fef0 0x7f73301570a0 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:58.372 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.359+0000 7f733715d700 1 -- 172.21.15.195:0/78458600 >> 172.21.15.195:0/78458600 conn(0x7f73300c6350 msgr2=0x7f73300c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:18:58.373 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.359+0000 7f733715d700 1 -- 172.21.15.195:0/78458600 shutdown_connections 2022-06-27T12:18:58.373 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.359+0000 7f733715d700 1 -- 172.21.15.195:0/78458600 wait complete. 2022-06-27T12:18:58.373 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.360+0000 7f733715d700 1 Processor -- start 2022-06-27T12:18:58.374 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.361+0000 7f733715d700 1 -- start start 2022-06-27T12:18:58.374 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.361+0000 7f733715d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f733006e850 0x7f733014da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:58.374 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.362+0000 7f733715d700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f73300c34a0 0x7f733014dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:58.375 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.362+0000 7f733715d700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f73300c2c00 0x7f733014c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:58.375 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.362+0000 7f733715d700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f733014c270 con 0x7f733006e850 2022-06-27T12:18:58.375 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.362+0000 7f733715d700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7330069820 con 0x7f73300c34a0 2022-06-27T12:18:58.376 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.362+0000 7f733715d700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f733014e5f0 con 0x7f73300c2c00 2022-06-27T12:18:58.376 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.362+0000 7f7334ef9700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f733006e850 0x7f733014da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:58.376 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.362+0000 7f7334ef9700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f733006e850 0x7f733014da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:53100/0 (socket says 172.21.15.195:53100) 2022-06-27T12:18:58.376 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.362+0000 7f7334ef9700 1 -- 172.21.15.195:0/2621563418 learned_addr learned my addr 172.21.15.195:0/2621563418 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:18:58.377 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.362+0000 7f73356fa700 1 --2- 172.21.15.195:0/2621563418 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f73300c2c00 0x7f733014c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:58.377 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.362+0000 7f732ffff700 1 --2- 172.21.15.195:0/2621563418 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f73300c34a0 0x7f733014dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:58.377 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.362+0000 7f7334ef9700 1 -- 172.21.15.195:0/2621563418 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f73300c2c00 msgr2=0x7f733014c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:58.377 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.362+0000 7f7334ef9700 1 --2- 172.21.15.195:0/2621563418 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f73300c2c00 0x7f733014c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:58.378 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.362+0000 7f7334ef9700 1 -- 172.21.15.195:0/2621563418 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f73300c34a0 msgr2=0x7f733014dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:58.378 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.362+0000 7f7334ef9700 1 --2- 172.21.15.195:0/2621563418 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f73300c34a0 0x7f733014dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:58.378 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.362+0000 7f7334ef9700 1 -- 172.21.15.195:0/2621563418 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f731c0093a0 con 0x7f733006e850 2022-06-27T12:18:58.378 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.362+0000 7f732ffff700 1 --2- 172.21.15.195:0/2621563418 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f73300c34a0 0x7f733014dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-06-27T12:18:58.379 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.362+0000 7f7334ef9700 1 --2- 172.21.15.195:0/2621563418 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f733006e850 0x7f733014da90 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f732400b440 tx=0x7f7324006f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:18:58.379 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.362+0000 7f732cff9700 1 -- 172.21.15.195:0/2621563418 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7324011630 con 0x7f733006e850 2022-06-27T12:18:58.379 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.362+0000 7f732cff9700 1 -- 172.21.15.195:0/2621563418 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7324011790 con 0x7f733006e850 2022-06-27T12:18:58.379 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.362+0000 7f732cff9700 1 -- 172.21.15.195:0/2621563418 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7324011b50 con 0x7f733006e850 2022-06-27T12:18:58.380 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.362+0000 7f733715d700 1 -- 172.21.15.195:0/2621563418 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f733014b660 con 0x7f733006e850 2022-06-27T12:18:58.380 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.362+0000 7f733715d700 1 -- 172.21.15.195:0/2621563418 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f733014bc20 con 0x7f733006e850 2022-06-27T12:18:58.380 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.363+0000 7f733715d700 1 -- 172.21.15.195:0/2621563418 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f733014c550 con 0x7f733006e850 2022-06-27T12:18:58.381 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.364+0000 7f732cff9700 1 -- 172.21.15.195:0/2621563418 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f732400f050 con 0x7f733006e850 2022-06-27T12:18:58.381 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.364+0000 7f732cff9700 1 --2- 172.21.15.195:0/2621563418 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7318063d00 0x7f73180661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:18:58.381 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.364+0000 7f732cff9700 1 -- 172.21.15.195:0/2621563418 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7750+0+0 (secure 0 0 0) 0x7f7324080990 con 0x7f733006e850 2022-06-27T12:18:58.382 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.364+0000 7f732ffff700 1 --2- 172.21.15.195:0/2621563418 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7318063d00 0x7f73180661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:18:58.382 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.364+0000 7f732ffff700 1 --2- 172.21.15.195:0/2621563418 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7318063d00 0x7f73180661b0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f733015e4a0 tx=0x7f731c016000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:18:58.382 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.368+0000 7f732cff9700 1 -- 172.21.15.195:0/2621563418 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f732404eee0 con 0x7f733006e850 2022-06-27T12:18:58.600 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.600+0000 7f733715d700 1 -- 172.21.15.195:0/2621563418 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f73300722c0 con 0x7f733006e850 2022-06-27T12:18:58.601 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.600+0000 7f732cff9700 1 -- 172.21.15.195:0/2621563418 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) v1 ==== 95+0+5371 (secure 0 0 0) 0x7f7324015090 con 0x7f733006e850 2022-06-27T12:18:58.602 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:18:58.602 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":118,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:18:58.402136+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7300},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7300":{"gid":7300,"name":"b","rank":0,"incarnation":114,"state":"up:active","state_seq":23,"addr":"172.21.15.202:6833/1965783412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1965783412},{"type":"v1","addr":"172.21.15.202:6833","nonce":1965783412}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:18:58.605 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.602+0000 7f733715d700 1 -- 172.21.15.195:0/2621563418 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7318063d00 msgr2=0x7f73180661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:58.606 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.602+0000 7f733715d700 1 --2- 172.21.15.195:0/2621563418 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7318063d00 0x7f73180661b0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f733015e4a0 tx=0x7f731c016000 comp rx=0 tx=0).stop 2022-06-27T12:18:58.606 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.602+0000 7f733715d700 1 -- 172.21.15.195:0/2621563418 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f733006e850 msgr2=0x7f733014da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:18:58.607 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.602+0000 7f733715d700 1 --2- 172.21.15.195:0/2621563418 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f733006e850 0x7f733014da90 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f732400b440 tx=0x7f7324006f60 comp rx=0 tx=0).stop 2022-06-27T12:18:58.607 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.602+0000 7f733715d700 1 -- 172.21.15.195:0/2621563418 shutdown_connections 2022-06-27T12:18:58.608 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.602+0000 7f733715d700 1 --2- 172.21.15.195:0/2621563418 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f7318063d00 0x7f73180661b0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:58.608 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.602+0000 7f733715d700 1 --2- 172.21.15.195:0/2621563418 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f733006e850 0x7f733014da90 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:58.609 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.602+0000 7f733715d700 1 --2- 172.21.15.195:0/2621563418 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f73300c2c00 0x7f733014c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:58.609 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.602+0000 7f733715d700 1 --2- 172.21.15.195:0/2621563418 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f73300c34a0 0x7f733014dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:18:58.610 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.602+0000 7f733715d700 1 -- 172.21.15.195:0/2621563418 >> 172.21.15.195:0/2621563418 conn(0x7f73300c6350 msgr2=0x7f73301504d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:18:58.610 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.602+0000 7f733715d700 1 -- 172.21.15.195:0/2621563418 shutdown_connections 2022-06-27T12:18:58.610 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:18:58.602+0000 7f733715d700 1 -- 172.21.15.195:0/2621563418 wait complete. 2022-06-27T12:18:58.611 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 118 2022-06-27T12:18:58.615 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=7165 2022-06-27T12:18:58.623 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 4 secs before reviving mds.d 2022-06-27T12:19:02.624 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.d 2022-06-27T12:19:02.625 INFO:tasks.ceph.mds.d:Restarting daemon 2022-06-27T12:19:02.626 DEBUG:teuthology.orchestra.run.smithi202:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2022-06-27T12:19:02.629 INFO:tasks.ceph.mds.d:Started 2022-06-27T12:19:02.629 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:19:02.719 INFO:tasks.ceph.mds.d.smithi202.stdout:starting mds.d at 2022-06-27T12:19:02.765 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:02.764+0000 7f4e79101700 1 Processor -- start 2022-06-27T12:19:02.765 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:02.765+0000 7f4e79101700 1 -- start start 2022-06-27T12:19:02.768 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:02.765+0000 7f4e79101700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f4e74150500 0x7f4e74063af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:19:02.768 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:02.765+0000 7f4e79101700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4e741508d0 0x7f4e740bbf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:19:02.769 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:02.765+0000 7f4e79101700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e74064030 con 0x7f4e7406a450 2022-06-27T12:19:02.769 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:02.765+0000 7f4e79101700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e740c4160 con 0x7f4e74150500 2022-06-27T12:19:02.769 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:02.765+0000 7f4e79101700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e740c4440 con 0x7f4e741508d0 2022-06-27T12:19:02.769 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:02.766+0000 7f4e72d9d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f4e74150500 0x7f4e74063af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:19:02.769 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:02.766+0000 7f4e7259c700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f4e7406a450 0x7f4e740c48d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:59302/0 (socket says 172.21.15.195:59302) 2022-06-27T12:19:02.770 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:02.766+0000 7f4e7259c700 1 -- 172.21.15.195:0/727221371 learned_addr learned my addr 172.21.15.195:0/727221371 (peer_addr_for_me v1:172.21.15.195:0/0) 2022-06-27T12:19:02.770 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:02.766+0000 7f4e7359e700 1 --2- 172.21.15.195:0/727221371 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4e741508d0 0x7f4e740bbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:19:02.770 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:02.766+0000 7f4e72d9d700 1 -- 172.21.15.195:0/727221371 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4e741508d0 msgr2=0x7f4e740bbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:19:02.770 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:02.766+0000 7f4e72d9d700 1 --2- 172.21.15.195:0/727221371 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4e741508d0 0x7f4e740bbf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:19:02.771 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:02.766+0000 7f4e72d9d700 1 -- 172.21.15.195:0/727221371 >> v1:172.21.15.195:6789/0 conn(0x7f4e7406a450 legacy=0x7f4e740c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:19:02.771 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:02.766+0000 7f4e71d9b700 1 -- 172.21.15.195:0/727221371 <== mon.0 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 52607469 0 0) 0x7f4e64001110 con 0x7f4e7406a450 2022-06-27T12:19:02.771 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:02.766+0000 7f4e72d9d700 1 -- 172.21.15.195:0/727221371 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e7415a800 con 0x7f4e74150500 2022-06-27T12:19:02.771 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:02.766+0000 7f4e72d9d700 1 --2- 172.21.15.195:0/727221371 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4e74150500 0x7f4e74063af0 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f4e5c009720 tx=0x7f4e5c009a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c51b808e0cc1be4 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:19:02.772 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:02.766+0000 7f4e71d9b700 1 -- 172.21.15.195:0/727221371 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e5c005ad0 con 0x7f4e74150500 2022-06-27T12:19:02.772 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:02.766+0000 7f4e71d9b700 1 -- 172.21.15.195:0/727221371 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e5c005c30 con 0x7f4e74150500 2022-06-27T12:19:02.772 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:02.766+0000 7f4e71d9b700 1 -- 172.21.15.195:0/727221371 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e5c006030 con 0x7f4e74150500 2022-06-27T12:19:02.772 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:02.767+0000 7f4e79101700 1 -- 172.21.15.195:0/727221371 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4e74150500 msgr2=0x7f4e74063af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:19:02.773 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:02.767+0000 7f4e79101700 1 --2- 172.21.15.195:0/727221371 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4e74150500 0x7f4e74063af0 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f4e5c009720 tx=0x7f4e5c009a30 comp rx=0 tx=0).stop 2022-06-27T12:19:02.773 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:02.767+0000 7f4e79101700 1 -- 172.21.15.195:0/727221371 shutdown_connections 2022-06-27T12:19:02.774 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:02.767+0000 7f4e79101700 1 --2- 172.21.15.195:0/727221371 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4e741508d0 0x7f4e740bbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:19:02.774 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:02.767+0000 7f4e79101700 1 --2- 172.21.15.195:0/727221371 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4e74150500 0x7f4e74063af0 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:19:02.774 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:02.767+0000 7f4e79101700 1 -- 172.21.15.195:0/727221371 >> 172.21.15.195:0/727221371 conn(0x7f4e740c9b00 msgr2=0x7f4e740c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:19:02.775 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:02.767+0000 7f4e79101700 1 -- 172.21.15.195:0/727221371 shutdown_connections 2022-06-27T12:19:02.775 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:02.767+0000 7f4e79101700 1 -- 172.21.15.195:0/727221371 wait complete. 2022-06-27T12:19:02.775 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:02.768+0000 7f4e79101700 1 Processor -- start 2022-06-27T12:19:02.775 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:02.768+0000 7f4e79101700 1 -- start start 2022-06-27T12:19:02.778 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:02.769+0000 7f4e79101700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4e7406a450 0x7f4e740c5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:19:02.779 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:02.769+0000 7f4e79101700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4e74150500 0x7f4e7416a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:19:02.779 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:02.769+0000 7f4e79101700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4e741508d0 0x7f4e74166be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:19:02.779 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:02.769+0000 7f4e79101700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e74169fb0 con 0x7f4e7406a450 2022-06-27T12:19:02.779 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:02.769+0000 7f4e79101700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e741667b0 con 0x7f4e741508d0 2022-06-27T12:19:02.780 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:02.769+0000 7f4e79101700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e7406ecf0 con 0x7f4e74150500 2022-06-27T12:19:02.780 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:02.769+0000 7f4e72d9d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4e7406a450 0x7f4e740c5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:19:02.780 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:02.769+0000 7f4e72d9d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4e7406a450 0x7f4e740c5250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:53118/0 (socket says 172.21.15.195:53118) 2022-06-27T12:19:02.780 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:02.769+0000 7f4e72d9d700 1 -- 172.21.15.195:0/3193496775 learned_addr learned my addr 172.21.15.195:0/3193496775 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:19:02.781 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:02.769+0000 7f4e7259c700 1 --2- 172.21.15.195:0/3193496775 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4e74150500 0x7f4e7416a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:19:02.781 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:02.769+0000 7f4e7359e700 1 --2- 172.21.15.195:0/3193496775 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4e741508d0 0x7f4e74166be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:19:02.781 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:02.769+0000 7f4e72d9d700 1 -- 172.21.15.195:0/3193496775 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4e74150500 msgr2=0x7f4e7416a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:19:02.781 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:02.769+0000 7f4e72d9d700 1 --2- 172.21.15.195:0/3193496775 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4e74150500 0x7f4e7416a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:19:02.781 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:02.769+0000 7f4e72d9d700 1 -- 172.21.15.195:0/3193496775 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4e741508d0 msgr2=0x7f4e74166be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:19:02.782 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:02.769+0000 7f4e72d9d700 1 --2- 172.21.15.195:0/3193496775 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4e741508d0 0x7f4e74166be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:19:02.782 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:02.769+0000 7f4e72d9d700 1 -- 172.21.15.195:0/3193496775 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e5c009290 con 0x7f4e7406a450 2022-06-27T12:19:02.782 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:02.769+0000 7f4e7259c700 1 --2- 172.21.15.195:0/3193496775 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4e74150500 0x7f4e7416a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:19:02.783 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:02.769+0000 7f4e72d9d700 1 --2- 172.21.15.195:0/3193496775 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4e7406a450 0x7f4e740c5250 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f4e5c012000 tx=0x7f4e5c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:19:02.783 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:02.770+0000 7f4e6affd700 1 -- 172.21.15.195:0/3193496775 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e5c01d780 con 0x7f4e7406a450 2022-06-27T12:19:02.783 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:02.770+0000 7f4e6affd700 1 -- 172.21.15.195:0/3193496775 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e5c01d920 con 0x7f4e7406a450 2022-06-27T12:19:02.783 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:02.770+0000 7f4e79101700 1 -- 172.21.15.195:0/3193496775 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e74154310 con 0x7f4e7406a450 2022-06-27T12:19:02.784 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:02.770+0000 7f4e6affd700 1 -- 172.21.15.195:0/3193496775 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e5c01dce0 con 0x7f4e7406a450 2022-06-27T12:19:02.784 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:02.770+0000 7f4e79101700 1 -- 172.21.15.195:0/3193496775 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e741548d0 con 0x7f4e7406a450 2022-06-27T12:19:02.784 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:02.771+0000 7f4e79101700 1 -- 172.21.15.195:0/3193496775 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e74169d80 con 0x7f4e7406a450 2022-06-27T12:19:02.784 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:02.771+0000 7f4e6affd700 1 -- 172.21.15.195:0/3193496775 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f4e5c02c020 con 0x7f4e7406a450 2022-06-27T12:19:02.784 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:02.771+0000 7f4e6affd700 1 --2- 172.21.15.195:0/3193496775 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4e58063d00 0x7f4e580661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:19:02.785 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:02.771+0000 7f4e6affd700 1 -- 172.21.15.195:0/3193496775 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7750+0+0 (secure 0 0 0) 0x7f4e5c082f00 con 0x7f4e7406a450 2022-06-27T12:19:02.785 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:02.771+0000 7f4e7259c700 1 --2- 172.21.15.195:0/3193496775 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4e58063d00 0x7f4e580661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:19:02.785 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:02.772+0000 7f4e7259c700 1 --2- 172.21.15.195:0/3193496775 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4e58063d00 0x7f4e580661b0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f4e740c4af0 tx=0x7f4e64007050 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:19:02.785 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:02.776+0000 7f4e6affd700 1 -- 172.21.15.195:0/3193496775 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f4e74169d80 con 0x7f4e7406a450 2022-06-27T12:19:03.004 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:03.004+0000 7f4e79101700 1 -- 172.21.15.195:0/3193496775 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4e74169d80 con 0x7f4e7406a450 2022-06-27T12:19:03.005 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:03.004+0000 7f4e6affd700 1 -- 172.21.15.195:0/3193496775 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) v1 ==== 95+0+5371 (secure 0 0 0) 0x7f4e74169d80 con 0x7f4e7406a450 2022-06-27T12:19:03.006 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:19:03.007 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":118,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:18:58.402136+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7300},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7300":{"gid":7300,"name":"b","rank":0,"incarnation":114,"state":"up:active","state_seq":23,"addr":"172.21.15.202:6833/1965783412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1965783412},{"type":"v1","addr":"172.21.15.202:6833","nonce":1965783412}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:19:03.010 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:03.006+0000 7f4e79101700 1 -- 172.21.15.195:0/3193496775 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4e58063d00 msgr2=0x7f4e580661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:19:03.011 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:03.006+0000 7f4e79101700 1 --2- 172.21.15.195:0/3193496775 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4e58063d00 0x7f4e580661b0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f4e740c4af0 tx=0x7f4e64007050 comp rx=0 tx=0).stop 2022-06-27T12:19:03.011 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:03.006+0000 7f4e79101700 1 -- 172.21.15.195:0/3193496775 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4e7406a450 msgr2=0x7f4e740c5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:19:03.011 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:03.006+0000 7f4e79101700 1 --2- 172.21.15.195:0/3193496775 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4e7406a450 0x7f4e740c5250 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f4e5c012000 tx=0x7f4e5c016000 comp rx=0 tx=0).stop 2022-06-27T12:19:03.011 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:03.006+0000 7f4e79101700 1 -- 172.21.15.195:0/3193496775 shutdown_connections 2022-06-27T12:19:03.012 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:03.006+0000 7f4e79101700 1 --2- 172.21.15.195:0/3193496775 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f4e58063d00 0x7f4e580661b0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:19:03.012 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:03.006+0000 7f4e79101700 1 --2- 172.21.15.195:0/3193496775 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4e7406a450 0x7f4e740c5250 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:19:03.012 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:03.006+0000 7f4e79101700 1 --2- 172.21.15.195:0/3193496775 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f4e741508d0 0x7f4e74166be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:19:03.012 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:03.006+0000 7f4e79101700 1 --2- 172.21.15.195:0/3193496775 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f4e74150500 0x7f4e7416a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:19:03.013 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:03.007+0000 7f4e79101700 1 -- 172.21.15.195:0/3193496775 >> 172.21.15.195:0/3193496775 conn(0x7f4e740c9b00 msgr2=0x7f4e74067ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:19:03.014 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:03.007+0000 7f4e79101700 1 -- 172.21.15.195:0/3193496775 shutdown_connections 2022-06-27T12:19:03.014 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:03.007+0000 7f4e79101700 1 -- 172.21.15.195:0/3193496775 wait complete. 2022-06-27T12:19:03.015 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 118 2022-06-27T12:19:03.024 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2022-06-27T12:19:03.604 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:19:03.604 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 83 -eq 48 ']' 2022-06-27T12:19:03.605 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.83 2022-06-27T12:19:03.605 INFO:tasks.workunit.client.0.smithi195.stdout:v0.83 2022-06-27T12:19:03.606 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.83 2022-06-27T12:19:03.607 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.83 2022-06-27T12:19:05.025 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-27T12:19:05.222 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.175+0000 7f37c7e33700 1 Processor -- start 2022-06-27T12:19:05.224 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.175+0000 7f37c7e33700 1 -- start start 2022-06-27T12:19:05.224 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.176+0000 7f37c7e33700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f37c0150950 0x7f37c0156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:19:05.224 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.176+0000 7f37c7e33700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f37c006f620 0x7f37c006d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:19:05.225 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.176+0000 7f37c7e33700 1 -- --> v1:172.21.15.202:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37c00c76b0 con 0x7f37c0150580 2022-06-27T12:19:05.225 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.176+0000 7f37c7e33700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37c0157430 con 0x7f37c006f620 2022-06-27T12:19:05.226 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.176+0000 7f37c7e33700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37c00c8250 con 0x7f37c0150950 2022-06-27T12:19:05.226 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.176+0000 7f37c63d0700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f37c006f620 0x7f37c006d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:19:05.227 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.177+0000 7f37c63d0700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f37c006f620 0x7f37c006d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:53140/0 (socket says 172.21.15.195:53140) 2022-06-27T12:19:05.227 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.177+0000 7f37c63d0700 1 -- 172.21.15.195:0/2271344240 learned_addr learned my addr 172.21.15.195:0/2271344240 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:19:05.228 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.177+0000 7f37c53ce700 1 --2- 172.21.15.195:0/2271344240 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f37c0150950 0x7f37c0156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:19:05.228 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.177+0000 7f37c63d0700 1 -- 172.21.15.195:0/2271344240 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f37c0150950 msgr2=0x7f37c0156ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-27T12:19:05.229 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.177+0000 7f37c63d0700 1 --2- 172.21.15.195:0/2271344240 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f37c0150950 0x7f37c0156ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:19:05.235 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.177+0000 7f37c63d0700 1 -- 172.21.15.195:0/2271344240 >> v1:172.21.15.202:6789/0 conn(0x7f37c0150580 legacy=0x7f37c00c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:19:05.235 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.177+0000 7f37c63d0700 1 -- 172.21.15.195:0/2271344240 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37c00bad70 con 0x7f37c006f620 2022-06-27T12:19:05.236 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.177+0000 7f37c53ce700 1 --2- 172.21.15.195:0/2271344240 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f37c0150950 0x7f37c0156ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:19:05.236 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.177+0000 7f37c63d0700 1 --2- 172.21.15.195:0/2271344240 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f37c006f620 0x7f37c006d6a0 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f37b8009f20 tx=0x7f37b8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c8306660a27d372 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:19:05.237 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.177+0000 7f37c4bcd700 1 -- 172.21.15.195:0/2271344240 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37b800e070 con 0x7f37c006f620 2022-06-27T12:19:05.237 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.177+0000 7f37c4bcd700 1 -- 172.21.15.195:0/2271344240 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37b800ba80 con 0x7f37c006f620 2022-06-27T12:19:05.238 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.177+0000 7f37c4bcd700 1 -- 172.21.15.195:0/2271344240 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37b80092e0 con 0x7f37c006f620 2022-06-27T12:19:05.238 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.178+0000 7f37c7e33700 1 -- 172.21.15.195:0/2271344240 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f37c006f620 msgr2=0x7f37c006d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:19:05.239 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.178+0000 7f37c7e33700 1 --2- 172.21.15.195:0/2271344240 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f37c006f620 0x7f37c006d6a0 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f37b8009f20 tx=0x7f37b8009840 comp rx=0 tx=0).stop 2022-06-27T12:19:05.239 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.178+0000 7f37c7e33700 1 -- 172.21.15.195:0/2271344240 shutdown_connections 2022-06-27T12:19:05.239 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.178+0000 7f37c7e33700 1 --2- 172.21.15.195:0/2271344240 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f37c006f620 0x7f37c006d6a0 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:19:05.240 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.178+0000 7f37c7e33700 1 --2- 172.21.15.195:0/2271344240 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f37c0150950 0x7f37c0156ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:19:05.240 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.178+0000 7f37c7e33700 1 -- 172.21.15.195:0/2271344240 >> 172.21.15.195:0/2271344240 conn(0x7f37c0063af0 msgr2=0x7f37c0063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:19:05.241 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.178+0000 7f37c7e33700 1 -- 172.21.15.195:0/2271344240 shutdown_connections 2022-06-27T12:19:05.241 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.178+0000 7f37c7e33700 1 -- 172.21.15.195:0/2271344240 wait complete. 2022-06-27T12:19:05.243 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.179+0000 7f37c7e33700 1 Processor -- start 2022-06-27T12:19:05.243 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.179+0000 7f37c7e33700 1 -- start start 2022-06-27T12:19:05.244 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.180+0000 7f37c7e33700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f37c006f620 0x7f37c014d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:19:05.244 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.180+0000 7f37c7e33700 1 --2- >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f37c0150580 0x7f37c00bcca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:19:05.245 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.180+0000 7f37c7e33700 1 --2- >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f37c0150950 0x7f37c00bd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:19:05.245 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.180+0000 7f37c7e33700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37c00c1e90 con 0x7f37c006f620 2022-06-27T12:19:05.246 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.180+0000 7f37c7e33700 1 -- --> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37c00c2170 con 0x7f37c0150950 2022-06-27T12:19:05.246 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.180+0000 7f37c7e33700 1 -- --> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37c00c2450 con 0x7f37c0150580 2022-06-27T12:19:05.247 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.180+0000 7f37c5bcf700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f37c006f620 0x7f37c014d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:19:05.247 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.180+0000 7f37c5bcf700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f37c006f620 0x7f37c014d670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:53142/0 (socket says 172.21.15.195:53142) 2022-06-27T12:19:05.248 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.180+0000 7f37c5bcf700 1 -- 172.21.15.195:0/1605031310 learned_addr learned my addr 172.21.15.195:0/1605031310 (peer_addr_for_me v2:172.21.15.195:0/0) 2022-06-27T12:19:05.248 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.180+0000 7f37c63d0700 1 --2- 172.21.15.195:0/1605031310 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f37c0150950 0x7f37c00bd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:19:05.248 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.180+0000 7f37c53ce700 1 --2- 172.21.15.195:0/1605031310 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f37c0150580 0x7f37c00bcca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:19:05.249 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.180+0000 7f37c5bcf700 1 -- 172.21.15.195:0/1605031310 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f37c0150580 msgr2=0x7f37c00bcca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:19:05.249 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.180+0000 7f37c5bcf700 1 --2- 172.21.15.195:0/1605031310 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f37c0150580 0x7f37c00bcca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:19:05.250 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.180+0000 7f37c5bcf700 1 -- 172.21.15.195:0/1605031310 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f37c0150950 msgr2=0x7f37c00bd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:19:05.250 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.180+0000 7f37c5bcf700 1 --2- 172.21.15.195:0/1605031310 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f37c0150950 0x7f37c00bd1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:19:05.251 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.180+0000 7f37c5bcf700 1 -- 172.21.15.195:0/1605031310 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37b8009ae0 con 0x7f37c006f620 2022-06-27T12:19:05.251 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.180+0000 7f37c63d0700 1 --2- 172.21.15.195:0/1605031310 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f37c0150950 0x7f37c00bd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-27T12:19:05.252 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.180+0000 7f37c5bcf700 1 --2- 172.21.15.195:0/1605031310 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f37c006f620 0x7f37c014d670 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f37b400c490 tx=0x7f37b4007e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:19:05.252 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.180+0000 7f37b1ffb700 1 -- 172.21.15.195:0/1605031310 <== mon.0 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37b40050d0 con 0x7f37c006f620 2022-06-27T12:19:05.254 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.180+0000 7f37b1ffb700 1 -- 172.21.15.195:0/1605031310 <== mon.0 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37b4005230 con 0x7f37c006f620 2022-06-27T12:19:05.255 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.180+0000 7f37c7e33700 1 -- 172.21.15.195:0/1605031310 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37c00c2730 con 0x7f37c006f620 2022-06-27T12:19:05.255 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.180+0000 7f37b1ffb700 1 -- 172.21.15.195:0/1605031310 <== mon.0 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37b40117a0 con 0x7f37c006f620 2022-06-27T12:19:05.256 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.181+0000 7f37c7e33700 1 -- 172.21.15.195:0/1605031310 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37c0156150 con 0x7f37c006f620 2022-06-27T12:19:05.256 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.181+0000 7f37c7e33700 1 -- 172.21.15.195:0/1605031310 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37c0159890 con 0x7f37c006f620 2022-06-27T12:19:05.257 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.182+0000 7f37b1ffb700 1 -- 172.21.15.195:0/1605031310 <== mon.0 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 81916+0+0 (secure 0 0 0) 0x7f37b4019030 con 0x7f37c006f620 2022-06-27T12:19:05.257 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.182+0000 7f37b1ffb700 1 --2- 172.21.15.195:0/1605031310 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f37a0063d30 0x7f37a00661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-06-27T12:19:05.258 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.182+0000 7f37b1ffb700 1 -- 172.21.15.195:0/1605031310 <== mon.0 v2:172.21.15.195:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7750+0+0 (secure 0 0 0) 0x7f37b407ffd0 con 0x7f37c006f620 2022-06-27T12:19:05.258 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.182+0000 7f37c53ce700 1 --2- 172.21.15.195:0/1605031310 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f37a0063d30 0x7f37a00661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-06-27T12:19:05.259 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.182+0000 7f37c53ce700 1 --2- 172.21.15.195:0/1605031310 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f37a0063d30 0x7f37a00661e0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f37c00c1950 tx=0x7f37bc008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-27T12:19:05.259 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.186+0000 7f37b1ffb700 1 -- 172.21.15.195:0/1605031310 <== mon.0 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+167038 (secure 0 0 0) 0x7f37b4049c90 con 0x7f37c006f620 2022-06-27T12:19:05.413 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.413+0000 7f37c7e33700 1 -- 172.21.15.195:0/1605031310 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f37c00722c0 con 0x7f37c006f620 2022-06-27T12:19:05.414 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.413+0000 7f37b1ffb700 1 -- 172.21.15.195:0/1605031310 <== mon.0 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) v1 ==== 95+0+6132 (secure 0 0 0) 0x7f37b4017070 con 0x7f37c006f620 2022-06-27T12:19:05.415 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-06-27T12:19:05.415 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":119,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6837/3305006841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6836","nonce":3305006841},{"type":"v1","addr":"172.21.15.202:6837","nonce":3305006841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/2060709822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2060709822},{"type":"v1","addr":"172.21.15.195:6835","nonce":2060709822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6838/1598751652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1598751652},{"type":"v1","addr":"172.21.15.195:6838","nonce":1598751652}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/1313800661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6837","nonce":1313800661},{"type":"v1","addr":"172.21.15.195:6839","nonce":1313800661}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":7405,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.202:6835/2972997888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6834","nonce":2972997888},{"type":"v1","addr":"172.21.15.202:6835","nonce":2972997888}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":119}],"filesystems":[{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-27T11:47:52.358095+0000","modified":"2022-06-27T12:18:58.402136+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7300},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7300":{"gid":7300,"name":"b","rank":0,"incarnation":114,"state":"up:active","state_seq":23,"addr":"172.21.15.202:6833/1965783412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.202:6832","nonce":1965783412},{"type":"v1","addr":"172.21.15.202:6833","nonce":1965783412}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-27T12:19:05.417 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.415+0000 7f37c7e33700 1 -- 172.21.15.195:0/1605031310 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f37a0063d30 msgr2=0x7f37a00661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:19:05.417 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.415+0000 7f37c7e33700 1 --2- 172.21.15.195:0/1605031310 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f37a0063d30 0x7f37a00661e0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f37c00c1950 tx=0x7f37bc008040 comp rx=0 tx=0).stop 2022-06-27T12:19:05.417 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.415+0000 7f37c7e33700 1 -- 172.21.15.195:0/1605031310 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f37c006f620 msgr2=0x7f37c014d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-27T12:19:05.418 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.415+0000 7f37c7e33700 1 --2- 172.21.15.195:0/1605031310 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f37c006f620 0x7f37c014d670 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f37b400c490 tx=0x7f37b4007e40 comp rx=0 tx=0).stop 2022-06-27T12:19:05.418 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.416+0000 7f37c7e33700 1 -- 172.21.15.195:0/1605031310 shutdown_connections 2022-06-27T12:19:05.419 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.416+0000 7f37c7e33700 1 --2- 172.21.15.195:0/1605031310 >> [v2:172.21.15.195:6832/93955,v1:172.21.15.195:6833/93955] conn(0x7f37a0063d30 0x7f37a00661e0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:19:05.419 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.416+0000 7f37c7e33700 1 --2- 172.21.15.195:0/1605031310 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f37c006f620 0x7f37c014d670 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:19:05.419 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.416+0000 7f37c7e33700 1 --2- 172.21.15.195:0/1605031310 >> [v2:172.21.15.202:3300/0,v1:172.21.15.202:6789/0] conn(0x7f37c0150950 0x7f37c00bd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:19:05.420 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.416+0000 7f37c7e33700 1 --2- 172.21.15.195:0/1605031310 >> [v2:172.21.15.202:3301/0,v1:172.21.15.202:6790/0] conn(0x7f37c0150580 0x7f37c00bcca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-06-27T12:19:05.420 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.416+0000 7f37c7e33700 1 -- 172.21.15.195:0/1605031310 >> 172.21.15.195:0/1605031310 conn(0x7f37c0063af0 msgr2=0x7f37c014f4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-27T12:19:05.420 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.416+0000 7f37c7e33700 1 -- 172.21.15.195:0/1605031310 shutdown_connections 2022-06-27T12:19:05.421 INFO:teuthology.orchestra.run.smithi195.stderr:2022-06-27T12:19:05.416+0000 7f37c7e33700 1 -- 172.21.15.195:0/1605031310 wait complete. 2022-06-27T12:19:05.422 INFO:tasks.ceph.mds.b.smithi202.stderr:2022-06-27T12:19:05.416+0000 7fcbea804700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2022-06-27T12:19:05.423 INFO:teuthology.orchestra.run.smithi195.stderr:dumped fsmap epoch 119 2022-06-27T12:19:05.433 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d reported in up:standby state 2022-06-27T12:19:05.434 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 111 secs before thrashing 2022-06-27T12:19:05.683 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:19:05.687 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:19:18.405 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 54% (118/218) Updating files: 55% (120/218) Updating files: 56% (123/218) Updating files: 57% (125/218) Updating files: 58% (127/218) Updating files: 59% (129/218) Updating files: 60% (131/218) Updating files: 61% (133/218) Updating files: 62% (136/218) Updating files: 63% (138/218) Updating files: 64% (140/218) Updating files: 65% (142/218) Updating files: 66% (144/218) Updating files: 67% (147/218) Updating files: 68% (149/218) Updating files: 69% (151/218) Updating files: 70% (153/218) Updating files: 71% (155/218) Updating files: 72% (157/218) Updating files: 73% (160/218) Updating files: 74% (162/218) Updating files: 75% (164/218) Updating files: 76% (166/218) Updating files: 77% (168/218) Updating files: 78% (171/218) Updating files: 79% (173/218) Updating files: 80% (175/218) Updating files: 80% (176/218) Updating files: 81% (177/218) Updating files: 82% (179/218) Updating files: 83% (181/218) Updating files: 84% (184/218) Updating files: 85% (186/218) Updating files: 86% (188/218) Updating files: 87% (190/218) Updating files: 88% (192/218) Updating files: 89% (195/218) Updating files: 90% (197/218) Updating files: 91% (199/218) Updating files: 92% (201/218) Updating files: 93% (203/218) Updating files: 94% (205/218) Updating files: 95% (208/218) Updating files: 96% (210/218) Updating files: 97% (212/218) Updating files: 98% (214/218) Updating files: 99% (216/218) Updating files: 100% (218/218) Updating files: 100% (218/218), done. 2022-06-27T12:19:18.584 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at 78ff1f0a5df 0.83 2022-06-27T12:19:18.588 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.83 2022-06-27T12:19:18.631 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:19:18.631 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 84 -eq 48 ']' 2022-06-27T12:19:18.632 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.84 2022-06-27T12:19:18.632 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.84 2022-06-27T12:19:18.632 INFO:tasks.workunit.client.0.smithi195.stdout:v0.84 2022-06-27T12:19:18.633 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.84 2022-06-27T12:19:24.887 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 28% (101/356) Updating files: 29% (104/356) Updating files: 30% (107/356) Updating files: 31% (111/356) Updating files: 32% (114/356) Updating files: 33% (118/356) Updating files: 34% (122/356) Updating files: 35% (125/356) Updating files: 36% (129/356) Updating files: 37% (132/356) Updating files: 38% (136/356) Updating files: 39% (139/356) Updating files: 40% (143/356) Updating files: 41% (146/356) Updating files: 42% (150/356) Updating files: 43% (154/356) Updating files: 44% (157/356) Updating files: 45% (161/356) Updating files: 46% (164/356) Updating files: 47% (168/356) Updating files: 48% (171/356) Updating files: 49% (175/356) Updating files: 50% (178/356) Updating files: 51% (182/356) Updating files: 52% (186/356) Updating files: 53% (189/356) Updating files: 54% (193/356) Updating files: 55% (196/356) Updating files: 56% (200/356) Updating files: 57% (203/356) Updating files: 58% (207/356) Updating files: 59% (211/356) Updating files: 60% (214/356) Updating files: 61% (218/356) Updating files: 62% (221/356) Updating files: 63% (225/356) Updating files: 64% (228/356) Updating files: 65% (232/356) Updating files: 66% (235/356) Updating files: 67% (239/356) Updating files: 68% (243/356) Updating files: 69% (246/356) Updating files: 70% (250/356) Updating files: 71% (253/356) Updating files: 72% (257/356) Updating files: 73% (260/356) Updating files: 74% (264/356) Updating files: 75% (267/356) Updating files: 76% (271/356) Updating files: 77% (275/356) Updating files: 78% (278/356) Updating files: 79% (282/356) Updating files: 80% (285/356) Updating files: 81% (289/356) Updating files: 82% (292/356) Updating files: 83% (296/356) Updating files: 84% (300/356) Updating files: 85% (303/356) Updating files: 86% (307/356) Updating files: 87% (310/356) Updating files: 88% (314/356) Updating files: 89% (317/356) Updating files: 90% (321/356) Updating files: 91% (324/356) Updating files: 92% (328/356) Updating files: 93% (332/356) Updating files: 94% (335/356) Updating files: 95% (339/356) Updating files: 96% (342/356) Updating files: 97% (346/356) Updating files: 98% (349/356) Updating files: 99% (353/356) Updating files: 100% (356/356) Updating files: 100% (356/356), done. 2022-06-27T12:19:24.921 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at 8336f81c5c7 0.84 2022-06-27T12:19:24.922 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.84 2022-06-27T12:19:25.052 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:19:25.052 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 85 -eq 48 ']' 2022-06-27T12:19:25.053 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.85 2022-06-27T12:19:25.053 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.85 2022-06-27T12:19:25.054 INFO:tasks.workunit.client.0.smithi195.stdout:v0.85 2022-06-27T12:19:25.055 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.85 2022-06-27T12:19:26.933 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 76% (185/243) Updating files: 77% (188/243) Updating files: 78% (190/243) Updating files: 79% (192/243) Updating files: 80% (195/243) Updating files: 81% (197/243) Updating files: 82% (200/243) Updating files: 83% (202/243) Updating files: 84% (205/243) Updating files: 85% (207/243) Updating files: 86% (209/243) Updating files: 87% (212/243) Updating files: 88% (214/243) Updating files: 89% (217/243) Updating files: 90% (219/243) Updating files: 91% (222/243) Updating files: 92% (224/243) Updating files: 93% (226/243) Updating files: 94% (229/243) Updating files: 95% (231/243) Updating files: 96% (234/243) Updating files: 97% (236/243) Updating files: 98% (239/243) Updating files: 99% (241/243) Updating files: 100% (243/243) Updating files: 100% (243/243), done. 2022-06-27T12:19:27.253 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at a0c22842db9 0.85 2022-06-27T12:19:27.257 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.85 2022-06-27T12:19:27.352 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:19:27.353 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 86 -eq 48 ']' 2022-06-27T12:19:27.353 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.86 2022-06-27T12:19:27.353 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.86 2022-06-27T12:19:27.354 INFO:tasks.workunit.client.0.smithi195.stdout:v0.86 2022-06-27T12:19:27.355 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.86 2022-06-27T12:19:34.176 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 24% (109/447) Updating files: 25% (112/447) Updating files: 26% (117/447) Updating files: 27% (121/447) Updating files: 28% (126/447) Updating files: 29% (130/447) Updating files: 30% (135/447) Updating files: 31% (139/447) Updating files: 32% (144/447) Updating files: 33% (148/447) Updating files: 34% (152/447) Updating files: 35% (157/447) Updating files: 36% (161/447) Updating files: 37% (166/447) Updating files: 38% (170/447) Updating files: 39% (175/447) Updating files: 40% (179/447) Updating files: 41% (184/447) Updating files: 42% (188/447) Updating files: 43% (193/447) Updating files: 44% (197/447) Updating files: 45% (202/447) Updating files: 46% (206/447) Updating files: 47% (211/447) Updating files: 48% (215/447) Updating files: 49% (220/447) Updating files: 50% (224/447) Updating files: 51% (228/447) Updating files: 52% (233/447) Updating files: 52% (236/447) Updating files: 53% (237/447) Updating files: 54% (242/447) Updating files: 55% (246/447) Updating files: 56% (251/447) Updating files: 57% (255/447) Updating files: 58% (260/447) Updating files: 59% (264/447) Updating files: 60% (269/447) Updating files: 61% (273/447) Updating files: 62% (278/447) Updating files: 63% (282/447) Updating files: 64% (287/447) Updating files: 65% (291/447) Updating files: 66% (296/447) Updating files: 66% (299/447) Updating files: 67% (300/447) Updating files: 68% (304/447) Updating files: 69% (309/447) Updating files: 70% (313/447) Updating files: 71% (318/447) Updating files: 72% (322/447) Updating files: 73% (327/447) Updating files: 74% (331/447) Updating files: 75% (336/447) Updating files: 76% (340/447) Updating files: 77% (345/447) Updating files: 78% (349/447) Updating files: 79% (354/447) Updating files: 80% (358/447) Updating files: 81% (363/447) Updating files: 82% (367/447) Updating files: 83% (372/447) Updating files: 84% (376/447) Updating files: 85% (380/447) Updating files: 86% (385/447) Updating files: 87% (389/447) Updating files: 88% (394/447) Updating files: 89% (398/447) Updating files: 90% (403/447) Updating files: 91% (407/447) Updating files: 92% (412/447) Updating files: 93% (416/447) Updating files: 94% (421/447) Updating files: 95% (425/447) Updating files: 96% (430/447) Updating files: 97% (434/447) Updating files: 98% (439/447) Updating files: 99% (443/447) Updating files: 100% (447/447) Updating files: 100% (447/447), done. 2022-06-27T12:19:35.151 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at 97dcc0539df 0.86 2022-06-27T12:19:35.157 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.86 2022-06-27T12:19:35.775 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:19:35.798 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:19:36.126 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:19:36.127 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 87 -eq 48 ']' 2022-06-27T12:19:36.127 INFO:tasks.workunit.client.0.smithi195.stdout:v0.87 2022-06-27T12:19:36.128 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.87 2022-06-27T12:19:36.129 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.87 2022-06-27T12:19:36.129 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.87 2022-06-27T12:19:36.495 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at c51c8f9d80f 0.87 2022-06-27T12:19:36.500 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.87 2022-06-27T12:19:36.575 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:19:36.576 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 88 -eq 48 ']' 2022-06-27T12:19:36.576 INFO:tasks.workunit.client.0.smithi195.stdout:v0.88 2022-06-27T12:19:36.577 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.88 2022-06-27T12:19:36.577 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.88 2022-06-27T12:19:36.578 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.88 2022-06-27T12:19:43.821 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 9% (32/335) Updating files: 10% (34/335) Updating files: 11% (37/335) Updating files: 12% (41/335) Updating files: 13% (44/335) Updating files: 14% (47/335) Updating files: 15% (51/335) Updating files: 15% (52/335) Updating files: 16% (54/335) Updating files: 17% (57/335) Updating files: 18% (61/335) Updating files: 19% (64/335) Updating files: 20% (67/335) Updating files: 21% (71/335) Updating files: 22% (74/335) Updating files: 23% (78/335) Updating files: 24% (81/335) Updating files: 25% (84/335) Updating files: 26% (88/335) Updating files: 27% (91/335) Updating files: 28% (94/335) Updating files: 29% (98/335) Updating files: 30% (101/335) Updating files: 31% (104/335) Updating files: 32% (108/335) Updating files: 33% (111/335) Updating files: 34% (114/335) Updating files: 35% (118/335) Updating files: 36% (121/335) Updating files: 37% (124/335) Updating files: 38% (128/335) Updating files: 39% (131/335) Updating files: 40% (134/335) Updating files: 41% (138/335) Updating files: 42% (141/335) Updating files: 43% (145/335) Updating files: 44% (148/335) Updating files: 45% (151/335) Updating files: 46% (155/335) Updating files: 47% (158/335) Updating files: 48% (161/335) Updating files: 49% (165/335) Updating files: 50% (168/335) Updating files: 51% (171/335) Updating files: 52% (175/335) Updating files: 53% (178/335) Updating files: 54% (181/335) Updating files: 55% (185/335) Updating files: 56% (188/335) Updating files: 57% (191/335) Updating files: 58% (195/335) Updating files: 59% (198/335) Updating files: 60% (201/335) Updating files: 61% (205/335) Updating files: 62% (208/335) Updating files: 63% (212/335) Updating files: 64% (215/335) Updating files: 65% (218/335) Updating files: 66% (222/335) Updating files: 67% (225/335) Updating files: 68% (228/335) Updating files: 69% (232/335) Updating files: 70% (235/335) Updating files: 71% (238/335) Updating files: 72% (242/335) Updating files: 73% (245/335) Updating files: 73% (246/335) Updating files: 74% (248/335) Updating files: 75% (252/335) Updating files: 76% (255/335) Updating files: 77% (258/335) Updating files: 78% (262/335) Updating files: 79% (265/335) Updating files: 80% (268/335) Updating files: 81% (272/335) Updating files: 82% (275/335) Updating files: 83% (279/335) Updating files: 84% (282/335) Updating files: 85% (285/335) Updating files: 86% (289/335) Updating files: 87% (292/335) Updating files: 88% (295/335) Updating files: 89% (299/335) Updating files: 90% (302/335) Updating files: 91% (305/335) Updating files: 92% (309/335) Updating files: 93% (312/335) Updating files: 94% (315/335) Updating files: 95% (319/335) Updating files: 96% (322/335) Updating files: 97% (325/335) Updating files: 98% (329/335) Updating files: 99% (332/335) Updating files: 100% (335/335) Updating files: 100% (335/335), done. 2022-06-27T12:19:43.849 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at 4be687bf448 0.88 2022-06-27T12:19:43.854 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.88 2022-06-27T12:19:43.949 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:19:43.949 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 89 -eq 48 ']' 2022-06-27T12:19:43.950 INFO:tasks.workunit.client.0.smithi195.stdout:v0.89 2022-06-27T12:19:43.951 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.89 2022-06-27T12:19:43.952 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.89 2022-06-27T12:19:43.952 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.89 2022-06-27T12:19:45.781 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 47% (93/197) Updating files: 48% (95/197) Updating files: 49% (97/197) Updating files: 50% (99/197) Updating files: 51% (101/197) Updating files: 52% (103/197) Updating files: 53% (105/197) Updating files: 54% (107/197) Updating files: 55% (109/197) Updating files: 56% (111/197) Updating files: 57% (113/197) Updating files: 58% (115/197) Updating files: 59% (117/197) Updating files: 60% (119/197) Updating files: 61% (121/197) Updating files: 62% (123/197) Updating files: 63% (125/197) Updating files: 64% (127/197) Updating files: 65% (129/197) Updating files: 66% (131/197) Updating files: 67% (132/197) Updating files: 68% (134/197) Updating files: 69% (136/197) Updating files: 70% (138/197) Updating files: 71% (140/197) Updating files: 72% (142/197) Updating files: 73% (144/197) Updating files: 74% (146/197) Updating files: 75% (148/197) Updating files: 76% (150/197) Updating files: 77% (152/197) Updating files: 78% (154/197) Updating files: 79% (156/197) Updating files: 80% (158/197) Updating files: 81% (160/197) Updating files: 82% (162/197) Updating files: 83% (164/197) Updating files: 84% (166/197) Updating files: 85% (168/197) Updating files: 86% (170/197) Updating files: 87% (172/197) Updating files: 88% (174/197) Updating files: 89% (176/197) Updating files: 90% (178/197) Updating files: 91% (180/197) Updating files: 92% (182/197) Updating files: 93% (184/197) Updating files: 94% (186/197) Updating files: 95% (188/197) Updating files: 96% (190/197) Updating files: 97% (192/197) Updating files: 98% (194/197) Updating files: 99% (196/197) Updating files: 100% (197/197) Updating files: 100% (197/197), done. 2022-06-27T12:19:45.830 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at 68fdc0f68e6 0.89 2022-06-27T12:19:45.835 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.89 2022-06-27T12:19:45.930 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:19:45.931 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 90 -eq 48 ']' 2022-06-27T12:19:45.932 INFO:tasks.workunit.client.0.smithi195.stdout:v0.90 2022-06-27T12:19:45.933 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.90 2022-06-27T12:19:45.934 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo v0.90 2022-06-27T12:19:45.934 INFO:tasks.workunit.client.0.smithi195.stderr:+ git reset --hard v0.90 2022-06-27T12:19:50.224 INFO:tasks.workunit.client.0.smithi195.stderr:Updating files: 29% (77/265) Updating files: 30% (80/265) Updating files: 31% (83/265) Updating files: 32% (85/265) Updating files: 33% (88/265) Updating files: 34% (91/265) Updating files: 35% (93/265) Updating files: 36% (96/265) Updating files: 37% (99/265) Updating files: 38% (101/265) Updating files: 39% (104/265) Updating files: 40% (106/265) Updating files: 41% (109/265) Updating files: 42% (112/265) Updating files: 43% (114/265) Updating files: 44% (117/265) Updating files: 45% (120/265) Updating files: 46% (122/265) Updating files: 47% (125/265) Updating files: 47% (127/265) Updating files: 48% (128/265) Updating files: 49% (130/265) Updating files: 50% (133/265) Updating files: 51% (136/265) Updating files: 52% (138/265) Updating files: 53% (141/265) Updating files: 54% (144/265) Updating files: 55% (146/265) Updating files: 56% (149/265) Updating files: 57% (152/265) Updating files: 58% (154/265) Updating files: 59% (157/265) Updating files: 60% (159/265) Updating files: 61% (162/265) Updating files: 62% (165/265) Updating files: 63% (167/265) Updating files: 64% (170/265) Updating files: 65% (173/265) Updating files: 66% (175/265) Updating files: 67% (178/265) Updating files: 68% (181/265) Updating files: 69% (183/265) Updating files: 70% (186/265) Updating files: 71% (189/265) Updating files: 72% (191/265) Updating files: 73% (194/265) Updating files: 74% (197/265) Updating files: 75% (199/265) Updating files: 76% (202/265) Updating files: 77% (205/265) Updating files: 78% (207/265) Updating files: 79% (210/265) Updating files: 80% (212/265) Updating files: 81% (215/265) Updating files: 82% (218/265) Updating files: 83% (220/265) Updating files: 84% (223/265) Updating files: 85% (226/265) Updating files: 86% (228/265) Updating files: 87% (231/265) Updating files: 88% (234/265) Updating files: 89% (236/265) Updating files: 90% (239/265) Updating files: 91% (242/265) Updating files: 92% (244/265) Updating files: 93% (247/265) Updating files: 94% (250/265) Updating files: 95% (252/265) Updating files: 96% (255/265) Updating files: 97% (258/265) Updating files: 98% (260/265) Updating files: 99% (263/265) Updating files: 100% (265/265) Updating files: 100% (265/265), done. 2022-06-27T12:19:50.263 INFO:tasks.workunit.client.0.smithi195.stdout:HEAD is now at 08bd1e1eee3 0.90 2022-06-27T12:19:50.268 INFO:tasks.workunit.client.0.smithi195.stderr:+ mkdir .snap/v0.90 2022-06-27T12:19:51.108 INFO:tasks.workunit.client.0.smithi195.stderr:+ for v in $versions 2022-06-27T12:19:51.109 INFO:tasks.workunit.client.0.smithi195.stderr:+ '[' 1 -eq 48 ']' 2022-06-27T12:19:51.110 INFO:tasks.workunit.client.0.smithi195.stderr:+ ver=v0.1 2022-06-27T12:19:51.110 INFO:tasks.workunit.client.0.smithi195.stderr:+ echo checking v0.1 2022-06-27T12:19:51.110 INFO:tasks.workunit.client.0.smithi195.stderr:+ cd .snap/v0.1 2022-06-27T12:19:51.111 INFO:tasks.workunit.client.0.smithi195.stdout:checking v0.1 2022-06-27T12:19:51.118 INFO:tasks.workunit.client.0.smithi195.stderr:+ git diff --exit-code 2022-06-27T12:19:54.103 INFO:tasks.workunit.client.0.smithi195.stdout:diff --git a/src/msg/Messenger.cc b/src/msg/Messenger.cc 2022-06-27T12:19:54.103 INFO:tasks.workunit.client.0.smithi195.stdout:index 5af83462b29..e69de29bb2d 100644 2022-06-27T12:19:54.104 INFO:tasks.workunit.client.0.smithi195.stdout:--- a/src/msg/Messenger.cc 2022-06-27T12:19:54.104 INFO:tasks.workunit.client.0.smithi195.stdout:+++ b/src/msg/Messenger.cc 2022-06-27T12:19:54.105 INFO:tasks.workunit.client.0.smithi195.stdout:@@ -1,39 +0,0 @@ 2022-06-27T12:19:54.105 INFO:tasks.workunit.client.0.smithi195.stdout:-// -*- mode:C++; tab-width:8; c-basic-offset:2; indent-tabs-mode:t -*- 2022-06-27T12:19:54.107 INFO:tasks.workunit.client.0.smithi195.stdout:-// vim: ts=8 sw=2 smarttab 2022-06-27T12:19:54.108 INFO:tasks.workunit.client.0.smithi195.stdout:-/* 2022-06-27T12:19:54.108 INFO:tasks.workunit.client.0.smithi195.stdout:- * Ceph - scalable distributed file system 2022-06-27T12:19:54.109 INFO:tasks.workunit.client.0.smithi195.stdout:- * 2022-06-27T12:19:54.109 INFO:tasks.workunit.client.0.smithi195.stdout:- * Copyright (C) 2004-2006 Sage Weil 2022-06-27T12:19:54.109 INFO:tasks.workunit.client.0.smithi195.stdout:- * 2022-06-27T12:19:54.110 INFO:tasks.workunit.client.0.smithi195.stdout:- * This is free software; you can redistribute it and/or 2022-06-27T12:19:54.110 INFO:tasks.workunit.client.0.smithi195.stdout:- * modify it under the terms of the GNU Lesser General Public 2022-06-27T12:19:54.111 INFO:tasks.workunit.client.0.smithi195.stdout:- * License version 2.1, as published by the Free Software 2022-06-27T12:19:54.112 INFO:tasks.workunit.client.0.smithi195.stdout:- * Foundation. See file COPYING. 2022-06-27T12:19:54.112 INFO:tasks.workunit.client.0.smithi195.stdout:- * 2022-06-27T12:19:54.113 INFO:tasks.workunit.client.0.smithi195.stdout:- */ 2022-06-27T12:19:54.113 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.114 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.114 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.115 INFO:tasks.workunit.client.0.smithi195.stdout:-#include 2022-06-27T12:19:54.115 INFO:tasks.workunit.client.0.smithi195.stdout:-#include "include/types.h" 2022-06-27T12:19:54.116 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.116 INFO:tasks.workunit.client.0.smithi195.stdout:-#include "Message.h" 2022-06-27T12:19:54.116 INFO:tasks.workunit.client.0.smithi195.stdout:-#include "Messenger.h" 2022-06-27T12:19:54.117 INFO:tasks.workunit.client.0.smithi195.stdout:-#include "messages/MGenericMessage.h" 2022-06-27T12:19:54.117 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.118 INFO:tasks.workunit.client.0.smithi195.stdout:-#include 2022-06-27T12:19:54.118 INFO:tasks.workunit.client.0.smithi195.stdout:-#include 2022-06-27T12:19:54.119 INFO:tasks.workunit.client.0.smithi195.stdout:-using namespace std; 2022-06-27T12:19:54.119 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.120 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.120 INFO:tasks.workunit.client.0.smithi195.stdout:-// --------- 2022-06-27T12:19:54.121 INFO:tasks.workunit.client.0.smithi195.stdout:-// incoming messages 2022-06-27T12:19:54.121 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.121 INFO:tasks.workunit.client.0.smithi195.stdout:-void Messenger::dispatch(Message *m) 2022-06-27T12:19:54.122 INFO:tasks.workunit.client.0.smithi195.stdout:-{ 2022-06-27T12:19:54.122 INFO:tasks.workunit.client.0.smithi195.stdout:- assert(dispatcher); 2022-06-27T12:19:54.123 INFO:tasks.workunit.client.0.smithi195.stdout:- dispatcher->dispatch(m); 2022-06-27T12:19:54.123 INFO:tasks.workunit.client.0.smithi195.stdout:-} 2022-06-27T12:19:54.124 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.125 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.125 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.126 INFO:tasks.workunit.client.0.smithi195.stdout:diff --git a/src/newsyn.cc b/src/newsyn.cc 2022-06-27T12:19:54.127 INFO:tasks.workunit.client.0.smithi195.stdout:index e67ff5eeb89..e69de29bb2d 100644 2022-06-27T12:19:54.127 INFO:tasks.workunit.client.0.smithi195.stdout:--- a/src/newsyn.cc 2022-06-27T12:19:54.128 INFO:tasks.workunit.client.0.smithi195.stdout:+++ b/src/newsyn.cc 2022-06-27T12:19:54.128 INFO:tasks.workunit.client.0.smithi195.stdout:@@ -1,477 +0,0 @@ 2022-06-27T12:19:54.129 INFO:tasks.workunit.client.0.smithi195.stdout:-// -*- mode:C++; tab-width:8; c-basic-offset:2; indent-tabs-mode:t -*- 2022-06-27T12:19:54.129 INFO:tasks.workunit.client.0.smithi195.stdout:-// vim: ts=8 sw=2 smarttab 2022-06-27T12:19:54.129 INFO:tasks.workunit.client.0.smithi195.stdout:-/* 2022-06-27T12:19:54.130 INFO:tasks.workunit.client.0.smithi195.stdout:- * Ceph - scalable distributed file system 2022-06-27T12:19:54.130 INFO:tasks.workunit.client.0.smithi195.stdout:- * 2022-06-27T12:19:54.131 INFO:tasks.workunit.client.0.smithi195.stdout:- * Copyright (C) 2004-2006 Sage Weil 2022-06-27T12:19:54.131 INFO:tasks.workunit.client.0.smithi195.stdout:- * 2022-06-27T12:19:54.132 INFO:tasks.workunit.client.0.smithi195.stdout:- * This is free software; you can redistribute it and/or 2022-06-27T12:19:54.132 INFO:tasks.workunit.client.0.smithi195.stdout:- * modify it under the terms of the GNU Lesser General Public 2022-06-27T12:19:54.133 INFO:tasks.workunit.client.0.smithi195.stdout:- * License version 2.1, as published by the Free Software 2022-06-27T12:19:54.133 INFO:tasks.workunit.client.0.smithi195.stdout:- * Foundation. See file COPYING. 2022-06-27T12:19:54.134 INFO:tasks.workunit.client.0.smithi195.stdout:- * 2022-06-27T12:19:54.134 INFO:tasks.workunit.client.0.smithi195.stdout:- */ 2022-06-27T12:19:54.135 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.135 INFO:tasks.workunit.client.0.smithi195.stdout:-#define intabs(x) ((x) >= 0 ? (x):(-(x))) 2022-06-27T12:19:54.136 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.136 INFO:tasks.workunit.client.0.smithi195.stdout:-#include 2022-06-27T12:19:54.136 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.137 INFO:tasks.workunit.client.0.smithi195.stdout:-#include 2022-06-27T12:19:54.137 INFO:tasks.workunit.client.0.smithi195.stdout:-#include 2022-06-27T12:19:54.138 INFO:tasks.workunit.client.0.smithi195.stdout:-#include 2022-06-27T12:19:54.138 INFO:tasks.workunit.client.0.smithi195.stdout:-using namespace std; 2022-06-27T12:19:54.139 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.139 INFO:tasks.workunit.client.0.smithi195.stdout:-#include 2022-06-27T12:19:54.140 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.140 INFO:tasks.workunit.client.0.smithi195.stdout:-#include "config.h" 2022-06-27T12:19:54.141 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.142 INFO:tasks.workunit.client.0.smithi195.stdout:-#include "mds/MDS.h" 2022-06-27T12:19:54.142 INFO:tasks.workunit.client.0.smithi195.stdout:-#include "osd/OSD.h" 2022-06-27T12:19:54.143 INFO:tasks.workunit.client.0.smithi195.stdout:-#include "mon/Monitor.h" 2022-06-27T12:19:54.143 INFO:tasks.workunit.client.0.smithi195.stdout:-#include "client/Client.h" 2022-06-27T12:19:54.146 INFO:tasks.workunit.client.0.smithi195.stdout:-#include "client/SyntheticClient.h" 2022-06-27T12:19:54.146 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.147 INFO:tasks.workunit.client.0.smithi195.stdout:-#include "msg/SimpleMessenger.h" 2022-06-27T12:19:54.147 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.148 INFO:tasks.workunit.client.0.smithi195.stdout:-#include "common/Timer.h" 2022-06-27T12:19:54.148 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.149 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.149 INFO:tasks.workunit.client.0.smithi195.stdout:-class C_Test : public Context { 2022-06-27T12:19:54.149 INFO:tasks.workunit.client.0.smithi195.stdout:-public: 2022-06-27T12:19:54.149 INFO:tasks.workunit.client.0.smithi195.stdout:- void finish(int r) { 2022-06-27T12:19:54.150 INFO:tasks.workunit.client.0.smithi195.stdout:- cout << "C_Test->finish(" << r << ")" << std::endl; 2022-06-27T12:19:54.150 INFO:tasks.workunit.client.0.smithi195.stdout:- } 2022-06-27T12:19:54.150 INFO:tasks.workunit.client.0.smithi195.stdout:-}; 2022-06-27T12:19:54.150 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.151 INFO:tasks.workunit.client.0.smithi195.stdout:-extern std::map g_fake_kill_after; 2022-06-27T12:19:54.151 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.152 INFO:tasks.workunit.client.0.smithi195.stdout:-bool use_existing_monmap = false; 2022-06-27T12:19:54.152 INFO:tasks.workunit.client.0.smithi195.stdout:-const char *monmap_fn = ".ceph_monmap"; 2022-06-27T12:19:54.153 INFO:tasks.workunit.client.0.smithi195.stdout:-/* 2022-06-27T12:19:54.153 INFO:tasks.workunit.client.0.smithi195.stdout:- * start up NewMessenger via MPI. 2022-06-27T12:19:54.154 INFO:tasks.workunit.client.0.smithi195.stdout:- */ 2022-06-27T12:19:54.154 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.154 INFO:tasks.workunit.client.0.smithi195.stdout:-pair mpi_bootstrap_new(int& argc, const char**& argv, MonMap *monmap) 2022-06-27T12:19:54.155 INFO:tasks.workunit.client.0.smithi195.stdout:-{ 2022-06-27T12:19:54.155 INFO:tasks.workunit.client.0.smithi195.stdout:- MPI_Init(&argc, (char***)&argv); 2022-06-27T12:19:54.156 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.156 INFO:tasks.workunit.client.0.smithi195.stdout:- int mpi_world; 2022-06-27T12:19:54.157 INFO:tasks.workunit.client.0.smithi195.stdout:- int mpi_rank; 2022-06-27T12:19:54.157 INFO:tasks.workunit.client.0.smithi195.stdout:- MPI_Comm_size(MPI_COMM_WORLD, &mpi_world); 2022-06-27T12:19:54.158 INFO:tasks.workunit.client.0.smithi195.stdout:- MPI_Comm_rank(MPI_COMM_WORLD, &mpi_rank); 2022-06-27T12:19:54.158 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.159 INFO:tasks.workunit.client.0.smithi195.stdout:- if (use_existing_monmap && mpi_rank < g_conf.num_mon) { 2022-06-27T12:19:54.159 INFO:tasks.workunit.client.0.smithi195.stdout:- int r = monmap->read(monmap_fn); 2022-06-27T12:19:54.159 INFO:tasks.workunit.client.0.smithi195.stdout:- assert(r >= 0); 2022-06-27T12:19:54.160 INFO:tasks.workunit.client.0.smithi195.stdout:- g_my_addr = monmap->get_inst(mpi_rank).addr; 2022-06-27T12:19:54.160 INFO:tasks.workunit.client.0.smithi195.stdout:- cout << "i am monitor, will bind to " << g_my_addr 2022-06-27T12:19:54.161 INFO:tasks.workunit.client.0.smithi195.stdout:- << " from existing " << monmap_fn << std::endl; 2022-06-27T12:19:54.161 INFO:tasks.workunit.client.0.smithi195.stdout:- } 2022-06-27T12:19:54.162 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.162 INFO:tasks.workunit.client.0.smithi195.stdout:- // first, synchronize clocks. 2022-06-27T12:19:54.163 INFO:tasks.workunit.client.0.smithi195.stdout:- if (g_conf.clock_tare) { 2022-06-27T12:19:54.163 INFO:tasks.workunit.client.0.smithi195.stdout:- if (1) { 2022-06-27T12:19:54.164 INFO:tasks.workunit.client.0.smithi195.stdout:- // use an MPI barrier. probably not terribly precise. 2022-06-27T12:19:54.164 INFO:tasks.workunit.client.0.smithi195.stdout:- MPI_Barrier(MPI_COMM_WORLD); 2022-06-27T12:19:54.165 INFO:tasks.workunit.client.0.smithi195.stdout:- g_clock.tare(); 2022-06-27T12:19:54.165 INFO:tasks.workunit.client.0.smithi195.stdout:- } else { 2022-06-27T12:19:54.166 INFO:tasks.workunit.client.0.smithi195.stdout:- // use wall clock; assume NTP has all nodes synchronized already. 2022-06-27T12:19:54.166 INFO:tasks.workunit.client.0.smithi195.stdout:- // FIXME someday: this hangs for some reason. whatever. 2022-06-27T12:19:54.166 INFO:tasks.workunit.client.0.smithi195.stdout:- utime_t z = g_clock.now(); 2022-06-27T12:19:54.167 INFO:tasks.workunit.client.0.smithi195.stdout:- MPI_Bcast( &z, sizeof(z), MPI_CHAR, 2022-06-27T12:19:54.167 INFO:tasks.workunit.client.0.smithi195.stdout:- 0, MPI_COMM_WORLD); 2022-06-27T12:19:54.168 INFO:tasks.workunit.client.0.smithi195.stdout:- cout << "z is " << z << std::endl; 2022-06-27T12:19:54.168 INFO:tasks.workunit.client.0.smithi195.stdout:- g_clock.tare(z); 2022-06-27T12:19:54.169 INFO:tasks.workunit.client.0.smithi195.stdout:- } 2022-06-27T12:19:54.169 INFO:tasks.workunit.client.0.smithi195.stdout:- } 2022-06-27T12:19:54.170 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.170 INFO:tasks.workunit.client.0.smithi195.stdout:- // start up all monitors at known addresses. 2022-06-27T12:19:54.170 INFO:tasks.workunit.client.0.smithi195.stdout:- entity_inst_t moninst[mpi_world]; // only care about first g_conf.num_mon of these. 2022-06-27T12:19:54.171 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.171 INFO:tasks.workunit.client.0.smithi195.stdout:- rank.bind(); // bind and listen 2022-06-27T12:19:54.172 INFO:tasks.workunit.client.0.smithi195.stdout:- rank.start(); 2022-06-27T12:19:54.172 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.173 INFO:tasks.workunit.client.0.smithi195.stdout:- if (mpi_rank < g_conf.num_mon) { 2022-06-27T12:19:54.173 INFO:tasks.workunit.client.0.smithi195.stdout:- moninst[mpi_rank].addr = rank.rank_addr; 2022-06-27T12:19:54.174 INFO:tasks.workunit.client.0.smithi195.stdout:- moninst[mpi_rank].name = entity_name_t(entity_name_t::TYPE_MON, mpi_rank); 2022-06-27T12:19:54.174 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.175 INFO:tasks.workunit.client.0.smithi195.stdout:- //cerr << mpi_rank << " at " << rank.get_listen_addr() << std::endl; 2022-06-27T12:19:54.175 INFO:tasks.workunit.client.0.smithi195.stdout:- } 2022-06-27T12:19:54.176 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.176 INFO:tasks.workunit.client.0.smithi195.stdout:- MPI_Gather( &moninst[mpi_rank], sizeof(entity_inst_t), MPI_CHAR, 2022-06-27T12:19:54.177 INFO:tasks.workunit.client.0.smithi195.stdout:- moninst, sizeof(entity_inst_t), MPI_CHAR, 2022-06-27T12:19:54.177 INFO:tasks.workunit.client.0.smithi195.stdout:- 0, MPI_COMM_WORLD); 2022-06-27T12:19:54.178 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.178 INFO:tasks.workunit.client.0.smithi195.stdout:- if (mpi_rank == 0) { 2022-06-27T12:19:54.179 INFO:tasks.workunit.client.0.smithi195.stdout:- for (int i=0; imon_inst[i] = moninst[i]; 2022-06-27T12:19:54.180 INFO:tasks.workunit.client.0.smithi195.stdout:- } 2022-06-27T12:19:54.180 INFO:tasks.workunit.client.0.smithi195.stdout:- } 2022-06-27T12:19:54.181 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.181 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.182 INFO:tasks.workunit.client.0.smithi195.stdout:- // distribute monmap 2022-06-27T12:19:54.182 INFO:tasks.workunit.client.0.smithi195.stdout:- bufferlist bl; 2022-06-27T12:19:54.183 INFO:tasks.workunit.client.0.smithi195.stdout:- if (mpi_rank == 0) { 2022-06-27T12:19:54.183 INFO:tasks.workunit.client.0.smithi195.stdout:- monmap->encode(bl); 2022-06-27T12:19:54.184 INFO:tasks.workunit.client.0.smithi195.stdout:- monmap->write(monmap_fn); 2022-06-27T12:19:54.184 INFO:tasks.workunit.client.0.smithi195.stdout:- } else { 2022-06-27T12:19:54.184 INFO:tasks.workunit.client.0.smithi195.stdout:- int l = g_conf.num_mon * 1000; // nice'n big. 2022-06-27T12:19:54.185 INFO:tasks.workunit.client.0.smithi195.stdout:- bufferptr bp(l); 2022-06-27T12:19:54.185 INFO:tasks.workunit.client.0.smithi195.stdout:- bl.append(bp); 2022-06-27T12:19:54.186 INFO:tasks.workunit.client.0.smithi195.stdout:- } 2022-06-27T12:19:54.186 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.187 INFO:tasks.workunit.client.0.smithi195.stdout:- MPI_Bcast(bl.c_str(), bl.length(), MPI_CHAR, 2022-06-27T12:19:54.187 INFO:tasks.workunit.client.0.smithi195.stdout:- 0, MPI_COMM_WORLD); 2022-06-27T12:19:54.188 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.188 INFO:tasks.workunit.client.0.smithi195.stdout:- if (mpi_rank > 0) { 2022-06-27T12:19:54.189 INFO:tasks.workunit.client.0.smithi195.stdout:- monmap->decode(bl); 2022-06-27T12:19:54.189 INFO:tasks.workunit.client.0.smithi195.stdout:- } 2022-06-27T12:19:54.190 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.190 INFO:tasks.workunit.client.0.smithi195.stdout:- // wait for everyone! 2022-06-27T12:19:54.191 INFO:tasks.workunit.client.0.smithi195.stdout:- MPI_Barrier(MPI_COMM_WORLD); 2022-06-27T12:19:54.191 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.191 INFO:tasks.workunit.client.0.smithi195.stdout:- return pair(mpi_rank, mpi_world); 2022-06-27T12:19:54.192 INFO:tasks.workunit.client.0.smithi195.stdout:-} 2022-06-27T12:19:54.192 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.193 INFO:tasks.workunit.client.0.smithi195.stdout:-utime_t tick_start; 2022-06-27T12:19:54.193 INFO:tasks.workunit.client.0.smithi195.stdout:-int tick_count = 0; 2022-06-27T12:19:54.194 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.194 INFO:tasks.workunit.client.0.smithi195.stdout:-class C_Tick : public Context { 2022-06-27T12:19:54.195 INFO:tasks.workunit.client.0.smithi195.stdout:-public: 2022-06-27T12:19:54.195 INFO:tasks.workunit.client.0.smithi195.stdout:- void finish(int) { 2022-06-27T12:19:54.196 INFO:tasks.workunit.client.0.smithi195.stdout:- utime_t now = g_clock.now() - tick_start; 2022-06-27T12:19:54.196 INFO:tasks.workunit.client.0.smithi195.stdout:- cout << "tick +" << g_conf.tick << " -> " << now << " (" << tick_count << ")" << std::endl; 2022-06-27T12:19:54.197 INFO:tasks.workunit.client.0.smithi195.stdout:- tick_count += g_conf.tick; 2022-06-27T12:19:54.197 INFO:tasks.workunit.client.0.smithi195.stdout:- utime_t next = tick_start; 2022-06-27T12:19:54.197 INFO:tasks.workunit.client.0.smithi195.stdout:- next.sec_ref() += tick_count; 2022-06-27T12:19:54.198 INFO:tasks.workunit.client.0.smithi195.stdout:- g_timer.add_event_at(next, new C_Tick); 2022-06-27T12:19:54.198 INFO:tasks.workunit.client.0.smithi195.stdout:- } 2022-06-27T12:19:54.199 INFO:tasks.workunit.client.0.smithi195.stdout:-}; 2022-06-27T12:19:54.200 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.200 INFO:tasks.workunit.client.0.smithi195.stdout:-class C_Die : public Context { 2022-06-27T12:19:54.201 INFO:tasks.workunit.client.0.smithi195.stdout:-public: 2022-06-27T12:19:54.201 INFO:tasks.workunit.client.0.smithi195.stdout:- void finish(int) { 2022-06-27T12:19:54.201 INFO:tasks.workunit.client.0.smithi195.stdout:- cerr << "die" << std::endl; 2022-06-27T12:19:54.202 INFO:tasks.workunit.client.0.smithi195.stdout:- _exit(1); 2022-06-27T12:19:54.202 INFO:tasks.workunit.client.0.smithi195.stdout:- } 2022-06-27T12:19:54.203 INFO:tasks.workunit.client.0.smithi195.stdout:-}; 2022-06-27T12:19:54.203 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.204 INFO:tasks.workunit.client.0.smithi195.stdout:-class C_Debug : public Context { 2022-06-27T12:19:54.204 INFO:tasks.workunit.client.0.smithi195.stdout:- public: 2022-06-27T12:19:54.205 INFO:tasks.workunit.client.0.smithi195.stdout:- void finish(int) { 2022-06-27T12:19:54.205 INFO:tasks.workunit.client.0.smithi195.stdout:- int size = (long)&g_conf.debug_after - (long)&g_conf.debug; 2022-06-27T12:19:54.206 INFO:tasks.workunit.client.0.smithi195.stdout:- memcpy((char*)&g_conf.debug, (char*)&g_debug_after_conf.debug, size); 2022-06-27T12:19:54.206 INFO:tasks.workunit.client.0.smithi195.stdout:- cout << "debug_after flipping debug settings" << std::endl; 2022-06-27T12:19:54.207 INFO:tasks.workunit.client.0.smithi195.stdout:- //g_conf.debug_ms = 1; 2022-06-27T12:19:54.207 INFO:tasks.workunit.client.0.smithi195.stdout:- } 2022-06-27T12:19:54.207 INFO:tasks.workunit.client.0.smithi195.stdout:-}; 2022-06-27T12:19:54.208 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.208 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.209 INFO:tasks.workunit.client.0.smithi195.stdout:-int main(int argc, const char **argv) 2022-06-27T12:19:54.209 INFO:tasks.workunit.client.0.smithi195.stdout:-{ 2022-06-27T12:19:54.210 INFO:tasks.workunit.client.0.smithi195.stdout:- vector args; 2022-06-27T12:19:54.210 INFO:tasks.workunit.client.0.smithi195.stdout:- argv_to_vec(argc, argv, args); 2022-06-27T12:19:54.210 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.211 INFO:tasks.workunit.client.0.smithi195.stdout:- map kill_osd_after; 2022-06-27T12:19:54.211 INFO:tasks.workunit.client.0.smithi195.stdout:- int share_single_client = 0; 2022-06-27T12:19:54.212 INFO:tasks.workunit.client.0.smithi195.stdout:- if (1) { 2022-06-27T12:19:54.212 INFO:tasks.workunit.client.0.smithi195.stdout:- vector nargs; 2022-06-27T12:19:54.213 INFO:tasks.workunit.client.0.smithi195.stdout:- for (unsigned i=0; i 0 ? g_conf.num_mon:0; 2022-06-27T12:19:54.224 INFO:tasks.workunit.client.0.smithi195.stdout:- int start_mds = g_conf.num_mds > 0 ? g_conf.num_mds:0; 2022-06-27T12:19:54.224 INFO:tasks.workunit.client.0.smithi195.stdout:- int start_osd = g_conf.num_osd > 0 ? g_conf.num_osd:0; 2022-06-27T12:19:54.225 INFO:tasks.workunit.client.0.smithi195.stdout:- int start_client = g_conf.num_client > 0 ? g_conf.num_client:0; 2022-06-27T12:19:54.225 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.226 INFO:tasks.workunit.client.0.smithi195.stdout:- //g_conf.num_mon = intabs(g_conf.num_mon); 2022-06-27T12:19:54.226 INFO:tasks.workunit.client.0.smithi195.stdout:- g_conf.num_mds = intabs(g_conf.num_mds); 2022-06-27T12:19:54.227 INFO:tasks.workunit.client.0.smithi195.stdout:- g_conf.num_client = intabs(g_conf.num_client); 2022-06-27T12:19:54.227 INFO:tasks.workunit.client.0.smithi195.stdout:- g_conf.num_osd = intabs(g_conf.num_osd); 2022-06-27T12:19:54.228 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.228 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.229 INFO:tasks.workunit.client.0.smithi195.stdout:- if (g_conf.kill_after) 2022-06-27T12:19:54.229 INFO:tasks.workunit.client.0.smithi195.stdout:- g_timer.add_event_after(g_conf.kill_after, new C_Die); 2022-06-27T12:19:54.230 INFO:tasks.workunit.client.0.smithi195.stdout:- if (g_conf.debug_after) 2022-06-27T12:19:54.230 INFO:tasks.workunit.client.0.smithi195.stdout:- g_timer.add_event_after(g_conf.debug_after, new C_Debug); 2022-06-27T12:19:54.230 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.231 INFO:tasks.workunit.client.0.smithi195.stdout:- if (g_conf.tick) { 2022-06-27T12:19:54.231 INFO:tasks.workunit.client.0.smithi195.stdout:- tick_start = g_clock.now(); 2022-06-27T12:19:54.232 INFO:tasks.workunit.client.0.smithi195.stdout:- g_timer.add_event_after(g_conf.tick, new C_Tick); 2022-06-27T12:19:54.232 INFO:tasks.workunit.client.0.smithi195.stdout:- } 2022-06-27T12:19:54.233 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.233 INFO:tasks.workunit.client.0.smithi195.stdout:- vector nargs; 2022-06-27T12:19:54.234 INFO:tasks.workunit.client.0.smithi195.stdout:- for (unsigned i=0; i mpiwho = mpi_bootstrap_new(argc, argv, monmap); 2022-06-27T12:19:54.241 INFO:tasks.workunit.client.0.smithi195.stdout:- int mpirank = mpiwho.first; 2022-06-27T12:19:54.241 INFO:tasks.workunit.client.0.smithi195.stdout:- int world = mpiwho.second; 2022-06-27T12:19:54.242 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.242 INFO:tasks.workunit.client.0.smithi195.stdout:- int need = 0; 2022-06-27T12:19:54.242 INFO:tasks.workunit.client.0.smithi195.stdout:- if (g_conf.ms_skip_rank0) need++; 2022-06-27T12:19:54.243 INFO:tasks.workunit.client.0.smithi195.stdout:- need += start_mds; 2022-06-27T12:19:54.243 INFO:tasks.workunit.client.0.smithi195.stdout:- if (g_conf.ms_stripe_osds) 2022-06-27T12:19:54.244 INFO:tasks.workunit.client.0.smithi195.stdout:- need++; 2022-06-27T12:19:54.244 INFO:tasks.workunit.client.0.smithi195.stdout:- else 2022-06-27T12:19:54.244 INFO:tasks.workunit.client.0.smithi195.stdout:- need += start_osd; 2022-06-27T12:19:54.245 INFO:tasks.workunit.client.0.smithi195.stdout:- if (start_client) { 2022-06-27T12:19:54.245 INFO:tasks.workunit.client.0.smithi195.stdout:- if (!g_conf.ms_overlay_clients) 2022-06-27T12:19:54.245 INFO:tasks.workunit.client.0.smithi195.stdout:- need += 1; 2022-06-27T12:19:54.246 INFO:tasks.workunit.client.0.smithi195.stdout:- } 2022-06-27T12:19:54.246 INFO:tasks.workunit.client.0.smithi195.stdout:- assert(need <= world); 2022-06-27T12:19:54.247 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.247 INFO:tasks.workunit.client.0.smithi195.stdout:- if (mpirank == 0) 2022-06-27T12:19:54.248 INFO:tasks.workunit.client.0.smithi195.stdout:- cerr << "nummds " << start_mds << " numosd " << start_osd << " numclient " << start_client << " .. need " << need << ", have " << world << std::endl; 2022-06-27T12:19:54.248 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.248 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.249 INFO:tasks.workunit.client.0.smithi195.stdout:- char hostname[100]; 2022-06-27T12:19:54.249 INFO:tasks.workunit.client.0.smithi195.stdout:- gethostname(hostname,100); 2022-06-27T12:19:54.250 INFO:tasks.workunit.client.0.smithi195.stdout:- int pid = getpid(); 2022-06-27T12:19:54.250 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.251 INFO:tasks.workunit.client.0.smithi195.stdout:- int started = 0; 2022-06-27T12:19:54.251 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.252 INFO:tasks.workunit.client.0.smithi195.stdout:- //if (mpirank == 0) g_conf.debug = 20; 2022-06-27T12:19:54.252 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.253 INFO:tasks.workunit.client.0.smithi195.stdout:- // courtesy symlinks 2022-06-27T12:19:54.253 INFO:tasks.workunit.client.0.smithi195.stdout:- char ffrom[100]; 2022-06-27T12:19:54.253 INFO:tasks.workunit.client.0.smithi195.stdout:- char fto[100]; 2022-06-27T12:19:54.254 INFO:tasks.workunit.client.0.smithi195.stdout:- sprintf(fto, "%s.%d", hostname, pid); 2022-06-27T12:19:54.254 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.255 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.255 INFO:tasks.workunit.client.0.smithi195.stdout:- // create mon 2022-06-27T12:19:54.255 INFO:tasks.workunit.client.0.smithi195.stdout:- if (mpirank < g_conf.num_mon) { 2022-06-27T12:19:54.256 INFO:tasks.workunit.client.0.smithi195.stdout:- Monitor *mon = new Monitor(mpirank, rank.register_entity(entity_name_t(entity_name_t::TYPE_MON, mpirank)), monmap); 2022-06-27T12:19:54.256 INFO:tasks.workunit.client.0.smithi195.stdout:- mon->init(); 2022-06-27T12:19:54.256 INFO:tasks.workunit.client.0.smithi195.stdout:- if (g_conf.dout_dir) { 2022-06-27T12:19:54.257 INFO:tasks.workunit.client.0.smithi195.stdout:- sprintf(ffrom, "%s/mon%d", g_conf.dout_dir, mpirank); 2022-06-27T12:19:54.257 INFO:tasks.workunit.client.0.smithi195.stdout:- ::unlink(ffrom); 2022-06-27T12:19:54.257 INFO:tasks.workunit.client.0.smithi195.stdout:- ::symlink(fto, ffrom); 2022-06-27T12:19:54.258 INFO:tasks.workunit.client.0.smithi195.stdout:- } 2022-06-27T12:19:54.258 INFO:tasks.workunit.client.0.smithi195.stdout:- } 2022-06-27T12:19:54.259 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.259 INFO:tasks.workunit.client.0.smithi195.stdout:- // wait for monitors to start. 2022-06-27T12:19:54.260 INFO:tasks.workunit.client.0.smithi195.stdout:- MPI_Barrier(MPI_COMM_WORLD); 2022-06-27T12:19:54.260 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.260 INFO:tasks.workunit.client.0.smithi195.stdout:- // okay, home free! 2022-06-27T12:19:54.261 INFO:tasks.workunit.client.0.smithi195.stdout:- MPI_Finalize(); 2022-06-27T12:19:54.261 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.262 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.262 INFO:tasks.workunit.client.0.smithi195.stdout:- // create mds 2022-06-27T12:19:54.263 INFO:tasks.workunit.client.0.smithi195.stdout:- map mds; 2022-06-27T12:19:54.263 INFO:tasks.workunit.client.0.smithi195.stdout:- map mdsosd; 2022-06-27T12:19:54.264 INFO:tasks.workunit.client.0.smithi195.stdout:- for (int i=0; iget_myaddr() << " " << hostname << "." << pid << std::endl; 2022-06-27T12:19:54.265 INFO:tasks.workunit.client.0.smithi195.stdout:- if (g_conf.dout_dir) { 2022-06-27T12:19:54.266 INFO:tasks.workunit.client.0.smithi195.stdout:- sprintf(ffrom, "%s/mds%d", g_conf.dout_dir, i); 2022-06-27T12:19:54.266 INFO:tasks.workunit.client.0.smithi195.stdout:- ::unlink(ffrom); 2022-06-27T12:19:54.267 INFO:tasks.workunit.client.0.smithi195.stdout:- ::symlink(fto, ffrom); 2022-06-27T12:19:54.267 INFO:tasks.workunit.client.0.smithi195.stdout:- } 2022-06-27T12:19:54.268 INFO:tasks.workunit.client.0.smithi195.stdout:- mds[i] = new MDS(i, m, monmap); 2022-06-27T12:19:54.268 INFO:tasks.workunit.client.0.smithi195.stdout:- mds[i]->init(); 2022-06-27T12:19:54.268 INFO:tasks.workunit.client.0.smithi195.stdout:- started++; 2022-06-27T12:19:54.269 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.269 INFO:tasks.workunit.client.0.smithi195.stdout:- if (g_conf.mds_local_osd) { 2022-06-27T12:19:54.270 INFO:tasks.workunit.client.0.smithi195.stdout:- int n = i+g_conf.num_osd; 2022-06-27T12:19:54.270 INFO:tasks.workunit.client.0.smithi195.stdout:- mdsosd[i] = new OSD(n, rank.register_entity(entity_name_t(entity_name_t::TYPE_OSD, n)), monmap); 2022-06-27T12:19:54.271 INFO:tasks.workunit.client.0.smithi195.stdout:- mdsosd[i]->init(); 2022-06-27T12:19:54.271 INFO:tasks.workunit.client.0.smithi195.stdout:- } 2022-06-27T12:19:54.271 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.272 INFO:tasks.workunit.client.0.smithi195.stdout:- if (g_fake_kill_after.count(entity_name_t::MDS(i))) { 2022-06-27T12:19:54.272 INFO:tasks.workunit.client.0.smithi195.stdout:- cerr << "mds" << i << " will die after " << g_fake_kill_after[entity_name_t::MDS(i)] << std::endl; 2022-06-27T12:19:54.273 INFO:tasks.workunit.client.0.smithi195.stdout:- g_timer.add_event_after(g_fake_kill_after[entity_name_t::MDS(i)], new C_Die); 2022-06-27T12:19:54.273 INFO:tasks.workunit.client.0.smithi195.stdout:- } 2022-06-27T12:19:54.273 INFO:tasks.workunit.client.0.smithi195.stdout:- } 2022-06-27T12:19:54.273 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.274 INFO:tasks.workunit.client.0.smithi195.stdout:- // create osd 2022-06-27T12:19:54.274 INFO:tasks.workunit.client.0.smithi195.stdout:- map osd; 2022-06-27T12:19:54.274 INFO:tasks.workunit.client.0.smithi195.stdout:- int max_osd_nodes = world - start_mds - g_conf.ms_skip_rank0; // assumes 0 clients, if we stripe. 2022-06-27T12:19:54.274 INFO:tasks.workunit.client.0.smithi195.stdout:- int osds_per_node = (start_osd-1)/max_osd_nodes + 1; 2022-06-27T12:19:54.275 INFO:tasks.workunit.client.0.smithi195.stdout:- for (int i=0; iget_myaddr() << " " << hostname << "." << pid << std::endl; 2022-06-27T12:19:54.277 INFO:tasks.workunit.client.0.smithi195.stdout:- if (g_conf.dout_dir) { 2022-06-27T12:19:54.278 INFO:tasks.workunit.client.0.smithi195.stdout:- sprintf(ffrom, "%s/osd%d", g_conf.dout_dir, i); 2022-06-27T12:19:54.278 INFO:tasks.workunit.client.0.smithi195.stdout:- ::unlink(ffrom); 2022-06-27T12:19:54.278 INFO:tasks.workunit.client.0.smithi195.stdout:- ::symlink(fto, ffrom); 2022-06-27T12:19:54.278 INFO:tasks.workunit.client.0.smithi195.stdout:- } 2022-06-27T12:19:54.279 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.279 INFO:tasks.workunit.client.0.smithi195.stdout:- osd[i] = new OSD(i, m, monmap); 2022-06-27T12:19:54.279 INFO:tasks.workunit.client.0.smithi195.stdout:- if (osd[i]->init() < 0) 2022-06-27T12:19:54.279 INFO:tasks.workunit.client.0.smithi195.stdout:- return 1; 2022-06-27T12:19:54.279 INFO:tasks.workunit.client.0.smithi195.stdout:- started++; 2022-06-27T12:19:54.280 INFO:tasks.workunit.client.0.smithi195.stdout:- } 2022-06-27T12:19:54.280 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.280 INFO:tasks.workunit.client.0.smithi195.stdout:- if (g_conf.ms_overlay_clients) sleep(5); 2022-06-27T12:19:54.281 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.281 INFO:tasks.workunit.client.0.smithi195.stdout:- // create client 2022-06-27T12:19:54.281 INFO:tasks.workunit.client.0.smithi195.stdout:- int skip_osd = start_osd; 2022-06-27T12:19:54.282 INFO:tasks.workunit.client.0.smithi195.stdout:- if (g_conf.ms_overlay_clients) 2022-06-27T12:19:54.282 INFO:tasks.workunit.client.0.smithi195.stdout:- skip_osd = 0; // put clients with osds too! 2022-06-27T12:19:54.282 INFO:tasks.workunit.client.0.smithi195.stdout:- int client_nodes = world - start_mds - skip_osd - g_conf.ms_skip_rank0; 2022-06-27T12:19:54.283 INFO:tasks.workunit.client.0.smithi195.stdout:- int clients_per_node = 1; 2022-06-27T12:19:54.283 INFO:tasks.workunit.client.0.smithi195.stdout:- if (start_client && client_nodes > 0) clients_per_node = (start_client-1) / client_nodes + 1; 2022-06-27T12:19:54.283 INFO:tasks.workunit.client.0.smithi195.stdout:- set clientlist; 2022-06-27T12:19:54.283 INFO:tasks.workunit.client.0.smithi195.stdout:- map client; 2022-06-27T12:19:54.284 INFO:tasks.workunit.client.0.smithi195.stdout:- map syn; 2022-06-27T12:19:54.284 INFO:tasks.workunit.client.0.smithi195.stdout:- int nclients = 0; 2022-06-27T12:19:54.285 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.285 INFO:tasks.workunit.client.0.smithi195.stdout:- // create the synthetic clients, and one Ceph client per synthetic client 2022-06-27T12:19:54.286 INFO:tasks.workunit.client.0.smithi195.stdout:- Client* single_client = 0; // unless share_single_client... 2022-06-27T12:19:54.286 INFO:tasks.workunit.client.0.smithi195.stdout:- for (int i=0; i::iterator it = clientlist.begin(); 2022-06-27T12:19:54.296 INFO:tasks.workunit.client.0.smithi195.stdout:- it != clientlist.end(); 2022-06-27T12:19:54.296 INFO:tasks.workunit.client.0.smithi195.stdout:- it++) { 2022-06-27T12:19:54.297 INFO:tasks.workunit.client.0.smithi195.stdout:- int i = *it; 2022-06-27T12:19:54.297 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.297 INFO:tasks.workunit.client.0.smithi195.stdout:- //cerr << "starting synthetic" << i << " on rank " << mpirank << std::endl; 2022-06-27T12:19:54.297 INFO:tasks.workunit.client.0.smithi195.stdout:- syn[i]->start_thread(); 2022-06-27T12:19:54.298 INFO:tasks.workunit.client.0.smithi195.stdout:- } 2022-06-27T12:19:54.298 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.298 INFO:tasks.workunit.client.0.smithi195.stdout:- // client status message 2022-06-27T12:19:54.299 INFO:tasks.workunit.client.0.smithi195.stdout:- if (nclients) { 2022-06-27T12:19:54.299 INFO:tasks.workunit.client.0.smithi195.stdout:- if (share_single_client) 2022-06-27T12:19:54.299 INFO:tasks.workunit.client.0.smithi195.stdout:- cerr << "In one-client-per-synclient mode:"; 2022-06-27T12:19:54.299 INFO:tasks.workunit.client.0.smithi195.stdout:- cerr << nclients << " clients at " << rank.rank_addr << " " << hostname << "." << pid << std::endl; 2022-06-27T12:19:54.300 INFO:tasks.workunit.client.0.smithi195.stdout:- } 2022-06-27T12:19:54.300 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.300 INFO:tasks.workunit.client.0.smithi195.stdout:- // wait for the synthetic clients to finish 2022-06-27T12:19:54.301 INFO:tasks.workunit.client.0.smithi195.stdout:- for (set::iterator it = clientlist.begin(); 2022-06-27T12:19:54.301 INFO:tasks.workunit.client.0.smithi195.stdout:- it != clientlist.end(); 2022-06-27T12:19:54.301 INFO:tasks.workunit.client.0.smithi195.stdout:- it++) { 2022-06-27T12:19:54.301 INFO:tasks.workunit.client.0.smithi195.stdout:- int i = *it; 2022-06-27T12:19:54.302 INFO:tasks.workunit.client.0.smithi195.stdout:- // cout << "waiting for synthetic client" << i << " to finish" << std::endl; 2022-06-27T12:19:54.302 INFO:tasks.workunit.client.0.smithi195.stdout:- syn[i]->join_thread(); 2022-06-27T12:19:54.302 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.303 INFO:tasks.workunit.client.0.smithi195.stdout:- // fix simplemessenger race before deleting synclients and clients 2022-06-27T12:19:54.303 INFO:tasks.workunit.client.0.smithi195.stdout:- // delete syn[i]; 2022-06-27T12:19:54.304 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.304 INFO:tasks.workunit.client.0.smithi195.stdout:- // if (!ALL_SYNCLIENTS_THROUGH_ONE_CLIENT) 2022-06-27T12:19:54.304 INFO:tasks.workunit.client.0.smithi195.stdout:- // delete client[i]; 2022-06-27T12:19:54.305 INFO:tasks.workunit.client.0.smithi195.stdout:- } 2022-06-27T12:19:54.305 INFO:tasks.workunit.client.0.smithi195.stdout:- // if (ALL_SYNCLIENTS_THROUGH_ONE_CLIENT) 2022-06-27T12:19:54.305 INFO:tasks.workunit.client.0.smithi195.stdout:- // delete client[0]; 2022-06-27T12:19:54.306 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.306 INFO:tasks.workunit.client.0.smithi195.stdout:- if (mpirank && !started) { 2022-06-27T12:19:54.307 INFO:tasks.workunit.client.0.smithi195.stdout:- //dout(1) << "IDLE" << dendl; 2022-06-27T12:19:54.307 INFO:tasks.workunit.client.0.smithi195.stdout:- cerr << "idle at " << rank.rank_addr << " mpirank " << mpirank << " " << hostname << "." << pid << std::endl; 2022-06-27T12:19:54.307 INFO:tasks.workunit.client.0.smithi195.stdout:- } 2022-06-27T12:19:54.308 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.308 INFO:tasks.workunit.client.0.smithi195.stdout:- // wait for everything to finish 2022-06-27T12:19:54.308 INFO:tasks.workunit.client.0.smithi195.stdout:- rank.wait(); 2022-06-27T12:19:54.309 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.309 INFO:tasks.workunit.client.0.smithi195.stdout:- cerr << "newsyn done on " << hostname << "." << pid << std::endl; 2022-06-27T12:19:54.310 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.310 INFO:tasks.workunit.client.0.smithi195.stdout:- // cd on exit, so that gmon.out (if any) goes into a separate directory for each node. 2022-06-27T12:19:54.310 INFO:tasks.workunit.client.0.smithi195.stdout:- char s[20]; 2022-06-27T12:19:54.311 INFO:tasks.workunit.client.0.smithi195.stdout:- sprintf(s, "gmon/%d", mpirank); 2022-06-27T12:19:54.311 INFO:tasks.workunit.client.0.smithi195.stdout:- mkdir(s, 0755); 2022-06-27T12:19:54.312 INFO:tasks.workunit.client.0.smithi195.stdout:- chdir(s); 2022-06-27T12:19:54.312 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.312 INFO:tasks.workunit.client.0.smithi195.stdout:- return 0; // whatever, cleanup hangs sometimes (stopping ebofs threads?). 2022-06-27T12:19:54.313 INFO:tasks.workunit.client.0.smithi195.stdout:- 2022-06-27T12:19:54.313 INFO:tasks.workunit.client.0.smithi195.stdout:- // cleanup 2022-06-27T12:19:54.314 INFO:tasks.workunit.client.0.smithi195.stdout:- for (map::iterator i = mds.begin(); i != mds.end(); i++) 2022-06-27T12:19:54.314 INFO:tasks.workunit.client.0.smithi195.stdout:- delete i->second; 2022-06-27T12:19:54.314 INFO:tasks.workunit.client.0.smithi195.stdout:- for (map::iterator i = mdsosd.begin(); i != mdsosd.end(); i++) 2022-06-27T12:19:54.315 INFO:tasks.workunit.client.0.smithi195.stdout:- delete i->second; 2022-06-27T12:19:54.315 INFO:tasks.workunit.client.0.smithi195.stdout:- for (map::iterator i = osd.begin(); i != osd.end(); i++) 2022-06-27T12:19:54.315 INFO:tasks.workunit.client.0.smithi195.stdout:- delete i->second; 2022-06-27T12:19:54.315 INFO:tasks.workunit.client.0.smithi195.stdout:- /* 2022-06-27T12:19:54.316 INFO:tasks.workunit.client.0.smithi195.stdout:- for (map::iterator i = client.begin(); i != client.end(); i++) 2022-06-27T12:19:54.316 INFO:tasks.workunit.client.0.smithi195.stdout:- delete i->second; 2022-06-27T12:19:54.316 INFO:tasks.workunit.client.0.smithi195.stdout:- for (map::iterator i = syn.begin(); i != syn.end(); i++) 2022-06-27T12:19:54.316 INFO:tasks.workunit.client.0.smithi195.stdout:- delete i->second; 2022-06-27T12:19:54.317 INFO:tasks.workunit.client.0.smithi195.stdout:- */ 2022-06-27T12:19:54.317 INFO:tasks.workunit.client.0.smithi195.stdout:- /* 2022-06-27T12:19:54.317 INFO:tasks.workunit.client.0.smithi195.stdout:- for (int i=0; i sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2022-06-27T12:19:54.561 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_082ae7ef4302fa54665ed0a2535e8e254118dcfd/teuthology/run_tasks.py", line 103, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_082ae7ef4302fa54665ed0a2535e8e254118dcfd/teuthology/run_tasks.py", line 82, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_5292901bc999daa0d7ac128764a3f27c4233c252/qa/tasks/workunit.py", line 148, in task cleanup=cleanup) File "/home/teuthworker/src/git.ceph.com_ceph-c_5292901bc999daa0d7ac128764a3f27c4233c252/qa/tasks/workunit.py", line 298, in _spawn_on_all_clients timeout=timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_082ae7ef4302fa54665ed0a2535e8e254118dcfd/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_git_teuthology_082ae7ef4302fa54665ed0a2535e8e254118dcfd/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_git_teuthology_082ae7ef4302fa54665ed0a2535e8e254118dcfd/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_git_teuthology_082ae7ef4302fa54665ed0a2535e8e254118dcfd/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_5292901bc999daa0d7ac128764a3f27c4233c252/qa/tasks/workunit.py", line 427, in _run_tests label="workunit test {workunit}".format(workunit=workunit) File "/home/teuthworker/src/git.ceph.com_git_teuthology_082ae7ef4302fa54665ed0a2535e8e254118dcfd/teuthology/orchestra/remote.py", line 510, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_082ae7ef4302fa54665ed0a2535e8e254118dcfd/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_082ae7ef4302fa54665ed0a2535e8e254118dcfd/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_082ae7ef4302fa54665ed0a2535e8e254118dcfd/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi195 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5292901bc999daa0d7ac128764a3f27c4233c252 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh' 2022-06-27T12:19:54.713 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=c8ea50dd901a406997fcd25f38296b8a Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_082ae7ef4302fa54665ed0a2535e8e254118dcfd/teuthology/run_tasks.py", line 103, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_082ae7ef4302fa54665ed0a2535e8e254118dcfd/teuthology/run_tasks.py", line 82, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_5292901bc999daa0d7ac128764a3f27c4233c252/qa/tasks/workunit.py", line 148, in task cleanup=cleanup) File "/home/teuthworker/src/git.ceph.com_ceph-c_5292901bc999daa0d7ac128764a3f27c4233c252/qa/tasks/workunit.py", line 298, in _spawn_on_all_clients timeout=timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_082ae7ef4302fa54665ed0a2535e8e254118dcfd/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_git_teuthology_082ae7ef4302fa54665ed0a2535e8e254118dcfd/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_git_teuthology_082ae7ef4302fa54665ed0a2535e8e254118dcfd/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_git_teuthology_082ae7ef4302fa54665ed0a2535e8e254118dcfd/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_5292901bc999daa0d7ac128764a3f27c4233c252/qa/tasks/workunit.py", line 427, in _run_tests label="workunit test {workunit}".format(workunit=workunit) File "/home/teuthworker/src/git.ceph.com_git_teuthology_082ae7ef4302fa54665ed0a2535e8e254118dcfd/teuthology/orchestra/remote.py", line 510, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_082ae7ef4302fa54665ed0a2535e8e254118dcfd/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_082ae7ef4302fa54665ed0a2535e8e254118dcfd/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_082ae7ef4302fa54665ed0a2535e8e254118dcfd/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi195 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5292901bc999daa0d7ac128764a3f27c4233c252 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh' 2022-06-27T12:19:54.717 DEBUG:teuthology.run_tasks:Unwinding manager mds_thrash 2022-06-27T12:19:54.726 INFO:tasks.mds_thrash:joining mds_thrasher 2022-06-27T12:19:54.730 INFO:tasks.mds_thrash.fs.[cephfs]:stat['max_mds'] = 0 2022-06-27T12:19:54.731 INFO:tasks.mds_thrash.fs.[cephfs]:stat['deactivate'] = 0 2022-06-27T12:19:54.731 INFO:tasks.mds_thrash.fs.[cephfs]:stat['kill'] = 19 2022-06-27T12:19:54.732 INFO:tasks.mds_thrash:done joining 2022-06-27T12:19:54.732 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2022-06-27T12:19:54.754 INFO:tasks.kclient:Unmounting kernel clients... 2022-06-27T12:19:54.755 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2022-06-27T12:19:54.755 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-06-27T12:19:54.756 DEBUG:teuthology.orchestra.run.smithi195:> sudo umount /home/ubuntu/cephtest/mnt.0 2022-06-27T12:20:05.885 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:20:05.896 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:20:35.977 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:20:35.986 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:21:06.053 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:21:06.078 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:21:36.168 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:21:36.172 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:22:06.238 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:22:06.247 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:22:36.407 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:22:36.439 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:23:06.592 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:23:06.599 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:23:36.880 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:23:36.884 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:24:06.966 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:24:06.970 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:24:37.231 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:24:37.266 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:24:54.782 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_082ae7ef4302fa54665ed0a2535e8e254118dcfd/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 747, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_082ae7ef4302fa54665ed0a2535e8e254118dcfd/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_082ae7ef4302fa54665ed0a2535e8e254118dcfd/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_082ae7ef4302fa54665ed0a2535e8e254118dcfd/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_082ae7ef4302fa54665ed0a2535e8e254118dcfd/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_082ae7ef4302fa54665ed0a2535e8e254118dcfd/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_082ae7ef4302fa54665ed0a2535e8e254118dcfd/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1376, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_082ae7ef4302fa54665ed0a2535e8e254118dcfd/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 749, in recv_stderr raise socket.timeout() socket.timeout 2022-06-27T12:24:54.816 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_082ae7ef4302fa54665ed0a2535e8e254118dcfd/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 699, in recv out = self.in_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_082ae7ef4302fa54665ed0a2535e8e254118dcfd/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_082ae7ef4302fa54665ed0a2535e8e254118dcfd/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_082ae7ef4302fa54665ed0a2535e8e254118dcfd/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_082ae7ef4302fa54665ed0a2535e8e254118dcfd/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_082ae7ef4302fa54665ed0a2535e8e254118dcfd/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_082ae7ef4302fa54665ed0a2535e8e254118dcfd/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1361, in _read return self.channel.recv(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_082ae7ef4302fa54665ed0a2535e8e254118dcfd/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 701, in recv raise socket.timeout() socket.timeout 2022-06-27T12:25:07.370 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:25:07.375 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:25:07.426 INFO:tasks.ceph.mds.f.smithi202.stderr:2022-06-27T12:25:07.423+0000 7f38e894d700 -1 received signal: Hangup from killall ceph-mds -1 (PID: 101959) UID: 0 2022-06-27T12:25:07.427 INFO:tasks.ceph.mds.b.smithi202.stderr:2022-06-27T12:25:07.424+0000 7fcbee80c700 -1 received signal: Hangup from killall ceph-mds -1 (PID: 101959) UID: 0 2022-06-27T12:25:07.428 INFO:tasks.ceph.mds.d.smithi202.stderr:2022-06-27T12:25:07.424+0000 7f2fdf3c4700 -1 received signal: Hangup from killall ceph-mds -1 (PID: 101959) UID: 0 2022-06-27T12:27:20.167 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:27:20.178 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:27:50.462 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:27:50.466 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:28:20.532 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:28:20.536 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:28:50.636 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:28:50.645 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:29:20.766 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:29:20.771 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:29:50.846 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:29:50.851 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:30:21.051 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:30:21.055 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:30:51.138 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:30:51.143 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:31:21.269 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:31:21.273 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:31:51.559 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:31:51.564 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:32:21.698 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:32:21.705 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:32:51.930 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:32:51.935 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:33:22.005 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:33:22.009 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:33:52.259 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:33:52.263 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:34:22.401 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:34:22.409 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:34:52.486 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:34:52.490 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:35:22.755 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:35:22.762 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:35:52.829 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:35:52.833 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:36:22.981 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:36:22.987 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:36:53.105 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:36:53.111 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:37:23.176 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:37:23.180 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:37:53.264 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:37:53.269 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:38:23.430 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:38:23.434 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:38:53.502 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:38:53.508 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:39:23.647 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:39:23.662 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:39:53.761 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:39:53.765 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:40:23.917 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:40:23.925 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:40:54.032 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:40:54.041 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:41:24.177 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:41:24.185 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:41:54.309 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:41:54.317 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:42:24.446 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:42:24.454 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:42:54.621 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:42:54.627 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:43:24.739 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:43:24.749 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:43:54.820 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:43:54.824 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:44:24.966 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:44:24.971 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:44:55.129 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:44:55.134 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:45:25.284 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:45:25.288 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:45:55.373 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:45:55.377 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:46:25.511 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:46:25.515 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:46:55.642 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:46:55.646 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:47:25.956 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:47:25.960 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:47:56.037 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:47:56.042 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:48:26.311 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:48:26.315 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:48:56.390 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:48:56.394 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:49:26.697 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:49:26.701 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:49:56.779 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:49:56.783 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:50:26.915 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:50:26.919 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:50:57.080 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:50:57.084 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:51:27.209 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:51:27.214 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:51:57.474 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:51:57.479 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:52:27.569 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:52:27.573 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:52:57.723 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:52:57.727 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:53:27.830 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:53:27.835 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:53:58.097 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:53:58.101 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:54:28.215 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:54:28.220 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:54:58.379 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:54:58.384 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:55:28.509 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:55:28.514 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:55:58.753 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:55:58.757 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:56:28.819 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:56:28.824 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:56:58.951 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:56:58.955 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:57:29.030 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:57:29.034 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:57:59.383 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:57:59.387 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:58:29.454 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:58:29.458 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:58:59.596 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:58:59.600 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:59:29.882 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:59:29.885 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:59:59.949 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T12:59:59.953 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:00:30.104 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:00:30.109 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:01:00.242 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:01:00.248 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:01:30.340 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:01:30.345 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:02:00.413 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:02:00.418 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:02:30.542 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:02:30.546 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:03:00.649 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:03:00.653 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:03:30.778 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:03:30.782 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:04:00.883 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:04:00.887 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:04:31.031 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:04:31.035 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:05:01.126 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:05:01.131 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:05:31.317 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:05:31.322 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:06:01.429 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:06:01.433 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:06:31.609 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:06:31.613 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:07:01.732 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:07:01.737 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:07:31.897 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:07:31.903 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:08:02.018 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:08:02.024 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:08:32.174 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:08:32.178 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:09:02.328 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:09:02.334 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:09:32.558 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:09:32.577 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:10:02.781 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:10:02.785 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:10:32.928 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:10:32.932 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:11:03.090 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:11:03.096 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:11:33.289 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:11:33.295 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:12:03.361 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:12:03.366 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:12:33.449 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:12:33.453 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:13:03.596 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:13:03.601 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:13:33.735 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:13:33.741 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:14:03.840 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:14:03.846 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:14:33.996 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:14:34.001 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:15:04.142 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:15:04.148 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:15:34.371 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:15:34.375 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:16:04.519 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:16:04.523 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:16:34.740 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:16:34.743 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:17:04.863 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:17:04.867 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:17:35.002 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:17:35.293 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:18:05.365 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:18:05.369 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:18:35.513 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:18:35.517 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:19:05.585 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:19:05.589 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:19:35.942 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:19:35.948 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:20:06.020 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:20:06.025 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:20:36.160 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:20:36.164 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:21:06.413 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:21:06.417 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:21:36.477 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:21:36.483 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:22:06.611 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:22:06.615 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:22:36.767 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:22:36.772 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:23:06.889 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:23:06.894 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:23:36.959 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:23:36.964 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:24:07.032 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:24:07.040 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:24:37.244 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:24:37.249 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:25:07.327 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:25:07.332 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:25:37.469 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:25:37.473 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:26:07.598 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:26:07.602 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:26:37.736 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:26:37.740 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:27:07.900 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:27:07.905 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:27:37.969 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:27:37.974 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:28:08.083 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:28:08.087 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:28:38.150 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:28:38.154 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:29:08.231 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:29:08.235 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:29:38.441 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:29:38.446 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:30:08.575 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:30:08.591 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:30:38.811 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:30:38.815 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:31:08.927 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:31:08.932 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:31:39.057 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:31:39.061 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:32:09.302 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:32:09.306 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:32:39.375 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:32:39.379 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:33:09.456 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:33:09.461 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:33:39.621 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:33:39.625 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:34:09.741 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:34:09.745 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:34:39.888 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:34:39.892 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:35:09.985 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:35:09.989 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:35:40.275 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:35:40.280 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:36:10.347 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:36:10.351 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:36:40.493 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:36:40.497 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:37:10.557 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:37:10.562 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:37:40.856 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:37:40.864 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:38:10.955 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:38:10.960 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:38:41.127 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:38:41.138 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:39:11.271 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:39:11.279 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:39:41.437 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:39:41.443 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:40:11.543 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:40:11.552 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:40:41.665 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:40:41.669 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:41:11.807 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:41:11.812 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:41:41.972 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:41:41.977 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:42:12.270 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:42:12.275 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:42:42.427 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:42:42.431 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:43:12.562 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:43:12.566 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:43:42.677 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:43:42.681 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:44:12.817 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:44:12.821 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:44:43.080 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:44:43.085 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:45:13.162 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:45:13.170 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:45:43.242 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:45:43.247 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:46:13.430 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:46:13.434 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:46:43.688 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:46:43.693 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:47:13.765 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:47:13.771 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:47:44.082 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:47:44.090 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:48:14.171 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:48:14.175 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:48:44.335 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:48:44.510 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:49:14.595 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:49:14.603 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:49:44.720 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:49:44.726 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:50:14.809 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:50:14.815 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:50:44.904 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:50:44.908 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:51:15.163 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:51:15.167 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:51:45.230 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:51:45.235 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:52:15.326 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:52:15.329 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:52:45.539 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:52:45.543 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:53:15.693 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:53:15.697 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:53:45.833 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:53:45.843 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:54:15.982 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:54:15.987 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:54:46.135 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:54:46.139 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:55:16.299 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:55:16.304 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:55:46.454 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:55:46.457 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:56:16.604 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:56:16.609 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:56:46.675 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:56:46.680 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:57:16.812 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:57:16.817 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:57:46.967 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:57:46.972 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:58:17.089 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:58:17.094 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:58:47.253 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:58:47.257 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:59:17.392 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:59:17.396 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:59:47.537 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T13:59:47.541 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:00:17.678 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:00:17.732 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:00:47.807 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:00:47.816 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:01:17.975 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:01:17.990 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:01:48.092 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:01:48.099 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:02:18.209 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:02:18.215 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:02:48.365 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:02:48.369 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:03:18.503 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:03:18.509 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:03:48.634 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:03:48.640 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:04:18.734 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:04:18.755 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:04:48.849 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:04:48.858 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:05:19.004 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:05:19.010 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:05:49.167 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:05:49.171 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:06:19.314 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:06:19.319 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:06:49.427 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:06:49.433 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:07:19.591 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:07:19.597 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:07:49.671 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:07:49.675 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:08:19.841 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:08:19.845 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:08:49.947 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:08:49.951 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:09:20.020 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:09:20.023 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:09:50.165 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:09:50.170 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:10:20.230 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:10:20.235 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:10:50.351 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:10:50.357 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:11:20.514 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:11:20.519 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:11:50.645 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:11:50.655 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:12:20.784 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:12:20.790 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:12:50.898 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:12:50.902 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:13:21.040 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:13:21.046 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:13:51.184 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:13:51.189 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:14:21.330 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:14:21.334 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:14:51.460 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:14:51.478 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:15:21.614 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:15:21.621 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:15:51.747 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:15:51.752 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:16:21.912 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:16:21.917 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:16:52.047 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:16:52.052 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:17:22.168 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:17:22.174 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:17:52.425 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:17:52.430 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:18:22.505 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:18:22.511 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:18:52.661 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:18:52.665 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:19:22.803 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:19:22.807 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:19:52.951 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:19:52.954 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:20:23.047 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:20:23.051 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:20:53.186 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:20:53.191 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:21:23.307 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:21:23.311 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:21:53.457 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:21:53.461 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:22:23.589 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:22:23.593 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:22:53.705 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:22:53.709 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:23:23.868 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:23:23.872 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:23:54.031 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:23:54.037 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:24:24.168 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:24:24.172 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:24:54.333 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:24:54.337 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:25:24.469 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:25:24.474 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:25:54.627 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:25:54.632 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:26:24.856 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:26:24.865 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:26:54.985 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:26:54.989 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:27:25.142 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:27:25.147 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:27:55.221 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:27:55.225 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:28:25.384 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:28:25.389 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:28:55.505 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:28:55.511 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:29:25.669 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:29:25.673 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:29:55.793 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:29:55.798 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:30:25.924 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:30:25.928 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:30:56.084 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:30:56.089 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:31:26.231 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:31:26.235 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:31:56.370 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:31:56.374 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:32:26.580 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:32:26.585 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:32:56.733 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:32:56.740 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:33:26.860 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:33:26.863 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:33:57.007 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:33:57.011 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:34:27.142 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:34:27.146 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:34:57.280 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:34:57.284 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:35:27.413 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:35:27.417 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:35:57.559 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:35:57.563 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:36:27.688 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:36:27.693 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:36:57.810 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:36:57.815 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:37:27.958 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:37:27.961 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:37:58.088 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:37:58.092 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:38:28.244 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:38:28.249 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:38:58.399 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:38:58.405 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:39:28.529 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:39:28.533 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:39:58.683 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:39:58.688 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:40:28.829 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:40:28.834 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:40:58.955 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:40:58.958 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:41:29.110 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:41:29.114 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:41:59.270 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:41:59.274 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:42:29.425 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:42:29.431 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:42:59.545 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:42:59.550 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:43:29.692 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:43:29.696 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:43:59.831 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:43:59.835 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:44:29.993 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:44:29.997 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:45:00.157 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:45:00.162 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:45:30.312 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:45:30.316 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:46:00.376 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:46:00.381 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:46:30.515 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:46:30.520 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:47:00.596 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:47:00.600 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:47:30.734 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:47:30.738 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:48:00.865 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:48:00.868 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:48:31.004 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:48:31.009 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:49:01.115 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:49:01.119 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:49:31.236 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:49:31.240 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:50:01.391 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:50:01.395 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:50:31.529 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:50:31.533 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:51:01.592 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:51:01.596 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:51:31.727 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:51:31.731 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:52:01.885 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:52:01.889 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:52:31.958 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:52:31.962 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:53:02.088 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:53:02.092 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:53:32.201 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:53:32.205 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:54:02.361 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:54:02.366 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:54:32.428 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:54:32.433 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:55:02.589 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:55:02.593 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:55:32.719 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:55:32.723 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:56:02.859 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:56:02.863 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:56:32.971 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:56:32.975 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:57:03.094 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:57:03.098 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:57:33.165 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:57:33.170 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:58:03.312 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:58:03.316 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:58:33.499 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:58:33.503 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:59:03.640 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:59:03.650 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:59:33.752 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T14:59:33.757 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:00:03.907 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:00:03.911 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:00:34.061 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:00:34.065 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:01:04.224 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:01:04.228 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:01:34.289 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:01:34.293 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:02:04.442 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:02:04.447 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:02:34.592 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:02:34.596 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:03:04.757 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:03:04.762 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:03:34.874 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:03:34.878 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:04:05.004 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:04:05.008 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:04:35.153 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:04:35.159 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:05:05.318 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:05:05.322 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:05:35.541 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:05:35.548 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:06:05.604 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:06:05.608 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:06:35.677 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:06:35.681 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:07:05.750 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:07:05.755 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:07:35.822 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:07:35.826 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:08:05.894 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:08:05.899 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:08:35.970 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:08:35.974 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:09:06.038 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:09:06.042 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:09:36.178 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:09:36.181 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:10:06.242 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:10:06.246 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:10:36.389 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:10:36.394 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:11:06.461 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:11:06.465 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:11:36.582 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:11:36.586 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:12:06.713 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:12:06.717 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:12:36.851 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:12:36.855 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:13:06.973 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:13:06.976 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:13:37.128 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:13:37.132 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:14:07.242 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:14:07.247 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:14:37.390 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:14:37.394 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:15:07.510 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:15:07.522 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:15:37.659 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:15:37.663 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:16:07.812 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:16:07.816 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:16:37.967 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:16:37.971 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:17:08.067 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:17:08.071 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:17:38.215 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:17:38.220 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:18:08.328 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:18:08.332 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:18:38.493 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:18:38.498 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:19:08.643 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:19:08.648 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:19:38.806 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:19:38.810 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:20:08.884 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:20:08.888 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:20:39.024 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:20:39.028 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:21:09.153 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:21:09.158 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:21:39.308 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:21:39.312 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:22:09.422 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:22:09.426 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:22:39.551 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:22:39.556 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:23:09.665 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:23:09.671 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:23:39.778 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:23:39.783 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:24:09.942 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:24:09.946 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:24:40.073 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:24:40.076 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:25:10.137 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:25:10.141 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:25:40.258 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:25:40.264 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:26:10.330 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:26:10.335 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:26:40.477 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:26:40.481 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:27:10.624 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:27:10.631 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:27:40.771 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:27:40.777 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:28:10.834 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:28:10.839 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:28:40.990 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:28:40.994 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:29:11.128 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:29:11.133 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:29:41.249 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:29:41.254 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:30:11.389 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:30:11.393 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:30:41.535 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:30:41.539 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:31:11.666 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:31:11.669 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:31:41.796 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:31:41.800 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:32:11.934 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:32:11.938 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:32:41.998 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:32:42.002 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:33:12.145 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:33:12.197 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:33:42.275 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:33:42.279 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:34:12.413 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:34:12.419 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:34:42.475 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:34:42.479 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:35:12.549 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:35:12.554 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:35:42.695 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:35:42.699 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:36:12.760 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:36:12.768 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:36:42.856 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:36:42.861 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:37:12.987 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:37:12.992 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:37:43.109 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:37:43.113 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:38:13.228 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:38:13.232 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:38:43.409 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:38:43.413 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:39:13.538 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:39:13.542 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:39:43.637 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:39:43.641 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:40:13.772 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:40:13.777 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:40:43.936 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:40:43.940 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:41:14.065 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:41:14.070 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:41:44.188 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:41:44.193 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:42:14.328 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:42:14.333 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:42:44.490 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:42:44.495 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:43:14.645 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:43:14.652 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:43:44.750 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:43:44.754 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:44:14.898 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:44:14.902 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:44:45.028 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:44:45.032 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:45:15.158 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:45:15.163 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:45:45.315 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:45:45.319 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:46:15.453 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:46:15.457 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:46:45.541 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:46:45.546 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:47:15.693 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:47:15.697 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:47:45.964 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:47:45.969 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:48:16.033 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:48:16.038 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:48:46.196 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:48:46.201 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:49:16.270 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:49:16.275 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:49:46.401 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:49:46.405 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:50:16.466 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:50:16.471 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:50:46.620 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:50:46.624 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:51:16.751 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:51:16.756 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:51:46.889 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:51:46.894 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:52:16.953 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:52:16.956 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:52:47.076 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:52:47.080 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:53:17.139 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:53:17.143 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:53:47.285 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:53:47.290 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:54:17.372 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:54:17.377 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:54:47.512 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:54:47.517 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:55:17.677 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:55:17.682 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:55:47.798 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:55:47.803 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:56:17.928 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:56:17.933 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:56:48.006 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:56:48.011 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:57:18.094 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:57:18.099 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:57:48.169 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:57:48.173 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:58:18.317 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:58:18.321 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:58:48.464 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:58:48.468 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:59:18.622 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:59:18.626 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:59:48.741 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T15:59:48.746 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:00:18.889 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:00:18.894 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:00:49.024 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:00:49.029 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:01:19.109 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:01:19.113 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:01:49.256 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:01:49.266 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:02:19.378 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:02:19.383 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:02:49.533 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:02:49.538 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:03:19.697 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:03:19.709 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:03:49.818 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:03:49.824 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:04:19.966 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:04:19.972 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:04:50.078 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:04:50.082 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:05:20.150 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:05:20.154 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:05:50.305 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:05:50.309 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:06:20.452 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:06:20.456 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:06:50.558 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:06:50.563 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:07:20.712 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:07:20.716 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:07:50.834 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:07:50.838 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:08:20.983 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:08:20.988 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:08:51.166 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:08:51.169 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:09:21.300 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:09:21.304 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:09:51.438 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:09:51.442 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:10:21.552 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:10:21.556 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:10:51.715 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:10:51.720 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:11:21.862 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:11:21.866 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:11:51.995 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:11:51.999 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:12:22.138 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:12:22.142 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:12:52.260 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:12:52.264 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:13:22.406 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:13:22.410 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:13:52.562 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:13:52.567 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:14:22.700 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:14:22.704 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:14:52.844 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:14:52.849 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:15:22.993 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:15:22.997 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:15:53.139 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:15:53.143 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:16:23.285 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:16:23.295 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:16:53.458 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:16:53.463 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:17:23.598 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:17:23.603 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:17:53.703 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:17:53.708 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:18:24.055 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:18:24.060 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:18:54.218 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:18:54.222 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:19:24.281 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:19:24.286 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:19:54.430 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:19:54.433 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:20:24.554 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:20:24.559 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:20:54.710 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:20:54.714 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:21:24.864 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:21:24.869 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:21:55.003 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:21:55.007 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:22:25.150 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:22:25.154 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:22:55.411 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:22:55.415 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:23:25.517 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:23:25.521 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:23:55.635 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:23:55.639 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:24:25.780 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:24:25.784 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:24:55.895 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:24:55.900 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:25:26.008 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:25:26.012 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:25:56.177 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:25:56.182 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:26:26.303 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:26:26.308 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:26:56.435 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:26:56.440 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:27:26.508 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:27:26.512 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:27:56.621 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:27:56.626 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:28:26.768 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:28:26.774 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:28:56.849 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:28:56.854 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:29:26.988 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:29:26.992 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:29:57.143 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:29:57.147 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:30:27.298 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:30:27.302 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:30:57.379 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:30:57.385 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:31:27.529 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:31:27.559 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:31:57.716 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:31:57.720 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:32:27.861 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:32:27.865 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:32:57.984 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:32:57.988 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:33:28.123 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:33:28.127 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:33:58.211 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:33:58.215 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:34:28.367 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:34:28.371 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:34:58.437 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:34:58.441 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:35:28.535 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:35:28.539 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:35:58.676 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:35:58.681 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:36:28.804 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:36:28.808 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:36:58.908 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:36:58.912 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:37:29.023 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:37:29.027 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:37:59.145 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:37:59.149 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:38:29.267 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:38:29.271 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:38:59.396 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:38:59.401 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:39:29.552 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:39:29.607 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:39:59.740 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:39:59.745 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:40:29.854 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:40:29.858 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:40:59.919 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:40:59.923 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:41:30.039 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:41:30.043 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:42:00.186 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:42:00.190 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:42:30.325 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:42:30.329 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:43:00.421 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:43:00.425 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:43:30.559 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:43:30.563 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:44:00.655 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:44:00.661 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:44:30.768 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:44:30.772 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:45:00.919 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:45:00.924 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:45:31.050 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:45:31.056 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:46:01.205 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:46:01.210 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:46:31.310 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:46:31.316 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:47:01.402 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:47:01.406 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:47:31.537 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:47:31.542 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:48:01.675 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:48:01.680 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:48:31.831 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:48:31.835 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:49:01.978 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:49:01.984 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:49:32.084 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:49:32.088 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:50:02.204 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:50:02.210 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:50:32.313 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:50:32.319 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:51:02.473 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:51:02.487 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:51:32.579 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:51:32.584 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:52:02.726 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:52:02.731 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:52:32.866 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:52:32.870 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:53:03.003 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:53:03.007 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:53:33.142 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:53:33.147 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:54:03.222 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:54:03.226 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:54:33.385 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:54:33.395 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:55:03.474 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:55:03.479 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:55:33.620 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:55:33.624 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:56:03.692 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:56:03.698 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:56:33.847 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:56:33.851 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:57:03.919 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:57:03.925 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:57:34.026 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:57:34.031 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:58:04.090 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:58:04.094 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:58:34.221 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:58:34.224 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:59:04.335 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:59:04.338 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:59:34.480 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T16:59:34.483 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:00:04.593 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:00:04.598 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:00:34.743 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:00:34.748 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:01:04.898 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:01:04.903 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:01:35.043 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:01:35.048 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:02:05.168 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:02:05.173 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:02:35.310 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:02:35.322 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:03:05.470 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:03:05.476 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:03:35.574 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:03:35.578 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:04:05.738 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:04:05.743 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:04:35.894 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:04:35.899 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:05:06.050 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:05:06.056 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:05:36.203 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:05:36.208 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:06:06.275 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:06:06.279 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:06:36.341 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:06:36.347 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:07:06.476 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:07:06.482 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:07:36.625 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:07:36.629 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:08:06.772 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:08:06.777 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:08:36.935 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:08:36.940 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:09:07.082 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:09:07.087 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:09:37.237 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:09:37.241 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:10:07.393 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:10:07.400 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:10:37.507 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:10:37.522 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:11:07.661 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:11:07.666 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:11:37.808 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:11:37.812 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:12:07.964 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:12:07.968 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:12:38.100 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:12:38.115 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:13:08.206 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:13:08.231 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:13:38.380 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:13:38.385 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:14:08.535 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:14:08.540 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:14:38.655 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:14:38.660 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:15:08.777 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:15:08.782 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:15:38.890 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:15:38.895 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:16:09.021 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:16:09.027 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:16:39.151 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:16:39.155 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:17:09.274 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:17:09.280 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:17:39.429 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:17:39.433 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:18:09.544 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:18:09.549 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:18:39.673 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:18:39.678 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:19:09.758 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:19:09.763 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:19:39.836 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:19:39.848 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:20:09.906 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:20:09.910 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:20:40.065 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:20:40.070 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:21:10.143 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:21:10.147 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:21:40.300 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:21:40.307 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:22:10.388 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:22:10.393 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:22:40.544 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:22:40.555 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:23:10.708 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:23:10.713 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:23:40.797 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:23:40.805 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:24:10.914 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:24:10.920 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:24:41.074 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:24:41.078 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:25:11.187 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:25:11.216 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:25:41.320 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:25:41.328 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:26:11.462 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:26:11.465 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:26:41.630 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:26:41.635 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:27:11.773 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:27:11.778 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:27:41.903 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:27:41.908 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:28:11.992 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:28:11.998 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:28:42.139 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:28:42.144 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:29:12.225 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:29:12.229 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:29:42.332 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:29:42.337 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:30:12.468 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:30:12.472 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:30:42.602 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:30:42.608 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:31:12.746 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:31:12.750 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:31:42.874 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:31:42.879 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:32:13.000 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:32:13.004 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:32:43.148 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:32:43.154 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:33:13.361 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:33:13.366 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:33:43.429 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:33:43.434 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:34:13.569 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:34:13.574 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:34:43.734 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:34:43.739 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:35:13.897 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:35:13.902 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:35:44.028 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:35:44.032 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:36:14.147 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:36:14.150 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:36:44.297 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:36:44.301 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:37:14.419 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:37:14.425 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:37:44.549 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:37:44.553 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:38:14.672 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:38:14.677 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:38:44.811 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:38:44.816 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:39:14.983 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:39:14.987 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:39:45.130 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:39:45.134 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:40:15.269 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:40:15.273 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:40:45.333 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:40:45.340 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:41:15.481 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:41:15.486 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:41:45.591 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:41:45.595 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:42:15.749 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:42:15.755 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:42:45.888 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:42:45.900 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:43:15.990 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:43:15.998 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:43:46.137 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:43:46.140 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:44:16.241 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:44:16.247 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:44:46.373 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:44:46.382 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:45:16.452 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:45:16.457 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:45:46.584 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:45:46.589 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:46:16.720 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:46:16.725 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:46:46.859 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:46:46.864 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:47:16.940 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:47:16.945 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:47:47.078 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:47:47.082 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:48:17.176 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:48:17.182 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:48:47.315 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:48:47.321 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:49:17.403 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:49:17.409 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:49:47.474 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:49:47.479 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:50:17.562 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:50:17.568 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:50:47.720 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:50:47.724 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:51:17.858 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:51:17.873 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:51:47.997 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:51:48.003 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:52:18.137 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:52:18.142 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:52:48.204 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:52:48.209 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:53:18.364 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:53:18.369 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:53:48.431 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:53:48.435 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:54:18.634 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:54:18.637 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:54:48.721 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:54:48.726 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:55:18.784 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:55:18.789 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:55:48.916 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:55:48.920 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:56:19.046 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:56:19.052 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:56:49.160 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:56:49.165 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:57:19.307 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:57:19.311 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:57:49.388 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:57:49.395 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:58:19.527 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:58:19.532 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:58:49.596 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:58:49.600 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:59:19.707 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:59:19.715 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:59:49.844 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T17:59:49.852 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:00:19.989 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:00:19.995 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:00:50.097 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:00:50.104 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:01:20.235 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:01:20.241 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:01:50.410 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:01:50.433 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:02:20.573 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:02:20.577 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:02:50.701 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:02:50.707 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:03:20.839 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:03:20.980 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:03:51.046 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:03:51.050 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:04:21.200 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:04:21.205 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:04:51.356 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:04:51.360 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:05:21.417 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:05:21.422 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:05:51.556 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:05:51.561 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:06:21.679 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:06:21.684 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:06:51.742 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:06:51.746 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:07:21.872 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:07:21.876 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:07:52.011 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:07:52.016 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:08:22.141 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:08:22.147 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:08:52.214 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:08:52.220 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:09:22.354 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:09:22.359 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:09:52.500 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:09:52.505 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:10:22.652 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:10:22.656 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:10:52.811 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:10:52.817 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:11:22.891 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:11:22.895 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:11:53.046 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:11:53.050 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:12:23.200 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:12:23.205 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:12:53.313 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:12:53.318 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:13:23.488 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:13:23.494 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:13:53.628 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:13:53.633 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:14:23.766 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:14:23.776 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:14:53.905 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:14:53.910 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:15:24.049 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:15:24.057 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:15:54.122 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:15:54.127 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:16:24.205 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:16:24.213 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:16:54.351 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:16:54.356 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:17:24.489 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:17:24.494 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:17:54.613 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:17:54.617 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:18:24.752 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:18:24.759 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:18:54.918 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:18:54.923 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:19:25.079 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:19:25.083 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:19:55.167 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:19:55.173 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:20:25.251 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:20:25.255 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:20:55.320 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:20:55.324 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:21:25.383 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:21:25.387 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:21:55.447 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:21:55.453 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:22:25.511 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:22:25.515 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:22:55.588 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:22:55.594 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:23:25.657 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:23:25.681 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:23:55.742 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:23:55.747 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:24:25.829 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:24:25.833 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:24:55.891 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:24:55.895 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:25:25.982 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:25:25.988 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:25:56.134 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:25:56.140 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:26:26.248 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:26:26.254 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:26:56.396 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:26:56.402 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:27:26.559 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:27:26.565 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:27:56.698 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:27:56.704 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:28:26.827 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:28:26.841 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:28:56.907 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:28:56.912 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:29:26.995 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:29:26.999 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:29:57.059 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:29:57.064 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:30:27.124 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:30:27.129 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:30:57.257 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:30:57.261 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:31:27.385 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:31:27.389 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:31:57.700 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:31:57.704 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:32:27.846 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:32:27.852 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:32:57.916 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:32:57.922 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:33:27.986 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:33:27.990 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:33:58.062 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:33:58.066 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:34:28.138 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:34:28.146 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:34:58.273 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:34:58.282 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:35:28.403 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:35:28.412 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:35:58.574 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:35:58.579 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:36:28.680 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:36:28.684 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:36:58.827 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:36:58.833 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:37:29.010 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:37:29.015 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:37:59.075 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:37:59.081 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:38:29.228 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:38:29.233 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:38:59.386 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:38:59.389 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:39:29.462 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:39:29.466 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:39:59.625 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:39:59.630 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:40:29.777 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:40:29.781 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:40:59.878 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:40:59.885 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:41:30.024 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:41:30.030 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:42:00.162 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:42:00.166 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:42:30.301 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:42:30.307 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:43:00.381 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:43:00.385 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:43:30.447 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:43:30.453 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:44:00.522 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:44:00.526 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:44:30.613 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:44:30.619 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:45:00.688 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:45:00.696 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:45:30.758 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:45:30.764 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:46:00.903 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:46:00.913 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:46:30.983 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:46:30.999 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:47:01.126 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:47:01.131 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:47:31.271 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:47:31.276 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:48:01.344 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:48:01.350 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:48:31.483 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:48:31.488 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:49:01.554 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:49:01.558 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:49:31.643 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:49:31.647 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:50:01.714 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:50:01.721 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:50:31.797 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:50:31.801 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:51:01.863 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:51:01.868 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:51:31.939 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:51:31.975 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:52:02.045 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:52:02.049 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:52:32.127 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:52:32.133 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:53:02.208 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:53:02.213 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:53:32.272 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:53:32.277 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:54:02.402 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:54:02.412 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:54:32.549 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:54:32.553 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:55:02.670 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:55:02.675 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:55:32.784 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:55:32.789 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:56:02.938 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:56:02.943 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:56:33.052 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:56:33.056 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:57:03.165 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:57:03.169 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:57:33.245 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:57:33.253 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:58:03.327 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:58:03.333 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:58:33.438 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:58:33.505 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:59:03.582 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:59:03.611 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:59:33.767 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T18:59:33.789 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:00:03.892 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:00:03.905 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:00:34.068 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:00:34.075 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:01:04.237 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:01:04.242 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:01:34.535 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:01:34.541 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:02:04.622 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:02:04.627 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:02:34.777 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:02:34.781 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:03:04.890 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:03:04.893 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:03:35.057 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:03:35.062 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:04:05.209 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:04:05.217 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:04:35.358 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:04:35.362 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:05:05.486 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:05:05.490 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:05:35.659 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:05:35.664 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:06:05.872 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:06:05.876 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:06:35.993 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:06:35.996 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:07:06.073 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:07:06.077 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:07:36.138 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:07:36.144 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:08:06.212 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:08:06.218 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:08:36.281 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:08:36.286 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:09:06.357 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:09:06.363 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:09:36.425 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:09:36.430 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:10:06.480 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:10:06.484 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:10:36.557 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:10:36.570 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:11:06.699 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:11:06.719 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:11:36.779 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:11:36.783 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:12:06.844 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:12:06.862 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:12:36.935 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:12:36.940 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:13:07.019 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:13:07.024 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:13:37.096 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:13:37.100 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:14:07.168 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:14:07.173 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:14:37.243 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:14:37.248 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:15:07.343 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:15:07.348 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:15:37.493 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:15:37.496 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:16:07.631 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:16:07.635 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:16:37.728 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:16:37.732 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:17:07.884 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:17:07.888 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:17:38.006 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:17:38.010 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:18:08.320 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:18:08.324 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:18:38.400 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:18:38.405 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:19:08.483 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:19:08.488 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:19:38.610 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:19:38.615 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:20:08.749 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:20:08.753 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:20:38.896 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:20:38.903 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:21:08.986 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:21:09.002 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:21:39.132 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:21:39.137 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:22:09.196 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:22:09.205 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:22:39.274 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:22:39.301 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:23:09.387 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:23:09.394 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:23:39.526 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:23:39.533 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:24:09.639 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:24:09.643 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:24:39.786 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:24:39.790 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:25:09.846 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:25:09.851 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:25:39.987 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:25:39.993 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:26:10.083 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:26:10.087 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:26:40.305 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:26:40.311 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:27:10.368 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:27:10.372 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:27:40.465 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:27:40.469 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:28:10.629 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:28:10.634 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:28:40.759 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:28:40.763 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:29:10.832 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:29:10.836 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:29:40.979 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:29:40.983 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:30:11.143 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:30:11.147 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:30:41.305 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:30:41.312 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:31:11.678 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:31:11.682 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:31:41.741 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:31:41.744 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:32:11.890 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:32:11.894 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:32:42.044 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:32:42.048 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:33:12.164 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:33:12.169 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:33:42.319 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:33:42.324 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:34:12.384 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:34:12.388 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:34:42.557 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:34:42.572 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:35:12.704 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:35:12.707 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:35:42.874 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:35:42.878 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:36:13.005 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:36:13.010 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:36:43.119 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:36:43.123 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:37:13.241 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:37:13.245 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:37:43.373 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:37:43.377 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:38:13.493 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:38:13.498 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:38:43.632 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:38:43.636 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:39:13.729 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:39:13.734 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:39:43.869 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:39:43.874 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:40:13.982 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:40:13.987 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:40:44.112 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:40:44.116 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:41:14.176 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:41:14.181 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:41:44.316 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:41:44.320 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:42:14.382 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:42:14.385 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:42:44.472 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:42:44.476 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:43:14.542 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:43:14.547 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:43:44.694 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:43:44.700 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:44:14.782 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:44:14.786 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:44:44.945 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:44:44.955 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:45:15.034 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:45:15.037 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:45:45.169 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:45:45.173 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:46:15.253 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:46:15.258 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:46:45.342 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:46:45.347 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:47:15.440 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:47:15.444 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:47:45.578 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:47:45.583 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:48:15.669 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:48:15.680 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:48:45.749 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:48:45.753 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:49:15.828 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:49:15.844 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:49:45.998 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:49:46.004 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:50:16.129 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:50:16.135 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:50:46.251 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:50:46.256 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:51:16.428 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:51:16.439 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:51:46.537 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:51:46.542 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:52:16.600 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:52:16.604 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:52:46.756 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:52:46.760 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:53:16.846 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:53:16.851 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:53:46.919 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:53:46.925 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:54:17.004 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:54:17.013 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:54:47.152 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:54:47.156 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:55:17.222 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:55:17.228 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:55:47.352 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:55:47.358 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:56:17.468 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:56:17.472 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:56:47.663 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:56:47.669 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:57:17.734 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:57:17.739 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:57:47.805 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:57:47.813 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:58:17.894 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:58:17.900 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:58:47.962 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:58:47.967 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:59:18.036 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:59:18.041 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:59:48.115 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T19:59:48.119 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:00:18.196 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:00:18.212 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:00:48.297 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:00:48.301 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:01:18.383 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:01:18.389 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:01:48.523 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:01:48.530 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:02:18.595 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:02:18.602 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:02:48.746 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:02:48.753 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:03:18.877 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:03:18.894 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:03:49.039 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:03:49.043 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:04:19.201 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:04:19.205 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:04:49.323 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:04:49.328 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:05:19.437 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:05:19.442 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:05:49.509 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:05:49.513 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:06:19.631 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:06:19.635 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:06:49.744 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:06:49.748 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:07:19.813 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:07:19.817 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:07:49.881 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:07:49.885 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:08:19.951 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:08:19.955 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:08:50.099 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:08:50.103 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:09:20.321 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:09:20.329 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:09:50.426 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:09:50.430 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:10:20.565 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:10:20.570 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:10:50.654 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:10:50.658 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:11:20.792 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:11:20.797 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:11:50.949 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:11:50.953 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:12:21.086 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:12:21.090 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:12:51.200 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:12:51.204 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:13:21.349 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:13:21.352 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:13:51.437 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:13:51.441 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:14:21.509 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:14:21.514 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:14:51.601 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:14:51.606 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:15:21.665 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:15:21.673 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:15:51.813 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:15:51.818 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:16:21.923 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:16:21.928 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:16:52.088 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:16:52.092 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:17:22.235 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:17:22.241 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:17:52.349 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:17:52.353 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:18:22.430 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:18:22.435 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:18:52.602 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:18:52.608 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:19:22.682 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:19:22.688 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:19:52.773 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:19:52.792 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:20:22.851 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:20:22.856 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:20:52.923 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:20:52.927 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:21:22.995 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:21:22.999 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:21:53.067 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:21:53.072 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:22:23.147 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:22:23.152 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:22:53.219 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:22:53.223 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:23:23.358 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:23:23.363 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:23:53.476 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:23:53.480 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:24:23.627 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:24:23.630 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:24:53.790 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:24:53.794 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:25:23.929 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:25:23.933 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:25:54.101 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:25:54.105 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:26:24.190 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:26:24.194 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:26:54.369 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:26:54.373 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:27:24.517 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:27:24.521 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:27:54.614 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:27:54.618 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:28:24.744 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:28:24.750 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:28:54.866 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:28:54.872 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:29:25.038 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:29:25.042 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:29:55.202 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:29:55.206 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:30:25.473 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:30:25.479 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:30:55.637 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:30:55.641 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:31:25.702 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:31:25.706 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:31:55.864 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:31:55.868 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:32:26.028 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:32:26.032 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:32:56.158 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:32:56.162 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:33:26.287 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:33:26.291 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:33:56.427 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:33:56.432 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:34:26.556 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:34:26.560 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:34:56.787 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:34:56.790 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:35:26.943 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:35:26.947 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:35:57.015 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:35:57.019 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:36:27.087 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:36:27.091 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:36:57.151 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:36:57.155 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:37:27.216 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:37:27.220 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:37:57.346 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:37:57.350 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:38:27.501 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:38:27.505 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:38:57.640 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:38:57.643 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:39:27.752 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:39:27.756 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:39:57.926 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:39:57.930 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:40:28.056 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:40:28.060 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:40:58.228 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:40:58.232 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:41:28.359 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:41:28.363 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:41:58.498 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:41:58.502 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:42:28.616 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:42:28.620 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:42:58.774 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:42:58.778 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:43:28.929 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:43:28.933 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:43:58.990 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:43:58.994 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:44:29.147 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:44:29.152 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:44:59.261 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:44:59.266 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:45:29.431 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:45:29.436 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:45:59.603 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:45:59.608 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:46:29.743 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:46:29.747 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:46:59.857 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:46:59.861 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:47:30.030 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:47:30.035 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:48:00.118 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:48:00.122 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:48:30.273 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:48:30.278 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:49:00.429 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:49:00.433 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:49:30.567 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:49:30.571 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:50:00.647 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:50:00.651 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:50:30.778 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:50:30.782 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:51:00.866 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:51:00.870 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:51:31.013 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:51:31.018 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:52:01.077 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:52:01.082 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:52:31.149 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:52:31.154 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:53:01.205 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:53:01.209 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:53:31.268 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:53:31.321 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:54:01.406 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:54:01.411 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:54:31.473 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:54:31.478 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:55:01.617 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:55:01.621 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:55:31.731 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:55:31.735 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:56:01.861 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:56:01.866 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:56:31.966 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:56:31.970 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:57:02.094 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:57:02.098 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:57:32.243 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:57:32.247 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:58:02.349 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:58:02.353 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:58:32.496 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:58:32.500 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:59:02.600 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:59:02.605 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:59:32.748 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T20:59:32.752 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:00:02.828 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:00:02.832 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:00:32.883 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:00:32.887 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:01:03.054 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:01:03.058 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:01:33.402 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:01:33.406 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:02:03.465 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:02:03.469 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:02:33.536 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:02:33.540 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:03:03.602 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:03:03.606 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:03:33.673 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:03:33.677 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:04:03.820 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:04:03.824 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:04:33.942 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:04:33.946 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:05:04.089 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:05:04.092 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:05:34.219 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:05:34.223 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:06:04.333 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:06:04.337 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:06:34.422 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:06:34.426 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:07:04.552 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:07:04.556 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:07:34.691 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:07:34.695 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:08:04.821 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:08:04.826 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:08:34.968 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:08:34.972 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:09:05.322 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:09:05.326 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:09:35.402 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:09:35.405 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:10:05.492 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:10:05.495 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:10:35.617 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:10:35.621 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:11:05.744 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:11:05.747 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:11:35.883 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:11:35.887 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:12:05.995 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:12:05.999 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:12:36.143 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:12:36.148 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:13:06.207 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:13:06.211 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:13:36.352 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:13:36.356 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:14:06.483 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:14:06.487 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:14:36.556 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:14:36.560 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:15:06.644 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:15:06.648 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:15:36.708 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:15:36.712 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:16:06.780 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:16:06.784 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:16:36.843 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:16:36.847 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:17:06.922 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:17:06.926 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:17:36.987 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:17:36.991 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:18:07.124 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:18:07.128 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:18:37.230 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:18:37.233 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:19:07.377 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:19:07.381 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:19:37.448 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:19:37.452 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:20:07.545 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:20:07.549 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:20:37.611 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:20:37.615 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:21:07.673 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:21:07.677 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:21:37.737 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:21:37.741 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:22:07.801 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:22:07.805 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:22:37.864 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:22:37.869 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:23:07.944 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:23:07.947 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:23:38.016 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:23:38.020 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:24:08.088 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:24:08.092 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:24:38.151 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:24:38.155 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:25:08.234 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:25:08.238 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:25:38.295 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:25:38.299 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:26:08.416 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:26:08.420 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:26:38.537 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:26:38.541 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:27:08.692 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:27:08.696 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:27:38.825 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:27:38.829 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:28:08.970 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:28:08.974 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:28:39.110 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:28:39.114 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:29:09.240 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:29:09.247 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:29:39.395 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:29:39.399 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:30:09.542 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:30:09.546 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:30:39.715 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:30:39.718 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:31:09.801 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:31:09.804 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:31:39.924 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:31:39.928 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:32:10.069 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:32:10.073 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:32:40.202 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:32:40.206 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:33:10.329 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:33:10.333 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:33:40.459 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:33:40.463 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:34:10.592 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:34:10.598 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:34:40.719 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:34:40.724 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:35:10.892 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:35:10.896 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:35:41.030 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:35:41.034 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:36:11.109 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:36:11.112 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:36:41.247 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:36:41.251 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:37:11.360 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:37:11.364 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:37:41.499 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:37:41.503 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:38:11.587 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:38:11.591 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:38:41.726 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:38:41.730 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:39:11.867 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:39:11.871 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:39:42.186 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:39:42.189 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:40:12.341 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:40:12.344 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:40:42.488 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:40:42.491 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:41:12.626 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:41:12.630 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:41:42.770 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:41:42.774 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:42:12.917 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:42:12.921 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:42:43.067 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:42:43.071 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:43:13.222 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:43:13.226 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:43:43.360 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:43:43.364 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:44:13.540 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:44:13.544 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:44:43.670 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:44:43.674 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:45:13.787 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:45:13.791 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:45:43.942 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:45:43.946 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:46:14.029 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:46:14.032 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:46:44.160 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:46:44.164 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:47:14.281 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:47:14.285 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:47:44.654 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:47:44.658 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:48:14.784 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:48:14.787 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:48:44.904 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:48:44.908 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:49:15.043 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:49:15.047 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:49:45.123 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:49:45.127 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:50:15.269 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:50:15.273 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:50:45.417 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:50:45.420 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:51:15.520 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:51:15.524 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:51:45.651 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:51:45.655 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:52:15.715 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:52:15.718 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:52:45.874 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:52:45.878 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:53:15.968 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:53:15.972 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:53:46.104 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:53:46.108 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:54:16.270 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:54:16.274 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:54:46.409 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:54:46.414 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:55:16.543 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:55:16.547 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:55:46.674 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:55:46.678 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:56:16.745 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:56:16.749 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:56:46.871 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:56:46.875 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:57:16.960 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:57:16.964 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:57:47.093 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:57:47.097 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:58:17.220 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:58:17.224 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:58:47.353 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:58:47.357 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:59:17.474 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:59:17.478 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:59:47.620 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T21:59:47.624 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:00:17.742 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:00:17.746 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:00:47.884 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:00:47.887 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:01:17.986 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:01:17.990 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:01:48.125 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:01:48.129 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:02:18.187 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:02:18.191 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:02:48.309 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:02:48.314 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:03:18.464 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:03:18.468 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:03:48.602 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:03:48.606 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:04:18.732 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:04:18.735 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:04:48.844 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:04:48.848 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:05:18.992 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:05:18.996 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:05:49.056 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:05:49.060 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:06:19.134 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:06:19.138 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:06:49.206 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:06:49.210 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:07:19.269 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:07:19.273 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:07:49.402 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:07:49.406 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:08:19.462 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:08:19.466 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:08:49.595 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:08:49.599 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:09:19.673 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:09:19.677 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:09:49.756 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:09:49.759 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:10:19.841 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:10:19.845 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:10:49.904 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:10:49.908 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:11:19.975 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:11:19.978 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:11:50.114 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:11:50.118 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:12:20.260 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:12:20.263 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:12:50.372 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:12:50.375 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:13:20.485 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:13:20.489 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:13:50.598 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:13:50.602 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:14:20.745 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:14:20.750 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:14:50.808 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:14:50.812 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:15:20.963 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:15:20.967 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:15:51.043 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:15:51.047 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:16:21.191 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:16:21.194 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:16:51.320 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:16:51.324 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:17:21.442 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:17:21.446 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:17:51.539 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:17:51.543 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:18:21.661 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:18:21.666 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:18:51.774 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:18:51.778 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:19:21.922 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:19:21.926 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:19:51.985 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:19:51.988 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:20:22.057 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:20:22.061 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:20:52.121 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:20:52.125 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:21:22.197 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:21:22.201 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:21:52.323 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:21:52.327 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:22:22.391 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:22:22.395 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:22:52.509 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:22:52.514 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:23:22.631 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:23:22.635 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:23:52.698 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:23:52.702 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:24:22.762 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:24:22.765 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:24:52.850 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:24:52.853 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:25:22.998 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:25:23.001 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:25:53.102 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:25:53.106 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:26:23.250 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:26:23.253 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:26:53.310 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:26:53.314 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:27:23.376 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:27:23.379 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:27:53.439 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:27:53.443 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:28:23.518 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:28:23.522 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:28:53.666 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:28:53.670 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:29:23.795 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:29:23.799 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:29:53.993 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:29:53.996 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:30:24.148 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:30:24.152 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:30:54.219 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:30:54.224 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:31:24.376 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:31:24.380 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:31:54.438 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:31:54.442 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:32:24.577 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:32:24.581 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:32:54.699 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:32:54.703 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:33:24.837 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:33:24.842 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:33:54.897 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:33:54.901 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:34:24.954 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:34:24.958 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:34:55.024 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:34:55.028 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:35:25.087 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:35:25.091 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:35:55.180 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:35:55.184 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:36:25.237 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:36:25.241 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:36:55.376 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:36:55.380 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:37:25.532 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:37:25.536 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:37:55.662 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:37:55.666 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:38:25.725 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:38:25.729 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:38:55.870 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:38:55.874 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:39:26.209 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:39:26.212 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:39:56.281 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:39:56.284 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:40:26.437 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:40:26.440 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:40:56.594 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:40:56.599 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:41:26.730 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:41:26.733 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:41:56.892 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:41:56.896 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:42:27.058 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:42:27.062 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:42:57.122 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:42:57.126 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:43:27.275 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:43:27.279 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:43:57.362 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:43:57.366 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:44:27.485 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:44:27.489 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:44:57.624 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:44:57.628 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:45:27.779 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:45:27.783 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:45:57.927 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:45:57.930 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:46:28.081 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:46:28.085 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:46:58.245 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:46:58.248 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:47:28.394 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:47:28.397 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:47:58.455 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:47:58.458 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:48:28.592 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:48:28.596 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:48:58.731 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:48:58.735 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:49:28.871 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:49:28.875 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:49:59.001 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:49:59.005 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:50:29.131 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:50:29.135 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:50:59.279 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:50:59.283 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:51:29.410 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:51:29.414 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:51:59.551 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:51:59.555 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:52:29.695 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:52:29.699 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:52:59.794 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:52:59.798 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:53:29.949 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:53:29.953 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:54:00.046 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:54:00.050 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:54:30.199 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:54:30.203 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:55:00.346 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:55:00.350 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:55:30.485 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:55:30.489 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:56:00.623 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:56:00.627 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:56:30.703 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:56:30.707 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:57:00.934 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:57:00.938 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:57:31.071 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:57:31.075 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:58:01.201 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:58:01.205 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:58:31.340 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:58:31.344 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:59:01.428 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:59:01.431 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:59:31.549 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T22:59:31.553 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:00:01.654 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:00:01.658 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:00:31.741 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:00:31.746 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:01:01.941 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:01:01.945 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:01:32.010 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:01:32.014 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:02:02.199 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:02:02.203 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:02:32.330 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:02:32.334 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:03:02.459 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:03:02.463 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:03:32.631 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:03:32.634 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:04:02.778 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:04:02.782 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:04:32.905 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:04:32.909 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:05:03.052 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:05:03.056 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:05:33.192 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:05:33.195 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:06:03.329 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:06:03.334 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:06:33.434 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:06:33.438 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:07:03.573 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:07:03.577 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:07:33.711 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:07:33.715 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:08:03.816 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:08:03.819 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:08:33.954 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:08:33.958 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:09:04.033 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:09:04.038 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:09:34.164 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:09:34.168 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:10:04.322 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:10:04.326 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:10:34.439 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:10:34.443 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:11:04.561 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:11:04.564 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:11:34.691 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:11:34.694 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:12:04.832 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:12:04.836 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:12:34.892 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:12:34.897 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:13:04.970 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:13:04.974 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:13:35.042 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:13:35.046 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:14:05.106 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:14:05.110 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:14:35.195 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:14:35.199 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:15:05.265 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:15:05.269 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:15:35.330 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:15:35.334 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:16:05.435 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:16:05.438 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:16:35.505 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:16:35.509 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:17:05.562 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:17:05.566 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:17:35.648 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:17:35.652 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:18:05.712 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:18:05.716 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:18:35.770 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:18:35.774 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:19:05.830 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:19:05.833 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:19:35.906 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:19:35.909 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:20:06.064 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:20:06.068 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:20:36.182 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:20:36.186 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:21:06.449 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:21:06.453 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:21:36.536 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:21:36.540 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:22:06.684 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:22:06.688 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:22:36.738 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:22:36.742 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:23:06.802 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:23:06.806 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:23:36.882 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:23:36.886 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:24:06.946 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:24:06.949 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:24:37.009 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:24:37.012 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:25:07.072 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:25:07.076 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:25:37.136 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:25:37.139 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:26:07.331 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:26:07.335 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:26:37.389 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:26:37.393 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:27:07.452 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:27:07.456 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:27:37.521 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:27:37.526 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:28:07.597 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:28:07.601 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:28:37.693 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:28:37.697 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:29:07.806 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:29:07.810 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:29:37.945 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:29:37.949 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:30:08.092 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:30:08.096 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:30:38.155 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:30:38.159 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:31:08.302 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:31:08.306 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:31:38.383 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:31:38.406 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:32:08.530 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:32:08.534 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:32:38.667 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:32:38.672 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:33:08.748 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:33:08.752 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:33:38.895 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:33:38.899 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:34:08.999 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:34:09.003 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:34:39.088 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:34:39.093 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:35:09.155 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:35:09.158 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:35:39.218 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:35:39.222 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:36:09.296 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:36:09.299 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:36:39.362 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:36:39.365 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:37:09.451 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:37:09.455 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:37:39.513 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:37:39.517 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:38:09.586 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:38:09.589 DEBUG:teuthology.orchestra.run.smithi202:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-27T23:38:26.617 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2022-06-27T23:38:26.620 DEBUG:teuthology.task.console_log:Killing console logger for smithi195 2022-06-27T23:38:26.621 DEBUG:teuthology.task.console_log:Killing console logger for smithi202 2022-06-27T23:38:26.622 DEBUG:teuthology.exit:Finished running handlers